.. _sec_custom_advancedhpo: Getting started with Advanced HPO Algorithms ============================================ Loading libraries ----------------- .. code:: python # Basic utils for folder manipulations etc import time import multiprocessing # to count the number of CPUs available # External tools to load and process data import numpy as np import pandas as pd # MXNet (NeuralNets) import mxnet as mx from mxnet import gluon, autograd from mxnet.gluon import nn # AutoGluon and HPO tools import autogluon as ag from autogluon.utils import load_and_split_openml_data Check the version of MxNet, you should be fine with version >= 1.5 .. code:: python mx.__version__ .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) .. parsed-literal:: :class: output '1.7.0' You can also check the version of AutoGluon and the specific commit and check that it matches what you want. .. code:: python ag.__version__ .. parsed-literal:: :class: output '0.0.13b20200814' Hyperparameter Optimization of a 2-layer MLP -------------------------------------------- Setting up the context ~~~~~~~~~~~~~~~~~~~~~~ Here we declare a few "environment variables" setting the context for what we're doing .. code:: python OPENML_TASK_ID = 6 # describes the problem we will tackle RATIO_TRAIN_VALID = 0.33 # split of the training data used for validation RESOURCE_ATTR_NAME = 'epoch' # how do we measure resources (will become clearer further) REWARD_ATTR_NAME = 'objective' # how do we measure performance (will become clearer further) NUM_CPUS = multiprocessing.cpu_count() Preparing the data ~~~~~~~~~~~~~~~~~~ We will use a multi-way classification task from OpenML. Data preparation includes: - Missing values are imputed, using the 'mean' strategy of ``sklearn.impute.SimpleImputer`` - Split training set into training and validation - Standardize inputs to mean 0, variance 1 .. code:: python X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data( OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False) n_classes .. parsed-literal:: :class: output Downloading ./org/openml/www/datasets/6/dataset.arff from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.arff... 100%|██████████| 704/704 [00:00<00:00, 37142.48KB/s] Downloading ./org/openml/www/datasets/6/dataset.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.pkl.py3... 100%|██████████| 2521/2521 [00:00<00:00, 40785.33KB/s] Downloading ./org/openml/www/datasets/6/description.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/description.xml... 3KB [00:00, 4211.15KB/s] Downloading ./org/openml/www/datasets/6/features.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/features.xml... 8KB [00:00, 8390.71KB/s] Downloading ./org/openml/www/datasets/6/qualities.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/qualities.xml... 15KB [00:00, 15130.97KB/s] Downloading ./org/openml/www/tasks/6/datasplits.arff from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.arff... 2998KB [00:00, 9384.87KB/s] Downloading ./org/openml/www/tasks/6/datasplits.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.pkl.py3... 881KB [00:00, 39661.07KB/s] Downloading ./org/openml/www/tasks/6/task.xml from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/task.xml... 3KB [00:00, 3988.24KB/s] Data pickle file already exists and is up to date. Data pickle file already exists and is up to date. .. parsed-literal:: :class: output 26 The problem has 26 classes. Declaring a model specifying a hyperparameter space with AutoGluon ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Two layer MLP where we optimize over: - the number of units on the first layer - the number of units on the second layer - the dropout rate after each layer - the learning rate - the scaling - the ``@ag.args`` decorator allows us to specify the space we will optimize over, this matches the `ConfigSpace `__ syntax The body of the function ``run_mlp_openml`` is pretty simple: - it reads the hyperparameters given via the decorator - it defines a 2 layer MLP with dropout - it declares a trainer with the 'adam' loss function and a provided learning rate - it trains the NN with a number of epochs (most of that is boilerplate code from ``mxnet``) - the ``reporter`` at the end is used to keep track of training history in the hyperparameter optimization **Note**: The number of epochs and the hyperparameter space are reduced to make for a shorter experiment .. code:: python @ag.args(n_units_1=ag.space.Int(lower=16, upper=128), n_units_2=ag.space.Int(lower=16, upper=128), dropout_1=ag.space.Real(lower=0, upper=.75), dropout_2=ag.space.Real(lower=0, upper=.75), learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True), batch_size=ag.space.Int(lower=8, upper=128), scale_1=ag.space.Real(lower=0.001, upper=10, log=True), scale_2=ag.space.Real(lower=0.001, upper=10, log=True), epochs=9) def run_mlp_openml(args, reporter, **kwargs): # Time stamp for elapsed_time ts_start = time.time() # Unwrap hyperparameters n_units_1 = args.n_units_1 n_units_2 = args.n_units_2 dropout_1 = args.dropout_1 dropout_2 = args.dropout_2 scale_1 = args.scale_1 scale_2 = args.scale_2 batch_size = args.batch_size learning_rate = args.learning_rate ctx = mx.cpu() net = nn.Sequential() with net.name_scope(): # Layer 1 net.add(nn.Dense(n_units_1, activation='relu', weight_initializer=mx.initializer.Uniform(scale=scale_1))) # Dropout net.add(gluon.nn.Dropout(dropout_1)) # Layer 2 net.add(nn.Dense(n_units_2, activation='relu', weight_initializer=mx.initializer.Uniform(scale=scale_2))) # Dropout net.add(gluon.nn.Dropout(dropout_2)) # Output net.add(nn.Dense(n_classes)) net.initialize(ctx=ctx) trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate': learning_rate}) for epoch in range(args.epochs): ts_epoch = time.time() train_iter = mx.io.NDArrayIter( data={'data': X_train}, label={'label': y_train}, batch_size=batch_size, shuffle=True) valid_iter = mx.io.NDArrayIter( data={'data': X_valid}, label={'label': y_valid}, batch_size=batch_size, shuffle=False) metric = mx.metric.Accuracy() loss = gluon.loss.SoftmaxCrossEntropyLoss() for batch in train_iter: data = batch.data[0].as_in_context(ctx) label = batch.label[0].as_in_context(ctx) with autograd.record(): output = net(data) L = loss(output, label) L.backward() trainer.step(data.shape[0]) metric.update([label], [output]) name, train_acc = metric.get() metric = mx.metric.Accuracy() for batch in valid_iter: data = batch.data[0].as_in_context(ctx) label = batch.label[0].as_in_context(ctx) output = net(data) metric.update([label], [output]) name, val_acc = metric.get() print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % ( epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc)) ts_now = time.time() eval_time = ts_now - ts_epoch elapsed_time = ts_now - ts_start # The resource reported back (as 'epoch') is the number of epochs # done, starting at 1 reporter( epoch=epoch + 1, objective=float(val_acc), eval_time=eval_time, time_step=ts_now, elapsed_time=elapsed_time) .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) **Note**: The annotation ``epochs=9`` specifies the maximum number of epochs for training. It becomes available as ``args.epochs``. Importantly, it is also processed by ``HyperbandScheduler`` below in order to set its ``max_t`` attribute. **Recommendation**: Whenever writing training code to be passed as ``train_fn`` to a scheduler, if this training code reports a resource (or time) attribute, the corresponding maximum resource value should be included in ``train_fn.args``: - If the resource attribute (``time_attr`` of scheduler) in ``train_fn`` is ``epoch``, make sure to include ``epochs=XYZ`` in the annotation. This allows the scheduler to read ``max_t`` from ``train_fn.args.epochs``. This case corresponds to our example here. - If the resource attribute is something else than ``epoch``, you can also include the annotation ``max_t=XYZ``, which allows the scheduler to read ``max_t`` from ``train_fn.args.max_t``. Annotating the training function by the correct value for ``max_t`` simplifies scheduler creation (since ``max_t`` does not have to be passed), and avoids inconsistencies between ``train_fn`` and the scheduler. Running the Hyperparameter Optimization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can use the following schedulers: - FIFO (``fifo``) - Hyperband (either the stopping (``hbs``) or promotion (``hbp``) variant) And the following searchers: - Random search (``random``) - Gaussian process based Bayesian optimization (``bayesopt``) - SkOpt Bayesian optimization (``skopt``; only with FIFO scheduler) Note that the method known as (asynchronous) Hyperband is using random search. Combining Hyperband scheduling with the ``bayesopt`` searcher uses a novel method called asynchronous BOHB. Pick the combination you're interested in (doing the full experiment takes around 120 seconds, see the ``time_out`` parameter), running everything with multiple runs can take a fair bit of time. In real life, you will want to choose a larger ``time_out`` in order to obtain good performance. .. code:: python SCHEDULER = "hbs" SEARCHER = "bayesopt" .. code:: python def compute_error(df): return 1.0 - df["objective"] def compute_runtime(df, start_timestamp): return df["time_step"] - start_timestamp def process_training_history(task_dicts, start_timestamp, runtime_fn=compute_runtime, error_fn=compute_error): task_dfs = [] for task_id in task_dicts: task_df = pd.DataFrame(task_dicts[task_id]) task_df = task_df.assign(task_id=task_id, runtime=runtime_fn(task_df, start_timestamp), error=error_fn(task_df), target_epoch=task_df["epoch"].iloc[-1]) task_dfs.append(task_df) result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True) # re-order by runtime result = result.sort_values(by="runtime") # calculate incumbent best -- the cumulative minimum of the error. result = result.assign(best=result["error"].cummin()) return result resources = dict(num_cpus=NUM_CPUS, num_gpus=0) .. code:: python search_options = { 'num_init_random': 2, 'debug_log': True} if SCHEDULER == 'fifo': myscheduler = ag.scheduler.FIFOScheduler( run_mlp_openml, resource=resources, searcher=SEARCHER, search_options=search_options, time_out=120, time_attr=RESOURCE_ATTR_NAME, reward_attr=REWARD_ATTR_NAME) else: # This setup uses rung levels at 1, 3, 9 epochs. We just use a single # bracket, so this is in fact successive halving (Hyperband would use # more than 1 bracket). # Also note that since we do not use the max_t argument of # HyperbandScheduler, this value is obtained from train_fn.args.epochs. sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion' myscheduler = ag.scheduler.HyperbandScheduler( run_mlp_openml, resource=resources, searcher=SEARCHER, search_options=search_options, time_out=120, time_attr=RESOURCE_ATTR_NAME, reward_attr=REWARD_ATTR_NAME, type=sch_type, grace_period=1, reduction_factor=3, brackets=1) # run tasks myscheduler.run() myscheduler.join_jobs() results_df = process_training_history( myscheduler.training_history.copy(), start_timestamp=myscheduler._start_time) .. parsed-literal:: :class: output max_t = 9, as inferred from train_fn.args scheduler_options: Key 'resume': Imputing default value False scheduler_options: Key 'keep_size_ratios': Imputing default value False scheduler_options: Key 'maxt_pending': Imputing default value False scheduler_options: Key 'searcher_data': Imputing default value rungs scheduler_options: Key 'do_snapshots': Imputing default value False scheduler_options: Key 'visualizer': Imputing default value none scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60 scheduler_options: Key 'delay_get_config': Imputing default value True search_options: Key 'random_seed': Imputing default value 31415927 search_options: Key 'opt_skip_init_length': Imputing default value 150 search_options: Key 'opt_skip_period': Imputing default value 1 search_options: Key 'profiler': Imputing default value False search_options: Key 'opt_maxiter': Imputing default value 50 search_options: Key 'opt_nstarts': Imputing default value 2 search_options: Key 'opt_warmstart': Imputing default value False search_options: Key 'opt_verbose': Imputing default value False search_options: Key 'opt_debug_writer': Imputing default value False search_options: Key 'num_fantasy_samples': Imputing default value 20 search_options: Key 'num_init_candidates': Imputing default value 250 search_options: Key 'initial_scoring': Imputing default value thompson_indep search_options: Key 'first_is_default': Imputing default value True search_options: Key 'opt_skip_num_max_resource': Imputing default value False search_options: Key 'gp_resource_kernel': Imputing default value matern52 search_options: Key 'resource_acq': Imputing default value bohb [GPMultiFidelitySearcher.__init__] - acquisition_class = - local_minimizer_class = - num_initial_candidates = 250 - num_initial_random_choices = 2 - initial_scoring = thompson_indep - first_is_default = True Starting Experiments Num of Finished Tasks is 0 Time out (secs) is 120 Starting get_config[random] for config_id 0 Start with default config: {'batch_size': 68, 'dropout_1': 0.375, 'dropout_2': 0.375, 'learning_rate': 0.001, 'n_units_1': 72, 'n_units_2': 72, 'scale_1': 0.1, 'scale_2': 0.1} [0: random] batch_size: 68 dropout_1: 0.375 dropout_2: 0.375 learning_rate: 0.001 n_units_1: 72 n_units_2: 72 scale_1: 0.1 scale_2: 0.1 /var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/distributed/worker.py:3385: UserWarning: Large object of size 1.30 MB detected in task graph: (, {'ar ... sReporter}, []) Consider scattering large objects ahead of time with client.scatter to reduce scheduler burden and keep data on workers future = client.submit(func, big_data) # bad big_future = client.scatter(big_data) # good future = client.submit(func, big_future) # good % (format_bytes(len(b)), s) .. parsed-literal:: :class: output Epoch 1 ; Time: 0.451235 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250 .. parsed-literal:: :class: output Update for config_id 0:1: reward = 0.53125, crit_val = 0.46875 config_id 0: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.860013 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247 Epoch 3 ; Time: 1.263851 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686 .. parsed-literal:: :class: output Update for config_id 0:3: reward = 0.6946858288770054, crit_val = 0.30531417112299464 config_id 0: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.667614 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063 Epoch 5 ; Time: 2.075626 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939 Epoch 6 ; Time: 2.480307 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321 Epoch 7 ; Time: 2.877022 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501 Epoch 8 ; Time: 3.282080 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202 Epoch 9 ; Time: 3.684766 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043 .. parsed-literal:: :class: output config_id 0: Terminating evaluation at 9 Update for config_id 0:9: reward = 0.766042780748663, crit_val = 0.23395721925133695 Starting get_config[random] for config_id 1 [1: random] batch_size: 49 dropout_1: 0.6118880110946948 dropout_2: 0.5692033531564655 learning_rate: 0.00010310483923558212 n_units_1: 126 n_units_2: 93 scale_1: 0.17664689253248303 scale_2: 0.5396711634456218 .. parsed-literal:: :class: output Epoch 1 ; Time: 0.617862 ; Training: accuracy=0.048005 ; Validation: accuracy=0.111743 .. parsed-literal:: :class: output config_id 1: Terminating evaluation at 1 Update for config_id 1:1: reward = 0.11174305787888926, crit_val = 0.8882569421211107 Starting get_config[BO] for config_id 2 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4740695831238606 - self.std = 0.25382661537049356 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 4 Current best is [0.46875] [2: BO] (1 evaluations) batch_size: 50 dropout_1: 0.21593234798980557 dropout_2: 0.2842099148996821 learning_rate: 0.12426792150054146 n_units_1: 61 n_units_2: 79 scale_1: 9.791854554777688 scale_2: 4.705654410614348 Started BO from (top scorer): batch_size: 50 dropout_1: 0.21593234798980557 dropout_2: 0.2842099148996821 learning_rate: 0.12426792150054146 n_units_1: 61 n_units_2: 79 scale_1: 9.791854554777688 scale_2: 4.705654410614348 Top score values: [-0.27440679 0.04673507 0.04701008 0.10393977 0.10463477] Labeled: 0:1, 0:3, 0:9, 1:1. Pending: Targets: [-0.02095755 -0.66484522 -0.94597 1.63177277] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.7572453035969371, 'kernel_inv_bw1': 0.9003213708321401, 'kernel_inv_bw2': 0.9671934120293088, 'kernel_inv_bw3': 3.9351561554818266, 'kernel_inv_bw4': 26.15373099715838, 'kernel_inv_bw5': 1.6599386110717227, 'kernel_inv_bw6': 4.378455276759229, 'kernel_inv_bw7': 35.45306859774744, 'kernel_inv_bw8': 1.6274469437660462, 'kernel_covariance_scale': 0.8177941757954876, 'mean_mean_value': 0.3168762072733891} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.616818 ; Training: accuracy=0.072975 ; Validation: accuracy=0.038319 .. parsed-literal:: :class: output config_id 2: Terminating evaluation at 1 Update for config_id 2:1: reward = 0.03831932773109244, crit_val = 0.9616806722689075 Starting get_config[BO] for config_id 3 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.57159180095287 - self.std = 0.2993070203278022 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 5 Current best is [0.46875] [3: BO] (1 evaluations) batch_size: 22 dropout_1: 0.6341443733804044 dropout_2: 0.04086583670464042 learning_rate: 0.12818863950369533 n_units_1: 19 n_units_2: 101 scale_1: 0.026860146662962745 scale_2: 4.431236744525342 Started BO from (top scorer): batch_size: 22 dropout_1: 0.6341443733804044 dropout_2: 0.04086583670464042 learning_rate: 0.12818863950369533 n_units_1: 19 n_units_2: 101 scale_1: 0.026860146662962745 scale_2: 4.431236744525342 Top score values: [-0.02496145 0.00037188 0.04140351 0.12359639 0.12398311] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1. Pending: Targets: [-0.3435997 -0.88964712 -1.12805433 1.05799437 1.30330679] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 6.2975129900938756, 'kernel_inv_bw1': 1.6684536123391513, 'kernel_inv_bw2': 1.817414219491645, 'kernel_inv_bw3': 1.4215981511230344, 'kernel_inv_bw4': 57.091085169764185, 'kernel_inv_bw5': 3.604498367053083, 'kernel_inv_bw6': 100.00000000000004, 'kernel_inv_bw7': 99.92474107423274, 'kernel_inv_bw8': 1.3393578509128972, 'kernel_covariance_scale': 0.7384716747026111, 'mean_mean_value': 0.38060991018158125} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.299096 ; Training: accuracy=0.041812 ; Validation: accuracy=0.039562 .. parsed-literal:: :class: output config_id 3: Terminating evaluation at 1 Update for config_id 3:1: reward = 0.03956228956228956, crit_val = 0.9604377104377104 Starting get_config[BO] for config_id 4 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6363994525336767 - self.std = 0.30927991991886417 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 6 Current best is [0.46875] [4: BO] (6 evaluations) batch_size: 110 dropout_1: 0.5744977554712186 dropout_2: 0.7047747941055489 learning_rate: 3.6223201928237886e-05 n_units_1: 92 n_units_2: 92 scale_1: 0.06664854483482407 scale_2: 0.022478620206736768 Started BO from (top scorer): batch_size: 110 dropout_1: 0.5744977553473402 dropout_2: 0.7047748951791846 learning_rate: 3.6221732543522435e-05 n_units_1: 92 n_units_2: 92 scale_1: 0.06664854527483652 scale_2: 0.017643835128767108 Top score values: [-0.02863203 0.00599656 0.042477 0.05395405 0.06651693] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1. Pending: Targets: [-0.54206381 -1.07050364 -1.30122328 0.81433508 1.05173727 1.04771838] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0001511285849645314, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.001199968812262379, 'kernel_inv_bw3': 0.00926838083535284, 'kernel_inv_bw4': 0.00010293036281036059, 'kernel_inv_bw5': 0.00017461620414159357, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 4.272493402708928, 'kernel_inv_bw8': 1.3164718962359343, 'kernel_covariance_scale': 0.7976260528462709, 'mean_mean_value': 0.023357588254215324} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.319956 ; Training: accuracy=0.042066 ; Validation: accuracy=0.109933 .. parsed-literal:: :class: output config_id 4: Terminating evaluation at 1 Update for config_id 4:1: reward = 0.10993265993265994, crit_val = 0.89006734006734 Starting get_config[BO] for config_id 5 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6726377221813429 - self.std = 0.2997807115320194 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 7 Current best is [0.46875] [5: BO] (8 evaluations) batch_size: 106 dropout_1: 0.38750926166961597 dropout_2: 0.04489600208822606 learning_rate: 0.0026794725095178663 n_units_1: 93 n_units_2: 20 scale_1: 0.48508438196776654 scale_2: 0.03480880867048884 Started BO from (top scorer): batch_size: 106 dropout_1: 0.38750926119776946 dropout_2: 0.04489600246106559 learning_rate: 0.002265665895170696 n_units_1: 93 n_units_2: 20 scale_1: 0.4850843918016997 scale_2: 0.03481291816391773 Top score values: [0.18489824 0.20828588 0.23691933 0.2416328 0.24992744] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1. Pending: Targets: [-0.68012288 -1.22530749 -1.46333799 0.71925648 0.96418128 0.96003504 0.72529556] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00011518068128664455, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.555364761942682, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00017086339060469206, 'kernel_inv_bw7': 0.035729865892209396, 'kernel_inv_bw8': 1.3111759816687758, 'kernel_covariance_scale': 0.6932464484239275, 'mean_mean_value': 0.1222573690401266} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.347613 ; Training: accuracy=0.355842 ; Validation: accuracy=0.601787 .. parsed-literal:: :class: output Update for config_id 5:1: reward = 0.6017874875868917, crit_val = 0.3982125124131083 config_id 5: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.618149 ; Training: accuracy=0.574892 ; Validation: accuracy=0.681397 Epoch 3 ; Time: 0.879449 ; Training: accuracy=0.622559 ; Validation: accuracy=0.727574 .. parsed-literal:: :class: output Update for config_id 5:3: reward = 0.7275736511089044, crit_val = 0.2724263488910956 config_id 5: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.142586 ; Training: accuracy=0.648047 ; Validation: accuracy=0.750083 Epoch 5 ; Time: 1.409948 ; Training: accuracy=0.672459 ; Validation: accuracy=0.760344 Epoch 6 ; Time: 1.672125 ; Training: accuracy=0.684542 ; Validation: accuracy=0.766137 Epoch 7 ; Time: 1.934993 ; Training: accuracy=0.701258 ; Validation: accuracy=0.792453 Epoch 8 ; Time: 2.192502 ; Training: accuracy=0.709782 ; Validation: accuracy=0.798080 Epoch 9 ; Time: 2.460412 ; Training: accuracy=0.713009 ; Validation: accuracy=0.812479 .. parsed-literal:: :class: output config_id 5: Terminating evaluation at 9 Update for config_id 5:9: reward = 0.8124793114862628, crit_val = 0.18752068851373715 Starting get_config[BO] for config_id 6 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5566623605087341 - self.std = 0.3107077976680413 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 10 Current best is [0.39821251] [6: BO] (8 evaluations) batch_size: 100 dropout_1: 0.4870263486471881 dropout_2: 0.59113405302611 learning_rate: 0.0051374723840914805 n_units_1: 105 n_units_2: 107 scale_1: 0.2432746016644627 scale_2: 3.384577528513164 Started BO from (top scorer): batch_size: 100 dropout_1: 0.4870263426644337 dropout_2: 0.5911340539952046 learning_rate: 0.009911435239249514 n_units_1: 105 n_units_2: 107 scale_1: 0.24327462238842057 scale_2: 3.38457748137845 Top score values: [0.19986271 0.27163002 0.37278358 0.37794283 0.37953738] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9. Pending: Targets: [-0.28294224 -0.80895359 -1.03861295 1.06722324 1.30353443 1.29953401 1.07304993 -0.50996418 -0.91480167 -1.18806697] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0005635684715621877, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.720114182169939, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.002059715378214448, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.1165087337394366, 'kernel_covariance_scale': 0.6850617791776692, 'mean_mean_value': 0.3838207647871803} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.430025 ; Training: accuracy=0.281405 ; Validation: accuracy=0.649833 .. parsed-literal:: :class: output Update for config_id 6:1: reward = 0.6498333333333334, crit_val = 0.3501666666666666 config_id 6: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.705111 ; Training: accuracy=0.446942 ; Validation: accuracy=0.693167 Epoch 3 ; Time: 0.973286 ; Training: accuracy=0.491240 ; Validation: accuracy=0.712833 .. parsed-literal:: :class: output config_id 6: Terminating evaluation at 3 Update for config_id 6:3: reward = 0.7128333333333333, crit_val = 0.2871666666666667 Starting get_config[BO] for config_id 7 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5169964115350562 - self.std = 0.297458835498879 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 12 Current best is [0.35016667] [7: BO] (5 evaluations) batch_size: 70 dropout_1: 0.6131746934906158 dropout_2: 0.022967783090604488 learning_rate: 0.008313446421825743 n_units_1: 24 n_units_2: 99 scale_1: 0.0028554704441241295 scale_2: 0.011971504396868799 Started BO from (top scorer): batch_size: 70 dropout_1: 0.6131746914586562 dropout_2: 0.022967791041196395 learning_rate: 0.007721593613613116 n_units_1: 24 n_units_2: 99 scale_1: 0.0028554705455487786 scale_2: 0.011971504865511351 Top score values: [0.25898139 0.30766019 0.30984522 0.31771857 0.32951799] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3. Pending: Targets: [-0.16219525 -0.71163541 -0.9515239 1.24810725 1.49494386 1.49076526 1.25419347 -0.39932886 -0.82219801 -1.10763468 -0.56084986 -0.77264387] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00018498994662793233, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00017831718840405247, 'kernel_inv_bw3': 6.785443295549769, 'kernel_inv_bw4': 0.0008660779309394507, 'kernel_inv_bw5': 0.0002996293975555951, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00015743086560692838, 'kernel_inv_bw8': 1.1136085510227574, 'kernel_covariance_scale': 0.6935746305208051, 'mean_mean_value': 0.45824701788781513} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.523046 ; Training: accuracy=0.271263 ; Validation: accuracy=0.543697 .. parsed-literal:: :class: output Update for config_id 7:1: reward = 0.5436974789915966, crit_val = 0.4563025210084034 config_id 7: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.929654 ; Training: accuracy=0.386210 ; Validation: accuracy=0.577479 Epoch 3 ; Time: 1.331224 ; Training: accuracy=0.424030 ; Validation: accuracy=0.582857 .. parsed-literal:: :class: output config_id 7: Terminating evaluation at 3 Update for config_id 7:3: reward = 0.5828571428571429, crit_val = 0.41714285714285715 Starting get_config[BO] for config_id 8 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5055287368979954 - self.std = 0.27692114408851204 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 14 Current best is [0.35016667] [8: BO] (3 evaluations) batch_size: 112 dropout_1: 0.41258052357180974 dropout_2: 0.07535360975884835 learning_rate: 1.0000000000000004e-06 n_units_1: 40 n_units_2: 33 scale_1: 0.1918197311813462 scale_2: 0.10218151730817446 Started BO from (top scorer): batch_size: 112 dropout_1: 0.4125805242468248 dropout_2: 0.07535360980344002 learning_rate: 2.48539902503145e-06 n_units_1: 40 n_units_2: 33 scale_1: 0.1918197310941 scale_2: 0.1021815173022151 Top score values: [0.30319667 0.30627468 0.31694949 0.33985494 0.34255644] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3. Pending: Targets: [-0.13281303 -0.72300209 -0.98068177 1.38208372 1.64722682 1.64273831 1.38862131 -0.38753352 -0.8417645 -1.14837041 -0.56103361 -0.7885352 -0.17776258 -0.31917346] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.0006690012947653791, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.764899094740552, 'kernel_inv_bw4': 0.0005796463402060587, 'kernel_inv_bw5': 0.000684237942280471, 'kernel_inv_bw6': 0.00028425765435826007, 'kernel_inv_bw7': 0.0001024953751392057, 'kernel_inv_bw8': 1.0593295856674674, 'kernel_covariance_scale': 0.7444010436536751, 'mean_mean_value': 0.6096884310053552} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.330547 ; Training: accuracy=0.037616 ; Validation: accuracy=0.032738 .. parsed-literal:: :class: output config_id 8: Terminating evaluation at 1 Update for config_id 8:1: reward = 0.03273809523809524, crit_val = 0.9672619047619048 Starting get_config[BO] for config_id 9 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5363109480889227 - self.std = 0.29127064516524426 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 15 Current best is [0.35016667] [9: BO] (8 evaluations) batch_size: 16 dropout_1: 0.49382843382784947 dropout_2: 0.17577032493560404 learning_rate: 0.004241709322261239 n_units_1: 75 n_units_2: 24 scale_1: 0.00112063957179567 scale_2: 0.013201132182141391 Started BO from (top scorer): batch_size: 16 dropout_1: 0.4938284333967552 dropout_2: 0.17577032689555921 learning_rate: 0.0044596526917615315 n_units_1: 75 n_units_2: 24 scale_1: 0.0011206396260318283 scale_2: 0.01320113262596052 Top score values: [0.34243971 0.34413616 0.34417665 0.34515598 0.34564232] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1. Pending: Targets: [-0.23195248 -0.79306576 -1.03805081 1.20831261 1.46039339 1.45612601 1.21452813 -0.47412411 -0.90597732 -1.19747824 -0.6390767 -0.85537038 -0.27468757 -0.40913183 1.47955506] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.678950645138259, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010001133434562772, 'kernel_inv_bw8': 1.0241379286848398, 'kernel_covariance_scale': 0.6939330217554913, 'mean_mean_value': 0.6890140220227132} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.766322 ; Training: accuracy=0.358256 ; Validation: accuracy=0.618112 .. parsed-literal:: :class: output Update for config_id 9:1: reward = 0.618111559139785, crit_val = 0.381888440860215 config_id 9: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 3.471594 ; Training: accuracy=0.533156 ; Validation: accuracy=0.696069 Epoch 3 ; Time: 5.177650 ; Training: accuracy=0.579824 ; Validation: accuracy=0.747144 .. parsed-literal:: :class: output Update for config_id 9:3: reward = 0.7471438172043011, crit_val = 0.2528561827956989 config_id 9: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 6.984765 ; Training: accuracy=0.598558 ; Validation: accuracy=0.758401 Epoch 5 ; Time: 8.700785 ; Training: accuracy=0.627321 ; Validation: accuracy=0.766297 Epoch 6 ; Time: 10.508648 ; Training: accuracy=0.640666 ; Validation: accuracy=0.764785 Epoch 7 ; Time: 12.191849 ; Training: accuracy=0.644977 ; Validation: accuracy=0.788306 Epoch 8 ; Time: 13.899998 ; Training: accuracy=0.654426 ; Validation: accuracy=0.799563 Epoch 9 ; Time: 15.659388 ; Training: accuracy=0.662218 ; Validation: accuracy=0.799395 .. parsed-literal:: :class: output config_id 9: Terminating evaluation at 9 Update for config_id 9:9: reward = 0.7993951612903226, crit_val = 0.20060483870967738 Starting get_config[BO] for config_id 10 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.49333409353885727 - self.std = 0.2844316122764386 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 18 Current best is [0.35016667] [10: BO] (12 evaluations) batch_size: 82 dropout_1: 0.14557280209760065 dropout_2: 0.5231236974598504 learning_rate: 0.0036698312743314717 n_units_1: 40 n_units_2: 101 scale_1: 5.981987507854231 scale_2: 10.0 Started BO from (top scorer): batch_size: 82 dropout_1: 0.14557280856507585 dropout_2: 0.5231231614760518 learning_rate: 0.0033261781877540063 n_units_1: 40 n_units_2: 101 scale_1: 5.981988181743532 scale_2: 6.375505195149974 Top score values: [0.32350358 0.34767498 0.34845628 0.35179445 0.35273396] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9. Pending: Targets: [-0.08643235 -0.66103736 -0.91191296 1.38846328 1.64660522 1.64223524 1.39482824 -0.3344269 -0.77666383 -1.07517376 -0.50334569 -0.72484006 -0.130195 -0.2678719 1.66622763 -0.3918188 -0.8454683 -1.02917272] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0002897017498538562, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0008011959809245235, 'kernel_inv_bw3': 7.974358785073643, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.006837139024467231, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.36100017660804706, 'kernel_inv_bw8': 1.2163207127105384, 'kernel_covariance_scale': 0.6947885806350937, 'mean_mean_value': 0.7327734829861489} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.392405 ; Training: accuracy=0.257004 ; Validation: accuracy=0.498162 .. parsed-literal:: :class: output config_id 10: Terminating evaluation at 1 Update for config_id 10:1: reward = 0.49816237888406284, crit_val = 0.5018376211159372 Starting get_config[BO] for config_id 11 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4937816476218615 - self.std = 0.2768519137992561 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 19 Current best is [0.35016667] [11: BO] (17 evaluations) batch_size: 79 dropout_1: 0.75 dropout_2: 0.42369903081916027 learning_rate: 0.0031743331533078427 n_units_1: 128 n_units_2: 98 scale_1: 0.4942778506833816 scale_2: 5.998936967743114 Started BO from (top scorer): batch_size: 79 dropout_1: 0.6243451100425594 dropout_2: 0.42338274050078417 learning_rate: 0.002139116508757636 n_units_1: 119 n_units_2: 57 scale_1: 0.49430045483873486 scale_2: 5.998675112100447 Top score values: [0.26995698 0.28539949 0.31800563 0.32117917 0.33470418] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1. Pending: Targets: [-0.0904153 -0.68075194 -0.93849605 1.42486028 1.69006968 1.68558005 1.43139951 -0.34519947 -0.79954404 -1.10622663 -0.51874296 -0.74630144 -0.13537608 -0.27682233 1.71022931 -0.40416266 -0.87023225 -1.05896616 0.02909849] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00013384115748560582, 'kernel_inv_bw1': 0.7164481059620007, 'kernel_inv_bw2': 0.007434905102630733, 'kernel_inv_bw3': 5.873920713629177, 'kernel_inv_bw4': 0.29616486376970397, 'kernel_inv_bw5': 0.42909627025008135, 'kernel_inv_bw6': 0.00093129006269352, 'kernel_inv_bw7': 0.0016781228072952883, 'kernel_inv_bw8': 1.0966551096400794, 'kernel_covariance_scale': 0.7390471995681659, 'mean_mean_value': 0.8736689829419473} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.409493 ; Training: accuracy=0.133284 ; Validation: accuracy=0.438208 .. parsed-literal:: :class: output config_id 11: Terminating evaluation at 1 Update for config_id 11:1: reward = 0.4382078614257162, crit_val = 0.5617921385742838 Starting get_config[BO] for config_id 12 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4971821721694826 - self.std = 0.2702486648299593 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 20 Current best is [0.35016667] [12: BO] (18 evaluations) batch_size: 52 dropout_1: 0.4187634264961476 dropout_2: 0.7247258372119212 learning_rate: 0.003204081133297433 n_units_1: 50 n_units_2: 128 scale_1: 0.24304339720298734 scale_2: 1.2689115862202556 Started BO from (top scorer): batch_size: 52 dropout_1: 0.5640354983588307 dropout_2: 0.7247257870729263 learning_rate: 0.0037061027371093794 n_units_1: 50 n_units_2: 114 scale_1: 0.24305486090288597 scale_2: 1.2689345703007509 Top score values: [0.28768416 0.29973203 0.31945222 0.34145763 0.37267305] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1. Pending: Targets: [-0.10520745 -0.70996836 -0.97401019 1.44709233 1.71878185 1.71418252 1.45379134 -0.36621702 -0.83166303 -1.14583909 -0.54400086 -0.77711949 -0.1512668 -0.29616914 1.73943406 -0.42662091 -0.90407843 -1.09742386 0.01722654 0.23907599] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.4571498209951534, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.032423290570723, 'kernel_inv_bw4': 0.001970176820657327, 'kernel_inv_bw5': 0.42480096030193, 'kernel_inv_bw6': 0.000384714205448538, 'kernel_inv_bw7': 0.00028464009509878525, 'kernel_inv_bw8': 1.0773759457871805, 'kernel_covariance_scale': 0.763639790879144, 'mean_mean_value': 1.0197102312028976} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.577715 ; Training: accuracy=0.251243 ; Validation: accuracy=0.619398 .. parsed-literal:: :class: output Update for config_id 12:1: reward = 0.6193979933110368, crit_val = 0.38060200668896316 config_id 12: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.187254 ; Training: accuracy=0.403515 ; Validation: accuracy=0.691639 Epoch 3 ; Time: 1.707295 ; Training: accuracy=0.469164 ; Validation: accuracy=0.708027 .. parsed-literal:: :class: output config_id 12: Terminating evaluation at 3 Update for config_id 12:3: reward = 0.7080267558528428, crit_val = 0.2919732441471572 Starting get_config[BO] for config_id 13 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4825553951920805 - self.std = 0.26213127511599504 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 22 Current best is [0.35016667] [13: BO] (13 evaluations) batch_size: 65 dropout_1: 0.4380660616295446 dropout_2: 0.1932438504773113 learning_rate: 0.00442799054709087 n_units_1: 57 n_units_2: 78 scale_1: 2.497201620763285 scale_2: 0.0075591995777766095 Started BO from (top scorer): batch_size: 65 dropout_1: 0.4169220946335453 dropout_2: 0.1932436849600857 learning_rate: 0.0035468887104147884 n_units_1: 57 n_units_2: 60 scale_1: 2.4972005561729547 scale_2: 0.007559210365656552 Top score values: [0.31963507 0.32597431 0.35312265 0.36624176 0.37228937] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3. Pending: Targets: [-0.05266596 -0.67615443 -0.94837282 1.54770371 1.82780661 1.82306486 1.55461017 -0.32175818 -0.80161761 -1.12552272 -0.50504744 -0.74538503 -0.10015163 -0.24954114 1.84909835 -0.3840326 -0.87627549 -1.07560823 0.07355943 0.30227886 -0.38894019 -0.7270485 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0010532467968579068, 'kernel_inv_bw1': 1.404342815957603, 'kernel_inv_bw2': 0.0005398724164910998, 'kernel_inv_bw3': 4.82418165734747, 'kernel_inv_bw4': 0.0006939553261438888, 'kernel_inv_bw5': 0.5188360353524706, 'kernel_inv_bw6': 0.00012187757927405632, 'kernel_inv_bw7': 0.00026183289101106727, 'kernel_inv_bw8': 1.014056003186076, 'kernel_covariance_scale': 0.796498374050167, 'mean_mean_value': 1.1471307714613548} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.498134 ; Training: accuracy=0.433912 ; Validation: accuracy=0.687124 .. parsed-literal:: :class: output Update for config_id 13:1: reward = 0.687123745819398, crit_val = 0.31287625418060205 config_id 13: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.925793 ; Training: accuracy=0.555335 ; Validation: accuracy=0.736957 Epoch 3 ; Time: 1.362534 ; Training: accuracy=0.593218 ; Validation: accuracy=0.749331 .. parsed-literal:: :class: output Update for config_id 13:3: reward = 0.7493311036789297, crit_val = 0.25066889632107026 config_id 13: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.780590 ; Training: accuracy=0.614888 ; Validation: accuracy=0.776254 Epoch 5 ; Time: 2.204054 ; Training: accuracy=0.628867 ; Validation: accuracy=0.776421 Epoch 6 ; Time: 2.620269 ; Training: accuracy=0.646071 ; Validation: accuracy=0.801505 Epoch 7 ; Time: 3.046810 ; Training: accuracy=0.657155 ; Validation: accuracy=0.821572 Epoch 8 ; Time: 3.506386 ; Training: accuracy=0.657734 ; Validation: accuracy=0.809197 Epoch 9 ; Time: 3.924699 ; Training: accuracy=0.669231 ; Validation: accuracy=0.820903 .. parsed-literal:: :class: output config_id 13: Terminating evaluation at 9 Update for config_id 13:9: reward = 0.8209030100334448, crit_val = 0.1790969899665552 Starting get_config[BO] for config_id 14 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.45435443338776 - self.std = 0.2581820075415383 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 25 Current best is [0.31287626] [14: BO] (16 evaluations) batch_size: 30 dropout_1: 0.3893679882377774 dropout_2: 0.34480399499247283 learning_rate: 0.006353695537618099 n_units_1: 122 n_units_2: 68 scale_1: 5.840391604925039 scale_2: 5.876385115956924 Started BO from (top scorer): batch_size: 30 dropout_1: 0.4222267082483692 dropout_2: 0.34480373634999584 learning_rate: 0.005673378598456798 n_units_1: 122 n_units_2: 93 scale_1: 5.7852734476136485 scale_2: 5.876374329027126 Top score values: [0.32319459 0.32494214 0.3334888 0.33862142 0.35233721] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9. Pending: Targets: [ 0.05575744 -0.5772682 -0.85365056 1.68060708 1.96499455 1.96018027 1.68761918 -0.21745094 -0.70465051 -1.03351023 -0.40354387 -0.64755778 0.0075454 -0.14412924 1.98661199 -0.28067793 -0.7804504 -0.98283222 0.18391362 0.41613165 -0.2856606 -0.62894076 -0.54797846 -0.78892228 -1.0661372 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.425836992553471, 'kernel_inv_bw2': 0.00023623419268202186, 'kernel_inv_bw3': 4.980182657080316, 'kernel_inv_bw4': 0.0011116128852937048, 'kernel_inv_bw5': 0.5922730036428987, 'kernel_inv_bw6': 0.01835734551762398, 'kernel_inv_bw7': 0.00010992267079371953, 'kernel_inv_bw8': 1.2350787994251735, 'kernel_covariance_scale': 0.7894787373295402, 'mean_mean_value': 1.19408549856321} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.000329 ; Training: accuracy=0.208789 ; Validation: accuracy=0.410943 .. parsed-literal:: :class: output config_id 14: Terminating evaluation at 1 Update for config_id 14:1: reward = 0.41094276094276094, crit_val = 0.5890572390572391 Starting get_config[BO] for config_id 15 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.45953531052889374 - self.std = 0.254490115617556 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 26 Current best is [0.31287626] [15: BO] (30 evaluations) batch_size: 128 dropout_1: 0.0 dropout_2: 0.75 learning_rate: 0.004462430596721465 n_units_1: 115 n_units_2: 54 scale_1: 0.003013621357331008 scale_2: 0.01592608866916495 Started BO from (top scorer): batch_size: 41 dropout_1: 0.6127517152216655 dropout_2: 0.679970947116099 learning_rate: 0.0026772865275651683 n_units_1: 115 n_units_2: 54 scale_1: 0.0030135746287843013 scale_2: 0.010357876640772819 Top score values: [0.10144235 0.19863885 0.21942546 0.22952127 0.24792889] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1. Pending: Targets: [ 0.03620844 -0.60600051 -0.88639235 1.6846298 1.97314289 1.96825876 1.69174362 -0.24096338 -0.73523076 -1.06886125 -0.42975596 -0.67730978 -0.01270301 -0.166578 1.99507393 -0.3051076 -0.81213028 -1.01744805 0.16622379 0.40181061 -0.31016255 -0.65842269 -0.57628586 -0.82072506 -1.10196154 0.50894679] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.46313143159054526, 'kernel_inv_bw1': 0.03799649053583399, 'kernel_inv_bw2': 0.6789335288357643, 'kernel_inv_bw3': 5.746987933410928, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010179136732314697, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 1.0217327913896106, 'kernel_inv_bw8': 1.2424868531875415, 'kernel_covariance_scale': 0.9901900556575884, 'mean_mean_value': 0.8505919846375775} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.263732 ; Training: accuracy=0.178783 ; Validation: accuracy=0.513797 .. parsed-literal:: :class: output config_id 15: Terminating evaluation at 1 Update for config_id 15:1: reward = 0.5137965425531915, crit_val = 0.4862034574468085 Starting get_config[BO] for config_id 16 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4605230196740017 - self.std = 0.24978365026042518 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 27 Current best is [0.31287626] [16: BO] (20 evaluations) batch_size: 111 dropout_1: 0.7489872782361482 dropout_2: 0.4112951825285804 learning_rate: 0.006030391154332713 n_units_1: 71 n_units_2: 64 scale_1: 0.4702460058234478 scale_2: 0.0022580834536106847 Started BO from (top scorer): batch_size: 94 dropout_1: 0.6232277607609598 dropout_2: 0.41130154566559785 learning_rate: 0.009744280412612874 n_units_1: 67 n_units_2: 64 scale_1: 0.47024490449176665 scale_2: 0.002261738493914154 Top score values: [0.14477379 0.29458184 0.30317129 0.30841421 0.33825946] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1. Pending: Targets: [ 0.03293642 -0.62137313 -0.90704816 1.71241761 2.00636692 2.00139076 1.71966548 -0.24945791 -0.75303836 -1.09295517 -0.44180775 -0.69402602 -0.01689662 -0.17367094 2.02871118 -0.31481075 -0.83138683 -1.04057323 0.16540154 0.40542733 -0.31996095 -0.67478306 -0.5910986 -0.84014355 -1.12667915 0.5145822 0.10281072] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.5873871913789199, 'kernel_inv_bw1': 0.6133906375366832, 'kernel_inv_bw2': 0.002425978535538196, 'kernel_inv_bw3': 5.967537223380306, 'kernel_inv_bw4': 1.3571819493400543, 'kernel_inv_bw5': 0.004172932574532522, 'kernel_inv_bw6': 0.0008343982783770331, 'kernel_inv_bw7': 0.005430228338931737, 'kernel_inv_bw8': 1.088895033568476, 'kernel_covariance_scale': 0.8472538274510384, 'mean_mean_value': 1.2046771368037217} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.301385 ; Training: accuracy=0.277296 ; Validation: accuracy=0.595929 .. parsed-literal:: :class: output Update for config_id 16:1: reward = 0.5959292625959293, crit_val = 0.40407073740407073 config_id 16: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.557506 ; Training: accuracy=0.391685 ; Validation: accuracy=0.646980 Epoch 3 ; Time: 0.806270 ; Training: accuracy=0.419456 ; Validation: accuracy=0.667668 .. parsed-literal:: :class: output config_id 16: Terminating evaluation at 3 Update for config_id 16:3: reward = 0.6676676676676677, crit_val = 0.3323323323323323 Starting get_config[BO] for config_id 17 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.45415602072187755 - self.std = 0.24233239022487874 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 29 Current best is [0.25066889] [17: BO] (22 evaluations) batch_size: 78 dropout_1: 0.3045576813904649 dropout_2: 0.06715019571519018 learning_rate: 0.003262408223475904 n_units_1: 75 n_units_2: 128 scale_1: 4.8273549499119905 scale_2: 0.18003298996589037 Started BO from (top scorer): batch_size: 19 dropout_1: 0.3045423041078955 dropout_2: 0.06715009483758164 learning_rate: 0.0011765168495392192 n_units_1: 53 n_units_2: 101 scale_1: 4.8273605985554235 scale_2: 0.17881078258405803 Top score values: [0.17956376 0.19205754 0.19215283 0.20628314 0.20852107] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3. Pending: Targets: [ 0.06022298 -0.61420535 -0.90866434 1.79134502 2.09433271 2.08920355 1.79881575 -0.23085444 -0.74991903 -1.10028763 -0.42911867 -0.68909218 0.00885767 -0.15273717 2.11736402 -0.29821676 -0.83067657 -1.04629506 0.19676115 0.44416728 -0.30352531 -0.66925753 -0.58299993 -0.83970254 -1.13504856 0.55667845 0.13224578 -0.2066801 -0.50271319] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.49302343927026276, 'kernel_inv_bw1': 0.003938199546954624, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.202454482506499, 'kernel_inv_bw4': 1.5032552926599776, 'kernel_inv_bw5': 0.15190273859891862, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.8110564636235771, 'kernel_inv_bw8': 1.0382408371530802, 'kernel_covariance_scale': 0.8305671541530133, 'mean_mean_value': 1.276143848952813} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.420562 ; Training: accuracy=0.490736 ; Validation: accuracy=0.732601 .. parsed-literal:: :class: output Update for config_id 17:1: reward = 0.7326007326007326, crit_val = 0.26739926739926745 config_id 17: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.785453 ; Training: accuracy=0.637469 ; Validation: accuracy=0.787046 Epoch 3 ; Time: 1.146320 ; Training: accuracy=0.684285 ; Validation: accuracy=0.805195 .. parsed-literal:: :class: output Update for config_id 17:3: reward = 0.8051948051948052, crit_val = 0.19480519480519476 config_id 17: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.499746 ; Training: accuracy=0.713978 ; Validation: accuracy=0.823177 Epoch 5 ; Time: 1.865865 ; Training: accuracy=0.730025 ; Validation: accuracy=0.828338 Epoch 6 ; Time: 2.220377 ; Training: accuracy=0.746898 ; Validation: accuracy=0.867133 Epoch 7 ; Time: 2.569184 ; Training: accuracy=0.764516 ; Validation: accuracy=0.870463 Epoch 8 ; Time: 2.923679 ; Training: accuracy=0.771547 ; Validation: accuracy=0.869131 Epoch 9 ; Time: 3.277197 ; Training: accuracy=0.780149 ; Validation: accuracy=0.885281 .. parsed-literal:: :class: output config_id 17: Terminating evaluation at 9 Update for config_id 17:9: reward = 0.8852813852813853, crit_val = 0.11471861471861466 Starting get_config[BO] for config_id 18 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4296077399330478 - self.std = 0.24374035511370012 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 32 Current best is [0.19480519] [18: BO] (29 evaluations) batch_size: 29 dropout_1: 0.6421558254588213 dropout_2: 0.1423270311953913 learning_rate: 0.0030835974069092385 n_units_1: 76 n_units_2: 77 scale_1: 0.012848600967693495 scale_2: 0.1695480375063522 Started BO from (top scorer): batch_size: 29 dropout_1: 0.6420664042353282 dropout_2: 0.14232698216264267 learning_rate: 0.0015986669988118994 n_units_1: 49 n_units_2: 98 scale_1: 0.012848630503362952 scale_2: 0.037627741428044 Top score values: [0.15164046 0.17197031 0.22818429 0.248012 0.25221713] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9. Pending: Targets: [ 0.16058999 -0.50994251 -0.80270056 1.88171221 2.18294969 2.17785016 1.88913978 -0.12880603 -0.64487225 -0.99321695 -0.32592499 -0.58439676 0.10952138 -0.05114 2.20584796 -0.19577923 -0.72516329 -0.93953626 0.29633944 0.54231643 -0.20105712 -0.56467669 -0.47891735 -0.73413713 -1.02777708 0.65417768 0.23219675 -0.10477134 -0.39909439 -0.66549699 -0.96333061 -1.29190394] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.004452078240875218, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 6.117482454382751, 'kernel_inv_bw4': 1.4186884023219986, 'kernel_inv_bw5': 0.5540479762921837, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.8030999527559353, 'kernel_inv_bw8': 1.0030579054261999, 'kernel_covariance_scale': 0.8512684794456711, 'mean_mean_value': 1.3586677548274406} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.028780 ; Training: accuracy=0.391827 ; Validation: accuracy=0.658032 .. parsed-literal:: :class: output Update for config_id 18:1: reward = 0.6580319596299411, crit_val = 0.3419680403700589 config_id 18: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 2.041691 ; Training: accuracy=0.547662 ; Validation: accuracy=0.730530 Epoch 3 ; Time: 2.989971 ; Training: accuracy=0.593253 ; Validation: accuracy=0.737763 .. parsed-literal:: :class: output Update for config_id 18:3: reward = 0.7377628259041211, crit_val = 0.2622371740958789 config_id 18: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 3.948495 ; Training: accuracy=0.616711 ; Validation: accuracy=0.768545 Epoch 5 ; Time: 4.891091 ; Training: accuracy=0.633040 ; Validation: accuracy=0.769386 Epoch 6 ; Time: 5.820230 ; Training: accuracy=0.644728 ; Validation: accuracy=0.775105 Epoch 7 ; Time: 6.756561 ; Training: accuracy=0.652851 ; Validation: accuracy=0.797309 Epoch 8 ; Time: 7.697622 ; Training: accuracy=0.659980 ; Validation: accuracy=0.797813 Epoch 9 ; Time: 8.649182 ; Training: accuracy=0.661223 ; Validation: accuracy=0.803532 .. parsed-literal:: :class: output config_id 18: Terminating evaluation at 9 Update for config_id 18:9: reward = 0.8035323801513877, crit_val = 0.19646761984861227 Starting get_config[BO] for config_id 19 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4156605860620594 - self.std = 0.23810796909043946 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 35 Current best is [0.19480519] [19: BO] (25 evaluations) batch_size: 30 dropout_1: 0.30051346754016905 dropout_2: 0.5149662501287657 learning_rate: 0.0033584644609969 n_units_1: 87 n_units_2: 128 scale_1: 0.007139985851794008 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 30 dropout_1: 0.19178842843083357 dropout_2: 0.514965733153878 learning_rate: 0.004608147162332131 n_units_1: 89 n_units_2: 92 scale_1: 0.0071400818684522515 scale_2: 0.04676604880546017 Top score values: [0.16827371 0.2183128 0.23697301 0.24806571 0.25096753] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9. Pending: Targets: [ 0.22296362 -0.46343016 -0.76311334 1.98479857 2.29316175 2.28794159 1.99240183 -0.07327799 -0.60155163 -0.95813634 -0.27505975 -0.53964561 0.170687 0.00622521 2.31660167 -0.14183543 -0.68374193 -0.90318585 0.3619242 0.6137197 -0.14723816 -0.51945906 -0.43167111 -0.69292805 -0.99351398 0.728227 0.29626422 -0.04867476 -0.34995995 -0.62266424 -0.92754305 -1.2638887 -0.30949214 -0.64434388 -0.92056124] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0013634816584363687, 'kernel_inv_bw1': 0.6190951630939251, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.676432070744685, 'kernel_inv_bw4': 1.3333351557353115, 'kernel_inv_bw5': 0.5290566741809684, 'kernel_inv_bw6': 0.00013338602715591287, 'kernel_inv_bw7': 0.4125592465974438, 'kernel_inv_bw8': 1.0277581812671168, 'kernel_covariance_scale': 0.9258641208621095, 'mean_mean_value': 1.4275200415657963} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.014797 ; Training: accuracy=0.424461 ; Validation: accuracy=0.669024 .. parsed-literal:: :class: output Update for config_id 19:1: reward = 0.6690235690235691, crit_val = 0.3309764309764309 config_id 19: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.990045 ; Training: accuracy=0.618657 ; Validation: accuracy=0.740404 Epoch 3 ; Time: 3.037812 ; Training: accuracy=0.660862 ; Validation: accuracy=0.761448 .. parsed-literal:: :class: output Update for config_id 19:3: reward = 0.7614478114478115, crit_val = 0.23855218855218852 config_id 19: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 4.226244 ; Training: accuracy=0.699420 ; Validation: accuracy=0.788721 Epoch 5 ; Time: 5.339609 ; Training: accuracy=0.715506 ; Validation: accuracy=0.819697 Epoch 6 ; Time: 6.393850 ; Training: accuracy=0.732836 ; Validation: accuracy=0.844949 Epoch 7 ; Time: 7.426750 ; Training: accuracy=0.737811 ; Validation: accuracy=0.848485 Epoch 8 ; Time: 8.461567 ; Training: accuracy=0.746683 ; Validation: accuracy=0.853367 Epoch 9 ; Time: 9.532964 ; Training: accuracy=0.764179 ; Validation: accuracy=0.864141 .. parsed-literal:: :class: output config_id 19: Terminating evaluation at 9 Update for config_id 19:9: reward = 0.8641414141414141, crit_val = 0.1358585858585859 Starting get_config[BO] for config_id 20 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4014080978305075 - self.std = 0.2347141628366402 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 38 Current best is [0.19480519] [20: BO] (16 evaluations) batch_size: 128 dropout_1: 0.3795842555697978 dropout_2: 0.72838833428696 learning_rate: 0.0030198784213265003 n_units_1: 83 n_units_2: 128 scale_1: 0.04408145378314061 scale_2: 0.09765656504766378 Started BO from (top scorer): batch_size: 128 dropout_1: 0.4048887065451759 dropout_2: 0.728421596605789 learning_rate: 0.006348230627072913 n_units_1: 78 n_units_2: 89 scale_1: 0.04413240076143723 scale_2: 0.09765665888004807 Top score values: [0.19344329 0.2160605 0.22133216 0.22332952 0.23784654] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9. Pending: Targets: [ 0.28691026 -0.4094083 -0.71342469 2.07422014 2.38704204 2.3817464 2.08193335 -0.0136148 -0.54952691 -0.91126759 -0.21831418 -0.48672577 0.23387776 0.06703796 2.41082089 -0.08316352 -0.63290563 -0.85552255 0.42788012 0.68331642 -0.08864438 -0.46624734 -0.37719004 -0.64222457 -0.94715677 0.79947941 0.36127074 0.01134418 -0.29429739 -0.5709448 -0.88023194 -1.22144092 -0.25324444 -0.5929379 -0.87314918 -0.30007421 -0.69384782 -1.13137405] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.3537938522065249, 'kernel_inv_bw1': 0.8378356556101776, 'kernel_inv_bw2': 0.0016915851893806095, 'kernel_inv_bw3': 5.980375819112083, 'kernel_inv_bw4': 1.399340131565785, 'kernel_inv_bw5': 0.4440378496097676, 'kernel_inv_bw6': 0.0038976552747363043, 'kernel_inv_bw7': 0.0017376830268557, 'kernel_inv_bw8': 1.0821242533040438, 'kernel_covariance_scale': 0.8307653793078447, 'mean_mean_value': 1.5267504731783406} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.269033 ; Training: accuracy=0.279359 ; Validation: accuracy=0.618850 .. parsed-literal:: :class: output Update for config_id 20:1: reward = 0.6188497340425532, crit_val = 0.38115026595744683 config_id 20: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.494673 ; Training: accuracy=0.500987 ; Validation: accuracy=0.690824 Epoch 3 ; Time: 0.713359 ; Training: accuracy=0.564638 ; Validation: accuracy=0.732214 .. parsed-literal:: :class: output config_id 20: Terminating evaluation at 3 Update for config_id 20:3: reward = 0.7322140957446809, crit_val = 0.2677859042553191 Starting get_config[BO] for config_id 21 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3975610971943012 - self.std = 0.22973470279782654 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 40 Current best is [0.19480519] [21: BO] (29 evaluations) batch_size: 67 dropout_1: 0.34686117566790153 dropout_2: 0.20201418691166012 learning_rate: 0.004738650794832996 n_units_1: 80 n_units_2: 128 scale_1: 0.028468999595052603 scale_2: 0.770211333225558 Started BO from (top scorer): batch_size: 38 dropout_1: 0.5118747656872447 dropout_2: 0.20201432747850842 learning_rate: 0.005354025339239537 n_units_1: 73 n_units_2: 54 scale_1: 0.028469106152755365 scale_2: 0.7702109535801622 Top score values: [0.19717544 0.2035116 0.2039406 0.21059317 0.23890782] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3. Pending: Targets: [ 0.3098744 -0.40153675 -0.71214264 2.13592391 2.45552617 2.45011575 2.1438043 0.00283551 -0.54469241 -0.91427375 -0.2063007 -0.48053006 0.25569243 0.0852364 2.47982042 -0.06822067 -0.62987835 -0.85732045 0.45389975 0.71487259 -0.07382033 -0.45960776 -0.36862016 -0.63939927 -0.95094082 0.8335534 0.38584663 0.02833547 -0.28393083 -0.56657452 -0.88256541 -1.23117004 -0.24198807 -0.58904433 -0.87532913 -0.28983286 -0.69214144 -1.13915098 -0.07143384 -0.56489155] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.854629066503119, 'kernel_inv_bw1': 0.5870978793204662, 'kernel_inv_bw2': 0.00010050459973374464, 'kernel_inv_bw3': 6.240918512920625, 'kernel_inv_bw4': 1.4475915916512152, 'kernel_inv_bw5': 0.203472904939617, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010114531074059352, 'kernel_inv_bw8': 1.184103642865395, 'kernel_covariance_scale': 0.8406326793341455, 'mean_mean_value': 1.6068406626882217} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.464360 ; Training: accuracy=0.529104 ; Validation: accuracy=0.757672 .. parsed-literal:: :class: output Update for config_id 21:1: reward = 0.7576723125943318, crit_val = 0.24232768740566824 config_id 21: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.881710 ; Training: accuracy=0.680100 ; Validation: accuracy=0.806809 Epoch 3 ; Time: 1.292391 ; Training: accuracy=0.726202 ; Validation: accuracy=0.839510 .. parsed-literal:: :class: output Update for config_id 21:3: reward = 0.8395103136005366, crit_val = 0.16048968639946337 config_id 21: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.709924 ; Training: accuracy=0.748259 ; Validation: accuracy=0.846721 Epoch 5 ; Time: 2.114368 ; Training: accuracy=0.763765 ; Validation: accuracy=0.842529 Epoch 6 ; Time: 2.518257 ; Training: accuracy=0.780265 ; Validation: accuracy=0.872044 Epoch 7 ; Time: 2.922713 ; Training: accuracy=0.788806 ; Validation: accuracy=0.871373 Epoch 8 ; Time: 3.332150 ; Training: accuracy=0.798425 ; Validation: accuracy=0.880429 Epoch 9 ; Time: 3.735568 ; Training: accuracy=0.801078 ; Validation: accuracy=0.888311 .. parsed-literal:: :class: output config_id 21: Terminating evaluation at 9 Update for config_id 21:9: reward = 0.8883112527251383, crit_val = 0.11168874727486167 Starting get_config[BO] for config_id 22 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.38178953508958235 - self.std = 0.2293797855695426 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 43 Current best is [0.11168875] [22: BO] (28 evaluations) batch_size: 103 dropout_1: 0.1833774875607099 dropout_2: 0.17319162379965122 learning_rate: 0.0038188313847388818 n_units_1: 79 n_units_2: 81 scale_1: 5.538443323522205 scale_2: 0.007745579111514982 Started BO from (top scorer): batch_size: 103 dropout_1: 0.4270479207850957 dropout_2: 0.19779706556964466 learning_rate: 0.005409909016125813 n_units_1: 82 n_units_2: 83 scale_1: 5.538450711761021 scale_2: 0.007745531590890991 Top score values: [0.12476872 0.1279559 0.13359992 0.14188824 0.14854048] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9. Pending: Targets: [ 3.79111283e-01 -3.33400625e-01 -6.44487113e-01 2.20798623e+00 2.52808300e+00 2.52266421e+00 2.21587880e+00 7.15973174e-02 -4.76777785e-01 -8.46930980e-01 -1.37862490e-01 -4.12516160e-01 3.24845477e-01 1.54125709e-01 2.55241484e+00 4.31187824e-04 -5.62095531e-01 -7.89889553e-01 5.23359483e-01 7.84736122e-01 -5.17712752e-03 -3.91561491e-01 -3.00433104e-01 -5.71631186e-01 -8.83654785e-01 9.03600566e-01 4.55201064e-01 9.71367301e-02 -2.15612734e-01 -4.98693760e-01 -8.15173577e-01 -1.16431760e+00 -1.73605074e-01 -5.21198329e-01 -8.07926099e-01 -2.21523898e-01 -6.24454967e-01 -1.07215616e+00 -2.78694625e-03 -4.97008185e-01 -6.07995370e-01 -9.64774852e-01 -1.17752655e+00] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.8264068686291545, 'kernel_inv_bw2': 0.3255986668562232, 'kernel_inv_bw3': 6.486586927673842, 'kernel_inv_bw4': 1.8061128745205743, 'kernel_inv_bw5': 0.5409780498842242, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9818858854378966, 'kernel_covariance_scale': 0.893924709445891, 'mean_mean_value': 1.5238415630704631} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.339283 ; Training: accuracy=0.567468 ; Validation: accuracy=0.753766 .. parsed-literal:: :class: output Update for config_id 22:1: reward = 0.7537663207231335, crit_val = 0.24623367927686646 config_id 22: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.616398 ; Training: accuracy=0.687839 ; Validation: accuracy=0.800301 Epoch 3 ; Time: 1.008949 ; Training: accuracy=0.724288 ; Validation: accuracy=0.813023 .. parsed-literal:: :class: output Update for config_id 22:3: reward = 0.8130231001004352, crit_val = 0.18697689989956479 config_id 22: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.401494 ; Training: accuracy=0.742718 ; Validation: accuracy=0.839471 Epoch 5 ; Time: 1.728578 ; Training: accuracy=0.763041 ; Validation: accuracy=0.860730 Epoch 6 ; Time: 1.996136 ; Training: accuracy=0.777851 ; Validation: accuracy=0.854871 Epoch 7 ; Time: 2.264200 ; Training: accuracy=0.778756 ; Validation: accuracy=0.867593 Epoch 8 ; Time: 2.536325 ; Training: accuracy=0.786655 ; Validation: accuracy=0.878641 Epoch 9 ; Time: 2.854653 ; Training: accuracy=0.799901 ; Validation: accuracy=0.891865 .. parsed-literal:: :class: output config_id 22: Terminating evaluation at 9 Update for config_id 22:9: reward = 0.8918647472380314, crit_val = 0.10813525276196856 Starting get_config[BO] for config_id 23 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.36865860523457483 - self.std = 0.2277360857031747 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 46 Current best is [0.10813525] [23: BO] (32 evaluations) batch_size: 70 dropout_1: 0.013863812496834233 dropout_2: 0.7352588908751994 learning_rate: 0.004293444810684013 n_units_1: 81 n_units_2: 51 scale_1: 3.139501529887402 scale_2: 0.212439505230771 Started BO from (top scorer): batch_size: 85 dropout_1: 0.31349832499476277 dropout_2: 0.7352587949202476 learning_rate: 0.00033486684036548067 n_units_1: 97 n_units_2: 51 scale_1: 3.1394908510247905 scale_2: 0.2124395580502697 Top score values: [0.07207588 0.0823837 0.11300674 0.14383435 0.15748913] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9. Pending: Targets: [ 0.43950608 -0.27814843 -0.5914802 2.28158105 2.60398814 2.59853024 2.28953059 0.12977261 -0.42256042 -0.79538522 -0.08119898 -0.35783498 0.38484861 0.21289666 2.6284956 0.05809284 -0.50849395 -0.73793209 0.5847954 0.84805854 0.05244404 -0.33672907 -0.24494296 -0.51809843 -0.83237408 0.9677809 0.51614505 0.15549636 -0.15951039 -0.44463458 -0.76339861 -1.11506259 -0.11719954 -0.46730157 -0.75609882 -0.16546422 -0.57130347 -1.02223597 0.05485148 -0.44293683 -0.55472508 -0.91407964 -1.12836689 -0.53757368 -0.79777302 -1.14397045] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.8027154333408109, 'kernel_inv_bw1': 0.6254275950337382, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 9.693844289179063, 'kernel_inv_bw4': 1.7398242155388026, 'kernel_inv_bw5': 0.0001150766226427339, 'kernel_inv_bw6': 0.0003244867382900531, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.1631557122246694, 'kernel_covariance_scale': 0.8250739669034771, 'mean_mean_value': 1.6079053152786353} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.445919 ; Training: accuracy=0.265896 ; Validation: accuracy=0.633109 .. parsed-literal:: :class: output Update for config_id 23:1: reward = 0.633109243697479, crit_val = 0.366890756302521 config_id 23: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.858005 ; Training: accuracy=0.392733 ; Validation: accuracy=0.693613 Epoch 3 ; Time: 1.256582 ; Training: accuracy=0.428984 ; Validation: accuracy=0.700000 .. parsed-literal:: :class: output config_id 23: Terminating evaluation at 3 Update for config_id 23:3: reward = 0.7, crit_val = 0.30000000000000004 Starting get_config[BO] for config_id 24 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.36719138743943674 - self.std = 0.2231565750796014 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 48 Current best is [0.10813525] [24: BO] (38 evaluations) batch_size: 85 dropout_1: 0.31051060860973106 dropout_2: 0.46498300377204105 learning_rate: 0.0037397376083877225 n_units_1: 75 n_units_2: 99 scale_1: 0.0010265288422456429 scale_2: 0.031194535210625002 Started BO from (top scorer): batch_size: 85 dropout_1: 0.5566114422886048 dropout_2: 0.46081974429643935 learning_rate: 0.011724856654620737 n_units_1: 91 n_units_2: 118 scale_1: 0.001029001714917599 scale_2: 0.031194407051080224 Top score values: [0.11614941 0.12104836 0.13073905 0.16401772 0.18226919] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3. Pending: Targets: [ 4.55100248e-01 -2.77281618e-01 -5.97043435e-01 2.33497738e+00 2.66400076e+00 2.65843085e+00 2.34309006e+00 1.39010580e-01 -4.24657165e-01 -8.05132893e-01 -7.62904735e-02 -3.58603464e-01 3.99321120e-01 2.23840457e-01 2.68901114e+00 6.58598270e-02 -5.12354183e-01 -7.46500741e-01 6.03371125e-01 8.72036825e-01 6.00951114e-02 -3.37064428e-01 -2.43394725e-01 -5.22155760e-01 -8.42880822e-01 9.94216063e-01 5.33311958e-01 1.65262215e-01 -1.56208954e-01 -4.47184314e-01 -7.72489865e-01 -1.13137053e+00 -1.13029818e-01 -4.70316473e-01 -7.65040275e-01 -1.62284963e-01 -5.76452649e-01 -1.03663897e+00 6.25519482e-02 -4.45451733e-01 -5.59534040e-01 -9.26263100e-01 -1.14494785e+00 -5.42030671e-01 -8.07569696e-01 -1.16087162e+00 -1.34717580e-03 -3.01095262e-01] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.7937617751816689, 'kernel_inv_bw2': 0.2653430479628214, 'kernel_inv_bw3': 6.492116862427172, 'kernel_inv_bw4': 1.680620562170976, 'kernel_inv_bw5': 0.7040478711700238, 'kernel_inv_bw6': 0.0017574484709384443, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9794468614287616, 'kernel_covariance_scale': 0.9366363563177954, 'mean_mean_value': 1.6260365596584596} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.394603 ; Training: accuracy=0.374814 ; Validation: accuracy=0.619328 .. parsed-literal:: :class: output config_id 24: Terminating evaluation at 1 Update for config_id 24:1: reward = 0.619327731092437, crit_val = 0.380672268907563 Starting get_config[BO] for config_id 25 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3674665074693985 - self.std = 0.22087595379903402 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 49 Current best is [0.10813525] [25: BO] (41 evaluations) batch_size: 94 dropout_1: 0.18069117583678673 dropout_2: 0.0 learning_rate: 0.003287309390802507 n_units_1: 76 n_units_2: 100 scale_1: 0.08806668911612683 scale_2: 0.22766609949966224 Started BO from (top scorer): batch_size: 66 dropout_1: 0.12832840740272275 dropout_2: 0.6942082513852165 learning_rate: 0.014601379220547455 n_units_1: 90 n_units_2: 99 scale_1: 0.05865040201561297 scale_2: 0.22766679901993808 Top score values: [-0.01010981 0.07318622 0.11750267 0.11968641 0.127114 ] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 24:1. Pending: Targets: [ 4.58553730e-01 -2.81390234e-01 -6.04453703e-01 2.35784125e+00 2.69026191e+00 2.68463449e+00 2.36603769e+00 1.39200327e-01 -4.30287485e-01 -8.14691757e-01 -7.83237854e-02 -3.63551756e-01 4.02198664e-01 2.24906101e-01 2.71553054e+00 6.52942665e-02 -5.18890005e-01 -7.55454208e-01 6.08355556e-01 8.79795323e-01 5.94700283e-02 -3.41790322e-01 -2.47153447e-01 -5.28792787e-01 -8.52829447e-01 1.00323611e+00 5.37573004e-01 1.65723019e-01 -1.59067452e-01 -4.53047235e-01 -7.81711679e-01 -1.14429791e+00 -1.15442476e-01 -4.76418241e-01 -7.74185169e-01 -1.65206198e-01 -5.83650310e-01 -1.04858821e+00 6.19522327e-02 -4.51296764e-01 -5.66557010e-01 -9.37072676e-01 -1.15801542e+00 -5.48872913e-01 -8.17153721e-01 -1.17410361e+00 -2.60667201e-03 -3.05449762e-01 5.97881354e-02] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.3678343883835809, 'kernel_inv_bw1': 0.7601706769574186, 'kernel_inv_bw2': 0.23850940582314611, 'kernel_inv_bw3': 7.049427409972382, 'kernel_inv_bw4': 1.6494545190974244, 'kernel_inv_bw5': 0.37602988534368015, 'kernel_inv_bw6': 0.5503915167679336, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9766869575576077, 'kernel_covariance_scale': 0.8534905298413838, 'mean_mean_value': 1.7392742840439546} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.358125 ; Training: accuracy=0.516164 ; Validation: accuracy=0.721742 .. parsed-literal:: :class: output Update for config_id 25:1: reward = 0.7217420212765957, crit_val = 0.2782579787234043 config_id 25: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.657087 ; Training: accuracy=0.721508 ; Validation: accuracy=0.796044 Epoch 3 ; Time: 0.960103 ; Training: accuracy=0.768761 ; Validation: accuracy=0.825465 .. parsed-literal:: :class: output Update for config_id 25:3: reward = 0.8254654255319149, crit_val = 0.17453457446808507 config_id 25: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.255456 ; Training: accuracy=0.801006 ; Validation: accuracy=0.853059 Epoch 5 ; Time: 1.550006 ; Training: accuracy=0.826076 ; Validation: accuracy=0.873005 Epoch 6 ; Time: 1.837561 ; Training: accuracy=0.840508 ; Validation: accuracy=0.880485 Epoch 7 ; Time: 2.140273 ; Training: accuracy=0.849250 ; Validation: accuracy=0.892121 Epoch 8 ; Time: 2.453575 ; Training: accuracy=0.859063 ; Validation: accuracy=0.904754 Epoch 9 ; Time: 2.742258 ; Training: accuracy=0.873330 ; Validation: accuracy=0.903757 .. parsed-literal:: :class: output config_id 25: Terminating evaluation at 9 Update for config_id 25:9: reward = 0.9037566489361702, crit_val = 0.09624335106382975 Starting get_config[BO] for config_id 26 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.35682489942799706 - self.std = 0.21941256255578964 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 52 Current best is [0.09624335] [26: BO] (69 evaluations) batch_size: 42 dropout_1: 0.1744349916309062 dropout_2: 0.0 learning_rate: 0.0038372046729851224 n_units_1: 79 n_units_2: 96 scale_1: 0.2658709475772431 scale_2: 0.10064586413418124 Started BO from (top scorer): batch_size: 73 dropout_1: 0.5357545241906819 dropout_2: 0.6253351021597766 learning_rate: 0.0005066579097652998 n_units_1: 35 n_units_2: 105 scale_1: 1.1089901159674307 scale_2: 0.1505131524731448 Top score values: [0.11596818 0.11649149 0.13559758 0.14573724 0.15898673] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 24:1, 25:1, 25:3, 25:9. Pending: Targets: [ 0.51011254 -0.23476654 -0.55998471 2.42206753 2.75670529 2.75104034 2.43031864 0.18862919 -0.38465687 -0.77162497 -0.03034572 -0.31747605 0.45338161 0.27490658 2.78214245 0.1142302 -0.47385034 -0.71199233 0.66091349 0.93416364 0.10836712 -0.29556947 -0.20030141 -0.48381917 -0.81001702 1.05842773 0.58965884 0.21532877 -0.11162792 -0.40756842 -0.73842492 -1.10342946 -0.06771198 -0.4310953 -0.73084821 -0.1178076 -0.53904257 -1.00708141 0.11086588 -0.40580628 -0.52183526 -0.89482211 -1.11723845 -0.50403322 -0.77410335 -1.13343395 0.04587639 -0.25898654 0.10868735 -0.3580785 -0.83081079 -1.18763277] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.42324491240627987, 'kernel_inv_bw1': 0.7156945449360881, 'kernel_inv_bw2': 0.2141060817290444, 'kernel_inv_bw3': 6.454815377465997, 'kernel_inv_bw4': 1.5658965803517961, 'kernel_inv_bw5': 0.36673513779368455, 'kernel_inv_bw6': 0.5249981960362449, 'kernel_inv_bw7': 0.00019286209585298096, 'kernel_inv_bw8': 0.9692032659730366, 'kernel_covariance_scale': 0.8850408718410175, 'mean_mean_value': 1.8051325387420518} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.741436 ; Training: accuracy=0.608383 ; Validation: accuracy=0.787223 .. parsed-literal:: :class: output Update for config_id 26:1: reward = 0.7872233400402414, crit_val = 0.21277665995975859 config_id 26: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.385048 ; Training: accuracy=0.774058 ; Validation: accuracy=0.838364 Epoch 3 ; Time: 2.020630 ; Training: accuracy=0.821759 ; Validation: accuracy=0.868042 .. parsed-literal:: :class: output Update for config_id 26:3: reward = 0.8680415828303152, crit_val = 0.13195841716968482 config_id 26: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 2.658254 ; Training: accuracy=0.847388 ; Validation: accuracy=0.881288 Epoch 5 ; Time: 3.296943 ; Training: accuracy=0.865410 ; Validation: accuracy=0.896714 Epoch 6 ; Time: 3.937221 ; Training: accuracy=0.873760 ; Validation: accuracy=0.905097 Epoch 7 ; Time: 4.632987 ; Training: accuracy=0.882440 ; Validation: accuracy=0.921194 Epoch 8 ; Time: 5.277716 ; Training: accuracy=0.895172 ; Validation: accuracy=0.918343 Epoch 9 ; Time: 5.912462 ; Training: accuracy=0.899140 ; Validation: accuracy=0.922535 .. parsed-literal:: :class: output config_id 26: Terminating evaluation at 9 Update for config_id 26:9: reward = 0.9225352112676056, crit_val = 0.07746478873239437 Starting get_config[BO] for config_id 27 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.3450380842930488 - self.std = 0.2193002583241897 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 55 Current best is [0.07746479] [27: BO] (46 evaluations) batch_size: 8 dropout_1: 0.0 dropout_2: 0.75 learning_rate: 0.0018806118959927038 n_units_1: 72 n_units_2: 128 scale_1: 0.37204093099055713 scale_2: 0.6836981951655413 Started BO from (top scorer): batch_size: 61 dropout_1: 0.020579246890917252 dropout_2: 0.16993870748769685 learning_rate: 0.002747913638632415 n_units_1: 67 n_units_2: 111 scale_1: 0.48539653881748507 scale_2: 0.7290186728766899 Top score values: [0.08114525 0.09766269 0.12373368 0.13443452 0.15749047] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 24:1, 25:1, 25:3, 25:9, 26:1, 26:3, 26:9. Pending: Targets: [ 0.56412116 -0.18113938 -0.5065241 2.47705526 2.81186439 2.80619654 2.4853106 0.24247317 -0.33110647 -0.71827273 0.02338612 -0.26389124 0.50736117 0.32879475 2.83731458 0.16803608 -0.42034561 -0.65860956 0.71499933 0.98838942 0.16217 -0.24197345 -0.1466566 -0.43031955 -0.75668445 1.11271713 0.64370819 0.26918643 -0.0579377 -0.35402976 -0.68505569 -1.05024714 -0.01399927 -0.37756869 -0.6774751 -0.06412055 -0.48557123 -0.95384976 0.16467004 -0.35226671 -0.46835511 -0.84153297 -1.06406321 -0.45054395 -0.72075239 -1.080267 0.09964727 -0.20537178 0.16249039 -0.30451449 -0.77748887 -1.13449357 -0.60310656 -0.97163436 -1.22012303] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.05130578743636919, 'kernel_inv_bw1': 0.551820318189915, 'kernel_inv_bw2': 0.2340075383592714, 'kernel_inv_bw3': 12.644248319806238, 'kernel_inv_bw4': 1.6113078304573207, 'kernel_inv_bw5': 0.21352907282436578, 'kernel_inv_bw6': 0.5487184266138397, 'kernel_inv_bw7': 0.009127759054027853, 'kernel_inv_bw8': 0.9820413088936553, 'kernel_covariance_scale': 0.9280838171874843, 'mean_mean_value': 1.808546926134167} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.401866 ; Training: accuracy=0.470491 ; Validation: accuracy=0.745458 .. parsed-literal:: :class: output Update for config_id 27:1: reward = 0.745457604306864, crit_val = 0.25454239569313597 config_id 27: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 6.888130 ; Training: accuracy=0.629310 ; Validation: accuracy=0.810229 Epoch 3 ; Time: 10.207375 ; Training: accuracy=0.674237 ; Validation: accuracy=0.833782 .. parsed-literal:: :class: output Update for config_id 27:3: reward = 0.8337819650067295, crit_val = 0.16621803499327048 config_id 27: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 13.520683 ; Training: accuracy=0.698773 ; Validation: accuracy=0.826884 Epoch 5 ; Time: 16.891933 ; Training: accuracy=0.716512 ; Validation: accuracy=0.849092 Epoch 6 ; Time: 20.292322 ; Training: accuracy=0.730852 ; Validation: accuracy=0.859690 Epoch 7 ; Time: 23.590390 ; Training: accuracy=0.751243 ; Validation: accuracy=0.867429 Epoch 8 ; Time: 27.000361 ; Training: accuracy=0.751243 ; Validation: accuracy=0.873318 Epoch 9 ; Time: 30.334838 ; Training: accuracy=0.758704 ; Validation: accuracy=0.868102 .. parsed-literal:: :class: output config_id 27: Terminating evaluation at 9 Update for config_id 27:9: reward = 0.8681022880215343, crit_val = 0.13189771197846567 Starting get_config[BO] for config_id 28 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.33671987549625093 - self.std = 0.21682166948961054 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 58 Current best is [0.07746479] [28: BO] (32 evaluations) batch_size: 34 dropout_1: 0.0 dropout_2: 0.0 learning_rate: 0.003154901164209692 n_units_1: 78 n_units_2: 128 scale_1: 0.5518292204244141 scale_2: 0.0025324347286775906 Started BO from (top scorer): batch_size: 95 dropout_1: 0.02527270344625121 dropout_2: 0.10529219762842898 learning_rate: 0.008783476078550585 n_units_1: 119 n_units_2: 109 scale_1: 0.012432133016247692 scale_2: 0.0018737671697008117 Top score values: [-0.03482335 0.09673004 0.10107539 0.13619202 0.14314731] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 24:1, 25:1, 25:3, 25:9, 26:1, 26:3, 26:9, 27:1, 27:3, 27:9. Pending: Targets: [ 0.60893418 -0.14484578 -0.47395012 2.54373591 2.8823724 2.87663976 2.55208562 0.28360928 -0.29652722 -0.68811935 0.06201775 -0.22854362 0.55152534 0.37091764 2.90811352 0.20832127 -0.38678649 -0.62777414 0.7615371 1.03805244 0.20238813 -0.20637527 -0.10996881 -0.39687444 -0.72697017 1.1638014 0.68943101 0.31062791 -0.02023572 -0.31971255 -0.65452259 -1.02388872 0.02420498 -0.34352056 -0.64685534 -0.02648926 -0.45275773 -0.92638937 0.20491674 -0.31792934 -0.43534481 -0.81278864 -1.03786272 -0.41733004 -0.69062735 -1.05425174 0.13915067 -0.16935519 0.20271218 -0.26963125 -0.74801242 -1.1090982 -0.57163666 -0.94437728 -1.19570653 -0.37900953 -0.78636901 -0.94465726] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.28953845288727814, 'kernel_inv_bw1': 0.49950899888421957, 'kernel_inv_bw2': 0.12457109989271442, 'kernel_inv_bw3': 5.22186115179259, 'kernel_inv_bw4': 1.1916036383166984, 'kernel_inv_bw5': 0.3724057688625036, 'kernel_inv_bw6': 0.6362441175203422, 'kernel_inv_bw7': 0.5019076543486825, 'kernel_inv_bw8': 0.8463917120707751, 'kernel_covariance_scale': 1.1440352398539448, 'mean_mean_value': 1.8331809509409112} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.086185 ; Training: accuracy=0.662386 ; Validation: accuracy=0.796303 .. parsed-literal:: :class: output Update for config_id 28:1: reward = 0.7963025210084034, crit_val = 0.20369747899159663 config_id 28: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.866524 ; Training: accuracy=0.834134 ; Validation: accuracy=0.849916 Epoch 3 ; Time: 2.746118 ; Training: accuracy=0.885584 ; Validation: accuracy=0.887059 .. parsed-literal:: :class: output Update for config_id 28:3: reward = 0.8870588235294118, crit_val = 0.11294117647058821 config_id 28: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 3.617511 ; Training: accuracy=0.904474 ; Validation: accuracy=0.895966 Epoch 5 ; Time: 4.499994 ; Training: accuracy=0.923695 ; Validation: accuracy=0.910588 Epoch 6 ; Time: 5.377641 ; Training: accuracy=0.930406 ; Validation: accuracy=0.926555 Epoch 7 ; Time: 6.235283 ; Training: accuracy=0.942833 ; Validation: accuracy=0.924874 Epoch 8 ; Time: 7.096413 ; Training: accuracy=0.950621 ; Validation: accuracy=0.922689 Epoch 9 ; Time: 7.969093 ; Training: accuracy=0.955758 ; Validation: accuracy=0.922857 .. parsed-literal:: :class: output config_id 28: Terminating evaluation at 9 Update for config_id 28:9: reward = 0.9228571428571428, crit_val = 0.07714285714285718 Analysing the results ~~~~~~~~~~~~~~~~~~~~~ The training history is stored in the ``results_df``, the main fields are the runtime and ``'best'`` (the objective). **Note**: You will get slightly different curves for different pairs of scheduler/searcher, the ``time_out`` here is a bit too short to really see the difference in a significant way (it would be better to set it to >1000s). Generally speaking though, hyperband stopping / promotion + model will tend to significantly outperform other combinations given enough time. .. code:: python results_df.head() .. parsed-literal:: :class: output /var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above. and should_run_async(code) .. raw:: html
bracket elapsed_time epoch error eval_time objective runtime searcher_data_size searcher_params_kernel_covariance_scale searcher_params_kernel_inv_bw0 ... searcher_params_kernel_inv_bw7 searcher_params_kernel_inv_bw8 searcher_params_mean_mean_value searcher_params_noise_variance target_epoch task_id time_since_start time_step time_this_iter best
0 0 0.453852 1 0.468750 0.449654 0.531250 0.637854 NaN 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.639817 1.597432e+09 0.483365 0.468750
1 0 0.861637 2 0.344753 0.401216 0.655247 1.045638 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.046564 1.597432e+09 0.407755 0.344753
2 0 1.265343 3 0.305314 0.401622 0.694686 1.449345 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.450430 1.597432e+09 0.403707 0.305314
3 0 1.669142 4 0.288937 0.399600 0.711063 1.853143 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.853880 1.597432e+09 0.403799 0.288937
4 0 2.077150 5 0.273061 0.405842 0.726939 2.261152 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 2.262300 1.597432e+09 0.408009 0.273061

5 rows × 26 columns

.. code:: python import matplotlib.pyplot as plt plt.figure(figsize=(12, 8)) runtime = results_df['runtime'].values objective = results_df['best'].values plt.plot(runtime, objective, lw=2) plt.xticks(fontsize=12) plt.xlim(0, 120) plt.ylim(0, 0.5) plt.yticks(fontsize=12) plt.xlabel("Runtime [s]", fontsize=14) plt.ylabel("Objective", fontsize=14) .. parsed-literal:: :class: output Text(0, 0.5, 'Objective') .. figure:: output_mlp_cb387f_18_1.png Diving Deeper ------------- Now, you are ready to try HPO on your own machine learning models (if you use PyTorch, have a look at :ref:`sec_customstorch`). While AutoGluon comes with well-chosen defaults, it can pay off to tune it to your specific needs. Here are some tips which may come useful. Logging the Search Progress ~~~~~~~~~~~~~~~~~~~~~~~~~~~ First, it is a good idea in general to switch on ``debug_log``, which outputs useful information about the search progress. This is already done in the example above. The outputs show which configurations are chosen, stopped, or promoted. For BO and BOHB, a range of information is displayed for every ``get_config`` decision. This log output is very useful in order to figure out what is going on during the search. Configuring ``HyperbandScheduler`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The most important knobs to turn with ``HyperbandScheduler`` are ``max_t``, ``grace_period``, ``reduction_factor``, ``brackets``, and ``type``. The first three determine the rung levels at which stopping or promotion decisions are being made. - The maximum resource level ``max_t`` (usually, resource equates to epochs, so ``max_t`` is the maximum number of training epochs) is typically hardcoded in ``train_fn`` passed to the scheduler (this is ``run_mlp_openml`` in the example above). As already noted above, the value is best fixed in the ``ag.args`` decorator as ``epochs=XYZ``, it can then be accessed as ``args.epochs`` in the ``train_fn`` code. If this is done, you do not have to pass ``max_t`` when creating the scheduler. - ``grace_period`` and ``reduction_factor`` determine the rung levels, which are ``grace_period``, ``grace_period * reduction_factor``, ``grace_period * (reduction_factor ** 2)``, etc. All rung levels must be less or equal than ``max_t``. It is recommended to make ``max_t`` equal to the largest rung level. For example, if ``grace_period = 1``, ``reduction_factor = 3``, it is in general recommended to use ``max_t = 9``, ``max_t = 27``, or ``max_t = 81``. Choosing a ``max_t`` value "off the grid" works against the successive halving principle that the total resources spent in a rung should be roughly equal between rungs. If in the example above, you set ``max_t = 10``, about a third of configurations reaching 9 epochs are allowed to proceed, but only for one more epoch. - With ``reduction_factor``, you tune the extent to which successive halving filtering is applied. The larger this integer, the fewer configurations make it to higher number of epochs. Values 2, 3, 4 are commonly used. - Finally, ``grace_period`` should be set to the smallest resource (number of epochs) for which you expect any meaningful differentiation between configurations. While ``grace_period = 1`` should always be explored, it may be too low for any meaningful stopping decisions to be made at the first rung. - ``brackets`` sets the maximum number of brackets in Hyperband (make sure to study the Hyperband paper or follow-ups for details). For ``brackets = 1``, you are running successive halving (single bracket). Higher brackets have larger effective ``grace_period`` values (so runs are not stopped until later), yet are also chosen with less probability. We recommend to always consider successive halving (``brackets = 1``) in a comparison. - Finally, with ``type`` (values ``stopping``, ``promotion``) you are choosing different ways of extending successive halving scheduling to the asynchronous case. The method for the default ``stopping`` is simpler and seems to perform well, but ``promotion`` is more careful promoting configurations to higher resource levels, which can work better in some cases. Asynchronous BOHB ~~~~~~~~~~~~~~~~~ Finally, here are some ideas for tuning asynchronous BOHB, apart from tuning its ``HyperbandScheduling`` component. You need to pass these options in ``search_options``. - We support a range of different surrogate models over the criterion functions across resource levels. All of them are jointly dependent Gaussian process models, meaning that data collected at all resource levels are modelled together. The surrogate model is selected by ``gp_resource_kernel``, values are ``matern52``, ``matern52-res-warp``, ``exp-decay-sum``, ``exp-decay-combined``, ``exp-decay-delta1``. These are variants of either a joint Matern 5/2 kernel over configuration and resource, or the exponential decay model. Details about the latter can be found `here `__. - Fitting a Gaussian process surrogate model to data encurs a cost which scales cubically with the number of datapoints. When applied to expensive deep learning workloads, even multi-fidelity asynchronous BOHB is rarely running up more than 100 observations or so (across all rung levels and brackets), and the GP computations are subdominant. However, if you apply it to cheaper ``train_fn`` and find yourself beyond 2000 total evaluations, the cost of GP fitting can become painful. In such a situation, you can explore the options ``opt_skip_period`` and ``opt_skip_num_max_resource``. The basic idea is as follows. By far the most expensive part of a ``get_config`` call (picking the next configuration) is the refitting of the GP model to past data (this entails re-optimizing hyperparameters of the surrogate model itself). The options allow you to skip this expensive step for most ``get_config`` calls, after some initial period. Check the docstrings for details about these options. If you find yourself in such a situation and gain experience with these skipping features, make sure to contact the AutoGluon developers -- we would love to learn about your use case.