.. _sec_custom_advancedhpo: Getting started with Advanced HPO Algorithms ============================================ Loading libraries ----------------- .. code:: python # Basic utils for folder manipulations etc import time import multiprocessing # to count the number of CPUs available # External tools to load and process data import numpy as np import pandas as pd # MXNet (NeuralNets) import mxnet as mx from mxnet import gluon, autograd from mxnet.gluon import nn # AutoGluon and HPO tools import autogluon as ag from autogluon.utils import load_and_split_openml_data Check the version of MxNet, you should be fine with version >= 1.5 .. code:: python mx.__version__ .. parsed-literal:: :class: output '1.6.0' You can also check the version of AutoGluon and the specific commit and check that it matches what you want. .. code:: python ag.__version__ .. parsed-literal:: :class: output '0.0.13b20200717' Hyperparameter Optimization of a 2-layer MLP -------------------------------------------- Setting up the context ~~~~~~~~~~~~~~~~~~~~~~ Here we declare a few "environment variables" setting the context for what we're doing .. code:: python OPENML_TASK_ID = 6 # describes the problem we will tackle RATIO_TRAIN_VALID = 0.33 # split of the training data used for validation RESOURCE_ATTR_NAME = 'epoch' # how do we measure resources (will become clearer further) REWARD_ATTR_NAME = 'objective' # how do we measure performance (will become clearer further) NUM_CPUS = multiprocessing.cpu_count() Preparing the data ~~~~~~~~~~~~~~~~~~ We will use a multi-way classification task from OpenML. Data preparation includes: - Missing values are imputed, using the 'mean' strategy of ``sklearn.impute.SimpleImputer`` - Split training set into training and validation - Standardize inputs to mean 0, variance 1 .. code:: python X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data( OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False) n_classes .. parsed-literal:: :class: output Downloading ./org/openml/www/datasets/6/dataset.arff from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.arff... 100%|██████████| 704/704 [00:00<00:00, 61920.23KB/s] Downloading ./org/openml/www/datasets/6/dataset.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.pkl.py3... 100%|██████████| 2521/2521 [00:00<00:00, 38835.29KB/s] Downloading ./org/openml/www/datasets/6/description.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/description.xml... 3KB [00:00, 4056.39KB/s] Downloading ./org/openml/www/datasets/6/features.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/features.xml... 8KB [00:00, 7113.51KB/s] Downloading ./org/openml/www/datasets/6/qualities.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/qualities.xml... 15KB [00:00, 16074.24KB/s] Downloading ./org/openml/www/tasks/6/datasplits.arff from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.arff... 2998KB [00:00, 69993.01KB/s] Downloading ./org/openml/www/tasks/6/datasplits.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.pkl.py3... 881KB [00:00, 65618.63KB/s] Downloading ./org/openml/www/tasks/6/task.xml from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/task.xml... 3KB [00:00, 3673.84KB/s] Data pickle file already exists and is up to date. Data pickle file already exists and is up to date. .. parsed-literal:: :class: output 26 The problem has 26 classes. Declaring a model specifying a hyperparameter space with AutoGluon ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Two layer MLP where we optimize over: - the number of units on the first layer - the number of units on the second layer - the dropout rate after each layer - the learning rate - the scaling - the ``@ag.args`` decorator allows us to specify the space we will optimize over, this matches the `ConfigSpace `__ syntax The body of the function ``run_mlp_openml`` is pretty simple: - it reads the hyperparameters given via the decorator - it defines a 2 layer MLP with dropout - it declares a trainer with the 'adam' loss function and a provided learning rate - it trains the NN with a number of epochs (most of that is boilerplate code from ``mxnet``) - the ``reporter`` at the end is used to keep track of training history in the hyperparameter optimization **Note**: The number of epochs and the hyperparameter space are reduced to make for a shorter experiment .. code:: python @ag.args(n_units_1=ag.space.Int(lower=16, upper=128), n_units_2=ag.space.Int(lower=16, upper=128), dropout_1=ag.space.Real(lower=0, upper=.75), dropout_2=ag.space.Real(lower=0, upper=.75), learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True), batch_size=ag.space.Int(lower=8, upper=128), scale_1=ag.space.Real(lower=0.001, upper=10, log=True), scale_2=ag.space.Real(lower=0.001, upper=10, log=True), epochs=9) def run_mlp_openml(args, reporter, **kwargs): # Time stamp for elapsed_time ts_start = time.time() # Unwrap hyperparameters n_units_1 = args.n_units_1 n_units_2 = args.n_units_2 dropout_1 = args.dropout_1 dropout_2 = args.dropout_2 scale_1 = args.scale_1 scale_2 = args.scale_2 batch_size = args.batch_size learning_rate = args.learning_rate ctx = mx.cpu() net = nn.Sequential() with net.name_scope(): # Layer 1 net.add(nn.Dense(n_units_1, activation='relu', weight_initializer=mx.initializer.Uniform(scale=scale_1))) # Dropout net.add(gluon.nn.Dropout(dropout_1)) # Layer 2 net.add(nn.Dense(n_units_2, activation='relu', weight_initializer=mx.initializer.Uniform(scale=scale_2))) # Dropout net.add(gluon.nn.Dropout(dropout_2)) # Output net.add(nn.Dense(n_classes)) net.initialize(ctx=ctx) trainer = gluon.Trainer(net.collect_params(), 'adam', {'learning_rate': learning_rate}) for epoch in range(args.epochs): ts_epoch = time.time() train_iter = mx.io.NDArrayIter( data={'data': X_train}, label={'label': y_train}, batch_size=batch_size, shuffle=True) valid_iter = mx.io.NDArrayIter( data={'data': X_valid}, label={'label': y_valid}, batch_size=batch_size, shuffle=False) metric = mx.metric.Accuracy() loss = gluon.loss.SoftmaxCrossEntropyLoss() for batch in train_iter: data = batch.data[0].as_in_context(ctx) label = batch.label[0].as_in_context(ctx) with autograd.record(): output = net(data) L = loss(output, label) L.backward() trainer.step(data.shape[0]) metric.update([label], [output]) name, train_acc = metric.get() metric = mx.metric.Accuracy() for batch in valid_iter: data = batch.data[0].as_in_context(ctx) label = batch.label[0].as_in_context(ctx) output = net(data) metric.update([label], [output]) name, val_acc = metric.get() print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % ( epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc)) ts_now = time.time() eval_time = ts_now - ts_epoch elapsed_time = ts_now - ts_start # The resource reported back (as 'epoch') is the number of epochs # done, starting at 1 reporter( epoch=epoch + 1, objective=float(val_acc), eval_time=eval_time, time_step=ts_now, elapsed_time=elapsed_time) **Note**: The annotation ``epochs=9`` specifies the maximum number of epochs for training. It becomes available as ``args.epochs``. Importantly, it is also processed by ``HyperbandScheduler`` below in order to set its ``max_t`` attribute. **Recommendation**: Whenever writing training code to be passed as ``train_fn`` to a scheduler, if this training code reports a resource (or time) attribute, the corresponding maximum resource value should be included in ``train_fn.args``: - If the resource attribute (``time_attr`` of scheduler) in ``train_fn`` is ``epoch``, make sure to include ``epochs=XYZ`` in the annotation. This allows the scheduler to read ``max_t`` from ``train_fn.args.epochs``. This case corresponds to our example here. - If the resource attribute is something else than ``epoch``, you can also include the annotation ``max_t=XYZ``, which allows the scheduler to read ``max_t`` from ``train_fn.args.max_t``. Annotating the training function by the correct value for ``max_t`` simplifies scheduler creation (since ``max_t`` does not have to be passed), and avoids inconsistencies between ``train_fn`` and the scheduler. Running the Hyperparameter Optimization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can use the following schedulers: - FIFO (``fifo``) - Hyperband (either the stopping (``hbs``) or promotion (``hbp``) variant) And the following searchers: - Random search (``random``) - Gaussian process based Bayesian optimization (``bayesopt``) - SkOpt Bayesian optimization (``skopt``; only with FIFO scheduler) Note that the method known as (asynchronous) Hyperband is using random search. Combining Hyperband scheduling with the ``bayesopt`` searcher uses a novel method called asynchronous BOHB. Pick the combination you're interested in (doing the full experiment takes around 120 seconds, see the ``time_out`` parameter), running everything with multiple runs can take a fair bit of time. In real life, you will want to choose a larger ``time_out`` in order to obtain good performance. .. code:: python SCHEDULER = "hbs" SEARCHER = "bayesopt" .. code:: python def compute_error(df): return 1.0 - df["objective"] def compute_runtime(df, start_timestamp): return df["time_step"] - start_timestamp def process_training_history(task_dicts, start_timestamp, runtime_fn=compute_runtime, error_fn=compute_error): task_dfs = [] for task_id in task_dicts: task_df = pd.DataFrame(task_dicts[task_id]) task_df = task_df.assign(task_id=task_id, runtime=runtime_fn(task_df, start_timestamp), error=error_fn(task_df), target_epoch=task_df["epoch"].iloc[-1]) task_dfs.append(task_df) result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True) # re-order by runtime result = result.sort_values(by="runtime") # calculate incumbent best -- the cumulative minimum of the error. result = result.assign(best=result["error"].cummin()) return result resources = dict(num_cpus=NUM_CPUS, num_gpus=0) .. code:: python search_options = { 'num_init_random': 2, 'debug_log': True} if SCHEDULER == 'fifo': myscheduler = ag.scheduler.FIFOScheduler( run_mlp_openml, resource=resources, searcher=SEARCHER, search_options=search_options, time_out=120, time_attr=RESOURCE_ATTR_NAME, reward_attr=REWARD_ATTR_NAME) else: # This setup uses rung levels at 1, 3, 9 epochs. We just use a single # bracket, so this is in fact successive halving (Hyperband would use # more than 1 bracket). # Also note that since we do not use the max_t argument of # HyperbandScheduler, this value is obtained from train_fn.args.epochs. sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion' myscheduler = ag.scheduler.HyperbandScheduler( run_mlp_openml, resource=resources, searcher=SEARCHER, search_options=search_options, time_out=120, time_attr=RESOURCE_ATTR_NAME, reward_attr=REWARD_ATTR_NAME, type=sch_type, grace_period=1, reduction_factor=3, brackets=1) # run tasks myscheduler.run() myscheduler.join_jobs() results_df = process_training_history( myscheduler.training_history.copy(), start_timestamp=myscheduler._start_time) .. parsed-literal:: :class: output max_t = 9, as inferred from train_fn.args scheduler_options: Key 'resume': Imputing default value False scheduler_options: Key 'keep_size_ratios': Imputing default value False scheduler_options: Key 'maxt_pending': Imputing default value False scheduler_options: Key 'searcher_data': Imputing default value rungs scheduler_options: Key 'do_snapshots': Imputing default value False scheduler_options: Key 'visualizer': Imputing default value none scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60 scheduler_options: Key 'delay_get_config': Imputing default value True search_options: Key 'random_seed': Imputing default value 31415927 search_options: Key 'opt_skip_init_length': Imputing default value 150 search_options: Key 'opt_skip_period': Imputing default value 1 search_options: Key 'profiler': Imputing default value False search_options: Key 'opt_maxiter': Imputing default value 50 search_options: Key 'opt_nstarts': Imputing default value 2 search_options: Key 'opt_warmstart': Imputing default value False search_options: Key 'opt_verbose': Imputing default value False search_options: Key 'opt_debug_writer': Imputing default value False search_options: Key 'num_fantasy_samples': Imputing default value 20 search_options: Key 'num_init_candidates': Imputing default value 250 search_options: Key 'initial_scoring': Imputing default value thompson_indep search_options: Key 'first_is_default': Imputing default value True search_options: Key 'opt_skip_num_max_resource': Imputing default value False search_options: Key 'gp_resource_kernel': Imputing default value matern52 search_options: Key 'resource_acq': Imputing default value bohb [GPMultiFidelitySearcher.__init__] - acquisition_class = - local_minimizer_class = - num_initial_candidates = 250 - num_initial_random_choices = 2 - initial_scoring = thompson_indep - first_is_default = True Starting Experiments Num of Finished Tasks is 0 Time out (secs) is 120 Starting get_config[random] for config_id 0 Start with default config: {'batch_size': 68, 'dropout_1': 0.375, 'dropout_2': 0.375, 'learning_rate': 0.001, 'n_units_1': 72, 'n_units_2': 72, 'scale_1': 0.1, 'scale_2': 0.1} [0: random] batch_size: 68 dropout_1: 0.375 dropout_2: 0.375 learning_rate: 0.001 n_units_1: 72 n_units_2: 72 scale_1: 0.1 scale_2: 0.1 /var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/distributed/worker.py:3385: UserWarning: Large object of size 1.30 MB detected in task graph: (, {'ar ... sReporter}, []) Consider scattering large objects ahead of time with client.scatter to reduce scheduler burden and keep data on workers future = client.submit(func, big_data) # bad big_future = client.scatter(big_data) # good future = client.submit(func, big_future) # good % (format_bytes(len(b)), s) .. parsed-literal:: :class: output Epoch 1 ; Time: 0.378139 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250 .. parsed-literal:: :class: output Update for config_id 0:1: reward = 0.53125, crit_val = 0.46875 config_id 0: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.725758 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247 Epoch 3 ; Time: 1.066662 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686 .. parsed-literal:: :class: output Update for config_id 0:3: reward = 0.6946858288770054, crit_val = 0.30531417112299464 config_id 0: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.413917 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063 Epoch 5 ; Time: 1.877906 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939 Epoch 6 ; Time: 2.217341 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321 Epoch 7 ; Time: 2.561606 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501 Epoch 8 ; Time: 2.907823 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202 Epoch 9 ; Time: 3.252361 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043 .. parsed-literal:: :class: output config_id 0: Terminating evaluation at 9 Update for config_id 0:9: reward = 0.766042780748663, crit_val = 0.23395721925133695 Starting get_config[random] for config_id 1 [1: random] batch_size: 49 dropout_1: 0.6118880110946948 dropout_2: 0.5692033531564655 learning_rate: 0.00010310483923558212 n_units_1: 126 n_units_2: 93 scale_1: 0.17664689253248303 scale_2: 0.5396711634456218 .. parsed-literal:: :class: output Epoch 1 ; Time: 0.510078 ; Training: accuracy=0.048005 ; Validation: accuracy=0.111743 .. parsed-literal:: :class: output config_id 1: Terminating evaluation at 1 Update for config_id 1:1: reward = 0.11174305787888926, crit_val = 0.8882569421211107 Starting get_config[BO] for config_id 2 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4740695831238606 - self.std = 0.25382661537049356 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 4 Current best is [0.46875] [2: BO] (1 evaluations) batch_size: 50 dropout_1: 0.21593234798980557 dropout_2: 0.2842099148996821 learning_rate: 0.12426792150054146 n_units_1: 61 n_units_2: 79 scale_1: 9.791854554777688 scale_2: 4.705654410614348 Started BO from (top scorer): batch_size: 50 dropout_1: 0.21593234798980557 dropout_2: 0.2842099148996821 learning_rate: 0.12426792150054146 n_units_1: 61 n_units_2: 79 scale_1: 9.791854554777688 scale_2: 4.705654410614348 Top score values: [-0.27440679 0.04673507 0.04701008 0.10393977 0.10463477] Labeled: 0:1, 0:3, 0:9, 1:1. Pending: Targets: [-0.02095755 -0.66484522 -0.94597 1.63177277] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.7572453035969384, 'kernel_inv_bw1': 0.9003213708321409, 'kernel_inv_bw2': 0.967193412029309, 'kernel_inv_bw3': 3.935156155481977, 'kernel_inv_bw4': 26.153730997146404, 'kernel_inv_bw5': 1.6599386110717311, 'kernel_inv_bw6': 4.378455276759351, 'kernel_inv_bw7': 35.45306859772872, 'kernel_inv_bw8': 1.6274469437661863, 'kernel_covariance_scale': 0.8177941757955258, 'mean_mean_value': 0.3168762072733417} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.477658 ; Training: accuracy=0.076116 ; Validation: accuracy=0.045378 .. parsed-literal:: :class: output config_id 2: Terminating evaluation at 1 Update for config_id 2:1: reward = 0.0453781512605042, crit_val = 0.9546218487394958 Starting get_config[BO] for config_id 3 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5701800362469877 - self.std = 0.2974747675593404 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 5 Current best is [0.46875] [3: BO] (1 evaluations) batch_size: 22 dropout_1: 0.6341443733804044 dropout_2: 0.04086583670464042 learning_rate: 0.12818863950369533 n_units_1: 19 n_units_2: 101 scale_1: 0.026860146662962745 scale_2: 4.431236744525342 Started BO from (top scorer): batch_size: 22 dropout_1: 0.6341443733804044 dropout_2: 0.04086583670464042 learning_rate: 0.12818863950369533 n_units_1: 19 n_units_2: 101 scale_1: 0.026860146662962745 scale_2: 4.431236744525342 Top score values: [-0.02336009 0.00183452 0.04264146 0.12438426 0.12476886] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1. Pending: Targets: [-0.34097022 -0.89038095 -1.13025659 1.06925676 1.292351 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 8.629109443351176, 'kernel_inv_bw1': 1.74250542457921, 'kernel_inv_bw2': 1.9309204792740968, 'kernel_inv_bw3': 1.3547866499369474, 'kernel_inv_bw4': 72.29401709148367, 'kernel_inv_bw5': 4.416632357086148, 'kernel_inv_bw6': 99.98620162991055, 'kernel_inv_bw7': 100.00000000000004, 'kernel_inv_bw8': 1.345177612206158, 'kernel_covariance_scale': 0.7394315650251619, 'mean_mean_value': 0.38000484142269936} .. parsed-literal:: :class: output Epoch 1 ; Time: 1.176405 ; Training: accuracy=0.040652 ; Validation: accuracy=0.034007 .. parsed-literal:: :class: output config_id 3: Terminating evaluation at 1 Update for config_id 3:1: reward = 0.034006734006734006, crit_val = 0.965993265993266 Starting get_config[BO] for config_id 4 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6361489078713674 - self.std = 0.3090342341807874 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 6 Current best is [0.46875] [4: BO] (6 evaluations) batch_size: 110 dropout_1: 0.5744977547648839 dropout_2: 0.7047749526514967 learning_rate: 3.622175990800583e-05 n_units_1: 92 n_units_2: 92 scale_1: 0.066648545659043 scale_2: 0.021088639611731623 Started BO from (top scorer): batch_size: 110 dropout_1: 0.5744977553473402 dropout_2: 0.7047748951791846 learning_rate: 3.6221732543522435e-05 n_units_1: 92 n_units_2: 92 scale_1: 0.06664854527483652 scale_2: 0.017643835128767108 Top score values: [-0.0169074 0.02032432 0.05571745 0.06113922 0.08197224] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1. Pending: Targets: [-0.54168403 -1.07054397 -1.30144704 0.81579322 1.03054259 1.06733922] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0035785923979742124, 'kernel_inv_bw3': 0.002069863224722539, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 4.569906122762831, 'kernel_inv_bw8': 1.3538924996406003, 'kernel_covariance_scale': 0.780966574930137, 'mean_mean_value': -0.06205703671802589} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.266611 ; Training: accuracy=0.041983 ; Validation: accuracy=0.114983 .. parsed-literal:: :class: output Update for config_id 4:1: reward = 0.11498316498316498, crit_val = 0.885016835016835 config_id 4: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.486770 ; Training: accuracy=0.067438 ; Validation: accuracy=0.208754 Epoch 3 ; Time: 0.702778 ; Training: accuracy=0.095785 ; Validation: accuracy=0.246633 .. parsed-literal:: :class: output config_id 4: Terminating evaluation at 3 Update for config_id 4:3: reward = 0.24663299663299662, crit_val = 0.7533670033670034 Starting get_config[BO] for config_id 5 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6819096607015054 - self.std = 0.28105513303685026 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 8 Current best is [0.46875] [5: BO] (33 evaluations) batch_size: 128 dropout_1: 0.0 dropout_2: 0.75 learning_rate: 0.003209138078256018 n_units_1: 128 n_units_2: 16 scale_1: 0.4851811706447043 scale_2: 0.022316347285376906 Started BO from (top scorer): batch_size: 106 dropout_1: 0.38750926119776946 dropout_2: 0.04489600246106559 learning_rate: 0.002265665895170696 n_units_1: 93 n_units_2: 20 scale_1: 0.4850843918016997 scale_2: 0.03481291816391773 Top score values: [0.17278973 0.18459205 0.1869189 0.19297214 0.21585366] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3. Pending: Targets: [-0.7584265 -1.33993457 -1.59382409 0.73418791 0.97031563 1.01077537 0.72265954 0.25424671] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.042529264186120797, 'kernel_inv_bw1': 0.02357832557647553, 'kernel_inv_bw2': 0.03973208528485137, 'kernel_inv_bw3': 6.923567210521894, 'kernel_inv_bw4': 0.037428426239917156, 'kernel_inv_bw5': 0.05885736295947688, 'kernel_inv_bw6': 0.00010000000395442647, 'kernel_inv_bw7': 0.025104636768339714, 'kernel_inv_bw8': 1.3833914683569162, 'kernel_covariance_scale': 0.8507522465523345, 'mean_mean_value': 0.05452222727231742} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.207044 ; Training: accuracy=0.110773 ; Validation: accuracy=0.450798 .. parsed-literal:: :class: output Update for config_id 5:1: reward = 0.4507978723404255, crit_val = 0.5492021276595744 config_id 5: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.399478 ; Training: accuracy=0.151974 ; Validation: accuracy=0.518617 Epoch 3 ; Time: 0.586444 ; Training: accuracy=0.161513 ; Validation: accuracy=0.574967 .. parsed-literal:: :class: output Update for config_id 5:3: reward = 0.574966755319149, crit_val = 0.425033244680851 config_id 5: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 0.770652 ; Training: accuracy=0.172122 ; Validation: accuracy=0.602227 Epoch 5 ; Time: 0.960417 ; Training: accuracy=0.186678 ; Validation: accuracy=0.569980 Epoch 6 ; Time: 1.140242 ; Training: accuracy=0.196217 ; Validation: accuracy=0.640957 Epoch 7 ; Time: 1.319475 ; Training: accuracy=0.200740 ; Validation: accuracy=0.646277 Epoch 8 ; Time: 1.506795 ; Training: accuracy=0.205921 ; Validation: accuracy=0.668218 Epoch 9 ; Time: 1.698087 ; Training: accuracy=0.212993 ; Validation: accuracy=0.672540 .. parsed-literal:: :class: output config_id 5: Terminating evaluation at 9 Update for config_id 5:9: reward = 0.6725398936170213, crit_val = 0.32746010638297873 Starting get_config[BO] for config_id 6 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6142702513032225 - self.std = 0.26813178318606906 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 11 Current best is [0.46875] [6: BO] (10 evaluations) batch_size: 54 dropout_1: 0.0 dropout_2: 0.17619505106431926 learning_rate: 1.0000000000000004e-06 n_units_1: 102 n_units_2: 98 scale_1: 0.42878232592151666 scale_2: 0.00332059417669923 Started BO from (top scorer): batch_size: 54 dropout_1: 0.17226648693648097 dropout_2: 0.17978321761415783 learning_rate: 2.8682166768783764e-06 n_units_1: 104 n_units_2: 98 scale_1: 0.4256582753177773 scale_2: 0.0045765439868914945 Top score values: [0.26910219 0.30657945 0.41944929 0.42287908 0.43152957] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9. Pending: Targets: [-0.54271914 -1.1522546 -1.41838102 1.02183593 1.26934447 1.31175428 1.00975192 0.51876264 -0.24267218 -0.70576119 -1.06966113] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0029912571288300016, 'kernel_inv_bw1': 0.053822932930128575, 'kernel_inv_bw2': 0.008761955733602828, 'kernel_inv_bw3': 5.99107398159901, 'kernel_inv_bw4': 0.020069365294496683, 'kernel_inv_bw5': 0.009927002027154538, 'kernel_inv_bw6': 0.012215919003796337, 'kernel_inv_bw7': 0.02089175923817651, 'kernel_inv_bw8': 1.2191341554478625, 'kernel_covariance_scale': 0.8403434819864579, 'mean_mean_value': 0.329076417554536} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.435742 ; Training: accuracy=0.060764 ; Validation: accuracy=0.076768 .. parsed-literal:: :class: output config_id 6: Terminating evaluation at 1 Update for config_id 6:1: reward = 0.07676767676767676, crit_val = 0.9232323232323232 Starting get_config[BO] for config_id 7 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6400170906306476 - self.std = 0.270546353047318 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 12 Current best is [0.46875] [7: BO] (27 evaluations) batch_size: 100 dropout_1: 0.4356280405489459 dropout_2: 0.2913279569764319 learning_rate: 0.0010090949832113998 n_units_1: 16 n_units_2: 28 scale_1: 10.0 scale_2: 0.0010000000000000002 Started BO from (top scorer): batch_size: 98 dropout_1: 0.46137011527956384 dropout_2: 0.3585507059711988 learning_rate: 0.0007500307333009615 n_units_1: 82 n_units_2: 29 scale_1: 5.647170184313445 scale_2: 0.0018416686798357334 Top score values: [0.33640367 0.34549649 0.3693474 0.39685807 0.40517526] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1. Pending: Targets: [-0.63304158 -1.23713706 -1.50088836 0.91755017 1.16284975 1.20488106 0.905574 0.4189667 -0.33567247 -0.79462851 -1.15528072 1.04682702] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.01817329221506528, 'kernel_inv_bw1': 0.015177120649564996, 'kernel_inv_bw2': 0.017523212840445543, 'kernel_inv_bw3': 5.599465675509502, 'kernel_inv_bw4': 0.039005983498097294, 'kernel_inv_bw5': 0.013380403236308833, 'kernel_inv_bw6': 0.039127311735895454, 'kernel_inv_bw7': 0.6501362624588456, 'kernel_inv_bw8': 1.172328518720046, 'kernel_covariance_scale': 0.8289394096816239, 'mean_mean_value': 0.4350393245176951} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.245011 ; Training: accuracy=0.151818 ; Validation: accuracy=0.396000 .. parsed-literal:: :class: output Update for config_id 7:1: reward = 0.396, crit_val = 0.604 config_id 7: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.488793 ; Training: accuracy=0.212727 ; Validation: accuracy=0.436000 Epoch 3 ; Time: 0.773323 ; Training: accuracy=0.235868 ; Validation: accuracy=0.481333 .. parsed-literal:: :class: output config_id 7: Terminating evaluation at 3 Update for config_id 7:3: reward = 0.48133333333333334, crit_val = 0.5186666666666666 Starting get_config[BO] for config_id 8 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6287765538738883 - self.std = 0.2525015149386042 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 14 Current best is [0.46875] [8: BO] (45 evaluations) batch_size: 128 dropout_1: 0.0 dropout_2: 0.75 learning_rate: 0.0020383468158007866 n_units_1: 128 n_units_2: 16 scale_1: 10.0 scale_2: 2.196319028546823 Started BO from (top scorer): batch_size: 67 dropout_1: 0.16473551439200332 dropout_2: 0.3332012884008315 learning_rate: 0.0025313573378859147 n_units_1: 108 n_units_2: 112 scale_1: 0.005389198200803596 scale_2: 2.4381701654026275 Top score values: [0.28911849 0.35186715 0.35969463 0.41547191 0.42135214] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3. Pending: Targets: [-0.63376473 -1.28103145 -1.56363155 1.02763894 1.29046867 1.33550372 1.01480691 0.49342456 -0.31514435 -0.80689935 -1.19332531 1.16615447 -0.09812438 -0.43607614] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.038837005523123434, 'kernel_inv_bw1': 0.027836144782973124, 'kernel_inv_bw2': 0.031119068717025092, 'kernel_inv_bw3': 4.991188656552736, 'kernel_inv_bw4': 0.034976644009979095, 'kernel_inv_bw5': 0.041701092068418495, 'kernel_inv_bw6': 0.025326558006529706, 'kernel_inv_bw7': 1.315654342341951, 'kernel_inv_bw8': 1.107046708582646, 'kernel_covariance_scale': 0.8772040809522454, 'mean_mean_value': 0.5823012477284242} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.193519 ; Training: accuracy=0.069490 ; Validation: accuracy=0.195645 .. parsed-literal:: :class: output config_id 8: Terminating evaluation at 1 Update for config_id 8:1: reward = 0.19564494680851063, crit_val = 0.8043550531914894 Starting get_config[BO] for config_id 9 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.6404817871617284 - self.std = 0.2478401141563974 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 15 Current best is [0.46875] [9: BO] (26 evaluations) batch_size: 122 dropout_1: 0.5937047326888085 dropout_2: 0.37303996058687555 learning_rate: 0.0008351845890106922 n_units_1: 106 n_units_2: 128 scale_1: 10.0 scale_2: 0.026486365357882662 Started BO from (top scorer): batch_size: 122 dropout_1: 0.5926878354262678 dropout_2: 0.3771519994860407 learning_rate: 0.003289546045796139 n_units_1: 106 n_units_2: 128 scale_1: 7.856450227222442 scale_2: 0.18615906855362926 Top score values: [0.37313461 0.39394155 0.41274531 0.41885784 0.42425513] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1. Pending: Targets: [-0.69291361 -1.35235419 -1.64026945 0.9997379 1.26751096 1.31339303 0.98666452 0.45547597 -0.36830059 -0.86930456 -1.26299845 1.14085864 -0.14719888 -0.49150688 0.66120558] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.018142034227149436, 'kernel_inv_bw1': 0.01360424503439145, 'kernel_inv_bw2': 0.01865090626139601, 'kernel_inv_bw3': 4.710716144212342, 'kernel_inv_bw4': 0.0182185999905586, 'kernel_inv_bw5': 0.021442449668858127, 'kernel_inv_bw6': 0.052111193469014445, 'kernel_inv_bw7': 2.0089750512467703, 'kernel_inv_bw8': 1.079644308068812, 'kernel_covariance_scale': 0.9142565484730996, 'mean_mean_value': 0.6331161028293205} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.226691 ; Training: accuracy=0.298559 ; Validation: accuracy=0.632988 .. parsed-literal:: :class: output Update for config_id 9:1: reward = 0.6329876212780194, crit_val = 0.36701237872198056 config_id 9: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.442504 ; Training: accuracy=0.443534 ; Validation: accuracy=0.707929 Epoch 3 ; Time: 0.652674 ; Training: accuracy=0.482530 ; Validation: accuracy=0.711777 .. parsed-literal:: :class: output Update for config_id 9:3: reward = 0.7117765138842422, crit_val = 0.28822348611575777 config_id 9: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 0.867149 ; Training: accuracy=0.504471 ; Validation: accuracy=0.730345 Epoch 5 ; Time: 1.097763 ; Training: accuracy=0.511095 ; Validation: accuracy=0.745065 Epoch 6 ; Time: 1.308775 ; Training: accuracy=0.538914 ; Validation: accuracy=0.752760 Epoch 7 ; Time: 1.538658 ; Training: accuracy=0.544958 ; Validation: accuracy=0.772666 Epoch 8 ; Time: 1.754248 ; Training: accuracy=0.552741 ; Validation: accuracy=0.777852 Epoch 9 ; Time: 1.972207 ; Training: accuracy=0.560772 ; Validation: accuracy=0.774005 .. parsed-literal:: :class: output config_id 9: Terminating evaluation at 9 Update for config_id 9:9: reward = 0.7740046838407494, crit_val = 0.22599531615925061 Starting get_config[BO] for config_id 10 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5826921104679397 - self.std = 0.26161121007299293 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 18 Current best is [0.36701238] [10: BO] (8 evaluations) batch_size: 35 dropout_1: 0.4189316406676796 dropout_2: 0.64785944140762 learning_rate: 0.0004923229512480123 n_units_1: 92 n_units_2: 107 scale_1: 0.06187773027317159 scale_2: 0.014919426971865563 Started BO from (top scorer): batch_size: 35 dropout_1: 0.4189316770736182 dropout_2: 0.647859383861638 learning_rate: 0.0004840516285227221 n_units_1: 92 n_units_2: 107 scale_1: 0.06187776130090886 scale_2: 0.03166162310123019 Top score values: [0.33639694 0.36297432 0.38126212 0.38645803 0.39695491] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9. Pending: Targets: [-0.43553986 -1.06026779 -1.33302732 1.16801123 1.42168884 1.4651557 1.15562603 0.652399 -0.12801433 -0.60264568 -0.9756157 1.30170344 0.08144869 -0.24473509 0.8472991 -0.82442848 -1.12559635 -1.36346143] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0012528653216046406, 'kernel_inv_bw1': 0.000351239318362345, 'kernel_inv_bw2': 0.00045058044207702105, 'kernel_inv_bw3': 6.68223995627897, 'kernel_inv_bw4': 0.0015479203428939742, 'kernel_inv_bw5': 0.0007301684228847166, 'kernel_inv_bw6': 0.0003142868289391012, 'kernel_inv_bw7': 2.612446594927186, 'kernel_inv_bw8': 1.2880199140786603, 'kernel_covariance_scale': 0.78600469004484, 'mean_mean_value': 0.69320155558555} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.675189 ; Training: accuracy=0.236605 ; Validation: accuracy=0.500504 .. parsed-literal:: :class: output Update for config_id 10:1: reward = 0.5005042016806722, crit_val = 0.4994957983193278 config_id 10: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 1.337997 ; Training: accuracy=0.413085 ; Validation: accuracy=0.582521 Epoch 3 ; Time: 2.003231 ; Training: accuracy=0.487619 ; Validation: accuracy=0.626387 .. parsed-literal:: :class: output config_id 10: Terminating evaluation at 3 Update for config_id 10:3: reward = 0.6263865546218488, crit_val = 0.3736134453781512 Starting get_config[BO] for config_id 11 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5680783616060197 - self.std = 0.2528133635392197 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 20 Current best is [0.36701238] [11: BO] (27 evaluations) batch_size: 128 dropout_1: 0.48567407913313465 dropout_2: 0.41288933361639724 learning_rate: 0.002041995112130015 n_units_1: 36 n_units_2: 128 scale_1: 5.784232732906726 scale_2: 0.025972116606279762 Started BO from (top scorer): batch_size: 105 dropout_1: 0.4856686881223919 dropout_2: 0.41288942543415097 learning_rate: 0.015537307956329766 n_units_1: 36 n_units_2: 118 scale_1: 5.784225726298944 scale_2: 0.1753674900097659 Top score values: [0.28009922 0.28997111 0.3217382 0.34170678 0.35464527] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3. Pending: Targets: [-0.39289205 -1.03936037 -1.32161187 1.26646225 1.52896778 1.57394727 1.25364605 0.73290683 -0.0746647 -0.56581312 -0.95176241 1.40480692 0.14208758 -0.19544732 0.93458941 -0.7953139 -1.10696235 -1.35310508 -0.27127744 -0.76920347] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.19035005072870462, 'kernel_inv_bw1': 0.0011561453841036151, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 4.326774650479719, 'kernel_inv_bw4': 0.00024049852257538023, 'kernel_inv_bw5': 0.5778725631767939, 'kernel_inv_bw6': 0.000175147876436982, 'kernel_inv_bw7': 1.6193948079704306, 'kernel_inv_bw8': 1.1099815661552153, 'kernel_covariance_scale': 0.8683275627500506, 'mean_mean_value': 0.7830009055909884} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.199573 ; Training: accuracy=0.278372 ; Validation: accuracy=0.572806 .. parsed-literal:: :class: output Update for config_id 11:1: reward = 0.5728058510638298, crit_val = 0.42719414893617025 config_id 11: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.397331 ; Training: accuracy=0.363816 ; Validation: accuracy=0.621011 Epoch 3 ; Time: 0.591078 ; Training: accuracy=0.400905 ; Validation: accuracy=0.642287 .. parsed-literal:: :class: output config_id 11: Terminating evaluation at 3 Update for config_id 11:3: reward = 0.6422872340425532, crit_val = 0.35771276595744683 Starting get_config[BO] for config_id 12 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5521124612279095 - self.std = 0.24650151773873644 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 22 Current best is [0.36701238] [12: BO] (30 evaluations) batch_size: 128 dropout_1: 0.3949306838852321 dropout_2: 0.37421973461517766 learning_rate: 0.0008619875734616166 n_units_1: 32 n_units_2: 60 scale_1: 0.19191121634570388 scale_2: 0.024011044036543073 Started BO from (top scorer): batch_size: 72 dropout_1: 0.39493055316577974 dropout_2: 0.37424420255786994 learning_rate: 0.0015053635333599907 n_units_1: 32 n_units_2: 65 scale_1: 0.1919113363111212 scale_2: 0.02258804603989569 Top score values: [0.39569149 0.41432932 0.43900397 0.44850582 0.46284489] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3. Pending: Targets: [-0.33818234 -1.00120394 -1.29068269 1.3636609 1.63288807 1.6790193 1.35051653 0.81644342 -0.01180655 -0.51553117 -0.91136297 1.50554798 0.21049582 -0.1356819 1.02329022 -0.75090849 -1.07053692 -1.3229823 -0.21345371 -0.72412948 -0.50676488 -0.78863488] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.2923585253961906, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0008488941270236102, 'kernel_inv_bw3': 4.8085961725692075, 'kernel_inv_bw4': 0.0009394039000798733, 'kernel_inv_bw5': 0.28904663988794216, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 1.8569050490473502, 'kernel_inv_bw8': 1.24560644192412, 'kernel_covariance_scale': 0.8430521769516296, 'mean_mean_value': 0.8725688626622445} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.195287 ; Training: accuracy=0.164145 ; Validation: accuracy=0.238863 .. parsed-literal:: :class: output config_id 12: Terminating evaluation at 1 Update for config_id 12:1: reward = 0.2388630319148936, crit_val = 0.7611369680851063 Starting get_config[BO] for config_id 13 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5612004832651789 - self.std = 0.24482270307015344 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 23 Current best is [0.36701238] [13: BO] (31 evaluations) batch_size: 85 dropout_1: 0.517318414125913 dropout_2: 0.3503723017238424 learning_rate: 0.004339501360450686 n_units_1: 127 n_units_2: 120 scale_1: 0.00503374773788621 scale_2: 0.04682843950719627 Started BO from (top scorer): batch_size: 85 dropout_1: 0.5173205997487473 dropout_2: 0.3503395161012028 learning_rate: 0.17877383202727398 n_units_1: 21 n_units_2: 38 scale_1: 0.005033730493003303 scale_2: 0.004129261303025423 Top score values: [0.21522201 0.24871193 0.24959443 0.2818383 0.29216209] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1. Pending: Targets: [-0.37762218 -1.04519029 -1.33665408 1.33589105 1.60696439 1.65341195 1.32265655 0.78492116 -0.04900835 -0.55618714 -0.95473326 1.47875109 0.17481841 -0.17373314 0.99318636 -0.7931785 -1.11499871 -1.36917517 -0.25203825 -0.76621586 -0.54736073 -0.83116359 0.81665827] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.0018009065340422222, 'kernel_inv_bw2': 0.00526751979250695, 'kernel_inv_bw3': 3.6751566246702496, 'kernel_inv_bw4': 0.6931462824479084, 'kernel_inv_bw5': 0.9862999198786682, 'kernel_inv_bw6': 0.0020835114772087687, 'kernel_inv_bw7': 3.171923971732715, 'kernel_inv_bw8': 0.9914290006573513, 'kernel_covariance_scale': 1.0347136339005742, 'mean_mean_value': 0.6951370896269443} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.297343 ; Training: accuracy=0.436205 ; Validation: accuracy=0.676303 .. parsed-literal:: :class: output Update for config_id 13:1: reward = 0.6763025210084034, crit_val = 0.3236974789915966 config_id 13: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.585487 ; Training: accuracy=0.617978 ; Validation: accuracy=0.755630 Epoch 3 ; Time: 0.867727 ; Training: accuracy=0.671914 ; Validation: accuracy=0.776807 .. parsed-literal:: :class: output Update for config_id 13:3: reward = 0.7768067226890756, crit_val = 0.22319327731092442 config_id 13: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.156833 ; Training: accuracy=0.701574 ; Validation: accuracy=0.809580 Epoch 5 ; Time: 1.442775 ; Training: accuracy=0.721541 ; Validation: accuracy=0.833445 Epoch 6 ; Time: 1.734089 ; Training: accuracy=0.737200 ; Validation: accuracy=0.837311 Epoch 7 ; Time: 2.021497 ; Training: accuracy=0.749544 ; Validation: accuracy=0.849076 Epoch 8 ; Time: 2.312035 ; Training: accuracy=0.753853 ; Validation: accuracy=0.860840 Epoch 9 ; Time: 2.596859 ; Training: accuracy=0.761060 ; Validation: accuracy=0.865546 .. parsed-literal:: :class: output config_id 13: Terminating evaluation at 9 Update for config_id 13:9: reward = 0.865546218487395, crit_val = 0.134453781512605 Starting get_config[BO] for config_id 14 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5226521404967017 - self.std = 0.2551554231756705 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 26 Current best is [0.22319328] [14: BO] (27 evaluations) batch_size: 8 dropout_1: 0.44083006806595904 dropout_2: 0.5666285599135659 learning_rate: 0.0039505467062455855 n_units_1: 128 n_units_2: 83 scale_1: 0.004122091602642879 scale_2: 0.06906901108503602 Started BO from (top scorer): batch_size: 89 dropout_1: 0.44083011379014075 dropout_2: 0.5666285180710813 learning_rate: 0.056261366616198794 n_units_1: 74 n_units_2: 70 scale_1: 0.004122085994468912 scale_2: 0.07838948698811744 Top score values: [0.08943119 0.09851775 0.15949493 0.16826384 0.18186243] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9. Pending: Targets: [-0.21125218 -0.8517866 -1.13144733 1.43287098 1.69296699 1.73753362 1.42017242 0.90421305 0.10405418 -0.38258601 -0.76499269 1.56994579 0.31881689 -0.01561979 1.10404439 -0.60998022 -0.91876806 -1.16265146 -0.09075387 -0.58410945 -0.37411704 -0.64642708 0.93466494 -0.77973911 -1.17363315 -1.5214192 ] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.19661690687917155, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 2.76708380111987, 'kernel_inv_bw4': 0.45302302511801334, 'kernel_inv_bw5': 1.0006757249815805, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 3.4402454237796487, 'kernel_inv_bw8': 0.9099949398435804, 'kernel_covariance_scale': 1.0560090446899353, 'mean_mean_value': 0.7294833011248456} .. parsed-literal:: :class: output Epoch 1 ; Time: 2.887999 ; Training: accuracy=0.453830 ; Validation: accuracy=0.709287 .. parsed-literal:: :class: output Update for config_id 14:1: reward = 0.7092866756393001, crit_val = 0.29071332436069985 config_id 14: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 5.793767 ; Training: accuracy=0.579078 ; Validation: accuracy=0.756057 Epoch 3 ; Time: 8.724383 ; Training: accuracy=0.601376 ; Validation: accuracy=0.781965 .. parsed-literal:: :class: output Update for config_id 14:3: reward = 0.7819650067294751, crit_val = 0.2180349932705249 config_id 14: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 11.667929 ; Training: accuracy=0.626824 ; Validation: accuracy=0.795929 Epoch 5 ; Time: 14.510981 ; Training: accuracy=0.638677 ; Validation: accuracy=0.790377 Epoch 6 ; Time: 17.419908 ; Training: accuracy=0.652188 ; Validation: accuracy=0.807032 Epoch 7 ; Time: 20.510548 ; Training: accuracy=0.654095 ; Validation: accuracy=0.820659 Epoch 8 ; Time: 23.343941 ; Training: accuracy=0.665202 ; Validation: accuracy=0.827221 Epoch 9 ; Time: 26.385818 ; Training: accuracy=0.670590 ; Validation: accuracy=0.822678 .. parsed-literal:: :class: output config_id 14: Terminating evaluation at 9 Update for config_id 14:9: reward = 0.822678331090175, crit_val = 0.17732166890982504 Starting get_config[BO] for config_id 15 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4922422634294929 - self.std = 0.2580920603410797 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 29 Current best is [0.21803499] [15: BO] (32 evaluations) batch_size: 28 dropout_1: 0.7067660247935745 dropout_2: 0.5077105068694533 learning_rate: 0.0060571749044140925 n_units_1: 128 n_units_2: 124 scale_1: 8.648184446960908 scale_2: 0.007892685448944886 Started BO from (top scorer): batch_size: 28 dropout_1: 0.7067660233501636 dropout_2: 0.5077105012206583 learning_rate: 0.6552783935929372 n_units_1: 25 n_units_2: 26 scale_1: 8.64818447109396 scale_2: 0.001987907158095328 Top score values: [0.08101922 0.18297088 0.22671446 0.24364789 0.26306334] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9. Pending: Targets: [-0.09102281 -0.72426905 -1.00074773 1.53439311 1.79152968 1.83558922 1.52183903 1.01175038 0.22069592 -0.26040715 -0.63846271 1.66990825 0.43301501 0.10238363 1.20930799 -0.48521401 -0.79048839 -1.03159681 0.02810445 -0.45963761 -0.25203454 -0.52124617 1.04185578 -0.65304134 -1.04245356 -1.3862824 -0.7808413 -1.06243977 -1.22018707] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0017239593234710502, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 3.570434407737034, 'kernel_inv_bw4': 0.7283941471136309, 'kernel_inv_bw5': 1.000855396404901, 'kernel_inv_bw6': 0.00010423808122954203, 'kernel_inv_bw7': 3.232910339204173, 'kernel_inv_bw8': 1.0262492248060313, 'kernel_covariance_scale': 0.8876604836153849, 'mean_mean_value': 0.8701089467021575} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.875654 ; Training: accuracy=0.170451 ; Validation: accuracy=0.267606 .. parsed-literal:: :class: output config_id 15: Terminating evaluation at 1 Update for config_id 15:1: reward = 0.2676056338028169, crit_val = 0.7323943661971831 Starting get_config[BO] for config_id 16 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.5002473335217492 - self.std = 0.257389745636864 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 30 Current best is [0.21803499] [16: BO] (44 evaluations) batch_size: 8 dropout_1: 0.6233223085212575 dropout_2: 0.4112965755062492 learning_rate: 0.001956180614449219 n_units_1: 96 n_units_2: 128 scale_1: 0.08489975369264192 scale_2: 0.004798678017280562 Started BO from (top scorer): batch_size: 94 dropout_1: 0.6232277607609598 dropout_2: 0.41130154566559785 learning_rate: 0.009744280412612874 n_units_1: 67 n_units_2: 64 scale_1: 0.47024490449176665 scale_2: 0.002261738493914154 Top score values: [0.12222099 0.18662739 0.19128603 0.20603738 0.21008631] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1. Pending: Targets: [-0.12237214 -0.75734626 -1.03457934 1.50747889 1.76531708 1.80949684 1.49489056 0.98341008 0.19019714 -0.29221867 -0.67130579 1.6433638 0.40309557 0.07156203 1.18150674 -0.51763894 -0.82374629 -1.0655126 -0.00291983 -0.49199275 -0.28382321 -0.55376941 1.01359762 -0.6859242 -1.07639897 -1.42116599 -0.81407287 -1.09643972 -1.25461744 0.90192806] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.02072833578446393, 'kernel_inv_bw1': 0.0007070265218982463, 'kernel_inv_bw2': 0.00010601560063883542, 'kernel_inv_bw3': 5.71608459837552, 'kernel_inv_bw4': 1.4945779216991482, 'kernel_inv_bw5': 0.3867168213153175, 'kernel_inv_bw6': 1.2627144379756745, 'kernel_inv_bw7': 0.6569959505220573, 'kernel_inv_bw8': 0.9141003039474879, 'kernel_covariance_scale': 1.0905645951879839, 'mean_mean_value': 0.912365266777159} .. parsed-literal:: :class: output Epoch 1 ; Time: 3.034433 ; Training: accuracy=0.429377 ; Validation: accuracy=0.677322 .. parsed-literal:: :class: output Update for config_id 16:1: reward = 0.677321668909825, crit_val = 0.32267833109017496 config_id 16: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 5.950719 ; Training: accuracy=0.556283 ; Validation: accuracy=0.738728 Epoch 3 ; Time: 8.878337 ; Training: accuracy=0.598060 ; Validation: accuracy=0.759253 .. parsed-literal:: :class: output Update for config_id 16:3: reward = 0.7592530282637954, crit_val = 0.24074697173620463 config_id 16: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 11.820351 ; Training: accuracy=0.622265 ; Validation: accuracy=0.774226 Epoch 5 ; Time: 14.735351 ; Training: accuracy=0.636771 ; Validation: accuracy=0.782806 Epoch 6 ; Time: 17.613175 ; Training: accuracy=0.652105 ; Validation: accuracy=0.789367 Epoch 7 ; Time: 20.488157 ; Training: accuracy=0.653017 ; Validation: accuracy=0.811743 Epoch 8 ; Time: 23.363082 ; Training: accuracy=0.655338 ; Validation: accuracy=0.807201 Epoch 9 ; Time: 26.267338 ; Training: accuracy=0.670093 ; Validation: accuracy=0.815949 .. parsed-literal:: :class: output config_id 16: Terminating evaluation at 9 Update for config_id 16:9: reward = 0.8159488559892328, crit_val = 0.18405114401076716 Starting get_config[BO] for config_id 17 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4774211046208977 - self.std = 0.2563816663344639 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 33 Current best is [0.21803499] [17: BO] (36 evaluations) batch_size: 128 dropout_1: 0.24051539331560012 dropout_2: 0.6439217787214787 learning_rate: 0.00403615530668573 n_units_1: 106 n_units_2: 61 scale_1: 0.01043239707029727 scale_2: 0.0485385641688307 Started BO from (top scorer): batch_size: 99 dropout_1: 0.2403693569347366 dropout_2: 0.643921725129281 learning_rate: 0.06780717783001759 n_units_1: 110 n_units_2: 84 scale_1: 0.01619971067719354 scale_2: 0.04848851707519957 Top score values: [9.18103952e-05 1.07671531e-01 1.51050172e-01 1.67672935e-01 2.31863384e-01] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9. Pending: Targets: [-0.03382108 -0.67129189 -0.94961504 1.60243844 1.86129044 1.90564391 1.58980061 1.07630901 0.27997721 -0.20433544 -0.58491311 1.73885764 0.49371274 0.16087563 1.27518458 -0.43064205 -0.73795299 -0.98066992 0.08610091 -0.40489502 -0.19590697 -0.46691458 1.10661526 -0.59958899 -0.99159909 -1.33772172 -0.72824154 -1.01171864 -1.17051831 0.99450661 -0.60356411 -0.92313205 -1.14427043] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.023086313292617164, 'kernel_inv_bw1': 0.004640112704300706, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 4.655431728543398, 'kernel_inv_bw4': 1.5147034320525727, 'kernel_inv_bw5': 0.4950980358598371, 'kernel_inv_bw6': 1.3729014795091468, 'kernel_inv_bw7': 0.005020068157930614, 'kernel_inv_bw8': 0.8831416045466082, 'kernel_covariance_scale': 1.147014605226413, 'mean_mean_value': 0.9577840686516704} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.202711 ; Training: accuracy=0.286266 ; Validation: accuracy=0.607048 .. parsed-literal:: :class: output Update for config_id 17:1: reward = 0.6070478723404256, crit_val = 0.39295212765957444 config_id 17: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.401124 ; Training: accuracy=0.491118 ; Validation: accuracy=0.693152 Epoch 3 ; Time: 0.591367 ; Training: accuracy=0.553043 ; Validation: accuracy=0.723903 .. parsed-literal:: :class: output Update for config_id 17:3: reward = 0.7239029255319149, crit_val = 0.27609707446808507 config_id 17: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 0.783518 ; Training: accuracy=0.594901 ; Validation: accuracy=0.749169 Epoch 5 ; Time: 0.973595 ; Training: accuracy=0.603783 ; Validation: accuracy=0.771609 Epoch 6 ; Time: 1.165268 ; Training: accuracy=0.630099 ; Validation: accuracy=0.785073 Epoch 7 ; Time: 1.354876 ; Training: accuracy=0.646135 ; Validation: accuracy=0.801862 Epoch 8 ; Time: 1.550017 ; Training: accuracy=0.657237 ; Validation: accuracy=0.813664 Epoch 9 ; Time: 1.744095 ; Training: accuracy=0.673026 ; Validation: accuracy=0.820977 .. parsed-literal:: :class: output config_id 17: Terminating evaluation at 9 Update for config_id 17:9: reward = 0.8209773936170213, crit_val = 0.17902260638297873 Starting get_config[BO] for config_id 18 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4611935628055629 - self.std = 0.25256294519747646 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 36 Current best is [0.21803499] [18: BO] (41 evaluations) batch_size: 105 dropout_1: 0.75 dropout_2: 0.19746761799270035 learning_rate: 0.0010894554312626916 n_units_1: 77 n_units_2: 128 scale_1: 10.0 scale_2: 0.002951115158859443 Started BO from (top scorer): batch_size: 126 dropout_1: 0.14441139566675065 dropout_2: 0.1974703086504215 learning_rate: 0.0007212733277897834 n_units_1: 71 n_units_2: 110 scale_1: 0.4701081990937349 scale_2: 0.022516111383238842 Top score values: [0.15113935 0.1618521 0.20883571 0.23071878 0.23280613] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9. Pending: Targets: [ 0.02991903 -0.61719027 -0.89972163 1.69091859 1.9536844 1.99870849 1.67808968 1.15683415 0.3484619 -0.14317349 -0.52950545 1.82940043 0.56542909 0.22755953 1.3587167 -0.37290183 -0.68485928 -0.93124606 0.15165422 -0.34676551 -0.13461759 -0.4097228 1.18759862 -0.54440323 -0.94234047 -1.29369643 -0.67500099 -0.96276423 -1.12396493 1.0737949 -0.54843846 -0.87283822 -1.09732019 -0.27019575 -0.7328727 -1.11723023] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.20953941985620783, 'kernel_inv_bw1': 0.048192726740054884, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 4.582304758795538, 'kernel_inv_bw4': 1.4502331780737872, 'kernel_inv_bw5': 0.06676427918254801, 'kernel_inv_bw6': 1.1459117819421119, 'kernel_inv_bw7': 0.7880349587551732, 'kernel_inv_bw8': 0.8158791108576938, 'kernel_covariance_scale': 1.2586479096561005, 'mean_mean_value': 1.094690485256033} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.249351 ; Training: accuracy=0.276190 ; Validation: accuracy=0.582623 .. parsed-literal:: :class: output Update for config_id 18:1: reward = 0.5826232247284879, crit_val = 0.41737677527151207 config_id 18: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.503005 ; Training: accuracy=0.328033 ; Validation: accuracy=0.615706 Epoch 3 ; Time: 0.734463 ; Training: accuracy=0.343354 ; Validation: accuracy=0.640936 .. parsed-literal:: :class: output config_id 18: Terminating evaluation at 3 Update for config_id 18:3: reward = 0.6409356725146199, crit_val = 0.3590643274853801 Starting get_config[BO] for config_id 19 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4573528779936094 - self.std = 0.24645695926305125 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 38 Current best is [0.21803499] [19: BO] (36 evaluations) batch_size: 8 dropout_1: 0.75 dropout_2: 0.39774784778258354 learning_rate: 0.001832140609847963 n_units_1: 114 n_units_2: 128 scale_1: 0.01076740318505519 scale_2: 0.047239059463260964 Started BO from (top scorer): batch_size: 11 dropout_1: 0.27773647423520337 dropout_2: 0.3977763768050508 learning_rate: 0.0011784010178056816 n_units_1: 117 n_units_2: 92 scale_1: 0.0025456093439067047 scale_2: 0.04677717904168564 Top score values: [0.05222215 0.16571061 0.18450087 0.1997315 0.20469509] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3. Pending: Targets: [ 0.04624386 -0.6168976 -0.90642869 1.74839479 2.01767064 2.06381021 1.73524805 1.20107838 0.37267866 -0.13113703 -0.52704039 1.89030753 0.59502123 0.24878092 1.40796258 -0.3665569 -0.68624312 -0.93873414 0.17099505 -0.33977305 -0.12236915 -0.40429011 1.23260504 -0.54230726 -0.95010342 -1.31016425 -0.67614059 -0.97103318 -1.13622764 1.11598183 -0.54644246 -0.87887924 -1.10892277 -0.26130628 -0.73544608 -1.12932608 -0.16220318 -0.39880615] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.21829910405242756, 'kernel_inv_bw1': 0.025265367905435814, 'kernel_inv_bw2': 0.00018208945300286777, 'kernel_inv_bw3': 5.645833577530412, 'kernel_inv_bw4': 1.0626331487883236, 'kernel_inv_bw5': 0.24181265317433792, 'kernel_inv_bw6': 1.0525747449661047, 'kernel_inv_bw7': 1.2753127010118737, 'kernel_inv_bw8': 0.8183279586931533, 'kernel_covariance_scale': 1.182063180806255, 'mean_mean_value': 1.1579050627358658} .. parsed-literal:: :class: output Epoch 1 ; Time: 2.876373 ; Training: accuracy=0.377072 ; Validation: accuracy=0.648890 .. parsed-literal:: :class: output Update for config_id 19:1: reward = 0.6488896366083445, crit_val = 0.35111036339165547 config_id 19: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 5.718347 ; Training: accuracy=0.502984 ; Validation: accuracy=0.706427 Epoch 3 ; Time: 8.549129 ; Training: accuracy=0.536638 ; Validation: accuracy=0.730989 .. parsed-literal:: :class: output Update for config_id 19:3: reward = 0.7309892328398385, crit_val = 0.2690107671601615 config_id 19: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 11.390956 ; Training: accuracy=0.560262 ; Validation: accuracy=0.741252 Epoch 5 ; Time: 14.409938 ; Training: accuracy=0.575348 ; Validation: accuracy=0.761945 Epoch 6 ; Time: 17.289553 ; Training: accuracy=0.587782 ; Validation: accuracy=0.756898 Epoch 7 ; Time: 20.207145 ; Training: accuracy=0.594910 ; Validation: accuracy=0.765478 Epoch 8 ; Time: 23.148990 ; Training: accuracy=0.596237 ; Validation: accuracy=0.774058 Epoch 9 ; Time: 26.027264 ; Training: accuracy=0.597066 ; Validation: accuracy=0.774899 .. parsed-literal:: :class: output config_id 19: Terminating evaluation at 9 Update for config_id 19:9: reward = 0.7748990578734859, crit_val = 0.2251009421265141 Starting get_config[BO] for config_id 20 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4445032057667191 - self.std = 0.24204871271645806 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 41 Current best is [0.13445378] [20: BO] (33 evaluations) batch_size: 128 dropout_1: 0.43608638717208853 dropout_2: 0.5726251067457891 learning_rate: 0.002088478516714575 n_units_1: 121 n_units_2: 128 scale_1: 0.021011121096350963 scale_2: 0.006870163092014335 Started BO from (top scorer): batch_size: 26 dropout_1: 0.37471147279126216 dropout_2: 0.5782873531954602 learning_rate: 0.0006132144475527039 n_units_1: 128 n_units_2: 76 scale_1: 0.762216918632655 scale_2: 0.006595249532295266 Top score values: [0.04338868 0.07714762 0.09796855 0.14017555 0.14090854] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3, 19:1, 19:3, 19:9. Pending: Targets: [ 0.1001732 -0.57504555 -0.86984964 1.83332409 2.10750405 2.15448392 1.81993791 1.27603983 0.4325531 -0.08043819 -0.48355184 1.97782137 0.65894502 0.30639891 1.48669185 -0.32014559 -0.645654 -0.90274345 0.22719639 -0.29287394 -0.07151063 -0.358566 1.30814066 -0.49909675 -0.91431979 -1.28093813 -0.63536748 -0.93563072 -1.10383374 1.18939348 -0.50330726 -0.84179846 -1.07603159 -0.21297811 -0.69575305 -1.09680649 -0.11207013 -0.35298216 -0.38584317 -0.72502942 -0.90643847] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.24140631994923895, 'kernel_inv_bw1': 0.8134641729505723, 'kernel_inv_bw2': 0.0070302629600143204, 'kernel_inv_bw3': 5.8639041470113975, 'kernel_inv_bw4': 1.1427048768692314, 'kernel_inv_bw5': 0.05156797663092838, 'kernel_inv_bw6': 1.0219083373997764, 'kernel_inv_bw7': 0.7573163628088571, 'kernel_inv_bw8': 0.8114423032606065, 'kernel_covariance_scale': 1.133285180700306, 'mean_mean_value': 1.358492378145194} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.217693 ; Training: accuracy=0.292763 ; Validation: accuracy=0.565326 .. parsed-literal:: :class: output config_id 20: Terminating evaluation at 1 Update for config_id 20:1: reward = 0.5653257978723404, crit_val = 0.4346742021276596 Starting get_config[BO] for config_id 21 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.44426918187055103 - self.std = 0.2391545157785912 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 42 Current best is [0.13445378] [21: BO] (29 evaluations) batch_size: 86 dropout_1: 0.29046327939080685 dropout_2: 0.56812963618158 learning_rate: 0.00971849783517772 n_units_1: 128 n_units_2: 123 scale_1: 0.0010000000000000002 scale_2: 0.007699153336710522 Started BO from (top scorer): batch_size: 84 dropout_1: 0.29045243854599945 dropout_2: 0.5681299155477846 learning_rate: 0.007410577045318821 n_units_1: 111 n_units_2: 123 scale_1: 0.11063448065875608 scale_2: 2.7172159701453054 Top score values: [-0.04052008 0.02178 0.0764601 0.1164286 0.12405922] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3, 19:1, 19:3, 19:9, 20:1. Pending: Targets: [ 0.10236402 -0.58102608 -0.87939783 1.85648914 2.13398716 2.18153558 1.84294096 1.29246074 0.43876632 -0.08043309 -0.48842513 2.00273509 0.66789798 0.31108543 1.50566202 -0.32304137 -0.65248902 -0.91268971 0.23092441 -0.29543969 -0.07139749 -0.36192675 1.32495004 -0.50415817 -0.92440615 -1.29546122 -0.64207802 -0.94597498 -1.11621356 1.20476581 -0.50841963 -0.85100718 -1.08807495 -0.21457698 -0.70319436 -1.10910126 -0.11244783 -0.35627533 -0.38953401 -0.73282503 -0.91642944 -0.04012042] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.24978167947420007, 'kernel_inv_bw1': 0.0008003164637168435, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 3.55902353920494, 'kernel_inv_bw4': 1.20011744943782, 'kernel_inv_bw5': 0.0010767226860239716, 'kernel_inv_bw6': 1.332453533412621, 'kernel_inv_bw7': 1.2456797098664971, 'kernel_inv_bw8': 0.7243831048153299, 'kernel_covariance_scale': 1.4911276462414562, 'mean_mean_value': 1.2398239116520988} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.291640 ; Training: accuracy=0.438562 ; Validation: accuracy=0.698339 .. parsed-literal:: :class: output Update for config_id 21:1: reward = 0.6983388704318937, crit_val = 0.3016611295681063 config_id 21: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 0.583856 ; Training: accuracy=0.631453 ; Validation: accuracy=0.780066 Epoch 3 ; Time: 0.865273 ; Training: accuracy=0.670790 ; Validation: accuracy=0.806146 .. parsed-literal:: :class: output Update for config_id 21:3: reward = 0.8061461794019934, crit_val = 0.1938538205980066 config_id 21: Reaches 3, continues to 9 .. parsed-literal:: :class: output Epoch 4 ; Time: 1.156060 ; Training: accuracy=0.699736 ; Validation: accuracy=0.822591 Epoch 5 ; Time: 1.457118 ; Training: accuracy=0.712024 ; Validation: accuracy=0.836545 Epoch 6 ; Time: 1.820592 ; Training: accuracy=0.724559 ; Validation: accuracy=0.847508 Epoch 7 ; Time: 2.165454 ; Training: accuracy=0.736104 ; Validation: accuracy=0.862458 Epoch 8 ; Time: 2.501467 ; Training: accuracy=0.743691 ; Validation: accuracy=0.856478 Epoch 9 ; Time: 2.836660 ; Training: accuracy=0.753670 ; Validation: accuracy=0.854983 .. parsed-literal:: :class: output config_id 21: Terminating evaluation at 9 Update for config_id 21:9: reward = 0.8549833887043189, crit_val = 0.1450166112956811 Starting get_config[BO] for config_id 22 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.4288852711116653 - self.std = 0.2387063356373742 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 45 Current best is [0.13445378] [22: BO] (31 evaluations) batch_size: 8 dropout_1: 0.5597623370437255 dropout_2: 0.43944490965643446 learning_rate: 0.0005004578038988256 n_units_1: 70 n_units_2: 89 scale_1: 8.515949425434965 scale_2: 0.07218167085678881 Started BO from (top scorer): batch_size: 71 dropout_1: 0.5597622717952918 dropout_2: 0.4394453462003949 learning_rate: 0.0009696499292183465 n_units_1: 97 n_units_2: 89 scale_1: 5.416358116563618 scale_2: 0.0010993055764807885 Top score values: [0.10849736 0.11287591 0.14163988 0.14519679 0.16528343] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3, 19:1, 19:3, 19:9, 20:1, 21:1, 21:3, 21:9. Pending: Targets: [ 0.16700323 -0.51766996 -0.81660192 1.92442178 2.20244082 2.25007851 1.91084817 1.35933439 0.50403713 -0.01613709 -0.42489515 2.07094232 0.733599 0.37611652 1.57293597 -0.25920088 -0.58926708 -0.8499563 0.295805 -0.23154738 -0.00708453 -0.29815926 1.3918847 -0.44065773 -0.86169474 -1.23344648 -0.57883653 -0.88330407 -1.05386228 1.27147482 -0.44492719 -0.78815796 -1.02567084 -0.15053284 -0.64006762 -1.04673663 -0.04821194 -0.29249724 -0.32581836 -0.66975392 -0.85370306 0.02425127 -0.53297346 -0.984605 -1.18919617] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.24306654143612832, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00017957180150249163, 'kernel_inv_bw3': 3.6946631814676305, 'kernel_inv_bw4': 1.2534435597612723, 'kernel_inv_bw5': 0.0026664377144599394, 'kernel_inv_bw6': 1.2801691515722537, 'kernel_inv_bw7': 1.20290572647708, 'kernel_inv_bw8': 0.7528836236798467, 'kernel_covariance_scale': 1.393286444244258, 'mean_mean_value': 1.3453057287510302} .. parsed-literal:: :class: output Epoch 1 ; Time: 2.920245 ; Training: accuracy=0.233173 ; Validation: accuracy=0.591184 .. parsed-literal:: :class: output Update for config_id 22:1: reward = 0.5911843876177658, crit_val = 0.4088156123822342 config_id 22: Reaches 1, continues to 3 .. parsed-literal:: :class: output Epoch 2 ; Time: 5.863423 ; Training: accuracy=0.372596 ; Validation: accuracy=0.640478 Epoch 3 ; Time: 8.919775 ; Training: accuracy=0.411555 ; Validation: accuracy=0.660162 .. parsed-literal:: :class: output config_id 22: Terminating evaluation at 3 Update for config_id 22:3: reward = 0.6601615074024226, crit_val = 0.33983849259757737 Starting get_config[BO] for config_id 23 Fitting GP model [GPMXNetModel._posterior_for_state] - self.mean = 0.42656364478733516 - self.std = 0.2339399337653385 BO Algorithm: Generating initial candidates. BO Algorithm: Scoring (and reordering) candidates. BO Algorithm: Selecting final set of candidates. [GPMXNetModel.current_best -- RECOMPUTING] - len(candidates) = 47 Current best is [0.13445378] [23: BO] (25 evaluations) batch_size: 128 dropout_1: 0.3134983827249526 dropout_2: 0.7352587393338911 learning_rate: 0.0008338466269583976 n_units_1: 88 n_units_2: 51 scale_1: 0.7518272595398855 scale_2: 0.019313089787206573 Started BO from (top scorer): batch_size: 85 dropout_1: 0.31349832499476277 dropout_2: 0.7352587949202476 learning_rate: 0.00033486684036548067 n_units_1: 97 n_units_2: 51 scale_1: 3.1394908510247905 scale_2: 0.2124395580502697 Top score values: [0.0300192 0.10144442 0.10614942 0.1195869 0.12029652] Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 4:3, 5:1, 5:3, 5:9, 6:1, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 10:3, 11:1, 11:3, 12:1, 13:1, 13:3, 13:9, 14:1, 14:3, 14:9, 15:1, 16:1, 16:3, 16:9, 17:1, 17:3, 17:9, 18:1, 18:3, 19:1, 19:3, 19:9, 20:1, 21:1, 21:3, 21:9, 22:1, 22:3. Pending: Targets: [ 0.18032986 -0.51829319 -0.82331572 1.97355488 2.25723841 2.30584669 1.95970471 1.39695414 0.52423065 -0.00654185 -0.42362814 2.1230607 0.75846972 0.39370372 1.61490773 -0.25455793 -0.59134905 -0.85734969 0.3117559 -0.226341 0.00269515 -0.29431007 1.43016764 -0.43971187 -0.86932729 -1.24865327 -0.58070599 -0.89137689 -1.06541013 1.30730447 -0.44406832 -0.79429224 -1.03664431 -0.14367584 -0.64318463 -1.0581393 -0.03927021 -0.28853269 -0.32253271 -0.67347577 -0.86117278 0.0346694 -0.53390848 -0.99474177 -1.20350138 -0.07586577 -0.37071547] GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.24680646738407225, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 3.812822596647508, 'kernel_inv_bw4': 1.2834061281882614, 'kernel_inv_bw5': 0.00029318259959548325, 'kernel_inv_bw6': 1.3105072234692994, 'kernel_inv_bw7': 1.1718830444446848, 'kernel_inv_bw8': 0.7597362042547287, 'kernel_covariance_scale': 1.3651439244674628, 'mean_mean_value': 1.371661215909066} .. parsed-literal:: :class: output Epoch 1 ; Time: 0.195983 ; Training: accuracy=0.141941 ; Validation: accuracy=0.393617 .. parsed-literal:: :class: output config_id 23: Terminating evaluation at 1 Update for config_id 23:1: reward = 0.39361702127659576, crit_val = 0.6063829787234043 Analysing the results ~~~~~~~~~~~~~~~~~~~~~ The training history is stored in the ``results_df``, the main fields are the runtime and ``'best'`` (the objective). **Note**: You will get slightly different curves for different pairs of scheduler/searcher, the ``time_out`` here is a bit too short to really see the difference in a significant way (it would be better to set it to >1000s). Generally speaking though, hyperband stopping / promotion + model will tend to significantly outperform other combinations given enough time. .. code:: python results_df.head() .. raw:: html
bracket elapsed_time epoch error eval_time objective runtime searcher_data_size searcher_params_kernel_covariance_scale searcher_params_kernel_inv_bw0 ... searcher_params_kernel_inv_bw7 searcher_params_kernel_inv_bw8 searcher_params_mean_mean_value searcher_params_noise_variance target_epoch task_id time_since_start time_step time_this_iter best
0 0 0.379979 1 0.468750 0.376011 0.531250 0.490698 NaN 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.492744 1.595025e+09 0.431452 0.468750
1 0 0.726648 2 0.344753 0.340125 0.655247 0.837368 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 0.838209 1.595025e+09 0.346649 0.344753
2 0 1.067602 3 0.305314 0.339053 0.694686 1.178321 1.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.179219 1.595025e+09 0.340955 0.305314
3 0 1.414792 4 0.288937 0.343092 0.711063 1.525511 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.526258 1.595025e+09 0.347189 0.288937
4 0 1.878843 5 0.273061 0.462348 0.726939 1.989563 2.0 1.0 1.0 ... 1.0 1.0 0.0 0.001 9 0 1.990429 1.595025e+09 0.464052 0.273061

5 rows × 26 columns

.. code:: python import matplotlib.pyplot as plt plt.figure(figsize=(12, 8)) runtime = results_df['runtime'].values objective = results_df['best'].values plt.plot(runtime, objective, lw=2) plt.xticks(fontsize=12) plt.xlim(0, 120) plt.ylim(0, 0.5) plt.yticks(fontsize=12) plt.xlabel("Runtime [s]", fontsize=14) plt.ylabel("Objective", fontsize=14) .. parsed-literal:: :class: output Text(0, 0.5, 'Objective') .. figure:: output_mlp_cb387f_18_1.png Diving Deeper ------------- Now, you are ready to try HPO on your own machine learning models (if you use PyTorch, have a look at :ref:`sec_customstorch`). While AutoGluon comes with well-chosen defaults, it can pay off to tune it to your specific needs. Here are some tips which may come useful. Logging the Search Progress ~~~~~~~~~~~~~~~~~~~~~~~~~~~ First, it is a good idea in general to switch on ``debug_log``, which outputs useful information about the search progress. This is already done in the example above. The outputs show which configurations are chosen, stopped, or promoted. For BO and BOHB, a range of information is displayed for every ``get_config`` decision. This log output is very useful in order to figure out what is going on during the search. Configuring ``HyperbandScheduler`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The most important knobs to turn with ``HyperbandScheduler`` are ``max_t``, ``grace_period``, ``reduction_factor``, ``brackets``, and ``type``. The first three determine the rung levels at which stopping or promotion decisions are being made. - The maximum resource level ``max_t`` (usually, resource equates to epochs, so ``max_t`` is the maximum number of training epochs) is typically hardcoded in ``train_fn`` passed to the scheduler (this is ``run_mlp_openml`` in the example above). As already noted above, the value is best fixed in the ``ag.args`` decorator as ``epochs=XYZ``, it can then be accessed as ``args.epochs`` in the ``train_fn`` code. If this is done, you do not have to pass ``max_t`` when creating the scheduler. - ``grace_period`` and ``reduction_factor`` determine the rung levels, which are ``grace_period``, ``grace_period * reduction_factor``, ``grace_period * (reduction_factor ** 2)``, etc. All rung levels must be less or equal than ``max_t``. It is recommended to make ``max_t`` equal to the largest rung level. For example, if ``grace_period = 1``, ``reduction_factor = 3``, it is in general recommended to use ``max_t = 9``, ``max_t = 27``, or ``max_t = 81``. Choosing a ``max_t`` value "off the grid" works against the successive halving principle that the total resources spent in a rung should be roughly equal between rungs. If in the example above, you set ``max_t = 10``, about a third of configurations reaching 9 epochs are allowed to proceed, but only for one more epoch. - With ``reduction_factor``, you tune the extent to which successive halving filtering is applied. The larger this integer, the fewer configurations make it to higher number of epochs. Values 2, 3, 4 are commonly used. - Finally, ``grace_period`` should be set to the smallest resource (number of epochs) for which you expect any meaningful differentiation between configurations. While ``grace_period = 1`` should always be explored, it may be too low for any meaningful stopping decisions to be made at the first rung. - ``brackets`` sets the maximum number of brackets in Hyperband (make sure to study the Hyperband paper or follow-ups for details). For ``brackets = 1``, you are running successive halving (single bracket). Higher brackets have larger effective ``grace_period`` values (so runs are not stopped until later), yet are also chosen with less probability. We recommend to always consider successive halving (``brackets = 1``) in a comparison. - Finally, with ``type`` (values ``stopping``, ``promotion``) you are choosing different ways of extending successive halving scheduling to the asynchronous case. The method for the default ``stopping`` is simpler and seems to perform well, but ``promotion`` is more careful promoting configurations to higher resource levels, which can work better in some cases. Asynchronous BOHB ~~~~~~~~~~~~~~~~~ Finally, here are some ideas for tuning asynchronous BOHB, apart from tuning its ``HyperbandScheduling`` component. You need to pass these options in ``search_options``. - We support a range of different surrogate models over the criterion functions across resource levels. All of them are jointly dependent Gaussian process models, meaning that data collected at all resource levels are modelled together. The surrogate model is selected by ``gp_resource_kernel``, values are ``matern52``, ``matern52-res-warp``, ``exp-decay-sum``, ``exp-decay-combined``, ``exp-decay-delta1``. These are variants of either a joint Matern 5/2 kernel over configuration and resource, or the exponential decay model. Details about the latter can be found `here `__. - Fitting a Gaussian process surrogate model to data encurs a cost which scales cubically with the number of datapoints. When applied to expensive deep learning workloads, even multi-fidelity asynchronous BOHB is rarely running up more than 100 observations or so (across all rung levels and brackets), and the GP computations are subdominant. However, if you apply it to cheaper ``train_fn`` and find yourself beyond 2000 total evaluations, the cost of GP fitting can become painful. In such a situation, you can explore the options ``opt_skip_period`` and ``opt_skip_num_max_resource``. The basic idea is as follows. By far the most expensive part of a ``get_config`` call (picking the next configuration) is the refitting of the GP model to past data (this entails re-optimizing hyperparameters of the surrogate model itself). The options allow you to skip this expensive step for most ``get_config`` calls, after some initial period. Check the docstrings for details about these options. If you find yourself in such a situation and gain experience with these skipping features, make sure to contact the AutoGluon developers -- we would love to learn about your use case.