Getting started with Advanced HPO Algorithms¶
Loading libraries¶
# Basic utils for folder manipulations etc
import time
import multiprocessing # to count the number of CPUs available
# External tools to load and process data
import numpy as np
import pandas as pd
# MXNet (NeuralNets)
import mxnet as mx
from mxnet import gluon, autograd
from mxnet.gluon import nn
# AutoGluon and HPO tools
import autogluon as ag
from autogluon.utils import load_and_split_openml_data
Check the version of MxNet, you should be fine with version >= 1.5
mx.__version__
/var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above. and should_run_async(code)
'1.7.0'
You can also check the version of AutoGluon and the specific commit and check that it matches what you want.
ag.__version__
'0.0.13b20200814'
Hyperparameter Optimization of a 2-layer MLP¶
Setting up the context¶
Here we declare a few “environment variables” setting the context for what we’re doing
OPENML_TASK_ID = 6 # describes the problem we will tackle
RATIO_TRAIN_VALID = 0.33 # split of the training data used for validation
RESOURCE_ATTR_NAME = 'epoch' # how do we measure resources (will become clearer further)
REWARD_ATTR_NAME = 'objective' # how do we measure performance (will become clearer further)
NUM_CPUS = multiprocessing.cpu_count()
Preparing the data¶
We will use a multi-way classification task from OpenML. Data preparation includes:
Missing values are imputed, using the ‘mean’ strategy of
sklearn.impute.SimpleImputerSplit training set into training and validation
Standardize inputs to mean 0, variance 1
X_train, X_valid, y_train, y_valid, n_classes = load_and_split_openml_data(
OPENML_TASK_ID, RATIO_TRAIN_VALID, download_from_openml=False)
n_classes
Downloading ./org/openml/www/datasets/6/dataset.arff from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.arff...
100%|██████████| 704/704 [00:00<00:00, 45583.22KB/s]
Downloading ./org/openml/www/datasets/6/dataset.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/dataset.pkl.py3...
100%|██████████| 2521/2521 [00:00<00:00, 49632.66KB/s]
Downloading ./org/openml/www/datasets/6/description.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/description.xml...
3KB [00:00, 4126.90KB/s]
Downloading ./org/openml/www/datasets/6/features.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/features.xml...
8KB [00:00, 8242.31KB/s]
Downloading ./org/openml/www/datasets/6/qualities.xml from https://autogluon.s3.amazonaws.com/org/openml/www/datasets/6/qualities.xml...
15KB [00:00, 17260.51KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.arff from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.arff...
2998KB [00:00, 28504.03KB/s]
Downloading ./org/openml/www/tasks/6/datasplits.pkl.py3 from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/datasplits.pkl.py3...
881KB [00:00, 43908.72KB/s]
Downloading ./org/openml/www/tasks/6/task.xml from https://autogluon.s3.amazonaws.com/org/openml/www/tasks/6/task.xml...
3KB [00:00, 3280.22KB/s]
Data pickle file already exists and is up to date.
Data pickle file already exists and is up to date.
26
The problem has 26 classes.
Declaring a model specifying a hyperparameter space with AutoGluon¶
Two layer MLP where we optimize over:
the number of units on the first layer
the number of units on the second layer
the dropout rate after each layer
the learning rate
the scaling
the
@ag.argsdecorator allows us to specify the space we will optimize over, this matches the ConfigSpace syntax
The body of the function run_mlp_openml is pretty simple:
it reads the hyperparameters given via the decorator
it defines a 2 layer MLP with dropout
it declares a trainer with the ‘adam’ loss function and a provided learning rate
it trains the NN with a number of epochs (most of that is boilerplate code from
mxnet)the
reporterat the end is used to keep track of training history in the hyperparameter optimization
Note: The number of epochs and the hyperparameter space are reduced to make for a shorter experiment
@ag.args(n_units_1=ag.space.Int(lower=16, upper=128),
n_units_2=ag.space.Int(lower=16, upper=128),
dropout_1=ag.space.Real(lower=0, upper=.75),
dropout_2=ag.space.Real(lower=0, upper=.75),
learning_rate=ag.space.Real(lower=1e-6, upper=1, log=True),
batch_size=ag.space.Int(lower=8, upper=128),
scale_1=ag.space.Real(lower=0.001, upper=10, log=True),
scale_2=ag.space.Real(lower=0.001, upper=10, log=True),
epochs=9)
def run_mlp_openml(args, reporter, **kwargs):
# Time stamp for elapsed_time
ts_start = time.time()
# Unwrap hyperparameters
n_units_1 = args.n_units_1
n_units_2 = args.n_units_2
dropout_1 = args.dropout_1
dropout_2 = args.dropout_2
scale_1 = args.scale_1
scale_2 = args.scale_2
batch_size = args.batch_size
learning_rate = args.learning_rate
ctx = mx.cpu()
net = nn.Sequential()
with net.name_scope():
# Layer 1
net.add(nn.Dense(n_units_1, activation='relu',
weight_initializer=mx.initializer.Uniform(scale=scale_1)))
# Dropout
net.add(gluon.nn.Dropout(dropout_1))
# Layer 2
net.add(nn.Dense(n_units_2, activation='relu',
weight_initializer=mx.initializer.Uniform(scale=scale_2)))
# Dropout
net.add(gluon.nn.Dropout(dropout_2))
# Output
net.add(nn.Dense(n_classes))
net.initialize(ctx=ctx)
trainer = gluon.Trainer(net.collect_params(), 'adam',
{'learning_rate': learning_rate})
for epoch in range(args.epochs):
ts_epoch = time.time()
train_iter = mx.io.NDArrayIter(
data={'data': X_train},
label={'label': y_train},
batch_size=batch_size,
shuffle=True)
valid_iter = mx.io.NDArrayIter(
data={'data': X_valid},
label={'label': y_valid},
batch_size=batch_size,
shuffle=False)
metric = mx.metric.Accuracy()
loss = gluon.loss.SoftmaxCrossEntropyLoss()
for batch in train_iter:
data = batch.data[0].as_in_context(ctx)
label = batch.label[0].as_in_context(ctx)
with autograd.record():
output = net(data)
L = loss(output, label)
L.backward()
trainer.step(data.shape[0])
metric.update([label], [output])
name, train_acc = metric.get()
metric = mx.metric.Accuracy()
for batch in valid_iter:
data = batch.data[0].as_in_context(ctx)
label = batch.label[0].as_in_context(ctx)
output = net(data)
metric.update([label], [output])
name, val_acc = metric.get()
print('Epoch %d ; Time: %f ; Training: %s=%f ; Validation: %s=%f' % (
epoch + 1, time.time() - ts_start, name, train_acc, name, val_acc))
ts_now = time.time()
eval_time = ts_now - ts_epoch
elapsed_time = ts_now - ts_start
# The resource reported back (as 'epoch') is the number of epochs
# done, starting at 1
reporter(
epoch=epoch + 1,
objective=float(val_acc),
eval_time=eval_time,
time_step=ts_now,
elapsed_time=elapsed_time)
/var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above. and should_run_async(code)
Note: The annotation epochs=9 specifies the maximum number of
epochs for training. It becomes available as args.epochs.
Importantly, it is also processed by HyperbandScheduler below in
order to set its max_t attribute.
Recommendation: Whenever writing training code to be passed as
train_fn to a scheduler, if this training code reports a resource
(or time) attribute, the corresponding maximum resource value should be
included in train_fn.args:
If the resource attribute (
time_attrof scheduler) intrain_fnisepoch, make sure to includeepochs=XYZin the annotation. This allows the scheduler to readmax_tfromtrain_fn.args.epochs. This case corresponds to our example here.If the resource attribute is something else than
epoch, you can also include the annotationmax_t=XYZ, which allows the scheduler to readmax_tfromtrain_fn.args.max_t.
Annotating the training function by the correct value for max_t
simplifies scheduler creation (since max_t does not have to be
passed), and avoids inconsistencies between train_fn and the
scheduler.
Running the Hyperparameter Optimization¶
You can use the following schedulers:
FIFO (
fifo)Hyperband (either the stopping (
hbs) or promotion (hbp) variant)
And the following searchers:
Random search (
random)Gaussian process based Bayesian optimization (
bayesopt)SkOpt Bayesian optimization (
skopt; only with FIFO scheduler)
Note that the method known as (asynchronous) Hyperband is using random
search. Combining Hyperband scheduling with the bayesopt searcher
uses a novel method called asynchronous BOHB.
Pick the combination you’re interested in (doing the full experiment
takes around 120 seconds, see the time_out parameter), running
everything with multiple runs can take a fair bit of time. In real life,
you will want to choose a larger time_out in order to obtain good
performance.
SCHEDULER = "hbs"
SEARCHER = "bayesopt"
def compute_error(df):
return 1.0 - df["objective"]
def compute_runtime(df, start_timestamp):
return df["time_step"] - start_timestamp
def process_training_history(task_dicts, start_timestamp,
runtime_fn=compute_runtime,
error_fn=compute_error):
task_dfs = []
for task_id in task_dicts:
task_df = pd.DataFrame(task_dicts[task_id])
task_df = task_df.assign(task_id=task_id,
runtime=runtime_fn(task_df, start_timestamp),
error=error_fn(task_df),
target_epoch=task_df["epoch"].iloc[-1])
task_dfs.append(task_df)
result = pd.concat(task_dfs, axis="index", ignore_index=True, sort=True)
# re-order by runtime
result = result.sort_values(by="runtime")
# calculate incumbent best -- the cumulative minimum of the error.
result = result.assign(best=result["error"].cummin())
return result
resources = dict(num_cpus=NUM_CPUS, num_gpus=0)
search_options = {
'num_init_random': 2,
'debug_log': True}
if SCHEDULER == 'fifo':
myscheduler = ag.scheduler.FIFOScheduler(
run_mlp_openml,
resource=resources,
searcher=SEARCHER,
search_options=search_options,
time_out=120,
time_attr=RESOURCE_ATTR_NAME,
reward_attr=REWARD_ATTR_NAME)
else:
# This setup uses rung levels at 1, 3, 9 epochs. We just use a single
# bracket, so this is in fact successive halving (Hyperband would use
# more than 1 bracket).
# Also note that since we do not use the max_t argument of
# HyperbandScheduler, this value is obtained from train_fn.args.epochs.
sch_type = 'stopping' if SCHEDULER == 'hbs' else 'promotion'
myscheduler = ag.scheduler.HyperbandScheduler(
run_mlp_openml,
resource=resources,
searcher=SEARCHER,
search_options=search_options,
time_out=120,
time_attr=RESOURCE_ATTR_NAME,
reward_attr=REWARD_ATTR_NAME,
type=sch_type,
grace_period=1,
reduction_factor=3,
brackets=1)
# run tasks
myscheduler.run()
myscheduler.join_jobs()
results_df = process_training_history(
myscheduler.training_history.copy(),
start_timestamp=myscheduler._start_time)
max_t = 9, as inferred from train_fn.args
scheduler_options: Key 'resume': Imputing default value False
scheduler_options: Key 'keep_size_ratios': Imputing default value False
scheduler_options: Key 'maxt_pending': Imputing default value False
scheduler_options: Key 'searcher_data': Imputing default value rungs
scheduler_options: Key 'do_snapshots': Imputing default value False
scheduler_options: Key 'visualizer': Imputing default value none
scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60
scheduler_options: Key 'delay_get_config': Imputing default value True
search_options: Key 'random_seed': Imputing default value 31415927
search_options: Key 'opt_skip_init_length': Imputing default value 150
search_options: Key 'opt_skip_period': Imputing default value 1
search_options: Key 'profiler': Imputing default value False
search_options: Key 'opt_maxiter': Imputing default value 50
search_options: Key 'opt_nstarts': Imputing default value 2
search_options: Key 'opt_warmstart': Imputing default value False
search_options: Key 'opt_verbose': Imputing default value False
search_options: Key 'opt_debug_writer': Imputing default value False
search_options: Key 'num_fantasy_samples': Imputing default value 20
search_options: Key 'num_init_candidates': Imputing default value 250
search_options: Key 'initial_scoring': Imputing default value thompson_indep
search_options: Key 'first_is_default': Imputing default value True
search_options: Key 'opt_skip_num_max_resource': Imputing default value False
search_options: Key 'gp_resource_kernel': Imputing default value matern52
search_options: Key 'resource_acq': Imputing default value bohb
[GPMultiFidelitySearcher.__init__]
- acquisition_class = <class 'autogluon.searcher.bayesopt.models.nphead_acqfunc.EIAcquisitionFunction'>
- local_minimizer_class = <class 'autogluon.searcher.bayesopt.tuning_algorithms.bo_algorithm_components.LBFGSOptimizeAcquisition'>
- num_initial_candidates = 250
- num_initial_random_choices = 2
- initial_scoring = thompson_indep
- first_is_default = True
Starting Experiments
Num of Finished Tasks is 0
Time out (secs) is 120
Starting get_config[random] for config_id 0
Start with default config:
{'batch_size': 68, 'dropout_1': 0.375, 'dropout_2': 0.375, 'learning_rate': 0.001, 'n_units_1': 72, 'n_units_2': 72, 'scale_1': 0.1, 'scale_2': 0.1}
[0: random]
batch_size: 68
dropout_1: 0.375
dropout_2: 0.375
learning_rate: 0.001
n_units_1: 72
n_units_2: 72
scale_1: 0.1
scale_2: 0.1
/var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/distributed/worker.py:3385: UserWarning: Large object of size 1.30 MB detected in task graph:
(<function run_mlp_openml at 0x7fd7e8f01200>, {'ar ... sReporter}, [])
Consider scattering large objects ahead of time
with client.scatter to reduce scheduler burden and
keep data on workers
future = client.submit(func, big_data) # bad
big_future = client.scatter(big_data) # good
future = client.submit(func, big_future) # good
% (format_bytes(len(b)), s)
Epoch 1 ; Time: 0.468991 ; Training: accuracy=0.260079 ; Validation: accuracy=0.531250
Update for config_id 0:1: reward = 0.53125, crit_val = 0.46875
config_id 0: Reaches 1, continues to 3
Epoch 2 ; Time: 0.874065 ; Training: accuracy=0.496365 ; Validation: accuracy=0.655247
Epoch 3 ; Time: 1.273944 ; Training: accuracy=0.559650 ; Validation: accuracy=0.694686
Update for config_id 0:3: reward = 0.6946858288770054, crit_val = 0.30531417112299464
config_id 0: Reaches 3, continues to 9
Epoch 4 ; Time: 1.674376 ; Training: accuracy=0.588896 ; Validation: accuracy=0.711063
Epoch 5 ; Time: 2.075989 ; Training: accuracy=0.609385 ; Validation: accuracy=0.726939
Epoch 6 ; Time: 2.478248 ; Training: accuracy=0.628139 ; Validation: accuracy=0.745321
Epoch 7 ; Time: 2.882975 ; Training: accuracy=0.641193 ; Validation: accuracy=0.750501
Epoch 8 ; Time: 3.328253 ; Training: accuracy=0.653751 ; Validation: accuracy=0.763202
Epoch 9 ; Time: 3.746055 ; Training: accuracy=0.665482 ; Validation: accuracy=0.766043
config_id 0: Terminating evaluation at 9
Update for config_id 0:9: reward = 0.766042780748663, crit_val = 0.23395721925133695
Starting get_config[random] for config_id 1
[1: random]
batch_size: 49
dropout_1: 0.6118880110946948
dropout_2: 0.5692033531564655
learning_rate: 0.00010310483923558212
n_units_1: 126
n_units_2: 93
scale_1: 0.17664689253248303
scale_2: 0.5396711634456218
Epoch 1 ; Time: 0.607219 ; Training: accuracy=0.048005 ; Validation: accuracy=0.111743
config_id 1: Terminating evaluation at 1
Update for config_id 1:1: reward = 0.11174305787888926, crit_val = 0.8882569421211107
Starting get_config[BO] for config_id 2
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4740695831238606
- self.std = 0.25382661537049356
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 4
Current best is [0.46875]
[2: BO] (1 evaluations)
batch_size: 50
dropout_1: 0.21593234798980557
dropout_2: 0.2842099148996821
learning_rate: 0.12426792150054146
n_units_1: 61
n_units_2: 79
scale_1: 9.791854554777688
scale_2: 4.705654410614348
Started BO from (top scorer):
batch_size: 50
dropout_1: 0.21593234798980557
dropout_2: 0.2842099148996821
learning_rate: 0.12426792150054146
n_units_1: 61
n_units_2: 79
scale_1: 9.791854554777688
scale_2: 4.705654410614348
Top score values: [-0.27440679 0.04673507 0.04701008 0.10393977 0.10463477]
Labeled: 0:1, 0:3, 0:9, 1:1. Pending:
Targets: [-0.02095755 -0.66484522 -0.94597 1.63177277]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.7572453035969371, 'kernel_inv_bw1': 0.9003213708321401, 'kernel_inv_bw2': 0.9671934120293088, 'kernel_inv_bw3': 3.9351561554818266, 'kernel_inv_bw4': 26.15373099715838, 'kernel_inv_bw5': 1.6599386110717227, 'kernel_inv_bw6': 4.378455276759229, 'kernel_inv_bw7': 35.45306859774744, 'kernel_inv_bw8': 1.6274469437660462, 'kernel_covariance_scale': 0.8177941757954876, 'mean_mean_value': 0.3168762072733891}
Epoch 1 ; Time: 0.625895 ; Training: accuracy=0.072975 ; Validation: accuracy=0.038319
config_id 2: Terminating evaluation at 1
Update for config_id 2:1: reward = 0.03831932773109244, crit_val = 0.9616806722689075
Starting get_config[BO] for config_id 3
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.57159180095287
- self.std = 0.2993070203278022
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 5
Current best is [0.46875]
[3: BO] (1 evaluations)
batch_size: 22
dropout_1: 0.6341443733804044
dropout_2: 0.04086583670464042
learning_rate: 0.12818863950369533
n_units_1: 19
n_units_2: 101
scale_1: 0.026860146662962745
scale_2: 4.431236744525342
Started BO from (top scorer):
batch_size: 22
dropout_1: 0.6341443733804044
dropout_2: 0.04086583670464042
learning_rate: 0.12818863950369533
n_units_1: 19
n_units_2: 101
scale_1: 0.026860146662962745
scale_2: 4.431236744525342
Top score values: [-0.02496145 0.00037188 0.04140351 0.12359639 0.12398311]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1. Pending:
Targets: [-0.3435997 -0.88964712 -1.12805433 1.05799437 1.30330679]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 6.2975129900938756, 'kernel_inv_bw1': 1.6684536123391513, 'kernel_inv_bw2': 1.817414219491645, 'kernel_inv_bw3': 1.4215981511230344, 'kernel_inv_bw4': 57.091085169764185, 'kernel_inv_bw5': 3.604498367053083, 'kernel_inv_bw6': 100.00000000000004, 'kernel_inv_bw7': 99.92474107423274, 'kernel_inv_bw8': 1.3393578509128972, 'kernel_covariance_scale': 0.7384716747026111, 'mean_mean_value': 0.38060991018158125}
Epoch 1 ; Time: 1.300784 ; Training: accuracy=0.041812 ; Validation: accuracy=0.039562
config_id 3: Terminating evaluation at 1
Update for config_id 3:1: reward = 0.03956228956228956, crit_val = 0.9604377104377104
Starting get_config[BO] for config_id 4
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6363994525336767
- self.std = 0.30927991991886417
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 6
Current best is [0.46875]
[4: BO] (6 evaluations)
batch_size: 110
dropout_1: 0.5744977554712186
dropout_2: 0.7047747941055489
learning_rate: 3.6223201928237886e-05
n_units_1: 92
n_units_2: 92
scale_1: 0.06664854483482407
scale_2: 0.022478620206736768
Started BO from (top scorer):
batch_size: 110
dropout_1: 0.5744977553473402
dropout_2: 0.7047748951791846
learning_rate: 3.6221732543522435e-05
n_units_1: 92
n_units_2: 92
scale_1: 0.06664854527483652
scale_2: 0.017643835128767108
Top score values: [-0.02863203 0.00599656 0.042477 0.05395405 0.06651693]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1. Pending:
Targets: [-0.54206381 -1.07050364 -1.30122328 0.81433508 1.05173727 1.04771838]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0001511285849645314, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.001199968812262379, 'kernel_inv_bw3': 0.00926838083535284, 'kernel_inv_bw4': 0.00010293036281036059, 'kernel_inv_bw5': 0.00017461620414159357, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 4.272493402708928, 'kernel_inv_bw8': 1.3164718962359343, 'kernel_covariance_scale': 0.7976260528462709, 'mean_mean_value': 0.023357588254215324}
Epoch 1 ; Time: 0.304145 ; Training: accuracy=0.042066 ; Validation: accuracy=0.109933
config_id 4: Terminating evaluation at 1
Update for config_id 4:1: reward = 0.10993265993265994, crit_val = 0.89006734006734
Starting get_config[BO] for config_id 5
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.6726377221813429
- self.std = 0.2997807115320194
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 7
Current best is [0.46875]
[5: BO] (8 evaluations)
batch_size: 106
dropout_1: 0.38750926166961597
dropout_2: 0.04489600208822606
learning_rate: 0.0026794725095178663
n_units_1: 93
n_units_2: 20
scale_1: 0.48508438196776654
scale_2: 0.03480880867048884
Started BO from (top scorer):
batch_size: 106
dropout_1: 0.38750926119776946
dropout_2: 0.04489600246106559
learning_rate: 0.002265665895170696
n_units_1: 93
n_units_2: 20
scale_1: 0.4850843918016997
scale_2: 0.03481291816391773
Top score values: [0.18489824 0.20828588 0.23691933 0.2416328 0.24992744]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1. Pending:
Targets: [-0.68012288 -1.22530749 -1.46333799 0.71925648 0.96418128 0.96003504
0.72529556]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00011518068128664455, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.555364761942682, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00017086339060469206, 'kernel_inv_bw7': 0.035729865892209396, 'kernel_inv_bw8': 1.3111759816687758, 'kernel_covariance_scale': 0.6932464484239275, 'mean_mean_value': 0.1222573690401266}
Epoch 1 ; Time: 0.330590 ; Training: accuracy=0.355842 ; Validation: accuracy=0.601787
Update for config_id 5:1: reward = 0.6017874875868917, crit_val = 0.3982125124131083
config_id 5: Reaches 1, continues to 3
Epoch 2 ; Time: 0.604294 ; Training: accuracy=0.574892 ; Validation: accuracy=0.681397
Epoch 3 ; Time: 0.870683 ; Training: accuracy=0.622559 ; Validation: accuracy=0.727574
Update for config_id 5:3: reward = 0.7275736511089044, crit_val = 0.2724263488910956
config_id 5: Reaches 3, continues to 9
Epoch 4 ; Time: 1.146298 ; Training: accuracy=0.648047 ; Validation: accuracy=0.750083
Epoch 5 ; Time: 1.408732 ; Training: accuracy=0.672459 ; Validation: accuracy=0.760344
Epoch 6 ; Time: 1.671629 ; Training: accuracy=0.684542 ; Validation: accuracy=0.766137
Epoch 7 ; Time: 1.933792 ; Training: accuracy=0.701258 ; Validation: accuracy=0.792453
Epoch 8 ; Time: 2.195195 ; Training: accuracy=0.709782 ; Validation: accuracy=0.798080
Epoch 9 ; Time: 2.459531 ; Training: accuracy=0.713009 ; Validation: accuracy=0.812479
config_id 5: Terminating evaluation at 9
Update for config_id 5:9: reward = 0.8124793114862628, crit_val = 0.18752068851373715
Starting get_config[BO] for config_id 6
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5566623605087341
- self.std = 0.3107077976680413
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 10
Current best is [0.39821251]
[6: BO] (8 evaluations)
batch_size: 100
dropout_1: 0.4870263486471881
dropout_2: 0.59113405302611
learning_rate: 0.0051374723840914805
n_units_1: 105
n_units_2: 107
scale_1: 0.2432746016644627
scale_2: 3.384577528513164
Started BO from (top scorer):
batch_size: 100
dropout_1: 0.4870263426644337
dropout_2: 0.5911340539952046
learning_rate: 0.009911435239249514
n_units_1: 105
n_units_2: 107
scale_1: 0.24327462238842057
scale_2: 3.38457748137845
Top score values: [0.19986271 0.27163002 0.37278358 0.37794283 0.37953738]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9. Pending:
Targets: [-0.28294224 -0.80895359 -1.03861295 1.06722324 1.30353443 1.29953401
1.07304993 -0.50996418 -0.91480167 -1.18806697]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0005635684715621877, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.720114182169939, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.002059715378214448, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.1165087337394366, 'kernel_covariance_scale': 0.6850617791776692, 'mean_mean_value': 0.3838207647871803}
Epoch 1 ; Time: 0.337567 ; Training: accuracy=0.281405 ; Validation: accuracy=0.649833
Update for config_id 6:1: reward = 0.6498333333333334, crit_val = 0.3501666666666666
config_id 6: Reaches 1, continues to 3
Epoch 2 ; Time: 0.617234 ; Training: accuracy=0.446942 ; Validation: accuracy=0.693167
Epoch 3 ; Time: 0.887761 ; Training: accuracy=0.491240 ; Validation: accuracy=0.712833
config_id 6: Terminating evaluation at 3
Update for config_id 6:3: reward = 0.7128333333333333, crit_val = 0.2871666666666667
Starting get_config[BO] for config_id 7
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5169964115350562
- self.std = 0.297458835498879
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 12
Current best is [0.35016667]
[7: BO] (5 evaluations)
batch_size: 70
dropout_1: 0.6131746934906158
dropout_2: 0.022967783090604488
learning_rate: 0.008313446421825743
n_units_1: 24
n_units_2: 99
scale_1: 0.0028554704441241295
scale_2: 0.011971504396868799
Started BO from (top scorer):
batch_size: 70
dropout_1: 0.6131746914586562
dropout_2: 0.022967791041196395
learning_rate: 0.007721593613613116
n_units_1: 24
n_units_2: 99
scale_1: 0.0028554705455487786
scale_2: 0.011971504865511351
Top score values: [0.25898139 0.30766019 0.30984522 0.31771857 0.32951799]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3. Pending:
Targets: [-0.16219525 -0.71163541 -0.9515239 1.24810725 1.49494386 1.49076526
1.25419347 -0.39932886 -0.82219801 -1.10763468 -0.56084986 -0.77264387]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00018498994662793233, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00017831718840405247, 'kernel_inv_bw3': 6.785443295549769, 'kernel_inv_bw4': 0.0008660779309394507, 'kernel_inv_bw5': 0.0002996293975555951, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00015743086560692838, 'kernel_inv_bw8': 1.1136085510227574, 'kernel_covariance_scale': 0.6935746305208051, 'mean_mean_value': 0.45824701788781513}
Epoch 1 ; Time: 0.466277 ; Training: accuracy=0.271263 ; Validation: accuracy=0.543697
Update for config_id 7:1: reward = 0.5436974789915966, crit_val = 0.4563025210084034
config_id 7: Reaches 1, continues to 3
Epoch 2 ; Time: 0.867361 ; Training: accuracy=0.386210 ; Validation: accuracy=0.577479
Epoch 3 ; Time: 1.347637 ; Training: accuracy=0.424030 ; Validation: accuracy=0.582857
config_id 7: Terminating evaluation at 3
Update for config_id 7:3: reward = 0.5828571428571429, crit_val = 0.41714285714285715
Starting get_config[BO] for config_id 8
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5055287368979954
- self.std = 0.27692114408851204
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 14
Current best is [0.35016667]
[8: BO] (3 evaluations)
batch_size: 112
dropout_1: 0.41258052357180974
dropout_2: 0.07535360975884835
learning_rate: 1.0000000000000004e-06
n_units_1: 40
n_units_2: 33
scale_1: 0.1918197311813462
scale_2: 0.10218151730817446
Started BO from (top scorer):
batch_size: 112
dropout_1: 0.4125805242468248
dropout_2: 0.07535360980344002
learning_rate: 2.48539902503145e-06
n_units_1: 40
n_units_2: 33
scale_1: 0.1918197310941
scale_2: 0.1021815173022151
Top score values: [0.30319667 0.30627468 0.31694949 0.33985494 0.34255644]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3. Pending:
Targets: [-0.13281303 -0.72300209 -0.98068177 1.38208372 1.64722682 1.64273831
1.38862131 -0.38753352 -0.8417645 -1.14837041 -0.56103361 -0.7885352
-0.17776258 -0.31917346]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.0006690012947653791, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.764899094740552, 'kernel_inv_bw4': 0.0005796463402060587, 'kernel_inv_bw5': 0.000684237942280471, 'kernel_inv_bw6': 0.00028425765435826007, 'kernel_inv_bw7': 0.0001024953751392057, 'kernel_inv_bw8': 1.0593295856674674, 'kernel_covariance_scale': 0.7444010436536751, 'mean_mean_value': 0.6096884310053552}
Epoch 1 ; Time: 0.405703 ; Training: accuracy=0.037616 ; Validation: accuracy=0.032738
config_id 8: Terminating evaluation at 1
Update for config_id 8:1: reward = 0.03273809523809524, crit_val = 0.9672619047619048
Starting get_config[BO] for config_id 9
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.5363109480889227
- self.std = 0.29127064516524426
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 15
Current best is [0.35016667]
[9: BO] (8 evaluations)
batch_size: 16
dropout_1: 0.49382843382784947
dropout_2: 0.17577032493560404
learning_rate: 0.004241709322261239
n_units_1: 75
n_units_2: 24
scale_1: 0.00112063957179567
scale_2: 0.013201132182141391
Started BO from (top scorer):
batch_size: 16
dropout_1: 0.4938284333967552
dropout_2: 0.17577032689555921
learning_rate: 0.0044596526917615315
n_units_1: 75
n_units_2: 24
scale_1: 0.0011206396260318283
scale_2: 0.01320113262596052
Top score values: [0.34243971 0.34413616 0.34417665 0.34515598 0.34564232]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1. Pending:
Targets: [-0.23195248 -0.79306576 -1.03805081 1.20831261 1.46039339 1.45612601
1.21452813 -0.47412411 -0.90597732 -1.19747824 -0.6390767 -0.85537038
-0.27468757 -0.40913183 1.47955506]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 7.678950645138259, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010000000000000009, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010001133434562772, 'kernel_inv_bw8': 1.0241379286848398, 'kernel_covariance_scale': 0.6939330217554913, 'mean_mean_value': 0.6890140220227132}
Epoch 1 ; Time: 1.796589 ; Training: accuracy=0.358256 ; Validation: accuracy=0.618112
Update for config_id 9:1: reward = 0.618111559139785, crit_val = 0.381888440860215
config_id 9: Reaches 1, continues to 3
Epoch 2 ; Time: 3.533327 ; Training: accuracy=0.533156 ; Validation: accuracy=0.696069
Epoch 3 ; Time: 5.221506 ; Training: accuracy=0.579824 ; Validation: accuracy=0.747144
Update for config_id 9:3: reward = 0.7471438172043011, crit_val = 0.2528561827956989
config_id 9: Reaches 3, continues to 9
Epoch 4 ; Time: 6.953095 ; Training: accuracy=0.598558 ; Validation: accuracy=0.758401
Epoch 5 ; Time: 8.659162 ; Training: accuracy=0.627321 ; Validation: accuracy=0.766297
Epoch 6 ; Time: 10.381943 ; Training: accuracy=0.640666 ; Validation: accuracy=0.764785
Epoch 7 ; Time: 12.095024 ; Training: accuracy=0.644977 ; Validation: accuracy=0.788306
Epoch 8 ; Time: 13.815813 ; Training: accuracy=0.654426 ; Validation: accuracy=0.799563
Epoch 9 ; Time: 15.536620 ; Training: accuracy=0.662218 ; Validation: accuracy=0.799395
config_id 9: Terminating evaluation at 9
Update for config_id 9:9: reward = 0.7993951612903226, crit_val = 0.20060483870967738
Starting get_config[BO] for config_id 10
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.49333409353885727
- self.std = 0.2844316122764386
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 18
Current best is [0.35016667]
[10: BO] (12 evaluations)
batch_size: 82
dropout_1: 0.14557280209760065
dropout_2: 0.5231236974598504
learning_rate: 0.0036698312743314717
n_units_1: 40
n_units_2: 101
scale_1: 5.981987507854231
scale_2: 10.0
Started BO from (top scorer):
batch_size: 82
dropout_1: 0.14557280856507585
dropout_2: 0.5231231614760518
learning_rate: 0.0033261781877540063
n_units_1: 40
n_units_2: 101
scale_1: 5.981988181743532
scale_2: 6.375505195149974
Top score values: [0.32350358 0.34767498 0.34845628 0.35179445 0.35273396]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9. Pending:
Targets: [-0.08643235 -0.66103736 -0.91191296 1.38846328 1.64660522 1.64223524
1.39482824 -0.3344269 -0.77666383 -1.07517376 -0.50334569 -0.72484006
-0.130195 -0.2678719 1.66622763 -0.3918188 -0.8454683 -1.02917272]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0002897017498538562, 'kernel_inv_bw1': 0.00010000000000000009, 'kernel_inv_bw2': 0.0008011959809245235, 'kernel_inv_bw3': 7.974358785073643, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.006837139024467231, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.36100017660804706, 'kernel_inv_bw8': 1.2163207127105384, 'kernel_covariance_scale': 0.6947885806350937, 'mean_mean_value': 0.7327734829861489}
Epoch 1 ; Time: 0.446839 ; Training: accuracy=0.257004 ; Validation: accuracy=0.498162
config_id 10: Terminating evaluation at 1
Update for config_id 10:1: reward = 0.49816237888406284, crit_val = 0.5018376211159372
Starting get_config[BO] for config_id 11
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4937816476218615
- self.std = 0.2768519137992561
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 19
Current best is [0.35016667]
[11: BO] (17 evaluations)
batch_size: 79
dropout_1: 0.75
dropout_2: 0.42369903081916027
learning_rate: 0.0031743331533078427
n_units_1: 128
n_units_2: 98
scale_1: 0.4942778506833816
scale_2: 5.998936967743114
Started BO from (top scorer):
batch_size: 79
dropout_1: 0.6243451100425594
dropout_2: 0.42338274050078417
learning_rate: 0.002139116508757636
n_units_1: 119
n_units_2: 57
scale_1: 0.49430045483873486
scale_2: 5.998675112100447
Top score values: [0.26995698 0.28539949 0.31800563 0.32117917 0.33470418]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1. Pending:
Targets: [-0.0904153 -0.68075194 -0.93849605 1.42486028 1.69006968 1.68558005
1.43139951 -0.34519947 -0.79954404 -1.10622663 -0.51874296 -0.74630144
-0.13537608 -0.27682233 1.71022931 -0.40416266 -0.87023225 -1.05896616
0.02909849]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00013384115748560582, 'kernel_inv_bw1': 0.7164481059620007, 'kernel_inv_bw2': 0.007434905102630733, 'kernel_inv_bw3': 5.873920713629177, 'kernel_inv_bw4': 0.29616486376970397, 'kernel_inv_bw5': 0.42909627025008135, 'kernel_inv_bw6': 0.00093129006269352, 'kernel_inv_bw7': 0.0016781228072952883, 'kernel_inv_bw8': 1.0966551096400794, 'kernel_covariance_scale': 0.7390471995681659, 'mean_mean_value': 0.8736689829419473}
Epoch 1 ; Time: 0.414171 ; Training: accuracy=0.133284 ; Validation: accuracy=0.438208
config_id 11: Terminating evaluation at 1
Update for config_id 11:1: reward = 0.4382078614257162, crit_val = 0.5617921385742838
Starting get_config[BO] for config_id 12
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4971821721694826
- self.std = 0.2702486648299593
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 20
Current best is [0.35016667]
[12: BO] (18 evaluations)
batch_size: 52
dropout_1: 0.4187634264961476
dropout_2: 0.7247258372119212
learning_rate: 0.003204081133297433
n_units_1: 50
n_units_2: 128
scale_1: 0.24304339720298734
scale_2: 1.2689115862202556
Started BO from (top scorer):
batch_size: 52
dropout_1: 0.5640354983588307
dropout_2: 0.7247257870729263
learning_rate: 0.0037061027371093794
n_units_1: 50
n_units_2: 114
scale_1: 0.24305486090288597
scale_2: 1.2689345703007509
Top score values: [0.28768416 0.29973203 0.31945222 0.34145763 0.37267305]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1. Pending:
Targets: [-0.10520745 -0.70996836 -0.97401019 1.44709233 1.71878185 1.71418252
1.45379134 -0.36621702 -0.83166303 -1.14583909 -0.54400086 -0.77711949
-0.1512668 -0.29616914 1.73943406 -0.42662091 -0.90407843 -1.09742386
0.01722654 0.23907599]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.4571498209951534, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.032423290570723, 'kernel_inv_bw4': 0.001970176820657327, 'kernel_inv_bw5': 0.42480096030193, 'kernel_inv_bw6': 0.000384714205448538, 'kernel_inv_bw7': 0.00028464009509878525, 'kernel_inv_bw8': 1.0773759457871805, 'kernel_covariance_scale': 0.763639790879144, 'mean_mean_value': 1.0197102312028976}
Epoch 1 ; Time: 0.584321 ; Training: accuracy=0.251243 ; Validation: accuracy=0.619398
Update for config_id 12:1: reward = 0.6193979933110368, crit_val = 0.38060200668896316
config_id 12: Reaches 1, continues to 3
Epoch 2 ; Time: 1.120297 ; Training: accuracy=0.403515 ; Validation: accuracy=0.691639
Epoch 3 ; Time: 1.658318 ; Training: accuracy=0.469164 ; Validation: accuracy=0.708027
config_id 12: Terminating evaluation at 3
Update for config_id 12:3: reward = 0.7080267558528428, crit_val = 0.2919732441471572
Starting get_config[BO] for config_id 13
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4825553951920805
- self.std = 0.26213127511599504
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 22
Current best is [0.35016667]
[13: BO] (13 evaluations)
batch_size: 65
dropout_1: 0.4380660616295446
dropout_2: 0.1932438504773113
learning_rate: 0.00442799054709087
n_units_1: 57
n_units_2: 78
scale_1: 2.497201620763285
scale_2: 0.0075591995777766095
Started BO from (top scorer):
batch_size: 65
dropout_1: 0.4169220946335453
dropout_2: 0.1932436849600857
learning_rate: 0.0035468887104147884
n_units_1: 57
n_units_2: 60
scale_1: 2.4972005561729547
scale_2: 0.007559210365656552
Top score values: [0.31963507 0.32597431 0.35312265 0.36624176 0.37228937]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3. Pending:
Targets: [-0.05266596 -0.67615443 -0.94837282 1.54770371 1.82780661 1.82306486
1.55461017 -0.32175818 -0.80161761 -1.12552272 -0.50504744 -0.74538503
-0.10015163 -0.24954114 1.84909835 -0.3840326 -0.87627549 -1.07560823
0.07355943 0.30227886 -0.38894019 -0.7270485 ]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0010532467968579068, 'kernel_inv_bw1': 1.404342815957603, 'kernel_inv_bw2': 0.0005398724164910998, 'kernel_inv_bw3': 4.82418165734747, 'kernel_inv_bw4': 0.0006939553261438888, 'kernel_inv_bw5': 0.5188360353524706, 'kernel_inv_bw6': 0.00012187757927405632, 'kernel_inv_bw7': 0.00026183289101106727, 'kernel_inv_bw8': 1.014056003186076, 'kernel_covariance_scale': 0.796498374050167, 'mean_mean_value': 1.1471307714613548}
Epoch 1 ; Time: 0.493533 ; Training: accuracy=0.433912 ; Validation: accuracy=0.687124
Update for config_id 13:1: reward = 0.687123745819398, crit_val = 0.31287625418060205
config_id 13: Reaches 1, continues to 3
Epoch 2 ; Time: 0.925874 ; Training: accuracy=0.555335 ; Validation: accuracy=0.736957
Epoch 3 ; Time: 1.352277 ; Training: accuracy=0.593218 ; Validation: accuracy=0.749331
Update for config_id 13:3: reward = 0.7493311036789297, crit_val = 0.25066889632107026
config_id 13: Reaches 3, continues to 9
Epoch 4 ; Time: 1.779695 ; Training: accuracy=0.614888 ; Validation: accuracy=0.776254
Epoch 5 ; Time: 2.335559 ; Training: accuracy=0.628867 ; Validation: accuracy=0.776421
Epoch 6 ; Time: 2.762544 ; Training: accuracy=0.646071 ; Validation: accuracy=0.801505
Epoch 7 ; Time: 3.208177 ; Training: accuracy=0.657155 ; Validation: accuracy=0.821572
Epoch 8 ; Time: 3.628639 ; Training: accuracy=0.657734 ; Validation: accuracy=0.809197
Epoch 9 ; Time: 4.055151 ; Training: accuracy=0.669231 ; Validation: accuracy=0.820903
config_id 13: Terminating evaluation at 9
Update for config_id 13:9: reward = 0.8209030100334448, crit_val = 0.1790969899665552
Starting get_config[BO] for config_id 14
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.45435443338776
- self.std = 0.2581820075415383
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 25
Current best is [0.31287626]
[14: BO] (16 evaluations)
batch_size: 30
dropout_1: 0.3893679882377774
dropout_2: 0.34480399499247283
learning_rate: 0.006353695537618099
n_units_1: 122
n_units_2: 68
scale_1: 5.840391604925039
scale_2: 5.876385115956924
Started BO from (top scorer):
batch_size: 30
dropout_1: 0.4222267082483692
dropout_2: 0.34480373634999584
learning_rate: 0.005673378598456798
n_units_1: 122
n_units_2: 93
scale_1: 5.7852734476136485
scale_2: 5.876374329027126
Top score values: [0.32319459 0.32494214 0.3334888 0.33862142 0.35233721]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9. Pending:
Targets: [ 0.05575744 -0.5772682 -0.85365056 1.68060708 1.96499455 1.96018027
1.68761918 -0.21745094 -0.70465051 -1.03351023 -0.40354387 -0.64755778
0.0075454 -0.14412924 1.98661199 -0.28067793 -0.7804504 -0.98283222
0.18391362 0.41613165 -0.2856606 -0.62894076 -0.54797846 -0.78892228
-1.0661372 ]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 1.425836992553471, 'kernel_inv_bw2': 0.00023623419268202186, 'kernel_inv_bw3': 4.980182657080316, 'kernel_inv_bw4': 0.0011116128852937048, 'kernel_inv_bw5': 0.5922730036428987, 'kernel_inv_bw6': 0.01835734551762398, 'kernel_inv_bw7': 0.00010992267079371953, 'kernel_inv_bw8': 1.2350787994251735, 'kernel_covariance_scale': 0.7894787373295402, 'mean_mean_value': 1.19408549856321}
Epoch 1 ; Time: 0.969953 ; Training: accuracy=0.208789 ; Validation: accuracy=0.410943
config_id 14: Terminating evaluation at 1
Update for config_id 14:1: reward = 0.41094276094276094, crit_val = 0.5890572390572391
Starting get_config[BO] for config_id 15
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.45953531052889374
- self.std = 0.254490115617556
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 26
Current best is [0.31287626]
[15: BO] (30 evaluations)
batch_size: 128
dropout_1: 0.0
dropout_2: 0.75
learning_rate: 0.004462430596721465
n_units_1: 115
n_units_2: 54
scale_1: 0.003013621357331008
scale_2: 0.01592608866916495
Started BO from (top scorer):
batch_size: 41
dropout_1: 0.6127517152216655
dropout_2: 0.679970947116099
learning_rate: 0.0026772865275651683
n_units_1: 115
n_units_2: 54
scale_1: 0.0030135746287843013
scale_2: 0.010357876640772819
Top score values: [0.10144235 0.19863885 0.21942546 0.22952127 0.24792889]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1. Pending:
Targets: [ 0.03620844 -0.60600051 -0.88639235 1.6846298 1.97314289 1.96825876
1.69174362 -0.24096338 -0.73523076 -1.06886125 -0.42975596 -0.67730978
-0.01270301 -0.166578 1.99507393 -0.3051076 -0.81213028 -1.01744805
0.16622379 0.40181061 -0.31016255 -0.65842269 -0.57628586 -0.82072506
-1.10196154 0.50894679]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.46313143159054526, 'kernel_inv_bw1': 0.03799649053583399, 'kernel_inv_bw2': 0.6789335288357643, 'kernel_inv_bw3': 5.746987933410928, 'kernel_inv_bw4': 0.00010000000000000009, 'kernel_inv_bw5': 0.00010179136732314697, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 1.0217327913896106, 'kernel_inv_bw8': 1.2424868531875415, 'kernel_covariance_scale': 0.9901900556575884, 'mean_mean_value': 0.8505919846375775}
Epoch 1 ; Time: 0.274473 ; Training: accuracy=0.178783 ; Validation: accuracy=0.513797
config_id 15: Terminating evaluation at 1
Update for config_id 15:1: reward = 0.5137965425531915, crit_val = 0.4862034574468085
Starting get_config[BO] for config_id 16
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4605230196740017
- self.std = 0.24978365026042518
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 27
Current best is [0.31287626]
[16: BO] (20 evaluations)
batch_size: 111
dropout_1: 0.7489872782361482
dropout_2: 0.4112951825285804
learning_rate: 0.006030391154332713
n_units_1: 71
n_units_2: 64
scale_1: 0.4702460058234478
scale_2: 0.0022580834536106847
Started BO from (top scorer):
batch_size: 94
dropout_1: 0.6232277607609598
dropout_2: 0.41130154566559785
learning_rate: 0.009744280412612874
n_units_1: 67
n_units_2: 64
scale_1: 0.47024490449176665
scale_2: 0.002261738493914154
Top score values: [0.14477379 0.29458184 0.30317129 0.30841421 0.33825946]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1. Pending:
Targets: [ 0.03293642 -0.62137313 -0.90704816 1.71241761 2.00636692 2.00139076
1.71966548 -0.24945791 -0.75303836 -1.09295517 -0.44180775 -0.69402602
-0.01689662 -0.17367094 2.02871118 -0.31481075 -0.83138683 -1.04057323
0.16540154 0.40542733 -0.31996095 -0.67478306 -0.5910986 -0.84014355
-1.12667915 0.5145822 0.10281072]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.5873871913789199, 'kernel_inv_bw1': 0.6133906375366832, 'kernel_inv_bw2': 0.002425978535538196, 'kernel_inv_bw3': 5.967537223380306, 'kernel_inv_bw4': 1.3571819493400543, 'kernel_inv_bw5': 0.004172932574532522, 'kernel_inv_bw6': 0.0008343982783770331, 'kernel_inv_bw7': 0.005430228338931737, 'kernel_inv_bw8': 1.088895033568476, 'kernel_covariance_scale': 0.8472538274510384, 'mean_mean_value': 1.2046771368037217}
Epoch 1 ; Time: 0.330650 ; Training: accuracy=0.277296 ; Validation: accuracy=0.595929
Update for config_id 16:1: reward = 0.5959292625959293, crit_val = 0.40407073740407073
config_id 16: Reaches 1, continues to 3
Epoch 2 ; Time: 0.594731 ; Training: accuracy=0.391685 ; Validation: accuracy=0.646980
Epoch 3 ; Time: 0.870386 ; Training: accuracy=0.419456 ; Validation: accuracy=0.667668
config_id 16: Terminating evaluation at 3
Update for config_id 16:3: reward = 0.6676676676676677, crit_val = 0.3323323323323323
Starting get_config[BO] for config_id 17
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.45415602072187755
- self.std = 0.24233239022487874
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 29
Current best is [0.25066889]
[17: BO] (22 evaluations)
batch_size: 78
dropout_1: 0.3045576813904649
dropout_2: 0.06715019571519018
learning_rate: 0.003262408223475904
n_units_1: 75
n_units_2: 128
scale_1: 4.8273549499119905
scale_2: 0.18003298996589037
Started BO from (top scorer):
batch_size: 19
dropout_1: 0.3045423041078955
dropout_2: 0.06715009483758164
learning_rate: 0.0011765168495392192
n_units_1: 53
n_units_2: 101
scale_1: 4.8273605985554235
scale_2: 0.17881078258405803
Top score values: [0.17956376 0.19205754 0.19215283 0.20628314 0.20852107]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3. Pending:
Targets: [ 0.06022298 -0.61420535 -0.90866434 1.79134502 2.09433271 2.08920355
1.79881575 -0.23085444 -0.74991903 -1.10028763 -0.42911867 -0.68909218
0.00885767 -0.15273717 2.11736402 -0.29821676 -0.83067657 -1.04629506
0.19676115 0.44416728 -0.30352531 -0.66925753 -0.58299993 -0.83970254
-1.13504856 0.55667845 0.13224578 -0.2066801 -0.50271319]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.49302343927026276, 'kernel_inv_bw1': 0.003938199546954624, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.202454482506499, 'kernel_inv_bw4': 1.5032552926599776, 'kernel_inv_bw5': 0.15190273859891862, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.8110564636235771, 'kernel_inv_bw8': 1.0382408371530802, 'kernel_covariance_scale': 0.8305671541530133, 'mean_mean_value': 1.276143848952813}
Epoch 1 ; Time: 0.413663 ; Training: accuracy=0.490736 ; Validation: accuracy=0.732601
Update for config_id 17:1: reward = 0.7326007326007326, crit_val = 0.26739926739926745
config_id 17: Reaches 1, continues to 3
Epoch 2 ; Time: 0.776846 ; Training: accuracy=0.637469 ; Validation: accuracy=0.787046
Epoch 3 ; Time: 1.132158 ; Training: accuracy=0.684285 ; Validation: accuracy=0.805195
Update for config_id 17:3: reward = 0.8051948051948052, crit_val = 0.19480519480519476
config_id 17: Reaches 3, continues to 9
Epoch 4 ; Time: 1.494883 ; Training: accuracy=0.713978 ; Validation: accuracy=0.823177
Epoch 5 ; Time: 1.849410 ; Training: accuracy=0.730025 ; Validation: accuracy=0.828338
Epoch 6 ; Time: 2.228908 ; Training: accuracy=0.746898 ; Validation: accuracy=0.867133
Epoch 7 ; Time: 2.581089 ; Training: accuracy=0.764516 ; Validation: accuracy=0.870463
Epoch 8 ; Time: 2.953403 ; Training: accuracy=0.771547 ; Validation: accuracy=0.869131
Epoch 9 ; Time: 3.320290 ; Training: accuracy=0.780149 ; Validation: accuracy=0.885281
config_id 17: Terminating evaluation at 9
Update for config_id 17:9: reward = 0.8852813852813853, crit_val = 0.11471861471861466
Starting get_config[BO] for config_id 18
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4296077399330478
- self.std = 0.24374035511370012
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 32
Current best is [0.19480519]
[18: BO] (29 evaluations)
batch_size: 29
dropout_1: 0.6421558254588213
dropout_2: 0.1423270311953913
learning_rate: 0.0030835974069092385
n_units_1: 76
n_units_2: 77
scale_1: 0.012848600967693495
scale_2: 0.1695480375063522
Started BO from (top scorer):
batch_size: 29
dropout_1: 0.6420664042353282
dropout_2: 0.14232698216264267
learning_rate: 0.0015986669988118994
n_units_1: 49
n_units_2: 98
scale_1: 0.012848630503362952
scale_2: 0.037627741428044
Top score values: [0.15164046 0.17197031 0.22818429 0.248012 0.25221713]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9. Pending:
Targets: [ 0.16058999 -0.50994251 -0.80270056 1.88171221 2.18294969 2.17785016
1.88913978 -0.12880603 -0.64487225 -0.99321695 -0.32592499 -0.58439676
0.10952138 -0.05114 2.20584796 -0.19577923 -0.72516329 -0.93953626
0.29633944 0.54231643 -0.20105712 -0.56467669 -0.47891735 -0.73413713
-1.02777708 0.65417768 0.23219675 -0.10477134 -0.39909439 -0.66549699
-0.96333061 -1.29190394]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.004452078240875218, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 6.117482454382751, 'kernel_inv_bw4': 1.4186884023219986, 'kernel_inv_bw5': 0.5540479762921837, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.8030999527559353, 'kernel_inv_bw8': 1.0030579054261999, 'kernel_covariance_scale': 0.8512684794456711, 'mean_mean_value': 1.3586677548274406}
Epoch 1 ; Time: 1.045400 ; Training: accuracy=0.391827 ; Validation: accuracy=0.658032
Update for config_id 18:1: reward = 0.6580319596299411, crit_val = 0.3419680403700589
config_id 18: Reaches 1, continues to 3
Epoch 2 ; Time: 2.006912 ; Training: accuracy=0.547662 ; Validation: accuracy=0.730530
Epoch 3 ; Time: 2.963032 ; Training: accuracy=0.593253 ; Validation: accuracy=0.737763
Update for config_id 18:3: reward = 0.7377628259041211, crit_val = 0.2622371740958789
config_id 18: Reaches 3, continues to 9
Epoch 4 ; Time: 3.932769 ; Training: accuracy=0.616711 ; Validation: accuracy=0.768545
Epoch 5 ; Time: 4.875020 ; Training: accuracy=0.633040 ; Validation: accuracy=0.769386
Epoch 6 ; Time: 5.846173 ; Training: accuracy=0.644728 ; Validation: accuracy=0.775105
Epoch 7 ; Time: 6.815998 ; Training: accuracy=0.652851 ; Validation: accuracy=0.797309
Epoch 8 ; Time: 7.842346 ; Training: accuracy=0.659980 ; Validation: accuracy=0.797813
Epoch 9 ; Time: 8.833176 ; Training: accuracy=0.661223 ; Validation: accuracy=0.803532
config_id 18: Terminating evaluation at 9
Update for config_id 18:9: reward = 0.8035323801513877, crit_val = 0.19646761984861227
Starting get_config[BO] for config_id 19
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4156605860620594
- self.std = 0.23810796909043946
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 35
Current best is [0.19480519]
[19: BO] (25 evaluations)
batch_size: 30
dropout_1: 0.30051346754016905
dropout_2: 0.5149662501287657
learning_rate: 0.0033584644609969
n_units_1: 87
n_units_2: 128
scale_1: 0.007139985851794008
scale_2: 0.0010000000000000002
Started BO from (top scorer):
batch_size: 30
dropout_1: 0.19178842843083357
dropout_2: 0.514965733153878
learning_rate: 0.004608147162332131
n_units_1: 89
n_units_2: 92
scale_1: 0.0071400818684522515
scale_2: 0.04676604880546017
Top score values: [0.16827371 0.2183128 0.23697301 0.24806571 0.25096753]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9. Pending:
Targets: [ 0.22296362 -0.46343016 -0.76311334 1.98479857 2.29316175 2.28794159
1.99240183 -0.07327799 -0.60155163 -0.95813634 -0.27505975 -0.53964561
0.170687 0.00622521 2.31660167 -0.14183543 -0.68374193 -0.90318585
0.3619242 0.6137197 -0.14723816 -0.51945906 -0.43167111 -0.69292805
-0.99351398 0.728227 0.29626422 -0.04867476 -0.34995995 -0.62266424
-0.92754305 -1.2638887 -0.30949214 -0.64434388 -0.92056124]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.0013634816584363687, 'kernel_inv_bw1': 0.6190951630939251, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 5.676432070744685, 'kernel_inv_bw4': 1.3333351557353115, 'kernel_inv_bw5': 0.5290566741809684, 'kernel_inv_bw6': 0.00013338602715591287, 'kernel_inv_bw7': 0.4125592465974438, 'kernel_inv_bw8': 1.0277581812671168, 'kernel_covariance_scale': 0.9258641208621095, 'mean_mean_value': 1.4275200415657963}
Epoch 1 ; Time: 0.986719 ; Training: accuracy=0.424461 ; Validation: accuracy=0.669024
Update for config_id 19:1: reward = 0.6690235690235691, crit_val = 0.3309764309764309
config_id 19: Reaches 1, continues to 3
Epoch 2 ; Time: 2.002579 ; Training: accuracy=0.618657 ; Validation: accuracy=0.740404
Epoch 3 ; Time: 3.064167 ; Training: accuracy=0.660862 ; Validation: accuracy=0.761448
Update for config_id 19:3: reward = 0.7614478114478115, crit_val = 0.23855218855218852
config_id 19: Reaches 3, continues to 9
Epoch 4 ; Time: 4.122421 ; Training: accuracy=0.699420 ; Validation: accuracy=0.788721
Epoch 5 ; Time: 5.381907 ; Training: accuracy=0.715506 ; Validation: accuracy=0.819697
Epoch 6 ; Time: 6.426985 ; Training: accuracy=0.732836 ; Validation: accuracy=0.844949
Epoch 7 ; Time: 7.527427 ; Training: accuracy=0.737811 ; Validation: accuracy=0.848485
Epoch 8 ; Time: 8.606051 ; Training: accuracy=0.746683 ; Validation: accuracy=0.853367
Epoch 9 ; Time: 9.661116 ; Training: accuracy=0.764179 ; Validation: accuracy=0.864141
config_id 19: Terminating evaluation at 9
Update for config_id 19:9: reward = 0.8641414141414141, crit_val = 0.1358585858585859
Starting get_config[BO] for config_id 20
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.4014080978305075
- self.std = 0.2347141628366402
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 38
Current best is [0.19480519]
[20: BO] (16 evaluations)
batch_size: 128
dropout_1: 0.3795842555697978
dropout_2: 0.72838833428696
learning_rate: 0.0030198784213265003
n_units_1: 83
n_units_2: 128
scale_1: 0.04408145378314061
scale_2: 0.09765656504766378
Started BO from (top scorer):
batch_size: 128
dropout_1: 0.4048887065451759
dropout_2: 0.728421596605789
learning_rate: 0.006348230627072913
n_units_1: 78
n_units_2: 89
scale_1: 0.04413240076143723
scale_2: 0.09765665888004807
Top score values: [0.19344329 0.2160605 0.22133216 0.22332952 0.23784654]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9. Pending:
Targets: [ 0.28691026 -0.4094083 -0.71342469 2.07422014 2.38704204 2.3817464
2.08193335 -0.0136148 -0.54952691 -0.91126759 -0.21831418 -0.48672577
0.23387776 0.06703796 2.41082089 -0.08316352 -0.63290563 -0.85552255
0.42788012 0.68331642 -0.08864438 -0.46624734 -0.37719004 -0.64222457
-0.94715677 0.79947941 0.36127074 0.01134418 -0.29429739 -0.5709448
-0.88023194 -1.22144092 -0.25324444 -0.5929379 -0.87314918 -0.30007421
-0.69384782 -1.13137405]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.3537938522065249, 'kernel_inv_bw1': 0.8378356556101776, 'kernel_inv_bw2': 0.0016915851893806095, 'kernel_inv_bw3': 5.980375819112083, 'kernel_inv_bw4': 1.399340131565785, 'kernel_inv_bw5': 0.4440378496097676, 'kernel_inv_bw6': 0.0038976552747363043, 'kernel_inv_bw7': 0.0017376830268557, 'kernel_inv_bw8': 1.0821242533040438, 'kernel_covariance_scale': 0.8307653793078447, 'mean_mean_value': 1.5267504731783406}
Epoch 1 ; Time: 0.271799 ; Training: accuracy=0.279359 ; Validation: accuracy=0.618850
Update for config_id 20:1: reward = 0.6188497340425532, crit_val = 0.38115026595744683
config_id 20: Reaches 1, continues to 3
Epoch 2 ; Time: 0.606417 ; Training: accuracy=0.500987 ; Validation: accuracy=0.690824
Epoch 3 ; Time: 0.825680 ; Training: accuracy=0.564638 ; Validation: accuracy=0.732214
config_id 20: Terminating evaluation at 3
Update for config_id 20:3: reward = 0.7322140957446809, crit_val = 0.2677859042553191
Starting get_config[BO] for config_id 21
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3975610971943012
- self.std = 0.22973470279782654
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 40
Current best is [0.19480519]
[21: BO] (29 evaluations)
batch_size: 67
dropout_1: 0.34686117566790153
dropout_2: 0.20201418691166012
learning_rate: 0.004738650794832996
n_units_1: 80
n_units_2: 128
scale_1: 0.028468999595052603
scale_2: 0.770211333225558
Started BO from (top scorer):
batch_size: 38
dropout_1: 0.5118747656872447
dropout_2: 0.20201432747850842
learning_rate: 0.005354025339239537
n_units_1: 73
n_units_2: 54
scale_1: 0.028469106152755365
scale_2: 0.7702109535801622
Top score values: [0.19717544 0.2035116 0.2039406 0.21059317 0.23890782]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3. Pending:
Targets: [ 0.3098744 -0.40153675 -0.71214264 2.13592391 2.45552617 2.45011575
2.1438043 0.00283551 -0.54469241 -0.91427375 -0.2063007 -0.48053006
0.25569243 0.0852364 2.47982042 -0.06822067 -0.62987835 -0.85732045
0.45389975 0.71487259 -0.07382033 -0.45960776 -0.36862016 -0.63939927
-0.95094082 0.8335534 0.38584663 0.02833547 -0.28393083 -0.56657452
-0.88256541 -1.23117004 -0.24198807 -0.58904433 -0.87532913 -0.28983286
-0.69214144 -1.13915098 -0.07143384 -0.56489155]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.854629066503119, 'kernel_inv_bw1': 0.5870978793204662, 'kernel_inv_bw2': 0.00010050459973374464, 'kernel_inv_bw3': 6.240918512920625, 'kernel_inv_bw4': 1.4475915916512152, 'kernel_inv_bw5': 0.203472904939617, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010114531074059352, 'kernel_inv_bw8': 1.184103642865395, 'kernel_covariance_scale': 0.8406326793341455, 'mean_mean_value': 1.6068406626882217}
Epoch 1 ; Time: 0.469977 ; Training: accuracy=0.529104 ; Validation: accuracy=0.757672
Update for config_id 21:1: reward = 0.7576723125943318, crit_val = 0.24232768740566824
config_id 21: Reaches 1, continues to 3
Epoch 2 ; Time: 0.893080 ; Training: accuracy=0.680100 ; Validation: accuracy=0.806809
Epoch 3 ; Time: 1.315032 ; Training: accuracy=0.726202 ; Validation: accuracy=0.839510
Update for config_id 21:3: reward = 0.8395103136005366, crit_val = 0.16048968639946337
config_id 21: Reaches 3, continues to 9
Epoch 4 ; Time: 1.743192 ; Training: accuracy=0.748259 ; Validation: accuracy=0.846721
Epoch 5 ; Time: 2.153153 ; Training: accuracy=0.763765 ; Validation: accuracy=0.842529
Epoch 6 ; Time: 2.564735 ; Training: accuracy=0.780265 ; Validation: accuracy=0.872044
Epoch 7 ; Time: 2.981216 ; Training: accuracy=0.788806 ; Validation: accuracy=0.871373
Epoch 8 ; Time: 3.393506 ; Training: accuracy=0.798425 ; Validation: accuracy=0.880429
Epoch 9 ; Time: 3.803730 ; Training: accuracy=0.801078 ; Validation: accuracy=0.888311
config_id 21: Terminating evaluation at 9
Update for config_id 21:9: reward = 0.8883112527251383, crit_val = 0.11168874727486167
Starting get_config[BO] for config_id 22
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.38178953508958235
- self.std = 0.2293797855695426
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 43
Current best is [0.11168875]
[22: BO] (28 evaluations)
batch_size: 103
dropout_1: 0.1833774875607099
dropout_2: 0.17319162379965122
learning_rate: 0.0038188313847388818
n_units_1: 79
n_units_2: 81
scale_1: 5.538443323522205
scale_2: 0.007745579111514982
Started BO from (top scorer):
batch_size: 103
dropout_1: 0.4270479207850957
dropout_2: 0.19779706556964466
learning_rate: 0.005409909016125813
n_units_1: 82
n_units_2: 83
scale_1: 5.538450711761021
scale_2: 0.007745531590890991
Top score values: [0.12476872 0.1279559 0.13359992 0.14188824 0.14854048]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9. Pending:
Targets: [ 3.79111283e-01 -3.33400625e-01 -6.44487113e-01 2.20798623e+00
2.52808300e+00 2.52266421e+00 2.21587880e+00 7.15973174e-02
-4.76777785e-01 -8.46930980e-01 -1.37862490e-01 -4.12516160e-01
3.24845477e-01 1.54125709e-01 2.55241484e+00 4.31187824e-04
-5.62095531e-01 -7.89889553e-01 5.23359483e-01 7.84736122e-01
-5.17712752e-03 -3.91561491e-01 -3.00433104e-01 -5.71631186e-01
-8.83654785e-01 9.03600566e-01 4.55201064e-01 9.71367301e-02
-2.15612734e-01 -4.98693760e-01 -8.15173577e-01 -1.16431760e+00
-1.73605074e-01 -5.21198329e-01 -8.07926099e-01 -2.21523898e-01
-6.24454967e-01 -1.07215616e+00 -2.78694625e-03 -4.97008185e-01
-6.07995370e-01 -9.64774852e-01 -1.17752655e+00]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.8264068686291545, 'kernel_inv_bw2': 0.3255986668562232, 'kernel_inv_bw3': 6.486586927673842, 'kernel_inv_bw4': 1.8061128745205743, 'kernel_inv_bw5': 0.5409780498842242, 'kernel_inv_bw6': 0.00010000000000000009, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9818858854378966, 'kernel_covariance_scale': 0.893924709445891, 'mean_mean_value': 1.5238415630704631}
Epoch 1 ; Time: 0.329885 ; Training: accuracy=0.567468 ; Validation: accuracy=0.753766
Update for config_id 22:1: reward = 0.7537663207231335, crit_val = 0.24623367927686646
config_id 22: Reaches 1, continues to 3
Epoch 2 ; Time: 0.616524 ; Training: accuracy=0.687839 ; Validation: accuracy=0.800301
Epoch 3 ; Time: 0.895412 ; Training: accuracy=0.724288 ; Validation: accuracy=0.813023
Update for config_id 22:3: reward = 0.8130231001004352, crit_val = 0.18697689989956479
config_id 22: Reaches 3, continues to 9
Epoch 4 ; Time: 1.174442 ; Training: accuracy=0.742718 ; Validation: accuracy=0.839471
Epoch 5 ; Time: 1.448048 ; Training: accuracy=0.763041 ; Validation: accuracy=0.860730
Epoch 6 ; Time: 1.719954 ; Training: accuracy=0.777851 ; Validation: accuracy=0.854871
Epoch 7 ; Time: 1.993462 ; Training: accuracy=0.778756 ; Validation: accuracy=0.867593
Epoch 8 ; Time: 2.272425 ; Training: accuracy=0.786655 ; Validation: accuracy=0.878641
Epoch 9 ; Time: 2.545978 ; Training: accuracy=0.799901 ; Validation: accuracy=0.891865
config_id 22: Terminating evaluation at 9
Update for config_id 22:9: reward = 0.8918647472380314, crit_val = 0.10813525276196856
Starting get_config[BO] for config_id 23
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.36865860523457483
- self.std = 0.2277360857031747
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 46
Current best is [0.10813525]
[23: BO] (32 evaluations)
batch_size: 70
dropout_1: 0.013863812496834233
dropout_2: 0.7352588908751994
learning_rate: 0.004293444810684013
n_units_1: 81
n_units_2: 51
scale_1: 3.139501529887402
scale_2: 0.212439505230771
Started BO from (top scorer):
batch_size: 85
dropout_1: 0.31349832499476277
dropout_2: 0.7352587949202476
learning_rate: 0.00033486684036548067
n_units_1: 97
n_units_2: 51
scale_1: 3.1394908510247905
scale_2: 0.2124395580502697
Top score values: [0.07207588 0.0823837 0.11300674 0.14383435 0.15748913]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9. Pending:
Targets: [ 0.43950608 -0.27814843 -0.5914802 2.28158105 2.60398814 2.59853024
2.28953059 0.12977261 -0.42256042 -0.79538522 -0.08119898 -0.35783498
0.38484861 0.21289666 2.6284956 0.05809284 -0.50849395 -0.73793209
0.5847954 0.84805854 0.05244404 -0.33672907 -0.24494296 -0.51809843
-0.83237408 0.9677809 0.51614505 0.15549636 -0.15951039 -0.44463458
-0.76339861 -1.11506259 -0.11719954 -0.46730157 -0.75609882 -0.16546422
-0.57130347 -1.02223597 0.05485148 -0.44293683 -0.55472508 -0.91407964
-1.12836689 -0.53757368 -0.79777302 -1.14397045]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.8027154333408109, 'kernel_inv_bw1': 0.6254275950337382, 'kernel_inv_bw2': 0.00010000000000000009, 'kernel_inv_bw3': 9.693844289179063, 'kernel_inv_bw4': 1.7398242155388026, 'kernel_inv_bw5': 0.0001150766226427339, 'kernel_inv_bw6': 0.0003244867382900531, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 1.1631557122246694, 'kernel_covariance_scale': 0.8250739669034771, 'mean_mean_value': 1.6079053152786353}
Epoch 1 ; Time: 0.459558 ; Training: accuracy=0.265896 ; Validation: accuracy=0.633109
Update for config_id 23:1: reward = 0.633109243697479, crit_val = 0.366890756302521
config_id 23: Reaches 1, continues to 3
Epoch 2 ; Time: 0.872883 ; Training: accuracy=0.392733 ; Validation: accuracy=0.693613
Epoch 3 ; Time: 1.270875 ; Training: accuracy=0.428984 ; Validation: accuracy=0.700000
config_id 23: Terminating evaluation at 3
Update for config_id 23:3: reward = 0.7, crit_val = 0.30000000000000004
Starting get_config[BO] for config_id 24
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.36719138743943674
- self.std = 0.2231565750796014
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 48
Current best is [0.10813525]
[24: BO] (38 evaluations)
batch_size: 85
dropout_1: 0.31051060860973106
dropout_2: 0.46498300377204105
learning_rate: 0.0037397376083877225
n_units_1: 75
n_units_2: 99
scale_1: 0.0010265288422456429
scale_2: 0.031194535210625002
Started BO from (top scorer):
batch_size: 85
dropout_1: 0.5566114422886048
dropout_2: 0.46081974429643935
learning_rate: 0.011724856654620737
n_units_1: 91
n_units_2: 118
scale_1: 0.001029001714917599
scale_2: 0.031194407051080224
Top score values: [0.11614941 0.12104836 0.13073905 0.16401772 0.18226919]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3. Pending:
Targets: [ 4.55100248e-01 -2.77281618e-01 -5.97043435e-01 2.33497738e+00
2.66400076e+00 2.65843085e+00 2.34309006e+00 1.39010580e-01
-4.24657165e-01 -8.05132893e-01 -7.62904735e-02 -3.58603464e-01
3.99321120e-01 2.23840457e-01 2.68901114e+00 6.58598270e-02
-5.12354183e-01 -7.46500741e-01 6.03371125e-01 8.72036825e-01
6.00951114e-02 -3.37064428e-01 -2.43394725e-01 -5.22155760e-01
-8.42880822e-01 9.94216063e-01 5.33311958e-01 1.65262215e-01
-1.56208954e-01 -4.47184314e-01 -7.72489865e-01 -1.13137053e+00
-1.13029818e-01 -4.70316473e-01 -7.65040275e-01 -1.62284963e-01
-5.76452649e-01 -1.03663897e+00 6.25519482e-02 -4.45451733e-01
-5.59534040e-01 -9.26263100e-01 -1.14494785e+00 -5.42030671e-01
-8.07569696e-01 -1.16087162e+00 -1.34717580e-03 -3.01095262e-01]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.00010000000000000009, 'kernel_inv_bw1': 0.7937617751816689, 'kernel_inv_bw2': 0.2653430479628214, 'kernel_inv_bw3': 6.492116862427172, 'kernel_inv_bw4': 1.680620562170976, 'kernel_inv_bw5': 0.7040478711700238, 'kernel_inv_bw6': 0.0017574484709384443, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9794468614287616, 'kernel_covariance_scale': 0.9366363563177954, 'mean_mean_value': 1.6260365596584596}
Epoch 1 ; Time: 0.389757 ; Training: accuracy=0.374814 ; Validation: accuracy=0.619328
config_id 24: Terminating evaluation at 1
Update for config_id 24:1: reward = 0.619327731092437, crit_val = 0.380672268907563
Starting get_config[BO] for config_id 25
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3674665074693985
- self.std = 0.22087595379903402
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 49
Current best is [0.10813525]
[25: BO] (41 evaluations)
batch_size: 94
dropout_1: 0.18069117583678673
dropout_2: 0.0
learning_rate: 0.003287309390802507
n_units_1: 76
n_units_2: 100
scale_1: 0.08806668911612683
scale_2: 0.22766609949966224
Started BO from (top scorer):
batch_size: 66
dropout_1: 0.12832840740272275
dropout_2: 0.6942082513852165
learning_rate: 0.014601379220547455
n_units_1: 90
n_units_2: 99
scale_1: 0.05865040201561297
scale_2: 0.22766679901993808
Top score values: [-0.01010981 0.07318622 0.11750267 0.11968641 0.127114 ]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 24:1. Pending:
Targets: [ 4.58553730e-01 -2.81390234e-01 -6.04453703e-01 2.35784125e+00
2.69026191e+00 2.68463449e+00 2.36603769e+00 1.39200327e-01
-4.30287485e-01 -8.14691757e-01 -7.83237854e-02 -3.63551756e-01
4.02198664e-01 2.24906101e-01 2.71553054e+00 6.52942665e-02
-5.18890005e-01 -7.55454208e-01 6.08355556e-01 8.79795323e-01
5.94700283e-02 -3.41790322e-01 -2.47153447e-01 -5.28792787e-01
-8.52829447e-01 1.00323611e+00 5.37573004e-01 1.65723019e-01
-1.59067452e-01 -4.53047235e-01 -7.81711679e-01 -1.14429791e+00
-1.15442476e-01 -4.76418241e-01 -7.74185169e-01 -1.65206198e-01
-5.83650310e-01 -1.04858821e+00 6.19522327e-02 -4.51296764e-01
-5.66557010e-01 -9.37072676e-01 -1.15801542e+00 -5.48872913e-01
-8.17153721e-01 -1.17410361e+00 -2.60667201e-03 -3.05449762e-01
5.97881354e-02]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.3678343883835809, 'kernel_inv_bw1': 0.7601706769574186, 'kernel_inv_bw2': 0.23850940582314611, 'kernel_inv_bw3': 7.049427409972382, 'kernel_inv_bw4': 1.6494545190974244, 'kernel_inv_bw5': 0.37602988534368015, 'kernel_inv_bw6': 0.5503915167679336, 'kernel_inv_bw7': 0.00010000000000000009, 'kernel_inv_bw8': 0.9766869575576077, 'kernel_covariance_scale': 0.8534905298413838, 'mean_mean_value': 1.7392742840439546}
Epoch 1 ; Time: 0.343432 ; Training: accuracy=0.516164 ; Validation: accuracy=0.721742
Update for config_id 25:1: reward = 0.7217420212765957, crit_val = 0.2782579787234043
config_id 25: Reaches 1, continues to 3
Epoch 2 ; Time: 0.648966 ; Training: accuracy=0.721508 ; Validation: accuracy=0.796044
Epoch 3 ; Time: 0.943018 ; Training: accuracy=0.768761 ; Validation: accuracy=0.825465
Update for config_id 25:3: reward = 0.8254654255319149, crit_val = 0.17453457446808507
config_id 25: Reaches 3, continues to 9
Epoch 4 ; Time: 1.242588 ; Training: accuracy=0.801006 ; Validation: accuracy=0.853059
Epoch 5 ; Time: 1.535863 ; Training: accuracy=0.826076 ; Validation: accuracy=0.873005
Epoch 6 ; Time: 1.828226 ; Training: accuracy=0.840508 ; Validation: accuracy=0.880485
Epoch 7 ; Time: 2.122348 ; Training: accuracy=0.849250 ; Validation: accuracy=0.892121
Epoch 8 ; Time: 2.417222 ; Training: accuracy=0.859063 ; Validation: accuracy=0.904754
Epoch 9 ; Time: 2.712334 ; Training: accuracy=0.873330 ; Validation: accuracy=0.903757
config_id 25: Terminating evaluation at 9
Update for config_id 25:9: reward = 0.9037566489361702, crit_val = 0.09624335106382975
Starting get_config[BO] for config_id 26
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.35682489942799706
- self.std = 0.21941256255578964
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 52
Current best is [0.09624335]
[26: BO] (69 evaluations)
batch_size: 42
dropout_1: 0.1744349916309062
dropout_2: 0.0
learning_rate: 0.0038372046729851224
n_units_1: 79
n_units_2: 96
scale_1: 0.2658709475772431
scale_2: 0.10064586413418124
Started BO from (top scorer):
batch_size: 73
dropout_1: 0.5357545241906819
dropout_2: 0.6253351021597766
learning_rate: 0.0005066579097652998
n_units_1: 35
n_units_2: 105
scale_1: 1.1089901159674307
scale_2: 0.1505131524731448
Top score values: [0.11596818 0.11649149 0.13559758 0.14573724 0.15898673]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 24:1, 25:1, 25:3, 25:9. Pending:
Targets: [ 0.51011254 -0.23476654 -0.55998471 2.42206753 2.75670529 2.75104034
2.43031864 0.18862919 -0.38465687 -0.77162497 -0.03034572 -0.31747605
0.45338161 0.27490658 2.78214245 0.1142302 -0.47385034 -0.71199233
0.66091349 0.93416364 0.10836712 -0.29556947 -0.20030141 -0.48381917
-0.81001702 1.05842773 0.58965884 0.21532877 -0.11162792 -0.40756842
-0.73842492 -1.10342946 -0.06771198 -0.4310953 -0.73084821 -0.1178076
-0.53904257 -1.00708141 0.11086588 -0.40580628 -0.52183526 -0.89482211
-1.11723845 -0.50403322 -0.77410335 -1.13343395 0.04587639 -0.25898654
0.10868735 -0.3580785 -0.83081079 -1.18763277]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.42324491240627987, 'kernel_inv_bw1': 0.7156945449360881, 'kernel_inv_bw2': 0.2141060817290444, 'kernel_inv_bw3': 6.454815377465997, 'kernel_inv_bw4': 1.5658965803517961, 'kernel_inv_bw5': 0.36673513779368455, 'kernel_inv_bw6': 0.5249981960362449, 'kernel_inv_bw7': 0.00019286209585298096, 'kernel_inv_bw8': 0.9692032659730366, 'kernel_covariance_scale': 0.8850408718410175, 'mean_mean_value': 1.8051325387420518}
Epoch 1 ; Time: 0.698000 ; Training: accuracy=0.608383 ; Validation: accuracy=0.787223
Update for config_id 26:1: reward = 0.7872233400402414, crit_val = 0.21277665995975859
config_id 26: Reaches 1, continues to 3
Epoch 2 ; Time: 1.352709 ; Training: accuracy=0.774058 ; Validation: accuracy=0.838364
Epoch 3 ; Time: 1.996558 ; Training: accuracy=0.821759 ; Validation: accuracy=0.868042
Update for config_id 26:3: reward = 0.8680415828303152, crit_val = 0.13195841716968482
config_id 26: Reaches 3, continues to 9
Epoch 4 ; Time: 2.638844 ; Training: accuracy=0.847388 ; Validation: accuracy=0.881288
Epoch 5 ; Time: 3.275313 ; Training: accuracy=0.865410 ; Validation: accuracy=0.896714
Epoch 6 ; Time: 3.913710 ; Training: accuracy=0.873760 ; Validation: accuracy=0.905097
Epoch 7 ; Time: 4.555463 ; Training: accuracy=0.882440 ; Validation: accuracy=0.921194
Epoch 8 ; Time: 5.292177 ; Training: accuracy=0.895172 ; Validation: accuracy=0.918343
Epoch 9 ; Time: 5.974089 ; Training: accuracy=0.899140 ; Validation: accuracy=0.922535
config_id 26: Terminating evaluation at 9
Update for config_id 26:9: reward = 0.9225352112676056, crit_val = 0.07746478873239437
Starting get_config[BO] for config_id 27
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.3450380842930488
- self.std = 0.2193002583241897
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 55
Current best is [0.07746479]
[27: BO] (46 evaluations)
batch_size: 8
dropout_1: 0.0
dropout_2: 0.75
learning_rate: 0.0018806118959927038
n_units_1: 72
n_units_2: 128
scale_1: 0.37204093099055713
scale_2: 0.6836981951655413
Started BO from (top scorer):
batch_size: 61
dropout_1: 0.020579246890917252
dropout_2: 0.16993870748769685
learning_rate: 0.002747913638632415
n_units_1: 67
n_units_2: 111
scale_1: 0.48539653881748507
scale_2: 0.7290186728766899
Top score values: [0.08114525 0.09766269 0.12373368 0.13443452 0.15749047]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 24:1, 25:1, 25:3, 25:9, 26:1, 26:3, 26:9. Pending:
Targets: [ 0.56412116 -0.18113938 -0.5065241 2.47705526 2.81186439 2.80619654
2.4853106 0.24247317 -0.33110647 -0.71827273 0.02338612 -0.26389124
0.50736117 0.32879475 2.83731458 0.16803608 -0.42034561 -0.65860956
0.71499933 0.98838942 0.16217 -0.24197345 -0.1466566 -0.43031955
-0.75668445 1.11271713 0.64370819 0.26918643 -0.0579377 -0.35402976
-0.68505569 -1.05024714 -0.01399927 -0.37756869 -0.6774751 -0.06412055
-0.48557123 -0.95384976 0.16467004 -0.35226671 -0.46835511 -0.84153297
-1.06406321 -0.45054395 -0.72075239 -1.080267 0.09964727 -0.20537178
0.16249039 -0.30451449 -0.77748887 -1.13449357 -0.60310656 -0.97163436
-1.22012303]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.05130578743636919, 'kernel_inv_bw1': 0.551820318189915, 'kernel_inv_bw2': 0.2340075383592714, 'kernel_inv_bw3': 12.644248319806238, 'kernel_inv_bw4': 1.6113078304573207, 'kernel_inv_bw5': 0.21352907282436578, 'kernel_inv_bw6': 0.5487184266138397, 'kernel_inv_bw7': 0.009127759054027853, 'kernel_inv_bw8': 0.9820413088936553, 'kernel_covariance_scale': 0.9280838171874843, 'mean_mean_value': 1.808546926134167}
Epoch 1 ; Time: 3.516380 ; Training: accuracy=0.470491 ; Validation: accuracy=0.745458
Update for config_id 27:1: reward = 0.745457604306864, crit_val = 0.25454239569313597
config_id 27: Reaches 1, continues to 3
Epoch 2 ; Time: 6.916026 ; Training: accuracy=0.629310 ; Validation: accuracy=0.810229
Epoch 3 ; Time: 10.248655 ; Training: accuracy=0.674237 ; Validation: accuracy=0.833782
Update for config_id 27:3: reward = 0.8337819650067295, crit_val = 0.16621803499327048
config_id 27: Reaches 3, continues to 9
Epoch 4 ; Time: 13.588085 ; Training: accuracy=0.698773 ; Validation: accuracy=0.826884
Epoch 5 ; Time: 16.957092 ; Training: accuracy=0.716512 ; Validation: accuracy=0.849092
Epoch 6 ; Time: 20.317139 ; Training: accuracy=0.730852 ; Validation: accuracy=0.859690
Epoch 7 ; Time: 23.678815 ; Training: accuracy=0.751243 ; Validation: accuracy=0.867429
Epoch 8 ; Time: 27.046151 ; Training: accuracy=0.751243 ; Validation: accuracy=0.873318
Epoch 9 ; Time: 30.456161 ; Training: accuracy=0.758704 ; Validation: accuracy=0.868102
config_id 27: Terminating evaluation at 9
Update for config_id 27:9: reward = 0.8681022880215343, crit_val = 0.13189771197846567
Starting get_config[BO] for config_id 28
Fitting GP model
[GPMXNetModel._posterior_for_state]
- self.mean = 0.33671987549625093
- self.std = 0.21682166948961054
BO Algorithm: Generating initial candidates.
BO Algorithm: Scoring (and reordering) candidates.
BO Algorithm: Selecting final set of candidates.
[GPMXNetModel.current_best -- RECOMPUTING]
- len(candidates) = 58
Current best is [0.07746479]
[28: BO] (32 evaluations)
batch_size: 34
dropout_1: 0.0
dropout_2: 0.0
learning_rate: 0.003154901164209692
n_units_1: 78
n_units_2: 128
scale_1: 0.5518292204244141
scale_2: 0.0025324347286775906
Started BO from (top scorer):
batch_size: 95
dropout_1: 0.02527270344625121
dropout_2: 0.10529219762842898
learning_rate: 0.008783476078550585
n_units_1: 119
n_units_2: 109
scale_1: 0.012432133016247692
scale_2: 0.0018737671697008117
Top score values: [-0.03482335 0.09673004 0.10107539 0.13619202 0.14314731]
Labeled: 0:1, 0:3, 0:9, 1:1, 2:1, 3:1, 4:1, 5:1, 5:3, 5:9, 6:1, 6:3, 7:1, 7:3, 8:1, 9:1, 9:3, 9:9, 10:1, 11:1, 12:1, 12:3, 13:1, 13:3, 13:9, 14:1, 15:1, 16:1, 16:3, 17:1, 17:3, 17:9, 18:1, 18:3, 18:9, 19:1, 19:3, 19:9, 20:1, 20:3, 21:1, 21:3, 21:9, 22:1, 22:3, 22:9, 23:1, 23:3, 24:1, 25:1, 25:3, 25:9, 26:1, 26:3, 26:9, 27:1, 27:3, 27:9. Pending:
Targets: [ 0.60893418 -0.14484578 -0.47395012 2.54373591 2.8823724 2.87663976
2.55208562 0.28360928 -0.29652722 -0.68811935 0.06201775 -0.22854362
0.55152534 0.37091764 2.90811352 0.20832127 -0.38678649 -0.62777414
0.7615371 1.03805244 0.20238813 -0.20637527 -0.10996881 -0.39687444
-0.72697017 1.1638014 0.68943101 0.31062791 -0.02023572 -0.31971255
-0.65452259 -1.02388872 0.02420498 -0.34352056 -0.64685534 -0.02648926
-0.45275773 -0.92638937 0.20491674 -0.31792934 -0.43534481 -0.81278864
-1.03786272 -0.41733004 -0.69062735 -1.05425174 0.13915067 -0.16935519
0.20271218 -0.26963125 -0.74801242 -1.1090982 -0.57163666 -0.94437728
-1.19570653 -0.37900953 -0.78636901 -0.94465726]
GP params:{'noise_variance': 1.0000000000000007e-09, 'kernel_inv_bw0': 0.28953845288727814, 'kernel_inv_bw1': 0.49950899888421957, 'kernel_inv_bw2': 0.12457109989271442, 'kernel_inv_bw3': 5.22186115179259, 'kernel_inv_bw4': 1.1916036383166984, 'kernel_inv_bw5': 0.3724057688625036, 'kernel_inv_bw6': 0.6362441175203422, 'kernel_inv_bw7': 0.5019076543486825, 'kernel_inv_bw8': 0.8463917120707751, 'kernel_covariance_scale': 1.1440352398539448, 'mean_mean_value': 1.8331809509409112}
Epoch 1 ; Time: 0.885678 ; Training: accuracy=0.662386 ; Validation: accuracy=0.796303
Update for config_id 28:1: reward = 0.7963025210084034, crit_val = 0.20369747899159663
config_id 28: Reaches 1, continues to 3
Epoch 2 ; Time: 1.666421 ; Training: accuracy=0.834134 ; Validation: accuracy=0.849916
Epoch 3 ; Time: 2.563686 ; Training: accuracy=0.885584 ; Validation: accuracy=0.887059
Update for config_id 28:3: reward = 0.8870588235294118, crit_val = 0.11294117647058821
config_id 28: Reaches 3, continues to 9
Epoch 4 ; Time: 3.444435 ; Training: accuracy=0.904474 ; Validation: accuracy=0.895966
Epoch 5 ; Time: 4.316014 ; Training: accuracy=0.923695 ; Validation: accuracy=0.910588
Epoch 6 ; Time: 5.225533 ; Training: accuracy=0.930406 ; Validation: accuracy=0.926555
Epoch 7 ; Time: 6.224707 ; Training: accuracy=0.942833 ; Validation: accuracy=0.924874
Epoch 8 ; Time: 7.102430 ; Training: accuracy=0.950621 ; Validation: accuracy=0.922689
Epoch 9 ; Time: 7.974301 ; Training: accuracy=0.955758 ; Validation: accuracy=0.922857
config_id 28: Terminating evaluation at 9
Update for config_id 28:9: reward = 0.9228571428571428, crit_val = 0.07714285714285718
Analysing the results¶
The training history is stored in the results_df, the main fields
are the runtime and 'best' (the objective).
Note: You will get slightly different curves for different pairs of
scheduler/searcher, the time_out here is a bit too short to really
see the difference in a significant way (it would be better to set it to
>1000s). Generally speaking though, hyperband stopping / promotion +
model will tend to significantly outperform other combinations given
enough time.
results_df.head()
/var/lib/jenkins/miniconda3/envs/autogluon_docs/lib/python3.7/site-packages/ipykernel/ipkernel.py:287: DeprecationWarning: should_run_async will not call transform_cell automatically in the future. Please pass the result to transformed_cell argument and any exception that happen during thetransform in preprocessing_exc_tuple in IPython 7.17 and above. and should_run_async(code)
| bracket | elapsed_time | epoch | error | eval_time | objective | runtime | searcher_data_size | searcher_params_kernel_covariance_scale | searcher_params_kernel_inv_bw0 | ... | searcher_params_kernel_inv_bw7 | searcher_params_kernel_inv_bw8 | searcher_params_mean_mean_value | searcher_params_noise_variance | target_epoch | task_id | time_since_start | time_step | time_this_iter | best | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0.471754 | 1 | 0.468750 | 0.465767 | 0.531250 | 0.653787 | NaN | 1.0 | 1.0 | ... | 1.0 | 1.0 | 0.0 | 0.001 | 9 | 0 | 0.655609 | 1.597435e+09 | 0.504280 | 0.468750 |
| 1 | 0 | 0.875604 | 2 | 0.344753 | 0.397097 | 0.655247 | 1.057637 | 1.0 | 1.0 | 1.0 | ... | 1.0 | 1.0 | 0.0 | 0.001 | 9 | 0 | 1.058451 | 1.597435e+09 | 0.403831 | 0.344753 |
| 2 | 0 | 1.275409 | 3 | 0.305314 | 0.397895 | 0.694686 | 1.457442 | 1.0 | 1.0 | 1.0 | ... | 1.0 | 1.0 | 0.0 | 0.001 | 9 | 0 | 1.458186 | 1.597435e+09 | 0.399805 | 0.305314 |
| 3 | 0 | 1.675770 | 4 | 0.288937 | 0.395817 | 0.711063 | 1.857803 | 2.0 | 1.0 | 1.0 | ... | 1.0 | 1.0 | 0.0 | 0.001 | 9 | 0 | 1.858569 | 1.597435e+09 | 0.400361 | 0.288937 |
| 4 | 0 | 2.077540 | 5 | 0.273061 | 0.399986 | 0.726939 | 2.259573 | 2.0 | 1.0 | 1.0 | ... | 1.0 | 1.0 | 0.0 | 0.001 | 9 | 0 | 2.260557 | 1.597435e+09 | 0.401771 | 0.273061 |
5 rows × 26 columns
import matplotlib.pyplot as plt
plt.figure(figsize=(12, 8))
runtime = results_df['runtime'].values
objective = results_df['best'].values
plt.plot(runtime, objective, lw=2)
plt.xticks(fontsize=12)
plt.xlim(0, 120)
plt.ylim(0, 0.5)
plt.yticks(fontsize=12)
plt.xlabel("Runtime [s]", fontsize=14)
plt.ylabel("Objective", fontsize=14)
Text(0, 0.5, 'Objective')
Diving Deeper¶
Now, you are ready to try HPO on your own machine learning models (if you use PyTorch, have a look at MNIST Training in PyTorch). While AutoGluon comes with well-chosen defaults, it can pay off to tune it to your specific needs. Here are some tips which may come useful.
Logging the Search Progress¶
First, it is a good idea in general to switch on debug_log, which
outputs useful information about the search progress. This is already
done in the example above.
The outputs show which configurations are chosen, stopped, or promoted.
For BO and BOHB, a range of information is displayed for every
get_config decision. This log output is very useful in order to
figure out what is going on during the search.
Configuring HyperbandScheduler¶
The most important knobs to turn with HyperbandScheduler are
max_t, grace_period, reduction_factor, brackets, and
type. The first three determine the rung levels at which stopping or
promotion decisions are being made.
The maximum resource level
max_t(usually, resource equates to epochs, somax_tis the maximum number of training epochs) is typically hardcoded intrain_fnpassed to the scheduler (this isrun_mlp_openmlin the example above). As already noted above, the value is best fixed in theag.argsdecorator asepochs=XYZ, it can then be accessed asargs.epochsin thetrain_fncode. If this is done, you do not have to passmax_twhen creating the scheduler.grace_periodandreduction_factordetermine the rung levels, which aregrace_period,grace_period * reduction_factor,grace_period * (reduction_factor ** 2), etc. All rung levels must be less or equal thanmax_t. It is recommended to makemax_tequal to the largest rung level. For example, ifgrace_period = 1,reduction_factor = 3, it is in general recommended to usemax_t = 9,max_t = 27, ormax_t = 81. Choosing amax_tvalue “off the grid” works against the successive halving principle that the total resources spent in a rung should be roughly equal between rungs. If in the example above, you setmax_t = 10, about a third of configurations reaching 9 epochs are allowed to proceed, but only for one more epoch.With
reduction_factor, you tune the extent to which successive halving filtering is applied. The larger this integer, the fewer configurations make it to higher number of epochs. Values 2, 3, 4 are commonly used.Finally,
grace_periodshould be set to the smallest resource (number of epochs) for which you expect any meaningful differentiation between configurations. Whilegrace_period = 1should always be explored, it may be too low for any meaningful stopping decisions to be made at the first rung.bracketssets the maximum number of brackets in Hyperband (make sure to study the Hyperband paper or follow-ups for details). Forbrackets = 1, you are running successive halving (single bracket). Higher brackets have larger effectivegrace_periodvalues (so runs are not stopped until later), yet are also chosen with less probability. We recommend to always consider successive halving (brackets = 1) in a comparison.Finally, with
type(valuesstopping,promotion) you are choosing different ways of extending successive halving scheduling to the asynchronous case. The method for the defaultstoppingis simpler and seems to perform well, butpromotionis more careful promoting configurations to higher resource levels, which can work better in some cases.
Asynchronous BOHB¶
Finally, here are some ideas for tuning asynchronous BOHB, apart from
tuning its HyperbandScheduling component. You need to pass these
options in search_options.
We support a range of different surrogate models over the criterion functions across resource levels. All of them are jointly dependent Gaussian process models, meaning that data collected at all resource levels are modelled together. The surrogate model is selected by
gp_resource_kernel, values arematern52,matern52-res-warp,exp-decay-sum,exp-decay-combined,exp-decay-delta1. These are variants of either a joint Matern 5/2 kernel over configuration and resource, or the exponential decay model. Details about the latter can be found here.Fitting a Gaussian process surrogate model to data encurs a cost which scales cubically with the number of datapoints. When applied to expensive deep learning workloads, even multi-fidelity asynchronous BOHB is rarely running up more than 100 observations or so (across all rung levels and brackets), and the GP computations are subdominant. However, if you apply it to cheaper
train_fnand find yourself beyond 2000 total evaluations, the cost of GP fitting can become painful. In such a situation, you can explore the optionsopt_skip_periodandopt_skip_num_max_resource. The basic idea is as follows. By far the most expensive part of aget_configcall (picking the next configuration) is the refitting of the GP model to past data (this entails re-optimizing hyperparameters of the surrogate model itself). The options allow you to skip this expensive step for mostget_configcalls, after some initial period. Check the docstrings for details about these options. If you find yourself in such a situation and gain experience with these skipping features, make sure to contact the AutoGluon developers – we would love to learn about your use case.