.. _sec_customobj: Searchable Objects ================== When defining custom Python objects such as network architectures, or specialized optimizers, it may be hard to decide what values to set for all of their attributes. AutoGluon provides an API that allows you to instead specify a search space of possible values to consider for such attributes, within which the optimal value will be automatically searched for at runtime. This tutorial demonstrates how easy this is to do, without having to modify your existing code at all! Example for Constructing a Network ---------------------------------- This tutorial covers an example of selecting a neural network's architecture as a hyperparameter optimization (HPO) task. If you are interested in efficient neural architecture search (NAS), please refer to this other tutorial instead: ``sec_proxyless``\ \_ . CIFAR ResNet in GluonCV ~~~~~~~~~~~~~~~~~~~~~~~ GluonCV provides `CIFARResNet `__, which allow user to specify how many layers at each stage. For example, we can construct a CIFAR ResNet with only 1 layer per stage: .. code:: python import pickle from gluoncv.model_zoo.cifarresnet import CIFARResNetV1, CIFARBasicBlockV1 layers = [1, 1, 1] channels = [16, 16, 32, 64] net = CIFARResNetV1(CIFARBasicBlockV1, layers, channels) .. parsed-literal:: :class: output /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/venv/lib/python3.9/site-packages/gluoncv/__init__.py:40: UserWarning: Both `mxnet==1.7.0` and `torch==1.7.1+cu101` are installed. You might encounter increased GPU memory footprint if both framework are used at the same time. warnings.warn(f'Both `mxnet=={mx.__version__}` and `torch=={torch.__version__}` are installed. ' We can visualize the network: .. code:: python import autogluon.core as ag from autogluon.vision.utils import plot_network plot_network(net, (1, 3, 32, 32)) .. figure:: output_object_d3e86d_3_0.svg Searchable Network Architecture Using AutoGluon Object ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :func:`autogluon.obj` enables customized search space to any user defined class. It can also be used within ``autogluon.Categorical()`` if you have multiple networks to choose from. .. code:: python @ag.obj( nstage1=ag.space.Int(2, 4), nstage2=ag.space.Int(2, 4), ) class MyCifarResNet(CIFARResNetV1): def __init__(self, nstage1, nstage2): nstage3 = 9 - nstage1 - nstage2 layers = [nstage1, nstage2, nstage3] channels = [16, 16, 32, 64] super().__init__(CIFARBasicBlockV1, layers=layers, channels=channels) Create one network instance and print the configuration space: .. code:: python mynet=MyCifarResNet() print(mynet.cs) .. parsed-literal:: :class: output Configuration space object: Hyperparameters: nstage1, Type: UniformInteger, Range: [2, 4], Default: 3 nstage2, Type: UniformInteger, Range: [2, 4], Default: 3 We can also overwrite existing search spaces: .. code:: python mynet1 = MyCifarResNet(nstage1=1, nstage2=ag.space.Int(5, 10)) print(mynet1.cs) .. parsed-literal:: :class: output Configuration space object: Hyperparameters: nstage2, Type: UniformInteger, Range: [5, 10], Default: 8 Decorate Existing Class ~~~~~~~~~~~~~~~~~~~~~~~ We can also use :func:`autogluon.obj` to easily decorate any existing classes. For example, if we want to search learning rate and weight decay for Adam optimizer, we only need to add a decorator: .. code:: python from mxnet import optimizer as optim @ag.obj() class Adam(optim.Adam): pass Then we can create an instance: .. code:: python myoptim = Adam(learning_rate=ag.Real(1e-2, 1e-1, log=True), wd=ag.Real(1e-5, 1e-3, log=True)) print(myoptim.cs) .. parsed-literal:: :class: output Configuration space object: Hyperparameters: learning_rate, Type: UniformFloat, Range: [0.01, 0.1], Default: 0.0316227766, on log-scale wd, Type: UniformFloat, Range: [1e-05, 0.001], Default: 0.0001, on log-scale Launch Experiments Using AutoGluon Object ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ AutoGluon Object is compatible with Fit API in AutoGluon tasks, and also works with user-defined training scripts using :func:`autogluon.autogluon_register_args`. We can start fitting: .. code:: python from autogluon.vision import ImagePredictor classifier = ImagePredictor().fit('cifar10', hyperparameters={'net': mynet, 'optimizer': myoptim, 'epochs': 1}, ngpus_per_trial=1) .. parsed-literal:: :class: output `time_limit=auto` set to `time_limit=7200`. Starting fit without HPO modified configs( != ): { root.img_cls.model resnet50_v1 != resnet50 root.train.rec_val ~/.mxnet/datasets/imagenet/rec/val.rec != auto root.train.early_stop_baseline 0.0 != -inf root.train.lr 0.1 != 0.01 root.train.rec_train_idx ~/.mxnet/datasets/imagenet/rec/train.idx != auto root.train.num_workers 4 != 8 root.train.rec_val_idx ~/.mxnet/datasets/imagenet/rec/val.idx != auto root.train.epochs 10 != 1 root.train.batch_size 128 != 16 root.train.early_stop_patience -1 != 10 root.train.num_training_samples 1281167 != -1 root.train.rec_train ~/.mxnet/datasets/imagenet/rec/train.rec != auto root.train.data_dir ~/.mxnet/datasets/imagenet != auto root.train.early_stop_max_value 1.0 != inf root.valid.batch_size 128 != 16 root.valid.num_workers 4 != 8 } Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/8c291d87/.trial_0/config.yaml Start training from [Epoch 0] Epoch[0] Batch [49] Speed: 72.258577 samples/sec accuracy=0.157500 lr=0.010000 Epoch[0] Batch [99] Speed: 72.579896 samples/sec accuracy=0.161250 lr=0.010000 Epoch[0] Batch [149] Speed: 72.004609 samples/sec accuracy=0.166250 lr=0.010000 Epoch[0] Batch [199] Speed: 71.669728 samples/sec accuracy=0.169687 lr=0.010000 Epoch[0] Batch [249] Speed: 71.164211 samples/sec accuracy=0.173500 lr=0.010000 Epoch[0] Batch [299] Speed: 70.555383 samples/sec accuracy=0.176667 lr=0.010000 Epoch[0] Batch [349] Speed: 69.966051 samples/sec accuracy=0.178750 lr=0.010000 Epoch[0] Batch [399] Speed: 69.486565 samples/sec accuracy=0.183594 lr=0.010000 Epoch[0] Batch [449] Speed: 69.124722 samples/sec accuracy=0.185417 lr=0.010000 Epoch[0] Batch [499] Speed: 68.603150 samples/sec accuracy=0.185875 lr=0.010000 Epoch[0] Batch [549] Speed: 68.262517 samples/sec accuracy=0.185114 lr=0.010000 Epoch[0] Batch [599] Speed: 67.778727 samples/sec accuracy=0.187083 lr=0.010000 Epoch[0] Batch [649] Speed: 67.319600 samples/sec accuracy=0.189327 lr=0.010000 Epoch[0] Batch [699] Speed: 66.997021 samples/sec accuracy=0.191607 lr=0.010000 Epoch[0] Batch [749] Speed: 66.709574 samples/sec accuracy=0.193833 lr=0.010000 Epoch[0] Batch [799] Speed: 66.607790 samples/sec accuracy=0.195469 lr=0.010000 Epoch[0] Batch [849] Speed: 68.958905 samples/sec accuracy=0.197794 lr=0.010000 Epoch[0] Batch [899] Speed: 70.448814 samples/sec accuracy=0.200208 lr=0.010000 Epoch[0] Batch [949] Speed: 71.484922 samples/sec accuracy=0.201645 lr=0.010000 Epoch[0] Batch [999] Speed: 71.748843 samples/sec accuracy=0.204063 lr=0.010000 Epoch[0] Batch [1049] Speed: 72.150712 samples/sec accuracy=0.205238 lr=0.010000 Epoch[0] Batch [1099] Speed: 72.339380 samples/sec accuracy=0.205227 lr=0.010000 Epoch[0] Batch [1149] Speed: 72.459271 samples/sec accuracy=0.206902 lr=0.010000 Epoch[0] Batch [1199] Speed: 72.563837 samples/sec accuracy=0.208750 lr=0.010000 Epoch[0] Batch [1249] Speed: 72.583615 samples/sec accuracy=0.210250 lr=0.010000 Epoch[0] Batch [1299] Speed: 72.588186 samples/sec accuracy=0.212452 lr=0.010000 Epoch[0] Batch [1349] Speed: 72.513160 samples/sec accuracy=0.213287 lr=0.010000 Epoch[0] Batch [1399] Speed: 72.495583 samples/sec accuracy=0.213839 lr=0.010000 Epoch[0] Batch [1449] Speed: 72.451813 samples/sec accuracy=0.214698 lr=0.010000 Epoch[0] Batch [1499] Speed: 72.343677 samples/sec accuracy=0.215625 lr=0.010000 Epoch[0] Batch [1549] Speed: 72.157853 samples/sec accuracy=0.216855 lr=0.010000 Epoch[0] Batch [1599] Speed: 71.981409 samples/sec accuracy=0.217227 lr=0.010000 Epoch[0] Batch [1649] Speed: 71.652033 samples/sec accuracy=0.219015 lr=0.010000 Epoch[0] Batch [1699] Speed: 71.638559 samples/sec accuracy=0.220184 lr=0.010000 Epoch[0] Batch [1749] Speed: 71.510581 samples/sec accuracy=0.220893 lr=0.010000 Epoch[0] Batch [1799] Speed: 71.471549 samples/sec accuracy=0.221632 lr=0.010000 Epoch[0] Batch [1849] Speed: 71.175401 samples/sec accuracy=0.222973 lr=0.010000 Epoch[0] Batch [1899] Speed: 70.938200 samples/sec accuracy=0.223586 lr=0.010000 Epoch[0] Batch [1949] Speed: 70.602699 samples/sec accuracy=0.223910 lr=0.010000 Epoch[0] Batch [1999] Speed: 70.134861 samples/sec accuracy=0.224562 lr=0.010000 Epoch[0] Batch [2049] Speed: 69.524083 samples/sec accuracy=0.225122 lr=0.010000 Epoch[0] Batch [2099] Speed: 69.059033 samples/sec accuracy=0.226131 lr=0.010000 Epoch[0] Batch [2149] Speed: 68.553819 samples/sec accuracy=0.226512 lr=0.010000 Epoch[0] Batch [2199] Speed: 68.040829 samples/sec accuracy=0.227330 lr=0.010000 Epoch[0] Batch [2249] Speed: 67.568369 samples/sec accuracy=0.228556 lr=0.010000 Epoch[0] Batch [2299] Speed: 67.086535 samples/sec accuracy=0.229755 lr=0.010000 Epoch[0] Batch [2349] Speed: 66.872884 samples/sec accuracy=0.231037 lr=0.010000 Epoch[0] Batch [2399] Speed: 66.440173 samples/sec accuracy=0.231615 lr=0.010000 Epoch[0] Batch [2449] Speed: 66.145407 samples/sec accuracy=0.231964 lr=0.010000 Epoch[0] Batch [2499] Speed: 65.775920 samples/sec accuracy=0.232100 lr=0.010000 Epoch[0] Batch [2549] Speed: 65.685150 samples/sec accuracy=0.232500 lr=0.010000 Epoch[0] Batch [2599] Speed: 65.612831 samples/sec accuracy=0.233101 lr=0.010000 Epoch[0] Batch [2649] Speed: 65.573857 samples/sec accuracy=0.233797 lr=0.010000 Epoch[0] Batch [2699] Speed: 65.574097 samples/sec accuracy=0.234560 lr=0.010000 Epoch[0] Batch [2749] Speed: 65.558847 samples/sec accuracy=0.235318 lr=0.010000 Epoch[0] Batch [2799] Speed: 65.647501 samples/sec accuracy=0.235871 lr=0.010000 Epoch[0] Batch [2849] Speed: 65.900436 samples/sec accuracy=0.236294 lr=0.010000 Epoch[0] Batch [2899] Speed: 66.148494 samples/sec accuracy=0.236983 lr=0.010000 Epoch[0] Batch [2949] Speed: 66.370391 samples/sec accuracy=0.237458 lr=0.010000 Epoch[0] Batch [2999] Speed: 66.502544 samples/sec accuracy=0.238208 lr=0.010000 Epoch[0] Batch [3049] Speed: 66.652083 samples/sec accuracy=0.239385 lr=0.010000 Epoch[0] Batch [3099] Speed: 66.672657 samples/sec accuracy=0.239315 lr=0.010000 Epoch[0] Batch [3149] Speed: 66.416612 samples/sec accuracy=0.239702 lr=0.010000 Epoch[0] Batch [3199] Speed: 66.027664 samples/sec accuracy=0.240020 lr=0.010000 Epoch[0] Batch [3249] Speed: 65.713096 samples/sec accuracy=0.240904 lr=0.010000 Epoch[0] Batch [3299] Speed: 65.584536 samples/sec accuracy=0.241288 lr=0.010000 Epoch[0] Batch [3349] Speed: 65.551247 samples/sec accuracy=0.241922 lr=0.010000 [Epoch 0] training: accuracy=0.242167 [Epoch 0] speed: 68 samples/sec time cost: 783.288290 [Epoch 0] validation: top1=0.317833 top5=0.839167 [Epoch 0] Current best top-1: 0.317833 vs previous -inf, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/8c291d87/.trial_0/best_checkpoint.pkl Unable to pickle object due to the reason: Can't pickle : it's not the same object as __main__.MyCifarResNet. This object is not saved. Applying the state from the best checkpoint... Unable to resume the state from the best checkpoint, using the latest state. Finished, total runtime is 807.73 s { 'best_config': { 'batch_size': 16, 'custom_net': MyCifarResNet( (features): HybridSequential( (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (3): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (3): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (4): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (5): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (6): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (7): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (4): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW) ) (output): Dense(64 -> 10, linear) ), 'custom_optimizer': <__main__.Adam object at 0x7f10f46df1f0>, 'dist_ip_addrs': None, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'early_stop_patience': 10, 'epochs': 1, 'final_fit': False, 'gpus': [0], 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/8c291d87', 'lr': 0.01, 'model': 'resnet50', 'ngpus_per_trial': 1, 'nthreads_per_trial': 128, 'num_trials': 1, 'num_workers': 8, 'problem_type': 'multiclass', 'scheduler': 'local', 'search_strategy': 'random', 'searcher': 'random', 'seed': 445, 'time_limits': 7200, 'wall_clock_tick': 1634599797.2190487}, 'total_time': 793.606636762619, 'train_acc': 0.24216666666666667, 'valid_acc': 0.31783333333333336} .. code:: python print(classifier.fit_summary()) .. parsed-literal:: :class: output {'train_acc': 0.24216666666666667, 'valid_acc': 0.31783333333333336, 'total_time': 793.606636762619, 'best_config': {'model': 'resnet50', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/8c291d87', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet( (features): HybridSequential( (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (3): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (3): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (4): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (5): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (6): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (7): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (4): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW) ) (output): Dense(64 -> 10, linear) ), 'custom_optimizer': <__main__.Adam object at 0x7f10f46df1f0>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 445, 'final_fit': False, 'wall_clock_tick': 1634599797.2190487, 'problem_type': 'multiclass'}, 'fit_history': {'train_acc': 0.24216666666666667, 'valid_acc': 0.31783333333333336, 'total_time': 793.606636762619, 'best_config': {'model': 'resnet50', 'lr': 0.01, 'num_trials': 1, 'epochs': 1, 'batch_size': 16, 'nthreads_per_trial': 128, 'ngpus_per_trial': 1, 'time_limits': 7200, 'search_strategy': 'random', 'dist_ip_addrs': None, 'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-course-v3/docs/_build/eval/tutorials/course/8c291d87', 'searcher': 'random', 'scheduler': 'local', 'custom_net': MyCifarResNet( (features): HybridSequential( (0): Conv2D(None -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(16 -> 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (3): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(16 -> 32, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (1): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (2): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (3): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (4): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (5): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (6): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) (7): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(32 -> 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (4): HybridSequential( (0): CIFARBasicBlockV1( (body): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) (2): Activation(relu) (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (4): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) (downsample): HybridSequential( (0): Conv2D(32 -> 64, kernel_size=(1, 1), stride=(2, 2), bias=False) (1): BatchNorm(axis=1, eps=1e-05, momentum=0.9, fix_gamma=False, use_global_stats=False, in_channels=None) ) ) ) (5): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW) ) (output): Dense(64 -> 10, linear) ), 'custom_optimizer': <__main__.Adam object at 0x7f10f46df1f0>, 'early_stop_patience': 10, 'early_stop_baseline': -inf, 'early_stop_max_value': inf, 'num_workers': 8, 'gpus': [0], 'seed': 445, 'final_fit': False, 'wall_clock_tick': 1634599797.2190487, 'problem_type': 'multiclass'}}}