Text Prediction - Customization (For MXNet backend only)¶
This advanced tutorial teaches you how to customize the hyperparameters
in TextPredictor by specifying:
A custom search space of candidate hyperparameter values to consider.
Which hyperparameter optimization (HPO) method should be used to actually search through this space.
import numpy as np
import warnings
import autogluon as ag
warnings.filterwarnings('ignore')
np.random.seed(123)
Stanford Sentiment Treebank Data¶
For demonstration, we use the Stanford Sentiment Treebank (SST) dataset.
from autogluon.core import TabularDataset
subsample_size = 1000 # subsample for faster demo, you may try specifying larger value
train_data = TabularDataset('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/train.parquet')
test_data = TabularDataset('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/dev.parquet')
train_data = train_data.sample(n=subsample_size, random_state=0)
train_data.head(10)
| sentence | label | |
|---|---|---|
| 43787 | very pleasing at its best moments | 1 |
| 16159 | , american chai is enough to make you put away... | 0 |
| 59015 | too much like an infomercial for ram dass 's l... | 0 |
| 5108 | a stirring visual sequence | 1 |
| 67052 | cool visual backmasking | 1 |
| 35938 | hard ground | 0 |
| 49879 | the striking , quietly vulnerable personality ... | 1 |
| 51591 | pan nalin 's exposition is beautiful and myste... | 1 |
| 56780 | wonderfully loopy | 1 |
| 28518 | most beautiful , evocative | 1 |
Configuring the TextPredictor¶
Pre-configured Hyperparameters¶
We provided a series of pre-configured hyperparameters. You may list the
keys from ag_text_presets via list_presets.
from autogluon.text.text_prediction.legacy_presets import ag_text_presets, list_presets
list_presets()
{'simple_presets': ['default',
'lower_quality_fast_train',
'medium_quality_faster_train',
'best_quality'],
'advanced_presets': ['electra_small_fuse_late',
'electra_base_fuse_late',
'electra_large_fuse_late',
'roberta_base_fuse_late',
'multi_cased_bert_base_fuse_late',
'electra_base_fuse_early',
'electra_base_all_text']}
There are two kinds of presets. The simple_presets are pre-defined
configurations recommended for most users, which allow you specify
whether you care more about predictive accuracy ('best_quality') or
more about training/inference speed ('lower_quality_fast_train')
The advanced_presets are pre-configured networks using different
Transformer backbones such as ELECTRA, RoBERTa, or Multilingual BERT,
and different feature fusion strategies. For example,
electra_small_fuse_late means we use the ELECTRA-small model as the
network backbone for text fields and use the late fusion strategy
described in “What’s happening inside?”. The default
preset is the same as electra_base_fuse_late. Now let’s train a
model on our data with specified presets.
from autogluon.text import TextPredictor
predictor = TextPredictor(path='ag_text_sst_electra_small', eval_metric='acc', label='label', backend='mxnet')
predictor.set_verbosity(0)
predictor.fit(train_data, presets='electra_small_fuse_late', time_limit=60, seed=123)
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_sst_electra_small/task0/training.log
<autogluon.text.text_prediction.predictor.TextPredictor at 0x7f0bc2b9a1f0>
Below we report both f1 and acc metrics for our predictions.
Note that if you really want to obtain the best F1 score, you should set
eval_metric='f1' when constructing the TextPredictor.
predictor.evaluate(test_data, metrics=['f1', 'acc'])
{'f1': 0.8030303030303031, 'acc': 0.7912844036697247}
To view the pre-registered hyperparameters, you can call
ag_text_presets.create(presets_name), e.g.,
import pprint
pprint.pprint(ag_text_presets.create('electra_small_fuse_late'))
{'models': {'MultimodalTextModel': {'backend': 'gluonnlp_v0',
'search_space': {'model.backbone.name': 'google_electra_small',
'model.network.agg_net.agg_type': 'concat',
'model.network.aggregate_categorical': True,
'model.use_avg_nbest': True,
'optimization.batch_size': 128,
'optimization.layerwise_lr_decay': 0.8,
'optimization.lr': Categorical[0.0001],
'optimization.nbest': 3,
'optimization.num_train_epochs': 10,
'optimization.per_device_batch_size': 8,
'optimization.wd': 0.0001,
'preprocessing.categorical.convert_to_text': False,
'preprocessing.numerical.convert_to_text': False}}},
'tune_kwargs': {'num_trials': 1,
'scheduler_options': None,
'search_options': None,
'search_strategy': 'local',
'searcher': 'local_random'}}
Another way to specify a custom TextPredictor configuration is via the
hyperparameters argument.
predictor = TextPredictor(path='ag_text_customize1', eval_metric='acc', label='label', backend='mxnet')
predictor.fit(train_data, hyperparameters=ag_text_presets.create('electra_small_fuse_late'),
time_limit=30, seed=123)
Problem Type="binary"
Column Types:
- "sentence": text
- "label": categorical
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_customize1/task0/training.log
Fitting and transforming the train data...
Done! Preprocessor saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_customize1/task0/preprocessor.pkl
Process dev set...
Done!
Max length for chunking text: 64, Stochastic chunk: Train-False/Test-False, Test #repeat: 1.
#Total Params/Fixed Params=13516290/0
Using gradient accumulation. Global batch size = 128
Local training results will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_customize1/task0/results_local.jsonl.
[Iter 1/70, Epoch 0] train loss=7.41e-01, gnorm=7.22e+00, lr=1.43e-05, #samples processed=128, #sample per second=331.10. ETA=0.44min
[Iter 2/70, Epoch 0] train loss=1.00e+00, gnorm=7.03e+00, lr=2.86e-05, #samples processed=128, #sample per second=519.03. ETA=0.36min
[Iter 2/70, Epoch 0] Validation f1=7.1280e-01, mcc=1.4917e-01, roc_auc=6.0192e-01, accuracy=5.8500e-01, log_loss=7.0598e-01, Time computing validation-score=0.125s, Total time spent=0.01min. Found improved model=True, Improved top-3 models=True
[Iter 3/70, Epoch 0] train loss=8.80e-01, gnorm=4.48e+00, lr=4.29e-05, #samples processed=128, #sample per second=261.77. ETA=0.42min
[Iter 4/70, Epoch 0] train loss=7.23e-01, gnorm=6.01e+00, lr=5.71e-05, #samples processed=128, #sample per second=546.62. ETA=0.37min
[Iter 4/70, Epoch 0] Validation f1=7.3469e-01, mcc=2.5191e-01, roc_auc=6.8778e-01, accuracy=6.1000e-01, log_loss=6.7474e-01, Time computing validation-score=0.126s, Total time spent=0.03min. Found improved model=True, Improved top-3 models=True
[Iter 5/70, Epoch 0] train loss=7.06e-01, gnorm=3.00e+00, lr=7.14e-05, #samples processed=128, #sample per second=237.03. ETA=0.41min
[Iter 6/70, Epoch 0] train loss=7.78e-01, gnorm=3.18e+00, lr=8.57e-05, #samples processed=128, #sample per second=535.37. ETA=0.38min
[Iter 6/70, Epoch 0] Validation f1=5.8101e-01, mcc=2.9705e-01, roc_auc=7.0081e-01, accuracy=6.2500e-01, log_loss=6.5234e-01, Time computing validation-score=0.124s, Total time spent=0.04min. Found improved model=True, Improved top-3 models=True
[Iter 7/70, Epoch 0] train loss=8.00e-01, gnorm=3.42e+00, lr=1.00e-04, #samples processed=128, #sample per second=236.80. ETA=0.40min
[Iter 8/70, Epoch 1] train loss=6.85e-01, gnorm=3.90e+00, lr=9.84e-05, #samples processed=128, #sample per second=513.62. ETA=0.38min
[Iter 8/70, Epoch 1] Validation f1=7.2727e-01, mcc=2.0996e-01, roc_auc=7.5747e-01, accuracy=5.9500e-01, log_loss=6.4385e-01, Time computing validation-score=0.127s, Total time spent=0.05min. Found improved model=False, Improved top-3 models=True
[Iter 9/70, Epoch 1] train loss=6.99e-01, gnorm=3.15e+00, lr=9.68e-05, #samples processed=128, #sample per second=263.85. ETA=0.39min
[Iter 10/70, Epoch 1] train loss=6.78e-01, gnorm=2.56e+00, lr=9.52e-05, #samples processed=128, #sample per second=549.71. ETA=0.37min
[Iter 10/70, Epoch 1] Validation f1=7.6296e-01, mcc=3.7689e-01, roc_auc=7.7091e-01, accuracy=6.8000e-01, log_loss=6.0942e-01, Time computing validation-score=0.124s, Total time spent=0.07min. Found improved model=True, Improved top-3 models=True
[Iter 11/70, Epoch 1] train loss=6.86e-01, gnorm=2.39e+00, lr=9.37e-05, #samples processed=128, #sample per second=220.37. ETA=0.38min
[Iter 12/70, Epoch 1] train loss=6.14e-01, gnorm=2.59e+00, lr=9.21e-05, #samples processed=128, #sample per second=519.29. ETA=0.36min
[Iter 12/70, Epoch 1] Validation f1=7.4021e-01, mcc=2.8401e-01, roc_auc=8.0919e-01, accuracy=6.3500e-01, log_loss=5.9405e-01, Time computing validation-score=0.134s, Total time spent=0.08min. Found improved model=False, Improved top-3 models=True
[Iter 13/70, Epoch 1] train loss=6.09e-01, gnorm=2.95e+00, lr=9.05e-05, #samples processed=128, #sample per second=253.16. ETA=0.36min
[Iter 14/70, Epoch 1] train loss=5.52e-01, gnorm=2.18e+00, lr=8.89e-05, #samples processed=128, #sample per second=526.45. ETA=0.35min
[Iter 14/70, Epoch 1] Validation f1=7.6316e-01, mcc=4.5160e-01, roc_auc=8.1212e-01, accuracy=7.3000e-01, log_loss=5.3268e-01, Time computing validation-score=0.125s, Total time spent=0.09min. Found improved model=True, Improved top-3 models=True
[Iter 15/70, Epoch 2] train loss=5.86e-01, gnorm=3.03e+00, lr=8.73e-05, #samples processed=128, #sample per second=221.07. ETA=0.35min
[Iter 16/70, Epoch 2] train loss=4.78e-01, gnorm=2.03e+00, lr=8.57e-05, #samples processed=128, #sample per second=535.60. ETA=0.34min
[Iter 16/70, Epoch 2] Validation f1=7.8431e-01, mcc=4.5580e-01, roc_auc=8.4333e-01, accuracy=7.2500e-01, log_loss=5.5335e-01, Time computing validation-score=0.123s, Total time spent=0.10min. Found improved model=False, Improved top-3 models=True
[Iter 17/70, Epoch 2] train loss=5.63e-01, gnorm=3.05e+00, lr=8.41e-05, #samples processed=128, #sample per second=262.59. ETA=0.34min
[Iter 18/70, Epoch 2] train loss=4.18e-01, gnorm=2.66e+00, lr=8.25e-05, #samples processed=128, #sample per second=526.87. ETA=0.33min
[Iter 18/70, Epoch 2] Validation f1=8.1481e-01, mcc=5.5044e-01, roc_auc=8.6758e-01, accuracy=7.7500e-01, log_loss=4.8310e-01, Time computing validation-score=0.125s, Total time spent=0.12min. Found improved model=True, Improved top-3 models=True
[Iter 19/70, Epoch 2] train loss=4.89e-01, gnorm=4.38e+00, lr=8.10e-05, #samples processed=128, #sample per second=221.10. ETA=0.33min
[Iter 20/70, Epoch 2] train loss=5.28e-01, gnorm=6.20e+00, lr=7.94e-05, #samples processed=128, #sample per second=498.01. ETA=0.32min
[Iter 20/70, Epoch 2] Validation f1=8.2524e-01, mcc=6.4776e-01, roc_auc=8.8788e-01, accuracy=8.2000e-01, log_loss=4.2861e-01, Time computing validation-score=0.130s, Total time spent=0.13min. Found improved model=True, Improved top-3 models=True
[Iter 21/70, Epoch 2] train loss=4.22e-01, gnorm=7.70e+00, lr=7.78e-05, #samples processed=128, #sample per second=212.43. ETA=0.32min
[Iter 22/70, Epoch 3] train loss=5.01e-01, gnorm=4.89e+00, lr=7.62e-05, #samples processed=128, #sample per second=479.27. ETA=0.31min
[Iter 22/70, Epoch 3] Validation f1=8.1600e-01, mcc=5.4829e-01, roc_auc=8.8515e-01, accuracy=7.7000e-01, log_loss=5.2490e-01, Time computing validation-score=0.130s, Total time spent=0.15min. Found improved model=False, Improved top-3 models=True
[Iter 23/70, Epoch 3] train loss=3.54e-01, gnorm=5.08e+00, lr=7.46e-05, #samples processed=128, #sample per second=259.10. ETA=0.31min
[Iter 24/70, Epoch 3] train loss=4.50e-01, gnorm=5.71e+00, lr=7.30e-05, #samples processed=128, #sample per second=495.21. ETA=0.30min
[Iter 24/70, Epoch 3] Validation f1=8.5124e-01, mcc=6.4498e-01, roc_auc=9.0202e-01, accuracy=8.2000e-01, log_loss=4.7084e-01, Time computing validation-score=0.131s, Total time spent=0.16min. Found improved model=True, Improved top-3 models=True
[Iter 25/70, Epoch 3] train loss=4.06e-01, gnorm=4.85e+00, lr=7.14e-05, #samples processed=128, #sample per second=206.48. ETA=0.30min
[Iter 26/70, Epoch 3] train loss=4.16e-01, gnorm=3.64e+00, lr=6.98e-05, #samples processed=128, #sample per second=502.22. ETA=0.29min
[Iter 26/70, Epoch 3] Validation f1=8.6758e-01, mcc=7.0740e-01, roc_auc=9.1273e-01, accuracy=8.5500e-01, log_loss=3.6598e-01, Time computing validation-score=0.130s, Total time spent=0.17min. Found improved model=True, Improved top-3 models=True
[Iter 27/70, Epoch 3] train loss=4.21e-01, gnorm=9.87e+00, lr=6.83e-05, #samples processed=128, #sample per second=213.15. ETA=0.28min
[Iter 28/70, Epoch 3] train loss=2.92e-01, gnorm=7.02e+00, lr=6.67e-05, #samples processed=128, #sample per second=504.93. ETA=0.27min
[Iter 28/70, Epoch 3] Validation f1=8.3544e-01, mcc=6.0854e-01, roc_auc=9.1515e-01, accuracy=8.0500e-01, log_loss=4.1337e-01, Time computing validation-score=0.132s, Total time spent=0.19min. Found improved model=False, Improved top-3 models=False
[Iter 29/70, Epoch 4] train loss=3.53e-01, gnorm=2.90e+00, lr=6.51e-05, #samples processed=128, #sample per second=326.60. ETA=0.27min
[Iter 30/70, Epoch 4] train loss=4.08e-01, gnorm=1.03e+01, lr=6.35e-05, #samples processed=128, #sample per second=502.21. ETA=0.26min
[Iter 30/70, Epoch 4] Validation f1=8.1124e-01, mcc=5.3591e-01, roc_auc=9.1677e-01, accuracy=7.6500e-01, log_loss=5.0395e-01, Time computing validation-score=0.130s, Total time spent=0.20min. Found improved model=False, Improved top-3 models=False
[Iter 31/70, Epoch 4] train loss=3.61e-01, gnorm=9.61e+00, lr=6.19e-05, #samples processed=128, #sample per second=332.66. ETA=0.25min
[Iter 32/70, Epoch 4] train loss=3.00e-01, gnorm=3.82e+00, lr=6.03e-05, #samples processed=128, #sample per second=506.33. ETA=0.24min
[Iter 32/70, Epoch 4] Validation f1=8.5463e-01, mcc=6.6598e-01, roc_auc=9.1182e-01, accuracy=8.3500e-01, log_loss=3.8519e-01, Time computing validation-score=0.131s, Total time spent=0.21min. Found improved model=False, Improved top-3 models=True
[Iter 33/70, Epoch 4] train loss=3.18e-01, gnorm=3.86e+00, lr=5.87e-05, #samples processed=128, #sample per second=253.86. ETA=0.24min
[Iter 34/70, Epoch 4] train loss=3.87e-01, gnorm=5.84e+00, lr=5.71e-05, #samples processed=128, #sample per second=497.66. ETA=0.23min
[Iter 34/70, Epoch 4] Validation f1=8.5714e-01, mcc=6.8813e-01, roc_auc=9.1707e-01, accuracy=8.4500e-01, log_loss=3.6984e-01, Time computing validation-score=0.130s, Total time spent=0.22min. Found improved model=False, Improved top-3 models=True
[Iter 35/70, Epoch 4] train loss=2.94e-01, gnorm=1.02e+01, lr=5.56e-05, #samples processed=128, #sample per second=240.26. ETA=0.23min
[Iter 36/70, Epoch 5] train loss=3.37e-01, gnorm=4.86e+00, lr=5.40e-05, #samples processed=128, #sample per second=504.08. ETA=0.22min
[Iter 36/70, Epoch 5] Validation f1=8.3817e-01, mcc=6.1207e-01, roc_auc=9.2717e-01, accuracy=8.0500e-01, log_loss=4.3120e-01, Time computing validation-score=0.131s, Total time spent=0.23min. Found improved model=False, Improved top-3 models=False
[Iter 37/70, Epoch 5] train loss=2.70e-01, gnorm=5.44e+00, lr=5.24e-05, #samples processed=128, #sample per second=328.39. ETA=0.21min
[Iter 38/70, Epoch 5] train loss=3.17e-01, gnorm=5.18e+00, lr=5.08e-05, #samples processed=128, #sample per second=498.50. ETA=0.20min
[Iter 38/70, Epoch 5] Validation f1=8.4337e-01, mcc=6.2323e-01, roc_auc=9.2889e-01, accuracy=8.0500e-01, log_loss=4.7520e-01, Time computing validation-score=0.132s, Total time spent=0.24min. Found improved model=False, Improved top-3 models=False
[Iter 39/70, Epoch 5] train loss=2.51e-01, gnorm=4.58e+00, lr=4.92e-05, #samples processed=128, #sample per second=327.53. ETA=0.20min
[Iter 40/70, Epoch 5] train loss=2.64e-01, gnorm=3.46e+00, lr=4.76e-05, #samples processed=128, #sample per second=498.44. ETA=0.19min
[Iter 40/70, Epoch 5] Validation f1=8.6580e-01, mcc=6.8771e-01, roc_auc=9.2960e-01, accuracy=8.4500e-01, log_loss=3.7012e-01, Time computing validation-score=0.135s, Total time spent=0.26min. Found improved model=False, Improved top-3 models=True
[Iter 41/70, Epoch 5] train loss=2.32e-01, gnorm=3.95e+00, lr=4.60e-05, #samples processed=128, #sample per second=258.91. ETA=0.18min
[Iter 42/70, Epoch 5] train loss=1.93e-01, gnorm=5.35e+00, lr=4.44e-05, #samples processed=128, #sample per second=523.28. ETA=0.18min
[Iter 42/70, Epoch 5] Validation f1=8.5981e-01, mcc=7.0007e-01, roc_auc=9.3061e-01, accuracy=8.5000e-01, log_loss=3.4404e-01, Time computing validation-score=0.124s, Total time spent=0.27min. Found improved model=False, Improved top-3 models=True
[Iter 43/70, Epoch 6] train loss=2.76e-01, gnorm=7.09e+00, lr=4.29e-05, #samples processed=128, #sample per second=273.36. ETA=0.17min
[Iter 44/70, Epoch 6] train loss=1.88e-01, gnorm=6.67e+00, lr=4.13e-05, #samples processed=128, #sample per second=550.21. ETA=0.16min
[Iter 44/70, Epoch 6] Validation f1=8.5586e-01, mcc=6.7625e-01, roc_auc=9.2960e-01, accuracy=8.4000e-01, log_loss=3.5068e-01, Time computing validation-score=0.123s, Total time spent=0.28min. Found improved model=False, Improved top-3 models=False
[Iter 45/70, Epoch 6] train loss=2.54e-01, gnorm=8.51e+00, lr=3.97e-05, #samples processed=128, #sample per second=360.57. ETA=0.16min
[Iter 46/70, Epoch 6] train loss=2.30e-01, gnorm=4.31e+00, lr=3.81e-05, #samples processed=128, #sample per second=545.72. ETA=0.15min
[Iter 46/70, Epoch 6] Validation f1=8.4337e-01, mcc=6.2323e-01, roc_auc=9.2636e-01, accuracy=8.0500e-01, log_loss=5.0588e-01, Time computing validation-score=0.122s, Total time spent=0.29min. Found improved model=False, Improved top-3 models=False
[Iter 47/70, Epoch 6] train loss=2.33e-01, gnorm=6.52e+00, lr=3.65e-05, #samples processed=128, #sample per second=359.25. ETA=0.14min
[Iter 48/70, Epoch 6] train loss=2.87e-01, gnorm=1.45e+01, lr=3.49e-05, #samples processed=128, #sample per second=558.63. ETA=0.14min
[Iter 48/70, Epoch 6] Validation f1=8.2677e-01, mcc=5.7751e-01, roc_auc=9.2525e-01, accuracy=7.8000e-01, log_loss=6.0640e-01, Time computing validation-score=0.121s, Total time spent=0.30min. Found improved model=False, Improved top-3 models=False
[Iter 49/70, Epoch 6] train loss=2.19e-01, gnorm=9.96e+00, lr=3.33e-05, #samples processed=128, #sample per second=353.07. ETA=0.13min
[Iter 50/70, Epoch 7] train loss=2.96e-01, gnorm=1.10e+01, lr=3.17e-05, #samples processed=128, #sample per second=541.12. ETA=0.12min
[Iter 50/70, Epoch 7] Validation f1=8.5366e-01, mcc=6.5067e-01, roc_auc=9.2576e-01, accuracy=8.2000e-01, log_loss=4.9763e-01, Time computing validation-score=0.123s, Total time spent=0.31min. Found improved model=False, Improved top-3 models=False
[Iter 51/70, Epoch 7] train loss=1.77e-01, gnorm=8.51e+00, lr=3.02e-05, #samples processed=128, #sample per second=355.22. ETA=0.12min
[Iter 52/70, Epoch 7] train loss=1.71e-01, gnorm=3.41e+00, lr=2.86e-05, #samples processed=128, #sample per second=542.92. ETA=0.11min
[Iter 52/70, Epoch 7] Validation f1=8.5714e-01, mcc=6.6715e-01, roc_auc=9.2657e-01, accuracy=8.3500e-01, log_loss=3.8948e-01, Time computing validation-score=0.123s, Total time spent=0.32min. Found improved model=False, Improved top-3 models=False
[Iter 53/70, Epoch 7] train loss=1.88e-01, gnorm=7.33e+00, lr=2.70e-05, #samples processed=128, #sample per second=356.88. ETA=0.10min
[Iter 54/70, Epoch 7] train loss=1.42e-01, gnorm=3.42e+00, lr=2.54e-05, #samples processed=128, #sample per second=529.51. ETA=0.10min
[Iter 54/70, Epoch 7] Validation f1=8.5714e-01, mcc=6.6715e-01, roc_auc=9.2606e-01, accuracy=8.3500e-01, log_loss=3.9039e-01, Time computing validation-score=0.122s, Total time spent=0.33min. Found improved model=False, Improved top-3 models=False
[Iter 55/70, Epoch 7] train loss=1.98e-01, gnorm=5.14e+00, lr=2.38e-05, #samples processed=128, #sample per second=361.14. ETA=0.09min
[Iter 56/70, Epoch 7] train loss=2.18e-01, gnorm=4.05e+00, lr=2.22e-05, #samples processed=128, #sample per second=546.78. ETA=0.08min
[Iter 56/70, Epoch 7] Validation f1=8.5593e-01, mcc=6.5989e-01, roc_auc=9.2545e-01, accuracy=8.3000e-01, log_loss=4.3357e-01, Time computing validation-score=0.125s, Total time spent=0.34min. Found improved model=False, Improved top-3 models=False
[Iter 57/70, Epoch 8] train loss=1.28e-01, gnorm=2.89e+00, lr=2.06e-05, #samples processed=128, #sample per second=356.21. ETA=0.08min
[Iter 58/70, Epoch 8] train loss=1.59e-01, gnorm=2.95e+00, lr=1.90e-05, #samples processed=128, #sample per second=543.69. ETA=0.07min
[Iter 58/70, Epoch 8] Validation f1=8.6192e-01, mcc=6.7316e-01, roc_auc=9.2465e-01, accuracy=8.3500e-01, log_loss=4.7564e-01, Time computing validation-score=0.123s, Total time spent=0.35min. Found improved model=False, Improved top-3 models=False
[Iter 59/70, Epoch 8] train loss=1.61e-01, gnorm=3.15e+00, lr=1.75e-05, #samples processed=128, #sample per second=356.93. ETA=0.07min
[Iter 60/70, Epoch 8] train loss=1.21e-01, gnorm=3.42e+00, lr=1.59e-05, #samples processed=128, #sample per second=548.44. ETA=0.06min
[Iter 60/70, Epoch 8] Validation f1=8.5833e-01, mcc=6.6375e-01, roc_auc=9.2414e-01, accuracy=8.3000e-01, log_loss=4.9380e-01, Time computing validation-score=0.125s, Total time spent=0.36min. Found improved model=False, Improved top-3 models=False
[Iter 61/70, Epoch 8] train loss=1.81e-01, gnorm=5.42e+00, lr=1.43e-05, #samples processed=128, #sample per second=355.07. ETA=0.05min
[Iter 62/70, Epoch 8] train loss=1.11e-01, gnorm=4.94e+00, lr=1.27e-05, #samples processed=128, #sample per second=549.02. ETA=0.05min
[Iter 62/70, Epoch 8] Validation f1=8.6192e-01, mcc=6.7316e-01, roc_auc=9.2465e-01, accuracy=8.3500e-01, log_loss=4.7258e-01, Time computing validation-score=0.126s, Total time spent=0.37min. Found improved model=False, Improved top-3 models=False
[Iter 63/70, Epoch 8] train loss=1.06e-01, gnorm=3.76e+00, lr=1.11e-05, #samples processed=128, #sample per second=353.86. ETA=0.04min
[Iter 64/70, Epoch 9] train loss=1.98e-01, gnorm=4.13e+00, lr=9.52e-06, #samples processed=128, #sample per second=542.08. ETA=0.04min
[Iter 64/70, Epoch 9] Validation f1=8.6441e-01, mcc=6.8071e-01, roc_auc=9.2495e-01, accuracy=8.4000e-01, log_loss=4.3706e-01, Time computing validation-score=0.124s, Total time spent=0.38min. Found improved model=False, Improved top-3 models=False
[Iter 65/70, Epoch 9] train loss=2.14e-01, gnorm=4.35e+00, lr=7.94e-06, #samples processed=128, #sample per second=357.50. ETA=0.03min
[Iter 66/70, Epoch 9] train loss=1.15e-01, gnorm=3.81e+00, lr=6.35e-06, #samples processed=128, #sample per second=551.30. ETA=0.02min
[Iter 66/70, Epoch 9] Validation f1=8.5714e-01, mcc=6.6715e-01, roc_auc=9.2505e-01, accuracy=8.3500e-01, log_loss=4.1776e-01, Time computing validation-score=0.123s, Total time spent=0.39min. Found improved model=False, Improved top-3 models=False
Early stopping patience reached!
Training completed. Auto-saving to "ag_text_customize1/". For loading the model, you can use predictor = TextPredictor.load("ag_text_customize1/")
<autogluon.text.text_prediction.predictor.TextPredictor at 0x7f0b1512e160>
Custom Hyperparameter Values¶
The pre-registered configurations provide reasonable default hyperparameters. A common workflow is to first train a model with one of the presets and then tune some hyperparameters to see if the performance can be further improved. In the example below, we set the number of training epochs to 5 and the learning rate to be 5E-5.
hyperparameters = ag_text_presets.create('electra_small_fuse_late')
hyperparameters['models']['MultimodalTextModel']['search_space']['optimization.num_train_epochs'] = 5
hyperparameters['models']['MultimodalTextModel']['search_space']['optimization.lr'] = ag.core.space.Categorical(5E-5)
predictor = TextPredictor(path='ag_text_customize2', eval_metric='acc', label='label', backend='mxnet')
predictor.fit(train_data, hyperparameters=hyperparameters, time_limit=30, seed=123)
Problem Type="binary"
Column Types:
- "sentence": text
- "label": categorical
The GluonNLP V0 backend is used. We will use 8 cpus and 1 gpus to train each trial.
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_customize2/task0/training.log
Fitting and transforming the train data...
Done! Preprocessor saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_customize2/task0/preprocessor.pkl
Process dev set...
Done!
Max length for chunking text: 64, Stochastic chunk: Train-False/Test-False, Test #repeat: 1.
#Total Params/Fixed Params=13516290/0
Using gradient accumulation. Global batch size = 128
Local training results will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_customize2/task0/results_local.jsonl.
[Iter 1/35, Epoch 0] train loss=7.79e-01, gnorm=7.51e+00, lr=1.25e-05, #samples processed=128, #sample per second=332.50. ETA=0.22min
[Iter 2/35, Epoch 0] train loss=8.90e-01, gnorm=6.69e+00, lr=2.50e-05, #samples processed=128, #sample per second=532.36. ETA=0.17min
[Iter 2/35, Epoch 0] Validation f1=6.9039e-01, mcc=8.4205e-02, roc_auc=5.7687e-01, accuracy=5.6500e-01, log_loss=7.1596e-01, Time computing validation-score=0.124s, Total time spent=0.01min. Found improved model=True, Improved top-3 models=True
[Iter 3/35, Epoch 0] train loss=7.16e-01, gnorm=4.34e+00, lr=3.75e-05, #samples processed=128, #sample per second=270.97. ETA=0.20min
[Iter 4/35, Epoch 0] train loss=7.47e-01, gnorm=5.52e+00, lr=5.00e-05, #samples processed=128, #sample per second=554.80. ETA=0.17min
[Iter 4/35, Epoch 0] Validation f1=7.2241e-01, mcc=1.7854e-01, roc_auc=6.7596e-01, accuracy=5.8500e-01, log_loss=6.8245e-01, Time computing validation-score=0.123s, Total time spent=0.03min. Found improved model=True, Improved top-3 models=True
[Iter 5/35, Epoch 0] train loss=6.84e-01, gnorm=3.43e+00, lr=4.84e-05, #samples processed=128, #sample per second=242.85. ETA=0.19min
[Iter 6/35, Epoch 0] train loss=7.91e-01, gnorm=3.24e+00, lr=4.68e-05, #samples processed=128, #sample per second=537.01. ETA=0.17min
[Iter 6/35, Epoch 0] Validation f1=6.4840e-01, mcc=2.2302e-01, roc_auc=7.0768e-01, accuracy=6.1500e-01, log_loss=6.2866e-01, Time computing validation-score=0.124s, Total time spent=0.04min. Found improved model=True, Improved top-3 models=True
[Iter 7/35, Epoch 0] train loss=7.47e-01, gnorm=3.70e+00, lr=4.52e-05, #samples processed=128, #sample per second=228.42. ETA=0.18min
[Iter 8/35, Epoch 1] train loss=6.63e-01, gnorm=3.52e+00, lr=4.35e-05, #samples processed=128, #sample per second=536.56. ETA=0.16min
[Iter 8/35, Epoch 1] Validation f1=7.4906e-01, mcc=3.3393e-01, roc_auc=7.2677e-01, accuracy=6.6500e-01, log_loss=6.1986e-01, Time computing validation-score=0.124s, Total time spent=0.05min. Found improved model=True, Improved top-3 models=True
[Iter 9/35, Epoch 1] train loss=6.87e-01, gnorm=2.58e+00, lr=4.19e-05, #samples processed=128, #sample per second=216.17. ETA=0.17min
[Iter 10/35, Epoch 1] train loss=6.26e-01, gnorm=2.42e+00, lr=4.03e-05, #samples processed=128, #sample per second=543.15. ETA=0.16min
[Iter 10/35, Epoch 1] Validation f1=7.3649e-01, mcc=2.6392e-01, roc_auc=7.6556e-01, accuracy=6.1000e-01, log_loss=6.3489e-01, Time computing validation-score=0.126s, Total time spent=0.07min. Found improved model=False, Improved top-3 models=True
[Iter 11/35, Epoch 1] train loss=6.58e-01, gnorm=3.08e+00, lr=3.87e-05, #samples processed=128, #sample per second=269.10. ETA=0.15min
[Iter 12/35, Epoch 1] train loss=6.10e-01, gnorm=2.67e+00, lr=3.71e-05, #samples processed=128, #sample per second=520.55. ETA=0.14min
[Iter 12/35, Epoch 1] Validation f1=7.4157e-01, mcc=3.0947e-01, roc_auc=7.8040e-01, accuracy=6.5500e-01, log_loss=5.9525e-01, Time computing validation-score=0.126s, Total time spent=0.08min. Found improved model=False, Improved top-3 models=True
[Iter 13/35, Epoch 1] train loss=6.16e-01, gnorm=2.21e+00, lr=3.55e-05, #samples processed=128, #sample per second=264.73. ETA=0.14min
[Iter 14/35, Epoch 1] train loss=6.34e-01, gnorm=2.54e+00, lr=3.39e-05, #samples processed=128, #sample per second=552.52. ETA=0.13min
[Iter 14/35, Epoch 1] Validation f1=7.2072e-01, mcc=3.7255e-01, roc_auc=7.8303e-01, accuracy=6.9000e-01, log_loss=5.7716e-01, Time computing validation-score=0.123s, Total time spent=0.09min. Found improved model=True, Improved top-3 models=True
[Iter 15/35, Epoch 2] train loss=6.41e-01, gnorm=3.17e+00, lr=3.23e-05, #samples processed=128, #sample per second=223.47. ETA=0.13min
[Iter 16/35, Epoch 2] train loss=6.02e-01, gnorm=3.42e+00, lr=3.06e-05, #samples processed=128, #sample per second=550.44. ETA=0.12min
[Iter 16/35, Epoch 2] Validation f1=7.2803e-01, mcc=3.3710e-01, roc_auc=8.0364e-01, accuracy=6.7500e-01, log_loss=5.6195e-01, Time computing validation-score=0.123s, Total time spent=0.10min. Found improved model=False, Improved top-3 models=True
[Iter 17/35, Epoch 2] train loss=6.52e-01, gnorm=3.29e+00, lr=2.90e-05, #samples processed=128, #sample per second=282.38. ETA=0.11min
[Iter 18/35, Epoch 2] train loss=5.71e-01, gnorm=2.41e+00, lr=2.74e-05, #samples processed=128, #sample per second=531.56. ETA=0.10min
[Iter 18/35, Epoch 2] Validation f1=7.4386e-01, mcc=2.9630e-01, roc_auc=8.2899e-01, accuracy=6.3500e-01, log_loss=5.9281e-01, Time computing validation-score=0.124s, Total time spent=0.11min. Found improved model=False, Improved top-3 models=False
[Iter 19/35, Epoch 2] train loss=6.23e-01, gnorm=4.63e+00, lr=2.58e-05, #samples processed=128, #sample per second=360.42. ETA=0.10min
[Iter 20/35, Epoch 2] train loss=6.12e-01, gnorm=5.09e+00, lr=2.42e-05, #samples processed=128, #sample per second=553.33. ETA=0.09min
[Iter 20/35, Epoch 2] Validation f1=7.5362e-01, mcc=3.3980e-01, roc_auc=8.3667e-01, accuracy=6.6000e-01, log_loss=5.6732e-01, Time computing validation-score=0.123s, Total time spent=0.12min. Found improved model=False, Improved top-3 models=False
[Iter 21/35, Epoch 2] train loss=5.44e-01, gnorm=3.74e+00, lr=2.26e-05, #samples processed=128, #sample per second=351.83. ETA=0.08min
[Iter 22/35, Epoch 3] train loss=5.84e-01, gnorm=2.74e+00, lr=2.10e-05, #samples processed=128, #sample per second=543.39. ETA=0.08min
[Iter 22/35, Epoch 3] Validation f1=7.9184e-01, mcc=4.8817e-01, roc_auc=8.3899e-01, accuracy=7.4500e-01, log_loss=5.2210e-01, Time computing validation-score=0.123s, Total time spent=0.14min. Found improved model=True, Improved top-3 models=True
[Iter 23/35, Epoch 3] train loss=5.54e-01, gnorm=4.01e+00, lr=1.94e-05, #samples processed=128, #sample per second=208.97. ETA=0.07min
[Iter 24/35, Epoch 3] train loss=5.14e-01, gnorm=2.96e+00, lr=1.77e-05, #samples processed=128, #sample per second=560.43. ETA=0.07min
[Iter 24/35, Epoch 3] Validation f1=8.0488e-01, mcc=5.2140e-01, roc_auc=8.4545e-01, accuracy=7.6000e-01, log_loss=5.1100e-01, Time computing validation-score=0.124s, Total time spent=0.15min. Found improved model=True, Improved top-3 models=True
[Iter 25/35, Epoch 3] train loss=4.82e-01, gnorm=2.45e+00, lr=1.61e-05, #samples processed=128, #sample per second=220.84. ETA=0.06min
[Iter 26/35, Epoch 3] train loss=6.02e-01, gnorm=2.97e+00, lr=1.45e-05, #samples processed=128, #sample per second=561.28. ETA=0.05min
[Iter 26/35, Epoch 3] Validation f1=7.9681e-01, mcc=4.9476e-01, roc_auc=8.5263e-01, accuracy=7.4500e-01, log_loss=5.1082e-01, Time computing validation-score=0.123s, Total time spent=0.16min. Found improved model=False, Improved top-3 models=True
[Iter 27/35, Epoch 3] train loss=5.21e-01, gnorm=2.73e+00, lr=1.29e-05, #samples processed=128, #sample per second=271.42. ETA=0.05min
[Iter 28/35, Epoch 3] train loss=4.59e-01, gnorm=2.82e+00, lr=1.13e-05, #samples processed=128, #sample per second=552.74. ETA=0.04min
[Iter 28/35, Epoch 3] Validation f1=7.8599e-01, mcc=4.5887e-01, roc_auc=8.5828e-01, accuracy=7.2500e-01, log_loss=5.1587e-01, Time computing validation-score=0.127s, Total time spent=0.17min. Found improved model=False, Improved top-3 models=False
[Iter 29/35, Epoch 4] train loss=5.88e-01, gnorm=4.39e+00, lr=9.68e-06, #samples processed=128, #sample per second=336.39. ETA=0.04min
[Iter 30/35, Epoch 4] train loss=6.09e-01, gnorm=4.37e+00, lr=8.06e-06, #samples processed=128, #sample per second=523.67. ETA=0.03min
[Iter 30/35, Epoch 4] Validation f1=8.0159e-01, mcc=5.0721e-01, roc_auc=8.6111e-01, accuracy=7.5000e-01, log_loss=5.0360e-01, Time computing validation-score=0.123s, Total time spent=0.18min. Found improved model=False, Improved top-3 models=True
[Iter 31/35, Epoch 4] train loss=4.90e-01, gnorm=3.25e+00, lr=6.45e-06, #samples processed=128, #sample per second=268.40. ETA=0.02min
[Iter 32/35, Epoch 4] train loss=4.00e-01, gnorm=2.28e+00, lr=4.84e-06, #samples processed=128, #sample per second=539.08. ETA=0.02min
[Iter 32/35, Epoch 4] Validation f1=8.1301e-01, mcc=5.4294e-01, roc_auc=8.6202e-01, accuracy=7.7000e-01, log_loss=4.9113e-01, Time computing validation-score=0.125s, Total time spent=0.20min. Found improved model=True, Improved top-3 models=True
[Iter 33/35, Epoch 4] train loss=4.95e-01, gnorm=3.40e+00, lr=3.23e-06, #samples processed=128, #sample per second=213.18. ETA=0.01min
[Iter 34/35, Epoch 4] train loss=5.04e-01, gnorm=3.25e+00, lr=1.61e-06, #samples processed=128, #sample per second=558.43. ETA=0.01min
[Iter 34/35, Epoch 4] Validation f1=8.1967e-01, mcc=5.6214e-01, roc_auc=8.6242e-01, accuracy=7.8000e-01, log_loss=4.8516e-01, Time computing validation-score=0.122s, Total time spent=0.21min. Found improved model=True, Improved top-3 models=True
[Iter 35/35, Epoch 4] train loss=4.90e-01, gnorm=2.87e+00, lr=0.00e+00, #samples processed=128, #sample per second=216.49. ETA=0.00min
[Iter 35/35, Epoch 4] Validation f1=8.1967e-01, mcc=5.6214e-01, roc_auc=8.6242e-01, accuracy=7.8000e-01, log_loss=4.8516e-01, Time computing validation-score=0.126s, Total time spent=0.22min. Found improved model=True, Improved top-3 models=True
Training completed. Auto-saving to "ag_text_customize2/". For loading the model, you can use predictor = TextPredictor.load("ag_text_customize2/")
<autogluon.text.text_prediction.predictor.TextPredictor at 0x7f0c72aa9ac0>
Register Your Own Configuration¶
You can also register your custom hyperparameter settings as new presets
in ag_text_presets. Below, the electra_small_fuse_late_train5
preset uses ELECTRA-small as its backbone and trains for 5 epochs with a
weight-decay of 1E-2.
@ag_text_presets.register()
def electra_small_fuse_late_train5():
hyperparameters = ag_text_presets.create('electra_small_fuse_late')
hyperparameters['models']['MultimodalTextModel']['search_space']['optimization.num_train_epochs'] = 5
hyperparameters['models']['MultimodalTextModel']['search_space']['optimization.wd'] = 1E-2
return hyperparameters
predictor = TextPredictor(path='ag_text_customize3', eval_metric='acc', label='label', backend='mxnet')
predictor.fit(train_data, presets='electra_small_fuse_late_train5', time_limit=60, seed=123)
Problem Type="binary"
Column Types:
- "sentence": text
- "label": categorical
The GluonNLP V0 backend is used. We will use 8 cpus and 1 gpus to train each trial.
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_customize3/task0/training.log
Fitting and transforming the train data...
Done! Preprocessor saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_customize3/task0/preprocessor.pkl
Process dev set...
Done!
Max length for chunking text: 64, Stochastic chunk: Train-False/Test-False, Test #repeat: 1.
#Total Params/Fixed Params=13516290/0
Using gradient accumulation. Global batch size = 128
Local training results will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_customize3/task0/results_local.jsonl.
[Iter 1/35, Epoch 0] train loss=8.07e-01, gnorm=7.51e+00, lr=2.50e-05, #samples processed=128, #sample per second=330.80. ETA=0.22min
[Iter 2/35, Epoch 0] train loss=8.39e-01, gnorm=5.76e+00, lr=5.00e-05, #samples processed=128, #sample per second=527.09. ETA=0.17min
[Iter 2/35, Epoch 0] Validation f1=7.2131e-01, mcc=1.7703e-01, roc_auc=6.3606e-01, accuracy=5.7500e-01, log_loss=7.8491e-01, Time computing validation-score=0.125s, Total time spent=0.01min. Found improved model=True, Improved top-3 models=True
[Iter 3/35, Epoch 0] train loss=8.71e-01, gnorm=7.10e+00, lr=7.50e-05, #samples processed=128, #sample per second=266.90. ETA=0.20min
[Iter 4/35, Epoch 0] train loss=7.54e-01, gnorm=5.47e+00, lr=1.00e-04, #samples processed=128, #sample per second=538.09. ETA=0.17min
[Iter 4/35, Epoch 0] Validation f1=5.6684e-01, mcc=2.1997e-01, roc_auc=6.9182e-01, accuracy=5.9500e-01, log_loss=6.6308e-01, Time computing validation-score=0.125s, Total time spent=0.03min. Found improved model=True, Improved top-3 models=True
[Iter 5/35, Epoch 0] train loss=9.12e-01, gnorm=7.70e+00, lr=9.68e-05, #samples processed=128, #sample per second=235.84. ETA=0.19min
[Iter 6/35, Epoch 0] train loss=7.47e-01, gnorm=3.46e+00, lr=9.35e-05, #samples processed=128, #sample per second=537.80. ETA=0.17min
[Iter 6/35, Epoch 0] Validation f1=7.5093e-01, mcc=3.3733e-01, roc_auc=7.0980e-01, accuracy=6.6500e-01, log_loss=6.4224e-01, Time computing validation-score=0.123s, Total time spent=0.04min. Found improved model=True, Improved top-3 models=True
[Iter 7/35, Epoch 0] train loss=7.03e-01, gnorm=3.15e+00, lr=9.03e-05, #samples processed=128, #sample per second=229.46. ETA=0.18min
[Iter 8/35, Epoch 1] train loss=6.77e-01, gnorm=3.17e+00, lr=8.71e-05, #samples processed=128, #sample per second=538.68. ETA=0.16min
[Iter 8/35, Epoch 1] Validation f1=7.4436e-01, mcc=3.2026e-01, roc_auc=7.5434e-01, accuracy=6.6000e-01, log_loss=6.1053e-01, Time computing validation-score=0.122s, Total time spent=0.05min. Found improved model=False, Improved top-3 models=True
[Iter 9/35, Epoch 1] train loss=6.43e-01, gnorm=2.72e+00, lr=8.39e-05, #samples processed=128, #sample per second=273.92. ETA=0.16min
[Iter 10/35, Epoch 1] train loss=6.74e-01, gnorm=4.69e+00, lr=8.06e-05, #samples processed=128, #sample per second=559.90. ETA=0.15min
[Iter 10/35, Epoch 1] Validation f1=6.7021e-01, mcc=4.1417e-01, roc_auc=8.1121e-01, accuracy=6.9000e-01, log_loss=5.9695e-01, Time computing validation-score=0.121s, Total time spent=0.07min. Found improved model=True, Improved top-3 models=True
[Iter 11/35, Epoch 1] train loss=6.42e-01, gnorm=3.14e+00, lr=7.74e-05, #samples processed=128, #sample per second=211.18. ETA=0.15min
[Iter 12/35, Epoch 1] train loss=5.93e-01, gnorm=2.45e+00, lr=7.42e-05, #samples processed=128, #sample per second=562.19. ETA=0.14min
[Iter 12/35, Epoch 1] Validation f1=7.3050e-01, mcc=2.4330e-01, roc_auc=8.3899e-01, accuracy=6.2000e-01, log_loss=5.8745e-01, Time computing validation-score=0.121s, Total time spent=0.08min. Found improved model=False, Improved top-3 models=False
[Iter 13/35, Epoch 1] train loss=6.51e-01, gnorm=4.53e+00, lr=7.10e-05, #samples processed=128, #sample per second=360.27. ETA=0.14min
[Iter 14/35, Epoch 1] train loss=6.23e-01, gnorm=4.33e+00, lr=6.77e-05, #samples processed=128, #sample per second=543.56. ETA=0.13min
[Iter 14/35, Epoch 1] Validation f1=7.7056e-01, mcc=4.6155e-01, roc_auc=8.4040e-01, accuracy=7.3500e-01, log_loss=5.1682e-01, Time computing validation-score=0.126s, Total time spent=0.09min. Found improved model=True, Improved top-3 models=True
[Iter 15/35, Epoch 2] train loss=5.80e-01, gnorm=2.11e+00, lr=6.45e-05, #samples processed=128, #sample per second=214.87. ETA=0.13min
[Iter 16/35, Epoch 2] train loss=5.27e-01, gnorm=4.19e+00, lr=6.13e-05, #samples processed=128, #sample per second=520.14. ETA=0.12min
[Iter 16/35, Epoch 2] Validation f1=7.9048e-01, mcc=5.6282e-01, roc_auc=8.3960e-01, accuracy=7.8000e-01, log_loss=5.0414e-01, Time computing validation-score=0.126s, Total time spent=0.10min. Found improved model=True, Improved top-3 models=True
[Iter 17/35, Epoch 2] train loss=6.33e-01, gnorm=4.59e+00, lr=5.81e-05, #samples processed=128, #sample per second=217.08. ETA=0.11min
[Iter 18/35, Epoch 2] train loss=5.05e-01, gnorm=2.31e+00, lr=5.48e-05, #samples processed=128, #sample per second=532.95. ETA=0.11min
[Iter 18/35, Epoch 2] Validation f1=7.7099e-01, mcc=4.0947e-01, roc_auc=8.5697e-01, accuracy=7.0000e-01, log_loss=5.4655e-01, Time computing validation-score=0.123s, Total time spent=0.12min. Found improved model=False, Improved top-3 models=True
[Iter 19/35, Epoch 2] train loss=5.27e-01, gnorm=3.97e+00, lr=5.16e-05, #samples processed=128, #sample per second=286.15. ETA=0.10min
[Iter 20/35, Epoch 2] train loss=5.85e-01, gnorm=6.78e+00, lr=4.84e-05, #samples processed=128, #sample per second=564.62. ETA=0.09min
[Iter 20/35, Epoch 2] Validation f1=7.7692e-01, mcc=4.2939e-01, roc_auc=8.6929e-01, accuracy=7.1000e-01, log_loss=5.3256e-01, Time computing validation-score=0.122s, Total time spent=0.13min. Found improved model=False, Improved top-3 models=True
[Iter 21/35, Epoch 2] train loss=4.85e-01, gnorm=4.01e+00, lr=4.52e-05, #samples processed=128, #sample per second=255.50. ETA=0.09min
[Iter 22/35, Epoch 3] train loss=5.54e-01, gnorm=2.76e+00, lr=4.19e-05, #samples processed=128, #sample per second=547.60. ETA=0.08min
[Iter 22/35, Epoch 3] Validation f1=8.3036e-01, mcc=6.1511e-01, roc_auc=8.7737e-01, accuracy=8.1000e-01, log_loss=4.3892e-01, Time computing validation-score=0.123s, Total time spent=0.14min. Found improved model=True, Improved top-3 models=True
[Iter 23/35, Epoch 3] train loss=4.51e-01, gnorm=4.47e+00, lr=3.87e-05, #samples processed=128, #sample per second=221.33. ETA=0.08min
[Iter 24/35, Epoch 3] train loss=4.62e-01, gnorm=4.61e+00, lr=3.55e-05, #samples processed=128, #sample per second=556.52. ETA=0.07min
[Iter 24/35, Epoch 3] Validation f1=8.3186e-01, mcc=6.1497e-01, roc_auc=8.8253e-01, accuracy=8.1000e-01, log_loss=4.2850e-01, Time computing validation-score=0.121s, Total time spent=0.15min. Found improved model=True, Improved top-3 models=True
[Iter 25/35, Epoch 3] train loss=3.74e-01, gnorm=2.92e+00, lr=3.23e-05, #samples processed=128, #sample per second=227.69. ETA=0.06min
[Iter 26/35, Epoch 3] train loss=4.14e-01, gnorm=2.72e+00, lr=2.90e-05, #samples processed=128, #sample per second=567.51. ETA=0.06min
[Iter 26/35, Epoch 3] Validation f1=8.1633e-01, mcc=5.5254e-01, roc_auc=8.8313e-01, accuracy=7.7500e-01, log_loss=4.7074e-01, Time computing validation-score=0.123s, Total time spent=0.16min. Found improved model=False, Improved top-3 models=False
[Iter 27/35, Epoch 3] train loss=3.75e-01, gnorm=2.73e+00, lr=2.58e-05, #samples processed=128, #sample per second=362.53. ETA=0.05min
[Iter 28/35, Epoch 3] train loss=3.57e-01, gnorm=4.08e+00, lr=2.26e-05, #samples processed=128, #sample per second=554.27. ETA=0.04min
[Iter 28/35, Epoch 3] Validation f1=7.8906e-01, mcc=4.6861e-01, roc_auc=8.8535e-01, accuracy=7.3000e-01, log_loss=5.1700e-01, Time computing validation-score=0.123s, Total time spent=0.17min. Found improved model=False, Improved top-3 models=False
[Iter 29/35, Epoch 4] train loss=4.50e-01, gnorm=6.86e+00, lr=1.94e-05, #samples processed=128, #sample per second=356.41. ETA=0.04min
[Iter 30/35, Epoch 4] train loss=4.46e-01, gnorm=9.60e+00, lr=1.61e-05, #samples processed=128, #sample per second=546.04. ETA=0.03min
[Iter 30/35, Epoch 4] Validation f1=8.1452e-01, mcc=5.4544e-01, roc_auc=8.8869e-01, accuracy=7.7000e-01, log_loss=4.7372e-01, Time computing validation-score=0.123s, Total time spent=0.18min. Found improved model=False, Improved top-3 models=False
[Iter 31/35, Epoch 4] train loss=3.92e-01, gnorm=4.70e+00, lr=1.29e-05, #samples processed=128, #sample per second=356.30. ETA=0.02min
[Iter 32/35, Epoch 4] train loss=3.66e-01, gnorm=3.09e+00, lr=9.68e-06, #samples processed=128, #sample per second=537.86. ETA=0.02min
[Iter 32/35, Epoch 4] Validation f1=8.2700e-01, mcc=5.8766e-01, roc_auc=8.9505e-01, accuracy=7.9500e-01, log_loss=4.2816e-01, Time computing validation-score=0.122s, Total time spent=0.20min. Found improved model=False, Improved top-3 models=True
[Iter 33/35, Epoch 4] train loss=3.99e-01, gnorm=4.60e+00, lr=6.45e-06, #samples processed=128, #sample per second=272.18. ETA=0.01min
[Iter 34/35, Epoch 4] train loss=3.90e-01, gnorm=2.93e+00, lr=3.23e-06, #samples processed=128, #sample per second=560.72. ETA=0.01min
[Iter 34/35, Epoch 4] Validation f1=8.3983e-01, mcc=6.2603e-01, roc_auc=8.9667e-01, accuracy=8.1500e-01, log_loss=4.1213e-01, Time computing validation-score=0.122s, Total time spent=0.21min. Found improved model=True, Improved top-3 models=True
[Iter 35/35, Epoch 4] train loss=3.76e-01, gnorm=3.44e+00, lr=0.00e+00, #samples processed=128, #sample per second=210.31. ETA=0.00min
[Iter 35/35, Epoch 4] Validation f1=8.3983e-01, mcc=6.2603e-01, roc_auc=8.9667e-01, accuracy=8.1500e-01, log_loss=4.1213e-01, Time computing validation-score=0.126s, Total time spent=0.22min. Found improved model=True, Improved top-3 models=True
Training completed. Auto-saving to "ag_text_customize3/". For loading the model, you can use predictor = TextPredictor.load("ag_text_customize3/")
<autogluon.text.text_prediction.predictor.TextPredictor at 0x7f0b126f68b0>
HPO over a Customized Search Space via Bayesian Optimization¶
To control which hyperparameter values are considered during fit(),
we specify the hyperparameters argument. Rather than specifying a
particular fixed value for a hyperparameter, we can specify a space of
values to search over via ag.core.space. We can also specify which
HPO method to use for the search via search_strategy. By default, we
will use Bayesian
Optimization as the searcher.
In this example, we search for good values of the following
hyperparameters:
warmup
number of hidden units in the final MLP layer that maps aggregated features to output prediction
learning rate
weight decay
def electra_small_basic_demo_hpo():
hparams = ag_text_presets.create('electra_small_fuse_late')
search_space = hparams['models']['MultimodalTextModel']['search_space']
search_space['optimization.per_device_batch_size'] = 8
search_space['model.network.agg_net.mid_units'] = ag.core.space.Int(32, 128)
search_space['optimization.warmup_portion'] = ag.core.space.Categorical(0.1, 0.2)
search_space['optimization.lr'] = ag.core.space.Real(1E-5, 2E-4)
search_space['optimization.wd'] = ag.core.space.Categorical(1E-4, 1E-3, 1E-2)
search_space['optimization.num_train_epochs'] = 5
return hparams
We can now call fit() with hyperparameter-tuning over our custom
search space. Below num_trials controls the maximal number of
different hyperparameter configurations for which AutoGluon will train
models (4 models are trained under different hyperparameter
configurations in this case). To achieve good performance in your
applications, you should use larger values of num_trials, which may
identify superior hyperparameter values but will require longer
runtimes.
predictor_sst_rs = TextPredictor(path='ag_text_sst_random_search', label='label', eval_metric='acc', backend='mxnet')
predictor_sst_rs.set_verbosity(0)
predictor_sst_rs.fit(train_data,
hyperparameters=electra_small_basic_demo_hpo(),
time_limit=60 * 2,
num_trials=4,
seed=123)
The GluonNLP V0 backend is used. We will use 8 cpus and 1 gpus to train each trial.
0%| | 0/4 [00:00<?, ?it/s]
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_sst_random_search/task0/training.log
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_sst_random_search/task1/training.log
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_sst_random_search/task2/training.log
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_text_sst_random_search/task3/training.log
<autogluon.text.text_prediction.predictor.TextPredictor at 0x7f0b12278f70>
We can again evaluate our model’s performance on separate test data.
test_score = predictor_sst_rs.evaluate(test_data, metrics=['acc', 'f1'])
print('Best Config = {}'.format(predictor_sst_rs.results['best_config']))
print('Total Time = {}s'.format(predictor_sst_rs.results['total_time']))
print('Accuracy = {:.2f}%'.format(test_score['acc'] * 100))
print('F1 = {:.2f}%'.format(test_score['f1'] * 100))
Best Config = {'model.backbone.name': 'google_electra_small', 'optimization.batch_size': 128, 'optimization.per_device_batch_size': 8, 'optimization.num_train_epochs': 5, 'optimization.lr': 0.00019309592449519557, 'optimization.wd': 0.001, 'optimization.layerwise_lr_decay': 0.8, 'model.use_avg_nbest': True, 'optimization.nbest': 3, 'model.network.agg_net.agg_type': 'concat', 'model.network.aggregate_categorical': True, 'preprocessing.categorical.convert_to_text': False, 'preprocessing.numerical.convert_to_text': False, 'model.network.agg_net.mid_units': 120, 'optimization.warmup_portion': 0.1}
Total Time = 54.745769739151s
Accuracy = 77.98%
F1 = 76.64%