Text Prediction - Quick Start¶
Here we introduce the TextPrediction task, which helps you
automatically train and deploy models for various Natural Language
Processing (NLP) problems. This tutorial presents two examples to
demonstrate how TextPrediction can be used for different NLP tasks
including:
The general usage is similar to AutoGluon’s TabularPredictor. We
treat NLP datasets as tables where certain columns contain text fields
and a special column contains the labels to predict. Here, the labels
can be discrete categories (classification) or numerical values
(regression). TextPrediction fits neural networks to your data via
transfer learning from pretrained NLP models like:
BERT,
ALBERT, and
ELECTRA.
TextPrediction also trains multiple models with different
hyperparameters and returns the best model, a process called
Hyperparameter Optimization (HPO).
import numpy as np
import warnings
warnings.filterwarnings('ignore')
np.random.seed(123)
Sentiment Analysis¶
First, we consider the Stanford Sentiment Treebank (SST) dataset, which consists of movie reviews and their associated sentiment. Given a new movie review, the goal is to predict the sentiment reflected in the text (in this case a binary classification problem, where reviews are labeled as 1 if they convey a positive opinion and labeled as 0 otherwise). Let’s first load the data and view some examples, noting the labels are stored in a column called label.
from autogluon.core.utils.loaders.load_pd import load
train_data = load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/train.parquet')
dev_data = load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/dev.parquet')
rand_idx = np.random.permutation(np.arange(len(train_data)))[:1000]
train_data = train_data.iloc[rand_idx]
train_data.head(10)
| sentence | label | |
|---|---|---|
| 2434 | goes by quickly | 1 |
| 27796 | reading lines from a teleprompter | 0 |
| 249 | degraded , handheld blair witch video-cam foot... | 0 |
| 12115 | reminds us how realistically nuanced a robert ... | 1 |
| 50834 | indulges in the worst elements of all of them . | 0 |
| 43622 | are nowhere near as vivid as the 19th-century ... | 0 |
| 3955 | throughout a film that is both gripping and co... | 1 |
| 51011 | to see over and over again | 1 |
| 31232 | that fails to match the freshness of the actre... | 0 |
| 32153 | this is an undeniably intriguing film from an ... | 1 |
Above the data happen to be stored in a
Parquet table
format, but you can also directly load() data from a
CSV file
instead. While here we load files from AWS S3 cloud
storage,
these could instead be local files on your machine. After loading,
train_data is simply a Pandas
DataFrame,
where each row represents a different training example (for machine
learning to be appropriate, the rows should be independent and
identically distributed).
To ensure this tutorial runs quickly, we simply call fit() with a
subset of 2000 training examples and limit its runtime to approximately
1 minute. To achieve reasonable performance in your applications, you
should set much longer time_limits (eg. 1 hour), or do not specify
time_limits at all.
from autogluon.text import TextPrediction as task
predictor = task.fit(train_data, label='label',
time_limits=60,
ngpus_per_trial=1,
seed=123,
output_directory='./ag_sst')
2021-02-23 19:25:00,603 - autogluon.text.text_prediction.text_prediction - INFO - All Logs will be saved to ./ag_sst/ag_text_prediction.log
INFO:autogluon.text.text_prediction.text_prediction:All Logs will be saved to ./ag_sst/ag_text_prediction.log
2021-02-23 19:25:00,614 - autogluon.text.text_prediction.text_prediction - INFO - Train Dataset:
INFO:autogluon.text.text_prediction.text_prediction:Train Dataset:
2021-02-23 19:25:00,615 - autogluon.text.text_prediction.text_prediction - INFO - Columns:
- Text(
name="sentence"
#total/missing=800/0
length, min/avg/max=4/53.7475/259
)
- Categorical(
name="label"
#total/missing=800/0
num_class (total/non_special)=2/2
categories=[0, 1]
freq=[366, 434]
)
INFO:autogluon.text.text_prediction.text_prediction:Columns:
- Text(
name="sentence"
#total/missing=800/0
length, min/avg/max=4/53.7475/259
)
- Categorical(
name="label"
#total/missing=800/0
num_class (total/non_special)=2/2
categories=[0, 1]
freq=[366, 434]
)
2021-02-23 19:25:00,615 - autogluon.text.text_prediction.text_prediction - INFO - Tuning Dataset:
INFO:autogluon.text.text_prediction.text_prediction:Tuning Dataset:
2021-02-23 19:25:00,616 - autogluon.text.text_prediction.text_prediction - INFO - Columns:
- Text(
name="sentence"
#total/missing=200/0
length, min/avg/max=4/50.505/245
)
- Categorical(
name="label"
#total/missing=200/0
num_class (total/non_special)=2/2
categories=[0, 1]
freq=[85, 115]
)
INFO:autogluon.text.text_prediction.text_prediction:Columns:
- Text(
name="sentence"
#total/missing=200/0
length, min/avg/max=4/50.505/245
)
- Categorical(
name="label"
#total/missing=200/0
num_class (total/non_special)=2/2
categories=[0, 1]
freq=[85, 115]
)
WARNING:autogluon.core.utils.multiprocessing_utils:WARNING: changing multiprocessing start method to forkserver
2021-02-23 19:25:00,623 - autogluon.text.text_prediction.text_prediction - INFO - All Logs will be saved to ./ag_sst/main.log
INFO:autogluon.text.text_prediction.text_prediction:All Logs will be saved to ./ag_sst/main.log
0%| | 0/3 [00:00<?, ?it/s]
(task:0) 2021-02-23 19:25:03,606 - root - INFO - All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task0/training.log
2021-02-23 19:25:03,606 - root - INFO - learning:
early_stopping_patience: 10
log_metrics: auto
stop_metric: auto
valid_ratio: 0.15
misc:
exp_dir: /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task0
seed: 123
model:
backbone:
name: google_electra_small
network:
agg_net:
activation: tanh
agg_type: concat
data_dropout: False
dropout: 0.1
feature_proj_num_layers: -1
initializer:
bias: ['zeros']
weight: ['xavier', 'uniform', 'avg', 3.0]
mid_units: 256
norm_eps: 1e-05
normalization: layer_norm
out_proj_num_layers: 0
categorical_net:
activation: leaky
data_dropout: False
dropout: 0.1
emb_units: 32
initializer:
bias: ['zeros']
embed: ['xavier', 'gaussian', 'in', 1.0]
weight: ['xavier', 'uniform', 'avg', 3.0]
mid_units: 64
norm_eps: 1e-05
normalization: layer_norm
num_layers: 1
feature_units: -1
initializer:
bias: ['zeros']
weight: ['truncnorm', 0, 0.02]
numerical_net:
activation: leaky
data_dropout: False
dropout: 0.1
initializer:
bias: ['zeros']
weight: ['xavier', 'uniform', 'avg', 3.0]
input_centering: False
mid_units: 128
norm_eps: 1e-05
normalization: layer_norm
num_layers: 1
text_net:
pool_type: cls
use_segment_id: True
preprocess:
max_length: 128
merge_text: True
optimization:
batch_size: 32
begin_lr: 0.0
final_lr: 0.0
layerwise_lr_decay: 0.8
log_frequency: 0.1
lr: 5e-05
lr_scheduler: triangular
max_grad_norm: 1.0
model_average: 5
num_train_epochs: 4
optimizer: adamw
optimizer_params: [('beta1', 0.9), ('beta2', 0.999), ('epsilon', 1e-06), ('correct_bias', False)]
per_device_batch_size: 16
val_batch_size_mult: 2
valid_frequency: 0.1
warmup_portion: 0.1
wd: 0.01
version: 1
2021-02-23 19:25:03,910 - root - INFO - Process training set...
2021-02-23 19:25:04,093 - root - INFO - Done!
2021-02-23 19:25:04,093 - root - INFO - Process dev set...
2021-02-23 19:25:04,136 - root - INFO - Done!
2021-02-23 19:25:09,393 - root - INFO - #Total Params/Fixed Params=13483522/0
2021-02-23 19:25:09,408 - root - INFO - Using gradient accumulation. Global batch size = 32
2021-02-23 19:25:10,355 - root - INFO - [Iter 3/100, Epoch 0] train loss=5.8952e-01, gnorm=9.8782e+00, lr=1.5000e-05, #samples processed=144, #sample per second=159.23
2021-02-23 19:25:10,593 - root - INFO - [Iter 3/100, Epoch 0] valid f1=6.1423e-01, mcc=-1.2789e-01, roc_auc=4.3212e-01, accuracy=4.8500e-01, log_loss=7.9937e-01, time spent=0.148s, total_time=0.02min
2021-02-23 19:25:10,971 - root - INFO - [Iter 6/100, Epoch 0] train loss=5.3909e-01, gnorm=1.2266e+01, lr=3.0000e-05, #samples processed=144, #sample per second=233.91
2021-02-23 19:25:11,095 - root - INFO - [Iter 6/100, Epoch 0] valid f1=2.6207e-01, mcc=4.9571e-02, roc_auc=4.8010e-01, accuracy=4.6500e-01, log_loss=9.0086e-01, time spent=0.124s, total_time=0.03min
2021-02-23 19:25:11,382 - root - INFO - [Iter 9/100, Epoch 0] train loss=6.1580e-01, gnorm=1.2433e+01, lr=4.5000e-05, #samples processed=144, #sample per second=350.90
2021-02-23 19:25:11,669 - root - INFO - [Iter 9/100, Epoch 0] valid f1=4.0237e-01, mcc=6.7208e-02, roc_auc=5.5642e-01, accuracy=4.9500e-01, log_loss=7.9209e-01, time spent=0.116s, total_time=0.04min
2021-02-23 19:25:11,974 - root - INFO - [Iter 12/100, Epoch 0] train loss=5.0367e-01, gnorm=6.6898e+00, lr=4.8889e-05, #samples processed=144, #sample per second=243.08
2021-02-23 19:25:12,213 - root - INFO - [Iter 12/100, Epoch 0] valid f1=7.0423e-01, mcc=7.8952e-02, roc_auc=5.9918e-01, accuracy=5.8000e-01, log_loss=7.0353e-01, time spent=0.116s, total_time=0.05min
2021-02-23 19:25:12,508 - root - INFO - [Iter 15/100, Epoch 0] train loss=5.1291e-01, gnorm=4.4318e+00, lr=4.7222e-05, #samples processed=144, #sample per second=269.88
2021-02-23 19:25:12,798 - root - INFO - [Iter 15/100, Epoch 0] valid f1=7.0922e-01, mcc=1.0832e-01, roc_auc=6.1483e-01, accuracy=5.9000e-01, log_loss=6.8048e-01, time spent=0.119s, total_time=0.06min
2021-02-23 19:25:13,096 - root - INFO - [Iter 18/100, Epoch 0] train loss=4.6790e-01, gnorm=6.9617e+00, lr=4.5556e-05, #samples processed=144, #sample per second=245.14
2021-02-23 19:25:13,331 - root - INFO - [Iter 18/100, Epoch 0] valid f1=7.1324e-01, mcc=1.6557e-01, roc_auc=6.3448e-01, accuracy=6.1000e-01, log_loss=6.5667e-01, time spent=0.114s, total_time=0.06min
2021-02-23 19:25:13,659 - root - INFO - [Iter 21/100, Epoch 0] train loss=4.7886e-01, gnorm=6.5739e+00, lr=4.3889e-05, #samples processed=144, #sample per second=255.54
2021-02-23 19:25:13,908 - root - INFO - [Iter 21/100, Epoch 0] valid f1=6.6667e-01, mcc=2.2493e-01, roc_auc=6.4471e-01, accuracy=6.2000e-01, log_loss=6.5432e-01, time spent=0.115s, total_time=0.07min
2021-02-23 19:25:14,218 - root - INFO - [Iter 24/100, Epoch 0] train loss=4.6309e-01, gnorm=6.7642e+00, lr=4.2222e-05, #samples processed=144, #sample per second=257.93
2021-02-23 19:25:14,496 - root - INFO - [Iter 24/100, Epoch 0] valid f1=6.7544e-01, mcc=2.4533e-01, roc_auc=6.7478e-01, accuracy=6.3000e-01, log_loss=6.3939e-01, time spent=0.115s, total_time=0.08min
2021-02-23 19:25:14,777 - root - INFO - [Iter 27/100, Epoch 1] train loss=4.4571e-01, gnorm=4.4769e+00, lr=4.0556e-05, #samples processed=144, #sample per second=257.60
2021-02-23 19:25:15,022 - root - INFO - [Iter 27/100, Epoch 1] valid f1=7.4552e-01, mcc=2.5537e-01, roc_auc=7.0813e-01, accuracy=6.4500e-01, log_loss=6.2081e-01, time spent=0.118s, total_time=0.09min
2021-02-23 19:25:15,290 - root - INFO - [Iter 30/100, Epoch 1] train loss=4.2172e-01, gnorm=4.9218e+00, lr=3.8889e-05, #samples processed=144, #sample per second=280.71
2021-02-23 19:25:15,578 - root - INFO - [Iter 30/100, Epoch 1] valid f1=7.4717e-01, mcc=2.9782e-01, roc_auc=7.4118e-01, accuracy=6.6500e-01, log_loss=5.9731e-01, time spent=0.115s, total_time=0.10min
2021-02-23 19:25:15,837 - root - INFO - [Iter 33/100, Epoch 1] train loss=4.0676e-01, gnorm=5.7487e+00, lr=3.7222e-05, #samples processed=144, #sample per second=263.39
2021-02-23 19:25:16,074 - root - INFO - [Iter 33/100, Epoch 1] valid f1=7.0370e-01, mcc=3.6262e-01, roc_auc=7.5652e-01, accuracy=6.8000e-01, log_loss=5.9500e-01, time spent=0.115s, total_time=0.11min
2021-02-23 19:25:16,359 - root - INFO - [Iter 36/100, Epoch 1] train loss=4.3250e-01, gnorm=5.8966e+00, lr=3.5556e-05, #samples processed=144, #sample per second=276.03
2021-02-23 19:25:16,616 - root - INFO - [Iter 36/100, Epoch 1] valid f1=7.1963e-01, mcc=4.0611e-01, roc_auc=7.7023e-01, accuracy=7.0000e-01, log_loss=5.8079e-01, time spent=0.117s, total_time=0.12min
2021-02-23 19:25:16,894 - root - INFO - [Iter 39/100, Epoch 1] train loss=4.6675e-01, gnorm=6.3319e+00, lr=3.3889e-05, #samples processed=144, #sample per second=269.08
2021-02-23 19:25:17,010 - root - INFO - [Iter 39/100, Epoch 1] valid f1=7.6448e-01, mcc=3.6493e-01, roc_auc=7.8107e-01, accuracy=6.9500e-01, log_loss=5.6165e-01, time spent=0.115s, total_time=0.13min
2021-02-23 19:25:17,301 - root - INFO - [Iter 42/100, Epoch 1] train loss=3.9259e-01, gnorm=9.2708e+00, lr=3.2222e-05, #samples processed=144, #sample per second=354.21
2021-02-23 19:25:17,553 - root - INFO - [Iter 42/100, Epoch 1] valid f1=7.7165e-01, mcc=3.9708e-01, roc_auc=7.8721e-01, accuracy=7.1000e-01, log_loss=5.5366e-01, time spent=0.114s, total_time=0.14min
2021-02-23 19:25:17,822 - root - INFO - [Iter 45/100, Epoch 1] train loss=3.6284e-01, gnorm=7.4090e+00, lr=3.0556e-05, #samples processed=144, #sample per second=276.15
2021-02-23 19:25:18,064 - root - INFO - [Iter 45/100, Epoch 1] valid f1=7.7165e-01, mcc=3.9708e-01, roc_auc=7.9386e-01, accuracy=7.1000e-01, log_loss=5.4907e-01, time spent=0.116s, total_time=0.14min
2021-02-23 19:25:18,329 - root - INFO - [Iter 48/100, Epoch 1] train loss=3.6125e-01, gnorm=7.3234e+00, lr=2.8889e-05, #samples processed=144, #sample per second=284.37
2021-02-23 19:25:18,599 - root - INFO - [Iter 48/100, Epoch 1] valid f1=7.7108e-01, mcc=4.0762e-01, roc_auc=8.0194e-01, accuracy=7.1500e-01, log_loss=5.3900e-01, time spent=0.116s, total_time=0.15min
2021-02-23 19:25:18,886 - root - INFO - [Iter 51/100, Epoch 2] train loss=4.1700e-01, gnorm=6.3041e+00, lr=2.7222e-05, #samples processed=144, #sample per second=258.70
2021-02-23 19:25:19,123 - root - INFO - [Iter 51/100, Epoch 2] valid f1=7.4312e-01, mcc=4.4068e-01, roc_auc=8.0931e-01, accuracy=7.2000e-01, log_loss=5.3795e-01, time spent=0.115s, total_time=0.16min
2021-02-23 19:25:19,375 - root - INFO - [Iter 54/100, Epoch 2] train loss=3.3013e-01, gnorm=6.0729e+00, lr=2.5556e-05, #samples processed=144, #sample per second=294.83
2021-02-23 19:25:19,658 - root - INFO - [Iter 54/100, Epoch 2] valid f1=7.5000e-01, mcc=4.3314e-01, roc_auc=8.1176e-01, accuracy=7.2000e-01, log_loss=5.2912e-01, time spent=0.116s, total_time=0.17min
2021-02-23 19:25:19,925 - root - INFO - [Iter 57/100, Epoch 2] train loss=2.8255e-01, gnorm=5.2432e+00, lr=2.3889e-05, #samples processed=144, #sample per second=261.89
2021-02-23 19:25:20,040 - root - INFO - [Iter 57/100, Epoch 2] valid f1=7.4667e-01, mcc=4.2186e-01, roc_auc=8.1524e-01, accuracy=7.1500e-01, log_loss=5.2117e-01, time spent=0.115s, total_time=0.18min
2021-02-23 19:25:20,301 - root - INFO - [Iter 60/100, Epoch 2] train loss=3.9490e-01, gnorm=6.4083e+00, lr=2.2222e-05, #samples processed=144, #sample per second=382.50
2021-02-23 19:25:20,417 - root - INFO - [Iter 60/100, Epoch 2] valid f1=7.4459e-01, mcc=3.9551e-01, roc_auc=8.2036e-01, accuracy=7.0500e-01, log_loss=5.1151e-01, time spent=0.115s, total_time=0.18min
2021-02-23 19:25:20,687 - root - INFO - [Iter 63/100, Epoch 2] train loss=3.6566e-01, gnorm=9.3752e+00, lr=2.0556e-05, #samples processed=144, #sample per second=373.54
2021-02-23 19:25:20,802 - root - INFO - [Iter 63/100, Epoch 2] valid f1=7.4678e-01, mcc=3.9382e-01, roc_auc=8.2568e-01, accuracy=7.0500e-01, log_loss=5.0415e-01, time spent=0.114s, total_time=0.19min
2021-02-23 19:25:21,054 - root - INFO - [Iter 66/100, Epoch 2] train loss=2.9910e-01, gnorm=6.5386e+00, lr=1.8889e-05, #samples processed=144, #sample per second=392.93
2021-02-23 19:25:21,356 - root - INFO - [Iter 66/100, Epoch 2] valid f1=7.5676e-01, mcc=4.5576e-01, roc_auc=8.3100e-01, accuracy=7.3000e-01, log_loss=5.0513e-01, time spent=0.115s, total_time=0.20min
2021-02-23 19:25:21,614 - root - INFO - [Iter 69/100, Epoch 2] train loss=3.5372e-01, gnorm=7.9419e+00, lr=1.7222e-05, #samples processed=144, #sample per second=257.16
2021-02-23 19:25:21,854 - root - INFO - [Iter 69/100, Epoch 2] valid f1=7.5962e-01, mcc=5.1761e-01, roc_auc=8.3468e-01, accuracy=7.5000e-01, log_loss=5.1635e-01, time spent=0.115s, total_time=0.21min
2021-02-23 19:25:22,116 - root - INFO - [Iter 72/100, Epoch 2] train loss=2.7875e-01, gnorm=8.6700e+00, lr=1.5556e-05, #samples processed=144, #sample per second=286.87
2021-02-23 19:25:22,232 - root - INFO - [Iter 72/100, Epoch 2] valid f1=7.6147e-01, mcc=4.8116e-01, roc_auc=8.3754e-01, accuracy=7.4000e-01, log_loss=4.9754e-01, time spent=0.115s, total_time=0.21min
2021-02-23 19:25:22,503 - root - INFO - [Iter 75/100, Epoch 2] train loss=3.5572e-01, gnorm=6.8519e+00, lr=1.3889e-05, #samples processed=144, #sample per second=371.79
2021-02-23 19:25:22,621 - root - INFO - [Iter 75/100, Epoch 2] valid f1=7.4678e-01, mcc=3.9382e-01, roc_auc=8.4010e-01, accuracy=7.0500e-01, log_loss=4.8494e-01, time spent=0.117s, total_time=0.22min
2021-02-23 19:25:22,887 - root - INFO - [Iter 78/100, Epoch 3] train loss=2.7614e-01, gnorm=6.3383e+00, lr=1.2222e-05, #samples processed=144, #sample per second=375.60
2021-02-23 19:25:23,005 - root - INFO - [Iter 78/100, Epoch 3] valid f1=7.7178e-01, mcc=4.3051e-01, roc_auc=8.4113e-01, accuracy=7.2500e-01, log_loss=4.8512e-01, time spent=0.118s, total_time=0.23min
2021-02-23 19:25:23,256 - root - INFO - [Iter 81/100, Epoch 3] train loss=3.1080e-01, gnorm=8.4711e+00, lr=1.0556e-05, #samples processed=144, #sample per second=390.80
2021-02-23 19:25:23,370 - root - INFO - [Iter 81/100, Epoch 3] valid f1=7.6151e-01, mcc=4.1051e-01, roc_auc=8.4184e-01, accuracy=7.1500e-01, log_loss=4.8329e-01, time spent=0.115s, total_time=0.23min
2021-02-23 19:25:23,623 - root - INFO - [Iter 84/100, Epoch 3] train loss=2.5932e-01, gnorm=7.3023e+00, lr=8.8889e-06, #samples processed=144, #sample per second=392.32
2021-02-23 19:25:23,738 - root - INFO - [Iter 84/100, Epoch 3] valid f1=7.5745e-01, mcc=4.1292e-01, roc_auc=8.4440e-01, accuracy=7.1500e-01, log_loss=4.8062e-01, time spent=0.115s, total_time=0.24min
2021-02-23 19:25:24,008 - root - INFO - [Iter 87/100, Epoch 3] train loss=3.1257e-01, gnorm=6.9703e+00, lr=7.2222e-06, #samples processed=144, #sample per second=374.00
2021-02-23 19:25:24,124 - root - INFO - [Iter 87/100, Epoch 3] valid f1=7.5862e-01, mcc=4.2544e-01, roc_auc=8.4573e-01, accuracy=7.2000e-01, log_loss=4.7916e-01, time spent=0.115s, total_time=0.24min
2021-02-23 19:25:24,361 - root - INFO - [Iter 90/100, Epoch 3] train loss=3.1674e-01, gnorm=6.6589e+00, lr=5.5556e-06, #samples processed=144, #sample per second=407.79
2021-02-23 19:25:24,477 - root - INFO - [Iter 90/100, Epoch 3] valid f1=7.6190e-01, mcc=4.3650e-01, roc_auc=8.4685e-01, accuracy=7.2500e-01, log_loss=4.7787e-01, time spent=0.115s, total_time=0.25min
2021-02-23 19:25:24,727 - root - INFO - [Iter 93/100, Epoch 3] train loss=2.6397e-01, gnorm=1.0188e+01, lr=3.8889e-06, #samples processed=144, #sample per second=394.12
2021-02-23 19:25:24,845 - root - INFO - [Iter 93/100, Epoch 3] valid f1=7.4783e-01, mcc=4.0665e-01, roc_auc=8.4747e-01, accuracy=7.1000e-01, log_loss=4.7692e-01, time spent=0.118s, total_time=0.26min
2021-02-23 19:25:25,119 - root - INFO - [Iter 96/100, Epoch 3] train loss=2.7495e-01, gnorm=8.1681e+00, lr=2.2222e-06, #samples processed=144, #sample per second=367.09
2021-02-23 19:25:25,234 - root - INFO - [Iter 96/100, Epoch 3] valid f1=7.4459e-01, mcc=3.9551e-01, roc_auc=8.4757e-01, accuracy=7.0500e-01, log_loss=4.7623e-01, time spent=0.114s, total_time=0.26min
2021-02-23 19:25:25,491 - root - INFO - [Iter 99/100, Epoch 3] train loss=2.9856e-01, gnorm=8.6445e+00, lr=5.5556e-07, #samples processed=144, #sample per second=387.04
2021-02-23 19:25:25,606 - root - INFO - [Iter 99/100, Epoch 3] valid f1=7.4678e-01, mcc=3.9382e-01, roc_auc=8.4757e-01, accuracy=7.0500e-01, log_loss=4.7598e-01, time spent=0.114s, total_time=0.27min
2021-02-23 19:25:25,609 - root - INFO - Early stopping patience reached!
(task:1) 2021-02-23 19:25:28,161 - root - INFO - All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task1/training.log
2021-02-23 19:25:28,161 - root - INFO - learning:
early_stopping_patience: 10
log_metrics: auto
stop_metric: auto
valid_ratio: 0.15
misc:
exp_dir: /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task1
seed: 123
model:
backbone:
name: google_electra_small
network:
agg_net:
activation: tanh
agg_type: concat
data_dropout: False
dropout: 0.1
feature_proj_num_layers: -1
initializer:
bias: ['zeros']
weight: ['xavier', 'uniform', 'avg', 3.0]
mid_units: 256
norm_eps: 1e-05
normalization: layer_norm
out_proj_num_layers: 0
categorical_net:
activation: leaky
data_dropout: False
dropout: 0.1
emb_units: 32
initializer:
bias: ['zeros']
embed: ['xavier', 'gaussian', 'in', 1.0]
weight: ['xavier', 'uniform', 'avg', 3.0]
mid_units: 64
norm_eps: 1e-05
normalization: layer_norm
num_layers: 1
feature_units: -1
initializer:
bias: ['zeros']
weight: ['truncnorm', 0, 0.02]
numerical_net:
activation: leaky
data_dropout: False
dropout: 0.1
initializer:
bias: ['zeros']
weight: ['xavier', 'uniform', 'avg', 3.0]
input_centering: False
mid_units: 128
norm_eps: 1e-05
normalization: layer_norm
num_layers: 1
text_net:
pool_type: cls
use_segment_id: True
preprocess:
max_length: 128
merge_text: True
optimization:
batch_size: 32
begin_lr: 0.0
final_lr: 0.0
layerwise_lr_decay: 0.8
log_frequency: 0.1
lr: 4.1195288877731504e-05
lr_scheduler: triangular
max_grad_norm: 1.0
model_average: 5
num_train_epochs: 4
optimizer: adamw
optimizer_params: [('beta1', 0.9), ('beta2', 0.999), ('epsilon', 1e-06), ('correct_bias', False)]
per_device_batch_size: 16
val_batch_size_mult: 2
valid_frequency: 0.1
warmup_portion: 0.1
wd: 0.01
version: 1
2021-02-23 19:25:28,316 - root - INFO - Process training set...
2021-02-23 19:25:28,497 - root - INFO - Done!
2021-02-23 19:25:28,497 - root - INFO - Process dev set...
2021-02-23 19:25:28,547 - root - INFO - Done!
2021-02-23 19:25:33,886 - root - INFO - #Total Params/Fixed Params=13483522/0
2021-02-23 19:25:33,901 - root - INFO - Using gradient accumulation. Global batch size = 32
2021-02-23 19:25:34,839 - root - INFO - [Iter 3/100, Epoch 0] train loss=5.9082e-01, gnorm=1.0008e+01, lr=1.2359e-05, #samples processed=144, #sample per second=160.89
2021-02-23 19:25:35,089 - root - INFO - [Iter 3/100, Epoch 0] valid f1=6.4469e-01, mcc=-7.0772e-02, roc_auc=4.3038e-01, accuracy=5.1500e-01, log_loss=8.0502e-01, time spent=0.170s, total_time=0.02min
2021-02-23 19:25:35,461 - root - INFO - [Iter 6/100, Epoch 0] train loss=5.4035e-01, gnorm=1.0983e+01, lr=2.4717e-05, #samples processed=144, #sample per second=231.60
2021-02-23 19:25:35,579 - root - INFO - [Iter 6/100, Epoch 0] valid f1=3.7349e-01, mcc=3.8869e-02, roc_auc=4.7059e-01, accuracy=4.8000e-01, log_loss=8.5606e-01, time spent=0.117s, total_time=0.03min
2021-02-23 19:25:35,868 - root - INFO - [Iter 9/100, Epoch 0] train loss=6.0367e-01, gnorm=1.2270e+01, lr=3.7076e-05, #samples processed=144, #sample per second=354.21
2021-02-23 19:25:35,984 - root - INFO - [Iter 9/100, Epoch 0] valid f1=3.5443e-01, mcc=8.0630e-02, roc_auc=5.3729e-01, accuracy=4.9000e-01, log_loss=8.1636e-01, time spent=0.116s, total_time=0.03min
2021-02-23 19:25:36,280 - root - INFO - [Iter 12/100, Epoch 0] train loss=5.1028e-01, gnorm=5.9195e+00, lr=4.0280e-05, #samples processed=144, #sample per second=349.48
2021-02-23 19:25:36,558 - root - INFO - [Iter 12/100, Epoch 0] valid f1=6.6932e-01, mcc=1.2576e-01, roc_auc=5.8588e-01, accuracy=5.8500e-01, log_loss=6.9569e-01, time spent=0.116s, total_time=0.04min
2021-02-23 19:25:36,865 - root - INFO - [Iter 15/100, Epoch 0] train loss=5.0118e-01, gnorm=4.7131e+00, lr=3.8907e-05, #samples processed=144, #sample per second=246.05
2021-02-23 19:25:37,109 - root - INFO - [Iter 15/100, Epoch 0] valid f1=7.1080e-01, mcc=9.0363e-02, roc_auc=6.0102e-01, accuracy=5.8500e-01, log_loss=6.9532e-01, time spent=0.118s, total_time=0.05min
2021-02-23 19:25:37,396 - root - INFO - [Iter 18/100, Epoch 0] train loss=4.7891e-01, gnorm=8.2039e+00, lr=3.7533e-05, #samples processed=144, #sample per second=271.31
2021-02-23 19:25:37,660 - root - INFO - [Iter 18/100, Epoch 0] valid f1=7.1864e-01, mcc=8.4287e-02, roc_auc=6.2107e-01, accuracy=5.8500e-01, log_loss=6.8352e-01, time spent=0.116s, total_time=0.06min
2021-02-23 19:25:37,990 - root - INFO - [Iter 21/100, Epoch 0] train loss=4.9256e-01, gnorm=5.6984e+00, lr=3.6160e-05, #samples processed=144, #sample per second=242.56
2021-02-23 19:25:38,231 - root - INFO - [Iter 21/100, Epoch 0] valid f1=6.9291e-01, mcc=1.7740e-01, roc_auc=6.3100e-01, accuracy=6.1000e-01, log_loss=6.5372e-01, time spent=0.117s, total_time=0.07min
2021-02-23 19:25:38,536 - root - INFO - [Iter 24/100, Epoch 0] train loss=4.6538e-01, gnorm=7.2384e+00, lr=3.4787e-05, #samples processed=144, #sample per second=264.04
2021-02-23 19:25:38,651 - root - INFO - [Iter 24/100, Epoch 0] valid f1=6.2673e-01, mcc=1.8918e-01, roc_auc=6.5708e-01, accuracy=5.9500e-01, log_loss=6.5432e-01, time spent=0.115s, total_time=0.08min
2021-02-23 19:25:38,926 - root - INFO - [Iter 27/100, Epoch 1] train loss=4.6364e-01, gnorm=6.0129e+00, lr=3.3414e-05, #samples processed=144, #sample per second=368.97
2021-02-23 19:25:39,175 - root - INFO - [Iter 27/100, Epoch 1] valid f1=7.3282e-01, mcc=2.6298e-01, roc_auc=6.8788e-01, accuracy=6.5000e-01, log_loss=6.2610e-01, time spent=0.116s, total_time=0.09min
2021-02-23 19:25:39,432 - root - INFO - [Iter 30/100, Epoch 1] train loss=4.1835e-01, gnorm=4.6893e+00, lr=3.2041e-05, #samples processed=144, #sample per second=284.84
2021-02-23 19:25:39,699 - root - INFO - [Iter 30/100, Epoch 1] valid f1=7.5093e-01, mcc=2.9923e-01, roc_auc=7.1519e-01, accuracy=6.6500e-01, log_loss=6.1173e-01, time spent=0.115s, total_time=0.10min
2021-02-23 19:25:39,967 - root - INFO - [Iter 33/100, Epoch 1] train loss=4.1839e-01, gnorm=5.5281e+00, lr=3.0668e-05, #samples processed=144, #sample per second=269.00
2021-02-23 19:25:40,237 - root - INFO - [Iter 33/100, Epoch 1] valid f1=7.4074e-01, mcc=3.4558e-01, roc_auc=7.3698e-01, accuracy=6.8500e-01, log_loss=5.9952e-01, time spent=0.122s, total_time=0.10min
2021-02-23 19:25:40,518 - root - INFO - [Iter 36/100, Epoch 1] train loss=4.3262e-01, gnorm=5.3979e+00, lr=2.9294e-05, #samples processed=144, #sample per second=261.42
2021-02-23 19:25:40,635 - root - INFO - [Iter 36/100, Epoch 1] valid f1=7.2174e-01, mcc=3.4527e-01, roc_auc=7.5018e-01, accuracy=6.8000e-01, log_loss=5.9112e-01, time spent=0.117s, total_time=0.11min
2021-02-23 19:25:40,913 - root - INFO - [Iter 39/100, Epoch 1] train loss=4.7685e-01, gnorm=6.9142e+00, lr=2.7921e-05, #samples processed=144, #sample per second=364.60
2021-02-23 19:25:41,188 - root - INFO - [Iter 39/100, Epoch 1] valid f1=7.5200e-01, mcc=3.5361e-01, roc_auc=7.5908e-01, accuracy=6.9000e-01, log_loss=5.7894e-01, time spent=0.119s, total_time=0.12min
2021-02-23 19:25:41,472 - root - INFO - [Iter 42/100, Epoch 1] train loss=3.9227e-01, gnorm=7.6583e+00, lr=2.6548e-05, #samples processed=144, #sample per second=257.83
2021-02-23 19:25:41,587 - root - INFO - [Iter 42/100, Epoch 1] valid f1=7.3333e-01, mcc=3.3689e-01, roc_auc=7.6481e-01, accuracy=6.8000e-01, log_loss=5.7272e-01, time spent=0.115s, total_time=0.13min
2021-02-23 19:25:41,852 - root - INFO - [Iter 45/100, Epoch 1] train loss=3.8390e-01, gnorm=7.8581e+00, lr=2.5175e-05, #samples processed=144, #sample per second=379.21
2021-02-23 19:25:42,105 - root - INFO - [Iter 45/100, Epoch 1] valid f1=7.6378e-01, mcc=3.7511e-01, roc_auc=7.7217e-01, accuracy=7.0000e-01, log_loss=5.6712e-01, time spent=0.116s, total_time=0.14min
2021-02-23 19:25:42,381 - root - INFO - [Iter 48/100, Epoch 1] train loss=3.8232e-01, gnorm=7.4005e+00, lr=2.3802e-05, #samples processed=144, #sample per second=272.17
2021-02-23 19:25:42,614 - root - INFO - [Iter 48/100, Epoch 1] valid f1=7.7344e-01, mcc=3.9755e-01, roc_auc=7.7821e-01, accuracy=7.1000e-01, log_loss=5.6293e-01, time spent=0.117s, total_time=0.14min
2021-02-23 19:25:42,904 - root - INFO - [Iter 51/100, Epoch 2] train loss=4.4792e-01, gnorm=5.7064e+00, lr=2.2429e-05, #samples processed=144, #sample per second=275.42
2021-02-23 19:25:43,022 - root - INFO - [Iter 51/100, Epoch 2] valid f1=7.3418e-01, mcc=3.4942e-01, roc_auc=7.8824e-01, accuracy=6.8500e-01, log_loss=5.5150e-01, time spent=0.117s, total_time=0.15min
2021-02-23 19:25:43,378 - root - INFO - [Iter 54/100, Epoch 2] train loss=3.3694e-01, gnorm=5.5086e+00, lr=2.1055e-05, #samples processed=144, #sample per second=303.91
2021-02-23 19:25:43,497 - root - INFO - [Iter 54/100, Epoch 2] valid f1=7.3276e-01, mcc=3.6385e-01, roc_auc=7.9110e-01, accuracy=6.9000e-01, log_loss=5.4583e-01, time spent=0.118s, total_time=0.16min
2021-02-23 19:25:43,748 - root - INFO - [Iter 57/100, Epoch 2] train loss=3.0911e-01, gnorm=5.9129e+00, lr=1.9682e-05, #samples processed=144, #sample per second=389.37
2021-02-23 19:25:43,863 - root - INFO - [Iter 57/100, Epoch 2] valid f1=7.3451e-01, mcc=3.9026e-01, roc_auc=7.9448e-01, accuracy=7.0000e-01, log_loss=5.4312e-01, time spent=0.115s, total_time=0.17min
2021-02-23 19:25:44,126 - root - INFO - [Iter 60/100, Epoch 2] train loss=4.0372e-01, gnorm=5.9834e+00, lr=1.8309e-05, #samples processed=144, #sample per second=381.56
2021-02-23 19:25:44,243 - root - INFO - [Iter 60/100, Epoch 2] valid f1=7.3684e-01, mcc=3.8815e-01, roc_auc=7.9877e-01, accuracy=7.0000e-01, log_loss=5.3672e-01, time spent=0.117s, total_time=0.17min
2021-02-23 19:25:44,522 - root - INFO - [Iter 63/100, Epoch 2] train loss=3.8651e-01, gnorm=8.3916e+00, lr=1.6936e-05, #samples processed=144, #sample per second=363.73
2021-02-23 19:25:44,640 - root - INFO - [Iter 63/100, Epoch 2] valid f1=7.3913e-01, mcc=3.8619e-01, roc_auc=8.0460e-01, accuracy=7.0000e-01, log_loss=5.3018e-01, time spent=0.117s, total_time=0.18min
2021-02-23 19:25:44,888 - root - INFO - [Iter 66/100, Epoch 2] train loss=3.2546e-01, gnorm=7.0137e+00, lr=1.5563e-05, #samples processed=144, #sample per second=393.67
2021-02-23 19:25:45,007 - root - INFO - [Iter 66/100, Epoch 2] valid f1=7.3778e-01, mcc=4.0153e-01, roc_auc=8.1125e-01, accuracy=7.0500e-01, log_loss=5.3143e-01, time spent=0.119s, total_time=0.18min
2021-02-23 19:25:45,264 - root - INFO - [Iter 69/100, Epoch 2] train loss=3.6854e-01, gnorm=6.8702e+00, lr=1.4189e-05, #samples processed=144, #sample per second=382.68
2021-02-23 19:25:45,548 - root - INFO - [Iter 69/100, Epoch 2] valid f1=7.3832e-01, mcc=4.4657e-01, roc_auc=8.1483e-01, accuracy=7.2000e-01, log_loss=5.3699e-01, time spent=0.115s, total_time=0.19min
2021-02-23 19:25:45,810 - root - INFO - [Iter 72/100, Epoch 2] train loss=2.9991e-01, gnorm=8.1640e+00, lr=1.2816e-05, #samples processed=144, #sample per second=264.16
2021-02-23 19:25:46,052 - root - INFO - [Iter 72/100, Epoch 2] valid f1=7.5676e-01, mcc=4.5576e-01, roc_auc=8.1739e-01, accuracy=7.3000e-01, log_loss=5.2387e-01, time spent=0.115s, total_time=0.20min
2021-02-23 19:25:46,320 - root - INFO - [Iter 75/100, Epoch 2] train loss=3.7902e-01, gnorm=6.5946e+00, lr=1.1443e-05, #samples processed=144, #sample per second=282.09
2021-02-23 19:25:46,438 - root - INFO - [Iter 75/100, Epoch 2] valid f1=7.4561e-01, mcc=4.0855e-01, roc_auc=8.2097e-01, accuracy=7.1000e-01, log_loss=5.1315e-01, time spent=0.117s, total_time=0.21min
2021-02-23 19:25:46,709 - root - INFO - [Iter 78/100, Epoch 3] train loss=3.0121e-01, gnorm=6.0926e+00, lr=1.0070e-05, #samples processed=144, #sample per second=370.37
2021-02-23 19:25:46,826 - root - INFO - [Iter 78/100, Epoch 3] valid f1=7.5105e-01, mcc=3.9089e-01, roc_auc=8.2240e-01, accuracy=7.0500e-01, log_loss=5.1090e-01, time spent=0.116s, total_time=0.21min
2021-02-23 19:25:47,079 - root - INFO - [Iter 81/100, Epoch 3] train loss=3.4079e-01, gnorm=7.8657e+00, lr=8.6968e-06, #samples processed=144, #sample per second=389.66
2021-02-23 19:25:47,199 - root - INFO - [Iter 81/100, Epoch 3] valid f1=7.5105e-01, mcc=3.9089e-01, roc_auc=8.2322e-01, accuracy=7.0500e-01, log_loss=5.0949e-01, time spent=0.120s, total_time=0.22min
2021-02-23 19:25:47,457 - root - INFO - [Iter 84/100, Epoch 3] train loss=2.8519e-01, gnorm=7.7474e+00, lr=7.3236e-06, #samples processed=144, #sample per second=381.10
2021-02-23 19:25:47,576 - root - INFO - [Iter 84/100, Epoch 3] valid f1=7.5105e-01, mcc=3.9089e-01, roc_auc=8.2332e-01, accuracy=7.0500e-01, log_loss=5.0756e-01, time spent=0.119s, total_time=0.23min
2021-02-23 19:25:47,838 - root - INFO - [Iter 87/100, Epoch 3] train loss=3.3696e-01, gnorm=6.8759e+00, lr=5.9504e-06, #samples processed=144, #sample per second=378.01
2021-02-23 19:25:47,958 - root - INFO - [Iter 87/100, Epoch 3] valid f1=7.5105e-01, mcc=3.9089e-01, roc_auc=8.2455e-01, accuracy=7.0500e-01, log_loss=5.0629e-01, time spent=0.120s, total_time=0.23min
2021-02-23 19:25:48,209 - root - INFO - [Iter 90/100, Epoch 3] train loss=3.4469e-01, gnorm=6.4928e+00, lr=4.5773e-06, #samples processed=144, #sample per second=387.82
2021-02-23 19:25:48,327 - root - INFO - [Iter 90/100, Epoch 3] valid f1=7.5105e-01, mcc=3.9089e-01, roc_auc=8.2578e-01, accuracy=7.0500e-01, log_loss=5.0501e-01, time spent=0.118s, total_time=0.24min
2021-02-23 19:25:48,586 - root - INFO - [Iter 93/100, Epoch 3] train loss=2.9676e-01, gnorm=8.4980e+00, lr=3.2041e-06, #samples processed=144, #sample per second=382.53
2021-02-23 19:25:48,701 - root - INFO - [Iter 93/100, Epoch 3] valid f1=7.4576e-01, mcc=3.8122e-01, roc_auc=8.2660e-01, accuracy=7.0000e-01, log_loss=5.0409e-01, time spent=0.114s, total_time=0.25min
2021-02-23 19:25:48,980 - root - INFO - [Iter 96/100, Epoch 3] train loss=2.9195e-01, gnorm=7.5293e+00, lr=1.8309e-06, #samples processed=144, #sample per second=365.90
2021-02-23 19:25:49,097 - root - INFO - [Iter 96/100, Epoch 3] valid f1=7.4576e-01, mcc=3.8122e-01, roc_auc=8.2680e-01, accuracy=7.0000e-01, log_loss=5.0348e-01, time spent=0.117s, total_time=0.25min
2021-02-23 19:25:49,357 - root - INFO - [Iter 99/100, Epoch 3] train loss=3.3066e-01, gnorm=8.3947e+00, lr=4.5773e-07, #samples processed=144, #sample per second=381.87
2021-02-23 19:25:49,472 - root - INFO - [Iter 99/100, Epoch 3] valid f1=7.4576e-01, mcc=3.8122e-01, roc_auc=8.2721e-01, accuracy=7.0000e-01, log_loss=5.0327e-01, time spent=0.115s, total_time=0.26min
2021-02-23 19:25:49,683 - root - INFO - [Iter 100/100, Epoch 3] valid f1=7.4576e-01, mcc=3.8122e-01, roc_auc=8.2721e-01, accuracy=7.0000e-01, log_loss=5.0327e-01, time spent=0.118s, total_time=0.26min
2021-02-23 19:25:49,685 - root - INFO - Early stopping patience reached!
2021-02-23 19:26:01,438 - autogluon.text.text_prediction.text_prediction - INFO - Results=
INFO:autogluon.text.text_prediction.text_prediction:Results=
2021-02-23 19:26:01,442 - autogluon.text.text_prediction.text_prediction - INFO - Best_config={'search_space▁optimization.lr': 5e-05}
INFO:autogluon.text.text_prediction.text_prediction:Best_config={'search_space▁optimization.lr': 5e-05}
(task:2) 2021-02-23 19:25:52,380 - root - INFO - All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task2/training.log
2021-02-23 19:25:52,380 - root - INFO - learning:
early_stopping_patience: 10
log_metrics: auto
stop_metric: auto
valid_ratio: 0.15
misc:
exp_dir: /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task2
seed: 123
model:
backbone:
name: google_electra_small
network:
agg_net:
activation: tanh
agg_type: concat
data_dropout: False
dropout: 0.1
feature_proj_num_layers: -1
initializer:
bias: ['zeros']
weight: ['xavier', 'uniform', 'avg', 3.0]
mid_units: 256
norm_eps: 1e-05
normalization: layer_norm
out_proj_num_layers: 0
categorical_net:
activation: leaky
data_dropout: False
dropout: 0.1
emb_units: 32
initializer:
bias: ['zeros']
embed: ['xavier', 'gaussian', 'in', 1.0]
weight: ['xavier', 'uniform', 'avg', 3.0]
mid_units: 64
norm_eps: 1e-05
normalization: layer_norm
num_layers: 1
feature_units: -1
initializer:
bias: ['zeros']
weight: ['truncnorm', 0, 0.02]
numerical_net:
activation: leaky
data_dropout: False
dropout: 0.1
initializer:
bias: ['zeros']
weight: ['xavier', 'uniform', 'avg', 3.0]
input_centering: False
mid_units: 128
norm_eps: 1e-05
normalization: layer_norm
num_layers: 1
text_net:
pool_type: cls
use_segment_id: True
preprocess:
max_length: 128
merge_text: True
optimization:
batch_size: 32
begin_lr: 0.0
final_lr: 0.0
layerwise_lr_decay: 0.8
log_frequency: 0.1
lr: 8.342656133136337e-05
lr_scheduler: triangular
max_grad_norm: 1.0
model_average: 5
num_train_epochs: 4
optimizer: adamw
optimizer_params: [('beta1', 0.9), ('beta2', 0.999), ('epsilon', 1e-06), ('correct_bias', False)]
per_device_batch_size: 16
val_batch_size_mult: 2
valid_frequency: 0.1
warmup_portion: 0.1
wd: 0.01
version: 1
2021-02-23 19:25:52,528 - root - INFO - Process training set...
2021-02-23 19:25:52,713 - root - INFO - Done!
2021-02-23 19:25:52,713 - root - INFO - Process dev set...
2021-02-23 19:25:52,754 - root - INFO - Done!
2021-02-23 19:25:58,252 - root - INFO - #Total Params/Fixed Params=13483522/0
2021-02-23 19:25:58,267 - root - INFO - Using gradient accumulation. Global batch size = 32
2021-02-23 19:25:59,179 - root - INFO - [Iter 3/100, Epoch 0] train loss=5.8502e-01, gnorm=9.4390e+00, lr=2.5028e-05, #samples processed=144, #sample per second=165.45
2021-02-23 19:25:59,402 - root - INFO - [Iter 3/100, Epoch 0] valid f1=5.5061e-01, mcc=-1.6868e-01, roc_auc=4.3959e-01, accuracy=4.4500e-01, log_loss=7.9017e-01, time spent=0.157s, total_time=0.02min
2021-02-23 19:25:59,784 - root - INFO - [Iter 6/100, Epoch 0] train loss=5.4727e-01, gnorm=1.5773e+01, lr=5.0056e-05, #samples processed=144, #sample per second=238.11
2021-02-23 19:26:00,049 - root - INFO - [Iter 6/100, Epoch 0] valid f1=1.6667e-01, mcc=4.4428e-02, roc_auc=5.3013e-01, accuracy=4.5000e-01, log_loss=1.0137e+00, time spent=0.121s, total_time=0.03min
2021-02-23 19:26:00,345 - root - INFO - [Iter 9/100, Epoch 0] train loss=6.1448e-01, gnorm=9.7110e+00, lr=7.5084e-05, #samples processed=144, #sample per second=256.42
2021-02-23 19:26:00,579 - root - INFO - [Iter 9/100, Epoch 0] valid f1=6.5546e-01, mcc=1.5122e-01, roc_auc=6.0512e-01, accuracy=5.9000e-01, log_loss=6.8790e-01, time spent=0.118s, total_time=0.04min
2021-02-23 19:26:00,895 - root - INFO - [Iter 12/100, Epoch 0] train loss=5.0382e-01, gnorm=9.6136e+00, lr=8.1573e-05, #samples processed=144, #sample per second=262.14
2021-02-23 19:26:01,164 - root - INFO - [Iter 12/100, Epoch 0] valid f1=7.3927e-01, mcc=1.6610e-01, roc_auc=6.4113e-01, accuracy=6.0500e-01, log_loss=7.3054e-01, time spent=0.115s, total_time=0.05min
Above we specify that: the label column of our DataFrame contains the label-values to predict, AutoGluon should run for 60 seconds, each training run of an individual model (with particular hyperparameters) should run on 1 GPU, a particular random seed should be used to facilitate reproducibility, and that trained models should be saved in the ag_sst folder.
Now you can use predictor.evaluate() to evaluate the trained model
on some separate test data.
dev_score = predictor.evaluate(dev_data, metrics='acc')
print('Total Time = {}s'.format(predictor.results['total_time']))
print('Accuracy = {:.2f}%'.format(dev_score['acc'] * 100))
/var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/venv/lib/python3.8/site-packages/mxnet/gluon/block.py:995: UserWarning: The 3-th input to HybridBlock is not used by any computation. Is this intended?
self._build_cache(*args)
Total Time = 60.83460736274719s
Accuracy = 76.26%
And you can easily obtain predictions from these models.
sentence1 = "it's a charming and often affecting journey."
sentence2 = "It's slow, very, very, very slow."
predictions = predictor.predict({'sentence': [sentence1, sentence2]})
print('"Sentence":', sentence1, '"Predicted Sentiment":', predictions[0])
print('"Sentence":', sentence2, '"Predicted Sentiment":', predictions[1])
"Sentence": it's a charming and often affecting journey. "Predicted Sentiment": 1
"Sentence": It's slow, very, very, very slow. "Predicted Sentiment": 1
For classification tasks, you can ask for predicted class-probabilities instead of predicted classes.
probs = predictor.predict_proba({'sentence': [sentence1, sentence2]})
print('"Sentence":', sentence1, '"Predicted Class-Probabilities":', probs[0])
print('"Sentence":', sentence2, '"Predicted Class-Probabilities":', probs[1])
"Sentence": it's a charming and often affecting journey. "Predicted Class-Probabilities": [0.03195439 0.9680456 ]
"Sentence": It's slow, very, very, very slow. "Predicted Class-Probabilities": [0.44929108 0.5507089 ]
Sentence Similarity¶
Next, let’s use AutoGluon to train a model for evaluating how semantically similar two sentences are. We use the Semantic Textual Similarity Benchmark dataset for illustration.
train_data = load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sts/train.parquet')[['sentence1', 'sentence2', 'score']]
dev_data = load('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sts/dev.parquet')[['sentence1', 'sentence2', 'score']]
train_data.head(10)
| sentence1 | sentence2 | score | |
|---|---|---|---|
| 0 | A plane is taking off. | An air plane is taking off. | 5.00 |
| 1 | A man is playing a large flute. | A man is playing a flute. | 3.80 |
| 2 | A man is spreading shreded cheese on a pizza. | A man is spreading shredded cheese on an uncoo... | 3.80 |
| 3 | Three men are playing chess. | Two men are playing chess. | 2.60 |
| 4 | A man is playing the cello. | A man seated is playing the cello. | 4.25 |
| 5 | Some men are fighting. | Two men are fighting. | 4.25 |
| 6 | A man is smoking. | A man is skating. | 0.50 |
| 7 | The man is playing the piano. | The man is playing the guitar. | 1.60 |
| 8 | A man is playing on a guitar and singing. | A woman is playing an acoustic guitar and sing... | 2.20 |
| 9 | A person is throwing a cat on to the ceiling. | A person throws a cat on the ceiling. | 5.00 |
In this data, the score column contains numerical values (which we’d like to predict) that are human-annotated similarity scores for each given pair of sentences.
print('Min score=', min(train_data['score']), ', Max score=', max(train_data['score']))
Min score= 0.0 , Max score= 5.0
Let’s train a regression model to predict these scores with
task.fit(). Note that we only need to specify the label column and
AutoGluon automatically determines the type of prediction problem and an
appropriate loss function. Once again, you should increase the short
time_limits below to obtain reasonable performance in your own
applications.
predictor_sts = task.fit(train_data, label='score',
time_limits='1min', ngpus_per_trial=1, seed=123,
output_directory='./ag_sts')
2021-02-23 19:26:08,427 - autogluon.text.text_prediction.text_prediction - INFO - All Logs will be saved to ./ag_sts/ag_text_prediction.log
INFO:autogluon.text.text_prediction.text_prediction:All Logs will be saved to ./ag_sts/ag_text_prediction.log
2021-02-23 19:26:08,457 - autogluon.text.text_prediction.text_prediction - INFO - Train Dataset:
INFO:autogluon.text.text_prediction.text_prediction:Train Dataset:
2021-02-23 19:26:08,457 - autogluon.text.text_prediction.text_prediction - INFO - Columns:
- Text(
name="sentence1"
#total/missing=4599/0
length, min/avg/max=16/57.77625570776256/367
)
- Text(
name="sentence2"
#total/missing=4599/0
length, min/avg/max=15/57.63513807349424/311
)
- Numerical(
name="score"
#total/missing=4599/0
shape=()
)
INFO:autogluon.text.text_prediction.text_prediction:Columns:
- Text(
name="sentence1"
#total/missing=4599/0
length, min/avg/max=16/57.77625570776256/367
)
- Text(
name="sentence2"
#total/missing=4599/0
length, min/avg/max=15/57.63513807349424/311
)
- Numerical(
name="score"
#total/missing=4599/0
shape=()
)
2021-02-23 19:26:08,458 - autogluon.text.text_prediction.text_prediction - INFO - Tuning Dataset:
INFO:autogluon.text.text_prediction.text_prediction:Tuning Dataset:
2021-02-23 19:26:08,460 - autogluon.text.text_prediction.text_prediction - INFO - Columns:
- Text(
name="sentence1"
#total/missing=1150/0
length, min/avg/max=17/57.43739130434783/267
)
- Text(
name="sentence2"
#total/missing=1150/0
length, min/avg/max=16/57.122608695652175/249
)
- Numerical(
name="score"
#total/missing=1150/0
shape=()
)
INFO:autogluon.text.text_prediction.text_prediction:Columns:
- Text(
name="sentence1"
#total/missing=1150/0
length, min/avg/max=17/57.43739130434783/267
)
- Text(
name="sentence2"
#total/missing=1150/0
length, min/avg/max=16/57.122608695652175/249
)
- Numerical(
name="score"
#total/missing=1150/0
shape=()
)
2021-02-23 19:26:08,462 - autogluon.text.text_prediction.text_prediction - INFO - All Logs will be saved to ./ag_sts/main.log
INFO:autogluon.text.text_prediction.text_prediction:All Logs will be saved to ./ag_sts/main.log
0%| | 0/3 [00:00<?, ?it/s]
2021-02-23 19:27:09,016 - autogluon.text.text_prediction.text_prediction - INFO - Results=
INFO:autogluon.text.text_prediction.text_prediction:Results=
2021-02-23 19:27:09,017 - autogluon.text.text_prediction.text_prediction - INFO - Best_config={'search_space▁optimization.lr': 5e-05}
INFO:autogluon.text.text_prediction.text_prediction:Best_config={'search_space▁optimization.lr': 5e-05}
(task:3) 2021-02-23 19:26:11,051 - root - INFO - All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sts/task3/training.log
2021-02-23 19:26:11,051 - root - INFO - learning:
early_stopping_patience: 10
log_metrics: auto
stop_metric: auto
valid_ratio: 0.15
misc:
exp_dir: /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sts/task3
seed: 123
model:
backbone:
name: google_electra_small
network:
agg_net:
activation: tanh
agg_type: concat
data_dropout: False
dropout: 0.1
feature_proj_num_layers: -1
initializer:
bias: ['zeros']
weight: ['xavier', 'uniform', 'avg', 3.0]
mid_units: 256
norm_eps: 1e-05
normalization: layer_norm
out_proj_num_layers: 0
categorical_net:
activation: leaky
data_dropout: False
dropout: 0.1
emb_units: 32
initializer:
bias: ['zeros']
embed: ['xavier', 'gaussian', 'in', 1.0]
weight: ['xavier', 'uniform', 'avg', 3.0]
mid_units: 64
norm_eps: 1e-05
normalization: layer_norm
num_layers: 1
feature_units: -1
initializer:
bias: ['zeros']
weight: ['truncnorm', 0, 0.02]
numerical_net:
activation: leaky
data_dropout: False
dropout: 0.1
initializer:
bias: ['zeros']
weight: ['xavier', 'uniform', 'avg', 3.0]
input_centering: False
mid_units: 128
norm_eps: 1e-05
normalization: layer_norm
num_layers: 1
text_net:
pool_type: cls
use_segment_id: True
preprocess:
max_length: 128
merge_text: True
optimization:
batch_size: 32
begin_lr: 0.0
final_lr: 0.0
layerwise_lr_decay: 0.8
log_frequency: 0.1
lr: 5e-05
lr_scheduler: triangular
max_grad_norm: 1.0
model_average: 5
num_train_epochs: 4
optimizer: adamw
optimizer_params: [('beta1', 0.9), ('beta2', 0.999), ('epsilon', 1e-06), ('correct_bias', False)]
per_device_batch_size: 16
val_batch_size_mult: 2
valid_frequency: 0.1
warmup_portion: 0.1
wd: 0.01
version: 1
2021-02-23 19:26:11,198 - root - INFO - Process training set...
2021-02-23 19:26:14,891 - root - INFO - Done!
2021-02-23 19:26:14,891 - root - INFO - Process dev set...
2021-02-23 19:26:17,747 - root - INFO - Done!
2021-02-23 19:26:22,966 - root - INFO - #Total Params/Fixed Params=13483265/0
2021-02-23 19:26:22,984 - root - INFO - Using gradient accumulation. Global batch size = 32
2021-02-23 19:26:25,690 - root - INFO - [Iter 15/576, Epoch 0] train loss=5.6454e+00, gnorm=4.9295e+01, lr=1.3158e-05, #samples processed=720, #sample per second=269.83
2021-02-23 19:26:26,717 - root - INFO - [Iter 15/576, Epoch 0] valid mean_squared_error=2.4207e+00, root_mean_squared_error=1.5558e+00, mean_absolute_error=1.3127e+00, time spent=0.948s, total_time=0.06min
2021-02-23 19:26:28,614 - root - INFO - [Iter 30/576, Epoch 0] train loss=1.7106e+00, gnorm=2.1737e+01, lr=2.6316e-05, #samples processed=720, #sample per second=246.25
2021-02-23 19:26:29,704 - root - INFO - [Iter 30/576, Epoch 0] valid mean_squared_error=1.5456e+00, root_mean_squared_error=1.2432e+00, mean_absolute_error=1.0504e+00, time spent=0.949s, total_time=0.11min
2021-02-23 19:26:31,522 - root - INFO - [Iter 45/576, Epoch 0] train loss=1.1062e+00, gnorm=3.7559e+01, lr=3.9474e-05, #samples processed=720, #sample per second=247.65
2021-02-23 19:26:32,639 - root - INFO - [Iter 45/576, Epoch 0] valid mean_squared_error=1.1685e+00, root_mean_squared_error=1.0810e+00, mean_absolute_error=8.9138e-01, time spent=0.948s, total_time=0.16min
2021-02-23 19:26:34,442 - root - INFO - [Iter 60/576, Epoch 0] train loss=9.6977e-01, gnorm=1.9567e+01, lr=4.9711e-05, #samples processed=720, #sample per second=246.55
2021-02-23 19:26:35,551 - root - INFO - [Iter 60/576, Epoch 0] valid mean_squared_error=8.5146e-01, root_mean_squared_error=9.2275e-01, mean_absolute_error=7.5173e-01, time spent=0.954s, total_time=0.21min
2021-02-23 19:26:37,736 - root - INFO - [Iter 75/576, Epoch 0] train loss=7.6825e-01, gnorm=2.1465e+01, lr=4.8266e-05, #samples processed=720, #sample per second=218.59
2021-02-23 19:26:38,708 - root - INFO - [Iter 75/576, Epoch 0] valid mean_squared_error=8.9216e-01, root_mean_squared_error=9.4454e-01, mean_absolute_error=7.5832e-01, time spent=0.971s, total_time=0.26min
2021-02-23 19:26:40,909 - root - INFO - [Iter 90/576, Epoch 0] train loss=8.0189e-01, gnorm=1.1030e+01, lr=4.6821e-05, #samples processed=720, #sample per second=226.94
2021-02-23 19:26:42,005 - root - INFO - [Iter 90/576, Epoch 0] valid mean_squared_error=7.1927e-01, root_mean_squared_error=8.4810e-01, mean_absolute_error=6.8095e-01, time spent=0.956s, total_time=0.32min
2021-02-23 19:26:43,708 - root - INFO - [Iter 105/576, Epoch 0] train loss=6.9309e-01, gnorm=1.1257e+01, lr=4.5376e-05, #samples processed=720, #sample per second=257.26
2021-02-23 19:26:44,853 - root - INFO - [Iter 105/576, Epoch 0] valid mean_squared_error=7.0311e-01, root_mean_squared_error=8.3852e-01, mean_absolute_error=6.6397e-01, time spent=0.961s, total_time=0.36min
2021-02-23 19:26:46,574 - root - INFO - [Iter 120/576, Epoch 0] train loss=6.4387e-01, gnorm=1.7313e+01, lr=4.3931e-05, #samples processed=720, #sample per second=251.28
2021-02-23 19:26:47,535 - root - INFO - [Iter 120/576, Epoch 0] valid mean_squared_error=7.6587e-01, root_mean_squared_error=8.7514e-01, mean_absolute_error=6.9315e-01, time spent=0.961s, total_time=0.41min
2021-02-23 19:26:49,140 - root - INFO - [Iter 135/576, Epoch 0] train loss=6.6454e-01, gnorm=3.6993e+01, lr=4.2486e-05, #samples processed=720, #sample per second=280.58
2021-02-23 19:26:50,100 - root - INFO - [Iter 135/576, Epoch 0] valid mean_squared_error=9.2323e-01, root_mean_squared_error=9.6085e-01, mean_absolute_error=7.6496e-01, time spent=0.960s, total_time=0.45min
2021-02-23 19:26:51,854 - root - INFO - [Iter 150/576, Epoch 1] train loss=6.3641e-01, gnorm=2.0107e+01, lr=4.1040e-05, #samples processed=711, #sample per second=262.04
2021-02-23 19:26:52,821 - root - INFO - [Iter 150/576, Epoch 1] valid mean_squared_error=7.4271e-01, root_mean_squared_error=8.6181e-01, mean_absolute_error=6.8223e-01, time spent=0.967s, total_time=0.50min
2021-02-23 19:26:55,256 - root - INFO - [Iter 165/576, Epoch 1] train loss=6.1940e-01, gnorm=1.3970e+01, lr=3.9595e-05, #samples processed=720, #sample per second=211.63
2021-02-23 19:26:56,374 - root - INFO - [Iter 165/576, Epoch 1] valid mean_squared_error=6.9922e-01, root_mean_squared_error=8.3619e-01, mean_absolute_error=6.6199e-01, time spent=0.982s, total_time=0.56min
2021-02-23 19:26:58,262 - root - INFO - [Iter 180/576, Epoch 1] train loss=5.7463e-01, gnorm=2.2712e+01, lr=3.8150e-05, #samples processed=720, #sample per second=239.51
2021-02-23 19:26:59,230 - root - INFO - [Iter 180/576, Epoch 1] valid mean_squared_error=7.2200e-01, root_mean_squared_error=8.4970e-01, mean_absolute_error=6.6850e-01, time spent=0.968s, total_time=0.60min
2021-02-23 19:27:01,138 - root - INFO - [Iter 195/576, Epoch 1] train loss=5.9736e-01, gnorm=1.0646e+01, lr=3.6705e-05, #samples processed=720, #sample per second=250.34
2021-02-23 19:27:02,295 - root - INFO - [Iter 195/576, Epoch 1] valid mean_squared_error=5.6824e-01, root_mean_squared_error=7.5382e-01, mean_absolute_error=5.9536e-01, time spent=0.978s, total_time=0.65min
2021-02-23 19:27:04,754 - root - INFO - [Iter 210/576, Epoch 1] train loss=6.2861e-01, gnorm=2.3337e+01, lr=3.5260e-05, #samples processed=720, #sample per second=199.15
2021-02-23 19:27:05,740 - root - INFO - [Iter 210/576, Epoch 1] valid mean_squared_error=8.1212e-01, root_mean_squared_error=9.0118e-01, mean_absolute_error=7.1468e-01, time spent=0.986s, total_time=0.71min
2021-02-23 19:27:07,663 - root - INFO - [Iter 225/576, Epoch 1] train loss=5.4069e-01, gnorm=1.6118e+01, lr=3.3815e-05, #samples processed=720, #sample per second=247.49
2021-02-23 19:27:08,641 - root - INFO - [Iter 225/576, Epoch 1] valid mean_squared_error=5.9569e-01, root_mean_squared_error=7.7181e-01, mean_absolute_error=5.9851e-01, time spent=0.977s, total_time=0.76min
We again evaluate our trained model’s performance on some separate test data. Below we choose to compute the following metrics: RMSE, Pearson Correlation, and Spearman Correlation.
dev_score = predictor_sts.evaluate(dev_data, metrics=['rmse', 'pearsonr', 'spearmanr'])
print('Best Config = {}'.format(predictor_sts.results['best_config']))
print('Total Time = {}s'.format(predictor_sts.results['total_time']))
print('RMSE = {:.2f}'.format(dev_score['rmse']))
print('PEARSONR = {:.4f}'.format(dev_score['pearsonr']))
print('SPEARMANR = {:.4f}'.format(dev_score['spearmanr']))
/var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/venv/lib/python3.8/site-packages/mxnet/gluon/block.py:995: UserWarning: The 3-th input to HybridBlock is not used by any computation. Is this intended?
self._build_cache(*args)
Best Config = {'search_space▁optimization.lr': 5e-05}
Total Time = 60.572534799575806s
RMSE = 0.80
PEARSONR = 0.8458
SPEARMANR = 0.8475
Let’s use our model to predict the similarity score among these sentences:
‘The child is riding a horse.’
‘The young boy is riding a horse.’
‘The young man is riding a horse.’
‘The young man is riding a bicycle.’
sentences = ['The child is riding a horse.',
'The young boy is riding a horse.',
'The young man is riding a horse.',
'The young man is riding a bicycle.']
score1 = predictor_sts.predict({'sentence1': [sentences[0]],
'sentence2': [sentences[1]]})
score2 = predictor_sts.predict({'sentence1': [sentences[0]],
'sentence2': [sentences[2]]})
score3 = predictor_sts.predict({'sentence1': [sentences[0]],
'sentence2': [sentences[3]]})
print(score1, score2, score3)
[2.8975334] [2.4277894] [1.6367726]
Save and Load¶
Here, we demonstrate how to easily save and load a trained TextPrediction model.
predictor_sts.save('saved_dir')
predictor_sts_new = task.load('saved_dir')
score3 = predictor_sts_new.predict({'sentence1': [sentences[0]],
'sentence2': [sentences[3]]})
print(score3)
[1.6367726]
/var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/venv/lib/python3.8/site-packages/mxnet/gluon/block.py:995: UserWarning: The 3-th input to HybridBlock is not used by any computation. Is this intended?
self._build_cache(*args)
Extract Embeddings¶
After you have trained a predictor, you can also use the predictor to extract embeddings that maps the input data to a real vector. This can be useful for integrating with other AutoGluon modules like TabularPredictor. We can just feed the embeddings to TabularPredictor.
embeddings = predictor_sts_new.extract_embedding(dev_data)
print(embeddings)
[[ 0.7491128 0.74637544 0.6407232 ... 0.93967164 0.2225879
-1.0021996 ]
[ 0.6876894 0.6852882 0.51317114 ... 1.0417908 0.22565487
-0.8999633 ]
[ 0.7459354 0.72983456 0.68254095 ... 0.71984047 0.28864622
-1.029607 ]
...
[-0.18616538 1.4507027 0.09135915 ... 2.2345812 0.41163737
-0.22406614]
[ 0.04140446 1.4527035 -0.7491549 ... 2.5234642 -0.11837799
0.35897902]
[ 0.23341516 1.5211928 -0.07434467 ... 2.5080233 0.23936479
-0.05783204]]
Note: TextPrediction depends on the
GluonNLP package. Due to an ongoing
upgrade of GluonNLP, we are currently using a custom version of the
package:
autogluon-contrib-nlp.
In a future release, AutoGluon will switch to using the official
GluonNLP, but the APIs demonstrated here will remain the same.