Text Prediction - Quick Start¶
Here we briefly demonstrate the TextPredictor, which helps you
automatically train and deploy models for various Natural Language
Processing (NLP) tasks. This tutorial presents two examples of NLP
tasks:
The general usage of the TextPredictor is similar to AutoGluon’s
TabularPredictor. We format NLP datasets as tables where certain
columns contain text fields and a special column contains the labels to
predict, and each row corresponds to one training example. Here, the
labels can be discrete categories (classification) or numerical values
(regression).
%matplotlib inline
import numpy as np
import warnings
import matplotlib.pyplot as plt
warnings.filterwarnings('ignore')
np.random.seed(123)
Sentiment Analysis Task¶
First, we consider the Stanford Sentiment Treebank (SST) dataset, which consists of movie reviews and their associated sentiment. Given a new movie review, the goal is to predict the sentiment reflected in the text (in this case a binary classification, where reviews are labeled as 1 if they convey a positive opinion and labeled as 0 otherwise). Let’s first load and look at the data, noting the labels are stored in a column called label.
from autogluon.core import TabularDataset
train_data = TabularDataset('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/train.parquet')
test_data = TabularDataset('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sst/dev.parquet')
subsample_size = 1000 # subsample data for faster demo, try setting this to larger values
train_data = train_data.sample(n=subsample_size, random_state=0)
train_data.head(10)
| sentence | label | |
|---|---|---|
| 43787 | very pleasing at its best moments | 1 |
| 16159 | , american chai is enough to make you put away... | 0 |
| 59015 | too much like an infomercial for ram dass 's l... | 0 |
| 5108 | a stirring visual sequence | 1 |
| 67052 | cool visual backmasking | 1 |
| 35938 | hard ground | 0 |
| 49879 | the striking , quietly vulnerable personality ... | 1 |
| 51591 | pan nalin 's exposition is beautiful and myste... | 1 |
| 56780 | wonderfully loopy | 1 |
| 28518 | most beautiful , evocative | 1 |
Above the data happen to be stored in a
Parquet table
format, but you can also directly load() data from a
CSV file
instead. While here we load files from AWS S3 cloud
storage,
these could instead be local files on your machine. After loading,
train_data is simply a Pandas
DataFrame,
where each row represents a different training example (for machine
learning to be appropriate, the rows should be independent and
identically distributed).
Training¶
To ensure this tutorial runs quickly, we simply call fit() with a
subset of 1000 training examples and limit its runtime to approximately
1 minute. To achieve reasonable performance in your applications, you
are recommended to set much longer time_limit (eg. 1 hour), or do
not specify time_limit at all (time_limit=None).
from autogluon.text import TextPredictor
predictor = TextPredictor(label='label', eval_metric='acc', path='./ag_sst')
predictor.fit(train_data, time_limit=60)
Problem Type="binary"
Column Types:
- "sentence": text
- "label": categorical
The GluonNLP V0 backend is used. We will use 8 cpus and 1 gpus to train each trial.
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task0/training.log
2022-01-21 06:03:18,224 - autogluon.text.text_prediction.mx.models - INFO - Fitting and transforming the train data...
Fitting and transforming the train data...
2022-01-21 06:03:18,258 - autogluon.text.text_prediction.mx.models - INFO - Done! Preprocessor saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task0/preprocessor.pkl
Done! Preprocessor saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task0/preprocessor.pkl
2022-01-21 06:03:18,261 - autogluon.text.text_prediction.mx.models - INFO - Process dev set...
Process dev set...
2022-01-21 06:03:18,271 - autogluon.text.text_prediction.mx.models - INFO - Done!
Done!
2022-01-21 06:03:18,280 - autogluon.text.text_prediction.mx.models - INFO - Max length for chunking text: 64, Stochastic chunk: Train-False/Test-False, Test #repeat: 1.
Max length for chunking text: 64, Stochastic chunk: Train-False/Test-False, Test #repeat: 1.
2022-01-21 06:03:21,670 - autogluon.text.text_prediction.mx.models - INFO - #Total Params/Fixed Params=108990466/0
#Total Params/Fixed Params=108990466/0
2022-01-21 06:03:21,682 - autogluon.text.text_prediction.mx.models - Level 15 - Using gradient accumulation. Global batch size = 128
Using gradient accumulation. Global batch size = 128
2022-01-21 06:03:21,787 - autogluon.text.text_prediction.mx.models - INFO - Local training results will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task0/results_local.jsonl.
Local training results will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sst/task0/results_local.jsonl.
2022-01-21 06:03:23,255 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 1/70, Epoch 0] train loss=8.80e-01, gnorm=7.51e+00, lr=1.43e-05, #samples processed=128, #sample per second=87.20. ETA=1.69min
[Iter 1/70, Epoch 0] train loss=8.80e-01, gnorm=7.51e+00, lr=1.43e-05, #samples processed=128, #sample per second=87.20. ETA=1.69min
2022-01-21 06:03:24,170 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 2/70, Epoch 0] train loss=7.25e-01, gnorm=4.69e+00, lr=2.86e-05, #samples processed=128, #sample per second=139.95. ETA=1.35min
[Iter 2/70, Epoch 0] train loss=7.25e-01, gnorm=4.69e+00, lr=2.86e-05, #samples processed=128, #sample per second=139.95. ETA=1.35min
2022-01-21 06:03:25,730 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 2/70, Epoch 0] Validation f1=3.0769e-01, mcc=1.4264e-01, roc_auc=5.9323e-01, accuracy=5.0500e-01, log_loss=7.9234e-01, Time computing validation-score=0.438s, Total time spent=0.07min. Found improved model=True, Improved top-3 models=True
[Iter 2/70, Epoch 0] Validation f1=3.0769e-01, mcc=1.4264e-01, roc_auc=5.9323e-01, accuracy=5.0500e-01, log_loss=7.9234e-01, Time computing validation-score=0.438s, Total time spent=0.07min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:03:26,507 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 3/70, Epoch 0] train loss=7.46e-01, gnorm=6.52e+00, lr=4.29e-05, #samples processed=128, #sample per second=54.81. ETA=1.76min
[Iter 3/70, Epoch 0] train loss=7.46e-01, gnorm=6.52e+00, lr=4.29e-05, #samples processed=128, #sample per second=54.81. ETA=1.76min
2022-01-21 06:03:27,220 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 4/70, Epoch 0] train loss=6.93e-01, gnorm=4.71e+00, lr=5.71e-05, #samples processed=128, #sample per second=179.66. ETA=1.49min
[Iter 4/70, Epoch 0] train loss=6.93e-01, gnorm=4.71e+00, lr=5.71e-05, #samples processed=128, #sample per second=179.66. ETA=1.49min
2022-01-21 06:03:28,888 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 4/70, Epoch 0] Validation f1=7.3139e-01, mcc=1.6281e-01, roc_auc=7.6076e-01, accuracy=5.8500e-01, log_loss=6.9661e-01, Time computing validation-score=0.439s, Total time spent=0.12min. Found improved model=True, Improved top-3 models=True
[Iter 4/70, Epoch 0] Validation f1=7.3139e-01, mcc=1.6281e-01, roc_auc=7.6076e-01, accuracy=5.8500e-01, log_loss=6.9661e-01, Time computing validation-score=0.439s, Total time spent=0.12min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:03:29,646 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 5/70, Epoch 0] train loss=7.43e-01, gnorm=6.06e+00, lr=7.14e-05, #samples processed=128, #sample per second=52.80. ETA=1.70min
[Iter 5/70, Epoch 0] train loss=7.43e-01, gnorm=6.06e+00, lr=7.14e-05, #samples processed=128, #sample per second=52.80. ETA=1.70min
2022-01-21 06:03:30,412 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 6/70, Epoch 0] train loss=6.51e-01, gnorm=4.72e+00, lr=8.57e-05, #samples processed=128, #sample per second=167.26. ETA=1.53min
[Iter 6/70, Epoch 0] train loss=6.51e-01, gnorm=4.72e+00, lr=8.57e-05, #samples processed=128, #sample per second=167.26. ETA=1.53min
2022-01-21 06:03:32,194 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 6/70, Epoch 0] Validation f1=4.4295e-01, mcc=3.3235e-01, roc_auc=8.5861e-01, accuracy=5.8500e-01, log_loss=6.4510e-01, Time computing validation-score=0.436s, Total time spent=0.17min. Found improved model=True, Improved top-3 models=True
[Iter 6/70, Epoch 0] Validation f1=4.4295e-01, mcc=3.3235e-01, roc_auc=8.5861e-01, accuracy=5.8500e-01, log_loss=6.4510e-01, Time computing validation-score=0.436s, Total time spent=0.17min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:03:32,969 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 7/70, Epoch 0] train loss=6.10e-01, gnorm=6.25e+00, lr=1.00e-04, #samples processed=128, #sample per second=50.08. ETA=1.68min
[Iter 7/70, Epoch 0] train loss=6.10e-01, gnorm=6.25e+00, lr=1.00e-04, #samples processed=128, #sample per second=50.08. ETA=1.68min
2022-01-21 06:03:33,738 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 8/70, Epoch 1] train loss=4.94e-01, gnorm=3.03e+00, lr=9.84e-05, #samples processed=128, #sample per second=166.74. ETA=1.54min
[Iter 8/70, Epoch 1] train loss=4.94e-01, gnorm=3.03e+00, lr=9.84e-05, #samples processed=128, #sample per second=166.74. ETA=1.54min
2022-01-21 06:03:35,751 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 8/70, Epoch 1] Validation f1=8.2528e-01, mcc=5.5657e-01, roc_auc=9.2046e-01, accuracy=7.6500e-01, log_loss=4.9082e-01, Time computing validation-score=0.440s, Total time spent=0.23min. Found improved model=True, Improved top-3 models=True
[Iter 8/70, Epoch 1] Validation f1=8.2528e-01, mcc=5.5657e-01, roc_auc=9.2046e-01, accuracy=7.6500e-01, log_loss=4.9082e-01, Time computing validation-score=0.440s, Total time spent=0.23min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:03:36,497 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 9/70, Epoch 1] train loss=4.96e-01, gnorm=7.11e+00, lr=9.68e-05, #samples processed=128, #sample per second=46.40. ETA=1.66min
[Iter 9/70, Epoch 1] train loss=4.96e-01, gnorm=7.11e+00, lr=9.68e-05, #samples processed=128, #sample per second=46.40. ETA=1.66min
2022-01-21 06:03:37,238 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 10/70, Epoch 1] train loss=4.22e-01, gnorm=4.46e+00, lr=9.52e-05, #samples processed=128, #sample per second=173.04. ETA=1.55min
[Iter 10/70, Epoch 1] train loss=4.22e-01, gnorm=4.46e+00, lr=9.52e-05, #samples processed=128, #sample per second=173.04. ETA=1.55min
2022-01-21 06:03:40,178 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 10/70, Epoch 1] Validation f1=8.8372e-01, mcc=7.5395e-01, roc_auc=9.4507e-01, accuracy=8.7500e-01, log_loss=3.0577e-01, Time computing validation-score=0.443s, Total time spent=0.31min. Found improved model=True, Improved top-3 models=True
[Iter 10/70, Epoch 1] Validation f1=8.8372e-01, mcc=7.5395e-01, roc_auc=9.4507e-01, accuracy=8.7500e-01, log_loss=3.0577e-01, Time computing validation-score=0.443s, Total time spent=0.31min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:03:41,029 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 11/70, Epoch 1] train loss=3.51e-01, gnorm=4.60e+00, lr=9.37e-05, #samples processed=128, #sample per second=33.78. ETA=1.72min
[Iter 11/70, Epoch 1] train loss=3.51e-01, gnorm=4.60e+00, lr=9.37e-05, #samples processed=128, #sample per second=33.78. ETA=1.72min
2022-01-21 06:03:41,738 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 12/70, Epoch 1] train loss=4.64e-01, gnorm=5.13e+00, lr=9.21e-05, #samples processed=128, #sample per second=180.66. ETA=1.61min
[Iter 12/70, Epoch 1] train loss=4.64e-01, gnorm=5.13e+00, lr=9.21e-05, #samples processed=128, #sample per second=180.66. ETA=1.61min
2022-01-21 06:03:43,734 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 12/70, Epoch 1] Validation f1=9.1736e-01, mcc=8.0335e-01, roc_auc=9.4751e-01, accuracy=9.0000e-01, log_loss=3.4959e-01, Time computing validation-score=0.432s, Total time spent=0.37min. Found improved model=True, Improved top-3 models=True
[Iter 12/70, Epoch 1] Validation f1=9.1736e-01, mcc=8.0335e-01, roc_auc=9.4751e-01, accuracy=9.0000e-01, log_loss=3.4959e-01, Time computing validation-score=0.432s, Total time spent=0.37min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:03:44,534 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 13/70, Epoch 1] train loss=4.11e-01, gnorm=1.03e+01, lr=9.05e-05, #samples processed=128, #sample per second=45.81. ETA=1.66min
[Iter 13/70, Epoch 1] train loss=4.11e-01, gnorm=1.03e+01, lr=9.05e-05, #samples processed=128, #sample per second=45.81. ETA=1.66min
2022-01-21 06:03:45,276 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 14/70, Epoch 1] train loss=3.27e-01, gnorm=7.17e+00, lr=8.89e-05, #samples processed=128, #sample per second=172.74. ETA=1.57min
[Iter 14/70, Epoch 1] train loss=3.27e-01, gnorm=7.17e+00, lr=8.89e-05, #samples processed=128, #sample per second=172.74. ETA=1.57min
2022-01-21 06:03:46,539 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 14/70, Epoch 1] Validation f1=8.8679e-01, mcc=7.6785e-01, roc_auc=9.6013e-01, accuracy=8.8000e-01, log_loss=2.7562e-01, Time computing validation-score=0.438s, Total time spent=0.41min. Found improved model=False, Improved top-3 models=True
[Iter 14/70, Epoch 1] Validation f1=8.8679e-01, mcc=7.6785e-01, roc_auc=9.6013e-01, accuracy=8.8000e-01, log_loss=2.7562e-01, Time computing validation-score=0.438s, Total time spent=0.41min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:03:47,288 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 15/70, Epoch 2] train loss=3.52e-01, gnorm=7.25e+00, lr=8.73e-05, #samples processed=128, #sample per second=63.66. ETA=1.56min
[Iter 15/70, Epoch 2] train loss=3.52e-01, gnorm=7.25e+00, lr=8.73e-05, #samples processed=128, #sample per second=63.66. ETA=1.56min
2022-01-21 06:03:48,069 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 16/70, Epoch 2] train loss=2.06e-01, gnorm=2.10e+00, lr=8.57e-05, #samples processed=128, #sample per second=163.93. ETA=1.48min
[Iter 16/70, Epoch 2] train loss=2.06e-01, gnorm=2.10e+00, lr=8.57e-05, #samples processed=128, #sample per second=163.93. ETA=1.48min
2022-01-21 06:03:50,115 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 16/70, Epoch 2] Validation f1=9.2500e-01, mcc=8.2215e-01, roc_auc=9.6104e-01, accuracy=9.1000e-01, log_loss=2.9698e-01, Time computing validation-score=0.443s, Total time spent=0.47min. Found improved model=True, Improved top-3 models=True
[Iter 16/70, Epoch 2] Validation f1=9.2500e-01, mcc=8.2215e-01, roc_auc=9.6104e-01, accuracy=9.1000e-01, log_loss=2.9698e-01, Time computing validation-score=0.443s, Total time spent=0.47min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:03:50,865 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 17/70, Epoch 2] train loss=2.57e-01, gnorm=3.11e+00, lr=8.41e-05, #samples processed=128, #sample per second=45.80. ETA=1.51min
[Iter 17/70, Epoch 2] train loss=2.57e-01, gnorm=3.11e+00, lr=8.41e-05, #samples processed=128, #sample per second=45.80. ETA=1.51min
2022-01-21 06:03:51,621 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 18/70, Epoch 2] train loss=2.19e-01, gnorm=7.29e+00, lr=8.25e-05, #samples processed=128, #sample per second=169.57. ETA=1.44min
[Iter 18/70, Epoch 2] train loss=2.19e-01, gnorm=7.29e+00, lr=8.25e-05, #samples processed=128, #sample per second=169.57. ETA=1.44min
2022-01-21 06:03:52,847 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 18/70, Epoch 2] Validation f1=9.1775e-01, mcc=8.0651e-01, roc_auc=9.5117e-01, accuracy=9.0500e-01, log_loss=3.1864e-01, Time computing validation-score=0.444s, Total time spent=0.52min. Found improved model=False, Improved top-3 models=True
[Iter 18/70, Epoch 2] Validation f1=9.1775e-01, mcc=8.0651e-01, roc_auc=9.5117e-01, accuracy=9.0500e-01, log_loss=3.1864e-01, Time computing validation-score=0.444s, Total time spent=0.52min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:03:53,581 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 19/70, Epoch 2] train loss=1.33e-01, gnorm=3.85e+00, lr=8.10e-05, #samples processed=128, #sample per second=65.34. ETA=1.42min
[Iter 19/70, Epoch 2] train loss=1.33e-01, gnorm=3.85e+00, lr=8.10e-05, #samples processed=128, #sample per second=65.34. ETA=1.42min
2022-01-21 06:03:54,328 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 20/70, Epoch 2] train loss=3.07e-01, gnorm=4.33e+00, lr=7.94e-05, #samples processed=128, #sample per second=171.77. ETA=1.36min
[Iter 20/70, Epoch 2] train loss=3.07e-01, gnorm=4.33e+00, lr=7.94e-05, #samples processed=128, #sample per second=171.77. ETA=1.36min
2022-01-21 06:03:54,768 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 20/70, Epoch 2] Validation f1=8.9431e-01, mcc=7.4479e-01, roc_auc=9.3287e-01, accuracy=8.7000e-01, log_loss=4.9610e-01, Time computing validation-score=0.439s, Total time spent=0.55min. Found improved model=False, Improved top-3 models=False
[Iter 20/70, Epoch 2] Validation f1=8.9431e-01, mcc=7.4479e-01, roc_auc=9.3287e-01, accuracy=8.7000e-01, log_loss=4.9610e-01, Time computing validation-score=0.439s, Total time spent=0.55min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:03:55,482 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 21/70, Epoch 2] train loss=1.40e-01, gnorm=6.67e+00, lr=7.78e-05, #samples processed=128, #sample per second=111.00. ETA=1.31min
[Iter 21/70, Epoch 2] train loss=1.40e-01, gnorm=6.67e+00, lr=7.78e-05, #samples processed=128, #sample per second=111.00. ETA=1.31min
2022-01-21 06:03:56,281 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 22/70, Epoch 3] train loss=1.50e-01, gnorm=6.06e+00, lr=7.62e-05, #samples processed=128, #sample per second=160.43. ETA=1.25min
[Iter 22/70, Epoch 3] train loss=1.50e-01, gnorm=6.06e+00, lr=7.62e-05, #samples processed=128, #sample per second=160.43. ETA=1.25min
2022-01-21 06:03:56,730 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 22/70, Epoch 3] Validation f1=8.6124e-01, mcc=7.2190e-01, roc_auc=9.4700e-01, accuracy=8.5500e-01, log_loss=4.3860e-01, Time computing validation-score=0.448s, Total time spent=0.58min. Found improved model=False, Improved top-3 models=False
[Iter 22/70, Epoch 3] Validation f1=8.6124e-01, mcc=7.2190e-01, roc_auc=9.4700e-01, accuracy=8.5500e-01, log_loss=4.3860e-01, Time computing validation-score=0.448s, Total time spent=0.58min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:03:57,472 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 23/70, Epoch 3] train loss=1.57e-01, gnorm=7.20e+00, lr=7.46e-05, #samples processed=128, #sample per second=107.56. ETA=1.22min
[Iter 23/70, Epoch 3] train loss=1.57e-01, gnorm=7.20e+00, lr=7.46e-05, #samples processed=128, #sample per second=107.56. ETA=1.22min
2022-01-21 06:03:58,258 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 24/70, Epoch 3] train loss=5.49e-01, gnorm=1.50e+01, lr=7.30e-05, #samples processed=128, #sample per second=163.08. ETA=1.17min
[Iter 24/70, Epoch 3] train loss=5.49e-01, gnorm=1.50e+01, lr=7.30e-05, #samples processed=128, #sample per second=163.08. ETA=1.17min
2022-01-21 06:03:58,700 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 24/70, Epoch 3] Validation f1=8.6792e-01, mcc=7.2751e-01, roc_auc=9.5240e-01, accuracy=8.6000e-01, log_loss=4.1723e-01, Time computing validation-score=0.441s, Total time spent=0.62min. Found improved model=False, Improved top-3 models=False
[Iter 24/70, Epoch 3] Validation f1=8.6792e-01, mcc=7.2751e-01, roc_auc=9.5240e-01, accuracy=8.6000e-01, log_loss=4.1723e-01, Time computing validation-score=0.441s, Total time spent=0.62min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:03:59,481 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 25/70, Epoch 3] train loss=2.56e-01, gnorm=6.44e+00, lr=7.14e-05, #samples processed=128, #sample per second=104.77. ETA=1.13min
[Iter 25/70, Epoch 3] train loss=2.56e-01, gnorm=6.44e+00, lr=7.14e-05, #samples processed=128, #sample per second=104.77. ETA=1.13min
2022-01-21 06:04:00,249 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 26/70, Epoch 3] train loss=1.89e-01, gnorm=3.32e+00, lr=6.98e-05, #samples processed=128, #sample per second=166.97. ETA=1.08min
[Iter 26/70, Epoch 3] train loss=1.89e-01, gnorm=3.32e+00, lr=6.98e-05, #samples processed=128, #sample per second=166.97. ETA=1.08min
2022-01-21 06:04:00,698 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 26/70, Epoch 3] Validation f1=8.7059e-01, mcc=6.8391e-01, roc_auc=9.1944e-01, accuracy=8.3500e-01, log_loss=6.4412e-01, Time computing validation-score=0.448s, Total time spent=0.65min. Found improved model=False, Improved top-3 models=False
[Iter 26/70, Epoch 3] Validation f1=8.7059e-01, mcc=6.8391e-01, roc_auc=9.1944e-01, accuracy=8.3500e-01, log_loss=6.4412e-01, Time computing validation-score=0.448s, Total time spent=0.65min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:04:01,397 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 27/70, Epoch 3] train loss=2.71e-01, gnorm=9.12e+00, lr=6.83e-05, #samples processed=128, #sample per second=111.52. ETA=1.05min
[Iter 27/70, Epoch 3] train loss=2.71e-01, gnorm=9.12e+00, lr=6.83e-05, #samples processed=128, #sample per second=111.52. ETA=1.05min
2022-01-21 06:04:02,149 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 28/70, Epoch 3] train loss=3.83e-01, gnorm=1.79e+01, lr=6.67e-05, #samples processed=128, #sample per second=170.41. ETA=1.01min
[Iter 28/70, Epoch 3] train loss=3.83e-01, gnorm=1.79e+01, lr=6.67e-05, #samples processed=128, #sample per second=170.41. ETA=1.01min
2022-01-21 06:04:02,589 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 28/70, Epoch 3] Validation f1=8.6486e-01, mcc=6.7039e-01, roc_auc=9.1578e-01, accuracy=8.2500e-01, log_loss=6.7868e-01, Time computing validation-score=0.439s, Total time spent=0.68min. Found improved model=False, Improved top-3 models=False
[Iter 28/70, Epoch 3] Validation f1=8.6486e-01, mcc=6.7039e-01, roc_auc=9.1578e-01, accuracy=8.2500e-01, log_loss=6.7868e-01, Time computing validation-score=0.439s, Total time spent=0.68min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:04:03,372 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 29/70, Epoch 4] train loss=3.92e-01, gnorm=1.27e+01, lr=6.51e-05, #samples processed=128, #sample per second=104.78. ETA=0.98min
[Iter 29/70, Epoch 4] train loss=3.92e-01, gnorm=1.27e+01, lr=6.51e-05, #samples processed=128, #sample per second=104.78. ETA=0.98min
2022-01-21 06:04:04,129 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 30/70, Epoch 4] train loss=1.27e-01, gnorm=5.87e+00, lr=6.35e-05, #samples processed=128, #sample per second=169.24. ETA=0.94min
[Iter 30/70, Epoch 4] train loss=1.27e-01, gnorm=5.87e+00, lr=6.35e-05, #samples processed=128, #sample per second=169.24. ETA=0.94min
2022-01-21 06:04:04,575 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 30/70, Epoch 4] Validation f1=9.0749e-01, mcc=7.8615e-01, roc_auc=9.5494e-01, accuracy=8.9500e-01, log_loss=2.8031e-01, Time computing validation-score=0.445s, Total time spent=0.71min. Found improved model=False, Improved top-3 models=False
[Iter 30/70, Epoch 4] Validation f1=9.0749e-01, mcc=7.8615e-01, roc_auc=9.5494e-01, accuracy=8.9500e-01, log_loss=2.8031e-01, Time computing validation-score=0.445s, Total time spent=0.71min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:04:05,367 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 31/70, Epoch 4] train loss=7.44e-02, gnorm=1.43e+00, lr=6.19e-05, #samples processed=128, #sample per second=103.52. ETA=0.91min
[Iter 31/70, Epoch 4] train loss=7.44e-02, gnorm=1.43e+00, lr=6.19e-05, #samples processed=128, #sample per second=103.52. ETA=0.91min
2022-01-21 06:04:06,123 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 32/70, Epoch 4] train loss=2.50e-01, gnorm=3.53e+00, lr=6.03e-05, #samples processed=128, #sample per second=169.63. ETA=0.88min
[Iter 32/70, Epoch 4] train loss=2.50e-01, gnorm=3.53e+00, lr=6.03e-05, #samples processed=128, #sample per second=169.63. ETA=0.88min
2022-01-21 06:04:06,567 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 32/70, Epoch 4] Validation f1=8.9908e-01, mcc=7.8110e-01, roc_auc=9.6175e-01, accuracy=8.9000e-01, log_loss=2.7385e-01, Time computing validation-score=0.443s, Total time spent=0.75min. Found improved model=False, Improved top-3 models=False
[Iter 32/70, Epoch 4] Validation f1=8.9908e-01, mcc=7.8110e-01, roc_auc=9.6175e-01, accuracy=8.9000e-01, log_loss=2.7385e-01, Time computing validation-score=0.443s, Total time spent=0.75min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:04:07,335 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 33/70, Epoch 4] train loss=9.06e-02, gnorm=2.33e+00, lr=5.87e-05, #samples processed=128, #sample per second=105.68. ETA=0.85min
[Iter 33/70, Epoch 4] train loss=9.06e-02, gnorm=2.33e+00, lr=5.87e-05, #samples processed=128, #sample per second=105.68. ETA=0.85min
2022-01-21 06:04:08,111 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 34/70, Epoch 4] train loss=4.41e-02, gnorm=8.74e-01, lr=5.71e-05, #samples processed=128, #sample per second=165.19. ETA=0.82min
[Iter 34/70, Epoch 4] train loss=4.41e-02, gnorm=8.74e-01, lr=5.71e-05, #samples processed=128, #sample per second=165.19. ETA=0.82min
2022-01-21 06:04:08,554 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 34/70, Epoch 4] Validation f1=9.0833e-01, mcc=7.8025e-01, roc_auc=9.4955e-01, accuracy=8.9000e-01, log_loss=4.2719e-01, Time computing validation-score=0.441s, Total time spent=0.78min. Found improved model=False, Improved top-3 models=False
[Iter 34/70, Epoch 4] Validation f1=9.0833e-01, mcc=7.8025e-01, roc_auc=9.4955e-01, accuracy=8.9000e-01, log_loss=4.2719e-01, Time computing validation-score=0.441s, Total time spent=0.78min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:04:09,312 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 35/70, Epoch 4] train loss=1.32e-01, gnorm=3.59e+00, lr=5.56e-05, #samples processed=128, #sample per second=106.65. ETA=0.79min
[Iter 35/70, Epoch 4] train loss=1.32e-01, gnorm=3.59e+00, lr=5.56e-05, #samples processed=128, #sample per second=106.65. ETA=0.79min
2022-01-21 06:04:10,062 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 36/70, Epoch 5] train loss=3.74e-02, gnorm=1.64e+00, lr=5.40e-05, #samples processed=128, #sample per second=170.91. ETA=0.76min
[Iter 36/70, Epoch 5] train loss=3.74e-02, gnorm=1.64e+00, lr=5.40e-05, #samples processed=128, #sample per second=170.91. ETA=0.76min
2022-01-21 06:04:10,507 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 36/70, Epoch 5] Validation f1=9.0909e-01, mcc=7.8227e-01, roc_auc=9.4914e-01, accuracy=8.9000e-01, log_loss=5.1023e-01, Time computing validation-score=0.443s, Total time spent=0.81min. Found improved model=False, Improved top-3 models=False
[Iter 36/70, Epoch 5] Validation f1=9.0909e-01, mcc=7.8227e-01, roc_auc=9.4914e-01, accuracy=8.9000e-01, log_loss=5.1023e-01, Time computing validation-score=0.443s, Total time spent=0.81min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:04:11,284 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 37/70, Epoch 5] train loss=5.53e-02, gnorm=2.08e+00, lr=5.24e-05, #samples processed=128, #sample per second=104.82. ETA=0.74min
[Iter 37/70, Epoch 5] train loss=5.53e-02, gnorm=2.08e+00, lr=5.24e-05, #samples processed=128, #sample per second=104.82. ETA=0.74min
2022-01-21 06:04:11,978 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 38/70, Epoch 5] train loss=8.88e-02, gnorm=5.91e+00, lr=5.08e-05, #samples processed=128, #sample per second=184.90. ETA=0.70min
[Iter 38/70, Epoch 5] train loss=8.88e-02, gnorm=5.91e+00, lr=5.08e-05, #samples processed=128, #sample per second=184.90. ETA=0.70min
2022-01-21 06:04:13,943 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 38/70, Epoch 5] Validation f1=9.2373e-01, mcc=8.1881e-01, roc_auc=9.5535e-01, accuracy=9.1000e-01, log_loss=4.3876e-01, Time computing validation-score=0.446s, Total time spent=0.87min. Found improved model=True, Improved top-3 models=True
[Iter 38/70, Epoch 5] Validation f1=9.2373e-01, mcc=8.1881e-01, roc_auc=9.5535e-01, accuracy=9.1000e-01, log_loss=4.3876e-01, Time computing validation-score=0.446s, Total time spent=0.87min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:04:14,667 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 39/70, Epoch 5] train loss=1.20e-01, gnorm=9.89e+00, lr=4.92e-05, #samples processed=128, #sample per second=47.61. ETA=0.70min
[Iter 39/70, Epoch 5] train loss=1.20e-01, gnorm=9.89e+00, lr=4.92e-05, #samples processed=128, #sample per second=47.61. ETA=0.70min
2022-01-21 06:04:15,430 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 40/70, Epoch 5] train loss=4.56e-02, gnorm=2.21e+00, lr=4.76e-05, #samples processed=128, #sample per second=168.11. ETA=0.67min
[Iter 40/70, Epoch 5] train loss=4.56e-02, gnorm=2.21e+00, lr=4.76e-05, #samples processed=128, #sample per second=168.11. ETA=0.67min
2022-01-21 06:04:17,379 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 40/70, Epoch 5] Validation f1=9.2308e-01, mcc=8.1772e-01, roc_auc=9.5585e-01, accuracy=9.1000e-01, log_loss=4.3021e-01, Time computing validation-score=0.442s, Total time spent=0.93min. Found improved model=True, Improved top-3 models=True
[Iter 40/70, Epoch 5] Validation f1=9.2308e-01, mcc=8.1772e-01, roc_auc=9.5585e-01, accuracy=9.1000e-01, log_loss=4.3021e-01, Time computing validation-score=0.442s, Total time spent=0.93min. Found improved model=True, Improved top-3 models=True
Training completed. Auto-saving to "./ag_sst/". For loading the model, you can use predictor = TextPredictor.load("./ag_sst/")
<autogluon.text.text_prediction.predictor.predictor.TextPredictor at 0x7f9dacdf5ee0>
Above we specify that: the column named label contains the label values to predict, AutoGluon should optimize its predictions for the accuracy evaluation metric, trained models should be saved in the ag_sst folder, and training should run for around 60 seconds.
Evaluation¶
After training, we can easily evaluate our predictor on separate test data formatted similarly to our training data.
test_score = predictor.evaluate(test_data)
print('Accuracy = {:.2f}%'.format(test_score * 100))
Accuracy = 87.39%
By default, evaluate() will report the evaluation metric previously
specified, which is accuracy in our example. You may also specify
additional metrics, e.g. F1 score, when calling evaluate.
test_score = predictor.evaluate(test_data, metrics=['acc', 'f1'])
print(test_score)
{'acc': 0.8738532110091743, 'f1': 0.8817204301075268}
Prediction¶
And you can easily obtain predictions from these models by calling
predictor.predict().
sentence1 = "it's a charming and often affecting journey."
sentence2 = "It's slow, very, very, very slow."
predictions = predictor.predict({'sentence': [sentence1, sentence2]})
print('"Sentence":', sentence1, '"Predicted Sentiment":', predictions[0])
print('"Sentence":', sentence2, '"Predicted Sentiment":', predictions[1])
"Sentence": it's a charming and often affecting journey. "Predicted Sentiment": 1
"Sentence": It's slow, very, very, very slow. "Predicted Sentiment": 0
For classification tasks, you can ask for predicted class-probabilities instead of predicted classes.
probs = predictor.predict_proba({'sentence': [sentence1, sentence2]})
print('"Sentence":', sentence1, '"Predicted Class-Probabilities":', probs[0])
print('"Sentence":', sentence2, '"Predicted Class-Probabilities":', probs[1])
"Sentence": it's a charming and often affecting journey. "Predicted Class-Probabilities": 0 0.000754
1 0.976570
Name: 0, dtype: float32
"Sentence": It's slow, very, very, very slow. "Predicted Class-Probabilities": 0 0.999246
1 0.023430
Name: 1, dtype: float32
We can just as easily produce predictions over an entire dataset.
test_predictions = predictor.predict(test_data)
test_predictions.head()
0 1
1 0
2 1
3 1
4 0
Name: label, dtype: int64
Intermediate Training Results¶
After training, you can explore intermediate training results in
predictor.results.
predictor.results.tail(3)
| iteration | report_idx | epoch | f1 | mcc | roc_auc | accuracy | log_loss | find_better | find_new_topn | nbest_stat | elapsed_time | reward_attr | eval_metric | exp_dir | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 18 | 38 | 19 | 5 | 0.923729 | 0.818814 | 0.955345 | 0.91 | 0.438755 | True | True | [[0.905, 0.91, 0.91], [18, 16, 38]] | 52 | 0.91 | accuracy | /var/lib/jenkins/workspace/workspace/autogluon... |
| 19 | 40 | 20 | 5 | 0.923077 | 0.817718 | 0.955854 | 0.91 | 0.430206 | True | True | [[0.91, 0.91, 0.91], [40, 16, 38]] | 55 | 0.91 | accuracy | /var/lib/jenkins/workspace/workspace/autogluon... |
| 20 | 40 | 21 | 5 | 0.923077 | 0.817718 | 0.955854 | 0.91 | 0.430206 | True | True | [[0.91, 0.91, 0.91], [40, 16, 38]] | 55 | 0.91 | accuracy | /var/lib/jenkins/workspace/workspace/autogluon... |
Save and Load¶
The trained predictor is automatically saved at the end of fit(),
and you can easily reload it.
loaded_predictor = TextPredictor.load('ag_sst')
loaded_predictor.predict_proba({'sentence': [sentence1, sentence2]})
| 0 | 1 | |
|---|---|---|
| 0 | 0.000754 | 0.999246 |
| 1 | 0.976570 | 0.023430 |
You can also save the predictor to any location by calling .save().
loaded_predictor.save('my_saved_dir')
loaded_predictor2 = TextPredictor.load('my_saved_dir')
loaded_predictor2.predict_proba({'sentence': [sentence1, sentence2]})
| 0 | 1 | |
|---|---|---|
| 0 | 0.000754 | 0.999246 |
| 1 | 0.976570 | 0.023430 |
Extract Embeddings¶
You can also use a trained predictor to extract embeddings that maps each row of the data table to an embedding vector extracted from intermediate neural network representations of the row.
embeddings = predictor.extract_embedding(test_data)
print(embeddings)
[[ 0.21358123 -0.44993764 -0.300163 ... -0.22978368 0.29806402
0.22377433]
[-0.24794428 0.3881194 -0.39479852 ... 0.34626406 0.57664824
0.3822652 ]
[ 0.13004313 -0.47476366 -0.18469483 ... -0.20439021 0.3409003
0.24067077]
...
[-0.02681284 0.10309694 -0.38451737 ... 0.03428023 0.09695421
0.8873183 ]
[ 0.21095227 0.21165359 -0.65412647 ... 0.1764061 0.4770543
0.49337402]
[ 0.114441 -0.5424684 -0.16822033 ... -0.19437121 0.21894938
0.12769817]]
Here, we use TSNE to visualize these extracted embeddings. We can see that there are two clusters corresponding to our two labels, since this network has been trained to discriminate between these labels.
from sklearn.manifold import TSNE
X_embedded = TSNE(n_components=2, random_state=123).fit_transform(embeddings)
for val, color in [(0, 'red'), (1, 'blue')]:
idx = (test_data['label'].to_numpy() == val).nonzero()
plt.scatter(X_embedded[idx, 0], X_embedded[idx, 1], c=color, label=f'label={val}')
plt.legend(loc='best')
<matplotlib.legend.Legend at 0x7f9cfe959130>
Continuous Training¶
You can also load a predictor and call .fit() again to continue
training the same predictor with new data.
new_predictor = TextPredictor.load('ag_sst')
new_predictor.fit(train_data, time_limit=30, save_path='ag_sst_continue_train')
test_score = new_predictor.evaluate(test_data, metrics=['acc', 'f1'])
print(test_score)
Continue training the existing model...
2022-01-21 06:04:41,004 - autogluon.text.text_prediction.mx.models - INFO - The GluonNLP V0 backend is used. We will use 8 cpus and 1 gpus to train each trial.
The GluonNLP V0 backend is used. We will use 8 cpus and 1 gpus to train each trial.
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/autonlp/task0/training.log
2022-01-21 06:04:42,297 - autogluon.text.text_prediction.mx.models - INFO - Done! Preprocessor saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/autonlp/task0/preprocessor.pkl
Done! Preprocessor saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/autonlp/task0/preprocessor.pkl
2022-01-21 06:04:42,300 - autogluon.text.text_prediction.mx.models - INFO - Process dev set...
Process dev set...
2022-01-21 06:04:42,311 - autogluon.text.text_prediction.mx.models - INFO - Done!
Done!
2022-01-21 06:04:42,320 - autogluon.text.text_prediction.mx.models - INFO - Max length for chunking text: 64, Stochastic chunk: Train-False/Test-False, Test #repeat: 1.
Max length for chunking text: 64, Stochastic chunk: Train-False/Test-False, Test #repeat: 1.
2022-01-21 06:04:42,500 - autogluon.text.text_prediction.mx.models - INFO - #Total Params/Fixed Params=108990466/0
#Total Params/Fixed Params=108990466/0
2022-01-21 06:04:42,513 - autogluon.text.text_prediction.mx.models - Level 15 - Using gradient accumulation. Global batch size = 128
Using gradient accumulation. Global batch size = 128
2022-01-21 06:04:42,567 - autogluon.text.text_prediction.mx.models - INFO - Local training results will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/autonlp/task0/results_local.jsonl.
Local training results will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/autonlp/task0/results_local.jsonl.
2022-01-21 06:04:43,550 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 1/70, Epoch 0] train loss=1.61e-01, gnorm=2.58e+00, lr=1.43e-05, #samples processed=128, #sample per second=128.06. ETA=1.13min
[Iter 1/70, Epoch 0] train loss=1.61e-01, gnorm=2.58e+00, lr=1.43e-05, #samples processed=128, #sample per second=128.06. ETA=1.13min
2022-01-21 06:04:44,332 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 2/70, Epoch 0] train loss=8.47e-02, gnorm=2.64e+00, lr=2.86e-05, #samples processed=128, #sample per second=163.95. ETA=1.00min
[Iter 2/70, Epoch 0] train loss=8.47e-02, gnorm=2.64e+00, lr=2.86e-05, #samples processed=128, #sample per second=163.95. ETA=1.00min
2022-01-21 06:04:45,610 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 2/70, Epoch 0] Validation f1=9.2035e-01, mcc=8.1691e-01, roc_auc=9.6236e-01, accuracy=9.1000e-01, log_loss=2.9213e-01, Time computing validation-score=0.450s, Total time spent=0.05min. Found improved model=True, Improved top-3 models=True
[Iter 2/70, Epoch 0] Validation f1=9.2035e-01, mcc=8.1691e-01, roc_auc=9.6236e-01, accuracy=9.1000e-01, log_loss=2.9213e-01, Time computing validation-score=0.450s, Total time spent=0.05min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:04:46,384 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 3/70, Epoch 0] train loss=2.64e-02, gnorm=1.38e+00, lr=4.29e-05, #samples processed=128, #sample per second=62.40. ETA=1.42min
[Iter 3/70, Epoch 0] train loss=2.64e-02, gnorm=1.38e+00, lr=4.29e-05, #samples processed=128, #sample per second=62.40. ETA=1.42min
2022-01-21 06:04:47,103 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 4/70, Epoch 0] train loss=5.82e-02, gnorm=2.79e+00, lr=5.71e-05, #samples processed=128, #sample per second=178.27. ETA=1.25min
[Iter 4/70, Epoch 0] train loss=5.82e-02, gnorm=2.79e+00, lr=5.71e-05, #samples processed=128, #sample per second=178.27. ETA=1.25min
2022-01-21 06:04:48,009 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 4/70, Epoch 0] Validation f1=8.9879e-01, mcc=7.5693e-01, roc_auc=9.3073e-01, accuracy=8.7500e-01, log_loss=6.2100e-01, Time computing validation-score=0.447s, Total time spent=0.09min. Found improved model=False, Improved top-3 models=True
[Iter 4/70, Epoch 0] Validation f1=8.9879e-01, mcc=7.5693e-01, roc_auc=9.3073e-01, accuracy=8.7500e-01, log_loss=6.2100e-01, Time computing validation-score=0.447s, Total time spent=0.09min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:04:48,746 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 5/70, Epoch 0] train loss=1.16e-01, gnorm=3.74e+00, lr=7.14e-05, #samples processed=128, #sample per second=77.97. ETA=1.34min
[Iter 5/70, Epoch 0] train loss=1.16e-01, gnorm=3.74e+00, lr=7.14e-05, #samples processed=128, #sample per second=77.97. ETA=1.34min
2022-01-21 06:04:49,499 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 6/70, Epoch 0] train loss=1.77e-01, gnorm=6.17e+00, lr=8.57e-05, #samples processed=128, #sample per second=170.24. ETA=1.23min
[Iter 6/70, Epoch 0] train loss=1.77e-01, gnorm=6.17e+00, lr=8.57e-05, #samples processed=128, #sample per second=170.24. ETA=1.23min
2022-01-21 06:04:50,391 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 6/70, Epoch 0] Validation f1=8.8235e-01, mcc=7.1612e-01, roc_auc=9.2768e-01, accuracy=8.6000e-01, log_loss=4.7454e-01, Time computing validation-score=0.443s, Total time spent=0.13min. Found improved model=False, Improved top-3 models=True
[Iter 6/70, Epoch 0] Validation f1=8.8235e-01, mcc=7.1612e-01, roc_auc=9.2768e-01, accuracy=8.6000e-01, log_loss=4.7454e-01, Time computing validation-score=0.443s, Total time spent=0.13min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:04:51,180 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 7/70, Epoch 0] train loss=6.82e-02, gnorm=2.97e+00, lr=1.00e-04, #samples processed=128, #sample per second=76.22. ETA=1.29min
[Iter 7/70, Epoch 0] train loss=6.82e-02, gnorm=2.97e+00, lr=1.00e-04, #samples processed=128, #sample per second=76.22. ETA=1.29min
2022-01-21 06:04:51,988 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 8/70, Epoch 1] train loss=8.84e-02, gnorm=2.93e+00, lr=9.84e-05, #samples processed=128, #sample per second=158.56. ETA=1.22min
[Iter 8/70, Epoch 1] train loss=8.84e-02, gnorm=2.93e+00, lr=9.84e-05, #samples processed=128, #sample per second=158.56. ETA=1.22min
2022-01-21 06:04:53,209 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 8/70, Epoch 1] Validation f1=8.8793e-01, mcc=7.3481e-01, roc_auc=9.3368e-01, accuracy=8.7000e-01, log_loss=4.3328e-01, Time computing validation-score=0.445s, Total time spent=0.18min. Found improved model=False, Improved top-3 models=True
[Iter 8/70, Epoch 1] Validation f1=8.8793e-01, mcc=7.3481e-01, roc_auc=9.3368e-01, accuracy=8.7000e-01, log_loss=4.3328e-01, Time computing validation-score=0.445s, Total time spent=0.18min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:04:53,969 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 9/70, Epoch 1] train loss=7.32e-02, gnorm=4.32e+00, lr=9.68e-05, #samples processed=128, #sample per second=64.67. ETA=1.29min
[Iter 9/70, Epoch 1] train loss=7.32e-02, gnorm=4.32e+00, lr=9.68e-05, #samples processed=128, #sample per second=64.67. ETA=1.29min
2022-01-21 06:04:54,718 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 10/70, Epoch 1] train loss=3.89e-01, gnorm=1.82e+01, lr=9.52e-05, #samples processed=128, #sample per second=171.12. ETA=1.22min
[Iter 10/70, Epoch 1] train loss=3.89e-01, gnorm=1.82e+01, lr=9.52e-05, #samples processed=128, #sample per second=171.12. ETA=1.22min
2022-01-21 06:04:55,167 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 10/70, Epoch 1] Validation f1=8.5271e-01, mcc=6.3414e-01, roc_auc=9.0154e-01, accuracy=8.1000e-01, log_loss=1.0126e+00, Time computing validation-score=0.448s, Total time spent=0.21min. Found improved model=False, Improved top-3 models=False
[Iter 10/70, Epoch 1] Validation f1=8.5271e-01, mcc=6.3414e-01, roc_auc=9.0154e-01, accuracy=8.1000e-01, log_loss=1.0126e+00, Time computing validation-score=0.448s, Total time spent=0.21min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:04:55,912 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 11/70, Epoch 1] train loss=2.30e-01, gnorm=1.59e+01, lr=9.37e-05, #samples processed=128, #sample per second=107.27. ETA=1.19min
[Iter 11/70, Epoch 1] train loss=2.30e-01, gnorm=1.59e+01, lr=9.37e-05, #samples processed=128, #sample per second=107.27. ETA=1.19min
2022-01-21 06:04:56,619 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 12/70, Epoch 1] train loss=1.26e-01, gnorm=9.30e+00, lr=9.21e-05, #samples processed=128, #sample per second=181.44. ETA=1.13min
[Iter 12/70, Epoch 1] train loss=1.26e-01, gnorm=9.30e+00, lr=9.21e-05, #samples processed=128, #sample per second=181.44. ETA=1.13min
2022-01-21 06:04:57,071 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 12/70, Epoch 1] Validation f1=8.2653e-01, mcc=6.9810e-01, roc_auc=9.4395e-01, accuracy=8.3000e-01, log_loss=5.0747e-01, Time computing validation-score=0.450s, Total time spent=0.24min. Found improved model=False, Improved top-3 models=False
[Iter 12/70, Epoch 1] Validation f1=8.2653e-01, mcc=6.9810e-01, roc_auc=9.4395e-01, accuracy=8.3000e-01, log_loss=5.0747e-01, Time computing validation-score=0.450s, Total time spent=0.24min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:04:57,884 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 13/70, Epoch 1] train loss=3.39e-01, gnorm=1.68e+01, lr=9.05e-05, #samples processed=128, #sample per second=101.25. ETA=1.12min
[Iter 13/70, Epoch 1] train loss=3.39e-01, gnorm=1.68e+01, lr=9.05e-05, #samples processed=128, #sample per second=101.25. ETA=1.12min
2022-01-21 06:04:58,635 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 14/70, Epoch 1] train loss=2.09e-01, gnorm=8.59e+00, lr=8.89e-05, #samples processed=128, #sample per second=170.76. ETA=1.07min
[Iter 14/70, Epoch 1] train loss=2.09e-01, gnorm=8.59e+00, lr=8.89e-05, #samples processed=128, #sample per second=170.76. ETA=1.07min
2022-01-21 06:04:59,844 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 14/70, Epoch 1] Validation f1=9.1630e-01, mcc=8.0652e-01, roc_auc=9.5412e-01, accuracy=9.0500e-01, log_loss=2.8884e-01, Time computing validation-score=0.445s, Total time spent=0.29min. Found improved model=False, Improved top-3 models=True
[Iter 14/70, Epoch 1] Validation f1=9.1630e-01, mcc=8.0652e-01, roc_auc=9.5412e-01, accuracy=9.0500e-01, log_loss=2.8884e-01, Time computing validation-score=0.445s, Total time spent=0.29min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:05:00,606 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 15/70, Epoch 2] train loss=1.56e-01, gnorm=3.86e+00, lr=8.73e-05, #samples processed=128, #sample per second=64.99. ETA=1.10min
[Iter 15/70, Epoch 2] train loss=1.56e-01, gnorm=3.86e+00, lr=8.73e-05, #samples processed=128, #sample per second=64.99. ETA=1.10min
2022-01-21 06:05:01,403 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 16/70, Epoch 2] train loss=1.18e-01, gnorm=3.08e+00, lr=8.57e-05, #samples processed=128, #sample per second=160.91. ETA=1.06min
[Iter 16/70, Epoch 2] train loss=1.18e-01, gnorm=3.08e+00, lr=8.57e-05, #samples processed=128, #sample per second=160.91. ETA=1.06min
2022-01-21 06:05:01,858 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 16/70, Epoch 2] Validation f1=8.9431e-01, mcc=7.4479e-01, roc_auc=9.4324e-01, accuracy=8.7000e-01, log_loss=6.5043e-01, Time computing validation-score=0.454s, Total time spent=0.32min. Found improved model=False, Improved top-3 models=False
[Iter 16/70, Epoch 2] Validation f1=8.9431e-01, mcc=7.4479e-01, roc_auc=9.4324e-01, accuracy=8.7000e-01, log_loss=6.5043e-01, Time computing validation-score=0.454s, Total time spent=0.32min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:05:02,619 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 17/70, Epoch 2] train loss=7.71e-02, gnorm=3.61e+00, lr=8.41e-05, #samples processed=128, #sample per second=105.36. ETA=1.04min
[Iter 17/70, Epoch 2] train loss=7.71e-02, gnorm=3.61e+00, lr=8.41e-05, #samples processed=128, #sample per second=105.36. ETA=1.04min
2022-01-21 06:05:03,388 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 18/70, Epoch 2] train loss=3.71e-02, gnorm=9.83e+00, lr=8.25e-05, #samples processed=128, #sample per second=166.71. ETA=1.00min
[Iter 18/70, Epoch 2] train loss=3.71e-02, gnorm=9.83e+00, lr=8.25e-05, #samples processed=128, #sample per second=166.71. ETA=1.00min
2022-01-21 06:05:04,565 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 18/70, Epoch 2] Validation f1=8.9655e-01, mcc=7.5535e-01, roc_auc=9.5026e-01, accuracy=8.8000e-01, log_loss=5.5848e-01, Time computing validation-score=0.446s, Total time spent=0.37min. Found improved model=False, Improved top-3 models=True
[Iter 18/70, Epoch 2] Validation f1=8.9655e-01, mcc=7.5535e-01, roc_auc=9.5026e-01, accuracy=8.8000e-01, log_loss=5.5848e-01, Time computing validation-score=0.446s, Total time spent=0.37min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:05:05,309 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 19/70, Epoch 2] train loss=1.30e-01, gnorm=1.93e+00, lr=8.10e-05, #samples processed=128, #sample per second=66.65. ETA=1.02min
[Iter 19/70, Epoch 2] train loss=1.30e-01, gnorm=1.93e+00, lr=8.10e-05, #samples processed=128, #sample per second=66.65. ETA=1.02min
2022-01-21 06:05:06,071 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 20/70, Epoch 2] train loss=1.03e-01, gnorm=4.32e+00, lr=7.94e-05, #samples processed=128, #sample per second=168.29. ETA=0.98min
[Iter 20/70, Epoch 2] train loss=1.03e-01, gnorm=4.32e+00, lr=7.94e-05, #samples processed=128, #sample per second=168.29. ETA=0.98min
2022-01-21 06:05:06,520 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 20/70, Epoch 2] Validation f1=8.8889e-01, mcc=7.3056e-01, roc_auc=9.4721e-01, accuracy=8.6500e-01, log_loss=8.0116e-01, Time computing validation-score=0.447s, Total time spent=0.40min. Found improved model=False, Improved top-3 models=False
[Iter 20/70, Epoch 2] Validation f1=8.8889e-01, mcc=7.3056e-01, roc_auc=9.4721e-01, accuracy=8.6500e-01, log_loss=8.0116e-01, Time computing validation-score=0.447s, Total time spent=0.40min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:05:07,239 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 21/70, Epoch 2] train loss=4.03e-02, gnorm=5.12e+00, lr=7.78e-05, #samples processed=128, #sample per second=109.70. ETA=0.96min
[Iter 21/70, Epoch 2] train loss=4.03e-02, gnorm=5.12e+00, lr=7.78e-05, #samples processed=128, #sample per second=109.70. ETA=0.96min
2022-01-21 06:05:08,043 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 22/70, Epoch 3] train loss=5.84e-02, gnorm=5.62e+00, lr=7.62e-05, #samples processed=128, #sample per second=159.54. ETA=0.93min
[Iter 22/70, Epoch 3] train loss=5.84e-02, gnorm=5.62e+00, lr=7.62e-05, #samples processed=128, #sample per second=159.54. ETA=0.93min
2022-01-21 06:05:09,234 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 22/70, Epoch 3] Validation f1=9.1139e-01, mcc=7.8834e-01, roc_auc=9.5016e-01, accuracy=8.9500e-01, log_loss=6.0883e-01, Time computing validation-score=0.449s, Total time spent=0.44min. Found improved model=False, Improved top-3 models=True
[Iter 22/70, Epoch 3] Validation f1=9.1139e-01, mcc=7.8834e-01, roc_auc=9.5016e-01, accuracy=8.9500e-01, log_loss=6.0883e-01, Time computing validation-score=0.449s, Total time spent=0.44min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:05:09,981 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 23/70, Epoch 3] train loss=3.03e-02, gnorm=3.67e+00, lr=7.46e-05, #samples processed=128, #sample per second=66.10. ETA=0.93min
[Iter 23/70, Epoch 3] train loss=3.03e-02, gnorm=3.67e+00, lr=7.46e-05, #samples processed=128, #sample per second=66.10. ETA=0.93min
2022-01-21 06:05:10,768 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 24/70, Epoch 3] train loss=1.62e-01, gnorm=2.59e+00, lr=7.30e-05, #samples processed=128, #sample per second=162.69. ETA=0.90min
[Iter 24/70, Epoch 3] train loss=1.62e-01, gnorm=2.59e+00, lr=7.30e-05, #samples processed=128, #sample per second=162.69. ETA=0.90min
2022-01-21 06:05:11,920 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 24/70, Epoch 3] Validation f1=9.1556e-01, mcc=8.0703e-01, roc_auc=9.4711e-01, accuracy=9.0500e-01, log_loss=5.1787e-01, Time computing validation-score=0.449s, Total time spent=0.49min. Found improved model=False, Improved top-3 models=True
[Iter 24/70, Epoch 3] Validation f1=9.1556e-01, mcc=8.0703e-01, roc_auc=9.4711e-01, accuracy=9.0500e-01, log_loss=5.1787e-01, Time computing validation-score=0.449s, Total time spent=0.49min. Found improved model=False, Improved top-3 models=True
Training completed. Auto-saving to "ag_sst_continue_train/". For loading the model, you can use predictor = TextPredictor.load("ag_sst_continue_train/")
{'acc': 0.8990825688073395, 'f1': 0.8986175115207374}
Sentence Similarity Task¶
Next, let’s use AutoGluon to train a model for evaluating how semantically similar two sentences are. We use the Semantic Textual Similarity Benchmark dataset for illustration.
sts_train_data = TabularDataset('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sts/train.parquet')[['sentence1', 'sentence2', 'score']]
sts_test_data = TabularDataset('https://autogluon-text.s3-accelerate.amazonaws.com/glue/sts/dev.parquet')[['sentence1', 'sentence2', 'score']]
sts_train_data.head(10)
Loaded data from: https://autogluon-text.s3-accelerate.amazonaws.com/glue/sts/train.parquet | Columns = 4 / 4 | Rows = 5749 -> 5749
Loaded data from: https://autogluon-text.s3-accelerate.amazonaws.com/glue/sts/dev.parquet | Columns = 4 / 4 | Rows = 1500 -> 1500
| sentence1 | sentence2 | score | |
|---|---|---|---|
| 0 | A plane is taking off. | An air plane is taking off. | 5.00 |
| 1 | A man is playing a large flute. | A man is playing a flute. | 3.80 |
| 2 | A man is spreading shreded cheese on a pizza. | A man is spreading shredded cheese on an uncoo... | 3.80 |
| 3 | Three men are playing chess. | Two men are playing chess. | 2.60 |
| 4 | A man is playing the cello. | A man seated is playing the cello. | 4.25 |
| 5 | Some men are fighting. | Two men are fighting. | 4.25 |
| 6 | A man is smoking. | A man is skating. | 0.50 |
| 7 | The man is playing the piano. | The man is playing the guitar. | 1.60 |
| 8 | A man is playing on a guitar and singing. | A woman is playing an acoustic guitar and sing... | 2.20 |
| 9 | A person is throwing a cat on to the ceiling. | A person throws a cat on the ceiling. | 5.00 |
In this data, the column named score contains numerical values (which we’d like to predict) that are human-annotated similarity scores for each given pair of sentences.
print('Min score=', min(sts_train_data['score']), ', Max score=', max(sts_train_data['score']))
Min score= 0.0 , Max score= 5.0
Let’s train a regression model to predict these scores. Note that we
only need to specify the label column and AutoGluon automatically
determines the type of prediction problem and an appropriate loss
function. Once again, you should increase the short time_limit below
to obtain reasonable performance in your own applications.
predictor_sts = TextPredictor(label='score', path='./ag_sts')
predictor_sts.fit(sts_train_data, time_limit=60)
Problem Type="regression"
Column Types:
- "sentence1": text
- "sentence2": text
- "score": numerical
2022-01-21 06:05:19,064 - autogluon.text.text_prediction.mx.models - INFO - The GluonNLP V0 backend is used. We will use 8 cpus and 1 gpus to train each trial.
The GluonNLP V0 backend is used. We will use 8 cpus and 1 gpus to train each trial.
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sts/task0/training.log
2022-01-21 06:05:20,307 - autogluon.text.text_prediction.mx.models - INFO - Fitting and transforming the train data...
Fitting and transforming the train data...
2022-01-21 06:05:22,748 - autogluon.text.text_prediction.mx.models - INFO - Done! Preprocessor saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sts/task0/preprocessor.pkl
Done! Preprocessor saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sts/task0/preprocessor.pkl
2022-01-21 06:05:22,760 - autogluon.text.text_prediction.mx.models - INFO - Process dev set...
Process dev set...
2022-01-21 06:05:22,814 - autogluon.text.text_prediction.mx.models - INFO - Done!
Done!
2022-01-21 06:05:22,829 - autogluon.text.text_prediction.mx.models - INFO - Max length for chunking text: 128, Stochastic chunk: Train-False/Test-False, Test #repeat: 1.
Max length for chunking text: 128, Stochastic chunk: Train-False/Test-False, Test #repeat: 1.
2022-01-21 06:05:23,099 - autogluon.text.text_prediction.mx.models - INFO - #Total Params/Fixed Params=108990337/0
#Total Params/Fixed Params=108990337/0
2022-01-21 06:05:23,115 - autogluon.text.text_prediction.mx.models - Level 15 - Using gradient accumulation. Global batch size = 128
Using gradient accumulation. Global batch size = 128
2022-01-21 06:05:23,168 - autogluon.text.text_prediction.mx.models - INFO - Local training results will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sts/task0/results_local.jsonl.
Local training results will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-text-v3/docs/_build/eval/tutorials/text_prediction/ag_sts/task0/results_local.jsonl.
2022-01-21 06:05:27,101 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 3/410, Epoch 0] train loss=1.53e+00, gnorm=2.06e+01, lr=7.32e-06, #samples processed=384, #sample per second=97.25. ETA=8.89min
[Iter 3/410, Epoch 0] train loss=1.53e+00, gnorm=2.06e+01, lr=7.32e-06, #samples processed=384, #sample per second=97.25. ETA=8.89min
2022-01-21 06:05:30,826 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 6/410, Epoch 0] train loss=1.45e+00, gnorm=3.90e+01, lr=1.46e-05, #samples processed=384, #sample per second=103.11. ETA=8.59min
[Iter 6/410, Epoch 0] train loss=1.45e+00, gnorm=3.90e+01, lr=1.46e-05, #samples processed=384, #sample per second=103.11. ETA=8.59min
2022-01-21 06:05:34,617 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 9/410, Epoch 0] train loss=8.92e-01, gnorm=2.22e+01, lr=2.20e-05, #samples processed=384, #sample per second=101.33. ETA=8.50min
[Iter 9/410, Epoch 0] train loss=8.92e-01, gnorm=2.22e+01, lr=2.20e-05, #samples processed=384, #sample per second=101.33. ETA=8.50min
2022-01-21 06:05:37,784 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 9/410, Epoch 0] Validation r2=2.7861e-01, root_mean_squared_error=1.2623e+00, mean_absolute_error=1.0087e+00, Time computing validation-score=2.103s, Total time spent=0.24min. Found improved model=True, Improved top-3 models=True
[Iter 9/410, Epoch 0] Validation r2=2.7861e-01, root_mean_squared_error=1.2623e+00, mean_absolute_error=1.0087e+00, Time computing validation-score=2.103s, Total time spent=0.24min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:05:42,321 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 12/410, Epoch 0] train loss=9.76e-01, gnorm=2.92e+01, lr=2.93e-05, #samples processed=384, #sample per second=49.86. ETA=10.59min
[Iter 12/410, Epoch 0] train loss=9.76e-01, gnorm=2.92e+01, lr=2.93e-05, #samples processed=384, #sample per second=49.86. ETA=10.59min
2022-01-21 06:05:45,948 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 15/410, Epoch 0] train loss=8.27e-01, gnorm=1.87e+01, lr=3.66e-05, #samples processed=384, #sample per second=105.89. ETA=10.00min
[Iter 15/410, Epoch 0] train loss=8.27e-01, gnorm=1.87e+01, lr=3.66e-05, #samples processed=384, #sample per second=105.89. ETA=10.00min
2022-01-21 06:05:50,199 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 18/410, Epoch 0] train loss=7.53e-01, gnorm=7.57e+00, lr=4.39e-05, #samples processed=384, #sample per second=90.36. ETA=9.81min
[Iter 18/410, Epoch 0] train loss=7.53e-01, gnorm=7.57e+00, lr=4.39e-05, #samples processed=384, #sample per second=90.36. ETA=9.81min
2022-01-21 06:05:53,578 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 18/410, Epoch 0] Validation r2=4.3065e-01, root_mean_squared_error=1.1214e+00, mean_absolute_error=9.0784e-01, Time computing validation-score=2.110s, Total time spent=0.51min. Found improved model=True, Improved top-3 models=True
[Iter 18/410, Epoch 0] Validation r2=4.3065e-01, root_mean_squared_error=1.1214e+00, mean_absolute_error=9.0784e-01, Time computing validation-score=2.110s, Total time spent=0.51min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:05:57,128 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 21/410, Epoch 0] train loss=5.94e-01, gnorm=1.25e+01, lr=5.12e-05, #samples processed=384, #sample per second=55.43. ETA=10.48min
[Iter 21/410, Epoch 0] train loss=5.94e-01, gnorm=1.25e+01, lr=5.12e-05, #samples processed=384, #sample per second=55.43. ETA=10.48min
2022-01-21 06:06:01,188 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 24/410, Epoch 0] train loss=4.46e-01, gnorm=5.99e+00, lr=5.85e-05, #samples processed=384, #sample per second=94.60. ETA=10.19min
[Iter 24/410, Epoch 0] train loss=4.46e-01, gnorm=5.99e+00, lr=5.85e-05, #samples processed=384, #sample per second=94.60. ETA=10.19min
2022-01-21 06:06:04,666 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 27/410, Epoch 0] train loss=4.10e-01, gnorm=3.17e+00, lr=6.59e-05, #samples processed=384, #sample per second=110.45. ETA=9.81min
[Iter 27/410, Epoch 0] train loss=4.10e-01, gnorm=3.17e+00, lr=6.59e-05, #samples processed=384, #sample per second=110.45. ETA=9.81min
2022-01-21 06:06:08,268 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 27/410, Epoch 0] Validation r2=6.3244e-01, root_mean_squared_error=9.0103e-01, mean_absolute_error=7.0495e-01, Time computing validation-score=2.072s, Total time spent=0.75min. Found improved model=True, Improved top-3 models=True
[Iter 27/410, Epoch 0] Validation r2=6.3244e-01, root_mean_squared_error=9.0103e-01, mean_absolute_error=7.0495e-01, Time computing validation-score=2.072s, Total time spent=0.75min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:06:12,316 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 30/410, Epoch 0] train loss=3.37e-01, gnorm=4.28e+00, lr=7.32e-05, #samples processed=384, #sample per second=50.21. ETA=10.38min
[Iter 30/410, Epoch 0] train loss=3.37e-01, gnorm=4.28e+00, lr=7.32e-05, #samples processed=384, #sample per second=50.21. ETA=10.38min
2022-01-21 06:06:15,908 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 33/410, Epoch 0] train loss=3.99e-01, gnorm=7.30e+00, lr=8.05e-05, #samples processed=384, #sample per second=106.92. ETA=10.04min
[Iter 33/410, Epoch 0] train loss=3.99e-01, gnorm=7.30e+00, lr=8.05e-05, #samples processed=384, #sample per second=106.92. ETA=10.04min
2022-01-21 06:06:20,175 - autogluon.text.text_prediction.mx.models - Level 15 - [Iter 36/410, Epoch 0] train loss=3.61e-01, gnorm=5.28e+00, lr=8.78e-05, #samples processed=384, #sample per second=90.03. ETA=9.87min
[Iter 36/410, Epoch 0] train loss=3.61e-01, gnorm=5.28e+00, lr=8.78e-05, #samples processed=384, #sample per second=90.03. ETA=9.87min
2022-01-21 06:06:23,806 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 36/410, Epoch 0] Validation r2=6.8104e-01, root_mean_squared_error=8.3934e-01, mean_absolute_error=6.7415e-01, Time computing validation-score=2.061s, Total time spent=1.01min. Found improved model=True, Improved top-3 models=True
[Iter 36/410, Epoch 0] Validation r2=6.8104e-01, root_mean_squared_error=8.3934e-01, mean_absolute_error=6.7415e-01, Time computing validation-score=2.061s, Total time spent=1.01min. Found improved model=True, Improved top-3 models=True
Training completed. Auto-saving to "./ag_sts/". For loading the model, you can use predictor = TextPredictor.load("./ag_sts/")
<autogluon.text.text_prediction.predictor.predictor.TextPredictor at 0x7f9cf05a4d30>
We again evaluate our trained model’s performance on separate test data. Below we choose to compute the following metrics: RMSE, Pearson Correlation, and Spearman Correlation.
test_score = predictor_sts.evaluate(sts_test_data, metrics=['rmse', 'pearsonr', 'spearmanr'])
print('RMSE = {:.2f}'.format(test_score['rmse']))
print('PEARSONR = {:.4f}'.format(test_score['pearsonr']))
print('SPEARMANR = {:.4f}'.format(test_score['spearmanr']))
RMSE = 0.71
PEARSONR = 0.8840
SPEARMANR = 0.8835
Let’s use our model to predict the similarity score between a few sentences.
sentences = ['The child is riding a horse.',
'The young boy is riding a horse.',
'The young man is riding a horse.',
'The young man is riding a bicycle.']
score1 = predictor_sts.predict({'sentence1': [sentences[0]],
'sentence2': [sentences[1]]}, as_pandas=False)
score2 = predictor_sts.predict({'sentence1': [sentences[0]],
'sentence2': [sentences[2]]}, as_pandas=False)
score3 = predictor_sts.predict({'sentence1': [sentences[0]],
'sentence2': [sentences[3]]}, as_pandas=False)
print(score1, score2, score3)
[4.2140903] [2.9765248] [0.91105884]
Although the TextPredictor is only designed for classification and
regression tasks, it can directly be used for many NLP tasks if you
properly format them into a data table. Note that there can be many text
columns in this data table. Refer to the TextPredictor
documentation
to see all of the available methods/options.
Unlike TabularPredictor which trains/ensembles many different kinds
of models, TextPredictor fits only Transformer neural network
models. These are fit to your data via transfer learning from pretrained
NLP models like: BERT,
ALBERT, and
ELECTRA.
TextPredictor also enables training on multi-modal data tables that
contain text, numeric and categorical columns, and the neural network
hyperparameter can be automatically tuned with Hyperparameter
Optimization (HPO), which will be introduced in the other tutorials.
Note: TextPredictor depends on the
GluonNLP package. Due to an ongoing
upgrade of GluonNLP, we are currently using a custom version of the
package:
autogluon-contrib-nlp.
In a future release, AutoGluon will support the official GluonNLP 1.0,
but the APIs demonstrated here will remain the same.