Multimodal Data Tables: Combining BERT/Transformers and Classical Tabular Models¶
Tip: If your data contains images, consider also checking out Multimodal Data Tables: Tabular, Text, and Image which handles images in addition to text and tabular features.
Here we introduce how to use AutoGluon Tabular to deal with multimodal
tabular data that contains text, numeric, and categorical columns. In
AutoGluon, raw text data is considered as a first-class citizen of
data tables. AutoGluon Tabular can help you train and combine a diverse
set of models including classical tabular models like
LightGBM/RF/CatBoost as well as our pretrained NLP model based
multimodal network that is introduced in Section
“What’s happening inside?” of
Text Prediction - Multimodal Table with Text (used by AutoGluon’s
TextPredictor).
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import pprint
import random
from autogluon.tabular import TabularPredictor
import mxnet as mx
np.random.seed(123)
random.seed(123)
mx.random.seed(123)
Product Sentiment Analysis Dataset¶
We consider the product sentiment analysis dataset from a MachineHack hackathon. The goal is to predict a user’s sentiment towards a product given their review (raw text) and a categorical feature indicating the product’s type (e.g., Tablet, Mobile, etc.). We have already split the original dataset to be 90% for training and 10% for development/testing (if submitting your models to the hackathon, we recommend training them on 100% of the dataset).
!mkdir -p product_sentiment_machine_hack
!wget https://autogluon-text-data.s3.amazonaws.com/multimodal_text/machine_hack_product_sentiment/train.csv -O product_sentiment_machine_hack/train.csv
!wget https://autogluon-text-data.s3.amazonaws.com/multimodal_text/machine_hack_product_sentiment/dev.csv -O product_sentiment_machine_hack/dev.csv
!wget https://autogluon-text-data.s3.amazonaws.com/multimodal_text/machine_hack_product_sentiment/test.csv -O product_sentiment_machine_hack/test.csv
--2022-01-21 06:09:15-- https://autogluon-text-data.s3.amazonaws.com/multimodal_text/machine_hack_product_sentiment/train.csv
Resolving autogluon-text-data.s3.amazonaws.com (autogluon-text-data.s3.amazonaws.com)... 52.216.237.67
Connecting to autogluon-text-data.s3.amazonaws.com (autogluon-text-data.s3.amazonaws.com)|52.216.237.67|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 689486 (673K) [text/csv]
Saving to: ‘product_sentiment_machine_hack/train.csv’
product_sentiment_m 100%[===================>] 673.33K 1.69MB/s in 0.4s
2022-01-21 06:09:16 (1.69 MB/s) - ‘product_sentiment_machine_hack/train.csv’ saved [689486/689486]
--2022-01-21 06:09:17-- https://autogluon-text-data.s3.amazonaws.com/multimodal_text/machine_hack_product_sentiment/dev.csv
Resolving autogluon-text-data.s3.amazonaws.com (autogluon-text-data.s3.amazonaws.com)... 54.231.140.169
Connecting to autogluon-text-data.s3.amazonaws.com (autogluon-text-data.s3.amazonaws.com)|54.231.140.169|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 75517 (74K) [text/csv]
Saving to: ‘product_sentiment_machine_hack/dev.csv’
product_sentiment_m 100%[===================>] 73.75K --.-KB/s in 0.1s
2022-01-21 06:09:17 (543 KB/s) - ‘product_sentiment_machine_hack/dev.csv’ saved [75517/75517]
--2022-01-21 06:09:18-- https://autogluon-text-data.s3.amazonaws.com/multimodal_text/machine_hack_product_sentiment/test.csv
Resolving autogluon-text-data.s3.amazonaws.com (autogluon-text-data.s3.amazonaws.com)... 54.231.140.169
Connecting to autogluon-text-data.s3.amazonaws.com (autogluon-text-data.s3.amazonaws.com)|54.231.140.169|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 312194 (305K) [text/csv]
Saving to: ‘product_sentiment_machine_hack/test.csv’
product_sentiment_m 100%[===================>] 304.88K 1.14MB/s in 0.3s
2022-01-21 06:09:18 (1.14 MB/s) - ‘product_sentiment_machine_hack/test.csv’ saved [312194/312194]
subsample_size = 2000 # for quick demo, try setting to larger values
feature_columns = ['Product_Description', 'Product_Type']
label = 'Sentiment'
train_df = pd.read_csv('product_sentiment_machine_hack/train.csv', index_col=0).sample(2000, random_state=123)
dev_df = pd.read_csv('product_sentiment_machine_hack/dev.csv', index_col=0)
test_df = pd.read_csv('product_sentiment_machine_hack/test.csv', index_col=0)
train_df = train_df[feature_columns + [label]]
dev_df = dev_df[feature_columns + [label]]
test_df = test_df[feature_columns]
print('Number of training samples:', len(train_df))
print('Number of dev samples:', len(dev_df))
print('Number of test samples:', len(test_df))
Number of training samples: 2000
Number of dev samples: 637
Number of test samples: 2728
There are two features in the dataset: the users’ review of the product and the product’s type, and four possible classes to predict.
train_df.head()
| Product_Description | Product_Type | Sentiment | |
|---|---|---|---|
| 4532 | they took away the lego pit but replaced it wi... | 0 | 1 |
| 1831 | #Apple to Open Pop-Up Shop at #SXSW [REPORT]: ... | 9 | 2 |
| 3536 | RT @mention False Alarm: Google Circles Not Co... | 5 | 1 |
| 5157 | Will Google reveal a new social network called... | 9 | 2 |
| 4643 | Niceness RT @mention Less than 2 hours until w... | 6 | 3 |
dev_df.head()
| Product_Description | Product_Type | Sentiment | |
|---|---|---|---|
| 3170 | Do it. RT @mention Come party w/ Google tonigh... | 3 | 3 |
| 6301 | Line for iPads at #SXSW. Doesn't look too bad!... | 6 | 3 |
| 5643 | First up: iPad Design Headaches (2 Tablets, Ca... | 6 | 2 |
| 1953 | #SXSW: Mint Talks Mobile App Development Chall... | 9 | 2 |
| 2658 | ÛÏ@mention Apple store downtown Austin open t... | 9 | 2 |
test_df.head()
| Product_Description | Product_Type | |
|---|---|---|
| Text_ID | ||
| 5786 | RT @mention Going to #SXSW? The new iPhone gui... | 7 |
| 5363 | RT @mention 95% of iPhone and Droid apps have ... | 9 |
| 6716 | RT @mention Thank you to @mention for letting ... | 9 |
| 4339 | #Thanks @mention we're lovin' the @mention app... | 7 |
| 66 | At #sxsw? @mention / @mention wanna buy you a ... | 9 |
AutoGluon Tabular with Multimodal Support¶
To utilize the TextPredictor model inside of TabularPredictor,
we must specify the hyperparameters = 'multimodal' in AutoGluon
Tabular. Internally, this will train multiple tabular models as well as
the TextPredictor model, and then combine them via either a weighted
ensemble or stack ensemble, as explained in AutoGluon Tabular
Paper. If you do not specify
hyperparameters = 'multimodal', then AutoGluon Tabular will simply
featurize text fields using N-grams and train only tabular models (which
may work better if your text is mostly uncommon strings/vocabulary).
from autogluon.tabular import TabularPredictor
predictor = TabularPredictor(label='Sentiment', path='ag_tabular_product_sentiment_multimodal')
predictor.fit(train_df, hyperparameters='multimodal')
Beginning AutoGluon training ...
AutoGluon will save models to "ag_tabular_product_sentiment_multimodal/"
AutoGluon Version: 0.3.2b20220121
Python Version: 3.9.9
Operating System: Linux
Train Data Rows: 2000
Train Data Columns: 2
Preprocessing data ...
AutoGluon infers your prediction problem is: 'multiclass' (because dtype of label-column == int, but few unique label-values observed).
4 unique label values: [1, 2, 3, 0]
If 'multiclass' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])
Train Data Class Count: 4
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
Available Memory: 17069.98 MB
Train Data (Original) Memory Usage: 0.34 MB (0.0% of available memory)
Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
Stage 1 Generators:
Fitting AsTypeFeatureGenerator...
Stage 2 Generators:
Fitting FillNaFeatureGenerator...
Stage 3 Generators:
Fitting IdentityFeatureGenerator...
Fitting IdentityFeatureGenerator...
Fitting RenameFeatureGenerator...
Fitting CategoryFeatureGenerator...
Fitting CategoryMemoryMinimizeFeatureGenerator...
Fitting TextSpecialFeatureGenerator...
Fitting BinnedFeatureGenerator...
Fitting DropDuplicatesFeatureGenerator...
Fitting TextNgramFeatureGenerator...
Fitting CountVectorizer for text features: ['Product_Description']
CountVectorizer fit with vocabulary size = 230
Stage 4 Generators:
Fitting DropUniqueFeatureGenerator...
Types of features in original data (raw dtype, special dtypes):
('int', []) : 1 | ['Product_Type']
('object', ['text']) : 1 | ['Product_Description']
Types of features in processed data (raw dtype, special dtypes):
('category', ['text_as_category']) : 1 | ['Product_Description']
('int', []) : 1 | ['Product_Type']
('int', ['binned', 'text_special']) : 30 | ['Product_Description.char_count', 'Product_Description.word_count', 'Product_Description.capital_ratio', 'Product_Description.lower_ratio', 'Product_Description.digit_ratio', ...]
('int', ['text_ngram']) : 231 | ['__nlp__.about', '__nlp__.all', '__nlp__.amp', '__nlp__.an', '__nlp__.an ipad', ...]
('object', ['text']) : 1 | ['Product_Description_raw_text']
0.5s = Fit runtime
2 features in original data used to generate 264 features in processed data.
Train Data (Processed) Memory Usage: 1.34 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 0.5s ...
AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'
To change this, specify the eval_metric parameter of Predictor()
Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 1600, Val Rows: 400
Fitting 9 L1 models ...
Fitting model: LightGBM ...
0.8925 = Validation score (accuracy)
2.15s = Training runtime
0.01s = Validation runtime
Fitting model: LightGBMXT ...
0.8625 = Validation score (accuracy)
1.06s = Training runtime
0.01s = Validation runtime
Fitting model: CatBoost ...
0.8825 = Validation score (accuracy)
1.09s = Training runtime
0.01s = Validation runtime
Fitting model: XGBoost ...
0.8925 = Validation score (accuracy)
1.7s = Training runtime
0.01s = Validation runtime
Fitting model: NeuralNetTorch ...
0.8825 = Validation score (accuracy)
2.12s = Training runtime
0.02s = Validation runtime
Fitting model: VowpalWabbit ...
0.675 = Validation score (accuracy)
0.7s = Training runtime
0.03s = Validation runtime
Fitting model: LightGBMLarge ...
0.88 = Validation score (accuracy)
3.36s = Training runtime
0.02s = Validation runtime
Fitting model: TextPredictor ...
All Logs will be saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-tabular-v3/docs/_build/eval/tutorials/tabular_prediction/ag_tabular_product_sentiment_multimodal/models/TextPredictor/task0/training.log
2022-01-21 06:09:45,678 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 3/130, Epoch 0] Validation accuracy=6.2250e-01, log_loss=8.6025e-01, accuracy=6.2250e-01, Time computing validation-score=1.178s, Total time spent=0.11min. Found improved model=True, Improved top-3 models=True
[Iter 3/130, Epoch 0] Validation accuracy=6.2250e-01, log_loss=8.6025e-01, accuracy=6.2250e-01, Time computing validation-score=1.178s, Total time spent=0.11min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:09:51,611 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 6/130, Epoch 0] Validation accuracy=8.7750e-01, log_loss=5.5224e-01, accuracy=8.7750e-01, Time computing validation-score=1.148s, Total time spent=0.20min. Found improved model=True, Improved top-3 models=True
[Iter 6/130, Epoch 0] Validation accuracy=8.7750e-01, log_loss=5.5224e-01, accuracy=8.7750e-01, Time computing validation-score=1.148s, Total time spent=0.20min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:09:57,556 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 9/130, Epoch 0] Validation accuracy=8.8250e-01, log_loss=4.4643e-01, accuracy=8.8250e-01, Time computing validation-score=1.146s, Total time spent=0.30min. Found improved model=True, Improved top-3 models=True
[Iter 9/130, Epoch 0] Validation accuracy=8.8250e-01, log_loss=4.4643e-01, accuracy=8.8250e-01, Time computing validation-score=1.146s, Total time spent=0.30min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:10:03,792 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 12/130, Epoch 0] Validation accuracy=8.8250e-01, log_loss=4.5360e-01, accuracy=8.8250e-01, Time computing validation-score=1.152s, Total time spent=0.41min. Found improved model=True, Improved top-3 models=True
[Iter 12/130, Epoch 0] Validation accuracy=8.8250e-01, log_loss=4.5360e-01, accuracy=8.8250e-01, Time computing validation-score=1.152s, Total time spent=0.41min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:10:10,030 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 15/130, Epoch 1] Validation accuracy=8.8250e-01, log_loss=5.5542e-01, accuracy=8.8250e-01, Time computing validation-score=1.160s, Total time spent=0.51min. Found improved model=True, Improved top-3 models=True
[Iter 15/130, Epoch 1] Validation accuracy=8.8250e-01, log_loss=5.5542e-01, accuracy=8.8250e-01, Time computing validation-score=1.160s, Total time spent=0.51min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:10:16,246 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 18/130, Epoch 1] Validation accuracy=8.8250e-01, log_loss=4.4113e-01, accuracy=8.8250e-01, Time computing validation-score=1.147s, Total time spent=0.61min. Found improved model=True, Improved top-3 models=True
[Iter 18/130, Epoch 1] Validation accuracy=8.8250e-01, log_loss=4.4113e-01, accuracy=8.8250e-01, Time computing validation-score=1.147s, Total time spent=0.61min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:10:22,486 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 21/130, Epoch 1] Validation accuracy=8.8250e-01, log_loss=4.0868e-01, accuracy=8.8250e-01, Time computing validation-score=1.156s, Total time spent=0.72min. Found improved model=True, Improved top-3 models=True
[Iter 21/130, Epoch 1] Validation accuracy=8.8250e-01, log_loss=4.0868e-01, accuracy=8.8250e-01, Time computing validation-score=1.156s, Total time spent=0.72min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:10:29,339 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 24/130, Epoch 1] Validation accuracy=8.9000e-01, log_loss=3.9177e-01, accuracy=8.9000e-01, Time computing validation-score=1.147s, Total time spent=0.83min. Found improved model=True, Improved top-3 models=True
[Iter 24/130, Epoch 1] Validation accuracy=8.9000e-01, log_loss=3.9177e-01, accuracy=8.9000e-01, Time computing validation-score=1.147s, Total time spent=0.83min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:10:35,657 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 27/130, Epoch 2] Validation accuracy=8.9500e-01, log_loss=4.1922e-01, accuracy=8.9500e-01, Time computing validation-score=1.149s, Total time spent=0.94min. Found improved model=True, Improved top-3 models=True
[Iter 27/130, Epoch 2] Validation accuracy=8.9500e-01, log_loss=4.1922e-01, accuracy=8.9500e-01, Time computing validation-score=1.149s, Total time spent=0.94min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:10:41,107 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 30/130, Epoch 2] Validation accuracy=8.8500e-01, log_loss=4.3018e-01, accuracy=8.8500e-01, Time computing validation-score=1.172s, Total time spent=1.03min. Found improved model=False, Improved top-3 models=True
[Iter 30/130, Epoch 2] Validation accuracy=8.8500e-01, log_loss=4.3018e-01, accuracy=8.8500e-01, Time computing validation-score=1.172s, Total time spent=1.03min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:10:46,448 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 33/130, Epoch 2] Validation accuracy=8.8750e-01, log_loss=3.8662e-01, accuracy=8.8750e-01, Time computing validation-score=1.185s, Total time spent=1.12min. Found improved model=False, Improved top-3 models=True
[Iter 33/130, Epoch 2] Validation accuracy=8.8750e-01, log_loss=3.8662e-01, accuracy=8.8750e-01, Time computing validation-score=1.185s, Total time spent=1.12min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:10:52,748 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 36/130, Epoch 2] Validation accuracy=8.9750e-01, log_loss=4.0357e-01, accuracy=8.9750e-01, Time computing validation-score=1.165s, Total time spent=1.22min. Found improved model=True, Improved top-3 models=True
[Iter 36/130, Epoch 2] Validation accuracy=8.9750e-01, log_loss=4.0357e-01, accuracy=8.9750e-01, Time computing validation-score=1.165s, Total time spent=1.22min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:10:59,011 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 39/130, Epoch 2] Validation accuracy=9.0000e-01, log_loss=3.6024e-01, accuracy=9.0000e-01, Time computing validation-score=1.184s, Total time spent=1.33min. Found improved model=True, Improved top-3 models=True
[Iter 39/130, Epoch 2] Validation accuracy=9.0000e-01, log_loss=3.6024e-01, accuracy=9.0000e-01, Time computing validation-score=1.184s, Total time spent=1.33min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:11:04,397 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 42/130, Epoch 3] Validation accuracy=8.9500e-01, log_loss=3.5098e-01, accuracy=8.9500e-01, Time computing validation-score=1.159s, Total time spent=1.42min. Found improved model=False, Improved top-3 models=True
[Iter 42/130, Epoch 3] Validation accuracy=8.9500e-01, log_loss=3.5098e-01, accuracy=8.9500e-01, Time computing validation-score=1.159s, Total time spent=1.42min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:11:10,664 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 45/130, Epoch 3] Validation accuracy=9.0750e-01, log_loss=3.6334e-01, accuracy=9.0750e-01, Time computing validation-score=1.180s, Total time spent=1.52min. Found improved model=True, Improved top-3 models=True
[Iter 45/130, Epoch 3] Validation accuracy=9.0750e-01, log_loss=3.6334e-01, accuracy=9.0750e-01, Time computing validation-score=1.180s, Total time spent=1.52min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:11:16,126 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 48/130, Epoch 3] Validation accuracy=8.9750e-01, log_loss=3.4912e-01, accuracy=8.9750e-01, Time computing validation-score=1.185s, Total time spent=1.61min. Found improved model=False, Improved top-3 models=True
[Iter 48/130, Epoch 3] Validation accuracy=8.9750e-01, log_loss=3.4912e-01, accuracy=8.9750e-01, Time computing validation-score=1.185s, Total time spent=1.61min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:11:22,414 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 51/130, Epoch 3] Validation accuracy=9.0750e-01, log_loss=4.0451e-01, accuracy=9.0750e-01, Time computing validation-score=1.182s, Total time spent=1.72min. Found improved model=True, Improved top-3 models=True
[Iter 51/130, Epoch 3] Validation accuracy=9.0750e-01, log_loss=4.0451e-01, accuracy=9.0750e-01, Time computing validation-score=1.182s, Total time spent=1.72min. Found improved model=True, Improved top-3 models=True
2022-01-21 06:11:27,838 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 54/130, Epoch 4] Validation accuracy=9.0500e-01, log_loss=3.7775e-01, accuracy=9.0500e-01, Time computing validation-score=1.188s, Total time spent=1.81min. Found improved model=False, Improved top-3 models=True
[Iter 54/130, Epoch 4] Validation accuracy=9.0500e-01, log_loss=3.7775e-01, accuracy=9.0500e-01, Time computing validation-score=1.188s, Total time spent=1.81min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:11:32,405 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 57/130, Epoch 4] Validation accuracy=8.8000e-01, log_loss=3.8270e-01, accuracy=8.8000e-01, Time computing validation-score=1.183s, Total time spent=1.88min. Found improved model=False, Improved top-3 models=False
[Iter 57/130, Epoch 4] Validation accuracy=8.8000e-01, log_loss=3.8270e-01, accuracy=8.8000e-01, Time computing validation-score=1.183s, Total time spent=1.88min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:11:37,737 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 60/130, Epoch 4] Validation accuracy=9.0500e-01, log_loss=3.6967e-01, accuracy=9.0500e-01, Time computing validation-score=1.188s, Total time spent=1.97min. Found improved model=False, Improved top-3 models=True
[Iter 60/130, Epoch 4] Validation accuracy=9.0500e-01, log_loss=3.6967e-01, accuracy=9.0500e-01, Time computing validation-score=1.188s, Total time spent=1.97min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:11:42,238 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 63/130, Epoch 4] Validation accuracy=9.0250e-01, log_loss=4.3535e-01, accuracy=9.0250e-01, Time computing validation-score=1.184s, Total time spent=2.05min. Found improved model=False, Improved top-3 models=False
[Iter 63/130, Epoch 4] Validation accuracy=9.0250e-01, log_loss=4.3535e-01, accuracy=9.0250e-01, Time computing validation-score=1.184s, Total time spent=2.05min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:11:46,764 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 66/130, Epoch 5] Validation accuracy=9.0250e-01, log_loss=3.5517e-01, accuracy=9.0250e-01, Time computing validation-score=1.180s, Total time spent=2.12min. Found improved model=False, Improved top-3 models=False
[Iter 66/130, Epoch 5] Validation accuracy=9.0250e-01, log_loss=3.5517e-01, accuracy=9.0250e-01, Time computing validation-score=1.180s, Total time spent=2.12min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:11:52,197 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 69/130, Epoch 5] Validation accuracy=9.0500e-01, log_loss=3.5995e-01, accuracy=9.0500e-01, Time computing validation-score=1.195s, Total time spent=2.21min. Found improved model=False, Improved top-3 models=True
[Iter 69/130, Epoch 5] Validation accuracy=9.0500e-01, log_loss=3.5995e-01, accuracy=9.0500e-01, Time computing validation-score=1.195s, Total time spent=2.21min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:11:56,705 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 72/130, Epoch 5] Validation accuracy=9.0250e-01, log_loss=4.2312e-01, accuracy=9.0250e-01, Time computing validation-score=1.185s, Total time spent=2.29min. Found improved model=False, Improved top-3 models=False
[Iter 72/130, Epoch 5] Validation accuracy=9.0250e-01, log_loss=4.2312e-01, accuracy=9.0250e-01, Time computing validation-score=1.185s, Total time spent=2.29min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:01,321 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 75/130, Epoch 5] Validation accuracy=8.9750e-01, log_loss=3.7594e-01, accuracy=8.9750e-01, Time computing validation-score=1.186s, Total time spent=2.37min. Found improved model=False, Improved top-3 models=False
[Iter 75/130, Epoch 5] Validation accuracy=8.9750e-01, log_loss=3.7594e-01, accuracy=8.9750e-01, Time computing validation-score=1.186s, Total time spent=2.37min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:05,855 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 78/130, Epoch 5] Validation accuracy=9.0000e-01, log_loss=3.7560e-01, accuracy=9.0000e-01, Time computing validation-score=1.185s, Total time spent=2.44min. Found improved model=False, Improved top-3 models=False
[Iter 78/130, Epoch 5] Validation accuracy=9.0000e-01, log_loss=3.7560e-01, accuracy=9.0000e-01, Time computing validation-score=1.185s, Total time spent=2.44min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:11,225 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 81/130, Epoch 6] Validation accuracy=9.0500e-01, log_loss=3.9447e-01, accuracy=9.0500e-01, Time computing validation-score=1.209s, Total time spent=2.53min. Found improved model=False, Improved top-3 models=True
[Iter 81/130, Epoch 6] Validation accuracy=9.0500e-01, log_loss=3.9447e-01, accuracy=9.0500e-01, Time computing validation-score=1.209s, Total time spent=2.53min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:12:16,703 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 84/130, Epoch 6] Validation accuracy=9.0500e-01, log_loss=4.0651e-01, accuracy=9.0500e-01, Time computing validation-score=1.181s, Total time spent=2.62min. Found improved model=False, Improved top-3 models=True
[Iter 84/130, Epoch 6] Validation accuracy=9.0500e-01, log_loss=4.0651e-01, accuracy=9.0500e-01, Time computing validation-score=1.181s, Total time spent=2.62min. Found improved model=False, Improved top-3 models=True
2022-01-21 06:12:21,257 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 87/130, Epoch 6] Validation accuracy=8.8250e-01, log_loss=4.2515e-01, accuracy=8.8250e-01, Time computing validation-score=1.200s, Total time spent=2.70min. Found improved model=False, Improved top-3 models=False
[Iter 87/130, Epoch 6] Validation accuracy=8.8250e-01, log_loss=4.2515e-01, accuracy=8.8250e-01, Time computing validation-score=1.200s, Total time spent=2.70min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:25,827 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 90/130, Epoch 6] Validation accuracy=8.9750e-01, log_loss=4.1722e-01, accuracy=8.9750e-01, Time computing validation-score=1.188s, Total time spent=2.77min. Found improved model=False, Improved top-3 models=False
[Iter 90/130, Epoch 6] Validation accuracy=8.9750e-01, log_loss=4.1722e-01, accuracy=8.9750e-01, Time computing validation-score=1.188s, Total time spent=2.77min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:30,427 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 93/130, Epoch 7] Validation accuracy=8.8000e-01, log_loss=4.4252e-01, accuracy=8.8000e-01, Time computing validation-score=1.196s, Total time spent=2.85min. Found improved model=False, Improved top-3 models=False
[Iter 93/130, Epoch 7] Validation accuracy=8.8000e-01, log_loss=4.4252e-01, accuracy=8.8000e-01, Time computing validation-score=1.196s, Total time spent=2.85min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:34,985 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 96/130, Epoch 7] Validation accuracy=8.3750e-01, log_loss=5.3797e-01, accuracy=8.3750e-01, Time computing validation-score=1.192s, Total time spent=2.93min. Found improved model=False, Improved top-3 models=False
[Iter 96/130, Epoch 7] Validation accuracy=8.3750e-01, log_loss=5.3797e-01, accuracy=8.3750e-01, Time computing validation-score=1.192s, Total time spent=2.93min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:39,550 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 99/130, Epoch 7] Validation accuracy=8.7000e-01, log_loss=4.6513e-01, accuracy=8.7000e-01, Time computing validation-score=1.191s, Total time spent=3.00min. Found improved model=False, Improved top-3 models=False
[Iter 99/130, Epoch 7] Validation accuracy=8.7000e-01, log_loss=4.6513e-01, accuracy=8.7000e-01, Time computing validation-score=1.191s, Total time spent=3.00min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:44,152 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 102/130, Epoch 7] Validation accuracy=8.9750e-01, log_loss=4.1996e-01, accuracy=8.9750e-01, Time computing validation-score=1.224s, Total time spent=3.08min. Found improved model=False, Improved top-3 models=False
[Iter 102/130, Epoch 7] Validation accuracy=8.9750e-01, log_loss=4.1996e-01, accuracy=8.9750e-01, Time computing validation-score=1.224s, Total time spent=3.08min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:48,812 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 105/130, Epoch 8] Validation accuracy=8.8250e-01, log_loss=4.4127e-01, accuracy=8.8250e-01, Time computing validation-score=1.205s, Total time spent=3.16min. Found improved model=False, Improved top-3 models=False
[Iter 105/130, Epoch 8] Validation accuracy=8.8250e-01, log_loss=4.4127e-01, accuracy=8.8250e-01, Time computing validation-score=1.205s, Total time spent=3.16min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:53,417 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 108/130, Epoch 8] Validation accuracy=8.8000e-01, log_loss=4.6713e-01, accuracy=8.8000e-01, Time computing validation-score=1.203s, Total time spent=3.23min. Found improved model=False, Improved top-3 models=False
[Iter 108/130, Epoch 8] Validation accuracy=8.8000e-01, log_loss=4.6713e-01, accuracy=8.8000e-01, Time computing validation-score=1.203s, Total time spent=3.23min. Found improved model=False, Improved top-3 models=False
2022-01-21 06:12:58,083 - autogluon.text.text_prediction.mx.models - Level 25 - [Iter 111/130, Epoch 8] Validation accuracy=8.8500e-01, log_loss=4.5202e-01, accuracy=8.8500e-01, Time computing validation-score=1.225s, Total time spent=3.31min. Found improved model=False, Improved top-3 models=False
[Iter 111/130, Epoch 8] Validation accuracy=8.8500e-01, log_loss=4.5202e-01, accuracy=8.8500e-01, Time computing validation-score=1.225s, Total time spent=3.31min. Found improved model=False, Improved top-3 models=False
0.9075 = Validation score (accuracy)
208.65s = Training runtime
1.64s = Validation runtime
Fitting model: ImagePredictor ...
No valid features to train ImagePredictor... Skipping this model.
Fitting model: WeightedEnsemble_L2 ...
0.9075 = Validation score (accuracy)
0.2s = Training runtime
0.0s = Validation runtime
AutoGluon training complete, total runtime = 227.71s ...
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("ag_tabular_product_sentiment_multimodal/")
<autogluon.tabular.predictor.predictor.TabularPredictor at 0x7fe722ed09a0>
predictor.leaderboard(dev_df)
model score_test score_val pred_time_test pred_time_val fit_time pred_time_test_marginal pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order
0 TextPredictor 0.897959 0.9075 3.227543 1.639314 208.649363 3.227543 1.639314 208.649363 1 True 8
1 WeightedEnsemble_L2 0.897959 0.9075 3.257015 1.639695 208.852792 0.029472 0.000381 0.203429 2 True 9
2 NeuralNetTorch 0.886970 0.8825 0.025769 0.020709 2.117186 0.025769 0.020709 2.117186 1 True 5
3 LightGBMLarge 0.886970 0.8800 0.061167 0.015823 3.357606 0.061167 0.015823 3.357606 1 True 7
4 CatBoost 0.885400 0.8825 0.016103 0.013558 1.088665 0.016103 0.013558 1.088665 1 True 3
5 LightGBM 0.885400 0.8925 0.030645 0.012096 2.154743 0.030645 0.012096 2.154743 1 True 1
6 XGBoost 0.882261 0.8925 0.141316 0.007339 1.699968 0.141316 0.007339 1.699968 1 True 4
7 LightGBMXT 0.869702 0.8625 0.010967 0.006428 1.057044 0.010967 0.006428 1.057044 1 True 2
8 VowpalWabbit 0.714286 0.6750 0.095918 0.026814 0.699500 0.095918 0.026814 0.699500 1 True 6
| model | score_test | score_val | pred_time_test | pred_time_val | fit_time | pred_time_test_marginal | pred_time_val_marginal | fit_time_marginal | stack_level | can_infer | fit_order | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | TextPredictor | 0.897959 | 0.9075 | 3.227543 | 1.639314 | 208.649363 | 3.227543 | 1.639314 | 208.649363 | 1 | True | 8 |
| 1 | WeightedEnsemble_L2 | 0.897959 | 0.9075 | 3.257015 | 1.639695 | 208.852792 | 0.029472 | 0.000381 | 0.203429 | 2 | True | 9 |
| 2 | NeuralNetTorch | 0.886970 | 0.8825 | 0.025769 | 0.020709 | 2.117186 | 0.025769 | 0.020709 | 2.117186 | 1 | True | 5 |
| 3 | LightGBMLarge | 0.886970 | 0.8800 | 0.061167 | 0.015823 | 3.357606 | 0.061167 | 0.015823 | 3.357606 | 1 | True | 7 |
| 4 | CatBoost | 0.885400 | 0.8825 | 0.016103 | 0.013558 | 1.088665 | 0.016103 | 0.013558 | 1.088665 | 1 | True | 3 |
| 5 | LightGBM | 0.885400 | 0.8925 | 0.030645 | 0.012096 | 2.154743 | 0.030645 | 0.012096 | 2.154743 | 1 | True | 1 |
| 6 | XGBoost | 0.882261 | 0.8925 | 0.141316 | 0.007339 | 1.699968 | 0.141316 | 0.007339 | 1.699968 | 1 | True | 4 |
| 7 | LightGBMXT | 0.869702 | 0.8625 | 0.010967 | 0.006428 | 1.057044 | 0.010967 | 0.006428 | 1.057044 | 1 | True | 2 |
| 8 | VowpalWabbit | 0.714286 | 0.6750 | 0.095918 | 0.026814 | 0.699500 | 0.095918 | 0.026814 | 0.699500 | 1 | True | 6 |
Improve the Performance with Stack Ensemble¶
You can improve predictive performance by using stack ensembling. One way to turn it on is as follows:
predictor.fit(train_df, hyperparameters='multimodal', num_bag_folds=5, num_stack_levels=1)
or using:
predictor.fit(train_df, hyperparameters='multimodal', presets='best_quality')
which will automatically select values for num_stack_levels (how
many stacking layers) and num_bag_folds (how many folds to split
data into during bagging). Stack ensembling can take much longer, so we
won’t run with this configuration here. You may explore more examples in
https://github.com/awslabs/autogluon/tree/master/examples/text_prediction,
which demonstrate how you can achieve top performance in competitions
with a stack ensembling based solution.