Predicting Multiple Columns in a Table (Multi-Label Prediction)#
In multi-label prediction, we wish to predict multiple columns of a table (i.e. labels) based on the values in the remaining columns. Here we present a simple strategy to do this with AutoGluon, which simply maintains a separate TabularPredictor object for each column being predicted. Correlations between labels can be accounted for in predictions by imposing an order on the labels and allowing the TabularPredictor for each label to condition on the predicted values for labels that appeared earlier in the order.
MultilabelPredictor Class#
We start by defining a custom MultilabelPredictor class to manage a collection of TabularPredictor objects, one for each label. You can use the MultilabelPredictor similarly to an individual TabularPredictor, except it operates on multiple labels rather than one.
from autogluon.tabular import TabularDataset, TabularPredictor
from autogluon.common.utils.utils import setup_outputdir
from autogluon.core.utils.loaders import load_pkl
from autogluon.core.utils.savers import save_pkl
import os.path
class MultilabelPredictor():
""" Tabular Predictor for predicting multiple columns in table.
Creates multiple TabularPredictor objects which you can also use individually.
You can access the TabularPredictor for a particular label via: `multilabel_predictor.get_predictor(label_i)`
Parameters
----------
labels : List[str]
The ith element of this list is the column (i.e. `label`) predicted by the ith TabularPredictor stored in this object.
path : str, default = None
Path to directory where models and intermediate outputs should be saved.
If unspecified, a time-stamped folder called "AutogluonModels/ag-[TIMESTAMP]" will be created in the working directory to store all models.
Note: To call `fit()` twice and save all results of each fit, you must specify different `path` locations or don't specify `path` at all.
Otherwise files from first `fit()` will be overwritten by second `fit()`.
Caution: when predicting many labels, this directory may grow large as it needs to store many TabularPredictors.
problem_types : List[str], default = None
The ith element is the `problem_type` for the ith TabularPredictor stored in this object.
eval_metrics : List[str], default = None
The ith element is the `eval_metric` for the ith TabularPredictor stored in this object.
consider_labels_correlation : bool, default = True
Whether the predictions of multiple labels should account for label correlations or predict each label independently of the others.
If True, the ordering of `labels` may affect resulting accuracy as each label is predicted conditional on the previous labels appearing earlier in this list (i.e. in an auto-regressive fashion).
Set to False if during inference you may want to individually use just the ith TabularPredictor without predicting all the other labels.
kwargs :
Arguments passed into the initialization of each TabularPredictor.
"""
multi_predictor_file = 'multilabel_predictor.pkl'
def __init__(self, labels, path=None, problem_types=None, eval_metrics=None, consider_labels_correlation=True, **kwargs):
if len(labels) < 2:
raise ValueError("MultilabelPredictor is only intended for predicting MULTIPLE labels (columns), use TabularPredictor for predicting one label (column).")
if (problem_types is not None) and (len(problem_types) != len(labels)):
raise ValueError("If provided, `problem_types` must have same length as `labels`")
if (eval_metrics is not None) and (len(eval_metrics) != len(labels)):
raise ValueError("If provided, `eval_metrics` must have same length as `labels`")
self.path = setup_outputdir(path, warn_if_exist=False)
self.labels = labels
self.consider_labels_correlation = consider_labels_correlation
self.predictors = {} # key = label, value = TabularPredictor or str path to the TabularPredictor for this label
if eval_metrics is None:
self.eval_metrics = {}
else:
self.eval_metrics = {labels[i] : eval_metrics[i] for i in range(len(labels))}
problem_type = None
eval_metric = None
for i in range(len(labels)):
label = labels[i]
path_i = self.path + "Predictor_" + label
if problem_types is not None:
problem_type = problem_types[i]
if eval_metrics is not None:
eval_metric = eval_metrics[i]
self.predictors[label] = TabularPredictor(label=label, problem_type=problem_type, eval_metric=eval_metric, path=path_i, **kwargs)
def fit(self, train_data, tuning_data=None, **kwargs):
""" Fits a separate TabularPredictor to predict each of the labels.
Parameters
----------
train_data, tuning_data : str or autogluon.tabular.TabularDataset or pd.DataFrame
See documentation for `TabularPredictor.fit()`.
kwargs :
Arguments passed into the `fit()` call for each TabularPredictor.
"""
if isinstance(train_data, str):
train_data = TabularDataset(train_data)
if tuning_data is not None and isinstance(tuning_data, str):
tuning_data = TabularDataset(tuning_data)
train_data_og = train_data.copy()
if tuning_data is not None:
tuning_data_og = tuning_data.copy()
else:
tuning_data_og = None
save_metrics = len(self.eval_metrics) == 0
for i in range(len(self.labels)):
label = self.labels[i]
predictor = self.get_predictor(label)
if not self.consider_labels_correlation:
labels_to_drop = [l for l in self.labels if l != label]
else:
labels_to_drop = [self.labels[j] for j in range(i+1, len(self.labels))]
train_data = train_data_og.drop(labels_to_drop, axis=1)
if tuning_data is not None:
tuning_data = tuning_data_og.drop(labels_to_drop, axis=1)
print(f"Fitting TabularPredictor for label: {label} ...")
predictor.fit(train_data=train_data, tuning_data=tuning_data, **kwargs)
self.predictors[label] = predictor.path
if save_metrics:
self.eval_metrics[label] = predictor.eval_metric
self.save()
def predict(self, data, **kwargs):
""" Returns DataFrame with label columns containing predictions for each label.
Parameters
----------
data : str or autogluon.tabular.TabularDataset or pd.DataFrame
Data to make predictions for. If label columns are present in this data, they will be ignored. See documentation for `TabularPredictor.predict()`.
kwargs :
Arguments passed into the predict() call for each TabularPredictor.
"""
return self._predict(data, as_proba=False, **kwargs)
def predict_proba(self, data, **kwargs):
""" Returns dict where each key is a label and the corresponding value is the `predict_proba()` output for just that label.
Parameters
----------
data : str or autogluon.tabular.TabularDataset or pd.DataFrame
Data to make predictions for. See documentation for `TabularPredictor.predict()` and `TabularPredictor.predict_proba()`.
kwargs :
Arguments passed into the `predict_proba()` call for each TabularPredictor (also passed into a `predict()` call).
"""
return self._predict(data, as_proba=True, **kwargs)
def evaluate(self, data, **kwargs):
""" Returns dict where each key is a label and the corresponding value is the `evaluate()` output for just that label.
Parameters
----------
data : str or autogluon.tabular.TabularDataset or pd.DataFrame
Data to evalate predictions of all labels for, must contain all labels as columns. See documentation for `TabularPredictor.evaluate()`.
kwargs :
Arguments passed into the `evaluate()` call for each TabularPredictor (also passed into the `predict()` call).
"""
data = self._get_data(data)
eval_dict = {}
for label in self.labels:
print(f"Evaluating TabularPredictor for label: {label} ...")
predictor = self.get_predictor(label)
eval_dict[label] = predictor.evaluate(data, **kwargs)
if self.consider_labels_correlation:
data[label] = predictor.predict(data, **kwargs)
return eval_dict
def save(self):
""" Save MultilabelPredictor to disk. """
for label in self.labels:
if not isinstance(self.predictors[label], str):
self.predictors[label] = self.predictors[label].path
save_pkl.save(path=self.path+self.multi_predictor_file, object=self)
print(f"MultilabelPredictor saved to disk. Load with: MultilabelPredictor.load('{self.path}')")
@classmethod
def load(cls, path):
""" Load MultilabelPredictor from disk `path` previously specified when creating this MultilabelPredictor. """
path = os.path.expanduser(path)
if path[-1] != os.path.sep:
path = path + os.path.sep
return load_pkl.load(path=path+cls.multi_predictor_file)
def get_predictor(self, label):
""" Returns TabularPredictor which is used to predict this label. """
predictor = self.predictors[label]
if isinstance(predictor, str):
return TabularPredictor.load(path=predictor)
return predictor
def _get_data(self, data):
if isinstance(data, str):
return TabularDataset(data)
return data.copy()
def _predict(self, data, as_proba=False, **kwargs):
data = self._get_data(data)
if as_proba:
predproba_dict = {}
for label in self.labels:
print(f"Predicting with TabularPredictor for label: {label} ...")
predictor = self.get_predictor(label)
if as_proba:
predproba_dict[label] = predictor.predict_proba(data, as_multiclass=True, **kwargs)
data[label] = predictor.predict(data, **kwargs)
if not as_proba:
return data[self.labels]
else:
return predproba_dict
Training#
Let’s now apply our multi-label predictor to predict multiple columns in a data table. We first train models to predict each of the labels.
train_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv')
subsample_size = 500 # subsample subset of data for faster demo, try setting this to much larger values
train_data = train_data.sample(n=subsample_size, random_state=0)
train_data.head()
| age | workclass | fnlwgt | education | education-num | marital-status | occupation | relationship | race | sex | capital-gain | capital-loss | hours-per-week | native-country | class | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 6118 | 51 | Private | 39264 | Some-college | 10 | Married-civ-spouse | Exec-managerial | Wife | White | Female | 0 | 0 | 40 | United-States | >50K |
| 23204 | 58 | Private | 51662 | 10th | 6 | Married-civ-spouse | Other-service | Wife | White | Female | 0 | 0 | 8 | United-States | <=50K |
| 29590 | 40 | Private | 326310 | Some-college | 10 | Married-civ-spouse | Craft-repair | Husband | White | Male | 0 | 0 | 44 | United-States | <=50K |
| 18116 | 37 | Private | 222450 | HS-grad | 9 | Never-married | Sales | Not-in-family | White | Male | 0 | 2339 | 40 | El-Salvador | <=50K |
| 33964 | 62 | Private | 109190 | Bachelors | 13 | Married-civ-spouse | Exec-managerial | Husband | White | Male | 15024 | 0 | 40 | United-States | >50K |
labels = ['education-num','education','class'] # which columns to predict based on the others
problem_types = ['regression','multiclass','binary'] # type of each prediction problem (optional)
eval_metrics = ['mean_absolute_error','accuracy','accuracy'] # metrics used to evaluate predictions for each label (optional)
save_path = 'agModels-predictEducationClass' # specifies folder to store trained models (optional)
time_limit = 5 # how many seconds to train the TabularPredictor for each label, set much larger in your applications!
multi_predictor = MultilabelPredictor(labels=labels, problem_types=problem_types, eval_metrics=eval_metrics, path=save_path)
multi_predictor.fit(train_data, time_limit=time_limit)
Fitting TabularPredictor for label: education-num ...
Fitting TabularPredictor for label: education ...
Fitting TabularPredictor for label: class ...
MultilabelPredictor saved to disk. Load with: MultilabelPredictor.load('agModels-predictEducationClass')
Beginning AutoGluon training ... Time limit = 5s
AutoGluon will save models to "agModels-predictEducationClassPredictor_education-num"
AutoGluon Version: 0.8.3b20231012
Python Version: 3.10.8
Operating System: Linux
Platform Machine: x86_64
Platform Version: #1 SMP Tue Nov 30 00:17:50 UTC 2021
Disk Space Avail: 231.00 GB / 274.87 GB (84.0%)
Train Data Rows: 500
Train Data Columns: 12
Label Column: education-num
Preprocessing data ...
/home/ci/autogluon/tabular/src/autogluon/tabular/learner/default_learner.py:215: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead.
with pd.option_context("mode.use_inf_as_na", True): # treat None, NaN, INF, NINF as NA
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
Available Memory: 31025.34 MB
Train Data (Original) Memory Usage: 0.26 MB (0.0% of available memory)
Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
Stage 1 Generators:
Fitting AsTypeFeatureGenerator...
Note: Converting 1 features to boolean dtype as they only contain 2 unique values.
Stage 2 Generators:
Fitting FillNaFeatureGenerator...
/home/ci/autogluon/features/src/autogluon/features/generators/fillna.py:58: FutureWarning: The 'downcast' keyword in fillna is deprecated and will be removed in a future version. Use res.infer_objects(copy=False) to infer non-object dtype, or pd.to_numeric with the 'downcast' keyword to downcast numeric results.
X.fillna(self._fillna_feature_map, inplace=True, downcast=False)
Stage 3 Generators:
Fitting IdentityFeatureGenerator...
Fitting CategoryFeatureGenerator...
Fitting CategoryMemoryMinimizeFeatureGenerator...
Stage 4 Generators:
Fitting DropUniqueFeatureGenerator...
Stage 5 Generators:
Fitting DropDuplicatesFeatureGenerator...
Types of features in original data (raw dtype, special dtypes):
('int', []) : 5 | ['age', 'fnlwgt', 'capital-gain', 'capital-loss', 'hours-per-week']
('object', []) : 7 | ['workclass', 'marital-status', 'occupation', 'relationship', 'race', ...]
Types of features in processed data (raw dtype, special dtypes):
('category', []) : 6 | ['workclass', 'marital-status', 'occupation', 'relationship', 'race', ...]
('int', []) : 5 | ['age', 'fnlwgt', 'capital-gain', 'capital-loss', 'hours-per-week']
('int', ['bool']) : 1 | ['sex']
0.1s = Fit runtime
12 features in original data used to generate 12 features in processed data.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '1054.6666666666665' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
Train Data (Processed) Memory Usage: 0.03 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 0.08s ...
AutoGluon will gauge predictive performance using evaluation metric: 'mean_absolute_error'
This metric's sign has been flipped to adhere to being higher_is_better. The metric score can be multiplied by -1 to get the metric value.
To change this, specify the eval_metric parameter of Predictor()
Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 400, Val Rows: 100
User-specified model hyperparameters to be fit:
{
'NN_TORCH': {},
'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],
'CAT': {},
'XGB': {},
'FASTAI': {},
'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],
'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],
'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],
}
Fitting 11 L1 models ...
Fitting model: KNeighborsUnif ... Training model for up to 4.92s of the 4.91s of remaining time.
-2.086 = Validation score (-mean_absolute_error)
0.01s = Training runtime
0.01s = Validation runtime
Fitting model: KNeighborsDist ... Training model for up to 4.89s of the 4.89s of remaining time.
-2.1856 = Validation score (-mean_absolute_error)
0.0s = Training runtime
0.01s = Validation runtime
Fitting model: LightGBMXT ... Training model for up to 4.87s of the 4.87s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '954.6666666666666' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
-1.7808 = Validation score (-mean_absolute_error)
0.26s = Training runtime
0.01s = Validation runtime
Fitting model: LightGBM ... Training model for up to 4.6s of the 4.6s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '954.6666666666666' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
-1.7854 = Validation score (-mean_absolute_error)
0.16s = Training runtime
0.0s = Validation runtime
Fitting model: RandomForestMSE ... Training model for up to 4.43s of the 4.43s of remaining time.
-1.7082 = Validation score (-mean_absolute_error)
0.44s = Training runtime
0.05s = Validation runtime
Fitting model: CatBoost ... Training model for up to 3.93s of the 3.93s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '954.6666666666666' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
-1.7377 = Validation score (-mean_absolute_error)
1.08s = Training runtime
0.01s = Validation runtime
Fitting model: ExtraTreesMSE ... Training model for up to 2.84s of the 2.84s of remaining time.
-1.8193 = Validation score (-mean_absolute_error)
0.38s = Training runtime
0.05s = Validation runtime
Fitting model: NeuralNetFastAI ... Training model for up to 2.4s of the 2.4s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '954.6666666666666' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/data/transforms.py:225: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if is_categorical_dtype(col):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
-1.9224 = Validation score (-mean_absolute_error)
2.21s = Training runtime
0.01s = Validation runtime
Fitting model: XGBoost ... Training model for up to 0.16s of the 0.16s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '954.6666666666666' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
/home/ci/opt/venv/lib/python3.10/site-packages/xgboost/data.py:440: FutureWarning: is_sparse is deprecated and will be removed in a future version. Check `isinstance(dtype, pd.SparseDtype)` instead.
if is_sparse(data):
-1.6316 = Validation score (-mean_absolute_error)
0.18s = Training runtime
0.01s = Validation runtime
Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.92s of the -0.04s of remaining time.
-1.6266 = Validation score (-mean_absolute_error)
0.35s = Training runtime
0.0s = Validation runtime
AutoGluon training complete, total runtime = 5.42s ... Best model: "WeightedEnsemble_L2"
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("agModels-predictEducationClassPredictor_education-num")
Beginning AutoGluon training ... Time limit = 5s
AutoGluon will save models to "agModels-predictEducationClassPredictor_education"
AutoGluon Version: 0.8.3b20231012
Python Version: 3.10.8
Operating System: Linux
Platform Machine: x86_64
Platform Version: #1 SMP Tue Nov 30 00:17:50 UTC 2021
Disk Space Avail: 230.98 GB / 274.87 GB (84.0%)
Train Data Rows: 500
Train Data Columns: 13
Label Column: education
Preprocessing data ...
/home/ci/autogluon/tabular/src/autogluon/tabular/learner/default_learner.py:215: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead.
with pd.option_context("mode.use_inf_as_na", True): # treat None, NaN, INF, NINF as NA
Warning: Some classes in the training set have fewer than 10 examples. AutoGluon will only keep 11 out of 15 classes for training and will not try to predict the rare classes. To keep more classes, increase the number of datapoints from these rare classes in the training data or reduce label_count_threshold.
Fraction of data from classes with at least 10 examples that will be kept for training models: 0.976
Train Data Class Count: 11
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
Available Memory: 30754.28 MB
Train Data (Original) Memory Usage: 0.25 MB (0.0% of available memory)
Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
Stage 1 Generators:
Fitting AsTypeFeatureGenerator...
Note: Converting 1 features to boolean dtype as they only contain 2 unique values.
Stage 2 Generators:
Fitting FillNaFeatureGenerator...
/home/ci/autogluon/features/src/autogluon/features/generators/fillna.py:58: FutureWarning: The 'downcast' keyword in fillna is deprecated and will be removed in a future version. Use res.infer_objects(copy=False) to infer non-object dtype, or pd.to_numeric with the 'downcast' keyword to downcast numeric results.
X.fillna(self._fillna_feature_map, inplace=True, downcast=False)
Stage 3 Generators:
Fitting IdentityFeatureGenerator...
Fitting CategoryFeatureGenerator...
Fitting CategoryMemoryMinimizeFeatureGenerator...
Stage 4 Generators:
Fitting DropUniqueFeatureGenerator...
Stage 5 Generators:
Fitting DropDuplicatesFeatureGenerator...
Types of features in original data (raw dtype, special dtypes):
('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
('object', []) : 7 | ['workclass', 'marital-status', 'occupation', 'relationship', 'race', ...]
Types of features in processed data (raw dtype, special dtypes):
('category', []) : 6 | ['workclass', 'marital-status', 'occupation', 'relationship', 'race', ...]
('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
('int', ['bool']) : 1 | ['sex']
0.1s = Fit runtime
13 features in original data used to generate 13 features in processed data.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '1042.6666666666665' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
Train Data (Processed) Memory Usage: 0.03 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 0.09s ...
AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'
To change this, specify the eval_metric parameter of Predictor()
Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 390, Val Rows: 98
User-specified model hyperparameters to be fit:
{
'NN_TORCH': {},
'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],
'CAT': {},
'XGB': {},
'FASTAI': {},
'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],
'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],
'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],
}
Fitting 13 L1 models ...
Fitting model: KNeighborsUnif ... Training model for up to 4.91s of the 4.91s of remaining time.
0.2653 = Validation score (accuracy)
0.01s = Training runtime
0.01s = Validation runtime
Fitting model: KNeighborsDist ... Training model for up to 4.89s of the 4.89s of remaining time.
0.2347 = Validation score (accuracy)
0.0s = Training runtime
0.01s = Validation runtime
Fitting model: NeuralNetFastAI ... Training model for up to 4.87s of the 4.87s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '944.6666666666666' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/data/transforms.py:225: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if is_categorical_dtype(col):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/data/transforms.py:225: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if is_categorical_dtype(col):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
0.8163 = Validation score (accuracy)
0.55s = Training runtime
0.01s = Validation runtime
Fitting model: LightGBMXT ... Training model for up to 4.29s of the 4.29s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '944.6666666666666' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
0.9694 = Validation score (accuracy)
0.7s = Training runtime
0.01s = Validation runtime
Fitting model: LightGBM ... Training model for up to 3.53s of the 3.53s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '944.6666666666666' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
1.0 = Validation score (accuracy)
0.4s = Training runtime
0.01s = Validation runtime
Fitting model: RandomForestGini ... Training model for up to 3.12s of the 3.12s of remaining time.
0.9082 = Validation score (accuracy)
0.85s = Training runtime
0.08s = Validation runtime
Fitting model: RandomForestEntr ... Training model for up to 2.16s of the 2.16s of remaining time.
0.9082 = Validation score (accuracy)
0.71s = Training runtime
0.08s = Validation runtime
Fitting model: CatBoost ... Training model for up to 1.35s of the 1.35s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '944.6666666666666' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
Ran out of time, early stopping on iteration 72.
0.8878 = Validation score (accuracy)
1.33s = Training runtime
0.01s = Validation runtime
Fitting model: ExtraTreesGini ... Training model for up to 0.01s of the 0.01s of remaining time.
Warning: Model is expected to require 1.1s to train, which exceeds the maximum time limit of 0.0s, skipping model...
Time limit exceeded... Skipping ExtraTreesGini.
Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.91s of the -0.16s of remaining time.
1.0 = Validation score (accuracy)
0.42s = Training runtime
0.0s = Validation runtime
AutoGluon training complete, total runtime = 5.6s ... Best model: "WeightedEnsemble_L2"
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("agModels-predictEducationClassPredictor_education")
Beginning AutoGluon training ... Time limit = 5s
AutoGluon will save models to "agModels-predictEducationClassPredictor_class"
AutoGluon Version: 0.8.3b20231012
Python Version: 3.10.8
Operating System: Linux
Platform Machine: x86_64
Platform Version: #1 SMP Tue Nov 30 00:17:50 UTC 2021
Disk Space Avail: 230.96 GB / 274.87 GB (84.0%)
Train Data Rows: 500
Train Data Columns: 14
Label Column: class
Preprocessing data ...
/home/ci/autogluon/tabular/src/autogluon/tabular/learner/default_learner.py:215: FutureWarning: use_inf_as_na option is deprecated and will be removed in a future version. Convert inf values to NaN before operating instead.
with pd.option_context("mode.use_inf_as_na", True): # treat None, NaN, INF, NINF as NA
Selected class <--> label mapping: class 1 = >50K, class 0 = <=50K
Note: For your binary classification, AutoGluon arbitrarily selected which label-value represents positive ( >50K) vs negative ( <=50K) class.
To explicitly set the positive_class, either rename classes to 1 and 0, or specify positive_class in Predictor init.
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
Available Memory: 30614.01 MB
Train Data (Original) Memory Usage: 0.29 MB (0.0% of available memory)
Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
Stage 1 Generators:
Fitting AsTypeFeatureGenerator...
Note: Converting 1 features to boolean dtype as they only contain 2 unique values.
Stage 2 Generators:
Fitting FillNaFeatureGenerator...
/home/ci/autogluon/features/src/autogluon/features/generators/fillna.py:58: FutureWarning: The 'downcast' keyword in fillna is deprecated and will be removed in a future version. Use res.infer_objects(copy=False) to infer non-object dtype, or pd.to_numeric with the 'downcast' keyword to downcast numeric results.
X.fillna(self._fillna_feature_map, inplace=True, downcast=False)
Stage 3 Generators:
Fitting IdentityFeatureGenerator...
Fitting CategoryFeatureGenerator...
Fitting CategoryMemoryMinimizeFeatureGenerator...
Stage 4 Generators:
Fitting DropUniqueFeatureGenerator...
Stage 5 Generators:
Fitting DropDuplicatesFeatureGenerator...
Types of features in original data (raw dtype, special dtypes):
('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
('object', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]
Types of features in processed data (raw dtype, special dtypes):
('category', []) : 7 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]
('int', []) : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
('int', ['bool']) : 1 | ['sex']
0.1s = Fit runtime
14 features in original data used to generate 14 features in processed data.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '1097.3333333333335' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
Train Data (Processed) Memory Usage: 0.03 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 0.1s ...
AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'
To change this, specify the eval_metric parameter of Predictor()
Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 400, Val Rows: 100
User-specified model hyperparameters to be fit:
{
'NN_TORCH': {},
'GBM': [{'extra_trees': True, 'ag_args': {'name_suffix': 'XT'}}, {}, 'GBMLarge'],
'CAT': {},
'XGB': {},
'FASTAI': {},
'RF': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],
'XT': [{'criterion': 'gini', 'ag_args': {'name_suffix': 'Gini', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'entropy', 'ag_args': {'name_suffix': 'Entr', 'problem_types': ['binary', 'multiclass']}}, {'criterion': 'squared_error', 'ag_args': {'name_suffix': 'MSE', 'problem_types': ['regression', 'quantile']}}],
'KNN': [{'weights': 'uniform', 'ag_args': {'name_suffix': 'Unif'}}, {'weights': 'distance', 'ag_args': {'name_suffix': 'Dist'}}],
}
Fitting 13 L1 models ...
Fitting model: KNeighborsUnif ... Training model for up to 4.9s of the 4.9s of remaining time.
0.73 = Validation score (accuracy)
0.01s = Training runtime
0.01s = Validation runtime
Fitting model: KNeighborsDist ... Training model for up to 4.88s of the 4.88s of remaining time.
0.65 = Validation score (accuracy)
0.0s = Training runtime
0.01s = Validation runtime
Fitting model: LightGBMXT ... Training model for up to 4.86s of the 4.86s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '997.3333333333334' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
0.83 = Validation score (accuracy)
0.19s = Training runtime
0.01s = Validation runtime
Fitting model: LightGBM ... Training model for up to 4.66s of the 4.66s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '997.3333333333334' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
0.85 = Validation score (accuracy)
0.22s = Training runtime
0.01s = Validation runtime
Fitting model: RandomForestGini ... Training model for up to 4.42s of the 4.42s of remaining time.
0.84 = Validation score (accuracy)
0.49s = Training runtime
0.06s = Validation runtime
Fitting model: RandomForestEntr ... Training model for up to 3.86s of the 3.86s of remaining time.
0.83 = Validation score (accuracy)
0.47s = Training runtime
0.06s = Validation runtime
Fitting model: CatBoost ... Training model for up to 3.31s of the 3.31s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '997.3333333333334' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
0.85 = Validation score (accuracy)
0.77s = Training runtime
0.01s = Validation runtime
Fitting model: ExtraTreesGini ... Training model for up to 2.53s of the 2.53s of remaining time.
0.82 = Validation score (accuracy)
0.48s = Training runtime
0.06s = Validation runtime
Fitting model: ExtraTreesEntr ... Training model for up to 1.98s of the 1.98s of remaining time.
0.81 = Validation score (accuracy)
0.48s = Training runtime
0.06s = Validation runtime
Fitting model: NeuralNetFastAI ... Training model for up to 1.42s of the 1.41s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '997.3333333333334' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/data/transforms.py:225: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if is_categorical_dtype(col):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/data/transforms.py:225: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if is_categorical_dtype(col):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
/home/ci/opt/venv/lib/python3.10/site-packages/fastai/tabular/core.py:233: FutureWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, CategoricalDtype) instead
if not is_categorical_dtype(c):
0.82 = Validation score (accuracy)
0.53s = Training runtime
0.01s = Validation runtime
Fitting model: XGBoost ... Training model for up to 0.87s of the 0.87s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '997.3333333333334' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
/home/ci/opt/venv/lib/python3.10/site-packages/xgboost/data.py:440: FutureWarning: is_sparse is deprecated and will be removed in a future version. Check `isinstance(dtype, pd.SparseDtype)` instead.
if is_sparse(data):
0.87 = Validation score (accuracy)
0.31s = Training runtime
0.01s = Validation runtime
Fitting model: NeuralNetTorch ... Training model for up to 0.54s of the 0.54s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '997.3333333333334' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
Not enough time to train first epoch. (Time Required: 0.52s, Time Left: 0.5s)
Time limit exceeded... Skipping NeuralNetTorch.
Fitting model: LightGBMLarge ... Training model for up to 0.3s of the 0.3s of remaining time.
/home/ci/autogluon/common/src/autogluon/common/utils/pandas_utils.py:50: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '997.3333333333334' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.
memory_usage[column] = (
Ran out of time, early stopping on iteration 269. Best iteration is:
[67] valid_set's binary_error: 0.17
0.83 = Validation score (accuracy)
0.32s = Training runtime
0.01s = Validation runtime
Fitting model: WeightedEnsemble_L2 ... Training model for up to 4.9s of the -0.03s of remaining time.
0.87 = Validation score (accuracy)
0.65s = Training runtime
0.0s = Validation runtime
AutoGluon training complete, total runtime = 5.71s ... Best model: "WeightedEnsemble_L2"
TabularPredictor saved. To load, use: predictor = TabularPredictor.load("agModels-predictEducationClassPredictor_class")
Inference and Evaluation#
After training, you can easily use the MultilabelPredictor to predict all labels in new data:
test_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv')
test_data = test_data.sample(n=subsample_size, random_state=0)
test_data_nolab = test_data.drop(columns=labels) # unnecessary, just to demonstrate we're not cheating here
test_data_nolab.head()
Loaded data from: https://autogluon.s3.amazonaws.com/datasets/Inc/test.csv | Columns = 15 / 15 | Rows = 9769 -> 9769
| age | workclass | fnlwgt | marital-status | occupation | relationship | race | sex | capital-gain | capital-loss | hours-per-week | native-country | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 5454 | 41 | Self-emp-not-inc | 408498 | Married-civ-spouse | Exec-managerial | Husband | White | Male | 0 | 0 | 50 | United-States |
| 6111 | 39 | Private | 746786 | Married-civ-spouse | Prof-specialty | Husband | White | Male | 0 | 0 | 55 | United-States |
| 5282 | 50 | Private | 62593 | Married-civ-spouse | Farming-fishing | Husband | Asian-Pac-Islander | Male | 0 | 0 | 40 | United-States |
| 3046 | 31 | Private | 248178 | Married-civ-spouse | Other-service | Husband | Black | Male | 0 | 0 | 35 | United-States |
| 2162 | 43 | State-gov | 52849 | Married-civ-spouse | Prof-specialty | Husband | White | Male | 0 | 0 | 40 | United-States |
multi_predictor = MultilabelPredictor.load(save_path) # unnecessary, just demonstrates how to load previously-trained multilabel predictor from file
predictions = multi_predictor.predict(test_data_nolab)
print("Predictions: \n", predictions)
---------------------------------------------------------------------------
FileNotFoundError Traceback (most recent call last)
Cell In[7], line 1
----> 1 multi_predictor = MultilabelPredictor.load(save_path) # unnecessary, just demonstrates how to load previously-trained multilabel predictor from file
3 predictions = multi_predictor.predict(test_data_nolab)
4 print("Predictions: \n", predictions)
Cell In[2], line 158, in MultilabelPredictor.load(cls, path)
156 if path[-1] != os.path.sep:
157 path = path + os.path.sep
--> 158 return load_pkl.load(path=path+cls.multi_predictor_file)
File ~/autogluon/common/src/autogluon/common/loaders/load_pkl.py:43, in load(path, format, verbose, **kwargs)
40 compression_fn_kwargs = {}
42 if compression_fn in compression_fn_map:
---> 43 with compression_fn_map[compression_fn]["open"](validated_path, "rb", **compression_fn_kwargs) as fin:
44 object = pickle.load(fin)
45 else:
FileNotFoundError: [Errno 2] No such file or directory: 'agModels-predictEducationClass/multilabel_predictor.pkl'
We can also easily evaluate the performance of our predictions if our new data contain the ground truth labels:
evaluations = multi_predictor.evaluate(test_data)
print(evaluations)
print("Evaluated using metrics:", multi_predictor.eval_metrics)
Accessing the TabularPredictor for One Label#
We can also directly work with the TabularPredictor for any one of the labels as follows. However we recommend you set consider_labels_correlation=False before training if you later plan to use an individual TabularPredictor to predict just one label rather than all of the labels predicted by the MultilabelPredictor.
predictor_class = multi_predictor.get_predictor('class')
predictor_class.leaderboard(silent=True)
Tips#
In order to obtain the best predictions, you should generally add the following arguments to MultilabelPredictor.fit():
Specify
eval_metricsto the metrics you will use to evaluate predictions for each labelSpecify
presets='best_quality'to tell AutoGluon you care about predictive performance more than latency/memory usage, which will utilize stack ensembling when predicting each label.
If you find that too much memory/disk is being used, try calling MultilabelPredictor.fit() with additional arguments discussed under “If you encounter memory issues” in the In Depth Tutorial or “If you encounter disk space issues”.
If you find inference too slow, you can try the strategies discussed under “Accelerating Inference” in the In Depth Tutorial.
In particular, simply try specifying the following preset in MultilabelPredictor.fit(): presets = ['good_quality', 'optimize_for_deployment']