.. _sec_tabularinterpretability:

Interpretable rule-based modeling
=================================


*Note*: This addition was made through collaboration with `the
Yu-Group <https://www.stat.berkeley.edu/~yugroup/>`__ at UC Berkeley.

**Tip**: Prior to reading this tutorial, it is recommended to have a
basic understanding of the TabularPredictor API covered in
:ref:`sec_tabularquick`.

In this tutorial, we will explain how to automatically use interpretable
models powered by integration with `🔍 the imodels
package <https://github.com/csinva/imodels>`__. This allows for
automatically learning models based on rules which are extremely concise
and can be useful for (1) understanding data or (2) building a
transparent predictive model.

Begin by loading in data to predict. Note: interpretable rule-based
modeling is currently only supported for binary classification.

.. code:: python

    from autogluon.tabular import TabularDataset, TabularPredictor
    train_data = TabularDataset('https://autogluon.s3.amazonaws.com/datasets/Inc/train.csv')
    subsample_size = 500  # subsample subset of data for faster demo, try setting this to much larger values
    train_data = train_data.sample(n=subsample_size, random_state=0)
    train_data.head()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>age</th>
          <th>workclass</th>
          <th>fnlwgt</th>
          <th>education</th>
          <th>education-num</th>
          <th>marital-status</th>
          <th>occupation</th>
          <th>relationship</th>
          <th>race</th>
          <th>sex</th>
          <th>capital-gain</th>
          <th>capital-loss</th>
          <th>hours-per-week</th>
          <th>native-country</th>
          <th>class</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>6118</th>
          <td>51</td>
          <td>Private</td>
          <td>39264</td>
          <td>Some-college</td>
          <td>10</td>
          <td>Married-civ-spouse</td>
          <td>Exec-managerial</td>
          <td>Wife</td>
          <td>White</td>
          <td>Female</td>
          <td>0</td>
          <td>0</td>
          <td>40</td>
          <td>United-States</td>
          <td>&gt;50K</td>
        </tr>
        <tr>
          <th>23204</th>
          <td>58</td>
          <td>Private</td>
          <td>51662</td>
          <td>10th</td>
          <td>6</td>
          <td>Married-civ-spouse</td>
          <td>Other-service</td>
          <td>Wife</td>
          <td>White</td>
          <td>Female</td>
          <td>0</td>
          <td>0</td>
          <td>8</td>
          <td>United-States</td>
          <td>&lt;=50K</td>
        </tr>
        <tr>
          <th>29590</th>
          <td>40</td>
          <td>Private</td>
          <td>326310</td>
          <td>Some-college</td>
          <td>10</td>
          <td>Married-civ-spouse</td>
          <td>Craft-repair</td>
          <td>Husband</td>
          <td>White</td>
          <td>Male</td>
          <td>0</td>
          <td>0</td>
          <td>44</td>
          <td>United-States</td>
          <td>&lt;=50K</td>
        </tr>
        <tr>
          <th>18116</th>
          <td>37</td>
          <td>Private</td>
          <td>222450</td>
          <td>HS-grad</td>
          <td>9</td>
          <td>Never-married</td>
          <td>Sales</td>
          <td>Not-in-family</td>
          <td>White</td>
          <td>Male</td>
          <td>0</td>
          <td>2339</td>
          <td>40</td>
          <td>El-Salvador</td>
          <td>&lt;=50K</td>
        </tr>
        <tr>
          <th>33964</th>
          <td>62</td>
          <td>Private</td>
          <td>109190</td>
          <td>Bachelors</td>
          <td>13</td>
          <td>Married-civ-spouse</td>
          <td>Exec-managerial</td>
          <td>Husband</td>
          <td>White</td>
          <td>Male</td>
          <td>15024</td>
          <td>0</td>
          <td>40</td>
          <td>United-States</td>
          <td>&gt;50K</td>
        </tr>
      </tbody>
    </table>
    </div>



Now, we create a predictor and fit it to the data. By specifying
``presets='interpretable'``, we tell the predictor to fit only
interpretable models.

.. code:: python

    predictor = TabularPredictor(label='class')
    predictor.fit(train_data, presets='interpretable')
    predictor.leaderboard()


.. parsed-literal::
    :class: output

    No path specified. Models will be saved in: "AutogluonModels/ag-20220512_215830/"
    Presets specified: ['interpretable']
    Beginning AutoGluon training ...
    AutoGluon will save models to "AutogluonModels/ag-20220512_215830/"
    AutoGluon Version:  0.4.1b20220512
    Python Version:     3.9.12
    Operating System:   Linux
    Train Data Rows:    500
    Train Data Columns: 14
    Label Column: class
    Preprocessing data ...
    AutoGluon infers your prediction problem is: 'binary' (because only two unique label-values observed).
    	2 unique label values:  [' >50K', ' <=50K']
    	If 'binary' is not the correct problem_type, please manually specify the problem_type parameter during predictor init (You may specify problem_type as one of: ['binary', 'multiclass', 'regression'])
    Selected class <--> label mapping:  class 1 =  >50K, class 0 =  <=50K
    	Note: For your binary classification, AutoGluon arbitrarily selected which label-value represents positive ( >50K) vs negative ( <=50K) class.
    	To explicitly set the positive_class, either rename classes to 1 and 0, or specify positive_class in Predictor init.
    Using Feature Generators to preprocess the data ...
    Fitting AutoMLPipelineFeatureGenerator...
    	Available Memory:                    22882.25 MB
    	Train Data (Original)  Memory Usage: 0.29 MB (0.0% of available memory)
    	Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
    	Stage 1 Generators:
    		Fitting AsTypeFeatureGenerator...
    			Note: Converting 1 features to boolean dtype as they only contain 2 unique values.
    	Stage 2 Generators:
    		Fitting FillNaFeatureGenerator...
    	Stage 3 Generators:
    		Fitting IdentityFeatureGenerator...
    		Fitting CategoryFeatureGenerator...
    			Fitting CategoryMemoryMinimizeFeatureGenerator...
    	Stage 4 Generators:
    		Fitting DropUniqueFeatureGenerator...
    	Types of features in original data (raw dtype, special dtypes):
    		('int', [])    : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
    		('object', []) : 8 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]
    	Types of features in processed data (raw dtype, special dtypes):
    		('category', [])  : 7 | ['workclass', 'education', 'marital-status', 'occupation', 'relationship', ...]
    		('int', [])       : 6 | ['age', 'fnlwgt', 'education-num', 'capital-gain', 'capital-loss', ...]
    		('int', ['bool']) : 1 | ['sex']
    	0.1s = Fit runtime
    	14 features in original data used to generate 14 features in processed data.
    	Train Data (Processed) Memory Usage: 0.03 MB (0.0% of available memory)
    Data preprocessing and feature engineering runtime = 0.08s ...
    AutoGluon will gauge predictive performance using evaluation metric: 'accuracy'
    	To change this, specify the eval_metric parameter of Predictor()
    Automatically generating train/validation split with holdout_frac=0.2, Train Rows: 400, Val Rows: 100
    Fitting 6 L1 models ...
    Fitting model: RuleFit ...
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    	0.73	 = Validation score   (accuracy)
    	1.73s	 = Training   runtime
    	0.03s	 = Validation runtime
    Fitting model: RuleFit_2 ...
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    	0.73	 = Validation score   (accuracy)
    	1.35s	 = Training   runtime
    	0.05s	 = Validation runtime
    Fitting model: RuleFit_3 ...
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    	0.81	 = Validation score   (accuracy)
    	1.5s	 = Training   runtime
    	0.07s	 = Validation runtime
    Fitting model: GreedyTree ...
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    	0.81	 = Validation score   (accuracy)
    	0.02s	 = Training   runtime
    	0.01s	 = Validation runtime
    Fitting model: BoostedRules ...
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    	0.78	 = Validation score   (accuracy)
    	0.02s	 = Training   runtime
    	0.01s	 = Validation runtime
    Fitting model: BoostedRules_2 ...
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    	0.82	 = Validation score   (accuracy)
    	0.03s	 = Training   runtime
    	0.01s	 = Validation runtime
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    /var/lib/jenkins/miniconda3/envs/autogluon-tutorial-tabular-v3/lib/python3.9/site-packages/sklearn/utils/validation.py:624: UserWarning: pandas.DataFrame with sparse columns found.It will be converted to a dense numpy array.
      warnings.warn(
    Fitting model: WeightedEnsemble_L2 ...
    	0.85	 = Validation score   (accuracy)
    	0.13s	 = Training   runtime
    	0.0s	 = Validation runtime
    AutoGluon training complete, total runtime = 5.21s ... Best model: "WeightedEnsemble_L2"
    TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20220512_215830/")


.. parsed-literal::
    :class: output

                     model  score_val  pred_time_val  fit_time  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer  fit_order
    0  WeightedEnsemble_L2       0.85       0.093246  1.678375                0.000478           0.125870            2       True          7
    1       BoostedRules_2       0.82       0.010010  0.026565                0.010010           0.026565            1       True          6
    2           GreedyTree       0.81       0.009116  0.023104                0.009116           0.023104            1       True          4
    3            RuleFit_3       0.81       0.073642  1.502836                0.073642           1.502836            1       True          3
    4         BoostedRules       0.78       0.010142  0.020632                0.010142           0.020632            1       True          5
    5              RuleFit       0.73       0.029439  1.733647                0.029439           1.733647            1       True          1
    6            RuleFit_2       0.73       0.054927  1.354842                0.054927           1.354842            1       True          2




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>model</th>
          <th>score_val</th>
          <th>pred_time_val</th>
          <th>fit_time</th>
          <th>pred_time_val_marginal</th>
          <th>fit_time_marginal</th>
          <th>stack_level</th>
          <th>can_infer</th>
          <th>fit_order</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>WeightedEnsemble_L2</td>
          <td>0.85</td>
          <td>0.093246</td>
          <td>1.678375</td>
          <td>0.000478</td>
          <td>0.125870</td>
          <td>2</td>
          <td>True</td>
          <td>7</td>
        </tr>
        <tr>
          <th>1</th>
          <td>BoostedRules_2</td>
          <td>0.82</td>
          <td>0.010010</td>
          <td>0.026565</td>
          <td>0.010010</td>
          <td>0.026565</td>
          <td>1</td>
          <td>True</td>
          <td>6</td>
        </tr>
        <tr>
          <th>2</th>
          <td>GreedyTree</td>
          <td>0.81</td>
          <td>0.009116</td>
          <td>0.023104</td>
          <td>0.009116</td>
          <td>0.023104</td>
          <td>1</td>
          <td>True</td>
          <td>4</td>
        </tr>
        <tr>
          <th>3</th>
          <td>RuleFit_3</td>
          <td>0.81</td>
          <td>0.073642</td>
          <td>1.502836</td>
          <td>0.073642</td>
          <td>1.502836</td>
          <td>1</td>
          <td>True</td>
          <td>3</td>
        </tr>
        <tr>
          <th>4</th>
          <td>BoostedRules</td>
          <td>0.78</td>
          <td>0.010142</td>
          <td>0.020632</td>
          <td>0.010142</td>
          <td>0.020632</td>
          <td>1</td>
          <td>True</td>
          <td>5</td>
        </tr>
        <tr>
          <th>5</th>
          <td>RuleFit</td>
          <td>0.73</td>
          <td>0.029439</td>
          <td>1.733647</td>
          <td>0.029439</td>
          <td>1.733647</td>
          <td>1</td>
          <td>True</td>
          <td>1</td>
        </tr>
        <tr>
          <th>6</th>
          <td>RuleFit_2</td>
          <td>0.73</td>
          <td>0.054927</td>
          <td>1.354842</td>
          <td>0.054927</td>
          <td>1.354842</td>
          <td>1</td>
          <td>True</td>
          <td>2</td>
        </tr>
      </tbody>
    </table>
    </div>



The rule-based models take slightly different forms (see below), but all
try to optimize predictive performance using as few rules as possible.
See `imodels package <https://github.com/csinva/imodels>`__ for more
details.

.. figure:: https://raw.githubusercontent.com/csinva/imodels/master/docs/img/model_table_rules.png

In addition to the usual functions in ``TabularPredictor``, this
predictor fitted with interpretable models has some additional
functionality. For example, we can now inspect the complexity of the
fitted models (i.e. how many rules they contain).

.. code:: python

    predictor.interpretable_models_summary()




.. raw:: html

    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }
    
        .dataframe tbody tr th {
            vertical-align: top;
        }
    
        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>model_types</th>
          <th>model_performance</th>
          <th>complexity</th>
          <th>model_best</th>
          <th>model_paths</th>
          <th>model_fit_times</th>
          <th>model_pred_times</th>
          <th>num_bag_folds</th>
          <th>max_stack_level</th>
          <th>num_classes</th>
          <th>model_hyperparams</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>BoostedRules_2</th>
          <td>BoostedRulesModel</td>
          <td>0.82</td>
          <td>20.0</td>
          <td>WeightedEnsemble_L2</td>
          <td>AutogluonModels/ag-20220512_215830/models/Boos...</td>
          <td>0.026565</td>
          <td>0.010010</td>
          <td>0</td>
          <td>2</td>
          <td>2</td>
          <td>{'random_state': 0, 'n_estimators': 10}</td>
        </tr>
        <tr>
          <th>GreedyTree</th>
          <td>GreedyTreeModel</td>
          <td>0.81</td>
          <td>19.0</td>
          <td>WeightedEnsemble_L2</td>
          <td>AutogluonModels/ag-20220512_215830/models/Gree...</td>
          <td>0.023104</td>
          <td>0.009116</td>
          <td>0</td>
          <td>2</td>
          <td>2</td>
          <td>{'random_state': 0, 'max_leaf_nodes': 20}</td>
        </tr>
        <tr>
          <th>RuleFit_3</th>
          <td>RuleFitModel</td>
          <td>0.81</td>
          <td>49.0</td>
          <td>WeightedEnsemble_L2</td>
          <td>AutogluonModels/ag-20220512_215830/models/Rule...</td>
          <td>1.502836</td>
          <td>0.073642</td>
          <td>0</td>
          <td>2</td>
          <td>2</td>
          <td>{'random_state': 0, 'max_rules': 20}</td>
        </tr>
        <tr>
          <th>BoostedRules</th>
          <td>BoostedRulesModel</td>
          <td>0.78</td>
          <td>10.0</td>
          <td>WeightedEnsemble_L2</td>
          <td>AutogluonModels/ag-20220512_215830/models/Boos...</td>
          <td>0.020632</td>
          <td>0.010142</td>
          <td>0</td>
          <td>2</td>
          <td>2</td>
          <td>{'random_state': 0, 'n_estimators': 5}</td>
        </tr>
        <tr>
          <th>RuleFit</th>
          <td>RuleFitModel</td>
          <td>0.73</td>
          <td>8.0</td>
          <td>WeightedEnsemble_L2</td>
          <td>AutogluonModels/ag-20220512_215830/models/Rule...</td>
          <td>1.733647</td>
          <td>0.029439</td>
          <td>0</td>
          <td>2</td>
          <td>2</td>
          <td>{'random_state': 0, 'max_rules': 7}</td>
        </tr>
        <tr>
          <th>RuleFit_2</th>
          <td>RuleFitModel</td>
          <td>0.73</td>
          <td>29.0</td>
          <td>WeightedEnsemble_L2</td>
          <td>AutogluonModels/ag-20220512_215830/models/Rule...</td>
          <td>1.354842</td>
          <td>0.054927</td>
          <td>0</td>
          <td>2</td>
          <td>2</td>
          <td>{'random_state': 0, 'max_rules': 12}</td>
        </tr>
      </tbody>
    </table>
    </div>



We can also explicitly inspect the rules of the best-performing model.

.. code:: python

    predictor.print_interpretable_rules() # can optionally specify a model name or complexity threshold


.. parsed-literal::
    :class: output

    BoostedRules:
    Rule → predicted probability (final prediction is weighted sum of all predictions)
      If marital-status_1.0 <= 0.5 → 0.00 (weight: 0.50)
      If marital-status_1.0 > 0.5 → 0.51 (weight: 0.56)
      If marital-status_1.0 <= 0.5 → 0.29 (weight: 0.46)
      If marital-status_1.0 > 0.5 → 0.06 (weight: 0.24)
      If education-num <= 11.5 → 0.79 (weight: 0.35)
    


In some cases, these rules are sufficient to accurately make
predictions. In other cases, they may just be used to gain a better
understanding of the data before proceeding with more black-box models.