AutoGluon Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Models¶
Foundation models have transformed landscapes across fields like computer vision and natural language processing. These models are pre-trained on extensive common-domain data, serving as powerful tools for a wide range of applications. However, seamlessly integrating foundation models into real-world application scenarios has posed challenges. The diversity of data modalities, the multitude of available foundation models, and the considerable model sizes make this integration a nontrivial task.
AutoMM is dedicated to breaking these barriers by substantially reducing the engineering effort and manual intervention required in data preprocessing, model selection, and fine-tuning. With AutoMM, users can effortlessly adapt foundation models (from popular model zoos like HuggingFace, TIMM, MMDetection) to their domain-specific data using just three lines of code. Our toolkit accommodates various data types, including image, text, tabular, and document data, either individually or in combination. It offers support for an array of tasks, encompassing classification, regression, object detection, named entity recognition, semantic matching, and image segmentation. AutoMM represents a state-of-the-art and user-friendly solution, empowering multimodal AutoML with foundation models.
In the following, we decompose the functionalities of AutoMM and prepare step-by-step guide for each functionality.
Text Data – Classification / Regression / NER¶
:gutter: 3
:link: text_prediction/beginner_text.html
How to train high-quality text prediction models with AutoMM.
:link: text_prediction/multilingual_text.html
How to use AutoMM to build models on datasets with languages other than English.
:link: text_prediction/ner.html
How to use AutoMM for entity extraction.
Image Data – Classification / Regression¶
:gutter: 3
:link: image_prediction/beginner_image_cls.html
How to train image classification models with AutoMM.
:link: image_prediction/clip_zeroshot.html
How to enable zero-shot image classification in AutoMM via pretrained CLIP model.
Image Data – Object Detection¶
:gutter: 3
:link: object_detection/quick_start/quick_start_coco.html
How to train high quality object detection model with AutoMM in under 5 minutes on COCO format dataset.
:link: object_detection/data_preparation/prepare_coco17.html
How to prepare COCO2017 dataset for object detection.
:link: object_detection/data_preparation/prepare_voc.html
How to prepare Pascal VOC dataset for object detection.
:link: object_detection/data_preparation/prepare_watercolor.html
How to prepare Watercolor dataset for object detection.
:link: object_detection/data_preparation/voc_to_coco.html
How to convert a dataset from VOC format to COCO format for object detection.
:link: object_detection/data_preparation/object_detection_with_dataframe.html
How to use pd.DataFrame format for object detection
Image Data – Segmentation¶
:gutter: 3
:link: image_segmentation/beginner_semantic_seg.html
How to train semantic segmentation models with AutoMM.
Document Data – Classification / Regression¶
:gutter: 3
:link: document_prediction/document_classification.html
How to use AutoMM to build a scanned document classifier.
:link: document_prediction/pdf_classification.html
How to use AutoMM to build a PDF document classifier.
Image / Text Data – Semantic Matching¶
:gutter: 3
:link: semantic_matching/text2text_matching.html
How to use AutoMM for text-to-text semantic matching.
:link: semantic_matching/image2image_matching.html
How to use AutoMM for image-to-image semantic matching.
:link: semantic_matching/image_text_matching.html
How to use AutoMM for image-text semantic matching.
:link: semantic_matching/zero_shot_img_txt_matching.html
How to use AutoMM for zero shot image-text semantic matching.
:link: semantic_matching/text_semantic_search.html
How to use semantic embeddings to improve search ranking performance.
Multimodal Data – Classification / Regression / NER¶
:gutter: 3
:link: multimodal_prediction/multimodal_text_tabular.html
How AutoMM can be applied to multimodal data tables with a mix of text, numerical, and categorical columns.
:link: multimodal_prediction/beginner_multimodal.html
How to use AutoMM to train a model on image, text, numerical, and categorical data.
:link: multimodal_prediction/multimodal_ner.html
How to use AutoMM to train a model for multimodal named entity recognition.
Advanced Topics¶
:gutter: 3
:link: advanced_topics/efficient_finetuning_basic.html
How to take advantage of larger foundation models with the help of parameter-efficient finetuning. In the tutorial, we will use combine IA^3, BitFit, and gradient checkpointing to finetune FLAN-T5-XL.
:link: advanced_topics/hyperparameter_optimization.html
How to do hyperparameter optimization in AutoMM.
:link: advanced_topics/model_distillation.html
How to do knowledge distillation in AutoMM.
:link: advanced_topics/continuous_training.html
How to continue training in AutoMM.
:link: advanced_topics/customization.html
How to customize AutoMM configurations.
:link: advanced_topics/presets.html
How to use AutoMM presets.
:link: advanced_topics/few_shot_learning.html
How to use foundation models + SVM for few shot learning.
:link: advanced_topics/focal_loss.html
How to use AutoMM to handle class imbalance.
:link: advanced_topics/tensorrt.html
How to use TensorRT in accelerating AutoMM model inference.