Object Detection - Quick Start¶
Object detection is the process of identifying and localizing objects in an image and is an important task in computer vision. Follow this tutorial to learn how to use AutoGluon for object detection.
Tip: If you are new to AutoGluon, review Image Prediction - Quick Start first to learn the basics of the AutoGluon API.
Our goal is to detect motorbike in images by YOLOv3 model. A tiny dataset is collected from VOC dataset, which only contains the motorbike category. The model pretrained on the COCO dataset is used to fine-tune our small dataset. With the help of AutoGluon, we are able to try many models with different hyperparameters automatically, and return the best one as our final model.
To start, import autogluon.vision and ObjectDetector:
import autogluon.core as ag
from autogluon.vision import ObjectDetector
/var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/core/src/autogluon/core/scheduler/jobs.py:132: SyntaxWarning: "is" with a literal. Did you mean "=="?
file = sys.stderr if out is 'err' else sys.stdout
Tiny_motorbike Dataset¶
We collect a toy dataset for detecting motorbikes in images. From the VOC dataset, images are randomly selected for training, validation, and testing - 120 images for training, 50 images for validation, and 50 for testing. This tiny dataset follows the same format as VOC.
Using the commands below, we can download this dataset, which is only
23M. The name of unzipped folder is called tiny_motorbike. Anyway,
the task dataset helper can perform the download and extraction
automatically, and load the dataset according to the detection formats.
url = 'https://autogluon.s3.amazonaws.com/datasets/tiny_motorbike.zip'
dataset_train = ObjectDetector.Dataset.from_voc(url, splits='trainval')
tiny_motorbike/
├── Annotations/
├── ImageSets/
└── JPEGImages/
Fit Models by AutoGluon¶
In this section, we demonstrate how to apply AutoGluon to fit our detection models. We use mobilenet as the backbone for the YOLOv3 model. Two different learning rates are used to fine-tune the network. The best model is the one that obtains the best performance on the validation dataset. You can also try using more networks and hyperparameters to create a larger searching space.
We fit a classifier using AutoGluon as follows. In each experiment
(one trial in our searching space), we train the model for 5 epochs to
avoid bursting our tutorial runtime.
time_limit = 60*30 # at most 0.5 hour
detector = ObjectDetector()
hyperparameters = {'epochs': 5, 'batch_size': 8}
hyperparamter_tune_kwargs={'num_trials': 2}
detector.fit(dataset_train, time_limit=time_limit, hyperparameters=hyperparameters, hyperparamter_tune_kwargs=hyperparamter_tune_kwargs)
WARNING:gluoncv.auto.tasks.object_detection:The number of requested GPUs is greater than the number of available GPUs.Reduce the number to 1
INFO:gluoncv.auto.tasks.object_detection:Randomly split train_data into train[145]/validation[25] splits.
INFO:gluoncv.auto.tasks.object_detection:Starting fit without HPO
INFO:SSDEstimator:modified configs(<old> != <new>): {
INFO:SSDEstimator:root.train.epochs 20 != 5
INFO:SSDEstimator:root.train.seed 233 != 509
INFO:SSDEstimator:root.train.batch_size 16 != 8
INFO:SSDEstimator:root.dataset_root ~/.mxnet/datasets/ != auto
INFO:SSDEstimator:root.num_workers 4 != 8
INFO:SSDEstimator:root.valid.batch_size 16 != 8
INFO:SSDEstimator:root.ssd.data_shape 300 != 512
INFO:SSDEstimator:root.ssd.base_network vgg16_atrous != resnet50_v1
INFO:SSDEstimator:root.gpus (0, 1, 2, 3) != (0,)
INFO:SSDEstimator:root.dataset voc_tiny != auto
INFO:SSDEstimator:}
INFO:SSDEstimator:Saved config to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/config.yaml
INFO:SSDEstimator:Using transfer learning from ssd_512_resnet50_v1_coco, the other network parameters are ignored.
INFO:SSDEstimator:Start training from [Epoch 0]
INFO:SSDEstimator:[Epoch 0] Training cost: 10.752224, CrossEntropy=3.568488, SmoothL1=1.019028
INFO:SSDEstimator:[Epoch 0] Validation:
bus=0.6363636363636365
motorbike=0.6693062911160786
cow=0.6363636363636365
person=0.6370631942591196
dog=1.0000000000000002
pottedplant=0.0
bicycle=0.02394678492239468
boat=1.0000000000000002
chair=0.0
car=0.7109225007144885
mAP=0.5313966043739355
INFO:SSDEstimator:[Epoch 0] Current best map: 0.531397 vs previous 0.000000, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:SSDEstimator:Pickled to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:SSDEstimator:[Epoch 1] Training cost: 9.368550, CrossEntropy=2.739953, SmoothL1=1.168742
INFO:SSDEstimator:[Epoch 1] Validation:
bus=1.0000000000000002
motorbike=0.8653889340484672
cow=0.0
person=0.6933378557071507
dog=1.0000000000000002
pottedplant=0.0
bicycle=0.051448551448551455
boat=1.0000000000000002
chair=0.0
car=0.7737771989673613
mAP=0.5383952540171532
INFO:SSDEstimator:[Epoch 1] Current best map: 0.538395 vs previous 0.531397, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:SSDEstimator:Pickled to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:SSDEstimator:[Epoch 2] Training cost: 9.539678, CrossEntropy=2.413765, SmoothL1=1.065030
INFO:SSDEstimator:[Epoch 2] Validation:
bus=0.8181818181818181
motorbike=0.8642826279334521
cow=0.7090909090909092
person=0.7967784439336412
dog=0.5000000000000001
pottedplant=0.0
bicycle=0.07170761428597028
boat=1.0000000000000002
chair=0.008695652173913044
car=0.8700587305629323
mAP=0.5638795796162637
INFO:SSDEstimator:[Epoch 2] Current best map: 0.563880 vs previous 0.538395, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:SSDEstimator:Pickled to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:SSDEstimator:[Epoch 3] Training cost: 10.124531, CrossEntropy=2.334373, SmoothL1=1.053449
INFO:SSDEstimator:[Epoch 3] Validation:
bus=0.8181818181818181
motorbike=0.8796913277774454
cow=0.909090909090909
person=0.8457892366038237
dog=1.0000000000000002
pottedplant=0.0
bicycle=0.2336987729173655
boat=1.0000000000000002
chair=0.0
car=0.8298320438846947
mAP=0.6516284108456057
INFO:SSDEstimator:[Epoch 3] Current best map: 0.651628 vs previous 0.563880, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:SSDEstimator:Pickled to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:SSDEstimator:[Epoch 4] Training cost: 9.657665, CrossEntropy=2.129051, SmoothL1=0.952331
INFO:SSDEstimator:[Epoch 4] Validation:
bus=1.0000000000000002
motorbike=0.8806782984388191
cow=1.0000000000000002
person=0.8381137021467983
dog=1.0000000000000002
pottedplant=0.0
bicycle=0.19137009035589156
boat=1.0000000000000002
chair=1.0000000000000002
car=0.7840739692463536
mAP=0.7694236060187862
INFO:SSDEstimator:[Epoch 4] Current best map: 0.769424 vs previous 0.651628, saved to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:SSDEstimator:Pickled to /var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b/.trial_0/best_checkpoint.pkl
INFO:gluoncv.auto.tasks.object_detection:Finished, total runtime is 89.91 s
INFO:gluoncv.auto.tasks.object_detection:{ 'best_config': { 'batch_size': 8,
'dist_ip_addrs': None,
'epochs': 5,
'final_fit': False,
'gpus': [0],
'log_dir': '/var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/docs/_build/eval/tutorials/object_detection/6482fb4b',
'lr': 0.001,
'ngpus_per_trial': 8,
'nthreads_per_trial': 128,
'num_trials': 1,
'num_workers': 8,
'search_strategy': 'random',
'seed': 509,
'time_limits': 1800,
'transfer': 'ssd_512_resnet50_v1_coco',
'wall_clock_tick': 1614109777.8227332},
'total_time': 74.24876022338867,
'train_map': 0.7694236060187862,
'valid_map': 0.7694236060187862}
<autogluon.vision.detector.detector.ObjectDetector at 0x7fc6481b60a0>
Note that num_trials=2 above is only used to speed up the tutorial.
In normal practice, it is common to only use time_limit and drop
num_trials. Also note that hyperparameter tuning defaults to random
search. Model-based variants, such as search_strategy='bayesopt' or
search_strategy='bayesopt_hyperband' can be a lot more
sample-efficient.
After fitting, AutoGluon automatically returns the best model among all models in the searching space. From the output, we know the best model is the one trained with the second learning rate. To see how well the returned model performed on test dataset, call detector.evaluate().
dataset_test = ObjectDetector.Dataset.from_voc(url, splits='test')
test_map = detector.evaluate(dataset_test)
print("mAP on test dataset: {}".format(test_map[1][-1]))
tiny_motorbike/
├── Annotations/
├── ImageSets/
└── JPEGImages/
mAP on test dataset: 0.0754835304429625
Below, we randomly select an image from test dataset and show the
predicted class, box and probability over the origin image, stored in
predict_class, predict_rois and predict_score columns,
respectively. You can interpret predict_rois as a dict of (xmin,
ymin, xmax, ymax) proportional to original image size.
image_path = dataset_test.iloc[0]['image']
result = detector.predict(image_path)
print(result)
INFO:numexpr.utils:NumExpr defaulting to 8 threads.
predict_class predict_score 0 motorbike 0.988815
1 person 0.899608
2 motorbike 0.293910
3 motorbike 0.127058
4 person 0.093919
.. ... ...
95 person 0.019599
96 person 0.019576
97 person 0.019567
98 person 0.019563
99 person 0.019538
predict_rois
0 {'xmin': 0.31304341554641724, 'ymin': 0.426152...
1 {'xmin': 0.3929811418056488, 'ymin': 0.3062253...
2 {'xmin': 0.0, 'ymin': 0.6646333932876587, 'xma...
3 {'xmin': 0.3813198506832123, 'ymin': 0.3355947...
4 {'xmin': 0.99007648229599, 'ymin': 0.443583607...
.. ...
95 {'xmin': 0.972899317741394, 'ymin': 0.0, 'xmax...
96 {'xmin': 0.8153835535049438, 'ymin': 0.5468932...
97 {'xmin': 0.49996355175971985, 'ymin': 0.280560...
98 {'xmin': 0.5031661987304688, 'ymin': 0.0202829...
99 {'xmin': 0.8568260073661804, 'ymin': 0.5384305...
[100 rows x 3 columns]
Prediction with multiple images is permitted:
bulk_result = detector.predict(dataset_test)
print(bulk_result)
predict_class predict_score 0 motorbike 0.988815
1 person 0.899608
2 motorbike 0.293910
3 motorbike 0.127058
4 person 0.093919
... ... ...
4300 person 0.024254
4301 person 0.024223
4302 person 0.024162
4303 person 0.024068
4304 person 0.023885
predict_rois 0 {'xmin': 0.31304341554641724, 'ymin': 0.426152...
1 {'xmin': 0.3929811418056488, 'ymin': 0.3062253...
2 {'xmin': 0.0, 'ymin': 0.6646333932876587, 'xma...
3 {'xmin': 0.3813198506832123, 'ymin': 0.3355947...
4 {'xmin': 0.99007648229599, 'ymin': 0.443583607...
... ...
4300 {'xmin': 0.8013302683830261, 'ymin': 0.1425867...
4301 {'xmin': 0.0, 'ymin': 0.18318504095077515, 'xm...
4302 {'xmin': 0.02827780321240425, 'ymin': 0.209180...
4303 {'xmin': 0.34089717268943787, 'ymin': 0.005420...
4304 {'xmin': 0.28278982639312744, 'ymin': 0.958871...
image
0 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
1 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
2 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
3 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
4 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
... ...
4300 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
4301 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
4302 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
4303 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
4304 /var/lib/jenkins/.gluoncv/datasets/tiny_motorb...
[4305 rows x 4 columns]
We can also save the trained model, and use it later.
savefile = 'detector.ag'
detector.save(savefile)
new_detector = ObjectDetector.load(savefile)
/var/lib/jenkins/workspace/workspace/autogluon-tutorial-object-detection-v3/venv/lib/python3.8/site-packages/mxnet/gluon/block.py:1512: UserWarning: Cannot decide type for the following arguments. Consider providing them as input:
data: None
input_sym_arg_type = in_param.infer_type()[0]