{"id":2384,"date":"2021-05-25T09:00:00","date_gmt":"2021-05-25T01:00:00","guid":{"rendered":"https:\/\/blog.ailia.ai\/%e6%9c%aa%e5%88%86%e9%a1%9e\/dpt-segmentation-model-using-vision-transformer\/"},"modified":"2025-05-14T15:31:10","modified_gmt":"2025-05-14T07:31:10","slug":"dpt-segmentation-model-using-vision-transformer","status":"publish","type":"post","link":"https:\/\/blog.ailia.ai\/en\/tips-en\/dpt-segmentation-model-using-vision-transformer\/","title":{"rendered":"DPT : Segmentation Model Using Vision Transformer"},"content":{"rendered":"\n

Overview<\/strong><\/p>\n\n\n\n

DPT (DensePredictionTransformers)<\/em> is a segmentation model released by Intel in March 2021 that applies vision transformers <\/em>to images. It can perform image semantic segmentation with 49.02% mIoU on ADE20K, and it can also be used for monocular depth estimation with an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network.<\/p>\n\n\n\n

<\/a>Vision Transformers for Dense Prediction<\/a><\/p>\n\n\n\n

Architecture<\/strong><\/h3>\n\n\n\n

In DPT<\/em>, vision transformers (ViT)<\/em>are used instead of convolutional network. Using transformers allows to make more detailed and globally consistent predictions compared to convolutional networks. In particular, performance is improved when a large amount of training data is available.<\/p>\n\n\n\n

\"\"
\u51fa\u5178\uff1ahttps:\/\/arxiv.org\/pdf\/2103.13413<\/a><\/figcaption><\/figure>\n\n\n\n

The encoder <\/em>divides the image into tiles, which are then tokenized (Embed<\/em> in the graph above), and transformers process it. The process marked as Embed<\/em>is a patch-based method to divide image into tiles, and tokenize the pixel feature map obtained by applying ResNet50 <\/em>to the input image.<\/p>\n\n\n\n

The decoder <\/em>in DPT converts the output of each resolution of the transformer into an image like representation and uses a convolutional network to generate the segmentation image.<\/p>\n\n\n\n

There are three model architectures defined in DPT: ViT-Base, ViT-Large<\/em>, and ViT-Hybrid<\/em>. ViT-Base<\/em> performs patch-based embedding and has 12 transformer layers. ViT-Large<\/em> performs the same embedding as ViT-Base<\/em>, but has 24 transformer layers and a larger feature size. ViT-Hybrid<\/em> performs embedding using ResNet50 <\/em>and has 12 transformer layers.<\/p>\n\n\n\n

DPT accuracy<\/strong><\/h3>\n\n\n\n

DPT <\/em>sets a new state of the art for the semantic segmentation task on ADE20K, a large data set with 150 classes<\/a>.<\/p>\n\n\n\n

\"\"
\u51fa\u5178\uff1ahttps:\/\/arxiv.org\/pdf\/2103.13413<\/a><\/figcaption><\/figure>\n\n\n\n

It is also the state of the art after some fine-tuning on smaller datasets such as NYUv2<\/em>, KITTI<\/em>, and Pascal Context<\/em>.<\/p>\n\n\n\n

\"\"
\u51fa\u5178\uff1ahttps:\/\/arxiv.org\/pdf\/2103.13413<\/a><\/figcaption><\/figure>\n\n\n\n

Below is a comparison of MiDaS<\/em><\/a> <\/em>and DPT <\/em>for depth estimation. DPT is able to predict the depth inmore detail. It can also improve the accuracy of large homogeneous regions and relative positioning within an image, which is a shortcoming of convolution networks.<\/p>\n\n\n\n

\"\"
\u51fa\u5178\uff1ahttps:\/\/arxiv.org\/pdf\/2103.13413<\/a><\/figcaption><\/figure>\n\n\n\n

Below is a comparison for the segmentation task. DPT tends to produce more detailed output at object boundaries, and it tends to produce less cluttered output in some cases.<\/p>\n\n\n\n

\"\"
\u51fa\u5178\uff1ahttps:\/\/arxiv.org\/pdf\/2103.13413<\/a><\/figcaption><\/figure>\n\n\n\n

DPT Usage<\/strong><\/h3>\n\n\n\n

You can use the following commands to perform segmentation and depth estimation on the input images with ailia SDK.<\/span><\/p>\n\n\n\n

$ python3 dense_prediction_transformers.py -i input.jpg -s output.png --task=segmentation -e 0\n$ python3 dense_prediction_transformers.py -i input.jpg -s output.png--task=monodepth -e 0\n<\/a><\/code><\/pre>\n\n\n\n
axinc-ai\/ailia-models<\/a><\/p>\n\n\n\n
Here is a result you can expect.<\/p>\n\n\n\n
\n