diff --git a/modelcenter/PP-Matting/benchmark_cn.md b/modelcenter/PP-Matting/benchmark_cn.md index 2ab6bcbba6a42f0f841c50fab19d9269a71b4d0b..375b55f16919e737b2eceda34319956b3817337f 100644 --- a/modelcenter/PP-Matting/benchmark_cn.md +++ b/modelcenter/PP-Matting/benchmark_cn.md @@ -1,5 +1,3 @@ -# 模型列表 - ## 1. 训练Benchmark ### 1.1 软硬件环境 @@ -29,7 +27,7 @@ ### 2.3 指标 | 模型 | 数据集 | SAD | MSE | Grad | Conn |Params(M) | FLOPs(G) | FPS | -| - | - | -| - | - | - | - | -| - | - | +| - | - | -| - | - | - | - | -| - | | ppmatting_hrnet_w48 | Composition-1k | 46.22 | 0.005 | 22.69 | 45.40 | 86.3 | 165.4 | 24.4 | | ppmatting_hrnet_w48 | Distinctions-646 | 40.69 | 0.009 | 43.91 |40.56 | 86.3 | 165.4 | 24.4 | | ppmatting_hrnet_w18 | PPM-AIM-195 | 31.56|0.0022|31.80|30.13| 24.5 | 91.28 | 28.9 | diff --git a/modelcenter/PP-Matting/benchmark_en.md b/modelcenter/PP-Matting/benchmark_en.md new file mode 100644 index 0000000000000000000000000000000000000000..40c8a463ca0c9091b457fcc331452e707c8d7678 --- /dev/null +++ b/modelcenter/PP-Matting/benchmark_en.md @@ -0,0 +1,36 @@ +## 1. Training Benchmark + +### 1.1 Environment + +* The training process of PP-Matting model uses one GPU and batch size 4. + +### 1.2 Datasets + +* The common object matting dataset is Compositon-1k or Distinctins-646 (the use of both datasets are requested from the author). COCO2017 and Pascal VOC 2012 are used as the background datasets. +* Human matting uses the private dataset. + +### 1.3 Benchmark +|Model | Description | Input shape | +|---|---|---| +|ppmatting_hrnet_w48 | Common object matting | 512 | +|ppmatting_hrnet_w18 | Human matting | 512 | + +## 2. Inference Benchmark + +### 2.1 Environment + +* The PP-Matting model's inference speed test is tested with one V100, batch size=1, CUDA 10.2, CUDNN 7.6.5, PaddlePaddle-gpu 2.3.2. + +### 2.2 Datasets +* Common object matting: the test dataset of Compositon-1k or Distinctions-646. +* Human matting: the human image of PPM-100 and AIM-500, total 195 images, named PPM-AIM-195. + +### 2.3 Benchmark +| Model | Dataset | SAD | MSE | Grad | Conn |Params(M) | FLOPs(G) | FPS | +| - | - | -| - | - | - | - | -| - | +| ppmatting_hrnet_w48 | Composition-1k | 46.22 | 0.005 | 22.69 | 45.40 | 86.3 | 165.4 | 24.4 | +| ppmatting_hrnet_w48 | Distinctions-646 | 40.69 | 0.009 | 43.91 |40.56 | 86.3 | 165.4 | 24.4 | +| ppmatting_hrnet_w18 | PPM-AIM-195 | 31.56|0.0022|31.80|30.13| 24.5 | 91.28 | 28.9 | + +## 3. Reference +1. https://github.com/PaddlePaddle/PaddleSeg/tree/develop/Matting diff --git a/modelcenter/PP-Matting/download_en.md b/modelcenter/PP-Matting/download_en.md new file mode 100644 index 0000000000000000000000000000000000000000..380a662b46c79ebf3804333c9d651cd8222a63e3 --- /dev/null +++ b/modelcenter/PP-Matting/download_en.md @@ -0,0 +1,12 @@ +# Model list + +## Common object matting +|Model | Dataset | Description | Model size | Download | +|---|---|---|---|---| +|ppmatting_hrnet_w48 | Composition-1k | Common object matting | 305M | [inference model](https://paddleseg.bj.bcebos.com/matting/models/deploy/ppmatting-hrnet_w48-composition.zip)/[model](https://paddleseg.bj.bcebos.com/matting/models/ppmatting-hrnet_w48-composition.pdparams) | +|ppmatting_hrnet_w48 | Distinctions-646 | Common object matting | 305M | [inference model](https://paddleseg.bj.bcebos.com/matting/models/deploy/ppmatting-hrnet_w48-distinctions.zip)/[model](https://paddleseg.bj.bcebos.com/matting/models/ppmatting-hrnet_w48-distinctions.pdparams) | + +## Human matting +| Model | Description | Model size | Download | +|---|---|---|---| +|ppmatting_hrnet_w18 | Human matting | 87M | [inference model](https://paddleseg.bj.bcebos.com/matting/models/deploy/ppmatting-hrnet_w18-human_512.zip)/[model](https://paddleseg.bj.bcebos.com/matting/models/ppmatting-hrnet_w18-human_512.pdparams) | diff --git a/modelcenter/PP-Matting/introduction_cn.ipynb b/modelcenter/PP-Matting/introduction_cn.ipynb index 99429f39713e7200232dfab05c3e9ca952631148..7fd30d24223686009c6c7966dca2961c674c9d2e 100644 --- a/modelcenter/PP-Matting/introduction_cn.ipynb +++ b/modelcenter/PP-Matting/introduction_cn.ipynb @@ -7,7 +7,7 @@ "source": [ "## 1. PP-Matting 模型简介\n", "\n", - "在众多图像抠图算法中，为了追求精度，往往需要输入trimap作为辅助信息，但这极大限制了算法的使用性。PP-Matting作为一种trimap-free的抠图方法，有效克服了辅助信息带来的弊端，在Composition-1k和Distinctions-646数据集中取得了SOTA的效果。PP-Matting利用语义分支（SCB）提取图片高级语义信息并通过引导流设计（Guidance Flow）逐步引导高分辨率细节分支（HRBP）对过度区域的细节提取，最后通过融合模块实现语义和细节的融合得到最终的alpha matte。\n", + "在众多图像抠图算法中，为了追求精度，往往需要输入trimap作为辅助信息，但这极大限制了算法的使用性。PP-Matting作为一种trimap-free的抠图方法，有效克服了辅助信息带来的弊端，在Composition-1k和Distinctions-646数据集中取得了SOTA的效果。PP-Matting利用语义分支（SCB）提取图片高级语义信息并通过引导流设计（Guidance Flow）逐步引导高分辨率细节分支（HRDB）对过度区域的细节提取，最后通过融合模块实现语义和细节的融合得到最终的alpha matte。\n", "\n", "更多细节可参考技术报告：https://arxiv.org/abs/2204.09433 。\n", "\n", @@ -153,7 +153,7 @@ "* 分支设计，明确语义预测和细节预测任务。\n", "* SCB(Semantic Context Branch)分支进行语义预测，保证图像整体预测正确性，其将图像粗略的分为三个部分，即前景、背景和过度区域。\n", "* HRDB（High-Resolution Detail Branch）维持高分辨率特征提取过程，保证细节不受损失。\n", - "* 引导流结构设计（Guidance Flow）使HRBD分支获取SCB分支提取的语义信息，使HRDB分支能专注与过度区域的细节预测。\n" + "* 引导流结构设计（Guidance Flow）使HRDB分支获取SCB分支提取的语义信息，使HRDB分支能专注与过度区域的细节预测。\n" ] }, { diff --git a/modelcenter/PP-Matting/introduction_en.ipynb b/modelcenter/PP-Matting/introduction_en.ipynb new file mode 100644 index 0000000000000000000000000000000000000000..4d21db9bb64d2d67f86413353450ed75779065c5 --- /dev/null +++ b/modelcenter/PP-Matting/introduction_en.ipynb @@ -0,0 +1,208 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "03cfffa3-2398-4d55-bf1e-fe3e01f10d68", + "metadata": {}, + "source": [ + "## 1. PP-Matting model introduction\n", + "\n", + "\n", + "In many image matting algorithms, in order to pursue precision, trimap is often provided as auxiliary information, but this greatly limits the application of the algorithm. PP-Matting, as a trimap-free image matting method, overcomes the disadvantages of auxiliary information and achieves SOTA performance in Composition-1k and Distinctions-646 datasets. PP-Matting uses Semantic Context Branch (SCB) to extract high-level semantic information of images and gradually guides high-resolution detail branch (HRDB) to extract details in transition area through Guidance Flow. Finally, alpha matte is obtained by fusing semantic map and detail map with fusion module.\n", + "\n", + "More details can be found in the paper: https://arxiv.org/abs/2204.09433.\n", + "\n", + "More about PaddleMatting，you can click https://github.com/PaddlePaddle/PaddleSeg/tree/develop/Matting to learn.\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "id": "8cbcf510-dc56-43f9-9864-5e1de3c7b272", + "metadata": {}, + "source": [ + "## 2. Model Effects and Application Scenarios\n", + "The human matting effects of PP-Matting are as follows:\n", + "

\n", + "\n", + "

\n" + ] + }, + { + "cell_type": "markdown", + "id": "8d9da224-edd4-4c9e-ab2d-cba7ee5a92a4", + "metadata": {}, + "source": [ + "## 3. How to Use the Model" + ] + }, + { + "cell_type": "markdown", + "id": "2ac57be7-c00f-441e-ad82-635f6268b6bd", + "metadata": {}, + "source": [ + "### 3.1 Model Inference\n", + "* Download" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "3ea22cb4-5bed-4ce0-858b-3bbb342e8ccf", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [], + "source": [ + "%cd ~/work\n", + "!git clone --depth 1 https://gitee.com/paddlepaddle/PaddleSeg" + ] + }, + { + "cell_type": "markdown", + "id": "fd7142e9-5202-4a4d-8a03-36fcc5ea0259", + "metadata": {}, + "source": [ + "* Installation" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a7fd9575-4116-4fa7-86ee-0b2db7417300", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [], + "source": [ + "%cd ~/work/PaddleSeg/Matting\n", + "!pip install -r requirements.txt" + ] + }, + { + "cell_type": "markdown", + "id": "a0dd2390-e635-432b-9e0b-d7c9323f81a9", + "metadata": {}, + "source": [ + "* Quick experience\n", + "\n", + "Congratulations! Now that you've successfully installed PaddleSeg, let's get a quick feel at human matting." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "81736dd2-e90d-4dac-b3c0-58424e2e8dc4", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [], + "source": [ + "# Download pretrained model\n", + "!wget https://paddleseg.bj.bcebos.com/matting/models/ppmatting-hrnet_w18-human_512.pdparams\n", + "# Download image\n", + "!wget https://user-images.githubusercontent.com/30919197/200645066-6898ec5a-f1c5-4bf7-aa41-473a29977a86.jpeg\n", + "# Predicd one image in GPU\n", + "!export CUDA_VISIBLE_DEVICES=0\n", + "!python tools/predict.py \\\n", + " --config configs/ppmatting/ppmatting-hrnet_w18-human_512.yml \\\n", + " --model_path ppmatting-hrnet_w18-human_512.pdparams \\\n", + " --image_path 200645066-6898ec5a-f1c5-4bf7-aa41-473a29977a86.jpeg \\\n", + " --save_dir ./output/results \\\n", + " --fg_estimate True" + ] + }, + { + "cell_type": "markdown", + "id": "3ab60ad4-2908-40e2-9c08-34ac7978ef16", + "metadata": {}, + "source": [ + "An alpha image and a rgba image are generated in the output/results folder.\n", + "\n", + "\n", + "The results are as follows：\n", + "
\n", + " \n", + "\n", + "*
\n" + ] + }, + { + "cell_type": "markdown", + "id": "98c083d9-acd1-4e0a-855b-0a048983f251", + "metadata": {}, + "source": [ + "### 3.2 Model Training\n", + " * Clone PaddleSeg repository（see 3.1 for details）。\n", + " \n", + " More details for training please refer the document in Matting folder under PaddleSeg repository.\n", + "https://github.com/PaddlePaddle/PaddleSeg/tree/develop/Matting 。" + ] + }, + { + "cell_type": "markdown", + "id": "e9b05c1e-cb59-4dd7-83c0-cb30dfefba60", + "metadata": {}, + "source": [ + "## 4. Model Principles\n", + "\n", + "
\n", + "\n", + "*
\n", + "\n", + "* Clear semantic prediction and detail prediction tasks by branch design.\n", + "* Semantic Context Branch(SCB) ensures the accuracy of the overall prediction of the image by semantic prediction, which roughly divides the image into three parts: foreground, background and transition area.\n", + "* High-Resolution Detail Branch (HRDB) keep high resolution while extracting features，which ensures that the details are not lost.\n", + "* The design of Guidance Flow enables the HRDB branch to obtain the semantic information extracted by the SCB branch, which enables the HRDB branch to focus on the detail prediction in transition region.\n" + ] + }, + { + "cell_type": "markdown", + "id": "ae7e1030-9044-44f5-a0d3-5148e59d4753", + "metadata": {}, + "source": [ + "## 5. Attention\n", + "* Training and prediction based on the public datasets Composition-1K and Distinctions-646 should be requested by email to the author.\n", + "* PP-Matting open-source pre-trained human matting model, which can be finetune using a small amount of annotated dataset according to specific human matting scenes." + ] + }, + { + "cell_type": "markdown", + "id": "7e238357-2f19-48c3-902e-9ba3c93f7818", + "metadata": {}, + "source": [ + "## 6. Related papers and citations\n", + "@article{chen2022pp,\n", + " title={PP-Matting: High-Accuracy Natural Image Matting},\n", + " author={Chen, Guowei and Liu, Yi and Wang, Jian and Peng, Juncai and Hao, Yuying and Chu, Lutao and Tang, Shiyu and Wu, Zewu and Chen, Zeyu and Yu, Zhiliang and others},\n", + " journal={arXiv preprint arXiv:2204.09433},\n", + " year={2022}\n", + "}" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "py35-paddle1.2.0" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.4" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}