introduction_en.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "03cfffa3-2398-4d55-bf1e-fe3e01f10d68",
   "metadata": {},
   "source": [
    "## 1. PP-Matting model introduction\n",
    "\n",
    "\n",
    "In many image matting algorithms, in order to pursue precision, trimap is often provided as auxiliary information, but this greatly limits the application of the algorithm. PP-Matting, as a trimap-free image matting method, overcomes the disadvantages of auxiliary information and achieves SOTA performance in Composition-1k and Distinctions-646 datasets. PP-Matting uses Semantic Context Branch (SCB) to extract high-level semantic information of images and gradually guides high-resolution detail branch (HRDB) to extract details in transition area through Guidance Flow. Finally, alpha matte is obtained by fusing semantic map and detail map with fusion module.\n",
    "\n",
    "More details can be found in the paper: https://arxiv.org/abs/2204.09433.\n",
    "\n",
    "More about PaddleMatting，you can click https://github.com/PaddlePaddle/PaddleSeg/tree/develop/Matting to learn.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cbcf510-dc56-43f9-9864-5e1de3c7b272",
   "metadata": {},
   "source": [
    "## 2. Model Effects and Application Scenarios\n",
    "The human matting effects of PP-Matting are as follows:\n",
    "<p align=\"center\">\n",
    "<img src=\"https://user-images.githubusercontent.com/30919197/179751613-d26f2261-7bcf-4066-a0a4-4c818e7065f0.gif\" width=\"100%\" height=\"100%\">\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d9da224-edd4-4c9e-ab2d-cba7ee5a92a4",
   "metadata": {},
   "source": [
    "## 3. How to Use the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ac57be7-c00f-441e-ad82-635f6268b6bd",
   "metadata": {},
   "source": [
    "### 3.1 Model Inference\n",
    "* Download"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ea22cb4-5bed-4ce0-858b-3bbb342e8ccf",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "!mkdir -p ~/work\n",
    "%cd ~/work\n",
    "!git clone --depth 1  https://gitee.com/paddlepaddle/PaddleSeg"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd7142e9-5202-4a4d-8a03-36fcc5ea0259",
   "metadata": {},
   "source": [
    "* Installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7fd9575-4116-4fa7-86ee-0b2db7417300",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%cd ~/work/PaddleSeg/Matting\n",
    "!pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0dd2390-e635-432b-9e0b-d7c9323f81a9",
   "metadata": {},
   "source": [
    "* Quick experience\n",
    "\n",
    "Congratulations! Now that you've successfully installed PaddleSeg, let's get a quick feel at human matting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81736dd2-e90d-4dac-b3c0-58424e2e8dc4",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Download pretrained model\n",
    "!wget https://paddleseg.bj.bcebos.com/matting/models/ppmatting-hrnet_w18-human_512.pdparams\n",
    "# Download image\n",
    "!wget https://user-images.githubusercontent.com/30919197/200645066-6898ec5a-f1c5-4bf7-aa41-473a29977a86.jpeg\n",
    "# Predicd one image in GPU\n",
    "!export CUDA_VISIBLE_DEVICES=0\n",
    "!python tools/predict.py \\\n",
    "    --config configs/ppmatting/ppmatting-hrnet_w18-human_512.yml \\\n",
    "    --model_path ppmatting-hrnet_w18-human_512.pdparams \\\n",
    "    --image_path 200645066-6898ec5a-f1c5-4bf7-aa41-473a29977a86.jpeg \\\n",
    "    --save_dir ./output/results \\\n",
    "    --fg_estimate True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ab60ad4-2908-40e2-9c08-34ac7978ef16",
   "metadata": {},
   "source": [
    "An alpha image and a rgba image are generated in the output/results folder.\n",
    "\n",
    "\n",
    "The results are as follows：\n",
    "<div align=\"center\">\n",
    "<img src=\"https://user-images.githubusercontent.com/30919197/200647513-e744118a-bd2b-4de2-b740-6eaeaccf47b6.png\"  width = \"40%\"  /> \n",
    "<img src=\"https://user-images.githubusercontent.com/30919197/200647492-7fd5fd80-2c2a-4775-ad9e-fd1d476cba48.png\"  width = \"40%\"  />\n",
    "* </div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98c083d9-acd1-4e0a-855b-0a048983f251",
   "metadata": {},
   "source": [
    "### 3.2 Model Training\n",
    " * Clone PaddleSeg repository（see 3.1 for details）。\n",
    " \n",
    " More details for training please refer the document in Matting folder under PaddleSeg repository.\n",
    "https://github.com/PaddlePaddle/PaddleSeg/tree/develop/Matting 。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9b05c1e-cb59-4dd7-83c0-cb30dfefba60",
   "metadata": {},
   "source": [
    "## 4. Model Principles\n",
    "\n",
    "<div align=\"center\">\n",
    "<img src=\"https://user-images.githubusercontent.com/30919197/200649929-3fa07082-7980-49f8-9f7f-38c31c46de20.png\"  width = \"80%\"  />\n",
    "* </div>\n",
    "\n",
    "* Clear semantic prediction and detail prediction tasks by branch design.\n",
    "* Semantic Context Branch(SCB) ensures the accuracy of the overall prediction of the image by semantic prediction, which roughly divides the image into three parts: foreground, background and transition area.\n",
    "* High-Resolution Detail Branch (HRDB) keep high resolution while extracting features，which ensures that the details are not lost.\n",
    "* The design of Guidance Flow enables the HRDB branch to obtain the semantic information extracted by the SCB branch, which enables the HRDB branch to focus on the detail prediction in transition region.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae7e1030-9044-44f5-a0d3-5148e59d4753",
   "metadata": {},
   "source": [
    "## 5. Attention\n",
    "* Training and prediction based on the public datasets Composition-1K and Distinctions-646 should be requested by email to the author.\n",
    "* PP-Matting open-source pre-trained human matting model, which can be finetune using a small amount of annotated dataset according to specific human matting scenes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e238357-2f19-48c3-902e-9ba3c93f7818",
   "metadata": {},
   "source": [
    "## 6. Related papers and citations\n",
    "@article{chen2022pp,\n",
    "  title={PP-Matting: High-Accuracy Natural Image Matting},\n",
    "  author={Chen, Guowei and Liu, Yi and Wang, Jian and Peng, Juncai and Hao, Yuying and Chu, Lutao and Tang, Shiyu and Wu, Zewu and Chen, Zeyu and Yu, Zhiliang and others},\n",
    "  journal={arXiv preprint arXiv:2204.09433},\n",
    "  year={2022}\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}