Skip to content

O
Opencv Zoo

张重言 / Opencv Zoo
10 个月 前同步成功

通知 1
Star 0
Fork 0
O
Opencv Zoo

前往新版Gitcode，体验更适合开发者的 AI 搜索 >>

提交 1f19f821 编写于 作者: W Wanli 提交者： GitHub
浏览文件
操作

add pose estimation model (#152)

上级 899346b1
隐藏空白更改
内联 并排
Showing with 888 addition and 8 deletion
README.md
浏览文件 @ 1f19f821
......@@ -86,6 +86,10 @@ Some examples are listed below. You can find more in the directory of each model
![person det](./models/person_detection_mediapipe/examples/mppersondet_demo.webp)
### Pose Estimation with [MP-Pose](models/pose_estimation_mediapipe)
![pose_estimation](models/pose_estimation_mediapipe/examples/mpposeest_demo.webp)
### QR Code Detection and Parsing with [WeChatQRCode](./models/qrcode_wechatqrcode/)
![qrcode](./models/qrcode_wechatqrcode/examples/wechat_qrcode_demo.gif)
......
benchmark/README.md
浏览文件 @ 1f19f821
......@@ -98,6 +98,7 @@ mean median min input size model
13.88 14.82 12.39 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
30.87 30.69 29.85 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
30.77 30.02 27.97 [128, 256] YoutuReID with ['person_reid_youtu_2021nov_int8.onnx']
7.72 8.84 6.13 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
1.35 1.37 1.30 [100, 100] WeChatQRCode with ['detect_2021nov.prototxt', 'detect_2021nov.caffemodel', 'sr_2021nov.prototxt', 'sr_2021nov.caffemodel']
75.82 75.37 69.18 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
74.80 75.16 69.05 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
......@@ -151,6 +152,7 @@ mean median min input size model
98.52 98.95 97.58 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
676.15 655.20 636.06 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
548.93 582.29 443.32 [128, 256] YoutuReID with ['person_reid_youtu_2021nov_int8.onnx']
90.26 92.06 88.80 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
8.18 8.15 8.13 [100, 100] WeChatQRCode with ['detect_2021nov.prototxt', 'detect_2021nov.caffemodel', 'sr_2021nov.prototxt', 'sr_2021nov.caffemodel']
2025.09 2046.92 1971.57 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
2041.85 2048.24 1971.57 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
......@@ -205,6 +207,7 @@ mean median min input size model
98.38 98.20 97.69 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
411.49 417.53 402.57 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
372.94 370.17 335.95 [128, 256] YoutuReID with ['person_reid_youtu_2021nov_int8.onnx']
74.36 75.15 72.22 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
5.62 5.64 5.55 [100, 100] WeChatQRCode with ['detect_2021nov.prototxt', 'detect_2021nov.caffemodel', 'sr_2021nov.prototxt', 'sr_2021nov.caffemodel']
1089.89 1091.85 1071.95 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
1089.94 1095.07 1071.95 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
......@@ -241,6 +244,7 @@ mean median min input size model
67.34 67.83 62.38 [192, 192] MPPalmDet with ['palm_detection_mediapipe_2023feb.onnx']
56.69 55.54 48.96 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
126.65 126.63 124.96 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
73.84 75.25 72.19 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
303.12 302.80 299.30 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
302.58 299.78 297.83 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
58.05 62.90 52.47 [1280, 720] CRNN with ['text_recognition_CRNN_CH_2021sep.onnx']
......@@ -271,6 +275,7 @@ mean median min input size model
212.86 213.21 210.03 [192, 192] MPPalmDet with ['palm_detection_mediapipe_2023feb.onnx']
221.12 255.53 217.16 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
96.68 94.21 89.24 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
73.68 77.30 69.17 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
343.38 344.17 337.62 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
344.29 345.07 337.62 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
48.91 50.31 45.41 [1280, 720] CRNN with ['text_recognition_CRNN_CH_2021sep.onnx']
......@@ -318,6 +323,7 @@ mean median min input size model
84.42 85.99 83.30 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
439.53 431.92 406.03 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
358.63 379.93 296.32 [128, 256] YoutuReID with ['person_reid_youtu_2021nov_int8.onnx']
68.51 66.87 66.53 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
5.29 5.30 5.21 [100, 100] WeChatQRCode with ['detect_2021nov.prototxt', 'detect_2021nov.caffemodel', 'sr_2021nov.prototxt', 'sr_2021nov.caffemodel']
973.75 968.68 954.58 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
961.44 959.29 935.29 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
......@@ -396,6 +402,7 @@ mean median min input size model
134.10 134.43 133.62 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
631.70 631.81 630.61 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
595.32 599.48 565.32 [128, 256] YoutuReID with ['person_reid_youtu_2021nov_int8.onnx']
108.55 117.88 106.66 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
1452.55 1453.75 1450.98 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
1433.26 1432.08 1409.78 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
299.36 299.92 298.75 [1280, 720] CRNN with ['text_recognition_CRNN_CH_2021sep.onnx']
......@@ -412,7 +419,7 @@ mean median min input size model
NPU:
```
$ python3 benchmark.py --all --fp32 --cfg_exclude wechat:dasiamrpn:crnn --cfg_overwrite_backend_target 4
$ python3 benchmark.py --all --fp32 --cfg_exclude wechat:dasiamrpn:crnn --model_exclude pose_estimation_mediapipe_2023mar.onnx --cfg_overwrite_backend_target 4
Benchmarking ...
backend=cv.dnn.DNN_BACKEND_CANN
target=cv.dnn.DNN_TARGET_NPU
......@@ -475,6 +482,7 @@ mean median min input size model
1326.56 1327.10 1305.18 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
11117.07 11109.12 11058.49 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
7037.96 7424.89 3750.12 [128, 256] YoutuReID with ['person_reid_youtu_2021nov_int8.onnx']
704.44 704.77 672.58 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
49065.03 49144.55 48943.50 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
49052.24 48992.64 48927.44 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
2200.08 2193.78 2175.77 [1280, 720] CRNN with ['text_recognition_CRNN_CH_2021sep.onnx']
......@@ -529,6 +537,7 @@ mean median min input size model
46.07 46.77 45.10 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
195.67 198.02 182.97 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
181.91 182.28 169.98 [128, 256] YoutuReID with ['person_reid_youtu_2021nov_int8.onnx']
35.47 37.63 33.55 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
394.77 407.60 371.95 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
392.52 404.80 367.96 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
77.32 77.72 75.27 [1280, 720] CRNN with ['text_recognition_CRNN_CH_2021sep.onnx']
......@@ -582,6 +591,7 @@ mean median min input size model
191.41 191.48 191.00 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
898.23 897.52 896.58 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
749.83 765.90 630.39 [128, 256] YoutuReID with ['person_reid_youtu_2021nov_int8.onnx']
158.50 160.55 155.64 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
1908.87 1905.00 1903.13 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
1922.34 1920.65 1896.97 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
470.78 469.17 467.92 [1280, 720] CRNN with ['text_recognition_CRNN_CH_2021sep.onnx']
......@@ -636,6 +646,7 @@ mean median min input size model
1835.97 1836.24 1835.34 [224, 224] MPPersonDet with ['person_detection_mediapipe_2023mar.onnx']
14886.02 14884.48 14881.73 [128, 256] YoutuReID with ['person_reid_youtu_2021nov.onnx']
10491.63 10930.80 6975.34 [128, 256] YoutuReID with ['person_reid_youtu_2021nov_int8.onnx']
987.30 992.59 982.71 [256, 256] MPPose with ['pose_estimation_mediapipe_2023mar.onnx']
65681.91 65674.89 65612.09 [640, 480] DB with ['text_detection_DB_IC15_resnet18_2021sep.onnx']
65630.56 65652.90 65531.21 [640, 480] DB with ['text_detection_DB_TD500_resnet18_2021sep.onnx']
3248.11 3242.59 3241.18 [1280, 720] CRNN with ['text_recognition_CRNN_CH_2021sep.onnx']
......
benchmark/color_table.svg
浏览文件 @ 1f19f821
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1546.489312pt" height="600.632471pt" viewBox="0 0 1546.489312 600.632471" xmlns="http://www.w3.org/2000/svg" version="1.1">
<svg xmlns:xlink="http://www.w3.org/1999/xlink" width="1546.489312pt" height="624.632471pt" viewBox="0 0 1546.489312 624.632471" xmlns="http://www.w3.org/2000/svg" version="1.1">
<metadata>
<rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<cc:Work>
......@@ -15,8 +15,8 @@
</defs>
<g id="figure_1">
<g id="patch_1">
<path d="M 0 600.632471
L 1546.489312 600.632471
<path d="M 0 624.632471
L 1546.489312 624.632471
L 1546.489312 0
L 0 0
z
......@@ -3713,14 +3713,183 @@ z
<g id="text_317">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="1460.320331" y="552.533665" transform="rotate(-0 1460.320331 552.533665)">16.45</text>
</g>
<g id="patch_317">
<a xlink:href="https://github.com/opencv/opencv_zoo/tree/master/models/pose_estimation_mediapipe">
<path d="M 0 585.498353
L 128.297812 585.498353
L 128.297812 561.498353
L 0 561.498353
z
" style="fill: #ffffff; stroke: #000000; stroke-linejoin: miter"/>
</a>
</g>
<g id="text_318">
<a xlink:href="https://github.com/opencv/opencv_zoo/tree/master/models/pose_estimation_mediapipe">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="12.829781" y="576.533665" transform="rotate(-0 12.829781 576.533665)">MP-Pose</text>
</a>
</g>
<g id="patch_318">
<path d="M 128.297812 585.498353
L 335.566688 585.498353
L 335.566688 561.498353
L 128.297812 561.498353
z
" style="fill: #ffffff; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_319">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="149.0247" y="576.533665" transform="rotate(-0 149.0247 576.533665)">Pose Estimation</text>
</g>
<g id="patch_319">
<path d="M 335.566688 585.498353
L 410.291063 585.498353
L 410.291063 561.498353
L 335.566688 561.498353
z
" style="fill: #ffffff; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_320">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="343.039125" y="576.533665" transform="rotate(-0 343.039125 576.533665)">256x256</text>
</g>
<g id="patch_320">
<path d="M 410.291063 585.498353
L 466.440562 585.498353
L 466.440562 561.498353
L 410.291063 561.498353
z
" style="fill: #006837; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_321">
<text style="fill: #ffffff; font: 700 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="415.906013" y="576.533665" transform="rotate(-0 415.906013 576.533665)">7.72</text>
</g>
<g id="patch_321">
<path d="M 466.440562 585.498353
L 576.029438 585.498353
L 576.029438 561.498353
L 466.440562 561.498353
z
" style="fill: #bde379; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_322">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="477.39945" y="576.533665" transform="rotate(-0 477.39945 576.533665)">90.26</text>
</g>
<g id="patch_322">
<path d="M 576.029438 585.498353
L 639.014063 585.498353
L 639.014063 561.498353
L 576.029438 561.498353
z
" style="fill: #a50026; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_323">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="582.3279" y="576.533665" transform="rotate(-0 582.3279 576.533665)">704.44</text>
</g>
<g id="patch_323">
<path d="M 639.014063 585.498353
L 744.40575 585.498353
L 744.40575 561.498353
L 639.014063 561.498353
z
" style="fill: #66bd63; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_324">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="649.553231" y="576.533665" transform="rotate(-0 649.553231 576.533665)">35.47</text>
</g>
<g id="patch_324">
<path d="M 744.40575 585.498353
L 881.238187 585.498353
L 881.238187 561.498353
L 744.40575 561.498353
z
" style="fill: #f2faae; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_325">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="758.088994" y="576.533665" transform="rotate(-0 758.088994 576.533665)">158.50</text>
</g>
<g id="patch_325">
<path d="M 881.238187 585.498353
L 982.407937 585.498353
L 982.407937 561.498353
L 881.238187 561.498353
z
" style="fill: #a50026; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_326">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="891.355163" y="576.533665" transform="rotate(-0 891.355163 576.533665)">987.30</text>
</g>
<g id="patch_326">
<path d="M 982.407937 585.498353
L 1071.918375 585.498353
L 1071.918375 561.498353
L 982.407937 561.498353
z
" style="fill: #abdb6d; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_327">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="991.358981" y="576.533665" transform="rotate(-0 991.358981 576.533665)">74.36</text>
</g>
<g id="patch_327">
<path d="M 1071.918375 585.498353
L 1168.705313 585.498353
L 1168.705313 561.498353
L 1071.918375 561.498353
z
" style="fill: #a2d76a; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_328">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="1081.597069" y="576.533665" transform="rotate(-0 1081.597069 576.533665)">68.51</text>
</g>
<g id="patch_328">
<path d="M 1168.705313 585.498353
L 1264.448625 585.498353
L 1264.448625 561.498353
L 1168.705313 561.498353
z
" style="fill: #cfeb85; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_329">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="1178.279644" y="576.533665" transform="rotate(-0 1178.279644 576.533665)">108.55</text>
</g>
<g id="patch_329">
<path d="M 1264.448625 585.498353
L 1353.959063 585.498353
L 1353.959063 561.498353
L 1264.448625 561.498353
z
" style="fill: #abdb6d; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_330">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="1273.399669" y="576.533665" transform="rotate(-0 1273.399669 576.533665)">73.84</text>
</g>
<g id="patch_330">
<path d="M 1353.959063 585.498353
L 1450.746 585.498353
L 1450.746 561.498353
L 1353.959063 561.498353
z
" style="fill: #ffffff; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_331">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="1363.637756" y="576.533665" transform="rotate(-0 1363.637756 576.533665)">---</text>
</g>
<g id="patch_331">
<path d="M 1450.746 585.498353
L 1546.489312 585.498353
L 1546.489312 561.498353
L 1450.746 561.498353
z
" style="fill: #ffffff; stroke: #000000; stroke-linejoin: miter"/>
</g>
<g id="text_332">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="1460.320331" y="576.533665" transform="rotate(-0 1460.320331 576.533665)">---</text>
</g>
</g>
</g>
<g id="axes_3">
<g id="text_318">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="0" y="585.300108" transform="rotate(-0 0 585.300108)">Units: All data in milliseconds (ms).</text>
<g id="text_333">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="0" y="609.300108" transform="rotate(-0 0 609.300108)">Units: All data in milliseconds (ms).</text>
</g>
<g id="text_319">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="0" y="598.344814" transform="rotate(-0 0 598.344814)">\*: Models are quantized in per-channel mode, which run slower than per-tensor quantized models on NPU.</text>
<g id="text_334">
<text style="font: 11px 'DejaVu Sans', 'Bitstream Vera Sans', 'Computer Modern Sans Serif', 'Lucida Grande', 'Verdana', 'Geneva', 'Lucid', 'Arial', 'Helvetica', 'Avant Garde', sans-serif; text-anchor: start" x="0" y="622.344814" transform="rotate(-0 0 622.344814)">\*: Models are quantized in per-channel mode, which run slower than per-tensor quantized models on NPU.</text>
</g>
</g>
</g>
......
benchmark/config/pose_estimation_mediapipe.yaml 0 → 100644
浏览文件 @ 1f19f821
Benchmark:
name: "Pose Estimation Benchmark"
type: "Recognition"
data:
path: "data/person_detection"
files: ["person1.jpg", "person2.jpg", "person3.jpg"]
sizes: # [[w1, h1], ...], Omit to run at original scale
- [256, 256]
metric:
warmup: 30
repeat: 10
backend: "default"
target: "cpu"
Model:
name: "MPPose"
confThreshold: 0.9
benchmark/table_config.yaml
浏览文件 @ 1f19f821
......@@ -157,6 +157,13 @@ Models:
acceptable_time: 1300
keyword: "person_detection_mediapipe"
- name: "MP-Pose"
task: "Pose Estimation"
input_size: "256x256"
folder: "pose_estimation_mediapipe"
acceptable_time: 700
keyword: "pose_estimation_mediapipe"
Devices:
- name: "Intel 12700K"
......
models/__init__.py
浏览文件 @ 1f19f821
......@@ -9,6 +9,7 @@ from .face_recognition_sface.sface import SFace
from .image_classification_ppresnet.ppresnet import PPResNet
from .human_segmentation_pphumanseg.pphumanseg import PPHumanSeg
from .person_detection_mediapipe.mp_persondet import MPPersonDet
from .pose_estimation_mediapipe.mp_pose import MPPose
from .qrcode_wechatqrcode.wechatqrcode import WeChatQRCode
from .object_tracking_dasiamrpn.dasiamrpn import DaSiamRPN
from .person_reid_youtureid.youtureid import YoutuReID
......@@ -82,6 +83,7 @@ MODELS.register(SFace)
MODELS.register(PPResNet)
MODELS.register(PPHumanSeg)
MODELS.register(MPPersonDet)
MODELS.register(MPPose)
MODELS.register(WeChatQRCode)
MODELS.register(DaSiamRPN)
MODELS.register(YoutuReID)
......
models/pose_estimation_mediapipe/LICENSE 0 → 100644
浏览文件 @ 1f19f821
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
\ No newline at end of file
models/pose_estimation_mediapipe/README.md 0 → 100644
浏览文件 @ 1f19f821
# Pose estimation from MediaPipe Pose
This model estimates 33 pose keypoints and person segmentation mask per detected person from [person detector](../person_detection_mediapipe). (The image below is referenced from [MediaPipe Pose Keypoints](https://github.com/tensorflow/tfjs-models/tree/master/pose-detection#blazepose-keypoints-used-in-mediapipe-blazepose))
![MediaPipe Pose Landmark](examples/pose_landmarks.png)
This model is converted from TFlite to ONNX using following tools:
- TFLite model to ONNX: https://github.com/onnx/tensorflow-onnx
- simplified by [onnx-simplifier](https://github.com/daquexian/onnx-simplifier)
**Note**:
- Visit https://github.com/google/mediapipe/blob/master/docs/solutions/models.md#pose for models of larger scale.
## Demo
Run the following commands to try the demo:
```bash
# detect on camera input
python demo.py
# detect on an image
python demo.py -i /path/to/image -v
```
### Example outputs
![webcam demo](examples/mpposeest_demo.webp)
## License
All files in this directory are licensed under [Apache 2.0 License](LICENSE).
## Reference
- MediaPipe Pose: https://developers.google.com/mediapipe/solutions/vision/pose_landmarker
- MediaPipe pose model and model card: https://github.com/google/mediapipe/blob/master/docs/solutions/models.md#pose
- BlazePose TFJS: https://github.com/tensorflow/tfjs-models/tree/master/pose-detection/src/blazepose_tfjs
models/pose_estimation_mediapipe/demo.py 0 → 100644
浏览文件 @ 1f19f821
import sys
import argparse
import numpy as np
import cv2 as cv
from mp_pose import MPPose
sys.path.append('../person_detection_mediapipe')
from mp_persondet import MPPersonDet
# Check OpenCV version
assert cv.__version__ >= "4.7.0", \
"Please install latest opencv-python to try this demo: python3 -m pip install --upgrade opencv-python"
# Valid combinations of backends and targets
backend_target_pairs = [
[cv.dnn.DNN_BACKEND_OPENCV, cv.dnn.DNN_TARGET_CPU],
[cv.dnn.DNN_BACKEND_CUDA, cv.dnn.DNN_TARGET_CUDA],
[cv.dnn.DNN_BACKEND_CUDA, cv.dnn.DNN_TARGET_CUDA_FP16],
[cv.dnn.DNN_BACKEND_TIMVX, cv.dnn.DNN_TARGET_NPU],
[cv.dnn.DNN_BACKEND_CANN, cv.dnn.DNN_TARGET_NPU]
]
parser = argparse.ArgumentParser(description='Pose Estimation from MediaPipe')
parser.add_argument('--input', '-i', type=str,
help='Path to the input image. Omit for using default camera.')
parser.add_argument('--model', '-m', type=str, default='./pose_estimation_mediapipe_2023mar.onnx',
help='Path to the model.')
parser.add_argument('--backend_target', '-bt', type=int, default=0,
help='''Choose one of the backend-target pair to run this demo:
{:d}: (default) OpenCV implementation + CPU,
{:d}: CUDA + GPU (CUDA),
{:d}: CUDA + GPU (CUDA FP16),
{:d}: TIM-VX + NPU,
{:d}: CANN + NPU
'''.format(*[x for x in range(len(backend_target_pairs))]))
parser.add_argument('--conf_threshold', type=float, default=0.8,
help='Filter out hands of confidence < conf_threshold.')
parser.add_argument('--save', '-s', action='store_true',
help='Specify to save results. This flag is invalid when using camera.')
parser.add_argument('--vis', '-v', action='store_true',
help='Specify to open a window for result visualization. This flag is invalid when using camera.')
args = parser.parse_args()
def visualize(image, poses):
display_screen = image.copy()
display_3d = np.zeros((400, 400, 3), np.uint8)
cv.line(display_3d, (200, 0), (200, 400), (255, 255, 255), 2)
cv.line(display_3d, (0, 200), (400, 200), (255, 255, 255), 2)
cv.putText(display_3d, 'Main View', (0, 12), cv.FONT_HERSHEY_DUPLEX, 0.5, (0, 0, 255))
cv.putText(display_3d, 'Top View', (200, 12), cv.FONT_HERSHEY_DUPLEX, 0.5, (0, 0, 255))
cv.putText(display_3d, 'Left View', (0, 212), cv.FONT_HERSHEY_DUPLEX, 0.5, (0, 0, 255))
cv.putText(display_3d, 'Right View', (200, 212), cv.FONT_HERSHEY_DUPLEX, 0.5, (0, 0, 255))
is_draw = False # ensure only one person is drawn
def _draw_lines(image, landmarks, keep_landmarks, is_draw_point=True, thickness=2):
def _draw_by_presence(idx1, idx2):
if keep_landmarks[idx1] and keep_landmarks[idx2]:
cv.line(image, landmarks[idx1], landmarks[idx2], (255, 255, 255), thickness)
_draw_by_presence(0, 1)
_draw_by_presence(1, 2)
_draw_by_presence(2, 3)
_draw_by_presence(3, 7)
_draw_by_presence(0, 4)
_draw_by_presence(4, 5)
_draw_by_presence(5, 6)
_draw_by_presence(6, 8)
_draw_by_presence(9, 10)
_draw_by_presence(12, 14)
_draw_by_presence(14, 16)
_draw_by_presence(16, 22)
_draw_by_presence(16, 18)
_draw_by_presence(16, 20)
_draw_by_presence(18, 20)
_draw_by_presence(11, 13)
_draw_by_presence(13, 15)
_draw_by_presence(15, 21)
_draw_by_presence(15, 19)
_draw_by_presence(15, 17)
_draw_by_presence(17, 19)
_draw_by_presence(11, 12)
_draw_by_presence(11, 23)
_draw_by_presence(23, 24)
_draw_by_presence(24, 12)
_draw_by_presence(24, 26)
_draw_by_presence(26, 28)
_draw_by_presence(28, 30)
_draw_by_presence(28, 32)
_draw_by_presence(30, 32)
_draw_by_presence(23, 25)
_draw_by_presence(25, 27)
_draw_by_presence(27, 31)
_draw_by_presence(27, 29)
_draw_by_presence(29, 31)
if is_draw_point:
for i, p in enumerate(landmarks):
if keep_landmarks[i]:
cv.circle(image, p, thickness, (0, 0, 255), -1)
for idx, pose in enumerate(poses):
bbox, landmarks_screen, landmarks_word, mask, heatmap, conf = pose
edges = cv.Canny(mask, 100, 200)
kernel = np.ones((2, 2), np.uint8) # expansion edge to 2 pixels
edges = cv.dilate(edges, kernel, iterations=1)
edges_bgr = cv.cvtColor(edges, cv.COLOR_GRAY2BGR)
edges_bgr[edges == 255] = [0, 255, 0]
display_screen = cv.add(edges_bgr, display_screen)
# draw box
bbox = bbox.astype(np.int32)
cv.rectangle(display_screen, bbox[0], bbox[1], (0, 255, 0), 2)
cv.putText(display_screen, '{:.4f}'.format(conf), (bbox[0][0], bbox[0][1] + 12), cv.FONT_HERSHEY_DUPLEX, 0.5, (0, 0, 255))
# Draw line between each key points
landmarks_screen = landmarks_screen[:-6, :]
landmarks_word = landmarks_word[:-6, :]
keep_landmarks = landmarks_screen[:, 4] > 0.8 # only show visible keypoints which presence bigger than 0.8
landmarks_screen = landmarks_screen
landmarks_word = landmarks_word
landmarks_xy = landmarks_screen[:, 0: 2].astype(np.int32)
_draw_lines(display_screen, landmarks_xy, keep_landmarks, is_draw_point=False)
# z value is relative to HIP, but we use constant to instead
for i, p in enumerate(landmarks_screen[:, 0: 3].astype(np.int32)):
if keep_landmarks[i]:
cv.circle(display_screen, np.array([p[0], p[1]]), 2, (0, 0, 255), -1)
if is_draw is False:
is_draw = True
# Main view
landmarks_xy = landmarks_word[:, [0, 1]]
landmarks_xy = (landmarks_xy * 100 + 100).astype(np.int32)
_draw_lines(display_3d, landmarks_xy, keep_landmarks, thickness=2)
# Top view
landmarks_xz = landmarks_word[:, [0, 2]]
landmarks_xz[:, 1] = -landmarks_xz[:, 1]
landmarks_xz = (landmarks_xz * 100 + np.array([300, 100])).astype(np.int32)
_draw_lines(display_3d, landmarks_xz,keep_landmarks, thickness=2)
# Left view
landmarks_yz = landmarks_word[:, [2, 1]]
landmarks_yz[:, 0] = -landmarks_yz[:, 0]
landmarks_yz = (landmarks_yz * 100 + np.array([100, 300])).astype(np.int32)
_draw_lines(display_3d, landmarks_yz, keep_landmarks, thickness=2)
# Right view
landmarks_zy = landmarks_word[:, [2, 1]]
landmarks_zy = (landmarks_zy * 100 + np.array([300, 300])).astype(np.int32)
_draw_lines(display_3d, landmarks_zy, keep_landmarks, thickness=2)
return display_screen, display_3d
if __name__ == '__main__':
backend_id = backend_target_pairs[args.backend_target][0]
target_id = backend_target_pairs[args.backend_target][1]
# person detector
person_detector = MPPersonDet(modelPath='../person_detection_mediapipe/person_detection_mediapipe_2023mar.onnx',
nmsThreshold=0.3,
scoreThreshold=0.5,
topK=5000, # usually only one person has good performance
backendId=backend_id,
targetId=target_id)
# pose estimator
pose_estimator = MPPose(modelPath=args.model,
confThreshold=args.conf_threshold,
backendId=backend_id,
targetId=target_id)
# If input is an image
if args.input is not None:
image = cv.imread(args.input)
# person detector inference
persons = person_detector.infer(image)
poses = []
# Estimate the pose of each person
for person in persons:
# pose estimator inference
pose = pose_estimator.infer(image, person)
if pose is not None:
poses.append(pose)
# Draw results on the input image
image, view_3d = visualize(image, poses)
if len(persons) == 0:
print('No person detected!')
else:
print('Person detected!')
# Save results
if args.save:
cv.imwrite('result.jpg', image)
print('Results saved to result.jpg\n')
# Visualize results in a new window
if args.vis:
cv.namedWindow(args.input, cv.WINDOW_AUTOSIZE)
cv.imshow(args.input, image)
cv.imshow('3D Pose Demo', view_3d)
cv.waitKey(0)
else: # Omit input to call default camera
deviceId = 0
cap = cv.VideoCapture(deviceId)
tm = cv.TickMeter()
while cv.waitKey(1) < 0:
hasFrame, frame = cap.read()
if not hasFrame:
print('No frames grabbed!')
break
# person detector inference
persons = person_detector.infer(frame)
poses = []
tm.start()
# Estimate the pose of each person
for person in persons:
# pose detector inference
pose = pose_estimator.infer(frame, person)
if pose is not None:
poses.append(pose)
tm.stop()
# Draw results on the input image
frame, view_3d = visualize(frame, poses)
if len(persons) == 0:
print('No person detected!')
else:
print('Person detected!')
cv.putText(frame, 'FPS: {:.2f}'.format(tm.getFPS()), (0, 15), cv.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255))
cv.imshow('MediaPipe Pose Detection Demo', frame)
cv.imshow('3D Pose Demo', view_3d)
tm.reset()
models/pose_estimation_mediapipe/mp_pose.py 0 → 100644
浏览文件 @ 1f19f821
import numpy as np
import cv2 as cv
class MPPose:
def __init__(self, modelPath, confThreshold=0.5, backendId=0, targetId=0):
self.model_path = modelPath
self.conf_threshold = confThreshold
self.backend_id = backendId
self.target_id = targetId
self.input_size = np.array([256, 256]) # wh
# RoI will be larger so the performance will be better, but preprocess will be slower. Default to 1.
self.PERSON_BOX_PRE_ENLARGE_FACTOR = 1
self.PERSON_BOX_ENLARGE_FACTOR = 1.25
self.model = cv.dnn.readNet(self.model_path)
self.model.setPreferableBackend(self.backend_id)
self.model.setPreferableTarget(self.target_id)
@property
def name(self):
return self.__class__.__name__
def setBackendAndTarget(self, backendId, targetId):
self._backendId = backendId
self._targetId = targetId
self.model.setPreferableBackend(self.backend_id)
self.model.setPreferableTarget(self.target_id)
def _preprocess(self, image, person):
'''
Rotate input for inference.
Parameters:
image - input image of BGR channel order
face_bbox - human face bounding box found in image of format [[x1, y1], [x2, y2]] (top-left and bottom-right points)
person_landmarks - 4 landmarks (2 full body points, 2 upper body points) of shape [4, 2]
Returns:
rotated_person - rotated person image for inference
rotate_person_bbox - person box of interest range
angle - rotate angle for person
rotation_matrix - matrix for rotation and de-rotation
pad_bias - pad pixels of interest range
'''
# crop and pad image to interest range
pad_bias = np.array([0, 0], dtype=np.int32) # left, top
person_keypoints = person[4: 12].reshape(-1, 2)
mid_hip_point = person_keypoints[0]
full_body_point = person_keypoints[1]
# get RoI
full_dist = np.linalg.norm(mid_hip_point - full_body_point)
full_bbox = np.array([mid_hip_point - full_dist, mid_hip_point + full_dist], np.int32)
# enlarge to make sure full body can be cover
center_bbox = np.sum(full_bbox, axis=0) / 2
wh_bbox = full_bbox[1] - full_bbox[0]
new_half_size = wh_bbox * self.PERSON_BOX_PRE_ENLARGE_FACTOR / 2
full_bbox = np.array([
center_bbox - new_half_size,
center_bbox + new_half_size], np.int32)
person_bbox = full_bbox.copy()
# refine person bbox
person_bbox[:, 0] = np.clip(person_bbox[:, 0], 0, image.shape[1])
person_bbox[:, 1] = np.clip(person_bbox[:, 1], 0, image.shape[0])
# crop to the size of interest
image = image[person_bbox[0][1]:person_bbox[1][1], person_bbox[0][0]:person_bbox[1][0], :]
# pad to square
left, top = person_bbox[0] - full_bbox[0]
right, bottom = full_bbox[1] - person_bbox[1]
image = cv.copyMakeBorder(image, top, bottom, left, right, cv.BORDER_CONSTANT, None, (0, 0, 0))
pad_bias += person_bbox[0] - [left, top]
# compute rotation
mid_hip_point -= pad_bias
full_body_point -= pad_bias
radians = np.pi / 2 - np.arctan2(-(full_body_point[1] - mid_hip_point[1]), full_body_point[0] - mid_hip_point[0])
radians = radians - 2 * np.pi * np.floor((radians + np.pi) / (2 * np.pi))
angle = np.rad2deg(radians)
# get rotation matrix
rotation_matrix = cv.getRotationMatrix2D(mid_hip_point, angle, 1.0)
# get rotated image
rotated_image = cv.warpAffine(image, rotation_matrix, (image.shape[1], image.shape[0]))
# get landmark bounding box
blob = cv.resize(rotated_image, dsize=self.input_size, interpolation=cv.INTER_AREA).astype(np.float32)
rotated_person_bbox = np.array([[0, 0], [image.shape[1], image.shape[0]]], dtype=np.int32)
blob = cv.cvtColor(blob, cv.COLOR_BGR2RGB)
blob = blob / 255. # [0, 1]
return blob[np.newaxis, :, :, :], rotated_person_bbox, angle, rotation_matrix, pad_bias
def infer(self, image, person):
h, w, _ = image.shape
# Preprocess
input_blob, rotated_person_bbox, angle, rotation_matrix, pad_bias = self._preprocess(image, person)
# Forward
self.model.setInput(input_blob)
output_blob = self.model.forward(self.model.getUnconnectedOutLayersNames())
# Postprocess
results = self._postprocess(output_blob, rotated_person_bbox, angle, rotation_matrix, pad_bias, np.array([w, h]))
return results # [bbox_coords, landmarks_coords, conf]
def _postprocess(self, blob, rotated_person_bbox, angle, rotation_matrix, pad_bias, img_size):
landmarks, conf, mask, heatmap, landmarks_word = blob
conf = conf[0][0]
if conf < self.conf_threshold:
return None
landmarks = landmarks[0].reshape(-1, 5) # shape: (1, 195) -> (39, 5)
landmarks_word = landmarks_word[0].reshape(-1, 3) # shape: (1, 117) -> (39, 3)
# recover sigmoid score
landmarks[:, 3:] = 1 / (1 + np.exp(-landmarks[:, 3:]))
# TODO: refine landmarks with heatmap. reference: https://github.com/tensorflow/tfjs-models/blob/master/pose-detection/src/blazepose_tfjs/detector.ts#L577-L582
heatmap = heatmap[0]
# transform coords back to the input coords
wh_rotated_person_bbox = rotated_person_bbox[1] - rotated_person_bbox[0]
scale_factor = wh_rotated_person_bbox / self.input_size
landmarks[:, :2] = (landmarks[:, :2] - self.input_size / 2) * scale_factor
landmarks[:, 2] = landmarks[:, 2] * max(scale_factor) # depth scaling
coords_rotation_matrix = cv.getRotationMatrix2D((0, 0), angle, 1.0)
rotated_landmarks = np.dot(landmarks[:, :2], coords_rotation_matrix[:, :2])
rotated_landmarks = np.c_[rotated_landmarks, landmarks[:, 2:]]
rotated_landmarks_world = np.dot(landmarks_word[:, :2], coords_rotation_matrix[:, :2])
rotated_landmarks_world = np.c_[rotated_landmarks_world, landmarks_word[:, 2]]
# invert rotation
rotation_component = np.array([
[rotation_matrix[0][0], rotation_matrix[1][0]],
[rotation_matrix[0][1], rotation_matrix[1][1]]])
translation_component = np.array([
rotation_matrix[0][2], rotation_matrix[1][2]])
inverted_translation = np.array([
-np.dot(rotation_component[0], translation_component),
-np.dot(rotation_component[1], translation_component)])
inverse_rotation_matrix = np.c_[rotation_component, inverted_translation]
# get box center
center = np.append(np.sum(rotated_person_bbox, axis=0) / 2, 1)
original_center = np.array([
np.dot(center, inverse_rotation_matrix[0]),
np.dot(center, inverse_rotation_matrix[1])])
landmarks[:, :2] = rotated_landmarks[:, :2] + original_center + pad_bias
# get bounding box from rotated_landmarks
bbox = np.array([
np.amin(landmarks[:, :2], axis=0),
np.amax(landmarks[:, :2], axis=0)]) # [top-left, bottom-right]
center_bbox = np.sum(bbox, axis=0) / 2
wh_bbox = bbox[1] - bbox[0]
new_half_size = wh_bbox * self.PERSON_BOX_ENLARGE_FACTOR / 2
bbox = np.array([
center_bbox - new_half_size,
center_bbox + new_half_size])
# invert rotation for mask
mask = mask[0].reshape(256, 256) # shape: (1, 256, 256, 1) -> (256, 256)
invert_rotation_matrix = cv.getRotationMatrix2D((mask.shape[1]/2, mask.shape[0]/2), -angle, 1.0)
invert_rotation_mask = cv.warpAffine(mask, invert_rotation_matrix, (mask.shape[1], mask.shape[0]))
# enlarge mask
invert_rotation_mask = cv.resize(invert_rotation_mask, wh_rotated_person_bbox)
# crop and pad mask
min_w, min_h = -np.minimum(pad_bias, 0)
left, top = np.maximum(pad_bias, 0)
pad_over = img_size - [invert_rotation_mask.shape[1], invert_rotation_mask.shape[0]] - pad_bias
max_w, max_h = np.minimum(pad_over, 0) + [invert_rotation_mask.shape[1], invert_rotation_mask.shape[0]]
right, bottom = np.maximum(pad_over, 0)
invert_rotation_mask = invert_rotation_mask[min_h:max_h, min_w:max_w]
invert_rotation_mask = cv.copyMakeBorder(invert_rotation_mask, top, bottom, left, right, cv.BORDER_CONSTANT, None, 0)
# binarize mask
invert_rotation_mask = np.where(invert_rotation_mask > 0, 255, 0).astype(np.uint8)
# 2*2 person bbox: [[x1, y1], [x2, y2]]
# 39*5 screen landmarks: 33 keypoints and 6 auxiliary points with [x, y, z, visibility, presence], z value is relative to HIP
# Visibility is probability that a keypoint is located within the frame and not occluded by another bigger body part or another object
# Presence is probability that a keypoint is located within the frame
# 39*3 world landmarks: 33 keypoints and 6 auxiliary points with [x, y, z] 3D metric x, y, z coordinate
# img_height*img_width mask: gray mask, where 255 indicates the full body of a person and 0 means background
# 64*64*39 heatmap: currently only used for refining landmarks, requires sigmod processing before use
# conf: confidence of prediction
return [bbox, landmarks, rotated_landmarks_world, invert_rotation_mask, heatmap, conf]
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或