diff --git a/README.md b/README.md index b5c0e8dc802a38ddfdecb956b7dae2d24d365de2..0371c8a465439c078bb416ec4db114e4db050d96 100644 --- a/README.md +++ b/README.md @@ -27,19 +27,17 @@ OpenPose represents the **first real-time multi-person system to jointly detect -## Latest News +## Latest Features - Sep 2017: **CMake** installer and **IP camera** support! - Jul 2017: [**Windows portable demo**](doc/installation.md#installation---demo)! - Jul 2017: **Hands** released! - Jun 2017: **Face** released! -- May 2017: **Windows** version! -- Apr 2017: **Body** released! -- Check all the [release notes](doc/release_notes.md). +For further details, check [all released features](doc/released_features.md) and [release notes](doc/release_notes.md). ## Contents -1. [Latest News](#latest-news) +1. [Latest Features](#latest-features) 2. [Results](#results) 3. [Installation, Reinstallation and Uninstallation](#installation-reinstallation-and-uninstallation) 4. [Quick Start](#quick-start) diff --git a/doc/library_how_to_debug.md b/doc/library_how_to_debug.md new file mode 100644 index 0000000000000000000000000000000000000000..5abc3fa1da4a3c14ba180d66531ba72052fe05e7 --- /dev/null +++ b/doc/library_how_to_debug.md @@ -0,0 +1,12 @@ +OpenPose C++ API - How to Debug OpenPose +====================================================== + +# Finding Segmentation Faults +This is the faster method to debug a segmentation fault problem. Usual scenario: You are editing OpenPose source code and suddenly OpenPose returns segmentation fault when executed. In order to find where it occurs: + 1. Select one of the 2 options: + 1. Switch to debug mode. + 2. Go to `openpose/utilities/errorAndLog.hpp` and modify `dLog`: + 1. Comment `#ifndef NDEBUG` and its else and endif. + 2. Call OpenPose with `--logging_level 0 --disable_multi_thread`. + 3. At this point you have an idea of in which file class the segmentation fault is coming from. Now you can further isolate the error by iteratively adding the following line all over the code until you find the exact position of the segmentation fault: `log("", Priority::Low, __LINE__, __FUNCTION__, __FILE__);` + 4. After you have found the segmentation fault, remember to remove all the extra `log()` calls that you temporaryly added. diff --git a/doc/output.md b/doc/output.md index 5647c15c2ceddfeca0ae132ddca70af53213f0b1..e9c936aa7f515079cfd3e1efc8215b9883153e89 100644 --- a/doc/output.md +++ b/doc/output.md @@ -4,44 +4,85 @@ OpenPose Demo - Output ## Output Format -There are 2 alternatives to save the **(x,y,score) body part locations**. The `write_keypoint` flag uses the OpenCV cv::FileStorage default formats (JSON, XML and YML). However, the JSON format is only available after OpenCV 3.0. Hence, `write_keypoint_json` saves the people pose data using a custom JSON writer. For the latter, each JSON file has a `people` array of objects, where each object has an array `pose_keypoints` containing the body part locations and detection confidence formatted as `x1,y1,c1,x2,y2,c2,...`. The coordinates `x` and `y` can be normalized to the range [0,1], [-1,1], [0, source size], [0, output size], etc., depending on the flag `keypoint_scale`. In addition, `c` is the confidence in the range [0,1]. +There are 2 alternatives to save the OpenPose output. + 1. The `write_json` flag saves the people pose data using a custom JSON writer. Each JSON file has a `people` array of objects, where each object has: + 1. An array `pose_keypoints` containing the body part locations and detection confidence formatted as `x1,y1,c1,x2,y2,c2,...`. The coordinates `x` and `y` can be normalized to the range [0,1], [-1,1], [0, source size], [0, output size], etc., depending on the flag `keypoint_scale`, while `c` is the confidence score in the range [0,1]. + 2. The arrays `face_keypoints`, `hand_left_keypoints`, and `hand_right_keypoints`, analogous to `pose_keypoints`. + 3. The body part candidates before being assembled into people (if `--part_candidates` is enabled). ``` { - "version":0.1, + "version":1.1, "people":[ - {"pose_keypoints":[1114.15,160.396,0.846207,...]}, - {"pose_keypoints":[...]}, + { + "pose_keypoints":[582.349,507.866,0.845918,746.975,631.307,0.587007,...], + "face_keypoints":[468.725,715.636,0.189116,554.963,652.863,0.665039,...], + "hand_left_keypoints":[746.975,631.307,0.587007,615.659,617.567,0.377899,...], + "hand_right_keypoints":[617.581,472.65,0.797508,0,0,0,723.431,462.783,0.88765,...] + } + ], + // If `--part_candidates` enabled + "part_candidates":[ + { + "0":[296.994,258.976,0.845918,238.996,365.027,0.189116], + "1":[381.024,321.984,0.587007], + "2":[313.996,314.97,0.377899], + "3":[238.996,365.027,0.189116], + "4":[283.015,332.986,0.665039], + "5":[457.987,324.003,0.430488,283.015,332.986,0.665039], + "6":[], + "7":[], + "8":[], + "9":[], + "10":[], + "11":[], + "12":[], + "13":[], + "14":[293.001,242.991,0.674305], + "15":[314.978,241,0.797508], + "16":[], + "17":[369.007,235.964,0.88765] + } ] } ``` + 2. (Deprecated) The `write_keypoint` flag uses the OpenCV cv::FileStorage default formats, i.e. JSON (available after OpenCV 3.0), XML, and YML. Note that it does not include any other information othern than keypoints. -The body part order of the COCO (18 body parts) and MPI (15 body parts) keypoints is described in `POSE_BODY_PART_MAPPING` in [src/openpose/pose/poseParameters.cpp](../src/openpose/pose/poseParameters.cpp). E.g., for COCO: + + +## Keypoint Ordering +The body part mapping order of any body model (e.g. COCO, MPI) can be extracted from the C++ API by using the `getPoseBodyPartMapping(const PoseModel poseModel)` function available in [poseParameters.hpp](../include/openpose/pose/poseParameters.hpp): ``` - POSE_COCO_BODY_PARTS { - {0, "Nose"}, - {1, "Neck"}, - {2, "RShoulder"}, - {3, "RElbow"}, - {4, "RWrist"}, - {5, "LShoulder"}, - {6, "LElbow"}, - {7, "LWrist"}, - {8, "RHip"}, - {9, "RKnee"}, - {10, "RAnkle"}, - {11, "LHip"}, - {12, "LKnee"}, - {13, "LAnkle"}, - {14, "REye"}, - {15, "LEye"}, - {16, "REar"}, - {17, "LEar"}, - {18, "Background"}, - } +// C++ API call +#include +const auto& poseBodyPartMappingCoco = getPoseBodyPartMapping(PoseModel::COCO_18) +const auto& poseBodyPartMappingMpi = getPoseBodyPartMapping(PoseModel::MPI_15) + +// Result for COCO (18 body parts) +// POSE_COCO_BODY_PARTS { +// {0, "Nose"}, +// {1, "Neck"}, +// {2, "RShoulder"}, +// {3, "RElbow"}, +// {4, "RWrist"}, +// {5, "LShoulder"}, +// {6, "LElbow"}, +// {7, "LWrist"}, +// {8, "RHip"}, +// {9, "RKnee"}, +// {10, "RAnkle"}, +// {11, "LHip"}, +// {12, "LKnee"}, +// {13, "LAnkle"}, +// {14, "REye"}, +// {15, "LEye"}, +// {16, "REar"}, +// {17, "LEar"}, +// {18, "Background"}, +// } ``` -For the **heat maps storing format**, instead of individually saving each of the 67 heatmaps (18 body parts + background + 2 x 19 PAFs) individually, the library concatenates them into a huge (width x #heat maps) x (height) matrix, i.e. it concats the heat maps by columns. E.g., columns [0, individual heat map width] contains the first heat map, columns [individual heat map width + 1, 2 * individual heat map width] contains the second heat map, etc. Note that some image viewers are not able to display the resulting images due to the size. However, Chrome and Firefox are able to properly open them. +For the **heat maps storing format**, instead of saving each of the 67 heatmaps (18 body parts + background + 2 x 19 PAFs) individually, the library concatenates them into a huge (width x #heat maps) x (height) matrix (i.e., concatenated by columns). E.g., columns [0, individual heat map width] contains the first heat map, columns [individual heat map width + 1, 2 * individual heat map width] contains the second heat map, etc. Note that some image viewers are not able to display the resulting images due to the size. However, Chrome and Firefox are able to properly open them. The saving order is body parts + background + PAFs. Any of them can be disabled with program flags. If background is disabled, then the final image will be body parts + PAFs. The body parts and background follow the order of `POSE_COCO_BODY_PARTS` or `POSE_MPI_BODY_PARTS`, while the PAFs follow the order specified on POSE_BODY_PART_PAIRS in `poseParameters.hpp`. E.g., for COCO: ``` @@ -121,7 +162,7 @@ There are 3 different keypoint Array elements on this class: ## Reading Saved Results -We use standard formats (JSON, XML, PNG, JPG, ...) to save our results, so there will be lots of frameworks to read them later, but you might also directly use our functions in [include/openpose/filestream/fileStream.hpp](../include/openpose/filestream/fileStream.hpp). In particular, `loadData` (for JSON, XML and YML files) and `loadImage` (for image formats such as PNG or JPG) to load the data into cv::Mat format. +We use standard formats (JSON, XML, PNG, JPG, ...) to save our results, so there are many open-source libraries to read them in most programming languages. From C++, but you might the functions in [include/openpose/filestream/fileStream.hpp](../include/openpose/filestream/fileStream.hpp). In particular, `loadData` (for JSON, XML and YML files) and `loadImage` (for image formats such as PNG or JPG) to load the data into cv::Mat format. diff --git a/doc/released_features.md b/doc/released_features.md new file mode 100644 index 0000000000000000000000000000000000000000..c2cffd7f112ce125fd65b54d403ed4a6bb102b4f --- /dev/null +++ b/doc/released_features.md @@ -0,0 +1,10 @@ +OpenPose Library - Latest Released Features +==================================== + +- Sep 2017: **CMake** installer and **IP camera** support! +- Jul 2017: [**Windows portable demo**](doc/installation.md#installation---demo)! +- Jul 2017: **Hands** released! +- Jun 2017: **Face** released! +- May 2017: **Windows** version! +- Apr 2017: **Body** released! +For further details, check the [release notes](./release_notes.md). diff --git a/include/openpose/face/faceParameters.hpp b/include/openpose/face/faceParameters.hpp index 348a4d33138605faa55e8f452d09e8fc951f3e51..42eb1b62e0f71360d156eb3a3baea9573c447b3c 100644 --- a/include/openpose/face/faceParameters.hpp +++ b/include/openpose/face/faceParameters.hpp @@ -8,13 +8,16 @@ namespace op const auto FACE_MAX_FACES = POSE_MAX_PEOPLE; const auto FACE_NUMBER_PARTS = 70u; - #define FACE_PAIRS_RENDER_GPU {0,1, 1,2, 2,3, 3,4, 4,5, 5,6, 6,7, 7,8, 8,9, 9,10, 10,11, 11,12, 12,13, 13,14, 14,15, 15,16, 17,18, 18,19, 19,20, \ - 20,21, 22,23, 23,24, 24,25, 25,26, 27,28, 28,29, 29,30, 31,32, 32,33, 33,34, 34,35, 36,37, 37,38, 38,39, 39,40, 40,41, \ - 41,36, 42,43, 43,44, 44,45, 45,46, 46,47, 47,42, 48,49, 49,50, 50,51, 51,52, 52,53, 53,54, 54,55, 55,56, 56,57, 57,58, \ - 58,59, 59,48, 60,61, 61,62, 62,63, 63,64, 64,65, 65,66, 66,67, 67,60} + #define FACE_PAIRS_RENDER_GPU \ + 0,1, 1,2, 2,3, 3,4, 4,5, 5,6, 6,7, 7,8, 8,9, 9,10, 10,11, 11,12, 12,13, 13,14, 14,15, 15,16, 17,18, 18,19, 19,20, \ + 20,21, 22,23, 23,24, 24,25, 25,26, 27,28, 28,29, 29,30, 31,32, 32,33, 33,34, 34,35, 36,37, 37,38, 38,39, 39,40, 40,41, \ + 41,36, 42,43, 43,44, 44,45, 45,46, 46,47, 47,42, 48,49, 49,50, 50,51, 51,52, 52,53, 53,54, 54,55, 55,56, 56,57, 57,58, \ + 58,59, 59,48, 60,61, 61,62, 62,63, 63,64, 64,65, 65,66, 66,67, 67,60 + #define FACE_SCALES_RENDER_GPU 1 const std::vector FACE_PAIRS_RENDER {FACE_PAIRS_RENDER_GPU}; #define FACE_COLORS_RENDER_GPU 255.f, 255.f, 255.f const std::vector FACE_COLORS_RENDER{FACE_COLORS_RENDER_GPU}; + const std::vector FACE_SCALES_RENDER{FACE_SCALES_RENDER_GPU}; // Constant parameters const auto FACE_CCN_DECREASE_FACTOR = 8.f; diff --git a/include/openpose/hand/handParameters.hpp b/include/openpose/hand/handParameters.hpp index 7f214fa86b28f86593e6b7f17a2c9c6e2d57f385..9cf6f206424b3b421d9140148686f88b23d7129c 100644 --- a/include/openpose/hand/handParameters.hpp +++ b/include/openpose/hand/handParameters.hpp @@ -8,7 +8,9 @@ namespace op const auto HAND_MAX_HANDS = 2*POSE_MAX_PEOPLE; const auto HAND_NUMBER_PARTS = 21u; - #define HAND_PAIRS_RENDER_GPU {0,1, 1,2, 2,3, 3,4, 0,5, 5,6, 6,7, 7,8, 0,9, 9,10, 10,11, 11,12, 0,13, 13,14, 14,15, 15,16, 0,17, 17,18, 18,19, 19,20} + #define HAND_PAIRS_RENDER_GPU \ + 0,1, 1,2, 2,3, 3,4, 0,5, 5,6, 6,7, 7,8, 0,9, 9,10, 10,11, 11,12, 0,13, 13,14, 14,15, 15,16, 0,17, 17,18, 18,19, 19,20 + #define HAND_SCALES_RENDER_GPU 1 const std::vector HAND_PAIRS_RENDER {HAND_PAIRS_RENDER_GPU}; #define HAND_COLORS_RENDER_GPU \ 100.f, 100.f, 100.f, \ @@ -33,6 +35,7 @@ namespace op 200.f, 0.f, 200.f, \ 255.f, 0.f, 255.f const std::vector HAND_COLORS_RENDER{HAND_COLORS_RENDER_GPU}; + const std::vector HAND_SCALES_RENDER{HAND_SCALES_RENDER_GPU}; // Constant parameters diff --git a/include/openpose/pose/poseParametersRender.hpp b/include/openpose/pose/poseParametersRender.hpp index 661c88877e9afffad68e7cfe5b77aab59532189e..a1f53479affeb9b6d7dea3f3d07b37f91008a70c 100644 --- a/include/openpose/pose/poseParametersRender.hpp +++ b/include/openpose/pose/poseParametersRender.hpp @@ -13,9 +13,9 @@ namespace op // Model-Dependent Parameters // CUDA-code Model-Dependent Parameters must be defined with #define // COCO - #define POSE_COCO_PAIRS_RENDER_GPU { \ - 1,2, 1,5, 2,3, 3,4, 5,6, 6,7, 1,8, 8,9, 9,10, 1,11, 11,12, 12,13, 1,0, 0,14, 14,16, 0,15, 15,17 \ - } + #define POSE_COCO_PAIRS_RENDER_GPU \ + 1,2, 1,5, 2,3, 3,4, 5,6, 6,7, 1,8, 8,9, 9,10, 1,11, 11,12, 12,13, 1,0, 0,14, 14,16, 0,15, 15,17 + #define POSE_COCO_SCALES_RENDER_GPU 1 #define POSE_COCO_COLORS_RENDER_GPU \ 255.f, 0.f, 85.f, \ 255.f, 0.f, 0.f, \ @@ -37,9 +37,9 @@ namespace op 85.f, 0.f, 255.f // MPI // MPI colors chosen such that they are closed to COCO colors - #define POSE_MPI_PAIRS_RENDER_GPU { \ - 0,1, 1,2, 2,3, 3,4, 1,5, 5,6, 6,7, 1,14, 14,8, 8,9, 9,10, 14,11, 11,12, 12,13 \ - } + #define POSE_MPI_PAIRS_RENDER_GPU \ + 0,1, 1,2, 2,3, 3,4, 1,5, 5,6, 6,7, 1,14, 14,8, 8,9, 9,10, 14,11, 11,12, 12,13 + #define POSE_MPI_SCALES_RENDER_GPU 1 #define POSE_MPI_COLORS_RENDER_GPU \ 255.f, 0.f, 85.f, \ 255.f, 0.f, 0.f, \ @@ -58,11 +58,12 @@ namespace op 0.f, 0.f, 255.f // BODY_18 #define POSE_BODY_18_PAIRS_RENDER_GPU POSE_COCO_PAIRS_RENDER_GPU + #define POSE_BODY_18_SCALES_RENDER_GPU POSE_COCO_SCALES_RENDER_GPU #define POSE_BODY_18_COLORS_RENDER_GPU POSE_COCO_COLORS_RENDER_GPU // BODY_19 - #define POSE_BODY_19_PAIRS_RENDER_GPU { \ - 1,8, 1,2, 1,5, 2,3, 3,4, 5,6, 6,7, 8,9, 9,10, 10,11, 8,12, 12,13, 13,14, 1,0, 0,15, 15,17, 0,16, 16,18 \ - } + #define POSE_BODY_19_PAIRS_RENDER_GPU \ + 1,8, 1,2, 1,5, 2,3, 3,4, 5,6, 6,7, 8,9, 9,10, 10,11, 8,12, 12,13, 13,14, 1,0, 0,15, 15,17, 0,16, 16,18 + #define POSE_BODY_19_SCALES_RENDER_GPU 1 #define POSE_BODY_19_COLORS_RENDER_GPU \ 255.f, 0.f, 85.f, \ 255.f, 0.f, 0.f, \ @@ -84,9 +85,9 @@ namespace op 255.f, 0.f, 255.f, \ 85.f, 0.f, 255.f // BODY_23 - #define POSE_BODY_23_PAIRS_RENDER_GPU { \ - 0,1, 0,4, 1,2, 2,3, 4,5, 5,6, 0,7, 7,8, 7,13, 8,9, 9,10,10,11,11,12,13,14,14,15,15,16,16,17, 0,18,18,19,18,21,19,20,21,22 \ - } + #define POSE_BODY_23_PAIRS_RENDER_GPU \ + 0,1, 0,4, 1,2, 2,3, 4,5, 5,6, 0,7, 7,8, 7,13, 8,9, 9,10,10,11,11,12,13,14,14,15,15,16,16,17, 0,18,18,19,18,21,19,20,21,22 + #define POSE_BODY_23_SCALES_RENDER_GPU 1 #define POSE_BODY_23_COLORS_RENDER_GPU \ 255.f, 0.f, 0.f, \ 255.f, 55.f, 0.f, \ @@ -112,12 +113,15 @@ namespace op 138.f, 43.f, 226.f, \ 75.f, 0.f, 130.f // BODY_59 - #define POSE_BODY_59_PAIRS_RENDER_GPU { \ + // Body + left hand + right hand + #define POSE_BODY_59_PAIRS_RENDER_GPU \ 1,8, 1,2, 1,5, 2,3, 3,4, 5,6, 6,7, 8,9, 9,10, 10,11, 8,12, 12,13, 13,14, 1,0, 0,15, 15,17, 0,16, 16,18, \ 7,19, 19,20, 20,21, 21,22, 7,23, 23,24, 24,25, 25,26, 7,27, 27,28, 28,29, 29,30, 7,31, 31,32, 32,33, 33,34, 7,35, 35,36, 36,37, 37,38, \ - 4,39, 39,40, 40,41, 41,42, 4,43, 43,44, 44,45, 45,46, 4,47, 47,48, 48,49, 49,50, 4,51, 51,52, 52,53, 53,54, 4,55, 55,56, 56,57, 57,58 \ - } - // Body + left hand + right hand + 4,39, 39,40, 40,41, 41,42, 4,43, 43,44, 44,45, 45,46, 4,47, 47,48, 48,49, 49,50, 4,51, 51,52, 52,53, 53,54, 4,55, 55,56, 56,57, 57,58 + #define POSE_BODY_59_SCALES_RENDER_GPU \ + 1.f,1.f,1.f,1.f,1.f, 1.f,1.f,1.f,1.f,1.f, 1.f,1.f,1.f,1.f,1.f, 1.f,1.f,1.f,1.f, \ + 0.5f,0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f,0.5f, \ + 0.5f,0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f,0.5f #define POSE_BODY_59_COLORS_RENDER_GPU \ 255.f, 0.f, 85.f, \ 255.f, 0.f, 0.f, \ @@ -138,50 +142,62 @@ namespace op 170.f, 0.f, 255.f, \ 255.f, 0.f, 255.f, \ 85.f, 0.f, 255.f, \ - \ - 100.f, 0.f, 0.f, \ - 150.f, 0.f, 0.f, \ - 200.f, 0.f, 0.f, \ + \ 255.f, 0.f, 0.f, \ - 100.f, 100.f, 0.f, \ - 150.f, 150.f, 0.f, \ - 200.f, 200.f, 0.f, \ + 191.f, 47.f, 47.f, \ + 127.f, 63.f, 63.f, \ + 63.f, 47.f, 47.f, \ + 255.f, 76.f, 0.f, \ + 191.f, 57.f, 0.f, \ + 127.f, 38.f, 0.f, \ + 63.f, 19.f, 0.f, \ + 255.f, 152.f, 0.f, \ + 191.f, 114.f, 0.f, \ + 127.f, 76.f, 0.f, \ + 63.f, 38.f, 0.f, \ 255.f, 255.f, 0.f, \ - 0.f, 100.f, 50.f, \ - 0.f, 150.f, 75.f, \ - 0.f, 200.f, 100.f, \ - 0.f, 255.f, 125.f, \ - 0.f, 50.f, 100.f, \ - 0.f, 75.f, 150.f, \ - 0.f, 100.f, 200.f, \ - 0.f, 125.f, 255.f, \ - 100.f, 0.f, 100.f, \ - 150.f, 0.f, 150.f, \ - 200.f, 0.f, 200.f, \ - 255.f, 0.f, 255.f, \ - \ - 100.f, 0.f, 0.f, \ - 150.f, 0.f, 0.f, \ - 200.f, 0.f, 0.f, \ - 255.f, 0.f, 0.f, \ - 100.f, 100.f, 0.f, \ - 150.f, 150.f, 0.f, \ - 200.f, 200.f, 0.f, \ - 255.f, 255.f, 0.f, \ - 0.f, 100.f, 50.f, \ - 0.f, 150.f, 75.f, \ - 0.f, 200.f, 100.f, \ - 0.f, 255.f, 125.f, \ - 0.f, 50.f, 100.f, \ - 0.f, 75.f, 150.f, \ - 0.f, 100.f, 200.f, \ - 0.f, 125.f, 255.f, \ - 100.f, 0.f, 100.f, \ - 150.f, 0.f, 150.f, \ - 200.f, 0.f, 200.f, \ - 255.f, 0.f, 255.f + 191.f, 191.f, 0.f, \ + 127.f, 127.f, 0.f, \ + 63.f, 63.f, 0.f, \ + 0.f, 255.f, 0.f, \ + 0.f, 191.f, 0.f, \ + 0.f, 127.f, 0.f, \ + 0.f, 63.f, 0.f, \ + \ + 255.f, 0.f, 153.f, \ + 191.f, 0.f, 114.f, \ + 127.f, 0.f, 76.f, \ + 63.f, 0.f, 38.f, \ + 203.f, 0.f, 255.f, \ + 152.f, 0.f, 191.f, \ + 101.f, 0.f, 127.f, \ + 50.f, 0.f, 63.f, \ + 50.f, 0.f, 255.f, \ + 37.f, 0.f, 191.f, \ + 25.f, 0.f, 127.f, \ + 12.f, 0.f, 63.f, \ + 0.f, 102.f, 255.f, \ + 0.f, 76.f, 191.f, \ + 0.f, 51.f, 127.f, \ + 0.f, 25.f, 63.f, \ + 0.f, 255.f, 255.f, \ + 0.f, 191.f, 191.f, \ + 0.f, 127.f, 127.f, \ + 0.f, 63.f, 63.f + + // Hand color selection + // http://www.perbang.dk/rgbgradient/ + // 1. Main color + // - Each finger of the right hand: 11 steps from FF0000 to FF0001 and pick last 5 from HSV gradient. + // - Each finger of the left hand: 21 steps from FF0000 to FF0001, choosing 4 among first 6 (HSV grad.), + // and then green. + // Note: Choosing first 5 from 11 steps was giving 2 very close greens + // 2. Gradient color from wrist to finger tips + // - Inside each finger: 5 steps from main color to 000000, and selecting first 4 from RGB gradient. + // Note: Used HSV gradient for red finger. // Rendering functions + OP_API const std::vector& getPoseScales(const PoseModel poseModel); OP_API const std::vector& getPoseColors(const PoseModel poseModel); OP_API const std::vector& getPoseBodyPartPairsRender(const PoseModel poseModel); } diff --git a/include/openpose/utilities/keypoint.hpp b/include/openpose/utilities/keypoint.hpp index 87bc84ceaf52f8cd0a29898a681f9169400cea4f..343d09ba66aea497a307c440bd913f509f3e4b14 100644 --- a/include/openpose/utilities/keypoint.hpp +++ b/include/openpose/utilities/keypoint.hpp @@ -19,7 +19,7 @@ namespace op OP_API void renderKeypointsCpu(Array& frameArray, const Array& keypoints, const std::vector& pairs, const std::vector colors, const float thicknessCircleRatio, const float thicknessLineRatioWRTCircle, - const float threshold); + const std::vector& poseScales, const float threshold); OP_API Rectangle getKeypointsRectangle(const Array& keypoints, const int person, const float threshold); diff --git a/include/openpose/utilities/render.hu b/include/openpose/utilities/render.hu index 3990728d6463bcfa3fcae2bad9116ecd9633938e..877506002f54795bc11fc281b758b633c71021c1 100644 --- a/include/openpose/utilities/render.hu +++ b/include/openpose/utilities/render.hu @@ -3,13 +3,16 @@ namespace op { - inline __device__ void renderKeypoints(float* targetPtr, float2* sharedMaxs, float2* sharedMins, float* sharedScaleF, - const int globalIdx, const int x, const int y, const int targetWidth, const int targetHeight, + inline __device__ void renderKeypoints(float* targetPtr, float2* sharedMaxs, float2* sharedMins, + float* sharedScaleF, const int globalIdx, const int x, const int y, + const int targetWidth, const int targetHeight, const float* const keypointsPtr, const unsigned int* const partPairsPtr, const int numberPeople, const int numberParts, const int numberPartPairs, - const float* const rgbColorsPtr, const int numberColors, - const float radius, const float stickwidth, const float threshold, const float alphaColorToAdd, - const bool blendOriginalFrame = true, const int googlyEye1 = -1, const int googlyEye2 = -1) + const float* const rgbColorsPtr, const int numberColors, const float radius, + const float lineWidth, const float* const keypointScalePtr, + const int numberScales, const float threshold, + const float alphaColorToAdd, const bool blendOriginalFrame = true, + const int googlyEye1 = -1, const int googlyEye2 = -1) { // Fill shared parameters if (globalIdx < numberPeople) @@ -40,7 +43,8 @@ namespace op { const auto averageX = sharedMaxs[globalIdx].x - sharedMins[globalIdx].x; const auto averageY = sharedMaxs[globalIdx].y - sharedMins[globalIdx].y; - sharedScaleF[globalIdx] = fastTruncate((averageX + averageY) / 400.f, 0.33f, 1.f); // (averageX + averageY) / 2.f / 400.f + // (averageX + averageY) / 2.f / 400.f + sharedScaleF[globalIdx] = fastTruncate((averageX + averageY) / 400.f, 0.33f, 1.f); const auto constantToAdd = 50.f; sharedMaxs[globalIdx].x += constantToAdd; sharedMaxs[globalIdx].y += constantToAdd; @@ -65,17 +69,19 @@ namespace op r = 0.f; } + const auto lineWidthSquared = lineWidth * lineWidth; + const auto radiusSquared = radius * radius; for (auto person = 0; person < numberPeople; person++) { // Make sure person x,y in the limits // Make sure person is not empty. Assume all joints are below threshold. Then // maxs = 0 and mins = width/height. So if statement would be false - if (x <= sharedMaxs[person].x && x >= sharedMins[person].x && y <= sharedMaxs[person].y && y >= sharedMins[person].y) + if (x <= sharedMaxs[person].x && x >= sharedMins[person].x + && y <= sharedMaxs[person].y && y >= sharedMins[person].y) { // Part pair connections for (auto partPair = 0; partPair < numberPartPairs; partPair++) { - const auto bSqrt = sharedScaleF[person] * sharedScaleF[person] * stickwidth * stickwidth; const auto partA = partPairsPtr[2*partPair]; const auto partB = partPairsPtr[2*partPair+1]; const auto indexA = person*numberParts*3 + partA*3; @@ -89,6 +95,12 @@ namespace op if (scoreA > threshold && scoreB > threshold) { + const auto keypointScale = keypointScalePtr[partB%numberScales] + * keypointScalePtr[partB%numberScales] + * keypointScalePtr[partB%numberScales]; + const auto lineWidthScaled = lineWidthSquared * keypointScale; + const auto bSqrt = sharedScaleF[person] * sharedScaleF[person] * lineWidthScaled; + const auto xP = (xA + xB) / 2.f; const auto yP = (yA + yB) / 2.f; const auto aSqrt = (xA - xP) * (xA - xP) + (yA - yP) * (yA - yP); @@ -103,27 +115,33 @@ namespace op const auto minV = 0.f; const auto maxV = 1.f; if (minV <= judge && judge <= maxV) - addColorWeighted(r, g, b, &rgbColorsPtr[(partB%numberColors)*3], alphaColorToAdd); // Before used partPair vs partB + // Before used partPair vs partB + addColorWeighted(r, g, b, &rgbColorsPtr[(partB%numberColors)*3], alphaColorToAdd); } } // Part circles - for (auto i = 0u; i < numberParts; i++) + for (auto part = 0u; part < numberParts; part++) { - const auto index = 3 * (person*numberParts + i); + const auto index = 3 * (person*numberParts + part); const auto localX = keypointsPtr[index]; const auto localY = keypointsPtr[index + 1]; const auto score = keypointsPtr[index + 2]; if (score > threshold) { + const auto keypointScale = keypointScalePtr[part%numberScales] + * keypointScalePtr[part%numberScales] + * keypointScalePtr[part%numberScales]; + const auto radiusScaled = radiusSquared * keypointScale; const auto dist2 = (x - localX) * (x - localX) + (y - localY) * (y - localY); // Googly eyes - if (googlyEye1 == i || googlyEye2 == i) + if (googlyEye1 == part || googlyEye2 == part) { - const auto ratio = 2.5f * radius; - const auto minr2 = sharedScaleF[person] * sharedScaleF[person] * (ratio - 2) * (ratio - 2); - const auto maxr2 = sharedScaleF[person] * sharedScaleF[person] * ratio * ratio; + const auto eyeRatio = 2.5f * sqrt(radiusScaled); + const auto minr2 = sharedScaleF[person] + * sharedScaleF[person] * (eyeRatio - 2) * (eyeRatio - 2); + const auto maxr2 = sharedScaleF[person] * sharedScaleF[person] * eyeRatio * eyeRatio; if (dist2 <= maxr2) { float colorToAdd [3] = {0., 0., 0.}; @@ -132,8 +150,9 @@ namespace op color = {255.f}; if (dist2 <= minr2*0.6f) { - const auto dist3 = (x-4 - localX) * (x-4 - localX) + (y - localY+4) * (y - localY+4); - if (dist3 > 3.75f*3.75f) + const auto dist3 = (x-4 - localX) + * (x-4 - localX) + (y - localY+4) * (y - localY+4); + if (dist3 > 14.0625f) // 3.75f^2 for (auto& color : colorToAdd) color = {0.f}; } @@ -145,9 +164,9 @@ namespace op else { const auto minr2 = 0.f; - const auto maxr2 = sharedScaleF[person] * sharedScaleF[person] * radius * radius; + const auto maxr2 = sharedScaleF[person] * sharedScaleF[person] * radiusScaled; if (minr2 <= dist2 && dist2 <= maxr2) - addColorWeighted(r, g, b, &rgbColorsPtr[(i%numberColors)*3], alphaColorToAdd); + addColorWeighted(r, g, b, &rgbColorsPtr[(part%numberColors)*3], alphaColorToAdd); } } } diff --git a/src/openpose/face/renderFace.cpp b/src/openpose/face/renderFace.cpp index a0a25e53c907314b883fb942fc5871c9564c52a1..239f8dc7f92a3b0c0a4946341e93bd9550e5df60 100644 --- a/src/openpose/face/renderFace.cpp +++ b/src/openpose/face/renderFace.cpp @@ -5,7 +5,8 @@ namespace op { - void renderFaceKeypointsCpu(Array& frameArray, const Array& faceKeypoints, const float renderThreshold) + void renderFaceKeypointsCpu(Array& frameArray, const Array& faceKeypoints, + const float renderThreshold) { try { @@ -15,9 +16,11 @@ namespace op const auto thicknessCircleRatio = 1.f/75.f; const auto thicknessLineRatioWRTCircle = 0.334f; const auto& pairs = FACE_PAIRS_RENDER; + const auto& scales = FACE_SCALES_RENDER; // Render keypoints - renderKeypointsCpu(frameArray, faceKeypoints, pairs, FACE_COLORS_RENDER, thicknessCircleRatio, thicknessLineRatioWRTCircle, renderThreshold); + renderKeypointsCpu(frameArray, faceKeypoints, pairs, FACE_COLORS_RENDER, thicknessCircleRatio, + thicknessLineRatioWRTCircle, scales, renderThreshold); } } catch (const std::exception& e) diff --git a/src/openpose/face/renderFace.cu b/src/openpose/face/renderFace.cu index 3098e24f0f64311bc0181611a84cf2907228b503..0d555e12b16c001b2a97f7934510c7871318b367 100644 --- a/src/openpose/face/renderFace.cu +++ b/src/openpose/face/renderFace.cu @@ -6,7 +6,8 @@ namespace op { - __constant__ const unsigned int PART_PAIRS_GPU[] = FACE_PAIRS_RENDER_GPU; + __constant__ const unsigned int PART_PAIRS_GPU[] = {FACE_PAIRS_RENDER_GPU}; + __constant__ const float SCALES[] = {FACE_SCALES_RENDER_GPU}; __constant__ const float COLORS[] = {FACE_COLORS_RENDER_GPU}; __global__ void renderFaceParts(float* targetPtr, const int targetWidth, const int targetHeight, @@ -24,15 +25,15 @@ namespace op // Other parameters const auto numberPartPairs = sizeof(PART_PAIRS_GPU) / (2*sizeof(PART_PAIRS_GPU[0])); + const auto numberScales = sizeof(SCALES) / sizeof(SCALES[0]); const auto numberColors = sizeof(COLORS) / (3*sizeof(COLORS[0])); const auto radius = fastMin(targetWidth, targetHeight) / 120.f; - const auto stickwidth = fastMin(targetWidth, targetHeight) / 250.f; + const auto lineWidth = fastMin(targetWidth, targetHeight) / 250.f; // Render key points - renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, - globalIdx, x, y, targetWidth, targetHeight, facePtr, PART_PAIRS_GPU, numberPeople, - FACE_NUMBER_PARTS, numberPartPairs, COLORS, numberColors, - radius, stickwidth, threshold, alphaColorToAdd); + renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, globalIdx, x, y, targetWidth, targetHeight, + facePtr, PART_PAIRS_GPU, numberPeople, FACE_NUMBER_PARTS, numberPartPairs, COLORS, + numberColors, radius, lineWidth, SCALES, numberScales, threshold, alphaColorToAdd); } void renderFaceKeypointsGpu(float* framePtr, const Point& frameSize, const float* const facePtr, diff --git a/src/openpose/hand/renderHand.cpp b/src/openpose/hand/renderHand.cpp index 7155ca068ecb3bb8e3f96a43851472c10aad4c46..8c11230864d0c8a607dc59081b1f5c671e816dbb 100644 --- a/src/openpose/hand/renderHand.cpp +++ b/src/openpose/hand/renderHand.cpp @@ -14,13 +14,14 @@ namespace op const auto thicknessCircleRatio = 1.f/50.f; const auto thicknessLineRatioWRTCircle = 0.75f; const auto& pairs = HAND_PAIRS_RENDER; + const auto& scales = HAND_SCALES_RENDER; // Render keypoints if (!frameArray.empty()) renderKeypointsCpu(frameArray, handKeypoints[0], pairs, HAND_COLORS_RENDER, thicknessCircleRatio, - thicknessLineRatioWRTCircle, renderThreshold); + thicknessLineRatioWRTCircle, scales, renderThreshold); if (!frameArray.empty()) renderKeypointsCpu(frameArray, handKeypoints[1], pairs, HAND_COLORS_RENDER, thicknessCircleRatio, - thicknessLineRatioWRTCircle, renderThreshold); + thicknessLineRatioWRTCircle, scales, renderThreshold); } catch (const std::exception& e) { diff --git a/src/openpose/hand/renderHand.cu b/src/openpose/hand/renderHand.cu index 88892eb20969841cd3b8115680b82a036bf7c309..60f89f28ef02d71bad7546b64e406af2105cf4cd 100644 --- a/src/openpose/hand/renderHand.cu +++ b/src/openpose/hand/renderHand.cu @@ -6,7 +6,8 @@ namespace op { - __constant__ const unsigned int PART_PAIRS_GPU[] = HAND_PAIRS_RENDER_GPU; + __constant__ const unsigned int PART_PAIRS_GPU[] = {HAND_PAIRS_RENDER_GPU}; + __constant__ const float SCALES[] = {HAND_SCALES_RENDER_GPU}; __constant__ const float COLORS[] = {HAND_COLORS_RENDER_GPU}; __global__ void renderHandsParts(float* targetPtr, const int targetWidth, const int targetHeight, @@ -24,15 +25,15 @@ namespace op // Other parameters const auto numberPartPairs = sizeof(PART_PAIRS_GPU) / (2*sizeof(PART_PAIRS_GPU[0])); + const auto numberScales = sizeof(SCALES) / sizeof(SCALES[0]); const auto numberColors = sizeof(COLORS) / (3*sizeof(COLORS[0])); const auto radius = fastMin(targetWidth, targetHeight) / 100.f; - const auto stickwidth = fastMin(targetWidth, targetHeight) / 80.f; + const auto lineWidth = fastMin(targetWidth, targetHeight) / 80.f; // Render key points - renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, - globalIdx, x, y, targetWidth, targetHeight, handsPtr, PART_PAIRS_GPU, numberHands, - HAND_NUMBER_PARTS, numberPartPairs, COLORS, numberColors, - radius, stickwidth, threshold, alphaColorToAdd); + renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, globalIdx, x, y, targetWidth, targetHeight, + handsPtr, PART_PAIRS_GPU, numberHands, HAND_NUMBER_PARTS, numberPartPairs, COLORS, + numberColors, radius, lineWidth, SCALES, numberScales, threshold, alphaColorToAdd); } void renderHandKeypointsGpu(float* framePtr, const Point& frameSize, const float* const handsPtr, diff --git a/src/openpose/pose/poseGpuRenderer.cpp b/src/openpose/pose/poseGpuRenderer.cpp index 9a23a3dd326eaca4466cdd73b1c50674036570ac..ca6d2f1b679bb0b42f40531bf25dc611b7077835 100644 --- a/src/openpose/pose/poseGpuRenderer.cpp +++ b/src/openpose/pose/poseGpuRenderer.cpp @@ -158,7 +158,8 @@ namespace op UNUSED(poseKeypoints); UNUSED(scaleNetToOutput); error("OpenPose must be compiled with the `USE_CUDA` macro definitions in order to run this" - " functionality.", __LINE__, __FUNCTION__, __FILE__); + " functionality. You can alternatively use CPU rendering (flag `--render_pose 1`).", + __LINE__, __FUNCTION__, __FILE__); #endif // Return result return std::make_pair(elementRendered, elementRenderedName); diff --git a/src/openpose/pose/poseParametersRender.cpp b/src/openpose/pose/poseParametersRender.cpp index 53ada87b9fe8a9f7f2ac79217adf80f508929954..edf1536cac86c44b028b50317e8b7c40bbd99989 100644 --- a/src/openpose/pose/poseParametersRender.cpp +++ b/src/openpose/pose/poseParametersRender.cpp @@ -2,6 +2,15 @@ namespace op { + const std::array, (int)PoseModel::Size> POSE_SCALES{ + std::vector{POSE_COCO_SCALES_RENDER_GPU}, + std::vector{POSE_MPI_SCALES_RENDER_GPU}, + std::vector{POSE_MPI_SCALES_RENDER_GPU}, + std::vector{POSE_BODY_18_SCALES_RENDER_GPU}, + std::vector{POSE_BODY_19_SCALES_RENDER_GPU}, + std::vector{POSE_BODY_23_SCALES_RENDER_GPU}, + std::vector{POSE_BODY_59_SCALES_RENDER_GPU} + }; const std::array, (int)PoseModel::Size> POSE_COLORS{ std::vector{POSE_COCO_COLORS_RENDER_GPU}, std::vector{POSE_MPI_COLORS_RENDER_GPU}, @@ -22,6 +31,19 @@ namespace op }; // Rendering functions + const std::vector& getPoseScales(const PoseModel poseModel) + { + try + { + return POSE_SCALES.at((int)poseModel); + } + catch (const std::exception& e) + { + error(e.what(), __LINE__, __FUNCTION__, __FILE__); + return POSE_SCALES[(int)poseModel]; + } + } + const std::vector& getPoseColors(const PoseModel poseModel) { try diff --git a/src/openpose/pose/renderPose.cpp b/src/openpose/pose/renderPose.cpp index f92f7237e056a2ddec33040c338579967927ce03..2fd2097342b71c35741dd785b03b3976ddf1a77c 100644 --- a/src/openpose/pose/renderPose.cpp +++ b/src/openpose/pose/renderPose.cpp @@ -23,10 +23,11 @@ namespace op const auto thicknessCircleRatio = 1.f/75.f; const auto thicknessLineRatioWRTCircle = 0.75f; const auto& pairs = getPoseBodyPartPairsRender(poseModel); + const auto& poseScales = getPoseScales(poseModel); // Render keypoints renderKeypointsCpu(frameArray, poseKeypoints, pairs, getPoseColors(poseModel), thicknessCircleRatio, - thicknessLineRatioWRTCircle, renderThreshold); + thicknessLineRatioWRTCircle, poseScales, renderThreshold); } } catch (const std::exception& e) diff --git a/src/openpose/pose/renderPose.cu b/src/openpose/pose/renderPose.cu index 6306751dcc1d7873437007bc698c199edf853ac0..b6ddfb15bc706960cf261ebd78fcbbf4808c86de 100644 --- a/src/openpose/pose/renderPose.cu +++ b/src/openpose/pose/renderPose.cu @@ -8,12 +8,21 @@ namespace op { // PI digits: http://www.piday.org/million/ __constant__ const float PI = 3.14159265358979323846264338327950288419716939937510582097494459230781640628620899862803482534211706798214808651328230664709384460955058223172535940812848111745f; - __constant__ const unsigned int COCO_PAIRS_GPU[] = POSE_COCO_PAIRS_RENDER_GPU; - __constant__ const unsigned int BODY_18_PAIRS_GPU[] = POSE_BODY_18_PAIRS_RENDER_GPU; - __constant__ const unsigned int BODY_19_PAIRS_GPU[] = POSE_BODY_19_PAIRS_RENDER_GPU; - __constant__ const unsigned int BODY_23_PAIRS_GPU[] = POSE_BODY_23_PAIRS_RENDER_GPU; - __constant__ const unsigned int BODY_59_PAIRS_GPU[] = POSE_BODY_59_PAIRS_RENDER_GPU; - __constant__ const unsigned int MPI_PAIRS_GPU[] = POSE_MPI_PAIRS_RENDER_GPU; + // Keypoint pairs + __constant__ const unsigned int COCO_PAIRS_GPU[] = {POSE_COCO_PAIRS_RENDER_GPU}; + __constant__ const unsigned int BODY_18_PAIRS_GPU[] = {POSE_BODY_18_PAIRS_RENDER_GPU}; + __constant__ const unsigned int BODY_19_PAIRS_GPU[] = {POSE_BODY_19_PAIRS_RENDER_GPU}; + __constant__ const unsigned int BODY_23_PAIRS_GPU[] = {POSE_BODY_23_PAIRS_RENDER_GPU}; + __constant__ const unsigned int BODY_59_PAIRS_GPU[] = {POSE_BODY_59_PAIRS_RENDER_GPU}; + __constant__ const unsigned int MPI_PAIRS_GPU[] = {POSE_MPI_PAIRS_RENDER_GPU}; + // Keypoint scales + __constant__ const float COCO_SCALES[] = {POSE_COCO_SCALES_RENDER_GPU}; + __constant__ const float BODY_18_SCALES[] = {POSE_BODY_18_SCALES_RENDER_GPU}; + __constant__ const float BODY_19_SCALES[] = {POSE_BODY_19_SCALES_RENDER_GPU}; + __constant__ const float BODY_23_SCALES[] = {POSE_BODY_23_SCALES_RENDER_GPU}; + __constant__ const float BODY_59_SCALES[] = {POSE_BODY_59_SCALES_RENDER_GPU}; + __constant__ const float MPI_SCALES[] = {POSE_MPI_SCALES_RENDER_GPU}; + // RGB colors __constant__ const float COCO_COLORS[] = {POSE_COCO_COLORS_RENDER_GPU}; __constant__ const float BODY_18_COLORS[] = {POSE_BODY_18_COLORS_RENDER_GPU}; __constant__ const float BODY_19_COLORS[] = {POSE_BODY_19_COLORS_RENDER_GPU}; @@ -115,15 +124,16 @@ namespace op // Other parameters const auto numberPartPairs = sizeof(COCO_PAIRS_GPU) / (2*sizeof(COCO_PAIRS_GPU[0])); + const auto numberScales = sizeof(COCO_SCALES) / sizeof(COCO_SCALES[0]); const auto numberColors = sizeof(COCO_COLORS) / (3*sizeof(COCO_COLORS[0])); const auto radius = fastMin(targetWidth, targetHeight) / 100.f; - const auto stickwidth = fastMin(targetWidth, targetHeight) / 120.f; + const auto lineWidth = fastMin(targetWidth, targetHeight) / 120.f; // Render key points renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, globalIdx, x, y, targetWidth, targetHeight, posePtr, COCO_PAIRS_GPU, numberPeople, 18, numberPartPairs, COCO_COLORS, - numberColors, radius, stickwidth, threshold, alphaColorToAdd, blendOriginalFrame, - (googlyEyes ? 14 : -1), (googlyEyes ? 15 : -1)); + numberColors, radius, lineWidth, COCO_SCALES, numberScales, threshold, alphaColorToAdd, + blendOriginalFrame, (googlyEyes ? 14 : -1), (googlyEyes ? 15 : -1)); } __global__ void renderPoseBody18(float* targetPtr, const int targetWidth, const int targetHeight, @@ -141,14 +151,15 @@ namespace op // Other parameters const auto numberPartPairs = sizeof(BODY_18_PAIRS_GPU) / (2*sizeof(BODY_18_PAIRS_GPU[0])); + const auto numberScales = sizeof(BODY_18_SCALES) / sizeof(BODY_18_SCALES[0]); const auto numberColors = sizeof(BODY_18_COLORS) / (3*sizeof(BODY_18_COLORS[0])); const auto radius = fastMin(targetWidth, targetHeight) / 100.f; - const auto stickwidth = fastMin(targetWidth, targetHeight) / 120.f; + const auto lineWidth = fastMin(targetWidth, targetHeight) / 120.f; // Render key points renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, globalIdx, x, y, targetWidth, targetHeight, - posePtr, BODY_18_PAIRS_GPU, numberPeople, 18, numberPartPairs, - BODY_18_COLORS, numberColors, radius, stickwidth, threshold, alphaColorToAdd, + posePtr, BODY_18_PAIRS_GPU, numberPeople, 18, numberPartPairs, BODY_18_COLORS, numberColors, + radius, lineWidth, BODY_18_SCALES, numberScales, threshold, alphaColorToAdd, blendOriginalFrame, (googlyEyes ? 14 : -1), (googlyEyes ? 15 : -1)); } @@ -167,14 +178,15 @@ namespace op // Other parameters const auto numberPartPairs = sizeof(BODY_19_PAIRS_GPU) / (2*sizeof(BODY_19_PAIRS_GPU[0])); + const auto numberScales = sizeof(BODY_19_SCALES) / sizeof(BODY_19_SCALES[0]); const auto numberColors = sizeof(BODY_19_COLORS) / (3*sizeof(BODY_19_COLORS[0])); const auto radius = fastMin(targetWidth, targetHeight) / 100.f; - const auto stickwidth = fastMin(targetWidth, targetHeight) / 120.f; + const auto lineWidth = fastMin(targetWidth, targetHeight) / 120.f; // Render key points renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, globalIdx, x, y, targetWidth, targetHeight, - posePtr, BODY_19_PAIRS_GPU, numberPeople, 19, numberPartPairs, - BODY_19_COLORS, numberColors, radius, stickwidth, threshold, alphaColorToAdd, + posePtr, BODY_19_PAIRS_GPU, numberPeople, 19, numberPartPairs, BODY_19_COLORS, numberColors, + radius, lineWidth, BODY_19_SCALES, numberScales, threshold, alphaColorToAdd, blendOriginalFrame, (googlyEyes ? 15 : -1), (googlyEyes ? 16 : -1)); } @@ -193,14 +205,15 @@ namespace op // Other parameters const auto numberPartPairs = sizeof(BODY_23_PAIRS_GPU) / (2*sizeof(BODY_23_PAIRS_GPU[0])); + const auto numberScales = sizeof(BODY_23_SCALES) / sizeof(BODY_23_SCALES[0]); const auto numberColors = sizeof(BODY_23_COLORS) / (3*sizeof(BODY_23_COLORS[0])); const auto radius = fastMin(targetWidth, targetHeight) / 100.f; - const auto stickwidth = fastMin(targetWidth, targetHeight) / 120.f; + const auto lineWidth = fastMin(targetWidth, targetHeight) / 120.f; // Render key points renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, globalIdx, x, y, targetWidth, targetHeight, - posePtr, BODY_23_PAIRS_GPU, numberPeople, 23, numberPartPairs, - BODY_23_COLORS, numberColors, radius, stickwidth, threshold, alphaColorToAdd, + posePtr, BODY_23_PAIRS_GPU, numberPeople, 23, numberPartPairs, BODY_23_COLORS, numberColors, + radius, lineWidth, BODY_23_SCALES, numberScales, threshold, alphaColorToAdd, blendOriginalFrame, (googlyEyes ? 19 : -1), (googlyEyes ? 21 : -1)); } @@ -219,14 +232,15 @@ namespace op // Other parameters const auto numberPartPairs = sizeof(BODY_59_PAIRS_GPU) / (2*sizeof(BODY_59_PAIRS_GPU[0])); + const auto numberScales = sizeof(BODY_59_SCALES) / sizeof(BODY_59_SCALES[0]); const auto numberColors = sizeof(BODY_59_COLORS) / (3*sizeof(BODY_59_COLORS[0])); const auto radius = fastMin(targetWidth, targetHeight) / 100.f; - const auto stickwidth = fastMin(targetWidth, targetHeight) / 120.f; + const auto lineWidth = fastMin(targetWidth, targetHeight) / 120.f; // Render key points renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, globalIdx, x, y, targetWidth, targetHeight, - posePtr, BODY_59_PAIRS_GPU, numberPeople, 59, numberPartPairs, - BODY_59_COLORS, numberColors, radius, stickwidth, threshold, alphaColorToAdd, + posePtr, BODY_59_PAIRS_GPU, numberPeople, 59, numberPartPairs, BODY_59_COLORS, numberColors, + radius, lineWidth, BODY_59_SCALES, numberScales, threshold, alphaColorToAdd, blendOriginalFrame, (googlyEyes ? 15 : -1), (googlyEyes ? 16 : -1)); } @@ -245,15 +259,15 @@ namespace op // Other parameters const auto numberPartPairs = sizeof(MPI_PAIRS_GPU) / (2*sizeof(MPI_PAIRS_GPU[0])); + const auto numberScales = sizeof(MPI_SCALES) / sizeof(MPI_SCALES[0]); const auto numberColors = sizeof(MPI_COLORS) / (3*sizeof(MPI_COLORS[0])); const auto radius = fastMin(targetWidth, targetHeight) / 100.f; - const auto stickwidth = fastMin(targetWidth, targetHeight) / 120.f; + const auto lineWidth = fastMin(targetWidth, targetHeight) / 120.f; // Render key points - renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, - globalIdx, x, y, targetWidth, targetHeight, posePtr, MPI_PAIRS_GPU, numberPeople, - 15, numberPartPairs, MPI_COLORS, numberColors, - radius, stickwidth, threshold, alphaColorToAdd, blendOriginalFrame); + renderKeypoints(targetPtr, sharedMaxs, sharedMins, sharedScaleF, globalIdx, x, y, targetWidth, targetHeight, + posePtr, MPI_PAIRS_GPU, numberPeople, 15, numberPartPairs, MPI_COLORS, numberColors, + radius, lineWidth, BODY_18_SCALES, numberScales, threshold, alphaColorToAdd, blendOriginalFrame); } __global__ void renderBodyPartHeatMaps(float* targetPtr, const int targetWidth, const int targetHeight, diff --git a/src/openpose/utilities/keypoint.cpp b/src/openpose/utilities/keypoint.cpp index 39f29dfa1f125e9f6f8de7837cc73ef51f7d2371..d43c92e7b1928bc21abf84fcda41fc4d831d20aa 100644 --- a/src/openpose/utilities/keypoint.cpp +++ b/src/openpose/utilities/keypoint.cpp @@ -133,7 +133,7 @@ namespace op void renderKeypointsCpu(Array& frameArray, const Array& keypoints, const std::vector& pairs, const std::vector colors, const float thicknessCircleRatio, const float thicknessLineRatioWRTCircle, - const float threshold) + const std::vector& poseScales, const float threshold) { try { @@ -150,14 +150,16 @@ namespace op const auto width = frame.size[2]; const auto height = frame.size[1]; const auto area = width * height; + const auto channelOffset = area * sizeof(float) / sizeof(uchar); cv::Mat frameB(height, width, CV_32FC1, &frame.data[0]); - cv::Mat frameG(height, width, CV_32FC1, &frame.data[area * sizeof(float) / sizeof(uchar)]); - cv::Mat frameR(height, width, CV_32FC1, &frame.data[2 * area * sizeof(float) / sizeof(uchar)]); + cv::Mat frameG(height, width, CV_32FC1, &frame.data[channelOffset]); + cv::Mat frameR(height, width, CV_32FC1, &frame.data[2 * channelOffset]); // Parameters const auto lineType = 8; const auto shift = 0; const auto numberColors = colors.size(); + const auto numberScales = poseScales.size(); const auto thresholdRectangle = 0.1f; const auto numberKeypoints = keypoints.getSize(1); @@ -173,7 +175,7 @@ namespace op const auto thicknessRatio = fastMax(intRound(std::sqrt(area) * thicknessCircleRatio * ratioAreas), 2); // Negative thickness in cv::circle means that a filled circle is to be drawn. - const auto thicknessCircle = (ratioAreas > 0.05 ? thicknessRatio : -1); + const auto thicknessCircle = (ratioAreas > 0.05f ? thicknessRatio : -1); const auto thicknessLine = intRound(thicknessRatio * thicknessLineRatioWRTCircle); const auto radius = thicknessRatio / 2; @@ -184,15 +186,17 @@ namespace op const auto index2 = (person * numberKeypoints + pairs[pair+1]) * keypoints.getSize(2); if (keypoints[index1+2] > threshold && keypoints[index2+2] > threshold) { + const auto thicknessLineScaled = thicknessLine + * poseScales[pairs[pair+1] % numberScales]; const auto colorIndex = pairs[pair+1]*3; // Before: colorIndex = pair/2*3; const cv::Scalar color{colors[colorIndex % numberColors], colors[(colorIndex+1) % numberColors], colors[(colorIndex+2) % numberColors]}; const cv::Point keypoint1{intRound(keypoints[index1]), intRound(keypoints[index1+1])}; const cv::Point keypoint2{intRound(keypoints[index2]), intRound(keypoints[index2+1])}; - cv::line(frameR, keypoint1, keypoint2, color[0], thicknessLine, lineType, shift); - cv::line(frameG, keypoint1, keypoint2, color[1], thicknessLine, lineType, shift); - cv::line(frameB, keypoint1, keypoint2, color[2], thicknessLine, lineType, shift); + cv::line(frameR, keypoint1, keypoint2, color[0], thicknessLineScaled, lineType, shift); + cv::line(frameG, keypoint1, keypoint2, color[1], thicknessLineScaled, lineType, shift); + cv::line(frameB, keypoint1, keypoint2, color[2], thicknessLineScaled, lineType, shift); } } @@ -202,15 +206,20 @@ namespace op const auto faceIndex = (person * numberKeypoints + part) * keypoints.getSize(2); if (keypoints[faceIndex+2] > threshold) { + const auto radiusScaled = radius * poseScales[part % numberScales]; + const auto thicknessCircleScaled = thicknessCircle * poseScales[part % numberScales]; const auto colorIndex = part*3; const cv::Scalar color{colors[colorIndex % numberColors], colors[(colorIndex+1) % numberColors], colors[(colorIndex+2) % numberColors]}; const cv::Point center{intRound(keypoints[faceIndex]), intRound(keypoints[faceIndex+1])}; - cv::circle(frameR, center, radius, color[0], thicknessCircle, lineType, shift); - cv::circle(frameG, center, radius, color[1], thicknessCircle, lineType, shift); - cv::circle(frameB, center, radius, color[2], thicknessCircle, lineType, shift); + cv::circle(frameR, center, radiusScaled, color[0], thicknessCircleScaled, lineType, + shift); + cv::circle(frameG, center, radiusScaled, color[1], thicknessCircleScaled, lineType, + shift); + cv::circle(frameB, center, radiusScaled, color[2], thicknessCircleScaled, lineType, + shift); } } }