提交 7c1e8e40 编写于 作者: G Gines Hidalgo

Sanity checks for flir camera with 3-d #1254

上级 13ee82fc
......@@ -210,7 +210,7 @@ Each flag is divided into flag name, default value, and description.
- DEFINE_bool(disable_blending, false, "If enabled, it will render the results (keypoint skeletons or heatmaps) on a black background, instead of being rendered into the original image. Related: `part_to_show`, `alpha_pose`, and `alpha_pose`.");
11. OpenPose Rendering Pose
- DEFINE_double(render_threshold, 0.05, "Only estimated keypoints whose score confidences are higher than this threshold will be rendered. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong detections).");
- DEFINE_double(render_threshold, 0.05, "Only estimated keypoints whose score confidences are higher than this threshold will be rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in the saved results. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong detections).");
- DEFINE_int32(render_pose, -1, "Set to 0 for no rendering, 1 for CPU rendering (slightly faster), and 2 for GPU rendering (slower but greater functionality, e.g., `alpha_X` flags). If -1, it will pick CPU if CPU_ONLY is enabled, or GPU if CUDA is enabled. If rendering is enabled, it will render both `outputData` and `cvOutputData` with the original image and desired body part to be shown (i.e., keypoints, heat maps or PAFs).");
- DEFINE_double(alpha_pose, 0.6, "Blending factor (range 0-1) for the body part rendering. 1 will show it completely, 0 will hide it. Only valid for GPU rendering.");
- DEFINE_double(alpha_heatmap, 0.7, "Blending factor (range 0-1) between heatmap and original frame. 1 will only show the heatmap, 0 will only show the frame. Only valid for GPU rendering.");
......
......@@ -18,6 +18,7 @@ OpenPose - Frequently Asked Question (FAQ)
12. [CMake-GUI Error While Getting Default Caffe](#cmake-gui-error-while-getting-default-caffe)
13. [Libgomp Out of Memory Error](#libgomp-out-of-memory-error)
14. [Runtime Error with Turing GPU (Tesla T4) or Volta GPU][#runtime-error-with-turing-gpu-teslat4-or-volta-gpu)
15. [Obscure CMake Error about Caffe or Pybind](#obscure-cmake-error-about-caffe-or-pybind).
2. [Speed Performance Issues](#speed-performance-issues)
1. [Speed Up, Memory Reduction, and Benchmark](#speed-up-memory-reduction-and-benchmark)
2. [How to Measure the Latency Time?](#how-to-measure-the-latency-time)
......@@ -171,6 +172,14 @@ git submodle update
#### Obscure CMake Error about Caffe or Pybind
**Q:** There appear some weird and obscure errors on CMake about Caffe and/or Pybind.
**A**: Check [doc/prerequisites.md#general-tips](./prerequisites.md#general-tips) to run the `git submodule` command.
### Speed Performance Issues
#### Speed Up, Memory Reduction, and Benchmark
......
......@@ -330,9 +330,10 @@ Also as a side note, if the default installation fails (i.e., the one explained
3. `Caffe_LIBS` set to `/usr/local/lib/libcaffe.dylib`.
4. Run `Configure` and `Generate` from CMake GUI.
You may also have to apply the following patch if you have the latest OSX 10.14. It can be done as follows:
`cd 3rdparty/caffe; git apply ../../scripts/osx/mac_opencl_patch.txt`
In addition, if you face an OpenCV error during compiling time similar to `fatal error: 'opencv2/highgui/highgui.hpp' file not found`, please apply the following patch (this error has been reported in the latest OSX 10.14):
```
cd 3rdparty/caffe; git apply ../../scripts/osx/mac_opencl_patch.txt
```
#### 3D Reconstruction Module
......
......@@ -100,7 +100,7 @@ The visual GUI should show 3 screens.
It should be similar to the following image.
<p align="center">
<img src="media/openpose3d.png">
<img src="../media/openpose3d.gif">
</p>
......
......@@ -184,9 +184,10 @@ DEFINE_bool(disable_blending, false, "If enabled, it will ren
" `alpha_pose`, and `alpha_pose`.");
// OpenPose Rendering Pose
DEFINE_double(render_threshold, 0.05, "Only estimated keypoints whose score confidences are higher than this threshold will be"
" rendered. Generally, a high threshold (> 0.5) will only render very clear body parts;"
" while small thresholds (~0.1) will also output guessed and occluded keypoints, but also"
" more false positives (i.e., wrong detections).");
" rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in"
" the saved results. Generally, a high threshold (> 0.5) will only render very clear body"
" parts; while small thresholds (~0.1) will also output guessed and occluded keypoints,"
" but also more false positives (i.e., wrong detections).");
DEFINE_int32(render_pose, -1, "Set to 0 for no rendering, 1 for CPU rendering (slightly faster), and 2 for GPU rendering"
" (slower but greater functionality, e.g., `alpha_X` flags). If -1, it will pick CPU if"
" CPU_ONLY is enabled, or GPU if CUDA is enabled. If rendering is enabled, it will render"
......
......@@ -23,6 +23,10 @@ namespace op
std::vector<cv::Mat> getRawFrames();
/**
* Note: The camera parameters are only read if undistortImage is true. This should be changed to add a
* new bool flag in the constructor, e.g., readCameraParameters
*/
std::vector<cv::Mat> getCameraMatrices() const;
std::vector<cv::Mat> getCameraExtrinsics() const;
......
......@@ -70,8 +70,13 @@ namespace op
float getKeypointsRoi(const Array<T>& keypoints, const int personA, const int personB, const T threshold);
template <typename T>
float getKeypointsRoi(const Array<T>& keypointsA, const int personA, const Array<T>& keypointsB, const int personB,
const T threshold);
float getKeypointsRoi(
const Array<T>& keypointsA, const int personA, const Array<T>& keypointsB, const int personB,
const T threshold);
template <typename T>
float getKeypointsRoi(
const Rectangle<T>& rectangleA, const Rectangle<T>& rectangleB);
}
#endif // OPENPOSE_UTILITIES_KEYPOINT_HPP
......@@ -54,7 +54,8 @@ namespace op
/**
* Rendering threshold. Only estimated keypoints whose score confidences are higher than this value will be
* rendered. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
* rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in the saved results.
* Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
* (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong
* detections).
*/
......
......@@ -73,7 +73,8 @@ namespace op
/**
* Rendering threshold. Only estimated keypoints whose score confidences are higher than this value will be
* rendered. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
* rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in the saved results.
* Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
* (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong
* detections).
*/
......
......@@ -143,7 +143,8 @@ namespace op
/**
* Rendering threshold. Only estimated keypoints whose score confidences are higher than this value will be
* rendered. Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
* rendered. Note: Rendered refers only to visual display in the OpenPose basic GUI, not in the saved results.
* Generally, a high threshold (> 0.5) will only render very clear body parts; while small thresholds
* (~0.1) will also output guessed and occluded keypoints, but also more false positives (i.e., wrong
* detections).
*/
......
......@@ -505,8 +505,8 @@ namespace op
{
try
{
return reconstructArray(std::vector<std::vector<Array<float>>>{keypointsVector},
cameraMatrices, imageSizes).at(0);
return reconstructArray(
std::vector<std::vector<Array<float>>>{keypointsVector}, cameraMatrices, imageSizes).at(0);
}
catch (const std::exception& e)
{
......@@ -515,6 +515,9 @@ namespace op
}
}
const std::string sFlirErrorMessage{
" If you are simultaneously using FLIR cameras (`--flir_camera`) and the 3-D reconstruction module"
" (`--3d), you should also enable `--frame_undistort` so their camera parameters are read."};
std::vector<Array<float>> PoseTriangulation::reconstructArray(
const std::vector<std::vector<Array<float>>>& keypointsVectors,
const std::vector<cv::Mat>& cameraMatrices,
......@@ -522,6 +525,20 @@ namespace op
{
try
{
// Sanity checks
if (cameraMatrices.size() < 2)
error("3-D reconstruction (`--3d`) requires at least 2 camera views, only found "
+ std::to_string(cameraMatrices.size()) + "camera parameter matrices." + sFlirErrorMessage,
__LINE__, __FUNCTION__, __FILE__);
for (const auto& cameraMatrix : cameraMatrices)
if (cameraMatrix.empty())
error("Camera matrix was found empty during 3-D reconstruction (`--3d`)." + sFlirErrorMessage,
__LINE__, __FUNCTION__, __FILE__);
if (cameraMatrices.size() != imageSizes.size())
error("The camera parameters and number of images must be the same ("
+ std::to_string(cameraMatrices.size()) + " vs. " + std::to_string(imageSizes.size()) + ").",
__LINE__, __FUNCTION__, __FILE__);
// Run 3-D reconstruction
bool keypointsReconstructed = false;
std::vector<Array<float>> keypoints3Ds(keypointsVectors.size());
// std::vector<std::thread> threads(keypointsVectors.size()-1);
......
......@@ -27,7 +27,7 @@ namespace op
__LINE__, __FUNCTION__, __FILE__);
// Warnings
if (!mHeatMapTypes.empty())
log("Note only keypoint heatmaps are available with face heatmaps (no background nor PAFs).",
log("Note that only the keypoint heatmaps are available with face heatmaps (no background nor PAFs).",
Priority::High);
}
catch (const std::exception& e)
......
......@@ -29,7 +29,7 @@ namespace op
__LINE__, __FUNCTION__, __FILE__);
// Warnings
if (!mHeatMapTypes.empty())
log("Note only keypoint heatmaps are available with hand heatmaps (no background nor PAFs).",
log("Note that only the keypoint heatmaps are available with hand heatmaps (no background nor PAFs).",
Priority::High);
}
catch (const std::exception& e)
......
......@@ -83,12 +83,12 @@ namespace op
void getRoiDiameterAndBounds(
Rectangle<int>& roi, int& diameter, int& partFirstNon0, int& partLastNon0,
const std::vector<int>& personVector, const T* const peaksPtr,
const int partInit, const int partEnd)
const int partInit, const int partEnd, const float margin)
{
try
{
// Find ROI, partFirstNon0, and partLastNon0
roi = Rectangle<int>{0,0,0,0};
roi = Rectangle<int>{std::numeric_limits<int>::max(),std::numeric_limits<int>::max(),0,0};
partFirstNon0 = -1;
partLastNon0 = -1;
for (auto part = partInit ; part < partEnd ; part++)
......@@ -96,7 +96,7 @@ namespace op
const auto x = peaksPtr[personVector[part]-2];
const auto y = peaksPtr[personVector[part]-1];
const auto score = peaksPtr[personVector[part]];
if (score > 0)
if (personVector[part] > 0 && score > 0)
{
// ROI
if (roi.x > x)
......@@ -105,7 +105,7 @@ namespace op
roi.y = y;
if (roi.width < x)
roi.width = x;
if (roi.height > y)
if (roi.height < y)
roi.height = y;
// First keypoint?
if (partFirstNon0 < 0)
......@@ -114,11 +114,24 @@ namespace op
partLastNon0 = part;
}
}
// From [p1, p2] to [p1, width, height]
roi.width -= roi.x;
roi.height -= roi.y;
// diameter
diameter = fastMax(roi.width, roi.height);
if (partLastNon0 > -1)
{
// Add margin
const auto marginX = roi.width * margin;
const auto marginY = roi.height * margin;
roi.x -= marginX;
roi.y -= marginY;
roi.width += 2*marginX;
roi.height += 2*marginY;
// partFirstNon0+1 for for loops
partLastNon0++;
// From [p1, p2] to [p1, width, height]
// +1 to account for 1-line keypoints
roi.width += 1 - roi.x;
roi.height += 1 - roi.y;
// diameter
diameter = fastMax(roi.width, roi.height);
}
}
catch (const std::exception& e)
{
......@@ -712,7 +725,8 @@ namespace op
faceValidSubsetIndexes.reserve(peopleVector.size());
// Face invalid sets
std::vector<int> faceInvalidSubsetIndexes;
faceInvalidSubsetIndexes.reserve(peopleVector.size());
if (numberBodyParts >= 135)
faceInvalidSubsetIndexes.reserve(peopleVector.size());
// For each person candidate
for (auto person = 0u ; person < peopleVector.size() ; person++)
{
......@@ -768,6 +782,77 @@ namespace op
error("Bad personCounter (" + std::to_string(personCounter) + "). Bug in this"
" function if this happens.", __LINE__, __FUNCTION__, __FILE__);
}
// Random standalone facial keypoints --> Merge into a more complete face
if (numberPeople > 0)
{
// Check invalid faces
for (const auto& personInvalid : faceInvalidSubsetIndexes)
{
// Get ROI of current face
Rectangle<int> roiInvalid;
int diameterInvalid = -1;
int partFirstNon0Invalid = -1;
int partLastNon0Invalid = -1;
getRoiDiameterAndBounds(
roiInvalid, diameterInvalid, partFirstNon0Invalid, partLastNon0Invalid,
peopleVector[personInvalid].first, peaksPtr, 65, 135, 0.2f);
// Check all valid faces to find best candidate
float keypointsRoiBest = 0.f;
auto keypointsRoiBestIndex = -1;
for (auto personId = 0u ; personId < faceValidSubsetIndexes.size() ; personId++)
{
const auto& personValid = faceValidSubsetIndexes[personId];
// Get ROI of current face
Rectangle<int> roiValid;
int diameterValid = -1;
int partFirstNon0Valid = -1;
int partLastNon0Valid = -1;
getRoiDiameterAndBounds(
roiValid, diameterValid, partFirstNon0Valid, partLastNon0Valid, peopleVector[personValid].first,
peaksPtr, 65, 135, 0.1f);
// Get ROI between both faces
const auto keypointsRoi = getKeypointsRoi(roiValid, roiInvalid);
// Update best so far
if (keypointsRoiBest < keypointsRoi)
{
keypointsRoiBest = keypointsRoi;
keypointsRoiBestIndex = personId;
}
}
// If invalid and best valid candidate overlap enough --> Merge them
if (keypointsRoiBest > 0.3f || (keypointsRoiBest > 0.01f && faceValidSubsetIndexes.size() < 3))
{
const auto& personValid = faceValidSubsetIndexes[keypointsRoiBestIndex];
// If it is from that face --> Combine invalid face keypoints into valid face
for (auto part = partFirstNon0Invalid ; part < partLastNon0Invalid ; part++)
{
auto& personVectorValid = peopleVector[personValid].first;
const auto scoreValid = peaksPtr[personVectorValid[part]];
const auto& personVectorInvalid = peopleVector[personInvalid].first;
const auto scoreInvalid = peaksPtr[personVectorInvalid[part]];
// If the new one has a keypoint...
if (personVectorInvalid[part] != 0)
{
// ... and the original face does not have it, then add it to it
if (personVectorValid[part] == 0)
{
if (personVectorInvalid[part] != 0)
{
personVectorValid[part] = personVectorInvalid[part];
peopleVector[personValid].second += scoreInvalid;
}
}
// ... and its score is higher than the original one, then replace it
else if (scoreValid < scoreInvalid)
{
personVectorValid[part] = personVectorInvalid[part];
peopleVector[personValid].second += scoreInvalid - scoreValid;
}
}
}
}
}
}
// If no people found --> Repeat with maximizePositives = true
// Result: Increased COCO mAP because we catch more foot-only images
if (numberPeople == 0 && !maximizePositives)
......
......@@ -716,7 +716,11 @@ namespace op
return 0.01f;
// Non-MPI models
else
return (maximizePositives ? 0.01f : 0.05f);
// return (maximizePositives ? 0.01f : 0.5f); // 0.485 but much less false positive connections
// return (maximizePositives ? 0.01f : 0.1f); // 0.518
// return (maximizePositives ? 0.01f : 0.075f); // 0.521
return (maximizePositives ? 0.01f : 0.05f); // 0.523
// return (maximizePositives ? 0.01f : 0.01f); // 0.527 but huge amount of false positives joints
}
catch (const std::exception& e)
{
......
......@@ -828,9 +828,9 @@ namespace op
}
}
void renderPosePAFGpu(float* framePtr, const PoseModel poseModel, const Point<int>& frameSize,
const float* const heatMapPtr, const Point<int>& heatMapSize, const float scaleToKeepRatio,
const int part, const float alphaBlending)
void renderPosePAFGpu(
float* framePtr, const PoseModel poseModel, const Point<int>& frameSize, const float* const heatMapPtr,
const Point<int>& heatMapSize, const float scaleToKeepRatio, const int part, const float alphaBlending)
{
try
{
......@@ -843,16 +843,16 @@ namespace op
}
}
void renderPosePAFsGpu(float* framePtr, const PoseModel poseModel, const Point<int>& frameSize,
const float* const heatMapPtr, const Point<int>& heatMapSize, const float scaleToKeepRatio,
const float alphaBlending)
void renderPosePAFsGpu(
float* framePtr, const PoseModel poseModel, const Point<int>& frameSize, const float* const heatMapPtr,
const Point<int>& heatMapSize, const float scaleToKeepRatio, const float alphaBlending)
{
try
{
const auto numberBodyPartPairs = (int)getPosePartPairs(poseModel).size()/2;
renderPosePAFGpuAux(
framePtr, poseModel, frameSize, heatMapPtr, heatMapSize, scaleToKeepRatio,
getPoseNumberBodyParts(poseModel) + (poseModel != PoseModel::BODY_25B ? 1 : 0),
getPoseNumberBodyParts(poseModel) + (addBkgChannel(poseModel) ? 1 : 0),
numberBodyPartPairs, alphaBlending);
}
catch (const std::exception& e)
......@@ -861,9 +861,9 @@ namespace op
}
}
void renderPoseDistanceGpu(float* framePtr, const Point<int>& frameSize, const float* const heatMapPtr,
const Point<int>& heatMapSize, const float scaleToKeepRatio, const unsigned int part,
const float alphaBlending)
void renderPoseDistanceGpu(
float* framePtr, const Point<int>& frameSize, const float* const heatMapPtr, const Point<int>& heatMapSize,
const float scaleToKeepRatio, const unsigned int part, const float alphaBlending)
{
try
{
......
......@@ -551,8 +551,9 @@ namespace op
const Array<double>& keypoints, const int personA, const int personB, const double threshold);
template <typename T>
float getKeypointsRoi(const Array<T>& keypointsA, const int personA, const Array<T>& keypointsB, const int personB,
const T threshold)
float getKeypointsRoi(
const Array<T>& keypointsA, const int personA, const Array<T>& keypointsB, const int personB,
const T threshold)
{
try
{
......@@ -566,13 +567,50 @@ namespace op
// Get ROI
const auto rectangleA = getKeypointsRectangle(keypointsA, personA, threshold);
const auto rectangleB = getKeypointsRectangle(keypointsB, personB, threshold);
return getKeypointsRoi(rectangleA, rectangleB);
}
catch (const std::exception& e)
{
error(e.what(), __LINE__, __FUNCTION__, __FILE__);
return 0.f;
}
}
template OP_API float getKeypointsRoi(
const Array<float>& keypointsA, const int personA, const Array<float>& keypointsB, const int personB,
const float threshold);
template OP_API float getKeypointsRoi(
const Array<double>& keypointsA, const int personA, const Array<double>& keypointsB, const int personB,
const double threshold);
template <typename T>
float getKeypointsRoi(const Rectangle<T>& rectangleA, const Rectangle<T>& rectangleB)
{
try
{
// Check if negative values, then normalize it
auto rectangleANorm = rectangleA;
auto rectangleBNorm = rectangleB;
// E.g., [-10,-10,W1,H1] and [-20,-20,W2,H2] should be equivalent to [10,10,W1,H1] and [0,0,W2,H2]
const auto biasX = std::min(std::min(T{0}, rectangleA.x), rectangleB.x);
if (biasX != 0)
{
rectangleANorm.x -= biasX;
rectangleBNorm.x -= biasX;
}
const auto biasY = std::min(std::min(T{0}, rectangleA.y), rectangleB.y);
if (biasY != 0)
{
rectangleANorm.y -= biasY;
rectangleBNorm.y -= biasY;
}
// Get ROI
const Point<T> pointAIntersection{
fastMax(rectangleA.x, rectangleB.x),
fastMax(rectangleA.y, rectangleB.y)
fastMax(rectangleANorm.x, rectangleBNorm.x),
fastMax(rectangleANorm.y, rectangleBNorm.y)
};
const Point<T> pointBIntersection{
fastMin(rectangleA.x+rectangleA.width, rectangleB.x+rectangleB.width),
fastMin(rectangleA.y+rectangleA.height, rectangleB.y+rectangleB.height)
fastMin(rectangleANorm.x+rectangleANorm.width, rectangleBNorm.x+rectangleBNorm.width),
fastMin(rectangleANorm.y+rectangleANorm.height, rectangleBNorm.y+rectangleBNorm.height)
};
// Make sure there is overlap
if (pointAIntersection.x < pointBIntersection.x && pointAIntersection.y < pointBIntersection.y)
......@@ -583,13 +621,14 @@ namespace op
pointBIntersection.x-pointAIntersection.x,
pointBIntersection.y-pointAIntersection.y
};
const auto areaA = rectangleA.area();
const auto areaB = rectangleB.area();
const auto areaA = rectangleANorm.area();
const auto areaB = rectangleBNorm.area();
const auto intersection = rectangleIntersection.area();
return float(intersection) / float(areaA + areaB - intersection);
}
// If non overlap --> Return 0
return 0.f;
else
return 0.f;
}
catch (const std::exception& e)
{
......@@ -598,9 +637,11 @@ namespace op
}
}
template OP_API float getKeypointsRoi(
const Array<float>& keypointsA, const int personA, const Array<float>& keypointsB, const int personB,
const float threshold);
const Rectangle<int>& rectangleA, const Rectangle<int>& rectangleB);
template OP_API float getKeypointsRoi(
const Array<double>& keypointsA, const int personA, const Array<double>& keypointsB, const int personB,
const double threshold);
const Rectangle<unsigned int>& rectangleA, const Rectangle<unsigned int>& rectangleB);
template OP_API float getKeypointsRoi(
const Rectangle<float>& rectangleA, const Rectangle<float>& rectangleB);
template OP_API float getKeypointsRoi(
const Rectangle<double>& rectangleA, const Rectangle<double>& rectangleB);
}
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册