提交 d4018ba9 编写于 作者: G gineshidalgo99

3D visualizer doesn't crash on exit

上级 4d3ba29c
OpenPose 3-D Reconstruction Module and Demo
=============================================
## Contents
1. [Introduction](#introduction)
2. [Features](#features)
3. [Required Hardware](#required-hardware)
4. [Camera Calibration](#camera-calibration)
5. [Camera Ordering](#camera-ordering)
6. [Installing the OpenPose 3-D Reconstruction Module](#installing-the-openpose-3-d-reconstruction-module)
7. [Known Bug](#known-bug)
## Introduction
This experimental module performs 3-D keypoint (body, face, and hand) reconstruction and rendering for 1 person. We will not keep updating it nor solving questions/issues about it at the moment. It requires the user to be familiar with computer vision and camera calibration, including extraction of intrinsic and extrinsic parameters.
......@@ -72,3 +84,10 @@ It should be similar to the following image.
<p align="center">
<img src="media/openpose3d.png">
</p>
## Known Bug
FreeGLUT is a quite light library. Due to that, there is a known bug in the 3D module:
1. The window must be closed with the <kbd>Esc</kbd> key. Clicking the close button will cause a core dumped or std::exception error in OpenPose. Reason: There is no way to control the behaviour of the exit button in a FreeGLUT program. Feel free to let us know or create a pull request if you find a workaround applicable to 3-D OpenPose. Another alternative is ussing `--disable_multi_thread` in OpenPose. This would avoid the issue but slow down the program, especially in multi-GPU systems.
......@@ -183,6 +183,7 @@ OpenPose Library - Release Notes
2. Main bugs fixed:
1. Slight speed up (~1%) for performing the non-maximum suppression stage only in the body part heatmaps channels, and not also in the PAF channels.
2. Fixed core-dumped in PoseRenderer with GUI when changed element to be rendered to something else than skeleton.
3. 3-D visualizer does not crash on exit anymore.
......
#ifndef OPENPOSE_EXPERIMENTAL_3D_RENDERER_HPP
#define OPENPOSE_EXPERIMENTAL_3D_RENDERER_HPP
#include <thread>
#include <openpose/core/common.hpp>
#include <openpose/experimental/3d/datum3D.hpp>
#include <openpose/pose/enumClasses.hpp>
......@@ -15,14 +14,11 @@ namespace op
public:
WRender3D(const PoseModel poseModel = PoseModel::COCO_18);
void initializationOnThread() {}
~WRender3D();
void workConsumer(const std::shared_ptr<std::vector<Datum3D>>& datumsPtr);
private:
std::thread mRenderThread;
void initializationOnThread();
void visualizationThread();
void workConsumer(const std::shared_ptr<std::vector<Datum3D>>& datumsPtr);
};
}
......
......@@ -14,9 +14,11 @@ namespace op
class OP_API Gui
{
public:
Gui(const Point<int>& outputSize, const bool fullScreen, const std::shared_ptr<std::atomic<bool>>& isRunningSharedPtr,
Gui(const Point<int>& outputSize, const bool fullScreen,
const std::shared_ptr<std::atomic<bool>>& isRunningSharedPtr,
const std::shared_ptr<std::pair<std::atomic<bool>, std::atomic<int>>>& videoSeekSharedPtr = nullptr,
const std::vector<std::shared_ptr<PoseExtractor>>& poseExtractors = {}, const std::vector<std::shared_ptr<Renderer>>& renderers = {});
const std::vector<std::shared_ptr<PoseExtractor>>& poseExtractors = {},
const std::vector<std::shared_ptr<Renderer>>& renderers = {});
void initializationOnThread();
......
......@@ -56,7 +56,8 @@ namespace op
__LINE__, __FUNCTION__, __FILE__);
Spinnaker::GenApi::CEnumEntryPtr ptrTriggerModeOff = ptrTriggerMode->GetEntryByName("Off");
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerModeOff) || !Spinnaker::GenApi::IsReadable(ptrTriggerModeOff))
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerModeOff)
|| !Spinnaker::GenApi::IsReadable(ptrTriggerModeOff))
error("Unable to disable trigger mode (enum entry retrieval). Aborting...",
__LINE__, __FUNCTION__, __FILE__);
......@@ -69,8 +70,10 @@ namespace op
// The trigger source must be set to hardware or software while trigger
// mode is off.
Spinnaker::GenApi::CEnumerationPtr ptrTriggerSource = iNodeMap.GetNode("TriggerSource");
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerSource) || !Spinnaker::GenApi::IsWritable(ptrTriggerSource))
error("Unable to set trigger mode (node retrieval). Aborting...", __LINE__, __FUNCTION__, __FILE__);
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerSource)
|| !Spinnaker::GenApi::IsWritable(ptrTriggerSource))
error("Unable to set trigger mode (node retrieval). Aborting...",
__LINE__, __FUNCTION__, __FILE__);
// Set trigger mode to hardware ('Line0')
Spinnaker::GenApi::CEnumEntryPtr ptrTriggerSourceHardware = ptrTriggerSource->GetEntryByName("Line0");
......@@ -88,7 +91,8 @@ namespace op
// Once the appropriate trigger source has been set, turn trigger mode
// on in order to retrieve images using the trigger.
Spinnaker::GenApi::CEnumEntryPtr ptrTriggerModeOn = ptrTriggerMode->GetEntryByName("On");
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerModeOn) || !Spinnaker::GenApi::IsReadable(ptrTriggerModeOn))
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerModeOn)
|| !Spinnaker::GenApi::IsReadable(ptrTriggerModeOn))
{
error("Unable to enable trigger mode (enum entry retrieval). Aborting...",
__LINE__, __FUNCTION__, __FILE__);
......@@ -128,12 +132,14 @@ namespace op
// restore the camera to a clean state.
//
Spinnaker::GenApi::CEnumerationPtr ptrTriggerMode = iNodeMap.GetNode("TriggerMode");
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerMode) || !Spinnaker::GenApi::IsReadable(ptrTriggerMode))
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerMode)
|| !Spinnaker::GenApi::IsReadable(ptrTriggerMode))
error("Unable to disable trigger mode (node retrieval). Non-fatal error...",
__LINE__, __FUNCTION__, __FILE__);
Spinnaker::GenApi::CEnumEntryPtr ptrTriggerModeOff = ptrTriggerMode->GetEntryByName("Off");
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerModeOff) || !Spinnaker::GenApi::IsReadable(ptrTriggerModeOff))
if (!Spinnaker::GenApi::IsAvailable(ptrTriggerModeOff)
|| !Spinnaker::GenApi::IsReadable(ptrTriggerModeOff))
error("Unable to disable trigger mode (enum entry retrieval). Non-fatal error...",
__LINE__, __FUNCTION__, __FILE__);
......@@ -204,7 +210,7 @@ namespace op
if (imagePtr->IsIncomplete())
{
log("Image incomplete with image status " + std::to_string(imagePtr->GetImageStatus()) + "...",
Priority::High, __LINE__, __FUNCTION__, __FILE__);
Priority::High, __LINE__, __FUNCTION__, __FILE__);
imagesExtracted = false;
break;
}
......@@ -389,7 +395,7 @@ namespace op
cameraPtr->DeInit();
}
log("Completed. Releasing...", Priority::High);
log("FLIR (Point-grey) capture completed. Releasing cameras...", Priority::High);
// Clear camera list before releasing upImpl->mSystemPtr
upImpl->mCameraList.Clear();
......@@ -398,7 +404,7 @@ namespace op
upImpl->mSystemPtr->ReleaseInstance();
}
log("Done! Exitting...", Priority::High);
log("Cameras released! Exitting program.", Priority::High);
}
catch (const Spinnaker::Exception& e)
{
......@@ -504,14 +510,14 @@ namespace op
// Remove buffer --> Always get newest frame
Spinnaker::GenApi::INodeMap& snodeMap = cameraPtr->GetTLStreamNodeMap();
Spinnaker::GenApi::CEnumerationPtr ptrBufferHandlingMode = snodeMap.GetNode("StreamBufferHandlingMode");
Spinnaker::GenApi::CEnumerationPtr ptrBufferHandlingMode = snodeMap.GetNode(
"StreamBufferHandlingMode");
if (!Spinnaker::GenApi::IsAvailable(ptrBufferHandlingMode)
|| !Spinnaker::GenApi::IsWritable(ptrBufferHandlingMode))
error("Unable to change buffer handling mode", __LINE__, __FUNCTION__, __FILE__);
Spinnaker::GenApi::CEnumEntryPtr ptrBufferHandlingModeNewest = ptrBufferHandlingMode->GetEntryByName(
"NewestFirstOverwrite"
);
"NewestFirstOverwrite");
if (!Spinnaker::GenApi::IsAvailable(ptrBufferHandlingModeNewest)
|| !IsReadable(ptrBufferHandlingModeNewest))
error("Unable to set buffer handling mode to newest (entry 'NewestFirstOverwrite' retrieval)."
......@@ -539,15 +545,15 @@ namespace op
auto cameraPtr = upImpl->mCameraList.GetByIndex(i);
// Set acquisition mode to continuous
Spinnaker::GenApi::CEnumerationPtr ptrAcquisitionMode = cameraPtr->GetNodeMap().GetNode("AcquisitionMode");
Spinnaker::GenApi::CEnumerationPtr ptrAcquisitionMode = cameraPtr->GetNodeMap().GetNode(
"AcquisitionMode");
if (!Spinnaker::GenApi::IsAvailable(ptrAcquisitionMode)
|| !Spinnaker::GenApi::IsWritable(ptrAcquisitionMode))
error("Unable to set acquisition mode to continuous (node retrieval; camera " + std::to_string(i)
+ "). Aborting...", __LINE__, __FUNCTION__, __FILE__);
error("Unable to set acquisition mode to continuous (node retrieval; camera "
+ std::to_string(i) + "). Aborting...", __LINE__, __FUNCTION__, __FILE__);
Spinnaker::GenApi::CEnumEntryPtr ptrAcquisitionModeContinuous = ptrAcquisitionMode->GetEntryByName(
"Continuous"
);
"Continuous");
if (!Spinnaker::GenApi::IsAvailable(ptrAcquisitionModeContinuous)
|| !Spinnaker::GenApi::IsReadable(ptrAcquisitionModeContinuous))
error("Unable to set acquisition mode to continuous (entry 'continuous' retrieval "
......@@ -571,7 +577,8 @@ namespace op
"DeviceSerialNumber"
);
if (Spinnaker::GenApi::IsAvailable(ptrStringSerial) && Spinnaker::GenApi::IsReadable(ptrStringSerial))
if (Spinnaker::GenApi::IsAvailable(ptrStringSerial)
&& Spinnaker::GenApi::IsReadable(ptrStringSerial))
{
strSerialNumbers[i] = ptrStringSerial->GetValue();
log("Camera " + std::to_string(i) + " serial number set to "
......@@ -607,7 +614,10 @@ namespace op
// Images to userDatum
auto datums3d = std::make_shared<std::vector<Datum3D>>(cvMats.size());
for (auto i = 0u ; i < cvMats.size() ; i++)
{
datums3d->at(i).cvInputData = cvMats.at(i);
datums3d->at(i).cvOutputData = (*datums3d)[i].cvInputData;
}
// Profiling speed
if (!cvMats.empty())
{
......
......@@ -2,6 +2,8 @@
#include <stdio.h>
#ifdef BUILD_MODULE_3D
#include <GL/glut.h>
#include <GL/freeglut_ext.h> // glutLeaveMainLoop
#include <GL/freeglut_std.h>
#endif
#include <opencv2/opencv.hpp>
#include <openpose/face/faceParameters.hpp>
......@@ -33,6 +35,7 @@ namespace op
Keypoints3D gKeypoints3D;
PoseModel sPoseModel = PoseModel::COCO_18;
int sLastKeyPressed = -1;
CameraMode gCameraMode = CameraMode::CAM_DEFAULT;
......@@ -223,7 +226,8 @@ namespace op
const auto gridNum = 10;
const auto width = 50.;//sqrt(Distance(gGloorPts.front(),gGloorCenter)*2 /gridNum) * 1.2;
const cv::Point3f origin = gGloorCenter - gGloorAxis1*(width*gridNum / 2) - gGloorAxis2*(width*gridNum / 2);
const cv::Point3f origin = gGloorCenter - gGloorAxis1*(width*gridNum / 2)
- gGloorAxis2*(width*gridNum / 2);
const cv::Point3f axis1 = gGloorAxis1 * width;
const cv::Point3f axis2 = gGloorAxis2 * width;
for (auto y = 0; y <= gridNum; ++y)
......@@ -354,6 +358,55 @@ namespace op
}
}
}
void keyPressed(const unsigned char key, const int x, const int y)
{
try
{
UNUSED(x);
UNUSED(y);
const std::lock_guard<std::mutex> lock{gKeypoints3D.mutex};
sLastKeyPressed = key;
}
catch (const std::exception& e)
{
error(e.what(), __LINE__, __FUNCTION__, __FILE__);
}
}
void initializeVisualization()
{
try
{
char *my_argv[] = { NULL };
int my_argc = 0;
glutInit(&my_argc, my_argv);
// setup the size, position, and display mode for new windows
glutInitWindowSize(1280, 720);
glutInitWindowPosition(200, 0);
// glutSetOption(GLUT_MULTISAMPLE,8);
// Ideally adding also GLUT_BORDERLESS | GLUT_CAPTIONLESS should fix the problem of disabling the `x`
// button, but it does not work (tested in Ubuntu)
glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH | GLUT_MULTISAMPLE);
// create and set up a window
glutCreateWindow(GUI_NAME.c_str());
initGraphics();
glutDisplayFunc(renderMain);
glutMouseFunc(mouseButton);
glutMotionFunc(mouseMotion);
glutIdleFunc(idleFunc);
// Full screen would fix the problem of disabling `x` button
// glutFullScreen();
// Key presses
glutKeyboardFunc(keyPressed);
}
catch (const std::exception& e)
{
error(e.what(), __LINE__, __FUNCTION__, __FILE__);
}
}
#endif
WRender3D::WRender3D(const PoseModel poseModel)
......@@ -363,10 +416,6 @@ namespace op
#ifdef BUILD_MODULE_3D
// Update sPoseModel
sPoseModel = poseModel;
// Init display
cv::imshow(GUI_NAME, cv::Mat( 500, 500, CV_8UC3, cv::Scalar{ 0,0,0 } ));
//Run OpenGL
mRenderThread = std::thread{ &WRender3D::visualizationThread, this };
#else
UNUSED(poseModel);
error("OpenPose must be compiled with `BUILD_MODULE_3D` in order to use this class.",
......@@ -379,6 +428,37 @@ namespace op
}
}
WRender3D::~WRender3D()
{
try
{
#ifdef BUILD_MODULE_3D
glutLeaveMainLoop();
#endif
}
catch (const std::exception& e)
{
error(e.what(), __LINE__, __FUNCTION__, __FILE__);
}
}
void WRender3D::initializationOnThread()
{
#ifdef BUILD_MODULE_3D
try
{
// Init display
cv::imshow(GUI_NAME, cv::Mat(500, 500, CV_8UC3, cv::Scalar{ 0,0,0 }));
// OpenGL - Initialization
initializeVisualization();
}
catch (const std::exception& e)
{
error(e.what(), __LINE__, __FUNCTION__, __FILE__);
}
#endif
}
void WRender3D::workConsumer(const std::shared_ptr<std::vector<Datum3D>>& datumsPtr)
{
#ifdef BUILD_MODULE_3D
......@@ -408,18 +488,27 @@ namespace op
gKeypoints3D.mLeftHandKeypoints = datumsPtr->at(0).leftHandKeypoints3D;
gKeypoints3D.mRightHandKeypoints = datumsPtr->at(0).rightHandKeypoints3D;
gKeypoints3D.validKeypoints = true;
// Esc pressed -> Close program
if (sLastKeyPressed == 27)
this->stop();
lock.unlock();
// Profiling speed
Profiler::timerEnd(profilerKey);
Profiler::printAveragedTimeMsOnIterationX(profilerKey, __LINE__, __FUNCTION__, __FILE__);
}
// Render images
// It sleeps 1 ms just to let the user see the output. Change to 33ms for normal 30 fps display if too fast
cv::waitKey(1);
// It sleeps 1 ms just to let the user see the 2D visual output
// Change to 33ms for normal 30 fps display if too fast
if ((char)cv::waitKey(1) == 27)
this->stop();
// OpenCL - Run main loop event
// It is run outside loop, or it would get visually stuck if loop to slow
glutMainLoopEvent();
// This alternative can only be called once, and it will block the thread until program exit
// glutMainLoop();
}
catch (const std::exception& e)
{
log("Some kind of unexpected error happened.");
this->stop();
error(e.what(), __LINE__, __FUNCTION__, __FILE__);
}
......@@ -427,38 +516,4 @@ namespace op
UNUSED(datumsPtr);
#endif
}
void WRender3D::visualizationThread()
{
#ifdef BUILD_MODULE_3D
try
{
char *my_argv[] = { NULL };
int my_argc = 0;
glutInit(&my_argc, my_argv);
// setup the size, position, and display mode for new windows
glutInitWindowSize(1280, 720);
glutInitWindowPosition(200, 0);
// glutSetOption(GLUT_MULTISAMPLE,8);
glutInitDisplayMode(GLUT_RGBA | GLUT_DOUBLE | GLUT_DEPTH | GLUT_MULTISAMPLE);
// create and set up a window
glutCreateWindow(GUI_NAME.c_str());
initGraphics();
glutDisplayFunc(renderMain);
glutMouseFunc(mouseButton);
glutMotionFunc(mouseMotion);
glutIdleFunc(idleFunc);
glutMainLoop();
this->stop();
}
catch (const std::exception& e)
{
error(e.what(), __LINE__, __FUNCTION__, __FILE__);
}
#endif
}
}
......@@ -18,7 +18,8 @@ namespace op
if (!helpCvMat.empty())
{
const auto fullScreen = false;
FrameDisplayer frameDisplayer{OPEN_POSE_TEXT + " - GUI Help", Point<int>{helpCvMat.cols, helpCvMat.rows}, fullScreen};
FrameDisplayer frameDisplayer{OPEN_POSE_TEXT + " - GUI Help",
Point<int>{helpCvMat.cols, helpCvMat.rows}, fullScreen};
frameDisplayer.displayFrame(helpCvMat, 33);
}
}
......@@ -28,8 +29,10 @@ namespace op
}
}
void handleWaitKey(bool& guiPaused, FrameDisplayer& frameDisplayer, std::vector<std::shared_ptr<PoseExtractor>>& poseExtractors,
std::vector<std::shared_ptr<Renderer>>& renderers, std::shared_ptr<std::atomic<bool>>& isRunningSharedPtr,
void handleWaitKey(bool& guiPaused, FrameDisplayer& frameDisplayer,
std::vector<std::shared_ptr<PoseExtractor>>& poseExtractors,
std::vector<std::shared_ptr<Renderer>>& renderers,
std::shared_ptr<std::atomic<bool>>& isRunningSharedPtr,
std::shared_ptr<std::pair<std::atomic<bool>, std::atomic<int>>>& spVideoSeek)
{
try
......@@ -91,13 +94,16 @@ namespace op
poseExtractor->increase(PoseProperty::NMSThreshold, 0.005f * (castedKey=='-' ? -1 : 1));
else if (castedKey=='_' || castedKey=='+')
for (auto& poseExtractor : poseExtractors)
poseExtractor->increase(PoseProperty::ConnectMinSubsetScore, 0.005f * (castedKey=='_' ? -1 : 1));
poseExtractor->increase(PoseProperty::ConnectMinSubsetScore,
0.005f * (castedKey=='_' ? -1 : 1));
else if (castedKey=='[' || castedKey==']')
for (auto& poseExtractor : poseExtractors)
poseExtractor->increase(PoseProperty::ConnectInterThreshold, 0.005f * (castedKey=='[' ? -1 : 1));
poseExtractor->increase(PoseProperty::ConnectInterThreshold,
0.005f * (castedKey=='[' ? -1 : 1));
else if (castedKey=='{' || castedKey=='}')
for (auto& poseExtractor : poseExtractors)
poseExtractor->increase(PoseProperty::ConnectInterMinAboveThreshold, (castedKey=='{' ? -0.1f : 0.1f));
poseExtractor->increase(PoseProperty::ConnectInterMinAboveThreshold,
(castedKey=='{' ? -0.1f : 0.1f));
else if (castedKey==';' || castedKey=='\'')
for (auto& poseExtractor : poseExtractors)
poseExtractor->increase(PoseProperty::ConnectMinSubsetCnt, (castedKey==';' ? -1 : 1));
......@@ -130,7 +136,8 @@ namespace op
}
void handleUserInput(FrameDisplayer& frameDisplayer, std::vector<std::shared_ptr<PoseExtractor>>& poseExtractors,
std::vector<std::shared_ptr<Renderer>>& renderers, std::shared_ptr<std::atomic<bool>>& isRunningSharedPtr,
std::vector<std::shared_ptr<Renderer>>& renderers,
std::shared_ptr<std::atomic<bool>>& isRunningSharedPtr,
std::shared_ptr<std::pair<std::atomic<bool>, std::atomic<int>>>& spVideoSeek)
{
try
......@@ -150,9 +157,11 @@ namespace op
}
}
Gui::Gui(const Point<int>& outputSize, const bool fullScreen, const std::shared_ptr<std::atomic<bool>>& isRunningSharedPtr,
Gui::Gui(const Point<int>& outputSize, const bool fullScreen,
const std::shared_ptr<std::atomic<bool>>& isRunningSharedPtr,
const std::shared_ptr<std::pair<std::atomic<bool>, std::atomic<int>>>& videoSeekSharedPtr,
const std::vector<std::shared_ptr<PoseExtractor>>& poseExtractors, const std::vector<std::shared_ptr<Renderer>>& renderers) :
const std::vector<std::shared_ptr<PoseExtractor>>& poseExtractors,
const std::vector<std::shared_ptr<Renderer>>& renderers) :
mFrameDisplayer{OPEN_POSE_TEXT, outputSize, fullScreen},
mPoseExtractors{poseExtractors},
mRenderers{renderers},
......
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册