Merge pull request #23575 from vovka643:4.x_aruco_calib3d_calibration

add ChArUco board pattern into calib3d/camera_calibration #23575

Added opportunity to calibrate camera using ChArUco board pattern in /samples/cpp/tutorial_code/calib3d/camera_calibration/caera_calibration.cpp 
### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake

doc/tutorials/calib3d/camera_calibration/camera_calibration.markdown

@@ -60,6 +60,7 @@ done through basic geometrical equations. The equations used depend on the chose
 objects. Currently OpenCV supports three types of objects for calibration:
 
 -   Classical black-white chessboard
+-   ChArUco board pattern
 -   Symmetrical circle pattern
 -   Asymmetrical circle pattern
 
@@ -88,7 +89,8 @@ Source code
 
 You may also find the source code in the `samples/cpp/tutorial_code/calib3d/camera_calibration/`
 folder of the OpenCV source library or [download it from here
-](https://github.com/opencv/opencv/tree/4.x/samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp). For the usage of the program, run it with `-h` argument. The program has an
+](https://github.com/opencv/opencv/tree/4.x/samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp).
+For the usage of the program, run it with `-h` argument. The program has an
 essential argument: the name of its configuration file. If none is given then it will try to open the
 one named "default.xml". [Here's a sample configuration file
 ](https://github.com/opencv/opencv/tree/4.x/samples/cpp/tutorial_code/calib3d/camera_calibration/in_VID5.xml) in XML format. In the
@@ -128,14 +130,23 @@ Explanation
 
     The formation of the equations I mentioned above aims
     to finding major patterns in the input: in case of the chessboard this are corners of the
-    squares and for the circles, well, the circles themselves. The position of these will form the
+    squares and for the circles, well, the circles themselves. ChArUco board is equivalent to
+    chessboard, but corners are mached by ArUco markers. The position of these will form the
     result which will be written into the *pointBuf* vector.
     @snippet samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp find_pattern
     Depending on the type of the input pattern you use either the @ref cv::findChessboardCorners or
-    the @ref cv::findCirclesGrid function. For both of them you pass the current image and the size
-    of the board and you'll get the positions of the patterns. Furthermore, they return a boolean
-    variable which states if the pattern was found in the input (we only need to take into account
-    those images where this is true!).
+    the @ref cv::findCirclesGrid function or @ref cv::aruco::CharucoDetector::detectBoard method.
+    For all of them you pass the current image and the size of the board and you'll get the positions
+    of the patterns. cv::findChessboardCorners and cv::findCirclesGrid return a boolean variable
+    which states if the pattern was found in the input (we only need to take into account
+    those images where this is true!). `CharucoDetector::detectBoard` may detect partially visible
+    pattern and returns coordunates and ids of visible inner corners.
+
+    @note Board size and amount of matched points is different for chessboard, circles grid and ChArUco.
+    All chessboard related algorithm expects amount of inner corners as board width and height.
+    Board size of circles grid is just amount of circles by both grid dimentions. ChArUco board size
+    is defined in squares, but detection result is list of inner corners and that's why is smaller
+    by 1 in both dimentions.
 
     Then again in case of cameras we only take camera images when an input delay time is passed.
     This is done in order to allow user moving the chessboard around and getting different images.

samples/cpp/tutorial_code/calib3d/camera_calibration/camera_calibration.cpp

@@ -11,6 +11,7 @@
 #include <opencv2/imgcodecs.hpp>
 #include <opencv2/videoio.hpp>
 #include <opencv2/highgui.hpp>
+#include "opencv2/objdetect/charuco_detector.hpp"
 
 using namespace cv;
 using namespace std;
@@ -19,7 +20,7 @@ class Settings
 {
 public:
     Settings() : goodInput(false) {}
-    enum Pattern { NOT_EXISTING, CHESSBOARD, CIRCLES_GRID, ASYMMETRIC_CIRCLES_GRID };
+    enum Pattern { NOT_EXISTING, CHESSBOARD, CHARUCOBOARD, CIRCLES_GRID, ASYMMETRIC_CIRCLES_GRID };
     enum InputType { INVALID, CAMERA, VIDEO_FILE, IMAGE_LIST };
 
     void write(FileStorage& fs) const                        //Write serialization for this class
@@ -28,7 +29,10 @@ public:
                   << "BoardSize_Width"  << boardSize.width
                   << "BoardSize_Height" << boardSize.height
                   << "Square_Size"         << squareSize
+                  << "Marker_Size"      << markerSize
                   << "Calibrate_Pattern" << patternToUse
+                  << "ArUco_Dict_Name"   << arucoDictName
+                  << "ArUco_Dict_File_Name" << arucoDictFileName
                   << "Calibrate_NrOfFrameToUse" << nrFrames
                   << "Calibrate_FixAspectRatio" << aspectRatio
                   << "Calibrate_AssumeZeroTangentialDistortion" << calibZeroTangentDist
@@ -48,10 +52,13 @@ public:
     }
     void read(const FileNode& node)                          //Read serialization for this class
     {
-        node["BoardSize_Width" ] >> boardSize.width;
+        node["BoardSize_Width"] >> boardSize.width;
         node["BoardSize_Height"] >> boardSize.height;
         node["Calibrate_Pattern"] >> patternToUse;
-        node["Square_Size"]  >> squareSize;
+        node["ArUco_Dict_Name"] >> arucoDictName;
+        node["ArUco_Dict_File_Name"] >> arucoDictFileName;
+        node["Square_Size"] >> squareSize;
+        node["Marker_Size"] >> markerSize;
         node["Calibrate_NrOfFrameToUse"] >> nrFrames;
         node["Calibrate_FixAspectRatio"] >> aspectRatio;
         node["Write_DetectedFeaturePoints"] >> writePoints;
@@ -147,6 +154,7 @@ public:
 
         calibrationPattern = NOT_EXISTING;
         if (!patternToUse.compare("CHESSBOARD")) calibrationPattern = CHESSBOARD;
+        if (!patternToUse.compare("CHARUCOBOARD")) calibrationPattern = CHARUCOBOARD;
         if (!patternToUse.compare("CIRCLES_GRID")) calibrationPattern = CIRCLES_GRID;
         if (!patternToUse.compare("ASYMMETRIC_CIRCLES_GRID")) calibrationPattern = ASYMMETRIC_CIRCLES_GRID;
         if (calibrationPattern == NOT_EXISTING)
@@ -198,8 +206,11 @@ public:
     }
 public:
     Size boardSize;              // The size of the board -> Number of items by width and height
-    Pattern calibrationPattern;  // One of the Chessboard, circles, or asymmetric circle pattern
+    Pattern calibrationPattern;  // One of the Chessboard, ChArUco board, circles, or asymmetric circle pattern
     float squareSize;            // The size of a square in your defined unit (point, millimeter,etc).
+    float markerSize;            // The size of a marker in your defined unit (point, millimeter,etc).
+    string arucoDictName;        // The Name of ArUco dictionary which you use in ChArUco pattern
+    string arucoDictFileName;    // The Name of file which contains ArUco dictionary for ChArUco pattern
     int nrFrames;                // The number of frames to use from the input for calibration
     float aspectRatio;           // The aspect ratio
     int delay;                   // In case of a video input
@@ -283,9 +294,6 @@ int main(int argc, char* argv[])
     fs.release();                                         // close Settings file
     //! [file_read]
 
-    //FileStorage fout("settings.yml", FileStorage::WRITE); // write config as YAML
-    //fout << "Settings" << s;
-
     if (!s.goodInput)
     {
         cout << "Invalid input detected. Application stopping. " << endl;
@@ -295,12 +303,63 @@ int main(int argc, char* argv[])
     int winSize = parser.get<int>("winSize");
 
     float grid_width = s.squareSize * (s.boardSize.width - 1);
+    if (s.calibrationPattern == Settings::Pattern::CHARUCOBOARD) {
+        grid_width = s.squareSize * (s.boardSize.width - 2);
+    }
+
     bool release_object = false;
     if (parser.has("d")) {
         grid_width = parser.get<float>("d");
         release_object = true;
     }
 
+    //create CharucoBoard
+    cv::aruco::Dictionary dictionary;
+    if (s.calibrationPattern == Settings::CHARUCOBOARD) {
+        if (s.arucoDictFileName == "") {
+            cv::aruco::PredefinedDictionaryType arucoDict;
+            if (s.arucoDictName == "DICT_4X4_50") { arucoDict = cv::aruco::DICT_4X4_50; }
+            else if (s.arucoDictName == "DICT_4X4_100") { arucoDict = cv::aruco::DICT_4X4_100; }
+            else if (s.arucoDictName == "DICT_4X4_250") { arucoDict = cv::aruco::DICT_4X4_250; }
+            else if (s.arucoDictName == "DICT_4X4_1000") { arucoDict = cv::aruco::DICT_4X4_1000; }
+            else if (s.arucoDictName == "DICT_5X5_50") { arucoDict = cv::aruco::DICT_5X5_50; }
+            else if (s.arucoDictName == "DICT_5X5_100") { arucoDict = cv::aruco::DICT_5X5_100; }
+            else if (s.arucoDictName == "DICT_5X5_250") { arucoDict = cv::aruco::DICT_5X5_250; }
+            else if (s.arucoDictName == "DICT_5X5_1000") { arucoDict = cv::aruco::DICT_5X5_1000; }
+            else if (s.arucoDictName == "DICT_6X6_50") { arucoDict = cv::aruco::DICT_6X6_50; }
+            else if (s.arucoDictName == "DICT_6X6_100") { arucoDict = cv::aruco::DICT_6X6_100; }
+            else if (s.arucoDictName == "DICT_6X6_250") { arucoDict = cv::aruco::DICT_6X6_250; }
+            else if (s.arucoDictName == "DICT_6X6_1000") { arucoDict = cv::aruco::DICT_6X6_1000; }
+            else if (s.arucoDictName == "DICT_7X7_50") { arucoDict = cv::aruco::DICT_7X7_50; }
+            else if (s.arucoDictName == "DICT_7X7_100") { arucoDict = cv::aruco::DICT_7X7_100; }
+            else if (s.arucoDictName == "DICT_7X7_250") { arucoDict = cv::aruco::DICT_7X7_250; }
+            else if (s.arucoDictName == "DICT_7X7_1000") { arucoDict = cv::aruco::DICT_7X7_1000; }
+            else if (s.arucoDictName == "DICT_ARUCO_ORIGINAL") { arucoDict = cv::aruco::DICT_ARUCO_ORIGINAL; }
+            else if (s.arucoDictName == "DICT_APRILTAG_16h5") { arucoDict = cv::aruco::DICT_APRILTAG_16h5; }
+            else if (s.arucoDictName == "DICT_APRILTAG_25h9") { arucoDict = cv::aruco::DICT_APRILTAG_25h9; }
+            else if (s.arucoDictName == "DICT_APRILTAG_36h10") { arucoDict = cv::aruco::DICT_APRILTAG_36h10; }
+            else if (s.arucoDictName == "DICT_APRILTAG_36h11") { arucoDict = cv::aruco::DICT_APRILTAG_36h11; }
+            else {
+                cout << "incorrect name of aruco dictionary \n";
+                return 1;
+            }
+
+            dictionary = cv::aruco::getPredefinedDictionary(arucoDict);
+        }
+        else {
+            cv::FileStorage dict_file(s.arucoDictFileName, cv::FileStorage::Mode::READ);
+            cv::FileNode fn(dict_file.root());
+            dictionary.readDictionary(fn);
+        }
+    }
+    else {
+        // default dictionary
+        dictionary = cv::aruco::getPredefinedDictionary(0);
+    }
+    cv::aruco::CharucoBoard ch_board({s.boardSize.width, s.boardSize.height}, s.squareSize, s.markerSize, dictionary);
+    cv::aruco::CharucoDetector ch_detector(ch_board);
+    std::vector<int> markerIds;
+
     vector<vector<Point2f> > imagePoints;
     Mat cameraMatrix, distCoeffs;
     Size imageSize;
@@ -308,8 +367,8 @@ int main(int argc, char* argv[])
     clock_t prevTimestamp = 0;
     const Scalar RED(0,0,255), GREEN(0,255,0);
     const char ESC_KEY = 27;
-
     //! [get_input]
+
     for(;;)
     {
         Mat view;
@@ -356,6 +415,10 @@ int main(int argc, char* argv[])
         case Settings::CHESSBOARD:
             found = findChessboardCorners( view, s.boardSize, pointBuf, chessBoardFlags);
             break;
+        case Settings::CHARUCOBOARD:
+            ch_detector.detectBoard( view, pointBuf, markerIds);
+            found = pointBuf.size() == (size_t)((s.boardSize.height - 1)*(s.boardSize.width - 1));
+            break;
         case Settings::CIRCLES_GRID:
             found = findCirclesGrid( view, s.boardSize, pointBuf );
             break;
@@ -367,8 +430,9 @@ int main(int argc, char* argv[])
             break;
         }
         //! [find_pattern]
+
         //! [pattern_found]
-        if ( found)                // If done with success,
+        if (found)                // If done with success,
         {
               // improve the found corners' coordinate accuracy for chessboard
                 if( s.calibrationPattern == Settings::CHESSBOARD)
@@ -388,7 +452,10 @@ int main(int argc, char* argv[])
                 }
 
                 // Draw the corners.
-                drawChessboardCorners( view, s.boardSize, Mat(pointBuf), found );
+                if(s.calibrationPattern == Settings::CHARUCOBOARD)
+                    drawChessboardCorners( view, cv::Size(s.boardSize.width-1, s.boardSize.height-1), Mat(pointBuf), found );
+                else
+                    drawChessboardCorners( view, s.boardSize, Mat(pointBuf), found );
         }
         //! [pattern_found]
         //----------------------------- Output Text ------------------------------------------------
@@ -530,15 +597,25 @@ static void calcBoardCornerPositions(Size boardSize, float squareSize, vector<Po
     {
     case Settings::CHESSBOARD:
     case Settings::CIRCLES_GRID:
-        for( int i = 0; i < boardSize.height; ++i )
-            for( int j = 0; j < boardSize.width; ++j )
+        for (int i = 0; i < boardSize.height; ++i) {
+            for (int j = 0; j < boardSize.width; ++j) {
                 corners.push_back(Point3f(j*squareSize, i*squareSize, 0));
+            }
+        }
+        break;
+    case Settings::CHARUCOBOARD:
+        for (int i = 0; i < boardSize.height - 1; ++i) {
+            for (int j = 0; j < boardSize.width - 1; ++j) {
+                corners.push_back(Point3f(j*squareSize, i*squareSize, 0));
+            }
+        }
         break;
-
     case Settings::ASYMMETRIC_CIRCLES_GRID:
-        for( int i = 0; i < boardSize.height; i++ )
-            for( int j = 0; j < boardSize.width; j++ )
-                corners.push_back(Point3f((2*j + i % 2)*squareSize, i*squareSize, 0));
+        for (int i = 0; i < boardSize.height; i++) {
+            for (int j = 0; j < boardSize.width; j++) {
+                corners.push_back(Point3f((2 * j + i % 2)*squareSize, i*squareSize, 0));
+            }
+        }
         break;
     default:
         break;
@@ -563,7 +640,12 @@ static bool runCalibration( Settings& s, Size& imageSize, Mat& cameraMatrix, Mat
 
     vector<vector<Point3f> > objectPoints(1);
     calcBoardCornerPositions(s.boardSize, s.squareSize, objectPoints[0], s.calibrationPattern);
-    objectPoints[0][s.boardSize.width - 1].x = objectPoints[0][0].x + grid_width;
+    if (s.calibrationPattern == Settings::Pattern::CHARUCOBOARD) {
+        objectPoints[0][s.boardSize.width - 2].x = objectPoints[0][0].x + grid_width;
+    }
+    else {
+        objectPoints[0][s.boardSize.width - 1].x = objectPoints[0][0].x + grid_width;
+    }
     newObjPoints = objectPoints[0];
 
     objectPoints.resize(imagePoints.size(),objectPoints[0]);
@@ -634,6 +716,7 @@ static void saveCameraParams( Settings& s, Size& imageSize, Mat& cameraMatrix, M
     fs << "board_width" << s.boardSize.width;
     fs << "board_height" << s.boardSize.height;
     fs << "square_size" << s.squareSize;
+    fs << "marker_size" << s.markerSize;
 
     if( !s.useFisheye && s.flag & CALIB_FIX_ASPECT_RATIO )
         fs << "fix_aspect_ratio" << s.aspectRatio;

samples/cpp/tutorial_code/calib3d/camera_calibration/in_VID5.xml

@@ -2,15 +2,16 @@
 <opencv_storage>
 <Settings>
   <!-- Number of inner corners per a item row and column. (square, circle) -->
-  <BoardSize_Width> 9</BoardSize_Width>
+  <BoardSize_Width>9</BoardSize_Width>
   <BoardSize_Height>6</BoardSize_Height>
   
   <!-- The size of a square in some user defined metric system (pixel, millimeter)-->
   <Square_Size>50</Square_Size>
-  
-  <!-- The type of input used for camera calibration. One of: CHESSBOARD CIRCLES_GRID ASYMMETRIC_CIRCLES_GRID -->
+  <Marker_Size>25</Marker_Size>
+  <!-- The type of input used for camera calibration. One of: CHESSBOARD CHARUCOBOARD CIRCLES_GRID ASYMMETRIC_CIRCLES_GRID -->
   <Calibrate_Pattern>"CHESSBOARD"</Calibrate_Pattern>
-  
+  <ArUco_Dict_Name>DICT_4X4_50</ArUco_Dict_Name>
+  <ArUco_Dict_File_Name></ArUco_Dict_File_Name>
   <!-- The input to use for calibration. 
 		To use an input camera -> give the ID of the camera, like "1"
 		To use an input video  -> give the path of the input video, like "/tmp/x.avi"