CSCI 512 / EENG 512
Computer Vision
Spring 2016
Lab 7
February 22, 2016
NAME ________________________
NAME ________________________
This lab assignment should be done in teams of two. Go through the exercises below and show me your results.
If you don’t finish by the end of today’s class, you may show me during the next class.
The task for today is to use OpenCV to detect an “ARTag” in images
from a live camera, or video, and compute the pose of the tag. The
dimensions of the tag are 2” x 2”.
Create an OpenCV project. If you have an existing OpenCV project, the
easiest thing is to copy the entire folder of that project; otherwise follow
the steps from the previous lecture to set up a new project.
FindingtheBlackSquare
Enter the following code for the main program. You can copy and paste the code directly from this pdf
document, or download the code from the website. If you do copy and paste, it is a good idea to “clean
up” the formatting. You can do this by going to Edit->Advanced->Format Document.
/* Detect ARTags. Bill Hoff February 2016 */ #include <iostream> #include <opencv2/opencv.hpp> std::vector<cv::Point2f> findSquare(cv::Mat imageInput); void readBitMatrix(cv::Mat imageMarker, int cellSize, char bitMatrix[10][10]); int main(int argc, char* argv[]) { printf("Hit ESC key to quit\n"); cv::VideoCapture cap(0); // open the camera if (!cap.isOpened()) { // check if we succeeded printf("error ‐ can't open the camera; hit any key to quit\n"); system("PAUSE"); return EXIT_FAILURE; } while (true) { cv::Mat imageInput; cap >> imageInput; if (imageInput.empty()) break; cv::Mat imageInputGray; 1
CSCI 512 / EENG 512
}
Computer Vision
Spring 2016
cv::cvtColor(imageInput, imageInputGray, cv::COLOR_BGR2GRAY); std::vector<cv::Point2f> squareCorners; squareCorners = findSquare(imageInputGray); if (squareCorners.size() == 0) { cv::imshow("My Image", imageInput); // Wait for xx ms (0 means wait until a keypress) if (cv::waitKey(1) == 27) break; // hit ESC (ascii code 27) to quit continue; } // Draw contour as a sequence of line segments. cv::Scalar color = cv::Scalar(0, 0, 255); for (int j = 0; j < 4; j++) { cv::Point p1 = squareCorners[j]; cv::Point p2 = squareCorners[(j + 1) % 4]; cv::line(imageInput, p1, p2, color, 2, // thickness 8); // line connectivity } // Show the image. cv::imshow("My Image", imageInput); } // Wait for xx ms (0 means wait until a keypress) if (cv::waitKey(1) == 27) break; // hit ESC (ascii code 27) to quit return EXIT_SUCCESS; Add an additional function to your project. To do this, go to the “Solution Explorer” panel, right click
on “Source Files”, and select Add->New Item. Name the file “findSquare.cpp”. This function
finds a black square in the image and returns the image points corresponding to its vertices, in counter
clockwise order.
The code for this function is given in the lecture slides, or you can download it from the course website.
Run the program and verify that it finds the black square.
TransformSquareMarkerto“Orthophoto”
Enter the following code in your main loop, after the point where you draw the contour around the black
square, but before displaying the image.
// Create a list of "ortho" square corner points. std::vector<cv::Point2f> squareOrtho; squareOrtho.push_back(cv::Point2f(0, 0)); squareOrtho.push_back(cv::Point2f(100, 0)); 2
CSCI 512 / EENG 512
Computer Vision
Spring 2016
squareOrtho.push_back(cv::Point2f(100, 100)); squareOrtho.push_back(cv::Point2f(0, 100)); // Find the perspective transfomation that brings current marker to rectangular form. cv::Mat M = cv::getPerspectiveTransform(squareCorners, squareOrtho); // Transform image to get an orthophoto square image. cv::Mat imageSquare; const int cellSize = 10; cv::Size imageSquareSize(10 * cellSize, 10 * cellSize); cv::warpPerspective(imageInputGray, imageSquare, M, imageSquareSize); cv::imshow("Marker", imageSquare);
Run the program and verify that it displays the orthophoto of the square marker.
IdentifytheMarker
Enter the following code in your main loop, after the code to transform and display the orthophoto.
// Read the bit matrix. Markers are divided into 10x10 cells, // of which the inner 6x6 belongs to marker info. The external border is // 2 cells wide, and should be entirely black. char bitMatrix[10][10]; readBitMatrix(imageSquare, cellSize, bitMatrix); // Identify the marker. int id; // marker id if (!arTag.identifyMarker(bitMatrix, id, squareCorners)) { // Show the image. cv::imshow("My Image", imageInput); } // Display the marker id number. char sz[80]; sprintf_s(sz, "%d", id); putText(imageInput, sz, cv::Point(320, 240), cv::FONT_HERSHEY_PLAIN, // font face 3.0, // font scale cv::Scalar(0, 255, 255), // font color 3);
// Wait for xx ms (0 means wait until a keypress) if (cv::waitKey(1) == 27) break; // hit ESC (ascii code 27) to quit continue; Add some additional files to your project. To do this, go to the “Solution Explorer” panel, right click on
“Source Files”, and select Add->New Item. Add these items:


readBitMatrix.cpp. This function extracts a 10x10 bit pattern from the orthophoto.
ARTag.cpp. This class holds the functions to identify the marker.
3
CSCI 512 / EENG 512

Computer Vision
Spring 2016
ARTag.h. This should go under “header files”.
The code is provided at the end of this document; or you can download it from the course website.
At the top of your main program, after the include statement for “opencv.hpp”, add this line:
#include "ARTag.h" Just before the main loop, add this line:
ARTag arTag; Run the program and verify that it displays the marker id number on top of the marker.
DeterminetheMarkerPose
Add the following code, after the code that identifies the marker:
// Refine corner locations. cv::cornerSubPix(imageInputGray, squareCorners, cv::Size(5, 5), cv::Size(‐1, ‐1), cv::TermCriteria(CV_TERMCRIT_ITER, 30, 0.1)); // Create a list of 3D marker corner points. std::vector<cv::Point3f> marker3D; marker3D.push_back(cv::Point3f(‐1.0f, ‐1.0f, 0)); marker3D.push_back(cv::Point3f(+1.0f, ‐1.0f, 0)); marker3D.push_back(cv::Point3f(+1.0f, +1.0f, 0)); marker3D.push_back(cv::Point3f(‐1.0f, +1.0f, 0)); // Compute pose of marker with respect to camera. cv::Mat rotVec, transVec; bool foundPose = cv::solvePnP(marker3D, squareCorners, K, // intrinsic camera parameter matrix dist, // distortion coefficients rotVec, transVec); // output rotation and translation
You will also need to define the camera parameters (the matrix K and the distortion parameters) at the
beginning of your program. Use the same values as you used in Lab 6.
Finally, display the coordinate axes of the marker as an overlay, as you did in Lab 6. Show the
instructor your solution.
4
CSCI 512 / EENG 512
Computer Vision
Spring 2016
Code for the file “readBitMatrix.cpp”:
#include <opencv2/opencv.hpp> // This function tries to read the bit pattern of a marker. void readBitMatrix(cv::Mat imageMarker, int cellSize, char bitMatrix[10][10]) { // Threshold the marker image. cv::threshold(imageMarker, imageMarker, // output thresholded image 0, // threshold value (not used; Otsu will compute) 255, // output value cv::THRESH_OTSU | cv::THRESH_BINARY); for (int iy = 0; iy<10; iy++) { for (int ix = 0; ix<10; ix++) { int x = ix * cellSize; int y = iy * cellSize; cv::Mat cell = imageMarker(cv::Rect(x, y, cellSize, cellSize)); int nZ = cv::countNonZero(cell); if (nZ > (cellSize*cellSize) / 2) bitMatrix[iy][ix] = 1; else bitMatrix[iy][ix] = 0; } } }
5
CSCI 512 / EENG 512
Computer Vision
Spring 2016
Code for the file “ARTag.h”:
const int NPATTERNS = 2; const int N = 6; // patterns are NxN bits class ARTag { char bits[NPATTERNS][N*N] = { { 1,1,1,1,1,0, 0,1,1,1,0,1, 0,1,0,1,1,1, 0,0,0,0,0,0, 1,0,0,1,0,0, 1,1,1,0,0,0, }, { 1,1,1,1,1,0, 0,0,0,1,0,1, 1,0,1,1,0,0, 0,0,1,1,1,1, 0,0,1,1,1,1, 0,1,0,1,0,0, }, }; char bits90[NPATTERNS][N*N]; char bits180[NPATTERNS][N*N]; char bits270[NPATTERNS][N*N]; public: //////////// Constructor ////////////// ARTag(); //////////// Methods ///////////////////// bool identifyMarker(char bitMatrix[10][10], int& id, std::vector<cv::Point2f>& squareCorners); }; 6
CSCI 512 / EENG 512
Computer Vision
Spring 2016
Code for the file “ARTag.cpp”:
#include <opencv2/opencv.hpp> #include "ARTag.h" /* A private function to check if two NxN bit patterns are equal. */ bool checkEqual(char b1[], char b2[]) { for (int i = 0; i < N*N; i++) if (b1[i] != b2[i]) return false; return true; } /* The constructor for the ARTag class. It generates the rotated versions of the bit patterns. */ ARTag::ARTag() { // Rotate by 90 degrees. for (int n = 0; n < NPATTERNS; n++) for (int i = 0; i < N; i++) for (int j = 0; j < N; j++) bits90[n][i*N + j] = bits[n][(N ‐ j ‐ 1)*N + i]; // Rotate again by 90 degrees to get 180 degree rotation. for (int n = 0; n < NPATTERNS; n++) for (int i = 0; i < N; i++) for (int j = 0; j < N; j++) bits180[n][i*N + j] = bits90[n][(N ‐ j ‐ 1)*N + i]; // Rotate again by 90 degrees to get 270 degree rotation. for (int n = 0; n < NPATTERNS; n++) for (int i = 0; i < N; i++) for (int j = 0; j < N; j++) bits270[n][i*N + j] = bits180[n][(N ‐ j ‐ 1)*N + i]; } /* This function takes as input a 10x10 bit matrix, extracted from an image. It checks the bit matrix against each of the pre‐stored patterns. If it matches, it returns true, and also the id of the pattern that matched. Also input is a list of image corner points. If the input pattern is rotated, the function rotates the corner points to put them in "canonical" form; ie, the same order as the pre‐stored pattern. */ bool ARTag::identifyMarker(char bitMatrix[10][10], int& id, std::vector<cv::Point2f>& squareCorners) { // First, make sure the borders are all black. 7
CSCI 512 / EENG 512
Computer Vision
Spring 2016
for (int i = 0; i < 10; i++) for (int j = 0; j < 2; j++) if (bitMatrix[i][j] != 0) return false; for (int i = 0; i < 10; i++) for (int j = 8; j < 10; j++) if (bitMatrix[i][j] != 0) return false; for (int i = 0; i < 2; i++) for (int j = 2; j < 8; j++) if (bitMatrix[i][j] != 0) return false; for (int i = 8; i < 10; i++) for (int j = 2; j < 8; j++) if (bitMatrix[i][j] != 0) return false; // Get the inner 6x6 bit pattern char bitsInput[N*N]; for (int i = 0; i < N; i++) for (int j = 0; j < N; j++) bitsInput[i*N + j] = bitMatrix[i+2][j+2]; // Compare the input bits against each of the stored patterns. for (id = 0; id < NPATTERNS; id++) { if (checkEqual(bitsInput, bits[id])) { return true; } if (checkEqual(bitsInput, bits90[id])) { // Rotate the points one position left. std::rotate(squareCorners.begin(), squareCorners.begin() + 1, squareCorners.end()); return true; } if (checkEqual(bitsInput, bits180[id])) { // Rotate the points 2 positions left. std::rotate(squareCorners.begin(), squareCorners.begin() + 2, squareCorners.end()); return true; } if (checkEqual(bitsInput, bits270[id])) { // Rotate the points 3 positions left. std::rotate(squareCorners.begin(), squareCorners.begin() + 3, squareCorners.end()); return true; } } // Didn't match any of them. return false; } 8