Question about using with with a tilted grid non-chessboard pattern

[tewarfel]

Hi Dima, thank you for writing and sharing this software with the open source community.
I have a question regarding using the python tools to calibrate a pair of thermal infrared cameras using an anchored (non-checkerboard) grid pattern with a moving rig holding two thermal cameras at a fixed separation of about 1 meter.

The calibration target is an array of stick-on resistive electric hotpads against a wooden board; e.g. visual camera appearance like this:

and thermal pair images like this:

I have a "dot detector" that can output vnl-format text files of the dot pattern centers, starting from top right dot, and
providing the x,y coordinates for the centers in order of top-to-bottom, right-to-left. An example file (not specifically from the above images) gives the center points as:

## generated with make_vnlfiles.py
# filename x y level
stereo_cam2_00007.png 704.3236 467.94455 0
stereo_cam2_00007.png 695.18475 533.9793 0
stereo_cam2_00007.png 686.5277 600.4337 0
stereo_cam2_00007.png 678.0138 667.6202 0
stereo_cam2_00007.png 667.8977 496.71448 0
stereo_cam2_00007.png 657.7925 563.66754 0
stereo_cam2_00007.png 649.7433 630.1107 0
stereo_cam2_00007.png 641.88007 696.6739 0
stereo_cam2_00007.png 637.7005 458.31003 0
stereo_cam2_00007.png 627.91504 524.61676 0
stereo_cam2_00007.png 619.01154 591.028 0
stereo_cam2_00007.png 610.5706 658.3819 0
stereo_cam2_00007.png 601.77106 486.57288 0
stereo_cam2_00007.png 591.06506 553.82135 0
stereo_cam2_00007.png 583.2952 620.97815 0
stereo_cam2_00007.png 574.2796 688.2082 0
stereo_cam2_00007.png 569.1609 448.06207 0
stereo_cam2_00007.png 559.0736 514.76953 0
stereo_cam2_00007.png 551.06335 582.1288 0
stereo_cam2_00007.png 542.4579 649.3663 0
stereo_cam2_00007.png 533.81104 476.6088 0
stereo_cam2_00007.png 524.1529 544.13086 0
stereo_cam2_00007.png 514.11597 611.4478 0
stereo_cam2_00007.png 505.4032 678.06744 0
stereo_cam2_00007.png 500.84088 438.70718 0
stereo_cam2_00007.png 491.37918 505.05344 0
stereo_cam2_00007.png 482.02417 572.33105 0
stereo_cam2_00007.png 473.0924 639.07385 0
stereo_cam2_00007.png 465.06732 466.6333 0
stereo_cam2_00007.png 455.16403 534.18567 0
stereo_cam2_00007.png 445.67804 601.48804 0
stereo_cam2_00007.png 437.5754 668.36255 0
stereo_cam2_00007.png 432.20752 427.99762 0
stereo_cam2_00007.png 422.10034 495.39325 0
stereo_cam2_00007.png 411.8926 562.6122 0
stereo_cam2_00007.png 403.84375 630.01996 0

There are some imprecisions in how the dot array is physically laid out, but I have a reasonable model of the real-world dot positions as a combination of an algorithmic generation plus a table of offsets:

from target_corrections import corrections_california as corrections
boardSize = (4,9) # 4 (staggered) rows  of  9 columns per staggered row#35 

#  example of one staggered row of 9 columns
#            *     *     *     *     *     
#                *     *     *     *


frameSize = (1280, 1024) # sensor is 1280 wide x1024 tall

# assuming sensor pixel is 11 micrometers square (0.001 cm per pixel)
# and estimated focal length is 25mm (2.50 cm), then focal length in pixels
# will be 2.5/0.001 pixels

est_focal_length = 2.5 / 0.001

est_cx = frameSize[0] / 2
est_cy = frameSize[1] / 2

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0). 

objp = np.zeros((boardSize[0] * boardSize[1], 3), np.float32)

# assuming:
# units in centimeters
# target is ~10 meters away
#
#                  |----9.5" ----|
#    *             *             *             *             *  --
#                                                               |
#           *             *             *             *         9.5"
#                                                               |
#    *             *             *             *             * --
#
#           *             *             *             *
# with board upright, targets are spaced 9.5" apart vertically and horizontally, so
# vertical/horizontal center-to-center distance is 24.13 cm apart
# and diagonal center-to-center distance is 17.062 cm apart


# indexing the pattern starts at top right, and goes top to bottom, then shifts
# 1 column left and repeats top-to-bottom, until all 9 columns are completed.

#initialize objp starting from top left, and going top-to-bottom, left-to-right
step_size = 24.13 # center-to-center distance in cm
half_step_size = step_size / 2.0

# the openCV  findCircleGrid returns sequence ordered as top->bottom, right-to-left !!!
# and openCV's  calibrateCamera expects that.

# *** objp units are in cm ***
target_world_X = 0
target_world_Y = 0
target_world_Z = 0
index = 0

if apply_known_offsets:
    for x in range(boardSize[1], 0, -1):
        for y in range(boardSize[0]):
            target_world_X = (x * half_step_size) + half_step_size + (corrections[index][0])
            target_world_Y = ((y * step_size) + (((x + 1) % 2) * half_step_size)) + (corrections[index][1])
            objp[index] = (target_world_X, target_world_Y, target_world_Z)
            index += 1
else:
    for x in range(boardSize[1], 0, -1):
        for y in range(boardSize[0]):
            target_world_X = (x * half_step_size) + half_step_size
            target_world_Y = ((y * step_size) + (((x+1)%2) * half_step_size))
            objp[index] = (target_world_X, target_world_Y, target_world_Z)
            index += 1

The final result is a list of X,Y,Z coordinates for the centers (where Z=0, and X,Y are horizontal and vertical distances in centimeters from top left corner of the board holding the thermal pads.

Where I'm having difficulty is how to pass this list of object-space centers to the test-surveyed-calibration.py example. Is there a simple way to give it the objp structure as a list of points (possibly as a vml file), or do I need to separate the points into two distinct 5 wide x 4 tall and 4 wide x 4 tall overlapped rectangular grid pairs and just process one of those grids?

Thank you,
-Tom

[dkogan]

Hi. First of all, test-surveyed-calibration.py solves only a toy problem: the board and camera cannot move. You can pass the 3d board points as "points" individually instead of "board" observations. This works today, but will make the uncertainty routine not work (will be fixed in mrcal 3.0)

So what I would suggest you do, is to define a regular 3d grid of points that's 2x the density of the points on your board, and tell mrcal to ignore half of them (x,y,level are "-"). Then you'll be able to communicate your layout to mrcal, as "board" observations, and everything will work. So if your object_spacing was 9.5, set it to 9.5/2. In the first row, all the even points would be missing, in the seccond row all the odd points would be missing, and so on. Does that make sense?

Unrelatedly, which thermal camera are you using? And have you tried other calibration board designs for IR? I'm working with this currently, and am looking for a solution.

[tewarfel]

(via email interface, edited to remove duplicate text) Thank you. Yes, that makes sense, and I will give it a try and see what happens. Previously, we had used FLIR A65 cameras (640x512 pixels, 17 micron pixel size, based on the FLIR “Tau” camera module https://www.flircameras.com/flir-a65-thermal-camera-series ) with a Pleora GeniCam gigabit ethernet interface board, using the Pleora GigE SDK (https://www.pleora.com/machine-vision-connectivity/ebus-sdk/ ) on Ubuntu Linux, with each camera having its own network interface and configured to send 8000-byte “jumbo” IPv4 ethernet packets. We are trying to now use the Sierra-Olympia Tenum 1280 (https://sierraolympia.com/product/tenum-1280/ ) with a similar Pleora interface board and (more or less) the same SDK interface with double the horizontal and vertical resolution. The GeniCam interface on the Tenum implementation is missing some features that were present on the FLIR A65, and that has required some low-level fiddling with the camera’s non-volatile registers to get the cameras to use an external sync signal. Since we’ve already had to dive into the low-level register fiddling, we are also considering just using their USB 3 interface for the next iteration (instead of ethernet) as a possibly less expensive hardware interface option, and supplying 12v directly to the camera (along with the sync signal) instead of using POE+. While I’m still working with the Tenum 1280 (and I really want to like it), the FLIR A65 has been a much more reliable and predictable camera and is the one I’d recommend to anyone just starting a new project. I wouldn’t recommend the Tenum to a friend just yet; it keeps surprising me with weird new failure modes, and it cannot really sustain continuous 30 frames-per-second 14-bit acquisitions without occasionally skipping frames. -Tom

[dkogan]

Interesting. Can you shoot me an email? We're doing similar things, and it'd be good to compare notes.

[tewarfel]

Okay, I'm getting the right and left individual camera model calculations to work.

I'm still unclear on how to best create the corners.vnl file for left cam images (of the stereo pairs) and the right cam images (of the stereo pairs) to generate extrinsics.

Right now I'm creating "solo" image collections, (a set of "left camera only" images + a set of "right camera only" images), and a set of stereo pair images as two directories: stereo/cam1/image001.png... and stereo/cam2/image001.png....

The images in the stereo pair subdirectories are a copied and renamed subset of the solo left camera and solo right camera images. I can calculate the intrinsic parameters with the (better coverage) solo images individually, and would like to calculate the extrinsics with the (matched but some areas of no coverage) stereo pair images.

To get the best extrinsics solution from the images I have:

1. Should I keep the non-stereo-paired solo images in the .vnl file to provide full FOV coverage for each camera, or just put the stereo pair images in the .vnl file?
2. How should I label the entries in the corners.vnl file so that left and right camera images are clearly distinguished from each other?
3. If I'm passing in the corners.vnl file, do the actual blob names generated have to match the filenames in corners.vnl, or simply be non-null?

I'll try to send a separate email to the address listed on the secretsauce.net site.

Thank you,
-Tom

[dkogan]

Hi. You should make one large image collection, containing ALL the images. Left-only and right-only and joint ones. You don't want to run separate intrinsics-only and then extrinsics-only solves: run a single joint solve instead. This is both the easiest and best way to do this.

1 Should I keep the non-stereo-paired solo images in the .vnl file to provide full FOV coverage for each camera, or just put the stereo pair images in the .vnl file?

You want one corners.vnl with everything in it.

2 How should I label the entries in the corners.vnl file so that left and right camera images are clearly distinguished from each other? 3 If I'm passing in the corners.vnl file, do the actual blob names generated have to match the filenames in corners.vnl, or simply be non-null?

The camera ids are inferred from the filename globs. In my usual case the filenames are frameAAAAA-camB.jpg. And I run mrcal-calibrate-cameras .... '*-cam0.jpg' '*-cam1.jpg' This tells mrcal that we're calibrating two cameras, and their images are matched by the given globs. For each camera, the changing part of the filenames (the "AAAAA" above) indicates which instant of time we're looking at, to match up images from the different cameras. Not everything must be matched up. For instance, it's perfectly acceptable to have frame00000-cam0.jpg frame00001-cam0.jpg frame00002-cam0.jpg frame00003-cam0.jpg frame00004-cam0.jpg frame00004-cam1.jpg frame00005-cam0.jpg frame00005-cam1.jpg frame00006-cam0.jpg frame00006-cam1.jpg frame00007-cam1.jpg frame00008-cam1.jpg frame00009-cam1.jpg frame00010-cam1.jpg frame00011-cam1.jpg Where frames 0-3 are cam0-only (presumably for intrinsics), frames 4-6 are joint (for extrinsics) and frames 7-11 are cam1-only. The solver will give you a full solution, and the validation routines will tell you if there are issues.

[tewarfel]

In the case where I'm trying to just pass data via the corners.vnl file, does there need to be a strict 1:1 correspondence between the filenames listed in the corners.vnl file and real files with that name in some findable subdirectory? If I have a fully detailed corners.vnl file but do not have a physically corresponding file for every entry, does it still process that corners entry, discard that entry (which it can't find the image for) from the computation, or return an invalid result? I have the "corners" pre-computed in this case and am not using a corner detector in the running of mrcal-calibrate-cameras. Thank you.

[dkogan]

You don't need to have files on disk at all. The mrcal-calibrate-cameras --corners tool only uses them to get the image dimensios which you can provide yourself with --imagersize