Room (Extrinsic) Calibration#
Implemented in C++, not Python: mosaic::room_frame::average()
(quaternion averaging) and mosaic::room_frame::bfs_resolve()
(pose-graph resolution), both in src/calibration/room_frame_solver.hpp.
See User guide’s calibration workflow for how to actually run this
from the app, and Camera calibration for intrinsic (per-camera lens)
calibration, which this depends on.
The problem#
Given a set of Basler cameras, each already intrinsically calibrated (Camera calibration), the goal is to find every camera’s extrinsic pose — its position and orientation relative to one shared “room” origin — so gaze rays, or any other per-camera 3D signal, from different cameras can be combined in one coordinate system (see Multi-Camera Gaze Fusion).
This is solved by moving a ChArUco board through the room, capturing
“shots” where two or more cameras see the board simultaneously. For each
camera that sees the board in a given shot, cv2.solvePnP against that
camera’s real intrinsics gives a board→camera pose. The remaining problem
is: given many pairwise board→camera poses scattered across many shots,
find one consistent camera→room pose per camera.
Rotation averaging#
A single camera pair sharing several shots yields several candidate relative poses (small variations from board-detection noise) that need averaging into one. Averaging translations is a plain arithmetic mean, \(\bar t = \frac{1}{K}\sum_k t_k\). Averaging rotations is not as simple, because unit quaternions double-cover \(SO(3)\): the map from the unit quaternion \(q\) to the rotation it represents is 2-to-1 — both \(q\) and \(-q\) represent the exact same rotation. A naive arithmetic mean of quaternions sampled around one physical orientation can therefore inadvertently sum quaternions from opposite hemispheres of that double cover and partially cancel toward zero, even though every sample represents nearly the same rotation.
Rotation → quaternion. Each sample’s 3×3 rotation matrix \(R\) is first converted to a unit quaternion via the numerically-robust branch of Shepperd’s method (branching on whichever of the trace or the three diagonal elements is largest, to avoid dividing by a near-zero term). For the common case \(\text{tr}(R) = R_{00}+R_{11}+R_{22} > 0\):
(the other three branches pivot on whichever diagonal element is largest, following the same standard derivation).
Sign alignment. Before averaging, every sample \(q_k\) is sign-aligned to the first sample \(q_{\text{ref}} = q_1\): if \(q_{\text{ref}} \cdot q_k < 0\), negate \(q_k\). Only after this alignment are the (now consistently-signed) quaternions summed and renormalized:
Important
(1) is a sign-aligned renormalized mean, not the exact chordal/Karcher mean of \(SO(3)\). It’s only a valid approximation of that true mean when the input rotations are already close together — which holds here specifically because the samples being averaged are repeated shots of the same static rig, not an arbitrary set of rotations. This is a deliberate, documented trade-off, not a claim of general correctness.
The averaged quaternion is converted back to a 3×3 rotation matrix by the standard formula, and combined with the arithmetic-mean translation into one averaged rigid transform.
Pose-graph resolution (BFS)#
With one averaged relative pose available for every camera pair that shared at least one shot, the remaining step is to compose these into one consistent pose per camera, relative to a single reference camera (camera 0, which is always the identity transform — see the Mathematical Background’s shared-conventions table).
Let \(T_{\text{cam} \to \text{room}}\) denote the transform that
defines a camera’s extrinsic pose (mapping a point in that camera’s local
frame into room coordinates). For a BFS edge from an already-resolved
parent to an unresolved child, using shot \(k\)’s
board→parent pose \(T_{\text{board}\to\text{parent}}^{(k)}\) and
board→child pose \(T_{\text{board}\to\text{child}}^{(k)}\):
Reading the right-hand side right-to-left makes the composition concrete: a point in the child camera’s local frame is mapped to the board’s frame by \((T_{\text{board}\to\text{child}}^{(k)})^{-1}\) (i.e. \(T_{\text{child}\to\text{board}}^{(k)}\)), then from the board’s frame to the parent camera’s frame by \(T_{\text{board}\to\text{parent}}^{(k)}\), then from the parent camera’s frame to the room frame by \(T_{\text{parent}\to\text{room}}\) — which is exactly \(T_{\text{child}\to\text{room}}\).
Every shared shot between the same camera pair produces one such candidate via (2); these are combined with the quaternion-mean (1) above into a single edge pose. Starting from the reference camera (identity), a breadth-first traversal of the shared-shot graph propagates a resolved pose outward to every reachable camera. A camera with no shot-chain back to the reference camera — e.g. it never shared a simultaneous board view with anything already resolved — is reported unresolved rather than silently assigned a meaningless identity pose, so the calibration UI can flag it and ask for more overlapping shots.
The room’s reference plane#
Once every camera’s extrinsic pose is known, the room’s reference plane (used by Multi-Camera Gaze Fusion’s target-point intersection) is defined for free: lay the ChArUco board flat on the target surface, capture one more shot, and reuse its already-solved board pose. For any resolved camera \(c\) that saw that shot directly, the board’s pose in room coordinates is \(T_{\text{board}\to\text{room}} = T_{c\to\text{room}} \circ T_{\text{board}\to c}\). The plane point is that transform’s translation column; the plane normal is its rotation part’s 3rd column (the board’s own printed-face normal, since ChArUco object points lie in the board’s local \(Z=0\) plane by construction).
Practical recommendations#
Every camera needs some path of shared shots back to camera 0 — not necessarily direct. Move the board through overlapping pairs of camera fields of view deliberately, not just wherever’s convenient; a camera left unresolved contributes nothing to any downstream 3D plugin (Multi-Camera Gaze Fusion, 3D Pose Reconstruction).
Multiple shared shots between the same camera pair feed (1)’s averaging step directly — a pair with only one shared shot has no noise-averaging benefit at all. A handful of varied-angle shots per overlapping pair meaningfully improves that pair’s resolved relative pose over a single shot.
A camera reporting “resolved” with a poor reprojection RMS still silently degrades every downstream 3D computation through it — treat a high RMS the same as an unresolved camera and recapture that camera’s shots with more care (steadier board, less motion blur, better lighting) rather than accepting a technically-non-null but low-quality pose.
The averaged extrinsics describe the cameras’ positions at calibration time. Any physical camera move — even a small bump — invalidates every pose computed from it, silently, with no runtime check catching the mismatch. Recalibrate after any physical rig change, not just when results start looking wrong.