Multi-Camera Gaze Fusion#

Implemented in gaze.estimator (per-camera head-pose solve) and gaze.ray_math (pure geometry — see Python Analysis API for the full API reference). This page derives the three stages: turning a solved head pose into a camera-local 3D ray, transforming that ray into room coordinates, and fusing several cameras’ rays into one triangulated point (optionally intersected with a target plane).

Stage 1 — camera-local ray construction#

camera_ray_from_pose() takes a head pose \((R, t)\) — solved per-camera via cv2.solvePnP against a generic 6-point 3D face model and that camera’s real calibrated intrinsics — plus the existing 2D iris-offset heuristic \((\Delta_x, \Delta_y) \in [-1, 1]^2\) (the same signal the live, single-camera gaze path already computes), and produces a 3D ray origin and direction in the camera’s local frame.

Origin. The ray starts at the eye-centre model point \(o_{\text{model}}\) (the midpoint of the two eye-outer-corner model points), mapped into camera space by the solved head pose:

(1)#\[\text{origin} = R \, o_{\text{model}} + t\]

Direction. Rather than trusting a monocular depth estimate for the eye itself (unobservable from iris landmarks alone), the direction composes two rotations: the head’s own solved orientation, and a small eye-in-socket perturbation bounded by tunable constants \(\psi_{\max}\) (max_eye_yaw_deg, default 30°) and \(\varphi_{\max}\) (max_eye_pitch_deg, default 20°):

\[\psi = \Delta_x \, \psi_{\max}, \qquad \varphi = \Delta_y \, \varphi_{\max}\]
\[\begin{split}R_{\text{yaw}}(\psi) = \begin{bmatrix} \cos\psi & 0 & \sin\psi \\ 0 & 1 & 0 \\ -\sin\psi & 0 & \cos\psi \end{bmatrix} \qquad R_{\text{pitch}}(\varphi) = \begin{bmatrix} 1 & 0 & 0 \\ 0 & \cos\varphi & -\sin\varphi \\ 0 & \sin\varphi & \cos\varphi \end{bmatrix}\end{split}\]

Applied to the face model’s own forward axis \(f = (0, 0, 1)^\mathsf{T}\), then rotated into camera space by the solved head pose:

(2)#\[d_{\text{model}} = R_{\text{yaw}}(\psi) \, \big( R_{\text{pitch}}(\varphi) \, f \big), \qquad d = \frac{R \, d_{\text{model}}}{\lVert R \, d_{\text{model}} \rVert}\]

When \(\Delta_x = \Delta_y = 0\), \(d\) reduces exactly to the head’s own forward direction \(R f\) — the eye perturbation vanishes.

Note

This is deliberately a heuristic, not a claim of true stereo eye depth. It separates two genuinely different signals: which way the head is pointing (metric, solved via solvePnP against real camera intrinsics) from which way the eyes are rotated within it (a bounded heuristic perturbation) — strictly more information than the live 2D-only estimator it replaces, which conflates the two.

Stage 2 — transform to room coordinates#

transform_ray_to_room() applies that camera’s extrinsic pose \([R_e \,|\, t_e]\) (from Room (Extrinsic) Calibration):

\[\text{origin}_{\text{room}} = R_e \, \text{origin} + t_e, \qquad d_{\text{room}} = \frac{R_e \, d}{\lVert R_e \, d \rVert}\]

(the direction is rotated only — no translation applies to a direction vector).

Stage 3 — multi-ray least-squares triangulation#

Given \(N \ge 2\) contributing cameras, each with a room-space ray \((o_i, d_i)\), closest_point_of_rays() finds the single 3D point \(x^\star\) minimizing the summed squared perpendicular distance to every ray.

For ray \(i\), the orthogonal projector onto the plane perpendicular to \(d_i\) is \(P_i = I - d_i d_i^\mathsf{T}\) (symmetric, idempotent: \(P_i^2 = P_i\)). The squared perpendicular distance from a point \(x\) to ray \(i\) is \(\lVert P_i (x - o_i) \rVert^2\), so the objective is:

(3)#\[F(x) = \sum_{i=1}^{N} (x - o_i)^\mathsf{T} P_i \, (x - o_i)\]

Setting the gradient to zero:

\[\nabla F(x) = 2 \sum_i P_i (x - o_i) = 0 \;\;\Longrightarrow\;\; \underbrace{\Big(\sum_i P_i\Big)}_{A} x = \underbrace{\sum_i P_i o_i}_{b} \;\;\Longrightarrow\;\; A x = b\]

\(A\) is a 3×3 matrix, symmetric positive semi-definite, and positive definite (invertible) whenever the contributing ray directions aren’t all parallel — the common case with \(\ge 2\) cameras viewing the same face from different angles. The implementation solves \(Ax=b\) directly via np.linalg.solve, falling back to the Moore–Penrose pseudo-inverse \(x^\star = A^{+}b\) (the minimum-norm least-squares solution) when \(A\) is near-singular — e.g. nearly-parallel rays, a genuinely degenerate configuration — rather than raising.

Fit quality. The reported residual is the RMS perpendicular distance from \(x^\star\) back to every contributing ray:

\[\text{residual}_{\text{rms}} = \sqrt{ \frac{1}{N} \sum_i \lVert P_i (x^\star - o_i) \rVert^2 }\]

For a single ray (\(N=1\)), there’s nothing to triangulate — the function returns that ray’s own origin with a residual of exactly 0.

Target-plane intersection#

ray_plane_intersection() finds where the fused ray \(x(u) = x^\star + u\,\bar d\) (using the mean of the contributing directions as the fused direction \(\bar d\)) crosses a plane defined by a point \(p_0\) and unit normal \(n\) (the room’s reference plane, set once during Room (Extrinsic) Calibration). Substituting into the plane equation \(n \cdot (x(u) - p_0) = 0\) and solving for \(u\):

\[u = \frac{n \cdot (p_0 - x^\star)}{n \cdot \bar d}\]

The intersection is reported as None (no target point) in two cases: \(|n \cdot \bar d|\) below a small epsilon (the ray runs parallel to the plane — no well-defined intersection), or \(u < 0\) (the intersection lies behind the ray’s origin, i.e. the gaze direction points away from the surface).

Practical recommendations#

📷 More cameras, better fusion

Below the --min-cameras threshold (default 2), a per-camera ray is still recorded but there’s nothing to triangulate. 3+ cameras viewing the subject from meaningfully different angles gives both a better-conditioned \(A\) matrix in Stage 3’s least-squares solve and a lower residual — treat 2-camera fusion as the practical minimum, not the target.

📐 Room calibration accuracy dominates

Every stage here is only as good as the room extrinsics from Room (Extrinsic) Calibration. A camera with a marginal reprojection RMS there silently degrades every fused ray computed through it — if target points look physically implausible, re-check calibration quality before suspecting the fusion math.

👁️ The eye-rotation term is a bounded heuristic

Remember the direction formula separates two genuinely different signals: a metric, solvePnP-solved head pose, and a bounded heuristic eye-in-socket perturbation (\(\pm 30°\)/\(\pm 20°\) by default). Don’t over-interpret gaze precision for subjects looking sharply off-axis from their own head direction — the heuristic’s bound is a real accuracy ceiling, not just a implementation detail.

📊 Reading the residual

residual_rms_mm is the direct fit-quality signal — a large residual means the contributing cameras’ rays didn’t actually converge well, and any target point derived from that fit deserves correspondingly less trust, independent of how “close” it looks to a plausible screen/table location.