Mathematical Background#

This section derives the math behind each analysis capability, alongside the code that implements it — for readers who want to know why an algorithm produces the numbers it does, not just how to call it. For the importable Python API itself, see Python Analysis API; for how to run each plugin from the app, see User guide.

Shared conventions#

Every page below that deals with 3D geometry (Multi-Camera Gaze Fusion, Room (Extrinsic) Calibration, 3D Pose Reconstruction) uses the same conventions, matching room_frame::Mat4 (C++) and ray_math.py’s module docstring (Python):

Convention

Meaning

Units

Positions/lengths in millimetres; angles in the formulas below in radians (APIs that take degrees convert internally).

Rigid transform

A 4×4 homogeneous matrix \([R\,|\,t]\), applied as \(p_{\text{out}} = R \, p_{\text{in}} + t\).

Storage

Row-major, flattened length-16 (index \(i \cdot 4 + j\) is row i, column j; the bottom row is implicitly \([0,0,0,1]\)).

Reference frame

Camera 0 is always the room-frame origin (identity transform); every other camera’s extrinsic pose is relative to it.

🎯 Multi-Camera Gaze Fusion

3D ray construction, least-squares triangulation, ray-plane intersection.

Multi-Camera Gaze Fusion
📐 Room (Extrinsic) Calibration

Quaternion averaging and BFS pose-graph resolution across cameras.

Room (Extrinsic) Calibration
🏃 Pose Kinematics

Gap-tolerant velocity/acceleration and why average speed is time-weighted.

Pose Kinematics
🙂 Facial Expression

Weighted blendshape scoring and the FER+ softmax.

Facial Expression
🗣️ Speaker Diarization

Max-overlap interval assignment between transcript and speaker turns.

Speaker Diarization
🫥 Face Masking

Box padding/clamping and blur-kernel sizing.

Face Masking
🐭 Motion Tracking

Image-moment centroids and greedy nearest-neighbour assignment.

Motion Tracking
🧍‍♂️ 3D Pose Reconstruction

Multi-view DLT triangulation, cross-camera person association, and 3D track identity.

3D Pose Reconstruction
❤️ Remote Heart Rate (rPPG)

Green/CHROM/POS pulse extraction and Welch-periodogram BPM estimation. Experimental.

Remote Heart Rate (rPPG)