MOSAIC Documentation#
MOSAIC · Multi-camera Observatory for Social & Activity Interaction Capture
Qt 6 · C++23 · Basler Pylon · OpenCV
Build MOSAIC, deploy to a recording station, and capture your first multi-camera session in under ten minutes.
How the subsystems fit together: video pipeline, audio, triggers, calibration, and the profile system.
Session folders, file formats, per-frame timestamp CSV, and the
session_meta.json sidecar.
Every Image/Exposure/Gain/Advanced/HW-Trigger field, its default, and how to change it for one camera or for every future one.
Checkerboard intrinsic calibration with OpenCV; results stored per-camera and per-profile.
Per-research-group profiles: isolated configs, PBKDF2 passwords, and the in-app profile switcher.
Task-oriented walkthrough: recording, browsing sessions, running each Analysis-tab plugin, and the calibration workflow end to end.
The analysis/ project’s importable library surface — Pose, Face
Masking, Diarization, Expression (incl. py-feat), Gaze Fusion, Motion,
3D Pose Reconstruction, and Remote Heart Rate — with full
numpydoc-rendered signatures.
The derivations behind each analysis plugin: gaze-ray triangulation, room-frame pose averaging, kinematics, and more.
A curated, subsystem-grouped tour of the app’s C++ classes, with a link out to the full Doxygen + Breathe generated reference.
Key capabilities#
Multi-camera |
Basler Pylon SDK with per-camera exposure, gain, ROI, pixel format, and hardware trigger control; test-pattern stub for development without hardware. |
Synchronised audio |
Per-microphone WAV recording via |
Precise timestamps |
Every video frame carries a monotonic |
Trigger logging |
Keyboard bindings, serial bytes, parallel-port TTL edges — all written to a CSV with nanosecond resolution, on the same clock as camera frames. |
EEG/trigger frame sync |
Post-hoc lookup of which camera frame corresponds to each parallel-port trigger event (e.g. an EEG amplifier’s trigger-out cable); parallel ports can also send a recording start/stop marker back to the EEG. |
Camera calibration |
OpenCV checkerboard intrinsic calibration, plus multi-camera ChArUco room (extrinsic) calibration for combining 3D signals across cameras. |
Eight offline analysis plugins |
Pose, Face Masking, Speaker Diarization, Facial Expression (3 backends incl. real FACS Action Units via py-feat), Multi-Camera Gaze Fusion, Motion Tracking, 3D Pose Reconstruction, and an experimental Remote Heart Rate (rPPG) estimator — see User guide. |
Live dashboard |
The Real-time tab shows live pose/gaze overlays, a live audio waveform, and live speech captions per camera/microphone, with automatic pause during active recording. |
Research profiles |
Isolated settings, recording directories, and calibration per group; PBKDF2-SHA256 login; per-user recording access control with an aggregated admin view. |
