API Reference#

MOSAIC#

page MOSAIC

Multi-camera Observatory for Social & Activity Interaction Capture

A synchronized multi-camera + audio recording suite for research labs, built around Basler GigE cameras, with live pose/gaze preview, post-recording analysis, and parallel-port/serial trigger integration for syncing with external systems (e.g. EEG amplifiers).

Capabilities#

  • Synchronized capture from multiple Basler GigE cameras (per-camera settings: exposure, gain, ROI, pixel format, hardware trigger input)

  • Multi-microphone audio recording alongside video

  • Post-hoc frame-accurate cross-camera sync (sync_manifest.json), with per-camera and per-frame timestamp logs

  • Keyboard, serial, and parallel-port trigger sources, with a session-wide trigger event log — parallel ports can also send a recording start/stop marker back out to an external device (e.g. an EEG amplifier’s trigger channel)

  • Post-hoc EEG-trigger-to-camera-frame lookup (Analysis tab’s “EEG/Trigger ↔ Frame Sync” plugin)

  • Live in-app pose & gaze preview (MediaPipe, CPU) during acquisition

  • Post-recording batch pose/motion analysis (analysis/: YOLOv8-pose, centroid tracking, heatmaps)

  • Checkerboard camera calibration

  • Per-user/lab-group login profiles, each with fully isolated settings

  • Session browser + synchronized multi-camera playback

Requirements#

The base build (UI, settings, profiles, tests) only needs Qt + CMake. Everything else is an opt-in MOSAIC_ENABLE_* flag — see Feature flags below.

Tool

Version

Needed for

CMake

≥ 3.25

always

C++ compiler

MSVC 2022 / GCC 13 / Clang 17 (C++23)

always

Qt

6.4+ (Core, Gui, Widgets, Network, Multimedia, Quick, QuickWidgets)

always

vcpkg

—

GTest, OpenCV, FFmpeg

Basler Pylon SDK

7.x

-EnableCameras

FFmpeg

via vcpkg (x264 feature)

-EnableFfmpeg

OpenCV

4.x via vcpkg

-EnableOpenCV (calibration)

CUDA + NVIDIA driver

—

-EnableNvenc

All optional features compile with stub fallbacks when disabled — you can develop and test the full UI without any lab hardware attached.

Building#

# 1. Clone
git clone https://github.com/fcbg-platforms/mosaic.git
cd mosaic

# 2. Install vcpkg packages
vcpkg install

# 3. Configure & build — base build, no hardware
.\scripts\configure.ps1 -BuildType Release -BuildTests
cmake --build build\Release --parallel

# Full build with cameras + FFmpeg + calibration
.\scripts\configure.ps1 -BuildType Release -EnableCameras -EnableFfmpeg -EnableOpenCV
cmake --build build\Release --parallel

# Deploy Qt DLLs so the .exe runs on other machines
windeployqt --qmldir src\qml build\Release\bin\mosaic.exe

macOS/Linux: ./scripts/configure.sh (see docs/quickstart.rst for the full flag reference).

Run tests: 

cd build\Release
ctest --output-on-failure

Feature flags#

Flag

Default

Requires

MOSAIC_ENABLE_CAMERAS

OFF

Basler Pylon SDK at PYLON_ROOT%

MOSAIC_ENABLE_FFMPEG

OFF

FFmpeg (vcpkg, x264 feature)

MOSAIC_ENABLE_NVENC

OFF

FFmpeg + CUDA + NVIDIA driver

MOSAIC_ENABLE_OPENCV

OFF

OpenCV 4.x (vcpkg)

MOSAIC_ENABLE_PARALLEL_PORT

OFF

Windows + InpOut32.dll next to the exe

MOSAIC_ENABLE_SERIAL

ON

Qt SerialPort (auto-detected)

MOSAIC_BUILD_TESTS

OFF

GTest (vcpkg)

MOSAIC_BUILD_DOCS

OFF

Doxygen + Sphinx (see Python environments)

CI (.github/workflows/ci.yml) builds and tests the hardware-free configuration only — Pylon is a licensed vendor SDK not fetchable via vcpkg, and no camera hardware exists on hosted runners. Camera/FFmpeg-touching changes need manual verification against real hardware; note how you tested in the PR description.

Python environments#

python/, analysis/, and docs/ are three independent uv projects (own pyproject.toml/uv.lock/.venv each) — not a shared workspace, since they have genuinely conflicting dependencies (e.g. python/ needs a light, headless OpenCV for the real-time capture path; analysis/ needs the full OpenCV build plus torch/ultralytics for batch pose analysis). Install only what you need:

cd python && uv sync      # real-time pose/gaze worker (spawned automatically by the app)
cd analysis && uv sync    # post-recording batch pose/motion analysis (YOLOv8-pose)
cd docs && uv sync        # Sphinx documentation build

Lint/format with ruff (config shared at repo-root ruff.toml; ruff isn’t a dependency of any of the three projects, so use uvx — an isolated, ad-hoc tool run — not uv run) from the repo root:

uvx ruff check --config ruff.toml .
uvx ruff format --config ruff.toml .

or install the .pre-commit-config.yaml hooks (pre-commit install) to run it automatically.

Project structure#

mosaic/
├── src/
│   ├── core/        # Application bootstrap, settings persistence
│   ├── auth/         # Login profiles, per-profile settings isolation
│   ├── video/         # Camera grabber (Pylon), encoder (FFmpeg), ring buffer feed
│   ├── audio/         # Microphone recorder, WAV writer
│   ├── trigger/       # Keyboard / serial / parallel-port triggers
│   ├── record/        # Session recording orchestration
│   ├── session/        # Session metadata
│   ├── analysis/       # Sync manifest, real-time pose/gaze worker, post-recording analysis launcher
│   ├── calibration/    # Checkerboard camera calibration
│   ├── ui/             # Qt widgets (video/audio/trigger settings, session browser/player, auth)
│   ├── qml/             # Live monitor view (Qt Quick)
│   └── utils/            # Logger, lock-free ring buffer, timestamps
├── python/          # uv-managed real-time pose/gaze worker (MediaPipe), spawned by src/analysis
├── analysis/        # Post-recording batch pose/motion analysis scripts (YOLOv8-pose, tracking)
├── tests/           # Google Test unit tests
├── docs/            # Sphinx + Doxygen documentation source
├── scripts/         # Build/setup helper scripts (configure, NIC/camera provisioning)
└── cmake/           # Find modules and compiler options

Documentation#

Full docs (architecture, quickstart, calibration, recording layout, profiles) live under docs/ — build them with MOSAIC_BUILD_DOCS=ON (see the table above), or start with docs/quickstart.rst directly.

Contributing#

See CONTRIBUTING.md — every PR needs a test, and CI must pass.

License#

[MIT](LICENSE)

../build-docs/doxygen/xml/docs/_static/fcbg-logo.png ../build-docs/doxygen/xml/docs/_static/neuro-logo.png

Developed by

File Hierarchy#

Full API#

Namespaces#

Classes and Structs#

Enums#

Functions#

Typedefs#