Event-Camera PRD
This page is the local source of truth for the synthetic IMX636 / EVK4 event-camera sensor and dataset pipeline.
Product Goal
SRB should let users record space-robotics policies with a neuromorphic equivalent of the Prophesee IMX636 / EVK4 sensor, validate synthetic fidelity, and publish event-camera datasets for robot-learning, machine-learning, and computer-vision workflows.
Scope
- Standalone
EventCameraCfgwith task-mounted poses, IMX636 presets, backend choice, representation choice, sub-render rate, and per-env RNG. - Pluggable event-generator backends:
noop,v2e,metavision, andv2ce. - Canonical
EVENT_HDF5storage plus manifest reproducibility metadata. - On-demand validation: statistical fidelity, reconstruction fidelity, and downstream train-on-synth / eval-on-real fidelity.
- Dataset interchange: LeRobot v3, Zenodo bundle, Prophesee RAW EVT3, and ROS bag export/import paths.
- Private staging mirrors for weights and fixtures until redistribution terms and release readiness are confirmed.
Milestones
| Milestone | Done when | Current state |
|---|---|---|
| M1 walking skeleton | Event-camera config, storage, manifest, and docs exist end to end. | Live |
| M2 training surface | Event observations can be wired into SRB tasks without changing downstream consumers. | Wired end to end (config, scene materialization, observation delivery, recorder); provider fidelity unproven |
| M3 dataset validation/release | Fidelity validators, interchange formats, and hosted artifacts are ready for publication. | Partly live; public release gated |
M2 is wired end to end: standalone EventCameraCfg, synthetic event
generation, storage, validation, and export paths are live, BaseEnvCfg
materializes the config into the Isaac task scene, and the runtime delivers
event_<name> / event_<name>_count observations. What remains open is
provider fidelity — every concrete backend materializes complete results, none
ships in the default container image, and the only end-to-end Isaac evidence is
a focused local runner with a strict test backend. No simulator-backed event
fidelity differential pass is claimed.
Current Blockers
| Blocker | Why it matters | What is possible without it |
|---|---|---|
| Labelled physical EVK4 task dataset | Required for downstream generalization numbers. | Validate schema, write label templates, run synthetic-only harness plumbing. |
| Public redistribution approval for mirrors | Required before private HF artifacts become public. | Keep private mirrors registered for permissioned collaborators. |
| Heavy optional-extra CI image | Required for routine OpenEB / E2VID / ROS bag smoke. | Keep opt-in private smoke and lightweight parser/contract tests. |
| Task model adapter | Required for real event-flow or detection train/eval. | Freeze scorecard, label contract, and adapter-facing CLI shape. |
| Concrete provider in the release image | Required for a provider-backed event differential in the SRB task runtime; the materialization and delivery seams already exist. | Run the wired path with the strict test backend, plus the live synthetic, storage, validation, and export paths. |
User Stories
- As a dataset creator, I can record synthetic event-camera runs with enough provenance to reproduce policy, seed, backend, preset, and task context.
- As a validation owner, I can compare synthetic and physical EVK4 data without running fidelity checks inside every policy-training loop.
- As a downstream researcher, I can train event-flow or detection models on SRB synthetic data and later evaluate on labelled physical EVK4 captures using the same scorecard schema.
- As a collaborator, I can fetch private HF mirrors after being granted access, while public release remains blocked until licensing and curation are ready.
References
- Status: Event-Camera Status Matrix
- Intake runbook: Physical EVK4 Intake
- Architecture:
docs/adr/0002-event-camera-sensor.md(in the repository, not in this book)