Event-Camera Status Matrix
Canonical readiness view for the synthetic IMX636 / EVK4 event-camera pipeline. Status terms on this page are the source of truth for the overview, mirror contracts, and dataset CLI reference.
Simulator pair and evidence scope
The target runtime is Isaac Sim 6.1.0 with Isaac Lab 3.0.0. Rows marked live or phase-complete describe SRB contracts and synthetic output; they do not certify event fidelity, real EVK4 capture, or hardware readiness. Native evidence is scoped and local; it does not establish broad task coverage or release acceptance.
The event-camera observation seam is independent of artifact persistence.
HDF5 and manifest persistence accepts either camera_record or
event_record_requested; RGB MP4 capture follows camera_record. Concrete
RGB-dependent providers fail closed when their source frame is absent or
malformed; noop and synthetic contract smokes do not stand in for those
provider checks.
Scoped native functional proof
A native CUDA camera in one velocity_tracking environment exercised
v2e_streaming through an automatic reset. Provider output, HDF5, and manifest
events agreed, with bounded chunks and workspace. The probe used
camera_record=False and event_record_requested=True; the full CLI
--record-events path and paired MP4 remain unaccepted after an RGB writer
failure. This local run does not establish event fidelity, throughput, peak
memory, all-task wiring, hardware readiness, or release acceptance. It also
predates the camera-convention alignment; orientation is covered by a unit
test. One native CUDA RGB camera in a velocity_tracking environment covered
4,096 control steps at 125 Hz simulated cadence. All 4,096 frames changed;
4,019,523 event rows agreed across provider output, HDF5, and manifests in two
nonempty episodes and one empty reset episode. All 130,811 chunks were at most
64 rows, with a provider workspace ceiling of 4,096 rows.
0.1.0 task-variant evaluation
An opt-in v2e_streaming collection on six task variants recorded nonempty
event episodes for velocity_tracking,
waypoint_navigation and sample_collection. Three did not, and are declared
limitations of this release:
landingandmoonfall(onboard, downward camera): v2e loggedno signal events generatedon every frame of a 32-step random rollout, so the episodes hold no events. A 200-steplandingrollout reached the per-episode event-observation element limit after 59 frames, still without events.excavation(wrist camera): one run exceeded the defaultobservation_window_eventsbound (3,686,400 rows) while finalizing the episode; a rerun stayed within it and recorded 27,540,315 events. The overflow depends on the rollout. Closing the environment now raises that failure instead of only logging it, so collection no longer reports success without the episode.
This local evaluation makes no fidelity claim.
| Area | Status | Current contract | Gate to next state |
|---|---|---|---|
| Sensor config | Live (scoped) | BaseEnvCfg._add_sensors materializes each EventCameraCfg into the active Isaac scene as a native RGB CameraCfg under the same logical name and keeps the original Isaac-free config in the scene-side event-camera bindings; DirectEnv binds those sensors after the scene is built. The scoped native proof exercised the camera and patched provider in velocity_tracking. | Acceptance of native wiring across all six tasks; no event-fidelity claim follows from the functional proof. |
| Task adoption | Live (scoped) | Opt-in task toggles retain EventCameraCfg and active-sensor intent; the runtime observation assembly discovers each bound sensor and registers its event representation plus event-count observations. One native velocity_tracking environment exercised event-only persistence across an automatic reset. | Acceptance of native wiring across all six tasks; full CLI --record-events proof and a provider-backed fidelity differential remain open. |
noop backend | Live | Built-in smoke adapter emits zero events. | None. |
v2e backend | Live (non-streaming) | Canonical default backend; install with srb[event-v2e] plus the source-event-v2e group (Git-pinned v2e). Upstream returns complete event arrays, so the adapter remains one-shot and strict_streaming=True rejects it before construction. | Live optional-extra smoke (host CPU unit evidence only so far). Frame interpolation is rejected until an adapter-owned path exists. |
v2e_streaming backend | Functional proof (scoped) | Explicit signal-only profile backed by the recorded EventEmulator.iter_events patch in srb/core/sensor/event_camera/backends/v2e_pin.toml. The installed-provider CI tests cover first-chunk timing, bounded chunks, equivalence, workspace failure, cancellation rebuild, and recorder integration. The scoped strict-image native CUDA run exercised 4,096 calls across an automatic reset and verified provider/HDF5/manifest agreement. Construction requires the patched source digest and disabled photoreceptor noise, CSDVS, SCIDVS, and shot noise; rejected chunks close the iterator. | Default images need the explicit patch build option. Native wiring across all six tasks remains open; event fidelity, throughput, and peak memory are unmeasured. |
metavision backend | Live (non-streaming) | Optional adapter and import contract are pinned; the provider materializes complete results, so it is rejected under strict_streaming=True. | Live optional-extra CI with OpenEB installed. |
v2ce backend | Live (non-streaming) | Optional adapter supports package-style V2CE providers and the script-style V2CE-Toolbox layout; it also materializes complete results and is rejected under strict_streaming=True. | Promote the opt-in V2CE checkpoint smoke into routine heavy CI; publish a public mirror. |
| Synthetic recording | Live (scoped) | srb dataset record-events writes canonical EVENT_HDF5 episodes and events_manifest.jsonl; its moving-patch output always uses task_id=fixture_moving_patch, policy_id=synthetic_fixture, and empty isaaclab_version, while retaining the detected SRB revision. Requested --task and --policy-id are compatibility inputs; the summary retains the requested task and labels the source synthetic_fixture. Only the exact task marker identifies fixtures; historical or metadata-free arrays are not inferred. Native event-only persistence passed in one strict-image velocity_tracking run with camera_record=False and event_record_requested=True; artifacts agreed with provider output. | Full CLI --record-events path and paired RGB MP4 remain unaccepted after an RGB writer failure. No event-fidelity pass is claimed; the default image needs the explicit patched-provider build option. |
| Statistical validation | Live | srb dataset validate-events stats compares rate, polarity, ISI, hot-pixel, and sparsity metrics. | Reference Gen4 fixture for routine CI. |
| Reconstruction validation | Live | srb dataset validate-events reconstruction provides baseline SSIM scoring and an E2VID-gated adapter path. | Promote the opt-in E2VID checkpoint smoke into routine heavy CI. |
| LeRobot export | Live | srb dataset export-lerobot-v3 wraps EVENT_HDF5 runs with an event_camera modality, warns when selected episodes carry the fixture marker, and preserves task/policy provenance. Its contains_synthetic_fixture flag covers mixed selections too. | Publication smoke against a hosted dataset. |
| Zenodo / RAW / ROS interop | Live | export-zenodo-bundle, export-raw-evt3, import-raw-evt3, and export-rosbag preserve the canonical event run layout. Zenodo warns and preserves fixture task/policy provenance, including a contains_synthetic_fixture flag that caller metadata cannot replace; NPZ also warns and retains HDF5 metadata. RAW EVT3 and ROS bag reject marked fixtures before publication because they cannot preserve the marker. | Heavy optional-extra smoke for RAW and ROS bag writers plus a licensed fixture mirror. |
| SuperSloMo weights | Private mirror | hf-andrejorsula-supersloMo is registered; fetching requires HF repo permission plus HF_TOKEN, HUGGING_FACE_HUB_TOKEN, or hf auth login. | Public availability pending. |
| E2VID weights | Private mirror | hf-andrejorsula-e2vid is registered; fetching requires HF repo permission plus HF_TOKEN, HUGGING_FACE_HUB_TOKEN, or hf auth login. | Public availability pending. |
| Gen4 reference fixture | Private mirror | hf-andrejorsula-active-marker-gen4 is registered; fetching requires HF dataset permission plus HF_TOKEN, HUGGING_FACE_HUB_TOKEN, or hf auth login. | Confirm Prophesee redistribution terms before public release; downstream harness still needs a labelled/task-specific fixture. |
| Preset retune | Phase A | Physical-ordering and distinguishability invariants are tested. | Phase B calibration needs srb dataset capture-real-evk4 data plus SuperSloMo weights. |
| Release smoke | Phase B opt-in | Optional-extra names are tested without heavy dependencies; tests/integration/test_event_camera_phase_b_smoke.py validates the private active-marker fixture plus E2VID and V2CE checkpoints when explicitly enabled. | Promote to routine CI after public/licensed mirrors and a heavy release image are ready. |
| Downstream validation | Gated (CLI rejects) | Only --synthetic-smoke runs. Every other srb dataset validate-events downstream invocation, including --write-label-template and --check-real-target, is refused by the CLI preflight before any output is written because no validated train-on-synth / eval-on-real harness or model adapter exists; the scorecard, label-template, and real-target helpers exist in-tree and are unit-tested. | Land the real harness with a labelled/task-specific Gen4 target and model adapter, then open the preflight gate. |
| EVK4 capture | Gated (CLI rejects) | srb dataset capture-real-evk4 parses its arguments and then exits through the CLI preflight before creating any directory, manifest, or protocol output: no validated hardware capture backend and device contract exists in this build. | Requires a validated capture backend, hardware capture campaign, and publication approval. |
Status Terms
| Term | Meaning |
|---|---|
| Live | The interface is implemented and usable from SRB today, subject to any documented optional extra. |
| Live (scoped) | The path is implemented and wired, and bounded local evidence exercises the stated task and backend combination. It is not broad task coverage, a provider-fidelity, throughput, memory, real-data, hardware, or release claim. |
| Functional proof (scoped) | A named provider and simulator path passed the stated functional checks in a local run. It does not imply fidelity, performance, broad task coverage, or release acceptance. |
| Live (non-streaming) | The backend adapter works, but the upstream provider materializes complete event results, so strict_streaming=True rejects it. |
| Gated (CLI rejects) | The command parses, then exits through a preflight capability gate before producing output. |
| Placeholder | The command or registry exists, but the default artifact intentionally refuses to download because SRB does not redistribute the data. |
| Gated | The interface is documented so downstream work can target it, but meaningful use waits on an explicit external artifact, hardware campaign, or harness. |
| Private mirror | A stable mirror is registered, but the HF repo is private and requires collaborator access plus an HF token. |
| Phase A | Lightweight invariant or contract is landed. |
| Phase B opt-in | Data-backed smoke exists in-tree, but runs only when explicitly enabled with local credentials/artifacts. |
| Phase B | End-to-end data-backed validation still waits on the task-specific artifacts or heavy CI image named in the row. |
User Paths
| Goal | Command | Status |
|---|---|---|
| List registered event-generator backends | srb event-camera backends --detailed | Live |
| List registered checkpoint mirrors | srb event-camera list-mirrors | Live |
| Fetch checkpoint by explicit URL | srb event-camera fetch-weights --url <url> --sha256 <sha> | Live |
| Fetch checkpoint by mirror | srb event-camera fetch-weights --mirror <short_name> | Private mirror; requires HF token |
| List registered fixture mirrors | srb event-camera list-fixtures | Live |
| Fetch fixture by explicit URL | srb event-camera fetch-fixture --url <url> --sha256 <sha> | Live |
| Fetch fixture by mirror | srb event-camera fetch-fixture --mirror <short_name> | Private mirror; requires HF token |
| Record synthetic event-camera data | srb dataset record-events --output OUT | Live (scoped) |
| Run statistical fidelity validation | srb dataset validate-events stats --synth SYN --real REAL --output OUT | Live |
| Run reconstruction fidelity validation | srb dataset validate-events reconstruction --events EVENTS --rgb-npy RGB --times-npy TIMES --output OUT | Live |
| Run downstream fidelity validation | srb dataset validate-events downstream --synth-events SYN --real-events REAL --output OUT | Rejected by preflight; no harness |
| Run synthetic downstream adapter smoke | srb dataset validate-events downstream --synth-events SYN --task event_flow --output OUT --synthetic-smoke | Synthetic contract smoke live; no real metrics |
| Check downstream real-target labels | srb dataset validate-events downstream --synth-events SYN --real-events REAL --task event_flow --output OUT --check-real-target | Rejected by preflight; contract helper exists in-tree |
| Write downstream label template | srb dataset validate-events downstream --synth-events SYN --real-events REAL --task event_flow --output OUT --write-label-template | Rejected by preflight; contract helper exists in-tree |
| Export to LeRobot v3 | srb dataset export-lerobot-v3 EVENT_RUN --output OUT | Live |
| Build a Zenodo bundle | srb dataset export-zenodo-bundle EVENT_RUN --output OUT | Live |
| Export Prophesee RAW EVT3 | srb dataset export-raw-evt3 EVENTS_H5 --output OUT | Live |
| Import Prophesee RAW EVT3 | srb dataset import-raw-evt3 EVENTS_RAW --output OUT_H5 | Live |
| Export ROS bag | srb dataset export-rosbag EVENTS_H5 --output OUT | Live |
| Capture real EVK4 data | srb dataset capture-real-evk4 --task TASK --policy POLICY --output OUT | Rejected by preflight; no capture backend |
The opt-in Phase-B smoke is:
SRB_EVENT_CAMERA_PHASE_B_SMOKE=1 \
SRB_EVENT_CAMERA_FETCH=1 \
pytest -q tests/integration/test_event_camera_phase_b_smoke.py