Event Camera (IMX636)

An event camera is a bio-inspired vision sensor: instead of capturing full frames at a fixed rate, each pixel independently reports brightness changes with microsecond timestamps. This makes it well-suited to the high-dynamic-range and fast-motion conditions common in space scenes (harsh lunar shadows, direct sun, rapid docking). SRB ships a synthetic one so you can generate event-stream datasets entirely in simulation.

SRB provides a synthetic neuromorphic-vision sensor that emits an asynchronous event stream modelled on a Prophesee IMX636 (EVK4), together with live synthetic storage, validation, and export paths. Isaac task-scene materialization and event-observation delivery are implemented and wired, but every concrete provider is one-shot and none ships in the default container image, so this guide does not claim provider-backed rollout fidelity or real EVK4 fidelity.

The sensor is standalone — it owns its own pose, intrinsics, internal high-frame-rate renderer, IMX636 sensor-model preset, and per-env RNG. It is not a Camera subclass; events are not frames.

Current rollout readiness is tracked in the Status Matrix. The local PRD is the event-camera source of truth, and Physical EVK4 Intake documents the future real-capture handoff.

Architectural decisions covered in docs/adr/0002-event-camera-sensor.md. PRD: local event-camera PRD.

Overview

Configuration

Construct an EventCameraCfg to describe the synthetic sensor configuration. BaseEnvCfg materializes it into the active Isaac scene as a native RGB camera under the same logical name, keeps the Isaac-free config in a scene-side binding, and the runtime registers the matching event observations.

from srb.core.sensor.event_camera import EventCameraCfg

cfg = EventCameraCfg(
    prim_path="/World/robot/ee/event_cam",
    resolution=(1280, 720),       # IMX636 native
    sub_render_hz=250.0,          # honored from S3 (#19) onward
    preset_name="imx636_nominal", # see "Presets" below
    backend="v2e",                # canonical default
    representation="voxel_grid",  # see "Representations" below
    rng_seed=None,                # None ⇒ derived from episode × env_id
)

Key fields:

Presets

Presets capture IMX636 photometric + noise parameters. Built-ins ship in srb.core.sensor.event_camera.preset:

Register a custom preset:

from srb.core.sensor.event_camera.preset import (
    IMX636Preset, register_preset,
)

register_preset(IMX636Preset(
    name="excavation_dust",
    pos_thres=0.18, neg_thres=0.18, sigma_thres=0.04,
    cutoff_hz=40.0, leak_rate_hz=0.15,
    shot_noise_rate_hz=12.0, refractory_period_s=5e-4,
    hot_pixel_frac=2e-4,
))

S15 status

S15 (#32) deliverable so far is the invariant framework, not an absolute recalibration. Tests in tests/unit/test_event_camera_preset.py + tests/unit/test_event_camera_preset_distinguishability.py pin:

1. The physical-ordering direction of each noise/threshold field between imx636_outdoor_sunlit → imx636_nominal → imx636_low_light. The direction is illumination-physics — a retune cannot silently invert it.
2. That the three presets produce measurably different event streams under the synth-vs-synth statistical scorecard. A retune that collapses them into one is a regression even if individual envelopes still pass.
3. That refractory_period_s and hot_pixel_frac stay constant across illumination presets (hardware-driven, not illumination-driven).

Absolute-value calibration against real EVK4 captures is gated on:

• S16 (#25) shipping real captures via srb dataset capture-real-evk4.
• The SuperSloMo mirror landing per Weight Mirrors so the v2e backend can run end-to-end.

Until both land, retunes should preserve the physical ordering and the distinguishability invariants — those are the contracts other code (and the fidelity scorecards) rely on.

Backends, extras, licenses

Backends register against the EventGenerator interface and are selected by cfg.backend = "<name>". Each non-trivial backend ships as an opt-in extra.

S9 status (env-runtime batched dispatch)

EventBackendDispatch (constructed by EventRecorder) checks factory.supports_batching at construction. For supports_batching=False backends (noop, v2e) it keeps the per-env shape — one EventGenerator instance per (env_id, sensor) slot in _per_env, looped once per env per sub-tick. For supports_batching=True backends (metavision, v2ce) it constructs one shared adapter instance per sensor in _shared, calls process_batch(rgb_batch, t_ns) once per sub-tick, and routes each per-env event tensor back into its per-env buffer. The shared adapter is still responsible for per-env temporal state; Metavision and V2CE lazily allocate one simulator/model slot per env inside that adapter. Episode boundaries call reset_env(env_id, seed=…, episode_id=…) on the shared backend so adapters can zero only the affected slice. Batched backends MUST override reset_env (the dispatch raises ValueError otherwise); only per-env (supports_batching=False) backends fall back to the base-class default, which forwards to reset() and globally reseeds. Per-env buffers, manifest rows, and HDF5 file layout are identical across both paths — only the dispatch shape changes.

AGPL forbidden. AGPL-licensed event repositories (notably ev-ultralytics) MUST NOT appear in any SRB extra or be vendored into the tree. See docs/adr/0002-event-camera-sensor.md §4.

Installing an extra:

pip install 'srb[event-v2e]'

If you select a backend whose extra is absent, the sensor raises MissingExtraError with the exact pip install invocation.

S19 status

S19 (#33) now has two release-smoke layers. tests/unit/test_release_smoke_extras.py is the always-on Phase-A invariant: every require_extra("X") call site under srb/ must have a matching X entry in [project.optional-dependencies]. Without the gate, a documented pip install srb[event-reconstruction] can silently fail with pip: warning: no such extra, and MissingExtraError then advertises an install hint that pip cannot satisfy.

tests/integration/test_event_camera_phase_b_smoke.py is the opt-in Phase-B route. It runs only with SRB_EVENT_CAMERA_PHASE_B_SMOKE=1; set SRB_EVENT_CAMERA_FETCH=1 to fetch from the registered private HF mirrors, or point it at local files with SRB_EVENT_CAMERA_FIXTURE_H5 and SRB_EVENT_CAMERA_E2VID_CHECKPOINT. The smoke validates the private hf-andrejorsula-active-marker-gen4 fixture as EVENT_HDF5, verifies the private hf-andrejorsula-e2vid and hf-andrejorsula-v2ce checkpoint hashes, and executes tiny E2VID reconstruction and V2CE backend forwards through the optional providers.

This is still not the downstream benchmark. The private active_marker mirror is enough for fixture fetch/import/schema and E2VID provider smoke, but the downstream-eval probe still needs a labelled/task-specific Gen4 target, which can come from the physical EVK4 dataset you collect later.

Representations

The observation builder turns events into a fixed-shape tensor consumable by an RL policy. Selected via cfg.representation:

B is the number of envs; T = cfg.time_bins; H/W are the effective resolution (after ROI + downsample). See S4 (#17) for the implementation.

Render-rate guidance

cfg.sub_render_hz is decoupled from the env step rate. Recommended floors:

By default v2e runs with SuperSloMo upsampling disabled (use_interp=False). The cfg.frame_interp field is currently not wired into the v2e backend (make_backend does not forward it), so setting it has no effect; enabling SuperSloMo requires constructing the backend with use_interp=True and a SloMo model path.

Recording during rollouts

Every srb agent subcommand (zero, rand, teleop, ros, train, eval, collect) accepts --record-events. The option, provenance fields, scene materialization, observation delivery, and recorder/storage plumbing are all wired: BaseEnvCfg materializes the EventCameraCfg into the task scene, DirectEnv binds it, and the recorder publishes per env-episode-sensor episodes. What is not established is provider-backed fidelity — no concrete backend (v2e, Metavision, V2CE) ships in the default container image, and the only end-to-end Isaac evidence is a focused local runner using a strict test backend. No event-camera fidelity differential pass is claimed.

The command shape is therefore live but provider-gated:

srb agent eval --env landing --algo skrl_ppo --record-events

For the available synthetic (non-Isaac) path, srb dataset record-events remains live. Its EVENT_HDF5 outputs can use the live storage, validation, and export tooling described below.

Dataset layout

The live synthetic srb dataset record-events path produces a timestamped output directory under record_dir:

20260521T140312/
├── events_manifest.jsonl
├── events_env0_ep0/
│   └── events_event_cam_env0_ep0.h5
├── events_env0_ep1/
│   └── events_event_cam_env0_ep1.h5
├── events_env1_ep0/
│   └── events_event_cam_env1_ep0.h5
├── events_env1_ep1/
│   └── events_event_cam_env1_ep1.h5
└── ...

Each manifest event_hdf5_path is relative to this timestamped run directory and points into the nested per-environment episode publication directory. rgb_mp4_path remains a flat filename relative to the same run directory.

The Isaac srb agent --record-events surface writes the same layout once a concrete backend is installed. The layout above does not prove paired RGB output from an Isaac rollout; an rgb_mp4_path is retained as an optional provenance field for the video recorder.

HDF5 layout

/events/t    int64    ns           (N,)   chunked, compressed
/events/x    uint16                (N,)
/events/y    uint16                (N,)
/events/p    int8     ±1            (N,)
/metadata    group    attrs:
                       schema_version, sensor_preset, resolution,
                       sub_render_hz, backend_name, backend_version,
                       backend_commit_sha, representation, time_bins,
                       episode_id, policy_id, policy_checkpoint_hash,
                       task_id, seed, srb_git_sha, isaaclab_version,
                       rgb_mp4_path (optional), success_flag, reward_summary,
                       compression, event_count, dropped_batches,
                       hot_pixel_mask, hf_repo_ids   (JSON-encoded;
                       ADR-0002 Amendment 1)

Schema version is 1 (frozen for v1). Breaking changes bump EVENT_HDF5_VERSION and trigger a converter + republish.

Manifest JSONL

One row per (env_id, episode_id, sensor_name):

{
  "sensor_name": "event_cam",
  "env_id": 0,
  "episode_id": 0,
  "task_id": "landing",
  "seed": 42,
  "event_hdf5_path": "events_env0_ep0/events_event_cam_env0_ep0.h5",
  "rgb_mp4_path": "",
  "event_count": 184321,
  "success_flag": 1,
  "reward_summary": "{\"total\": 2.5}",
  "srb_git_sha": "deadbeef",
  "isaaclab_version": "2.x.y",
  "backend_name": "v2e",
  "backend_version": "1.5.0",
  "backend_commit_sha": "",
  "sensor_preset": "imx636_nominal",
  "resolution": [1280, 720],
  "representation": "voxel_grid",
  "time_bins": 5,
  "sub_render_hz": 250.0,
  "policy_id": "ppo_landing_v3",
  "policy_checkpoint_hash": "abc123…"
}

The manifest row and the HDF5 /metadata group share a single source-of-truth struct (EventEpisodeProvenance); manifest rows additionally carry the cross-episode routing fields (sensor_name, env_id, event_hdf5_path, event_count). Both also carry the ADR-0002 Amendment 1 provenance fields hot_pixel_mask and hf_repo_ids (JSON-encoded in the HDF5 attrs).

The provenance fields (srb_git_sha, isaaclab_version, backend_version, policy_checkpoint_hash, and seed) make a configured synthetic run auditable and provide inputs for a future replay. They are not, by themselves, a deterministic-replay proof, and they do not establish Isaac task delivery or real-data fidelity.

Reading events

For tests and small analyses:

from srb.core.sensor.event_camera import read_event_hdf5

ev = read_event_hdf5("events_env0_ep0/events_event_cam_env0_ep0.h5")
ev["t"], ev["x"], ev["y"], ev["p"]      # numpy arrays
ev["metadata"]                           # dict of attrs

For streaming over large datasets, open with h5py directly; the file is opened SWMR-aware so you can attach while the writer is still active.

Runtime producer bounds

event_chunk_rows bounds every chunk crossing the backend/dispatch seam; observation_window_events separately bounds events retained for the current online observation. With a writer sink, dispatch hands off one validated chunk synchronously before pulling the next, so sink backpressure reaches a genuine provider iterator without accumulating a dispatch result mapping. Batched iterators must yield exactly one entry per environment. Provider iterator, shape, cardinality, sink, or size failures fail the affected environment and abort its staged writer.

Streaming yields must make progress. A per-env iterator cannot yield an empty chunk; a batched yield may contain idle environments but must contain events for at least one active environment. A provider represents an empty interval by ending that iterator. Otherwise dispatch raises event_camera.stream_no_progress, preventing an empty-yield loop from hanging outside every row and writer bound.

strict_streaming=True rejects a backend before construction unless it both declares streaming support and overrides the iterator dispatch will call (iter_frame for per-env backends, iter_batch for batched backends). The current v2e, Metavision, and V2CE upstream APIs return complete event arrays/lists, so their adapters deliberately remain non-streaming. Splitting those results after materialization would bound SRB retention, not upstream allocation, and is not accepted as strict streaming. A future capability flip requires a provider iterator/callback that directly yields bounded, chronologically ordered chunks without first creating the complete result.

Async writer

By default EventHDF5Writer writes asynchronously on a daemon thread to keep the env step non-blocking:

• Bounded queue.Queue(maxsize=256) between the env thread and the writer.
• On saturation: oldest batch dropped, dropped_batches incremented, a WARNING logged. The final drop count is persisted under /metadata/dropped_batches.
• close() enqueues a sentinel and joins the thread, draining everything still queued.

Force synchronous mode (tests, deterministic CI):

from srb.core.sensor.event_camera import EventHDF5Writer

with EventHDF5Writer(path, metadata, async_writes=False) as w:
    w.append(events)

Validation

Three on-demand validation modules (run via CLI, not in the inner loop):

Validation runs against explicit EVENT_HDF5 inputs. Reference captures are fetched by URL or registered fixture mirror; SRB does not commit capture data to git.

Reconstruction-fidelity (S13):

srb dataset validate-events reconstruction \
  --events events.h5 \
  --rgb-npy frames.npy \
  --times-npy frame_times_ns.npy \
  --output recon_card.json \
  --operator baseline      # 'baseline' (pure-numpy) or 'e2vid' (gated)
  # --lpips                  # optional, requires srb[event-reconstruction]

The baseline operator bins events between successive RGB frame timestamps and min-max normalises the per-pixel polarity sum — pure numpy, always available. The e2vid operator is gated on srb[event-reconstruction]. A private E2VID checkpoint mirror is registered; fetch it with srb event-camera fetch-weights --mirror hf-andrejorsula-e2vid before using the operator. SRB uses the MIT e2vid provider from uzh-rpg/e2calib and does not vendor GPL rpg_e2vid code.

Downstream-fidelity (S14):

srb dataset validate-events downstream \
  --synth-events events.h5 \
  --real-events real_gen4.h5 \
  --task event_flow         # or 'detection'
  --output downstream_card.json
  # --synthetic-smoke       # no real data; adapter-contract smoke only
  # --write-label-template  # create a valid sidecar skeleton for labelling
  # --check-real-target     # validate physical EVK4 labels before harness use

S14 ships as a protocol stub today: the default CLI prints the requirements for a paper-ready downstream-fidelity probe (model family, metrics, gen-gap reporting convention) and writes a schema-stable DownstreamScoreCard with protocol_status="stub". --synthetic-smoke is the no-real-data adapter path: it validates the synthetic EVENT_HDF5 and writes protocol_status="synthetic_smoke" with synthetic contract facts only, not fidelity metrics. The actual train-on-synth + eval-on-real harness lands once a labelled/task-specific real Gen4 target and model adapter are available — the JSON shape will be the same so consumers do not need to rewrite their parsing logic.

The strict real-target contract is live and can be run before the model harness exists:

• --write-label-template creates the expected sidecar beside --real-events. The generated file is a valid contract skeleton, not a finished annotation.
• event_flow: real_gen4.h5 must be canonical EVENT_HDF5 and have an adjacent real_gen4.flow.npz with flow_xy shaped (T,H,W,2), valid_mask shaped (T,H,W), and integer timestamps_ns shaped (T,).
• detection: real_gen4.h5 must be canonical EVENT_HDF5 and have an adjacent real_gen4.detections.jsonl; each record has integer timestamp_ns, boxes_xyxy as 4-coordinate boxes, and matching class_ids.

SuperSloMo weights

v2e’s temporal upsampler needs a SuperSloMo checkpoint. SRB does not redistribute this checkpoint in-repo. The registered Hugging Face mirror is private until release; collaborators need HF repo access plus HF_TOKEN or hf auth login. Two paths exist today:

# Recommended for collaborators with HF access:
srb event-camera list-mirrors                       # show registered mirrors
srb event-camera fetch-weights --mirror hf-andrejorsula-supersloMo

# Fallback for other private mirrors or one-off downloads:
srb event-camera fetch-weights \
  --url https://example.com/SuperSloMo39.ckpt \
  --sha256 <hex digest> \
  --name SuperSloMo39.ckpt

The fetcher downloads to ${XDG_CACHE_HOME}/srb/event_weights/ and verifies the downloaded bytes against the supplied SHA256. Mismatches abort with a non-zero exit so corrupted / substituted weights can never silently produce events.

Private mirrors registered. The default artifact is still a guarded placeholder; srb event-camera fetch-weights with no --mirror / --url refuses to run. See Weight mirrors for the mirror names, hashes, and publishing contract.

Publishing datasets

Two complementary bundlers wrap an EVENT_HDF5 run directory into a publish-ready layout:

# LeRobot v3 (HuggingFace Hub / behaviour-cloning pipelines) — S17 / #26
srb dataset export-lerobot-v3 <event_hdf5_dir> --output ./bundles/lerobot

# Zenodo archival deposit — S18 / #27
srb dataset export-zenodo-bundle <event_hdf5_dir> --output ./bundles/zenodo.tar.gz

The LeRobot v3 bundle materialises an SRB-extended event_camera modality because LeRobot has no canonical event-camera schema. The on-disk layout:

bundle/
├── meta/
│   ├── info.json            # codebase_version v3.0 + srb_event_camera schema
│   ├── stats.json
│   ├── tasks.parquet
│   └── episodes/chunk-000/file-000.parquet  # one row per (env, episode, sensor)
├── data/chunk-000/file-000.parquet          # one row per episode, event pointers
└── events/<sensor_name>/chunk-000/episode-{n:06d}.h5  # canonical EVENT_HDF5

info.json adds an event_path template alongside LeRobot’s standard data_path / video_path, plus an srb_event_camera block with the schema version, registered sensors, presets, backends, and resolutions present in the bundle. The episode parquet carries the provenance fields from the source manifest (srb_git_sha, policy_checkpoint_hash, backend_version, seed, sub_render_hz, etc.) so downstream tooling can filter without re-reading HDF5 metadata; these fields are not a deterministic-replay proof.

fps is set to the max sub_render_hz across rows — for mixed-task bundles (e.g. landing 1000 Hz + traversal 250 Hz) the dataset’s declared temporal resolution matches the highest rate present.

RAW EVT3 import

Existing Prophesee RAW EVT3 recordings can be normalized into the canonical EVENT_HDF5 schema without Metavision/OpenEB:

srb dataset import-raw-evt3 active_marker.raw \
  --output active_marker.h5 \
  --task active_marker \
  --policy-id prophesee_source

The importer decodes EVT3 address and vector CD events, unwraps the 24-bit EVT3 timestamp, writes nanosecond t values, and preserves the source resolution in /metadata/resolution. Trigger, marker, and continued words are ignored because they are not CD events.

Forward compatibility

Real EVK4 capture (S16 #25) remains gated. The future hardware path is planned to target the same StorageFormat.EVENT_HDF5 schema so downstream consumers can share one layout, but no real capture or hardware fidelity is available today.

A CLI stub reserves the on-disk layout today:

srb dataset capture-real-evk4 \
  --task landing \
  --policy /path/to/policy.ckpt \
  --episodes 8 \
  --output ./captures/evk4 \
  --preset imx636_nominal \
  --resolution 1280x720

The stub prints a 6-step trajectory-replay protocol (mount EVK4, calibrate intrinsics against the synthetic preset, configure Metavision streaming, replay policy deterministically, append manifest rows, validate with validate-events stats) and writes a skeleton events_manifest.jsonl under a timestamped subdirectory. Real hardware capture lands when the Metavision driver is wired; the layout it lands into is the one this stub reserves.

References

• ADR: docs/adr/0002-event-camera-sensor.md (in the repository, not in this book)
• PRD: local event-camera PRD
• Module: srb/core/sensor/event_camera/
• Tests: tests/unit/test_event_camera_*.py
• v2e — Hu et al., “v2e: From Video Frames to Realistic DVS Events”.
• Prophesee IMX636 / EVK4 product page and datasheet.