Event Camera (IMX636)

SRB provides a standalone synthetic event camera modeled on the Prophesee IMX636 / EVK4. Each pixel reports brightness changes asynchronously with microsecond timestamps. The sensor owns its pose, intrinsics, renderer, preset, and per-environment RNG; it is not a Camera subclass. See the Status Matrix for rollout readiness and Physical EVK4 Intake for the real-capture handoff.

Overview

Synthetic IMX636; defaults: v2e, imx636_nominal, StorageFormat.EVENT_HDF5. One tick per rendered frame, capped at 250 Hz; interpolation rejected; bounded async writes, HDF5 SWMR reads, Blosc:zstd-3 compression (gzip-6 fallback).

Configuration

Construct an EventCameraCfg to describe the synthetic sensor configuration. BaseEnvCfg materializes it into the active Isaac scene as a native RGB camera under the same logical name, keeps the Isaac-free config in a scene-side binding, and the runtime registers the matching event observations.

from srb.core.sensor.event_camera import EventCameraCfg
cfg = EventCameraCfg(
    prim_path="/World/robot/ee/event_cam", resolution=(1280, 720),
    sub_render_hz=250.0, preset_name="imx636_nominal", backend="v2e",
    representation="voxel_grid", rng_seed=None,
)

Key fields: resolution: (W, H) is pre-ROI/downsample (native (1280, 720)); roi_crop: (x, y, w, h)? crops before the backend and affects observations and saved events; downsample: int applies after ROI. preset_name copies its IMX636Preset threshold/noise values, with per-field overrides winning. sub_render_hz defaults to 250: each new frame is sent once at simulation time and events span the interval since that sensor’s previous frame; frames sooner than 1 / sub_render_hz are skipped. The cap does not raise render rate. frame_interp=True is rejected for pinned v2e; slomo_model_path is reserved and unused while interpolation is unsupported. representation is voxel_grid, histogram, time_surface, stacked, or raw; time_bins defaults to 5. event_chunk_rows defaults to 65,536 and limits transport chunks; it cannot exceed provider_workspace_rows or 1,000,000, and dispatch slices larger one-shot results. The only per-step memory bound, observation_window_events, defaults to None, resolving at use to 4·W·H (3,686,400 events / about 118 MB per env and sensor at 1280x720); larger steps fail closed. provider_workspace_rows defaults to 1,048,576 and separately bounds each provider timestamp/event/permutation workspace. backend selects the registered EventGenerator. Torch providers (v2e, V2CE) use backend_device=None to select cuda when available, else cpu; V2CE requires a fetched checkpoint at v2ce_model_path. hot_pixel_rate_hz defaults to 200 per masked pixel: each tick adds a Poisson count of at most 64 ON events and records the rate. The first post-reset frame is only a reference. rng_seed seeds per-reset backends from the base seed, env index, and episode id; None uses env.cfg.seed via torch.initial_seed(). Backends never touch process-global random generators.

Presets

Presets capture IMX636 photometric + noise parameters. Built-ins ship in srb.core.sensor.event_camera.preset:

Register a custom preset:

from srb.core.sensor.event_camera.preset import (
    IMX636Preset, register_preset,
)
register_preset(IMX636Preset(
    name="excavation_dust",
    pos_thres=0.18, neg_thres=0.18, sigma_thres=0.04,
    cutoff_hz=40.0, leak_rate_hz=0.15,
    shot_noise_rate_hz=12.0, refractory_period_s=5e-4,
    hot_pixel_frac=2e-4,
))

Backends, extras, licenses

Backends register against the EventGenerator interface and are selected by cfg.backend = "<name>". Each non-trivial backend ships as an opt-in extra.

AGPL forbidden. AGPL-licensed event repositories (notably ev-ultralytics) MUST NOT appear in any SRB extra or be vendored into the tree.

Installing an extra:

pip install 'srb[event-v2e]' 'v2e @ git+https://github.com/SensorsINI/v2e.git@cac638444557a224ba6ee5c29260916f152c00bd'

v2e (and E2VID for srb[event-reconstruction]) is pinned to a Git revision. PyPI rejects direct references in published metadata, so the pinned source is not part of the extra; install it next to the extra as above. From a checkout, the locked equivalent is uv sync --all-extras --group source-event-v2e (the source-event-reconstruction group carries E2VID, and vcs-sources selects every pinned source). If you select a backend whose extra or pinned source is absent, the sensor raises MissingExtraError with the exact pip install invocation. Release-smoke coverage and its limits are summarized in the Status Matrix.

Representations

cfg.representation selects a fixed-shape policy tensor. B is env count, T = cfg.time_bins, and H/W are effective resolution after ROI/downsample:

Render-rate guidance

The effective rate is the lower of render rate and cfg.sub_render_hz; raising the cap alone does not raise rendering, and long intervals lose intermediate motion. Manifest/HDF5 record measured sub_render_hz (the cap until two frames are consumed). Recommended render-rate floors:

Recording during rollouts

Every srb agent subcommand accepts --record-events. BaseEnvCfg materializes the EventCameraCfg into the task scene. For supported-provider requirements and recording evidence, see the Status Matrix.

srb agent eval --env landing --algo skrl_ppo --record-events

The Isaac-free srb dataset record-events command writes a moving-patch fixture, not task or real-sensor events. It always persists task_id="fixture_moving_patch", policy_id="synthetic_fixture", and an empty isaaclab_version, plus the detected SRB revision. --task and --policy-id remain compatibility inputs: the summary retains the requested task and reports source: "synthetic_fixture", but persisted provenance uses the reserved values. Only the exact task marker identifies a fixture; missing metadata, historical rows without the marker, and backend names alone do not. Fixtures may be exported to LeRobot, Zenodo, and NPZ for synthetic testing; exports and validation retain their identity, and LeRobot/Zenodo set contains_synthetic_fixture if any selected episode is a fixture. This is not task or real-data fidelity evidence. Use --seed INT to seed the synthetic environment and camera; omitted, the process run seed is used, with per-environment and per-episode seeds derived and recorded:

srb dataset record-events --output events --backend noop --seed 42

Dataset layout

The live synthetic srb dataset record-events path creates a timestamped record_dir containing events_manifest.jsonl and one nested HDF5 publication directory per environment and episode, for example:

20260521T140312/events_env0_ep0/events_event_cam_env0_ep0.h5
20260521T140312/events_env0_ep1/events_event_cam_env0_ep1.h5
20260521T140312/events_env1_ep0/events_event_cam_env1_ep0.h5

Each manifest event_hdf5_path is relative to the run directory and points into that nested directory; rgb_mp4_path is a flat filename relative to the same directory. Isaac srb agent --record-events uses this layout when a concrete backend is installed. It does not prove paired RGB output; the optional rgb_mp4_path is provenance for the video recorder.

HDF5 layout

The frozen v1 EVENT_HDF5 schema stores chunked, compressed /events/t (int64 nanoseconds), /events/x and /events/y (uint16), and /events/p (int8, ±1), each shaped (N,). /metadata attributes are schema_version, sensor_preset, resolution, sub_render_hz, backend_name, backend_version, backend_commit_sha, representation, time_bins, episode_id, policy_id, policy_checkpoint_hash, task_id, seed, srb_git_sha, isaaclab_version, optional rgb_mp4_path, success_flag, reward_summary, compression, event_count, dropped_batches, hot_pixel_mask, JSON-encoded hf_repo_ids, and optional hot_pixel_rate_hz. In files written before 2026-09-25, an absent hot_pixel_rate_hz means unknown. Breaking changes bump EVENT_HDF5_VERSION and require conversion and republishing. Before writing any HDF5 attribute, the writer checks metadata attribute count, UTF-8/JSON byte sizes, and cumulative storage; an over-limit snapshot aborts its private staging file.

Manifest JSONL

Each events_manifest.jsonl row represents one (env_id, episode_id, sensor_name) and contains sensor_name, env_id, episode_id, task_id, seed, event_hdf5_path, rgb_mp4_path, event_count, success_flag, reward_summary, srb_git_sha, isaaclab_version, backend_name, backend_version, backend_commit_sha, sensor_preset, resolution, representation, time_bins, sub_render_hz, policy_id, policy_checkpoint_hash, hot_pixel_mask, and hf_repo_ids. HDF5 stores hf_repo_ids and the mask as JSON-encoded metadata. EventEpisodeProvenance is the shared source for manifest and HDF5 provenance; manifest rows also add routing fields. srb_git_sha, isaaclab_version, backend_version, policy_checkpoint_hash, and seed support audit and future replay; alone they prove neither deterministic replay, Isaac task delivery, nor real-data fidelity. Directory checkpoints (Dreamer) hash a sorted manifest of member paths, sizes, and file hashes; symlinks are recorded, not followed. Empty or unreadable directories yield an empty hash and warning. Random and human collection record the selected controller as policy_id. Strict validation allows an empty checkpoint hash only for random, human, zero, rand, teleop, ros, and synthetic_fixture; unknown and learned policies require one. Missing SRB or Isaac Lab provenance fails strict validation. In a strict source image, srb_git_sha comes from the verified build manifest beside the package; in a checkout it comes from Git. It identifies build source but does not attest runtime files are unchanged. isaaclab_version is the installed package version, which may differ from the separately pinned source version.

Reading events

For tests and small analyses, read_event_hdf5 returns NumPy arrays and metadata, after checking row count, decoded column sizes, and cumulative bytes. Over-limit full reads fail explicitly without truncation; use the bounded iterator for larger files:

from srb.core.sensor.event_camera import read_event_hdf5
from srb.core.sensor.event_camera.storage import iter_event_hdf5
ev = read_event_hdf5("events_env0_ep0/events_event_cam_env0_ep0.h5")
for chunk in iter_event_hdf5("events_env0_ep0/events_event_cam_env0_ep0.h5", 4096):
    consume(chunk["t"], chunk["x"], chunk["y"], chunk["p"])

The full reader provides ev["t"], ev["x"], ev["y"], ev["p"] and ev["metadata"]. Each iterator chunk is bounded by the requested rows and repository event-batch ceiling; it validates final count and digest without assembling the entire file in memory.

Bounded streaming

event_chunk_rows, observation_window_events, and provider_workspace_rows bound transport chunks, per-step observations, and provider workspaces. Invalid or oversized chunks and writer failures abort the episode. strict_streaming=True accepts only streaming backends; legacy v2e, Metavision, and V2CE materialize complete results. A streaming iterator may yield no chunks for an empty interval; an empty chunk is invalid and fails the stream. The optional v2e_streaming profile uses a recorded local patch. Install the pinned base, then apply and verify the patch:

python -m srb.core.sensor.event_camera.backends.v2e_install --apply
python -m srb.core.sensor.event_camera.backends.v2e_install --check

Docker builds can apply it with --build-arg SRB_EVENT_V2E_STREAMING=true. Set shot_noise_rate_hz=0; construction and reset verify the patch and required disabled noise options.

Async writer

EventHDF5Writer defaults to a daemon thread and bounded queue.Queue(maxsize=256) to keep env steps non-blocking. Saturation raises EventHDF5BackpressureError and aborts the staged episode; published datasets never silently drop batches. close() queues a sentinel, joins, and drains. Use synchronous mode for tests and deterministic CI:

from srb.core.sensor.event_camera import EventHDF5Writer
with EventHDF5Writer(path, metadata, async_writes=False) as w:
    w.append(events)

Validation

Run validation on explicit EVENT_HDF5 inputs. Reference captures come from a URL or registered fixture mirror; SRB does not commit captures.

stats requires matching native resolutions unless --resolution supplies missing metadata. It rejects out-of-grid events and compares rate, polarity, ISI, hot-pixel, and sparsity metrics. per_pixel_rate_emd is the 1-D Wasserstein distance between per-pixel counts divided by each stream’s duration; fewer than two timestamps or non-positive duration makes the rate undefined (null in JSON). Reconstruction:

srb dataset validate-events reconstruction --events events.h5 \
  --rgb-npy frames.npy --times-npy frame_times_ns.npy \
  --output recon_card.json \
  --operator baseline      # 'baseline' (pure-numpy) or 'e2vid' (gated)
  # --lpips                  # optional, requires srb[event-reconstruction]

baseline bins events between successive RGB frame timestamps and min-max normalises per-pixel polarity sums (pure NumPy). e2vid requires srb[event-reconstruction] and the private checkpoint fetched with srb event-camera fetch-weights --mirror hf-andrejorsula-e2vid. SRB uses the MIT e2vid provider from uzh-rpg/e2calib, not GPL rpg_e2vid code. Fixture inputs warn and retain their identity in scorecards.

RGB frame dimensions must match the event resolution. Non-finite RGB, reconstruction or score values are rejected, so scorecards never contain NaN or infinity.

Downstream validation

srb dataset validate-events downstream --synth-events events.h5 \
  --real-events real_gen4.h5 --task event_flow --output downstream_card.json  # or --task detection
  # --synthetic-smoke       # no real data; adapter-contract smoke only
  # --write-label-template  # sidecar skeleton for labelling (rejected in this build)
  # --check-real-target     # validate physical EVK4 labels before harness use

Only --synthetic-smoke currently runs; it records contract facts, not fidelity metrics. Other modes are rejected before output until real-data harness and model adapter exist, and smoke mode rejects --write-label-template and --check-real-target. Real-data labels follow this layout: for event_flow, the adjacent real_gen4.flow.npz must contain flow_xy (T,H,W,2), valid_mask (T,H,W), and integer timestamps_ns (T,). For detection, adjacent real_gen4.detections.jsonl records require integer timestamp_ns, 4-coordinate boxes_xyxy, and matching class_ids; both tasks require canonical EVENT_HDF5 real_gen4.h5.

SuperSloMo weights

No backend consumes this checkpoint because interpolation is unsupported. SRB does not redistribute it; its Hugging Face mirror is private and requires repository access and an HF token. Fetch options:

# Recommended for collaborators with HF access:
srb event-camera list-mirrors                       # show registered mirrors
srb event-camera fetch-weights --mirror hf-andrejorsula-supersloMo

# Fallback for other private mirrors or one-off downloads:
srb event-camera fetch-weights \
  --url https://example.com/SuperSloMo39.ckpt \
  --sha256 <hex digest> \
  --name SuperSloMo39.ckpt

Downloads go to $XDG_CACHE_HOME/srb/event_weights/, or ~/.cache/srb/event_weights/ when XDG_CACHE_HOME is unset or empty; SHA256 is checked. The default artifact is a guarded placeholder, so fetching with no --mirror or --url refuses to run. See the weight mirror contract.

Publishing datasets

Wrap an EVENT_HDF5 run directory for LeRobot v3 or Zenodo:

srb dataset export-lerobot-v3 <event_hdf5_dir> --output ./bundles/lerobot
srb dataset export-zenodo-bundle <event_hdf5_dir> --output ./bundles/zenodo.tar.gz

Both accept marked fixtures with a warning and preserve provenance. LeRobot and Zenodo set contains_synthetic_fixture if any selected episode is a fixture. RAW EVT3 and ROS bag cannot preserve the marker and reject fixture input; use ordinary task data with explicit task provenance, for example:

srb dataset import-raw-evt3 landing.raw --output landing.h5 \
  --task landing --policy-id prophesee_landing
srb dataset export-rosbag landing.h5 --output ./landing_bag

Do not export srb dataset record-events output to RAW EVT3 or ROS bag: it always carries the reserved fixture marker, regardless of compatibility --task and --policy-id values. LeRobot v3 uses an SRB event_camera modality because LeRobot has no canonical event-camera schema. The bundle contains meta/info.json (codebase_version v3.0 and srb_event_camera schema), meta/stats.json, meta/tasks.parquet, meta/episodes/chunk-000/file-000.parquet (one row per env/episode/sensor), data/chunk-000/file-000.parquet (one row per episode with event pointers), and events/<sensor_name>/chunk-000/episode-{n:06d}.h5 (canonical EVENT_HDF5). It records an event_path template, schema, and source provenance; fps is the maximum recorded sub_render_hz.

RAW EVT3 import

Normalize existing Prophesee RAW EVT3 without Metavision/OpenEB:

srb dataset import-raw-evt3 active_marker.raw --output active_marker.h5 \
  --task active_marker --policy-id prophesee_source

The importer decodes EVT3 address and vector CD events, unwraps timestamps to nanoseconds, and preserves source resolution. It skips and reports words before the first EVT_TIME_HIGH; non-empty files without one and malformed words fail closed. SRB export headers preserve timestamp origin on re-import. RAW EVT3 and ROS bag exports rebase time to the first event; EVT3 floors to whole microseconds. Gaps over 60 seconds are rejected.

Real capture

Real EVK4 capture is unavailable. The CLI rejects it before creating output; see the physical intake runbook. Intended command:

srb dataset capture-real-evk4 --task landing --policy /path/to/policy.ckpt \
  --episodes 8 --output ./captures/evk4 --preset imx636_nominal \
  --resolution 1280x720

See the Status Matrix for the current capture gate.

References

• Module: srb/core/sensor/event_camera/
• Tests: tests/unit/test_event_camera_*.py
• v2e — Hu et al., “v2e: From Video Frames to Realistic DVS Events”.
• Prophesee IMX636 / EVK4 product page and datasheet.