srb dataset — Manage Dataset Artifacts

srb dataset inspects, validates, converts, previews, and publishes SRB JSONL, robomimic HDF5, LeRobot, and real-validation datasets.

Usage

srb dataset <action> [options]

Actions

Event-camera actions: record-events, validate-events, capture-real-evk4, export-lerobot-v3, export-zenodo-bundle, export-raw-evt3, import-raw-evt3, and export-rosbag. Validation checks are stats, reconstruction, and downstream; unsupported paths fail in preflight. See the event-camera status matrix.

Replay backends are summary and rerun (--backend summary and --backend rerun).

Algorithm Compatibility

compatibility DATASET --purpose train --algo ALGO reports implemented training input (training_dataset_mode: offline, replay_prefill, or unsupported). Collection, evaluation, and publication support do not imply demonstration training support. Unsupported focused reports return compatible=false and exit 1; omit --algo for the matrix. Compatibility checks manifest metadata and format; the trainer validates payload and action and observation semantics before use.

Publication Format

Canonical srb_jsonl is preferred for publication. lerobot publication needs its optional dependencies. Export robomimic HDF5 before publication.

LeRobot v3 export requires a finite positive nominal fps in the source metadata. Missing timing is an error; SRB does not infer a frame rate. For image-heavy datasets, export to srb_jsonl while LeRobot export remains slow.

Export and publication validate a private snapshot, leaving caller data unchanged. push --dry-run previews the plan but does not validate every payload or guarantee upload success.

For downloads, pull --output requires an absent or empty directory; SRB does not merge downloaded revisions into an existing dataset.

Input Limits

• JSONL: each line is limited to 16 MiB.
• HDF5: at most 100,000 storage entries and 16 GiB per input file.

Action Metadata

Actions are flat vectors. When the source exposes grouped actions, SRB records ordered names, paths, shapes, dimensions, and offsets in metadata.action_schema. info surfaces action_dim, action_components, and action_schema; export preserves the schema across supported formats, and validation checks it against metadata.action_dim. LeRobot writes srb_action_schema to meta/info.json; deep validation checks it against the manifest. Dataset cards include a schema table, and replay displays it in the summary backend or per-component values under actions/<component> in Rerun.

Count validation reconciles declared episode, sample, and action counts with physical JSONL, HDF5, or Parquet lengths. Missing h5py or pandas Parquet readers fail validation explicitly; unchecked data is never reported valid.

Demonstration I/O Metadata

New robomimic collection runs record metadata.demo_io_contract version srb.demo_io.v1, describing the observation projection and, when available, ordered action components, dimensions, bounds, and runtime scales. Export preserves it in the canonical manifest. Observation rows use contract shapes; simulated depth uses uint8. The collection output page gives the exact layout and its v0.1.0 limits. HDF5 collected before 2026-10-01 kept a leading single-environment axis, (T, 1, *shape); validation accepts it and exports from HDF5 drop it, so re-exporting migrates such a dataset (ADR-0035).

Deep validation checks observation shapes, compatible dtypes and value ranges; obs and next_obs leaves must agree. HDF5’s data attribute and LeRobot’s srb_demo_io_contract must match the manifest; LeRobot must also include the implied srb_visual_views and matching feature shapes. The contract records action smoothing and observation stacking under env_wrappers; stored actions are policy commands before smoothing.

Certified rows must contain explicit next_obs, terminated and truncated values. Every nested observation leaf and shape must match the explicit next observation; certified rows never use legacy fallback.

Certified LeRobot v3 exports carry the SRB transition extension srb.lerobot.transitions.v1. Each action row includes paired observation.* and next_observation.* features, including the actual terminal endpoint. The manifest and meta/info.json contain the same srb_transition_extension path mapping. Component-wise percent encoding distinguishes literal dots, slashes and percent characters from nested paths. Strict validation checks this mapping, paired shapes, physical counts and the three Boolean flags; SRB replay reconstructs the original nested observations.

The LeRobot 0.4 scalar convention stores one-element features as scalars. For certified exports, manifest obs_shapes retains each original observation shape, including [] versus [1], so SRB replay restores that distinction. LeRobot feature ranks above five reject before publication. Stock-consumer compatibility is specific to the pinned LeRobot 0.4 reader; the extra endpoint columns do not make every upstream policy consume demonstration transitions.

Uncertified exports keep their existing fields. Markerless LeRobot replay does not invent a next observation or missing termination flags. Legacy datasets remain usable through their existing readers, but conversion cannot establish a missing demonstration contract or recover termination flags that were never recorded. A strict replay consumer checks this metadata against its live wrapper before inserting transitions.

Examples

# Validate a canonical dataset thoroughly before publishing
srb dataset validate /data/srb_excavation --deep --strict

# Convert a robomimic dataset to canonical SRB JSONL
srb dataset export /logs/excavation/collect/datasets/demos.hdf5 \
  --output /data/srb_excavation_policy --format srb_jsonl

# Preview a dataset with Rerun and save the replay
srb dataset replay /data/srb_excavation_policy \
  --backend rerun --save /tmp/srb_excavation.rrd

# Inspect upload contents without network access
srb dataset push /data/srb_excavation_policy --dry-run

# Resolve a Hub repo from artifact identity and inspect the plan
srb dataset pull --task excavation --embodiment kinova_gen3 \
  --action-mode smooth_osc --dry-run

# Download an SRB dataset artifact by repo-id
srb dataset pull --repo-id space_robotics_bench/srb_excavation_kinova_gen3_smooth_osc \
  --output /data/srb_excavation_policy

# Turn a real validation session into a dataset
srb dataset from-validation logs/real_validation/excavation/<session> \
  --output /data/srb_excavation_real \
  --embodiment kinova_gen3 --action-mode smooth_osc

from-validation keeps real-world data in split=real. By default, sessions with a validation algorithm or checkpoint are labeled controller=policy. Supported controller labels are human, policy, random, zero, and real.