SimForge

SimForge is a framework for creating diverse virtual environments through procedural generation.

Motivation

Procedural generation is a powerful technique for creating unique scenarios with an infinite number of variations. Although the gaming industry has embraced this approach for creating diverse and realistic environments, other fields, such as robotics and space exploration, have yet to leverage its potential fully. SimForge aims to bridge this gap by providing a unified and extensible framework for generating assets and integrating them into external game engines and physics simulators.

Overview

The framework implements a modular approach with three ecosystem-agnostic concepts:

Getting Started

To get started with SimForge, please refer to:

Instructions

Detailed instructions for specific components can be found in the respective sections:

Development

Guidelines for implementing your components are available here:

Assets

Assets are the registered building blocks that range from simple images and meshes to complex articulated models. Their definitions reside in external repositories that can be shared and reused across projects.

General

Templates

Issue tracker: #1 Assets

Generators

Generators are responsible for automating the creation of Assets from their definitions in a deterministic manner. They interface with external tools and libraries to produce the desired output.

Internal

Issue tracker: #2 Generators

Generator — Blender

Blender is an extensive open-source 3D creation suite. SimForge leverages Blender's generative capabilities to create a wide range of assets via BlGenerator using its Python API (bpy).

Requirements

Python 3.11

For python_version=='3.11', is it enough to install SimForge with the bpy extra:

# Install SimForge with bpy extra
pip install simforge[bpy]

Other Environments | --subprocess

For other environments, BlGenerator can be run in a subprocess using the embedded Python interpreter. This requires a local Blender installation with the blender executable in PATH.

CLI: Generation in a subprocess can be achieved by simforge gen --subprocess.

BlGenerator

All Bl* assets are automatically registered for generation using the BlGenerator class, which is a subclass of Generator and provides a unified interface for generating Blender assets.

BlGeometry

Geometry generated by Blender is specified using the BlGeometry class, which includes a sequence of BlGeometryOp operations that create and modify the geometry of a mesh object.

The BlGeometryOp operations can use arbitrary bpy calls to manipulate the mesh object. For instance, BlGeometryNodesModifier is a subclass of BlGeometryOp that supports Blender's Geometry Nodes, which can be loaded prior to the generation from a Python file (NodeToPython). Additional typed inputs can be defined as attributes of the operation, and they will be automatically passed as the inputs of the node group.

Example:

from pathlib import Path
from typing import List, Tuple

from pydantic import PositiveFloat, PositiveInt
from simforge import (
    BlGeometry,
    BlGeometryNodesModifier,
    BlGeometryOp,
    BlMaterial,
    BlNodesFromPython,
)


class ExampleNodes(BlGeometryNodesModifier):
    nodes: BlNodesFromPython = BlNodesFromPython(
        name="ExampleNodeGroup",
        python_file=Path(__file__).parent.joinpath("example_nodes.py"),
    )

    input1: PositiveInt = 42
    input2: Tuple[PositiveFloat, PositiveFloat, PositiveFloat] = (0.1, 0.1, 0.1)
    input3: BlMaterial | None = None


class ExampleGeo(BlGeometry):
    ops: List[BlGeometryOp] = [ExampleNodes()]

BlMaterial

Materials of Blender assets are defined using the BlMaterial class, which includes BlShader that specifies the appearance of the object.

The BlShader class must utilize Shader Nodes to define the appearance of the object. Similar to BlGeometryNodesModifier, the nodes can be loaded from a Python file (NodeToPython) and additional typed inputs can be defined as attributes of the shader.

Example:

from pathlib import Path
from typing import Tuple

from pydantic import NonNegativeFloat, PositiveFloat
from simforge import BlMaterial, BlNodesFromPython, BlShader


class ExampleShader(BlShader):
    nodes: BlNodesFromPython = BlNodesFromPython(
        name="ExampleShader",
        python_file=Path(__file__).parent.joinpath("example_nodes.py"),
    )

    input1: PositiveFloat = 1.0
    input2: Tuple[
        NonNegativeFloat, NonNegativeFloat, NonNegativeFloat, NonNegativeFloat
    ] = (
        0.0,
        0.2,
        0.8,
        1.0,
    )


class ExampleMat(BlMaterial):
    shader: BlShader = ExampleShader()

BlModel

The BlModel class combines geo: BlGeometry and an optional mat: BlMaterial to define a model. Furthermore, texture_resolution attribute specifies the resolution of the baked textures for the model.

BlGeometry with BlGeometryNodesModifier operation can internally set the material of the object via BlMaterial inputs. In this case, the mat attribute can be omitted as it would otherwise override the material set by the geometry nodes modifier.

Example:

from pydantic import InstanceOf, SerializeAsAny
from simforge import BakeType, BlGeometry, BlMaterial, BlModel, TexResConfig


class ExampleModel(BlModel):
    geo: SerializeAsAny[InstanceOf[BlGeometry]] = ExampleGeo()
    mat: SerializeAsAny[InstanceOf[BlMaterial]] | None = ExampleMat()
    texture_resolution: TexResConfig = {
        BakeType.ALBEDO: 2048,
        BakeType.EMISSION: 256,
        BakeType.METALLIC: 256,
        BakeType.NORMAL: 4096,
        BakeType.ROUGHNESS: 512,
    }

New Assets

Geometry Nodes and Shader Nodes are the preferred methods for defining new Blender-based geometry and materials because they provide a high level of flexibility via artist-friendly node-based interfaces. The nodes can be exported to Python files using NodeToPython and integrated into the SimForge asset definitions, as shown in the examples above. Furthermore, automatic randomization can be exposed by including a seed attribute in the node group that will be automatically detected and updated by the generator.

If you are new to Blender, there is a vast number of free Blender tutorials and resources available online that can help you get started with creating your own assets. Consider creating your own Donut as the first step!

If you are somewhat familiar with Blender but never used Geometry Nodes, consider watching the Geometry Nodes Fundamentals that provides an excellent introduction to the topic.

If you are a seasoned Blender artist, then please teach us your ways! The contributors of SimForge are mostly non-artists with limited creativity, so we would greatly appreciate your help in improving and expanding the asset library.

Integrations

Integrations seamlessly bridge the gap between the Generators and external frameworks such as game engines and physics simulators. These modules leverage domain-specific APIs to import and configure the generated Assets.

Internal

Issue tracker: #3 Integrations

Integration — Isaac Lab

Isaac Lab is a robot learning framework built on top of Isaac Sim. SimForge integrates with Isaac Lab through SimforgeAssetCfg to configure the spawning of assets within interactive scenes.

Requirements

SimforgeAssetCfg

The SimforgeAssetCfg class is a SpawnerCfg (FileCfg) subclass that streamlines the generation and spawning of SimForge assets within Isaac Lab. The primary attributes include:

  • assets: Sequence of asset types to spawn
  • num_assets: The number of asset variants distributed among assets to generate (default: 1)
  • seed: The initial seed used to generate the first variant of assets (default: 0)
  • use_cache: Use cached assets instead of generating new ones (default: True)
  • random_choice: Randomly select variants instead of sequentially (default: False)

Example:

from omni.isaac.lab import sim as sim_utils
from omni.isaac.lab.assets import AssetBaseCfg, RigidObjectCfg
from omni.isaac.lab.scene import InteractiveSceneCfg
from omni.isaac.lab.utils import configclass
from simforge import AssetRegistry
from simforge.integrations.isaaclab import SimforgeAssetCfg
from simforge_foundry import ExampleModel


@configclass
class ExampleSceneCfg(InteractiveSceneCfg):
    num_envs: int = 64

    asset1: AssetBaseCfg = AssetBaseCfg(
        prim_path="{ENV_REGEX_NS}/asset1",
        spawn=SimforgeAssetCfg(
            assets=[
                AssetRegistry.by_name("example_geo"),
                AssetRegistry.by_name("example_model"),
            ],
            num_assets=64,
            collision_props=sim_utils.CollisionPropertiesCfg(),
        ),
    )
    asset2: RigidObjectCfg = RigidObjectCfg(
        prim_path="{ENV_REGEX_NS}/asset2",
        spawn=SimforgeAssetCfg(
            assets=[ExampleModel],
            num_assets=8,
            seed=42,
            collision_props=sim_utils.CollisionPropertiesCfg(),
            rigid_props=sim_utils.RigidBodyPropertiesCfg(),
            mass_props=sim_utils.MassPropertiesCfg(),
        ),
    )

Installation

All releases of SimForge are available at PyPI and can be installed using your preferred Python package manager:

# Install SimForge with all extras
pip install simforge[all]

Extras

SimForge specifies several optional dependencies to enhance its functionality. These can be specified as extras:

  • all - Include all other SimForge extras (recommended)
  • assets - Primary collection of SimForge assets
  • bpy - Enable Blender generator via its Python API
  • cli - Utilities for enhancing the CLI experience
  • dev - Utilities for development and testing

Multiple extras can be specified at once by separating them with commas:

# Install SimForge with assets and CLI extras
pip install simforge[assets,cli]

Docker

  • Docker Engine is required to use SimForge in a containerized environment.
  • NVIDIA Container Toolkit is recommended to automatically utilize available NVIDIA GPUs for workloads such as texture baking.

For convenience, install_docker.bash script is included to setup Docker on a Linux host.

A minimal Dockerfile is provided for convenience with all extras included. Pre-built images for every release are available on Docker Hub and GitHub Container Registry for easy access:

# Docker Hub
docker pull andrejorsula/simforge

# [ALTERNATIVE] GitHub Container Registry
docker pull ghcr.io/andrejorsula/simforge

For convenience, .docker/run.bash script is included to run the Docker container with appropriate arguments, environment variables, and volumes for persistent cache storage:

# Path to a cloned repository
simforge/.docker/run.bash $TAG $CMD

# [ALTERNATIVE] Raw content via wget
WITH_DEV_VOLUME=false bash -c "$(wget -qO - https://raw.githubusercontent.com/AndrejOrsula/simforge/refs/heads/main/.docker/run.bash)" -- $TAG $CMD

# [ALTERNATIVE] Raw content via curl
WITH_DEV_VOLUME=false bash -c "$(curl -fsSL https://raw.githubusercontent.com/AndrejOrsula/simforge/refs/heads/main/.docker/run.bash)" -- $TAG $CMD

Usage of SimForge

SimForge can be used in two primary ways:

Command Line Interface (CLI)

The simforge CLI is the most straightforward way to get started with SimForge. It allows you to generate and export assets directly from the command line without needing to write any code. You can then use the generated assets in your application by importing them as needed. Furthermore, the CLI provides a convenient way to list and manage the available assets.

The CLI should be used as a starting point. It supports exporting assets in a variety of formats that are compatible with most external applications — even if a direct integration with SimForge is not yet available.

Integrations

Integrations offer a more streamlined experience for using SimForge within a specific game engine or physics simulator. These modules provide APIs that automate the on-demand process of generating, caching and importing assets into external frameworks while abstracting away the underlying complexity.

Integrations are the recommended way to use SimForge for specific applications in the long term, as they provide a more ergonomic and efficient workflow. However, they may require additional setup and configuration to get started.

The number of available integrations is currently limited but expected to grow over time as the framework matures. Contributions are always welcome!

Command Line Interface (CLI)

After installing SimForge, the CLI can be accessed via the following commands:

# Run as package entrypoint
python3 -m simforge

# Run via installed command
simforge

Subcommands

The CLI provides a number of subcommands that leverage the underlying SimForge API. Each subcommand has its own set of options and arguments that can be accessed via the -h or --help flag.

simforge gen

Generate variants of registered assets based on the provided arguments (simforge gen -h).

Examples

Generate a single variant of a registered asset named "model1" (exported to ~/.cache/simforge):

simforge gen model1

Generate -n 2 variants, starting at random seed -s 100, of an asset named "geo1":

simforge gen -n 2 -s 100 geo1

Generate -n 3 variants for two different assets named "geo1" and "model2" (3 each) while exporting them to a file format with the extension -e stl:

simforge gen -n 3 geo1 model2 -e stl

Generate -n 5 variants of an asset named "geo2" in a custom output directory -o custom/cache:

simforge gen -n 5 geo2 -o custom/cache

Use a --subprocess to generate -n 8 variants of "model1":

simforge gen -n 8 --subprocess model1

Running in a subprocess is especially useful if the generator is non-compatible with the current environment. For example, Blender requires a specific version of Python that might differ from the system's default. The --subprocess flag thus allows the generator to run in a separate process with the embedded Python interpreter of the locally-installed Blender application.

simforge ls

List all registered assets in a tabular format.

❯ simforge ls
                SimForge Asset Registry
┏━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━┓
┃ # ┃   Type   ┃ Package  ┃ Name   ┃ Semantics ┃ Cached ┃
┡━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━┩
│ 1 │ geometry │ package1 │ geo1   │           │        │
│ 2 │ geometry │ package1 │ geo2   │           │        │
├───┼──────────┼──────────┼────────┼───────────┼────────┤
│ 3 │ material │ package2 │ mat1   │           │        │
│ 4 │ material │ package3 │ mat2   │           │        │
├───┼──────────┼──────────┼────────┼───────────┼────────┤
│ 5 │  model   │ package1 │ model1 │           │        │
│ 6 │  model   │ package3 │ model2 │           │        │
└───┴──────────┴──────────┴────────┴───────────┴────────┘

Note: This subcommand requires the rich package to be installed (included in the cli extra)

simforge clean

Remove all generated assets that are cached on the system.

❯ simforge clean
[HH:MM:SS] WARNING  This will remove all SimForge assets cached on your system under /home/USER/.cache/simforge (X.YZ GB)
Are you sure you want to continue? [y/n] (n):

New Assets

SimForge Assets are always registered with a specific Generator that is responsible for creating them. This means that you first need to select one of the available generators before you define new assets. Each generator leverages a different set of tools and libraries to create assets, so the process of creating and defining assets will vary depending on your choice. Please refer to the generator documentation you intend to use for more information on how to create new assets.

In case you want to create assets with an external tool or library that does not have a SimForge generator yet, consider visiting the New Generator guide to learn how to implement one.

New Generators

Implementing a new Generator is an involved process that requires inheriting from the core SimForge classes and implementing the necessary methods. Once you know which external tool or library you want to leverage, you can start by creating a new Python module and defining the following classes:

  • Generation [required]:

    • ExModelExporter(ModelExporter) - Class that exports generated model assets
    • ExGenerator(Generator) - The main class that handles the generation of assets

  • Generation [optional]:

    • ExBaker(Baker) - Class that bakes textures for the generated assets

  • Assets [standard]:

    • ExGeometry(Geometry) - Class for geometry assets
    • ExMaterial(Material) - Class for material assets
    • ExModel(Model) - Class for model assets

  • Assets [extra]:

    • ExImage(Image) - Class for image assets
    • ExArticulation(Articulation) - Class for articulation assets

First, make sure to read the documentation of the external tool or library you want to integrate with SimForge. This will help you understand the API and any limitations or constraints that you need to consider. Then, take a look at the existing generators in the SimForge codebase to get an idea of how you could structure your generator.

Template:

from __future__ import annotations

from pathlib import Path
from typing import Any, ClassVar, Dict, Mapping, Tuple

from simforge import (
    Articulation,
    Baker,
    BakeType,
    Generator,
    Geometry,
    Image,
    Material,
    Model,
    ModelExporter,
)
from simforge._typing import ExporterConfig


class ExModelExporter(ModelExporter):
    def export(self, filepath: Path | str, **kwargs) -> Path:
        raise NotImplementedError


class ExBaker(Baker):
    def setup(self):
        pass

    def bake(self, texture_resolution: int | Dict[BakeType, int]):
        if self.enabled:
            raise NotImplementedError

    def cleanup(self):
        pass


class ExGenerator(Generator):
    EXPORTERS: ClassVar[ExporterConfig] = ExModelExporter()
    BAKER: ClassVar[ExBaker] = ExBaker()

    def _setup_articulation(
        self,
        asset: ExArticulation,
        **kwargs,
    ) -> Dict[str, Any]:
        return kwargs

    def _generate_articulation(
        self,
        asset: ExArticulation,
        seed: int,
        **setup_kwargs,
    ) -> Dict[str, Any]:
        raise NotImplementedError

    def _export_articulation(
        self,
        asset: ExArticulation,
        seed: int,
        export_kwargs: Mapping[str, Any] = {},
        **generate_kwargs,
    ) -> Tuple[Path, Dict[str, Any]]:
        return self.__export(asset=asset, seed=seed, **export_kwargs), generate_kwargs

    def _cleanup_articulation(
        self,
        asset: ExArticulation,
    ):
        asset.cleanup()

    def _setup_geometry(
        self,
        asset: ExGeometry,
        **kwargs,
    ) -> Dict[str, Any]:
        return kwargs

    def _generate_geometry(
        self,
        asset: ExGeometry,
        seed: int,
        **setup_kwargs,
    ) -> Dict[str, Any]:
        raise NotImplementedError

    def _export_geometry(
        self,
        asset: ExGeometry,
        seed: int,
        export_kwargs: Mapping[str, Any] = {},
        **generate_kwargs,
    ) -> Tuple[Path, Dict[str, Any]]:
        return self.__export(asset=asset, seed=seed, **export_kwargs), generate_kwargs

    def _cleanup_geometry(
        self,
        asset: ExGeometry,
    ):
        asset.cleanup()

    def _setup_image(
        self,
        asset: ExImage,
        **kwargs,
    ) -> Dict[str, Any]:
        return kwargs

    def _generate_image(
        self,
        asset: ExImage,
        seed: int,
        **setup_kwargs,
    ) -> Dict[str, Any]:
        raise NotImplementedError

    def _export_image(
        self,
        asset: ExImage,
        seed: int,
        export_kwargs: Mapping[str, Any] = {},
        **generate_kwargs,
    ) -> Tuple[Path, Dict[str, Any]]:
        return self.__export(asset=asset, seed=seed, **export_kwargs), generate_kwargs

    def _cleanup_image(
        self,
        asset: ExImage,
    ):
        asset.cleanup()

    def _setup_material(
        self,
        asset: ExMaterial,
        **kwargs,
    ) -> Dict[str, Any]:
        return kwargs

    def _generate_material(
        self,
        asset: ExMaterial,
        seed: int,
        **setup_kwargs,
    ) -> Dict[str, Any]:
        raise NotImplementedError

    def _export_material(
        self,
        asset: ExMaterial,
        seed: int,
        export_kwargs: Mapping[str, Any] = {},
        **generate_kwargs,
    ) -> Tuple[Path, Dict[str, Any]]:
        return self.__export(asset=asset, seed=seed, **export_kwargs), generate_kwargs

    def _cleanup_material(
        self,
        asset: ExMaterial,
    ):
        asset.cleanup()

    def _setup_model(
        self,
        asset: ExModel,
        **kwargs,
    ) -> Dict[str, Any]:
        return kwargs

    def _generate_model(
        self,
        asset: ExModel,
        seed: int,
        **setup_kwargs,
    ) -> Dict[str, Any]:
        raise NotImplementedError

    def _export_model(
        self,
        asset: ExModel,
        seed: int,
        export_kwargs: Mapping[str, Any] = {},
        **generate_kwargs,
    ) -> Tuple[Path, Dict[str, Any]]:
        return self.__export(asset=asset, seed=seed, **export_kwargs), generate_kwargs

    def _cleanup_model(
        self,
        asset: ExModel,
    ):
        asset.cleanup()


class ExArticulation(Articulation, asset_metaclass=True, asset_generator=ExGenerator):
    def setup(self):
        raise NotImplementedError

    def cleanup(self):
        pass

    def seed(self, seed: int):
        pass


class ExGeometry(Geometry, asset_metaclass=True, asset_generator=ExGenerator):
    def setup(self):
        raise NotImplementedError

    def cleanup(self):
        pass

    def seed(self, seed: int):
        pass


class ExImage(Image, asset_metaclass=True, asset_generator=ExGenerator):
    def setup(self):
        raise NotImplementedError

    def cleanup(self):
        pass

    def seed(self, seed: int):
        pass


class ExMaterial(Material, asset_metaclass=True, asset_generator=ExGenerator):
    def setup(self):
        raise NotImplementedError

    def cleanup(self):
        pass

    def seed(self, seed: int):
        pass


class ExModel(Model, asset_metaclass=True, asset_generator=ExGenerator):
    def setup(self):
        raise NotImplementedError

    def cleanup(self):
        pass

    def seed(self, seed: int):
        pass

If you have any questions regarding a specific generator, feel free to open a new issue.

New Integrations

Supporting a new external framework via SimForge Integration does not follow a strict pattern, as it depends on the specific requirements of that framework. Before getting started, make sure to read its documentation to understand its API and any limitations or constraints. Then, take a look at the existing integrations in the SimForge codebase to get an idea of how to structure your integration.

In a nutshell, the integration should consume any Asset definition and generate/export the requested number of asset variants to an intermediate file format supported by the external framework. Then, it should spawn the assets in the framework and optionally configure them based on the metadata provided by the generator. The integration can either be implemented as a Python function or a class, depending on the desired ergonomics.

If the framework requires a specific environment that is not compatible with the generator, consider generating the assets in a subprocess via Generator.generate_subprocess() instead of Generator.generate(). This way, the generator can run in a separate process with the required environment, and the integration can spawn the assets in the framework without any compatibility issues.

Template:

from simforge import Asset


def spawn_simforge_asset(
    asset: Asset,
    num_assets: int = 1,
    seed: int = 0,
    subprocess: bool = False,
):
    # Select an intermediate model format supported by the framework
    FILE_FORMAT: ModelFileFormat = ...

    # Instantiate the generator associated with the asset
    generator = asset.generator_type(
        num_assets=num_assets,
        seed=seed,
        file_format=FILE_FORMAT,
    )

    # Generate the assets
    if subprocess:
        generator_output = generator.generate_subprocess(asset)
    else:
        generator_output = generator.generate(asset)

    # Iterate over the generator output
    for filepath, metadata in generator_output:
        # TODO: Spawn the asset from filepath
        # TODO: Configure the asset from metadata

If you have any questions regarding a specific integration, feel free to open a new issue.

Dev Container

SimForge includes a Dev Container configuration under .devcontainer/devcontainer.json that you can customize for your development needs.

VS Code

Alongside the configuration, .devcontainer/open.bash script is provided to streamline the process of building and opening the repository as a Dev Container in Visual Studio Code (VS Code).

# Path to a cloned repository
simforge/.devcontainer/open.bash

Contributors