Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 4 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
tutorials/sim2real-blogs/**/*.svg filter=lfs diff=lfs merge=lfs -text
tutorials/sim2real-blogs/**/reference-benchmark/**/*.csv -text whitespace=cr-at-eol
tutorials/sim2real-blogs/**/*.gz filter=lfs diff=lfs merge=lfs -text
tutorials/sim2real-blogs/**/*.npz filter=lfs diff=lfs merge=lfs -text
*.pptx filter=lfs diff=lfs merge=lfs -text
*.pdf filter=lfs diff=lfs merge=lfs -text
*.png filter=lfs diff=lfs merge=lfs -text
Expand Down
105 changes: 68 additions & 37 deletions tutorials/sim2real-blogs/README.md
Original file line number Diff line number Diff line change
@@ -1,56 +1,87 @@
# Sim2Real Blogs

Companion tutorials for the simulation-to-real robotics blog series. Each article
has its own lessons, exercises, and completed solutions.
Companion tutorials for the simulation-to-real robotics blog series. Article 2
runs the same receiving-box task on native MuJoCo CPU and MuJoCo Warp GPU:
each arm grasps both cubes, carries and releases them into the box, then withdraws.
The lessons check both SO-101 and Seeed reBot DevArm. They do not deploy a policy
to a physical robot.

## Articles
| Notebook | Description | Colab |
|---|---|---|
| [MuJoCo fundamentals](notebooks/mujoco/part1/01__mujoco_fundamentals.ipynb) | Inspect a model, state and CPU stepping. | [Open](https://colab.research.google.com/github/johnnynunez/accelerated-computing-hub/blob/0a5cd94b208e79dd443a5cf6393cd29df0d43d56/tutorials/sim2real-blogs/notebooks/mujoco/part1/01__mujoco_fundamentals.ipynb) |
| [Pick and place](notebooks/mujoco/part1/02__pick_and_place.ipynb) | Both robots place both cubes in the box. | [Open](https://colab.research.google.com/github/johnnynunez/accelerated-computing-hub/blob/0a5cd94b208e79dd443a5cf6393cd29df0d43d56/tutorials/sim2real-blogs/notebooks/mujoco/part1/02__pick_and_place.ipynb) |
| [MuJoCo Warp](notebooks/mujoco/part2/03__mujoco_warp.ipynb) | The same task on one selected GPU. | [Open](https://colab.research.google.com/github/johnnynunez/accelerated-computing-hub/blob/0a5cd94b208e79dd443a5cf6393cd29df0d43d56/tutorials/sim2real-blogs/notebooks/mujoco/part2/03__mujoco_warp.ipynb) |
| [CPU/GPU benchmark](notebooks/mujoco/part2/04__cpu_gpu_benchmark.ipynb) | Preflight first; optionally measure 1 through 2048 environments. | [Open](https://colab.research.google.com/github/johnnynunez/accelerated-computing-hub/blob/0a5cd94b208e79dd443a5cf6393cd29df0d43d56/tutorials/sim2real-blogs/notebooks/mujoco/part2/04__cpu_gpu_benchmark.ipynb) |

| Article | Lessons |
| --- | --- |
| 2. How to Use NVIDIA Warp and MJWarp to Accelerate Robotics Simulation and Learning Workflows | [SO-101 simulation: from MuJoCo to MuJoCo Warp](notebooks/mujoco/README.md) |
See the [lesson guide](notebooks/mujoco/README.md), [benchmark method and results](notebooks/mujoco/BENCHMARK.md),
and [validation scope](VALIDATION.md). Reference implementations run by default;
student exercises remain editable and optional stacking stays separate.

The current lessons cover simulation and physics throughput; they do not deploy
a policy to a physical robot.
## Quickstart (local)

## Notebooks
Linux (tested on Ubuntu) or macOS with `git` and internet access for the first run.
An NVIDIA GPU is needed only for Part 2 and the GPU benchmark. This is the only local
setup path; the lesson guide links back here.

| Notebook | Description | Colab |
| --- | --- | --- |
| [01. MuJoCo Fundamentals](notebooks/mujoco/part1/01__mujoco_fundamentals.ipynb) | Load the SO-101 model, inspect state, and control a CPU simulation. | [Open in Colab](https://colab.research.google.com/github/NVIDIA/accelerated-computing-hub/blob/main/tutorials/sim2real-blogs/notebooks/mujoco/part1/01__mujoco_fundamentals.ipynb) |
| [02. Pick and Place](notebooks/mujoco/part1/02__pick_and_place.ipynb) | Complete the physics loop, validate a stack, and optionally place both cubes in a box. | [Open in Colab](https://colab.research.google.com/github/NVIDIA/accelerated-computing-hub/blob/main/tutorials/sim2real-blogs/notebooks/mujoco/part1/02__pick_and_place.ipynb) |
| [03. MuJoCo Warp](notebooks/mujoco/part2/03__mujoco_warp.ipynb) | Seed GPU state, validate one world, capture CUDA work, and benchmark a batch. | [Open in Colab](https://colab.research.google.com/github/NVIDIA/accelerated-computing-hub/blob/main/tutorials/sim2real-blogs/notebooks/mujoco/part2/03__mujoco_warp.ipynb) |
```bash
git clone https://github.com/NVIDIA/accelerated-computing-hub.git
cd accelerated-computing-hub/tutorials/sim2real-blogs
curl -LsSf https://astral.sh/uv/install.sh | sh # installs uv once; open a new terminal afterwards
uv venv --allow-existing --python 3.12 notebooks/mujoco/.venv && uv pip install --python notebooks/mujoco/.venv/bin/python --require-hashes -r notebooks/mujoco/requirements.lock.txt
source notebooks/mujoco/.venv/bin/activate # again in every new terminal
python doctor.py # checks the setup; prints a fix: line per problem
cd notebooks/mujoco/part1
python solutions/so101_pick_place_solution.py --task box --sim-substeps 20
```

The last command opens the viewer (add `--headless-steps 2000` without a display),
downloads the pinned robot model once on first use, closes by itself after the
40-second task and ends with `Task complete: both cubes are in the receiving box (verified).`
Esc, Q or Ctrl+C stop it early. Part 2 runs the same task on the GPU:
`cd ../part2 && python solutions/so101_mjwarp_solution.py --task box --sim-substeps 20 --device cuda:0`
(the first GPU run compiles kernels for a few minutes). Add `--robot rebot` for the
second arm. Notebooks: `uv pip install jupyterlab && jupyter lab`.

Follow the [local Python 3.12 setup and validation instructions](notebooks/mujoco/README.md)
to run the lessons. Colab links target upstream `main` and become available after merge.
No environment variable is needed for the robot models. A valid cache from an
earlier run is reused; anything else in `~/.cache/mujoco_menagerie` is left alone
or moved aside, and a verified copy is downloaded into a folder named after the
pinned commit. Offline machines: copy `~/.cache/mujoco_menagerie` and
`~/.cache/mujoco_menagerie_rebot` from a machine where the lessons already ran.

The optional receiving-box task runs with either the SO-101 or Seeed reBot
DevArm, on CPU MuJoCo or single-world MuJoCo Warp. Its checks require each cube
to be grasped, lifted, carried and released inside the box, then settled after
the gripper withdraws. The original stacking task remains the default.
With conda instead of uv: `conda create -y -n sim2real python=3.12 git`,
`conda activate sim2real`, then
`pip install --require-hashes -r notebooks/mujoco/requirements.lock.txt` and `python doctor.py`.

## Brev and Docker
## Local and hosted setup

This tutorial has its own [Dockerfile](brev/dockerfile),
[dependencies](brev/requirements.txt), [Docker Compose configuration](brev/docker-compose.yml),
and [test entrypoint](brev/test.bash). Use an NVIDIA CUDA GPU such as an L40S, L4,
or T4 and a provider with Flexible Ports for Brev.
Use Python 3.12 with the module's unchanged hash lock. This update deliberately
uses the measured MuJoCo 3.8.0 / MuJoCo Warp 3.8.0.3 / Warp 1.15.0 line, replacing
the previous Hub tutorial's 3.12 / 1.17 dependencies. Newton uses a separate environment.

From the repository root, build and run the dedicated environment:
The existing Brev Docker Compose services and shared Hub entrypoint are retained.
The image installs the same physics lock before adding notebook/UI tools under
constraints. Build and test it through the Hub scripts:

```bash
./brev/dev-build.bash sim2real-blogs
./brev/dev-start.bash sim2real-blogs
./brev/dev-test.bash sim2real-blogs
brev/dev-build.bash sim2real-blogs
brev/dev-test.bash sim2real-blogs
```

The Compose configuration uses `ghcr.io/nvidia/sim2real-blogs-tutorial:latest`;
build locally until that image is published by the repository's CI.
A container build, Brev deployment or Colab run is separate validation; imported
workstation measurements do not establish those results. The Colab links select
a fixed public notebook revision; setup downloads the separately pinned runtime source. Setup prints its resolved Git SHA,
refuses to overwrite a checkout at another revision, and skips LFS downloads for
teaching code. Teaching cells need a Python 3.12 kernel; the benchmark notebook
uses an isolated Python 3.12 environment for hosted subprocesses.

See the [validation record](VALIDATION.md) for the CPU/dual-GPU matrix,
container checks and remaining environment coverage.
## Evidence and attribution

## Attribution
[Migration provenance](notebooks/mujoco/MIGRATION_MANIFEST.json) records relocation
and source identities. The original result, notebook, viewer and reference packages
are retained byte-for-byte, including failed cases and their historical paths.
Pull the required LFS objects when inspecting archived evidence; notebook physics
does not require downloading those archives.

The imported lessons retain their [MIT license](notebooks/mujoco/LICENSE).
See the [third-party notices](notebooks/mujoco/THIRD_PARTY_NOTICES.md) for
source attribution and robot asset licenses.
The imported materials retain their [MIT license](notebooks/mujoco/LICENSE) and
[third-party notices](notebooks/mujoco/THIRD_PARTY_NOTICES.md), including model
assets preserved inside diagnostic archives.
116 changes: 49 additions & 67 deletions tutorials/sim2real-blogs/VALIDATION.md
Original file line number Diff line number Diff line change
@@ -1,67 +1,49 @@
# Validation — 4 October 2026

These checks use this tutorial's Python 3.12, MuJoCo **3.12.0**, MuJoCo Warp
**3.12.0**, Warp **1.17.0** and NumPy **2.5.3** environment. The earlier companion
repository's MuJoCo 3.8 results were checked separately and are not substituted
for this version line.

## Task matrix

| Backend | Robot | Original stack | Receiving box |
|---|---|---|---|
| Native MuJoCo CPU | SO-101 | Pass, 600 frames | Pass, 1,800 frames |
| Native MuJoCo CPU | reBot | Pass, 600 frames | Pass, 1,800 frames |
| MJWarp CUDA 0 | SO-101 | Pass, 600 frames | Pass, 2,000 frames |
| MJWarp CUDA 0 | reBot | Pass, 600 frames | Pass, 2,000 frames |
| MJWarp CUDA 1 | SO-101 | Pass, 600 frames | Pass, 2,000 frames |
| MJWarp CUDA 1 | reBot | Pass, 600 frames | Pass, 2,000 frames |

CPU task checks ran on macOS ARM64. CUDA checks ran on Linux x86-64 with two
NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPUs and driver 615.71.09.
Each process selected one device; this is not a multi-GPU scaling benchmark.
The checked runtime snapshot contained 31 Python sources, all matching the
submitted versions.

The box runs were independently audited from measured 50 Hz observations.
Both cubes established opposing loaded jaw contacts, sustained whole-object
lift and airborne carry, opening over the box, detached release and settling,
whole-cube containment and final jaw withdrawal. GPU observations preserve
forces from the GPU solver. The contact adapter reads `contact.geom`, because
MJWarp exports that modern pair while legacy `geom1`/`geom2` can retain stale IDs.

Capacity checks run after each GPU substep. Peak box-task counts were 33 contacts
and 180 constraints for SO-101 (capacities 128/300), and 20/69 for reBot
(capacities 256/500). No capacity overflow occurred. Solver iteration-limit
flags are retained as convergence diagnostics and are not treated as capacity
flags. Passing the task does not establish full solver convergence or identical
CPU/GPU trajectories.

The published commands allow 2,000 frames: 40 simulated seconds at 50 Hz with
a 0.002 s physics timestep. That budget includes settling margin. During the
separate MuJoCo 3.8 validation, one reBot run at 36 seconds had only 35 of the
required 50 consecutive settled frames; the controller and acceptance thresholds
were left unchanged. All four final CUDA cases in the 3.12 matrix were checked
at 40 seconds, with at least 62 consecutive detached settled frames at the end.

## Regression and packaging checks

- Local pinned-environment suite: **24 passed, 3 CUDA checks explicitly skipped**.
- Dedicated Docker image built on Linux using the published CUDA 13.1/Python 3.12
recipe. Its dependency check passed; the installed interpreter was Python 3.12.15.
- Inside that image, `bash tutorials/sim2real-blogs/brev/test.bash` passed
**27 tests and 5 subtests, with no skips**, in 101.87 seconds. This includes the
CUDA stack, captured benchmark, actual constraint-overflow rejection, both CPU
box references and the new acceptance/contact-export regression tests.
- Both SO-101 and reBot box references also passed at 2,000 frames inside the
image on one selected GPU, with JSON reports confirming task success.
- All three notebooks passed the Hub canonical-format checker. The two existing
task notebooks retain all 18 previous code cells and each add one opt-in box cell.
- Python syntax, 26 relative documentation links, Compose configuration, shell
syntax and `git diff --check` passed. The existing Warp tutorial tree matches
the PR base.

Robot assets are downloaded from the pinned upstream revisions. The container
was tested directly with one GPU; a hosted Brev deployment, Compose service
lifecycle, interactive viewers and Colab execution were not tested. Colab links
point to upstream `main` and become usable after merge. Repository CI on this
fork still requires the maintainers' normal review process.
# Validation scope

This update relocates the reviewed Article 2 source into the Hub tutorial. Its
physics scripts and dependency lock are unchanged; notebooks add Hub checkout
setup, use the Hub naming convention, and retain their reference/task defaults.
The [migration manifest](notebooks/mujoco/MIGRATION_MANIFEST.json) maps every
imported file and records each adaptation.

## Imported workstation evidence

- [Task-specific CPU/GPU study](notebooks/mujoco/benchmark-results/2026-10-09-validated-box-gpu0/README.md): both robots, nine environment counts, one complete warm-up and five repetitions. **35 of 36 configurations passed**. The reBot GPU 512 configuration failed its physical gate and supplies no accepted timing or ratio.
- [Notebook execution](notebooks/mujoco/notebook-validation/2026-10-10/README.md): four source notebooks passed actual kernels, including both box lessons and measurement-disabled benchmark preflight. Fundamentals produced finite image pixels; this is not image visual QA.
- [Viewer lifecycle](notebooks/mujoco/VIEWER_VALIDATION.md): finite completion and clean cancellation on the original Linux NVIDIA desktop, with the exact source relationship documented. The ordinary-Python macOS launch guard has regression coverage, not macOS GUI validation.
- [Viewer exit, Ctrl+C and benchmark output, 13 October](notebooks/mujoco/validation/viewer-exit-and-benchmark-2026-10-13/README.md): reproduces a tester's report against the previous `launch_passive` lessons under a real X server, and validates the fix (every run ends with a verified result and exit code 0/1, Ctrl+C exits 130 within about a second, self-describing benchmark output). Imported from [NVIDIA-dev/newton_blogs#2](https://github.com/NVIDIA-dev/newton_blogs/pull/2).
- [Legacy solver diagnostic](notebooks/mujoco/validation/legacy-linesearch-2026-10-10/README.md): original warning and diagnostic outcomes remain available. This is separate from the box benchmark.

All original archives, result JSON, figures and recorded hashes are unchanged.
They retain original source paths and historical links. Current tutorial links
and commands are in the [module README](notebooks/mujoco/README.md) and
[benchmark guide](notebooks/mujoco/BENCHMARK.md). These are community workstation
measurements; importing them is not a new hardware or container execution.

## Version boundary

The prior Hub [4 October validation record](history/VALIDATION-2026-10-04.md)
is preserved verbatim. It describes MuJoCo 3.12.0 / MuJoCo Warp 3.12.0 / Warp
1.17.0, including its own container checks. The current port intentionally uses
the source study's **3.8.0 / 3.8.0.3 / 1.15.0** environment and hash lock. The
historical container result cannot validate this updated image.

## Checks for this layout

With the tutorial's Python 3.12 locked environment active:

```bash
python -m pytest tutorials/sim2real-blogs/test -v
python brev/test-notebook-format.py sim2real-blogs
bash tutorials/sim2real-blogs/brev/test.bash
```

The full suite includes real task/recording checks and may download pinned
robot assets. CUDA checks skip explicitly if no GPU is available. Viewer unit
tests mock graphics; they do not open a desktop or establish GL validation.
Set `ARTICLE2_NOTEBOOKS_EXECUTE=1` to execute the benchmark notebook's preflight
in an actual kernel; measurements remain disabled. The optional CPU viewer
integration check uses `RUN_ARTICLE2_VIEWER_CPU_TESTS=1` and is not a GUI test.
`test/test_article2_viewer_x11.py` is a real-window GUI test: it runs the lesson scripts under an X server (Xvfb is enough) and sends real key presses, a window-manager close and Ctrl+C. It needs `DISPLAY` and `xdotool` and skips cleanly without them, so the container suite skips it. `test/test_article2_ctrl_c.py` checks the Ctrl+C exit contract with real signals and needs no display.

The [fresh Hub container validation](hub-validation/2026-10-10/README.md) built the updated image and passed **151 tests with one optional viewer-integration skip**, including both robots on CPU/GPU and the real-kernel benchmark preflight. Its initial collection failure and corrected test-only rerun are preserved. Compose lifecycle, live Brev/Colab, lower-end GPU and macOS GUI validation remain unverified.
13 changes: 9 additions & 4 deletions tutorials/sim2real-blogs/brev/dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,17 @@ RUN curl -fsSL https://keyserver.ubuntu.com/pks/lookup?op=get\&search=0xBA693236
&& rm -f /usr/lib/python3.*/EXTERNALLY-MANAGED \
&& rm -rf /var/lib/apt/lists/*

# Copy only requirements.txt first for better Docker layer caching.
# Use the measured Blog 2 dependency line, unchanged from the source archive.
# The previous Hub image used MuJoCo 3.12 / Warp 1.17; those are not these results.
COPY tutorials/${ACH_TUTORIAL}/notebooks/mujoco/requirements.lock.txt /opt/physics.lock.txt
COPY tutorials/${ACH_TUTORIAL}/brev/requirements.txt /opt/requirements.txt

# Install Python packages.
RUN pip install --no-cache-dir --root-user-action=ignore -r /opt/requirements.txt \
&& rm -f /opt/requirements.txt
# UI tools may add packages but cannot upgrade the locked physics dependencies.
RUN pip install --no-cache-dir --root-user-action=ignore --require-hashes -r /opt/physics.lock.txt \
&& pip freeze > /opt/locked-constraints.txt \
&& pip install --no-cache-dir --root-user-action=ignore -c /opt/locked-constraints.txt -r /opt/requirements.txt \
&& pip check \
&& rm -f /opt/requirements.txt /opt/physics.lock.txt /opt/locked-constraints.txt

# Disable unnecessary default Jupyter extensions.
RUN python -m jupyter labextension disable "@jupyterlab/apputils-extension:announcements" \
Expand Down
25 changes: 4 additions & 21 deletions tutorials/sim2real-blogs/brev/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,29 +1,12 @@
# Warp
warp-lang==1.17.0

# Robotics simulation (same versions as notebooks/mujoco/requirements.txt)
mujoco==3.12.0
mujoco-warp==3.12.0

# Arrays
numpy==2.5.3

# Scientific/numeric
# Notebook/UI and test tools only. The Dockerfile first installs the unchanged
# notebooks/mujoco/requirements.lock.txt with --require-hashes (MuJoCo 3.8.0,
# MuJoCo Warp 3.8.0.3, Warp 1.15.0), then constrains these additions to that lock.
scipy

# Visualization
matplotlib==3.11.2
ipympl
Pillow

# Dark mode detection
jupyter-dark-detect == 0.1.0

# Jupyter
jupyter-dark-detect==0.1.0
jupyter
jupyter-server-proxy
jupyterlab
jupyterlab-execute-time

# Tutorial validation
pytest
4 changes: 3 additions & 1 deletion tutorials/sim2real-blogs/brev/test.bash
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,6 @@ set -euo pipefail

TUTORIAL_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
nvidia-smi
python -m pytest "$TUTORIAL_DIR/test" "$@"
# Bound the complete correctness suite; this never enables benchmark measurements.
# GNU timeout terminates its process group on expiry; Docker owns container cleanup.
timeout --signal=TERM --kill-after=15s 1800 python -m pytest "$TUTORIAL_DIR/test" "$@"
Loading