Setting Up Your Physical AI Laboratory

Infrastructure

Docker

Isaac Sim

Isaac Lab

Tutorial

A complete beginner’s guide to NVIDIA Isaac Sim & Isaac Lab on Ubuntu 24.04 with Docker

Author

Rajesh

Published

December 12, 2025

TL;DR - What You’ll Build

By the end of this guide, you will have:

NVIDIA Isaac Sim 5.1.0 running in Docker (robotics simulator)
NVIDIA Isaac Lab for reinforcement learning experiments
A launcher script that handles everything - NGC login, image downloads, setup
A reproducible setup you can share with others

Time required: ~2 hours (mostly waiting for downloads)

The Easy Path: After installing Docker (Section 3), download our launcher script (v2.2.0) - it handles everything else!

wget https://raw.githubusercontent.com/myidentity/physical-ai-lab/main/scripts/auto_start.sh
chmod +x auto_start.sh
./auto_start.sh

1. Introduction

Why This Guide Exists

When I started learning robotics simulation, I hit wall after wall:

Documentation assumed I knew things I didn’t
Tutorials were outdated or incomplete
Small configuration mistakes wasted hours

This guide documents every step - including the mistakes - so you don’t have to struggle like I did.

What You’ll Learn

Section	What You’ll Do	Can Skip If…
2. Prerequisites	Install NVIDIA drivers	Already have working drivers
3. Docker Setup	Install Docker & NVIDIA toolkit	Already have Docker with GPU
4. Quick Start	Use launcher script for easy setup	Want manual control
5. Verification	Test everything works	Confident it works
6. Troubleshooting	Fix common problems	Everything works
Appendix A	Manual installation steps	Using the launcher
Appendix B	Docker & Isaac Sim concepts	Already know Docker

2. Prerequisites

Hardware Requirements

Minimum Hardware

GPU: NVIDIA RTX series (RTX 2070 or better)
VRAM: 8GB minimum, 12GB+ recommended
RAM: 32GB recommended
Storage: 50GB+ free space (Docker images are large!)
OS: Ubuntu 22.04 or 24.04 LTS

Install NVIDIA Drivers

First, check if drivers are already installed:

nvidia-smi

What You Should See

If drivers are installed, you’ll see a table showing your GPU model, driver version, and CUDA version. If you see command not found, follow the steps below.

If nvidia-smi fails, install drivers:

# Step 1: Update your system
sudo apt update && sudo apt upgrade -y

# Step 2: See which drivers are available for your GPU
sudo ubuntu-drivers list --gpgpu

# Step 3: Install the recommended driver automatically
sudo ubuntu-drivers install --gpgpu

# Step 4: Reboot (required!)
sudo reboot

After reboot, verify:

nvidia-smi

Secure Boot Users

If your computer uses Secure Boot, you’ll see extra steps during installation. Ubuntu will ask you to create a temporary password. After reboot, a blue “MOK Management” screen appears - select “Enroll MOK”, enter the password, and reboot.

If you miss this screen, reinstall: sudo ubuntu-drivers install --gpgpu

Our Test System

This guide was tested on:

Component	Specification
OS	Ubuntu 24.04.3 LTS
Kernel	6.14.0-37-generic
GPU	NVIDIA GeForce RTX 5090 Laptop GPU
Driver	580.95.05
Docker	29.1.2

3. Docker Setup

Goal

Install Docker and configure it to use your NVIDIA GPU. This is required before using the launcher script.

Step 3.1: Install Docker

# Update package list
sudo apt update

# Install Docker
sudo apt install -y docker.io docker-compose-v2

# Add yourself to docker group (avoids needing sudo)
sudo usermod -aG docker $USER

Important!

After adding yourself to the docker group, you must log out and log back in (or reboot) for it to take effect.

Alternatively, run this to apply immediately (current terminal only):

newgrp docker

Step 3.2: Verify Docker Works

docker run hello-world

You should see “Hello from Docker!” message.

Step 3.3: Install NVIDIA Container Toolkit

This lets Docker containers access your GPU:

# Add NVIDIA repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install the toolkit
sudo apt update
sudo apt install -y nvidia-container-toolkit

# Configure Docker to use NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 3.4: Verify GPU Access in Docker

docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi

You should see the same nvidia-smi output as on your host.

Common Error: “could not select device driver”

If you see this error:

sudo systemctl restart docker

Still not working? Reboot your computer.

4. Quick Start with Launcher Script

The Easy Way

Now that Docker is set up, use our launcher script to handle everything else: NGC login, downloading Isaac Sim, cloning Isaac Lab, and running containers.

Download the Launcher

Current version: v2.2.0

cd ~
wget https://raw.githubusercontent.com/myidentity/physical-ai-lab/main/scripts/auto_start.sh
chmod +x auto_start.sh

Run the Launcher

./auto_start.sh

You’ll see a menu like this:

═══════════════════════════════════════════════════════
     Isaac Sim 5.1.0 & Isaac Lab Launcher v2.2.0
═══════════════════════════════════════════════════════

─────────── SETUP (First Time) ─────────────────────────
s) NGC Login (for pulling images)        [manual check]
p) Pull Isaac Sim 5.1.0 Image (~15GB)    [NOT FOUND]
c) Clone Isaac Lab Repository            [NOT FOUND]

─────────── 🎮 ISAAC SIM (Standalone Simulation) ─────────
  Start Simulator:
    1) With GUI (Desktop Mode)
    2) Headless (No Display)
  ...
  💾 Save Your Work:
    7) Save Installed Packages ⭐

─────────── 🤖 ISAAC LAB (RL Training Environment) ───────
  Start & Access:
    8) Start/Build Container ⭐
    9) Enter Container
  🛑 Shutdown Options:
    h) Hibernate (Safe) ⭐ - Preserves everything
    a) DELETE Everything ⚠️ - Destroys all data

─────────── ⚙️  UTILITIES ─────────────────────────────────
    0) Toggle Display Driver (Dummy ↔ NVIDIA)
    b) Python Package Guide

First-Time Setup Walkthrough

Follow these steps in order:

1. NGC Login (Option ‘s’):

You need an NVIDIA NGC account to download Isaac Sim:

Go to https://ngc.nvidia.com and create a free account
Go to Setup → API Key and generate a new key
Select option ‘s’ in the launcher

When prompted: - Username: $oauthtoken (literally type this) - Password: Your API key from NGC

2. Pull Isaac Sim Image (Option ‘p’) - ~15GB download:

Coffee Time

This download takes 15-30 minutes depending on your internet speed.

3. Clone Isaac Lab (Option ‘c’):

4. Menu after setup complete - all components [FOUND]:

Launcher Menu Reference

Setup Options

Option	Name	Description
s	NGC Login	Login to NVIDIA GPU Cloud (required to pull images)
p	Pull Isaac Sim Image	Download Isaac Sim 5.1.0 (~15GB)
c	Clone Isaac Lab	Clone Isaac Lab repository

🎮 Isaac Sim Options

Option	Name	Description
1	With GUI	Launch Isaac Sim with graphical interface
2	Headless	Launch without display (for servers/remote)
3	Compatibility Check	Verify system meets requirements
4	Setup Directories	Create/reset cache directories
5	System Information	View GPU, storage, and image info
6	Check Prerequisites	Verify Docker, GPU access, etc.
7	Save Installed Packages ⭐	Save pip-installed packages permanently

Option 7: Save Your Packages

When you pip install packages inside Isaac Sim, they’re lost when you exit. Use Option 7 while the container is running to save them permanently to a custom image.

🤖 Isaac Lab Options

Option	Name	Description
8	Start/Build Container	Build and start the Isaac Lab Docker container
9	Enter/Resume Container	Enter running container or resume hibernated one
h	Hibernate (Safe) ⭐	Safely pause container - preserves everything
a	DELETE Everything ⚠️	Destroys container + all training logs

Safe Shutdown vs Destructive Delete

Option h (Hibernate): Use this before shutting down your computer. Everything is preserved - training logs, installed packages, container state. Resume with Option 9 when you return.
Option a (DELETE): Only use when you want a clean slate or need to free disk space. This permanently deletes all training logs and container data! Requires typing “DELETE” to confirm.

⚙️ Utilities

Option	Name	Description
0	Toggle Display Driver	Switch between NVIDIA (HDMI) and Dummy (headless)
b	Python Package Guide	Instructions for managing packages

Isaac Lab vs Isaac Sim

Isaac Sim (Options 1-7): The base simulator for rendering and physics
Isaac Lab (Options 8, 9, h, a): Reinforcement learning framework built on Isaac Sim

5. Verification

Run Compatibility Check

Select Option 3 from the launcher to verify your system:

Look for “System checking result: PASSED” at the bottom.

Start Isaac Sim

Select Option 1 to start Isaac Sim with GUI. You’ll enter an interactive shell:

Container shell before running Isaac Sim

Inside the container, run ./runapp.sh to start the simulator:

First-Time Startup

The first time you run Isaac Sim, it compiles shaders for your GPU. This can take 5-10 minutes. Subsequent starts are much faster.

Your First Simulation

Once Isaac Sim is running:

Go to Window → Browsers → Isaac Sim Assets
Search for “franka” or browse the Robots category
Double-click a Franka robot to load it
Click Play to start simulation

Start Isaac Lab

Now let’s verify Isaac Lab works by training a reinforcement learning agent!

Step 1: Start Isaac Lab Container

Select Option 8 from the launcher to build and start the Isaac Lab container:

┌─────────────────────────────────────────────────────────────────┐
│        Isaac Sim 5.1.0 & Isaac Lab Launcher                     │
├─────────────────────────────────────────────────────────────────┤
│  ISAAC LAB OPTIONS                                              │
│  8) Start/Build Container    [Run Isaac Lab]                    │
│  9) Quick Access                                                │
│  a) Stop Container                                              │
└─────────────────────────────────────────────────────────────────┘

First-Time Build

The first time you run Option 8, Isaac Lab will build its Docker container. This takes 10-15 minutes as it installs all RL dependencies (PyTorch, RSL-RL, etc.). Subsequent starts are instant.

Step 2: Run Cartpole Training

Once inside the Isaac Lab container, you’ll see the workspace prompt:

root@physical-ai-lab:/workspace/isaaclab#

Run the Cartpole training with 64 parallel environments:

./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --num_envs=64

What the command does:

Part	Meaning
`./isaaclab.sh -p`	Run a Python script through Isaac Lab
`scripts/reinforcement_learning/rsl_rl/train.py`	The RSL-RL training script
`--task=Isaac-Cartpole-v0`	The Cartpole balancing task
`--num_envs=64`	Run 64 environments in parallel

Step 3: Watch the Training!

You should see 64 cartpoles training simultaneously - each one learning to balance the pole through trial and error!

Isaac Lab running Cartpole training with 64 parallel environments

What You’re Seeing

64 parallel environments: The GPU simulates all 64 cartpoles simultaneously
Real-time learning: Watch the poles start wobbly and gradually stabilize
Training metrics: The terminal shows reward values increasing as learning progresses

Other Isaac Lab Commands

# List all available environments
./isaaclab.sh -p scripts/environments/list_envs.py

# Train a different task (Humanoid)
./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Humanoid-v0 --num_envs=64

# Train with more environments (if you have VRAM)
./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --num_envs=256

🎉 Congratulations! You now have a fully working Physical AI Laboratory with both Isaac Sim and Isaac Lab!

6. Troubleshooting

GPU Not Detected in Container

Symptom: nvidia-smi works on host but not in container

# Restart Docker
sudo systemctl restart docker

# If still not working, reboot
sudo reboot

Display Errors

Symptom: “Cannot open display” or blank window

# Allow local connections to X server
xhost +local:

# Check DISPLAY variable
echo $DISPLAY

Permission Denied

Symptom: “Permission denied” when running Docker

# Add yourself to docker group
sudo usermod -aG docker $USER
# Log out and log back in

Isaac Sim Won’t Start

Symptom: Container exits immediately

# Check Docker logs
docker logs <container_id>

# Run with verbose output
docker run --rm -it --gpus all nvcr.io/nvidia/isaac-sim:5.1.0 bash
./isaac-sim.sh --verbose

Out of Disk Space

# Check disk space
df -h

# Clean up Docker (removes ALL unused images)
docker system prune -a

7. Resources

Resource	Description
Isaac Sim Documentation	Complete user guide and API reference
Isaac Lab Documentation	RL environments and training guides
NVIDIA Developer Forums	Community support
NGC Catalog	Official container images

Conclusion

You now have a fully functional Physical AI laboratory!

What you’ve accomplished:

Docker with GPU support
Isaac Sim 5.1.0 for robotics simulation
Isaac Lab for reinforcement learning
A launcher script for easy access

Remember:

Use ./auto_start.sh for easy access
First runs are slow (shader compilation)
Check troubleshooting if things go wrong

Happy simulating!

Appendices

Appendix A: Manual Installation

When to Use This

These manual steps are an alternative to using the launcher script. Use this if you prefer full control or need to customize the setup.

A.2: Create Directory Structure

mkdir -p ~/isaac-sim/{cache/kit,cache/ov,cache/pip,cache/glcache,cache/computecache,logs,data,documents}

A.3: Pull Isaac Sim Image

docker pull nvcr.io/nvidia/isaac-sim:5.1.0

A.4: Run Isaac Sim Manually

xhost +local:
docker run --rm -it --gpus all \
    -e DISPLAY \
    -e PRIVACY_CONSENT=Y \
    -v $HOME/.Xauthority:/root/.Xauthority \
    -v ~/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \
    -v ~/isaac-sim/cache/ov:/root/.cache/ov:rw \
    -v ~/isaac-sim/cache/pip:/root/.cache/pip:rw \
    -v ~/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
    -v ~/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \
    -v ~/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \
    -v ~/isaac-sim/data:/root/.local/share/ov/data:rw \
    -v ~/isaac-sim/documents:/root/Documents:rw \
    --network=host \
    nvcr.io/nvidia/isaac-sim:5.1.0

A.5: Clone Isaac Lab Manually

mkdir -p ~/docker/isaac-lab
cd ~/docker/isaac-lab
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab

A.6: Build and Start Isaac Lab

python3 docker/container.py start
python3 docker/container.py enter

Appendix B: Core Concepts

Optional Reading

This section explains Docker and Isaac Sim concepts. Skip if you’re already familiar.

What is Docker?

Think of Docker as a shipping container for software:

┌─────────────────────────────────────────────────┐
│                Your Computer                     │
│  ┌─────────────────┐  ┌─────────────────┐       │
│  │  Container 1    │  │  Container 2    │       │
│  │  Isaac Sim      │  │  Isaac Lab      │       │
│  │  Python 3.11    │  │  PyTorch, etc   │       │
│  └─────────────────┘  └─────────────────┘       │
│         Docker Engine (manages containers)       │
└─────────────────────────────────────────────────┘

Why use Docker?

Reproducibility: Same environment everywhere
Isolation: Won’t mess up your system Python
Easy cleanup: Delete container, everything gone
NVIDIA support: GPU access inside containers

What is Isaac Sim?

NVIDIA Isaac Sim is a robotics simulator that:

Renders photorealistic environments
Simulates accurate physics
Supports many robot models
Runs on NVIDIA GPUs

What is Isaac Lab?

Isaac Lab builds on Isaac Sim and adds:

Reinforcement learning training
Pre-built robot environments
Training algorithms (PPO, SAC, etc.)
Parallel environment simulation

Key Terms

Term	Meaning
Container	Isolated environment running an application
Image	Template used to create containers
Volume	Persistent storage that survives container restarts
NGC	NVIDIA GPU Cloud - where NVIDIA hosts container images

--- title: "Setting Up Your Physical AI Laboratory" description: "A complete beginner's guide to NVIDIA Isaac Sim & Isaac Lab on Ubuntu 24.04 with Docker" author: "Rajesh" date: "2025-12-12" categories: [Infrastructure, Docker, Isaac Sim, Isaac Lab, Tutorial] image: "thumbnail.jpg" format: html: code-fold: false code-tools: true toc: true toc-depth: 3 --- ::: {.callout-tip} ## TL;DR - What You'll Build {.unlisted} By the end of this guide, you will have: - **NVIDIA Isaac Sim 5.1.0** running in Docker (robotics simulator) - **NVIDIA Isaac Lab** for reinforcement learning experiments - **A launcher script** that handles everything - NGC login, image downloads, setup - **A reproducible setup** you can share with others **Time required:** ~2 hours (mostly waiting for downloads) **The Easy Path:** After installing Docker (Section 3), download our launcher script (v2.2.0) - it handles everything else! ```bash wget https://raw.githubusercontent.com/myidentity/physical-ai-lab/main/scripts/auto_start.sh chmod +x auto_start.sh ./auto_start.sh ``` ::: --- ## 1. Introduction {#sec-intro} ### Why This Guide Exists When I started learning robotics simulation, I hit wall after wall: - Documentation assumed I knew things I didn't - Tutorials were outdated or incomplete - Small configuration mistakes wasted hours This guide documents **every step** - including the mistakes - so you don't have to struggle like I did. ### What You'll Learn | Section | What You'll Do | Can Skip If... | |---------|---------------|----------------| | [2. Prerequisites](#sec-prerequisites) | Install NVIDIA drivers | Already have working drivers | | [3. Docker Setup](#sec-docker-setup) | Install Docker & NVIDIA toolkit | Already have Docker with GPU | | [4. Quick Start](#sec-quickstart) | Use launcher script for easy setup | Want manual control | | [5. Verification](#sec-verification) | Test everything works | Confident it works | | [6. Troubleshooting](#sec-troubleshooting) | Fix common problems | Everything works | | [Appendix A](#sec-manual) | Manual installation steps | Using the launcher | | [Appendix B](#sec-concepts) | Docker & Isaac Sim concepts | Already know Docker | --- ## 2. Prerequisites {#sec-prerequisites} ### Hardware Requirements ::: {.callout-important} ## Minimum Hardware {.unlisted} - **GPU**: NVIDIA RTX series (RTX 2070 or better) - **VRAM**: 8GB minimum, 12GB+ recommended - **RAM**: 32GB recommended - **Storage**: 50GB+ free space (Docker images are large!) - **OS**: Ubuntu 22.04 or 24.04 LTS ::: ### Install NVIDIA Drivers First, check if drivers are already installed: ```bash nvidia-smi ``` ::: {.callout-tip} ## What You Should See {.unlisted} If drivers are installed, you'll see a table showing your GPU model, driver version, and CUDA version. If you see `command not found`, follow the steps below. ::: **If `nvidia-smi` fails, install drivers:** ```bash # Step 1: Update your system sudo apt update && sudo apt upgrade -y # Step 2: See which drivers are available for your GPU sudo ubuntu-drivers list --gpgpu # Step 3: Install the recommended driver automatically sudo ubuntu-drivers install --gpgpu # Step 4: Reboot (required!) sudo reboot ``` After reboot, verify: ```bash nvidia-smi ``` ::: {.callout-warning} ## Secure Boot Users {.unlisted} If your computer uses Secure Boot, you'll see extra steps during installation. Ubuntu will ask you to create a temporary password. After reboot, a blue "MOK Management" screen appears - select "Enroll MOK", enter the password, and reboot. If you miss this screen, reinstall: `sudo ubuntu-drivers install --gpgpu` ::: ### Our Test System This guide was tested on: | Component | Specification | |-----------|--------------| | **OS** | Ubuntu 24.04.3 LTS | | **Kernel** | 6.14.0-37-generic | | **GPU** | NVIDIA GeForce RTX 5090 Laptop GPU | | **Driver** | 580.95.05 | | **Docker** | 29.1.2 | --- ## 3. Docker Setup {#sec-docker-setup} ::: {.callout-tip} ## Goal {.unlisted} Install Docker and configure it to use your NVIDIA GPU. This is required before using the launcher script. ::: ### Step 3.1: Install Docker ```bash # Update package list sudo apt update # Install Docker sudo apt install -y docker.io docker-compose-v2 # Add yourself to docker group (avoids needing sudo) sudo usermod -aG docker $USER ``` ::: {.callout-warning} ## Important! {.unlisted} After adding yourself to the docker group, you must **log out and log back in** (or reboot) for it to take effect. Alternatively, run this to apply immediately (current terminal only): ```bash newgrp docker ``` ::: ### Step 3.2: Verify Docker Works ```bash docker run hello-world ``` You should see "Hello from Docker!" message. ### Step 3.3: Install NVIDIA Container Toolkit This lets Docker containers access your GPU: ```bash # Add NVIDIA repository curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \ sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \ sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list # Install the toolkit sudo apt update sudo apt install -y nvidia-container-toolkit # Configure Docker to use NVIDIA runtime sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker ``` ### Step 3.4: Verify GPU Access in Docker ```bash docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi ``` You should see the same `nvidia-smi` output as on your host. ::: {.callout-warning} ## Common Error: "could not select device driver" {.unlisted} If you see this error: ```bash sudo systemctl restart docker ``` Still not working? Reboot your computer. ::: --- ## 4. Quick Start with Launcher Script {#sec-quickstart} ::: {.callout-tip} ## The Easy Way {.unlisted} Now that Docker is set up, use our launcher script to handle everything else: NGC login, downloading Isaac Sim, cloning Isaac Lab, and running containers. ::: ### Download the Launcher Current version: **v2.2.0** ```bash cd ~ wget https://raw.githubusercontent.com/myidentity/physical-ai-lab/main/scripts/auto_start.sh chmod +x auto_start.sh ``` ### Run the Launcher ```bash ./auto_start.sh ``` You'll see a menu like this: ``` ═══════════════════════════════════════════════════════ Isaac Sim 5.1.0 & Isaac Lab Launcher v2.2.0 ═══════════════════════════════════════════════════════ ─────────── SETUP (First Time) ───────────────────────── s) NGC Login (for pulling images) [manual check] p) Pull Isaac Sim 5.1.0 Image (~15GB) [NOT FOUND] c) Clone Isaac Lab Repository [NOT FOUND] ─────────── 🎮 ISAAC SIM (Standalone Simulation) ───────── Start Simulator: 1) With GUI (Desktop Mode) 2) Headless (No Display) ... 💾 Save Your Work: 7) Save Installed Packages ⭐ ─────────── 🤖 ISAAC LAB (RL Training Environment) ─────── Start & Access: 8) Start/Build Container ⭐ 9) Enter Container 🛑 Shutdown Options: h) Hibernate (Safe) ⭐ - Preserves everything a) DELETE Everything ⚠️ - Destroys all data ─────────── ⚙️ UTILITIES ───────────────────────────────── 0) Toggle Display Driver (Dummy ↔ NVIDIA) b) Python Package Guide ``` ### First-Time Setup Walkthrough Follow these steps in order: **1. NGC Login (Option 's'):** You need an NVIDIA NGC account to download Isaac Sim: 1. Go to [https://ngc.nvidia.com](https://ngc.nvidia.com) and create a free account 2. Go to **Setup** → **API Key** and generate a new key 3. Select option **'s'** in the launcher ![NGC login screen](images/02_ngc_login_screen.png) When prompted: - **Username**: `$oauthtoken` (literally type this) - **Password**: Your API key from NGC ![NGC login success](images/03_ngc_login_success.png) **2. Pull Isaac Sim Image (Option 'p') - ~15GB download:** ![Image pull starting](images/04_image_pull_start.png) ::: {.callout-note} ## Coffee Time {.unlisted} This download takes 15-30 minutes depending on your internet speed. ::: ![Image pull progress](images/05_image_pull_progress.png) ![Image pull complete](images/06_image_pull_complete.png) **3. Clone Isaac Lab (Option 'c'):** ![Cloning Isaac Lab](images/07_clone_isaaclab.png) ![Clone complete](images/08_clone_complete.png) **4. Menu after setup complete - all components [FOUND]:** ![Main menu after setup](images/09_main_menu_after.png) ### Launcher Menu Reference #### Setup Options | Option | Name | Description | |--------|------|-------------| | **s** | NGC Login | Login to NVIDIA GPU Cloud (required to pull images) | | **p** | Pull Isaac Sim Image | Download Isaac Sim 5.1.0 (~15GB) | | **c** | Clone Isaac Lab | Clone Isaac Lab repository | #### 🎮 Isaac Sim Options | Option | Name | Description | |--------|------|-------------| | **1** | With GUI | Launch Isaac Sim with graphical interface | | **2** | Headless | Launch without display (for servers/remote) | | **3** | Compatibility Check | Verify system meets requirements | | **4** | Setup Directories | Create/reset cache directories | | **5** | System Information | View GPU, storage, and image info | | **6** | Check Prerequisites | Verify Docker, GPU access, etc. | | **7** | Save Installed Packages ⭐ | Save pip-installed packages permanently | ::: {.callout-tip} ## Option 7: Save Your Packages When you `pip install` packages inside Isaac Sim, they're lost when you exit. Use **Option 7** while the container is running to save them permanently to a custom image. ::: #### 🤖 Isaac Lab Options | Option | Name | Description | |--------|------|-------------| | **8** | Start/Build Container | Build and start the Isaac Lab Docker container | | **9** | Enter/Resume Container | Enter running container or resume hibernated one | | **h** | Hibernate (Safe) ⭐ | Safely pause container - preserves everything | | **a** | DELETE Everything ⚠️ | Destroys container + all training logs | ::: {.callout-important} ## Safe Shutdown vs Destructive Delete - **Option h (Hibernate)**: Use this before shutting down your computer. Everything is preserved - training logs, installed packages, container state. Resume with Option 9 when you return. - **Option a (DELETE)**: Only use when you want a clean slate or need to free disk space. This **permanently deletes** all training logs and container data! Requires typing "DELETE" to confirm. ::: #### ⚙️ Utilities | Option | Name | Description | |--------|------|-------------| | **0** | Toggle Display Driver | Switch between NVIDIA (HDMI) and Dummy (headless) | | **b** | Python Package Guide | Instructions for managing packages | ::: {.callout-note} ## Isaac Lab vs Isaac Sim {.unlisted} - **Isaac Sim** (Options 1-7): The base simulator for rendering and physics - **Isaac Lab** (Options 8, 9, h, a): Reinforcement learning framework built on Isaac Sim ::: --- ## 5. Verification {#sec-verification} ### Run Compatibility Check Select **Option 3** from the launcher to verify your system: ![Compatibility check showing PASSED](images/10_compatibility_check.png) Look for **"System checking result: PASSED"** at the bottom. ### Start Isaac Sim Select **Option 1** to start Isaac Sim with GUI. You'll enter an interactive shell: ![Container shell before running Isaac Sim](images/11_container_before_runapp.png) Inside the container, run `./runapp.sh` to start the simulator: ![Isaac Sim GUI running](images/12_isaac_sim_gui.png) ::: {.callout-note} ## First-Time Startup {.unlisted} The first time you run Isaac Sim, it compiles shaders for your GPU. This can take **5-10 minutes**. Subsequent starts are much faster. ::: ### Your First Simulation Once Isaac Sim is running: 1. Go to **Window** → **Browsers** → **Isaac Sim Assets** 2. Search for **"franka"** or browse the **Robots** category 3. Double-click a Franka robot to load it 4. Click **Play** to start simulation ![Franka robot loaded in Isaac Sim](images/13_isaac_sim_franka.png) ### Start Isaac Lab Now let's verify Isaac Lab works by training a reinforcement learning agent! **Step 1: Start Isaac Lab Container** Select **Option 8** from the launcher to build and start the Isaac Lab container: ``` ┌─────────────────────────────────────────────────────────────────┐ │ Isaac Sim 5.1.0 & Isaac Lab Launcher │ ├─────────────────────────────────────────────────────────────────┤ │ ISAAC LAB OPTIONS │ │ 8) Start/Build Container [Run Isaac Lab] │ │ 9) Quick Access │ │ a) Stop Container │ └─────────────────────────────────────────────────────────────────┘ ``` ::: {.callout-note} ## First-Time Build {.unlisted} The first time you run Option 8, Isaac Lab will build its Docker container. This takes **10-15 minutes** as it installs all RL dependencies (PyTorch, RSL-RL, etc.). Subsequent starts are instant. ::: **Step 2: Run Cartpole Training** Once inside the Isaac Lab container, you'll see the workspace prompt: ```bash root@physical-ai-lab:/workspace/isaaclab# ``` Run the Cartpole training with 64 parallel environments: ```bash ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --num_envs=64 ``` **What the command does:** | Part | Meaning | |------|---------| | `./isaaclab.sh -p` | Run a Python script through Isaac Lab | | `scripts/reinforcement_learning/rsl_rl/train.py` | The RSL-RL training script | | `--task=Isaac-Cartpole-v0` | The Cartpole balancing task | | `--num_envs=64` | Run 64 environments in parallel | **Step 3: Watch the Training!** You should see 64 cartpoles training simultaneously - each one learning to balance the pole through trial and error! ![Isaac Lab running Cartpole training with 64 parallel environments](images/16_isaac_lab_cartpole.png) ::: {.callout-tip} ## What You're Seeing {.unlisted} - **64 parallel environments**: The GPU simulates all 64 cartpoles simultaneously - **Real-time learning**: Watch the poles start wobbly and gradually stabilize - **Training metrics**: The terminal shows reward values increasing as learning progresses ::: ### Other Isaac Lab Commands ```bash # List all available environments ./isaaclab.sh -p scripts/environments/list_envs.py # Train a different task (Humanoid) ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Humanoid-v0 --num_envs=64 # Train with more environments (if you have VRAM) ./isaaclab.sh -p scripts/reinforcement_learning/rsl_rl/train.py --task=Isaac-Cartpole-v0 --num_envs=256 ``` 🎉 **Congratulations!** You now have a fully working Physical AI Laboratory with both Isaac Sim and Isaac Lab! --- ## 6. Troubleshooting {#sec-troubleshooting} ### GPU Not Detected in Container **Symptom**: `nvidia-smi` works on host but not in container ```bash # Restart Docker sudo systemctl restart docker # If still not working, reboot sudo reboot ``` ### Display Errors **Symptom**: "Cannot open display" or blank window ```bash # Allow local connections to X server xhost +local: # Check DISPLAY variable echo $DISPLAY ``` ### Permission Denied **Symptom**: "Permission denied" when running Docker ```bash # Add yourself to docker group sudo usermod -aG docker $USER # Log out and log back in ``` ### Isaac Sim Won't Start **Symptom**: Container exits immediately ```bash # Check Docker logs docker logs <container_id> # Run with verbose output docker run --rm -it --gpus all nvcr.io/nvidia/isaac-sim:5.1.0 bash ./isaac-sim.sh --verbose ``` ### Out of Disk Space ```bash # Check disk space df -h # Clean up Docker (removes ALL unused images) docker system prune -a ``` --- ## 7. Resources {#sec-resources} | Resource | Description | |----------|-------------| | [Isaac Sim Documentation](https://docs.isaacsim.omniverse.nvidia.com/) | Complete user guide and API reference | | [Isaac Lab Documentation](https://isaac-sim.github.io/IsaacLab/) | RL environments and training guides | | [NVIDIA Developer Forums](https://forums.developer.nvidia.com/c/omniverse/simulation/69) | Community support | | [NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/isaac-sim) | Official container images | --- ## Conclusion You now have a fully functional Physical AI laboratory! **What you've accomplished**: - Docker with GPU support - Isaac Sim 5.1.0 for robotics simulation - Isaac Lab for reinforcement learning - A launcher script for easy access **Remember**: - Use `./auto_start.sh` for easy access - First runs are slow (shader compilation) - Check troubleshooting if things go wrong Happy simulating! --- # Appendices {.unnumbered} ## Appendix A: Manual Installation {#sec-manual} ::: {.callout-note} ## When to Use This {.unlisted} These manual steps are an **alternative** to using the launcher script. Use this if you prefer full control or need to customize the setup. ::: ### A.1: NGC Login (Manual) ```bash docker login nvcr.io ``` When prompted: - **Username**: `$oauthtoken` - **Password**: Your NGC API key ### A.2: Create Directory Structure ```bash mkdir -p ~/isaac-sim/{cache/kit,cache/ov,cache/pip,cache/glcache,cache/computecache,logs,data,documents} ``` ### A.3: Pull Isaac Sim Image ```bash docker pull nvcr.io/nvidia/isaac-sim:5.1.0 ``` ### A.4: Run Isaac Sim Manually ```bash xhost +local: docker run --rm -it --gpus all \ -e DISPLAY \ -e PRIVACY_CONSENT=Y \ -v $HOME/.Xauthority:/root/.Xauthority \ -v ~/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw \ -v ~/isaac-sim/cache/ov:/root/.cache/ov:rw \ -v ~/isaac-sim/cache/pip:/root/.cache/pip:rw \ -v ~/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw \ -v ~/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw \ -v ~/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw \ -v ~/isaac-sim/data:/root/.local/share/ov/data:rw \ -v ~/isaac-sim/documents:/root/Documents:rw \ --network=host \ nvcr.io/nvidia/isaac-sim:5.1.0 ``` ### A.5: Clone Isaac Lab Manually ```bash mkdir -p ~/docker/isaac-lab cd ~/docker/isaac-lab git clone https://github.com/isaac-sim/IsaacLab.git cd IsaacLab ``` ### A.6: Build and Start Isaac Lab ```bash python3 docker/container.py start python3 docker/container.py enter ``` --- ## Appendix B: Core Concepts {#sec-concepts} ::: {.callout-note} ## Optional Reading {.unlisted} This section explains Docker and Isaac Sim concepts. Skip if you're already familiar. ::: ### What is Docker? Think of Docker as a **shipping container for software**: ``` ┌─────────────────────────────────────────────────┐ │ Your Computer │ │ ┌─────────────────┐ ┌─────────────────┐ │ │ │ Container 1 │ │ Container 2 │ │ │ │ Isaac Sim │ │ Isaac Lab │ │ │ │ Python 3.11 │ │ PyTorch, etc │ │ │ └─────────────────┘ └─────────────────┘ │ │ Docker Engine (manages containers) │ └─────────────────────────────────────────────────┘ ``` **Why use Docker?** 1. **Reproducibility**: Same environment everywhere 2. **Isolation**: Won't mess up your system Python 3. **Easy cleanup**: Delete container, everything gone 4. **NVIDIA support**: GPU access inside containers ### What is Isaac Sim? **NVIDIA Isaac Sim** is a robotics simulator that: - Renders photorealistic environments - Simulates accurate physics - Supports many robot models - Runs on NVIDIA GPUs ### What is Isaac Lab? **Isaac Lab** builds on Isaac Sim and adds: - Reinforcement learning training - Pre-built robot environments - Training algorithms (PPO, SAC, etc.) - Parallel environment simulation ### Key Terms | Term | Meaning | |------|---------| | **Container** | Isolated environment running an application | | **Image** | Template used to create containers | | **Volume** | Persistent storage that survives container restarts | | **NGC** | NVIDIA GPU Cloud - where NVIDIA hosts container images |