Agentic DT

①Data Generation — Simulation Capture

Inside MuJoCo, PickSkill performs the grasp automatically while every joint state, gripper state, camera frame, and the high-level instruction are recorded frame by frame. Successful episodes are kept; failed ones are deleted outright.

vector-os-nano/scripts/collection/tasks/sim/collect_cube_pick.py vector-os-nano/vector_os_nano/data/recorder.py

1.1 One episode, end to end

1.2 Sampling positions & diversity

The default run is --episodes=50. Five fixed XY positions are cycled in blocks of RUNS_PER_POSITION=2 (the position index advances every 2 episodes), so 50 episodes yield 10 per XY. The file also contains a deprecated hand-written trajectory _run_episode() (with XY/timing noise) kept only for reference — not used for new captures.

CUBE_POSITIONS_XY = [
    (0.18,  0.00),   # close center
    (0.22,  0.12),   # center-left
    (0.22, -0.12),   # center-right
    (0.28,  0.06),   # far-left
    (0.28, -0.06),   # far-right
]
RUNS_PER_POSITION = 2          # collect_cube_pick.py:70–77
# pos_idx = ep_idx // RUNS_PER_POSITION  → block of 2 episodes per XY

1.3 sim_sync — atomic state/image alignment

The hard part of capture is keeping images and joint state aligned. Simulation uses mode="sim_sync": no background threads — a hook fires on each MuJoCo step and, from the same mjData, renders the cameras and reads the joints together, so the image-to-state time delta is exactly 0.

1.4 Output: raw_episodes/ layout

raw_episodes/
└── episode_0000/
    ├── meta.json              # instruction, spec_version, t_start_ns, sampling_mode, cube_initial_pos …
    ├── commands.parquet       # every move_joints / gripper command + timestamp
    ├── observations.parquet   # 30Hz: 5 joint angles + 1 gripper + t_ns + t_sim_s
    └── videos/
        ├── camera1.mp4        # overhead   (camera1 = OBS_IMAGE_1)
        ├── camera1_meta.parquet
        ├── camera2.mp4        # wrist_cam  (camera2 = OBS_IMAGE_2)
        ├── camera2_meta.parquet
        ├── camera3.mp4        # front      (camera3 = OBS_IMAGE_3)
        └── camera3_meta.parquet

Camera-name mapping: CAMERA_MAP (collect_cube_pick.py:93). MP4 encoded via imageio-ffmpeg (libx264, crf 23).

②Export & Convert — raw → LeRobot v3

Convert raw_episodes/ into a standard LeRobot Dataset v3. This step is task-agnostic; its core job is to resample each episode onto a strict 30Hz timeline and to generate the action labels used for training.

vector-os-nano/scripts/collection/export_to_lerobot.py vector-os-nano/vector_os_nano/data/exporter.py

2.1 30Hz timeline resampling

A uniform set of timestamps is generated across the observation span; at each timestamp the nearest observation and image are taken. The exporter handles arbitrary raw episode timing in general; note that current sim_sync cube_pick data is already uniformly stamped at 30Hz, so the resampling is close to a pass-through for this data.

t_start, t_end = obs_ts[0], obs_ts[-1]
period_ns = 1e9 / fps                       # fps = 30
n_frames  = max(1, (t_end - t_start) / period_ns)
frame_times = [t_start + i*period_ns for i in range(n_frames)]
# per frame_time: nearest observation → observation.state (6,)
#                 nearest image       → observation.images.*
#                                       exporter.py:160–217

2.2 Two timestamp domains

2.3 Action label — next_state mode

Default --action-mode next_state: the next observed state is used as the current action target.

2.4 Output: LeRobot v3 dataset layout

$DATASET_ROOT/                            # = export_to_lerobot.py --output-root
├── meta/
│   ├── info.json            # feature spec (observation.state / action / images)
│   ├── stats.json           # per-dimension stats (used for normalization at train time)
│   ├── tasks.parquet        # instruction text table
│   └── episodes/chunk-000/file-000.parquet
├── data/chunk-000/file-000.parquet      # per-frame state / action / action_is_pad
└── videos/observation.images.<cam>/chunk-000/file-000.mp4

Built by LeRobotDataset.create(robot_type="so101_follower"); frames added via dataset.add_frame(), finalized with dataset.save_episode(). Feature spec: _build_features() (exporter.py:246).

③Training — SmolVLA + LoRA

Fine-tune SmolVLA on the LeRobot dataset with lerobot-train. Base lerobot/smolvla_base, LoRA r=32, inference chunk of 50 steps. The three run environments share the same hyperparameters.

scripts/run_pipeline.sh scripts/train_cube_pick.sh hpc/train_cube_pick.job

3.1 Core training command

lerobot-train \
  --policy.type=smolvla \
  --policy.pretrained_path=lerobot/smolvla_base \
  --policy.push_to_hub=false \
  --dataset.repo_id=local/cube_pick \
  --dataset.root=$DATASET_ROOT \
  --output_dir=$CKPT_DIR \
  --steps=100000  --batch_size=64 \
  --save_freq=10000  --eval_freq=0 \
  --policy.use_amp=true \
  --policy.chunk_size=50 \
  --peft.method_type=LORA  --peft.r=32

3.2 OOM auto-downgrade (shell retry, not a trainer feature)

The script first tries bs=64. If the process fails and the log matches the literal lowercase string "out of memory", it retries at bs=32, then bs=16. To keep the effective number of gradient updates constant, steps scale up proportionally.

steps = BASE_STEPS * BASE_BS / bs        # BASE_STEPS=100000, BASE_BS=64
  bs=64 → 100000 steps
  bs=32 → 200000 steps   (×2)
  bs=16 → 400000 steps   (×4)
# detect + retry: run_pipeline.sh:110–123 / train_cube_pick.sh:67–82

3.3 Three run environments

④Evaluation — closed-loop rollout

After training, eval_and_serve.py runs a closed-loop evaluation in simulation: it loads the LoRA checkpoint, rolls out over several seeds, records videos, computes a success rate, and (by default) starts a web viewer.

scripts/eval_and_serve.py

python scripts/eval_and_serve.py \
    --model $CKPT_DIR/checkpoints/<step>/pretrained_model \
    --instruction "pick up the red cube" \
    --seeds 0 1 2 3 4 --max-steps 150 \
    --out-dir $EVAL_DIR