Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Logic
|
5ae9f5fa48 |
@@ -0,0 +1 @@
|
||||
{"reason":"idle timeout","timestamp":1776932392614}
|
||||
@@ -0,0 +1 @@
|
||||
31521
|
||||
@@ -0,0 +1,48 @@
|
||||
<h2>新任务场景草图：双臂空中穿环插入</h2>
|
||||
<p class="subtitle">保持双 Diana + 同一张桌子；左侧放环状木块，右侧放长条木块，先各自抓起，再在空中完成穿孔插入。</p>
|
||||
|
||||
<div class="split">
|
||||
<div class="mockup">
|
||||
<div class="mockup-header">俯视布局（初始）</div>
|
||||
<div class="mockup-body">
|
||||
<div style="position:relative;height:340px;background:#f3f4f6;border-radius:16px;border:1px solid #d1d5db;overflow:hidden;">
|
||||
<div style="position:absolute;left:12%;top:18%;width:22%;height:64%;border:2px dashed #60a5fa;border-radius:18px;background:rgba(96,165,250,.08);display:flex;align-items:flex-start;justify-content:center;padding-top:8px;font-weight:700;">左臂采样区（环）</div>
|
||||
<div style="position:absolute;right:12%;top:18%;width:22%;height:64%;border:2px dashed #f59e0b;border-radius:18px;background:rgba(245,158,11,.08);display:flex;align-items:flex-start;justify-content:center;padding-top:8px;font-weight:700;">右臂采样区（长条）</div>
|
||||
<div style="position:absolute;left:50%;top:50%;transform:translate(-50%,-50%);width:120px;height:120px;border:10px solid #ef4444;background:white;box-sizing:border-box;display:flex;align-items:center;justify-content:center;font-weight:700;color:#ef4444;">环块</div>
|
||||
<div style="position:absolute;left:72%;top:50%;transform:translate(-50%,-50%) rotate(0deg);width:110px;height:22px;background:#10b981;border:2px solid #065f46;border-radius:6px;display:flex;align-items:center;justify-content:center;color:white;font-size:12px;font-weight:700;">长条</div>
|
||||
<div style="position:absolute;left:50%;top:12%;transform:translateX(-50%);padding:6px 10px;background:#111827;color:white;border-radius:999px;font-size:12px;">桌面中心 / 空中插入区上方</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="mockup">
|
||||
<div class="mockup-header">任务阶段</div>
|
||||
<div class="mockup-body">
|
||||
<div class="section">
|
||||
<div class="label">Phase 1</div>
|
||||
<h3>左右臂分别接近目标</h3>
|
||||
<p>左臂只抓环状木块，右臂只抓长条木块。两物体初始姿态固定，只随机平面位置。</p>
|
||||
</div>
|
||||
<div class="section">
|
||||
<div class="label">Phase 2</div>
|
||||
<h3>同时抓起并离桌</h3>
|
||||
<p>两个物体都需要离开桌面后，才允许触发最终成功奖励。</p>
|
||||
</div>
|
||||
<div class="section">
|
||||
<div class="label">Phase 3</div>
|
||||
<h3>空中对齐并插入</h3>
|
||||
<p>在桌面上方的会合区完成对齐，让长条沿自身长轴穿过环孔；一旦穿孔成功即判成功。</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h3>奖励草案（总分 5）</h3>
|
||||
<div class="cards">
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>左臂碰到环块</p></div></div>
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>右臂碰到长条</p></div></div>
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>环块离桌</p></div></div>
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>长条离桌</p></div></div>
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>两者离桌时，长条穿过环孔</p></div></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -0,0 +1 @@
|
||||
<div style="display:flex;align-items:center;justify-content:center;min-height:60vh"><p class="subtitle">Continuing in terminal...</p></div>
|
||||
@@ -0,0 +1,39 @@
|
||||
# Repository Guidelines
|
||||
|
||||
## Project Structure & Module Organization
|
||||
- `roboimi/` is the main package: simulation code in `roboimi/envs/`, assets/helpers in `roboimi/assets/`, VLA models/config in `roboimi/vla/`, and runnable scripts in `roboimi/demos/` plus `roboimi/demos/vla_scripts/`.
|
||||
- `tests/` holds `unittest` regressions for training, rollout, dataset loading, and asset resolution.
|
||||
- `docs/superpowers/` stores design specs and plans.
|
||||
- `docs/experiment-guide.md` store how beginning experiment
|
||||
- `runs/`, `outputs/`, `swanlog/`, `.worktrees/`, and `checkpoints/` are generated experiment artifacts, not source modules.
|
||||
|
||||
## Build, Test, and Development Commands
|
||||
- `conda env create -f environment.yml && conda activate roboimi` — create the standard environment.
|
||||
- `pip install -e .` — editable install.
|
||||
- `python -m unittest discover -s tests -p 'test_*.py'` — run all tests.
|
||||
- `python -m unittest tests.test_train_vla_rollout_validation -v` — run one focused test module.
|
||||
- `python roboimi/vla/scripts/calculate_stats.py` — regenerate normalization stats.
|
||||
- `python roboimi/demos/vla_scripts/train_vla.py train.max_steps=1000` — quick training smoke test.
|
||||
- `python roboimi/demos/vla_scripts/eval_vla.py eval.ckpt_path=checkpoints/vla_model_best.pt eval.num_episodes=5` — evaluate a checkpoint.
|
||||
|
||||
## Coding Style & Naming Conventions
|
||||
- Use Python, 4-space indentation, and PEP 8–style naming.
|
||||
- Prefer `snake_case` for files, functions, variables, and Hydra keys; use `CamelCase` for classes.
|
||||
- Match existing Hydra namespaces such as `train.*`, `eval.*`, and `agent.*`.
|
||||
- Keep diffs focused and imports tidy. Experiment names usually encode host/GPU, e.g. `...-l20g3-...`, `...-5880g1-...`, or `...-5090-...`.
|
||||
|
||||
## Testing Guidelines
|
||||
- Tests use `unittest` and `unittest.mock`.
|
||||
- Name files `tests/test_*.py` and classes `*Test`.
|
||||
- Add direct regression coverage for changes to training, rollout, config, or dataset code paths.
|
||||
- No strict coverage target is defined, but changed behavior should be tested and affected suites should pass before review.
|
||||
|
||||
## Commit & Pull Request Guidelines
|
||||
- Follow existing prefixes: `feat`, `fix`, `chore`, `docs`, `debug`, optionally with scope, e.g. `feat(eval): export rollout video timing`.
|
||||
- Keep commit messages imperative and specific.
|
||||
- PRs should state the goal, touched configs/scripts, exact test commands, and dataset/checkpoint assumptions. Attach visual artifacts only when outputs change.
|
||||
|
||||
## Experiment Host Reference
|
||||
- **Local workstation (`droid-z790eagleax`)** — local shell; **1× RTX 5090 32 GB**; env: `/home/droid/.conda/envs/roboimi/bin/python`.
|
||||
- **5880 node (`droid-System-Product-Name`)** — username: `droid`; IP: `100.73.14.65`; connect with `ssh droid@100.73.14.65`; **2× RTX 5880 Ada 48 GB**; env: `/home/droid/miniforge3/envs/roboimi/bin/python`; trusted tailnet host.
|
||||
- **L20 node (`droid-G5500-V7`)** — username: `droid`; IP: `100.119.99.14`; connect with `ssh droid@100.119.99.14`; **8× NVIDIA L20 46 GB**; env: `/home/droid/miniforge3/envs/roboimi/bin/python`; store large experiment data under `/data`.
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
Mon Mar 30 13:35:31 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 13:35:31 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 13:39:46 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 13:39:46 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 14:30:24 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 14:30:24 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 14:41:26 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 14:41:26 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 15:21:19 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 15:21:19 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 20:58:52 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 20:59:05 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Mon Mar 30 22:05:37 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:00:10 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:00:10 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:03:41 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:03:41 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:05:09 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:05:09 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:06:00 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:06:00 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:09:24 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:09:24 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:14:36 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:14:36 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:20:44 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:23:07 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:29:49 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 11:55:22 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 11:55:22 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sun May 24 21:50:49 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sun May 24 21:51:10 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sun May 24 21:53:08 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sun May 24 21:53:28 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Capture first and last (success) frames from front camera for both sim tasks."""
|
||||
import os
|
||||
import time
|
||||
import numpy as np
|
||||
import cv2
|
||||
from roboimi.envs.double_pos_ctrl_env import make_sim_env
|
||||
from roboimi.demos.diana_policy import TestPickAndTransferPolicy
|
||||
from roboimi.demos.diana_air_insert_policy import TestAirInsertPolicy
|
||||
from roboimi.utils.act_ex_utils import sample_transfer_pose, sample_air_insert_socket_peg_state
|
||||
|
||||
OUTPUT_DIR = "sim_frames"
|
||||
IMG_SIZE = (256, 256)
|
||||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||||
|
||||
|
||||
def capture_task(task_name, policy, task_state, episode_len):
|
||||
env = make_sim_env(task_name, headless=True)
|
||||
env.reset(task_state)
|
||||
time.sleep(1)
|
||||
|
||||
env._update_camera_images_sync()
|
||||
first_frame = cv2.resize(env.front, IMG_SIZE)
|
||||
|
||||
last_success_frame = None
|
||||
max_reward = 0
|
||||
|
||||
for step in range(episode_len):
|
||||
action = policy.predict(task_state, step)
|
||||
env.step(action)
|
||||
env._update_camera_images_sync()
|
||||
if env.rew is not None and env.rew >= max_reward:
|
||||
max_reward = env.rew
|
||||
if env.rew == env.max_reward:
|
||||
last_success_frame = cv2.resize(env.front, IMG_SIZE)
|
||||
|
||||
if last_success_frame is None:
|
||||
last_success_frame = cv2.resize(env.front, IMG_SIZE)
|
||||
print(f" [WARN] {task_name}: max_reward={max_reward}/{env.max_reward}, using last frame")
|
||||
else:
|
||||
print(f" {task_name}: success! max_reward={max_reward}")
|
||||
|
||||
return first_frame, last_success_frame
|
||||
|
||||
|
||||
def main():
|
||||
# Transfer task (episode_len=700, policy covers full range)
|
||||
print("Running sim_transfer...")
|
||||
task_state_transfer = sample_transfer_pose()
|
||||
policy_transfer = TestPickAndTransferPolicy(inject_noise=False)
|
||||
first_t, last_t = capture_task('sim_transfer', policy_transfer, task_state_transfer, 700)
|
||||
cv2.imwrite(os.path.join(OUTPUT_DIR, "transfer_front_first.png"), first_t)
|
||||
cv2.imwrite(os.path.join(OUTPUT_DIR, "transfer_front_last.png"), last_t)
|
||||
|
||||
# Air insert / peg-in-socket (policy EPISODE_END_T=600)
|
||||
print("Running sim_air_insert_socket_peg...")
|
||||
task_state_insert = sample_air_insert_socket_peg_state()
|
||||
policy_insert = TestAirInsertPolicy(inject_noise=False)
|
||||
first_i, last_i = capture_task('sim_air_insert_socket_peg', policy_insert, task_state_insert, 600)
|
||||
cv2.imwrite(os.path.join(OUTPUT_DIR, "air_insert_front_first.png"), first_i)
|
||||
cv2.imwrite(os.path.join(OUTPUT_DIR, "air_insert_front_last.png"), last_i)
|
||||
|
||||
print(f"\nDone! Images saved to {OUTPUT_DIR}/")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,498 @@
|
||||
# 实验操作指南
|
||||
|
||||
## 0. 适用范围与当前起点
|
||||
|
||||
这份文档面向 **当前 `main` 分支** 的通用训练 / 验证 / 评估流程，不绑定某一条历史实验分支。
|
||||
|
||||
当前默认 Hydra 起点：
|
||||
- agent：`resnet_transformer`
|
||||
- data：`simpe_robot_dataset`
|
||||
- eval：`eval`
|
||||
|
||||
当前仓库里常见可选 agent 配置包括：
|
||||
- `resnet_transformer`
|
||||
- `resnet_diffusion`
|
||||
- `resnet_gr00t_dit`
|
||||
- `resnet_imf_attnres`
|
||||
- `resnet_imf_attnres_multitoken`
|
||||
- `siglip2_imf_attnres`
|
||||
- `lewm_imf_attnres`
|
||||
|
||||
说明：
|
||||
- `experiment_suites/` 下的历史目录仍然可以作为参考，但**正文以当前 `main` 的通用流程为准**。
|
||||
- 如果你只是想快速开始，优先从默认 `resnet_transformer` 配置出发，再按实验需求替换 agent。
|
||||
|
||||
---
|
||||
|
||||
## 1. 关键文件与入口
|
||||
|
||||
| 路径 | 作用 |
|
||||
| --- | --- |
|
||||
| `roboimi/demos/vla_scripts/train_vla.py` | 主训练入口；负责数据集、checkpoint、`val/loss`、held-out action MSE、训练期 rollout 验证、SwanLab |
|
||||
| `roboimi/demos/vla_scripts/eval_vla.py` | 单次 rollout / 离线评估入口；支持 headless、summary、trajectory image、video artifact、多进程并行 rollout |
|
||||
| `roboimi/vla/conf/config.yaml` | 全局 Hydra 训练配置 |
|
||||
| `roboimi/vla/conf/data/simpe_robot_dataset.yaml` | 默认数据集配置；数据路径、相机名、图像 resize 都在这里 |
|
||||
| `roboimi/vla/conf/eval/eval.yaml` | eval 默认配置；`eval.ckpt_path`、`eval.num_episodes`、artifact 开关、多进程 rollout 配置都在这里 |
|
||||
| `roboimi/vla/conf/agent/*.yaml` | 可选 agent 配置集合 |
|
||||
| `roboimi/vla/data/simpe_robot_dataset.py` | HDF5 懒加载数据集；支持 `episode_indices` 过滤与 `available_episode_indices` 元信息 |
|
||||
| `roboimi/vla/scripts/calculate_stats.py` | 重算 `dataset_stats.pkl` |
|
||||
| `experiment_suites/` | 历史实验目录；可参考，但不作为当前主流程的唯一事实来源 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 三台机器与环境
|
||||
|
||||
| 机器 | GPU | Python | 常用数据集路径 | 备注 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 本地 `droid-z790eagleax` | 1× RTX 5090 32GB | `/home/droid/.conda/envs/roboimi/bin/python` | `/home/droid/project/diana_sim/sim_transfer` | 适合 smoke、单条主跑、本地调参 |
|
||||
| 5880 节点 `100.73.14.65` | 2× RTX 5880 Ada 48GB | `/home/droid/miniforge3/envs/roboimi/bin/python` | `/home/droid/sim_dataset/sim_transfer` | 适合 2 条并行主跑 |
|
||||
| L20 节点 `100.119.99.14` | 8× NVIDIA L20 46GB | `/home/droid/miniforge3/envs/roboimi/bin/python` | `/data/simtransfer/current` | 适合大规模 grid；建议数据与 run 放 `/data` |
|
||||
|
||||
连接：
|
||||
- 5880：`ssh droid@100.73.14.65`
|
||||
- L20：`ssh droid@100.119.99.14`
|
||||
|
||||
说明：
|
||||
- repo / worktree 路径可能会随当前 checkout 或同步目录变化，**以你当前机器上的实际 repo 路径为准**。
|
||||
- 如果训练目录会被 Hydra 自动切走，建议显式设置 `hydra.run.dir=...` 以固定输出目录。
|
||||
|
||||
---
|
||||
|
||||
## 3. 训练流怎么走
|
||||
|
||||
`train_vla.py` 当前主流程如下：
|
||||
|
||||
1. 读取 Hydra 配置并打印完整 cfg
|
||||
2. 根据配置确定数据集图像 resize：
|
||||
- 默认用 `data.image_resize_shape`
|
||||
- 如果 `agent.vision_backbone.dataset_image_resize_shape` 存在，则优先用 backbone 覆盖值
|
||||
3. 通过 `build_train_val_datasets()` 构建 dataset / train dataset / val dataset
|
||||
- 若设置 `train.val_episode_indices`：按显式 episode 切出 held-out val
|
||||
- 否则按 `train.val_split` 做随机划分
|
||||
4. 用 `DataLoader` 建 train / val loader
|
||||
5. 从 `dataset_dir/dataset_stats.pkl` 读取归一化统计
|
||||
6. 实例化当前指定的 agent：`instantiate(cfg.agent, dataset_stats=...)`
|
||||
7. 可选加载：
|
||||
- `train.pretrained_ckpt`：微调起点
|
||||
- `train.resume_ckpt`：断点续训（支持显式路径或 `auto`）
|
||||
8. 建 optimizer 与 scheduler
|
||||
9. 训练循环里按 `log_freq` 记录 train loss / lr
|
||||
10. 按 `save_freq` 保存 `checkpoints/vla_model_step_*.pt`，并在有 val loader 时计算 `val/loss`
|
||||
11. 若设置了显式 held-out val，并且 `train.action_mse_val_freq_epochs > 0`，则按 epoch 计算 held-out `action MSE`
|
||||
12. 按 `train.rollout_val_freq_epochs` 跑训练期 rollout 验证
|
||||
13. 最后写：
|
||||
- `checkpoints/vla_model_best.pt`
|
||||
- `checkpoints/vla_model_final.pt`
|
||||
|
||||
当前 best model 选择逻辑：
|
||||
- **第一次拿到 rollout reward 之前**：先用 `val_loss`（或 train loss 回退）挑 best
|
||||
- **第一次 rollout 之后**：优先用 `rollout_avg_reward` 挑 best
|
||||
|
||||
输出目录：
|
||||
- 当前代码实际输出写到 **Hydra runtime output dir** 下
|
||||
- checkpoint 默认在：`<hydra_output_dir>/checkpoints/`
|
||||
- 如果不显式指定 `hydra.run.dir`，Hydra 会自动生成目录
|
||||
|
||||
---
|
||||
|
||||
## 4. 验证流怎么走
|
||||
|
||||
### 4.1 常规 `val/loss`
|
||||
|
||||
常规验证有两种来源：
|
||||
|
||||
1. **随机划分验证集**
|
||||
- 用 `train.val_split > 0`
|
||||
- 训练脚本会在保存 checkpoint 时计算 `val/loss`
|
||||
|
||||
2. **显式 held-out episode 验证集**
|
||||
- 用 `train.val_episode_indices=[...]`
|
||||
- 训练集 = 全部 episode - held-out episode
|
||||
- 验证集 = held-out episode
|
||||
- checkpoint 保存时同样会计算 `val/loss`
|
||||
|
||||
说明：
|
||||
- 若 `train.val_split=0.0` 且未设置 `train.val_episode_indices`，则不会有 `val/loss`。
|
||||
|
||||
### 4.2 held-out action MSE
|
||||
|
||||
如果想对固定 held-out episode 做更稳定、可复现的动作预测误差验证，推荐使用：
|
||||
|
||||
- `train.val_split=0.0`
|
||||
- `train.val_episode_indices=[100]`（或你指定的 episode 列表）
|
||||
- `train.action_mse_val_freq_epochs=1`
|
||||
|
||||
当前 `main` 中这套逻辑会：
|
||||
- 每隔 `action_mse_val_freq_epochs` 个 epoch
|
||||
- 对 held-out val loader 调 `agent.predict_action_chunk(...)`
|
||||
- 与 batch 中的 `action` 计算 masked MSE
|
||||
- 如果存在 `action_is_pad`，会自动 mask 掉 padding 部分
|
||||
|
||||
日志 key：
|
||||
- 控制台 / `train_vla.log`：`held-out action MSE`
|
||||
- SwanLab：`val/action_mse`
|
||||
|
||||
重要约束：
|
||||
- `train.action_mse_val_freq_epochs > 0` **必须搭配** `train.val_episode_indices`
|
||||
- 如果只设了 `action_mse_val_freq_epochs` 而没有设 `val_episode_indices`，训练会直接报错
|
||||
|
||||
### 4.3 rollout 验证
|
||||
|
||||
训练内 rollout 验证由：
|
||||
- `train_vla.py -> run_rollout_validation() -> eval_vla._run_eval()`
|
||||
|
||||
当前训练内 rollout 会强制：
|
||||
- `headless=true`
|
||||
- `verbose_action=false`
|
||||
- `record_video=false`
|
||||
- `save_trajectory_image=true`
|
||||
- `trajectory_image_camera_name=front`
|
||||
- `save_summary_json=true`
|
||||
|
||||
当前这套路径是**配置驱动**的：
|
||||
- `train.rollout_device`：默认跟随 `train.device`
|
||||
- `train.rollout_num_workers`：默认 `null`
|
||||
- 当 rollout 设备是 CPU 时，自动退化为 `1`
|
||||
- 当 rollout 设备是 CUDA 时，自动推断为 `min(train.rollout_num_episodes, 8)`
|
||||
- `train.rollout_cuda_devices`：默认 `null`，等价于逻辑 GPU `[0]`
|
||||
- `train.rollout_response_timeout_s`
|
||||
- `train.rollout_server_startup_timeout_s`
|
||||
|
||||
所以现在：
|
||||
- 训练在 CUDA 上时，训练期 rollout 默认也会走 GPU
|
||||
- `rollout_num_workers > 1` 时，会走并行 rollout
|
||||
- 可以是 **单 GPU 多 worker 共用一个 inference server**
|
||||
- 也可以是 **多 GPU 多 server 分摊 worker**
|
||||
|
||||
训练内 rollout artifact 默认落到：
|
||||
- `<hydra_output_dir>/rollout_artifacts/<checkpoint_stem>/`
|
||||
|
||||
常见文件：
|
||||
- `rollout_summary.json`
|
||||
- `rollout_front_ep01_trajectory.png` ...
|
||||
|
||||
日志重点看：
|
||||
- `Epoch X rollout 平均奖励`
|
||||
- `最佳模型已更新`
|
||||
|
||||
---
|
||||
|
||||
## 5. 数据集加载与 `val_episode_indices` 机制
|
||||
|
||||
### 5.1 数据集格式
|
||||
|
||||
`SimpleRobotDataset` 读取 `dataset_dir` 下的 `*.hdf5` / `episode_*.hdf5`，每个 episode 文件至少要有：
|
||||
- `action`
|
||||
- `observations/qpos`
|
||||
- `observations/images/{cam_name}`
|
||||
|
||||
默认数据配置中的相机：
|
||||
- `r_vis`
|
||||
- `top`
|
||||
- `front`
|
||||
|
||||
### 5.2 懒加载行为
|
||||
|
||||
`roboimi/vla/data/simpe_robot_dataset.py` 是按帧懒加载，不会一次性把整套 HDF5 全读进内存。
|
||||
|
||||
它会：
|
||||
- 扫描目录下的 HDF5 文件
|
||||
- 在 worker 内做 HDF5 文件句柄 LRU 缓存
|
||||
- 根据文件名中的 `episode_XXX` 建立 `available_episode_indices`
|
||||
|
||||
### 5.3 `val_episode_indices` 怎么切
|
||||
|
||||
`build_train_val_datasets()` 的逻辑是：
|
||||
|
||||
1. 先 instantiate 一次完整 dataset
|
||||
2. 读取 `dataset.available_episode_indices`
|
||||
3. 检查 `train.val_episode_indices` 是否都存在
|
||||
4. 用 `episode_indices=` 再各 instantiate 一次：
|
||||
- train dataset = 全部 episode - held-out episode
|
||||
- val dataset = 只包含 held-out episode
|
||||
|
||||
因此：
|
||||
- `train.val_episode_indices=[100]` 的意思是把 `episode_100.hdf5` 整个拿去做 held-out val
|
||||
- 如果 episode 不存在，会直接报错
|
||||
- 如果你把所有 episode 都塞进 `val_episode_indices`，也会直接报错，因为训练集会变空
|
||||
|
||||
### 5.4 图像 resize 与返回字段
|
||||
|
||||
dataset 侧 resize 默认来自：
|
||||
- `data.image_resize_shape`
|
||||
- 如果 backbone 额外覆盖，则优先 `agent.vision_backbone.dataset_image_resize_shape`
|
||||
|
||||
当前通用 batch 返回字段包括：
|
||||
- `observation.state`
|
||||
- `observation_is_pad`
|
||||
- `observation.<cam>`
|
||||
- `action`
|
||||
- `action_is_pad`
|
||||
- `task`
|
||||
|
||||
### 5.5 统计文件
|
||||
|
||||
训练和推理都默认依赖 `dataset_stats.pkl`。数据集更新后需要重算：
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/vla/scripts/calculate_stats.py \
|
||||
--dataset_dir /home/droid/project/diana_sim/sim_transfer
|
||||
```
|
||||
|
||||
远端只要把：
|
||||
- Python 路径
|
||||
- `--dataset_dir`
|
||||
|
||||
替换成对应机器上的实际路径即可。
|
||||
|
||||
---
|
||||
|
||||
## 6. SwanLab 行为
|
||||
|
||||
当前默认配置里：
|
||||
- `train.use_swanlab=false`
|
||||
|
||||
如果要开启，通常显式设置：
|
||||
- `train.use_swanlab=true`
|
||||
- `train.swanlab_project=roboimi-vla`
|
||||
- `train.swanlab_run_name=<run_name>`
|
||||
|
||||
`train_vla.py` 当前会记录：
|
||||
- 初始化时上传 `train` / `data` / `agent` 三段 config
|
||||
- 训练中：
|
||||
- `train/loss`
|
||||
- `train/lr`
|
||||
- `train/best_loss`
|
||||
- `train/step`
|
||||
- checkpoint 验证时：
|
||||
- `val/loss`
|
||||
- held-out 数值验证时：
|
||||
- `val/action_mse`
|
||||
- rollout 验证时：
|
||||
- `rollout/avg_reward`
|
||||
- `rollout/epoch`
|
||||
- 训练结束时：
|
||||
- `final/checkpoint_path`
|
||||
- `final/best_checkpoint_path`
|
||||
|
||||
训练期 rollout 生成的前视图轨迹 PNG 会 best-effort 上传到 SwanLab；上传失败只会 warning，不会让训练中断。
|
||||
|
||||
---
|
||||
|
||||
## 7. 并行 rollout 说明
|
||||
|
||||
### 7.1 这套能力现在在哪里
|
||||
|
||||
当前主仓库已经**内置**多进程并行 rollout 能力，入口就是：
|
||||
- `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
|
||||
控制参数：
|
||||
- `eval.num_workers`
|
||||
- `eval.cuda_devices`
|
||||
|
||||
语义：
|
||||
- `eval.num_workers`：环境 worker 数，按 episode 切分
|
||||
- `eval.cuda_devices`：推理 server 绑定到哪些逻辑 GPU
|
||||
|
||||
### 7.2 两种常见模式
|
||||
|
||||
1. **单机单卡，多 worker 共用同一张 GPU**
|
||||
- 典型：5090 只有 1 卡，但想让 4 个 rollout worker 并行跑环境
|
||||
- 形式：`eval.device=cuda eval.num_workers=4 'eval.cuda_devices=[0]'`
|
||||
- 这时是 **1 个 CUDA inference server + 4 个 env worker**
|
||||
|
||||
2. **单机多卡，多 server 分摊 worker**
|
||||
- 典型：5880 / L20 有多卡
|
||||
- 形式：`eval.device=cuda eval.num_workers=8 'eval.cuda_devices=[0,1]'`
|
||||
- worker 会按 round-robin 分到多个 server 上
|
||||
|
||||
### 7.3 操作约束
|
||||
|
||||
- 并行 rollout 依赖 **多进程 eval 路径**，不是 `train.num_workers`
|
||||
- `train.num_workers` 是 DataLoader worker，和 rollout 并行不是一回事
|
||||
- `eval.num_workers > 1` 时必须 `eval.headless=true`
|
||||
- worker 数会自动 cap 到 `eval.num_episodes`
|
||||
- 多 worker 时**不支持**同时导出：
|
||||
- `eval.record_video=true`
|
||||
- `eval.save_trajectory=true`
|
||||
- `eval.save_trajectory_npz=true`
|
||||
- `eval.save_trajectory_image=true` 可以开，适合并行 reward + 定性检查一起做
|
||||
|
||||
---
|
||||
|
||||
## 8. 常用命令模板
|
||||
|
||||
下面的模板都以 **当前 `main` 通用流程** 为准。远端机器只需要替换：
|
||||
- Python 路径
|
||||
- repo 路径
|
||||
- `data.dataset_dir`
|
||||
- `hydra.run.dir`
|
||||
|
||||
### 8.1 本地 smoke train
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.batch_size=8 \
|
||||
train.max_steps=1000 \
|
||||
train.num_workers=4 \
|
||||
train.save_freq=200 \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/smoke-resnet-transformer
|
||||
```
|
||||
|
||||
### 8.2 通用训练模板
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.batch_size=16 \
|
||||
train.lr=0.0001 \
|
||||
train.max_steps=100000 \
|
||||
train.num_workers=4 \
|
||||
train.save_freq=2000 \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.swanlab_run_name=<run_name> \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/<run_name>
|
||||
```
|
||||
|
||||
### 8.3 带 held-out episode 验证的训练模板
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.batch_size=16 \
|
||||
train.lr=0.0001 \
|
||||
train.max_steps=100000 \
|
||||
train.num_workers=4 \
|
||||
train.val_split=0.0 \
|
||||
'train.val_episode_indices=[100]' \
|
||||
train.action_mse_val_freq_epochs=1 \
|
||||
train.rollout_val_freq_epochs=5 \
|
||||
train.rollout_num_episodes=10 \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.swanlab_run_name=<run_name> \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/<run_name>
|
||||
```
|
||||
|
||||
### 8.4 断点续训模板
|
||||
|
||||
`train_vla.py` 支持通过 CLI override 使用 `train.resume_ckpt`：
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.resume_ckpt=auto \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/<run_name>
|
||||
```
|
||||
|
||||
也可以把 `train.resume_ckpt` 换成显式 checkpoint 路径。
|
||||
|
||||
### 8.5 单次离线评估
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/eval_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda eval.device=cuda \
|
||||
eval.ckpt_path=/home/droid/project/roboimi/runs/<run_name>/checkpoints/vla_model_best.pt \
|
||||
eval.num_episodes=10 \
|
||||
eval.headless=true \
|
||||
eval.verbose_action=false \
|
||||
eval.save_summary_json=true \
|
||||
eval.save_trajectory_image=true \
|
||||
eval.trajectory_image_camera_name=front \
|
||||
eval.artifact_dir=/tmp/roboimi_eval_front
|
||||
```
|
||||
|
||||
### 8.6 并行离线评估
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/eval_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda eval.device=cuda \
|
||||
eval.ckpt_path=/home/droid/project/roboimi/runs/<run_name>/checkpoints/vla_model_best.pt \
|
||||
eval.num_episodes=10 \
|
||||
eval.num_workers=4 \
|
||||
'eval.cuda_devices=[0]' \
|
||||
eval.headless=true \
|
||||
eval.verbose_action=false \
|
||||
eval.save_summary_json=true \
|
||||
eval.save_trajectory_image=true \
|
||||
eval.trajectory_image_camera_name=front \
|
||||
eval.artifact_dir=/tmp/roboimi_parallel_eval
|
||||
```
|
||||
|
||||
### 8.7 训练内启用并行 GPU rollout
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.batch_size=16 \
|
||||
train.lr=0.0001 \
|
||||
train.max_steps=100000 \
|
||||
train.num_workers=4 \
|
||||
train.rollout_val_freq_epochs=5 \
|
||||
train.rollout_num_episodes=10 \
|
||||
train.rollout_device=cuda \
|
||||
train.rollout_num_workers=4 \
|
||||
'train.rollout_cuda_devices=[0]' \
|
||||
train.rollout_validate_on_checkpoint=false \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.swanlab_run_name=<run_name> \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/<run_name>
|
||||
```
|
||||
|
||||
### 8.8 重算统计文件
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/vla/scripts/calculate_stats.py \
|
||||
--dataset_dir /home/droid/project/diana_sim/sim_transfer
|
||||
```
|
||||
|
||||
### 8.9 监控日志
|
||||
|
||||
```bash
|
||||
tail -f runs/<run_name>/train_vla.log
|
||||
```
|
||||
|
||||
如果你显式设置了 `hydra.run.dir`，优先直接 tail 对应绝对路径下的日志文件。
|
||||
|
||||
---
|
||||
|
||||
## 9. 操作建议
|
||||
|
||||
- **优先固定 `hydra.run.dir`**，避免输出目录分散在 Hydra 自动生成路径里
|
||||
- 如果只需要常规验证，使用：
|
||||
- `train.val_split > 0`
|
||||
- 如果需要稳定、可复现的 held-out 指标，使用：
|
||||
- `train.val_split=0.0`
|
||||
- `train.val_episode_indices=[...]`
|
||||
- `train.action_mse_val_freq_epochs=1`
|
||||
- `train.num_workers` 是 DataLoader worker，不等于 rollout 并行度
|
||||
- rollout 并行优先看：
|
||||
- `train.rollout_num_workers`
|
||||
- `train.rollout_cuda_devices`
|
||||
- `eval.num_workers`
|
||||
- `eval.cuda_devices`
|
||||
- 数据集更新后记得重算 `dataset_stats.pkl`
|
||||
- 若同时设置了：
|
||||
- `train.pretrained_ckpt`
|
||||
- `train.resume_ckpt`
|
||||
当前逻辑会优先走 `resume_ckpt`
|
||||
- 建议 run name 中带上关键信息：
|
||||
- agent / horizon / batch size / lr / host / gpu / date
|
||||
- `experiment_suites/` 下的历史目录适合作为参考资料，但不要默认把某一条旧 suite 当作当前主流程的唯一标准
|
||||
@@ -0,0 +1,256 @@
|
||||
# Align ResNet Transformer Diffusion To External Repo Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Do not switch to inline execution. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 将当前仓库的 `resnet_transformer` 对齐到 external repo `/home/droid/project/diffusion_policy` 的原生 DDPM Transformer Diffusion（非 PMF、非 DiT、非 UNet）实现，并采用 external 中已存在的 full-attention / nocausal 变体（`causal_attn=false`），固定为三相机图像条件输入，同时保持 EE action 语义的推理执行路径正确。
|
||||
|
||||
**Architecture:** 保留当前仓库较轻量的训练脚本和数据集组织，但把 Transformer denoiser 本体对齐到 external repo 的 `TransformerForDiffusion` 实现与接口语义；视觉编码器保留当前仓库的 ResNet+SpatialSoftmax 路线，仅保证其输出维度与三相机条件输入兼容。评估路径统一按 EE action 调 `env.step(action)`，训练/推理配置显式固定三相机 `r_vis/top/front` 且图像永远作为条件输入。
|
||||
|
||||
**Tech Stack:** Python, PyTorch, diffusers DDPM/DDIM, Hydra/OmegaConf, unittest, h5py, OpenCV
|
||||
|
||||
---
|
||||
|
||||
### Task 0: 执行前提与分支约束
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: Confirm execution stays on the feature branch**
|
||||
|
||||
Run: `git branch --show-current`
|
||||
Expected: `feat-align-dp-transformer-ee`
|
||||
|
||||
If the current branch is not `feat-align-dp-transformer-ee`, create/switch to it before continuing:
|
||||
|
||||
Run: `git checkout -b feat-align-dp-transformer-ee || git checkout feat-align-dp-transformer-ee`
|
||||
Expected: working branch becomes `feat-align-dp-transformer-ee`
|
||||
|
||||
- [ ] **Step 2: Confirm implementation is executed with subagents**
|
||||
|
||||
Use `superpowers:subagent-driven-development` for implementation tasks and reviews. Do not switch to inline/manual execution unless the human explicitly changes course.
|
||||
|
||||
- [ ] **Step 3: Confirm the current branch is the only target branch for this work**
|
||||
|
||||
Do not create or switch to another branch/worktree during implementation unless the human explicitly asks for it. All changes for this migration stay on `feat-align-dp-transformer-ee`.
|
||||
|
||||
### Task 1: 验证 EE action 推理执行语义已固定
|
||||
|
||||
**Files:**
|
||||
- Verify: `roboimi/vla/eval_utils.py`
|
||||
- Verify: `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
- Verify: `tests/test_eval_vla_execution.py`
|
||||
|
||||
- [ ] **Step 1: Verify the test still passes on the feature branch**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_eval_vla_execution -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 2: Verify the real eval script no longer executes joint-action stepping**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python - <<'PY'
|
||||
from pathlib import Path
|
||||
src = Path('roboimi/demos/vla_scripts/eval_vla.py').read_text()
|
||||
assert 'execute_policy_action(env, action)' in src
|
||||
assert 'env.step_jnt(action)' not in src
|
||||
print('eval_vla execution path verified')
|
||||
PY
|
||||
```
|
||||
Expected: PASS
|
||||
|
||||
### Task 2: 通过对拍测试把当前 Transformer head 对齐到 external repo 的 `TransformerForDiffusion`
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/models/heads/transformer1d.py`
|
||||
- Modify: `roboimi/vla/conf/head/transformer1d.yaml`
|
||||
- Test: `tests/test_transformer1d_external_alignment.py`
|
||||
|
||||
- [ ] **Step 1: Confirm the alignment test exists and captures the intended parity contract**
|
||||
|
||||
Write a test that imports external repo's `TransformerForDiffusion` from:
|
||||
`/home/droid/project/diffusion_policy/diffusion_policy/model/diffusion/transformer_for_diffusion.py`
|
||||
and verifies the local `Transformer1D` can load the external model's `state_dict` and produce numerically identical outputs for the same inputs when configured equivalently. Use the full-attention / nocausal configuration (`causal_attn=False`) as the target behavior.
|
||||
|
||||
Required assertions:
|
||||
- same parameter key structure (or compatible `load_state_dict(strict=True)`)
|
||||
- same output shape `(B, T, action_dim)`
|
||||
- `torch.allclose(local_out, external_out, atol=1e-6, rtol=1e-5)`
|
||||
- local model exposes `get_optim_groups(weight_decay=...)` like external repo
|
||||
- use an explicit import path / loader that does not depend on installing the external repo as a package
|
||||
- account for external `ModuleAttrMixin`-style optimizer grouping expectations, including `_dummy_variable` / no-decay bookkeeping if needed for strict state-dict parity
|
||||
- set both models to `eval()` and fix the random seed inside the test so dropout does not create false mismatches
|
||||
|
||||
- [ ] **Step 2: If the test still fails on this branch, use that red state as the TDD starting point**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_transformer1d_external_alignment -v`
|
||||
Expected: either FAIL before implementation on a fresh checkout, or PASS if Task 2 has already been completed on this branch.
|
||||
|
||||
- [ ] **Step 3a: Port API and state-dict compatibility first**
|
||||
|
||||
Match constructor args, parameter names, embeddings, masks, and strict `state_dict` layout with external `TransformerForDiffusion`.
|
||||
|
||||
- [ ] **Step 3b: Port `_init_weights` and optimizer grouping**
|
||||
|
||||
Match external `_init_weights`, `get_optim_groups`, and `configure_optimizers`.
|
||||
|
||||
- [ ] **Step 3c: Port forward semantics**
|
||||
|
||||
Match external `forward(sample, timestep, cond)` behavior under the full-attention / nocausal configuration.
|
||||
|
||||
- [ ] **Step 3d: Port the minimal external implementation**
|
||||
|
||||
Update `roboimi/vla/models/heads/transformer1d.py` so it matches external repo's native DDPM transformer implementation semantics:
|
||||
- same constructor arguments and defaults where relevant
|
||||
- same token accounting (`time_as_cond`, `obs_as_cond`, `T_cond`)
|
||||
- same parameter naming/layout for embeddings, encoder, decoder, masks
|
||||
- same `_init_weights`
|
||||
- same `get_optim_groups` / `configure_optimizers`
|
||||
- same `forward(sample, timestep, cond)` behavior
|
||||
|
||||
Do **not** port PMF / IMF / DiT branches.
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_transformer1d_external_alignment -v`
|
||||
Expected: PASS with strict state-dict load and matching outputs.
|
||||
|
||||
### Task 3: 将当前 Agent 与配置固定到“三相机图像作为条件”的 Transformer diffusion 路线
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/agent.py`
|
||||
- Modify: `roboimi/vla/conf/agent/resnet_transformer.yaml`
|
||||
- Modify: `roboimi/vla/conf/data/simpe_robot_dataset.yaml`
|
||||
- Modify: `roboimi/vla/conf/eval/eval.yaml`
|
||||
- Modify: `roboimi/vla/models/backbones/resnet_diffusion.py`
|
||||
- Test: `tests/test_resnet_transformer_agent_wiring.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Write a wiring test that instantiates the Transformer agent config and checks:
|
||||
- `head_type == "transformer"`
|
||||
- `cfg.data.camera_names == cfg.eval.camera_names == ["r_vis", "top", "front"]`
|
||||
- `num_cams == 3`
|
||||
- Transformer `cond_dim == single_cam_feat_dim * 3 + obs_dim`
|
||||
- `predict_action(...)` accepts image conditions and returns `(B, pred_horizon, action_dim)`
|
||||
- test setup must not download weights; override `pretrained_backbone_weights=null` or stub the backbone
|
||||
- assert camera order used for conditioning is tied to the required three cameras, not a stray constant
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_resnet_transformer_agent_wiring -v`
|
||||
Expected: FAIL if configs/head wiring do not yet guarantee three-camera conditional setup.
|
||||
|
||||
- [ ] **Step 3: Implement minimal alignment**
|
||||
|
||||
Make the transformer path explicit and stable:
|
||||
- keep image observations always as condition (`cond_dim > 0`, `obs_as_cond` semantics)
|
||||
- keep exactly three cameras: `r_vis`, `top`, `front`
|
||||
- keep current ResNet+SpatialSoftmax backbone unless a test proves incompatibility
|
||||
- make the camera feature order deterministic and aligned to the required three-camera list, not generic key sorting
|
||||
- ensure `agent.per_step_cond_dim` and config `head.cond_dim` stay consistent
|
||||
- ensure eval config uses the same three camera names as training
|
||||
- ensure transformer config follows the external full-attention variant (`causal_attn=false`) instead of the external default causal setting
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_resnet_transformer_agent_wiring -v`
|
||||
Expected: PASS
|
||||
|
||||
### Task 4: 让训练脚本在 Transformer 路线上尽量遵循 external repo 的 optimizer/head 使用方式
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Test: `tests/test_train_vla_transformer_optimizer.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Write a test that builds a transformer agent and verifies the training script prefers the head/model supplied optimizer grouping when available (via `get_optim_groups`) instead of blindly using one flat `AdamW(agent.parameters(), ...)`.
|
||||
The test must also prove that every remaining trainable non-head parameter is included exactly once in the optimizer (no silent drops, no duplicates).
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_train_vla_transformer_optimizer -v`
|
||||
Expected: FAIL because current training script uses flat optimizer construction.
|
||||
|
||||
- [ ] **Step 3: Implement minimal optimizer alignment**
|
||||
|
||||
Update `train_vla.py` so that:
|
||||
- for transformer head paths, if `agent.noise_pred_net` exposes `get_optim_groups`, build optimizer groups like external repo
|
||||
- explicitly include the remaining trainable non-head parameters (for example the ResNet backbone's non-frozen projection / pooling layers) in the optimizer instead of accidentally dropping them
|
||||
- keep the rest of the simple training loop intact (no EMA unless required later)
|
||||
- do not over-port external workspace abstractions
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_train_vla_transformer_optimizer -v`
|
||||
Expected: PASS
|
||||
|
||||
### Task 5: 端到端实例化与最小推理验收
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: Run all focused unit tests**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
mamba run -n roboimi python -m unittest \
|
||||
tests.test_eval_vla_execution \
|
||||
tests.test_transformer1d_external_alignment \
|
||||
tests.test_resnet_transformer_agent_wiring \
|
||||
tests.test_train_vla_transformer_optimizer -v
|
||||
```
|
||||
Expected: all PASS
|
||||
|
||||
- [ ] **Step 2: Run syntax checks on changed training/eval/model files**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
mamba run -n roboimi python -m py_compile \
|
||||
roboimi/vla/eval_utils.py \
|
||||
roboimi/vla/models/heads/transformer1d.py \
|
||||
roboimi/vla/agent.py \
|
||||
roboimi/demos/vla_scripts/train_vla.py \
|
||||
roboimi/demos/vla_scripts/eval_vla.py
|
||||
```
|
||||
Expected: no syntax errors
|
||||
|
||||
- [ ] **Step 3: Run a local instantiation smoke test**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
import torch
|
||||
from hydra import compose, initialize_config_dir
|
||||
from hydra.utils import instantiate
|
||||
from pathlib import Path
|
||||
|
||||
config_dir = str((Path.cwd() / "roboimi" / "vla" / "conf").resolve())
|
||||
with initialize_config_dir(version_base=None, config_dir=config_dir):
|
||||
cfg = compose(config_name="config", overrides=[
|
||||
"agent=resnet_transformer",
|
||||
"agent.vision_backbone.pretrained_backbone_weights=null",
|
||||
])
|
||||
|
||||
agent = instantiate(cfg.agent, dataset_stats=None)
|
||||
images = {
|
||||
"r_vis": torch.rand(1, cfg.agent.obs_horizon, 3, 224, 224),
|
||||
"top": torch.rand(1, cfg.agent.obs_horizon, 3, 224, 224),
|
||||
"front": torch.rand(1, cfg.agent.obs_horizon, 3, 224, 224),
|
||||
}
|
||||
qpos = torch.rand(1, cfg.agent.obs_horizon, cfg.agent.obs_dim)
|
||||
out = agent.predict_action(images, qpos)
|
||||
assert out.shape == (1, cfg.agent.pred_horizon, cfg.agent.action_dim), out.shape
|
||||
print("smoke_shape=", out.shape)
|
||||
PY
|
||||
```
|
||||
|
||||
that:
|
||||
- instantiates `agent=resnet_transformer`
|
||||
- constructs fake three-camera input (`r_vis`, `top`, `front`)
|
||||
- calls `agent.predict_action(...)`
|
||||
- verifies output shape is `(1, pred_horizon, 16)`
|
||||
|
||||
Expected: PASS
|
||||
@@ -0,0 +1,202 @@
|
||||
# VLA Experiment Sweep Execution Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Probe the largest safe shared batch size, prepare the 10 unique VLA sweep runs, and launch the serial GPU experiment queue with SwanLab logging and 20-epoch headless rollout validation.
|
||||
|
||||
**Architecture:** Use the existing `train_vla.py` / `eval_vla.py` path without changing model code unless launch-blocking issues appear. A small external launcher contract will manage per-run directories, logs, pids, and serial execution while Hydra places trainer-local `checkpoints/` under each run directory.
|
||||
|
||||
**Tech Stack:** zsh, mamba env `roboimi`, Hydra overrides, PyTorch CUDA, SwanLab, existing `train_vla.py`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Reconfirm schedule math and current runtime assumptions
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/superpowers/specs/2026-03-30-vla-experiment-sweep-design.md` (only if execution reveals a spec mismatch)
|
||||
- Test: none
|
||||
|
||||
- [ ] **Step 1: Print dataset size and epoch math for the baseline batch size**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python - <<'PY'
|
||||
num_samples = 70000
|
||||
for batch_size in (32, 48, 64, 80):
|
||||
steps_per_epoch = num_samples // batch_size
|
||||
print(batch_size, steps_per_epoch, 20 * steps_per_epoch, 200 * steps_per_epoch)
|
||||
PY
|
||||
```
|
||||
|
||||
Expected: integer epoch-step mappings for candidate batch sizes.
|
||||
|
||||
- [ ] **Step 2: Reconfirm no stale training process is already active**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
ps -ef | grep -E 'train_vla.py|python .*roboimi/demos/vla_scripts/train_vla.py' | grep -v grep || true
|
||||
```
|
||||
|
||||
Expected: no active training process unless intentionally started by this session.
|
||||
|
||||
- [ ] **Step 3: Record the execution assumptions**
|
||||
|
||||
Capture in session notes:
|
||||
- dataset path
|
||||
- camera list
|
||||
- current branch
|
||||
- no active conflicting training process
|
||||
|
||||
|
||||
### Task 2: Probe the largest safe shared batch size
|
||||
|
||||
**Files:**
|
||||
- Create: `runs/batch-probe-<timestamp>/` (runtime artifact)
|
||||
- Test: probe command output only
|
||||
|
||||
- [ ] **Step 1: Write the candidate probe command for the largest model**
|
||||
|
||||
Use overrides:
|
||||
- `data.dataset_dir=/home/droid/project/diana_sim/sim_transfer`
|
||||
- `data.camera_names=[r_vis,top,front]`
|
||||
- `agent.head.n_emb=384`
|
||||
- `agent.head.n_layer=12`
|
||||
- `agent.head.n_head=4`
|
||||
- `agent.vision_backbone.pretrained_backbone_weights=IMAGENET1K_V1`
|
||||
- `agent.vision_backbone.freeze_backbone=true`
|
||||
- `agent.vision_backbone.use_separate_rgb_encoder_per_camera=true`
|
||||
- `train.device=cuda`
|
||||
- `train.val_split=0.0`
|
||||
- `train.seed=42`
|
||||
- `train.use_swanlab=false`
|
||||
- `train.rollout_val_freq_epochs=0`
|
||||
- `train.rollout_validate_on_checkpoint=false`
|
||||
- `train.max_steps=4`
|
||||
|
||||
- [ ] **Step 2: Run the probe for `batch_size=32`**
|
||||
|
||||
Expected pass condition:
|
||||
- no CUDA OOM
|
||||
- no crash/abort
|
||||
- no NaN/Inf loss
|
||||
- 4 steps complete
|
||||
|
||||
- [ ] **Step 3: Repeat for `batch_size=48`, `64`, then optionally `80`**
|
||||
|
||||
Stop increasing once one candidate fails.
|
||||
|
||||
- [ ] **Step 4: Compute the shared LR and 200-epoch step budget**
|
||||
|
||||
Using:
|
||||
```text
|
||||
lr = 1e-4 * (batch_size / 16)
|
||||
max_steps = 200 * floor(70000 / batch_size)
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Write down the chosen shared batch size and derived max_steps**
|
||||
|
||||
This becomes the execution contract for all runs.
|
||||
|
||||
|
||||
### Task 3: Materialize the 10 unique run matrix
|
||||
|
||||
**Files:**
|
||||
- Create: `runs/vla-sweep-<timestamp>/manifest.txt` (runtime artifact)
|
||||
- Create: `runs/vla-sweep-<timestamp>/launch_queue.sh` (runtime artifact)
|
||||
- Test: shell syntax / manifest inspection
|
||||
|
||||
- [ ] **Step 1: Enumerate the 2-point pretraining comparison**
|
||||
|
||||
Canonical runs:
|
||||
1. `sim-transfer-baseline-pretrain-on-emb128-layer4`
|
||||
2. `sim-transfer-pretrain-off`
|
||||
|
||||
- [ ] **Step 2: Enumerate the remaining 8 unique architecture-sweep runs**
|
||||
|
||||
Skip the duplicate `(pretrain-on, emb128, layer4)` because it reuses the baseline run.
|
||||
|
||||
- [ ] **Step 3: For each run, define exact Hydra overrides**
|
||||
|
||||
Every run must pin:
|
||||
- dataset path
|
||||
- three cameras
|
||||
- `train.device=cuda`
|
||||
- `train.val_split=0.0`
|
||||
- `train.num_workers=12`
|
||||
- `train.seed=42`
|
||||
- `train.rollout_val_freq_epochs=20`
|
||||
- `train.rollout_validate_on_checkpoint=false`
|
||||
- `train.rollout_num_episodes=3`
|
||||
- `train.use_swanlab=true`
|
||||
- a unique `train.swanlab_run_name`
|
||||
- shared batch size
|
||||
- shared LR
|
||||
- derived `max_steps`
|
||||
- `agent.head.n_head=4`
|
||||
- `agent.vision_backbone.freeze_backbone=true`
|
||||
- `agent.vision_backbone.use_separate_rgb_encoder_per_camera=true`
|
||||
|
||||
- [ ] **Step 4: Write the queue launcher script**
|
||||
|
||||
Launcher responsibilities:
|
||||
- create run directory
|
||||
- export `LD_LIBRARY_PATH` CUDA/cuDNN fixup
|
||||
- export `MPLCONFIGDIR=/tmp/mpl`
|
||||
- set `hydra.job.chdir=true`
|
||||
- set `hydra.run.dir=<run_dir>`
|
||||
- tee logs to `<run_dir>/train.log`
|
||||
- write `<run_dir>/train.pid`
|
||||
- run jobs serially
|
||||
|
||||
- [ ] **Step 5: Verify the queue script syntax**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
zsh -n runs/vla-sweep-<timestamp>/launch_queue.sh
|
||||
```
|
||||
|
||||
Expected: exit 0.
|
||||
|
||||
|
||||
### Task 4: Launch the first probe-selected production run and verify startup
|
||||
|
||||
**Files:**
|
||||
- Create: `runs/vla-sweep-<timestamp>/<run-name>/` (runtime artifact)
|
||||
- Test: live log inspection / process inspection
|
||||
|
||||
- [ ] **Step 1: Start the first run in the queue**
|
||||
|
||||
Use the generated launcher contract.
|
||||
|
||||
- [ ] **Step 2: Inspect the first 100-200 log lines**
|
||||
|
||||
Expected:
|
||||
- config resolves correctly
|
||||
- dataset loads
|
||||
- agent initializes on CUDA
|
||||
- training loop starts
|
||||
- SwanLab initializes
|
||||
|
||||
- [ ] **Step 3: Confirm process identity and run directory**
|
||||
|
||||
Collect:
|
||||
- PID
|
||||
- run dir
|
||||
- log path
|
||||
- SwanLab run name
|
||||
|
||||
- [ ] **Step 4: If startup is clean, continue the remaining serial queue**
|
||||
|
||||
If startup fails, stop and debug before launching more runs.
|
||||
|
||||
|
||||
### Task 5: Report the launched sweep contract back to the user
|
||||
|
||||
**Files:**
|
||||
- Test: log evidence only
|
||||
|
||||
- [ ] **Step 1: Summarize the chosen batch size, LR, max_steps, and rollout cadence**
|
||||
|
||||
- [ ] **Step 2: Provide the active run directory / pid / first-run status**
|
||||
|
||||
- [ ] **Step 3: State the remaining queued runs by name**
|
||||
@@ -0,0 +1,454 @@
|
||||
# VLA Training + Headless Rollout + SwanLab Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 补齐当前 `Transformer1D` 训练依赖，在 `/home/droid/project/diana_sim/sim_transfer` 上启动训练，接入 SwanLab 标量日志，并提供训练期可选的 headless rollout validation 路径。
|
||||
|
||||
**Architecture:** 保持现有 `train_vla.py` / `eval_vla.py` 主体不变，只做最小必要改造：补依赖、补 stats 生成入口、在训练脚本里加轻量 SwanLab logger 和可选 checkpoint-time rollout wrapper、在环境侧把图像更新与 GUI 显示解耦。训练默认仍走当前 `resnet_transformer + Transformer1D` 路线，rollout validation 作为薄封装默认关闭。
|
||||
|
||||
**Tech Stack:** Python, mamba/conda, pip, PyTorch, Hydra, diffusers, torchvision, einops, SwanLab, MuJoCo, OpenCV, unittest
|
||||
|
||||
---
|
||||
|
||||
### Task 0: 执行前提与分支/环境确认
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: 确认当前分支仍是目标分支**
|
||||
|
||||
Run: `git branch --show-current`
|
||||
Expected: `feat-align-dp-transformer-ee`
|
||||
|
||||
- [ ] **Step 2: 记录当前 Python 解释器与环境名**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
import sys
|
||||
print(sys.executable)
|
||||
PY`
|
||||
Expected: 输出 `/home/droid/.conda/envs/roboimi/bin/python`
|
||||
|
||||
- [ ] **Step 3: 记录当前数据集目录存在性与 episode 数量**
|
||||
|
||||
Run: `/usr/bin/zsh -lc 'echo DATASET=/home/droid/project/diana_sim/sim_transfer; find /home/droid/project/diana_sim/sim_transfer -maxdepth 1 -name "episode_*.hdf5" | wc -l'`
|
||||
Expected: 输出目录路径与 `100`
|
||||
|
||||
### Task 1: 补齐训练依赖并把 resolved versions 写回环境定义
|
||||
|
||||
**Files:**
|
||||
- Modify: `environment.yml`
|
||||
- Verify only: local `roboimi` env
|
||||
|
||||
- [ ] **Step 1: 写出缺失依赖的最小清单**
|
||||
|
||||
需要补齐：
|
||||
- `diffusers`
|
||||
- `torchvision`
|
||||
- `einops`
|
||||
- `swanlab`
|
||||
|
||||
- [ ] **Step 2: 先用 dry-run 解析候选版本，确认不会升级 Torch**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python -m pip install --dry-run \
|
||||
diffusers torchvision einops swanlab
|
||||
```
|
||||
Expected: 输出候选版本；若显示会升级/替换 `torch`，则停止并改用显式兼容版本
|
||||
|
||||
- [ ] **Step 3: 安装与当前 Torch 兼容的缺失依赖到现有环境**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python -m pip install \
|
||||
diffusers torchvision einops swanlab
|
||||
```
|
||||
Expected: 安装成功，且不替换当前 `torch==2.4.0`
|
||||
|
||||
- [ ] **Step 4: 运行 import 验证**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
mods=['torch','hydra','omegaconf','diffusers','torchvision','einops','cv2','h5py','swanlab','mujoco']
|
||||
for m in mods:
|
||||
__import__(m)
|
||||
print('OK', m)
|
||||
PY
|
||||
```
|
||||
Expected: 每个模块都输出 `OK <module>`
|
||||
|
||||
- [ ] **Step 5: 记录实际安装版本**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
import diffusers, torchvision, einops, swanlab
|
||||
print('diffusers', getattr(diffusers,'__version__',''))
|
||||
print('torchvision', getattr(torchvision,'__version__',''))
|
||||
print('einops', getattr(einops,'__version__',''))
|
||||
print('swanlab', getattr(swanlab,'__version__',''))
|
||||
PY
|
||||
```
|
||||
Expected: 输出四个包的 resolved versions
|
||||
|
||||
- [ ] **Step 6: 将 resolved versions 写回 `environment.yml`**
|
||||
|
||||
把新增依赖补到 `environment.yml` 的现有依赖列表（若使用 `pip:` 段则更新该段）里，使用 Step 5 得到的**实际 resolved versions**，避免环境漂移，并避免重复 package 条目。
|
||||
|
||||
- [ ] **Step 7: 语法检查环境定义文件仅作结构确认**
|
||||
|
||||
Run: `python - <<'PY'
|
||||
from pathlib import Path
|
||||
text = Path('environment.yml').read_text()
|
||||
assert 'diffusers' in text
|
||||
assert 'torchvision' in text
|
||||
assert 'einops' in text
|
||||
assert 'swanlab' in text
|
||||
print('environment.yml updated')
|
||||
PY`
|
||||
Expected: `environment.yml updated`
|
||||
|
||||
### Task 2: 让统计脚本支持外部数据目录并生成 dataset stats
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/scripts/calculate_stats.py`
|
||||
- Test: `tests/test_calculate_stats_cli.py`
|
||||
|
||||
- [ ] **Step 1: 写 failing test，验证统计脚本可接受外部 `--dataset_dir` 并输出目标路径**
|
||||
|
||||
Test file should:
|
||||
- 用临时目录创建最小 HDF5 episode
|
||||
- 调用脚本入口/函数时传入外部目录
|
||||
- 断言输出 `dataset_stats.pkl` 出现在该目录
|
||||
- 断言 pickle 内包含 `action_mean/qpos_mean/...`
|
||||
|
||||
- [ ] **Step 2: 跑测试确认它先失败**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_calculate_stats_cli -v`
|
||||
Expected: FAIL（当前脚本写死默认目录）
|
||||
|
||||
- [ ] **Step 3: 最小实现 `--dataset_dir` 支持**
|
||||
|
||||
要求：
|
||||
- 保留现有统计逻辑
|
||||
- 仅增加 CLI 参数解析
|
||||
- 输出仍写入 `<dataset_dir>/dataset_stats.pkl`
|
||||
|
||||
- [ ] **Step 4: 重新跑测试确认转绿**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_calculate_stats_cli -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: 用真实数据集生成 stats**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/vla/scripts/calculate_stats.py \
|
||||
--dataset_dir /home/droid/project/diana_sim/sim_transfer
|
||||
```
|
||||
Expected: 生成 `/home/droid/project/diana_sim/sim_transfer/dataset_stats.pkl`
|
||||
|
||||
- [ ] **Step 6: 验证 stats 文件结构**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
import pickle
|
||||
path='/home/droid/project/diana_sim/sim_transfer/dataset_stats.pkl'
|
||||
with open(path,'rb') as f:
|
||||
stats=pickle.load(f)
|
||||
for k in ['action_mean','action_std','action_min','action_max','qpos_mean','qpos_std','qpos_min','qpos_max']:
|
||||
assert k in stats, k
|
||||
print('stats_ok')
|
||||
PY
|
||||
```
|
||||
Expected: `stats_ok`
|
||||
|
||||
### Task 3: 增加 SwanLab 训练日志集成
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Modify: `roboimi/vla/conf/config.yaml`
|
||||
- Test: `tests/test_train_vla_swanlab_logging.py`
|
||||
|
||||
- [ ] **Step 1: 写 failing test，验证训练脚本在 `train.use_swanlab=true` 时会初始化 SwanLab 并记录标量**
|
||||
|
||||
Test should:
|
||||
- stub `swanlab`
|
||||
- 调用训练脚本中抽取出的非 Hydra helper（如 `_run_training(cfg)`）的最小路径（`max_steps=0` 或很小）
|
||||
- 断言调用了：
|
||||
- `swanlab.init(project='roboimi-vla', ...)`
|
||||
- 至少一次 `swanlab.log({...})`
|
||||
- 断言当 `use_swanlab=false` 时不会 import/初始化 SwanLab
|
||||
- 断言当 `use_swanlab=true` 且 import `swanlab` 失败时会 fail fast
|
||||
- 断言当 `use_swanlab=true` 且认证/登录状态不可用时会 fail fast
|
||||
- 断言会记录最终/最佳 checkpoint 路径到 log/summary
|
||||
|
||||
- [ ] **Step 2: 跑测试确认它先失败**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_train_vla_swanlab_logging -v`
|
||||
Expected: FAIL（当前无 SwanLab 集成）
|
||||
|
||||
- [ ] **Step 3: 在配置中增加最小 SwanLab 契约**
|
||||
|
||||
在 `roboimi/vla/conf/config.yaml` 添加：
|
||||
- `train.use_swanlab: true`
|
||||
- `train.swanlab_project: roboimi-vla`
|
||||
- 可选 `train.swanlab_run_name: null`
|
||||
|
||||
- [ ] **Step 4: 从 Hydra 入口提取可测试的训练 helper**
|
||||
|
||||
要求：
|
||||
- 新增类似 `_run_training(cfg)` 的普通函数
|
||||
- `main()` 只做 Hydra 入口转发
|
||||
- 测试只调用 helper，不直接调用 Hydra-decorated `main(cfg)`
|
||||
|
||||
- [ ] **Step 5: 在训练脚本中实现 SwanLab 初始化的 fail-fast 逻辑**
|
||||
|
||||
要求：
|
||||
- `use_swanlab=true` 时：
|
||||
- import 失败 -> 直接报错
|
||||
- 本地未登录/认证失败 -> 直接报错
|
||||
- 成功后执行 `swanlab.init(project=cfg.train.swanlab_project, ...)`
|
||||
|
||||
- [ ] **Step 6: 在训练脚本中实现轻量 scalar logger**
|
||||
|
||||
要求：
|
||||
- 仅 scalar logging，不引入自定义 callback 框架
|
||||
- 训练时记录 `train/loss`, `train/lr`, `train/best_loss`, `train/step`
|
||||
- 验证时记录 `val/loss`
|
||||
- 训练结束时记录：
|
||||
- `train/final_checkpoint_path`
|
||||
- `train/best_checkpoint_path`
|
||||
- 若库支持则显式 `finish/close`
|
||||
|
||||
- [ ] **Step 7: 重新跑测试确认转绿**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_train_vla_swanlab_logging -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 8: 用提供的 API key 完成本地登录**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/usr/bin/zsh -lc 'SWANLAB_API_KEY="<user-provided>"; /home/droid/.conda/envs/roboimi/bin/swanlab login -k "$SWANLAB_API_KEY"'
|
||||
```
|
||||
Expected: 登录成功并保存本地凭证
|
||||
|
||||
### Task 4: 为评估/rollout 路径增加 headless 模式
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/envs/double_base.py`
|
||||
- Modify: `roboimi/envs/double_pos_ctrl_env.py`
|
||||
- Modify: `roboimi/vla/conf/eval/eval.yaml`
|
||||
- Modify: `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
- Test: `tests/test_eval_vla_headless.py`
|
||||
|
||||
- [ ] **Step 1: 写 failing test，验证 headless 路径不触发 GUI 调用**
|
||||
|
||||
Test should stub:
|
||||
- `cv2.namedWindow`
|
||||
- `cv2.imshow`
|
||||
- `cv2.waitKey`
|
||||
- viewer launch/render path
|
||||
|
||||
And assert:
|
||||
- `eval.headless=true` 时不调用这些 GUI 接口
|
||||
- 仍能获取图像观测并走到 policy action 执行前/后关键路径
|
||||
|
||||
- [ ] **Step 2: 跑测试确认它先失败**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_eval_vla_headless -v`
|
||||
Expected: FAIL（当前 env/eval 默认会开 viewer 和 cv2 窗口）
|
||||
|
||||
- [ ] **Step 3: 统一配置开关为 `eval.headless`**
|
||||
|
||||
在 `roboimi/vla/conf/eval/eval.yaml` 添加：
|
||||
- `headless: false`
|
||||
|
||||
不要再引入第二个同义开关。
|
||||
|
||||
- [ ] **Step 4: 在 env 工厂中接入 `headless`**
|
||||
|
||||
在 `make_sim_env(...)` 中：
|
||||
- `eval.headless=true` -> `is_render=False`
|
||||
- 保留图像观测更新
|
||||
- 不创建 MuJoCo viewer
|
||||
|
||||
- [ ] **Step 5: 将相机更新与 GUI 显示解耦**
|
||||
|
||||
在 `double_base.py`：
|
||||
- 图像更新逻辑保留
|
||||
- `cv2.namedWindow/imshow/waitKey` 仅在非 headless 下执行
|
||||
|
||||
- [ ] **Step 6: eval 脚本中在 headless 下跳过 `env.render()`**
|
||||
|
||||
只在 `eval.headless=false` 时调用 `env.render()`。
|
||||
|
||||
- [ ] **Step 7: 重新跑测试确认转绿**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_eval_vla_headless -v`
|
||||
Expected: PASS
|
||||
|
||||
### Task 5: 给训练脚本加可选 checkpoint-time rollout validation 薄封装
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Possibly modify: `roboimi/demos/vla_scripts/eval_vla.py`（仅当需要提取可复用入口）
|
||||
- Modify: `roboimi/vla/conf/config.yaml`
|
||||
- Test: `tests/test_train_vla_rollout_validation.py`
|
||||
|
||||
- [ ] **Step 1: 写 failing test，验证 checkpoint 保存点可选调用 rollout validation，且会传 `eval.headless=true`**
|
||||
|
||||
Test should:
|
||||
- stub rollout/eval helper
|
||||
- 开启 `train.rollout_validate_on_checkpoint=true`
|
||||
- 设置小 `save_freq`
|
||||
- 断言训练脚本在 checkpoint 时调用验证 helper
|
||||
- 断言调用参数带 `headless=true`
|
||||
|
||||
- [ ] **Step 2: 跑测试确认它先失败**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_train_vla_rollout_validation -v`
|
||||
Expected: FAIL（当前无该 hook）
|
||||
|
||||
- [ ] **Step 3: 增加最小配置键**
|
||||
|
||||
在 `config.yaml` 中添加：
|
||||
- `train.rollout_validate_on_checkpoint: false`
|
||||
- `train.rollout_num_episodes: 1`
|
||||
|
||||
- [ ] **Step 4: 提取一个最小 rollout validation helper 接口**
|
||||
|
||||
要求：
|
||||
- helper 输入至少包括 `cfg`, `ckpt_path`, `num_episodes`, `headless`
|
||||
- 默认训练侧调用时强制 `headless=True`
|
||||
- 优先复用现有 eval 逻辑，不引入第二套 validator 类
|
||||
|
||||
- [ ] **Step 5: 在 checkpoint 保存路径中接入 rollout helper**
|
||||
|
||||
要求：
|
||||
- 不重写第二套验证框架
|
||||
- 优先复用现有 eval 逻辑/工具
|
||||
- 默认关闭
|
||||
- 仅在 checkpoint 时少量调用
|
||||
- 强制 `eval.headless=true`
|
||||
|
||||
- [ ] **Step 6: 重新跑测试确认转绿**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_train_vla_rollout_validation -v`
|
||||
Expected: PASS
|
||||
|
||||
### Task 6: 启动训练前的集成 smoke verification
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: 跑所有新增/相关测试**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python -m unittest \
|
||||
tests.test_calculate_stats_cli \
|
||||
tests.test_train_vla_swanlab_logging \
|
||||
tests.test_eval_vla_headless \
|
||||
tests.test_train_vla_rollout_validation -v
|
||||
```
|
||||
Expected: 全部 PASS
|
||||
|
||||
- [ ] **Step 2: 对关键修改文件做语法检查**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python -m py_compile \
|
||||
roboimi/vla/scripts/calculate_stats.py \
|
||||
roboimi/demos/vla_scripts/train_vla.py \
|
||||
roboimi/demos/vla_scripts/eval_vla.py \
|
||||
roboimi/envs/double_pos_ctrl_env.py \
|
||||
roboimi/envs/double_base.py
|
||||
```
|
||||
Expected: 无语法错误
|
||||
|
||||
- [ ] **Step 3: 运行训练 smoke run**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
SWANLAB_API_KEY='<user-provided>' \
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.max_steps=20 \
|
||||
train.log_freq=1 \
|
||||
train.save_freq=10 \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.rollout_validate_on_checkpoint=false
|
||||
```
|
||||
Expected:
|
||||
- 训练启动成功
|
||||
- 产生 `checkpoints/vla_model_step_10.pt` 或 `vla_model_final.pt`
|
||||
- 本地日志中无 ImportError
|
||||
|
||||
- [ ] **Step 4: 运行一个最小 headless rollout-validation smoke run**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
SWANLAB_API_KEY='<user-provided>' \
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.max_steps=10 \
|
||||
train.log_freq=1 \
|
||||
train.save_freq=5 \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.rollout_validate_on_checkpoint=true \
|
||||
train.rollout_num_episodes=1 \
|
||||
eval.headless=true
|
||||
```
|
||||
Expected:
|
||||
- 到达 checkpoint-time rollout 调用
|
||||
- 不弹 MuJoCo viewer
|
||||
- 不执行 `cv2.namedWindow/imshow/waitKey`
|
||||
|
||||
- [ ] **Step 5: 验证 checkpoint 文件已写出**
|
||||
|
||||
Run: `/usr/bin/zsh -lc 'ls -lah checkpoints | sed -n "1,120p"'`
|
||||
Expected: 存在 `vla_model_step_10.pt` 或 `vla_model_final.pt`
|
||||
|
||||
- [ ] **Step 6: 验证 SwanLab 已收到标量**
|
||||
|
||||
验证方式：
|
||||
- 终端日志中确认 `swanlab.init` / run URL / run id
|
||||
- 若工具支持，确认 dashboard 中 `roboimi-vla` 项目下出现本次 run
|
||||
|
||||
### Task 7: 启动正式训练
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: 用真实配置启动正式训练**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
SWANLAB_API_KEY='<user-provided>' \
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.rollout_validate_on_checkpoint=true \
|
||||
eval.headless=true
|
||||
```
|
||||
Expected:
|
||||
- 训练持续运行
|
||||
- checkpoint 周期性写出
|
||||
- SwanLab 周期性收到 train/val 标量
|
||||
- 若 checkpoint-time rollout 打开，则不弹 GUI
|
||||
|
||||
- [ ] **Step 2: 记录启动信息并向用户汇报**
|
||||
|
||||
汇报内容至少包括：
|
||||
- 使用的数据集路径
|
||||
- 训练命令/关键 overrides
|
||||
- checkpoint 输出目录
|
||||
- SwanLab project 名称
|
||||
- rollout validation 是否已启用以及是否 headless
|
||||
@@ -0,0 +1,43 @@
|
||||
# Held-out Episode Validation Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add optional held-out episode validation to main training flow, including explicit val episode selection and periodic action MSE evaluation without merging LEWM-only model features.
|
||||
|
||||
**Architecture:** Extend the generic dataset with optional episode filtering metadata, extend train config with explicit held-out validation knobs, and wire train_vla to choose between random split and explicit episode split. Reuse agent.predict_action_chunk for action MSE so the metric stays model-agnostic.
|
||||
|
||||
**Tech Stack:** Python, Hydra/OmegaConf, PyTorch, unittest
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add failing tests for dataset episode filtering and config semantics
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/test_simple_robot_dataset_image_loading.py`
|
||||
- Modify: `tests/test_train_vla_rollout_validation.py`
|
||||
|
||||
- [ ] Add dataset tests for `episode_indices` and `available_episode_indices`.
|
||||
- [ ] Add training tests for explicit held-out episode splitting and fail-fast config validation.
|
||||
- [ ] Run focused tests and verify they fail for the expected missing behavior.
|
||||
|
||||
### Task 2: Implement minimal dataset and training support
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/data/simpe_robot_dataset.py`
|
||||
- Modify: `roboimi/vla/conf/config.yaml`
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
|
||||
- [ ] Add config keys `train.val_episode_indices` and `train.action_mse_val_freq_epochs`.
|
||||
- [ ] Add optional dataset filtering by episode index plus `available_episode_indices` metadata.
|
||||
- [ ] Add explicit train/val dataset builder and held-out action MSE computation.
|
||||
- [ ] Log `val/action_mse` only when explicit held-out episode validation is configured.
|
||||
|
||||
### Task 3: Verify focused coverage
|
||||
|
||||
**Files:**
|
||||
- Test: `tests/test_simple_robot_dataset_image_loading.py`
|
||||
- Test: `tests/test_train_vla_rollout_validation.py`
|
||||
- Test: `tests/test_train_vla_swanlab_logging.py`
|
||||
|
||||
- [ ] Run focused unittest targets for dataset filtering, held-out MSE, and SwanLab logging.
|
||||
- [ ] Fix any regressions with minimal code changes.
|
||||
@@ -0,0 +1,242 @@
|
||||
# VLA Experiment Sweep Design
|
||||
|
||||
## Goal
|
||||
|
||||
Run a matched-configuration training sweep on `/home/droid/project/diana_sim/sim_transfer` for the aligned Transformer diffusion policy, comparing:
|
||||
|
||||
1. ResNet ImageNet pretraining vs no pretraining under matched settings.
|
||||
2. Transformer width/depth choices across a 3×3 grid.
|
||||
|
||||
The sweep should use the full dataset, headless rollout validation every 20 epochs, SwanLab logging, and a total budget of about 200 epochs per run.
|
||||
|
||||
## Current Context
|
||||
|
||||
- Dataset: `100` episodes, `70000` frame-level samples.
|
||||
- Cameras used as image conditions: `r_vis`, `top`, `front`.
|
||||
- Current policy family: ResNet visual encoder + Transformer DDPM head.
|
||||
- Current training path already supports:
|
||||
- full-dataset training (`val_split=0.0`)
|
||||
- checkpoint-time and epoch-time rollout hooks
|
||||
- headless eval path
|
||||
- SwanLab logging
|
||||
- best-checkpoint selection by rollout reward once rollout metrics exist
|
||||
- Current trainer behavior that matters for execution:
|
||||
- step checkpoints are written to `checkpoints/` relative to the Hydra job working directory
|
||||
- epoch-time rollout validation is controlled by `train.rollout_val_freq_epochs`
|
||||
- checkpoint-time rollout validation is separately controlled by `train.rollout_validate_on_checkpoint`
|
||||
- per-run log files, pid files, and run directories must be created by the external launcher, not by the trainer itself
|
||||
|
||||
## Derived Schedule
|
||||
|
||||
For a chosen global batch size `B`, define:
|
||||
|
||||
- `steps_per_epoch = floor(70000 / B)` because training currently uses `drop_last=True` when the train set is larger than the batch.
|
||||
- `rollout_freq_steps = 20 * steps_per_epoch`
|
||||
- `max_steps = 200 * steps_per_epoch`
|
||||
|
||||
Example at `B = 32`:
|
||||
|
||||
- `steps_per_epoch = 2187`
|
||||
- `rollout_freq_steps = 43740`
|
||||
- `max_steps = 437400`
|
||||
|
||||
This schedule must be recomputed after the batch-size probe chooses the final shared batch size.
|
||||
|
||||
## Experiment Strategy
|
||||
|
||||
### Recommended approach
|
||||
|
||||
Use one **shared batch size across all runs**, chosen by probing the largest model that will appear in the sweep. This preserves fairness while still using as much GPU memory as safely available.
|
||||
|
||||
### Why not per-run max batch size
|
||||
|
||||
Allowing each model to use a different batch size would add another changing variable, making the 11-run comparison harder to interpret.
|
||||
|
||||
### Unique executions vs logical comparisons
|
||||
|
||||
There are `11` logical comparison points in the user request (`2` for the pretraining comparison and `9` for the emb/layer grid), but only `10` **unique** training configurations because:
|
||||
|
||||
- the pretrained baseline in Section A (`pretrained_backbone_weights=IMAGENET1K_V1`, `n_emb=128`, `n_layer=4`) is also one cell of the Section B sweep.
|
||||
|
||||
This design will therefore run **10 unique training jobs**, and the pretrained-on baseline will be reused as the `(128, 4)` architecture-sweep point. This avoids wasting one full 200-epoch run on an identical configuration.
|
||||
|
||||
The reused baseline has one canonical identity throughout execution:
|
||||
|
||||
- run name stem: `sim-transfer-baseline-pretrain-on-emb128-layer4`
|
||||
- this same run is counted both as:
|
||||
- the pretrained-on arm of the 2-point pretraining comparison
|
||||
- the `(n_emb=128, n_layer=4)` point of the 9-point architecture sweep
|
||||
|
||||
## Batch-Size Probe
|
||||
|
||||
Before launching the full sweep:
|
||||
|
||||
1. Probe the largest planned model:
|
||||
- `n_emb=384`
|
||||
- `n_layer=12`
|
||||
- ResNet pretrained enabled
|
||||
2. Try candidate batch sizes in ascending order:
|
||||
- `32`
|
||||
- `48`
|
||||
- `64`
|
||||
- optionally `80` if `64` is clearly safe
|
||||
3. For each candidate, run a short GPU smoke training with:
|
||||
- real dataset
|
||||
- three cameras
|
||||
- no SwanLab
|
||||
- no rollout validation
|
||||
- `max_steps=4`
|
||||
4. A candidate **passes** if all 4 steps finish without:
|
||||
- CUDA OOM
|
||||
- process crash / abort
|
||||
- NaN / Inf loss
|
||||
5. Choose the **highest passing** candidate and use it for all sweep runs.
|
||||
|
||||
Learning rate should scale linearly using the current training rule:
|
||||
|
||||
- config baseline: `lr=1e-4` at `batch_size=16`
|
||||
- equivalent rule: `lr = 1e-4 * (batch_size / 16)`
|
||||
- equivalently: `lr = 2e-4 * (batch_size / 32)`
|
||||
|
||||
## Experiment Matrix
|
||||
|
||||
Total logical comparison points: `11`
|
||||
|
||||
Total unique training executions: `10`
|
||||
|
||||
### A. ResNet pretraining comparison (2 runs)
|
||||
|
||||
Fixed Transformer settings:
|
||||
|
||||
- `n_emb=128`
|
||||
- `n_layer=4`
|
||||
- `n_head=4`
|
||||
- `freeze_backbone=true`
|
||||
- `use_separate_rgb_encoder_per_camera=true`
|
||||
|
||||
Interpretation note:
|
||||
|
||||
- This comparison keeps the current frozen-backbone recipe unchanged.
|
||||
- Therefore, the `pretrained_backbone_weights=null` run measures **random frozen visual features vs ImageNet frozen visual features**.
|
||||
- It does **not** test end-to-end training of the ResNet from scratch; that would be a separate experiment.
|
||||
|
||||
Runs:
|
||||
|
||||
1. `pretrained_backbone_weights=IMAGENET1K_V1`
|
||||
2. `pretrained_backbone_weights=null`
|
||||
|
||||
### B. Transformer architecture sweep (9 runs)
|
||||
|
||||
Fixed visual backbone setting:
|
||||
|
||||
- `pretrained_backbone_weights=IMAGENET1K_V1`
|
||||
- `freeze_backbone=true`
|
||||
- `use_separate_rgb_encoder_per_camera=true`
|
||||
- `n_head=4`
|
||||
|
||||
Rationale:
|
||||
|
||||
- The 9-point emb/layer sweep fixes `pretrained_backbone_weights=IMAGENET1K_V1`.
|
||||
- The separate 2-run comparison in Section A isolates the effect of ResNet pretraining at the baseline architecture.
|
||||
|
||||
Grid:
|
||||
|
||||
- `n_emb ∈ {128, 256, 384}`
|
||||
- `n_layer ∈ {4, 8, 12}`
|
||||
|
||||
## Shared Training Configuration
|
||||
|
||||
All full runs should use:
|
||||
|
||||
- dataset: `/home/droid/project/diana_sim/sim_transfer`
|
||||
- cameras: `[r_vis, top, front]`
|
||||
- device: `cuda`
|
||||
- `val_split=0.0`
|
||||
- `num_workers=12`
|
||||
- `seed=42`
|
||||
- headless rollout validation
|
||||
- `rollout_val_freq_epochs=20`
|
||||
- `rollout_validate_on_checkpoint=false`
|
||||
- `rollout_num_episodes=3`
|
||||
- `use_swanlab=true`
|
||||
- shared batch size from probe
|
||||
- shared learning-rate rule from probe
|
||||
- `max_steps` recomputed to represent about `200` epochs
|
||||
|
||||
Reproducibility note:
|
||||
|
||||
- `train.seed=42` will be fixed and recorded for all runs.
|
||||
- Under the current `train_vla.py`, this does **not** make the runs bitwise deterministic when `val_split=0.0`, because model initialization and shuffled training order are not fully seeded by the trainer today.
|
||||
- The sweep is therefore treated as a **matched-config comparison**, not a strict deterministic benchmark.
|
||||
|
||||
## Rollout / Validation Policy
|
||||
|
||||
- No held-out val split will be used.
|
||||
- Loss remains a fallback best-model metric until the first rollout reward exists.
|
||||
- After rollout metrics begin, the best checkpoint should be chosen by **higher average rollout reward**.
|
||||
- Rollout should run with:
|
||||
- no GUI
|
||||
- CPU eval device
|
||||
- current evaluation helper path
|
||||
|
||||
## Execution Model
|
||||
|
||||
Runs should be launched **serially**, not concurrently, because:
|
||||
|
||||
- the user currently has one GPU available
|
||||
- concurrent jobs would reduce throughput and complicate memory management
|
||||
- serial execution preserves clean logs and easier resume semantics
|
||||
|
||||
Each run gets:
|
||||
|
||||
- a dedicated run directory under `runs/`
|
||||
- its own log file
|
||||
- its own pid / launcher metadata if backgrounded
|
||||
- a unique SwanLab run name
|
||||
- `hydra.job.chdir=true`
|
||||
- `hydra.run.dir=<run_dir>` so the trainer-local `checkpoints/` land inside the run directory
|
||||
|
||||
## Naming Convention
|
||||
|
||||
Suggested run name pattern:
|
||||
|
||||
- pretraining comparison:
|
||||
- `sim-transfer-baseline-pretrain-on-emb128-layer4-bs{B}-lr{LR}-e200`
|
||||
- `sim-transfer-pretrain-off-bs{B}-lr{LR}-e200`
|
||||
- architecture sweep:
|
||||
- `(128,4)` reuses `sim-transfer-baseline-pretrain-on-emb128-layer4-bs{B}-lr{LR}-e200`
|
||||
- `sim-transfer-emb{E}-layer{L}-bs{B}-lr{LR}-e200`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
The sweep is successful when:
|
||||
|
||||
1. A shared safe batch size has been found.
|
||||
2. All 10 unique runs have launchable commands and isolated run directories.
|
||||
3. Each run trains for about 200 epochs worth of steps.
|
||||
4. Rollout validation occurs every 20 epochs.
|
||||
5. SwanLab records train metrics and rollout rewards.
|
||||
6. Final comparison can be made using best rollout average reward and training traces across:
|
||||
- 2-point pretraining comparison
|
||||
- 9-point emb/layer sweep
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No change to dataset format.
|
||||
- No change to model architecture beyond the requested pretraining toggle and Transformer width/depth sweep.
|
||||
- No attempt to equalize wall-clock time across models.
|
||||
- No multi-GPU scheduling.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
### Risk: larger models may OOM
|
||||
|
||||
Mitigation: probe the largest model first and choose one shared safe batch size.
|
||||
|
||||
### Risk: long runs may be interrupted
|
||||
|
||||
Mitigation: use per-run directories, periodic checkpoints, and resumable launch commands.
|
||||
|
||||
### Risk: rollout validation increases total training time
|
||||
|
||||
Mitigation: keep rollout headless, CPU-based, and limited to 3 episodes every 20 epochs.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 346 KiB |
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"akita_black_bowl_1": {
|
||||
"min": [
|
||||
-0.10497,
|
||||
-0.014979,
|
||||
0.898407
|
||||
],
|
||||
"max": [
|
||||
-0.075125,
|
||||
0.013868,
|
||||
0.899216
|
||||
],
|
||||
"range": [
|
||||
0.029845,
|
||||
0.028847,
|
||||
0.000809
|
||||
],
|
||||
"std": [
|
||||
0.009227,
|
||||
0.008342,
|
||||
0.000113
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"min": [
|
||||
0.035635,
|
||||
-0.034313,
|
||||
0.902506
|
||||
],
|
||||
"max": [
|
||||
0.065008,
|
||||
-0.008729,
|
||||
0.902506
|
||||
],
|
||||
"range": [
|
||||
0.029373,
|
||||
0.025583,
|
||||
1e-06
|
||||
],
|
||||
"std": [
|
||||
0.009018,
|
||||
0.007293,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"min": [
|
||||
-0.214447,
|
||||
-0.064733,
|
||||
0.898875
|
||||
],
|
||||
"max": [
|
||||
-0.186022,
|
||||
-0.035164,
|
||||
0.898875
|
||||
],
|
||||
"range": [
|
||||
0.028425,
|
||||
0.029569,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.007991,
|
||||
0.009218,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"min": [
|
||||
-0.069231,
|
||||
0.111593,
|
||||
0.908921
|
||||
],
|
||||
"max": [
|
||||
-0.030195,
|
||||
0.146634,
|
||||
0.908921
|
||||
],
|
||||
"range": [
|
||||
0.039036,
|
||||
0.035042,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.011587,
|
||||
0.009411,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"wooden_cabinet_1": {
|
||||
"min": [
|
||||
0.020628,
|
||||
-0.249667,
|
||||
0.905
|
||||
],
|
||||
"max": [
|
||||
0.039668,
|
||||
-0.23086,
|
||||
0.905
|
||||
],
|
||||
"range": [
|
||||
0.01904,
|
||||
0.018807,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.005624,
|
||||
0.005887,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"flat_stove_1": {
|
||||
"min": [
|
||||
-0.41998,
|
||||
0.200256,
|
||||
0.905
|
||||
],
|
||||
"max": [
|
||||
-0.400671,
|
||||
0.219361,
|
||||
0.905
|
||||
],
|
||||
"range": [
|
||||
0.019309,
|
||||
0.019105,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.00542,
|
||||
0.005313,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"wine_rack_1": {
|
||||
"min": [
|
||||
-0.269738,
|
||||
-0.269776,
|
||||
0.92
|
||||
],
|
||||
"max": [
|
||||
-0.250473,
|
||||
-0.250372,
|
||||
0.92
|
||||
],
|
||||
"range": [
|
||||
0.019265,
|
||||
0.019404,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.005695,
|
||||
0.005698,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"eef_pos": {
|
||||
"min": [
|
||||
-0.218347,
|
||||
-0.022207,
|
||||
1.155793
|
||||
],
|
||||
"max": [
|
||||
-0.191731,
|
||||
0.016482,
|
||||
1.188032
|
||||
],
|
||||
"range": [
|
||||
0.026616,
|
||||
0.038689,
|
||||
0.03224
|
||||
],
|
||||
"std": [
|
||||
0.005112,
|
||||
0.009859,
|
||||
0.007417
|
||||
]
|
||||
},
|
||||
"joint_pos": {
|
||||
"min": [
|
||||
-0.032644,
|
||||
-0.185945,
|
||||
-0.035387,
|
||||
-2.486189,
|
||||
-0.01145,
|
||||
2.185933,
|
||||
0.745322
|
||||
],
|
||||
"max": [
|
||||
0.021994,
|
||||
-0.132522,
|
||||
0.029516,
|
||||
-2.375572,
|
||||
0.008342,
|
||||
2.274589,
|
||||
0.815841
|
||||
],
|
||||
"range": [
|
||||
0.054637,
|
||||
0.053423,
|
||||
0.064903,
|
||||
0.110617,
|
||||
0.019791,
|
||||
0.088656,
|
||||
0.070519
|
||||
],
|
||||
"std": [
|
||||
0.013218,
|
||||
0.012722,
|
||||
0.015234,
|
||||
0.021384,
|
||||
0.004568,
|
||||
0.020009,
|
||||
0.017693
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,552 @@
|
||||
[
|
||||
{
|
||||
"init_id": 0,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.097626,
|
||||
-0.008583,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.062074,
|
||||
-0.009382,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.204233,
|
||||
-0.061651,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.046026,
|
||||
0.134473,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 1,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.096428,
|
||||
0.013749,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.058727,
|
||||
-0.034313,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.203176,
|
||||
-0.053714,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.040864,
|
||||
0.121312,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
-0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 2,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.103282,
|
||||
0.002253,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.052456,
|
||||
-0.012705,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.193374,
|
||||
-0.036331,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.066908,
|
||||
0.118864,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
-0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 3,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.090777,
|
||||
-0.002232,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.040521,
|
||||
-0.022222,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.20432,
|
||||
-0.056655,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.031539,
|
||||
0.123188,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 4,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.104225,
|
||||
0.013797,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.03824,
|
||||
-0.010652,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.19591,
|
||||
-0.039959,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.037586,
|
||||
0.133424,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 5,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.08334,
|
||||
0.002911,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.064465,
|
||||
-0.028594,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.198764,
|
||||
-0.053643,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.063907,
|
||||
0.120699,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 6,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.076496,
|
||||
-0.00786,
|
||||
0.898414
|
||||
],
|
||||
"quat": [
|
||||
0.706935,
|
||||
-0.000206,
|
||||
-0.00017,
|
||||
0.707279
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.036954,
|
||||
-0.031686,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707659,
|
||||
-3.5e-05,
|
||||
1e-06,
|
||||
0.706554
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.19987,
|
||||
-0.056163,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.053796,
|
||||
0.120126,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
-0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 7,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.075125,
|
||||
-0.003472,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.046896,
|
||||
-0.014595,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.1875,
|
||||
-0.0641,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.060927,
|
||||
0.116114,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
-0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 8,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.086798,
|
||||
0.010183,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.057509,
|
||||
-0.011733,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.20364,
|
||||
-0.035474,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.040355,
|
||||
0.128839,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 9,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.081038,
|
||||
-0.000113,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.049027,
|
||||
-0.013357,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.203822,
|
||||
-0.057315,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.06023,
|
||||
0.136331,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"checkpoint": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/checkpoints/vla_model_step_83124.pt",
|
||||
"num_rollouts": 5,
|
||||
"episodes": [
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01",
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_summary.json"
|
||||
},
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02",
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_summary.json"
|
||||
},
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03",
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_summary.json"
|
||||
},
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04",
|
||||
"avg_reward": 2.0,
|
||||
"avg_max_reward": 2.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_summary.json"
|
||||
},
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05",
|
||||
"avg_reward": 1408.0,
|
||||
"avg_max_reward": 4.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_summary.json"
|
||||
}
|
||||
],
|
||||
"mean_avg_reward": 282.0,
|
||||
"best_avg_reward": 1408.0
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
0.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
0.0
|
||||
],
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 416.328579450197,
|
||||
"control_fps": 16.62784691072735,
|
||||
"avg_obs_read_time_ms": 2.911593945131504,
|
||||
"avg_preprocess_time_ms": 6.928545483055392,
|
||||
"avg_inference_time_ms": 2.401948963774235,
|
||||
"avg_env_step_time_ms": 47.72096303665811,
|
||||
"avg_total_time_ms": 60.14007738758145,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 0.0,
|
||||
"episode_max_reward": 0.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.911593945131504,
|
||||
"std": 0.2611419950299626,
|
||||
"min": 2.712091023568064,
|
||||
"max": 7.493758981581777
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.928545483055392,
|
||||
"std": 0.5543077871256182,
|
||||
"min": 6.473146029748023,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.401948963774235,
|
||||
"std": 12.794224178610442,
|
||||
"min": 0.49940100871026516,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.72096303665811,
|
||||
"std": 2.1171372236113726,
|
||||
"min": 39.931224018801004,
|
||||
"max": 76.46449701860547
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 60.14007738758145,
|
||||
"std": 13.188809002297118,
|
||||
"min": 56.274613016285,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.911593945131504,
|
||||
"std": 0.2611419950299626,
|
||||
"min": 2.712091023568064,
|
||||
"max": 7.493758981581777
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.928545483055392,
|
||||
"std": 0.5543077871256182,
|
||||
"min": 6.473146029748023,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.401948963774235,
|
||||
"std": 12.794224178610442,
|
||||
"min": 0.49940100871026516,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.72096303665811,
|
||||
"std": 2.1171372236113726,
|
||||
"min": 39.931224018801004,
|
||||
"max": 76.46449701860547
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 60.14007738758145,
|
||||
"std": 13.188809002297118,
|
||||
"min": 56.274613016285,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8758058099562978,
|
||||
"std": 0.0751506551331651,
|
||||
"min": 2.712091023568064,
|
||||
"max": 3.0596989672631025
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.1382775310088284,
|
||||
"std": 1.3865518440871207,
|
||||
"min": 6.516602006740868,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 15.048258656778755,
|
||||
"std": 33.453789958947034,
|
||||
"min": 11.253663979005069,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.788851250185296,
|
||||
"std": 1.684109686362168,
|
||||
"min": 39.931224018801004,
|
||||
"max": 56.31602002540603
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 73.03679807624907,
|
||||
"std": 34.000319709963435,
|
||||
"min": 67.7422959706746,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 416.328579450197,
|
||||
"avg_control_fps": 16.62784691072735,
|
||||
"avg_obs_read_time_ms": 2.911593945131504,
|
||||
"avg_preprocess_time_ms": 6.928545483055392,
|
||||
"avg_inference_time_ms": 2.401948963774235,
|
||||
"avg_env_step_time_ms": 47.72096303665811,
|
||||
"avg_total_time_ms": 60.14007738758145,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.911593945131504,
|
||||
"std": 0.2611419950299626,
|
||||
"min": 2.712091023568064,
|
||||
"max": 7.493758981581777
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.928545483055392,
|
||||
"std": 0.5543077871256182,
|
||||
"min": 6.473146029748023,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.401948963774235,
|
||||
"std": 12.794224178610442,
|
||||
"min": 0.49940100871026516,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.72096303665811,
|
||||
"std": 2.1171372236113726,
|
||||
"min": 39.931224018801004,
|
||||
"max": 76.46449701860547
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 60.14007738758145,
|
||||
"std": 13.188809002297118,
|
||||
"min": 56.274613016285,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8758058099562978,
|
||||
"std": 0.0751506551331651,
|
||||
"min": 2.712091023568064,
|
||||
"max": 3.0596989672631025
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.1382775310088284,
|
||||
"std": 1.3865518440871207,
|
||||
"min": 6.516602006740868,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 15.048258656778755,
|
||||
"std": 33.453789958947034,
|
||||
"min": 11.253663979005069,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.788851250185296,
|
||||
"std": 1.684109686362168,
|
||||
"min": 39.931224018801004,
|
||||
"max": 56.31602002540603
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 73.03679807624907,
|
||||
"std": 34.000319709963435,
|
||||
"min": 67.7422959706746,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
0.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
0.0
|
||||
],
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 423.2530657749397,
|
||||
"control_fps": 16.369630991018912,
|
||||
"avg_obs_read_time_ms": 2.922768201263222,
|
||||
"avg_preprocess_time_ms": 7.034949533491662,
|
||||
"avg_inference_time_ms": 2.362652703221621,
|
||||
"avg_env_step_time_ms": 48.58740326665741,
|
||||
"avg_total_time_ms": 61.088731966447085,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 0.0,
|
||||
"episode_max_reward": 0.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.922768201263222,
|
||||
"std": 0.20864628356732534,
|
||||
"min": 2.7384720160625875,
|
||||
"max": 4.775333974976093
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.034949533491662,
|
||||
"std": 0.6020329090344461,
|
||||
"min": 6.419215991627425,
|
||||
"max": 10.473316011484712
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.362652703221621,
|
||||
"std": 12.69272670328858,
|
||||
"min": 0.4969470319338143,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.58740326665741,
|
||||
"std": 3.577976984950618,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 61.088731966447085,
|
||||
"std": 13.227582693702194,
|
||||
"min": 56.584795005619526,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.922768201263222,
|
||||
"std": 0.20864628356732534,
|
||||
"min": 2.7384720160625875,
|
||||
"max": 4.775333974976093
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.034949533491662,
|
||||
"std": 0.6020329090344461,
|
||||
"min": 6.419215991627425,
|
||||
"max": 10.473316011484712
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.362652703221621,
|
||||
"std": 12.69272670328858,
|
||||
"min": 0.4969470319338143,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.58740326665741,
|
||||
"std": 3.577976984950618,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 61.088731966447085,
|
||||
"std": 13.227582693702194,
|
||||
"min": 56.584795005619526,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.89267532604175,
|
||||
"std": 0.11125532360169263,
|
||||
"min": 2.7624600334092975,
|
||||
"max": 3.5272870445623994
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.164920194720088,
|
||||
"std": 0.6480595307828021,
|
||||
"min": 6.461958982981741,
|
||||
"max": 10.244207980576903
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.722178172881037,
|
||||
"std": 33.26832928567316,
|
||||
"min": 10.894106002524495,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.64498521717244,
|
||||
"std": 3.7269116621985248,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 73.61584814432057,
|
||||
"std": 32.98871154259634,
|
||||
"min": 67.3744379892014,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 423.2530657749397,
|
||||
"avg_control_fps": 16.369630991018912,
|
||||
"avg_obs_read_time_ms": 2.922768201263222,
|
||||
"avg_preprocess_time_ms": 7.034949533491662,
|
||||
"avg_inference_time_ms": 2.362652703221621,
|
||||
"avg_env_step_time_ms": 48.58740326665741,
|
||||
"avg_total_time_ms": 61.088731966447085,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.922768201263222,
|
||||
"std": 0.20864628356732534,
|
||||
"min": 2.7384720160625875,
|
||||
"max": 4.775333974976093
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.034949533491662,
|
||||
"std": 0.6020329090344461,
|
||||
"min": 6.419215991627425,
|
||||
"max": 10.473316011484712
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.362652703221621,
|
||||
"std": 12.69272670328858,
|
||||
"min": 0.4969470319338143,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.58740326665741,
|
||||
"std": 3.577976984950618,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 61.088731966447085,
|
||||
"std": 13.227582693702194,
|
||||
"min": 56.584795005619526,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.89267532604175,
|
||||
"std": 0.11125532360169263,
|
||||
"min": 2.7624600334092975,
|
||||
"max": 3.5272870445623994
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.164920194720088,
|
||||
"std": 0.6480595307828021,
|
||||
"min": 6.461958982981741,
|
||||
"max": 10.244207980576903
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.722178172881037,
|
||||
"std": 33.26832928567316,
|
||||
"min": 10.894106002524495,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.64498521717244,
|
||||
"std": 3.7269116621985248,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 73.61584814432057,
|
||||
"std": 32.98871154259634,
|
||||
"min": 67.3744379892014,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
0.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
0.0
|
||||
],
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 428.38141787855113,
|
||||
"control_fps": 16.74224842048907,
|
||||
"avg_obs_read_time_ms": 2.900395128443571,
|
||||
"avg_preprocess_time_ms": 6.847853548159557,
|
||||
"avg_inference_time_ms": 2.3343682948533178,
|
||||
"avg_env_step_time_ms": 47.4696740059049,
|
||||
"avg_total_time_ms": 59.7291340377082,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 0.0,
|
||||
"episode_max_reward": 0.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.900395128443571,
|
||||
"std": 0.2916878594282876,
|
||||
"min": 2.68459296785295,
|
||||
"max": 7.19653902342543
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.847853548159557,
|
||||
"std": 0.21243417443892795,
|
||||
"min": 6.388033041730523,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3343682948533178,
|
||||
"std": 11.800465870815552,
|
||||
"min": 0.49768202006816864,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.4696740059049,
|
||||
"std": 1.7681566119287178,
|
||||
"min": 41.32056800881401,
|
||||
"max": 82.69464899785817
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 59.7291340377082,
|
||||
"std": 11.80948193858811,
|
||||
"min": 56.34060001466423,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.900395128443571,
|
||||
"std": 0.2916878594282876,
|
||||
"min": 2.68459296785295,
|
||||
"max": 7.19653902342543
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.847853548159557,
|
||||
"std": 0.21243417443892795,
|
||||
"min": 6.388033041730523,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3343682948533178,
|
||||
"std": 11.800465870815552,
|
||||
"min": 0.49768202006816864,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.4696740059049,
|
||||
"std": 1.7681566119287178,
|
||||
"min": 41.32056800881401,
|
||||
"max": 82.69464899785817
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 59.7291340377082,
|
||||
"std": 11.80948193858811,
|
||||
"min": 56.34060001466423,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.850530921105846,
|
||||
"std": 0.06315892633423309,
|
||||
"min": 2.729182015173137,
|
||||
"max": 2.99588602501899
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.963662055733783,
|
||||
"std": 0.3317973295122339,
|
||||
"min": 6.484561017714441,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.542899646171877,
|
||||
"std": 30.613461495155526,
|
||||
"min": 11.056077957618982,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.46852555862543,
|
||||
"std": 0.9486706652557946,
|
||||
"min": 41.32056800881401,
|
||||
"max": 49.64756895788014
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 72.01395506920313,
|
||||
"std": 30.190598270939223,
|
||||
"min": 67.28241901146248,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 428.38141787855113,
|
||||
"avg_control_fps": 16.74224842048907,
|
||||
"avg_obs_read_time_ms": 2.900395128443571,
|
||||
"avg_preprocess_time_ms": 6.847853548159557,
|
||||
"avg_inference_time_ms": 2.3343682948533178,
|
||||
"avg_env_step_time_ms": 47.4696740059049,
|
||||
"avg_total_time_ms": 59.7291340377082,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.900395128443571,
|
||||
"std": 0.2916878594282876,
|
||||
"min": 2.68459296785295,
|
||||
"max": 7.19653902342543
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.847853548159557,
|
||||
"std": 0.21243417443892795,
|
||||
"min": 6.388033041730523,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3343682948533178,
|
||||
"std": 11.800465870815552,
|
||||
"min": 0.49768202006816864,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.4696740059049,
|
||||
"std": 1.7681566119287178,
|
||||
"min": 41.32056800881401,
|
||||
"max": 82.69464899785817
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 59.7291340377082,
|
||||
"std": 11.80948193858811,
|
||||
"min": 56.34060001466423,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.850530921105846,
|
||||
"std": 0.06315892633423309,
|
||||
"min": 2.729182015173137,
|
||||
"max": 2.99588602501899
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.963662055733783,
|
||||
"std": 0.3317973295122339,
|
||||
"min": 6.484561017714441,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.542899646171877,
|
||||
"std": 30.613461495155526,
|
||||
"min": 11.056077957618982,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.46852555862543,
|
||||
"std": 0.9486706652557946,
|
||||
"min": 41.32056800881401,
|
||||
"max": 49.64756895788014
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 72.01395506920313,
|
||||
"std": 30.190598270939223,
|
||||
"min": 67.28241901146248,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
2.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
2.0
|
||||
],
|
||||
"avg_reward": 2.0,
|
||||
"avg_max_reward": 2.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 425.1027328138825,
|
||||
"control_fps": 17.116880794537646,
|
||||
"avg_obs_read_time_ms": 2.8568991100681678,
|
||||
"avg_preprocess_time_ms": 6.912527064227366,
|
||||
"avg_inference_time_ms": 2.3523725509377464,
|
||||
"avg_env_step_time_ms": 46.12416496566896,
|
||||
"avg_total_time_ms": 58.42185921626099,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 2.0,
|
||||
"episode_max_reward": 2.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8568991100681678,
|
||||
"std": 0.20988493064877758,
|
||||
"min": 2.6854430325329304,
|
||||
"max": 5.505044013261795
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.912527064227366,
|
||||
"std": 0.3468982959934229,
|
||||
"min": 6.422812002710998,
|
||||
"max": 13.460688991472125
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3523725509377464,
|
||||
"std": 12.345277540311537,
|
||||
"min": 0.4927710397168994,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.12416496566896,
|
||||
"std": 1.6786026968893601,
|
||||
"min": 40.27245403267443,
|
||||
"max": 69.19108401052654
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 58.42185921626099,
|
||||
"std": 12.395693709763066,
|
||||
"min": 54.73730497760698,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8568991100681678,
|
||||
"std": 0.20988493064877758,
|
||||
"min": 2.6854430325329304,
|
||||
"max": 5.505044013261795
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.912527064227366,
|
||||
"std": 0.3468982959934229,
|
||||
"min": 6.422812002710998,
|
||||
"max": 13.460688991472125
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3523725509377464,
|
||||
"std": 12.345277540311537,
|
||||
"min": 0.4927710397168994,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.12416496566896,
|
||||
"std": 1.6786026968893601,
|
||||
"min": 40.27245403267443,
|
||||
"max": 69.19108401052654
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 58.42185921626099,
|
||||
"std": 12.395693709763066,
|
||||
"min": 54.73730497760698,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8195713563367133,
|
||||
"std": 0.05933493418683723,
|
||||
"min": 2.6968690217472613,
|
||||
"max": 2.953937044367194
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.006697107333986,
|
||||
"std": 0.4376351299601935,
|
||||
"min": 6.50148798013106,
|
||||
"max": 10.386412963271141
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.704165457012344,
|
||||
"std": 32.214912229767435,
|
||||
"min": 11.083905992563814,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.088106907105114,
|
||||
"std": 1.088099613967191,
|
||||
"min": 40.27245403267443,
|
||||
"max": 49.108261009678245
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 70.80417735150232,
|
||||
"std": 31.96448918725213,
|
||||
"min": 65.67699601873755,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 425.1027328138825,
|
||||
"avg_control_fps": 17.116880794537646,
|
||||
"avg_obs_read_time_ms": 2.8568991100681678,
|
||||
"avg_preprocess_time_ms": 6.912527064227366,
|
||||
"avg_inference_time_ms": 2.3523725509377464,
|
||||
"avg_env_step_time_ms": 46.12416496566896,
|
||||
"avg_total_time_ms": 58.42185921626099,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8568991100681678,
|
||||
"std": 0.20988493064877758,
|
||||
"min": 2.6854430325329304,
|
||||
"max": 5.505044013261795
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.912527064227366,
|
||||
"std": 0.3468982959934229,
|
||||
"min": 6.422812002710998,
|
||||
"max": 13.460688991472125
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3523725509377464,
|
||||
"std": 12.345277540311537,
|
||||
"min": 0.4927710397168994,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.12416496566896,
|
||||
"std": 1.6786026968893601,
|
||||
"min": 40.27245403267443,
|
||||
"max": 69.19108401052654
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 58.42185921626099,
|
||||
"std": 12.395693709763066,
|
||||
"min": 54.73730497760698,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8195713563367133,
|
||||
"std": 0.05933493418683723,
|
||||
"min": 2.6968690217472613,
|
||||
"max": 2.953937044367194
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.006697107333986,
|
||||
"std": 0.4376351299601935,
|
||||
"min": 6.50148798013106,
|
||||
"max": 10.386412963271141
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.704165457012344,
|
||||
"std": 32.214912229767435,
|
||||
"min": 11.083905992563814,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.088106907105114,
|
||||
"std": 1.088099613967191,
|
||||
"min": 40.27245403267443,
|
||||
"max": 49.108261009678245
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 70.80417735150232,
|
||||
"std": 31.96448918725213,
|
||||
"min": 65.67699601873755,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
1408.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
4.0
|
||||
],
|
||||
"avg_reward": 1408.0,
|
||||
"avg_max_reward": 4.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 421.5143683898348,
|
||||
"control_fps": 15.98697584872221,
|
||||
"avg_obs_read_time_ms": 2.916401576450361,
|
||||
"avg_preprocess_time_ms": 6.9587834345709,
|
||||
"avg_inference_time_ms": 2.372398368814694,
|
||||
"avg_env_step_time_ms": 50.12335285793857,
|
||||
"avg_total_time_ms": 62.55091703787912,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 1408.0,
|
||||
"episode_max_reward": 4.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.916401576450361,
|
||||
"std": 0.19681075249117103,
|
||||
"min": 2.723354031331837,
|
||||
"max": 4.304414032958448
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.9587834345709,
|
||||
"std": 0.32918428959573737,
|
||||
"min": 6.440367025788873,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.372398368814694,
|
||||
"std": 12.755981125405425,
|
||||
"min": 0.4968460416421294,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 50.12335285793857,
|
||||
"std": 2.0541933730540083,
|
||||
"min": 41.243349958676845,
|
||||
"max": 59.02870395220816
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 62.55091703787912,
|
||||
"std": 12.685506605997542,
|
||||
"min": 57.05423402832821,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.916401576450361,
|
||||
"std": 0.19681075249117103,
|
||||
"min": 2.723354031331837,
|
||||
"max": 4.304414032958448
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.9587834345709,
|
||||
"std": 0.32918428959573737,
|
||||
"min": 6.440367025788873,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.372398368814694,
|
||||
"std": 12.755981125405425,
|
||||
"min": 0.4968460416421294,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 50.12335285793857,
|
||||
"std": 2.0541933730540083,
|
||||
"min": 41.243349958676845,
|
||||
"max": 59.02870395220816
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 62.55091703787912,
|
||||
"std": 12.685506605997542,
|
||||
"min": 57.05423402832821,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8879472850927743,
|
||||
"std": 0.09183274389315361,
|
||||
"min": 2.758314018137753,
|
||||
"max": 3.517468983773142
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.111402045914226,
|
||||
"std": 0.5243812452094909,
|
||||
"min": 6.641154002863914,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.798059454320041,
|
||||
"std": 33.432189809551524,
|
||||
"min": 10.985016997437924,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 49.97409828435841,
|
||||
"std": 2.1747279952000875,
|
||||
"min": 41.243349958676845,
|
||||
"max": 56.928464968223125
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 74.9630101153426,
|
||||
"std": 32.74896535841012,
|
||||
"min": 67.58100300794467,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 421.5143683898348,
|
||||
"avg_control_fps": 15.98697584872221,
|
||||
"avg_obs_read_time_ms": 2.916401576450361,
|
||||
"avg_preprocess_time_ms": 6.9587834345709,
|
||||
"avg_inference_time_ms": 2.372398368814694,
|
||||
"avg_env_step_time_ms": 50.12335285793857,
|
||||
"avg_total_time_ms": 62.55091703787912,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.916401576450361,
|
||||
"std": 0.19681075249117103,
|
||||
"min": 2.723354031331837,
|
||||
"max": 4.304414032958448
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.9587834345709,
|
||||
"std": 0.32918428959573737,
|
||||
"min": 6.440367025788873,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.372398368814694,
|
||||
"std": 12.755981125405425,
|
||||
"min": 0.4968460416421294,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 50.12335285793857,
|
||||
"std": 2.0541933730540083,
|
||||
"min": 41.243349958676845,
|
||||
"max": 59.02870395220816
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 62.55091703787912,
|
||||
"std": 12.685506605997542,
|
||||
"min": 57.05423402832821,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8879472850927743,
|
||||
"std": 0.09183274389315361,
|
||||
"min": 2.758314018137753,
|
||||
"max": 3.517468983773142
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.111402045914226,
|
||||
"std": 0.5243812452094909,
|
||||
"min": 6.641154002863914,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.798059454320041,
|
||||
"std": 33.432189809551524,
|
||||
"min": 10.985016997437924,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 49.97409828435841,
|
||||
"std": 2.1747279952000875,
|
||||
"min": 41.243349958676845,
|
||||
"max": 56.928464968223125
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 74.9630101153426,
|
||||
"std": 32.74896535841012,
|
||||
"min": 67.58100300794467,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"status": "success",
|
||||
"task_name": "sim_transfer",
|
||||
"success_video": "/data/roboimi_rollout_videos/scripted_success_20260529/sim_transfer/success_front.mp4",
|
||||
"max_reward_required": 4,
|
||||
"attempts": [
|
||||
{
|
||||
"attempt": 1,
|
||||
"task_name": "sim_transfer",
|
||||
"success": true,
|
||||
"sum_reward": 1238.0,
|
||||
"max_reward": 4.0,
|
||||
"max_reward_required": 4,
|
||||
"max_reward_step": 503,
|
||||
"episode_len": 700,
|
||||
"elapsed_s": 29.355701208114624,
|
||||
"sim_steps_per_wall_s": 23.845453223461178,
|
||||
"video_path": "/data/roboimi_rollout_videos/scripted_success_20260529/sim_transfer/attempt_001_front.mp4",
|
||||
"task_state": [
|
||||
0.0338307802064175,
|
||||
0.9599788062599779,
|
||||
0.47
|
||||
],
|
||||
"reward_trace": [
|
||||
{
|
||||
"step": 0,
|
||||
"reward": 0.0
|
||||
},
|
||||
{
|
||||
"step": 278,
|
||||
"reward": 2.0
|
||||
},
|
||||
{
|
||||
"step": 503,
|
||||
"reward": 4.0
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"created_at": "2026-05-29T19:28:23+0800"
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,80 @@
|
||||
{
|
||||
"status": "success",
|
||||
"task_name": "sim_air_insert_socket_peg",
|
||||
"success_video": "/data/roboimi_rollout_videos/scripted_success_20260529/sim_air_insert_socket_peg/success_front.mp4",
|
||||
"max_reward_required": 5,
|
||||
"attempts": [
|
||||
{
|
||||
"attempt": 1,
|
||||
"task_name": "sim_air_insert_socket_peg",
|
||||
"success": true,
|
||||
"sum_reward": 1942.0,
|
||||
"max_reward": 5.0,
|
||||
"max_reward_required": 5,
|
||||
"max_reward_step": 583,
|
||||
"episode_len": 750,
|
||||
"elapsed_s": 35.83620357513428,
|
||||
"sim_steps_per_wall_s": 20.92855618557775,
|
||||
"video_path": "/data/roboimi_rollout_videos/scripted_success_20260529/sim_air_insert_socket_peg/attempt_001_front.mp4",
|
||||
"task_state": {
|
||||
"socket_pos": [
|
||||
-0.14154230058193207,
|
||||
0.9299893975257874,
|
||||
0.47200000286102295
|
||||
],
|
||||
"socket_quat": [
|
||||
1.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0
|
||||
],
|
||||
"peg_pos": [
|
||||
0.13059355318546295,
|
||||
0.881460428237915,
|
||||
0.46000000834465027
|
||||
],
|
||||
"peg_quat": [
|
||||
1.0,
|
||||
0.0,
|
||||
0.0,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"reward_trace": [
|
||||
{
|
||||
"step": 0,
|
||||
"reward": 0.0
|
||||
},
|
||||
{
|
||||
"step": 132,
|
||||
"reward": 1.0
|
||||
},
|
||||
{
|
||||
"step": 171,
|
||||
"reward": 2.0
|
||||
},
|
||||
{
|
||||
"step": 185,
|
||||
"reward": 3.0
|
||||
},
|
||||
{
|
||||
"step": 186,
|
||||
"reward": 2.0
|
||||
},
|
||||
{
|
||||
"step": 190,
|
||||
"reward": 3.0
|
||||
},
|
||||
{
|
||||
"step": 351,
|
||||
"reward": 4.0
|
||||
},
|
||||
{
|
||||
"step": 583,
|
||||
"reward": 5.0
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"created_at": "2026-05-29T19:31:20+0800"
|
||||
}
|
||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,191 @@
|
||||
{
|
||||
"agent": {
|
||||
"vision_backbone": {
|
||||
"_target_": "roboimi.vla.models.backbones.resnet_diffusion.ResNetDiffusionBackbone",
|
||||
"vision_backbone": "resnet18",
|
||||
"pretrained_backbone_weights": null,
|
||||
"vision_backbone_mode": "resnet",
|
||||
"freeze_backbone": false,
|
||||
"input_shape": [
|
||||
3,
|
||||
224,
|
||||
224
|
||||
],
|
||||
"crop_shape": null,
|
||||
"crop_is_random": true,
|
||||
"use_group_norm": true,
|
||||
"spatial_softmax_num_keypoints": 32,
|
||||
"use_separate_rgb_encoder_per_camera": true,
|
||||
"output_tokens_per_camera": false,
|
||||
"num_cameras": 3,
|
||||
"attnres_stem_dim": 64,
|
||||
"attnres_stage_dims": [
|
||||
64,
|
||||
128,
|
||||
256,
|
||||
512
|
||||
],
|
||||
"attnres_stage_depths": [
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"attnres_stage_heads": [
|
||||
4,
|
||||
4,
|
||||
8,
|
||||
8
|
||||
],
|
||||
"attnres_stage_kv_heads": [
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1
|
||||
],
|
||||
"attnres_stage_window_sizes": [
|
||||
7,
|
||||
7,
|
||||
7,
|
||||
7
|
||||
],
|
||||
"attnres_dropout": 0.0,
|
||||
"attnres_ffn_mult": 2.667,
|
||||
"attnres_eps": 1e-06,
|
||||
"attnres_rope_theta": 10000.0,
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top",
|
||||
"front"
|
||||
]
|
||||
},
|
||||
"state_encoder": {
|
||||
"_target_": "roboimi.vla.modules.encoders.IdentityStateEncoder"
|
||||
},
|
||||
"action_encoder": {
|
||||
"_target_": "roboimi.vla.modules.encoders.IdentityActionEncoder"
|
||||
},
|
||||
"head": {
|
||||
"_target_": "roboimi.vla.models.heads.transformer1d.Transformer1D",
|
||||
"_partial_": true,
|
||||
"n_layer": 18,
|
||||
"n_head": 4,
|
||||
"n_emb": 384,
|
||||
"p_drop_emb": 0.05,
|
||||
"p_drop_attn": 0.05,
|
||||
"causal_attn": false,
|
||||
"time_as_cond": true,
|
||||
"obs_as_cond": true,
|
||||
"n_cond_layers": 1,
|
||||
"input_dim": 16,
|
||||
"output_dim": 16,
|
||||
"horizon": 16,
|
||||
"n_obs_steps": 2,
|
||||
"cond_dim": 208
|
||||
},
|
||||
"_target_": "roboimi.vla.agent.VLAAgent",
|
||||
"action_dim": 16,
|
||||
"obs_dim": 16,
|
||||
"normalization_type": "min_max",
|
||||
"pred_horizon": 16,
|
||||
"obs_horizon": 2,
|
||||
"num_action_steps": 8,
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top",
|
||||
"front"
|
||||
],
|
||||
"num_cams": 3,
|
||||
"diffusion_steps": 100,
|
||||
"inference_steps": 100,
|
||||
"head_type": "transformer"
|
||||
},
|
||||
"data": {
|
||||
"_target_": "roboimi.vla.data.simpe_robot_dataset.SimpleRobotDataset",
|
||||
"dataset_dir": "/home/droid/project/diana_sim/sim_transfer",
|
||||
"pred_horizon": 16,
|
||||
"obs_horizon": 2,
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top",
|
||||
"front"
|
||||
],
|
||||
"image_resize_shape": [
|
||||
224,
|
||||
224
|
||||
]
|
||||
},
|
||||
"eval": {
|
||||
"ckpt_path": "/home/droid/project/roboimi/runs/sim-transfer-no-pretrain-ph16-emb384-l18-infer100-unfreeze-150k-roll5x5-20260331-235043/checkpoints/vla_model_best.pt",
|
||||
"num_episodes": 3,
|
||||
"num_workers": 1,
|
||||
"cuda_devices": null,
|
||||
"response_timeout_s": 300.0,
|
||||
"server_startup_timeout_s": 300.0,
|
||||
"max_timesteps": 700,
|
||||
"device": "cuda",
|
||||
"task_name": "sim_transfer",
|
||||
"task_description": null,
|
||||
"num_queries": 8,
|
||||
"obs_horizon": 2,
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top",
|
||||
"front"
|
||||
],
|
||||
"use_smoothing": false,
|
||||
"smooth_method": "ema",
|
||||
"smooth_alpha": 0.3,
|
||||
"headless": true,
|
||||
"verbose_action": false,
|
||||
"artifact_dir": null,
|
||||
"save_artifacts": false,
|
||||
"save_timing": false,
|
||||
"save_trajectory": false,
|
||||
"save_summary_json": false,
|
||||
"save_trajectory_npz": false,
|
||||
"save_trajectory_image": false,
|
||||
"trajectory_image_camera": null,
|
||||
"trajectory_image_camera_name": null,
|
||||
"record_video": false,
|
||||
"video_camera": null,
|
||||
"video_camera_name": null,
|
||||
"video_fps": 30
|
||||
},
|
||||
"train": {
|
||||
"batch_size": 16,
|
||||
"lr": 0.0001,
|
||||
"max_steps": 100000,
|
||||
"device": "cuda",
|
||||
"disable_cudnn": false,
|
||||
"num_workers": 12,
|
||||
"val_split": 0.0,
|
||||
"val_episode_indices": null,
|
||||
"action_mse_val_freq_epochs": 0,
|
||||
"seed": 42,
|
||||
"log_freq": 100,
|
||||
"save_freq": 2000,
|
||||
"use_swanlab": false,
|
||||
"swanlab_project": "roboimi-vla",
|
||||
"swanlab_run_name": null,
|
||||
"rollout_val_freq_epochs": 50,
|
||||
"rollout_validate_on_checkpoint": false,
|
||||
"rollout_num_episodes": 3,
|
||||
"rollout_device": "cuda",
|
||||
"rollout_num_workers": null,
|
||||
"rollout_cuda_devices": null,
|
||||
"rollout_response_timeout_s": 300.0,
|
||||
"rollout_server_startup_timeout_s": 300.0,
|
||||
"warmup_steps": 2000,
|
||||
"scheduler_type": "cosine",
|
||||
"min_lr": 1e-06,
|
||||
"weight_decay": 1e-05,
|
||||
"grad_clip": 1.0,
|
||||
"pretrained_ckpt": null
|
||||
},
|
||||
"experiment": {
|
||||
"name": "vla_diffusion",
|
||||
"notes": "",
|
||||
"tags": []
|
||||
}
|
||||
}
|
||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,198 @@
|
||||
{
|
||||
"agent": {
|
||||
"vision_backbone": {
|
||||
"_target_": "roboimi.vla.models.backbones.resnet_diffusion.ResNetDiffusionBackbone",
|
||||
"vision_backbone": "resnet18",
|
||||
"pretrained_backbone_weights": null,
|
||||
"vision_backbone_mode": "resnet",
|
||||
"freeze_backbone": false,
|
||||
"input_shape": [
|
||||
3,
|
||||
224,
|
||||
224
|
||||
],
|
||||
"crop_shape": null,
|
||||
"crop_is_random": true,
|
||||
"use_group_norm": true,
|
||||
"spatial_softmax_num_keypoints": 32,
|
||||
"use_separate_rgb_encoder_per_camera": true,
|
||||
"output_tokens_per_camera": false,
|
||||
"num_cameras": 3,
|
||||
"attnres_stem_dim": 64,
|
||||
"attnres_stage_dims": [
|
||||
64,
|
||||
128,
|
||||
256,
|
||||
512
|
||||
],
|
||||
"attnres_stage_depths": [
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"attnres_stage_heads": [
|
||||
4,
|
||||
4,
|
||||
8,
|
||||
8
|
||||
],
|
||||
"attnres_stage_kv_heads": [
|
||||
1,
|
||||
1,
|
||||
1,
|
||||
1
|
||||
],
|
||||
"attnres_stage_window_sizes": [
|
||||
7,
|
||||
7,
|
||||
7,
|
||||
7
|
||||
],
|
||||
"attnres_dropout": 0.0,
|
||||
"attnres_ffn_mult": 2.667,
|
||||
"attnres_eps": 1e-06,
|
||||
"attnres_rope_theta": 10000.0,
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top",
|
||||
"front"
|
||||
]
|
||||
},
|
||||
"state_encoder": {
|
||||
"_target_": "roboimi.vla.modules.encoders.IdentityStateEncoder"
|
||||
},
|
||||
"action_encoder": {
|
||||
"_target_": "roboimi.vla.modules.encoders.IdentityActionEncoder"
|
||||
},
|
||||
"head": {
|
||||
"_target_": "roboimi.vla.models.heads.imf_transformer1d.IMFTransformer1D",
|
||||
"_partial_": true,
|
||||
"input_dim": 16,
|
||||
"output_dim": 16,
|
||||
"horizon": 32,
|
||||
"n_obs_steps": 2,
|
||||
"cond_dim": 208,
|
||||
"n_layer": 12,
|
||||
"n_head": 1,
|
||||
"n_emb": 384,
|
||||
"p_drop_emb": 0.1,
|
||||
"p_drop_attn": 0.1,
|
||||
"causal_attn": false,
|
||||
"time_as_cond": true,
|
||||
"obs_as_cond": true,
|
||||
"n_cond_layers": 0,
|
||||
"backbone_type": "attnres_full",
|
||||
"n_kv_head": 1,
|
||||
"attn_res_ffn_mult": 2.667,
|
||||
"attn_res_eps": 1e-06,
|
||||
"attn_res_rope_theta": 10000.0
|
||||
},
|
||||
"_target_": "roboimi.vla.agent_imf.IMFVLAAgent",
|
||||
"action_dim": 16,
|
||||
"obs_dim": 16,
|
||||
"normalization_type": "min_max",
|
||||
"pred_horizon": 32,
|
||||
"obs_horizon": 2,
|
||||
"num_action_steps": 16,
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top",
|
||||
"front"
|
||||
],
|
||||
"num_cams": 3,
|
||||
"diffusion_steps": 100,
|
||||
"inference_steps": 1,
|
||||
"head_type": "transformer",
|
||||
"loss_type": "pseudo_huber",
|
||||
"pseudo_huber_delta": 1.0
|
||||
},
|
||||
"data": {
|
||||
"_target_": "roboimi.vla.data.simpe_robot_dataset.SimpleRobotDataset",
|
||||
"dataset_dir": "/home/droid/project/diana_sim/sim_transfer",
|
||||
"pred_horizon": 32,
|
||||
"obs_horizon": 2,
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top",
|
||||
"front"
|
||||
],
|
||||
"image_resize_shape": [
|
||||
224,
|
||||
224
|
||||
]
|
||||
},
|
||||
"eval": {
|
||||
"ckpt_path": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/runs/sim-transfer-imf-attnres-ph32-exec16-emb384-l12-infer1-unfreeze-step50k-roll5x5-20260403-141304/checkpoints/vla_model_best.pt",
|
||||
"num_episodes": 3,
|
||||
"num_workers": 1,
|
||||
"cuda_devices": null,
|
||||
"response_timeout_s": 300.0,
|
||||
"server_startup_timeout_s": 300.0,
|
||||
"max_timesteps": 700,
|
||||
"device": "cuda",
|
||||
"task_name": "sim_transfer",
|
||||
"task_description": null,
|
||||
"num_queries": 16,
|
||||
"obs_horizon": 2,
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top",
|
||||
"front"
|
||||
],
|
||||
"use_smoothing": false,
|
||||
"smooth_method": "ema",
|
||||
"smooth_alpha": 0.3,
|
||||
"headless": true,
|
||||
"verbose_action": false,
|
||||
"artifact_dir": null,
|
||||
"save_artifacts": false,
|
||||
"save_timing": false,
|
||||
"save_trajectory": false,
|
||||
"save_summary_json": false,
|
||||
"save_trajectory_npz": false,
|
||||
"save_trajectory_image": false,
|
||||
"trajectory_image_camera": null,
|
||||
"trajectory_image_camera_name": null,
|
||||
"record_video": false,
|
||||
"video_camera": null,
|
||||
"video_camera_name": null,
|
||||
"video_fps": 30
|
||||
},
|
||||
"train": {
|
||||
"batch_size": 16,
|
||||
"lr": 0.0001,
|
||||
"max_steps": 100000,
|
||||
"device": "cuda",
|
||||
"disable_cudnn": false,
|
||||
"num_workers": 12,
|
||||
"val_split": 0.0,
|
||||
"val_episode_indices": null,
|
||||
"action_mse_val_freq_epochs": 0,
|
||||
"seed": 42,
|
||||
"log_freq": 100,
|
||||
"save_freq": 2000,
|
||||
"use_swanlab": false,
|
||||
"swanlab_project": "roboimi-vla",
|
||||
"swanlab_run_name": null,
|
||||
"rollout_val_freq_epochs": 50,
|
||||
"rollout_validate_on_checkpoint": false,
|
||||
"rollout_num_episodes": 3,
|
||||
"rollout_device": "cuda",
|
||||
"rollout_num_workers": null,
|
||||
"rollout_cuda_devices": null,
|
||||
"rollout_response_timeout_s": 300.0,
|
||||
"rollout_server_startup_timeout_s": 300.0,
|
||||
"warmup_steps": 2000,
|
||||
"scheduler_type": "cosine",
|
||||
"min_lr": 1e-06,
|
||||
"weight_decay": 1e-05,
|
||||
"grad_clip": 1.0,
|
||||
"pretrained_ckpt": null
|
||||
},
|
||||
"experiment": {
|
||||
"name": "vla_diffusion",
|
||||
"notes": "",
|
||||
"tags": []
|
||||
}
|
||||
}
|
||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
@@ -113,6 +113,7 @@ def prepare_observation(
|
||||
obs: Dict,
|
||||
camera_names: list,
|
||||
image_resize_shape: Optional[tuple[int, int]] = (224, 224),
|
||||
task_description: Optional[str] = None,
|
||||
) -> Dict:
|
||||
"""
|
||||
将环境观测转换为 agent 格式。
|
||||
@@ -139,7 +140,32 @@ def prepare_observation(
|
||||
# 转换 qpos: numpy -> tensor
|
||||
qpos = torch.from_numpy(obs['qpos']).float()
|
||||
|
||||
return {'qpos': qpos, 'images': images}
|
||||
observation = {'qpos': qpos, 'images': images}
|
||||
if 'task' in obs:
|
||||
observation['task'] = obs['task']
|
||||
elif task_description is not None:
|
||||
observation['task'] = task_description
|
||||
return observation
|
||||
|
||||
|
||||
def _normalize_resize_shape(shape) -> Optional[tuple[int, int]]:
|
||||
if shape is None:
|
||||
return None
|
||||
normalized = tuple(int(v) for v in shape)
|
||||
if len(normalized) != 2:
|
||||
raise ValueError(f'image resize shape must contain exactly two values, got {normalized}')
|
||||
return normalized
|
||||
|
||||
|
||||
def _resolve_eval_image_resize_shape(cfg: DictConfig) -> Optional[tuple[int, int]]:
|
||||
image_resize_shape = cfg.get('data', {}).get('image_resize_shape', (224, 224))
|
||||
agent_cfg = cfg.agent
|
||||
for backbone_key in ('vision_backbone', 'condition_encoder'):
|
||||
backbone_cfg = agent_cfg.get(backbone_key, None)
|
||||
if backbone_cfg is not None and 'eval_image_resize_shape' in backbone_cfg:
|
||||
image_resize_shape = backbone_cfg.get('eval_image_resize_shape')
|
||||
break
|
||||
return _normalize_resize_shape(image_resize_shape)
|
||||
|
||||
|
||||
def _resolve_policy_camera_names(cfg: DictConfig) -> list[str]:
|
||||
@@ -159,6 +185,7 @@ def _new_local_policy_queues(obs_horizon: int) -> dict[str, deque]:
|
||||
return {
|
||||
'qpos': deque(maxlen=int(obs_horizon)),
|
||||
'images': deque(maxlen=int(obs_horizon)),
|
||||
'task': deque(maxlen=int(obs_horizon)),
|
||||
'action': deque(),
|
||||
}
|
||||
|
||||
@@ -174,6 +201,8 @@ def _populate_local_policy_queues(
|
||||
camera_name: image.detach().clone()
|
||||
for camera_name, image in observation['images'].items()
|
||||
})
|
||||
if 'task' in observation:
|
||||
queues['task'].append(observation['task'])
|
||||
|
||||
|
||||
def _prepare_local_policy_batch(
|
||||
@@ -202,7 +231,10 @@ def _prepare_local_policy_batch(
|
||||
).unsqueeze(0)
|
||||
for camera_name in ordered_camera_names
|
||||
}
|
||||
return {'qpos': batch_qpos, 'images': batch_images}
|
||||
batch = {'qpos': batch_qpos, 'images': batch_images}
|
||||
if queues.get('task'):
|
||||
batch['task'] = [list(queues['task'])[-1]]
|
||||
return batch
|
||||
|
||||
|
||||
def _enqueue_predicted_actions(
|
||||
@@ -226,23 +258,29 @@ def _enqueue_predicted_actions(
|
||||
|
||||
|
||||
def _serialize_policy_batch(batch: Dict[str, torch.Tensor]) -> dict[str, Any]:
|
||||
return {
|
||||
serialized = {
|
||||
'qpos': batch['qpos'].detach().cpu().numpy().astype(np.float32, copy=True),
|
||||
'images': {
|
||||
camera_name: image.detach().cpu().numpy().astype(np.float32, copy=True)
|
||||
for camera_name, image in batch['images'].items()
|
||||
},
|
||||
}
|
||||
if 'task' in batch:
|
||||
serialized['task'] = batch['task']
|
||||
return serialized
|
||||
|
||||
|
||||
def _deserialize_policy_batch(batch: dict[str, Any], device: str) -> Dict[str, torch.Tensor]:
|
||||
return {
|
||||
deserialized = {
|
||||
'qpos': torch.as_tensor(batch['qpos'], dtype=torch.float32, device=device),
|
||||
'images': {
|
||||
camera_name: torch.as_tensor(image, dtype=torch.float32, device=device)
|
||||
for camera_name, image in batch['images'].items()
|
||||
},
|
||||
}
|
||||
if 'task' in batch:
|
||||
deserialized['task'] = batch['task']
|
||||
return deserialized
|
||||
|
||||
|
||||
class _LocalPolicyRunner:
|
||||
@@ -986,7 +1024,7 @@ def _print_eval_config(cfg: DictConfig):
|
||||
|
||||
def _build_episode_plans(
|
||||
num_episodes: int,
|
||||
box_poses: Optional[list[np.ndarray]] = None,
|
||||
box_poses: Optional[list[Any]] = None,
|
||||
) -> list[dict[str, Any]]:
|
||||
if box_poses is None:
|
||||
return [
|
||||
@@ -998,12 +1036,25 @@ def _build_episode_plans(
|
||||
return [
|
||||
{
|
||||
'episode_index': int(episode_index),
|
||||
'box_pos': np.asarray(box_pos, dtype=np.float32).copy(),
|
||||
'box_pos': _json_friendly(np.asarray(box_pos, dtype=np.float32).copy()),
|
||||
}
|
||||
for episode_index, box_pos in enumerate(box_poses)
|
||||
]
|
||||
|
||||
|
||||
def _episode_plan_reset_state(plan: dict[str, Any], task_name: str):
|
||||
if 'task_state' in plan:
|
||||
return _json_friendly(plan['task_state'])
|
||||
|
||||
if str(task_name) == 'sim_air_insert_socket_peg':
|
||||
return _sample_task_reset_state(str(task_name))
|
||||
|
||||
task_state = plan.get('box_pos')
|
||||
if task_state is None:
|
||||
return _sample_task_reset_state(str(task_name))
|
||||
return np.asarray(task_state, dtype=np.float32)
|
||||
|
||||
|
||||
def _run_eval_episode_plans(
|
||||
cfg: DictConfig,
|
||||
episode_plans: list[dict[str, Any]],
|
||||
@@ -1015,6 +1066,8 @@ def _run_eval_episode_plans(
|
||||
eval_cfg = cfg.eval
|
||||
device = str(eval_cfg.device)
|
||||
camera_names = list(eval_cfg.camera_names)
|
||||
image_resize_shape = _resolve_eval_image_resize_shape(cfg)
|
||||
task_description = eval_cfg.get('task_description', None)
|
||||
artifact_paths = artifact_paths or _resolve_artifact_paths(eval_cfg)
|
||||
video_recorder = _RolloutVideoRecorder(
|
||||
output_path=artifact_paths['video_mp4'],
|
||||
@@ -1041,12 +1094,7 @@ def _run_eval_episode_plans(
|
||||
episode_max_rewards = []
|
||||
for plan in episode_plans:
|
||||
episode_idx = int(plan['episode_index'])
|
||||
task_state = plan.get('box_pos')
|
||||
if task_state is None:
|
||||
task_state = _sample_task_reset_state(str(eval_cfg.task_name))
|
||||
elif isinstance(task_state, np.ndarray):
|
||||
task_state = np.asarray(task_state, dtype=np.float32)
|
||||
|
||||
task_state = _episode_plan_reset_state(plan, str(eval_cfg.task_name))
|
||||
|
||||
if show_progress:
|
||||
print(f"\n{'='*60}")
|
||||
@@ -1089,7 +1137,12 @@ def _run_eval_episode_plans(
|
||||
video_recorder.write(video_frame)
|
||||
|
||||
# 准备给 agent 的观测
|
||||
observation = prepare_observation(obs, camera_names)
|
||||
observation = prepare_observation(
|
||||
obs,
|
||||
camera_names,
|
||||
image_resize_shape=image_resize_shape,
|
||||
task_description=task_description,
|
||||
)
|
||||
end_preprocess = time.perf_counter()
|
||||
|
||||
# 选择动作（本地 agent 或远端 inference server）
|
||||
@@ -1497,7 +1550,9 @@ def _build_parallel_worker_payloads(
|
||||
num_episodes=int(eval_cfg.num_episodes),
|
||||
num_workers=requested_workers,
|
||||
)
|
||||
box_poses = _plan_episode_box_poses(int(eval_cfg.num_episodes))
|
||||
box_poses = None
|
||||
if 'sim_transfer' in str(eval_cfg.task_name):
|
||||
box_poses = _plan_episode_box_poses(int(eval_cfg.num_episodes))
|
||||
resolved_cfg = OmegaConf.to_container(cfg, resolve=True)
|
||||
payloads = []
|
||||
workers_dir = None
|
||||
@@ -1523,7 +1578,15 @@ def _build_parallel_worker_payloads(
|
||||
'episode_plans': [
|
||||
{
|
||||
'episode_index': int(episode_index),
|
||||
'box_pos': np.asarray(box_poses[episode_index], dtype=np.float32).tolist(),
|
||||
**(
|
||||
{
|
||||
'box_pos': _json_friendly(
|
||||
np.asarray(box_poses[episode_index], dtype=np.float32).copy()
|
||||
),
|
||||
}
|
||||
if box_poses is not None
|
||||
else {}
|
||||
),
|
||||
}
|
||||
for episode_index in episode_indices
|
||||
],
|
||||
|
||||
@@ -0,0 +1,474 @@
|
||||
"""Record realtime-ish front-view rollout videos for socket insertion checkpoints.
|
||||
|
||||
This script intentionally decouples video time from policy inference time. During
|
||||
an inference call the environment is not advanced; the current front frame is
|
||||
written repeatedly. Once the action is ready, exactly one environment action step
|
||||
is applied. The resulting video therefore makes slow inference visible as
|
||||
holds/pauses, matching the user's requested visualization.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
# Set EGL before importing eval_vla/mujoco helpers on headless hosts.
|
||||
os.environ.setdefault("MUJOCO_GL", "egl")
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from hydra import compose, initialize_config_dir
|
||||
from omegaconf import DictConfig, ListConfig, OmegaConf
|
||||
|
||||
# Make direct script execution work from the repository root.
|
||||
sys.path.append(os.getcwd())
|
||||
|
||||
from roboimi.demos.vla_scripts.eval_vla import ( # noqa: E402
|
||||
ActionSmoother,
|
||||
_close_env,
|
||||
_get_video_frame,
|
||||
_is_cuda_device,
|
||||
_open_video_writer,
|
||||
_resolve_policy_camera_names,
|
||||
_sample_task_reset_state,
|
||||
load_checkpoint,
|
||||
prepare_observation,
|
||||
)
|
||||
from roboimi.envs.double_pos_ctrl_env import make_sim_env # noqa: E402
|
||||
from roboimi.vla.eval_utils import execute_policy_action # noqa: E402
|
||||
|
||||
|
||||
def _jsonable(value: Any) -> Any:
|
||||
if isinstance(value, dict):
|
||||
return {str(k): _jsonable(v) for k, v in value.items()}
|
||||
if isinstance(value, (list, tuple)):
|
||||
return [_jsonable(v) for v in value]
|
||||
if isinstance(value, (np.integer, np.floating)):
|
||||
return value.item()
|
||||
if isinstance(value, np.ndarray):
|
||||
return value.tolist()
|
||||
if isinstance(value, Path):
|
||||
return str(value)
|
||||
return value
|
||||
|
||||
|
||||
def _parse_override_items(items: Optional[list[str]]) -> list[str]:
|
||||
return [str(item) for item in (items or []) if str(item)]
|
||||
|
||||
|
||||
def _load_cfg(agent_name: str, overrides: list[str]) -> DictConfig:
|
||||
config_dir = str((Path.cwd() / "roboimi" / "vla" / "conf").resolve())
|
||||
with initialize_config_dir(version_base=None, config_dir=config_dir):
|
||||
return compose(config_name="config", overrides=[f"agent={agent_name}", *overrides])
|
||||
|
||||
|
||||
def _listify(value: Any) -> list[str]:
|
||||
if isinstance(value, (list, tuple, ListConfig)):
|
||||
return [str(v) for v in value]
|
||||
if isinstance(value, str):
|
||||
# Be permissive for older SwanLab configs that persisted lists as strings.
|
||||
stripped = value.strip()
|
||||
if stripped.startswith("[") and stripped.endswith("]"):
|
||||
inner = stripped[1:-1].strip()
|
||||
if not inner:
|
||||
return []
|
||||
return [part.strip().strip("'\"") for part in inner.split(",")]
|
||||
return [value]
|
||||
return [str(value)]
|
||||
|
||||
|
||||
def _normalize_resize_shape(shape) -> Optional[tuple[int, int]]:
|
||||
if shape is None:
|
||||
return None
|
||||
normalized = tuple(int(v) for v in shape)
|
||||
if len(normalized) != 2:
|
||||
raise ValueError(f"image resize shape must contain exactly two values, got {normalized}")
|
||||
return normalized
|
||||
|
||||
|
||||
def _resolve_eval_image_resize_shape_local(cfg: DictConfig) -> Optional[tuple[int, int]]:
|
||||
image_resize_shape = cfg.get("data", {}).get("image_resize_shape", (224, 224))
|
||||
agent_cfg = cfg.agent
|
||||
for backbone_key in ("vision_backbone", "condition_encoder"):
|
||||
backbone_cfg = agent_cfg.get(backbone_key, None)
|
||||
if backbone_cfg is not None and "eval_image_resize_shape" in backbone_cfg:
|
||||
image_resize_shape = backbone_cfg.get("eval_image_resize_shape")
|
||||
break
|
||||
return _normalize_resize_shape(image_resize_shape)
|
||||
|
||||
|
||||
def _copy_success_artifacts(output_dir: Path, attempt_dir: Path) -> dict[str, str]:
|
||||
final_video = output_dir / "success_front.mp4"
|
||||
final_summary = output_dir / "success_summary.json"
|
||||
src_video = attempt_dir / "front.mp4"
|
||||
src_summary = attempt_dir / "summary.json"
|
||||
if src_video.exists():
|
||||
shutil.copy2(src_video, final_video)
|
||||
if src_summary.exists():
|
||||
shutil.copy2(src_summary, final_summary)
|
||||
return {
|
||||
"success_video": str(final_video),
|
||||
"success_summary": str(final_summary),
|
||||
}
|
||||
|
||||
|
||||
def _write_json(path: Path, payload: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("w", encoding="utf-8") as f:
|
||||
json.dump(_jsonable(payload), f, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
def _record_hold_frames(
|
||||
writer,
|
||||
frame: Optional[np.ndarray],
|
||||
*,
|
||||
elapsed_s: float,
|
||||
video_fps: float,
|
||||
min_frames: int = 1,
|
||||
) -> int:
|
||||
if frame is None:
|
||||
return 0
|
||||
# Round up so an inference wait of 80 ms at 30 FPS becomes 3 held frames.
|
||||
num_frames = max(int(min_frames), int(np.ceil(max(0.0, elapsed_s) * float(video_fps))))
|
||||
for _ in range(num_frames):
|
||||
writer.write(frame)
|
||||
return num_frames
|
||||
|
||||
|
||||
def _run_one_attempt(
|
||||
*,
|
||||
cfg: DictConfig,
|
||||
agent: torch.nn.Module,
|
||||
attempt_dir: Path,
|
||||
attempt_index: int,
|
||||
args: argparse.Namespace,
|
||||
) -> dict[str, Any]:
|
||||
eval_cfg = cfg.eval
|
||||
camera_names = _resolve_policy_camera_names(cfg)
|
||||
camera_names = _listify(camera_names)
|
||||
if "front" not in camera_names:
|
||||
raise ValueError(f"front camera is required for recording, got camera_names={camera_names}")
|
||||
|
||||
image_resize_shape = _resolve_eval_image_resize_shape_local(cfg)
|
||||
task_name = str(eval_cfg.task_name)
|
||||
task_state = _sample_task_reset_state(task_name)
|
||||
video_path = attempt_dir / "front.mp4"
|
||||
summary_path = attempt_dir / "summary.json"
|
||||
attempt_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
env = None
|
||||
writer = None
|
||||
smoother = ActionSmoother(alpha=float(eval_cfg.get("smooth_alpha", 0.3))) if bool(eval_cfg.get("use_smoothing", False)) else None
|
||||
|
||||
episode_reward = 0.0
|
||||
episode_max_reward = float("-inf")
|
||||
inference_times_ms: list[float] = []
|
||||
env_step_times_ms: list[float] = []
|
||||
obs_read_times_ms: list[float] = []
|
||||
preprocess_times_ms: list[float] = []
|
||||
recorded_frames = 0
|
||||
executed_steps = 0
|
||||
reward_trace: list[float] = []
|
||||
step_records: list[dict[str, Any]] = []
|
||||
|
||||
start_wall = time.perf_counter()
|
||||
try:
|
||||
env = make_sim_env(task_name, headless=True)
|
||||
# Optional runtime override. The factory defaults socket insertion to 30 Hz;
|
||||
# this script keeps that default unless the caller explicitly changes it.
|
||||
if args.control_freq is not None and callable(getattr(env, "_initializeTime", None)):
|
||||
env._initializeTime(float(args.control_freq))
|
||||
if hasattr(env, "base_time"):
|
||||
env.base_time = env.control_timestep
|
||||
|
||||
env.reset(task_state)
|
||||
if callable(getattr(agent, "reset", None)):
|
||||
agent.reset()
|
||||
if smoother is not None:
|
||||
smoother.reset()
|
||||
|
||||
obs = env._get_image_obs()
|
||||
qpos_obs = env._get_qpos_obs()
|
||||
obs["qpos"] = qpos_obs["qpos"]
|
||||
frame = _get_video_frame(obs, "front")
|
||||
if frame is None:
|
||||
raise RuntimeError("front frame is unavailable")
|
||||
writer = _open_video_writer(str(video_path), (int(frame.shape[1]), int(frame.shape[0])), int(args.video_fps))
|
||||
writer.write(frame)
|
||||
recorded_frames += 1
|
||||
|
||||
with torch.inference_mode():
|
||||
for t in range(int(args.max_timesteps)):
|
||||
iter_start = time.perf_counter()
|
||||
|
||||
obs_start = time.perf_counter()
|
||||
obs = env._get_image_obs()
|
||||
qpos_obs = env._get_qpos_obs()
|
||||
obs["qpos"] = qpos_obs["qpos"]
|
||||
obs_end = time.perf_counter()
|
||||
current_frame = _get_video_frame(obs, "front")
|
||||
|
||||
prep_start = time.perf_counter()
|
||||
observation = prepare_observation(
|
||||
obs,
|
||||
camera_names,
|
||||
image_resize_shape=image_resize_shape,
|
||||
)
|
||||
prep_end = time.perf_counter()
|
||||
|
||||
infer_start = time.perf_counter()
|
||||
action = agent.select_action(observation)
|
||||
if _is_cuda_device(str(eval_cfg.device)) and torch.cuda.is_available():
|
||||
torch.cuda.synchronize()
|
||||
infer_end = time.perf_counter()
|
||||
inference_s = infer_end - infer_start
|
||||
hold_frames = _record_hold_frames(
|
||||
writer,
|
||||
current_frame,
|
||||
elapsed_s=inference_s,
|
||||
video_fps=float(args.video_fps),
|
||||
min_frames=int(args.min_hold_frames),
|
||||
)
|
||||
recorded_frames += hold_frames
|
||||
|
||||
raw_action = action.detach().cpu().numpy().astype(np.float32, copy=True) if isinstance(action, torch.Tensor) else np.asarray(action, dtype=np.float32).copy()
|
||||
executed_action = raw_action.copy()
|
||||
if smoother is not None:
|
||||
executed_action = smoother.smooth(executed_action)
|
||||
|
||||
step_start = time.perf_counter()
|
||||
execute_policy_action(env, executed_action)
|
||||
step_end = time.perf_counter()
|
||||
executed_steps += 1
|
||||
|
||||
reward = getattr(env, "rew", None)
|
||||
reward_value = float(reward) if reward is not None else 0.0
|
||||
reward_trace.append(reward_value)
|
||||
episode_reward += reward_value
|
||||
episode_max_reward = max(episode_max_reward, reward_value)
|
||||
|
||||
obs_read_ms = (obs_end - obs_start) * 1000.0
|
||||
preprocess_ms = (prep_end - prep_start) * 1000.0
|
||||
inference_ms = inference_s * 1000.0
|
||||
env_step_ms = (step_end - step_start) * 1000.0
|
||||
obs_read_times_ms.append(obs_read_ms)
|
||||
preprocess_times_ms.append(preprocess_ms)
|
||||
inference_times_ms.append(inference_ms)
|
||||
env_step_times_ms.append(env_step_ms)
|
||||
step_records.append({
|
||||
"step": int(t),
|
||||
"reward": reward_value,
|
||||
"episode_max_reward_so_far": float(episode_max_reward),
|
||||
"obs_read_time_ms": obs_read_ms,
|
||||
"preprocess_time_ms": preprocess_ms,
|
||||
"inference_time_ms": inference_ms,
|
||||
"env_step_time_ms": env_step_ms,
|
||||
"held_video_frames": int(hold_frames),
|
||||
"raw_action_min": float(raw_action.min()),
|
||||
"raw_action_max": float(raw_action.max()),
|
||||
"raw_action_mean": float(raw_action.mean()),
|
||||
})
|
||||
|
||||
if args.verbose_every > 0 and (t % int(args.verbose_every) == 0 or episode_max_reward > args.success_threshold):
|
||||
print(
|
||||
f"attempt={attempt_index} step={t} reward={reward_value:.1f} "
|
||||
f"max={episode_max_reward:.1f} infer={inference_ms:.1f}ms "
|
||||
f"hold_frames={hold_frames} video_frames={recorded_frames}",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
if bool(args.stop_on_success) and episode_max_reward > float(args.success_threshold):
|
||||
# Add a short tail so the successful state remains visible.
|
||||
final_obs = env._get_image_obs()
|
||||
final_frame = _get_video_frame(final_obs, "front")
|
||||
tail_frames = int(round(float(args.success_tail_s) * float(args.video_fps)))
|
||||
for _ in range(max(0, tail_frames)):
|
||||
writer.write(final_frame)
|
||||
recorded_frames += max(0, tail_frames)
|
||||
break
|
||||
|
||||
# Safety: keep files bounded if a very slow model produces many duplicate frames.
|
||||
if args.max_video_frames is not None and recorded_frames >= int(args.max_video_frames):
|
||||
print(
|
||||
f"attempt={attempt_index} reached max_video_frames={args.max_video_frames}; stopping attempt",
|
||||
flush=True,
|
||||
)
|
||||
break
|
||||
|
||||
wall_s = time.perf_counter() - start_wall
|
||||
success = bool(episode_max_reward > float(args.success_threshold))
|
||||
summary = {
|
||||
"attempt_index": int(attempt_index),
|
||||
"success": success,
|
||||
"checkpoint": str(args.ckpt_path),
|
||||
"agent": str(args.agent),
|
||||
"task_name": task_name,
|
||||
"task_state": task_state,
|
||||
"video_path": str(video_path),
|
||||
"video_camera": "front",
|
||||
"video_fps": float(args.video_fps),
|
||||
"recording_semantics": "Video duplicates the last observed front frame while inference runs; env.step is not called, so the simulation state is not advanced or modified during inference waits.",
|
||||
"control_freq_hz": float(getattr(env, "control_freq", np.nan)),
|
||||
"mujoco_model_timestep_s": float(getattr(env, "model_timestep", np.nan)),
|
||||
"max_theoretical_physics_hz": float(1.0 / getattr(env, "model_timestep", np.nan)),
|
||||
"executed_steps": int(executed_steps),
|
||||
"recorded_frames": int(recorded_frames),
|
||||
"video_duration_s": float(recorded_frames) / float(args.video_fps),
|
||||
"wall_time_s": wall_s,
|
||||
"episode_reward": float(episode_reward),
|
||||
"episode_max_reward": float(episode_max_reward) if episode_max_reward != float("-inf") else None,
|
||||
"avg_inference_time_ms": float(np.mean(inference_times_ms)) if inference_times_ms else 0.0,
|
||||
"avg_inference_fps": float(1000.0 / np.mean(inference_times_ms)) if inference_times_ms and np.mean(inference_times_ms) > 0 else 0.0,
|
||||
"avg_env_step_time_ms": float(np.mean(env_step_times_ms)) if env_step_times_ms else 0.0,
|
||||
"avg_obs_read_time_ms": float(np.mean(obs_read_times_ms)) if obs_read_times_ms else 0.0,
|
||||
"avg_preprocess_time_ms": float(np.mean(preprocess_times_ms)) if preprocess_times_ms else 0.0,
|
||||
"inference_time_ms_min": float(np.min(inference_times_ms)) if inference_times_ms else 0.0,
|
||||
"inference_time_ms_max": float(np.max(inference_times_ms)) if inference_times_ms else 0.0,
|
||||
"reward_trace": reward_trace,
|
||||
"steps": step_records,
|
||||
}
|
||||
_write_json(summary_path, summary)
|
||||
print(
|
||||
f"attempt={attempt_index} done success={success} reward={episode_reward:.1f} "
|
||||
f"max_reward={summary['episode_max_reward']} video={video_path}",
|
||||
flush=True,
|
||||
)
|
||||
return summary
|
||||
finally:
|
||||
if writer is not None:
|
||||
writer.release()
|
||||
_close_env(env)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--ckpt-path", required=True)
|
||||
parser.add_argument("--agent", required=True, help="Hydra agent config name, e.g. resnet_imf_attnres or resnet_transformer")
|
||||
parser.add_argument("--output-dir", required=True)
|
||||
parser.add_argument("--task-name", default="sim_air_insert_socket_peg")
|
||||
parser.add_argument("--dataset-dir", default=None)
|
||||
parser.add_argument("--camera-names", default=None, help="Comma-separated policy camera names. Defaults depend on task.")
|
||||
parser.add_argument("--max-attempts", type=int, default=80)
|
||||
parser.add_argument("--max-timesteps", type=int, default=750)
|
||||
parser.add_argument("--success-threshold", type=float, default=4.0)
|
||||
parser.add_argument("--video-fps", type=float, default=30.0)
|
||||
parser.add_argument("--min-hold-frames", type=int, default=1)
|
||||
parser.add_argument("--success-tail-s", type=float, default=2.0)
|
||||
parser.add_argument("--max-video-frames", type=int, default=None)
|
||||
parser.add_argument("--control-freq", type=float, default=None, help="Optional env control frequency override. Default uses make_sim_env default (30 Hz for socket).")
|
||||
parser.add_argument("--device", default="cuda")
|
||||
parser.add_argument("--seed", type=int, default=42)
|
||||
parser.add_argument("--stop-on-success", action=argparse.BooleanOptionalAction, default=True)
|
||||
parser.add_argument("--verbose-every", type=int, default=50)
|
||||
parser.add_argument("overrides", nargs="*", help="Additional Hydra overrides, e.g. agent.inference_steps=3")
|
||||
args = parser.parse_args()
|
||||
|
||||
np.random.seed(int(args.seed))
|
||||
torch.manual_seed(int(args.seed))
|
||||
if torch.cuda.is_available():
|
||||
torch.cuda.manual_seed_all(int(args.seed))
|
||||
|
||||
output_dir = Path(args.output_dir).expanduser().resolve()
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
default_dataset_dirs = {
|
||||
"sim_air_insert_socket_peg": "/data/roboimi_datasets/sim_air_insert_socket_peg",
|
||||
"sim_transfer": "/home/droid/project/diana_sim/sim_transfer",
|
||||
}
|
||||
default_camera_names = {
|
||||
"sim_air_insert_socket_peg": ["l_vis", "r_vis", "front"],
|
||||
"sim_transfer": ["r_vis", "top", "front"],
|
||||
}
|
||||
task_name = str(args.task_name)
|
||||
dataset_dir = str(args.dataset_dir or default_dataset_dirs.get(task_name, ""))
|
||||
if not dataset_dir:
|
||||
raise ValueError(f"--dataset-dir is required for task {task_name!r}")
|
||||
if args.camera_names is None:
|
||||
camera_names = default_camera_names.get(task_name)
|
||||
else:
|
||||
camera_names = [item.strip() for item in str(args.camera_names).split(",") if item.strip()]
|
||||
if not camera_names:
|
||||
raise ValueError(f"--camera-names is required for task {task_name!r}")
|
||||
camera_names_override = "[" + ",".join(camera_names) + "]"
|
||||
|
||||
overrides = [
|
||||
f"data.dataset_dir={dataset_dir}",
|
||||
f"data.camera_names={camera_names_override}",
|
||||
f"eval.camera_names={camera_names_override}",
|
||||
f"eval.task_name={task_name}",
|
||||
f"eval.max_timesteps={int(args.max_timesteps)}",
|
||||
f"eval.device={args.device}",
|
||||
"eval.headless=true",
|
||||
"eval.verbose_action=false",
|
||||
*_parse_override_items(args.overrides),
|
||||
]
|
||||
cfg = _load_cfg(args.agent, overrides)
|
||||
cfg.eval.ckpt_path = str(Path(args.ckpt_path).expanduser().resolve())
|
||||
cfg.eval.device = str(args.device)
|
||||
cfg.eval.headless = True
|
||||
cfg.eval.task_name = task_name
|
||||
cfg.eval.max_timesteps = int(args.max_timesteps)
|
||||
if cfg.get("train") is not None:
|
||||
cfg.train.device = str(args.device)
|
||||
|
||||
_write_json(output_dir / "resolved_config.json", OmegaConf.to_container(cfg, resolve=True))
|
||||
print("=" * 80)
|
||||
print("Realtime front rollout config")
|
||||
print(OmegaConf.to_yaml(cfg))
|
||||
print("=" * 80, flush=True)
|
||||
|
||||
agent, _stats = load_checkpoint(str(cfg.eval.ckpt_path), cfg.agent, device=str(args.device))
|
||||
|
||||
all_summaries: list[dict[str, Any]] = []
|
||||
for attempt_index in range(1, int(args.max_attempts) + 1):
|
||||
attempt_dir = output_dir / f"attempt_{attempt_index:03d}"
|
||||
summary = _run_one_attempt(
|
||||
cfg=cfg,
|
||||
agent=agent,
|
||||
attempt_dir=attempt_dir,
|
||||
attempt_index=attempt_index,
|
||||
args=args,
|
||||
)
|
||||
all_summaries.append(summary)
|
||||
_write_json(output_dir / "attempts_summary.json", {"attempts": all_summaries})
|
||||
if summary.get("success"):
|
||||
final_paths = _copy_success_artifacts(output_dir, attempt_dir)
|
||||
final_summary = {
|
||||
"success": True,
|
||||
"winning_attempt": int(attempt_index),
|
||||
"final_paths": final_paths,
|
||||
"summary": summary,
|
||||
}
|
||||
_write_json(output_dir / "success_result.json", final_summary)
|
||||
print(json.dumps(_jsonable(final_summary), ensure_ascii=False, indent=2), flush=True)
|
||||
return 0
|
||||
|
||||
failure = {
|
||||
"success": False,
|
||||
"max_attempts": int(args.max_attempts),
|
||||
"best_episode_max_reward": max(
|
||||
[
|
||||
(
|
||||
s.get("episode_max_reward")
|
||||
if s.get("episode_max_reward") is not None
|
||||
else float("-inf")
|
||||
)
|
||||
for s in all_summaries
|
||||
],
|
||||
default=None,
|
||||
),
|
||||
"attempts": all_summaries,
|
||||
}
|
||||
_write_json(output_dir / "failure_result.json", failure)
|
||||
print(json.dumps(_jsonable(failure), ensure_ascii=False, indent=2), flush=True)
|
||||
return 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -118,6 +118,109 @@ def recursive_to_device(data, device):
|
||||
return data
|
||||
|
||||
|
||||
def build_agent_input(batch_data):
|
||||
images = {}
|
||||
for key, value in batch_data.items():
|
||||
if key.startswith('observation.') and key != 'observation.state':
|
||||
images[key.replace('observation.', '')] = value
|
||||
|
||||
agent_input = {
|
||||
'images': images,
|
||||
'qpos': batch_data['observation.state'],
|
||||
'action': batch_data['action'],
|
||||
}
|
||||
if 'action_is_pad' in batch_data:
|
||||
agent_input['action_is_pad'] = batch_data['action_is_pad']
|
||||
if 'task' in batch_data:
|
||||
agent_input['task'] = batch_data['task']
|
||||
return agent_input
|
||||
|
||||
|
||||
def _instantiate_dataset(cfg, dataset_image_resize_shape, episode_indices=None):
|
||||
kwargs = {'image_resize_shape': dataset_image_resize_shape}
|
||||
if episode_indices is not None:
|
||||
kwargs['episode_indices'] = episode_indices
|
||||
return instantiate(cfg.data, **kwargs)
|
||||
|
||||
|
||||
def build_train_val_datasets(cfg, dataset_image_resize_shape):
|
||||
val_episode_indices = cfg.train.get('val_episode_indices', None)
|
||||
if val_episode_indices:
|
||||
dataset = _instantiate_dataset(cfg, dataset_image_resize_shape)
|
||||
available_episode_indices = list(getattr(dataset, 'available_episode_indices', []))
|
||||
if not available_episode_indices:
|
||||
raise ValueError('显式 val_episode_indices 需要数据集暴露 available_episode_indices')
|
||||
requested_val_episode_indices = sorted(int(idx) for idx in val_episode_indices)
|
||||
available_set = set(available_episode_indices)
|
||||
missing = sorted(set(requested_val_episode_indices) - available_set)
|
||||
if missing:
|
||||
raise ValueError(
|
||||
f'val_episode_indices {missing} 不存在于数据集可用 episodes {available_episode_indices}'
|
||||
)
|
||||
val_set = set(requested_val_episode_indices)
|
||||
train_episode_indices = [
|
||||
idx for idx in available_episode_indices
|
||||
if idx not in val_set
|
||||
]
|
||||
if not train_episode_indices:
|
||||
raise ValueError('显式 val_episode_indices 不能覆盖全部 episodes，训练集将为空')
|
||||
|
||||
train_dataset = _instantiate_dataset(
|
||||
cfg,
|
||||
dataset_image_resize_shape,
|
||||
episode_indices=train_episode_indices,
|
||||
)
|
||||
val_dataset = _instantiate_dataset(
|
||||
cfg,
|
||||
dataset_image_resize_shape,
|
||||
episode_indices=requested_val_episode_indices,
|
||||
)
|
||||
return dataset, train_dataset, val_dataset, requested_val_episode_indices
|
||||
|
||||
dataset = _instantiate_dataset(cfg, dataset_image_resize_shape)
|
||||
val_split = float(cfg.train.get('val_split', 0.1))
|
||||
seed = int(cfg.train.get('seed', 42))
|
||||
val_size = int(len(dataset) * val_split)
|
||||
train_size = len(dataset) - val_size
|
||||
if val_size > 0:
|
||||
train_dataset, val_dataset = random_split(
|
||||
dataset,
|
||||
[train_size, val_size],
|
||||
generator=torch.Generator().manual_seed(seed)
|
||||
)
|
||||
else:
|
||||
train_dataset, val_dataset = dataset, None
|
||||
return dataset, train_dataset, val_dataset, None
|
||||
|
||||
|
||||
def compute_action_mse_validation(agent, val_loader, device):
|
||||
if val_loader is None:
|
||||
return None
|
||||
|
||||
was_training = agent.training
|
||||
agent.eval()
|
||||
total_squared_error = 0.0
|
||||
total_count = 0.0
|
||||
with torch.no_grad():
|
||||
for val_batch in val_loader:
|
||||
val_batch = recursive_to_device(val_batch, device)
|
||||
val_input = build_agent_input(val_batch)
|
||||
pred_actions = agent.predict_action_chunk(val_input)
|
||||
target_actions = val_input['action']
|
||||
squared_error = (pred_actions - target_actions).pow(2)
|
||||
action_is_pad = val_input.get('action_is_pad', None)
|
||||
if action_is_pad is not None:
|
||||
mask = (~action_is_pad).unsqueeze(-1).to(squared_error.dtype)
|
||||
total_squared_error += (squared_error * mask).sum().item()
|
||||
total_count += mask.sum().item() * squared_error.shape[-1]
|
||||
else:
|
||||
total_squared_error += squared_error.sum().item()
|
||||
total_count += target_actions.numel()
|
||||
if was_training:
|
||||
agent.train()
|
||||
return total_squared_error / max(total_count, 1.0)
|
||||
|
||||
|
||||
def resolve_resume_checkpoint(resume_ckpt, checkpoint_dir):
|
||||
"""
|
||||
解析恢复训练用的 checkpoint 路径。
|
||||
@@ -369,6 +472,11 @@ def _run_training(cfg: DictConfig):
|
||||
_configure_cuda_runtime(cfg)
|
||||
swanlab_module = _init_swanlab(cfg)
|
||||
try:
|
||||
action_mse_val_freq_epochs = int(cfg.train.get('action_mse_val_freq_epochs', 0) or 0)
|
||||
explicit_val_episode_indices = cfg.train.get('val_episode_indices', None)
|
||||
if action_mse_val_freq_epochs > 0 and not explicit_val_episode_indices:
|
||||
raise ValueError('action_mse_val_freq_epochs > 0 requires train.val_episode_indices')
|
||||
|
||||
# 创建检查点目录
|
||||
run_output_dir = _resolve_run_output_dir()
|
||||
checkpoint_dir = run_output_dir / "checkpoints"
|
||||
@@ -381,33 +489,35 @@ def _run_training(cfg: DictConfig):
|
||||
log.info("📦 加载数据集...")
|
||||
try:
|
||||
dataset_image_resize_shape = cfg.data.get('image_resize_shape', (224, 224))
|
||||
vision_backbone_cfg = cfg.agent.get('vision_backbone', None)
|
||||
if vision_backbone_cfg is not None and 'dataset_image_resize_shape' in vision_backbone_cfg:
|
||||
dataset_image_resize_shape = vision_backbone_cfg.get('dataset_image_resize_shape')
|
||||
dataset = instantiate(
|
||||
cfg.data,
|
||||
image_resize_shape=dataset_image_resize_shape,
|
||||
for backbone_key in ('vision_backbone', 'condition_encoder'):
|
||||
backbone_cfg = cfg.agent.get(backbone_key, None)
|
||||
if backbone_cfg is not None and 'dataset_image_resize_shape' in backbone_cfg:
|
||||
dataset_image_resize_shape = backbone_cfg.get('dataset_image_resize_shape')
|
||||
break
|
||||
dataset, train_dataset, val_dataset, explicit_val_episode_indices = (
|
||||
build_train_val_datasets(cfg, dataset_image_resize_shape)
|
||||
)
|
||||
log.info(f"✅ 数据集加载成功。总样本数: {len(dataset)}")
|
||||
except Exception as e:
|
||||
log.error(f"❌ 数据集加载失败: {e}")
|
||||
raise
|
||||
|
||||
# 训练/验证集划分
|
||||
val_split = float(cfg.train.get('val_split', 0.1))
|
||||
seed = int(cfg.train.get('seed', 42))
|
||||
val_size = int(len(dataset) * val_split)
|
||||
train_size = len(dataset) - val_size
|
||||
if val_size > 0:
|
||||
train_dataset, val_dataset = random_split(
|
||||
dataset,
|
||||
[train_size, val_size],
|
||||
generator=torch.Generator().manual_seed(seed)
|
||||
if explicit_val_episode_indices is not None:
|
||||
log.info(
|
||||
"✅ 数据集划分: 训练集=%s, 验证集=%s (显式 held-out episodes=%s)",
|
||||
len(train_dataset),
|
||||
len(val_dataset),
|
||||
explicit_val_episode_indices,
|
||||
)
|
||||
log.info(f"✅ 数据集划分: 训练集={train_size}, 验证集={val_size} (验证比例={val_split})")
|
||||
else:
|
||||
train_dataset, val_dataset = dataset, None
|
||||
log.info("✅ 数据集划分: 全部用于训练, 验证集=0 (验证比例=0)")
|
||||
val_split = float(cfg.train.get('val_split', 0.1))
|
||||
val_size = len(val_dataset) if val_dataset is not None else 0
|
||||
if val_size > 0:
|
||||
log.info(
|
||||
f"✅ 数据集划分: 训练集={len(train_dataset)}, 验证集={val_size} (验证比例={val_split})"
|
||||
)
|
||||
else:
|
||||
log.info("✅ 数据集划分: 全部用于训练, 验证集=0 (验证比例=0)")
|
||||
|
||||
train_batch_size = int(cfg.train.batch_size)
|
||||
train_drop_last = len(train_dataset) >= train_batch_size
|
||||
@@ -643,22 +753,6 @@ def _run_training(cfg: DictConfig):
|
||||
# =========================================================================
|
||||
log.info("🏋️ 开始训练循环...")
|
||||
|
||||
def build_agent_input(batch_data):
|
||||
"""构建 agent 输入格式"""
|
||||
images = {}
|
||||
# SimpleRobotDataset 返回 observation.{cam_name} 格式
|
||||
for cam_name in cfg.data.camera_names:
|
||||
key = f"observation.{cam_name}"
|
||||
if key in batch_data:
|
||||
images[cam_name] = batch_data[key]
|
||||
|
||||
return {
|
||||
'images': images,
|
||||
'qpos': batch_data['observation.state'], # SimpleRobotDataset 使用 observation.state
|
||||
'action': batch_data['action'],
|
||||
'action_is_pad': batch_data.get('action_is_pad', None) # 传递padding mask
|
||||
}
|
||||
|
||||
def save_checkpoint(checkpoint_path: Path, step: int, loss_value, val_loss=None, rollout_avg_reward=None):
|
||||
agent_stats = agent.get_normalization_stats()
|
||||
torch.save({
|
||||
@@ -753,6 +847,9 @@ def _run_training(cfg: DictConfig):
|
||||
steps_per_epoch = len(train_loader)
|
||||
rollout_val_freq_epochs = int(cfg.train.get('rollout_val_freq_epochs', 0) or 0)
|
||||
rollout_validation_enabled = rollout_val_freq_epochs > 0
|
||||
action_mse_validation_enabled = (
|
||||
explicit_val_episode_indices is not None and action_mse_val_freq_epochs > 0
|
||||
)
|
||||
best_loss = resume_best_loss
|
||||
best_rollout_reward = resume_best_rollout_reward
|
||||
last_loss = resume_loss
|
||||
@@ -911,6 +1008,26 @@ def _run_training(cfg: DictConfig):
|
||||
and completed_epoch > 0
|
||||
and completed_epoch % rollout_val_freq_epochs == 0
|
||||
)
|
||||
should_run_action_mse_validation = (
|
||||
action_mse_validation_enabled
|
||||
and steps_per_epoch > 0
|
||||
and completed_steps % steps_per_epoch == 0
|
||||
and completed_epoch > 0
|
||||
and completed_epoch % action_mse_val_freq_epochs == 0
|
||||
)
|
||||
if should_run_action_mse_validation:
|
||||
action_mse = compute_action_mse_validation(agent, val_loader, cfg.train.device)
|
||||
if action_mse is not None:
|
||||
log.info(
|
||||
f"步骤 {step}/{cfg.train.max_steps} | Epoch {completed_epoch} held-out action MSE: "
|
||||
f"{action_mse:.6f}"
|
||||
)
|
||||
_log_to_swanlab(
|
||||
swanlab_module,
|
||||
{'val/action_mse': action_mse},
|
||||
step=step,
|
||||
)
|
||||
|
||||
if should_run_epoch_rollout:
|
||||
if checkpoint_path is None:
|
||||
checkpoint_path = checkpoint_dir / f"vla_model_step_{step}.pt"
|
||||
|
||||
@@ -15,14 +15,46 @@ except ImportError: # pragma: no cover
|
||||
|
||||
|
||||
class IMFVLAAgent(VLAAgent):
|
||||
def __init__(self, *args, inference_steps: int = 1, **kwargs):
|
||||
if inference_steps != 1:
|
||||
def __init__(
|
||||
self,
|
||||
*args,
|
||||
inference_steps: int = 1,
|
||||
loss_type: str = 'pseudo_huber',
|
||||
pseudo_huber_delta: float = 1.0,
|
||||
**kwargs,
|
||||
):
|
||||
inference_steps = int(inference_steps)
|
||||
if inference_steps < 1:
|
||||
raise ValueError(
|
||||
'IMFVLAAgent only supports one-step inference; '
|
||||
f'inference_steps must be 1, got {inference_steps}.'
|
||||
'IMFVLAAgent inference_steps must be >= 1, '
|
||||
f'got {inference_steps}.'
|
||||
)
|
||||
self._validate_loss_config(loss_type=loss_type, pseudo_huber_delta=pseudo_huber_delta)
|
||||
super().__init__(*args, inference_steps=inference_steps, **kwargs)
|
||||
self.inference_steps = 1
|
||||
self.inference_steps = inference_steps
|
||||
self.loss_type = str(loss_type)
|
||||
self.pseudo_huber_delta = float(pseudo_huber_delta)
|
||||
|
||||
@staticmethod
|
||||
def _validate_loss_config(loss_type: str, pseudo_huber_delta: float) -> None:
|
||||
if loss_type not in {'pseudo_huber', 'mse'}:
|
||||
raise ValueError(
|
||||
"IMFVLAAgent loss_type must be one of {'pseudo_huber', 'mse'}, "
|
||||
f'got {loss_type!r}.'
|
||||
)
|
||||
if float(pseudo_huber_delta) <= 0.0:
|
||||
raise ValueError(
|
||||
'IMFVLAAgent pseudo_huber_delta must be > 0, '
|
||||
f'got {pseudo_huber_delta}.'
|
||||
)
|
||||
|
||||
def _velocity_loss_from_error(self, error: torch.Tensor) -> torch.Tensor:
|
||||
if self.loss_type == 'mse':
|
||||
return error.square()
|
||||
if self.loss_type != 'pseudo_huber':
|
||||
raise ValueError(f'Unsupported IMFVLAAgent loss_type: {self.loss_type!r}.')
|
||||
delta = float(self.pseudo_huber_delta)
|
||||
return delta**2 * (torch.sqrt(1.0 + (error / delta).square()) - 1.0)
|
||||
|
||||
@staticmethod
|
||||
def _broadcast_batch_time(value: torch.Tensor, reference: torch.Tensor) -> torch.Tensor:
|
||||
@@ -142,7 +174,7 @@ class IMFVLAAgent(VLAAgent):
|
||||
V = self._compound_velocity(u, du_dt, r, t)
|
||||
target = e - x
|
||||
|
||||
loss = F.mse_loss(V, target, reduction='none')
|
||||
loss = self._velocity_loss_from_error(V - target)
|
||||
if action_is_pad is not None:
|
||||
mask = (~action_is_pad).unsqueeze(-1).to(loss.dtype)
|
||||
valid_count = mask.sum() * loss.shape[-1]
|
||||
@@ -157,5 +189,26 @@ class IMFVLAAgent(VLAAgent):
|
||||
proprioception = self.normalization.normalize_qpos(proprioception)
|
||||
cond = self._build_cond(images, proprioception)
|
||||
z_t = torch.randn((batch_size, self.pred_horizon, self.action_dim), device=cond.device, dtype=cond.dtype)
|
||||
action = self._sample_one_step(z_t, cond=cond)
|
||||
action = z_t
|
||||
time_grid = torch.linspace(
|
||||
1.0,
|
||||
0.0,
|
||||
steps=self.inference_steps + 1,
|
||||
device=cond.device,
|
||||
dtype=cond.dtype,
|
||||
)
|
||||
for step_index in range(self.inference_steps):
|
||||
t = torch.full(
|
||||
(batch_size,),
|
||||
float(time_grid[step_index].item()),
|
||||
device=cond.device,
|
||||
dtype=cond.dtype,
|
||||
)
|
||||
r = torch.full(
|
||||
(batch_size,),
|
||||
float(time_grid[step_index + 1].item()),
|
||||
device=cond.device,
|
||||
dtype=cond.dtype,
|
||||
)
|
||||
action = self._sample_one_step(action, r=r, t=t, cond=cond)
|
||||
return self.normalization.denormalize_action(action)
|
||||
|
||||
@@ -0,0 +1,282 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import deque
|
||||
from typing import Dict, Optional, Sequence
|
||||
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
|
||||
from roboimi.vla.agent_imf import IMFVLAAgent
|
||||
from roboimi.vla.models.normalization import NormalizationModule
|
||||
|
||||
|
||||
class SmolVLAIMFAttnResAgent(IMFVLAAgent):
|
||||
"""IMF-AttnRes action expert conditioned by SmolVLA-style VLM prefix tokens.
|
||||
|
||||
Unlike ``VLAAgent`` this agent does not concatenate ResNet features and raw
|
||||
state at every observation step. A ``condition_encoder`` receives images,
|
||||
normalized state, and variable task language, and returns a condition token
|
||||
sequence `(B, S, D)` that is passed directly to the IMF head.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
condition_encoder: nn.Module,
|
||||
action_encoder,
|
||||
head,
|
||||
action_dim: int,
|
||||
obs_dim: int,
|
||||
pred_horizon: int = 16,
|
||||
obs_horizon: int = 2,
|
||||
diffusion_steps: int = 100,
|
||||
inference_steps: int = 1,
|
||||
num_cams: int = 3,
|
||||
camera_names: Optional[Sequence[str]] = None,
|
||||
dataset_stats=None,
|
||||
normalization_type: str = 'min_max',
|
||||
num_action_steps: int = 8,
|
||||
head_type: str = 'transformer',
|
||||
condition_dim: int | None = None,
|
||||
condition_sequence_length: int | None = None,
|
||||
task_description: str | None = None,
|
||||
loss_type: str = 'pseudo_huber',
|
||||
pseudo_huber_delta: float = 1.0,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
# Intentionally bypass VLAAgent.__init__; its condition dimensions are
|
||||
# tied to ResNet-style visual+state concatenation.
|
||||
nn.Module.__init__(self)
|
||||
if inference_steps != 1:
|
||||
raise ValueError(
|
||||
'SmolVLAIMFAttnResAgent only supports one-step IMF inference; '
|
||||
f'inference_steps must be 1, got {inference_steps}.'
|
||||
)
|
||||
if head_type != 'transformer':
|
||||
raise ValueError(f'SmolVLAIMFAttnResAgent requires head_type="transformer", got {head_type!r}')
|
||||
self._validate_loss_config(loss_type=loss_type, pseudo_huber_delta=pseudo_huber_delta)
|
||||
del diffusion_steps, kwargs
|
||||
|
||||
self.action_dim = int(action_dim)
|
||||
self.obs_dim = int(obs_dim)
|
||||
self.pred_horizon = int(pred_horizon)
|
||||
self.obs_horizon = int(obs_horizon)
|
||||
self.num_cams = int(num_cams)
|
||||
self.num_action_steps = int(num_action_steps)
|
||||
self.inference_steps = 1
|
||||
self.head_type = head_type
|
||||
self.loss_type = str(loss_type)
|
||||
self.pseudo_huber_delta = float(pseudo_huber_delta)
|
||||
self.camera_names = tuple(camera_names) if camera_names is not None else None
|
||||
if self.camera_names is not None and len(self.camera_names) != self.num_cams:
|
||||
raise ValueError(f'camera_names length({len(self.camera_names)}) does not match num_cams({self.num_cams})')
|
||||
|
||||
self.normalization = NormalizationModule(stats=dataset_stats, normalization_type=normalization_type)
|
||||
self.condition_encoder = condition_encoder
|
||||
# Compatibility aliases used by some utilities/tests.
|
||||
self.vision_encoder = condition_encoder
|
||||
self.action_encoder = action_encoder
|
||||
self.state_encoder = None
|
||||
self.task_description = task_description
|
||||
|
||||
encoder_dim = getattr(condition_encoder, 'output_dim', None)
|
||||
if encoder_dim is None:
|
||||
encoder_dim = getattr(condition_encoder, 'joint_output_dim', None)
|
||||
if condition_dim is None:
|
||||
if encoder_dim is None:
|
||||
raise ValueError('condition_dim must be provided when condition_encoder has no output_dim')
|
||||
condition_dim = int(encoder_dim)
|
||||
self.per_step_cond_dim = int(condition_dim)
|
||||
self.raw_per_step_cond_dim = self.per_step_cond_dim
|
||||
|
||||
encoder_seq_len = getattr(condition_encoder, 'condition_sequence_length', None)
|
||||
if condition_sequence_length is None:
|
||||
if encoder_seq_len is None:
|
||||
raise ValueError(
|
||||
'condition_sequence_length must be provided when condition_encoder has no condition_sequence_length'
|
||||
)
|
||||
condition_sequence_length = int(encoder_seq_len)
|
||||
self.condition_sequence_length = int(condition_sequence_length)
|
||||
self.condition_tokens_per_step = self.condition_sequence_length
|
||||
self.global_cond_dim = self.per_step_cond_dim * self.condition_sequence_length
|
||||
|
||||
if isinstance(head, nn.Module):
|
||||
self.noise_pred_net = head
|
||||
else:
|
||||
self.noise_pred_net = head(
|
||||
input_dim=self.action_dim,
|
||||
output_dim=self.action_dim,
|
||||
horizon=self.pred_horizon,
|
||||
n_obs_steps=self.condition_sequence_length,
|
||||
cond_dim=self.per_step_cond_dim,
|
||||
)
|
||||
self.reset()
|
||||
|
||||
def _get_model_device(self) -> torch.device:
|
||||
return next(self.parameters()).device
|
||||
|
||||
def _move_to_device(self, data, device: torch.device):
|
||||
if torch.is_tensor(data):
|
||||
return data.to(device)
|
||||
if isinstance(data, dict):
|
||||
return {k: self._move_to_device(v, device) for k, v in data.items()}
|
||||
if isinstance(data, list):
|
||||
return [self._move_to_device(v, device) for v in data]
|
||||
if isinstance(data, tuple):
|
||||
return tuple(self._move_to_device(v, device) for v in data)
|
||||
return data
|
||||
|
||||
def _order_images(self, images: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:
|
||||
if self.camera_names is None:
|
||||
names = tuple(sorted(images.keys()))
|
||||
if len(names) != self.num_cams:
|
||||
raise ValueError(f'image camera count({len(names)}) does not match num_cams({self.num_cams})')
|
||||
return {name: images[name] for name in names}
|
||||
missing = [name for name in self.camera_names if name not in images]
|
||||
if missing:
|
||||
raise ValueError(f'image condition missing required cameras. missing={missing}, expected={list(self.camera_names)}')
|
||||
return {name: images[name] for name in self.camera_names}
|
||||
|
||||
def _resolve_task(self, task, batch_size: int):
|
||||
def is_missing_task(item) -> bool:
|
||||
return item is None or (isinstance(item, str) and item.strip().lower() in {'', 'unknown'})
|
||||
|
||||
def fallback_or(item):
|
||||
if self.task_description is not None and is_missing_task(item):
|
||||
return self.task_description
|
||||
if item is None:
|
||||
return ''
|
||||
return item
|
||||
|
||||
if task is None:
|
||||
task = self.task_description
|
||||
if task is None:
|
||||
return [''] * batch_size
|
||||
if isinstance(task, str):
|
||||
return [fallback_or(task)] * batch_size
|
||||
task = list(task)
|
||||
if len(task) != batch_size:
|
||||
raise ValueError(f'task batch size ({len(task)}) must match batch size ({batch_size})')
|
||||
return [fallback_or(item) for item in task]
|
||||
|
||||
def _build_cond(self, images: Dict[str, torch.Tensor], states: torch.Tensor, task=None) -> torch.Tensor:
|
||||
ordered_images = self._order_images(images)
|
||||
batch_size = states.shape[0]
|
||||
tasks = self._resolve_task(task, batch_size=batch_size)
|
||||
cond = self.condition_encoder(ordered_images, state=states, task=tasks)
|
||||
if cond.ndim != 3:
|
||||
raise RuntimeError(f'condition_encoder must return (B,S,D), got {tuple(cond.shape)}')
|
||||
if cond.shape[0] != batch_size:
|
||||
raise RuntimeError(f'condition batch mismatch: got {cond.shape[0]}, expected {batch_size}')
|
||||
if cond.shape[1] != self.condition_sequence_length:
|
||||
raise RuntimeError(
|
||||
f'condition sequence length mismatch: got {cond.shape[1]}, expected {self.condition_sequence_length}'
|
||||
)
|
||||
if cond.shape[2] != self.per_step_cond_dim:
|
||||
raise RuntimeError(f'condition dim mismatch: got {cond.shape[2]}, expected {self.per_step_cond_dim}')
|
||||
head_dtype = next(self.noise_pred_net.parameters()).dtype
|
||||
return cond.to(dtype=head_dtype)
|
||||
|
||||
def compute_loss(self, batch):
|
||||
actions, states, images = batch['action'], batch['qpos'], batch['images']
|
||||
action_is_pad = batch.get('action_is_pad', None)
|
||||
batch_size = actions.shape[0]
|
||||
|
||||
states = self.normalization.normalize_qpos(states)
|
||||
actions = self.normalization.normalize_action(actions)
|
||||
cond = self._build_cond(images, states, task=batch.get('task', None))
|
||||
|
||||
x = actions
|
||||
e = torch.randn_like(x)
|
||||
t = torch.rand(batch_size, device=x.device, dtype=x.dtype)
|
||||
r = torch.rand(batch_size, device=x.device, dtype=x.dtype)
|
||||
t, r = torch.maximum(t, r), torch.minimum(t, r)
|
||||
|
||||
t_broadcast = self._broadcast_batch_time(t, x)
|
||||
z_t = (1 - t_broadcast) * x + t_broadcast * e
|
||||
|
||||
v = self.fn(z_t, t, t, cond=cond)
|
||||
u, du_dt = self._compute_u_and_du_dt(z_t, r, t, cond=cond, v=v)
|
||||
V = self._compound_velocity(u, du_dt, r, t)
|
||||
target = e - x
|
||||
|
||||
loss = self._velocity_loss_from_error(V - target)
|
||||
if action_is_pad is not None:
|
||||
mask = (~action_is_pad).unsqueeze(-1).to(loss.dtype)
|
||||
valid_count = mask.sum() * loss.shape[-1]
|
||||
loss = (loss * mask).sum() / valid_count.clamp_min(1.0)
|
||||
else:
|
||||
loss = loss.mean()
|
||||
return loss
|
||||
|
||||
@torch.no_grad()
|
||||
def predict_action(self, images, proprioception, task=None):
|
||||
batch_size = proprioception.shape[0]
|
||||
proprioception = self.normalization.normalize_qpos(proprioception)
|
||||
cond = self._build_cond(images, proprioception, task=task)
|
||||
z_t = torch.randn((batch_size, self.pred_horizon, self.action_dim), device=cond.device, dtype=cond.dtype)
|
||||
action = self._sample_one_step(z_t, cond=cond)
|
||||
return self.normalization.denormalize_action(action)
|
||||
|
||||
@torch.no_grad()
|
||||
def predict_action_chunk(self, batch: Dict[str, torch.Tensor]) -> torch.Tensor:
|
||||
return self.predict_action(batch['images'], batch['qpos'], task=batch.get('task', None))
|
||||
|
||||
def reset(self):
|
||||
self._queues = {
|
||||
'qpos': deque(maxlen=self.obs_horizon),
|
||||
'images': deque(maxlen=self.obs_horizon),
|
||||
'task': deque(maxlen=self.obs_horizon),
|
||||
'action': deque(maxlen=self.pred_horizon - self.obs_horizon + 1),
|
||||
}
|
||||
|
||||
def _populate_queues(self, observation: Dict[str, torch.Tensor]) -> None:
|
||||
if 'qpos' in observation:
|
||||
self._queues['qpos'].append(observation['qpos'].clone())
|
||||
if 'images' in observation:
|
||||
ordered_images = self._order_images(observation['images'])
|
||||
self._queues['images'].append({k: v.clone() for k, v in ordered_images.items()})
|
||||
if 'task' in observation:
|
||||
self._queues['task'].append(observation['task'])
|
||||
|
||||
def _prepare_observation_batch(self) -> Dict[str, torch.Tensor]:
|
||||
qpos_list = list(self._queues['qpos'])
|
||||
if not qpos_list:
|
||||
raise ValueError('observation queue is empty.')
|
||||
while len(qpos_list) < self.obs_horizon:
|
||||
qpos_list.append(qpos_list[-1])
|
||||
batch_qpos = torch.stack(qpos_list, dim=0).unsqueeze(0)
|
||||
|
||||
images_list = list(self._queues['images'])
|
||||
if not images_list:
|
||||
raise ValueError('image queue is empty.')
|
||||
while len(images_list) < self.obs_horizon:
|
||||
images_list.append(images_list[-1])
|
||||
names = self.camera_names if self.camera_names is not None else tuple(sorted(images_list[0].keys()))
|
||||
batch_images = {
|
||||
name: torch.stack([item[name] for item in images_list], dim=0).unsqueeze(0)
|
||||
for name in names
|
||||
}
|
||||
batch = {'qpos': batch_qpos, 'images': batch_images}
|
||||
if self._queues['task']:
|
||||
batch['task'] = [list(self._queues['task'])[-1]]
|
||||
elif self.task_description is not None:
|
||||
batch['task'] = [self.task_description]
|
||||
return batch
|
||||
|
||||
@torch.no_grad()
|
||||
def select_action(self, observation: Dict[str, torch.Tensor]) -> torch.Tensor:
|
||||
device = self._get_model_device()
|
||||
observation = self._move_to_device(observation, device)
|
||||
self._populate_queues(observation)
|
||||
if len(self._queues['action']) == 0:
|
||||
batch = self._prepare_observation_batch()
|
||||
actions = self.predict_action_chunk(batch)
|
||||
start = self.obs_horizon - 1
|
||||
end = start + self.num_action_steps
|
||||
executable_actions = actions[:, start:end]
|
||||
for i in range(executable_actions.shape[1]):
|
||||
self._queues['action'].append(executable_actions[:, i].squeeze(0))
|
||||
return self._queues['action'].popleft()
|
||||
|
||||
def get_normalization_stats(self):
|
||||
return self.normalization.get_stats()
|
||||
@@ -25,6 +25,8 @@ vision_backbone:
|
||||
diffusion_steps: 100
|
||||
inference_steps: 1
|
||||
head_type: "transformer"
|
||||
loss_type: "pseudo_huber"
|
||||
pseudo_huber_delta: 1.0
|
||||
|
||||
head:
|
||||
input_dim: ${agent.action_dim}
|
||||
|
||||
@@ -24,6 +24,8 @@ vision_backbone:
|
||||
diffusion_steps: 100
|
||||
inference_steps: 1
|
||||
head_type: "transformer"
|
||||
loss_type: "pseudo_huber"
|
||||
pseudo_huber_delta: 1.0
|
||||
|
||||
head:
|
||||
input_dim: ${agent.action_dim}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user