Compare commits
19 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Logic
|
5ae9f5fa48 | ||
|
Logic
|
acbd7c605a | ||
|
Logic
|
73f5b6e3d9 | ||
|
Logic
|
bc4caf355b | ||
|
Logic
|
b1116e489f | ||
|
Logic
|
a2c018acce | ||
|
Logic
|
ff7c9c1f2a | ||
|
Logic
|
d51b3ecafa | ||
|
Logic
|
2033169840 | ||
|
Logic
|
a78006808a | ||
|
Logic
|
0586a6e6c7 | ||
|
Logic
|
48f0eb8dd0 | ||
|
Logic
|
3a17744dcf | ||
|
Logic
|
0514f86c36 | ||
|
Logic
|
dffd92f82d | ||
|
Logic
|
c2000b5533 | ||
|
Logic
|
8d6060224a | ||
|
Logic
|
8a8193fe7e | ||
|
Logic
|
1a92c5e8a6 |
@@ -0,0 +1 @@
|
||||
{"reason":"idle timeout","timestamp":1776932392614}
|
||||
@@ -0,0 +1 @@
|
||||
31521
|
||||
@@ -0,0 +1,48 @@
|
||||
<h2>新任务场景草图：双臂空中穿环插入</h2>
|
||||
<p class="subtitle">保持双 Diana + 同一张桌子；左侧放环状木块，右侧放长条木块，先各自抓起，再在空中完成穿孔插入。</p>
|
||||
|
||||
<div class="split">
|
||||
<div class="mockup">
|
||||
<div class="mockup-header">俯视布局（初始）</div>
|
||||
<div class="mockup-body">
|
||||
<div style="position:relative;height:340px;background:#f3f4f6;border-radius:16px;border:1px solid #d1d5db;overflow:hidden;">
|
||||
<div style="position:absolute;left:12%;top:18%;width:22%;height:64%;border:2px dashed #60a5fa;border-radius:18px;background:rgba(96,165,250,.08);display:flex;align-items:flex-start;justify-content:center;padding-top:8px;font-weight:700;">左臂采样区（环）</div>
|
||||
<div style="position:absolute;right:12%;top:18%;width:22%;height:64%;border:2px dashed #f59e0b;border-radius:18px;background:rgba(245,158,11,.08);display:flex;align-items:flex-start;justify-content:center;padding-top:8px;font-weight:700;">右臂采样区（长条）</div>
|
||||
<div style="position:absolute;left:50%;top:50%;transform:translate(-50%,-50%);width:120px;height:120px;border:10px solid #ef4444;background:white;box-sizing:border-box;display:flex;align-items:center;justify-content:center;font-weight:700;color:#ef4444;">环块</div>
|
||||
<div style="position:absolute;left:72%;top:50%;transform:translate(-50%,-50%) rotate(0deg);width:110px;height:22px;background:#10b981;border:2px solid #065f46;border-radius:6px;display:flex;align-items:center;justify-content:center;color:white;font-size:12px;font-weight:700;">长条</div>
|
||||
<div style="position:absolute;left:50%;top:12%;transform:translateX(-50%);padding:6px 10px;background:#111827;color:white;border-radius:999px;font-size:12px;">桌面中心 / 空中插入区上方</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="mockup">
|
||||
<div class="mockup-header">任务阶段</div>
|
||||
<div class="mockup-body">
|
||||
<div class="section">
|
||||
<div class="label">Phase 1</div>
|
||||
<h3>左右臂分别接近目标</h3>
|
||||
<p>左臂只抓环状木块，右臂只抓长条木块。两物体初始姿态固定，只随机平面位置。</p>
|
||||
</div>
|
||||
<div class="section">
|
||||
<div class="label">Phase 2</div>
|
||||
<h3>同时抓起并离桌</h3>
|
||||
<p>两个物体都需要离开桌面后，才允许触发最终成功奖励。</p>
|
||||
</div>
|
||||
<div class="section">
|
||||
<div class="label">Phase 3</div>
|
||||
<h3>空中对齐并插入</h3>
|
||||
<p>在桌面上方的会合区完成对齐，让长条沿自身长轴穿过环孔；一旦穿孔成功即判成功。</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="section">
|
||||
<h3>奖励草案（总分 5）</h3>
|
||||
<div class="cards">
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>左臂碰到环块</p></div></div>
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>右臂碰到长条</p></div></div>
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>环块离桌</p></div></div>
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>长条离桌</p></div></div>
|
||||
<div class="card"><div class="card-body"><h3>+1</h3><p>两者离桌时，长条穿过环孔</p></div></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -0,0 +1 @@
|
||||
<div style="display:flex;align-items:center;justify-content:center;min-height:60vh"><p class="subtitle">Continuing in terminal...</p></div>
|
||||
@@ -0,0 +1,39 @@
|
||||
# Repository Guidelines
|
||||
|
||||
## Project Structure & Module Organization
|
||||
- `roboimi/` is the main package: simulation code in `roboimi/envs/`, assets/helpers in `roboimi/assets/`, VLA models/config in `roboimi/vla/`, and runnable scripts in `roboimi/demos/` plus `roboimi/demos/vla_scripts/`.
|
||||
- `tests/` holds `unittest` regressions for training, rollout, dataset loading, and asset resolution.
|
||||
- `docs/superpowers/` stores design specs and plans.
|
||||
- `docs/experiment-guide.md` store how beginning experiment
|
||||
- `runs/`, `outputs/`, `swanlog/`, `.worktrees/`, and `checkpoints/` are generated experiment artifacts, not source modules.
|
||||
|
||||
## Build, Test, and Development Commands
|
||||
- `conda env create -f environment.yml && conda activate roboimi` — create the standard environment.
|
||||
- `pip install -e .` — editable install.
|
||||
- `python -m unittest discover -s tests -p 'test_*.py'` — run all tests.
|
||||
- `python -m unittest tests.test_train_vla_rollout_validation -v` — run one focused test module.
|
||||
- `python roboimi/vla/scripts/calculate_stats.py` — regenerate normalization stats.
|
||||
- `python roboimi/demos/vla_scripts/train_vla.py train.max_steps=1000` — quick training smoke test.
|
||||
- `python roboimi/demos/vla_scripts/eval_vla.py eval.ckpt_path=checkpoints/vla_model_best.pt eval.num_episodes=5` — evaluate a checkpoint.
|
||||
|
||||
## Coding Style & Naming Conventions
|
||||
- Use Python, 4-space indentation, and PEP 8–style naming.
|
||||
- Prefer `snake_case` for files, functions, variables, and Hydra keys; use `CamelCase` for classes.
|
||||
- Match existing Hydra namespaces such as `train.*`, `eval.*`, and `agent.*`.
|
||||
- Keep diffs focused and imports tidy. Experiment names usually encode host/GPU, e.g. `...-l20g3-...`, `...-5880g1-...`, or `...-5090-...`.
|
||||
|
||||
## Testing Guidelines
|
||||
- Tests use `unittest` and `unittest.mock`.
|
||||
- Name files `tests/test_*.py` and classes `*Test`.
|
||||
- Add direct regression coverage for changes to training, rollout, config, or dataset code paths.
|
||||
- No strict coverage target is defined, but changed behavior should be tested and affected suites should pass before review.
|
||||
|
||||
## Commit & Pull Request Guidelines
|
||||
- Follow existing prefixes: `feat`, `fix`, `chore`, `docs`, `debug`, optionally with scope, e.g. `feat(eval): export rollout video timing`.
|
||||
- Keep commit messages imperative and specific.
|
||||
- PRs should state the goal, touched configs/scripts, exact test commands, and dataset/checkpoint assumptions. Attach visual artifacts only when outputs change.
|
||||
|
||||
## Experiment Host Reference
|
||||
- **Local workstation (`droid-z790eagleax`)** — local shell; **1× RTX 5090 32 GB**; env: `/home/droid/.conda/envs/roboimi/bin/python`.
|
||||
- **5880 node (`droid-System-Product-Name`)** — username: `droid`; IP: `100.73.14.65`; connect with `ssh droid@100.73.14.65`; **2× RTX 5880 Ada 48 GB**; env: `/home/droid/miniforge3/envs/roboimi/bin/python`; trusted tailnet host.
|
||||
- **L20 node (`droid-G5500-V7`)** — username: `droid`; IP: `100.119.99.14`; connect with `ssh droid@100.119.99.14`; **8× NVIDIA L20 46 GB**; env: `/home/droid/miniforge3/envs/roboimi/bin/python`; store large experiment data under `/data`.
|
||||
+78
-24
@@ -1,72 +1,126 @@
|
||||
Sat Apr 25 10:03:15 2026
|
||||
Mon Mar 30 13:35:31 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:12:58 2026
|
||||
Mon Mar 30 13:35:31 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:13:59 2026
|
||||
Mon Mar 30 13:39:46 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:14:52 2026
|
||||
Mon Mar 30 13:39:46 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:15:27 2026
|
||||
Mon Mar 30 14:30:24 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:16:03 2026
|
||||
Mon Mar 30 14:30:24 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:18:11 2026
|
||||
Mon Mar 30 14:41:26 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:19:03 2026
|
||||
Mon Mar 30 14:41:26 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:20:06 2026
|
||||
Mon Mar 30 15:21:19 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:20:38 2026
|
||||
Mon Mar 30 15:21:19 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:21:23 2026
|
||||
Mon Mar 30 20:58:52 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:21:55 2026
|
||||
Mon Mar 30 20:59:05 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:22:29 2026
|
||||
Mon Mar 30 22:05:37 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:23:01 2026
|
||||
Tue Mar 31 16:00:10 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:23:33 2026
|
||||
Tue Mar 31 16:00:10 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:25:01 2026
|
||||
Tue Mar 31 16:03:41 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:25:59 2026
|
||||
Tue Mar 31 16:03:41 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:26:52 2026
|
||||
Tue Mar 31 16:05:09 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:27:20 2026
|
||||
Tue Mar 31 16:05:09 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:27:49 2026
|
||||
Tue Mar 31 16:06:00 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:28:18 2026
|
||||
Tue Mar 31 16:06:00 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:37:53 2026
|
||||
Tue Mar 31 16:09:24 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:38:56 2026
|
||||
Tue Mar 31 16:09:24 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sat Apr 25 10:41:12 2026
|
||||
Tue Mar 31 16:14:36 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:14:36 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:20:44 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:23:07 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Tue Mar 31 16:29:49 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 11:55:22 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 11:55:22 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Thu Apr 23 12:43:50 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sun May 24 21:50:49 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sun May 24 21:51:10 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sun May 24 21:53:08 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
Sun May 24 21:53:28 2026
|
||||
WARNING: OpenGL error 0x502 in or before mjr_makeContext
|
||||
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Capture first and last (success) frames from front camera for both sim tasks."""
|
||||
import os
|
||||
import time
|
||||
import numpy as np
|
||||
import cv2
|
||||
from roboimi.envs.double_pos_ctrl_env import make_sim_env
|
||||
from roboimi.demos.diana_policy import TestPickAndTransferPolicy
|
||||
from roboimi.demos.diana_air_insert_policy import TestAirInsertPolicy
|
||||
from roboimi.utils.act_ex_utils import sample_transfer_pose, sample_air_insert_socket_peg_state
|
||||
|
||||
OUTPUT_DIR = "sim_frames"
|
||||
IMG_SIZE = (256, 256)
|
||||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||||
|
||||
|
||||
def capture_task(task_name, policy, task_state, episode_len):
|
||||
env = make_sim_env(task_name, headless=True)
|
||||
env.reset(task_state)
|
||||
time.sleep(1)
|
||||
|
||||
env._update_camera_images_sync()
|
||||
first_frame = cv2.resize(env.front, IMG_SIZE)
|
||||
|
||||
last_success_frame = None
|
||||
max_reward = 0
|
||||
|
||||
for step in range(episode_len):
|
||||
action = policy.predict(task_state, step)
|
||||
env.step(action)
|
||||
env._update_camera_images_sync()
|
||||
if env.rew is not None and env.rew >= max_reward:
|
||||
max_reward = env.rew
|
||||
if env.rew == env.max_reward:
|
||||
last_success_frame = cv2.resize(env.front, IMG_SIZE)
|
||||
|
||||
if last_success_frame is None:
|
||||
last_success_frame = cv2.resize(env.front, IMG_SIZE)
|
||||
print(f" [WARN] {task_name}: max_reward={max_reward}/{env.max_reward}, using last frame")
|
||||
else:
|
||||
print(f" {task_name}: success! max_reward={max_reward}")
|
||||
|
||||
return first_frame, last_success_frame
|
||||
|
||||
|
||||
def main():
|
||||
# Transfer task (episode_len=700, policy covers full range)
|
||||
print("Running sim_transfer...")
|
||||
task_state_transfer = sample_transfer_pose()
|
||||
policy_transfer = TestPickAndTransferPolicy(inject_noise=False)
|
||||
first_t, last_t = capture_task('sim_transfer', policy_transfer, task_state_transfer, 700)
|
||||
cv2.imwrite(os.path.join(OUTPUT_DIR, "transfer_front_first.png"), first_t)
|
||||
cv2.imwrite(os.path.join(OUTPUT_DIR, "transfer_front_last.png"), last_t)
|
||||
|
||||
# Air insert / peg-in-socket (policy EPISODE_END_T=600)
|
||||
print("Running sim_air_insert_socket_peg...")
|
||||
task_state_insert = sample_air_insert_socket_peg_state()
|
||||
policy_insert = TestAirInsertPolicy(inject_noise=False)
|
||||
first_i, last_i = capture_task('sim_air_insert_socket_peg', policy_insert, task_state_insert, 600)
|
||||
cv2.imwrite(os.path.join(OUTPUT_DIR, "air_insert_front_first.png"), first_i)
|
||||
cv2.imwrite(os.path.join(OUTPUT_DIR, "air_insert_front_last.png"), last_i)
|
||||
|
||||
print(f"\nDone! Images saved to {OUTPUT_DIR}/")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,498 @@
|
||||
# 实验操作指南
|
||||
|
||||
## 0. 适用范围与当前起点
|
||||
|
||||
这份文档面向 **当前 `main` 分支** 的通用训练 / 验证 / 评估流程，不绑定某一条历史实验分支。
|
||||
|
||||
当前默认 Hydra 起点：
|
||||
- agent：`resnet_transformer`
|
||||
- data：`simpe_robot_dataset`
|
||||
- eval：`eval`
|
||||
|
||||
当前仓库里常见可选 agent 配置包括：
|
||||
- `resnet_transformer`
|
||||
- `resnet_diffusion`
|
||||
- `resnet_gr00t_dit`
|
||||
- `resnet_imf_attnres`
|
||||
- `resnet_imf_attnres_multitoken`
|
||||
- `siglip2_imf_attnres`
|
||||
- `lewm_imf_attnres`
|
||||
|
||||
说明：
|
||||
- `experiment_suites/` 下的历史目录仍然可以作为参考，但**正文以当前 `main` 的通用流程为准**。
|
||||
- 如果你只是想快速开始，优先从默认 `resnet_transformer` 配置出发，再按实验需求替换 agent。
|
||||
|
||||
---
|
||||
|
||||
## 1. 关键文件与入口
|
||||
|
||||
| 路径 | 作用 |
|
||||
| --- | --- |
|
||||
| `roboimi/demos/vla_scripts/train_vla.py` | 主训练入口；负责数据集、checkpoint、`val/loss`、held-out action MSE、训练期 rollout 验证、SwanLab |
|
||||
| `roboimi/demos/vla_scripts/eval_vla.py` | 单次 rollout / 离线评估入口；支持 headless、summary、trajectory image、video artifact、多进程并行 rollout |
|
||||
| `roboimi/vla/conf/config.yaml` | 全局 Hydra 训练配置 |
|
||||
| `roboimi/vla/conf/data/simpe_robot_dataset.yaml` | 默认数据集配置；数据路径、相机名、图像 resize 都在这里 |
|
||||
| `roboimi/vla/conf/eval/eval.yaml` | eval 默认配置；`eval.ckpt_path`、`eval.num_episodes`、artifact 开关、多进程 rollout 配置都在这里 |
|
||||
| `roboimi/vla/conf/agent/*.yaml` | 可选 agent 配置集合 |
|
||||
| `roboimi/vla/data/simpe_robot_dataset.py` | HDF5 懒加载数据集；支持 `episode_indices` 过滤与 `available_episode_indices` 元信息 |
|
||||
| `roboimi/vla/scripts/calculate_stats.py` | 重算 `dataset_stats.pkl` |
|
||||
| `experiment_suites/` | 历史实验目录；可参考，但不作为当前主流程的唯一事实来源 |
|
||||
|
||||
---
|
||||
|
||||
## 2. 三台机器与环境
|
||||
|
||||
| 机器 | GPU | Python | 常用数据集路径 | 备注 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 本地 `droid-z790eagleax` | 1× RTX 5090 32GB | `/home/droid/.conda/envs/roboimi/bin/python` | `/home/droid/project/diana_sim/sim_transfer` | 适合 smoke、单条主跑、本地调参 |
|
||||
| 5880 节点 `100.73.14.65` | 2× RTX 5880 Ada 48GB | `/home/droid/miniforge3/envs/roboimi/bin/python` | `/home/droid/sim_dataset/sim_transfer` | 适合 2 条并行主跑 |
|
||||
| L20 节点 `100.119.99.14` | 8× NVIDIA L20 46GB | `/home/droid/miniforge3/envs/roboimi/bin/python` | `/data/simtransfer/current` | 适合大规模 grid；建议数据与 run 放 `/data` |
|
||||
|
||||
连接：
|
||||
- 5880：`ssh droid@100.73.14.65`
|
||||
- L20：`ssh droid@100.119.99.14`
|
||||
|
||||
说明：
|
||||
- repo / worktree 路径可能会随当前 checkout 或同步目录变化，**以你当前机器上的实际 repo 路径为准**。
|
||||
- 如果训练目录会被 Hydra 自动切走，建议显式设置 `hydra.run.dir=...` 以固定输出目录。
|
||||
|
||||
---
|
||||
|
||||
## 3. 训练流怎么走
|
||||
|
||||
`train_vla.py` 当前主流程如下：
|
||||
|
||||
1. 读取 Hydra 配置并打印完整 cfg
|
||||
2. 根据配置确定数据集图像 resize：
|
||||
- 默认用 `data.image_resize_shape`
|
||||
- 如果 `agent.vision_backbone.dataset_image_resize_shape` 存在，则优先用 backbone 覆盖值
|
||||
3. 通过 `build_train_val_datasets()` 构建 dataset / train dataset / val dataset
|
||||
- 若设置 `train.val_episode_indices`：按显式 episode 切出 held-out val
|
||||
- 否则按 `train.val_split` 做随机划分
|
||||
4. 用 `DataLoader` 建 train / val loader
|
||||
5. 从 `dataset_dir/dataset_stats.pkl` 读取归一化统计
|
||||
6. 实例化当前指定的 agent：`instantiate(cfg.agent, dataset_stats=...)`
|
||||
7. 可选加载：
|
||||
- `train.pretrained_ckpt`：微调起点
|
||||
- `train.resume_ckpt`：断点续训（支持显式路径或 `auto`）
|
||||
8. 建 optimizer 与 scheduler
|
||||
9. 训练循环里按 `log_freq` 记录 train loss / lr
|
||||
10. 按 `save_freq` 保存 `checkpoints/vla_model_step_*.pt`，并在有 val loader 时计算 `val/loss`
|
||||
11. 若设置了显式 held-out val，并且 `train.action_mse_val_freq_epochs > 0`，则按 epoch 计算 held-out `action MSE`
|
||||
12. 按 `train.rollout_val_freq_epochs` 跑训练期 rollout 验证
|
||||
13. 最后写：
|
||||
- `checkpoints/vla_model_best.pt`
|
||||
- `checkpoints/vla_model_final.pt`
|
||||
|
||||
当前 best model 选择逻辑：
|
||||
- **第一次拿到 rollout reward 之前**：先用 `val_loss`（或 train loss 回退）挑 best
|
||||
- **第一次 rollout 之后**：优先用 `rollout_avg_reward` 挑 best
|
||||
|
||||
输出目录：
|
||||
- 当前代码实际输出写到 **Hydra runtime output dir** 下
|
||||
- checkpoint 默认在：`<hydra_output_dir>/checkpoints/`
|
||||
- 如果不显式指定 `hydra.run.dir`，Hydra 会自动生成目录
|
||||
|
||||
---
|
||||
|
||||
## 4. 验证流怎么走
|
||||
|
||||
### 4.1 常规 `val/loss`
|
||||
|
||||
常规验证有两种来源：
|
||||
|
||||
1. **随机划分验证集**
|
||||
- 用 `train.val_split > 0`
|
||||
- 训练脚本会在保存 checkpoint 时计算 `val/loss`
|
||||
|
||||
2. **显式 held-out episode 验证集**
|
||||
- 用 `train.val_episode_indices=[...]`
|
||||
- 训练集 = 全部 episode - held-out episode
|
||||
- 验证集 = held-out episode
|
||||
- checkpoint 保存时同样会计算 `val/loss`
|
||||
|
||||
说明：
|
||||
- 若 `train.val_split=0.0` 且未设置 `train.val_episode_indices`，则不会有 `val/loss`。
|
||||
|
||||
### 4.2 held-out action MSE
|
||||
|
||||
如果想对固定 held-out episode 做更稳定、可复现的动作预测误差验证，推荐使用：
|
||||
|
||||
- `train.val_split=0.0`
|
||||
- `train.val_episode_indices=[100]`（或你指定的 episode 列表）
|
||||
- `train.action_mse_val_freq_epochs=1`
|
||||
|
||||
当前 `main` 中这套逻辑会：
|
||||
- 每隔 `action_mse_val_freq_epochs` 个 epoch
|
||||
- 对 held-out val loader 调 `agent.predict_action_chunk(...)`
|
||||
- 与 batch 中的 `action` 计算 masked MSE
|
||||
- 如果存在 `action_is_pad`，会自动 mask 掉 padding 部分
|
||||
|
||||
日志 key：
|
||||
- 控制台 / `train_vla.log`：`held-out action MSE`
|
||||
- SwanLab：`val/action_mse`
|
||||
|
||||
重要约束：
|
||||
- `train.action_mse_val_freq_epochs > 0` **必须搭配** `train.val_episode_indices`
|
||||
- 如果只设了 `action_mse_val_freq_epochs` 而没有设 `val_episode_indices`，训练会直接报错
|
||||
|
||||
### 4.3 rollout 验证
|
||||
|
||||
训练内 rollout 验证由：
|
||||
- `train_vla.py -> run_rollout_validation() -> eval_vla._run_eval()`
|
||||
|
||||
当前训练内 rollout 会强制：
|
||||
- `headless=true`
|
||||
- `verbose_action=false`
|
||||
- `record_video=false`
|
||||
- `save_trajectory_image=true`
|
||||
- `trajectory_image_camera_name=front`
|
||||
- `save_summary_json=true`
|
||||
|
||||
当前这套路径是**配置驱动**的：
|
||||
- `train.rollout_device`：默认跟随 `train.device`
|
||||
- `train.rollout_num_workers`：默认 `null`
|
||||
- 当 rollout 设备是 CPU 时，自动退化为 `1`
|
||||
- 当 rollout 设备是 CUDA 时，自动推断为 `min(train.rollout_num_episodes, 8)`
|
||||
- `train.rollout_cuda_devices`：默认 `null`，等价于逻辑 GPU `[0]`
|
||||
- `train.rollout_response_timeout_s`
|
||||
- `train.rollout_server_startup_timeout_s`
|
||||
|
||||
所以现在：
|
||||
- 训练在 CUDA 上时，训练期 rollout 默认也会走 GPU
|
||||
- `rollout_num_workers > 1` 时，会走并行 rollout
|
||||
- 可以是 **单 GPU 多 worker 共用一个 inference server**
|
||||
- 也可以是 **多 GPU 多 server 分摊 worker**
|
||||
|
||||
训练内 rollout artifact 默认落到：
|
||||
- `<hydra_output_dir>/rollout_artifacts/<checkpoint_stem>/`
|
||||
|
||||
常见文件：
|
||||
- `rollout_summary.json`
|
||||
- `rollout_front_ep01_trajectory.png` ...
|
||||
|
||||
日志重点看：
|
||||
- `Epoch X rollout 平均奖励`
|
||||
- `最佳模型已更新`
|
||||
|
||||
---
|
||||
|
||||
## 5. 数据集加载与 `val_episode_indices` 机制
|
||||
|
||||
### 5.1 数据集格式
|
||||
|
||||
`SimpleRobotDataset` 读取 `dataset_dir` 下的 `*.hdf5` / `episode_*.hdf5`，每个 episode 文件至少要有：
|
||||
- `action`
|
||||
- `observations/qpos`
|
||||
- `observations/images/{cam_name}`
|
||||
|
||||
默认数据配置中的相机：
|
||||
- `r_vis`
|
||||
- `top`
|
||||
- `front`
|
||||
|
||||
### 5.2 懒加载行为
|
||||
|
||||
`roboimi/vla/data/simpe_robot_dataset.py` 是按帧懒加载，不会一次性把整套 HDF5 全读进内存。
|
||||
|
||||
它会：
|
||||
- 扫描目录下的 HDF5 文件
|
||||
- 在 worker 内做 HDF5 文件句柄 LRU 缓存
|
||||
- 根据文件名中的 `episode_XXX` 建立 `available_episode_indices`
|
||||
|
||||
### 5.3 `val_episode_indices` 怎么切
|
||||
|
||||
`build_train_val_datasets()` 的逻辑是：
|
||||
|
||||
1. 先 instantiate 一次完整 dataset
|
||||
2. 读取 `dataset.available_episode_indices`
|
||||
3. 检查 `train.val_episode_indices` 是否都存在
|
||||
4. 用 `episode_indices=` 再各 instantiate 一次：
|
||||
- train dataset = 全部 episode - held-out episode
|
||||
- val dataset = 只包含 held-out episode
|
||||
|
||||
因此：
|
||||
- `train.val_episode_indices=[100]` 的意思是把 `episode_100.hdf5` 整个拿去做 held-out val
|
||||
- 如果 episode 不存在，会直接报错
|
||||
- 如果你把所有 episode 都塞进 `val_episode_indices`，也会直接报错，因为训练集会变空
|
||||
|
||||
### 5.4 图像 resize 与返回字段
|
||||
|
||||
dataset 侧 resize 默认来自：
|
||||
- `data.image_resize_shape`
|
||||
- 如果 backbone 额外覆盖，则优先 `agent.vision_backbone.dataset_image_resize_shape`
|
||||
|
||||
当前通用 batch 返回字段包括：
|
||||
- `observation.state`
|
||||
- `observation_is_pad`
|
||||
- `observation.<cam>`
|
||||
- `action`
|
||||
- `action_is_pad`
|
||||
- `task`
|
||||
|
||||
### 5.5 统计文件
|
||||
|
||||
训练和推理都默认依赖 `dataset_stats.pkl`。数据集更新后需要重算：
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/vla/scripts/calculate_stats.py \
|
||||
--dataset_dir /home/droid/project/diana_sim/sim_transfer
|
||||
```
|
||||
|
||||
远端只要把：
|
||||
- Python 路径
|
||||
- `--dataset_dir`
|
||||
|
||||
替换成对应机器上的实际路径即可。
|
||||
|
||||
---
|
||||
|
||||
## 6. SwanLab 行为
|
||||
|
||||
当前默认配置里：
|
||||
- `train.use_swanlab=false`
|
||||
|
||||
如果要开启，通常显式设置：
|
||||
- `train.use_swanlab=true`
|
||||
- `train.swanlab_project=roboimi-vla`
|
||||
- `train.swanlab_run_name=<run_name>`
|
||||
|
||||
`train_vla.py` 当前会记录：
|
||||
- 初始化时上传 `train` / `data` / `agent` 三段 config
|
||||
- 训练中：
|
||||
- `train/loss`
|
||||
- `train/lr`
|
||||
- `train/best_loss`
|
||||
- `train/step`
|
||||
- checkpoint 验证时：
|
||||
- `val/loss`
|
||||
- held-out 数值验证时：
|
||||
- `val/action_mse`
|
||||
- rollout 验证时：
|
||||
- `rollout/avg_reward`
|
||||
- `rollout/epoch`
|
||||
- 训练结束时：
|
||||
- `final/checkpoint_path`
|
||||
- `final/best_checkpoint_path`
|
||||
|
||||
训练期 rollout 生成的前视图轨迹 PNG 会 best-effort 上传到 SwanLab；上传失败只会 warning，不会让训练中断。
|
||||
|
||||
---
|
||||
|
||||
## 7. 并行 rollout 说明
|
||||
|
||||
### 7.1 这套能力现在在哪里
|
||||
|
||||
当前主仓库已经**内置**多进程并行 rollout 能力，入口就是：
|
||||
- `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
|
||||
控制参数：
|
||||
- `eval.num_workers`
|
||||
- `eval.cuda_devices`
|
||||
|
||||
语义：
|
||||
- `eval.num_workers`：环境 worker 数，按 episode 切分
|
||||
- `eval.cuda_devices`：推理 server 绑定到哪些逻辑 GPU
|
||||
|
||||
### 7.2 两种常见模式
|
||||
|
||||
1. **单机单卡，多 worker 共用同一张 GPU**
|
||||
- 典型：5090 只有 1 卡，但想让 4 个 rollout worker 并行跑环境
|
||||
- 形式：`eval.device=cuda eval.num_workers=4 'eval.cuda_devices=[0]'`
|
||||
- 这时是 **1 个 CUDA inference server + 4 个 env worker**
|
||||
|
||||
2. **单机多卡，多 server 分摊 worker**
|
||||
- 典型：5880 / L20 有多卡
|
||||
- 形式：`eval.device=cuda eval.num_workers=8 'eval.cuda_devices=[0,1]'`
|
||||
- worker 会按 round-robin 分到多个 server 上
|
||||
|
||||
### 7.3 操作约束
|
||||
|
||||
- 并行 rollout 依赖 **多进程 eval 路径**，不是 `train.num_workers`
|
||||
- `train.num_workers` 是 DataLoader worker，和 rollout 并行不是一回事
|
||||
- `eval.num_workers > 1` 时必须 `eval.headless=true`
|
||||
- worker 数会自动 cap 到 `eval.num_episodes`
|
||||
- 多 worker 时**不支持**同时导出：
|
||||
- `eval.record_video=true`
|
||||
- `eval.save_trajectory=true`
|
||||
- `eval.save_trajectory_npz=true`
|
||||
- `eval.save_trajectory_image=true` 可以开，适合并行 reward + 定性检查一起做
|
||||
|
||||
---
|
||||
|
||||
## 8. 常用命令模板
|
||||
|
||||
下面的模板都以 **当前 `main` 通用流程** 为准。远端机器只需要替换：
|
||||
- Python 路径
|
||||
- repo 路径
|
||||
- `data.dataset_dir`
|
||||
- `hydra.run.dir`
|
||||
|
||||
### 8.1 本地 smoke train
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.batch_size=8 \
|
||||
train.max_steps=1000 \
|
||||
train.num_workers=4 \
|
||||
train.save_freq=200 \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/smoke-resnet-transformer
|
||||
```
|
||||
|
||||
### 8.2 通用训练模板
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.batch_size=16 \
|
||||
train.lr=0.0001 \
|
||||
train.max_steps=100000 \
|
||||
train.num_workers=4 \
|
||||
train.save_freq=2000 \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.swanlab_run_name=<run_name> \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/<run_name>
|
||||
```
|
||||
|
||||
### 8.3 带 held-out episode 验证的训练模板
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.batch_size=16 \
|
||||
train.lr=0.0001 \
|
||||
train.max_steps=100000 \
|
||||
train.num_workers=4 \
|
||||
train.val_split=0.0 \
|
||||
'train.val_episode_indices=[100]' \
|
||||
train.action_mse_val_freq_epochs=1 \
|
||||
train.rollout_val_freq_epochs=5 \
|
||||
train.rollout_num_episodes=10 \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.swanlab_run_name=<run_name> \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/<run_name>
|
||||
```
|
||||
|
||||
### 8.4 断点续训模板
|
||||
|
||||
`train_vla.py` 支持通过 CLI override 使用 `train.resume_ckpt`：
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.resume_ckpt=auto \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/<run_name>
|
||||
```
|
||||
|
||||
也可以把 `train.resume_ckpt` 换成显式 checkpoint 路径。
|
||||
|
||||
### 8.5 单次离线评估
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/eval_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda eval.device=cuda \
|
||||
eval.ckpt_path=/home/droid/project/roboimi/runs/<run_name>/checkpoints/vla_model_best.pt \
|
||||
eval.num_episodes=10 \
|
||||
eval.headless=true \
|
||||
eval.verbose_action=false \
|
||||
eval.save_summary_json=true \
|
||||
eval.save_trajectory_image=true \
|
||||
eval.trajectory_image_camera_name=front \
|
||||
eval.artifact_dir=/tmp/roboimi_eval_front
|
||||
```
|
||||
|
||||
### 8.6 并行离线评估
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/eval_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda eval.device=cuda \
|
||||
eval.ckpt_path=/home/droid/project/roboimi/runs/<run_name>/checkpoints/vla_model_best.pt \
|
||||
eval.num_episodes=10 \
|
||||
eval.num_workers=4 \
|
||||
'eval.cuda_devices=[0]' \
|
||||
eval.headless=true \
|
||||
eval.verbose_action=false \
|
||||
eval.save_summary_json=true \
|
||||
eval.save_trajectory_image=true \
|
||||
eval.trajectory_image_camera_name=front \
|
||||
eval.artifact_dir=/tmp/roboimi_parallel_eval
|
||||
```
|
||||
|
||||
### 8.7 训练内启用并行 GPU rollout
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_transformer \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.device=cuda \
|
||||
train.batch_size=16 \
|
||||
train.lr=0.0001 \
|
||||
train.max_steps=100000 \
|
||||
train.num_workers=4 \
|
||||
train.rollout_val_freq_epochs=5 \
|
||||
train.rollout_num_episodes=10 \
|
||||
train.rollout_device=cuda \
|
||||
train.rollout_num_workers=4 \
|
||||
'train.rollout_cuda_devices=[0]' \
|
||||
train.rollout_validate_on_checkpoint=false \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.swanlab_run_name=<run_name> \
|
||||
hydra.run.dir=/home/droid/project/roboimi/runs/<run_name>
|
||||
```
|
||||
|
||||
### 8.8 重算统计文件
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/vla/scripts/calculate_stats.py \
|
||||
--dataset_dir /home/droid/project/diana_sim/sim_transfer
|
||||
```
|
||||
|
||||
### 8.9 监控日志
|
||||
|
||||
```bash
|
||||
tail -f runs/<run_name>/train_vla.log
|
||||
```
|
||||
|
||||
如果你显式设置了 `hydra.run.dir`，优先直接 tail 对应绝对路径下的日志文件。
|
||||
|
||||
---
|
||||
|
||||
## 9. 操作建议
|
||||
|
||||
- **优先固定 `hydra.run.dir`**，避免输出目录分散在 Hydra 自动生成路径里
|
||||
- 如果只需要常规验证，使用：
|
||||
- `train.val_split > 0`
|
||||
- 如果需要稳定、可复现的 held-out 指标，使用：
|
||||
- `train.val_split=0.0`
|
||||
- `train.val_episode_indices=[...]`
|
||||
- `train.action_mse_val_freq_epochs=1`
|
||||
- `train.num_workers` 是 DataLoader worker，不等于 rollout 并行度
|
||||
- rollout 并行优先看：
|
||||
- `train.rollout_num_workers`
|
||||
- `train.rollout_cuda_devices`
|
||||
- `eval.num_workers`
|
||||
- `eval.cuda_devices`
|
||||
- 数据集更新后记得重算 `dataset_stats.pkl`
|
||||
- 若同时设置了：
|
||||
- `train.pretrained_ckpt`
|
||||
- `train.resume_ckpt`
|
||||
当前逻辑会优先走 `resume_ckpt`
|
||||
- 建议 run name 中带上关键信息：
|
||||
- agent / horizon / batch size / lr / host / gpu / date
|
||||
- `experiment_suites/` 下的历史目录适合作为参考资料，但不要默认把某一条旧 suite 当作当前主流程的唯一标准
|
||||
@@ -0,0 +1,256 @@
|
||||
# Align ResNet Transformer Diffusion To External Repo Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan task-by-task. Do not switch to inline execution. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 将当前仓库的 `resnet_transformer` 对齐到 external repo `/home/droid/project/diffusion_policy` 的原生 DDPM Transformer Diffusion（非 PMF、非 DiT、非 UNet）实现，并采用 external 中已存在的 full-attention / nocausal 变体（`causal_attn=false`），固定为三相机图像条件输入，同时保持 EE action 语义的推理执行路径正确。
|
||||
|
||||
**Architecture:** 保留当前仓库较轻量的训练脚本和数据集组织，但把 Transformer denoiser 本体对齐到 external repo 的 `TransformerForDiffusion` 实现与接口语义；视觉编码器保留当前仓库的 ResNet+SpatialSoftmax 路线，仅保证其输出维度与三相机条件输入兼容。评估路径统一按 EE action 调 `env.step(action)`，训练/推理配置显式固定三相机 `r_vis/top/front` 且图像永远作为条件输入。
|
||||
|
||||
**Tech Stack:** Python, PyTorch, diffusers DDPM/DDIM, Hydra/OmegaConf, unittest, h5py, OpenCV
|
||||
|
||||
---
|
||||
|
||||
### Task 0: 执行前提与分支约束
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: Confirm execution stays on the feature branch**
|
||||
|
||||
Run: `git branch --show-current`
|
||||
Expected: `feat-align-dp-transformer-ee`
|
||||
|
||||
If the current branch is not `feat-align-dp-transformer-ee`, create/switch to it before continuing:
|
||||
|
||||
Run: `git checkout -b feat-align-dp-transformer-ee || git checkout feat-align-dp-transformer-ee`
|
||||
Expected: working branch becomes `feat-align-dp-transformer-ee`
|
||||
|
||||
- [ ] **Step 2: Confirm implementation is executed with subagents**
|
||||
|
||||
Use `superpowers:subagent-driven-development` for implementation tasks and reviews. Do not switch to inline/manual execution unless the human explicitly changes course.
|
||||
|
||||
- [ ] **Step 3: Confirm the current branch is the only target branch for this work**
|
||||
|
||||
Do not create or switch to another branch/worktree during implementation unless the human explicitly asks for it. All changes for this migration stay on `feat-align-dp-transformer-ee`.
|
||||
|
||||
### Task 1: 验证 EE action 推理执行语义已固定
|
||||
|
||||
**Files:**
|
||||
- Verify: `roboimi/vla/eval_utils.py`
|
||||
- Verify: `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
- Verify: `tests/test_eval_vla_execution.py`
|
||||
|
||||
- [ ] **Step 1: Verify the test still passes on the feature branch**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_eval_vla_execution -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 2: Verify the real eval script no longer executes joint-action stepping**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python - <<'PY'
|
||||
from pathlib import Path
|
||||
src = Path('roboimi/demos/vla_scripts/eval_vla.py').read_text()
|
||||
assert 'execute_policy_action(env, action)' in src
|
||||
assert 'env.step_jnt(action)' not in src
|
||||
print('eval_vla execution path verified')
|
||||
PY
|
||||
```
|
||||
Expected: PASS
|
||||
|
||||
### Task 2: 通过对拍测试把当前 Transformer head 对齐到 external repo 的 `TransformerForDiffusion`
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/models/heads/transformer1d.py`
|
||||
- Modify: `roboimi/vla/conf/head/transformer1d.yaml`
|
||||
- Test: `tests/test_transformer1d_external_alignment.py`
|
||||
|
||||
- [ ] **Step 1: Confirm the alignment test exists and captures the intended parity contract**
|
||||
|
||||
Write a test that imports external repo's `TransformerForDiffusion` from:
|
||||
`/home/droid/project/diffusion_policy/diffusion_policy/model/diffusion/transformer_for_diffusion.py`
|
||||
and verifies the local `Transformer1D` can load the external model's `state_dict` and produce numerically identical outputs for the same inputs when configured equivalently. Use the full-attention / nocausal configuration (`causal_attn=False`) as the target behavior.
|
||||
|
||||
Required assertions:
|
||||
- same parameter key structure (or compatible `load_state_dict(strict=True)`)
|
||||
- same output shape `(B, T, action_dim)`
|
||||
- `torch.allclose(local_out, external_out, atol=1e-6, rtol=1e-5)`
|
||||
- local model exposes `get_optim_groups(weight_decay=...)` like external repo
|
||||
- use an explicit import path / loader that does not depend on installing the external repo as a package
|
||||
- account for external `ModuleAttrMixin`-style optimizer grouping expectations, including `_dummy_variable` / no-decay bookkeeping if needed for strict state-dict parity
|
||||
- set both models to `eval()` and fix the random seed inside the test so dropout does not create false mismatches
|
||||
|
||||
- [ ] **Step 2: If the test still fails on this branch, use that red state as the TDD starting point**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_transformer1d_external_alignment -v`
|
||||
Expected: either FAIL before implementation on a fresh checkout, or PASS if Task 2 has already been completed on this branch.
|
||||
|
||||
- [ ] **Step 3a: Port API and state-dict compatibility first**
|
||||
|
||||
Match constructor args, parameter names, embeddings, masks, and strict `state_dict` layout with external `TransformerForDiffusion`.
|
||||
|
||||
- [ ] **Step 3b: Port `_init_weights` and optimizer grouping**
|
||||
|
||||
Match external `_init_weights`, `get_optim_groups`, and `configure_optimizers`.
|
||||
|
||||
- [ ] **Step 3c: Port forward semantics**
|
||||
|
||||
Match external `forward(sample, timestep, cond)` behavior under the full-attention / nocausal configuration.
|
||||
|
||||
- [ ] **Step 3d: Port the minimal external implementation**
|
||||
|
||||
Update `roboimi/vla/models/heads/transformer1d.py` so it matches external repo's native DDPM transformer implementation semantics:
|
||||
- same constructor arguments and defaults where relevant
|
||||
- same token accounting (`time_as_cond`, `obs_as_cond`, `T_cond`)
|
||||
- same parameter naming/layout for embeddings, encoder, decoder, masks
|
||||
- same `_init_weights`
|
||||
- same `get_optim_groups` / `configure_optimizers`
|
||||
- same `forward(sample, timestep, cond)` behavior
|
||||
|
||||
Do **not** port PMF / IMF / DiT branches.
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_transformer1d_external_alignment -v`
|
||||
Expected: PASS with strict state-dict load and matching outputs.
|
||||
|
||||
### Task 3: 将当前 Agent 与配置固定到“三相机图像作为条件”的 Transformer diffusion 路线
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/agent.py`
|
||||
- Modify: `roboimi/vla/conf/agent/resnet_transformer.yaml`
|
||||
- Modify: `roboimi/vla/conf/data/simpe_robot_dataset.yaml`
|
||||
- Modify: `roboimi/vla/conf/eval/eval.yaml`
|
||||
- Modify: `roboimi/vla/models/backbones/resnet_diffusion.py`
|
||||
- Test: `tests/test_resnet_transformer_agent_wiring.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Write a wiring test that instantiates the Transformer agent config and checks:
|
||||
- `head_type == "transformer"`
|
||||
- `cfg.data.camera_names == cfg.eval.camera_names == ["r_vis", "top", "front"]`
|
||||
- `num_cams == 3`
|
||||
- Transformer `cond_dim == single_cam_feat_dim * 3 + obs_dim`
|
||||
- `predict_action(...)` accepts image conditions and returns `(B, pred_horizon, action_dim)`
|
||||
- test setup must not download weights; override `pretrained_backbone_weights=null` or stub the backbone
|
||||
- assert camera order used for conditioning is tied to the required three cameras, not a stray constant
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_resnet_transformer_agent_wiring -v`
|
||||
Expected: FAIL if configs/head wiring do not yet guarantee three-camera conditional setup.
|
||||
|
||||
- [ ] **Step 3: Implement minimal alignment**
|
||||
|
||||
Make the transformer path explicit and stable:
|
||||
- keep image observations always as condition (`cond_dim > 0`, `obs_as_cond` semantics)
|
||||
- keep exactly three cameras: `r_vis`, `top`, `front`
|
||||
- keep current ResNet+SpatialSoftmax backbone unless a test proves incompatibility
|
||||
- make the camera feature order deterministic and aligned to the required three-camera list, not generic key sorting
|
||||
- ensure `agent.per_step_cond_dim` and config `head.cond_dim` stay consistent
|
||||
- ensure eval config uses the same three camera names as training
|
||||
- ensure transformer config follows the external full-attention variant (`causal_attn=false`) instead of the external default causal setting
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_resnet_transformer_agent_wiring -v`
|
||||
Expected: PASS
|
||||
|
||||
### Task 4: 让训练脚本在 Transformer 路线上尽量遵循 external repo 的 optimizer/head 使用方式
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Test: `tests/test_train_vla_transformer_optimizer.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
Write a test that builds a transformer agent and verifies the training script prefers the head/model supplied optimizer grouping when available (via `get_optim_groups`) instead of blindly using one flat `AdamW(agent.parameters(), ...)`.
|
||||
The test must also prove that every remaining trainable non-head parameter is included exactly once in the optimizer (no silent drops, no duplicates).
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_train_vla_transformer_optimizer -v`
|
||||
Expected: FAIL because current training script uses flat optimizer construction.
|
||||
|
||||
- [ ] **Step 3: Implement minimal optimizer alignment**
|
||||
|
||||
Update `train_vla.py` so that:
|
||||
- for transformer head paths, if `agent.noise_pred_net` exposes `get_optim_groups`, build optimizer groups like external repo
|
||||
- explicitly include the remaining trainable non-head parameters (for example the ResNet backbone's non-frozen projection / pooling layers) in the optimizer instead of accidentally dropping them
|
||||
- keep the rest of the simple training loop intact (no EMA unless required later)
|
||||
- do not over-port external workspace abstractions
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `mamba run -n roboimi python -m unittest tests.test_train_vla_transformer_optimizer -v`
|
||||
Expected: PASS
|
||||
|
||||
### Task 5: 端到端实例化与最小推理验收
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: Run all focused unit tests**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
mamba run -n roboimi python -m unittest \
|
||||
tests.test_eval_vla_execution \
|
||||
tests.test_transformer1d_external_alignment \
|
||||
tests.test_resnet_transformer_agent_wiring \
|
||||
tests.test_train_vla_transformer_optimizer -v
|
||||
```
|
||||
Expected: all PASS
|
||||
|
||||
- [ ] **Step 2: Run syntax checks on changed training/eval/model files**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
mamba run -n roboimi python -m py_compile \
|
||||
roboimi/vla/eval_utils.py \
|
||||
roboimi/vla/models/heads/transformer1d.py \
|
||||
roboimi/vla/agent.py \
|
||||
roboimi/demos/vla_scripts/train_vla.py \
|
||||
roboimi/demos/vla_scripts/eval_vla.py
|
||||
```
|
||||
Expected: no syntax errors
|
||||
|
||||
- [ ] **Step 3: Run a local instantiation smoke test**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
import torch
|
||||
from hydra import compose, initialize_config_dir
|
||||
from hydra.utils import instantiate
|
||||
from pathlib import Path
|
||||
|
||||
config_dir = str((Path.cwd() / "roboimi" / "vla" / "conf").resolve())
|
||||
with initialize_config_dir(version_base=None, config_dir=config_dir):
|
||||
cfg = compose(config_name="config", overrides=[
|
||||
"agent=resnet_transformer",
|
||||
"agent.vision_backbone.pretrained_backbone_weights=null",
|
||||
])
|
||||
|
||||
agent = instantiate(cfg.agent, dataset_stats=None)
|
||||
images = {
|
||||
"r_vis": torch.rand(1, cfg.agent.obs_horizon, 3, 224, 224),
|
||||
"top": torch.rand(1, cfg.agent.obs_horizon, 3, 224, 224),
|
||||
"front": torch.rand(1, cfg.agent.obs_horizon, 3, 224, 224),
|
||||
}
|
||||
qpos = torch.rand(1, cfg.agent.obs_horizon, cfg.agent.obs_dim)
|
||||
out = agent.predict_action(images, qpos)
|
||||
assert out.shape == (1, cfg.agent.pred_horizon, cfg.agent.action_dim), out.shape
|
||||
print("smoke_shape=", out.shape)
|
||||
PY
|
||||
```
|
||||
|
||||
that:
|
||||
- instantiates `agent=resnet_transformer`
|
||||
- constructs fake three-camera input (`r_vis`, `top`, `front`)
|
||||
- calls `agent.predict_action(...)`
|
||||
- verifies output shape is `(1, pred_horizon, 16)`
|
||||
|
||||
Expected: PASS
|
||||
@@ -0,0 +1,202 @@
|
||||
# VLA Experiment Sweep Execution Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Probe the largest safe shared batch size, prepare the 10 unique VLA sweep runs, and launch the serial GPU experiment queue with SwanLab logging and 20-epoch headless rollout validation.
|
||||
|
||||
**Architecture:** Use the existing `train_vla.py` / `eval_vla.py` path without changing model code unless launch-blocking issues appear. A small external launcher contract will manage per-run directories, logs, pids, and serial execution while Hydra places trainer-local `checkpoints/` under each run directory.
|
||||
|
||||
**Tech Stack:** zsh, mamba env `roboimi`, Hydra overrides, PyTorch CUDA, SwanLab, existing `train_vla.py`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Reconfirm schedule math and current runtime assumptions
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/superpowers/specs/2026-03-30-vla-experiment-sweep-design.md` (only if execution reveals a spec mismatch)
|
||||
- Test: none
|
||||
|
||||
- [ ] **Step 1: Print dataset size and epoch math for the baseline batch size**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python - <<'PY'
|
||||
num_samples = 70000
|
||||
for batch_size in (32, 48, 64, 80):
|
||||
steps_per_epoch = num_samples // batch_size
|
||||
print(batch_size, steps_per_epoch, 20 * steps_per_epoch, 200 * steps_per_epoch)
|
||||
PY
|
||||
```
|
||||
|
||||
Expected: integer epoch-step mappings for candidate batch sizes.
|
||||
|
||||
- [ ] **Step 2: Reconfirm no stale training process is already active**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
ps -ef | grep -E 'train_vla.py|python .*roboimi/demos/vla_scripts/train_vla.py' | grep -v grep || true
|
||||
```
|
||||
|
||||
Expected: no active training process unless intentionally started by this session.
|
||||
|
||||
- [ ] **Step 3: Record the execution assumptions**
|
||||
|
||||
Capture in session notes:
|
||||
- dataset path
|
||||
- camera list
|
||||
- current branch
|
||||
- no active conflicting training process
|
||||
|
||||
|
||||
### Task 2: Probe the largest safe shared batch size
|
||||
|
||||
**Files:**
|
||||
- Create: `runs/batch-probe-<timestamp>/` (runtime artifact)
|
||||
- Test: probe command output only
|
||||
|
||||
- [ ] **Step 1: Write the candidate probe command for the largest model**
|
||||
|
||||
Use overrides:
|
||||
- `data.dataset_dir=/home/droid/project/diana_sim/sim_transfer`
|
||||
- `data.camera_names=[r_vis,top,front]`
|
||||
- `agent.head.n_emb=384`
|
||||
- `agent.head.n_layer=12`
|
||||
- `agent.head.n_head=4`
|
||||
- `agent.vision_backbone.pretrained_backbone_weights=IMAGENET1K_V1`
|
||||
- `agent.vision_backbone.freeze_backbone=true`
|
||||
- `agent.vision_backbone.use_separate_rgb_encoder_per_camera=true`
|
||||
- `train.device=cuda`
|
||||
- `train.val_split=0.0`
|
||||
- `train.seed=42`
|
||||
- `train.use_swanlab=false`
|
||||
- `train.rollout_val_freq_epochs=0`
|
||||
- `train.rollout_validate_on_checkpoint=false`
|
||||
- `train.max_steps=4`
|
||||
|
||||
- [ ] **Step 2: Run the probe for `batch_size=32`**
|
||||
|
||||
Expected pass condition:
|
||||
- no CUDA OOM
|
||||
- no crash/abort
|
||||
- no NaN/Inf loss
|
||||
- 4 steps complete
|
||||
|
||||
- [ ] **Step 3: Repeat for `batch_size=48`, `64`, then optionally `80`**
|
||||
|
||||
Stop increasing once one candidate fails.
|
||||
|
||||
- [ ] **Step 4: Compute the shared LR and 200-epoch step budget**
|
||||
|
||||
Using:
|
||||
```text
|
||||
lr = 1e-4 * (batch_size / 16)
|
||||
max_steps = 200 * floor(70000 / batch_size)
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Write down the chosen shared batch size and derived max_steps**
|
||||
|
||||
This becomes the execution contract for all runs.
|
||||
|
||||
|
||||
### Task 3: Materialize the 10 unique run matrix
|
||||
|
||||
**Files:**
|
||||
- Create: `runs/vla-sweep-<timestamp>/manifest.txt` (runtime artifact)
|
||||
- Create: `runs/vla-sweep-<timestamp>/launch_queue.sh` (runtime artifact)
|
||||
- Test: shell syntax / manifest inspection
|
||||
|
||||
- [ ] **Step 1: Enumerate the 2-point pretraining comparison**
|
||||
|
||||
Canonical runs:
|
||||
1. `sim-transfer-baseline-pretrain-on-emb128-layer4`
|
||||
2. `sim-transfer-pretrain-off`
|
||||
|
||||
- [ ] **Step 2: Enumerate the remaining 8 unique architecture-sweep runs**
|
||||
|
||||
Skip the duplicate `(pretrain-on, emb128, layer4)` because it reuses the baseline run.
|
||||
|
||||
- [ ] **Step 3: For each run, define exact Hydra overrides**
|
||||
|
||||
Every run must pin:
|
||||
- dataset path
|
||||
- three cameras
|
||||
- `train.device=cuda`
|
||||
- `train.val_split=0.0`
|
||||
- `train.num_workers=12`
|
||||
- `train.seed=42`
|
||||
- `train.rollout_val_freq_epochs=20`
|
||||
- `train.rollout_validate_on_checkpoint=false`
|
||||
- `train.rollout_num_episodes=3`
|
||||
- `train.use_swanlab=true`
|
||||
- a unique `train.swanlab_run_name`
|
||||
- shared batch size
|
||||
- shared LR
|
||||
- derived `max_steps`
|
||||
- `agent.head.n_head=4`
|
||||
- `agent.vision_backbone.freeze_backbone=true`
|
||||
- `agent.vision_backbone.use_separate_rgb_encoder_per_camera=true`
|
||||
|
||||
- [ ] **Step 4: Write the queue launcher script**
|
||||
|
||||
Launcher responsibilities:
|
||||
- create run directory
|
||||
- export `LD_LIBRARY_PATH` CUDA/cuDNN fixup
|
||||
- export `MPLCONFIGDIR=/tmp/mpl`
|
||||
- set `hydra.job.chdir=true`
|
||||
- set `hydra.run.dir=<run_dir>`
|
||||
- tee logs to `<run_dir>/train.log`
|
||||
- write `<run_dir>/train.pid`
|
||||
- run jobs serially
|
||||
|
||||
- [ ] **Step 5: Verify the queue script syntax**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
zsh -n runs/vla-sweep-<timestamp>/launch_queue.sh
|
||||
```
|
||||
|
||||
Expected: exit 0.
|
||||
|
||||
|
||||
### Task 4: Launch the first probe-selected production run and verify startup
|
||||
|
||||
**Files:**
|
||||
- Create: `runs/vla-sweep-<timestamp>/<run-name>/` (runtime artifact)
|
||||
- Test: live log inspection / process inspection
|
||||
|
||||
- [ ] **Step 1: Start the first run in the queue**
|
||||
|
||||
Use the generated launcher contract.
|
||||
|
||||
- [ ] **Step 2: Inspect the first 100-200 log lines**
|
||||
|
||||
Expected:
|
||||
- config resolves correctly
|
||||
- dataset loads
|
||||
- agent initializes on CUDA
|
||||
- training loop starts
|
||||
- SwanLab initializes
|
||||
|
||||
- [ ] **Step 3: Confirm process identity and run directory**
|
||||
|
||||
Collect:
|
||||
- PID
|
||||
- run dir
|
||||
- log path
|
||||
- SwanLab run name
|
||||
|
||||
- [ ] **Step 4: If startup is clean, continue the remaining serial queue**
|
||||
|
||||
If startup fails, stop and debug before launching more runs.
|
||||
|
||||
|
||||
### Task 5: Report the launched sweep contract back to the user
|
||||
|
||||
**Files:**
|
||||
- Test: log evidence only
|
||||
|
||||
- [ ] **Step 1: Summarize the chosen batch size, LR, max_steps, and rollout cadence**
|
||||
|
||||
- [ ] **Step 2: Provide the active run directory / pid / first-run status**
|
||||
|
||||
- [ ] **Step 3: State the remaining queued runs by name**
|
||||
@@ -0,0 +1,454 @@
|
||||
# VLA Training + Headless Rollout + SwanLab Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 补齐当前 `Transformer1D` 训练依赖，在 `/home/droid/project/diana_sim/sim_transfer` 上启动训练，接入 SwanLab 标量日志，并提供训练期可选的 headless rollout validation 路径。
|
||||
|
||||
**Architecture:** 保持现有 `train_vla.py` / `eval_vla.py` 主体不变，只做最小必要改造：补依赖、补 stats 生成入口、在训练脚本里加轻量 SwanLab logger 和可选 checkpoint-time rollout wrapper、在环境侧把图像更新与 GUI 显示解耦。训练默认仍走当前 `resnet_transformer + Transformer1D` 路线，rollout validation 作为薄封装默认关闭。
|
||||
|
||||
**Tech Stack:** Python, mamba/conda, pip, PyTorch, Hydra, diffusers, torchvision, einops, SwanLab, MuJoCo, OpenCV, unittest
|
||||
|
||||
---
|
||||
|
||||
### Task 0: 执行前提与分支/环境确认
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: 确认当前分支仍是目标分支**
|
||||
|
||||
Run: `git branch --show-current`
|
||||
Expected: `feat-align-dp-transformer-ee`
|
||||
|
||||
- [ ] **Step 2: 记录当前 Python 解释器与环境名**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
import sys
|
||||
print(sys.executable)
|
||||
PY`
|
||||
Expected: 输出 `/home/droid/.conda/envs/roboimi/bin/python`
|
||||
|
||||
- [ ] **Step 3: 记录当前数据集目录存在性与 episode 数量**
|
||||
|
||||
Run: `/usr/bin/zsh -lc 'echo DATASET=/home/droid/project/diana_sim/sim_transfer; find /home/droid/project/diana_sim/sim_transfer -maxdepth 1 -name "episode_*.hdf5" | wc -l'`
|
||||
Expected: 输出目录路径与 `100`
|
||||
|
||||
### Task 1: 补齐训练依赖并把 resolved versions 写回环境定义
|
||||
|
||||
**Files:**
|
||||
- Modify: `environment.yml`
|
||||
- Verify only: local `roboimi` env
|
||||
|
||||
- [ ] **Step 1: 写出缺失依赖的最小清单**
|
||||
|
||||
需要补齐：
|
||||
- `diffusers`
|
||||
- `torchvision`
|
||||
- `einops`
|
||||
- `swanlab`
|
||||
|
||||
- [ ] **Step 2: 先用 dry-run 解析候选版本，确认不会升级 Torch**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python -m pip install --dry-run \
|
||||
diffusers torchvision einops swanlab
|
||||
```
|
||||
Expected: 输出候选版本；若显示会升级/替换 `torch`，则停止并改用显式兼容版本
|
||||
|
||||
- [ ] **Step 3: 安装与当前 Torch 兼容的缺失依赖到现有环境**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python -m pip install \
|
||||
diffusers torchvision einops swanlab
|
||||
```
|
||||
Expected: 安装成功，且不替换当前 `torch==2.4.0`
|
||||
|
||||
- [ ] **Step 4: 运行 import 验证**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
mods=['torch','hydra','omegaconf','diffusers','torchvision','einops','cv2','h5py','swanlab','mujoco']
|
||||
for m in mods:
|
||||
__import__(m)
|
||||
print('OK', m)
|
||||
PY
|
||||
```
|
||||
Expected: 每个模块都输出 `OK <module>`
|
||||
|
||||
- [ ] **Step 5: 记录实际安装版本**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
import diffusers, torchvision, einops, swanlab
|
||||
print('diffusers', getattr(diffusers,'__version__',''))
|
||||
print('torchvision', getattr(torchvision,'__version__',''))
|
||||
print('einops', getattr(einops,'__version__',''))
|
||||
print('swanlab', getattr(swanlab,'__version__',''))
|
||||
PY
|
||||
```
|
||||
Expected: 输出四个包的 resolved versions
|
||||
|
||||
- [ ] **Step 6: 将 resolved versions 写回 `environment.yml`**
|
||||
|
||||
把新增依赖补到 `environment.yml` 的现有依赖列表（若使用 `pip:` 段则更新该段）里，使用 Step 5 得到的**实际 resolved versions**，避免环境漂移，并避免重复 package 条目。
|
||||
|
||||
- [ ] **Step 7: 语法检查环境定义文件仅作结构确认**
|
||||
|
||||
Run: `python - <<'PY'
|
||||
from pathlib import Path
|
||||
text = Path('environment.yml').read_text()
|
||||
assert 'diffusers' in text
|
||||
assert 'torchvision' in text
|
||||
assert 'einops' in text
|
||||
assert 'swanlab' in text
|
||||
print('environment.yml updated')
|
||||
PY`
|
||||
Expected: `environment.yml updated`
|
||||
|
||||
### Task 2: 让统计脚本支持外部数据目录并生成 dataset stats
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/scripts/calculate_stats.py`
|
||||
- Test: `tests/test_calculate_stats_cli.py`
|
||||
|
||||
- [ ] **Step 1: 写 failing test，验证统计脚本可接受外部 `--dataset_dir` 并输出目标路径**
|
||||
|
||||
Test file should:
|
||||
- 用临时目录创建最小 HDF5 episode
|
||||
- 调用脚本入口/函数时传入外部目录
|
||||
- 断言输出 `dataset_stats.pkl` 出现在该目录
|
||||
- 断言 pickle 内包含 `action_mean/qpos_mean/...`
|
||||
|
||||
- [ ] **Step 2: 跑测试确认它先失败**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_calculate_stats_cli -v`
|
||||
Expected: FAIL（当前脚本写死默认目录）
|
||||
|
||||
- [ ] **Step 3: 最小实现 `--dataset_dir` 支持**
|
||||
|
||||
要求：
|
||||
- 保留现有统计逻辑
|
||||
- 仅增加 CLI 参数解析
|
||||
- 输出仍写入 `<dataset_dir>/dataset_stats.pkl`
|
||||
|
||||
- [ ] **Step 4: 重新跑测试确认转绿**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_calculate_stats_cli -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: 用真实数据集生成 stats**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/vla/scripts/calculate_stats.py \
|
||||
--dataset_dir /home/droid/project/diana_sim/sim_transfer
|
||||
```
|
||||
Expected: 生成 `/home/droid/project/diana_sim/sim_transfer/dataset_stats.pkl`
|
||||
|
||||
- [ ] **Step 6: 验证 stats 文件结构**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python - <<'PY'
|
||||
import pickle
|
||||
path='/home/droid/project/diana_sim/sim_transfer/dataset_stats.pkl'
|
||||
with open(path,'rb') as f:
|
||||
stats=pickle.load(f)
|
||||
for k in ['action_mean','action_std','action_min','action_max','qpos_mean','qpos_std','qpos_min','qpos_max']:
|
||||
assert k in stats, k
|
||||
print('stats_ok')
|
||||
PY
|
||||
```
|
||||
Expected: `stats_ok`
|
||||
|
||||
### Task 3: 增加 SwanLab 训练日志集成
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Modify: `roboimi/vla/conf/config.yaml`
|
||||
- Test: `tests/test_train_vla_swanlab_logging.py`
|
||||
|
||||
- [ ] **Step 1: 写 failing test，验证训练脚本在 `train.use_swanlab=true` 时会初始化 SwanLab 并记录标量**
|
||||
|
||||
Test should:
|
||||
- stub `swanlab`
|
||||
- 调用训练脚本中抽取出的非 Hydra helper（如 `_run_training(cfg)`）的最小路径（`max_steps=0` 或很小）
|
||||
- 断言调用了：
|
||||
- `swanlab.init(project='roboimi-vla', ...)`
|
||||
- 至少一次 `swanlab.log({...})`
|
||||
- 断言当 `use_swanlab=false` 时不会 import/初始化 SwanLab
|
||||
- 断言当 `use_swanlab=true` 且 import `swanlab` 失败时会 fail fast
|
||||
- 断言当 `use_swanlab=true` 且认证/登录状态不可用时会 fail fast
|
||||
- 断言会记录最终/最佳 checkpoint 路径到 log/summary
|
||||
|
||||
- [ ] **Step 2: 跑测试确认它先失败**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_train_vla_swanlab_logging -v`
|
||||
Expected: FAIL（当前无 SwanLab 集成）
|
||||
|
||||
- [ ] **Step 3: 在配置中增加最小 SwanLab 契约**
|
||||
|
||||
在 `roboimi/vla/conf/config.yaml` 添加：
|
||||
- `train.use_swanlab: true`
|
||||
- `train.swanlab_project: roboimi-vla`
|
||||
- 可选 `train.swanlab_run_name: null`
|
||||
|
||||
- [ ] **Step 4: 从 Hydra 入口提取可测试的训练 helper**
|
||||
|
||||
要求：
|
||||
- 新增类似 `_run_training(cfg)` 的普通函数
|
||||
- `main()` 只做 Hydra 入口转发
|
||||
- 测试只调用 helper，不直接调用 Hydra-decorated `main(cfg)`
|
||||
|
||||
- [ ] **Step 5: 在训练脚本中实现 SwanLab 初始化的 fail-fast 逻辑**
|
||||
|
||||
要求：
|
||||
- `use_swanlab=true` 时：
|
||||
- import 失败 -> 直接报错
|
||||
- 本地未登录/认证失败 -> 直接报错
|
||||
- 成功后执行 `swanlab.init(project=cfg.train.swanlab_project, ...)`
|
||||
|
||||
- [ ] **Step 6: 在训练脚本中实现轻量 scalar logger**
|
||||
|
||||
要求：
|
||||
- 仅 scalar logging，不引入自定义 callback 框架
|
||||
- 训练时记录 `train/loss`, `train/lr`, `train/best_loss`, `train/step`
|
||||
- 验证时记录 `val/loss`
|
||||
- 训练结束时记录：
|
||||
- `train/final_checkpoint_path`
|
||||
- `train/best_checkpoint_path`
|
||||
- 若库支持则显式 `finish/close`
|
||||
|
||||
- [ ] **Step 7: 重新跑测试确认转绿**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_train_vla_swanlab_logging -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 8: 用提供的 API key 完成本地登录**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/usr/bin/zsh -lc 'SWANLAB_API_KEY="<user-provided>"; /home/droid/.conda/envs/roboimi/bin/swanlab login -k "$SWANLAB_API_KEY"'
|
||||
```
|
||||
Expected: 登录成功并保存本地凭证
|
||||
|
||||
### Task 4: 为评估/rollout 路径增加 headless 模式
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/envs/double_base.py`
|
||||
- Modify: `roboimi/envs/double_pos_ctrl_env.py`
|
||||
- Modify: `roboimi/vla/conf/eval/eval.yaml`
|
||||
- Modify: `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
- Test: `tests/test_eval_vla_headless.py`
|
||||
|
||||
- [ ] **Step 1: 写 failing test，验证 headless 路径不触发 GUI 调用**
|
||||
|
||||
Test should stub:
|
||||
- `cv2.namedWindow`
|
||||
- `cv2.imshow`
|
||||
- `cv2.waitKey`
|
||||
- viewer launch/render path
|
||||
|
||||
And assert:
|
||||
- `eval.headless=true` 时不调用这些 GUI 接口
|
||||
- 仍能获取图像观测并走到 policy action 执行前/后关键路径
|
||||
|
||||
- [ ] **Step 2: 跑测试确认它先失败**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_eval_vla_headless -v`
|
||||
Expected: FAIL（当前 env/eval 默认会开 viewer 和 cv2 窗口）
|
||||
|
||||
- [ ] **Step 3: 统一配置开关为 `eval.headless`**
|
||||
|
||||
在 `roboimi/vla/conf/eval/eval.yaml` 添加：
|
||||
- `headless: false`
|
||||
|
||||
不要再引入第二个同义开关。
|
||||
|
||||
- [ ] **Step 4: 在 env 工厂中接入 `headless`**
|
||||
|
||||
在 `make_sim_env(...)` 中：
|
||||
- `eval.headless=true` -> `is_render=False`
|
||||
- 保留图像观测更新
|
||||
- 不创建 MuJoCo viewer
|
||||
|
||||
- [ ] **Step 5: 将相机更新与 GUI 显示解耦**
|
||||
|
||||
在 `double_base.py`：
|
||||
- 图像更新逻辑保留
|
||||
- `cv2.namedWindow/imshow/waitKey` 仅在非 headless 下执行
|
||||
|
||||
- [ ] **Step 6: eval 脚本中在 headless 下跳过 `env.render()`**
|
||||
|
||||
只在 `eval.headless=false` 时调用 `env.render()`。
|
||||
|
||||
- [ ] **Step 7: 重新跑测试确认转绿**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_eval_vla_headless -v`
|
||||
Expected: PASS
|
||||
|
||||
### Task 5: 给训练脚本加可选 checkpoint-time rollout validation 薄封装
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Possibly modify: `roboimi/demos/vla_scripts/eval_vla.py`（仅当需要提取可复用入口）
|
||||
- Modify: `roboimi/vla/conf/config.yaml`
|
||||
- Test: `tests/test_train_vla_rollout_validation.py`
|
||||
|
||||
- [ ] **Step 1: 写 failing test，验证 checkpoint 保存点可选调用 rollout validation，且会传 `eval.headless=true`**
|
||||
|
||||
Test should:
|
||||
- stub rollout/eval helper
|
||||
- 开启 `train.rollout_validate_on_checkpoint=true`
|
||||
- 设置小 `save_freq`
|
||||
- 断言训练脚本在 checkpoint 时调用验证 helper
|
||||
- 断言调用参数带 `headless=true`
|
||||
|
||||
- [ ] **Step 2: 跑测试确认它先失败**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_train_vla_rollout_validation -v`
|
||||
Expected: FAIL（当前无该 hook）
|
||||
|
||||
- [ ] **Step 3: 增加最小配置键**
|
||||
|
||||
在 `config.yaml` 中添加：
|
||||
- `train.rollout_validate_on_checkpoint: false`
|
||||
- `train.rollout_num_episodes: 1`
|
||||
|
||||
- [ ] **Step 4: 提取一个最小 rollout validation helper 接口**
|
||||
|
||||
要求：
|
||||
- helper 输入至少包括 `cfg`, `ckpt_path`, `num_episodes`, `headless`
|
||||
- 默认训练侧调用时强制 `headless=True`
|
||||
- 优先复用现有 eval 逻辑，不引入第二套 validator 类
|
||||
|
||||
- [ ] **Step 5: 在 checkpoint 保存路径中接入 rollout helper**
|
||||
|
||||
要求：
|
||||
- 不重写第二套验证框架
|
||||
- 优先复用现有 eval 逻辑/工具
|
||||
- 默认关闭
|
||||
- 仅在 checkpoint 时少量调用
|
||||
- 强制 `eval.headless=true`
|
||||
|
||||
- [ ] **Step 6: 重新跑测试确认转绿**
|
||||
|
||||
Run: `/home/droid/.conda/envs/roboimi/bin/python -m unittest tests.test_train_vla_rollout_validation -v`
|
||||
Expected: PASS
|
||||
|
||||
### Task 6: 启动训练前的集成 smoke verification
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: 跑所有新增/相关测试**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python -m unittest \
|
||||
tests.test_calculate_stats_cli \
|
||||
tests.test_train_vla_swanlab_logging \
|
||||
tests.test_eval_vla_headless \
|
||||
tests.test_train_vla_rollout_validation -v
|
||||
```
|
||||
Expected: 全部 PASS
|
||||
|
||||
- [ ] **Step 2: 对关键修改文件做语法检查**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python -m py_compile \
|
||||
roboimi/vla/scripts/calculate_stats.py \
|
||||
roboimi/demos/vla_scripts/train_vla.py \
|
||||
roboimi/demos/vla_scripts/eval_vla.py \
|
||||
roboimi/envs/double_pos_ctrl_env.py \
|
||||
roboimi/envs/double_base.py
|
||||
```
|
||||
Expected: 无语法错误
|
||||
|
||||
- [ ] **Step 3: 运行训练 smoke run**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
SWANLAB_API_KEY='<user-provided>' \
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.max_steps=20 \
|
||||
train.log_freq=1 \
|
||||
train.save_freq=10 \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.rollout_validate_on_checkpoint=false
|
||||
```
|
||||
Expected:
|
||||
- 训练启动成功
|
||||
- 产生 `checkpoints/vla_model_step_10.pt` 或 `vla_model_final.pt`
|
||||
- 本地日志中无 ImportError
|
||||
|
||||
- [ ] **Step 4: 运行一个最小 headless rollout-validation smoke run**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
SWANLAB_API_KEY='<user-provided>' \
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.max_steps=10 \
|
||||
train.log_freq=1 \
|
||||
train.save_freq=5 \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.rollout_validate_on_checkpoint=true \
|
||||
train.rollout_num_episodes=1 \
|
||||
eval.headless=true
|
||||
```
|
||||
Expected:
|
||||
- 到达 checkpoint-time rollout 调用
|
||||
- 不弹 MuJoCo viewer
|
||||
- 不执行 `cv2.namedWindow/imshow/waitKey`
|
||||
|
||||
- [ ] **Step 5: 验证 checkpoint 文件已写出**
|
||||
|
||||
Run: `/usr/bin/zsh -lc 'ls -lah checkpoints | sed -n "1,120p"'`
|
||||
Expected: 存在 `vla_model_step_10.pt` 或 `vla_model_final.pt`
|
||||
|
||||
- [ ] **Step 6: 验证 SwanLab 已收到标量**
|
||||
|
||||
验证方式：
|
||||
- 终端日志中确认 `swanlab.init` / run URL / run id
|
||||
- 若工具支持，确认 dashboard 中 `roboimi-vla` 项目下出现本次 run
|
||||
|
||||
### Task 7: 启动正式训练
|
||||
|
||||
**Files:**
|
||||
- Verify only
|
||||
|
||||
- [ ] **Step 1: 用真实配置启动正式训练**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
SWANLAB_API_KEY='<user-provided>' \
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.rollout_validate_on_checkpoint=true \
|
||||
eval.headless=true
|
||||
```
|
||||
Expected:
|
||||
- 训练持续运行
|
||||
- checkpoint 周期性写出
|
||||
- SwanLab 周期性收到 train/val 标量
|
||||
- 若 checkpoint-time rollout 打开，则不弹 GUI
|
||||
|
||||
- [ ] **Step 2: 记录启动信息并向用户汇报**
|
||||
|
||||
汇报内容至少包括：
|
||||
- 使用的数据集路径
|
||||
- 训练命令/关键 overrides
|
||||
- checkpoint 输出目录
|
||||
- SwanLab project 名称
|
||||
- rollout validation 是否已启用以及是否 headless
|
||||
@@ -0,0 +1,268 @@
|
||||
# IMF-AttnRes Policy Migration Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 将 external `diffusion_policy@185ed659` 的 IMF-AttnRes 模型、训练目标和一步推理机制迁移到 RoboIMI，并在保持三相机视觉条件输入与现有训练/rollout 工作流的前提下启动同参数训练。
|
||||
|
||||
**Architecture:** 保留 RoboIMI 现有 ResNet 三相机观测编码、normalization、queue-based online rollout 和训练脚本；新增 AttnRes 组件与 IMF transformer head，并新增 IMF 专用 agent 以覆盖 DDPM loss / DDIM inference 语义。训练脚本只做最小接线修改，让新 head/agent 能用现有 optimizer、checkpoint、SwanLab 和 headless rollout。
|
||||
|
||||
**Tech Stack:** PyTorch, Hydra, diffusers schedulers (仅保留兼容初始化), MuJoCo rollout, unittest, SwanLab
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
### New files
|
||||
- `roboimi/vla/models/heads/attnres_transformer_components.py` — 本地 IMF AttnRes 基础组件
|
||||
- `roboimi/vla/models/heads/imf_transformer1d.py` — IMF transformer head，暴露 `forward(sample, r, t, cond=None)`
|
||||
- `roboimi/vla/agent_imf.py` — IMF 专用 VLA agent，复用现有观测/队列/normalization 逻辑并覆盖 loss / inference
|
||||
- `roboimi/vla/conf/head/imf_transformer1d.yaml` — IMF head 配置
|
||||
- `roboimi/vla/conf/agent/resnet_imf_attnres.yaml` — IMF agent + backbone/head 组合配置
|
||||
- `tests/test_imf_transformer1d_external_alignment.py` — external `185ed659` 对齐测试
|
||||
- `tests/test_imf_vla_agent.py` — IMF agent 的 loss / inference / queue 语义测试
|
||||
|
||||
### Modified files
|
||||
- `roboimi/demos/vla_scripts/train_vla.py` — 优化器参数分组接线；确保新 agent 能无缝训练
|
||||
- `roboimi/vla/conf/config.yaml` — 保持默认配置不变，仅支持通过 override 启用 IMF agent
|
||||
- `tests/test_train_vla_transformer_optimizer.py` — 覆盖 IMF head 的 optimizer-group 行为
|
||||
- （如需要）`roboimi/vla/models/heads/__init__.py` 或相近导出文件 — 暴露新 head
|
||||
|
||||
---
|
||||
|
||||
### Task 1: 写 IMF transformer 对齐测试
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/test_imf_transformer1d_external_alignment.py`
|
||||
- Reference: `/home/droid/project/diffusion_policy/diffusion_policy/model/diffusion/attnres_transformer_components.py`
|
||||
- Reference: `/home/droid/project/diffusion_policy/diffusion_policy/model/diffusion/imf_transformer_for_diffusion.py`
|
||||
|
||||
- [ ] **Step 1: 写失败测试，验证 local IMF head 与 external `185ed659` 的 state-dict key、前向 shape、forward 数值、optim groups 对齐**
|
||||
|
||||
```python
|
||||
with torch.no_grad():
|
||||
external_out = external_model(sample=sample, r=r, t=t, cond=cond)
|
||||
local_out = local_model(sample=sample, r=r, t=t, cond=cond)
|
||||
assert torch.allclose(local_out, external_out, atol=1e-6, rtol=1e-5)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: 运行单测，确认当前失败**
|
||||
|
||||
Run: `python -m unittest tests.test_imf_transformer1d_external_alignment -v`
|
||||
Expected: FAIL，提示 `imf_transformer1d` / `attnres` 模块不存在
|
||||
|
||||
- [ ] **Step 3: 若测试需要复用现有 external-loader 逻辑，则从 `tests/test_transformer1d_external_alignment.py` 复制最小必要 helper，避免重复依赖 session context**
|
||||
|
||||
- [ ] **Step 4: 提交测试骨架**
|
||||
|
||||
```bash
|
||||
git add tests/test_imf_transformer1d_external_alignment.py
|
||||
git commit -m "test: add IMF transformer external alignment coverage"
|
||||
```
|
||||
|
||||
### Task 2: 实现 AttnRes 组件与 IMF transformer head
|
||||
|
||||
**Files:**
|
||||
- Create: `roboimi/vla/models/heads/attnres_transformer_components.py`
|
||||
- Create: `roboimi/vla/models/heads/imf_transformer1d.py`
|
||||
- Modify: `tests/test_imf_transformer1d_external_alignment.py`
|
||||
|
||||
- [ ] **Step 1: 按 external `185ed659` 迁移 AttnRes 基础组件，保持命名和参数语义一致**
|
||||
|
||||
必须包含：
|
||||
- `RMSNorm`
|
||||
- `RMSNormNoWeight`
|
||||
- `precompute_rope_freqs`
|
||||
- `apply_rope`
|
||||
- `GroupedQuerySelfAttention`
|
||||
- `SwiGLUFFN`
|
||||
- `AttnResOperator`
|
||||
- `AttnResSubLayer`
|
||||
- `AttnResTransformerBackbone`
|
||||
|
||||
- [ ] **Step 2: 在 `imf_transformer1d.py` 中实现本地 IMF head**
|
||||
|
||||
必须满足：
|
||||
- `forward(sample, r, t, cond=None)`
|
||||
- 默认支持 `backbone_type='attnres_full'`
|
||||
- token 序列为 `[r_token, t_token, cond_tokens..., sample_tokens...]`
|
||||
- 输出只切回 sample token 段
|
||||
- 保留 `get_optim_groups()` 供 AdamW 分组
|
||||
|
||||
- [ ] **Step 3: 运行对齐测试，修正 state-dict key / init / no-decay 参数分组不一致问题**
|
||||
|
||||
Run: `python -m unittest tests.test_imf_transformer1d_external_alignment -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 4: 提交模型组件实现**
|
||||
|
||||
```bash
|
||||
git add roboimi/vla/models/heads/attnres_transformer_components.py \
|
||||
roboimi/vla/models/heads/imf_transformer1d.py \
|
||||
tests/test_imf_transformer1d_external_alignment.py
|
||||
git commit -m "feat: add IMF AttnRes transformer head"
|
||||
```
|
||||
|
||||
### Task 3: 写 IMF agent 行为测试
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/test_imf_vla_agent.py`
|
||||
- Reference: `roboimi/vla/agent.py`
|
||||
- Reference: `tests/test_resnet_transformer_agent_wiring.py`
|
||||
|
||||
- [ ] **Step 1: 写失败测试，覆盖 IMF agent 的核心契约**
|
||||
|
||||
需要覆盖：
|
||||
1. `compute_loss()` 接受当前 batch 结构并返回标量 loss
|
||||
2. `predict_action()` 输出 `(B, pred_horizon, action_dim)`
|
||||
3. `select_action()` 仍按 queue/chunk 语义工作
|
||||
4. `predict_action()` 不走 DDIM 多步循环，而是只触发一步 IMF sample
|
||||
5. `action_is_pad` 存在时仅在有效 action 上计 loss
|
||||
|
||||
- [ ] **Step 2: 用 stub backbone / stub head 记录调用参数，验证 `r,t,cond` 的传递与 observation conditioning 维度正确**
|
||||
|
||||
```python
|
||||
self.assertEqual(recorded['cond'].shape, (B, obs_horizon, expected_cond_dim))
|
||||
self.assertTrue(torch.allclose(recorded['r'], torch.zeros(B)))
|
||||
self.assertTrue(torch.allclose(recorded['t'], torch.ones(B)))
|
||||
```
|
||||
|
||||
- [ ] **Step 3: 运行测试，确认当前失败**
|
||||
|
||||
Run: `python -m unittest tests.test_imf_vla_agent -v`
|
||||
Expected: FAIL，提示 `roboimi.vla.agent_imf` 不存在
|
||||
|
||||
- [ ] **Step 4: 提交测试骨架**
|
||||
|
||||
```bash
|
||||
git add tests/test_imf_vla_agent.py
|
||||
git commit -m "test: add IMF VLA agent behavior coverage"
|
||||
```
|
||||
|
||||
### Task 4: 实现 IMF agent 与 Hydra 接线
|
||||
|
||||
**Files:**
|
||||
- Create: `roboimi/vla/agent_imf.py`
|
||||
- Create: `roboimi/vla/conf/head/imf_transformer1d.yaml`
|
||||
- Create: `roboimi/vla/conf/agent/resnet_imf_attnres.yaml`
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Modify: `tests/test_train_vla_transformer_optimizer.py`
|
||||
- Modify: `tests/test_imf_vla_agent.py`
|
||||
|
||||
- [ ] **Step 1: 以 `VLAAgent` 为基础实现 `IMFVLAAgent`**
|
||||
|
||||
实现策略：
|
||||
- 复用 `VLAAgent.__init__`、`_build_cond()`、`reset()`、`_populate_queues()`、`_prepare_observation_batch()`、`select_action()`、`get_normalization_stats()`
|
||||
- 覆盖：
|
||||
- `compute_loss()` -> IMF objective
|
||||
- `predict_action()` -> one-step sample
|
||||
- 提供内部 helper：
|
||||
- `_broadcast_batch_time`
|
||||
- `_apply_conditioning`（如需）
|
||||
- `_compute_u_and_du_dt`
|
||||
- `_compound_velocity`
|
||||
- `_sample_one_step`
|
||||
|
||||
- [ ] **Step 2: 在 JVP 路径中加入 CUDA math SDPA fallback，保持 external repo 的稳定性策略**
|
||||
|
||||
- [ ] **Step 3: 新增 Hydra 配置，让 `agent=resnet_imf_attnres` 可实例化**
|
||||
|
||||
关键默认值：
|
||||
- `_target_: roboimi.vla.agent_imf.IMFVLAAgent`
|
||||
- `head._target_: roboimi.vla.models.heads.imf_transformer1d.IMFTransformer1D`
|
||||
- `head.backbone_type: attnres_full`
|
||||
- `head.causal_attn: false`
|
||||
- `head.time_as_cond: true`
|
||||
- `head.n_cond_layers: 0`
|
||||
- `inference_steps: 1`
|
||||
- `camera_names: ${data.camera_names}`
|
||||
- `vision_backbone.camera_names: ${agent.camera_names}`
|
||||
|
||||
- [ ] **Step 4: 让训练脚本对任何带 `get_optim_groups()` 的 head 复用参数分组，而不是硬编码旧 transformer head_type**
|
||||
|
||||
推荐最小改法：
|
||||
```python
|
||||
use_head_groups = callable(getattr(noise_pred_net, 'get_optim_groups', None))
|
||||
```
|
||||
|
||||
- [ ] **Step 5: 运行测试并修复 wiring 问题**
|
||||
|
||||
Run:
|
||||
- `python -m unittest tests.test_imf_vla_agent -v`
|
||||
- `python -m unittest tests.test_train_vla_transformer_optimizer -v`
|
||||
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 6: 提交 agent / config / train-script 接线**
|
||||
|
||||
```bash
|
||||
git add roboimi/vla/agent_imf.py \
|
||||
roboimi/vla/conf/head/imf_transformer1d.yaml \
|
||||
roboimi/vla/conf/agent/resnet_imf_attnres.yaml \
|
||||
roboimi/demos/vla_scripts/train_vla.py \
|
||||
tests/test_imf_vla_agent.py \
|
||||
tests/test_train_vla_transformer_optimizer.py
|
||||
git commit -m "feat: add IMF VLA agent and training wiring"
|
||||
```
|
||||
|
||||
### Task 5: 集成验证与训练启动
|
||||
|
||||
**Files:**
|
||||
- Modify: none required unless验证暴露真实问题
|
||||
- Use run artifacts under: `runs/`
|
||||
|
||||
- [ ] **Step 1: 运行聚焦测试集**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python -m unittest \
|
||||
tests.test_imf_transformer1d_external_alignment \
|
||||
tests.test_imf_vla_agent \
|
||||
tests.test_resnet_transformer_agent_wiring \
|
||||
tests.test_train_vla_transformer_optimizer -v
|
||||
```
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 2: 运行一个最小 GPU 训练冒烟任务（不必长跑）**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_imf_attnres \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
data.camera_names=[r_vis,top,front] \
|
||||
train.device=cuda train.max_steps=2 train.batch_size=4 train.num_workers=2 \
|
||||
train.use_swanlab=false train.rollout_val_freq_epochs=0
|
||||
```
|
||||
Expected: 成功完成 2 steps，生成 checkpoint / log，无 shape 或 JVP 错误
|
||||
|
||||
- [ ] **Step 3: 用正式参数启动 IMF 训练**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
/home/droid/.conda/envs/roboimi/bin/python roboimi/demos/vla_scripts/train_vla.py \
|
||||
agent=resnet_imf_attnres \
|
||||
data.dataset_dir=/home/droid/project/diana_sim/sim_transfer \
|
||||
data.camera_names=[r_vis,top,front] \
|
||||
train.device=cuda train.val_split=0.0 train.seed=42 \
|
||||
train.batch_size=80 train.lr=5e-4 train.num_workers=12 train.max_steps=150000 \
|
||||
train.log_freq=100 train.save_freq=10000 train.use_swanlab=true \
|
||||
train.swanlab_project=roboimi-vla \
|
||||
train.rollout_val_freq_epochs=5 train.rollout_validate_on_checkpoint=false \
|
||||
train.rollout_num_episodes=5 train.warmup_steps=2000 \
|
||||
train.scheduler_type=cosine train.min_lr=1e-6 train.weight_decay=1e-5 train.grad_clip=1.0 \
|
||||
agent.pred_horizon=16 agent.inference_steps=1 \
|
||||
agent.head.n_emb=384 agent.head.n_layer=18 agent.head.n_head=1 agent.head.n_kv_head=1 \
|
||||
agent.vision_backbone.pretrained_backbone_weights=null \
|
||||
agent.vision_backbone.freeze_backbone=false \
|
||||
agent.vision_backbone.use_separate_rgb_encoder_per_camera=true
|
||||
```
|
||||
Expected: 训练启动成功，SwanLab 记录完整 config，5 epoch 一次 headless rollout
|
||||
|
||||
- [ ] **Step 4: 记录 run 路径、训练 PID、SwanLab 运行名并向用户汇报**
|
||||
|
||||
- [ ] **Step 5: 提交最终收尾改动（如果 smoke fix 需要额外 patch）**
|
||||
|
||||
```bash
|
||||
git add <changed files>
|
||||
git commit -m "chore: verify IMF AttnRes training launch"
|
||||
```
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
# IMF Rollout Trajectory Images and Short-Horizon Training Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add training-time rollout front trajectory image export plus SwanLab image logging, then start a new local IMF training run with `emb=384`, `layer=12`, `pred_horizon=8`, `num_action_steps=4`, `max_steps=50000`.
|
||||
|
||||
**Architecture:** Extend `eval_vla.py` so a rollout can emit one per-episode static front-view image with red EE trajectory overlay. Extend `train_vla.py` so rollout validation forces image export, forces video off, and uploads those per-episode images to SwanLab. Launch the requested new run through explicit command-line overrides rather than branch-default config changes.
|
||||
|
||||
**Tech Stack:** Python, PyTorch, Hydra/OmegaConf, MuJoCo, OpenCV, SwanLab.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add and validate rollout image tests
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/test_eval_vla_rollout_artifacts.py`
|
||||
- Modify: `tests/test_train_vla_swanlab_logging.py`
|
||||
- Modify: `tests/test_train_vla_rollout_validation.py`
|
||||
|
||||
- [ ] Add/adjust eval tests so they assert per-episode trajectory image paths are produced without requiring video export.
|
||||
- [ ] Add/adjust training tests so they assert training-time rollout validation forces `record_video=false`.
|
||||
- [ ] Add/adjust training tests so they assert trajectory image paths flow from eval summary into SwanLab media logging.
|
||||
- [ ] Add/adjust training tests so they assert image media is logged, not only scalar reward metrics.
|
||||
|
||||
### Task 2: Implement per-episode front trajectory image export in eval
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
- Reuse/Read: `roboimi/utils/raw_action_trajectory_viewer.py`
|
||||
- Modify: `roboimi/vla/conf/eval/eval.yaml`
|
||||
|
||||
- [ ] Add config plumbing for `save_trajectory_image` and `trajectory_image_camera_name`.
|
||||
- [ ] Ensure the default training-time camera resolution path is pinned to `front`.
|
||||
- [ ] Implement distinct per-episode image naming so 5 rollout episodes create 5 distinct PNGs.
|
||||
- [ ] Reuse the existing red trajectory representation logic when composing the PNG.
|
||||
- [ ] Ensure headless eval works under EGL even on machines with `DISPLAY` set.
|
||||
|
||||
### Task 3: Implement SwanLab rollout image logging in training
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Modify: `tests/test_train_vla_swanlab_logging.py`
|
||||
- Modify: `tests/test_train_vla_rollout_validation.py`
|
||||
|
||||
- [ ] Make `run_rollout_validation()` force `record_video=false`.
|
||||
- [ ] Make `run_rollout_validation()` force `save_trajectory_image=true` and `trajectory_image_camera_name=front`.
|
||||
- [ ] Ensure rollout validation still uses 5 episodes per validation event for the requested run.
|
||||
- [ ] Add a best-effort helper that converts per-episode image paths into SwanLab image media payloads.
|
||||
- [ ] Keep image-upload failures non-fatal and warning-only.
|
||||
|
||||
### Task 4: Verify action-chunk semantics for the new run
|
||||
|
||||
**Files:**
|
||||
- Verify: `roboimi/vla/agent.py`
|
||||
- Verify: `roboimi/vla/agent_imf.py`
|
||||
- Test: `tests/test_imf_vla_agent.py`
|
||||
|
||||
- [ ] Confirm the existing queue logic still means “predict 8, execute first 4”.
|
||||
- [ ] Do not change branch defaults unless strictly necessary; prefer launch-time overrides.
|
||||
|
||||
### Task 5: Verify and launch the requested local training run
|
||||
|
||||
**Files:**
|
||||
- Use: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Use: `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
|
||||
- [ ] Run the targeted verification suite.
|
||||
- [ ] Run one real headless smoke eval and confirm a front trajectory PNG is produced while `video_mp4` stays null.
|
||||
- [ ] Launch the new local training run with explicit overrides including:
|
||||
- `agent=resnet_imf_attnres`
|
||||
- `agent.head.n_emb=384`
|
||||
- `agent.head.n_layer=12`
|
||||
- `agent.pred_horizon=8`
|
||||
- `agent.num_action_steps=4`
|
||||
- `train.max_steps=50000`
|
||||
- `train.rollout_num_episodes=5`
|
||||
- `train.use_swanlab=true`
|
||||
- current local baseline dataset/camera/CUDA/batch/lr/num_workers/backbone settings
|
||||
- [ ] Verify PID, GPU allocation, log tail, and SwanLab run URL.
|
||||
@@ -0,0 +1,68 @@
|
||||
# IMF Horizon Grid and AttnRes Ablation Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Run a 6-run Phase-1 IMF horizon/action-step experiment grid across available GPUs, monitor progress and collect best rollout metrics, then use the best horizon setting for a Phase-2 visual-attnres ablation.
|
||||
|
||||
**Architecture:** Use the current IMF training code as-is for Phase-1 by sweeping explicit `(pred_horizon, num_action_steps)` overrides while keeping emb=384, layer=12, and max_steps=50k fixed. Maintain a local experiment suite directory with a manifest and machine-readable status snapshots so progress can be resumed and summarized across turns. After Phase-1 completes, compare the current head-only attnres setup against a variant that also adds attnres into the visual ResNet path.
|
||||
|
||||
**Tech Stack:** Python, Hydra/OmegaConf, PyTorch, SSH/Tailscale, JSON/CSV status files, SwanLab.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Prepare the experiment suite manifest and state tracking
|
||||
|
||||
**Files:**
|
||||
- Create: `experiment_suites/2026-04-04-imf-horizon-grid/manifest.json`
|
||||
- Create: `experiment_suites/2026-04-04-imf-horizon-grid/status.json`
|
||||
- Create: `experiment_suites/2026-04-04-imf-horizon-grid/notes.md`
|
||||
|
||||
- [ ] Define the 6 legal Phase-1 combinations: `(8,8)`, `(16,8)`, `(16,16)`, `(32,8)`, `(32,16)`, `(32,32)`.
|
||||
- [ ] Record for each run: name, host, GPU slot, command, log path, SwanLab run name, and completion criteria.
|
||||
- [ ] Define the comparison metric as the maximum rollout average reward seen during training (`max avg_reward`), preferably read from the best-checkpoint metadata and cross-checked against logs.
|
||||
- [ ] Keep `status.json` updated with per-run state: queued / running / finished / failed plus latest parsed progress.
|
||||
|
||||
### Task 2: Prepare the remote 8-GPU execution target
|
||||
|
||||
**Files:**
|
||||
- Remote working directory under `/home/droid/`
|
||||
- Reuse or create a synced code directory for this suite
|
||||
|
||||
- [ ] Verify the remote dataset path and environment path.
|
||||
- [ ] Verify GPU availability and reserve 6 GPUs for Phase-1 launches.
|
||||
- [ ] Sync the required code to a dedicated remote suite directory.
|
||||
- [ ] Record exact remote paths back into the local suite manifest.
|
||||
|
||||
### Task 3: Launch the 6 Phase-1 experiments in parallel
|
||||
|
||||
**Files:**
|
||||
- Reuse: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- Modify only local suite tracking files unless a launch bug is discovered
|
||||
|
||||
- [ ] Launch 6 runs concurrently with fixed settings: IMF, emb=384, layer=12, max_steps=50k.
|
||||
- [ ] Keep all other relevant training hyperparameters aligned to the current strong baseline unless a concrete blocker appears.
|
||||
- [ ] Assign one GPU per run on the 8xL20 host.
|
||||
- [ ] Capture PID, log path, and SwanLab URL for each run in `status.json`.
|
||||
|
||||
### Task 4: Monitor and summarize Phase-1 until all 6 finish
|
||||
|
||||
**Files:**
|
||||
- Update: `experiment_suites/2026-04-04-imf-horizon-grid/status.json`
|
||||
- Update: `experiment_suites/2026-04-04-imf-horizon-grid/notes.md`
|
||||
|
||||
- [ ] Periodically parse each run’s log/checkpoints to extract latest step, latest rollout reward, and best rollout reward so far.
|
||||
- [ ] Keep a resumable local summary so progress can be continued in later turns without rediscovery.
|
||||
- [ ] After all 6 runs finish, rank them by `max avg_reward` and write a compact Phase-1 summary.
|
||||
|
||||
### Task 5: Prepare the Phase-2 visual-attnres ablation
|
||||
|
||||
**Files:**
|
||||
- Likely modify: vision backbone implementation and config files (to be confirmed after code inspection)
|
||||
- Add/update targeted tests for the visual backbone path if code changes are needed
|
||||
|
||||
- [ ] Use the best Phase-1 `(pred_horizon, num_action_steps)` combination as the fixed rollout setting for Phase-2.
|
||||
- [ ] Compare:
|
||||
1. current setup: attnres only in the IMF head
|
||||
2. ablation setup: attnres in both IMF head and visual encoder path
|
||||
- [ ] Keep the rest of the training settings fixed.
|
||||
- [ ] Launch and monitor the Phase-2 pair after Phase-1 summary is complete.
|
||||
@@ -0,0 +1,92 @@
|
||||
# LEWM ViT Backbone Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the current ResNet visual encoder in roboimi VLA training with a frozen LEWM ViT visual backbone (encoder + projector) that consumes the three camera views jointly and outputs one 192-d CLS embedding per timestep, then launch two 50k runs on the 5880 machine.
|
||||
|
||||
**Architecture:** Add a new joint-multiview LEWM backbone that fuses `front/top/r_vis` into one LEWM-style image, reproduces LEWM preprocessing, loads frozen weights from the trained checkpoint, and exposes a `joint_output_dim=192`. Add a minimal `VLAAgent` compatibility branch so conditions can be sized from joint visual dim instead of `output_dim * num_cams`, while leaving the rest of the diffusion pipeline unchanged.
|
||||
|
||||
**Tech Stack:** PyTorch, transformers `ViTModel`, Hydra configs, existing roboimi VLA training/eval scripts, remote SSH/rsync to 100.73.14.65.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add failing tests for LEWM joint-vision backbone contract
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/test_lewm_vit_backbone.py`
|
||||
- Modify: `tests/test_imf_vla_agent.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing backbone shape/load test**
|
||||
- [ ] **Step 2: Run `pytest tests/test_lewm_vit_backbone.py -q` and verify it fails**
|
||||
- [ ] **Step 3: Extend `tests/test_imf_vla_agent.py` with a failing joint-output backbone case**
|
||||
- [ ] **Step 4: Run `pytest tests/test_imf_vla_agent.py -q` and verify it fails**
|
||||
|
||||
### Task 2: Implement LEWM joint-multiview frozen backbone
|
||||
|
||||
**Files:**
|
||||
- Create: `roboimi/vla/models/backbones/lewm_vit_backbone.py`
|
||||
- Modify: `roboimi/vla/models/backbones/__init__.py` only if exports are needed
|
||||
|
||||
- [ ] **Step 1: Create `LEWMViTBackbone` with public attrs `camera_names`, `num_cameras`, `joint_output_dim=192`**
|
||||
- [ ] **Step 2: Reproduce LEWM preprocessing and joint multiview fusion**
|
||||
- [ ] **Step 3: Load checkpoint weights from `model.encoder.*` and `model.projector.*`**
|
||||
- [ ] **Step 4: Freeze encoder/projector and keep them in eval mode via `train()` override**
|
||||
- [ ] **Step 5: Run `pytest tests/test_lewm_vit_backbone.py -q` and verify green**
|
||||
|
||||
### Task 3: Add minimal agent support for joint visual dim
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/agent.py`
|
||||
- Test: `tests/test_imf_vla_agent.py`
|
||||
|
||||
- [ ] **Step 1: Add a `joint_output_dim` branch in `VLAAgent.__init__` for `per_step_cond_dim` / `global_cond_dim`**
|
||||
- [ ] **Step 2: Keep `_build_cond()` semantics unchanged except for matching the new dim contract**
|
||||
- [ ] **Step 3: Run `pytest tests/test_imf_vla_agent.py -q` and verify green**
|
||||
|
||||
### Task 4: Add Hydra configs for LEWM backbone training
|
||||
|
||||
**Files:**
|
||||
- Create: `roboimi/vla/conf/backbone/lewm_vit_diffusion.yaml`
|
||||
- Create: `roboimi/vla/conf/agent/lewm_imf_attnres.yaml`
|
||||
|
||||
- [ ] **Step 1: Add backbone config pointing to the new LEWM backbone**
|
||||
- [ ] **Step 2: Add `agent=lewm_imf_attnres` config with 3 cameras and `head.cond_dim=208`**
|
||||
- [ ] **Step 3: Verify Hydra instantiation with a one-shot compose smoke**
|
||||
|
||||
### Task 5: Verify focused local tests
|
||||
|
||||
**Files:**
|
||||
- Reuse the above
|
||||
|
||||
- [ ] **Step 1: Run `pytest tests/test_lewm_vit_backbone.py tests/test_imf_vla_agent.py tests/test_eval_vla_headless_import.py -q`**
|
||||
- [ ] **Step 2: If needed, run one tiny local import/forward smoke**
|
||||
|
||||
### Task 6: Sync to 5880 and remote smoke with real checkpoint
|
||||
|
||||
**Files:**
|
||||
- Remote target: `/home/droid/roboimi_suite_20260404`
|
||||
|
||||
- [ ] **Step 1: Rsync modified source/config files to `100.73.14.65:/home/droid/roboimi_suite_20260404`**
|
||||
- [ ] **Step 2: Run a 2-step smoke on GPU0 with `agent.head.n_emb=384`, `train.rollout_num_episodes=10`, real LEWM checkpoint**
|
||||
- [ ] **Step 3: Run a 2-step smoke on GPU1 with `agent.head.n_emb=256`, same checkpoint**
|
||||
|
||||
### Task 7: Launch two real 50k runs on the 5880 machine
|
||||
|
||||
**Files:**
|
||||
- Remote logs under `/home/droid/roboimi_suite_20260404/experiment_suite_launch_logs/`
|
||||
|
||||
- [ ] **Step 1: Launch embed384/layer12 on GPU0**
|
||||
- [ ] **Step 2: Launch embed256/layer12 on GPU1**
|
||||
- [ ] **Step 3: Ensure both use `data.camera_names=[r_vis,top,front]`, `pred_horizon=16`, `num_action_steps=8`, `train.rollout_num_episodes=10`, `max_steps=50000`**
|
||||
- [ ] **Step 4: Record run names, pids, log paths, SwanLab URLs**
|
||||
|
||||
### Task 8: Update experiment tracking docs and commit
|
||||
|
||||
**Files:**
|
||||
- Create: `experiment_suites/2026-04-05-lewm-vit-transfer/manifest.json`
|
||||
- Create: `experiment_suites/2026-04-05-lewm-vit-transfer/status.json`
|
||||
- Create: `experiment_suites/2026-04-05-lewm-vit-transfer/notes.md`
|
||||
|
||||
- [ ] **Step 1: Record checkpoint path, frozen LEWM design, rollout=10, and both run configs**
|
||||
- [ ] **Step 2: Record running status after launch**
|
||||
- [ ] **Step 3: Commit implementation + docs with a focused message**
|
||||
@@ -0,0 +1,64 @@
|
||||
# Phase-2 Full-AttnRes Vision Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace all ResNet residual units in the vision backbone with AttnRes-based image blocks while preserving the current IMF agent interfaces and launch a Phase-2 experiment anchored on the best Phase-1 horizon setting.
|
||||
|
||||
**Architecture:** Keep the current multi-camera encoder shell and per-camera output contract, but introduce a new ResNet-like 2D AttnRes backbone that preserves stage-wise downsampling and final SpatialSoftmax conditioning. Wire it into the existing `ResNetDiffusionBackbone` via an opt-in mode and keep the agent/head/data interfaces unchanged.
|
||||
|
||||
**Tech Stack:** PyTorch, Hydra/OmegaConf, existing IMF AttnRes transformer components, pytest.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add failing tests for the new full-AttnRes visual backbone
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/test_attnres_resnet2d_backbone.py`
|
||||
- Update: `tests/test_imf_vla_agent.py`
|
||||
|
||||
- [ ] **Step 1: Write a failing backbone shape test**
|
||||
- [ ] **Step 2: Run it to confirm the new backbone/config does not exist yet**
|
||||
- [ ] **Step 3: Add a failing IMF agent wiring test for unchanged cond_dim=208**
|
||||
- [ ] **Step 4: Run the targeted tests and capture the failure**
|
||||
|
||||
### Task 2: Implement a ResNet-like 2D AttnRes backbone
|
||||
|
||||
**Files:**
|
||||
- Create: `roboimi/vla/models/backbones/attnres_resnet2d.py`
|
||||
- Modify: `roboimi/vla/models/backbones/resnet_diffusion.py`
|
||||
|
||||
- [ ] **Step 1: Add minimal 2D tokenization helpers and positional encoding / bias handling**
|
||||
- [ ] **Step 2: Implement `AttnResImageBlock2D` for feature maps**
|
||||
- [ ] **Step 3: Implement `AttnResResNetLikeBackbone2D` with stage-wise downsampling**
|
||||
- [ ] **Step 4: Wire `_SingleRgbEncoder` to choose between original ResNet trunk and the new full-AttnRes trunk**
|
||||
- [ ] **Step 5: Run the new backbone tests**
|
||||
|
||||
### Task 3: Expose config switches and agent wiring
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/conf/backbone/resnet_diffusion.yaml`
|
||||
- Modify: `roboimi/vla/conf/agent/resnet_imf_attnres.yaml`
|
||||
|
||||
- [ ] **Step 1: Add a backbone mode/config flag for the full-AttnRes vision trunk**
|
||||
- [ ] **Step 2: Add defaults for attnres image depth/heads/etc. if needed**
|
||||
- [ ] **Step 3: Add a Phase-2 launch override path that enables the new visual trunk**
|
||||
- [ ] **Step 4: Run agent wiring tests again**
|
||||
|
||||
### Task 4: Smoke-verify training path
|
||||
|
||||
**Files:**
|
||||
- Reuse existing training scripts and configs
|
||||
|
||||
- [ ] **Step 1: Run a short CPU or tiny-step smoke instantiation / `compute_loss` test**
|
||||
- [ ] **Step 2: If needed, run a very short training smoke launch**
|
||||
- [ ] **Step 3: Verify no cond-dim or rollout-loading regressions**
|
||||
|
||||
### Task 5: Launch the Phase-2 experiment
|
||||
|
||||
**Files:**
|
||||
- Update experiment tracking under `experiment_suites/`
|
||||
|
||||
- [ ] **Step 1: Use Phase-1 best setting (`pred_horizon=16`, `num_action_steps=8`)**
|
||||
- [ ] **Step 2: Launch baseline reference or reuse existing result**
|
||||
- [ ] **Step 3: Launch full-AttnRes vision experiment**
|
||||
- [ ] **Step 4: Track rollout metrics and compare max avg_reward**
|
||||
@@ -0,0 +1,81 @@
|
||||
# ResNet Multitoken IMF Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Implement a standard-ResNet-18 multiview IMF variant that emits three condition tokens per obs step and launch four L20 experiments for `n_emb in {256,384}` and `n_layer in {12,16}`.
|
||||
|
||||
**Architecture:** The ResNet backbone will optionally return one token per camera instead of concatenating all cameras into one token. `VLAAgent` will pair each camera token with the current state, project each pair into a condition token, flatten the per-step camera tokens into one cond sequence, and feed that sequence into the existing IMF/AttnRes head.
|
||||
|
||||
**Tech Stack:** PyTorch, torchvision ResNet-18, Hydra, pytest, SwanLab, SSH/Tailscale.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add failing tests for multi-token conditioning
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/test_imf_vla_agent.py`
|
||||
- Modify: `tests/test_resnet_transformer_agent_wiring.py`
|
||||
|
||||
- [ ] **Step 1: Add a direct agent test**
|
||||
- Stub a vision backbone returning `(B,T,3,D)` and assert `_build_cond()` yields `(B, T*3, D_cond)`.
|
||||
- Assert state is paired with each camera token, not concatenated across cameras first.
|
||||
- [ ] **Step 2: Add Hydra wiring test**
|
||||
- Instantiate a new `agent=resnet_imf_attnres_multitoken` config with small dims.
|
||||
- Assert `condition_tokens_per_step == 3`, `condition_sequence_length == obs_horizon * 3`, and head `n_obs_steps` receives that sequence length.
|
||||
- [ ] **Step 3: Run focused tests and verify RED**
|
||||
- `python -m pytest tests/test_imf_vla_agent.py tests/test_resnet_transformer_agent_wiring.py -q`
|
||||
|
||||
### Task 2: Implement multi-token ResNet conditioning path
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/models/backbones/resnet_diffusion.py`
|
||||
- Modify: `roboimi/vla/agent.py`
|
||||
- Create: `roboimi/vla/conf/agent/resnet_imf_attnres_multitoken.yaml`
|
||||
|
||||
- [ ] **Step 1: Extend ResNet backbone**
|
||||
- Add an opt-in flag to return `(B,T,num_cams,D)` camera tokens instead of one concatenated `(B,T,num_cams*D)` token.
|
||||
- Keep standard ResNet-18 vision mode; do not switch to AttnRes vision.
|
||||
- [ ] **Step 2: Extend VLAAgent condition building**
|
||||
- Support visual features with rank 4 `(B,T,K,D)`.
|
||||
- Broadcast state to `(B,T,K,D_state)`, concatenate per camera, apply projector per token, then flatten to `(B,T*K,D_cond)`.
|
||||
- Track `condition_tokens_per_step` and `condition_sequence_length`.
|
||||
- [ ] **Step 3: Update transformer-head instantiation**
|
||||
- Pass `n_obs_steps=condition_sequence_length` when building transformer heads.
|
||||
- [ ] **Step 4: Add Hydra config**
|
||||
- New agent config uses:
|
||||
- separate ResNet-18 per camera
|
||||
- standard residual vision trunk (`vision_backbone_mode=resnet`)
|
||||
- condition projector output dim tied to `${agent.head.n_emb}`
|
||||
- rollout episodes `10`, `pred_horizon=16`, `num_action_steps=8`
|
||||
|
||||
### Task 3: Verify locally
|
||||
|
||||
**Files:**
|
||||
- Modify only if verification reveals issues
|
||||
|
||||
- [ ] **Step 1: Run focused tests and make them pass**
|
||||
- `python -m pytest tests/test_imf_vla_agent.py tests/test_resnet_transformer_agent_wiring.py -q`
|
||||
- [ ] **Step 2: Run regression subset**
|
||||
- `python -m pytest tests/test_eval_vla_headless.py tests/test_train_vla_rollout_validation.py tests/test_simple_robot_dataset_image_loading.py -q`
|
||||
- [ ] **Step 3: Run local smoke instantiation**
|
||||
- instantiate the new Hydra config and verify cond shape / sequence length
|
||||
|
||||
### Task 4: Launch 4 L20 experiments
|
||||
|
||||
**Files:**
|
||||
- Remote repo copy under `/home/droid/roboimi_suite_20260404`
|
||||
|
||||
- [ ] **Step 1: Sync code to `100.119.99.14`**
|
||||
- [ ] **Step 2: Smoke the new config on remote**
|
||||
- [ ] **Step 3: Launch runs**
|
||||
- `(n_emb=256, n_layer=12)`
|
||||
- `(n_emb=256, n_layer=16)`
|
||||
- `(n_emb=384, n_layer=12)`
|
||||
- `(n_emb=384, n_layer=16)`
|
||||
- [ ] **Step 4: Keep fixed across runs**
|
||||
- rollout episodes `10`
|
||||
- `pred_horizon=16`
|
||||
- `num_action_steps=8`
|
||||
- standard ResNet-18 vision trunk
|
||||
- three separate camera weights
|
||||
- [ ] **Step 5: Record PIDs, GPUs, log paths, SwanLab URLs**
|
||||
@@ -0,0 +1,78 @@
|
||||
# SigLIP2 Multiview VLA Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Integrate a frozen shared SigLIP2 multiview encoder into the IMF/AttnRes policy, preserve raw-256 image handling, and launch two 50k-step experiments on the 5880 host with per-view projection dims 96 and 192.
|
||||
|
||||
**Architecture:** A new backbone will independently encode each camera view with SigLIP2 and project each 768-d pooled feature to a configurable per-view dimension. `VLAAgent` will concatenate visual features with robot state, then optionally project the combined per-step condition to the head's required 384-d interface before diffusion training/inference.
|
||||
|
||||
**Tech Stack:** PyTorch, transformers SigLIP2, Hydra, pytest, SSH/Tailscale, SwanLab.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add failing tests for SigLIP2 backbone and projected conditioning
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/test_siglip2_diffusion_backbone.py`
|
||||
- Modify: `tests/test_imf_vla_agent.py`
|
||||
|
||||
- [ ] **Step 1: Write failing backbone tests**
|
||||
- Instantiate the new backbone with a stub SigLIP2 vision model.
|
||||
- Assert raw dataset resize is `None`, eval resize is `(256, 256)`, output shape is `(B, T, 3 * per_view_output_dim)`.
|
||||
- Assert three views are encoded independently and projected.
|
||||
- [ ] **Step 2: Run focused tests and verify RED**
|
||||
- Run `pytest tests/test_siglip2_diffusion_backbone.py tests/test_imf_vla_agent.py -q`
|
||||
- Expect failure because the backbone/config/projector do not exist yet.
|
||||
- [ ] **Step 3: Extend agent wiring tests**
|
||||
- Add a Hydra/instantiate test for a new SigLIP2 IMF config.
|
||||
- Assert raw condition dim `3 * per_view_output_dim + obs_dim`, projected cond dim `384`, and head `cond_dim == 384`.
|
||||
|
||||
### Task 2: Implement SigLIP2 backbone and optional condition projector
|
||||
|
||||
**Files:**
|
||||
- Create: `roboimi/vla/models/backbones/siglip2_diffusion_backbone.py`
|
||||
- Create: `roboimi/vla/conf/backbone/siglip2_diffusion.yaml`
|
||||
- Create: `roboimi/vla/conf/agent/siglip2_imf_attnres.yaml`
|
||||
- Create: `roboimi/vla/conf/modules/linear_condition_projector.yaml`
|
||||
- Modify: `roboimi/vla/models/backbones/__init__.py`
|
||||
- Modify: `roboimi/vla/agent.py`
|
||||
|
||||
- [ ] **Step 1: Implement backbone**
|
||||
- Load `SiglipVisionModel.from_pretrained("google/siglip2-base-patch16-256")`.
|
||||
- Normalize `[0,1]` pixels with mean/std `0.5` and encode each view independently.
|
||||
- Project each 768-d pooled feature to configurable per-view dim and concatenate across cameras.
|
||||
- [ ] **Step 2: Implement optional condition projector**
|
||||
- Allow `VLAAgent` to accept `cond_projector`.
|
||||
- Track `raw_per_step_cond_dim` and projected `per_step_cond_dim` / `global_cond_dim`.
|
||||
- Apply the projector in `_build_cond()` after visual+state concatenation.
|
||||
- [ ] **Step 3: Add Hydra configs**
|
||||
- New agent config should default to `n_emb=384`, `n_layer=12`, `pred_horizon=16`, `num_action_steps=8`, `head.cond_dim=384`.
|
||||
- Backbone config should set `dataset_image_resize_shape: null` and `eval_image_resize_shape: [256, 256]`.
|
||||
|
||||
### Task 3: Verify locally and prepare remote execution
|
||||
|
||||
**Files:**
|
||||
- Modify as needed only if tests/smoke reveal issues
|
||||
|
||||
- [ ] **Step 1: Run focused tests and make them pass**
|
||||
- `pytest tests/test_siglip2_diffusion_backbone.py tests/test_imf_vla_agent.py tests/test_eval_vla_headless.py tests/test_train_vla_rollout_validation.py tests/test_simple_robot_dataset_image_loading.py -q`
|
||||
- [ ] **Step 2: Run a local smoke instantiation**
|
||||
- Instantiate the new Hydra config with stubbed optional modules or offline-safe monkeypatching.
|
||||
- [ ] **Step 3: Review diffs for unintended LEWM/raw256 regressions**
|
||||
|
||||
### Task 4: Sync to 5880 and launch experiments
|
||||
|
||||
**Files:**
|
||||
- Remote repo copy under `/home/droid/roboimi_suite_20260404`
|
||||
|
||||
- [ ] **Step 1: Stop superseded remote jobs**
|
||||
- [ ] **Step 2: Sync updated code to remote**
|
||||
- Prefer `rsync` or `git push/pull` without overwriting unrelated files.
|
||||
- [ ] **Step 3: Remote smoke test**
|
||||
- Confirm SigLIP2 model download/import works in `/home/droid/miniforge3/envs/roboimi/bin/python`.
|
||||
- Confirm headless rollout path still uses `256x256` eval resize.
|
||||
- [ ] **Step 4: Launch experiment A**
|
||||
- `per_view_output_dim=96`, `embed=384`, `layer=12`, `pred=16`, `exec=8`, `steps=50000`.
|
||||
- [ ] **Step 5: Launch experiment B**
|
||||
- `per_view_output_dim=192`, same other hyperparameters.
|
||||
- [ ] **Step 6: Record PIDs, GPUs, log paths, and SwanLab run URLs.**
|
||||
@@ -1,72 +0,0 @@
|
||||
# Socket/Peg Air Insert Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Convert the air-insert task from ring/bar objects to the XML-defined socket/peg objects and rename the canonical task to `sim_air_insert_socket_peg`.
|
||||
|
||||
**Architecture:** Keep the existing Diana dual-arm air-insert environment and policy flow, but replace all task-state keys, MuJoCo joint/geom names, reward names, and registration names with socket/peg semantics. Use the socket's internal `pin` geom as the success detector: reward reaches max only when `red_peg` contacts `pin` while both objects are airborne. Keep a narrowly scoped backward-compatible robot class alias only if needed to avoid import breakage, but make socket/peg the canonical name.
|
||||
|
||||
**Tech Stack:** Python unittest, NumPy, MuJoCo XML assets, existing `PolicyBase` interpolation behavior.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Rename task registration and reset state API
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/utils/constants.py`
|
||||
- Modify: `roboimi/utils/act_ex_utils.py`
|
||||
- Modify: `roboimi/demos/diana_record_sim_episodes.py`
|
||||
- Modify: `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
- Modify: `roboimi/envs/double_pos_ctrl_env.py`
|
||||
- Test: `tests/test_air_insert_env.py`
|
||||
- Test: `tests/test_eval_vla_headless.py`
|
||||
|
||||
- [ ] Write/adjust tests to expect `sim_air_insert_socket_peg`, `sample_air_insert_socket_peg_state()`, and task-state keys `socket_pos/socket_quat/peg_pos/peg_quat`.
|
||||
- [ ] Run targeted tests and confirm they fail before implementation.
|
||||
- [ ] Implement the canonical task-name and sampler rename across registration, rollout, and eval paths.
|
||||
- [ ] Run targeted tests and confirm they pass.
|
||||
|
||||
### Task 2: Switch environment helpers/reward to socket/peg MuJoCo names
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/envs/double_air_insert_env.py`
|
||||
- Test: `tests/test_air_insert_env.py`
|
||||
|
||||
- [ ] Write/adjust tests for `set_socket_peg_task_state()`, `get_socket_peg_env_state()`, joint names `blue_socket_joint/red_peg_joint`, geom names `socket-1..4`, `red_peg`, and success contact `red_peg` ↔ `pin`.
|
||||
- [ ] Run targeted tests and confirm they fail before implementation.
|
||||
- [ ] Implement helper and reward rename; remove ring/bar aperture geometry success logic.
|
||||
- [ ] Run targeted tests and confirm they pass.
|
||||
|
||||
### Task 3: Switch robot/XML asset names to socket/peg
|
||||
|
||||
**Files:**
|
||||
- Move/Modify: `roboimi/assets/models/manipulators/DianaMed/ring_bar_objects.xml` -> `roboimi/assets/models/manipulators/DianaMed/socket_peg_objects.xml`
|
||||
- Move/Modify: `roboimi/assets/models/manipulators/DianaMed/bi_diana_ring_bar_ee.xml` -> `roboimi/assets/models/manipulators/DianaMed/bi_diana_socket_peg_ee.xml`
|
||||
- Modify: `roboimi/assets/robots/diana_med.py`
|
||||
- Test: `tests/test_robot_asset_paths.py`
|
||||
- Test: `tests/test_air_insert_env.py`
|
||||
|
||||
- [ ] Write/adjust tests to expect `BiDianaMedSocketPeg` and `bi_diana_socket_peg_ee.xml` including `socket_peg_objects.xml`.
|
||||
- [ ] Run targeted tests and confirm they fail before implementation.
|
||||
- [ ] Rename XML assets, update include/model names, set default object positions on the table, and update robot class/path.
|
||||
- [ ] Run targeted tests and confirm they pass.
|
||||
|
||||
### Task 4: Switch scripted policy to socket/peg
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/demos/diana_air_insert_policy.py`
|
||||
- Test: `tests/test_air_insert_env.py`
|
||||
|
||||
- [ ] Write/adjust tests to ensure left hand grasps socket, right hand grasps peg, and the insertion trajectory drives peg toward socket/pin.
|
||||
- [ ] Run targeted policy tests and confirm they fail before implementation.
|
||||
- [ ] Rename policy constants/locals from ring/bar to socket/peg, keep the existing timing/interpolation style, and use socket hold + peg insert waypoints.
|
||||
- [ ] Run targeted policy tests and confirm they pass.
|
||||
|
||||
### Task 5: Full verification
|
||||
|
||||
**Files:**
|
||||
- All modified files above.
|
||||
|
||||
- [ ] Run `python -m unittest tests.test_air_insert_env tests.test_robot_asset_paths tests.test_eval_vla_headless -v`.
|
||||
- [ ] Run a grep check that canonical code no longer references `sim_air_insert_ring_bar`, `sample_air_insert_ring_bar_state`, `ring_block`, or `bar_block` except deliberate compatibility aliases/comments if any.
|
||||
- [ ] Report exact verification output and any remaining caveats.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Held-out Episode Validation Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add optional held-out episode validation to main training flow, including explicit val episode selection and periodic action MSE evaluation without merging LEWM-only model features.
|
||||
|
||||
**Architecture:** Extend the generic dataset with optional episode filtering metadata, extend train config with explicit held-out validation knobs, and wire train_vla to choose between random split and explicit episode split. Reuse agent.predict_action_chunk for action MSE so the metric stays model-agnostic.
|
||||
|
||||
**Tech Stack:** Python, Hydra/OmegaConf, PyTorch, unittest
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add failing tests for dataset episode filtering and config semantics
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/test_simple_robot_dataset_image_loading.py`
|
||||
- Modify: `tests/test_train_vla_rollout_validation.py`
|
||||
|
||||
- [ ] Add dataset tests for `episode_indices` and `available_episode_indices`.
|
||||
- [ ] Add training tests for explicit held-out episode splitting and fail-fast config validation.
|
||||
- [ ] Run focused tests and verify they fail for the expected missing behavior.
|
||||
|
||||
### Task 2: Implement minimal dataset and training support
|
||||
|
||||
**Files:**
|
||||
- Modify: `roboimi/vla/data/simpe_robot_dataset.py`
|
||||
- Modify: `roboimi/vla/conf/config.yaml`
|
||||
- Modify: `roboimi/demos/vla_scripts/train_vla.py`
|
||||
|
||||
- [ ] Add config keys `train.val_episode_indices` and `train.action_mse_val_freq_epochs`.
|
||||
- [ ] Add optional dataset filtering by episode index plus `available_episode_indices` metadata.
|
||||
- [ ] Add explicit train/val dataset builder and held-out action MSE computation.
|
||||
- [ ] Log `val/action_mse` only when explicit held-out episode validation is configured.
|
||||
|
||||
### Task 3: Verify focused coverage
|
||||
|
||||
**Files:**
|
||||
- Test: `tests/test_simple_robot_dataset_image_loading.py`
|
||||
- Test: `tests/test_train_vla_rollout_validation.py`
|
||||
- Test: `tests/test_train_vla_swanlab_logging.py`
|
||||
|
||||
- [ ] Run focused unittest targets for dataset filtering, held-out MSE, and SwanLab logging.
|
||||
- [ ] Fix any regressions with minimal code changes.
|
||||
@@ -0,0 +1,242 @@
|
||||
# VLA Experiment Sweep Design
|
||||
|
||||
## Goal
|
||||
|
||||
Run a matched-configuration training sweep on `/home/droid/project/diana_sim/sim_transfer` for the aligned Transformer diffusion policy, comparing:
|
||||
|
||||
1. ResNet ImageNet pretraining vs no pretraining under matched settings.
|
||||
2. Transformer width/depth choices across a 3×3 grid.
|
||||
|
||||
The sweep should use the full dataset, headless rollout validation every 20 epochs, SwanLab logging, and a total budget of about 200 epochs per run.
|
||||
|
||||
## Current Context
|
||||
|
||||
- Dataset: `100` episodes, `70000` frame-level samples.
|
||||
- Cameras used as image conditions: `r_vis`, `top`, `front`.
|
||||
- Current policy family: ResNet visual encoder + Transformer DDPM head.
|
||||
- Current training path already supports:
|
||||
- full-dataset training (`val_split=0.0`)
|
||||
- checkpoint-time and epoch-time rollout hooks
|
||||
- headless eval path
|
||||
- SwanLab logging
|
||||
- best-checkpoint selection by rollout reward once rollout metrics exist
|
||||
- Current trainer behavior that matters for execution:
|
||||
- step checkpoints are written to `checkpoints/` relative to the Hydra job working directory
|
||||
- epoch-time rollout validation is controlled by `train.rollout_val_freq_epochs`
|
||||
- checkpoint-time rollout validation is separately controlled by `train.rollout_validate_on_checkpoint`
|
||||
- per-run log files, pid files, and run directories must be created by the external launcher, not by the trainer itself
|
||||
|
||||
## Derived Schedule
|
||||
|
||||
For a chosen global batch size `B`, define:
|
||||
|
||||
- `steps_per_epoch = floor(70000 / B)` because training currently uses `drop_last=True` when the train set is larger than the batch.
|
||||
- `rollout_freq_steps = 20 * steps_per_epoch`
|
||||
- `max_steps = 200 * steps_per_epoch`
|
||||
|
||||
Example at `B = 32`:
|
||||
|
||||
- `steps_per_epoch = 2187`
|
||||
- `rollout_freq_steps = 43740`
|
||||
- `max_steps = 437400`
|
||||
|
||||
This schedule must be recomputed after the batch-size probe chooses the final shared batch size.
|
||||
|
||||
## Experiment Strategy
|
||||
|
||||
### Recommended approach
|
||||
|
||||
Use one **shared batch size across all runs**, chosen by probing the largest model that will appear in the sweep. This preserves fairness while still using as much GPU memory as safely available.
|
||||
|
||||
### Why not per-run max batch size
|
||||
|
||||
Allowing each model to use a different batch size would add another changing variable, making the 11-run comparison harder to interpret.
|
||||
|
||||
### Unique executions vs logical comparisons
|
||||
|
||||
There are `11` logical comparison points in the user request (`2` for the pretraining comparison and `9` for the emb/layer grid), but only `10` **unique** training configurations because:
|
||||
|
||||
- the pretrained baseline in Section A (`pretrained_backbone_weights=IMAGENET1K_V1`, `n_emb=128`, `n_layer=4`) is also one cell of the Section B sweep.
|
||||
|
||||
This design will therefore run **10 unique training jobs**, and the pretrained-on baseline will be reused as the `(128, 4)` architecture-sweep point. This avoids wasting one full 200-epoch run on an identical configuration.
|
||||
|
||||
The reused baseline has one canonical identity throughout execution:
|
||||
|
||||
- run name stem: `sim-transfer-baseline-pretrain-on-emb128-layer4`
|
||||
- this same run is counted both as:
|
||||
- the pretrained-on arm of the 2-point pretraining comparison
|
||||
- the `(n_emb=128, n_layer=4)` point of the 9-point architecture sweep
|
||||
|
||||
## Batch-Size Probe
|
||||
|
||||
Before launching the full sweep:
|
||||
|
||||
1. Probe the largest planned model:
|
||||
- `n_emb=384`
|
||||
- `n_layer=12`
|
||||
- ResNet pretrained enabled
|
||||
2. Try candidate batch sizes in ascending order:
|
||||
- `32`
|
||||
- `48`
|
||||
- `64`
|
||||
- optionally `80` if `64` is clearly safe
|
||||
3. For each candidate, run a short GPU smoke training with:
|
||||
- real dataset
|
||||
- three cameras
|
||||
- no SwanLab
|
||||
- no rollout validation
|
||||
- `max_steps=4`
|
||||
4. A candidate **passes** if all 4 steps finish without:
|
||||
- CUDA OOM
|
||||
- process crash / abort
|
||||
- NaN / Inf loss
|
||||
5. Choose the **highest passing** candidate and use it for all sweep runs.
|
||||
|
||||
Learning rate should scale linearly using the current training rule:
|
||||
|
||||
- config baseline: `lr=1e-4` at `batch_size=16`
|
||||
- equivalent rule: `lr = 1e-4 * (batch_size / 16)`
|
||||
- equivalently: `lr = 2e-4 * (batch_size / 32)`
|
||||
|
||||
## Experiment Matrix
|
||||
|
||||
Total logical comparison points: `11`
|
||||
|
||||
Total unique training executions: `10`
|
||||
|
||||
### A. ResNet pretraining comparison (2 runs)
|
||||
|
||||
Fixed Transformer settings:
|
||||
|
||||
- `n_emb=128`
|
||||
- `n_layer=4`
|
||||
- `n_head=4`
|
||||
- `freeze_backbone=true`
|
||||
- `use_separate_rgb_encoder_per_camera=true`
|
||||
|
||||
Interpretation note:
|
||||
|
||||
- This comparison keeps the current frozen-backbone recipe unchanged.
|
||||
- Therefore, the `pretrained_backbone_weights=null` run measures **random frozen visual features vs ImageNet frozen visual features**.
|
||||
- It does **not** test end-to-end training of the ResNet from scratch; that would be a separate experiment.
|
||||
|
||||
Runs:
|
||||
|
||||
1. `pretrained_backbone_weights=IMAGENET1K_V1`
|
||||
2. `pretrained_backbone_weights=null`
|
||||
|
||||
### B. Transformer architecture sweep (9 runs)
|
||||
|
||||
Fixed visual backbone setting:
|
||||
|
||||
- `pretrained_backbone_weights=IMAGENET1K_V1`
|
||||
- `freeze_backbone=true`
|
||||
- `use_separate_rgb_encoder_per_camera=true`
|
||||
- `n_head=4`
|
||||
|
||||
Rationale:
|
||||
|
||||
- The 9-point emb/layer sweep fixes `pretrained_backbone_weights=IMAGENET1K_V1`.
|
||||
- The separate 2-run comparison in Section A isolates the effect of ResNet pretraining at the baseline architecture.
|
||||
|
||||
Grid:
|
||||
|
||||
- `n_emb ∈ {128, 256, 384}`
|
||||
- `n_layer ∈ {4, 8, 12}`
|
||||
|
||||
## Shared Training Configuration
|
||||
|
||||
All full runs should use:
|
||||
|
||||
- dataset: `/home/droid/project/diana_sim/sim_transfer`
|
||||
- cameras: `[r_vis, top, front]`
|
||||
- device: `cuda`
|
||||
- `val_split=0.0`
|
||||
- `num_workers=12`
|
||||
- `seed=42`
|
||||
- headless rollout validation
|
||||
- `rollout_val_freq_epochs=20`
|
||||
- `rollout_validate_on_checkpoint=false`
|
||||
- `rollout_num_episodes=3`
|
||||
- `use_swanlab=true`
|
||||
- shared batch size from probe
|
||||
- shared learning-rate rule from probe
|
||||
- `max_steps` recomputed to represent about `200` epochs
|
||||
|
||||
Reproducibility note:
|
||||
|
||||
- `train.seed=42` will be fixed and recorded for all runs.
|
||||
- Under the current `train_vla.py`, this does **not** make the runs bitwise deterministic when `val_split=0.0`, because model initialization and shuffled training order are not fully seeded by the trainer today.
|
||||
- The sweep is therefore treated as a **matched-config comparison**, not a strict deterministic benchmark.
|
||||
|
||||
## Rollout / Validation Policy
|
||||
|
||||
- No held-out val split will be used.
|
||||
- Loss remains a fallback best-model metric until the first rollout reward exists.
|
||||
- After rollout metrics begin, the best checkpoint should be chosen by **higher average rollout reward**.
|
||||
- Rollout should run with:
|
||||
- no GUI
|
||||
- CPU eval device
|
||||
- current evaluation helper path
|
||||
|
||||
## Execution Model
|
||||
|
||||
Runs should be launched **serially**, not concurrently, because:
|
||||
|
||||
- the user currently has one GPU available
|
||||
- concurrent jobs would reduce throughput and complicate memory management
|
||||
- serial execution preserves clean logs and easier resume semantics
|
||||
|
||||
Each run gets:
|
||||
|
||||
- a dedicated run directory under `runs/`
|
||||
- its own log file
|
||||
- its own pid / launcher metadata if backgrounded
|
||||
- a unique SwanLab run name
|
||||
- `hydra.job.chdir=true`
|
||||
- `hydra.run.dir=<run_dir>` so the trainer-local `checkpoints/` land inside the run directory
|
||||
|
||||
## Naming Convention
|
||||
|
||||
Suggested run name pattern:
|
||||
|
||||
- pretraining comparison:
|
||||
- `sim-transfer-baseline-pretrain-on-emb128-layer4-bs{B}-lr{LR}-e200`
|
||||
- `sim-transfer-pretrain-off-bs{B}-lr{LR}-e200`
|
||||
- architecture sweep:
|
||||
- `(128,4)` reuses `sim-transfer-baseline-pretrain-on-emb128-layer4-bs{B}-lr{LR}-e200`
|
||||
- `sim-transfer-emb{E}-layer{L}-bs{B}-lr{LR}-e200`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
The sweep is successful when:
|
||||
|
||||
1. A shared safe batch size has been found.
|
||||
2. All 10 unique runs have launchable commands and isolated run directories.
|
||||
3. Each run trains for about 200 epochs worth of steps.
|
||||
4. Rollout validation occurs every 20 epochs.
|
||||
5. SwanLab records train metrics and rollout rewards.
|
||||
6. Final comparison can be made using best rollout average reward and training traces across:
|
||||
- 2-point pretraining comparison
|
||||
- 9-point emb/layer sweep
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No change to dataset format.
|
||||
- No change to model architecture beyond the requested pretraining toggle and Transformer width/depth sweep.
|
||||
- No attempt to equalize wall-clock time across models.
|
||||
- No multi-GPU scheduling.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
### Risk: larger models may OOM
|
||||
|
||||
Mitigation: probe the largest model first and choose one shared safe batch size.
|
||||
|
||||
### Risk: long runs may be interrupted
|
||||
|
||||
Mitigation: use per-run directories, periodic checkpoints, and resumable launch commands.
|
||||
|
||||
### Risk: rollout validation increases total training time
|
||||
|
||||
Mitigation: keep rollout headless, CPU-based, and limited to 3 episodes every 20 epochs.
|
||||
@@ -0,0 +1,272 @@
|
||||
# IMF-AttnRes Policy Migration Design
|
||||
|
||||
**Date:** 2026-04-01
|
||||
**Status:** Approved in chat, written spec pending review
|
||||
|
||||
## Goal
|
||||
|
||||
将 `/home/droid/project/diffusion_policy` 中提交 `185ed659` 的 IMF-AttnRes diffusion policy 迁移到当前 `roboimi` 仓库，作为当前 DiT / Transformer diffusion policy 的替代训练选项；同时迁移其训练目标与一步推理机制，并保持 RoboIMI 现有的仿真环境、三相机视觉输入、数据集格式、训练脚本和 rollout 验证工作流可继续使用。
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- 不迁移 external repo 中与当前任务无关的 obs encoder、dataset、env wrapper、PushT 专用逻辑。
|
||||
- 不强行复刻 external repo 中全部目录结构；仅迁移当前 RoboIMI 训练所必需的模型、loss、inference 语义。
|
||||
- 不在本次工作中同时保留旧 DiT 为默认训练目标；旧配置继续可用，但新模型单独提供 config 入口。
|
||||
|
||||
## User-Confirmed Requirements
|
||||
|
||||
1. 迁移对象是 `185ed659` 中的 **IMF-AttnRes 模型相关代码**。
|
||||
2. 不只是迁移骨架，还要迁移：
|
||||
- **训练目标**
|
||||
- **一步推理机制**
|
||||
3. 视觉输入与当前 RoboIMI diffusion policy 一致：
|
||||
- 使用三个相机图像作为条件输入
|
||||
- 图像观测必须作为条件，而不是拼进输出预测目标
|
||||
4. 当前任务里，IMF policy 用来替代现有 DiT/Transformer diffusion policy 训练。
|
||||
5. 训练参数沿用最近一次训练的大体设置（后续由训练命令显式覆盖），但推理方式改为 IMF 的 one-step 机制。
|
||||
6. 用户接受 IMF 中“全注意力 / 非因果注意力”的实现约束。
|
||||
|
||||
## External Source of Truth
|
||||
|
||||
迁移语义以 external repo 的以下文件为准：
|
||||
|
||||
- `diffusion_policy/model/diffusion/attnres_transformer_components.py`
|
||||
- `diffusion_policy/model/diffusion/imf_transformer_for_diffusion.py`
|
||||
- `diffusion_policy/policy/imf_transformer_hybrid_image_policy.py`
|
||||
- 参考配置：`image_pusht_diffusion_policy_dit_imf_attnres_full.yaml`
|
||||
|
||||
其中最关键的差异是：该策略并非 DDPM/DDIM 多步去噪，而是 IMF 训练目标 + one-step 推理。
|
||||
|
||||
## Current RoboIMI Baseline
|
||||
|
||||
当前 RoboIMI 中与该任务直接相关的基线如下：
|
||||
|
||||
- 视觉编码：`ResNetDiffusionBackbone`
|
||||
- 三相机：`r_vis`, `top`, `front`
|
||||
- 每个时间步将相机特征与 `qpos` 拼接为 per-step condition
|
||||
- 策略主体：`VLAAgent`
|
||||
- `compute_loss()` 使用 DDPM 噪声预测损失
|
||||
- `predict_action()` 使用 DDIM 多步采样
|
||||
- 在线控制通过动作队列机制在 `select_action()` 中按 chunk 触发预测
|
||||
- 训练脚本：`roboimi/demos/vla_scripts/train_vla.py`
|
||||
- 支持 GPU 训练、SwanLab 日志、headless rollout 验证
|
||||
|
||||
因此，本次迁移的核心不是换视觉 backbone，而是替换 **head + loss + inference semantics**。
|
||||
|
||||
## Recommended Integration Approach
|
||||
|
||||
采用 **最小侵入式集成**：
|
||||
|
||||
1. **保留当前 RoboIMI 的视觉编码、数据读取、rollout/eval、训练脚本主框架**。
|
||||
2. **新增 IMF 专用 head 模块**，在 RoboIMI 内本地实现：
|
||||
- AttnRes 组件
|
||||
- IMF transformer 主体
|
||||
3. **新增 IMF 专用 agent**，复用当前 `VLAAgent` 的：
|
||||
- 归一化逻辑
|
||||
- 相机顺序管理
|
||||
- 观测缓存 / 动作 chunk 缓存
|
||||
- rollout 接口
|
||||
但覆盖：
|
||||
- `compute_loss()`
|
||||
- `predict_action()`
|
||||
4. **新增独立 Hydra config**，让 IMF policy 作为新的 agent 选项，不破坏已有 resnet_transformer / gr00t_dit 配置。
|
||||
|
||||
这样做的原因：
|
||||
|
||||
- 迁移 IMF 语义时不必把当前 DDPM agent 搅乱；
|
||||
- rollout / eval / checkpoint 逻辑仍然可复用；
|
||||
- 便于和现有 Transformer / DiT 直接做 A/B 对比训练。
|
||||
|
||||
## Architecture
|
||||
|
||||
### 1. Observation / Conditioning Path
|
||||
|
||||
沿用当前 RoboIMI 的视觉路径：
|
||||
|
||||
- 输入观测：`images={r_vis, top, front}` + `qpos`
|
||||
- `ResNetDiffusionBackbone` 对每个相机编码，得到 per-camera feature
|
||||
- `state_encoder` 编码 `qpos`
|
||||
- 将三相机特征与 state feature 按时间步拼接，形成 `per_step_cond`
|
||||
|
||||
这里不迁移 external repo 的 obs_encoder 实现；我们只对齐 **“图像作为条件 token 输入 transformer”** 这一语义。
|
||||
|
||||
### 2. Condition Tokenization
|
||||
|
||||
对齐 external IMF transformer 的 token 使用方式：
|
||||
|
||||
- action trajectory token：由 `(B, pred_horizon, action_dim)` 通过线性层映射到 `n_emb`
|
||||
- time token：两个标量 `r` 与 `t`，分别通过 sinusoidal embedding + linear projection 得到 token
|
||||
- observation token：`per_step_cond` 通过线性层映射到 `n_emb`
|
||||
- 最终 token 序列为：
|
||||
- `[r_token, t_token, obs_cond_tokens..., action_tokens...]`
|
||||
|
||||
在当前任务中，obs token 数量等于 `obs_horizon`，且图像观测始终作为条件输入。
|
||||
|
||||
### 3. IMF-AttnRes Backbone
|
||||
|
||||
在 RoboIMI 内新增 AttnRes backbone 实现，保持 external commit 的关键语义：
|
||||
|
||||
- `RMSNorm` / `RMSNormNoWeight`
|
||||
- RoPE
|
||||
- Grouped Query Self-Attention
|
||||
- SwiGLU FFN
|
||||
- AttnRes operator / residual source aggregation
|
||||
- `AttnResTransformerBackbone`
|
||||
|
||||
并保持：
|
||||
|
||||
- **full attention**（不使用因果注意力）
|
||||
- `backbone_type='attnres_full'`
|
||||
- 输出仅切回 action token 部分，再经过最终 norm + head 得到 velocity-like 输出
|
||||
|
||||
### 4. Training Objective
|
||||
|
||||
训练目标从当前 DDPM epsilon prediction 改为 external IMF 目标：
|
||||
|
||||
给定真实轨迹 `x` 与随机噪声 `e`：
|
||||
|
||||
1. 采样 `t ~ U(0,1)`、`r ~ U(0,1)`，并排序为 `t >= r`
|
||||
2. 构造插值状态：
|
||||
- `z_t = (1 - t) x + t e`
|
||||
3. 用模型计算：
|
||||
- `v = f(z_t, t, t, cond)`
|
||||
4. 对 `g(z, r, t) = f(z, r, t, cond)` 做 JVP，得到：
|
||||
- `u, du_dt`
|
||||
5. 构造 compound velocity：
|
||||
- `V = u + (t - r) * du_dt`
|
||||
6. 目标为：
|
||||
- `target = e - x`
|
||||
7. 用 action 维度上的 MSE 作为最终损失
|
||||
|
||||
RoboIMI 现有 batch 中的 `action_is_pad` 仍要保留支持；如果存在 padding，只在有效 action 上计算损失。
|
||||
|
||||
### 5. One-Step Inference
|
||||
|
||||
推理改为 external IMF 的一步采样语义：
|
||||
|
||||
1. 从标准高斯初始化 action trajectory `z_t`
|
||||
2. 计算 `u = f(z_t, r=0, t=1, cond)`
|
||||
3. 一步更新：
|
||||
- `x_hat = z_t - (t-r) * u = z_t - u`
|
||||
4. 反归一化得到动作序列
|
||||
|
||||
这意味着：
|
||||
|
||||
- `num_inference_steps` 对 IMF policy 固定为 `1`
|
||||
- 不再调用 DDIM scheduler 的多步 `step()`
|
||||
- 在线控制中仍沿用当前 chunk 机制：
|
||||
- 动作队列为空时触发一次 `predict_action_chunk()`
|
||||
- 取预测序列中 `[obs_horizon-1 : obs_horizon-1+num_action_steps]` 这一段入队
|
||||
|
||||
也就是说，**触发模型前向的规则不变，改变的是每次触发后的动作序列生成方式**。
|
||||
|
||||
## API / Code Structure
|
||||
|
||||
计划中的主要代码边界如下：
|
||||
|
||||
- `roboimi/vla/models/heads/attnres_transformer_components.py`
|
||||
- IMF AttnRes 基础组件
|
||||
- `roboimi/vla/models/heads/imf_transformer1d.py`
|
||||
- RoboIMI 版本 IMF transformer head
|
||||
- 对外暴露 `forward(sample, r, t, cond=None)`
|
||||
- 暴露 `get_optim_groups()` 供 AdamW 分组使用
|
||||
- `roboimi/vla/agent_imf.py`
|
||||
- 复用 `VLAAgent` 的观测处理 / normalization / queue 基础设施
|
||||
- 覆盖 IMF 的训练损失与 one-step 预测逻辑
|
||||
- Hydra config
|
||||
- `roboimi/vla/conf/head/imf_transformer1d.yaml`
|
||||
- `roboimi/vla/conf/agent/resnet_imf_attnres.yaml`
|
||||
|
||||
训练脚本主流程尽量不改；只要求它能 instantiate 新 agent 并继续使用当前 rollout / checkpoint / swanlab 逻辑。
|
||||
|
||||
## Compatibility Decisions
|
||||
|
||||
## Initial Config Defaults To Preserve
|
||||
|
||||
为避免迁移时语义漂移，首版 IMF 配置默认值明确固定为：
|
||||
|
||||
- `backbone_type: attnres_full`
|
||||
- `n_head: 1`
|
||||
- `n_kv_head: 1`
|
||||
- `n_cond_layers: 0`
|
||||
- `time_as_cond: true`
|
||||
- `causal_attn: false`
|
||||
- `num_inference_steps: 1`
|
||||
|
||||
这些默认值与 external `185ed659` 的 IMF-AttnRes 使用方式保持一致；后续调参可以覆盖，但首版迁移必须先以该语义跑通。
|
||||
|
||||
### Reuse From RoboIMI
|
||||
|
||||
保留：
|
||||
|
||||
- 三相机数据读取方式
|
||||
- ResNet visual backbone
|
||||
- qpos / action normalization
|
||||
- 训练循环、优化器、scheduler、SwanLab、headless rollout
|
||||
- `select_action()` 的在线 chunk 执行方式
|
||||
|
||||
### Replace With External IMF Semantics
|
||||
|
||||
替换：
|
||||
|
||||
- transformer head 实现
|
||||
- diffusion training objective
|
||||
- inference sampling semantics
|
||||
|
||||
### Intentionally Not Mirrored 1:1
|
||||
|
||||
不强行与 external repo 一致的部分：
|
||||
|
||||
- external repo 的整体 policy 基类继承体系
|
||||
- external repo 的 obs encoder 模块树
|
||||
- external repo 的 normalizer / mask generator 框架
|
||||
|
||||
原因是当前 RoboIMI 已有稳定的数据接口和 rollout 流程，直接嫁接进去更稳。
|
||||
|
||||
## Testing / Verification Strategy
|
||||
|
||||
迁移完成后至少验证以下内容：
|
||||
|
||||
1. **单元 / 冒烟验证**
|
||||
- IMF head 前向 shape 正确
|
||||
- IMF agent `compute_loss()` 在真实 batch 上可前向、反向
|
||||
- IMF agent `predict_action()` 能输出 `(B, pred_horizon, action_dim)`
|
||||
2. **训练链路验证**
|
||||
- 使用 GPU 跑一个短训练任务，确认：
|
||||
- dataloader 正常
|
||||
- optimizer / lr scheduler 正常
|
||||
- SwanLab 正常记录配置和训练指标
|
||||
3. **rollout 验证**
|
||||
- 训练中周期性 headless rollout 能跑通
|
||||
- 环境仍按 EE-style `step()` 接收动作
|
||||
4. **最终交付**
|
||||
- 用用户指定的同类超参数启动正式训练
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
### Risk 1: JVP 在 CUDA 注意力内核上不稳定
|
||||
|
||||
缓解：沿用 external repo 的策略，在 JVP 路径上切换到 math SDP kernel，必要时 fallback 到 `torch.autograd.functional.jvp`。同时，JVP 的切线构造与 `u, du_dt` 计算流程必须严格对齐 external source，不在本次迁移中自行改写其数学语义。
|
||||
|
||||
### Risk 2: Optimizer 参数分组遗漏新模块
|
||||
|
||||
缓解：IMF head 提供 `get_optim_groups()`，并在训练脚本中按“只要 head 提供该接口就使用”的策略统一处理，而不是绑定旧 `head_type`。
|
||||
|
||||
### Risk 3: 现有 rollout 逻辑假定 DDIM 多步采样
|
||||
|
||||
缓解：保持 `select_action()` / `predict_action_chunk()` 接口不变，只替换 `predict_action()` 内部实现，确保 eval 代码无需理解 IMF 细节。
|
||||
|
||||
### Risk 4: 训练命令参数与新 config 不一致
|
||||
|
||||
缓解：新增独立 agent config，并保留此前训练参数作为显式 CLI override 模板。
|
||||
|
||||
## Success Criteria
|
||||
|
||||
以下条件全部满足，视为本次迁移成功：
|
||||
|
||||
1. RoboIMI 中新增 IMF-AttnRes policy，可通过 Hydra config 单独启用。
|
||||
2. 训练时使用 external IMF 的 loss，而不是当前 DDPM epsilon loss。
|
||||
3. 推理时使用 one-step IMF 采样，而不是 DDIM 多步采样。
|
||||
4. 三相机图像始终作为条件输入参与模型前向。
|
||||
5. 在线 rollout 能在 headless 仿真环境中跑通。
|
||||
6. 能按最近一次实验参数模板成功启动训练。
|
||||
@@ -0,0 +1,75 @@
|
||||
# IMF Rollout Trajectory Images + Short-Horizon Training Design
|
||||
|
||||
## Background
|
||||
The current RoboIMI IMF training flow can perform rollout validation and log scalar reward metrics to SwanLab, but it does not yet emit the qualitative rollout artifacts now required for analysis. The user wants training-time rollout validation to save front-view trajectory images with the model-generated trajectory drawn in red, upload those images to SwanLab, and then start a new local short-horizon IMF training run.
|
||||
|
||||
## Goals
|
||||
1. During training-time rollout validation, save one **front-camera** trajectory image per rollout episode.
|
||||
2. The image must show the rollout EE trajectory in red.
|
||||
3. Reuse the existing repository trajectory visualization logic as much as practical, especially the existing red capsule-marker trajectory representation.
|
||||
4. Save 5 rollout images locally for each validation event and upload the same 5 images to SwanLab.
|
||||
5. Do **not** record rollout videos for this training-time validation flow.
|
||||
6. Start a new local IMF-AttnRes training run with:
|
||||
- `agent.head.n_emb=384`
|
||||
- `agent.head.n_layer=12`
|
||||
- `agent.pred_horizon=8`
|
||||
- `agent.num_action_steps=4`
|
||||
- `train.max_steps=50000`
|
||||
- `train.rollout_num_episodes=5`
|
||||
- `train.use_swanlab=true`
|
||||
|
||||
## Non-Goals
|
||||
- No IMF architecture or loss-function change.
|
||||
- No dataset schema change.
|
||||
- No rollout video generation for the new training flow.
|
||||
- No interactive viewer requirement.
|
||||
|
||||
## Existing Relevant Code
|
||||
- `roboimi/demos/vla_scripts/eval_vla.py`
|
||||
- already supports rollout summaries, optional trajectory export, and optional video export.
|
||||
- `roboimi/utils/raw_action_trajectory_viewer.py`
|
||||
- already contains the red trajectory capsule-marker construction logic.
|
||||
- `roboimi/demos/vla_scripts/train_vla.py`
|
||||
- already performs periodic rollout validation and scalar SwanLab logging.
|
||||
- `roboimi/vla/agent.py`
|
||||
- already implements “predict pred_horizon, execute first num_action_steps” queue semantics.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### 1. Artifact contract
|
||||
Each rollout episode will emit one distinct PNG file under the eval artifact directory. The file naming/path contract must be per-episode, not shared, so a 5-episode validation event yields 5 stable image paths without overwriting.
|
||||
|
||||
### 2. Trajectory definition
|
||||
The red trajectory corresponds to the **actually executed model action sequence** over the rollout loop: the raw EE actions returned and consumed step-by-step by the policy loop. For the requested short-horizon run, this means the visualization reflects repeated execution of the first 4 actions from each predicted 8-action chunk, not every discarded future prediction from replanning.
|
||||
|
||||
### 3. Camera choice
|
||||
The training-time image export path is explicitly pinned to the repo’s concrete `front` camera key. It must not silently use `camera_names[0]` if that is not `front`.
|
||||
|
||||
### 4. Rendering path
|
||||
`eval_vla.py` will add a lightweight headless image-export path that:
|
||||
- renders the `front` camera frame,
|
||||
- overlays the trajectory using the existing red trajectory representation,
|
||||
- saves a static PNG per episode.
|
||||
|
||||
The implementation may reuse the existing marker-construction logic directly and add a minimal helper for final image composition/export.
|
||||
|
||||
### 5. Training-time behavior
|
||||
`train_vla.py` rollout validation must explicitly:
|
||||
- request/save trajectory images,
|
||||
- keep `record_video=false`,
|
||||
- return the 5 per-episode image paths in the rollout summary payload,
|
||||
- upload those 5 images to SwanLab,
|
||||
- keep image-upload failures non-fatal.
|
||||
|
||||
## Expected User-Visible Outcome
|
||||
For each scheduled validation event in the new training run:
|
||||
- 5 rollout episodes execute,
|
||||
- 5 front-view PNG trajectory images are saved locally,
|
||||
- the same 5 images are uploaded to SwanLab,
|
||||
- scalar reward metrics continue to be logged,
|
||||
- no rollout videos are generated.
|
||||
|
||||
## Risks and Mitigations
|
||||
- **Headless rendering conflicts from desktop env vars**: force headless eval onto EGL when `headless=true`.
|
||||
- **Image overwrite risk**: use explicit per-episode artifact paths.
|
||||
- **SwanLab media API mismatch**: isolate media logging in a small best-effort helper.
|
||||
@@ -0,0 +1,138 @@
|
||||
# LEWM ViT Backbone Replacement Design
|
||||
|
||||
## Goal
|
||||
将当前 roboimi VLA policy 中的 ResNet 视觉编码器替换为来自 LEWM checkpoint 的冻结 ViT 视觉编码器（encoder + projector），仅使用最终 CLS token 的 192 维 embedding 作为视觉特征。
|
||||
|
||||
## User constraints
|
||||
- 使用 `/home/droid/下载/lewm_sim_transfer_checkpoint_usage.md` 中确认的训练好 checkpoint
|
||||
- 只使用视觉编码部分：`encoder + projector`
|
||||
- 权重冻结
|
||||
- 维持“视觉特征 + state 拼接，再送入 diffusion transformer”这一总体处理方式
|
||||
- 输入使用三视角：`[r_vis, top, front]`
|
||||
- 在 5880 机器上启动两个训练：`embed=384/layer=12` 和 `embed=256/layer=12`
|
||||
- `pred_horizon=16`
|
||||
- `num_action_steps=8`
|
||||
- 每个训练 `50k` steps
|
||||
- rollout 验证每次用 `10` 个 episodes，不是之前的 `5`
|
||||
|
||||
## Trusted existing facts
|
||||
1. LEWM checkpoint 路径：
|
||||
- `/home/droid/le-wm/lewm-sim-transfer/pa1w85md8jop6bvol8oxp/checkpoints/epoch=99-step=47800.ckpt`
|
||||
2. 需要加载的 state_dict 前缀：
|
||||
- `model.encoder.*`
|
||||
- `model.projector.*`
|
||||
3. LEWM ViT 配置：
|
||||
- encoder scale: `tiny`
|
||||
- hidden size: `192`
|
||||
- layers: `12`
|
||||
- attention heads: `3`
|
||||
- patch size: `14`
|
||||
- projector: `MLP(192 -> 2048 -> 192)` with `BatchNorm1d + GELU`
|
||||
4. LEWM 训练时三视角先拼成单图，再送入单个 ViT encoder；输出整体视觉 embedding 是 **192 维**。
|
||||
|
||||
## Key design decision
|
||||
### Chosen design: fuse 3 cameras into one LEWM-style image, output one 192-d visual vector per timestep
|
||||
不是把 LEWM ViT 当成“每相机一个 192-d encoder”，而是按 LEWM 原训练方式：
|
||||
- 输入三视角图像字典 `{r_vis, top, front}`
|
||||
- 按固定顺序拼成一张 fused image
|
||||
- 走单个 frozen ViT + projector
|
||||
- 得到一个 **192 维总视觉特征**
|
||||
|
||||
### Why this is the right replacement
|
||||
当前 ResNet backbone 对外给到 policy head 的**总视觉特征维度**是：
|
||||
- 每相机 `64`
|
||||
- 三相机总计 `192`
|
||||
|
||||
而 LEWM checkpoint 输出的 CLS/projector embedding 也是：
|
||||
- 总计 `192`
|
||||
|
||||
因此，最自然的“直接平替当前 ResNet 视觉编码器”的方式是：
|
||||
- 用 LEWM backbone 直接产出一个 192-d 总视觉向量
|
||||
- 后续和 state `16-d` 拼接后，依旧得到 `208-d` 条件向量
|
||||
- 不改 diffusion head 的总体接口和语义
|
||||
|
||||
## Interface compatibility plan
|
||||
现有 `VLAAgent` 假设 backbone 暴露：
|
||||
- `camera_names`
|
||||
- `num_cameras`
|
||||
- `output_dim`（语义上是“每相机特征维度”）
|
||||
- `forward(images_dict) -> (B, T, total_visual_dim)`
|
||||
|
||||
为了最小改动兼容现有 agent：
|
||||
- 新 LEWM backbone 的 `forward()` 返回 `(B, T, 192)`
|
||||
- `camera_names = ('r_vis', 'top', 'front')`
|
||||
- `num_cameras = 3`
|
||||
- `output_dim = 64`
|
||||
|
||||
这样 `VLAAgent` 内部仍会计算：
|
||||
- `per_step_cond_dim = output_dim * num_cams + obs_dim = 64*3 + 16 = 208`
|
||||
与实际 `forward()` 输出的 `192 + 16 = 208` 保持一致。
|
||||
|
||||
> 也就是说：`output_dim` 在这个 backbone 里保留为“与旧 ResNet 总特征等价的单相机占位维度”，而不是“真实 projector 输出维度”。这是一个兼容性 shim，用来避免改 agent 主逻辑。
|
||||
|
||||
## Image preprocessing design
|
||||
当前 roboimi dataset 已经把每个相机图像读成：
|
||||
- `(C, 224, 224)`
|
||||
- 值域 `[0, 1]`
|
||||
|
||||
新 LEWM backbone 将：
|
||||
1. 按顺序取 `r_vis`, `top`, `front`
|
||||
2. 在宽度方向拼接，得到 fused image：
|
||||
- `(C, 224, 672)`
|
||||
3. 使用 LEWM 一致的 ImageNet normalize：
|
||||
- mean `[0.485, 0.456, 0.406]`
|
||||
- std `[0.229, 0.224, 0.225]`
|
||||
4. 调用 `ViTModel(..., interpolate_pos_encoding=True)`
|
||||
5. 取 `last_hidden_state[:, 0]`
|
||||
6. 送入 frozen projector，得到 `(B*T, 192)`
|
||||
|
||||
## Files to create / modify
|
||||
### New files
|
||||
- `roboimi/vla/models/backbones/lewm_vit_backbone.py`
|
||||
- `roboimi/vla/conf/backbone/lewm_vit_diffusion.yaml`
|
||||
- `roboimi/vla/conf/agent/lewm_imf_attnres.yaml`
|
||||
- `tests/test_lewm_vit_backbone.py`
|
||||
|
||||
### Modified files
|
||||
- `roboimi/vla/models/backbones/__init__`（如果需要导出）
|
||||
- `tests/test_imf_vla_agent.py`（增加新 backbone 集成用例）
|
||||
- `roboimi/demos/vla_scripts/train_vla.py`（如需仅调整 rollout 默认/日志；如果命令覆盖足够，则尽量不改主逻辑）
|
||||
- 训练/实验 suite 文档（新增本次 LEWM ViT 训练记录）
|
||||
|
||||
## Testing plan
|
||||
1. **Unit test: load + forward**
|
||||
- 用 synthetic checkpoint 验证新 backbone 能正确加载 `model.encoder.*` 与 `model.projector.*`
|
||||
- 输入 3 相机 `(B,T,C,224,224)`
|
||||
- 输出 `(B,T,192)`
|
||||
2. **Agent integration test**
|
||||
- backbone.output_dim=64, num_cameras=3
|
||||
- agent `_build_cond()` 输出最后维度为 `208`
|
||||
3. **Remote smoke test on 5880**
|
||||
- 使用真实 checkpoint
|
||||
- `max_steps=2`
|
||||
- 两个实验各自 smoke 一次
|
||||
4. **Full run**
|
||||
- GPU0: `embed=384, layer=12`
|
||||
- GPU1: `embed=256, layer=12`
|
||||
- `rollout_num_episodes=10`
|
||||
|
||||
## Training launch contract
|
||||
- host: `100.73.14.65`
|
||||
- code dir: `/home/droid/roboimi_suite_20260404`
|
||||
- python: `/home/droid/miniforge3/envs/roboimi/bin/python`
|
||||
- dataset: `/home/droid/sim_dataset/sim_transfer`
|
||||
- cameras: `[r_vis, top, front]`
|
||||
- agent: new `lewm_imf_attnres`
|
||||
- max_steps: `50000`
|
||||
- rollout every `5` epochs
|
||||
- rollout episodes: `10`
|
||||
|
||||
## Risks
|
||||
1. LEWM 训练时的 fused image 预处理如果方向实现错了（224x672 vs 672x224），会导致分布偏移。
|
||||
2. 当前 roboimi env 需确保安装 `transformers`；从 `environment.yml` 看本地已有该依赖，但远端训练环境要 smoke 确认。
|
||||
3. 因为这是 frozen ViT + projector，若 projector BN 仍保持 train 模式，统计量会漂移，所以必须整体 `eval()` 并冻结。
|
||||
|
||||
## Recommended first implementation path
|
||||
- 先实现一个独立 `LEWMViTBackbone` 类，不改现有 `ResNetDiffusionBackbone` 主逻辑。
|
||||
- 再通过新的 hydra backbone/agent 配置接入。
|
||||
- 优先做到“最少侵入 + smoke 可跑 + 远端可训”。
|
||||
@@ -0,0 +1,81 @@
|
||||
# Phase-2 Full-AttnRes Vision Design
|
||||
|
||||
## Goal
|
||||
在当前 roboimi IMF policy 中，把视觉 backbone 里原先由 ResNet BasicBlock/Bottleneck 提供的残差单元全部替换为 AttnRes 风格单元，同时尽量保持现有 agent / cond / rollout / 训练脚本接口不变。
|
||||
|
||||
## User requirement interpretation
|
||||
这里按最严格解释执行：
|
||||
- 不是“在 ResNet 后面再加一个 AttnRes 模块”
|
||||
- 也不是“只在某几个 stage 加 AttnRes 混合”
|
||||
- 而是：视觉主干网络中原本依赖 ResNet residual block 的地方，统一改成 AttnRes residual operator 驱动的 block
|
||||
- 最终仍然输出与现有 `ResNetDiffusionBackbone` 相同的每相机特征接口，以便复用 `SpatialSoftmax -> Linear -> ReLU`、多相机拼接、state concat、IMF head 条件输入
|
||||
|
||||
## Recommended design
|
||||
### Option A (recommended)
|
||||
保留 ResNet 的宏观 stage/stem 结构与通道/步幅规划，但把每个 stage 内的 BasicBlock/Bottleneck 替换为新的 `AttnResImageBlock2D`：
|
||||
- 输入仍是 `(B, C, H, W)` feature map
|
||||
- block 内先把空间维 flatten 成 token 序列 `(B, H*W, C)`
|
||||
- 用二维位置编码 / 可学习位置偏置 + AttnRes self-attention + AttnRes FFN 完成 block 变换
|
||||
- 再 reshape 回 `(B, C, H, W)`
|
||||
- stage 间下采样仍由显式 stride/downsample path 完成
|
||||
|
||||
优点：
|
||||
- 最接近“ResNet 中所有残差都由 AttnRes 代替”的要求
|
||||
- 保留现有视觉输出接口和 cond_dim，不用改 agent/head/data pipeline
|
||||
- 仍可沿用现有多相机编码器框架
|
||||
|
||||
缺点：
|
||||
- 需要新写 2D 版 AttnRes image block，而不是直接复用 1D IMF head block
|
||||
|
||||
### Option B
|
||||
完全移除 ResNet stage，换成 patchify + ViT/AttnRes 图像 transformer，再接 SpatialSoftmax/MLP。
|
||||
|
||||
优点：实现概念更统一。
|
||||
缺点：已经不算“把 ResNet 中残差替换掉”，而是直接换 backbone，和用户要求不完全一致。
|
||||
|
||||
### Option C
|
||||
保留现有 ResNet block，只在 block 外层加 AttnRes mixing。
|
||||
|
||||
不推荐，因为不满足“所有残差均由 AttnRes 替代”。
|
||||
|
||||
## Concrete architecture choice
|
||||
采用 Option A：
|
||||
1. 保留 stem（conv/bn-or-gn/relu/maxpool）与 stage 边界
|
||||
2. 新增 `AttnResImageBlock2D`
|
||||
3. 新增 `AttnResResNetLikeBackbone2D`，负责堆叠 stage/block
|
||||
4. 在 `ResNetDiffusionBackbone` 中增加可选 backbone mode，例如：
|
||||
- `vision_backbone_mode: resnet`
|
||||
- `vision_backbone_mode: attnres_resnet`
|
||||
5. `resnet_imf_attnres` agent 配置新增一个 Phase-2 变体，默认打开 `attnres_resnet`
|
||||
6. 仍保持：
|
||||
- 每相机输出 `64`
|
||||
- 多相机总视觉输出 `3 * 64`
|
||||
- 与 state 拼接后 `cond_dim = 208`
|
||||
|
||||
## Files likely to change
|
||||
- `roboimi/vla/models/backbones/resnet_diffusion.py`
|
||||
- `roboimi/vla/conf/backbone/resnet_diffusion.yaml`
|
||||
- `roboimi/vla/conf/agent/resnet_imf_attnres.yaml`
|
||||
- new: `roboimi/vla/models/backbones/attnres_resnet2d.py`
|
||||
- tests:
|
||||
- new: `tests/test_attnres_resnet2d_backbone.py`
|
||||
- update/add wiring test for agent cond dims
|
||||
|
||||
## Test plan
|
||||
1. New backbone instantiates and forwards `(B,T,C,H,W)` multi-camera input
|
||||
2. Output shape unchanged vs current backbone
|
||||
3. `output_dim == 64`
|
||||
4. 3-camera cond path still yields `208`
|
||||
5. Phase-2 config instantiates full IMF agent successfully
|
||||
6. One short CPU smoke forward for `compute_loss`
|
||||
|
||||
## Phase-2 experiment plan
|
||||
固定使用 Phase-1 最优组合：
|
||||
- `pred_horizon=16`
|
||||
- `num_action_steps=8`
|
||||
|
||||
比较：
|
||||
1. baseline: current IMF head-only AttnRes + original ResNet vision backbone
|
||||
2. phase2: IMF head AttnRes + full AttnRes-replaced vision backbone
|
||||
|
||||
训练超参保持与 Phase-1 最优设置一致，先跑一组 50k step 对比。
|
||||
@@ -0,0 +1,32 @@
|
||||
# ResNet Multitoken IMF Design
|
||||
|
||||
**Status:** user-specified architecture, treated as approved on 2026-04-06.
|
||||
|
||||
## Goal
|
||||
Keep a standard ResNet-18 visual trunk (no AttnRes in vision), but change IMF conditioning from one concatenated multiview token per obs step into three camera-specific condition tokens per obs step.
|
||||
|
||||
## Approved architecture
|
||||
- Vision trunk: standard `resnet18` residual network
|
||||
- Cameras: `front`, `top`, `r_vis`
|
||||
- Each camera uses its **own** ResNet-18 weights (`use_separate_rgb_encoder_per_camera=true`)
|
||||
- Each camera produces one visual token
|
||||
- For each obs step and each camera:
|
||||
1. take that camera visual token
|
||||
2. concatenate robot state
|
||||
3. project to one condition token
|
||||
- IMF input should receive **3 condition tokens per obs step**, not one concatenated token
|
||||
- With `obs_horizon=2`, IMF cond sequence length becomes `2 * 3 = 6`
|
||||
- IMF head remains on the existing IMF/AttnRes implementation path
|
||||
- Vision trunk remains standard ResNet; **no AttnRes vision replacement**
|
||||
|
||||
## Design choices
|
||||
- Extend `ResNetDiffusionBackbone` with an opt-in mode that returns per-camera tokens shaped `(B, T, num_cams, D)` instead of concatenating camera features into `(B, T, num_cams * D)`.
|
||||
- Teach `VLAAgent` to detect multi-token visual features, broadcast state per camera token, apply the existing condition projector on each token, then flatten `(T, num_cams)` into one cond sequence for the IMF head.
|
||||
- Keep `per_step_cond_dim` as the width of a single condition token, and add explicit token-count metadata so transformer heads get the correct cond-sequence length.
|
||||
- For the new experiments, set the condition-token width equal to `n_emb` via `cond_projector.output_dim=${agent.head.n_emb}`.
|
||||
|
||||
## Files expected to change
|
||||
- `roboimi/vla/models/backbones/resnet_diffusion.py`
|
||||
- `roboimi/vla/agent.py`
|
||||
- new Hydra agent config for the multitoken ResNet IMF variant
|
||||
- focused tests in `tests/test_imf_vla_agent.py` and/or `tests/test_resnet_transformer_agent_wiring.py`
|
||||
@@ -0,0 +1,41 @@
|
||||
# SigLIP2 Multiview VLA Design
|
||||
|
||||
**Status:** user-specified architecture, treated as approved on 2026-04-06
|
||||
|
||||
## Goal
|
||||
Replace the current vision encoder for the IMF/AttnRes diffusion policy with a frozen SigLIP2 image encoder while preserving the downstream action-diffusion stack and rollout behavior.
|
||||
|
||||
## Approved architecture
|
||||
- Backbone model: `google/siglip2-base-patch16-256`
|
||||
- Camera inputs: three views, encoded **independently** with a **shared** SigLIP2 vision encoder
|
||||
- Input size:
|
||||
- dataset images stay at native `256x256` (no dataset-side resize)
|
||||
- eval/rollout images resize to `256x256` before SigLIP2 because env renders are larger
|
||||
- Per-view feature: use the global pooled image feature (`pooler_output`, 768-d)
|
||||
- Per-view projection experiments:
|
||||
1. `768 -> 96`
|
||||
2. `768 -> 192`
|
||||
- Conditioning pipeline:
|
||||
1. concatenate 3 projected camera vectors
|
||||
2. concatenate robot state
|
||||
3. project concatenated condition to `384`
|
||||
4. feed that `384`-d per-step condition into the existing IMF/AttnRes diffusion head
|
||||
- Training/run defaults for requested experiments:
|
||||
- `n_emb=384`
|
||||
- `n_layer=12`
|
||||
- `pred_horizon=16`
|
||||
- `num_action_steps=8`
|
||||
- rollout count for validation: keep current requested behavior on this branch unless explicitly overridden later
|
||||
|
||||
## Design decisions
|
||||
- The condition projector lives in `VLAAgent._build_cond()` so the backbone owns only visual features, while the agent owns the final conditioning contract expected by the diffusion head.
|
||||
- The SigLIP2 backbone is frozen by default; only the per-view projectors and downstream policy layers train.
|
||||
- The backbone exposes `dataset_image_resize_shape=None` and `eval_image_resize_shape=(256, 256)` so existing train/eval plumbing can reuse the raw-256 path already added in this branch.
|
||||
- One shared vision encoder is used across cameras to keep memory and download size reasonable and to match the user's request for per-view independent encoding rather than a fused multiview image.
|
||||
|
||||
## Files expected to change
|
||||
- `roboimi/vla/models/backbones/` for the new SigLIP2 backbone
|
||||
- `roboimi/vla/agent.py` for optional post-concat condition projection
|
||||
- Hydra configs under `roboimi/vla/conf/{agent,backbone,modules}`
|
||||
- tests for backbone wiring and agent conditioning dims
|
||||
- remote launch commands/scripts only as needed for training
|
||||
@@ -0,0 +1,63 @@
|
||||
# Phase-1 Final Report and Phase-2 Handoff
|
||||
|
||||
- Finalized: 2026-04-05 00:34:20 CST
|
||||
- Scope: IMF AttnRes policy horizon/action-step grid on `sim_transfer`
|
||||
- Fixed setup: `n_emb=384`, `n_layer=12`, batch size `80`, learning rate `2.5e-4`, `max_steps=50k`, rollout every 5 epochs with 5 episodes, 3 cameras `[r_vis, top, front]`.
|
||||
- Main metric: `checkpoints/vla_model_best.pt` 中记录的训练期最大 `rollout_avg_reward`。
|
||||
|
||||
## Final leaderboard
|
||||
|
||||
| Rank | Run ID | pred_horizon | executed action steps | Best avg_reward | Best step | Final loss |
|
||||
|---:|---|---:|---:|---:|---:|---:|
|
||||
| 1 | `ph16_ex8` | 16 | 8 | **610.8** | 21874 | 0.0034 |
|
||||
| 2 | `ph16_ex16` | 16 | 16 | 561.2 | 48124 | 0.0045 |
|
||||
| 3 | `ph32_ex32` | 32 | 32 | 513.2 | 43749 | 0.0040 |
|
||||
| 4 | `ph8_ex8` | 8 | 8 | 415.6 | 48124 | 0.0070 |
|
||||
| 5 | `ph32_ex8` | 32 | 8 | 361.6 | 43749 | 0.0048 |
|
||||
| 6 | `ph32_ex16` | 32 | 16 | 239.6 | 48124 | 0.0038 |
|
||||
|
||||
## Final conclusions
|
||||
|
||||
1. **最佳组合是 `pred_horizon=16` + `num_action_steps=8`**，最佳平均奖励为 **610.8**，出现在 **step 21874**。
|
||||
2. 在 `pred_horizon=16` 下，执行 8 步优于执行 16 步，优势约 **+8.8%**（610.8 vs 561.2）。
|
||||
3. `pred_horizon=32` 时，对执行步长非常敏感：`32/32` 明显优于 `32/8` 和 `32/16`；特别是 `32/16` 退化最明显。
|
||||
4. 更长的预测窗口并不会自动带来更高 reward；**预测窗口与实际执行窗口的匹配关系** 是关键。
|
||||
5. 最佳 checkpoint 并不在训练结束时出现，而是在 50k 训练中较早的 **21.9k step** 出现，说明 rollout 验证比仅看 train loss 更重要。
|
||||
6. 因而 Phase-2 的比较基线固定为 **`ph16_ex8`**。
|
||||
|
||||
## Recommended baseline for follow-up experiments
|
||||
|
||||
- Baseline run: `ph16_ex8`
|
||||
- Baseline best checkpoint: `step 21874`
|
||||
- Baseline best avg_reward: `610.8`
|
||||
- Baseline run dir: `/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223`
|
||||
|
||||
## Phase-2 target: full-AttnRes vision backbone
|
||||
|
||||
本阶段按你的要求，不再只是 IMF head 中使用 AttnRes，而是把**之前视觉 ResNet 主干中的残差单元全部替换为 AttnRes 残差单元**。当前实现保留了 ResNet 风格的 stage / downsample 宏观结构，但视觉残差 trunk 已切换到 AttnRes：
|
||||
|
||||
- implementation: `roboimi/vla/models/backbones/attnres_resnet2d.py`
|
||||
- wiring: `roboimi/vla/models/backbones/resnet_diffusion.py`
|
||||
- config: `roboimi/vla/conf/backbone/resnet_diffusion.yaml`
|
||||
|
||||
相关代码已提交：
|
||||
|
||||
- `a780068` — headless rollout 修复 + Phase-1 汇总
|
||||
- `2033169` — full-AttnRes vision backbone
|
||||
|
||||
## Phase-2 launch status (observed on 2026-04-05 00:36 CST)
|
||||
|
||||
- Run: `imf-p2-full-attnres-vision-ph16-ex08-emb384-l12-b40-lr1p25e4-ms50k-l20g3-20260405-002424`
|
||||
- Host: `100.119.99.14`, GPU `3`
|
||||
- Config anchor: `pred_horizon=16`, `num_action_steps=8`
|
||||
- Vision backbone: `attnres_resnet`
|
||||
- Because batch size `80` OOMed on both local 5090 and remote L20, Phase-2 currently uses:
|
||||
- batch size: `40`
|
||||
- learning rate: `1.25e-4`
|
||||
- Latest confirmed progress: **step 1300**
|
||||
- First rollout has **not happened yet** at this observation point.
|
||||
- SwanLab: https://swanlab.cn/@game-loader/roboimi-vla/runs/xy7fjdmn0stdr19eu3gub
|
||||
|
||||
## Next action
|
||||
|
||||
继续监控 Phase-2 full-AttnRes 训练，待其完成后直接与 Phase-1 baseline `610.8` 做对比，判断“视觉主干全部替换为 AttnRes”是否优于“仅 IMF 中使用 AttnRes”。
|
||||
@@ -0,0 +1,7 @@
|
||||
rank,run_id,status,pred_horizon,num_action_steps,best_rollout_avg_reward,best_step,final_step,final_loss,host,run_dir,latest_step
|
||||
1,ph16_ex8,running,16,8,610.8,21874,50000,0.0034315965604037046,100.73.14.65,/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223,50000
|
||||
2,ph16_ex16,running,16,16,561.2,48124,50000,0.004544622730463743,100.119.99.14,/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex16-emb384-l12-ms50k-l20g0-20260404-131223,50000
|
||||
3,ph32_ex32,finished,32,32,513.2,43749,50000,0.003953303210437298,local,/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/runs/imf-p1-ph32-ex32-emb384-l12-ms50k-5090-20260404-131223,49900
|
||||
4,ph8_ex8,running,8,8,415.6,48124,50000,0.007008877582848072,100.73.14.65,/home/droid/roboimi_suite_20260404/runs/imf-p1-ph08-ex08-emb384-l12-ms50k-5880g0-20260404-131223,50000
|
||||
5,ph32_ex8,running,32,8,361.6,43749,50000,0.004788532387465239,100.119.99.14,/home/droid/roboimi_suite_20260404/runs/imf-p1-ph32-ex08-emb384-l12-ms50k-l20g1-20260404-131223,50000
|
||||
6,ph32_ex16,running,32,16,239.6,48124,50000,0.0038348555099219084,100.119.99.14,/home/droid/roboimi_suite_20260404/runs/imf-p1-ph32-ex16-emb384-l12-ms50k-l20g2-20260404-131223,50000
|
||||
|
@@ -0,0 +1,115 @@
|
||||
{
|
||||
"suite_name": "2026-04-04-imf-horizon-grid",
|
||||
"created_at": "2026-04-04 13:19:52",
|
||||
"updated_at": "2026-04-04 13:19:52",
|
||||
"phase": "phase1_launching",
|
||||
"metric": "max_avg_reward",
|
||||
"baseline": {
|
||||
"agent": "resnet_imf_attnres",
|
||||
"batch_size": 80,
|
||||
"lr": 0.00025,
|
||||
"num_workers": 12,
|
||||
"max_steps": 50000,
|
||||
"rollout_val_freq_epochs": 5,
|
||||
"rollout_num_episodes": 5,
|
||||
"val_split": 0.0,
|
||||
"seed": 42,
|
||||
"scheduler_type": "cosine",
|
||||
"warmup_steps": 2000,
|
||||
"min_lr": 1e-06,
|
||||
"weight_decay": 1e-05,
|
||||
"grad_clip": 1.0,
|
||||
"inference_steps": 1,
|
||||
"embed_dim": 384,
|
||||
"n_layer": 12,
|
||||
"n_head": 1,
|
||||
"n_kv_head": 1,
|
||||
"freeze_backbone": false,
|
||||
"pretrained_backbone_weights": null,
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top",
|
||||
"front"
|
||||
]
|
||||
},
|
||||
"runs": [
|
||||
{
|
||||
"id": "ph8_ex8",
|
||||
"pred_horizon": 8,
|
||||
"num_action_steps": 8,
|
||||
"host": "100.73.14.65",
|
||||
"host_label": "tailnet-5880",
|
||||
"gpu": 0,
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"python": "/home/droid/miniforge3/envs/roboimi/bin/python",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"run_name": "imf-p1-ph08-ex08-emb384-l12-ms50k-5880g0-20260404-131223",
|
||||
"launch_state": "ready"
|
||||
},
|
||||
{
|
||||
"id": "ph16_ex8",
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"host": "100.73.14.65",
|
||||
"host_label": "tailnet-5880",
|
||||
"gpu": 1,
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"python": "/home/droid/miniforge3/envs/roboimi/bin/python",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"run_name": "imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223",
|
||||
"launch_state": "ready"
|
||||
},
|
||||
{
|
||||
"id": "ph16_ex16",
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 16,
|
||||
"host": "100.119.99.14",
|
||||
"host_label": "tailnet-l20",
|
||||
"gpu": 0,
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"python": "/home/droid/miniforge3/envs/roboimi/bin/python",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"run_name": "imf-p1-ph16-ex16-emb384-l12-ms50k-l20g0-20260404-131223",
|
||||
"launch_state": "provisioning_required"
|
||||
},
|
||||
{
|
||||
"id": "ph32_ex8",
|
||||
"pred_horizon": 32,
|
||||
"num_action_steps": 8,
|
||||
"host": "100.119.99.14",
|
||||
"host_label": "tailnet-l20",
|
||||
"gpu": 1,
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"python": "/home/droid/miniforge3/envs/roboimi/bin/python",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"run_name": "imf-p1-ph32-ex08-emb384-l12-ms50k-l20g1-20260404-131223",
|
||||
"launch_state": "provisioning_required"
|
||||
},
|
||||
{
|
||||
"id": "ph32_ex16",
|
||||
"pred_horizon": 32,
|
||||
"num_action_steps": 16,
|
||||
"host": "100.119.99.14",
|
||||
"host_label": "tailnet-l20",
|
||||
"gpu": 2,
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"python": "/home/droid/miniforge3/envs/roboimi/bin/python",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"run_name": "imf-p1-ph32-ex16-emb384-l12-ms50k-l20g2-20260404-131223",
|
||||
"launch_state": "provisioning_required"
|
||||
},
|
||||
{
|
||||
"id": "ph32_ex32",
|
||||
"pred_horizon": 32,
|
||||
"num_action_steps": 32,
|
||||
"host": "local",
|
||||
"host_label": "local-5090",
|
||||
"gpu": 0,
|
||||
"workdir": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy",
|
||||
"python": "/home/droid/.conda/envs/roboimi/bin/python",
|
||||
"dataset_dir": "/home/droid/project/diana_sim/sim_transfer",
|
||||
"run_name": "imf-p1-ph32-ex32-emb384-l12-ms50k-5090-20260404-131223",
|
||||
"launch_state": "ready"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
# IMF Horizon Grid Suite Notes
|
||||
|
||||
- Created: 2026-04-04 13:19:52
|
||||
- Phase-1 matrix: (8,8), (16,8), (16,16), (32,8), (32,16), (32,32)
|
||||
- Fixed baseline: IMF AttnRes, n_emb=384, n_layer=12, batch_size=80, lr=2.5e-4, max_steps=50k, rollout every 5 epochs with 5 episodes.
|
||||
- Host allocation:
|
||||
- local RTX 5090: ph32_ex32
|
||||
- 100.73.14.65 RTX 5880 GPU0: ph8_ex8
|
||||
- 100.73.14.65 RTX 5880 GPU1: ph16_ex8
|
||||
- 100.119.99.14 L20 GPU0: ph16_ex16
|
||||
- 100.119.99.14 L20 GPU1: ph32_ex8
|
||||
- 100.119.99.14 L20 GPU2: ph32_ex16
|
||||
- 100.119.99.14 still needs env + dataset + swanlab credential copy before launch.
|
||||
|
||||
- 2026-04-04 13:23:43: launched local ph32_ex32 (pid 1437836), remote 100.73 ph8_ex8 (pid 931824), ph16_ex8 (pid 931826); started 100.119 bootstrap (local pid 1437837).
|
||||
- 2026-04-04 13:25:43: first status sync — local ph32_ex32 step≈500; remote ph8_ex8 step≈400; remote ph16_ex8 step≈400.
|
||||
- 2026-04-04 13:27:41: second status sync — 100.119 bootstrap finished env copy and entered dataset copy; local ph32_ex32 step≈900; remote ph8_ex8 step≈800; remote ph16_ex8 step≈800.
|
||||
- 2026-04-04 13:35:31: 100.119 bootstrap data/env copy finished. Original validation command hit a quoting bug, then I manually revalidated torch+mujoco+swanlab and launched ph16_ex16/ph32_ex8/ph32_ex16 with pids 81129/81130/81131.
|
||||
- 2026-04-04 13:37:36: all 6 Phase-1 runs are now up. SwanLab links recorded in status.json; latest observed steps ~ local 900 / 5880 runs 800 / L20 runs 100.
|
||||
- 2026-04-04 14:41:08: diagnosed remote first-rollout crash as early mujoco import before MUJOCO_GL=egl in eval_vla.py via raw_action_trajectory_viewer. Added regression test tests/test_eval_vla_headless_import.py, fixed import to lazy-load, verified 20-step headless eval on 5880 and L20, then resumed 5 failed runs from step 4374. Current resumed pids: ph8_ex8=938714, ph16_ex8=938717, ph16_ex16=90169, ph32_ex8=90173, ph32_ex16=90175.
|
||||
@@ -0,0 +1,38 @@
|
||||
# Phase-1 IMF Horizon Grid Summary
|
||||
|
||||
- Generated: 2026-04-04 23:43:38
|
||||
- Fixed baseline: IMF AttnRes head, n_emb=384, n_layer=12, batch_size=80, lr=2.5e-4, max_steps=50k, rollout every 5 epochs with 5 episodes, 3 cameras `[r_vis, top, front]`.
|
||||
- Primary metric: `checkpoints/vla_model_best.pt -> rollout_avg_reward` (max training-time rollout average reward).
|
||||
|
||||
## Ranked results
|
||||
|
||||
| Rank | Run ID | pred_horizon | num_action_steps | Best avg_reward | Best step | Final loss | Host |
|
||||
|---:|---|---:|---:|---:|---:|---:|---|
|
||||
| 1 | `ph16_ex8` | 16 | 8 | 610.8 | 21874 | 0.0034 | 100.73.14.65 |
|
||||
| 2 | `ph16_ex16` | 16 | 16 | 561.2 | 48124 | 0.0045 | 100.119.99.14 |
|
||||
| 3 | `ph32_ex32` | 32 | 32 | 513.2 | 43749 | 0.0040 | local |
|
||||
| 4 | `ph8_ex8` | 8 | 8 | 415.6 | 48124 | 0.0070 | 100.73.14.65 |
|
||||
| 5 | `ph32_ex8` | 32 | 8 | 361.6 | 43749 | 0.0048 | 100.119.99.14 |
|
||||
| 6 | `ph32_ex16` | 32 | 16 | 239.6 | 48124 | 0.0038 | 100.119.99.14 |
|
||||
|
||||
## Main observations
|
||||
|
||||
- Best overall setting was **`pred_horizon=16`, `num_action_steps=8`** with **max avg_reward = 610.8** at step **21874**.
|
||||
- Comparing horizon 16: executing 8 steps outperformed executing 16 steps (`ph16_ex8` > `ph16_ex16`).
|
||||
- Comparing horizon 32: executing the full 32-step chunk was much better than executing 16 or 8 steps (`ph32_ex32` > `ph32_ex8` > `ph32_ex16`).
|
||||
- Short horizon 8 with 8-step execution was competitive but clearly below the best 16/8 and 32/32 settings.
|
||||
- In this sweep, increasing prediction horizon helped only when the executed chunk length matched a good control cadence; mismatch could hurt a lot (especially `ph32_ex16`).
|
||||
|
||||
## Raw results
|
||||
|
||||
- `ph16_ex8`: best avg_reward=610.8 @ step 21874, final_loss=0.0034, run_dir=`/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223`
|
||||
- `ph16_ex16`: best avg_reward=561.2 @ step 48124, final_loss=0.0045, run_dir=`/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex16-emb384-l12-ms50k-l20g0-20260404-131223`
|
||||
- `ph32_ex32`: best avg_reward=513.2 @ step 43749, final_loss=0.0040, run_dir=`/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/runs/imf-p1-ph32-ex32-emb384-l12-ms50k-5090-20260404-131223`
|
||||
- `ph8_ex8`: best avg_reward=415.6 @ step 48124, final_loss=0.0070, run_dir=`/home/droid/roboimi_suite_20260404/runs/imf-p1-ph08-ex08-emb384-l12-ms50k-5880g0-20260404-131223`
|
||||
- `ph32_ex8`: best avg_reward=361.6 @ step 43749, final_loss=0.0048, run_dir=`/home/droid/roboimi_suite_20260404/runs/imf-p1-ph32-ex08-emb384-l12-ms50k-l20g1-20260404-131223`
|
||||
- `ph32_ex16`: best avg_reward=239.6 @ step 48124, final_loss=0.0038, run_dir=`/home/droid/roboimi_suite_20260404/runs/imf-p1-ph32-ex16-emb384-l12-ms50k-l20g2-20260404-131223`
|
||||
|
||||
## Recommendation for Phase-2 anchor
|
||||
|
||||
- Use **`pred_horizon=16`, `num_action_steps=8`** as the strongest Phase-1 baseline if the goal is purely maximizing rollout reward.
|
||||
- If phase-2 needs a more conservative action execution budget, `ph16_ex8` is the strongest non-full-32 execution setting and may still be a good comparison anchor.
|
||||
@@ -0,0 +1,167 @@
|
||||
{
|
||||
"suite_name": "2026-04-04-imf-horizon-grid",
|
||||
"updated_at": "2026-04-05 00:34:20",
|
||||
"phase": "phase1_completed",
|
||||
"provisioning": {
|
||||
"100.119.99.14": {
|
||||
"state": "completed_manual_launch",
|
||||
"bootstrap_pid_local": 1437837,
|
||||
"log_path": "experiment_suites/2026-04-04-imf-horizon-grid/provision_logs/100.119.99.14-bootstrap-20260404-131223.log",
|
||||
"env_copy": "completed",
|
||||
"dataset_copy": "completed",
|
||||
"launch_watcher_pid_local": null,
|
||||
"launch_watcher_log": "experiment_suites/2026-04-04-imf-horizon-grid/launch_logs/100.119.99.14-launch-watcher-20260404-131223.log",
|
||||
"swanlab_copy": "completed",
|
||||
"bootstrap_validation_note": "initial validation command had a quoting bug; manual validation passed and launches were started successfully"
|
||||
}
|
||||
},
|
||||
"runs": {
|
||||
"ph8_ex8": {
|
||||
"status": "finished",
|
||||
"host": "100.73.14.65",
|
||||
"gpu": 0,
|
||||
"run_name": "imf-p1-ph08-ex08-emb384-l12-ms50k-5880g0-20260404-131223",
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph08-ex08-emb384-l12-ms50k-5880g0-20260404-131223/train_vla.log",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph08-ex08-emb384-l12-ms50k-5880g0-20260404-131223",
|
||||
"pred_horizon": 8,
|
||||
"num_action_steps": 8,
|
||||
"pid": 938714,
|
||||
"launch_log": "experiment_suite_launch_logs/imf-p1-ph08-ex08-emb384-l12-ms50k-5880g0-20260404-131223.restartfix-20260404-143827.log",
|
||||
"latest_step": 50000,
|
||||
"latest_log_sync": "2026-04-05 00:34:20",
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/i5syc57b6zq7rbkrtqy7b",
|
||||
"process_running": false,
|
||||
"best_step": 48124,
|
||||
"best_rollout_avg_reward": 415.6,
|
||||
"final_loss": 0.007008877582848072
|
||||
},
|
||||
"ph16_ex8": {
|
||||
"status": "finished",
|
||||
"host": "100.73.14.65",
|
||||
"gpu": 1,
|
||||
"run_name": "imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223",
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223/train_vla.log",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223",
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"pid": 938717,
|
||||
"launch_log": "experiment_suite_launch_logs/imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223.restartfix-20260404-143827.log",
|
||||
"latest_step": 50000,
|
||||
"latest_log_sync": "2026-04-05 00:34:20",
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/4rusbrpfxmw4ffii1ul5w",
|
||||
"process_running": false,
|
||||
"best_step": 21874,
|
||||
"best_rollout_avg_reward": 610.8,
|
||||
"final_loss": 0.0034315965604037046
|
||||
},
|
||||
"ph16_ex16": {
|
||||
"status": "finished",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 0,
|
||||
"run_name": "imf-p1-ph16-ex16-emb384-l12-ms50k-l20g0-20260404-131223",
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex16-emb384-l12-ms50k-l20g0-20260404-131223/train_vla.log",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex16-emb384-l12-ms50k-l20g0-20260404-131223",
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 16,
|
||||
"pid": 90169,
|
||||
"launch_log": "experiment_suite_launch_logs/imf-p1-ph16-ex16-emb384-l12-ms50k-l20g0-20260404-131223.restartfix-20260404-143827.log",
|
||||
"latest_log_sync": "2026-04-05 00:34:20",
|
||||
"latest_step": 50000,
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/wwm232k6190gexnze8mg6",
|
||||
"process_running": false,
|
||||
"best_step": 48124,
|
||||
"best_rollout_avg_reward": 561.2,
|
||||
"final_loss": 0.004544622730463743
|
||||
},
|
||||
"ph32_ex8": {
|
||||
"status": "finished",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 1,
|
||||
"run_name": "imf-p1-ph32-ex08-emb384-l12-ms50k-l20g1-20260404-131223",
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph32-ex08-emb384-l12-ms50k-l20g1-20260404-131223/train_vla.log",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph32-ex08-emb384-l12-ms50k-l20g1-20260404-131223",
|
||||
"pred_horizon": 32,
|
||||
"num_action_steps": 8,
|
||||
"pid": 90173,
|
||||
"launch_log": "experiment_suite_launch_logs/imf-p1-ph32-ex08-emb384-l12-ms50k-l20g1-20260404-131223.restartfix-20260404-143827.log",
|
||||
"latest_log_sync": "2026-04-05 00:34:20",
|
||||
"latest_step": 50000,
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/o5y2xjb2rsb3lmfcuhy4p",
|
||||
"process_running": false,
|
||||
"best_step": 43749,
|
||||
"best_rollout_avg_reward": 361.6,
|
||||
"final_loss": 0.004788532387465239
|
||||
},
|
||||
"ph32_ex16": {
|
||||
"status": "finished",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 2,
|
||||
"run_name": "imf-p1-ph32-ex16-emb384-l12-ms50k-l20g2-20260404-131223",
|
||||
"workdir": "/home/droid/roboimi_suite_20260404",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph32-ex16-emb384-l12-ms50k-l20g2-20260404-131223/train_vla.log",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph32-ex16-emb384-l12-ms50k-l20g2-20260404-131223",
|
||||
"pred_horizon": 32,
|
||||
"num_action_steps": 16,
|
||||
"pid": 90175,
|
||||
"launch_log": "experiment_suite_launch_logs/imf-p1-ph32-ex16-emb384-l12-ms50k-l20g2-20260404-131223.restartfix-20260404-143827.log",
|
||||
"latest_log_sync": "2026-04-05 00:34:20",
|
||||
"latest_step": 50000,
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/54cjpgba9eqsopdm0l8d3",
|
||||
"process_running": false,
|
||||
"best_step": 48124,
|
||||
"best_rollout_avg_reward": 239.6,
|
||||
"final_loss": 0.0038348555099219084
|
||||
},
|
||||
"ph32_ex32": {
|
||||
"status": "finished",
|
||||
"host": "local",
|
||||
"gpu": 0,
|
||||
"run_name": "imf-p1-ph32-ex32-emb384-l12-ms50k-5090-20260404-131223",
|
||||
"workdir": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy",
|
||||
"dataset_dir": "/home/droid/project/diana_sim/sim_transfer",
|
||||
"log_path": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/runs/imf-p1-ph32-ex32-emb384-l12-ms50k-5090-20260404-131223/train_vla.log",
|
||||
"run_dir": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/runs/imf-p1-ph32-ex32-emb384-l12-ms50k-5090-20260404-131223",
|
||||
"pred_horizon": 32,
|
||||
"num_action_steps": 32,
|
||||
"pid": 1437836,
|
||||
"launch_log": "experiment_suites/2026-04-04-imf-horizon-grid/launch_logs/imf-p1-ph32-ex32-emb384-l12-ms50k-5090-20260404-131223.launch.log",
|
||||
"latest_step": 49900,
|
||||
"latest_log_sync": "2026-04-05 00:34:20",
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/ajs2m218jd260hawhy5ns",
|
||||
"process_running": false,
|
||||
"latest_rollout_avg_reward": 513.2,
|
||||
"best_rollout_avg_reward": 513.2,
|
||||
"best_step": 43749,
|
||||
"final_loss": 0.003953303210437298
|
||||
}
|
||||
},
|
||||
"monitor": {
|
||||
"state": "stopped",
|
||||
"pid_local": null,
|
||||
"log_path": "experiment_suites/2026-04-04-imf-horizon-grid/monitor_logs/status-sync-20260404-131223.log",
|
||||
"interval_seconds": 300,
|
||||
"stopped_at": "2026-04-05 00:34:20",
|
||||
"stop_reason": "phase1 suite finalized after all six runs completed"
|
||||
},
|
||||
"debug": {
|
||||
"remote_rollout_failure_20260404": {
|
||||
"root_cause": "eval_vla.py imported raw_action_trajectory_viewer at module import time, which imported mujoco before MUJOCO_GL=egl was set; remote headless rollout then fell back to GLFW/X11 and crashed with mujoco.FatalError: gladLoadGL error during env.reset()->mj.Renderer(...)",
|
||||
"fixed_file": "roboimi/demos/vla_scripts/eval_vla.py",
|
||||
"verification": {
|
||||
"pytest": "tests/test_eval_vla_headless_import.py passed",
|
||||
"remote_eval_5880": "1 episode x 20 steps headless eval passed",
|
||||
"remote_eval_l20": "1 episode x 20 steps headless eval passed"
|
||||
}
|
||||
}
|
||||
},
|
||||
"phase1_summary_md": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/experiment_suites/2026-04-04-imf-horizon-grid/phase1_summary.md"
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
# Camera Ablation Summary (`pred_horizon=16`, `num_action_steps=8`, ResNet IMF)
|
||||
|
||||
- Generated: 2026-04-05
|
||||
- Common setup: original ResNet vision backbone, `n_emb=384`, `n_layer=12`, `batch_size=80`, `lr=2.5e-4`, `max_steps=50k`, rollout every 5 epochs with 5 episodes, headless eval.
|
||||
- Metric for comparison: `checkpoints/vla_model_best.pt -> rollout_avg_reward`.
|
||||
|
||||
## Leaderboard
|
||||
|
||||
| Rank | Cameras | Best avg_reward | Best step | Final loss | Run name |
|
||||
|---:|---|---:|---:|---:|---|
|
||||
| 1 | `top + front` | **274.8** | 48124 | 0.0056 | `imf-resnet-topfront-2cam-ph16-ex08-emb384-l12-ms50k-5090-20260405-085023` |
|
||||
| 2 | `top` | **271.2** | 43749 | 0.0052 | `imf-resnet-top-1cam-ph16-ex08-emb384-l12-ms50k-l20g4-20260405-125844` |
|
||||
| 3 | `r_vis + front` | **244.0** | 21874 | 0.0043 | `imf-resnet-frontrvis-2cam-ph16-ex08-emb384-l12-ms50k-l20g1-20260405-102029` |
|
||||
| 4 | `r_vis` | **6.4** | 17499 | 0.0047 | `imf-resnet-rvis-1cam-ph16-ex08-emb384-l12-ms50k-l20g3-20260405-125844` |
|
||||
| 5 | `r_vis + top` | **1.2** | 4374 | 0.0047 | `imf-resnet-rvistop-2cam-ph16-ex08-emb384-l12-ms50k-l20g2-20260405-125844` |
|
||||
| 6 | `front` | **0.0** | 4374 | 0.0074 | `imf-resnet-front-1cam-ph16-ex08-emb384-l12-ms50k-l20g0-20260405-095607` |
|
||||
|
||||
## Main takeaways
|
||||
|
||||
1. **`top` 是最关键的单相机视角**：`top only = 271.2`，几乎与 `top + front = 274.8` 持平。
|
||||
2. **`front` 单独几乎没有效用**：`front only = 0.0`。
|
||||
3. **`r_vis` 单独也基本无效**：`r_vis only = 6.4`。
|
||||
4. **`r_vis + front` 可以显著优于单独 `front` / `r_vis`**，说明这两个视角有一定互补性，但仍明显弱于任何包含 `top` 且表现正常的配置。
|
||||
5. **`r_vis + top` 的结果异常差**：只有 `1.2`，远低于 `top only = 271.2`。这说明简单加入 `r_vis` 并不保证增益，甚至可能破坏当前设置下的学习。
|
||||
6. **训练 loss 与 rollout reward 明显不一致**：例如 `r_vis + top` 和 `r_vis only` 的 final loss 都不高，但 reward 很差，因此本组实验必须以 rollout reward 而不是 loss 选型。
|
||||
|
||||
## Horizontal comparison views
|
||||
|
||||
### Single-camera comparison
|
||||
|
||||
- `top`: **271.2**
|
||||
- `r_vis`: **6.4**
|
||||
- `front`: **0.0**
|
||||
|
||||
结论：**`top >>> r_vis > front`**。
|
||||
|
||||
### Two-camera comparison
|
||||
|
||||
- `top + front`: **274.8**
|
||||
- `r_vis + front`: **244.0**
|
||||
- `r_vis + top`: **1.2**
|
||||
|
||||
结论：
|
||||
- **最稳妥的双相机组合是 `top + front`**。
|
||||
- `r_vis + front` 有效，但不如 `top + front`。
|
||||
- `r_vis + top` 在当前设置下几乎失效。
|
||||
|
||||
### Incremental effect of adding a second view
|
||||
|
||||
- 在 `top` 基础上加 `front`：`271.2 -> 274.8`，**增益很小**。
|
||||
- 在 `front` 基础上加 `r_vis`：`0.0 -> 244.0`，**增益很大**。
|
||||
- 在 `top` 基础上加 `r_vis`：`271.2 -> 1.2`，**显著退化**。
|
||||
|
||||
## Practical recommendation
|
||||
|
||||
如果只从这 6 个实验里选：
|
||||
|
||||
- **首选**：`top + front`
|
||||
- **次选**：`top only`
|
||||
- 如果必须不用 `top`：`r_vis + front` 明显优于 `front only` / `r_vis only`
|
||||
- **不建议**：`r_vis + top`
|
||||
|
||||
## Note relative to previous 3-camera baseline
|
||||
|
||||
此前 3 相机 `[r_vis, top, front]` 的最佳 reward 为 **610.8**。
|
||||
因此这次 6 个 camera ablation 的最佳结果（`top + front = 274.8`）说明：
|
||||
|
||||
- 当前这个训练批次里，**去掉任意一个视角都会显著低于之前的 3 相机最优结果**；
|
||||
- 但在去掉视角的约束下，**`top` 仍然是最核心的保留对象**。
|
||||
@@ -0,0 +1,8 @@
|
||||
# CHECKLIST
|
||||
|
||||
- [x] Confirm remote free GPU
|
||||
- [x] Create front-only run contract
|
||||
- [x] Remote smoke test passes
|
||||
- [x] Launch 50k run on remote GPU0
|
||||
- [x] Record pid / log / SwanLab
|
||||
- [x] Report status back to user
|
||||
@@ -0,0 +1,28 @@
|
||||
# PLAN
|
||||
|
||||
## Goal
|
||||
Train a 50k-step IMF baseline with the original ResNet vision backbone, using only the `front` camera as image conditioning.
|
||||
|
||||
## Fixed comparison contract
|
||||
- Same as the active `top/front` run except image input is reduced to `[front]`
|
||||
- Agent: `resnet_imf_attnres`
|
||||
- Vision backbone mode: `resnet`
|
||||
- `pred_horizon=16`, `num_action_steps=8`
|
||||
- `n_emb=384`, `n_layer=12`, `n_head=1`, `n_kv_head=1`
|
||||
- `inference_steps=1`
|
||||
- `batch_size=80`, `lr=2.5e-4`, cosine, warmup=2000
|
||||
- dataset: `/home/droid/sim_dataset/sim_transfer`
|
||||
- cameras: `[front]` only
|
||||
- rollout every 5 epochs with 5 episodes, headless
|
||||
|
||||
## Resource plan
|
||||
- Host: `100.119.99.14`
|
||||
- GPU: `0`
|
||||
|
||||
## Important dimension override
|
||||
- Single-camera visual cond dim = `64 + 16 = 80`, so override `agent.head.cond_dim=80` and `agent.num_cams=1`.
|
||||
|
||||
## Execution path
|
||||
1. 2-step smoke test on remote GPU0.
|
||||
2. If smoke passes, launch 50k main run with SwanLab.
|
||||
3. Record pid / run_dir / log / URL locally.
|
||||
@@ -0,0 +1,6 @@
|
||||
# Notes
|
||||
|
||||
- 2026-04-05 09:55:27: remote 2-step smoke passed on `100.119.99.14` GPU0 with `front` only, batch=80, no OOM.
|
||||
- 2026-04-05 09:56:26: launched main run `imf-resnet-front-1cam-ph16-ex08-emb384-l12-ms50k-l20g0-20260405-095607`.
|
||||
- 2026-04-05 09:57:36: confirmed training is stable through step 200, latest loss 0.2830.
|
||||
- SwanLab: https://swanlab.cn/@game-loader/roboimi-vla/runs/7kdii8oc6tjkcyu5y0lwq
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"suite_name": "2026-04-05-front-only-resnet-1cam",
|
||||
"updated_at": "2026-04-05 09:57:36",
|
||||
"phase": "running",
|
||||
"baseline_reference": {
|
||||
"source_run": "imf-resnet-topfront-2cam-ph16-ex08-emb384-l12-ms50k-5090-20260405-085023",
|
||||
"notes": "Same hyperparameters as the active top/front run, but image input is reduced to [front] only."
|
||||
},
|
||||
"smoke_test": {
|
||||
"status": "passed",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 0,
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/smoke-frontonly-resnet-ph16-ex08-20260405-095509",
|
||||
"batch_size": 80,
|
||||
"max_steps": 2,
|
||||
"note": "2-step remote CUDA smoke passed on L20 GPU0 without OOM."
|
||||
},
|
||||
"main_run": {
|
||||
"status": "running",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 0,
|
||||
"launch_pid": 158874,
|
||||
"pid": 158877,
|
||||
"run_name": "imf-resnet-front-1cam-ph16-ex08-emb384-l12-ms50k-l20g0-20260405-095607",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-front-1cam-ph16-ex08-emb384-l12-ms50k-l20g0-20260405-095607",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-front-1cam-ph16-ex08-emb384-l12-ms50k-l20g0-20260405-095607/train_vla.log",
|
||||
"launch_log": "/home/droid/roboimi_suite_20260404/experiment_suite_launch_logs/imf-resnet-front-1cam-ph16-ex08-emb384-l12-ms50k-l20g0-20260405-095607.launch.log",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"camera_names": [
|
||||
"front"
|
||||
],
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"head_cond_dim": 80,
|
||||
"head_n_emb": 384,
|
||||
"head_n_layer": 12,
|
||||
"vision_backbone_mode": "resnet",
|
||||
"pretrained_backbone_weights": null,
|
||||
"freeze_backbone": false,
|
||||
"batch_size": 80,
|
||||
"lr": 0.00025,
|
||||
"num_workers": 12,
|
||||
"max_steps": 50000,
|
||||
"rollout_val_freq_epochs": 5,
|
||||
"rollout_num_episodes": 5,
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/7kdii8oc6tjkcyu5y0lwq",
|
||||
"latest_step": 200,
|
||||
"latest_loss": 0.283,
|
||||
"process_running": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
# CHECKLIST
|
||||
|
||||
- [x] Confirm camera mapping (`right` -> `r_vis`)
|
||||
- [x] Create front+r_vis run contract
|
||||
- [x] Remote smoke test passes
|
||||
- [x] Launch 50k run on remote GPU1
|
||||
- [x] Record pid / log / SwanLab
|
||||
- [x] Report status back to user
|
||||
@@ -0,0 +1,23 @@
|
||||
# PLAN
|
||||
|
||||
## Goal
|
||||
Train a 50k-step IMF baseline with the original ResNet vision backbone, using `front` + `r_vis` cameras only.
|
||||
|
||||
## Fixed comparison contract
|
||||
- Same hyperparameters as the active top/front and front-only runs
|
||||
- Agent: `resnet_imf_attnres`
|
||||
- Vision backbone mode: `resnet`
|
||||
- `pred_horizon=16`, `num_action_steps=8`
|
||||
- `n_emb=384`, `n_layer=12`, `n_head=1`, `n_kv_head=1`
|
||||
- `inference_steps=1`
|
||||
- `batch_size=80`, `lr=2.5e-4`, cosine warmup 2000
|
||||
- dataset: `/home/droid/sim_dataset/sim_transfer`
|
||||
- cameras: `[r_vis, front]`
|
||||
- rollout every 5 epochs with 5 episodes, headless
|
||||
|
||||
## Important dimension override
|
||||
- Two-camera visual cond dim = `64*2 + 16 = 144`, so set `agent.num_cams=2`, `agent.head.cond_dim=144`.
|
||||
|
||||
## Resource plan
|
||||
- Host: `100.119.99.14`
|
||||
- GPU: `1`
|
||||
@@ -0,0 +1,6 @@
|
||||
# Notes
|
||||
|
||||
- 2026-04-05 10:20:09: remote 2-step smoke passed on `100.119.99.14` GPU1 with `r_vis + front`, batch=80, no OOM.
|
||||
- 2026-04-05 10:20:49: launched main run `imf-resnet-frontrvis-2cam-ph16-ex08-emb384-l12-ms50k-l20g1-20260405-102029`.
|
||||
- 2026-04-05 10:22:03: confirmed training is stable through step 200, latest loss 0.3321.
|
||||
- SwanLab: https://swanlab.cn/@game-loader/roboimi-vla/runs/3fyzjfdcbiq7frtbqv6ss
|
||||
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"suite_name": "2026-04-05-front-rvis-resnet-2cam",
|
||||
"updated_at": "2026-04-05 10:22:03",
|
||||
"phase": "running",
|
||||
"interpretation": {
|
||||
"right_camera_name": "r_vis"
|
||||
},
|
||||
"baseline_reference": {
|
||||
"source_run": "imf-resnet-topfront-2cam-ph16-ex08-emb384-l12-ms50k-5090-20260405-085023",
|
||||
"notes": "Same hyperparameters as the active top/front run, replacing top with r_vis."
|
||||
},
|
||||
"smoke_test": {
|
||||
"status": "passed",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 1,
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/smoke-frontrvis-resnet-ph16-ex08-20260405-102001",
|
||||
"batch_size": 80,
|
||||
"max_steps": 2,
|
||||
"note": "2-step remote CUDA smoke passed on L20 GPU1 without OOM."
|
||||
},
|
||||
"main_run": {
|
||||
"status": "running",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 1,
|
||||
"launch_pid": 159910,
|
||||
"pid": 159913,
|
||||
"run_name": "imf-resnet-frontrvis-2cam-ph16-ex08-emb384-l12-ms50k-l20g1-20260405-102029",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-frontrvis-2cam-ph16-ex08-emb384-l12-ms50k-l20g1-20260405-102029",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-frontrvis-2cam-ph16-ex08-emb384-l12-ms50k-l20g1-20260405-102029/train_vla.log",
|
||||
"launch_log": "/home/droid/roboimi_suite_20260404/experiment_suite_launch_logs/imf-resnet-frontrvis-2cam-ph16-ex08-emb384-l12-ms50k-l20g1-20260405-102029.launch.log",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"front"
|
||||
],
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"head_cond_dim": 144,
|
||||
"head_n_emb": 384,
|
||||
"head_n_layer": 12,
|
||||
"vision_backbone_mode": "resnet",
|
||||
"pretrained_backbone_weights": null,
|
||||
"freeze_backbone": false,
|
||||
"batch_size": 80,
|
||||
"lr": 0.00025,
|
||||
"num_workers": 12,
|
||||
"max_steps": 50000,
|
||||
"rollout_val_freq_epochs": 5,
|
||||
"rollout_num_episodes": 5,
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/3fyzjfdcbiq7frtbqv6ss",
|
||||
"latest_step": 200,
|
||||
"latest_loss": 0.3321,
|
||||
"process_running": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"suite_name": "2026-04-05-full-attnres-vision-phase2",
|
||||
"created_at": "2026-04-05 00:12:14",
|
||||
"phase": "phase2_running",
|
||||
"baseline_reference": {
|
||||
"run_id": "ph16_ex8",
|
||||
"best_rollout_avg_reward": 610.8,
|
||||
"best_step": 21874,
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223"
|
||||
},
|
||||
"candidate": {
|
||||
"run_name": "imf-p2-full-attnres-vision-ph16-ex08-emb384-l12-ms50k-20260405-001214",
|
||||
"host": "local",
|
||||
"gpu": 0,
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"vision_backbone_mode": "attnres_resnet",
|
||||
"notes": "Full-AttnRes vision backbone replacing ResNet residual units; IMF head unchanged."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
# Full-AttnRes Vision Phase-2
|
||||
|
||||
- Created: 2026-04-05 00:12:14
|
||||
- Baseline reference: ph16_ex8 best avg_reward=610.8
|
||||
- Candidate run: imf-p2-full-attnres-vision-ph16-ex08-emb384-l12-ms50k-20260405-001214
|
||||
- 2026-04-05 00:23:03: batch=80 OOM on both 5090 and L20; using validated fallback batch=40, lr=1.25e-4 on remote L20 GPU3.
|
||||
- 2026-04-05 00:24:24: launching candidate imf-p2-full-attnres-vision-ph16-ex08-emb384-l12-b40-lr1p25e4-ms50k-l20g3-20260405-002424 on 100.119.99.14 GPU3 with batch=40 lr=1.25e-4.
|
||||
- 2026-04-05 00:27:17: remote phase2 run is active on 100.119.99.14 GPU3, validated at least to step 200. SwanLab: https://swanlab.cn/@game-loader/roboimi-vla/runs/xy7fjdmn0stdr19eu3gub
|
||||
- 2026-04-05 00:36:54: latest confirmed progress is step 1300 on 100.119.99.14 GPU3; first rollout not reached yet.
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"suite_name": "2026-04-05-full-attnres-vision-phase2",
|
||||
"updated_at": "2026-04-05 00:36:54",
|
||||
"phase": "phase2_running",
|
||||
"baseline_reference": {
|
||||
"run_id": "ph16_ex8",
|
||||
"best_rollout_avg_reward": 610.8,
|
||||
"best_step": 21874,
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223"
|
||||
},
|
||||
"candidate": {
|
||||
"run_name": "imf-p2-full-attnres-vision-ph16-ex08-emb384-l12-b40-lr1p25e4-ms50k-l20g3-20260405-002424",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 3,
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"vision_backbone_mode": "attnres_resnet",
|
||||
"notes": "Full-AttnRes vision backbone replacing ResNet residual units; IMF head unchanged.",
|
||||
"status": "running",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-p2-full-attnres-vision-ph16-ex08-emb384-l12-b40-lr1p25e4-ms50k-l20g3-20260405-002424",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-p2-full-attnres-vision-ph16-ex08-emb384-l12-b40-lr1p25e4-ms50k-l20g3-20260405-002424/train_vla.log",
|
||||
"pid": 151187,
|
||||
"batch_size": 40,
|
||||
"lr": 0.000125,
|
||||
"num_workers": 12,
|
||||
"launch_log": "/home/droid/roboimi_suite_20260404/experiment_suite_launch_logs/imf-p2-full-attnres-vision-ph16-ex08-emb384-l12-b40-lr1p25e4-ms50k-l20g3-20260405-002424.launch.log",
|
||||
"note": "Local 5090 and remote L20 both OOM at batch=80; switched to batch=40 and linearly scaled lr to 1.25e-4 after smoke validation on L20.",
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/xy7fjdmn0stdr19eu3gub",
|
||||
"latest_step": 1300,
|
||||
"latest_log_sync": "2026-04-05 00:36:54"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"date": "2026-04-06",
|
||||
"branch": "feat-imf-attnres-policy",
|
||||
"worktree": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy",
|
||||
"model": "LEWM ViT frozen visual encoder + IMF AttnRes diffusion head",
|
||||
"checkpoint_path": "/home/droid/le-wm/lewm-sim-transfer/pa1w85md8jop6bvol8oxp/checkpoints/epoch=99-step=47800.ckpt",
|
||||
"visual_contract": {
|
||||
"input_camera_names": ["r_vis", "top", "front"],
|
||||
"fused_camera_names": ["front", "top", "r_vis"],
|
||||
"joint_output_dim": 192,
|
||||
"freeze_backbone": true,
|
||||
"dataset_image_resize_shape": null,
|
||||
"eval_image_resize_shape": [256, 256],
|
||||
"fused_short_side_resize": 224
|
||||
},
|
||||
"training_contract": {
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"max_steps": 50000,
|
||||
"rollout_val_freq_epochs": 5,
|
||||
"rollout_num_episodes": 10,
|
||||
"batch_size": 80,
|
||||
"lr": 0.00025,
|
||||
"num_workers": 12,
|
||||
"scheduler_type": "cosine",
|
||||
"warmup_steps": 2000,
|
||||
"min_lr": 1e-06,
|
||||
"weight_decay": 1e-05,
|
||||
"grad_clip": 1.0
|
||||
},
|
||||
"verification": {
|
||||
"local_tests": "38 passed",
|
||||
"remote_dataset_shape": [2, 3, 256, 256],
|
||||
"remote_eval_prepared_shape": [3, 256, 256],
|
||||
"remote_smoke_run": {
|
||||
"run_name": "smoke-lewm-imf-rawpath-emb384-20260406-002002",
|
||||
"result": "passed",
|
||||
"details": "2-step train + checkpoint-triggered 1-episode headless rollout succeeded with corrected raw256 path"
|
||||
}
|
||||
},
|
||||
"superseded_runs": [
|
||||
{
|
||||
"run_name": "lewm-vit-imf-sim-transfer-emb384-l12-ph16-ex08-step50k-roll10-5880g0-20260405-201914",
|
||||
"reason": "stopped due to incorrect early per-camera 224 resize"
|
||||
},
|
||||
{
|
||||
"run_name": "lewm-vit-imf-sim-transfer-emb256-l12-ph16-ex08-step50k-roll10-5880g1-20260405-201914",
|
||||
"reason": "stopped due to incorrect early per-camera 224 resize"
|
||||
}
|
||||
],
|
||||
"full_runs": [
|
||||
{
|
||||
"host": "100.73.14.65",
|
||||
"gpu": 0,
|
||||
"run_name": "lewm-vit-imf-raw256fix-sim-transfer-emb384-l12-ph16-ex08-step50k-roll10-5880g0-20260406-002124",
|
||||
"pid": 1058589,
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/experiment_suite_launch_logs/lewm-vit-imf-raw256fix-sim-transfer-emb384-l12-ph16-ex08-step50k-roll10-5880g0-20260406-002124.launch.log",
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/y5tzgqe0u966w9ak41i31",
|
||||
"head_n_emb": 384,
|
||||
"head_n_layer": 12
|
||||
},
|
||||
{
|
||||
"host": "100.73.14.65",
|
||||
"gpu": 1,
|
||||
"run_name": "lewm-vit-imf-raw256fix-sim-transfer-emb256-l12-ph16-ex08-step50k-roll10-5880g1-20260406-002124",
|
||||
"pid": 1058590,
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/experiment_suite_launch_logs/lewm-vit-imf-raw256fix-sim-transfer-emb256-l12-ph16-ex08-step50k-roll10-5880g1-20260406-002124.launch.log",
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/2esr9y7t2dgesstgrn5i6",
|
||||
"head_n_emb": 256,
|
||||
"head_n_layer": 12
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
# 2026-04-06 LEWM ViT Transfer Notes
|
||||
|
||||
## Root-cause fix
|
||||
|
||||
The first LEWM runs were stopped because the data path still resized each camera view to `224x224` **before** multiview fusion. That preserved the final tensor shape but broke the original LEWM geometry.
|
||||
|
||||
Corrected path now is:
|
||||
|
||||
- **Training dataset**: keep stored per-view `256x256` images (`data.image_resize_shape=null` at launch; dataset instantiate override is `None` for LEWM)
|
||||
- **Eval rollout input**: resize live MuJoCo `480x640` camera images to `256x256` per view
|
||||
- **Backbone**: fuse `front, top, r_vis` on the LEWM axis, then resize fused short side to `224`
|
||||
|
||||
## Verification
|
||||
|
||||
- Local tests passed (`38 passed` across the focused suite)
|
||||
- Remote check:
|
||||
- dataset sample image shape: `(2, 3, 256, 256)`
|
||||
- eval-prepared live frame shape: `(3, 256, 256)`
|
||||
- Remote smoke passed with real checkpoint:
|
||||
- `smoke-lewm-imf-rawpath-emb384-20260406-002002`
|
||||
|
||||
## Current runs
|
||||
|
||||
- `lewm-vit-imf-raw256fix-sim-transfer-emb384-l12-ph16-ex08-step50k-roll10-5880g0-20260406-002124`
|
||||
- `lewm-vit-imf-raw256fix-sim-transfer-emb256-l12-ph16-ex08-step50k-roll10-5880g1-20260406-002124`
|
||||
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"status": "running",
|
||||
"updated_at": "2026-04-06T00:22:10+08:00",
|
||||
"remote_host": "100.73.14.65",
|
||||
"runs": [
|
||||
{
|
||||
"run_name": "lewm-vit-imf-raw256fix-sim-transfer-emb384-l12-ph16-ex08-step50k-roll10-5880g0-20260406-002124",
|
||||
"pid": 1058589,
|
||||
"gpu": 0,
|
||||
"state": "running"
|
||||
},
|
||||
{
|
||||
"run_name": "lewm-vit-imf-raw256fix-sim-transfer-emb256-l12-ph16-ex08-step50k-roll10-5880g1-20260406-002124",
|
||||
"pid": 1058590,
|
||||
"gpu": 1,
|
||||
"state": "running"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
# CHECKLIST
|
||||
|
||||
- [x] Create run contract
|
||||
- [x] Remote smoke test passes
|
||||
- [x] Launch 50k main run
|
||||
- [x] Record pid / log / SwanLab
|
||||
- [x] Report status back to user
|
||||
@@ -0,0 +1,12 @@
|
||||
# PLAN
|
||||
|
||||
## Goal
|
||||
Train a 50k-step IMF baseline with the original ResNet vision backbone using r_vis only as the only image conditioning.
|
||||
|
||||
## Fixed comparison contract
|
||||
- same hyperparameters as the active top/front run
|
||||
- cameras: ['r_vis']
|
||||
- num_cams=1
|
||||
- head.cond_dim=80
|
||||
- host: 100.119.99.14
|
||||
- gpu: 3
|
||||
@@ -0,0 +1,6 @@
|
||||
# Notes
|
||||
|
||||
- 2026-04-05 12:58:22: smoke passed for ['r_vis'] on 100.119.99.14 GPU3.
|
||||
- 2026-04-05 12:59:24: launched main run `imf-resnet-rvis-1cam-ph16-ex08-emb384-l12-ms50k-l20g3-20260405-125844`.
|
||||
- 2026-04-05 13:01:20: latest confirmed progress step=400, loss=0.1165.
|
||||
- SwanLab: https://swanlab.cn/@game-loader/roboimi-vla/runs/qnuh7vln9mqomxxldyecq
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"suite_name": "2026-04-05-rvis-only-resnet-1cam",
|
||||
"updated_at": "2026-04-05 13:01:20",
|
||||
"phase": "running",
|
||||
"smoke_test": {
|
||||
"status": "passed",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 3,
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/smoke-rvisonly-resnet-ph16-ex08-20260405-125812",
|
||||
"batch_size": 80,
|
||||
"max_steps": 2,
|
||||
"note": "2-step remote CUDA smoke passed without OOM."
|
||||
},
|
||||
"main_run": {
|
||||
"status": "running",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 3,
|
||||
"launch_pid": 164812,
|
||||
"pid": 164816,
|
||||
"run_name": "imf-resnet-rvis-1cam-ph16-ex08-emb384-l12-ms50k-l20g3-20260405-125844",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-rvis-1cam-ph16-ex08-emb384-l12-ms50k-l20g3-20260405-125844",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-rvis-1cam-ph16-ex08-emb384-l12-ms50k-l20g3-20260405-125844/train_vla.log",
|
||||
"launch_log": "/home/droid/roboimi_suite_20260404/experiment_suite_launch_logs/imf-resnet-rvis-1cam-ph16-ex08-emb384-l12-ms50k-l20g3-20260405-125844.launch.log",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"camera_names": [
|
||||
"r_vis"
|
||||
],
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"head_cond_dim": 80,
|
||||
"head_n_emb": 384,
|
||||
"head_n_layer": 12,
|
||||
"vision_backbone_mode": "resnet",
|
||||
"pretrained_backbone_weights": null,
|
||||
"freeze_backbone": false,
|
||||
"batch_size": 80,
|
||||
"lr": 0.00025,
|
||||
"num_workers": 12,
|
||||
"max_steps": 50000,
|
||||
"rollout_val_freq_epochs": 5,
|
||||
"rollout_num_episodes": 5,
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/qnuh7vln9mqomxxldyecq",
|
||||
"latest_step": 400,
|
||||
"latest_loss": 0.1165,
|
||||
"process_running": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
# CHECKLIST
|
||||
|
||||
- [x] Create run contract
|
||||
- [x] Remote smoke test passes
|
||||
- [x] Launch 50k main run
|
||||
- [x] Record pid / log / SwanLab
|
||||
- [x] Report status back to user
|
||||
@@ -0,0 +1,12 @@
|
||||
# PLAN
|
||||
|
||||
## Goal
|
||||
Train a 50k-step IMF baseline with the original ResNet vision backbone using r_vis + top as the only image conditioning.
|
||||
|
||||
## Fixed comparison contract
|
||||
- same hyperparameters as the active top/front run
|
||||
- cameras: ['r_vis', 'top']
|
||||
- num_cams=2
|
||||
- head.cond_dim=144
|
||||
- host: 100.119.99.14
|
||||
- gpu: 2
|
||||
@@ -0,0 +1,6 @@
|
||||
# Notes
|
||||
|
||||
- 2026-04-05 12:58:22: smoke passed for ['r_vis', 'top'] on 100.119.99.14 GPU2.
|
||||
- 2026-04-05 12:59:24: launched main run `imf-resnet-rvistop-2cam-ph16-ex08-emb384-l12-ms50k-l20g2-20260405-125844`.
|
||||
- 2026-04-05 13:01:20: latest confirmed progress step=200, loss=0.2845.
|
||||
- SwanLab: https://swanlab.cn/@game-loader/roboimi-vla/runs/umsm6402eb81et7wx7z4a
|
||||
@@ -0,0 +1,48 @@
|
||||
{
|
||||
"suite_name": "2026-04-05-rvistop-resnet-2cam",
|
||||
"updated_at": "2026-04-05 13:01:20",
|
||||
"phase": "running",
|
||||
"smoke_test": {
|
||||
"status": "passed",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 2,
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/smoke-rvistop-resnet-ph16-ex08-20260405-125812",
|
||||
"batch_size": 80,
|
||||
"max_steps": 2,
|
||||
"note": "2-step remote CUDA smoke passed without OOM."
|
||||
},
|
||||
"main_run": {
|
||||
"status": "running",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 2,
|
||||
"launch_pid": 164745,
|
||||
"pid": 164749,
|
||||
"run_name": "imf-resnet-rvistop-2cam-ph16-ex08-emb384-l12-ms50k-l20g2-20260405-125844",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-rvistop-2cam-ph16-ex08-emb384-l12-ms50k-l20g2-20260405-125844",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-rvistop-2cam-ph16-ex08-emb384-l12-ms50k-l20g2-20260405-125844/train_vla.log",
|
||||
"launch_log": "/home/droid/roboimi_suite_20260404/experiment_suite_launch_logs/imf-resnet-rvistop-2cam-ph16-ex08-emb384-l12-ms50k-l20g2-20260405-125844.launch.log",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"camera_names": [
|
||||
"r_vis",
|
||||
"top"
|
||||
],
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"head_cond_dim": 144,
|
||||
"head_n_emb": 384,
|
||||
"head_n_layer": 12,
|
||||
"vision_backbone_mode": "resnet",
|
||||
"pretrained_backbone_weights": null,
|
||||
"freeze_backbone": false,
|
||||
"batch_size": 80,
|
||||
"lr": 0.00025,
|
||||
"num_workers": 12,
|
||||
"max_steps": 50000,
|
||||
"rollout_val_freq_epochs": 5,
|
||||
"rollout_num_episodes": 5,
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/umsm6402eb81et7wx7z4a",
|
||||
"latest_step": 200,
|
||||
"latest_loss": 0.2845,
|
||||
"process_running": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
# CHECKLIST
|
||||
|
||||
- [x] Confirm baseline hyperparameters from trusted prior run
|
||||
- [x] Confirm local GPU availability
|
||||
- [x] Smoke test with `top/front` cameras only
|
||||
- [x] Launch 50k run
|
||||
- [x] Record pid / run dir / log path / SwanLab URL
|
||||
- [x] Report status back to user
|
||||
@@ -0,0 +1,30 @@
|
||||
# PLAN
|
||||
|
||||
## Goal
|
||||
Train a 50k-step IMF baseline with the original ResNet vision backbone (no full-AttnRes vision replacement), using only `top` and `front` cameras as image conditioning.
|
||||
|
||||
## Fixed comparison contract
|
||||
- Agent: `resnet_imf_attnres`
|
||||
- Vision backbone mode: `resnet`
|
||||
- `pred_horizon=16`
|
||||
- `num_action_steps=8`
|
||||
- `n_emb=384`, `n_layer=12`, `n_head=1`, `n_kv_head=1`
|
||||
- `inference_steps=1`
|
||||
- `batch_size=80`, `lr=2.5e-4`, cosine scheduler, warmup 2000
|
||||
- dataset: `/home/droid/project/diana_sim/sim_transfer`
|
||||
- cameras: `[top, front]` only
|
||||
- training budget: `max_steps=50000`
|
||||
- rollout validation: every 5 epochs, 5 episodes, headless
|
||||
|
||||
## Resource plan
|
||||
- Host: local
|
||||
- GPU: RTX 5090 (GPU 0)
|
||||
|
||||
## Execution path
|
||||
1. Run a short 2-step smoke test on GPU with the exact 2-camera config.
|
||||
2. If smoke passes, launch the 50k main run with durable log redirection.
|
||||
3. Record run name, pid, log path, and SwanLab URL into suite status.
|
||||
|
||||
## Fallbacks
|
||||
- If batch 80 OOMs, fall back to batch 64 with scaled lr 2.0e-4.
|
||||
- If dataloader startup is unstable, reduce num_workers from 12 to 8.
|
||||
@@ -0,0 +1,5 @@
|
||||
# Notes
|
||||
|
||||
- 2026-04-05 08:50:04: 2-step smoke test passed locally on RTX 5090 with `top/front` cameras, batch=80, no OOM.
|
||||
- 2026-04-05 08:50:42: launched main run `imf-resnet-topfront-2cam-ph16-ex08-emb384-l12-ms50k-5090-20260405-085023` on local GPU0.
|
||||
- SwanLab: https://swanlab.cn/@game-loader/roboimi-vla/runs/vi77mn5dwd19z4nttxab8
|
||||
@@ -0,0 +1,51 @@
|
||||
{
|
||||
"suite_name": "2026-04-05-top-front-resnet-2cam",
|
||||
"updated_at": "2026-04-05 08:52:12",
|
||||
"phase": "running",
|
||||
"baseline_reference": {
|
||||
"source_run": "imf-p1-ph16-ex08-emb384-l12-ms50k-5880g1-20260404-131223",
|
||||
"best_rollout_avg_reward": 610.8,
|
||||
"best_step": 21874,
|
||||
"notes": "Same IMF baseline as Phase-1 best, but switch cameras from [r_vis, top, front] to [top, front] and keep the original ResNet vision backbone."
|
||||
},
|
||||
"smoke_test": {
|
||||
"status": "passed",
|
||||
"run_dir": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/runs/smoke-topfront-resnet-ph16-ex08-20260405-085000",
|
||||
"batch_size": 80,
|
||||
"num_workers": 4,
|
||||
"max_steps": 2,
|
||||
"note": "2-step local CUDA smoke passed without OOM using top/front only."
|
||||
},
|
||||
"main_run": {
|
||||
"status": "running",
|
||||
"host": "local",
|
||||
"gpu": 0,
|
||||
"pid": 1693348,
|
||||
"run_name": "imf-resnet-topfront-2cam-ph16-ex08-emb384-l12-ms50k-5090-20260405-085023",
|
||||
"run_dir": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/runs/imf-resnet-topfront-2cam-ph16-ex08-emb384-l12-ms50k-5090-20260405-085023",
|
||||
"log_path": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/runs/imf-resnet-topfront-2cam-ph16-ex08-emb384-l12-ms50k-5090-20260405-085023/train_vla.log",
|
||||
"launch_log": "/home/droid/project/roboimi/.worktrees/feat-imf-attnres-policy/experiment_suites/2026-04-05-top-front-resnet-2cam/launch_logs/imf-resnet-topfront-2cam-ph16-ex08-emb384-l12-ms50k-5090-20260405-085023.launch.log",
|
||||
"dataset_dir": "/home/droid/project/diana_sim/sim_transfer",
|
||||
"camera_names": [
|
||||
"top",
|
||||
"front"
|
||||
],
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"head_n_emb": 384,
|
||||
"head_n_layer": 12,
|
||||
"vision_backbone_mode": "resnet",
|
||||
"pretrained_backbone_weights": null,
|
||||
"freeze_backbone": false,
|
||||
"batch_size": 80,
|
||||
"lr": 0.00025,
|
||||
"num_workers": 12,
|
||||
"max_steps": 50000,
|
||||
"rollout_val_freq_epochs": 5,
|
||||
"rollout_num_episodes": 5,
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/vi77mn5dwd19z4nttxab8",
|
||||
"latest_step": 500,
|
||||
"latest_loss": 0.0978,
|
||||
"process_running": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
# CHECKLIST
|
||||
|
||||
- [x] Create run contract
|
||||
- [x] Remote smoke test passes
|
||||
- [x] Launch 50k main run
|
||||
- [x] Record pid / log / SwanLab
|
||||
- [x] Report status back to user
|
||||
@@ -0,0 +1,12 @@
|
||||
# PLAN
|
||||
|
||||
## Goal
|
||||
Train a 50k-step IMF baseline with the original ResNet vision backbone using top only as the only image conditioning.
|
||||
|
||||
## Fixed comparison contract
|
||||
- same hyperparameters as the active top/front run
|
||||
- cameras: ['top']
|
||||
- num_cams=1
|
||||
- head.cond_dim=80
|
||||
- host: 100.119.99.14
|
||||
- gpu: 4
|
||||
@@ -0,0 +1,6 @@
|
||||
# Notes
|
||||
|
||||
- 2026-04-05 12:58:22: smoke passed for ['top'] on 100.119.99.14 GPU4.
|
||||
- 2026-04-05 12:59:24: launched main run `imf-resnet-top-1cam-ph16-ex08-emb384-l12-ms50k-l20g4-20260405-125844`.
|
||||
- 2026-04-05 13:01:20: latest confirmed progress step=400, loss=0.1233.
|
||||
- SwanLab: https://swanlab.cn/@game-loader/roboimi-vla/runs/egzo29l3z9ftsaunhf025
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"suite_name": "2026-04-05-top-only-resnet-1cam",
|
||||
"updated_at": "2026-04-05 13:01:20",
|
||||
"phase": "running",
|
||||
"smoke_test": {
|
||||
"status": "passed",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 4,
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/smoke-toponly-resnet-ph16-ex08-20260405-125812",
|
||||
"batch_size": 80,
|
||||
"max_steps": 2,
|
||||
"note": "2-step remote CUDA smoke passed without OOM."
|
||||
},
|
||||
"main_run": {
|
||||
"status": "running",
|
||||
"host": "100.119.99.14",
|
||||
"gpu": 4,
|
||||
"launch_pid": 164808,
|
||||
"pid": 164813,
|
||||
"run_name": "imf-resnet-top-1cam-ph16-ex08-emb384-l12-ms50k-l20g4-20260405-125844",
|
||||
"run_dir": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-top-1cam-ph16-ex08-emb384-l12-ms50k-l20g4-20260405-125844",
|
||||
"log_path": "/home/droid/roboimi_suite_20260404/runs/imf-resnet-top-1cam-ph16-ex08-emb384-l12-ms50k-l20g4-20260405-125844/train_vla.log",
|
||||
"launch_log": "/home/droid/roboimi_suite_20260404/experiment_suite_launch_logs/imf-resnet-top-1cam-ph16-ex08-emb384-l12-ms50k-l20g4-20260405-125844.launch.log",
|
||||
"dataset_dir": "/home/droid/sim_dataset/sim_transfer",
|
||||
"camera_names": [
|
||||
"top"
|
||||
],
|
||||
"pred_horizon": 16,
|
||||
"num_action_steps": 8,
|
||||
"head_cond_dim": 80,
|
||||
"head_n_emb": 384,
|
||||
"head_n_layer": 12,
|
||||
"vision_backbone_mode": "resnet",
|
||||
"pretrained_backbone_weights": null,
|
||||
"freeze_backbone": false,
|
||||
"batch_size": 80,
|
||||
"lr": 0.00025,
|
||||
"num_workers": 12,
|
||||
"max_steps": 50000,
|
||||
"rollout_val_freq_epochs": 5,
|
||||
"rollout_num_episodes": 5,
|
||||
"swanlab_url": "https://swanlab.cn/@game-loader/roboimi-vla/runs/egzo29l3z9ftsaunhf025",
|
||||
"latest_step": 400,
|
||||
"latest_loss": 0.1233,
|
||||
"process_running": true
|
||||
}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 40 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 44 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 45 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 46 KiB |
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 346 KiB |
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"akita_black_bowl_1": {
|
||||
"min": [
|
||||
-0.10497,
|
||||
-0.014979,
|
||||
0.898407
|
||||
],
|
||||
"max": [
|
||||
-0.075125,
|
||||
0.013868,
|
||||
0.899216
|
||||
],
|
||||
"range": [
|
||||
0.029845,
|
||||
0.028847,
|
||||
0.000809
|
||||
],
|
||||
"std": [
|
||||
0.009227,
|
||||
0.008342,
|
||||
0.000113
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"min": [
|
||||
0.035635,
|
||||
-0.034313,
|
||||
0.902506
|
||||
],
|
||||
"max": [
|
||||
0.065008,
|
||||
-0.008729,
|
||||
0.902506
|
||||
],
|
||||
"range": [
|
||||
0.029373,
|
||||
0.025583,
|
||||
1e-06
|
||||
],
|
||||
"std": [
|
||||
0.009018,
|
||||
0.007293,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"min": [
|
||||
-0.214447,
|
||||
-0.064733,
|
||||
0.898875
|
||||
],
|
||||
"max": [
|
||||
-0.186022,
|
||||
-0.035164,
|
||||
0.898875
|
||||
],
|
||||
"range": [
|
||||
0.028425,
|
||||
0.029569,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.007991,
|
||||
0.009218,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"min": [
|
||||
-0.069231,
|
||||
0.111593,
|
||||
0.908921
|
||||
],
|
||||
"max": [
|
||||
-0.030195,
|
||||
0.146634,
|
||||
0.908921
|
||||
],
|
||||
"range": [
|
||||
0.039036,
|
||||
0.035042,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.011587,
|
||||
0.009411,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"wooden_cabinet_1": {
|
||||
"min": [
|
||||
0.020628,
|
||||
-0.249667,
|
||||
0.905
|
||||
],
|
||||
"max": [
|
||||
0.039668,
|
||||
-0.23086,
|
||||
0.905
|
||||
],
|
||||
"range": [
|
||||
0.01904,
|
||||
0.018807,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.005624,
|
||||
0.005887,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"flat_stove_1": {
|
||||
"min": [
|
||||
-0.41998,
|
||||
0.200256,
|
||||
0.905
|
||||
],
|
||||
"max": [
|
||||
-0.400671,
|
||||
0.219361,
|
||||
0.905
|
||||
],
|
||||
"range": [
|
||||
0.019309,
|
||||
0.019105,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.00542,
|
||||
0.005313,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"wine_rack_1": {
|
||||
"min": [
|
||||
-0.269738,
|
||||
-0.269776,
|
||||
0.92
|
||||
],
|
||||
"max": [
|
||||
-0.250473,
|
||||
-0.250372,
|
||||
0.92
|
||||
],
|
||||
"range": [
|
||||
0.019265,
|
||||
0.019404,
|
||||
0.0
|
||||
],
|
||||
"std": [
|
||||
0.005695,
|
||||
0.005698,
|
||||
0.0
|
||||
]
|
||||
},
|
||||
"eef_pos": {
|
||||
"min": [
|
||||
-0.218347,
|
||||
-0.022207,
|
||||
1.155793
|
||||
],
|
||||
"max": [
|
||||
-0.191731,
|
||||
0.016482,
|
||||
1.188032
|
||||
],
|
||||
"range": [
|
||||
0.026616,
|
||||
0.038689,
|
||||
0.03224
|
||||
],
|
||||
"std": [
|
||||
0.005112,
|
||||
0.009859,
|
||||
0.007417
|
||||
]
|
||||
},
|
||||
"joint_pos": {
|
||||
"min": [
|
||||
-0.032644,
|
||||
-0.185945,
|
||||
-0.035387,
|
||||
-2.486189,
|
||||
-0.01145,
|
||||
2.185933,
|
||||
0.745322
|
||||
],
|
||||
"max": [
|
||||
0.021994,
|
||||
-0.132522,
|
||||
0.029516,
|
||||
-2.375572,
|
||||
0.008342,
|
||||
2.274589,
|
||||
0.815841
|
||||
],
|
||||
"range": [
|
||||
0.054637,
|
||||
0.053423,
|
||||
0.064903,
|
||||
0.110617,
|
||||
0.019791,
|
||||
0.088656,
|
||||
0.070519
|
||||
],
|
||||
"std": [
|
||||
0.013218,
|
||||
0.012722,
|
||||
0.015234,
|
||||
0.021384,
|
||||
0.004568,
|
||||
0.020009,
|
||||
0.017693
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,552 @@
|
||||
[
|
||||
{
|
||||
"init_id": 0,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.097626,
|
||||
-0.008583,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.062074,
|
||||
-0.009382,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.204233,
|
||||
-0.061651,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.046026,
|
||||
0.134473,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 1,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.096428,
|
||||
0.013749,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.058727,
|
||||
-0.034313,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.203176,
|
||||
-0.053714,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.040864,
|
||||
0.121312,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
-0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 2,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.103282,
|
||||
0.002253,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.052456,
|
||||
-0.012705,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.193374,
|
||||
-0.036331,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.066908,
|
||||
0.118864,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
-0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 3,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.090777,
|
||||
-0.002232,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.040521,
|
||||
-0.022222,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.20432,
|
||||
-0.056655,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.031539,
|
||||
0.123188,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 4,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.104225,
|
||||
0.013797,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.03824,
|
||||
-0.010652,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.19591,
|
||||
-0.039959,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.037586,
|
||||
0.133424,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 5,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.08334,
|
||||
0.002911,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.064465,
|
||||
-0.028594,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.198764,
|
||||
-0.053643,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.063907,
|
||||
0.120699,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 6,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.076496,
|
||||
-0.00786,
|
||||
0.898414
|
||||
],
|
||||
"quat": [
|
||||
0.706935,
|
||||
-0.000206,
|
||||
-0.00017,
|
||||
0.707279
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.036954,
|
||||
-0.031686,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707659,
|
||||
-3.5e-05,
|
||||
1e-06,
|
||||
0.706554
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.19987,
|
||||
-0.056163,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.053796,
|
||||
0.120126,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
-0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 7,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.075125,
|
||||
-0.003472,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.046896,
|
||||
-0.014595,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.1875,
|
||||
-0.0641,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.060927,
|
||||
0.116114,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
-0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 8,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.086798,
|
||||
0.010183,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.057509,
|
||||
-0.011733,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.20364,
|
||||
-0.035474,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.040355,
|
||||
0.128839,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"init_id": 9,
|
||||
"akita_black_bowl_1": {
|
||||
"pos": [
|
||||
-0.081038,
|
||||
-0.000113,
|
||||
0.898407
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.4e-05,
|
||||
2e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"plate_1": {
|
||||
"pos": [
|
||||
0.049027,
|
||||
-0.013357,
|
||||
0.902506
|
||||
],
|
||||
"quat": [
|
||||
0.707107,
|
||||
-1.3e-05,
|
||||
6e-06,
|
||||
0.707107
|
||||
]
|
||||
},
|
||||
"wine_bottle_1": {
|
||||
"pos": [
|
||||
-0.203822,
|
||||
-0.057315,
|
||||
0.898875
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
-6e-06,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
},
|
||||
"cream_cheese_1": {
|
||||
"pos": [
|
||||
-0.06023,
|
||||
0.136331,
|
||||
0.908921
|
||||
],
|
||||
"quat": [
|
||||
0.0,
|
||||
0.0,
|
||||
0.0,
|
||||
1.0
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"checkpoint": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/checkpoints/vla_model_step_83124.pt",
|
||||
"num_rollouts": 5,
|
||||
"episodes": [
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01",
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_summary.json"
|
||||
},
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02",
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_summary.json"
|
||||
},
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03",
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_summary.json"
|
||||
},
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04",
|
||||
"avg_reward": 2.0,
|
||||
"avg_max_reward": 2.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_summary.json"
|
||||
},
|
||||
{
|
||||
"episode_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05",
|
||||
"avg_reward": 1408.0,
|
||||
"avg_max_reward": 4.0,
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_front.mp4",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_summary.json"
|
||||
}
|
||||
],
|
||||
"mean_avg_reward": 282.0,
|
||||
"best_avg_reward": 1408.0
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
0.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
0.0
|
||||
],
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 416.328579450197,
|
||||
"control_fps": 16.62784691072735,
|
||||
"avg_obs_read_time_ms": 2.911593945131504,
|
||||
"avg_preprocess_time_ms": 6.928545483055392,
|
||||
"avg_inference_time_ms": 2.401948963774235,
|
||||
"avg_env_step_time_ms": 47.72096303665811,
|
||||
"avg_total_time_ms": 60.14007738758145,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 0.0,
|
||||
"episode_max_reward": 0.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.911593945131504,
|
||||
"std": 0.2611419950299626,
|
||||
"min": 2.712091023568064,
|
||||
"max": 7.493758981581777
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.928545483055392,
|
||||
"std": 0.5543077871256182,
|
||||
"min": 6.473146029748023,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.401948963774235,
|
||||
"std": 12.794224178610442,
|
||||
"min": 0.49940100871026516,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.72096303665811,
|
||||
"std": 2.1171372236113726,
|
||||
"min": 39.931224018801004,
|
||||
"max": 76.46449701860547
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 60.14007738758145,
|
||||
"std": 13.188809002297118,
|
||||
"min": 56.274613016285,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.911593945131504,
|
||||
"std": 0.2611419950299626,
|
||||
"min": 2.712091023568064,
|
||||
"max": 7.493758981581777
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.928545483055392,
|
||||
"std": 0.5543077871256182,
|
||||
"min": 6.473146029748023,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.401948963774235,
|
||||
"std": 12.794224178610442,
|
||||
"min": 0.49940100871026516,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.72096303665811,
|
||||
"std": 2.1171372236113726,
|
||||
"min": 39.931224018801004,
|
||||
"max": 76.46449701860547
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 60.14007738758145,
|
||||
"std": 13.188809002297118,
|
||||
"min": 56.274613016285,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8758058099562978,
|
||||
"std": 0.0751506551331651,
|
||||
"min": 2.712091023568064,
|
||||
"max": 3.0596989672631025
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.1382775310088284,
|
||||
"std": 1.3865518440871207,
|
||||
"min": 6.516602006740868,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 15.048258656778755,
|
||||
"std": 33.453789958947034,
|
||||
"min": 11.253663979005069,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.788851250185296,
|
||||
"std": 1.684109686362168,
|
||||
"min": 39.931224018801004,
|
||||
"max": 56.31602002540603
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 73.03679807624907,
|
||||
"std": 34.000319709963435,
|
||||
"min": 67.7422959706746,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep01/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 416.328579450197,
|
||||
"avg_control_fps": 16.62784691072735,
|
||||
"avg_obs_read_time_ms": 2.911593945131504,
|
||||
"avg_preprocess_time_ms": 6.928545483055392,
|
||||
"avg_inference_time_ms": 2.401948963774235,
|
||||
"avg_env_step_time_ms": 47.72096303665811,
|
||||
"avg_total_time_ms": 60.14007738758145,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.911593945131504,
|
||||
"std": 0.2611419950299626,
|
||||
"min": 2.712091023568064,
|
||||
"max": 7.493758981581777
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.928545483055392,
|
||||
"std": 0.5543077871256182,
|
||||
"min": 6.473146029748023,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.401948963774235,
|
||||
"std": 12.794224178610442,
|
||||
"min": 0.49940100871026516,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.72096303665811,
|
||||
"std": 2.1171372236113726,
|
||||
"min": 39.931224018801004,
|
||||
"max": 76.46449701860547
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 60.14007738758145,
|
||||
"std": 13.188809002297118,
|
||||
"min": 56.274613016285,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8758058099562978,
|
||||
"std": 0.0751506551331651,
|
||||
"min": 2.712091023568064,
|
||||
"max": 3.0596989672631025
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.1382775310088284,
|
||||
"std": 1.3865518440871207,
|
||||
"min": 6.516602006740868,
|
||||
"max": 19.883478991687298
|
||||
},
|
||||
"inference": {
|
||||
"mean": 15.048258656778755,
|
||||
"std": 33.453789958947034,
|
||||
"min": 11.253663979005069,
|
||||
"max": 327.08223100053146
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.788851250185296,
|
||||
"std": 1.684109686362168,
|
||||
"min": 39.931224018801004,
|
||||
"max": 56.31602002540603
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 73.03679807624907,
|
||||
"std": 34.000319709963435,
|
||||
"min": 67.7422959706746,
|
||||
"max": 389.8623649729416
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
0.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
0.0
|
||||
],
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 423.2530657749397,
|
||||
"control_fps": 16.369630991018912,
|
||||
"avg_obs_read_time_ms": 2.922768201263222,
|
||||
"avg_preprocess_time_ms": 7.034949533491662,
|
||||
"avg_inference_time_ms": 2.362652703221621,
|
||||
"avg_env_step_time_ms": 48.58740326665741,
|
||||
"avg_total_time_ms": 61.088731966447085,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 0.0,
|
||||
"episode_max_reward": 0.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.922768201263222,
|
||||
"std": 0.20864628356732534,
|
||||
"min": 2.7384720160625875,
|
||||
"max": 4.775333974976093
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.034949533491662,
|
||||
"std": 0.6020329090344461,
|
||||
"min": 6.419215991627425,
|
||||
"max": 10.473316011484712
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.362652703221621,
|
||||
"std": 12.69272670328858,
|
||||
"min": 0.4969470319338143,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.58740326665741,
|
||||
"std": 3.577976984950618,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 61.088731966447085,
|
||||
"std": 13.227582693702194,
|
||||
"min": 56.584795005619526,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.922768201263222,
|
||||
"std": 0.20864628356732534,
|
||||
"min": 2.7384720160625875,
|
||||
"max": 4.775333974976093
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.034949533491662,
|
||||
"std": 0.6020329090344461,
|
||||
"min": 6.419215991627425,
|
||||
"max": 10.473316011484712
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.362652703221621,
|
||||
"std": 12.69272670328858,
|
||||
"min": 0.4969470319338143,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.58740326665741,
|
||||
"std": 3.577976984950618,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 61.088731966447085,
|
||||
"std": 13.227582693702194,
|
||||
"min": 56.584795005619526,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.89267532604175,
|
||||
"std": 0.11125532360169263,
|
||||
"min": 2.7624600334092975,
|
||||
"max": 3.5272870445623994
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.164920194720088,
|
||||
"std": 0.6480595307828021,
|
||||
"min": 6.461958982981741,
|
||||
"max": 10.244207980576903
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.722178172881037,
|
||||
"std": 33.26832928567316,
|
||||
"min": 10.894106002524495,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.64498521717244,
|
||||
"std": 3.7269116621985248,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 73.61584814432057,
|
||||
"std": 32.98871154259634,
|
||||
"min": 67.3744379892014,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep02/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 423.2530657749397,
|
||||
"avg_control_fps": 16.369630991018912,
|
||||
"avg_obs_read_time_ms": 2.922768201263222,
|
||||
"avg_preprocess_time_ms": 7.034949533491662,
|
||||
"avg_inference_time_ms": 2.362652703221621,
|
||||
"avg_env_step_time_ms": 48.58740326665741,
|
||||
"avg_total_time_ms": 61.088731966447085,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.922768201263222,
|
||||
"std": 0.20864628356732534,
|
||||
"min": 2.7384720160625875,
|
||||
"max": 4.775333974976093
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.034949533491662,
|
||||
"std": 0.6020329090344461,
|
||||
"min": 6.419215991627425,
|
||||
"max": 10.473316011484712
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.362652703221621,
|
||||
"std": 12.69272670328858,
|
||||
"min": 0.4969470319338143,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.58740326665741,
|
||||
"std": 3.577976984950618,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 61.088731966447085,
|
||||
"std": 13.227582693702194,
|
||||
"min": 56.584795005619526,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.89267532604175,
|
||||
"std": 0.11125532360169263,
|
||||
"min": 2.7624600334092975,
|
||||
"max": 3.5272870445623994
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.164920194720088,
|
||||
"std": 0.6480595307828021,
|
||||
"min": 6.461958982981741,
|
||||
"max": 10.244207980576903
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.722178172881037,
|
||||
"std": 33.26832928567316,
|
||||
"min": 10.894106002524495,
|
||||
"max": 324.9893220490776
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 48.64498521717244,
|
||||
"std": 3.7269116621985248,
|
||||
"min": 41.04071902111173,
|
||||
"max": 67.934600985609
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 73.61584814432057,
|
||||
"std": 32.98871154259634,
|
||||
"min": 67.3744379892014,
|
||||
"max": 378.0319819925353
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
0.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
0.0
|
||||
],
|
||||
"avg_reward": 0.0,
|
||||
"avg_max_reward": 0.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 428.38141787855113,
|
||||
"control_fps": 16.74224842048907,
|
||||
"avg_obs_read_time_ms": 2.900395128443571,
|
||||
"avg_preprocess_time_ms": 6.847853548159557,
|
||||
"avg_inference_time_ms": 2.3343682948533178,
|
||||
"avg_env_step_time_ms": 47.4696740059049,
|
||||
"avg_total_time_ms": 59.7291340377082,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 0.0,
|
||||
"episode_max_reward": 0.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.900395128443571,
|
||||
"std": 0.2916878594282876,
|
||||
"min": 2.68459296785295,
|
||||
"max": 7.19653902342543
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.847853548159557,
|
||||
"std": 0.21243417443892795,
|
||||
"min": 6.388033041730523,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3343682948533178,
|
||||
"std": 11.800465870815552,
|
||||
"min": 0.49768202006816864,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.4696740059049,
|
||||
"std": 1.7681566119287178,
|
||||
"min": 41.32056800881401,
|
||||
"max": 82.69464899785817
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 59.7291340377082,
|
||||
"std": 11.80948193858811,
|
||||
"min": 56.34060001466423,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.900395128443571,
|
||||
"std": 0.2916878594282876,
|
||||
"min": 2.68459296785295,
|
||||
"max": 7.19653902342543
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.847853548159557,
|
||||
"std": 0.21243417443892795,
|
||||
"min": 6.388033041730523,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3343682948533178,
|
||||
"std": 11.800465870815552,
|
||||
"min": 0.49768202006816864,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.4696740059049,
|
||||
"std": 1.7681566119287178,
|
||||
"min": 41.32056800881401,
|
||||
"max": 82.69464899785817
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 59.7291340377082,
|
||||
"std": 11.80948193858811,
|
||||
"min": 56.34060001466423,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.850530921105846,
|
||||
"std": 0.06315892633423309,
|
||||
"min": 2.729182015173137,
|
||||
"max": 2.99588602501899
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.963662055733783,
|
||||
"std": 0.3317973295122339,
|
||||
"min": 6.484561017714441,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.542899646171877,
|
||||
"std": 30.613461495155526,
|
||||
"min": 11.056077957618982,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.46852555862543,
|
||||
"std": 0.9486706652557946,
|
||||
"min": 41.32056800881401,
|
||||
"max": 49.64756895788014
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 72.01395506920313,
|
||||
"std": 30.190598270939223,
|
||||
"min": 67.28241901146248,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep03/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 428.38141787855113,
|
||||
"avg_control_fps": 16.74224842048907,
|
||||
"avg_obs_read_time_ms": 2.900395128443571,
|
||||
"avg_preprocess_time_ms": 6.847853548159557,
|
||||
"avg_inference_time_ms": 2.3343682948533178,
|
||||
"avg_env_step_time_ms": 47.4696740059049,
|
||||
"avg_total_time_ms": 59.7291340377082,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.900395128443571,
|
||||
"std": 0.2916878594282876,
|
||||
"min": 2.68459296785295,
|
||||
"max": 7.19653902342543
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.847853548159557,
|
||||
"std": 0.21243417443892795,
|
||||
"min": 6.388033041730523,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3343682948533178,
|
||||
"std": 11.800465870815552,
|
||||
"min": 0.49768202006816864,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.4696740059049,
|
||||
"std": 1.7681566119287178,
|
||||
"min": 41.32056800881401,
|
||||
"max": 82.69464899785817
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 59.7291340377082,
|
||||
"std": 11.80948193858811,
|
||||
"min": 56.34060001466423,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.850530921105846,
|
||||
"std": 0.06315892633423309,
|
||||
"min": 2.729182015173137,
|
||||
"max": 2.99588602501899
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.963662055733783,
|
||||
"std": 0.3317973295122339,
|
||||
"min": 6.484561017714441,
|
||||
"max": 9.000881982501596
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.542899646171877,
|
||||
"std": 30.613461495155526,
|
||||
"min": 11.056077957618982,
|
||||
"max": 300.0851860269904
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 47.46852555862543,
|
||||
"std": 0.9486706652557946,
|
||||
"min": 41.32056800881401,
|
||||
"max": 49.64756895788014
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 72.01395506920313,
|
||||
"std": 30.190598270939223,
|
||||
"min": 67.28241901146248,
|
||||
"max": 353.5165900248103
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
2.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
2.0
|
||||
],
|
||||
"avg_reward": 2.0,
|
||||
"avg_max_reward": 2.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 425.1027328138825,
|
||||
"control_fps": 17.116880794537646,
|
||||
"avg_obs_read_time_ms": 2.8568991100681678,
|
||||
"avg_preprocess_time_ms": 6.912527064227366,
|
||||
"avg_inference_time_ms": 2.3523725509377464,
|
||||
"avg_env_step_time_ms": 46.12416496566896,
|
||||
"avg_total_time_ms": 58.42185921626099,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 2.0,
|
||||
"episode_max_reward": 2.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8568991100681678,
|
||||
"std": 0.20988493064877758,
|
||||
"min": 2.6854430325329304,
|
||||
"max": 5.505044013261795
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.912527064227366,
|
||||
"std": 0.3468982959934229,
|
||||
"min": 6.422812002710998,
|
||||
"max": 13.460688991472125
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3523725509377464,
|
||||
"std": 12.345277540311537,
|
||||
"min": 0.4927710397168994,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.12416496566896,
|
||||
"std": 1.6786026968893601,
|
||||
"min": 40.27245403267443,
|
||||
"max": 69.19108401052654
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 58.42185921626099,
|
||||
"std": 12.395693709763066,
|
||||
"min": 54.73730497760698,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8568991100681678,
|
||||
"std": 0.20988493064877758,
|
||||
"min": 2.6854430325329304,
|
||||
"max": 5.505044013261795
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.912527064227366,
|
||||
"std": 0.3468982959934229,
|
||||
"min": 6.422812002710998,
|
||||
"max": 13.460688991472125
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3523725509377464,
|
||||
"std": 12.345277540311537,
|
||||
"min": 0.4927710397168994,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.12416496566896,
|
||||
"std": 1.6786026968893601,
|
||||
"min": 40.27245403267443,
|
||||
"max": 69.19108401052654
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 58.42185921626099,
|
||||
"std": 12.395693709763066,
|
||||
"min": 54.73730497760698,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8195713563367133,
|
||||
"std": 0.05933493418683723,
|
||||
"min": 2.6968690217472613,
|
||||
"max": 2.953937044367194
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.006697107333986,
|
||||
"std": 0.4376351299601935,
|
||||
"min": 6.50148798013106,
|
||||
"max": 10.386412963271141
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.704165457012344,
|
||||
"std": 32.214912229767435,
|
||||
"min": 11.083905992563814,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.088106907105114,
|
||||
"std": 1.088099613967191,
|
||||
"min": 40.27245403267443,
|
||||
"max": 49.108261009678245
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 70.80417735150232,
|
||||
"std": 31.96448918725213,
|
||||
"min": 65.67699601873755,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep04/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 425.1027328138825,
|
||||
"avg_control_fps": 17.116880794537646,
|
||||
"avg_obs_read_time_ms": 2.8568991100681678,
|
||||
"avg_preprocess_time_ms": 6.912527064227366,
|
||||
"avg_inference_time_ms": 2.3523725509377464,
|
||||
"avg_env_step_time_ms": 46.12416496566896,
|
||||
"avg_total_time_ms": 58.42185921626099,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8568991100681678,
|
||||
"std": 0.20988493064877758,
|
||||
"min": 2.6854430325329304,
|
||||
"max": 5.505044013261795
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.912527064227366,
|
||||
"std": 0.3468982959934229,
|
||||
"min": 6.422812002710998,
|
||||
"max": 13.460688991472125
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.3523725509377464,
|
||||
"std": 12.345277540311537,
|
||||
"min": 0.4927710397168994,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.12416496566896,
|
||||
"std": 1.6786026968893601,
|
||||
"min": 40.27245403267443,
|
||||
"max": 69.19108401052654
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 58.42185921626099,
|
||||
"std": 12.395693709763066,
|
||||
"min": 54.73730497760698,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8195713563367133,
|
||||
"std": 0.05933493418683723,
|
||||
"min": 2.6968690217472613,
|
||||
"max": 2.953937044367194
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.006697107333986,
|
||||
"std": 0.4376351299601935,
|
||||
"min": 6.50148798013106,
|
||||
"max": 10.386412963271141
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.704165457012344,
|
||||
"std": 32.214912229767435,
|
||||
"min": 11.083905992563814,
|
||||
"max": 315.18344301730394
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 46.088106907105114,
|
||||
"std": 1.088099613967191,
|
||||
"min": 40.27245403267443,
|
||||
"max": 49.108261009678245
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 70.80417735150232,
|
||||
"std": 31.96448918725213,
|
||||
"min": 65.67699601873755,
|
||||
"max": 368.8187550287694
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
@@ -0,0 +1,216 @@
|
||||
{
|
||||
"num_episodes": 1,
|
||||
"episode_rewards": [
|
||||
1408.0
|
||||
],
|
||||
"episode_max_rewards": [
|
||||
4.0
|
||||
],
|
||||
"avg_reward": 1408.0,
|
||||
"avg_max_reward": 4.0,
|
||||
"episodes": [
|
||||
{
|
||||
"inference_fps": 421.5143683898348,
|
||||
"control_fps": 15.98697584872221,
|
||||
"avg_obs_read_time_ms": 2.916401576450361,
|
||||
"avg_preprocess_time_ms": 6.9587834345709,
|
||||
"avg_inference_time_ms": 2.372398368814694,
|
||||
"avg_env_step_time_ms": 50.12335285793857,
|
||||
"avg_total_time_ms": 62.55091703787912,
|
||||
"num_inferences": 88,
|
||||
"num_model_forwards": 88,
|
||||
"num_steps": 700,
|
||||
"episode_reward": 1408.0,
|
||||
"episode_max_reward": 4.0,
|
||||
"artifact_paths": {
|
||||
"video": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_front.mp4",
|
||||
"trajectory": null,
|
||||
"timing": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_summary.json"
|
||||
},
|
||||
"timing_breakdown_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.916401576450361,
|
||||
"std": 0.19681075249117103,
|
||||
"min": 2.723354031331837,
|
||||
"max": 4.304414032958448
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.9587834345709,
|
||||
"std": 0.32918428959573737,
|
||||
"min": 6.440367025788873,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.372398368814694,
|
||||
"std": 12.755981125405425,
|
||||
"min": 0.4968460416421294,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 50.12335285793857,
|
||||
"std": 2.0541933730540083,
|
||||
"min": 41.243349958676845,
|
||||
"max": 59.02870395220816
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 62.55091703787912,
|
||||
"std": 12.685506605997542,
|
||||
"min": 57.05423402832821,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
},
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.916401576450361,
|
||||
"std": 0.19681075249117103,
|
||||
"min": 2.723354031331837,
|
||||
"max": 4.304414032958448
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.9587834345709,
|
||||
"std": 0.32918428959573737,
|
||||
"min": 6.440367025788873,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.372398368814694,
|
||||
"std": 12.755981125405425,
|
||||
"min": 0.4968460416421294,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 50.12335285793857,
|
||||
"std": 2.0541933730540083,
|
||||
"min": 41.243349958676845,
|
||||
"max": 59.02870395220816
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 62.55091703787912,
|
||||
"std": 12.685506605997542,
|
||||
"min": 57.05423402832821,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8879472850927743,
|
||||
"std": 0.09183274389315361,
|
||||
"min": 2.758314018137753,
|
||||
"max": 3.517468983773142
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.111402045914226,
|
||||
"std": 0.5243812452094909,
|
||||
"min": 6.641154002863914,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.798059454320041,
|
||||
"std": 33.432189809551524,
|
||||
"min": 10.985016997437924,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 49.97409828435841,
|
||||
"std": 2.1747279952000875,
|
||||
"min": 41.243349958676845,
|
||||
"max": 56.928464968223125
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 74.9630101153426,
|
||||
"std": 32.74896535841012,
|
||||
"min": 67.58100300794467,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"artifact_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05",
|
||||
"artifacts": {
|
||||
"output_dir": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05",
|
||||
"summary_json": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_summary.json",
|
||||
"timing_json": null,
|
||||
"trajectory_npz": null,
|
||||
"video_mp4": "/home/droid/roboimi_imf_attnres/runs/sim-transfer-imf-attnres-emb384-l12-ph16-step150k-roll5x5-lr2p5e4-restart-20260402-104553/rollout-step83124-front-5x-20260403-090744/ep05/rollout_front.mp4",
|
||||
"video_camera_name": "front"
|
||||
},
|
||||
"avg_inference_fps": 421.5143683898348,
|
||||
"avg_control_fps": 15.98697584872221,
|
||||
"avg_obs_read_time_ms": 2.916401576450361,
|
||||
"avg_preprocess_time_ms": 6.9587834345709,
|
||||
"avg_inference_time_ms": 2.372398368814694,
|
||||
"avg_env_step_time_ms": 50.12335285793857,
|
||||
"avg_total_time_ms": 62.55091703787912,
|
||||
"timing_summary": {
|
||||
"count": 700,
|
||||
"model_forward_count": 88,
|
||||
"all_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.916401576450361,
|
||||
"std": 0.19681075249117103,
|
||||
"min": 2.723354031331837,
|
||||
"max": 4.304414032958448
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 6.9587834345709,
|
||||
"std": 0.32918428959573737,
|
||||
"min": 6.440367025788873,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 2.372398368814694,
|
||||
"std": 12.755981125405425,
|
||||
"min": 0.4968460416421294,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 50.12335285793857,
|
||||
"std": 2.0541933730540083,
|
||||
"min": 41.243349958676845,
|
||||
"max": 59.02870395220816
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 62.55091703787912,
|
||||
"std": 12.685506605997542,
|
||||
"min": 57.05423402832821,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
},
|
||||
"model_forward_steps_ms": {
|
||||
"obs_read": {
|
||||
"mean": 2.8879472850927743,
|
||||
"std": 0.09183274389315361,
|
||||
"min": 2.758314018137753,
|
||||
"max": 3.517468983773142
|
||||
},
|
||||
"preprocess": {
|
||||
"mean": 7.111402045914226,
|
||||
"std": 0.5243812452094909,
|
||||
"min": 6.641154002863914,
|
||||
"max": 10.836776986252517
|
||||
},
|
||||
"inference": {
|
||||
"mean": 14.798059454320041,
|
||||
"std": 33.432189809551524,
|
||||
"min": 10.985016997437924,
|
||||
"max": 326.6250739688985
|
||||
},
|
||||
"env_step": {
|
||||
"mean": 49.97409828435841,
|
||||
"std": 2.1747279952000875,
|
||||
"min": 41.243349958676845,
|
||||
"max": 56.928464968223125
|
||||
},
|
||||
"loop_total": {
|
||||
"mean": 74.9630101153426,
|
||||
"std": 32.74896535841012,
|
||||
"min": 67.58100300794467,
|
||||
"max": 379.86371299484745
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user