feat: add vision transfer backbones and IMF variants
This commit is contained in:
25
experiment_suites/2026-04-05-lewm-vit-transfer/notes.md
Normal file
25
experiment_suites/2026-04-05-lewm-vit-transfer/notes.md
Normal file
@@ -0,0 +1,25 @@
|
||||
# 2026-04-06 LEWM ViT Transfer Notes
|
||||
|
||||
## Root-cause fix
|
||||
|
||||
The first LEWM runs were stopped because the data path still resized each camera view to `224x224` **before** multiview fusion. That preserved the final tensor shape but broke the original LEWM geometry.
|
||||
|
||||
Corrected path now is:
|
||||
|
||||
- **Training dataset**: keep stored per-view `256x256` images (`data.image_resize_shape=null` at launch; dataset instantiate override is `None` for LEWM)
|
||||
- **Eval rollout input**: resize live MuJoCo `480x640` camera images to `256x256` per view
|
||||
- **Backbone**: fuse `front, top, r_vis` on the LEWM axis, then resize fused short side to `224`
|
||||
|
||||
## Verification
|
||||
|
||||
- Local tests passed (`38 passed` across the focused suite)
|
||||
- Remote check:
|
||||
- dataset sample image shape: `(2, 3, 256, 256)`
|
||||
- eval-prepared live frame shape: `(3, 256, 256)`
|
||||
- Remote smoke passed with real checkpoint:
|
||||
- `smoke-lewm-imf-rawpath-emb384-20260406-002002`
|
||||
|
||||
## Current runs
|
||||
|
||||
- `lewm-vit-imf-raw256fix-sim-transfer-emb384-l12-ph16-ex08-step50k-roll10-5880g0-20260406-002124`
|
||||
- `lewm-vit-imf-raw256fix-sim-transfer-emb256-l12-ph16-ex08-step50k-roll10-5880g1-20260406-002124`
|
||||
Reference in New Issue
Block a user