Too Much TrashMuJoCo ↗
- Free, open source; macOS universal build.
- Simple arm, gripper, rigid objects.
- Replay / retarget first; small policies next.
PROJECT 001 / TRASH COLLECTION
Project Plan
A path from phone-mounted capture to useful manipulation data.
01 / CAPTURE & POTENTIAL VALUE
What we collect. Why it matters.
Ranked by potential value for manipulation research
| Signal or Product | Capture route | Benefits & limits | Demand & public benchmarks |
|---|---|---|---|
| 01 / Reviewed task episodes & outcomes✓ iPhone 18 · source videoNot Started |
| Benefits
|
|
| 02 / Calibrated interaction episodes✓ iPhone 18 · estimated poseNot Started |
| Benefits
|
|
| 03 / Object masks & tracks✓ iPhone 18 · source video onlyIn ProgressPlanning · SAM3 |
| Benefits
|
|
| 04 / Depth & geometry✓ iPhone 18 Pro · LiDARNot Started |
| Benefits
|
|
| 05 / Inertial motion✓ iPhone 18Not Started |
| Benefits
|
|
| 05 / Trigger squeeze events✓ iPhone 18 · active captureIn ProgressPlanning |
| Benefits
|
|
| 06 / RGB video & timing✓ iPhone 18In ProgressPlanning |
| Benefits
|
|
| 06 / Context✓ iPhone 18 · with permissionNot Started |
| Benefits
|
|
02 / INTEROPERABILITY
Reuse the ecosystem’s formats.
| Format | Best fit | Backend design choice |
|---|---|---|
| 01 / LeRobot v3 ↗INITIAL SUPPORT · EPISODES | MP4 + Parquet + episode metadata | First export target: aligned video / events with explicit observations. Never invent robot actions. |
| 02 / COCO ↗INITIAL SUPPORT · MASKS | Per-frame instance masks in lossless RLE | Save SAM3-generated masks now; preserve object IDs and frame timing separately. |
| 03 / ASAM OpenLABEL ↗ | Temporal / multimodal JSON | Add when richer event / object annotation is needed. |
| 04 / RLDS ↗ | Episodes / steps; TFDS ecosystem | Consumer-requested research adapter. |
| 05 / robomimic ↗ | HDF5 demonstrations | Consumer-requested imitation-learning export. |
Initial plan: LeRobot for episodes + COCO for SAM3 masks. OpenLABEL / RLDS / HDF5 remain optional adapters. Processing and exports are planned—not implemented.
03 / SIMULATION SETUP
Start small. Start with MuJoCo.
Build the simulation harness before testing policy value. Start with a small CPU-based task on the M1 Pro—not foundation-model training.
NVIDIA Isaac Sim / Isaac Lab ↗
- Robotics simulation + learning workflows.
- Supported NVIDIA RTX host required; not local M1.
- Consider a compatible rented host only if needed.
- Check trajectory reach, timing, and collisions.
- Establish robomimic BC on the public Lift PH dataset.
- Randomize objects, friction, lighting, and noise.
Simulation validates the tested model only. A borrowed robot or partner lab must confirm real-world transfer. Grass, flexible litter, and contact are especially hard to model.
04 / SPECIFIC BASELINE
Start with BC on robosuite Lift.
Baseline: robomimic feed-forward behavioral cloning (BC), trained on the public Lift proficient-human low-dimensional dataset; evaluate in its matching MuJoCo / robosuite environment.
Available today
- Public Lift PH dataset and BC implementation.
- Lift simulation: randomized cube pickup.
- Small CPU experiment first; pin matching versions and smoke-test on M1.
Then test our data
- Lift validates the pipeline—not litter-data value.
- Next: public Can pick-and-place baseline, then a matched litter task.
- Compare the same BC model with / without task-matched, retargeted demonstrations.
Success gate
- Held-out task success, drops, collisions.
- Same compute, splits, seeds; confidence intervals.
- Video + squeeze events alone do not supply BC actions.
05 / LOW-EXPENSE UPGRADES
Add only what improves the result.
| Priority | Added signal | Validation |
|---|---|---|
| 1 / Printed calibration + jaw markers | Scale, camera-to-tool alignment, visible jaw opening. Very low materials cost. | Compare to measured motion / calipers. |
| 2 / Potentiometer or Hall sensor + logger | Continuous opening proxy. Example component; electronics / assembly extra. | Measure actual jaw opening, hysteresis, clock offset. |
| 3 / Contact or load sensor | Contact / load evidence. Low-cost parts; integration uncertain. | Repeat known-load tests; not full tactile sensing. |
| 4 / Borrowed second phone | Outcome visibility + reference view. | Synchronize views; measure review benefit. |
Cost bands are planning judgments, not kit quotes. Use paired before / after tests; keep only useful additions.
06 / PROJECT TRAIL
Follow the evidence.
Repeat where it matters.
Hover, focus, or tap a node for details and its feedback path. Progress depends on passing each gate—not a calendar.
Define the acceptance spec
- Choose task, target user, and success metric.
- Measure phone capabilities and calibration.
- Agree consent, license, and quality gates.
Gate → observable signals and testable requirements.
Build recording. Review a small pilot.
- Synchronize streams; retain originals and failures.
- Check drift, missing frames, and label agreement.
- Freeze an independent held-out split.
Loop ↺ Capture fails QA → revise the contract and recollect.
Prove model and policy benefit
- Compare perception and simulated policies with / without our data.
- Report held-out improvement and uncertainty.
- Count accepted episodes and review cost.
Loop ↺ No improvement → refine collection / labels before adding volume.
Package for existing tools
- Version episodes, labels, units, and frames.
- Validate LeRobot episodes and COCO SAM3 masks together.
- Round-trip a sample in a consumer loader.
Gate → a partner can load the data without bespoke tooling.
Build a small MuJoCo harness
- Validate retargeting and jaw-state assumptions.
- Smoke-test BC / Lift PH; freeze randomized test scenes.
- Add a sensor only for a measured gap.
Loop ↺ Weak action signal → instrument, recalibrate, and repeat the pilot.
Validate transfer and demand
- Borrow robot access through a lab / partner.
- Agree sample acceptance and paid-pilot terms.
- Scale, refine, or pivot based on evidence.
Loop ↺ Transfer fails → revisit benchmarks and simulation; do not scale yet.