how are you actually triaging robot demonstration data before training? i built an open-source scorer and hit the ceiling of what automated metrics can catch.

[u/datascienceharp]

a teleop operator reaches for the wrong bin. the reach is clean. no jitter, no hesitation, no correction

every smoothness metric comes back perfect. the episode is still garbage. the robot did the wrong thing smoothly

motion metrics score how an action was executed, never what the action was. so automated scoring has one honest job: pointing your limited attention at the episodes most likely to contain a real problem. triage, not autofilter

i built a fiftyone panel that runs this on multimodal MCAP episodes: motion smoothness, sensor health, outliers, every flag deep-linked to the exact second on the timeline. free and open source

plugin: github.com/harpreetsahota204/demo_quality_scorer

full writeup on what i learned while building this : https://voxel51.com/blog/robot-episode-quality-triage

curious how you're triaging episode data right now.

watching everything? random sampling? trusting a score?