ClawMetry
Developer guide
Robot cameras, telemetry, and replay: how to investigate a failed task
A step-by-step investigation method for camera freshness, state mismatches, recording gaps, and historical review.
Begin with the first thing you cannot explain
A robot misses a grasp. The video is blurry, the model says it reached the object, and the joint chart looks ordinary. Start with a small question: at what point did the requested action stop matching the evidence? Investigating that interval is more productive than reading a whole run as one undifferentiated log.
Our SO-101 work put camera views next to leader and follower representations. The BluPe pilot added another complication: station feedback and run-specific evidence could arrive independently. Both experiences point to the same method. Keep sources distinct, align them as carefully as the clocks allow, and resist filling gaps with assumptions.
Check whether the image belongs to the moment
For each camera, inspect the source identity, capture time if known, receipt time, and the displayed frame's age. Confirm that replay is showing the selected interval rather than continuing to follow the newest image. A correct picture of the wrong moment can be more misleading than a missing frame.
If two cameras disagree, check their clocks and viewpoints before drawing a conclusion. One may see an object hidden from the other. Neither image automatically identifies the command that caused a movement. Preserve the original frame references so a later reviewer can examine the same evidence.
Compare the action request with measured feedback
Choose the relevant joint, target, or trajectory segment and identify what the interface actually exposed. A requested position and a measured position should have different labels. An accepted packet is another separate event. If the interface provides no measurement, mark the physical response unconfirmed.
Then inspect timing around the mismatch. Was a new request sent before a previous one completed? Did the source stop reporting? Was the task interrupted? These are investigation questions, not automatic diagnoses. The evidence may support several explanations until the controller logs or operator review narrows them.
Treat observer failures as part of the record
A failed task can coincide with a failed recording. Count missing or dropped events where the recorder can detect them, preserve reconnects, and explain when the observer stopped receiving data. Do not turn the end of the recording into a successful end of the task.
For example, an application may record that a run ended while the last image still shows an object in the gripper. That establishes an application event and an earlier visual state. It does not tell you whether the object was placed afterward. The right outcome may be unknown until another source resolves the gap.
Keep inspection replay separate from motion replay
Inspecting saved telemetry should never silently resend commands. Label a historical view clearly and make its selected time visible. If the software also supports executing a recorded trajectory, put that operation behind a distinct control and the controller's required authorization.
This is particularly useful when debugging a shared or remote robot. Another person may currently own the live session. Reviewing yesterday's failure should not compete with their task or change the machine's state. A read-only history view is an investigation tool in its own right.
Write a conclusion someone else can challenge
Finish the review with a short statement of the task, the observed mismatch, supporting evidence, and remaining alternatives. Include the relevant frame and event references in the local report. If a calibration change or software update is suspected, compare a controlled follow-up run rather than rewriting the earlier record.
ClawMetry's historical screenshots show this workflow's early foundations, including camera history and arm-state views. The goal for any integration is a record another engineer can inspect and question. A good replay should make an uncertain conclusion visible, not make an incomplete recording look continuous.
Sources and evidence
External sources support the attributed background. Integration accounts describe historical project evidence. Guidance and proposed tests are identified in the article.