Applying the 9D action to tactile video — and what the gel actually does

From each sensor's 9D action [Δp(3), Rot6D(ΔR)(6)] you can rasterize a dense per-pixel 3D displacement field — the apparent motion of a world-static contact:

r6D → ΔR_t        R^app = ΔR_tᵀ        p^app = −ΔR_tᵀ Δp_t
M_t(u,v) = (R^app − I) P_t(u,v) + p^app          P_t = flat gel plane, z = 0

episode_001 · 1524 frames @ 6 fps · left + right sensors · 40×30 mm gel

The prior does not describe the measured tactile flow. It is 542–632× larger than the AllTracker flow measured on the same frames, correlates only +0.22 / +0.36 per frame, and does not beat a zero-motion baseline at any scale. It faithfully describes how the sensor moves; the gel image shows friction-limited slip, which is 2–3 orders of magnitude smaller.

Measured flow (left)

0.260 px

mean per frame

Action prior (left)

164.5 px

632× larger

Per-frame correlation

+0.22 / +0.36

left / right — weak

Sensor still (bottom 20%)

12.8 px

prior, vs 0.09 px measured

Side by side, full episode

Left pane: the measured AllTracker flow overlay. Right pane: the action-derived field, drawn at its own true scale. The prior's arrows saturate the frame because the sensor really is moving that far — the gel simply does not show it.

LEFT sensor — measured | action-derived

1524 frames, 254 s. Axis map +x,−y.

RIGHT sensor — measured | action-derived

Axis map +y,−x — the two sensors are mounted differently.

Rescaled so the fields can be compared at all

10 s from the highest-motion window (t = 68.7–78.7 s). Three panes: measured flow, the action prior divided by its mean ratio, and the prior at true scale. Even after removing the scale difference, the directions do not track: the prior is a smooth global sweep, the measurement is sparse and localized to where contact actually occurs.

LEFT — measured · prior ÷ 367 · prior true scale

Same frames, same arrow spacing; only the scaling differs between panes.

RIGHT — measured · prior ÷ 714 · prior true scale

Magnitudes over the whole episode

Per-frame magnitude traces on log axes and a scatter of prior vs measured. — Log-scale traces and per-frame scatter

The two bands never overlap. In the scatter every point sits far above y = x; if the prior were merely mis-scaled the cloud would be a tight line parallel to it — it is a diffuse blob.

Sensor	measured mean	prior mean	ratio	corr	EPE vs baseline
left (+x,−y)	0.260 px	164.5 px	632×	+0.22	164.4 vs 0.26
right (+y,−x)	0.178 px	96.4 px	542×	+0.36	96.4 vs 0.18

Why — and what the map is still good for

The formula assumes a world-static contact that the gel slides freely beneath. In reality the gel is bonded to the sensor and friction makes the contact patch stick: it travels with the sensor, so the image barely changes. Measured 0.26 px ≈ 0.04 mm, while the sensor genuinely moves tens of millimetres per 1/6 s frame. The prior predicts sensor motion; the tactile image records the residual slip.

That makes it a valid, cheap spatial encoding of the action — a per-pixel layout of the 6-DoF command, decomposed below into a uniform translation term and a rotation term that vanishes at the image centre. It is not a predictor of tactile optical flow, and should not be used as a flow supervision target.

Motion map decomposed into rotation and translation terms for both sensors. — The two terms of the formula (episode_004, one frame)

Rotation term (R^app−I)P_t swirls and vanishes at the centre; translation term p^app is uniform. Here translation dominates 24:1.

Reproduce

# build the maps at the latent frame rate (all 36 episodes)
python data_preprocessing/build_tactile_motion_maps.py --action-file sensor_actions_fps6.pt
→ <episode>/tactile_motion_maps_fps6.npz   motion_{left,right}: (N,60,80,3) float16, metres

# score against measured flow, testing the axis convention rather than assuming it
python data_preprocessing/compare_motion_map_to_flow.py --episode motherboard_0510_episode_001

One episode (1524 frames, both sensors). The same comparison on episode_004 gives ratios of 158–184× and correlations of +0.06/+0.12 — same conclusion, different magnitude. The Δz channel has no optical-flow counterpart and remains unvalidated.