SemAnCorr: Semantic Anchored Correspondence for Zero-Shot Manipulation Skill Transfer

Overview video

A short overview of SemAnCorr: the semantic‑anchored correspondence problem, our training‑free pipeline, and real‑world manipulation skill demonstrations and transfer executions.

Transferring manipulation skills across object instances that share functionality but differ in geometry remains a fundamental challenge in robot learning. While recent correspondence methods leverage dense visual descriptors and 3D feature fields, nearest‑neighbor feature matching often produces spatially incoherent correspondences that fail to recover the local geometric frames required for reliable skill transfer.

We introduce SemAnCorr, a training‑free framework that establishes dense correspondence by selecting semantically consistent anchor regions through joint pose‑correspondence optimization and propagating these constraints over the object surface using functional maps. The resulting correspondences preserve both semantic consistency and geometric coherence, enabling object‑centric manipulation skills to transfer across geometrically diverse instances.

We evaluate SemAnCorr on a dense correspondence benchmark built on PartNet‑Mobility, achieving 90.8% semantic accuracy while substantially improving geometric coherence over existing correspondence and affordance‑transfer baselines. Finally, we show these improvements translate directly into real‑world manipulation performance: using a single demonstration, SemAnCorr enables substantially more reliable zero‑shot manipulation skill transfer to previously unseen objects than existing correspondence methods.

Evaluated on a dense correspondence benchmark built from seven PartNet‑Mobility categories, against functional‑map, 2D semantic, and 3D feature‑field baselines. SemAnCorr leads on semantic accuracy in every category and is the only method that achieves both high continuity and meaningful surface coverage.

Within‑category semantic accuracy (sAcc ± std, %). SemAnCorr improves 6.2 points over the strongest baseline, D3Fields.

Beyond category boundaries

Many manipulation skills transfer across functionally related but visually distinct categories. SemAnCorr's anchor confidence also doubles as a lightweight, training‑free signal for whether a demonstrated skill is likely to transfer between two objects at all.

Every component earns its place

From one demonstration to a novel object

Five kinesthetically demonstrated skills — peeling, opening, pouring, twisting, and cutting — are transferred zero‑shot to unseen target objects using only the computed dense correspondence. A contact keyframe (end‑effector pose + contact region) and relative motion segments are transferred via Procrustes alignment to the corresponding region on the target.

Task	D3Fields	SemAnCorr
Task 1: Banana Peeling	7/10	8/10
Task 2: Pen Cap Opening	9/10	9/10
Task 3: Kettle Pouring	3/10	7/10
Task 4: Bottle Twisting	1/10	7/10
Task 5: Scissor Cutting	3/10	6/10

Real‑world task success rate, 10 trials per task with varied object pose and viewpoint.

SemAnCorr correspondence on real‑world object pairs, Tasks 1–5.

D3Fields fails on Tasks 3 & 4: spatially incoherent correspondence yields incorrect local frames, causing task failure.

Both methods perform comparably on simpler tasks where the correct interaction region is relatively unambiguous. On tasks that additionally require preserving a local geometric frame — approach direction, end‑effector orientation — D3Fields' spatially incoherent correspondence produces erroneous frames despite often identifying the correct semantic region. SemAnCorr's remaining failures stem mainly from mesh‑to‑workspace alignment error rather than correspondence estimation.