FIT-MAC

Vision-language-action (VLA) models have achieved strong performance on standard robotic manipulation benchmarks. However, their reliability can degrade substantially when task conditions deviate from the nominal setting. Perturbations in object positions and task instructions may lead to complete task failure. Existing VLA policies generally lack an explicit runtime mechanism for distinguishing these failure modes and selecting an appropriate recovery strategy.

To systematically mitigate this issue, we propose FIT-MAC, a failure-aware inference-time multi-agent control framework for robust VLA manipulation. FIT-MAC augments a frozen VLA policy with an inference-time control layer. First, the Task Planner Agent decomposes the language instruction into an ordered sequence of manipulation subtasks, and the Grounder Agent resolves each instruction-level reference to a visual target. Next, the Monitor-Diagnosis Agent tracks phase-specific execution progress and distinguishes target-binding failures from local execution failures. Finally, FIT-MAC routes each detected failure to a corresponding recovery mechanism, target rebinding for target failures and bounded grasping or placement recovery for execution failures. When no failure is detected, the original VLA remains the default action generator.

Extensive experiments on LIBERO-PRO demonstrate that FIT-MAC achieves state-of-the-art performance under diverse perturbations and consistently improves success rate across multiple VLA backbones. The largest gains occur under position, task, and environment perturbations, while high success rates are maintained under object and semantic perturbations.