HYDRA: a HYper agent
for Dynamic compositional visual ReAsoning

Fucai Ke*, Zhixi Cai*, Simindokht Jahangard*,
Weiqing Wang, Pari Delir Haghighi, Hamid Rezatofighi
Monash University
*Equal Contribution
arXiv

Abstract

Recent advances in visual reasoning (VR), particularly with the aid of Large Vision-Language Models (VLMs), show promise but require access to large-scale datasets and face challenges such as high computational costs and limited generalization capabilities. Compositional visual reasoning approaches have emerged as effective strategies; however, they heavily rely on the commonsense knowledge encoded in Large Language Models (LLMs) to perform planning, reasoning, or both, without considering the effect of their decisions on the visual reasoning process, which can lead to errors or failed procedures. To address these challenges, we introduce HYDRA, a multi-stage dynamic compositional visual reasoning framework designed for reliable and incrementally progressive general reasoning. HYDRA integrates three essential modules: a planner, a Reinforcement Learning (RL) agent serving as a cognitive controller, and a reasoner. The planner and reasoner modules utilize an LLM to generate instruction samples and executable code from the selected instruction, respectively, while the RL agent dynamically interacts with these modules, making high-level decisions on selection of the best instruction sample given information from the historical state stored through a feedback loop. This adaptable design enables HYDRA to adjust its actions based on previous feedback received during the reasoning process, leading to more reliable reasoning outputs and ultimately enhancing its overall effectiveness. Our framework demonstrates state-of-the-art performance in various VR tasks on four different widely-used datasets.

Contribution

A multi-stage dynamic compositional visual reasoning framework designed for reliable and incrementally progressive general reasoning.

  1. Cognitive Agent. Integrating a cognitive reinforcement learning-based agent as a controller into a framework to foster hyper decision-making and behavior across diverse environments, enhancing system cohesion, performance, and reasoning capabilities.
  2. Interative Processing. Employing LLM as a natural language planner that enables the dynamic generation of valid instruction samples for iterative processing. The samples are vary in both the complexity and scope of perception tasks assigned with validity probabilities.
  3. Incremental Reasoning. Applying incremental reasoning, storing information from previous states aids both the LLM and RL agent in acquiring fine-grained visual information through VFMs and the visual-perception-to-text module, thereby refining their reasoning processes.
  4. Open-source. Our model and code base are publicly available.

BibTeX


@misc{ke2024hydra,
      title={HYDRA: A Hyper Agent for Dynamic Compositional Visual Reasoning}, 
      author={Fucai Ke and Zhixi Cai and Simindokht Jahangard and Weiqing Wang and Pari Delir Haghighi and Hamid Rezatofighi},
      year={2024},
      eprint={2403.12884},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
-->