Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model
Authors: Tianle Li, Jihai Zhang, Yongming Rao, Yu Cheng
NeurIPS 2025 | Venue PDF | LLM Run Details | Input Tokens: 18,841 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 3,963 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct comprehensive experiments across multiple post-training strategies to evaluate the compositional capabilities of VLMs. |
| Researcher Affiliation | Collaboration | Tianle Li, Jihai Zhang, Yongming Rao, Yu Cheng The Chinese University of Hong Kong, Tencent Hunyuan Research EMAIL |
| Pseudocode | No | The paper describes methodologies such as Supervised Fine-Tuning and Reinforcement Learning with GRPO using mathematical formulations and descriptive text (Section 3.1, 3.2), but does not present any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | https://github.com/ltl3A87/Com PA |
| Open Datasets | Yes | We introduce Com PABench, a diagnostic benchmark that systematically evaluates compositional generalization in VLMs across modalities, reasoning tasks, and distribution shifts. |
| Dataset Splits | Yes | More specifically, we generate 4K samples for each individual type of data in training and 500 samples for evaluation. For instance, for Cross-Model Composition task, we mix 4K PT-GR and 4K PT-SR data to train a VLM, and test it on 500 PT-GR, 500 PT-SR, etc. |
| Hardware Specification | Yes | All experiments are conducted on 4 NVIDIA H100 GPUs. |
| Software Dependencies | No | Our implementation of GRPO is based on the open-source R1-V1. |
| Experiment Setup | Yes | For training configurations, we apply consistent hyperparameters: a per-device batch size of 1, a learning rate of 1e 6, and a total of 1 training epoch. For RL experiments, we generate 8 completions per prompt in training. And we set the scale before KL divergence constraints to 0, as we observe a dramatic performance degradation with KL divergence in the objective of optimization. |