AI Reproducibility Index

Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.


A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.


The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.

Journal of Artificial Intelligence Research (JAIR) - 2015

Documentation Rate of Empirical Papers by Reproducibility Variable

Distribution of Empirical Papers by Number of Documented Variables

Website:

Venue Year Papers
Reproducibility Score Reproducibility Score based on Gundersen et al. (2025). See Methods for details.
Documentation Score Documentation Score is the average score over the seven reproducibility variables for empirical research papers. See Methods for details.
% Empirical Percentage of papers that are empirical research vs theoretical research.
% Industry Percentage of empirical research papers with at least one author from Industry.
Website
JAIR 2015 48 0.39 4.18 68.75% 21.21%
Pseudocode
Open Source Code
Open Datasets
Dataset Splits
Hardware Specification
Software Dependencies
Experiment Setup
A Case-Based Reasoning Framework to Choose Trust Models for Different E-Marketplace Environments ❌ ❌ ❌ ✅ ❌ ❌ ✅ 2
A Compositional Framework for Grounding Language Inference, Generation, and Acquisition in Video ❌ ✅ ✅ ✅ ❌ ❌ ✅ 4
Achieving Goals Quickly Using Real-time Search: Experimental Results in Video Games ✅ ✅ ✅ ✅ ✅ ✅ ✅ 7
Agnostic Pointwise-Competitive Selective Classification ✅ ❌ ✅ ✅ ❌ ❌ ✅ 4
Approximate Value Iteration with Temporally Extended Actions ✅ ❌ ✅ ❌ ✅ ✅ ✅ 5
AutoFolio: An Automatically Configured Algorithm Selector ✅ ✅ ✅ ✅ ✅ ✅ ✅ 7
Belief Change with Uncertain Action Histories ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Bypassing Combinatorial Protections: Polynomial-Time Algorithms for Single-Peaked Electorates ✅ ❌ ❌ ❌ ❌ ❌ ❌ 1
Clause Elimination for SAT and QSAT ✅ ✅ ✅ ❌ ✅ ✅ ✅ 6
Coactive Learning ✅ ❌ ✅ ✅ ❌ ❌ ✅ 4
Coherent Predictive Inference under Exchangeability with Imprecise Probabilities ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Compressing Optimal Paths with Run Length Encoding ❌ ❌ ✅ ✅ ✅ ✅ ✅ 5
Computing Convex Coverage Sets for Faster Multi-objective Coordination ✅ ❌ ❌ ❌ ✅ ❌ ✅ 3
Constraining Information Sharing to Improve Cooperative Information Gathering ❌ ❌ ❌ ❌ ❌ ❌ ✅ 1
Continuing Plan Quality Optimisation ✅ ✅ ✅ ❌ ✅ ❌ ✅ 5
Decision Making with Dynamic Uncertain Events ✅ ❌ ❌ ❌ ❌ ❌ ✅ 2
Deterministic Oversubscription Planning as Heuristic Search: Abstractions and Reformulations ✅ ❌ ✅ ❌ ❌ ❌ ✅ 3
Distributed Evaluation of Nonmonotonic Multi-context Systems ✅ ✅ ❌ ❌ ✅ ✅ ✅ 5
Evolutionary Dynamics of Multi-Agent Learning: A Survey ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Expressiveness of Two-Valued Semantics for Abstract Dialectical Frameworks ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
ITSAT: An Efficient SAT-Based Temporal Planner ✅ ❌ ✅ ❌ ✅ ✅ ✅ 5
Inferring Team Task Plans from Human Meetings: A Generative Modeling Approach with Logic-Based Prior ❌ ❌ ❌ ❌ ❌ ✅ ✅ 2
Knowledge-Based Textual Inference via Parse-Tree Transformations ✅ ❌ ✅ ✅ ❌ ❌ ✅ 4
Lazy Model Expansion: Interleaving Grounding with Search ✅ ❌ ✅ ❌ ✅ ✅ ✅ 5
Learning Relational Event Models from Video ✅ ❌ ✅ ✅ ❌ ❌ ✅ 4
Leveraging Online User Feedback to Improve Statistical Machine Translation ✅ ❌ ✅ ✅ ✅ ❌ ✅ 5
Mechanisms for Multi-unit Combinatorial Auctions with a Few Distinct Goods ✅ ❌ ❌ ❌ ❌ ❌ ❌ 1
Model Theory of XPath on Data Trees. Part I: Bisimulation and Characterization ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Modeling the Lifespan of Discourse Entities with Application to Coreference Resolution ❌ ❌ ✅ ✅ ❌ ✅ ✅ 4
On a Practical, Integer-Linear Programming Model for Delete-Free Tasks and its Use as a Heuristic for Cost-Optimal Planning ✅ ❌ ✅ ❌ ✅ ✅ ✅ 5
On the Subexponential-Time Complexity of CSP ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
PAGOdA: Pay-As-You-Go Ontology Query Answering Using a Datalog Reasoner ✅ ✅ ✅ ❌ ✅ ✅ ❌ 5
Pay-As-You-Go Description Logic Reasoning by Coupling Tableau and Saturation Procedures ✅ ✅ ✅ ❌ ✅ ✅ ✅ 6
Placement of Loading Stations for Electric Vehicles: No Detours Necessary! ❌ ❌ ✅ ❌ ✅ ❌ ✅ 3
Possible and Necessary Winners of Partial Tournaments ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Probabilistic Inference Techniques for Scalable Multiagent Decision Making ❌ ❌ ✅ ❌ ✅ ❌ ✅ 3
Regular Path Queries in Lightweight Description Logics: Complexity and Algorithms ✅ ❌ ❌ ❌ ❌ ❌ ❌ 1
Relations Between Spatial Calculi About Directions and Orientations ❌ ✅ ❌ ❌ ❌ ❌ ❌ 1
Revision by History ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Satsisfiability and Systematicity ✅ ✅ ✅ ✅ ✅ ✅ ✅ 7
Scheduling Conservation Designs for Maximum Flexibility via Network Cascade Optimization ✅ ❌ ✅ ❌ ❌ ❌ ✅ 3
Solving #SAT and MAXSAT by Dynamic Programming ✅ ✅ ❌ ❌ ✅ ✅ ✅ 5
The Ceteris Paribus Structure of Logics of Game Forms ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Tree-Width and the Computational Complexity of MAP Approximations in Bayesian Networks ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Using Machine Translation to Provide Target-Language Edit Hints in Computer Aided Translation Based on Translation Memories ❌ ✅ ✅ ✅ ❌ ✅ ✅ 5
Weighted Electoral Control ✅ ❌ ❌ ❌ ❌ ❌ ❌ 1
Weighted Regret-Based Likelihood: A New Approach to Describing Uncertainty ❌ ❌ ❌ ❌ ❌ ❌ ❌ 0
Word vs. Class-Based Word Sense Disambiguation ✅ ❌ ✅ ✅ ❌ ❌ ✅ 4