Notice: The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates. Full accuracy metrics and methodology are described in Coakley et alK. L. Coakley, T. Snelleman, H. Hoos, and O. E. Gundersen, "The Shift Toward Open and Reproducible AI Research," Under Review, 2026. doi: 10.48550/arXiv.2606.16974..
Transformers as Soft Reasoners over Language
Authors: Peter Clark, Oyvind Tafjord, Kyle Richardson
IJCAI 2020 | Venue PDF | LLM Run Details | Input Tokens: 18,367 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 5,349 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We train transformers to reason (or emulate reasoning) over these sentences using synthetically generated data. Our models, that we call Rule Takers, provide the ο¬rst empirical demonstration that this kind of soft reasoning over language is learnable, can achieve high (99%) accuracy, and generalizes to test data requiring substantially deeper chaining than seen during training (95%+ scores). |
| Researcher Affiliation | Industry | Peter Clark , Oyvind Tafjord and Kyle Richardson Allen Institute for AI, Seattle, WA EMAIL |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | A live demo and all our datasets are available at https://allenai.org/data/ruletaker |
| Open Datasets | Yes | Each dataset contains 100k examples (25k of each Type without/with negation)... A live demo and all our datasets are available at https://allenai.org/data/ruletaker |
| Dataset Splits | Yes | Data is randomly split 70/10/20 into train/dev/test partitions, ensuring no overlap of theories between each partition. |
| Hardware Specification | No | The paper does not provide specific hardware details (e.g., GPU/CPU models, memory, or cloud instance types) used for running the experiments. |
| Software Dependencies | No | The paper mentions models like RoBERTa and BERT, but does not specify software versions for libraries, frameworks, or programming languages (e.g., Python, PyTorch, TensorFlow versions). |
| Experiment Setup | No | We use ο¬xed hyperparameters (learning rate etc), inheriting the settings from Ro BERTa on RACE [Liu et al., 2019]. |