Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
On the Markov Property of Neural Algorithmic Reasoning: Analyses and Methods
Authors: Montgomery Bohde, Meng Liu, Alexandra Saxton, Shuiwang Ji
ICLR 2024 | Venue PDF | LLM Run Details | Input Tokens: 24,236 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 7,405 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Our extensive experiments, based on the CLRS-30 algorithmic reasoning benchmark, demonstrate that both Forget Net and G-Forget Net achieve better generalization capability than existing methods. |
| Researcher Affiliation | Academia | Montgomery Bohde , Meng Liu , Alexandra Saxton, Shuiwang Ji Department of Computer Science & Engineering Texas A&M University College Station, TX 77843, USA EMAIL |
| Pseudocode | No | The paper provides mathematical formulations and architectural diagrams (Figure 1) but does not include structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code is publicly available at https://github.com/divelab/Forget Net. |
| Open Datasets | Yes | Our extensive experiments, based on the CLRS-30 algorithmic reasoning benchmark, demonstrate that both Forget Net and G-Forget Net achieve better generalization capability than existing methods. |
| Dataset Splits | Yes | We perform experiments on the standard out-of-distribution (OOD) splits present in the CLRS-30 algorithmic reasoning benchmark (Veliหckovi c et al., 2022a). To be specific, we train on inputs with 16 or fewer nodes, and use inputs with 16 nodes for validation. |
| Hardware Specification | No | The paper mentions training models but does not specify any particular hardware (e.g., GPU models, CPU types) used for the experiments. |
| Software Dependencies | No | The paper mentions using "Adam optimizer (Kingma & Ba, 2015)" and a "cosine learning rate scheduler," but it does not provide specific version numbers for these or any other software libraries/dependencies. |
| Experiment Setup | Yes | Specifically, we employ the Adam optimizer (Kingma & Ba, 2015) with a cosine learning rate scheduler and an initial learning rate of 0.0015. The models are trained for 10,000 steps with a batch size of 32. |