Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Relation-Aware Language-Graph Transformer for Question Answering
Authors: Jinyoung Park, Hyeong Kyu Choi, Juyeon Ko, Hyeonjin Park, Ji-Hoon Kim, Jisu Jeong, Kyungmin Kim, Hyunwoo Kim
AAAI 2023 | Venue PDF | LLM Run Details | Input Tokens: 17,010 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 5,221 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We validate the effectiveness of QAT on commonsense question answering datasets like Commonsense QA and Open Book QA, and on a medical question answering dataset, Med QA-USMLE. On all the datasets, our method achieves state-of-the-art performance. |
| Researcher Affiliation | Collaboration | 1Korea University 2 NAVER 3 NAVER Cloud 4 NAVER AI Lab |
| Pseudocode | No | The paper describes its proposed methods using mathematical formulations and textual explanations but does not include any explicitly labeled pseudocode or algorithm blocks. |
| Open Source Code | Yes | Our code is available at http://github.com/mlvlab/QAT. |
| Open Datasets | Yes | We evaluate our method on three question-answering datasets: Commonsense QA (Talmor et al. 2019), Open Book QA (Mihaylov et al. 2018), and Med QA-USMLE (Jin et al. 2021). |
| Dataset Splits | Yes | Since the official test set labels are not publicly available, we mainly report performance on the in-house development (IHdev) and test (IHtest) sets following (Lin et al. 2019).; we experiment on the official data split from (Mihaylov and Frank 2018).; We use the same data split as (Jin et al. 2021). |
| Hardware Specification | No | The main paper does not explicitly describe the hardware used for experiments. It mentions 'The details on the backbone LMs and technical details are in the supplement' but this is not provided in the main text. |
| Software Dependencies | No | The paper mentions using specific Language Models like RoBERTa and SapBERT, but it does not specify version numbers for these LMs or other software dependencies (e.g., Python, PyTorch, TensorFlow versions) in the main text. |
| Experiment Setup | No | The paper states, 'The details on the backbone LMs and technical details are in the supplement.' However, no specific hyperparameters, training configurations, or system-level settings are provided in the main body of the paper. |