Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Fact Checking in Community Forums
Authors: Tsvetomila Mihaylova, Preslav Nakov, Lluís Màrquez, Alberto Barrón-Cedeño, Mitra Mohtarami, Georgi Karadzhov, James Glass
AAAI 2018 | Venue PDF | LLM Run Details | Input Tokens: 17,617 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,777 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Evaluation results show a MAP value of 86.54, which is 21 points absolute above the baseline. and Evaluation and Results Settings We train an SVM classifier (Joachims 1999) on the 249 examples as described above, where each example is one question answer pair. For the evaluation, we use leave-one-thread-out cross validation, where each time we exclude and use for testing one of the 71 questions together with all its answers. We report Accuracy, Precision, Recall, and F1 for the classification setting. We also calculate Mean Average Precision (MAP). |
| Researcher Affiliation | Collaboration | Tsvetomila Mihaylova,1 Preslav Nakov,2 Llu ıs M arquez,2 Alberto Barr on-Cede no,2 Mitra Mohtarami,3 Georgi Karadzhov,1 James Glass 3 1Sofia University St. Kliment Ohridski , Sofia, Bulgaria 2Qatar Computing Research Institute, Hamad bin Khalifa University, Doha, Qatar 3Massachusetts Institute of Technology, Cambridge, MA, USA |
| Pseudocode | No | The paper does not contain any structured pseudocode or algorithm blocks. |
| Open Source Code | Yes | The dataset and the source code are available online at https: //github.com/qcri/QLFact Checking |
| Open Datasets | Yes | we create a new high-quality dataset CQA-QL-2016-fact , which we release to the research community;3 to the best of our knowledge, this is the first publicly-available dataset specifically targeting factuality in a c QA setting; and 3The dataset and the source code are available online at https: //github.com/qcri/QLFact Checking |
| Dataset Splits | Yes | Evaluation and Results Settings We train an SVM classifier (Joachims 1999) on the 249 examples as described above, where each example is one question answer pair. For the evaluation, we use leave-one-thread-out cross validation, where each time we exclude and use for testing one of the 71 questions together with all its answers. |
| Hardware Specification | No | The paper does not specify the hardware used for running the experiments. |
| Software Dependencies | No | The paper mentions using an 'SVM classifier (Joachims 1999)' but does not provide specific version numbers for any software dependencies or libraries. |
| Experiment Setup | No | The paper mentions training an SVM classifier but does not provide specific experimental setup details such as hyperparameters, optimizer settings, or other system-level training configurations. |