Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Unbiased Multi-Label Learning from Crowdsourced Annotations
Authors: Mingxuan Xia, Zenan Huang, Runze Wu, Gengyu Lyu, Junbo Zhao, Gang Chen, Haobo Wang
ICML 2024 | Venue PDF | LLM Run Details | Input Tokens: 28,462 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 3,831 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | Experiments on various CMLL scenarios demonstrate the effectiveness of our proposed method. |
| Researcher Affiliation | Collaboration | 1School of Software Technology, Zhejiang University, Ningbo, China 2State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China 3Fuxi AI Lab, Net Ease Inc., Hangzhou, China 4Faculty of Information Technology, Beijing University of Technology, Beijing, China. |
| Pseudocode | Yes | Algorithm 1 Pseudo-code of CLEAR. Algorithm 2 Pseudo-code of Transition Matrix Estimation. |
| Open Source Code | Yes | The source code is available at https: //github.com/Mingxuan Xia/CLEAR. |
| Open Datasets | Yes | We conduct our experiments on five benchmark multi-label image datasets1, including Image, Scene, Corel5K, Mirflickr, NUS-WIDE. For these datasets, we corrupt the training sets according to true transition matrices {T mk}M,K m=1,k=1. 1http://mulan.sourceforge.net/datasets-mlc.html |
| Dataset Splits | Yes | For all the experiments, we perform ten-fold cross-validation and report the mean as well as the standard deviation for metric values. |
| Hardware Specification | No | The paper does not provide specific details about the hardware used for running the experiments, such as GPU/CPU models or memory specifications. |
| Software Dependencies | No | The paper mentions using 'Adam optimizer' and 'BCE loss' and indicates that models are trained with 'neural networks,' implying use of a deep learning framework like PyTorch. However, it does not specify exact version numbers for any software libraries, programming languages (e.g., Python), or frameworks. |
| Experiment Setup | Yes | The encoder and decoder for CLEAR are parameterized as three fully connected layer neural networks with hidden sizes 512 and 256. ... we train the models with Adam optimizer ... with a learning rate of 7.5 10 4 and a weight decay of 1e 5. ... the confident-sample number C and the momentum parameter η are fixed as 20 and 0.9 for all settings. The Gaussian subspace dimensionality d is set as 100 for Corel5K and NUS-WIDE, and 50 otherwise. The trade-off parameters α and β are set by 1.0 and 1.1 by default. |