Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Class Distribution Shifts in Zero-Shot Learning: Learning Robust Representations
Authors: Yuli Slavutsky, Yuval Benjamini
NeurIPS 2024 | Venue PDF | LLM Run Details | Input Tokens: 34,163 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 4,550 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We show that our algorithm improves generalization to diverse class distributions in both simulations and experiments on real-world datasets. |
| Researcher Affiliation | Academia | Yuli Slavutsky Department of Statistics and Data Science The Hebrew University of Jerusalem Jerusalem, Israel EMAIL Yuval Benjamini Department of Statistics and Data Science The Hebrew University of Jerusalem Jerusalem, Israel EMAIL |
| Pseudocode | Yes | Algorithm 1 Robust Zero-Shot Representation |
| Open Source Code | Yes | Code to reproduce our results is available at https://github.com/YuliSl/Zero_Shot_Robust_Representations. |
| Open Datasets | Yes | We used the ETHEC dataset [11] which contains 47,978 butterfly images... We used the Celeb A dataset [30] which contains 202,599 images of 10,177 celebrities. |
| Dataset Splits | No | The paper specifies class distributions for training and testing but does not provide explicit percentages or counts for a separate validation dataset split. It mentions that "hyperparameters for all methods were chosen via grid-search in a single experiment repetition" which implies a validation process, but the specific split ratios or methodology for a validation set are not detailed. |
| Hardware Specification | Yes | We ran all experiments on a single A100 cloud GPU. |
| Software Dependencies | Yes | All the code in this work was implemented in Python 3.10. We used the Tensor Flow 2.13 and Tensor Flow Addons 0.21 packages. For evaluation we used the auc function from scikitlearn 1.2. The Celeb A dataset was loaded through Tensor Flow Datasets 4.9 and pandas 1.5 was used to process the ETHEC dataset. Statistical tests were performed using ttest_rel and false_discovery_control functions from scipy.stats 1.11.4. All figures were generated using Matplotlib 3.7. |
| Experiment Setup | Yes | In all our experiments we used margin of m = 0.5 for the contrastive loss and Adam (Kingma & Ba, 2014) optimizer to train all models. ...hyperparameters for all methods were chosen via grid-search... All hyper-parameters are reported in Table 5. |