Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
FrugalML: How to use ML Prediction APIs more accurately and cheaply
Authors: Lingjiao Chen, Matei Zaharia, James Y. Zou
NeurIPS 2020 | Venue PDF | LLM Run Details | Input Tokens: 17,810 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 6,790 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We conduct systematic experiments using ML APIs from Google, Microsoft, Amazon, IBM, Baidu and other providers for tasks including facial emotion recognition, sentiment analysis and speech recognition. Across various tasks, Frugal ML can achieve up to 90% cost reduction while matching the accuracy of the best single API, or up to 5% better accuracy while matching the best API s cost. |
| Researcher Affiliation | Academia | 1Department of Computer Sciences, 2 Department of Biomedical Data Science Stanford University |
| Pseudocode | Yes | Algorithm 1 Frugal ML Strategy Training. |
| Open Source Code | Yes | We release our code and our dataset1 of 612,139 samples annotated by commercial APIs as a resource to aid future research in this area. 1https://github.com/lchen001/Frugal ML |
| Open Datasets | Yes | Table 2: Datasets sample size and number of classes... We release our code and our dataset1 of 612,139 samples annotated by commercial APIs as a resource to aid future research in this area. 1https://github.com/lchen001/Frugal ML. |
| Dataset Splits | No | The paper mentions 'training on half of FER+' but does not explicitly describe a separate validation dataset split or how it was used. |
| Hardware Specification | No | The paper does not provide specific hardware details such as GPU/CPU models, processor types, or memory amounts used for running experiments. |
| Software Dependencies | No | The paper refers to various open-source models and tools (e.g., 'Bixin', 'Vader', 'Deep Speech', 'PyTorch' indirectly), but it does not specify version numbers for any software dependencies or libraries. |
| Experiment Setup | No | The paper describes the Frugal ML algorithm but does not provide specific experimental setup details such as learning rates, batch sizes, or other hyperparameters for its training or for the underlying models. |