Notice: This website is continuously updated as new conference proceedings and journal volumes are published. Consequently, the figures presented here may differ from those reported in our publications.
A brief overview of our methodology is provided on the Methods page, along with references to publications documenting our approach and links to the code and data used in each study.
The reproducibility variables underlying each score are classified using an automated LLM-based pipeline, validated against a manually labeled dataset. LLM-based classification introduces uncertainty and potential bias; scores should be interpreted as estimates.
Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation
Authors: Matthew O'Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, John C. Duchi
NeurIPS 2018 | Venue PDF | LLM Run Details | Input Tokens: 18,656 Total number of tokens sent to the LLM as input for this paper's analysis. | Output Tokens: 5,569 Total number of tokens produced by the LLM (including reasoning/thinking tokens) for this paper's analysis.
| Reproducibility Variable | Result | LLM Response |
|---|---|---|
| Research Type | Experimental | We demonstrate our framework on a highway scenario, accelerating system evaluation by 2-20 times over naive Monte Carlo sampling methods and 10-300P times (where P is the number of processors) over real-world testing. |
| Researcher Affiliation | Academia | Matthew O Kelly University of Pennsylvania EMAIL Aman Sinha Stanford University EMAIL Hongseok Namkoong Stanford University EMAIL John Duchi Stanford University EMAIL Russ Tedrake Massachusetts Institute of Technology EMAIL |
| Pseudocode | Yes | Algorithm 1 Cross-Entropy Method |
| Open Source Code | No | The paper refers to 'our open-source toolchain' and 'open-source framework', implying the code's nature, but does not provide a direct link or an explicit statement about releasing the code for the described methodology. |
| Open Datasets | Yes | Using the highway traffic dataset NGSim [36], we train policies of human drivers via imitation learning... We use public traffic data collected by the US Department of Transportation [36]. |
| Dataset Splits | No | The paper does not explicitly provide specific training/validation/test dataset splits for reproducibility in the standard sense (e.g., percentages or sample counts for different subsets of a fixed dataset). |
| Hardware Specification | No | The paper mentions 'distributed among available CPUs and GPUs' and 'heterogeneous GPU compute clusters', but does not provide specific hardware details such as GPU or CPU models. |
| Software Dependencies | Yes | Using the asynchronous messaging library Zero MQ [21], our implementation is fully-distributed among available CPUs and GPUs; our rollouts are up to 30P times faster than real time, where P is the number of processors... which uses Unreal Engine 4 [17]. |
| Experiment Setup | Yes | We fix the number of iterations at K = 100, number of samples taken per iteration at Nk = 5000, step size for updates at αk = 0.8, and γ = 0.14. |