ICLR 2024: Numbers and My To Be Read List
The twelfth International Conference on Learning Representations (ICLR 2024) took place during the second week of May in Vienna, Austria. ICLR is widely recognized as one of the premier conferences in the field of machine learning, with a high volume of submissions annually. This post will first present the conference key statistics, followed by a list of papers that received outstanding paper awards and honorable mentions. Lastly, a selection of papers that I find intriguing, based solely on their titles and abstracts, with a few exceptions, will be shared.
Conference Numbers
Here is the ICLR ‘24 stats:
- Submitted papers: 7262
- Accepted paper: 2260
- Acceptance rate: 31%
- Papers selected for oral talk: 86 (3.8% of accepted papers)
- Papers selected as the “spotlight posters”: 366 (16.2% of the accepted papers)
- Number of reviewers: 8950
If you are not familiar with the major machine learning conferences, the aforementioned figures provide a sense of their magnitude and organizational intricacy. This level of magnitude also brings unique challenges. For example, a large number of reviewers are required, some of whom may be undergraduate students who have previously published in these conferences. This situation could potentially result in some inconsistent or unpredictable decisions.
Outstanding Paper Awards
- Generalization in diffusion models arises from geometry-adaptive harmonic representations
- Learning Interactive Real-World Simulators
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
- Protein Discovery with Discrete Walk-Jump Sampling
- Vision Transformers Need Registers
Honorable Mentions
- Amortizing intractable inference in large language models
- Approximating Nash Equilibria in Normal-Form Games via Stochastic Optimization
- Beyond Weisfeiler-Lehman: A Quantitative Framework for GNN Expressiveness
- Flow Matching on General Geometries
- Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled video
- Meta Continual Learning Revisited: Implicitly Enhancing Online Hessian Approximation via Variance Reduction
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
- Proving Test Set Contamination in Black-Box Language Models
- Robust agents learn causal world models
- The mechanistic basis of data dependence and abrupt learning in an in-context classification task
- Towards a statistical theory of data selection under weak supervision
My TBR List
Here are the papers that initially caught my interest, selected based on their titles and abstracts. As my analysis progresses, this list may subsequently shrink.
| Title | Category | Misc. |
|---|---|---|
| Towards a Statistical Theory of Data Selection under Weak Supervision | Efficient ML, Data Manipulation | |
| Tackling the Data Heterogeneity in Asynchronous Federated Learning with Cached Update Calibration | Federated Learning, Data Manipulation | |
| Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging | Efficient ML | |
| Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN | Spiking Neural Network, Efficient ML | |
| The Need for Speed: Pruning Transformers with One Recipe | Efficient ML | |
| Towards Energy Efficient Spiking Neural Networks: An Unstructured Pruning Framework | Efficient ML, Spiking Neural Network | |
| VONet: Unsupervised Video Object Learning With Parallel U-Net Attention and Object-wise Sequential VAE | Efficient ML | |
| Zero Bubble Pipeline Parallelism | Efficient ML | |
| FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning | Efficient ML | |
| SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores | Scalable ML, Reinforcement Learning | |
| Proving Test Set Contamination in Black-Box Language Models | Attacks and Mitigation, LLM | |
| Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platform | Reinforcement Learning, Scalable ML | |
| Backdoor Federated Learning by Poisoning Backdoor-Critical Layers | Attacks and Mitigation, Federated Learning | |
| FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity | Federated Learning, Scalable ML | |
| Understanding Convergence and Generalization in Federated Learning through Feature Learning Theory | Federated Learning | |
| Federated Wasserstein Distance | Federated Learning, Data Manipulation | |
| Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boosting | Federated Learning, Data Manipulation | |
| FedDA: Faster Adaptive Gradient Methods for Federated Constrained Optimization | Federated Learning, Efficient ML | PDF-short - PDF-long |
| FedCDA: Federated Learning with Cross-Rounds Divergence-Aware Aggregation | Federated Learning, Efficient ML | |
| Federated Q-Learning: Linear Regret Speedup with Low Communication Cost | Federated Learning, Reinforcement Learning, Scalable ML | |
| Fake It Till Make It: Federated Learning with Consensus-Oriented Generation | Federated Learning, Data Manipulation, Knowledge Distillation | |
| FedInverse: Evaluating Privacy Leakage in Federated Learning | Federated Learning, Attacks and Mitigation | |
| A Mutual Information Perspective on Federated Contrastive Learning | Federated Learning, Unsupervised Learning | |
| Incentivized Truthful Communication for Federated Bandits | Federated Learning, Attacks and Mitigation | |
| Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning | Federated Learning, Continuous Learning | |
| Hiding in Plain Sight: Disguising Data Stealing Attacks in Federated Learning | Federated Learning, Attacks and Mitigation | |
| Momentum Benefits Non-IID Federated Learning Simply and Provably | Federated Learning, Efficient ML | |
| FedHyper: A Universal and Robust Learning Rate Scheduler for Federated Learning with Hypergradient Descent | Federated Learning, Efficient ML | |
| FedCompass: Efficient Cross-Silo Federated Learning on Heterogeneous Client Devices Using a Computing Power-Aware Scheduler | Federated Learning, Efficient ML | |
| Adversarial Feature Map Pruning for Backdoor | Attacks and Mitigation | |
| Divide and not forget: Ensemble of selectively trained experts in Continual Learning | Ensemble, Continuous Learning | |
| The Curse of Diversity in Ensemble-Based Exploration | Reinforcement Learning, Ensemble | |
| Reward Model Ensembles Help Mitigate Overoptimization | Reinforcement Learning, Ensemble | |
| Fast Ensembling with Diffusion Schrödinger Bridge | Ensemble |
Machine Learning Conference Summary Efficient ML Attacks and Mitigation LLM Federated Learning Data Manipulation Spiking Neural Network Scalable ML Knowledge Distillation Continuous Learning Ensemble