I am a Research Scientist at Anthropic. Formerly at Google Deepmind and Disney Research. My current focus is Scaling and the intersection of Information Theory and Deep Learning. I got my Ph.D. in Theoretical Condensed Matter Physics at Cornell University, supervised by Jim Sethna. I got my B.S. at Caltech, where I majored in Physics.
Below you'll find a list of my publications and other writings, for my recent writing, see my blog.
K Everett, L Xiao, M Wortsman, AA Alemi, R Novak, PJ Liu, I Gur, J Sohl-Dickstein, LP Kaelbling, J Lee, J Pennington
Understanding parameterizations and how to scale them.
B Lester, J Lee, AA Alemi, J Pennington, A Roberts, J Sohl-Dickstein, N Constant
Trying to train transformers on top of transformers with arithmetic compression.
PAGI
Squeezing more performance out of models by fine-tuning on filtered generated responses.
PAGI
It's easy to get models to perform arithmetic incorrectly, if you just ask nicely.
M Wortsman & PAGI
Studying problems of large scale models in the small.
I Seroussi, AA Alemi, M Helias, Z Ringel
Working out thermodynamic speed limits for learning.
AA Alemi, B Poole
Targetting the predictive distribution directly with a variational method.
Y Ruan, S Singh, WR Morningstar, AA Alemi, S Ioffe, I Fischer, JV Dillon
Ensembling the heads of SSL methods gives nice gains.
L Anderson-Conway, V Birodkar, S Singh, H Mobahi, AA Alemi
Ensembling within a trajectory gives some simple gains.
Y Du, D Ho, AA Alemi, E Jang, M Khansari
Using VIB to help robots open doors.
I Korshunova, D Stutz, AA Alemi, O Wiles, S Gowal
Looking more carefully, IB models aren't fully robust to adversarial examples.
S Stanton, P Izmailov, P Kirichenko, AA Alemi, AG Wilson
Knowledge distillation doesn't seem to work as well as people assume it does.
AA Alemi, WR Morningstar, B Poole, I Fischer, JV Dillon
VIB can be rederived as a half-Bayesian half-Maximum likelihood method.
WR Morningstar, AA Alemi, JV Dillon
Multisample bound that does better than Bayes at prediction for misspecified models.
WR Morningstar, C Ham, AG Gallagher, B Lakshminarayanan, AA Alemi, JV Dillon
Simple density-of-states inspired out of distribution detection.
DS Karls, M Bierbaum, AA Alemi, RS Elliot, JP Sethna, EB Tadmor
Database for Interatomic Potentials.
R Novak, L Xiao, J Hron, J Lee, AA Alemi, J Sohl-Dickstein, SS Schoenholz
Simple to use python package for training infinitely wide neural networks.
AA Alemi
Most modern inference procedures can be rederived as a simple variational bound on a predictive information bottleneck objective.
R Shwartz-Ziv, AA Alemi
While they seem complex, infinite ensembles of infinitely-wide networks are simple enough to enable tractable calculations of many information theoretic quantities.
I Fischer, AA Alemi
A class conditional version of VIB shows good robustness.
Z Dong, D Oktay, B Poole, AA Alemi
Modern RNNs do not optimally capture predictive information in sequences.
T Conte, E DeBenedictis, N Ganesh, T Hylton, JP Strachan, RS Williams, AA Alemi, L Altenberg, G Crooks, J Crutchfield, L del Rio, J Deutsch, M DeWeese, K Douglas, M Esposito, M Frank, R Fry, P Harsha, M Hill, C Kello, J Krichmar, S Kumar, SC Liu, S Lloyd, M Marsili, I Nemenman, A Nugent, N Packard, D Randall, P Sadowski, N Santhanam, R Shaw, A Stieg, E Stopnitzky, C Teuscher, C Watkins, D Wolpert, J Yang, Y Yufik
A position paper on the future of thermodynamic computing.
B Poole, S Ozair, A van den Oord, AA Alemi, G Tucker
Overview of recent advances in variationally bounding mutual information.
B Seybold, E Fertig, AA Alemi, I Fischer
Sometimes a worse decoder gives better representations.
I Fischer, AA Alemi, JV Dillon, TFP Team
TFP makes VAEs easy.
CB Clement, M Bierbaum, KP O'Keeffe, AA Alemi
More people should use the ArXiv as a dataset.
E Fertig, A Arbabi, AA Alemi
Some rich decoder VAEs can magically focus on salient information.
LX Hayden, R Chachra, AA Alemi, PH Ginsparg, JP Sethna
Matrix factorization gives automatic and continous sector assignments to stocks.
H Choi, E Jang, AA Alemi
Even though it shouldn't work, robust likelihoods can detect OOD data in practice.
AA Alemi, I Fisher
Modern variational latent variable modelling looks a lot like Thermodynamics.
AA Alemi, I Fischer, JV Dillon
VIB builds robust classifiers which are aware of what they don't know.
S Abu-El-Haija, B Perozzi, R Al-Rfou, AA Alemi
Building better graph representations.
AA Alemi, I Fischer
A variational lower bound on the mutual informations in GANs highlight some of their problems.
AA Alemi, B Poole, I Fischer, JV Dillon, RA Saurous, K Murphy
Adopting a representational view of VAEs can help explain away some of their problems.
JV Dillon, I Langmore, D Tran, E Brevdo, S Vasudevan, D Moore, B Patton, AA Alemi, M Hoffman, RA Saurous
Paper accompanying library.
M Bierbaum, BD Leahy, AA Alemi, I Cohen, JP Sethna
Better featuring of colloids.
LX Hayden, AA Alemi, PH Ginsparg, JP Sethna
Jeffrey's prior doesn't really work for neural networks.
K Fragkiadaki, J Huang, AA Alemi, S Vijayanarasimhan, S Ricco, R Sukthankar
Pedestrian motion is stochastic which creates certain challenges.
C Szegedy, S Ioffe, V Vanhoucke, AA Alemi
Residual connections improve the inception family of classifiers.
AA Alemi, I Fischer, JV Dillon, K Murphy
A modern formulation of the Information Bottleneck which is friendly towards neural networks.
B Poole, AA Alemi, J Sohl-Dickstein, A Angelova
You can target separate divergences for the generator and discriminator of a GAN.
R Shin, AA Alemi, G Irving, O Vinyals
Attempting to learn tree-structured representations.
AA Alemi
Blogpost accompanying open source release of Inception Resnet V2.
G Irving, C Szegedy, AA Alemi, N Eén, F Chollet, J Urban
Using neural networks to improve automatic theorem proving.
CJM Mathy, F Gonda, D Schmidt, N Derbinsky, AA Alemi, J Bento, FM Delle Fave, JS Yedidia
Using ADMM to do fast trajectory planning.
AA Alemi, M Bierbaum, CR Myers, JP Sethna
A fun pedadogical introduction to epidemiology and statistical mechanics.
AA Alemi
A collection of four of my graduate student projects.
JM Cashore, X Zhao, AA Alemi, Y Liu, PI Frazier
Better clustering through content conditioning.
AA Alemi, P Ginsparg
Using word2vec vectors to do automatic text segmentation.
A Taloni, AA Alemi, E Ciusani, JP Sethna, S Zapperi, CAM La Porta
Elastic models of skin cancer.
E Tadmor, R Elliott, D Karls, A Ludvik, J Sethna, M Bierbaum, AA Alemi, T Wennblom
R Elliott, E Tadmor, D Karls, A Ludvik, J Sethna, M Bierbaum, AA Alemi, T Wennblom
Building a website to collect interatomic potentials and score them.
PY Huang, S Kurasch, JS Alden, A Shekhawat, AA Alemi, PL McEuen, JP Sethna, U Kaiser, DA Muller
Applying elastic theory to the atomic scale.
MM Baraldi, AA Alemi, JP Sethna, S Caracciolo, CAM La Porta, S Zapperi
Simple models of skin cancer growth.
RS Ottens, Volker Quetschke, Stacy Wise, AA Alemi, Ramsey Lundock, Guido Mueller, David H Reitze, David B Tanner, Bernard F Whiting
Exploration of quantum tunnelling as a mechanism for cooling the next generation LIGO detectors.
AA Alemi
Undergraduate project on the history of the Runge Vector.
AA Alemi
Undergraduate research project on synchronization in nano cantilevers.
AA Alemi, DJ Stevenson
Undergraduate research investigating whether two collisions in the opposite direction could explain Venus' lack of moon and slow rotation.
For more writing, see my blog.
I've contributed to various Google Opensource projects including: NanoDo, JAX, Neural Tangents, Tensorflow Probability, Tensorflow Datasets, and TensorFlow.
I also opensourced a top class image classification network: Inception Resnet V2.
A self-contained Diffusion colab is available here.
An open source demo of VIB is available here.
pychebfun is an open source reimplimentation of ChebFun.
texpad.alexalemi.com is a simple static MathJAX formula as copyable image generator.
alexalemi/advent are my solutions to Advent of Code.
A popular overview of LLMs and how to think about AI.
Introduction to Statistical Thinking through Randomization.
Dimensional analysis and basic order of magnitude physics.
An overview of information theory for machine learning.
Everything is KL divergence minimization.
Thinking about the future of science and machine learning.
A variational way to directly target the posterior predictive.
The variational approach to machine learning.
Viewing VAEs as four stroke engines.
Bayesian inference doesn't optimize for prediction in mispecified models.
Another version of the relationship between thermodynamics and machine learning.
The Variational Information Bottleneck can be viewed as a sort of half-Bayesian approach.
Thermodynamics from a Probabilistic perspective and machine learning from a thermodynamic perspective.
Another version of my TherML talk.
I attempt to show that most modern forms of inference can be viewed as optimizing a variational bound on a predictive information bottleneck objective.
I offer arguments both for and against learning compressed representations in the form of a generalized information bottleneck.
Drawing an analogy between Thermodynamics and modern deep variational latent variable generative modelling
An overview of my work, which often amounts to reinterpreting existing techniques in a representational light.
An earlier talk relating thermodynamics and machine learning for a physics audience.
A representational reinterpretation of VAEs that help clarify issues such as posterior collapse.
VIB classifiers capture uncertainty effectively.
I used to be quite active on the Physics Stackexchange.
Espdic is a simple Esperanto Dictionary.
One Look: A ludum dare 45 entry.
A Zombie Simulator for our paper.
The Physics Virtuosi was a blog I ran with some friends from graduate school.
A Mirror of My Old Homepage at Cornell University has some random things.