Here are some non-exhaustive notes on the things I do and don’t understand across a variety of topics. I read about Feynman doing something similar; I’ve found that keeping these notes has been helpful for me so far.

If you know about something that I list here and you want to tell me about it, I’d love to hear your explanation!

## Paul Christiano’s work

Concepts I don't

- Meta-execution
- learning the right model
- Annotated functional programming
- benign model-free RL
- benign model-based RL (https://ai-alignment.com/aligned-search-366f983742e9)

Concepts I understand

- reliability and robustness
- oversight/reward learning
- Two approaches: IRL, learning from human feedback
- deliberation and amplification
- secure
- competitive
- scalable
- adversarial training
- ensembling and consensus
- Against mimicry
- When can mimicry go wrong? robot and blocks example
- Weak and strong HCH
- KWIK learning— knows what it knows

## Reinforcement learning

- Generalized advantage estimation
- vanishing gradients
- I’ve forgotten the RL stuff I learned before. At some point I should draw a list of connections between RL ideas and things in deep RL.
- Eg td(lambda)
- WRT policy gradients: Finite difference methods, likelihood ratio methods, REINFORCE, natural policy gradients
- Variance reduction, in policy gradients
- local response normalization
- q-prop
- control variate

things formerly on this list:

- linear policies
- I wasn't quite clear on how convolution across multiple timesteps works. Now I understand
- Conv nets—I don’t really understand them very well and should do the tutorial
- I feel a little better now
- Action advantage/Advantage function
- I think I know this now! This is how much better an action is than other actions would be.
- Specifics of convolution: what are the arguments? I should consider rewriting these functions myself.
- Max pooling
- Python’s named_tuple things
- qfunc network vs target network
- Dueling DQN
- Double DQN
- model-free RL
- model-based RL
- batch normalization
- TRPO

## Math

- calculus of variations, Lagrange multipliers
- I understood this for a brief period once!
- Lagrange multiplier type things for constrained optimization
- Why is group theory cool? This is embarrassing to admit: I like abstract algebra but I don’t actually know of concrete situations where it’s very helpful.
- Contour integration

## Statistics

- Time series modelling
- ANOVA

## Statistical mechanics

- Equipartition theorem
- Apparently this implies the Dulong-Petit law—the heat capacity of a solid is proportional to 3 times the “gas constant” times the number of molecules. So elements with lower atomic weights have higher heat capacity per unit mass.
- This law overestimates the heat capacity, because nearby molecules have correlated motion.
- Thermal wavelength
- Maximum entropy thermodynamics
- Suppose I have a bunch of oxygen and a bunch of nitrogen in tanks next to each other, at the same temperature. This has lower entropy than the state where the gasses are at diffusive equilibrium. How in general do I extract energy from the process of mixing them?
- I don’t know about the different kind of cycles and stuff
- I have now decided that I don’t care much about this
- Deriving the Sackur-Tetrode equation for entropy from information theory

- "In addition to using the thermodynamic perspective of entropy, the tools of information theory can be used to provide an information perspective of entropy. The Sackur–Tetrode equation for entropy can be derived in information theoretic terms. The equation can be seen to consist of the sum of four entropies (missing information) due to positional uncertainty, momenta uncertainty, the quantum mechanical uncertainty principle and the indistinguishability of the particles.” this sounds pretty dope and I don’t understand it
- Calculating a black hole radius from no-hair and thermodynamics.
- $$dU = T \cdot dS - P \cdot dV + \mu \cdot dN$$
- I am sort of familiar with these facts now
- How do we use fundamental physical models to calculate:
- thermal conductivity
- phase transitions
- Why does a mix of oxygen and nitrogen start to liquify at below the freezing point of either oxygen or nitrogen? (This is explained in 5.4 of Schroeder.)
- I still don’t know anything about how engines and refrigerators work, but I don’t know if I care

Things I now understand:

- Some systems have their temperature fall as their energy rises. And allegedly some have their entropy fall as their energy rises? I don’t get this.
- Negative temperatures. For example, a paramagnet with more than half its bits in the odd side of things. These try to give their energy to any finite-temperature system that’s nearby.
- $$\frac1T = \frac{\partial S}{\partial U}$$
- Gibbs free energy is defined as $$G = U + PV - TS$$: this is the amount of energy required to create the thing and make space for it, if you’re allowed to use temperature from the environment.
- What proportion of atoms in a paramagnet have aligned spins, as a function of temperature?
- Enthalpy: the energy of a system plus the work required to make room for it, in an environment with constant pressure. Enthalpy H = U + P V.
- What precisely is a degree of freedom? How come I can’t turn my degree of freedom into more degrees of freedom by interpreting “horizontal position” with some weird ugly function from R -> R^2?
- How do fractional degrees of freedom work?
- The answer to this is that “degrees of freedom” are just an approximation to the underlying reality. What actually matters is all of the definite energy wavefunctions available, and their energy levels. You can’t turn your degree of freedom into more degrees of freedom with a space-filling curve because your degree of freedom was just being approximated as a continuous space and if you apply a space-filling curve then you break the approximation. The reason we bother talking about “degrees of freedom” at all is that often in practice we actually do have degrees of freedom that can be modelled pretty well as uncorrelated. If our degrees of freedom of a system (let’s call them a and b) are "orthogonal", this means that we can write the energy of the system as the sum of a function of a and a function of b.
- Fractional degrees of freedom occur when your energy function can’t be separated out into a function of many of its degrees of freedom.
- An example model of fractional degrees of freedom is the Ising model.
- Imagine you have a cylinder that's a centimeter wide and 1000km tall, with a base here on Earth somewhere. You have some amount of gas in it. How does pressure vary with height in the cylinder?
- I think this is the barometric pressure equation.
- What is the distribution of velocities and positions in this cylinder?
- When we extract energy from increasing entropy (eg by mixing two substances), where does the energy come from? Kinetic energy of the molecules?
- yes

## Machine learning

- Deep learning
- Adam
- RmsProp
- Details of attentional models like the Transformer
- Variational auto encoders
- global average pooling
- Inception score
- resnets
- Variational inference, evidence lower bound, etc
- Markov chain monte carlo
- SVMs

## Biology

- What exactly is protein folding?
- What exactly is Adenosine triphosphate (ATP)? How does it work?
- I should more clearly understand how DNA works. It’s something like:
- DNA gets turned into RNA somehow
- You use ribosomes to express the DNA as proteins somehow?
- The base pairs code for proteins by using three consecutive letters to give you one of twenty amino acids. How does this work?
- How does DNA get reproduced?
- How do we sequence DNA?

## Physics

- General relativity
- What are the equations that describe how light moves through space?
- it follows geodesics? What are those?
- How do you get gravitational waves? What are the waves “in”?
- There must be some second order PDE, I just don’t quite get what it is
- Cosmology
- I don’t quite know how we measure some stuff.
- Things I think I know: We know how far away stars are because many of them happen to be a very similar size to each other? (Why is that?) And then we know how fast they’re going because they have hydrogen spectra that is redshifted.
- Why did our solar system form planets instead of remaining a cloud of stuff? Why does mass congeal?
- I should read the first few chapters of the Loeb book about this
- Quantum field theory
- How long does positronium last? how do I calculate that?
- What does it mean when the Hamiltonian “couples” to something?
- Nonrelativistic quantum mechanics
- I still don’t totally get Slater determinants
- I don’t really understand bosons
- Why does spin “add” so that helium nuclei or whatever are bosons? seems weird
- relatedly, I don’t really understand what bose-einstein condensates are.
- Why does degeneracy matter in QM?
- What’s an example where the energy level of a system is different between two bosons and two distinguishable particles?
- I vaguely recall it matters to perturbation theory
- Are there QM potentials for every distribution in the exponential family? harmonic oscillator gives Gaussian
- What is second-order pertubation theory?
- Why do interactions between electrons and photons have to take particles to eigenstates?
- creation and annihilation operators
- I should revise the use of time-dependent perturbation theory to predict transition probabilities
- Interaction part of the Hamiltonian
- Ladder operators for the harmonic oscillator
- Scattering
- Rotation group as applied to the quantum mechanics of spin
- Nuclear physics
- There’s a shell model and a liquid drop model. The shell model takes into account the fact that the nuclei have to occupy states with energy that increases quadratically and capacity that increases according to 2n**2. But these models aren’t super solid. How not accurate are they? Is the inaccuracy just because we don’t know how the strong nuclear force works? How did we try to model them in the 1940s?
- I now understand these better. In particular I understand that the models are semi-empirical—we don’t even try to model the interactions from first principles, we just try to fit empirical parameters into our models to match reality as well as we can.
- How do atomic bombs work? How complicated are the rules required to see what reactions are possible?
- Classical physics
- I don’t totally understand Hamiltonian and Lagrangian mechanics. I forget the Euler-Lagrange equation.
- Computing stuff
- I still don’t understand the rules about fundamental limits on computation. How does the Landauer limit work? Why is entropy a thing? I know it’s something to do with the unitarity of physics but I don’t quite understand it.
- I don’t remember anything about quantum computing
- I don’t understand what negentropy is
- Acoustics
- Why does Jeff Kaufman’s pipe sound like it does? What is the physics that tells me how to guess the resonant frequency of any pipe system? What is the analogy between air pipe resonance and circuit resonance?
- Helmholtz resonance
- Misc
- How do lasers work?
- How do LEDs work?
- How do semiconductors work?