I’m grateful for John Schulman’s Opinionated Guide to ML Research, if only for articulating the difference between idea-driven and goal-driven research. This is something that took me the first year and a half of my Ph.D. to figure out, and I’m still trying to pin down my own “research taste” and long-term goals.

I’ve decided to post a few of my goals here, forcing myself to organize and critique my thoughts while simultaneously putting myself on the public record (which will be fun to look back at in a few years). They’re vaguely ordered by the likeliness that I’ll spend time on them.

  1. Develop more compression methods with provable approximation guarantees.

  2. Make AI Dungeon 2 more semantically coherent and game-like.

  3. Achieve BERT-level performance from scratch for less than $100 of compute. (Currently $7,000).
  4. Align the compression community on comparable metrics.

  5. Solve MNIST with a quantum computer simulator
  6. Improve machine translation in low-resource domains.
    • Why does domain adaptation help?
    • What is a principled way of doing domain adaptation?
    • How is this related to fine-tuning pre-trained models?

  7. Bring theory back to DL (not my field)

Writing this list revealed how very few real or novel problems I am aware of. Most problems I know about (especially #6 and #7) come from reading the literature. They’re idea-driven rather than goal-driven research, almost by definition. I’m excited to get back out into the real world this summer and get some exposure to more novel problems.