The Bitter Lesson

If you're building anything that involves AI, then you have to know about the bitter lesson.

The bitter lesson is an article by Rich Sutton from March 13, 2019. This sounds like ancient AI lore but is only six years old.

Rich Sutton, a pioneer of reinforcement learning and a Turing Award winner, wrote a short essay saying that 70 years of AI has taught him and the best quote from it is also the first sentence.

"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."

In other words, as the cost per unit of computation falls (as computation gets cheaper), you can throw more computing power at the problem, and it will generally outperform any optimizations that you've done on top.

Most AI research has been conducted as if the computation available to the agents were constant. We're behaving as if the ground we're standing on is solid, but it's not. It doesn't matter if you found this crazy new technique for improving your AI app. Eventually, the computers will get so powerful, the AI will get so powerful that we'll outperform your solution.

Any time spent optimizing will probably just be a waste of time. Time that you could have just spent by waiting for the models to get better.

This is the bitter lesson.

We are all just wasting our time. Leveraging domain knowledge is bad. Nothing really matters in AI research, apart from improving the power of the machines.

This is controversial, to say the least.

If you see someone on the internet talking about the bitter lesson, now you know what they're talking about.

Here's the original essay https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf