The model pyramid
In software engineering, we often talk about the Testing Pyramid - the idea that different types of automated tests have different values, and so your test suite should be made up of a combination of these in different proportions.
The general principle is:
You should have lots of fast unit tests, some integration tests, and few slow end-to-end tests.
I think the current machine learning / AI landscape could learn from this.
The current hype cycle would have you believe that all problems can and should be solved by the biggest models we have ever created - enormous transformer-based models with parameters into the billions. But in reality, most problems can be solved with much simpler models. In fact, systems often benefit from many small, simple models more than from a few giant ones.
Let's look at the distribution of problems that we typically apply ML to, and draw an analogy. It's not a perfect one, but I like to think there's some learned wisdom in it.
Solving Problems
In general in engineering, most problems are actually easy. At least, the distribution of problems that we choose to solve is skewed easy.
Think about it this way - there's a huge number of tasks where you wouldn't even reach for ML. There are probably millions of problems that we solve in code in any application. The majority of them never even touch ML.
- "What controller should I invoke from this URL?"
- "How many times has this user logged in?"
- "Can this user view this page?"
Some problems require a more complex approach. Perhaps a statistical or nondeterministic approach is required.
- "What content should I show this user?"
- "Which button is most likely to increase CTR?"
- "Is this user likely to disengage?"
Some problems are very hard indeed.
- "What’s the answer to any arbitrary question the user asks?"
- "Is this a photo of a bird?"
Notice that with the increase in complexity comes a decrease in the number of those problems that we have. We choose to solve much more easy problems than hard ones.
The Analogy
At the bottom of the model pyramid, you have many simple models. Linear regressions, logistic classifiers, Bayesian logistic regressions all work extremely well. They're fast, interpretable, easy to train and easy to run. You can design and ship them in an afternoon.
In the middle sit moderately complex models. Gradient boosted trees, Bayesian mixed models and small neural networks are able to capture richer patterns for those problems that need it.
At the very top of the pyramid is a small number of heavy models - the transformers, the language models - used only when absolutely necessary. They're slow to build, slow to train and very intensive to run.
Why many small models work
Small models are faster to build, easier to implement and reason about. You don't need billions of datapoints for a logistic regression with a few well-engineered features.
Small models can cover a surprisingly wide range of behaviours, while the few large ones handle the genuinely hard problems. The result is a modeling stack that's modular, interpretable, and resilient, instead of one giant model that's expensive, fragile, and hard to debug.
If you reach for a large model for each problem, each problem will take exponentially more time and compute to solve. With small models, you have the budget - both time and resources - to tackle more problems.
The bitter lesson
The bitter lesson is often understood as telling us that in the long run, scale swallows all problems. That's not strictly true, the actual lesson is slightly narrower - that general methods, given enough compute, tend to beat hand-crafted domain knowledge.
The analogy here isn't "never use a big model". The lesson of the model pyramid is orthogonal to the bitter lesson.
The lesson is use the smallest model that makes sense for your problem, and then more problems become solvable.