Lessons from the Netflix Prize: Changing Requirements and Cost-Effectiveness
Last updated:- The Winning Solution (1st Place) was an Ensemble of more than 100 separate models
- Too Complex for the Improvements on the other Solutions
- Requirements had changed by the time Netflix actually implemented it
- Recommending movies for rental vs recommending streaming content
Original Article on Techdirt: Why Netflix Never Implemented the Algorithm that won the Netflix $1 Million Challenge
The Winning Solution (1st Place) was an Ensemble of more than 100 separate models
Out of all 107 component models, the two best individual algorithms were based on
Matrix Factorization and
RBM (Restricted Bolztmann Machines)
These two were the only parts actually used by Netflix.
Too Complex for the Improvements on the other Solutions
The Netflix team made a simplified model using just the two best performing individual models (as above) and used that in actual production systems.
The full model had slightly better performance than the simplified model, but this came at a much higher engineering effort (the full model was much more complicated to build).
Requirements had changed by the time Netflix actually implemented it
The challenge was made with movie rentals in mind but, by the time it was going to be used, streaming had become a much more important part of the Netflix business model.
Also, Netflix had recently gone international and become available on many more devices, thus affecting the way viewers consumed the products.
Apparently, data science is software too.
Recommending movies for rental vs recommending streaming content
Recommending streaming services is fundamentally different from recommending movies for rental.
When you rent something, you want to be more certain that it's good, because of the time/money you'll put into it, but When viewing streaming content, you can afford to experiment a little bit, but you are more likely to give up on something if you don't immediately like it.
Also, there's much more data available (implicit feedback e.g. half-watched movies, movies watched multiple times) in a streaming service context and this information can be leveraged to improve recommendations.
This short post is part of the data newsletter. Click here to sign up.
References
Netflix Blog: Netflix Recommendations: Beyond the 5 Stars (Part 1)
- Numerous insights into the (then) way Netflix recommended stuff: movie rows, recommendation diversity, recommendation awareness