Lessons from the Netflix Prize: Changing Requirements and Cost-Effectiveness

Last updated:

Original Article on Techdirt: Why Netflix Never Implemented the Algorithm that won the Netflix $1 Million Challenge

The Winning Solution (1st Place) was an Ensemble of more than 100 separate models

Out of all 107 component models, the two best individual algorithms were based on

  • Matrix Factorization and

  • RBM (Restricted Bolztmann Machines)

These two were the only parts actually used by Netflix.

Too Complex for the Improvements on the other Solutions

The Netflix team made a simplified model using just the two best performing individual models (as above) and used that in actual production systems.

The full model had slightly better performance than the simplified model, but this came at a much higher engineering effort (the full model was much more complicated to build).

Requirements had changed by the time Netflix actually implemented it

The challenge was made with movie rentals in mind but, by the time it was going to be used, streaming had become a much more important part of the Netflix business model.

Also, Netflix had recently gone international and become available on many more devices, thus affecting the way viewers consumed the products.

dilbert-requirements Apparently, data science is software too.

Recommending movies for rental vs recommending streaming content

Recommending streaming services is fundamentally different from recommending movies for rental.

When you rent something, you want to be more certain that it's good, because of the time/money you'll put into it, but When viewing streaming content, you can afford to experiment a little bit, but you are more likely to give up on something if you don't immediately like it.

Also, there's much more data available (implicit feedback e.g. half-watched movies, movies watched multiple times) in a streaming service context and this information can be leveraged to improve recommendations.

This short post is part of the data newsletter. Click here to sign up.


References

Dialogue & Discussion