Personalization – Makes Sense(s) But It’s Really Difficult to Do

Designing algorithms to provide personalized information is really hard, actually much harder than one would assume. My dissertation was on personalization agents for information retrieval, and the field is still pretty much at the same stage.

One of the most interesting academic challenges in the personalization of information area is the Netflix Challenge. Netflix offered $1 million to anyone who can improve the accuracy of its DVD recommendation engine by at least 10 percent. As of this posting, the prize remains unclaimed.

Netflix, like Amazon and other major online retailers, has applications that learn customers' preferences by the ratings customers assign to movies when they log into their accounts. These applications then recommend movies (or products) to customers. They do evaluations to gauge the accuracy of the recommender systems. The Netflix Challenge is to improve the recommendation accuracy by at least 10%.

So far, the biggest improvement has been approximately 6% by the Simon Funk team, using a singular value decomposition, which basically results in a clustering approach.

So, there is a long way to go to reach the 10% improvement mark. Naturally, the easier stuff has already been done and the remaining portions are running into the 'hard to define' movies and the changing taste of customers over time.

I also believe part of the issue is that ‘like’ and ‘dislike’ are really irrational processes and using a rational algorithm is always going to have limited application.

Maybe the answer is to use something that is ‘irrational’ like simulated annealing or even more crazy, a semi-random algorithm.

Read the full NY Times article, If You Liked This, You’re Sure to Love That, by Clive Thompson