Welcome to TimesPeople

Get Started
The latest activity in your network...

Latest Activity

Tuesday, July 28, 2009

Technology



July 27, 2009, 4:59 pm

Netflix Challenge Ends, But Winner Is in Doubt

The $1 million contest for the Netflix Prize ended Sunday with two teams in the computing version of a photo finish. The outcome, it seems, remains in some doubt.

The competition, begun in 2006, really shifted into high gear in the last month, after an international team of statisticians, machine learning experts and computer engineers declared that they had come up with algorithms that could improve the movie recommendations made by Netflix’s internal software, Cinematch, by at least 10 percent.

The announcement by that team, BellKor’s Pragmatic Chaos, set off a 30-day race, under the contest rules, for other teams to try to best them. Entries flowed fast and furious since then. And just minutes before the Sunday deadline, a team composed of other former teams, appropriately called the Ensemble, nudged ahead of BellKor’s, on the public Web leaderboard.

So The Ensemble won, right? Not necessarily. In an e-mail message Sunday night, Chris Volinsky, a scientist at AT&T Research and a leader of the BellKor’s team, said: “Our team is in first place as we were contacted by Netflix to validate our entry.” And in an online forum, another member of the BellKor team, Yehuda Koren, a researcher for Yahoo in Israel, said his team had “a better test score than the Ensemble,” despite what the rival team submitted for the leader board.

So is BellKor the winner? Certainly not yet, according to a Netflix spokesman, Steve Swasey. “There is no winner,” he said.

A winner, Mr. Swasey said, will probably not be announced until sometime in September at an event hosted by Reed Hastings, Netflix’s chief executive. The movie rental company is not holding off for maximum P.R. effect, Mr. Swasey said, but because the winner has not yet been determined.

The Web leader board, he explained, is based on what the teams submitted. Next, Netflix’s in-house researchers and outside experts have to validate the teams’ submissions, poring over the submitted code, design documents and other materials. “This is really complex stuff,” Mr. Swasey said.

A leading member of the Ensemble, Domonkos Tikk, a Hungarian computer scientist, did not sound too hopeful. “We didn’t get any notification from Netflix,” Mr. Tikk said in a phone interview from Hungary. “So I think the chances that we won are very slight. It was a nice try.”


17 Comments

  1. 1. July 27, 2009 7:08 pm Link

    Without any doubt whatsoever the losing will sue.

    — Stephen Andrus
  2. 2. July 27, 2009 7:37 pm Link

    I don’t think customers really care…I know I am perfectly capable of deciding what movies I want to see. While Netflix is certainly helpful with its recommendation, I don’t think a 10% improvement is really going to affect the quality of my movie-rental experience that much.

    — C
  3. 3. July 27, 2009 8:14 pm Link

    This is journalistic reporting at its worst. The author assumes that the reader is already fully aware of the Netflix contest by giving out just the barest of details about it - only focusing on the outcome. Oh wait - I take it back…the author provides links to stories that were already written. LINKS! Since when is a link to another story considered a journalistic tool for a publication such as the New York Times? Lazy, lazy!!

    — Abbie
  4. 4. July 27, 2009 8:44 pm Link

    as long as netflix is not offering a meaningful porno selection, i can’t say how this algorithm will improve MY experience

    — Mikey Likes It
  5. 5. July 27, 2009 8:56 pm Link

    That million dollars would have been better spent sending Hollywood’s studio chiefs back to film school.

    — James
  6. 6. July 27, 2009 8:58 pm Link

    I think that it would impress Netflix’s advertisers more than their customers.

    — alboyjr
  7. 7. July 27, 2009 9:00 pm Link

    Now I know how Netflix is using the Blu-Ray surcharge they are gouging me with.

    — Alan
  8. 8. July 27, 2009 10:17 pm Link

    The Netflix contest has certainly focused the effort of algorithm engineers, but has it really achieved an improvement in user recommendation experience? The underlying assumption in the contest is that the algorithm can be used and scaled by Netflix to predict what their users will like for movies which they haven’t seen–usually considered the most interesting problem for recommendation systems. But based on the methodologies of the contest and the teams, it’s not clear that this will be the case. The danger of Netflix’s narrow definition of what constitutes “recommendations” is that you end up with a decent system for predicting how a user will rate a movie on a 1-5 scale.

    Many open questions remain, such as:

    How is the ability to predict viewer ratings (movies already seen) different from the problem of movie discovery (movies not yet seen)?

    Will a 10% increase in Cinematch actually translate into noticeable product improvements for Netflix subscribers?

    Does the Netflix contest ask a question with direct relevance to Netflix’s overall business goals and financial bottom-line?

    Are 1-5 star ratings objective? Are there better ways of soliciting input from viewers that would yield more meaningful data with less tedium?

    /Michael Papish
    CEO, MediaUnbound

    — michael papish
  9. 9. July 27, 2009 10:33 pm Link

    Haha. (abbie) Well, actually the readers do know what they’re talking about because we follow tech news on this biz/tech blog. Also, links are the ubiquitous currency of the internet, it seems to be the appropriate tools for a blog post. Blogs aren’t meant to be long Sunday magazine articles.

    — tamonline
  10. 10. July 27, 2009 11:04 pm Link

    I hope Netflix puts the algorithm to work as soon as possible. If I have to check Not Interested on one more Katt Williams video, I’m going to lose it.

    — Satorical
  11. 11. July 27, 2009 11:32 pm Link

    The claim by a commenter that this is “journalistic reporting at its worst” is ridiculous.

    This is a blog entry, not an article. Different standards apply, as blogs are by convention briefer and more informal than articles.

    For an ideal comparison, just read the companion article to this blog, written by the same author:
    http://www.nytimes.com/2009/07/28/technology/internet/28netflix.html

    — Jian
  12. 12. July 28, 2009 12:21 am Link

    On their forum, Netflix states that both teams met the minimum requirement to win the prize. The actual performance difference on the “test” set is probably around 0.01% - statistically insignificant. This is a TIE!

    — Insider
  13. 13. July 28, 2009 12:23 am Link

    They should do this for dating sites to match people and save billions of dollars from unnecessary divorces and broken hearts and negatively affected children.

    Who cares about movie recommendations?

    — Miss Understood
  14. 14. July 28, 2009 1:04 am Link

    I’m encouraged by the exceptional quality of work that the teams put in, no doubt fanned by not only fame, but spoils of $1 mil certainly doesn’t hurt.

    We’ve hurt so many times before, about the power (or wisdom or whatever) of the crowd. What we mostly see from are simply silly, silly stuffs (eg. star gazing and celebrity chasing on twitter, highly opinionated but otherwise useless blogs). That’s why this netflix competition is such a breath of fresh air.

    To top it off, netflix doesn’t skim it like google does. Afterall, netflix will very likely use the winner’s algo. Remember what google does, telling illustration artists to submit their work to be used by google as wallpaper for free? Now, THAT is what I call cheap.

    — tiddle
  15. 15. July 28, 2009 6:14 am Link

    They should do the same for the congress, to see if we improve! : )

    — Chris
  16. 16. July 28, 2009 7:56 am Link

    Mike of [generic media company] wrote:

    “Does the Netflix contest ask a question with direct relevance to Netflix’s overall business goals and financial bottom-line?”

    Yes, because the system is not only about recommending better choices, it’s about *not* recommending relatively worse choices. A customer gets half-way through a movie that doesn’t really interest him, decides he’ll get back to the rest when he can, ultimately sends the DVD back, unfinished. But that may be two weeks later, during which time that copy is otherwise unavailable, and Netflix still has to eat the postage.

    The good recs are still good recs. The real payoff comes with reducing the bad recs. That customer above would have been just as happy with 9 movies as with 10, given the 10th was (for him) a clunker. (Or yeah, an improved rec for the 10th movie would make him happier.)

    — Yasha
  17. 17. July 28, 2009 9:43 am Link

    THE LOSER WILL WIN — more than $1M. Much like the winner of American Idol is tied into a so-so contract, the winner of this contest must make all their algorithms publicly available. They can license them to non-netflix firms but that assumes they can turn them into patents or other protected IP. The “loser” does not have to reveal their “secrets” (although there is certainly significant overlap w/ the winning algorithms) and pawn them off as “trade secrets” or even keep them in-house and certainly get a few major contracts.

    In the end I won’t be surprised if Netflix gives both teams $1M (but will the second team need to reveal its codes?)

    Paul Hamilton

    — pvh

Add your comments...

Required

Required, will not be published

What We're Reading

Driven to Distraction

VIDEO
Distracted Drivers

75 ThumbnailMost people are aware of the dangers of trying to multitask while driving, but most continue to do it anyway.

INTERACTIVE FEATURE
Gauging Your Distraction

75 ThumbnailA game illustrates the potential consequences of distractions like texting on your driving ability.

RELATED ARTICLES

About Bits

Bits offers a steady stream of news and analysis on the technology industry throughout the day from New York Times writers and freelancers. We cover start-ups, tech leaders like Google and Apple, enterprise technology, government policies and the way the Internet is changing how we live and work. Read more.

Feedback

Tell us what you like, don't like and want to read more about. Send us e-mail with your comments

For news tips and press announcements, please use the e-mail links on the blog home page to reach our writers and editors.

Subscribe

Follow us on

Contributors

Damon Darlin
Damon Darlin
Technology editor, San Francisco

Consumer electronics, consumer issues, pricing

David F. Gallagher
David F. Gallagher
Deputy technology editor, New York

Internet, blogs, search, cellphones

Vindu Goel
Vindu Goel
Deputy Technology Editor, New York

Social networking, telecommunications, enterprise technology, blogs

Sam Grobart
Sam Grobart
Assistant Technology Editor, New York

Cellphones, cameras, computers, home & entertainment technology

Saul Hansell
Saul Hansell
Reporter, New York

Telecommunications, government policy, cellphones, cable TV

Miguel Helft
Miguel Helft
Reporter, San Francisco

Search, Internet, online marketing, Google, Yahoo

Steve Lohr
Steve Lohr
Reporter, New York

Enterprise computing, economics of technology, Microsoft, I.B.M.

Claire Cain Miller
Claire Cain Miller
Reporter, San Francisco

Venture capital, technology start-ups

Matt Richtel
Matt Richtel
Reporter, San Francisco

Consumer electronics, video game business, Silicon Valley, Internet gambling, Internet pornography

Brad Stone
Brad Stone
Reporter, San Francisco

Internet, technology and society, policy and law, security, social networks, electronic commerce, eBay, Amazon.com

Ashlee Vance
Ashlee Vance
Reporter, San Francisco

Enterprise computing, software, network technology, semiconductors, trends in corporate technology

Jenna Wortham
Jenna Wortham
Reporter, New York

Internet, Web start-ups, digital culture, communications, convergence, N.Y. tech scene

Multimedia

Interactive Feature
75 ThumbnailPolaroid Photographers Plead Their Case

Five photographers who regularly shoot with Polaroid instant cameras and film explain their hopes for a revival of Polaroid.

Audio Slide Show
75 ThumbnailMaking a Magazine With MagCloud

College students produce a fashion magazine more cheaply and easily using MagCloud, a new self-publishing site from Hewlett-Packard.

Archive

Recent Posts

July 27
(4)

Booyah Wants You to Level Up, Online and Off

Three alums of Blizzard, creator of World of Warcraft, are hoping to move quests into the real world with a new iPhone game.

July 27
(17)

Netflix Challenge Ends, But Winner Is in Doubt

The $1 million contest to improve Netflix's movie recommendation software ended with two teams in something of a photo finish, but the outcome remains in doubt.

July 26
(5)

The ‘Mayors’ of Manhattan Meet and Compete

Top players of a social networking game called Foursquare hold their first offline gathering.

July 24
(90)

Eleven Years of Ambition and Failure at AOL

Tim Armstrong is just the latest of a long stream of executives who have promised great things for AOL. Here's a review of more than a decade of corporate promises.

July 24
(19)

Twitter Takes a Step Toward Commercial Accounts

Twitter unveiled Twitter 101, a series of case studies that explain how businesses use Twitter.