Our taste in movies is highly idiosyncratic, and at odds with critics' preferences

"What we find enjoyable in movies is strikingly subjective -- so much so that the industry's targeting of film goers by broad demographic categories seems off the mark," says Pascal Wallisch, a clinical assistant professor in New York University's Department of Psychology and the senior author of the study, which appears in the journal Projections.

"Critics may be adept at evaluating films, but that doesn't mean their assessments will accurately predict how much the public will like what they see," adds co-author Jake Whritner, a graduate of the Cinema Studies Program at NYU's Tisch School of the Arts and currently part of the Cognitive and Data Science Lab at Rutgers University-Newark.

Over the past century, filmmakers have sought to control viewers' attention through different editing and shooting techniques, with the assumption that the audience will respond in the same way.

However, while neural and attentional processing has been extensively examined, the level of agreement in the appraisal of movies among viewers has not been studied. Similarly, while past research has analyzed the relationship between reviews and box-office success as well as agreement among critics for films, none have explored agreement between critics and the general public.

To address these questions, the researchers considered more than 200 major motion pictures, taking into account popularity, financial success, and critics' reviews. They then surveyed over 3,000 participants, asking them to give a rating of how much they liked each of the films in the sample that they had previously seen. The researchers also asked participants to provide demographic information (e.g., age, gender) and whether they consider movie critics' recommendations in choosing which movies to see.

Finally, Wallisch and Whritner gathered reviews from 42 publicly accessible critics or rating sites (e.g., IMDb) for each of the films in the sample.

The results generally showed low levels of correlation in movie preferences among study participants. However, there were some patterns. As the number of jointly seen films increased, so did the correlation of the ratings for such films -- at least up to a point. When the number of ratings for a given film reached between 100 and 120, correlation grew to its highest point -- but as this number continued to increase, correlation for that film's ratings began to dip, before spiking up again at around 180 commonly seen films.

Looking at demographics, the survey showed greater agreement in film ratings among male participants than among females -- and this difference between genders was statistically significant. However, agreement among both men and women in films was relatively low. There was also little correlation between movie ratings and age -- however, because the overall sample skewed younger, the significance of this result is limited. In general, neither gender nor age had much of an effect on inter-subjective agreement. Overall, the low inter-subjective agreement could account for all the vehement disagreements between people on whether or not a given movie was good: on average, one could expect a 1.25-star difference in disagreement on a scale from 0 to 4 stars.

Turning to correlations with movie critics, the connection between the ratings of critics and any given participant was no better than the average correlation between participants. Even a critic as well regarded as the late Roger Ebert did no better in predicting how well someone would like a movie than a randomly picked participant in the sample. In contrast, critics agreed with each other relatively strongly. In fact, the best predictor of a critic's response to a film was that of other critics while the best predictor of a non-critics' response were the aggregated evaluations of other non-critics such as those on the Internet Movie Database (IMDB) -- but not the aggregated ratings of critics such as Rotten Tomatoes. So it is not the aggregation of ratings per se that improves predictability, but aggregation of non-critic ratings.

"Something about being a critic seems to make the recommendations of critics unsuitable for predicting the movie taste of regular people," the authors conclude. "This study is the first to quantify this in an adequately powered fashion, and it helps to explain why people often perceive critics to be out of touch.

"There are some people in our sample who are 'superpredictors' -- they perform as well as the best aggregated non-critic ratings when it comes to predicting average non-critics will like. Short of these exceptional predictors, if someone seeks recommendations about what to see, their best bet is to either consult sites that aggregate individual judgements, or to find other individuals or critics with similar tastes."