Art by the Numbers is Mediocrity

I apologize ahead of time if you have read this essay before. Portions were previously published on David Poland's The Hot Button page. I was hoping to finish reading The Math Gene by Keith Devlin and write a review as Part 3 of my What is Mathematics? series. Unfortunately, other things got in the way of me finishing it, let alone writing a comprehensive review. So, to avoid going too long without a new monthly column, I present a case of an industry, specifically the movie industry, which cares more about numbers than art. While some have described mathematics as an art form, no one would ever describe art as a branch of mathematics.

---

I also gave each movie a Critical score, which is basically the "cream of the crop" percentage from Rotten Tomatoes divided by ten. Rotten Tomatoes has designated their big market and distinguished critics as "cream of the crop" critics. The advantage is that the percentage of critics that like or dislike a movie is thus given to the same group of about 15 critics. Otherwise, each movie would have a critical score based on different groups of critics, which would throw in an unnecessary chance factor. 

So by the numbers, here is how the movie blockbusters of 2001 performed with the critics and American audiences:

 

Title	Popularity	Critics
Shrek	10.0	8.6
The Fast and the Furious	7.4	6.7
The Mummy Returns	9.4	3.3
Pearl Harbor	9.3	2.5
Baby Boy	3.2	8.6
A.I.	4.4	7.0
Planet of the Apes	8.1	3.5
Moulin Rouge	3.5	7.1
Jurassic Park 3	6.6	4.0
The Score	5.1	5.3
A Knights Tale	3.6	5.7
Dr Doolittle 2	5.1	4.1
Legally Blonde	4.3	4.7
Atlantis: The Lost Empire	4.4	4.4
Kiss of the Dragon	2.8	5.7
crazy/beautiful	2.2	6.0
Tomb Raider	6.1	1.5
America's Sweethearts	5.2	2.2
The Animal	3.6	3.8
Swordfish	4.5	2.5
Cats and Dogs	4.0	2.7
Angel Eyes	1.6	4.3
Evolution	2.5	2.8
Scary Movie 2	3.0	0.8
Final Fantasy	1.3	2.2
The Worst That Could Happen?	2.1	0.5

The movies are listed in order of their combined scores. To make patterns easier to see, high scores are highlighted in green, low scores in red.

The interesting thing is when a statistical correlation is calculated the result is only 13%. Remember that statistical correlation  measures how one affect the other. So movie critics only have a 13% effect on how well a movie will do.

That is a very low number statistically speaking. And, this percentage is due to movies on the low end of both scales that were universally panned and rejected by audiences (those eight movies on the bottom highlighted red). In fact only two movies, Shrek and The Fast and the Furious could qualify as both popular and critically acclaimed. Note, I am only including movies that were released universally, art house and foreign films would no doubt demonstrate a negative correlation (since critics love art house movies, and audiences are generally small) 

So, while not scientific, this snapshot poll demonstrates that critical acclaim has almost nothing to do with a movie's popularity. Statistically, movie critics are out of step with the viewing public. Surprise! Surprise!  

I was thinking of making a big deal about this until I read a column by David Poland on July 19th relating to this very issue about critics being out of touch. I think it said it all: "Criticism that is reduced to a statistic is worthless, with due respect to Rotten Tomatoes.  The job of a critic, it seems to me, is to add some serious thought to the conversation of films."  

That in a nutshell is the difference between us movie buffs, and the general population. Most see movies as a form of entertainment, thus the general public goes to the movies to be entertained. We movie buffs, and all movie critics fit into this category, understand that movies are the dominant medium of artistic statement. Entertainment value is just an added bonus.  

Criticism has little to do with box office success, except where it provides pull quotes for advertisers (and when critics don't create good pull quotes, studios like Sony can just invent them). Marketing is everything, and we movie buffs know it. I knew I would not like Pearl Harbor before I saw it, so why did I? So I could join my movie snob friends in conversations about how bad it was. To me, the real joy of sites like Rotten Tomatoes is to read the full length reviews of movies you have just seen, especially the ones that disagree with my view.  

The question worth asking is not "Are critics out of touch?", it should be "What are critics for?"

***

It is unfortunate that the industries of Movie and Television are more interested in how much money they can make than in whether or not their product is any good. Is this news to anyone? 

Part of the problem is that there is no way to quantify whether art is good or bad. We either like it or we don't, and everyone's opinions are going to differ in this regard. Hence industries that create art are forced to rely on what is popular or what trends are showing. If this also means that they make more money in the process, then the better for them.

Of course, creating art based on what is popular could backfire. Last year, a few "banal comedies" American Pie, Road Trip and Scary Movie came out and were very popular, and the fact that these movies had no major stars meant that they did not cost much to make. So studios rushed in to production a ton of similar movies: Dude, Where's my Car?, Sugar and Spice, Double Take, Saving Silverman, Head Over Heels, Monkeybone, See Spot Run, Tomcats, Say It Isn't So, The Adventures of Joe Dirt, The Animal, What's the Worst That Could Happen?, Just Visiting, Freddy Got Fingered, Scary Movie 2, and American Pie 2. None of these were very popular, and most lost money (a few made money due to the fact that they were so cheap to make). Movie audiences tastes change that quickly, and thank you America for stopping an ugly trend in its tracks!

The entertainment industry is controlled almost entirely by seven companies: AOL Time Warner, Disney, Newscorp (Fox), GE (NBC), Viacom (Paramount), Vivendi (Universal) and Sony (Columbia). Almost every major movie studio and TV broadcast and cable network is a subsidiary of one of these seven. All of them are more interested in making money than creating good art, and all of them are very aware of the statistics I presented above. For a movie to make money, it does not have to be good, only marketed well. Why they can not create good movies and programs and advertise them well, is a rhetorical question I am not sure I want answered.

That is why critics are necessary, they are the ones that can sift through the chaff generated by these seven mega corporations a find the occasional grain of wheat.

Art by the numbers is mediocrity, and you can quote me on that.

I am not the first one to say that profit margin has ruined the movie industry, the recently deceased Pauline Kael (1919-2001) wrote a famous essay back in 1980 called "Why Are Movies So Bad? or, The Numbers" which you can read here, which is just as true today as it was then.

Update August 2002:

So how do this summer's movies stack up vs. critics? See for yourself:

Correlation between critics and popularity is 27% this year due to there being more good "critically acclaimed" movies this year. I guess this just proves movie watchers will go see whatever is playing. One slight problem is that the popularity scores of Spider-Man and Star Wars Episode II are way out of proportion to the rest of the field. Eliminating those two raises the correlation to 36%

This year is decidedly different. This years crop of movies are much better than last years, and it shows in the critic ratings. With more good films to choose from, audiences are actually going to the good films and ignoring the bad ones. Hence a correlation score of 27%. Much higher than last years 13%. 

Obviously, there is luck involved. These numbers are not scientific. Only three movies were widely praised by critics, but ignored by viewers (Insomnia, Stuart Little 2, and K-19), and only two widely popular film were panned by critics (Scooby Doo, Star Wars Ep. II). That is only five disagreements out of 33 films, which is not typical.

As a final update, I got this nice response from Paul Lee one of the founders of Rotten Tomatoes:

Hi Paul,

I read your article about math and movies on your site and found it to be a great read. Even though I'm one of the Rotten Tomatoes' co-founders, I do agree with you in your statement that "art by numbers is mediocrity." All of us here also feel that RT's greatest value is in helping to continually answer the question, "What are critics for?"

That being said, I do feel that David Poland somehow missed the mark in his Hot Button piece about RT. Yes, the Tomatometer and the whole rating thing is a simplified approach to rating movies, but we feel it is merely a starting point for some great discussions and learning. The past several years have supported that belief. August was our best month ever in terms of traffic despite the lackluster box office numbers (2.73 million unique visitors and almost 19 million page views). Furthermore, I believe that our message boards are, by far, the most active movie boards on the Net - we average some 30,000 new posts each week! Without meaning to be conceited, no other movie site comes close in terms of the quantity and quality of discussions.

One great example was that we once had the screenwriter from "Final Fantasy" come in and start a month long discussion with other users of the movie. He didn't like the fact that FF was basically trashed by the critics, but his input into the whole process of how it went from his script to the final theatrical release was very enlightening. Even users who didn't enjoy "Final Fantasy" developed a new appreciation of the film, but the Tomatometer often served as a starting point for such discussions.

Clearly, there are many, many people out there who spend a lot of time on the RT boards discussing movies, videos, and the other topics under the sun. IMDB is the largest movie site bar none (they are roughly five times larger than we are) but it's rare to see in-depth discussions occurring on their boards. A "good" thread on IMDB will get some 30 replies. A good thread on RT gets 100+ replies and we have many of those every day. The best threads will get hundreds or even thousands of replies. And it all ads up to 30,000 new message each and every week.

I'm constantly amazed by the sense of community that has developed around RT. In fact, users will - of their own accord - arrange "RT Get-Togethers" on the Off-Topic Discussion boards so they can meet each other in real life. One user went to Seattle meet last month, and met a fellow user for the first time. He ended up proposing to her that weekend, and their wedding is now set for next year. It's the first "RT marriage" that we know of, but I find it fascinating as well as fantastic that people can connect to each other via a movie site. So clearly, there is something special about RT that users develop such an affinity to the site.

In fact, I would say that the critical community has gained more influence because of sites like RT. This is just conjecture on my part, but before RT, I would think studios had an easier time ignoring or manipulating critical opinion. They couldn't ignore big-name critics like the Roger Eberts and the Janet Maslins, but they could probably safely ignore the opinions of critics of small town, but wholly reputable papers. Now those small-town critics have as much visibility as Roger Ebert on the site.  We constantly field requests from critics for the right to get their reviews added to the RT database, and The Tomatometer is simply the glue that binds all those individual opinions together into one cohesive, and eminently quotable, object. 

Now, I think a truly interesting question is, is the low statistical correlation between critical opinion and box office performance dependent on how widespread the Tomatometer is? That is, if Rotten Tomatoes was a household name, and you had the Tomatometer printed up next to all the box office lists of every web site, newspaper, and magazine, would the statistical correlation be higher?

The reason I ask this is, in our experience, many users learn how to "trust" the Tomatometer after a period of using it. The Tomatometer becomes a bigger factor - if not the deciding factor - in determining whether they go see a movie or not. That's true for small, independent films that you would normally never hear about but noticed because of it 93% rating on RT or for intensely marketed media monsters like Pearl Harbor (but which you end up going to see precisely because it's so badly rated).

So even though the Cream of the Crop critics collectively reach a large audience, without a site like RT, their voices tend to be disparate and less effective in terms of swaying opinions. When one of your trusted friend from down the street says a movie is  a must-see, that's one thing. When a dozen of them living in different cities all call you up to say the same thing, it's quite another. It's probably the reason why politicians all live and die by public opinion polls. There's something to be sad about consolidated opinions.

Anyway, sorry for the long email, but I really enjoyed the analysis you did. I also think your mathmistakes website is great - I'm always depressed at the sorry state of math and science education in the U.S. Not only are we not teaching kids to appreciate and understand mathematics, but the atrocious quality of math textbooks seem to be teaching kids to actually hate math with a passion. Even worse, to consider math as irrelevant to their lives.

Thanks,

Paul Lee

Back to the Glossary
Back to Archives