Caveat Emptor, Analysts
February 18th, 2008I enjoyed this Kotaku piece, partially because it’s a vindication of what I’ve previously written. Basically, it analyzes 10 predictions from 10 analysts and finds that even the best analysts are correct in their predictions only 60% of the time. I’m not sure yet what the predictions were about, and 10 predictions is hardly a decent sample size, but it should be concerning that analysts are essentially no better than raw chance at making estimates about the future… since it’s what they get paid to spend all their time doing. Amusingly, the worst analysts - e.g., Billy Pidgeon - are so wrong that you’d be better off assuming the opposite of what they tell you.
Not that I really pay attention to them in the first place… but caveat emptor.
Posted in Business, Geoff, Idiocy |
February 19th, 2008 at 12:05 am
I share your low opinion of analysts, but I hesitate to be quite so harsh on them.
First, I posit that analysts are necessary. I agree with Pachter that they’re essentially required to convince a generally ignorant investing public to finance game development, so if we want investment and thus quality games (and in reasonable quantity), analysts cease to be optional.
Now, Divnich says, all you need is an understanding of financial markets and knowledge of the game industry. I’d add strong reasoning ability to that list. That may seem like a low bar to set, but really, think about any gaming forum you’ve ever read. How many times have you seen someone claim that a company’s sales problems are easy to solve — they should just lower their prices, and then they’d sell a ton of units?
Analysts may be incredibly bad at their jobs, but let’s face it: the general public would be even worse. You need some way to guide investors who can’t or won’t do the research themselves, and analysts are about the only way to do it.
What annoys me are predictions pulled out of thin air. I’m still waiting to hear what Pachter based his “new DS redesign” prediction on.
February 19th, 2008 at 1:27 am
“since it’s what they get paid to spend all their time doing. ”
Except that its really not that simple. They don’t get paid to predict if a console will sell 10 million or 5 million. They don’t get paid to predict which console will even sell the most or when there will be a price drop. They get paid to advise clients to buy, sell, hold, etc. They get paid to predict gains and losses in revenue and the sales prediction is more a “side business” that they do to entertain the media. This analysis of the analysts is little more than fanboy service for those who love to hate analysts. I would say that at least 90 percent of the erroneous “predictions” we as gamers get “upset” about occur outside the purview of the jobs they ACTUALLY get paid to do. Sure, some of the prediction probably factor into the advice they give clients, but that doesn’t mean much. If they predict a price drop as part of an increased revenue stream leading to a “buy” rating and the price drop doesn’t happen, its only meaninful if the revenue increase also doesn’t happen. And since no sane investment adviser would base that kind of advice on that type of singular prediction, odds are good, it wouldn’t effect their job performance anyway.
February 20th, 2008 at 12:05 am
Cisco, I’m not sure I see much of a distinction. Presumably, analysts’ forecasts of a firm’s profitability (and thus the value of its stock) are based on some sort of revenue and cost model. In turn, those forecasts are based on estimates of what they think the company is going to be able to sell (e.g., a revenue forecast at its most basic is a prediction of the amount of product that a company and the price point(s) at which it is going to sell them).
The GIGO principle applies: if you can’t accurately determine what those inputs are, you can’t decide what the revenues and costs are either, and thus can’t determine what the company is worth. Which is undeniably the job of an analyst.
I agree that some of these predictions are sideshows to the main issue at hand. But to the extent that they indicate that analysts don’t really know what inputs they should be using, they’re definitely relevant.
February 20th, 2008 at 1:31 am
@Geoff.
But there IS a distinction. A big one. And they DO know what inputs they should be using. It just so happens that those inputs are not relevant to gaming media interviews and gamer blog circle jerks because they are boring.
Console sales numbers (particularly weekly and monthly numbers), price drops, new SKUs, now, those are interesting to us on a level that an investor would NEVER care about. And those are the very predictions we are holding up as an example of their job performance.
The point is that the actual clients of these analysts don’t pay attention to which console they say will sell the most in an interview at IGN, or that they changed their prediction 3 weeks later in an interview with 1up. In fact, I would guess that most of their client don’t even know they make such statements in many cases. Their clients are not reading their interviews with gaming media to make stock buying decisions. Yet those interviews are the only thing gamers are concerned with. Their clients on the other hand pay attention to the overall guidance for a company/sector/industry. If they get an input or two wrong, only the gamers examining the minutiae of their predictions will even notice. As long as the overall guidance is pretty accurate, they are doing a good enough job. The analysis over at Kotaku doesn’t reflect on that level of accuracy and is not relevant to making a decision as to their success “on the job”. And believe me, if their guidance was as poor as the Kotaku report implies, they would long be out of a job, simply because all their clients would go elsewhere.n And this doesn’t even factor into it, that when a guidance is not met, the company is often blamed as much as the analyst. WHY didn’t it meet the guidance? The presumption is often that the company has failed to meet expectations, rather than the idea that the expectations were flawed to be begin with.
I’ll admit, its fun to use them as whipping boys, but the depth of the Kotaku analysis and the fact that people put stock into it, as reflecting on the actual jobs of these individuals, only makes gamers look like simpletons. These are just regular people like you and me who get out of bed every morning and go to work to make a living. I’m certain they take it seriously when they are behind a desk writing a report that will influence large amounts of money, even if they don’t appear to do so from the perspective of a hardcore gamer.
In the end, it doesn’t matter if they say the moon is made of green cheese, as long as they get the big picture right. The big picture also happens to be exactly what we as gamers have trouble seeing, because we’re too close to the subject matter.