by guest bloggers Mike Joel, Quantitative Specialist, London, and Matthew Van Der Weide, Quantitative Specialist, Amsterdam

Now that the craze of the World Cup is over, vuvuzelas have silenced, and Paul the Octopus has retired from making predictions, it seems like the time to evaluate. Could it have been luck, or was Paul the Octopus really skilled, and, if so, where would that rank him against some of the greatest fund managers?

The first argument of sceptics is that there are potentially thousands, if not millions, of animals with assumed predictive skills and only those who predict correctly come to the surface. That does sound like a plausible story; had Paul been wrong in the group phase, surely we would not have heard as much of our squishy oracle. In a way this reminds us of the (e)mail scams that recommend a stock every week, building up a track record week after week, trying to persuade you to invest. Although these stories sound compelling if there are a number of "correct" predictions in a row, it cannot be ignored that these frauds simply stop writing the people for whom they made a wrong prediction. Something that Nassim Taleb has shown all to clear in his book Fooled by Randomness: in the end this is simply a numbers game. Eight out of eight predictions right and given 50/50 odds (we will address both assumptions later), that would mean that there is a 1 in 256 chance for a random animal in a German zoo becomes the next Oracle.

But that is not the complete story, Paul actually has a bit of a track record. Although it is doubtful that it improves his statistics, at least it shows he has been around and we can conduct some analysis. Back in 2008, he made predictions for the Euro 2008 Championship and got four out of six right then. Still assuming 50/50 odds, would you have invested with a manager that has Paul’s track record? A simple coin flip would have given you three out of six (assuming 50/50 odds, mutual exclusivity of events, and the caveat that you might need to quit your job to be able to repeat the experiment enough to obtain stable results).

So when we look at the total track record of the eight-legged oracle, he predicted 12 out of 14 matches correctly (again, of these mutually exclusive events!). Through the use of our friend, the factorial, we can calculate the probability of achieving this:

 where x = number of occurrences and y = number of wins

Which ultimately translates to 91/(2^14)=0.0055, or 0.55% probability.

So what about those odds? As we pointed out earlier, they are assumed to be 50/50. Ever wondered why sometimes the bets at a bookmaker do not seem to add up? Effectively for the group phase there is a one in three chance; teams can draw and bookies can make money. Hence, given the extra choice, the odds of making the right prediction on 12 out of 14 matches plummets to 0.0019%, which is still slightly better than getting five out of six numbers (not including the bonus number) correct in the National Lottery (~0.0018%)= £1,500. Although when placing a bet with a bookie, one (usually) has detailed knowledge of team, hence there is more than luck in play and the probabilities are probably biased.

So how does Paul hold up against a good or even average fund manager? Grinold and Kahn define the Information Ratio (IR) as the Information Coefficient (IC) times the Square Root of Opportunity 
(). If the IC is our measure of skill, we will assign this the probability of Paul correctly predicting 12 out of the 14 matches (for lack of a better measure). This gives us an IR of 12/14 * Sqrt(14) = 3.2, which is remarkable, although given the light track record, not reliable. To maintain the same IR going forward, Paul does not have to maintain his skill, but merely place more bets!

Now to compare him to a fund manager, how can we describe Paul the Octopus’ investment style? He is not your typical long-only equity investor; his style more resembles that of a hedge fund: long team A, short team B (given that one of the co-authors is Dutch we will not disclose the teams). If one has to put a label on what the World Cup is, "Event Driven" probably fits best. Looking at the information ratio over the last 14 months, for the Lipper Tass Event Driven hedge fund universe, where would that put Paul the Octopus? His performance is not too bad as he ranks comfortably amongst the best names in the top quartile.

This brings us back to the main question: is Paul’s ability the result of skill or luck (putting aside any bias that may influence the numbers, such as shape or colour of the flag, an inclination to favour Germany, etc)? Given Paul’s predictions and where he falls relative to other funds of his style, he is a highly skilled manager. But whether this is true skill, whether there is a correlation between the winning team and characteristics that attract Paul or whether this is pure luck, the jury is still out; there are just not enough data points to quantify skill vs. luck. Perhaps Paul’s situation is a case of survivorship bias to the extreme (also quite literal), but one thing is certain: the next time one peruses a marketing brochure, it may be worth contemplating Paul and his ability. Any number of calculations/statistics will make Paul look like one of best managers, and although the numbers may look good, one must delve into the details behind the numbers. The risks for managers involved in pursuing the stats and making it to a top quartile ranking, though, are minimal compared to being part of our next dish.

Subscribe by e-mail to receive new Taking Risk posts as they are published and follow @FactSet on Twitter.