Regression to the mean, or why second albums suck

I'm currently reading Daniel Kahneman's Thinking, fast and slow.  It's excellent.  It's taking me a while to get through, because the book is long and has lots of great insights that take a while to digest.

He looks like he's thinking at a medium pace here.

One such insight is the oft-quoted but less understood "regression to the mean".  Kahneman gives the clearest explanation I've ever read.  Put simply, he says that many outcomes are a result of a combination of what we might call "skill" and "luck".  (If you don't like the word "luck", just substitute "randomness").

Because many outcomes are the result of skill + luck, when there is a particularly good outcome (say an investment manager's portfolio return in a year), it is likely that the next year will be less impressive.

For argument's sake, let's be generous to investment managers and say that their portfolio return for a single year is a result of 30% skill and 70% luck.  If they are a skilled manager, who gets lucky, they will have a very high return that year.  Let's say they were in the top 10% of luckiness that year.  The next year, they are still highly skilled (their skill transports over time), but the 70% luck outcome resets every year (of course, it resets every nanosecond, but for our purposes we can think of it resetting each year as that is our measurement period).

When the random number generator called life rolls the investment manager's portfolio dice, the probability that they repeat their good luck or receives better luck is 10%.  The probability that their luck "reverts to the mean" is whatever's left - 90%.

God rolling the dice of life, and determining your investment results. 

That's why it's interesting for the purposes of investments (and of course it applies to individual stock picks, yearly results, and all sorts of decisions that need to be made).

But it occurred to me there's another area where it goes a long way to explain (and to my mind probably completely explains) a commonly debated and observed pattern - bands' second albums are almost always worse than their first.

The Strokes - it wasn't because Julian got a long term girlfriend, it was a statistical artifact called regression to the mean!

So the first thing you have to realise is that the only reason you have heard of a band's first album is because it was excellent, right?  There are a lot of bands making first albums in the world, and you hear a miniscule fraction of a tenth of a hundredth of one percent of those albums.  For that to happen, the band has to be skilled, but of course there are lots of skilled bands.  On top of that, the band has to get lucky - not a bit lucky, but massively lucky.  For whatever reason, be it coincidence of time and space, the music they make has to resonate with the public, and probably with you, for you to think the band is good and to worry about the band's second album.  Right time, right place, whatever - the band got a bit lucky.

So we know that luck played a large part in determining the outcome of their first album (this does not undermine the skill/taste/coolness of the band, it just notes that there is an aspect of luck to their success).  When luck is reset for their second album, the chances of them repeating their outstanding skill + luck combo are tiny!  Of course it's going to be a worse album!  The only reason you heard of them in the first place was their crazy luck to make it out of the swamp of average debut albums.

So next time someone at a party starts talking about bands with disappointing second albums, impress them by talking about regression to the mean.  If that doesn't impress them, really wow them by bringing up this blog post on your iphone or android device!