This Time is Different vs Excel

Summary:

– Reinhart and Rogoff’s finding that countries with a debt-to-GDP ratio above 90% experience a sharp slowdown in growth provided academic support for arguments in favor of fiscal austerity.

– An attempt to replicate the results by Hendon, Ash, and Pollin (U. Mass.) showed that this threshold is the result of errors in methodology and calculation.

– However, the influence of the 90% threshold on policy design should not be overestimated. The other major aspects of the R&R study remain valid, and the incident should draw attention to the risks associated with using inappropriate tools (in this case, a spreadsheet) for critical tasks involving the production of figures.

Source: Hendon, Ash, and Pollin (2013)

Since the publication of a paper by the Roosevelt Institute [1], social media and the economic blogosphere have been in turmoil: a finding by Reinhart and Rogoff that countries’ growth is penalized when their debt-to-GDP ratio exceeds 90% appears to have been caused by both questionable methodological choices and an error in a spreadsheet formula [2].

Much of the debate has focused on this 90% threshold, which had been a major topic of public debate and, according to many commentators, provided academic support for austerity policies. However, this is not the main lesson to be learned from this episode. Their study contains much more interesting insights into the process leading up to banking crises and the financial repression that follows. The episode itself is instructive in terms of how public and academic debate works, and has the merit of drawing attention to the risks associated with using spreadsheets for calculations where reliability is important.

This Time is Different: The decisive role of credit bubbles and the analysis of financial repression

There is undoubtedly no reason to overlook the importance of This Time is Different on the basis of this error alone. Determining a debt threshold above which debt becomes an obstacle to growth is not the main contribution of this book. It is probably its weakest and least interesting part. Reinhart and Rogoff’s great contribution is their in-depth historical analysis of banking crises.

This has highlighted the decisive role of credit bubbles, particularly real estate bubbles, in triggering crises. Above all, and this is what gave the book its title, they show very effectively how the formation of these bubbles is fundamentally based on biased expectations, on the renewed belief that this time it is different, and that a given innovation (the Internet and securitization being the two latest examples) profoundly changes the functioning of the economy by rapidly increasing growth prospects.

Another important aspect of their work is the section on financial repression. Just as much as debt-to-GDP ratios, this toolbox and its use are likely to shape the functioning of the European, American, and Japanese economies for the next ten years. It seems that this aspect of the book has not received the attention it deserves.

This Time is Not Different: The fragility of the 90% threshold was known and documented by the authors themselves.

Furthermore, we should probably not exaggerate the real significance of this 90% threshold. It has certainly served as a reference in certain debates, but any other similar threshold, calculated using a similar method, would probably have served as academic justification. We did not wait for this work to be completed before setting a debt-to-GDP threshold in the Maastricht Treaty (60% of GDP, for the record).

The violent reaction from academic circles can probably be explained less by the effect of policies using this 90% threshold (which is not clear, as debt sustainability analyses seem more complex) than by a feeling of betrayal. Reinhart and Rogoff’s work was received as a genuine academic contribution, in which the authors had set aside their opinions to exploit a very rich (and particularly difficult to construct) data set. Learning that their results are linked to methodological biases and an error in a formula is therefore disappointing, and all the more so because their reaction does not seem to be up to the task.

That said, we must also keep our reactions in perspective. The fact that this threshold was calculated on the basis of relatively few data points, and was therefore fragile, was a known factor, documented by the authors themselves. Even before the errors in the calculation were revealed, there was ample reason to qualify the significance of this result, which Paul Krugman did not hesitate to do, with good arguments [3].

Excel, an operational risk factor

One point that I think is important to emphasize is the part related to the error in a formula. The mistake is minor: in an average, one of the researchers did not pull the selection frame low enough, thus leaving out some of the data. Yes, it happens to everyone. No, it should not happen, either in academic work or in industry or finance.

The problem here lies in the choice of tool. Yes, a spreadsheet is a fantastic tool for doing back-of-the-envelope calculations and getting a rough idea of what’s in the data. No, a spreadsheet should not be used in production. The fundamental flaw of the spreadsheet is that it combines data, formulas, and results in a single interface. This makes it easy to edit data without realizing it (one of the no-nos of statistics: never change data without documenting it), replace a formula with a value, or edit a result.

The latter error is all the easier to make because, in most cases, the formulas are not visible, hidden by their results. A copy-paste value, and a formula is replaced by a fixed result. A faulty formula can remain hidden for a long time (as happened with the Thames Whale [4]) or, in the case of Reinhart and Rogoff, only affect some of the results.

However, let those who have never made this type of error cast the first stone. This shows how common these errors are, and how most of them go unnoticed despite their significant consequences. Omnipresent in industry and finance, spreadsheets are probably one of the major factors of operational risk—perhaps even more significant than high-frequency trading algorithms.

Notes: