Reinhart-Rogoff
(2010)
Five countries: Australia, Austria, Belgium, Canada, Denmark: were left out.
The corrected figure flipped sign.
The paper claimed countries with debt above 90% of GDP saw −0.1% median real growth. The corrected calculation: roughly +2.2%. In the three years between publication and discovery, the paper had been cited by the European Commissioner for Economic Affairs, the UK Chancellor, and the US House Budget Committee: all justifying austerity programs.
Sources · Herndon, Ash & Pollin (2013), “Does High Public Debt Consistently Stifle Economic Growth?” PERI Working Paper 322 · Reinhart & Rogoff (2010), “Growth in a Time of Debt,” NBER Working Paper 15639 · authors' subsequent corrections, NYT 2013-04-17.
The working paper computed, across twenty advanced economies and roughly two centuries of data, the relationship between sovereign debt levels and real GDP growth. The headline finding aggregated countries into four debt-to-GDP buckets and reported median and mean growth in each. A single Excel workbook held the country-year observations, the bucket assignments, and the bucketed averages.
In the worksheet that computed the high-debt-bucket average, the formula referenced rows L30:L44 : fifteen rows. The data block actually extended through row 49: twenty rows. Five countries that belonged in the high-debt bucket were silently excluded from the average. The omitted rows held growth observations for Australia, Austria, Belgium, Canada, and Denmark.
The bucketed average was the paper's headline number. The headline number became Figure 2. Figure 2 became a one-line summary in the abstract. The one-line summary entered the policy bloodstream: the European Commissioner cited the 90%-of-GDP threshold; the Chancellor of the Exchequer cited the negative-growth finding; the House Budget Committee cited it in its 2013 budget proposal. The number was the empirical anchor for austerity arguments across two continents.
Three years after publication, a graduate student at the University of Massachusetts, Thomas Herndon, requested the original spreadsheet for a replication assignment. He reproduced the calculation and found his numbers did not match. He emailed the authors. They replied with the workbook. The discrepancy was the range. Herndon, Ash, and Pollin published the correction.
The cost is contested and depends on what counter-factual one assigns to policy choices that would have been made differently. Estimates of the fiscal-multiplier effect of the austerity programs that cited the paper run from hundreds of billions of dollars in lost GDP to single-digit percentage points of recession depth in several European economies. We do not litigate the counter-factual here. The architectural point stands independent of it.
The architectural cause is not that the researchers were careless. The researchers were among the most careful in their field. The architectural cause is that Excel offered no structural way to express the invariant that the range must cover all of the rows that hold high-debt observations.There is no type system that would have known “this range should be a list of country-years with debt > 90%” and there is no assertion engine that would have checked “count of bucketed observations equals count of high-debt country-years in the source.” The substrate gave the operator no way to encode the invariant they were trying to enforce. The operator did everything right within the bounds of what the substrate allowed. The bounds were wrong.
=AVERAGE(L30:L44)[ should have been =AVERAGE(L30:L49) ]An assertion would have failed
at the moment of compile.
In Flatland, the bucket would be a typed Set(CountryYear) and the average would be a computed driver whose dependency on the bucket is declared by name, not by row range. A structural assertion: len(high_debt_bucket) == count(country_years where debt > 0.9) would have failed the compile the moment the high-debt observations were filtered. The model would not have produced a number. The headline finding would not have been generated.
The point is not that Reinhart and Rogoff would have caught the error with a different tool. The point is that the substrate they were working in offered no way to express the invariant they meant to enforce. A typed compilation substrate does.
Compile a model
whose assertions can fire.
Three questions. A typed compile. A real Excel export. The same six-sheet artifact Reinhart and Rogoff would have produced: but with assertions that can fail loudly when the math is wrong.