THE PERILS OF NATURAL EXPERIMENTS AND RANDOMIZED CONTROLLED TRIALS: DR. VIVEK MOORTHY
– Prof. Vivek Moorthy
The RCTs that have swept economics and won Nobel recognition remain rather unreliable in comparison with clinical trials
My previous article of 19 October, ‘Better ways to help workers than minimum wages’ (bit.ly/3nr0vnD), mentioned that conclusions from a 1994 paper of Card and Krueger that employment rose in New Jersey after its minimum wage was hiked, compared to adjacent Pennsylvania where it was not, were questionable. This article adumbrates the data problems with their paper, and, more so, with basing economic policies on such research.
To begin with, natural experiments can provide robust conclusions, provided the data-sets are accurate. The move in economic policy research, away from estimating regression equations derived from prior hypothesized models, to testing based on natural experiments is welcome. Specifically, the randomized control trial of Angrist and Imbens, using public records of birth dates, conclusively established the link between years of schooling and subsequent earnings. The Nobel Committee justifiably praised their astute way of forming a control group from birth dates that happened to fall in the fourth quarter of the year.
However, we should be sceptical about the policy conclusions of randomized control trials used in economics and social sciences. These are a far cry from the randomized clinical trials that have revolutionized medicine and led to many life-saving treatments, drugs and vaccines. The human body, the object of clinical trials, is a very complex but generally closed system. The impact of a trial drug on one person is independent of its impact on others. The results of a randomized clinical trial for, say, a vaccine, after it has gone through the quality checks of three phases, can be safely applied to the whole population. Not so for randomized controlled trials (RCTs). As economic agents, individuals are part of a complex and open system, their decisions and interactions impinging upon one another. Thus, at the economy-wide or macroeconomic level, the fallacy of composition may hold. This fallacy was a cornerstone of Keynes’ General Theory. If wages are cut for workers in one sector, employment is likely to go up; but an economy wide synchronized money wage cut will lead to a drop in wage income and so also nominal demand, with a possible fall in employment.
On the cost or supply side, similar considerations may apply. A rise in the minimum wage in one district or industry may not have an adverse impact upon employment and costs and prices, since firms invariably have some margin of profits to absorb the hike. But an economy-wide minimum wage rise may push up prices, unless the central bank tightens policy to offset that.
These fallacy-of-composition considerations even apply in the case of RCTs pertaining to education. A certain subsidy leading to more education for some individuals may benefit them, but nationally, it may not. Specific skills apart, often relative education matters a lot for getting a better job and higher wages. Thus, when everyone is better educated thanks to schooling subsidies, studying hard for years to get a degree may not be fruitful, ignoring its non-pecuniary benefits.
The second vital and delicate matter is the accuracy of the data that researchers generate by conducting surveys to conduct their RCTs—as distinct from using publicly and routinely available data from national agencies and other reputed organizations. In their pioneering 1994 study, Card and Krueger generated their own data from phone interviews. Given ideological biases, researcher- generated survey data is intrinsically suspect.
Neumark and Wascher, in their 2000 rejoinder to Card and Krueger, exposed these data problems. Using a payroll data-set from Employment Policies Institute in the US, they found substantial negative effects. However, this Institute being a think-tank funded by companies which were campaigning against the minimum wage, their results in turn were questionable. In response, Card and Krueger then used payroll data from the ES-202 form of the US Bureau of Labor Statistics. Accordingly, their conclusions regarding the rise in employment were considerably watered down (“small and statistically insignificant”, Card and Krueger, AER, December 2000). The Nobel citation’s silence on this matter reminds one of a great line from a Simon & Garfunkel song, The Boxer: Still a man hears what he wants to hear, and disregards the rest.
The academic debate continues. The first question to ask is: Of all the studies using RCTs published in the top 10 economics journals in the past two decades, how many generated their own data versus used publicly-available data? I have no idea, but it would broadly indicate the reliability of non-medical RCTs that have swept the field.
Professor Card is an exceptionally skilled and imaginative researcher. One of the most intriguing data puzzles that I have come across was posed in his co-authored 1986 paper on the sudden post-1981 US-Canada unemployment gap (see chart). Based on a wide-ranging investigation of many potential causes, including unemployment insurance systems, the authors concluded that more generous benefits could not explain the gap.
However, data on the unemployed by demographic groups and by job leavers versus job losers, clearly revealed far more job leavers in Canada. Further, total benefits paid out doubled in Canada from 2% to about 4% of wage income in 1972, following its 1971 Unemployment Insurance Act, an economy-wide legal event (details in my FRBNY 1989-90 article; see economicsperiscope.com).
Limitations of economics RCTs apart, research is badly skewed toward using sophisticated techniques, leading to flimsy and controversial results that generate flurries of further papers. It is far more useful to uncover crucial and robust stylized facts that can hopefully be understood by voters to make choices and used as inputs by policymakers.