Naoki Watanabe: Experimental Economics is Not Yet a Science

2022/03/25

I earned my PhD in game theory, a fundamental theory of microeconomics, and I still consider theory to be my primary focus. However, in recent years, I have been writing papers based on experiments. This is because, in the practical application phase of systems designed based on game theory, it has become commonplace to conduct subject experiments in "some form" and present the results of their performance evaluation. However, when asked for the name of the field, I choose to write "economic experiments" in the sense of subject experiments related to the economy, separating the "science" from "experimental economics." This is because the results are often lacking sufficient evidence to be called scientific knowledge.

Over the last two years, the clinical trial process for COVID-19 vaccines has been introduced in detail by various media outlets, and I believe many people have gained a broader understanding of how experimental results are evaluated scientifically or statistically. It is a given that experiments should be planned to minimize bias in subject attributes and conducted with specifications that can withstand statistical scrutiny (internal validity). However, for many economic experiments, there is no guarantee that the results detected will be found under the same conditions at another site (external validity). This is because academic journals tend to highly value and accept proposals for new methods that improve the precision of internal validity rather than examining the reproducibility of results.

Such evaluations by leaders in economic experiments are certainly important for asserting that experimental economics is a field independent of theory. Since there are many theories awaiting experimental verification, the secondary justification—that conducting experiments in "some form" (guaranteeing internal validity) and showing the results is better than not doing so at all—is a reasonable perspective if experimental economics is still only at the stage of solidifying its scientific foundation. However, many academic journals in economics reject even papers that question whether the same results can be observed under slightly different conditions (robustness), based on the aforementioned evaluation criteria.

One reason for this current situation is the lack of quantitative evaluation of systems that have begun operation based on experimental results. It has been less than 20 years since various systems designed based on game theory began to be applied in practice, but operational data for them has been accumulating worldwide. As the analysis and sharing of these results progress, the day when economic experiments begin to focus on guaranteeing external validity should be near.