Challenges in Randomized Controlled Trials

[math] \newcommand{\indep}[0]{\ensuremath{\perp\!\!\!\perp}} \newcommand{\dpartial}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\abs}[1]{\left| #1 \right|} \newcommand\autoop{\left(} \newcommand\autocp{\right)} \newcommand\autoob{\left[} \newcommand\autocb{\right]} \newcommand{\vecbr}[1]{\langle #1 \rangle} \newcommand{\ui}{\hat{\imath}} \newcommand{\uj}{\hat{\jmath}} \newcommand{\uk}{\hat{k}} \newcommand{\V}{\vec{V}} \newcommand{\half}[1]{\frac{#1}{2}} \newcommand{\recip}[1]{\frac{1}{#1}} \newcommand{\invsqrt}[1]{\recip{\sqrt{#1}}} \newcommand{\halfpi}{\half{\pi}} \newcommand{\windbar}[2]{\Big|_{#1}^{#2}} \newcommand{\rightinfwindbar}[0]{\Big|_{0}^\infty} \newcommand{\leftinfwindbar}[0]{\Big|_{-\infty}^0} \newcommand{\state}[1]{\large\protect\textcircled{\textbf{\small#1}}} \newcommand{\shrule}{\\ \centerline{\rule{13cm}{0.4pt}}} \newcommand{\tbra}[1]{$\bra{#1}$} \newcommand{\tket}[1]{$\ket{#1}$} \newcommand{\tbraket}[2]{$\braket{1}{2}$} \newcommand{\infint}[0]{\int_{-\infty}^{\infty}} \newcommand{\rightinfint}[0]{\int_0^\infty} \newcommand{\leftinfint}[0]{\int_{-\infty}^0} \newcommand{\wavefuncint}[1]{\infint|#1|^2} \newcommand{\ham}[0]{\hat{H}} \newcommand{\mathds}{\mathbb}[/math]

A major issue with randomized controlled trials (RCT) is that we must experiment with subjects. This raises many issues that are not necessarily related to causal inference itself but are more broadly about ethics and legality. For instance, the “Tuskegee Study of Untreated Syphilis in the Negro Male” was the widely-known and widely-condemned study for investigating the effect of untreated syphilis~. As RCT requires careful, double blinding, the trial administrators did not reveal to the study participants that they were diagnosed with (latent) syphilis. The study was originally designed (and the participants were told) to run for six months but lasted for 40 years until the details of the study were leaked to the press. During these decades, the treatment for syphilis was made available but none of the participants were treated properly, resulting in the death of more than 100 participants due to syphilis, out of approximately 400 participants, the syphilis infection of the wives of fourty participants and the congenital syphilis infection of 19 children. It took more than half a century for the US government to formally issue apology.

A similar issue persists throughout medicine when it comes to RCT which is de facto standard for establishing any causal effect of a treatment on the outcome of a patient. Due to the necessity of randomization, some patient participants will inevitably receive placebo rather than the actual treatment. Even if the tested treatment ultimately turns out to be causally effective, by then it may be already late for those patients who were put on the control arm to receive and benefit from this new treatment. How ready are you to put patients into suffering because we want to (and often need to) establish the causal effect of a new treatment?

Sometimes, it is impossible to design a placebo that ensures double blindness of a trial. Consider for instance an RCT on the effectiveness of masking on preventing respiratory diseases. Participants will understandably alter their behaviours based on their assignments; treatment (masking) or control (no masking), as their perception of risk is altered, which violates the stationarity of the causal effect [math]p^*(y|a,x)[/math]. In order to avoid this, we must ensure that participants cannot tell whether they are in the treatment or control arm, but it is pretty much impossible to design a placebo mask that looks and feels the same as an actual mask but does not filter any particle in the air. In other words, RCT is only possible when placebos can be effectively designed and deployed.

In this example, we run into yet another problem; how do we enforce the treatment on subjects? In the case of vaccination, subjects come into clinics and are for instance injected on the spot under the supervision of a clinician, after which the subjects cannot get rid of injected vaccine. In the case of masking, for instance, we cannot ensure that participants wear masks as they are instructed, as this requires non-stop monitoring throughout the trial period. Finally, some actions take long to have measurable impact on the outcome. For instance, consider a policy proposal of introducing a new course on programming at elementary schools (1-6 grades) with the goal of improving students' job prospects and growing the information technology (IT) sector. It will take anywhere between 12 to 20 years for these students to finish their education and participate in society, and we will have to wait another 4 to 15 years to see any measurable economic impact on the IT sector. Such a long duration between the action and the outcome further complicates RCT, as it is often impossible to ensure the stationarity of underlying distributions over that duration. RCT is thus not suitable for such actions that require a significant amount of time to have any measurable impact.

In this chapter, we instead consider an alternative approach to RCT, where we rely on existing data to infer the causal relationship between the action and outcome. As we use already collected data, we can often avoid the issues arising from actively experimentation, although we are now faced with another set of challenges, such as the existence of spurious correlations arising from various unobserved confounders that affected the choice of actions earlier. We will discuss how we can avoid these issues in this chapter. It is however important to emphasize that there is no silver bullet in causal inference.

General references

Cho, Kyunghyun (2024). "A Brief Introduction to Causal Inference in Machine Learning". arXiv:2405.08793 [cs.LG].