Rethinking historical controls (2001)(en)(14s)

Biostatistics (2001), 2, 4, pp. 383–396 Printed in Great Britain

Rethinking historical controls STUART G. BAKER∗ Biometry Research Group, Division of Cancer Prevention, National Cancer Institute, EPN 344, 6130 Executive Blvd MSC 7354, Bethesda, MD 20892-7354, USA [email protected] KAREN S. LINDEMAN Department of Anesthesiology, The Johns Hopkins Medical Institutions S UMMARY Inference from traditional historical controls, i.e. comparing a new treatment in a current series of patients with an old treatment in a previous series of patients, may be subject to a strong selection bias. To avoid this bias, Baker and Lindeman (1994) proposed the paired availability design. By applying this methodology to estimate the effect of epidural analgesia on the probability of Cesarean section, we made two important contributions with the current study. First, we generalized the methodology to include different types of availability and multiple time periods. Second, we investigated how well the paired availability design reduced selection bias by comparing results to those from a meta-analysis of randomized trials and a multivariate analysis of concurrent controls. The confidence interval from the paired availability approach differed considerably from that of the multivariate analysis of concurrent controls but was similar to that from the meta-analysis of randomized trials. Because we believe the multivariate analysis of concurrent controls omitted an important predictor and the meta-analysis of randomized trials was the gold standard for inference, we concluded that the paired availability design did, in fact, reduce selection bias. Keywords: All-or-none compliance; Concurrent controls; Meta-analysis; Propensity scores; Randomized trials.

1. I NTRODUCTION The gold standard for evaluating the effects of medical interventions is the randomized controlled trial. Unfortunately, in some situations, a satisfactory randomized trial is not feasible. Expanding on Byar (1980), some reasons include (i) enrollment of sufficient numbers of patients would take too long, (ii) the cost or required effort would be excessive, (iii) controversy exists over various ethical issues or (iv) the study could not be blinded, and an unblinded analysis would introduce bias. When a randomized trial cannot be implemented, general approaches for inference with comparative studies include (i) multivariate adjustments for concurrent controls and (ii) historical controls. Unfortunately, results from both approaches can be seriously biased. Multivariate adjustments can be biased if they fail to adjust for an important predictor (see, for example, Byar (1980)). Traditionally when making inference from historical controls, one compares the new treatment in a current series of patients with the old treatment in a previous series of patients, which can give biased results if subjects with certain risk factors related to outcome are more or less likely to receive the new treatment (see, for example, Pocock (1983)). ∗ To whom correspondence should be addressed

c Oxford University Press (2001) 

384

S. G. BAKER AND K. S. L INDEMAN

To strengthen inference from historical controls Baker and Lindeman (1994) proposed the paired availability design in the context of a proposed study of the effect of labor epidural analgesia on the probability of Cesarean section (CS). This is an important topic because (i) CS, especially during labor, increases maternal morbidity and mortality when compared with vaginal delivery (National Institute of Child Health and Human Development, 1981), (ii) epidural analgesia represents an effective form of pain relief not matched by other agents, and (iii) each year 1.6 million women in the United States receive epidural analgesia for pain relief during labor (Hawkins et al., 1997; Lieberman, 1999). Because of insufficient data to implement their analysis, Baker and Lindeman (1994) focused on design issues and sample size calculations for the number of required studies. More recently, many studies of the effect of epidural analgesia on CS have been published. These include studies which could be analyzed via the methods for the paired availability design, as well as randomized trials, and multivariate observational studies. Using data from these studies, we extended our previous work in two ways. First, we generalized the paired availability design to include two different types of availability and to accommodate data from multiple time periods. Second, we compared the results of the current paired availability analysis with those from a multivariate analysis of concurrent controls and from a meta-analysis of randomized trials. 2. PAIRED AVAILABILITY DESIGN The basic paired availability design has three characteristics: (i) a comparison of all eligible subjects at a later time with all eligible subjects at previous time using as an intervention the availability, not the receipt, of treatment at a hospital, (ii) a model to estimate the effect of receipt of treatment, and (iii) a meta-analysis to combine results from all the hospitals. In Section 5, we extend (i) to multiple time periods. In applying the paired availability design to study the effect of epidural analgesia on the probability of CS, we identified 11 hospitals or medical institutions which reported changes in availability of epidural analgesia (Table 1). Because the technique of epidural analgesia stabilized in the late 1970s, we only included studies with data after 1980. We first consider the case of two time periods as in the original formulation of Baker and Lindeman (1994). To put the analysis on firmer foundation, we consider the following scenarios for a change in availability of epidural analgesia over time. Fixed availability. Epidural analgesia is available at a certain time during the first time period, such as the daytime, and is also available at another time, such as the evening, during the second time period. Alternatively, epidural analgesia is only available for a certain class of patients in the first time period, such as very sick patients, and is more widely available in the second time period. Random availability. More anesthesiologists are available to provide epidural analgesia during the second time period than the first. When a patient arrives at the hospital, the availability of epidural analgesia is random, depending on whether or not the anesthesiologist is working elsewhere in the hospital. 3. T HOUGHT EXPERIMENT With the two scenarios for availability in mind, consider the following thought experiment. Suppose each woman who delivered in the first time period had entered the hospital at a random time during the second time period and vice versa. Under fixed availability, we assume that a women would arrive at the same time of day in either time period. Under random availability, we assume the same probability

Rethinking historical controls

385

Table 1. Paired availability design. Studies ordered from smallest to largest sample size Study

Times

Number Fraction who received Epidural CS Johnson and Rosenfeld (1995) 1993 116 0.586 0.172 1994 103 0.223 0.184 Frolich et al. (1997) 1985–90∗ 180 0.29 0.08 1991–96∗ 180 0.44 0.09 Lyon et al. (1997) 1992–93 373 0.131 0.110 1993–94 421 0.587 0.100 Robson et al. (1993) 1987 1000 0.10 0.04 1992 1000 0.45 0.05 Gribble and Meier (1991) 1986–87 1298 0.00 0.074 1989–91 1084 0.48 0.065 Larsen (1992) 1989–90 1919 0.000 0.275 1990–91 2073 0.316 0.229 Fogel et al. (1998) 1992–93 3195 0.01 0.030 1993–94 3733 0.29 0.031 Yancy et al. (1999) 1992–93 4778 0.008 0.194 1995–96 4859 0.586 0.190 Mancuso (1993) 1988–90 4685 0.187 0.149 1990–93∗ 6170 0.551 0.125 Newman et al. (1995) 1988–90 8108 0.467 0.248 1991–94 9918 0.678 0.280 Dailey (1999) 1989 2175 0.18 0.207 1990 2239 0.27 0.197 1991 2115 0.35 0.233 1992 2226 0.43 0.215 1993 2404 0.40 0.197 1994 2476 0.467 0.209 1995 2450 0.49 0.213 1996 2334 0.521 0.207 1997 2320 0.483 0.197 1998 2289 0.535 0.218

Estimate (s.e.) Both −0.033 (0.143) 0.067 (0.196) −0.022 (0.048) 0.029 (0.026) −0.019 (0.022) −0.146 (0.044) 0.004 (0.015) −0.006 (0.014) −0.066 (0.018) 0.152 (0.031) 0.035 0.066 0.081 0.092 0.074 0.089 0.103 0.114 0.102 0.123

0.011 (0.025)∗∗

∗ Data read from a graph ∗∗ Based on model for multiple times.

of availability throughout the time period. Under either fixed availability or random availability, we can classify subjects by whether or not they would have received epidural analgesia in each time period. The classification depends on the preference of the subject and the availability of epidural analgesia. Let R be a random variable which denotes this classification; it has four realizations: a, always-receivers, who would receive epidural analgesia in either time period; c, consistent-receivers, who would not receive epidural analgesia in the time period with less availability and would receive it in the time period with greater availability; i, inconsistent-receivers, who would receive epidural analgesia in the time period with less availability and would not receive it in the time period with greater availability; n, never-receivers, who would not receive epidural analgesia in either time period. For some subjects we will not be able to deduce the realization for R but this is not necessary for our analysis. The variable R has its origins in Baker and Lindeman (1994) and is similar to a variable independently proposed by Angrist et al. (1996).

386

S. G. BAKER AND K. S. L INDEMAN 4. F RAMEWORK FOR INFERENCE

Having presented the thought experiment, we can now outline our framework for inference. Let Y = 1 if a subject has a CS and 0 otherwise. Let D = 1 if a subject receives epidural analgesia and 0 otherwise. Ideally one would compare outcomes after all the patients received D = 1, and then, after turning back the clock, the same patients received D = 0. In our notation the comparison would be pr(Y = 1| D = 1) − pr(Y = 1| D = 0). This study design would eliminate the possibility that different outcomes for D = 1 and D = 0 resulted from differences inherent to the patients. Although this study design is impossible to implement, it sets a standard for evaluation. Using a traditional approach to historical controls, one would compare CS probabilities among subjects who received epidural in the later time, T = 1, with those who received it in the earlier time, T = 0. In our notation the comparison is pr(Y = 1 |D = 1, T = 1) − pr(Y = 1 | D = 0, T = 0). Estimation based on this comparison is subject to selection bias because subjects who receive epidural analgesia may have certain risk factors, such as painful labor, that make it more likely from them to have CS even if epidural analgesia has no effect (Hess et al., 2000). Now consider the paired availability design. Let Z = 0 denote the time period before a change in the availability of epidural analgesia, and let Z = 1 denote the time period after a change in availability. The paired availability design directly compares the probability of CS in all eligible subjects at Z = 0 versus the probability of CS in all eligible subjects when Z = 1, namely, pr(Y = 1|Z = 1) −pr(Y = 1|Z = 0). Because this comparison is based on all subjects, it avoids the selection bias associated with traditional historical controls. However pr(Y = 1|Z = 1) −pr(Y = 1|Z = 0) estimates the effect of increased availability on the probability of CS, not the effect of epidural analgesia on probability of CS, which is of primary interest. We will show that with additional assumptions, we can use the paired availability design to estimate the effect of epidural analgesia on the probability of CS among a special subset of the population, namely consistent-receivers. In our notation, this comparison is θ = pr(Y = 1 |D = 1, R = c) − pr(Y = 1|D = 0, R = c).

(4.1)

In an ideal situation, (4.1) is a comparison of outcomes after all the consistent-receivers received D = 1, and then, after turning back the clock, the same consistent-receivers received D = 0. In practice (4.1) is still a causal effect because of assumptions which state that ‘assignment’ to Z (which determines D in consistent-receivers) is essentially random and some additional assumptions. Because R is a baseline covariate, estimation based on θ avoids selection bias associated with traditional historical controls. Note that θ strictly applies only to consistent-receivers. In practice, one might argue that θ correctly measures the effect in all subjects. We will argue that the assumptions necessary for valid inference from θ are plausible, much more so than those required for valid inference with traditional historical controls. 5. A SSUMPTIONS To appropriately base inference on (4.1) we require the following assumptions. These assumptions modify those in Baker and Lindeman (1994), putting them on firmer foundation by incorporating both fixed and random changes in availability and including more supporting evidence. The following assumption, or a variant, is necessary with any type of study involving possible confounding over time. A SSUMPTION 1 (S TABLE CARE ) There were no systematic changes obstetric practice, unrelated to epidural analgesia, which could affect the probability of CS. We restrict Assumption 1 to systematic changes, because combining estimates from multiple hospitals will average the effect of random changes in obstetric practice over time. In support of Assumption 1,

Rethinking historical controls

387

with the exception of abstracts by Robson et al. (1993) and Newman et al. (1995) which did not discuss the matter, all of the studies reported that obstetric practice did not vary over time. In addition, three considerations mitigate the possibility of substantial bias from a systematic change in obstetric practice over time. First, the studies spanned a period of 15 years and included hospitals in both North America and Europe, so it is unlikely the same systematic change, if any, occurred in all studies. Second, in all the studies utilization of epidural analgesia increased over time with the exception of Johnson and Rosenfeld (1995) in which epidural availability decreased following initiation of state-funded health care insurance. Therefore, a systematic change over time would have the opposite effect in Johnson and Rosenfeld (1995) compared with the other studies. Third, as we will discuss, in the study by Dailey (1999) with multiple time periods, we tested for a change in effect over time and found it was not statistically significant. The second assumption, or a variant, is also necessary with any study with possible confounding over time. Here it has important implications regarding the thought experiment. A SSUMPTION 2 (S TABLE POPULATION ) There are no changes between the two time periods in the characteristics of the population related to the probability of CS in the absence of epidural analgesia. The joint distribution of whether or not a subject requests epidural analgesia, the arrival time at the hospital, and time course of labor is the same in both time periods, so the distribution of R does not depend on the time period: pr(R = r |Z = 0) = pr(R = r | Z = 1). In support of Assumption 2, with the exception of abstracts by Robson et al. (1993) and Newman et al. (1995) which did not discuss the matter, all the studies reported that the patient population did not vary over time. In addition, many of the studies involved a closed population served by an army medical center or the only hospital in a geographic region. It would be unlikely that a women in labor would go to another, considerably less convenient, hospital in order to receive epidural analgesia. As in Baker and Lindeman (1994) and Angrist et al. (1996), in which it is called an exclusion restriction, we make the following assumption which allows us to ‘peel away’ information from subjects who would not change treatment in the thought experiment and thus estimate the quantity of interest. A SSUMPTION 3 (N O EFFECT OF TREATMENT TIMING ) For always-receivers and never-receivers, the probability of CS does not depend on the time period. In our notation, pr(Y = 1| R = r, Z = 1) = pr(Y = 1| R = r, Z = 0), for R = a, n. For always-receivers, the time during the course of labor when epidural analgesia is initiated depends on the availability of epidural analgesia and hence differs by time period. For example, under random availability, a subject in Z = 1 might receive an epidural sooner than if she were in Z = 0, simply because an anesthesiologist was available sooner. Therefore Assumption 3 would be violated if the initiation time of epidural analgesia affected the probability of CS. However, after randomizing subjects to early or late initiation of epidural analgesia, Chestnut et al. (1994) found that time of initiation did not affect the probability of CS. For never-receivers, one possible complication is that some women will not receive epidural analgesia due to rapid delivery and might have a different probability of CS than women who receive opioid analgesia. However, if Assumption 2 holds, the fraction of women with rapid deliveries should be similar for Z = 0 or Z = 1, so the probability of CS will be the same for all never-receivers. We require one final assumption. A SSUMPTION 4 (S TABLE PREFERENCES ) There is no change in preference for epidural analgesia over time. For fixed availability, there are no inconsistent receivers: namely, pr(R = i) = 0. For random availability, the effect of epidural analgesia on consistent-receivers is the same as on inconsistentreceivers: namely, θ = γ , where γ = pr(Y = 1|D = 1, R = i) − pr(Y = 1|D = 0, R = i). Assumption 4 would be violated if new information became available in the second time period, such as a report warning of the risks from epidural analgesia, which would alter preferences for epidural

388

S. G. BAKER AND K. S. L INDEMAN Table 2.

Thought experiment: fixed availability and stable preferences

R = a always-receiver R = c consistent-receiver R = n never-receiver

Request epidural yes yes  no no

Time of day daytime evening daytime evening

Available in Z = 0 yes no yes no

Available in Z = 1 yes yes yes yes

R = i is not possible with stable preferences and fixed availability

Table 3.

Thought experiment: random availability and stable preferences

R = a always-receiver R = c consistent-receiver R = i inconsistent-receiver R = n never-receiver

Request epidural yes yes yes  yes no

Available in Z = 0 yes no yes no yes/no

Available in Z = 1 yes yes no no yes/no

analgesia. Technically, Assumption 4 is also needed to justify Assumption 2 that the distribution of R is the same over time. We do not know of any violations of Assumption 4. For fixed availability, Assumption 4 is the monotonicity assumption in Angrist et al. (1996) and was the assumption used in Baker and Lindeman (1994). Under fixed availability, the only way for some subjects to become inconsistent-receivers is for preferences to change over time (Table 2), which Assumption 4 rules out. For random availability, Assumption 4 is predicated on the fact that both inconsistent- and consistent-receivers would always request epidural regardless of time period (Table 3); the only difference is whether or not, by chance, an anesthesiologist were available to provide the epidural analgesia. 6. E STIMATION FOR TWO TIME PERIODS Invoking the assumptions, we can obtain a simple estimate for (4.1) Starting with the identity pr(Y = 1|Z = z) = r pr(Y = 1|Z = z, R = r ) pr(R = r |Z = z), and invoking Assumption 2, we obtain pr(Y = 1|Z = z) = r pr(Y = 1|Z = z, R = r ) pr(R = r ). Invoking Assumption 3, gives  pr(Y = 1|Z = z) = pr(Y = 1| R = r ) pr(R = r ) r =a,n

+



pr(Y = 1|Z = z, R = r ) pr(R = r )

(6.1)

r =c,i

Using (6.1), the difference in probabilities of CS between the two time periods is pr(Y = 1|Z = 1) − pr(Y = 1|Z = 0) = [pr(Y = 1|Z = 1, R = c) − pr(Y = 1|Z = 0, R = c)] pr(R = c) + [pr(Y = 1|Z = 1, R = i) − pr(Y = 1|Z = 0, R = i)] pr(R = i).

(6.2)

Rethinking historical controls

389

Essentially, Assumption 3 has ‘peeled away’ information from always-receivers and never-receivers, which was uninformative because there was no change in treatment. Noting that realizations for Z and R imply a realization for D, we rewrite (6.2) as pr(Y = 1|Z = 1) − pr(Y = 1|Z = 0) = θ pr(R = c) − ψ pr(R = i),

(6.3)

where ψ = pr(Y = 1|D = 1, R = i) − pr(Y = 1|D = 0, R = i) and θ was defined in (4.1). Invoking Assumption 2 and the definition of a, i, and c, we can write pr(D = 1|Z = 0) = pr(R = a) + pr(R = i) and pr(D = 1|Z = 1) = pr(R = a) + pr(R = c), which together imply pr(D = 1|Z = 1) − pr(D = 1|Z = 0) = pr(R = c) − pr(R = i).

(6.4)

Dividing (6.4) by (6.3) gives pr(Y = 1|Z = 1) − pr(Y = 1|Z = 0) pr(R = c) θ − pr(R = i) ψ = . pr(D = 1|Z = 1) − pr(D = 1|Z = 0) pr(R = c) − pr(R = i) By invoking Assumption 4, which says that pr(R = i) = 0 for fixed availability or θ = ψ for random availability, we can express θ as the ratio of the difference in probabilities of CS to the difference in probabilities of epidural analgesia, θ=

pr(Y = 1|Z = 1) − pr(Y = 1|Z = 0) . pr(D = 1|Z = 1) − pr(D = 1|Z = 0)

(6.5)

Baker and Lindeman (1994) gave a related proof for fixed availability. In a different context, Angrist et al. (1996) gave a related proof based on equivalent assumptions. Let n zdy denote the number of subjects in time period z who receive d and have outcome y, and let pzdy = n zdy /n z++ . The maximumlikelihood estimate is  θ = ( p1+1 − p0+1 )/( p+11 − p+01 ), with approximate asymptotic variance of (z pz+1 pz+0 /n z++ )/( p+11 − p+01 )2 (Baker and Lindeman, 1994). 7. E STIMATION FOR MULTIPLE TIME PERIODS In some studies the change in epidural use occurred gradually and results were reported at multiple time periods, Z = 0, 1, 2, . . . , k. One approach to analyzing these data is to separately combine the data from the first k/2 time periods and the last k/2 time periods and treat it as a two-period design. However, if data {n zdy } are available, as in the largest study by Dailey (1999), we can utilize all the information to obtain better estimates. For each pair of successive time periods, z and z + 1, we define a variable Rz with realizations {a, c, i, n}, as with variable R. Let θz = pr(Y = 1| D = 1, Rz = c) − pr(Y = 1|D = 0, Rz = c). If the assumptions cover multiple time periods, we can extend (6.5) to θz =

pr(Y = 1|Z = z + 1) − pr(Y = 1|Z = z) . pr(D = 1|Z = z + 1) − pr(D = 1|Z = z)

(7.1)

To summarize the effect of epidural analgesia on the probability of CS, we want to combine the estimates for θz into an overall estimate for θ . One possibility is to estimate θz separately for each time period and combine the results by taking a weighted average. The difficulty is that the estimates for each time period z are not independent, making it difficult to determine the optimal weights. Instead we formulate the following model and compute the maximum-likelihood estimate when θz = θ. Define the marginal probabilities, pr(Y = 1|Z = 1) = γ and pr(D = 1|Z = z) = αz , so pr(Y = 1|Z = z) = γ + θ(αz − α1 ).

390

S. G. BAKER AND K. S. L INDEMAN

Including a nuisance parameter βz for one of the joint probabilities of Y and D, we can write the joint probabilities of outcome and analgesia as pr(Y = 1, D = 1|Z = z) = βz , pr(Y = 0, D = 1|Z = z) = αz − βz , pr(Y = 1, D = 0 |Z = z) = γ + θ (αz − α1 ) − βz , pr(Y = 0, D = 0|Z = z) = 1 − γ − θ(αz − α1 ) − αz − βz . The kernel of the likelihood is L=

k 

(1 − γ − θ(αz − α1 ) − αz − βz )n z00

z=0

(γ + θ (αz − α1 ) − βz )n z01

(αz − βz )n z10

βzn z11 ,

(7.2)

which we maximized using a Newton–Raphson algorithm. For the data in Dailey (1999), we obtained  θ = 0.011 with a standard error of 0.025. In order to investigate whether the effect of epidural on the probability of CS changed over time, we also fit a model with θz . There was no evidence to reject the more parsimonious model: the deviance increased by 14.8 on nine degrees of freedom, ( p = 0.10), and { θz } = {−0.11, 0.15, 0.03, −0.04, 0.01, 0.02, 0.00, −0.03, 0.03} indicated no trend over time. As expected, fitting the model to multiple time periods reduced the standard error. With five time periods of 2 years each, the estimated effect was 0.015 with a standard error of 0.026. With two time periods, it was −0.004 with a standard error of 0.031. 8. C OMBINING ESTIMATES OVER ALL STUDIES In ten studies we used the formulation for two time periods because either (i) the availability of epidural analgesia changed suddenly (Johnson and Rosenfeld, 1995; Lyon et al., 1997; Gribble and Meier, 1991; Larsen, 1992; Fogel et al., 1998), (ii) only data from two time periods were reported (Robson et al., 1993; Frolich et al., 1997; Yancy et al., 1999), or (iii) only marginal data {n z+y , n zd+ } were reported (Mancuso, 1993; Newman et al., 1995). As discussed previously, in the largest study by (Dailey, 1999), we had the necessary data to use the formulation for multiple time periods. To combine the estimates from each study (see Table 1 and Figure 1), we used a weighted sum based on a random-effects model (Dersimonian and Laird, 1986) and a paired permutation distribution, treating the time periods as in the observed order or reversed. For each estimate, let p denote the probability of the same order and hence the same estimate, and 1 − p the probability of the reverse order and hence negative of the estimate. Under the null hypothesis, p = 0.5. Applying the computational technique of Follman and Proschan (1999), we inverted hypothesis testing to obtain a confidence interval. Our estimate was 0.00 with a 95% confidence interval of (−0.06, 0.05). 9. M ULTIVARIATE ADJUSTMENT FOR CONCURRENT CONTROLS An alternative method for estimating the effect of epidural analgesia on the probability of CS is a multivariate adjustment for concurrent controls. Let S = 1 denote the subjects who received epidural and S = 0 denote those who did not. We include S to indicate that different subjects received D = 1 and D = 0. A comparison is based on pr(Y = 1|D = 1, S = 1, w) − pr(Y = 1|D = 0, S = 0, w) where, ideally, given risk factors w, there are no unknown risk factors with different distributions for D = 0 and D = 1. The best published multivariate adjustment for studying the effect of epidural analgesia on the

Rethinking historical controls Paired Availability Design

study

95% CI

Johnson

size

difference in epidural rates

219 – 36% 360 15% 794 46% 2000 35% 2382 48% 3992 32% 6928 28% 9637 58% 10855 36% 18026 21% 23028 17%

Frolich Lyon Robson Gribble Larsen Fogel Yancy Mancuso Newman Dailey summary

391

Meta-Analysis of Randomized Trials difference in epidural 95% trial size rates CI Muir1

50

50%

Thorp

93

96%

Bofill

100

76%

Philipson Muir2

111 100% 185 100%

Clark

318

45%

Sharma2

342

93%

Loughnan

614

26%

Sharma1 Gambling

715

66%

1223

39%

summary – .3 0 .3

– .3 0 .3

Fig. 1. Paired availability design and meta-analysis of randomized trials.

probability of CS in observational data was a propensity score analysis of 1733 subjects by Lieberman et al. (1996). This study considered many more variables than any other observational study of epidural analgesia and CS: maternal age, race, insurance, pre-pregnant weight, height, birth weight, gestational duration, dilation at admission, initial rate of cervical dilation, station of fetal head at admission, active management, ruptured membrane at admission, maternal chronic hypertension, and maternal pregnancy hypertension. Unlike many of the other observational studies, Lieberman et al. (1996) did not include oxytocin as a predictor of the probability of CS. Because epidural analgesia likely affects the utilization of oxytocin, oxytocin is an outcome variable and thus inappropriate as a baseline predictor. To convert the results in Lieberman et al. (1996) from an odds ratio to a difference, we applied the method of subclassification of propensity scores (Rosenbaun and Rubin, 1984) to data from Figure 2 in Lieberman et al. (1996). With this approach we estimated that epidural analgesia increases the probability of CS by 0.10 with 95% confidence interval of (0.07, 0.13). Unlike the paired availability analysis, the multivariate adjustment for concurrent controls indicated that epidural analgesia has a large effect on the probability of CS. We believe the reason for this discrepancy is that the multivariate analysis omitted an important predictor. According to DanilenkoDixon and Van Winter (1997), ‘ . . . even the most meticulous statistical methods will be unable to recognize, quantify, and account for all potential confounding variables. For instance, potential confounders that are very difficult to measure or impossible to quantify and control include the parturient’s perception of and physiological response to pain and the size, shape, and degree of laxity of the maternal pelvis—especially in relation to the size, shape, presentation, and station of the fetal head, both during the onset of labor and with respect to changes occurring the progression of labor.’ Lieberman et al. (1996) admitted ‘we cannot rule out the possibility of residual confounding by an unmeasured and uncontrolled factor. For example we did not have information regarding pelvic size.’ However, Lieberman et al. (1996) argued ‘that for some other factor not controlled in our analysis to be responsible for the association we have noted, it would have to be very strongly associated with epidural analgesia use and cesarean delivery. There are no obvious candidates apart from the factors we have measured . . . . The association of such a factor with epidural analgesia and cesarean delivery would need to be at least as large as the association

392

S. G. BAKER AND K. S. L INDEMAN Table 4. Randomized trials. Studies ordered from smallest to largest sample size

Study Muir et al. (1994) Thorp et al. (1993) Bofill et al. (1997) Philipsen and Jensen (1989) Muir et al. (2000) Clark et al. (1998) Sharma et al. (2000) Loughnan et al. (2000) Sharma et al. (1997) Gambling et al. (1998)

control study control study control study control study control study control study control study control study control study control study

Size 22 28 45 48 51 49 54 57 88 97 162 156 172 170 310 304 357 358 607 616

Numbers Epidural CS 11 2 28 3 1 1 47 12 12 3 48 5 0 6 57 10 0 9 97 11 84 22 151 15 12 15 170 11 175 40 257 36 5 16 243 13 159 34 400 39

Proportions Epidural CS 0.50 0.05 1.00 0.07 0.02 0.02 0.98 0.25 0.24 0.06 1.00 0.10 0.00 0.11 1.00 0.18 0.00∗ 0.10 1.00∗ 0.11 0.52 0.10 0.97 0.06 0.07 0.09 1.00∗ 0.06 0.56 0.12 0.83 0.13 0.01 0.03 0.68 0.03 0.26 0.04 0.65 0.04

Estimate (s.e.) 0.052 (0.132) 0.238 (0.069) 0.057 (0.071) 0.064 (0.066) 0.011 (0.046) −0.105 (0.068) −0.024 (0.031) −0.040 (0.010) −0.004 (0.018) −0.010 (0.029)

∗ Compliance not reported in the abstract, so full compliance was assumed.

we observed between epidural analgesia and cesarean delivery for it to be a competing explanation for our findings.’ We believe that intense pain early in labor could be such a factor. Women with greater pain early in labor are more likely to request epidural analgesia and are also at higher risk for operative delivery (National Institute of Child Health and Human Development, 1981; Wuitchik et al., 1989; Hess et al., 2000), which would increase the apparent effect the effect of epidural analgesia on the probability of CS (Rosenbaum, 1989). 10. VALIDATION : A META - ANALYSIS OF RANDOMIZED TRIALS Our gold standard for evaluation is the randomized trial. We identified ten randomized controlled trials of epidural analgesia since 1980 which reported data on CS rates in each randomized group, either in an article or an abstract (Table 4). We dropped from our meta-analysis the study by Nikkola et al. (1997) because there were no cases of CS in either arm, which made it difficult to include in the calculations. Because Nikkola et al. (1997) randomized only 20 patients, we believe the information loss was minimal. In this setting, let Z = 0 if a subject were randomized to no epidural, and Z = 1 if randomized to epidural. The intent-to-treat estimate is pr(Y = 1|Z = 1) − pr(Y = 1|Z = 0). By virtue of the randomization, the distribution of unknown risk factors is the same in the two groups. In some trials many subjects randomized to the epidural group did not receive epidural analgesia, and many randomized to the control group received epidural analgesia. Previous meta-analyses were based on intent-to-treat (Halpern et al., 1998; Howell, 1999; Zhang et al., 1999). In the presence of noncompliance, analysis by intent-to-treat estimates the effect of assignment to the epidural group, not quantity of interest, the effect of epidural analgesia. To adjust for this noncompliance, we used the same framework as with the paired

Rethinking historical controls

393

Table 5. Effect of epidural analgesia on the probability of CS Paired availability design Meta-analysis of randomized trials Multivariate adjustment of concurrent controls

Estimate 0.00 0.02 0.10

95% confidence interval (−0.06, 0.05) (−0.02, 0.08) (0.07, 0.13)

availability design. This approach is similar to that in Angrist et al. (1996). For each study, we obtained an estimate using (4.1) (see Figure 1 and Table 4). Randomization ensures that Assumptions 1 and 2 hold. Assumption 3 holds for the same reason as with the paired availability design. In Loughnan et al. (2000), a small number of subjects received both opioid and epidural analgesia, and it is necessary to also assume no interactive effect on the probability of CS. Under randomization, Assumption 4 says no subject would receive epidural if randomized to no epidural and receive no epidural if randomized to epidural, for which we believe there would be few exceptions. Our summary estimate, based on a random effects meta-analysis with a paired permutation distribution (Follman and Proschan, 1999), was 0.02 with a 95% confidence interval of (−0.02, 0.08). The estimate and confidence interval were similar to those from the paired availability design, providing strong support for the validity of inference based on the paired availability design. 11. D ISCUSSION As with traditional historical controls, the paired availability design assumes no changes over time which could confound the effect of treatment. We considered three types of temporal assumptions. Assumption 1 (stable care) ensures one is measuring the effect of treatment. Assumption 2 (stable population) ensures the same distribution of unknown covariates. Assumption 4 (stable preference), which is not often considered, ensures no new information which would alter the type of subject selecting a treatment. There was no evidence of any gross violation of these assumptions. Traditional historical controls must also assume that subjects who receive the old treatment in the previous time period have the same distribution of unknown covariates as subjects who receive new treatment current time period. As a much more plausible alternative, the paired availability design requires Assumption 3 (no effect of treatment timing), which makes it possible to ‘peel away’ the effect of treatment in always- and neverreceivers. By combining the results of multiple studies, the paired availability design dampens the effect of random violations of the assumptions. The striking similarity between the confidence intervals based on the meta-analysis of randomized trials and the paired availability design, particularly in comparison with confidence interval from the multivariate adjustment for concurrent controls (Table 5), provides strong support that the paired availability design reduces selection bias. In interpreting results from all the approaches, one should realize that it was not possible to blind the obstetricians to the type of analgesia. An obstetrician who knew that a patient received epidural analgesia might have been more likely to perform a CS because of (i) a preconceived notion that epidural analgesia increases the probability of CS or (ii) a perception that the patient was ready for anesthesia and surgery for CS by the presence of the epidural catheter. The lack of blinding would most likely have increased the apparent association between epidural analgesia and the probability of CS. Because we found only a small effect of epidural analgesia on the probability of CS, we do not believe the lack of blinding had much, if any, noticeable effect. As one referee pointed out, the key idea is that ‘the availability of treatment may vary with time in ways that have little to do with the characteristics of the patients’, so it is better to base the analysis of historical controls on changes in availability of treatment rather than on changes in receipt of treatment

394

S. G. BAKER AND K. S. L INDEMAN

as is traditionally done. Another referee noted that because ‘differences in availability are more plausibly like ‘a coin toss’ compared to differences in the actual taking of treatments’, one could use multivariate adjustments, such as propensity scores, to better estimate the effect of changes in availability of treatment, and then make additional adjustments to estimate the effect of receipt of treatment. We agree that it best to adjust for observed covariates, if they are available and note that, alternatively, one could directly incorporate covariates into the likelihood in (7.2). However, in this application, the additional covariate data were not generally available and would be difficult or impossible to obtain in many of the studies. Our findings of a small effect of epidural analgesia on the probability of CS are clinically important because a small increase in the probability of CS would likely deter few women from selecting the most effective form of pain relief for labor. According to Chestnut (1997), ‘Even if it were shown that the contemporary use of epidural analgesia results in a small increase in the CS rate, it is unclear how many women—who now choose epidural analgesia—would voluntarily opt for nothing or intravenous opioid analgesia, which is less effective and provides marked sedation.’ It is also economically important because some insurance companies have considered withholding reimbursement for epidural analgesia because of perceived high probability of CS (Chestnut, 1997).

ACKNOWLEDGEMENTS The authors thank two anonymous referees for their insights, Patricia Daily for invaluable data from the Mills-Pennisula hospital, and Barbara Leighton and Steven Segal for helpful comments and assistance in locating appropriate studies.

R EFERENCES A NGRIST , J. D., I MBENS , G. W. AND RUBIN , D. B. (1996). Identification of causal effects using instrumental variables (with discussion). Journal of the American Statistical Association 91, 444–472. BAKER , S. G. AND L INDEMAN , K. S. (1994). The paired availability design: a proposal for evaluating epidural analgesia during labor. Statistics in Medicine 13, 2269–2278. BAKER , S. G. (1997). Compliance, all-or-none. In Kotz, S., Read, C. R. and Banks, D. L. (eds), The Encyclopedia of Statistical Science, Update Vol. 1. New York: Wiley, pp. 134–138. BAKER , S. G. (1998). The paired availability design: an update. In Abel, U. and Koch, A. (eds), Nonrandomized Comparative Clinical Studies, Dusseldorf: Symposion, pp. 79–84. B IRNBACH , D. J., G RUNEBAUM , A., S TEIN , D. J., K ARGACIN , B., K URODA , M. M. AND T HIS , D. M. (1997). Does epidural analgesia protect against Cesarean section in nulliparous patients? A retrospective review of 13 203 patients, Abstract (Presented at the Annual Meeting of Society of Obstetric Anesthesiology and Perinatology) (Hamilton, Bermuda). B OFILL , J. A., V INCENT , R. D., ROSS , E. L., M ARTIN , R. W., N ORMAN , P. F., W ERHAN , C. F. AND M ORRISON , J. C. (1997). Nulliparous active labor, epidural analgesia, and cesarean delivery for dystocia. American Journal of Obstetrics and Gynecology 177, 1462–1470. B YAR , D. (1980). Why data bases should not replace randomized clinical trials. Biometrics 36, 337–342. C HESTNUT , D. H. (1997). Epidural analgesia and the incidence of Cesarean section. Time for another close look. Anesthesiology 87, 472–476. C HESTNUT , D. H., M C G RATH , J. M., V INCENT , R. D., P ENNING , D. H., C HOI , W. W., BATES , J. N. AND M C FARELANE , C. (1994). Does early administration of epidural analgesia affect obstetric outcome in nulliparous women who are in spontaneous labor? Anesthesiology 80, 1201–1208.

Rethinking historical controls

395

C LARK , A., C ARR , D., L OYD , G., C OOK , V. AND S PINNATO , J. (1998). The influence of epidural analgesia on cesarean delivery rates: a randomized, prospective clinical trial. American Journal of Obstetrics and Gynecology 179, 1527–1533. DAILEY , P. (1999). Personal communication. DANILENKO -D IXON , D. AND VAN W INTER , J. T. (1997). The effect of epidural analgesia on labour and delivery. Postgraduate Obstetrics and Gynecology 17, 1–8. D ERSIMONIAN , R. AND L AIRD , N. M. (1986). Meta analysis of clinical trials. Controlled Clinical Trials 7, 177–188. F OLLMAN , D. A. 737.

AND

P ROSCHAN , M. A. (1999). Inference in random effects meta-analysis. Biometrics 55, 732–

F OGEL , S., S HYKEN , J. M., L EIGHTON , B. L., M ORMOL , J. S. AND S MELTZER , J. (1998). Epidural labor analgesia and the incidence of cesarean delivery for dystocia. Anesthesia and Analgesia 87, 119–123. F ROLICH , M. A., O RTH , V., K NITZA , R., F INSTERER , U., H EPP , H. AND P ETER , K. (1997). Does epidural analgesia reduce the incidence of operative delivery? Abstract (Presented at the Annual Meeting of Society of Obstetric Anesthesiology and Perinatology) (Hamilton, Bermuda). G AMBLING , D. R., S HARMA , S. K., R AMIN , S. M., L UCAS , M. J., L EVENO , K. J., W ILEY , J. AND S IDAWI , J. E. (1998). A randomized study of combined spinal-epidural analgesia versus intravenous meperidine during labor. Anesthesiology 89, 1336–1344. G RIBBLE , R. K. AND M EIER , P. R. (1991). Effect of epidural analgesia on the primary Cesarean rate. Obstetrics and Gynecology 78, 231–234. H ALPERN , S., L EIGHTON , B. L., O HISSON , A., BARRETT , J. F. R. AND R ICE , A. (1998). Effect of epidural verses parenteral opioid analgesia on the progress of labor. A meta-analysis. Journal of the American Medical Association 289, 2105–2110. H AWKINS , J. L., G IBBS , C. P., O RLEANS , M., M ARTIN -S ALVAJ , G. AND B EATY , B. (1997). Obstetric anesthesia work force survey, 1981 versus 1992. Anesthesiology 87, 135–143. H ESS , P. E., P RATT , S. E., S ONI , A., S ARNA , M. C. AND O RIOL , N. E. (2000). An association between severe labor pain and Cesarean delivery. Obstetric Anesthesia 90, 881–886. H OWELL , H. J. (1999) Epidural versus non-epidural analgesia for pain relief in labour, Cochrane Review Oxford: The Cochrane Library, update software. J OHNSON , S. AND ROSENFELD , J. A. (1995). The effect of epidural anesthesia on the length of labor. The Journal of Family Practice 40, 244–247. L ARSEN , D. D. (1992). The effect of initiating an obstetric anesthesiology service on rate of Cesarean section and rate of forceps delivery. Abstract (Presented at the Annual Meeting of the Society of Obstetric Anesthesia and Perinatology). L IEBERMAN , E. (1999). No free lunch on labor day. The risks and benefits of epidural analgesia during labor. Journal of Nurse-Midwifery 44, 394–398. L IEBERMAN , E., L ANG , J., C OHEN , A., D’ AGOSTINO , R., DATTA , S. AND F RIGOLETTO , F. (1996). Association of epidural analgesia with cesarean delivery in nulliparas. Obstetrics and Gynecology 88, 993–1000. L OUGHNAN , R. A., C ARLI , F., ROMNEY , M., D ORE , C. J. AND G ORDON , H. (2000). Randomized controlled comparison of epidural bupivicaine versus pethidine for analgesia in labor. British Journal of Anesthesia 84, 715–719. LYON , D., K NUCKLES , G., W HITAKER , E. AND S ALGADO , S. (1997). The effect of instituting an elective labor epidural program on the operative delivery rate. Obstetrics and Gynecology 90, 135–141. M ANCUSO , J. J. (1993). Epidural analgesia in an army medical center: impact on Cesareans and instrumental

396

S. G. BAKER AND K. S. L INDEMAN

deliveries, Abstract 13 (Presented at the Annual Meeting of the Society for Obstetric Anesthesiology and Perinatology) (Palm Springs). M UIR , H. A., S HULKLA , R., L ISTON , R. AND W RITER , D. (1996). Randomized trial of labor analgesia: a pilot study to compare patient-controlled intravenous analgesia with patient-controlled epidural analgesia to determine if analgesic method affects delivery outcome. Abstract. Canadian Journal of Anaesthesia 43, A60. M UIR , H., B REEN , T., C AMPBELL , D., H ALPERN , S., L ISTON , R. AND B LANCHARD , W. (2000). A multi-center study of the effect of analgesia on the progress of labor, Abstract A 23 (Presented at the Annual Meeting of the Society of Obstetric Anesthesiology and Perinatology) Montreal Canada. NATIONAL I NSTITUTE MA: NICHD.

OF

C HILD H EALTH

AND

H UMAN D EVELOPMENT (1981). Report No 82-2067. Bethesda,

N EWMAN , L. M., P EREZ , E. C., K ROLICK , T. J. AND I VANKOVICH , A. D. (1995). Labor analgesia, Cesarean anesthesia, and cesarean delivery rates for 18 000 deliveries from 1988 through 1994. Abstract. Anesthesiology 83, 3A, A967. N IKKOLA , E. M., E KBLAD , U. U., K ERO , O. O., A LIHANKA , J. J. M., M ARKKU , A. O. AND S ALONEN , M. A. O. (1997). Intravenous fentanyl PCA during labour. Canadian Journal of Anaesthesia 44, 1248–1255. P HILIPSEN , T. AND J ENSEN , N. (1989). Epidural block or parenteral pethidine as analgesia in labour; a randomized study concerning progress in labour and instrumental deliveries. European Journal of Obstetrics and Gynecology and Reproductive Biology 30, 27–33. P OCOCK , S. J. (1983). Clincial trials. A Practical Approach. Chichester: Wiley. ROBSON , M., B OYLAN , P., M C PARLAND , P., M C Q UILLAN , C. AND O’N EILL , M. (1993). Epidural analgesia need not influence the spontaneous vaginal delivery rate. American Journal of Obstetrics and Gynecology January, Abstract 240, 364. ROSENBAUN , P. R. AND RUBIN , D. R. (1984). Reducing bias in observational studies using subclassification of the propensity score. Journal of the American Statistical Association 79, 516–524. ROSENBAUM , P. R. (1989). The role of known effects in observational studies. Biometrics 45, 557–569. S HARMA , S. K., S IDAWI , J. E., R AMIN , S. M., L UCAS , M. J., L EVNO , K. J. AND C UNNINGHAM , G. (1997). Cesarean delivery: a randomized trial of epidural versus patient-controlled meperidine analgesia during labor. Anesthesiology 87, 487–494. S HARMA , S. K., L EVENO , K. J., M ESSICK , G., A LEXANDER , J. M., S IDAWI , J. E. AND W ILEY , J. (2000). A randomized trial of patient-controlled epidural versus patient-controlled intravenous analgesia during labor. Abstract A 22 (Presented at the Annual Meeting of the Society of Obstetric Anesthesia and Perinatology) (Montreal, Canada). T HORP , J. A., H U , D. H., A LBIN , R. M., M C N ITT , J., M EYER , B. A., C OHEN , G. AND Y EAST , J. D. (1993). The effect of intrapartum epidural analgesia on nulliparous labor: a randomized, controlled, prospective trial. American Journal of Obstetrics and Gynecology 169, 851–858. W UITCHIK , M., BAKAL , D. AND L IPSHITZ , J. (1989). The clinical significance of pain and cognitive activity in latent labor. Obstetrics and Gynecology 73, 35–42. YANCY , M. K., P IERCE , B., S CHWEITZER , D. AND DANIELS , D. (1999). Observations on labor epidural analgesia and operative delivery rates. American Journal of Obstetrics and Gynecology 180, 353–359. Z HANG , J., K LEBANOFF , M. AND D ERSIMONIAN , R. (1999). Epidural analgesia in association with duration of labor and mode of delivery: a quantitative review. American Journal of Obstetrics and Gynecology 180, 970–977. [Received 30 June, 2000; revised 11 September, 2000; accepted for publication 28 September, 2000]