Craft Of Political Research

Seventh Edition

W. Phillips Shively University

of Minnesota

PEARSON

Upper Saddle River, NJ 07458

Library of Congress Cataloging-in-Publication Data Shively.W. Phillips, 1942The craft of political research / W. Phillips Shively. — 7th ed. p. cm. Includes bibliographical references and index. ISBN-13: 978-0-13-602948-9 (alk. paper) ISBN-10: 0-13-602948-5 (alk. paper) 1. Political science—Methodology.

2. Political science—Research.

I. Title.

JA71.S45 2009 320.072—dc22 2007047170 Executive Editor: Dickson Musslewhite Editor-in-Chief: Yolanda de Rooy Associate Editor: Rob De George Editorial Assistant: Synamin Ballatt Senior Marketing Manager: Kate Mitchell Marketing Assistant: Jennifer Lang Managing Editor: Joanne Riker Manufacturing Buyer: Wanda Rockwell Creative Director: Christy Mahon Cover Design Director: Jayne Conte Cover Design: Jon Boylan Cover Illustration/Photo: Getty Images, Inc. Composition: Integra

To

Full-Service Project Management: Integra Printer/Binder: Courier-Stoughton Credits and acknowledgments borrowed from other sources and reproduced, with permission, in this textbook appear on appropriate page within text.

Copyright © 2009, 2005, 2 0 0 2 , 1 9 9 8 , 1 9 9 0 , 1 9 8 0 by Pearson Education, Inc., Upper Saddle River, New Jersey, 07458. All rights reserved. Printed in the United States of America. This publication is protected by Copyright and permission should be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permission(s), write to: Rights and Permissions Department. Pearson Prentice Hall™ is a trademark of Pearson Education, Inc. Pearson® is a registered trademark of Pearson pic Prentice Hall® is a registered trademark of Pearson Education, Inc. Pearson Pearson Pearson Pearson

Education Ltd. Education Singapore, Pte. Ltd. Education, Canada, Ltd. Education-Japan

Pearson Pearson Pearson Pearson

Education Australia PTY, Ltd. Education North Asia Ltd. Educacion de Mexico, S.A. de C.V. Education Malaysia, Pte. Ltd.

ISBN-13: ISBN-ID:

c

10 9 8 7 6 5 4 3 2 1 l7fl-D-13-bD5T4fl- 1 D-ig-bDSTMfl-S c

L

BARBARA

Preface

xi

Doing Research Social Research

1 2

Types of Political Research Research Mix

4

8

Evaluating Different Types of Research E t h i c s of P o l i t i c a l R e s e a r c h

10

11

Political Theories and Research Topics C a u s a l i t y and P o l i t i c a l T h e o r y

14

What Does Good Theory L o o k L i k e ?

15

Example of Elegant Research: Philip Converse To Quantify or Not C h o i c e of a T o p i c

20 22

Engineering Research

22

Theory-Oriented Research D e v e l o p m e n t of a R e s e a r c h D e s i g n

23 23

Observations, Puzzles, and the Construction of Theories 25

Contents

Contents

Machiavellian Guide to Developing R e s e a r c h Topics

A F e w B a s i c s of Research Design

28

Further D i s c u s s i o n

31

True Experiment

32

83

D e s i g n s for P o l i t i c a l R e s e a r c h E n g l i s h as a L a n g u a g e for R e s e a r c h

Ordinary Language

33

34

Dimensional

Further D i s c u s s i o n

37

Analysis

U s e o f V a r i e d D e s i g n s and M e a s u r e s

38

Conclusion

Further D i s c u s s i o n

Reliability as a Characteristic of Concepts

Validity

92

93

Holding a Variable Constant

41

45

Testing the Reliability of a Measure

90

Example of Varied Designs and Measures

40

Problems of Measurement: Accuracy Reliability

84

Special Design Problem for Policy Analysis: Regression to the Mean 88

Proper U s e of Multidimensional Words

of

79

Use of a Control Group 82

Importance of Dimensional Thinking

Example

78

Designs Without a Control Group

94

96

46

7

47

Selection of Observations for Study

97

S a m p l i n g f r o m a Population of Potential O b s e r v a t i o n s

48

Some Examples

Random Sampling

49

Checks for Validity

Quasi-random Sampling

51

Purposive Sampling I m p a c t o f R a n d o m and N o n r a n d o m E r r o r s Importance of A c c u r a c y

53

101 102

Selection of Cases for Case Studies

103

54 Censored Data

Further D i s c u s s i o n

56

P r e c i s i o n in M e a s u r e s

103

When Scholars Pick the Cases They 're Interested In

Problems of Measurement: Precision

When Nature Censors Data

57

D o n ' t D o It!

106

62 S e l e c t i o n o f C a s e s for C a s e Studies ( A g a i n )

Innate Nature of Levels of Measurement The Sin of Wasting Information

A n o t h e r Strategy, H o w e v e r : S i n g l e - C a s e Studies S e l e c t e d

64

for the R e l a t i o n s h i p B e t w e e n the Independent

Further D i s c u s s i o n

72

73

8

Causal Thinking and Design of Research

77

74

109

76

110

Introduction to Statistics: Measuring Relationships for Interval Data 111 Statistics

75

E l i m i n a t i o n o f A l t e r n a t i v e C a u s a l Interpretations

Summary

and D e p e n d e n t V a r i a b l e s

67

Quantifiers and Nonquantifiers Again

C a u s a l i t y : A n Interpretation

108

63

Enrichment of the Level of Precision in Measurement Examples of Enrichment

104

106

S e l e c t i o n A l o n g the D e p e n d e n t V a r i a b l e :

58

Precision in Measurement

Further D i s c u s s i o n

99

99

112

Importance of L e v e l s of Measurement W o r k i n g w i t h Interval D a t a

113

112

ix

Contents Regression Analysis

113

Correlation Analysis

122

Correlation and Regression Compared Problem

of

Measurement

Further D i s c u s s i o n

Error

127 130

131

Introduction to Statistics: Further Topics on Measurement of Relationships 132 M e a s u r e s o f R e l a t i o n s h i p for O r d i n a l D a t a

132

M e a s u r e s o f R e l a t i o n s h i p for N o m i n a l D a t a

Dichotomies and Regression Analysis L o g i t and Probit A n a l y s i s Multivariate A n a l y s i s Conclusion

136

136

139 141

146

Introduction to Statistics: Inference, or How to Gamble on Your Research 148 L o g i c of Measuring Significance

149

E x a m p l e o f Statistical I n f e r e n c e

I wrote this little b o o k in 1970, w h e n I w a s an assistant p r o f e s s o r at Y a l e U n i v e r s i t y . In teaching a n u m b e r of sections of Introduction to R e s e a r c h to undergraduates there, I had f o u n d that the students benefited f r o m an introduction that e m p h a s i z e d

150

the internal l o g i c of r e s e a r c h methods and the c o l l e c t i v e , cooperative nature of the Hypothesis Testing

151

r e s e a r c h p r o c e s s . I c o u l d not find a b o o k that presented things in this w a y at a suffi-

Null Hypothesis

152

ciently elementary l e v e l to be r e a d i l y a c c e s s i b l e by undergraduates. A n d so I wrote

2

Example: %

this book.

153

Sampling Distribution Importance of'N Problem

of

It has f o l l o w e d me through the rest of my career so far, and has g i v e n me

155

enormous p l e a s u r e . It has a l w a y s s e e m e d to me that it fills a needed n i c h e , and it h a s

158 Independent

been a thrill w h e n students have told me that they have benefited f r o m it. I am

Observations

Significance Test: A l w a y s N e c e s s a r y ?

161 162

pleased that it still s e e m s to be w o r k i n g for them. W h i l e the general p r i n c i p l e s of good argument and investigation don't c h a n g e , I have m a d e a n u m b e r of additions and deletions over the last c o u p l e of editions to

P o l l i n g and S i g n i f i c a n c e T e s t s

163

U s e s and L i m i t a t i o n s of Statistical T e s t s

reflect n e w p o s s i b i l i t i e s in technique. In this seventh edition, I have updated a n u m 163

ber of e x a m p l e s , I h a v e i m p r o v e d on my treatment of the use of n o m i n a l variables in r e g r e s s i o n a n a l y s i s , and I have added a n e w d i s c u s s i o n of the p r o b l e m of c o m p a r i n g

Conclusion

164

Further D i s c u s s i o n

units in r e g r e s s i o n a n a l y s i s . I have also c h a n g e d significantly the position I take on the question of selecting c a s e s for a n a l y s i s .

164

I must a l s o admit to a recent c h a n g e of heart (or, perhaps, a return to old l o v e ) .

Where Do Theories Come From?

166

R e g u l a r users of this text m a y r e c a l l that a c o u p l e of editions a g o , under prodding f r o m r e v i e w e r s , I c h a n g e d m y e x a m p l e o f elegant r e s e a r c h f r o m P h i l i p C o n v e r s e ' s

Selected Bibliography

170

"Of Time

and Partisan

Stability"

to

R o b e r t P u t n a m ' s Making Democracy

Work.

P u t n a m ' s b o o k a n d the resultant r e s e a r c h p r o g r a m on s o c i a l c a p i t a l are of c o u r s e

Index

174

splendid, but I a l w a y s regretted s a c r i f i c i n g C o n v e r s e ' s article. It is still the most

xi

xii

Preface

Chapter I

beautiful p i e c e of p o l i t i c a l s c i e n c e r e s e a r c h I know. A n d so I returned in the sixth e d i t i o n — a n d continue i n this seventh e d i t i o n — t o " O f T i m e and P a r t i s a n Stability," and i n c l u d e w i t h it the s a m e e x p l i c a t i o n of M a r k o v c h a i n s that I u s e d in those earlier editions. As y o u c a n no doubt tell f r o m the tone of this preface, this is a book for w h i c h I have great affection. I hope y o u w i l l e n j o y it as m u c h as I have e n j o y e d it.

ACKNOWLEDGMENTS

T h a n k s to the f o l l o w i n g r e v i e w e r s for their helpful suggestions and c o m m e n t s : Terri S u s a n F i n e , U n i v e r s i t y o f C e n t r a l F l o r i d a ; Garrett G l a s g o w , U n i v e r s i t y o f C a l i f o r n i a a t S a n t a B a r b a r a ; and P a t r i c k J a m e s , U n i v e r s i t y o f Southern C a l i f o r n i a .

W.

Phillips Shively

S c h o l a r l y r e s e a r c h is e x c i t i n g and is f u n to do. S o m e students, caught in the grind of daily and term a s s i g n m e n t s , m a y not see it this way. B u t for people w h o c a n carry on r e s e a r c h in a m o r e r e l a x e d w a y , for professors or for students w h o c a n involve t h e m s e l v e s in a l o n g - r a n g e project, r e s e a r c h m a y be a source of fascination and great satisfaction.

Francis's preoccupation with D N A quickly became full-time. The first afternoon following the discovery that A - T and G - C base pairs had similar shapes, he went back to his thesis measurements, but his effort was ineffectual. Constantly he would pop up from his chair, worriedly look at the cardboard models, fiddle with other combinations, and then, the period of momentary uncertainty over, look satisfied and tell me how important our work was. I enjoyed Francis's words, even though they lacked the casual sense of understatement known to be the correct way to behave in Cambridge. (Watson, 1968, p. 198)

1

T h i s i s the w a y J a m e s D . W a t s o n d e s c r i b e s his and F r a n c i s C r i c k ' s s e a r c h for the structure of the D N A m o l e c u l e .

The Double Helix, h i s account of their w o r k ,

gives a good picture of the excitement of r e s e a r c h . It is m o r e gripping than most mystery novels. A l t h o u g h r e s e a r c h c a n be e x c i t i n g in this w a y , the s a d fact is that w r i t i n g papers for c o u r s e s is too often something of a drag. F i r s t of a l l , c o u r s e papers are tied to all sorts of r e w a r d s and p u n i s h m e n t s — y o u r future e a r n i n g s , the approval of others, and s o o n . A l l o f the anxiety a s s o c i a t e d w i t h these vulnerabilities c o m e s , indirectly, to lodge on the paper. Y e t this is probably a l e s s e r c a u s e for frustration in student r e s e a r c h . A f t e r a l l , e a c h of these anxieties m a y a l s o be present for 'Reprinted with permission from Watson, James D. The Double Helix (New York: Atheneum, 1968).

1

2

Chapter

1

3

Doing Research

p r o f e s s i o n a l s c h o l a r s . A more important r e a s o n for the student's l a c k of e n t h u s i a s m

c o n f l i c t s o v e r l a p , there w i l l be m o r e than two distinct p o l i t i c a l positions in the c o u n -

is the s i m p l e fact that a paper is g e n e r a l l y regarded, by both teacher and student, as a

try. F o r e x a m p l e , a c o n f l i c t b e t w e e n w o r k e r s and the m i d d l e c l a s s might o c c u r at the

practice r u n , going through the motions of s c h o l a r s h i p . U s u a l l y , not e n o u g h time is

s a m e time as a c o n f l i c t b e t w e e n C a t h o l i c s and n o n - C a t h o l i c s . T h e n , if these groups

a l l o w e d for the student to think l o n g and s e r i o u s l y about the subject, e s p e c i a l l y w i t h

overlapped s o that s o m e o f the C a t h o l i c s w e r e w o r k e r s and s o m e w e r e m i d d l e c l a s s ,

other papers c o m p e t i n g for attention. A n d e v e n w h e n adequate time is a l l o w e d , there

w h i l e s o m e o f the n o n - C a t h o l i c s w e r e w o r k e r s and s o m e w e r e m i d d l e c l a s s , there

u s u a l l y is a feeling on both sides that this is " j u s t a student p a p e r " — t h a t it d o e s n ' t

w o u l d be four distinct political positions in the country:

r e a l l y matter h o w g o o d it i s , that a student w i l l learn f r o m d o i n g the thing w r o n g .

position, the n o n - C a t h o l i c w o r k e r p o s i t i o n , the C a t h o l i c m i d d l e - c l a s s p o s i t i o n , and

Students must have the c h a n c e to learn f r o m their o w n m i s t a k e s , but this attitude

the n o n - C a t h o l i c m i d d l e - c l a s s position. T h e appropriate n u m b e r o f parties w o u l d

toward the r e s e a r c h w o r k cheats them of the p l e a s u r e and excitement that r e s e a r c h

then tend to r i s e , w i t h one party c o r r e s p o n d i n g to e a c h distinct p o s i t i o n .

c a n b r i n g , of the f e e l i n g of creating s o m e t h i n g that no one ever s a w before.

the C a t h o l i c w o r k e r

H o w e v e r , D u v e r g e r thought that this tendency c o u l d be s h o r t - c i r c u i t e d if the

T h e r e is probably no w a y out of this d i l e m m a . In a b o o k s u c h as this, I cannot

electoral s y s t e m w e r e set up in s u c h a w a y as to p e n a l i z e s m a l l p a r t i e s — b y requiring

give y o u the d r a m a and excitement of o r i g i n a l r e s e a r c h . I c a n o n l y give my o w n

that a candidate have a majority, rather than a plurality, of votes in a district, for

testimony, as one for w h o m r e s e a r c h is v e r y e x c i t i n g . B u t I c a n introduce y o u to

instance. T h i s requirement w o u l d f o r c e s o m e of the distinct groups to c o m p r o m i s e

s o m e selected p r o b l e m s y o u s h o u l d be a w a r e of if y o u want to do good r e s e a r c h

their positions and merge into larger parties that w o u l d h a v e a better c h a n c e of

y o u r s e l f or to evaluate the w o r k of others. I a l s o hope to m a k e y o u a w a r e of w h a t a

w i n n i n g e l e c t i o n s . S u c h a p r o c e s s of c o n s o l i d a t i o n l o g i c a l l y w o u l d c u l m i n a t e in a

c h a l l e n g i n g g a m e it c a n be, and of h o w important i n v e n t i v e n e s s , originality, and

two-party s y s t e m . To s u m m a r i z e the theory: A country w i l l develop a two-party

b o l d n e s s are to g o o d r e s e a r c h .

s y s t e m (1) if there are only two distinct p o l i t i c a l positions in the country, or (2) if despite the p r e s e n c e of m o r e than two distinct p o l i t i c a l p o s i t i o n s , the electoral l a w forces people of d i v e r s e positions to c o n s o l i d a t e into two large political parties so as

SOCIAL RESEARCH

to gain an electoral advantage.

S o c i a l r e s e a r c h is an attempt by s o c i a l scientists to develop and sharpen theories that

taneously with a great number of idiosyncratic party systems. He needed to think only

give us a handle on the universe. R e a l i t y unrefined by theory is too c h a o t i c for us to

about a single developmental p r o c e s s , of w h i c h t i l those party systems were examples.

H a v i n g formulated this theory, D u v e r g e r no longer h a d to concern h i m s e l f s i m u l -

absorb. S o m e people vote and others do not; in s o m e elections there are m a j o r shifts,

S o m e t h i n g i s a l w a y s lost w h e n w e s i m p l i f y reality i n this way. B y restricting

in others there are not; s o m e b i l l s are p a s s e d by C o n g r e s s , others are not; e c o n o m i c

his attention to the

development p r o g r a m s s u c c e e d i n s o m e c o u n t r i e s , but fail i n others; s o m e t i m e s w a r

D u v e r g e r h a d to forget about m a n y other potentially interesting things, s u c h as

c o m e s , s o m e t i m e s it does not. To have any hope of controlling w h a t h a p p e n s , we

whether any one of the parties w a s revolutionary, or h o w m a n y of the parties h a d any

m u s t understand w h y these things h a p p e n . A n d t o have any h o p e o f understanding

c h a n c e of getting a majority of the votes.

w h y they h a p p e n , w e must s i m p l i f y our perceptions o f reality. S o c i a l scientists c a r r y out this s i m p l i f i c a t i o n by d e v e l o p i n g theories. A theory

n u m b e r of parties c o m p e t i n g in the s y s t e m ,

for e x a m p l e ,

Note, too, that D u v e r g e r restricted h i m s e l f in more than j u s t his c h o i c e of a theme: He c h o s e deliberately to play d o w n exceptions to his theory, although these

takes a set of s i m i l a r things that h a p p e n — s a y , the development of party s y s t e m s in

exceptions might have provided interesting additional information.

d e m o c r a c i e s — a n d finds a c o m m o n pattern a m o n g them that a l l o w s us to treat e a c h

instance, that a country for w h i c h his theory had predicted a two-party s y s t e m devel-

S u p p o s e , for

of these different o c c u r r e n c e s as a repeated e x a m p l e of the s a m e thing. Instead of

oped a multiparty s y s t e m instead. W h y w a s this s o ? D u v e r g e r might have cast around

h a v i n g to think about a large n u m b e r of disparate h a p p e n i n g s , we n e e d only think of

to find an explanation for the exception to his theory, and he c o u l d have then incorpo-

a single pattern w i t h s o m e variations.

rated that explanation into the original theory to produce a larger theory. Instead, w h e n

F o r e x a m p l e , i n his book o n political parties, M a u r i c e D u v e r g e r w a s c o n c e r n e d

faced with exceptions s u c h as these, he c h o s e to accept them as accidents. It w a s

w i t h the question of w h y s o m e countries develop two-party s y s t e m s and others

necessary for h i m to do this in order to keep the theory simple and to the point.

develop multiparty s y s t e m s ( 1 9 6 3 , pp. 2 0 6 - 2 8 0 ) . T h e initial reality w a s c h a o t i c ;

O t h e r w i s e , it might have grown as c o m p l e x as the reality that it sought to simplify.

s c o r e s o f countries w e r e i n v o l v e d , w i t h v a r y i n g n u m b e r s and types o f parties present

As y o u c a n see, there are costs in setting up a theory. B e c a u s e the theory

at different times in their histories. D u v e r g e r d e v i s e d the theory that (1) if s o c i a l

s i m p l i f i e s reality for u s , it a l s o generally requires that we both narrow the range of

c o n f l i c t s o v e r l a p , and (2) if the electoral s y s t e m of the country does not p e n a l i z e

reality we l o o k at and o v e r s i m p l i f y e v e n the portion of reality that falls w i t h i n that

s m a l l parties, the country w i l l develop a multiparty s y s t e m ; o t h e r w i s e , the country

n a r r o w e d range. As theorists, we a l w a y s have to strike a b a l a n c e b e t w e e n the

w i l l develop a two-party s y s t e m . H i s i d e a w a s that w h e r e there is m o r e than one sort of political c o n f l i c t going on s i m u l t a n e o u s l y in a country, and w h e r e the groups of people i n v o l v e d in these

s i m p l i c i t y of a theory and the n u m b e r of exceptions we are w i l l i n g to tolerate. We do not r e a l l y h a v e any c h o i c e . W i t h o u t theories, we are f a c e d w i t h the unreadable c h a o s of reality.

4

Chapter

1

Doing Research

A c t u a l l y , w h a t s o c i a l scientists do in developing theories is not different f r o m w h a t we n o r m a l l y do every day in perceiving, or interpreting, our environment. S o c i a l

is

John

Stuart

Mill's

argument

in

Considerations

on

Representative

5 Government,

in w h i c h he urges the adoption of d e m o c r a t i c representative g o v e r n m e n t b e c a u s e

scientists m e r e l y interpret reality in a m o r e s y s t e m a t i c a n d explicit way. Without

(1) the c h i e f end of g o v e r n m e n t s h o u l d be to facilitate the d e v e l o p m e n t in e a c h

theories, students of society are trapped. T h e y are r e d u c e d to m e r e l y observing

citizen of h i s full potential ( m o r a l a r g u m e n t ) , a n d (2) d e m o c r a t i c government, by

events, without c o m m e n t . I m a g i n e a p h y s i c i s t — o r a fruit p i c k e r for that m a t t e r —

giving the people responsibility, w i l l do this (factual a s s u m p t i o n ) .

operating in the a b s e n c e of theory. A l l she c o u l d do if she s a w an apple falling f r o m a

Like

normative

philosophy,

engineering

research

is

geared

to

s o l v i n g prob-

l e m s . H o w e v e r , its stance is e m p i r i c a l ; it is c o n c e r n e d w i t h a s c e r t a i n i n g the facts

tree w o u l d be to d u c k , a n d she w o u l d not even k n o w w h i c h w a y to m o v e . S o c i a l theory, then, is the s u m total of a l l those theories d e v e l o p e d by s o c i a l

needed to s o l v e p o l i t i c a l p r o b l e m s . S o m e e x a m p l e s w o u l d be m e a s u r i n g the effects

scientists to e x p l a i n h u m a n behavior. P o l i t i c a l theory, a subset of s o c i a l theory, c o n -

of various reapportionment m e t h o d s , trying to d e s i g n a d i p l o m a t i c strategy to effect

sists o f a l l theories that have been d e v e l o p e d to e x p l a i n political behavior.

d i s a r m a m e n t p r o c e d u r e s , a n d d e s i g n i n g methods o f riot c o n t r o l . T h e s e two forms of applied r e s e a r c h exist in s o m e estrangement f r o m a c a d e m i c political s c i e n c e . Political engineering is a thriving industry a n d m a n y courses relevant

Types of Political R e s e a r c h

to it are taught in political s c i e n c e departments, but research in it is often relegated to a

T h e w a y a particular p o l i t i c a l scientist c o n d u c t s r e s e a r c h w i l l depend both o n the

separate institute or " s c h o o l of public policy." Normative philosophy is taught exten-

u s e s that she v i s u a l i z e s for the project a n d on the w a y she m a r s h a l s e v i d e n c e .

sively, and r e s e a r c h is carried on under that n a m e , but generally this m e a n s the history

R e s e a r c h m a y be c l a s s i f i e d a c c o r d i n g to these two criteria.

of normative philosophy a n d its development, not the active formulation of normative

T h e two m a i n w a y s b y w h i c h t o d i s t i n g u i s h o n e p i e c e o f r e s e a r c h f r o m

arguments. F o r both forms of applied r e s e a r c h , we must look largely outside a c a d e m i c life to such sources as the R A N D C o r p o r a t i o n a n d the Weekly Standard.

another are:

At the other e n d of the c o n t i n u u m f r o m applied r e s e a r c h is recreational 1. Research may be directed toward providing the answer to a particular problem, or it

r e s e a r c h . It is u s u a l l y c a l l e d " p u r e " or " b a s i c " r e s e a r c h , but this c a r r i e s the u n p l e a s -

may be carried on largely for its own sake, to add to our general understanding of poli-

ant implication that applied r e s e a r c h is either i m p u r e or of l i m i t e d v a l u e . T h i s type of

tics. This distinction, based on the uses for which research is designed, may be thought

research is r e a l l y not as flippant as the c h o i c e of the term recreational might m a k e it

of as applied versus basic research.

s e e m , for this is r e s e a r c h c a r r i e d on for its o w n s a k e , to i m p r o v e p o l i t i c a l theory.

2. Research may also be intended primarily to discover new facts, or it may be intended to provide new ways of looking at old facts. Thus, political research can be characterized

by the extent to which it seeks to provide new factual information (empirical versus nonempirical).

P o l i t i c a l scientists pursue this type of r e s e a r c h for the t w i n p l e a s u r e s of e x e r c i s i n g their m i n d s a n d i n c r e a s i n g their understanding of things. In a h i g h s e n s e of the w o r d , it is "recreation." Formal theory, l a r g e l y a p o s t - W o r l d W a r II p h e n o m e n o n , is the m o s t r e c e n t l y

A g l a n c e at T a b l e 1-1 s h o w s us the four types of p o l i t i c a l r e s e a r c h b a s e d on different

combinations

of these

two

dimensions.

arguments about what should be in p o l i t i c s .

Normative philosophy c o n s i s t s

of

P r o b a b l y the oldest f o r m of p o l i t i c a l

r e s e a r c h , i t i n c l u d e s a m o n g its practitioners P l a t o , K a r l M a r x , A y n R a n d , P a u l K r u g m a n , G e o r g e W i l l , a n d others. It is a p p l i e d r e s e a r c h ; that i s , its goal is p r o b l e m s o l v i n g . T h i s m e a n s that it is not intended so m u c h to develop p o l i t i c a l theory as to u s e what political theory tells us about society and p o l i t i c s as a b a s i s for m a k i n g p o l i t i c a l d e c i s i o n s . It is a l s o n o n e m p i r i c a l in that it does not c o n s i s t p r i m a r i l y of investigating matters of fact. It t y p i c a l l y takes certain p o l i t i c a l facts as given and c o m b i n e s t h e m w i t h m o r a l arguments to p r e s c r i b e p o l i t i c a l action. A good e x a m p l e

introduced f o r m o f political r e s e a r c h . L i k e normative p h i l o s o p h e r s , f o r m a l theorists posit certain facts about p o l i t i c s ; but in contrast to normative p h i l o s o p h e r s , they posit facts as e m p i r i c a l conditions rather than as the foundation for m o r a l arguments. A n d they distinctively operate by d e r i v i n g further i m p l i c a t i o n s of the posited conditions by p r e c i s e l o g i c a l and m a t h e m a t i c a l operations. T h e i r c o n c e r n is to take the posited facts, or a s s u m p t i o n s , a n d derive theories f r o m them. T h e i r e n d goal is to develop reasonably b r o a d a n d general theories b a s e d on a s m a l l n u m b e r of agreed-upon assumptions. A g o o d e x a m p l e of f o r m a l t h e o r y — i n d e e d , a w o r k by w h i c h m a n y w o u l d date the e m e r g e n c e of f o r m a l theory as a distinct field in p o l i t i c a l s c i e n c e — i s A n t h o n y Downs'

T A B L E 1-1

An Economic

Theory

of Democracy

(1957).

Downs

builds

a wide-ranging

theory f r o m a set of a s s u m p t i o n s s u c h as: (1) voters a n d parties b e h a v e rationally;

Types of Political R e s e a r c h

(2) p o l i t i c a l c o n f l i c t o c c u r s on o n l y one i s s u e at a t i m e ; a n d (3) p o l i t i c a l events are Applied

Recreational

Nonempirical

Normative philosophy

Formal theory

Empirical

Engineering research

Theory-oriented research

not perfectly predictable. S o m e of the predictions generated f r o m h i s theory are (1) in a two-party s y s t e m , parties w i l l tend to agree very c l o s e l y on i s s u e s , w h e r e a s in a multiparty s y s t e m , they w i l l not; (2) it m a y be rational for the voter to r e m a i n uninformed;

and

(3)

democratic

governments

tend

to

redistribute

income.

6

Doing Research

Chapter 1

7

( O f c o u r s e , one m u s t r e c o g n i z e that excerpts s u c h as these do e v e n m o r e than

interests existed i n s o c i e t y — r a c i a l m i n o r i t i e s , b u s i n e s s e s , p r o f e s s i o n s , groups w i t h

the u s u a l v i o l e n c e to a r i c h net of theories.) It is important to e m p h a s i z e that this sort

special c o n c e r n s s u c h as historic p r e s e r v a t i o n — p o l i t i c a l organizations c o u l d be

o f w o r k i s a l m o s t s o l e l y a n e x e r c i s e i n deduction. A l l o f the c o n c l u s i o n s derive

expected to emerge naturally to represent those i n t e r e s t s . We s h o u l d thus expect to

2

l o g i c a l l y f r o m a limited set of e x p l i c i t a s s u m p t i o n s . D o w n s ' p u r p o s e in this is s i m p l y

see a w i d e range of parties and interest groups engaged in p o l i t i c s . A w h o l e s c h o o l of

to see w h e r e the a s s u m p t i o n s he started w i t h w i l l lead h i m . P r e s u m a b l y , if the

political s c i e n c e , the

a s s u m p t i o n s p r o d u c e d an untenable result, he w o u l d go b a c k and r e e x a m i n e them.

most of the time, m o s t of s o c i e t y ' s interests w o u l d be actively o r g a n i z e d .

pluralist school,

w a s organized ar ound the expectation that

T h e m a i n u s e of f o r m a l theory, as in the e x a m p l e above, is explanation; a

B a s e d on the rational c h o i c e a s s u m p t i o n , h o w e v e r , O l s o n r e a s o n e d that there

f o r m a l theory is u s e d to construct a set of conditions f r o m w h i c h the thing we w i s h

w a s nothing natural about organization at a l l . F r o m the standpoint of any i n d i v i d u a l

to e x p l a i n w o u l d have l o g i c a l l y f l o w e d . S u c h explanatory f o r m a l theories are then

in a group w i t h a shared object, he c o n c l u d e d , participation in the group is u s u a l l y

often tested e m p i r i c a l l y through theory-oriented r e s e a r c h . B u t b e c a u s e f o r m a l theory

nonrational. R e m e m b e r that the rational c h o i c e a s s u m p t i o n states that i n d i v i d u a l s

consists of taking a set of a s s u m p t i o n s and w o r k i n g out w h e r e they l e a d — t h a t i s ,

choose their actions in order to m a x i m i z e a v a l u e d object, w h i l e m i n i m i z i n g the c o s t

what they l o g i c a l l y i m p l y — i t is a l s o u s e f u l for d e v e l o p i n g and a n a l y z i n g strategies

expended in a c h i e v i n g it. If I am a p e r s o n c o n c e r n e d w i t h h i s t o r i c preservation,

for p o l i t i c a l action. T h a t i s , we c a n u s e f o r m a l theory to construct a n a l y s e s of the

I k n o w that u n l e s s I have v e r y u n u s u a l r e s o u r c e s , my i n d i v i d u a l contribution to an

f o l l o w i n g f o r m : If we want to a c h i e v e X, c a n we d e v i s e a set of r e a s o n a b l y true

interest group p u r s u i n g preservation w i l l not m a k e a m e a s u r a b l e difference. L e t us

a s s u m p t i o n s and an action w h i c h , in the context of those a s s u m p t i o n s , w i l l l o g i c a l l y

say there are 3 0 0 , 0 0 0 people around the country w h o share my interest; if e a c h of us

lead to XI F o r m a l theory is u s e d in this w a y , for e x a m p l e , to argue for v a r i o u s w a y s

contributes $ 100 to the c a u s e , the difference if I do or do not contribute is a budget of

to set up e l e c t i o n s , or for v a r i o u s w a y s to arrange taxes so as to get the o u t c o m e s we

$ 2 9 , 9 9 9 , 9 0 0 v e r s u s $ 3 0 , 0 0 0 , 0 0 0 . To the organization this a m o u n t w o u l d be trivially

want. F l a t - t a x p r o p o s a l s are a g o o d e x a m p l e : T h e y originated in argument of the

s m a l l , but to me $ 1 0 0 m a k e s a real difference. If I contribute, I w i l l h a v e e x p e n d e d a

f o l l o w i n g f o r m : (a) I f w e want t o m a x i m i z e investment and e c o n o m i c growth, and

significant cost without getting any m o r e of my v a l u e d good, w h i c h is not rational.

(b) if we a s s u m e that governmental investment is inefficient and that i n d i v i d u a l

W h a t is rational, instead, is to be

taxpayers act so as to m a x i m i z e their i n c o m e , then ( c ) c a n we d e d u c e what sort of

contributions. H o w e v e r , O l s o n pointed out, s i n c e every potential m e m b e r of s u c h an

taxes in the context of the a s s u m p t i o n s of (b) w o u l d best a c h i e v e ( a ) ?

organization is in this s a m e situation, the m a r v e l s h o u l d be that any interest or gani -

L i k e normative p h i l o s o p h y , f o r m a l theory interacts w i t h e m p i r i c a l r e s e a r c h .

a free rider, and let all those other people m a k e the

zations exist at a l l .

F o r m a l theorists u s u a l l y try to start w i t h a s s u m p t i o n s that are in a c c o r d w i t h existing

O l s o n laid out several conditions under w i i i c h organizations might nonetheless

k n o w l e d g e about p o l i t i c s , and at the end they m a y c o m p a r e their final m o d e l s w i t h

arise. O n e s u c h condition is that one potential m e m b e r might h a v e s u c h large

this body of k n o w l e d g e . B u t they are not t h e m s e l v e s c o n c e r n e d w i t h turning up n e w

resources that she k n o w s no organization is p o s s i b l e without her participation. T h e

factual i n f o r m a t i o n .

largest department store in town, for i n s t a n c e , k n o w s that a D o w n t o w n M e r c h a n t s '

G o o d work in formal theory w i l l take a set of seemingly reasonable assumptions

A s s o c i a t i o n cannot function i f i t does not j o i n and contribute. T h e B i j o u x T e e - S h i r t

and w i l l show by logical deduction that those assumptions lead inescapably to c o n -

S h o p on the corner, though, is not in that situation. U n d e r these c i r c u m s t a n c e s , we

clusions that surprise the reader. T h e reader must then either accept the surprising

c a n count on an organization b e i n g set up, b e c a u s e rationally, the large store cannot

conclusion or reexamine the assumptions that had seemed plausible. T h u s , formal theory

get its v a l u e d g o o d u n l e s s it takes the lead in setting up the a s s o c i a t i o n .

provides insights by logical argument, not by a direct examination of political facts. F o l l o w i n g f r o m D o w n s , a great deal of f o r m a l theory in political s c i e n c e has

No theory c a n ever be a l l - e n c o m p a s s i n g , and in fact one function of theory m a y be that it highlights exceptions for c l o s e r e x a m i n a t i o n . We k n o w that m a n y

the a s s u m p t i o n

people do contribute to political organizations even though, as O l s o n has p r o v e d , it is

that i n d i v i d u a l s c h o o s e their actions in order to m a x i m i z e s o m e v a l u e d object, and

irrational for t h e m to do s o . T h e virtue of O l s o n ' s theory in this c a s e is that instead of

m i n i m i z e the cost e x p e n d e d in a c h i e v i n g it. ( I n e c o n o m i c s the v a l u e d object is gen-

v i e w i n g s u c h contributions as " n a t u r a l " and therefore i gnor i ng t h e m , we are f o r c e d

e r a l l y taken to be m o n e y ; in political s c i e n c e it m a y be m o n e y — a s in theories of w h y

to treat the contributions as a p u z z l e requiring further investigation.

b a s e d itself on the e c o n o m i s t s ' c o r e a s s u m p t i o n of

rational choice:

and h o w c o m m u n i t i e s s e e k pork-barrel s p e n d i n g — b u t theories m a y also posit that

However, in a w i d e array of settings O l s o n ' s theory predicts behavior rather w e l l .

the v a l u e d object is a nonmonetary p o l i c y s u c h as abortion, or p o l i t i c a l p o w e r itself.

T h e excruciating efforts of public television stations to get their viewers to j o i n rather

S o m e t i m e s the object m a y e v e n be left u n s p e c i f i e d in the theory.)

than be free riders ("Please! O n l y one in ten of our viewers is a member. If y o u j o i n

A good e x a m p l e of f o r m a l theory that illustrates the rational c h o i c e a s s u m p -

K X X X - T V today we w i l l send y o u this beautiful coffee m u g ! " ) bears testimony to the

The Logic of Collective Action ( 1 9 6 5 ) . T h e rational c h o i c e

power of O l s o n ' s logic. In the next chapter y o u w i l l see that it m a y also help to explain

tion is M a n c u r O l s o n ' s

a s s u m p t i o n pointed O l s o n to a question no one h a d a s k e d before, and a l l o w e d h i m to

w h y s m a l l nations typically do not pull their "fair" weight in international alliances.

stand r e c e i v e d w i s d o m o n its h e a d . O l s o n wrote o n the v e r y b a s i c question o f 2

political organization in society. B e f o r e his b o o k , s c h o l a r s had a s s u m e d that w h e n

For example, Duverger (1963) assumed this in the theory I described on pp. 2 - 3 .

8

Chapter

1

Doing Research

9

A l t h o u g h f o r m a l theory is the fastest g r o w i n g type of political r e s e a r c h , most

p e r s o n a l i n v e s t i g a t i o n o f this f a c t u a l b a s i s . S u c h a c t i v i t y w i l l , o f c o u r s e , i n v o l v e

r e s e a r c h and teaching in p o l i t i c a l s c i e n c e is still of the fourth type suggested in

h i m t o s o m e degree i n e n g i n e e r i n g r e s e a r c h . I t i s c h a r a c t e r i s t i c o f n o r m a t i v e

Table

with

p h i l o s o p h y , h o w e v e r , that the r e s e a r c h e r n e e d not f e e l required to p r o d u c e the f u l l

e x p a n d i n g our k n o w l e d g e of w h a t happens in p o l i t i c s and of w h y it happens as it

factual b a s i s for h i s argument. I n this r e s p e c t n o r m a t i v e p h i l o s o p h y differs f r o m

does. L i k e p o l i t i c a l e n g i n e e r i n g , it is e m p i r i c a l ; it is c o n c e r n e d w i t h d i s c o v e r i n g facts

the e m p i r i c a l types o f p o l i t i c a l r e s e a r c h .

1-1,

theory-oriented

research.

This

type

of

research

is

concerned

about p o l i t i c s . B u t u n l i k e e n g i n e e r i n g , w h i c h deals w i t h facts o n l y for their u s e f u l -

T h e distinction is an important one. F o r one thing, the fact that normative

n e s s in s p e c i f i c p o l i t i c a l p r o b l e m s , this r e s e a r c h deals w i t h them to develop n e w

p h i l o s o p h e r s are not required to p r o v i d e e v i d e n c e for all their a s s u m p t i o n s l e a v e s

p o l i t i c a l theories or to c h a n g e or c o n f i r m old ones. A c c o r d i n g l y , the m o s t important

t h e m free to devote m o r e energy to other parts of the r e s e a r c h task. M o r e important,

activity in this r e s e a r c h is the development of theories l i n k i n g o b s e r v e d facts about

they often n e e d to a s s u m e facts that cannot p o s s i b l y be tested against reality. T h e

politics. In e n g i n e e r i n g , facts are sought out if they are n e e d e d to s o l v e a p r o b l e m ;

normative p h i l o s o p h e r m u s t be free to i m a g i n e realities that have never existed

here they are sought out if they w i l l be u s e f u l in d e v e l o p i n g theories.

before, and these, o f c o u r s e , cannot b e "tested." I f normative p h i l o s o p h e r s w e r e h e l d

D u v e r g e r ' s study of p o l i t i c a l parties is an e x a m p l e of theory-oriented r e s e a r c h . A n o t h e r g o o d e x a m p l e is a test by D i e h l and K i n g s t o n ( 1 9 8 7 ) of the theory that arms

to the s a m e standards of factual e v i d e n c e as e m p i r i c a l r e s e a r c h e r s , all U t o p i a n dreams w o u l d h a v e to be thrown out.

buildups l e a d t o m i l i t a r y confrontations. T h e y e x a m i n e d c h a n g e s i n military e x p e n -

A s a s e c o n d e x a m p l e o f the w a y i n w h i c h types o f r e s e a r c h are m i x e d i n any

diture by m a j o r p o w e r s f r o m 1816 to 1976 to see w h e t h e r i n c r e a s e s tended to be

one w o r k , let us l o o k at a c a s e in w h i c h r e s e a r c h e r s w o r k i n g on a p r i m a r i l y e n g i -

f o l l o w e d b y i n v o l v e m e n t i n war. N o matter h o w they adjusted t h i n g s — l o o k i n g for

neering project f o u n d they h a d to develop a theory to m a k e s e n s e out of their w o r k .

d e l a y e d r e a c t i o n s , l o o k i n g only a t " a r m s r a c e s " i n w h i c h two rivals s i m u l t a n e o u s l y

A group of s o c i o l o g i s t s l e d by S a m u e l Stouffer w a s e m p l o y e d by the A r m y to study

i n c r e a s e d their military f o r c e s , and so o n — t h e y found no relationship. M i l i t a r y

the m o r a l e o f A m e r i c a n soldiers during W o r l d W a r I I (Stouffer and others, 1949).

engagements w e r e no m o r e or l e s s l i k e l y to o c c u r f o l l o w i n g military buildups than

Stouffer and his c o w o r k e r s w e r e p u z z l e d by the fact that often a s o l d i e r ' s m o r a l e h a d

under other c i r c u m s t a n c e s . T h e y c o n c l u d e d by e x p l o r i n g the i m p l i c a t i o n s of this

little to do w i t h h i s objective situation.

finding, one of w h i c h is that a r m s expenditure m u s t therefore be determined m o r e by d o m e s t i c p o l i t i c a l c o n s i d e r a t i o n s than by the international situation.

F o r instance, M P s were objectively less likely to be promoted than were members o f the A r m y A i r C o r p s . O f Stouffer's sample o f M P s , 2 4 percent were n o n c o m m i s sioned officers ( N C O s ) , c o m p a r e d with 47 percent of the air c o r p s m e n . Paradoxically, however, the M P s w e r e m u c h more likely than the air c o r p s m e n to think that soldiers

Research

Mix

with ability h a d a good chance to advance in the A r m y . T h i s sort of paradox occurred a

P r a c t i c a l l y no r e s e a r c h is a pure e x a m p l e of any of the types I h a v e presented here.

number of times in their study, and the researchers felt they had to m a k e some sense of

T h e s e are abstract distinctions, types o f e m p h a s i s found i n particular p i e c e s o f

it if their efforts were to help the A r m y improve morale.

r e s e a r c h . G e n e r a l l y , any s p e c i f i c p i e c e of w o r k is a m i x of m o r e than one of the

T h e y d i d this by d e v e l o p i n g the theory of relative deprivation to a c c o u n t for

types. A l t h o u g h one m e t h o d w i l l u s u a l l y predominate, there w i l l a l m o s t a l w a y s b e

their s e e m i n g l y contradictory findings. A c c o r d i n g to this theory, satisfaction w i t h

s o m e interaction b e t w e e n the different types i n any g i v e n w o r k . T w o e x a m p l e s m a y

o n e ' s c o n d i t i o n is not a function of h o w w e l l - o f f a p e r s o n is objectively, but of whether her c o n d i t i o n c o m p a r e s favorably or unfavorably w i t h a standard that she

help illustrate this point. F i r s t , let us l o o k a bit m o r e c l o s e l y at normative p h i l o s o p h y , u s i n g K a r l M a r x ' s

p e r c e i v e s as n o r m a l .

w o r k as an e x a m p l e . M a r x ' s theory of the d i a l e c t i c is p r i m a r i l y a w o r k in normative

T h e fact that s o m a n y air c o r p s m e n w e r e N C O s apparently m a d e the c o r p s m e n

philosophy. H i s argument takes the s a m e general f o r m as that in M i l l ' s e s s a y on rep-

feel that p r o m o t i o n w a s the n o r m a l thing. T h o s e w h o w e r e not p r o m o t e d w e r e d i s a p -

resentative government: " B e c a u s e

pointed, and those w h o w e r e p r o m o t e d d i d not feel particularly h o n o r e d . A m o n g the

aspects of the h u m a n condition today are

b a d , and b e c a u s e the state and the e c o n o m y function in

w a y s to p r o d u c e these

b a d effects, we s h o u l d strive to c h a n g e the state and the e c o n o m y in

ways,

M P s , on the other h a n d , promotion w a s sufficiently infrequent that not being promoted w a s seen as the n o r m . T h o s e w h o w e r e not p r o m o t e d w e r e not disappointed,

w h i c h w i l l eliminate the b a d effects." B u t M a r x w a s less w i l l i n g than M i l l t o s i m p l y

and those w h o w e r e promoted felt h o n o r e d . T h u s , p a r a d o x i c a l l y , the air c o r p s m e n ,

assume the factual portions of his argument. I n s t e a d , he spent y e a r s of r e s e a r c h

w h o w e r e m o r e l i k e l y to be p r o m o t e d , felt that c h a n c e s for p r o m o t i o n in the A r m y

trying to w o r k out the p r e c i s e e c o n o m i c effects of c a p i t a l i s m .

w e r e poor, and the M P s , w h o w e r e l e s s l i k e l y to be p r o m o t e d , felt that c h a n c e s for

It s h o u l d be e v i d e n t that a n y o n e d e v e l o p i n g n o r m a t i v e theories about p o l i t i c s

promotion i n the A r m y w e r e g o o d !

must begin with some factual assumptions. A researcher m a y be relatively more

I h a v e m e n t i o n e d these two e x a m p l e s to illustrate my point that m o s t r e s e a r c h

w i l l i n g t o a s s u m e these facts f r o m g e n e r a l e x p e r i e n c e a n d / o r f r o m the r e s e a r c h

w o r k i n v o l v e s s o m e m i x of the four types of r e s e a r c h . I n d e e d , a m i x is so m u c h the

o f others, a s M i l l w a s ; o n the other h a n d , h e m a y w i s h , l i k e M a r x , t o c o n d u c t a

u s u a l situation that w h e n I tried to m a k e a r o u g h h e a d count of the f r e q u e n c y of the

10

Chapter 1

Doing Research

different types of r e s e a r c h in p o l i t i c a l s c i e n c e j o u r n a l s , I w a s u n a b l e to do s o . I w a s

11

engineering. T h e r e are two r e a s o n s for this: (1) It is the m o r e c o m m o n k i n d of

s i m p l y u n w i l l i n g to a s s i g n most articles to one or another of the categories. It is j u s t

research in p o l i t i c a l s c i e n c e , and (2) it poses rather m o r e difficult instructional

not often the c a s e that a r e s e a r c h e r c a n e a s i l y be labeled a normative philosopher, an

problems than does e n g i n e e r i n g .

engineer, a f o r m a l theorist, or a theory-oriented e m p i r i c a l researcher. T h e s e types interact in the w o r k of every political scientist. T h a t most r e s e a r c h i n v o l v e s a m i x of the types does not p r e c l u d e the i m p o r -

ETHICS OF POLITICAL RESEARCH

tance of the d i s t i n c t i o n s , however. G e n e r a l l y , one type of r e s e a r c h is dominant in any given p i e c e of w o r k , depending on the goals of the researcher. T h e s e goals have a lot

C o n d u c t i n g r e s e a r c h is an act by y o u . Y o u m u s t therefore be c o n c e r n e d about the

to do w i t h the w a y a study s h o u l d be set up and the criteria a c c o r d i n g to w h i c h it

ethics of y o u r r e s e a r c h , j u s t as y o u are w i t h all of y o u r a c t i o n s . T h e r e are two broad

should be judged.

c l a s s e s o f e t h i c a l questions regarding our r e s e a r c h . F i r s t , w e m u s t c o n c e r n o u r s e l v e s w i t h the effects on society of what we discover. F o r i n s t a n c e , if y o u study t e c h n i q u e s of political p e r s u a s i o n , it is p o s s i b l e that w h a t y o u learn c o u l d be u s e d by a political

Evaluating Different Types of Research

charlatan to do b a d things. A c o l l e a g u e o n c e p u b l i s h e d a study of the effects of It is dangerous to set d o w n s i m p l e standards for good r e s e a r c h . L i k e any creative

electoral s y s t e m s on representation, only to learn later that it w a s u s e d by a military

w o r k , r e s e a r c h s h o u l d be evaluated subjectively, a c c o r d i n g to i n f o r m a l and rather

j u n t a in a L a t i n A m e r i c a n country to figure out h o w to p r o d u c e a controllable

flexible criteria. B u t I w i l l r i s k suggesting t w o standards for r e s e a r c h that w i l l s e r v e

"democracy."

as e x a m p l e s of the w a y in w h i c h the type (or types) of r e s e a r c h we are doing dictates the w a y we s h o u l d c o n d u c t that r e s e a r c h .

held

A l s o , the results of a r e s e a r c h c a n be d e m e a n i n g or d e h u m a n i z i n g . R e c e n t results in p s y c h o l o g y suggesting that a w i d e range of b e h a v i o r s are genetically c o n -

In the first p l a c e , in either f o r m of e m p i r i c a l r e s e a r c h , the r e s e a r c h e r s h o u l d be

trolled go against our prevailing d i s p o s i t i o n to think of h u m a n s as free agents in w h a t

responsible

they do. R e s e a r c h on r a c i a l or ethnic groups is particularly s e n s i t i v e , as we m a y fear

for demonstrating

the factual basis

of his

conclusions.

In

either

f o r m of n o n e m p i r i c a l r e s e a r c h , this is not n e c e s s a r y , although a normative argument

that innocent r e s e a r c h results might reinforce preexisting stereotypes.

m a y b e m a d e m o r e c o n v i n c i n g , o r a n e x e r c i s e i n f o r m a l theory m a y b e m a d e m o r e

E t h i c a l q u e s t i o n s o f this sort are e s p e c i a l l y d i f f i c u l t b e c a u s e the r e s u l t s o f

interesting, by p r o v i d i n g e v i d e n c e for the factual b a s i s on w h i c h its a s s u m p t i o n s rest.

our r e s e a r c h are s o h a r d t o p r e d i c t . A n o t h e r c o l l e a g u e , i n b i o l o g y , w a s upset w h e n

In the s e c o n d p l a c e , good r e s e a r c h of any sort s h o u l d be directed to an inter-

h e l e a r n e d that h i s r e s e a r c h o n f r o g s ' e y e s t u r n e d out t o h a v e a p p l i c a t i o n s i n the

on the

d e s i g n o f g u i d a n c e s y s t e m s for m i s s i l e s ! I n the c a s e o f d e m e a n i n g o r d e h u m a n i z -

motivation of the study. F o r either sort of a p p l i e d r e s e a r c h , p r o b l e m s s h o u l d be

i n g r e s e a r c h , a further p r o b l e m a r i s e s , in that w h a t s e e m s " d e h u m a n i z i n g " to a

c h o s e n w h i c h are of r e a l i m p o r t a n c e for c o n t e m p o r a r y p o l i c y . T o d a y , an argument

p e r s o n d e p e n d s o n w h a t the p e r s o n t h i n k s " h u m a n " m e a n s — t h a t i s , i t i s v e r y

esting problem.

B u t w h a t sort of p r o b l e m is

"interesting" depends largely

about c i v i l d i s o b e d i e n c e , for e x a m p l e , m a k e s a m o r e interesting p r o b l e m in n o r m a -

much

tive theory than a p r o b l e m dealing w i t h an argument about d y n a s t i c s u c c e s s i o n ; a

r e s e a r c h noted a b o v e m a y s e e m w h o l l y a p p r o p r i a t e , for i n s t a n c e , d e p e n d i n g o n

f e w h u n d r e d y e a r s ago the reverse w o u l d probably h a v e been the c a s e . In other

o n e ' s v i e w o f h u m a n n e s s . A s a n o t h e r e x a m p l e , the t h e o r y o f e v o l u t i o n a p p e a r s

w o r d s , applied r e s e a r c h s h o u l d be relevant, in the c o m m o n usage of the w o r d .

d e h u m a n i z i n g to m a n y f u n d a m e n t a l i s t C h r i s t i a n s but d o e s not a p p e a r so to m a n y

R e c r e a t i o n a l r e s e a r c h , on the other h a n d , requires p r o b l e m s that w i l l have a substantial i m p a c t on existing bodies of theory. M a n y topics that are of c o n s i d e r a b l e importance

to

an

engineer

show

little

promise

for

theory-oriented

research.

a

matter

of personal

beliefs

and

cultural

context.

The

psychological

other p e o p l e . O n e r e s p o n s e to s u c h difficulties might be to take a "pure s c i e n c e " a p p r o a c h , arguing that b e c a u s e it is so hard to j u d g e the results of k n o w l e d g e a n y w a y , we

S i m i l a r l y , m a n y p r o m i s i n g topics for recreational r e s e a r c h are not directly relevant.

s h o u l d let the c h i p s fall w h e r e they may. We s h o u l d s i m p l y s e e k truth and not w o r r y

F o r e x a m p l e , r e s e a r c h o n the difference b e t w e e n m e n ' s and w o m e n ' s voting i n

about its effects. As we w i l l see throughout this book, h o w e v e r , the s o c i a l scientist

I c e l a n d in the 1920s and 1930s w o u l d s o u n d absurd f r o m the standpoint of an

rarely deals i n u n q u e s t i o n e d truths. W e w o r k under sufficient difficulties, e s p e c i a l l y

engineer. B u t these voting patterns, o c c u r r i n g j u s t after the extension of the vote to

the fact that we u s u a l l y cannot operate by experimentation (see C h a p t e r 6 ) , that our

w o m e n , might be important for theories of h o w voting patterns b e c o m e established

results are to s o m e extent a subjective interpretation of reality. We operate w i t h i n

a m o n g new voters. H o w to c h o o s e an interesting p r o b l e m is one of the most difficult

rules of e v i d e n c e in interpreting reality, so we are c o n s t r a i n e d in w h a t we c a n assert

and c h a l l e n g i n g parts of e m p i r i c a l r e s e a r c h . I w i l l d i s c u s s this in s o m e detail in

and cannot s i m p l y p u l l findings out of a hat; but still our results i n v o l v e i n d i v i d u a l

Chapter 2.

c h o i c e s and j u d g m e n t by u s . We are not s i m p l y neutral agents of truth; we must take

I n g e n e r a l , this b o o k i s c o n c e r n e d w i t h e m p i r i c a l r e s e a r c h . W i t h i n e m p i r i c a l r e s e a r c h , I devote s o m e w h a t m o r e attention to theory-oriented r e s e a r c h than to

personal r e s p o n s i b i l i t y for the results of our r e s e a r c h , difficult though these e t h i c a l questions m a y be.

12

Chapter 1 A s e c o n d c l a s s of questions, more specific than the ones described earlier but not

necessarily

easier to

answer,

deals

with

our treatment

of the people

we

are

studying.

We are responsible to treat the subjects of our study fairly and decently. Particular problems arise in the f o l l o w i n g w a y s : 1. Harm to subjects. Harming the subjects of your study, either by doing harmful things to them or by withholding good things from them, should generally be avoided. Is it ethically right, for example, in evaluating the effects of a program to get people off the welfare rolls and into jobs, to withhold the program from some deserving people while administering it to others, in order to see how effective it is? 2. Embarrassment or psychological stress. You should avoid shaming people into participating in your study, or submitting them to embarrassing situations. 3. Imposition. You are asking your subjects to help you. Don't demand more of them than is reasonable. Public officials may get a hundred questionnaires a year; keep yours short. Dinnertime is a good time to reach people by phone, but it is also an annoying time if you have 15 minutes' worth of questions to ask. 4. Confidentiality. Generally, the subjects of your study will wish to have their privacy protected. It is not enough just to withhold publishing their names. Relevant details you include in your report might make it easy to identify the subject (a member of Congress, female, from the South, the senior member of her committee). You should

In this chapter we look more c l o s e l y at the nature of political theories and at the f a c -

take care to truly mask the subjects who have helped you.

tors that i n f l u e n c e the d e c i s i o n to do r e s e a r c h on a particular theory. A l o n g the w a y

5. Fooling or misleading the subjects. As an overall rule, you should make certain that your subjects know exactly what they will be doing and what use you will make of them. As you will see in Chapter 6, the results of your study might well be more valid if the people you study are unaware that they are being studied. However, everyone has the right not to be fooled and not to be used without his or her consent.

I w i l l d i s c u s s s o m e standards to u s e in d e c i d i n g w h e t h e r a theory is w e a k or strong. A l t h o u g h this chapter deals w i t h p o l i t i c a l theories, y o u s h o u l d not a s s u m e that it is important only for w h a t I h a v e c a l l e d theory-oriented r e s e a r c h . I n d e e d , as I pointed out in C h a p t e r 1, the k e y to s o l v i n g m a n y engineering p r o b l e m s m a y be a political theory of s o m e sort. To effect a c h a n g e in s o m e given p h e n o m e n o n , y o u m a y need to develop a theory that a c c o u n t s for several factors and a l l o w s y o u to

T h e p r o b l e m s I have noted here p o s e difficult ethical questions of the " e n d s and m e a n s " sort. If a r e s e a r c h that w i l l benefit society c a n be c o n d u c t e d only by mistreating s u b j e c t s , s h o u l d it be d o n e ? T h e r e is no c l e a r a n s w e r . If the costs to s u b j e c t s are slight ( i n c o n v e n i e n c e , p a i n of w h i c h they are i n f o r m e d in a d v a n c e ) and the social benefits great, we w o u l d g e n e r a l l y s a y y e s , it s h o u l d be done. B u t w h a t if it puts subjects in danger of death, as m a y be true of political r e s e a r c h that d e l v e s into racketeering or c o r r u p t i o n ? T h e most horrible historic e x a m p l e of s c i e n c e gone bad is that of the N a z i doctors w h o k i l l e d prisoners b y i m m e r s i n g t h e m i n i c e water t o see h o w long people c o u l d s u r v i v e in f r e e z i n g water. A p a i n f u l ethical question today is w h e t h e r e v e n to u s e the results of that r e s e a r c h , w h i c h w a s p u r c h a s e d at great h u m a n p a i n , but w h i c h m a y potentially help i n s a v i n g lives a n d — w e h o p e — w i l l never b e available again f r o m any s o u r c e . D o e s u s i n g the results o f the r e s e a r c h j u s t i f y it? I f s o , perhaps w e s h o u l d destroy the results. B u t might that not lead to greater h u m a n pain for v i c t i m s o f f r e e z i n g and exposure w h o m w e might have h e l p e d ? T h e one f i r m rule, for me at least, is that people s h o u l d never be c o e r c e d or tricked into participation and s h o u l d a l w a y s be fully i n f o r m e d before they agree to participate.

manipulate them to produce the d e s i r e d c h a n g e . M u c h a p p l i e d r e s e a r c h on the probl e m of e n r i c h i n g the education of underprivileged c h i l d r e n , for e x a m p l e , has h a d to c o n c e r n itself w i t h d e v e l o p i n g theories to e x p l a i n w h y one c h i l d learns things m o r e q u i c k l y than another. T h e Stouffer study, c i t e d in C h a p t e r 1, is another e x a m p l e of an engineering study in w h i c h it w a s n e c e s s a r y to develop a theory. In that c a s e , Stouffer and his collaborators h a d t o e x p l a i n w h y M P s h a d higher m o r a l e than air c o r p s m e n . T h i s w a s n e c e s s a r y if they w e r e to devise w a y s to raise the m o r a l e of A r m y p e r s o n n e l i n general. On the other h a n d , m a n y e n g i n e e r i n g studies do not require that a theory be developed; they s i m p l y i n v o l v e m e a s u r i n g things that need to be m e a s u r e d . T a k i n g the U . S . c e n s u s i s one e x a m p l e o f s u c h engineering r e s e a r c h . O t h e r s i n c l u d e the G a l l u p P o l l , studies m e a s u r i n g the malapportionment of state legislatures, and c o m parisons of the relative military strength of v a r i o u s countries. I n s u m , e n g i n e e r i n g r e s e a r c h m a y o r m a y not i n v o l v e the development o f political theories; theory-oriented r e s e a r c h a l w a y s does. T h e o r y is a tool in one type of r e s e a r c h ; it is an e n d in itself in the other. B u t no matter w h i c h type of r e s e a r c h one is currently e n g a g e d i n , it is w o r t h taking a c l o s e r l o o k at the nature of theory.

13

14

Chapter

2

CAUSALITY AND

Political Theories and Research Topics POLITICAL THEORY

15

W H A T D O E S G O O D T H E O R Y L O O K LIKE?

In the s o c i a l s c i e n c e s , theories g e n e r a l l y are stated in a c a u s a l m o d e : " I f X h a p p e n s ,

T h r e e things are important if we are to develop g o o d , effective theories:

then Y w i l l f o l l o w as a result." T h e e x a m p l e s we l o o k e d at in C h a p t e r 1 w e r e all of this f o r m . In the D u v e r g e r e x a m p l e , if a certain configuration of political c o n f l i c t s

1. Simplicity.

A theory should give us as simple a handle on the universe as possible.

e x i s t s , a n d if the country adopts a certain electoral law, then the n u m b e r of political

It should use no more than a few independent variables. It would not be very useful to

parties in the c o u n t r y c a n be e x p e c t e d to grow or s h r i n k to a certain number. In the

develop a theory that used 30 variables, in intricate combinations, to explain why

D i e h l a n d K i n g s t o n study, the authors tested a theory that if a country i n c r e a s e s its m i l i t a r y expenditure, then it might be e x p e c t e d to go to war. A c a u s a l theory a l w a y s i n c l u d e s s o m e p h e n o m e n o n that is to be e x p l a i n e d or a c c o u n t e d for.

T h i s is the dependent variable.

In D u v e r g e r ' s theory, the dependent

v a r i a b l e w a s the n u m b e r of parties. A c a u s a l theory a l s o i n c l u d e s one or m o r e factors that are thought to affect the dependent variable. T h e s e are c a l l e d the independent variables.

D u v e r g e r u s e d t w o independent v a r i a b l e s in h i s theory:

people vote the way they do. Such a theory would be about as chaotic and as difficult to absorb as the reality it sought to simplify.

the nature of

2. Predictive accuracy.

A theory should make accurate predictions. It does not help to

have a simple, broad theory which gives predictions that are not much better than one could get by guessing. 3. Importance.

A theory should be important. However, what makes a theory important is

different in engineering research than in theory-oriented research, so we shall consider them separately.

s o c i a l c o n f l i c t s in a country a n d the c o u n t r y ' s electoral s y s t e m . A l l of these factors are c a l l e d " v a r i a b l e s " s i m p l y b e c a u s e it is the variation of

In engineering r e s e a r c h , a theory s h o u l d address a p r o b l e m that is currently

e a c h that m a k e s it of interest to u s . If party s y s t e m s h a d not v a r i e d — t h a t i s , if e a c h

p r e s s i n g . T h i s is a subjective j u d g m e n t , of c o u r s e , but before y o u begin y o u r

country h a d had e x a c t l y the s a m e n u m b e r o f p a r t i e s — t h e r e w o u l d h a v e b e e n nothing

r e s e a r c h , y o u s h o u l d try to j u s t i f y y o u r c h o i c e of topic, not o n l y to y o u r s e l f but a l s o

for D u v e r g e r to e x p l a i n . If one or the other of h i s independent v a r i a b l e s h a d not

to y o u r a u d i e n c e . Y o u r r e s e a r c h report s h o u l d i n c l u d e s o m e d i s c u s s i o n of the i m p o r -

v a r i e d , that factor w o u l d have b e e n u s e l e s s i n e x p l a i n i n g the dependent variable. F o r

tance of the p r o b l e m a n d of p o s s i b l e applications for y o u r findings. It m a y s e e m

i n s t a n c e , if all countries h a d h a d the s a m e electoral s y s t e m , the variations in party

u n n e c e s s a r y to point this out, but it is an important part of the e n g i n e e r i n g r e s e a r c h

s y s t e m s that p u z z l e d h i m c o u l d not h a v e b e e n due to differences in the c o u n t r i e s '

project, one that is often c a r r i e d out s l o p p i l y a n d in an i n c o m p l e t e way. Students

electoral s y s t e m s , i n a s m u c h a s there w e r e n o differences.

have been k n o w n , for e x a m p l e , s i m p l y to turn in a c o m p u t e r printout as a paper,

T h e dependent v a r i a b l e is so n a m e d b e c a u s e in terms of the particular theory

b e c a u s e "the applications are o b v i o u s . " T r u e , the obvious a p p l i c a t i o n s are o b v i o u s ,

used it is thought to be the result of other factors (the independent v a r i a b l e s ) .

but an i m a g i n a t i v e r e s e a r c h e r w h o sits d o w n a n d thinks about it for a w h i l e m a y be

T h e shape it takes " d e p e n d s " on the configuration of the other factors. S i m i l a r l y , the

able to point up additional, m o r e v a r i e d w a y s in w h i c h the results c a n be u s e d .

independent v a r i a b l e s are thus d e s i g n a t e d b e c a u s e in terms of the particular theory,

In theory-oriented r e s e a r c h , the theory s h o u l d give a h a n d l e on as big a portion

they are not taken as determined by any other factor u s e d in this particular theory.

of the u n i v e r s e as p o s s i b l e ; that i s , it s h o u l d apply broadly a n d generally. It is e a s y to

T h e s a m e v a r i a b l e m a y b e a n i n d e p e n d e n t v a r i a b l e i n o n e theory a n d a

develop a trivial theory. A theory of the organization of borough p r e s i d e n c i e s in N e w

d e p e n d e n t v a r i a b l e i n another. F o r i n s t a n c e , o n e theory m i g h t u s e the s o c i a l status

Y o r k C i t y , for e x a m p l e , might predict quite a c c u r a t e l y for that s p e c i f i c situation. B u t

of a p e r s o n ' s father (the i n d e p e n d e n t v a r i a b l e ) to e x p l a i n the p e r s o n ' s s o c i a l status

i n a s m u c h as the borough presidents h a v e little power, it w o u l d not help us very

(the dependent v a r i a b l e ) . A n o t h e r theory might u s e the p e r s o n ' s s o c i a l status as an

m u c h to r e d u c e the c h a o s of N e w Y o r k C i t y p o l i t i c s , let alone the c h a o s of p o l i t i c s in

independent v a r i a b l e to e x p l a i n s o m e t h i n g e l s e , p e r h a p s the w a y the p e r s o n v o t e s .

general.

T h u s , no variable is innately either independent or dependent. I n d e p e n d e n c e

W h e n we say that a theory s h o u l d apply " b r o a d l y " a n d "generally," we are

and d e p e n d e n c e are the two roles a variable m a y p l a y in a c a u s a l theory, a n d it is not

referring not o n l y to h o w large a selection of items f r o m reality the theory deals w i t h ,

s o m e t h i n g about the variable itself. It all depends on the theory:

but also to h o w great a variety of preexisting theories are affected by the n e w theory. A theory c a n attain great generality rather e c o n o m i c a l l y if it h e l p s to recast older the-

Theory 1:

Democracies do not tend to initiate wars.

ories, e a c h o f w h i c h i n v o l v e s its o w n portion o f reality. T h u s , a theory o f electoral

Theory 2:

Countries with high per capita incomes are more likely to be democracies

change might take on importance partly f r o m the p h e n o m e n a it e x p l a i n e d d i r e c t l y —

than poor countries are.

c h a n g e s in p e o p l e ' s votes; but it w o u l d be a m o r e v a l u a b l e tool if it c o u l d be s h o w n to h a v e significant i m p l i c a t i o n s for other areas of s o c i a l t h e o r y — d e m o c r a t i c theory,

In theory 1, democracy functions as an independent variable; the tendency to w a g e w a r

general theories of attitude c h a n g e , or whatever. In effect, it w o u l d p e r f o r m two s i m -

depends on whether or not a country is a democracy. In theory 2, democracy functions

p l i f y i n g f u n c t i o n s : It w o u l d not o n l y give us a handle on the rather l i m i t e d portion of

as a dependent variable; whether or not a country is l i k e l y to be a d e m o c r a c y depends

our environment that it sought to e x p l a i n directly, but it w o u l d a l s o shed light on the

on its per capita i n c o m e .

w i d e r u n i v e r s e dealt w i t h by the other theories.

16

Political

Chapter 2

Theories

and Research

Topics

17

In the e x a m p l e j u s t c i t e d , a theory to e x p l a i n the o r g a n i z a t i o n of b o r o u g h

i s u s u a l l y b e c a u s e the s c o p e o f the theory i s r e s t r i c t e d o r b e c a u s e m a n y o f the

p r e s i d e n c i e s i n N e w Y o r k , the theory a c c r u e s s o little i m p o r t a n c e d i r e c t l y a s t o

e x c e p t i o n s to the theory h a v e b e e n a b s o r b e d into it as a d d i t i o n a l v a r i a b l e s ,

l o o k a b s u r d . B u t i t m i g h t b e p o s s i b l e , i f the b o r o u g h p r e s i d e n c i e s w e r e taken

making it very c o m p l e x .

2

a s e x a m p l e s o f s o m e b r o a d e r c o n c e p t i n u r b a n p o l i t i c s , that the study w o u l d bor-

T h e fact that most social s c i e n c e theory is not very elegant does not m e a n that it

r o w i m p o r t a n c e f r o m this u n d e r l y i n g p h e n o m e n o n . T h e b o r o u g h p r e s i d e n c i e s

is not good. T h e r e a l test of a theory's v a l u e is whether its subject matter is important

m i g h t , for e x a m p l e , s e r v e as a u s e f u l m i c r o c o s m for s t u d y i n g the w o r k i n g s of

and h o w c l o s e it has c o m e to elegance, given that subject matter. If it is important to

patronage.

understand h u m a n k i n d ' s behavior, it is important to try to develop theories about it,

If a theory c a n s u c c e e d r e a s o n a b l y w e l l at meeting these three c r i t e r i a —

even if things do not fall as neatly into p l a c e as we w o u l d like. 1 am a l w a y s a m u s e d w h e n people s a y of a question that is b e i n g m a d e to look

i m p o r t a n c e , s i m p l i c i t y , and predictive a c c u r a c y — i t w i l l be u s e f u l as a tool for s i m p l i f y i n g reality. S u c h a theory is s o m e t i m e s d e s c r i b e d as elegant. O n e difficulty

more difficult than it r e a l l y i s , " T h i s s h o u l d n ' t be that h a r d ; w h a t the h e c k , it's not

in creating an elegant theory is that trying to meet any one of the three b a s i c criteria

rocket s c i e n c e " — i m p l y i n g that rocket s c i e n c e is the e s s e n c e of difficulty and

tends to m a k e it harder to meet the other two. In the e x a m p l e of D u v e r g e r ' s theory,

complexity. N o t to take a w a y f r o m the difficulty of rocket s c i e n c e , but plotting the

we s a w that he might h a v e i m p r o v e d the a c c u r a c y of h i s t h e o r y ' s predictions by

trajectory of an object in a v a c u u m is far s i m p l e r than understanding the motivation

1

bringing in additional explanatory v a r i a b l e s ; but this w o u l d have r e d u c e d the

of a h u m a n b e i n g . P e r h a p s one day the old s a w w i l l b e c o m e , " T h i s s h o u l d n ' t be that

s i m p l i c i t y of the theory. S i m i l a r l y , an attempt to m a k e a theory m o r e general often

hard; w h a t the h e c k , it's not s o c i a l s c i e n c e . "

w i l l cost us s o m e t h i n g in either the s i m p l i c i t y of the theory or the a c c u r a c y of its predictions.

Example of Elegant R e s e a r c h : Philip C o n v e r s e

A s i d e f r o m its utility and simplicity, there is also an element of "beautiful s u r p r i s e " to elegant r e s e a r c h . A p i e c e of r e s e a r c h that goes against our expectations,

I n his article " O f T i m e and P a r t i s a n S t a b i l i t y " ( 1 9 6 9 ) , P h i l i p C o n v e r s e c a m e about

that m a k e s us rethink our w o r l d , gives us a s p e c i a l k i n d of pleasure. P o l i t i c a l scientists

as c l o s e to d e v e l o p i n g an "elegant" theory as one c a n c o m m o n l y do in the s o c i a l

often j o k i n g l y refer to this element as the "interocular subjectivity test" of r e s e a r c h —

s c i e n c e s . H i s study is worth l o o k i n g at in s o m e detail. C o n v e r s e took as his dependent variable the strength of the "party identification"

D o e s it hit us b e t w e e n the e y e s ? A g o o d e x a m p l e of r e s e a r c h w i t h beautiful surprise is a study of the i m p a c t of

of i n d i v i d u a l s — t h e i r sense that they are supporters of one or another of the political

"get-tough" p o l i c i e s against i l l e g a l i m m i g r a t i o n a c r o s s the U n i t e d S t a t e s - M e x i c a n

parties. In an earlier study, he and G e o r g e s D u p e u x h a d found that, whereas about

border. In the late 1980s and early 1 9 9 0 s , the U . S . I m m i g r a t i o n and N a t u r a l i z a t i o n

75 percent of A m e r i c a n s w h o were polled identified with some political party, a s i m i -

S e r v i c e added extra guards and i m p o s e d p u n i s h m e n t s on e m p l o y e r s found to be

lar poll conducted in F r a n c e s h o w e d that less than 45 percent of the respondents did

h i r i n g i l l e g a l i m m i g r a n t s . O n e thousand extra border patrol officers w e r e then added

so ( C o n v e r s e & D u p e u x , 1962). Other studies h a d s h o w n high levels of party identifi-

e a c h y e a r for several y e a r s afterward. D o u g l a s S . M a s s e y and K r i s t i n E . E s p i n o s a

cation in B r i t a i n and N o r w a y , and l o w e r levels of party identification in G e r m a n y and

( 1 9 9 7 ) found that s i n c e the border c r o s s i n g h a d been m a d e tougher, illegal i m m i -

Italy. B e c a u s e the overall extent to w h i c h citizens of a particular country felt bound to

grants w h o o r i g i n a l l y w o u l d h a v e c o m e to the U n i t e d States for only a f e w months of

the existing parties s e e m e d likely to have something to do w i t h h o w stable politics in

s e a s o n a l labor n o w stayed y e a r - r o u n d b e c a u s e they k n e w it w o u l d be h a r d to get

that country w o u l d be, C o n v e r s e wanted to k n o w w h y the level of party identification

b a c k into the U n i t e d States if they w e n t h o m e to M e x i c o . T h e e n d result w a s that the

varied as it did from country to country. At the time of their e a r l i e r study, he a n d D u p e u x h a d f o u n d that the difference

n u m b e r of illegal i m m i g r a n t s present at any g i v e n time w a s i n c r e a s e d , not d e c r e a s e d , by the stepped-up enforcement.

in percentage of party identifiers b e t w e e n F r a n c e a n d the U n i t e d States s e e m e d to

I t a p p e a r s t o b e p a r t i c u l a r l y h a r d t o a c h i e v e elegant r e s e a r c h i n the s o c i a l

b e e x p l a i n e d a l m o s t w h o l l y b y the fact that m o r e A m e r i c a n s than F r e n c h h a d s o m e

s c i e n c e s , c o m p a r e d w i t h other s c i e n t i f i c a r e a s . H u m a n b e h a v i o r i s m o r e c o m p l e x

i d e a o f w h a t party their fathers h a d identified w i t h . A s w e c a n see i n T a b l e 2 - 1 ,

than the b e h a v i o r o f p h y s i c a l o b j e c t s — i n fact, s o m e t h i n k i t m a y p e r h a p s b e

within

l a r g e l y b e y o n d e x p l a n a t i o n . O n the other h a n d , i t m a y b e that h u m a n b e h a v i o r c a n

and A m e r i c a n l e v e l s of party identification. In both countries about 50 percent of

be u n d e r s t o o d , but that we h a v e not y e t c o m e up w i t h a s o c i a l theory that c o u l d

those

s h o w the true potential of our f i e l d . At any rate, it is rare for theory in the s o c i a l

s o m e party t h e m s e l v e s .

s c i e n c e s to a c h i e v e e l e g a n c e . If a t h e o r y ' s p r e d i c t i o n s are r e a s o n a b l y a c c u r a t e , it

'The choice of this word typifies the aesthetic pleasure—and the vanity—with which researchers approach their work.

each

who

row

did

there

not

was

know 3

practically

their

father's

no

difference

party

between

expressed

the

French

identification

with

A b o u t 80 p e r c e n t of those w h o did k n o w their father's

2

Another reason for the difficulty of attaining elegance in social research is simply that most social science terms are imprecise and ambiguous. This problem is addressed in Chapter 3. 3

Remember that this was an early study, done in 1962. It was not long before further work showed similar effects for mothers!

18

Chapter

T A B L E 2-1

Political Theories and Research Topics

2

19

the older he w a s the m o r e l i k e l y he w a s to identify strongly w i t h a party. M o r e o v e r ,

Percent Having Same Sort of Party Identification

this h a d b e e n s h o w n to be a result of h o w long he h a d b e e n e x p o s e d to the party by France

USA

being able to vote for it, rather than of h i s age itself ( s e e , for e x a m p l e , C a m p b e l l

K n o w father's party

79.4

81.6

e t a l . , 1 9 6 0 , pp. 1 6 1 - 1 6 4 ) .

Do not know father's party

47.7

50.7

W o r k i n g f r o m these two a n g l e s , C o n v e r s e d e v e l o p e d a s i m p l e theory that predicts the strength of a voter's party identification f r o m j u s t two things: (1) the number of y e a r s the p e r s o n has b e e n eligible to vote ( w h i c h is a d u a l f u n c t i o n of age

party w e r e p o l i t i c a l party adherents. T h u s , the difference b e t w e e n the two c o u n t r i e s

and h o w l o n g e l e c t i o n s have b e e n h e l d in h i s / h e r c o u n t r y ) ; a n d (2) the l i k e l i h o o d that

w a s a result of the fact that the A m e r i c a n s k n e w their father's party so m u c h m o r e

the i n d i v i d u a l ' s father h a d identified w i t h a party ( w h i c h in turn depends on w h a t

frequently than the F r e n c h d i d .

portion of the father's adult life e l e c t i o n s w e r e h e l d in w h i c h he w a s e l i g i b l e to vote).

A t the time, C o n v e r s e a n d D u p e u x a c c e p t e d this a s a n interesting finding and

T h e first of these derived f r o m the earlier r e s e a r c h on i n d i v i d u a l development, the

d i d not elaborate o n it. B u t i n " O f T i m e and P a r t i s a n Stability," C o n v e r s e u s e d the earlier f i n d i n g to suggest a general theory of the p r o c e s s by w h i c h countries developed stable patterns of party preference. I n d o i n g s o h e b r o u g h t t w o strands o f theory together. F i r s t , h e r e a s o n e d that the d i f f e r e n c e b e t w e e n F r a n c e a n d the U n i t e d States c o u l d b e e x p l a i n e d e a s i l y i f the p r e v i o u s g e n e r a t i o n i n F r a n c e h a d i n d e e d i n c l u d e d v e r y f e w voters w h o i d e n t i f i e d w i t h a party. It c o u l d h a v e b e e n , of c o u r s e , that the d i f f e r e n c e w a s due to the fact that the F r e n c h d i d not talk to their c h i l d r e n about p o l i t i c s as m u c h as the A m e r i c a n s d i d . B u t for the p u r p o s e s o f a r g u m e n t , C o n v e r s e c h o s e t o a s s u m e that this w a s not the c a s e . H e then s h o w e d that i f h i s a s s u m p t i o n about the p r e v i o u s g e n e r a t i o n ' s l o w l e v e l o f party i d e n t i f i c a t i o n w e r e true, o n e c o u l d e x p e c t the next g e n e r a t i o n i n F r a n c e t o b e m u c h m o r e l i k e the A m e r i c a n s . A l s o , i f the a s s u m p t i o n

Markov Chains Converse's reasoning is based on some cute, simple mathematics that you can play with for yourself. If the rates of transferring identifications are in fact the same in two countries, then even though the countries differ greatly in the level of identification at present, we would expect them to converge rapidly. For example, Converse and Dupeux estimated for France and the United States that about 80 percent of those whose fathers had identified with a party developed an identification of their own, and that, of those whose fathers had not identified with a party, about 50 percent developed an identification of their own. Given these figures, and assuming that party identifiers have the same number of children as nonidentifiers, then if 30 percent of the population of country A presently identify with a party, and 90 percent of the population of country B presently identify with a party, in the next generation we would expect to see

w e r e true, F r a n c e m u s t b e m o v i n g t o w a r d the l e v e l o f party i d e n t i f i c a t i o n f o u n d i n the U n i t e d S t a t e s , B r i t a i n , a n d N o r w a y . ( T h i s d e v e l o p m e n t c a n b e seen i n the b o x , " M a r k o v c h a i n s , " on p. 19) C o n v e r s e further r e a s o n e d that the 80 percent a n d 50 percent figures might be

(0.8 X 30%) + (0.5 X 70%) = 59% of country A having an identification, and

u n i v e r s a l l y true. ( H e knew o n l y that they h e l d for F r a n c e a n d the U n i t e d States.) If this w e r e s o , then both F r a n c e a n d the U n i t e d States m i g h t s i m p l y b e e x a m p l e s o f

(0.8 X 9 0 % ) + (0.5 X 10%) = 77%

a g e n e r a l p r o c e s s that a l l c o u n t r i e s undergo w h e n their c i t i z e n s are first g i v e n the vote. In the first e l e c t i o n , s c a r c e l y any voters in a g i v e n c o u n t r y w o u l d identify w i t h a party, but 50 percent of s e c o n d - g e n e r a t i o n voters w o u l d e x p r e s s i d e n t i f i c a -

of country B having an identification. In the next generation after that, we would expect to see

tion. ( S i n c e o f those w h o s e fathers h a d not identified w i t h any party, 5 0 percent d e v e l o p e d a n identification o f their o w n . ) T h u s , g r a d u a l l y party a d h e r e n c e w o u l d r e a c h a stable l e v e l . A c c o r d i n g to this s c h e m e , the r e l a t i v e l y l o w l e v e l of party i d e n -

(0.8 X 59%) + (0.5 X 41%) = 67.7% of country A having an identification, and

tification i n F r a n c e m u s t h a v e r e s u l t e d b e c a u s e the vote w a s extended later a n d l e s s c o m p l e t e l y there than i n A m e r i c a . ( F r e n c h w o m e n , for one thing, w e r e f i r s t

(0.8 X 77%) + (0.5 X 23%) = 73.1%

given the vote in 1945.) T h u s , F r a n c e m u s t be at an e a r l i e r stage of the p r o c e s s than America. T h e s e c o n d strand o f theory c a m e into play w h e n C o n v e r s e tied h i s theory o f national d e v e l o p m e n t to s o m e older findings on i n d i v i d u a l voters in the A m e r i c a n electorate. Voting studies c o m m o n l y h a d s h o w n that w i t h i n an i n d i v i d u a l ' s life s p a n ,

of country B having an identification. Thus, in two generations the two countries, which had started out being quite different, would have moved to similar levels of party identification. The process involved here, called a "Markov chain," is described in J. Kemeny, J. Snell, and G. Thompson, Introduction to Finite Mathematics (Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1957), pp. 171-178.

20

Chapter 2

Political Theories and Research Topics 21

s e c o n d f r o m h i s a n d D u p e u x ' s c o m p a r a t i v e study o f F r a n c e and the U n i t e d States.

A n y t h i n g i n p o l i t i c a l s c i e n c e c a n b e stated w i t h v a r y i n g degrees o f q u a n t i f i -

T h u s , essentially, party identification is predicted f r o m the i n d i v i d u a l ' s age and the

c a t i o n . T o g i v e a c r u d e e x a m p l e : " T h e length o f s e r v i c e o f m e m b e r s o f the U . S .

length of time that the country h a s been h o l d i n g e l e c t i o n s . A f e w e x a m p l e s of predictions f r o m his theory are (1) at the time e l e c t i o n s are

H o u s e i n c r e a s e d f r o m 1880 t o 2 0 0 5 o n the a v e r a g e b y 0.78 y e a r s e v e r y d e c a d e ; the rate o f i n c r e a s e w a s 0.68 y e a r s p e r d e c a d e before 1 9 2 2 , a n d 0 . 8 6 y e a r s p e r d e c a d e

first h e l d in a country, the pattern we t y p i c a l l y o b s e r v e in E u r o p e a n d A m e r i c a (the

after 1 9 2 2 , " s a y s a p p r o x i m a t e l y the s a m e thing a s " F r o m 1 8 8 0 t o 2 0 0 5 , r e p r e s e n -

y o u n g b e i n g w e a k l y identified, the o l d strongly) s h o u l d not h o l d ; all s h o u l d identify

tatives s e r v e d a s t e a d i l y i n c r e a s i n g length of time in the H o u s e ; the c h a n g e

at the s a m e l o w l e v e l s ; (2) if e l e c t i o n s w e r e interrupted in a country (as in G e r m a n y

p r o c e e d e d a bit m o r e r a p i d l y in the latter h a l f of that p e r i o d . " T h e first f o r m of the

f r o m 1933 to 1 9 4 5 ) , l e v e l s of party identification s h o u l d d e c l i n e at a predictable rate;

statement g i v e s m o r e p r e c i s e i n f o r m a t i o n , but the s e n s e of the two statements is

(3) (/the transition rates for a l l countries w e r e r o u g h l y the s a m e as for F r a n c e a n d the

the s a m e .

U n i t e d States, then party identification levels in a l l electoral d e m o c r a c i e s s h o u l d converge over the next f e w generations toward a s i n g l e v a l u e of about 72 percent.

E a c h a p p r o a c h i n v o l v e s c o s t s a n d benefits for r e s e a r c h . M o s t people w o u l d agree that p r e c i s e i n f o r m a t i o n is m o r e u s e f u l than i m p r e c i s e i n f o r m a t i o n , a l l other

T h u s , although C o n v e r s e ' s theory w a s quite simple, it w a s applicable to a w i d e

things b e i n g e q u a l . B u t it m a y be that the time a n d attention spent in gathering

variety of questions. It simultaneously explained individual behavior and characteristics

p r e c i s e data m a k e it difficult for the r e s e a r c h e r to appreciate the larger a s p e c t s of a

of political s y s t e m s . It i m p l i e d a more or less universal f o r m of political development

theory. A l s o , b e c a u s e s o m e k i n d s of data by their very nature are m o r e a m e n a b l e to

at the m a s s l e v e l — w i t h a prediction of initial, but rapidly decreasing, potential for

p r e c i s e f o r m u l a t i o n , there is the danger that o v e r c o n c e r n w i t h p r e c i s i o n m a y restrict

electoral instability in a new electorate. A n d it included the startling suggestion of a

our c h o i c e of r e s e a r c h to those v a r i a b l e s we c a n m o r e e a s i l y quantify. It is s t r i k i n g ,

convergence of "mature" electorates to a c o m m o n level of party identification approxi-

for i n s t a n c e , h o w little the U . S . p r e s i d e n c y i s studied b y p o l i t i c a l s c i e n t i s t s . T h i s

mately equal to that of B r i t a i n , N o r w a y , or the U n i t e d States. T h e theory w a s s i m p l e , a n d i t w a s b r o a d l y a p p l i c a b l e . W h a t w a s m o r e , i t

oversight m a y w e l l be due to the difficulty of getting " h a r d " data on w h a t goes on in that office.

s e e m e d to predict fairly accurately, thus f u l f i l l i n g the third criterion for " e l e g a n c e . "

I deal w i t h the i s s u e of proper l e v e l s of p r e c i s i o n in C h a p t e r 5. F o r our p u r p o s -

U s i n g data f r o m B r i t a i n , G e r m a n y , Italy, the U n i t e d States, and M e x i c o to test the

es here, the important thing is to see w h a t relationship degrees of quantification bear

theory, C o n v e r s e f o u n d that the theory predicted quite w e l l for a l l five c o u n t r i e s .

to e l e g a n c e in r e s e a r c h .

O v e r the y e a r s after it appeared, the C o n v e r s e article stimulated a great deal of

F i r s t o f a l l , the p a r t i c u l a r s u b j e c t w e are s t u d y i n g affects the extent t o w h i c h

further r e s e a r c h , w h i c h i s w h a t one w o u l d expect o f elegant w o r k . H i s f i n d i n g s

i t i s p o s s i b l e for u s t o quantify. I n e l e c t i o n s t u d i e s , there i s c o n s i d e r a b l e s c o p e for

s e r v e d a s a s s u m p t i o n s for f o r m a l theoretic w o r k ( P r z e w o r s k i , 1 9 7 5 ) . T h e y a l s o s t i m -

q u a n t i f i c a t i o n . R e c o r d s f r o m e a r l i e r e l e c t i o n s are u s u a l l y kept i n f a i r l y g o o d

ulated r e s e a r c h e r s to investigate w h e t h e r in fact the transition probabilities on w h i c h

order; the r e s u l t s o f m a n y attitude s u r v e y s are a l s o a v a i l a b l e , a n d m o s t voters d o

the M a r k o v c h a i n is b a s e d are the s a m e in all i n d u s t r i a l i z e d countries ( B u t l e r &

not regard their a c t i o n s a s s o m e t h i n g about w h i c h they n e e d t o m a i n t a i n s e c r e c y .

5 3 ) , and to test w h e t h e r n e w electorates actually b e h a v e as

T h u s , the quantitative r e s e a r c h e r is a b l e to do a great d e a l . On the other h a n d ,

C o n v e r s e ' s theory predicts they w o u l d ( S h i v e l y , 1972; L e i t n e r , 1997; T i l l e y , 2 0 0 3 ;

Stokes,

1 9 6 9 , p.

i n C h i n e s e s t u d i e s , o r i n s t u d i e s d e a l i n g w i t h the U . S . p r e s i d e n c y , s o u r c e s o f

D a l t o n & W e l d o n , 2 0 0 7 ) . It is in this w a y that a good p i e c e of theoretical w o r k f e e d s ,

quantitative data are quite r e s t r i c t e d , a n d m o s t r e s e a r c h m u s t b e r e l a t i v e l y n o n -

and b e c o m e s e n m e s h e d i n , the w h o l e b o d y o f theoretical exploration.

quantitative. I n v i r t u a l l y every f i e l d o f political r e s e a r c h , however, w o r k c a n b e c o n d u c t e d in either a p r i m a r i l y quantitative or a p r i m a r i l y nonquantitative m o d e . It is p r o b a b l y

To Quantify or Not

best that studies w i t h

A side issue in the question of h o w to develop elegant theory is the o l d chestnut: S h o u l d political s c i e n c e be "quantitative" or not? T h e r e has been m u c h rhetoric spilled over this. As long ago as 1956, J a m e s Prothro c a l l e d the dispute "the nonsense fight over scientific method," but it has not c o o l e d sufficiently in the intervening y e a r s .

quantitative m e a n s , but generally, numerical m e a s u r e s of things, a n d

It is a bit hard to pin d o w n w h a t the term r e s e a r c h that p a y s a g o o d d e a l of attention to tends to m a k e

mathematical

statements about them, is c o n s i d e r e d quantitative.

R e s e a r c h that is less c o n c e r n e d w i t h m e a s u r i n g things n u m e r i c a l l y , a n d tends to make

verbal statements

about t h e m , is c o n s i d e r e d relatively less quantitative.

varying degrees of quantification be c a r r i e d on in any g i v e n

field of p o l i t i c a l r e s e a r c h , for the different levels of quantification c o m p l e m e n t e a c h other. T y p i c a l l y , less quantitative r e s e a r c h provides greater breadth, greater openness to totally n e w theories, and a greater a w a r e n e s s of the c o m p l e x i t y of s o c i a l p h e n o m ena. S t u d i e s e m p l o y i n g m o r e quantitative data, however, are m o r e l i k e l y to p r o d u c e s i m p l e , u s a b l e theories; and they are certainly m o r e l i k e l y to give us a c l e a r i d e a of h o w accurate a theory's predictions are. T h u s , e a c h a p p r o a c h has its o w n costs a n d benefits, a n d it is w e l l to r e m e m b e r that no particular degree of quantification h a s a corner on e l e g a n c e .

22

Political Theories and Research Topics

Chapter 2

CHOICE OF ATOPIC

23

Theory-Oriented Research C h o o s i n g a topic that w i l l p r o d u c e important results for theory is undoubtedly m o r e

T h e c h o i c e of a r e s e a r c h topic is intimately b o u n d up w i t h the elegance of what c o m e s out of the r e s e a r c h effort. In selecting a topic, of c o u r s e , the first step is to c h o o s e a general area that is interesting and significant for y o u . By c h o o s i n g to study political s c i e n c e y o u h a v e already begun to n a r r o w the field, and y o u certainly w i l l narrow things m o r e before y o u are ready to begin. T h e r e is no difficulty in this; y o u s i m p l y f o l l o w y o u r interests.

difficult than formulating a question that m a y y i e l d important practical applications. Y o u w i l l r e c a l l that if theory-oriented r e s e a r c h is to be important, it s h o u l d have a broad and general effect on theory. T h i s effect c a n be a c h i e v e d either directly through the p h e n o m e n a it e x p l a i n s , or indirectly through the variety of other theories it affects. S i m i l a r l y , to be "new," the r e s e a r c h results must either p r o d u c e totally n e w theories or lead to s o m e c h a n g e in the status of older theories.

B u t o n c e y o u h a v e c h o s e n a general area to w o r k i n , p i c k i n g a particular topic to r e s e a r c h is difficult. T h i s is the c r i t i c a l d e c i s i o n in doing r e s e a r c h . It is a l s o the most difficult aspect of r e s e a r c h to teach a n y o n e . It is at this s t e p — s e e i n g that a p r o b l e m exists and that there is a good c h a n c e y o u c a n provide n e w insight into i t — that originality and talent are most c r i t i c a l . T h e important thing in c h o o s i n g a topic is to p i c k one that s h o w s p r o m i s e of g i v i n g y o u n e w and elegant results. T h i s i m p l i e s two things: (1) Y o u want t o f o r m u late y o u r topic question so that y o u r results w i l l be l i k e l y to alter existing o p i n i o n on a subject, and (2) y o u want y o u r results, as m u c h as p o s s i b l e , to attain the three criteria for e l e g a n c e : s i m p l i c i t y , predictive a c c u r a c y , and importance.

T h i s m e a n s that in f r a m i n g any topic for r e s e a r c h , y o u are i n v o l v e d at o n c e in the full body of political s c i e n c e theory, for a single p i e c e of r e s e a r c h m a y s i m u l t a n e o u s l y affect m a n y different theories. R e s e a r c h on h o w a c o n g r e s s i o n a l c o m m i t t e e reaches its d e c i s i o n s , for e x a m p l e , c a n affect theories about p o w e r in C o n g r e s s , general theories about c o m m i t t e e s and organizations, theories about e x e c u t i v e c o n g r e s s i o n a l relations, theories about elite political behavior, and so o n . T h u s , what the r e s e a r c h e r in this area must do is to decide w h i c h r e s e a r c h topic is going to p r o d u c e the greatest c h a n g e in the status of e x i s t i n g theories. T h i s task requires not o n l y that she be f a m i l i a r w i t h as broad a range of existing theories as p o s s i b l e , but that she also have s o m e i d e a of w h e r e an existing body of r e s e a r c h is weakest and most needs to be supported or c h a n g e d . D e c i d i n g w h e r e y o u are l i k e l y to produce theoretical results that are s i m p l e

Engineering Research C h o o s i n g a topic is s o m e w h a t s i m p l e r in e n g i n e e r i n g r e s e a r c h than it is in theoryoriented r e s e a r c h . H e r e , it is p r i m a r i l y a question of u s i n g y o u r time and talents efficiently. To y i e l d elegant results, the topic s h o u l d be one that deals w i t h a p r e s s i n g p r o b l e m and one on w h i c h y o u think y o u are l i k e l y to c o m e up w i t h findings that are

and predict a c c u r a t e l y requires the s a m e sort of g u e s s i n g as in engineering r e s e a r c h , but in theory-oriented r e s e a r c h it is harder to d e c i d e h o w important the results of a study are l i k e l y to be. Y o u must j u g g l e all of these d e c i s i o n s around so as to get the best m i x — a topic that w i l l produce results that are as new and as elegant as p o s s i b l e . T h i s is something for w h i c h no rules c a n be laid d o w n . It is an art.

both accurate and s i m p l e enough to be u s e f u l . At the s a m e time, y o u w i l l w a n t to state y o u r thesis so that y o u r results w i l l not duplicate an earlier study, or at least point up w h e r e that w o r k p r o d u c e d m i s t a k e n results. T h e r e is no s e n s e in w a s t i n g

DEVELOPMENT OF A RESEARCH

DESIGN

y o u r time r u n n i n g over ground that has already been w o r k e d u n l e s s y o u think y o u are l i k e l y to d i s c o v e r d i s c r e p a n c i e s . O n e difficulty in c h o o s i n g the topic is that y o u probably w i l l have to c o m p r o m i s e a m o n g y o u r g o a l s . Y o u m a y decide that for the p r o b l e m nearest y o u r heart, there s i m p l y is not e n o u g h material a v a i l a b l e to let y o u study it satisfactorily. M a n y topics relating to defense or to the executive are of this sort. Or it m a y be that a topic interests y o u not b e c a u s e it deals w i t h the most p r e s s i n g p r o b l e m y o u c a n think of, but b e c a u s e y o u h a v e seen s o m e r e s e a r c h on it that y o u think w o u l d be rather e a s y to

It m a y be true, as I say, that c h o o s i n g a topic is not s o m e t h i n g for w h i c h rules c a n be laid d o w n . B u t it is certainly something for w h i c h rules have been l a i d d o w n . B e c a u s e of an exaggerated fear of ex post facto argument, s o m e s o c i a l scientists h a v e developed a very restrictive p r o c e d u r e to serve as a standard in c a r r y i n g on r e s e a r c h .

4

A c c o r d i n g to this procedure, the r e s e a r c h e r should first frame a theory, stating it in the f o r m of a set of h y p o t h e s e s to be tested. T h e s e h y p o t h e s e s p r e s u m a b l y are b a s e d on w o r k others h a v e done in the past. T h e r e s e a r c h e r s h o u l d then gather fresh data

correct. 4

T h e m a i n thing t o d o i n l o o k i n g for a t o p i c i s t o r e a d . Y o u s h o u l d r e a d s o that y o u are c e r t a i n y o u are p i c k i n g a n i m p o r t a n t p r o b l e m , a n d y o u s h o u l d r e a d t o f i n d out h o w l i k e l y i t i s that y o u r t o p i c w i l l y i e l d u s e f u l r e s u l t s . F i n a l l y , y o u s h o u l d r e a d t o see w h a t other w o r k h a s b e e n d o n e o n the p r o b l e m , o r o n s i m i l a r p r o b l e m s , s o that y o u w i l l s e e w h e r e y o u are m o s t l i k e l y t o p r o d u c e r e s u l t s that are new.

Ex post facto argument results when an investigator forms a theory on the basis of certain evidence, then uses that evidence to affirm the theory. If a political scientist formed a theory of congressional committees on the basis of intimate experience with the House Appropriations Committee, for example, and then carried out a study of the House Appropriations Committee to test the theory, this would be ex post facto argument. The danger in this is that any given situation has certain unique aspects, and these are likely to be included in any theory based on it. If the same situation is then used to test the theory, it will look as if the unique aspects are indeed general, whereas if a different test situation had been used, those parts of the theory would have been found wanting.

24

Chapter

Political Theories and Research Topics

2

w i t h w h i c h to test the theory. F i n a l l y , h a v i n g tested the theory, the r e s e a r c h e r s h o u l d

25

it w i l l be c o m p l e t e d . A l s o , it is h a r d to teach s o m e o n e to grope for interesting topics

either reject it or e n s h r i n e it solely on the basis of those new data. It is true that this

and theories. P e r h a p s a good w a y to learn is by starting w i t h the m o r e c l e a r and

procedure erects f o r m i d a b l e barriers to protect us f r o m ex post facto argument, but it

obvious p r o c e d u r e s , then gradually l o o s e n i n g up as e x p e r i e n c e is gained.

has a n u m b e r of serious d r a w b a c k s . In the first p l a c e , it lends an exaggerated significance to the results of the new study. E v e n in c a s e s w h e r e a variety of previously existing evidence favors a particular theory, that evidence p r e s u m a b l y is to be ignored if the new test gives contradictory

O b s e r v a t i o n s , P u z z l e s , and the C o n s t r u c t i o n of Theories O n e w a y to l o o k at c h o o s i n g topics and developing theories is w i t h the realization that

results. S e c o n d and m o r e important, the u s u a l procedure deters researchers f r o m

they are at heart v e r y c o m m o n s e n s i c a l p r o c e s s e s , b a s e d on our daily e x p e r i e n c e s .

casting about creatively for research topics and theories. B e c a u s e it requires that

T h i s is often lost in the forest of s c h o l a r s h i p , in w h i c h s c h o l a r s frequently deal w i t h

hypotheses be fixed firmly at the beginning of the research p r o c e s s , it effectively

abstractions (and w i t h e a c h other's rival abstractions). B u t at heart all theory-oriented

reduces the research task to a selection of obvious hypotheses. It offers researchers no

research in the s o c i a l s c i e n c e s is of the f o l l o w i n g sort:

encouragement to think about their theories once research has begun. R e s e a r c h e r s are not supposed to r e m o l d theory as they go along, learning more about the subject. T h e y are m e r e l y supposed to react to old theories and concepts rather than to think up e n tirely new problems for explanation. In short, this approach encourages the researcher to function as a clerk. T h e epitome of this type of thinking is the research design—a c o m m o n student exercise in w h i c h students are instructed to frame s o m e hypotheses (presumably based on their reading) and s h o w h o w they might gather data to test those hypotheses.

1. Something in our lives puzzles us, and we try to think of an explanation to account for it. 2. To account for it, we put it in a broader, general category of causal relationships (that is, a theory). 3. To test whether the broader theory is valid as an explanation, we draw other specific predictions from the theory and test these to see whether the theory's predictions are generally true. If they are, the theory qualifies as a plausible explanation of the thing we are trying to explain.

A doctoral candidate w h o m I o n c e talked w i t h s e e m e d to me the perfect e x a m p l e of repeated exposure to e x e r c i s e s s u c h as these. He needed to find a topic for his dissertation and he thought that a good w a y to do this w o u l d be to look through a b o o k l a y i n g out theories of elections, p i c k a f e w propositions about voting behavior, and test them w i t h s o m e data. T h i s is h o w we train p e o p l e to do r e s e a r c h , but m o s t of us h a v e better sense than to f o l l o w our o w n precepts. A s e a r c h of articles in p o l i t i c a l s c i e n c e j o u r n a l s w i l l turn up o n l y a f e w that report r e s e a r c h that f o l l o w s the r u l e s . O n e of the betterkept secrets in p o l i t i c a l s c i e n c e is that g o o d p o l i t i c a l scientists g e n e r a l l y do not

A s a n e x a m p l e , c o n s i d e r the p u z z l e that the U n i t e d States h a s a l w a y s c o n tributed p r o p o r t i o n a l l y m o r e than a l m o s t all other m e m b e r s o f the N A T O m i l i t a r y a l l i a n c e , e s p e c i a l l y a s c o m p a r e d w i t h the s m a l l e r m e m b e r s o f the a l l i a n c e , a s s e e n in Table 2-2. O n l y G r e e c e and Turkey, w h i c h historically have armed heavily because of their c o n f l i c t s w i t h e a c h other, contribute a n y t h i n g n e a r as h i g h a p e r c e n t a g e of their g r o s s n a t i o n a l product t o the c o m m o n a l l i a n c e a s the U n i t e d States d o e s . T h e other s m a l l a l l i e s all appear to v a r y i n g degrees to ride on the coattails of the

d r a w up r e s e a r c h d e s i g n s before they start to w o r k on a topic. N o r do they u s u a l l y " f r a m e h y p o t h e s e s " in any f o r m a l s e n s e before they start to w o r k , although they u s u a l l y h a v e s o m e operational h u n c h e s about w h a t they e x p e c t t o f i n d . A n d they

T A B L E 2-2

theory to t h e m in the first p l a c e . T h e i r procedure is m u c h l e s s f o r m a l than the one they p r e s c r i b e for students. T h e y p l a y w i t h data, i m m e r s e t h e m s e l v e s in w h a t other people h a v e written, argue w i t h c o l l e a g u e s , and think. In doing s o , they grope for interesting theories, theories that are elegant and give a n e w slant to things. A l t h o u g h I h a v e c o n d e m n e d the f o r m a l p r o c e d u r e for d e s i g n i n g r e s e a r c h , I hasten to add that it s h o u l d not be rejected completely. O n e of its advantages is safeguarding against ex post facto argument. F u r t h e r m o r e , e v e n though the r e s e a r c h d e s i g n undoubtedly stifles initiative and creativity, it is m o r e m e t h o d i c a l and easier to apply for the b e g i n n i n g researcher. B e c a u s e students u s u a l l y operate under stricter deadlines than other r e s e a r c h e r s , it m a y m a k e sense for t h e m to w o r k w i t h s p e c i f i c goals in m i n d so that they c a n estimate a c c u r a t e l y at the b e g i n n i n g of a project w h e n

U . S . Defense Spending, C o m p a r e d with the Seven Smallest M e m b e r s of N A T O

m o s t c e r t a i n l y do not i g n o r e older e v i d e n c e , e v e n the e v i d e n c e that suggested a

G r o s s National Product, 2 0 0 5 (billions of $)

Defense Spending as Percentage of G r o s s National Product, 2 0 0 5

12,500

4.0

Turkey

363

3.2

Belgium

297

1.6

Norway

295

1.6

Denmark

258

1.4

Greece

226

3.0

Portugal

184

1.6

37

0.7

United States

Luxembourg

Source: International Institute for Strategic Studies, The Military Balance, 2006.

26

Chapter 2

Political Theories and Research Topics

United States. One way to explain this would be to treat it as a specific instance of a more general relationship—that in any voluntary cooperative group the member with the greatest resources always tends to make disproportionate contributions. That is, a member who sees that the group would fail without her contribution will come through strongly; a relatively insignificant member will see that the group would do about equally well whether or not he contributes, and will tend to sit back and be a free rider. In the NATO example, if the United States does not contribute vigorously, the alliance languishes, but little Denmark hardly makes a difference one way or another. A lot of the political scientist's creativity will then come into play in devising other, testable predictions from the theory to see whether it is generally valid. In this example we might examine chambers of commerce to see whether the biggest merchants in town usually carry most of the freight. Or we might look at trade union-supported political parties such as the Labour Party of Great Britain to see whether the largest unions carry a disproportionate share of the burden. If the theory holds up well across a variety of such tests, it will be a plausible potential explanation for the "Denmark problem." Lest working out a theory or a puzzle should seem too easy or too pat from these examples, let me review for you a puzzle currently in real play, to show the uncertainty and the progression of steps by which scientific discussion of a puzzle usually proceeds in real life. A great deal of attention has been directed over the past decade to the so-called democratic peace—the observation that no democracy has ever been observed to initiate war with another democracy. It is extremely rare in the social sciences for any relationship to be as invariant as this one, and the relationship is also of great importance in the assessment of democracy, so it is not surprising that this has attracted the attention of some very good minds. First of all, a number of scholars have questioned whether the observation is as infallible as it is claimed to be. Both "democracy" and "war" are subject to a certain amount of interpretation. Since no country is a pure democracy, how much democracy is enough to have the country count as democratic? When the NATO alliance attacked the Yugoslavian government of Slobodan Milosevic in 1998 (a government which had been elected, but in an election without much real competition), was that a case of democracies attacking another democracy? Similarly, "war" is not always a black-and-white concept. When the U.S. Central Intelligence Agency gave covert support to a military coup against democratically elected President Allende of Chile, was that war? 5

6

27

Granting the gray areas of definition, however, most political scientists consider the democratic peace to be an enduring relationship of quite unusual regularity. This observation, then, poses the puzzle: Why is it that democracies do not attack one another? There have been two primary explanations offered. First, many scholars argue that democracies share a number of values in common, such as respect for individual liberties and a high value on competition. They also share an international norm of compromise and bounded resolutions to conflicts. These characteristics, it is said, make democracies unlikely to wage war on those with whom they agree in so many areas. Alternatively, a number of scholars point to the institutional structure of democracies as the explanation. People generally do not want to fight wars, and in democracies the wishes of the broad populace count for more than in nondemocracies. Accordingly, the chance that two democracies will enter a war, if in both of them the people's concerns count for a good deal, is low. But beyond these two broad positions, as the diffuse scientific "conversation" has proceeded, many subtly nuanced theories have been developed. As one example, Bueno de Mesquita et al. (1999) point out that both of the broad explanations are hard to reconcile with the fairly strong willingness of democracies to initiate wars of conquest or empire. In the late nineteenth century, for instance, democratic France and Great Britain frequently sent armed forces to help acquire additions to their empires. And in the early twentieth century, the democratic United States enthusiastically engaged Spain in a war intended to lift some parts of the Spanish Empire such as Puerto Rico and the Philippines. These were not wars waged on democracies, but it was certainly the case that democratic values, and the influence of "the neople" in these democracies, did not prevent these countries from entering into warfare rather happily. Bueno de Mesquita and his coauthors propose instead a slightly complicated explanation based on rational choice theory. 1. In a democracy, a leader must satisfy a fairly large number of people in order to stay in power; in an autocracy, leaders typically need satisfy only a small number—the army, for instance—or a single party. 2. A war involves costs, but also benefits if won. A democratic leader needs a higher ratio of potential benefits to costs to justify going to war than an autocrat does, because the democratic leader has to reward more people. (See [1.] above.) Therefore, the stakes are higher for democratic leaders. An autocrat might lose a war yet stay in power (Saddam Hussein of Iraq was an example, continuing in office after he lost the first Gulf War in 1991; he was ousted only later by direct action of American troops in 2003.) It is hard to imagine a democratic leader losing a war and not being voted out of office. 3. Since the stakes are higher for democratic leaders, they will be more careful about entering wars than will autocrats; and once involved in wars, democratic leaders will

The theory used in this example derives from the broader theoretical structure of Mancur Olson's The Logic of Collective Action (1965), which was discussed in pp. 6-7. The structure of argument discussed in this section—see a puzzle, frame an explanation based on a more general principle, devise other unrelated predictions from the general principle in order to test it—is presented skillfully by Charles A. Lave and James G. March in Introduction to Models in the Social Sciences (1975), especially in the first three chapters. 6

T h e discussion that follows draws heavily on the useful review of the democratic peace literature in Bueno de Mesquita et al. (1999).

fight them more determinedly. 4. Given reasonably good information about each other, it is unlikely that either of two democrats involved in a conflict will think that he or she has a preponderant chance of winning a war; and each will know that the other (because it is a democracy) will fight very determinedly if there is a war. Therefore, the countries can be expected to try to negotiate their disputes rather than go to war over them.

28

Political Theories and Research Topics

Chapter 2

This is a good explanation, but it certainly has not represented the last word in political scientists' collective effort to work out an explanation of the puzzle. (A good exercise for you would be to frame rebuttals of the rational choice explanation from the standpoint of the first two explanations—the "shared values" explanation and the "institutional role of the people" explanation.) Note that throughout this discussion of puzzles and explanations. I have said that what is produced is a plausible potential explanation. It typically is the case that more than one plausible potential explanation can be produced for anything needing explanation. It then becomes the task of the scholar to decide among them on the basis of how broadly they apply, how simple they are (remember, a theory that is as complicated as the reality it is meant to explain has not gotten you very far), how accurately they predict, and so on. But the basic building blocks of political explanation are plausible potential explanations, derived in just the way that I have outlined here.

29

them, you will be doing your best job when you maximize your effect on theory with a given investment of time and money. To do this, you must: Maximize the generality of the theory you intend to examine. This is basically a restatement of the first criterion for elegant research. Note, though, that this rule is not something absolute, for any phenomenon can be examined at different levels of generality. One person may be hit on the head by an apple and form a theory of falling apples; another may have the same experience and form a theory of universal gravitation. The physical activity of the "study" is the same in both cases; the difference lies solely in the level at which the researcher works. As an example from political science research, consider the variety of studies done on the presidency. The narrowest range of theory is found in biographies of particular presidents. The researcher in such a biography generally is concerned only with explaining what happened during a particular president's life, especially during his term in office. A broader range of theory is aimed at in studies of the U.S. presidency, which may analyze the nature of the office, the sources of executive power, the way in which presidents' personalities can influence their behavior in office, and so on. A still broader range of theory is seen in studies that use the U.S. presidency as an example of sovereigns in general and seek to explain the sources and limitations of sovereign power. Richard Neustadt's Presidential Power and the Modern Presidents (1991), for instance, often operates at this level of generality. /.

8

MACHIAVELLIAN GUIDE TO DEVELOPING RESEARCH TOPICS There are really no guidelines that I can give you for developing a research topic other than to remind you once again that you are working toward results that are both new and elegant. Perhaps if we view the development task from the perspective of political research in general, however, we will gain some clue as to its place in the entire scheme. Implicit in this chapter is the view that scholarly research represents a loose cooperative effort among many people. I mentioned earlier the pleasure that researchers feel in creating something that no one has seen before. This is mixed, however, with a sense of pride in being part of an ongoing tradition. One's work is something brand new, but it also draws on Karl Marx, or Emile Durkheim, or Robert Putnam, and modifies the meaning of their work. Scholars involved in developing theory form a kind of priesthood—admittedly sometimes run less on faith and more according to the laws of laissez-faire and caveat emptor—focused on the common goal of perfecting elegant theories. As we have seen, the celebrants carry on this process by developing new theories and adapting old ones, fitting these theories to the real world to see how accurately they predict things, and feeding the results of such research back into the body of theory. From this description of the process of research, we can derive a set of rules to guide the individual researcher. If empirical research is motivated by a desire to affect the state of theories, either by confirming them or by working changes in 7

'Needless to say, it is not quite as neat as this. For one thing, a given person usually does not handle all these aspects of a particular problem. One person may work simply at clarifying theories, another may do a descriptive study of a partcular case, and a third may relate the new evidence to the body of older theory. Most researchers can expect to carry on any or all activities at any time.

2. Pick a weak theory to work on. The weaker the previous confirmations of a theory have been, the greater your contribution will be. Of course, you have a greater probability of refuting a weak theory. But also, if your research does confirm the theory, your work will again be more significant than if the theory already had a good deal of confirming evidence. Perhaps the best way to use the strategy of picking a weak theory is to state a new, original theory yourself. In this case, your hypotheses are necessarily in need of proof, and any evidence you can buttress them with will be important. Remember, though, that "new, original theories" that are also elegant are hard to come up with. Another way to follow this strategy is to pick an anomaly—that is, a question on which previous research has been contradictory. A good example of research stimulated by an anomaly is the study by Sidney Tarrow (1971) of the political participation of French peasants. Tarrow was struck by the fact that although French peasants regularly responded to surveys by stating that they were not very interested in politics, they also regularly turned out to vote in greater numbers than did the urban French. This anomaly led Tarrow to probe more deeply into what political involvement means to the French, so as to resolve the apparent contradiction. His conclusions led to the rejection of the traditional "interest in politics" measure. They also shed new light on 8

F o r example, a study of the nature of the presidential office (such as Koenig, 1968), a study of presidents' political personalities (Barber, 1972), or an analysis of presidents' leadership (Greenstein, 2000).

30

Chapter 2

Political Theories and Research Topics 31

the nature of F r e n c h voters' attachment to political parties. On this hinged the apparent

O n e person m a y find a tool that measures a variable better than had been done before

contradiction, since F r e n c h peasants were fiercely independent and understood "interest

and then s i m p l y apply it to sharpen previously e x a m i n e d relationships. A n o t h e r m a y

in politics" to i m p l y some sort of approval of the existing parties. T h o u g h they were in

note an a n o m a l y in a theory and organize an experiment to resolve the problem. A third

fact interested in politics, they denied this because of their antipathy to the parties.

m a y look over previous research findings and place a new, broader, or simpler inter-

A n o m a l i e s s u c h as this are hard to c o m e by, because earlier investigators generally have

pretation on them. A l l are following the rule of m a x i m i z i n g their impact on theory.

noticed them already and have tried to resolve them. If y o u c a n find an anomaly having to do with a significant area of political theory, however, y o u c a n be certain that any plausible efforts at resolution w i l l be interesting. A n o t h e r e x a m p l e of a n i c e a n o m a l y j u s t waiting to be a n a l y z e d is the fact that in the 2 0 0 0 e l e c t i o n , the h i g h e r voters' i n c o m e s w e r e , the m o r e l i k e l y they w e r e to vote for G e o r g e W. B u s h . B u t if y o u look at the " r e d s t a t e s " — t h e states he c a r r i e d — they are on the w h o l e poor states w i t h low average i n c o m e s . W h a t sort of theory might y o u d e v i s e to e x p l a i n t h i s ? B e s i d e s anomalies, y o u might choose a problem y o u believe has just not been sufficiently researched, perhaps one in w h i c h all variables have not been covered. T h u s , y o u might replicate a study in a different context from the original one. Arthur A. G o l d s m i t h (1987) noted that M a n c u r O l s o n ' s (1982) w e l l - k n o w n theory, that prolonged political stability produces rigid social and e c o n o m i c structures that retard e c o n o m i c growth, w a s based mostly on the experiences of the West and Japan. T h e T h i r d W o r l d w o u l d provide a laboratory of less-developed countries with a wider range of cultures

4. Present your theory as clearly and vividly as possible. A

Machiavellian

researcher wants to i n f l u e n c e as m a n y people as p o s s i b l e , so it m a k e s sense to m a k e y o u r reader's life e a s i e r and y o u r m e s s a g e more c o m p e l l i n g . T h i s m e a n s , write w e l l and present any g r a p h i c information w e l l . People often think how y o u say s o m e t h i n g is separable f r o m

what y o u

say, but that is s i m p l y not true. I f the purpose o f theory

is to c h a n g e p e o p l e ' s understanding of the w o r l d , then the w a y the theory is c o m m u nicated to them is an integral part of the development of the theory. H o w to write w e l l and d e s i g n g r a p h i c d i s p l a y s w e l l are b e y o n d the scope of this book; e a c h r e a l l y requires a b o o k in its o w n right. Fortunately, I c a n suggest two truly good b o o k s that w i l l help y o u . F o r w r i t i n g , I r e c o m m e n d W i l l i a m K n o w l t o n Zinsser's

On Writing Well: The Classic Guide to Writing Nonfiction ( N e w Y o r k :

HarperCollins,

1998).

is E d w a r d R. T u f t e ' s

An

excellent

introduction

to

good

graphic

presentation

The Visual Display of Quantitative Information ( C h e s h i r e ,

C o n n . : G r a p h i c s P r e s s , 1983).

and e c o n o m i c circumstances. In fact, G o l d s m i t h found that examination of these c o u n tries did not support O l s o n ' s theory, though he hastened to add that his measure of "instability" w a s questionable and might have been a source of the disparity. T h e most

FURTHER DISCUSSION

likely reason, at least to my taste, is that the context is different. O l s o n ' s theory relies on the negative e c o n o m i c effects of organized interest groups. S u c h groups are strong and active in the countries O l s o n studied, but not in the T h i r d World context that G o l d s m i t h studied. G o l d s m i t h ' s negative results m a y thus point up the importance of organized interest groups in O l s o n ' s theory.

A n intriguing b o o k relevant to this chapter is A r t h u r K o e s t l e r ' s

appropriate w a y s to build and test theory is R o n a l d B r u n n e r ' s " A n ' I n t e n t i o n a l ' Alternative i n P u b l i c O p i n i o n R e s e a r c h " ( 1 9 7 7 ) . A n o t h e r perspective i s offered persuasively by J.

3. Make the connection between the general theory and your specific operations as clear as possible. T h i s really j u s t boils d o w n to m a k i n g sure y o u say w h a t y o u

The Act of Creation

( 1 9 6 9 ) . An article that presents s o m e interesting, rather c o n t r o v e r s i a l ideas about

Donald Moon

(1975).

A n e x c e l l e n t introduction t o b u i l d i n g

elegant theories is found in the first three chapters of L a v e and M a r c h ( 1 9 7 5 ) . S o m e questions y o u might c o n s i d e r are:

think y o u are s a y i n g . It i n v o l v e s s u c h things as the a c c u r a c y of y o u r deductions f r o m the theory to the s p e c i f i c situation, the a c c u r a c y with w h i c h y o u have m e a s u r e d things, and so o n . M u c h of the rest of this b o o k f o c u s e s on s u c h p r o b l e m s . Y o u m a y have noticed that these three rules r e s e m b l e the criteria for elegance fairly c l o s e l y . Y o u also m a y have noticed that the b a s i c p h i l o s o p h y b e h i n d t h e m — " D o r e s e a r c h that m a k e s as big a s p l a s h as p o s s i b l e " — r e a d s like a guide for ruthless and h u n g r y assistant p r o f e s s o r s . B u t e a c h o f the r u l e s , d e r i v e d f r o m m y u n d e r l y i n g

1. Presumably, work in normative philosophy or in formal theory could be evaluated in terms of elegance, just as empirical research is. What changes would this require in the definition of elegance? 2. This chapter has implied that the usual way to come up with a theory is to focus on a body of observations and look for regular patterns in them. Although this is the usual procedure, it is neither the only nor necessarily the best approach. What drawbacks might it involve? In what alternative ways might one develop a theory?

M a c h i a v e l l i a n outlook, also has a b e n e f i c i a l effect on the field as a w h o l e . If i n d i v i d -

3. I stated in this chapter that most social science theories are causal. What would a

uals c h o o s e those p r o b l e m s of theory that have so far h a d the w e a k e s t v e r i f i c a t i o n ,

noncausal theory look like? Under what circumstances would it be likely to be used?

for e x a m p l e , the entire field w i l l benefit f r o m an e x a m i n a t i o n of those theories m o s t

{Hint: Consider Einstein's famous theory, E = mc .)

in need of investigation. N e e d l e s s to say, these guidelines s h o u l d remain flexible enough to allow different m i x e s of research strategy. T h e r e is no one "scientific method" involved here.

Importance of Dimensional Thinking

Chapter 3

33

ENGLISH AS A L A N G U A G E FOR RESEARCH

I have a s s u m e d so far that if p o l i t i c a l r e s e a r c h is to be u s e f u l , a m i n i m a l requirement is that its results s h o u l d m e a n the s a m e thing to

any

two different readers.

Unfortunately, the E n g l i s h language is b a d l y d e s i g n e d to function as a m e d i u m for

m

stating r e s e a r c h results c l e a r l y and without ambiguity. M o s t E n g l i s h w o r d s c a n take on a variety of m e a n i n g s , depending on the context and on the m o o d of the reader. To use an e v e r y d a y e x a m p l e , the statement, " T h e w e l f a r e b i l l failed in the H o u s e

mkM

because the administration did not fight for it," might m e a n : 1. A w e l f a r e bill w a s p a s s e d , but n o t t h e o n e the w r i t e r w a n t e d . 2. No w e l f a r e bill p a s s e d at all. W h i c h e v e r of these is the c a s e , the failure c o u l d have resulted f r o m any n u m b e r of other events, for e x a m p l e : 3 . T h e a d m i n i s t r a t i o n p r i v a t e l y p a s s e d t h e w o r d t o its s u p p o r t e r s t o u n d e r c u t the bill. 4 . T h e a d m i n i s t r a t i o n w a n t e d t h e bill, but did not c a m p a i g n a s h a r d a s i t u s u a l l y d o e s for bills it w a n t s .

In C h a p t e r 2, I a r g u e d that f l e x i b i l i t y a n d o r i g i n a l i t y — i n a w o r d , f r e e d o m — a r e i m p o r t a n t in d o i n g g o o d r e s e a r c h . In this c h a p t e r I s h a l l stress the n e e d to state t h e o r i e s i n a c l e a r , u n a m b i g u o u s w a y — a f o r m o f s e l f - d i s c i p l i n e that m u s t a c c o m pany such freedom.

5. T h e a d m i n i s t r a t i o n c a m p a i g n e d as h a r d as it u s u a l l y d o e s for bills it w a n t s , but not as hard as the writer would have done. 6. T h e a d m i n i s t r a t i o n w e n t all-out for t h e bill, but failed to get it p a s s e d — a n d t h e w r i t e r is in t h e o p p o s i n g party.

In particular, we shall be concerned with framing theories in terms of concepts that cannot easily be subdivided into further distinct meanings, either denotative or

Note that in these statements there are many additional phrases that are ambiguous:

connotative—in other words, very simple concepts w h o s e meaning is unequivocal.

"pass the word," "undercut the bill," "want the bill," "campaign for the bill," "go all-out."

I shall c a l l such simple concepts unidimensional, in contrast to concepts w h o s e meaning

As

another e x a m p l e

of a m u l t i d i m e n s i o n a l

concept,

c o n s i d e r political party.

m a y vary f r o m reader to reader and even from time to time, that is, multidimensional

T h i s c o n c e p t c a n be b r o k e n d o w n into at least six c o m p o n e n t d i m e n s i o n s : those

concepts.

people w h o vote for the party in a given election, those people w h o are registered

C o n s i d e r the f o l l o w i n g e x a m p l e : As a description of c l i m a t e s , "temperature" is

with the party, those people w h o identify t h e m s e l v e s in their m i n d s w i t h the party,

a u n i d i m e n s i o n a l concept. Temperature c a n vary along only one d i m e n s i o n , f r o m hot

those people w h o do voluntary w o r k for the party at s o m e g i v e n time, the officers of

to c o l d . T h e r e f o r e , if a theory states that a temperature b e l o w x degrees c a u s e s the

the party, and those w h o are r u n n i n g or elected as candidates under the party l abel .

spotted ibex to stop breeding, that statement w i l l mean the s a m e thing to every reader.

W h e n r e s e a r c h e r s write about "the political party," they m a y be referring to any or all

" G o o d weather," on the other h a n d , is a m u l t i d i m e n s i o n a l concept that involves

of these v a r i o u s d i m e n s i o n s of the term. T h u s , u n l e s s p o l i t i c a l scientists s p e c i f y

a m o n g other things temperature, w i n d velocity, humidity, amount of precipitation,

w h i c h d i m e n s i o n s apply, they m a y m e r e l y s u c c e e d in creating c o n f u s i o n . A f e w

and degree of c l o u d cover. If there is a variation along any one of these separate

further e x a m p l e s

d i m e n s i o n s , the " g o o d n e s s " of weather varies. If the theory stated that " b a d weather"

economic development,

of m u l t i d i m e n s i o n a l

intelligent,

and

terms

are: power,

enjoyment,

conservative,

love.

c a u s e d the spotted ibex to stop breeding, the m e a n i n g of that theory w o u l d be left to

Debates in political science sometimes resolve themselves into the fact that

the reader's judgment. Is it rain that discourages the i b e x ' s ardor? Or do high w i n d s

participants are using the same term in different w a y s . T h e w o r d liberal, for instance, has

m a k e h i m think of other things? Is it the heat? T h e h u m i d i t y ?

many dimensions, some of w h i c h are probably even inconsistent. Just to look at one of its

As we c a n see f r o m this e x a m p l e , it is preferable to frame t h e o r i e s i n terms of

dimensions, it often connotes "free-spending," although in its original usage, w h i c h is

u n i d i m e n s i o n a l rather than multidimensional concepts. T h e E n g l i s h language does not

still fairly current in E u r o p e , it means "favoring a limited role for government." S i n c e

a l w a y s m a k e this easy, however. In this chapter we shall look at the E n g l i s h language

liberal has generally unpopular connotations in the United States and is multidimension-

in general and at the varied theories that m a y be hidden in ordinary language.

al in complex w a y s , it is natural to use this flexible w o r d as a club in describing candidates. In the election of 1996, for instance, the late Senator Paul Wellstone of Minnesota

32

34

Chapter

Importance of Dimensional ininmng

3

w a s described by his opponent as "embarrassingly liberal" for his vote against a bill reducing welfare payments. He responded that his vote w a s actually "deeply conservative" because it reflected the moral values of Minnesotans. W a s Wellstone's vote "liberal" or "conservative"? T h e argument proceeded almost irrespective of its merits. T h e strategy I am introducing here, framing theories in terms of unidimensional concepts, can help us to keep clear the meanings of the words we use and to avoid this kind of confusion.

3.

Ambiguous associations.

->3

From the standpoint of the social scientist, another fault in

multidimensional words is that each such word is itself a theory of sorts. By associating several distinct things with a single term, a multidimensional word implies the theory that these things go together. But unless its component dimensions are made explicit, it

is a poorly articulated and poorly controlled theory. For example, the word wise has long connoted (1) broad practical knowledge, (2) a highly developed understanding of relationships, (3) a contemplative bent, and (4) a background of long practical experience. Among other things, it has implied the theory that the older (and thus the more experienced) one got, the better one understood things. So, if I use the word wise rather than intelligent, my reader picks up this little

Ordinary Language

theory subliminally embedded in my statement.

A s w e have j u s t seen, m a n y E n g l i s h words involve more than one d i m e n s i o n , and these component d i m e n s i o n s are rarely specified in ordinary language. M u l t i d i m e n s i o n a l

T h e o r i e s such as this, embedded in multidimensional words, change as

concepts are valuable and useful in ordinary language, as is indicated by their popular-

c o m m o n u s a g e r e f l e c t s c h a n g e d m o o d s a n d n e w e x p e r i e n c e s . B u t the p r o c e s s b y

ity. T h e connotations accruing to a w o r d that means many different things simultane-

w h i c h they c h a n g e is not a v e r y s a t i s f a c t o r y o n e ; it is a p r o c e s s that g i v e s us little

ously add richness to our language. A r t and rhetoric c o u l d not be restricted to

c o n f i d e n c e in the n e w t h e o r i e s that are p r o d u c e d . It m a y be an u n a r t i c u l a t e d

unidimensional vocabularies.

d i s l i k e for the theory i m p l i e d by the w o r d wise, for e x a m p l e , w h i c h h a s l e d to

B u t in the social s c i e n c e s , we pay a high price for this benefit. R i c h connotations,

d i m i n i s h e d u s e o f the w o r d i n e g a l i t a r i a n A m e r i c a , a n d its d e g r a d a t i o n i n s u c h

w h i c h are so important to art, get in the w a y of analytic thinking. A poet is pleased if she

s l a n g a s " D o n ' t get w i s e w i t h m e ! " a n d " w i s e guy." I t i s i n j u s t s u c h a w a y that

chooses a w o r d w h o s e meanings give the reader pause for thought; it is the poet's j o b to

w o r d u s a g e g r a d u a l l y c h a n g e s , r e f l e c t i n g different v i e w s o f the w o r l d . T h e polite

create a rich confusion of varied nuances. T h e j o b of the social analyst, however, is to

substitution of the t e r m developing nations for underdeveloped countries in the 1 9 5 0 s a n d 1 9 6 0 s i s another e x a m p l e . T h i s c h a n g e i n u s a g e r e f l e c t e d the c h a n g e d

bring order out of social chaos by means of simple theories. M u l t i d i m e n s i o n a l w o r d s h a m p e r s o c i a l scientists in at least three w a y s :

status o f the T h i r d W o r l d a n d a t least s o m e h o p e f u l t h i n k i n g about w h a t w a s happening there.

1. Poor communication.

Because a reader can never be certain what meaning an author

intended, multidimensional words in effect hamper communication. 2. Difficulty with measurement.

1

T h i s p r o c e s s of theory d e v e l o p m e n t is u n c o n t r o l l e d . In a s e n s e it is v e r y d e m o c r a t i c — p r o b a b l y m o r e d e m o c r a t i c than w e w o u l d l i k e . E v e r y o n e w h o u s e s a

Whenever social scientists want to measure a variable,

p a r t i c u l a r w o r d takes part in the p r o c e s s , and a p e r s o n ' s i n f l u e n c e is m o r e or l e s s

they confront an impossible task unless they have defined the variable in such a way

p r o p o r t i o n a l to the n u m b e r of p e o p l e w h o h e a r her u s e the w o r d . T h e r e is no p r o -

that it consists of a single dimension. For example, consider a political scientist

v i s i o n m a d e for greater i n f l u e n c e o n the p r o c e s s b y those w h o k n o w m o r e about

who wanted to measure the "amount of interaction" between various nations. Now,

the s u b j e c t . D i c t i o n a r i e s , b y c o d i f y i n g w o r d u s a g e , s e r v e t o s l o w d o w n the p r o c e s s

"interaction" involves a number of dimensions: alliances, volume of trade, tourism, exchanges of mail, and so on. Suppose that this political scientist discovered the following things about countries A, B, and C:

of c h a n g e , but they do not affect the quality of the c h a n g e . A n o t h e r p r o b l e m w i t h the p r o c e s s is that these c h a n g e s often p r o c e e d at a s n a i l ' s p a c e . T h u s , t h e o r i e s that are e m b e d d e d i n m u l t i d i m e n s i o n a l w o r d s s u r v i v e p r a c t i c a l refutation m u c h l o n g e r than i f they w e r e m a d e e x p l i c i t . T h e w o r d s

Between Countries

Volume Trade/Year

Items of Mail/Year

Alliance

A and B

$10 billion

20 million

Yes

v i d u a l s , p a c i f i s m , and i n t e r n a t i o n a l i s m often s e e m e d t o g o together w h e n o n e

B and C

$5 billion

30 million

Yes

l o o k e d at the p o l i t i c a l elite in the U n i t e d S t a t e s , the t e r m liberal c a m e to denote

A and C

$20 billion

10 million

No

their p r e s e n c e a n d conservative their a b s e n c e . ( T h e w o r d s w e r e b o r r o w e d f r o m a

liberal a n d conservative are a c a s e in point. B e c a u s e s u c h attitudes as a d e s i r e for e c o n o m i c a c t i v i s m o n the part o f g o v e r n m e n t , c o n c e r n for the legal rights o f i n d i -

E u r o p e a n c o n t e x t , but their m e a n i n g s w e r e c o n s i d e r a b l y c h a n g e d , w h i c h o n l y Between which two countries is there greater interaction? A and C trade together a great deal but are not allies; B and C exchange a great deal of mail but do not trade together as much as A and B. Clearly, "amount of interaction" is not a unidimensional variable. Had

a d d e d t o the s u b s e q u e n t c o n f u s i o n . ) T h e i m p l i e d t h e o r y — t h a t these attitudes tended to c o i n c i d e — w a s n e v e r v e r y a c c u r a t e as a d e s c r i p t i o n e v e n of the elite a n d

the investigator chosen three separate measures—trade, mail flow, and alliance—there would have been no problem in comparing the countries. We can confidently state, for instance, that A and C share the greatest trade, followed by A and B and by B and C. We return to such problems of measurement in Chapter 4.

'Today, of course, reality has caught up with rhetoric in the explosive economic growth of countries such as China, India, and South Korea.

36

Chapter

37

Importance of Dimensional Thinking

3

i t h a s p r o v e d t o b e quite i n a c c u r a t e i n d e s c r i b i n g p e o p l e i n g e n e r a l . I t t u r n s out,

"hurtles through space," and the m i n d does not rebel. B u t w h e n the political scientist

for e x a m p l e , that p e o p l e w h o are l i b e r a l o n i n d i v i d u a l s ' rights are often c o n s e r v a -

describes politics as consisting of " s y s t e m inputs," " s y s t e m outputs," and "feedback

tive o n e c o n o m i c i s s u e s . A l l t h i s w e n t u n n o t i c e d u n t i l a t l e a s t the late 1 9 4 0 s , s o

loops," the mind does rebel. T h i s is because the social scientist deals with people, the

that

thing we care most passionately about. It does not bother us w h e n a p h y s i c i s t tries to

even

today

there

are

still

many

people

who

use

conservative in this w a y , as if they h a d g e n e r a l v a l i d i t y .

the

words

liberal a n d

2

reduce the complex motion of a particle to unidimensional concepts, but w h e n social

T h u s , w h i l e c h a n g e s in usage do p r o d u c e c h a n g e s in the theories i m p l i e d by m u l t i d i m e n s i o n a l w o r d s , the p r o c e s s by w h i c h these c h a n g e s are made is c a p r i c i o u s ;

scientists try to simplify the c o m p l e x reality of politics, the family, or of protests, we are disturbed.

this is p r e s u m a b l y not the w a y we want to d e v e l o p s o c i a l theory. S o c i a l scientists s h o u l d be uncomfortable if hidden a w a y in their statement of a theory are additional little theories that are implicit a n d uncontrolled.

PROPER USE OF MULTIDIMENSIONAL W O R D S

L e t me s u m up the argument thus far: S o c i a l scientists s h o u l d use u n i d i m e n sional language for three r e a s o n s :

I have a r g u e d that we s h o u l d try to use o n l y i n d i v i s i b l e d i m e n s i o n s as c o n c e p t s in p o l i t i c a l s c i e n c e . B u t there is another side to the q u e s t i o n , w h i c h l e a d s me to

1. T h e m e a n i n g of a theory is not u n a m b i g u o u s l y clear if it is c o u c h e d in m u l t i d i m e n s i o n a l words.

temper that s t a n c e s o m e w h a t . A l t h o u g h it is a l w a y s n e c e s s a r y to think and w o r k

2. Variables c a n n o t be m e a s u r e d u n a m b i g u o u s l y if they h a v e b e e n defined in a m u l t i d i m e n s i o n a l way.

separate d i m e n s i o n s together in explicit m u l t i d i m e n s i o n a l c o m b i n a t i o n s .

3. I n c l u s i o n of m u l t i d i m e n s i o n a l w o r d s in a t h e o r y c o n f u s e s that t h e o r y w i t h a d d i t i o n a l

many d i m e n s i o n s i m p l i e d in the term: political c o n s e n s u s w i t h i n the nation; w i d e -

t h e o r i e s that are i m p l i e d b y t h e e x i s t e n c e o f the m u l t i d i m e n s i o n a l w o r d s t h e m s e l v e s .

w i t h u n i d i m e n s i o n a l c o n c e p t s in the s o c i a l s c i e n c e s , it m a y be c o n v e n i e n t to put

" N a t i o n a l integration" m a y provide an e x a m p l e of s u c h a concept. T h e r e are

spread c o m m u n i c a t i o n and personal interchanges w i t h i n the nation; a f e e l i n g , a m o n g the nation's people, that they all b e l o n g together; legal integration; and so on. It

D e s p i t e our prescription, however, we must a c k n o w l e d g e the fact that the E n g l i s h language

contains

many

words

that

hold

a

rich

variety

of

meanings

and

connotations. T h e s e are to be v a l u e d in their o w n right, yet s o c i a l scientists often must create their o w n vocabulary. If ordinary language does not provide u n i d i m e n sional w o r d s for the things s o c i a l scientists want to say, they must invent the w o r d s t h e m s e l v e s . T h i s is one r e a s o n that s o c i a l s c i e n c e writing so often strikes readers as flat and c o l d . W h a t most people m e a n b y " s o c i a l s c i e n c e j a r g o n " i s the u n i d i m e n sional v o c a b u l a r y invented by s o c i a l scientists for their use in analytic r e s e a r c h . W r i t i n g f r o m w h i c h the r i c h n e s s o f v a r i e d connotations h a s been r e m o v e d i s flat. T h i s is s i m p l y one of the costs we must pay to write a n a l y t i c a l l y . Of c o u r s e , this is not meant to e x c u s e poor writing in the s o c i a l s c i e n c e s or a n y w h e r e e l s e . U n i d i m e n s i o n a l language is a m i n o r h a n d i c a p under w h i c h s o c i a l scientists operate, but it need not prevent them f r o m w r i t i n g c l e a r and graceful prose. Actually, writers in any analytic field suffer the same handicap. Natural scientists must create their o w n vocabularies, too. B u t somehow the loss of richness seems more painful in the social sciences than in other fields. T h e physicist m a y describe a body's motion in terms of " m a s s , " "velocity," and "acceleration" rather than saying that it

w o u l d be p o s s i b l e to do without the term national integration, and w o r k directly with these d i m e n s i o n s , but it w o u l d be a w k w a r d . T h e r e are a great n u m b e r of d i m e n s i o n s i n v o l v e d , and they m a y c o m b i n e in odd w a y s . At the s a m e time, there is a w i d e s p r e a d feeling a m o n g political scientists that these things tend to go together to constitute a general p r o c e s s . T h i s p r o c e s s might not be at all e a s y to d i s c e r n if we were f o r c e d to w o r k s i m u l t a n e o u s l y w i t h the large n u m b e r of u n i d i m e n s i o n a l c o n cepts i n v o l v e d in it. A c c o r d i n g l y , political scientists o v e r the last f e w d e c a d e s have explicitly

developed

the

term

national

integration,

composed

of

these

various

d i m e n s i o n s , to refer to the general p r o c e s s . T h e result h a s not been fully satisfactory, but there does appear to be a need for s u c h a s u m m a r y term. T h e use of an explicit c o m b i n a t i o n of separate d i m e n s i o n s s u c h as this has s o m e of the advantages of both " o r d i n a r y l a n g u a g e " and u n i d i m e n s i o n a l l a n g u a g e , and s o m e of the disadvantages of e a c h . S u c h s u m m a r y constructs m a y add grace and interest to the presentation of y o u r results and ideas. T h e y m a y add clarity, too, if y o u are w o r k i n g w i t h so m a n y d i m e n s i o n s s i m u l t a n e o u s l y that readers w o u l d have difficulty k e e p i n g track o f t h e m . A n d they help t o keep y o u r theory p a r s i m o n i o u s . S u c h explicit c o m b i n a t i o n s of d i m e n s i o n s have the advantage, as c o m p a r e d w i t h ordinary l a n g u a g e , that y o u have built y o u r o w n j u x t a p o s i t i o n o f d i m e n s i o n s into the w o r d . H o w e v e r , these contrived concepts retain m a n y of the disadvantages

"Another example of meanings of words being slow to change, an amusing one, is offered by the old saying "the exception that proves the rule." Taken at face value—as it usually is—this phrase is idiotic. An exception does not prove a rule, but should cause us to reconsider the rule. The secret is that centuries ago to "prove" meant to submit to a test; in fact, the word has the same root as "prove," and was used in phrases such as "submit to a proof by fire." The phrase about exceptions has continued, with " p r o o f included in it, even though the changed meaning of the word has rendered the phrase meaningless.

of ordinary l a n g u a g e . A reader cannot tell, for e x a m p l e , w h e n f a c e d with a h i g h score on a variable that c o m b i n e s d i m e n s i o n s A, B, and C, w h e t h e r this means that A and B w e r e h i g h and C low, or that B w a s h i g h and A and C w e r e low, or whatever. A l t h o u g h e x p l i c i t m u l t i d i m e n s i o n a l w o r d s are more u s e f u l than ordinary language, y o u s h o u l d not use them c a s u a l l y .

38

Chapter

3

Importance of Dimensional Thinking

39

I h a v e u s e d as my " b a d " e x a m p l e s w o r d s f r o m ordinary language rather than

interests but rather in determining w h a t leads t h e m to define their v e r y identity. If

w o r d s s p e c i f i c a l l y d e s i g n e d for u s e by s c h o l a r s . In g e n e r a l , as I have i n d i c a t e d , w h e n

the N e w H a v e n w o r k e r s c a n be l e d to identify t h e m s e l v e s as w h i t e s rather than as

s c h o l a r s d e s i g n t e r m i n o l o g y it is a good deal tighter than ordinary language.

w o r k e r s , that determines a l l that f o l l o w s .

H o w e v e r , e v e n s c h o l a r l y v o c a b u l a r y is often not as tight as it c o u l d be. It m a y

A l t h o u g h the scholars involved in these arguments have generally v i e w e d their

h a p p e n , for i n s t a n c e , that several s c h o l a r s w o r k i n g independently of e a c h other

o w n version of power as ruling out one or more of the others, dimensional analysis c a n

d e v i s e related u n i d i m e n s i o n a l w o r d s , e a c h set up for one particular r e s e a r c h u s e .

show h o w they all w o r k together. E a c h of these conceptualizations actually emphasizes

W h e n these related w o r d s are taken as a group, they m a y p r o d u c e c o n f u s i o n as great

a different d i m e n s i o n of power:

as that r e s u l t i n g f r o m ordinary language. In one study, K r o e b e r a n d K l u c k h o h n

decision; B a c h r a c h and Baratz, whether one has helped set the agenda for public decisions; L u k e s , whether one has influenced people's understanding of their interests; and Digeser, whether one has influenced people's political identities.

( 1 9 5 2 ) counted several h u n d r e d different w a y s i n w h i c h the w o r d culture w a s u s e d by anthropologists.

D a h l e m p h a s i z e s whether one has participated in a

A c c o r d i n g l y , e x p l i c i t d i m e n s i o n a l a n a l y s i s o f a body o f s c h o l a r l y w o r k , e v e n i f

F i g u r e 3-1 l a y s these out as four d i m e n s i o n s of power. T h e p o w e r any person

it is already at a quite abstract l e v e l , c a n help us to c l a r i f y the m e a n i n g of the w h o l e

holds in a particular political d e c i s i o n m a y be a n a l y z e d in terms of her position on the

body

development.

four d i m e n s i o n s . On a question of park policy, for instance, a n e w s p a p e r editor m a y

C o n c e p t u a l a n a l y s i s of this sort is terribly important to r e s e a r c h . P e r h a p s the most

of work

and

may

point u p

new

directions

for theoretical

have b e e n i n v o l v e d in the setting of identities, in i n f l u e n c i n g p e o p l e ' s understanding

frequent f a i l i n g in r e s e a r c h is that this part of the w o r k is done c a s u a l l y a n d sloppily.

of their interests, and in determining the agenda of i s s u e s f a c i n g the city but m a y not

T h e f o l l o w i n g a n a l y s i s of different w a y s of u s i n g the c o n c e p t power is an extended e x a m p l e o f h o w s u c h d i m e n s i o n a l a n a l y s i s might p r o c e e d . Figure 3-1

Four D i m e n s i o n s of Power

Example of Dimensional Analysis T h e b a s i c b u i l d i n g b l o c k o f all p o l i t i c a l s c i e n c e a n a l y s i s i s the c o n c e p t o f power. B u t this c o n c e p t is

m u l t i d i m e n s i o n a l and notoriously difficult to pin d o w n .

A Political Decision

Dimension

Major

debates o f the last f e w d e c a d e s c a n b e c l a r i f i e d b y d i m e n s i o n a l a n a l y s i s . R o b e r t D a h l l e d off the m o d e r n debate o v e r the nature o f p o w e r w i t h a study

Participation

participated

did not

o f p o w e r i n l o c a l p o l i t i c s i n N e w H a v e n , C o n n e c t i c u t ( D a h l , 1 9 6 1 ) . H e noted that o n different i s s u e s , different sorts o f p e o p l e w e r e i n v o l v e d , s o h e c o n c l u d e d that l o c a l p o l i t i c a l p o w e r w a s not c o n c e n t r a t e d i n a n elite but w a s w i d e l y d i s t r i b u t e d . B a c h r a c h a n d B a r a t z ( 1 9 6 2 ) r e s p o n d e d that the true e s s e n c e o f p o w e r l a y not i n p a r t i c i p a t i o n i n d e c i s i o n s but i n b e i n g a b l e t o l a y out the a g e n d a for w h a t q u e s t i o n s w e r e to be d i s c u s s e d . D e s p i t e the fact that a w i d e r a n g e of p e o p l e h a d p a r t i c i p a t e d i n N e w H a v e n d i s c u s s i o n s , i t m i g h t b e that a s m a l l g r o u p o f p o w e r f u l f i g u r e s h a d d e t e r m i n e d w h a t i s s u e s w o u l d b e o p e n for d i s c u s s i o n . T h e q u e s t i o n o f the c i t y g o v e r n m e n t t a k i n g o v e r the p u b l i c utilities h a d not b e e n o n the t a b l e , for i n s t a n c e ; nor at that t i m e h a d there b e e n a n y interest in the r e l a t i v e s i t u a t i o n of women vis-a-vis men.

Interests

influenced people's understanding of interests

did not

did

did not

did

did not

did

did not

In 1974, S t e v e n L u k e s a d d e d a further c o m p l i c a t i o n . T r u e p o w e r lay not in the m e c h a n i c s of d e c i s i o n m a k i n g , either in the d e c i s i o n itself or in setting the agenda. Rather, true p o w e r l a y in the ability to define for people w h a t their interests w e r e . If one c o u l d c o n v i n c e white w o r k e r s in N e w H a v e n that the m o s t important thing to them w a s education for their c h i l d r e n rather than defending w h i t e s ' prerogatives against b l a c k s — o r i f y o u d i d the reverse o f t h i s — y o u w o u l d h a v e already pretty w e l l determined the p o l i t i c a l o u t c o m e s . A s a f i n a l h o o k , Peter D i g e s e r ( 1 9 9 2 ) drew o n the w o r k o f M i c h e l F o u c a u l t t o suggest that real p o w e r c o n s i s t s not of determining w h a t people w i l l see as in their

Identity influenced did did did did did did did did did did did people's not not not not not not definition of their identities

did did did did not not

40

Chapter

3

Chapter 4

have been i n v o l v e d in the actual d e c i s i o n s . A m e m b e r of the city park board m a y have been i n v o l v e d in the d e c i s i o n but m a y not have e x e r c i s e d p o w e r in any of its other d i m e n s i o n s . A n elementary s c h o o l teacher m a y have been i n v o l v e d i n the p r o c e s s o f f o r m i n g identities but m a y not have e x e r c i s e d p o w e r in its other d i m e n s i o n s ; another s c h o o l teacher, interested and active on park questions, m a y have been i n v o l v e d in the p r o c e s s of f o r m i n g identities, and m a y have participated in the d e c i s i o n s , but not have e x e r c i s e d p o w e r in the other two d i m e n s i o n s . D i m e n s i o n a l a n a l y s i s of this sort a l l o w s us to see the interrelations of v a r i o u s

WClLTTTOClLTl]

a p p r o a c h e s to a question and c a n a l s o give us a r i c h f r a m e w o r k w i t h w h i c h to apply a m u l t i d i m e n s i o n a l c o n c e p t . It a l s o a l l o w s us to c o m p a r e the i m p o r t a n c e of the d i m e n s i o n s . I m p l i c i t in e a c h later s c h o l a r in the d i s c u s s i o n s k e t c h e d above w a s the

Accuracy

i d e a that h i s notion of p o w e r w a s deeper and more b a s i c than those that went before. A d i m e n s i o n a l a n a l y s i s g i v e s us a structure w i t h i n w h i c h we c a n address this q u e s t i o n .

FURTHER DISCUSSION

F o r m a l treatment o f d i m e n s i o n a l a n a l y s i s w a s i n t r o d u c e d b y A l l a n H . B a r t o n , " T h e

In this chapter I e x p l o r e p r o b l e m s of accurate m e a s u r e m e n t . T h e s e are p r o b l e m s

Concept of Property-Space in Social R e s e a r c h " (1955). Philip E. Jacob's "A Multi-

that arise w h e n we try to relate the a c t u a l operations in a p i e c e of r e s e a r c h — t h a t i s ,

d i m e n s i o n a l C l a s s i f i c a t i o n o f A t r o c i t y S t o r i e s " ( 1 9 5 5 ) f u r n i s h e s a good e x a m p l e o f

m e a s u r e s of t h i n g s — t o the c o n c e p t s that f o r m the b a s i s of our u n d e r l y i n g theory.

dimensional analysis in practice.

C o n c e p t s , o f c o u r s e , exist o n l y i n the m i n d . O n e n e c e s s a r y a s s u m p t i o n , i f w e are t o

S o m e e x a m p l e s f r o m p o l i t i c a l s c i e n c e are C h a p t e r Political Oppositions in

Western Democracies ( 1 9 6 6 ) ,

11

of R o b e r t D a h l ' s

a first-rate a n a l y s i s of the r e l -

c l a i m that a p i e c e of r e s e a r c h h a s tested a given theory, is that the things m e a s u r e d in the r e s e a r c h c o r r e s p o n d to the things in the theorist's m i n d .

Group

T h i s is often a difficult a s s u m p t i o n to m a k e . In the p r e c e d i n g chapter, y o u s a w

Politics ( 1 9 6 0 ) , e s p e c i a l l y pp. 1 5 - 4 0 , in w h i c h he c l a s s i f i e s pressure group activities;

one k i n d o f p r o b l e m that c a n stand i n the w a y o f it. T h e p o l i t i c a l scientist w h o

and the third chapter of h i s Division and Cohesion in Democracy ( 1 9 6 6 ) , an e x c e l -

w a n t e d to m e a s u r e the a m o u n t of interaction b e t w e e n nations f o u n d that there w a s

lent

Representation

no s i n g l e satisfactory indicator of "interaction." A n u m b e r of t h i n g s — t r a d e , m a i l

( 1 9 6 9 ) , a c o l l e c t i o n of v a r i o u s w r i t i n g s on the c o n c e p t of representation, a m o n g

e x c h a n g e s , a l l i a n c e , and so o n — p a r t o o k of "interaction," but no one of t h e m alone

w h i c h P i t k i n ' s o w n e s s a y is p a r t i c u l a r l y insightful;

w a s s y n o n y m o u s w i t h the mental construct.

evant

dimensions

dimensional

for c l a s s i f y i n g

analysis

Representation ( 1 9 6 7 ) ;

of

"opposition";

"political

Harry

division";

Eckstein's

Hanna

Pressure

Pitkin's

also P i t k i n ' s The

Concept of

and G i o v a n n i S a r t o r i ' s Parties and Party Systems ( 1 9 7 6 ) .

In the social s c i e n c e s , only rarely are we able to measure our concepts directly.

O n e b r a n c h o f p o l i t i c a l p h i l o s o p h y i s the " a n a l y t i c p o l i t i c a l p h i l o s o p h y "

C o n s i d e r , for example, the concepts " s o c i a l c l a s s , " "respect for the presidency," and

a p p r o a c h , w h i c h s e e k s t o s t u d y p o l i t i c a l i d e a s b y a c l o s e e x a m i n a t i o n o f the

"power in the community." A n y variables we w o u l d c h o o s e to measure these concepts

meaning of concepts used to describe politics. T h i s approach is reviewed in

correspond only indirectly or in part to our mental constructs. T h i s is the basic problem

Richard

of measurement in the social s c i e n c e s .

B e r n s t e i n ' s Restructuring

Oppenheim

of Social and Political

Theory

(1978)

and

in

(1975).

F i n a l l y , a s a n e x e r c i s e , y o u m i g h t c o n s i d e r the c o n c e p t u a l p r o b l e m s i n v o l v e d

C o n s i d e r the concept " s o c i a l status." A m o n g s o c i a l scientists there are two popular v e r s i o n s of this concept: " s u b j e c t i v e s o c i a l status," the c l a s s that i n d i v i d u a l s

i n the w e l l - w o r n a p h o r i s m o f L o r d A c t o n : " P o w e r tends t o corrupt a n d absolute

c o n s i d e r t h e m s e l v e s as b e l o n g i n g to; and "objective s o c i a l status," an i n d i v i d u a l ' s

p o w e r c o r r u p t s a b s o l u t e l y . " H o w w o u l d y o u a n a l y z e this c o n c e p t u a l l y ? Can i t b e

rank w i t h regard to prestige along s o c i a l h i e r a r c h i e s s u c h as education, i n c o m e , and

analyzed?

o c c u p a t i o n . N e i t h e r v e r s i o n of the c o n c e p t c a n be m e a s u r e d directly. I n the c a s e o f " s u b j e c t i v e s o c i a l status," w e c a n n o t m e a s u r e d i r e c t l y w h a t i n d i v i d u a l s f e e l about their status. W e k n o w w h a t they report, but their r e p l i e s t o our i n q u i r i e s m a y not b e w h a t w e are l o o k i n g for. T h e y m a y not k n o w w h a t they " r e a l l y " f e e l , for i n s t a n c e ; o r they m a y m i s u n d e r s t a n d the q u e s t i o n a n d g i v e a

41

42

Problems of Measurement

Chapter 4

43

m i s l e a d i n g a n s w e r . T h e n a g a i n , a p e r s o n m a y f e e l d i f f e r e n t l y f r o m one d a y t o the

W h e t h e r w h a t we say about the m e a s u r e s is an accurate statement of the relationship

next, i n w h i c h c a s e the m e a s u r e o f h i s status w i l l d e p e n d o n our rather arbitrary

b e t w e e n the c o n c e p t s depends on the u n o b s e r v e d relationships b e t w e e n c o n c e p t A and m e a s u r e A and between c o n c e p t B and m e a s u r e B. We c a n only a s s u m e what

c h o i c e o f day. In the c a s e of "objective s o c i a l status," again we cannot m e a s u r e the variable

these relationships are. L i k e the theory itself, the relationships cannot be o b s e r v e d .

directly. " O b j e c t i v e status" has something to do w i t h i n c o m e , something to do w i t h

A s a n e x a m p l e , suppose y o u want t o a s s e s s the theory tested b y D i e h l and

e d u c a t i o n , s o m e t h i n g to do w i t h o c c u p a t i o n , and s o m e t h i n g to do w i t h various other

K i n g s t o n , that countries that are i n c r e a s i n g their a r m a m e n t s tend to engage in

h i e r a r c h i e s , s o m e of w h i c h we m a y not k n o w about. N o n e of these provides a suffi-

aggressive international p o l i c i e s (see above, p. 8). Y o u might be f a c e d w i t h the

cient m e a s u r e in itself. F o r e x a m p l e , if we tried to u s e o c c u p a t i o n alone as a m e a s u r e

f o l l o w i n g relationships:

of s o c i a l status, we w o u l d be f a c e d w i t h the difficult question of whether a p l u m b e r w h o m a d e $ 4 0 , 0 0 0 a y e a r w a s r e a l l y of l o w e r s o c i a l status than a bank teller w h o made $ 2 5 , 0 0 0 a year. S i m i l a r l y , if we tried to use i n c o m e alone as a m e a s u r e , we w o u l d be f a c e d w i t h the p r o b l e m of what to do w i t h a retired teacher, w h o s e i n c o m e might be b e l o w the poverty l e v e l . " S o c i a l status" in this c a s e is a c o n c e p t that is related to a n u m b e r of m e a s u r a b l e things but is related only i m p e r f e c t l y to e a c h of them. T h e best we c a n do in m e a s u r i n g it is to c o m b i n e the v a r i o u s m e a s u r a b l e

1. Relationship between the concept "increasing armaments " and the measure of "increasing armaments." You clearly cannot measure increases in a nation's armaments directly; only a national intelligence apparatus has the facilities to do that, and even then, the result is imperfect. Therefore, you might take as your measure the country's reported expenditures on armaments. Now, a country that is not preparing to launch an aggressive military venture would have less reason to lie about an arms buildup than would a country (such as Germany in 1933) that is consciously preparing for aggression. Therefore, the relationship between

indicators into a p o o l e d m e a s u r e that is r o u g h l y related to the c o n c e p t "objective

concept and measure in this case might be: When a country is not building up its arma-

s o c i a l status."

ments, or when it is building them up in order to launch an aggressive action, its reported

We encounter s i m i l a r p r o b l e m s in m e a s u r i n g the other concepts I have cited as e x a m p l e s . L i k e m a n y other v a r i a b l e s i n p o l i t i c a l s c i e n c e , these concepts are o f c o n -

expenditures on arms will not increase.

2. Relationship between the concept "increasing armaments" and the concept "aggressive

siderable interest and use in theories but are by their nature i m p o s s i b l e to measure

international policies." Let us assume, for this example, that countries that are increasing

directly. T h e general p r o b l e m p o s e d b y s u c h v a r i a b l e s i s presented s c h e m a t i c a l l y i n

their armaments do tend to engage in aggressive international policies.

3. Relationship between the concept "aggressive international policies" and the measure

Figure 4-1. A s y o u s a w i n C h a p t e r 2 , i n p o l i t i c a l r e s e a r c h w e are c o m m o n l y interested i n relating c o n c e p t s through a theory. T h i s is a l w a y s true in theory-oriented r e s e a r c h , and it is often true in e n g i n e e r i n g r e s e a r c h as w e l l . If we cannot measure directly the

of "aggressive international policies." Let us assume, for this example, that we are able to develop a measure that corresponds almost perfectly to the concept "aggressive international policies." (In practice, of course, this would be a difficult variable to measure, and it certainly would be necessary first to analyze the varied dimensions

concepts w e w i s h t o u s e , w e find o u r s e l v e s i n the position depicted i n F i g u r e 4 - 1 .

involved in "aggression" and "policies" in order to state more clearly just what was

We w a n t to say, " C o n c e p t A bears a relationship of t h u s - a n d - s o f o r m to c o n c e p t B . "

meant by the concept.)

B u t all that we c a n o b s e r v e is the relationship between m e a s u r e A and m e a s u r e B. W e n o w find o u r s e l v e s i n the position depicted i n F i g u r e 4 - 2 . H e r e , b e c a u s e o f Figure 4-1

peculiarities in the relationships b e t w e e n the concepts a n d the m e a s u r e s of these

The Problem of Measurement

c o n c e p t s , the relationship y o u c a n o b s e r v e between the m e a s u r e s turns out to be the opposite of the true relationship b e t w e e n the c o n c e p t s . W o r s e yet, i n a s m u c h as the Theory Concept A

-«

>-

Concept B

two m e a s u r e s and the c o n n e c t i o n between them are all that y o u c a n o b s e r v e , y o u w o u l d h a v e no w a y of k n o w i n g that this w a s h a p p e n i n g . T h i s is w h y I h a v e c a l l e d indirect m e a s u r e m e n t of concepts the p r o b l e m of m e a s u r e m e n t . O n e solution to the p r o b l e m might be to m e a s u r e v a r i a b l e s only directly. S o m e

9 I

I 9

concepts are directly m e a s u r a b l e . A few e x a m p l e s are p e o p l e ' s votes if an election is nonsecret and y o u tabulate the result y o u r s e l f ; statements y o u hear m a d e on the Senate floor; a b o m b y o u see dropped or t h r o w n .

Relationship Measure A

•*

*-

Measure B

1

"it is reasonable, also, to include here reliable observers' accounts of such events. Even though the measurement of these things is technically indirect, if you accept another observer's account of them, you should be able to achieve a very tight fit between the concept "event happens" and the measure "reliable observer says that event happens."

44

Chapter

4

Problems of Measurement

Concept Increases in Armaments

Concept Increases in armaments tend to be followed by aggression

As concept increases, measure increases

Measure

Figure 4 - 2

elections are held with a secret ballot). Therefore, most of the time we must work with variables that are indirect measures of the concepts in which we are interested. This means that there are interposed, between our (concrete) operations and the (abstract) theory we want to work on, the relationships between our concrete concepts and their abstract measures. This is the situation illustrated graphically in Figures 4-1 and 4-2. 2

The chief problem of measurement is to ensure, as much as possible, that the relationships between concepts and measures are such that the relationship between the measures mirrors the relationship between the concepts.

As concept increases, measure increases except when policies are aggressive

Increases in Reported Arms Expenditures

Aggressive International Policies

45

Measure Increased reported expenditures are never followed by aggression

Aggressive International Policies

Example of the Problem of Measurement

The difficulty with such a solution is that concepts that can be measured directly are usually trivial in and of themselves. They are too idiosyncratic to use in general, interesting theories. I would hate to say that particular statements by U.S. senators lead nowhere, but as far as political theory is concerned, it is true. Any given statement can apply only to itself. It takes on a general meaning only if it is placed in a category, so that it can be compared with other statements. For instance, Senator 's statement, "The president's policies are bankrupting the people of my state," is not intrinsically of theoretical interest. It can be placed in various categories, however: "statements of opposition to the president," "statements of concern for constituents' needs," "bombastic statements." Placing it in one of these categories allows us to compare it with other senatorial statements and to develop theories about the causes and effects of such statements. Note, though, that by placing it into a category, we have used the statement as an indirect measure of the concept which the category represents. No given statement is a perfect case of the "bombastic statement" or of the "statement of opposition to the president." Rather, a number of statements approximate each category, and we choose to use these statements as indirect measures of the abstract concept we cannot measure. To sum up the argument thus far: For a concept to be useful in building theories, it usually must be an abstraction, which cannot be measured directly. Further, of those interesting concepts that are in principle directly measurable (how individuals vote in an election, for example), many cannot be measured directly for practical reasons (the

Problems we may encounter in trying to achieve this correspondence between measures and concepts fall under two headings: problems of measure reliability and problems of measure validity.

RELIABILITY A measure is reliable to the extent that it gives the same result again and again if the measurement is repeated. For example, if people are asked several days in a row whether they are married, and their answers vary from one day to the next, the measure of their marital state is unreliable. If their answers are stable from one time to the next, the measure is reliable. The analogy of measuring with a yardstick may help make the meaning of reliability clear. If an object is measured a number of times with an ordinary wooden yardstick, it will give approximately the same length each time. If the yardstick were made of an elastic material, its results would not be so reliable. It might say that a chair was 20 inches high one day, 16 the next. Similarly, if it were made of a material that expanded or contracted greatly with changes in temperature, its results would not be reliable. On hot days it would say that the chair was shorter than on cold days. In fact, the choice of wood as a material for yardsticks is in part a response to the problem of reliability in measurement, a problem certainly not confined to the social sciences. Wood is cheap, rigid, and relatively unresponsive to changes in temperature. There are many sources of unreliability in social science data. The sources vary, depending on what kinds of data are used. Official statistics, for example, may be unreliable because of an unusual number of clerical errors or because of variability in how categories are defined from one time to the next. ("Votes cast in an election," for instance, may mean "all votes, including spoiled ballots" at one time, "all valid votes" at another.) Attitude measures may be unreliable because a question is hard for respondents to understand, and they interpret it one way at one time, another way the next. Or the people entering their responses into the computer may make mistakes.

t e c h n i c a l l y , these relationships are called epistemic correlations. "Correlation" means relationship, and "epistemic" has the same root as "epistemology"—the study of how we know.

46

Problems of Measurement 47

Chapter 4

As an illustration, let us list the various sources of unreliability that might be involved in our example of tabulating responses to a simple question about marital status: 1 . T h e q u e s t i o n m i g h t b e p h r a s e d badly, s o that r e s p o n d e n t s s o m e t i m e s interpreted i t t o m e a n , ' A r e y o u n o w m a r r i e d ? " a n d s o m e t i m e s , " H a v e y o u ever b e e n m a r r i e d ? " I t m i g h t not b e c l e a r h o w p e o p l e w h o w e r e s e p a r a t e d from their s p o u s e s , but not d i v o r c e d , should answer. 2. R e s p o n d e n t s m i g h t be playing g a m e s with the interviewer, a n s w e r i n g questions randomly. 3 . D i s h o n e s t i n t e r v i e w e r s m i g h t b e p l a y i n g g a m e s w i t h t h e r e s e a r c h e r b y filling o u t t h e f o r m s t h e m s e l v e s i n s t e a d o f g o i n g t h r o u g h t h e t r o u b l e o f getting the r e s p o n d e n t s t o answer them. 4 . R e s p o n d e n t s ' a n s w e r s m i g h t d e p e n d o n their m o o d . P e r h a p s t h e y w o u l d a n s w e r " y e s " w h e n t h e y h a d h a d a g o o d day, or " n o " w h e n t h e y h a d h a d a b a d day. 5. R e s p o n d e n t s ' a n s w e r s m i g h t d e p e n d on the c o n t e x t of t h e interview. A p e r s o n m i g h t s a y " n o " t o a n attractive i n t e r v i e w e r a n d " y e s " t o e v e r y o n e e l s e . 6. T h e r e m i g h t be simple clerical errors in copying d o w n the answers, either by the interviewer on the spot or by the person w h o transcribes the interviewers' copy into a

In our previous example, perhaps some people got married, or divorced, from one time to the next. This source of unreliability would be easily distinguishable from others, however, because it would show up as a recognizable pattern of stable answers up to a certain point, followed by changed, but once again stable, answers. A more interesting case is presented when the true value itself varies randomly. This situation sometimes provides the basis for interesting theories. In one study, Converse (1964) noted that on many standard questions of political policy, people's attitudes appeared to vary randomly across time. He concluded that on certain issues, the mass public simply did not form stable opinions; and he went on to draw interesting comparisons between elite and mass opinion, based on that conclusion. Note that to reach this conclusion, Converse had to assume that he had effectively eliminated other sources of unreliability, such as interviewer error and confusion about the meaning of questions. Having first eliminated these sources of unpredictability in the relationship between concept and measure, he could then treat the unreliability in his measure as a reflection of unreliability in the concept. Christopher Achen (1975) later challenged Converse's conclusions on just these grounds.

computer.

Admittedly, some of these possibilities are farfetched. The example itself is a bit strained, inasmuch as straightforward informational items like this one can usually be measured with reasonable reliability. But the same sorts of conditions affect the reliability of less straightforward survey questions, such as "What do you like about candidate X?" "What social class do you consider yourself to be a member of?" and "Do you feel people generally can be trusted?" A few examples will help give you a sense of the magnitude of this problem, at least in American survey research. Even on attributes that should be relatively easy to measure reliably, such as gender and race, some errors appeared when the same respondents were interviewed at two-year intervals by a well-administered survey. On the average, the reported gender of respondents changed 0.5 percent of the time from one interview to the next, whereas race did not change at all. Characteristics on which it is somewhat easier to be vague or mistaken showed substantial unreliability. For example, the report of respondents' educational background showed lower education two years later (which is logically impossible) an average of 13 percent of the time. Presumably, questions that permit a considerable degree of interpretation, such as attitudinal questions, would show even more unreliability. 3

Reliability as a Characteristic of Concepts Thus far I have treated the unreliability of a measure as if it were a result of unpredictability in the relationship between the concept and its measure. An additional source of unreliability in the measure is variability in the "true value" of the concept. 3

These are drawn from Asher (1974).

Testing the Reliability of a Measure Although unreliability may sometimes spur on further theoretical research, as it did in this case, it is usually a barrier we want to elm inate. Careful work is the best way to achieve reasonable reliability—double-checki rig all clerical work, trying out the questionnaire on a small pilot study in order to catch and correct unclear questions, and so on. We often wish to know how successfully we have reduced unreliability. A number of tests have been developed to help researchers check the reliability of a measure. I shall describe two of them briefly. The test-retest check for reliability simply consists of repeating the measurement a second time, allowing for a suitable interval of time between the two measurements. If the second measure strongly resembles the first—that is, if the measure is stable over the elapsed time—it is considered relatively reliable. One problem with this test, of course, is that there is no way to distinguish instability that stems from "real" unreliability in the concept being measured from instability due to problems in the measurement process. Another test, the split-half check for reliability, avoids this problem. It is particularly useful whenever a measure is multidimensional—for instance, a measure of "social status," which is made by combining such items as an individual's income, occupation, education, house size, and neighborhood into a single summary measure; or a measure of "welfare policy expenditures," comprising such disparate items as welfare payments, unemployment relief, hospital subsidies, and school lunch programs. In the split-half test, the researcher randomly divides these assorted items into two groups and then composes a summary "measure" out of each of the groups. Because all of the items are taken to be measures of the same thing, the

48

Problems of Measurement 49

Chapter 4

two s u m m a r y m e a s u r e s s h o u l d tend to be the s a m e . A m e a s u r e of h o w c l o s e they

either high or low, depending on the reason for the arms buildup. T h e m e a s u r e might

are to e a c h other p r o v i d e s a c h e c k on h o w r e l i a b l e the total s u m m a r y m e a s u r e i s .

be reliable (a country that reported l o w increases in one year, for instance, s h o u l d be

A s a n e x a m p l e , c o n s i d e r a state-by-state m e a s u r e o f " w e l f a r e p o l i c y e x p e n d i -

l i k e l y to report low increases the next y e a r a l s o ) , but it w o u l d still be invalid, b e c a u s e

tures." It m i g h t be that one p a r t i c u l a r i t e m — d i s a s t e r relief, for i n s t a n c e — v a r i e s greatly f r o m state to state a n d f r o m one y e a r to the next in any o n e state. In one y e a r

the m e a s u r e does not mirror the concept accurately. T h e relationship between validity and reliability c a n b e c l a r i f i e d b y introduc-

random error a n d nonrandom error, w h i c h w i l l also be important

there m i g h t be no natural d i s a s t e r s ; in another there m i g h t be floods or a h u r r i c a n e .

ing the notion of

T h a t p a r t i c u l a r i t e m w o u l d be a s o u r c e of u n r e l i a b i l i t y in the overall m e a s u r e .

w h e n we l o o k at m e a s u r i n g the strength of relationships in C h a p t e r 8. R a n d o m error

It a l s o s h o u l d c a u s e the s p l i t - h a l f test to s h o w a r e l a t i v e l y l o w reliability, for its

is the sort of error we have a d d r e s s e d in d i s c u s s i n g reliability. If in the l o n g r u n , on

erratic variation w o u l d m a k e the s c o r e b a s e d o n the group i n w h i c h i t w a s i n c l u d e d

the average, the m e a s u r e s of a c o n c e p t tend to be true, we c a n a s s u m e that any error

l e s s l i k e l y to e q u a l the s c o r e b a s e d on the group that d i d not i n c l u d e it.

i n the m e a s u r e i s r a n d o m . I n m e a s u r i n g education, for e x a m p l e (see p . 4 6 ) , w e

T h e s e two c h e c k s for reliability c o m p l e m e n t e a c h other. T h e test-retest c h e c k

encountered a good deal of r a n d o m error; people tended to report their l e v e l of e d u -

is appropriate for any sort of m e a s u r e that c a n be replicated. It c h e c k s for all s o u r c e s

cation differently f r o m one time to the next. T h e r e w a s no r e a s o n to expect that peo-

of unreliability, but this often i n c l u d e s c h a n g e s in the true v a l u e of the c o n c e p t rather

ple w e r e deliberately m i s r e p r e s e n t i n g their educational l e v e l , however, s o w e w o u l d

than o n l y the instability that is due to the m e a s u r e m e n t p r o c e s s .

expect that in the long r u n , a c c i d e n t a l reporting errors w o u l d c a n c e l e a c h other out

T h e s p l i t - h a l f c h e c k is appropriate for m e a s u r e s c o m p r i s i n g a group of subitems. It c h e c k s o n l y for those s o u r c e s of unreliability that do not operate over

and the average of m a n y s u c h reports w o u l d give a true m e a s u r e of the c o n c e p t for a particular group of people.

time, i n a s m u c h as a l l of the subitems p r e s u m a b l y are m e a s u r e d at the s a m e time.

By contrast, n o n r a n d o m error is s y s t e m a t i c error that tends in the l o n g r u n , on

A c c o r d i n g l y , i t c a n m i s s s o m e s o u r c e s o f instability i n the m e a s u r e m e n t p r o c e s s ,

the average, to distort a g i v e n m e a s u r e of a c o n c e p t . T h u s , if we a s k e d people

s u c h as the effect of the length of time s i n c e p a y d a y or of c h a n g e s in the weather, on

w h e t h e r they h a d a p r i s o n r e c o r d , it is l i k e l y that there w o u l d be a g o o d deal of

respondents' a n s w e r s to an i n t e r v i e w question. B u t this is actually an important

n o n r a n d o m error in the m e a s u r e , as people s y s t e m a t i c a l l y tried to suppress their

benefit: If we are able to s c r e e n out true c h a n g e over time in the c o n c e p t , we w i l l

p r i s o n r e c o r d s . E v e n in the l o n g r u n , on the average, this m e a s u r e w o u l d not give an

have a m u c h better i d e a of any instability due to the m e a s u r e m e n t p r o c e s s .

accurate estimate of the true v a l u e . Q u i t e s i m p l y , a m e a s u r e is v a l i d to the extent that it is free of both sorts of error. A m e a s u r e is reliable to the extent that it is free of r a n d o m error alone. T h u s ,

VALIDITY

reliability is a n e c e s s a r y but not a sufficient c o n d i t i o n of validity.

R e l i a b i l i t y has to do w i t h h o w d e p e n d a b l y a m e a s u r e m i r r o r s its c o n c e p t . In thinking

a r c h e r w h o is t r y i n g to p r o d u c e " v a l i d " s h o t s , that i s , shots at the c e n t e r of a

about reliability, we a s s u m e d i m p l i c i t l y that the m e a s u r e tended to m i r r o r the

target. T h e s h o o t i n g m a y s u f f e r f r o m either r a n d o m error (the a r c h e r i s e r r a t i c ) o r

c o n c e p t faithfully a n d that the p r o b l e m of reliability w a s s i m p l y that this tendency

n o n r a n d o m error (the a r c h e r h a s s o m e s y s t e m a t i c p r o b l e m — p e r h a p s there i s a

T h e t w o sorts o f error are p r e s e n t e d v i s u a l l y i n F i g u r e 4 - 3 . T h i n k o f a n

m a y be a rather loose one. We a s s u m e d , in other w o r d s , that if the c o n c e p t w e r e

w i n d f r o m the left, a n d the a r c h e r h a s not y e t l e a r n e d to c o r r e c t for i t ) , or both.

m e a s u r e d a large n u m b e r of t i m e s , the average of those m e a s u r e s w o u l d reflect our

T h e s e t w o sorts o f error r e s u l t i n the f o u r p o s s i b l e c o m b i n a t i o n s s h o w n i n F i g u r e

" i d e a l " c o n c e p t . T h e p r o b l e m in reliability is that s i n c e the m e a s u r e s vary, any one of

4 - 3 . T h e a r c h e r a c h i e v e s " r e l i a b i l i t y " on b o t h targets B

them c o u l d be rather far f r o m the true v a l u e of the concept.

" v a l i d i t y " o n l y o n target D .

a n d D but a c h i e v e s

A m o r e s e r i o u s f a i l i n g of our m e a s u r e m e n t s , h o w e v e r , c o u l d result if they lack

validity. A m e a s u r e is

v a l i d i f it a c t u a l l y m e a s u r e s w h a t it purports to m e a s u r e .

T h a t i s , i f there is in p r i n c i p l e a

relationship of equivalence b e t w e e n a m e a s u r e and

Some Examples

its c o n c e p t , the m e a s u r e is v a l i d . A m e a s u r e c a n n o t be v a l i d a n d yet not be r e l i a b l e .

A f e w e x a m p l e s are in order. T h e r e are m a n y w a y s that a m e a s u r e c a n be i n v a l i d . We

B u t it c a n be r e l i a b l e a n d yet not be v a l i d . If it g i v e s the s a m e result repeatedly, the

have already d i s c u s s e d s e v e r a l i n s t a n c e s o f r a n d o m error, s o w e w i l l c o n f i n e our-

m e a s u r e is r e l i a b l e , but it c o u l d distort the c o n c e p t in the s a m e w a y e a c h of these

selves here to e x a m p l e s of n o n r a n d o m error.

t i m e s , so that it does not tend to m i r r o r it faithfully. In effect, it is " r e l i a b l y i n v a l i d . "

O n e c o m m o n s o u r c e of i n v a l i d m e a s u r e s is extrapolation f r o m a s a m p l e to a

T h e relationship between the measure " i n c r e a s e s in reported arms expenditures"

population that is not r e a l l y represented by that s a m p l e . U s i n g letters to the editor as

and the concept " i n c r e a s e s in a r m a m e n t s " in F i g u r e 4 - 2 is an example of invalid

a n indicator o f p u b l i c o p i n i o n w o u l d b e u n w i s e , for i n s t a n c e , b e c a u s e the people

measurement. T h e relationship between the concept and the m e a s u r e is s u c h that w h e n

w h o write s u c h letters are not an accurate c r o s s s e c t i o n of the p u b l i c as a w h o l e .

"increases in a r m a m e n t s " are h i g h , " i n c r e a s e s in reported arms expenditures" m a y be

T h e i r o p i n i o n s w o u l d not be a v a l i d m e a s u r e of " p u b l i c o p i n i o n . "

50

Problems of Measurement 51

Chapter 4

Random Error

High

Figure

Low

4-3 Random and Nonrandom Error

A comic case of sampling problems from the early days of opinion polls is the Literary Digest poll. The Literary Digest was a giant magazine in the United States in the early part of the twentieth century. Starting in 1924, the Digest ran an ambi-

tious poll in presidential election years. Virtually everyone who owned a car or a telephone was reached by the poll, which was sent out to a mailing list obtained from telephone directories and state automobile registration lists. Only about 20 percent of the sample ballots mailed out were returned, but even at that, the Digest had over 2 million responses each time. The Digest sample distorted the U.S. population in two ways. First, it essentially sampled only the upper and middle classes, inasmuch as those who did not have a car or a telephone—at a time when cars and telephones were far less universally owned than today—did not get onto the mailing list. Also, it sampled only those who were interested enough and energetic enough to return the sample ballot. Because only 20 percent of those who received the ballot returned it, this seems to have been a rather select sample. In 1924, 1928, and again in 1932, the Digest poll was very successful, coming within a few percentage points of the actual outcome in each of those elections. By 1932 the poll had become an institution; it was attacked in the Congressional Record and featured in New Yorker cartoons. Thus it was a shock when the poll's prediction of a landslide victory for Landon in 1936 was gainsaid by FDR's decisive victory. When the Literary Digest went out of business the next year, it was thought that the shock and loss of reputation from having called the election so badly was a factor in the magazine's demise.

Apparently, the interested, middle-class sample the Digest drew upon did not vote much differently from the rest of the country from 1924 through 1932. Accordingly, its sympathies were a valid measure of the way the country was going to vote in those elections. Between 1932 and 1936, however, Roosevelt initiated the New Deal, which broadened his support among the poor and drove the middle class to the Republicans. After 1936, the sympathies of the middle class were no longer a valid measure of the way the country would vote. A questionnaire item may also result in an invalid measure when respondents attach a significantly different meaning to the question than was intended by the researcher. It had always been thought, for example, that French farmers were essentially apolitical. When asked in surveys "How interested are you in politics?" they had generally responded, "Not at all." At the same time, it was striking that voting participation was higher among farmers than among most groups in the French population. If they were not interested in politics, why did they vote? In a study designed to explore this paradox (see p. 29), Sidney Tarrow discovered that the innocent question about political interest had been spreading confusion. French farmers apparently interpreted "interest in politics" to mean commitment to some particular party, with many of them vehemently rejecting political parties. Thus, many farmers who were interested in politics but considered themselves independents responded "Not at all" to this invalid measure of "interest in politics." Checks for Validity Taking precautions. Our problem in checking the validity of a measure is similar to the general problem of measurement, depicted in Figure 4-1. We say that a measure is valid if it is a true measure of its concept. But the general problem of measurement is precisely the fact that usually all we can observe is the measures. We cannot

know what the relationship between a concept and its measure is. How, then, can we assess the validity of the measure? The answer, of course, is that there is no pat way to do so. Part of the "craft" in the craft of political research is cleverness and care in developing measures that appear likely to be valid. Some techniques are available to help in developing valid measures. These deceptively simple strategies consist of taking various precautions against invalidity during the construction of a measure. The most important thing is to think through the measurement process carefully and to be on guard against any way in which the relationship between concept and measure might be distorted. For example, we now know that drawing a sample in certain ways (drawing a random sample, for instance) guards against a fiasco like that which destroyed the Literary Digest poll. Also, in determining the final form of a questionnaire that you hope to use in a study, you should ask a few people to answer your questions and then to relate their understanding of the questions themselves. This may alert you to questions that mean something different to your respondents than they mean to you.

52

Chapter

Problems of Measurement

4

S u c h p r e l i m i n a r y testing of o n e ' s m e a s u r e s is c a l l e d a "pilot study." S i m i l a r l y , in

53

IMPACT O F R A N D O M A N D N O N R A N D O M ERRORS

u s i n g o f f i c i a l d o c u m e n t s , y o u s h o u l d go thoroughly into the b a c k g r o u n d of the things reported t h e r e — h o w they w e r e d e v e l o p e d , what the terms m e a n , how broadly

It s h o u l d be o b v i o u s that the o n l y g o o d m e a s u r e is one that is v a l i d , that i s , one that has in it neither r a n d o m nor n o n r a n d o m error. B e c a u s e s o c i a l scientists often operate

they are intended to apply, a n d so o n . T h e s e techniques s i m p l y require that the investigator think ahead to p r o b l e m s that c o u l d o c c u r in the relationship b e t w e e n c o n c e p t and m e a s u r e and act either to prevent these or to c h e c k to see whether they are present. At the m o s t general l e v e l ,

w i t h m e a s u r e s that suffer f r o m one or the other sort of error, however, it is worth c o n s i d e r i n g what happens under those c i r c u m s t a n c e s . A s i t h a p p e n s , the t w o sorts o f error have different effects on the development of theory. T h e effect of n o n r a n d o m error is s i m p l e and severe. If a m e a s u r e is s y s t e m a t i -

the strategy I have suggested here requires o n l y that the investigator c o n s i d e r c a r e -

c a l l y i n v a l i d , there is no r e a s o n for us to expect any c o r r e s p o n d e n c e b e t w e e n the

fully h o w p l a u s i b l e it is that the m e a s u r e mirrors the concept.

relationship w e actually o b s e r v e f r o m our m e a s u r e s and the i d e a l i z e d relationship Test of validity.

T h u s far our strategies have not a c t u a l l y p r o v i d e d a test of the

we w i s h to investigate.

validity of the m e a s u r e . S u c h a test c a n be m a d e , although it is n e c e s s a r i l y subjective

T h e effect of unreliability ( i f the m e a s u r e s are o t h e r w i s e valid) is m o r e subtle.

and o p e n to v a r y i n g interpretations. L e t us say that we want to decide whether

To the extent that m e a s u r e s are unreliable, the relationship at the m e a s u r e l e v e l

m e a s u r e a is a v a l i d m e a s u r e of c o n c e p t A. If there is s o m e m e a s u r e /3 that we are

w i l l tend to be looser and w e a k e r than the true relationship. It w i l l p a r a l l e l the true

certain is strongly related to c o n c e p t A, we c a n c h e c k to see whether m e a s u r e ji is

relationship but w i l l appear w e a k e r than is actually the c a s e . If the m e a s u r e s are

related to m e a s u r e a. If it is not, a n d our a s s u m p t i o n of a relationship between /3 and

sufficiently u n r e l i a b l e , the b a s i c relationship c a n be so w e a k e n e d that it w i l l appear,

A is true, then a c o u l d not be a v a l i d m e a s u r e of A.

f r o m w h a t we c a n o b s e r v e , as if there is no relationship at a l l .

T h e study b y T a r r o w cited earlier provides a n e x a m p l e o f this l o g i c . T a r r o w ' s

T h i s is illustrated in Table 4 - 1 . E a c h set of two c o l u m n s tabulates the c l o s e n e s s

initial c o n c l u s i o n that the u s u a l question " H o w interested are y o u in p o l i t i c s ? " w a s

of elections in ten congressional districts, as w e l l as their representatives' seniority. It

p r o v i d i n g a n i n v a l i d m e a s u r e o f "political interest" a m o n g F r e n c h farmers c a m e

is apparent from the figures in the first set of c o l u m n s (True V a l u e s ) that there is a

f r o m his observation that farmers h a d in fact one of the highest levels of electoral

relationship between the two, i n a s m u c h as representatives f r o m safe districts tend to

participation a m o n g F r e n c h c i t i z e n s . B e c a u s e h e c o u l d not c o n c e i v e o f high electoral

have greater seniority than those f r o m marginal districts. T h e relationship is also quite

participation o c c u r r i n g in the a b s e n c e of h i g h political interest, he c o n c l u d e d that the

strong; seniority increases without exception as the representatives' margins of victory

conventional m e a s u r e s , w h i c h had s h o w e d l o w political interest c o i n c i d i n g w i t h h i g h

increase.

participation, must not have b e e n m e a s u r i n g political interest v a l i d l y .

4

In the third and fourth c o l u m n s , r a n d o m error, s u c h as might o c c u r f r o m clerical

As another e x a m p l e of this k i n d of test, c o n s i d e r a measure of nations' hostility

errors or other sources of unreliability, has been added to the original m e a s u r e s ,

to e a c h o t h e r — b a s e d on content a n a l y s i s of the nations' n e w s p a p e r s ( a ) . If we found

m a k i n g them less reliable. In the fifth and sixth c o l u m n s , an even greater degree of

that two of the nations went to w a r against e a c h other (/3) yet the measure did not show an a c c o m p a n y i n g increase in feelings of hostility between the two, we w o u l d be

TABLE 4-1

Safe Districts Related to Seniority, Using Simulated D a t a

s u s p i c i o u s of the validity of the m e a s u r e s . True Values

S u c h an indirect test of validity is p o s s i b l e only w h e n a r e s e a r c h e r is quite

usually, p o s s i b l e ; and it is a l w a y s rather subjective. B u t a s s e s s i n g the validity of m e a s u r e s is so important that an indirect test, w h e n it c a n be u s e d , w i l l greatly strengthen y o u r findings. T h e m o s t general test of validity is w h a t is c a l l e d face validity. T h i s is j u s t a f a n c y term for whether a m e a s u r e l o o k s right to y o u . Is it v a l i d "on its f a c e " ? A f t e r a l l , y o u have c o n s i d e r a b l e e x p e r i e n c e w i t h politics and m u s t j u d g e for y o u r s e l f whether the m e a s u r e does w h a t y o u want it to do. If y o u think it does (and people w h o read y o u r w o r k agree w i t h y o u ) , i t has face validity.

4

In this example, "political interest" corresponds to A, farmers' responses to the question on political interest correspond to a, and farmers' electoral participation corresponds to B.

Very Unreliable Measures

Margin of Victory (%)

Seniority

Margin of Victory (%)

Seniority

32

18

32.0

21.6

12.8

9.0

24

12

19.2

12.0

12.0

12.6

23

11

11.5

12.1

30.2

8.8

20

11

22.0

4.4

22.4

6.6

14

8

14.0

5.6

21.2

12.0

11

6

11.0

6.0

13.1

3.6

10

6

9.0

10.8

2.5

0.1

6

4

3.6

4.0

13.8

2.8

5

3

4.5

3.6

21.3

1.5

2

1

2.4

0.5

0.2

12.9

certain that /3 must go along w i t h A. T h a t k i n d of certainty is u n c o m m o n and m a y not be shared e q u a l l y by every observer. T h u s , this test is not a l w a y s , or even

Less Reliable Measures

Seniority

Margin of Victory (%)

54

Chapter 4

Problems of Measurement

55

random error has been added. Note that the relationship b e c o m e s w e a k e r in the s e c o n d

U n d e r the b a d , old s y s t e m both R e p u b l i c a n suburbs and D e m o c r a t i c c i t i e s h a d been

set of data (there are more exceptions to the general tendency) and virtually disappears

d i s e n f r a n c h i s e d , w h i l e r u r a l a r e a s , w h i c h had benefited, w e r e not a l l that c l e a r l y

in the last. If we w e r e to test the relationship between safe districts and seniority and

either D e m o c r a t i c o r R e p u b l i c a n . S o r e d r e s s i n g the i m b a l a n c e s w a s a p p r o x i m a t e l y

had only the data f r o m the last two c o l u m n s in hand, we w o u l d probably c o n c l u d e that

a w a s h in terms of party strength. A n s o l a b e h e r e a n d h i s c o a u t h o r s pointed out, t h o u g h , that the true test of

there w a s no relationship.

w h e t h e r e l e c t o r a l institutions affected p o l i c y w a s not the effect on the t w o parties, or on the o v e r a l l s p e n d i n g l e v e l of the state, but w h e t h e r c o u n t i e s that g a i n e d fair representation thereafter got a m o r e equal share of state e x p e n d i t u r e s . In other

IMPORTANCE OF ACCURACY

w o r d s , the true test of w h e t h e r e q u a l i t y of representation affected p o l i c y w a s not I n a c c u r a c y in measurement is a critical p r o b l e m w h o s e potential for m i s c h i e f does not

w h e t h e r the l e v e l o f s p e n d i n g r o s e — i t w a s pretty o b v i o u s w h y i t d i d n o t — b u t

yet s e e m w e l l - e n o u g h understood by political scientists. B e c a u s e the variables we

w h e t h e r it w a s distributed m o r e e q u a l l y as a result of m o r e e q u a l r e p r e s e n t a t i o n .

w o r k w i t h are difficult to m e a s u r e , we have in m a n y c a s e s c o m e to accept measures

W h e n A n s o l a b e h e r e a n d h i s c o a u t h o r s tested the effect i n this w a y , they f o u n d

that we k n o w full w e l l are inadequate. M u c h of survey r e s e a r c h tends to fall into this

d r a m a t i c p o l i c y effects f r o m r eappor t i onm ent . T h r e e d e c a d e s o f r e s e a r c h and

category. T h e loose acceptance of cross-nationa l indicators (for instance, using

c o m m e n t a r y h a d m i s s e d the point.

" n e w s p a p e r s read per 1,000 population" as a measure of the political awareness of various electorates) is another e x a m p l e of this problem. To the extent that our measures are not valid, what we do with them is irrelevant.

measures. Fortunately, political scientists are now b e c o m i n g more aware of the problem of measurement, but its importance must be constantly underscored.

Be sure that the measures you

choose

fit

the relationship among concepts

affected

public

a

measurement

policy

had

i n a d e q u a c i e s in the c h o s e n m e a s u r e s m a y debilitate a theory so that it b e c o m e s a l m o s t a trivial e x e r c i s e . C o n s i d e r a test for the s i m p l e theory: " T o the extent that they understand p o l i t i c s , i f p e o p l e ' s n e e d for p u b l i c s e r v i c e s i s r e l a t i v e l y great,

1. "Understanding of politics" indicated by years of education. This would seem reasonable; at least, understanding should be fairly closely related to education.

S t e p h e n A n s o l a b e h e r e , A l a n G e r b e r , and J a m e s M . S n y d e r , Jr. ( 2 0 0 2 ) d e m o n -

reapportionment

when

that

O f t e n , an interesting question is lost through m i s t a k e s in

strated this p r o b l e m w e l l in a paper s h o w i n g that three d e c a d e s of r e s e a r c h on electoral

Even

t i o n a l i z e the three v a r i a b l e s of this theory in a w a y s u c h as the f o l l o w i n g :

setting up e m p i r i c a l operations to parallel the theoretical question.

whether

inadequacies.

they w i l l be m o r e l i b e r a l . " It m i g h t w e l l be that a p o l i t i c a l s c i e n t i s t w o u l d o p e r a -

L e t me r e e m p h a s i z e the important pitfalls to be c a r e f u l of in m e a s u r e m e n t :

/.

Test your measures for possible

p r o b l e m is not so central as to n u l l i f y the results of a study, recurrent n a g g i n g

T h i s simple fact has tended to be forgotten in a general ethos of " m a k i n g d o " with poor

you wish to examine.

2.

been

subtly

m i s d i r e c t e d and h a d , as a result, r e a c h e d e x a c t l y the w r o n g c o n c l u s i o n . In the w a k e

2. "Need for public services" indicated by the size of the person's family. Again, although this is a rough measure, it would seem that the more dependents a person had, the more that person would depend on a variety of public services. 3. "Liberalism" indicated by voting Democratic.

of the Baker v. Carr d e c i s i o n , s c h o l a r s had l o o k e d to see w h e t h e r taking a w a y the unfair overrepresentation of rural c o u n t i e s h a d led to an e x p a n s i o n of state s p e n d -

N o w , the e m p i r i c a l a n a l o g of the theory b e c o m e s : " T h e more educated a

i n g . T h e i m p l i c i t a s s u m p t i o n w a s that rural areas w o u l d h a v e preferred t o k e e p

person i s , the stronger the relationship between the s i z e of that p e r s o n ' s f a m i l y and

5

the probability that the person w i l l vote D e m o c r a t i c . " T h i s statement is r i d i c u l o u s . I

W h e n they f o u n d that s p e n d i n g l e v e l s after reapportionment w e r e no higher than

have exaggerated here slightly, but only slightly, the extent to w h i c h u n i m a g i n a t i v e

s p e n d i n g d o w n , s o i f they lost i n f l u e n c e o n p o l i c y , s p e n d i n g s h o u l d h a v e r i s e n .

they h a d b e e n before reapportionment,

scholars

c o n c l u d e d that c h a n g i n g the

s c h o l a r s w i l l a l l o w moderate errors of m e a s u r e m e n t to a c c u m u l a t e in a statement

electoral r u l e s h a d h a d n o i m p a c t o n p u b l i c p o l i c y . T h e r e a s o n w a s pretty c l e a r :

until the statement loses m u c h of its m e a n i n g . T h e c u r e for this p r o b l e m is s i m p l y to u s e c a r e and imagination i n d e v e l o p i n g m e a s u r e s . In this chapter I have d i s c u s s e d problems in the a c c u r a c y of measurement. T h e s e

5

Baker v. Carr 369 US 186 (1962). In this decision the Supreme Court ruled that it was unconstitutional to have state legislative districts with widely varying populations. Prior to the decision many states had done no legislative redistricting for half a century, so backward rural areas that had not experienced much growth in their populations were hugely advantaged relative to rapidly growing cities and suburbs. For instance, in Florida, before the Court's decision. Jefferson County, with a population of 9,543, had had one seat in the state senate and one seat in the state house of representatives. Miami's Dade County, population 935,047, had had one seat in the state senate and three seats in the state house of representatives.

problems turn out to be of two basic types, depending on whether they stem from flux in the measure (the problem of reliability) or from a basic lack of correspondence between measure and concept (the problem of nonrandom error). In the next chapter we look at another aspect of measurement, the question of how precisely a measure should be calibrated. In Chapter 8 we w i l l return to tackle the problem of measuring relationships w h e n random or nonrandom error is present.

56

Chapter 4

Chapter 5

FURTHER DISCUSSION D

A delightful book with a unique approach to handling certain problems of validity is Unobtrusive Measures, by E u g e n e J. Webb, D o n a l d T. C a m p b e l l , R i c h a r d D. S c h w a r t z , and L e e Sechrist (1966). T h e ideas presented in the book are both creative and sound, and the text itself is filled w i t h interesting and highly unusual examples. Frederick Mosteller's

article " E r r o r s , " in the International Encyclopedia of the Social

Sciences (1968), is w e l l worth reading. Mosteller details a variety of possible sources of invalidity and unreliability. Herbert A s h e r (1974) presents some good examples of

Precision

reliability problems, with a rather technical d i s c u s s i o n of w a y s to handle them. S e e also

K i r k and Miller, Reliability and Validity in Qualitative Research (1985). S e v e r a l good examples of specific measurement problems are N i e m i and K r e h b i e l ( 1 9 8 4 ) , " T h e Quality of S u r v e y i n g R e s p o n s e s A b o u t Parents and the F a m i l y " ; H a m m o n d and F r a s e r (1984), " S t u d y i n g Presidential Performance in C o n g r e s s " : F e l d m a n ( 1 9 8 3 ) , " T h e M e a s u r e m e n t and M e a n i n g of Trust in G o v e r n m e n t " ; and C o n v e r s e and Pierce ( 1 9 8 5 ) , " M e a s u r i n g Partisanship." A u s e f u l e x e r c i s e w o u l d be to list as m a n y factors as y o u c a n that w o u l d l e a d t o r a n d o m o r n o n r a n d o m error for e a c h o f the f o l l o w i n g m e a s u r e s : i n t e n d e d vote (in s u r v e y s ) ; strength o f a r m e d f o r c e s ; a g r e e m e n t w i t h the p r e s i d e n t (in c o n g r e s -

The

s i o n a l v o t i n g ) ; t r i b a l i s m ; u n e m p l o y m e n t ; h i e r a r c h i c a l c o n t r o l i n a n a g e n c y ; and

m e a s u r e s . T h o s e p r o b l e m s c o n c e r n e d the relationship b e t w e e n a m e a s u r e and the

preceding

chapter

dealt

with

problems

of the

reliability

and

validity

of

personal income.

c o n c e p t that m e a s u r e is intended to mirror. In this chapter we deal w i t h the " q u a l i t y " of the m e a s u r e i t s e l f — h o w p r e c i s e it s h o u l d be, or h o w finely calibrated, if it is to be u s e f u l . In a study of N o r w e g i a n p o l i t i c s , H a r r y E c k s t e i n felt that it w a s n e c e s s a r y to a p o l o g i z e for the fact that s o m e m e a s u r e s he w o u l d u s e w e r e subjective intangibles ( " w a r m t h i n s o c i a l relations," for i n s t a n c e , and " s e n s e o f c o m m u n i t y " ) rather than p r e c i s e n u m e r i c a l quantities.

[M]any of the indicators used in the text may not be readily recognized as such by contemporary social scientists. By an indicator we usually mean nowadays a precisely ascertainable quantity that stands for some imprecise quantity (as G N P may indicate level of economic development, or as the number of casualties in revolutionary violence may indicate its intensity). I do use such quantities in what follows. More often, however, readily observable "qualities" are used as indicators of not-so-readily observable qualities. This strikes me as both defensible and desirable, for quantitative indicators are not always as "indicative" of what one wants to know as other observations, nor always obtainable. In overemphasizing quantities we sometimes miss the most telling data—in any case, data that may be reliable in their own right or used as checks on the inferences drawn from quantitative data. I conceive of all social behavior as a vast "data bank," only some of which is quantitatively aggregated in yearbooks and the like, and much of the rest of which may speak volumes to our purposes, if used circumspectly. (Eckstein, 1966, footnote pp. 7 9 - 8 0 )

1

'From Eckstein, Harry, Division and Cohesion in Democracy: A Study of Norway (Copyright 1966 by Princeton University Press). Reprinted by permission of Princeton University Press.

57

58

Chapter

5

Problems of Measurement

59

It is wrong that E c k s t e i n should have felt the need to apologize. T h i s should have been an unnecessary justification, but unfortunately, precise measurement has become enough of a fetish in political science that it is no doubt a l w a y s possible to find someone w h o w i l l d i s m i s s a piece of w o r k because it is not "quantitative." In this chapter I sort out j u s t what kinds of precise quantification are helpful in political research. T h e C a r d i n a l R u l e of P r e c i s i o n might read: Use measurements that are as precise

as possible,

given the subject you are studying;

do not waste

information by

imprecise

measurement. O n e theme of this chapter w i l l be that this rule is as susceptible to violation by "quantifiers" as by "nonquantifiers." To discuss it further, I must first distinguish between two kinds of precision with w h i c h we shall be concerned. I c a l l the

first of these precision in measures, and the second, precision in measurement.

PRECISION IN MEASURES P r e c i s i o n in m e a s u r e s c o r r e s p o n d s r o u g h l y to our c o l l o q u i a l u s e of the w o r d precision—that i s , k e e p i n g the units of m e a s u r e m e n t relatively fine.

For example,

reporting a p e r s o n ' s i n c o m e in dollars is m o r e p r e c i s e than rounding it off to the nearest thousand dollars, and r o u n d i n g i n c o m e off to the nearest ten thousand dollars is still l e s s p r e c i s e . S i m i l a r l y , reporting a p e r s o n ' s r e l i g i o n as " P r e s b y t e r i a n , "

18

30

40

" R e f o r m e d J e w i s h , " " G r e e k O r t h o d o x , " and so on is m o r e p r e c i s e than reporting it as

50

60

70

80

85

Age

"Protestant," " C a t h o l i c , " and "other." A l t h o u g h as a general rule, p r e c i s i o n in m e a s u r e s is o b v i o u s l y a g o o d thing, its importance c a n be overrated. F i r s t , the degree of p r e c i s i o n we n e e d is determined by what we w i s h to do w i t h the data. If we w e r e registering voters, for i n s t a n c e , any

Figure 5-1

Age and Participation in 2 0 0 2 Election: Age Measured by Years

Source: 2002 Congressional-Election Survey, National Election Study. Data provided by the Interuniversity Consortium for Political and Social Research.

p r e c i s i o n in m e a s u r i n g age that w e n t b e y o n d l a b e l i n g people "under 18" and " o v e r 18" w o u l d be u n n e c e s s a r y and p o s s i b l y a n u i s a n c e as w e l l . G e n e r a l l y , though, in theory-oriented r e s e a r c h , we are not able to limit the n e c e s s a r y l e v e l of p r e c i s i o n in this way. I n s t e a d , we are interested in l o o k i n g at the entire range of variation and

nor i n m a n y c a s e s p o s s i b l e . I f w e study U . S . presidents, for e x a m p l e , w e are limited to a population of 4 5 . A n o t h e r w a y to handle the problem is to decrease the precision of the measure, 2

have no particular cutoff point in m i n d . S o m e t i m e s , even i n theory-oriented r e s e a r c h , too m u c h p r e c i s i o n i n m e a s u r e s c a n be a n u i s a n c e . C o n s i d e r F i g u r e 5 - 1 , w h i c h s h o w s the relationship b e t w e e n age and participation in the 2 0 0 2 c o n g r e s s i o n a l election. T h e figure is so chaotic that it is h a r d to d e c i p h e r a relationship. G i v e n the l i m i t e d n u m b e r o f i n d i v i d u a l s (about 1,300) c o n s u l t e d i n the p o l l , there i s o n l y a s m a l l n u m b e r o f r e s p o n d e n t s for e a c h p a r t i c u l a r age. T h i s m e a n s that the p e r c e n t voting fluctuates a g o o d d e a l f r o m one age group to the next. ( S e e the b o x " L a w o f L a r g e N u m b e r s , " p . 6 1 ) T h u s , i n the e n s u i n g , l a r g e l y r a n d o m

creating a smaller number of categories with a larger number of cases in e a c h . T h i s has been done in F i g u r e 5-2, with age measured to the nearest half-decade rather than to the nearest year. W i t h the larger number of cases in e a c h age c l a s s , the measures of percent voting fluctuate less and the f o r m of the relationship between age and participation becomes clearer, with the most rapid increases c o m i n g in the first couple of decades, and a drop-off among the very aged. If it is true that s o m e t i m e s we m a y be better off w i t h l e s s p r e c i s i o n in our m e a s u r e s , then it appears l i k e l y that this sort of p r e c i s i o n is not so important that r e s e a r c h s h o u l d b e j u d g e d s o l e l y , o r e v e n largely, o n h o w p r e c i s e its m e a s u r e s are.

fluctuation of p e r c e n t v o t i n g , it is difficult to p i c k out a s y s t e m a t i c pattern in the r e l a t i o n s h i p b e t w e e n age a n d p a r t i c i p a t i o n i n e l e c t i o n s , a l t h o u g h w e c a n see that participation g e n e r a l l y r i s e s w i t h age. O n e w a y to handle this p r o b l e m , of c o u r s e , w o u l d be to e x p a n d the study by c o n s i d e r i n g m o r e i n d i v i d u a l s i n e a c h category. B u t this u s u a l l y i s neither p r a c t i c a l

2

Reducing precision in this way to eliminate random noise in the data is appropriate only for chartmaking and visual presentation. Data analysis techniques such as regression analysis handle random noise in their own way. Reducing precision in measures is thus u n n e c e s s a r y when one is analyzing data, and it may cause such techniques to give systematically inaccurate results. (See Chapter 7 for a description of regression analysis.)

60

Chapter 5

Problems of Measurement

61

100 L a w of Large Numbers The fact that the smaller the group of individuals sampled, the more any measure based on that group will deviate from its norm is one part of the L a w of Large Numbers. (This law forms the basis for a great deal of statistics.) The fact is intuitively obvious. If a research team selected groups of 1,000 people randomly and calculated the percentage male, they would expect to get very nearly the national figure in all of them. With groups of 100, they would get increasingly erratic measurement. (It would not be unlikely for there to be 60 males in 100 people, for example.) With groups of 10, there would be wild fluctuation; and with groups of 1 (the smallest possible "group") all groups would be either 0 or 100 percent male.

80

WASP,

3

white

ethnics,

blacks,

Puerto Ricans,

and Chicanos.

Had you

simply

drawn a random sample of the population, you probably would have drawn too few Puerto Ricans and Chicanos, and you would have had to lump t h e m together with either the blacks or the white ethnics. A n o t h e r pitfall to a v o i d at t h e d a t a - g a t h e r i n g s t a g e is c a s u a l l y t h r o w i n g a w a y precision that m i g h t later p r o v e useful. W h e n asking people their age, record the n u m b e r of years old, not " u n d e r 30," " 3 0 - 4 0 , " and the like. You can always g r o u p 30

40

50

60

70

80

8

Age Figure 5-2

t h e d a t a later i f y o u w a n t t o ; a t this p o i n t y o u s h o u l d s a v e all the i n f o r m a t i o n y o u get. If y o u ask people their religion, write d o w n "Piesbyterian," " R o m a n Catholic," and

Age and Participation in 2 0 0 2 Election: Age Measured by H a l f - D e c a d e s

so o n — n o t "Protestant," "Catholic," or "Jewish ' In short, do not group the informat i o n y o u h a v e u n t i l y o u h a v e f i n i s h e d g a t h e r i n g it. Y o u w i l l h a v e a b e t t e r i d e a t h e n o f h o w fine a g r o u p i n g y o u w a n t to e n d up with. If it is true, as the Cardinal R u l e states, that it is important to be as precise as

O n t h e o t h e r h a n d , a l t h o u g h t h e r e a r e s i t u a t i o n s i n w h i c h w e m a y w a n t t o b a c k off from precise measures, one should not conclude that precision is not helpful or important.

To continue with the example of age and participation, if age were

m e a s u r e d still less p r e c i s e l y t h a n i n F i g u r e 5-2, w e c o u l d a g a i n l o s e s o m e o f t h e pattern that a p p e a r e d there. If age w e r e m e a s u r e d in units of 33 years, for e x a m p l e , w e w o u l d find that 5 4 p e r c e n t o f t h o s e i n t h e first unit ( 1 8 - 5 0 ) v o t e d , a n d that 7 7 p e r c e n t o f t h o s e i n t h e s e c o n d u n i t ( 5 1 - 8 4 ) v o t e d . A l t h o u g h w e w o u l d still s e e a strong relationship in these figures, m u c h of the interesting detail of Figure 5-2

possible and not to waste information, these suggestions will help meet that rule. B u t t h e r u l e a l s o s t a t e s t h a t w e s h o u l d b e a s p r e c i s e a s w e c a n , given the subject w e are studying. B e c a u s e o f t h e l i m i t a t i o n s i m p o s e d b y a s p e c i f i c r e s e a r c h t o p i c , w e m u s t be careful not to overemphasize the importance of precision in measures. M a n y questions in political science simply do not admit of great precision of this type. E c k s t e i n ' s study, w h i c h he d e f e n d e d in the quotation given at the start of this chapter, is a case in point. He wanted to study the degree of " c o m m u n i t y " in Norway. That c o n c e p t d o e s not lend itself to finely calibrated m e a s u r e s .

w o u l d be lost. T h e time to ensure that your measures will be as precise as you want is w h e n y o u are gathering data. A n u m b e r of p r e c a u t i o n s are in order. First, m a k e sure that you include as large a n u m b e r of cases as is practical, so that you will not have to r e d u c e the precision m o r e than you w o u l d like. For e x a m p l e , if y o u r study c o n c e r n s a specific group within the population, it m a k e s sense to get an adequate n u m b e r of the group into your study, even if the resultant data are not

It w o u l d be foolish to give up such studies on this account. Precision in measures is important, but not indispensable.

No one should stop studying the

presidency because the n u m b e r of cases is limited; or stop studying past historical periods b e c a u s e data are limited and m a n y kinds are " p r e g r o u p e d " in inconvenient ways;

or

stop

studying political

corruption because

grouped together.

"typical" of the entire population. This sort of selection is k n o w n as d r a w i n g a stratified sample.

Thus,

if y o u

w a n t e d to

study

the

effects

of different k i n d s

of discrimination, you might draw up a sample consisting of equal numbers of

'WASP is the acronym for white, Anglo-Saxon Protestant.

many

facts

are hidden or

62

Chapter 5

Problems of Measurement

63

PRECISION IN MEASUREMENT

T h e sort o f p r e c i s i o n d i s c u s s e d i n the p r e c e d i n g s e c t i o n , p r e c i s i o n i n m e a s u r e s , c a n be o v e r e m p h a s i z e d . B u t the next sort, p r e c i s i o n in m e a s u r e m e n t , cannot. " P r e c i s i o n in m e a s u r e s " referred to k e e p i n g distinctions as fine as w a s p o s s i b l e and p r a c t i c a l . B u t w e a l s o f i n d v a r y i n g l e v e l s o f information i n the p r o c e s s o f m e a s u r e m e n t itself. T h e r e are b a s i c a l l y three different w a y s to m e a s u r e , e a c h of w h i c h is qualitatively different f r o m the others, in that it contains information not i n c l u d e d in the others. T h e most primitive w a y to measure a variable is s i m p l y to a s s i g n the individuals being studied to categories of that variable. T h i s is c a l l e d nominal measurement. F o r e x a m p l e , to measure religion, we m a y label people " C a t h o l i c , " "Protestant," "Jew," or "other." T o measure nationality, w e m a y label them " B r i t i s h , " " G e r m a n , " " R u s s i a n , " and so on. N o m i n a l measurement places individuals into distinct categories of a variable, but it does not tell us anything about how the categories relate to e a c h other. If, in addition to a s s i g n i n g categories, we c a n rank the categories a c c o r d i n g to Figure 5-3

" h o w m u c h " of the variable they e m b o d y , we have a c h i e v e d ordinal m e a s u r e m e n t .

Levels of Precision

In s u c h m e a s u r e m e n t , we have s o m e i d e a of a s c a l e that might ideally represent the variable, and the scores we a s s i g n to i n d i v i d u a l s s h o w w h e t h e r they fall h i g h e r or

r a n k i n g on the variable, we w o u l d still have a n o m i n a l m e a s u r e , a set of distinct c l a s s e s

l o w e r than others on s u c h a s c a l e . E x a m p l e s are (1) s o c i a l status, m e a s u r e d in s o m e

to w h i c h individuals w e r e a s s i g n e d .

w a y , s u c h as " l o w e r / w o r k i n g / m i d d l e / u p p e r " ; and (2) party identification, m e a s ured

as

"strong

Democrat/weak

Democrat/independent/weak

Republican/strong

Republican."

4

To put it a little differently, if two i n d i v i d u a l s h a v e different s c o r e s on a v a r i a b l e , then (1) if the v a r i a b l e is m e a s u r e d n o m i n a l l y , we k n o w that the t w o differ on the v a r i a b l e ; (2) if the v a r i a b l e is m e a s u r e d o r d i n a l l y , we k n o w that the t w o dif-

F u r t h e r p r e c i s i o n in m e a s u r e m e n t is p o s s i b l e if in addition to r a n k i n g the

fer on the v a r i a b l e and w h i c h one has the h i g i e r s c o r e ; a n d (3) if the v a r i a b l e is

scores a c c o r d i n g to " h o w m u c h " of the variable they represent, we c a n say h o w great

m e a s u r e d intervally, we k n o w that the two differ on the v a r i a b l e , w h i c h one h a s the

the differences b e t w e e n the s c o r e s are. To do this, we must have s o m e c o m m o n unit

h i g h e r s c o r e , and h o w m u c h h i g h e r that s c o r e is than the other p e r s o n ' s .

of m e a s u r e m e n t for our s c a l e . Note that s u c h a unit w a s l a c k i n g in the two e x a m p l e s

T h e three major levels of measurement, then, are n o m i n a l , ordinal, and interval

of ordinal m e a s u r e m e n t . We c o u l d not say whether the difference in status b e t w e e n

measurement. As a definition of " p r e c i s i o n in measurement," we c a n say that measure-

" w o r k i n g " and " m i d d l e " w a s greater than that between " l o w e r " and " w o r k i n g , " nor

ment is relatively precise insofar as it operates at a level that is relatively informative.

c o u l d w e say whether the difference between " w e a k R e p u b l i c a n " and "strong

Interval measurement is more precise than ordinal measurement, w h i c h is in turn more

R e p u b l i c a n " w a s greater than that b e t w e e n " i n d e p e n d e n t " a n d " w e a k R e p u b l i c a n . "

precise than n o m i n a l measurement.

I f units exist b y w h i c h w e c a n m e a s u r e the intervals b e t w e e n s c o r e s l i k e these, w e have a c h i e v e d interval m e a s u r e m e n t . S o m e variables c o m m o n l y m e a s u r e d i n intervals are i n c o m e ( w i t h the unit e x p r e s s e d in d o l l a r s ) , percent voting by districts

Innate Nature of Levels of Measurement

( w i t h the unit e x p r e s s e d in percentage points), governmental expenditure on a

O b v i o u s l y , the l e v e l at w h i c h we m e a s u r e things is not j u s t a c h o i c e we m a k e ; if it

p r o g r a m ( w i t h the unit e x p r e s s e d either in dollars or in percentage points if m e a s u r e d

w e r e , we w o u l d s i m p l y m e a s u r e everything at the interval l e v e l . S o m e m e a s u r e s are

as a percentage of the budget), air p o w e r (with the unit e x p r e s s e d in kilotons of

continuous; that i s , they c o n s i s t of gradations w h i c h , in p r i n c i p l e , c a n be s u b d i v i d e d

b o m b i n g c o v e r a g e , n u m b e r of p l a n e s , or w h a t e v e r ) , body counts (with the unit

infinitely. A n e x a m p l e i s i n c o m e , w h i c h c a n b e e x p r e s s e d i n thousands o f dollars but

e x p r e s s e d in n u m b e r of dead p e o p l e ) , and so on. It is clear that these levels of precision c o m p r i s e a nesting progression, as s h o w n in F i g u r e 5-3. A l l interval measurements are also ordinal measurements and n o m i n a l measurements, and all ordinal measurements are also n o m i n a l measurements. T h a t i s , if we had an interval measure but c h o s e to ignore our k n o w l e d g e of the distances involved between s c o r e s , w e w o u l d still have scores o n a n ordinal measure. A n d i f w e had scores on an ordinal measure but c h o s e to ignore our knowledge of the relative

4

A further refinement in precision is possible if in addition to measuring the length of intervals along a scale. We can assign a score of zero to some point on the scale. We are then said to have "ratio" measurement (because we can then take one score as a multiple of another score, which is not possible if there is no zero point). I have not included ratio measurement in my discussion here because it has not, as yet, figured importantly in data analysis in the social sciences. All of the interval scales I noted above could in fact be treated as ratio scales. Generally, however, this is ignored and they are treated simply as interval scales.

64

Chapter

Problems of Measurement

5

65

w h i c h c o u l d be s u b d i v i d e d into hundreds of d o l l a r s , into dollars, into c e n t s , into

qualitatively different things w i t h data m e a s u r e d at different levels of p r e c i s i o n . W i t h

tenths of a cent, and so o n . A l l continuous m e a s u r e s lend t h e m s e l v e s naturally to

more p r e c i s e m e a s u r e m e n t we c a n do a greater variety of things w i t h our data a n d

interval m e a s u r e m e n t b e c a u s e their infinite d i v i s i b i l i t y requires that there be s o m e

thus h a v e more of an opportunity to develop interesting theories. It is for this r e a s o n

unit of m e a s u r e m e n t that c a n be d i v i d e d .

that p r e c i s i o n i n m e a s u r e m e n t i s more important than p r e c i s i o n i n m e a s u r e s . L e t m e

M e a s u r e s that are not continuous are c a l l e d discrete. T h e s e consist naturally of

provide an e x a m p l e .

separate points that cannot be further divided. T h e r e are a few discrete m e a s u r e s that

I n F i g u r e 5 - 4 , the relationship between age a n d percent voting, w h i c h w e

are interval. E l e c t i o n s , for e x a m p l e , are periodic events that cannot be subdivided. We

l o o k e d at earlier, is presented in three f o r m s , a c c o r d i n g as ( A ) age is c o l l a p s e d to a

c o u l d use the n u m b e r of elections that a person h a d experienced as a m e a s u r e of his

n o m i n a l m e a s u r e and all k n o w l e d g e of r a n k i n g and unit distance in it is lost; ( B ) age

political exposure, w h i c h w o u l d produce an interval m e a s u r e , with "five e l e c t i o n s "

is c o l l a p s e d to an ordinal m e a s u r e and the k n o w l e d g e of unit distance in it is lost; a n d

being two units higher than "three elections." T h e measure w o u l d be interval but

( C ) age is maintained at an interval l e v e l of m e a s u r e m e n t .

w o u l d not be continuous; we c o u l d not subdivide elections into half-elections, or anything like that. F o r most discrete variables, however, s u c h as " r a c e , " "religion," "region," or " s o c i a l c l a s s , " no unit presents itself by w h i c h categories ( s u c h as

Figure 5 - 4

" A f r i c a n - A m e r i c a n " and " w h i t e " ) c a n be c o m p a r e d . T h u s , with a few exceptions,

100

Age and Participation in 2 0 0 2 Election 100

s u c h as "elections," discrete variables are by their nature either ordinal (if they have an ordering to them) or n o m i n a l . W e s e e , then, that all c o n t i n u o u s m e a s u r e s c a n b e e x p r e s s e d a s i n t e r v a l m e a s u r e s , w h e r e a s a l m o s t all d i s c r e t e v a r i a b l e s are n a t u r a l l y either o r d i n a l o r n o m i n a l . A s a r e s u l t , the l e v e l o f m e a s u r e m e n t w e u s e i s g e n e r a l l y not a p e r s o n a l c h o i c e but i n h e r e s in the thing we are m e a s u r i n g . T h e r e are t w o c a v e a t s to this statement, h o w e v e r :

(1) W e c o u l d a l w a y s i g n o r e i n f o r m a t i o n w e h a v e about a

m e a s u r e , a n d treat i t a s a l o w e r - l e v e l m e a s u r e (the next s e c t i o n , " T h e S i n o f W a s t i n g I n f o r m a t i o n , " a r g u e s a g a i n s t t h i s ) ; a n d ( 2 ) sometimes w e c a n a d d other i n f o r m a t i o n or theory to a n a t u r a l l y n o m i n a l or o r d i n a l m e a s u r e a n d e n r i c h it to a h i g h e r l e v e l than its natural l e v e l : this is d i s c u s s e d in the s e c t i o n after next, " E n r i c h i n g the L e v e l o f P r e c i s i o n i n M e a s u r e m e n t . "

High

Low Age

Age The Sin of Wasting Information 100

T i m e and again data c o l l e c t e d at a higher l e v e l of p r e c i s i o n are c o l l a p s e d to a l o w e r

(C) Age measured intervally

level to s i m p l i f y h a n d l i n g the data, writing reports, and so o n . A g e is often grouped into categories s u c h as " y o u n g e s t , " " y o u n g e r m i d d l e - a g e d , " "older m i d d l e - a g e d , " 75

and " o l d e s t " — a n ordinal m e a s u r e . O r i n c o m e i s grouped into "low," " m i d d l e , " and "high."

c

Esthetically, of course, it w o u l d seem better to k n o w more about a subject rather than less. T h i s alone should be enough to justify our C a r d i n a l R u l e : " U s e measurements that are as precise as possible, given the subject y o u are studying; do not waste informa-

•£ 5 0 CD O CD Q.

tion by imprecise measurement." T h i s esthetic consideration applies both to precision in measures and to precision in measurement. B u t i n a s m u c h as I argued earlier that for

25

practical purposes precision in measures might sometimes be sacrificed, it appears that by itself the esthetic consideration is not compelling. A m o r e important r e a s o n for f o l l o w i n g the C a r d i n a l R u l e , one that applies only to p r e c i s i o n in m e a s u r e m e n t , arises if we are interested in u s i n g the m e a s u r e s to study a relationship b e t w e e n two or more variables. In this c a s e , we c a n do

30

40

60

50 Age

70

80

90

94

66

Problems of Measurement

Chapter 5

In the first case, with age measured nominally, we can see that there is a relationship between age and participation in elections. This is indicated by the fact that different age groups participate at different rates. Only 16 percent of the individuals in category C vote, for instance, as compared with 86 percent of those in category F. If age and participation were not related, we would expect about the same percentage of individuals in each category to vote. With nominal measurement, this is all we can

67

However, the information lost through the low precision in measures in part C is slight compared with the information lost by lowering the precision in measurement in parts A and B. Preserving a high level of precision in measurement deserves a strong priority

in

research.

say—that the variables are or are not related.

Enrichment of the Level of Precision in Measurement

In the second case, with ordinal measurement, we can say a good deal more. We now know the ranking of the different categories of age: C is youngest, E the next to youngest, and then come B, G, A, F, and D. In addition to seeing whether or not the variables are related, we can see the pattern of the relationship. In this case, that pattern is one of increasing participation with increasing age up to some point, and then a reversal, with participation decreasing as age increases. If age were measured intervally, we could say still more about the relationship. Interval measurement adds information about the magnitude of differences in age expressed by the categories. If age were measured intervally, we could see whether or not there was a relationship, we could see the pattern of the relationship, and we also could see the rate at which one variable changed in response to changes in the

So far I have argued that we should try not to drop data carelessly to a lower level of measurement. By a similar argument, we should always try to raise data to a higher level when this is possible. Given a certain amount of boldness and ingenuity, we may sometimes be able to do this. Often, we can enrich our data in this way if we know something more about the data than is reflected in our measurement—in other words, if we have information about the data that would otherwise be wasted. This information may not be enough to provide neat, clean measurement at the higher level. If it were, we probably would have measured at the higher level in the first place. Generally, though, untidy measurement at a higher level is better than neat measurement at a lower level. A few examples may be the best way to show how measurement can be "enriched" in this way, by raising the level of measurement.

other

variable.

In Figure 5-4(c), the bars in the graph have been stretched to take into account the added information that the categories represent the following age groups, measured in a common unit, years: C

18-20

E

21-23

B

24-30

G

31-50

A

51-64

F

65-80

D

81-91

We can now see that the initial increase in participation, between the ages of 20 and 30, is slow; that there is a big jump over about the next ten years; and that participation peaks around age 70, after which there is a decline. The dashed line tracing the path of the bars approximates the pattern we have already seen in Figure 5-2. This suggests an interesting combination of processes: Learning and acquiring the habit of voting during the first years of eligibility causes participation to rise rapidly at first after an initial pause; as this increase levels off, it is followed by a decline, perhaps due to enfeeblement. This is a richer description of the relationship than we could have gotten using the ordinal measurement. Notice that in part C of Figure 5-4, greater precision in measures also would have been useful, especially a finer breakdown of ages above 40. With the greater precision in measures of Figure 5-2, we can see that the peak occurs in the late 70s.

Examples of Enrichment Example 1. You are studying the relationship between the colonial experience of new nations and the stability of democratic institutions in those nations. That is, you want to compare the stability of democratic institutions in former French, British, Dutch, and American colonies, for example. One way to do this would be simply to treat the problem as one of a relationship involving nominal measurement. But it is likely that you have in mind some underlying scale along which the colonial experience varied, depending on which country had done the colonizing, and that you are really interested in the relationship between this scale and the stability of democratic institutions. If you can array the different mother countries along this scale, you can use them as an ordinal or interval measure of the scale. You might, for example, be interested in whether or not the colonizing country tried to assimilate the native population to its own culture and the effect this had on the stability of the resulting institutions. On the scale "level of assimilation," the British colonial experience would rank low and the French quite high. The Dutch and American experiences would fall somewhere between these, with the United States' possibly lower than the Dutch. For this particular purpose, then, the colonizing countries comprise at least an ordinal measure of the level of assimilation. If you could make a reasonably informed guess at the relative differences (such as Dutch rather close to the French; large gap between British and American), you might even be able to bring your data up to the level of rough interval measurement.

68

Chapter 5

Problems of Measurement

Further, you well might want to use a given nominal m e a s u r e m e n t twice or

educational programs:

military academy;

general university,

69

with participation in

m o r e in the s a m e analysis, as a m e a s u r e of different underlying scales. T h u s y o u

R O T C ; general university, without participation in R O T C . T h e s e form an ordinal

m i g h t be interested in predicting the stability of d e m o c r a t i c institutions from t w o

measure of "militariness" readily enough, with the military academy highest and the

variables simultaneously: (1) the level of assimilation at w h i c h the colonizer a i m e d ,

n o n - R O T C p r o g r a m lowest. C a n y o u m a k e a n interval m e a s u r e out o f this r a n k i n g ?

and

(2) the extent of oppression

and violence

during the colonial period.

The

Fortunately, you might have s o m e additional information on the subject. In a

colonizers w o u l d be arrayed differently along these t w o scales, and thus each w o u l d

study of A n n a p o l i s m i d s h i p m e n , R O T C , a n d n o n - R O T C students, for instance, it

represent a different m i x of the t w o variables. T h e U n i t e d States, for e x a m p l e ,

w a s found that 39 percent of the m i d s h i p m e n thought the A m e r i c a n military b u d g -

p r o b a b l y w o u l d fall a b o v e t h e m i d d l e i n a s s i m i l a t i o n , b u t l o w i n o p p r e s s i o n ; F r a n c e ,

et was too small; 10 percent of the R O T C students agreed, as did 4 percent of the

high in assimilation and toward the middle in oppression; Belgium, low on assimila-

n o n - R O T C students (Karsten and others, 1971). If you were willing to a s s u m e that

tion but high on oppression; a n d so o n .

5

at that time the Naval A c a d e m y was typical of the military academies and that there was a linear relationship between the promilitary

Example

2.

You

are

studying

the

relationship

between

religion

and

political

alienation. Religion could simply be measured nominally (Jewish, old-Reformation Protestant, R o m a n Catholic, and so on). Perhaps, however, you have in mind an underlying d i m e n s i o n that causes y o u to expect a relationship b e t w e e n the two variables.

w a s too small, this information w o u l d have a l l o w e d y o u to reconstruct the interval measure implicit in the ordered variable. (For a m o r e detailed explanation of "linear relationship," see the box on p. 71)

You might be thinking in terms of the extent to which one's religion

promotes an apocalyptic view of the universe. If so, then as in E x a m p l e 1, you could take the different religions ranked by their apocalyptic content as an ordinal o r i n t e r v a l m e a s u r e o f t h i s s c a l e . J e w s w o u l d fall n e a r t h e b o t t o m o f s u c h a s c a l e and fundamental Protestants, near the top. Alternatively, your theory might be that religious groups, simply as social

If the degree of promilitary orientation in R O T C programs increased support of higher budgets by 6 percentage points over n o n - R O T C students, but the Naval A c a d e m y increased budget support by 29 percentage points over R O T C students, then (given your assumptions) the interval b e t w e e n military academies and R O T C must have been 4.83 times as great as the interval between R O T C and n o n - R O T C . This is demonstrated geometrically in Figure 5-5.

organizations, build involvement a m o n g their m e m b e r s . In this case you could array the different religions in t e r m s of the size a n d " c l o s e n e s s " of their congregations, h o w d e m o c r a t i c their legal structure is, or whatever. N o t e that the "right" ordinal or interval arrangement of the n o m i n a l categories d e p e n d s on which underlying scale y o u w a n t t o t a p . T h u s , J e w s w o u l d fall l o w o n t h e " a p o c a l y p t i c " s c a l e b u t h i g h o n t h e "closeness" scale. In working with ordinal variables (either "enriched" nominal variables or variables which obviously are ordered from the outset), a c o m m o n unit m a y appear immediately, and you can treat the variables readily and directly as interval m e a s ures. F o r instance, abstracting the d e g r e e of " c l o s e n e s s " from religious affiliation suggests o n e ready interval m e a s u r e — a v e r a g e size of c o n g r e g a t i o n s — a l t h o u g h this admittedly w o u l d be only a r o u g h m e a s u r e w h o s e validity w o u l d be questionable. Often, to enrich ordinal data, we m u s t use a g o o d deal of ingenuity, as in the next example.

orientation of an educational

p r o g r a m a n d the p e r c e n t a g e o f its s t u d e n t s w h o b e l i e v e d that t h e m i l i t a r y b u d g e t

Here I have drawn two of the m a n y possible linear relationships between the "military c o n t e n t " of e d u c a t i o n a n d support for the defense budget. It should be clear that the s a m e p r i n c i p l e w o u l d h o l d for a n y linear relationship. B e c a u s e y o u know

the

scores

on

budget

support

for

non-ROTC,

ROTC,

and

the

Naval

A c a d e m y , y o u c a n p r o j e c t a c r o s s t h e g r a p h t o s e e w h e r e t h e y s h o u l d fall o n t h e l i n e showing

the

relationship.

Non-ROTC

falls

at A,

ROTC

at

B,

and

the

Naval

A c a d e m y at C. N o w look d o w n the graph to see what scores on "military content" must be associated with positions A, B, and C on the line. These scores are A ' , B ' , and C. Of course, you do not have units in advance by which to measure military c o n t e n t , b u t i t i s a p p a r e n t t h a t w h a t e v e r u n i t s y o u c o n c e i v e of, a n d w h a t e v e r l i n e a r relationship between the t w o variables exists, the distance from B' to C must be a p p r o x i m a t e l y five t i m e s t h e d i s t a n c e f r o m A ' t o B . (To b e exact, t h e ratio o f t h e distances is 4.83.) T h i s g i v e s y o u all y o u n e e d t o c o n s t r u c t a n interval m e a s u r e — t h a t is, a n e s t i m a t e

You w a n t to m e a s u r e the extent to w h i c h different educational p r o g r a m s

of the relative differences (intervals) b e t w e e n scores of the measure. B e c a u s e an interval

tend to encourage pro- or antimilitary attitudes. You can distinguish three types of

m e a s u r e d o e s not a s s u m e that a zero point is k n o w n , y o u can arbitrarily pick the unit in

Example 3 .

w h i c h t o e x p r e s s y o u r m e a s u r e . Y o u m i g h t a s s i g n 1.0 t o n o n - R O T C a n d 2 . 0 t o R O T C , in w h i c h case Naval A c a d e m y w o u l d have to be 6.83. You might assign 4.0 to non5

Notice that the manner in which measurement of nominal variables has been enriched in these examples parallels the "dimensional analysis" that I urged in Chapter 3. Like vague, multidimensional concepts, nominal classifications involve an infinite number of dimensions. The various types of colonial experience we cited vary on degree of assimilation, extent of oppression, speed of economic development, geopolitical position, or what have you. Abstracting the appropriate ordered variable(s) from a nominally measured variable is much the same as abstracting the appropriate dimension(s) from a set of multidimensional concepts.

R O T C a n d 8.0 t o R O T C ; N a v a l A c a d e m y w o u l d t h e n h a v e t o b e 2 7 . 3 2 . M o s t s i m p l y , t h e scores a s s i g n e d m i g h t b e n o n - R O T C , 4 ; R O T C , 10; a n d a c a d e m y , 3 9 . Example 4 .

In E x a m p l e 3, a rather p r e c i s e p i e c e of o u t s i d e i n f o r m a t i o n w a s u s e d

to tell h o w long t h e intervals s h o u l d b e . O f t e n s u c h a p r e c i s e g u i d e is s i m p l y

70

Chapter

5

Problems of Measurement

71

than others. Y o u c o u l d create a n a c c e p t a b l e , t h o u g h quite r o u g h , interval m e a s u r e o f the n a t i o n a l p o w e r o f the v a r i o u s c o u n t r i e s s i m p l y b y a s s i g n i n g arbitrary interval lengths i n a c c o r d a n c e w i t h y o u r g e n e r a l i d e a o f the m a g n i t u d e s i n v o l v e d . Y o u m i g h t c h o o s e , for e x a m p l e , C o s t a R i c a , 1 ; S p a i n , 7 ; G r e a t B r i t a i n , 10; a n d R u s s i a , 4 0 . T h e m e a s u r e i s r o u g h a n d s u b j e c t i v e , but i t c o u l d b e r e f i n e d b y b r i n g i n g t o bear the i m p r e s s i o n s o f m o r e k n o w l e d g e a b l e o b s e r v e r s w i t h r e g a r d t o r e l a tive n a t i o n a l p o w e r . M o s t i m p o r t a n t , m e a s u r i n g i n this w a y d o e s not i g n o r e the v a l u a b l e c o l l e c t i o n o f o u t s i d e i n f o r m a t i o n — m o s t o f i t admittedly u n d i g e s t e d — that y o u a l r e a d y p o s s e s s . T h e interval measures y o u derive in E x a m p l e s 3 and 4 m a y not leave y o u fully satisfied. B e c a u s e of the indirect c h a i n of reasoning in E x a m p l e 3, and the assumptions required for it, and because of the arbitrary decisions to be made in E x a m p l e 4, there is plenty of r o o m for errors to occur. T h e question remains: A r e y o u better off w i t h s u c h rough interval measures, or s h o u l d y o u opt to ignore the size of intervals and treat the variables as ordinal m e a s u r e s ? G i v e n the wider range of theoretical c o n c e r n s that one c a n c o v e r using interval data, I think the a n s w e r is obvious. It m a k e s sense to use all the information we have about a question, even if we must supplement that information with assumptions and h u n c h e s . T h e alternative is to ignore a portion of w h a t we know. T a k i n g the " s a f e r " course a l l o w s us to state our results with more confidence but reduces the importance of what we have to say. E n r i c h i n g m e a s u r e m e n t in this w a y is a r e s e a r c h technique for w h i c h the r e s e a r c h e r ' s creativity and ingenuity are c r i t i c a l . O n e m a r k of a good r e s e a r c h e r is that she boldly seeks out all c h a n c e s — n o t j u s t tl le o b v i o u s or safe o n e s — t o r a i s e the level of m e a s u r e m e n t in her w o r k .

Linear Relationship

u n a v a i l a b l e , but y o u still c o u l d k n o w m o r e than n o t h i n g about the length o f the i n t e r v a l s . C o n s i d e r a p r o b l e m i n w h i c h y o u w a n t t o m e a s u r e the n a t i o n a l p o w e r o f Costa R i c a , Spain, Great Britain, and R u s s i a .

6

T h e order i s c l e a r , a n d a t i m i d

r e s e a r c h e r w o u l d be content to treat this as an o r d i n a l v a r i a b l e . On the other h a n d , y o u k n o w that the intervals are not e q u a l , and y o u k n o w w h i c h i n t e r v a l s are greater 'Although I have dubbed the variable "national power," for most purposes this would be broken down more profitably into component dimensions, such as "diplomatic influence," "economic power," and "military power."

A r e l a t i o n s h i p b e t w e e n t w o v a r i a b l e s is linear if o n e v a r i a b l e c h a n g e s by a c o n s t a n t a m o u n t w i t h a c h a n g e of o n e unit in the second variable, regardless of the value of the s e c o n d v a r i a b l e . T h e r e l a t i o n s h i p in g r a p h A is linear; y inc r e a s e s b y o n e - h a l f unit w h e n x c h a n g e s b y o n e unit, n o m a t t e r w h a t v a l u e x starts w i t h . T h e r e l a t i o n s h i p in g r a p h B is n o n l i n e a r . H e r e , if x is h i g h , a unit c h a n g e in x p r o d u c e s relatively little c h a n g e in y; if x is low, a u n i t c h a n g e in x p r o d u c e s relatively g r e a t c h a n g e in y. T h u s , the r e l a t i o n s h i p i n g r a p h A can b e e x p r e s s e d b y a s t r a i g h t line, i n a s m u c h as t h e r e l a t i o n s h i p is the s a m e at all v a l u e s of x. T h e r e l a t i o n s h i p in g r a p h B m u s t be e x p r e s s e d by a c u r v e w h i c h tilts differently at different v a l u e s of x, inasmuch as the relationship between the two variables is different for different v a l u e s of x.

72

Chapter

5

Problems of Measurement

Quantifiers and Nonquantifiers Again

FURTHER DISCUSSION

This discussion of precision in measures and precision in measurement has centered on the work of "quantifiers" (those who work by preference with objective data, especially with numerical data), simply because it was more convenient to introduce it in that way. But the same general conclusions hold for "nonquantifiers" as well. To return to the quotation from Eckstein with which I introduced this chapter, it is clear that he was interested in doing the same things as quantifiers do, but in a different way. He wanted to measure variables and to assess relationships between those variables. His subject matter, however, rarely allowed him to use objective or numerical measures. For many variables in which he was interested, he could use only vague descriptions, such as "rather strong sense of community," "highly developed team spirit," and so on. Most of these descriptions had to depend on his subjective judgment rather than on impersonal outside indicators. Can this sort of work be described in terms of precision in measures and precision in measurement? Do the rules I have prescribed in this chapter hold for this sort of work as they do for more quantitative studies? First, we may note that measuring variables nonquantitatively requires the same sort of boldness that I urged in enriching data to a higher level of measurement. More important, we can see that this is primarily a problem of precision in measures, not of precision in measurement. A nonquantifier can measure things only approximately, but there is no particular reason why such research cannot approximate interval measurement. For example, a student of Chinese politics might suggest that the higher a Chinese politician rises in the hierarchy, the less likely he is to press for the interests of his region. This statement approximates a relationship between two variables measured at the ordinal level ("height in the hierarchy" and "likelihood of lobbying"). Working more boldly and imaginatively, another student might approximate interval measurement. She might state, for example, that the higher a Chinese politician rises in the hierarchy, the less likely he is to press for the interest of his region; that this shift occurs very sharply as he moves from some particular level of the hierarchy to another; that the shift continues from there on up the hierarchy at a diminished rate; and finally, that near the top of the hierarchy, the process reverses itself, with the men at the top feeling secure enough to concern themselves more than those immediately below them with the interests of their regions. This is certainly a richer statement than the earlier version, but it is no more "quantified" than the other. It is still based on subjective descriptions, made in the absence of objective indicators. The only difference is that in the second case the researcher enriches the level of measurement at which she is working. 7

8

7

Let me repeat my earlier admonition. Being a "quantifier" or "nonquantifier" is not an either/or question. The extent to which a researcher uses objective data is generally not a matter of personality but rather of the subject he is studing and the data available for that subject. 8

73

I have no idea whether this is a reasonable theory. I have constructed it merely as an example of the wide applicability of pseudo-interval measurement.

A readable presentation along the lines developed in this chapter is Edward R. Tufte's "Improving Data Analysis in Political Science" (1968-1969). You should note that in this chapter I have presented a fairly radical position in favor of upgrading levels of measurement, a position that would not be accepted by all social scientists. To the extent that one is worried about measurement error, either random or nonrandom, one should be concerned about the possibility that "enriching" a level of measurement may insert either sort of error into the new measure. Personally, I think the risk is often justified; however, many scholars believe that the costs in carrying out such "enrichment" are greater than the benefits. A fairly technical paper that urges researchers to take seriously the risk of inserting measurement error by treating ordinal variables as if they were interval (an extrapolation similar to enrichment but riskier and less well thought out) is that by Wilson (1971). Another excellent article, a bit technical, is by Carter (1971). Two questions you might consider are: 1. Why would the technique in Example 3 not be appropriate if the relationship between "military content" and "support for defense budget" were nonlinear? What might you do in this case? 2. "Ratio" measurement is interval measurement for which, in addition to being able to measure the relative interval between different values of the measure, it is possible to assign the value zero to some point on the measure (see footnote 3). One difference between interval and ratio measurement is that whereas it is possible to add and subtract interval measures, it is possible to add and subtract, multiply, and divide ratio measures. It is possible to say that a given score is twice as great as another score, given ratio measurements; this is not possible with simple interval measurement. Fahrenheit temperature is an example of an interval measure that is not a true ratio measure, even though "zero" is arbitrarily assigned to one point on the scale. It is not true, for example, that a temperature of +20°F is half as "hot" as +40°F. In this chapter I pointed out that the different levels of measurement were important because it was possible to make a greater variety of statements about relationships between variables that were measured in "advanced" ways. What statements might be made about relationships between ratio-measured variables that could not be made about relationships between interval-measured variables?

Causal Thinking and Design of Research

Chapter 6

75

CAUSALITY: AN INTERPRETATION

T h e m o s t important thing to note about c a u s a l t h i n k i n g is that it is an interpretation of reality. In this regard, the a s s e r t i o n of a causal relationship differs f r o m the m e r e a s s e r t i o n of a relationship, w h i c h is a rather o b j e c t i v e statement. In the e x a m p l e of s o c i a l c l a s s and the vote, for i n s t a n c e , there is o b j e c t i v e e v i d e n c e for the e x i s t e n c e of a r e l a t i o n s h i p , but a c a u s a l interpretation of this r e l a t i o n s h i p is m u c h m o r e

QoJ)

s u b j e c t i v e . It might be a r g u e d , for i n s t a n c e , that c l a s s d o e s not produce the vote, but that both are p r o d u c e d by s o m e t h i n g e l s e — t h e e t h n i c , r e l i g i o u s , and r a c i a l c o n f l i c t s that h a v e s u r f a c e d often i n A m e r i c a n p o l i t i c s . T h u s s o m e o n e c o u l d a r g u e — a n d i t w o u l d not be an u n r e a s o n a b l e a r g u m e n t — t h a t c l a s s is not a c a u s e of the vote. R a t h e r , that p e r s o n might say, c e r t a i n ethnic groups tend to vote for the D e m o c r a t s ; and these

same groups,

s i m p l y by a c c i d e n t ,

a l s o tend to be w o r k i n g c l a s s .

T h e r e f o r e , the c o i n c i d e n c e o f c l a s s and party votes i s j u s t t h a t — a c o i n c i d e n c e . D i s t i n g u i s h i n g b e t w e e n this v e r s i o n of the r e l a t i o n s h i p and the m o r e c o m m o n v e r s i o n is at least partly a matter of j u d g m e n t . A l m o s t every situation in w h i c h we w i s h to m a k e c a u s a l statements is similar to this example. T h e question of whether or not there is a relationship is objectively T h u s far everything we have l o o k e d at in this book has been j u s t i f i e d in terms of h o w

testable. T h e question of whether the relationship is a c a u s a l one, and of w h i c h variable

u s e f u l it is in establishing relationships b e t w e e n v a r i a b l e s . It is time n o w to l o o k

c a u s e s w h i c h , requires an interpretation. G e n e r a l l y speaking, all that we k n o w directly

m o r e c l o s e l y at what a " r e l a t i o n s h i p " i s , and h o w we interpret it.

from our data is that two variables tend to o c c u r together. To read causality into s u c h

T w o variables are related if certain v a l u e s of one variable tend to c o i n c i d e w i t h certain v a l u e s of the other v a r i a b l e . T h u s , s o c i a l c l a s s and party vote are related in

c o - o c c u r r e n c e , we must add something more, although as y o u w i l l see, it is possible to design the research in w a y s that c a n help us significantly in doing this.

the U n i t e d States, b e c a u s e those w h o vote R e p u b l i c a n tend to be f r o m the m i d d l e

C o n s i d e r two further e x a m p l e s : I f w e see that people w h o s m o k e regularly

and upper c l a s s e s . W e have seen m a n y s u c h e x a m p l e s o f relationships i n earlier

h a v e a greater i n c i d e n c e of heart d i s e a s e than n o n s m o k e r s , we might c o n c l u d e that s m o k i n g c a u s e s heart d i s e a s e . If we see that those U . S . senators w h o c o n f o r m to the

chapters. the

i n f o r m a l rules of the Senate tend to be the ones w h o s e bills get p a s s e d , we might

T h e example

c o n c l u d e that c o n f o r m i t y is r e w a r d e d in the Senate. In both c a s e s , we o b s e r v e that

o f c l a s s a n d v o t i n g , noted a b o v e , i s a n e x a m p l e o f a c a u s a l r e l a t i o n s h i p . W e f e e l

two p h e n o m e n a tend to o c c u r together ( s m o k i n g and heart d i s e a s e , c o n f o r m i t y and

If,

in

addition,

we c o n s i d e r that the

values

of one

v a r i a b l e produce

v a l u e s of the other v a r i a b l e , the r e l a t i o n s h i p is a causal relationship.

that there is s o m e t h i n g about a p e r s o n ' s s o c i a l c l a s s that m a k e s that p e r s o n

s u c c e s s ) . O u r interpretation of this is that one of the p h e n o m e n a c a u s e s the other.

m o r e l i k e l y to vote in a c e r t a i n w a y . T h e r e f o r e , we s a y that s o c i a l c l a s s is a

T h i s interpretation is a subjective one.

" c a u s e " of party vote. It is not m e r e l y true that the t w o v a r i a b l e s tend to c o i n c i d e ;

We cannot a l w a y s m a k e a c a u s a l interpretation w h e n two p h e n o m e n a tend to

they tend to c o i n c i d e because values of the one tend to produce distinct values of the other.

c o i n c i d e . T h e notion of c a u s e i n v o l v e s m o r e than that. W i n t e r does not c a u s e s p r i n g , although the one f o l l o w s the other regularly. S i m i l a r l y , hair c o l o r does not c a u s e

E m p i r i c a l , theory-oriented p o l i t i c a l r e s e a r c h i s almost e x c l u s i v e l y c o n c e r n e d

political party preference, although it is probably true in the U n i t e d States that

w i t h causal relationships. As I pointed out earlier, a theory in its s i m p l e s t f o r m

b l o n d s , w h o are relatively l i k e l y to be w h i t e and Protestant, are m o r e apt than

u s u a l l y c o n s i s t s of three things: independent variables (those we think of as d o i n g

brunettes to be R e p u b l i c a n s . To qualify as a " c a u s a l " relationship, the c o i n c i d e n c e of

the " p r o d u c i n g " ) , dependent v a r i a b l e s (those we think of as being " p r o d u c e d " ) , and

two p h e n o m e n a must i n c l u d e the idea that one of them produces the other.

c a u s a l statements l i n k i n g the t w o (refer again to p. 14).

A good e x a m p l e of the difficulty of a s c r i b i n g c a u s a l direction is the relationship

In this chapter we first d i s c u s s the i d e a of c a u s a t i o n , and then f o l l o w this

between central bank independence and inflation. In general, e c o n o m i s t s think that if

u p w i t h s o m e ideas o n r e s e a r c h d e s i g n . R e s e a r c h d e s i g n — t h e w a y i n w h i c h w e

central b a n k s ( s u c h as the F e d e r a l R e s e r v e in the U n i t e d States) are relatively

structure the gathering of d a t a — s t r o n g l y affects the c o n f i d e n c e w i t h w h i c h we c a n

independent of political control, they w i l l use monetary p o l i c y more a g g r e s s i v e l y to

put a c a u s a l interpretation on the results of r e s e a r c h .

fight inflation. C e r t a i n l y , w h e r e central b a n k s are independent, inflation is low. In a

74

76

Causal Thinking and Design of Research 77

Chapter 6

1993 study, the three countries with the most politicized banks had averaged almost 8 percent inflation from 1955 to 1985; the three countries whose central banks were most independent had averaged only about 3.75 percent inflation.' We see, then, that there is a relationship between the two; but is it true, as the economists believe, that bank independence causes low inflation? Batalla (1993) suggests that the opposite causal interpretation may be true. Reviewing the history of the 1920s and 1930s in Latin America, he notes that many countries had established autonomous central banks by the late 1920s, but that when high rates of inflation hit in the mid-1930s, most of them took away that autonomy. That is, low inflation rates allowed governments to tolerate independent central banks, while high inflation rates led the governments to pull the banks under their control. So, which causes which? Do independent central banks give us low inflation, or does low inflation give us independent central banks? If ascribing cause to the coincidence of two things is so tricky, why do we bother with the notion of causation in our theories? What difference does it make whether class causes voting for a certain party, or whether the coincidence of class and party is due to something else that causes both of them? Remember that the ultimate purpose of theories is to give us levers on reality, some basis for choosing how to act. If A and B coincide but A does not cause B, changing A will not change B. Coincidence without cause gives you no lever.

ELIMINATION OF ALTERNATIVE CAUSAL INTERPRETATIONS A causal interpretation is something that cannot come solely from our observations. But by setting up a study in certain ways and by manipulating the data appropriately, we can settle some of the problems in making causal interpretations. In our previous example of hair color, for instance, we might have looked only at WASPs, and compared blonds and brunettes. If we then found that blond WASPs did not tend more often than brunette WASPs to be Republican, we could infer that hair color did not cause party preference. In general, where we think that a third variable is causing two other variables to coincide accidentally, as in this hair color example, we can use our data to test the relationship. By holding constant the third variable, we can see whether it has led the original two variables to coincide in such a way as to resemble a causal relationship. Thus, if we artificially hold social position constant by looking only at WASPs, and now blonds and brunettes no longer differ in their politics, we can infer that the difference in voting was due not to the difference in hair color (which was merely a coincident variable) but to the difference in socioethnic background. We can then conclude that hair color does not cause political preference. 2

Thus there are techniques by which we can manipulate our data to eliminate some of the alternative causal interpretations of a relationship. But there is always one question about causation that remains purely subjective and that cannot be resolved completely by any techniques of data handling. Given that two variables are causally related, which of the variables causes which? Suppose that two phenomena coincide and that there appears to be a causal relationship between them. Only you can decide which is the cause and which, the result. One useful convention in Western culture—but it is only a convention, even though it is so well established that it seems "natural" to us—is that causation works forward in time. Accordingly, if we can establish that change in one of the variables precedes change in the other, and if we are sure that there is causation between the two, it is clear which variable must be the cause. For example, we assume that pulling a trigger causes the shot that follows, rather than vice versa. Although this convention frequently simplifies things for the researcher, there are many instances in which it cannot be used. Survey research, in which variables usually are measured just once and in a single interview, is a case in point. If voters who like the Republican party also tend to oppose welfare programs, which causes which? We might think that voters choose the party that offers the policies they prefer, but it might also be that voters choose the Republican party for other reasons, such as foreign policy, and then are influenced by the party's leaders to adopt its position on welfare programs as their own. 3

Summary Let me pull together the argument to this point. It generally is not enough for us to note that two phenomena coincide. We generally also want to interpret why they coincide. There are three interpretations available to us, and our choice from among them is ultimately subjective. /. Causation is not involved at all. The phenomena coincide because of logical necessity; that is, their coincidence is tautologically determined. Thus, by definition, spring follows winter, yet we do not think of winter as producing spring. A slight variation of this often occurs in the social sciences. It often happens that two slightly different measures of the same concept coincide. We would expect them to coincide, simply because they measure the same thing; we do not think of either of them as causing the other. For example, members of Congress who vote to increase aid to education tend also to support increases in welfare spending. This is not because their votes on one issue cause them to vote the way they do on the other. Rather, both votes are an expression of their general disposition to spend money on social programs. We must decide from outside the data at hand whether a coincidence of two phenomena is of this type or whether it involves causation. 3

'The Economist, November 20. 1993, p. 94. 2

The technique of holding constant is discussed in greater detail at the conclusion of the chapter.

A n example of a cultural tradition in which causation does not necessarily work forward in time is that of the Old Testament, whose writers believed that some people could prophesy what was to come in the future. In effect, the future event caused the prior prophecy to occur.

78

Causal Thinking and Design of Research

Chapter 6

79

2. The relationship we observe is a result of outside factors that cause the two phenomena at hand, and thus neither of these phenomena causes the

decentralize decision making. After the reform, another week's tabulation shows increased output. Conclusion: Decentralized decision making increases output.

other. The study of hair color and party preference was an example of this. By setting up the study appropriately, we can control for various outside factors in order to concentrate on the relationship in question. In the hair color example, such a control was used. To this extent, we can see from the data whether the coincidence of the phenomena is of this sort. But we are still not exempt from making assumptions, for we must first have assumed the outside factor(s) causally prior to the two coincident phenomena. This is not always an easy decision to make. If it is possible to set up a true experiment (described in the next section), we can eliminate this possibility. But this is often not possible in "field" social sciences such as political science or sociology.

2. Reagan's victory over the Soviet Union. During the Reagan administration (1980-1988) the United States steadily increased its military spending. Over the three years from 1989 to 1991 the Soviet Union collapsed, which conservatives hailed as a victory for Reagan's policies. Conclusion: The economic strain of matching Reagan's military buildup had been too much for the Soviet system, and had led to its collapse and the end of the Cold War.

3. One of the phenomena causes the other. Here we have a true causal statement. We are still not finished making assumptions, of course, for we must decide which of the phenomena is the cause and which the effect. That is ultimately a subjective decision, though often we are aided in making it by the convention that causation must run forward in time. As I have said so often in this book, one of the pleasures of research is that nothing in it is automatic. Even the most "quantitative" techniques do not take away our obligation and our right to be creative and imaginative. The fact that causal analysis is ultimately subjective may trouble us—objectivity always seems more comforting than the responsibility imposed by subjective judgment—but in a way it is also a great comfort, inasmuch as it keeps us, and what we do with our minds, at the heart of our research.

A FEW BASICS OF RESEARCH DESIGN It should be evident from the discussion so far that the basic problem in causal analysis is that of eliminating alternative causal interpretations. Whenever two variables vary together (are related, coincide), there is a variety of causal sequences that might account for their doing so. A might cause B, B might cause A, both A and B might be caused by something else, or there might be no causation involved. Our task is to eliminate all but one of these, thus leaving an observed relationship, together with a single causal interpretation of it. Some of these alternatives can be eliminated only if we make assumptions from outside the actual study. But we also can design the study in such a way that certain alternatives are impossible. This will leave an interpretation that is dependent on fewer subjective assumptions and can thus lend a greater measure of certainty to the results. Consider these examples:

3. Organizing the poor. In anticipation of a major campaign to organize the poor of a city, a survey is taken among them to measure their interest in politics. At the end of the organizing campaign, the same people are asked the same questions a second time. It turns out that those who were contacted by the campaign workers have indeed acquired an increased interest in politics, compared with those who were not. Conclusion: The campaign has increased the political awareness of the poor.

4. Tax-reform mail.

The Congressional Quarterly reports the proportion of each senator's mail which favored tax reform. Comparing these figures with the senators' votes on a tax-reform bill, we see that senators who had received relatively favorable mail tended to vote for the bill, whereas those who had received relatively unfavorable mail tended to vote against it. Conclusion: How favorable a senator's mail was on tax reform affected whether or not she voted for it. 5. Presidential lobbying. In an attempt to measure his influence over Congress, the president randomly selects half the members of the House. He conducts a straw vote to find out how all the members of the House intend to vote on a bill important to him. He then lobbies intensively among the half he has randomly selected. In the final vote in the House, the group that he had lobbied shifted in his favor compared with what he could have expected from the earlier straw vote; the other half voted as predicted from the straw vote. Conclusion: His lobbying helped the bill. Let us look at the design of these studies to see how many alternative causal interpretations each can eliminate.

Designs Without a Control Group In the first two examples, the design is of the form: 1. M e a s u r e the d e p e n d e n t v a r i a b l e . 2 . O b s e r v e that t h e i n d e p e n d e n t v a r i a b l e o c c u r s . 3. M e a s u r e the dependent variable again.

/. Agency study. An organizational analysis of a government agency is made in which workers keep track of their output for a week. The organization is then restructured to

4 . I f t h e d e p e n d e n t v a r i a b l e h a s c h a n g e d , a s c r i b e that t o t h e o c c u r r e n c e o f t h e i n d e p e n d e n t variable.

80

Causal Thinking and Design of Research 81

Chapter 6 T h u s , in " A g e n c y Study," (1) the w o r k e r s ' o u t p u t is tabulated; (2) the o r g a n i -

Nations has affected the foreign policy of every country in the world since 1945.

zational structure is d e c e n t r a l i z e d ; (3) the w o r k e r s ' o u t p u t is o n c e a g a i n tabulated;

H o w c a n a s t u d e n t o f i n t e r n a t i o n a l p o l i t i c s d i s t i n g u i s h its effect o n f o r e i g n p o l i c y

a n d ( 4 ) t h e c o n c l u s i o n i s r e a c h e d . T h i s k i n d o f d e s i g n o p e r a t e s without a control

from

group. A s a r e s u l t , t h e r e a r e a n u m b e r o f a l t e r n a t i v e c a u s a l s e q u e n c e s t h a t m i g h t h a v e

w e a p o n s , the l i b e r a t i o n o f f o r m e r c o l o n i e s , a n d s o o n , all o f w h i c h h a v e h a p p e n e d

p r o d u c e d the s a m e result.

at the s a m e time? O n e cannot, of course. It is simply not possible to provide a

F o r e x a m p l e , a plausible alternative explanation for the increased productivity m i g h t b e that the initial m e a s u r e m e n t o f p r o d u c t i o n , i n w h i c h e a c h w o r k e r k e p t track

the

effects

of the

Soviet-American

rivalry,

the

development

of atomic

control g r o u p of c o n t e m p o r a r y countries for w h i c h the U n i t e d N a t i o n s has not existed.

of output for a w e e k , focused the w o r k e r s ' attention on productivity in a w a y that had

T h e s a m e general alternative explanation (that other things h a p p e n i n g at the

not been done before, leading them to improve their productivity. In other words,

s a m e time could have been the true cause) also might have applied to "Agency

i t w a s n o t t h e d e c e n t r a l i z a t i o n o f t h e a g e n c y , b u t t h e s t u d y itself, w h i c h c a u s e d productivity to rise.

Study."

If

something

else

had

happened

between

the

two

measurements

of

productivity—the weather improved, Christmas came, the president urged greater

4

H a d the study included a second agency as a control (see the next section), o n e in

productivity, or whatever—this might have been the true cause of the increased

w h i c h o u t p u t w a s m e a s u r e d a t t h e s a m e t i m e s a s i n t h e first a g e n c y b u t i n w h i c h t h e r e

production. Again, using a second agency as a control could eliminate such alterna-

w a s no decentralization, the alternative explanation w o u l d not have been plausible.

tive e x p l a n a t i o n s .

T h a t is, i f t h e i n c r e a s e d p r o d u c t i v i t y i n " A g e n c y S t u d y " h a d b e e n d u e s i m p l y t o t h e act

S t u d i e s w i t h o u t a c o n t r o l g r o u p p o p u p all t h e t i m e . O n J a n u a r y 4 , 2 0 0 4 , t h e

o f m e a s u r i n g , p r o d u c t i v i t y i n t h e c o n t r o l a g e n c y (in w h i c h t h e s a m e m e a s u r e m e n t s

Minneapolis Star Tribune h e a d l i n e d a f r o n t - p a g e s t o r y : " S t a t e S e x E d N o t W o r k i n g ,

w e r e t a k e n a s i n t h e first a g e n c y ) a l s o s h o u l d h a v e i n c r e a s e d . A c c o r d i n g l y , i f w e f o u n d

Study Finds." T h e story reported a study by the M i n n e s o t a D e p a r t m e n t of Health,

that productivity increased m o r e in the reorganized agency than in the control agency,

which in 2001 had asked students in grades 7 and 8 of three schools that h a d adopted

w e w o u l d k n o w that this c o u l d not h a v e b e e n b e c a u s e o f t h e act o f m e a s u r i n g , for

an abstinence-only sex education curriculum (not teaching contraception or safe sex,

both

but rather providing materials to encourage students to abstain), whether "at any time

agencies

would

have

undergone

the

same

measurements.

That

particular

alternative interpretation w o u l d h a v e b e e n eliminated by the design of the study. In

i n y o u r life h a v e y o u e v e r h a d s e x ( i n t e r c o u r s e ) ? " T h e y t h e n c a m e b a c k i n 2 0 0 2 a n d

conducting the study without a control, the alternative interpretation can be eliminated

r e p e a t e d the q u e s t i o n for t h e s a m e g r o u p of students, n o w in g r a d e s 8 a n d 9. F r o m

only by a s s u m i n g it away, w h i c h s e e m s very risky.

2001 to 2 0 0 2 the n u m b e r saying that they had ever h a d sexual intercourse rose from

"Reagan's victory over the Soviet U n i o n " provides an example of another sort of alternative explanation that m a y be plausible in studies without a control group. It is quite possible that other things that occurred b e t w e e n the m e a s u r e -

5.8 to

12.4 percent.

T h e conclusion of the article w a s that the abstinence-only

c u r r i c u l u m h a d failed. An

abstinence-only

c u r r i c u l u m m i g h t o r m i g h t n o t fail t o r e d u c e

sexual

m e n t of the two events ( 1 9 8 0 - 1 9 8 8 and 1 9 8 9 - 1 9 9 1 ) w e r e the cause of the Soviet

i n t e r c o u r s e , but we c a n n o t tell f r o m this s t u d y w h i c h is the c a s e . S t u d e n t s of that

Union's collapse. Reform in C o m m u n i s t China, the growth of Muslim minorities

age, as they grow a year older, are in any case probably m o r e likely to have sexual

in

as

intercourse than when they were younger. A control group of students of the same

alternative c a u s e s of the Soviet U n i o n ' s collapse. If it w e r e possible to find as a

a g e w h o h a d h a d a different c u r r i c u l u m for sex e d u c a t i o n w o u l d h a v e b e e n easy to

control g r o u p a similar system that w e n t through the s a m e other factors but was

add to the study, but without it we have no idea w h e t h e r the students with the

not

the

Soviet Union,

engaged

explanations

in

an

feuds

arms

could be

among

race

tested

with

Soviet leaders—all

the

and perhaps

United

might

States,

eliminated.

then

be

proposed

these

alternative

But of course,

no

such

system exists. T h i s is a g o o d e x a m p l e of h o w difficult it m a y s o m e t i m e s be to i n c l u d e a

abstinence-only

curriculum initiated

sexual

activity

at a greater rate than they

would have done without the curriculum, at a lesser rate, or at the s a m e rate. We s i m p l y c a n n o t tell

whether the abstinence-only curriculum resulted in reduced

sexual intercourse.

control group in a design. S o m e circumstances just do not yield parallel cases that

(The problem of interpreting results w a s also exacerbated in this case by the

can be c o m p a r e d . As another example, consider that the existence of the United

q u e s t i o n , w h i c h a s k e d s t u d e n t s w h e t h e r a t a n y t i m e i n t h e i r life t h e y h a d e v e r h a d s e x u a l i n t e r c o u r s e . T h e 5.8 p e r c e n t w h o a n s w e r e d affirmatively i n 2 0 0 1 m u s t h a v e a n s w e r e d affirmatively again in 2002, so the percent responding " y e s " in 2 0 0 2

4

A famous example of this sort is the Hawthorne study, in which an attempt was made to measure how much productivity increased when factories were made brighter and more pleasant. As it turned out, the groups of workers who were placed in better surroundings did show major increases in productivity. But so did control groups whose surroundings had not been improved. The novelty of taking part in an experiment, the attention paid to the workers, and increased social cohesiveness among those groups chosen for the experiment—all these raised productivity irrespective of the experimental changes in physical working conditions that were made for some (but not all) of the groups. See Roethlisberger and Dickson (1939).

c o u l d h a v e only risen or s t a y e d the s a m e ; it w a s n o t p o s s i b l e for the p e r c e n t a g e to decline. So, apparent failure of the p r o g r a m w a s baked into the study from the start. A better q u e s t i o n w o r d i n g , w h i c h w o u l d h a v e b e e n m o r e sensitive to the impact of the new program, w o u l d have been: " D u r i n g the past year, have you had sex [intercourse]?")

82

Chapter 6

Causal Thinking and Design of Research 83 a c c o m p l i s h e s this i s a

U s e of a Control G r o u p

true experiment; but b e f o r e g o i n g o n to d i s c u s s t h i s , let m e

d i s c u s s a p o o r c o u s i n o f the n a t u r a l e x p e r i m e n t .

The natural experiment.

" O r g a n i z i n g the P o o r " i s an e x a m p l e o f a d e s i g n i n

w h i c h a control group h a s b e e n added to handle the sorts of p r o b l e m s we j u s t encountered. It i s a

natural experiment,

a d e s i g n i n w h i c h a test group (that i s , a

The natural experiment without premeasurement. T h i s i s a d e s i g n i n w h i c h no m e a s u r e m e n t s are m a d e before subjects are e x p o s e d to the independent variable. T h i s d e s i g n f o l l o w s the f o r m :

group e x p o s e d to the independent v a r i a b l e ) a n d a control group (a group not e x p o s e d to the independent v a r i a b l e ) are u s e d , but in w h i c h the investigator has no control over w h o

falls

into

the

test

group

and

who

falls

into

the

control

group.

1. Measure the dependent variable for subjects, some of whom have been exposed to the

In

independent variable (the test group) and some of whom have not (the control group).

" O r g a n i z i n g the Poor," the matter of w h o w a s contacted by the c a m p a i g n w o r k e r s

2. If the dependent variable differs between the groups, ascribe this to the effect of the

w a s d e c i d e d b y the w o r k e r s ' o w n c h o i c e o f w h o m t o contact a n d b y the extent

independent variable.

to w h i c h different people m a d e t h e m s e l v e s a v a i l a b l e for contact by the c a m p a i g n w o r k e r s . T h e natural e x p e r i m e n t d e s i g n i s o f the form:

T h e " T a x - R e f o r m M a i l " e x a m p l e i s o f this sort. I n this d e s i g n , (1) s e n a t o r s ' votes on a t a x - r e f o r m b i l l w e r e noted, a n d (2) the votes of senators w h o h a d

1. Measure the dependent variable for a specific population before it is exposed to the independent variable. 2. Wait until some among the population have been exposed to the independent variable. 3. Measure the dependent variable again. 4. If between measurings the group that was exposed (called the test group) has changed relative to the control group, ascribe this to the effect of the independent variable on the dependent variable.

r e c e i v e d f a v o r a b l e m a i l w e r e c o m p a r e d w i t h the votes o f those w h o h a d not. T h e s a m e k i n d of alternative e x p l a n a t i o n that has to be dealt w i t h in natural e x p e r i m e n t s has t o b e dealt w i t h i n this d e s i g n a l s o . A s i n " O r g a n i z i n g the Poor," i t m a y b e that h e a v i e r p r o - t a x - r e f o r m m a i l w e n t to senators w h o a l r e a d y w e r e m o v i n g into a taxr e f o r m p o s i t i o n e v e n w i t h o u t the m a i l . S u c h s e n a t o r s , about w h o m there m i g h t h a v e b e e n a great d e a l of s p e c u l a t i o n in the p r e s s , c o u l d h a v e attracted m o r e m a i l than did other senators.

T h u s , in " O r g a n i z i n g the Poor," (1) a n u m b e r of poor people w e r e s u r v e y e d as to their interest in p o l i t i c s ; (2) the c a m p a i g n o c c u r r e d ; (3) the s a m e poor people w e r e s u r v e y e d a g a i n ; and (4) those w h o h a d b e e n contacted by the c a m p a i g n w e r e c o m p a r e d w i t h those w h o h a d not. T h i s design e l i m i n a t e s m a n y o f the alternative explanations that c a n c r o p up in w o r k i n g without a control group. F o r i n s t a n c e , it c o u l d not have b e e n the initial m e a s u r e m e n t that c h a n g e d the group that h a d b e e n contacted, c o m p a r e d w i t h the control group, b e c a u s e both groups h a d b e e n m e a s u r e d in the s a m e w a y . N e v e r t h e l e s s , the n a t u r a l e x p e r i m e n t s t i l l a l l o w s alternative e x p l a n a t i o n s . B e c a u s e the r e s e a r c h e r d o e s not h a v e c o n t r o l o v e r w h o i s e x p o s e d t o the i n d e p e n d e n t v a r i a b l e , it m a y be that the e x p o s e d g r o u p h a s a different p r e d i s p o s i tion to c h a n g e than the c o n t r o l g r o u p . In " O r g a n i z i n g the P o o r , " for i n s t a n c e , the c a m p a i g n w o r k e r s are l i k e l y t o h a v e a p p r o a c h e d t h o s e p o o r w h o m they thought

Moreover,

this

design

permits

many

additional

alternative

explanations

b e y o n d those that a p p l y to a natural e x p e r i m e n t . In the t a x - r e f o r m e x a m p l e , it is p r o b a b l e that p e o p l e w e r e m o r e l i k e l y to w r i t e 1 ;tters f a v o r i n g r e f o r m to senators they thought w o u l d agree w i t h t h e m . I n other w o r d s , i t m i g h t b e that s e n a t o r s ' m a i l d i d not c a u s e their v o t e s , but that their l i k e l y vote c a u s e d t h e m to get c e r t a i n k i n d s of m a i l . H e n c e , the r e l a t i o n s h i p b e t w e e n a s e n a t o r ' s vote on the b i l l a n d the m a i l she r e c e i v e d m i g h t not be a c a u s a t i v e r e l a t i o n s h i p at a l l — m e r e l y a c o i n c i d e n c e b e t w e e n p r o - r e f o r m a n d p r o - m a i l a n d a n t i - r e f o r m a n d a n t i - m a i l . T h i s alternative c o u l d not a p p l y to a natural e x p e r i m e n t . In a natural e x p e r i m e n t it w o u l d h a v e b e e n c l e a r f r o m the i n i t i a l m e a s u r e m e n t w h e t h e r or not those w h o fell into the test group h a d i n i t i a l l y b e e n different f r o m those f a l l i n g into the c o n t r o l group. I n fact, w h a t i s c o m p a r e d i n the natural e x p e r i m e n t i s not the m e a s u r e d v a r i a b l e s t h e m s e l v e s , but h o w the two groups c h a n g e b e t w e e n m e a s u r e m e n t s .

they c o u l d m o s t e a s i l y get i n t e r e s t e d i n p o l i t i c s . A l s o , t h o s e a m o n g the p o o r w h o

T o s u m u p , the n a t u r a l e x p e r i m e n t w i t h o u t p r e m e a s u r e m e n t i s a d e s i g n i n

w e r e m o s t r e s i s t a n t t o c h a n g e m i g h t not h a v e let the c a m p a i g n w o r k e r s i n the

w h i c h the test group a n d the c o n t r o l group are c o m p a r e d w i t h r e s p e c t to a d e -

d o o r o r m i g h t h a v e b e e n c h r o n i c a l l y a b s e n t w h e n the c a m p a i g n w o r k e r s t r i e d t o

p e n d e n t v a r i a b l e o n l y after they h a v e b e e n e x p o s e d to the i n d e p e n d e n t v a r i a b l e . It

c o n t a c t t h e m . A c c o r d i n g l y , a p l a u s i b l e alternative e x p l a n a t i o n i n " O r g a n i z i n g the

i n v o l v e s the s a m e sorts o f a l t e r n a t i v e e x p l a n a t i o n s a s the n a t u r a l e x p e r i m e n t d o e s ,

P o o r " i s that the p o o r w h o w e r e c o n t a c t e d b y the c a m p a i g n i n c r e a s e d their

p l u s s o m e others that r e s u l t f r o m the fact that the i n v e s t i g a t o r d o e s not k n o w w h a t

simply because they were the ones who showed more potential to become more interested in politics at that time, regardless of the campaign. T h i s alternative m u s t be either a s s u m e d

the test g r o u p a n d the c o n t r o l g r o u p l o o k e d l i k e b e f o r e the w h o l e t h i n g started.

interest i n p o l i t i c s m o r e than t h o s e w h o w e r e not c o n t a c t e d

True Experiment

a w a y o r c o n t r o l l e d b y u s i n g a d e s i g n i n w h i c h the r e s e a r c h e r c a n d e t e r m i n e w h o f a l l s into the test g r o u p a n d w h o f a l l s into the c o n t r o l g r o u p . A d e s i g n that

In neither of the two v e r s i o n s of the natural e x p e r i m e n t j u s t outlined d i d the i n v e s t i gator h a v e any control over w h o fell into the test group a n d w h o fell into the control

84

Chapter

Causal

6

g r o u p . I f t h e i n v e s t i g a t o r d o e s h a v e s u c h c o n t r o l , s h e c a n p e r f o r m a true experiment.

T A B L E 6-1

and

Design

of Research

85

Selected Research Designs

A true e x p e r i m e n t i n c l u d e s t h e f o l l o w i n g s t e p s : 1. Assign at random s o m e s u b j e c t s to t h e test g r o u p a n d s o m e to the c o n t r o l g r o u p .

Thinking

Type

Graphic Presentation

3

Example from This Chapter

Selected Alternative C a u s a l Interpretations

Agency Study, Reagan's Victory

T h e first m e a s u r e m e n t itself m a y h a v e c a u s e d the c h a n g e observed in the s e c o n d measurement;

2 . M e a s u r e the d e p e n d e n t v a r i a b l e for both g r o u p s . 3 . A d m i n i s t e r the i n d e p e n d e n t v a r i a b l e t o t h e test g r o u p . 4 . M e a s u r e t h e d e p e n d e n t v a r i a b l e a g a i n for both g r o u p s . 5. If t h e test g r o u p h a s c h a n g e d b e t w e e n t h e first a n d s e c o n d m e a s u r e m e n t s in a w a y that

Observation with no control group

Test g r o u p : M * M

is different from t h e c o n t r o l g r o u p , a s c r i b e t h i s difference to t h e p r e s e n c e of t h e independent variable.

or s o m e t h i n g e l s e that h a p p e n e d at the s a m e t i m e as * m a y h a v e c a u s e d the change.

B e c a u s e investigators can control w h o falls into w h i c h group, they c a n set u p theoretically equivalent g r o u p s . ( T h e best, and s i m p l e s t , w a y to do this is to assign subjects r a n d o m l y t o o n e g r o u p o r the other.) T h e a d v a n t a g e o f m a k i n g the g r o u p s e q u i v a l e n t i s that r e s e a r c h e r s c a n t h e r e b y e l i m i n a t e a l m o s t all o f t h e a l t e r n a t i v e c a u s a l i n t e r p r e t a t i o n s t h a t h a d t o b e a s s u m e d a w a y i n t h e v a r i o u s natural e x p e r i m e n t d e s i g n s . I f t h e g r o u p s are e q u i v a l e n t t o start w i t h , f o r e x a m p l e , a d i f f e r e n c e i n h o w

Natural e x p e r i m e n t w i t h o u t premeasurement

Test g r o u p : * M

Tax-Reform

Control:

Mail

M

t h e g r o u p s c h a n g e c a n n o t b e d u e t o t h e f a c t that t h e i n d i v i d u a l s i n t h e t e s t g r o u p w e r e m o r e p r o n e to c h a n g e in certain w a y s than w e r e t h o s e in the control group. T h u s the p r o b l e m that t h e i n v e s t i g a t o r s f a c e d i n " O r g a n i z i n g t h e P o o r " i s e l i m i n a t e d .

or

"Presidential L o b b y i n g " is an e x a m p l e of the true e x p e r i m e n t . H e r e ( 1 ) the

t h o s e w h o m a d e their w a y into t h e test g r o u p m a y h a v e b e e n different from t h o s e in t h e c o n t r o l group even before they e x p e r i e n c e d *.

president c h o s e half of the H o u s e r a n d o m l y to be the test group, l e a v i n g the other half as the control; (2) he t o o k a straw vote to m e a s u r e the d e p e n d e n t variable (vote) for both g r o u p s ; (3) he l o b b i e d the test g r o u p ; (4) the bill w a s v o t e d o n ; and (5) he c o m p a r e d the voting of the t w o groups to see whether his lobbying had m a d e a d i f f e r e n c e . W o r k i n g w i t h t h i s d e s i g n , t h e p r e s i d e n t w o u l d b e pretty c e r t a i n that a disproportionately favorable c h a n g e a m o n g those he l o b b i e d w a s d u e to his efforts. If he had not b e e n able to control w h o w a s lobbied, he w o u l d have been faced with

T h o s e w h o m a d e their w a y into t h e test g r o u p m a y h a v e b e e n m o r e likely t o c h a n g e than t h o s e w h o m a d e their w a y into t h e control group;

Natural e x p e r i ment

Test g r o u p : M * M Control: M M

Organizing t h e Poor

T h o s e w h o m a d e their w a y into t h e test g r o u p m a y h a v e b e e n m o r e likely t o c h a n g e than t h o s e w h o m a d e their w a y into t h e control g r o u p .

Test group: R M * M Control: RM M

Presidential Lobbying

N o n e of the alternatives d i s c u s s e d in this c h a p t e r a p p l i e s . This d e s i g n permits o n l y a very f e w alternative e x p l a n a t i o n s . Consult Cook and Campbell (1979), cited at t h e e n d o f this chapter.

plausible alternative causal interpretations. Table 6-1 s u m m a r i z e s the various research d e s i g n s d i s c u s s e d in this chapter and s o m e of the alternative explanations applicable in each case. Each d e s i g n is p r e s e n t e d t h e r e i n a s y m b o l i c s h o r t h a n d that i s e x p l a i n e d i n t h e f o o t n o t e t o t h e t a b l e . I n t h e last c o l u m n o f t h e t a b l e , o n e o r t w o o f t h e a l t e r n a t i v e i n t e r p r e t a t i o n s left o p e n b y e a c h d e s i g n are h i g h l i g h t e d .

D E S I G N S FOR POLITICAL RESEARCH

True e x p e r i m e n t

T h e natural e x p e r i m e n t w i t h o u t p r e m e a s u r e m e n t i s the s i n g l e m o s t c o m m o n l y u s e d d e s i g n in political research. A f e w e x a m p l e s will indicate the broad u s e of the d e s i g n : ( 1 ) a n y v o t i n g s t u d y that s h o w s that p e r s o n s o f a c e r t a i n t y p e ( w o r k i n g c l a s s , e d u c a t e d , male, or w h a t have y o u — t h e test g r o u p ) vote differently from t h o s e w h o are n o t o f t h i s t y p e ( t h e c o n t r o l g r o u p ) ; ( 2 ) a n y o f t h e l a r g e n u m b e r o f s t u d i e s estimating h o w m u c h better i n c u m b e n t presidents do in s e e k i n g reelection w h e n the

"Notation a d a p t e d from D o n a l d T. Campbell and Julian C. Stanley, Experimental and Quasiexperimental Designs for Research (Chicago: Rand McNally, 1963). In presenting the designs graphically, an asterisk (*) indicates that a group has been exposed to a stimulus or is distinguished in s o m e other way so as to constitute a "test group"; M indicates that the d e p e n d e n t variable has been measured for t h e group; and R (used to describe the "true experiment") indicates that individuals have b e e n assigned randomly to the groups.

86

Chapter

6

Causal Thinking and Design of Research

e c o n o m y h a s i m p r o v e d i n t h e p r e c e d i n g y e a r (test g r o u p ) c o m p a r e d w i t h t h o s e for w h o m the economy has

anticipated: the introduction of a poverty p r o g r a m in a town, high school graduates'

output" studies, such as one by R o g o w s k i and Kayser (2002), w h i c h s h o w e d that

entrance into college, regularly scheduled political events such as elections, and so

majoritarian

on. T h e s e lend themselves readily to a natural experimental design.

systems

(the

test

(control group);

group;

also

known

(3)

fit i n t o a n a t u r a l e x p e r i m e n t d e s i g n . O n t h e o t h e r h a n d , s o m e e v e n t s a r e m o r e e a s i l y

most "policy

electoral

not been improving

87

as

single-member

district, plurality s y s t e m s , the k i n d of s y s t e m u s e d in U . S . H o u s e elections and

A true experiment requires even greater control over the subjects of the study

British a n d C a n a d i a n elections) will tend to p r o d u c e e c o n o m i c policies favorable to

t h a n d o e s a natural e x p e r i m e n t . T h e latter requires only that the investigator be able to

c o n s u m e r s , while proportional representation systems (the control group, electoral

anticipate events, but the true experiment requires that the researcher be able to

systems like those of Germany, Sweden, and the Netherlands) should strengthen the

m a n i p u l a t e t h o s e e v e n t s — s h e m u s t d e c i d e w h o i s t o fall i n t o t h e c o n t r o l g r o u p a n d

interests of producers;

Uhlaner and

w h o i s t o fall i n t o t h e e x p e r i m e n t a l g r o u p . T o d o t h i s i n a f i e l d s i t u a t i o n r e q u i r e s

S c h l o z m a n ( 1 9 8 6 ) , w h o t e s t e d w h e t h e r w o m e n c a n d i d a t e s for office w e r e less w e l l

p o w e r over p e o p l e in their n o r m a l lives. T h u s it is no accident that the e x a m p l e I u s e d

and

(4)

studies

of discrimination,

such as

financed than m e n b e c a u s e of experiential attributes such as seniority, or b e c a u s e of

of a true experiment, "Presidential Lobbying," w a s carried out by the president. It is

direct discrimination.

rare in real political situations that a researcher can exercise this kind of control over

In

short,

any

research

is

an

example

of the

natural

experiment

without

events. However, the experiment is so m u c h m o r e powerful than other d e s i g n s —

p r e m e a s u r e m e n t if it (1) takes t w o or m o r e types of subjects and c o m p a r e s their

rejects

values on a d e p e n d e n t variable; and (2) infers that the difference on the d e p e n d e n t

possible, and probably should use it a good deal m o r e than is now d o n e .

variable is the result of their difference on whatever it is that distinguishes t h e m as "types."

5

This really describes the bulk of political research.

As we s a w in earlier s e c t i o n s , this d e s i g n is far f r o m satisfactory. It p e r m i t s relatively m a n y

alternative causal interpretations,

alternative interpretations

so decisively—that we

should use it whenever 6

A c h e n (1986, p. 6), citing Gilbert, Light, and Mosteller (1975), points out that a p p a r e n t l y only r a n d o m i z a t i o n p r o v i d e s a c l e a r - e n o u g h c a u s a l i n t e r p r e t a t i o n t o s e t t l e issues of social-scientific research conclusively:

w h i c h are difficult t o h a n d l e .

Nevertheless, this r e m a i n s the m o s t widely u s e d d e s i g n in political science. O t h e r

W i t h o u t it, e v e n t h e c l e v e r e s t statistical a n a l y s i s m e e t s s t r o n g r e s i s t a n c e f r o m o t h e r

designs that have the advantage of control groups are to be preferred because they

s c h o l a i s , w h o s e p r o f e s s i o n a l s k e p t i c i s m i s q u i t e n a t u r a l . W h e n t h e forces t h a t deter-

r e q u i r e less difficult a s s u m p t i o n s , b u t t h e s e d e s i g n s c a n b e u s e d o n l y w h e n the

m i n e t h e e x p e r i m e n t a l a n d c o n t r o l g r o u p s a r e u n k n o w n , t h e i m a g i n a t i o n h a s full p l a y t o

r e s e a r c h e r h a s m o r e control over the test variables than m o s t political scientists can

c r e a t e alternative e x p l a n a t i o n s for t h e d a t a . I n v e n t i n g h y p o t h e s e s of this sort is e n j o y -

usually achieve.

a b l e , u n s t r e n u o u s l a b o r t h a t i s r a r e l y resisted.

In o r d e r to u s e a n a t u r a l e x p e r i m e n t d e s i g n , for i n s t a n c e , r e s e a r c h e r s m u s t be able t o a n t i c i p a t e the o c c u r r e n c e o f the test factor. T h e y a l s o m u s t g o t o the e x p e n s e

It m a y be that in a s m a l l o r g a n i z a t i o n , s u c h as a local political c a u c u s or a

in time and m o n e y of m a k i n g t w o m e a s u r e m e n t s of the subjects. E v e n then, the

portion of a c a m p u s , the investigator can carry out a true experimental design. For

a t t e m p t m a y misfire; for e x a m p l e , it m a y be that t h e test factor (especially if it is o n e

instance, students might c o n d u c t a political c a m p a i g n a m o n g a r a n d o m l y selected

that affects only a small part of the p o p u l a t i o n ) will a p p l y to o n l y a few of the p e o p l e

portion

i n c l u d e d i n t h e study. T h a t is, t h e i n v e s t i g a t o r m a y b e left w i t h a c o n t r o l g r o u p b u t n o

r a n d o m l y selected control.

test g r o u p .

of the

campus

community

and c o m p a r e that portion

over time with

a

It is s o m e t i m e s possible to carry out true experiments on a larger scale than

M a n y i m p o r t a n t v a r i a b l e s i n t h e s o c i a l s c i e n c e s a r e n o t a t all s u s c e p t i b l e t o

this, but only if the experimenter can control significant policies and m a n i p u l a t e

s t u d y b y n a t u r a l e x p e r i m e n t s . S o m e lie i n t h e past: p e o p l e ' s e x p e r i e n c e s d u r i n g the

t h e m for the purposes of investigation. A g o o d e x a m p l e of this sort of e x p e r i m e n t is

Depression, the colonial histories of nations, the educational backgrounds and past

Gerber

professions of m e m b e r s of C o n g r e s s , and so on. Others, such as assassinations, riots,

C o n n e c t i c u t , w e r e e a c h r a n d o m l y assigned o n e of three different a p p r o a c h e s in a

and

Green

(2000),

in

a n d c h a n g e s in the b u s i n e s s c y c l e , are u n p r e d i c t a b l e . S u c h variables are difficult to

nonpartisan

get-out-the-vote

which

program.

30,000

It

registered

turned

out

voters

that

in

personal

New

Haven,

visits

were

effective in getting p e o p l e to vote, direct m a i l a p p e a l s h e l p e d slightly, a n d t e l e p h o n e calls m a d e no difference at all. 5

It is apparent here and in the examples directly preceding this that I am taking some liberty with the notion of "control group." Where rates of participation in the middle class and working class are compared, for instance, it is not at all clear which class is the "test" group and which is the "control." The distinction is even muddier when one compares several groups simultaneously, such as voters from several age groups. But the logic of what is done here is the same as in the strict test/control situation, where the dependent variable is compared among groups of subjects distinguished by their values on the independent variable. It is convenient and revealing to treat this sort of analysis in terms of the analogy to experiments.

A n o t h e r interesting e x a m p l e of a natural situation that p r o d u c e d something a p p r o x i m a t i n g a true e x p e r i m e n t is t h e s t u d y by H o w e l l et al. ( 2 0 0 2 ) of t h e effects of a private education on poor children. P r o g r a m s w e r e initiated in Dayton, Ohio:

6

Kinder and Palfrey (1993, especially the Introduction) argue persuasively that we in political science have overrated the problems and barriers to true experimentation in our field.

88

Chapter 6

Washington, D.C.; and New York City that offered children in public schools partial scholarship vouchers to attend private schools. Since there were more applicants than could be funded, children were chosen by lottery to receive the vouchers. This selection process approximated a true experiment; the scholarships were assigned randomly, and there was a control group of children who applied for scholarships but did not receive them. Thus, the two groups should have been equivalent in all respects other than the experimental treatment. The result of the study was that African American students who received the scholarships did significantly better on standardized tests two years later than African American students who did not receive the scholarships. No other ethnic group appeared to benefit in the same way. This study comes very close to a true experiment, and therefore offers very convincing results on an important policy issue. One remaining alternative explanation is that something akin to the problem we saw in 'Agency Study" was operating. The children who received scholarships knew they had won something special. Also, they knew that their parents were paying extra money beyond the partial scholarship to further their education. This may have left them more highly motivated than the students who had not won in the lottery. (In an ideal experiment, people do not know whether they are in the experimental group or in the control; the New Haven study by Gerber and Green is an example.) This worry might be lessened, however, by the fact that it was only African Americans who appeared to benefit from the vouchers; if the benefit were an artifact of children's knowing they had won in the lottery, one would think that would affect all ethnic groups in the same way. When it is possible to construct true (or nearly true) experiments in real field situations like this, the results can be compelling. More often, true experimentation is useful for studying general aspects of small-group interactions or individual thought processes relevant to politics. For example, a group of subjects may be placed together and told to reach a decision on some question. The investigator then manipulates the way individuals in the group may communicate with each other, to see how this influences the result. In such studies, of course, true experiments are the most appropriate design, inasmuch as the investigator generally can control all the relevant variables. A good example of this sort of experiment is that of Iyengar and Kinder (1987), in which the investigators tested experimentally the effects of variations in the format of TV news reporting on subjects' political perceptions. Special Design Problem for Policy Analysis: Regression to the Mean I have introduced you above to some simple sources of alternative causal interpretations and some simple designs to control for them. A slightly more complex problem, while it has many applications in general explanation, is especially important in evaluating the impact of policy initiatives. This is a problem most public officials and journalists do not appear to understand at all, but it is well within your range of understanding at this point.

Causal Thinking and Design of Research 89 The problem is called regression to the mean. If we assume that essentially everything we observe has some element of randomness to it—that is, in addition to its true core value it varies somewhat from time to time—regression to the mean will always be present. Consider a student in a course, for example. When she is tested, her measured level of knowledge is generally about right, but if she has had a bad day (isn't feeling good, has bad luck in the instructor's choice of questions, and so on), she will score somewhat below her usual level; on a good day, she will score somewhat higher. Note that this sort of random variation in a measure is closely related to what we referred to as unreliability in Chapter 4. For our purposes here, the important thing about this is that if we observe cases at two points in time, we can expect high initial values to drop somewhat from one time to the next, and low initial values to rise somewhat. Not all of them will do this; some of the high scores no doubt are genuinely high and may even rise by the next time they are observed. But a disproportionate number of the high cases are probably cases for whom things had lined up unusually well the first time; that is, the random part of their measure is likely to have been positive. We should expect that it is unlikely they would be so lucky twice in a row, so on the average they are likely to go down the next time they are observed. Similarly, the lowest scores probably include a disproportionate number of cases that were unusually low because of a negative random factor; on the average, we should see them rise. We should therefore be a little suspicious of a statement.like: "Team learning strategies are good in that the weakest students improve, but they are bad in that the strongest students appear to be dragged down by working with the weaker ones." This could be true. But an alternative explanation could be that this is simply regression to the mean. It might be that the stronger students' scores declined because when they were measured at the beginning of the experiment with team learning, a number of them had done unusually well and then just drifted back to their normal level of performance over time. Similarly, some of the poorer students might have drifted up to their normal levels. In other words, the scores of the weaker students might have improved, and the scores of the stronger students might have declined, even if the students had never been involved in team learning. The two alternative causal interpretations can only be sorted out by using an appropriate research design. (One is given on p. 90) One variant of this problem shows up time and again in assessing the impact of new governmental policies. How often have you seen a statement of the sort: "After foot patrols were introduced in the entertainment district, our city's murder rate declined by a full 18 percent"? What is often missed in such statements is that this effect might also be due to regression to the mean. When do cities typically institute new policies? When the problem they are concerned about has flared up! But cities' problems, like students taking tests, probably have some element of randomness to their measures. If the murder rate has shot up in one year, it is more likely to go down the next year than to go still higher, whether or not the city institutes foot patrols in the entertainment district. In other words, governments are likely to pick a time when a problem is at a high point (which may or may not include some random element)

90

Causal Thinking and Design of Research

Chapter 6

91

to initiate a p o l i c y to s o l v e the p r o b l e m . T h u s , in the c a s e of the murder rates, on the b a s i s of the information in the statement, we cannot tell whether or not foot patrols r e a l l y c u r b e d the m u r d e r s , b e c a u s e r e g r e s s i o n to the m e a n provides a p l a u s i b l e

(A)

alternative explanation. D o e s this m e a n that w e cannot a s s e s s the i m p a c t o f c h a n g e s i n p o l i c i e s ? O f c o u r s e not. B u t it does m e a n that we must find a r e s e a r c h d e s i g n that a l l o w s us to d i s t i n g u i s h b e t w e e n the t w o alternative explanations. T h e d e s i g n in the statement about foot patrols and m u r d e r rates is our old friend: M * M

O u r p r o b l e m is that we suspect that the first M m a y have been u n u s u a l l y h i g h , m a k i n g it l i k e l y that the s e c o n d m e a s u r e m e n t w o u l d s h o w a d e c r e a s e whether or not AlZ,

the intervention between the two m e a s u r e s has had any impact. A d e s i g n that a l l o w s us to sort this out is an

interrupted time series, that i s , a

series of m e a s u r e m e n t s over time that is interrupted by the p o l i c y intervention: 1 MMMMM

2

3

4

5

6

7

8

9

10

11

12

13

14

8

9

10

11

12

13

14

Year

*MMMMM

If the measurement that c a m e just before the intervention w a s unusually high, the m e a s urements preceding it should tend to be lower than it i s . T h e test for whether the (B)

average of the several measurements preceding the intervention differs f r o m the average of the measurements intervention has had an effect in this design is whether the

following the intervention.

7

T h e interrupted time series i s g r a p h i c a l l y illustrated i n F i g u r e 6 - 1 , w i t h the intervention indicated by an asterisk and a d a s h e d v e r t i c a l line. G r a p h A illustrates an interrupted time series in w h i c h the intervention appears to have c a u s e d a c h a n g e . G r a p h B illustrates one in w h i c h it did not. Note that in graph B, if we h a d l o o k e d only at the adjacent "intervention and after" m e a s u r e s , r e g r e s s i o n to the m e a n w o u l d have l e d us to think the intervention had had an i m p a c t .

USE OF VARIED DESIGNS A N D MEASURES

I s u p p o s e that the easiest c o n c l u s i o n to draw f r o m our d i s c u s s i o n thus far is that any

1

k i n d of r e s e a r c h in political s c i e n c e is difficult and that the results of p o l i t i c a l

2

3

4

5

6

7 Year

r e s e a r c h are, in the last a n a l y s i s , unreliable. B u t this w o u l d be far too sour. I did not d i s c u s s these selected r e s e a r c h d e s i g n s to c o n v i n c e y o u not to do p o l i t i c a l r e s e a r c h

Figure 6-1

Examples of Interrupted Time Series

but to s h o w y o u s o m e of the p r o b l e m s y o u must d e a l w i t h .

E v e n if y o u are f o r c e d to rely solely on one of the w e a k e r d e s i g n s (operating 7

A n even better design, where possible, would add a control to rule out the possibility that it was something else that happened at the same time as the intervention that caused the change: MMMMM * MMMMM MMMMM MMMMM

without a control group, or u s i n g a control group w i t h no p r e m e a s u r e m e n t ) , y o u are better off if y o u set out y o u r d e s i g n f o r m a l l y and m e a s u r e those variables that c a n be m e a s u r e d , a c k n o w l e d g e alternative c a u s a l interpretations, and try to a s s e s s j u s t h o w

92

Chapter 6

likely it is that each alternative is true. The choice is between doing this or giving up and relying on your intuition and impressions. But there is a more hopeful side to this chapter. The various weaknesses of different designs should suggest a solution: Wherever possible, try to work simultane-

ously with a variety of designs, which can at least in part cancel out each other's weaknesses. For example, in working solely with a randomized experiment in an isolated situation, we may wonder whether the result we have obtained might be a result of something about the way the experiment was set up, rather than a "true" result. Relying totally on a natural experiment, we might wonder whether our results simply reflect a test group comprised of unusual people, rather than a "true" result. But if we could use both of these designs, employing either the same or closely related measures, each result would bolster the other. We could be more confident of the experimental result if we had gotten a similar result in a field situation. And we could be more confident of the result of the natural experiment if we had gotten a similar result in an artificial true experiment. If we can mix research strategies in this way, the end result is more than the sum of its parts. The strengths of the various designs complement each other. The likelihood that the alternative explanations associated with each design are true decreases because of the confirmation from the other design. It will seem less likely to us that the result of the natural experiment is due to unusual people entering the test group, for example, if we obtain similar results in a true experiment, whose test group we controlled. Note that the effect would not be the same if a single research design were repeated twice—if, say, a natural experiment were repeated in a different locality. Repeating the same design twice would increase our confidence somewhat, by showing us that the first result was not an accident or a result of the particular locality chosen. But the alternative causal interpretations associated with natural experimental design would in no way be suspended. Consider the "Organizing the Poor" example once again. If the campaign in one city tended to reach mainly those who were already becoming more interested in politics, thus creating the illusion that the campaign had gotten them interested in politics, there is no reason to think that a similar campaign would not do the same thing in a second city. Getting the same result in two different cities would not make it any less likely that the alternative causal interpretation was true. Example of Varied Designs and Measures In his Making Democracy Work, Robert Putnam (1993) uses the varied-design strategy rather well. His study sought to explain variation in the quality of governmental institutions; his theory was that traditions of civic cooperation, based on what he calls social capital (networks of social cooperation), are what lead to effective government. First, in what is really a problem of valid measurement (see Chapter 4) rather than one of research design, he approached the problem of measuring "government performance" by using multiple measures. To the extent that he got the same results

Causal Thinking and Design of Research

93

across a variety of methods of measurement, all with their own varied sources of invalidity, he could be more confident that he had measured governmental effectiveness accurately. To this end, he first gathered a number of statistics about governmental performance in the 20 regions that he was studying, such as the promptness with which the regional governments developed their budget documents. He then checked and supplemented the official statistics by having research assistants call government agencies in each region with fictitious requests for help on such things as completing a claim for sickness benefits or applying for admission to vocational school. (Governments' response quality varied from responses in under a week after a single letter, to responses that required numerous letters, numerous phone calls, and a personal visit over several weeks.) Finally, these objective measures of performance were supplemented by several opinion surveys in which people were asked how well they thought the regional government performed. Once he had decided how to measure governmental performance, Putnam was faced with the question of research design. His basic design was the natural experiment with no premeasurement. The 20 regions were compared as to their level of civic engagement and the quality of their governments' performance, and those with high engagement proved to be those with a high quality of governmental performance. To test the relationship more richly, he also added tests of linking relationships using the same design. For instance, he showed that the higher civic engagement was in communities, the less political leaders feared compromise. This makes sense as a component of the relationship between civic engagement and governmental effectiveness, since willingness to compromise should in turn lead to effective government. The fact that such linking relationships proved to be true under testing makes us more confident that the overall relationship was not a design fluke. This design still suffered, though, from the possibility that the relationship was a matter of other contemporary things associated with civic engagement, such that those regions high in civic engagement were for some other reasons primed for governmental effectiveness. Putnam countered this, in part, by adding analyses over time of the form "observation with no control group." In these, he traced the development of civic involvement in regions over the past century and showed that high civic involvement in the latter part of the nineteenth century led to civic involvement and governmental effectiveness a century later. By combining the complementary strengths of varied measures and varied designs, Putnam was thus able to generate conclusions of which the reader is more confident than if they were based on a single measure or on a single design.

CONCLUSION To pick up once again the refrain of this book, creativity and originality lie at the heart of elegant research. Anybody, or almost anybody, can take a research problem, carry out a fairly obvious test on it using one or two obvious measures, and either

94

Chapter

6

Causal Thinking and Design of Research

ignore or a s s u m e a w a y the ever-present alternative causal interpretations. A creative

T A B L E 6-2

95

Life Expectancy U n d e r Dictatorships and D e m o c r a c y

r e s e a r c h e r w i l l n o t b e s a t i s f i e d w i t h t h i s b u t w i l l try v e r y h a r d t o a c c o u n t f o r all plausible alternative interpretations. S u c h a student will muster logical arguments, w i l l cite e v i d e n c e f r o m related studies, and m a y e v e n vary the d e s i g n s and m e a s u r e s

Per C a p i t a Income, $

i n h i s o w n study. A l l o f t h e s e t e c h n i q u e s will h e l p the investigator limit the n u m b e r

Life Expectancy, Democracies

Life Expectancy, Dictatorships

Difference

0-1,000

47.2

(215)

0.8

1,001-2,000

(11) (39)

46.4

56.3

52.2

(144)

4.1

T h e s u g g e s t i o n m a d e i n t h e p r e c e d i n g s e c t i o n — t o vary m e a s u r e s a n d d e s i g n s — i s ,

2,001-3,000

63.6

(34)

59.2

(59)

4.4

I think, a u s e f u l o n e , but it s h o u l d n o t be t h o u g h t of as t h e o n l y a n s w e r . V a r y i n g d e s i g n s

3,001-4,000

67.3

(30)

64.2

(44)

3.1

i s g e n e r a l l y u s e f u l , but there are o t h e r w a y s t o e l i m i n a t e alternative interpretations, s u c h

4,001-5,000

70.2

(19)

65.0

(25)

5.2

(18)

2.7

of alternative interpretations of his findings.

a s b y l o g i c a l a r g u m e n t o r b y indirect e v i d e n c e o f v a r i o u s sorts. N o interpretation o f a r e s e a r c h result is cut a n d dried; interpreting a result a n d h a n d l i n g alternative interpreta-

5,001-6,000

71.3

(23)

68.6

6,001-

73.2

(239)

67.6

(18)

3.6

69.3

(395)

53.3

(521)

16.0

Total

t i o n s o f t h e result are difficult a n d c h a l l e n g i n g c o n s t i t u e n t s o f research.

H O L D I N G A VARIABLE C O N S T A N T

e x p e c t a n c y i n b o t h d e m o c r a c i e s a n d d i c t a t o r s h i p s i s l o w , but t h e d e m o c r a c i e s h a v e a n a d v a n t a g e o f e i g h t - t e n t h s o f a year. A t all l e v e l s o f p r o s p e r i t y , d e m o c r a c i e s h a v e s o m e

I h a v e talked about h o l d i n g a variable constant m o r e than o n c e in this chapter. If we w a n t to k n o w w h e t h e r a relationship b e t w e e n t w o variables c a n be a c c o u n t e d for by a third v a r i a b l e that i s r e l a t e d t o b o t h o f t h e m , w e c a n h o l d t h e third v a r i a b l e c o n s t a n t

a d v a n t a g e , r e a c h i n g a s h i g h a s a n i n c r e a s e o f 5 . 2 y e a r s o f life e x p e c t a n c y for c o u n t r i e s with per capita i n c o m e of $ 4 , 0 0 1 - $ 5 , 0 0 0 , and an increase of 5.6 years for countries w i t h p e r c a p i t a i n c o m e g r e a t e r than $ 6 , 0 0 0 .

t o s e e w h e t h e r t h e r e l a t i o n s h i p b e t w e e n t h e first t w o c o n t i n u e s t o e x i s t w h e n t h e third variable

is

not

controlling for

the

free

to

vary.

Holding

a

variable

constant

is

often

also

called

variable.

T h e s i m p l e s t w a y to do this is to divide the subjects into separate g r o u p s , e a c h having a distinct value on the variable to be held constant, and then observe whether within e a c h of these g r o u p s there is a relationship b e t w e e n the first t w o variables. If

T h e number of countries falling into e a c h type appears in parentheses, and f r o m t h e s e w e c a n s e e w h y the difference w a s s o great b e f o r e w e c o n t r o l l e d for the countries' per capita i n c o m e s .

A g o o d e x a m p l e of h o l d i n g a variable constant is provided by the Przeworski e t al. ( 2 0 0 0 ) s t u d y o f w h e t h e r p e o p l e l i v e b e t t e r i n d e m o c r a c i e s t h a n i n d i c t a t o r s h i p s .

dictatorships w i t h per capita i n c o m e o v e r $ 6 , 0 0 0 , but 2 3 9 d e m o c r a c i e s ( m o s t l y f r o m North A m e r i c a and western Europe). The conclusion

c o u l d e x p e c t to live 16 years longer, on the average, than those in dictatorships—a big difference.

w e d r a w f r o m h o l d i n g p e r c a p i t a i n c o m e c o n s t a n t i s that

although m o s t of the difference in life e x p e c t a n c i e s w a s due to the difference in h o w w e l l o f f d e m o c r a c i e s a n d d i c t a t o r s h i p s are, a s u b s t a n t i a l d i f f e r e n c e still e x i s t s b e t w e e n d e m o c r a c i e s and dictatorships e v e n w h e n per capita i n c o m e is taken into account.

T h e y f o u n d that t h e l i f e e x p e c t a n c y o f p e o p l e l i v i n g i n d e m o c r a c i e s w a s 6 9 . 3 y e a r s w h i l e that o f p e o p l e living i n dictatorships w a s j u s t 5 3 . 3 y e a r s . T h o s e i n d e m o c r a c i e s

T h e poorest group has 2 1 5 dictatorships, compared

w i t h o n l y e l e v e n d e m o c r a c i e s ( c o u n t r i e s s u c h a s I n d i a ) . A t t h e o t h e r e n d are o n l y 1 8

t h e r e i s , i t c a n n o t b e d u e t o v a r i a t i o n i n t h e third v a r i a b l e , f o r w i t h i n e a c h o f t h e s e g r o u p s t h e third v a r i a b l e l i t e r a l l y i s c o n s t a n t .

9

T h e technique demonstrated here of h o l d i n g a variable constant—literally s e p a r a t i n g t h e s u b j e c t s i n t o n e w little g r o u p s a n d d o i n g t h e s a m e a n a l y s i s w i t h i n e a c h o f t h e s e g r o u p s — i s t h e s i m p l e s t t e c h n i q u e o n e c a n u s e t o e l i m i n a t e a third v a r i a b l e . S t a t i s t i c a l t e c h n i q u e s s u c h a s m u l t i p l e r e g r e s s i o n artificially h o l d v a r i a b l e s

H o w e v e r , a n o b v i o u s a l t e r n a t i v e e x p l a n a t i o n o f f e r e d itself. D i c t a t o r s h i p s w e r e m u c h m o r e likely than d e m o c r a c i e s to be poor, and p o o r countries in general have w o r s e h e a l t h a n d l o w e r l i f e e x p e c t a n c i e s t h a n c o u n t r i e s that are b e t t e r off. D o t h e l o n g e r life e x p e c t a n c i e s in d e m o c r a c i e s indicate anything about w h a t d e m o c r a c y d o e s f o r p e o p l e , o r i s i t j u s t t h a t d e m o c r a c i e s a r e r i c h e r c o u n t r i e s , a n d that i s w h y people in democracies live longer?

constant, and their effect is a p p r o x i m a t e l y the s a m e as the direct t e c h n i q u e I u s e d here.

1 0

T h e a d v a n t a g e o f s u c h s t a t i s t i c a l t e c h n i q u e s i s that t h e y c a n c o n v e n i e n t l y

h o l d several variables c o n s t a n t s i m u l t a n e o u s l y . D o i n g this w i t h the direct t e c h n i q u e w o u l d not be easy. First, it w o u l d result in a h o r r e n d o u s n u m b e r of tiny tables. M o r e s e r i o u s l y , m a n y o f t h e s e w o u l d b e b a s e d o n rather s m a l l n u m b e r s o f c a s e s , a n d their m e a n i n g w o u l d thus be uncertain.

As indicated in Table 6-2, if we divide countries into groups with approximately t h e s a m e p e r c a p i t a i n c o m e s , there still r e m a i n s a n a d v a n t a g e for d e m o c r a c i e s , a l t h o u g h 8

i t i s n o t h i n g l i k e 1 6 y e a r s . F o r c o u n t r i e s w i t h p e r c a p i t a i n c o m e s l e s s than $ 1 , 0 0 1 , l i f e

"Based on Przeworski et al. (2000), p. 229.

9

T h e reason the number of countries is so high is that the authors counted each country at several different points in its history; some countries appear in the data set four or more times. S e e Chapter 9 for an elementary discussion of multiple regression. l0

96

Chapter

6

FURTHER DISCUSSION

I have presented here only a f e w examples of the m o r e c o m m o n l y u s e d research designs, and my analysis of t h e m has included only a sample of the possible alternative explanations for each design. A more complete treatment of research design, excellent and readable, is provided in T h o m a s D. C o o k and D o n a l d T. C a m p b e l l ' s Quasi-

experimentation: Design and Analysis Issues for Field Setting ( 1 9 7 9 ) ; or more recent and even m o r e complete (but not quite as readable), W i l l i a m R. S h a d i s h , T h o m a s D. Cook,

and

Donald

T.

Campbell

Experimental

and

Quasi-Experimental

Designs for

Generalized Causal Inference ( 2 0 0 2 ) . I have relied h e a v i l y on their approach in this chapter.

C h r i s t o p h e r H.

Achen's

The

Statistical Analysis

of Quasi-experiments

(1986)

explores these issues with great w i s d o m ; Chapters 1 and 2 are easily a c c e s s i b l e to readers with limited technical preparation. A good treatment of the concept of " c a u s e " in relation to research design is found in B l a l o c k ( 1 9 6 4 , Chapters 1 and 2 ) . Tingsten (1937)

provides

an

example

of

unusually

careful

and

creative

interpretations,

developed in the face of quite limited research designs. S e v e r a l interesting e x a m p l e s of i m p r o v i s e d research designs in field research on animal behavior c a n be found in Tinbergen ( 1 9 6 8 ) . T h e literature on the use of true experiments in p o l i t i c a l s c i e n c e is burgeoning.

So far in this book we h a v e l o o k e d at h o w y o u c a n develop a r e s e a r c h question,

G o o d e x a m p l e s are K i n d e r and P a l f r e y ( 1 9 9 3 ) ; a s p e c i a l i s s u e of the j o u r n a l Political

m e a s u r e the variables i n v o l v e d , and look for relationships a m o n g them to provide

Analysis ( v o l u m e 10, n o . 4 , A u t u m n 2 0 0 2 ) ; a n d a s p e c i a l i s s u e , edited b y D o n a l d

a n s w e r s to the r e s e a r c h question. A further important factor u n d e r l i e s this w h o l e

G r e e n a n d A l a n G e r b e r , of American Behavioral Scientist ( v o l u m e 4 8 , no.

p r o c e s s , however. To do all these things, y o u must select a set of observations to look

1, J a n u a r y

2004).

at, and y o u r selection of c a s e s c a n sharply affect or e v e n distort w h a t y o u w i l l find. O n e question that y o u might c o n s i d e r i n c o n n e c t i o n w i t h this chapter i s : W h y

Further, it is often the c a s e that the " s e l e c t i o n " o c c u r s by subtle p r o c e s s e s other than

s h o u l d theory-oriented, e m p i r i c a l political r e s e a r c h b e b a s e d a l m o s t e x c l u s i v e l y o n

y o u r o w n c h o i c e . In this chapter I want to alert y o u to the i m p o r t a n c e of c a s e

c a u s a l relationships rather than on relationships in g e n e r a l ?

s e l e c t i o n (whether it is done by y o u or by nature), and to s h o w y o u s o m e b a s i c p r i n c i p l e s that w i l l help y o u to select c a s e s in w a y s that w i l l a l l o w y o u a c l e a n e x a m i n a t i o n o f y o u r r e s e a r c h question. A central theme in this b o o k , w h i c h y o u s a w e s p e c i a l l y in the two chapters on measurement and w h i c h y o u w i l l see again in later chapters, is that we need to construct our research operations so that the relationships we observe a m o n g the variables we have m e a s u r e d mirror faithfully the theoretical relationships we are interested in. ( T h i s w a s the point of F i g u r e 4 - 1 , for instance.) It w i l l not surprise y o u , then, that case selection is j u d g e d by h o w w e l l it produces observed relationships that faithfully mirror the theoretical relationships we are trying to test. C o n s i d e r the f o l l o w i n g e x a m p l e s :

•

I d e s c r i b e d earlier the strange c a s e o f the Literary Digest, w h i c h in 1936

predicted that F r a n k l i n R o o s e v e l t w o u l d l o s e the e l e c t i o n by a l a n d s l i d e on the basis of questionnaires sent to s u b s c r i b e r s to their m a g a z i n e , car o w n e r s , and those h a v i n g telephone s e r v i c e (not a representative group of A m e r i c a n voters in those d a y s ) (see pp. 5 0 - 5 1 ) . T h e c a s e s they h a d c h o s e n to l o o k at did not faithfully mirror the A m e r i c a n electorate.

97

98

Chapter

7

• College students taking psychology tests are routinely used for experiments to measure psychological processes. For instance, an experiment using college students might study the effect of drinking coffee on one's ability to memorize long strings of numbers. Researchers justify doing this by arguing that the processes they are studying are universal, so even though their test subjects are not at all representative of the human population, that does not matter. The relationship they are looking for would be expected to be the same in any kind of group of people, so their students are as good as any other for the test. • Most graduate departments in political science admit students to their programs based in part on each student's scores on the Graduate Record Examination (GRE). From time to time, a department will consider dropping the exam because when one looks at the performance of graduate students in the department, how they did on the GRE has little to do with how well they have done in the graduate program. Therefore, it is argued, the test is a nearly irrelevant predictor of success in the graduate program and should be dropped as a tool for assessment. However, this argument relies on data from those admitted to the program to generalize to the population of all students who might apply to the program. Clearly, the students admitted into the program are an atypical sample from the population; they all did well enough on the GRE to get into the program. Would students who did badly on the GRE do well in the program? We cannot know, because no such students are in the group we are able to observe. In this case, selection of a potentially biased sample did not result from a decision the researcher made, but inheres in the situation. Nature did it. • John Zaller (1998), studying whether incumbent members of Congress had extra advantages in garnering votes, had to work with another selection problem dealt by nature. His theory predicted that incumbent members' safety from electoral defeat, as measured by their share of the vote, would increase with every term they served, but at a decreasing rate. So he examined the vote margins of incumbents running for reelection, at varying levels of seniority. However, a number of seats were uncontested, because the incumbent was so safe that no one wanted to take her on. So the available set of congressional races consisted only of those districts in which someone felt it made sense to challenge the incumbent. Zaller stated his problem:

Selection of Observations for Study

99

their safety. (His approximated solution was to estimate from other factors the vote that the unchallenged incumbents would have gotten if they had been challenged, and then to insert those simulated estimates as the observations for the unchallenged incumbents—not a perfect solution, but the one that made the best approximation.) • The general understanding of the new democracies of eastern Europe has been that unless they are culturally homogeneous as Poland is, they are "ethnic powderkegs" that had been held in check only by Soviet repression, and are now prone to explode in ethnic violence. As Mihaela Mihaelescu (2004) has pointed out, however, this general impression has come about because almost all scholars who looked at ethnic conflict in new eastern European democracies were drawn to the dramatic outbreaks of violence in the former Yugoslavia. In fact, of fourteen new states in Eastern Europe with significant ethnic minorities, only four experienced violent ethnic conflict, and three of these were various parts of former Yugoslavia. In this case, scholars' attraction to the dramatic cases led to a strangely "selected" body of scholarship, in which the full range of possibilities did not get examined.

SAMPLING FROM A POPULATION OF POTENTIAL OBSERVATIONS Both the Literary Digest example and the discussion of using college students to generalize to all adults are examples of the problem of sampling. We usually cannot study all possible observations to which our theory applies. We cannot, for example, ask every American adult how he or she expects to vote in an election. And, though it would be (barely) possible to do detailed analysis and fieldwork in each country of the world, limitations of time and resources usually bar us from doing so. As a result, we usually work with a sample drawn from the universe of all possible observations to which a theory applies. As noted earlier, the guiding rule in choosing observations for study is that the relationships we are looking for in the full universe of possible cases should be mirrored faithfully among the observations we are using.

Random Sampling To set [these races] aside, as is sometimes done, would be to set aside those cases in which incumbents have been most successful in generating electoral security, thereby understating the amount of electoral security that develops. On the other hand, to regard victory in an uncontested race as evidence that the MC [member of congress] has captured 100 percent of the vote would probably exaggerate M C s ' actual level of support, (p. 136)

In other words, if he ignored the selection problem, he would understate the amount of safety that long-time incumbents accrue. But if instead he treated the unchallenged incumbents as having gotten 100 percent of the vote, he would be overstating

When we are able to draw a fairly large number of observations, the "Cadillac" method is to draw a random sample from the full population of possible cases. In random sampling it is purely a matter of chance which cases from the full population end up in the sample for observation; that is, each member of the population has an equal chance of being drawn for the sample. It is as if we had flipped coins for each possible case and, say, admitted into the sample any case that got heads ten times in a row. (In reality, scholars use computer-generated random numbers to identify which members of the full population should join the sample for observation.)

100

Chapter 7

If a sample has been drawn randomly, we are assured that any relationship in which we are interested should be mirrored faithfully in the sample, at least in the sense that, across repeated samplings of this sort, on the average the relationships we would see would be the same as the relationship in the full population. Even random samples will diverge by accident from the full population. For instance, although there is a "gender gap" in American voting, with men favoring the Republican party more than women do, it would not be unusual for a sample of ten Americans to include a group of women who were more Republican than the men in the sample. Since in fact men are more likely to be Republicans than women are, most samples of ten Americans would show men to be the more Republican, but it would still happen fairly often, by chance, that the ten people you drew for a sample would show the reverse of that. If you took a large number of such samples, however, and averaged them, the average expression of the gender gap would almost surely mirror faithfully the gender gap in the full population. The principle here is the same as that in randomized experiments, which we looked at in Chapter 6. In randomized experiments, two groups are randomly chosen, so they cannot be expected to differ in any significant way that could interfere with a causal interpretation from the experiment. In other words, randomization ensures that any two groups chosen are essentially alike. But if that is true in experiments, it must also be true when trying to generalize to a population from a sample. If any group chosen randomly from the population can be expected to be essentially the same as any other, then they must all be essentially the same as the full population. How much variation there is from one random sample to another is hugely influenced by the size of the sample. The larger the sample, the less variation there will be from one time to another when we draw samples and look at a relationship. If you drew samples of a thousand Americans, for instance, it would happen only rarely that you could have come up with a group among whom the women were more Republican than the men. (Review the discussion of the Law of Large Numbers, p. 61. We saw there that variation in anything we are observing becomes less variable as we use larger and larger samples to examine it.) One nice thing about random samples is that there is a very precisely worked out mathematical system to ascertain, for a sample of any given size, exactly how likely it is that the sample result is any given distance from what you would have seen in the population. It is this mathematical system that allows pollsters to say of a national poll, for example, that it shows support of 48 percent for George Bush, within an error of plus or minus 3 percentage points. It always seems surprising that samples do not really need to be all that large to give an accurate rendition of a population. Typically, surveys of American citizens are considered to have sufficient cases for accurately mirroring the population of 302,000,000 if they have questioned a sample of two or three thousand people. It is kind of amazing that just a few thousand out of hundreds of millions would be sufficient to estimate the population to within a couple of percentage points.

Selection

of Observations for

Study

101

Now, while the true random sample is the Cadillac of sampling, it is rarely used, at least for large populations. For relatively small populations, such as the population of a city, true random sampling is often feasible, and is then, of course, the method of choice. But you can imagine what it would cost to sample the U.S. population in a truly random way—sending an interviewer to interview a single person in Menominee, Wisconsin, another to interview a person in San Diego, and so on. Quasi-random Sampling When a large population is involved, various methods exist to draw "quasi-random" approximations to a random sample. The National Election Study at the University of Michigan uses what is called a "cluster sample": About 100 localities are first selected, and then, within each of these, a certain number of individuals are randomly chosen for the sample. This allows the survey to use approximately 100 interviewers, one based in each of the localities chosen. Then, instead of having interviewers fly or drive hundreds of miles for a single interview, each interviewer can readily reach the 20 or 30 subjects randomly selected from her cluster. The end result approximates a random sample closely. And statistical adjustments can be made to take into account the clustered structure of the sample, so that it operates almost exactly like a random sample. An alternative system is to draw a truly random sample of telephones by having a computer dial telephones randomly ("random digit dialing," or RDD sampling). This allows access to a random sample without the difficulty or expense of getting an interviewer to the person's door, but it is increasingly difficult to draw a sample in this way that represents the broad population accurately. In 1932, at the time of the Literary Digest poll described earlier, the problem with telephone lists was that only the middle class had telephones. Today almost everyone has a telephone, so in principle RDD sampling could work well. But in practice, people's use of their phones varies enough so that the group that can actually be reached by phone at a given time is problematic as a sample. Some people are at work all day, so their telephones will ring at any daytime hour but not be answered unless by a machine. Others are always gone in the evenings. Some telephones have devices to block unsolicited calls. Some families have two phone lines, and therefore would be twice as likely to end up in the sample. RDD samples are rather problematic as approximations of a random sample of the population. Looking back now at the first two examples stated, we can see from our consideration of random sampling how they might (or might not) be questionable as samples. The Literary Digest poll was drawn before sampling was well understood. Its sample of telephone owners, subscribers to the magazine, and those appearing on state lists of registered automobiles was simply a terrible sample, very much skewed toward the middle class, and would have produced an erroneous prediction any time the middle class voted differently than the population as a whole. The sample of college students, on the other hand, might provide a reasonable basis for drawing psychological conclusions about people in general, even though they are clearly not

102

Chapter 7

Selection of Observations for Study

a random sample of the full population if we are confident that their psychological processes are the same as psychological processes among the full population. In this case, even though the sample is not random, it is much cheaper than trying to draw a random sample of all people, and can serve as a satisfactory quasi-random sample if processes are indeed the same as for the full population. Purposive Sampling Sometimes we deliberately want to sample in a nonrandom way, not just for reasons of efficiency or cost, but because a deliberately constructed, nonrandom sample serves a purpose for our research. If we wanted to compare the political choices of African Americans and whites, for instance, a purely random sample (or good approximation, such as a national cluster sample) would probably not yield enough African American citizens for a good comparison. If it faithfully mirrored the population, it should have about one-eighth of the sample African American, too small a group for reliable comparisons in a sample of the usual size. One solution would be to draw a monstrously large sample, but that would, of course, be very expensive. Another solution, if the purpose of the study is to look at the politics of race, is to draw what is called a "purposive" sample. A purposive sample does not attempt to replicate the full population. Rather, it draws subjects to maximize variation in the independent variable of interest, so that the relationships we are looking for will be very clear. In the preceding example, we could sample African Americans randomly and sample whites randomly, combining them to construct a sample that is half white and half African American. Like all purposive samples, this would not work as a general reflection of the full population. If African Americans were more likely to vote Democratic than whites, for instance, the sample would give an unreasonably high estimate of the Democratic vote. But the one thing for which the sample is going to be used, estimating a relationship or set of relationships based on race, is fine. Since both African Americans and whites have been randomly (or quasi-randomly) selected, an estimate of, say, the news-watching habits of the two races should reflect accurately the difference in news-watching habits in the full population. One reason that most sampling is random or quasi-random, rather than purposive, is that usually when we draw a sample, it is intended to serve multiple purposes. And we should probably also anticipate that as we proceed with analysis, we might want to look at things in the data other than what we had anticipated when we were first drawing up the sample. A purposive sample works only for the particular independent variable or variables on which we have based the sample. So it is usually safer and more useful to draw up a more general sample, random or quasi-random, rather than one focused just on the single task we start with. 1

Selection of Cases for Case Studies Another kind of purposive sampling comes into play when we are dealing with such small numbers of cases that random sampling would not be very helpful in any case. A case study is an intensive study of one or a few cases. It might be a study of a particular president's administrative style, an intensive study of the attitudes and ideologies of a few people, or, as it is most frequently seen, a study of some aspect of politics in one or a few countries. In principle, even if we are looking at only two or three cases, random selection of the case or cases would reflect the full population in the long run. But in practice, the Law of Large Numbers is working so weakly when we study only a few cases that the divergence of any case from the full population, though it will be "only random," will still be huge. Under these circumstances there is much more to be gained by intelligently choosing the case(s) to pick up the relationship of interest, rather than by randomly drawing the case(s). If we want to study the effect of authoritarian government on economic growth, for instance, it makes sense to seek a couple of authoritarian states and a couple of nonauthoritarian ones for comparison, rather than just taking four randomly chosen states. 2

CENSORED DATA The last three of the five examples in the introduction to this chapter were examples of "censored data," instances in which part of tue range of cases to which a theory applies are cut off and unavailable to the researcher, either by the researcher's choice or by circumstances. In the case of GRE scores, any generalization about the importance of test scores is obviously meant to apply to all students who could apply for admission, but the researcher can see the performance only of students who were admitted to the program. John Zaller's theory of congressional elections was obviously meant to apply to all members of Congress, but he could observe election outcomes only for those members who were challenged by an opponent. In the case of ethnic violence in eastern Europe, the theory that ancient hatreds would flare up after Soviet repression was lifted from the region obviously was meant to apply to all eastern European democracies with large ethnic minorities, but scholars had chosen to look only at the dramatic cases in which conflict had actually occured. The problem of censored data is always severe, because it requires researchers and their readers to make strong assumptions about what "might have been" in the range of data that are not available. Sometimes there are reasonable ways to fill in the spaces. Zaller, for instance, estimated on the basis of other variables what vote unchallenged members could have been expected to receive

2

'We will see (pp. 106-108) why we do not draw the sample to maximize variation in the dependent variable.

103

This is a situation that seems to fit Keynes' well-known comment that in the long run we shall all be dead. In principle and in the long run, a randomly selected sample of even just a couple of cases will mirror the full population. But any given couple of cases—for instance, the ones you plan to study—are likely to diverge sharply from the full population.

104

Selection

Chapter 7

had they been challenged, and inserted those estimates into his study as imputed data. And sometimes the answer is easy: If investigators have ignored some of the range but can study it, that just makes a neat and interesting study for another investigator to conduct, filling in for their short-sightedness. Mihaelescu was able to do a useful and rewarding study, just picking up on what others had not noticed: that most eastern European states with ethnic minorities had not exploded when the Soviets left.

10 -

of Observations for

105

Study

a

b •Singapore

Q O

-

. 'Taiwan South Korea

6 -

Growth in G D P per capita

•Brazil

Mexico*

4 -

d

c

2 -

When Scholars Pick the Cases They're Interested In 0 -

Often, as in this example of ethnic conflict in eastern Europe, investigators doing case studies gravitate to the outcomes in which they are interested, and pick cases for which the outcome was strong or dramatic. They then look at those cases to see what caused the thing in which they are interested. Scholars of revolution are prone to look at the Soviet Union or Cuba. Scholars of economic growth are prone to look at countries such as Korea, Taiwan, or Brazil, which have grown rapidly. Scholars of the presidency are likely to look at the administrations of dramatically successful presidents such as Franklin Roosevelt or Ronald Reagan, rather than less showy presidents such as Calvin Coolidge. I will argue later that choosing cases to maximize variation in the dependent variable distorts relationships, and that one should instead select to maximize variation in the independent variable or variables. But another result of choosing cases that exhibit most strongly the thing we are interested in is that it often also results in a censored data set. Barbara Geddes (2003) gives a nice example of this. She points out that a strong argument arose in the 1980s, based on case studies of rapidly developing countries, that it was important for developing countries' governments to repress organized labor to let industry develop rapidly. Most case studies of economic growth at the time focused on Mexico, Brazil, and the "Asian Tigers": Singapore, South Korea, and Taiwan. As Geddes points out, the focus on these several cases of rapid development led to a very wrong-headed conclusion. Geddes developed a measure for how much labor organization was repressed by governments, and then placed the five frequently studied countries on a graph of labor repression, related to growth in per capita GDP from 1970 to 1981. As you can see in her graph, reproduced in Figure 7-1, all five cases fall in the upper -right-hand part of the graph (b: high labor repression, rapid economic growth); and the lowest growth (Mexico's) also happens to coincide with a lower repression of labor. As Geddes showed in work adapted here in Figure 7-2, however, if we had all of the developing countries of the 1970s before us, we would see that there was little or no advantage for the countries that repressed labor. Looking at the full range of labor repression and the full range of economic outcomes, we find about as many low-growth countries with low levels of labor repression as with high labor repression. There is very little pattern in the figure, indicating little

-2 -4 0.0

0.5

1.0

1.5

2.0

2.5

1

1

1

3.0

3.5

4.0

4.5

5.0

Labor R e p r e s s i o n

Labor Repression and Growth in the Most Frequently Studied Cases, 1970-1981. (GDP per Capita from Penn World Tables.) Source: Geddes (2003), p. 101

Figure 7-1

relationship between the two variables. There may be a slight relationship; the upper left-hand part of the figure is a little less thickly filled with cases, indicating that there were somewhat fewer low-repression countries with high growth. But overall, there is nothing like the kind of relationship that appeared from the censored data in Figure 7-1. Figure 7-2 Labor Repression and Growth in the Full Universe of Developing Countries, 1970-1981. (GDP per Capital from Penn World Tables.) Adapted from Geddes (2003), p. 103 10 •Singapore 8 . »Taiwan South Korea

6

•Brazil ••

4 Growth in G D P per capita

Mexico* .

2 0 -2 -4 0.0

0.5

1.0

1.5

2.0

2.5

3.0

Labor R e p r e s s i o n

3.5

4.0

4.5

5.0

106

Selection of Observations for Study

Chapter 7

When Nature Censors Data What can one do about censored data? If the data have been censored because of decisions of researchers, the problem is easy to solve, and can in fact offer a nice chance to another researcher to make a significant contribution. This is what Mihaelescu or Geddes did, for example. All one has to do in this case is to seek out a broader range of information, and bring it into the analysis. But if the data have been censored by nature, as in the example of GRE scores, or Zaller's work on congressional elections, there are no easy solutions. The missing cases are gone and are not going to reappear. One cannot under ordinary circumstances admit to graduate school a group of students who have done badly on the GRE test to see how they perform. And one cannot decree that all congressional incumbents will be challenged for reelection. In such cases, we are forced to draw conclusions of what might have been had the data not been censored. This means we must make some assumptions, and justify them, for drawing "what if?" conclusions. We might, for example, seek out those graduate departments that for whatever reasons do not use the GRE as a criterion, find the GRE scores of students who attended there, and see how they did. This would not be a perfect solution; it would require the assumption that departments that did not require the GRE were like other departments in every other way, which is, of course, unlikely. And it would ignore the fact that the low-GRE students admitted to those departments would be atypical as well; since they would have been rejected by other departments but ended up being accepted into the non-GRE departments, they are probably unusually high achievers in other ways. Nonetheless, despite the fact that assumptions are required, this would be better than ignoring the initial problem of censored data. Similarly, Zaller had to make a number of assumptions to predict what vote unchallenged members would have gotten had they been challenged. A good deal of artfulness is required to analyze data that have been censored by nature, but a creative imagination will be rewarded by substantially improved estimation. And it's a nice challenge.

Numbers: White

Support Mayor

90,000

800,000

890,000

Don't Support Mayor

110,000

200,000

310,000

200,000

1,000,000

Nonwhite

White

Support

45%

80%

Don't

55%

20%

100%

100%

7-3 Race and Support for the Mayor: Hypothetical Polarized City

true population figures. Taking percentages, we find that 80 percent of whites support the mayor, but only 45 percent of nonwhites support the mayor. In Figure 7-4 I have drawn two samples of 400 from this population, one a purposive sample to maximize variation in race, and the other a purposive sample to maximize variation in support or opposition to the mayor. Figure 7-4(A) shows the results of drawing 200 nonwhites randomly from the pool of nonwhites shown in Figure 7-3, and similarly drawing 200 whites from the pool of whites. While this Figure

7-4 Selection Along Independent, Dependent Variables

(A) Selection Along Independent Variable Numbers:

Support

Don't

SELECTION ALONG THE DEPENDENT VARIABLE: DON'T DO IT! When we looked at selection of samples, we noted that you might usefully draw a purposive sample to get sufficient variation in the variables of interest. A very important rule, however, is that this can work well when you select cases to maximize variation on the independent variable, but that you should never select to maximize variation on the dependent variable. If you maximize variation in the independent variable, the relationship you observe will mirror nicely the true relationship, at least if you have enough cases so that the Law of Large Numbers can work for you. But if you maximize variation in the dependent variable, you will distort the true relationship. And it does not matter in this case whether you have a large or a small sample. Let us illustrate this with a hypothetical city with significant racial polarization. In Figure 7-3 we see the breakdown of support for the mayor, by race. These are the

Percentages:

Nonwhite

Figure

107

Percentages:

Nonwhite

White

86

171

262

114

29

138

200

200

Nonwhite

White

Support

43%

85%

Don't

57%

15%

(B) Selection Along Dependent Variable Percentages:

Numbers: Nonwhite

White

Support

39

161

200

Don't

64

136

200

103

297

Nonwhite

White

Support

38%

54%

Don't

62%

46%

108

Chapter 7

Selection of Observations for Study

sample is on the small side, it draws enough nonwhites to make fairly reliable comparisons with whites; and since it is drawn along the independent variable, it replicates faithfully the relationship in the full population. In the sample, 43 percent of nonwhites support the mayor (compared with 45 percent in the full population), as do 85 percent of whites (compared with 80 percent in the full population). The numbers diverge by about the amount we would normally expect in drawing a random sample of 400 cases, but the relationship is recognizably the same. In fact, it is more accurate than what we would have gotten from a purely random sample of the population, since that would have produced only a small number of nonwhites, with a widely variable estimate of nonwhites' support for the mayor. Figure 7-4(B) shows the results of a purposive sample of the same size that maximizes variation in the dependent variable, support for the mayor. As we see, drawing this sort of sample does not produce a faithful reflection of the relationship in the full population. In the full population, 80 percent of whites support the mayor, compared with 45 percent of nonwhites, a relationship of sharp polarization. But the sample maximizing variation in support appears to indicate a considerably weaker relationship, with 54 percent of whites supporting the mayor, compared with 38 percent of nonwhites. This does not present the same picture of sharp polarization. Why is it that sampling along the independent variable produces accurate estimates of the relationship, but sampling along the dependent variable does not? It makes sense, because the purpose of the analysis is to compare nonwhites' and whites' support for the mayor. Drawing groups from the two and comparing them fits this very well. But drawing groups of supporters and opponents of the mayor and then comparing whites' and nonwhites' support does not parallel the research question in the same way. And operationally, changing the relative numbers of supporters and opponents from what one sees in the full population changes the percent of supporters in both racial groups, in ways that distort the percentages in both groups. (Note that sampling on the independent variable distorts the proportions of whites and nonwhites in the population, but that this is actually helpful; it increases the number of nonwhites available for analysis. It leaves unaffected the percentages in which we are interested.) 3

SELECTION OF CASES FOR CASE STUDIES (AGAIN) The preceding example clarifies why one should not sample on the dependent variable, but it may puzzle the reader a bit. It looks a little artificial—why would anyone do this in the first place? Actually, this does not come up often with regard to large-scale sampling. It does come up time and again, however, with regard to 3

This sample, by the way, offers a good tangible example of about how much variability one gets with random sampling of this size.

109

intensive studies of one or a few cases (so-called case studies), and the logic there is just the same as the logic of the preceding example. I chose to introduce the idea through a large-scale survey example because the problem is easier to see there, but the real importance of the argument is with regard to case studies. It is so easy when studying a political outcome to choose a case in which the outcome occurred, and see whether the things you expected to have been the causes were present. Or, if you properly wanted to avoid the problem of censored data, you might choose two cases, one in which the outcome had occurred and another in which it had not, and see whether the things you expected to have been the causes were present. In fact, doing something like this is the most obvious and intuitive thing to do; see where the thing you're trying to explain has occurred, and try to account for it. It is intuitive, but it is wrong, for the same reason we saw in the earlier example—namely, that it distorts the likelihood that the outcome occurs or does not occur. Choosing instead cases that represent varying instances of your explanatory variable allows you to examine the full range over which your explanation is meant to apply, but it does not fiddle at all with the likelihood that the outcome occurs, and so allows you to examine straightforwardly where the chips fall under varying circumstances.

ANOTHER STRATEGY, HOWEVER: SINGLE-CASE STUDIES SELECTED FORTHE RELATIONSHIP BETWEEN THE INDEPENDENT AND DEPENDENT VARIABLES So far in this chapter I have dealt with issues having to do with: the selection of cases; the question of sampling from a larger population; the problem of censored data; and the question of whether to select cases on the basis of what you are trying to explain or on the basis of your independent variable. This has all been aimed at research in which we are comparing cases (anywhere from two cases to thousands) to look at the pattern of the relationship between independent and dependent variables. For such research, the selection of cases affects profoundly what results you see. Case-selection forms the foundation for all of your examination of evidence, and it will work well as long as your procedures fulfill the basic principle enunciated so far in this chapter: The relationship to be observed among the cases you have selected must mirror faithfully the true relationship among all potential cases. There is another kind of case study, however, in which you do not look at a relationship between variables across cases, but instead look at a single case to help clarify or illuminate a theory. For instance, Gerhard Loewenberg (1968) was struck by the fact that Germany by the late 1960s looked just the opposite of what we would have expected from theories of electoral systems. It had a proportional representation electoral system, and according to theory, this should lead to a multiparty system if the country had multiple divisions in society (see the discussion of Duverger's theory, pp. 2-3); and,

110

Chapter 7

Chapter 8

G e r m a n y had demonstrated in the 1920s and 1930s that it did indeed have a richly divided society. ( A t its highest, the number of parties represented in the prewar G e r m a n R e i c h s t a g had been 32!) B u t by the late 1960s G e r m a n y had almost a two-party 4

s y s t e m . H o w could this b e ? L o e w e n b e r g studied the G e r m a n case in depth to see what had produced this exception to the theory, in order to refine the theory and illuminate how it worked.

©

Note that L o e w e n b e r g did not p i c k this c a s e j u s t in terms of the dependent variable. He did not s i m p l y p i c k a c a s e w i t h a dramatic or interesting outcome. R a t h e r , he p i c k e d it b e c a u s e of the combination of the electoral s y s t e m and historic divisions

in

society

(independent v a r i a b l e s )

and the party

system

Measuring Relationships for Interval D a t a

(dependent

v a r i a b l e ) . W h e n a c a s e study is b e i n g u s e d in this w a y — n o t to test a theory by trying it out to see h o w w e l l it w o r k s , but rather to develop the theory by l o o k i n g in detail at a c a s e in w h i c h the theory is w o r k i n g in particular w a y s — we s h o u l d not ignore the dependent variable in s e l e c t i n g our c a s e . B u t we s h o u l d also not select the c a s e j u s t in terms of the dependent variable. Rather, we s h o u l d p i c k the c a s e b e c a u s e of how the theory appears to be w o r k i n g , as indicated by the c o m b i n a t i o n of dependent and independent variables. A n d , we s h o u l d then be c l e a r that we are not p r o v i d i n g a test of the theory. Rather, this is a strategy for further refinement of the theory.

5

In C h a p t e r 6 we w e r e c o n c e r n e d generally w i t h " r e l a t i o n s h i p s " b e t w e e n independent and dependent v a r i a b l e s . T h a t i s , we w a n t e d to see whether the p r e s e n c e or absence FURTHER DISCUSSION

of a test factor affected the v a l u e of the dependent variable. T h i s w a s too s i m p l e , for two reasons.

J o h n G e r r i n g (2007) provides a comprehensive and intelligent review of case selection.

F i r s t , as noted in footnote 5 in that chapter, the test factor in field r e s e a r c h is

G e d d e s ( 2 0 0 3 ) has a very nice chapter, " H o w the C a s e s Y o u C h o o s e A f f e c t the

frequently not something that is s i m p l y present or absent; rather, it takes on a variety

A n s w e r s Y o u Get." A good, though fairly technical, treatment of c e n s o r e d data is

of v a l u e s . O u r task then is not j u s t to see whether the presence or absence of a test

presented in P r z e w o r s k i et a l . ( 2 0 0 0 ) in an appendix, " S e l e c t i o n M o d e l s . " A n o t h e r

factor affects the value of the dependent variable, but instead, to see whether and h o w

good treatment of case selection is K i n g , K e o h a n e , and V e r b a ( 1 9 9 4 ) , Chapter 4.

the value of the independent variable affects the value of the dependent variable. F o r e x a m p l e , in relating education to i n c o m e , we do not treat people s i m p l y as " e d u c a t e d " or "not educated." T h e y have v a r y i n g amounts of education, and our task is to see whether the amount of education a person has affects his or her i n c o m e . S e c o n d , " r e l a t i o n s h i p " cannot be d i c h o t o m i z e d , although I treated it as a d i c h o t o m y in C h a p t e r 6, to ease the presentation there. T w o v a r i a b l e s are not s i m p l y related or not related. R e l a t i o n s h i p s v a r y in t w o w a y s : first, in h o w strongly the independent v a r i a b l e affects the dependent v a r i a b l e . F o r i n s t a n c e , e d u c a t i o n might h a v e o n l y a m i n o r effect o n i n c o m e . T h e average i n c o m e o f c o l l e g e graduates might b e $ 3 3 , 0 0 0 and the average i n c o m e o f h i g h s c h o o l dropouts m i g h t b e $ 3 0 , 0 0 0 . O r , it m i g h t h a v e a m a j o r effect. C o l l e g e graduates might average $ 5 0 , 0 0 0 , w h i l e h i g h s c h o o l dropouts a v e r a g e $ 1 5 , 0 0 0 . Relationships

also

v a r y in h o w

completely the

independent v a r i a b l e

deter-

m i n e s scores on the dependent variable. C o l l e g e graduates might average $ 5 0 , 0 0 0 4

Since then the system has moved back somewhat toward multipartism. Three parties were represented in the parliament in the 1960s, with the two largest (the Social Democrats and the Christian democrats) holding about 90 percent of the seats. Currently there are five parties represented, and the "big two" have 73 percent of the seats. This argument is developed in Shively (2006). 5

i n c o m e and h i g h s c h o o l dropouts $ 1 5 , 0 0 0 , for i n s t a n c e , yet there might still b e m u c h variation in i n c o m e s that c o u l d not be attributed to variation in p e o p l e ' s education. S o m e c o l l e g e graduates might m a k e only $ 1 0 , 0 0 0 a y e a r and s o m e h i g h s c h o o l dropouts might m a k e $ 8 0 , 0 0 0 or $ 9 0 , 0 0 0 a y e a r , e v e n though the average i n c o m e of

111

112

Chapter 8

the college graduates was higher than the average income of the dropouts. This would indicate that although education affected incomes sharply, it was relatively incomplete as an explanation of people's incomes. Because income still varied a good deal within each level of the independent variable, there must be other things affecting income in important ways, and we often would guess incorrectly if we tried to predict a person's income solely on the basis of education. This is what it means to say that education is not a very "complete" explanation of income. Thus variables are not simply "related" or "not related." Their relationship may be such that the independent variable has a greater or lesser effect on the dependent variable; and it may be such that the independent variable determines the dependent variable more or less completely. Generally speaking, political research is not so much concerned with whether or not two variables are related but with whether or not they have a "strong" relationship (in one or both of the senses used earlier). This can be seen in our examples of research design in Chapter 6. Although for the sake of simplicity these were presented as if we were interested only in whether or not a relationship existed, it is clear that what was of interest to the investigators in each case was finding out how strong a relationship existed. In "Presidential Lobbying," for instance, the president was not simply concerned with whether or not he was able to influence voting for the bill, but with how many votes he could swing. Our task in evaluating the results of research, then, is to measure how strong a relationship exists between the independent variable(s) and the dependent variable. The tools we need to accomplish this task are found in the field of statistics.

STATISTICS Although modern political scientists have begun to use statistics extensively only in the past several decades, it was actually political scientists of a sort who first developed the field, for statistics originally grew out of the need to keep records for the state. The name statistics derives from the Latin statisticus, "of state affairs." Statistics includes two main activities: statistical inference and statistical measurement (including the measurement of relationships, with which we are concerned in this chapter). Statistical inference consists of estimating how likely it is that a particular result could be due to chance; it tells us how reliable the results of our research are. I discuss inference in Chapter 10. In this chapter and in Chapter 9, I introduce some statistical techniques for measuring the strength of relationships.

Introduction to Statistics

113

relationships between variables are different depending on the level at which the variables were measured. If we know more about a relationship, we should be able to measure a greater variety of things about it. Just as higher levels of measurement yield relatively richer information about a variable, so techniques for measuring relationships at high levels of measurement give relatively richer information about those relationships. Basically, there are two major types of techniques: those appropriate for data measured at the interval and those suited for lower-level measurements. Recall that we mentioned two ways to measure the "strength" of a relationship between two variables: (1) by how great a difference the independent variable makes in the dependent variable, that is, how greatly values of the dependent variable differ, given varying scores on the independent variable; or (2) by how completely the independent variable determines the dependent variable, that is, how complete an explanation of the dependent variable is provided by the independent variable. I shall call the first way of measuring a relationship effect-descriptive, and the second correlational. The critical difference between working with interval-measured data and working with data measured at a lower level is that effect-descriptive measurement can apply only to interval-scale data. Correlational measurement of one sort or another can apply to data measured at any level. Ordinal and nominal measurement techniques cannot tell us how great a difference in the dependent variable is produced by a given difference in the independent variable, although this is precisely what is required to measure the relationship in an effect-descriptive way. The whole point of nomi lal and ordinal measurement is that in neither do we have available a unit by which to measure the difference between two values of a variable. This means that we cannot measure how great a difference is induced in the dependent variable by a change in the independent variable. If we are using an ordinal-scale variable, we know whether one value is higher than another, but we do not know how much higher it is. If we are using a nominal-scale variable, of course, all we know is whether the two values are distinct. This becomes a particularly important distinction in political research, because under most circumstances, effect-descriptive ways of measuring the strength of a relationship are more useful than correlational ways. I demonstrate this in the next few sections.

WORKING WITH INTERVAL DATA Regression Analysis

IMPORTANCE OF LEVELS OF MEASUREMENT In Chapter 5 we saw that we have more information about a relationship between variables if we work at a higher level of measurement than if we work at a lower level of measurement. It should not be too surprising that methods of measuring

A convenient way to summarize data on two interval-scale variables so that we can easily see what is going on in the relationship between them is to plot all the observations on a scattergram, as in Figure 8 - 1 . Each dot in the scattergram represents one observation (a person, state, or country, for example), placed on the graph according to its scores on the two variables. For instance, dot A represents an observation that

114

Introduction to Statistics

Chapter 8 (A)

115

(D)

Independent Variable Figure 8-2

Assorted Scattergrams

separate relationships. It w o u l d be painful to read such a paper if e a c h relationship w e r e presented in the f o r m of a scattergram. W h a t is m o r e , c o m p a r i n g t w o scattergrams gives us o n l y an approximate idea of the differences b e t w e e n t w o relationJ

I

I

I

I

I

I

L

Independent Variable

s h i p s . C o m p a r i n g t h e g r a p h s i n F i g u r e s 8 - 1 a n d 8 - 2 ( B ) , w e c a n s a y t h a t i n t h e first graph the i n d e p e n d e n t variable c a u s e s greater shifts in the d e p e n d e n t variable than in t h e s e c o n d , b u t w e c a n n o t s a y p r e c i s e l y h o w m u c h g r e a t e r t h e s h i f t s are. I f t h e

Figure 8-1

Scattergram

differences w e r e m o r e subtle, or if we w e r e c o m p a r i n g several relationships, the j o b would become well-nigh impossible.

c o m b i n e s scores of 3.5 on the independent variable and 5 on the d e p e n d e n t variable.

F i n a l l y a n d m o s t i m p o r t a n t , w e o f t e n m e a s u r e t h e strength o f a r e l a t i o n s h i p

D o t B represents an observation with scores of 12 on the independent variable and 17

b e t w e e n t w o v a r i a b l e s w h i l e h o l d i n g a third v a r i a b l e c o n s t a n t ( s e e t h e d i s c u s s i o n o f this

on the dependent variable.

t o p i c o n p p . 9 4 - 9 5 ) . T o d o t hi s u s i n g s c a t t e r g r a m s m a y b e e x t r e m e l y c u m b e r s o m e .

By l o o k i n g at the pattern f o r m e d by the d o t s , we c a n tell a g o o d d e a l about the

F o r all o f t h e s e r e a s o n s , i t i s u s e f u l t o d e v i s e a p r e c i s e n u m e r i c a l m e a s u r e t o

r e l a t i o n s h i p b e t w e e n t h e t w o v a r i a b l e s . F o r i n s t a n c e , i n F i g u r e 8 - 1 , w e n o t e that

s u m m a r i z e the relevant characteristics of a relationship s h o w n in a scattergram. T h e

t h e r e are f e w d o t s i n t h e l o w e r - r i g h t a n d u p p e r - l e f t c o r n e r s o f t h e g r a p h . T h i s m e a n s

measure c o m m o n l y u s e d to summarize the effect-descriptive characteristics of a

that h i g h s c o r e s o n t h e d e p e n d e n t v a r i a b l e t e n d t o c o i n c i d e w i t h h i g h s c o r e s o n t h e

s c a t t e r g r a m is t h e regression coefficient.

independent variable, and l o w scores on the dependent variable tend to c o i n c i d e with

T h e l i n e a r r e g r e s s i o n c o e f f i c i e n t i s d e r i v e d i n t h e f o l l o w i n g w a y . First, t h e pattern

l o w s c o r e s o n t h e i n d e p e n d e n t v a r i a b l e . T h u s w e k n o w that t h e t w o v a r i a b l e s are

i n t h e d o t s o f a s c a t t e r g r a m i s s u m m e d u p b y t h e s i n g l e l i n e that b e s t a p p r o x i m a t e s it.

p o s i t i v e l y related. Furthermore, this relationship appears to be a p p r o x i m a t e l y linear

For a linear relationship, the m a t h e m a t i c a l l y best procedure is to c h o o s e the unique line

( s e e the d i s c u s s i o n of "linear relationships" in the b o x on p. 7 1 ) .

that m i n i m i z e s t h e s q u a r e d d i f f e r e n c e s b e t w e e n o b s e r v e d v a l u e s o f t h e d e p e n d e n t

W e h a v e d o n e t w o t h i n g s s o far. W e h a v e o b s e r v e d w h i c h w a y t h e d e p e n d e n t

v a r i a b l e a n d its i d e a l i z e d v a l u e s a s g i v e n b y t h e s i m p l i f y i n g line. T h i s i s illustrated i n

v a r i a b l e m o v e s w i t h c h a n g e s i n t h e i n d e p e n d e n t v a r i a b l e , a n d w e h a v e o b s e r v e d that

Figure 8-3, w h e r e a simplifying line has b e e n drawn through a scattergram with obser-

i t m o v e s a t a s t e a d y rate a t a l l v a l u e s o f t h e i n d e p e n d e n t v a r i a b l e ( a l i n e a r r e l a t i o n -

v a t i o n s o n s e v e n h y p o t h e t i c a l c o u n t r i e s t o s u m m a r i z e t h e pattern a c r o s s t h e c o u n t r i e s .

ship) rather than at c h a n g i n g rates (a n o n l i n e a r r e l a t i o n s h i p ) . T h e s e are b o t h part of

It has b e e n drawn to m i n i m i z e the squared differences b e t w e e n e a c h of the observed

an effect-descriptive m e a s u r e m e n t of the relationship.

p o i n t s , s u c h a s A , a n d t h e p o i n t B a t w h i c h a c o u n t r y h a v i n g A's s c o r e o n t h e i n d e -

T h e scattergrams i n F i g u r e 8 - 2 illustrate various other patterns w e m i g h t have

p e n d e n t v a r i a b l e w o u l d b e e x p e c t e d t o fall o n t h e i d e a l i z e d s i m p l i f y i n g l i n e .

observed. Graph A s h o w s a nonlinear relationship (the dependent variable increases

T h e s i m p l i f y i n g l i n e m a y be t h o u g h t of as a rule for predicting s c o r e s on the

faster w i t h increases in the i n d e p e n d e n t variable if the i n d e p e n d e n t variable h a s a

d e p e n d e n t v a r i a b l e f r o m s c o r e s o n t h e i n d e p e n d e n t v a r i a b l e s . Its u s e f u l n e s s a s a

high value). Graph B s h o w s a linear relationship in w h i c h the d e p e n d e n t variable

predictor d e p e n d s on k e e p i n g the average squared v a l u e of "deviant" s c o r e s on the

i n c r e a s e s m o r e gradually than in the graph in F i g u r e 8 - 1 . Graph C s h o w s a negative

dependent variable as l o w as possible. A single s u m m a r i z i n g line can be described

linear relationship in w h i c h the d e p e n d e n t variable d e c r e a s e s as the i n d e p e n d e n t

m o r e e a s i l y than a pattern of dots. In particular, a straight line s u c h as this c a n be

variable increases. Graph D s h o w s a pattern in w h i c h there is no relationship.

fully described by the equation \

A l t h o u g h the scattergram tells us a g o o d deal about a relationship, it can be u n w i e l d y to w o r k with. It is not u n c o m m o n in a research report to d i s c u s s 30 or 40

y = a + bx

116

Chapter 8

Introduction to Statistics

117

value of the independent variable, you must add a (the predicted value of y when x equals zero) to b times the number of units by which x exceeds zero. The slope, often simply called the regression coefficient, is the most valuable part of this equation for most purposes in the social sciences. By telling how great a shift we can expect in the dependent variable if the independent variable shifts by one unit, the slope provides a single, precise summary measure of how great an impact the independent variable has on the dependent variable. Let's assume, for instance, that the relationship between income and electoral participation is linear and can be summarized by the regression equation percent voting = 50.5 + — (income in thousands of dollars) This means that with every additional thousand dollars of income, V percent more potential voters vote. For example, if income is $20,000, the predicted percent voting equals 50.5 + (1/2 x 20), or 60.5. If income is $30,000, the predicted percent voting equals 50.5 + (1/2 x 30), or 65.5. Remember, however, that even though we work with neat, impersonal numbers, we do not escape the scholar's obligation to think. If we have guessed the direction of causation between x and y incorrectly, plugging in our data and getting numbers out will not make the results valid. If y causes x rather than vice versa, the formulas will still give us an a and a b. but the shift of one unit in x in the real world will not be followed by a shift of b units in y. Thus the arguments made in Chapter 6 apply even when we work with simple numbers like these. The problem of comparing units. It may be seen from an examination of the concept slope (the number of units by which y changes with a change of one unit in x) that the slope has meaning only with regard to the units in which x and y are measured. For example, if there is a regression coefficient of—10.5 for nations' diplomatic involvement with the United States (measured by the number or magnitude of exchanges per year) predicted from their distance from the United States measured in thousands of miles, there would be a regression coefficient of —0.0105 for the same dependent variable if distance were measured in miles. That is, if diplomatic involvement could be expected to decrease by 10.5 with every 1,000 miles of distance from the United States, it would be expected to decrease by 0.0105 with every mile of distance. If we are working with just two variables, this poses no real difficulty. But often we may be interested in comparing the effects of two or more independent variables on a particular dependent variable. If the two independent variables are measured in different sorts of units, this can be difficult. Continuing with our example, we might want to know which variable—nations' distance from the United States or the volume of their trade with the United States—has a greater impact on their diplomatic interaction with the United States. If the regression coefficient for volume of trade, measured in millions of dollars, is +0.024, how can we tell whether it has a greater impact on diplomatic interaction than does geographic distance, with its slope of—10.5? 2

Figure 8-3

The Regression Line

where y is the predicted value of the dependent variable, x is the value of the independent variable, and a and b are numbers relating y to x. The number a, the expected value of y when x equals zero, is called the intercept of the regression equation; it is the value of y where the regression line crosses the y axis, that is, where x equals zero (see Figure 8-4). The number b, or the slope of the regression equation, shows by how many units y increases as x increases one unit. (If b is negative, y decreases as x increases; there is a negative relationship between the variables.) In other words, to find the predicted value of the dependent variable for any specified Figure 8 - 4

The Regression Equation

The equation of this line is y = 6 + 3x. The predicted value of y w h e n x is 4, for instance, is 6 + (4 x 3), or 1 8.

Independent Variable

118 Chapter 8 The units—thousands of miles and millions of dollars—are not comparable; therefore, the coefficients based on those units are not comparable either. The real answer, of course, is that the numbers do have meaning, but only with relationship to their original concepts. If the concepts are not easy to compare, then regression coefficients referring to them will also not be easy to compare. The solution to the problem requires going back to theory development, to consider on what grounds your concepts can be related to each other. Apples and oranges actually can be compared, contrary to the old saying about "comparing apples and oranges," but to do so we must first specify along what dimensions our theory calls for them to be compared. We can compare them in a nutritional theory, for instance, by measuring how much vitamin C each fruit contains. Or, we could compare them in a theory of gravity by comparing their mass. To compare the effects of diplomatic interaction and geographic distance in the example above, we would have to develop in our theory dimensions along which they could be meaningfully compared. Checking for linearity. We must be careful to make certain in using linear regression analysis that the data do fit a more or less linear pattern. Otherwise, the regression equation will not summarize the pattern in the data, but will distort it. Figure 8-5 shows a linear regression equation passed through the scattergram of a nonlinear relationship. This regression line fits the data very badly and is not a useful summary of the relationship. You should always check your data before using linear regression analysis. The best way to do this is simply to draw a scattergram (or rather, let the computer do it for you) and see whether it looks linear. Many relationships in political science do turn out to be linear. But if the relationship you are investigating turns out to be nonlinear, that is no reason to give up analyzing it. It merely means that the relationship is more complex than you anticipated—and probably more interesting. A nonlinear regression equation may be found to fit the pattern fairly well.

Introduction to Statistics 1 1 9

0

Figure 8-6

Nonlinear Regression Equation 2

Figure 8-5

In Figure 8-6, a nonlinear regression equation y - a + b x — b x has been passed through the scattergram from Figure 8-5. It summarizes the pattern in the data more accurately. In this case, two coefficients, b\ and b , are required to express the impact that a change in x will have on y, since that change is not the same at all values of x. We can see that y increases with x but decreases with the square of x. The regression equation provides a handy summary description of the effect, now a bit more complicated, that x has on y. Formulas are available to calculate equations for regression lines satisfying the least-squares criteria. This is particularly true for linear regression; the formulas for a and b are found in every standard statistics text, including the ones cited at the end of this chapter. But there are no set "formulas" for nonlinear regression equations, for there is an infinite variety of nonlinear equations that you might fit to any set of data. It usually is necessary to play around with alternative nonlinear equations for a while. But these, too, can be worked out readily enough. To the extent that your research is based on a well-thought-out theory, this will help you to design an appropriate nonlinear equation. For instance, if your theory predicts that the dependent variable will always increase with increases in the independent variable, but at a constantly diminishing rate (a "diminishing marginal returns" model), the equation depicted in Figure 8-6 would be inappropriate because it must inevitably reverse direction at some point. An equation such as y = a + b log x, as depicted in Figure 8-7, would be appropriate. One important warning: Remember that the presentation I have made here is only a broad, introductory overview. Competence in using measures like those presented here requires more thorough training than is within the scope of this book.

0

Examining the residuals. Any regression line is actually a reflection of the stage that we have reached in developing a theory. Because our theory anticipates a relationship between two variables, we measure the relationship between them by

Linear Regression on a Nonlinear Relationship

x

2

2

120

Introduction to Statistics

Chapter 8

121

that it w o u l d h a v e b e e n difficult to identify the p r e s e n c e of a n o t h e r factor if you - a + b log x

h a d n o t first r e g r e s s e d e d u c a t i o n a l s p e n d i n g o n G N P . T h e r i c h e s t d e m o c r a c i e s i n F i g u r e 8-3 s h o w a h i g h e r rate of e d u c a t i o n a l s p e n d i n g t h a n do t h e p o o r e s t n o n d e m o c r a c i e s , so that it m i g h t n o t h a v e b e e n at all o b v i o u s that " d e m o c r a c y " w a s a variable you should use to explain levels of educational spending. T h e technique of e x a m i n i n g residuals is illustrated in Figure

8-8, adapted

f r o m V . O . K e y , J r . ' s Southern Politics ( 1 9 5 0 , p . 4 8 ) . K e y w a n t e d t o m e a s u r e t h e i m p a c t of factions in A l a b a m a p r i m a r i e s , so he related c o u n t i e s ' votes for F o l s o m , a progressive

candidate

in

the

1946

gubernatorial

primary,

to

their

votes

for

Sparkman, a progressive candidate in the 1946 senatorial primary. He found a m o d erately strong relationship b e t w e e n the t w o . B e c a u s e this m e a n t that counties tended to lean the same way in both elections, it indicated the presence of conservative 0

and progressive

Figure 8-7

factions

structuring

the

vote,

as

Key had expected.

Model of Diminishing Marginal Returns Figure 8-8

c a l c u l a t i n g t h e r e g r e s s i o n e q u a t i o n for t h e line t h a t b e s t s u m m a r i z e s t h e p a t t e r n o f the

Example of Residual Analysis

Source: V. O. Key, Jr., Southern Politics (New York: Alfred A. Knopf, 1 950), p. 4 8 .

relationship. In this sense, regression analysis is an e x p r e s s i o n of w h a t we already think about the subject. It m a y surprise u s — w h e r e we expected a relationship there

100r

m a y be none, or it m a y be nonlinear, and so on. But it is an expression of what we already h a v e b e e n thinking about the subject. However,

the

regression

line

can

serve

a further important

function;

by

pointing out important independent variables that had not yet occurred to us, it can help refine our theory. L o o k i n g b a c k at Figure 8-3, note that the regression line d o e s n o t p r o v i d e perfect p r e d i c t i o n of the v a l u e s on the d e p e n d e n t variable for the c a s e s i n t h e s c a t t e r g r a m . T h i s m e a n s t h a t t h e r e i s still u n e x p l a i n e d v a r i a t i o n i n the values of the dependent variable. S o m e of the observations are higher on the dependent variable than variable,

also

we would expect from their value on the independent

and s o m e are lower.

affecting

the

dependent

Something

else,

beyond the

independent variable,

is

variable.

This difference b e t w e e n the observed value a n d the predicted value is called the residual. E x a m i n i n g t h e s e r e s i d u a l s p o i n t s o u t t o u s t h o s e c a s e s i n w h i c h t h e " s o m e t h i n g e l s e " h a s t h e effect o f r a i s i n g (or, c o n v e r s e l y , l o w e r i n g ) the d e p e n d e n t v a r i a b l e . I n F i g u r e 8-3, for i n s t a n c e , a c a s e s u c h as A is o n e in w h i c h the effect of t h e " s o m e t h i n g e l s e " is to raise the v a l u e of t h e d e p e n d e n t variable; t h e actual value for c a s e A is h i g h e r than the v a l u e for B , w h i c h w o u l d h a v e b e e n p r e d i c t e d from the r e g r e s s i o n line. N o w , o n c e the cases have been sorted out in this way, we m a y notice that cases with similar residuals have s o m e additional characteristic in c o m m o n ; this may

then

suggest an additional variable that m a y be brought into our theory.

C o n s i d e r Figure 8-3. On e x a m i n i n g the residuals in that figure, you m i g h t notice that all o f t h e c o u n t r i e s for w h i c h e d u c a t i o n a l s p e n d i n g w a s h i g h e r t h a n p r e d i c t e d w e r e not d e m o c r a c i e s , a n d all o f t h e c o u n t r i e s for w h i c h i t w a s u n e x p e c t e d l y l o w were democracies.

In this w a y

you might have discovered the identity of the

" s o m e t h i n g else," b e y o n d G N P , that acts as an independent variable. Notice also

Percent for Sparkman, Special Primary, July 3 0 , 1 9 4 6

Had

such

122

Introduction to Statistics

Chapter 8

factions not existed, there would have been no particular reason to expect a county to vote in much the same way in the two primaries. But the relationship was not very tight. Many counties, such as the one that gave 90 percent of its vote to Folsom but only 45 percent to Sparkman, voted far differently from what one would have expected simply on the basis of conservative and progressive factions. This indicated the presence of other variables, which were causing additional variations in the vote. By examining the residuals around the regression line, Key got some idea of what those variables might be. In this case it turned out that the residuals could be explained in part by the effect of "friends and neighbors." Counties in Folsom's home part of the state voted for him more enthusiastically than would have been expected on the basis of their vote for Sparkman. Counties in Sparkman's home part of the state voted less enthusiastically for Folsom than would have been expected from their vote for Sparkman (which presumably was high because he was a local boy). Similarly, the "home" counties of Folsom's opponent went less heavily for Folsom than would have been expected on the basis of their vote for Sparkman. This pointed out to Key the importance of local solidarity, one of the major forces retarding the development of stable statewide factions in Alabama politics at that time. Much of the looseness in the relationship between the votes for two candidates of the same faction was shown to be a result of people's tendency to vote for a candidate on the basis of where he came from in the state rather than the faction with which he was identified. Users of regression analysis in political science far too rarely go on to the creative and exploratory labor of examining the residuals to see what additional variables affect the dependent variable. Usually, the spread of dots around the regression line is treated as an act of God, or as a measure of the basic uncertainty of human affairs. On the contrary, it is a trove in which new variables lie waiting to be discovered. I suspect the reason most of us do not go on to examine this trove is that we have developed a proprietorial sense toward our theories before we ever get to the point of testing them. There is a certain completeness about one's own theory, and it does not occur to us to use our theory as a "mere" starting point in the search for explanations. Correlation Analysis At the beginning of this chapter I pointed out that there are two ways to measure the strength of a relationship: by measuring how much difference the independent variable makes in the dependent variable and by measuring how completely the independent variable determines the dependent variable. For interval-scale data, the regression coefficient accomplishes the first of these; the correlation coefficient accomplishes the second. Consider the graphs in Figure 8-9. Both relationships can be summarized by the same regression line, but the value of the dependent variable in graph B is less closely determined by the independent variable than in graph A. A change in the independent

(B)

(A)

Figure 8-9

123

Two Correlations

variable tends to produce the same change in the dependent variable, on the average, in both graphs. But this tendency is weaker, and more likely to be disturbed by "other factors," in graph B; in other words, the residuals tend to be larger in graph B than in graph A. In one sense, then, the relationship in graph B is weaker than that in graph A. The dependent variable is less a result of the independent variable, compared to "other factors" (the unknown things that cause the residuals to exist), in B than in A. The correlation coefficient, r, measures how widely such a body of data spreads around a regression line. This coefficient compares a set of data with ideal models of a perfect relationship and a perfect lack of relationship, and assigns to the relationship a score ranging in absolute value from 0 to 1, depending on how closely the data approximate a perfect relationship. The two extreme models are illustrated in Figure 8-10. In graph A of Figure 8-10, the data all fall on a straight line through the scattergram. A regression line passed through them would leave no residual variation at

Figure 8-10

Extreme Models of Correlation

(A)

(B)

124

Chapter 8

Introduction to Statistics

all i n t h e d e p e n d e n t v a r i a b l e . T h u s the i n d e p e n d e n t v a r i a b l e d e t e r m i n e s t h e d e p e n d e n t

2

(-i -

v a r i a b l e c o m p l e t e l y . T h e c o r r e l a t i o n c o e f f i c i e n t f o r this t y p e h a s a n a b s o l u t e v a l u e o f 1 .

y) + 3

2

(4

-y) + 3

(5 -

7)

125

2

3

3

I n g r a p h B , o n t h e o t h e r h a n d , v a l u e s o f t h e d e p e n d e n t v a r i a b l e are c o m b i n e d

2

4

2

( - "AO + ( / ) + (7 )

r a n d o m l y w i t h v a l u e s o f t h e i n d e p e n d e n t v a r i a b l e , s o that a n y g i v e n v a l u e o f t h e

3

2

3

3

dependent variable is as likely to coincide with a l o w value on the independent 121

variable as w i t h a h i g h o n e . T h u s there is no pattern to the relationship. T h i s indicates

= —

that t h e i n d e p e n d e n t v a r i a b l e h a s n o e f f e c t o n t h e d e p e n d e n t v a r i a b l e . T h e c o r r e l a t i o n

4

A, + " 7 9 + 7

9

= 6.89

3

coefficient for this t y p e has a b s o l u t e v a l u e of 0. M o s t r e l a t i o n s h i p s , o f c o u r s e , fall s o m e w h e r e b e t w e e n t h e s e t w o e x t r e m e s ; the

T h e f o r m u l a for the variance of any variable x is

c l o s e r a r e l a t i o n s h i p a p p r o a c h e s t h e s i t u a t i o n i n g r a p h A , t h e h i g h e r its c o r r e l a t i o n

E(x -

N

m e a s u r e by w h i c h the strengths of various relationships can be c o m p a r e d , in the correlational s e n s e o f " s t r e n g t h o f r e l a t i o n s h i p . " I h a v e referred here o n l y to the absolute v a l u e of the correlation coefficient. In

xf

variance-, =

coefficient will be in absolute value. Thus the correlation coefficient provides a

w h e r e x is the m e a n of x and N is the n u m b e r of observations we w i s h to average. T h e 2

s i g n s i m p l y m e a n s that for all t h e o b s e r v a t i o n s , we are to c a l c u l a t e

and then add t h e s e results together. T h e e x p r e s s i o n 2 (x

-

t h e c o r r e l a t i o n c o e f f i c i e n t i n d i c a t e s b y its s i g n w h e t h e r t h e r e l a t i o n s h i p i s p o s i t i v e o r

(x

is

negative. T h e coefficient ranges from - 1 . 0 (a perfect negative relationship, such as

first c a s e , x t h e o b s e r v a t i o n for t h e s e c o n d c a s e , a n d s o o n .

addition to s h o w i n g h o w c l o s e l y a relationship approaches the situation in graph A,

x

-

xf

+

(x

2

-

xf

+

...

+

(x

N

-

xf,

where x

x

(x - xf

xf is equivalent to writing the

observation

for the

2

t h e o n e i n g r a p h A o f F i g u r e 8-10) t h r o u g h 0 . 0 ( g r a p h B ) t o + 1 . 0 ( a p e r f e c t p o s i t i v e

T h e variance is a measure of h o w w i d e l y the observed values of a variable

relationship, similar to the o n e in graph A but tilted up). In a positive relationship,

vary a m o n g t h e m s e l v e s . I f t h e y d o n o t vary a t all but e a c h h a s t h e s a m e v a l u e , the

increases in the dependent variable produce increases in the independent variable; in

variance w i l l b e zero. T h i s i s true b e c a u s e e a c h v a l u e w i l l e q u a l the m e a n , and

a negative relationship, increases in the independent variable produce decreases in

thus the s u m of the squared deviations from the m e a n will equal zero. T h e more

the d e p e n d e n t variable.

the v a l u e s vary a m o n g t h e m s e l v e s , the further e a c h will b e f r o m the m e a n o f t h e m

Interpreting the correlation coefficient.

A l t h o u g h it i s c l e a r e n o u g h w h a t

correlation coefficients o f - 1 , 0 , o r + 1 m e a n , there i s n o direct w a y o f interpreting w h a t c o e f f i c i e n t s b e t w e e n t h e s e v a l u e s m e a n . I t i s true t h a t t h e h i g h e r t h e a b s o l u t e v a l u e o f t h e c o e f f i c i e n t , t h e c l o s e r i t a p p r o a c h e s t h e m o d e l i n g r a p h A o f F i g u r e 8-10, s o that i f w e w i s h t o c o m p a r e t w o different r e l a t i o n s h i p s , w e c a n s a y w h i c h i s s t r o n g e r . B u t i t i s n o t e a s y t o s e e w h a t t h e d i f f e r e n c e b e t w e e n t h e m m e a n s . I t i s not true, f o r i n s t a n c e , that t h e d i f f e r e n c e b e t w e e n r = 0.8 a n d r = 0 . 6 is t h e s a m e as t h e d i f f e r e n c e b e t w e e n r = 0 . 4 a n d r = 0.2. A n d it is a l s o n o t true that r = - 0 . 6 is t w i c e as s t r o n g a s r = -0.3. T h i s i s r e m i n i s c e n t o f t h e d i f f e r e n c e b e t w e e n o r d i n a l a n d i n t e r v a l m e a s u r e m e n t : We k n o w that t h e h i g h e r the a b s o l u t e v a l u e of r, the stronger the relat i o n s h i p ; b u t w e d o n o t k n o w how much s t r o n g e r o n e r e l a t i o n s h i p i s t h a n a n o t h e r . Fortunately, the square of the correlation coefficient ( s o m e t i m e s c a l l e d the 2

coefficient of determination, b u t m o r e o f t e n j u s t r ) d o e s h a v e a u s a b l e i n t e r p r e t a t i o n a t all v a l u e s o f r . B e f o r e w e c a n c o n s i d e r t h i s , h o w e v e r , I m u s t first i n t r o d u c e t h e concept

all, and c o n s e q u e n t l y , the greater the s u m o f s q u a r e d d e v i a t i o n s f r o m the m e a n . T h u s , the m o r e that v a l u e s vary a m o n g t h e m s e l v e s , the h i g h e r their variance will be. T h e variance of the dependent variable y can be depicted in a scattergram by d r a w i n g a h o r i z o n t a l l i n e at y = y, a n d d r a w i n g in the r e s i d u a l s f r o m this l i n e , a s i n F i g u r e 8-11. T h e a v e r a g e s q u a r e d v a l u e o f t h e r e s i d u a l s , b e c a u s e i t i s the a v e r a g e squared d e v i a t i o n of the v a l u e s of y f r o m their o w n m e a n , is the v a r i a n c e of y. O n e w a y t o v i e w o u r g o a l i n t h e o r e t i c a l s o c i a l s c i e n c e r e s e a r c h i s t o n o t e that our task is u s u a l l y to a c c o u n t for the variance in a d e p e n d e n t variable. It is the variance in s o m e t h i n g that p u z z l e s us and c h a l l e n g e s us to p r o d u c e an explanation. W h y i s i t that s o m e p e o p l e m a k e m o r e m o n e y than other p e o p l e ? T h a t s o m e nations are f r e q u e n t l y i n v o l v e d i n w a r s a n d o t h e r s are n o t ? T h a t s o m e m e m b e r s o f C o n g r e s s v o t e f o r a b i l l a n d o t h e r s o p p o s e it? T h a t s o m e p e o p l e are m o r e p o l i t i c a l l y a l i e n a t e d than others? A l l o f t h e s e q u e s t i o n s s i m p l y ask: T o w h a t c a n the v a r i a n c e i n this

o f variance.

v a r i a b l e be a t t r i b u t e d — w h y does this variable vary?

Variance. T h e v a r i a n c e o f a variable i s t h e a v e r a g e s q u a r e d d e v i a t i o n o f v a l u e s o f that 1

C o m p a r i n g F i g u r e s 8-3 a n d 8-11, we s h o u l d s e e at l e a s t a s u p e r f i c i a l s i m i l a r i t y

v a r i a b l e f r o m their o w n m e a n . F o r i n s t a n c e , i f t h e r e are j u s t three c a s e s , w i t h s c o r e s o f

b e t w e e n the residuals around the least-squares line and the variance of the dependent

- 1 , 4, a n d 5 for a v a r i a b l e , their m e a n is (- 1 + 4 + 5)/3 = 8/3, a n d their v a r i a n c e is

variable. T h e least-squares line is a line p a s s e d through the scattergram in any direction s u c h that t h e s q u a r e d d e v i a t i o n s o f v a l u e s o f t h e d e p e n d e n t v a r i a b l e f r o m that l i n e

'The mean of any variable is its arithmetic mean, or average: the sum of all the values divided by the number of cases.

are m i n i m i z e d . T h e l i n e y = y is a horizontal l i n e p a s s e d t h r o u g h t h e d a t a in t h e s c a t t e r g r a m , a n d i n s p e c t i o n w i l l s u g g e s t w h a t i s , i n fact, m a t h e m a t i c a l l y true: T h i s

126

Chapter 8

Introduction to Statistics

127

that i s , the proportion of the total v a r i a n c e in the dependent v a r i a b l e that c a n be 2

a s c r i b e d to the independent v a r i a b l e . T h e e x p l a i n e d v a r i a n c e , therefore, gives us a

y T T

! ! y=y

T 1 1

!

useful interpretation of r at all v a l u e s . If y o u read that an author h a s f o u n d a c o r r e l a tion o f - 0 . 3 0 b e t w e e n two v a r i a b l e s , y o u s h o u l d mentally square the correlation a n d interpret that statement: " T h e two v a r i a b l e s are negatively related, a n d 9 percent of the v a r i a n c e in one is due to the other." A n o t h e r h e l p f u l w a y to l o o k at t h i s interpretation is to t h i n k in terms of

i

i i

p r e d i c t i o n . O p e r a t i n g w i t h o u t any k n o w l e d g e o f the i n d e p e n d e n t v a r i a b l e , our 1

b e s t strategy i n t r y i n g t o p r e d i c t v a l u e s o f the d e p e n d e n t v a r i a b l e for p a r t i c u l a r

1

i

c a s e s w o u l d b e t o g u e s s that the v a l u e i n a n y g i v e n c a s e i s the m e a n . W e w o u l d b e 3

l e s s w r o n g m o r e o f the t i m e than w i t h a n y other g u e s s w e c o u l d m a k e . I f w e n o w add k n o w l e d g e o f the i n d e p e n d e n t v a r i a b l e , o u r best g u e s s b e c o m e s the v a l u e p r e d i c t e d f r o m the r e g r e s s i o n e q u a t i o n . T h e m a g n i t u d e o f the m i s t a k e s i n e a c h

0

c a s e i s n o w r e p r e s e n t e d b y s q u a r e d d e v i a t i o n s a r o u n d the p r e d i c t i o n s . T h e v a l u e r m e a s u r e s the p r o p o r t i o n b y w h i c h w e h a v e r e d u c e d o u r m i s t a k e s i n p r e d i c t i n g 2

Figure 8 - 1 1

The Variance of y

the d e p e n d e n t v a r i a b l e b y i n t r o d u c i n g k n o w l e d g e o f the i n d e p e n d e n t v a r i a b l e .

line is the one horizontal line that m i n i m i z e s the squared deviations of values of the

Correlation and Regression Compared

dependent variable f r o m itself (see B l a l o c k , 1979, p. 5 8 ) . T h a t i s , any other horizontal line p a s s e d through the data w o u l d y i e l d a greater s u m of squared deviations in the

O u r d i s c u s s i o n s o far l e a v e s u s w i t h the q u e s t i o n , " I s c o r r e l a t i o n o r r e g r e s s i o n

dependent variable.

a n a l y s i s the better w a y to m e a s u r e the strength of a r e l a t i o n s h i p ? " O b v i o u s l y , the

T h i s similarity suggests that the squared deviations around the regression line

a n s w e r m u s t be, " S o m e t i m e s one i s , s o m e t i m e s the other." T h e k e y t o d e c i d i n g

m a y be treated as the variance of the dependent variable around the values predicted

w h e n to u s e e a c h m e a s u r e lies in the fact that the correlation c o e f f i c i e n t reflects the

for it by the regression equation. As already noted, this is the variance in the dependent

v a r i a b i l i t y of the independent variable directly, w h e r e a s the r e g r e s s i o n coefficient

variable that is still left unaccounted for after the effect of the independent variable has

d o e s not.

been estimated.

C o n s i d e r the t w o s c a t t e r g r a m s i n F i g u r e 8 - 1 2 . T h e s c a t t e r g r a m f r o m g r a p h

T h u s we have two v a r i a n c e s for the dependent variable: its v a r i a n c e around its

A h a s b e e n r e p r o d u c e d i n g r a p h B , e x c e p t that a l l o b s e r v a t i o n s for w h i c h the

o w n m e a n (the "total v a r i a n c e " ) and its v a r i a n c e around the r e g r e s s i o n line ("vari-

i n d e p e n d e n t v a r i a b l e is l e s s than 2 or greater than 4 h a v e b e e n e l i m i n a t e d . T h e

a n c e left u n e x p l a i n e d by the independent v a r i a b l e " ) . To the extent that the dependent

effect is to r e d u c e the v a r i a b i l i t y of the i n d e p e n d e n t v a r i a b l e w h i l e l e a v i n g the

variable is determined by the independent variable, this u n e x p l a i n e d v a r i a n c e w i l l be

b a s i c r e l a t i o n s h i p b e t w e e n the i n d e p e n d e n t a n d d e p e n d e n t v a r i a b l e s u n c h a n g e d .

s m a l l c o m p a r e d to the total v a r i a n c e . If the dependent variable c a n be predicted

Under these circumstances, the regression coefficient in B will be approximately the same as that in A, but the correlation coefficient will be sharply lowered in B.

perfectly f r o m the independent variable, as in F i g u r e 8 - 1 0 ( A ) , the u n e x p l a i n e d v a r i a n c e w i l l be zero. If the dependent variable is unrelated to the independent

L e t us see w h y this should be so. T h e regression line that m i n i m i z e d squared

variable, as in F i g u r e 8 - 1 0 ( B ) , the regression line w i l l be h o r i z o n t a l , indicating

deviations in the full set of data in graph A should continue to m i n i m i z e squared

that the s a m e value of the dependent variable is predicted at all values of the i n d e -

deviations in the partial set of data in graph B. T h e r e f o r e , we w o u l d expect the

pendent v a r i a b l e ; i n a s m u c h as the line y = y is the horizontal line that m i n i m i z e s

regression coefficient to be about the s a m e in both graphs. On the other h a n d , because

squared deviations around itself, the regression line w i l l equal the line y = y in this

the variability of the independent variable has been reduced in graph B, the extent of

c a s e . T h u s the u n e x p l a i n e d v a r i a n c e w i l l equal the total v a r i a n c e .

its possible effects on the dependent variable are reduced. R e l a t i v e to other c a u s e s of

D i v i d i n g the u n e x p l a i n e d variance by the total variance tells us w h a t proportion of the total v a r i a n c e is left after we have a l l o w e d the independent variable to explain as m u c h as it c a n e x p l a i n . As it happens, r equals 1 m i n u s this proportion, or 2

^

2

For a good presentation of this interpretation, including the proof that r

unexplained variance total v a r i a n c e

= 1

unexplained variance ; : total variance

see Blalock (1979, pp. 4 0 5 - 4 0 9 ) . 3

A t least this is true if we think of "mistakes" as squared deviations from the true value.

128 Chapter 8

Introduction to Statistics 1 2 9

1 0 0

c o > c o

(1)

Percent Male

0

Figure 8-13

t h e

d e p e n d e n t

v a r i a b l e ,

w h i c h

a r e

j u s t

a s

f r e e

t o

o p e r a t e

a s

t h e y

w e r e

i n

g r a p h

A ,

d e c l i n e .

o f

B u t

t h e

t h i s

i n d e p e n d e n t i s

s i m p l y

t o

v a r i a b l e

s a y

t h a t

a s

t h e

a

c a u s e

p r o p o r t i o n

o f

o f

t h e

t h e

d e p e n d e n t

t o t a l

v a r i a b l e

t

a

b

l

et

ot

h

T h i s

ei

n

d

e

p

e

b e c o m e s

n

a

d

e

n

tv

a

m a t t e r

r

o f

i

a

b

l

ed

e

c

l

i

c o n s i d e r a b l e

n

e

s

.I

no

t

h

i m p o r t a n c e

e

v a r i a n c e

rw

o

i n

r

d

s

f i e l d

,ri

t h a t sr

e

i s

d

u

a

m e a s u r e

c a u s e

o f

t h e

v a r i a b i l i t y

o f

i n d e p e n d e n t

v a r i a b l e s

i s

b e y o n d

t h e

e

d

i n s t a n c e ,

a

r e s e a r c h e r

m i g h t

b e

i n t e r e s t e d

i n

k n o w i n g

r e s e a r c h ,

b e c a u s e

i n v e s t i g a t o r ' s

c o n t r o l .

w h e t h e r

s e x

h a d

m o r e

w h i l e

t h e

t h e

t o a

d o

w i t h

l a r g e

w h e t h e r

c i t y ,

a

p e r s o n

c o r r e l a t i n g

v o t e d .

p e r c e n t

T h e

b l a c k

r e s e a r c h e r

w i t h

p e r c e n t

g e n d e r

o r

p e r c e n t

v o t i n g .

B u t

t h e

d i s t r i b u t i o n

o f

p e o p l e

i n

m o s t

m i g h t

u s e

c e n s u s

v o t i n g

a n d

p e r c e n t

c i t i e s

i s

s u c h

t h a t

w o u l d

v a r y

g r e a t l y

w h i l e

t h e

p e r c e n t

m a l e

w o u l d

n o t .

( B l a c k s

a r e

p r o b l e m

a t

a

i n

s o m e

a l l

t h e

t r a c t s

a n d

a l m o s t

a b s e n t

i n

o t h e r s ;

m e n

a r e

s p r e a d

m o r e

T h e

l o c a t e d

t h e

h o w

h i g h e r )

t h a t

t h a t

p e r c e n t

t h i s

p e r c e n t n e a r v o t i n g

p e r c e n t

o r

l e s s

s h o w

m a l e

r e s e a r c h e r

v o t i n g .

z e r o c a n m a l e

u p

t V

o

t

i

n

g f

r

o

m

P

e

r

c

e

n

t M

a

l

e

f r o m

t h i s

t h a t

r a c e

i s

a

m o r e

i m p o r t a n t

5

b y

t h e

c o e f f i c i e n t s

c a s e .

I n

p a r t i c u l a r

i n

b o t h

c l e a r l y

t h e

v a r i a n c e

i n t e n c ' s

c e n s u s

t r a c t

t o

o f

o f

t h e s e

t h e

c a s e s

i s

i n d e p e n d e n t

t h a t

e x t r a p o l a t e

s t u d y ,

t h e

f r o m

t h e s e

r e s e a r c h e r

v a r i a b l e

p a r t i c u l a r

w a n t s

o f

r a c e

o r

s e x

g e n d e r

o n

v o t i n g ,

r e g a r d l e s s

t o

o f

m a k e

h o w

a

p e o p l e

c i t y . n e w

I n

t h e

G R E

a p p l i c a n t s

s t u d y ,

t h e

b a s i s

( a m o n g

t h e

w h o m

r e s e a r c h e r t h e

w a n t s

t o

v a r i a b i l i t y

o f

m a k e G R E

a

j u d g m e n t

s c o r e s

o f t h e

r e l a t i o n s h i p

b e t w e e n

G R E

s c o r e s

a n d

g r a d e s

w o u l d

a m o n g

i n

t h e

t h e

d e p a r t m e n t .

e v e n l y

s c a r c e l y

w o u l d

f i n d

v a r i e s

f r o m

p r a c t i c a l l y

o n e

n o

c e n s u s

t r a c t

c o r r e l a t i o n

t o

b e t w e e n

g o o d

r u l e

A s

i n d i c a t e d

i n

t h e

h y p o t h e t i c a l

s c a t t e r g r a m

i n

v a r i a n c e

i n

p e r c e n t

m a l e ;

h e n c e

v e r y

l i t t l e

o f

i s

b e

d u e

c o u l d

t o v a r y

i t . a s

O n

t h e

m u c h

o t h e r

a s

h a n d ,

p e r c e n t

i f

r e s i d e n t i a l

b l a c k

d o e s ,

i t

t h a t

b e

w e r e

t h a t

a s

a

m a j o r

d e t e r m i n a n t

o f

v o t e r

t u r n o u t .

4

T h e r e

i s

l i m i t e d

d a t a

t o

i n

a

a n y

m o r e

s i t u a t i o n

g e n e r a l

i n

w h i c h

c a s e ,

y o u

y o u

w i s h

s h o u l d

u s e

t o

e x t r a p o l a t e

r e g r e s s i o n

f r o m

a

c o e f f i c i e n t s .

t h e o r e t i c a l u s u a l l y

r e s e a r c h

w i l l

b e

a l m o s t

s u p e r i o r

a l w a y s

t o

i s

i n t e r e s t e d

c o r r e l a t i o n

i n

a n a l y s i s .

g e n e r a l i z i n g , ( T h i s

a d v i c e

r e g r e s s i o n

h o l d s

d e s p i t e

8 - 1 3 ,

v a r i a n c e

p a t t e r n s

m i g h t

o f

p e r c e n t

F i g u r e

t h e

s e t

a n o t h e r

f a c t ,

w h i c h

I

m e n t i o n e d

e a r l i e r ,

t h a t

i n

r e g r e s s i o n

a n a l y s i s

t h e r e

i s

a l w a y s

s o m e

i n i n

h a n d l i n g

v a r y i n g

u n i t s . )

s u c h a r e

c i r c u m s t a n c e s ,

h o w e v e r ,

i n

w h i c h

y o u

m a y

n o t

i n t e n d

t o

g e n e r a l i z e ,

b u t

g e n d e r o n l y

w o u l d

n

c o r r e l a t i o n

i m p a c t

t h e

c u r r e n t l y

T h e r e t h a t

e

c o n c l u d e

i n v e s t i g a t o r

t r e a t

o n

d i f f i c u l t y p e r c e n t

c

t r a c t s . )

f a c t

i s

r

c o n c e n t r a t e d

t h e t h e r e

e

g e n d e r .

a f f e c t e d

t h e

t h e

i n t o

a n a l y s i s a n d

g P

p e r c e n t

B e c a u s e

m a l e

n

u s

s e x

u s i n g

i s

g e n e r a l

a b o u t

p a r t i c u l a r

g u a r a n t e e s

i

m a l e

A a c r o s s

h a v e

t h a n

w i t h

h a n d ,

m o r e

s t u d e n t s h i g h l y

t

t r a c t

b e b l a c k

c

r a c e

a b o u t w i t h

w o u l d

c o e f f i c i e n t

d a t a

t o

a r e o n

t h a t

p a r t i c i p a t i o n

s t a t e m e n t

d a t a

i

.

d a t a F o r

d

a t t r i b c

i n g e n e r a l l y

e

0

m u s t

T h e u

r

0

t h e t o

i m p o r t a n c e

P

1

t o

d e s c r i b e

a

p a r t i c u l a r

s i t u a t i o n .

F o r

i n s t a n c e ,

s o m e o n e

m i g h t

w a n t

t o

d e s c r i b e

h o w

u s e f u l n e s s a

p a r t i c u l a r

n o t e

t h a t

b e r s ' p

This example also incorporates a major statistical problem in some correlation and regression analyses, the ecological fallacy. This can occur when data on aggregate units (such as percent black, median income, and so on, for census tracts, countries, or states) are used to infer how variables are related among individuals living in those aggregate units. See Robinson (1950), Stokes (1969) and Achen and Shively (1995).

o

r

t

i n

r a c e a

n

t f

p o s s i b i l i t y

C o n g r e s s , t h a t

a n d a

c

t o f

P r o b l e m s

r a

t h e

p a r t i c u l a r

t h e i r o

s a y

s i

n

v o t e s f

l

u

e

c

t h i s

o r

t h e

C o n g r e s s ,

o n n

r e l a t i o n s h i p

s u c h a s

8 0 t h

i

v a r i o u s n

g o

u

t

c

b e t w e e n

9 2 n d ,

t h e r e b i l l s .

o

m

e

s

r a c e

o p e r a t e d . w a s

T h i s

a

w o u l d

v o t i n g

a r e a r e s u l t o f c e n s o r e d d a t a ( s e e

w o u l d

n e g l i g i b l e

i n that Congress. a n d

I t

H

h e l p o

w

e

v

r e c o r d

a b o v e ,

t h e n

b e

a p p r o p r i a t e

c o r r e l a t i o n

e s t a b l i s h

w h a t

e

o

r

, t

f o r

h

i

s w

u

l

C o n g r e s s e s

pp. 103-106)

b e t w e e n w e r e d n

o i n

t d

t h e e

t o

m e m -

n

i m y t

h

e

g e n e r a l .

130

Chapter 8

Introduction to Statistics

Problem of Measurement Error Our understanding of relationships between variables is intimately bound up with the extent to which we have been able to measure the variables validly. To the extent that there is nonrandom error in our variables, we are simply unable to state accurately what the relationship between those variables is. This should be clear from our earlier discussion of measurement error in Chapter 4. Random measurement error also distorts the relationship between variables. Refer back to Table 4-1 (p. 53). The relationship between the two variables is increasingly attenuated from left to right, as it might appear under conditions of increasing levels of random measurement error. The scattergrams for the three sets of data drawn from Table 4-1 are presented in Figure 8-14. As progressively greater amounts of random measurement error are present in two variables, the true form of

Figure 8-14

Relationship under Varying Degrees of Random Measurement Er

No Random Measurement Error

5

10 15 Margin

20

Moderate Random Measurement Error

25

5

10 15 Margin

Great Random Measurement Error 40

30

I

20

10

0

5

J10

L 15 Margin

20

25

20

25

131

the relationship between the variables is more and more lost to us. Not only is it lost to us, but it is systematically distorted. To the extent that there is random error in two variables, they will appear to us to be unrelated. Thus there is a real danger that the only relationships we will be able to perceive are those between variables that are easy to measure accurately, a possibility that bodes ill for social science. If we are willing to make some assumptions regarding the nature of the random error, it is possible to correct for it, and thus reconstruct the true relationship between two variables. A good example of this technique is seen in Bartels (1993). (Unfortunately, this article and most other discussions of measurement error are difficult for the untrained reader.) FURTHER DISCUSSION In this chapter I have drawn only the broad outlines of correlation and regression analysis for interval data. In fact, I have purposely refrained from giving formulas for calculating these measures, so that anyone wishing to put these techniques to use would be forced to seek out more detailed training. David Knoke, George W. Bohrnstedt, and Alissa Potter Mee's Statistics for Social Data Analysis (2002) is a good, standard text in statistics—one of many—for social scientists. One question you might consider is this: What would happen in a regression analysis if the independent variable did not vary at all—if, say, we wanted to relate voting turnout to education, but everyone in our study had the same amount of education? This question looks ahead to Chapter 10.

Chapter 9

Introduction to Statistics T A B L E 9-1

133

Relationship Between Support for Military Intervention and Support for E c o n o m i c Aid Willingness to Have United States G i v e Financial Aid to Countries in Crisis

Willingness to Have United States U s e Military Force to Solve Problems

F u r t h e r Topics on Measurement of Relationships

Not

Willing

Somewhat Willing

Very Willing

Extremely Willing

Extremely w i l l i n g

18

66

33

Very w i l l i n g

47

152

75

14

120

355

112

34

48

119

38

5

Somewhat willing Not willing

29

Source: 1998 Congressional Election Survey, National Election Study, University of Michigan.

In tins chapter I build on the introduction in Chapter 8 to discuss in brief how to measure three types of relationships: (1) those among ordinal and nominal variables(2) those that involve dichotomous variables; and (3) those that involve several variables simultaneously. 1

MEASURES OF RELATIONSHIP FOR ORDINAL DATA As we saw at the beginning of Chapter 8, there are two main ways in which we can measure the strength of a relationship. We can measure how much change is produced in the dependent variable by a change in the independent variable, or we can measure how strongly the dependent variable is determined by the independent variable relative to other things that also help determine it. Only for the second of these is it possible to develop a standard measure for use with ordinal-scale data. To measure how much change is produced in the dependent variable, it is necessary to have some sort of unit by which to measure that change. But this is not available under ordinal measurement. Accordingly, the most we can hope for from ordinal variables is some form of correlational analysis. A cross-tabulation is to the analysis of ordinal variables what a scattergram is to the analysis of interval variables. Consider Table 9-1, in which voters' willingness to have the United States use military force to solve problems abroad is related to their willingness to have the United States give financial aid to countries 'This chapter presents advanced material that can be skipped over without any loss in comprehension of the remaining material covered in this book.

132

in economic crisis. This is an interesting question. There might be a negative relationship, in which voters who favored the use of military force did not want to see more "soft" involvement, and voters who favored helping those in economic crises did not want to see other sorts of involvement. Or there might be a positive relationship if both questions were tapping an "internationalist" dimension such that those favoring one sort of involvement would favor other sorts as well, and those opposing U.S. involvement abroad in one form opposed it in all other ways as well. Like a scattergram, the table shows how frequently the various combinations of the two variables occur. And just as in a scattergram that shows the relationship between two intervally measured variables, a positive relationship would mean that relatively few cases should fall in the upper left corner of the table (not many cases willing to intervene militarily but unwilling to send aid) or the lower right corner (not many cases willing to send aid but unwilling to intervene militarily). A negative relationship would imply the opposite: relatively few cases in the lower left or upper right. And no relationship at all would, of course, imply an even spread across the table. Each number in the table gives the number of people in the 1998 study who exhibited a given combination of attitudes toward the two sorts of international intervention. These are actual numbers of cases, not percentages. The table reveals a pattern between these variables, just as a scattergram does for interval-scale data. In general, we can say that there is a modest positive relationship in the table, with willingness to use military force increasing somewhat as voters are more willing to give financial aid. For instance, of the 82 voters who are extremely willing to give financial aid, 29 are extremely willing to have the United States use military force, while of the 233 who are not willing to give aid, only 18 are extremely willing to have

134

Chapter 9

Introduction to Statistics

the United States use military force. There are t w o differences b e t w e e n a table such as

T A B L E 9-2

135

Independent Version of Table 9-1

this a n d a scattergram, however: Willingness to Have United States G i v e Financial Aid to Countries in Crisis 1. Because there is a limited number of values available for each variable (four for the independent variable and four for the dependent variable), it is not necessary to use the infinite space of a plane to locate the observations. Instead, the observations are located in one or another of the cells of a table. Because so many observations are tied on each value, their presence is indicated by counting the number of observations falling in each cell and printing the number there rather than by printing a dot for each observation. If the observations are indicated by dots, as in Figure 9-1, the similarity to a scattergram becomes more obvious.

Willingness to Have United States Use Military Force to Solve Problems

Not Willing

Extremely willing Very willing

2. Because there are no units by which to measure the difference between two values of a variable, however, the precise pattern in the table is meaningless. How great a change there is in attitudes as we move from "Very willing" to "Extremely willing" tells us

Somewhat willing Not willing

Somewhat Willing

Extremely Willing

Very Willing

27 53 114

80 157 340

30 59 127

39

115

42

9 19 40 14

nothing specific, because for all we know, "Very willing" might represent only slightly less support than "Extremely willing."

T h u s it is not possible to m e a s u r e the strength of relationships in a w a y analogous

O n e s u c h m e a s u r e for u s e w i t h ordinal variables i s G o o d m a n a n d K r u s k a l ' s 3

to regression. On the other hand, it is possible to develop measures analogous to the

gamma.

correlation coefficient. T o d o this, w e m u s t d e v e l o p m o d e l s o f t h e perfect relationship

distributed u n i f o r m l y t h r o u g h o u t t h e t a b l e ; that is, t h e s a m e p e r c e n t o f e a c h cate-

a n d of perfect i n d e p e n d e n c e , as w a s d o n e w i t h the correlation coefficient, a n d also

gory of the dependent variable occurs at each category of the independent variable,

develop a measure to indicate h o w close the data in our table c o m e to these two m o d e l s .

2

I t t a k e s a s its m o d e l o f p e r f e c t i n d e p e n d e n c e a t a b l e i n w h i c h t h e d a t a are

and vice versa. U n d e r such "perfect i n d e p e n d e n c e , " the data in Table 9-1 w o u l d appear as in Table 9-2. T h e n u m b e r s in Table 9-2 are printed in italics to e m p h a s i z e that they are b a s e - l i n e c o u n t e r p a r t s for c o m p a r i s o n w i t h the o b s e r v e d n u m b e r s in Table 9-1.

Figure 9-1

Scattergram Version of Table 9-1

* •*•****• * * • • • • • * * * • * \ * • !•*•*; •**••*••>* *••*****•• «• * !!•••„••** • • • • # • •*•*# . •.. .: • •' •• •• •. ..• ••"»••• *••*•• • • * • * • • . • • . . • •••••• . •. • • • • ••. : • • •;••• *•*»*»•••• • . • '•*•..••*•*••*"•.*. .. . .• . . .*. • ;•* • ••• : « £ S 3 K .**.•••*••• • •..-•* • %: •:•..*...*%•:•*. • • ••*".*•. • ' ••• •.*• ••••». *lw.'S3* * * . • «• • .vv.»:^v;.;:; * * • . *•.:.:'.:*:

111

Measures

like

the

Goodman-Kruskal

gamma

are

designed

to

compare

the

distributions in Tables 9-1 a n d 9-2 to see h o w different they are in a positive or negative direction. For instance, there are fewer cases in the upper-left corner (8 vs. 27) and l o w e r - r i g h t c o r n e r ( 5 v s . 14) t h a n o n e w o u l d e x p e c t i f t h e r e w e r e n o r e l a t i o n s h i p ; a n d t h e r e a r e m o r e i n t h e l o w e r left ( 4 8 v s . 39) a n d u p p e r r i g h t ( 2 9 v s . 9 ) t h a n e x p e c t e d . T h i s indicates

a pattern consistent with a mildly positive relationship.

A m e a s u r e like

G o o d m a n - K r u s k a l ' s w i l l u s e all c e l l s o f t h e t a b l e , n o t j u s t t h e c o r n e r s , b u t t h i s i s t h e general principle of h o w they work. F r o m a m e a s u r e b a s e d on cell-by-cell c o m p a r i s o n s like this, we can develop a m e a s u r e o f c o r r e l a t i o n . B u t n o n e o f t h e m e a s u r e s that h a v e b e e n d e v e l o p e d i s all that satisfactory, and often t w o rather different patterns will p r o d u c e the s a m e result. T h e result is useful, but a bit a m b i g u o u s . T h i s is simply the price we p a y for the fact that ordinal m e a s u r e s are naturally less informative than interval m e a s u r e s .

(See also

footnote 4 on p a g e 138, which notes a w a y to incorporate ordinal measures as independent variables in combination with interval variables.)

Even though there is not a precise regression line to define residuals, however, it might still be useful to look at the "residuals" in Table 9-1—the cases that do not fit the generally positive relationship. Is there anything special, for instance, about those who buck the general trend by favoring the use of military force but opposing financial aid, or vice versa? 3

This measure was described in Goodman and Kruskal (1954).

136

Chapter 9

Introduction to Statistics

MEASURES OF RELATIONSHIP FOR NOMINAL DATA In relating two nominally measured variables, cross-tabulation again approximates the function of a scattergram—that is, it shows which values of one variable tend to coincide with which values of the other. However, in contrast to a table relating ordinal variables, the position of a cell in the table no longer tells us anything, for the order of values of a nominal variable means nothing. Consider the example in Table 9-3, in which versions A and B are equivalent. The table could not be rearranged in this way if the variables were measured ordinally. As with ordinally measured data, all we can do in measuring the strength of a relationship between two nominally measured variables is to assess the degree to which one variable is determined by the other. Various measures have been developed to do this, just as has been done with ordinal variables. But the problem is really better dealt with by adaptations of regression analysis to nominal variable analysis. Regression adaptations for analyzing nominal variables are dealt with in the next two sections. Dichotomies and Regression Analysis Dichotomies are classifications that involve only two categories: for example, gender (male or female), referendum vote (yes or no), or participation (voter or nonvoter). Such classifications have a useful property, which at first glance appears to be a bit mysterious: Although dichotomies are nominal scales, they can quite properly be treated as interval scales and can be analyzed using measures appropriate for such scales. This greatly expands the value and applicability of regression analysis, for obvious reasons. It means that not only intervally measured variables, but also dichotomous nominal scale variables—and mixtures of the two—can be analyzed using regression analysis. The trick is that because a dichotomous variable can take on only one of two values, there is by definition a common unit in which differences between values of the variable can be measured, namely the difference between the two categories. Let

T A B L E 9-3

Interchangeable O r d e r in Nominal Data (A) Religion

Candidate Preferred

(B) Religion Candidate

Cath.

Prot.

X

20

0

z

5

30

Y

0

0

Other

Preferred

Prot.

Cath.

Other

0

X

0

20

0

4

Y

0

0

26

Z

30

5

4

26

137

us call the categories A and B. Then the distance between two observations falling into category A is zero units, and the distance between two observations, one of which is in A and the other in B, is one unit. Because there usually is no natural ordering to the dichotomy, the decision of whether A is one unit greater than B, or one unit less, must be arbitrary. An alternative way to look at this situation is to think of a dichotomy as measuring the degree to which an observation embodies one of its categories. The category is either present or not present for each observation. For instance, the dichotomy "gender: male or female" can also be viewed as a variable "femaleness" (or alternatively, "maleness"). Each subject is either totally "female" or totally "not female." If the difference between totally female and totally not female is taken to be one unit, women have values of +1.0 for the variable "femaleness" and men have values of 0.0. Because this is a true interval measure, it can be used together with other interval measures, such as income or age, in regression and correlation analysis. (The same interpretation cannot be placed on nominal-scale variables with more than two categories. We would have no common unit, for example, in which to place Catholics relative to Protestants or Jews in the variable "religion." As you will see in footnote 4, however, there is yet another trick we can use in this case that will allow us to work with multiple-category variables.) For example, we might regress income at a low-paying factory on gender, with gender equal to 0 for men and 1 for women. The best-fitting regression line might be income = 14,000 - 2,000 gender This would mean that the expected income of men at the factory was $14,000, since the value of "gender" for men is zero. Similarly, the expected income for women would be $14,000 - ($2,000 x 1) = $12,000. Since the independent variable can take on only two values, the scattergram for this analysis will look odd, with a set of data arrayed vertically above "gender = 0" and another set arrayed vertically above "gender = 1." A scattergram that would be consistent with these hypothetical regression results is shown in Figure 9-2. The regression line will actually pass directly through the mean income of men at gender = 0 and the mean income of women at gender = 1. Since a regression line serves as a convenient way of summarizing the central pattern in a set of means, there is no particular advantage to using regression analysis in cases like these, in which we are interested in the relationship between an interval-scaled variable and a single dichotomous variable. (Exactly the same result could be presented, more simply, by noting that the average income of men at the factory is $14,000, while the average income of women is $12,000.) Where this technique of treating dichotomous variables does become valuable is when we wish to combine a dichotomous variable with other independent variables in an analysis. This would be the case if we wished to look at the simultaneous effects of sex and education on income. There is a well-developed technique described in the section "Multivariate Analysis"

138

Chapter 9

Introduction to Statistics

139

$18,000

16,000

14,000

0 £ 8 c

12,000 10,000

8,000

-

6,000 Gender

Figure 9-2

LOGIT AND PROBIT ANALYSIS

Hypothetical Regression of Income on Gender

on pp. 141-146 that allows us to use interval-level measures simultaneously in this way, but there is no such technique for nominal measures. The ability to include dichotomous nominal variables in such analyses is extraordinarily valuable. Such variables generally are known as dummy variables or binary variables. The value of this trick, whereby we can include nominal scale dichotomies in the powerful technique of regression analysis, cannot be stated too strongly. However, this works only for dichotomous nominal measures used as independent variables. We cannot appropriately use them as dependent variables in ordinary regression. One indication that this would be problematic is that it would be impossible to construct a meaningful correlation coefficient for such an analysis. Consider the scattergram in Figure 9-3. In any reasonable sense, there is a perfect relationship here. Above levels of x = 10, y is always 1; and when x is less than 10, y is always 0. But there is still "error" around the regression line. Since the regression line cannot bend, it cannot follow the jump from y - 0 to y = 1 at the point at which x equals 10 and thus cannot register the full extent to which values of y are determined by x. It is therefore incapable of serving as the "best summarizing line" to describe the relationship between x and y. 4

Techniques have also been developed for including ordinal variables in regression, but presentation of these techniques goes beyond the scope of this book. The techniques involve "ordered logit" and "ordered probit." They are variants of the logit and probit analysis presented in the next section.

To handle the problem of describing the relationship between a continuous independent variable and a dichotomous dependent variable, statisticians have developed an alternative model that is found in two very similar variations, called logit analysis and probit analysis. I describe logit analysis here, but the general explanation holds for both. The regression line, as you will recall, can be interpreted as the path of expected values of the dependent variable across varying values of the independent variable. From the regression line in Figure 8-4, for instance, we would predict that y would equal 9 when x equaled 1, that y would equal 18 when x equaled 4, and so on. This does not make sense for a dichotomous variable, since it can take on values of only 0 or 1. It would not make sense in Figure 9-3, for instance, to predict that the dependent variable would have a value of 0.3 when the independent variable equaled 8; this would be nonsensical. An analogy to the expected value of a dependent variable, however, would be the probability that the dependent variable equals 1. Such a model would portray, across the variation in an independent variable, the varying probability that a dichotomous variable equals 1. It would need to be S-shaped and bounded vertically by 0 and 1, since a probability must be positive and less than or equal to 1. Figure 9-4 provides an example. The model would predict, for example, that at x = 5, about 70 percent of the cases would have y = 1. That is, P(y= 1), the probability that y equals 1, is about 0.7. We do not have any way to work directly with such a model, but the logit model allows us to derive estimates of the sort such a model would yield. Letting p = P (y = 1), the logit is the logarithm to base e, that is the natural logarithm, of

140

Introduction to Statistics

Chapter 9

Age 1.0

.6

141

Estimated Probability of Voting

25

0.37

35

0.58

45

0.68

55

0.70

65

0.71

75

0.71

l o g i t ( t h e n a t u r a l l o g a r i t h m o f p / ( l - p)) i s a t e a c h o f s e v e r a l v a l u e s o f x . T h e n , i t i s s t r a i g h t f o r w a r d t o s o l v e f o r p ( t h e p r o b a b i l i t y t h a t t h e d e p e n d e n t v a r i a b l e i s 1 rather t h a n 0 ) , f r o m t h e n a t u r a l l o g a r i t h m o f p / ( l -p). F o r i n s t a n c e , i f w e w e r e i n t e r e s t e d in the effect of age on voting participation, we m i g h t perform a logit analysis and then present the results:

.2

A table like this approximates the results o n e m i g h t have gotten f r o m a nonlin-

-

e a r r e g r e s s i o n l i n e . T h e r e are a l s o v a r i o u s a n a l o g s t o t h e c o r r e l a t i o n c o e f f i c i e n t f o r logit a n a l y s e s but n o n e that is readily p r e s e n t e d at this l e v e l . T h e ability to incorporate d i c h o t o m o u s variables into regression analysis, either d i r e c t l y i f t h e y are i n d e p e n d e n t v a r i a b l e s , o r t h r o u g h t h e l o g i t a n d p r o b i t v a r i a n t s o f regression analysis if the d i c h o t o m y is a d e p e n d e n t variable, has m a d e regression analysis a p o w e r f u l statistical tool and the d o m i n a n t tool for data a n a l y s i s in political science. Figure 9-4

Path-of-Probabilities M o d e l MULTIVARIATE ANALYSIS

p / ( l — p). T h i s i s a d i f f i c u l t q u a n t i t y t o c o n c e i v e ! B u t i t d o e s e m b o d y t h e p r o b a bility i n w h i c h w e are interested; and m o s t important, the l o g i t m o d e l a l l o w s u s t o use regular regression

analysis

to analyze a d i c h o t o m o u s

dependent variable.

It turns o u t that the l o g i t c a n appropriately be taken as a linear f u n c t i o n of a c o n tinuous independent variable, in the standard form

I n C h a p t e r 6 w e s a w that f r e q u e n t l y i t i s i m p o r t a n t t o h o l d o n e v a r i a b l e c o n s t a n t w h i l e m e a s u r i n g the r e l a t i o n s h i p b e t w e e n t w o other variables. T h e e a s i e s t w a y t o d o this i s literally t o " h o l d t h e variable constant," a s i n t h e e x a m p l e o n p p . 9 4 - 9 5 , b y separating the subjects into g r o u p s a l o n g t h e control variable ( a g r o u p o f D e m o c r a t s , a g r o u p o f R e p u b l i c a n s , a n d a g r o u p o f i n d e p e n d e n t s , for e x a m p l e , i f registration i s t h e variable t o b e

logit of y

bx

M e c h a n i c a l l y , t h e w a y t h i s w o r k s i s that t h e d e p e n d e n t v a r i a b l e h a s v a l u e s o f e i t h e r 0 or 1. T h e c o m p u t e r calculates f r o m the n u m b e r of 0s and 1 s at g i v e n values of the i n d e p e n d e n t variable w h a t the e x p e c t e d logit of the d e p e n d e n t variable is across the range of the independent variable. B e c a u s e the analysis is so indirect, the a and b in t h e l o g i t e q u a t i o n are n o t i n t u i t i v e l y i n t e r p r e t a b l e . R e m e m b e r , i t i s n o t y , o r e v e n t h e p r o b a b i l i t y o f y , that i s p r e d i c t e d f r o m x i n t h i s e q u a t i o n ; rather, i t i s t h e n a t u r a l l o g a r i t h m o f p / ( l - p). T h e b e s t w a y t o i n t e r p r e t l o g i t a n a l y s i s , a n d t h e w a y i t i s u s u a l ly presented in studies, is to calculate by a series of transformations on the logit coefficients what the e x p e c t e d probability of y is at various values of x and to present t h o s e d i s c r e t e l y . T h i s i s d o n e b y first c a l c u l a t i n g f r o m a a n d b w h a t t h e e x p e c t e d

h e l d c o n s t a n t ) . T h e r e l a t i o n s h i p b e t w e e n t h e o t h e r t w o variables i s t h e n m e a s u r e d for e a c h group. T h i s t e c h n i q u e h a s s e r i o u s d i s a d v a n t a g e s , t h o u g h . First, i f w e w a n t t o control for an intervally m e a s u r e d variable, the n u m b e r of "groups" we set up m i g h t be too l a r g e . F o r i n s t a n c e , i n c o n t r o l l i n g f o r i n c o m e , w e m i g h t h a v e a g r o u p o f all t h o s e w i t h a n i n c o m e o f $ 1 8 , 2 4 3 , a n o t h e r o f all t h o s e w i t h a n i n c o m e o f $ 4 3 , 4 0 2 , a n d s o o n ; in this c a s e there probably w o u l d be as m a n y groups as there w e r e subjects in the study. T h i s p r o b l e m c a n be averted by l u m p i n g subjects into a f e w broad categories of the independent variable, but d o i n g so throws a w a y a great deal of the precision in the m e a s u r e — s o m e t h i n g o n e m a y not w a n t to do. A s e c o n d a n d m o r e s e r i o u s p r o b l e m i n literally h o l d i n g a v a r i a b l e c o n s t a n t i s that the process quite rapidly leaves the researcher w i t h o n l y small numbers of subjects

142

Chapter 9

Introduction to Statistics

143

among w h o m to measure a relationship. Suppose that someone wanted to examine the relationship between occupation and voting, holding constant race and age. T h e investigator might divide the population into groups s u c h as blacks aged 2 0 - 3 0 , whites aged 2 0 - 3 0 , b l a c k s aged 31—40, and so on, and then look at the occupation/vote relationship for e a c h of these groups. B u t the typical national survey, with about 2,000 respondents, might include no more than five or six whites aged 2 0 - 3 0 , blacks aged 5 1 - 6 0 , or whatever. S u c h s m a l l groups w o u l d give very unreliable estimates of the relationship between occupation and vote.

5

A final disadvantage of literal controlling is that it produces a series of measures of the relationship (one measure for e a c h group) that is u n w i e l d y and difficult to absorb. Particularly, if we wanted to control simultaneously for m o r e than one variable, as in our example of race and age, this might leave us with as m a n y as 20 or 30 separate measures to consider. E s p e c i a l l y , as noted above, any multicategory n o m i n a l variables included in the a n a l y s i s w i l l show up as a set of several dichotomous variables. F o r t u n a t e l y , there is an e a s i e r w a y to cont rol for a v a r i a b l e — a t least if we are u s i n g i n t e r v a l - s c a l e v a r i a b l e s ( i n c l u d i n g d i c h o t o m i e s that are taken as i n d e p e n d e n t Figure 9-5

v a r i a b l e s ) . T h e t e c h n i q u e of multivariate r e g r e s s i o n a l l o w s us to l o o k at the pattern

Three-Dimensional Scattergram

i n s e v e r a l v a r i a b l e s a m o n g a l l o f our o b s e r v a t i o n s ( w i t h o u t b r e a k i n g these d o w n into

separate

subgroups)

and to estimate

w h a t the r e l a t i o n s h i p b e t w e e n the

dependent v a r i a b l e a n d any p a r t i c u l a r i n d e p e n d e n t v a r i a b l e w o u l d b e i f n o n e o f

T h i s regression plane w o u l d look l i k e a flat p i e c e of c a r d b o a r d set at s o m e tilt

the other v a r i a b l e s v a r i e d — t h a t i s , i f e a c h o f the other v a r i a b l e s w e r e " h e l d

in the m i d d l e of the t h r e e - d i m e n s i o n a l s p a c e over a g r i d c o n t a i n i n g all p o s s i b l e

constant."

c o m b i n a t i o n s of v a l u e s of w and x. By c o u n t i n g out a certain distance along x and

To s k e t c h the t e c h n i q u e in a general w a y , I w i l l u s e as an e x a m p l e the c a s e of

then a certain distance along w, we c a n locate a particular c o m b i n a t i o n of w and x on

t w o i n d e p e n d e n t v a r i a b l e s (w and x) and a dependent v a r i a b l e ( y ) . Y o u w i l l r e c a l l

this grid. T h e height of the c a r d b o a r d plane above the grid at this point is the value of

that the r e l a t i o n s h i p b e t w e e n two v a r i a b l e s c a n be plotted in a flat s p a c e of two

y predicted f r o m the g i v e n v a l u e s of w a n d x by the regression p l a n e . T h e r e g r e s s i o n

d i m e n s i o n s ( r e p r e s e n t e d by a s c a t t e r g r a m ) . S i m i l a r l y , the r e l a t i o n s h i p a m o n g three

equation a s s o c i a t e d w i t h the plane is of the f o r m

v a r i a b l e s c a n b e plotted i n a t h r e e - d i m e n s i o n a l s p a c e , a s seen i n F i g u r e 9 - 5 . T h e v e r t i c a l d i m e n s i o n i s the dependent v a r i a b l e , a n d e a c h o f the h o r i z o n t a l d i m e n -

y

= a + b\w + b x 2

s i o n s is one of the i n d e p e n d e n t v a r i a b l e s . E a c h o b s e r v a t i o n is a dot floating in t h r e e - d i m e n s i o n a l s p a c e , l o c a t e d a c c o r d i n g to its v a l u e s on e a c h of the three

As in the t w o - v a r i a b l e c a s e , a is the i n t e r c e p t — t h e predicted v a l u e of y w h e n both x

v a r i a b l e s . D o t A h a s h a d c o o r d i n a t e s d r a w n to s h o w h o w it is l o c a t e d in the three-

and w equal zero. S i m i l a r l y , b tells by h o w m a n y units y c a n be e x p e c t e d to i n c r e a s e

d i m e n s i o n a l s p a c e by its v a l u e s on w, x, a n d y, w h e r e these v a l u e s are p, q, and r,

if w i n c r e a s e s by one unit and x does not change; a n d b tells by h o w m a n y units y

respectively.

c a n be expected to i n c r e a s e if x i n c r e a s e s by one unit and w does not change. H e r e fe,

J u s t as a o n e - d i m e n s i o n a l l i n e t h r o u g h the t w o - d i m e n s i o n a l s c a t t e r g r a m c o u l d s u m m a r i z e the pattern in a t w o - v a r i a b l e r e l a t i o n s h i p , so a t w o - d i m e n s i o n a l p l a n e through the t h r e e - d i m e n s i o n a l scattergram c a n s u m m a r i z e the pattern in its

x

2

and b are the r e g r e s s i o n coefficients of the equation. 2

To c a l c u l a t e f r o m the multivariate r e g r e s s i o n the y v a l u e y o u w o u l d expect f r o m a particular c o m b i n a t i o n of w and x:

t h r e e - v a r i a b l e r e l a t i o n s h i p . T h e plane i s p i c k e d o n the b a s i s o f the s a m e c r i t e r i o n b y w h i c h the r e g r e s s i o n line w a s c h o s e n . I t m u s t b e the p l a n e that m i n i m i z e s

1. Start with a, the expected value of y when both x and w equal zero.

the s u m of s q u a r e d d e v i a t i o n s of a c t u a l y v a l u e s f r o m the y v a l u e s p r e d i c t e d

2. Add to this b\ times w, or the amount by which y could be expected to change if

f r o m the p l a n e .

the value of w shifted from zero to the particular value you are using. At this point, you have the expected value of y when x equals zero and w equals the particular value. 3. To this sum, add b times x, the amount by which y could be expected to change if x 2

shifted from zero to the particular value you are using and w remained unchanged. You 'Refer again to the box "Law of Large Numbers" on p. 6 1 .

now have the expected value of y, given these particular values of w and x.

1 4 4 Chapter 9

Introduction to Statistics 1 4 5

N o t i c e in this that b d e s c r i b e s the relationship b e t w e e n w and y if x does not {

vary, and that b d e s c r i b e s the relationship b e t w e e n x and y if w does not vary. In this 2

w a y , without requiring us to break our observations into separate s m a l l and a w k w a r d groups, multivariate r e g r e s s i o n a l l o w s us to e x a m i n e relationships w i t h certain other v a r i a b l e s controlled. F o r e x a m p l e , i f w e had data for U . S . counties g i v i n g the percentages o f A f r i c a n A m e r i c a n students attending a l l - b l a c k s c h o o l s , the m e d i a n i n c o m e o f e a c h county, and the percentage o f A f r i c a n A m e r i c a n s i n e a c h c o u n t y ' s population, w e might find the f o l l o w i n g situation: L e t S be the percentage of A f r i c a n A m e r i c a n students attending a l l - b l a c k

Interaction in Research and Theory Although interaction among variables is a nuisance from the standpoint of multivariate regression analysis, in itself it is an interesting phenomenon. It is exciting to think that a relationship itself may depend on a further variable. Because such interaction occurs frequently in the social sciences, it is important that you be alert to it in your research. Because "common sense" rarely gets so complicated as to suggest this sort of thing, interaction is almost always unexpected, and discovering it is fun and dramatic. As one example, Philip Converse was able to argue very neatly for the importance of a measure of ideological sophistication by showing that the relationship between social class and voting was strongly affected by it (1964, esp. pp. 2 3 1 - 2 3 4 ) . Among those who scored lowest in ideological sophistication, there was no relationship between social class and voting. As the score on ideological sophistication increased, however, a relationship appeared, reaching its apex among those with the highest sophistication score. 8

schools. L e t M be the c o u n t y ' s m e d i a n i n c o m e (in thousands of d o l l a r s ) . L e t A be the percentage of A f r i c a n - A m e r i c a n s in the c o u n t y ' s population. On e x a m i n i n g j u s t the relationship b e t w e e n S and M, we m i g h t find that the r e g r e s s i o n equation w a s

S =

68.2 - 2.1 M

technique for use with ordinal data, we c a n add this to our list of arguments for trying to measure variables intervally as often as possible. N o m i n a l variables m a y be included in

T h i s w o u l d indicate that s c h o o l integration i s m o r e w i d e s p r e a d i n r i c h e r counties. T h e percentage o f A f r i c a n A m e r i c a n s attending a l l - b l a c k s c h o o l s d e c r e a s e s b y 2.1 percent as the m e d i a n i n c o m e of the county i n c r e a s e s by 1,000 d o l l a r s .

multiple regression equations by treating them as dichotomies (refer to p. 136). As in simple linear regression, it is important in multiple linear regression that the data actually fit a m o d e l in w h i c h all the relationships between the dependent variable

O b v i o u s l y , however, we need to control for percent A f r i c a n A m e r i c a n in

and the independent variables are linear. O t h e r w i s e , the regression plane described by

a s s e s s i n g this relationship. It might be that the apparent integration of the w e l l - h e e l e d

the equation we c o m e up with w i l l be an inaccurate representation of the relationships.

counties is s i m p l y a result of the fact that in the richer counties there are so few

There is also an additional requirement in multiple linear regression. It is important

A f r i c a n A m e r i c a n s that there are not l i k e l y to be any a l l - b l a c k s c h o o l s . We c o u l d test

that the data best fit a model in w h i c h the relationship between the dependent variable

for this

and each independent variable is the same no matter what value the other independent

by

adding

the

variable

"percent A f r i c a n

American"

as

an

additional

independent variable. T h e regression equation

variable(s) takes on. In our three-variable example of w, x, and y, we had to assume that an increase of one unit in x produced the same change in y, no matter what the value of w

S = 49.5 - 0 . 1 M + 0 . 6 / 1

w a s . Otherwise, we could not have calculated the expected value of y simply by adding a plus b\ times w plus b times x. We w o u l d have had to pick a value for b\ that w a s 2

w o u l d indicate that this is true. W i t h the cont rol for "percent A f r i c a n A m e r i c a n "

appropriate for whatever value of x we were using, and a value of b that w a s appropriate 2

added, the relationship b e t w e e n m e d i a n i n c o m e and percentage o f A f r i c a n A m e r i c a n students attending a l l - b l a c k s c h o o l s largely disappears. In the m u l t i p l e regression equation, a d e c r e a s e of only 0.1 percent c a n be expected f r o m an i n c r e a s e of 1,000 dollars i n m e d i a n i n c o m e .

for whatever value of w we were using. It is obvious that this w o u l d be a complicated procedure. In fact, it is often impossible to operate under these conditions. W h e n this h a p p e n s — w h e n the relationship b e t w e e n v a r i a b l e s A and B differs,

6

depending on the v a l u e of a variable C—there is s a i d to be " i n t e r a c t i o n " a m o n g the

T h i s freedom f r o m the need to "hold variables constant" physically is an important advantage of multivariate regression analysis. T h i s is especially true w h e n working

variables. A c o n c r e t e e x a m p l e m a y m a k e m o r e c l e a r what interaction is and h o w it might arise. C o n s i d e r the relationship b e t w e e n party affiliation and h o w m e m b e r s of

7

with several variables simultaneously. B e c a u s e there is no satisfactory analog to this

C o n g r e s s vote on a b i l l . T h i s relationship c o u l d be strong (all D e m o c r a t s vote one w a y , all R e p u b l i c a n s the other), or w e a k (less c l e a r - c u t party d i f f e r e n c e s ) . N o w , the

6

strength of the relationship might be related to a m e m b e r ' s seniority. N e w m e m b e r s ,

This little analysis is wholly fictitious. For a multiple regression analysis of black voting registration, using county data for southern states in the 1950s, see Matthews and Prothro (1963).

w h o are rather dependent on party l e a d e r s h i p for f a v o r s , might vote obediently along

'Although I have used the case of three variables in this section because that case can be represented by perspective drawings, the general multivariate regression technique applies to any number of variables. The same logic would apply to a four-variable case, with the equation y = a + b w + b x + b z, although no scattergram can be drawn in these four dimensions.

This example is a particularly interesting one because Converse demonstrated yet a second level of interaction (the interaction itself varies with gender) that I have chosen to ignore here.

{

2

}

8

146

Introduction to Statistics

Chapter 9 T A B L E 9-4

147

Example of Results from a Multivariate Logit Analysis Gender

Age

Male

25

0.52* 0.48

Female 0.62

0.46

0.58 0.55

55 65

0.45

0.53

0.44

0.51

75

0.44

0.50

35 45

'Estimated probability of voting Democratic, for various combinations of age and gender.

straight party lines. Those with more seniority, who had built up respect among fellow members, might vote more independently. Thus, there would be interaction among the variables. The relationship between party and vote would change as the third variable, seniority, changed. Under these circumstances, a regression equation of the form vote = a + b\ party + b seniority 2

Figure 9-6

would not be appropriate. It would not be true that "vote" always increased by the same amount with a given change in "party." How much "vote" changed with a given change in "party" would depend on the value of "seniority." Like ordinary regression analysis, logit and probit analyses may readily incorporate a number of independent variables. The coefficients for the variables are difficult to interpret, since they are predicting not directly to the variable or its probability but (in the case of the logit) to the natural logarithm of pi (l-p) where p is the probability that y = 1. As in logit analysis with a single independent variable, the results are usually made understandable by calculating from transformations of the coefficients what the probability of y is estimated to be, given varying combinations of the independent variables. For instance, if we were predicting voting for the Democratic Party from age and gender, we might present the results as in Table 9-4.

Four Alternative Scattergrams with the Same Regression Equations

9

CONCLUSION You might expect me to conclude, "Go out and use these measures." However, I am more interested in cautioning you against using them unwisely. All of these techniques simplify what is going on in a set of data by screening out certain aspects 9

It is possible to add "interaction terms" to the regression equation to take into account certain types of interaction, but that goes beyond the scope of this presentation. See, for example, Blalock (1969, Appendix A, "Theory Building and the Statistical Concept of Interaction").

of relationships. This is their great virtue, and it is a virtue. Still, it is your responsibility to delve a bit deeper and see just what is being screened out. For instance, each of the scattergrams in Figure 9-6 would give approximately the same regression equation, but the relationships depicted mean quite different things. It is critical that you examine as carefully as possible each scattergram on which you base a regression equation, checking for nonlinearity, noting which observations are mavericks in the relationship, and so on. A computer can chug out hundreds of regression equations in a few seconds. It is tempting simply to call for a mound of results, especially given the uncertainty involved in working with most social science theories. Usually, however, you will be better served if you first think carefully about what you want to do, pick a limited number of relationships of interest to you, and examine each of these relationships in detail. Your measures will then be useful summarizing tools, not substitutes for a full look at what is going on. The important thing is that you stay in charge. Use your measures and computer analyses as tools rather than ends in themselves, and keep your eye on the question you want to answer. Data will talk to you, but you must let them speak softly.

Introduction to Statistics

C h a p t e r 10

149

subset from that in the Senate as a w h o l e ) . If it is of the first type, y o u r c o n c l u s i o n w i l l be true; if it is of the s e c o n d (a distorted s a m p l e ) , your c o n c l u s i o n w i l l be false. T h i s is a c o m m o n problem that must be dealt with w henever anyone tries to describe a w h o l e population of c a s e s by looking only at a sample drawn out of the population. It c o m e s up in conducting social s c i e n c e research, opinion polling, assigning television ratings, conducting market a n a l y s e s , administering quality control in manufacturing, and in performing numerous other activities. Fortunately, there are w a y s to determine j u s t h o w l i k e l y it is that a particular result c o u l d have o c c u r r e d b y c h a n c e e v e n i f its opposite w e r e gener al l y true. B y

Inference, or H o w to G a m b l e on Y o u r Research

means of these techniques we c a n c a l c u l a t e h o w m u c h stock to put in our findings. It is a very c l o s e a n a l o g y to say that we c a n g a m b l e intelligently on the results of our study by c a l c u l a t i n g the odds that the results are false. W h e n we do this, we are s a i d to

measure

(or "test") the statistical significance of our results.

In l o o k i n g at h o w to m e a s u r e relationships in C h a p t e r 9, we found that there w e r e qualitatively different w a y s to go about it, depending on the l e v e l at w h i c h the variables w e r e m e a s u r e d . In m e a s u r i n g the statistical s i g n i f i c a n c e of a relationship, one broad procedure h o l d s for all l e v e l s of data, although there are differences in s p e c i f i c techniques. A s I p o i n t e d out i n C h a p t e r 8 , there are t w o m a i n areas o f s t a t i s t i c s : ( 1 ) m e a s u r i n g things, especially measuring relationships between variables; and (2) estimating h o w l i k e l y i t i s that the m e a s u r e y o u h a v e gotten c o u l d h a v e o c c u r r e d b y c h a n c e .

LOGIC OF MEASURING SIGNIFICANCE

In C h a p t e r s 8 and 9 we l o o k e d at the first of these a s p e c t s , and n o w we e x a m i n e the s e c o n d . (I m u s t note p a r e n t h e t i c a l l y that, as in p r e c e d i n g c h a p t e r s , I do not i n t e n d to t e a c h y o u how to use s t a t i s t i c a l tests. I n s t e a d I p r e s e n t the g e n e r a l l o g i c o f s u c h tests t o h e l p y o u u n d e r s t a n d w h a t they m e a n w h e n y o u s e e t h e m . T h i s a p p r o a c h m i g h t a l s o s e r v e as a u s e f u l s u p p l e m e n t to a s t a t i s t i c s c o u r s e , w h i c h s o m e t i m e s t e a c h e s students h o w to u s e the tests w i t h o u t putting the tests in a broader perspective.) It is quite p o s s i b l e to get a particular result by c h a n c e , even though the result is not true in general. If y o u c h o s e ten U . S . senators at r a n d o m , for instance, it w o u l d be p o s s i b l e to c o m e up w i t h a group c o n s i s t i n g largely of c o n s e r v a t i v e D e m o c r a t s and liberal R e p u b l i c a n s . F r o m that y o u might c o n c l u d e — e r r o n e o u s l y — t h a t there w a s a relationship b e t w e e n party and ideology in the Senate, w i t h R e p u b l i c a n s the m o r e liberal group. H o w e v e r , if y o u repeatedly drew groups of ten senators, selecting the ten at r a n d o m f r o m the w h o l e Senate, most of the time y o u w o u l d find that the D e m o c r a t s in y o u r group were more liberal than the R e p u b l i c a n s , simply because D e m o c r a t i c

T h e general procedure by w h i c h we c a l c u l a t e t i e probability that a g i v e n result c o u l d h a v e o c c u r r e d by c h a n c e , even though the true state of affairs w a s something other than what is i n d i c a t e d by the result, is the s a m e procedure that we u s e in c a l c u l a t i n g the odds of any event h a p p e n i n g . As usual w h e n talking about probability, let us start w i t h a d e c k of cards. G i v e n an honest d e c k of c a r d s , thoroughly shuffled, y o u are instructed to draw a c a r d . Y o u k n o w that the probability that y o u w i l l draw the k i n g of hearts is 1/52, for there are 52 cards and only one of them is the k i n g of hearts. Next, y o u are instructed to do this two times, replacing the c a r d and reshuffling after the first draw. Y o u k n o w that the probability of d r a w i n g a k i n g of hearts both times is 1/52 x 1/52 = 1/2704, or 0 . 0 0 0 3 7 , and so o n .

1

Y o u w e r e able to calculate these probabilities b e c a u s e , g i v e n a sufficient set of assumptions, the probability that any particular result w o u l d o c c u r c o u l d be determined. T h e s e assumptions had to account for all of the factors that c o u l d influence the o u t c o m e of the d r a w i n g . W i t h s u c h a set of a s s u m p t i o n s , it w a s possible

senators do tend to be more liberal than R e p u b l i c a n senators. B u t some of the time y o u w o u l d find that the R e p u b l i c a n s in your sample were more liberal than the D e m o c r a t s , or that the two parties did not differ at all. T h e problem in drawing a single group and describing the Senate f r o m it is that y o u cannot k n o w whether your group is of the first type (the relationship between the variables is the same in your subset as in the Senate as a w h o l e ) or of the s e c o n d (the relationship between the variables is different in the

148

'The logic of this is as follows: In order to draw two kings of hearts in two draws, you must accomplish two things. First, you must draw the king of hearts on the first round. This can be expected to happen only one fifty-second of the time. Now, assuming that you have succeeded in the first round, you still can expect success in the second round only one fifty-second of the time. In other words, in the full two-draw process, you can expect to proceed to final success only one fifty-second of one fifty-second of the time.

150

Chapter 10

Introduction to Statistics

to calculate exactly how likely any particular result was. The set of assumptions in this case was as follows: 1. T h e deck has 52 cards, of w h i c h only o n e is the king of hearts. 2. The deck is thoroughly and honestly shuffled before each drawing. 3. The drawing is blind, so that each card is equally likely to be drawn ( y o u don't peek, none of the cards stick together, and so on).

If any of these assumptions were not true, your calculation of probabilities would be incorrect. For instance, if there were two kings of hearts (that is, assumption 1 above was wrong), the probability of drawing one in a single draw would have been 2/52, not 1/52. Or if the deck had been stacked so as to make drawing the king of hearts likely (that is, assumption 2 above was false), the probability of drawing one would have been greater than 1/52. This is the way in which we normally use odds and probability. We have a set of assumptions of which we are confident. Those assumptions determine the probability that any particular event will happen. From this we know with how much confidence we can predict that the event will occur, that is, at what odds we should bet that it will occur. Statistical inference uses this same logical structure, but turns it on its head. An example may be the best way to demonstrate this.

EXAMPLE OF STATISTICAL INFERENCE Let us return to our previous example, taking a sample of ten senators to see whether there is a difference in how liberal the two senatorial parties are. Let us say that you have drawn such a sample and that your results are shown in Table 10-1. Of the ten senators in the sample, 75 percent of the Democrats, but only 33 percent of the Republicans, are liberal. How safe would you be to conclude from this limited sample that senatorial Democrats are on the whole more liberal? To find out, you must proceed just as you did in our card-drawing example, by setting a list of assumptions that would be sufficient to let you calculate the probability of drawing ten senators that look like those you have, if in fact Democratic and Republican senators do not differ in the degree to which they are liberal. These assumptions would be: 1. T h e ten senators have been drawn at random. 2. There is no relationship b e t w e e n party and i d e o l o g y in the "full" Senate from w h i c h these have been drawn.

TABLE 10-1

Sample Result Democrats

Republicans

Liberal

3

2

Conservative

I

4

151

Just from these two assumptions, using probability theory, it is possible for you to calculate how likely it is that you would have gotten any particular result in your sample. In this particular case, using a "chi-square" test (which I introduce later in the chapter), you can calculate that the probability of having gotten at least as great a difference as you found between the parties, given the assumptions, is 0.264. Thus there is a good chance that you could have gotten a relationship as strong as this in a sample of ten even if the assumptions that you set up were true—that is, even if there were no difference between Democrats and Republicans. How you would then treat your research results depends on whether you are a long-shot gambler at heart, what kind of risks ride on your decision, and so on. Approximately three times out of four you would not have gotten as strong a relationship as you did if the assumptions you wish to test were true. Will you reject that set of assumptions on the basis of your sample result, accepting the one chance in four that you are wrong in doing so? If you are willing to reject the assumptions, notice that the truth is questionable with regard to only one of the assumptions in the set; you purposely have set up the test this way. You know whether or not you have drawn the ten senators at random. Therefore, in rejecting the set of assumptions, you really are rejecting only one assumption, the assumption that there is no relationship between party and ideology in the Senate. What you are saying is: "By rejecting its opposite, I am making the statement, There is an ideological difference between the senatorial parties.' I know that given the amount of evidence I have gathered, I am running considerable risk in making the statement. There is a probability of 0.264 that I am wrong." 2

HYPOTHESIS TESTING The example we have used is a typical problem in statistical inference: We have a particular result in hand, and we wish to calculate the probability that this event could have occurred given some set of assumptions. We cannot be confident of one of these assumptions, for it involves the very thing we are trying to infer. If there is a sufficiently small probability that the event we have in mind could have occurred, given the set of assumptions, we decide to treat the set of assumptions as false. (This usually is referred to as "rejecting the hypothesis" that the assumptions are true.) And since we are confident of all but one of the assumptions (the thing we want to test), rejecting the set of assumptions really means that we are rejecting that one assumption. We run some risk of being wrong in treating the assumptions as false. After all, the result we have observed could have occurred even if the assumptions were true.

2

There is a great variety of statistical tests available to fit different circumstances of research. Statistical tests vary in the level of measurement that they require and in the particular set of assumptions that they embody. The chi-square test used here is particularly designed for testing the significance of a relationship between two nominal-scale variables. See the discussion on pp. 153-155.

152

Chapter 10

Introduction to Statistics

153

The probability we have calculated tells us precisely how likely it is that we are wrong. It tells us the probability that the result we have observed could have occurred if the assumptions we are rejecting were in fact true—that is, the probability that we are making a mistake in rejecting the assumptions. We normally will reject a set of assumptions only when the probability that they could have produced the result we have observed is comfortably low. You can see now why I said that statistical inference uses the same logic as setting odds, but turns it on its head. Normally, in setting odds, we select a set of assumptions from which we can confidently predict how likely it is that certain things will happen in the future. In statistical inference, we calculate how likely it is that an observed result could have occurred if some hypothetical and contrary assumption were true. If the probability is sufficiently low, we use the result to reject the assumption. Just as in everyday setting of odds, the probability shows us how likely it is that we are making a mistake.

range. However, because you do not know what the true value is, this is a test that is impossible to set up. In using the null hypothesis, on the other hand, a single alternative is chosen, which negates all possible true values that would be consistent with the statement you wish to make. For instance, in the Senate example, if you wanted to assert that your sample result was sufficiently close to the true relationship for you to accept the sample result as valid, you would have to choose among an infinite variety of possible true relationships (80 percent of Democrats liberal, 75 percent of Republicans liberal; 100 percent of Democrats liberal, 51 percent of Republicans liberal; 100 percent of Democrats liberal, 1 percent of Republicans liberal; and so on). By using a null hypothesis ("there is no difference between the parties"), you can deal with a single statement that negates all possible versions of the statement you want to make. It is simply easier to disprove a specific hypothesis than to prove an open-ended hypothesis.

Null Hypothesis

Example: %

It is clear from our description of statistical inference that we do not test directly the statement we wish to make about a relationship. Instead, we insert its opposite— which is called the null hypothesis, the hypothesis we should like to reject—into the set of assumptions from which we calculate the probability. Because it is the only questionable member of the set of assumptions, if we decide to reject the set, we really are rejecting only the null hypothesis. By rejecting the null hypothesis we in effect assert its opposite, which is the statement we wished to make in the first place. The twisted and convoluted way one must think to understand this process is discouraging to most people when they see it for the first time. "Why," they ask, "can't we just test our hypothesis directly instead of having to turn it inside out?" But think about how you might go about doing this. First of all, you could never test the hypothesis that your sample result was exactly the same as the true value in the population from which you had drawn the sample. Taken to a sufficient number of decimal places, it would virtually always not be the same as the true population value. For instance, 50.7309834 percent of voters in the United States voted for George W. Bush in 2004; no sample would have produced exactly this figure. But because your sample was not that precise a reflection of reality, you would not want automatically to reject it. Accordingly, you would pick some range of values around the true value and test whether or not the sample result was in that

One popular significance test is % (chi-square), which we used in our liberal— conservative example. It is designed for use with a table relating two nominal-scale variables. Given the strength of the relationship between the variables in the table, % allows us to estimate the probability that there is no relationship between the variables in the full population from which the sample has been drawn. In Table 10-2, a hypothetical relationship 'between ethnicity and policy priorities has been drawn, based on a sample of 200 respondents to a survey. We want to know whether it is safe to assert, on the evidence of this sample, that there is a relationship between the two variables. To calculate % this table must be compared with a construct designed to show a lack of relationship between the variables. Such a construct is shown in Table 10-3. The entries in Table 10-3 are arrived at by calculating how many WASPs, say, would give first priority to employment and welfare if exactly the same proportion of WASPs as of African Americans and "other ethnics" favored the employment/welfare

3

2

2

2

2

T A B L E 10-2

Ethnicity Preferred Priority

WASP

African American

Other

Total

53

110 30

30

27

Foreign affairs

11

5

14

Environment

29

CO

3

Whether the probability is "comfortably low," of course, is a difficult thing to judge. In setting odds, one usually has objective outside criteria to "help" in making the decision—an amount of money to be lost if the person is wrong, for instance. In much engineering research, there also will be such criteria available. In theory-oriented research, however, there usually are no fixed criteria, because the result is supposed to hold not just for some particular occasion, but in general. It is supposed to apply to occasions as yet unforeseen. Therefore, because objective criteria are lacking, a convention usually is followed of rejecting a set of assumptions only if the probability is less than 0.05.

Ethnicity and Preferred G o v e r n m e n t Action Priorities

23

60

Total

70

40

90

200

Employment and welfare

154

Chapter 10 T A B L E 10-3

Introduction to Statistics Table of Nonrelationship

2

together (you will recall that this is what the sign "X" means) to give us the %. The more the tables differ, the greater % will be. If the tables are exactly the same, % will equal zero, inasmuch as each cell's result will be 2

Ethnicity

Preferred Priority

WASP

Employment and welfare

38.5

Foreign affairs

10.5

Environment Total

155

African American

Other

Total

22.0

49.5

110

6.0

13.5

30

21.0

12.0

27.0

70

40

90

(Fp ~ F f F h h

=

2

=

F

h

In Table 10-4, % is calculated for the present example. Cell A is the upper-left cell of Table 10-2 or 10-3 (the cell in which WASPs choosing employment/welfare fall), cell B is the next cell to the right, cell C is the upper-right cell, cell D is the middle left cell, cell E falls at the exact center of the table, and so on. 2

60 200

Sampling Distribution 4

priority. Thus, because there are 70 WASPs in the sample, and 110/200 of the sample favor the employment/welfare priority, we would expect to find that the sample contained 110/200 x 70 = 38.5 WASPs who chose employment/welfare. Similarly, we would expect to find that 110/200 of the 40 African Americans, or 22.0, chose employment/welfare, and so on, cell by cell, until the hypothetical table was filled. Notice that to construct a table in which the variables are completely unrelated, it was necessary to maintain the fiction that there could have been one-half of a WASP in favor of the employment/welfare priority. Table 10-3 embodies the null hypothesis that we want to test—that there is no relationship between the variables. Each cell in the table has the characteristic (1) that its entry is the same proportion of its row frequency as its column frequency is of the total sample, and (2) that its entry is the same proportion of its column frequency as its row frequency is of the total sample. That is, the members of each row are equally likely to fall into any given column, and vice versa. This is what we should expect if the two variables were unrelated. The question we want to ask is: Are the two tables sufficiently different that we can say, on the basis of what we found when we looked at the sample, that the full population does not look like the distribution in the hypothetical table? Chi-square is a measure of how different the two tables are. It is calculated from the formula

Before I can show how we go from measuring the difference between the observed and null hypothesis tables to calculating the probability that the null hypothesis is true, I must first introduce the idea of a sampling distribution. For any particular set of assumptions, a sampling distribution shows what proportion of the time each particular result could be expected to occur if the sampling technique chosen were repeated a very large number of times and the set of assumptions (including the null hypothesis) were true. That is, it gives the probability of getting a particular result if you applied the techniques you are using to a population for which the null hypothesis is true. A sampling distribution embodies all the assumptions you make in a test, including the null hypothesis. It represents the application of probability theory to those assumptions, to calculate the probability that each possible result would occur. Some simple sampling distributions can be calculated easily. For instance, in the card-drawing example on p. 149, the sampling distribution states that the probability of drawing the ace of hearts is 1/52, the probability of drawing the

T A B L E 10-4

Calculations for %

(F ~ F )

Cell

0

h

2

(F -F )

(F - F ) /F

1.877

2

0

h

0

h

h

38.5

-8.5

72.25

22.0

+ 5.0

25.00

1.136

C

49.5

+ 3.5

12.25

0.247

D

10.5

+ 0.5

0.25

0.024

-

1.0

1.00

0.167

+ 0.5

0.25

0.019

+ 8.0

64.00

3.048

-4.0

16.00

1.333

-4.0

16.00

0.593

E

G

I WASP is the acronym for white, Anglo-Saxon Protestant.

h

B

H

4

0

A

LL.

where for each cell in the table, F is the observed frequency and F is the frequency predicted for the hypothetical table. Thus for each cell in the table, (1) the prediction from the hypothetical table is subtracted from the actual figure, (2) this figure is squared, and (3) the squared difference is then divided by the prediction from the hypothetical table. The results of this, from all the cells of the table, are added

2

6.0 13.5 21.0 12.0 27.0

X = total = 8.444 2

156

Introduction to Statistics

Chapter 10

157

k i n g o f hearts i s 1/52, a n d s o o n , t h r o u g h a l l the c a r d s o f the d e c k . O f c o u r s e , this 1.00

s a m p l i n g d i s t r i b u t i o n d e p e n d s o n the set o f a s s u m p t i o n s y o u m a d e i n that e x a m p l e . T h e s a m p l i n g d i s t r i b u t i o n for a different set o f a s s u m p t i o n s w o u l d l o o k different.

.75

In another s i m p l e c a s e , suppose that y o u h a d to predict the probability of

+^

getting j u s t one head in two flips of a c o i n . A s s u m e that (1) the c o i n is honest, (2) the flipper is honest, and (3) the c o i n w i l l not stand on its edge. Y o u c a n now calculate

a .50 n o

that the probability of getting a head f o l l o w e d by a tail is 0.25, the probability of

a.

getting a head f o l l o w e d by a head is 0.25, the probability of getting a tail f o l l o w e d by .25

a h e a d is 0.25, and the probability of getting a tail f o l l o w e d by a tail is 0.25. B e c a u s e two of the possibilities involve getting j u s t one head in the two throws, the probability of getting j u s t one head in either of the two w a y s is 0.50, the probability of getting

.00

no heads is 0 . 2 5 , and the probability of getting two heads is 0.25. T h i s s a m p l i n g

0

distribution is presented g r a p h i c a l l y in F i g u r e 10-1. T h e same s a m p l i n g distribution c o u l d be presented in a cumulative graph, s h o w i n g the probability of getting

at least no

1

2

Number of Heads Figure 1 0 - 2

heads, at least one h e a d , or at least two

Cumulative Sampling Distribution for Two

Flips of a C o i n

heads. F i g u r e 10-2 charts the sampling distribution in this way. We a l w a y s w i l l get at

C h i - s q u a r e sampling distributions vary depending on the number of categories in

least zero heads ( w e cannot get a negative n u m b e r ) , so the probability of that outcome

the rows and c o l u m n s of the table used. T h e probabilities given are the probability that

is 1.0; we c o u l d get at least one head either by getting no heads (probability of 0.25) or

a

by getting one head (probability of 0.50), so the probability of at least one head is 0.75;

drawn by random selection from a parent population in w h i c h the variables were unre-

we c o u l d only get at least two heads by getting exactly two heads, so the probability of

lated. (Note that this is a cumulative sampling distribution, like the one depicted in

at least two heads is 0.25.

F i g u r e 10-2.) T h u s , if y o u had found a % of 4.9, y o u w o u l d k n o w that the probability of

2

x at least as high as the one observed w o u l d have been obtained if the sample were 2

T h e s e s a m p l i n g distributions are s i m p l e . T h e s a m p l i n g distributions for most statistical tests are m o r e c o m p l e x , a n d we must u s e tables developed by statisticians 2

to v i s u a l i z e them. T h e c u m u l a t i v e s a m p l i n g distribution of % for a three-by-three table (a table w i t h three categories in the r o w s and three categories in the c o l u m n s , s u c h a s T a b l e s 10-2 and 10-3) i s presented i n F i g u r e 1 0 - 3 .

getting x as high as this w a s 0.3 if the variables y o u were w o r k i n g w i t h w e r e , in fact, 2

not related.

( T h e sampling distribution is based on assumptions of (1) the null

hypothesis y o u w i s h to test, and (2) random selection, w h i c h y o u k n o w y o u have done.) F r o m this s a m p l i n g distribution we see that there is a probability of between 2

0.05 and 0.10 that we w o u l d have f o u n d a % as h i g h as 8.44 f r o m a three-by-three table if there w e r e really no relationship b e t w e e n ethnicity and c h o i c e of priority (that i s , if the n u l l h y p o t h e s i s w e r e true). T h i s m e a n s that we w o u l d r u n a r i s k of

Figure 1 0 - 1

Sampling Distribution for Two Flips of a C o i n 2

Figure 1 0 - 3

Sampling Distribution of % , for Three-by-Three Tables

1.00

.75

> ro .50

no i _

a. .25 0 .00 0

1 Number of Heads

2

1

2

3

4

5

6

7

8

9

10 11

12 13

14 15

16

17

18

19

1 5 8 Chapter 10

Introduction to Statistics 1 5 9

b e t w e e n 1 c h a n c e in 10 and 1 c h a n c e in 20 of b e i n g w r o n g in asserting on the b a s i s of T a b l e 10-2 that the n u l l h y p o t h e s i s is f a l s e — t h a t there is a relationship between ethnicity and the c h o i c e of priorities. To s u m m a r i z e these sections: A sampling distribution is a list of the probabilities that e a c h possible event (out of a set of events) w i l l happen. T h e probabilities in a sampling distribution are based on a set of assumptions. If we choose our significance test appropriately, the set of assumptions on w h i c h its s a m p l i n g distribution is based w i l l consist of (1) a null hypothesis (the assumption that what we really want to say about the data is false), and (2) a group of other assumptions that we feel certain are true. If we find f r o m the sampling distribution that it is very unlikely, given those assumptions, that the event we have observed (in our ethnicity/choice of priorities e x a m p l e above, finding a yf of 8.44) c o u l d have o c c u r r e d , we reject the assumptions on w h i c h the s a m p l i n g distribution is based. B e c a u s e we are sure that all but one of those assumptions (the null hypothesis) are true, this amounts to rejecting the null

Independent Variable

hypothesis and asserting its opposite, w h i c h is what we had originally wanted to say about the data. T h e probability that we c o u l d have gotten the observed result if all the

Figure 10-4

assumptions underlying the sampling distribution were true tells us h o w likely it is that

Regression Analysis with Zero V a r i a n c e in

the Independent Variable

we are w r o n g to reject the null hypothesis. T h u s a significance test furnishes a useful c h e c k on our research. It tells us how likely it is that we c o u l d have gotten our results by c h a n c e a l o n e — t h e probability that in fact the opposite of what we are asserting is true.

m a n y c a s e s we have used but also of h o w m u c h the independent variable varies. In the extreme c a s e of zero variance in the independent variable, depicted in the scattergram in F i g u r e 10-4, we are s i m p l y unable to c h o o s e any regression line. B e c a u s e all the data points fall above a single value of the independent variable, an infinite n u m b e r of lines (a few of w h i c h are indicated on the graph) c a n be p a s s e d through the data, e a c h

I m p o r t a n c e of N

of w h i c h has an e q u a l s u m of squared deviations about itself. In other w o r d s , it is N, the number of c a s e s in a s a m p l e , is a l w a y s a factor in s i g n i f i c a n c e tests. A l l other things being e q u a l , the greater the n u m b e r of c a s e s on w h i c h a statement is based, the m o r e certain y o u c a n be that the statement is true.

5

Every

s i g n i f i c a n c e test,

i m p o s s i b l e to c h o o s e a single " b e s t " l i n e by least-squares criteria. T h i s is the extreme c a s e . In g e n e r a l , the greater the variation in the i n d e p e n d ent v a r i a b l e , the m o r e firmly fixed the estimate of the r e g r e s s i o n l i n e c a n be a n d ,

a c c o r d i n g l y , takes the n u m b e r of c a s e s into account. If e a c h n u m b e r in T a b l e s 10-2

a c c o r d i n g l y , the m o r e stock we c a n put in our findings. O n e w a y to l o o k at this is to

and

w o u l d double for

think o f the r e g r e s s i o n line a s w o b b l i n g o n a f u l c r u m . A s i t h a p p e n s , every r e g r e s -

w o u l d thus be double 8.444, or 16.888. F r o m F i g u r e 10-3 we see

sion l i n e must p a s s through point x, y in the scattergram; this is determined mathe-

10-3 were doubled, for e x a m p l e , the quantity (F — F ) /F 2

0

each cell. The %

h

h

that if the null hypothesis of no relationship w e r e true, the probability of getting a %

2

m a t i c a l l y by the f o r m u l a s for a a n d b. ( I n other w o r d s , the e x p e c t e d v a l u e of y w h e n

at least this high w o u l d be less than 0.01. T h u s f r o m a table s h o w i n g e x a c t l y the s a m e

x equals its o w n m e a n must be the m e a n of y.) If there is relatively little variation in

pattern as T a b l e 10-2 but based on a s a m p l e of 4 0 0 c a s e s instead of the 2 0 0 used in

the independent v a r i a b l e , as in F i g u r e 1 0 - 5 , the l i n e is free to w o b b l e a g o o d d e a l ,

that table, we w o u l d run a risk of less than 1 c h a n c e in 100 of being w r o n g in assert-

u s i n g this point as a f u l c r u m . A c o n s i d e r a b l e c h a n g e in the angle at w h i c h the line

ing that there is a relationship between the v a r i a b l e s .

6

p a s s e s through the point x, y w i l l not c h a n g e the s i z e of the s q u a r e d deviations f r o m

N is the most o b v i o u s factor affecting the statistical s i g n i f i c a n c e of findings. In

the l i n e by v e r y m u c h . T h i s is due to the fact that for observations w h o s e v a l u e on x

m a n y sorts of statistical tests, it is the only factor we need to w o r r y about. B u t there

is c l o s e to x, w o b b l e in the line does not c h a n g e the e x p e c t e d v a l u e of y v e r y greatly;

m a y be others. F o r e x a m p l e , in regression a n a l y s i s the amount of c o n f i d e n c e we c a n

thus the difference b e t w e e n the o b s e r v e d and e x p e c t e d v a l u e s of y also is not greatly

h a v e in our estimate of w h a t the relationship l o o k s like is a matter not only of how

changed. No one of the observations is able significantly to affect the regression line in F i g u r e 10-5. T h e l i n e i s s i m p l y not h e l d f i r m l y i n p l a c e b y these observations, w h o s e

5

6

S e e the box "Law of Large Numbers" on p. 61.

T h e following five paragraphs present advanced material that can be skipped over without any loss in comprehension of other material covered in this book.

values on x fall so c l o s e together. O b s e r v a t i o n s w i t h w i d e l y v a r y i n g v a l u e s on x, h o w ever, w o u l d h o l d a m u c h firmer grip on the regression line. T w o s u c h observations have been added in F i g u r e 10-6.

160 Chapter 10

Introduction to Statistics a regression has been due to chance, we must take into account not only how many observations have been used in the analysis, but also how widely they are spread on the independent variable. Significance tests designed for use with regression analysis take into account both the number of cases and the amount of variation in the independent variable. 7

Problem of Independent Observations

Figure 10-5 Regression Analysis with Little Variance in the Independent Variable

If the regression line were now to wobble through the same angles as in Figure 10-5, the size of the deviations of observations A and B (in Figure 10-6) from the line would increase dramatically. Remember, the regression line must be the line that minimizes the sum of squared deviations about itself. The regression line in Figure 10-6 cannot stray very far from points A and B without sharply increasing that sum. Thus a few points whose x values are extreme have more impact in determining the slope of the line than do a larger number of points whose x values fall close to the mean. Accordingly, in calculating how likely it is that our estimate of Figure 1 0 - 6 The Stabilizing Independent Variable

y

Effect of Variance

in

the

It often happens that a researcher draws a conclusion based on a set of observations that appear to be independent but are really not. Because there are in effect fewer observations in the study than the scholar believes, he or she may overestimate the strength of the evidence. The relationship in question might be due to chance, without the researcher realizing it. The difficulty here is that statistical tests will not be able to alert us to the danger, because statistical tests already assume that the data to be analyzed consist of independent observations. If the observations are not independent, all statistical tests will underestimate the probability that a relationship might be due to chance. Thus we might be inappropriately confident of our assertions. An example may help. Consider a study that seeks to determine the extent to which "police brutality" is a function of the explicitness of guidelines issued to police patrols, with close and explicit directions presumed to be associated with low "brutality." The investigator might measure the explicitness of guidelines to patrols in the 11 towns and 63 townships of a county and relate this measure by regression analysis to the roughness with which police in each of those units handle suspects. Statistical tests might be applied to the regression results, with N stated as 74. Unknown to the investigator, however, it might be that some of these units shared a coordinated police force. For example, all township police forces might be under the direction of the county sheriff. In this case, although the 63 townships looked like separate units, they would all be separate parts of a single police administration, with one set of guidelines and one set of expectations with regard to roughness. Thus there would in fact be only 12 independent observations (each of the 11 town forces, plus the sheriff's office), which should raise a specter of doubt about the correctness of an assertion based on these data. The problem of nonindependent observations does not often occur in a simple form such as this, and if it does arise, it can generally be avoided by simple common sense. However, the problem occurs frequently in a more subtle (and more technical) way under the guise of autocorrelation. Autocorrelation occurs when observations are related to some extent but are not (as in the study of police brutality) identical to each other. For example, if we look at the relationship between trends in the economy and trends in the U.S. president's popularity, month by month, we will probably have a 7

T h e general extension of this problem to multivariate regression is the problem of multicollinearity. For a good expository discussion of multicollinearity, see King, Keohane, and Verba (1994, ch. 4, part 1).

162

Chapter 10

Introduction to Statistics

problem of autocorrelation. February 1943 had much in common with May 1943, for instance. Though one would not go so far as to say that gathering information from these two months is simply equivalent to having information about a single month (as we could say about any two townships in the other example), neither can we say that we have as much information here as we would have with two widely separated months. I mention the problem at this point only to alert you to it. Partial solutions are available, but they are beyond the scope of this book. Most texts in statistics or econometrics, such as the one by Knoke, Bohrnstedt, and Mee (2002) or Kmenta (1997), discuss the problem.

163

Sometimes an author claims that she need not be concerned about the possibility of chance results because her sample is synonymous with the total population of subjects (all the states, all the nations in the United Nations, all the U.S. senators, or what have you). This is a statement that should arouse suspicion. If the author is trying to draw general conclusions from the study, propositions that one would expect to be as true of states or nations or senators a decade from now as they are today, she must be concerned with the problem of chance results. With infrequent exceptions, then, this is a problem all of us must take seriously in our research.

POLLING AND SIGNIFICANCE TESTS SIGNIFICANCE TEST: ALWAYS NECESSARY? Even when we do not work with a limited sample drawn from a larger group of subjects, there is a sense in which all research involves "sampling" of a sort and in which misleading results can occur by chance. For instance, we might look at all 50 states and find that the generosity of their welfare budgets is related to the degree of party competition in the states' elections. In one sense, there is no question but that this is the "true" result. Because we have included all the states in our sample, anything we find out about them is by definition "true generally." But suppose we think of ourselves as trying to say something not just about the 50 states as they exist at this one point in time, but about "state politics." Then we must regard these 50 states as a sample drawn for us by chance and accident from a larger metaphysical population of "states"—all states as they might be in the future or as they might have been had their boundaries been drawn differently or had history proceeded differently. The latter view seems to reflect more accurately what we try to do in developing political theories out of empirical research. There are times when we wish to describe a specific population as it exists at one point in time; this is particularly likely in what I have termed "engineering" research. In such research we frequently are interested in measuring some condition of a population so as to react to it or make adjustments in it, rather than to develop explanatory theories from it. For instance, a tax administrator may want to know how many states administer sales taxes, how great a portion of national income these states involve, and so on. What the administrator is concerned with is a description of the tax situation as it currently exists, not, by and large, with developing theories to explain why things are as they are. In most theory-oriented research, however, we should seek a more inclusive sort of generality. 8

Note, by the way, that to the extent that an engineer sees her task as changing and reforming patterns rather than continuing ongoing administrative procedures, explanatory theory will be relatively more important. A tax administrator who wanted to change the extent to which states relied on sales taxes to generate income would first have to try to find out why states use sales taxes. Status quo administrators, on the other hand, can be more content with purely descriptive information, because their chief concern is with plugging established procedures into any given state of affairs, and what they need is simply to know what the "state of affairs" is. 8

A ubiquitous activity in our society is polling, asking a sample of the population whom they will vote for, whether they support regulation of handguns, or whatever. Typically, about 2,000 respondents are polled to describe a population such as that of the United States, and the results are reported as "accurate to within plus or minus 3 percent"—or 2 percent, or some other level of accuracy. This operation is a variant of statistical significance testing. Based on a given number of cases, which of course can be set as desired by the person doing the polling, and with knowledge of how the individuals have been chosen for the sample (usually some variant of random sampling), the pollster can conclude that according to the sampling distribution of percent approval that one would get using repeated samples of this sort and of this size, sample results would fall within plus or minus X percent of the true population value 95 percent of the time. Thus the pollster claims that the result is accurate to "within plus or minus X percent." Note that this does not mean that the result absolutely falls within plus or minus X percent of the true value; it means that 95 percent of the time it would fall within that range of the true value. Note here, also, that changing N changes the width of the "plus or minus" range. If an N of about 2,000 respondents gave you plus or minus 3 percent, increasing A' to 4,000 would give you a lower range, perhaps plus or minus 1 percent. 9

USES AND LIMITATIONS OF STATISTICAL TESTS I have tried in this chapter to present a significance test for what it is—a useful check on research results. The real meat of research, however—the way we find out about politics—is by looking at data and seeking out relationships between variables, the sorts of things I discussed in Chapters 2 through 9. Unfortunately, researchers often place undue emphasis on significance tests. It is a pity that looking at data requires less formal training (but more practice) than does

''See the discussion of sampling on pp. 9 9 - 1 0 3 .

164

Introduction

Chapter 10

calculating significance tests. Perhaps because they have spent so much time in courses learning to use significance tests, many researchers give the tests an undue emphasis in their research. The status of these tests should be strictly that of a secondary check on the creative work the researcher has done in looking at relationships. A recent article illustrates the problem. In this study, the authors reported that two variables were more strongly related among the working-class portion of their sample than among the middle-class portion. Their evidence for this was that among the working-class respondents, the relationship was more highly significant, with a probability of less than 0.01 that there was no relationship. Among the middle-class respondents, by contrast, the relationship was less highly significant, with probability between 0.01 and 0.05. Because previous theorists had said that one should expect a stronger relationship between these variables among the middle class than among the working class, these authors were understandably excited about their findings. Unfortunately, they were using significance tests for something the tests were not meant to do. The significance test is not itself a measure of the strength of a relationship but rather, a check on how likely it is that a given measure is due to chance. In this example, as it happened, the middle-class portion of the authors' sample was only about half as large as the working-class portion. It was because of the smaller number of cases in the middle-class portion that its relationship showed up as less significant than the same relationship for the working-class portion. In fact, when the strength of the relationship was measured by an appropriate technique (the Goodman-Kruskal gamma, for instance), it turned out that there was a Wronger relationship among the middle class. The authors' conclusion from their own data was wrong.

CONCLUSION I have tried in this chapter to give you some idea of the general logic of inference, which is the common thread running through all statistical tests. These tests vary among themselves, however, in terms of their suitability for a given level of measurement and the particular mix of assumptions (other than the null hypothesis) they require the investigator to guarantee. This is why there are so many different tests. For a description of particular tests, the reader should consult a statistics text such as the one cited at the end of Chapter 8.

FURTHER DISCUSSION A particularly good discussion of the misapplication of significance tests, along the lines of the example I used, is Duggan and Dean, "Common Misinterpretations of Significance Levels in Sociological Journals" (1968). See also Winch and Campbell, "Proof? No. Evidence? Yes. The Significance of Tests of Significance" (1969).

to

Statistics

Anyone thinking of using these tests should first take a course in statistics; barring that, however, useful presentations of the techniques are given in the text cited at the end of Chapter 8. The second chapter of Siegel (1956) is a particularly useful review of the overall logic of significance tests. It aims at much the same presentation as I have attempted in this chapter, but in rather more technical detail. Another very good treatment is that of Simon (1985). For further consideration, think about the two following situations. What is wrong in each situation ? 1. A researcher studying Congress examines the vote on 100 or so bills. For two of these votes, she finds a statistically significant relationship (at the 0.05 level) between a representative's height and the way he voted on the bill. This strikes her as a surprising finding, and she uses it as the basis for a chapter and a half of her book. 2. A large number of scholars study committee systems to see whether democracy in committee decision making leads the members to be satisfied with their work on the committee. All but one of these scholars fail to find a statistically significant relationship between these two variables. Most of those who fail to find a relationship leave that question and start to work on other things. A few of them write up their results, but these are rejected by journal editors because they are negative, not positive, findings. The one scholar who did find a statistically significant result publishes it. Net result: one published article, reporting a statistically significant relationship between committee democracy and members' satisfaction.

Chapter 11

Where Do

Theories

Come

167

From?

follow, then, that the m o s t appropriate w a y to devise a theory is simply to look at the j u m b l e

of observations

and

pick

out

the

most

prominent

pattern

running

through them? Unfortunately, "observations" in political science are riddled with problems of inaccurate m e a s u r e m e n t ; they are often m e a s u r e d imprecisely, and the nonexperi-

®d4

m e n t a l c i r c u m s t a n c e s u n d e r w h i c h w e g a t h e r t h e m m a k e i t difficult t o isolate the

1

©DTTD

effects o f s i n g l e v a r i a b l e s . T h e s e a r e all p r o b l e m s I d i s c u s s e d e a r l i e r i n t h e b o o k . T h e i r c o m b i n e d effect is to m a k e it very difficult to l o o k at a b a t c h of o b s e r v a t i o n s and pick out the best simplifying pattern. There is so m u c h going on in a group of

Lr®LnjTj9o

observations, m u c h of which is extraneous to what we want to do, that the pattern we ideally w o u l d h o p e to pick out is obscured. This problem is not confined to the social sciences. An amusing example of the difficulty of c h o o s i n g a m o n g potential e x p l a n a t i o n s w i t h o u t the benefit of prior theory

is offered by

some

of their friends

a discussion among Samuel Johnson, James in

1769

about

the

mystery

of w h y

Boswell, and

the residents

S c o t t i s h island of St. K i l d a c a u g h t c o l d w h e n e v e r a s h i p arrived t h e r e .

1

of the

There was

at that time already speculation that illnesses were transmitted from one person to another, This has b e e n a b o o k a b o u t d e v e l o p i n g theories a n d trying t h e m out on reality. In C h a p t e r 2, I e q u a t e d the research process with a search for elegant theories. In

but

Johnson

regarded

this

as

a

"prejudice,"

and

not

"smart

modern

thought." Eventually the explanation favored by B o s w e l l w a s offered in a letter from a L a d y of Norfolk:

succeeding chapters, I have discussed various aspects of research: concept formation, m e a s u r e m e n t , d a t a analysis. M y criterion for t h e usefulness o f any o f t h e s e t e c h n i q u e s has been the extent to which that technique aided in the creation of elegant theories. A high d e g r e e of "precision in m e a s u r e m e n t " is important, I h a v e argued, b e c a u s e it a l l o w s u s m o r e f l e x i b i l i t y i n s t a t i n g t h e theory w e c h o o s e t o w o r k w i t h . M e a s u r e m e n t accuracy, w h e t h e r it is a matter of reliability or validity, is important b e c a u s e without accurate m e a s u r e m e n t we cannot establish the connection b e t w e e n the data we are

Now for the explication of this seeming mystery, which is so very obvious as, for that reason, to have escaped the penetration of Dr. Johnson and his friend, as well as that of the author. Reading the book with my ingenious friend, the late Reverend Mr. Christian, of Docking—after ruminating a little, "The cause, (says he,) is a natural one. T h e situation of St. K i l d a renders a North-East Wind indispensably necessary before a stranger can land. The wind, not the stranger, occasions an epidemic cold."

h a n d l i n g a n d t h e theory w i t h w h i c h w e a r e w o r k i n g . R e g r e s s i o n a n a l y s i s u s u a l l y i s m o r e useful than correlation analysis, b e c a u s e the results of regression analysis apply g e n e r a l l y t o a theory, w h e r e a s t h e r e s u l t s o f c o r r e l a t i o n a n a l y s i s c a n s e r v e o n l y a s a description of a particular situation; and so on.

In a similar vein, J a m e s S. C o l e m a n described the w a y the theory of gravity m i g h t not h a v e b e e n d e v i s e d h a d G a l i l e o g o n e a b o u t h i s t a s k i n t h e w a y m o s t d a t a analyzing social scientists do:

In all of this, I h a v e n e v e r dealt w i t h t h e critical q u e s t i o n : J u s t h o w d o e s o n e find a theory on w h i c h to w o r k ? T h e r e is o n e deceptively s i m p l e a n s w e r to this question,

A simple example will illustrate some of the difficulties which might arise by this

w h i c h i s t o c h o o s e t h e o r i e s for s t u d y f r o m t h e a l r e a d y existing b o d y o f w o r k i n politi-

kind of "brick-by-brick" approach to theory. Suppose that early mechanics had

cal science. T h a t is, take an existing t h e o r y a n d try it out on s o m e data. I d i s c u s s e d

developed by the use of regression equations. Suppose, specifically, that an investiga-

s o m e o f t h e l i m i t a t i o n s o f this p r o c e d u r e i n C h a p t e r 2 , but t h e i m p o r t a n t p o i n t for m y

tion had been carried out relating the length of time a body had fallen through the air

discussion here is that this a n s w e r begs the question of h o w the original theorist found

and the velocity it attained. The relation in mechanics is that the velocity attained is

the theory.

equal to the acceleration due to gravity times the time the object has fallen, or

W h a t m a k e s a political scientist describe the relationship a m o n g a group of variables

in

a

particular

way?

Probably

it

most

frequently

happens

that

v = gt

the

researcher observes a b o d y of data and tries to see a pattern a m o n g them. This is the most obvious way to go about devising a theory; it follows quite naturally from the things a theory is s u p p o s e d to achieve. T h e p u r p o s e of a theory is to p r o v i d e a

'G. Birkbeck Hill, Be/swell's Life of Johnson, Vol. 2 (London: Clarendon Press, 1887), pp. 51-52.

simplified pattern to describe a complicated j u m b l e of observations. W o u l d it not

2

166

l b i d . , p . 52.

168

Where Do Theories Come From?

Chapter 11 where g is the acceleration due to gravity. Now if there had been numerous investiga-

of epidemic growth and population dynamics, from such fields as epidemiology

tions involving different-sized bodies, different velocities, and bodies with differing

and ecology, m a y suggest theories to political scientists or sociologists.

densities, the investigators would have ended with numerous pairs of observations

microeconomic theory; as noted in Chapter 1, rational choice theory drawn from

( V j , t ), which they would locate on a scatter diagram in order to find the line of best

m i c r o e c o n o m i c t h e o r y h a s b e e n p a r t i c u l a r l y fruitful for political s c i e n c e . A h e a l t h y

t

fit. But in every case, and especially for high velocities (i.e., objects which fell a great distance) and low-density objects (i.e., feathers), the observed velocity would fall considerably below that which the theoretical equation predicts. The resulting regression equation might have ended up including other variables, such as mass or density of the object; and there would have been indications that at high velocities the

So may

a w a r e n e s s o f m a j o r t h e o r i e s i n f i e l d s s u c h a s t h e s e , s o m e o f w h i c h are n o t all t h a t closely related to our field, can be a helpful source of theory. T h e art of building

a theory r e m a i n s in

flux in political

science—partly

deductive, but largely inductive. T h e resulting confusion can be both enjoyable and

relation of velocity to time was not even linear. The reason, of course, would be air

fruitful, b e c a u s e m o r e t h a n in m o s t d i s c i p l i n e s , it a l l o w s a p l a c e for every sort of

resistance, which has different effects as a function of the density of the object, its

imagination to work:

shape, its velocity, and other things. The regression equation would of course have

tion, m a t h e m a t i c a l imagination. It is in this spirit that I h a v e tried to stress the

been

"craft" in the "craft of political research."

empirically

correct,

but

it

wouldn't

have

corresponded

to

the

simple

velocity-time relation which served as the basis for Galileo's remarkable contribution to the science of mechanics. They might even have served to confound the issue, by bringing in too soon a factor—i.e., air resistance—which was irrelevant to the fundamentals of mechanics.

3

T h e m o r a l o f t h e s e s t o r i e s i s t h a t w e a l m o s t a l w a y s a r e b e t t e r off i f w e h a v e s o m e i d e a o f w h a t k i n d o f pattern w e w a n t t o l o o k for b e f o r e w e start t o l o o k a t data. If we h a v e in m i n d a p a r t i c u l a r k i n d of pattern, it is e a s y to tell, a m o n g the j u m b l e of t h i n g s w e find, w h a t i s relevant t o that p a t t e r n a n d w h a t i s e x t r a n e o u s . G a l i l e o w a s able to i g n o r e the effects of air resistance b e c a u s e he k n e w that they w e r e not the thing he should use to explain the speed of falling objects. The distinction here is between "inductive" and "deductive" theory building. To

169

build

theory

inductively,

the

researcher

scans

the

observations,

looking

for

patterns. To build theory deductively, the researcher deduces (from something else, s o m e prior e x p e c t a t i o n s ) w h a t sort of a pattern to e x p e c t a n d t h e n l o o k s for it a m o n g the observations. A c c o r d i n g to the a r g u m e n t I h a v e p r e s e n t e d so far in this chapter, d e d u c t i v e t h e o r y building is clearly t h e better of t h e t w o . T h e p r o b l e m lies with that " s o m e t h i n g else" from w h i c h the student is supposed to d e d u c e theories. T h e r e simply are not m a n y w e l l - e s t a b l i s h e d premises i n t h e s o c i a l s c i e n c e s f r o m w h i c h t o d e d u c e a n y t h i n g . O n e w a y of distinguishing an o n g o i n g " s c i e n c e " from a " p r e - s c i e n c e " is that the f o r m e r i n c l u d e s a g e n e r a l l y a g r e e d - u p o n b o d y o f a s s u m p t i o n s f r o m w h i c h m o s t o f its theories can be d e d u c e d (Kuhn, 1962). Political science, sociology, and similar social sciences c a n n o t be said to possess such a b o d y of assumptions. L a c k i n g this b a s e for d e d u c t i o n , it is h a r d to a r g u e p a t l y that t h e o r y b u i l d i n g must be done deductively. On the other hand, it is important to bear in mind the advantages of deduction, w h e r e it is feasible. T h e r e are s o m e tricks that m a y heighten your awareness of situations in w h i c h deduction is feasible. O n e of the best s o u r c e s of d e d u c t i v e t h e o r y is a w e l l - e s t a b l i s h e d t h e o r y f r o m a n o t h e r field. T h e o r i e s

3

James S. Coleman, Introduction to Mathematical Sociology (New York: The Free Press, 1964), pp. 100-101. Copyright 1964 by the Free Press.

literary imagination, scientific imagination, m o r a l imagina-

Selected Bibliography CAMPBELL, DONALD T, and JULIAN C. STANLEY.

171

1963. Experimental and Quasi-experimental Designs for

Research. Chicago: Rand McNally. CARTER, LEWIS F. 1971. "Inadvertent Sociological Theory," Social Forces, 50, 12-25. COLEMAN, JAMES S. 1964. Introduction to Mathematical Sociology. New York: Free Press. CONVERSE, PHILIP E. 1964. "The Nature of Belief Systems in Mass Publics." In David Apter, ed., Ideology and Discontent, pp. 2 0 6 - 2 6 1 . New York: Free Press. . 1969. "Of Time and Partisan Stability," Comparative Political Studies, 2, 139-171. , and GEORGES DUPEUX. 1962. "Politicization of the Electorate in France and the United States," Public Opinion Quarterly, 26 (Spring), 1-23. , and ROY PIERCE. 1985. "Measuring Partisanship," Political Methodology, 11, 143—166. COOK, THOMAS D., and DONALD T. CAMPBELL.

1979. Quasi-experimentation: Design and Analysis Issues

for Field Setting. Chicago: Rand McNally. DAHL, ROBERT. 1961. Who Governs! New Haven, Conn.: Yale University Press. . 1966. Political Oppositions in Western Democracies. New Haven, Conn.: Yale University Press. . 1984. Modern Political Analysis, 4th ed. Upper Saddle River, N.J.: Prentice Hall. DALTON, RUSSELL J., and STEVEN WELDON. 2007. "Partisanship and Party System Institutionalization,"

Party Politics, 13, 179-196. DIEHL, PAUL F, and JEAN KINGSTON. 1987. "Messenger or Message? Military Buildups and the Initiation of Conflict," Journal of Politics, 49 (August), 801-813. DIGESER, PETER. 1992. "The Fourth Face of Power," Journal of Politics, 54 (November), 977-1007. DOWNS, ANTHONY. 1957. An Economic Theory of Democracy. New York: Harper & Row. DRAPER, N. R., and H. SMITH. 1981. Applied Regression Analysis, 2nd ed. New York: John Wiley & Sons. DUGGAN, THOMAS J., and CHARLES W. DEAN. 1968. "Common Misinterpretations of Significance Levels

ACHEN, CHRISTOPHER H. 1975. "Attitudes and the Survey Response," American Political Science Review, 69 (December), 1218-1231. . 1986. The Statistical Analysis of Quasi-experiments. Berkeley, Calif.: University of California Press. , and W. PHILLIPS SHIVELY. 1995. Cross-Level Inference. Chicago: University of Chicago Press. ANSOLABEHERE, STEPHEN, ALAN GERBER, and JAMES M. SNYDER, JR. 2002. "Equal Votes, Equal Money:

Court-Ordered Redistricting and the Distribution of Public Expenditures in the American States," American Political Science Review, 96, 767-777'. ASHER, HERBERT B. 1974. "Some Consequences of Measurement Error in Survey Data," American Journal of Political Science, 18 (May), 469^185. BACHRACH, PETER, and MORTON BARATZ. 1962. "The Two Faces of Power," American Political Science

Review, 56 (December), 947-953. BARBER, JAMES D. 1972. The Presidential Character: Predicting Performance in the White House. Upper Saddle River, N.J.: Prentice Hall. BARTELS, LARRY M. 1993. "Messages Received: The Political Impact of Media Exposure," American Political Science Review, 87 (June), 267-285. BARTON, ALLAN H. 1955. "The Concept of Property-Space in Social Research." In Paul F. Lazarsfeld and Morris Rosenberg, eds., The Language of Social Research, pp. 4 0 - 5 3 . New York: Free Press. BATALLA, JAVIER ORTIZ. 1993. "Essays on the Early History of Latin American Central Banking," Ph.D. dissertation in economics, University of California-Los Angeles. BERNSTEIN, RICHARD J. 1978. Restructuring of Social and Political Theory. Philadelphia: University of Pennsylvania Press. BLALOCK, HUBERT M. 1964. Causal Inferences in Nonexperimental Research. Chapel Hill, N.C.: University of North Carolina Press. . 1969. Theory Construction: Prom Verbal to Mathematical Formulations. Upper Saddle River, N.J.: Prentice Hall. . 1979. Social Statistics, rev. 2nd ed. New York: McGraw-Hill. BRUNNER, RONALD D. 1977. "An 'Intentional' Alternative in Public Opinion Research," American Journal of Political Science, 21 (August), 435-464. BUENO DE MESQUITA, BRUCE, JAMES D. MORROW, RANDOLPH M. SIVERSON, and ALASTAIR SMITH. 1999.

"An Institutional Explanation of the Democratic Peace," American Political Science Review, 93 (December), 791-808. BUTLER, DAVID, and DONALD STOKES. 1969. Political Change in Britain. New York: St. Martin's Press. CAMPBELL, ANGUS, and others. 1960. The American Voter. New York: John Wiley & Sons.

170

in Sociological Journals," American Sociologist, 3 (February), 45^16. DUVERGER, MAURICE. 1963. Political Parties. New York: Science Editions. ECKSTEIN, HARRY. 1960. Pressure Group Politics. London: George Allen & Unwin. . 1966. Division and Cohesion in Democracy. Princeton, N.J.: Princeton University Press. EDWARDS, ALLEN L. 1973. Statistical Methods, 3rd ed. New York: Holt, Rinehart & Winston. FELDMAN, STANLEY. 1983. "The Measurement and Meaning of Trust in Government," Political Methodology, 9, 341-354. GEDDES, BARBARA. 2003. Paradigms and Sand Castles: Theory Building and Research Design in Comparative Politics. Ann Arbor: University of Michigan Press. GERBER, ALAN S., and DONALD P. GREEN. 2000. "The Effects of Personal Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment," American Political Science Review, 94 (September), 653-664. GERRING, JOHN. 2007. Case Study Research: Principles and Practices. New York: Cambridge University Press. GILBERT, JOHN P., RICHARD J. LIGHT, and FREDERICK MOSTELLER. 1975. "Assessing Social Innovations: An

Empirical Basis for Policy." In Carl A. Bennett and Arthur A. Lumsdaine, eds., Evaluation and Experiment, pp. 39-193. New York: Academic Press. GOLDSMITH, ARTHUR A. 1987. "Does Political Stability Hinder Economic Development?" Comparative Politics, 19,471^180. GOODMAN, LEO A., and WILLIAM H. KRUSKAL. 1954. "Measures of Association for Cross Classifications,"

Journal of the American Statistical Association, 49 (December), 747-754. GREEN, DONALD, and ALAN GERBER, eds. 2004. American Behavioral Scientists, 48, 1 (January). GREENSTEIN, FRED I. 2000. The Presidential Difference: Leadership and Style from FDR to Clinton. New York: Free Press. HAMMOND, THOMAS H., and JANE M. FRASER. 1984. "Studying Presidential Performance in Congress,"

Political Methodology,

10, 211-244.

HILDEBRAND, DAVID K., JAMES D. LAING, and HOWARD ROSENTHAL. 1977. Analysis of Nominal Data.

Beverly Hills, Calif.: Sage. HOWELL, WILLIAM G., PATRICK J. WOLF, DAVID E. CAMPBELL, and PAUL E. PETERSON, 1966. "Test-Score

Effects of School Vouchers in Dayton, Ohio, New York City, and Washington, D.C.: Evidence from Randomized Field Trials." Paper presented at the annual meetings of the American Political Science Association, Washington, D.C., September 2000. 2002. "School Vouchers and Academic Performance: Results from Three Randomized Field Trials" Journal of Policy Analysis and Management, 21(2), 191-217.

172

Selected Bibliography

Selected Bibliography

173

INTERNATIONAL INSTITUTE FOR STRATEGIC STUDIES. 2006. The Military Balance, 2006. Oxford: Oxford University Press. IYENGAR, SHANTO, and DONALD R. KINDER. 1987. News That Matters. Chicago: University of Chicago Press. JACOB, PHILIP E. 1955. "A Multi-dimensional Classification of Atrocity Stories." In Paul F. Lazarsfeld and Morris Rosenberg, eds., The Language of Social Research, pp. 5 4 - 5 7 . New York: Free Press. KARSTEN, PETER, and others. 1971. "ROTC, Mylai and the Volunteer Army," Foreign Policy, 2 (Spring), 135-161. K E Y , V. O., JR. 1950. Southern Politics. New York: Alfred A. Knopf.

REYNOLDS, H. T. 1977. Analysis of Nominal Data. Beverly Hills, Calif.: Sage. ROBINSON, W. S. 1950. "Ecological Correlations and the Behavior of Individuals," American Sociological Review, 15 (June), 351-357. ROETHLISBERGER, F. J., and W. J. DICKSON. 1939. Management and the Worker. Cambridge, Mass.: Harvard University Press.

KINDER, DONALD R., and THOMAS R. PALFREY, eds. 1993. Experimental Foundations of Political Science.

SHADISH, WILLIAM R., THOMAS D. C O O K , and DONALD T . CAMPBELL. 2002. Experimental and Quasi-

Ann Arbor, Mich.: University of Michigan Press. KING,

GARY,

ROBERT O.

KEOHANE,

and SIDNEY VERBA.

1994. Designing Social Inquiry:

Scientific

Inference in Qualitative Research. Princeton, N.J.: Princeton University Press. KIRK, JEROME, and M A R C L. MILLER. 1985. Reliability and Validity in Qualitative Research. Beverly Hills, Calif.: Sage. KMENTA, JAN. 1997. Elements of Econometrics, 2nd ed. Ann Arbor: University of Michigan Press. KNOKE, DAVID, GEORGE W. BOHRNSTEDT, and ALISSA POTTER M E E . 2002. Statistics for Social Data

Analysis, 4th ed. Belmont, Calif.: Wadsworth. KOENIG, LOUIS W. 1968. The Chief Executive, rev. ed. New York: Harcourt Brace Jovanovich. KOESTLER, ARTHUR. 1969. The Act of Creation. London: Hutchinson. KROEBER, A. L., and CLYDE KLUCKHOHN. 1952. Culture: A Critical Review of Concepts and Definitions. Cambridge, Mass.: The Museum. KUHN, THOMAS S. 1962. The Structure of Scientific Revolutions. Chicago: University of Chicago Press. LAVE, A. CHARLES, and JAMES G. M A R C H . 1975. Introduction to Models in the Social Sciences. New York: Harper & Row. LEITNER, CHRISTOPHER. 1997. "Of Time and partisan Stability Revisited: Australia and New Zealand 1905-90," American Journal of Political Science, 4 1 , 1 1 0 4 - 1 1 2 7 . LUKES, STEVEN. 1974. Power: A Radical View. London: Macmillan. MASSEY, DOUGLAS S„ and KRISTIN E. ESPINOSA.

1997. "What's Driving M e x i c o - U . S . Migration?

A Theoretical, Empirical, and Policy Analysis," American Journal of Sociology, 102 (January), 939-999. MATTHEWS, DONALD R., and JAMES W. PROTHRO.

1963. "Social and Economic Factors and Voter

Registration in the South," American Political Science Review, 57 (March), 24ff. MIHAELESCU, MIHAELA. 2004. "Interethnic Cooperation in Post-Communist Eastern Europe." Ph.D. dissertation in political science, University of Minnesota, expected completion 2005. M O O N , J. DONALD 1975. "The Logic of Political Inquiry: A Synthesis of Opposed Perspectives." In Fred Greenstein and Nelson Polsby, eds., Handbook of Political Science, Vol. 1, pp. 131-228. Reading, Mass.: Addison-Wesley. MOSTELLER, FREDERICK. 1968. "Errors," International Encyclopedia of the Social Sciences. New York: Macmillan. NEUSTADT, RICHARD E. 1991. Presidential Power and the Modern Presidents. New York: Free Press. NIEMI, RICHARD G., and KEITH KREHBIEL. 1984. "The Quality of Surveying Responses About Parents and

the Family: A Longitudinal Analysis," Political Methodology, 10, 193-210. OLSON, MANCUR. 1965. The Logic of Collective Action. Cambridge, Mass.: Harvard University Press. . 1982. The Rise and Decline of Nations. New Haven, Conn.: Yale University Press. OPPENHEIM, FELIX. 1975. "The Language of Political Inquiry: Problems of Clarification." In Fred Greenstein and Nelson Polsby, eds., Handbook of Political Science, Vol. 1, pp. 283-336. Reading, Mass.: Addison-Wesley. PITKIN, HANNA. 1967. The Concept of Representation. Berkeley, Calif.: University of California Press. . 1969. Representation. New York: Atherton. PROTHRO, JAMES W. 1956. "The Nonsense Fight over Scientific Method: A Plea for Peace," Journal of Politics, 18, 565-570. PRZEWORSKI, ADAM. 1975. "Institutionalization of Voting Patterns, or Is Mobilization the Source of Decay?" American Political Science Review, 69 (March), 4 9 - 6 7 . , MICHAEL E. ALVAREZ, JOSE ANTONIO CHEIBUB, and FERNANDO LONGI. 2000. Democracy and

Development: Political Institutions and Well-Being in the World, 1950-1990. New York: Cambridge University Press. PUTNAM, ROBERT D. 1993. Making Democracy Work. Cambridge, Mass.: Harvard University Press.

ROGOWSKI, RONALD, and M A R K ANDREAS KAYSER. 2002. "Majoritarian Electoral Systems and Consumer

Power: Price-Level Evidence from the OECD Countries," American Journal of Political Science, 46, 526-539. SARTORI, GIOVANNI. 1976. Parties and Party Systems. London: Cambridge University Press. Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin. SHIVELY, W. PHILLIPS. 1972. "Party Identification, Party Choice, and Voting Stability: The Weimar Case," American Political Science Review, 66 (December), 1203-1225. . 2006. "Case Selection: Insights from Rethinking Social Inquiry',' Political Analysis, 14, 344-347. SIEGEL, SIDNEY. 1956. Nonparametric Statistics for the Behavioral Sciences. New York: McGraw-Hill. SIMON, JULIAN. 1985. Basic Research Methods in Social Science, 3rd ed. New York: Random House. STOKES, DONALD E. 1969. "Cross-Level Inference as a Game Against Nature." In Joseph L. Bernd, ed., Mathematical Applications in Political Science, Vol. 4, pp. 6 2 - 8 3 . Charlottesville, Va.: The University of Virginia Press. STOUFFER, SAMUEL, and others. 1949. The American Soldier, Vol. 1. Princeton, N.J.: Princeton University Press. TARROW, SIDNEY. 1996. "Making Social Science Work Across Space and Time: A Critical Reflection on Robert Putnam's Making Democracy Work" American Political Science Review, 90 (June), 389-398. . 1971. "The Urban-Rural Cleavage in Political Involvement: The Case of France," American Political Science Review, 65 (June), 341-357. TILLEY, JAMES R. 2003. "Party Identification in Britain: Does Length of Time in the Electorate Affect Strength of Partisanship?" British Journal of Political Science, 33, 332-344. TINBERGEN, NIKO. 1968. Curious Naturalists. Garden City. N.Y.: Doubleday. TINGSTEN, HERBERT. 1937. Political Behavior. London: Longmans. TUFTE, EDWARD R. 1968-1969. "Improving Data Analysis in Political Science," World Politics, 2 1 , 641-654. . 1983. The Visual Display of Quantitative Information. Cheshire, Conn.: Graphics Press. UHLANER, CAROLE JEAN, and KAY LEHMAN SCHLOZMAN. 1986. "Candidate Gender and Congressional

Campaign Receipts," Journal of Politics, 48 (February), 30-50. V A N D E T H , JAN W. 2000. "Interesting but Irrelevant: Social Capital and the Saliency of Politics in Europe," European Journal of Political Science, 115-147. WATSON, JAMES D. 1968. The Double Helix. New York: Atheneum. W E B B , EUGENE J., DONALD T. CAMPBELL, RICHARD D. SCHWARTZ, and L E E SECHRIST. 1966. Unobtrusive

Measures. Chicago: Rand McNally. WILSON, THOMAS P. 1971. "Critique of Ordinal Variables." In Hubert M. Blalock, Jr., ed., Causal Models in the Social Sciences, pp. 415^131. Chicago: Aldine-Atherton. W I N C H , ROBERT E , and DONALD T. CAMPBELL. 1969. "Proof? No. Evidence? Yes. The Significance of

Tests of Significance," American Sociologist, 4 (May), 140-143. ZALLER, JOHN. 1998. "Politicians as Prize Fighters: Electoral Selection and Incumbency Advantage." In John Geer, ed., Politicians and Party Politics. Baltimore: Johns Hopkins University Press.

Index

D

Dahl, Robert, 38, 40 Dalton, Russell J., 20 Dean, Charles W., 164 Dependent variable, 14 Dichotomies, 136-137 in regression analysis, 137-138 Dickson, W. J., 80n Diehl, Paul E, 8 Digeser, Peter, 38 Diminishing marginal returns model, 119-120 Discrete measures, 64 Downs, Anthony, 5-6 Duggan, Thomas J., 164 Duverger, Maurice, 2-3, 7n Abstinence-only sex education, 81 Achen, Christopher H., 47, 87, 96, 128n Acton, Lord, 40 Alvarez, Michael E., 94, 110 Analytic political philosophy, 40 Anomalies, 29-30 Ansolabehere, Stephen, 54-55 Applied research, 4 Asher, Herbert B., 46n, 56 Autocorrelation, 161-162 B

Bachrach, Peter, 38 Baker v. Carr, 54n

Baratz, Morton, 38 Barber, James David, 29 Bartels, Larry M., 131 Barton, Allan H., 40 Batalla, Javier Ortiz, 76 Bernstein, Richard J., 40 Blalock, Hubert M„ 96, 126, 127n, 146n Bohrnstedt, George W., 131,162 Boswell, James, 167 Brunner, Ronald D., 31 Bueno de Mesquita, Bruce, 26-27 Butler, David, 20 174

Campbell, Angus, 18-19 Campbell, David E„ 87 Campbell, Donald T., 56, 85n, 96, 164 Carter, Lewis F., 73 Case studies, 103 Causal interpretations, 77-78 eliminating alternative, 78-84 Causation, 14, 74-76 Censored data, 103-106 Cheibub, Jose Antonio, 94, 110 Cluster sample, 101 Coefficient of determination, 124 Coleman, James S., 167-168 Continuous measures, 63-64 Control group, 80, 82 designs without, 79-81 Controlling for a variable, 94-95 Converse, Philip E., 17-20, 47, 56, 145 Cook, Thomas D„ 85, 96 Correlational measures of relationship, 113 Correlation analysis, 122-127 compared with regression analysis, 127-130 Correlation coefficient, 123 Cross-tabulation, 132-134

E

Eckstein, Harry, 40, 57 Ecological fallacy, 128n Effect-descriptive measures of relationship, 113 Elegant research, 16-17 Empirical research, 4 Engineering research, 5, 15, 23 English language, 32-33, 34 Epistemic correlation, 45n Espinosa, Kristin E., 16 Ethics of research, 11-12 Experiment, true, 83-84 Ex post facto argument, 23 Extrapolation, 49 F Face validity, 52 Feldman, Stanley, 56 Formal theory, 5-8 Fraser, Jane M., 56 Free rider problem, 7 G

Galileo, 167-168 Geddes, Barbara, 104-105

Gerber, Alan S., 54-55, 87, 96 Gerring, John, 110 Gilbert, John P., 87 Goldsmith, Arthur A., 30 Goodman, Leo A., 135n Goodman and Kruskal gamma, 135 Green, Donald P., 87, 96 Greenstein, Fred I., 29 H Hammond, Thomas H., 56 Hawthorne study, 80n Hill, G. Birkbeck, 167n Holding a variable constant, 76, 94-95 Howell, William G„ 87 Hypothesis testing, 151-153 I

Independence of observations, 161-162 Independent variable, 14 Interaction, statistical, 145-146, 146n Intercept of regression equation, 116 Interesting problems, 10 Interrupted time series, 90-91 Interval measures, 62 Iyengar, Shanto, 88 J Jacob, Philip E., 40 Johnson, Samuel, 167 K

Karsten, Peter, 69 Kayser, Mark Andreas, 86 Kemeny, John, 19 Keohane, Robert O., 110, 161n Key, V. O., Jr., 121-122 Kinder, Donald R., 87n, 88, 96

175

176

Index

Index

King, Gary, 110, 161n

Multidimensional concepts. 32

Putnam, Robert D., 92-93

Kingston, Jean, 8

Multidimensional words, 34-35

Puzzles, 25-28

Kirk, Jerome, 56

Multivariate regression, 141-146

Selection on the dependent variable 106-108 Shadish, William R., 96

Kluckhohn, Clyde, 38

Shively, W. Phillips, 20, 110n, I28n

Q

Kmenta, Jan, 162 Knoke, David, 131, 162

N

Siegel, Sidney, 165 Significance, statistical, 149

Quantitative research, 20-21

Koenig, Louis W., 29

Significance tests, 162-164

Koestler, Arthur, 31

N (number of cases), 158-161

Simon, Julian, 165

Krehbiel, Keith, 56

National Election Study, 101

Simplicity, of a theory, 15

Kroeber, A. L . , 38

National integration, 37

Kruskal, William H., 135n

N A T O , 25

Kuhn, Thomas S . , 168

Natural experiment, 82-83

R

Slope of regression equation, 116

r, 123-127

Social research, 2 - 3

Snyder, James M., 54-55

2

r , 124, 126-127

Social status, 4 1 - 4 2

Neustadt, Richard E . , 29

Random error, 49, 53-54, 130-131

Split-half test for reliability, 47

without premeasurement, 83 Niemi, Richard G . , 56

Random sample, 99-101

Stanley, Julian C , 85n

Nominal measures, 62

Ratio measures, 63n, 73

Statistics, 112

Lave, Charles A . , 26n

Nonempirical research, 4

Rational choice, 6

Stokes, Donald E . , 20, 128n

L a w of Large Numbers, 58, 61,

Nonrandom error, 49, 53-54

R D D sampling, 101

Stouffer, Samuel, 9

Normative philosophy, 4 - 5

Reagan, Ronald, 79

Stratified sample, 60

Null hypothesis, 152-153

Recreational research, 4, 5-8

L

100, 106 Leitner, Christopher, 20

Regression analysis, 113-122

Liberal, 33

compared with correlation,

Light, Richard J . , 87 Linear relationships, 71,

o

118-119

T

127-130 and measurement error, 130-131

Tarrow, Sidney, 29, 51, 52 Tsst group, 82

Literary Digest, 50-51

Olson, Mancur, 6-7, 26n, 30

Regression coefficient, 117

Loewenberg, Gerhard, 109-110

Oppenheim, Felix, 40

Regression to the mean, 88-90

Test-retest check for reliability, 47

Relative deprivation, 9

Theory, 2, 3, 14-17

Reliability of measures, 45^18

Theory-oriented research, 8, 15, 23

Ordinal measures, 62

Logit analysis, 139-141 Longi, Fernando, 94, 110

related to validity, 49-50

Lukes, Steven, 38

Tilley, James R., 20

P

Research design, 24, 78-84

Tinbergen, Niko, 96

Residuals, in regression, 119-122

Tingsten, Herbert, 96

Robinson, W. S . , 128n

Topics for research, 22-23

M

Palfrey, Thomas R„ 87n, 96 Peterson, Paul E . , 87

Roethlisberger, F. J . , 80n

Trivial theory, 15

March, James G . , 26n

Pierce, Roy, 56

Rogowski, Ronald, 86

True experiment, 83-84

Markov chain, 19

Pilot study, 5 1 - 5 2

Marx, K a r l , 8

Pitkin, Hanna, 40

Massey, Douglas S . , 16

Pluralism, 7

Matthews, Donald R., 144n

Political party, 33

Measurement, general problem of,

Polling, 163

42^15

Power, 38-40

Measurement error,

130-131

Mee, A l i s s a Potter, 131, 162 Mihaelescu, Mihaela, 99, 104

Precision, 58-67 and describing relationships between variables, 65-67

Tufte, Edward R., 3 1 , 7 3

s

u

Sampling, 99-102 purposive, 102 quasi-random, 101-102

Sampling distribution, 155-158

enrichment of, 67-71 in measurement, 62-67

Sartori, Giovanni, 40

in measures, 58-61

Scattergram, 113-114

Tribune,

81

analysis, 117

R D D , 101

Miller, Marc L . , 56 Star

Units, comparability in regression

random, 99-101

Mill, John Stuart, 5 Minneapolis

Uhlaner, Carole Jean, 86 Unidimensional concepts, 32

Moon, J. Donald, 31

Probit analysis, 139-141

Schlozman, K a y Lehman, 86

Mosteller, Frederick, 56, 87

Prothro, James W., 144n

Schwartz, Richard D., 56

Multicollinearity, 161 n

Przeworski, Adam, 20, 94, 110

Sechrist, L e e , 56

V Validity, 48^19, 130-131 checking for, 51-53 Variables, 14

111

178

Index

Variance, 124-127

X

Verba, Sidney, 110, 16In X\

153-155

w Watson, James D., 1 Webb, Eugene J . , 56 Weldon, Steven, 20 Wilson, Thomas P., 73 Winch, Robert E, 164 Wolf, Patrick J . , 87

Z a l l

e r , John, 98

Zinsser, William Knowlton, 31