The Survey Kit, 2nd edition, How to Assess and Interpret Survey Psychometrics 8

MarkS. Litwin

How to Assess and nterpret Survey Psychometr·cs

),723

THE SURVEY KIT, Second Edition Purposes: The purposes of this to-volume Kit are to enable readers to prepare and conduct surveys and to help reader) become better users of survey results. Surveys are conducted to collect information; surveyors ask questions of people on the telephone, face-to-face, and by mai1. The ques­ tions can be about attitudes, beliefs, and behavior as well as socioeco­ nomic and health status. To do a good sUIvey, one must know how to plan and budget fo r all survey tasks, how to ask questions, how to design the survey (research) project, how to sample respondents, how to collect reli­ able and valid information, and how to analyze and report the results. Users: The Kit is for students in undergraduate and graduate classes in the social and health sdences and for individuals in the public and pri vate sectors who are responsible for conducting and using surveys. Its primary goal is to enable users to prepare surveys and collect data that are accura te and useful for primarily practical purposes. Sometimes, these practical pur­ poses overlap with the ob jectives of scientific research, and so survey researchers will also find the Ki t useful. Format of the Kit: All books in the series contain instructional ob jec­ tives, exercises and answers, examples of surveys in use and il lustrations of survey questions, guidelines for action, checklists of dos and don'ts, and annotated references. Volumes in The Survey Kit: 1. The Survey Handbook, 2nd Arlene Fink

2. How to Ask Survey Questions, 2nd Arlme Fink

3. How to Conduct Self-Adm:inistered and Mail Surveys, 2nd Linda B. Bourque and Elle P. Fielder

4. How to Conduct Telephone Surveys, 2nd Linda B. Bourque and Eve P. Fielder S. How to Conduct In-Person Interviews for Surveys, 2nd Sabine Mertells Oishi

6. How to Design Survey Studies, 2nd Arlene Fink

7. How to Sample in Surveys, 2nd Arlene Fillk

8. How to Assess and Interpret Survey Psychometrics. 2nd Mark S. Litwill

~

Mark S. Litwin

C\J

How to Assess and Interpret Survey

Psychometrics

-~

~

(l)

2nd edition

::>

L

~



CfJ

Q)

~

8

9. How to Manage, Analyze, and Interpret Survey Data, 2nd Arle/1e Fink

10. How to Report on Surveys, 2nd ·1rlt/lff! Flllk

't ~!~~/~JJ~1~~~9~3essJonal ~ Thousand Oaks •

London • New Delhi

Publisher

l

opyright © 2003 by Sage PubUcations, Inc. All rights reserved. No part of this book may be reproduced o r utilized in any foml or by any means, electronic o r mechanical, including photo­ copying, record ing, or by any information storage and retrieval system, without oennission in writing from the publisher.

,

For il/formatiol/:

Sage Publica tions, Ine. 2455 Teller Road

Thousand Oaks, California 91320

~ -mail: [email protected]

Sage Publications Ltd.

6 Bonbill Street

London EC2A 4PU

United Kingdom

Sage Publications India Pvt. Ltd.

M-32 Market

Greater Kailash I

New Delhi l l 0048 India

Printed in tl1e United States of America

Library of Congress Cataloging-in-Publication Data The survey kit.- 2nd ed. p. cm. Includes bibliographicaJ references. ISBN 0-76 19-25 10-4 (set : pbk.) 1. Social surveys . 2. Hea lth surveys. L Fink, Arlene.

HN29 .S724 2002

300'. 723-dc21 20020 12405

This book is printed on acid-free paper. 02 03 04 05 10 9 8 7 6 5 4 3 2 1

Acqllisitions Editor: Editorial Assistant: ;op>, Editor: Production Editor:

Typesetter: Proofreader: 'over Desigller: Prodllction Designer:

C. Deborah Laughton

Veronica Novak Judy Selhorst Diane S. f-osler Bramble Books Cheryl Rivard Ravi BaJasuriya Michelle Lee

~ Contents How to Assess and Interpret Survey Psychometrics:

Learning Objectives 1. Psychometrics 2. Reliability Types of Reliability Test-Retest Reliability Alternate-Form Reliability Internal Consistency Reliability Interobserver Reliability 3. Validlty Types of Validity Face Validity Content Validity Criterion Validity Construct Validity 4. Scaling and Scoring 5. Creating and Using a Codebook 6. Pilot Testing Checklist for Pilot Testing 7. Multicultural Issues 8. Translations Exercises Answers

Suggested Readings Glossary About the Author

vi

1

5

7

7

13

20

26

31

32

32

33

34

41

45

51

57

67

69

72

75

78

81

83

87

vi

THE SURVEY HANDBOOK

How to Assess and Interpret

Survey Psychometrics: Learning Objectives

This book is intendt>d to guide you in assessing and interpreting the quality of collected survey data by thor­ oughly examining the survey instrument used. It also addresses important considerations in the coding and pilot testing of surveys. The specific objectives are as fol­ lows:

~ PsychoDletrics

• Select and apply reliability criteria, including the following: -

Test-retest reliability

Alternate-form reliability

Internal consistency reliability

Interobserver reliability

lntraobserver reliability

• Select and apply validity criteria, including the fol­ lowing: -

Face validity

Content validity

Criterion validity

Constmct validity

• Outline the fundamental principles of scaling and scoring • Create and use a codebook for survey data • Pilot-test new and established surveys • Address cross-cultural issues in survey research

A

successful data collection survey is much more than a set of well-designed questions that are written down and administered to a sample population. There are good surveys and bad surveys. Bad surveys produce bad data- th at is, data that are unreliable, irreproduCible, or invalid, or that waste resources. Good surveys, on the other hand, can yield critical information and provide important windows in to the heart of the topic of interest. Psychometrics is the branch of survey research that enables you to determine how good your survey is. Psychometric principles were originally developed in the field of meaSUIement psychology, but they have been bor­ rowed by many other disciplines. Psychometrics provides survey researchers with a way to quantify the precision of the measurement of qualitative concepts, such as consumer satisfaction, depreSSion, and marital interaction. These phe­ nomena are inherently more difficult to measure than are discrete variables such as a person's zip code, marital status, or height. Example 1.1 presents two surveys of hotel guest satisfaction. It is not obvious whether one or the other is bet­ ter, but the two are clearly different and will yield different data . The aim of this book is to teach you how to evaluate differences in surveys such as these. 1

2

HOW TO ASSESS A!'ffi INTERPRE"I SURVEY PSYCHOMETRICS

EXAMPLE 1.1 Two Surveys of Hotel Guest Satisfaction Purpose of tile slIrvq: To assess guest satisfaction at a downtown hotel with a 3-item survey at the tIme of checkout. SIII1'CY J:

Circle one number for each item.

1. Did you enjoy your stay?

Yes .. .. . .. .. 1 No ...... .. 2 2. How good was the service at the hotel? Good ....... 1

Poor ....... 2

3. Would you stay here again? Yes .. . .. . .. 1 NO ...... . .. 2

Survey 2: Circle Ofle Plumber for each item. 1. Overall. considering the service, food, and

aJl other aspects of our hotel, how would

you describe your stay here?

Very enjoyable .......... .... ...... 1

Somewhat enjoyable .... . . ......... 2 Neither enjoyable nor unenjoyable .... 3 Somewhat unenjoyable . .... . . ...... 4 Very unenjoyable .... .. . .. ......... 5 2. How would you describe our service. More efficient than other hotels

I have staved in .. ............ 1

l'~ yel /Owe tries

f:..tlllllplt!

3

1. J (Oil/iII/it'd

Equally efficient to other hotels I have stayed in ...... .. ...... .... Less effitient than otller hotels I have stayed in .. ...... .. . .. . 3 3. How likely are YOll to stay here again. Highly likely ....... 1

I.ikely . ........... . 2

Not sllre .. .. . .. . .. . 3

Unlikely . . .. . .. .... 4

Highly unlikely ..... 5

Which of the two surveys in Example 1.1 is better? Survey 2 appears as thollgh it might produce more useful information because it provides more than just the yes/no­ type qu estions we see in Survey 1. But perhaps the simplicity of Survey 1 will encourage more hotel guests to complete it. Perhaps only the guests who really hated their stays will take the time to complete Survey 2, causing results that overlook the guests who reaIJy enjoyed their stays. Perhaps business guests are the hotel's most important customers, but because they are in a greater rush than pleasure travelers to catch their flights, they are much less likely to take the time at checkout to complete Survey 2. So we really don't know quantitatively which survey is better. What exactly do we mean by better? The better survey will more accurately mea­ sure guest satisfaction, producing more useful data from which to draw conclusions about the hotel's performance. Strictly speaking, it is difficult to assess the quality of the data we colJect. It is easier to assess the accuracy of the sur­ vey instrument that is used to collect those data. This assess­ ment consists primarily of looking at the reliability and the validity of the survey instrument. Example 1.2 demonstrates that different tools used to measure electrical resistance may produce completely different results . The only way to deter­

4

HOW TO ASSESS AND INfERf'RET SURVEY PSYCHOMETRICS

mine which, if either, is correct is by looking directly at the accuracy of the measurement tools and at potential sources of measurement error, such as response variation and non­ response. The resistance meters in ExampJe 1.2 demonstrate an important concept. Before we can use a survey instrument to collect meaningful data, we must test it to ensure its accu­ racy. This is true regardless of whether we are dealing with resistance meters, guest satisfaction questionnaires, market­ ing surveys, crime assessments, depression scales, or any other survey instrument. What matters is not how quantita­ tive the data are but how well the survey instrument per­ forms. In Chapters 2 and 3 of this book, which address the reliability and the validity of survey instruments, this idea is explored in greater detail.

EXAMPLE 1.2 Resistance Meters Two licensed electridans lise differen t resistance meters

to measure the ohms in four brand-new drcuits during a

new product anaJysis. Pat uses the old meter she has

been using for the past 15 years. Jerry uses a new model

that he just bought from HomeLine, Inc., a reputable

mall-order company. After they take their measure­

ments, Pat's data are 6, 16, 38, and 119 ohms, and Jerry's

data are 19, 28, 73, and 184 ohms. Since there is no way

to determine which, if either, data set is correct, we must

assess lhc accuracy of the resistance meters themselves.

We do this by asking both Pat and Jerry to measure the

resistance in a series of quality-control cirCUits in which

the ohms are known with certalnty in advance. After Pat

and Jerry measure the resistance three different times in

the quaUty-controJ ciIcuits, we find Pat's meter to be

more accurate. Therefore, we use her data in the new

product anaJysis of the four new circulls.

~

Reliability

I

n any set of data we collect, there will be some amount of error. Naturally, we want to rnlnimize this error so that the data provide as accurate a reflection of the truth as possible. In survey research, error comprises two components: ra n­ dom error and measuremen t error. Random error is the unpredictable error that occurs in all research. It may be caused by many different factors, but it is affected primarily by sampling techniques. To lower the chance of random error, we could select a larger and more representative sam­ ple. Collecting a larger sample will increase the cost of the study, however, so it is often neither practical nor feasible. Instead, we could use statistics to calculate the probability that a particular result is due to random error. If that proba­ bility falls below the limit we set, then we I/reject the null hypothesis" and draw inferences about our population. (Recall that in statistical analysis, we make a conservative assumption, called the 11ull hypothesis, that our two groups of interest do not differ in the particular variable we are study­ ing. For instance, if we were comparing men's and women's responses to the surveys in Example 1.1 concerning hotel guest satisfaction, the null hypothesis would be that there is 5

6

HOW TO ASSESS AND INTERPREl' SURVEY PSYCHOMETRlCS

no difference. We would design the survey research to test that null hypothesis and see if we could reasonably reject it, thus allowing us to draw inferences about differences between male and female hotel guests. (For more informa­ tion on hypothesis testing, see How to Manage, Analyze, and Interpret Survey Data, Volume 9 in this series.) Measurement error refers to how well or poorly a partic­ ular instrument performs in a given population. No instru­ ment is perfect, so we always expect at least some error to occur during the measurement process. For example, a stop­ watch with a minute hand but no second hand cannot meas­ ure runners' times in a 5-kilometer race to the nearest second; the best it can do is count minutes. Differences of less than 60 seconds in the runners' times will be lost in the measurement error of the stopwatch . The smaller the mea­ surement error, the closer our data are to the truth. However, even when random error is though t to be zero, some mea­ surement error will occur. This measurement error reflects the precision (or lack of precision) of the survey instrument itself. Reliability is a statistical measure of the reproducibility or stability of the data gathered by the survey instrument. Example 2.1 illustrates the principle of reliability with a comparison of two different tools used to measure fabric. In the example, the meta l yardsticks used at Honor Guard are more reliable and have less measurement error than the rub­ ber tape measures used at Notso. Thus Honor Guard's results are a more accurate reflection of the truth. Even so, there may still be some degree of measurement error. If the sales­ person at Honor Guard doesn't see very well and cannot read the difference between 36 inches and 35 inches, then Lane may still get shorted. Neither the metal yardsticks nor the rubber tape measures are perfectly reliable, and there is always some possibility of measurement error.

7

Rd/avility

EXAMPLE 2 .1

Fabric Measurcnlcnt

Honor Guard FilhriL Company sells its fabric by the yard and uses expensive aluminum yardsticks to quantify the amount of each purchase. Notso Fabrics also sells its fab­ ric by the yard but Llses tape measures made from inex­ pensive thin rubber strips to measure its quantities. Lane needs exactly 3 yards of bl ue cotton fabric to make a set of bumpers for his twins' cribs. At Notso, Lane has found over the years lhat the lengths of measured fabric are inconsisten t. Because the rubber tape measures can be stretched to varying degrees, depending on the strength of the salesperson, a "yard" of fabric sold at Notso may actually be anywhere from 35 inches to 41 inches long. At Honor Guard, all measured yards of fabric are always very nearly the same length; the measurements vary only with the visual acuity of the salesperson. Lane goes to Honor Guarc.l for his purchase, because he knows that there, a yard is always a yard.

Types of Reliability No survey instrument or test is perfectly reliable, but some are clearly more reliable than others. When evaluating the value of a data set, we begin by looking at the reliability characteristics of the measurement instrument. Reliability is commonly assessed in three forms: test-retest, alternate­ form, and inlernal consistency. Intraobserver and interob­ server reliability are also addressed below. TEST-RETEST RELIABILITY Test-retest reliability is the most commonly used indica­ tor of survey instrument reliability. It is a measure of how

8

HOW

ro ASSESS AND INTERPRFT SURVEY pSYC HOMETRICS

reproducible a set of results is. Survey researchers test a sur­ vey instrument by having the same respondents complete the instrument at two different points in time to see how sta­ ble their responses are. They then calculate correlation coef­ ficients, or r values, to compare the two sets of responses. (See How to Manage, Analyze, and Interpret Survey Data Volume 9 in this series.) These correlation coefficients are collectively referred to as the survey instrument's test-retest reliability. In general, the correlation coefficients are consid­ ered good if they are at least 0.70. This implies that the sur­ vey responses are reasonably consistent from one pOint in time to another. Sometimes data are not collected from a group of subjects but are recorded by one observer. In such a case, the survey researcher assesses test-retest reliability by having that indi­ vidual make two separate measurements. The researcher then compares the two data sets from the same observer. The correlation between two data sets from the same individual is common1y known as intraobserver reliability. This mea­ sure of the stability of responses from the same respondent is a form of test-retest reliability. In Example 2.2, the measurement of test-retest reliability Is demonstrated in a survey item that asks about crime rates in different cities. Example 2.3 presents a situation where test-retest reliability is low in a survey item that asks about education attitudes in the same respondents as those in Example 2.2.

EXAMPLE 2.2

Test-Retest Reliability: Crime Rates

Adam has designed a new survey instrument to measure grand larceny rates in a group of urban centers at high risk for various types of crime. One of the items on the survey asks the deputy chief of police in eadl urban cen­ ter whether grand larceny rates have been mild, moder-

9 Reliability

I~t"'IIf1/('

2.2 cUl/till/lcd

ate, or severe during the past month. In order to gauge whether respondents' answers to this item are consistent over lime, Adam administers the survey instrument once to a sample of SO deputy police chiefs in different urban centers and records the data. He then administers the identical survey instrument a second time, 4 weeks later, to the same SO deputy police chiefs and records those data. Because actual grand larceny rates tend to be stable over short periods of time, Adam expects that any differences in the survey responses will rcllect measure­ ment error of the survey instrument and not actual changes in the crime rales. When Adam compares the sets of data from the two different time points, be finds Lhat the correlation coefficient is 0.89. He knows that this item produces responses that are stable over moder­ ate time periods, because the correlation coefficient exceeds 0.70. Therefore, his item has good test-retest reliability.

EXAMPLE 2.3 Test-Retest Reliability: Education Attitudes Adam also wants to assess attitudes in the same deputy police chiefs to find out whether their police school cur­ ricula adequately prepared them to avoid professional bumout, a common problem among law enforcement officers. He designs a 3-item survey that asks how exten­ sive their poHce training in avoiding professional burnoul was, how relevant those lessons are at present, and how easy it is to put those techniques into practice. He administers the same 3-item attitude survey at two time points 4 weeks apart but finds that the correlation

11 H OW TO ASSESS AND JNT£illP RET SURVEY r SYC HOM ETRlCS

LO

EXlIl1Ijlh'

Reliability

EX,llllple 2.-1 Cll1Itil/llcd

2.3 CIIlltilllWd

coefficients for the items arc only 0.32, 0.12. and 0.17 respectively. Adam thus concludes that this survey does not produce responses that are stable over lime; he real­ izes that the deputy police chiefs' attitudes about their training may be subject to recall bias and are likely to change from week to week. Variolls factors may be int1u­ encing the changing responses. Perhaps the education attitudes are not dependent only on the quality of the education. Perhaps the attitudes are more related to salary or to how surly the boss is acting on any given day. Perhaps there are other factors in£luencing educa­ tion attitudes in this population. Because Adam does not know what these other factors are, he cannot control for them. He is forced to drop these items because their test­ retest reliability is too low.

You can calculate test-retest reliability not only for single items, but also for groups of items. In fact, test-retest reliabil­ ity is most often reported fo r entire survey instruments or for scales (more on scales in Chapter 4) within survey instru­ ments. Example 2.4 demonstrates test-retest reliability in a series of written items that are administered together to measure pain in men undergoing hernia surgery.

EXAMPLE 2.4

Test-Retest Reliability: A Pain Scale

Jackie wants to measure postoperative pain in a group of 12 adult men undergoing hernia surgery. She deSigns a series of 4 separate items to assess tile pain in terms of its intensity, quality, impact, and duration over the past 10 minutes, each rated as a number from 1 to 10. She cre-

tltes a score for the 4-item scale by summing the

responses to the items. She administers the survey instrument 2 hours after surgery (Time 1) and again -l hours after surgery (Time 2), t\"JO points when pain lev­ els should be similar. Sh e then compares the two sets of pain scale scores by calculating a correlation coefficient and finds it to be 0.79 . She concludes that her pain scale has good test-retest reliability during th e immediate postoperative period in this population. When measuring test-retest reliability, you must be care­ ful not to select items or scales that measure variables that are likely to change over short periods of time. Variables th at are likely to change over a given period of time will produce low test-retest reliability in measurement instruments. This does not mean that the survey instrument is performing poorly; it simply means that the attribute itself has changed. Example 2.5 illustrates how to select an appropriate time interval over which to assess test-retest reliability.

EXAMPLE 2 .5

Test-Retest Reliability: An xiety

Len uses a we ll-known short survey instrument to mea­ sure anxiety in a group of coUege students before and after midterm examinations. He administers the items 4 days prior to the exams (Time 1), 3 days prior to the exams (Time 2), and 2 days after the exams (Time 3), He calculates the correla tion coefficient between Times 1 and 2 to be 0.84, the correlation coefficient between Times 1 and 3 to be 0.12, and the correlation coefficient between Times 2 and 3 to be 0.09. The correct test-retest reliability figure for Len to report is 0.84. This reflects

12

HOW TO ASSESS AND INTERPIUT SURVEY PSYCHOMETRICS

Exam/lit' 2.5 contillul'd

thl? stability of the responses to the survev instrumenl during a period when L.en would expect the responses to remain fairly stable. Len deduces that the other two fig­ ures represent true changes in anxiety levels and not poor test-retest reliability. Based on hiS reliable survey instnunent, he concludes that anXiety levels go down after exams. Len's colleague, Bryan, designs his own new survey instmment to measure anxiety in the same group of col­ lege students at the same time PO i11tS. His correlation coefficients are 0.34, 0.43, and 0.50, respectively. Bryan canuot make any sound deductions about anXiety levels because his suryey instrument does not have good test­ retest reliability.

13

Re/ia/Jilit"

ALTERNATE-FORM REUABILITY

Alternate-form reliability provides one way to escape the problem of the practice effect. It involves using differ­ ently worded items to measure the same attributes. Questions and responses are reworded or their order is changed so that items are similar but not iden tical. When you are testing reliability using this method, you must be careful to create items that address the exact same aspect of behavior and also use the same vocabulary level and have the same level of difficulty. The items must differ only in their wording. You then administer the items or scales to the same population at different time points and calculate the correla tion coefficients. If these are high, the survey instru­ ment or item is said to have good alternate-foml reliability. One common way to test alternate-form reliability is simply to change the order of the response set. Example 2.6 shows two differently ordered response sets for the same item ques­ tion. Either response set can be used without changing the meaning of the question.

EXAMPLE 2 .6

You can certainly measure characteristics that tend to change over time. In fact, this is often the purpose of survey research. But test-retest reliability must be documented over shorter periods to decrease the degree of measurement error attributable to the survey itself. When measuring test-retest reliability, you must also consider that individuals may become familiar with the iteIl15 and so may answer partly based on their memory of what they answered the previous time. This phenomenon, called the practice effect, presents a challenging problem that must be addressed in measures of test-retest reliability over short periods of time. A5 a result of the practice effect, test-retest reliability figures can be falsely inflated.

Alternate-Form Reliability:

Response Sets for Depression

The following are equivalent but differently ordered response sets to single items on depression. For each of these three items, the two response sets differ only in their sequence. They are good items for this method of measuring alternate-form reliability. Item 1: Circle une /lumber ill each respUlIse set Version A

How often during the past 4- weeks have YOll felt sad and blue?

14

HOW TO ASSESS AND INTERPRET SU1{VEY PSYCHOMETRICS

E)',(Implc 2.6 (cmtill//('c/

15

Rt'110biii t}"

tixll/"plt!~.6

Version A

All of the time . .. . .. . 1

Most of the time .. .. .. 2

Some of the time ... .. . 3

Occasionally .. ...... . 4

Never............... 5

During the past 4 weeks, have you felt like hurting your­ self? Yes ......••• 1

No ......... 2

Version B How often during the past 4: weeks have blue?

conlil/llfti

Version B YOli

felt sad and

Never. ..... .... ..... 1

Occasionally .. . . ... .. 2

Some of the time .. .. .. 3

Most of the time ..... . 4:

All of the time ........ 5

Item 2: Circle one Hlimber i11 each response set. Version A During the past 4 weeks, I have felt downhearted Every day . . . 1

Some days ... 2

Never ...... 3

During the past 4 weeks, have YOLI felt like hurting your­ self? No ........ . 1

Yes ... . ..... 2

Changing the order of the response sets is most effective when the two time points for administration of the items are close together. This approach forces respondents to read the items and response sets very carefully, thereby decreasing the practice effect. Another way to test alternate-form reliability is to change the wording of the response sets without chang­ ing the meaning. Example 2. 7 presents two items on urinary function, each with two different response sets that collect the same information with different but synonymous wording.

Version B During the past 4 weeks, r have felt downhearted Never . .. .. . .l

Some days ... 2 Every day ... 3

Item 3: Circle olle Illlmber in eac" re!lpOl1Se set.

EXAMPLE 2.7

Alternate-Form Reliability:

Response Set Wording

I'hc following are equival~nt but differently worded response sets to two singJe items on urinary function . The response sets for each item are differently worded

J6

HOW TO ASSESS AND INTERPRET SUR\EY PSYCHOMETRICS

f:.vlimJllt· 2.7 colltilluetf

17

Relia/;i/lty

E.'
but functionally equivalent. This makes them good can­ didates for use in a test at alLernate-foffil reliability.

Item 1: Circle onc: Illl1llber in each respollse set. Version A During the past week, how often did you usually empty your bladder?

1 to 2 times per day ....... . .. .. 1 3 to 4 times per day .... ..... . .. 2 5 to 8 times per day .. .. . .... ... 3 12 times per day . ... .. ........ 4 More than 12 times per day .. . .. 5 Version B During tile past week, how often did you usually empt'

YOllr bladder?

Every 12 to 24 hours ..... . 1

Every 6 to 8 hours .. .. .... 2

Every 3 to 5 hours .. ...... 3

Every 2 hours .......... . 4

More than every 2 hours .. 5

Version B During the past -1 weeks, how much did you leak urine?

I used no pads in my underwear ... 1 I used 1 pad per day. . . . . . . . . . ... 2 I used 2 to 3 pads per day . .... ... 3 I used 4 to 6 pads per day .. ... ... 4 I used 7 or more pads per day ... .. . 5 Another common method used to test alternate-form reliability is to change the act ual wording of the items them­ selves. Again, if you use this method, you m ust be very care­ ful to design items that are truly equivalen t to each other. Items that are worded with different degrees of diffi culty do not measure the same attribute. Th ey are m ore likely to measure the reading comprehension or cognitive function of the respondent. Exam ple 2.8 contains two questions that are not equivalen t even though they are about the same topic. The items in this example would not provide a good test of alterna te-form reliability. Despite t he same response sets, these two versions are not equivalent and therefore cannot be used to test alternate-form reliability.

Item 2: Circle ulle /lumber in each respollse set.

EXAMPLE 2.8 Version A During the past 4 weeks, how much did you lea/.. urine? Never .. .. ...... ... ..... 1

A little bit . .. ... ..... . .. 2

A moderate amqunt . .. . .. 3

A lot ..... .... ......... 4

Constantly .... ......... 5

Nonequivalent Item Rewording The following are di fferently worded versions of the same Item intended to measure assertiveness at work. If you look closely at the wording, you can see that the two versions do ask basically the same question. In fact, the response sets are identical. But the two items are clearly not equivalent because their vocabulary levels are

18

HOW TO ASSESS AND lNfERPRET SURVEY PSYCHOMETRICS

£!.xumph' 2.B ull/lirlllL'd

profoundl}, different. They would Iwt be good items \,,,ith which to test alternate-form reliability. The first is a measure of assertiveness, whereas the. second is as much a measure of reading comprehension as of assertiveness. Item 1: Circle olle 11IImber.

When your boss blames you for something you did not do, how often do you .stick lip for yourself? All ofthe ti m .. . . . . . 1

Some of the time .... 2

None of the time .. . . 3

Item 2: Circle one number. When presented with difficult professional situations in which a superior censures you for an act for which you are not responsible, how frequently do you respond in an assertive way? All of the tim . .. .... 1

Some of the time .... 2

None of the time .... 3

Example 2.9 displays two questions on the same subject that are also differently worded, but these two are equiva­ lent and can be used to test alternate-form reliability. Notice that even though the response sets are different, the items are much closer in style and reading level than those in Example 2.B. When testing alternate-form reliability, you can adminis­ ter the different forms at separate time points to the same population, or if the sample is large enough, you can djvide it in half and administer each of the two alternate forms to half of the group. You would then compare the results from the two halves. This technique, called the split-halves

19

Rei i tI/) iI it)'

EXAMPLE 2.9

Equivalent ItCDl Rewording: Depression

n1e following are equivalent but differently worded items that measure loneliness. Notice that although the items are worded differently and have different response sets, they ask about the exact same issue in very similar ways. A high correlation coefficient, or r value, between the responses to these two items would indicate that the item has good alternate-form reliability. Item 1: Circle VIle Illllnber.

How often in the past month have you felt all alone in the world? Every day .. ..... . 1

Some days ........ 2

Occasionally ... .. . 3

Never .... . .. . .. . 4­ Item 2: Circle one 1lumber.

During the past 4 weeks, how often have you felt a sense of loneliness? All of the time ..... 1

Sometimes ....... 2

From time to time.. 3

Never .. ......... 4:

method, is generally accepted as being as good as adminis­ tering the different forms to the same sample at different time paints. When you use the split-halves method, you must make sure that you select the halves of the sample ran­ domly. You must also measure the sociodemographic charac­ teristics of both halves of the sample to make sure there are no group differences that might account for any disparities

22

HOW TO ASSESS AND INI£RPRET SUIWI.:Y PSYCHOMETRICS

£.wIIII'/I! 2.10 Ltmli/lll('d

It is easy to see how this series of items provides much more information on physical function than a single itemsucil as the following: Circle 01ll.' '/limber.

How .Ilrnil",d are YOII

ill

Y(lur dar·

to-day ph) ~Jcal activities?

I.Im\led a 1.01

Limite,", 3 Lillii.'

Not limited

I

2

:;

at All

j I

The measurement of internal consistency involves the calculation of a statistic known as Cronbach's coefficient alpha, named for the 20th-century psychometrician who first reported it in 1951. Coefficient alpha measures internal consistency reliability among a group of items that are com­ bined to form a single scale. It is a statistic that reflects the homogeneity of the scale. That is, it is an indication of how well the different items complement each other in their measurement of different aspects of the same variable or quality. The formula for calculating coefficient alpha can be found in any textbook of test theory or psychometric statis­ tics (see also How to Manage, Analyze, and Interpret Survey Data, Volume 9 in this series). Example 2.11 provides a demonstration of coefficient alpha calculation. For simplicity, the example contains a scale with three yes/no items, but coefficient alpha can also be calculated for longer scales containing items with more than two responses (most scales are longer, and most items on most scales have more than two responses). TI1e calcula­ tion of coefficient alpha for longer scales is greatly facilitated by a good statistician and a computer.

23

Reliability

EXAMPLE 2.11 Calculating Internal Consistency In the HAND 36-1tem Health Survey, emotional well­ being is assessed with 5 items, called the Mental Health Index or MHJ-S. Suppose we created a smaller mental healUl scale consisting of the following subset of 3 items and wanted to test its internal consistency reliability.

Circle olle Illi/llber 011 ene/dine. During the past month:

Yes

No

Have you been a very nervous person?

1

0

Have YOli felt downhearted and blue?

1

0

Have you felt so down in the dumps tha t nothing could cheer you up?

1

0

The response set for each of these items results in a number of points that are summed to form the scale score. A low total score reflects poorer emotional health, and a higher total score indicates better emoti.onal bealth. for simplicity, the response sets have been reduced to yes/no answers and scored with 1 pOint for a yes and () points for a no. In order to calculate coefficient alpha, we administer the scale to a sample of five nursing home patients and obtain the following results:

24

I lOW TO ASSESS AND lNTERPRl:. r SURVl-.Y

rsycr IOMETIUCS

ExaINI'it' 2 II (O/lti/l,/('d

PaUen t

EWfl//'11! 2.11

ltem A

.1

U

2

I

;1

Item

n

Summed I Scale SCOTt! I

Item C

..

I

I

2

1

1

0

II

Il

4

I

I

1

(:=-1 .,

s

1

)

U

2

3/5 =.6

J,/S = .X

I'cra:n t.lge

positive

-

:JjS

=.6

l -__

I

-1

J

First we must calcu late the sample mean and the sam­ ple variance (these formulas can be found in any basic statistics text). We also note the percentage of positive responses for each item and that the total number of items in the scale is 3.

Calculations: The sample mean score is (2 + 3 + 0 + 3 + 2) 5 = 2.

The sample variance is (2 - 2): + (3 - 2)" + (0 - 2)2 + (3 - 2f + (2 - 2)2 = ~ = J.5. (5 - I)

4

Coefficient alpha for a series of dichotomous i terns is

as follows (k = number of items in the scale): [ 1- I - ( .6)(.4) + <;~~{.2) + (.6)(.4 )] [ 3 /3 _ J ]

=0.86

---

25

Reliability

COIlfjllllt'ti

rhe internal consistency coefficient a,l pha is 0.86, sug­ gesting very good reliability in this scale of three dichotomous items.

If a scale's internal conSistency reliability is low, you can often improve it by adding more items or by reexamining the existing items for clarity. Reliability testing of items and scales provides quantita­ tive measurements of how well an instrument performs in a given population. If you develop a new survey instrument, it is imperative that you test it for reliability before you use it to collect actual data from which you will draw inferences. One of the major drawbacks of new survey instruments is that they are often nothing more than collections of q ues­ tions that seem to the surveyors to fit well together. Even when you are using an established survey instrument with a long and successful track record, it is important that yo u cal­ culate its internal consistency reliability and, if pOSSible, test­ retes t reliability to document its performance in your population. Established survey instruments have typically undergone extensive psychometriCevaluation, but the origi­ nal author's sample population may have been quite differ­ ent from yours. When multicultural issues or language barriers are relevant considerations, it is especially important to carry out reliability testing. If you are collecting data from a group of subjects with whom that survey instrument has not previously been used, you must document how well it performs in the new population by measuring its psycho­ metric properties, induding reliability.

26

I [OW TO ASSESS AND lNTERPRET SURVEY PSYCHOMETRlCS

INTEROBSERVER RELIABILITY

lnterobserver (or interrater) reliability provides a mea­ sure of how well two or more evaluators agree in their assess­ ment of a variable. It is usually reported as a correlation coefficient between different data collectors. When survey instruments are self-administered and designed to measure the respondents' own behaviors or attitudes, interobserver reliability is not used. However, whenever there is a subjec­ tive component in the measurement of an external variable, it is important to calculate this statistic. Sometimes, interob­ server reliability is used as a psychometric property of a sur­ vey instrument, whereas at other times, it is itself the variable of interest. Example 2.12 demonstrates the measure­ ment of interobserver reliability in a survey designed to measure efficiency in the workplace.

EXAMPLE 2.12

In terobserver Reliability:

Impact of Job Sharing

Dana designs a survey instrument to assess the impact of a new job-sharing policy at a newspaper production plant. the policy allows employees with smail children to share a -to-hour workweek to allow each worker more time at home. The survey instrument is a questionnaire completed by a "peer judge," who answers a series of 20 questions on the effldency of the 100-member work­ force on each shift. Dana plans to ask three differenl workers to act as peer judges and complele the survey instrument by quietly making observations during a reg­ ular shift. Dana ~ill figure the efficiency scale score as determined by each peer judge. She will then compare the three data sets by calculating correlation coefficients to detennine the interobserver reliability of the survey instrument. If the correlations are high, Dana will know

2

Relulbillty

E\al/lple 2.12 co"ti"lted

that her survey instrument has high reliability (rom one observer to another. She may conclude that it can be l15ed by variotls peer judges to measure the impact of the new policy on efficiency in this workforce. Tf the correla­ tions are low, Dana must consider that the survey instru­ ment may be operator dependent, not a good quality for such a survey. She may then conclude that the survey instrument is not stable among different judges. She rna) decide to assess the new policy's impact by using one judge for all of the work shifts.

Example 2.13 provides another demonstration that inter­ observer reliability may be more than a psychometric statis­ tic- it may also be the primary variable of interest. In this example, interobserver reliability is not only a psychometric property of the test, it is an outcome variable of primary interest to the data collectors.

EXAMPLE 2.13 Interobserver Reliability: Mammography Rose and Max are very interested in determining whether mammography is a good test to diagnose early­ stage breast cancer. They design a research project in which they will show the same 10 mammograms to a series of 12 different radiologists, They will ask each radiologist individually to rate each mammogram as sus­ piciOUS for cancer, indeterminate, or not suspicious for cancer, They will then compare the responses from the 12 radiologist'; and calculate correlation coefficients. This will provide a measure of intcrobserver reliability. If

28

HOW TO ASSESS AND INTERPRET SUHVI::Y PSYCHOMETRICS

29

Reliability

'xwn"h! 2. J3 cm /lilil/c'd

lhe correlations are high, Hose and Max will conclude that mammography has a high degree of interobserver reliability, but if they arl' low, they will conclude that the interobserver reliability of mammography fs suspect. Rose and Max plan to reporl their findings in lhe med­ italliterature.

Interobserver reliability is often used when the measure­ ment process is less quantitative than the variable being measured. The other forms of reliability discussed in th is chapter are more often used when the variable itself is more qualitative. There are, of course, exceptions to this general rule .

Type of Rc liab illty

Characteristics

Comments

Test-retesl

Measures the stability of responses over time, typ icall y in the saml! group of respoJl dent~

Req uires administration o f survey to a sample a l two d!fferenl an d appropri.a le poi nts in lim e (Time poi n ts lha t arc too fa r apart may prod uce d im in is hed rellabilitv estim ates th at reflect actua l ch ange over time In the variable of interest. )

Jn lraobserver

Me,lsures th e stability of responses oVt!r time in the saml' Individ ual respo n cien t

Requires comple tio n o f

a su rvey by a n in d ividual at two different anci appropriate points in tim e (T ime po ints th at arc to o far apa rt may p rodu ce d imi ni sh ed rcliabi li tv est imates th at reflect actual ch ange over ti me in th e variable of j nt e r ~ s t .)

Alte rnate-form

Uses differen tly wo rded lte m~ o r respon se sets to obtain t he sa me in form at ion abo ut a spec ific top iC

Requires two Items in which the wo rd ing is d ifferen t but aimed at th e same speCific v
Inte rnal consistent)'

Measures how well severa I items in " sca Ie vary together in a sam ple

Usua lly req uires a co mpu ter and statistician to ca rry out C(llcu lati ons

In terobserver

Measures how well two o r m o re di ffere n t responden ts ratl! the same phenomenon

May be u sed to demo n strate reliability of a su rvey o r may itse lf be th e va riable of m terest in a study

NOT E: ReJlabili ly i~ usua lly expressed as a correlation coeffiCient, o r] va lu e, between two sets o f e1ata . Levels of 0.70 or more are generall y accepted as represe nting good re liabil ity.

Q

Validity

In

addition to determining reliability, you must assess the validity of items, scales, and whole survey instruments-that is, how well they measure what they are intended to measure. An item that is supposed to measure pain should measure pain and not some related variable such as anXiety. A scale that claims to measure emotional quality of life should not measure depreSSion, which is a related but different variable. Reliability assessments are nec­ essary, but they are not the whole picture when you examine the psychometric properties of a survey instrument. Once you have documented that a scale is reliabJe over time and in alternate forms, you must then make sure that it is reliably measuring the truth. Example 3.1 takes another look at the problem of mea­ suring fabric that we explored in Example 2.1. Before, we were interested in reliability, but now we are concerned wj th validity.

31

32

HOW TO ASSESS AND TNTERPRLT SURVI:.Y PSYCHOMETRICS

EXAMPLE 3.1

Fabric N(easurement Revisited

Recall from Example 2.1 in Chapter 2 that salespersons at the Honor Guard fabric Company use expensive alu­ minum yardsticks to measure the lengths of all the pieces of fabric the company sel1s. We previously deter­ mined that these meta l yardsticks are very reliablei that is, they measure out the exact same length of cloth every time. Suppose that during a store audit, Honor Guard finds that each of its lI yardsticks" is actually 40 inches long. Every single time a clerk sells a yard of fabric. the measuring stick reliably counts out 40 inches of fabric. ll1e company realizes that over the years, it has given away thousands of inches of eXlra fabric because its alu­ minum measuring sticks are too long. Like a broken clock. the measurement instrument is reliable but not valid.

Types of Validity You must documen t validity when evaluating new survey instruments or when applying established survey instru­ ments to new populations. Validity is an important measure of a survey instrument's accuracy. Survey researchers typi­ cally measure several types of validity when assessing the performance of a survey instrument: face, content, criterion, and construct. FACE VALIDITY Face validity is based on a cursory review of items by untrained judges, such as your Auntie Mame, your boyfriend, your roommate's stepfather, or your squash part­ ner. Assessing face validity might involve simply showing

Vn/irlll)

33

your survey to a few untrained individuals to see whether they think the items look all right to them. This measure, which is often confused with content validity, is the least sci­ entific of all the validity measures. Although face and con­ tent validity are similar, face validity involves a much more casual assessment of item appropriateness. In fact, ma n y researchers do not consider face validity a measure of valid­ ity at all. CONTENT VALIDITY Content validity is a subjective measure of how appro­ priate items or scales seem to a set of reviewers who have some knowledge of the subject matter. The assessment of content validity typically involves an organ ized review of the survey's contents to ensure t hat it includes everythi ng it should and does not include anything it should not. If you arc examining the conten t valid ity of m edical scales, for example, it is important that you include actual patients and their families in the evaluation process. Doctors and nurses may be unaware of the subtle nuances perceived by patients who live day to day with medical conditions. Patien ts' fami­ lies also may provide helpful insights into dimensions th at might be overlooked by "experts." That said, it remains important for clinicians to review the items for relevance to and focus on the variables of interest. Conten t validity is not quantified with statistics. Rather, it is presented as an overall opinion of a group of trained judges. Strictly speaking, it is not a scientific measure of a survey instrument's accuracy. Nevertheless, it provides a good foundation on which to build a methodologically rig­ orous assessment of a survey instrument's validity. Example 3.2 shows how content validity is assessed for a sociological survey on interactions between spouses.

34

HOW TO ASSESS AND INTERPRP.TSlJRVEY rSYCHOMETIUC5

EXAMPLE 3.2 Content Validity: Marital Interaction Kristen designs a new scale to collect data on marital interaction as a dimension of health-related quality of life. She develops a series of 16 iLems about spousal com­

munication, interpersonal confidence, and discussions wilhin the marriage. She plans to use her new scale to assess the impact of social slIpport on a large population of married cancer patients who are ul1dergoing a diffi­ cult and stressful chemotherapy protocol. Before administering ber new scale, Kristen asks three oncologists, three psychologists, two social workers, an oncology nurse-practitioner, four cancer patients, and two spouses of cancer patients to review each of the items. She asks these reviewers to rate each item and the scale as a whole for approprialeness and relevance to the issue of marital interaction. She also asks each reviewer to list any areas that are pertinent to marital interaction but not covered in the 16 items. Once all the reviews are complete, Kristen studies them to determine whether her new survey instrument has content validit) . If she had wished to a'Ssess face validity, Kristen might have asked her coUege roommate or her grandfather to take a look at the survey and tell her whether the items seemed appropriate . Kristen decides to bypass face valid­ ity because she has chosen to look more carefully at con­ tent validity and because she knows that face validity is basically worthless.

35

Validll:)

of a survey instrument than does content or face validity. It may be measured in various ways, depending on how much previous work is available in the area of study. Criterion validity may be broken down into two components: concur­ rent validity and predictive validity.

COllcurrent Validity Concurrent validity requires that the survey instrument in question be judged against some other method that is acknowledged as a gold standard for assessing the same vari ­ able. This other method may be a published psychometriC index, a scientific measurement of some factor, or another generally accepted test. The fundamental reqUirement is that most survey scholars regard it as a good way to measure the same concept. The statistic is calculated as a correlation coef­ ficient with that test. A high correlation suggests good con ­ current validity. Alternatively, a test may be selected for comparison that is expected to measure an attribute or behavior that is opposite to the dimension of interest. In this case, a low correlation indicates good concurrent validity. There are times when you would not simply use an estab­ lished gold standard as your measure of choice; for instance, t he test may be too cumbersome, too expensive, or too inva­ sive to apply. Example 3.3 demonstrates the use of an established scale to assess concurrent validity in a new scale, and Example 3.4 illustrates the assessment of a new survey instrument's valid­ ity in measuring water supply through a comparison with a more standard measure of water supply.

EXAMPLE 3.3 CRITERION VALIDITY Criterion validity is a measure of how well one instru­ ment stacks up against another instrument or predictor. It provides much more quantitative evidence on the accuracy

Concurrent Validity: Pain Tolerance Alisha develops a new 4-item index to assess pain tolE'r­ ance in a group of patients scheduled lor surgery. The items draw information from patients' memories of

36

HOW TO ASSESS AND lNTERPIU::T SURVEY rSYCl rOME'llUCS

Validity

E.:valllplc 3.3 (tmtll/llet!

F.xWTlpl£· 3.4 contli/Ut'd

their past ex.periences with pain. Alisha sums the results from the 4 items to form a Pain Tolerance Index score The higher the score, the greater the tolerance for pain. The index is self-admlnlstered and takes about a minute for a patient to complete. To assess concurrent validity, Alisha administers her 4 items together with a published pain-tolerance survey instrument that has been in use for more lhan a decade in anesthesiology research and is generaUy accepted as tile gold standard in the field. It contains ..J:S items, requlres an interviewer, and takes an average of an hour to complete. 1t is also scored as a sum of item responses. Alisha is able to gather data with both survey instru­ ments in a sample of 24 patients. She calculates the cor­ relation coet1icient to be 0.92 between heI new test of pain tolerance and the gold-standard test of pain toler­ ance. She concludes that her index has high can curren l validity with tile gold standard. Because hers is much shorter and easier to administer, she convinces the prin­ cipal investigator in a large national study of postopera tive pain to use her more efficient index rather than the older, longer sUIvey. Alisha publishes her flndings and is awarded a generous academic scholarship as a result of her work.

monthly water pressure in the kltdlen of the local ele­ mentary school. The higher the index, the greater the water supply in the lown. Luis coUel:ts data for 12 con­ secutive months and uses his formula to calculate his water supply index. During that year, he also records the number of days each month that the 10Lal Department of Water declares as drought days. At the end of the year, Luis plots his index against the number of drought days for each month. He calculates the correlation coefficient between the two data sets to be -0.88. Luis reasons that these two variables should have an inverse relationship, and thus concludes that his index has good criterion validity.

EXAMPLE 3.4 Concurrent Validity: Water Supply Luis develops an index of overall water supply in desert

towns. It is a number based on a mathematical formula

that includes average monthly town rainfall, average

month Iv deptil of the town reservoir, and average

37

Although it is important to evaluate concurrent validity, you must make sure that you select a gold-standard test that is truly a good criterion against which to judge your new sur­ vey instrument. It is not helpful to show good correlations with some obscure index just because it happens to have been published in a journal or book. You should always select gold standards that are relevan t, well known, and accepted as being good meaSUIes of the vaIiable of interest. When testing concurrent validity, you should select gold standards that have been demonstrated to have psychomet­ ric properties of their own. Otherwise, you will be comparing your new scales to a substandard criterion. Predictive Validity

A survey instrument's predictive validity is its useful­ ness in the forecasting of future events, behaviors, attitudes, or outcomes. Predictive validity may be used during the course of a study to predict response to a stimulus, election winners, success of an intervention, time to a medical end­

8

HOW

ro ASSESS AND INTF.HPRET S URVFY PSYCHOMETRJCS

point, or other objective criteria. Over a brief interval, pre­ dictive validity is similar to concurrent validity in that it involves correlating the results of one test with the results of another administered around the same time. If the time frame is longer and the second test occurs much later, then the assessment is of predictive validity. Like concurrent validity, predictive validity is calculated as a correlation coef­ ficient between the initial test and the secondary outcome. Example 3.5 demonstrates that the Pain Tolerance Index that Alisha tested for concurrent validity in Example 3.3 may also be tested for predictive validity.

EXAMPLE 3.5 Predictive Validity: Pain Tolerance Fourteen years after her initial success, AUsha from Example 3.3 becomes professor of gynecology at a well­ respected research university. She decides to use her Pain Tolerance Index to predict narcotic requirements in patients undergoing hysterectomy. Having tested her index for reliability and concurrent validity, she now wants to test it for predictive validity. She administers her index to 100 of her preoperative patients and calcu­ lates an index score for each individual. (Recall that a high score reflects a high tolerance for pain.) Once all the surgeries haye been completed, Alisha reviews the medical records. She noles the total number of doses of narcotic that were administered for postoperative pain in each patient. She then calculate<; a correlation coeffi­ cient between the two data elements: index score and number of narcotic doses. She finds U1at the statistic Is - 0.84. As expected, there is a strong inverse correlation between the Pain Tolerance Index and the amount of narcotic required after surgery. Alisha is pleased to find that her index has high predictive validity in clinical

39

Validity

EX(l/1/1'11! J.S frll1tillllt!d

practice. She publishes her results in a national medical journal and is later promoted to the position of chief of the department of gynecology at an even more presli­ gious university.

Continuing with this theme, researchers can use prediC­ tive validity in a variety of settings to measure the accuracy of survey instruments. One of the most well-known survey instruments is the Scholastic Assessment Test (formerly the Scholastic Aptitude Test), or the SAT. Example 3.6 illustrates the use of the SAT to predict academic success among a pop­ ulation of COllege students.

EXAMPLE 3.6 Predictive Validity: SAT Scores Richard is the dean of students at Shadowmoss College a small liberal arts school in South Carolina. He decides to look into whether the SAT scores of entering fresh­ men predict how well the students will perform during their first semester at Shadowmoss. He looks back into the registrar's records for the past 5 years and gathers two data elements for each freshman: SAT score and first-semester grade point average. Richard then directs his assistant, Meryl. to enter the two data sets into a sta­ tistical program on her laptop computer and calculate a correlation coefficient between the two. To hIs surprise, he finds the statistic to be OAs. The students' SAT scores do nol appear to have high predictive validity for early success at Shadowmoss College. lIe immediately writes a memo to Pamela, the dean of admissioll5 at Shadow­

40

HOW TO ASSESS AND INrER PRllT SURVry PSYCHOMETRICS

',\II"'pi!! 3.6 C/I/llill/lf!.1

moss, to ask that evaluation policies be revised to reflect this important new information. When Pamela receives Richard's memo, she decides to do a little investigating of her own. She takes Richard's data and splits them out year by year. Using the same formulc:l Meryl used, she calculates correlation coeffi­ cien ts with her own laptop computer and finds that over the past S years, the predictive validity of SAT scores has dlanged; the correta lion coefficients for the 5 years, from least to most recent, are 0.21,0.36,0.48,0.57, and 0.72. Pamela writes a memo back to Richard in which she suggests that although SAT scores did not previously have much predictive validity, they have become increasingly more useful for (Jredlction in recent years. Pamela proudly sends a copy of her memo to Chancellor Kara in hopes that the dlancellor will take Pamela's work on this subject into consideration when she is making her upcoming decision on who should be pro­ moted to provost.

Predictive validity is one of the most important ways to measure a test's accuracy in practical applications; however, it is seldom used in longitudinal medical experiments that rely on surveys. Because the time frames in such trials are often several years long, secondary interventions may be implemented during a trial to alter the course of a disease or medical condition. rf the final outcomes were compared with a test score from the start of the study, their correlation may be diminished. This would falsely decrease the mea­ sured predictive validity of the test and perhaps call into question the statistical qualities of an otherwise valid survey instrument.

Validity

41

The situation described in Example 3.6 demonstrates that predictive validity (or any other psychometriC statistic) may be used in various ways to support different hypotheses. You must be careful with the conclusions you draw from any of these measured psychometric properties. A good exercise is to ask peers in your area of study who are unfamiliar with your hypothesis to look at a summary of your data and draw conclusions. If enough people draw the same conclusions, you may be somewhat reassured that your inferences are cor­ rect. You may also ask peers to take your data and statistics and use them to try to support a point that is opposite to YOUI conclusions. This may open up your mind to different interpretations. It may unhinge your argument, or it may guide your approach to collecting evidence that is less easily refuted. CONSTRUCT VALIDITY Construct validity is the most valuable and yet the most difficult way of assessing a survey instrument. It is difficult to understand, to measure, and to report. This form of valid­ ity is often determined only after years of experience with a survey instrument. It is a meaSUIe of how meaningfu l the scale or survey instrument is when in practical use. Often, construct validity is not calculated as a quantifia ble statistic. Rather, it may be seen as a gestalt of how well a survey instrument performs in a multitude of settings and popula­ tions over a number of years. Construct validity is often thought to comprise two other forms of validity: convergent and divergent. Convergent Validity

Convergent validity implies that several different meth­ ods for obtaining the same information about a given trait or concept produce similar results. Evaluating convergent valid­ ity is analogous to measuring alternate-form reliability, except that the former is more theoretical and reqUires a

42

HOW TO ASSESS AND INT ER PRET SURVEY PSYCHOMETRICS

great deal of work, usually by multiple investigators using different approaches.

Type of Validity

Divergent Validity

Divergent (or discriminant) validity is another theoreti­ cally based way of thinking about the ability of a measure to estimate the underlying truth in a given area. For a survey instrument to have divergent validity, it must be shown not to correlate too closely with similar but distinct concepts or traits. This, too, requires much effort over many years of evaluation. To summarize, testing a survey instrument for construct validity is more li ke hypothesis testing than like calculating correlation coefficients . Demonstrating construct validity is much more difficult and usualJy requires a great deal of effort in many different experiments. Construct validity may be said to result from the continued use of a survey instru­ ment to measure some trait, quality, or "construct." Indeed, over a period of years} the su rvey instrument itself may define the way we think about the variable. It is difficult to present a specific example of construct validity, because its measurement an d documen tation req Uire such an all­ encompassing and multifaceted research strategy.

43

Validity

I

I

Characteristics

Comments

Face

Casual review of how good an item or group of items appears

Assessed by inctivid uals with no formal training in the subject under st udy

Content

Fonnal expert review of how good an item or sedes of items appears

Usually assessed by individuals with expertise in some aspect of the subject under study

Criterion : Con current

Measures how well the item or sca le correlates with gol d-stan uard measures of the same variable

Requires the identification of an established, generally accepted gold sta ndard

Criterio n: Predicti ve

Measures how well the item or sca le predicts expected fu ture observations

Used Lo pred ict outcomes Or events of significance that the item or scale m ight subsequently be used to predict

Construct

A theoretical measure of how meaningful a survey instrument is; usually established after yea rs of experience by numero us investigators

Not easily quantifia ble

NOTE: Vahdity is usually expressed as a correlation coeffiCien t, or r value, between two sets of data. Levels of 0.70 or more are generally accepted as representi ng good validity.

I

~

Q

Scalin~ andScorin~

SCales and indexes are not merely collec­ tions of reliable and valid items about the same topic. In fact, most established scales represent months or years of work spent refining the list of items down to the critical ones that provide a rich view of a single attribute. The most common method used to assess whether different items belong together in a scale is a technique called factor analysis. A factor is a hypothetical trait that is thought to be measured by the items in a scale. In factor analysis, a compu ter­ executed algorithm is used to test many different possible combinations of items to determine which of them vary together. Survey researchers use factor analysis to evaluate and select items from larger pools for inclusion ill, or exclusion from, scales or indexes. They then use the resulting scales to produce scores, which in tum are thought to reflect the fac­ tors. Factors themselves are theoretical traits or attributes that are only approximated by scales. Researchers sometimes use another advanced, computer-executed technique called multitrait scaling analysis to measure how well groups of items hold together as scales. 45

46

HOW TO ASSESS AND ~RPRET SURVEY PSYCI JOMETlUCS

Example 4.1 provides an illustration of how a researcher might use factor analysis to select items for a scale that pur­ ports to measure viewer hostility toward sportscasters during broadcasts of the Olympic Games.

Sa/fins m Id Sermllg

47

;mlllph: oJ. I ent/tilll/C'''

scale to the president of UBC, who must decide how to use it against GBS during negotiations for exclUSive rights to the air the upcoming summer Olympic Games.

EXAMPLE 4.1 Factor Analysis:

Sportscasters at the Olympics

Kareem works for United Broadcasting Company (UBC) and wishes to measure television \iewer hostility toward the sportscasters at Good Broadcasting System (GBS) during that network's coverage of the winter Olympic Games. Kareem knows that viewer hostility will be a dif­ ficult concept to define, measure, and package in to a scale. He begins by admInistering a reliable IOO-item survey to 350 adult viewers on 10 nights of television coverage during the winter Olympics. The questions include a wide range of viewer responses, including pleasure, boredom, anger, identification wilh athletes, patriotism, and exasperation . After the winter Olympics are over, Kareem compiles all his data and works with a statistician to enter them into the mainframe computer at USc. He performs a fac­ tor analysis in which he hypothesizes which items will measure his trait of interest- hostility- as well as vari­ ous other traits lhat the survey might measure. The com­ puter puts out a matrix of statistics, called a factor anaJysis, that examines how well the items hypothe­ Sized to measure hostility vary with each other and how well or poorly they vary with the other hypothesized traits. Of the 12 items Kareem has selected, the factor analysis determines that only 7 of them vary together well enough that they can collectively be called a hostil­ ity scale. The other 5 turn out to vary more closely with other hypothesized traits. Kareem presents his hostility

One popular way of structuring response sets in survey research is the Likert scale. This method employs a series of response choices (typically 5) that convey various levels of agreement or disagreement with a statement. For example, the survey instructions might ask a respondent how much he or she agrees with the statement "The hotel guest services were satisfactory." The set of Likert scale responses might be as follows: "strongly agree, somewhat agree, neith er agree nor disagree, somewhat disagree, strongly disagree." Or an item reading, "How often do you feel your self-esteem is low?/I migh t have a Likert scale response set that reads, "all of the time, much of the time, about half the time, a little of the time, none of the time." (There are numerous other ways to structure response sets; these are discussed more fully in How to Ask Survey Questions, Volume 2 in this series.) Scoring a survey instrument is usually fairly straightfor­ ward and amenable to the creation of computer-driven algo­ rithms. Most established survey instruments have published scoring manuals that instruct the user on how many paints to coun t for each response to each item. When you are using an established instrument, it is important that you read the scoring rules carefully, because there are many different ways to convert raw scores to standardized scale scores. This allows researchers to compare different populations fro m dif­ ferent studies. In some survey instruments, a high score is better, whereas in others, a high score is worse. Some are converted to a standard 0-100 range; others may use a range that goes from 0 to I, 4, 25, and so on.

48

I row TO ASSESS AND INT£lU'RCT SURVEY I'S'r'llIOMETRICS

When you are creating a set of scoring rules for a new SUI­ vey instrument, you must determine whether to use a sum an average, or another formula to derive the scale score. Because different items may have 2, 3, 4, 5, or more responses, you must decide whether to value each response or each item the same.

EXAMPLE 4.2

Scoring: Beach Quality

To gather information that will help them select a site for next year's beachfront intramural volleyball tourna­ ment, Fred and Barne} design a new 2-item index that measures the quality of different beaches. The ilems and lhe numbers of points for the responses'are as follows:

ircle one IlIlmber for each item. 1. the average summer temperature at this beach is < /0° ............ .. 0

70' -800 • • • • • • • • • • • • • 1

> 80 0 • • • • • • • • • • • • • • 2

2. The average number of clear days per sum­ mer month at Lhis beach is < 7 ... . . ..... ' .....

a

7 to 1:~ .... . .. . .. . .. 1

14 to 20 . .. . .. . .. . .. 2

... 1 to 25 . .. , ... .. . .. 3

> 25 ............... -1

After testing tht: reliability and validity of the survey instrument extensively al various sample beaches, Fred and Barney are satisfied that it is both reliable and valid. However, they disagree on how to carry out the scoring. Prell wants to score the lndex by Slimming the total

Scalill;'[ alld Scorills

49

EX.ITllPit' 4.2 CrJlltilllWti

number of points in both of the items and then dividing by 6 (the perfect total score). Barney wants to score the index by calculating the percentage scores for both of the items and then averaging them together. Barney would divide the number of points in Item 1 by 2 (a per­ fect score for that item), divide the number of points in Item 2 by 4 (a perfect score for that item), and then cal­ rulate a mean score for the two items. Both Fred's and Barney's scores would be reported on lOa-point scales but they would be calculated differently. To see whether the results from the two methods would be djfferent, Fred and Barney convince t h e tournament adm inistrator to fly them to Cape Cod, Massachusetts, for a trial. While on Cape Cod, Fred and Barn ey t ravel to Provincetown and rate the beach at Herring Cove with scores of 2 and 3, respectively, for the two items. By Fred's scoring method, Herring Cove would score (2 + 3) / 6 = 83 (on a lOa-point scale). By Barney's method Herring Cove would score [(2 / 2) + (3 / 4)] / 2 = 88. The reason Barney's method yields a h igher score is that he values both items equally by scoring t hem indi­ vidually and averaging the item results. Herring Cove has a perfect score o n Item I , and Barney's method val­ ues Item 1 more highly. Fred's m ethod values each item response, not each item, equally. Therefore, the responses in Item 1 are relatively discounted in the total score.

Example 4.2 shows how using different scoring tech­ niqu es can produce very different survey results from the same data. This example sh ows that seemingly subtle alter­ ations in scoring methods can lead to significant differences in scale scores. Both Fred's and Barney's methods are correct, but they place d ifferent values on the items. When designing

SO

HOWTO ASSESS AND INTERPRET :-)URVEY PSYCHOMETRICS

a scoring system for a new survey instrument, you must rec­ ognize these differences and plan accordingly. Remember, if you score a scale by simply adding up the total number of points from each item, those items with larger response sets will be valued relatively higher than those with smaller response sets. One way of aVOiding this problem and valuing all responses equally is to assign increased paints to each response in items with smaller response sets. This common technique is demonstrated in Example 4.3.

~

Creatin;;­ and Usin;;-a

EXAMPLE 4.3

Codebool~

Beach Quality Revisited After much discussion, Barney wins the scoring debate with Fred, and they dedde to score their 2-item beach quality index in a manner that corucrs equal weight to each item. The scoring system they produce is as follows:

Response

Points COllnted

Hem 1

0 1 2

0 25 50

Item 2

0 1 2 3 4

0 12.5 25 37.5

SO

With this method, scoring is accomplished simply through the summing of the total number of assIgned points for each response. The Herring Cove beach from Example 4.2 would be scored by adding SO (Hem 1 := 2) and 3-7.5 (Item 2 = 3) to yield a total score of 87.5, which we round to 88, the exact same answer as Harney got in Example 4.2.

I

n survey research, coding includes the process of going through each respondent's questionnaire and looking for conflicting answers, missing data, handwrit­ ten notes, and other variations from desired "circle one answer" responses. Before you can enter the survey respon ses onto a data tape or into a computer file for analYSiS, you must decide how you are going to categorize any ambiguous answers. No matter how clearly you spell out th e instruc­ tions to respondents to "circle only one answer for each item," you will have unclear responses to some items. Some respondents may not follow instructions for skip patterns correctly, and some may circle more than one answer in response sets that are supposed to be mutually exclusive and collectively exhaustive. No matter how thoroughly you think through your items and response sets before you test your survey, issues will always arise during data collection that will require you to make subjective decisions or correc­ tions. Of course, maintaining a codebook is greatly facili­ tated by the use of a computer- or Internet-based tracking system. Example 5.1 shows one coding issue that came up during a recent survey on family support systems. 51

S2

HOW TO ASSESS AND IN1ER f' RLT SURVFY PSYCHOMETRICS

EXAMPLE 5.1

How Many Kids?

In a recent self-admirustered survey concerning family support systems for men with emphysema, one seem­ ingly straightforward question caused J moderate amount of confusion in several respondents. TIle item read: How many of your. children under age 18 live in your home!

None . . . ... . .. . 0

1 .... . . ... .... 1

2t04 ..... . . ... 2

More than 4 ..... 3

Three respondents answered Ilone, but then wrote in the margin that two to four of their wives' school·age children from previous marriages lived in the home. During the coding of this survey, the researchers decided to CQWlt the wives' young children, because stepchil­ dren in the home fulfill a role similar enough to that of biological children that lhere should be no distinctiol1 in this study. The research context determined how these children were classified . If the same item had appeared on a survey assessing ma le fertility, it wou ld not have been appropriate to include the stepchildren. The researchers recorded their deciSion in the study's codebook for future reference.

The most common data problem arises from the fact that respondents often skip items, either intentionally or by m is­ take . Usually, you will have to code those items as missing data and deCide later how to treat them in the analYSis. Occasionally, yo u can infer the answer to an item from other responses or from other sources, partlcularly if it is a socio­ demographic item, such as age. The best way to handle miss­

51

'reating
ing data is to contact the respondent and ask for the answer to the skipped item . However, thls is not always possible. Example 5.2 demonstrates one way to attempt to solve the problem of missing data.

EXAMPLE 5.2

Missing Da ta

tn the survey described in Example 5.1, concerning fam­ ily support systems for men with emphysema, the researchers included the fo llowing item on family outings: During the past -l weeks, how often clid you and all the other family members who live in your home go out to dinner together? Not at all . .... . ........ Once or twice .. . . . .. . .. Three to fi ve t imes ...... Six or more times ........

0

1

2

3

After all the surveys had been returned, they were coded by an undergraduate student research assistant, who found that 15 respondents had failed to answer this item at all. When he reported tills to the study'S princi­ pal investigator, she asked him to telephone these IS res pondents to try to fi lJ in the missing data. Only 9 of the respondents were available and provided answers to this item; 6 were not reachable, despite numerous attempts. '111e researchers recorded information about the telephone follow-up in the study codebook for future reference. Missing data constitute one of the most problematic areas in survey research. It is critically important that you make extensive efforts to minimize the amount of data miss­ ing from your survey, because missing data detract from the

54

HOW TO ASSESS AND INTIlRPRFl SURVEY P_'iYCHOMETRICS

overall quality of the survey results. The fact that data are missing should also serve to alert you to the possibility that your research methods need a quality check: The more data are missing, the poorer the quality of the methodology. Data may be missing for a variety of reasons. For example, respondents may omit data intentionally or because they have misunderstood skip patterns, failed to grasp the language used, been unable to read the type, or grown tired of a lengthy sur­ vey. Whatever the explanation, missing data can be devastat­ ing to an otherwise well-planned survey project. Example 5.3 demonstrates the value of the codebook when a surveyor makes a decision regarding data after the data have been collected but before they are coded on the survey.

EXAMPLES.3 Counting Homicides Sheila performed a review of police records in the SO largest cities in North America during the past 3 years to identity crime rates. One of the variables she collected was the total number of homiCides in each City during each year. She found the homicide counts to range from zero to 1,045. During the data collection stage of her project, Sheila decided to enter the exact number on a blank line next to the question. Laler, when she coded the surveys, she decided to collapse the hom icide counts into sLx groups that she felt were meaningful: Number of Homiddes

Response Code

o ........ . .. .. . . . .. . 1

1-10 ........... .. . .. . 2

11-50 . . .. .. .......... 3

51-IOO . ... .. ......... 4

101-250 . .... .. . ... . .. 5 >250 ...... . ........ . 6

Creating (I lld

US i n:s

a Corlebook

5S

E.vomph" S.J colltilll/ed

Sheila made a research decision lhat she did not need to analyze her data based on the e.xact number of homi­ cides. She recorded the above table in her study code­ book for future reference. She knew that if she decided to add more cities to her series several years later, she would not be able to remember where she had drawn the lines between groups. She also knew that she would be able to refer to her thorough codebook to get that information.

In some senses, the code book is a log or documentation of the research decisions that are made during the coding or review of surveys. Despite your best efforts, you will be unable to remember all th e small decisions that you make along the way in carrying out the collection, processing, and analysis of survey data. The codebook is a summary of all those decisions. It functions not only as a record but also as a rule book for future analyses using the same or similar data. The importance of keeping meticulous records cannot be overemphasized. Inevitably, in survey research, questions arise about how you collected a data element, where you went for resources, how you documented certain variables, whether you completed follOW-Up in a certain way, and so on. The only way to guarantee that you can reproduce your methods accu­ rately is to maintain excellent records. This includes your codebook as well as records of the way you structured each aspect of your methodology and data analys is.

~ Pilot Testin;;~

Before a new feature film appears in movie theaters, the movie's producers often show it to sample audi­ ences in various cities to observe their reactions to the char­ acters, plot, ending, and other aspects of the entertainment experience. If these sample audiences consistently dislike some aspect of the film, it is usually changed before the movie is released. Similarly, before a new product is intro­ duced into retail stores, the manufacturer always test-markets it to gauge consumer satisfaction. And before you buy a new car, you almost always take it for a test drive, even if you feel certain about your choice of models. In survey research, pilot testing, or pretesting the survey instrument with a small sample population, serves a func­ tion similar to these other kinds of tests. Pilot testing is one of the most important stages in the development of a new survey instrument, and time and again you will find that it proves to be worth the energy you spend on it. Pilot testing has three main benefits: • It helps you to identify errors in the survey.

• It allows you to learn where your survey instrument

may need redesign . 57

58

HOW TO ASSESS AND INTERPRET SURV EY PSYCHOMETRICS

• It predicts possible problems you may encounter in

using the instrument. Pilot testing almost always identifies errors in a survey's form and presentation. Inevitably, despite extensive thought and planning, errors occur in the final versions of question­ naires. They range from confusing typographical mistakes to overlapping response sets to ambiguous instructions. When you are creating a survey instrument, you are usually so close to the project that you may overlook even the most obvious errors. Pilot testing allows you a chance to correct these errors before the survey is mass-produced or used on a wider scope to gather real data. It allows you the time and opportu­ nity to redesign any problematic parts of the survey before it is actually used. Pilot testing also predicts difficulties that may arise during subsequent data collection that might oth­ erwise have gone unnoticed. At this early stage, most prob­ lems are still correctable. Sometimes the issues identified during pilot testing are problems of form. For older respondents, many of whom may have impaired vision, the type size and font are espe­ cially lmportant. Pilot testing may also expose respondents' di fficulties with the reading comprehension level of the sur­ vey. Many survey respondents are less technically educated than survey researchers. If a survey is written at a level above the understanding of the intended respondents, the result­ ing data will be spurious or incomplete. Respondents must be able to understand th e semantics of the survey items in order to provide honest and thoughtful answers. The general rule is that surveys should be written at no higber than a Sixth-grade reading leve1. Software applications are available that can help you to estimate the reading level of your sur­ vey. Likewise, many populations today are not homoge­ neous in their primary language. Even if a survey is written at a comprehensible level for native English speakers, indi­ viduals who are not completely fluent in or comfortable with English may answer with data that are not usable.

Pilot Testil1g

59

Language may present significant challenges, but just because your intended respondents do not speak English flu­ ently, that does not mean you cannot gather data from them. If there is information that you want to gather that is available reliably and validly only in the respondents' native tongue, and if you are very industriOUS, you may undertake the creation of a foreign -language version of your survey. In addition, pilot testing can help you to make sure that the items in your survey are culturally sensitive. Certain areas that are covered in a questionnaire may represent con­ cepts that are unfamiliar to some respondents. Pilot testing allows you to identify many such potential impediments in advance, so you can correct them before you have expended too many resources. (For discussion of how you can avoid the pitfalls associated with crossing cultural and language barriers in survey research, see Chapters 7 and 8). Pilot testing will not eliminate any of the problems men­ tioned above, but it will allow you to identify them so that you can consider the implications and decide prospectively how to handle them . Example 6.1 illustrates one of the simple but common problems often identified during pilot testing of a survey. When designing questionnaire layout, many researchers are tempted to squeeze as many items as possible onto one or two pages. They think that if a survey looks short, their respondents will believe it to be easy and quick to complete. But shortening the length of a questionnaire by making the type size smaller only makes it more difficult to read. If respondents have trouble just reading the words on the page, they will not have much energy left to think about th e meaning of the questions. Ramon in Example 6.1 learned that sometimes a survey that is a little longer but much eas­ ier to read can yield more valuable results than one that fits nicely into a short forma t and is difficult to read. Two pages are effectively shorte r than one if the type is easier to read. One trick that some researchers use is the creative num­ bering of items. Instead of simply numbering every question

60

HOW TO ASSESS AND lNTERPR[ I SURVE\ PSYCHOMETRICS

6J

Pilot Testing

EXAMPLE 6.1

EXAMPLE 6.2

Type Size

Arizona Geological Survey

Ramon was very pleased that his new lO-item question­ naire on atUtudes loward aging fit onto one side of a sin­ gle page. He estimated that it should require about 5 minutes to complete. Before Ramon set out to assess IllS survey's reliability and validity in a sample of 250 older New Yorkers, his professor urged him to conduct a small pilot test \,'lith six senior citizens. Ramon set up the pilot test by calling the activities office of a local community senior center and recrujting volunteers. On the appointed date, the four women and two men met Ramon at the community center for the short pilot test. To his surprise, Ramon found that the average time they took to fill out hls survey was 20 minutes. When he queried them as to why, five of the six stated that they had a great deal of trouble reading the questions, even with their glasses. They had 110 problems with reading comprehension or wjth English; they simply could not see the type on the page clearly Ramon revised the type size on his survey from 8­ point to 12-point. Instead of fitting onto a single page the survey now took up two pages. Ramon mailed the new version to his six pilot-test respondents and asked for their comments. All of them replied that it was now much easier to read and fill out.

Marissa and Jacob are geology students who wish to col­ lect data on the types of rocks found in the various counties of Arizona. They design a -lO-item question­ naire to be completed by the interior commissioners of all the counties, but they know Ulat these individuals are qUite busy and may not want to take the time to flU out a lengthy survey. Marissa and Jacob do some basic research and djscover that by far the three most com­ mon rock fonns in the state are quartz, limestone, and shale. Before sending out their survey, they perform a pilot test with two of the commissioners they know per­ sonally. They learn tha t the commissioners' main com­ plaint about Ule survey is that 40 items seem like quite a lot. Because t hey do not want to delete any of the items, Marissa and Jacob carefully craft a survey fonnat that relies heavily on groupings of simllar items. rn the cover letter they send out with the questionnaires, they intro­ duce their research project and ask for the commission­ ers' belp in complying with a "short lO-item survey." The survey's first item is as follows :

from ] to whatever, they break items up into groups or sec­ tions to create the illusion that the survey is shorter than it really is. If several items in a survey rely on the same response set, you can make them appear as a single item with several subitems. Rather than numbering each item sepa­ rately, you number the instruction and define the items with letters. This technique is demonstrated in Example 6.2.

62

HOW TO ASSESS AND INTERPRET SURVEY PSYCHOMP.TlUCS

XIWlp!t·

6.2 ("(It/till/It'd

1. Please circle the number corresponding to the. most common type of rock at each of these sites in your county.

Quartz

Limestone

Shale

a. Lakes

1

2

3

b. Hills

1

2

3

1

2

3

d. Suburbs

1

2

3

c. Forests

1

2

3

f. Riverbeds

1

2

3

c. Mounlains ---."

By combining their items together into groups as shown above, Marissa and Jacob were able to reduce 40 items to 10 items without shortening their survey at all. II simply appears ~horLer. Marissa and Jacob hope that the county interior commissioners will be cooperative in filling out this IIshort" questionnaire despite their bus) schedules.

!

63

Pilot Testin,'(

Surveys do not have to be filled with page after page of boring and monotonous questions. You can spice up the demeanor of your survey instrument by using some very simple techniques. Varying the response sets into columns for like groups, as in Example 6.2, is one commonly used method. Another is the incorporation of graphics into the response sets. To provide mental breaks for your respon­ dents, you can place a few short, easy questions after series of more difficult questions. Skip patterns are anot her area of questionnaire design that presents challenges for survey researchers. Using instructions such as "If you answered yes to the previous question, then skip to page 4" may be a wonderful way to use the same questionnaire for different groups, but it may cause you and your respondents untold fru stra tion and can lead to devastating amounts of missin g data. Often, pilot testing will h elp you to identify problems with skip patterns and other simple visual problems with your survey's layout and format . At this stage, you can easily correct what is wrong and then retest the instrument before you proceed to your primary effort at data collection. Example 6.3 illustrates the problem of confusing skip patterns. As the example shows, online and computer-aided survey instruments allow you to tailor the presentation of items and to incorporate skip patterns much more unobtrusively than you can in pen­ and-paper surveys.

EXAMPLE 6.3 Algebra Attitudes Estelle wishes to survey the attitudes of ninth-grade stu­ dents in Chicago toward different approaches to learn­ ing math. She designs a comprehensive questionnaire with several branch points that direct students to differ­ ent sets of questions. The firs t item in the questionnaire is as follows:

64

HOW TO ASSESS AND TNTERl'Rl:.1 SURVEV PSYCHOMErRICS

Etl1llll'll!

63 ((Jl/tilllled

1. Would you prefer to learn algehra by Listening to a teacher present new material and work problems ..... .............. A Listening to a teacher but working problems on your own ..... . B Reading new material and working problems all on you r own .... C

ifyou answered A, continlle with item 2, but skip items 12-36, and thell cOl/lplete items 37-50. If YOll allswered B, skip items 2-11, cOlltinlle with items 12-23, skip items 24-36, bllt complete items 37-50. If you allswered C, skip items 2-23, but c01llplete ite1lls 24-50. Betore taking her survey into the entire school dis­ trict, Estelle pilot-tested it in ber daughter's ninth-grade class of 30 girls and boys. She found that the students had numerous questions about the SkIP patterns. Despite their best efforts, Lhey simply found the instmc­ tions too confusing. Estelle was forced to take her survey back to the drawing board to make it more user-friendly for Lhese respondents. She was able to design an online version that presented the appropriate next item each time an item was answered, so the respondents did not have to deal directly with the skip patterns. When she returned Lo the same class to repeat her pilot lest, she found that the students logged on and sailed through the survey without any apparent difficulty.

65

Pilot Testing

Pilot testing is not limited to new survey instruments. It is beneficial to perform pilot tests even when you are using often-published and well-established survey instruments. Unless you are working with a population that is exactly the same as the other populations in which a survey instrument has been validated, you will probably be introducing some new twist in you r particular sample. Pilot-testing established survey instruments is extraordinarily helpful in all survey research. You may find that, for whatever reason, even an established survey instrument does not perform well in your pilot-test sample. This will afford you the opportunity to select a different survey instrument or even develop your own if you have the time and resources to do so. Example 6.4 provides an illustration of a case in which pilot testing revealed that an established survey instrumen t was not rele­ vant for the population of interest.

EXAMPLE 6.4 Type A Personalities The Jenkins Activity Scale is a published and well­ established survey instrument that distinguishes indi­ vi dual s with relatively high levels of daily anxiety and compulsiveness, so-called Type A personalities. from those with Lower levels of stress, so-called Type B person­ alities. It is intended to identify people at risk for heart attacks, strokes, and high blood pressure. In an effort to find potential clien ts with certain investment habits, the marketing director at a money management firm of dubious reputation decided to mail the Jenkins Activity Scale, which is a self-administered instrument, to a national sample of 5,000 members of the American Association of Retired Persons (AARP). The marketing director had an idea that Type B personalities would be more likely than Type A personalities to hand over their savings for investment without asking too

..

66

HOW TO ASSESS AND INr~RPRET 'iUTWLY PSYCHOMETRICS

E,\ill1lllle 6.'1 ((II/tilllleil

many questions. After a thorough literature review, the marketing director selected the Jenkins '\ctivity SUlvey because it is short and easily scored, and its reJiabilit} and validity are well documented. The marketing direc. tor submilled a proposal concerning the use of the sur· vey to the fi rm 's vice presiuent for marketing. Before committing Lbe fum's resources to conducting a survey using this particular survey instrument, the vice president asked her assistant to conduct a small pilottest by sending the Jenkins Activity Survey to 12 members of the local AARr chapter. Of the 8 surveys that were returned, 7 of the respondents had completed only a small fraction of the items. It turned out that most of the items on ti,e survey referred to the respondent's con. duct at work, interactions with professional colleagues and number of vacation days taken from the job. Of tile 8 people who participated in the pilot test, only '1 was currently employed. The remainder were retired and could not answer many of the questions-not surpris· ing. given the target audience of retired persons. The vice president fired the marketing director and hired an expensive outside consulting firm to develop and evaluate psychometrically a new survey instrument that assesses daiJy anxiety levels in retired persons as a predictor of investment strategies.

Pilot testing is a necessary and important part of survey development. It provides you with useful information about how your survey instrument actually plays in the field. Although performing a pilot test requires extra time and energy, it is a critical step in assessing the practical applica. tion of your survey instrument .

67

Pilot Testing

Checklist for Pilot Testing ./ Are there any typographical errors or misspelled words in the instrument? ./ Do the item numbers make sense? ./ Is the type size big enough to be read easily? ./ Is the vocabulary appropriate for the respondents? ./ Is the survey too long? ./ Are the styles of the items too monotonous? ./ Are there easy questions mixed in with the difficult questions? ./ Are the skip pa tterns too difficult to follow? ./ Does the survey forma t flow well? ./ Are the items appropriate for th e respondents? ./ Are the items sensitive to possible cultural barriers? ./ Is the survey in the best language for the respon­ dents?

~

~lIltic•• I~lll

ISslIeS

When you are designing new survey instruments or applying establish ed ones in populations of different ethnicities, creeds, or nationalities, yo u must make sure that your items translate well into both the language and the culture of your target audience. Although you may be able to translate a survey instrument's items into a n ew language, you may fi nd that they do n ot measure the same dimension in different cultures. This is particularly relevan t when you are studying social attitudes and health behaviors. Different cultures have very different concepts of health, well-being, illness, and disease, for example. A concept that is well developed in one culture may not even exist in another. Even if you start with a well-validated survey instru­ ment in English, you need to be aware that various popula­ tions within the United States and elsewhere in the world may not approach the concepts it measures with the same ideas. Failure to be attentive to multicultural issues may result in significant bias in the collection of data. For example,

69

o

HOW TO ASSESS AND lNTERPRET SURVEY PSYCHOMETRICS

when classifying ethnicities, many survey researchers catego­ rize all Asians together. For some projects, this may be acceptable, but many attitudes and behaviors vary a great deal among Chinese, Japanese, Filipino, Korean, and other cultures within Asia. If you lump all such ethnicities into one class, you may overlook differences that are important to your conclusions. Example 7.1 illustrates how problems can arise not only from language barriers but also from conceptual differences due to culture. EXAMPLE 7.1

Elderly Wonlen Olivia decided to lise an established, well·validated 24­ item survey instrument to measure family attitudes in elderly American women of Anglo and Latino descent. In order to facilitate her survey's croSSing the potential language barrier, Olh1a asked her friend Juanita, a native Mexican, to translate the survey into Spanish. After the translation was complete, Olivia administered the instrument to a sample of 300 elderly women, of whom ] SO classified themselves as Latina and 150 classified Ulemselves as Anglo. After she tabulated her data, Olivia was disappointed to discover Ihat, despite the perfect translation into Spanish , the Latina respondents' surveys were missing a tremendous amount of data. When she researched the matter more closely, Olhria realized that she had failed to take into account the multicultural issues raised by her project. She found out that the Latinas had a con­ cept of family ti1at was completely different from that of the Anglo-American women. For example, the Anglos tended to include only parents and children in Uleir concept of family, whereas the Latinas tended to con­ sider grandparents, aunts and uncles, and cousins as

Mlllticu/tural Issues

71

well. The survey instrument had b~en validated only in Anglo-American populations. Although the words in the survey had been translated quite well into \panish, the philosophies of family in the two groups were so differ­ ent that most of the Anglo concepts had no real mean­ ing to the Latinas. Therefore, they lett many ilems blank, and Olivia was unable to analyze her data in a useful way.

~

Translations

T ranslating a survey into a foreign language is not as simple as Olivia in Example 7.1 thought. You must consider n ot only issues of culture, but also issues of linguis­ tics. The correct way to translate your survey is to start by identifying two or three bilingual individuals and having them perform independen t "forward translations" into the new language. Next, you have these translators reconcile any differences in word choices in the new language. You then find two or three more bilingual individuals and have them perform independent "back-translations" in which they con­ vert the original reconciled translation back into English. Next, you have one or two bilingual individuals perform independent reviews of the forward, reconciled, and back­ translated versions in order to iron out any language differ­ ences and agree on the final wording of each item in the new language. Finally, after you pilot-test the instrument with a few bilingual subjects, you conduct a larger-scale test to measure the instrument's reliability and validity in the new language (see Examples 8.1 and 8.2).

73

74

HOW TO ASSESS AND INTERPRET SURVEY PSYCHOMETRICS

EXAMPLE 8.1

How Often in Spanish

Recently, while testing a linguistic translation of a sur­ vey instrument designed to measure symptoms in men with chronic prostatitis, researchers coUected data from many Spanish-speaking patients, Induding 15 in Argentina, 15 in Mexico, 4 in the United States, and 3 in Spain. The translation had high overall reliability, but the respondents noted t hat they had difficulty distin­ guishing between the response categories a merllldo (often) and norma/mente (llsually) in one of the items. The researchers decided to revise a menudo to mucJlas veces (milch uflile time) and nornmlmen te to casi siempre (almost always) to improve the distinctiveness of the response categories.

EXAMPLE 8.2 Pain in Span ish The original English version of the survey described in Example 8.1 contained the phrase paill or discomfort six times. The recommended translation was dolor 0 illcomo­ didad, which was back-translated as pain or discom(ort. However, one reviewer noted that illcornodidad is not generally used in the medical sensei and that mole5tia, which was also back~translated as pain or discomfort, con­ veyed the proper meaning in the medical sense. The other reviewers agreed. The final Spanish-language ver­ sion of the survey used the phrase doLor 0 mo/estia, which seemed to be tile simplest and most appropriate transla­ tion for the medical context.

~ Exercises

1. Northwest Cable administers a television prefer­

ence survey to 100 viewers in the company's

southeast region and includes the following two

items at different pOints in the same question­ naire: • How many hours of TV did you watch in the past 7 days? • How many programs of what length did you watch during the past 7 days? Which of the follOWing is the company trying to document? a. Alternate-form reliability b. Test-retest reliability c. Internal consistency reliability d. All of the above 2. Southwest Cable administers a new television preference questionnaire to 50 viewers in the company's northeast region on January 15, then repeats the identical survey with the same 50 viewers on February 15 and finds close correla­

7S

76

HOW TO ASSESS AND INTERPRET SURVEY PSYCHOMETRICS

tions between the two data sets. Southwest Cable's survey instrument can be said to have good a. Alternate-form reliability b. Test-retest reliability c. Internal consistency reliability d. All of the above 3. When you read that a scale has a coefficient

alpha of 0.90, you can be assured that the scale is

a high degree of

a. Alternate-form reliability b. Test-retest reliability c. Internal consistency reliability d. Concurrent validity 4. Suppose you are developing a measure of st udent

satisfaction with school lunches in a 7S0-student

elementary school. How might you design an

experiment to assess test-retest reliability?

S. Design two different response sets to test alternate­

form reliability for the fo llowing item from a

meteorological survey instrument about regional

precipitation:

How much rain did your region have

during the past 4 weeks?

6. A new survey instrument is published for assess­

ing safety in automobiles. It includes 12 items

about a wide range of qualities, such as strength

of the seat belt straps, the temperature at which

the engine overheats, how the windshield shat­

ters during impact, thickness of the steel in the

doors, and adequacy of the ventilation system.

The survey instrument is tested with several

£xC!'(i~C!~

models of cars, and it is determined that it has

good test-retest and alternate-form reliability, but its coeffident alpha is on ly 0.23. What does this mean? 7. The housing office of a large university wants to measure student satisfaction with various aspects of the campus dormitories. After researdling the relevant published literature, the housing direc­ tor cannot fi nd a survey instrument that she thinks is appropriate, so she decides to develop her own. She remembers from her survey research course in college that her index must be reliable and valid. She also remembers that her index must have good content validity. How would you advise her to begin her project? 8. Brian has designed and pilot-tested a new 20­ item self-administered survey instrument that measures religiOUS observance in a group of mid­ westerners. How might he go about assessing it for concurrent validity? 9. Obsessions Unlimited is a company tha t pro­ vides personal organization services; customers pay a fee in return for receiving individualized help in organizing their personal lives. The com­ pany is looking for a better way to assess the quality of applicants for employment as personal organization assistants. The director of personnel wants to devise a screening questionnaire that has excellent predictive validity in selecting per­ sons for employment as personal organizational assistants who will produce customer satisfac­ tion. He puts together a reliable self-adminis­ tered index of 20 items that job applicants can complete easUy. How should he test his index's predictive validity?

77

78

HOw TO ASSESS AND IN TERPRET SURVEY PSYCHOMETR ICS

ANSWERS

1. Alternate-form reliability

2. Test-retest reliability 3. All of the above 4. Because the students' satisfacti on with particular

lunches may be different on differen t days, you

must administer your survey at separate times on

the same afternoon after lunchtime. For exam­ ple, you might have students complete question ­ naires right after they have had lunch an d then

again right before school lets out.

5. Response Set 1: In this region, the average total

rainfall (to the nearest 10th of an inch) during

the past 4 weeks was

None 0.1 to 4 inches 4.1 to 8 inches 8.1 to 12 inches

More than 12 inches

Response Set 2: In this region, the n umber of days it rained during the past 4 weeks was No days

1-7 days

8-14 ays

15-21 days

22-28 days

Response Set 3: ill this region, rainfall during the past 4 weeks has been

79

Exercises

Below average for this time of year About average for this time of year Above average for this time of year 6. The survey instrument has low internal consis­

tency reliability. This means that although the

items seem to focus on various aspects of the

same concept, they may actually be quite differ­

ent. The author should consider the possibility

that the items may not be measuring the same

concept.

7. If she wants to ensure content validity, she must tailor her survey instrument to th e needs of the students themselves. The best way to start would be to put together a focus group of students cur­ rently living in the campus dorms. During this exploratory seSSion, she could get an idea of what issues are important to the students. She m ight then put together a first draft of her ques­ tionnaire and show it to these students for their comments. This wou ld provide initial testing of content validity. 8. In order to assess concurrent validity, Brian must identify some gold-standard method of assessing religious observance. He should start by looking in the sociology research literature. There may not be any such established measure, however, so he may have to select a measure that appears to be a gold standard. For example, he might use actual attendance at religiOUS services or religious functions during a 4-week period for a randomly selected group of midwesterners.

9. In order to test the predictive validity of his screening questionnaire, the personnel director must also select an established outcome measure

80

HOW TO ASSESS AND INTER PRET SURVEY PSYC HOMETRICS

of customer satisfaction. He should administer his questionnaire to all employment applicants over a defined period of time. He should then use the selected outcome measure to document the satis­ faction of those customers served by the different employees. By using the results from the screen­ ing index to try to predict which employees will have good results on the satisfaction outcomes measure, he can assess the predictive validity of his questionnaire. If those who score well on the screening index also score well on the satisfaction index, then his screening index can be said to have good predictive validity.

~

Su&'geste(l Readings

Coons, s. L Rao, S., Keininger, D. L., & Hays, R. D. (2000) . A comparative review of generic quali ty-of-life instruments. PharmacoEcol1omics, 17, 13­ 35. Guyatt, G. H., Kirshner, B., & Jaeschke, R. (1992). Measuring health status: What are the necessary measurement properties? journal of Clinical Epidemiology, 45, 1341-1345. Hays, R. D., Sherbourne, C. D., & Maze!, R. M. (1993). The RAND 36-Item Health Survey 1.0. Health Economics, 2, 217-227. Hays, R. D., & Woolley, J. M. (2000). The concept of clinically meaningful dif­ ference in health-related quality-of-life research: How meaningful is it? Pharmaco Eco l1omics, 18, 419-423. Kagawa-Singer, M ., & Blackhall, L. J. (2001). Negotiating cross-cultural issues a t the end of life: "You got to go where he lives." journal of the American Medical Association, 286, 2993-3001. Kaplan, R. (1990). Utility assessment for estimating quality-adjusted life years. In F. Sloan (Ed .), Va luing healtlz care: Costs, bel1efits, alld effecti velless of phar­ maceuticals and other medical technologies (pp. 31-60). Boston: Cambridge University Press. Litwin , M . S, H ays, R D., Fink, A., Ganz, P. A., Leake, B., Leach, G. E., & Brook, R H . (1995). Quality-of-life outcomes in men treated for localized prostate cancer. jOllrnal of tile American Medical Association, 273, 129-1 35.

81

82

HOW TO ASSESS AND INTERPRI::.T SURVEY PSYCHOMETRICS

Reiser, S. J. (1 993). Th e era of the pahent: Using the experience of ill ness in shaping the missions of health care. Journal of the Americall Medical Assoriatioll , 269,1012-1017. Rossi, P H., Wright, J. D., & Anderson, A. B. (Eds .). (J 983). Handbook of su rvey research. San Diego, CA: Academic Press. Slevin, M. L., Plant, H., Lynch, D., Drinkwater, J., & Gregory, W. M. (1988). Who shou ld measure quality o f life, the doctor or the patien t? British !aumal of Ca 11 cer, 57, ] 09-112. Ware, J. E., Jr., Kosinski, M., Gandek, B., Aaronson, N. K, Apolone, G., Bech, P., et a!. (1 998). The factor structure of the SF-36 Health Survey in 10 coun­ tries: Results from the IQOLA Project. Jo umal of CIillical Epidemiologtj, 51 , 1159-11 65. Ware, J. E., Jr., Kosinski, M., & Keller, S. D. (1994). SF-36 pllysical atld lIIel1tal health summary scales: A user's maltllal. Boston: New England Medica l Center, Heal th [nstitu te. Ware, J. E., Jr., Kosinski, M., & Keller, S. D. (1996) . A 12-ilem short-form health survey: Construction of scales and p reliminary tests of reliability and validity. Medical Care, 34, 220-233. Ware, J. E., Jr., & Sherboume, C. D. (1992) . The MOS 36-ltem Short-Form Health Survey (SF-36): 1. Concep tual framework and item selection. Medical Care, 3D, 473-483.

~

Glossary

Alternate-form reliability-A measure of survey repro­ ducibility in which an item and/or response set is worded in two or more different ways and the different versions are compared for consistency in responses. Codebook- A collection of rules developed during the trans­ lation of survey responses into numerical codes for analysis. For example, a codebook might contain a rule on ethnicity that assigns the number 1 to African Americans, 2 to Anglos, 3 to Asians, 4 to Latinos, and so on. Another rule might assign the number 9 to all items for which data are missing. The codebook IS a summary of all such rules to be used as a reference during data analysis. Concurrent validity-A measure of survey accuracy in which the results of a new surveyor scale are compared with the results from a generally accepted gold-standard test after both tests are administered to the same group of respondents. Construct validity-A theoretical gestalt-type measure of how meaningful a survey instrument is, usually after many years of experience by numerous investigators in many varied settings.

83

84

HOW TO ASSE.SS AND INTERPRET SU RV£Y PSYCHOMETRICS

Ci lossClry

85

Content validity-A measure of survey accuracy that involves formal review by individ uals who are experts in the subject matter of the survey.

Intraobserver reliability- The reproducibility of a set of observations on one variable made by the same observer at different times.

Convergent validity-A measure of survey accuracy that involves using differen t tools to obtain information about a particular variabJe and seeing how well the results correlate. Evaluating con vergen t validity is anal­ ogous to measurin g alternate-form reliability, but with different established instruments rather than different wordings of single items.

Item-A question that appears on a surveyor in an index.

Correlation coefficient- A statistical measure of how closely two variables or measures are related to each other. Correlation coefficients are usually calculated and reported as rvalues. Criterion validity- A measure of survey accuracy that involves comparing the survey to other tests. Criterion validity may be categorized as convergen t or divergent. Divergent validity- A measure of survey accuracy that involves using different tools for obtaining information about similar but discrete variables and seeing if they differ. Face validity- The most casual measure of a survey's accu­ racy, usually assessed informally by nonexperts.

Measurement error- The degree to which instruments yield data that are incorrect due to the measurement process. Multitrait scaling analysis-An advanced computer­ assisted method of measuring how well various items go together in a particular scale (similar to factor analysis). Pilot testing- The practice of trying out a surveyor index during the development phase with a small sample one or more times to see how well the survey works, to expose errors, and to identify areas of difficulty for respondents. Practice effect- A phenomenon in which a respondent becomes familiar with items on a surveyor index taken at several time pOints. Over time, the individual/s responses correlate highly with each other simply because he or she is remembering previous answers, and not because the variable being measured is unchanged. Predictive validity- A measure of survey accuracy in which an item or scale is correlated with future observations of behavior, survey responses, or other events.

Factor anaJysis- A computer-assisted method of analysis used to assess whether different items on a survey belong together in one scale.

Psychometrics-The science of measuring psychological or qualitative phenomena.

Index- See Scale.

r value-The statistic that is used to report correlations. See

Internal conSistency reliability- A measure of survey accu­ racy that re fl ects how well different items in a scale vary together when applied to a group of respondents. Interobserver reliability- The reproducibility of a set of observations on one variable made by different observers .

Correlation coefficient. Random error- The degree to which instruments yield data that are incorrect not due to the measurement process, but due to uncontrollable fluctuations in responses. Reliability-The reproducibility or stability of data or obser­ vations. When using a surveyor index, one wants to achieve high reliability, which implies that the data are highly reproducible.

86

HOW TO ASSESS AND INTERPRET SURVEY PSYCHOMETRICS

ScaJe--A series of items measuring a single variable, trait, or domain. ScaIing- A process in which different items are placed together in a single index that pertains to one variable of interest. This process often involves factor analysis and multitrait scaling analysis. Scoring- The conversion of an individual's survey answers into a numerical value for comparison with the answers of other individuals or of the same individual at differ­ ent times. Split-haJves rnethod- A technique used to assess alternate­ form reliability in which a large sample is divided equally into two smaller samples, each of the two h alves of the sample are administered differen t forms of the same question , and the responses of the two halves of the sample are compared. See Alternate-form reliability. Survey instrument (or Survey)- A series of items that typi­ cally contains several scales. A survey may be self­ administered or may require a trained interviewer. It may be very long or contain a single item. It may be about issues as personal as sexual fu nction or as imper­ sonal as rain fall. Test -retest reliability- A measure of t he stability of responses over time in th e same group of respon dents. Ma n y investigators report test-retest reliability by administering the same survey at two different time points (often 4 weeks apart) to the same group of indi­ viduals.

~ Abollt theAI.thor Mark S. Litwin, M.D., M.P.H., is Professor of Health Services and Urology at the UCLA School of Public Health and the David Geffen School of Medicine at UCLA, where he teaches and conducts outcomes research and practices urologic oncology. He received his M.D. [rom Emory University and trained in urology at Harvard Medical School's Brigham and Women's Hospital. He obtained h is M.P.H. from UCLA, where h e was a Robert Wood Johnson Clinical Scholar. In 2001, h e received the Gold Cystoscope Award, the American Urological Association's highest honor for the academjc con ­ tributions of a young investigator. He has extensive experi­ ence in health services research , includin g studies concerning health-related quality of life, patient utilities, quality of care, and medical resource use. His research is funded by the NTH, the American Cancer Society, and the California Departmen t of Health Services.

Validity- An assessment of how well a surveyor index measures what it is intended to measure.

87

"Pilot testing is very good and the coverage of reliability and validity is well-developed and well-written.n --Bonnie Rader, California State University, Long Beach "Mark Litwin does a great job of constructing 'friendly' examples, and keeping the tone of the text more of a conversation than an excerpt from a graduate level psychometrics or statistics textbook. He also includes a chapter on multicultural issues and how they affect every phase of survey design and use. -Patrick R. Powaser, Personnel Psychology II

Full of examples and clear explanations, this book will show you how to assess and interpret the quality of collected survey data by thoroughly examining the survey instrument used. Litwin covers: • Measuring reliability (including test-retest, alternate-form , internal consistency, interobserver and intraobserver reliability) • Measuring validity (including content, criterion, and construct validity) • Creating and using a codebook • How to pilot test new and established surveys • How to scale and score a survey

"Kudos to the author. These topics represent a potentially complicated and difficult set of concepts that was well (and painlessly) explained." -Carol J. Lancaster, Medical University of South Carolina "Very useful because, in addition to providing the standard definitions of the different methods to assess reliability and validity of a survey instrument, it gives concrete and clear examples of each type. The examples illustrating content and criterion validity are superior to most seen in research methodology texts." -Sohmi Sengupta, Health Education Research

tboks in The Survey Krt, 2nd edrtion: The Survey Handbook, 2nd Arlene Fink Volume 1 /ISBN o· 76 19·2580·5 1pb How to Ask Survey Questions, 2nd Arlene Fink Volume 2/ 1SBN 0·7619·2579· 11pb How to Conduct SelfAdministered and Mail Surveys, 2nd Linda B. Bourque, Eve P. Fielder Volume 3/1SBN 0·76 19·2562·7/pb How to Conduct Telephone Surveys, 2nd Linda B. Bourque, Eve P. Fielder Volume 4/ 1SBN 0-7619 -2591 -0 1pb

How to Conduct In-Person Interviews for Surveys, 2nd Sabine Mertens Oishi Volume 51 ISBN 0·7619·2570·8 1pb

How to Manage, Analyze,

and Interpret Survey Data,

2nd

Arlene Fink

Volume 9/ 1SBN 0·76 19·2576·71 pb

How to Design Survey Studies, 2nd Arlene Fink Volume 6/ 1SBN 0·7619·2578·3 1 pb

How to Report on Surveys,

2nd

Arlene Fink

Volume 10 IISBN 0·7619-2575-9 1pb

How to Sample in Surveys, 2nd Arlene Fink Volume 7/ 1SBN 0·7619·2577·5 / pb

TSC 2 YKIT

How to Assess and Interpret Survey Psychometrics, 2nd Mark S Litwin Volume 8/ 1SBN 0-7619-2561-9 1pb

I

THE COMPLETE SURVEY KIT Volumes 1-10/iSBN 0-7619-2510-4

ISBN 0-7619-2561-9

Visit our Web site at www .sagepublications.com

f$\ ~!~o~a/~d~~)i~~;1!l91)3essiona/ ~ Thousand Oaks . London • New Delhi

I

Publisher 9

~ 111 1111111

780761

925613

90000>

11 11111111111/ 11