_f'( ./l
ii_
StataReferenceManual Relea 7 Volume2 H-P Stata Press CollegeStation,Texas
Stata Press, 4905 Lakeway Dri...
35 downloads
2929 Views
56MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
_f'( ./l
ii_
StataReferenceManual Relea 7 Volume2 H-P Stata Press CollegeStation,Texas
Stata Press, 4905 Lakeway Drive, College Station, Texas 77845 Copyright C) 1985-2001 All fights reserved Version 7.0
by Stata Corporation
Typeset in TEX Printed in the United States of America l0 9 8 7 6 ISBN ISBN 1SBN ISBN ISBN
5 4 3 2
1-881228-47-9 1-88t228-48-7 1-881228-49-5 1-881228-50-9 1-881228-51-7
1
(volumes 1-4) (volume I) (volume 2) (volume 3) (volume 4)
This manual is protected by copyright. All rights are reserved. No part of this manual may be reproduced, stored in a retrieval system, or transcribed, in any form or by any means--electronic, mechanical, photocopying, recording, or otherwise--without the prior written permission of Stata Corporation (StataCorp), StataCorp provides this manual "as is" without warranty of any kind, either expressed or implied, including, bul not limited to the implied warranties of merchantability and fitness for a particular purpose. StataCorp may make improvements and/or changes in the product(s) and the programCsl described in this manual at any time and without notice. The software described in this manual is furnished under a license agreement or nondiscIosure agreement. The software may be copied only in accordance with the terms of the agreement. It is against the law m copy the software onto CD. disk, diskette, tape, or any other medium for any purpose other than backup or archival purposes. The automobile dataset appearing on the accompanying media is Copyright (_) 1979, 1993 by Consumers Union of U.S.. Inc.. Yonkers. NY 10703-1057 and is reproduced by permission from CONSUMER REPORTS, April 1979, April 1993. The Stata for Windows installation software was produced using Wise Installation System, which is Copyright @ 1994-2000 Wise Solutions. Inc, Portions of the Macintosh installation software are Copyright (_ 1990-2000 Aladdin Systems, Inc.. and Raymond Lau. Stata is a registered trademark and NetCourse is a trademark of Stata Corporation. Alpha and DEC are trademarks of Compaq Computer Corporation. AT&T is a registered trademark of American Telephone and Telegraph Company. HP-UX and HP LaserJet are registered trademarks of Hewlett-Packard Company. IBM and 0S/2 are registered trademarks and AIX. PowerPC. and RISC System/6000 are trademarks of International Business Machines Corporation. Linux is a registered trademark of Linus Torvalds. Macintosh is a registered trademark and Power Macintosh is a trademark of Apple Computer. Inc. MS-I)OS. Microsoft, and Windows are registered trademarks of Microsoft Corporation. Pentium is a trademark of Intel Corporation. PostScript and Display PostScript are registered trademarks of Adobe Systems. Inc. SPARC is a registered trademark of SPARC International. Inc Star/Transfer is a trademark of Circle Systems. Sun, SunOS. Sunview, Solaris. and NFS are trademarks or registered trademarks of Sun Microsysrems. Inc. TEN is a trademark of the American Mathematical Society. UNIX and OPEN LOOK are registered trademarks and X Window System is a trademark of The Open Group Limited. WordPerfect is a registered trademark of Corel Corporation.
The suggested citation for this software is StataCorp. 2001_ Stata Statistical Software: Release 7.0. College Station. TX: Stata Corporation.
i I
i
Title hadimvo -- Identify multivariate outliers
Syntax hadiravo varlist [if exp] [in range], g_en_rate(newvarl
[newvar2]) [p(#)]
;Description hadimvoidentifies multiple outtiers in multiv_ate data using the method of Hadi (1992, 1994), creating newvarl equal to 1 if an observation is an "outlier" and 0 otherwise. Optionally, newvar2 can also be created containing the distances from!the basic subset.
Options E
generate (newvarl [newvar2]) is not optional; it identifiesthe new variable(s) to be created. Whether you specify two variables or one, however, is dptional, newvarl--which is required--will create newvarl containing 1 if the observation is an outlier in the Hadi sense and 0 otherwise: Specifying gen (odd) would call this variable odd. newvai'2,if specified, will also create newvar2 containing the distances (not the distances squared) from the basic subset. Specifying gen (odd dist) creates odd and also creates dist containing the Hadi distances. p(#) specifies the sl_,mficance level for outlier cutoff; 0 < # < 1. The default is p(.05). Larger numbers identify a larger proportion of the sample as outliers. If # is specified greater than I. it is interpreted as a percent. Thus, p(5) is the Sameas p(.05).
Remarks Multivariate analysis techniques are commonly used to analyze data from many fields of study. The data often contain outliers. The search for subsets of the data which, if deleted, would change results markedly is known as the search for outli+rs,hadimvo provides one computer-intensive but practical method for identifying such observations. Classical outlier detection methods (e.g., Mahalhnobisdistance and Wilks' test) are powerful when the data contain only one outlier, but the power Of these methods decreases drastically when more than one outlying observation is present. The losisof power is usually due to what are known as masking and swamping problems (false negative and false positive decisions), but in addition, these methods often fail simply because they are affecied by the very observations they are supposed to identih,. Solutions to these problems often involve an unreasonable amount of calculation and therefore computer time. (Solutions involving hundreds of! millions of calculations for samples as small as 30 have been suggested.) The method developed:,{bY Hadi (1992, I994) attempts to surmount these problems and produce an answer, albeit second b{st, in finite time. A basic outline of the procedure is as follows_A measure of distance from an observation to a cluster of points is defined. A base cluster of r pdints is selected and then that cluster is continually redetined by taking the r + 1 points "closest" to the cluster as the new base cluster. This continues until some rule stops the redefinition of the clustei.
d:
.au=mvo
_
=o_nt.y
mUlZlVarlale
OUlllers
Ignoring many of the fine details, given k variables, the initial base cluster is defined as r = k + 1 points. The distance that is minimized in selecting these k + 1 points is a covariance-matrix distance on the variables with their medians removed. (We wilt use the language loosely; if we were being more precise, we would have said the distance is based on a matrix of second moments, but remember, the medians of the variables have been removed. We would also discuss how the k -r- 1 points must be of full column rank and how they would be expanded to include additional points if they are not.) Given the base cluster, a more standard mean-based center of the r-observation cluster is defined and the r + 1 observations closest in the covariance-matrix sense are chosen as a new base cluster. This is then repeated until the base cluster has r = int{(n + k + 1)/2} points. At this point, the method continues in much the same way, except a stopping rule based on the distance of the additional point and the user-specified p() is introduced. Simulation results are presented in Hadi (1994).
Examples hadimvo
price
• list
if odd
• summ
price
drop
gen(odd) /*
weight
if -odd
price
weight,
gen(odd
the
*/
outliers stats
for
clean
data
*/
D)
id=_n
graph
/* make
Did
graph price weight [w=D] graph price weight [w=i/D] summarize D, detail sort D list
list
/* summary
odd
hadimvo gen
weight,
make
hadimvo • regress
price
weight
price weight ... if -odd2
an index
/* index
plot
/* 2-way /* same,
scatter, outliers
variable
,/
of D
,/ outliers small
big
*/ ,/
D odd
mpg,
gen(odd2
Di)
p(.01)
Identifying outliers You have a theory about xz, x2, ..., xk which we will write as F(xl, x2,..., xk). Your theory might be that xl, x2, ..., xk are jointly distributed normally, perhaps with a particular mean and covariance matrix: or your theory might he that
xl =/31 T/32x2 + .-- + _kxk + u where u _ N(0, or2); or your theory might be
xl -/310 +/312x2+ ,,_14x4+ ua x2 -/320 + 1321xl+ _23xz + u2 or your theory might be anything else it does not matter. You have some data on x_, x2, ..., xk, which you will assume are generated by F(.). and from that data you plan to estimate the parameters (if any) of your theory and then test your theory in the sense of how well it explains the observed data.
h_dimvo-- Identifymultivariateoutliers
l
3
What if, however, some of your data are generated not by F(-) but by G(.), a different process? a a wages example, you For have theory on how are_assignedto employees in firm and. for the bulk of employees, that theory is correct. There are, hdwever, six employees at the top of the hierarchy for whom wages are set by a completely different process. Or, you have a theory on how individuals select different health insurance options except that, for a handful of individuals already diagnosed with serious illness, a different process controls ihe selection process. Or, you are testing a drug that reduces trauma after surgery except that, for a few patients with a high level of a particular protein, the drug has no effect. Or, in another drug experiment, some of the historical data are simply misrecorded. The data values generated by G(.) rather than F(.) are called contaminant observations. Of course, the analysis should be based only on the observations generated by F(.), but in practice we do not know which observations those are. In addition, if it happened by chance that some of the observations are within a reasonable distance _om the center of F(.), it becomes impossible to detennine whether they are contaminants. Accordingly.we adopt the following operational definition: Outliers are observations that do not conformto the pattern suggestedbythe majority of the observations in a dataset. Accordingly, observations generated b_ F(.) but located at the tail of F(.) are considered outliers. On the other hand, contaminants that are within a statistically reasonable distance from the center of F(.) are not considered outliers. It is well worth noting that outliership is strongly related to the completeness of the theory--a grand unified theory would have no outliers becafise it would explain all processes (including, one supposes, errors in recording the data). Grand uni_d theories, however, are difficult to come by and are most often developed by synthesizingthe results of many special theories. i / Theoretical work has tended to focus on one !pecial case: data containing only one outlier. As mentioned above, the single-outlier techniques ofteh fail to identify multiple outliers, even if applied recursively. One of the classic examples is the star t cluster data (a.k.a. Hertzsprung-Russell diagram) shown in the figure below (Rousseeuw and Leroy 1987, 27). For 47 stars, the data contain the (log) light intensity and the (log) effective temperature _t the star's surface. (For the sake of illustration, we treat the data here as bivafiate data--not as r_gression data--i.e., the two variables are treated ..... i . similarly with no dlstmcuon between which vanatile is dependent and which is independent.) This graph presents a scatter of the data along with two ellipses expected to contain 95% of the data. The larger ellipse is based on the mean and dovariance matrix of the full data. All 47 stars are inside the larger ellipse, indicating that classical iingle-case analysis fails to identify any outliers, The smaller ellipse is based on the mean and co_,afiancematrix of the data without the five stars identified by had±taro as outliers. These observaiions are located outside the smaller ellipse. The i dramatic effects of the outliers can be seen by comparing the two ellipses. The volume of the larger ellipse is much greater than that of the smaller one and the two ellipses have completely different orientations. In fact, their major axes are nearly orthogonal to each other; the larger ellipse indicates a negative correlation (r = -0.2) whereas the smalle_rellipse indicates a positive correlation (r = 0.7] (Theory would suggest a positive correlation: hot ihings glow.)
:i
(Graph on _ext page)
i
•
_r
Itc_u..vu
"-;"
lUt:lluly
inuiLivuna|e
OUtllers
I
I
8-
/
/ "_ =
/
o o
0 0
_o_ o -4
4"
0
\\
_.
// /
@__2"j
2
/
.//
F_Log
lemperalute
The single-outlier techniques make calculations for each observation under the assumption that it is the only outlier and the remaining n - 1 observations are generated by .F('.) producing a statistic for each of the n observations. Thinking about multiple oufliers is no more difficult. In the case of two outliers, consider all possible pairs of observations (there are n(nI)/2 of them) and, for each pair, make a calculation assuming the remaining n 2 observations are generated by F(-). For the three-outlier case, consider all possible triples of observations (there are rz(_ - 1)(n - 2)/(3 x 2) of them) and, for each triple, make a calculation assuming the remaining rL 3 observations are generated by F(-). Conceptually, this is easy but practically, it is difficult because of the rapidly increasing number of calculations required (there are also theoretical problems in determining how many outliers to test simultaneously). Techniques designed for detecting multiple outliers, therefore, make various simplifying assumptions to reduce the calculation burden and, along the way, lose some of the theoretical foundation. This loss, however, is no reason for ignoring the problem and the (admittedly second best) solutions available today. It is unreasonable to assume that outtiers do not occur in real data. If outliers exist in the data, they can distort parameter estimation, invalidate test statistics, and lead to incorrect statistical inference. The search for outliers is not merely to improve the estimates of the current model but also to provide valuable insight into the shortcomings of the current model. In addition, outliers themselves can sometimes provide valuable clues as to where more effort should be expended. In a drug experiment, for example, the patients excluded as outliers might well be further researched to understand why they do not fit the theory.
Multivariate, multiple outliers hadimvo is an example of a multivariate, multiple outlier technique. The multivariate aspect deserves some attention, In the single-equation regression techniques for identifying outliers, such as residuals and leverage, an important distinction is drawn between the dependent and independent variables--the b' and the x's in y = x/3 + u. The notion that the ff is a linear function of x can be exploited and. moreover, the fact that some point (Yi-xi) is "far" from the bulk of the other points has different meanings if that "farness" is due to ;ti or xi. A point that is far due to xi but, despite that, still close in the _/i given xi metric adds precision to the measurements of the coefficients and may not indicate a problem at all. In fact, if we have the luxury of designing the experiment, which means choosing the values of x a priori, we attempt to maximize the distance between the x's (within
h_dimvo-- Identify multivariateoutliers
5
the bounds of x we believe are covered by our {inear model) to maximize that precision. In that extreme case, the distance of xi carries no information as we set it prior to running the experiment. More recently, Hadi and Simonoff (1993) exploit _he structure of the linear model and suggest two
i
diagnostics). methods for identifying muhiple outliers v,hen the inodel is fitted to the data (also see [R] regression In the multivariate case, we do not know the strhcture of the model, so (y,, x+) is just a point and the y is treated no differently from an5"of the x's_a, fact which we emphasize by+writin_ the point as (xaz,x2i) or simply (Xi). The technique doeg assume, however, that the X's are multivariate normal o1"at least elliptically symmetric. This lead_ to a problem if some of the X's are functionally related to the other X's, such as the inclusion of x _nd x 2, interactions such as .r_x> or even dummy variables for multiple categories (in which one of the dummies being 1 means the other dummies must be 0). There is no good solution to this problrm. One idea, however, is to perform the analysis with and without the functionally related variables _ind to subject all observations identified for further study (see What to do with outliers below).
]
An implication of hadJ.mvo being a multivariaie technique is that it would be inappropriate to apply it to (9,x) when x is the result of experi_ntal design. The technique would know nothing of our design of x and would inappropriately treat "distance" in the x-metric the same as distance in the +j-metric. Even when x is inuhivariate norNal, unless y and x are treated similarly it may still be inappropriate to apply had±taro to (9, x)because of the different roles that _/ and x play in regression. However, one may apply had±taro on x to identify outliers which, in this case. are called leverage points. (We should also mention here that if had:i.mvo is applied to x when it contains + constants or any collinear variables, those variables will be correctly i'_nored, allowine_ the analxs_. ,_'s to continue.) It is also inappropriate to use hadimvo (and other outlier detection techniques) when the sample + size is too small, had:i.mvo uses a small-sample correction factor to adjust the covariance matrix of the "clean" subset. Because the quantity n - (3L:@ I) appears in the denominator of the correction factor, the sample size must be larger than 31,:+ _. Some authors would require the sample size to be at least 5h, i.e., at least five observations per vhriable, With these warnings, it is difficult to misapply {his tool assuming that you do not take the results as more than suggestive, hadimvo has a p () optioh that is a "significance level" for the outliers that i
are chosen. quote the term level b_cause, although has that beeninterpretation expended to really make We a significance level,significance approximations a_e involved and itgreat will effort not have i in all cases. It can be thought of as an index between 0 and 1, with increasing values resulting in the labeling of more obse_,'ations as outliers and _ith the suggestion that you select a number much as you would a significance level--it is roughly the probability of identifying any given point as an outlier if the data truly were multivariate normal. Nevertheless, the terms significance level or critical values should be taken with a grain of shlt. It is suggested that one examine a graphical display (e.g., an index plotl of the distance with berhaps different values of p(). The graphs give more information than a simple yes/no answer. FOr example, the graph may indicate that some of the observations (inliers or outliers) are only mar_nally so,
i }
What to do with outliers After a reading of the literature on outlier ddtection., many people are left with the incorrect impression that once outliers are identified, they, s_ould be deleted from the data and analysis should be continued. Automatic deletion (or even automatic down-weighting) of outliers is not ahvavs correct because outliers are not necessarily bad obser 'atlons. On the contrary,, if they are correct, they' may bc thc most informative points in the data. For _xample, they may indicate that the data do not
!:
wuaunmwvv
v
--
Igl_lltlly
illglIiVarla|e
come from a normally distributed techniques.
OUtllers
population,
as is commonly
assumed
by almost all multivariate
The proper use of this tool is to label the outliers and then subject the outtiers to further study, not simply to discard them and continue the analysis with the rest of the data. After further study, it may indeed turn out to be reasonable to discard the outliers, but some mention of the oufliers must certainly be made in the presentation of the final results. Other corrective actions may include correction of errors in the data, deletion or down-weighting of outliers, redesigning the experiment or sample survey, collecting more data, etc.
Saved Results hadimvo saves in r(): Scalars r(N)
number of outliers remaining
Methodsand Formulas hadimvo
is implemented
as an ado-file. Formulas are given in Hadi (I992,
1994).
Acknowledgment We would like to thank Ali S. Hadi of Comell University for his assistance
in writing hadimvo.
References Gould, W. W. and A. S. Hadi. 1993. smv6: Identifying multivariate outliers. Stata Technical Bulletin 11: 28-32. Reprinted in Stata TechnicalBulletin Reprints, vol. 2. pp. 163-168 Hadi, A. S. 1992. Identifying multiple outliers in multivariatedata. Journal of the Royal Statistical Society, Series B 54: 761-771. 1994. A modification of a method for the detection of outliers in multivariate samples. Journal of the Royal Statistical SocieD',Series B 56: 393-396. Hadi, A. S. and J. S. Simonoff. 1993. Procedures for the identificationof multiple outtiers in linear models. Journal of the American Statistical Association 88:1264 1272. Rousseeuw,P. J. and A. M. Leroy. 1987. Robust Regression and Outlier Detection. New York: John Wiley & Sons.
Also See Related:
JR] mvreg, [R] regression diagnostics,
[R] sureg
Title hausman -- Hausman specification test
Syntax hausman, save
hausman
[, [more
l!ess ] constant a_leqs skipeqs(eqtixt) i
sigmamore
p_rior(string)current (string)equations (matchlist)] hausman, clear where matchlist in equat ions () is
#:#[,#i#[,...]] For instance, equations (:1.: 1), equations(1
:_, 2: 2), or equations (1 :2).
i Description hausmanperforms Hausman's (t978) specificition test. /
Options save requests that Stata save the current estimation results, hausman will later compare these results with the estimation results from another model. A model must be saved in this fashion before a test against other models can be performed. more specifies that the most recently estimated model is the more efficient estimate. This is the default, less specifies that the most recently estimated model is the less efficient estimate. constant specifies that the estimated intercept(s) are to be included in the model comparison: by default, they are excluded. The default behavi_ is appropriate for models where the constant does not have a common interpretation across the two models. i alleqs specifies that all the equations in the mod_l be used to perform the Hausman test: by default. only the first equation is used. skipeqs(eqlist) specifies in eqlist the names of equations to be excluded from the test. Equation numbers are not allowed in this context as it is the equation names, along with the variable names. that are used to identify common coefficients. sigmamore allows you to specify that the two Cbvariancematrices used in the test be based on a common estimate of disturbance variance (cr2i--the variance from the fully efficient estimator. This option provides a proper estimate of the :contrast variance for so-called tests of e×ogeneity and over-identification in instrumental variablegregression: see Baltagi (1998, 29!). Note that this option can only be specified when both estimators save e(sigma) or e(rmse). !
prior (string) and current (string) are formattin_options that allow you to specify alternate wording for the "Prior" and "Current" default labels used to identify the columns of coefficients.
i i J
v
,JlauoIIIglll
equations
--
I ll;Ig;_lll(:ill
(matchlist)
If equations
:tl.l_l.;llli.;dl.l[.}l]
test
specifies• by number, the pairs of equations
that are to be compared.
() is not specified, then equations are matched on equation names.
equations() handles the situation where one does not. For instance, equations(l:2) means equations(i:1, 2:2) means equation 1 is to to be tested against equation 2. If equations() ignored.
estimator ases equation names and the other equation 1 is to be tested against equation 2. be tested against equation 1 and equation 2 is is specified, options alleqs and skipeqs are
clear discards the previously saved estimation resuhs and frees some memory; it is not necessary to specify hausmaxl, clear before specifying hausman, save.
Remarks hausm_n is a general implementation of Hausman's estimator that is known to be consistent with an estimator tested. The null hypothesis is that the efficient estimator the true parameters. If it is• there should be no systematic
(1978) specification test that compares an that is efficient under the assumption being is a consistent and efficient estimator of difference between the coefficients of the
efficient estimator and a comparison estimator that is known to be consistent for the true parameters. If the two models display a systematic difference in the estimated coefficients, then we have reason to doubt the assumptions on which the efficient estimator is based. To use hausman, you (estimate
the less efficient
• hausman,
save
(estimate • hausman
the fully
Alternatively,
efficient
model
)
model)
you can turn this around:
(estimate the fully efficient model) • hausman, • (estimate • hausman,
save the tess efficient
model)
less
> Example We are studying the factors that affect the wages of young women in the United States between 1970 and 1988 and have a panel-data sample of individual women over that time span.
(Continued
on next page)
lausman -- Hausman specificationtest
9
• describe Contains
data
from nlswork.dta
obs:
28,534
National i
Longitudinal
Young Women in 1968
14-26
Survey. years
of age
Z
vars :
6
size:
485,078
I Aug 2000 (88.4_ of memory
storage
09:48
fr_e)
display
value i
type
format
label!
idcode
imt
_8.0g
NLS id
year
byte
_8.0g
interview
age
byte
_8.0g
age in current
year
msp
byte
_8.0g
1 if married,
spouse
ttl_exp
float
_9.0g
total
work
experience
In_wage
float
_9.0g
In(wage/GNP
deflator)
variable
Sorted
name
by:
Note:
idcode
year
dataset
has
changed
since
last i
variable
label
year present
saved
We believe that a random-effects specification is @ppropriate for individual-level effects in our model. We estimate a fixed-effects model that will capture all temporally constant individual-level effects. . xtreg
In_wage
Fixed-effects
age msp ttl_exp,
(within)
fe Number
of obs
=
28494
(i) : idcode
Number
of groups
=
4710
within
= 0.1373
Obs per
between overall
= 0.2571 = 0.1800
Group
variable
_-sq:
corr(u_i,
Xb)
in_wage
regression
= 0.1476
Coef.
Std. Err.
i
t
group:
min =
1
avg = max =
6.0 15
F(3,23781)
=
Prob
=
P>[tl
> F
[95Z Conf.
1262.01 0.0000
Interval]
age
-.005485
.000837
_6.55
0.000
-.0071256
-.0038443
msp
.0033427
.0054868
!0.61
0.542
-.0074118
.0140971
ttl_exp
,0383604
.0012416
_0.90
0.000
.0359268
.0407941
_cons
1.593953
.0177538
_9.78
0.000
1.559154
1.628752
sigma_u
.37674223
sigma_e rho
.29751014 .61591044
F test
that
all u_i=O:
! (fraction
i of!variance ii i
F(4709,23781)ii =
due to u_i)
7.76
Prob
> F = 0.0000
{J
We assume that this model is consistent for the true parameters and save the results by typing . hausman,
save
Now, we estimate a random-effects model _s a fully efficient specification of the individual effects under the assumption that they follow a rahdom-normal distribution. These estimates are then compared to the previously saved results using tile hausman command.
1
•_
Jtuu_w
mvow I --
! lOU_I
1I_1
i _}.ll_'_ll
II,;BllOn
Ie$l
I
• xtreg
in_wage
age
msp
ttl_exp,
re
Random-effects
GLS
regression
Number
of
obs
=
28494
Group
variable
(i)
: idcode
Number
of
groups
=
4710
R-sq:
within
= 0.1373
between overall
= 0.2552 = 0.1Z97
effects
u_i
Random
corr(u_i,
X)
in_wage
Obs per
group:
min
=
1
avg max
= =
6.0 15
~ Gaussian
Wald
chi2(S)
=
5100.33
= 0 (assumed)
Prob
> chi2
=
0.0000
Coef.
Std.
Err.
z
P>Izl
[95Z
Conf.
Interval]
age
-.0069749
.0006882
-10.13
0.000
-.0083238
-.0056259
msp
.0046594
.0051012
0.91
0.361
-.0053387
.0146575
ttlexp _cons
.0429635 1.609916
.0010169 .0159176
42.25 101.14
0.000 0.000
.0409704 1.578718
.0449567 1.641114
sigma_u
.32648519
sigma_e rho
.29751014 .54633481
(fraction
of
variance
to
due
u_i)
• hausman Coefficients (b) Prior
(B) Current
(b-B) Difference
sqrt(diag(V_b-V_B)) S.E.
age
-.005485
-.0069749
msp
.0033427
.0046594
-.0013167
.0020206
.0383604
.0429635
-.0046031
.0007124
tt1_exp
b = less
efficient
B = fully Test:
Ho:
difference chi2(3)
.0014899
estimates
efficient
obtained
estimates
in coefficients
obtained
not
.0004764
previously from
from
xtreg
xtreg
systematic
= (b-B)'[(V_b-V_B)'(-I)](b-B) = 275.44 Prob>chi2
=
0.0000
Using the current specification, our initial hypothesis that the individual-level effects are adequately modeled by a random-effects model is resoundingly rejected. We realize, of course, that this result is based on the rest of our model specification and that it is entirely possible that random effects might be appropriate for some alternate model of wages. hausman is a generic implementation of the Hausman test and assumes that the user knows exactly what they want tested. The test between random- and fixed-effects is so common that Stata provides a special command for use after xtreg. We could have obtained the above test in a slightly different format by typing xthausman Hausman
specification
test -
Coefficients Fixed
in_wage
Random
Effects
Effects
Difference
age
-. 005485
-. 0069749
msp
.0033427
.0046594
-.0013167
tt l_exp
.0383604
.0429635
-.0046031
.0014899
t hausman--
Hausmanspecificationtest
11
i Test:
Ho:
difference
in coefficients
not
chi2(3)
= (b-B)'ES'(-!)]Cb-8),
Prob>chi2
= =
systematic
S = (S_fe - S_re)
275.44 O.0000
q
Example A stringent assumption of multinomial and cbnditional logit models is that outcome categories • i for the model have the property of independence of irrelevant alternatives (IIA). Stated simply, this assumption requires that the inclusion or exclusmn of categories does not affect the relative risks associated with the regressors in the remaining citegories. One classic example of a situation where thi_ assumption would be violated involves choice of transportation mode: see McFadden (1974). For s_mplicity, postulate a transportation model with the four possible outcomes: rides a train to work, t_es a bus to work, drives the Ford to work, and drives the Chevrolet to work• Clearly "drives the _ord" is a closer substitute to' drives the Chevrolet" than it is to "rides a train" (at least for most people). This means that excluding "drives the Ford" from the model could be expected to affect the relative risks of the remaining options and the model would not obey the IIA assumption. i Using the data presented in [R] mlogit, we w_l use a simplified model to test for IIA. Choice of insurance type among indemnity, prepaid, and un_sured is modeled as a function of age and gender. The indemnity category is allowed to be the base _ategory and the model including all three outcomes is estimated. i
• mlogit insure age male Iteration O: Iteration I: Iteration 2:
log likelihood = -555.854_6 log likelihood = -551.329t3 log likelihood = -551.32802
Multinomial regression
Number of obs LR chi2(4) Prob > chi2
= = =
615 9,05 0.0598
Log likelihood = -551.32802
Pseudo R2
=
0.0081
,i
insure
Coef.
Std. Err.
Prepai6 age male
-. 0100251 .5095767
.0060181 ,1977893
_cons
.2633838
.2787574
Uninsure age male _cons
z i i _1.67 2.58
I
O, 94
P>lz]
[95Y,Conf. Interval]
O. 096 O. 010
-. 0218204 .1219148
.0017702 .8972345
O. 345
-. 2829708
.8097383
0.648 O. 189 0.001
-.027501 -.2343477 -2.797504
.017116 i.184057 -.7161824
i
i -.0051925 .4748547 -1.756843
.0113821 .3618446 .5309591
40.46 iI.31 43.31 :i
(Outcome insure==Indem is the comparisonlgrouP)
]
• hausman, save
i i
!
Under the IIA assumption, we would expect no _;ystematic change in the coefficients if we excluded one of the outcomes from the model. (For an ektensive discussion, see Hausman and McFadden. 1984.) We re-estimate the model, excluding the!uninsured outcome, and perform a Hausman test
i i
. mlogit insure age male if insure-=
against the fully efficient full model.
"U_insure":insure
I
_:r,,
..
..u_.,..--n_usman
specmcaIlontest
Iteration
O:
log
likelihood
=
-394.8693
Iteration
I:
log likelihood
=
-390.4871
I_eration
2:
log likelihood
= -390.48643
Multinomial
Log
regression
likelihood
= -390.48643
Number of obs LR chi2(2)
= =
Prob
=
0.0125
=
0.0111
> chi2
Pseudo
Std.
Err.
z
R2
P>Iz}
[95Z
Conf.
570 8.77
insure
Coef.
Interval]
age male
-.0101521 .5144003
.0060049 .1981735
-1.69 2.60
0.091 0.009
-.0219214 .1259875
.0016173 .9028132
_cons
.2678043
.2775562
0.96
0.335
-.2761959
.8118046
Prepaid
(Outcome
insure==Indem
hausman,
alleqs
is the
less
comparison
group)
constant
Coefficients (b) Current
(B) Prior
(b-B) Difference
sqrt (diag (V_b-V_B)) S.E.
age male
-.0101521 .5144003
-.0100251 .509574Z
-.0001269 .0048256
_cons
.2678043
.2633838
.0044205
b = less
efficient
B = fully Test:
Ho:
efficient
difference
estimates
in coefficients
chi2(3)
obtained
estimates
from
obtained
not
.012334
mlogit
previously
from
mlogit
systematic
= (b-B)'[(V_b-g_B)_(-l)](b-B) = 0.08 Prob>chi2
=
0.9944
First, note that the somewhat used to identify the "Uninsured"
subtle syntax of the if condition on the mlogit command was simply category using the insure value label; see [U] 15.6.3 Value label.
On examining the output has been violated.
hausman,
Second, mlogit
from
since the Hausman
requires
test is a standardized
that the base category
most frequem category use the basecategory()
we see that there
be the same
is no evidence
comparison
of model
in both competing
that
the IIA assumption
coefficients,
models.
using
it with
In particular,
if the
(the default base category) is being removed to test for IIA, then you must option in mlogit to manually set the base category to something else.
The missing values for the square root of the diagonal of the covariance matrix of the differences is not comforting but is also not surprising. This covariance matrix is guaranteed to be positive definite only asymptotically and assurances are not made about the diagonal elements, Negative values _ong the diagonal are possible, and the fourth column of the table is provided maitdy for descriptive use. We can also perform the Hausman • mlogit
insure
age
male
IIA
if insure
test against the remaining alternative in the model. ~=
"Prepaid":insure
Iteration
O:
log
likelihood
= -132.59915
Iteration
i:
log
likelihood
= -131.78009
Iteration
2:
log
likelihood
= -131.76808
Iteration
3:
log
likelihood
= -131.76807
Multinomial
Log
regression
likelihood
= -131.76807
Number of obs LR chi2(2)
= =
Prob
> chi2
=
0.4356
R2
=
0.0063
Pseudo
338 i.66
lausman -- Hausmanspecificationtest
it
13
i
insure
Coef.
Std. Err.
z
P>Jzl
[95Z Conf. Interval]
Uninsure age male _cons
-.0041055 .4591072 -1. 801774
.0115807 .3595663 .5474476
_0.35 I.28 43.29
O.723 O. 202 O. 001
-.0268033 -.2456298 -2. 874752
.0185923 I.163844 -, 7287968
(Outcome insure==Indem is the comparison group)
• hausman, alleqs less constant -Coefficients -(b) Current age male _cons
-.0041055 .4591072 -1.801774
i
(B) Prior
(b-B) sqrt(diag(V b-V_B) ) Difference S.E.
-.0051925 .4748547 -1.756843
.001087 -.0157475 -.0449311
i
.0021357 .1333464
I
b = less efficient estim_ttesobtained from mlogit B = fully efficient estiz_atesobtained previously from mlogit Test:
Ho:
difference in coefficients not systematic chi2(3)
= (b-B)'[(V_b-__B)'(-I)](b-B) = -0.18 dhi2 model estimated on these } _ata fails to meet the asymptotic $ssumptions of the Hausman test i
/
In this case, the X2 statistic is actually negati_'e. We might interpret this as strong evidence that we cannot reject the null hypothesis. Such a result is not an unusual outcome for the Hausman test, particularly when the sample is relatively small'there are only 45 uninsured individuals in this dataset. Z
Are we surprised by the results of the Hausn_an test in this example? Not really. Judging from the z-statistics on the original multinomiat logit model, we were struggling to identify any structure in the data with the current specification. Even when we were willing to make the assumption of IIA I and estimate the most efficient model under this assumption, few of the effects could be identified as statistically different from those on the base category. Trying to base a Hausman test on a contrast (difference) between two poor estimates is just a_king too much of the existing data. <1 For an example applying the Hausman test to the endogeneity of variables in a simultaneous system, see [R] irreg.
]
,,
Saved Results hausman
saves
in r ()"
i ]
Scalars
i
r (chi2)
_2
r(df)
degrees of freOdom for the statistic
r(p)
p-value for the x -_
! ]
1:,,
!_
,,_
nausman -- Nausman specification test
Acknowledgment Portions of hausman Netherlands.
'
are based on an early implementation
by Jeroen Weesie, Utrecht University,
Methods and Formulas hausman
is implemented
as an ado-file.
The Hausman statistic is distributed
as X 2 and is computed
as
//- (/3c-/_e)'(_ - _)-1(/_c 9e) where tic 3e V_ V_
is is is is
the the the the
coefficient vector from the consistent estimator coefficient vector from the efficient estimator covariance matrix of the coefficients from the consistent estimator covariance matrix of the coefficients from the efficient estimator
In cases where the difference in the variance matrices is not positive definite, a Moore-Penrose generalized inverse is used. As noted in Gourieroux and Monfort (1989, 125-128), choice of generalized inverse is not important asymptotically. The degrees of freedom for the statistic are the rank of the difference in the variance matrices. When the difference is positive definite, this is the number of common coefficients in the models being compared.
References Baltagi, B. H. 1998. Econometrics. New York:Springer-Verlag. Gourieroux, H. and A. Monfort. 1989. Statistics and Econometric Modets, Vol. 2. New York: Springer-Verlag. Hausman, J. 1978. Specificationtests in econometrics. Econometrica46: 1251-1271. Hausman, J. and D. McFadden. 1984. Specificationtests in eco_ometncs. Econometrica 52. 12t9-1240. McFadden, D. 1974. Measurement of urban travel demand. Journal of Public Economics 3: 303-328.
Also See Related:
i:
[R| lrtest, [R] test, [R] xtreg, [R] xtregar
Title
[
i
hq_-kman -- Heckman selection model II1
[I
l
J
l
i
_
/I
I
I
'
11
,Syntax
i
Basic syntax heckman depvar
[vartist],
se!oct(varlis4)
[varlist],
setect(depvars!=
[ twostep
]
or
hectman
depvar
varlists)
[ twostep
]
Full syntax for maximum likelihood estimates only heckraan depvar
[varlist] [weight] [if
exp]] [in range],
select(
[ depvars = ] varlists [, off_et(varname)
[ robust
cluster
noconstant
] )
('varname) score (nev_'varlist) n_sshazard (newvarname)
L--
!
mills(newvarname)
offset(varname)
_oconstant constraints(numlist)
first
noskip level(#) iterate(O)nolog ldaximize_options ] /
Full syntax for Heckman's two-step consistent es_mates only i heckman depvar select(
[varlist] [ifexp] [in range],
[depvars
= ]varlists
,[ nshazard(newvalwame) L --
[ rhosigma by ...
I rhotrunc
: may be used with heckman;
[,
twostep
noccalstant
mills(newvamame) --
] rholimited
]
)
noconstant
rhoforce
first
level(#)
--
] ]
see [R] by.
pweights, aweights, fweights, and i_eights are allowed with maximum likelihood estimation: see [U] 14.1.6 weight. No weights are allowed if twostep is specified. heckman
shares the features of all estimation commands; _ee [U] 23 Estimation
and post-estimation
command_.
i ] i
!Syntaxfor predict predict [type] newvarname where statistic is
[il
exp] [in range]
[, statistic nooffset ]
-Ii _''
1_
neci(man -- Heckman selection model
xb
xjb.
ycond
E (yj IYj observed)
__ yexpected
E
nshazard psel xbsel stdpsel
or mills
* (yj),
fitted values (the default)
yj taken to be 0 where unobserved
nonselection hazard (also called inverse of Mills" ratio) P(yj observed) linear prediction for selection equation standard error of the linear prediction for selection equation
pr(a,b)
Pr(yj
[ a < yj < b)
e(a,b)
E(yj
ystar(a,b) __ strip stdf
E (yj), * yj* = max{a, min(yj,b)} standard error of the prediction standard error of the forecast
t a < yj < b)
where a and b may be numbers or variables; a equal to ' " means -oc: These statistics are available the estimation sample.
both in and om of sample;
wpe
predict
...
if
b equal to ".' means +oo.
e(sa_'nple)
. ..
if wanted only for
Description hecl_nan estimates regression models with selection using either Heckman's estimator or full maximum-likelihood.
two-step consistent
Options select (...) specifies the variables and options for the selection equation. specifying a Heckman model and is not optional. twostep specifies that Heckman's (1979) two-step errors, and covariance matrix are to be produced.
efficient estimates
It is an integral part of
of the parameters,
standard
robust specifies that the Huber/White/sandwich estlmator of the variance is to be used in place of the conventional MLE variance estimator, robust combined with cluster() further allows observations clusters).
which are not independent
If you specify pweights,
robust
within cluster (although they must be independent
is implied; see IU] 23,11 Obtaining
robust
variance
between estimates.
cluster(varname) specifies that the observations are independent across groups (clusters) but not necessarily independent within groups, varname specities to which group each observation belongs. cluster() affects the estimated standard errors and variance-covariance matrix of the estimators (VCE), but not the estimated coefficients, cluster() estimates for unstratified cluster-sampled data. cluster() cluster()
implies robust; by itself.
that is. specifying
can be used with pweights robust
cluster()
is equivalent
to produce to typing
score (newvarlist) creates a new variable, or a set of new variables, containing the contributions the scores for each equation and ancillary parameter in the model.
to
_heckman- Heckman selection model
17
....
The first new variable specified will contain uij = OlnLj/O(xj_) for each observation j in the sample, where lnLj is the jth observation's contribution to the log likelihood. • i The second new variable: u2j = OlnLj/O(zj_) The third: u3j = OlnLj /O(atanh p) The fourth: u4j = OInL.j/O(ln a)
i If only one variable is specified, only the first score is computed; if two variables are specified, only the first two scores are computed: and soon. i The jth observation's contribution to the score! vector is i
{01,,L/o 01nLj/0(-y) 01nL/0(atanh p) 01nLj/O(ln}= ( ljxj , jzj ',3:j ) !i
The score vector can be obtained by summing over j; see [U] 23.12 Obtaining
scores.
nshazard(newvarname) and mills (newvarnamk) are synonyms; either will create a new variable containing the nonselection hazard--what Hecl_an i (1979) referred to as the inverse of the Mills' ratio--from the selection equation. The nonselection hazard is computed from the estimated parameters of the selection equation. offset (varname) is a rarely used option that spdcifies a variable to be added directly to Xb. This option may be specified on the regession equOtion, the selection equation, or both. noconstant omits the constant term from the equations. This option may be specified on the regression equation, the selection equation, or both• i constraints(numtist) specifies by .number • the • linear constraints to be applied during estimation, i The default is to perform unconstrained est_matmn. Constraints are specified using the constraint command; see [R] constraint. See [R] reg3 for ihe use of constraints in multiple-equation contexts. constraints (numtist) may not be specified With twostep. I
first specifies that the first-step probit estimatds of the selection equation be displayed prior to estimation, i noskip specifies that a full maximum-likelihood model with only a constant for the regression equation be estimated. This model is not di@ayed but is used as the base model to compute a likelihood-ratio test for the model test statistic displayed in the estimation header. By default, the overall model test statistic is an asympto!ically equivalent Watd test of all the parameters in the regression equation being zero (exceptthe constant). For many models, this option can substantially increase estimation time.
_ ]
level(#)specifies the confidence level, in percent, for confidence intervals. The default is level(95) or as set by set level; see [U] 23.5 Specifying the width of confidence intervals. iterate(0) produces Heckman's (1979) two-step parameter estimates with standard errors computed from the inverse Hessian of the full information! matrix at the two-step solution for the parameters. As an alternative, the twostep option comput¢s Heckman's two-step consistent estimates of the standard errors, iterate(#) can also be used tb restrict the maximum number of iterations during optimization; see [R] rholimited, maximize. rhosigma, rhotrunc, and rhofdrce are rarely used options to specify how the two-step estimator, option twostep, handles _nusual cases where the two-step estimate of p is outside the admissible range for a correlation, [_ 1, 1]. When abs(p) > 1, the two-step estimate of the coefficient variance-covariance matrix may'not be positive definite, and thus may be unusable for testing. The default is rhosigma. :i rhotrunc specifies that p be truncated to lie in the range [- 1, 1]. If the two-step estimate is less than -1, p is set to -I: if the two-step estimate is greater than 1, p is set to 1. This truncated value of p is used in all computations to estimate the two-step covariance matrix. i
]
i
i
1 i
_:l:
,..
,,,_..R,,,,:,. -- nuuKman SeleC'[ion moclel rhosigmaspecifies that p be truncated, as with option rhotrunc,and that the estimate of o"be made consistent with _', the truncated estimate of p. So, _ =/3,._; see Methods and Formulas for the definition of/3 m. Both the truncated p and the new estimate of _ are used in all computations to estimate the two-step covariance matrix.
rholimited specifies that p be truncated only in the computation of the diagonal matrix D as it enters Vtwostep and Q; see Methods and Formulas. In all other computations, the untruncated estimate of p is used. rhoforee specifies that the two-step estimate of p be retained even if it is outside the admissible range for a correlation. This may, in rare cases, lead to a nonpositive-definite covariance matrix. These options have no effect when estimation is by maximum likelihood, the default. They also have no effect when the two-step estimate of p is in the range [ - 1,1 ], nolog suppresses the iteration log. maximize._options control the maximization process; see [R] maximize. You will likely never need to specify any of these options except for iterate(0) and possibly difficult.If the iteration log shows many "not concave" messages and it is taking many iterations to converge, you may want to try using the difficult option and see if that helps it converge in fewer steps.
Options for predict xb, the default, calculates the linear prediction xjb. ycond calculates the expected value of the dependent variable conditional on the dependent variable being observed, i.e., selected; E(_Ij [ yj observed). yexpected calculates the expected value of the dependent variable (y_) where that value is taken to be 0 when it is expected to he unobserved: y; = Pr(yj observed)E(yj _ yj observed). The assumption of 0 is valid for many cases where nonselection implies nonparticipation (e.g., unobserved wage levels, insurance claims from those who are uninsured, etc.) but may be inappropriate for some problems (e.g., unobserved disease incidence). nshazardand millsare synonyms; either calculates the nonselection hazard what Heckman (1979) referred to as the inverse of the Mills' ratio--from the selection equation. psel calculates the probability of selection (or being observed): P(yj observed) = Pr(zj7 + u2j > 0). xbsel calculates the linear prediction for the selection equation. stdpsel
calculates the standard error of the linear prediction for the selection equation.
pr(a,b) calculates Pr(a < xjb q- Ul < b), the probability that yjlxj interval (a, b).
would be observed in the
a and b may be specified as numbers or variable names; lb and ub are variable names; pr(20,30) calculates Pr(20 < xjb + Ul < 30); pr(lb, ub) calculates Pr(lb < x_b--r ul < ub); and pr(20,ub) calculates Pr(20 < xjb + zh < ub). a =. means -oc; pr(. ,30) calculates Pr(xjb + "u-1< 30); pr(lb,30) calculates Pr(xjb + ul < 30) in observations for which lb = . (and calculates Pr(/b < xjb + ul < 30) elsewhere).
•
iii
b =. means +ec; pr(20,. ) calculates Pr(xjb + ul > 20): pr(20,ub) calculates Pr(xjb + zh > 20) in observations for which ub -- . (and calculates Pr(20 < xjb + ul < ub) elsewhere).
"
_eckman -- Heckman selection model e(a,b) calculates E(xjb-1-ulla < xjb + ul _ b), the expected value of yjlxj yj Ixj being in the interval (a, b), which is to shy, yj [xj is censored. a and b are specified as ffley are for prO.
19
conditional on
ystar(a,b) calculates E(y_) where yj = a ifixjb + uj _ a, yj. = b if xjb + uj > b, and y_ = xjb + uj otherwise, which is to say, y_ i s truncated, a and b are specified as they are for prO. stdp calculates the standard error of the prediction. It can be thought of as the standard error of the predicted expected value or mean for the observation's covariate pattern. This is also referred to as the standard error of the fitted value. stdf calculates the standard error of the forecast. !This is the standard error of the point prediction for a single observation. It is commonly referrefl to as the standard error of the future or forecast value. By construction, the standard errors pr_uced by stdf are always larger than those by stdp; see [R] regress Methods and Formulas. noofgset is relevant when you specify offset (vilrname) for heckman. It modifies the calculations made by predict so that they ignore the offsdt variable; the linear prediction is treated as xjb rather than xjb + offsetj.
Remarks The Heckman selection model (Gronau 1974, Lewis 1974, Heckman 1976) assumes that there exists an underlying regression relationship Yj = xj_ + ulj
regression
equation
The dependent variable, however, is not always observed. Rather, the dependent variable for observation j is observed if zj'y + u2j > 0
selection
equation
where
corr( ,
=p
When p ¢i O, standard regression techniques applied to the first equation yield biased results. hectman provides consistenL asymptotically efficient estimates for all the parameters in such models.
t i I
In one classic example, the first equation describes the wages of women. Women choose whether to work and thus, from our point of view as researdhers, whether we observe their wages in our data. tf women made this decision randomly, we could ignore the fact that not all wages are observed and use ordinary regression to estimate a wage model. Such a random-participation-in-the-labor-force assumption, however, is unlikely to be true; wome_ who would have tow wages may be unlikely to choose to work and thus the sample of observed wages is biased upward. In the jargon of economics, wage greater women choose Thus, not toit work their that personal than have the wage by employers. is alsowhen possible wome_reservation who choose not isto work could even offered higher reservation wages. One could tell a story that competency is related to wages, but competency is rewarded more at home than in the labor force. offer wages than those who do work--they
may _ave high offer wages, but they have even higher
.,
r.u
.¢-,,;_lu|all
--
nls;Klllafl
lseteCl[ion moael
In any case, in this problem--which is the paradigm for most such problems--a solution can be found if there are some variables that strongly affect the chances for observation (the reservation wage) but not the outcome under study (the offer wage). Such a variable might be the number of children in the home. (Theoretically, one does not need such identifying variables, but without them, one is depending on functional form to identify the model. It would be difficult for anyone to take such results seriously since the functional-form assumptions have no firm basis in theory.)
_>Example In the syntax for heckman, depvar and varlist are the dependent variable and regressors for the underlying regression model to be estimated (y = X/_) and varlis% are the variables (Z) thought to determine whether depvar is observed or unobserved (selected or not selected). In our female wage example, number of children at home would be included in the second list. By default, heckman will assume that missing values (see [U]15.2.1 Missing values) of depvar imply that the dependent variable is unobserved (not selected). With some datasets, it is more convenient to specify a binary variable (depvars) that identifies the observations for which the dependent is observed/selected (depvars¢ O) or not observed (depvars= 0); beckman will accommodate either type of data. We have a (fictional) dataset on 2,000 women, of whom 1.343 work: • summarize age educ married children wage Variable
Obs
Mean
age education married children wage
2000 2000 2000 2000 1343
36.208 13.084 .6705 1.6445 23.69217
Std. Dev. 8. 28656 3. 045912 .4701492 1. 398963 6.305374
Min
Max
20 10 0 0 5,88497
59 20 1 5 45.80979
We will assume that the hourly wage is a function of education and age whereas the likelihood of working (the likelihood of the wage being observed) is a function of marital status, number of children at home, and (implicitly) the wage (via the inclusion of age and education which we think determine the wage):
(Continued on next page)
hbckman-Heckman selecUonmodel i! • heckman
wage educ
Iteration Iteration Iteration
O: t: 2:
log log log
Heckman selection
age, select(married likelihood likelihood likelihood
ch_dren
21
educ age)
= -5178.7009 = -5178.3049 = -5178.3045
model
(regression model wi_h sample selection)
Log likelihood = -5178.304 Coef.
Std, Err.
wage education
.9899537
.0532565
age _cons
.2131294 ,4857752
.0206031 I.077037
z
Number of obs
=
2000
Censored obs Uncensored obs
= =
657 1343
Weld chi2(2)
=
508.44
Prob > chi2
=
0.0000
P>Izl
[95_ Conf. Interval]
18.59
O,000
.8855729
I.094334
I0.B4 0.45
O,000 O,652
.1727481 -I.625179
.2535108 2.59673
.5772647 .4931601
'I
select married
.44_51721
.0673954
6 .B1
O.000
.3130794
children education age _cons
,43,87068 .05.57318 .0365098 -2.491015
.0277828 .0107349 .0041533 .1893402
15.79 5.19 8.79 -13.116
O.000 0.000 O.000 O.000
.3842534 .0346917 ,0283694 -2.862115
.0767718 .0446502 -2.119915
i i
/athrho
.8742086
.1014225
/Insigma
I.792559
,027598
rho
.7035061
sigma lambda
6.004797 4.224412
........
.6754241
I.072993
I.738468 /
I,84665
.0512264
.5885365
.7905862
.1657202 .3992265
5.68862 3.441942
6.338548 5.006881
LR test of indep, eqns. (rho = 0):
8 .!62 O.000 64.95
O.000
chi2(i) =
61.20
Prob > chi2 = 0.0000
heckman assumes thatwage is the dependent vada_]e and thatthe firstvariablelist(educ and age) are the determinants of wage. The variablesspecifiedin the select() option (married, children.
educ,andage)areassumedtodetermine whetherihedependent variable isobserved(theselection equation). Thus, we estimated the model wage = fi0+ flledUc + fl2age+ ul andwe assumedthatwage isobservedif 70 + 71married
+ 72children
_ 73educ + 74age + u2 > 0
where ut and u2 have correlation p.
l t
The reported results for the wage equation are interpreted exactly as though we observed wage data for all women in the sample; the coefficients on age and education level represent the estimated marginal effects of the regressors in the underlying regression equation. The results lbr the two ancillary parameters require some explanation, hec_anan does not directly estimate p: to constrain ,o within its valid limits, and for numerical stabiliiv during optimization, it estimates the inverse hyperbolic tangent of p:
atanh p = _
1-_/
I,
22
heckman -- Heckman selection model
This estimate is reported as /athrho. In the bottom panel of the output, heckman undoes this transformation for you: the estimated value of p is .7035061. The standard error for p is computed using the delta method and its confidence intervals are the transformed intervals of/aghrho. Similarly, or, the standard error of the residual in the wage equation, numerical stability, heckman instead estimates In or. The untransformed of the output: 6.004797.
is not directly estimated; for sigma is reported at the end
Finally, some researchers--especially economists are used to the selectivity effect summarized not by p but by A = per. heckman reports this, too, along with an estimate of the standard error and confidence interval.
q
[] Technical Note If each of the equations in the model had contained many regressors, the heckman command could become quite long. An alternate way of specifying our wage model would make use of Stata's global macros. The following lines are an equivalent way of estimating our model. global
wageeq
global
seleq
. heckman
"wage "married
$wageeq,
educ
age"
children
edue
age"
select($seleq)
[]
o Technical Note The reported model X _ test is a Wald test of all coefficients in the regression model (except the constant) being 0. heckman is an estimation command, so you can use test, testnl, or lrtest to perform tests against whatever nested alternate model you choose; see [R] test, [R] testnl, and [R] lrtest. The estimation of P and cr in the form atanh p and In cr extends the range of these parameters to infinity in both directions, thus avoiding boundary problems during the maximization. Tests of p must be made in the transformed units. However, since atanh(0) - 0, the reported test for atanh p = 0 is equivalent to the test for ,o = O. The likelihood-ratio test reported at the bottom of the output is an equivalent test for p = 0 and is computationally the comparison of the joint likelihood of an independent probit model for the selection equation and a regression model on the observed wage data against the heckman model likelihood. The z -- 8.619 and X _ of 61.20, both significantly different from zero. clearly justify the Heckman selection equation with this data. []
Example heckman supports the HuberAVhite/sandwich estimator of variance under the robust and cluster() options, or when the pweights are used for population weighted data: see IU] 23.11 Obtaining robust variance estimates. We can obtain robust standard errors for our wage model by specifying clustering on county of residence (the county variable).
h$ckman-- Heckmanselection model
23
i
• heckman
wage
educ
Iteration
O:
log likelihood
= -5178.7009
Iteration
I:
log likelihood
= -5178.3049
Iteration
2:
log likelihood
= -5178,3045
Heckman
selection
(regression
select(married
children
model
model
Log likelihood
age,
with
educ age)
Number sample
selection)
= -5178.304
Coef.
cluster(county)
of obs
=
2000
Censored obs Uncensored obs
= =
657 1343
Wald
=
272.17
=
0.0000
chi2(1)
Prob > chi2 (standard
errorsiladjusted :!
Robust
i!
Std,
Err.
for clustering
P>lzl
[957 Conf.
on county)
Interval]
! wage education age _cons
.9899537
.0600061
16.$0
0.000
.8723438
1.107564
.2131294 .4857752
.020995 1.302103
I0,i5 0._7
0.000 0.709
.17198 -2.066299
.2542789 3.03785
.4451721 .4387068
.0731472 .0312386
6.09 14.04
0.000 0.000
.3018062 .3774802
.5885379 .4999333
5._6
0.000
.0341645
.0772991
0.000 0.000
.0285954 -2,717059
.0444242 -2.264972
0.000
.5991596
1.149258
0.000
1.741902
1.843216
select married children education age _cons
.0110039 •004038 .1153305
9.$4 -21._0
.8742086
.1403337
6._3
1.792559
.0258458
69._6
rho
•7035061
,0708796
.5364513
.817508
sigma lambda
6.004797 4.224412
.155199 .5186709
5.708189 3.207835
6.316818 5.240988
Prob
= 0.0000
/athrho /Insigma
Wald
.0557318 •0365098 -2.491015
test
of indep,
eqns.
(rho = 0): chi2(l!
=
38.81
> chi2
The robust standard errors tend to be a bit larger, bit we do not notice any systematic differences. This is not surprising since the data were not constructed to have any county-specific correlations or other characteristics that would deviate from the assumptions of the Heckman model. q
Example The default statistic produced by predict after tieckman is the expected value of the dependenl variable from the underlying distribution of the regression model. In our wage model, this is the expected wage rate among all women, regardless of whether they were observed to participate in the labor force. • predict heckwage (option xb assumed;
fitted
values) }
It is instructive to compare these predicted wage v_lues from the Heckman model with an ordinary regression model--a model without the selection adiustment:
;_r;
24
heckman-Heckmanselectionmodel wage educ age
. regress
]
Source
13524.0337
wage
age _cons education
(option
MS
2
39830.8609
Total
• predict
df
i
Model Residual
)
SS
1340
53354,8948
1342
I
Coef.
Std.
I
.1465739 6.084875 .8965829
Number of obs F( 2, 1340)
= =
1343 227.49
6762.01687
Prob
=
0.0000
29.7245231
R-squared
=
0.2535
39.7577456
Adj R-squared Root MSE
= =
0.2524 5.452
Err.
t
.0187135 .8896182 .0498061
7.83 6.84 18.00
> F
P> It J
[95Y, Conf.
0.000 O. 000 0.000
.109863 4.339679 .7988765
Interval]
.1832848 7. 830071 .9942893
re.age xb assumed;
. summarize
fitted
heckwage
values)
re.age
'
Variable
0bs
Mean
Std.
Dev.
Min
Max
(
heckwage
2000
21. 15532
3. 83965
14.6479
32. 85949
regwage
2000
23. 12291
3. 241911
17. 98218
32. 66439
Since this dataset was concocted, we know the true coefficients of the wage regression equation to be 1, 0.2, and 1, respectively. We can compute the true mean wage for our sample. • gen
truewage
• sum
truewage
= i +
,2*age
+ l*educ
Variable
I
0bs
Mean
truewage
I
2000
21. 3256
Std.
Dev.
3.797904
Min
Max
15
32.8
Whereas the mean of the predictions from heckmanis within 18 cents of the true mean wage, ordinary regression yields predictions that are on average about $1.80 per hour too high due to the selection effect. The regression predictions also show somewhat less variation than the true wages. The coefficients from heckman are so close to the true values that they are not worth testing. Conversely, the regression equation is significantly off, but seems to give the right sense. Would we be led far astray if we relied on the OLS coefficients? The effect of age is off by over 5 cents per year of age and the coefficient on education level is off by about 10%. We can test the OLS coefficient on education level against the true value using test. • test (I)
educ
= 1
education F(
= 1.0
1, 1340) = Prob > F =
4.31 0.0380
Not only is the OL$ coefficient on education substantially lower than the true parameter, the difference from the true parameter is statistically significant beyond the 5% level. We can perform a similar test for the OLS age coefficient: • test (1)
age
=
.2
age
=
.2
F(
1, 1340) = Prob > F =
8.15 0.0044
We find even stronger evidence that the OLS regression results are biased away from the true parameters. <1 !
!
tw,_..
heckman-- Heckmanselection model
25
> Example Several other interesting aspects of the Heckmah model can be explored with predict;. Continuing with our wage model, the expected wages for wOmen conditional on participating in the labor force can be obtained with the ycond option. Let's gdt these predictions and compare them with actual wages for women participating in the labor forcel • predict
hcndwage,
• stmm_lize
wage
ycond
hcndwage
Variable wage hcndwage
if wage
-=
Obs
Mean
Std ! Dev.
1343
23.69217
6.3_5374,
1343
23.68239
3.355087 i
Min
Max
5.88497
45.80979
16. 18337
33.7567
We see that the average predictions from beckman are very close to the observed levels but do not have exactly the same mean. These conditional w'age predictions are available for all observations in the dataset, but can only be directly compared with observed wages where individuals are participating in the labor force. What if we were interested in making predictions about mean wages for all women? In this case, the expected wage is 0 for those who are not exp_ted to participate in the labor force, with expected participation determined by the selection equation.: These values can be obtained with the yexpected option of predict. For comparison, a variable can be generated where the wage is set to 0 for nonparticipants. . predict
hexpwage,
yexpected
• gen wageO=
wage
(657 missing
values
generated)
. replace
wageO=
0 if wage
(657 real
changes
made)
• summarize Variable hexpwage wageO
hexpwage
== .
wageO
fibs
Mean
Stdi Dev.
2000
15. 92511
5.949336
2000
15,90929
12. _7081
Min 2.492469
Max 32.45858
0
45.80979
i
i
Again, we note that the predictions from heckman are very' close to the observed mean hourly wage rate for all women. Why aren't the predictions uging ycond and yexpected exactly equal to their observed sample equivalents? For the Heckman _odel, unlike linear regression, the sample moments implied by the optimal solution to the model likelihood do not require that these predictions exactly match observed data. Properly accounting for thh additional variation from the selection equation ,-quires that the model use more information thar_just the sample moments of the observed wao_es. q
Example Stata wilt also produce Heckman's (1979) two-step efficient estimator of the model with the twostep option. Maximum likelihood estinaation of the parameters can be time-consuming with large datasets and the two-step estimates may provide a_ood alternative in such cases. Continuing with the women's wage model, we can obtain the two-step estimates with Heckman's consistent covariance estimates by typing
!
I
',,
26
heckman m Heckman selection model
• heckman wage educ age, select(married children ednc age) twostep Heckman selection model -- two-step estimates (regression model with sample selection)
Coef, wage education age _cons
Std, Err.
z
Number of obs Censored obs Uncensored obs
= = =
2000 657 1343
Wald chi2(4) Prob > chi2
= =
551.37 0.0000
P>Izl
[95_ Conf. Interval]
.9825259 .2118695 .7340391
.0538821 .0220511 1.248331
18.23 9.61 0.59
0.000 0.000 0.557
.8789189 .1686502 -1.712645
1.088133 .2550888 3.180723
.4308575 .4473249 .0583645 .0347211 -2.467365
.074208 .0287417 .0109742 .0042293 .1925635
5.81 15.56 5.32 8.21 -12.81
0.000 0.000 0.000 0.000 0.000
.2854125 .3909922 .0368555 .0264318 -2.844782
.5763025 .5036576 .0798735 .0430105 -2.089948
select married children education age _cons mills lambda
4.001615
rho sigma lambda
0.67284 5.9473529 4.0016155
.6065388
6.60
0.000
2.812821
5.19041
.6065388
q
t] Technical Note The Heckman selection mode] depends strongly on the model being correct; much more so than ordinary regression. Running a separate probit or ]ogit for sample inclusion followed by a regression, referred to in the literature as the two-part model (Manning, Duan, and Rogers 1987) not to be confused with Heckman's two-step procedure--is an especially attractive alternative if the regression part of the model arose because of taking a logarithm of zero values. When the goal is to analyze an underlying regression model or predict the value of the dependent variable that would be observed in the absence of selection, however, the Heckman model is more appropriate. When the goal is to predict an actual response, the two-part model is usually the better choice. The Heckman selection model can be unstable when the model is not properly specified, or if a specific dataset simply does not support the model's assumptions. For example, let's examine the solution to another simulated problem.
(Continued
on next page)
heckman-- _man
• heckman
yt xl x2 x3,
Iteration Iteration Iteration Iteration Iteration
O: i: 2: 3: 4:
log log log log log
selec¢(zl
likelihood likelihood likelihood likelihood likelihood
selection model
27
z2) = = = = =
-t11.94996 -110.82258 -II0.17707 -107.70663 (not concave) -107.07729 (not concave)
(outputo_.ed ) Iteration 31: Iteration 32:
log likelihood = -104.08268 log likelihood = -104.08267 (backed up)
Heckman selection model
Number of obs
=
150
(regression model with sample selection)
Censored obs Uncensored obs
= =
87 63
Wald chi2(3)
=
8.64e+07
Prob
=
0.0000
Log likelihood
= -104.0827 Coef.
Std. Err.
z
> chi2
P>Izl
[957,Conf, Interval]
yt xl x2
.8974192 -2,525302
.0006338 1415._ .0003934 -6418.57
O.000 O.000
.896177 -2.526074
.8986615 -2.524531 2. 856651
x3 _cons
2.855786 .6971255
.0004417 6465.84 .0851518 8.I_
O.000 O.000
2.85492 ,5302311
zl
-.6830377
.0824049
-8.29
O.000
-.8445484
-.521527
z2 _cons
1.004249 -.361413
.1211501 .1165081
8. _ -3,ID
O. 000 O.002
.7667993 -.589"/647
1. 241699 -. 1330613
/athrho /insigma
15.12596 -.5402571
151.3627 ,1206355
0.10 -4._
0.920 O.000
-281.5395 -.7766984
311.7914 -.3038158
.8640198
select
i
rho sigma lambda
1 .5825984 .5825984
4.40e-
LR test ol indep, eqns. (rho = 0):
I
I !
11
-1
.0702821 .0702821
.459922 .4448481 chi2(i) =
25.67
1 .7379968 .7203488
Prob > chi2 = 0,0000
the form of the likelihood for the Heckman selectioh model, this implies a division by zero and it is surprising that the model solution turns out as will as it does. Reparameterizing p has allowed The model has converged to a value of p that is 1.0--within machine rounding tolerances. Given the estimation to converge, but we clearly have problems with the estimates. Moreover, if this had occurred in a large dataset, waiting over 32 iteration_ for convergence might take considerable time. This dataset was not intentionally developed to cause problems. It is actually generated by a "Heckman process" and when generated starting fromidifferent random values can be easily estimated. The luck of the draw in this case merely led to daia that, despite its source, did not support the assumptions of the Heckman model. The two-step model is generally more stable in chses where the data are problematic. It is even tolerant of estimates of p less than -1 and greater !than t. For these reasons, the two-step model may be preferred when exploring a large dataset. Still, if the maximum likelihood estimates cannot converge, or converge to a value of p that is at the bouhdary of acceptable values, there is scant support for estimating a Heckman selection model on the d_ta. Heckman (1979) discusses the implications of p being exactly t or 0, together with the implica!ions of other possible covariance relationships among the model's determinants.
_
l_,T
28
heckman --
Saved Results heckman
saves
Heckman selection model
in e():
Scalars e (N)
number of observations
e (re)
return code
e (k)
number of variables
e (chi2)
x2
e(k_eq)
number of equations
e(chi2_c)
X2 for comparison test
e(k_dv) e (N_eens)
number of dependent variables number of censored observations
e(p_c) e (p)
p-value for comparison test significance of comparison test
e (dr_m)
model degrees of freedom
e (rho)
p
e(11)
log likelihood
e(£c)
number of iterations
e(ll_O) e(N_clust)
log likelihood, constant-only model number of clusters
e(rank) e(rankO)
rank of e(V) rank of e(V) for constant-only
e(lambda)
A
model
e(selambda) standard errorof A
e(sigma)
sigma
typeof optimization
Macros e(cmd)
heckman
e(opt)
e(depv_')
name(s)of dependent variable(s)
e(chi2type) Wald or Lit; typeof modcl x2 test
e(title) e(title2)
title in estimation output secondary title in estimation output
e(chi2_ct)
Wald or LR; type of model )c2 test corresponding to e(chi2_c)
e(utype)
weight type
e(offset#)
offset for equation #
e (wexp) e(clustvar)
weight expression name of cluster variable
e (mills)
variable containing nonselection hazard (inverse of Mills')
e (method)
requested estimation method
e(predict)
program used to implement predict
e(vcetype) e (user)
covanance estimation method name of tikelihood-evaluator program
e(cnslist)
constraint numbers
e(b)
coefficient vector
e(V)
variance-covariance
e(ilog)
iteration log (up to 20 iterations)
Matrices matrix of
the estimators
Functions e(sample)
marks estimation sample
Methods and Formulas heckma_n is implemented 446-450)
provide
as an ado-file.
an introduction
Greene
Regression estimates using the nonselection maximum likelihood estimation. The regression
equation
(2000,
928-933)
to the Heckmm-a selection hazard
(Heckman
is
yj = xjO + ulj The selection
equation
is zj'7 + u2j
> 0
where
ul _ N(O, a) u2 _ N(0,
1)
1
I_'i''1 :-
cor_(_l,u_)= p
or Johnston
and DiNardo
(1997,
model. t979')
provide
starting
values
for
|
!
--
i necKman- Heclcmanselection model
2g
The log likelihood for observation j is
observed lj =
V/1-- "_
-_
a
/
-- Wj
ln(
wjln @(-zjT)
yj
yj not observed i
where _0
is the standard cumulative normal and wj is an optional weight for observation j.
In the maximum likelihood estimation, o-and p are not directly estimated. Directly estimated are In a and atanh p:
(
l+p
_
i i i
atanh p = _ ln\ _] The standard error of ,_ = #a is approximated through the propagation of error (delta) method: that is, Var(A) _ D Var{(atanh
p lncr)} D'
where D is the Jacobian of )_ with respect to at_h p and In a. The two-step estimates are computed using H_ckman's (1979) procedure. Probit estimates of the selection equation Pr(yj
observed I zj)-
_(zj")')
are obtained. From these estimates the nonselection hazard, what Heckman (t979) referred to as the inverse of the Mills' ratio, m j, for each observa¢ion 3 is computed as
¢(zjS) mj where ¢ is the normal density. We also define
Following Heckman, the two-step parameter estimates of /3 are obtained by augmenting the regression equation with the nonselection hazard m. Thus, the regressors become [X m] and we obtain the additional parameter estimate/3,a on the variable containing the nonselection hazard. A consistent estimate of the regression disturbance variance is obtained using the residuals from the augmented regression and the parameter estimate on the nonselecfion hazard. e'e +/3_ _--]j=l N N
5j
The two-step estimate of p is then _ = /3r,L c3 Heckman derived consistent estimates of the coefficient covariance matrix based on the augmented regression.
]
.......
--.o_..
.,vv,-..,t,ua|
.O_I¢_I_.I.I1JII
IIIUUI_I
Let W = [X m] and D be a square diagonal matrix of rank N with (1 _ P^2o_ j) on the diagonal elements.
Vtwostep - 2(W'W)-I(W'DW + Q)(W'W)-1 where q = _2(W'DW)Vp(W'DW) and Vp is the variance-covariance
estimate from the probit estimation of the selection equation.
References Greene. W. H. 2000. Econometric Analysis. 4th ed. Upper Saddle River. NJ: Prentice-Hall. Gronau. R. 1974. Wage comparisons: A selectivity bias. Joumat of Political Economy 82: 1119-1155. Heckman, J. 1976. The common structure of statistical models of truncation, sample selection, and limited dependent variables and a simple estimator for such models. The Annals of Economic and Social Measurement 5: 475--492. 1979. Sample selection bias as a specificationerror. Econometrica47: 153-16t. Johnston. J, and J. DiNardo. 1997. EconometricMethods. 4th ed. New York: McGraw-Hill. Lewis, H. 1974. Comments on selectivity biases in wage comparisons. Journal of Political Economy 82: 1119-1155. Manning, W. G., N. Duan. and W. H. Rogers. 1987, Monte Carlo evidence on the choice between sample selection and two-part models. Journal of Econometrics 35: 59-82.
Also See Complementary:
[R] adjust, [R] constraint, JR] lincom, [R] lrtest, [R] mfx, [R] predict, [R] test, [R] testnl, JR] vce, [R] xi
Related:
[R] heckprob,
Background:
[U] [u] [U] [U]
[R] regress,
[R] tobit, [R] treatreg
16.5 Accessing coefficients and standard errors. 23 Estimation and post-estimation commands, 23.11 Obtaining robust variance estimates. 23.12 Obtaining scores
---Title heckprob -- Maximumd_kelihood probit estimation with selection
Syntax heckprob
dewar
[vartist] [,,eight]
[if
exp] [in
range],
select( [ depva,'s = ] varlists [, ,gffse_(varname) } [ robust
] )
cl__uster (vamame)
constraints by .,.
noconstant
(numlist)
s qcore(newv_rlist) first noconstant i noskip level(#) _ffset (varname) maximize_options
: may be used with heckprob; and i_eights
see [R] by.
_eights,
f_eights,
are allowed; see [U] 1_1.6
weight.
heckprob
shares the featuresof all estimationcommands;see [U] 23 Estimationand post-estimationcommands.
Syntaxforpredict predict
[type] newvarname
[if exp] [in range]
[, statistic nooffset
]
where statistic is /
pmargin
q'(xjb),
success probability (th_ default)
pll
_2(xjb,
_/.probit = 1, yj _ select zjg, p), predicted prolJability P'tyj
plO
_52(xjb,-z/g,-p),
predicted ,robability Pr(_/pr°bit = 1,_/;elect = O)
pO1
_52(-x3b,
predicted _robability P_yj
pO0
_2(-xjb,--zjg,
psel pcond
_(zjg), selection probability _52(xjb, zig, p)/_5(zjg), prob_ility
xb stdp
xyb, fitted values standard error of fitted values
xbsel
linear prediction for selection equation
stdpsel
standard error of the linear prediction for selection equation
zjg,-p),
_/_ probit
p),
predicted )robability
Pr(y
= l)
= O, yj
_ select
p.r°bit = O, y;elect
= 1) = O)
of success conditional on selection
q)() is the standard normal distribution function and q52() is the bivariate normal distribution function. These statistics are available both in and out of sample; type predict the estimation
...
if
e(sample)
...
sample.
Description heckprob
estimates maximum-likelihood probit models with sample selection. 31
if wanted only for
Options select(...) specifiesthe va_ables and optionsfor the selectionequation. It is an integral part of specifying a selection model and is not optional. robust specifies that the Huber/White/sandwich estimator of the variance is to be used in place of the conventional MLE variance estimator, robust combined with cluster() further allows observations which are not independent within cluster (although they must be independent between clusters). If you specify pweights, robustis implied; see [U] 23.11 Obtaining robust variance estimates. clust;er (varname) specifies that the observations are independent across groups (clusters) but are not necessarily independent within groups, varname specifies to which group each observation belongs, cluster() affects the estimated standard errors and variance-covariance matrix of the estimators (VCE),but not the estimated coefficients, cluster() can be used with pweights to produce estimates for unstratified cluster-sampled data. cluster() cluster()
implies robust; by itself.
that is, specifying robust
cluster()
is equivalent to typing
score(newvarlist) creates a new variable, or a set of new variables, containing the contributions to the scores for each equation and the ancillary parameter in the model. The first new variable specified will contain ul_ -- OtnLj/O(xj/3) for each observation j in the sample, where lnLj is the 3th observation's contribution to the log likelihood. The second new variable: u2j = OlnLj/O(zj_') The third: u3j = OlnLj / O(atanh p) If only one variable is specified, only the first score is computed; if two variables are specified, only the first two scores are computed; and so on. The jth observation's contribution to the score vector is { OtnLj/Ol30lnLj/O("/)
OlnLj/O(atanhp)}
= (UljXj
u2jzj
u3j)
The score vector can be obtained by summing over j; see [U] 23.12 Obtaining scores. first specifies that the first-step probit estimates of the selection equation be displayed prior to estimation. noconstant omits the constant term from the equation. This option may be specified on the regression equation, the selection equation, or both. constraints (numlist) specifies by number the linear constraints to be applied during estimation. The default is to perform unconstrained estimation. Constraints are specified using the constraint command; see [R] constraint. See [R] reg3 for the use of constraints in multiple-equation contexts. noskS.p specifies that a full maximum likelihood model with only a constant for the regression equation be estimated. This model is not displayed but is used as the base model to compute a likelihood-ratio test for the model test statistic displayed in the estimation header. By default, the overall model test statistic is an asymptotically equivalent Wald test of all the parameters in the regression equation being zero (except the constant). For many models, this option can substantially increase estimation time. level (#) specifies the confidence level, in percent, for confidence intervals The default is level (95) or as set by set level; see [U] 23.5 Specifying the width of confidence intervals. ogfset (w_rname) is a rarely used option that specifies a variable to be added directly to Xb. This option may be specified on the regression equation, the selection equation, or both.
hect(prob--:Maximum-likelih_od_pmbit estimationwith selection
33
_,
mca:imize_.options control the maximization process; see [R] maximize. With the possible exception of iterate (0) and trace, you should never ha_e to specify them.
Optionsfor predict pma.rgin, the default, calculates the univariate (ma@nal) predicted probability of success _e, probit 1). ptty j _. calculates the bivariate predicted probability P_yj
probit
=
__ probit plO calculates the bivariate predicted probability P,_y)
--
pll
i
1
- 1, _yjselect
_}_,probit
p01 calculates the bivariate predicted probability
P_yj
._. probit pO0 calculates the bivariate predicted probability P_(yj
psel
_ select , yj
=
l).
___ 0).
o select
= O, yj
-----1).
. select = O, yj
:
0).
calculates the univariate (marginal) predicted probability of selection Pr(y_ elect = l).
pcond calculates the conditional (on selection) predicted probability of success P_tYj-'" probit = l, yj-select = t)/Pr(y_ elect = 1). xb calculates the probit linear prediction xjb. stdp calculates the standard error of the prediction, it can be thought of as the standard error of the predicted expected value or mean for the obsel_'_tion's covariate patfern. This is also referred to as the standard error of the fitted value. xbsel
calculates the linear prediction for the selectibn equation.
stdpsel
calculates the standard error of the linear irediction for the selection equation.
nooffset is relevant only if you specified offset (varname) for heckprob. It modifies the calculations made by predict so that they ignore th_ offset variable; the linear prediction is treated as xj b rather than xj b + offsetj.
Remarks The probit model with sample selection (Van d_ Yen and Van Pragg 1981) assumes that there exists an underlying relationship y_ = xjj3 + ulj
latent
equation
probit
equation
such that we observe only the binary outcome , probit
yj
= (y_ > O)
The dependent variable, however, is not always observed. Rather, the dependent variable for observation j is observed if ffselect j
I =-
(Zjy-4:-U2j
>
0)
selection
where ul _ 2?{0, 1)
Nio,1) corr(ul,
=p
equation
i
F
F
o,_
necKproo
m MaxlmumqlKellhood
probit estimation
with
selection
When p _ 0, standard probit techniques applied to the first equation yield biased results, heckprob provides consistent, asymptotically efficient estimates for _t the parameters in such models.
> Example We use the data from Pindyck and Rubinfeld (1998). In this dataset, the variables are whether children a_end private school (private), number of years the family has been at the present residence (years), log of property tax (logptax), log of income (loginc), and whether one voted for an increase in prope_y taxes (vote). In this example, we alter the meaning of the data. Here we assume that we observe whether children attend private school only if the family votes for increasing the property taxes. This is not true in the dataset and we make this untrue assumption only to illustrate the use of this command. We observe whether children attend private school only if the head of household voted for an increase in property taxes. We assume that the vote is affected by the number of years in residence, the current prope_y taxes p_d. and the household income. We wish to model whether children are sent to private school based on the number of years spent in the current residence and the cu_ent prope_y taxes paid. . heckprob Fitting
private
probit
years
logptax,
sel(vote=years
Iteration
O:
log
likelihood
Iteration
I:
log
likelihood
= -18.407192
Iteration
2:
log
likelihood
= -16.1412S4
Iteration
3:
log
likelihood
= -15.953354
Iteration
4:
log
likelihood
= -15.887269
Iteration
5:
log
likelihood
= -15.883886
Iteration
6:
log
likelihood
= -15.883655
Fitting
selection
model:
O:
log likelihood
= -63.036914
Iteration
I:
log likelihood
= -58.581911
Iteration
2:
log
likelihood
= -58.497419
Iteration
3:
log
likelihood
= -58.497288
log
likelihood
= -74.380943
Comparlson: starting
values:
Iteration
O:
log
likelihood
Iteration
I:
log
likelihood
= -17.920826
Iteration
2:
log
likelihood
= -18.375362
= -40.895884
Iteration
3:
log likelihood
= -16.067451
Iteration
4:
log
likelihood
=
Iteration
5:
log
likelihood
= -15.760354
Iteration
6:
log
likelihood
= -15.753805
Iteration
7:
log
likelihood
= -15.753785
full
model
Iteration
0:
log
likelihood
= -75.010619
Iteration
I:
log
likelihood
= -74.287753
Fitting
-15.84787
Iteration
2:
log
likelihood
= -74.250148
Iteration
3:
log
likelihood
= -74.245088
Iteration
4:
log
likelihood
= -74.244973
Iteration
5:
log
likelihood
= -74.244973
Probit
Log
model
likelihood
logptax)
= -IZ.122381
Iteration
Fitting
loginc
model:
with
sample
= -74.24497
selection
(not
concave)
Number
=
95
Censored obs Uncensored obs
of obs
= =
36 59
Wald
chii(2)
=
Prob
> chi2
=
1.04 0.5935
heckprob- Maximum-likelihoodprobitestimationwith selection
Coef.
Std.
Err.
P> lzl
!z
[95_.
Conf.
35
Interval]
)
private years logptax _cons
-. 1142597
.1461711
-0 i78
0.434
-.4007498
.1722304
.3516095 -2,780664
1.01648 6.905814
0 i 35 -0_40
O. 729 0.687
-1.640655 -16.31581
2.343874 10.75448
-,0167511
.0147735
-li13
0.257
-.0457067
vote years
.0122045
loginc
.9923025
.4430003
2 i24
O.025
,1240378
I.860567
logptax _cons
-1. 278783 -.5458214
.5717544 4.070415
-2 _24 -0.13
O.025 O.893
-2.399401 -8,523689
-. 1581649 7.432046
/athrho
-.8663147
1.449995
-0.60
O.550
-3. 708253
1.975623
-. 6994969
.7405184
-. 9987982
.9622642
rho LR test
of indep,
eqns.
(rho = 0):
chi2(_)
=
0.27
Prob
> chi2
= 0.6020
J
The output shows several iteration logs. The first iteration log corresponds to running the probit model for those observations in the sample where we hav< observed the outcome. The second iteration log corresponds to running the selection probit model, _hich models whether we observe our outcome of interest. If p = 0, then the sum of the log likelihoods from these two models will equal the log likelihood of the probit model with sample selectioh; this sum is printed in the iteration log as the comparison log likelihood. The third iteration log shows starting values 'for the iterations. The finn iteration log is for estimating the full probit model with sample selection. A likelihoodratio test of the log likelihood for this model and the comparison log likelihood is presented at the end of the output. If we had specified the robust option, then this test would be presented as a Wald test instead of a likelihood-ratio test.
q
Example In the previous example, we could obtain robust standard errors by specifying the robust We also eIiminate the iteration logs using the nolog option. • heckprob Probit
private
model
Log likelihood
with
years
lo_ptax,
sample
selection
eel(vote=years
= -74.24497
loginc
lo_q_tax) nolog
robust
Number of obs Censored obs
= =
95 36
Uncensored
=
59
obs
Wald
chi2(2)
=
Prob
> chi2
=
2,55 0.2798
Kobust Coef.
Std. Err.
2
P>Iz[
[95_ Conf.
Interval]
i
private years
-.1142597
.1113949
i -1.03
0.305
-.3325896
•1040702
logptax _cons
.3516095 -2.780664
.735811 4.786602
0.48 -0.58
0.633 0.561
-I.090553 -12.16223
1.793773 6.600903
)
Vote
! i
years
-.0167511
.0173344
-0.97
0.334
-.0507258
.0172237
loginc
.9923025
.4228035
2.35
0.019
.1636229
1.820982
lo_ptax _cons
-1.2T8783 .5458214
.5095157 4.543884
-2._1 -0._2
0.012 0.904
-_.277416 -9.45167
-.280t505 8.360027
option.
!L
t_ir_
_
,=©_Rp, uu --
/athrho rho
maA..um-.Ke.nooa
-.8663147
1.630569
-. 6994969
.8327381
prODlI estimation
-0,53
Wald test of indep, eqns, (rho = 0): chi2(1) =
with
O.595
0.28
selection
-4,062171
2.329541
-, 9994077
.9812276
Prob > chi2 = 0.5952
Regardless of whether we specify the robustoption, it is clear that the outcome is not significantly different from the outcome obtained by estimating the probit and selection models separately. This is not surprising since the selection mechanism estimated was invented for the example rather than born from any economic theory.
> Example It is instructive to compare the marginal predicted probabilities with the predicted probabilities we would obtain ignoring the selection mechanism. To compare the two approaches, we will synthesize data so that we know the "true" predicted probabilities. First, we need to generate correlated error terms, which we can do using a standard Cholesky decomposition approach. For oar example, we will clear any data from memory and then generate errors that have correlation of .5 using the following commands. Note that we set the seed so that interested readers might type in these same commands and obtain the same results. clear set
seed
set
obs 5000
gen ci
12309
= invnorm(uniform())
gen c2 = invnorm(uniform()) matrix P = (1,.5\.5,1) matrix A = cholesky(P) local facl = A[2,1] local fac2 = A[2,2] gen ul = cl gen u2 = "facl"*cl + "fac2"*c2
We can check that the errors have the correct correlation using the corr command. We will also normalize the errors such that they have a standard deviation of one so that we can generate a bivariate probit model with known coefficients. We do that with the following commands. summarize ul replace ul = ul/sqrt(r(Var)) summarize u2 replace u2 = u2/sqrt(r(Var)) drop cl c2 gen xl = u_uifomn{)-.5 gen x2 = uniform()+i/3 gen yls = 0.5 gen
y2s
gen yl gen
+ 4.x1
= 3 - 3.x2 = (yls>0)
y2 = (y2s>0)
+ ul + .5*xl
+ u2
heckpmb -- Maximum-likelihood, , probit estimationwith selection
37
We have now created two dependent variables yl aM y2 that are defined by our specified coefficients. We also included error terms for each equation and ]he error terms are correlated. We run heckprob to verify that the data have been correctly _oenerate(]according to the model Yl -- .5 -}-4Xl _ ul Y2 = 3 + .5xl _ 3x2 + u2 where we assume that Yl is observed only if Y2 = J. • heckprob yl xl, sel(y2 = xl x2) nolog Probit model with sample selection
Log likelihood = -3600.854 Coef.
Std. Err.
Number of obs Censored obs Uncemsored obs
= = =
5000 1790 3210
Wald chi2(1) Prob > chi2
= =
941.68 0.0000
P>Iz]
[95_ Conf. Interval]
xl _cons
3.985923 .4852946
.1298904 .0464037
30._9 i0•_6
0.000 0.000
3.73i342 •3943449
4.240503 .5762442
xl x2 _cons
.5998148 -3.004937 3.0_1587
.0716655 .0829469 .0782817
8.37 -36•23 38.47 i
0.000 0.000 0.000
.4593531 -3.1/6751 2.858157
.7402765 -2.842364 3.165016
0.000
.4053964
.7427295
.3845569
.6307914
y2
/athrho rho
,574063
.0860559
,5183369
.062935
LR test of indep, eqns. (rho = 0):
6._7
chi2(I) =
46.58
Prob > chi2 = 0.0000
Now that we have verified that we generated data according to a known model, we can obtain and then compare predicted probabilities from the pi'obit model with sample selection and a (usual) probit model. predict pmarg (option pmargin assumed; Pr(yl=l)) probit yl xl if y2==l
(outputomitted) predict phat (option p assumed;
Pr(yl))
Using the (marginal) predicted probabilities from the probit model with sample selection (pmarg) and the predicted probabilities from the (usual) prob!t model (phat), we can also generate the "true" predicted probabilities from the synthesized yls variOble and then compare the predicted probabilities: • gen ptrue
= norm(yls)
• summarize pmarg ptrue phat Variable Obs
Mean
Std. Dev. i
Min
Max
.0658356 1.02e-06
.9933942 1
pmarg ptrue
5000 5000
.619436 .6090161
.32094_4 .34990_5
phat
5000
.6723897
.30632_)7
.096498
.9971064
i
_
38
heckprob m Maximum-likelihood
Here
we see that ignoring
the selection
probit estimation with selection
mechanism
(comparing
the phat
variable
with the true
ptrue variable) results in predicted probabilities that are much higher than the true values. Looking at the marginal predicted probabilities from the model with sample selection, however, results in more accurate
predictions.
<1
Saved Results in e():
heckprob saves Scalars e (N)
number of observations
e (re)
return code
e (k)
number of variables
e (chi2)
x_
e(k_eq)
number of equations
e (ch±2_c)
X_ for comparison test
e (k_dv) e (N_cens)
number of dependent variables number of censored observations
e (p_c) e (p)
p-value for comparison test significance of comparison test
e (df_.m) e(ll)
model degrees of freedom log likelihood
e (rho) e(ic)
p number of iterations
e(ll_O)
log likelihood, constant-only model
e (rank)
rank of
e(ll_c)
log likelihood, comparison model number of clusters
e(rank0)
rank of e(V) for constant-only model
e (cmd)
heckprob
e (opt)
e(depvar)
name(s) of dependent variable(s)
e(chi2type)
type of optimization Wald or LR; type of model x 2 test
e(title)
title in estimation output weight type
e(chi2_ct)
type of comparison X_ test
e (wtype)
e (offset)
offset
e(uexp) e(clustvar)
weight expression name of cluster variable
e(predict) e(cnslist)
program used to implement predict constraint numbers
e(vcetype)
covariance estimation method
e (user)
name of likelihood-evaluator program e (V)
variance-covariance
e (N_clust)
e (V)
Macros
Matrices e (b)
coefficient vector
e(ilog)
iteration log (up to 20 iterations)
matrix
of the estimators
Functions e(sample)
marks estimation sample
Methods and Formulas heckprob is implementedas an ado-file. Vande Venand VanPragg (1981)provide an introduction and an explanation The
probit
of this model.
equation
is
vj = (x9 + ulj > 0) The selection equation is zj'T + u2i
> 0
where
ul _ N(0, 1) uz _ N(O. 1)
corr(ul,u2)- p
heckprob-- Maximum-!ikelih0odprobit estimationwith selection
39
The log likelihood is
IES
_ti=0
+
{1- (z,-y+ offsetT)}
where S is the set of observations for which 9i is observed, (1)20 is the cumulative bivariate normal distribution function (with mean [0 0 ]'), _0 is the standard cumulative normal, and wi is an optional weight for observation i. In the maximum likelihood estimation, p is not directly estimated. Directly estimated is atanh p: i
7
From the form of the likelihood, it is clear that if p = 0, then the log likelihood for the probit model with sample selection is equal to the sum of the probit model for the outcome 9 and the selection model. A likelihood-ratio test may therefore be performed by comparing the likelihood of the full model with the sum of the log likelihoods fo_ the probit and selection models,
References Greene,W. H. 2000. EconometricAnalysis.4th ed. Upper Sa_le River, NJ: Prentice-Hall. Beckman.J. t979. Sampleselectionbias as a specificationerror. Economettica47: 153-161. Pindyek.R. and D. Rubinfeld.1998. EconometricModelsand EconomicForecasts.4th ed. New York:McGraw-Hill. Vande Ven,W. R M. M. and B. M. S. VanPragg. 1981.The demandfor deductiblesin private health insurance:A probit modelwith sample selection.JournaIof Econometric_17: 229-252.
Also See Complementary:
[R] adjust, [R] constraint, [R] l_com, [R] lrtest, [R] mfx, [R] predict, [R] test, [R] testnl, [R] vce, [R] Xi
Related:
[R] heckman, JR] probit, [R] treatreg
Background:
[u] [U] Iv] [u]
16.5 Accessing coefficients and standard errors, 23 Estimation and post-est_ation commands, 23.H Obtaining robust var] ance estimates, 23.12 Obta_iningscores
vw __'° ti"
Title he
u Obtain on-line help In
I
I
I
Syntax Windows, Macintosh, and Unix(GUI): help [ command or topic name ]
whelp[command or fopicname] Unix(console & GUI):
{help lma.n} [command or topic name ]
Description The help command displays help information on the specified command or topic. If help is not followed by a command or a topic name, a description of how to use the help system is displayed. Stata for Unix users may type help or mmamthey mean the same thing. Stata for Windows, Stata for Macintosh, and Stata for Unix(GUI) users may click on the Help menu. They may also type whelp something to display the help topic for something in Stata's Viewer. whelp typed by itself displays the table of contents for the on-line help.
Remarks See [U] 8 Stata's on-line help and search facilities for a complete description of how to use help. Q Technical Note When you type help something, Stata first looks along the S_ADOpath for something.hip; [U] 20.5 Where does Stata look for ado-files?. If nothing is found, it then looks in state.hip the topic.
Also See Complementary:
[R] search
Related:
[R] net search
Background:
[GSM]4 Help, [GSW]4 Help, [GSU] 4 Help, [U] 8 Stata's on-line help and search facilities
40
see for vl
Title i [ hetprOb - llliMaximum-l_etih°°d r 1 _ heter°skedastic "'i!_pr°bit estimati°n l llllllll I ,I I I
II I
i
i
ntax
eerar het(varlist,
[offset(varname)
c!luster(varname) nolrtest
'd [noconstant
level(#)
asis_robust
score (newvarl [newvar2:]) noskip offset (varname)
constraints
by ... : may be used with hetFob; fweights, iweights,
])
(numlist) nolqg maximize_options ] see [R] by.
and pweights are allowed; see [U] 14il.6 weight.
This command shares the features of all estimation commands;see [U] 23 Estimation and post-estimation commands.
Syntaxforpredict
i
predict[O?e]newvarname[ifexp][in r_ge] [, { p I xb [ sigma } nooffset] These statistics are available both in and out of sample; type predict the estimation sample.
...
if e(samp!e)
...
i
if wanted only for
scription hetprob
estimates a maximum-likelihoodhetero_kedasticprobit model.
See [R] logistic for a list of related estimation commands.
Options het(varlist [, of.fset(varname)]) specifies the independent variables and the offset variable, if there is one, in the variance function, her () is not optional. noconstant
suppresses the constant term (intercept}in the model.
level (#) specifiesthe confidencelevel, in percent, foi confidenceintervals. The default is level (95) or as set by set level; see [U] 23.5 Specifying !he width of confidence intervals. as is forces retention of perfect predictor variablesand their associatedperfectly predictedobservations and may produce instabilities in maximization; see [R] probit. robust specifies that the Huber/White/sandwichestinmtor of variance is to be used in place of the traditional calculation; see [U] 23.11 Obtaining robust varianee estimates, robust combined with cluster () allows observations which are not independent within cluster (although the)' must be independent between clusters). If you specify pweights, robust is implied: see[U] 23.13 Weighted estimation, 41
i
42
hetprob -- Maximum-likelihood heteroskedastic probit estimation
cluster(varname) specifies that the observations are independent across groups (clusters) but not necessarily within groups, varname specifies to which group each observation belongs; e,g., cluster(personid) in data with repeated observations on individuals, cluster() affects the estimated standard errors and variance-covariance matrix of the estimators (VCE), but not the estimated coefficients; see [U] 23.11 Obtaining robust variance estimates, cluster() can be used with pweights to produce estimates for unstratified cluster-sampled data. but see the svyprobit command in [r<] svy estimators for a command designed especially for survey data. cluster() by itself.
implies robust;
specifying
robust
cluster()
is equivalent
to typing cluster()
score (newvarl [newvar2 ] ) creates newvarl containing uj = OlnLj/0(xj b) for each observation j in the sample. The score" vector is _ OlnLj/Ob = _ wjujxj; i.e.. the product of newvar with each covariate summed over observations. The second new variable, newvar2, contains vj = OlnLj/0(zjq,).
See [U] 23.12 Obtaining
scores.
noskip requests fitting of the constant-only model and calculation of the corresponding likelihood-ratio X 2 statistic for testing significance of the full model. By default, a Wald X 2 statistic is computed for testing the significance of the full model. offset (varname) to be 1.
specifies that varname is to be included in the model with coefficient constrained
nolrtest specifies that a Wald test of whether lnsigma2 LR test.
-- 0 should be performed
instead of the
constraints(numlist) specifies by number the linear constraints to be applied during estimation. The default is to perform unconstrained estimation. Constraints are specified using the constraint command; see [R] constraint. See [R] reg3 for the use of constraints in multiple-equation contexts. nolog
suppresses
maximize_options specify them.
the iteration log. control the maximization
process:
see [R] maximize.
You should never have to
Options for predict p, the default, calculates
the probability
of a positive outcome.
xb calculates the linear prediction. sigma
calculates the standard deviation.
noof fset is relevant only if you specified off set (varname) for het prob. It modifies the calculations made by predict so that they ignore the offset variable: the linear prediction is treated as xjb rather than xjb + offsetj.
Remarks hetprob performs maximum likelihood estimation of the heteroskedastic probit model, a generalization of the probit model. Let yj, j = 1,.., N be a binary outcome variable taking on the value 0 (failure) or 1 (success). In the probit model, the probability that yj takes on the value 1 is modeled as a nonlinear function of a linear combination of thc k independent variables xj -- (z 1j, x2_ .... , xkj): Pr(yj
- 1) - _b(xjb)
hetprob- Maximum-likelihoo_heteroskedasticprobitestimation
43
in which _0 is the cumulative distribution function (CDF) of a standard normal random variable. that is, a normally distributed (Gaussian) random varihble with mean 0 and variance 1. The linear combination of the independent variables, xjb, is cdmmonly called the index fimction or index. Heteroskedastic probit generalizes the probit model b_ generalizing _I,0 to a normal CDF with a variance no longer fixed at 1 but allowed to vary as a h_nction of the independent variables, hetprob models the variance as a multiplicative function of _ese m variables zj = (zlj, z2j,..., Zmj), following Harvey (1976): 2 i 2
={exp(zj )} :i I
Thus, the probability of success as a function of all the independent variables is Pr(yj=l
=@
xj
expzj-y Z
From this expression it is clear that, unlike the index Xjb, no constant term can be present in zj7 ]f the model is to be identifiable. i Suppose the binary outcomes yj are generated by tltresholding an unobserved random variable t_, which is normally distributed with mean xjb and varihnce 1 such that
YJ=
01
ifw_ 0
;This process gives the probit model: Pr(yj = 1) = Pr(wj
N 0) = _(xjb)
Now suppose that the unobserved wj are heteroskedasiic with variance crj2= {exp(zjb)}
2
Relaxing the homoskedastic assumption of the probit rhodel in this manner yields our muItiplicative heteroskedastic probit model:
Pr(yj
= 1)=
_{xj_/exp(zj'7)}
Z
> Example For this example, we generate simulated data for a simple heteroskedastic probit model and then estimate the coefficients using hetprob: • set obs
obs
was
O,
1000 now
1000
set
seed
1234567
gen
X = l-2*uniform()
gen
xhet
= uniform()
gen
sigma
gen
p = norm((O.3+2*x)/s±gma)
= exp(1.5*xhet)
gen
y = cond(un±form()<=p,l,O)
_,
_t;'
44
hetprob -- Maximum-likelihood heteroskedastic probit estimation
• hetprob Fitting
y x, het(xhet) comparison
model:
Iteration
O:
log
likelihood
= -688.53208
Iteration
1 :
log likelihood
= -592.23614
Iteration
2:
log likelihood
= -591.50687
Iteration
3:
log likelihood
= -591.50674
Fitting
full
model:
Iteration
O:
log
likelihood
= -591.50674
Iteration
I:
log
likelihood
= -572. 12219
Iteration
2:
log
likelihood
=
Iteration
3:
log
likelihood
= -569.48921
Iteration
4:
log
likelihood
= -569.47828
Iteration
5:
log
likelihood
= -569.47827
Heteroskedastic
probit
-570.7742
model
Number of obs Zero outcomes Nonzero
Log
likelihood
= -569.4783
y
Coef.
x
Std.
Err.
z
= =
outcomes
1000 452
=
548
Wald
chi2 (I)
=
78.66
Prob
> chi2
=
0.0000
P>Izl
[95Y, Conf.
Interval]
Y 2. 228031
.2512073
8,87
O. 000
i, 735673
2. 720388
_cons
.2493822
.0862833
2.89
O. 004
.08027
.4184943
xhet
I. 602537
.2640131
6.07
O. 000
I.085081
2. 119993
Prob
= 0.0000
insiEma2
Likelihood
ratio
test
of Insigma2=O:
chi2(1)
=
44.06
Above we created two variables, x and xhet,and then simulated
> chi2
the model
Pr(y=11=F{(80+ ,x)/exp( lxhet)} for/3o = 0.3,/31 = 2, and 71 -- 1.5. According to hetprob's output, all coefficients are significant and, as we would expect, the Wald test of the full model versus the constant-only model, e.g., the index consisting of/3o + filx versus that of just /30, is significant with X2(1) = 79. Likewise, the likelihood-ratio test of heteroskedasticity which tests the full model with heteroskedasticity against the full model without is significant with X2(1) = 44. See [R] maximize for further explanation of the output. Note that for this simple model hetprob took five iterations to converge. As stated elsewhere (Greene 2000, 829), this is a difficult model to fit and it is not uncommon for it to require many iterations or for the optimizer to print out warning and informative messages during the optimization. Slow convergence is especially common for models in which one or more of the independent variables appear in both the index and variance functions.
<1
Q Technical Note Stata interprets a value of 0 as a negative outcome (failure) and missing) as positive outcomes (successes). Thus, if your dependent and 1, 0 is interpreted as failure and 1 as success. If your dependent 1, and 2, 0 is still interpreted as failure, but both 1 and 2 are treated
treats all other values (except variable takes on the values 0 variable takes on the values 0, as successes.
O
.............
S'
RObust standard errors If you [tJ] 23.11 re-estimate robust to
Maximum-likelihooq heteroskedasticprobit estimation specify the hetprob robust --option, hetprob reports robust standard errors as described 45 in Obtaining robust variance estimates. TO illustrate the effect of this option we will our coefficients using the same model and data in our example, only this time adding our hetprob command:
Example • hetprob
y x,
het(xhet)
robust
nolog
Heteroskedastic probit model
Log likelihood = -569.4783
Number of obs Zero outcomes Nonzero outcomes
= = =
I000 452 548
Wald chi2(I) Prob > chi2
= =
65.23 0.0000
Robust y
Coef.
x _cons
2. 22803 .249382 t
Std. Err.
z
P>Izl
[95_,Conf. Interval]
8.08 2.96
O,000 O.003
l. 687355 .0840853
Y .2758597 .0843367
2.768705 .4146789
i insigma2 xhet
i 1. 602537
Wald test of insigma2=O:
i
.2671326
6. O0 chi2(1) =
O.000 35.99
1.078967
2. 126107
Prob > chi2 = 0.0000
The robust standard errors for two of the three parameters are larger than the previously reported conventional standard errors. This is to be expected, even though (by construction) we have perfect model specification, since this option trades off efficient estimation of the coefficient variancecovariance matrix for robustness against misspecificat_on. 4 Specifying the cluster() option relaxes the usual assumption of independence between observations to the weaker assumption of independence jusi between clusters, that is, hetprob, robust cluster() is robust with respect to within-cluster coffelation, There is a cost in terms of efficiency with this option, since in this case hetprob inefficiefitly sums within cluster for the standard error calculation rather than attempting to exploit what might be assumed about the within-cluster correlation (as do the xtgee population-averaged models).
Obtaining predicted values
]
Once you have estimated a model, you can use the predict command to obtain the predicted probabilities for both the estimation sample and other samples; see [U] 23 Estimation and postestimation commands and [R] predict, predict without arguments calculates the predicted probability of a positive outcome. With the xb option, it calculates the index function combination xjb, where x7 are the independent variables in the jth observation and b is the estimated parameter vector. With the sigma option, predict calculates the predicted standard deviation oj = exp(zj2().
_;:,,,-
46
hetprob -- Maximum-likelihood
heteroskedastic
probit estimation
> Example We use predict to corapute the predicted model in order to compare these with the actual
probabilities and standard
deviations
based
on our
values:
predict phat (optionp assumed; Pr(y)) gen diff_p = phat - p • summarize diff_p Variable I
Obs
Mean
diff_p ]
1000
-.0107081
Std. Dev,
Min
Max
.0131869 -.0466331
.010482
• predict sigmahat, sigma gen diff_s = sigmahat - sigma . summarize diff s Variable
Ohs
Mean
Std. Dev.
diff_s
i000
.1558882
.1363698
Min
Max
,0000417
.4819107
Saved Results hetprob
saves
in e():
Scalars e (N)
number of observations
e (re)
return code
e (k)
number of variables
e (chi2)
2
e(k_eq)
number of equations
e(chi2_c)
X 2 for heteroskedasticity LR test
e(k_dv)
number of dependent variables
e (p_c)
p-value for heteroskedasticity LR test
e(N..f) e(N_s)
number of zero outcomes number of nonzero outcomes
e(df..m_c)
degrees of freedom for heteroskedasticity LR test
e(df..m) e (11)
model degrees of freedom log likelihood
e(p) e (ic)
significance number of iterations
e(ll_O) e(ll_c) e(N_clust)
log likelihood, constant-only model log likelihood, comparison model number of clusters
e(rank) e(rankO)
rank of e(V) rank of e(V) for constant-only model
e (cmd)
hetprob
e (opt)
type of optimization
e(depvar) e(title)
name of dependent variable title in estimation output
e(chi2type) e(chi2_ct)
Watd or LR; type of model x 2 test Wald or LR; type of model x 2 test
e(clustvar)
name of cluster variable
e(method) e(vcetype) e(user)
requested estimation method covariance estimation method name of iike]ihood-evaluator program
e(offset#) e(predict) e(cnslist)
offset for equation # program used to implement predict constraint numbers
e(b)
coefficient vector
e(V)
variance-covariance
e(ilog)
iteration log (up to 20 iterations)
Macros
corresponding to e(chi2_c)
Matrices
Functions e(sample)
marks estimation sample
the estimators
matrix of
I i i
hetprob--Maximum-likelih¢od heteroskedasticprobit estimation
47
i
Methodsand Formulas ?
hetprobis implemented as an ado-file. !
The heteroskedastic probit model is a generalizaiion of the probit model since it allows the scale
t
variables. of the inverse link function to vary from observation to observation as a function of the independent The log-likelihood function for the heteroskedas|ic probit model is
lnL = _ wj In_;{xjb/exp(zT)}+ }-'_wj In [1- _{xjb/exp(zT)}] jeS
jffS
where S is the set of all observations j such that yj -¢ 0 and is maximized as described in IN] maximize.
Wj
denotes the optional weights. In L
References Greene, W. H. 2000. Econometric Harvey. A. 1976. Estimating
Analysis. 4th ed. Upper _addle River. NJ: Prentice-Hall. i regression models with multiplicative heteroscedasticity. Ecooometrica
44: 461-465,
AlsoSee Complementary:
[R] adjust, [R] constraint, [R! lincom, [R] linktest, [R] lrtest, [R] mfx, [R] predict, [R] test, [R] testnl, [R] vce, [R] xi
Related:
[R] biprobit, [R] ciogit, [R] casum, [R] glm, [R] glogit, JR] logistic, [R] logit, [R] mlogit, [R] olog!t, [R] probit, [R] scobit, [R] xtprobit
Background:
[U] 16.5 Accessing coefficienb and standard errors, [U] 23 Estimation and post-_timation commands, [u] 23.11 Obtaining robust variance estimates, [u] 23.12 Obtaining scores, [R] maximize
i
....... "_'"',F J
Title hilite -- Highlight a subset of points in a two-way scatterplot
Syntax hilito
yvar xvar [ifexp] [in range], hilite(exp2)
[ graph_options ]
Description The hilitecommand draws a two-way scatterplot highlighting the observations selected by exp2.
Options hilite(exp2) is not optional. It specifies an expression identifying the observations to be highlighted. graph_options are any of the options allowed with graph, twoway; see [G] graph options.
Remarks > Example You have data on 956 U.S. cities, including average January temperature, average July temperature, and region. The region variable is coded 1, 2, 3, and 4, with 4 standing for the West. You wish to make a graph showing the relationship between January and July temperatures, highlighting the fourth region: . hilite tempjan tempjuly, hilite(region==4)
region==4
ylabel xlabel
highlighted
80=
= =
©
60"
'
_ :,"'.3 =_ 4e
._.¢'""
C
•
g (g _'
",
20
io_ a •
.....
g=*
" .
i"_r
• J _
". ,tNi'_': " : "; _'.2"_.: , "-.:
=
,:
4
.; .
e
0
1 Average
July
48
Temperalure
,, :_
hilite -- Highlighta subset of pointsin a two-way scatterplot It is possible to use graph
to product graphs like ffiis, but hilite is often more convenient.
49
q
[3Technical Note By default, hilite uses'.' for the plotting Lvmbbl and additionally highlights using the o symbol. Its default is equivalent to specifying sTabol(.o)as one of the graph_options. You can vary the symbols used, but you must specify exactly two symbols. The first is used to plot all the data and the second is used for overplotting the highlighted Subset.
Methodsand Formulas hilite is implemented as an ado-file.
References Weesie,J. 1999.dr38: Enhancementto the hilite command.Stata TechnicalBulletin 50: 17-20. Reprintedin Stara TechnicalBulletin Reprints,vol. 9, pp. 98-101.
AlsoSee Related:
[R] separate
Background:
Stata Graphics Manual
'"::"
Title I
hist-
Categorical
variable histogram
[
II
Syntax hist
varna,ne [weight]
[if exp] [in range]
[. i._ncr(#) graph_options
]
fweights are allowed; see [U] 14.1.6 weight.
Description hist graphs a histogram of varname, the result being quite similar to graph hist, however, is intended for use with integer-coded categorical variables. hist determines the number of bins automatically, labels are centered below the corresponding bar.
the z-axis
hist may only be used with categorical variables maximum(varname) - minimum(varname) < 50.
with
varname,
is automatically a range
of less
histogram.
labeled, and the than
50;
i.e.,
Options incr(#) specifies how the z-axis is to be labeled, incr(1), the default if varname reflects 25 or fewer categories, labels the minimum, minimum + 1, minimum 4- 2..... maximum, incr (2), the default if there are more than 25 categories, would label the minimum, minimum + 2, ..., etc. graph_options xlabel(), saving().
refers to any of the options allowed with graph's histogram style excluding bin (), and xscale(). These do include, for instance, freq, ylabel(), by(), total, and See [G] histogram.
Remarks b, Example You have a categorical variable rep78 reflecting the repair records of automobiles. 1 = Poor, 2 = Fair, 3 = Average, 4 = Good, and 5 - Excellent. You could type
(Continued
on next page)
5O
It is coded
h_t -- Categoricalvariablehistogram
51
• graph rep78, histogram bin(5)
to obtain a histogram. Youshould specie, bin(5) because your categorical variable takes on 5 values and you want one bar per value. (You could omit the option in this case, but only because the default value of bin() is 5; if you had 4 or 6 bars, you would have to specify it; see [G]histogram.) In any case, the resulting graph, while technically correct, ii aesthetically displdasing because the numeric code 1 is on the left edge of the first bar while the numeric code 5 is on the fight edge of the last bar. Using hist; is better: • hist rep78
434783
0
Repair
Record
1878
not only centers the numeric codes underneath the corresponding bar, it also automatically labels all the bars. hist
You are cautioned: hist is not a general replacement for graph, histogram, hist is intended for use with categorical data only, which is to say, floncontinuousdata. If you wanted a histogram of automobile
prices,
for instance,
you
would
still want
_o use the graph,
histogram
command.
;:
,_r,.
52
hist -- Categorical variable histogram
I> Example You may use any of Research and Survevs on election day data in Lipset (1993)--you • hist
candi
the options you would with graph, histogram. Using data collected by Voter based on questionnaires completed by 15,490 voters from 300 polling places originally printed in the New York Times• November 5. 1992 and reprinted draw the following graph:
[freq=pop],
by(inc) total ylab ytine noaxis title (Exit Polling By Family Income)
$50-75k
$75k+
6
= o o It.
.6
o,,_' .... L, ' Candidale
Exit Polling
voted
for,
1992
by Family
Income
[] Technical Note In both of these examples, each bar is labeled; if vour categorical you may not want to label them all. Typing
variable takes on many values,
hist myvar, incr(2)
would labeleveryotherbar.Specifying incr(3) would labeleverythirdbar,and so on,
Methods and Formulas hist is implemented
as an ado-file.
References Lipset, S. M. ]993. The significance
of the I992
Also See Related:
[R] spikeplot, [G] histogram
Background:
Stata Graphics Manual
election.
Political
Science
and Politic,_ 26_1_: 7-16,
Title hotel -- Ho_lling's
generalized means test
Syntax hotel varlist [weigh_ [iJ exp] [in range] [, by(varname)notable aweights
and fweights
]
are allow _d: see [U] 14,1.6 weight
DescriptiOn hotelperforms Hotelling's T-squared test for testing whether a set of means is zero or, alternatively, equal between two groups
i
Options by(varname) specifies a var able identifying two groups; the test of equality of means between groups is performed. If by '.) is not specified, a test of means being jointly zero is performed. i
t
notablesuppresses printing
table of the means being compared.
Remarks hotel performs Hotelling's T-squared test of whether a set of means is zero, or two sets of means are equal. It is a multivariate est that reduces to a standard t test if only one variable is specified.
I, t i I
_ Example You wi_h to _est whether a new fuel additive improves gas mileage in both stop-and-go and highway situatiotls. Taking tw_lye cars, you fill them with gas and run them on a highway-style track, recordingtheir gas mileage. Y_)uthen refill them and run them on a stop-and-go style track. Finally, you repeat the two runs but this time use fuel with the additive. Your dataset:is . describe Contains d_ta from gasexp.dta obS : 12 vats : size:
i
variable n_me
i ! i
id bmpgl ampgl bmpg2 ampg2
_
5 288 (c _.9% of memory free) storage type float float float float float;
lisplay _ormat
value label
/,9. Og /,9.Og /,9.0g /,9.0g /,9.0g
13 Jul
2000
13:22
variable label car id trackl before additiv_ trackl after additive_ track 2 before additive track 2 after additive
Sortdd by :
53
r'_
54
hotel -- Hoteiling's T-squared generalized means test [
To perfor_ zero:
the statistical test, you jointly test whether the differences
• g_n |
diffl
= ampgl
- bmpgl
g_n dill2 = ampg2 | hgtel diffl dill2
bmpg2
Variable
0bs
diffl dill2
12 12
I
Mean
Std.
1.75 2.083333
Dev.
2.70101 2.906367
1-g_oup Hotelling's T-squared = 9.6980676 F t, st statistic: ((12-2)/(12-1)(2)) x 9.6980676 H0:
Vector
of means is equal ¢o a vector F(2,10) = 4.4082
Prob
The meat
> F(2,10)
=
in before-and-after
Min
Max
-3 -3.5
5.5
results are
5
= 4.4082126
of zeros
0.0424
are different at the 4.24% significance level.
[] Technical _ote We used Hotelling's T-squared test because we were testing two differences jointly. Had there been onlvlone difference, we could have used a standard t test which would have yielded the same results as_totelling's''
test:
* W_ could have performed ampgl = bmpgl
the
test
like
this:
t test
Vari
Obs
Mean 22.75
12
• tt_st
20.68449
24.81551
2.730301
19.26525
22.73475
12
1.75
•7797144
2.70101
.0338602
3.46614
mean(ampgl
- bmpgl)
Ha:
= mean(diff)
mean(dill) ~= 0 t = 2,2444
P >
Itl =
mean(dill) > 0 _ = 2.2444
P > t =
0,0232
this: = 0
t test Std.
dlffl
12
1.75
.7797144
of freedom:
Ha: mean < 0 t = 2.2444 0.9768 this:
Err.
Std.
Dev.
2.70101
[95_
Conf,
.0338602
Interval] 3.46614
11 He:
Or like
= 0 Ha:
0.0463
Mean
t =
Interval]
3.250874
Obs
PI<
Conf,
.7881701
Variable
Degrees
[95_
.9384465
0,9768
dill1
One-_ample [
Dev.
21
Ha_ mean(dill) < 0 t = 2.2444
* Or like
Std.
Err.
12
Ho:
P < t =
Std.
meam(diffl) Ha:
P >
mean t =
Itl =
= 0
-= 0 2.2444 0.0463
Ha: mean > 0 t = 2.2444 P > t =
0.0232
""---
hotel-- Hotellin_sT-squaredgeneralizedmeanstest
55
. _otel dill1 i
Variable
i
[
0
Mean
diffl
Std.
1.75
Dev.
Min
Max
2.70101
-3
5
1-Sroup _otelling's T squared = 5.0373832 F _est statistic: ((i!2-I)/(12-I)(I))x 5.0373832 = 5.0373832
I
HO{ Vecter of means i 3 equal to a vector of zeros F(I,II) = 5.0374 Prob > F(I,II) = 0.0463
> Example Now"donsider a variation _n the experiment: rather than using 12 cars and running each car with and without the fuel additiv, you run 24 cars, 12 with the additive and 12 without. You have the following!dataset: . d_scribe
! i
I i
I
Contains data from ga: _xp2.dta o_s: 24 vats: 4 size: 480 97.4_ of memory free)
8 Sep 2000 12:19
[
! storage variable name type
display format
id mpgl mpg2 additive
_9.Og Z9.0g _9.0g _9.0g
float float float float
value label
Variable label
yesno
car id _rack 1 track 2 additive?
Sorted by: • tab additive additive?
Fr_q.
Percent
_um. i
no
12
50.:00
50.00
yes
12
50.00
100.00
Total
24
I00,00
jr
I
i
This is an_unpaired expefime_t because there is no natural pairing of the cars; we want to test that the rneanslof mpgl are equal for the two groups specified by additive as are the means of mpg2:
{
(Continued on next page)
_
=_
60
d9 -- ICD-9-CM diagnostic and procedure codes
] !
t ICD-9 codes and, if it does, icd9_] clean modifies the _ariable to contain the codes in either of two standard formats. Use of icd9[p]_clean is optional: all icd9[p] commands work equally well with cleaned or uncleaned codes. 'I_e_e are numerous ways of writing the same ICD-9 code, and icd9[p] clean is designed (1) to ensure c insistency, and (2) to make subsequent output look better. icd9[p] uncleaned) icd9[p] ge code. icd9 ICD-9 code
generate produces new variables based on existing variables containing (cleaned or ICD-9 codes, icd9[p] generate, main produces newvar containing the main code. aerate, description produces newvar containing a textual description of the ICD-9 p] generate, range() produces numeric newvar containing I if varname records an in the range listed and 0 otherwise.
icd9_p] lookup and icd9[p] search are utility routines that are useful interactively, icd9[p] lookup sin ply displays descriptions of the codes specified on the command line, so to find out what diagnostic _:913.1 means, you can type icd9 lookup e913.1. The data you have in memory are . I lrrelevant-_and remain unchanged when using icd9[p] lookup, icd9[p] search is like icd9[p] lookup ex!:ept that it turns the problem around; icd9[p] search looks for relevant ICD-9 codes from the dd_cription given on the command line. For instance, you could type icd9 search liver or icd9p
s._arch
liver
to obtain a list of codes containing the word "liver".
icd9[p] query displays the identity of the source from which the leD-9 codes were obtained and the textual _escription that icdg[p] uses. Note that! ICD-9 codes are commonly written in two bays,, with and without periods. For instance, with diagnostic codes, you can write 001, 86221. E8008. and V822, or you can write 001., 862.21, E800.8, and V82.2. With procedure codes, you can write 01, 50. 502. and 502l, or 0l., 50., 50.2. and 50.21. _he icd9[p] command does not care which syntax you use or even whether you are consistent. _ase also is irrelevant: v822, v82.2, v822. and v82.2 are all equivalent. Codes may be recorded w_h or without leading and trailing blanks.
Optionsfor use with icd9[p]check any tells ic the code, 230.52 option, conside list
59[p] check to verify that the codes fit the format of leD-9 codes but not to check whether are actually valid. This makes icd9_p] check run faster. For instance, diagnostic code _r 23052 if you prefer) looks valid, but there is no such ICD-9 code. Without the any 30.52 (or 23052) would be flagged as an error. With any. 230.52 (and 23052) is not d an error.
tells cd9[p] chock that invalid codes were found in the data. 1. 1.1.1. and perhaps 230.52 if any is n )t specified, are to be individually listed.
genaratet
ewvar) specifies that icd9[p]
check
is to create new variable newvar containing,
for
each observation, 0 if the code is valid and a number from 1 to I0 otherwise. The positive numberslindicate the kind of problem and correspond to the listing produced by icd9[p] check. For instance. 10 means the code could be valid, but it turns out not to be on the official list.
Options for use with icd9[p] clean dots specifies whether periods are to be included in the final format. Do you want the diagnostic codes recorded, for instance, as 86221 or 862.21? Without the dots option, the 86221 format would b_ used. With the dots option, the 862.21 format would be used. pad specifids that the codes are to be padded with spaces, front and back. to make the codes line up vertically.: in listings. Specifying pad makes the resulting codes look better when used with most other Stata commands.
i
icd9 -- ICD-9-CM diagnoSticand procedure codes
Options fOr i
61
with icd )[p]generate
main,descrip}ion,and ra _ge(icd9rangelist) specify what icd9[p] generate is to calculate. In all cases, varname specifies variable containing ICD,9 codes. main specifies ihat the malt .'ode is to be extracted from the IED-9 code. For procedure codes, the
i
i i
main code i_ the first tw_ characters. For diagnostic codes, the main Code is usually the first three or four characters (tlie characters before the dot if the code has dots) In any case, icdg[p] generate &Ses not care _hether the code is padded With blanks in front or how strangely it might be written; :icd9[p] gene rate will find the main code and extract it. The resulting variable is itself an ICD-Ocode and may be used with the other icd9[p] subcommands. This includes icd9[p] generate, ilain.
i ! i i_
descriptiondreates newva, containing descriptions of the ICD-9 codes.
I i{ i I
long is for _e with desc: iption.It specifies thai the new variable, in addition to containing the text describing the code, is to contain the code, too. Without long, newvar in an observation might contain "bro1_chus injury-closed". With long, it would contain " 862.21 t_ronchus injury-closed". end modifie_ long (speci _'ying end implies long) and places the code at the end of the string: "bronchus injury-closed 8I 2.21".
! i
i
range(icd9ran_elist) allows you to create indicator variables equal to l when the ICD-9 code is in the inclusive! range specifi _d.
Optionsfor usewith icd )[p]search
!
i
or specifies thai ICD-9 codes are to be searched for entries that contain any of the words specified after icd9[p I search,Th, default is to list only entries that contain all the words specified.
i
Remarks
I
code is
Let us begin _withdiagnost
codes--the
codes icd9 processes. The format of an ICD-9 diagnostic
[blanks { O-9,V,v} {0-9} {0-9} [. ] _0-9 r [0-9] jl [blanks] or
I
i i ; .
i
[blanks! E.e } { 0-9 } { 0-9 } {0-9 }[. ][0-9 [0-91 ] [blanl_s1 icd9 can dell with ICD-9 tiagnOstic codes written in any of the ways the above allows. Items in square brackets tare optional, the code might start with some number of blanks. Braces { } indicate required items. _he code the_ either has a digit from 0 to 9 the letter V (utmercase or lowercase) (first hne), or thei letter E (upl_ercase or lowercase ' '_.. second line).' After that, it has--two or mo re d_'g"_ts s, perhaps followed b a enod and th v u to tw e dvm ha s tollowed b ' more b' " i ! Y P " _t en it may ha "e p omor "_'siper p. " y
,anks .
l
_;;::
62
icd9 --
ICD-9-CM diagnostic and procedure codes
All of the following
meet the above
definition:
00: ool
')ol 001,9 O019 862 ._2 862,22 E80_). 2 e8092 V82|
Meeting t_ae above the above[definition, Examl_les|
definition of which
of currently
does not make the code valid. 15,186
defined
are currently
diagnostic
codes
There
are 233.100
possible
codes meeting
defined include
l
code
I
t i
I
description
001 001.0
cholera* cholera d/t vib cholerae
001.9 001.1
cholera cholera nos d/t vib el tot
999
complic medical care nec*
VOl
communicable dis contact*
V01. i VOI. 701.20 VOI. 3 VOl. 4 VOI. 5 VO1.6 VOl. 7 YO1.8 V01.9
tuberculosis contact cholera contactcontact poliomyelitis smallpox contact rubella contact rabies contact venereal dis contact viral dis contact nec communic dis contact nec communic dis contact nos
. . .
E800 E800.0 E800.1 E800.2 E800.3 E800.8 E800.9
rr rr rr rr rr rr rr
collision nos* collision nos-employ coll nos-passenger coll nos-pedestrian coll nos-ped cyclist coil nos-person nec coil nos-person nos
o , o
"Main ,eodes" refer to the part of the code to the left of the period. v82, and !_800 ..... E999 are main codes. There are 1.182 diagnostic
001,002 ..... main codes.
999. v0] .....
The m'Ain code corresponding to a detailed code can be obtained by taking the part of the code to the left lof the period, except for codes beginning with 176. 764. 765. v29. and v69. Those main codes are not defined, yet there are more detailed codes under them:
icd9 -- ICD-9-CM diagnostic and procedure codes I
cdde
d,:scription
[
1}'6 if(6.0 176,1
CDDE DOES NOT EXIST, but $ codes starting with 176 do exist: sl in - ka_si's sarcoma sl tisue - kpsi's srcma
764 754.0 754. O0
C )DE DOES NOT EXIST, but 44 codes starting with 7i54 do exist: It for-dates w/o let real* li :ht-for-dates winos
63
i
.5.
755 7_5.0 7_5. O0
C )DE DOES NOT EXIST, but 22 codes starting with %5 do exist: extreme immaturity* e_treme immatur wtnos
I
V_9 V:_9.0
O )DES DOES NOT EXIST, but 6 codes stating with V29 do exist: nt obsrv suspct infect
i
V_9.1
nt obsrv
I
V69 V_9.0 V619.1
O )DE DOES NOT EXIST, but 6 codes starting with V69 do exist: la k of physical exercse inirpprt diet eat habits
suspct
neurlgcl
"'_"
!
Our solution iis to define f)ur new codes:
t ! i
!
code
description
176 764 765 729 g69
kaposi's sarcoma (Stata)* light-for-dates (Stata)* immat & preterm (Stata)* nb suspct cnd (St,am)* lifestyle (stata)*
Thus, there are 15,186 + 5 = 15,191 diagnostic
i
Things are less confusing format of I CD-9iprocedure
!
I I
I
'
codes, of which
_'ith respect to procedure
codes_the
1,181 + 5 = 1,186 are main codes. codes processed
by icd9p.
The
co :les is [banks]
{0-9}{0-9}
[. ] [0-9 [0-9]]
[blanks]
Thus, there are i0,000 possil: e procedure codes, of which 4,275 are currently valid. The first two digits represent _e main co& of which 100 are feasible and 98 are currently used (00 and 17 are not used).
Descriptions The degcriptidns given for each of the codes is as found in the original source. The procedure codes • contain' the addition of fve new codes b_, us. An asterisk on the end of a description n_ d_cate_ "" _
i_ I
that the c°trespoiding ICD-9 tiagnostic code has subcategories. icd9[pJi quer_ reports thebriginal source of the information
1
on the codes:
r F J
64
icd9 -- ICD-9-CM diagnostic and procedure codes
• _cd9
query
_dta: I
i 1
Dataset from http://www.hcfa.gov/stats/pufiles.htm obtained 24aug1999 file http://www.hcfa,gov/stats/icdgv16.exe Codes 176, 764, 765, V29, and V69 defined
I
-- 765 176 kaposi's immat _ preterm sarcoma (Stata)* (Stata)* V29 nb suspct end (Stata)* V69 lifestyle (Stata)* cd9
query
J _d_a: Dataset obtained 24aug1999 • from http://www.hcfa.gov/stats/pufiles.htm file http://www.hcfa.gov/stats/icd9vl6.exe
> Example You t_ve a dataset containing up to three diagnostic codes and up to two procedures on a sample of 1,000 Ipatients: _se patients, _ist in 1/10 7.
patid I
I:
I_.
clear diagl 65450
diag2
3 2
710.02 23v.6
5 6 7 8 9
861.01 38601 705 v53.32 20200
procl 9383
proc2
37456
8383
17
2969
9337 7309 7878 0479
8385 951
i0
464.11
7548
diag3
E8247
20197
!
4641
Do not tD, to make sense of these data because, in constructing procedure codes were randomly selected.
this example,
the diagnostic
and
- _,-_Be_inlbvnoting that variable diagl is recorded sloppily--sometimes the dot notation is used and sometimes not, and sometimes there are leading blanks. That does not matter. We decide to begin by using icd9 cd9
clean clean
to clean up this variable:
diagl
di_gl contains invalid ICD-9 codes r (459) ;
icd9 clean refused because there are invalid codes among the 1.000 observations, check to find and flag the problem observations (or observation, as in this easel:
We can use icd9 :
i-|
!
[
) ) i
i
_....
icd9-_-ICD-9-CMdiagnostic and proce
. icd9 check diagl, gen(prob) diagllcontains i invalid ¢odes: I. Invalid placemer_ of period 2_ Too)many periods
t
0 0
I
3, 4_ 5i
Cod_ too short Cod# too long Invalid let chaz (not 0-9, E, or V)
0 0 0
i
6_
Invalid 2nd chax (not 0-9)
0
I
81 7_ 9.
Invalid 4th chat (not 0-9) Invalid Invalid 3rd 5th chat chat (not (not 0-9) 0-9)
0 0i
I
10,
Cod_ not defined
0
,ot.
i
i
. list pati_ diagl prob Lf prob
I
[
2.
patid 2
diagl 23v. 6
65
prob 7
Let's assume that _ve go back t_ the patient records and determine that this should have been coded 230.6: • replace d_agl = "230.6 (i re_l change made) . drop prob
if patid==2
We now tD,_againlto clean up t e formatting of the variable: • icd9 cleam diagl (643 dhange_ made) • lis_ in 1/10
_;
1, 2. 3. 4.
patid 1 2 3 4
diagl 65450 2306 71002 1026
diag2
I
5.
5
86101
6 7 8 9
38601 705 V5332 20200
2969
I )
6. 7. 8. 9.
[
10.
10
46411
20197
diag3
37456
procl 9383 8383
proc2 17
629
7548
E8247
9337 7309 7878 0479
8385 951
4641
)
Perhaps we prefer!the dot notati )n. icd9 clean can be used again on diagl, and now we will clean
l
Up diag2
and diag3:
• ted9 clea_ diagl, dots (936 _he/Ige_made) • icd9 clean diag2, dot_ (551 Changes made) • icd9 clea_ diag3, dote (i00 Changes made)
i
)
(Continued on next page)
_"
! i_d9 -- ICD-9-CM diagnostic and procedure codes
66
• lit
in
1
|
!
1/10
1
diagl 654.50
diag2
i.
patid
diag3
procl 9383
proc2
2.
2
230.6
374.56
3.
3
V10.02
8383
17
4.
4
102.6
5.
5
861.01
6.
6
386.01
7.
7
705
7309
8385
8.
8
V53.32
7878
951
9.
9
202.00
754.8
10.
10
464.11
201.97
629
We now turn to cleaning codes:
296.9
9337
E824.7
0479 4641
the procedure
codes.
We use icd9p
diag3
procl 93.83 83.83
(emphasis
on the p) to clean
these
l
. iccl9P clean procl, (816|changes made) |
dots
. ic_9p clean proc2, (140|changes made) !
dots
• li_t
in
1/10
I. i 2.
patid 1 2
diagl 654.50 230.6
diag2
3.
3
V10.02
4.
4
102.6
5.
5
861.01
6.
6
386.01
7.
7
705
73,09
83.85
8.
8
V53.32
78.78
95.1
9.
9
202.00
754.8
10.
10
464.11
201.97
374.56
296.9
icdl p check
E824.7
04.79 46.41
rules
clean
and icdgp
for the code:
clean
il does
not check
valid
ICD-9
procedure
codes;
168 missing
the code
is itself
proc2
contains
invalid
codes:
Invalid
2 3
Too many Code too
4 5
Code too long Invalid 1st char
(not
0-9)
0 0
6
Invalid
2nd
char
(not
0-9)
0
7
Invalid
3rd
char
(not
0-9)
0
8
Invalid
4th
char
(not
0-9)
0
Code
that
that the variable
values)
1
10
only verify
procl
contains
icd p check proc2
93,37
that both icd9
cleaned follows the construction icd_[p] check does that:
(proc
17
62.9
It is imDDKant to understand being valid.
proc2
not
placement
of period
0
periods short
0 0
1
defined
1
Total
Note that d_ag2 has an invalid generate( code. We;ould did above _ith icd9 check, .
find it using
icdg_] han create codes. For _nstance.
textual
new variables
containing
icd9p
descriptions
check,
generate(), just as we
of our diagnostic
and procedure
_
icd9 -- ICD-9-cM
i
diagnostic
and
proc_lure codes
67
• icd9 gen!tdl = diagl, desc . sort pared • list pared diagl tdl m 1/10 1. 2. 3. 4. 5 6
pa_id 1 2 3 4 5 6
diagl 654.50 230.6 VlO.02 102.6 861.01 386.01
tdl cerv incompet preg-unsp ca in situlanus nos hx-oral/ph_aTnxmalg nec yaws of bo_e _ joint heart contUsion-closed meniere dis co¢hlvestib
7 8 9
705 V53.32 202.00
disorders of sweat gland* ftng autmt¢ dfibrilla_or ndlr lymunsp xtrndl ors
7 8 9
I !
I
10 iCd9_] I0 464.11 ac tracheitis w obstruct Notethat _enerate, escription does notpreserve thesort order ofthedata(andneither does icdg[p] cheek unless you specify the any option),
Recall that pro6edure-codep:ioc2 had an invalid codel Even so, icdgp generate, is willing to_create a textual de: cription variable:
descript
. iedgp gen!tp2 : proc2, desc (I nonmissidg value inva!_idand so could not be labeled) . sortlpatid listipatid proc2 tp2 i:I1/10 i
pared
proc2
i
I.
' _:
2. 3,
i2 17 _3 5
i
5. 6. 7. 8.
_i
83.85 95.1
Y B
9.
itp2
musc/tend Img change nec form _ structur eye exam*
D
lo.
io
tp2 contains nothing when F "oc2 is 17 because 17 is not a valid procedure code. icdg[p] g*ner_te
can also reate variables containing main codes:
. icdg!gen main1 : diagl
I
main
listlpatid!diagl pati_ dinE1mainl [n I/I0 mainl 1. 1 654.50 654
I
3.
3 vtoo2 2 4
230.6 102.6
rio
5, 6,
_
861.01 386.01
861 386
7. 8.
7 8
705 V53.32
705 V53
10. 9.
109
464.11 202.00
464 202
2. 4. i
i
icdgp generate, :
230 102
_ain can sit ilarlygenerate main procedure codes. i
Sometimeslone i_ merely exa_fining an observation: • list
dins*
_f patid==56_
ion
_-
68
icd9-
ICD-9-CM diagnostic and procedure codes
I ...........
diagl 56 I.
diag2
diag3
526.4
If we woladered what 526.4 was, we could type ! . i_d9 |
lookup
1 m_tch
found:
526.4
icd9[p]
526.4
inflammation of jaw
]_ookup has the ability to list ranges of codes: I • i_d9
lookup
526/527
12 _atches found: 526 jaw diseases* 526.0 devel odontogenic cysts 526.1 fissural cysts of jaw 526.2 cysts of jaws nec 526.3 cent giant cell granulom 526.4 infl_mmation of jaw 526,5 alveolitis of jaw 526.8 other jaw diseases* 526.81 exostosis of jaw 526.89 [526.9 527
icd9[p]
st_arch /
jaw disease nec jaw disease nos salivary gland diseases*
has the ability to go from description to code:
• i_d9 search jaw
disease
|
4 m_tches found: |526 jaw diseases* 1526.8 other jaw diseases* 526.89 jaw disease nec
q 526.9
jaw disease nos
I
Saved Results icd9
ahd icd9p save in r()" Scalars r(e#) r(esum)
number of errors of type # total number
of errors
References Gould, W. 2p00. din76:ICD-9 diagnostic and procedure Technica I Bulletin Reprints. vol. 9. pp. 77-87. t l
codes.
Slate
Technical
Bulletir_ 54: 8-16.
Reprinted
in State
impute--- Predictmissingvalues
i
F
Syntax imputedepvdr
varlist [w_'ght]
Iif
exp] [in range],
g_enerate(newva,'l)
[ _ rp(nevar2)] aweights
and fweiShts
are allow_
see [V] 14.L6 weight.
Description impute mils in
missing vail _s; depvar is the variable whose missing values are to he imputed. vartist is tile list of variables m which the imputations are to be based and newvarl is the new '
i
i
variable to contain the imputations.
!
conducted dfficientty: this nece sitates a liner of 31 variables in varlist.
impute organi_zesthe casesby pa_ternsof missingdata so the missing-valheregressionscan be
Options generate(t/ewva'r
i [
I i
1) specifies he name of the new' variable to be created and is not optional.
varp(newvar2) specifies the n_me of a new variable to contain the variance (not the standard error'_ of the pr_edicti_n.
Remarks
i
In observation s where depva F is not equal to missing; newvarl is set equal to depvar and newvar2 (if specified) is set to zero. Whlere depvar is missing, neuvarl is imputed using the prediction from
i ! !
the best available subset of othelrwise present data. r_ewVar2(if specified) is setto the variance of the prediction. This variance is in tl_esense of predicts stdp option, although squared: see [R] predict. It is an estimate df how far thel prediction of the mean would differ from the actual data point were
l
_t known. This is not the only method or coping with missing data, but it is often much better than deleting cases with any migsing data, wl_ich is the default. For a discussion of different methods of imputation. see, for example, Little and Rubin (1987).
! i
Example imputemay be used in conjunction with, for instance, regression (or an5' estimation technique) to avoid the lo_s of an unacceptal:le number of cases due to missing data. Bear in mind, however, lhat the subsequent estimates may b : biased because any variable imputed by impute is only an estimate of the unknown, true value. In he case of linear regression, a reasonable bound (in fractional terms) for the bias!is gix,!enby the ratio of the mean of newvar2 to the variance of newvarl. Usualh'. the bias is toward zerO, meaning tl-at the effect of the variable will be underestimated. 69
.....
,v
,,,,_ut_
u
rr_olc(
missing
values
You have been hired by the restaurant industry to study expenditures on eating and drinking. You, have &ta on 898 U,S. cities: describe C_ntains data from emd.dta obs: _ars:
898 9 34,124 (96.6_
ize: v
1980 City Data 13 Jul 2000 14:00
iable name
type storage long float int float float
f_s it.eat iz:ome_pc l_.rsales_pc jaltemp
of memory free)
format
label
display _10.Og _9.0g _8.0g _9.0g _9.0g
value
variable label
precipitation
float
_9.0g
state/place code In(Dining sales per capita) Per capita money income in(retail sales per capita) Median January temperature (Fahrenheit) Annual precipitation (in.)
in,income me_ian_age hh_ize
float float float
_9.0g _9.0g _9.0g
in(median per capita income) Median age Persons per household
...... 1
So,ted by: !
You
beglnby running theregression: | • _egress In_eat In_rsales jantemp precip in_inc median_age Source
I
SS
df
MS
I Model I Residual ]
87.7285014 45.1948678
6 657
14.6214169 .068789753
E 132.923369
663
.200487736
Total I In_eat
Coef.
Std. Err.
t
P>It[
hhsize
Number of obs = F( 6, 657) =
664 212.55
Prob > F R-squared Adj R-squared Root MSE
0.0000 0.6600 0.6569 .26228
=
[95Z Conf. Interval]
In_:sales_pc jantemp pre:ipitat~n i.n_income
.6611241 .0019624 -.0014811 .I158486
.026623 .0007601 .0008433 .056352
24.83 2.58 -1.70 2.06
0,000 0.010 0.090 0-040
.6088476 .0004698 -.0030869 .0051969
.7134006 .003455 .000224Z .2265003
m,_dian_age hhsize .cons
-.0010863 -.0050407 -I.377592
.0002823 .0004243 .4777641
-3.85 -11.88 -2.88
0.000 0.000 0.004
-.0016407 -.0058739 -2.31572
-.0005319 -.0042076 -.459463
Despit, having data on 898 cities, your regression was estimated on only 664 cities_74% of the original 8 )8. Some 234 observations were unused due to missing data. In this case, when you type snrnmari: e, you discover that each of the independent variables has missing values, so the problem is not that one variable is missing in 26% of the observations, but that each of the variables is missing in some _servations. In fact, summarize revealed that each of the variables is missing in roughly 5% of th_ observations. We lost 26% of our data because, in aggregate. 26% of the observations have one )r more missing variables. Thus, |"+'eimpute each independent variable on the basis of the other independent variables: . i_pute In_rtl jantemp precip In_inc medage hhsize, gen(i_In_rtl) 4._0Y, (44) observations imputed impute } jantemp in rtl precip In_inc medage hhsize, gen(i_]antmp) 5.B0_, (53) observations imputed
f
impute -- Predict missing values
71
. impute _recip In rtL jantemp In_inc medage hhsize, gen(i_precip) 4.i56_(41) observati)ns imputed . impute In_inc In rtL jantemp precip medage!hl_size,gen(i_in_inc) 4.!34_ (39) observati)ns imputed . impute Medage In rt jantemp precip In inc hhsize, gen(i_medage) 4._5% (40) observati .ns imputed . impute lihsize In rt jantemp precip in_inc medage, gen(i_hhsize) 5._3_ (47) observati ,nsimputed
Thatdone,we can now re-estmatethe recession on the imputedvariables: • regress !in,eat i_Injsales Soul4ce Mod_l Residual ! _ Total
: i
In_eat
!
i _
S_
df
108.8_923 63.792_
i_jantmp i_precip i_in_inc i_median_ase i hhsize
.45
172.65:i145 Conf.
MS
6
18.1432051
891
,071596986
897
.192477308
Std. Err.
t
Number of obs = F( 6, 89_) = Prob > F =
898 253.41 0.0000
R-squared = Adj R-squexed = Root MSE =
0.6305 0.6280 .26758
P>ItI
[95% Conf. Interval]
i_im_rsales i_jantmp i_precip i_in_inc
.660_ )6 .0021G .9 -.0013_88 .095883
.0245827 .0006932 .0007646 .0510231
26,89 3.03 -1.74 1.88
0.000 0.002 0.083 0,061
.6126593 .0007414 -.0028275 -.0042764
.7091528 .0034625 .0001739 .1960024
i_median_age i_khsize _cons
-,0011_34 -.0052508 -I.143142
.0002584 .0003953 .4304284
-4.35 -13.28 -2.66
0.000 0.000 0,008
-.0016304 .0060267 -1.987914
-.0006163 -,004475 -.2983702
Notethat the regressionis no_ estimatedon all 898observations. <1
> Example impute canalsobe used 4th factor to extend[actorscoreestimatesto caseswith missing data.Forinstance, we havea /afiantof the automobile dataset (see[U] 9 Stata'son-linetutorials and sampledataSets)that conltinsa few additionalvariables.Wewill begin by factoringall but the price
vadable;
see [R] factor
• factor m_g-foreign, f ctors(4) (obs=66) (principal _actors; 4 factors retained) Eigenvalue Difference Proportion
Factor
Cumulative
3
i
1 2 3 4 5 6 7 8 9 I0
6. 99066 1.39528 O. 58952 O. 29870 0.24252 O. 12598 0.03628 -0.01457 -0.02732 -0.05591
5. 59588 0.80576 O. 29082 O. 05618 0.11654 O. 08970 0.05085 0.01275 O. 02860 0.05736
O. 7596 0.1516 O. 0641 O. 0325 O. 0264 O. 0137 0.0039 -0.0016 -0.0030 -0.006i
O.7596 0.9112 O. 9753 1. 0077 1. 0341 1.0478 1.0517 1.0501 1.0472 1.0411
tt I2 13
-0.11327 -0.11891 -0. 14605
O.00564 0,02714
-0.0i23 -0.0129 -0.0i59
1. 0288 1.0159 1.0000
r
72
impute
--
Predict missing values Factor Loadings
i Variable
1
mpg rep78 I rep77 headroom Irear_seat trunk
2
3
4
Uniqueness
-0.78200 -0.51076 -0.27332 0.56480 0.66135 0.72935
-0.02985 0.68322 0.70653 0.26549 0.20473 0.37095
-0.06546 -0.1i181 -0.32005 0.29651 0.36471 0.28176
0.33951 -0.01428 0.04710 0.16485 0.02062 0.12140
0,26803 0.25963 0.32145 0.49542 0.38727 0.23633
weight length turn ! displacement
0.95127 0.94621 0.88264 0.92199
0.10135 0.19595 -0.05607 0.06333
-0.18056 -0.05372 -0.08502 -0.17349
-0.09179 -0.10325 0.01169 -0.02554
0.04378 0.05274 0.21043 0.11518
_ar_ratio | order |foreign
-0.82782 -0.25907 -0.75728
0.06672 0.15344 0.30756
0.24558 0.01622 0.19130
-0.10994 0.14668 -0.29188
0.23787 0.88756 0.21014
I
There appear interpreta_on
to
be two we might
factors interpret
here. the
Let's pretend that we have given first factor as size. We now obtain
the first the factor
two factors scores:
an
||
. s_ore fl f2 (based on unrotated factors) (2 scorings not used) Scoring Coefficients 1 2
Variable I
mpg rep78 rep77 headroom _ear_seat I trunk
-0.02094 -0.03224 -0.11150 0.05530 0,03355 0.04603
0.11107 0.44562 0.27942 0.10017 0.02812 0.20622
I
0.12250 0.39997 0.04562 0.19281 -0.08534 0.00638
-0.13040 0.60223 -0.12825 0.11611 0.03528 0.06433
weight length turn displacement g_ar_ratio order
-0.06469
foreign Although nfissing observati(
is not v ]ues is:
(we
0.28292
revealed
by
this
output,
in 8 cases
would
see
that
if we typed
the
scores
summarize).
could
To
not
impute
. i_ _ute fl mpg-foreign, gen(i_fl) 10.91_ (8) observations imputed i, _ute f2 mpg-foreign, gen(i_f2) I0._1Z (8) observations imputed And
we _ ight
now
run
a regression
of price
(Continued
in terms
on next
of the
page)
two
thctors:
the
be calculated factor
scores
because
of
to all the
impute -- Predict imissingvalues i
. regre_s
price
i_f3
Source
i_f2 SS
df
MS
Number
of obs =
F( 2, Model
15¢_._23103
Residual
47£ 342293
Total
63_ )65396
price i_fl
t
73
74
71)=
2
79611551.5
Prob
71
6702004.13
R-squared:
=
0.2507
8699525.97
kdj R-squ_red Root MSE
= =
0.2296 2588.8
73
Err.
t
P>lt
> F
3oef.
Std.
3.347
315.7177
3.88
0.000
595.8234
{
[95Y. Conf.
=
ti.88 0.0000
Interval] 1854.87
i_f2
I
911:2878
339.9821
2.68
0.009
233.3827
1589.193
icons
J
626 1.285
301.7093
20.76
0.000
5660.694
6863,877
Methodsand Formulas imputeis implemented
Lsan ado-file.
Consider the command
repute y xl X2 ... Xk, gen(_)
When y is not missing,
varp(_).
=yand_=0.
Let y9 be an observatiol br which y is missing. A regressor list is formed containirig all x's for whic_ xij is not missing. If _e resulting list is empty, missing. !OtherWise a regres!iion of y on the list is dsdmated (see [R] regress) the predicted Value of yj (si,'e IN] predict), t,"j is defined as the square of the prediction, as Calculated by _redict, stdp; see [Ri predict.
from xl, x2 ..... xk _.3 and _j are set to and _j is defined as standard error of the
References i
Goldstein,R. 1996.sedl0: Patters of missingdata, Stata TechnicalBulletin32: 12-13, Reprintedin Stata Technical Bulletin Reprints.vol. 6. p. I 5.
i[
_.
1996. sedl0.I: Patternsof!missing data. update. Stata TechnicalBulletin 33: 2, Reprintedin Stata Technical BulletinReprints,vol. 6, pp. 15-116.
l,ittle.R. i. A. and D. B. Rubin. 1987.StatisticalAnalysis u4OJMissingData. New York:John Wiley& Sons. }
.
Mander.Ai asd D. Clayton.I999 sg116:Hotdeckimputation.Stata TechnicalBulletin 51: 32-34. Reprintedin Srata TechniralBulletin Reprints, v,)t.9, pp. 196-199. -_.
2000. sgll6A: Update to hotdeck imputation.Stata TechnicalBulletin 54: 26. Reprinted in Smta Technical BulletirtReprints, vol. 9, p. )9,
AlsoSee
i
Complementary:
[R] pr, diet
Related:
[R] ipelate, JR]regress
..... Title
,t ;
i
Quick reference for reading data into Stata
Description This er_try provides a quick reference for determining which method to use for reading non-Stata data into _hemoD,. See [U] 24 Commands to input clam for more details.
Remarks Summary bfthe different methods insheet o inshe, t reads text (ASCII) files created by a spreadsheet o The da a must be tab-separated space-s ,_parated. o A sing] _ observation
or comma-separated,
or a database program.
but not both simultaneously,
nor can it be
must be on only one line.
o The fir t line in the file can optionally contain the names of the variables. infile (fre_ format)--infile without a dictionary o The da
can be space-separated,
o Strings with embedded separat, d). o A singl _observation line.
tab-separated,
or comma-separated.
spaces or commas must be enclosed
in quotes (even if tab- or comma-
can be on more than one line or there can even be multiple observations
infix (fixe(_ format) o The da!
must be in fixed-column
format.
o A singl
observation
o infix
as simpler syntax than infile
can be on more than one line. (fixed format).
infile (fixe 1 format)--infile with a dictionary o The daa may be in fixed-column o A singl _ observation o infil_
format.
can be on more than one line.
(fixed format) has the most capabilities
74
for reading data.
per
infile-- Quicl_referencefor readingdata intoStata
75
I
Examples
I
l
> Example
topof exampl.raw i
1 0 0 0
0 0 I O
1 I 0
John Smith Paul Lin Jan Doe f Julie McDonald
m m f
endof exampl.raw-contains tab-separated data. The type command with the showtabs
option shows the tabs:
type eXampl.rau, slowtabs 1O1John Smithm OO.IPaulLin_T>m OIOJan Doe<3>f oO.Julie Mc[onaldf Z
It could be read in by • insheet a b c name gender using exampl
Example topof examp2.raw--
i
a,b, c,name, gender 1,0,I ,John Smith,m 0,0,I ,Pa_l Lin,m O,l,O,Jan Doe,f 0,0, Julie McDonald,:
!
endof examp2.rawcould be read in by
i
" insheet
using
examl,2
q
Example topof examp3.raw 1 0 0 0
0 0 I 0
i 1 0
"John Smith" m "Paul Lin" m "Jan Doe" f "Julie McDonald"
f
t
endof examp3.raw
contains tab-separated data _'ith strings in double quotes. ;
. type
examp3.raw, s]lowtabs
lO"John Sm th"m OO"Paul Li:"m O<:T>IO"JanDoe f OO."Julie M _Donald"f
It could be read in by i
• infile byte (a b c
I strl5 name strl gender hsing examp3
_
76
infile -- Quick reference for reading data into Stata
or
I
!
• zlsheet
a b c name
gender
using
examp3
!
Or
i_file
using
dict3
where the dictionary
dict3.dct
contains top of dict3.dct
/ infJle
dictionary
using
byte
a
byte
b
byte str15
c name
strl
gender
examp3
{
} end of dict3.dct
..
q
> Example top of examp4.raw 1 0 1 "John
Smith"
0 0 1 "Paul
Lin"
0 1
"Jan
Doe"
0 0 .
"3ulie
m
m
! !
f
McDonald"
f
end of examp4.rawcould be _ad in by • ii file
byte
• infile
using
(a b c) strl5
name
strl
gender
using
examp4
or dict4
i
l where
the _dictionary dict4,
dct
contains
i
-- top of dict4.dct
| inf_le
dictionary
using
byte
a
byte
b
byte strl5
c name
strl
gender
examp4
{
} end of dict4.dct
<3
> Example mp of examp5.raw I01_ John
Smith
O01z Paul
Lin
010J Jan Doe O0
Julie
McDonald end of examp5.raw
• i_fix
could be
a I b 2 c 3 str gender
:ad in by
4 str
name
5-19
using
examp5
i
infile-- Quickirdferencefor readingdata into Stata
77
or • imfix
}
using
dict5a
where d_ct5a.dct
contains -- topof dict5aAct--
infix didtionary
usinl a
str sir
examp5 1
b
2
c
3
gende: name
4 5-19
{
i
i
endof dict5a.dct--
or . i_file Using
dictSb
where dict5b.dctcontains ! !
topof dict5b.dct--_ infile dictionary
using
examp5
{
%If
I
byte
a
!
byte
b
_.If
i
byte strl
c gent er
Zlf Xis
strl5
name
%15s
} endof dict5b.dct i
> Example top of examp6.raw line There
I : a heading are a
total
of 4 lines
of heading.
The next line contains a useful heading: ---4+ .... I.... + .... 2 .... + .... 3.... + .... 4 .... +1
0
1
m
Jo_hn Smith
i
0
0
I
m
Paul Lin
i
0 0
i 0
0
f f
Jan Doe Julie McDonald
i i• • !
--
endofexamp6.raw
could be read in by . i_file using
where diCt6a.dct
dict6a
contains mpof dM6a.dct
i _
infi_le didtionary -firstline (5)
_ i
ex_mp6
{
byte byte _col_m_(17)
I i
usin_
-co!kunn(33)
%lf
byte strl
ender
strl5
ame
Y.15s
} endof dict6aAc_
q
11
_
78
_nfile
_
Ouick
_n_
_r
reading
d_ta
ir'_to
Stli_l_
or could bte read in by | • i_ix
5 first a I b 9 c 17 str gender 25 sir name 33-46 using examp6
or could @ read in by • infix using dict6b l where dict6b.dct contains ,_
top of dict6b.dct
infi_ dictionary using examp6 5 fifst a 1
l
b str str
{
9
c
17
gender name
25 33-46
} end ofdict6b.dct
> Example I a b _ gender name I 0 1
top of examp7.raw
John m I Smith
ooI Paul|Lin 010
't
oe
Jan O0
'4 Juli
McDonald
I
end of exampT.raw
could be --r_adin by • in_ile using dictTa
where dic}Ta.dctcontains
---
top of dict7a.dct
infi_ dictionary using examp7 { _firs_line (2) byte a byte b i _linel(2)
byte
c
_line (3)
strl str15
gender name _.15s
} end of dict7a.dct
Or, if you _,'anted to include variable labels: • fl._1 e using dict7b • in
infile -- Quick reference for reading data into Stata
where dictTb,
dct
79
contair mp of dict7b.dct
ififile dictionaryu-cLngexamp7{ __irstline (2) byte "Question 1" byte "question2" byte "Question3" _iine(2) str! mder "Genderof subject" _line (3) strl5
ime
_.15s
} end of dict7b.dct infix
could also read this data: • infix
2 first
3 lines
a i b 3 c 5 str gender
2:1 str name
3:1-15
using
examp7
or itcould be read in by
. :-infixusing
dict7 i
where dictZc.dct contair_s | iz[fixdictionaryusing examp7{ 2 ifirst I a
str str
I
I
b I
3
g der name
2:1 3 :1-15
top of dict7c.dct
end of dictTcAct or it could be read in by
where" i i
s ; i_fix dictionaryusing examp7 { 2 first a I b 3 c 5 / str
g_
1
str
n_
1-15
top of dict7d.dct
/ !
}
i
end of dict7d.dct
AlsoSee Complemental:
Background:
[R]
tit, [R] infile (fixed format),
[R]
_fix (fixed format),
[u]
4 Commands
[R] input,
to input data
[R] infile (free format), [R] insheet
Title _,_
I
infile _fixed format)|
Read ASCII (text) data in fixed format Iwith a dictionary I
]
Syntax infile
using
dfilename
[if exp] [in range]
[, _automatic
If dfilename
is specified
without
an extension,
.dot
is assumed.
;
lf filename2
is specified
without
an extension,
.raw
is assumed.
1
The synta
u_sing(fitename_)
clear
]
for a dictionary, a file created with an editor or word processor outside of Stata, is
[inf
.le]
top of dictionary
file
end of dictionary
file
dictionary [using filename] { * comments may be included freely _irecl(#) _firstlineof file (#) _lines (#) _line (#)
_newline[(') J
I
}
[_.pe]
varname
[:lblname]
['/.infmt]
["variable
labet"]
(you_data_might appearhere)
where ',in(mr is { 7,[#[.#]]{flgle} If using
ill,name /
is not specified,
If using ill, name is specified, extensiofi., raw is assumed.
] X[#]s }
the data are assumed
the data are assumed
to begin
on the line following
to be located
in filename.
the close brace.
If filename
is specified
without
an
Descriptk n infilc using reads data from a disk dataset that is not in Stata format, infile using does this by firs reading dfilename, called a dictionary, that describes the format of the data file, and then reads the ile containing the data. The dictionary is a file you create in an editor or word processor outside of Stata. The da
may be in the same file as the dictionary or in another file.
Anothe_ variation on infile omits the intermediate dictionary; see [R] infile (free format). This variation i_ easier to use. but will not read fixed-format files. On the other hand. although infile using will read free-format files, the variation is even better at it. An alternative to infile using for reading fixed-format files is infix; see [R] infix (fixed format). _nfix provides fewer features than infile using but is easier to use. I Stata his other commands for reading data. If you are not certain that infile are lookinl_ for, see [R] infile and [U] 24 Commands to input data.
80
i
using
is what you
6
•
i
infile (fixed format) -- Read ASCII (text)data in fixed format with a dictionary
81
Options automaticcauses Stata tc create value labels from the nonnumeric data it reads. It also automatically widens the display forn mt to fit the longest label.
i
using(filenamei) specific:s the name of a file containing the data. If using() is not specified, the data are assumed to fc low the dictionary in dfilename or, if the dictionary, specifies the name of Some other file, tha file is assumed to contain the data. If using(filenamei) is specified, fitename2 is used to ob din the data even if the dictionary itself says otherwise.
}! ; i
clear specifies that it is )kay for the new data tO replace what is currently in memory. To ensure that you do not lose solnething important, infi_le using will refuse to read new data if data are already in memory, cl_iar is one way you can tell infile using that it is okay. The other is to drop the data yourself _,_,typing drop _all before reading new data.
Dictionarydirective,, * marl_s comment lines. "_herever you wish to place a comment, begin the line with a *. Comments can !_appearmany times in the same dictionary. _lreci (#) is used only f_r reading da_asets that do not have end-of-litle delimiters (carriage return, line_'eed_or some coml:ination). Such files are often produced by mainframe computers and have bee n poorly translated from EBCDIC into ASCII. _lrecl() specifies the logical record !ength _lrecl()
[ i
requests thai infile
act as if a line ends every # characters.
_l_ecl() appears onl I once. and typically not at all, in a dictionara,. _firsttineoffile(#) _bbreviation ._.first())is also rarely specified. It states the line of the file where the data be_in. _:first() is not specified when the data follow the dictionary: Stata can!fi_ure that out for itself. _first () is instead specified when reading data from another file in _+hich the first line loes not contain data because of headers or other markers. F
_f_rst
() appears onl
once. and typically not at all, in a dictionary.
_line_. (#) states the nu: _ber of lines per observation in the file. Simple datasets typically have _li_aes(i). Large dat;lsets often have many lines (sometimes called records) per observation. _lines() is optional :yen when there is more ihan one line per observation because in:file can isometimes figure il out for itself. Still. if Alines(i) is not right for your data. it is best to spe_ifv the directive. _lines()
i
appears onl' once in a dictionary.
-line(#) tells infile tc .jump to line # of the observation. Distinguish _lines () from _line (). and consider a file with _lines (4). meaning four lines per observation. _line (2) says to go to the Second line of the observation. _line(4) says to go to the fourth line of the observation. You may.jump forward or b_ckward, infile does not care nor is there any inefficiency, in =,,oin_,_. forward to 21ine(a), reading: few variables, jumping back to _line(l), reading another variable, and jumping forward again to _line (3). It is not your responsib lity to ensure that, at the et_d of your dictionary, you are on _he last line of the !observation. infile knows how to get to |he next observation because it knows where you are iand it knows _lin._s(), the total number of lines per observation. _l_ne()
may appear, tnd typically does, many times in a dictionary.
f
I I I
II t
.................
I
.... ,
82 ...newlix goes _new to get
........
,
infile (fixed format) -- Read ASCII (text) data in fixed format with a dictionary e [(#) ] is an alternative to _line (). _newl ine (1), which may be abbreviated _newline, orward one line. _newline(2) goes forward two lines. We do not reconmlend the use of ine() because _line() is better. If you are currently on line 2 of an observation and want to line 6, you could type _newline (4), but your meaning is clearer if you type _line (6).
_new: ine () may appear many times in a dictionary. _colnmr (#) jumps to column # on the current line. You may jump forward or backward within a line. _ column() may appear many times in a dictionary. _skip (#_) jumps forward # columns on the current line. _skip _skil_ () may appear many times in a dictionary.
() is just an alternative to _column
().
[t)_e] va/_ame [: lblname] [%infint] ["variable label"] instructs inf i le to read a variable. The simplest form f this instruction is the variable name itself: varname. First _nderstand
that at all times infile
is on some column
of some line of an observation.
infi_e starts by being on column 1 of line 1, so pretend that is where we are. Given the simplest directiige 'varname, infile goes through the following logic: Is the.lcurrent column blank? If so, skip forward until there is a nonblank column (or until the end o_the line). If we just skipped all the way to the end of the line, store a missing value in varna/_e. If we skipped to a nonblank column, be_in collecting what is there until we come to a blankbolumn or the end of the line. That is the data for varname, Now set the current column |
to wherever we are. !
The l@ic is a bit more complicated. For instance, when skipping forward to find the data, infile might _ncounter a quote. If so, it then collects the characters for the data by skipping forward until it find_ the matching quote. If you specified a %infmt, in:file skips the skipping-forward step and simpl_ collects the specified number of characters. Nevertheless, the general logic is (optionally) skip, _oltect, and reset. ! I
Remarks i ,!
in_il using follows a two-step process using d_ cript and
i_
1. infil and
to read your data. You type something
like ingile
using reads the file descript.dct, which tells infile about the format of the data:
1
2. infil_
using
then reads the data according to the instructions
recorded
in descript.dct.
descrip_.dct ! (the file could be named anything) is called a dictionary and descript, a text file I _ou create with an editor or word processor outside of Stata. As forithe data themselves, does not aatter.
they can be in the same file as the dictionary
det is just
or in a different file. It
Readingfi ee-formatfiles 1
' ;, I
Another variation of infile for reading free-format files is described in [R] infile (free format). We will _efer to the variation as infile without a dictionary. The distinction between the two variations is in the treatment of line breaks, infile without a dictionary does not consider them significant, infile with a dictionary does.
infile(fixedformat) -- ReadASCII(tAx!)data in fixed formatwith a dictionary
83
A' line, also known a', a record, physical record, or physical line (as opposed to observations, or logical records, or logica lines), is a string of characters followed by the line terminator. If you were to type the file, a line is _,hatwould appear on y0ur screen if your screen were infinitely wide. Your screen would have to be infinitely wide so that th_erewould be no possibility that a single line could take more than one line 9f 3,our screen, thus fooling you into thinking there are multiple lines when I
i i
i
i.
i
!
i
i
therei is only one. A logical line, on the other hand, is a sequence of one or more physical lines that represents a singl+ observation of yoar data. infile with a dictionary does not willy-nilly go to new physical linesi it goes to a new lJne between obser_,ations and it goes to a new line when you tell it to, but that is all. infile without a dictionary, on the other hand, goes to a new line whenever it needs to, which can be right in tl-e middIe of an observation. Thus, consider the following little bit of data which, we will tell you, is for three variables: 54 193 2
How do you interpret th,_se data? H, re is one interpretation: There are three observations. The first is 5, 4. and missing. The second is 1, 9, and 3. The thirc is 2. missing, and missing. That is the interpretation that ±nfile with a dictionary makes. H,re is another interp etation: There are two observations. The first is 5, 4, and 1. The second is 9, 3, and 2. That is the interpretation that infile without a dictionary makes. W_aichis right? We would have to ask the person who entered these data. The question is. are the line t_reaks significant? )o they mean anything? It' the line breaks are significant, we use infile with a dictionary. If the line breaks are not significant, we use infile without a dictionary The other distinction _etween the two infiles is that infile with a dictionary does not process comma-separated-value ormat. If your data are c0mma-separated, see [R] infile (free format) or [R] Msheet.
Example Omside of Stata yout ave typed into the file -,-highway"dct information on the accident rate per million vehicle miles aldng a stretch of highwav_ the speed limit on that highway, and the number of access points (on-ram_s and off-ramps) per rnile, Your file contains [
' "
top ofhigh_ay.dct,,example]
infile dictionary i{ acc rate Ispdlimit acc_pts
} 4.58 55 4.6 2.86 6O 4.4 1.61 . 2.2 3.02 60 4.7
endof highway.dct,examplet This file can be read by yping infile data: •
using
infile using hil;hvay
infile dictionary 1{ acc_rate Ispdlimit acc_pts }(4observations r4ad) |
!
highway.
Stata displays the dictionary and reads the
1 ...........
lnfile
J ............. 84
| st . II !
.................. (fixed format) -- Read ASCII (text) data in fixed format with a dictionary
ace_rate
I. 2 I. 3_. 4_
. '_ _
4.58 2.86 1.61 3.02
spdlimit 55 60 60
acc_pts 4.6 4.4 2.2 4.7
.Z
(> Example We ca: include variable labels in a dictionary so that after we infile the data, the data will be fully labe ed. We could change highway, dot to read i
"
;
I : i I ;
.
top of highway.dct,
example 2
end of highway.dct,
example 2
inf: le dictionary { * T]is is a comment and will be ignored by Stata * Y,u might type the source of the data here. acc_rate "Acc. Rate/Million Miles"
} 4.5_ 2.8_ 1.6_ 3.0"-
spdlimit acc_pts
"Speed Limit (mph)" "Access Pts/Mile"
55 4.6 60 4.4 . 2.2 60 4.7
Now wheJb we type infile
using
highway,
Stata not only reads the data but labels the variables.
<1 I > Example
l
We caniindicate the variable types in the dictionary. For instance, if we wanted to store acc_rate as a doub..e and spdlimit as a byte, we could change highway.dct to read top of highway.dct,
example 3
infi .e dictionary { * Th s is a comment and will be ignored by Stata * dm Yo imight type the source of the data here. le acc_rate "Acc. Rate/Million Miles" byt@ !
spdlimit acc_pts
"Speed Limit (mph)" "Access Pts/Mile"
2.861 60 4.4 1.61 . 2.2 3.02
60 4.7
Since we c 3 not indicate the variable type for acc_pts, (or the typ, specified by the set type command).
end of highway.dct
example 3
it is given the default variable type float
<1
"
tnfile (fixed for at) -- Read ASCII (text) data in fixed format with a dictionary
85
Example i
By specifyingthe typ,is, we can read stringvariablesas well as numericvariables.For instance, +
topof emp.dct
iinfile dictionary '
• data on employe( str20 name
I
int
age sex
"Name" "Age" "Sex coded
0 male
I female"
} I
!"Lisa Gilmore"
25
!Branten 32 1 "Bill Ross"
27 0 end of" emp.dct
The stringscan be delimiLedby singleor doublequotesand quotesmay be omittedaltogetherif the string _:contains no blanks or other special characters,
q t [3Technical Note You may attach value abels to variables in the dictionary using the colon notation: _opof emp2.dct infile
dictionary-
data on name, strl6 name i
se: , and age "Name"
sex: sexlbl int age
"Sex" "Age"
} _Arthur Doyle" Malt 22 _Mary Hope" Female 37 #Guy Fawkes" Male _ 8 #Sherry
Crooks"
Fel ale 25 end of emp2.dct
i
If you _want the value labels to be created automatically, you must specify the automatic option on the infile command. Tl-ese data could be read by typing infile using person2, automatic assuming the dictionary alld data are stored in the file person2.dct.
i
J
}
I t
1
i
_"Example The data need not be n the same file as the dictionary. We might leave the highway data in highway.raw and write dictionary called highway.dct describing it: topof highway.dct,example4 infile
dictionary
u _ing highway
* This
dictionary
r._ads the file highway.raw.
* file
were
* read
"dictionary ace_rate
called
spdlimit
}
aee_pts
lighway.txt,
{ the first
If the
line would
1sing highway.txt" _cc. Rate/Million Miles!' Speed " ccess
Limit
(mph)"
Pts/Mile" --
end of highway.dcl,example 4
I
86 intrile(fixed format) -- Read ASCII (text) data in fixed format with a dictionary _, Example The fir_tlineoffile following rakv dataset:
() directive allows you to ignore lines at the top of the file. Consider the top of mydata.raw
The f( flowing data was entered by Marsha Martinez. Helen troy. id in( _me educ sex age 1024 5000 HS Male 28 1025 27000 C Female 24
It was checked by
end of mydata.raw
Your diction1 u'y might read top of mydata.dct infil_ dictionary using mydata { _first (4) int id "Identification Number" income "Annual income" str2 educ "Highest educ level" str6 sex byte age
} end of mydata.dct
q
1 i
> Example The _lir_e () and _lines () directives instruc! Stata how to read your data when there are multiple records per _Sbservation. You have the following in mydata2 .raw: ri
top of mvdata2.raw
id incpme educ sex age 1024 2_000 HS Male | 28 1025 2_7000 C Femalei 1035 2 000 HS Male 32 1036 25000 C Female 25
1
You can read this with a dictionary mydata2, reads the daia: • infi_e using mydata2, clear
end of mydata2.raw
dct, which we will just let Stata list as it simultaneously
i infile(fixed formal -- Read ASCII (text) data in fixed format with a dictionary
87
z
in_ile dictionary usiag mydata2 { _first(2) _lines(3) int id "Identific ttion Number" income "Annual in :ome" sir2 educ "Highes _line(2) sir6 sex _line(3)
* * * *
Begin reading on line 2 Each obbervatiOn takes 3 lines. Since __ine is not specified, Stata assumes that it is I.
educ level" * Go to line 2 of the observation. * (values for sex are located on line 2) * Go to llne 3 of the observation. * (values for age are located on line 3)
int age
} (4 bbservations read)
. list Ii 2! 3_
id 1024 1025 1035
inc(ime 251)00 27_i00 26_I00
4i
1036
25( 00
Now, here is the really good ii
could jus( as wdll have
educ _S C HS
sex Male Female Male
age 28 24 32
C
Female
25
art: We read these variables in order but that was not necessaD_. We
usedrhedictionary:
top of mydata2p.dct
inf_le dictionary using mydata2 { _first (2) _lines (3) _line (1)
int
id "Identification number" income "_ual income"
_line(3) _line(2)
sti int st_
educ age sex
"Highest educ level"
} end of mydata2p.dct
We would obtain the same re_ults--and just as quickly--the only difference being that our variables in the fin_ dataset would be n the order specified: id, income, educ, age, and sex. q
Technical!Note i.
You can use _newline tO specify where breaks occur, if you prefer: Z
........ i
i
!_
topof highway.dct,example5--
infile dictionary { acc_rate "Ac :. Rate/Million Miles" spdlimit "S )eed Limit (mph)"
>
_newline acc_pts
"Ac :essPts/Mile"
4.58 55 4.6 2.861 60 4.4 1.61. 2.2 3.02 i 60 4.7 end of highway.dct, example 5
The line th)at reads '1. 61 .' ould have been read 1.61 (without the period), and the results would have been unchanged. Since _tictionaries do not go to new lines automatically, a missing value is assumed for all values not foulnd in the record.
!
88 ---
i_file (fixed format) -- Read ASCII (text) data in fixed format with a dictionary =
]
Readingfied-format files Values _n formatted data are sometimes packed one against the other with no intervening For instande, the highway data might appear as I top of highway.raw,example 6
,,
:',
blanks.
4.58_54.6
2.86 04.4 1.61| 2.2 3.02604.7
":
end of highway.raw,example6 The first f_ur columns of each record represent the accident rate; the next two columns, the speed limit; and _he last three columns, the number of access points per mile. To read:: these data, you must specify the %infmt in the dictionary. Numeric Y,infints are denoted by a leadir_g percent sign (%) followed optionally by a string of the form w or w.d, where w and d stand for @o integers. The first integer, w, specifies the width of the format. The second integer, d, specifies ti_enumber of digits that are to follow the decimal point. Logic requires that d be less than or equal tqw. Finally, a character denoting the format type (f, g, or e) is appended. For example, %9.2f spe_zifies an f format that is nine characters wide and has two digits following the decimal point.
Numericformats The f f_rmat indicates that infile is to attempt to read the data as a number. When you do not specify th_%infmt in the dictionary, infile assumes the %f format. The missing width w means that infille is to attempt to read the data in free format. At the _mrt of each observation, to 1, indic moves the occurrence is left at tl
infile
reads a record into its buffer and sets a column pointer
ating that it is currently on the first column. When infile processes a %f format, it "olumn pointer forward through white space. It then collects the characters up to the next of white space and attempts to interpret those characters as a number. The column pointer e first occurrence of white space following those characters, If the next variable is also
free forma I, the logic repeats. When ypu space. Instead, the result @ a is, on the first
explicitly specify the field width w, as in %wf, infile does not skip leading white it collects the next w characters starting at the column pointer and attempts to interpret number. The column pointer is left at the old value of the column pointer plus w, that character following the specified field.
Example If the d tta above are stored in highway, the data:
raw, you could create the following
infi e dictionary using highway { acc_rate Y,4f "Acc. Rate/Million spdlimit acc_pts
dictionary to read
top of highway.dct,
example 6
end of highwa?.dct,
example 6
Miles"
Y,2f "Speed Limit (mph)" Y,3f "Access Pts/Mile
} 1
Wh:ncolu_s you explicitly field width, not skip intervening and characters. The first are usedindicate for the the variable ace_rate,infile the does next two for spdlim-it, the last three for acc_pts.
<1
i
•
Q
|
....
infile (fixed format,l-- Read ASCII (text) data in fixed format with a dictionary
89
The d specification in the i'/,w.df indicates the number of implied decimal places in the data. For Technica_ Note instance, the string 212 read tin a _3.2f
format repre_ems the number 2.12. You should not specifv
d unless _¢ourdata have ele@nts of this form. The w alone is sufficient to tell ±nfile
how to read
i '_ i
data in which the decimal P_]int is explicitly indicated" When iyou specify d, it is taken on13 as a su_,_estion. If the decimal point is explicitly indicated in the data, ihat decimal point a_wa3s m errides the d specification. Decimal points are also not implied
t 1 I
if the data contain an E, e, I], or d, indicating scientific notation. Fields i!are right-just, fled Otefore lmptymo dec,mal points. Thus, as 0 2 by the _3. If format.
I
2
,
2 . and
2 are all read
a TechnicalNote The g and e formats are the same as the f format. YoUCanspecify any of these letters interchangeably. The letters g and e are inch ided as a convenience to those familiar with Fortran. In Fortran. the e format i_icates scientific n, .ration. For example, the number 250 could be indicated as 2,5E+02 or 2.5D402. Fortran prograr Imers would refer to this as an ET. 5 format, and in Stata. this format would _ indicated as 7'7.5. _. In Stata. however, you need specify only the field width w. so you could react this number usin 7'7f, 7,7g. or '/,7e.
i ! i I
The gi format is really a :or_ran output format that indicates a freer format than f. In Stata. the two formtats are identical. !
i ! i i
i !
Throughout this section,
a
Technical
ou may freely substitute the g or e formats for the f format.
Note
Be careful to distinguish b__tween'/,tints and '/,infints: '/,tints are also known as display formats--_hey describe how a variable is :o look when it is outputted; see [u] 15.5 Formats: controlling how data are!displayed. 7,ilTfi,ts are also known as input formats--they describe how a variable looks when it _s inputted. For instance, there is an output date format 7,d, but there is no corresponding input format. (See [U] 27 C, remands for dealing Mth dates for recommendations on how to read dates.) Fbr the other formats we have attempted to make the input and output definitions as similar as possible. Thus. we includ g. e. and f }',i_!fints,even though they all mean the same thing, since g, e, and f are also '/,tints.
String formats The s format is for read ng strings. The syntax is gu:s where the w is optional. If you do not specify the field width, your strings must be enclosed in quotes (single or double) or the_ must not contain Nny characters other than letters, numbers, and '_. This may surprise you, ,ut the s format can be: used for reading numeric variables and the f tbrmat c_n be used for rea ring string variables! When you specify the field width u, in the '/,'u,f format, all embedded blank., in the field are removed before the result is interpreted: They are not removed by the Xws /ormat
90
' ,I
_nfile(fixed format) -- Read ASCII (text) data in fixed format with a dictionary
For instance, the _3f format would read '- 2', '-2 ', or ' -2' as the number -2. The _3s format would notl be able to read '- 2' as a number, since the sign is separated from the digit, but it could read ' -2" or '-2 '. The %wf format removes blanks; datasets written by some FORTRAN programs separate the sign from the number. There ge, however, some slde-effects of this practice. The stnng 2 2 will be read as 22 by %3f format. Most FORTRAN compilers would read this number as 202. The %3s format would issue a wamingland store a missing value. Now c6nsider reading the string 'a b' into a string variable. Using a Z3s format, it will store as it appears t a b. Using a Y,3f format, however, it wilt be stored as ab--the middle blank will be removed. I Examples using the Xs format are provided line numbers.
below, right after we discuss specifying column and
|
Specifying column and line numbers _colu_() jumps to the specified column. For instance, the documentation of some dataset indicates that the variable age is recorded as a 2-digit number in column 47. You could read this by coding
I
_column(47)
age Y.2f
After this,i you are now at column recording _ex as 0 or 1,
you
[
_column(47)
49, so if immediately
_column(47) _column(49)
I
were a 1-digit number
age Y.2f sex Y.lf
could instead code
age Y.2f sex Y, lf
It makes np difference. If at column 50 were a 1-digit code for race, skip readirlg the sex code, you could code _column(47)
age
could code
or, if you tvanted to be explicit about it, you I I
following
and you wanted to read it but
age Y,2f
column(50) race Y, lf
You couldlequivalently
skip forward using _skip ():
1
_colunm(47)
age _2f
I
_skip(1)
race Zlf
One advar_tage of column() over _skip is that it lets you jump forward or backward in a record. |
If you war,ted to read race
and then age, you could code
_column(50) race Y, lf _column(47) age Y,2f
If the d tta you are reading have multiple lines per observation (sometimes said as multiple records per observ _tion), tell infile how many lines per record there are using _lines (): _ lines (4)
_lines () appears only ()nee in a dictionary. Good style says it should be placed near the top of the dictionary, but Stata does not care.
;infile (fixed format -- Read ASCII (text) d_ta in fixed format with a dictionary
91
When you want to go to a particular line, includb the _line() directive. In our example, let's assume race, sex, and age are recorded on the second line of each observation: _lines(4) _line (2) _column(47)
a_e Y,2f
_column(50)
Tce
}'.If
Let's assume id is recordedlon line 1. 1 lines (4)
|
_line(l) i I
i
_column(I) _line(2)
d
Y,4f
Y,tf _co1_(47) ace goX2_ _column(50)
_line() works like _colu as well be read by
m() in that you can jump forward or backwardl so these data could just
_lines(4) _line(2) _column(47) _column(S0)
_ge %2f ,race %If
_line (I) _colnmn(1)
_d
7,4f
Remember that this dataset aas 4 lines per observatibn and yet we have never referred to line (3) or line(4). That is okay./Jso note that, at the endof our dictionary, we are on line t. not 4. That is okay, loo, infile will stll get to the next observation correctly.
E3TechnicalNote
i l.
Anotl!er way to move bet' _een records'is _newline ().._newline () is to _line () as _skip () is Io _column(), which is to say, ..mewline () can only go forward. There is one difference: _skip() has its uses; ._.newline () L' useful only for backward capability with older versions of Stata. _skip()has its uses bec Lusesometimes one thinks in terms of columns and sometimes one thinks in tern'ts:of widths. Some d Ia documentation might very well include the sentence "At column 54 are recorded the answers to the 25 questions, one column alloted to each." If we want to read the answers io questions 1 and 5, it would indeed be na_tutalto code _column(54)
tl
_.If
_skip(3)
is %1_ [
i
Nobody has ever read data _ocumentation with the siatement, "Demographics are recorded on record 2and, 2 records after that, _re the income " values. " The ' " documentanon " would instead " " sa3,' " Record ,_ '_ contains the demographic ir formation and record 4, irJcome." The _newline() way of thinking Is based on what is convenien for the computer which does, after all, have to eject a certain number of records. That, however, no reason for making yot_ think that way.
i
Before that thought occ fred to us, Stata users specified _mewline() to go forward records. They stiil can, so their old [ictionaries will work. When you use _neTaline() and do not speci_ _ -lines(), it is your respoasibility to eject the right number of records so that, at the end of the dictionary, you are on the last record. In'this mode. when Stata re-executes the dictionary to process
i
the next iobservation, it doe: forward one record.
I
!
[
...................
B
I
...........
92
infile (fixed format) -- Read ASCII (text) data in fixed format with a dictionary
'i':
Example of reading fixed-format files
:
> Example In thisIexample, "i each observation
i
Joh_ Dunbar 101 111111
P
i.
Sam! g. Newey,
i'
OlObOOOOO
!:
occupies two lines. The first two observations
Jr.
10001
101
North
10002
15663
42nd
in the dataset are
Street
Roustabout
Boulevard
The first )bservation tells us that the name of the respondent is John Dunbar; his id is 10001; his address i., 101 North 42nd Street; and that his answers to questions 1 through 10 are yes, no, yes, no, yes, _s, yes, yes, yes, and yes. The sei:ond observation tells us the name of the respondent is Sam K. Newey, Jr.; his id is 10002; his addre_ is 15663 Roustabout Boulevard; and that his answers to questions l through 10 were no, yes, no, _s, no, no, no, no, no, and no. (Probably John and Sam are not best friends.) In ord r to see the layout within the file, we can temporarily the appro!briate columns: ....
+ ....
Jol_
I ....
+ ....
2....
+ ....
3 ....
Dunbar
add two rulers to help our eyes see
+ ....
4 ....
+ ....
5 ....
+ ....
i0001
I01
North
42nd
Street
6 ....
10002
15663
+ ....
7 ....
+ ....
8
+ ....
7 ....
+ ....
8
lOlq 111111 Sam 010 ....
K. Newey, 000000 + ....
1....
Jr. + ....
2 ....
+ ....
3 ....
+ ....
4 ....
Roustabout + ....
5 ....
Boulevard + ....
6 ....
Each observation in the data appears in two physical lines within our text file. We had to check in our editorlto be sure that there really were newline characters (i.e., "hard returns") after the address. This is irr_ortant because some programs will wrap output for you and a single line may appear as many line_. The two seemingly identical files will differ in that one has a hard return and the other has a soft ireturn added only for display purposes. In our tidata, the name occupies columns 1 through 32; a person identifier occupies columns 33 through 37; and the address occupies columns 40 through 80. Our worksheet revealed that the widest address erred
in column 80.
The teat file containing these data is called fname, txt.
Our dictionary
file looks like
t
infi t e dictionary , I * Example * th_n
one
using
reading line.
fname.txt
in data The
where
next
line
{
top of fname.dct
observations
extend
tells
there
infile
across are
more
2 lines/obs:
_lin ,s(2)
,
_col mm(33) _ski _(2)
sir50
name
Z32s
"Name
long str50
id addr
Y,Sf Y,41s
"Person id" "Address"
of respondent"
_lin :(2) _col mm(1)
byte
ql
ZIg
"Question
!"
I
b}_te
q2
Y,lf
"Question
2"
i If
1
byte byte byte
q3 q4 q5
7,1f Y,lf Zlf
"Question "Question "Question
3" 4" 5"
byte
q6
Y,lf
"Question
6"
, I}
I"
7
infile (fixed form ) -- Read ASCII (text) data in fixed format with a dictionary byte
q8
%if
"Durst ion 8"
byte
q9
%If
"Du_stion
9"
byte
qlO
%If
"Question
I0"
93
} end of fname.dct
i
Up t6 five pieces of information may be supplied in the dictional3, for each variable: the location of the data, the storage tylie of the variable, the name of the variable, the input format, and the variable!label. !
i ! !
Thusl the str50 line sa_s that the first, variable, is to be given a.,, storage type of str50, should be called name, and have the '_ariable label Name of respondent. The %32s is the input format--this tells Stata how to read the _lata. The s tells Stata not to remove any embedded blanks: the 32 tells
li
Stata to go across 32 colur_ns when reading the data. The next line says that the second -,ariable- is to be assigned a storage type of long, named id, and labeled "Person id". Stata should start reading the information for this variable in column 33 The f tells Stata to remove any embedded blanks, and the 5 says to read across 5 columns.
I i • i i
The third variable is to De given a storage type of str50, called add.r, and labeled "Address". The _skip(2) directs Stat_ to skip 2 columns bef6re beginning to read the data for this variable, and the g4J.s instructs Stat_ to read across 41 colurnng and to not remove embedded blanks. line::(2)
instructs Stata o go to line 2 of the observation.
The remainder of the dzta is 0/1 coded--the answers to the questions. It would be convenient if we could use a shorthan to specify this portion of the dictionary, but we must supply explicit directives q
'3TechnicalNote i ! i
ii ! !
i
i
In the preceding exampl._, there were two pieces of information about location: where the data begin fo_ each variable (the _column(), _skip(), ,line ()) and how many columns the data span (the %32s, %5f, %41s, %lf). In our dictionary, some of this information was redundant. After reading name, Siata had finished with 32 columns of information. Unless instructed otherwise, Stata would _roceed to the next column--column 33--to begin reading information about id. The _column (33) was unnecessary. The _skip(2) was not.• however, unnecessary. Stata had read 37 columns of information and was ready to look at columl 38. Although the address information does not begin until column 40, columns "_8and 39 contain )lanks. Since these are leading blanks, instead of embedded blanks. $tata would jtist ignore them. Th .'re is no problem so far. The problem is with the %4is. If Stata begins reading _he address informa ion from column 38 and reads 41 columns, Stata would stop reading in column 78 (78 - 41 + 1 = 38), and the widest add?ess ends in column 80. We could have omitted the -skip(2) if we had specified an input format of X43s. The __l±ne(2) was necessary although we could have gotten to the second line by coding --newlineinstead. The _column(1)
could _ave been omitted. Afte_ the _line(),
Stata begins in column 1
See the following examp e for a dataset where both pieces of location infomaation are required.
r t
! mt.e (rlxeamrmatj -- Heaa ASCII (text) data in fixed format with a dictionary
7
D Example! The llowing file contains six variables in a variety of formats. Note that in the dictionary we read the _,ariables fifth and sixth out of order by forcing the column pointer. _
'r
topofexample.dct
in ile dictionary { -_I i double i ! i:
i .skip(2) ,column(21) ,_column(18)
i'_ _,
first second third
str4
%3f Zi.lf %6f
fourth %4s sixth 7,4.If fifth Y, if
1.2 L25.Te+252abcd 1 .232 1.3:35,7 52efgh2 5 1.4.457 52abcd 3 100. 1.5 155.TD+252efgh04 1.7 16 6 .57 52abed 5 t.71 [
end of example.dot
l
Assumin_| the above is stored in a file called example, dct, it can be infiled and listed by typing • i_file using example infile dictionary { i ! double i.skip(2)
str4
first second third fourth
7,Sf 7,2.if 7,6f
7,4s
sixth 7,4. If fifth _,2f
i.column(21) i_column(18)
} (5 _bservations
read)
list first i.2
second i.2
third 570
fourth abcd
sixth .232
fifth 1
2;{ 3 .i
1.3 I,4
1.3 I.4
5.7 57
efgh abcd
.5 i00
2 3
4.i 5.i
i.5 16.
i.5 1.6
570 .57
efgh abcd
I.7 1.71
4 5
1.
q {
Reading fi_(ed-block files
u Technical_ote The _l_'ecl (#) directive is used for reading datasets that do not have end,of-line delimiters (carriage return, lindfeed, or some combination): Such datasets are typical of IBM mainframes--where they are known as fixed block or FB. The word LRECLis IBM-mainframe jargon for logical record length. Fixed'block datasets are datasets where each # characters are to be interpreted as a record. For instance, @nsider the data 2 3 63
infile (fixed fo mat) -- Read ASCII (te_) data in fixed format with a dictionary'
95
In fixed-block format, tlq.'se data might be recorded top of mydata.ibm 1
212 423 63 end of mvdata.ibm
!
and you wouldbe told, m the side, thatthe LRECL is 4. If you then pass along that informationto inside,
it will be able
read the data: top of mydata.dct
infile dictionary using mydata.ibm { _Irecl(4)] int i_ int
}
a_e end of mydata.dct
When you do not spe( iCythe _lre¢l(#)
directive, in:file
i
assumes that each line ends with the
standardASCII delimiter whichcan be linefeed0r carriagereturn or linefeedfollowedby carriage return or carriage return 9llowed by finefeed). When you do specify _1reel data ih blocks of # char_ :ters and then acts as if that is a line.
i
(#), infile
reads the
A i:ornmon mistake ir processing fixed-block datasets is to be incorrect about the LRECL value, for instance, thinking the I.RECLis I60 when it is really 80. To understand what can happen, pretend we thought the LRECLin )ur data was 6 rather than 4, Taking the characters in groups of 6, the data
i
appearas 212
i
423 63 Stata has no way of verify ng that you have specifi_ the correct LRECLso, if the data appear incorrect,
verifyiyouhave the corre :t number. ThemaximumLRECLnfile allowsis 18,998withStatafor Unix,7,998with StataforWindows, and 3.998with Statafor /lacintosh.
References Gleason:, J. R. 1998. dm54: C tpturing comments from data dictionaries. in Siata Technical Bulletin Reprints, vol. 7. pp. 55-57. f
Stata Technical Bulletin 42: 3-4.
Gould, "iV,W. 1992. dml0: Inf ling data: Automatic dictionary' creation. Stata Technical Bulletin 9: 4-8. Stata Technical Bulletin Re_rints, vol. 2, pp. 28-34. Nash. J. D. 1994. dml9:
Me
4
ng raw data and dictionary, files. Stata Technical Bulletin 20: 3-5.
Technical Bulletin Reprints 1 vol. 4, pp. 22-25.
| AlsoSee Compl_mentary:
[R] utfile, [R] outsheet. [R] save
Related:
[R] afix (fixed format)
Background:
[u] ;4 Commands to input data, [R] afile
Reprinted
Reprinted in
Reprinted
in Stata
....................
i
Title '
...................
i
I infile ]i free format)-
Read unformatted ASCII (text)data [
i
]
Z'
Syntax ingil_
!i
]
varlist ['skip[(#)]
[varlist [_skip[(#)].
,.]I]
using filename [if exp] [in range]
If filename is specified without an extension, .raw is assumed
Descriptiln i
infil_reads into memory a disk dataset that is not in Stata format. Here _x!ediscuss using infile to read free-format data, meaning datasets where the knowledge of the forr_atting information is not necessary to make sense of them. Another variation on infile allows rea_ing fixed-format data; see [R] intile (fixed format). Yet another alternative is insheet, which is dasier to use if your data are tab- or comma, separated and contains one observation per line. Statalhas other commands for reading data, too. If you are not certain that infile is what you are lookin_ for, see IN] infile and [U] 24 Commands to input data, After tl_edata are read into Stata, the data can be saved as a Stata-format dataset; see [R] save.
Options
i
automati_causes Stata to create value labels from the nonnumeric data it reads. It also automatically widens the display format to fit the longest label. yvariable (#) specifies that the external data file is organized by variables rather than by observations. All the bbservations on the first variable appear, followed by all the observations on the second vanable_/ and so on. Time-series datasets sometimes come in this format 1
clear
spe}ifies that it is okay for the new data to replace what is currently in memory. To ensure
that youido not lose something important, infile will refuse to read new data if data are already in mem;ry, clear is one wa} you can tell infile that it is okay The other is to drop the data yourself_by typing drop _all before reading new data. i
Remarks infile_or,at least, the infilefeatures discussed here value forrn_tt.
reads data in free or comma-separated-
Remarkl are presented under the headings I I , ! t t
i 1
I
Reading free format data Reading comma-separated data Specifying variable types Reading string variables Skipping variables Skipping observations Reading time-series data
96
i I
infile (free formal -,--Read unformattedASCII (text) data
Readi
97
free format cmta
In ifree format, data • e separated by one or more white-space characters. White-space characters are blanks, tabs, and ne vlines (carriage return, linefeed, or carriage-retum_inefeed combinations). Thus. a single observatic may span any number:of lines. Numeric missing valu
Example
are indicated by single periods ('.').
J
In the file highway, r_w, you have information oft the accident rate per million vehicle miles along a stretch of highway, thelsoeed limit on that highway, and the number of access points (on-ramps and off-ramps) per mile.---Ifour file contains top of highway:raw, example1 _.58 55 4.6 2.86 60 4.4 i.61. 2.2
3.026o
4.7' endof highway•raw, example1 !
You can read these data by typing i infile ace_rate _ pdlimit {4 observations re_d)
acc_pts
using
highway
list I. 2. 3. 4.
acc_rate 4.58 2.86 1.61 3.02
s dlimit 55 60 60
acc.pts 4,6 4,4 2.2 4.7
Note that the spacing of tl e numbers in the original file is irrelevant.
[3TechniCalNote It isl not necessary that missing values be indicated by a single period. The third observation on the speed limit is missing in the previous example• The raw data file indicates this by recording a single period. Let's assume that instead the missing value was indicated by the word unknown. Thus, the raw data file appears a:
1
---
_ 4_58 55 4.6 2i86 60 4.4 1161 unknown 3 i02 60
top ofhighwayyaw,example22.2
4i7 endof highway.raw, example2 :
Here is the result of infilin
these data:
• _lnfite ace_rate s] llimit acc_pts using h_gh_ay ""/nk_ows" cannot be read as a number for spdlimit[3] (_ observations read
![
l
[1
98
iqfile (free format) m Read unformatted ASCII (text) data
infile wa_ed us that it did not know what to make of the word unknown, stored a missing, and then continhed to read the rest of the dataset. Thus, aside from the warning message, results are unchanged. ! Since not all packages indicate missing data in the same way, this feature can be useful when reading dat_ created by them. Whenever infile sees something it does not understand, it warns you, record_ a missing, and continues. If, on the other hand, the missing value were recorded not as unknown Nit as. say, 99. Stata would have no difficulty reading the number, but it would also store 99 rather than r_ssing. To convert such coded missing values to true missing values, see [R] mvencode.
i' .
[]
]
t
Reading comma-separated data In co_m_a-separated-value are are separated by either commas. You may intermix separated_'llue and free format.format, Missingdatavalues indicated bv single periods or by commamultiple commas which serve as place holders, or both. As with free format, a single observation may span any numbed of input lines.
Example We can imodify the format of highway, raw used in the previous example without affecting infile's ability to read it. The dataset can be read with the same command and the results would be the samd if the file instead contains 1
--
--
top of highway,raw. example 3
2.86, 4.58,1560,4.4 4.6 1.61, 12.2 3.02,_0 4.7 end of highway.raw,
example 3
<1
Specifying ,ariable types The variable names you type following the word infile are new variables. variable is
The syntax for a new
[type] new_varname[Llabel_name] A full discuJ;sion of this syntax can be found in [U] 14.4 varlists. As a quick review, new variables are, by defaldt, of type :float. This default can be overridden by preceding the variable name with a storage ty ,e (byte, int, long, float, double, or str#) or by using the set type command. A list of varia: _les placed in parentheses will be given the same type. For example. double_
_rst_var second_var...
causes first, var second_var _:
Ii
Iast_var)
... tast_var to all be of type double.
There is _lso a shorthand syntax for variable names with numeric suffixes. The varlist varl-var4 is equivalent to specifying varl vat2 var3 vat4.
I
,
infile (free format) -- Read unformattedASCII (text) data
99
> Example In!the highway exam _le, we could infile i
the data acc__rate,
spdtimit,
and acc_pts
and
force the variable spdli:Litto be of type int by typing •
infile
acc_rate
i(4 observations
int spdlimit
ace_pts
uaing
highway,
clear
r_ad)
We could force all vafia]:les to be of type double by typing • infile double(acc_rate (4 observations read)
spdlimit
aec_pts)
using
highway,
clear
We could call the three _ariables vl, v2, and v3 and make them all doubles bv typing i.
infile
double(vt-v3)
!(4 observations
using
highway,
clear
read)
q
Reading string variables B} explicitly specifyir g the types, we can read string variables as well as numeric variables.
Example Typing infile
str2(
; "Sherri Holliday" !Branton 32 1 "Bill
Ross"
name age sex using
myfile
would read top of myfile,raw
25 1
27,0
topof myfile.raw or even topof myfile.raw,variation2 "Sherri l,'Bill
golliday" 25,1 Ross', 27,
32
"Branton"
end of myfile.raw,
i
variation
2
Note ihat the spacing is i Televant and either single or double quotes may be used to delimit strings. The quotes_do not count when calculating the leng|h of strings, Quotes may be omitted altogether if the 'string contains no )lanks or other special characters (anything other than letters, numbers, or undergcores). Typing • infile str20 nam_ age sex using (3 observations re _d)
makes
name
a str20
an(
age
and
sex
myfile
floats.
• infile sir20 nam_ age int sex using (3 observations re Ld)
We
might have typed
myrtle
tomake sex an int or • infile striO nam._ int(age (3 observations re _d)
to make both age and sex ints.
seX) using
myfile
d
! ;_
100
infile (free format) -- Read unformatted ASCII (text) data
13Technical Note infile ican also handle nonnumeric data by using value labels. We will briefly review value labels, but you should see [U] 15.6.3 Value labels for a complete description. A value hbel
is a mapping from the set of integers to words. For instance, if you had a variable
J
called sex in your data that represented the sex of the individual, you might code 0 for mate and 1 for female. Yofi could then just remember that every time you see a value of 0 for sex, that observation refers to a male, whereas 1 refers to a female. Even better, you could inform State that 0 represents males and 1 represents lab_l
define
sexfmt
0 "Male"
females by typing
1 "Female"
Then you must tell State that this coding scheme is to be associated with the variable sex. This is typically ddne by typing • lab_l
values
sex
sex,mr
Thereafter, State will print Male rather than 0 and Female rather than 1 for this variable. State is :unique in that it has the ability to turn a value label around. Not only can it go from numeric c@es to words like "Male" and "Female", it can go from the words to the numeric code. We tell infite which value label goes with which variable by placing a colon (:) after the variable name and ts{ping the name of the value label. Before we do that, we use the label to inform S_tataof the coding.
define
command
Let's assume that we wish to infile a dataset containing the words Male and Female and that we wish to store numeric codes rather than the strings themselves. This will result in considerable data compression, especially if we store the numeric code as a byte. We have a dataset named persons .raw that contains name, sex, and age: top of persons.raw "ArthUrDoyle"Male 22 "Mary! Hope"Female37 "GuyFawkes"Male48 "Carrke
House"
Female
25 end of persons.raw
Here is hoW we read and encode it at the same time: label inf_le
define str16
sexfmt name
(4 observatlons
0 "Male" sex:sexfmt
i "Female" age using
persons
read)
list
i.
name
sex
age
Doyle
Male
22
Hope
Female
37
Guy Fawkes Carrie House
Male Female
48 25
Arthur
2.
Mary
3. 4.
The strl61in
the infile
command applies
only to the name variable: sex is a numeric variable.
as we can Orove by 1
lis_,
1. 2. 3. 4.
nolabel name
sex
age
Doyle
0
22
Mary Hope Guy Fawkes
1 0
37 48
1
25
Arthur
Carrie
House
_1
!
Int.e (_
tormm)-- -eaaumormmmaA:su. _jext)ama
,u,
D Technidal Note
i l
When infile is direct_d to use a value label arid it finds an entry in the file that does not match any ofithe codings record ;d in the label, it prints a warning message and stores missing for the observation. By specifyin_ the automatic option,you can instead have infileautomatically add entries to the value la _el, new Say!you have a dataset containing three variables. The first, region of the countr7, is a character string; the remaining two eariables, which we will just call varl and vat2, contain numbers. You have stored the data in a le called geog. raw: top of geog.raw
l
'_NE" _NCntrl"
31.23 29.52
South West
29.62 28.28
- -
87.78 98.92 114.69 218.92
.E
17.5o
44.3a
/_Cntrl
22.51
55.21 end of geog.raw
The easiest way to read tlris dataset is . infile str6 regica varl vat2 using geog
making region a string affable. You do not want to do this, however, because you are practicing for reaNng a dataset like his containing 20,000 observations. If region were numerically encoded and stored as a byte, th _.rewould be a 5-byte _aring per observation, reducing the size of the data by 100,000 bytes. Y( also do not want to bother with first creating the value label. Using the automatic option, infi: e creates the value label automatically as it encounters new regions. infile byte regi( a:regfmt varl vat2 usiflggeog, automatic (6 observations re_5) , list 1. 2. 3. 4. 5. 6,
vat1 31.23 29.52 29.62 28.28 17.5 22.51
region NE NCntrl South West NE NCntrl
vat2 87.78 98.92 114.69 218.92 44.33 55.21
±nfi_e automatically ere _tedand defined a new value label cal/ed regfmt. We can use tbe label list izomrnandto view i is contents: • label list regfmt :
regfm I 2 3 4
NE NCn ;rl Sou,;h Wes
It is not necessary that he value label be undefi_edprior to the use of infile with the automatic option. If the value label regfmt had been previOu._]ydefined as ;. label define reg Ymt 2 "West" i
the result of labellistafter the in_ilewould have been reEfmr : 2 3 4 5
West NE NCntrl South
•
.............
i
......
The automatic option is so convenient that you may see no reason for not using it. Here is one. 102 iinfile(free format)-- Read unformattedASCII(text) data Suppose _ou had a dataset containing, among other things, an individual's sex. You know that the sex variat_le is supposed to be coded male and female. If you read the data using the automatic 1 option and if one of the records contains fmlae, infile will blindly create a third sex rather than print a wdrning. 1 iIi Cl :
i t
.
Skippingariables Specifying _skip instead of a variable name directs infile to ignore the variable in that location. This feature makes it possible to extract manageable subsets from large disk datasets. A number of contiguou_ variables can be skipped by specifying _skip(#) Ignore. {
where # is the number of variables to
> Example In •the. _ighway example that started this section, the data file contained three variables: acc...xate, ] spcllamt; and acc_pts. You can read just the first two variables by typing t
• in_ile ace_rate
spdlimit _skip using highway
You can r_ad the first and last variables by typing in_ile ace_rate _skip acc_pts using highway, clear
You can r_ad just the first variable by typing :
• in_ile ace_rate _skip(2)using highway, clear {
ma_ be specified more than once. If you had a dataset containing four variables, say a, b, c, and d, _nd you wanted to read just a and c, you could type infile a _skip c _skip using filename, i _
,,
_skip
i I
1
Skipping observations Subsets! of observations can be extracted by specifying if exp, which also makes it possible to extract manageable subsets from large disk datasets. Do not, however, use the _variable __Nin exp. Use the ifirange modifier to refer to observation numbers within the disk dataset.
•
> Example
'
i
.Again r_ferring to the highway example, if you type • in] ile ace_rate spdlimit (2 oiservationsread)
ace_pts
iI
ace_rate>3
only obser 1 'ations for which ace_rate is greater than 3 will be infiled. You can type •
iniile
(30_
'
ace_rate
servations
to read onl_ the second,
spdlimit
acc_pts
in
2/4,
clear
read)
third,
and fourth
observations.
q }
I
infile (free format)-- Read unformattedASCII (text) data
Reading time-series ! ! ! i
103
clata
If you are dealing wilh time-series data, you may receive datasets organized by variables rather than by obser_'ations. All the observations on the first variable appear, followed by all the observations on the second variable, _nd so on. The byvariable(#)option specifies that the external data file is organized in this way. You specify the number of obseWations in the parentheses, since infile needs to know that numter in order to read the data properly. Alternatively, you can mark the end of one variable's data an:l the beginning of anottier's by placing a semicolon (';') in the raw data file. You may then specif a number larger than the number of observations in the dataset and leave it to 4nfile to determin, the actual number of observations. This method can also be used to read unbalanced data.
> Example YoU have time-series data on four years recorded in the file time.raw. information on year, amount, and cost and is organized by variable:
The dataset contains
top of time.raw i1980
1981 17 25
14 120
135
1982
198
30 150
180 end of time.raw
I
You can read these data _y typing :. infi!e year amot at cost using (4 observations re_d)
time,
byvariable(4)
list
I.
year 1980
amount 14
cost 120
2. 3.
1981
17
135
1982
25
4.
150
1983
30
180
If the ldata instead contai_ed semicolons marking the end of each series and had no information for amoum in 1983, the raw data might appear as 1980
I981
1982
14 17 25 ; i20 135 150 180
;
1983
; i
i
You could read these datI by typing • infile year amount (4 observations re_d) t . list
cost using
time,
I amount 14
cost 120
t.
year 1980
2.
1981
17
135
3.
1982
25
150
4.
1983
byvariable(lO0)
180
4
104
_
i:
infile (free format) -- Read unformatted ASCII (text) data
Also See! Complementary:
JR] outfile, JR] outsheet, [R] save
Related:
JR] infile (fixed format), JR] input, JR] insheet
Backgrodnd:
[U] 24 Commands to input data, [R] infile
[,n,x,,,.e0,oroa,,Re.d A=.extinxod fo nat t
_
i,,
i
i
i i
il
i
,
Syntax
in!fix specification sing filename [if exp] [in range] [, clear i i wherespecificationis # _irstlineoflile # lines #:
/ E
[byte lint Ifloat ilong ]double I str ] varlist and dfilename,
if it exists
[#-]#[-#]
contains
t
[
t
_
top of dictionary file --
I
infix dictionary Lsingfitename] { * comment, _recededby asterisk may appear freely specificafion_ _(yourdata might appear 5ere) end of dictionary file ......
4
If dfile_ame is specified wit
tan ex)ensiom .dot is assumed.
If fileng_me2or filename is specified without an extension, .rawis assumed. In the first svntax, if usingJ_lename 2 is not specified onthe command line and using file,atne is not specified in the _dictionarv_the data ard assumed to begin on the lifie following the close brace.
Description infix reads into memory
a disk dataset that is not in Stata format,
infix requires
that the data
_!
be in fixed-column forma You have alternatives t_ infix,
! !
(fixed format)--and it can read data in free format--see JR] infile (free format). Most people think infix is easier to use for reading fixed-format data, but infile has more features. If your data are not fi_ed-format,
another
is what you are looking In its first syntax, i !
i
is one. It can also read data in fixed-format--see
is insheet; See [R] insheet.
)r. see [R] irdile and [u]24 x reads the data in a two,step
Commands process.
[R] infile
If you are not certain that infix
to
input data.
You first create a disk file describing
how t_e data are recorde, t. You tell infix to read that file--called a dictionary--and from there infix goes on to read th_ data. The data can be in the same file as the dictionary, or a different file. In its second intermediate
i
inf
alternative
infile
syntax,
ou tell infix
how to read the data right on the command
file.
105
line with no
'i
106
:infix (fixedformat)
Read ASCII (text) data in fixed format
Options using(fi!ename2) specifies the name of a file containing the data. If using() is not specified, the data ate assumed to follow the dictionary in dfitename or, if the dictionary specifies the name of some other file, that file is assumed to contain the data. If using(fiIename2) is specified, filenamez is used to obtain the data even if the dictionary itself says otherwise. clear specifies that it is okay for the new data to replace what is currently in memory. To ensure that y+u do not lose something important, inf ix will refuse to read new data if data are already i in memory, clear is one way you can tell infix that it is okav. The other is to drop the data yourself by typing drop _all before reading new data. I
Specifications # first_ineoffile
(abbreviation first) is rarely specified. It states the line of the file where the
for its#lf, first is instead specified when only the data appear in a file and the first few lines of that fillecontain headers or other markers. data begin, first is not specified when the data follow the dictionary; infix can figure that out i firstl appears only once in the specifications.
i! i" :
# lines!states the number of lines per observation in the file. Simple datasets typically have '1 ; lines!. Large datasets often have many lines (sometimes called records) per observation, lines is optional even when there is more than one line per observation because infixcan sometimes figure _t out for itself. Still, if 1 lines is not fight for your data, it is best to specify the directive.
'
i, ,
lines Iappears only once in the specifications. #: tells infix to jump to line # of the observation. Consider a file with 4 lines, meaning four lines per observation. 2: says to go to the second line of the observation. 4: says to go to the fourth line of_the observation. You may jump forward or backward: infix does not care nor is there any inefficiency in going forward to 3:, reading a few variables, jumping back to 1:, reading anothei" variable, and jumping back again to 3 :. It is n0t your responsibility to ensure that, at the end of your specification, you are on the last line of!the observation, infix knows how to get to the next observation because it knows where you are and it knows lines, the total number of lines per observation #: may appear, and typically does, many times in the specifications. / is an alternative to #:. / goes forward one line. //goes forward two lines. We do not recommend the usd of / because #: is better. If you are currently on line 2 of an observation and want to get [_ "'
to linei6, you could type////, but your meaning is clearer if you type 6:. / may!appear many times in the specifications.
: : i
[byte I int Ifloat j long I double and, sdmetimes,_more than one.
I str ]varlist [#-]#[-#]
instructs infix
to read a variable
'_
Begin _y realizing that the simplest form of this is varname #, such as sex 20. That says that variabl_ varname is to be read from column # of the current line: variable sex is to be read from
: '_ t
column20 and here, sex is a one-digit number. varn " rn m fr m the column ran e s eclfied read ar_e#-#, such as age 21-23, says to readva a e o " g p " ; age frtm columns 21 through 23 and here, age is a three-digit number. You cab prefix the variable with a storage type. str name 25-44 means to read the string variable name _rom columns 25 through 44. If you do not specify str. the variable is assumed to be numeriC. "Youcan specify the numeric subtype if you wish.
infix (fixed format) _ Read A_Cll (text) data in fixeclformat-
i i
You can specify more than one variable, with or without a type. byte ql-q5 51-55 means read va_ables ql, q2, q3, q4. and q5 from column_; 51 through 55 and store the five variables as b_tes. Finally, you can spec fy the line on which the Variable(S) appear, age 2:21-23
i
107
says that age is
tO:be obtained from }he second line, column_ 21 through 23. Another way to do this is to put together the #: direct}ve _ith the input-_afiabte directive: 2: age 21-23. There is a difference. but not with respect t_ reading the variable age, Let s consider two alternatives: ;1: str name 25-4_ age 2:21-23 ql-q5 51-55 1:
[
str
name
25-44
2:
age
21"23
ql-q5
51-55
The difference is thai the first directive says variables ql through q5 are on line I whereas the seCond says they are an line 2. When the colon is p_t out front it says on which line variables are to be found when we do not explicitly say otherwise. Vc'hen the colon is put inside, it applies only to the variable under consideration.
Remafks There are two ways t9 use "infix il
One is to type the specifications that describe how to read the
fixed_format data on thelcommand line: .
infix
ace
rate
_-4
spdlimit
6-7
acc_pts
9-11
using
highway.raw /
The other is to type the specifications into a file Z
topof highway.dcI,exampleI
--i infix
dictionary acc rate spdlimit acc_pts
asing highway.raw t-4
{
3-7 I-II
} end of highway.dct,
example
I
and {hen, inside Stata. t, _e . infix
{
i
using
hig way.dct
Which you use makes r_o difference to Stata. The first form is more convenient if there are only a few variables and the second form is less prone to error if you are reading a big, complicated file The second form alkws two variations, the one we just showed--where file_and one where the data are in the same file as the dictionary:
the data are in another
topof highwav.dct,example2i
infix
dictionary acc_rate
{ i-4
spdlimit
_-7
acc_pts
)-II
} 4.58
55 .46
2.8660 1.61 3.02
4.4
2.2 60 4.7 --
>a ot6 that
in the first ex
ple, the top line of the file read infix
wheieas in the second toe line reads simply iMix {
dictionary.
data _.are it is implied t_at__the data follow the dictionary.
end of highway.tier
dictionary
example
using
2
highway.raw where the When you do not say.
108
infi]K(fixed format) -- Read ASCII (text) data in fixed format
'!.
> Example So let's complete the example we started. You have a dataset on the accident rate per million vehicle miles along a stretch of highway, the speed limit on that highway, and the number of access points per mile. You have created the dictionary file highway, dct which contains the dictionary and the data: top of highway.dct,example 2 infix d_ctionary { ace_rate I-4 spdlimit 6-7 acc_pts 9-11
} 4.58 2.86
55_ .46 6_ 4.4
1.61 3.02
i 2.2 6_ 4.7
|
! ! end of highway.dct,
example 2
You created this file outside of Stata using an editor or word processor. Inside Stata. you now read the data. infix lists the dictionary so you will know the directives it follows:
! i
: !
• infix_using highway infix dictionary { ace_rate 1-4 spdlimit 6-7 acc_pts 9-11
} (4 observations
read)
list 1. 2. 3. 4.
ace_rate 4.58 2.86 1.61 3.02
spdlimit 55 60 60
acc_pts .46 4.4 2.2 4.7
Note that we simply typed infix using highway rather than infix using highway.dct, When we do not specify the file extension, infix assumes we mean .dot. <1
Reading string variables When you do not say otherwise in your specification either on the command line or in the dictionary infix assumes variables are numeric. You specify that a variable is a string by placing str in front 9f its name: infix
id t-6
str name 7-36
age 38-39
str sex 41
uslng employee.raw
or top of emptoyee.dct infix d_etionary using employee.raw id t-6 isir name 7-36 age s_r sex
{
38-39 40
} end of empIoyee.dct
infix (fixedformat)--- Read ASCII(text) data in fixedformat
109
f Reading!multiple-lines-er-observation When a dataset has muir le lines per observation, sometimes said multiple records per observation. you specify the number ol lines per observation using lines and you specify on which line the elements appear using #:. . infix
2 lines
1: id 1-6
str name 7-36
2: age I-2
str sex 4
using emp2.raw
oF topofemp2Act iz_fixdictionary using emp2.raw { 2I:lines id sir name 2: age str sex
i"6 7' 36 1"i2 4
} end of emp2,dct
There _e lots of different _,ays to say the same thing.
,
> Example Consider the following
l
aw data:
i_ income educ / se_ age / rcode, 1024 25000 HS | Male 28 119503
top of mydata.raw answers
to questions
--
1-5
1025 27000 C Female 24 022113 1035 26000 HS Male 32 110321
f36 2sooo c Female 25 131232 ;
--
end of mydata.mw
This dmaset has 3 lines oer observation and the first line is just a comment. One possible set of specifi+ations to mad _is ktata'_is infix dictionary u i 2 first 3 lines I: id income str educ 2: str sex 3:
4
topof mydatalAct
ing mydata {
I-4 6-10 12-13 6-11
int age !13-14 rcode 16 ql-q5
7-16
I end of mydatal,dct
----_
although we pre_r i
110
infix (fixed format) -- Read ASCII (text) data in fixed format top of mydata2.dct infi_
dictionary
using
mydata
{
2 first 3 lines id
1:I-4
income
I: 6-10
' E_
sir
educ
1:12-13
i!i
sir
sex
2:6-11
I '
age rcode
2:13-14 3:6
ql-q5
3:7-16
} •end of mydata2.dct Either will read these data, so we will use the first and then explain why we prefer the second. • infix
using
mydatal
infix dictionary 2 first I:
using
lines id
mydata
{
1-4
income
6-10
str
educ
12-13
2:
str
sex
6-11
3:
int age rcode
13-14 6
ql-q5
7-16
} (4 observations • list
in
read)
I/2
Observation
1 id
1024
income
sex
Male
age
28
q2 q5
9 3
q! q4 Observation
1 0
25000
educ
HS
rcode
1
q3
5
2 id
1025
sex
Female
income
educ
C
age
27000 24
rcode
0
q3
1
ql
2
q2
2
q4
1
q5
3
Now, what is better about the second? What is better is that the location of each variable is completely documented on each line, in terms of both line number and column, Since infix does not care about the order in which we read the variables, we could take the dictionary, jumble the lines, and it would still work. For instance, .... infi:
dictionary first
using
mydata
top of mydata3.dct
{
lines str
sex
1
rcode
!
sir age id
educ
ql-q5 income
2:6-11 3:6 1:12-13 2:13-14 I: i-4 3:7-16 i: 6-10
}
t
end of mydam3.dct
!
[ ]
I
infix(fixedformat)--Read ASCII(text)datain fixedformat
111
wilt also read these data even though•for each observation, we start on line 2, go forward to line 3, jump back to line l, and end up on line 1. It is not even inefficient to do this because infix does not really jump to record 2, then record 3, then record 1 again, etc, infix takes what we say and organizes it efficiently. The order in which we say it makes no difference. Well, it does make one: the order of the variables in the resulting Stata dataset will be the order we specify. In this case the reordering is senselessbut, in real datasets, reordering variables is often desirable. Moreover, we often construct dictionaries, realize _at we omitted a variable, and then go back and modify them. By making each line complete in and of itself, we can add new variables anywhere in the dictionary and not worry that. because of our addition, something that occurs later will no longer read correctly. <1
Readingsubsetsof observations
i
If you wanted to read only the males from some raw data file, you might type • infix
id i-6
sir name 7-36
age 38-39
str sex 41
using employee.raw if sex=="M"
If your specification was instead recorded in a dictionary, you could type infix
using employee.dct i_ sex=="M"
In another dataset, if you wanted to read just the first t00 observations, you could type (
infix 2 lines > in i/i00
1:
id I-6
str name 7-36
2: age i-2
str sex 4
using empi.raw
Or, if the specification was instead recorded in a dictionary and you wanted observations 10l through 573, you could type • infix using emp2.dct in 101/573
Also See Complementary:
[R]outfile, [R] outsheet, [R] save
Related:
[R]intile (fixed format), [R]insheet
B_ckground:
[L] 24 Commands to input data, [R]intile
i
F °'; e
input -- Enter data from keyboard I
I II I
I III
I
I
I
Syntax input
[varlist]
[,_automatic label]
Description input allows you to type data directly into the dataset in memo_• alternative to input.
Also see [R] edit for a windowed
Options automatic causes Stata to create
value labels from the nonnumeric
data it encounters•
automatically widens the display format to fit the longest label. Specifying label even if you do not explicitly type the label option.
automatic
It also implies
label allows you to type the labels (strings) instead of the numeric values for variables associated with value labels. New value labels are not automatically created unless automatic is specified.
Remarks If there are no data in memory, when you type input you must specify a vartist• Stata will then prompt you to enter the new observations until you type end.
> Example You have data on the accident rate per million vehicle miles along a stretch of highway along with the speed limit on that highway. You wish to type these data directly into Stata: • input nothing to input r (104) ;
Typing input by itself does not provide enough information know the names of the variables you wish to create. • input ace_rate spdlimit 1. 2. 3. 4.
ace_rate 4.58 55 2.86 60 1.61 end
spdlimit
112
about your intentions.
Stata needs to
input -- Enter data from keyboard
113
:
! _ !
i
We typed input acc_rate spdlimit and Stata responded by repeating the variable names and then prompting us for the first observation. We then typed 4.58 and 55 and pressed Retth,'n. Stata prompte_ktusfor the second obsen, ation. We entered it and pressed Return. Stata prompted us for the third 6bservation. We knew that the accident rate is 1.61 per million vehicle miles, but we did not know the corresponding speed limit for the highway. We typed the number we knew, 1.61, followed by a period, the missing value indicator. When we pressed Return, Stata prompted us for the fourth 6bservation. We were finished entering our data, so we typed end in lowercase letters.
i i
We can now list
the data to verify that we have entered it correctly:
. list i. 2. 3.
acc_rate 4.58 2.86 1.61
spdlimit 55 60 Q
If you have data in memory and type input without a vartist, you will be prompted to enter adklitional information on all the variables. This continues until you type end. :
i
i
Examp You now have an additional observation you wish to add to the dataset. Typing input by itself tells Stata that you wish to add new observations: • i_ut 4, 5,
act_rate 3.02 60 end
spdlimit
St_ta rem/nded us of the names of our v_-iables and prompted us for the fomth observation. We entered 'the numbers 3,02 and 60 and pressed Return. Stats then prompted us for the fifth observation. We could add as many new observations as we wish. Since we needed to add only one observation, we typ_ _nd, Our dataset now has four observations. "xl
You may add new variables to the data in memory by typing input followed by the names of the new variables. Stata will begin by prompting yGu for the first observation, then the second, and so on, until you type end or enter the last observation.
'iExample i
! ,
In addition to the accident rate and speed limit, we now obtain data on the number of access points (omramps and off-ramps) per mile along each stretcl of highway. We wish to enter the new data.
I
• input acc_pts acc_pts t. 4.6 2. 4.4
3 2.2 I
i
4. _4.7
F
114 input -- Enter data from keyboard When we typed input acc_pts, Stata responded by prompting us for the first observation. There are 4.6 access points per mile for the first highway, so we entered 4.6 and pressed Return. Stata then prompted us for the second observation, and so on. We entered each of the numbers. When we entered the final observation, Stata automatically stopped prompting us--we did not have to type end. Stata knows that there are four observations in memory, and since we are adding a new variable, it stops automatically. We can, however, type end anytime we wish. If we do so, Stata fills the remaining observations on the new variables with m/ssing. To illustrate this, we enter one more variable to our data and then list the result: • input
junk
jun_ 1. 1 2. 2 3. end • list acc_rate 4.58
I.
spdlimit 55
acc_pts 4.6
60
4.4
2.86
2. 3,
1.61
4.
3• 02
junk 1 2
2.2 60
4.7
q
You can input string variables using input, but you must remember to explicitly indicate that the variables are strings by specifying the type of the variable before the variable's name.
> Example String variables are indicated by the types str#, where #represents the storage length, or maximum length, of the variable. For instance, a str4 variable has maximum length 4, meaning it can contain the strings a, ab, abe, and abed but not abede. Strings shorter than the maximum length can be stored in the variable, but strings longer than the maximum length cannot. You can create variables up to str80 in Stata. Since a str80 variable can store strings shorter than 80 characters, you might wonder why you should not make all your string variables str80. You do not want to do this because Stata allocates space for strings based on their maximum length. It would waste the computer's memory. Let's assume that we have no data in memory and wish to enter the following input
strl6
name
age
str6
name i.
"Arthur
2.
"Mary
3. Guy "Fawkes" 3.
"Guy
Hope"
Fawkes cannot
We first typed input sex a str6 variable.
sex age
sex
22 male
37
"female"
48 male be read
Fawkes"
4. "Kriste 5. end
:
Doyle"
data:
as a number
48 male
Yeager"
25 female
strl6 name age str6 sex, meaning that name is to be a strl6 variable and Since we did not specify anything about age, Stata made it a numeric variable.
Stata then prompted us to enter our data. On the first line, the name is Arthur Doyle, which we typed in double quotes. The double quotes are not really part of the string; they merely delimit the
_!lput
_
_l_,t;'w uam
llVli!
hVyL_)awu
J ,_,
beginning and end of the str ng. We followed that with Mr Doyle's age, 22, and his sex, male. We did not bpther to type doubk quotes around the word male because it contained no blanks or special characters. For the second o _servation,we did type the double quotes around female;it changed _othing. In the third observation w omitted the double quotes around the name, and Stata informed us that Fawkes c_uld not be read as number and repromptddus for the observation. When we omitted the double q_otes, Stata interpre:ed Guy as the name, Fa_rl_esas the age, and 48 as the sexl All of this would have been okay with Stata except for one problem: Fawkes looks nothing like a number, so Stata complained and gave :s another chance. This lime, we remembered to put the double quotes around ttie name.
i
Stata was satisfied, and _ continued. We entered lhe fourth observation and then typed end. Here is our dataset: • _ist 1. 2. 3. 4.
1 nam_ Arthur Doyle Mary Hope Guy Fawke. _ Kriste Yeagez
age 22 37 48 25
sex male female male female
q
I
>
I
Example Just as we indicated whic Lvariables Werestrings by placing a storage type in front of the variable name, we can indicate the .torage type of our numeric variables as well. Stata has five numeric storage types: byte, int, 1c ng, float, and double. When you do not specify the storage type. Stata assumes the variable is afl _at. Youmay want to review the definitions of numbers in [U] 15 Data.
! ' i i i i !i i i
,'_dditional Therei are two reasons you might The wantdefault to explicitly specify storage type: toforinduce precision or to co_vhy aserve memory. type float has the plenty of precision most circumstances because Stata performs all calculations in double precision no matter how the data are stored. I[ you were storing 9-digit Social Security Numbers, however, you would want to coerce a different storage type or else the last digit would be r0uhded, long would be the best choice: double would _,_orkequally well, b_]tit would waste memory. Sometimes you do not need to store a variable as float.If the variable contains only integers between -32,768 and 32,7i_6,it can be stored as an int and would take only half the space. If a variable contains only inti',gersbetween -127 and 126, it can be stored as a byte which would _:akeonly half again as mu( i space. For instance, in tile previous example we entered data for age ,_ithout explicitly specifyin, the storage type; hence, it was a float. It would have been better to _tore it as a byte.To do ti" tt. we would have typed input
strl6
name b _te age str6 nam _
_. "Arthur Doyle"
sex sex
12male
°I i
2. "Mary Hope" 37 'female" _. "Guy Fawkes" 48 male
i
4. "Kriste
Yeager"
age
25 female
5. end
i
Stata understands a number of shorthands. For instance,
_I
input
int(a
b) c
allows you to input three variables, Remember .{input
int
and c a float•
a b c
would make a an int *
a, b, and c, and makes both a and b ints
but both b and c floats.
. inputa longb double(cd) e would make a a float,b a long,c and d doubles,and e a float. Statahas a shorthandforvariable names withnumericsuffixes. Typingvl-v4 isequivalent to typing Vl v2 v3 v4. Thus, • linput
'
int(vl-v4)
inputs f6ur variables and stores them as ints. q
Q Technic,_l Note You may want to stop reading now. The rest of this section deals with using input with value labels. If you are not familiar with value labels, you should first read [U] 15.6.3 Value labels. Remdmber that value labels map numbers into words and vice versa. There are two aspects to the process. !First, we must define the association between numbers and words. We might tell Stata that 0 corresponds to male and 1 corresponds to female by typing label define sexlbl 0 "male" 1 "female". The correspondences are named, and in this case we have named the O_male l++female correspondence sexlbl. Next, iwe must associate this value label with a variable. If we had already entered the data and the variable was called sex, we would do this by typing label values sex sexlbl. We would have entered the data by typing O's and !'s, but at least now when we list the data, we would see the words rather than the underlying.numbers. We cab do better than that. After defining the value label, we can associate the value label with the type:variable at the time we input the data and tell Stata to use the value label to interpret what we l_bel • i_put
define strl6
I.
"Arthur
2.
"Mary
3.! "Guy
sexlbl name
byte(age
Hope"
1 "female"
sex:sexlbl),
name Doyle" 22 male
Fawkes"
4. "Kriste 5. end
0 "male"
age
label sex
37 "female" 48 male
Yeager"
25 female
After deft ing the value label, we typed our input command. Two things are noteworthy: We added the label option at the end of the command, and we typed sex:sexlbl for the name of the sex variable, T_e byte(...) around age and sex:sexlbl was not really necessary: it merely forced both age _nd sex to be stored as bytes. Let's first decipher sex : sexlbl, sex is the name of the variable we want to input. The : sexlbl part tells Stata thal the new variable is Lo be associated with the value label named sexlbl. The label option tells Stata that it is to look up any strings we type for labeled variables in their
input- Enter datafrom keyboard
117
corresponding value label and substitute the number when it stores the data. Thus, when we entered the first observation of ou • data, we typed male for Mr Doyle's sex even though the corresponding variable is numeric. Rather than complaining that ""mate" could not be read as a number", Stata accepted what we typed, 3oked up the number corresponding to male, and stored that number in the data.
i
The! fact that Stata has lctually stored a number rather than the words male or female is almost irrelevant. Whenever we ist the data or make a table, Stata will use the words male and female just as if those words were actually stored in the dht/set rather than their numeric codings: • list
I.
nm _e
age
se_
DoylLe
22
male
Ho],e
37
female
Guy Fawb is
48
male
95
female
Arthur
2.
Mary
3. i
, Kriste Yeag_ r tabulate sex sex
] req.
Percent
Cure,
male
2
50. O0
50.00
female
2
50. O0
I00. O0
Total
4
I00. O0
It is only almost irreleva at since we can make use of the underlying numbers in statistical analyses. For instance, if we were to ask Stata to calculate the mean of sex by typing sumrnarize sex, Stata would report 0.5. We woul interpret that to mean that one-half of our sample is female.
i i
Value labels are perman_ Itly associated with variables. Thus, once we associate a value label with a variaNe, we never have ti do so again. If we wanted to add another observation to these data, we
i
could type . input,
i i
label
i5. "Mark
Esman"
nam _ 26 male
age
sex
_. end
!_ i
[3Technical Note The automatic option ',utomates the definition of the value label. In the previous example, we _nformed Stata that male c, ,rresponds to 0 and female corresponds to 1 by typing label define sexlbl 0 "male" :t "female". It was not necessary to explicitly specify the mapping. Speci_,ing the aut6maticoption tells ;tara to interpret what we type as follows:
i i
ii
First, ;see if it is a numbeI If so, store that number and be done with it. If it is not a number, check
I I ! i_ i
I
the value label associated u th the variable in an attempt to interpret it. If an interpretation exists, store theIcorresponding nun: .tic code• If one does not exist, add a new numeric code corresponding to what was typed. Store th_ new number and update the value label so that the new correspondence is never t'orgotten. We can use these feature to reenter our age and sex data. Before reentering the data, we drop -all and label drop _all to prove tha_ we have nothing up our sleeve:
atop_an _abel
drop
_all
....
i
118
i input -- Enterdata from keyboard input
strl6
name
!
byte(age
sex:sexlbl),
name
i.
"Arthur
_.
"Mary
3.
"Guy
4.
"Kriste
Doyle" Hope"
22
37
Fawkes"
age
48
Yeager"
automatic sex
male
"female" male 25
female
. end i I
•
T We previouslydefinedthevaluelabelsexlbl so thatmale correspondedto 0 and female corresponded
to 1. Th+ label that Stata automatically created is slightly different but just as good: i
Sabel list sexlbl se: Ibl : I
male
0 2
female
Also See
'
Complementary:
[R] save
i
Related: i
[R] edit, [R] infile
Background:
[U] 24 Commands to input data
, i
t
I
i
, I
!
!_ !
i
_ in_iheet -- Read AS II (text)data created by a spreadsheet i i iHll i r i i iJ ii i iil ill [
i
Syntax i i
i i
i
insheet
[varlist] using
filename
[, _double [no]n_ames comma t__abclear
]
If filen_me is specifiedwithmt an extension, .raw is assumed.
Description
in_heet reads into rremory a disk dataset that is not in Stata format. ±nsheet is intended for readir_g files created by a spreadsheet or database program. Regardless of the creator, :i,nsheet reads text (ASCII) files where here is one observation per line and the values are separated by tabs or commas. In addition, the first line of the file can contain the variable names or not. The best thing
I i
[ about!insheet is that if you type . insheetusingill, name
i
insheet will read your lata; that's all there is to it. Stata has other comrr ands for reading data. If you are not sure that insheet
i
lookingfor, see [R] infih and [U] 24 Commands to input data. If y/ou want to save your data in "spregdsheet-style" forma
Options
is what you are
see [R] outsheet.
i
double forces Stata to st_age types.
t
tore variables as double_
rather than floats:
see IV] 15.2.2 Numeric
:-
[no]names informs Stata whether variable names are included on the first line of the file. Speci_,ing this option will speed insheet's processing--assuming you are right--but that is all. ±nsheet can determine for itse!f whether the file includes variable names.
1
comma tells Stata that the values are comma-separated. Specifying this option will speed insheet's pr0cessing--assumin_ you are right--but thai is all. insheetcan determine for itself whether the separation charact_ is a comma or a tab.
i i
!
tab prOcessing--assumin_ tells Stata the v_lues are right--but tab-separated. this can option will speed insheet's you are that Specifying ig all. insheet determine for itself whether the separation charact_:r is a tab or a comma.
i
clear specifies that it is okay for the new data |o replace what is currently in memory. To ensure that you do not lose sc mething important, insheetwill refuse to read new data if data are already in memory I clear
is _ne way you can tell ±nsheet
x_ourselfb_ typing drip
_all
that it is okay. The other is to drop _he data
before reading new data.
119
i
12o
Remarks
insheet-
Read ASCII (text) data created by a spreadsheet
There i_ nothing to using insheet.You type insheet and
insheet
using
filename
will read your data, That is, it will read your data if
1. It can find the file and 2. The file meets insheet's
expectations
as to the format in which it _s written.
Assuring I is easy enough; just realize that if you type infix using myfile, Stata interprets this as an instruction to read myfile.raw. If your file is called myfile.txt, type infix using myf ile. t,btt. As for the file's fo,-mat, most spreadsheets and some database programs write data in the form insheet expects, It is easy enough to look as we will show you--and it is even easier simply to try and see what happens. If typing • insheet
using
filenarrle
does not produce the desired result, you will have to try one of Stata's other infile commands: see [R] infile.
> Example You ha*e a raw data file on automobiles and can bd read by typing (5
called auto.raw.
This file was saved by a spreadsheet
insheet using auto vars, I0 obs)
That done, we can now look at what we just loaded: • describe Contains
data
obs:
I0 5
vats: size:
310
(99.8%
storage
of memory
free)
display
value
type
format
label
make_ price
strt3 int
%13s %8.0g
mpg
byte
Z8.0g
rep78
byte
%8.0g
foreign
strlO
ZlOs
variable
name
Sorted by: |Note:
dataset
has
changed
since
last
variable
label
saved
li_t
I. i 2._I 3,
make
price
mpg
AMC Concord AMC Pacer
4099 4749
22 17
3 3
foreign Domestic Domestic
Spirit
3799
22
4. Buick 5. Buick
Century Electra
4816 7827
20 15
3 4
Domestic Domestzc
6. Buick
LeSabre
5788
18
3
Domestzc
4453
26
7. !
AMC
rep78
Buick
Opel
Domestic
Domestic
insheet 8. BuickRegal 9. Buick Riviera 10. Buick
Read ASCII (text) data created by a spreadsheet
5189 10372
20 16
3 3
Domestic Domestic
4082
19
3
Domestic
Skylark
Note that these data contain a combination of string and numeric variables, insheet out by i_elfi
121
figured all that
i
i
3 Technical Note Now let's back up and look at the auto.raw screen: • Sype mal_e
These invisible
file. Stata's type command will display files to the
auto.raw mpg
rep78
foreign
AM¢ Concord
4099
22
3
Domestic
AMC Pacer
4749
17
3
Domestic
AMC Spirit
3799
22
Buick
Century
4816
20
3
Domestic
Buick Buick
Electra LeSabre
7827 5788
15 18
4 3
Domestic Domestic
Buick
Opel
4453
26
Buick Buick
Regal Riviera
5189 10372
20 16
3 3
Domestic Domestic
Buick
Skylark
4082
19
3
Domestic
data and
i
price
have
tab
hence
characters
Domestic
Domestic
between
indistinguishable
i !
]
i
values.
from
blanks,
Tab
characters
type's
showtabs
are
difficult option
to makes
see
since the
tabs
thev
are
visible: I
_ype
auto.raw,
showtabs
1
makepricempgrepZ8foreign AMC Concord4099223Domestic AMC Pacer4749173Domestic AMC Spirit379922.Domestic Buick Century4816203Domestic Buick Electra7827lS4Domestic Buick
LeSabre5788lS3Domestic
Buick
Opel4453<_>26.Domestic
Buick Buick
Rega!5189203Domestic KivieralO372i63Domestic
Buick
Skylark4082193Domestic
:
This is an example of the kind of data insheet is willing to read. The first line contains the variable names, Nthough that is not necessm/. What is nedessary is that the data values have tab characters between them. + insheet would be just as happy if the data values were separated by commas. Here is another vafiationi on auto. raw that insheet can read: type
auto2.raw
make,price,mpg,rep78,foreign AMC Concord,4099,22,3,Domestic AMC Pacer,4749,17,3,Domestic AMC Spirit,3799,22,,,Domestic Buick Buick
Century,48i6+,20,3,Domestic Electra,7827i, 15,4,Domestic
Buick
LeSabre,5788,18,3,Domestic
Buick
Opel,4453,26!,,Domestic
i
Buick Buick
Regal,5189,20,3,Domesti¢ Riviera,lO37_,16,3,Domestic
i
Buick
Skylark,4082.,19,3,Dom_sZic
["
It is way one easieror for the us other. human beings to see the commas rather than the tabs. but computers 122
insheet-
do not care O
Read ASCII (text) data created by a spreadsheet
!
D Example The file does not have to contain variable names. Here is another variation on auto. the first line. this time with commas rather than tabs separating the values:
raw without
type auto3, raw AMC Concord, 4099,22,3, Domestic AMC Pacer, 4749,17,3 ,Domestic (output omitted ) Buick
Skylark. 4082,19,3, Domestic
Here is what happens when we read it: insheet using auto3 you must start with an empty dataset r(I8);
Oops; we still have the data from the last example in memory. • insheet using auto3, clear (5 vars, I0 obs) . describe Contains data obs : vats : size:
variable name
10 5 310 (99.8Y,of memory free) storage type
display format
vl
strl3
Y,13s
v2 v3 v4 v5
int byte byte strlO
7,8.0g ]/,,8.0g XS. 0g Y, IOs
Sorted by : Note:
value label
variable label
dataset has changed since last saved
list vl AMC Concord AMC Pacer
v2 4099 4749
v3 22 17
v4 3 3
v5 Domestic Domestic
(output omitted ) i0. Buick Skylark
4082
19
3
Domestic
I. 2.
'j
The only difference is that rather than the variables being nicely named make, price, mpg, rep78, and foreign,they are named vl,v2, ..., v5. We could now give our variables nicer names: • rename vl make . rename v2 price
insheet-- Read ASCII (text) data created by a spreadsheet
123
!
Another alternative is to specie' the variable names when we read the data: • insheet make price mpg rep78 foreign u_ing auto3, clear
(5 vats,I0obs) list make AMC Concord AMC Pacer
i. 2.
price 4099 4749
mpg 22 17
rep78 3 3
4082
19
3
foreign Domestic Domestic
!
i
(outputomi,ed ) 10.
,i
Buick Skylark
Domestic
ii
If we use this approach, we must not specify too few variables • insheet make price mpg rep78 using aut03, clear too few variables specified error in line 11 of file r,(102) ;
i 4 I
|
or too many.
|
. insheet make price mpg rep78 foreign weight using auto3, clear too many variables specified e_ror in line 11 of file r,(103);
mat is why we recommend . insheet using
i
filename
/
It is not difficult to rename your variables afterwards should that be necessary,
q
> Example
I
About the only other thing that can go wrong is that the data are not appropriate for reading by insheet. Here is yet another version of the automobile data: type auto4.raw, showZabs "_AMCConcord" 4099 22 '_,AMC Pacer" 4749 17
3 3
Domestic Domestic
3 4
Domestic Domestic Domestic
"!AMCSpirit" '_BuickCentury" "Buick Electra"
3799 4816 7827
22 20 15
"Buick LeSabre" "Buick 0pel" "Buick Regal" ".'Buick Riviera"
5788 4453 5189 10372
18 26 20 16
3 3 3
Domestic Domestic Domestic Domestic
"_uick Skylark"
4082
19
3
Domestic
] ]
i
Note that we specified type's showtabs option and no tabs are shown. These data are not tabdelimited or comma-delimited and are not the kind of data insheet is designed to read. I,et's try insheetanyway:
] 1
(Continued on next page)
i
124
insheet -- Read ASCII (text) data created by a spreadsheet insheet using auto4, clear (I vat, I0 obs) describe Contains data obs : vats:
10 1
size:
430 (99.8Y,of memory free)
variable name
storage type
vl
display format
sir39
Sorted by: Note:
value label
variable label
7,39s
dataset has changed since last saved
• list vl I. AMC Concord 2. AMC Pacer (output omitted) 10, Buick Skylark
4099 4749
22 17
3 3
Domestic Domestic
4082
19
3
Domestic
When £nsheet tries to readdatathathave no tabsor commas, itisfooledintothinkingthedata contain justone variable. Ifyou had thesedata,you would have toreadthedatawithone ofStata's other commands such as infile (free format). <3
Also See Complementary:
[R] outfile, [R] outsheet, [R] save
Related:
[R] infile (free format)
Background:
[U] 24 Commands [R] intile
to input data,
Ilrt"w
1
e
r
I i;insect -- Display simple summary of data's ci laracteristics I IU ( i i II t _ _
m I,
_
I / "
I
i
yntax, :in.nspeict [varlist_ [±fexp] [in range] i byi ...
i
: may be used
with
inspect;
see [R] by.
)esCriiOn
i 1
i
The inspect command provides a quick summary of a numeric variable that differs from that provided by summ_arizeor tabulate. It reports ttie number of negative, zero, and positive values;
I
the nunlber of integers and nonintegers; the number of unique values: the number of missin_ and produces _asmall histogram. Its purpose is not anal)tical but to allow you to quickly gain familiarity with u_own data.
'
! l
ilt_vartisl, i
! i 1
Typing inspect
Example inspect
by itself produces an inspectio_ for all the variables in the dataset. If you specify
aninspectionofjustthosevafiablesisp resented. is not a replacement or substitute for s!!rnma/'ize and tabulate.
It is instead a data
management or information tool that lets you quickly g_in insight into the values stored ir_a variable, For instance, you receive data that purport to be on automobiles, and among the variables in tile dataset is one called mpg.Its variable label is l_ileage (mpg), which is surely suggestive. You i_aspect the variable • use
auto,
(1978
Automobile
inspect m]_g:
Data)
mpg
Mileage
! (mpg)
Number
of
Observations Non-
# #
Negative Zero
Total-
# #
Positive
74 .....
74
-
# #
#
#
74
74
-
#
#
#
,
i
clear
Total #
12
Missing
-
Integers -
-
41 (21 unique
Integers
74
values)
and you discover that the variable _s never missiflg; all 74 observations in the dataset have some value for rapg. Moreover. the values are all positive and they are all integers as well. Among those 74 observations are 2t unique (different) values. The variable ranges from 12 to 41. and you are provided with a small histogram that suggests the variable appears to be what it claims
i
lzs
[
_zo
znspec[ -- ulspiay slmpm summary ot data's characteristics
> Example Bob, a co-worker, presents you with some census data. Among the variables in the dataset is one called region which is labeled Census Region and is evidently a numeric variable. You inspect this variable. • use bobsdata • inspect region region:
Census region
Number of Observations Non-
#
# #
# # # #
Negative Zero Positive
#
#
#
#
Total
#
#
#
#
Hissing
# # #
5
Total 50 50
Integers
Integers
50 50
-
50
(5 unique values) region is labeled but I value is NOT documented in the label.
In this dataset something may be wrong, region takes on five unique values. The variable has a value label, however, and one of the observed values is not documented in the label. Perhaps there is a typographical error. A call to Bob would be in order.
> Example You call Bob and there was indeed an erron He fixes it and returns the data to you. Here is what inspect produces now: inspect region region:
# # #
Census region
# # # #
# # # # # #
Number of Observations NonNegative Zero Positive
# # # #
1
Total Missing 4
Total 50 50 -
Integers
Integers -
50 50
50
(4 unique values) region is labeled and all values are documented in the label.
4
I
inspect-- Displaysimplesummaryof data'scharacteristics
i
'
127
Example You receive data on the climate in 956 U.S. Cities. The variable tempjan records the Average January temperature in degrees Fahrenheit. The results of inspectare • inspect tempjan tempjan:
Average January temperature
Number of Observations Non-
i Negativ@ Zero Positiv4
# # # # # #
# # #
# # #
Total 954
Total Hissing
954 2
2.2 72.6 (More than 99 unique values)
Integers 78 78
Integers 876 ........ ! 876
i i
956
, i
In two of the 956 observations, tempjan is nlissiQg. Of the 954 cities that have a recorded tempjan, all are positive and 78 of them are integer valuesl tempjanvaries between 2.2 and 72.6. There are more than 99 unique values of tempj an in the dathset. (Stata stops counting unique values after 99.) <1
i
1
SaVediResults inspect
saves
i in r()"
I
Scalars r(N) r(N..Jaeg)
number of observatiqns number of negative 0bservations
r(N_O)
number of observations equal to 0
i ]
r (N_pc_s) number of positive observations r (N_negint) number of negative, Integer observations r(N_posint)
number of positive, integer observations
r(N_tmique) r(N_undoc)
number of unique values or. if more than 99 number of undocumented values or. if not labeled
AlsoSee
t
i i
Related: i
i !
[R] codebook, [R] compare, [R] describe, [R] Iv, [R] summarize, [R] table, [R] mbsmn, [R] tabulate
i ii
1
1 lue [ ipolate -- Linearly interpolate(extrapolate) I I values
]
I
]
I
Syntax ipolate war war by ...
[if
exp]
: may be used with ipolate;
[in range],
generate(newvar) [ epolate
]
see [R] by.
1
Description ipolate interpolated
creates newvar = yvar where yvar is not missing and fills in newvar (and optionally extrapolated) values of yvar where yvar is missing.
with linearly
Options generate (newvar) epolaZe
is not optional; it specifies the name of the new variable to be created.
specifes values are to be both interpolated and extrapolated.
Interpolation
only is the default.
Remarks > Example You have data points on y and x although, in some cases the observations on y are missing. You believe y is a function of x justifying filling in the missing values by linear interpolation: 1ist 1. 2. 3. 4. 5. 6. 7.
x 0 1
y 3
1.5 2 3 3.5 4
6
18
ipolate y x, gen(yl) (I missing value generated) • ipolate y x, gen(y2) epolate • list i. 2. 3. 4. 5. 6. 7.
x 0 1 1.5 2 3 3.5 4
y
yl
3
3 4.5 6 12 15 18
6
18
y2 0 3 4.5 6 12 15 18
128
8
!
!
i_i
l l
ipolate-- i_inearlyinterpolate(extrapolate)values
129
]
> E_amNe i
i
You have a dataset of circulation of a magazine from 1970 through 1993. Circulation is recorded in a Variablecalled circ and the year in year. in a few of the years, the circulation is not known
i
_o you want to fill it in by linear interpolation: . ipo!ate
!
cite year, gen(icirc)
Now assume you have data on the circulations of 50 magazines; the identity of the magazines is i recorded in magazine (which might be a string variable--it does not matter): i
by magazine: ipolate circ year, gen(ic_rc)
!i
! if the by ... : prefix is specified, interpolation is performed separately for each group.
]
> E_ample You have data on y and z although some of the y values are missing. You wish to smooth y(x) using lowess (see [R] ksm) and then fill in missifigwdues of y using interpolated values: I
. ksm y x, gen(yls) lowess 'ipolate yls x, gen(iyls)
i i i
q
"
:i]
/
]
MethodsandFormulas
:
i
ipolateis implemented as an ado-file. The value Y at x is found by finding the closest points (xo, yo) and (xt.yl), and xl > x, where Yo and 91 are observed, and +alculating
such that x0 < x
Y] - Yo y= _ (x-- Xo) +yo X 1 -- X 0
if epoZate is specified and if (xo, Yo) and (xt,yl) cannot be found on both sides of x, the two closest points on the same side of x are found arid the same formula is applied.
t
!
AlSO _e Complementary:
[t_]ksm
i
I I
i
ii
I Itle
[ ivreg -- Instrumental ,, variablesmidtwo-stageleast " squaresregression
i
Syntax ivreg
depvar
[vartish
] (varlist2
= varlistiv)
[weight]
[if
exp]
[in
range]
[, level(#) _beta_hasconsnoconstant _robustcl.uster(varname) firs_ noheader eform(string)depname(varname) msel 1 by ... : may be used with ivreg; see [R] by. aweights, freights, iweights, and pweights are allowed; see [U] 14.1.6 weight. depvar, varlistt, varlist2, and varlistiv may contain time-series operators; see [U] 14.4.3 Time-series varlists. ivreg shares the features of all estimation commands; see [U] 23 Estimation and post-estimation commands.
Syntaxfor predict predict
[l-ype] newvarname
[if
exp]
[in range!
[, statistic j
where statistic is
xb re s iduals
xjb, fitted values (the default) residuals
p_r(a,b)
Pr(a < Yi < b)
e(a,b)
E(yjla < Yi < b)
ystar(a,b) stdp stdf
(Yj-), yj = max{a, min(yj,b)} standard error of the prediction standard error of the forecast
where a and b may be numbers or variables; a equal to '.' These aresample. available both in and out of sample; type predict the statistics estimation
means -2; ...
b equal to ' ' means -roe.
if e(sample)
...
if wanted only for
Description ivreg estimates a linear regression model using instrumental variables (or two-stage least squares) of depvar on varlistl and varlist2 using varIisti,, (along with varlish) as instruments for varlist2. In the language of two-stage least squares, varlist_ and varlistiv are the exogenous varlist2 the endogenous variables.
130
variables and
ivreg -- Instrumentalvariablesand two-stage least squares regression
131
OptiOns level (#) specifies the confidence level, in percent, for confidence intervals. The default is level or as set by set level: see [U] 23.5 Specifying the width of confidence intervals.
(95)
i
beta asks that normalized beta coefficients be reported instead of confidence intervals,
i I
hascons indicates that a user-defined constant or its equivalent is specified among the independent variables. Some caution is recommended when specifying this option as resulting estimates may not be as accurate as they otherwise would be. See [R] regress for more information. noconstant
suppresses the constant term (interCept) in the regression.
robust specifies that the Huber/White/sandwieh estimator of variance is to be used in place of the traditional calculation (White, 1980). This alternative variance estimator produces consistent standard errors even if the data are weighted or if the residuals are not identically distributed. robust combined with cluster() further allows residuals which are not independent within ctu_ter (although the3, must be independent between clusters). See [u] 23.11 Obtaining robust variance estimates.
i i
i t
If you specie, pweights,
I
robust
is implied[ see [u] 23.13 Weighted estimation.
cluster (varname) specifies that the observations are independent across groups (clusters) but not necessarily independent within groups, varname specifies to which group each observation belongs; e.g., cluster(personid) in data with repeated observations on individuals, cluster () affects the estimated standard errors and variance-covariance matrix of the estimators (VCE), but not the estimated coefficientsL see [U] 23.11 Obtaining robust variance estimates, cluster () can be used with pweights to produce estimates for unstratified cluster-sampled data, see [U] 23.13 Weighted estimation. Also see [R] svy estimators for a command designed especially for sur_'ey data. c!us't;er() by itself. first
implies robust:
specifying robust
cluster()
is equivalent to typing cluster()
requests that the first-stage regression results be displayed.
noheader suppresses the display of the ANOVAtable and summary statistics at the top of the output, di:splaying only the coefficient table, This option is often used in programs and ado-files,
'
eform(string) is used only in programs and ad0-files that employ ivreg to estimate models other than instrumental variable regression, eformO specifies the coefficient table is to be displayed in "exponentiated lbrm" as defined in [R] maximize and that string is to be used to label the exponentiated coefficients in the table. depname (varname) is used only in programs and ado-files that employ ivreg to estimate models other than instrumental variable regression, depname () may be specified only at estimation time. varname is recorded as the identity of the dependent variable even though the estimates are ca'tculated using depvar. This affects the labeling of the output--not the results calculated_but could affec_ subsequem calculations made by predict, where the residual would be calculated as deviations from varname rather than depvar, depname() is most typically used when depvar is a temporary variable (see [el macro) used as a proxy for varname. msel is used only in programs a_d ado-files that employ ivreg to estimate models other than instrumental variables regression, msel sets the mean square error to 1, thus forcing the variancecovariance matrix of the estimators to be (X'DX) -1 (see [R] matsize Methods and Fonnulas) and so affects calculated standard errors. Degrees of freedom for t statistics are calculated as _ ra_her than n - k.
t
132
ivreg -- Instrumental variables and two-stage least squares regression
Options for predict xb, the default, calculates
the linear prediction.
res±duals calculates the residuals; that is, gj -xjb. These are based on the estimated equation when the observed values of the endogenous variables are used--not the projections of the instruments onto the endogenous variables. pr(a,b) calculates interval (a, b).
Pr(a < xjb
+ uj < b), the probability
that yjlxj
would be observed
in the
a and b may be specified as numbers or variable names; tb and ub are variable names; pr(20,30) calculates Pr(20 < xjb + uj < 30); pr(lb,ub) calculates Pr(/b < xjb -+.uj < ub); and pr(20,ub) calculates Pr(20 < xjb + uj < ub). a =. means -_zxz; pr(. ,30) calculates Pr(xjb + uj < 30); pr(/b,30) calculates Pr(xjb + uj <: 30) in observations for which It) =. (and calculates Pr(/b < xjb + uj < 30) elsewhere). b =. means +c_; pr(20, .) calculates Pr(xjb + uj > 20); pr(20,ub) calculates Pr(xjb + uj > 20) in observations for which ub =. (and calculates Pr(20 < x3b + uj < ub) elsewhere). e(a,b) calculates E(xjb + uj I a < xjb + uj < b), the expected value of 9jlxj yj[xj being in the interval (a,b), which is to say, 9jlxj is censored. a and b are specified as they are for pr ().
conditional
on
ystar(a,b) calculates E(y_) where y_ = a if xjb + uj < a, y_ = b if xjb + uj > b, and 9j" = xjb + uj otherwise, which is to say, #_ is truncated, a and b are specified as they are for prO. strip calculates the standard error of the prediction. It can be thought of as the standard error of the predicted expected value or mean for the observation's covarlate pattern. This is also referred to as the standard error of the fitted value. sgdf calculates the standard error of the forecast. This is the standard error of the point prediction for a single observation. It is commonly referred to as the standard error of the future or forecast value. By construction, the standard errors produced by stdl are always larger than those by strip; see [R] regress Methods and Formulas.
Remarks ivreg performs instrumental variable_ regression (or two-stage least squares), and weighted instrumental variables regression. For a general discussion of two-stage least squares, see Johnston and DiNardo (1997), Kmenta (1997), and Wooldridge (2000). While computationally identical. Davidson and MacKinnon (1993, 209-224) present their discussion using instrumental variables terminology. Some of the earliest work on simultaneous systems can be found in Cowles Commission monographs--Koopmans and Marschak (1950) and Koopmans and Hood (1953)--with the first development of two-stage least squares appearing in Theil (1953) and Basmann (1957). The syntax for ±vreg assumes you want to estimate a single equation from a system of equations, or an equation for which you do not want to specify the functional form of the remaining system, if you want to estimate a full system of equations, either using two-stage least squares equation-by-equation or using three-stage least squares, see [R] reg3. An advantage of ±vreg is lhat you can estimate a single equation of a multiple-equation equations.
system without speci_ing
the functional form of the remaining
i;] ivreg -- Instrumentalvariables and two-stage least squares regression
133
EXample i
Let us assume that you wish to estimate hsngval = so + _lfaminc + _2reg2 + _3reg3 + _4reg4 rent = _0 + _lhsngval+ fi2pcturban + u
'
1
i 5%u have state data from the 1980 Census. housing, and rent is median monttfly gross income (famine) and region of the country flmction of hsngval and the percentage of
} i
+ E
hsng#al is the mextian dollar value of owner-occupied rent. You postulate that hsngval is a function of family (reg2 through reg4). You also postulate that rent is a the p_ulation living in urban areas (pcturban). i
If you are familiar with multi-equation modelS, you have probably already noted the triangular structure of our model. This triangular structure ]is not required. In fact, given the triangular (or i
recursive) structure of the model, if we were to assume that c and u were uncorrelated, either of the equations could be consistently estimated by ordinaD' least squares. This is strictly a characteristic of triangular systems and would not hold if hsngval were assumed to also depend on rent,regardless of assumptions about e and u. For a more detailed discussion of triangular systems see Kmenta (1997. _19-720). "
i
You tell Stata to estimate the rent equation by specifying the structural equation and the additional exogenous variables in a specific form. The depeddenT variable appears first and is followed by the exogenous variables in the structural model for rent, These are followed by a group of variables in parentheses separated by an equal sign. The variables to the left of the equal sign are the endogenous regressors in the structural model for rent and those to the right are the additional exogenous variables that will instrument for the endogenous variables. Only the additional exogenous variables need to be specified to the right of the equal sign: those already in the structural model wilt be automatically included as instruments, • As the following command shows, this is more difficult to describe than to perform. In this example, rent is the endogenous dependent variable, hsngvat is an endogenous regressor, and
i
• ivreg
i
I
rent
pcturban
(hsng_al = famine
:
i
'
]
i
r_g2-reg4)
famine,Instrumental reg2, reg3,variables reg4, and peturban are th_ exogenous variables. (2SLS) regression
R-squared = Number of obs = Adj R-squared =
0.5989 50 0.5818
1249.851)59
Root MSE
221862
18338.7_)17
Repidual Source
24565.7167 SS
47 df
Total
61243.12
49
F( 2, Prob > F
i
h_ngval pc_urban _cons Ins¢runented: Instrunents:
i
522,6741_23 MS
2
rent
1
42.66 0.0000
30677.4033
i .
47) = =
Model
Coef.
Std. Err.
t
P>lt[
.0022398 .081516
.0003388 '. 3081528
_. 61 _. 26
0.000 O. 793
120.7065
15.70688
_.68
O.000
=
[957,Conf. Interval] .0015583 -. 5384074 89.10834
.00292i3 .7014394 152.3047
hsngval i pc_urban fam_nc reg2 rag3 #eg4
i
I,
134
ivreg -- Instrumental variables and two-stage least squares regression
> Example Given the triangular nature of the estimated system, we might wonder if there is sufficient correlation between the disturbances to warrant estimation by instrumental variables. (We might have a similar question, even if the system were fully simultaneous.) Stata's hausman command (see [R] hausma.) will allow us to test whether there is sufficient difference between the coefficients of the instrumental variables regression and standard OLS to indicate that OLS would be inconsistent for our model. To perform the Hausman test, we use hausman to save the ivreg estimates, perform an OLS regression, and compare the two using hausmma. •
hausma11,
save
• regress
renl
hsngval
Source
pcturban SS
Model Residual
df
MS
Number
of obs
50
40983.5269
2
20491.7635
20259.5931
47
431.055172
K-squared
=
0.6692
1249.85959
Adj R-squared Root MSE
= =
0.6551 20.762
Total
61243,12
rent
49
Coef.
Std.
Err.
t
47)
=
F( 2, Prob > F
P>It _
[95_, Conf.
= =
47.54 0.0000
Interval]
hsngval
.0015205
.0002276
6.68
O. 000
.0010627
.0019784
pcturban _cons
.5248216 125.9033
.2490782 14. 18537
2.11 8.88
O. 040 O. 000
.0237408 97. 36603
i.025902 154.4406
hausman,
constant
sigmamore Coefficients Prior
Current
Difference
S.E.
hsngval
.0022398
.0015205
.0007193
pcturban _cons
.081516 120.7065 (b)
.5248216 125.9033 (B)
-.4433056 -5. 196801 (b-B)
I
b = less
efficient
B = fully Test:
Ho:
efficient
difference
estimates
in coefficients
chi2(I)
obtained
estimates not
The Hausman test clearly indicates that
=
OLS
previously
obtained
from
from
ivreg
regress
systematic
= (b-B)" [(V_b-V_B)'(-I)] = 12.08 Prob>chi2
.000207 .1275655 I. 49543 sqrt (diag(Vib-V_B))
(b,B)
O. 0005
is an inconsistent
estimator for this equation.
As opposed to a direct test of hsngval's endogeneity, Davidson and MacKinnon (1993. 236-242) have noted that this Hausman test is best interpreted as evaluating whether O[S is a consistent estimator for the model. The null hypothesis is that the model was generated by an OLS process and the test is performed under the assumption that the instrumental variables estimates are consistent. As an alternative to the Hausman test, Davidson and MacKinnon suggest an augmented regression test that is based on the same asymptotic requirements as the Hausman test. We can easily fornl the augmented regression by including the predicted values of each endogenous right-hand-side (rhs) variable, as a function of all exogenous variables, in a regression of the original model. For our hsngval model, we regress hsngval on all exogenous variables and include the prediction from this regression in an OLS regression of the hsngval equation. • regress hsngval (outputormttcd ) predict (option
faminc
reg2-reg4
hsng_hat xb assumed;
fitted
values)
pcturban
r
ivreg -
Instrumentalvadables and two-stage least squaresregression
.prediCt
hsng_res,
. regress
rent
res
hsng_al
Source
pcturbknhsng_hat SS
df
Number of obs F( 3 46) Prob > F R-squared Adj R-squared Root MSE
MS
: Model Residual
46189.152 15053.968
Total
61243.12
rent
Coef.
hs_val pctttrban hsns_hat _cons
135
.0006509 .0815159 .0015889 120.7065
3 46
15396.384 327.260173
49
1249.85959
Std. Err.
t
P>ttl
.0002947 .2438355 .0003984 12.42856
2.21 0.$3 3.99 9._1
0.032 0.740 0.000 0.000
= = = = = =
50 47.05 0.0000 0.7542 0.7382 18.09
[95_ Cong. Interval] .0000577 -.4092994 .000787 95.68912
.00124_2 .57233113 .0023908 145.72_9
Since we have only a single endogenous rhs variable, our test statistic is just the t statistic for the hsng__hat variable. If there were more than one endogenous rhs variable, we would need to perform a joint test of _illtheir predicted value regressors being zero. For this simple case, the test statement w_ld be • ,test _sng_hat (1) ,
Itsng_hat= 0.0 _( 1, 46) = Prob > F =
15.91 0.0002
While the p-value from the augmented regression test is somewhat lower than the p-value from the Hausman test, both tests clearly show that OLS is nor indicated for the rent equation (under the assumption that the instrumental variables estimator is a consistent estimator for our rent modeD.
_!Example Robust sta_ard • ivreg
rent
errors are availab_ with ±vreg: pcturban
(hsngval
= famine
reg_-reg4),
robust
IV (2SLS) regression with robust standard errors
--_
rent
Coef.
hsngval pcturban _cons
.0022398 .081516 120.7065
Robust Std. Err.
t
P>It I
.0006931 .4585635 15.7348
3.23 O.18 7.67
O.002 O.860 O.000
Number of obs = F( 2, 47) = Prob > F =
50 21._4 O.O(YO0
R-squared Root MSE
O.5989 22.882
= =
[95_ Conf. Interva_l] .0008455 -. 8409949 89.05217
.0036342 i.004027 152.3609
T
InstzRlmented: hsngval In_tra_ments: pcturban famine reg2 reg3 re$4
The robust star_darderror for the coefficiem on housing value is double what was previously estimated.
_
_
13u
wreg -- mstrumemal variables and two-stage least squares regression
Q Technical Note You may perform weighted two-stage instrumental variables qualifier with irreg. You may perform weighted or unweighted variable estimation, suppressing the constant, by specifying the constant is excluded from both the structural equation and the
estimation by specifying the [weight] two-stage least squares or instrumental noconstant option. In this case, the instrument list. Cl
Acknowledgments The robust estimate of variance with instrumental Mead Over, Dean Jolliffe, and Andrew Foster (1996).
variables was first implemented
in Stata by
Saved Results ivreg saves in e() : Scalars e (N) e (ross) e(df_m) e(rss) e(df.._r) Macros
number of observations mode] sum of squares mode] degrees of freedom residual sum of squares, residual degrees of freedom
e(r2) e (r2_a) e(F) e(rmse) e(N_clust)
e(cmd)
ivreg
e(clustvar)
e(version)
version number of ivreg name of dependent variable iv weight type weight expression
e(vcetype)
e(depva.r)
e(model) e(wtype) e (wexp) Matrices e (b)
coefficientvector
e(instd)
e(insts) e(predict)
e (V)
R-squared
adjusted R-squared F statistic root mem_square error number of clusters name of cluster variable covariance estimation method instrumented variable instruments program used to implement predict
variance-covanance matrix of the estimators
Functions e (sample)
marks estimation sample
Methods and Formulas ivreg
is implemented
as an ado-file.
Variables printed in lowercase and not boldfaced (e.g., x) are scalars. Variables printed in lowercase and boldfaced (e.g., x) are column vectors. Variables printed in uppercase and boldfaced (e.g., X) are matrices. Let v be a column vector of weights specified by the user. If no weights are specified, then v -- 1. Let w be a column vector of normalized weights. If no weights are specified or if the user specified fweights or iweights, w= v. Otherwise, w = {v/(I'v)}(ltl).
i
i
1 i
J
!
ivreg -- Instrumentalvariablesand two-stageleast squares regression
137
The number of observations, n, is defined as l'w. In the case of iweights, this is truncated to an integer. The sum of the weights is l'v. Define e = t if there is a constant in the regression and zero otherwise. Define k as the number of right-hand-side (rhs) variables (including the constant). Let X denote the matrix of observations on the ths variables, y the vector of observations on the left-hap,d-side (lhs) variable, and Z the matrix of observations on the instruments. In the following formulas, if the user specifies weights, then X'X, X ! y, y'y, Z'Z, Z'X, . and Z'y are replaced by X'DX; X'Dy, y'Dy, Z'DZ, Z'DX, and Z'Dy, respectively; where D is a diagonal matrix whose diagonal elements are the elements of w. We suppress the D below to simpli_ the notation. Define A as X'Z(Z'Z)-I(x'z)
j
i
' and a as X'Z(Z'Z)-IZ'y.
The coefficient vector b is defined as A-la. Although not shown in the notation, unless hascons is specified, A and a are accumulated in deviation form and the constant is calculated separately. This comment applies toall statistics listed below. The total sum of squares, ySS, equals y'y if there is no intercept and y'y - { (l'y)2/n The degrees of freedom are n - c. The error sum of squares, ESS, is defined as y'_ - 2bX'y
k.
i
aren
The mode/sum
+ b'X'Xb.
} otherwise.
The degrees of freedom
of squares, MSS, equals TSS- ESS. The degrees of freedom are k - c.
The mean square error, s2. is defined as ESS/(n 2 k). The root mean square error is s, its square root. If c - 1, then F is defined as F = (b - c)iA(b - c) (k _ 1)s 2 where e is a veclor of k - 1 zeros and h'th element l'y/n. this case, you may use the test
Otherwise, F is defined as missing. (In
command to construct any F test you wish )
]
The R-squared, R 2, is defined as R 2 = 1 - ESS/TSS.
i
The. adjusted R-squared, R2a, is 1 - (1 - R2)(n- c)/(n - k). If robust is not specified, the conventional estimate of variance is s2A -1.
i ]
For a discussion of robust variance estimates in the context of recession and regression with instrumental Variables. see [R] regress. Methods and Formulas. See this same section for a discussion of: the formulas for predict after irreg,
i i i]
References Baltagi,B. H. 1998. Econometrics.New York: Springer-Verlag. Basmann. R. L. t057. A generalizedclassical method of iinear estimationof coefficientsin a structural equation. -
Econometrica25: 77-83. Davidson.R, and J. G. MacKinnon.t993. Estimationand Inferencein Econometrics.New York:OX%rdUniver_iU Press:
Johnston.J. and J. DiNardo.I99Z EconometricMethods.4lh ed New York:McGraw-Hill. Kmenta,J. 1997. Elememsof Economemcs.2d ed. Ann Arbor:Universityof MichiganPress. Koopmans,T. C. and W C, Hood. I953. Studiesin EconometrkMethod,New York:John Witey& Son_. Koopmans.T. C and J. Marschak.I950. StatisticalInferencein DynamicEconomic Models. New York:John Wiley & SOns.
I
138
ivreg -- Instrumental variables and two-stage least squares regression
Over, M.. D. Jolliffe. and A. Foster. 1996. sg46: Huber correction for two-stage least squares estimates. Stata Technical Bulletin 29: 24-25. Reprinted in Sta_aTechnical Bulletin Reprints, vo]. 5. pp. 140-142. Theil, H. 1953. Repeated Least Squares Applied _oCompleteEquation Systems. Mimeographfrom the Central Planning Bureau, Hague. White, H. 1980. A heteroskedasticity-consistentcovariance matrix estimator and a direct test for heteroskedasticity. Econome_ca 48: 817-.838. Wooldridge, J. M. 2000. Introductory Econome;_cs: A Modern Approach. Cincinnati. OH: South-Western College Publishing.
Also See Complementary:
[R] adjust, [R] lincom, JR] linktest, [R] mfx, [R] predict. [R] testnl, [R] vce, [R] xi
Related:
[R] anova, [R] areg, JR] ensreg,
[R] mvreg, [R] qreg, [R] reg3, JR] regress,
[R] rreg, [R] sureg, [R] svy estimators, [P] _robust Background:
[U] [U] [U] [U]
[R] test,
[R] xtreg, [R] xtregar,
16.5 Accessing coefficients and standard errors. 23 Estimation and post-estimation commands, 23.11 Obtaining robust variance estimates, 23.13 Weighted estimation
1
!
Title
i jknife_-- J_kknife ]
i i
i 'i
,i
estimation
'
i
I
i
i ,I
_
exp_list [if exp] [in ra,ge]
[,
i
N I
,, f
i
Syntax jtmife
"cmd"
[r_class
1 e_class
t n(exp) ]
!
level(#) keep ] expJist
Contains
] i i
newvarnarne = (exp) (exp) eexp
] i i
eexp is specname [eqno]specname specnarne is _b
i I
_b[]
1
_se _se []
eqno is ## /lan'td
Distinguish between [ ], which are to be typed, and [], which indicate optional arguments.
Iscription jknife
performs jack&nile estimation.
cmd defines the statistical command to be executed, cmd must be bound in double quotes. Compound double quotes ("" and "") are needed if the command itself contains double quotes exp_[ist specifies the statistics to be retrieved after the execution of cmd. on which jackknife statistics will lie calculated.
Qptions rclass, eclaSS, and n(exp) specify where crnd saves the number of observations on which it based the calculated results. You are strongly advised tO specify one of these options.
i i
rclass
specifies that cmd saves the number of dbservations in r(N).
ectass
specifies that cmd saves the number of observations in e(N).
n(exp) allows you to specify an}, other expression that evaluates to the number of obser_'ations used. Specifying n(r(N)) is equivalent to spedifying option rclass. Speci_'ing n(e(N)) is equi'falent to specifying option eclass. If cmd saved the number of observations in r(Nl), specify n(_(Ni) ). Y
139
140
Jknife-- Jackknifeestimation
If you specify none of these options, jknife assumes that all observations in the dataset contribute to the calculated result. If that assumption is incorrect, the reported standard errors wilt be incorrect. For instance, say you specify • jknife "regress y xl x2 x3" coef=_b[x2]
and pretend that observation 42 in the dataset has x3 equal to missing. The 42nd observation plays no role in obtaining the estimates, but jknife has no way of knowing that and will use the wrong N. If, on the other hand, you specify jknife "regress y xl x2 x3" coef=_b[x2], e
will correctly notice that observation 42 plays no role. Option e is specified because regress is an estimation command and saves the number of observations used in e (N). When jknife runs the regression omitting the 42nd observation, jknife will observe that e(N) has the same value as when jknife previously ran the regression using all the observations. Thus, jknife will know that regress did not use the observation. jknife
In this example, it does not matter whether you specify option eclass eclass is easier.
or n (e (N)). but specifying
level(#) specifies the confidence level, in percent, for the confidence intervals. The default is level(95) or as set by set level; see [U] 23.5 Specifying the width of confidence intervals. keep specifies that new variables are to be added to the dataset containing the pseudo-values of the requested statistics. For instance, if you typed . jknife "regress y xl x2 x3" coef=_b[x2], e keep
new variable cool would be added to the dataset containing the pseudo-values for _b [x2]. Let b be defined as the value of _b [x2] when all observations are used to estimate the model, and let b(j) be the value when the jth observation is omitted. The pseudo-values are defined as pseudovaluej = N • {b- b(j)} + b(j) where N is the number of observations used to produce b.
Remarks While the jackknife--developed in the late 1940s and earl}, 1950s--is of largely historical interest today, it is still useful in searching for overly influential observations. This feature is often forgotten. In any case, the jackknife is 1. an alternative, first-order unbiased estimator for a statistic; 2. a data-dependent way to calculate the standard error of the statistic and to obtain significance levels and confidence intervals; and 3. a way of producing measures called pseudo-values for each observation, reflecting the observation's influence on the overall statistic. The idea behind the simplest form of the jackknife the one implemented here is to calculate the statistic in question N times, each time omitting just one of the dataset's observations. Write S for the statistic calculated on the overall sample and S(j) for the statistic calculated when the jth observation is removed. If the statistic in question were the mean, then S=
(N - 1)5'(3) + sj N
i
_
jknife-- J_ffe where
estinl_.i_
141
is the value of the data in the jth observation. Solving for sj, we obtain
sj
sj = NS - (N-
1)S(j)
These are the pseudo-values the jackknife calculates, even though the statistic in question is not the mean. The jac_nife estimate is _, the average of the sj's, and its estimate of the standard error of the statistic is the corresponding standard error of the mean (Tukey 1958). The jackknife estimate of variance has been largely replaced by the bootstrap (see [R] bstrap), which is widely viewed as more efficient and robust. The use of iackknife pseudo-values to detect outliers is too o[ten forgotten and is something the bootstrap is unable to provide. See M0stetler and ,Tukey (1977, 133-t63) and Mooney and Duval (1993., 22-27) for more information.
I
JaCkknifedStandard deviation Example Moste!ler and Tukey (1977, 139-140) request a 95% confidence interval for the standard deviation of the eleven v_ilues: 0.t,
0.1,
0,1,
0.4,
0.5,
t.0,
1.1,
1.3,
1.9,
1.9,
4.7
Stata's summarize command calculates the mean and standard deviation and saves them as r (mean) and r (sd), To obtain the jackknifed standard deviation of the eleven values and to save the pseudovalues asia new variable sd, type • i input
x X
1.0.1 2. 0.1 3. 0.1 4. 0.4 5.0.5 6. i.O 7.!.1 8. t.3 9. 1.9
lo. 1.9 11. 12.
4.7 end
j:knife"summarize x" sd=r(sd), r keep command: statistic: n():
summarize x sd=r(sd) r(N)
Variable
Obs
jknife sd
overall
Statistic
i.489364 II
Std. Err.
[95Y,Conf. Interval]
.6244049
.0981028
2.880625
I.343469
Interpreting the, oulpu[, the standard deviation repoded by snmmaxize mpg is 1.34. The jackknife estimate is 1.49 with standard error 0.62. The 95% confidence interval for the standard deviation is .10 to 2.88. By spectfyii_g the keep option, jknife pseudo-valu_es.
creates a new variable in our dataset, sd. for the
• list -' ,
142
jknife-- Jackknife estimation x sd 1. 2. 3. 4. 5. 6. 7. 8. 9, 10. 11.
•I •1 •1 •4 •5 1 i.1 I.3 1.9 1.9 4.7
1.1399778 1.1399778 1,1399778 . 88931508 .8242672 • 63248882 •62031914 •62188884 .8354195 . 8354195 7.7039493
The jackknife estimate is the average of sd,so sd contains the individual "values" of our statistic. We can see that the last observation is substantially larger than the others, The last observation is certainly an outlier, but whether that reflects the considerable information it contains or indicates that it should be excluded from analysis is a decision that must be based on the context of the problem. In this case, Mosteller and Tukey created the dataset by sampling from an exponential distribution. so the observation is quite informative. ,q
> Example Let us repeat the above example using the automobile dataset, obtaining the standard error of the standard deviation of mpg. • use auto, clear (1978 Automobile Data) jknife "summarize mpg" sd=r(sd), r keep command: statistic: n() :
smmmarize mpg sd=r(sd) r(N)
Variable
Obs
Statistic
74
5.785503
Std. Err.
[95_ Conf. Interval]
sd overall jknife
5.817373
,607251
4.607124
Looking at sd more carefully, summarize sd, detail r(sd) pseudovalues
i_ 57 107 25_ 50Z 75_ 90_ 95_ 997
Percentiles 2.870485 2.870485 2.906249 3•328494
Smallest 2.870485 2.870485 2.870485 2.870485
3.948327 6.844408 9.597005 17.34316 38.60905
Obs Sum of Wgt. Mean
Largest 17.34316 19.76169 19.76169 38.60905
Std. Dev. Variance Skewness Kurtosis
74 74 5.817373 5.22377 27.28778 4.07202 23.37822
7.027623
_
jknife-- Jackknifeest_ • ]/istimake mpg
sd if sd > 30
,.pg
_ake 71•
_
143
Diesel
41
sd 38.60@052
Inthi s case,the_v Dieselistheonlydiesel carinourdataset.
q
!Collectingmultiple statistics i>Example : jknife is not limited to collecting just one ;tatistic. For instance, you can use s-T.marize, detail and then obtain the jackknife estimate of _e standard deviation and skewness, m_rnmarize, detail saves the standard deviation in r(sd) and the skewness in r(skewness), so you might type i
• _se (I_78
auto, clear _utomobile
• jkni_e
"summarize
comm_: statistic n():
Data) mpg, detail"
summarize :
sd=r(sd)skew=r(skewness),
r
mpg, detail
sd=r (sd) skew=r (skewness) r(N)
Variable
Obs
Statistic
74
5.78550_
Std. Err.
[95_, Conf.
Interval]
sd overall jknif
e
5. 817379
•607251
4. 607124
7• 027623
.3525056
1. 694686
skew 74
overall
.948717_ 1. 023596
jknife
.3367242
q
!Collectingcoefficients and standard errors Example
, jkni_eCan also collect coefficients and standard errors from estimation commands. For instance, using auto. klta we wish to obtain the jackknife e,_timate of the coefficients and standard errors from a regression in which we model the mileage of a _ar by its weight and trunk space. To do this. we refer io the Coefficients and the standard errors as _b [weight], _b [trunk], _se [weight], and _se [ttrumk] in the exp_list, or we could simplify them by using the extended expressions _b and
i 1
_SO. • Use _uto, clear (1978 iAutomobile Data) • _kniife "reg mpg weight
trunk"
co_iman61:
reg mpg weight
statistic:
b_weight=_b
_b _se, e
trunk
[weight]
se_weight=_se [weight] b_trunk=_b [trunk] b cons=
n() :
b[_cons]
se
Zrtmk=_se
se
cons=_se
_(_)
[trunk] [_cons]
] i
144
jknife -- Jackknife estimation Variable
1
Obs
Statistic
74
-.0056527
Std. Err.
[95X Conf. Interval]
b_weight overall jknife
-.0056325
.O010216
-.0076684
-.0035965
se_weight overall
74
.0007016
jkaife
.0003496
.000111
.0001284
.0005708
b_trunk overall
74
-.096229 -.0973012
jknife
.1486236
-.3935075
.1989052
b_cons overall
74
39.68913
jknife
39.65612
1.873323
35.92259
43.38965
.0218525
.0196476
.1067514
.2690423
.2907921
1.363193
se_trunk overall
74
.1274771
jknife
.0631995
se_cons overall
1.65207
74
jknife
.8269927
q
Saved Results jknife saves in r(): of observations
used in calculating
#
r(N#)
number
r(stat#)
value of statistic # using all observations
statistic
f
r(me_n#) r(se#)
jackknife estimate of statistic # (mean of pseudo-values) standard error of the mean of statistic #
|
Methods and Formulas jknife is implemented
as an ado-file.
References Gould, W. 1995. sg34: Jackknife estimation. Reprints, vol. 4, pp. 165-170.
Stata Technical
Bulletin
24: 25-29.
Mooney, C. Z. and R. D. Duval. Park, CA: Sage Publications.
1993. Bootstrapping:
A Nonparametric
Mosteller, E Company.
1977. Data Amdysis
and Regression.
and J. W. Tukey.
Tukey, J. W. 1958. Bias and confidence 614.
in not-quite
large samples.
Reprinted
Approach Reading.
Abstract
to Statistical
Related:
[R] bstrap, [R] statsby
Background:
[U] 16.5 Accessing coefficients and standard errors, [u] 23 Estimation and post-estimation commands
Inference.
MA: Addison-Wesley
in Annals
Also See
in Stata Technical
of Mat,_ematical
Bulletin
Newbury Publishing
Statistics
29:
rT!tle jointly --- l_orm all p_rwise combinations within groups
Syntax joinby
[varIist] using
nqlabe_. ....... , update
filena,ne
replace
[, _atehed({none
_merge(varname)
] b_oth [ re_aster
using
})
]
DescripUon j oinby joiqs, within groups formed by varlist, observations of the dataset in memory withfiiename, a Stata-format dataset. By join we mean "form all parwise combinations", filename is required to be sorted by varti_t. Iffilename is specified without an extension, '. dta' is assumed. If rarlist isnot specified, joinby memory antt in filename.
takes as varligt the set of variables common to the dataset in
Observations unique to one or the other dataset are ignored unless unmatched () specifies differently. Whether you load one dataset and join the other or vice versa makes no difference in terms of the number of resalting observations. If there ar_ common variables between the two datasets, however, the combined dataset will contain the va]ues from the master data for those observations. This behavior can be modified with the update and replace options.
Options
z
unmatched({llone I both !master I using }) specifies whether observations unique to one of the datasets are to be kept, with the variables from the other dataset set to missing. Valid values are none both m_.stier using
all unmatched observations are ignored (default) unmatched observations from the master and using data are included unmatched obse_'ations from ihe master data are included unmatched observations from the using data are included
}
no!abe! Nevents Stata from copying the value label definitions from the disk dataset into the dataset in memory. Even if you do not specify this option, in no event do label definitions from the disk dataset tephce label definitions already in memory. update i
varies the action that joinby
data_et is tield inviolate--values
takes when an observation is matched. By default, the master from the master data are retained when the same variables are
found in both datasets. If update is specified, however, the values from the using dataset are retained in cases where the master dataset contains missing. i '-
replace,
fillowed with update only, specifies that even when the master dataset contains nonmissin_ values, the_' are to be replaced with corresponding values from the using dataset when the corresp0ndjng values are not equal. A nonmissi_g value, however, will never be replaced with a
missing value. -merge(varname) specifies the name of the variable that will mark the source of the resulting observation. The default is _.merge (_merge) . To preserve compatibility with earlier versions of joir_b_, ___erge is only generated if unmatched is specified. 145
l
Remarks
146 joinby -- Form all pairwise combinations within groups The following, admittedly artificial, example illustrates j oinby.
> Example You have two datasets: child, dta and parent .dta, identifies the people who belong to the s_ne family. .
use
Both contain a family_id
child on
(Data
Children)
describe Contains
data
from
child.dta
obs:
5
Data
vats:
4
13
size:
50
(99.9_
storage variable
name
type
of memory
on Children
Jul
display
value
format
label
variable
family_id
int
_8.0g
Family
child_id
byte
_8.0g
Child
xl
byte
_8.0g
x2
int
_8.0g
by:
Sorted
2000
15:29
free)
label Id Number
Id Number
family_id
list family~d 1025
I.
child_id 3
xl II
x2 320
2.
1025
1
12
300
3.
1025
4
10
275
4.
1026
2
13
280
5.
1027
5
15
210
. use (Data
parent, clear on Parents)
describe Contains obs:
data
from
parent.dta 6
Data
vats:
4
13 Jul
size:
108 storage
(99.9_
of memory
on Parents 2000
15:31
free)
display
value
type
format
label
family_id
int
_8.0g
Family
Id Number
parent_id
float
_9.0g
Parent
Id Number
xl
float
_9.0g
x3
float
_9.0g
variable
Sorted
name
variable
by:
list
1.
fsmily-d 1030
2. 3.
1025 1025
4. 5. 6.
parent_id I0
xl 39
x3 600
11 12
20 27
643 721
1026
13
30
Z60
1026
14
26
668
1030
15
32
684
label
variable which
joinby-- Form ail pairwisecombinationswithin groups
147
You want tO "join" the information for the parents and their children. The data on parents are in memory;the data on children are on disk. dhild.dta has been sorted by family_id, but parerit._ti has not, so first we sort the parent _data on famity_id: • Sort
i family_id
• joinby
family_id
using
child
• describe Co,tails
data
o.bs:I
8
vats:
6 168
Data (99.4_, of memory
free)
i
i
storage
on Parents
,
display
value
type
format
label
family¢id
int
Y,8.0g
Family
Id Number
paz_nt_id
float
Y,9.0g
Parent
Id Number
Xl
float
%9.0g
x3
float
Y.9.0g
child__d
byte
XS.0g
x2
int
Y,8.0g
variable
name
Sorted iby : Npte:
dataset
has changed
variable
Child
since
last
label
Id Number
saved
l_st 1.
family-d 1025
parent_id 12
xl 27
x3 721
child_id 3
2;.
1025
11
20
643
3
320
3. 4.
1025 1025
12 11
27 20
721 643
1 1
300 300
5,
1025
li
20
643
4
275
6. 7.
1025 1026
12 13
27 30
721 760
4 2
275 280
8.
1026
14
26
668
2
280
x2 320
Notice that
I , I
1. fami_y__d of I027, which appears only in child.dta, and family_id only in Narent. dta, are not in the combined dataset. Observations variable(s) are not in both datasets are omitted.
of 1030, which appears for which the matching
2. The x_ v_riable is in both datasets. Values for this variable in the joined dataset are the values from par_nt.dta--the dataset in memory when we issued the joinby command. If we had cMld.d_a in memory and parent.dta on di_k when we requested joinby, the values for xl wouldiha_'e been from child.dta. Values from the dataset in memory take precedence over the datasel o_i disk. q
Methods joinby
Formulas ii implemented as an ado-file.
t
148
joinby-- Formall pairwisecombinationswithin groups
Acknowledgment joinbywas written by Jeroen Weesie, Department of Sociology, Utrecht University, Netherlands.
Also See Complementary:
[R] save
Related:
JR] append, [R] cross, JR] fillin, JR] merge
Background:
[U] 25 Commands for combining data
1
!
-¥itle kappa,
interrater agreement 4
1
i Syntax :
}
kap va_ai_el varname2 varnarne3 [...] [weigh3] [if exp] [in range] kappa i,ariist [if exp] [in range] fweights
i
a_e aliowed; see [U] 14,1.6 weight.
DescriptiOn kap (first s_,ntax)calculates the kappa-statistic measure of interrater agreement when there are two unique raters and two or more ratings. /
kapwgt: defines weights for use by kap in measuring the importance of disagreements. kap (secoqd syntax) and kappa calculate the kappa-statistic measure in the case of two or more (nonuniqu¢) r_atersand two outcomes, more than two outcomes when the number of raters is fixed, and more thah two outcomes when the number of raters varies, kap (second syntax) and kappa produce the same results: they merely differ in how they expect the data to be organized. kap assurrie's that each observation is a subject, varnamel contains the ratings by the first rater, varname2 'by ihe second rater, and so on. kappa also_assumesthat each obse_'ation is a subject. The variables,however, record the frequencies with which r@ngs were assigned. The first variable records the number of times the first rating was assigned, the gecond variable records the number of times the second rating was assigned, and so on.
Options tab displays a tabulation of the assessmentsby the two raters. wgt(wgtid) _pecifies that wgtid is to be used to weight disagreements. User-defined weights can be created using kapwgt: in that case, wgt() specifies the name of the user-defined matrix. For instance, you might define . kapwg_ mine i \, .8 1 \ 0 .8 I \ 0 0 .8 I'
and them . k_p
rgta
ratb,
wgt(mine)
14g
i i
150
kappa -- lnterrater agreement
In addition, two prerecorded
weights are available.
wgt(w) specifies weights 1 - [i -jl/(k - 1), where i and j index the rows and columns of the ratings by the two raters and k is the maximum number of possible ratings. wgt(w2)
specifies weights 1-{(i-
j)/(k-
1)} 2.
absolute is relevant only if wgt () is also specified; see wgt () above. Option absolute modifies how i, j, and k in the formulas below are defined and how corresponding entries are found in a user-defined weighting matrix. When absolute is not specified, i and j refer to the row and column index, not the ratings themselves. Say the ratings are recorded as {0, 1, 1.5, 2}. There are 4 ratings; k = 4 and i and j are still 1, 2, 3, and 4 in the formulas below. Index 3, for instance. corresponds to rating = 1.5. This is convenient but can, with some data, lead to difficulties. When absolute is specified, all ratings must be integers and they must be coded from the set {1,2, 3,...}. Not all values need be used; integer values that do not occur are simply assumed to be unobserved.
Remarks The kappa-statistic measure of agreement is scaled to be 0 when the amount of agreement is what would be expected to be observed by chance and 1 when there is perfect agreement. For intermediate values, Landis and Koch (1977a, 165) suggest the following interpretations: below 0.0 0.00-0.20 0.21-0.40 0.41-0.60 0.61- 0.80 0.81- 1.00
Poor Slight Fair Moderate Substantial Almost Perfect
The case of 2 raters > Example Consider the classification by two radMogists of 85 xeromammograms as normal, benign disease. suspicion of cancer, or cancer (a subset of the data from Boyd et al. 1982 and discussed in the context of kappa in Altman 1991, 403-405). . tabulate Radiologist A's assessment
rada
radb
Radiologist Normal
B's
benign
suspect
cancer
Total
12
0
0
33
benign
4
17
1
0
22
suspect cancer
3 0
9 0
15 0
2 1
29 1
38
16
3
85
Normal
Total
21
assessment
28
Our dataset contains two variables: Each observation is a patient.
rada,
radiologist A's assessment: radb.
We can obtain the kappa measure of interrater agreement
by typing
radiologisl B's assessment.
kap
-- lnterrat_ agreement
151
• kap rada radb ; Agreement i
Expected Agreement
Kappa
Std. Err,
30. 2z 0.472a 0.0694 !
Prob>Z
6.81
0.0000
Had each radiologist made his determination randomly (but with probabilities equal to the overall proportions), _we would expect the two radiologist_ to agree on 30.8% of the patients. In fact, they agreed on 6}.5% of the patients, or 47.3% of the way between random agreement and perfect a_reemenL _he amount of agreement indicates that we can reject the hypothesis that the?, are making their detetrni lations randomty.
Example l
Z
,I
i
There is a difference between two radiologists disagreeing whether a xeromammogram indicates cancer or th_ ,_uspicion of cancer and disagreeing whether it indicates cancer or is normal. The weighted kappa attempts to deal with this. kap provides two "prerecorded" weights, w and w2: . k_p _ada radb, wgt(_;) Ratlng_ weighted by: 1.0¢00 O, 666? 0.61_67 1.0000 O. 3_33 O. 6667 O. od,oo o. 3333
O.3333 0.6667 1.0000 o. 6667
i Expected /{gr¢_em_nt Agreement
l i i I
i !
O. 0000 0.3333 O. 6667 1. 0000
Kappa
Std. Err.
Z
Prob>Z
The w vJe_ghts are given by 1 - li - jt/(k - 1) where i and j index the rows of columns of the ratings by th_ two raters and k is the maxinmm _umber of possible ratings. The weighting matrix
i i
ratings normal, benign, suspicious, and cancerous. i the table. In our "case, the rows and columns of the 4 × 4 matrix correspond to the is prin_ed above A weight Of 1 indicates an observation should count as perfect agreement. The matrix has l s down the dia_ofials!--when both radiologists make the s_me assessment, they are in agreement. A weight of, say_0J66_7 means they are in two-thirds agreement. In our matrix they get that score if they are one aparl -+one radiologist assesses cancer and the other is merely suspicious, or one is suspicious and the o_herisays bemgn, and so on, An emry of 0.3333 means they are in one-third agreement or, } if you prefer,!two-thirds disagreement. That is the gcore attached when they are "two apart". Finally, they are in c_mplete disaueement when the weighi is zero, which happens only when the3, are three apart--one says cancer and the other says normal.: <1
i!
Example The other prerecorded weight is w2 where the weights are given by 1 - {(i• kap ttating_
i
_ada radb,
wgt(w2)
weighted by:
l.oqoo
0.8889
0.5556
0.0000
.s 89 oooo o.88s9 o.sss6 55_56 (),.oclooo. O. 8889 1. 0000 O.8889 s56 o.8889 oooo
j)/(_-
1)}2:
152
kappa -- lnterrater agreement Expected Agreement
Agreement 94.77X
The w2weight
Kappa
84.09X
0.6714
makes the categories
Std.
Err.
0.1079
even morealike
Z 6.22
Prob>Z 0.0000
and is probably inappropriate
here.
> Example In addition to prerecorded weights, you can define your own weights with the kapwgt command. For instance, you might feel that suspicious and cancerous are reasonably similar, benign and normal reasonably similar, but the suspicious/cancerous group is nothing like the benign/normal group: • kapwgt xm i \ .8 1 \ 0 0 1 \ 0 0 .8 1 • kapw_ xm 1.0000 0.8000 1.0000 O. 0000 O.0000 i.0000 O. 0000 O.0000 O.8000 1.0000
You name the weights--we named ours xm and after the weight name, you enter the lower triangle of the weighting matrix, using \ to separate rows. In our example we have four outcomes and so continued entering numbers until we had defined the fourth row of the weighting matrix. If you type kapwgt followed by a name and nothing else, it shows you the weights recorded under that name. Satisfied that we have entered them correctly, we now use the weights to recalculate kappa: . kap rada radb, wgt (xm) Ratings weighted by: I.0000 O.8000 O.8000 I.0000 O.0000 O.0000 O.0000 O.0000
Agreement 80.47)',
O.0000 O.0000 I.0000 O.8000
O.0000 0.0000 O. 8000 1.0000
Expected Agreement
Kappa
52.677.
O. 5874
Std, Err, O.0865
Z
Prob>Z
6.79
O.0000
4
[3 Technical Note In addition to weights for weighting the differences in categories, you can specify Stata's traditional weights for weighting the data. In the examples above, we have 85 observations in our dataset one for each patient. If all we knew was the table of outcomes that there were 21 patients rated normal by both radiologists, etc. it would be easier to enter the table into Stata and work from it. The easiest way to enter the data is with tabi; see [R] tabulate.
) ) ) ( ,
kappa-- lnterrateragreement
!
153
. tabi 21 12 0 0 \ 4 17 I 0 \ 3 9 15 2 \ 0 0 0 I, replace col row
1
2
3
4
Total
1 2
21 4
12 17
0 1
0 0
33 22
3 4
3 0
9 0
15 0
2 1
:29 1
3
85
,(
T_tal
28
Pearson cM2(9)
38 =
16
77.8111
Pr _ 0.000
tabi
felt obligated to tell us the Pearson X2 for this table, but we do not care about it. The important thing is tlaat,with the replace option, tabi left the table in memory: • list )in I/5 row 1 1
col 1 2
pop 21 12
3_
I
3
0
4_ 5,
1 2
4 1
0 4
1, 2.
The variable row is radiologist A's assessment: so assesse_ _ both. Thus, •:kap _ow col [freq=pop] ; Expected Agreement Agreement Kappa : j
Std. Err.
!
63.53'/,
30.827,
col,. radiologist B's assessment: and pop the number
O.4728
O.0694
Z
Prob>Z
6.81
O.0000
If we are going to keep these data, the names row and col are not indicative of what the data reflects. We could (seb [U] 15.6 Dam.set, variable, a,d value labels) • rename row rada • rename col radb . label var rada "Radiologist A's assessment" label var radb "Radiologist B's assessment" . label define assess I normal 2 benign 3 suspect 4 cancer l&be] values rada assess label values radb assess l&be] data "Altman p. 403"
kap's
tab
option, which can be used with or withont weighted data. shows the table of assessments: i
• kap _ada radb
[freq=pop],
Radiolqgist iA's assessment
Radiologist B's assessment normal benign suspect cancer
!
tab
Total
21
n
0
o
33
bez_ign
4
17
1
0
22
Suspect
3
9
15
2
29
cancer ) T_tal
0
0
0
I
1
28
38
18
3
85
_o_mal
)
]:_
Kappa -- mmrramr agreement Expected Agreement
Agreement 63.53_
Kappa
30.82_
Std.
0.4728
Err.
Z
0.0694
Prob>Z
6.81
0.0000
0
Q Technical Note You have data on individual patients. There are two raters and the possible
ratings are I, 2, 3,
and 4, but neither rater ever used rating 3: . tabulate ratera raterb raterb •
ratera
I
2
4
Total
1 2 4
6 5 1
4 3 1
3 3 26
13 11 28
12
8
32
52
Total
In this case, kap would determine the ratings are from the set {1,2, 4} because those were the only values observed, kap would expect a use_defined weighting matrix to be 3 x 3 and, were it not, kap would issue an error message. In the formula-based weights, the calculation would be based on i,j -- I, 2, 3 corresponding to the three observed ratings {1,2, 4}. Specifying the absolute option would make it clear that the ratings are 1, 2, 3, and 4; it just so happens that rating = 3 was never assigned. Were a use_defined weighting matrix also specified, kap would expect it to be 4 × 4 or larger (larger because one can think of the ratings being 1, 2, 3, 4, 5, ... and it just so happens that ratings 5, 6, ... were never observed just as rating -- 3 was not observed). In the formula-based weights, the calculation would be based on i,j -- I, 2, 4. • kap ratera raterb, wgt(w) Ratings weighted by: 1.0000 0.5000 0.0000 0,5000 1.0000 0.5000 0.0000 0.5000 1.0000
Agreement 79.81_
Expected Agreement 57.17Z
Kappa
Z
Prob>Z
4.52
0.0000
Z
Prob>Z
Std. Err.
0.5285
0.1169
. kap ratera raterb, wgt(w) absolute Ratings weighted by: 1.0000 0.6667 0.0000 0.6667 1.0000 0.3333 0.0000 0,3333 1.0000
Agreement
Expected Agreement
81.41Z
55.08X
Kappa 0.5862
Std. Err. 0.1209
4.85
0.0000
If all conceivable ratings are observed in the data, then whether absolute is specified makes no difference. For instance, if rater A assigns ratings { 1,2, 4} and rater B assigns {1,2, 3, 4}, then the complete set of assigned ratings is {1,2, 3, 4}, the same as absolute would specify. Without absolut e, it makes no difference whether the ratings are coded { 1,2, 3, 4}, {0.1.2, 3 }, {1,7, 9, 100}, {0, 1, t.5, 2.0}, or otherwise.
O
kappa-- lnterrateragreement
The case
155:,
more than two raters
In the c,se of more than two raters, the matha aatics are such that the two raters are not considered unique.!Fol " instance, if there are three raters, there is no assumption that the three raters who rate I
the are the the three ratersraters that rate thanflrst_suSject two r_iters case, it same can beasused with two whenthe thesecond. raters' Although identities we vary.call this the more The 'norlunique rater case can be usefully broken down into three subcases: (a) there are two possible raiings which we will call positive and negative; (b) there are more than two possible ratings but _thenumber of raters per subject is the same for all subjects; and (c) there are more than two possiblle ratings and the number of raters per subject varies, kappa handles all these cases. To emphasize that there is no assumption of constant identity of raters across subjects, the variables specified contain counts of the number of raters rating the subject into a particular category.
!
{ i
_ Example (Two; ratings.) Fleiss (1981, 227) offers the following hypothetical ratings by different sets of raters on 25}subjects:
Subject 1 2 3 4 5 6 7 8 9 t0 11 i
l
NO.of No. of raters pos. ratings 2 2 2 0 3 2 4 3 3 3 4 1 3 0 5 0 2 0 4 4 5 5
12 13
34
34
No. of No. of Subject raters pos. ratings 14 4 3 15 2 0 16 2 ' 2 17 3 1 18 2 t 19 4 t 20 5 4 21 3 2 22 4 0 23 3 0 24 3 3 25
2
2
We have entered these data into Stata and the variables are called subject, raters, and pos. kappa, however, re@ires that we specify variables containing the number of positive ratings and negative ratings; that i_s,pos and raters-pos: gen
_eg
kapp_
= raters-pos
pos neg
TWo4ou_comes, Kappa 0.5415
multiple
raters: Z 5.28
Prob>Z 0.0000
We wouldlha_e obtained the same results if we had typed kappa neg pos.
Example (More thanitwo ratings, constant number of raters,) Each of ten subjects is rated into one of three categories by five raters (Fleiss 1981, 230): li_t
I i
156
kappa-- Interrateragreement subject 1. 2. S. 4. 5. 6, 7. 8. 9. 10.
cat1 1 2 3 4 5 6 7 8 9 10
cat2 1 2 0 4 S 1 5 0 1 3
cats 4 0 0 0 0 4 0 4 0 0
0 S 5 1 2 0 0 1 4 2
We obtain the kappa statistic: • kappa earl-cat3 Outcome
Kappa
Z
Prob>Z
catI cat2 cat3
O.2917 0.6711 0.3490
2.92 6.71 3.49
O. 0018 0.0000 O. 0002
combined
0.4179
5.83
O.0000
The first part of the output shows the results of calculating kappa for each of the categories separately against an amalgam of the remaining categories. For instance, the cat I line is the two-rating kappa where positive is carl and negative is cat2 or catS. The test statistic, however, is calculated differently (see Methods and Formulas). The combined kappa is the appropriately weighted average of the individual kappas. Note that there is considerably less agreement about the rating of subjects into the first category than there is for the second. q
> Example Now suppose that we have the same data as in the previous example, but that it is organized differently: • list 1. 2. 3. 4. 5. 6. 7. 8. 9. i0.
subject 1 2 3 4 5 6 7 8 9 i0
raterl 1 1 3 I 1 1 1 2 1 1
In this case, you would kap
use
kap
rater2 2 I 3 1 1 2 I 2 3 1
rater3 2 3 3 1 1 2 1 2 3 1
rather than
kappa:
raterl rater2 raterS rater4 rater5
There are 5 raters per subject: Outcome
Kappa
Z
Prob>Z
1 2 3
0.2917 0.6711 O. 3490
2.92 6.71 3.49
0.0018 0.0000 O. 0002
combined
O. 4179
5.83
O. 0000
rater4 2 3 3 1 3 2 1 2 3 3
rater5 2 3 3 3 3 2 1 3 3 3
_
,
kappa -- Interrater agreement
157
Note that thfe information of which rater is which is not exploited when there are more than two raters.
q
_, Example (More' tha_ two ratings, vmo,ing number of raters!) In this unfortunate case, kappa can be calculated, but there is _o test statistic for testing against _ > 0. You do nothing differently--kappa calculates the total nun{bet of raters for each subject and, if it is not a constant, it suppresses the calculation of test statisttics[ .
1,ist
1,
subject 1
cat 1 1
cat 2 3
2.
2
2
0
3
3.
3
0
0
5
4.
4
4
0
1
5.
5
3
0
2
6.
6
1
4
0
7.
7
5
0
0
8_
8
0
4
1
9;
9
1
0
2
10.
10
3
0
2
• k_pp_
0
catl-cat3 Outcome
Kappa
cat i
O. 2685
cat2
O. 64,57
cat3
O. 2938
combined note:
cat3
Z
Prob>Z
O. 3816
}Number of ratings
per
subject
vary;: cannot
calculate
test
Istatistics,
q
Example This case _s similar to the previous example, but the data are organized differently: • list
I.
_ubject i
raterl I
rater2 2
rater3 2
2.
2
3.
3
4.
1
1
3
3
3
3
3
3
3
3
4
1
1
t
1
3
5-
5
1
1
1
3
3
6. 7.
6 7
1 1
2 1
2 1
2 1
2 1
8.
8
2
2
2
2
3
9.
9
1
3
10.
10
1
t
1
3
In this
cas_,
|
we
specify
kap,
instead
of
kappa:
rater4
rater5 2
3 3
i i
158
kappa -- Interrater agreement • kap raterl-rater5 There are between 3 and 5 (median = 5.00) raters per subject: 0utcome
Kappa
1 2 3
0.2685 0.6457 0.2938
Prob>Z
0.3816
combined note:
Z
Number of ratings per subject vary; cannot calculate test statistics.
Saved Results kap and kappa save in r(): Scalars r(N)
number
of subjects
(kap only)
r(prop_o) observed proportion of agreement (kap only) r(prop_e)expected proportion of agreement (kap only)
r(kappa)
kappa
r(z) r(se)
z statistic standard error for kappa statistic
Methods and Formulas kap, kapwgt,
and kappa
are implemented
as ado-files.
The kappa statistic was first proposed by Cohen (1960). The generalization for weights reflecting the relative seriousness of each possible disagreement is due to Cohen (1968). The analysis-of-variance approach for k = 2 and ra _> 2 is due to Landis and Koch (1977b). See Altman (1991, 403-409) or Dunn (2000. chapter 2) for an introductory treatment and Fleiss (198t, 212--236) for a more detailed treatment. All formulas below are as presented in Fleiss (1981). Let rn be the number of raters and let k be the number of rating outcomes.
kap: m = 2 Define wij (i = 1.... , k, j = 1,..., k) as the weights for agreement and disagreement (wgt ()) or, if not weighted, define wiz = 1 and wij = 0 for i ¢ j. If wgt (_r) is specified, u'ij -- 1-l i-jt/(k1). If wgt (_r2) is specified, wij -- 1 - {(i-j)/(k The observed proportion of agreement
1)} 2.
is k
k
Po = _ _ wijPij i=1 3=1
where Pij is the fraction of ratings i by the first rater and j by the second. The expected proportion of agreement is k 'De-
_ i=1
k _wijPi-P.j j=l
Ii
_
-- Interrater agreement
159
where Pi. = ___jPij and p.j = Y'_-iP/J" f Kappa is _iven by _ = (Po - Pe) / (I - Pe). The standard error of _ for testing against 0 is
s0 (1-
j
where n is th, number of subjects being rated and Ni. = _j statistic Z= _/'so is assumed to be distributed N(0, 1).
p.jwij
and ¥.j = _i Pi':wiJ" The test
kappa: m > 2,!k = 2 Each sUbjeCt i, i = 1,...,
n, is found by' xi of mi raters to be positive (the choice as to what is
labeled:positiVe being arbitrary). The overail proportion of positive ratings is _ = _i xi/(nN), between-s_bj@s mean square is (approximately)
B
where _
= _-_i rni/n.
_
The
!1 1
--
n
t
i
r°'i
and the w_thla-subject mean square is
W = n(_--
1 1i
E i
xi(mimi
xi) ] i
Kappa is thent defined
i
i
The standard !error for testing against 0 (Fleiss arid Cuzick 1979) is approximately calculated as 1 _'0 = (N.-
1)_
/
(_-
(2(_H z 1)+
_H)(1
equal to and
- @-_)
mpq
ii
i
where Nh, is the harmonic mean of rni and _ = 1 - _. The test st',ttistic Z = "/^_/Sois assumed to be distributed N(0, 1).
i
kappa: m >2,ik> 2 Let .rij be ithe number or ratings on subject i, i = 1,...,
n, into category j, j = 1,...,
k. Define
i
_j as the overall proportion of ratings in category j, _j = 1 - _._, and let _j be the kappa statistic given above f_r k = 2 when category j is compared with the amalgam of all other categories. Kappa
I
is (Landis an_ Koch 19778)
_=
A____PJ-qjr'JJ
t
160 kappa m lnterrater agreement In the case where the number of raters per subject _j xij is a constant m for all i, Fleiss, Nee, and Landis (1979) derived the following formulas for tile approximate standard errors. The standard error for testing
_j against
0 is
/ V /
and the standard
error
for testing
1)
g is
_= _. _f_jx/nm(m-J
2
i)
PJqJ
j
PJ'qJ('qJ-
_j)
References Altman, D. G. t991. Practical Statistics for Medical Research. London: Chapman & Hall. Boyd, N. F., C. Wolfson, M. Moskowitz, T. Carlile, M. Petitclerc, H. A. Ferri, E. Fishell. A. Gregoire, M. Kieman. J. D. Longley, I. S. Simor, and A. B. Miller. 1982. Observer variation in the interpretation of xeromammograms. lournaI of the National Cancer Institute 68: 357-63. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20: 37-46. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement Psychological Bulletin 70: 213-220.
or partial credit.
Dunn. G. 2000. Statistics in Psychiatry. London: Arnold. Fleiss, J. L. 1981. Statistical Methods for Rates and Proportions. 2d ed. New York: John Wiley & Sons. Fleiss, J. L. and J. Cuzick. 1979. The reliability of dichotomous judgments: unequal numbers of judges per subject. Applie 4 Psychological Measurement 3: 537-542. Fleiss, J. L., J. C. M. Nee, and J. R. Landis. 1979. Large sample variance of kappa in the case of different sets of raters. Psychological BuIletin 86: 974-977. Gould, W. 1997. stata49: Interrater agreement. Stata Technical Bulletin 40: 2-8. Reprinted in Stata TechnicaJ Bulletin Reprints, rot. 7, pp. 20-28. Landis, J. R. and G. G. Koch. 1977a. The measurement of observer agreement for categorical data. Biometrics 33: 159-174. 1977b. A one-way components of variance model for categorical data. Biometdcs 33: 671-679. Steichen, T. J. and N. J. Cox. 1998a. sg84: Concordance correlation coefficient. Stata Technical Bulletin 43: 35-39. Reprinted in Stata Technical Bultetin Reprints, vol. 8, pp. 137-143. 1998b. sg84.t: Concordance correlation coefficient, revisited. Stata Technical Bulletin 45: 21-23. Reprinted in Stata Technical Bulletin Reprints, vol. 8, pp. 143-145. 2000. sg84.2: Concordance correlation coefficient: update for Stata 6. Stata Technical Bulletin 54: 25-26. Reprinted in Stata Technical Bulletin Reprints, vol. 9. pp. 169-170.
Also See Related:
[R] tabulate
i
Title !
i
• kdenslty
Univariate kernel density estimation
i
Syntax kdensity varname [weight] [ifexp][inrange] [, nJ _raphgenerate(neuwarznewvardinsity) n(#) _width(#) [ ibiweight I cQsineI eppanlgausl] parzenl rectangle I triangle ] ! n(_rmal stud(#) at(varz) s_ymbol(...) _connect(...) title(string) [ [
fweigh_s
ggoph_optwns ? i J i and _weights are allowed;see [U] 14.1.6weighi.
Descriptien kdensity!produces
kernel density estimates and graphs the result. /
Options i
nograph suppresses the graph. This option is often Used in combination with the generate () option. generate(n_wvarz newvardensitv) stores the results of the estimation, newvardensity will contain
i t_ I
the densit>l estimate, newvarz will contain the pbints at which the density is estimated. n(#) specifie_ the number of points at which the d_nsity estimate is to be evaluated. The default is min(N; 501, where N is the number of observations in memory.
i i ! i
width(#) sNcifies the halfwidth of the kernel, the width of the density window around each point. If w() is hot specified, then the "optimal" width is calculated and used. The optimal width is the width [hat would minimize the mean integrated square error if the data were Gaussian and a Gaussiar_ kernel were used, so is not optimal in any global sense. In fact, for multimodal and highly skeived densities, this width is usually too wide and oversmooths the density (Silverman
i
1986). bi_reight,
I cbsine,
default, e_t_, [ l i
I i
i
gauss,
parzen,
rectangle,
and triangle
specify the kernel. By
specifying the Epanechnikov kernel, is used.
normal requd?ts that a normal density be overlaid on the density estimate for comparison. stud(#)
i.
speOfies that a Student's _ distribution with # degrees of freedom be overlaid on the density
estimate f_r comparison, at(varz) specifies a variable that contains the v_lues at which the density should be estimated. This optiot0 allows you to more easity obtain density estimates for different variables or different subsamples of a variable and then overlay the e_t_mated densmes for comparison. symbol(...)
i
epan,
!is graph,
is symbollo);
two,ray's symbol()
see [G]graph options.
°pti°h for specifying the plotting symbol. Tile default :
i
(
connect(...)isgraph, twoway'sconnect ()estimation optionforhow pointsareconnected. The default is 162 kdensity -- Univariate kernel density connect (1), meaning points are connected with straight lines: see [G] graph options. title(string) is graph, twoway's title() option for speci_'ing the title. The default title is "Kernel Density Estimate"; see [G] graph options. graph_options
are any of the other options allowed with graph,
twoway; see [G] graph
options.
Remarks Kernel density estimators approximate the density f(z) from observations on z. Histograms do this too, and the histogram itself is a kind of kernel density estimate. The data are divided into nonoverlapping intervals and counts are made of the number of data points within each interval. Histograms are bar graphs that depict these frequency counts the bar is centered at the midpoint of each intervalwand its height reflects the average number of data points in the interval. In more general kernel density estimates, the range is still divided into intervals and estimates of the density at the center of intervals are produced. One difference is that the intervals are allowed to overlap, One can think of sliding the intervalPcaUed a window along the range of the data and collecting the center-point density estimates. The second difference is that, rather than merely counting the number of observations in a window, a weight between 0 and 1 is assigned--based on the distance from the center of the window and the weighted values are summed. The function that determines these weights is called the kernel. Kernel density estimates have the advantages of being smooth and of being independent choice of origin (corresponding to the location of the bins in a histogram).
of the
See Salgado-Ugarte, Shimizu, and Taniuchi (1993) and Fox (1990) for discussions of kernel density estimators that stress their use as exploratory data analysis tools.
Example Goeden investigate histogram. is roughly
(1978) reports data consisting of 316 length observations of coral trout. We wish to the underlying density of the lengths. To begin on familiar ground, we might draw a In [G] histogram, we suggest setting the bins to min(v/-_. 10-loglcn ). which for n = 316 18:
graph length, xlab ylab bin(18)
2-
15"
05"
m 0
length
kdensity -- Univariatekernel density estimation
163
The kernel density estimate, on the other hand, is smooth. . kdens_ty
length,
xlab
ylab
006 -_
004
121
£,02
i
\
1
o
Kernel
Density
tdngth Estimate
Kernel den_ity)stimators are. however, sensitive to an assumption just as are histograms. In histograms, we specify a _umber of bins. For kernel density estimators, we specify a width. In the graph above, we used the d_fault width, kdensity is smarter than graph, histogram in that its default width is not a fixed _:onstant. Even so, the default width is not necessarily bei. i kder_sity !ayes the width in the return scalar width, so typing display Doing this, wd discover that the width is approximately 20.
r(width)
reveals it.
! i
Widths are(ketail. isimilarThe to units the inverse of thearenumber of ofz, bins in histogram in analyzed. that smaller provide more of the width the uhits the avariable being The widths width is specified as ia halfwidth, meaning that the kernel density estimator with halfwidth 20 corresponds to sliding a w!ndow of size 40 across the data.
I
We can specify halfwidths for ourselves using the t the density as imuch. • kdens_ty
length,
epan
width()
i ]
option. Smaller widths do not smooth
w(lO)
xlab ylab
i
I
]
OOB I
oo6_I
\
/
,004
/
e
_oo
5
_Jo
,oo I_ng(h
Kernel
Density
Estimate
s_o
do
• kdensity length, epam xlab ylab w(15)
164
kdensity -- Univariate kernel density estimation
•
006
/_
.004 >.
j
0 200
"\ 3(_0
Kernel
4(_0 length
Density
560
6;0
Estimate
q
> Example When widths are held constant, different kernels can produce surprisingly different results. This is really an attribute of the kernel and width combination; for a given width, some kernels are more sensitive than others at identifying peaks in the density estimate. We can see this when using a dataset with lots of peaks. In the automobile dataset, we characterize the density of we £ght, the weight of the vehicles. Below, we compare the Epanechnikov and Parzen kernels. kdensity weight, epan nogr g(x epan) kdensity weight, parzen nogr g(x2 parzen) • label vat epan "Epamechnikov Density Estimate" • label vat parzen "Parzen Density Estimate" • gr epan parzen x, xlab ylab c(ll)
o Epanechnikov .0008
Density
Estimate
_ Parzen
Density
Estimate
"_
oooo
i'll!
0
1ooo
2o'oo
3ooo Weight
(l_s.)
4otoo
5ooo
!
kdensRy-_ Univariatekerneldensityestimation
165
!
We did not s_ecify a width and so obtained the d_fault width. That width is not a function of the selected kerneil but of data. See the Methods and Formulas section for the calculation of the optimal
I
width.
q
> Example In examining the density estimates, we may wi_h to overlay a normal density or a Student's t • ari_" Mng automobile weights, we can get an idea of the distance from normality density for col_p _u,,. U__ with the normal option. , kdens_ty
weight,
epam
normal
xlab ylab
,0006
i
.ooo4
,ooo2 t
°t 1 1000
il 2000
I 3_00 Weigh!
Kernel
Density
I 4000
5000
(}bs,)
IEstimate
Example Another conmon desire in examining density estimates is to compare two or more densities. In this example, _,e will compare the density estimatesof the weights for the foreign and domestic cars. I
kdensi_y
i
.• kdensi_y kdens@y
i
weight,
negr
weight weight
gen(x
fx)
if gen(f_O) if foreign==O, foreigxl==l, nogr nogr gen(fXl)
label
_ar fxO
"Domestic
label
_ar fxl
"Foreign
at(x) at(x)
cars" cars"
(Continued on twxt page)
"
166
• gr
fxO fxl c(ll) s(TS) xlab ylab kdensity --x, Univariate kernel density estimation
i :
Domestic
cals
D Foreign
cars
OOl.
{
I !
.0005
l
_a_
fX
'
0" 1000
20100
3000' Weight
40t00
5000r
(Ibs.)
<1
0 Technical Note Although all the examples we included had densities
less than I. the density may exceed t.
The probability density f(z) of a continuous variable z has the units and dimensions of the reciprocal of z. If z is measured in meters, f(x) has units 1/meter. The density is thus not measured on a probability scale, so it is possible for f(x) to exceed I. To see this, think of a uniform density on the interval 0 to 1. The area under the density curve is 1: this is the product of the density, which is constant at 1, and the range, which is 1. If then the variable is transformed by doubling, the area under the curve remains 1. and is the product of the density, constant at 0.5, and the range, which is 2. If conversely the variable is transformed by halving, the area under the curve also remains at 1, and is the product of the density, constant at 2, and the range, which is 0.5. (Strictly, the range is measured in certain units, and the density in the reciprocal of those units, but the units cancel on multiplication.) D
Saved Results kdensity saves in r(): Scalars r(width) kernel bandwidth r(n) number of points at which the estimate was evaluated r(scale) density bin width Macros r(kernel)
name of kernel
g
i
kdensity_ Univariatekerneldensityestimation
167
Methods iar,d Formulas kdensit_r is implemented as an ado-file. A kernel density estimate is formed by summihg the weighted values calculated with the kernel
}
function K is in
= nh"i=1 --
l
:
where we may define various kernel functions, kdens±ty includes seven different kernel functions, The Epanec_nikov is the default function if no Otherkernel is specified and the most efficient in minimizing _he mean integrated squared error. Kernel
l i
FOrmula
Biweight
K[z] =
ig 0ta(1
Cosine
K[z] = {1 + eds(27rz) 0
z2) _
= i
i !
l i
1I
g 0 { 3(1
lz2]/x/_
if lzl < 1/2 otherwise iflzl
Epanechnikov
K[z]
Gaussian
K[z]=
Parzen
K[z] =
8(1 _ [z[)3/3 { 04 _ 8_zz + 8[z13
if 1/2 < !z[ < 1 if [z[ _<1/2 otherwise
Rectangular
K[z]
,,f1/2 t 0
if Iz! < 1 otherwise
0
otherwise
l
_
,,
if Izl < 1 otherwise
_-z_ /2
Triangular K[z] = { 1 -[z I if Izi < 1 From the definitions given in the table one can see that the choice of h will drive how many values are included in estimating the density at each point. This value is called the window width or bandwidth I_fthe window width is not specified, then it is determined as
m = rain i
h=
varian_,
interquartile 1.349 rang%
0.9m
1
i i
.. i
)
i
where :r is tile variable tbr which we wish to estimate thekernel and n is the number of observations. Most researchers agree that the choice of kernel is not as important as the choice of bandwidth,
i
i
There is a grea_deal of literature on choosing bandwidths under various conditions: see. for example, Parzen (196_) or Tapia and Thompson (1978}. Als6 see Newton (1988) for a comparison with sample
i i
i
spectral den, itv estimation in time-series applications.
i
i
Acknowledgments 168 kdensity -- Univariate kernel density estimation We gratefully acknowledge the previous work by Isa/as H. Salgado-Ugarte Autonoma de Mexico. and Makoto Shimizu and Tom Taniuchi Ugarte, Shimizu. and Taniuchi (1993), Their article provides subject of univariate analysis.
kernel
density
estimation
and presents
of Universidad
Nacional
of the University of Tokyo; see Salgadothe reader with a good overview of the arguments
for its use in exploratory
data
References Fox, J. 1990. Describing univariate distributions. In Modem Methods of Data Analysis. ed. J. Fox and J• S. Long, 58-125. Newbury Park, CA: Sage Publications. Goeden, G. B. t978. A monograph of the coral trout, Plectropomus leopardus (Lac6pbde). Res. Bull. Fish• Serv. Queens1. 1:42 p. Newton, H. J. t988, TIMESLAB: A Time Series Analysis Laboratory. Belmont, CA: Wadsworth & Brooks/Cole. Parzen, E. 1962. On estimation of a probability density function and mode. Annals of Mathematical Statistics 32: 1065-1076. Salgado-Ugarte, I. H., M. Shimizu, and "12,Taniuchi. 1993. snp6: Exploring the shape of univariate data using kernel density estimators. Stata Technical Bulletin 16: 8-19. Reprinted in Stata Technical Bulletin Reprints, vol. 3, pp. 155-173. 1995. snp6•t: ASH, WARPing, and kernel density estimation for univariate data. Stata Technical Bulletin 26: 23-31. Reprinted in Stata Technical Bulletin Reprints, vol. 5, pp. 161-172. 1995. snp6.2: Practical rules for bandwidth selection in univariate density estimation. Stata Technical Bulletin 27: 5-19. Reprinted in Stata Technical Bulletin Reprints, vot. 5, pp, 172-190. • 1997. snpl3: Nonparametric assessment of multimodality for univariate data. Stata Technical Butletin 38: 27-35. Reprinted in Stata Technical Bulletin Reprints, vol. 7, pp. 232-243. Scott, D. W. I992. Multivariate Density Estimation: Theory, Practice, and Visualization. New York: John Wiley & Sons, Silverman, B. W. 1986. Density Estimation for Statistics and Data AnaIysis. London: Chapman & Hall. Simonoff, J. S. 1996. Smoothing Methods in Statistics. New York: Springer-Verlag, Steichen, T. J. 1998. gr33: VioIin plots. Smm Technical Butletin 46: 13-18. Reprinted in Stata Technical Bulletin Reprints, vol. 8, pp. 57-65, Tapia, R. A. and J. R. Thompson. 1978. Nonparametric Probability Density Estimation. Baltimore: Johns Hopkins University Press. Wand, M. P and M, C. Jones. 1995. Kernel Smoothing. London: Chapman and Hall.
Also See Related:
[R] hist
Background:
Smta Graphics
Manual
Title I
ksm
--
Sm°°thing
iJ_cluding
l°v_;ess
i
Syntax ksm 3,v,tr xvar [ifexp] ]
[inrange]
[, line weight !o,wessb_wwidth(#) loogit
a_dj_stgenerate(newvar)nograph graph_options ]
i
Description
! ii
ksm car'ies out unweighted and locally weighted smoothing of yvar on .n,ar, displays the graph, and optionally saves the smoothed variable. Among ksm's capabilities are lowess (robust locally weighted r,:gression, Cleveland 1979). See Cleveland (1993, 94-101) for a discussion of lowess.
i i
%Vsmini: ksm is computationally intensive and may therefore take a tong time to run on a slow computer. Lowess calculations on 1,000 observations, for instance, require estimating 1.000
l
regressions_.
i
Options line speci_es running-line least-squares smoothing; the default is running mean. weight
specifies use of Cleveland's (1979) tricube weighting function: the default is unweighted.
lowess is_equivatent to specifying line I smootheir. bwidt;h(#)i
weight
and requests Cleveland's
specifies the bandwidth. Centered subsets of bwidth.
lowess running-line
3\r obse_'ations
are used for
} i
calcula@g smoothed values for each point in the data except for the end points, where smaller, uncente_d subsets are used. The greater the bwidth, the greater the smoothing. The default is
i i
0.8, logit transforms the smoothed war into logits. Predicted values less than .000t or greater than .9999 ar_ set to I/N and 1 - l/N. respectively, before taking logits, adjust adiusts the mean of the smoothed war to equal the mean of yvar by multiplying by an appropriate factor. This is useful when smoothing binary (0/1) data. i generate (inewvar) creates newvar containing the smoothed values of yvar in addition to or instead of displaying the graph. i t
t
i
nograph s_ppresses displaying the graph. ,eraph_optiSns are any of the options allowed wiih graph, 1{69
twoway; see [G] graph options.
] i
i ] i i ]
170
,,
ksm-
Smoothing including Iowess
Remarks The most common use of ksm is to provide lowess--locally weighted scatterplot smoothing. The basic idea is to create a new variable (newvar) that, for each yvar Yi, contains the corresponding smoothed value, The smoothed values are obtained by running a regression of war on xvar using only the data (zi, yi) and a small amount of the data near the point. In lowess, the regression is weighted so that the central point (zi,Yi) gets the highest weight and points farther away (based on the distance tzj - zil) receive less weight. The estimated regression is then used to predict the smoothed value _'i for Yi only. The procedure is repeated to obtain the remaining smoothed values, which means that a separate weighted regression is estimated for ever 3, point in the data. Lowess is a desirable smoother because of its localitymit tends to follow the data. Polynomial smoothing methods, for instance, are global in that what happens on the extreme left of a scatterplot can affect the fitted values on the extreme right.
Example The amount of smoothing is affected by the bwidth different values. For instance. • ksm hl
depth,
lowess
ylab
Lowess
xlab
and you are warned
s(Oi)
smoother,
bandwidth
= .8
14 o co
o
o
oo 0
12
o
o
-
o
0 o
000
°o
% 10
O0
0
0 0
0
_:
0
/i,
o
O0
_/
cO cOo
o o
o
8 o
o o
o
o
co
o
6
0
,60
26° depth
Now compare that with
(Continued
on next page)
360
,;o
to experiment
with
r i
,.o_
,,,
]
. ksmlhl depth, lowess ylab xlab s(Oi) bWidth(.4) Lowess
smoother,
bandwidth=
.4
i
o cl_
o
___oo
0
10
O0
_
0
0 0
O 0
0
o o oo fc_
o
eo
o°o
0
0
0%00//" o_ ooo
/ 0
C
0
O
0
oo
0
8
I c
c
o
o
co
6
I 300
i
l
4{_0
depth
In the first c _se,the default bandwidthof 0.8 is used, meaning that 80% of the data is used in smoothing each ,p°int- In the second case, we explicitly specifieda bandwidth of 0.4. Smaller bandwidths follow the ofigmalldata more cteselv.
Example !
Two ks_ options are especially useful with binary (0/1) data: adjust and logit, adjust adjusts the resultin_ curve (by multiplication) so that themean of the smoothed values is equal to the mean
l l
of the unsm_oothedvalues, logit specifies the smoothed curve is to be in terms of the log of _he odds ratio: i
i
. ksmiforeign mpg, lowess ylab xlab jitter(5) Lowess
smoother,
bandwidth
adjust
= .8
75
_//
& • o u.
5 /
i
// 25
/P--/
,
c
i i
1
,
X(o ,o ,o
o 2;
_ o
M_)leage (mpg)
_'o
,;
I
• ksm
f
172
foreign
mpg,
lowess
ylab
xlab
logit
ksm -- Smoothing LowesS including smoother, Iowess bandwidth
yline(O)
= .8
)l !
_g,
o
/ /t-
_
/
o,
/
/
/ -4
lo
2'0
3_o Mileage
4_
(mpg)
With binary data, if you do not use the logit option, it is a good idea to specify graph's jitter() option; see [G] graph options. Since the underlying data (whether the car was manufactured outside the United States in this case) take on only two values, raw data points are more likely to be on top of each other, thus making it impossible to tell how many points there are. graph's jitter() option adds some noise to the data to shift the points around. This noise affects only the location of points on the graph, not the lowess curve. When you do specify the logit
option, the display of the raw data is suppressed. <1
Q Technical Note ksm can be used for other than lowess smoothing. Lowess can be usefully thought of as a combination of two smoothing concepts: the use of predicted values from regression (rather than means) for imputing a smoothed value and the use of the tricube weighting function (rather than a constant weighting function), ksm allows you to combine these concepts freely. You can use line smoothing without weighting (specify line), or mean smoothing without weighting (specify no options), or mean smoothing with tricube weighting (specify weight). Specifying both weight and line is the same as specifying lowess. Q
Methodsand Formulas ksm is implemented
as an ado-file.
Let Yi and z_ be the two variables and assume the data are ordered i = 1.... , N 1. For each Yi, a smoothed value _t_ is calculated.
_'
so that z_ _< zi+l
for
The subset used in calculation of fl_ is indices i_ -- max(l, i k) through i.__= min(i + k, N), where k - L(N.bwidth-0.5)/2j. The weights for each of the observations between j -- i ...... i+ are either 1 (default) or the tricube (weight):
ksm -- Smoothingincludingi_
i
1_
where A = 1.000t max(z+ - x_, x, - z_). The smoothed value y] is then the (weighted) mean or the (weighted) regression prediction (line). !
Ackledgrnent
]
ksm was written by Patrick Royston of tile MRC Clinical Trials Unit, London.
i
ReferenCes ! Chamber_,J. M., W. S. Clevi_land,B. Kleiner,and E A. Tukey.I983. GraphicalMethodsfor Data AnalySis.Belmont, CA: WadsworthInternatiOnalGroup. Clevetan&W. S. 1979. Robust locally weightedregressionand smoothingscatterplots.Journal of t_ Amer/can Staris_catAssocia6on 74i 829-836. , 1993. VisualizingData. Summit,NJ: HobartPress. ............ . 1994. The Elementsdf GraphingData. Summit,NJ! HobartPress. Goodatl,i'C. 1990.A surveyOfsmoothingtechniques.In ModemMethodsof Data Analysis. ed. L Fox and J. S. Long, 126-!76. NewburyPark;CA: Sage Publications. Royston, P. 199t. gr6: Lowess smoothing.Stata TechnicalBulletin 3: 7-9. Reprinted in Stata TechnicalBulletin
i
Re ,s, voL1,or.41- 4.
Salgado'Ugarte,1. H. and M. Shimizu.1995.snp8: Robusiscatterplotsmoothing:enhancementsto Stata's ksm. Staia TechhicalBulletin25: 23-26. Reprintedin Stata TechnicalBulletin Reprints,vol, 5, pp. 190-194. Sasieni.P. 1994. snpT:Naturalcubic splines.Stata TechnicalBulletin22: 19-22. Repnntedm Stata TechnrcalBulletin Reprints,vol. 4, pp. 17b-174.
AlSo [
!
Related:
[R] ipOlate, [lk] smooth
BackgrOund:
Stata Graphics Manual
I i(le
ksmirnov -- Kolmogorov-Smimov I
equality of distributions I test
II
]
Syntax ksmirnov
varname = exp [if exp] [in range]
ksmkrnov varname [if exp] [in range], by(groupvar)
[ e_xact ]
Description ksmirnovperforms one- and two-sample Kolmogorov-Smirnov tests of the equality of distributions. In the first syntax, varname is tile variable whose distribution is being tested and exp must evaluate to the corresponding (theoretical) cumulative. In the second syntax, groupvar must take on two distinct values. The distribution of varname for the first value of groupvar is compared with that of the second value. When testing for normality, please see [R] sktest and [R] swilk.
Options by (groupvar) is not optional, in the second syntax. It specifies a binaD' variable that identifies the two groups. exact specifies the exact p-value is to be computed. This may take a long time if n > 50.
Remarks D Example You have data on x that resulted from two different experiments, labeled as group==1 group==2. Your data contain
and
list group 1. 2. 3. 4. 5. 6, 7.
x 2 0 3 4 5 8 10
2 1 2 1 1 2 2
You wish to use the two-sample Kolmogorov-Smirnov in the distribution of z for these two groups:
test to determine if there are any differences
- ksmirnov x, by(group) Two-sample Kolmogorov-Smirnov Smaller group 1: 2: Combined K-S :
D O.5000 -0. 1667 0,5000
test for equality of distribution P-value
Corrected
O.424 O.909 O. 785
O.735
174
functions:
ksmimov-- Kolmogorov-Smirnov equalityof distributions test
175
Thefirst lineteststhe hypothesis thatx for group] containssmaJtervaluesthangroup2. Thelargest difference between the distribution functions is 0.5. The approximatep-value for this is 0.424, which is not fignificant. The second line tests the hypothesis that x for group 1 contains Iarger values than group 2. The largest_ difference between the distribution p-vaLuefor this small difference is 0.909. functions in this direction is 0.1667. The approximate Finally,the approximate p-value for the combined test is 0.785, corrected to 0.735. The p-values ksmirnov calculates are based on the asymptotic distributions derived by Smimov (1939). These approximations are not very good for small samples (n < 50). They are too con_servative--real p-values tend to be substantially smaller. We have also included a less conservative approximationfor the nonidirectionalhypothesis based on an empirical continuity correction. That is the 0.734 reported in the third column• That number, too, is only an approximation. An exact value can be calculated using the exact option: • ksmirnov x, by(group) exact Two-sample Kolmogorov-Smirnov test for eo_ualityof distribution functions: Smaller group D P-value Exact 1;: 2: Combined K-S :
O.5000 -0.1667 O.5000
O. 424 0.909 O.785
O.657
I> Example Lefs now test whether x in the example above is distributed normally. Kolmogorov-Smirnov is not a particularly powerful test in testing for normality and we do not endorse such use of it; see [R] sktest and [R] swilk for better tests. In any case. we will test against a normal distribution with the same mean and standard deviation: • Summarize x Variable
Obs
Mean
Std. Dev.
x 7 4.571429 3.457222 _smirnov x = norm((x-4.571429)/3.457222)
Min
Max
0
10
One-sample Kolmogorov-Smirnov test against theoretical distribution norm( (x-4.571429)/3.457222) Smaller group x: Cumulative : Combined K-S:
D O. 1650 -0.1250 O.1650
P-value
Corrected
0.683 O. 803 O,991
O. 978
Since Stata has no way of knowing that you based this calculationon the calculated mean and standard deviation of x. the test statistics will be slightly conservative in addition to being approximations, Nevertheless. they cleartv indicate that the data cannot be distinguished from normally distributed data. q
]
, .v
i
rtoJls.
Jwvw
--
._v_lv_jvzvv--_.TlUnllUlVV
V_lMClllty
UI
Ul_itrlDU[lOrl$
[eS[
Saved Results ksmirnov
saves in r():
Scalars _
r(D_l)
D from line l
r(D)
combined D
r(p_l)
p-value from line 1
r(p)
combined ;,-value
r(D._2) r(p_2)
D from line 2 p-value from line 2
r(p_exact)
combined significance (x 2 or exact)
name of group from line 1
r(group2)
name of group from line 2
Macros r(groupl)
Methodsand Formulas ksmirnov is implemented as an ado-file. In general, the Kolmogorov-Smimov test (Kolmo_orov 1933; Smirnov 1939; also see Conover 1999, "Statistics of the Kolmogorov-Smimov type", 42)-465) is not very powerful against differences in the tails of distributions. In return for this, it is f_irly powerful for alternative hypotheses that involve lumpiness or clustering in the data. The directional hypotheses are evaluated with the statistics
where Fix ) and G(x) are the empirical distribution t_unctions for the sample being compared. The combined statistic is
The p-value for this statistic may be obtained by evaluating the asymptotic limiting distribution. Let m be the sample size for the first sample, and let rt be the sample size for the second sample. Smirnov (1939) shows that C_
lira rr_, _z_--40G
Pr{v/mn/(m.
n)Dra,n< z} = i-
2'
i=1
(-
1) i-1 exp (-
2i2z 2)
The first 5 terms form the approximation Pa used b_ Stata. The exact p-value is calculated by a counting algorithm; see Gibbons (1971, 127-131). A Corrected p-value wa_ obtained by modifying the asymptotic p-value using a numerical approximatic_n technique Z = ¢-1 (Pa)+
1.04/rain(m,
n)+ 2.09i/max(re,
p-value =
n)-
].35/v/_m/(m
+ n)
IL ksmimov
•
Kolmogorov-Smirnov
equality of dJstdbution,s test
177
References Conover, W. J. 1999. Pr_cOcaINonparameNc Statistics. 3d ed. New York: John Wiley & Sons. Gibb0n_, J. D. Y971.Nonparametric S_6srical Inference. New York: McGraw-Hill. Kolrn_g0rov, A. N. 1933. Sulla determinazione empirica di una legge di distribuzione. Giomale dell' fstituto Italiano degli Am_ari 4:83-91 Smirr_v, N. V. 1939. Estimate of deviation between ernt_irical distribution functions m two independeat samples (in R_sst_). Bulletin Moscow University 2(2): 3-16.
I
i
AlSo See
'
Related:
[R]runtest,[R]sktest,[R]swilk
ll_
! ltle [ kwallis -- [ Kruskal-Waltis equality of population,_•i rank test [ I /Ill
]
I
i
Syntax kwallis
varname [if exp] [in rangeJ, by(grou_var)
Description kwallistests the hypothesis that several sample_ are from the same population. In the syntax diagram above, varname refers to the variable recortling the outcome and groupvar refers to the variable denoting the population. Note that the by () "option" is not optional.
Remarks > Example You have data on the 50 states. The data contain tlhemedian age of the population medage and the region of the country region for each state. You _,vishto test for the equality of the median age distribution across all four regions simultaneously: • kwallis medage, by(region) Test: Equality of populations region NE N Cntrl South West chi-squared = probability =
_Obs 9 12 16 13
(Kruskal-Wallis _test)
_KankSum 376.50 294.O0 398.00 206.50
17.041 with 3 d.f. 0.0007
chi-squared with ties = probability = 0.0007
17.062 with 3 d.f.
From the output we see that we can reject the hypotl_esis that the populations are the same at any level below 0.07%.
Saved Results kwallissaves in r(): Scalars r(df)
degrees of _freedom
r (chi2)
X2
r(chi2__adj)
X 2 adjustedfor ties
178
kwallis -- Kruskal-Wallis equality of populationsrank test
179
MMhodSand Formulas kUallisis implemented as an ado-file. The Kruskat-Wallis test (Kruskat and Wallis 1952; atso see Conover 1999. 288-297 or Airman 1991, 213-21'5) is a multiple-sample generalization of the two-sample Wilcoxon (also called Mann-Whitney) rank sum test (Wilcoxon 1945: Mann and Whitney 1947). Samples of sizes n3, 3 = I,..., m, are combined and ranked in ascending order of magnitude. Tied values are assigned the average ranks. Let n denote the overall sample size and let Rj denote the sum of the ranks for the jth sample. The Kruskat-Wallis one-way analysis-of-variance test H is defined as
H=
12 _., P_ - 3(n + 1) n(n + l ) j_l' nj
The sampling distribution of H is approximately X2 with m - 1 degrees of freedom.
tl raneas A_ttman. D. G. I991. Practical Satistics
for Medical Research.
Conover. W. J. 1099. Practical Nonparametric
London: Chapman & Hall.
Statistics. 3d ed. New York: John Wiley & Sons.
Ktuskal, W. H and W A. Wallis. 1952. Use of ranks in one-criterion Stalistical Association 47: 583-621.
variance anah,sis. Journal
of the American
Mann. H?B. and D. R. Whitney. 1947. On a test of whether one of two random variables is stochastically the other. Annals of Mathematical Sta6stics 18: 50-60. Wilcoxon. F. 1945. Individual
comparisons by ranking methods. Biometrics
Also Sea Related:
[R] nptrend, [R] oneway, [R] runtest, [R] signrank
1: 80-83.
larger than
Title label -- Label manipulation I I O
II
g_
I
i
Syntax la_bel data
["label"!
label define Iblname # "label"
[# "label"...
] [, _addmodify nofix ]
label dir label drop {Ibtname label
list
[lblname
... ]l-all
[Iblname [Iblname ... ] ]
label save [Iblname [Iblname.,. label
}
values
varname
[lblname]
label variable varname
]] using [, nofix
filen_me
[, replace ]
]
["label"]
Description label data attaches a label (up to 80 characters) to the dataset in memory. Dataset labels are displayed when you use the dataset and when you describe it. If no label is specified, any existing label is removed. label define defines a list of up to 65,536 (1,000 for Small Stata) associations of integers and text called a value label. The value label is attached te variables by label values. label
dir
lists the names of value labels stored in memory.
label
drop
eliminates value labels.
label
list
lists the names and contents of value labels stored in memory.
label
save
saves value labels in a do-file.
label values attaches a value label to a variable. If no value label is specified, any existing value label is detached. The value label, however, is r_ot deleted. label variable attaches a label (up to 80 characters) existing variable label is removed.
to a variable.
If no label is specified, any
Options add allows additional #++ label correspondences to be added to Iblname. If add is not specified. only new Iblnames may be created. If add is specified, you may create new lblnames or add new entries to existing Iblnames.
i
180
i
allows modification or deletion of existing #_ label correspondences and also allows additional correspondences to be added, Specifying modify implies add even if you do not type tl_eiadd option.
modify
no:fix prevents display formats from being widened according to the maximum length of the value label. Consider label values myvar mylab and pretend that myvar has a Y.9.0gdisplay format right now. Pretend that the maximum length Ofthe strings in mylab is t2 characters Then label values would change the format of myvar from Y.9.0g to %t2.0g. no:fix prevents this, nofix is also allowed with label define, but it is relevant only when you are modifying an existing value label. Withoutthe no:fix option, label define finds all the variables that use this value label and considers widening their display formats, no:fix prevents this. replace
allows filenalne to be replaced even if it already exists.
Remarks See:[U] 15.6 Dataset, variable, and value labels for a complete description of labels. This entry deals Onlywith details not covered there. label dir lists the names of all defined value labels, The label list contents of_a value label.
command displays the
Example Although describe shows the names of the value labels, those value labels may not exist. Stata does not consider it an error to label the values of a variable with a nonexistent label. When this occurs. Stata still shows the association on describe but otherwise acts as if the variable's values are Unlabele&This way, you can associate a value label name with a variable before creating the corresponding label, Similarly, you can define labels that you have not yet use& label dir shows you the labels that you have actually defined: label dir y_sno _exlbl
We have two value labels stored in memory: one called yesnoand the other called sexlbl We can display the contents of a value label using the labellistcommand: label list yesno
y_sne: 1 yes 2 no
The value label yesno labels the values l as yes and 2 as no. If you do not specify the name of the value label on the label list value labels is produced: label list y,esno_: 1 yes 2 no sexlb_
: 0 Male 1 Pemaie
command, a listing of all
r
182
label -- Label manipulation
[] Technical Note Since Stata can have more value labels stored in memory than are actually
used in the dataset,
you may wonder what happens when you save the dat_set, tn that case. Stata stores with the dataset only those value labels actually associated with variables. When you use a dataset, Stata eliminates the dataset.
all the _mlue labels stored in memory before loading []
You can add new codings to an existing value label _sing the add option with the label define command. You can modify existing codings using the modify option.
_>Example The label yesno codes 1 as yes and 2 as no. Perhaps at some later time you wish to add a third coding: 3 as maybe. Typing label define without any options results in an error: label label
define
yesno
yesno
already
3 maybe defined
r(llO) ;
If you do not specify the add or modify options, label define can only be used to create new value labels. The add option lets you add codings to _n existing label: . label label
define list
yesno
3 maybe,
add
yesno
yesno : 1 yes 2 no 3 maybe
Perhaps you have accidentally mislabeled a value. For instance. 3 may not mean "maybe" may instead mean "don't know", add will not allow you to change an existing label: label
define
yesno
invalid attempt r(180) ;
3 "don't
to modify
know",
but
add
label
Instead, you specify the modify option: label
define
label
list
yesno
3 "don't
know",
modify
yesno
yesno: I yes no 3 don_t
know
In this way, Stata attempts to protect you from yourself. If you type label define without any options, you can only create a new value label--you cannot accidentally mutilate an existing one. If you specify the add option, you can add new labelings to a label, but you cannot accidentally change one of the existing labelings. If you specify the modify optiom which you may not abbreviate, you can do whatever you want. You can even use the modify option to eliminate numeric code to a null string, that is. '....
existing labelings.
To do this. you map the
label-
Label m_mipul_lo.
183
1
. label define yesno 3 .... , modify label list yesno yesno: i yes 2 no
You can eliminate entire value labels using the label
drop command.
Example We currentlY have two value labels stored in memory--sexlbl and yesno.The label dir comman_ reports that: • label,dir
y_sno sexlbl
The da_a_t that we have in memory uses only one of the labels
sexlbl,
describe
reports that:
describe Centains data from emp.dta obs: vats: size:
7 4 224 (99.87,o_ memory free)
variable name
storage type
displ&y format
n_me
str16
7,16s
empno sex salary
float float float
X9.0g X9.0g Y,9.0g
S_rted
value label
sexlbl
1992 Employee Data 14 Jul 2000 14:28
variable label
Employee mumber O--male;l=female Annual salary, exclusive of bonus
by:
We can:eliminate the yesno label by typing label drop yesno: • ilabelidrop yesno • !la_el dir sexlbl
We could elin_inate all the value labels in memory by typing • .label_rop _ail label Idir
Remember that the value label sexlbl,which no longer exists, was associated with the variable sex. E_en after dropping the value label, sexlbl is still associated with The variable:
_04
taoe_-- Lane_manlpulatton . describe
i i
obs: Contains vats: size:
data
variable
name
7 :from emp.dta 4 224 (99.8_
1992 of memory
2000
Data 14:28
free)
storage
display
value
t_e
format
label
name
sir16
empno
float
7.16s _9.0g
sex
float
7,9.0g
salary
float
_,9.0g
Sorted
Employee
14 Jul
sexlbl
variable
label
Employee
number
O--_nale; 1=re, hale Annual salary, bonus
exclusive
of
by :
As stated earlier, Stats does not mind if a nonexistent value label is associated with a variable. When Stats uses such a variable, it simply acts as if it is not labeled: • list
1. 2.
in
1/4 name
empno
sex
salary
Hank Rogers Pat Welch
57213 47229
0 1
24000 27000
57323
0
24000
57401
0
24500
3.
Bob
4.
Richard
Underhill Doyle
q The label save command creates a do-file containing label define commands for each label you specify. If you do not specify the Iblnames, all value labels are stored in the file. If you do not specify the extension for fitename, . do is assumed.
_, Example Labels are automatically stored with your dataset when you save it. Conversely, the use command drops all labels before loading the new dataset. You may occasionally wish to move a value label from one dataset to another. The label save command allows you to do this. For example, assume we currently have the value label yesno label
list
in memory:
yesno
yesno: I yes 2 no 3 maybe
You have a dataset stored on disk called survey, dta to which you wish to add this value label. One alternative is to use survey and then retype the label define yesno command. Retyping the label would not be too tedious in this case, but if the value label in memory mapped, say, the 50 states of the union, retyping it would be irksome, label save provides an alternative: label save yesno using file ynfile.do saved
ynfile
Typing label save yesno using ynfile the definition of the yesno label.
caused Stata to create a doqilc called ynfile,
If we want to see the contents of the file, we can use the Stata type
command:
do containing
label-
Label manip?iation
IFK
type ynfile.do l_be_ define yesno 1 ""yes'", modify labe_ define yesno 2 "no"", modify label define yesno 3 ""maybe"', modify
Weean:now use our new dataset, survey.dta: • :us_ survey (]Ioudeholdsurvey data) • la_el dir
Using the new dataset causes Stata to eliminate all value labels stored in memory. _ label yesno isnow gone.Sincewe saveditinthefile ynfile,do,however,we cangetitbackby typingeither do runexecute ynfile.Ifwe execute. Ifwe type run,y_file:,or _he file will silently: typedo.we willsecthecommandsinthefile , runiynfile . -lab_l yeSno
dir
The libel islnow just as if we had typed it from the keyboard. q
0 Techni_aiNbte Yola can also use the label save command to make the editing of value labels easier. You can save a: label in a file. leave Stata and use your word processor or editor to edit the label, and then return ,to Stata. Using do or run, you can load the edited values,
Gleason, J. R. J998. dm56: labels editor in Stata Technical ButletmA Reprints, vol, for 8, Windows pp. 5-t0. and Macintosh. Stata Technical Bulletin 43: 3-6.
--'vo1.1999"9, p.dm_6"l:]_.Update to labedk
Stata Technical Bulletin 5I: -.') Reprinted in Stata Technical Bulletin Reprints,
Weesie, L 1997. din47:vot. Verifying value label mappings. Stats Technical Bulletin 37: 7-8. Bull_tin Reprints, 7, pp. 39-40.
Also:See Background:
Reprinted
[u] 15.6 Dataset, variable, and value labels
Reprinted in Stata Technical
ladder -- Ladder of powers
Title []
Syntax ladder
varname [if
gladder
varname
qladder
varname
symbol(string)
exp]
[in range]
[if exp] [in range] [if
exp]
[in
margin(string)
range]
[" g_enerate(newvar) [, bin(#)
graph_options
noadjust
]
]
[, grid scale(#)
saving([ilename[,
replace])
]
by ... : may be used with ladder; see [R] by.
Description ladder searches a subset of the ladder of powers (Tukey 1977) for a transform that converts varname into a normally distributed variable, sktes_: is used to test for normality; see [R] sktest. Also see [el boxcox. gladder
displays nine histograms
of transforms Of varname according
to the ladder of powers.
qladder displays the quanfites of transforms of vamame according to the ladder of powers against the quantiles of a normal distribution.
Options generate(newvar) saves the transformed values co_esponding to the minimum chi-squared value from the table. Its use is not, in general, recommended since generate() is quite literal in its interpretation noadjust
of minimum, thus ignoring nearly equal but perhaps more interpretable
is the noadjust
option to sktest;
see [R] sktest.
bin (#) specifies the number of bins for the histograms. for you (see Methods and Formulas below). graph_options grid scale
transforms.
If not specified, an intelligent choice is made
are any of the options allowed with graph,
histogram;
see [G] histogram.
adds grid lines at the .05, .I0, .25, .50, .75, .90, and .95 quantiles. (#) specifies the size of text used to label the graphs, scale(1.25)
symbol(string)
is the default.
specifies the symbol used in the graph.
margin(string) specifies the margin to be placed around each graph as a percenl of graphical area. The default is 0. saving_Iename[,
replace
]) saves the graph.
i
186
,
lad_ler-- Ladder of powers
18/
IrrrkS Example You have data on the mileage rating of 74 automobiles and wish to find a transform that makes the varihble n_rmally distributed:
.-ila e =pg T_an_Io_mation
formula
chi2(2)
P(chi2)
cube square raw squaxe-_oot log reciprocal root r_icil_rodal
mpg'3 mpg'2 mpg sqrt(mpg) log (_g) I/sqrt(mpg) llmpg
43.59 27.03 10.95 4.94 O.87 0.20 2.36
(}. 000 O. 000 O.004 O.084 O. 647 0.905 O.307
recipro_al square rdci_rodal cube
i/(mpg-2) I/(mpg'3)
1I. 99 24.30
O. 002 O, 000
Had we t>ped '_adder mpg, gen (mpgx). the variable mpgx would have been automatically generated for us containing 1/_ m,'_pg.This is the perfect example of why you should not. in general, specify the generate() option. Note that we also cannot r_iect the hypothesis that the reciprocal of mpg is normally distributed and i/mpg gallons per mile has a better interpretation. I! is a measure of energy ¢onSun_ption. q
Example glad_lerexplores the same transforms as ladderbut presents results graphically: • gladd_
mpg
Histograms
Mi{eage(@pg) by Transformation
q
188
ladder -- Ladder of powers
Q Technical Note
!_
gladder is useful pedagogically, but some caution must be exercised when using it for research work, especially with large numbers of observations; For instance, consider the following data on the average July temperature in degrees Fahrenheit for 1954 U.S. cities:
ii+.i
. ladder tempjuty Transformation
formula
chi2 (2)
cube square raw square-root log reciprocal root reciprocal reciprocal square reciprocal cube
tempjuly_3 tempjuly'2 tempj uly sqrt (tempiuly) log (tempjuly) 1/sqrt (tempjuly) I/tempjuly 1/(tempjuly'2) I/(tempjuly'3)
47.49 19.70 3.83 1.83 5.40 13.72 26.36 64.43
P (chi2) O.000 O.000 0.147 O.400 O. 067 O.001 O. 000 O. 000 O. 000
The period in the last line indicates that the ;g2 is very large; see [R] sktest. From the table, we see that there is certainly a difference, normality-wise, between the square and square-root transform. If, however, you can see the difference between the transforms in the diagram below, you have better eyes than we do: • gladder tempjuly
eubo
squir*
_9_23
B_eo28
menl_+_/
337_
sqn
5B,_
,og
_
_q+l
+_ .+?044
+ + 7 S+2_14
-+
+ i ++7471
o 4.0+++P'
m¥@rll
'
*
• ++1+m3
+
l/Ioum+l
_S89_
i . °,+++,,
+++055
.2704
t
t I +o,o,,,+,
. +_+3+31+p
!tcm_l
:+J..... oo_.,+
Average
Histograms
duty
+,_°, +J
tem_eralure
by Trafisformation
CI
Example
+i
A better graph for seeing normality is the quantile-normal
graph which can be produced by qladder.
ladder-- Ladder of powers
,1
qladder tempjuly
g2_}0_6 I
--a_
1381B1 15 _ 13B'_31
B7 BO._6
.... 721058
r
3110.81 311:0 81
sqrl
7 68
58,1 5B.1481
8215.65
Jog ....
53
9.63357
4.0J_963
_nverse
-.017212 _ ..016441
93,6
tlsqrt
4.54141
-.128761
I tsquere
$.
t -000296 - 01035
-.00{_263
Average
Ouantile-Normat
-,000098
91.9594
-.102562
1/cube
-5.1e.06 -4.2e-06
-7 4e-07
Ju y temperature
PlOts by Transformation
This graph shows that for the square transform, the upper tail, and only the upper tail, diverges from,what would be expected, This is detected bs_sktest as a problem with skewness, as we would learn from using sktest to examine tempjuly squared and square-rooted, ,3
:SavedtResults l_dd_r saves in r(): Scalars r(l_)
number of observations
r(cube)
X_ for cube transformation
r(P_cube)
significance level for cube transformation
r(square)
_2 for square transformation
r(P_square)
significance level for square transformation
r (raw)
X_ for untransfom_ed: data
r(P_raw)
significance level for iuntransformed data
r(sqrt)
_:e for square-root
r(P_sqrt)
significance level for square-root
r(log)
.x: for log transformation
r(P_log)
significance level for log transformation
r(invsqrt)
:ts for reciprocal root transformation
r(P_invsqrt)
significance level for reciprocal
r(inv)
X2 for reciprocal transformation
r(P_inv)
significance level for reciprocal
r(invsq)
;_2 for reciprocal square transformation
r(P_invsq)
significance
r(invcube)
k 2 [br reciprocal cube transformation
r(P_invcube)
s_gnificance level for reciprocal cube transformation
transformation transformation
root transformation transfommtion
level tbr reciprocal square transformation
190
ladder -- Ladder of powers
Methods and Formulas ladder,gladder,and qladder are implemented For ladder, results are as reported by sktest; transform with the minimum X 2 value is chosen.
as ado-files. see [R] sktest. If generate
() is specified, the
gladder sets the number of bins to min(v/-_ , 10 log_o n), rounded to the closest integer, where n is the number of unique values of varname. See [G] histogram for a discussion of the optimal number of bins. Also see Findley (1990) for a ladder-of-powers variable transformation program that produces one-way graphs with overlaid box plots, in addition to histograms with overlaid normals. Buchner and Findley (1990) discuss ladder-of-powers transformations as one aspect of preliminary data analysis. Also see Hamilton (1992, 18-23).
Acknowledgment qladder
was written by Jeroen Weesie, Utrecht University,
Netherlands.
References Buchner, D. M. and T. W. Findley. 1990. Research in physical medicine and rehabilitation: viii preliminary data analysis. American Journal of Physical Medicine and Reliabilitation 69:154-169. Findley, T. W. 1990. sod3: Variable transformation and evaluation. Smta TechnicalBu!letin 2: t5. Reprinted in Smm TechnicalBulletin Reprints, vol. 1, pp. 85-86. Hamilton. L. C. 1992. Regression with Graphics. Pacific Gro_'e.CA: Brooks/Cole Publishing Company. Tukey, J. W. 1977. Exploratory Dam Analysis. Reading, MA: Addison-Wesley Publishing Company.
Also See Related:
IR] hoxcox, [R] diagplots,
Background:
Stata Graphics Manual
[R] Insl_ew0, [R] Iv, [R] sktest
Title
level _ Set default confidence level
Syntatx S_t!leVel#
DescriptiOn •
i
set It#el specifies the default confidence level for confidence intervals for all commands that repoia Confidence intervals. The initial value is 95. meaning 95ck confidence lnter_a],. " , _ # may . be betweeri 10 and 99.
Remarks i
I
TO change the width of confidence intervals reported by a particular command, it is not necessary to re_et!theldefault confidence level. All commands that report confidence intervals have a level (#) option. 'W_en you do not specify the option, the confidence intervals are calculated for the default level sei b_ set level
or 95% if you have not reset it.
> Example You tlse the c± command to obtain the confidence interval for the mean of mpg: . ,ci
Impg
mpg _ariable
I
74 Obs
21.2973 Mean
.6725511 Std. Err.
19.9569 [95Y, Conf.
22.63769 Interval]
[90]/,Conf.
Interval]
To obtain _0% confidence intervals, you could type • {el_pg, level(90) _ariable
Obs
mpg
74
Mean 21.2973
Std.
Err.
.6725511
20.17683
Std.
[90Y, Conf.
22.41776
Or iset i level
90
• :ci _pg g,_riable
Obs
mpg
74
Mean 21.2973
Err.
,6725511
20. 17683
Interval] 22.41776
If y'ou opt for the second alternative, the next time you estimate a model (.say with regress). OOe/_ confidence intervals will be reported. If you wanted 95% confidence intervals, you could specify' level (95) I on the estimation command or you dould reset the default by typing set level 95. 1
191
Also See
192 level -- Set default confidence level Complementary: [R] query
ii
Related:
[R] ci
Background:
[U] 23 Estimation and post-estimation commands. [O] 23.5 Specifying the width of confidence intervals
•
,
!
' J i
-- Quick reference for limits ,i i, ri i I i r i_lll
H
I,,,,lliHillllll,
i
i
!
Descdp{tidn ' i T!fisientry provides a quick reference for the size limits in Stata. Note that most of these limits are So _igt that you will never encounter them.
Remarks Maximum size limits for Small Stata and Intercooled Stata
_
l
_
,
Small Stata
Nu_be[ of observations Nu_bei of variables WiSh 0f a dataset Valde df matsize : I Numberof characters _n a command
about t,000 99 200 40
67,800
50
50
t,600 100 8
1,600 100 8
66 50
200 t 50
256 80 5
512 80 5
3,400
67,784
32
32
1,000 37,296
3,500 135,600
_ngthlof variable name Length tof ado-command name
32 32
32 32
Eer_gthtof global macro name _ngth!ofi local macro name
32 31
32 3t
Lengthi of a string variable
80
80
Numberf 1)ilirr ited°fconditionSby memory.inan if statement (Confi,ued on nero page)
30
100
Nu,be t of elements in a numlist Nurbbe i of unique time-series operators in a command Numbe_cof seasonal suboperators per time-series operator Nur0be [i of dyadic operators in an expression Nurpbe I of numeric literals in an expression Nur0'be_-,of string literals in an expression _ngth of string in string expression Numbe[ of sum functions in an expression Numbe_ of characters in a macro Nu_be_ of nested do-files Nu,be_ of lines in a program Nu_be[ of characters in a program
I
2.147,483,647 (1) 2,047 8,192 8(30
3,500
Nu_be i of options for a command
[i
Intercooled Stata
1
]
1 ' 1
i
1 ] t
Z
193 i i
!
194
limits -- Quick reference for limits Maximum size limits for specific commands Small State
Intercooled Stata
8 4
8 4
anova
Number of terms in anova model test statement Number of terms in the repeated() option char
Maximum length of a single characteristic
3.400
67,784
constraint Number of constraints encode
1,000
1,000
1,000
65,536
and decode
Number of uniquevaluesfora string variable estimates hold Number of stored estimation results
10
10
13
13
_.N/2
_.NI2
graph (See State Graphics Manual for graph limits) greigen Number of eigenvalues
plotted by greigen
grmeanby Number of unique values in varlist hist Number of unique values in varname impute Number of variables in varlist
(Table continued
on next page)
50
50
31
3t
!
V
limits -- Quick reference for lhnits
195
Maximum size limits for _pecific commands, continued
Small Intercooted St,am Stata iRecord length without data dictionary Record length with data dictionary (_)
none 7.998
none 7,998
none 7,998
none 7.998
80 80 80 32 1.000
80 80 80 32 65.536
Infix !Record length without data dictionary Record length with data dictionary (2) labal Length of dataset label Length of variable tabel Length of value label Length of name of value label Number of codings within a single Value label mat :ix Size of a single matrix
40x40
tnajimize options Number of iterations specified with iterate()
16.000
16.000
10
10
20
50
8
8
t.000 9.999 9.999
67.784 9.999 9.999
1.600
1.600
_l_git Number of outcomes in model
20
50
Op_obit Number of outcomes in model
20
50
800×800
mer_e
, ,
Number of variables that you can specify in a match-merge ml_git Number of outcomes in model nl4git and nlogittree Number of levels in model i_o_es
,
! Maximum length of a single note _ Number of notes attached to _dta Number of notes attached to each variable
nu_list i Number of elements in the numeric list
(2) :or Stata for Unix. the maximum record length is 19.998.
i'l
i _!
Maximum size limits for specific commands, limits -- Quick reference for limits
196
continued
plot Number of columns specified with column() option Number of lines specified with lines() option reg3, set
sureg, and other system estimators Number of equations
Small Stata
[ntercoolcd Stata
133 83
133 83
40
800
500K
500K
5
5
4
4
3.000
3,000
500 160 20
3,000 300 20
375
375
40
800
adosize
Maximum
amount of memory that ado-files may consume
sts graph Number of by variables (3) tabdisp and table Number of by variables Number of margins; that is, the sum of the rows, columns, supercolumns, and by groups tabulate Number of rows for a one-way table (4) Number of rows for a two-way table (4) Number of columns for a two-way table tabulate,
summarize
Number of cells xt estimation commands Number of time periods
(3) May be restricted to fewer depending on other options specified. (4) For lntercooled Stata for the Macintosh, limits are 2.000 for the number of rows for a one-way table and 180 for number of rows for a two-way table.
Also See
[I
Related:
[R] matsize,
[R] memory
Background:
[U] 7 Setting the size of memory
Title
,
[
i
I
II
0
I
U
I[
I
lin_o,exp [,level
iF
I
IU i
i
I
[I "
(#) or hr irr rrr e__form]
exp is apy 1:near combination of coefficients that is valid syntax for test: riot _ont_fin any additive constants or equal signs.
S¢¢ [R] test, Note, however, that exp must
Descrilti(m
'
li_o_, computes point estimates, standard errors, t or z statistics, p-values, and confidence intervals )r a linear combination of coefficients after any estimation command except a.nova. Results can optionally be displayed as odds ratios, hazard ratios, incidence rate ratios, or relative risk ratios. I
The. s_ y estimation commands for survey data have their own special command, svylc, estimating linear combinations: see [R] svylc.
OptiOnS leveli( ) specifies the confidence level, in percent, for confidence intervals. I
'
i
"
l
,
l i
[
Syntax
I
i
c m -- Linear combinations of estimators
or ;Is s_t by set level:
The default is level
for
(95)
see [U] 23.5 Specifying the width of confidence intervals.
or. hr_, ilr, rrr, and eform all do the same thing:, they all report coefficient estimates as exp(3) rather than ft. Standard errors and c?nfidenc_ intervals are similarly transformed. Note that or is the default after logistic.The onl 3 differehce in these options is how the output is labeled.
, :
Option
Label
Exl_lanation
Example commands
or
Odds Ratio
Odds ratio
logistic,
hr
_Iaz. Ratio
Ha_d
stcox,
irr
IRR
Incidence Rate Ratio
poisson
rrr
_
Relmive Rate Ratio
mlogit
eform
exp(b)
Gerieric label
ratio
logit streg
Remarks After itting a model and obtaining estimate_ for coefficients/3t,/32, .... 3_-, one often wants to view esti! aates for linear combinations of the 3i!, such as 31 32. lincomcan display estimates for an)" lilaem combination of the form ¢131 + C2/_2 + " '- * Ck'3k. Any valid works. estimation command for which iinco: works after ans' test except anova expressio_ tbr test Syntax l (see [R] test-_is a valid expression for lincom,There is only one exception to this rule: lincom does not allow ddditive constants: i.e.. it cannot display estimates for co -t- C131_-" "+ ck/3_,-when co _ O. line@
is useful for viewing odds ratios, lhazard ratios, etc.. for one group ti.e., one set _f
covariatel)1 relative to another group (i.e.. another set of covariates). See examples below 197
D Example We estimate a linear regTession: i!I_
- regress y xl x2 x3
l Source I
!" ili_ !i
Model Residual
,
Total
l
SS
.
df
MS
3259,3561 1627.56282
3 144
i086,45203 II .3025196
4886.91892
147
33.24434_4
Y
Coef.
t
/ i
= = = = =
148 96.12 0.0000 0.6670 0.6600 3.3619
Std. Err.
I
l xt x2 x3 _cons
Number of obs F( 3, 144) Prob > F R-sql/ared Adj R-squared Root MSE
1.457113 2. 221682 -.006139 36.10135
1. 07461 .8610358 .0005543 4.382693
1.36 2.58 -11,08 8,24
P> [t I O. 177 O. 01I O,000 O,000
[95Y,Conf, Interval] -. 6669339 • 5197797 -,0072345 27.43863
3. 581161 3. 923583 -.0050435 44. 76407
To see the difference of the coefficients of x2 and xl, type lincom x2 - xl (1)
- xl
+ x2 = 0.0
y
Coef.
(1)
Std. Err.
.7645682
• 9950282
The expression can be any linear combination
t O. 77
P> ItI O. 444
[95Y,Coal. Interval] -1.20218
2. 731316
without a constant.
lineom 3,xl + 500,x3 (I)
3.0 xl + 500,0 x3 = 0.0
y
Coef.
(1)
1. 301825
Std. Err. 3. 396624
t O. 38
P>It I O. 702
[957,Conf. Interval] -5. 411858
8.015507
Expressions with additive constants are not allowed lincom xl
-
1
additive constant terms not allowed r (198) ;
norarenonlinear expressions. • lincom X2/xl not possible with test r(131) ;
<3 Q Technical Note lincom uses the same shorthands for coefficients as does test (see [R] test). When you type xl, for instance, lincom knows that you mean the coefficient of xl. The formal syntax for referencing this coefficient actually _b [xl], or alternatively _coef [xl]. So, more forma]ly, in the last example we could have istyped 2incom 3*_b[xl] + 500*_b[x3]
l
rl_
iincom-
!
Linear combinationsof estimators
t99
OddsAfter rati(,s andregression, incidence ratios l_gistic the orrate option can be specified I i
, _
with lincom to display odds ratios for any effect.: InCidence rate ratios after commands such as poisson can be obtained in a similar fashion by spe_cif_ing the irr option.
> Example t
Consider the low birth weight dataset from H0smer and Lemeshow (1989, Table 4. I). We estimate a logistic regression of low birth weight (variable low) on the following variables:
i
_
Vail _ble
Description
Coding
age
age in vears
bla,:k
race black
1 if black, 0 otherwise
Oth,;r
race other
1 if race other, 0 otherwise
Smol:e
smoking status
1 if smoker, 0 if nonsmoker
tit
history of hypertension
1 if yes. 0 if no
u±
uterine irritability
t if yes, 0 if no
t'wd
maternal weight before pregnancy
] if weight < t10 lb., 0 otherwise
aige_.wd
age × twd
smol:elwd ptd
smoke history x of lwd premature labor
l if yes.. 0 if no
i . l
We firsI estimate a model without the interaction terms agelwd
and smoketwd (Hosmer and
Lemest_ovq1 1989, Table 4.8) using logit ,
Io [it low
age lwd black
other
smoke p_d
ht ui
-117.336
I%er .tion O:
log likelihood
=
I_er _tion i:
log likelihood
= -99.4311]_4
l_eration
2:
log likelihood
= -98.785718
Ieerltion
3:
log likelihood
=
I%er _tion 4:
log likelihood
= -98.777998
L_gi
-98.7_8
estimates
Number
! i
=
189
LR chi2(8)
=
37.12
> chi2
=
0.0000
:
0.1582
Prob Log
.ikelihood
: -98.777998
Pseudo
of obs
R2
, I
:
,
low
i
age twd
i
-. 0464796 .8420615
Std. Err. .0373888 .4055338
z
P>[zI
-1.24 2.08
0.214 O. 038
[957, Conf. -. 1197603 ,0472299
Interval] .0268011 1,636893
black
I.073456
.5150752
2.08
O. 037
.0639273
2.082985
other smoke
.815367 .8071996
.4452979 .404446
1.83 2. O0
0.067 O. 046
-. 0574008 .0145001
I. 688135 i.599899
ptd
i i
Coef.
i .281678
.4621157
2.77
0,006
.3759478
2. 187408
ht
:1. 435227
.6482699
2.21
0.027
.1646415
2. 705813
ui _cons
.65762S6 -1.216781
.4666192 .9556797
I. 41 _1.27
-. 2569313 -3.089878
1. 572182 .6563!7
O. 159 0.203
To _et the odds ratio for black smokers relative to white nonsmokers (the reference group), type
• lincom (1) i: I
200
black
black
+ smoke,
+ smoke
or
= 0.0
Iincom -- Linear combinations of estimators
ili
low
0dds
Ratio
Std.
z
Err.
il
P>,z,
[957, Conf.
Interval]
o0o, lincom computedcxp(_black+ blacknonsmokers,type lincom (I)
smoke
- black
- black, + smoke
low
Odds
(1)
_smoke)
_-
6.56.To seetheoddsratioforwhitesmokersrelative to
or
= 0.0
Ratio
Bid.
.7662425
Err.
z
.4430176
-0.46
P>IzJ
[95% Conf.
0.645
.2467334
Interval] 2.379603
Now let's add the interaction terms to the model (Hosmer and Lemeshow 1989, Table 4.10). This time we will use logistic rather than legit. By defaulL logistic displays odds ratios. . logistic Logit
Log
low
age
black
other
smoke
ht ui
Iwd
estimates
likelihood
=
low
-96.00616
Odds
Ratio
Std.
Err.
z
ptd
agelwd
smokelwd
Number of obs LR chi2(10) Prob > chi2
= = =
189 42.66 0.0000
Pseudo
=
0.1818
R2
P>|zl
[95_ Conf.
Interval]
,8408967 1.068277
1.005344 8,167462
age black
.9194513 2.95383
.041896 1.532788
-1.84 2.09
0.065 0.037
other
2.137589
.9919132
1.64
0.102
.8608713
5,307749
smoke
3.168096
1.452377
2.52
0.012
1.289956
7.780755
ht
3.893141
2.5752
2.05
0.040
1.064768
14.2346
ui
2.071284
.9931385
1.52
0.129
.8092928
5.301191
lwd
.1772934
.3312383
-0.93
0.354
.0045539
6.902359
ptd
3.426633
1.615282
2.61
0.009
1.360252
8.632086
1.15883 .2447849
.09602 .2003996
1.78 -1.72
0.075 0.086
.9851216 .0491956
1.36317 1.217988
agelwd smokelwd
Hosmer and Lemeshow (1989, Table 4.13) consider the effects of smoking (smoke -:- 1) and low maternal weight prior to pregnancy (lwd = 1). The effect of smoking among non-low-weight mothers (lwd -- 0) is given by the odds ratio 3.17 for smoke in the logistic output. The effect of smoking among low-weight mothers is given by • lincom (I)
smoke
smoke
low (1)
+ smokelwd
+ smokelwd
Odds
= 0.0
Ratio
.7755022
Std.
Err.
.5749508
z
P>Izl
[957 Conf.
-0.34
0.732
.1813465
Note that we did not have to specify the or option
After
logistic,
lincom
Interval] 3.316322
assumes or by default.
The effect of low-weight (Iwd = 1) is more complicated since we fit an age x lwd interaction. We must specify' the age of mothers for the effect. The effect among 30-year old nonsmokers is given by t
i _• !|_
i
'
_
i
lin¢om-- Linearcombinations of estimators
i _incom l_d + 30*agelwd (ii) lwd + 30.0 agelwd
i
i.
low
I
(t)
i
t
=
201
0.0
Odds Ratio 14. ?669
Std,
Err.
13. 56689
z 2.93
P>lz[
[95X Conf.
O.003
2. 439266
Interval] 89. 39625
..........
"
lincom _omputed exp(fllwd+30,_agelwd) = 14_.8.It seemsodd that we entered it as lwd+ 30*agelwd. but remember that lwd and agelwd are just'lincom's (and test'S) shorthand for _b[twd] and
i
_b [age_wd].
i
I
_ i
We could
i
i !
typed
(ii) incomlwd _b[1wd] + 30.0+ agelwd 30*_b[agelwd] = 0.0
low (1)
i
!
have
I
Odds Ratio 14. 7669
Std. Err. 13. 56689
z 2.93
P> Iz I O. 003
[957,Conf. Interval] 2. 439266 89. 39625
,
Multiple- quation models lincpm also works with multiple-equation models. The only difference is how"y'ou refer to the coefficiehts. Recall thai for multiple-equation models, coefficients are referenced using the syntax [e_no] varname where e,_nois the equation number or equation_nameand varname is the corresponding variable name for the cpefficient: see [U] 16.5 Accessing coefficients and standard errors and JR] test for detaih. ExampM, Consider the example from [R] mlogit (Taflov et al. 1989: Wells et al. I989).
!
. _logit insure age male nonwhite site2 site3,
nolog
Mu_tinomial regresszon
Number of obs LR chi2(I0)
= =
615 42.99
Lo i likelihood = -534.36165
Pseudo Prob > R2 chi2
= =
0,0387 O.0000
} !
insure
Coef.
age
-.Ol 1745
.0061946
-I.90
O.058
-.0238862
.0003962
male nonwhite
.5616934 .9747768
.2027465 .2363213
2.77 4.12
O.006 0.000
.1643175 .5115955
.9590693 1,437958
site2 site3 cons
.1130359 -. 5879879 .2697127
•2101903 .2279351 .3284422
O,54 -2.58 O.82
O.591 O. 010 O.412
.2989296 -1.034733 -.3740222
.5250013 -. 1412433 .9134476
I
age male
-.0077961 .4518496
.01t4416 .3674867
-0.68 1.23
0.496 0.219
-.0302217 -.268411
.0146294 I.17211
i
nonwhit e
.2170589
.4256361
O.51
O.610
-.6171725
i.05129
i
site2 site3 _cons
-1.211563 -.2078123 -1.286943
.4705127 .3662926 .5923219
-2.57 -0.57 -2.17
O.010 0.570 0.030
-2.133751 -.9257327 -2.447872
-.2893747 .510108 -. 1260135
i "
'
Std. Err.
z
P> Izl
[95Y,Conf. Interval]
Prepaid
i
Un insure
l (C _tcome insur!==Indem iB the comparison group) i
i' _
202 Linear combinations of estimators To see thelincom estimate-- of the sum of the coefficient of male and the coefficient Prepaid outcome, type • lincom (I)
of nonwhite
for the
[Prepaid]male + [Prepaid]nonwhite
[Prepaid]male + [Prepaid]nonwhite = 0.0
insure I (1)
Coef.
I
Std. Err.
1.53647
.3272489
z
P>Izf
4.70
0. 000
[95Y,Conf. Interval] lr
.
.8950741
2.177866
To view the estimate as a ratio of relative risks (see [R] miogit for the definition and interpretation), specify the rrr option. lincom (1)
[Prepaid]
male
[Prepaid]male
+ [Prepaid]
insure I (i) I
nonwhite,
+ [Prepaid]nonwhite
KPd{
rrr = 0.0
Std. Err.
4.648154
z
1.521103
4.70
P> [zl
[95X Conf. Interval]
0.000
2.447517
8.827451
q
Saved Results lincom saves in r(): Scalars r (estimate) r(se)
point estimate estimate of standard
r(df)
degrees
error
of freedom
Methods and Formulas lincom is implemented
as an ado-file.
References Hosmer, D. W., Jr,, and S. Lemeshow. edition forthcoming in 2001,)
f989. Applied
Logistic
Regression.
Tarlov, A. R,, J. E. Ware. Jr,, S. Greenfield. E. C. Nelson, E. Perrin. study, Journal of the American Medical Association 262: 925-930,
New York: John Wiley & Sons. (Second
and M. Zubkoff.
t989.
The medical
outcomes
Wells. K. E., R. D. Hays, M, A. Burnam, W, H. Rogers, S. Greenfield, and J. E. Ware. Jr. t989. Detection of depressive disorder for patients receiving prepaid or fee-for-service care. Journal of the American Medical Association 262: 3298-3302,
Also See Related:
[R] svylc, [R] svytest, [R] test, [R] testni
Background:
[u] 16.5 Accessing coefficients and standard errors, [U] 23 Estimation and pOst-estimation commands
I
rR r-
Title
II
l[nktes -- Specification link test for single-equation models i
!
i
I [
I
I
J
[
iI
I
i
I [
I
T[
I
I I
Syntax
i
lin.kt,
st [if exp] [in range] [, estimation_options ]
_qaen i: cap and in range are not specified,the link _estis performedon the same sample as the previous estimation.
Descripti(,n iin.kt_ st performs a link test for model specificationafter any single-equationestimationcommand such as 1, .gistic.
i
etc.;
regress,
see [R]
estimation commands.
I
:: ! ;
Options estimation_options must be the same option_ specified with the underlying estimation command.
] :
i
Remarks • i
The fotm of the linkltest implemented here if based on an idea of Tukey (1949) which was further descfibedlby Pregibon !(1980), elaborating on ,/,ork in his unpublished tl_esis (Pregibon t979). See Methods _nd Formulas! below for more details.
We at mpt to explifin Exampletl , the mileage ratings Of cars in our automobile dataset using the weight. engine displacement, a_d whether the car is manufactured outside the U.S.: r,_gress mpg
Source Model ;
w_ight
i !1619,71935
Residual
_ Total
mpg i
weight
23.740114 ; 12443.45946 _ SS
'
! i
foreign _cons
Coef.
-.0067745
displacement I
displ
.0019286 i-1.600631 41.84795
t
foreign
3
539.906448
F( 3, Prob > F
70
Ii.7_77159 33.4720474 MS
73 dI
Std. Err. .0011665 .0100701 1.113648 2.350704
t
P>It I
70)
= =
45.88 0.0000
R-squared
=
0,6629
Adj R-squared Root Number MSE of obs
= = =
0.6484 3.4304 74
[95X Conf.
Interval]
-5.81
0.000
-.0091011
0.19
0.849
-.0181556
.0220129
-1.44 17.80
0.155 O. 000
-3.821732 37.15962
.6204699 46.53628
-.0044479
204
linktest -- Specification link test for single-equation models
Based on the R 2. we are reasonably pleased with this model. If our model really is specified correctly, then were we to regress mpg on the prediction and the prediction squared, the prediction squared should have no explanatory power. This is what link"cost does: linktest Source
_
SS
df
f
Model Residual
] |
mpg
Number of F( 2,
1670.71514
2
835.357572
Prob
772.744316
71
10.8837228
73
33.4720474
I Total
MS
2
443
I
.45946
Coef.
I
Std.
Err.
obs 71)
= =
> F
74 76.75
=
0.0000
R-squared
=
0.6837
Adj K-squared Root MSE
= =
0.6748 3.299
t
P>Itl
[95_
Conf.
-0.63
0,532
-i.724283
2.09 2.16
0.041 0.034
.6211541 .0026664
Interval]
i
_hat _cons _hatsq
]
-.4127198
t
14.00705 .0338198
We find that the prediction good as we thought.
.6577736 6,713276 .015624
.8988434 27.39294 .0649732
squared does have explanatory, power, so our specification
is not as
Although linktest is formally a test of the specification of the dependent variable, it is often interpreted as a test that. conditional on the specification, the independent variables are specified incorrectly. We will follow that interpretation and now include weight-squared in our model: • gen
weight2
regress
= weight*weight
mpg
Source
weight I
Model
weight2 SS
displ
foreign
df
MS
Number F( 4,
of obs 69)
74 39.37
1699.02634
4
424.756584
Prob
=
0.0000
744.433124
69
10.7888859
K-squared
=
0.6953
Total
2443.45946
73
33.4720474
Adj R-squared Root MSE
= =
0.6777 3.2846
mpg
Coef.
Std.
Residual
Err.
t
P>rtl
> F
= =
[95_
Conf.
Interval]
weight
-.0173257
.0040488
-4.28
0.000
-.0254028
-.0092486
weight2
1.87e-06
6.89e-07
2.71
0.008
4.93e-07
3.24e-08
-.0101625
.0106236
-0.96
0.342
-.031356
foreign
-2.560016
1.123506
-2.28
0.026
-4.801349
-.3186832
_cons
58.23575
6.449882
9,03
0.000
45.36859
71.10291
displacement
.011031
And now we perform the link test on our new model: linktest
Model
1699.39489
Kesidual
i
744. 06457
Total
I
Source
[
2443.45946
SS
2
849.697445
Prob
=
0.0000
71
I0.4797827
R-squared
=
O. 6955
33.4720474
Adj R-squared Root MSE F( 2, 71)
= =
0.6869 3.2372 81.08
Number
=
73
df
MS
> F
of obs
74
'
gi
linktest-- specifi_ationlink test for single-equationmodels
i !•
i
mpg
Coef.
1
i,
hat hatsq
I 141987
.7612218
1.50
0.138
-.3758456
2.659821
i
_cons
- .0031916 -i.50305
.0170194 8.196444
-0.19 -0.18
O.852 O.855
-.0371272 -17.84629
.0307441 14.84019
'
t
P>Itl
[957,Conf. Interval]
! We now pass the link!'test.
! i
Std. Err.
205
•
> Exampl Abo_ we followe_ a standard misinterpretation of the link test when we discovered a problem, we focusted on the exl_lanatory variables of our model. It is at least worth considering varying exactly
i
what thi link test testS. The link test told us it]at our dependent variable was misspecified. For those
i
with _engineeringconsurr_tion__gallonsbackground, mpgperis mile--inindeed a terms strangeofmeasure.andIt woulddisptacement:make more sense to modelan@ergy weight ! _egress gpm _eight displ foreign i Source I 85 df i
i
] Model i Residual
i
Prob > F R_squared
=
0.7659
.01t957628
73
.000163803
Root MSE Adj R-squared
: =
.00632 0.7558
weight displacement I foreign
_cons
Std. Err.
t
P>lt I
[957.Conf, Interval]
.0000144
2.15e-06
6.72
O.000
.0000102
.0000187
.0000186 .0066981
.0000186 .0020531
I.O0 3.26
O.319 O. 002
-.0000184 .0026034
.0000557 .0107928
.0008917
.0043337
0.21
O. 838
-. 0077515
.009535
looks eve _ bit as reasonable as our original model.
inkiest _.
[ [
.003052654 .000039995
Coef.
_
,
Source
.
I li
SS
df
Residual
i .002782409 I Total .011957628 Model l ) i .009175219
gpm [i
Coef.
hat hatsq
I i I i
.6608413 3.275857
li
-_cons
))
.008365
irt a m( eparsimonio_s
MS
Number of obs =
F( 2,
!
i
74 76.33 0. 0000
3 70
gpm
This re+el
Number of obs = F( 3, 70) =
: .009157962 ;: .002799666
Total
! !
i
MS
71
.008039189
73 2
.000163803 .00_587609
Std. Err.:
t
P> It{
74
71) :
117.06
R-squared = Adj R-squ_red : Root = Prob MSE > F =
0.7673 0.7608 .00626 0.0000
[95_ Conf. Interval]
.5152751 4.936655 ;
1.28 0.66
0.204 0.509
-.3665877 -6.567553
1.68827 13.11927
.0130468
0.64
0.523
-.0176496
.0343795
speecmcanon " .
206
linktest -- Specification link test for single-equation models
> Example The link test can be used with any single-equatio/q estimation procedure, not solely regression. Let's mm our problem around and attempt to explain whether a car is manufactured outside the U.S. by its mileage rating and weight. To save paper, we will specify logit's nolog option, which suppresses the iteration log: . logit foreign mpg weight, nolog Logit estimates
Number of obs LR chi2 (2) Prob > chi2
= = =
74 35.72 0.0000
Log likelihood = -27.175156
Pseudo R2
=
0.3966
foreign
Coef.
mpg weight _cons
-.1685869 -.0039067 13.70837
Std. Err.
z
P>]z_
.0919174 .0010116 4.518707
-1.83 -3.86 _ 3.03
0.067 O.000 O.002
[95_. Conf. Interval] -.3487418 -.0058894 4.851864
.011568 -.001924 22. 56487
When you run linktest after logit,the result is another logit specification: linktest, nolog Logit estimates
Number of obs LR chi2(2) Pro5 > chi2
= = =
74 36.83 0.0000
Log likelihood = -26.615714
Pseudo R2
=
0.4090
foreign
Coef.
_hat _hatsq _cons
.8438531 -.1559115 .2630557
Std. Err. .2738759 .1568642 .4299598
z 3.08 -0.99 0.61
P>Iz_ 0.002 0.320 0.541
[95_ Conf. Interval] .3070661 -.4633596 -.57965
1.38064 .1515366 1.105761
The link test reveals no problems with our specification. If there had been a problem, we would have been Virtually forced to accept the misinterpretation of the link test we would have reconsidered our specification of the independent variables. When using logit, we have no control over the specification of the dependent variable other than to change likelihood functions. We admit to seeing a dataset once where the link test rejected the logit specification. We did change the likelihood function, re-estimating the model using probit, and satisfied the link test. Probit has thinner tails than logit. In general, however, you will not be so lucky. q
_3Technical Note You should specify exactly the same options with linktest as you do with the estimation command, although you do not have to follow this advice as literally as we did in the preceding example, logit's nolog option merely suppresses a part of the output, not what is estimated. We specified nolog both times to save paper. : '_
If you are testing a cox model with censored observations, however, you must specify the dead() option on linktest as well. If you are testing a tobit model, you must _pecify the censoring points
i
just as you do with the tobit;
command.
T
_
linktest
!
Specification linktest for single-equation models
207
i I . !
If youiare not sure which options are important, duplicate exactly what you specified on the command. estimatio_ _ If youido not specie' if exp or in range :with li_ktest, Stata will by default perform the link test _n the same s_unpleas the previous estimation. Suppose that you omitted some data when performin_ your estimation, but want to calculate the link test on all the data, which you might do if you belleved the moclelis appropriate for all !thedata. To do this, you would type 'linktest if
:{
e(
i
pl4) -=.'.
SavedRemrs linkt,mtsaves in ::(): Scalars r(t)
t statisticon _!aats_
r(df)degreesof freedom
linkt_stis not an estimation command in the sense that it leaves previous estimation results unchangeql.For instan@, one runs a regressiofi and then performs the link test. %,ping regress without a_gumentsarid the link test still replays the original regression.
i
In ternls of integrati g an estimation commafid with linktest, linktestassumes that the name of the estimation com_nand is stored in e(cmtt) and that the name of the dependent variable in e (depval_). After estirhation, it assumes that the number of degrees of freedom for the t test is given
i
by o(df_._)if the ma!ro is defined. If the estimation co_amandreports Z statistics instead of t statistics, then tinktestwill also report Z aatistics. TheiZ statistic, however, is still returned in r(t) and r(df) is set to a missing
i
vai ,e
I i
!
Methods ,nd ForMulas, linkt.=st is implemented as an ado-file. The link test is based on the idea that if a regression or
i , !
regressior-like equatioriis properlyspecified,on_ should not be able to find any additional independent variables :hatare signil_cantexcept by chance. One kind of specificationerr'or is called,a link error. In regression, this means that the dependent vmiable needs a transformation or "link function to
i !
properly relate to the i_dependent variables. Th_ idea of a link test is to add an independent variable to the eqt ation that is l_specialb likely to be significantif there is a link error,
i
Let
l
Y = f (X/3) be the m( del and _ be!the parameter estimatesl linktest
l } I I ! ! ,i
calculates
_hat= Xa and _hat_q= _hat2 The mod_l is then refit with these two variablesl and the test is based on the significance of _hatsa. This is tNe form suggelted by Pregibon (1979/based on an idea of Tukey (t949). Pregibon (1980) su_ests slightly different method that has tome to be known as "Pregibon s goodness-of-link tesf'. We _referredthe!olderversion because it is universally applicable, straightforward, and a good second-or ier approximation. It is universally applicable in the sense that it can be applied Io any' single-eq_,ationestimation technique whereas Pregibon s more recent tests are estimation-technique specific.!
|
i
=__
Pregibon, D. 1979. Data Analytic Methods for Generalized Linear Models. Ph.D. Dissertation 1980. Goodness of link tests for generalized linear modelS. Applied Statistics 29: 15-24. Tukey, J. W. 1949. One degree of freedom for non-additivity. Biometrics 5: 232-242.
Also See Related:
[R] estimation
commands,
JR] lrtest,
[R] test, [R] testnl
University of Toronto.
_ !
Title
i
1
I
list --
Sy ntax I!
f
i
I i l
'
_list
Iv_rlist! [i:fe_]
i
[n o]_display
nolabel
noobs doublespace
]
Descrlptic_n di ;plays the v
es of variables, If no v_rlist is specified, the values of all the variables are
_,lso see bro_vsein [R] edit.
displayed.
Options . [nojdisplgy forces th_format into display or tabular (nodisplay) format. If you do not specify one its judgment of which would be most one of t_ese two options, then Stata chooses based on re.adabk nolabel
[
[in range][,
by ... : mai, be used with kist; see [R] by. The second vadist may _ntain_ime-seriesoperators;see [U114.4.3"l'ime.seHesvaNi_s.
list I
I
, s v lues of variables
uses the nur eric codes rather than the label values (strings) to be displayed.
noobs sup _ressesprintiJ g of the observation numbers. doublesp_tce requests mt a blank line be inserted between observations. This option is ignored in displa format.
Remarks l
i° ! i
I ! I
list,t,. )ed by itself lists all the observations and all the variables in the dataset. If you specify
varlist, onl those vafia_tes are listed. Specifyifig one or both of in range and if e_p limits the observatiot listed.
:;
_ Examplei
list h. s two outpu
listing a f_w variables, whereas the display format is suitable for listing an unlimited number of variables. _tata chooses automatically between those two formats: Obse_ :vation 1 li_t in 1/2 make rep78 weight
I
formats, known as tabular and display. The tabular format is suitable for
I _ispla-t
AMC Concord 3 2,930 121
price headroom
4,099 2.5
mpg trunk
22 11
length
186
turn
40
gear_r-o
3.58
foreign
Domestic
Observation ri
'
I ,
--,-
2
.,,o..
_,.r._ vauu_ Ul vitrlODleS make AMC Pacer price rep78 3 headroom
weight
3,350
displa-t . list
make
258 mpg
weight
displ
make I. AMC Concord 2. AMC Pacer ;
3. AMC
The
Spirit
length
_
mpg trunk
17S
gear_r~o rep78
_
4,749 3.0
turn
2.53
40
foreign
Domestic
in I/5
mpg 22 17
weight 2,930 3,350
displa~t 121 258 121
rep78 3 3
22
2,640
4. Buick
Century
20
3,250
196
3
5. Buick
Electra
15
4,080
350
4
first case is an example
of display
17 II
format;
[he second
is an example
of tabular
format.
The
tabular format is more readable and takes less space, but it is effective only if the variables can fit on a single line across the screen. Stata chose to list all twelve variables in display format, but when the varlist was restricted to five variables, Stata chose tabular format. tf you are dissatisfied with Stata's choice, you can make the decision yourself. Specify the display option to force display format and the nodisplay option to force tabular format.
<1 0 Technical Note When Stata lists a string variable in tabular output format, it always lists the variable right-justified. When Stata lists a string variable in display output format, it decides whether to li st the variable rightjustified or left-justified according to the display forrnht for the string variable; see [U] 15,5 Formats: controlling how data are displayed. In our previous! example, make has a display format of 7.-18s. describe
make storage
variable
name
make
display
value
type
format
label
strl8
7.-18s
variable
label
Make
Model
and
The negative sign in the 7'-18sinstructs Stata to left+justify this variable. If the display format had been 7,18s, Stata would have fight-justified the variable. Note that it appears from our listing above that :foreign describe it, we see that it is not: describe
foreign
but if we
foreign storage
variable
is also a string variable,
name
display
value
type
format
label
variable
byte
7.8.Og
origin
Car type
label
foreign is stored as a byte, but it has an associated value label named origin;see [U] 15.6.3 Value labels. Stata decides whether to right-justify or left-justify a numeric variable with an associated value label using the same rule as Stata uses for stnng variables: it looks at the display format of the variable. In this case, the display format of 7.8. Og tells Stata to right-justify the variable. If the display format had been 7,-8.0g, Stata would have left-justified this variable.
[3
i
iI
i
i "
_
_
Xst -- List values of variables
_ Technical _ote
! You car_ list the v_riables in any order that you desire. When you specify the varlist, list makes the ttisplay in the order you specify. You 'may also include variables more than once in the vartist.
Example In some !cases, you m_y wish to suppress the Observation numbers. You do this by specifying the
lie
make
mpg wight
noons make opti,,n:
i
displ
foreign
mpg
weight
in 51/55
noobs
: displa-t
foreign
Pont.
Sunbird
24
2,690
151
Domestic
Audi Pont. Audi
_000 Phoenix ;ox
17 19 23
2,830 3,420 2,070
131 231 97
Foreign Domestic Foreign
BWW
.>Oi
25
2,650
121
Foreign
1 _ Example
1
You can Imake the list easier to read by specifOng the doublespaceoption:
I
i lis_ make make '
i
Pont. iPhoenix
19
3,420
231
Domestic
i
Pont. Igunbird
24
2,690
151
Domestic
Audi_000
17
2,830
131 Foreign
Audi
'ox
23
2,070
97
Foreign
BMW 3:!0i
25
2,650
121
Foreign
}
i } ! i_
mpg weight displ foreign in 51/55, noobs double mpg weight 4ispla~t foreign
21Technical Note
You can !suppress the use of value labels by specifying the nolabel option. For instance, the variable foreign in the _:xamples above really contains numeric codes. 0 meaning Domestic and 1 meaning Foreign.When you list the variable however, you see the corresponding value labels rather than the underlyin_ numeric code: lis_
foreign
I
51
iforeign _omestic
i
52.
_omesl;ic
I
211
53. 54.
iF°reign _Foreign
55.
IForeign
Specifying t e nolabel
in
1/55
i 1 1
ption displays the underlying numeric codes:
list
_!
212
_,
_
5i. 52. 53. 54. 55.
foreign
in
51/55,
nolabel
listforeign -- List values of variables 0 0 1 1 1
0
References Riley, A. R. 1993. dml5: Interactively list values of variable,s.Stata Technical Bulletin 16: 2-6. Reprinted in Stata TechnicalBulletin Reprints. vol. 3, pp. 37-41. Royston, P. and P. Sasieni. 1994. dml6: Compact listing Of a single variable. Stata Technical Bulletin 17: 7-8. Reprinted in Smta Technical Bulletin Reprints, vol. 3, pp_41-43. Weesie, J. t999. din68: Display of variablesin blocks. Stata TechnicalBulletin 50: 3-4. Reprinted in Stata Technical Bulletin Reprints. vol. 9, pp. 27-29.
Also See Related:
[R] edit, [P] display,
[P] tabdisp
i
Ins
! i
; i i i _I I
j i
"_
I
0 -- Find z
iit
1
_l
I
I
ire-skewness log or BoxLCox transform
lnske'_O ,,ewvar = ._xp [if exp] [in range] [, level(#) I
_delta(#)
_zero(#) 1
delta(#)
--
Syntax bcskei_O newvar = _.rp [if 1
e_,7_ ] [ill range] [ .
m
level(#)
--
--
zero(#)
] d
Deseripti n of inske_10creates n@var = ln(+exp - k). choosing k and the sign of exp so that the skewness newvar is zero. bcske_FO creates n 'vat= (exp _ - 1)/A, .the Box-Cox power transformation (B x and Cox 1964), ch_osing A so t_at the skewness of ned,vat is zero. exp must be strictly positi_°c. Also see
[R] boxeo
for maximu_n likelihood estimation of A
Options level (#) specifies the confidence level for a confidence interval for k (lnskewO) or A (bcskewO). Unlike usual, the ccnfidence interval is calculated only if level() is specified. As usual, # is _ecified as an integ,.'r; 95 means 95% confidence intervals. The level() option is honored onl>_ if the umber of observations exceeds 7. delta(#) specifies the increment used for calculating the derivative of the skewness function with respect to k (lnske'gO) or A (bcskewO). The default values are 0.02 for lnskewO and 0.0I for bcske_O. zero(#) s_ecifies a vah Eefor skewness to determine convergence that is small enough to be considered zero arld is by defau it 0.001.
Remarks
Example
1
Using dur automobih_ dataset (see [U] 9 Statai's on-line tutorials and sample datasets), we want to generatd a new variab le equal to ln(mpg- k) t6 be approximately normally distributed, mpg records the miles r gallon for _ach of our cars. One feature of the normal distribution is that it has skewness
• in_kewO lnmpg Transfor_
mpg k
[95Y,Cdnf. Interval]
Skewness
(not calculated)
-7.05e-06
....
in(mpg-k)
5.383659
214
InskewO-- Find zero-skewness log or Box-Cox transform
This created the new variable lnmpg = ln(mpg - 5.384): describe Inmpg
variable name
storage type
Inmpg
display format
float
value label
X9.0g
Since we did not specify the level we could have typed
variable label in(mpg-5.383659)
() option, no confidence
interval was calculated.
At the outset,
InskewO inmpg = mpg, level(95) Transform
I
In(mpg-k)
[
k 5.383659
[95_
Conf. Interval]
-17. 12339
Skewness
9.892416
-7.05e-06
The confidence interval is calculated under the assumption that In(mpgk) really does have a normal distribution. It would be perfectly reasonable to use Inskew0 even if we did not believe the transformed variable would have a normal distribution--if we literally wanted the zero-skewness transform--although in that case the confidence inte_'al would be an approximation of unknown quality to the true confidence interval. If we now wanted to test the believability of the confidence interval, we could also test our new variable lnmpg u!sing swilk with the !nnormal option. q
El Technical Note lnskewO (and bcskewO) reports the resulting skewness of the variable merely to reassure you of the accuracy of its results. In our above example, lnskew0 found k such that the resulting skewness was -7- 10-6, a number close enough to zero for all practical purposes. If you wanted to make it even smaller, you could specify the zero () option. Typing lnskewO new=mpg, zero (le-8) changes the estimated k to -5.383552 from -5.383659 and reduces the calculated skewness to -2.10 -11 When you request a confidence interval, it is possibl+ that lnskew0 will report the lower confidence interval as '. ', which should be taken as indicating the lower confidence limit kL = -oc. (This cannot happen with bcskew0.) As an example, consider a sample of size n on z and assume the skewness of z is positive, but not significantly so at the desired significance level, say 5%. Then no matter how large and negative you make kz,, there is no value extreme enough to make the skewness of ln(x - kL) equal the corresponding percentile (97.5 for a 95% confidence interval) of the distribution of skewness in a normal distribution of the same sample size. You cannot because the distribution of ln(z - kL) tends to that of zpapart from location and scale shift--as z --+ oc. This "problem" never applies to the upper confidence limit ku because the skewness of ln(:c - ku) tends to -co as k tends upwards to the minimum value of z.
Example In the above example, using lnskewO a natural
zero
and
we
are
shifting
variable such as temperature zero is indeed arbitrary.
that
with a variable like mpg is probably zero
arbitrarily.
measured in Fahrenheit
On
the
other
hrmd,
use
undesirable,
mpg has
of tnskew0
with
or Celsius would be more appropriate
a
as the
i i
_
lnskewO-- Find zerO-skewnesSlog or Box-Cox transform 215 For a var able like mpg it makes more sense touse the Box-Cox power transform (Box and Cox 1964): y(_)=
Y_-I
A is free to :ake on any ,,alue. but note that y(1) _ y_ bcskew0 works like 1:1skew0:
1. y(0) _ In(y), and yf-l_
= 1 - 1/y.
• bcs_ewO bcmpg = ipg, level(95) 4 i Transform (_pg'L-1)/L
L
[95_,Conf. Interval]
-. 3673283
-1. 212752
Skewness
,4339645
0001898
,i
It is worth n( ting that the _ _,%confidence interval i,cludes ), = - ] (A is labeled L in the output), which has a rather nore pteasin_ interpretation--gallons iper mile--rather than (mpg-'3673 - 1)/(-3673).• The confide;_ce interval, however, is calculated under the assumption that the power transformed
_
variable is rormally distributed. It makes perfect sense to use bcskewO even when one does not believe that the transforrred variable will be normallv distributed, but in that case the confidence interval is an approximaticn of unknown quality, ffone believes thai the transformed data are normally <1
i
distributed, me can alterwatively use boxcox to egtimate 3,: see [R] boxeox.
Saved ReslJits .
lnskewO and bcskew(, save in r() Scalars
!
mma)
h-(InskewO)
r(g_ !} },.
r (I__bda) r(ll_) r(ul_)
)_(bcskewO) lower bound of tonfidence interval upper lxmnd of confidence interval
r(sl:ewness)
resulting skewness of transformed
variable
Methods armdFormulas lnskewOiand
bcskew(
are implemented as ad0-files.
i
Skewness is as calcul_ :ed by summarize: see [R] summarize. Newton's method with numeric. uncentered cerivatives is _sed to estimate k (lnskew0) and )_ (bcskew0?. In the case of lnskew0.
i 1
lhe initial value i_ chosen so that the minimum of :r - k is l and thus ln(z with )_ = 1
Acknowle lnskewOiand
k) is 0. bcskewO star1_
ment bcskewC were written by Patrick Royston of the
MRC
Clinical Trial< [_nit. London.
216
Inskew0 m Find zero-skewness log or Box-Cox transform
References Box. G. E. E and D, R. Cox. 1964. An analysis of transformations. Journal of the Royal Statistical Society, Series B 26: 211-243.
Also See Related:
[R] boxcox, JR] swilk
Complementary:
[R] ladder
•
Title log -- E_:ho copy of
!
Syntax:
_
l_og
!
_
usi,g
to file or device
filename
, append
{ol
!
_eplace
[ Zext
smcl ] ]
}
cmdl og cmdlog _sing filenal,e [, append replace ] I
cmdlog
!
set
I
k
{onlofflcl
log_ype
{text_
set lin_size
lffilename
se}
smcl}
#
is ._pecified withoul an extension, one of the suffixes .smcl..log.
The extensibn
.smct
or .txt
is added.
or ,] og is added by log depending on whether the file format is SMCL or ASCII text
The extensibn .txt is add,',dby .cmdlog. In addition to 'ommandlog. _oumay accessthe capabilitie,_of log by pullingdow_ File and choo_ingLog.
Description
[
log allowis you to mal]e a full record of your Stata session. A log is a file containing what you !_
type a_d Staia's output. 1 cmdlog 4lows you to!make a record of what you type during your Stata session. A command
; i
log contains bnly what yo type and so is a subse! of a full log. You can rOake full logs md command logs simttltaneously, one or the other, or neither. Neither is
!
produced un_l you tell St;.ta to start I¢>gging. Command logs are ah,,ays straight ASCII text files and this makes them easy to convert into
, !
do-files. (In ihis respect, it would make more sen.4e if the default extension of a command log file was .do be&use commaz]d lo_osare do-files. The default is .txt. nOt .do. howe_er, to keep you
i i
from accidenialty overwriting your important do-files.) Full logs !are recordedlin one of two formats: SMCL (Stata Markup and Control Language) or
_° i
text (meaning ASCII). The default is SMCL. but s_t logtype can change that, or you can specify an option to state the forrrm you wish. We recommend SMCL because it preserves fonts and colors. SMCL logs c_n be convert,_d to ASCII text or to other formats using the translate command: see [R] translate; translate can also be used to produce printable versions of SMCL IO_S.or you can print SMCL l_gs by pullin_ down File and choosing Log. SMCL logs can be viewed in the viewer, as can any file: !see [R] view.
: ! i
21_
_
zl _
tog -- Ec.o copy of session to file or device
log or cmdlog,
typed without arguments, reports the status of logging.
log using and cmdlog using open a log file. log close and cmdlog close close the file. Between times, log off and cmdlog off, and log on and cmdlog on can temporarily suspend and resume logging. set logtype specifies the default format in which full logs are to be recorded. Initially, full logs are set to be recorded in SMCL format. set linesize specifies the width of the screen currently being used (and so really has nothing to do with logging). Few Stata commands currently respect linesize, but this will change in the future.
Options Optionsfor use with both log and logcmd append specifies that results are to be appended onto the end of an already existing file. If the file does not already exist, a new file is created. replace specifies that filename, if it already exists, is to be overwritten, and so is an alternative to append. When you do not specify either replace or append, the file is assumed to be new and an error message is issued and logging is not started.
Options for use with log text and smcl specify the format in which the log is to be recorded. The default is complicated describe but is what you would expect: If you specify the file as filename.smcl, (regardless of the value of set logtype).
to
then the default is to write the log in SMCL format
If you specify the file asfilename, log, then the default is to write the log in text format (regardless of the value of the set logtype). If you type filename without an extension and specify neither the smcl or text options, the default is to write the file according to the value of set logtype. If you have not reset set logtype, then that default is SMCL. In addition, the filename you specified will be fixed to read filename, sracl if a SMCL log is being created or fiiename, log if a text log is being created. If you specify either of the options text or smcl, then what you specify determines how the log is written. Iffilename was specified without an extension, the appropriate extension is added for you.
Remarks For a detailed explanation
of logs, see [U] 18 Printing
and preserving
output.
Note that when you open a fulI log, the default is to show the name of the file and a time and date stamp: log
using
log
log: type :
opened
L
on:
myfile
C: \data\proj smcl 12 Dec
2000,
Ikmyfile. 19:28:23
smcl
log _ Echo copy of sessionto file or device
i
219
The above information ' ,'ill appear in the log. If you do not want this information to appear, precede
i
the comm_ nd by quiet . qu etly
l
quietly
!
Ly:
log using myfile
'_ill not suppr,;ss an}, error messages qr anything else you need to know.
i
Simila@ when you :lose a fuel log, the default is to show the full information:
i
. lo*
I
i
close
i log- c:\_t_\proj l\_y_ile, s_l
clo!ed on
12
c 2000,
12:32:41
and lhat information wili appear in the log. If you want to suppress that, type quietly log close,
i I
SavedReSults log
and cmdlog sav_ in r ()" Macros
i
r (filename) name of file I
AlsoSee
} _ i {
I
! i
r(s_atus)
on or off
r(type)
text or smcl
i
Complemehtary:
[Ri translate; [R] more, [R] query
Baekgrounh:
17 Logs: Printing and saving output [G:;W] 17 Logs: Printing and saving output, [G:;U] 17 Logs: Printing and saving output, [G!M]
[U 10 -more-- conditions, [Ui 14.6 File-naming conventions, [UI 18 Printing and preserving output
' 1
Title [ I°gistic
-- L°gisfic , regressi°n
,
i
t
Syntax logistic
depvar varlist [weight]
cluster
(varname)
maximize_options lfit
[depvar] all
lstat lroc all
[depvar]
genprob
asis
[if
exp]
[in
range]
[if exp] _in range]
[weight]
[if
(varname)
coef
[, group(#)table
outsample
expl
[in
[. cutoff(*)all
range]
[, nograph
beta(mamame)
]
graph_options
]
[weight]
beta(matname)
offset
robust
]
[weight]
(varname)
[, _level(#)
]
[weight]
beta(mamame)
Isens [depvar]
all
score (newvarname)
beta(matname) [depvar]
[i£ exp] [in range]
[if
gensens
expl
[in
(varname)
range]
[. nograph
genspec
(varname)
graph_options replace
]
by ... : may be used with logistic; see [R] by. logistic allows fweights andpweights; lfit, lstat, lro¢, and lsens allow only fweights; see [U] 14.1.6weight. logistic shares the features of all estimation commands; see [U] 23 Estimation and post-estimation commands. logistic may be used with sw to perform stepwise estimation; see [R] sw.
(Continued
on next page)
220
yntax fir predict predict
[type] ,ewvarname [if exp] [in range] [, statistic rules
asif
nooffset
]
where slatistic is p xb strip * d_eta * deviance
, ___2 , ddeviemce , hat , number r,esiduals , rstandard
probability of a positive outcome (the default) xib, fitted values standard error of the prediction Pregiborl(198t) A 13influence statistic deviance residual Hosmer and Lemeshow (1989) A X2 influence statistic Hosmer and Lemeshow (1989) A D influence statistic Pregibon (1981) leverage sequential number of the covariate pattern Pearson residuals; adjusted for number sharing covafiate pattern standardized Pearson residuals: adjusted for number sharing covariate pattern
Unstarred statistics are available both in and out of sample; type predict ... if e(sataple) ... if wanted only for the estimation sample. Starred statistics are calculated only for the estimation sample even when if e(sample) is not specified.
DeScription logisticesumates a logistic regression of _lepvaron vartist, where depvar is a 0tl variable lot, more precisely, a 0/non-0 variable). Withoutarguments, logistic redisplays the last logistic estimates, logistic displays estimatesas odds ratios; to view coefficients,type logit after running logistic. To obtain odds ratios for any covariate pattern relative to another, see JR] lineom. ].fi_ displays either the Pearson goodness-of-fit test or the Hosmer-Lemeshow goodness-of-fit test is'_at displays various summary statistics, including the classification table, lroc graphs and calculates the area under the ROe curve. lsens graphs sensitivity and specificity versus probability cutoff and optionally creates new variables containing these data lfit, lstat, lroc, and lsens can produce Statisticsand graphs either for the estimation sample or for;any set of observations. However, they always use the estimation sample by default. When weights, if, or in are used with logistic, it ig not necessary to repeat them with,these commands when you want statistics computed for the estimition sample. Specify if, in. or the all option onb,' whe_nyou want statistics computed for a set of obsen_ationsother than the estimation sample. Specify wmghts (only fweights are allowed with these commands) only when you want to use a different set oftweights. Bydefault, if it. lstat, lroc, and lsens use the fastmodelestimated by logistic. Alternatively, the model can be specified by inputting a vector of coefficients with the beta() option and passing the name of the dependent variable depvar to ttie commands, The lfit,
lstat,
lroc. and lsens commands may also be used after logit
or probit.
Here is a list of other estimation commands that may be of interest. See |R] estimation commands for a complete list. See Gould (2000_for a discussion of the interpretation of logistic regression.
I)
222
lOgistic --
LOgiStiC regression
blogit
[R] glogit
Maximum-likelihood logit regression for grouped data
bprobit
[R] glogit
Maximum-likelihood probit regression for grouped data
clogit
[R] ciogit
Conditional (fixed-effects) logistic regression
cloglog
[R] cloglog
Maximum-likelihood complementary log-log estimation
glra
[R] glm
Generalized linear models
glogit
[R] glogit
Weighted least-squares togit regression for grouped data
gprobit
[R] glogit
Weighted least-squares probit regression for grouped data
heckprob
[R] heekproh
Maximum-likelihood probit estimation with selection
hetprob
[R] hetprob
Maximum-likelihood heteroskedastic probit estimation
logit
IR] logit
Maximum-likelihood logit regression
mlogit
[R] mlogit
Maximum-likelihoo_l multinomial (polytomous) logistic regression
nlogit
[R] nlogit
Maximum-likelihood nested logit estimation
ologit
[R] ologit
Maximum-likelihood ordered logit regression
oprobit
[RI oprobit
Maximum-likelihood ordered probit regression
probit
[R] probit
Maximum-likelihood probit regression
scobit
[R] scobit
Maximum-likelihood skewed logit estimation
svylogit
[R] svy estimators
Survey version of logit
svymlogit
[R] svy estimators
Survey version of mlogit
svyologit
[R] svy estimators
Survey version of ologit
svyoprobit
[R] svy estimators
Survey version of oprobit
svyprobit
[R] svy estimators
Survey version of probit
xtclog
[R] xtclog
Random-effects and population-averaged ctoglog models
xtlogit
[R] xtlogit
Fixed-effects, random-effects, and population-averaged
xtprobit
[R] xtprobit
Random-effects and population-averaged
xtgee
[R] xtgee
GEE population-averaged generalized linem: models
logit models
probit models
Options Optionsfor logistic level
(#) specifies
the confidence
or as set by set
level;
level, in percent,
for confidence
see [U] 23.5 Specifying
the width
intervals.
Tile default
of confidence
is level
(95)
intervals,
robust specifies that the Huber/White/sandwich estimator of variance is to be used in place of the traditional calculation; see [U] 23.11 Obtaining robust variance estimates, robust combined with cluster() be independent
allows between
If you specify
pweights,
cluster(varname) not necessarily cluster(personid) estimated estimated
observations clusters).
which
are not independent
robust
is implied;
see [U] 23.13
specifies within
that
groups, in data
the
observations
Weighted
independent
(although
they must
estimation. across
groups
varname specifies to which group each observation with repeated observations on individuals, cluster()
standard errors and variance-covariance coefficients; see [U] 23.11 Obtaining
used with pweights to produce command in [R] svy estimators
are
within cluster
matrix of the estimators robust variance estimates,
(clusters)
but
belongs; e.g., affects the
(VCE) but not the cluster() can be
estimates for unstratified cluster-sampled data, but see the svylogit for a command designed especially for survey data.
logistic-- Logisticregremsion c_hister () implies robust' by itself.
specifying robust cluster()
is equivalent to typing cluster
223 ()
scorei(newvarname) creates newvar containing uj = 01nLj/0(xjb) for each observation j in the sample. The score vector is _ 01nLj/ab = _ujxj; i.e., the product of ne_'var with each covariate summed over observations. See [U] 23.12 Obtaining scores. asis forces retentionof perfectpredictor variables and their associatedperfectly predictedobservations and may produce instabilities in maximization; see [R] probit (sic). offset (varname) specifies that varname is to be included in the model with coefficientconstrained tolJe 1. coef causes logistic to report the estimated coefficients rather than the ratios (exponentiated coefficieJas),coef may be specified when the _odel is estimated or used later to redisplay results. c0ef affects only how resuks are displayed ahd not how they are estimated. marimize_options control the maximization process; see [RI maximize. You should never have to specify Uhem.
'
Options!forlilt, Istat,troc,andIsens group(#) (ifit onl_y)specifies the number of quantiles to be used to group the data for the Hosmer-Lemeshow goodness-of-fit test. groqp(lO) is typically specified. If this option is not _iven, ttie Pearson goodness-of-fit test is computed using the covariate patterns in the data as groups.
I_
table (If it only) displays a table of the groups used for the Hosmer-Lemeshow or Pearson goodness-of-fit test '_,ithpredicted probabilitieS,observed and expected counts for both outcomes. anldtotal_ for each group, oulzsample (lfit only) adjusts the degrees of freedom for the Pearson and Hosmer-Lemeshow goodness-of-fittests for samples outside of the estimation sample. See the section Samples other thsn_estimation sample later in this entry. all requests that the statistic be computed for all observations in the data ignoring any if or in restrictions specified with logistic. beta(matn_lme) specifies a row vector containing coefficients for a logistic model. The columns of the row vector must be labeled with the corresponding names of the independent variables in the data. The dependent variable depvar must be specified immediately after the command name. See the section Models o/her than last estimated model later in this entry. cutoff (#) (1star only) specifies the value for determining whether an observation has a predicted positive outcome. An observation is classified as positive if its predicted probability is > #. The default is_0.5. nograph (1roe and lsens) suppresses graphical output. eraph_options (1roe and lsens_ are any of the options allowed with graph, lzwoway;see [G] graph options. genprob (va'rname). gensens (varname), and gelaspec (varname) (lsens only) specily the names of new variables created to contain, respectively, the probability cutoffs and the corresponding ser_sitivityand specificity. replace (lsens only) requests tha) if existing variables are specified for genprob(), or geaspec (), they should be ovem'ritten.
'i
gensens(),
1
Optionsfor predict p, the default, calculates the probability
of a positive outcome.
xb calculates the linear prediction. std_p calculates the standard error of the linear prediction. dbeta calculates the Pregibon (1981) A_ influence statistic, a standardized measure of the difference in the coefficient vector due to deletion of the observation along with all others that share the same covariate pattern. In Hosmer and Lemeshow (1989)jargon, this statistic is M-asymptotic, that is, adjusted for the number of observations that share the same covariate pattern. deviance calculates the deviance residual. dx2 calculates the Hosmer and Lemeshow (1989) AX2 influence statistic reflecting the decrease in the Pearson X2 due to deletion of the observation and all others that share the same covariate pattern. ddeviance calculates the Hosmer and Lemeshow (1989) AD influence statistic, which is the change in the deviance residual due to deletion of the observation and all others that share the same covariate pattern. hat calculates the Pregibon (1981) leverage or the diagonal elements of the hat matrix adjusted for the number of observations that share the same covariate pattern. number numbers the covariate patterns observations with the same covariate pattern have the same number. Observations not used in estimation have number set to missing. The "first" covariate pattern is numbered l, the second 2, and so on. residuals calculates the Pearson residual as given by Hosmer and Lemeshow for the number of observations that share the same covariate pattern.
(1989) and adjusted
rstandard calculates the standardized Pearson residual as given by Hosmer and Lemeshow and adjusted for the number of observations that share the same covariate pattern.
(1989)
rules requests that Stata use any "rules" that were used to identify the model when making the prediction. By default, Stata calculates missing fot excluded observations. See JR] legit for an example. asif requests that Stata ignore the rules and the exclusion criteria and calculate predictions for all observations possible using the estimated parameter from the model. See [R] logit for an example. nooffset is relevant only if you specified offset (vamame) for logistic. It modifies the calculations made by predict so that they ignore the Offset variable: the linear prediction is treated as x3b rather than x ab + offset a.
Remarks Remarks are presented under the headings logistic and logit Robust estimate of variance lilt lstat lroc lsens Samples other than estimation sample Models other than last estimated model predict after logistic
]
1 225
- uxjmtm lOgisciadd Iogit i
logistic provides an alternative and preferr_ way to estimate maximum-likelihood logit models, the other Choice being logit described in [R] iogit, First, let us dispose of some confusing terminology. We use the words logit and logistic to mean the same thing: maximum likelihood estimation. To some, one or the other of these words connotes trarisfOrming the dependent variable and using weighted least squares to estimate the model, but that is riot ho'& we use either word here. Thus, the logit and logistic commands produce the same res_tlts, The logistic command is generally preferred to logit because logistic presents the estimates in terms 6f odds ratios rather than coefficients. To a few, this may seem a disadvantage, but you can type logb:t without arguments after logistic to see the underlying coefficients. Nevertfieless. [R] log'it is still worth reading because logistic shares the same features as logit. incl_ud_ngomitting variables due to collinearity or one-way causation. For an introduction to logistic regression, see Lemeshow and Hosmer (1998) or Pagano and Gauvreau (2000, 470-487); for a thorough discussion, see Hosmer and Lemeshow (t989: second edition_ foghcoming in 2001).
> Example Colisidtr the following dataset from a study of risk factors associated with low birth weight des¢ribed ]n Hosmer and Lemeshow (1989, appendix 1). ., ddscribe Contains data from Ibw.dta ob_: 189 ,vaz]s: Ii :size:
_ari_ble name . ]
3,402 storage type
Hosmer _ Lemeshow data 18 Jul 2000 16:27
(95.!% of memory f_ee) di6play format
valu_ label
variable label
race
fd
int
_8,0g
identification code
]_bw v(ge l%_t Z_tce
byte byte int byte
_8.Og XS.0g _8.Og _8.0g
birth weight<25004 age of mother weight at last menstrual period race
s_nok_ 1_2 _t f_v
b_e byte byte byte byte
_8.04 _8.04 _8.04 _8.04 _,8.04
b_t
int
_8.04
, !
smoked during pregnancy prematttrelabor history (count) has history of hypertension presence, u_erine irritability number of visits to physician during let trimester birth weight (grams)
Sbrt_d by :
They _ant!to investigate thecausesoflow bi_hWeightInthis dataset, race isa categorical variable indicating _vhether a person is white (race = 1), black (race --- 2), or other (race -- 3). The authors want irldichtor_(dummy) variables for race included in the regression. (One of the dummies, of course, 'mu)st be omitted.) Thus, before we can _stimate the model, we must create the race dummy The_e ale a number of ways we could do this.: but the easiest is to let another Stata command, xi. do i! fdr uI. we type xi: in front of our logistic command and in our varlist include not race ) l
;; ._i:i
_o i. race,ug,suc -- cogmuc regression but to indicate we want the indicator
variables for this categorical
variable;
see [R] xi for
the full details. . xi: logistic low age lwt i.race smoke ptl ht ui i.race _Irace_l-3 (naturally coded; _Irace_l omitted) Legit estimates
Log likelihood =
-100.724
Number of obs LK chi2(8) Prob > chi2
= = =
189 33.22 0.0001
Pseudo K2
=
0.1416
,r
low
Odds Katie
Std. Err.
age lwt _Irace_2 _Irate_3
.9732636 .9849634 3.534767 2.368079
.0354759 .0068217 1.860737 1.039949
ptl smoke ht ui
1.719161 6.249602 2.517698 2.1351
.5952579 4.322408 1.00916 .9808153
z
P>lz[
[95_ Conf. Interval]
-0.74 -2.19 2.40 1.9_
0.457 0.029 0.016 0.050
.9061578 .9716834 1.259736 1.001356
1.045339 .9984249 9.918406 5.600207
1.5: 2.36i 2. 1.6_
0.118 0.021 0.008 0.099
.8721455 1.611152 1.147676 .8677528
3.388787 5.523162 24.24199 5.2534
The odds ratios are for a one-unit change in the variable. If we wanted the odds ratio for age to be in terms of 4-year intervals, we would gen age4 = age/4 . xi: logistic Io_ age4 lwt i.race smoke ptl ht ui (ou_utomit_d)
After logistic,
we can type logit
to see the mode in terms of coet_cients
and standard errors:
logit Legit estimates
Log likelihood =
-100.724
low
Coef.
age lwt _Irace_2 _Irace_3 smoke
-.0271003 -.0151508 1.262647 .8620792 .9233448
ptl ht ui cons
.5418366 1.832518 .7585135 .4612239
Std. Err. .0364504 .0069259 .5264101 .4391531 .4008266
z
Number of obs LRchi2(8) Prob > chi2
= = =
189 33.22 0.0001
Pseudo R2
=
0.1416
P>Jzl
[95_ Conf. Interval]
-0.74 -2.t9 2.40 1.96 2.30
0.457 0.029 0.016 0.050 0.021
-.0985418 -.0287253 .2309024 .0013548 .1377391
.0443412 -.0015763 2.294392 1.722804 1.708951
.346249 .6916292 .4593768
1.56 2.65 1.65
0.118 0.008 0.099
-.136799 .4769494 -.1418484
1.220472 3.188086 1.658875
1.20459
0.38
0.702
-t.899729
2.822176
If we wanted to see the logistic output again, we would type logistic without arguments. <3
> Example You can specie' the confidence interval for the odds ratios with the level () option, and you can do this either at estimation time or when you replay the model. For instance, to see our previous models with narrower, 90% confidence intervals,
_
logistic-- Logistic reg_
!
• lqgistic, Logft
227
level(90)
estimates
Log likelihood
=
-100.724
Number of obs LR chi2 {8)
= =
Prob
=
0,0001
=
O. 1416
> chi2
Pseudo
R2
189 33.22
Robust low
Odds
age lwt _Irate
2
Ratio
Std.
Err.
.9732636 .9849634
.0329376 .0070209
z -0.80 -2.13
P>Izl
[95_, Conf.
Interval]
O. 423 O. 034
.9108015 .9712984
1.040009 .9988206
3.534767
I.793616
2.49
O.013
I.307504
9.556051
_Irace_3
2.368079
I.026563
1.99
O.047
i.012512
5. 538501
smoke
2. 517698
,9736416
2.39
O. 017
1.179852
5,372537
ptl ht
1. 719161 6.249602
.7072902 4.102026
1.32 2.79
O. 188 O. 005
.7675715 1. 726445
3. 850476 22. 6231
ul
2.1351
I. 042775
1.55
O. 120
.8197749
5. 560858
<]
RobuStestimateof variance If you specify robust. Stata reports the robust estimate of variance described in [U]23,11 Obtaining rob_ist variance estimates Here is the model previously estimated with the robust estimate of variance: xi: logistic
LOgi_
smoke _ptl ht
low age lwt i.race
i.rate
_trace_l-3 estimates
i_)g likelihood
ui, robust
(liaturaally coded;
_Irace_1
omitted)
Number of obs _ald chi2 (8)
=
-100.724
= =
189 29.02
Proh > chi2
=
0. 0003
Pseudo
=
0.1416
R2
Robust low
Odds Ratio
Std.
Err.
z
P>}zl
[957, Conf.
Interval]
0.423 0.034
.9108015 .9712984
1.040009 ,9988206
r
age lwt iIrace_2 JIrace_3 smoke ptl ht ui
.9732636 .9849634
.0329376 .0070209
-0.80 -2, 13
3. 534767 2. 368079
I.793616 1.026563
2.49 i.99
0.013 O.047
I.307504 1.012512
9.556051 5.538501
2. 517698
.9736416
2.39
0. 017
1.179852
5.372537
1.7t9161
.7072902
1.32
O. 188
.7675715
3. ff50476
6,249602 2. 1351
4. 102026 1.042775
2,79 I. 55
0.005 O. 120
1.726445 .8197749
22.6231 5.560858
Additionally, robust allows you to specify cluster() and is then able, within cluster, to relax the assumpiion of independence. To illustrate this, we have made some fictional additions to the low-birth-Weight data. Pretend [hat these data are not a random sample of mothers but instead are a random sample of mothers+from a random sztmple of hospitals. In fact, that may be true--we do not know the history of these_dam but we can pretend in any case.
i i
H0spital$ specialize and it would not be too incorrect to say that some hospitals specialize in more difficult cases. We are going to show two extremes. In one, all hospitals are alike but we are going to estimate bnder the possibility that they might differ. In the other, hospitals are strikingly different In bc/_hCases, we assume patients are drawn from 20 hospitals. In both examples, we will estimate the same model and we will type the same command to estim_ate!it. !Below are the same data we have been using but with a new variable hospid, which ident_fie_ frbm which of the 20 hospitals each patient was drawn (and which we have made up):
_r.o
" F_
,ug,_uu-
Logm.c
regressl0N
. xi: logistic low age lwt i.race smoke ptliht ui, robust cluster(hospid) i.race _Irace_1-3 (naturally coded; _Irace_l omitted) Logit estimates
Log likelihood =
-100,724
Number of obs Wald chii(8) Prob > chi2
= = =
189 49.67 0.0000
Pseudo R2
=
0.1416
(standard errors adjusted for clustering on hospid) Robust low
Ratio
Std. Err.
age lwt _Irace_2 _Irace_3 smoke
.9732636 .9849634 3,534767 2.368079 2.517698
.0397476 .0057101 2.013285 .8451325 .8284259
ptl ht ui
1.719161 6.249602 2,1351
.6676221 4.066275 1o093144
z
P>_zl
[957 Conf. Interval]
-0.66 -2.6!1 2.2_ 2.42 2.81
0,507 0,009 0.027 0.016 0.005
.898396 .9738352 1,157563 1.176562 1.321062
1.05437 .9962187 10,79386 4.766257 4.79826
1.40 2.82 1.48
0.163 0.005 0,138
.8030814 1,745911 .7827337
3.680219 22.37086 5.824014
The standard errors are quite similar to the standard etTors we have previously obtained, whether we used the robust or the conventional estimators. In this example, we invented the hospital ids randomly. Here are the results of the estimation with the same data but with a different set of hospital ids: . xi: logistic low age lwt i.race smoke ptl ht ui, robust cluster(hospid) i.race _Irace_l-3 (naturally coded; _Irace_1 omitted) Logit estimates
Log likelihood =
-100.724
Number of obs Wald chii(8) Prob > chi2
= = =
189 7,19 0.5167
Pseudo R2
=
0.1416
(standard errors adjusted for clustering on hospid) Robust Std. Err.
low
0dds Ratio
age lwt _Irace 2 _Irate 3 smoke
.9732636 .9849634 3.534767 2.368079 2.517698
.0293064 .0106123 3.120338 1.297738 1.570287
ptl ht ui
1.719161 6.249602 2.1351
.6799153 7.165454 1.411977
z
P>[zJ
[957 Conf. Interval]
-0,90 -1,41 1.43 1.57 1.48
0.368 0.160 0.153 0,116 0.139
.9174862 ,9643817 .6265521 .8089594 .7414969
1.032432 1.005984 19.9418 6.932114 8.548654
1.37: 1.60 1.15
0.171 0.110 0.251
.7919046 .660558 .5841231
3.732161 59.12808 7.804266
Note the strikingly larger standard errors. What happened? In these data, women most likely to have low-birth-weight babies are sent to certain hospitals and the decision on likeliness is based not just on age, smoking history, etc., but on other things that doctors can see but that are not recorded in our data. Thus, merely because a woman is at one of the centers identifies her to be more likely to have a low-birth-weight baby. So much for our fictional example. The rest of this' section uses the real low-birth-weight data. To remind you, the last model we left off with was
r"
logistic-- Loglstlcregression i
i •
229
,
• Xi: logistic low age lwt i.race smoRe ptl ht ui i._ace _Irace_1-3 (naturally coded; _Irace_l omitted) Logit estimates
Log likelihood =
Number of obs LR chi2(8) Prob > chi2 Pseudo R2
-100.724
low
Odds Ratio
age l'.*t _!race_2 _Irace_3 smoke
.9732636 .9849634 3.534767 2.368079 2. 517698
.0354759 .0068217 1.860737 1.039949 1. 00916
1.719161 6.249602 2.1351
.5952579 4.322408 .9808153
ptl ht ui
Std. Err.
z -0.74 -2.19 2.40 1.96 2.30 I.56 2.65 1,65
= = = =
189 33.22 0.0001 0.1416
P> iz I
[95_ Conf. l_terval]
O.457 O.029 O.O16 O.050 0.021
.9061578 .9716834 1.259736 1.001356 1. 147676
t. 045339 .9984249 9.918406 5.600207 5. 523162
O.118 0.008 O. 099
.8721455 1.611152 .8677528
3,388787 24.24199 5. 2534
lilt Ifit Computes i goodness-of-fit tests, either the Pearson X2 test or the Hosmer-Lemeshow i i
test.
By _de(ault, lfit. Istat,lroc, and lsenS compute statistics for the estimation sample using the llast rdodel estimated by logistic. However, samples other than the estimation sample can be spetifibd;t'l.seethe section Samples other than esflmation sample later in this entry. Models other thanthe )last mbdel estimated by logistic can also be specified; see the section Models other than last estimated )model
> Example
i 1
lfi_c, fyped without options, presents the Pearson X2 goodness-of-fit test for the .estimated model. The: Pdarsbn k 2 goodness-of-fit test is a test of:the observed against expected number of responses usir_g ¢elI_ defined by the covafiate patterns; see predict with the numbers option below for the defiiaitibn bfcovariate patterns. ._if_.t L_gi_tic model for low, goodness-of-fit test I number of observations = 189 _umi_r of covariate patterns = Pearson chi2(t73) = Prob > chi2 =
182 i_9.24 0.3567
Our mddell fits reasonably well. We should note, however, that the number of covafiate patter