2151_C000.fm Page i Thursday, October 26, 2006 1:39 PM
Stream of Variation Modeling and Analysis for Multistage Manufa...
121 downloads
1796 Views
12MB Size
Report
This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
Report copyright / DMCA form
2151_C000.fm Page i Thursday, October 26, 2006 1:39 PM
Stream of Variation Modeling and Analysis for Multistage Manufacturing Processes
2151_C000.fm Page ii Thursday, October 26, 2006 1:39 PM
2151_C000.fm Page iii Thursday, October 26, 2006 1:39 PM
Stream of Variation Modeling and Analysis for Multistage Manufacturing Processes Jianjun Shi University of Michigan, Michigan
Boca Raton London New York
CRC Press is an imprint of the Taylor & Francis Group, an informa business
2151_C000.fm Page iv Thursday, October 26, 2006 1:39 PM
CRC Press Taylor & Francis Group 6000 Broken Sound Parkway NW, Suite 300 Boca Raton, FL 33487-2742 © 2007 by Taylor & Francis Group, LLC CRC Press is an imprint of Taylor & Francis Group, an Informa business No claim to original U.S. Government works Printed in the United States of America on acid-free paper 10 9 8 7 6 5 4 3 2 1 International Standard Book Number-10: 0-8493-2151-4 (Hardcover) International Standard Book Number-13: 978-0-8493-2151-1 (Hardcover) This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use. No part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers. For permission to photocopy or use material electronically from this work, please access www. copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC) 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment has been arranged. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe. Library of Congress Cataloging-in-Publication Data Shi, Jianjun, 1963Stream of variation modeling and analysis for multistage manufacturing processes / Jianjun Shi. p. cm. Includes bibliographical references and index. ISBN 0-8493-2151-4 1. Manufacturing processes--Statistical methods. I. Title. TS183.S524 2006 658.5--dc22 Visit the Taylor & Francis Web site at http://www.taylorandfrancis.com and the CRC Press Web site at http://www.crcpress.com
2006049164
2151_C000.fm Page v Thursday, October 26, 2006 1:39 PM
Preface The aim of this book is to summarize major achievements in stream of variation (SoV) methodology research and implementation in various manufacturing processes. The SoV methodology focuses on developing and implementing a unified, systematic, and generic methodology for variation management and reduction of multistage manufacturing processes (MMPs). An MMP is a complex manufacturing system with common characteristics as: (1) multiple operations/stations are involved to produce a product, (2) the product quality can be quantitatively characterized by a set of features or attributes, and (3) the product quality deviations/variations are contributed by the errors generated at the current station, as well as the accumulated errors transmitted from previous stations. The major challenges of variation reduction in an MMP are to model, analyze, control, and reduce the process-induced variation on final products. In an MMP, the output-workpiece deviation of one station is the input-workpiece deviation of the next station. Each station adds design inherent process variation due to tooling tolerance when no fault occurs, and special assignable process variations due to tooling errors when a fault occurs, to the part variation. The final product variation is an accumulation of variations from all stations. Due to the complexity of an MMP, the variation propagation depends on both product design and process design. The variation stack-up is normally not simply additive, and could be nonlinear. This book addresses those fundamental issues by developing the unified methodology, which includes: (1) model variation propagation in an MMP using a state space model, (2) expanding the concepts of system and control theory for variation management through design synthesis and optimization to achieve an affordable minimum variability at the design stage, (3) developing statistical methods driven by engineering models for root cause identification and quick failure recovery in the manufacturing stage; and (4) integrating process tool degradation, reliability model, and product quality information for prognostics and defect prevention to maintain the minimum variability throughout the manufacturing lifetime. The book can be used as a major reference or text book for researchers, engineers, and students who are interested in manufacturing design and analysis, manufacturing process control, advanced quality control, industrial statistics, or general areas of manufacturing process modeling, monitoring, diagnostics, and control.
OUTLINE OF THE BOOK The book is organized into 5 parts and 18 chapters. A brief summary of those 5 sections is summarized below:
2151_C000.fm Page vi Thursday, October 26, 2006 1:39 PM
•
•
•
•
•
Part I (Chapter 2 to Chapter 5) provides a review of some bases of matrix theory and multivariate statistics, which will be used in later sections in the SoV study. Topics include basics of matrix theory (Chapter 2), multivariate statistics (Chapter 3), statistical inference (Chapter 4), and principal component analysis and factor analysis (Chapter 5). Part II (Chapter 6 to Chapter 8) discusses variation propagation modeling issues, including SoV modeling based on the first principles (engineering knowledge) for assembly processes (Chapter 6), machining processes (Chapter 7), and SoV modeling based on observational data (Chapter 8). Part III addresses issues on variation source diagnosis. The section starts with diagnosability study (Chapter 9), then follows with two diagnostic methods as pattern matching methods (Chapter 10) and estimation based methods (Chapter 11). Part IV investigates issues related to design for variation reduction, which includes optimal sensor placement and distribution (Chapter 12), design evaluation and process capability analysis (Chapter 13), optimal fixture layout (Chapter 14), and process-oriented tolerance synthesis (Chapter 15). The last part introduces additional SoV related topics, including QR chain modeling and analysis (Chapter 16), quality-ensured maintenance (Chapter 17), and a review of other SoV research achievements (Chapter 18).
In addition to serving as a major reference book for researchers and engineers, this book can also be used as a text book for senior undergraduate students or graduate students in advanced quality control courses or manufacturing design courses. The book can be organized into three different tracks for a one-semester course according to students’ backgrounds and interests: 1. Data-driven track (fits industrial engineering–style quality and manufacturing for senior undergraduate and junior graduate students): include Chapter 2 to Chapter 5 and Chapter 8 to Chapter 11. 2. Physics-model-based track (fits mechanical engineering–style manufacturing process control for senior undergraduate and junior graduate students): includes Chapter 2, Chapter 3, Chapter 5, Chapter 6 or Chapter 7, Chapter 10, Chapter 14, and Chapter 15. 3. Advanced track (senior graduate and Ph.D. students): includes Chapter 6 (or Chapter 7) and any chapter from Chapter 8 to Chapter 17.
2151_C000.fm Page vii Thursday, October 26, 2006 1:39 PM
Acknowledgments I have been fortunate to have had three advisors in my graduate study: Professor Jun Zhang (M.S. degree advisor in the Automatic Control Department, Beijing Institute of Technology), who introduced me to the control and system theories; Prof. Zhifang Zhang (Ph.D. advisor in electrical engineering at Beijing Institute of Technology), who trained me with system identification and adaptive control, and Prof. S. M. Wu (Ph.D. advisor in mechanical engineering at the University of Michigan), who provided me with the opportunities to study at the University of Michigan and led me into the area of manufacturing science and engineering. Many of the research ideas and achievements described in this book originated from their guidance and training. I am indebted to many of my former and current students with whom I have collaborated, or am still collaborating, in regard to the technical ideas described in this book. The individuals include, but are not limited to, D. Ceglarek, D. Khorzad, B. W. Shiu, D. W. Apley, J. Jin, Y. Ding, S. Zhou, Y. Chen, Q. Huang, F. Tsung, Q. Rong, P. Chaipradubkiat, J. Li, J. Liu, L. E. Izquierdo, J. Zhong, R. Jin, and H. Zheng. I am also indebted to other friends and collaborators, including S. J. Hu, J. Ni, Y. Koren, G. Ulsoy, C. F. J. Wu, V. Nair, D. Djurdjanovic, J. Camelio, C. W. Wampler, and Z. Kong. The experiences of working with them are invaluable, and I have learned a great deal from them, which benefitted me both personally and professionally. My appreciation is also due to the many research associates and students as well as managers and engineers in companies who have directly contributed to the technology transfer and the implementation of the SoV methodologies in industrial practices. Special acknowledgement goes to three individuals who directly helped in contributing or writing some of the chapters: Y. Ding for Chapter 6 and Chapter 11 to Chapter 14; S. Zhou for Chapter 7, Chapter 9, and Chapter 10; and Y. Chen for Chapter 15 to Chapter 17. They also reviewed the manuscript draft of the book and provided many invaluable comments for improvement. Without their strong support and contribution, this manuscript would not have been completed. I am also grateful for the many students, collaborators, more senior colleagues, and others who provided valuable critiques and feedbacks on the manuscript. I have been fortunate to have research support from many sources, including the National Science Foundation (NSF), NSF Engineering Research Center for Reconfigurable Manufacturing Systems, NIST-Advanced Technology Program, General Motors, Ford, DaimlerChrysler, Auto Body Consortium, and many other industrial companies. Their generous funding support made many of these SoV research initiatives feasible.
2151_C000.fm Page viii Thursday, October 26, 2006 1:39 PM
Finally, a very special note of appreciation is extended to my wife, Liping Luo, and my two daughters, Helen and Katherine, who have provided continuous support in the past years.
2151_C000.fm Page ix Thursday, October 26, 2006 1:39 PM
Author Jianjun Shi is a professor of the Department of Industrial and Operations Engineering and a professor of Department of Mechanical Engineering at the University of Michigan. He received his bachelor of science and master of science in electrical engineering at the Beijing Institute of Technology in 1984 and 1987, respectively, and his Ph.D. in mechanical engineering at the University of Michigan in 1992. His research interests focus on the fusion of advanced statistical and domain knowledge to develop methodologies for modeling, monitoring, diagnosis, and control for complex systems in a data-rich environment. He is one of the early pioneers in the field. He has guided 19 Ph.D. graduates and published more than 100 papers. He has also been closely working with industrial companies. He has led various research projects funded by National Science Foundation, NIST Advanced Technology Program, General Motors, DaimlerChrysler, Ford, Lockheed-Martin, Honeywell, Pfizer, and various other industrial companies and funding agencies. The technologies developed in his research group have been implemented in various production systems with significant economic impacts. Professor Shi is the founding chairperson of the Quality, Statistics and Reliability (QSR) Subdivision, Institute for Operations Research and Management Science (INFORMS). He also serves as the director of the laboratory for In-Process Quality Improvement Research (IPQI) at the University of Michigan. He is currently serving as the departmental editor, IIE Transactions on Quality and Reliability, and associate editor for the International Journal of Flexible Manufacturing Systems. He is a member of ASME, ASQ, IIE, SME, and ASA. More information about Dr. Shi can be found on his web site: http://www-personal.engin.umich.edu/~shihang/index.html
2151_C000.fm Page x Thursday, October 26, 2006 1:39 PM
2151_C000.fm Page xi Thursday, October 26, 2006 1:39 PM
Table of Contents Chapter 1
What Is Stream of Variation for Multistage Manufacturing Processes?.............................................................................................1
1.1
History and Current Status of SoV .................................................................1 1.1.1 SoV Initiation and Impact in Industry.................................................1 1.1.2 The History of SoV Methodology Development ................................3 1.2 Overview of SoV Methodology.......................................................................5 1.3 Relationship of SoV Methodologies with Other Existing Methods ...............9 References................................................................................................................11
PART I Basis of Matrix Theory and Multivariate Statistics Chapter 2
Basics of Matrix Theory ....................................................................15
2.1 2.2
Introduction ....................................................................................................15 Definitions of Vector, Matrix, and Operations ..............................................15 2.2.1 Definitions ..........................................................................................15 2.2.2 Partitioned Matrices ...........................................................................18 2.3 Quadratic Forms.............................................................................................19 2.4 Vector Space and Geometrical Interpretations ..............................................20 2.5 Eigenvalues and Eigenvectors of a Matrix ....................................................22 2.6 Vector and Matrix Differentiation, Maximization, and Operators................24 2.6.1 Differentiation with Vectors...............................................................24 2.6.2 Matrix Inequalities and Maximization ..............................................26 2.6.3 Vec Operator, Kronecker Product, and Hadamard Product ..............27 2.6.3.1 Vec Operator and Kronecker Product ................................27 2.6.3.2 Hadamard Product ..............................................................30 2.7 Exercises.........................................................................................................30 References................................................................................................................32 Chapter 3 3.1 3.2
Basics of Multivariate Statistical Analysis ........................................33
Introduction ....................................................................................................33 Multivariate Distribution and Properties .......................................................33 3.2.1 Random Vectors, Cumulative Distribution Functions (CDF), and Probability Density Functions (PDF) .........................................34 3.2.2 Marginal and Conditional Distributions ............................................35 3.2.3 Population Moments ..........................................................................37 3.2.4 Correlation Coefficients .....................................................................40
2151_C000.fm Page xii Thursday, October 26, 2006 1:39 PM
3.3
Multivariate Normal Distribution and Quadratic Forms ...............................40 3.3.1 Multivariate Normal Distribution and Its Properties.........................43 3.3.2 Quadratic Forms.................................................................................44 3.3.3 Noncentral χ2 and F Distributions.....................................................45 3.4 Sampling Theory............................................................................................45 3.4.1 Sample Geometry...............................................................................49 3.4.2 Random Samples and the Expected Values of the Sample Mean and Covariance Matrix.......................................................................50 3.4.3 Some Important Results of Sampling from Multivariate Normal Distributions .......................................................................................50 3.5 The Wishart Distribution and Some Properties.............................................51 3.6 Exercises.........................................................................................................53 References................................................................................................................55 Chapter 4
Statistical Inferences on Mean Vectors and Linear Models..............57
4.1
Statistical Inferences on Mean Vectors..........................................................57 4.1.1 Hotelling’s T 2 Test .............................................................................57 4.1.2 Confidence Regions and Simultaneous Confidence Intervals...........59 4.1.2.1 Confidence Regions ............................................................59 4.1.2.2 Simultaneous Confidence Intervals ....................................61 4.1.3 Inference on a Population Mean Vector with Large Sample Size .....63 4.1.4 Multivariate Quality Control Charts ..................................................64 4.1.4.1 Control Charts for One Multivariate Sample.....................64 4.1.4.2 Control Charts Based on Subgroup Means........................66 4.2 Multiple Linear Regression ...........................................................................68 4.2.1 Model Description..............................................................................69 4.2.2 Least-Squares Estimates ....................................................................70 4.2.3 Inferences about the Regression Model ............................................73 4.2.4 Model Checking: Normality Checking and Outlier Detection .........75 4.2.5 Inferences from the Estimated Regression Model ............................76 4.2.6 Multivariate Linear Regression..........................................................78 4.2.6.1 Multivariate Linear Regression Model...............................78 4.2.6.2 Least-Squares Estimation (LSE) and Maximum Likelihood Estimation (MLE) of Parameters.....................79 4.2.6.3 Inferences for Multivariate Regression Model under Normality Assumption........................................................82 4.2.6.4 Predictions from Multivariate Regression..........................83 4.2.6.5 Selection Method of Independent Variables ......................85 4.3 Exercises.........................................................................................................86 References................................................................................................................89 Chapter 5 5.1
Principal Component Analysis and Factory Analysis .......................91
Principal Component Analysis.......................................................................91 5.1.1 Mathematical Model of PCA.............................................................91 5.1.2 Geometrical Interpretation .................................................................95
2151_C000.fm Page xiii Thursday, October 26, 2006 1:39 PM
5.1.3 Inferences on PCs ..............................................................................96 5.1.4 Applying PCs to Process Control......................................................97 5.2 Factor Analysis...............................................................................................99 5.2.1 Introduction ........................................................................................99 5.2.2 Comparison of Factor Analysis and PCA .........................................99 5.2.3 Orthogonal Factor Model.................................................................100 5.2.4 Statistical Interpretations of Factor Loadings and Communality ...102 5.2.4.1 Interpretation of Factor Loadings.....................................102 5.2.4.2 Interpretation of Communality .........................................102 5.2.5 Estimation of Loading Matrix .........................................................103 5.2.6 Factor Rotation.................................................................................105 5.2.7 Factor Score .....................................................................................109 5.2.8 General Procedures for Factor Analysis..........................................111 5.3 Exercises.......................................................................................................111 References..............................................................................................................113
PART II Variation Propagation Modeling in MMP Chapter 6
State Space Modeling for Assembly Processes...............................117
6.1 6.2
Introduction of Multistage Assembly Processes .........................................117 Variation Factors and Assumptions .............................................................119 6.2.1 Station-Level Variation Factors........................................................119 6.2.2 Across-Station Variation Factors .....................................................120 6.2.3 Summary and Assumptions for Modeling.......................................121 6.3 State Space Modeling ..................................................................................122 6.3.1 Representation of Part Position and Its Deviation State.................122 6.3.2 Some Preliminary Results................................................................123 6.3.3 State Space Representation..............................................................125 6.4 Model Validation ..........................................................................................131 6.5 Exercises.......................................................................................................133 Appendix 6.1: Determination of the Deviations of a Part (Lemma 6.1) .............137 Appendix 6.2: Effect of Fixture Deviation on Part Deviation (Lemma 6.2) .......138 References..............................................................................................................140 Chapter 7 7.1
7.2
State Space Modeling for Machining Processes .............................143
Introduction ..................................................................................................143 7.1.1 Introduction to Machining Processes and Dimensional Variation Sources .............................................................................................143 7.1.2 Modeling of Variation Propagation in Multistage Machining Processes ..........................................................................................145 7.1.3 Model Formulation ..........................................................................147 Derivation of Variation Propagation Model ................................................148 7.2.1 Basics of Kinematic Analysis of Machining Operations ................148 7.2.2 Representation of Workpiece Geometric Deviation ........................152
2151_C000.fm Page xiv Thursday, October 26, 2006 1:39 PM
7.2.3
Single-Stage Modeling of Dimensional Variation...........................153 7.2.3.1 Analysis of Datum-Induced Error ....................................154 7.2.3.2 Analysis of Fixture Errors ................................................158 7.2.3.3 Identify the Overall Dimensional Error by Combining Error Sources Together.....................................................159 7.2.3.4 Modeling Variation Propagation in Multistage Machining Processes ........................................................161 7.3 Model Validation ..........................................................................................162 7.3.1 Introduction to the Experimental Machining Process.....................162 7.3.2 Comparison between the Real Measurement and the Model Prediction .........................................................................................164 7.4 Summary ......................................................................................................166 7.5 Exercises.......................................................................................................166 7.6 Appendix: The System Matrices Used in Equation 7.27............................169 References..............................................................................................................173 Chapter 8
A Factor Analysis Method for Variability Modeling ......................175
8.1 8.2
Introduction ..................................................................................................175 A Factor Analysis Model for Process Variability........................................176 8.2.1 Model Structure................................................................................176 8.2.2 Interpretation of Variation Patterns..................................................177 8.3 Limitations of PCA and Factor Rotation ....................................................180 8.4 Estimating the Number of Faults.................................................................181 8.4.1 Likelihood Ratio Test.......................................................................181 8.4.2 AIC and MDL Information Criteria ................................................182 8.4.3 An Illustrative Example ...................................................................183 8.5 Unique Identification of Multiple Faults .....................................................187 8.5.1 Estimating the Fault Geometry Vectors...........................................187 8.5.2 Identifying Subgroups......................................................................189 8.5.3 Fault Interpretation and Illustrative Example..................................190 8.6 Statistical Properties.....................................................................................193 8.7 Summary ......................................................................................................195 8.8 Exercises.......................................................................................................196 8.9 Appendix 8.1: Discussion of the Distributions of Test Statistics ...............196 References..............................................................................................................197
PART III Variation Source Diagnosis Chapter 9 9.1 9.2
Diagnosability Analysis for Variation Source Identification...........201
Motivation and Formulation of Diagnosability Study ................................201 Definitions of Diagnosability.......................................................................203
2151_C000.fm Page xv Thursday, October 26, 2006 1:39 PM
9.3 9.4 9.5
Criterion of Fault Diagnosability.................................................................205 Minimal Diagnosable Class .........................................................................207 Gauging System Evaluation Based on Minimal Diagnosable Class ..........210 9.5.1 Information Quantity........................................................................210 9.5.2 Information Quality..........................................................................211 9.5.3 System Flexibility ............................................................................211 9.6 Case Study....................................................................................................212 9.6.1 Case Study of A Multistage Assembly Process ..............................212 9.6.2 Case Study of A Multistage Machining Process.............................219 9.7 Summary ......................................................................................................221 9.8 Exercises.......................................................................................................224 Reference ...............................................................................................................226 Chapter 10 Diagnosis through Variation Pattern Matching ...............................229 10.1 Introduction to Variation Patterns ................................................................229 10.2 Links between the Fault-Quality Model and Variation Patterns.................231 10.3 Procedure of Pattern Matching for Variation Source Identification ...........233 10.3.1 Disturbance due to Unstructured Noises .........................................235 10.3.2 Disturbance due to Sampling Uncertainty ......................................236 10.3.3 A Robust Pattern Matching Procedure ............................................237 10.4 Case Study....................................................................................................240 10.4.1 A Machining Process and Its Variation Propagation Model...........240 10.4.2 Pattern Matching for Root Cause Identification in the Machining Process ...........................................................................241 10.5 Summary ......................................................................................................244 10.6 Exercises.......................................................................................................244 References..............................................................................................................248 Chapter 11 Estimation-Based Diagnosis ............................................................249 11.1 LS Estimators for Variance Components ....................................................249 11.1.1 Deviation LS Estimator....................................................................250 11.1.2 Variation LS Estimator.....................................................................251 11.1.3 Other Variation LS Estimators.........................................................252 11.2 Relationship among Variance Estimators ....................................................254 11.3 Comparison of Variance Estimators ............................................................262 11.3.1 Unbiasedness of Variance Estimators ..............................................262 11.3.2 Dispersion of Variance Estimators...................................................263 11.3.3 Comparison of Variance Estimators ................................................264 11.4 Chapter Summary ........................................................................................270 11.5 Exercises.......................................................................................................270 References..............................................................................................................272
2151_C000.fm Page xvi Thursday, October 26, 2006 1:39 PM
PART IV Design for Variation Reduction Chapter 12 Optimal Sensor Placement and Distribution ...................................275 12.1 Introduction ..................................................................................................275 12.2 Design Criteria for Sensor Placement .........................................................278 12.2.1 Diagnosability Index as Design Criterion .......................................278 12.2.2 Sensitivity Index As Design Criterion.............................................279 12.3 Single-Station Sensor Placement .................................................................282 12.3.1 Optimization Formulation................................................................282 12.3.2 Exchange Algorithms from Optimal Experimental Design ............283 12.3.3 Fast Exchange Algorithm with Sort-and-Cut ..................................284 12.3.4 Comparison among Alternative Algorithms ....................................286 12.3.5 Results of Optimal Sensor Layout on a Single Station ..................288 12.4 Multiple-Station Sensor Distribution...........................................................290 12.4.1 Optimization Formulation................................................................290 12.4.2 Variation Transmissibility Ratio ......................................................291 12.4.3 Detection Power on An Individual Station......................................293 12.4.4 Optimal Strategy of Sensor Distribution .........................................296 12.4.4.1 Strategy of Sensor Distribution ........................................297 12.4.5 Example............................................................................................298 12.5 Summary ......................................................................................................301 12.6 Exercises.......................................................................................................301 References..............................................................................................................302 Chapter 13 Design Evaluation and Process Capability Analysis.......................305 13.1 13.2 13.3 13.4
Introduction ..................................................................................................305 Sensitivity-Based Design Evaluation...........................................................307 Multivariate Process Capability Analysis ....................................................313 Examples ......................................................................................................315 13.4.1 Sensitivity-Based Design Evaluation...............................................315 13.4.2 Multivariate Process Capability Analysis ........................................319 13.5 Exercises.......................................................................................................321 13.6 Appendix 13.1: System Matrices of Configuration C1...............................322 13.7 Appendix 13.2: System Matrices of Configuration C2...............................326 13.8 Appendix 13.3: System Matrices of Configuration C3...............................327 13.9 Appendix 13.4: System Matrices of Configuration C4...............................330 References..............................................................................................................331 Chapter 14 Optimal Fixture Layout Design.......................................................333 14.1 Introduction ..................................................................................................333 14.2 Design Criteria for Variation Reduction......................................................335 14.3 Data-Mining-Aided Design Algorithm........................................................338
2151_C000.fm Page xvii Thursday, October 26, 2006 1:39 PM
14.3.1 Overview of the Data-Mining-Aided Design ..................................339 14.3.2 Candidate Design Space ..................................................................341 14.3.3 Uniform Coverage Selection............................................................342 14.3.4 Feature and Feature Function ..........................................................344 14.3.5 Clustering Method............................................................................346 14.3.6 Classification Method ......................................................................347 14.3.7 Selection of K and J ........................................................................349 14.3.8 An Overall Description of the Data-Mining-Aided Design............352 14.4 Example and Performance Comparison ......................................................353 14.5 Summary ......................................................................................................356 14.6 Exercises.......................................................................................................356 References..............................................................................................................358 Chapter 15 Process-Oriented Tolerance Synthesis .............................................361 15.1 Concept of Process-Oriented Tolerancing ...................................................361 15.2 Framework of Process-Oriented Tolerancing ..............................................362 15.2.1 Overview ..........................................................................................362 15.2.2 Variation Propagation Model ...........................................................363 15.2.3 Relationship between Tolerance and Variation................................364 15.2.4 Process Degradation Model .............................................................366 15.2.5 Cost Function ...................................................................................368 15.2.6 Optimization Formulation and Optimality ......................................368 15.3 Case Study for Process-Oriented Tolerancing.............................................368 15.3.1 Tolerance Allocation when Tooling Degradation Is Not Considered........................................................................................368 15.3.2 Tolerance Allocation Considering Tooling Degradation .................370 15.4 Integration of Process-Oriented Tolerance Synthesis and Maintenance Planning........................................................................................................372 15.4.1 Decision Variables of Tolerance and Maintenance Design.............372 15.4.2 Cost Components .............................................................................373 15.4.2.1 Tolerance Cost ..................................................................373 15.4.2.2 Maintenance Cost .............................................................373 15.4.2.3 Quality Loss Function ......................................................374 15.4.3 Formulation of Optimization Problems...........................................374 15.4.3.1 Optimization Formulation Using a Quality Loss Function (F1) ....................................................................374 15.4.3.2 Optimization Formulation with a Quality Constraint ......375 15.5 Integrated Tolerance and Maintenance Design in BIW Assembly Processes ......................................................................................................375 15.6 Optimizations and Optimality for Integrated Tolerance and Maintenance Design...........................................................................................................377 15.6.1 Optimality Analysis of Optimization Formulation F1 ....................377 15.6.2 Optimality Analysis of Optimization Formulation F2 ....................379 15.7 Case Study for Integrated Tolerance and Maintenance Design..................380
2151_C000.fm Page xviii Thursday, October 26, 2006 1:39 PM
15.7.1 Optimal Tolerance and Maintenance Design for Optimization Formulation F1.................................................................................380 15.7.2 Optimal Tolerance and Maintenance Design for Optimization Formulation F2.................................................................................381 15.7.3 Cost Comparison under Different Design Schemes........................382 15.8 Exercises.......................................................................................................384 References..............................................................................................................385
PART V Quality and Reliability Integration and Advanced Topics Chapter 16 Quality and Reliability Chain Modeling and Analysis ...................389 16.1 Introduction ..................................................................................................389 16.1.1 Example 1: Machining Processes ....................................................391 16.1.2 Example 2: Transfer or Progressive Die-Stamping Processes ........392 16.2 QR-Chain Modeling.....................................................................................393 16.2.1 Relationship between Component Performance and Product Quality ..............................................................................................394 16.2.2 System Component Degradation .....................................................394 16.2.3 Product Quality Assessment and System Failure due to Nonconforming Products .................................................................395 16.2.4 Component Catastrophic Failure and Its Induced System Catastrophic Failure .........................................................................396 16.3 System Reliability Evaluation......................................................................396 16.3.1 Challenges in System Reliability Evaluation ..................................397 16.3.2 System Reliability Evaluation of MMPs.........................................397 16.3.3 Self-Improvement of Product Quality and the Upper Bound of System Reliability ............................................................................400 16.4 Implementation of QR-Chain Modeling and Analysis in Body-in-White Assembly Processes .....................................................................................401 16.4.1 QR-Chain in Multistation BIW Assembly Processes .....................402 16.4.2 QR-Chain Model of a BIW Assembly Process...............................402 16.4.2.1 Locating-Pin Degradation Model .....................................403 16.4.2.2 Relationship between Process Variables and Deviations of Quality Characteristics .................................................404 16.4.2.3 Product Quality Assessment and Pin Catastrophic Failure ...............................................................................407 16.4.2.4 System Reliability Evaluation ..........................................408 16.4.3 Case Study........................................................................................408 16.5 Exercises.......................................................................................................410 16.6 Appendix 16.1: Derivation of Equation 16.4 ..............................................410 16.7 Appendix 16.2: Proof of Result 16.1 ..........................................................412
2151_C000.fm Page xix Thursday, October 26, 2006 1:39 PM
16.8 Appendix 16.3: Derivation of Equation 16.10...........................................413 16.9 Appendix 16.4: Proof of Lemma 16.1 .......................................................414 16.10 Appendix 16.5: Proof of Result 16.4 .........................................................415 16.11 Appendix 16.6: Distribution of Random Variable ξ(tk) ............................415 References..............................................................................................................416 Chapter 17 Quality-Oriented Maintenance for Multiple Interactive System Components......................................................................................419 17.1 Quality-Oriented Maintenance Model .........................................................419 17.2 Multicomponent Maintenance Policies .......................................................422 17.2.1 Simple Block Resetting (Replacement)...........................................422 17.2.2 Modified Block Resetting ................................................................425 17.2.3 Age Resetting...................................................................................427 17.3 Further Discussion on Solutions of Optimization Problems ......................429 17.3.1 Optimal Solutions of SBR Policy....................................................430 17.3.2 Optimal Solutions of Modified Block Resetting Policy .................432 17.4 Case Study....................................................................................................432 17.5 Exercises.......................................................................................................436 17.6 Appendix 17.1: Proof of Lemma 17.7 ........................................................437 References..............................................................................................................438 Chapter 18 Additional Topics on Stream of Variation .......................................439 18.1 SoV Modeling for Multistation Assembly Process of Compliant Parts.....439 18.1.1 Introduction ......................................................................................439 18.1.2 SoV Modeling of Compliant Parts in an MMP ..............................440 18.1.2.1 Single Station Assembly Modeling ..................................441 18.1.2.2 Multistation Assembly Modeling .....................................442 18.1.3 Additional Comments ......................................................................444 18.2 SoV Modeling for Serial-Parallel Multistage Manufacturing Systems ......445 18.2.1 Introduction ......................................................................................445 18.2.2 SoV Modeling for Serial-Parallel MMPs........................................446 18.2.2.1 State Space Modeling for Multiple SoVs in an SP-MMP ...........................................................................446 18.2.2.2 Model Dimension Reduction............................................448 18.2.3 Additional Comments ......................................................................449 18.3 SoV-Based Quality-Ensured Setup Planning...............................................449 18.3.1 Introduction ......................................................................................449 18.3.2 Quality-Ensured Setup Planning Methodologies ............................450 18.3.2.1 Variation Propagation Modeling for Setup Planning.......450 18.3.2.2 Optimization Formulation ................................................451 18.3.3 Additional Comments ......................................................................452 18.4 Active Control for Variation Reduction of Multistage Manufacturing Processes ......................................................................................................453
2151_C000.fm Page xx Thursday, October 26, 2006 1:39 PM
18.4.1 Introduction ......................................................................................453 18.4.2 Active Control for Variation Reduction of MMP ...........................454 18.4.3 Additional Comments ......................................................................456 References..............................................................................................................457 Index ......................................................................................................................459 Related Titles .........................................................................................................469
2151_book.fm Page 1 Friday, October 20, 2006 5:04 PM
1
What Is Stream of Variation for Multistage Manufacturing Processes?
This chapter will present some background information about the Stream of Variation (SoV) methodology and its applications. We will particularly discuss the history of SoV initiation and development, the topics addressed in SoV, and the relationship of SoV with other existing methods.
1.1 HISTORY AND CURRENT STATUS OF SOV 1.1.1 SOV INITIATION
AND IMPACT IN INDUSTRY
SoV research was initiated by the need for variation reduction in automotive body assembly processes in the late 1980s. In those days, U.S. automobile producers were losing market share to Asian and European competitors. Quality, mainly reflected as the dimensional variability, of a body-in-white (BIW) was identified as one of the most important factors that impacted the competitiveness of U.S. automobile industries [1]. Variation reduction in automotive body assembly is a very challenging task [2]. A typical BIW has about 100 to 150 sheet metal parts, which are assembled at 80 to 120 assembly stations. A BIW assembly line normally has about 1500 to 2000 fixture locators and approximately 4000 welding spots. Figure 1.1 shows several layers of a BIW assembly process. If any deviations (or failures) occur at a locator, at a welding spot, or from an incoming part, a dimensional variation will be generated on the assembled part at this station, which will propagate to downstream assembly stations, and finally accumulate at the BIW. After assembly is finished, the door, hood, and deck-lid panels are installed into the openings of a BIW; and after the painting process, the windshield and backlight are installed into the appropriate openings. The dimensional variations accumulated at the BIW openings, compounded by variations associated with panels or other subassemblies, will significantly increase the production complexity, leading to more tooling failures and unexpected downtime and reducing both product quality and production throughput. In addition, the dimensional variation in auto body assembly also directly impacts gaps and flushes of panel fitting, which leads to the increase of wind noise and the risk of water leakage. Variation reduction demands effective process control techniques through monitoring and diagnosis based on the dimensional measurement data.
1
2151_book.fm Page 2 Friday, October 20, 2006 5:04 PM
2
Stream of Variation Modeling and Analysis for MMPs
FIGURE 1.1 A body-in-white (BIW) and major closure panels.
In the late 1980s, the inline optical coordination measurement machine (OCMM) was introduced to the automotive body assembly plant. The OCMM was installed at the end of an assembly line and used optical laser sensors to measure critical features of a BIW. An OCMM had about 100 laser sensors. Each sensor targeted a critical feature of a BIW, and provided dimension readings on the body coordinates. As a result, a large amount of measurement data were obtained from each auto body produced in an assembly line. The tremendous amount of inline quality data provided significant opportunities for more effective process control. However, it also challenged existing process control technologies. With hundreds of quality attributes being measured, some out-of-control conditions were invariably detected by using SPC control charts. In general, reacting to those out-of-control conditions was not easy using SPC techniques alone: they could not be quickly resolved because of the complexity of the process and time-consuming efforts in root cause identification. Therefore, the massive dimensional data generated from the OCMM was not fully utilized for variation reduction in body assembly. More effective models and data analysis methods were needed to address this challenging problem. Professor S.M. Wu at the University of Michigan took up the challenge and initiated the “Two Millimeter Program.” The main goal of this program was to increase the competitiveness of U.S. automakers by reducing the BIW dimensional variation. In the early 1990s, the dimensional variation (by Six Sigma) was about 2.0 mm for Japanese automakers, 2.5 mm for European automakers, and more than 3.0 mm (even close to 8.0 mm) for U.S. automakers. The 2 mm program aimed to reduce car body variation to the lowest level possible: the Six Sigma of all key dimensional quality attributes on a car body should be reduced to below 2 mm — a benchmark of the best quality of Japanese cars at that time. Under the leadership of Professor Wu and with strong support from GM, Chrysler, Ford, Auto Body Consortium, and NIST-Advanced Technology Programs (NIST-ATP), a group of researchers and engineers devoted much effort to studying various critical issues on variation reduction for automotive body assembly and enabled concurrent technology transfer and implementation in the automotive industry.
2151_book.fm Page 3 Friday, October 20, 2006 5:04 PM
What Is Stream of Variation for Multistage Manufacturing Processes?
3
The 2 mm program made a significant impact on automotive body assembly plants. In December 1992, one U.S. assembly plant located in Detroit, MI, achieved the “2 mm” variation level and marked the first success of the 2 mm program. In 1995, a “process navigator” software package was developed by Auto Body Consortium and the University of Michigan that provided an integrated environment for BIW dimensional data monitoring, analysis, and root cause diagnosis for the entire body assembly process. In 1998, distributed sensing strategy was deployed in a body shop for more effective data acquisition for process control. By 2000, the developed variation reduction methodologies had been successfully implemented in design improvement and the product launch of 32 new vehicle programs [1]. The implementation of those methodologies significantly improved the quality of assembly and also reduced the ramp-up time during the new production launch in automotive body assembly processes. The SoV methodology was expanded to the multistage machining processes in 1998 with the support of the NSF Engineering Research Center on Reconfigurable Manufacturing Systems (NSF ERC on RMS) and its industrial member companies. Unlike assembly, which is a part addition process, the machining is a materialremoving process. A set of new technologies was developed for the SoV methodology in machining. A test bed with four CNC machines, one CMM, and one reconfigurable inspection machine (RIM) was built in the NSF-ERC for RMS for the SoV research and validation in 1999. The developed SoV-based technologies for machining processes were validated and implemented in several powertrain plants from 2002 to 2006. The SoV-based sensitivity study was conducted to evaluate a new engine head machining line with 22 stations to identify critical machining stations that contributed more variability to the process in 2004. Gauging strategies were studied for quality feature selection and measurement of engine and transmission processes in 2005. Root cause diagnosis techniques were successfully implemented in process control in a diesel engine plant and adopted as the standard practice for the system in 2005. In 2003 the DCS Company won a grant from the NIST-Advanced Technology Program to develop commercial software based on the SoV methodology to support product and process modeling, design analysis, and diagnostic control of multistage manufacturing processes.
1.1.2 THE HISTORY
OF
SOV METHODOLOGY DEVELOPMENT
SoV methodology development attempts to develop a math-based technique for variation reduction in both design and manufacturing of a multistage manufacturing process. The methodology development can be divided into two phases: initial learning and technologies development (1990–1997) and a unified SoV methodology development based on state space models (1997–2006). Phase I research (1990–1997) targeted the development of advanced methods to analyze massive OCMM data to find root causes of variability. Some representative achievements in the early phase of technology development included (1) proposing the concept of using principal component analysis (PCA) to analyze the multivariate OCMM data [3], (2) developing a hierarchical representation or model and associated ad hoc diagnostic rules for failures in multistation assembly systems [4], (3) developing
2151_book.fm Page 4 Friday, October 20, 2006 5:04 PM
4
Stream of Variation Modeling and Analysis for MMPs
a theoretical basis by proving the inherent relationships between the fixture failure pattern obtained from design and the first principal component calculated from the OCMM data [5], (4) developing multiple fault diagnosis techniques based on geometric modeling and least-squares estimation [6], and (5) variation modeling and analysis of complaint parts by integrating the part dimensional variability and material properties [7,8]. In 1997, Hu discussed the variation and its propagation problem in multistage manufacturing processes using the term stream of variation (SoV), as suggested by Yoram Koren [8]. Although various significant technological advancements were achieved in Phase I, the methodologies in this phase were specifically devised to address particular variation reduction objectives. In addition, most of them focused on a singlestage manufacturing system (e.g., single fixture, single tooling). The hierarchical representation of multistation systems [4] was still an engineering knowledge “representation,” rather than a mathematical model, for body assembly system. Thus, there was no systematic, unified methodology for variation reduction for multistage manufacturing systems at that time. The concept of using a state space model to describe the variation and its propagation ushered in a new era of SoV methodology development [9,10]. The state space model was typically used in system or automatic control theories, but not in discrete-part manufacturing. The introduction of a state space modeling structure to describe variation and its propagation successfully resolved two fundamental issues in variation reduction of multistage manufacturing processes: (1) the lack of analytical models for variation propagation modeling and (2) the lack of systematic methodologies for variation propagation analysis and optimization. The development of state space models to describe variation propagation provided a solid scientific foundation for manufacturing information (product design, process design, and quality measurements) integration for variation reduction in multistage manufacturing processes. With the state space model as the basis, various SoV-related research efforts have been conducted in both product and process design and the manufacturing process control. The major technological milestones in SoV development are summarized in Figure 1.2, which consists of three major paths: SoV modeling, SoVbased design, and SoV-based process control. More than 40 papers on those topics were published in various journals, including ASME Transactions, IIE Transactions, IEEE Transactions, Technometrics, etc., which represent a truly multidisciplinary research effort contributing mechanical engineering, industrial engineering, system and control engineering, and applied statistics to the knowledge base. The work was also broadly recognized by different research disciplines, which were evidenced by the Best Paper Awards received from American Society of Mechanical Engineering (ASME), North American Manufacturing Engineering Institute (NAMRI), Institute of Industrial Engineering (IIE), and Institute of Operations Research and Management Science (INFORMS). The SoV methodology, as a unified approach, has made fundamental contributions and has had a significant impact on variation reduction of multistage manufacturing processes.
2151_book.fm Page 5 Friday, October 20, 2006 5:04 PM
What Is Stream of Variation for Multistage Manufacturing Processes?
5
FIGURE 1.2 Important milestones in SoV methodology development.
1.2 OVERVIEW OF SOV METHODOLOGY Although SoV methodology originated from dimensional variation control of auto body assembly processes, it has been expanded to address variation management and reduction issues for generic complex multistage manufacturing processes (MMPs). An MMP is referred to as a process involving multiple stations to produce a product. Examples of such processes include: (1) an automotive body assembly that has multiple parts assembled at multiple stations, (2) an engine head production that involves multiple machining operations of a single part at multiple stations, (3) a transfer or progressive stamping process that involves multiple die stations to form a part, and (4) semiconductor manufacturing processes in which a silicon wafer develops in several stages with several layers to form a chip. In an MMP, workpiece dimensional errors are contributed by the errors generated at the current stage, as well as accumulated errors transmitted from previous stages. Thus, the nature of variation and its propagation introduce significant challenges to variation management and reduction for an MMP. This book will discuss how to conduct variation reduction by presenting SoV methodologies. SoV attempts to describe the complex production stream and data stream involved in modeling and analysis of variation and its propagation in an MMP. The production stream is referred to as the physical layout of the multistage manufacturing processes. As shown in Figure 1.3, multiple stations form a subassembly line, and multiple subassembly lines merge into the main assembly line in an automotive body assembly process. Each station generates a dimensional variation on the subassembly, and the subassembly is transferred to the next station to be assembled with more parts. As the parts or subassemblies go through all production
2151_book.fm Page 6 Friday, October 20, 2006 5:04 PM
6
Stream of Variation Modeling and Analysis for MMPs
FIGURE 1.3 Layout of a multistage automotive body assembly process.
lines, a stream of part flow, as well as an associated flow of part variation, will be generated. One interpretation of SoV reflects these multiple stations and multiple line configurations, which leads to the variation and its propagation through the MMP. An interpretation of SoV reflects the complex data relationships in an MMP. As shown in Figure 1.4, the x-axis represents the manufacturing stages; the y axis, time; the z-axis, the quality attributes; and Mi, the quality features. In an MMP, there are three types of correlations among those data streams: (1) the quality attributes are autocorrelated in terms of the stages along the production line and are shown as M2 along the x-axis; (2) the quality attributes are crosscorrelated among them within the same stage and represented as [M1, M2, …, Mm] at stage N along the z-axis; and (3) each quality attribute is also autocorrelated in terms of time owing to the
FIGURE 1.4 Complex data relationships in an MMP.
2151_book.fm Page 7 Friday, October 20, 2006 5:04 PM
What Is Stream of Variation for Multistage Manufacturing Processes?
7
FIGURE 1.5 Variation propagation and notations in SoV modeling.
degradation or wear of production tooling over time and represented as Mi (i = 1, 2, … m) along the y axis. Those three types of correlations, observed as a stream of data, introduce significant challenges in variation modeling, analysis, and control. To investigate the variations of those data streams, this book presents the development of the SoV methodology and its applications. The foundation of the SoV methodology is a mathematical model that links the key product quality characteristics (KPC) with key control characteristics (KCC) (e.g., fixture error, machine error, etc.). This model has a state space representation that describes the deviation and its propagation in an N-station process (as shown in Figure 1.5), i.e., x k = A k −1x k −1 + B k u k + w k , k = 1, 2, …, N,
(1.1)
y k = C k x k + v k , {k} ⊂ {1, 2, …, N},
(1.2)
where k = 1, 2, …, N and is the stage index. xk is the state vector representing the key quality characteristics of the product (or intermediate workpiece) after stage k. uk is the control vector representing the tooling errors (e.g., tolerance when no faults occur, or deviation when failures occur on the tooling) at stage k. yk is the measurement vector representing product quality measurements at stage k. wk and vk are the modeling error and sensing error, respectively. The coefficient matrices Ak, Bk, and Ck are determined by product and process design information: Ak represents the impact of deviation transition from stage k – 1 to stage k, Bk represents the impact of the local tooling deviation on product quality at stage k, and Ck is the measurement matrix, which can be obtained from the defined key product quality features at stage k. If we repeat the modeling efforts for each stage from k = 1 to N, we will get the deviation and its propagation throughout the MMPs. By taking variations on both sides of Equation 1.1 and Equation 1.2, and making certain assumptions, we will obtain the “variation” and its propagation model for the MMPs. With the mathematical model, variation reduction can be achieved in both the design and manufacturing stages through rigorous mathematical decision making. However, significant challenges must be overcome in model development and utilization to realize the benefits of the analytical capability of this model. These challenges are addressed in this book. SoV methodology will address the following important questions for variation reduction in an MMP in detail:
2151_book.fm Page 8 Friday, October 20, 2006 5:04 PM
8
Stream of Variation Modeling and Analysis for MMPs
•
•
•
•
How to model and represent the product and process design information to achieve variation reduction? In this book, we introduce two basic methods: the physical modeling method and data-driven modeling methods. In the former method, the kinematics relationship between KCC and KPC is identified through a detailed physical analysis of the manufacturing process; whereas in the latter method, the model is achieved through a statistical fitting based on the historical process measurement data. Details of SoV modeling will be discussed in Chapter 6 to Chapter 8. How to systematically find the root causes of variability in terms of which manufacturing station, and what in the station introduce the variability? During continuous production, a larger product variation may occur at any stage of an MMP because of worn-out tooling, tooling breakage, and incoming part variation. This book will present a systematic approach to root cause identification. In this approach, a new concept of “statistical methods driven by engineering models” is proposed to integrate the product and process design knowledge with online statistics. The variation model, developed from the design information, is used to link the product variation (quality attributes) with the tooling variation (or potential failure). During production, the product features are measured and the data are used to conduct statistical analysis — based on Equation 1.1 and Equation 1.2 — to identify root causes. Advanced statistics and estimation theory will be used in these efforts. This book presents two types of diagnosis techniques for root cause identification in MMPs: variation pattern matching (Chapter 10) and estimation-based diagnosis (Chapter 11). How to distribute measurement gauges for effective process control in an MMP by determining when, where, and what to measure on the final and intermediate workpieces? One of the major tasks in variation reduction is to design the gauging strategies to measure the product features in a manufacturing process. Most existing industrial practices focus on product coherence inspection (i.e., product-oriented measurements), which is effective in detecting product imperfection but may not be as effective in identifying root causes of product variation. This book proposes a “process-oriented” measurement concept with distributed sensing strategy. In this strategy, selected key control and product characteristics are measured at specific stages to both detect product defects and identify their root causes. The diagnosability issue, i.e., the capability of identifying potential root causes of the process variation for a given measurement strategy, is discussed in Chapter 9. The related issues of optimal sensor placement and distribution are discussed in Chapter 12. How to conduct design evaluation and tolerance synthesis for an MMP? Variation analysis and design evaluation are conducted on the product and during the process design stage to identify the critical components, features, and manufacturing operations. With the SoV model defined in Equation 1.1, the following three tasks can be performed: (1) tolerance
2151_book.fm Page 9 Friday, October 20, 2006 5:04 PM
What Is Stream of Variation for Multistage Manufacturing Processes?
•
9
analysis by allocating the part tolerance (x0) and tooling tolerance (uk) and then predicting the final product tolerance (xN) by solving the difference equations, (2) tolerance synthesis by fixing the final product tolerance (xN) and then assigning the tolerance for individual parts (x0) and tooling components (uk) with minimized cost objective functions, and (3) sensitivity study by identifying the critical parts (xk) or tooling components (uk) that have significant impacts on the final part variation through the evaluation of defined sensitivity indices. The details of those topics will be discussed in Chapter 13 to Chapter 15 of this book. How to integrate product quality and production tooling reliability for effective system design and maintenance decision making? There is a complex and intriguing relationship between product quality and tooling reliability. A degraded (or failed) production tool will lead to greater product variability or number of defects; in addition, the variability of product quality features will impact the degradation rate or failure rate of production tooling. For an MMP, those interactions are more complex, as variations propagate from one stage to the next stage. Thus, a “chain effect” between the product quality and tooling reliability can be observed, and this is known as the “QR Chain” effect. The modeling of the QR Chain is an integrated effort of the SoV model and the semi-Markov process model. Chapter 16 will discuss the modeling of the QR Chain for MMPs. Chapter 17 investigates the applications of the QR Chain effect in reliability and maintenance decisions.
1.3 RELATIONSHIP OF SOV METHODOLOGIES WITH OTHER EXISTING METHODS A typical product realization involves product design, process design, production ramp-up, production and process control and, finally, system maintenance. Various methodologies have been developed and implemented in each stage of the product realization. However, very few techniques exist to systematically address issues related to variation reduction and management. There is a lack of a unified methodology to conduct variation management and reduction over all the different steps from design and manufacturing to maintenance in MMPs. Currently, computer-aided design (CAD) and computer-aided manufacturing/engineering (CAM or CAE) have been widely used in product and process design, and have significantly impacted product realization. CAD mainly focuses on product design in terms of functionality, part geometry, component interactions, and their tolerances. CAM addresses issues of process planning, sequencing, and tooling design for a given product. There is usually an efficient interface for data communication and data transfer among CAD and CAM systems. However, few of them address issues related to “variation” modeling and analysis. The closest efforts in variation modeling and analysis are Variation Simulation Analysis (VSA), or similar industrial packages such as DVA, 3DCS, or Valysis. These technologies use homogeneous transformation matrices to model the variation and its propagation based on geometrical data from the product and tooling design. Then, numerical
2151_book.fm Page 10 Friday, October 20, 2006 5:04 PM
10
Stream of Variation Modeling and Analysis for MMPs
simulations, mainly based on the Monte-Carlo method, are conducted to predict the final product variation with the given process sequence, tooling layout, and tooling tolerance. In general, these efforts can be time consuming and only lead to the “best available” solution from different simulation trials. The VSA-based technology mainly serves as a tolerance analysis and design evaluation tool rather than a tolerance synthesis and design optimization tool for MMPs. Unlike VSA-based technologies, SoV models the variation and its propagation with a station-indexed state space model based on the product and process design. The SoV model in Equation 1.1 contains all essential design information that impacts the product variability. That information includes production sequence information, fixture locator layout and datum changes along the process, tooling and part interactions, part and tooling tolerance, and key product quality features. All those design parameters are integrated into the SoV model in an analytical format. Thus, it facilitates effective computation to obtain analytical solutions for design evaluations (unlike Monte-Carlo simulations). It also provides capabilities for tolerance synthesis (or allocation) and design optimizations for process sequencing, tooling layout, and variation management. In production and process control, statistical process control (SPC) plays an essential role in monitoring process data and detecting mean or variation changes. Although SPC is effective in terms of process change detection, it does not systematically integrate the product and process design data into the process control efforts. Thus, human experience plays a major role in identifying root causes of variation. These efforts could be very challenging and time consuming, especially for a new production line with experience which is limited, or for MMPs in which the process and product parameters have complex and intricate relationships. The SoV models the product measurement data with an observation equation (Equation 1.2), which can be used jointly with the state space equation (Equation 1.1) representing the variation and its propagations based on the product design. As a result, SoV-based process control can systematically integrate production measurement data with the process and product design information. This leads to effective process monitoring, change detection, and root cause identification in terms of which stage introduces product variability, and what process in that stage introduces the variability. In addition, the SoV-based process control also leads to systematic distribution of sensors (e.g., gauging strategies) to ensure identification of the root causes of variability. This effort leads to the transformation of the gauging strategy from the end-of-line “product-inspection”-oriented sensing placement to distributed “process-diagnosis”-oriented sensing strategy. Therefore, more effective process control capabilities can be achieved. Variation reduction and management also affects tooling reliability and maintenance decisions during production. Tooling reliability modeling and analysis are typically separated from the product quality and process control in current industrial practice. However, recent research has indicated that product quality and tooling reliability are inherently related. The QR Chain is an important aspect of an MMP. The SoV model provides a basis for integrated modeling and analysis for product quality and tooling reliability integration, which further leads to integrated design
2151_book.fm Page 11 Friday, October 20, 2006 5:04 PM
What Is Stream of Variation for Multistage Manufacturing Processes?
11
of tolerance optimization and maintenance decision making with consideration of the life cycle of production. In summary, SoV methodology expands the current technological base by developing a unified tool for variation modeling, design analysis, and process control with a focus on variation reduction and management in an MMP.
References 1. Long, B., March 1999, A Systems Solution to a Quality Problem in Auto Body Manufacturing, NIST Special Pub 950-1. 2. Ceglarek, D. and Shi, J., 1995, Dimensional variation reduction for automotive body assembly, Journal of Manufacturing Review, 8, 139–154. 3. Hu, S.J. and Wu, S.M., May 1992, Identifying sources of variation in automobile body assembly using principal component analysis, Transactions of NAMRI, XX, 311–316. 4. Ceglarek, D., Shi, J., and Wu, S.M., 1994, A knowledge-based diagnostic approach for the launch of the auto-body assembly process, ASME Transactions, Journal of Engineering for Industry, 116, 491–499. (Also published in 1993, PED-Vol. 64, Manufacturing Science and Engineering, 401–412.) 5. Ceglarek, D. and Shi, J., 1996, Fixture failure diagnosis for auto body assembly using pattern recognition, ASME Transactions, Journal of Engineering for Industry, 118, 55–65. 6. Apley, D. and Shi, J., 1998, Diagnosis of multiple fixture faults in panel assembly, ASME Transactions, Journal of Manufacturing Science and Engineering, 120, 793–801. (Also Proceedings of the 95’ ASME Winter Annual Meeting, ASME MEDVol. 4, November 1996, 575–581.) 7. Liu, S.C. and Hu, S.J., 1997, Variation simulation for deformable sheet metal assembly using finite element methods, ASME Journal of Manufacturing Science and Engineering, 119, 368–374. 8. Hu, S.J., 1997, Stream of variation theory for automotive body assembly, Annals of the CIRP, 46(1), 1–6. 9. Shi, J. and Jin, J., 1997, Modeling and diagnosis for automotive body assembly process using state space models, Proceedings of International Intelligent Manufacturing System’97, Seoul, Korea, pp.189–196. 10. Jin, J. and Shi, J., November 1999, State space modeling of sheet metal assembly for dimensional control, ASME Transactions, Journal of Manufacturing Science and Engineering, 121, 756–762. 11. Huang, Q., Zhou, N., and Shi, J., 2000, Stream of variation modeling and diagnosis of multi-station machining processes, Proceedings of the 2000 ASME International Mechanical Engineering Congress and Exposition, MED-Vol. 11, November 5–10, Orlando, FL, pp. 81–88 12. Zhou, S., Huang, Q., and Shi, J., April 2003, State space modeling for dimensional monitoring of multistage machining process using differential motion vector, IEEE Transactions on Robotics and Automation, 19(2), 296–309. 13. Camelio, J., Hu, S.J., and Ceglarek, D., 2003, Modeling variation propagation of multi-station assembly systems with compliant parts, ASME Transactions, Journal of Mechanical Design, 125, 673–681.
2151_book.fm Page 12 Friday, October 20, 2006 5:04 PM
12
Stream of Variation Modeling and Analysis for MMPs 14. Chen, Y., Jin, J., and Shi, J., 2004, Integration of dimensional quality and locator reliability in design and evaluation of multi-station body-in-white assembly processes, IIE Transaction, 36(9), 827–839. 15. Ding. Y., Jin. J., Ceglarek, D., and Shi, J., 2000, Process-oriented tolerance synthesis for multi-station manufacturing systems, Proceedings of the 2000 International Mechanical Engineering Congress and Expositions, November 5–10, Orland, FL, pp. 15–22. 16. Ding, Y., Jin, J., Ceglarek, D., Shi, J., 2005, Process-oriented tolerancing for multistation assembly systems, IIE Transactions, 37(6), 493–508. 17. Ding, Y., Ceglarek, D., and Shi, J., 2002, Design evaluation of multi-station assembly processes by using state space approach, ASME Transactions, Journal of Mechanical Design, 124(2), 408–418. 18. Kim, P. and Ding Y., 2004, Optimal design of fixture layout in multi-station assembly processes, IEEE Transactions on Automation Science and Engineering, 1(2), 133–145. 19. Ding, Y., Ceglarek, D., Shi, J., 2002, Fault diagnosis of multistage manufacturing processes by using state space approach, ASME Transactions, Journal of Manufacturing Science and Engineering, 124(2), 313–322. 20. Ding, Y., Shi, J., and Ceglarek, D., 2002, Diagnosability analysis of multi-station manufacturing processes, ASME Transactions, Journal of Dynamics Systems, Measurement, and Control, 124, 1–13. 21. Ding, Y., Kim, P., Ceglarek, D., and Jin, J., 2003, Optimal sensor distribution for variation diagnosis for multi-station assembly processes, IEEE Transactions on Robotics and Automation, 19(4), 543–556. 22. Zhou, S., Chen, Y., and Shi, J., 2004, Root cause identification for quality improvement of multistage machining processes, IEEE transactions on Robotics and Automation, 1(1), 73–83.
2151_book.fm Page 13 Friday, October 20, 2006 5:04 PM
Part I Basis of Matrix Theory and Multivariate Statistics
2151_book.fm Page 14 Friday, October 20, 2006 5:04 PM
2151_book.fm Page 15 Friday, October 20, 2006 5:04 PM
2
Basics of Matrix Theory
2.1 INTRODUCTION Matrix concepts, techniques, and theorems are crucial tools for multivariate analysis and its applications, e.g., process monitoring and quality improvement for multiple significant characteristics. By employing matrix notation, results can be expressed in a clear and compact manner. It can also facilitate geometrical interpretation and algebraic explanation of calculations. In this chapter, we will summarize the concepts, operations, and theorems of matrix algebra that will be required in later sections of this book. More readings on the topic can be found in References 1–5.
2.2 DEFINITIONS OF VECTOR, MATRIX, AND OPERATIONS 2.2.1 DEFINITIONS Definition 2.1 (Matrix): A matrix of dimension m × n is defined as a rectangular array of elements arranged in m rows and n columns, denoted by an uppercase boldface letter as follows:
( )
A = aij
a11 a = 21 am 1
a12 a22 am 2
a1n a2 n ∈ R m×n , amn m × n
(2.1)
where aij denotes the element located in the i th row and the j th column of the matrix A. Definition 2.2 (Vector): A vector is a matrix with only a single row or column, named row vector or column vector, respectively. Remark 2.1: In this book, without special indication, we will refer to a vector as a column vector, which is customarily denoted by a lowercase boldface letter as follows: T
a = a1 a2 am ∈ R m×1.
(2.2)
Remark 2.2: From Definition 2.1 and Definition 2.2, a matrix can also be viewed as a collection of row or column vectors, i.e., A = [a 1T a T2 a Tm ]T = [a1 a 2 a n ],
(2.3)
15
2151_book.fm Page 16 Friday, October 20, 2006 5:04 PM
16
Stream of Variation Modeling and Analysis for MMPs
where a~ i = [ai1 ai2 ain] is the i th row vector with dimension n, and aj = [a1j a2j amj]T is the j th column vector with dimension m. Definition 2.3 (Determinant of a square matrix): Let A be an m × m square matrix. The determinant of A, denoted by A or det(A), is defined as: A =
∑ ( −1)
α
a1 j1 a2 j2 amjm .
(2.4)
m
All products consist of one element from each row and column and multiplied by (–1) if the number of inversions of the particular permutation j1, j2 ,..., jm from the standard order 1, 2, …, m. The summation extends through all m! permutations of the column subscripts. The number of inversions α in a particular permutation is defined as the total number of times an element is followed by numbers that would ordinarily precede it in the standard order 1, 2, …, m. Matrix A is referred to as a singular matrix if A = 0. Example 1: A = a11 , A = a11 a Example 2: A = 11 a21
a12 , A = a11a22 − a12 a21 a22
a11 Example 3: A = a21 a31
a12 a22 a32
a13 a23 , a33
A = a11a22 a33 + a12 a23 a31 + a13 a21a32 − a13 a22 a31 − a11a23 a32 − a12 a21a33 Definition 2.4 (Trace): Let A be an m × m square matrix. The sum of diagonal elements is called the trace of A, denoted by m
tr ( A) =
∑a
ii
(2.5)
i =1
Definition 2.5 (Inverse): The inverse of the m × m square matrix A = ( aij ) , denoted by A–1, is the unique matrix such that AA–1 = A–1A = I
(2.6)
2151_book.fm Page 17 Friday, October 20, 2006 5:04 PM
Basics of Matrix Theory
17
If such an A −1 exists, then A is nonsingular; otherwise, A is singular. Some useful results related to inverses are summarized below. Here, we assume that all the inverses exist. 1. ( A −1 )T = ( A T )−1 2. ( A −1 )−1 = A 3. ( AB)−1 = B−1A −1 1 4. ( kA)−1 = A −1, k is a nonzero scalar k −1 a11 1 / a11 0 0 0 0 a22 0 0 0 5. = 0 0 0 0 0 0 amm 0 0
0 1 / a22 0 0
0 0 0
0 0 0 1 / amm
6. If A is orthogonal, then A −1 = A T Definition 2.6 (Reduced row echelon form of a matrix): A matrix is said to be in reduced row echelon form if the following conditions are satisfied: • • • •
The first nonzero number in a row is 1 (we call it the leading 1). All rows of zeros (if any) are together at the bottom of the matrix. Each column that contains a leading 1 has only zeros below it. Each column that contains a leading 1 has zeros everywhere else.
If all these properties except the last are satisfied, then the matrix is in row echelon form. Remark 2.3: The reduced row echelon form of a matrix is unique. Remark 2.4: The reduced row echelon form of a matrix can be obtained through row operations, i.e., trading two rows, multiplying a row by a nonzero scalar, or adding a scalar multiple of one row to another row. For example, Matrix 1 A = 2 1
1 3 3
1 7 −2
3 0 17
can be transformed into reduced row echelon form as 1 0 0
0 1 0
0 0 1
1 4 . −2
2151_book.fm Page 18 Friday, October 20, 2006 5:04 PM
18
Stream of Variation Modeling and Analysis for MMPs
2.2.2 PARTITIONED MATRICES In certain cases, some blocks in a matrix possess common characteristics. Thus, it is sometimes convenient to partition the whole matrix into several blocks, for example; A11 A A = ( A ij ) = 21 A m 1
A12 A 22 Am 2
A1n A2 n A mn
(2.7)
where each submatrix A ij has pi rows and q j columns. Remark 2.5: In Equation 2.7, submatrices in the same row of A have the same number of rows; so do the submatrices in the same column of A. Remark 2.6: The operations of partitioned matrices are similar to those of unpartitioned ones, except that the elements in partitioned matrices are submatrices. For example, if A = ( Aij) and B = (Bij) are partitioned into submatrices with similar dimensions, then A + B = ( Aij + Bij) , n A1k B k 1 k =1 AB= n A mk B k 1 = k 1
∑
∑
A1k B kp k =1 n A mk B kp k =1 n
∑
∑
in which the dimensions of each submatrix product must conform. By employing submatrices, we can simplify computations of the inverse and determinant. Consider a particular important partition of the form: A A = 11 A 21
A12 A 22
(2.8)
If both A11 and A 22 are square, then the inverse and determinant of A can be readily calculated in terms of submatrices: 1. If A 22 has an inverse, then A −1 = ( A11 − A12 A −221A 21 )−1 −1 −1 −1 − A 22 A 21 ( A11 − A12 A 22 A 21 )
(2.9) A + A A 21 ( A11 − A12 A A 21 ) A12 A
−( A11 − A12 A −221A 21 )−1 A12 A −221 −1 22
−1 22
−1 22
−1
−1 22
2151_book.fm Page 19 Friday, October 20, 2006 5:04 PM
Basics of Matrix Theory
2.
19
−1 A = A11 ⋅ A 22 − A 21A11 A12 if A11 has an inverse
−1 A 21 if A 22 has an inverse = A 22 ⋅ A11 − A12 A 22
(2.10)
3. If both A11 and A 22 have inverses, then −1 −1 −1 −1 −1 ( A11 − A12 A 22 A 21 )−1 = A11 + A11 A12 ( A 22 − A 21A11 A12 )−1 A 21A11 .
(2.11)
As a special case, if A is an m × m nonsingular matrix, a is an m × 1 vector, and k is a scalar, then we have the identity: ( A + kaa T )−1 = A −1 −
k A −1aa T A −1 T −1 1 + ka A a
(2.12)
2.3 QUADRATIC FORMS Definition 2.7 (Quadratic Form): A quadratic form of the variables xi (i = 1, 2, …, n) is of the type: f ( x1 , x 2 ,..., x n ) = a11x12 + a22 x 22 + + ann x n2 + 2a12 x1x 2 + + 2a1n x1x n + + 2a( n−1) n x n−1x n
(2.13)
Let x T = x1 x 2 x n and A = (aij) be a symmetric matrix of order n × n . Then Equation 2.13 can readily be expressed in an equivalent matrix notation: f (x) = x T A x
(2.13P′)
Remark 2.7: If x is a random vector, the quadratic form plays an important role in univariate and multivariate analysis. Definition 2.8 (Positive definite and positive semidefinite matrix): A symmetric matrix A = (aij) and its associate quadratic form in Equation 2.13 are called positive definite if xTAx > 0 for ∀x ≠ 0
(2.14)
xTAx ≥ 0 for ∀x ≠ 0
(2.15)
and positive semidefinite if
2151_book.fm Page 20 Friday, October 20, 2006 5:04 PM
20
Stream of Variation Modeling and Analysis for MMPs
2.4 VECTOR SPACE AND GEOMETRICAL INTERPRETATIONS As a column vector a = [a1 a2 am]T can be viewed as a special matrix of m × 1 order, as defined in Definition 2.2, addition and multiplication by a scalar also hold for vectors. Hence, we can introduce the definition of vector space as follows: Definition 2.9 (Vector space): Vector space is the space of all the real vectors a = [a1 a2 am]T with the operations of addition and multiplication by a scalar. Definition 2.10 (Linear combination and linear span): The linear combination of vectors ai (i = 1, 2, …, n) is defined as a vector b of the form: n
b=
∑k a
(2.16)
i i
i =1
where all ki ‘s are scalars. The linear span of ai (i = 1, 2, …, n) is formed by the set of all linear combinations of ai (i = 1, 2, …, n). Remark 2.8: An m-dimensional vector a = [a1 a2 am]T can be geometrically viewed as a directed line in m-dimensional space with component ai along the i-axis, i = 1, 2, …, m. The 2-D case is illustrated in Figure 2.1. Thus, we can define the length and angle between two vectors. Definition 2.11 (Length of a vector): Let a = [a1 a2 am]T be an m-dimensional vector. Its length, la , is defined by: la = a12 + a22 + + am2
(2.17)
Remark 2.9: For the scalar multiplication ka, the direction of a does not change if k > 0, but reverses if k < 0.
a2 a
1
0 FIGURE 2.1 A 2-D vector.
a1
a1 a2
2151_book.fm Page 21 Friday, October 20, 2006 5:04 PM
Basics of Matrix Theory
21
Remark 2.10: For the scalar multiplication ka, its length is changed in the form: lka = ( ka1 )2 + ( ka2 )2 + + ( kam )2 = k
a12 + a22 + + am2 = k la (2.18)
Thus, length increases if k >1, and decreases if 0 < k <1. Before introducing the angle between two vectors, we will first define their inner product. Definition 2.12 (Inner product): The inner product of two m-dimensional vectors a and b is defined by: m
T
T
(a, b) = a b = b a =
∑a b
i i
(2.19)
i =1
Remark 2.11: la = (a, a ) = a T a Definition 2.13 (Angle): The angle θ between two m-dimensional vectors a and b is defined by: cos(θ) =
(a, b) (a, a ) ⋅ (b, b)
=
aT b bT a = la ⋅ lb la ⋅ lb
(2.20)
From Equation 2.20, cos(θ) = 0 only if aT b = 0 and we call a and b perpendicular, denoted by a ⊥ b . Given two m-dimensional vectors a and b, we can project one on the other. For example, when m = 2, Figure 2.2 illustrates the projection of a on b.
FIGURE 2.2 Angle between two vectors and projection of a on b.
2151_book.fm Page 22 Friday, October 20, 2006 5:04 PM
22
Stream of Variation Modeling and Analysis for MMPs
Definition 2.14 (Projection of vectors): The projection of a on b is defined by (a, b) aT b b ⋅ b= (b, b) lb lb where
(2.21)
b is the unit vector along the direction of b. The length of the projection is lb aT b aT b = la ⋅ = la | cos(θ) | lb la ⋅ lb
(2.22)
2.5 EIGENVALUES AND EIGENVECTORS OF A MATRIX Let A be an m × m matrix. It is said to have an eigenvalue λ and corresponding eigenvector x ≠ 0 if Ax = λx
(2.23)
Remark 2.12: Equation 2.23 has a clear geometrical interpretation. The left side A x indicates a linear transformation of vector x. The right side λ x means the new length of x. So Equation 2.23 indicates that when a linear transformation A is applied to an eigenvector, its length is multiplied by λ; however, its direction remains unchanged if λ > 0 and is reversed if λ < 0. The set of eigenvalues, denoted by λ(A), can be obtained by solving the socalled mth order characteristics polynomial or equation: A − λI = 0
(2.24)
A − λI = (− λ)m + t1 (− λ)m −1 + ... + tm −1 (− λ)1 + A
(2.25)
By using Laplace expansion,
where ti is the sum of all i × i principal minor determinants. So: m
t1 =
∑a
ii
= tr ( A)
(2.26)
i =1
Based on the theory of polynomial equations, it follows that: m
∏λ = A i
i =1
(2.27)
2151_book.fm Page 23 Friday, October 20, 2006 5:04 PM
Basics of Matrix Theory
23
m
∑ λ = tr (A)
(2.28)
i
i =1
Some properties of eigenvalues are summarized as follows: 1. The eigenvalues of a real symmetric matrix are all real. 2. The eigenvalues of a positive definite matrix are all positive. 3. For an m × m positive semidefinite symmetric matrix with rank r, i.e., | A | ≥ 0 and rank (A) = r, A has exactly r positive eigenvalues and m – r zero eigenvalues. 4. λ( AB) = λ(BA). 5. λ diag ( a11, a22 ,..., amm ) = a11, a22 ,..., amm .
(
) {
} After obtaining the set of eigenvalues λ( A) = {λ , λ , ..., λ } , the corresponding 1
2
m
eigenvectors can be calculated by solving the equation: A − λ i I ⋅ x i = 0
(2.29)
Remark 2.13: From Equation 2.29, it is apparent that the eigenvector x i corresponding to the eigenvalue λ i is not unique. For a symmetric matrix A, the eigenvalues and eigenvectors have the following properties: 1. If λ i ≠ λ j are two distinct eigenvalues of symmetric matrix A, then the associated eigenvectors x i and x j are perpendicular, i.e., x i ⊥ x j . 2. For any real symmetric matrix A , there exists an orthogonal matrix P such that
(
PTAP = diag λ1, λ 2 , …, λ m
)
(2.30)
Remark 2.14: Using the preceding properties, the quadratic form of symmetric matrices can be simplified by orthogonal transformation: x = Py
(2.31)
It follows immediately that m
x Ax = y P APy = T
T
T
∑λ y
2 i i
(2.32)
i =1
Remark 2.15: If A is not full rank, say rank(A) = r, then only r terms are left in the summation. Generally, the so-called spectral decomposition of an m × m symmetric matrix A is given by:
2151_book.fm Page 24 Friday, October 20, 2006 5:04 PM
24
Stream of Variation Modeling and Analysis for MMPs
A = λ1e1e1T + λ 2 e2 eT2 + ... + λ m em emT
(2.33)
where λi’s, i = 1, 2, …, m are eigenvalues of A and ei’s, i = 1, 2, …, m are associated normalized eigenvectors. So 1, i = j , i, j = 1, 2, ..., m eTi e j = 0, i ≠ j
(2.34)
Remark 2.16: Comparing Equation 2.32 and Equation 2.33, we can see that yi = x T ei , i = 1, 2, ..., m
(2.35)
2.6 VECTOR AND MATRIX DIFFERENTIATION, MAXIMIZATION, AND OPERATORS 2.6.1 DIFFERENTIATION
WITH
VECTORS
In later chapters, we will need to determine the partial derivatives of likelihoods and other scalar functions of vectors and matrices. Let f ( x ) = f ( x1, x2 ,..., xm ) be a continuous function whose first and second partial derivatives ∂f ( x ) ∂2 f ( x ) and ∂xi ∂xi ∂x j
(2.36)
exist for all points x in the span of m-dimensional Euclidean space. Definition 2.15 (Partial differentiation with vectors): The partial differentiation of function f ( x ) = f ( x1, x2 ,..., xm ) with respect to (w.r.t.) vector x is defined by: ∂f ( x ) ∂x 1 ∂f ( x ) ∂f ( x ) = ∂x2 . ∂x ∂f ( x ) ∂xm
(2.37)
The partial derivatives of some special forms of f ( x ) are summarized as follows.
2151_book.fm Page 25 Friday, October 20, 2006 5:04 PM
Basics of Matrix Theory
25
1. f (x) = constant for all x ∂f ( x ) =0 ∂x
(2.38)
∂f ( x ) =a ∂x
(2.39)
2. f (x) = a T x = x T a
3. Quadratic form: f ( x ) = x T Ax
(
)
∂f ( x ) = A + AT x ∂x
(2.40)
4. General quadratic form: f ( x ) = (a − Bx )T A(a − Bx ) where a is an m × 1 vector, B = [B1 B2 Bn] is an m × m matrix, and A is an m × m symmetric matrix. By using the chain rule: ∂f ( x ) = −2 BT A(a − Bx ) ∂x
(2.41)
Definition 2.16 (Hessian matrix): A Hessian matrix is defined by the matrix of second-order partial derivatives of a function f ( x ) = f ( x1, x2 ,..., xm ) w.r.t. vector x, which is ∂2 f ( x ) 2 ∂x1 ∂2 f ( x ) = H= ∂x T ∂x 2 ∂ f (x) ∂x ∂x 1 m
∂2 f ( x ) ∂x1∂xm ∂2 f ( x ) ∂xm2
(2.42)
As an example, the Hessian matrix of the quadratic form f ( x ) = x T Ax is 2A, if A is symmetric. After introducing definitions of first- and second-order partial derivatives, we can state the necessary condition for a maximum or minimum of f ( x ) at x = x*: ∂f (x) =0 ∂x x= x*
(2.43)
2151_book.fm Page 26 Friday, October 20, 2006 5:04 PM
26
Stream of Variation Modeling and Analysis for MMPs
Remark 2.17: This kind of maximum or minimum is called a stationary maximum or minimum because a global extreme may be located in the boundary of the feasible region of x, or as a cusp or another form at which the first-order derivative does not exist. A sufficient condition for a maximum (or minimum) at the point x* satisfying Equation 2.43 is that the Hessian matrix at x* H=
∂ 2 f (x) ∂x T ∂x x= x*
(2.44)
be negative (or positive) definite. Remark 2.18: If the Hessian matrix is indefinite at x*, then the point x* is not an extreme point.
2.6.2 MATRIX INEQUALITIES
AND
MAXIMIZATION
The matrix inequalities help obtain the maximization results. First, some important matrix inequalities are introduced. Cauchy–Schwarz Inequality: Let a and b be any two m × 1 vectors. Then (a, b)2 ≤ (a, a ) ⋅ (b, b) or (a T b)2 ≤ (a T a ) ⋅ (b T b)
(2.45)
The equality holds if and only if a = kb or b = ka, where k is a constant. An important extension of Cauchy–Schwarz Inequality is given as follows: Extended Cauchy–Schwarz Inequality: Let a and b be any two m × 1 vectors, and D be an m × m positive definite matrix. Then (a, b)2 ≤ (a, Da ) ⋅ (b, D −1b) or (a T b)2 ≤ (a T Da ) ⋅ (b T D −1b)
(2.46)
The equality holds if and only if a = kD–1b or b = kDa, where k is a constant. The proofs of Equation 2.45 and Equation 2.46 can be found in Reference 1. Based on the extended Cauchy–Schwarz Inequality, we have the following maximization theorem: Theorem 2.1 (Maximization): Suppose A is a positive definite matrix of order m × m and a is a given m × 1 vector. Then for ∀ x ≠ 0 of dimension m × 1 , max x ≠0
( x, a )2 = a T A −1a ( x, Ax )
(2.47)
The maximum is obtained when x = kA −1a for any constant k ≠ 0 . The following theorem gives the maximization result of quadratic forms for points on the unit sphere:
2151_book.fm Page 27 Friday, October 20, 2006 5:04 PM
Basics of Matrix Theory
27
Theorem 2.2 (Maximization of quadratic forms for points on the unit sphere): Suppose A is a positive definite matrix of dimension m × m with eigenvalues λ1 ≥ λ 2 ≥ ≥ λ m > 0
(2.48)
and corresponding eigenvectors ei , i = 1, 2, ..., m . Then ( x, Ax ) = λ1 obtained when x = e1 ( x, x ) ( x, Ax ) 2. min = λ m obtained when x = em x ≠ 0 ( x, x ) ( x, Ax ) 3. max = λ i+1 obtained when x = ei+1 i = 1, 2, ..., m − 1 x ⊥ e1,...,ei ( x, x ) 1. max x ≠0
(2.49) (2.50) (2.51)
The proofs can be found in Reference 1.
2.6.3 VEC OPERATOR, KRONECKER PRODUCT, AND HADAMARD PRODUCT 2.6.3.1 Vec Operator and Kronecker Product The concepts and results of the Kronecker product, which is also often called the direct product, are mentioned in many papers on variation modeling and analysis. Before introducing the definitions and some of the important properties of this product, we will first cover the concept of a vector operator, vec(⋅) . Definition 2.17 (Vector operator or Vec operation): Let X = [X1 X2 Xq] be a p × q matrix with columns X i ∈ R p ×1, i = 1, 2, …, q. Then the vec operator, vec(X) , is defined by stacking all the q columns of X in one column, i.e., X1 X2 vec (X) = ∈ R pq ×1 X q
(2.52)
Definition 2.18 (Kronecker product, or direct product): Given matrices A = ( aij ) ∈ R p ×q and B = (bij ) ∈ R r ×t , the Kronecker product of A and B, denoted by A ⊗ B, is the pr × qt matrix, a11B a21B A⊗B = a p1B
a12 B
a22 B
a p2B
a1q B a2 q B a pq B
(2.53) pr × qt
2151_book.fm Page 28 Friday, October 20, 2006 5:04 PM
28
Stream of Variation Modeling and Analysis for MMPs
Remark 2.19: Based on this definition, we can derive a special result about blockdiagonal matrices as follows: B 0 0
0 B 0
0 0 = Ip ⊗ B B pr × pt
(2.54)
where B ∈ R r ×t occurs p times on the diagonal. From the definition, it is easy to prove some important properties about the Kronecker product, which are summarized as follows: Property 1: ( aA) ⊗ (bB) = ab ( A ⊗ B) holds for any scalars a and b. Property 2: If A ∈ R p ×q , B ∈ R p ×q , and C ∈ R s ×t , then ( A + B) ⊗ C = A ⊗ C + B ⊗ C Property 3: ( A ⊗ B) ⊗ C = A ⊗ (B ⊗ C) Property 4: If A ∈ R m × n , B ∈ R p ×q , C ∈ R n ×s , and D ∈ R q ×t , then
( )
( A ⊗ B)(C ⊗ D) = AC ⊗ (BD) Property 5: ( A ⊗ B)T = A T ⊗ BT Property 6: If A and B are both nonsingular matrices, then ( A ⊗ B)−1 = A −1 ⊗ B−1 Property 7: If A ∈ R p × p and B ∈ R p × p , then tr ( A ⊗ B) = tr ( A) ⋅ tr (B) Property 8: If A and B are both orthogonal matrices, then A ⊗ B is also an orthogonal matrix. Property 9: If A ∈ R p × p , B ∈ R q ×q , then det( A ⊗ B) = [det( A)]q ⋅ [det(B)] p Property 10: If A > 0 and B > 0, then A ⊗ B > 0
2151_book.fm Page 29 Friday, October 20, 2006 5:04 PM
Basics of Matrix Theory
29
Property 11: If A ∈ R p × p and B ∈ R q ×q have eigenvalues αi, i = 1, 2, …, p and βj, i = 1, 2, …, q, respectively, then A ⊗ B has eigenvalues αiβj, i = 1, 2, …, p, j = 1, 2, …, q. The relationships between the vec operator and the Kronecker products are given in the following lemmas. Lemma 2.1: Let A ∈ R p ×m , X ∈ R m × n , and B ∈ R n ×q . Then vec ( AXB) = (BT ⊗ A) ⋅ vec (X)
(2.55)
The following remarks provide an application of this lemma. Remark 2.20: If X = [X1 X 2 X n ] ∈ R m × n is a random matrix whose columns are independent m × 1 random vectors each with the same covariance matrix Σ, i.e., cov(X i ) = Σ for i = 1, 2, …, n, then X1 X vec (X) = 2 ∈ R mn ×1 X n ∑ 0 cov[ vec (X)] = 0
0 ∑ 0
0 0 = In ⊗ ∑ ∈ R mn× nm ∑ mn× nm
(2.56)
(2.57)
Equation 2.57 holds because all the Xi’s are all independent with the same covariance matrix. Remark 2.21: Let A ∈ R p ×m and B ∈ R n ×q be constant matrices, and X ∈ R m × n be a random matrix whose columns are independent. Then applying this lemma and properties of the Kronecker product, the transformed random matrix Y = AXB has the following relationships: 1. E[vec( Y)] = (BT ⊗ A) ⋅ E[vec(X)] 2. cov[vec( Y)] = cov[(BT ⊗ A) ⋅ vec(X)] = (BT ⊗ A) ⋅ cov[vec(X)] ⋅ (BT ⊗ A)T = (BT ⊗ A) ⋅ (In ⊗ Σ) ⋅ (B ⊗ AT ) = (BT B) ⊗ ( AΣAT )
2151_book.fm Page 30 Friday, October 20, 2006 5:04 PM
30
Stream of Variation Modeling and Analysis for MMPs
Lemma 2.2: For appropriate sizes of matrices, the following statements hold: 1. vec( AB) = (I ⊗ A) ⋅ vec(B) = (BT ⊗ I) ⋅ vec( A) = (BT ⊗ A) ⋅ vec(I) 2. tr( ABC) = [ vec ( A T )]T (I ⊗ B) vec (C) 3. tr ( AX T BXC) = [ vec (X)]T ( A T C T ⊗ B) vec (X) = [ vec (X)]T (CA ⊗ BT ) vec (X) 2.6.3.2 Hadamard Product Definition 2.19 (Hadamard product): Suppose A and B are two m × m matrices. Their Hadamard product is the entrywise product of A and B, i.e., the m × n matrix A B whose (i,j)th entry is aijbij. For the following properties, suppose A, B, and C are m × m matrices and λ is a scalar. Property 1: A B = B A. Property 2: A (B + C) = A B + A C. Property 3: A (λB) = λ( A B). Property 4: If A and B are diagonal matrices, then A B = AB. Property 5 (Oppenheim inequality): If A and B are positive definite matrices and aii’s are the diagonal entries of A, then AB ≥
m
∏ a = B ii
i =1
with equality if and only if A is a diagonal matrix.
2.7 EXERCISES 1. Suppose aT = [3, 1, 4] and bT = [2, –3, 1]. a.Determine the length of a and b. b.What is the angle between a and b? c.What is the projection of a on b? d.Find the length of the projection in (c). 2. Let 3 A = 5 4 5
4 5 3 − 5
2151_book.fm Page 31 Friday, October 20, 2006 5:04 PM
Basics of Matrix Theory
31
a. Show if AA T = A T A = I , i.e., whether A is an orthogonal matrix. b. If A is an orthogonal matrix, verify that A −1 = A T . 3. Let 3 A= −2
−2 6
a. Check whether A is symmetric. b. Verify that A is positive definite. 4. Suppose 2 A= 2
2 . 5
Determine the following: a. The eigenvalues and eigenvectors of A. b. Calculate A–1. c. Calculate the eigenvalues and eigenvectors of A–1. 5. Let 1 A= 4
3 5
a. What is the determinant of A? b. Determine the trace of A. c. Determine whether the product of the eigenvalues of A is the same as the determinant of A. d. Determine whether the sum of the eigenvalues of A is the same as the trace of A. 6. Consider the symmetric matrix 5 A = 2 3
2 9 −2
3 −2 6
Verify whether A = λ1e1e1T + λ 2 e2 eT2 + λ 3e3eT3 . This is the so-called spectral decomposition of a symmetric matrix A. 7. Suppose f ( x ) = 2 x12 + 3 x1 x23 + 7 x32 . Find a.
∂f ( x ) ∂x
b. The Hessian matrix or
∂ 2 f (x) . ∂x T ∂x
2151_book.fm Page 32 Friday, October 20, 2006 5:04 PM
32
Stream of Variation Modeling and Analysis for MMPs
8. Suppose f (x) is in a quadratic form ( f (x) = xT Ax), i.e., 2 f ( x ) = x1 x2 x3 4 −1 Determine whether
(
3 −1 0
1 x1 5 x2 2 x3
)
∂f ( x ) = A + AT x . ∂x
References 1. Johnson, R.A. and Wichern, D.W., Applied Multivariate Statistical Analysis, 4th ed., Prentice Hall, Upper Saddle River, NJ, 1998. 2. Anderson, T.W., An Introduction to Multivariate Statistical Analysis, John Wiley & Sons, New York, 1958. 3. Graybill, F.A., Introduction to Matrices with Applications in Statistics, Wadsworth Publishing Company, Inc., Belmont, CA, 1969. 4. Harville, D.A., Matrix Algebra from a Statistician’s Perspective, Springer-Verlag, New York, 1997. 5. Noble, B. and Daniel, J.W., Applied Linear Algebra, 3rd ed., Prentice Hall, Englewood Cliffs, NJ, 1987.
2151_book.fm Page 33 Friday, October 20, 2006 5:04 PM
3
Basics of Multivariate Statistical Analysis
3.1 INTRODUCTION In variation modeling and analysis, it is common to consider multiple random variables. Thus, multivariate statistical analysis is essential to understand and master variation reduction methodologies. This chapter provides some basics of multivariate statistical analysis, including underlying theoretical population settings for random phenomena involving multivariate distributions, marginal distributions, conditional distributions, population means, variances, covariances, correlations, and other characteristics. The most important multivariate distribution, the multivariate normal distribution, is discussed in detail and some important results are summarized. An introduction of sampling theory is also included in this chapter. It should be noted that all the proofs are not provided here. Readers who want to learn more about the proofs may please refer to related mathematical statistics materials [1–10].
3.2 MULTIVARIATE DISTRIBUTION AND PROPERTIES 3.2.1 RANDOM VECTORS, CUMULATIVE DISTRIBUTION FUNCTIONS (CDF), AND PROBABILITY DENSITY FUNCTIONS (PDF) A random vector is a vector comprising elements that are random variables. Uppercase letters denote random variables and random vectors, and corresponding lowercase letters denote realizations of the random variables or random vectors. Definition 3.1 (Cumulative distribution function): Let X = [ X1, X2 , , X p ]T be a random vector with X1, X2 , , X p as the p random variables. The associated joint CDF is the function F defined by: F ( x ) = Pr(X ≤ x ) = Pr( X1 ≤ x1 , X2 ≤ x2 , , X p ≤ x p ) ,
(3.1)
where x = [ x1, x2 , , x p ]T is a realization of random vector X. Two important cases are absolutely continuous and discrete distributions. Definition 3.2 (Absolutely continuous distributions): A random vector X is absolutely continuous if there exists a probability density function (PDF), f ( x ) , s.t., F (x) =
∫
x
−∞
∫ ∫ xp
f (u) du =
−∞
x1
−∞
f (u1, u2 , , u p ) du1 du p
33
2151_book.fm Page 34 Friday, October 20, 2006 5:04 PM
34
Stream of Variation Modeling and Analysis for MMPs
Here, f ( x ) is called a joint PDF. Note that for any measurable set D ⊆ R p , Pr(X ∈ D ) =
∫ f (u)du
(3.2)
D
and
∫
+∞
−∞
f (u) du = 1
(3.3)
Definition 3.3 (Discrete distributions): For a discrete random vector X, the total probability is concentrated on a countable (or finite) set of points {χi ; i = 1, 2, }. Its probability mass function (PMF) is defined as: f (x i ) = Pr(X = x i ) for i = 1, 2, ; and f (x) = 0, otherwise.
(3.4)
Similarly, Pr(X ∈ D) =
∑ f (x ) i
i:xi ∈D
Then, for both cases, the support S of X is defined as the set:
{
S = x ∈ R p : f (x) > 0
3.2.2 MARGINAL
AND
}
CONDITIONAL DISTRIBUTIONS
Consider the partitioned vector X X = 1, X2 where X1 ∈ R k ×1 and X 2 ∈ R ( p − k )×1 , respectively. Definition 3.4 (Marginal CDF and PDF): Under the pervious partition, the function:
(
)
(
Pr X1 ≤ x10 = Pr X11 ≤ x10 , X 21 ≤ x 20 ,, X k1 ≤ x k0 , X k1+1 ≤ ∞,, X 1p ≤ ∞
)
(3.5)
is called the marginal CDF of X1. Let f ( x ) be the joint PDF of X. Then the marginal PDF of X1 is given by: f1 ( x 1 ) =
∫
+∞
−∞
f ( x 1 , x 2 ) dx 2
The marginal PDF of X2 can be defined in a similar manner.
(3.6)
2151_book.fm Page 35 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
35
Definition 3.5 (Conditional PDF): Under the previous partition, the conditional PDF of X2, given X1 = x10 , is defined as:
(
f x 2 | X1 = x
0 1
) = (f ( x ) ) f x10 , x 2
(3.7)
0 1
1
(Assume f1 ( x10 ) ≠ 0). Similarly, we can define the conditional PDF of X1 given X2 = x 02 . Based on the conditional distribution, we can introduce the important concept statistical independence as follows: Definition 3.6 (Statistical independence): When the conditional PDF f (x2|X1 = x 10 ) is the same for all possible values of x10 , then X1 and X2 are said to be statistically independent of each other, denoted as: X1 ⊥⊥ X2. Theorem 3.1: If X1 ⊥⊥ X2, then f ( x ) = f1 ( x 1 ) f 2 ( x 2 )
(3.8)
3.2.3 POPULATION MOMENTS Definition 3.7 (Population mean vector): Suppose X = [ X1, X2 , , X p ]T is a random vector with PDF f ( x ) . Then the expectation or mean of random vector X is defined as: E ( X1 ) E(X2 ) µX = E X = E ( X p )
( )
∫
(3.9)
+∞
xi fi ( xi ) dxi , i = 1, 2, , p . µ X is also called where µX = [µ1, µ2, …, µp]T and µ i = −∞ the population mean vector. It is easy to show that the expectation operator has the following properties:
(
)
( ) X Partition: Let X = , then E ( X ) = ∫ x X If X ⊥⊥ X , then E ( X X ) = E ( X ) E ( X ) .
1. Linearity: E AX + b = AE X + b for any A ∈ R q × p and b ∈R q . 2.
+∞
1
1
2
3.
1
2
1
2
1
−∞
f ( x 1 ) dx 1 .
1 1
2
Definition 3.8 (Population covariance matrix): The covariance matrix (or variancecovariance matrix, or dispersion matrix) of random vector X = [ X1, X2 , , X p ]T is defined as:
(
∑ X = var(X) = E (X − µ X )(X − µ X )T
)
(3.10)
2151_book.fm Page 36 Friday, October 20, 2006 5:04 PM
36
Stream of Variation Modeling and Analysis for MMPs
For simplicity, if random vector X = [ X1, X2 , , X p ]T has mean vector µ X and covariance matrix ∑ X , we denote it as X ~ ( µ X , ∑ X ) or ( µ , ∑ ). It can be proved that the covariance has the following properties: 1. ∑ X is semidefinite, i.e., for all a ∈R p , a T ∑ X a ≥ 0. 2. ∑ X is symmetric, i.e., ∑TX = ∑ X with diagonal elements σ ii = var( Xi ) and the (i, j)th element σ ij = cov( Xi , X j ) for i ≠ j. Thus,
( )
∑ X = σ ij σ11 σ 21 σ p1
p× p
=
σ12
σ 22
σ p2
σ1 p var( X1 ) σ 2 p cov( X1 , X 2 ) = σ pp cov( X1 , X p )
cov( X1 , X 2 )
var( X 2 )
cov( X 2 , X p )
cov( X1 , X p ) cov( X 2 , X p ) . var( X p )
∑ X = E (XXT ) − µ Xµ TX . For all constant vectors, a ∈R p , var(a T X) = a T ∑ X a. var( AX + b) = A ∑ X AT , where A is a matrix of an appropriate size. cov(X, X) = ∑ X. cov(X1 + X 2 , Y) = cov(X1, Y) + cov(X 2 , Y) . T Let X, Y ∈R p, then cov(X, Y) = cov( Y, X) and var(X + Y) = ∑ X + cov(X, Y) + cov( Y, X) + ∑ Y . 9. cov( AX, BY) = A cov(X, Y)BT, where A and B are matrices of appropriate sizes. 10. If X ⊥⊥ Y, then cov( X , Y ) = 0, but the converse may not be true. 3. 4. 5. 6. 7. 8.
Based on the covariance matrix, we can define the statistical distance between two random vectors. Clearly, it is a special case of Mahalanobis distance. Definition 3.9 (Statistical distance or Mahalanobis distance): If X, Y ∈R p , then the statistical distance between X and Y with matrix ∑ is defined as: d ∑ (X, Y) = (X − Y)T ∑ −1 (X − Y)
(3.11)
For convenience, the matrix ∑ is usually chosen to be the covariance matrix. For example: µ, ∑). The Mahalanobis distance between X and µ, d ∑ (X, µ ), is 1. Let X ~ (µ a random variable. 2. Let X ~ ( µ X , ∑) and Y ~ ( µ Y , ∑). Then d ∑ (µ X , µ Y ) is a Mahalanobis distance between the parameters. Also, for any nonsingular matrix A ∈ R p × p , d ∑ (µ X , µ Y ) is invariant under the following transformations:
2151_book.fm Page 37 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
37
= AX + b X :→ X = AY + b Y :→ Y = A ∑ AT ∑ :→ ∑ i.e., d ∑ (µ X , µ Y ) = d ∑ (µ X , µ Y ).
3.2.4 CORRELATION COEFFICIENTS There are three types of correlation coefficients: simple correlation coefficients, partial correlation coefficients, and multiple correlation coefficients. It should be noted that partial correlation coefficients and multiple correlation coefficients are special cases of simple correlation coefficients. µx, ∑x). As before, suppose X = [ X1, X2 , , X p ]T is a random vector with X ~ (µ Definition 3.10 (Simple correlation coefficients): The correlation coefficient between any two random variables Xi and Xj is defined as: ρij =
cov( Xi , X j ) var( Xi ) ⋅ var( X j )
=
σ ij σ ii ⋅ σ jj
(3.12)
Clearly, ρij is the normalized covariance of Xi and Xj , for i, j = 1, 2, …, p. Properties of ρij : 1. ρij is dimensionless. 2. –1 ≤ ρij ≤ +1. 3. If ρij = –1 or +1, there is a line in the Xi and X j sample space of the form X j = a + bXi such that all the nonzero probability values for the bivariate distribution of Xi and X j lie on the line. Furthermore, if ρij > 0, the slope is greater than zero; if ρij < 0, the slope is less than zero. 4. If –1 < ρij < +1, the value of ρij is a relatively subjective measure of the tendency of Xi’s and Xj’s probabilities to concentrate about the same line. It can be seen that the simple correlation coefficient between any two random variables Xi and X j describes their joint behavior, whereas the partial correlation coefficient reflects the behavior of two random variables when other random variables are held fixed. For example, considering random variables Y , X1, X2 , , Xl , the partial correlation between Y and X1 reflects the behavior between them when the other random variables X2 , , Xl are held fixed, which is commonly denoted by ρX1Y•X2X3Xl or ρ X1Y |X2 X3Xl . Without loss of generality, let X1, X2 , , X k and X k +1, X k +2 , , X k +r be any two subsets of a collection of random variables X1, X2 , , X k , X k +1, , X p (2 ≤ k < p).
2151_book.fm Page 38 Friday, October 20, 2006 5:04 PM
38
Stream of Variation Modeling and Analysis for MMPs
Definition 3.11 (Partial correlation coefficients or conditional correlation coefficients): Suppose Xi and Xj are any two of the random variables X1, X2 , , X k , and σ ij•k +1,k +2,,k +r is the conditional covariance of Xi and Xj in the joint conditional distribution, the partial correlation coefficient of Xi and Xj given X k +1, X k +2 , , X k +r is defined as: ρ X i X j • X k +1 X k + 2 X k + r =
σ ij•k +1,k +2,,k +r σ ii•k +1,k +2,,k +r ⋅ σ jj•k +1,k +2,,k +r
(3.13)
The partial correlation coefficient can be calculated from simple correlation coefficients. Let Z be the collection of random variables X k +1, X k +2 , , X k +r ; then Equation 3.13 can be expressed in a more explicit but equivalent form: ρ Xi X j •Z =
σ Xi X j •Z σ Xi Xi •Z ⋅ σ X j X j •Z
ρ Xi X j − ρ XiZ ⋅ ρ X j Z
=
(1 − ρ2XiZ )(1 − ρ2X jZ )
(3.13)
Equation (3.13) also gives the calculation approach. The interpretation of a partial correlation coefficient is the same as that of a simple correlation coefficient except that the partial correlation coefficient generally depends on the conditional values of the conditional random variables. Graphically, the simple correlation coefficient describes the data in an ordinary scatter plot, whereas the partial correlation coefficient describes the data in the partial regression residual plot. For example, considering the collection of random variables Y, X1, X2, , Xl, suppose Y and X1 are of interest and the remaining variables are fixed values. The residuals of Y and X1 are calculated by regressing on X2 , , Xl , respectively. These residuals are parts of Y and X1 that cannot be predicted from X2 , , Xl . The partial correlation coefficient between Y and X1 after adjusting X2 , , Xl is the correlation between these two sets of residuals. Furthermore, the regression coefficient when Y’s residuals are regressed on X1’s residuals is equal to the regression coefficient of X1 in the multiple regression equation when Y is regressed on X1, X2 , , Xl . For more details, please refer to the regression section in Chapter 4. The simple correlation coefficient is a measure of the linear relationship between two random variables. The partial correlation coefficient is a measure of the linear relationship between two random variables when other random variables are held fixed, and the multiple correlation coefficients are a measure of the linear relationship between a random variable and a set of other random variables. Definition 3.12 (Multiple correlation coefficients): Without loss of generality, suppose X1 is a random variable and X2, X3, , Xp are a set of other random variables. Let p
Y=
∑a X i
i=2
i
2151_book.fm Page 39 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
39
be an arbitrary linear combination of X2, X3, , Xp. Then the multiple correlation coefficient of X1 and X2, X3, , Xp is the maximum simple correlation coefficient of X1 and Y for variation over ai, i = 2, 3, , p. For example, let ρ X1Y ( a2 , a3 , , a p ) =
cov( X1, Y ) var( X1 ) ⋅ var(Y )
(3.14)
Then the multiple correlation coefficient of X1 and X2 , X3 , , X p , denoted by ρ1•2 3p, is defined as the maximum of ρ X1Y ( a2 , a3 , , a p ) w.r.t. a2 , a3 , , a p , i.e.,
{
}
ρ1•23p = max ρX1Y (a2 , a3 , , a p ) ai ,2≤i ≤ p
It can be calculated according to the following procedure: 1. Let X = [ X1, X2 , , X p ]T ; partition ∑ in the following manner ∑ ∑ = var(X) = 11 T ∑12
∑12 ∑ 22
where Σ11 = var( X1 ) = σ11 ∈ R 1×1 , ∑12 = [cov( X1 , X 2 ) cov( X1, X p )] = [σ12 σ1 p ] and ∑ 22 = cov(X −1, X −1 ) , where X −1 = [ X2 , X3 , , X p ]T . 2. Based on the definition of ρ X1Y ( a2 , a3 , , a p ) , compute
ρ2 X1Y ( a2 , a3 , , a p ) =
(Σ
(
a
12 −1
)
2
σ11 a T−1Σ 22a −1
)
where a −1 = [a2 a p ]. 3. Calculate ρ21•23p by
{
}
ρ21•23p = max ρ2 X1Y ( a2 , a3 , , a p ) . ai ,2 ≤i≤ p
It can be shown that ρ21•23p =
(
)
1 T . ∑12 ∑ −221 ∑12 σ11
(3.15)
2151_book.fm Page 40 Friday, October 20, 2006 5:04 PM
40
Stream of Variation Modeling and Analysis for MMPs
4. Determine ρ1•23p by taking the square root of ρ21•23p , i.e., ρ1•23p = ± ρ21•23p . Remark 3.1: Let S S = 11 T S12
S12 S22
be the sample covariance matrix. Then the sample multiple correlation coefficient is: r 21•23 p =
(
)
1 T S12S−221S12 . S11
Remark 3.2: The multiple correlation coefficient can be used to test whether X1 is independent of X2 , X3 , , X p by the F-test with appropriate degrees of freedom. Remark 3.3: The multiple correlation coefficient describes the correlation between one dependent variable X1 and its best estimate from a regression on several independent variables X2 , X3 , , X p. The squared multiple correlation coefficient is the proportion of variance accounted for the multiple regression.
3.3 MULTIVARIATE NORMAL DISTRIBUTION AND QUADRATIC FORMS The basic and central distribution in multivariate analysis is the multivariate normal distribution. This is due to two main factors. First, multivariate observations are often normally distributed, or approximately normally distributed, due to the central limit theorem. This fact lays the foundation for formal inferences. Second, the multivariate normal distribution and the sampling distributions are tractable. This is not generally the case with other multivariate distributions, even for those that are close to normality.
3.3.1 MULTIVARIATE NORMAL DISTRIBUTION
AND ITS
PROPERTIES
Definition 3.13 (Multivariate normal distribution): Let X = [X1, X2, , Xp]T ∈R p µx, ∑ X ), where all the p random variables Xi are be a random vector with X ~ (µ continuous s.t. –∞ < Xi < ∞, and ∑ X is a positive definite matrix. Thus S = {( x1 , x 2 ,, x p )T | − ∞ < xi < ∞, i = 1, 2, ..., p} forms the sample space for X. Then random vector X is said to have a p-variate normal distribution if its multivariate probability distribution is given by: f (X) =
1 (2π )
p/ 2
∑X
1/ 2
1 exp − X − µ X 2
(
)
T
∑ X−1 X − µ X
(
Often, it is denoted as X ~ N p (µ X , ∑ X ), or simply N p (µ , ∑).
)
(3.16)
2151_book.fm Page 41 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
41
Properties of multivariate normal distribution: 1. If X follows a p-variate normal distribution or X ~ N p (µ X , ∑ X ), then both µX ≡ E(X) and ∑ X ≡ cov(X) exist and the distribution of X is uniquely determined by µ X and ∑ X. 2. If X ~ N p (µ , ∑), and A ∈ R q × p , b ∈R q , then Y = AX + b has a q-variate µ + b, A ∑ AT). normal distribution with Y ~ N q (Aµ 3. If X ~ N p (µ , ∑), then the marginal distribution of any subset of k(k < p) components of X is k-variate normal. 4. If X ~ N p (µ , ∑), for the partition: X µ ∑ X = 1 , µ = 1 , ∑ = 11 X2 µ 2 ∑ 21
∑12 , ∑ 22
where X1 ∈R k, µ1 ∈R k , and ∑11 ∈ R k × k, the subvectors X1 and X2 are independent iff ∑12 = 0, i.e., X1 ⊥⊥ X 2 ⇔ ∑12 = 0
(3.17)
Remark 3.4: Property 4 can be easily extended to the case that X is partitioned into a number of blocks with more than two subvectors. Remark 3.5: Property 4 shows that, in order to determine whether two subvectors of a normally distributed vector are independent, it suffices to check that covariance matrices between these two subvectors are zero. 5. Contours of N p (µ , ∑) are defined as:
(X − µ )
T
(
)
Σ −1 X − µ = C 2
(3.18)
where C is a position constant. So Equation 3.18 defines an ellipsoid with its center at µ and axes ±C λ i ei , where λ i and ei are the eigenvalues and associated eigenvectors of ∑ . So µ determines its position and ∑ defines its shape and orientation. 6. (Regression) Given X ~ N p (µ , ∑), and the partition: X µ Σ X = 1 , µ = 1 , and Σ = 11 X2 µ 2 Σ 21
Σ12 Σ 22
where X1 ∈R k , µ1 ∈R k , and Σ11 ∈ R k × k . Let ∑ −22 be the generalized inverse (1) − − of ∑ 22 , i.e., ∑ 22 = ∑ 22 ∑ 22 ∑ 22 , and define ∑11 = ∑11 − ∑12 ∑ 22 ∑ 21 . Then:
2151_book.fm Page 42 Friday, October 20, 2006 5:04 PM
42
Stream of Variation Modeling and Analysis for MMPs − − (1) a. X1 − ∑12 ∑ 22 X 2 ~ N k (µ1 − ∑12 ∑ 22 µ 2 , ∑11 ); furthermore, − X1 − ∑12 ∑ 22 X2 ⊥⊥ X 2
(3.19)
b. Conditional distribution of X1 given X 2 = x 2 is: − (1) X1 X 2 = x 2 ~ N k (µ1 + ∑12 ∑ 22 (x 2 − µ 2 ), ∑11 ) − Remark 3.6: E( X1 X 2 ) = µ1 + ∑12 ∑ 22 (X 2 − µ 2 )
(3.20) (3.21)
(1) − var( X1 X 2 ) = ∑11 = ∑11 − ∑12 ∑ 22 ∑ 21
(3.22)
Rewrite E( X1 X 2 ) as: E(X1 X 2 ) = (µ1 − ∑12 ∑ −22 µ 2 ) + ∑12 ∑ −22 X 2 = b + AX 2
(3.23)
− − µ 2 ), A = ∑12 ∑ 22 . So E( X1 X 2 ) is usually called the regreswhere b = (µ1 − ∑12 ∑ 22 sion function of X1 on X2 with the intercept vector b and the regression matrix A. It is a linear regression function because it depends linearly on the variable X2 being held constant, whereas var( X1 X 2 ) is constant without depending on X2.
7. For random vectors X and Y ∈R p , if X ⊥⊥ Y and X + Y is p-variate normally distributed, then both X and Y are p-variate normal. 8. Linear combination of random vectors: If X1, X2, …, XN ∈R p are all independent random vectors and Xi ~ N p (µ i , ∑i ) for i = 1, 2, …, N, then: a. For any fixed constant a T = [a1 aN ], N
∑ i =1
ai X i ~ N p
N
N
∑
aiµ i ,
i =1
∑ i =1
ai2 ∑ i
(3.24)
b. As a special case of Equation 3.24, suppose Xi iid ~ N p (µ , ∑) for i = 1, 2, , N and a1 = = aN = 1/N, then: 1 X ~ N p µ, ∑ , N where X=
1 N
N
∑X i =1
is called the sample mean vector.
i
(3.25)
2151_book.fm Page 43 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
43
9. Let X ~ N p (µ , ∑), where ∑ is nonsingular, so that there exists a nonsingular matrix D such that ∑ = DDT . Then the transformation Y = D–1 (X – µ) ~ N p (0, I p ).
3.3.2 QUADRATIC FORMS Quadratic forms in multivariate normal random variables arise very often and play an important role in multivariate analysis. Some important results of certain quadratic forms under various parametric conditions are given in this subsection. Theorem 3.2: Suppose X ~ N p (µ , ∑), where ∑ is nonsingular, then: 1. (X − µ )T ∑ −1 (X − µ ) ~ χ2p and Prob. {(X – µ)T ∑ −1 (X − µ ) ≤ χp2(α)} = 1 – α, (3.26) where χ2p (α ) is the upper α percentile of the χ2 distribution with p degree of freedom. 2. XT ∑ −1 X ~ χ2p ( ∆) (3.27) T −1 where ∆ = µ ∑ µ . Remark 3.7: This result is used in testing the hypothesis about the mean of a multivariate normal distribution when the covariance matrix is known, e.g., as a χ2 control chart in a quality control problem. Remark 3.8: From this theorem and Property 8, it is easy to show that: N (X − µ )T ∑ −1 (X − µ ) ~ χ2p
(3.28)
Remark 3.9: If X ~ N p (µ , ∑), then (X − µ )T ∑ −1 (X − µ ) is the squared statistical distance (squared Mahalanobis distance) from X to the population mean vector µ . Remark 3.10: If σ ii > σ jj , then Xi contributes less to the statistical distance than Xj; if σij > σst, then Xi and X j contribute less to the statistical distance than Xs and Xt . Theorem 3.3: Suppose X ~ N p (µ , ∑), where ∑ is nonsingular. Then for the partition: X µ ∑ X = 1 , µ = 1 , and ∑ = 11 X2 µ 2 ∑ 21
∑12 , ∑ 22
where X1 ∈R k , , µ1 ∈R k , and ∑11 ∈ R k × k , the following result holds:
(
)
−1 (X − µ )T ∑ −1 (X − µ ) − (X1 − µ1 )T ∑11 X1 − µ1 ~ χ2p − k
(3.29)
Theorem 3.4 Suppose X ~ N p (0, ∑), where rank (∑) = r (r ≤ p), and ∑ − is the generalized inverse of Σ, i.e., ΣΣ − Σ = Σ , then XT ∑ − X ~ χ2r
(3.30)
2151_book.fm Page 44 Friday, October 20, 2006 5:04 PM
44
Stream of Variation Modeling and Analysis for MMPs
3.3.3 NONCENTRAL χ2
AND
F DISTRIBUTIONS
The noncentral χ2 distribution can be expressed as a mixture of central χ2 distributions where the weights are Poisson probabilities. Definition 3.14 (Noncentral χ2 distribution): If X ~ N p (µ , I p ), then the random variable Z = X T X has the CDF:
∞
F ( z; p, δ) =
∑ i=0
1 i δ δ 2 e − 2 Pr(χ2 ≤ z) p +2 i i!
(3.31)
where δ = µ T µ. Z is said to have the noncentral χ2 distribution with degree of freedom (d.o.f.) p and noncentrality parameter δ denoted by χ2p (δ) . Corollary: If Z ~ χ2p (δ) , then its PDF can be expressed as: ∞
∑ P (K = k )g
( z) , z > 0
(3.32)
1 e − z / 2 z r / 2 −1 Γ ( 21 r )
(3.33)
p +2 k
k =0
where K ~ Poisson (mean = δ/2), and gr ( z) =
2
r /2
is the density function of the χ2r distribution. Property 1: If Z ~ χ2p (δ) , then E ( Z ) = p + δ , var( Z ) = 2 p + 4δ . Property 2: If Z1 ~ χ2p1 (δ1 ) and Z 2 ~ χ2p2 (δ 2 ) , then Z1 + Z 2 ~ χ2p1+ p2 (δ1 + δ 2 ) . Recall that the central F distribution is obtained by taking the ratio of two independent χ2 variables divided by their degrees of freedom. The noncentral F distribution is obtained by allowing the numerator variable to be noncentral χ2. Definition 3.15 (Noncentral F distribution): If Z1 ~ χ2p1 (δ) , Z 2 ~ χ2p2 , and Z1 is independent of Z2, then F=
Z1 /p1 Z 2 /p2
(3.34)
is said to have the noncentral F distribution with d.o.f. p1 and p2, and noncentrality parameter δ, denoted by Fp1 , p2 (δ).
2151_book.fm Page 45 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
45
Property: If F ~ Fp1, p2 (δ) , then E (F ) =
p2 ( p1 + δ) , p2 > 2 p1 ( p2 − 2 )
(3.35)
2
p ( p + δ)2 + ( p1 + 2 δ)( p2 − 2 ) var( F ) = 2 2 1 , p2 > 4 p1 ( p2 − 2 )2 ( p2 − 4)
(3.36)
Remark 3.11: The noncentral χ2 distribution can be calculated by the Matlab® function NCX2CDF(x,df,delta) or SAS function PROBCHI(x,df,delta). Both return the noncentral chi-square probability with df degrees of freedom and noncentrality parameter, delta, at the values in x. Remark 3.12: The noncentral F distribution can be calculated by the Matlab function NCFCDF(x,ndf,ddf,delta) or SAS function PROBF(x,ndf,ddf,delta). Both return the noncentral F CDF with numerator degree of freedom, ndf; denominator degree of freedom, ddf; and noncentrality parameter, delta, at the values in x. For more details about these functions please refer to the related Matlab [11] and SAS manual [12].
3.4 SAMPLING THEORY 3.4.1 SAMPLE GEOMETRY Assume X1, X2, …, Xn ∈R p are n random vectors and x1, x 2 ,..., x n are associated realizations. Let x be an n × p matrix: x1T T x x= 2= T x n
x11 x21 x n1
x12
x22
xn 2
x1 p x2 p x np
(3.37) n× p
Definition 3.16 (Observation matrix, sample matrix): An observation matrix x with n multivariate observations (or sample size n) is defined as shown in Equation 3.37; each row is called a single multivariate observation and is the collection of measurements of p different variables taken at the same trial. Remark 3.13: Matrix x is also referred to as a sample of size n from a p-variate population. From Definition 3.16, each row of an observation matrix is a realization of p-variate random vectors.
2151_book.fm Page 46 Friday, October 20, 2006 5:04 PM
46
Stream of Variation Modeling and Analysis for MMPs
Remark 3.14: In observation matrix x, all the rows are independent of each other. For simplicity, we introduce the 1-vector: T
1n = 1 1 1 ∈ R n
(3.38)
We define the sample mean vector x, sample covariance matrix s n , and sample correlation matrix as follows: Sample mean vector: 1 x= n
n
∑ x = n1 x 1 T
(3.39)
i
i =1
Sample covariance matrix: 1 sn = n −1
n
∑ (x − x)(x − x) i
T
i
i =1
(3.40)
(
1 x − 1x T = n −1
) ( x − 1x ) T
T
1 T 1 = x I − 11T x n −1 n
Sample correlation matrix: s ij R n = rij = sii s jj
( )
(3.41)
where sij is the (i, j) entry of sn. To simplify the notation, let n
A=
∑ (x − x)(x − x) i
i
T
(3.42)
i =1
So Equation 3.40 becomes
sn =
1 A n −1
(3.43)
1 Remark 3.15: sn is a symmetric matrix with p variances and p ( p − 1) potential 2 different covariances.
2151_book.fm Page 47 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
47
Remark 3.16: sn can be denoted by
( )
s n = sij , i, j = 1, 2, …, p
(3.44)
where sij =
1 n −1
n
∑ (x
ki
− xi )( x kj − x j ) .
k =1
The observation matrix defined in Equation 3.37 can be geometrically interpreted in the following ways: 1. Express x1T T x x = 2 = [x1 x 2 x n ]T . T x n According to this expression, the total n data points, xi = [xi1 xi2 xip]T (i = 1, 2, …, n) can be plotted in R p space. This scatter plot contains rich information about the locations and variability of these points. 2. Express x as x = [x 1 x 2 x p ], where x1 j x2 j x j = , j = 1, 2, …, p x nj Here, there are a total of p vectors in R n space. Each x j can be decomposed into two mutually perpendicular parts, x1 j − x j x2 j − x j d j = x j − x j 1 n = and x j 1 n , x nj − x j which is illustrated in Figure 3.1.
2151_book.fm Page 48 Friday, October 20, 2006 5:04 PM
48
Stream of Variation Modeling and Analysis for MMPs
~ xj
~ xj
x j 1n
x j 1n FIGURE 3.1 Illustration of the decomposition of each column vector x~ j of data matrix x = [x~ 1 x~ 2 x~ p], where x–j is the mean value of the jth column vector x~ j, i.e., xj =
1 n
∑x n
ij
=
i =1
1 n
T x j ⋅ 1n.
In the decomposition x j = x j 1 n + d j , j = 1, 2, …, p
(3.45)
each x j 1 n is called a mean vector and d j is called the associated residual vector. The residual matrix is defined by:
d1
x1T T x d p = 2 T x n
d2
− xT − xT = x − 1n x T − x T
The length of each residual vector d j is given by: n
ld j = d j = d Tj d j =
∑ (x
kj
− x j )2 = ( n − 1)s jj
(3.46)
k =1
For any two residual vectors d i and d j , the inner product (d i , d j ) = d Ti d j n
=
∑
( x ki − xi )( x kj − x j ) = (n − 1)sij
(3.47)
k =1
= l di l d j cos(Q ij )
(3.48)
2151_book.fm Page 49 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
49
where θij is the angle between two residual vectors d i and d j . If we substitute Equation 3.48 with Equation 3.46, then: n
(d i , d j ) =
∑ (x
ki
− xi )( x kj − x j )
k =1
n
=
∑
n
( x ki − xi )2
∑ (x
kj
− x j )2 cos(θij )
k =1
k =1
This implies that: cos(θij ) =
sij sii s jj
= rij
(3.49)
i.e., cos(θij) is the sample correlation coefficient. Note the following: 1. If di is nearly parallel to dj with the same direction, i.e., θij ≈ 0°, then ρij ≈ 1; if di is nearly parallel to dj but with opposite direction, i.e., θij ≈ 180°, then ρij ≈ –1. 2. If di and dj are mutually perpendicular, then ρij = 0.
3.4.2 RANDOM SAMPLES AND THE EXPECTED VALUES OF THE SAMPLE MEAN AND COVARIANCE MATRIX The definition data matrix x is clearly constructed by the observations or realizations of a p-variate random vector. But before taking the actual measurements, we cannot predict these values exactly. So each set of measurements X i on p variables is a random vector and the matrix X =[X1 X2 Xn]T is actually a random matrix. Definition 3.17 (Random sample): A p-variate Xi, for i = 1, 2, …, n, is said to be a random sample taken from a common joint distribution with density function f ( x ) = f ( x1, x2 , , x p ) if the row vectors in X =[X1 X2 Xn]T, i.e., XTi, i = 1, 2, …, n, are independent observations from f (x), i.e., the joint PDF of X1 X2 Xn is given by n
∏ f (x ) , i
i =1
where f ( x i ) = f ( xi1, xi 2 , , xip ) is the PDF for the ith row vector.
2151_book.fm Page 50 Friday, October 20, 2006 5:04 PM
50
Stream of Variation Modeling and Analysis for MMPs
As the sample mean vector X is also a random vector, Theorem 3.5 gives its expected value and covariance matrix. Before introducing the theorem, we need the following lemma. Lemma 3.1: For any random vector Y with mean vector E( Y) = µ and covariance matrix cov( Y) = ∑ . The following relationship holds: E ( YY T ) = ∑ +µµ T
(3.50)
Theorem 3.5: Let X1 X2 Xn ∈R p be a random sample taken from a joint µ, ∑). Then the sample mean vector X is an unbiased estimator distribution with (µ of µ and its covariance matrix is n1 ∑, i.e, E(X) = µ
cov(X) =
(3.51)
1 ∑ n
(3.52)
The sample variance-covariance matrix s n defined in Equation 3.40 is an unbiased estimator for ∑, i.e., E (s n ) = ∑
3.4.3 SOME IMPORTANT RESULTS OF SAMPLING MULTIVARIATE NORMAL DISTRIBUTIONS
(3.53) FROM
For multivariate normal distributed random vectors, there exist the following fundamental results: Theorem 3.6 (Distributions of sample means and covariance matrices): Suppose X1, X2, , Xn are independently identically distributed (i.i.d.) random samples drawn from N p (µ , ∑), then: 1. Sample mean vector X =
1 n
n
∑ X ~ N (µ, n1 ∑). So i
p
i =1
n (X − µ ) ~ N p (0, ∑).
(3.54)
2. (n – 1)sn is distributed as a Wishart random matrix with d.o.f. = n – 1. – 3. X and sn are independent where
sn =
1 n −1
n
∑ (X − X)(X − X) . T
i
i =1
i
2151_book.fm Page 51 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
51
Theorem 3.7 (Multivariate central limit theorem, Cramer): Let X1, X2, , Xn be i.i.d random vectors with mean µ and positive definite ∑, and let 1 X= n
n
∑X . i
i =1
Then, (1) n (X − µ ) → N p (0, ∑), as n → ∞. (2) n(X − µ )T s −n1 (X − µ ) → χ 2p , for n – p is large.
(3.55)
3.5 THE WISHART DISTRIBUTION AND SOME PROPERTIES In many cases, observations are collected on k independent groups of sampling units. The responses are described by a multinormal random variable with mean vector µh being the hth group and a covariance matrix common to all groups. So the observation matrix for the hth group is: x1Th x11h T x x21h x h = 2 h = T x nh h x nh 1h
x12 h
x22 h
x nh 2 h
x1 ph x2 ph x nh ph
(3.56)
The maximum likelihood estimation (MLE) of µh is the sample mean vector x h of the hth group, and the unbiased estimate of ∑ is given by: 1 s= n−k
k
nh
∑∑ h =1 i =1
1 ( x ih − x h )( x ih − x h ) = n−k T
k
∑A
h
h =1
where k is the total number of groups, and nh
Ah =
∑ (x
ih
− x h )( x ih − x h )T
i =1
is the sums of squares and cross products within the hth group and k
n=
∑n . h
h =1
(3.57)
2151_book.fm Page 52 Friday, October 20, 2006 5:04 PM
52
Stream of Variation Modeling and Analysis for MMPs
Remark 3.17: From the definition of A and Ah, they can be expressed as the sums of products of n – 1 and n – k independent p-variate random vectors with the common distribution N p (0,∑ ∑). In general, any symmetric positive definite matrix A ∈ R p × p of quadratic and bilinear forms which can be transformed to the sum: n
∑YY , T i
i
(3.58)
i=1
with p-variate vectors Yi independently distributed according to the distribution N p (0, ∑) is said to have the Wishart distribution [13]. The density function of a positive definite A with d.o.f. = n is given by: ( n − p−1)/ 2
A
Wn ( A | ∑) = 2
/2 np
π
(
exp − 12 tr ( A ∑ −1 )
p ( p−1)/ 4
∑
n / 2
)
p
∑ Γ(n + 1 − i)
(3.59)
i =1
where Γ(⋅) is the gamma function. If A is not positive definite, then Wn ( A | ∑) = 0. In Equation 3.59, n = n − 1 for a single-sample matrix, n = n − k for the k sample within-groups matrix. The d.o.f. n and ∑ specifies the form of the density function. Remark 3.18: If p = 1 and ∑ =1, then the Wishart density reduces to χ2n . Wishart-distributed matrices have many similar properties of χ2 variates. Some properties are summarized as follows: Property 1: If A1, A 2 , , A k are independent Wishart-distributed matrices with the common parameter matrix ∑, i.e., A i ~ Wni ( Ai | ∑) for i =1, 2, ..., k, then k
∑ i =1
A i ~ Wn
k
∑ i =1
Ai | ∑ , where n =
k
∑n . i
i =1
Property 2: If A ~ Wn ( A | ∑), then CAC T ~ Wn (CACT C ∑ CT ), where C is a constant matrix with proper orders. Property 3: The sample mean vector x and the matrix of sums of squares and cross products n
A=
∑ (x − x)(x − x) i
T
i
i =1
computed from the same sample are independently distributed.
2151_book.fm Page 53 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
53
3.6 EXERCISES 1. A transportation company gives the list of ticket prices in dollars (x1) for different travel distances in miles (x2) as follows: x1
2
6
6
8
8
10
12
15
17
20
x2
4
7
10
15
20
25
35
45
55
70
a. Construct a scatter plot of the data. b. Calculate the mean of the ticket price x1 and travel distances x2 . c. By looking at the scatter plot, what will the sign of the covariance s12 be? d. Calculate the variance s11 and s22. e. Calculate the covariance s12 and the correlation coefficient r12. Explain the physical meaning of these values. 2. Suppose two variables x1 and x2 are measured and shown to be the following: x1 x2
2 4
5 6
3 5
9 13
6 7
1 3
8 11
4 5
3 4
7 9
a. Construct a scatter plot. x 2T , where b. Express x as xT = x 1T x1 j x x j = 2 j , j = 1, 2. x10 j c. Determine d j = x j − x j 1 n for j = 1, 2, where n = 10 in this case. dj is called the associated residual vector. d. Find the lengths of each residual vector dj for j = 1, 2. e. Find the angle between two residual vectors d1 and d1. f. Verify that cos(θ12) = r12. 3. The measurements of two quality characteristics are as follows: Time
1
2
3
4
5
6
7
8
9
10
x1 x2
4.5 3
2 1.5
2.5 2
4 3
3.5 2.5
4.5 4
3 2
4 3.5
5 4
3 2.5
a. Find the correlation coefficient r between x1 and x2. b. Find the covariance matrix S for x1 and x2, determine the eigenvalues λ1 and λ2, and the eigenvectors {u1, u2} of S.
2151_book.fm Page 54 Friday, October 20, 2006 5:04 PM
54
Stream of Variation Modeling and Analysis for MMPs
c. Use the preceding eigenvectors to form the transformation matrix Q. Then the variables x1 and x2 can be transformed into independent variables z1, and z2, by using z1 T x1 = Q . z 2 x2 Calculate the corresponding values of z1 for all time steps. 4. Suppose 3 X = 9 3
5 7 6
4 2 , aT = [3 –1 2], bT = [2 4 –2], and cT = [–3 –1 1]. 3
There are p = 3 variables X1, X2, and X3. Each variable has n = 3 observations. a. Find the sample means of aT X, bT X, and cT X. b. Find the variances of aT X, bT X, and cT X. c. Calculate the covariances of aT X and bT X, bT X and cT X, and aT X and cT X. 5. Suppose the random vector X T = X1, X2 , X3 , X 4 has µ Tx = [5,1,4,6] and covariance matrix 1 2 ∑X = 3 −4
2 3 0 2
−4 2 0 8
3 0 7 0
X is partitioned as X1 X1 X = X2 = X3 X 2 X4 Suppose 2 A = and B = 1 1
2
−2
2151_book.fm Page 55 Friday, October 20, 2006 5:04 PM
Basics of Multivariate Statistical Analysis
55
Determine the following: a. E(X1) and E(X2). b. E(AX1) and E(BX2). c. cov(X1) and cov(X2). d. cov(AX1) and cov(BX2). e. cov(X2, X1). f. cov(BX2, AX1). 6. Suppose x1 and x2 are two random measurements from a process and each follows a Gaussian distribution x1 ~ N(2,1) and x2 ~ N(5,4). Suppose also that x1 and x2 are not independent and have covariance cov(x1,x2) = 1.5. Calculate: a. The mean vector and covariance matrix of the 2-D random vector x ≡ [x1 x2]T. b. The joint probability density function of x and marginal probability density functions of x1 and x2, respectively. c. Draw the equidensity contour of x (i.e., the curves over which pdf(x) assumes a constant value) and pdfs of x1 and x2 on the same figure. d. If cov(x1,x2) = 0, repeat (a) to (c).
References 1. Kshirsagar, A.M., Multivariate Analysis, Marcel Dekker, New York, 1972. 2. Mardia, K.V. et al., Multivariate Analysis, Academic Press, London, 1979. 3. Kleinbaum, D.G. et al., Applied Regression Analysis and Other Multivariable Methods, 2nd ed., PWS-Kent Publishing Company, Boston, MA, 1987. 4. Hair, J.F. et al., Multivariate Data Analysis, 3rd ed., Macmillan Publishing Company, New York, 1992. 5. Muirhead, R.J., Aspects of Multivariate Statistical Theory, John Wiley & Sons, New York, 1982. 6. Karson, M.J., Multivariate Statistical Methods, The Iowa State University Press, Ames, IA, 1982. 7. Morrison, D.F., Multivariate Statistical Methods, 2nd ed., McGraw-Hill, New York, 1976. 8. Takeuchi, K. et al., The Foundations of Multivariate Analysis, Wiley Eastern Limited, New Delhi, India, 1982. 9. Srivastava, M.S., Methods of Multivariate Statistics, John Wiley & Sons, New York, 2002. 10. Manly, B.F.J., Multivariate Statistical Methods: A Primer, Chapman and Hall, London, 1986. 11. The MathWorks, Getting Started with Matlab Version 7, The MathWorks, Inc., Natick, MA, 2006. 12. SAS, SAS 9.1 Documentation, http://support.sas.com/documentation/index.html, SAS Institute, Inc., Cary, NC, 2006. 13. Wishart, J., Biometrika, A., The Generalized Product Moment Distribution in Samples from a Normal Multivariate Population, 20(1928), pp. 32–52.
2151_book.fm Page 56 Friday, October 20, 2006 5:04 PM
2151_book.fm Page 57 Friday, October 20, 2006 5:04 PM
4
Statistical Inferences on Mean Vectors and Linear Models
4.1 STATISTICAL INFERENCES ON MEAN VECTORS In this section, we will discuss systematic approaches to making inferences about mean vectors based on the sampling information from a multivariate distribution. The necessary assumption is that the data should follow a multivariate normal distribution.
4.1.1 HOTELLING’S T 2 TEST Let Xi, i = 1, 2, …, n, be a sequence of independently identically distributed (i.i.d) p-dimensional random vectors with N p (µ , ∑), and xi, i = 1, 2, …, n, be their associated realizations. For a multivariate hypothesis testing problem, we are required to test whether a vector µ0 ∈R p is a true mean vector for the multivariate normal distribution. The squared distance in a multivariate case is defined as: −1
1 T 2 = (X − µ 0 )T S (X − µ 0 ) = n(X − µ 0 )T S−1 (X − µ 0 ), n
(4.1)
where µ0 = [µ10 µ20 … µp0]T ∈Rp and X and S are the sample mean vector and sample covariance matrix, respectively, which are defined as: X=
1 n
n
∑
Xi ∈ R p , S =
i =1
1 n −1
n
∑ (X − X)(X − X) i
i
T
∈ R p× p
i =1
1
Remark 4.1: In Equation 4.1, n S is the estimate of cov( X). Remark 4.2: The statistic T 2 defined in Equation 4.1 is called Hotelling’s T 2 in honor of Harold Hotelling, and is distributed as ( n − 1) p Fp , n − p , n− p when µ = µ 0 holds. Its sampling distribution is as follows: 57
2151_book.fm Page 58 Friday, October 20, 2006 5:04 PM
58
Stream of Variation Modeling and Analysis for MMPs
For a random sample xi, i = 1,2, …, n, which is drawn from N p (µ , ∑) population, no matter what the true µ and ∑ are, (n − 1) p Pr T 2 > Fp,n− p (α) = n − p (n − 1) p Pr n( x − µ )T S−1 ( x − µ ) > Fp,n− p (α) = α n− p
(4.2)
where Fp,n − p (α) is the upper (100α)th percentile of the F-distribution with degrees of freedom (d.o.f.) p and n – p. Remark 4.3 (Property of the T 2 statistic): It is easy to verify that the T 2 statistic is invariant for different scales of observations, i.e., for the nonsingular transformation Y = CX + d T 2 is identical for both X and Y, where C and d are a constant matrix and a vector, respectively, with proper dimensions. For the multivariate hypothesis testing problem, we have the following result. Theorem 4.1: For the multivariate hypothesis testing problem with estimated covariance matrix S: H 0 : µ = µ 0 , H1: µ ≠ µ 0
(4.3)
H0 is rejected if T 2 = n( x − µ 0 )T S−1 ( x − µ 0 ) >
(n − 1) p Fp,n− p (α) n− p
(4.4)
Remark 4.4: The preceding Hotelling’s T 2 test is based on the assumption that the covariance matrix ∑ is unknown. But if ∑ is known, the statistic becomes: T 2 = n(X − µ 0 )T ∑ −1 (X − µ 0 ),
(4.5)
where T 2 follows chi-square distribution with d.o.f. p under H0. This is also true for S ≈ ∑, if there are a large number of samples. Remark 4.5: As p univariate tests do not account for the dependency between these variables in multivariate measurement, a multivariate hypothesis should not be tested using p separate univariate tests, because it can lead to wrong conclusions.
2151_book.fm Page 59 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
59
Remark 4.6: It can be proved [1] that: n
T2 =
ˆ (n − 1) ∑ 0 ˆ ∑
∑ (X − µ )(X − µ ) 0
i
− (n − 1) = (n − 1)
i
i =1
n
∑ (X − x)(X − x) i
T
0
− (n − 1).
(4.6)
T
i
i =1
Equation 4.6 facilitates the calculation of T 2 by avoiding the calculation of S–1.
4.1.2 CONFIDENCE REGIONS AND SIMULTANEOUS CONFIDENCE INTERVALS 4.1.2.1 Confidence Regions Definition 4.1 (Confidence region): Let θ ∈Θ, where Θ is the parameter space. Given the data matrix x = [x1 x2 xn]T and the significance level α, before the sample is selected, the 100 (1 − α )% confidence region CR(x) is defined by 1− α = Pr(θ ∈CR(x)) = Pr(CR(x) will contain the trrue θ)
(4.7)
which is calculated under the unknown true parameter θ. For example, before the sample is selected, from Equation 4.2 and Equation 4.7, we have
{
}
1 − α = Pr µ ∈CR(x)
(n − 1) p = Pr n( x − µ )T S−1 ( x − µ ) ≤ Fp,n− p (α) − n p
(4.8)
regardless of the values of unknown µ and ∑. Therefore, for the p-dimensional normal cases, CR(x) can be determined by the following theorem. Theorem 4.2 (Confidence region): Given sample data xi, i = 1, 2, …, n, a 100(1 – α)% confidence region for the mean of a p-dimensional normal distribution is the space within the ellipsoid centered at x , i.e., CR(x): n( x − µ )T S−1 ( x − µ ) ≤
(n − 1) p Fp,n− p (α) = Tα2 n− p
where 1 x= n
n
∑ i =1
1 xi ∈ R , S = n −1 p
n
∑ (x − x)(x − x) i
i =1
i
T
∈ R p× p .
(4.9)
2151_book.fm Page 60 Friday, October 20, 2006 5:04 PM
60
Stream of Variation Modeling and Analysis for MMPs
~ Remark 4.7: To determine whether µ ∈CR(x), we need to compare the values of T −1 n( x − µ ) S ( x − µ ) and
Tα2 =
( n − 1) p Fp,n − p (α) . n− p
If n( x − µ )T S−1 ( x − µ ) ≤ Tα2 , then µ ∈CR(x); otherwise, µ ∉CR(x). Remark 4.8: The application of the confidence region CR(x) defined in Equation 4.9 is equivalent to testing H 0 : µ = µ 0 , H1 : µ ≠ µ 0 because CR(x) contains all possible µ 0 vectors for which the T 2-test fails to reject H0 at significance level α. Remark 4.9: The joint confidence region for µ cannot be graphed when the dimension p ≥ 4. It can be determined by the directions and lengths of the axes of the ellipsoid defined in Equation 4.9 under the scaling factor Tα n
λ i , i = 1, 2, …, p,
along the eigenvectors ei , where λ i and ei are eigenvalues and associated eigenvectors of S. The procedure for determining the ellipsoid involves the following steps: 1. Determine x , the center of ellipsoid; 2. Determine the axes of ellipsoid:
± λi
p(n − 1) Fp,n− p (α ) ei , i = 1, 2, …, p n(n − p)
(4.10)
where λi’s and ei’s are calculated by solving S ei = λ i ei From Equation 4.10, the half-length of axes are
λi
p ( n − 1) Fp,n − p (α ) for i = 1, 2, …, p. n(n − p)
(4.11)
2151_book.fm Page 61 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
61
So the elongation of the confidence ellipsoid is given by the ratio of the lengths of the axes, i.e.,
2 λi 2 λj
p ( n − 1) Fp,n − p (α ) n(n − p) p ( n − 1) Fp,n − p (α) n(n − p)
=
λi λj
, i≠ j
(4.12)
4.1.2.2 Simultaneous Confidence Intervals Based on the matrix theory presented in Chapter 2, it can be shown that
max a≠0
[a T ( x − µ )]2 = ( x − µ )T S−1 ( x − µ ) a T Sa
(4.13)
Thus, Equation 4.8 is equivalent to:
{
{
}
1 − α = Pr µ ∈CR(x) = Pr n( x − µ )T S−1( x − µ ) ≤ Tα2
}
[a T (x − µ )]2 = Pr n ≤ Ta2 , ∀a ∈ R p , a ≠ 0 T a Sa
(4.14)
where Tα2 =
( n − 1) p Fp,n − p (α) . n− p
Hence, Equation 4.14 becomes a T x − Tα
a T Sa n
≤ a T µ ≤ a T x + Tα
a T Sa n
, ∀a ∈ R p and a ≠ 0
(4.15)
This region consists of all plausible values of µ at the significance level α. In summary, we have the following theorem: Theorem 4.3 (Simultaneous intervals, or T 2-intervals): Suppose xi, i = 1, 2, …, n, is a random sample drawn from N p (µ , ∑) population with ∑ positive definite. Then simultaneously, for all a, the interval: a T x − Tα
a T Sa n
, a T x + Tα
a T Sa n
(4.16)
2151_book.fm Page 62 Friday, October 20, 2006 5:04 PM
62
Stream of Variation Modeling and Analysis for MMPs
will contain aT µ with probability 1 – α, where Tα2 =
( n − 1) p Fp,n − p (α). n− p
The confidence interval in Equation 4.16 is called Roy’s confidence interval. This theorem states that the confidence coefficient 1–α remains unchanged for any a ∈R p. Some special cases are discussed in the following remarks: Remark 4.10: Construct 100(1–α)% simultaneous confidence intervals for µi , i = 1, 2, …, p. If we select aT in the following forms: 1
0
0 , 0
1
0 , …, 0
0
1 .
Then the T 2-intervals become: xi − Tα
Sii n
, xi + Tα
Sii , for i = 1, 2, ..., p n
(4.17)
All intervals hold simultaneously with confidence coefficient 1–α. Remark 4.11: Construct 100(1–α)% confidence intervals for differences µ i − µ j , i ≠ j, i, j = 1, 2, …, p. If we select aT = [0 0 ai 0 aj 0 0], where ai = 1 and aj = –1, then T a Sa = sii − 2 sij + s jj . So the T 2-intervals become: ( xi − x j ) − Tα
sii − 2sij + s jj n
, ( xi − x j ) + Tα
sii − 2sij + s jj . n
(4.18)
It holds with confidence 1–α. Remark 4.12: Construct 100(1–α)% confidence ellipses for pairs of means. (µ i , µ j ) is within the sample mean-centered ellipses
n xi − µ i
sii x j − µ j sij
sij s jj
−1
xi − µ i (n − 1) p Fp,n− p (α) = Tα2 (4.19) ≤ n− p x j − µ j
It holds with confidence 1–α. Remark 4.13: If we want to obtain 100(1–α)% simultaneous confidence intervals for aiTµ, i = 1, 2, …, k, and if k is small, then Roy’s confidence intervals defined by
2151_book.fm Page 63 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
63
Equation 4.16 will be too wide. But the Bonferroni method will give narrower and more precise intervals if k (the number of linear combinations of interest) is small. More discussions on the Bonferroni confidence interval can be found in Reference 1.
4.1.3 INFERENCE ON A POPULATION MEAN VECTOR WITH LARGE SAMPLE SIZE So far, the hypothesis testing and confidence regions for µ have been constructed based on the assumption of a normal population. But when the sample size n is large, this normality assumption can be ignored even though there is a serious departure from normality. All large-sample inferences on µ are based on a chi-square distribution. Because n(X − µ )T S−1 (X − µ ) is approximately χ2p with d.o.f. p, when the sample size n is large enough,
{
}
Pr n( x − µ)Τ S−1 ( x − µ ) ≤ χ 2p (α ) ≈ 1 − α ,
(4.20)
where χ2p (α ) is the upper (100α)th percentile of the χ2p distribution. From Equation 4.20, we can obtain the hypothesis test and simultaneous confidence regions in the case of the large sample size. Theorem 4.4: Suppose xi, i = 1, 2, …, n, is a random sample drawn from a population with mean vector µ and positive definite covariance matrix ∑, and assume n − p is large. Then we have the following: 1. Hypothesis testing: Hypothesis H 0 : µ = µ 0 H1 : µ ≠ µ 0 will reject H 0 at a level of significance approximately α, if n( x − µ 0 )T S−1 ( x − µ 0 ) > χ 2p (α).
(4.21)
where χ2p (α ) is the upper (100α)th percentile of the χ2p distribution. 2. Simultaneous confidence regions: For any a ∈R p, the confidence intervals a T Sa T a T Sa a T x − χ 2p (α )· , a x + χ 2p (α )· n n will contain aT µ with probability approximately 1–α.
(4.22)
2151_book.fm Page 64 Friday, October 20, 2006 5:04 PM
64
Stream of Variation Modeling and Analysis for MMPs
Remark 4.14: As with the remarks in Theorem 4.3, the 100(1–α)% simultaneous confidence intervals for µi are: s s xi − χ2p (α ) ⋅ ii , xi + χ2p (α ) ⋅ ii , for i = 1, 2, ..., p n n
(4.23)
Remark 4.15: 100(1–α)% confidence ellipses for pairs of means (µ i , µ j ) can be constructed by: −1 sii sij xi − µ i 2 ≤ χ p (α ) . (µ i , µ j ) : n xi − µ i x j − µ j s s x j − µ j ij jj
(4.24)
4.1.4 MULTIVARIATE QUALITY CONTROL CHARTS Quality is of great importance in any industry. To improve the quality and performance of products and services, the process should be stable and no special causes leading to variation. A quality control chart can be effectively employed to continuously monitor the process and help identify special causes of variation. A typical control chart is composed of data (e.g., individual observations, or sample means) plotted in time order and control limits that indicate the quantity of variation due to common causes. A univariate control chart is used to monitor the individual characteristics in a process. But when more than one important feature occurs, it cannot monitor such processes due to the correlation between these features. Hence, it is necessary to develop multivariate control charts to monitor process stability. The multivariate control chart has two main tasks. One is to monitor the stability of a process with multivariate observations; the other is to set a control region for future observations. T 2 control charts will be introduced in this section. 4.1.4.1 Control Charts for One Multivariate Sample Two useful control charts are described in the following text: 1. T 2 control chart to monitor stability Suppose xi, i = 1, 2, …, n, is a random sample drawn from a normal µ,∑). After simple calculation, population Np (µ
xi − x = xi −
1 n
n
∑ i =1
1 1 x i = 1 − xi − n n
n
∑x. j
j =1, j ≠ i
2151_book.fm Page 65 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
65
It is easy to verify that 2
2
1 (n − 1) 1 E (x i − x ) = 0 and cov(x i − x ) = 1 − ∑ + ⋅ (n − 1) ∑ = ∑. n n n Thus, x i − x is also a multivariate normal distribution with zero mean vector, but not independent of the sample covariance matrix S. A chisquare distribution can be used to set the control limits. A T 2 control chart can be obtained through the following three steps: Step 1: Construct control limits Set up the 100 (1 − α )% control limit: UCL = χ 2p (α ) , LCL = 0
(4.25)
where p is the total number of monitored features. Step 2: Determine the points to be plotted Calculate and plot the T 2 statistics for the ith point: Ti 2 = (x i − x )T S−1 (x i − x ).
(4.26)
Step 3: Identify special causes If some points are out of control limits, we need to identify which variables are responsible for these points. This can be done by the Bonferroni intervals. For example, the jth variable of the ith sample is out of control if
(
xij ∉ x j − tn−1 (α / 2 p) s jj , x j + tn−1 (α / 2 p) s jj
)
(4.27)
2. T 2 control chart for future individual measurements If a T 2 control chart indicates that the process is stable, we can employ the collected data to construct another control chart, that can be used to predict future measurements. For the T 2 control chart, the control limits are set: (n − 1) p UCL = n − p Fp,n− p (α) , LCL = 0
(4.28)
2151_book.fm Page 66 Friday, October 20, 2006 5:04 PM
66
Stream of Variation Modeling and Analysis for MMPs
and the following T 2 calculated from the new measurement x is then plotted: T2 =
n ( x − x )T S −1 ( x − x ) n +1
(4.29)
where x=
n
∑ x ∈R
1 n
p
i
i =1
and S=
1 n −1
n
∑ (x − x)(x − x) i
i
T
∈ R p× p
i =1
are the sample mean vector and covariance matrix, respectively. If the T 2 from the future measurement x exceeds the upper control limit, we can conclude that it is an out-of-control signal and further action may be taken to identify the special causes. 4.1.4.2 Control Charts Based on Subgroup Means Two useful control charts are described in the following text: 1. Control ellipse and T 2 chart for monitoring stability A multivariate control chart can also be constructed based on subgroup means. The basic assumption is that each multivariate sample is drawn from N p (µ , ∑). The subgroup size is m > 1. Suppose the subgroup mean vectors and covariance matrices are xi , Si , i = 1, 2, …, n, where n is the sample size and xi =
1 m
m
∑x
ij
.
j =1
Because the population is a multivariate normal distribution, xi and S i are independent of each other. The grand mean vector and pooled sample covariance is calculated by x=
1 n
n
∑x
i
i =1
and Sp =
1 n
n
∑S . i
i =1
2151_book.fm Page 67 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
67
Following procedures similar to those in Subsection 4.1.4.1, it is easy to verify that E ( xi − x ) = 0 and 2
n −1 n −1 1 ∑, cov( xi − x ) = 1 − cov( xi ) + 2 cov( xi ) = mn n n where m > 1 is the subgroup size and n is the sample size. Remark 4.16: The pooled S p is an estimate of ∑ . Remark 4.17: n (m − 1)S p is independent of each xi , and thus the grand mean x. Remark 4.18: n (m − 1)S p is distributed as a Wishart random matrix with d.o.f. n(m – 1). As ∑ can be estimated by each subgroup, when all the estimators are pooled together, a single estimator is obtained and thus has a large number of degrees of freedom. Consequently, T2 =
mn n(m − 1) p ( xi − x )T S−p1 ( xi − x ) ~ Fp,mn− n− p+1 , n −1 mn − n − p + 1
(4.30)
where Fp,mn − n − p +1 is an F-distribution with d.o.f p and mn − n − p + 1. For different dimensions of p, we can construct a T 2 control chart with control limits as: (n − 1)(m − 1) p UCL = mn − n − p + 1 Fp,mn− n− p+1 (α) . LCL = 0
(4.31)
For a large sample size n, (n − 1)(m − 1) p Fp,mn− n− p+1 (α) ≈ χ 2p (α ). mn − n − p + 1 The plotted points are: Ti 2 = m( xi − x )T S−p1 ( xi − x ), i = 1, 2, ..., n
(4.32)
Any Ti 2 exceeding the UCL may indicate a potential out-of-control signal, and appropriate action should be taken to identify the responsible special causes.
2151_book.fm Page 68 Friday, October 20, 2006 5:04 PM
68
Stream of Variation Modeling and Analysis for MMPs
2. T 2 Control chart for future subgroup measurements Once we have determined that the process is stable based on the T 2 control chart, we can use the measurements drawn from this stable process to set up the control limits for future measurement means. n +1 Suppose X is the future subgroup mean. Because X − X ~ N p (0, ∑), we mn have: mn n (m − 1) p (X − X)T S −p1 (X − X) ~ Fp,mn − n − p +1 n +1 mn − n − p + 1
(4.33)
Based on Equation 4.33, we can obtain the following T 2 control chart for future data monitoring: For 100 (1 − α )% prediction T 2 control chart, the control limits are: (n + 1)(m − 1) p UCL = mn − n − p + 1 Fp,mn− n− p+1 (α) . LCL = 0
(4.34)
For a large sample size m, (n + 1)(m − 1) p Fp,mn− n− p+1 (α) ≈ χ 2p (α ) . mn − n − p + 1 The plotted points in time order are the T 2 statistic defined by T 2 = m( x − x )T S−1 p ( x − x ). Any point exceeding the UCL may indicate a potential out-of-control signal, and appropriate corrective action should be taken to eliminate the special causes. Remark 4.19: So far, we have discussed the hypothesis testing of a multivariate one-sample mean vector from a single multivariate population. In some situations, we are required to compare two or more populations. Interested readers can find approaches to making inferences on multivariate two-sample testing problems in Reference 1.
4.2 MULTIPLE LINEAR REGRESSION Regression analysis is a collection of techniques to help assess how much variation of certain variables can be explained as being due to the effects of another set of variables. Similar to a simple linear regression analysis, a multiple regression analysis is a statistical technique that can be used to analyze the relationship between dependent (or response) variables and independent (or controlled) variables. The
2151_book.fm Page 69 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
69
objective of the multiple regression analysis is to use the independent variables whose values are known to predict the dependent values. The result is a linear combination of the independent variables that “best” predicts the dependent variables. The variables are weighted by the weights denoting their relative contribution to the overall prediction, so that the regression model has the maximum prediction power.
4.2.1 MODEL DESCRIPTION The accuracy of prediction of y based on a single variable x is determined by the correlation coefficient ρxy . If ρ xy is low, it is necessary to employ multiple variables as explanatory variables to increase the accuracy of prediction. So multiple regression analysis is a method to predict the value of a dependent variable y based on a set of independent variables, x1, x2, …, xp. If y is centered, i.e., E ( y ) = 0 , then a multiple linear regression model can be described by: y = a1 x1 + a2 x2 + a p x p + ξ
(4.35)
where y is called dependent variable (or response variable, or criterion); p variables xi , i = 1, 2, …, p, are called independent variables (or controlled, or explanatory variables), having values that are considered known or fixed, not random; ai, i = 1, 2, …, p, are called regression parameters whose values are unknown. Some basic assumptions are made regarding the error term ξ: (A1) E(ξ) = 0; (A2) var(ξ) = σ 2 So from Equation 4.35 and the assumptions (A1) and (A2), we obtain: E ( y ) = a1 x1 + a2 x2 + a p x p + E (ξ) = a1 x1 + a2 x2 + a p x p ; var( y ) = var(ξ) = σ 2 Suppose n observations are made, yi = a1 xi1 + a2 xi 2 + a p xip + ξ i , i = 1, 2, …, n. We assume that these observations are uncorrelated, i.e., (A3) cov( yi , y j ) = cov(ξ i , ξ j ) = 0 for i ≠ j .
(4.36)
2151_book.fm Page 70 Friday, October 20, 2006 5:04 PM
70
Stream of Variation Modeling and Analysis for MMPs
To simplify the notation, we rewrite Equation 4.36 in the equivalent matrix form: Y = Xa + ξ,
(4.36)
where y1 x11 y2 x 21 Y = , X = X1 X 2 X p = x n1 y n
x12
x 22
xn 2
x1 p x2 p and Xi ∈ R n x np
a1 ξ1 a2 ξ a = , ξ = 2 . ξ n a p Here, matrix X is usually referred to as a design matrix with full column rank. Thus, assumptions (A1)~(A3) can be rewritten in a matrix form known as Gauss–Markov conditions (GM conditions): (A1′) E(ξξ) = 0; (A2′) cov(ξξ) = σ2I. So Equation 4.36′ together with (A1′) and (A2′) is called a multivariate linear regression model. The unknown parameters are regression parameters vector a and variance σ2. Efforts are made to estimate and evaluate these parameters.
4.2.2 LEAST-SQUARES ESTIMATES We need to find the linear composite vector of p
ˆ = Xa = Y
∑a X i
i
i =1
ˆ is an approximation of vector Y, i.e., find as close to vector Y as possible where Y p a ∈R to minimize: ˆ = Y − Xa , ξ = Y−Y 2
2
2
(4.37)
2151_book.fm Page 71 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
71
or minimize ˆ )T ( Y − Y ˆ ) = ( Y − Xa )T ( Y − Xa ). ξT ξ = (Y − Y
(4.38)
Definition 4.2 (Least-squares estimate [LSE], residual vector, and residual sum of squares [RSS]): The coefficient vector a selected by minimizing Equation 4.38 is called the least-squares estimate of the regression parameters a, denoted by aˆ , i.e., aˆ = arg min( Y − Xa )T ( Y − Xa ) = arg min Y − Xa a ∈R
a ∈R
p
p
2 2
(4.39)
The difference between Y and its predictors Xaˆ is called residual vector, denoted by ξˆ . So, ˆ = Y − Xaˆ , ξˆ = Y − Y and ξˆ T ξˆ = ( Y − Xaˆ )T ( Y − Xaˆ )
(4.40)
are called the residual sum of squares, or RSS. The LSE of a can be obtained by the following theorem: Theorem 4.5 (LSE): If X has full rank, i.e., rank(X) = p ≤ n, then the LSE of a in Equation 4.39 and the residual vector ξˆ are given by: aˆ = (XT X)−1 XT Y ˆ = I − X(XT X)−1 XT Y ξˆ = Y − Y n
(4.41)
ˆ = a and cov(a) ˆ = σ2 (XT X)–1. Further, E(a) Remark 4.20 (BLUE: the best linear unbiased estimator): From Equation 4.41, aˆ is a linear function of Y. It is also an unbiased estimator under GM conditions. Moreover, the Gauss–Markov theorem (2, Theorem 2.6, p.41) ensures that aˆ is the best estimate because it has the smallest covariance among all linear unbiased estimators of a. Remark 4.21 (Collinearity): If X is not full column rank, the inverse in Equation 4.41 is replaced by a generalized inverse. In this case, some combinations, e.g., Xa~ , must be 0, and the columns are said to be collinear. In most regression analyses, Xa~ is not likely to be exactly 0. But a very small magnitude of |Xa~ | will lead to numerically unstable (XT X)–1. Some diagonal elements of (XT X)–1 will be large and, consequently, the estimated variance for the corresponding âi’s are large. This will make it difficult to detect the “significant” regression coefficient ai .
2151_book.fm Page 72 Friday, October 20, 2006 5:04 PM
72
Stream of Variation Modeling and Analysis for MMPs
The following two methods can be used to solve the problems caused by collinearity: 1. Deleting one predictor variable from a strongly correlated pair 2. Representing the original predictor variables using their principal components and then regressing Y on these new predictor variables Remark 4.22 (Hat matrix): Define matrix: H = X(XT X)−1 XT ,
(4.42)
ˆ = HY and ξˆ = Y – Y ˆ = (I – where H is usually called the “hat” matrix, then: Y n H)Y = (In – H)ξξ. Regarding the linear function of a, suppose p
cT a =
∑c a , i i
i =1
the following Gauss theorem [1, p.389] gives its “best” estimator. Theorem 4.6 (Gauss least-squares theorem): For the multiple regression model: Y = Xa + ξ , where ξ ~ (0, σ 2I) and X has full column rank p. Then for any vector c ∈R p , the estimator with the smallest possible variance among all linear estimators given by c T aˆ is unbiased for c T a. ˆ = Theorem 4.7: The residual vector ξˆ defined in Equation 4.41, i.e., ξˆ = Y − Y [In − H]Y, has the following property:
( )
E ξˆ T ξˆ = (n − p)σ 2 .
(4.43)
Remark 4.23: From Theorem 4.7, an unbiased estimator of σ2 is given by: s2 =
1 Y T ( I − H) Y n− p
=
1 ˆ )T ( Y − Y ˆ) (Y − Y n− p
=
1 1 ( Y − Xaˆ )T ( Y − Xaˆ ) = ( Y T Y − Y T Xaˆ ). n− p n− p
(4.44)
2151_book.fm Page 73 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
73
Remark 4.24 (Decomposition of sum of squares): The total sum of squares (TSS) can be decomposed into the regression sum of squares (RegSS) and the RSS, i.e., 2
2 2 ˆ + ξˆ . Y = Y 2
2
(4.45)
2
ˆ and variance of ξˆ as sY2, s 2 , If we denote the total variance of Y, variance of Y, Yx 2 and sY⊥ , respectively, then, x
sY2 = sY2 x + sY2 ⊥ . x
4.2.3 INFERENCES
ABOUT THE
(4.46)
REGRESSION MODEL
In this subsection, we give methods for testing the general linear hypothesis and obtaining the confidence intervals under the conditions in Theorem 4.8. It can be proved that the MLE for regression parameters is the same as the LSE under normality condition [1, pp.390–392], which is established by the following theorems: Theorem 4.8 (MLE for a and σ2): Given regression model Y = Xa + ξ, where a1 y1 ξ1 a2 y2 ξ2 Y = , a = , ξ == , a y ξ n n p design matrix X = [X1 X 2 X p ], Xi ∈ R n and X has full column rank, and the error vector follows multinormal distribution, ξ ~ N n (0, σ 2 I) . Then: 1. The MLE of a is the same as the LSE aˆ . Moreover, aˆ = (X T X)−1 X T Y ~ N p (a, σ 2 (X T X)−1 )
(4.47)
2. aˆ is distributed independently of the residuals ξˆ = Y − Xaˆ . Furthermore, ξˆ T ξˆ = n σˆ 2 ~ σ 2χ 2n− p where σˆ 2 =
1 ( Y − Xaˆ )T ( Y − Xaˆ ) is the MLE of σ2. n
(4.48)
2151_book.fm Page 74 Friday, October 20, 2006 5:04 PM
74
Stream of Variation Modeling and Analysis for MMPs
To draw inferences about regression parameters, the confidence ellipsoid for a is constructed based on the following theorem in terms of the estimated covariance matrix s 2 (X T X)−1 . Theorem 4.9 (Confidence region and confidence interval for parameters a): According to the assumption in Theorem 4.8, the 100 (1 − α )% confidence region for a is given by: (a − aˆ )T (X T X)(a − aˆ ) ≤ ps 2 Fp,n − p (α)
(4.49)
where s2 =
1 ˆT ˆ ξ ξ. n− p
Simultaneous 100 (1 − α )% confidence intervals for ai , i = 1, 2, ..., p are given by: aˆi ± var (aˆi ) pFp,n− p (α).
(4.50)
Remark 4.25: If the regression model contains a constant term, e.g., Y = a0 + a1X1 + + ar Xr + ξ
(4.51)
then the 100 (1 − α )% confidence region for a and simultaneous 100 (1 − α )% confidence intervals for ai , i = 0, 1, 2, ..., r are defined as: (a − aˆ )T (X T X)(a − aˆ ) ≤ (r + 1)s 2 Fr +1,n −r −1 (α)
(4.52)
aˆi ± var (aˆi ) (r + 1) Fr +1,n−r −1 (α),
(4.53)
where s2 =
1 ξˆ T ξˆ . n − r −1
Remark 4.26: The confidence ellipsoid is centered at the MLE â, and its orientation and size are determined by the eigenvalues and eigenvectors pair (λ, e) of XT X, i.e., λe = (X T X)e . If λ ≈ 0, then the confidence ellipsoid will be very long in the direction of the corresponding eigenvector. Remark 4.27: If ignoring the “simultaneous” confidence properties defined in Equation 4.50 or Equation 4.53, the pFp,n − p (α) in Equation 4.50 or (r + 1)Fr+1,n–r–1(α) in
2151_book.fm Page 75 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
75
Equation 4.53 can be replaced by a one-at-a-time t value, tn–p(α/2) or tn–r–1(α/2). This can be used to search for important independent variables. Remark 4.28: If the confidence interval for ai contains 0, then the associated independent variable xi may not have an influence on the dependent variable. In other words, it might contribute nothing to the response. Hence, it can be dropped from the regression model.
4.2.4 MODEL CHECKING: NORMALITY CHECKING AND OUTLIER DETECTION Regression modeling requires the assumption of normality and no outliers in the ˆ = (I – H)Y = (I – H)ξ. If the model is valid, then ξˆ is a data. We have ξˆ = Y – Y normally distributed random vector with E(ξˆ ) = 0 and var(ξˆ ) = σ2 (I – H). Although the covariance matrix is not diagonal, the off-diagonal elements are usually small and the diagonal elements are nearly equal. Clearly if the diagonal elements of H vary greatly, then the variance of each ξˆ i will also accordingly differ to a large extent from one another. Using residual mean square s2 =
1 Y − Xaˆ n− p
2 2
as the estimate of σ2, we obtain the estimation of var(ξˆ i ) : var (ξˆ i ) = s 2 (1 − hii ), i = 1, 2, …, n,
(4.54)
and the studentized residuals: ξˆ − E (ξˆ i ) ξˆ *i = i = var (ξˆ i )
ξˆ i s 2 (1 − hii )
, i = 1, 2, …, n.
(4.55)
The studentized residuals are expected to be approximately independently distributed as N (0, 1) . By ranking the ξˆ *i , i = 1, 2, …, n, we get the order statistics ξˆ *(i ), i = 1, 2, …, n. Then normality can be checked visually or analytically. The commonly used graphical approaches include: 1. Plot ( yˆi , ξˆ i ): If there is a trend on the plot (e.g., incorrect calculations, or if a constant term is ignored in the regression model) or the variance is not constant (e.g., transformations or weighted least squares are required), then the model assumptions do not hold. 2. Plot (xji, ξˆ i) or (xji xki , ξˆ i ): If there is a systematic pattern occurring in the plot, then it suggests that more terms are needed in the regression model. 3. Plot (i , ξˆ i ): This plot can be used to check independence over time. A systematic pattern suggests the data are time dependent.
2151_book.fm Page 76 Friday, October 20, 2006 5:04 PM
76
Stream of Variation Modeling and Analysis for MMPs
4. Q-Q plot: The Q-Q plot can be used to examine severe departures from normality or the presence of abnormal observations. If the sample size n is large, the extreme points may depart from normality but will not affect inferences about the regression parameter a. Statistical approaches include the Shapiro–Wilk test or Kolmogorov’s test. Interested readers may refer to Reference 3 in Chapter 3. An outlier can be detected using the methods described in Reference 3 (p. 274): 1. Construct the test statistic: P = max ξˆ *i , 1≤ i ≤ n
(4.56)
where ξˆ *i is defined in Equation 4.55. 2. Make inferences: There is an outlier in the data if P > F1,n–1(α/n); furthermore, if i0 = arg max ξˆ *i , 1≤ i ≤ n
(4.57)
then the i0th observation is an outlier.
4.2.5 INFERENCES
FROM THE
ESTIMATED REGRESSION MODEL
After having checked the fitted regression model and evaluated it as proper, one can use it to make some predictions, given a particular predictor vector x0 = [x10 x20 xp0 ]T. Based on x0 and MLE â, we can estimate the regression function and forecast a new observation at x0: 1. Estimating the regression function x0Ta at x0: Suppose y0 is the value of the response when the predictor vector x0 is given. Then:
(
)
E y0 x 0 = x T0 a
(4.58)
Its LSE is x0T â. Theorem 4.10 (Estimating): For the multiple regression model given in Equation 4.36 or Equation 4.36′: Y = Xa + ξ , E (ξ ) = 0, cov(ξ ) = σ 2I
2151_book.fm Page 77 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
77
x0T â is the best linear unbiased estimator (BLUE) of E(y0 |x0) in terms of minimum variance. Its variance is given by: var (x T0 aˆ ) = σ 2x T0 (XT X)−1 x 0
(4.59)
Further, if ξ is normally distributed, then a 100(1–α)% confidence interval for E(y0 |x0) is given by:
(
α x T0 aˆ ± tn− p s 2 x T0 (XT X)−1 x 0 2
)
(4.60)
where tn-p(α/2) is the upper 100(α/2)th percentile of a t-distribution with d.o.f. n – p. 2. Predicting a new observation y0 at x0: Predicting a new observation is more uncertain than estimating the expected value of y0. For the regression model: y0 = x T0 a + ξ 0 , ξ 0 ~ N (0, σ 2 ) where y0 is the new response; x0Ta, the expected value of y0 at x0; and ξ0, the new error. Theorem 4.11 (Predicting): For the multiple regression model given in Equation 4.36 or Equation 4.36′: Y = Xa + ξ , E (ξ ) = 0, cov(ξ ) = σ 2I a new observation of y0 has the unbiased predictor x T0 aˆ . The variance of the forecast error y0 − x T0 aˆ is given by: var ( y0 − x T0 aˆ ) = σ 2 (1 + x T0 (XT X)−1 x 0 )
(4.61)
Further, if error ξ is normally distributed, then a 100(1–α)% prediction interval for y0 is given by:
(
α x T0 aˆ ± tn− p s 2 1 + x T0 (XT X)−1 x 0 2
)
(4.62)
where tn–p(α/2) is the upper 100(α/2)th percentile of a t-distribution with d.o.f. of n – p. The proofs for Theorem 4.10 and Theorem 4.11 can be found in [1, pp. 400~402].
2151_book.fm Page 78 Friday, October 20, 2006 5:04 PM
78
Stream of Variation Modeling and Analysis for MMPs
Remark 4.29: By comparing Equation 4.60 with Equation 4.62, we can find:
(
)
(
α α tn− p s 2 1 + x T0 (XT X)−1 x 0 > tn− p s 2 x T0 (XT X)−1 x 0 2 2
)
(4.63)
i.e., the half width of the prediction interval for y0 is longer than the half width of the confidence interval for estimating the value of the regression function E(y0 |x0) = x0Ta. This additional uncertainty in forecasting y0 comes from the presence of the new unknown error term ξ0.
4.2.6 MULTIVARIATE LINEAR REGRESSION 4.2.6.1 Multivariate Linear Regression Model Multiple linear regression is used to model the linear relationship between one response, Y, and a set of independent variables X1, X2, , Xp. But in some cases, there are many response variables, Y(1), Y(2), …, Y(q). We need to model the relationship between the q variables and independent variables. Each response follows its own multiple regression model: Y( j ) = Xa ( j ) + ξ ( j ) , j = 1, 2, …, q. If each response has n observations, we introduce the notation: x11 x 21 X= x n1 y11 y21 Y = yn1 a11 a21 A= a p1
y12 y22
yn 2
a12
a22
a p2
x12
x 22
xn 2
x1 p x2 p x np
, n× p
y11 y2 q = Y(1) Y( 2) Y( q ) , ynq n× q a1q a2 q a pq
= a (1) a ( 2) a ( q ) , p× q
(4.64)
2151_book.fm Page 79 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
ξ11 ξ 21 ξ= ξ n1
ξ12
ξ 22
ξn2
ξ1q ξ 2q ξ nq
n× q
79
ξ1T ξT 2 = ξ (1) ξ ( 2) ξ ( q ) = , T ξ n
Thus, the multivariate linear regression model is defined as: Y(i ) = Xa (i ) + ξ (i ) , i = 1, 2, …, q, or Y = XA + ξ , E (ξ (i ) ) = 0, cov(ξ (i ) , ξ ( j ) ) = σ ij I for i, j = 1, 2, …, q,
(4.65)
where Y ∈ R n ×q , X ∈ R n × p , A ∈ R p ×q , ξ ∈ R n ×q . Remark 4.30: The n observations on the kth sample have the covariance matrix ∑, but observations for different samples are uncorrelated. Remark 4.31: In the multivariate linear regression model, A and σij, i, j = 1, 2, …, q are unknown parameters. Remark 4.32: The error term ξ corresponding to different responses may be correlated. 4.2.6.2 Least-Squares Estimation (LSE) and Maximum Likelihood Estimation (MLE) of Parameters Based on the results of multiple regression analysis, the LSE of a(i) is given by minimizing Y(i) – Xa(i)22 as: aˆ (i ) = (XT X)−1 XT Y(i ) , i = 1, 2, …, q
(4.66)
Thus, the criterion to be minimized for multivariate regression analysis is given by:
(
ˆ ) = tr Y − XA ˆ f (A
) ( Y − XAˆ ) T
(4.67)
ˆ by differentiating f(A) ˆ w.r.t. each element of A: We can obtain the LSE A ˆ = (XT X)−1 XT Y A
(4.68)
2151_book.fm Page 80 Friday, October 20, 2006 5:04 PM
80
Stream of Variation Modeling and Analysis for MMPs
ˆ we can obtain the matrices of predicted values and residuals: So, using the LSE A, ˆ = X (XT X)−1 XT Y ˆ = XA Y
(4.69)
ˆ = [I − X (XT X)−1 XT ] Y. ξˆ = Y − Y
(4.70)
We know that multiple regression analysis possesses orthogonality properties among residuals, predicted values, and columns of X. In multivariate regression analysis, these properties also hold. Property 1: Columns of X and residual ξˆ are orthogonal, i.e., XT ξˆ = 0
(4.71)
It can be easily verified by XT ξˆ = XT [I − X(XT X)−1 XT ] Y = 0. ˆ and residual ξˆ are orthogonal, i.e., Property 2: Predicted value Y ˆ T ξˆ = 0 Y
(4.72)
ˆ T ξˆ = (X (XT X)–1 XT Y)T [I – X (XT X)–1 XT] Y = 0. This property holds because Y Similar to multiple regression analysis, the total sum of squares can also be decomposed into two components because: ˆ + (Y − X A ˆ) ˆ + ξˆ = X A Y=Y Using Property 2, we have: ˆ TY ˆ + ξˆ T ξˆ + 0 T + 0 = Y ˆ TY ˆ + ξˆ T ξˆ ˆ + ξˆ ) = Y ˆ + ξˆ )T ( Y YT Y = (Y
(4.73)
= Y T X(XT X)−1 XT Y + ( Y T Y − Y T X(XT X)−1 XT Y)
(4.74)
Thus, the TSS and cross products can be decomposed into two components: Predicted sum of squares (RegSS) and cross products, and residual sum of squares (ReSS) and cross product, i.e., ˆ TY ˆ + ξˆ T ξˆ YT Y = Y
(4.75)
ˆ T X XT A ˆ ˆ TY ˆ = YT Y − A ξˆ T ξˆ = Y T Y − Y
(4.76)
So,
2151_book.fm Page 81 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
81
The following results give the LSE ξˆ its properties, as in the multiple regression model. Theorem 4.12 (Mean vectors and covariance matrices): Let the multivariate multiple regression model in Equation 4.65 have full rank, i.e., rank(X) = p < n. Then: ˆ = [aˆ aˆ aˆ ] is an unbiased estimator of 1. The LSE A (1) ( 2) (q) A = [a (1) a ( 2) a ( q ) ], i.e., ˆ = A. E(A) 2. cov( aˆ (i ) , aˆ ( j ) ) = σij (XT X)–1, i, j = 1, 2, …, q. ˆ satisfies 3. The residual ξˆ = ξˆ (1) ξˆ ( 2) ξˆ ( q ) = Y – XA
( )
(
)
E ξˆ (i ) = 0, E ξˆ T(i ) ξˆ ( j ) = (n − p)σ ij
(4.77) (4.78)
(4.79)
Thus,
()
1 ˆT ˆ E ξˆ = 0, E ξ (i ) ξ ( j ) = ∑ . n− p ˆ and the LSE A ˆ are 4. The residual ξˆ = ξˆ (1) ξˆ ( 2) ξˆ ( q ) = Y – XA uncorrelated. The detailed proof can be found in Reference 1 (p. 417). Similar to multiple regression analysis, when the error ξ follows a normal distribution, we can obtain the MLE and its distribution. This result is stated in the following theorems: Theorem 4.13 (MLE): Assume that the multivariate regression model in Equation 4.65 has full rank, i.e., rank(X) = p, p + q ≤ n, and the error ξ is normally distributed. Then: 1. The MLE of A is: ˆ = (XT X)–1 XT Y A
(4.80)
ˆ is normally distributed with 2. A ˆ ) = A, and cov( aˆ (i ), aˆ ( j )) = σij (XT X)–1, i, j = 1, 2, …, q E(A
(4.81)
2151_book.fm Page 82 Friday, October 20, 2006 5:04 PM
82
Stream of Variation Modeling and Analysis for MMPs
ˆ of positive definite ∑, where ˆ is independent of the MLE ∑ 3. A ˆ = 1 ξˆ T ξˆ = 1 ( Y − X A ˆ )T ( Y − X A ˆ) ∑ n n ˆ ~ W (∑ ) 4. n ∑ n− p
(4.82) (4.83)
4.2.6.3 Inferences for Multivariate Regression Model under Normality Assumption Suppose we want to test the hypothesis that the responses do not depend on some independent variables. Without loss of generality, we can split the matrices in the following manner: A(1) A = , A(1) ∈ Rl × q , A( 2) ∈ R( p−l )× q A( 2) p× q X = X1 X 2 , X1 ∈ R n×l , X 2 ∈ R n×( p−l ) n× p So the general model can be rewritten as: A(1) E(Y) = XA = X1 X 2 = X1A(1) + X 2 A( 2) A( 2)
(4.84)
Under the null hypothesis H 0 : A (2 ) = 0, the model becomes Y= X1A(1) + ξ
(4.85)
Then the extra sum of squares and cross products is given by: ˆ ˆ )T ( Y − X A ˆ ˆ T ˆ ( Y − X1 A (1) 1 (1) ) − ( Y − X A) ( Y − X A) = n(∑1 − ∑ ) where ˆ = 1 (Y − X A ˆ T ˆ ˆ = (XT X )−1 XT Y and ∑ A (1) 1 1 1 1 1 (1) ) ( Y − X1 A(1) ). n
(4.86)
2151_book.fm Page 83 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
83
Thus, the likelihood ratio, Λ, can be expressed based on generalized variances: ˆ ) ˆ ,∑ L(A 1 (1) Λ= = = ˆ) ˆ ,∑ max L ( A, ∑) L ( A A ,∑ max L ( A(1) , ∑)
A(1) ,∑
ˆ ∑ ˆ ∑ 1
n/2
(4.87)
or equivalently Wilks’ lambda statistics Λ 2/n =
ˆ ∑ ˆ ∑ 1
(4.88)
Theorem 4.14 (Likelihood ratio test): Suppose the multivariate regression model in Equation 4.65 holds with X full rank, i.e., rank(X) = p, p + q ≤ n. Assume that the ˆ ~ W (∑) and is indeerror ξ follows normal distribution. Under H0: A(2) = 0, n∑ n–p ˆ pendent of n(∑1 − ∑), which follows Wp−l (∑). The likelihood ratio test of H 0 is equivalent to rejecting H 0 for large values of −2 ln( Λ) = − n ln
ˆ ˆ n∑ ∑ = − n ln ˆ ˆ + n(∑ ˆ −∑ ˆ) n∑ ∑ 1 1
(4.89)
When the sample size n is large, the distribution of modified statistics: ˆ ∑ 1 − n − p − (q − p + l + 2 ) ln ˆ 2 ∑ 1
(4.90)
is close to a χ2 distribution with d.o.f. q(p – l – 1). The theorem and its proof can be found on page 422 in Reference 1, where the possibility that X is not full rank is also discussed. Other than the likelihood ratio test, there are also other multivariate test statistics for testing H0: A(2) = 0, e.g., Wilks’ lambda, Pillai’s trace, Hotelling–Lawley trace, and Roy’s greatest root. These four multivariate test statistics are commonly used in computer-based packages. Interested readers may refer to Reference 1. 4.2.6.4 Predictions from Multivariate Regression As in multiple regression analysis, the multivariate regression model can also be used for predictions if the model has been fit and checked for any inadequacies. Suppose the multivariate regression model has the form Y = XA + ξ
2151_book.fm Page 84 Friday, October 20, 2006 5:04 PM
84
Stream of Variation Modeling and Analysis for MMPs
where ξ is normally distributed. If the model is adequate, two predictive problems can be solved using the fitted model. 1. Predicting the mean responses corresponding to fixed values x0: Based on the theorem (MLE) and the remarks therein, we have
(
ˆ T x ~ N AT x , x T (XT X)−1 x ∑ A 0 0 0 0 q
)
(4.91)
ˆ ~ W (∑ ) n∑ n− p
(4.92)
ˆ T x . So Clearly, the unknown value of the regression function at x0 is A 0 2 the T statistic can be defined as: A ˆ T x − AT x 0 0 T2 = x T0 (XT X)−1 x 0
T
−1 n ˆ T x − AT x A 0 0 ˆ ∑ n − p x T0 (XT X)−1 x 0
(4.93)
The 100(1-α)th confidence ellipsoid for AT x0 is given by the inequality:
(
ˆ Tx AT x 0 − A 0
)
T
n ˆ n − p ∑
−1
(A x T
0
)
ˆ Tx ≤ −A 0
q(n − p) x ( X X) x 0 Fq ,n− p+1− q (α) 1 n − p + − q T 0
T
(4.94)
−1
where Fq,n − p +1−q (α) is the upper (100α)% percentile of an F-distribution with d.o.f q, n − p + 1 − q. The 100(1-α)% simultaneous confidence intervals for E(Yi) = x T0 a (i ) are: q(n − p) n x T0 aˆ (i ) ± Fq ,n− p+1− q (α) x T0 (XT X)−1 x 0 σˆ ii , n − p + 1− q n− p
(4.95)
i = 1, 2, …, q ˆ. ˆ and σˆ is the ith diagonal element of ∑ where aˆ (i ) is the ith column of A ii 2. Forecasting new responses Y0 at x0: If the model is adequate, it can be used to forecast new responses Y0 = AT x0 + ξ0 at x0, where ξ0 is independent of ξ.
2151_book.fm Page 85 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
85
Because ˆ T x = AT x + ξ – A ˆ T x = (AT – A ˆ T) x + ξ Y0 – A 0 0 0 0 0 0 its distribution is: ˆ T x ~ N (0, ∑ + x T (XT X)–1 x ∑) Y0 – A 0 q 0 0
(4.96)
ˆ. which is independent of n ∑ Thus, the 100(1–α)% prediction region is within the ellipsoid centered at Y0: −1
ˆ ˆ T x )T n ∑ ˆT ( Y0 − A 0 n − p ( Y0 − A x 0 ) ≤ q(n − p) Fq ,n− p+1− q (α) (1 + x (X X) x 0 ) n − p + 1 − q T 0
T
(4.97)
−1
The 100(1–α)% simultaneous prediction intervals for the individual responses y0i are: q(n − p) n x T0 aˆ (i ) ± Fq ,n− p+1− q (α) (1 + x T0 (XT X)−1 x 0 ) σˆ ii , n − p + 1− q n − p 4.98) i = 1, 2, …, q ˆ. ˆ and σˆ is the ith diagonal element of ∑ where aˆ (i ) is the ith column of A ii Remark 4.33: By comparing the previous two 100(1–α)% simultaneous intervals, ˆ 0i, the prediction we can see that, because of the presence of the random error σ intervals for the actual values of the response variables are wider than the corresponding intervals for the expected values. 4.2.6.5 Selection Method of Independent Variables In the previous chapters, we have discussed that the multiple correlation coefficient increases with an increase in the number of independent variables involved in the regression model. If the number of independent variables exceeds n – 1, where n is the number of samples, the values of multiple correlation coefficient become irrelevant to the magnitude of the correlation between X1, X2, …, Xp. To deal with this problem, some methods are useful in determining a set of extra variables to be included in the regression function, and the multiple correlation
2151_book.fm Page 86 Friday, October 20, 2006 5:04 PM
86
Stream of Variation Modeling and Analysis for MMPs
coefficient is adjusted for d.o.f. Among these methods, the AIC criterion, which was first proposed by Akaike [4], and the PSS criterion ([5]) are commonly used. Detailed descriptions are available in Reference 6.
4.3 EXERCISES 1. Let X be a sequence of i.i.d 2-D random vectors with a multivariate normal distribution. Suppose that the observations x are: 3 5 7 x= 2 4 6
6 4 5 7 10 9
Given that µ 0 = [5 7]T : a. Conduct the hypothesis testing for H0: µ = µ0. Determine T 2 using the given data. b. From (a), what is the distribution of T 2 ? Also, determine all parameters of this distribution. c. Given α = .05 level and based on the given data, what will be the conclusion for the test in (a)? 2. The following data are the off-target measurements (in mm) of an automobile body in the panel assembly process. X1, X2, X3, and X4 are the associated deviations from their nominal positions in mm (based on sensors). Twenty-five parts are measured. Part
X1
X2
X3
X4
1 2 3 4 5 6 7 8 9 10 11 12
0.0388 –0.0322 –0.0195 0.0254 0.0054 0.0425 –0.0059 –0.0352 –0.0457 –0.0620 –0.0076 –0.0181
0.0050 0.0004 –0.0146 –0.0747 –0.0476 –0.0504 0.0047 0.0404 –0.0094 0.0531 –0.0063 0.0219
0.0073 0.0161 0.0581 0.0644 –0.0065 –0.0158 0.0156 –0.1069 –0.0309 0.0007 –0.0101 0.0274
0.4319 0.4479 0.3606 0.4115 0.3864 0.3508 0.4185 0.3803 0.4165 0.3587 0.3819 0.4491
2151_book.fm Page 87 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
Part
X1
X2
X3
X4
13 14 15 16 17 18 19 20 21 22 23 24 25
0.0142 0.0166 0.0151 0.0358 –0.0166 0.0298 –0.0040 –0.0175 0.0403 0.0274 –0.0154 0.0026 0.0128
0.0290 0.0805 –0.0351 0.0091 0.0375 0.0137 0.0458 0.0117 0.0355 –0.0487 –0.0581 –0.0097 –0.0092
0.0157 –0.0255 –0.0156 –0.0223 0.0181 0.0736 0.0412 0.0271 0.0196 0.0115 0.0120 0.0253 0.0272
0.3799 0.3346 0.4253 0.3699 0.3903 0.3818 0.3279 0.5460 0.3710 0.3898 0.5007 0.4102 0.3958
Given that the true mean vector is µ0 = [0 0]T, Use the data on X1 and X2 to do the following: a. Conduct the hypothesis testing for H0: µ = µ0. Determine T 2 using the given data. b. Calculate T 2 using Equation 4.6. Is it the same as the result from (a)? c. Find the distribution of T 2. d. Given α = .05 level and based on the given data, what will be the conclusion of the test in (a)? 3. Using the data on X1 and X2 in Problem 2, assume that α = 0.05 and µ0 = [0 0]T. The hypothesis testing is H 0 : µ = µ 0 . H1: µ ≠ µ 0
4. 5. 6. 7.
Construct a confidence ellipse. Is the result from the confidence ellipse the same as the conclusion in Problem 2d? Using the data in Problem 2 and given α = 0.05, construct a T 2 chart and a control ellipse for X3 and X4. From the data on X1 and X2 in Problem 2, construct a control ellipse for a future observation xT = (x1, x2). Comment on the result. Repeat Problem 5, but use the data on X3 and X4 in Problem 2. Suppose 1 X = 3 1 T
1 5 3
1 8 4
1 7 4
1 9 5
1 4 and Y T = 15 9 20 18 25 16 . 2
87
2151_book.fm Page 88 Friday, October 20, 2006 5:04 PM
88
Stream of Variation Modeling and Analysis for MMPs
The multiple linear regression model is Y = Xa + ξ, where ξ1 a0 ξ a = a1 and ξ = 2 . a2 ξ 6 a. Determine the least-squares estimates â. ˆ b. Determine the fitted values Y. ˆ c. Calculate the residuals ξ. d. Calculate the residual sum of squares, ξˆ T ξˆ . 8. The relationship between measurement deviation (y) and the pins deviations (u) in a single-station assembly process can be represented in a linear relationship as the following: y = Γ ⋅u + ε where Γ is the geometric relationship between y and u, and ε is the model error. Suppose y = δx1 u = δP1x
δz1
δx3
δP1z
δz3
δx5
δx9
T
δP2 z ,
,
σ 2ε = 1.7361 × 10 −5 , and 0.9972 −0.0093 1 Γ= 0 −0.0150 1.0072 −0.0246
−0.1157 0.6196 0 1 0.3846 0.2961 −0.0049
T
δz9 ,
0.1157 0.3804 0 0 . 0.6154 −0.2961 1.0049
Let N = 25, i.e., there are 25 parts observed.
2151_book.fm Page 89 Friday, October 20, 2006 5:04 PM
Statistical Inferences on Mean Vectors and Linear Models
89
The observation data for y is provided in the table below: Part
δx 1
δz 1
δx 3
δz 3
δx 5
δx 9
δz 9
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
0.4231 0.4337 0.3563 0.4174 0.3748 0.3472 0.4237 0.3879 0.4197 0.3532 0.3686 0.4454 0.3769 0.3320 0.4306 0.3921 0.3849 0.3738 0.3237 0.5418 0.3687 0.3925 0.4846 0.3888 0.3828
0.0073 0.0161 0.0581 0.0644 –0.0065 –0.0158 0.0156 –0.1069 –0.0309 0.0007 –0.0101 0.0274 0.0157 –0.0255 –0.0156 –0.0223 0.0181 0.0736 0.0412 0.0271 0.0196 0.0115 0.0120 0.0253 0.0272
0.4319 0.4479 0.3606 0.4115 0.3864 0.3508 0.4185 0.3803 0.4165 0.3587 0.3819 0.4491 0.3799 0.3346 0.4253 0.3699 0.3903 0.3818 0.3279 0.5460 0.3710 0.3898 0.5007 0.4102 0.3958
0.0331 0.0351 0.0834 0.0404 0.0496 –0.0077 –0.0119 –0.1330 –0.0251 –0.0085 0.0113 0.0329 0.0078 –0.0219 –0.0107 –0.0895 0.0078 0.0890 0.0652 0.0439 0.0279 0.0119 0.0294 0.0713 0.0618
0.4226 0.4396 0.3513 0.4137 0.3769 0.3411 0.4228 0.3965 0.4195 0.3510 0.3736 0.4412 0.3786 0.3292 0.4301 0.3885 0.3949 0.3748 0.3167 0.5421 0.3687 0.3846 0.4868 0.3875 0.3805
0.4496 0.4669 0.3873 0.4032 0.4227 0.3683 0.4078 0.3644 0.4231 0.3496 0.3968 0.4569 0.3707 0.3217 0.4360 0.3265 0.3669 0.3861 0.3464 0.5565 0.3781 0.3875 0.5132 0.4248 0.4221
–0.0348 –0.0380 0.0088 0.0856 –0.0881 –0.0430 0.0399 –0.0595 –0.0486 0.0010 –0.0478 0.0202 0.0407 –0.0262 –0.0249 0.0688 0.0463 0.0584 0.0032 –0.0107 0.0088 –0.0038 –0.0238 –0.0426 –0.0199
a. b. c. d.
Use least-squares estimation to find uˆ . From Theorem 4.7, find an unbiased estimator of σ 2ε ? Given that α = 0.05, find the confidence region for u. Determine the simultaneous 100 (1 − α )% confidence intervals for the ui, i = 1, 2, 3. Use α = 0.05.
REFERENCES 1. Johnson, R.A. and Wichern, D.W., Applied Multivariate Statistical Analysis, 4th ed., Prentice Hall, Upper Saddle River, NJ, 1998. 2. Sen, A. and Srivastava, M., Regression Analysis: Theory, Methods and Applications, Springer-Verlag, New York, 1990. 3. Srivastatva, M.S., Methods of Multivariate Statistics (Wiley Series in Probability and Statistics), John Wiley & Sons, New York, 2002. 4. Akaike, H., Autoregressive model fitting for control, Annals of the Institute of Statistical Mathematics, 23, 163–180, 1971.
2151_book.fm Page 90 Friday, October 20, 2006 5:04 PM
90
Stream of Variation Modeling and Analysis for MMPs 5. Allen, D.M., Mean squared error of prediction as a criterion for selecting variables, Technometrics, 13, 469–475, 1971. 6. Takeuchi, K., Yanai, H., and Mukherjee, B.N., The Foundations of Multivariate Analysis: A Unified Approach by Means of Projection onto Linar Subspaces, John Wiley & Sons, New York, 1982. 7. Muirhead, R.J., Aspects of Multivariate Statistical Theory (Wiley Series in Probability and Statistics), John Wiley & Sons, New York, 1982. 8. Morrison, D.F., Multivariate Statistical Methods, 2nd ed., McGraw-Hill, New York, 1976. 9. Kleinbaum, D.G. et al., Applied Regression Analysis and Other Multivariate Methods, 2nd ed., PWS-KENT Publishing Company, Boston, MA, 1988. 10. Manly, B.F.J., Multivariate Statistical Methods: A Primer, Chapman and Hall, London, 1986. 11. Kshirsagar, A.M., Multivariate Analysis, Marcel Dekker, New York, 1972. 12. Karson, M.J., Multivariate Statistical Methods, The Iowa State University Press, Ames, IO, 1982. 13. Hair, J.F. et al., Multivariate Data Analysis, 3rd ed., Macmillan Publishing Company, New York, 1992.
ADDITIONAL READING Hair, J.F. et al., Multivariate Data Analysis, 3rd ed., Macmillan Publishing Company, New York, 1992. Karson, M.J., Multivariate Statistical Methods, The Iowa State University Press, Ames, IO, 1982. Kleinbaum, D.G. et al., Applied Regression Analysis and Other Multivariate Methods, 2nd ed., PWS-KENT Publishing Company, Boston, MA, 1988. Kshirsagar, A.M., Multivariate Analysis, Marcel Dekker, New York, 1972. Manly, B.F.J., Multivariate Statistical Methods: A Primer, Chapman and Hall, London, 1986. Morrison, D.F., Multivariate Statistical Methods, 2nd ed., McGraw-Hill, New York, 1976. Muirhead, R.J., Aspects of Multivariate Statistical Theory (Wiley Series in Probability and Statistics), John Wiley & Sons, New York, 1982.
2151_book.fm Page 91 Friday, October 20, 2006 5:04 PM
5
Principal Component Analysis and Factory Analysis
5.1 PRINCIPAL COMPONENT ANALYSIS A complex system may involve large volumes of data with more than hundreds of different variables. In such situations, it is desirable to find ways to effectively reduce the dimensions of the data with minimum loss of information, and interpret the data with physical meanings. Principal component analysis (PCA) provides the capability to identify patterns in high-dimensional data and reconstruct it to highlight similarities and differences. The concept of PCA was first introduced by Karl Pearson in 1901 and formally treated by Hotelling [1] and Rao [2]. PCA has been widely and successfully applied in various fields by analyzing the variance-covariance structure through linear combinations of multivariate data. In this section, the basic mathematical model of PCA, its geometrical interpretation, as well as related statistical tests will be discussed. A potential application in process control is also presented.
5.1.1 MATHEMATICAL MODEL
OF
PCA
Suppose X T = [X1 X2 Xp] is a random vector with variance-covariance
(
)
∑ x = var(X) = E (X − µ X )(X − µ X )T ,
(5.1)
Clearly, ∑x is at least positive semidefinite with rank r ≤ p; thus, all the eigenvalues are real and nonnegative. Without loss of generality, let the eigenvalues be λ1 ≥ λ2 ≥ ≥ λp ≥ 0 and the standardized eigenvectors be e1, e2, …, ep. Denote
E = e1
e2
e11 e21 e p = e p1
e12
e22
ep2
e1 p e2 p e pp
T
(5.2)
then EET = ETE = I and ET∑xE = Λ, or ∑x = EΛET where Λ = diag(λ1, λ2, , λp). Consider the linear combination: 91
2151_book.fm Page 92 Friday, October 20, 2006 5:04 PM
92
Stream of Variation Modeling and Analysis for MMPs
Yi = ai1 X1 + ai 2 X2 + + aip X p = a Ti X , (i = 1, 2, …, p)
(5.3)
where a Ti = ai1 ai 2 aip , a Ti a i = 1 and X T = [X1 X2 Xp]. The variance-covariance structure of Yi is determined by: var(Yi ) = var(a Ti X) = a Ti ∑ X a i cov(Yi , Yj ) = cov(a iT X, a Tj X) = a Ti ∑ X a j , for i, j = 1, 2, …, p.
(5.4)
As ∑ X = EΛET and λ1 ≥ λ2 ≥ ≥ λp ≥ 0 p
var(Yi ) = a Ti ∑ x a i = a Ti EΛET a i =
∑ λ (a e ) i
T i
2
j
j =1
(5.5)
p
≤ λ1
∑ (a e ) T i
j
2
= λ1a Ti EET a i = λ1a Ti a i = λ1
j =1
when a i = e1, “=” holds. So a i = e1 maximizes var(Yi ) and, λ1 = var(e1T X) = e1T ∑ X e1T = max a T ∑ X a a:a T a =1
(5.6)
It can be shown that for any i = 1, 2, …, p λ i = var(eTi X) , and cov(eTi X, eTj X) = eTi ∑ X e j = e Ti EΛET e j = 0 for i ≠ j
(5.7)
So the principal components (PCs) of Xi, i = 1, 2, …, p, are the uncorrelated linear combinations of Xi, i = 1, 2, …, p with the components of eigenvectors as combination coefficients. Property 1: Let ∑X be the covariance matrix of random vector X T = [X1 X2 Xp] and its eigenvalue-eigenvector pairs be (λi, ei) i = 1, 2, …, p with λ1 ≥ λ2 ≥ ≥ λp ≥ 0. Then the ith PC is defined by the linear combination: Yi = ei1 X1 + ei 2 X 2 + ⋅ ⋅ ⋅ + eip X p = eTi X, i = 1, 2, …, p
(5.8)
and var(Yi ) = λ i . Remark 5.1: var(Y1 ) ≥ var(Y2 ) ≥ ≥ var(Y p ) ≥ 0 because λ1 ≥ λ2 ≥ ≥ λp ≥ 0.
2151_book.fm Page 93 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
93
Remark 5.2: If some eigenvalues are identical, the choice of ei , and hence Yi , is not unique. P
P
Property 2:
∑
var(Yi ) =
i =1
∑ var( X ) . i
i =1
Property 3: The correlation between PCs and original variables is:
ρ(Yi , X j ) =
cov(Yi , X j ) var(Yi ) var( X j )
=
eij λ i σ ii
, i, j = 1, 2, …, p.
From Property 2, we can see that the total variance will not be significantly affected by ignoring components with smaller variances. This is why the PCA approach can be used to reduce variables without losing too much information. The contribution rate of the ith principal component can be defined as: var(Y j )
=
p
λi
(5.9)
p
∑ var(Y ) ∑ λ j
j
j =1
j =1
The larger this rate, the larger the capability of this PC to summarize the information in Xi , i = 1, 2, ..., p . The accumulated contribution rate of the first k PCs is given by: k
k
∑ var(Y ) ∑ λ j
j =1 p
=
∑ var(Y ) ∑ λ j
j =1
j
j =1 p
(5.10) j
j =1
If this rate becomes 85%, it indicates that the first k PCs extract 85% original variation information in Xi , i = 1, 2, ..., p . Thus, many variables are reduced effectively without losing too much information. We define the Loss of Information (LoI) using the first k principal components instead of all the variables Xi , i = 1, 2, ..., p as: p
LoI =
∑ i=1
p
k
var( Xi ) –
∑
var(Yi ) =
i =1
where k is selected by minimizing LoI.
∑ i =1
p
k
λi –
∑ i =1
λi =
∑λ
i = k +1
i
(5.11)
2151_book.fm Page 94 Friday, October 20, 2006 5:04 PM
94
Stream of Variation Modeling and Analysis for MMPs
p
p
k
k
∑ var( X ) – ∑ var(a X) ≥ ∑ var( X ) – ∑ var (e X) T i
i
i =1
i=1
T i
i
(5.12)
i =1
i=1
Equality holds if and only if ai = ei. PCs can also be calculated using “standardized” variables. Remark 5.3: In general, the eigenvalue–eigenvector pairs obtained from a covariance matrix are different from those obtained from the correlation matrix. This leads to different PCs from the covariance matrix ∑ and correlation matrix R. The covariance matrix ∑ is recommended when the variables are measured in the same, or comparable, units. Sometimes, the population variance-covariance ∑ X is not known. In this case, the sample covariance (or correlation) matrix is used in PCA. Corresponding to the results from the population, there are similar results from the random sample. Property 4: Let x be the sampling data matrix and S be the sample covariance matrix with eigenvalue–eigenvector pairs ( λˆ i , eˆ i ), i = 1, 2, …, p, with λˆ 1 ≥ λˆ 2 ≥ ≥ λˆ p ≥ 0 . Then the ith PC is defined by: yˆi = eˆi1 x1 + eˆi 2 x2 + … + eˆip x p = eˆ Ti x , i = 1, 2, …, p .
(5.13)
The variance-covariance matrix of yˆ T = [ yˆ1 , yˆ2 ,..., yˆ p ] is: var( yˆ ) = diag (λˆ 1, λˆ 2 ,..., λˆ p )
(5.14)
The total sample variance is: p
∑
p
var( yˆi ) =
i=1
∑
p
var( xi ) =
i =1
∑
p
sii =
i=1
∑ λˆ
i
(5.15)
i =1
and the sample correlation matrix R between PCs and x with (i, j) entry is:
ρ( yˆi , x j ) =
cov( yˆi , x j ) var( yˆi ) var( x j )
=
eˆij λˆ i sii
, i, j = 1, 2, ..., p
(5.16)
In summary, PCA procedures using sampling measurements are: Step 1: Obtain sample matrix x and standardize data if the variables are not of the same unit. Step 2: Calculate the sample covariance matrix S or correlation matrix R.
2151_book.fm Page 95 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
95
X2
Y2
Y1
O
X1
FIGURE 5.1 Relationship between PCs Y1, Y2 and original variables X1, X2.
Step 3: Compute the sample eigenvalue–eigenvector pairs (λˆ i , eˆ i ), i = 1, 2, …, p, with λˆ 1 ≥ λˆ 2 ≥ ≥ λˆ p ≥ 0 . Step 4: Obtain the sample PCs: yˆ i = eˆi1 x1 + eˆi 2 x2 + + eˆip x p = eˆ Ti x, i = 1, 2, …, p. Remark 5.4: From the derivation of PCs, it is evident that they only depend on the covariance matrix ∑ X and multinormal distribution is not required. However, normality is required for statistical tests.
5.1.2 GEOMETRICAL INTERPRETATION From the previous subsection, the p PCs can be algebraically viewed as special linear combinations of original variables Xi , i = 1, 2, …, p, i.e. Yi = ei1 X 1 + ei 2 X 2 + + eip X p = eTi X , i = 1, 2, …, p
(5.17)
From a geometrical point of view, these uncorrelated linear combinations Yi , i = 1, 2, …, p, form a new coordinate system by rotating the old coordinate system spanned by Xi , i = 1, 2, …, p. The PCs have the maximum variance. For simplicity, we will illustrate this transformation using p = 2. The 2-D scatter plot of data is shown in Figure 5.1. Suppose we have n samples. Let the original variables be X1, X2 and the two PCs be Y1,Y2. Further, let us assume XT = (X1, X2) ~ N(µ , ∑). Without loss of generality, we can assume that the data are centered at the origin, i.e., µ = 0. For a bivariate normal distributed random variable X, the n points are scattered in an ellipse with the direction of Y1 being its long-axis direction and the direction of Y2 being its short-axis direction. This is obtained by rotating the original coordinate system to a new coordinate system with angle θ. Let cos(θ) E= − sin(θ)
sin(θ) cos(θ)
(5.18)
2151_book.fm Page 96 Friday, October 20, 2006 5:04 PM
96
Stream of Variation Modeling and Analysis for MMPs
Clearly, E is an orthogonal matrix. Denote the preceding relationship in the matrix form: Y1 cos(θ) = Y2 − sin(θ)
sin(θ) X1 ⋅ = E⋅X cos(θ) X2
(5.19)
From Figure 5.1, the first PC, Y1, exhibits the maximum variance and the second PC, Y2 exhibits the second largest variance. So by choosing the linear combination with the maximum dispersion, most variation in X1, X2 can be explained. If the ellipse is very flat, then the total sample variation can be approximated by the dispersion in the Y1 direction, and thus the effect of Y2 on total variation can be ignored. In this sense, the one-dimensional PC is obtained by reducing the original 2-D data. In general, p variables Xi, i = 1, 2, …, p span a p-dimensional space, and n samples can be viewed as n points in this p-dimensional space. Then, selecting p PCs is equivalent to identifying the p pivotal axes of the p-dimensional ellipse.
5.1.3 INFERENCES
ON
PCS
In this subsection, we will consider two types of inferences in which normal distribution assumption about X ~ N p (µ , ∑) is required: 1. Statistical testing for independence of the original variables, i.e., H0: R = I, where R is the correlation matrix of multinormal variables X. For a large sample size n, a likelihood ratio test for H0 is given by p
|R| =
∏ λˆ
i
(5.20)
i=1
where λˆ i , i = 1, 2, …, p, are eigenvalues of correlation matrix R. H0 is rejected if:
( )
− m ln R > χ 2p( p−1)/ 2 (α)
5.21)
where m = n − 1−
2p +5 . 6
It should be noted that PCA can only be performed if H0 is rejected. 2. Statistical testing for isotropy for eigenvalues, i.e., H0: λq+1 = λq+2 = … = λp = λ, which can be used to determine the number of PCs.
2151_book.fm Page 97 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
97
Equation 5.10 or Equation 5.11 can be used to determine the number of PCs, i.e., to choose the first few PCs to replace the original p (usually very large) variables without much loss of information. Here, an alternative method is given based on the testing of the isotropy: H0: λ q +1 = λ q +2 = ... = λ p = λ
(5.22)
H a : λ i ≠ λ for at least one i = q + 1, q + 2, ..., p where λ is an unknown constant. If H0 is not rejected, then in addition to the first q PCs, all the remaining PCs must be added if any one of the PCs is added, because they are isotropic. The likelihood ratio test for H0 is based on the statistic: 1/( p − q )
p λˆ i i=q +1 Q= p 1 λˆ i p − q i=q +1
∏
∑
(5.23)
where λˆ i is an eigenvalue of the sample covariance matrix S, i =1, 2, …, p. Lawley [3] showed that 2 1 Q * = − n − 1 − q − 2( p − q) + 1 + ⋅ ln(Q ) 6 p − q
(5.24)
is asymptotically distributed as χ 2f (α ) where f =
1 ( p − q )( p − q + 1) − 1 . 2
So H0 is rejected if Q* > χ 2f (α ). When H0 is rejected, more than q PCs are needed. If H0 is not rejected, then at the significance level α, the first q PCs are enough to replace the p original variables without much loss of information.
5.1.4 APPLYING PCS
TO
PROCESS CONTROL
It is known from previous chapters that quality ellipses and T 2 charts can be used as multivariate control charts. With a large number of variables, PCs can be used to construct a control chart.
2151_book.fm Page 98 Friday, October 20, 2006 5:04 PM
98
Stream of Variation Modeling and Analysis for MMPs
Consider data matrix x11 x 21 x= x n1
x12
x 22
xn 2
x1 p x2 p x np
with (µ , ∑). The first two sample PCs yˆ1 and yˆ2 are considered because they account for the largest cumulative proportion of the total sample variance and can be visualized easily. Richard Johnson [4] suggested two types of control charts: 1. Ellipse format chart based on the first two PCs for large n: The first two PCs yˆ1 and yˆ2 are independent, so the control ellipse is constructed by: yˆ12 yˆ22 + ≤ χ 22 (α ) ˆλ λˆ 2 1
(5.25)
This ellipse can be used to check the stability of measurements. If there are out-of-control points, special causes may exist in the process. 2. T 2 chart based on the remaining PCs: Assume X − µ ∼ N p (0, ∑). Let Yi, i = 1, 2, …, p, be centered PCs with mean 0. A T 2 chart is constructed from the residual information not contained in the first two PCs. Based on the last p – 2 PCs, the statistic p
∑λ
Y j2
j =3
~ χ2p−2 .
j
As Yˆj , j = 3, 4, ..., p, λˆ j
are independent and normally distributed, the T 2 chart can be created based on the statistic p
Ti 2 =
∑ λˆ j =3
yˆij2 j
2151_book.fm Page 99 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
99
and the upper control limit (UCL) can be defined as UCL = χ2p−2 (α ) . This chart can be used to identify special causes of variation.
5.2 FACTOR ANALYSIS 5.2.1 INTRODUCTION In many areas such as psychology, sociology, and health care, it is not always possible to directly measure concepts of major interest such as intelligence, satisfaction, or quality of life. In the manufacturing and service industries, the quality of a product or service is the critical issue and is usually affected by many direct or indirect factors. The effective identification and interpretation of these common factors become the key to the success. The early stages of most statistical investigations in the categories listed earlier are characterized by little prior quantitative information being available about the area of study. Consequently, early research efforts are hit-and-miss by nature — many variables are studied in the hope of identifying some kind of order that will deepen understanding of the subject. Although correlation analysis can be used to identify the interrelationship among variables, practical problems arise when we have to deal with a large number of variables. The problem is that the number of coefficients keeps accelerating as the number of variables increases. So, a higherorder data reduction technique is required that can systematically summarize large covariance or correlation matrices. Factor analysis is one of the data reduction techniques. It is a family of procedures for removing redundancy from a set of correlated variables and representing the variables with a smaller set of “derived” variables, or factors. Alternatively, the factor analysis procedure can be thought of as removing duplicated information from a set of variables. We may also think of it loosely as a group of similar variables, although these variables are often unobservable.
5.2.2 COMPARISON
OF
FACTOR ANALYSIS
AND
PCA
Factor analysis and PCA are functionally very similar and can be used for data dimension reduction. However, they are quite different in terms of underlying assumptions. PCA is concerned with explaining the variance-covariance structure of a set of variables (XT = [X1 X2 Xp]) through a few linear combinations of these variables (YT = [Y1 Y2 Yp]). The PCs are those uncorrelated linear combinations that can be defined as Y = UX. Algebraically, PCs (Yi’s) are particular (uncorrelated) linear combinations of the random variables (Xi’s). Geometrically, these linear combinations represent the selection of a new coordinate system obtained by rotating the original system with U. Therefore, PCA is a method for analyzing the structure of the data covariance, but not for modeling the relationship between data and some latent variables. Factor analysis assumes that the variance of a single variable can be decomposed into a common variance that is shared by other variables included in the model, and
2151_book.fm Page 100 Friday, October 20, 2006 5:04 PM
100
Stream of Variation Modeling and Analysis for MMPs
a specific variance that is unique to a particular variable, including the error component. Factor analysis analyzes only the common variance of the observed variables, whereas PCA considers the total variance and makes no distinction between common and unique variances [5]. Stevens [6] described two criteria that could be used to choose one technique over the other: •
•
The objective of the analysis: Factor analysis and PCA are similar as the purpose of both methods is to reduce the original variables into fewer composite variables, called factors or PCs. However, they are distinct because the obtained composite variables serve different purposes. In factor analysis, a small number of factors are extracted to account for the intercorrelations among the observed variables, i.e., to identify the latent variables that explain why they are correlated with one another. The objective of PCA is to account for the maximum portion of the variance present in the original set of variables with a minimum number of composite variables, called PCs. The assumption about the variance in the original variables: If the observed variable measurements are relatively error free (for example, age, years of education, or number of family members), or if it is assumed that the error and specific variance represent a small portion of the total variance in the original set of variables, then PCA is appropriate. But if the observed variables are only indicators of the latent structures to be measured (such as test scores or responses to attitude scales), or if the error variance represents a significant portion of the total variance, then factor analysis will be the more appropriate technique.
5.2.3 ORTHOGONAL FACTOR MODEL Factor analysis is a multivariate analysis technique aiming at reducing the number of variables while preserving as much original information as possible. In other words, it is used to uncover the latent structure (dimensions) of a set of variables. The essential purpose of factor analysis is to describe the covariance relationship among many variables in terms of a few underlying, but unobservable, and random quantities. The factor model is motivated by the argument that variables can be grouped according to their correlations. That is, suppose all variables within a particular group are highly correlated among themselves, but have relatively small correlations with variables in a different group. Then it is conceivable that each group of variables represents a single underlying factor that is responsible for the observed correlations. We suppose X1, X2, …, Xp are p observable random variables. Remember that if these p variables do not have the same unit, standardization should be done to avoid unreasonable results. The factor model postulates that these variables are linearly dependent on a few unobservable random variables F1, F2, …, Fm, and p additional sources of variation ε1, ε2, …, εp, which cannot be explained by Fj , j = 1, 2, …, m. In particular, the factor analysis model in a matrix notation is:
2151_book.fm Page 101 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
X1 a11 X2 = a21 : X p a p1
a12 a22
... ...
a p2
...
a1m F1 ε1 a2 m F2 ε 2 + ... a pm Fm ε p
101
(5.26)
or X p×1 = A p× m Fm×1 + ε p×1
(5.26′)
where F1, F2, …, Fm are called common factors and ε1, ε2, …, εp are called residual errors or specific factors. The coefficient aij is called the loading of the ith variable on the jth factor. So the matrix A is the loading matrix of factor analysis. Because the p variables, X1, X2, …, Xp, are expressed in terms of p + m unobservable random variables, F1, F2, …, Fm and ε1, ε2, …, εp, it is impossible to give a direct verification of the factor model from the observations on X1, X2, …, Xp with so many unobservable quantities. Thus, additional assumptions on F1, F2, …, Fm and ε1, ε2, …, εp must be made to simplify the covariance relationships: 1. m ≤ p. 2. E(εε) = 0, ∑ ε = Cov(ε ) = diag σ12 , σ 22 ,…, σ 2p , i.e., ε1, ε2, …, εp are not correlated and the variances are different. 3. Cov(F, ε ) = 0, i.e., F and ε are uncorrelated. 4. E(F) = 0, ∑ F = Cov(F) = diag 1, 1, …, 1 = Im , i.e., the F1, …, Fm are uncorrelated and their variances are 1.
(
)
(
)
The preceding assumptions make the model Equation 5.26 an orthogonal factor model. Remark 5.5: The models of Equation 5.26 or Equation 5.26′ give the objective of factor analysis, i.e., to use F as a surrogate of X such that the dimension of the variable space is reduced from p to m. Remark 5.6: In PCA, the coefficient matrix A is determined and unique. But in factor analysis, the loading matrix A is not unique, because for any m × m orthogonal matrix Γ , Equation 5.26′ can be rewritten as: X p×1 = ( A p× m Γ m× m )(Γ Tm× m Fm×1 ) + ε p×1
(5.26″)
It is easy to verify that Equation 5.26″ still satisfies the assumptions, i.e., ∑ Γ T F = Cov(Γ T F) = Im , Cov(Γ T F, ε ) = 0 So Γ Tm× m Fm×1 is also a common factor, and A p× m Γ m× m is the loading matrix.
2151_book.fm Page 102 Friday, October 20, 2006 5:04 PM
102
Stream of Variation Modeling and Analysis for MMPs
Remark 5.7: This nonuniqueness provides a nice property. When the structure of A is complex, it can be simplified by an appropriate transformation such that the new factors have meanings that can be better interpreted. Unlike the coefficients in a regression model, the common factor F is unobservable in the factor analysis model.
5.2.4 STATISTICAL INTERPRETATIONS AND COMMUNALITY
OF
FACTOR LOADINGS
5.2.4.1 Interpretation of Factor Loadings Under assumptions listed in the previous section, we know that E(F) = 0, σ i2 = var(εi) = 1, µ = E(X) = 0, and var( Xi ) = σ 2ii , i= 1, 2, …, p. The orthogonal factor model given in Equation 5.26 implies a covariance structure for X: cov(X, F) = E (X − µ )(F − 0)T = E ( AF + ε )FT = A ∑ F + E (εFT ) = A ∑ F = A or cov( Xi , F j ) = aij
(5.27)
So the factor loadings are the covariance coefficients between the observable variables and factors. Similarly, if the data X is standardized, i.e., var(Xi) = σii2 = 1, factor loadings are the correlation coefficients between the observable variables and factors, i.e., ρ( Xi , F j ) = aij . Define p
Sj =
∑ a , j = 1, 2, ..., m 2 ij
i =1
where the squared factor loading aij2 is the percentage of variance in variable Xi accounted for by the factor Fj. Sj is the contribution of Fj to X, i.e., the sum of variance is contributed by Fj to all X. To obtain the percentage of variance in all the variables explained by each factor, the sum of the squared factor loadings for that factor is calculated and then divided by the number of variables. 5.2.4.2 Interpretation of Communality As mentioned earlier, we have: ∑ X = Cov(X) = E (X − µ )(X − µ )T = AAT + ∑ ε or 2 σ 2ii = var( Xi ) = ai21 + ... + aim + σ i2 , Cov ( Xi , X j ) = ai1a j 1 + ... + aim a jm
(5.28)
2151_book.fm Page 103 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
103
The portion of the variance of the ith variable var(Xi) contributed by the m 2 common factors, i.e., ai21 + ... + aim , is called the ith communality. The portion of var(Xi) due to the specific factor, i.e., σ 2i , is called the uniqueness or specific variance. Thus, the variance of the ith variable can be decomposed into two parts as follows: var(Xi) = communality + specific variance, i = 1, 2, …, p
(5.29)
2 Denoting the ith communality by hi2 = ai21 + ... + aim , the ith communality is the sum of squares of the loadings of the ith variable on the m common factors. Hence, from Equation 5.28, the “standardization” yields:
σ 2ii = hi2 + σ i2 = 1
(5.29)
If hi2 ≈ σ ii2 , then almost all the original information contained in the ith variable Xi is extracted by common factors, and thus hi2 is critical component of var(Xi). However, if hi2 ≈ 0, then the common factors have a small impact on Xi, and the specific factor ε i dominates the Xi. So the communality measures the percentage of variance in a given variable accounted for by all the factors.
5.2.5 ESTIMATION
OF
LOADING MATRIX
Suppose we are given n observations x1, x2, …, xn on p generally correlated variables. To construct the factor analysis model, we need to estimate the factor loading matrix A based on these observations. There are some popular methods of parameter estimation, such as, the principal component method and the maximum likelihood method. Here, we will only concentrate on the first method. The MLE method can be seen in Reference 4. Let ∑ X = Cov(X) have eigenvalue λ1 ≥ λ 2 ≥ ... ≥ λ p ≥ 0 and associated normalized eigenvectors ei , i = 1, 2, …, p then:
p
∑X =
∑λ e e
T i i i
i =1
= λ1 e1
λ eT 1 1 λp ep T λ p e p
(5.30)
If m = p, then from Equation 5.28 and Equation 5.30, ∑ X = AAT + 0 = AAT
(5.31)
A comparison of Equation 5.30 and Equation 5.31 yields A = λ1 e1
λpep
(5.32)
2151_book.fm Page 104 Friday, October 20, 2006 5:04 PM
104
Stream of Variation Modeling and Analysis for MMPs
If m < p and the last p – m eigenvalues are small, we neglect the contribution of p
∑λee
T i i i
to ∑ X ;
i = m +1
then:
∑ X ≈ λ1 e1
λ1 e1T λ m em T λ m e m
(5.33)
λ1 e1T λ m e m + ∑ ε = AAT + ∑ ε T λ m e m
(5.34)
If we consider the specific factors, then:
∑ X ≈ λ1 e1
To use this method in a data set x1, x2, …, xn, if the variables do not have the same unit, it is first customary to standardize the data by x ji − xi
, j = 1, 2, …, n, i = 1, 2, …, p
(5.35)
sii If ∑ X is unknown, we can use the sample variance-covariance matrix S obtained from the standardized data. So the estimated factor loading matrix is given by ˆ = λˆ eˆ A 1 1
λˆ m eˆ m
(5.36)
where λˆ 1 ≥ λˆ 2 ≥ ≥ λˆ p are eigenvalues of sample covariance matrix S (or correlation matrix R), and eˆ i , i = 1, 2, …, p, are the associated normalized orthogonal eigenvectors of S or R. The estimated specific variances are provided by the diagonal ˆ = diag σˆ 2 , σˆ 2 , …, σˆwith ˆ ˆ T and, thus, ∑ elements of the matrix S − AA ε m +1 m+2
(
m
σˆ 2i = sii −
∑ aˆ
ij
2
.
j =1
2 The communalities are estimated by hˆi2 = aˆi21 + ... + aˆim .
2151_book.fm Page 105 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
105
Fk
~ Fk
kj
Fj ~ Fj
FIGURE 5.2 Orthogonal rotation of common factors.
5.2.6 FACTOR ROTATION The objective of factor modeling is not only to find the common factors Fj, j = 1, 2, …, m, and group the original variables Xi, i = 1, 2, …, p, but also to make each common factor more interpretable and meaningful. Thus, factor rotation is usually necessary to facilitate the interpretation of factors. The nonuniqueness of a loading matrix makes the rotation feasible. Through factor rotation, the structure of the loading matrix is simplified in the sense of making each variable have larger loading only on one common factor, and smaller or moderate loading on other common factors. There are two categories of rotational methods to choose from: orthogonal rotation or oblique rotation. Oblique rotations allow the factors to be correlated, and so a factor correlation matrix is generated when oblique is requested. Normally, however, with orthogonal rotation, the axes are kept perpendicular as they are rotated and no factor correlation matrix is produced, as the correlation of any factor with another is zero. In this subsection, the commonly used maximum variance orthogonal rotation is introduced. The oblique rotation methods is illustrated in Figure 5.2 and can be found in Reference 4. We start with m = 2. Let the factor loading matrix a11 a21 A= a p1
a12 a22 . a p 2
To avoid imbalance caused by communality hi2 = ai21 + ai22 , i = 1, 2, …, p, the ith row of this loading matrix is preprocessed by dividing hi2 for i = 1, 2, …, p. For simplification, we still use A. Let cos ϕ T= sin ϕ
− sin ϕ cos ϕ
2151_book.fm Page 106 Friday, October 20, 2006 5:04 PM
106
Stream of Variation Modeling and Analysis for MMPs
be an orthogonal matrix and define the new matrix a11 cos ϕ + a12 sin ϕ a21 cos ϕ + a22 sin ϕ B = AT = a p1 cos ϕ + a p 2 sin ϕ
− a11 sin ϕ + a12 cos ϕ − a21 sin ϕ + a22 cos ϕ − a p1 sin ϕ + a p2 cos ϕ
b11 b 21 b p1
b12 b22 (5.37) b p 2
The rotation angle ϕ is selected by making the structure of loading matrix A as simple as possible. In other words, we hope to divide the variables Xi , i = 1, 2, …, p into two categories, one related to F1 and the other related to F1. This is achieved by making the two variances V1 and V2 as large as possible, where V1 and V2 are obtained from the two column vectors of B, respectively. Thus, we select rotation angle ϕ such that ϕ = arg max(V ) = arg max(V1 + V2 ) ϕ
(5.38)
ϕ
where
1 Vj = p
p
∑ i =1
1 (b ) − p 2 2 ij
Using the principle of maximization
p
∑ i =1
2
b , j = 1, 2. 2 ij
dV = 0, we have: dϕ D−
tan( 4ϕ) =
1 (2 AB) p
(5.39)
1 C − ( A2 − B2 ) p
where p
A=
∑
p
αi , B =
i =1
∑
p
βi , C =
i =1
2
∑ i =1
2
p
(α 2i − β2i ) , D = 2
∑ α β , and i
i
i =1
a a a a α i = i1 − i 2 , βi = 2 i1 i 2 for i = 1, 2, …, p hi hi hi hi
2151_book.fm Page 107 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
107
TABLE 5.1 The Range of Rotation Angle ϕ Sign of Nominator
Sign of Denominator
+
+
+
–
Range of 4 ϕ
π
+
–
8 π
~π
–π ~ − π
π
0~
2
–
–
π
0~
2 –
Range of ϕ
~
8 π
−
2
−
~0
2
π 4
π 4 π
~−
π 8
~0
8
The range of ϕ is determined according to the signs of denominators and nominators of tan(4ϕ), which are summarized in Table 5.1. If there are m > 2 common factors, we need to conduct the preceding orthogonal rotation on each pair of common factors, i.e., orthogonally rotate the factor space span {Fk, Fj} with ϕ kj for k = 1, 2, …, (m – 1) and j = k + 1, …, m. Each ϕ kj is determined by rotating a11 a21 A= a p1
a12 a22
... ...
a p2
...
a1m a2 m ... a pm
to b11 b21 B = ATkj = b p1
b12 b22 bp2
... ... ...
b1m b2 m ... b pm
by maximizing m
m
V=
1 p
p
∑ V =∑ ∑ j
j =1
j =1
i =1
1 (b ) − p 2 2 ij
p
∑ i =1
b 2 ij
2
.
2151_book.fm Page 108 Friday, October 20, 2006 5:04 PM
108
Stream of Variation Modeling and Analysis for MMPs
where Tkj is the fundamental rotation matrix, and bik = aik cos ϕ + aij sin ϕ bij = − aik sin ϕ + aij sin ϕ
(5.40)
bil = ail where i = 1, 2, …, p and j, k, l = 1, 2, …, m. The rotation angle ϕ is determined as before. m For m factors, the total number of rotations is ; thus, the rotated loading 2 matrix is given by: B(1) = AT12 T1m T(m −1)m = AC1
(5.41)
where C1 = T12 T1m T(m −1)m . Let V(1) be the variance of B(1). Continuing the earlier procedure, we get the sequences: B(1) , B( 2) , B(3) … and
(5.42)
V(1) ≤ V(2 ) ≤ V(3) ≤
(5.43)
The sequence has an upper bound because the absolute value of factor loading cannot exceed 1. Thus lim V( k ) exists; let k →∞
V * = lim V( k ) . k →∞
For a given precision δ, when k is sufficiently large, | V( k ) − V * |< δ
(5.44)
In practice, if | V( k ) − V( k +1) | < ε (ε > 0 is a small number), then we stop the rotation and take B( k ) = AC1C 2 C 3 C k = AC
(5.45)
as the factor loading matrix. Remark 5.8: The sum of eigenvalues is not affected by rotation, but the rotation will alter the eigenvalues of particular factors and will change the factor loadings.
2151_book.fm Page 109 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
109
Because multiple rotations may explain the same variance (have the same total eigenvalue) but have different factor loadings, and because factor loadings are used to intuit the meaning of factors, this implies that different meanings may be ascribed to the factors depending on the rotation — a problem some cite as a drawback of factor analysis.
5.2.7 FACTOR SCORE The factor model is given in Equation 5.26, Xi = ai1F1 + ai 2 F2 + aim Fm, i = 1, 2, …, p The common factors Fj , j = 1, 2, …, m, reflect the relationship between original variables Xi , i = 1, 2, …, p. Usually, it is convenient to study the characteristics of the problem using the common factors in place of original variables. Suppose common factors can be linearly represented by the original variables, i.e., F j = β j 1 X1 + β j 2 X2 + + β jp X p , j = 1, 2, …, m
(5.46)
Equation 5.46 is called the factor score function and can be used to compute the common factor score for each sample. For example, when m = 2, we can plug in the p variables of each sample and compute F1 and F2. The scatter plot of F2 vs. F1 can be used to classify the samples or conduct further analysis. When m < p, the factor scores cannot be precisely computed. There are many methods, such as, weighted least-squares regression, that can be used to estimate the factor scores. Here, Thomson’s [7] regression method is introduced. Suppose the original variables and common factors have been standardized. If common factors Fj , j = 1, 2, …, m, can be regressed on Xi , i = 1, 2, …, p, then the regression equation can be expressed as: Fˆ j = b j 1 X1 + b j 2 X2 + + b jp X p , j = 1, 2, …, m
(5.47)
where the common factors are unknown and need to be estimated. From Equation 5.27 and Equation 5.47, for i = 1, 2, …, p and j = 1, 2, …, m, we can obtain: aij = cov( X i , Fj ) = E ( X i Fj ) = E[ X i (b j1X1 + b j 2 X 2 + + b jp X p )] = b j1E ( X i X1 ) + b j 2 E ( X i X 2 ) + + b jp E ( X i X p ) = b j1 cov( X i , X1 ) + b j 2 cov( X i , X 2 ) + + b jp cov( X i , X p ) = b j1ri1 + b j 2ri 2 + + b jprip
2151_book.fm Page 110 Friday, October 20, 2006 5:04 PM
110
Stream of Variation Modeling and Analysis for MMPs
i.e., b j 1r11 + b j 2 r12 + + b jp r1 p = a1 j b j 1r21 + b j 2 r22 + + b jp r2 p = a2 j b j 1rp1 + b j 2 rp 2 + + b jp rpp = a pj
(5.48)
In matrix notation, Equation 5.48 can be expressed as: Rb j = a j
(5.49)
where b j = (b j 1, b j 2 ,..., b jp )T , a j = ( a1 j , a2 j ,..., a pj )T , and R is the correlation matrix. By solving Equation 5.49, we get: b j = R −1a j
(5.50)
Let b1T b11 T b b21 B= 2 = T b m bm 1
b12
b22
bm 2
b1 p b2 p bmp
Then: (R −1a1 )T −1 T (R a 2 ) B= = A T R −1 −1 T (R a m )
(5.51)
Thus, from Equation 5.47 and Equation 5.51, the common factors can be estimated by: Fˆ1 b T X 1T b X Fˆ Fˆ = 2 = 2 = BX = A T R −1X Fˆm b Tm X
(5.52)
2151_book.fm Page 111 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
5.2.8 GENERAL PROCEDURES
FOR
111
FACTOR ANALYSIS
The general procedures for factor analysis are summarized as follows: Step 1: Standardize original data if the variables do not have the same unit. Step 2: Derive the covariance matrix S or correlation matrix R. Step 3: Derive the eigenvalue and eigenvector pairs of S or R to get the factor loadings according to the cumulative contribution of the eigenvalues. Step 4: Rotate the factor loading. Step 5: Name and interpret the factors.
5.3 EXERCISES 1. It is given that the covariance matrix of the three (p = 3) standardized random variables Z1, Z2, and Z3 is 1.0 ρ = .07 .63
.07 1.0 .09
.63 .09 1.0
a. In order to generate Z1, Z2, and Z3 by an m = 1 factor model, list the necessary assumptions regarding ε ( ε = [ε1 ε2 ε3]T) and F1. b. Show that, based on the assumptions listed in (1), Z1, Z2, and Z3 can be generated by Z1 = .7F1 + ε1
.51 Z 2 = .1F1 + ε 2 and Ψ = cov(ε ) = 0 0 Z 3 = .9F1 + ε 3
0 .99 0
0 0 . .19
That is, write ρ in the form ρ = LLT + Ψ. 2. The eigenvalues and eigenvectors of the correlation matrix ρ in Problem 1 are λ1 = 1.64 , e1 = [.6953
.1716
.6980 ]T
λ 2 = .98 , e2 = [.1375
−.9849
.1052 ]T
λ 3 = .37 , e3 = [.7055
.0228
−.7084]T
2151_book.fm Page 112 Friday, October 20, 2006 5:04 PM
112
Stream of Variation Modeling and Analysis for MMPs
a. Assuming an m = 2 factor model, calculate the loading matrix L, and specific variance Ψ using the principal component solution method. b. What proportion of the total variance is explained by the m = 2 common factors? 3. Using the basic factor analysis model and assumptions, proof the following statements: a. If T is an m × m orthogonal matrix, then cov(TTF) = I and cov(TTF,εε) = 0. b. The factor model is X− µ = L
( p×1)
( p×1)
F + ε (m < p),
( p× m ) ( m ×1)
( p×1)
let L* = LT and F* = T T F, where T is an m × m orthogonal matrix, then cov(X) = L*L*T + Ψ. c. If the factor score is given by Fˆ = LT ∑ −1 (X − µ ), where L is estimated from covariance matrix ∑ by the principal component decomposition method (i.e., L = [ λ 1 e1 λ 2 e2 λ m em]), then E(Fˆ ) = 0 and Cov(Fˆ ) = I . Hint: i. cov(A) = E[(A – E(A)A – E(A))(A – E(A))T], where E(A) is the expectation of matrix A. ii. L and ∑ are deterministic, i.e., E(L) = L and E(∑) = ∑. iii. {λj, ej}, j = 1, 2, …, m are the eigenvalue and eigenvector pairs of ∑, which has the property that λj ej = ∑ej for j = 1, 2, …, m. 4. Consider an assembly process. A side panel of a car body is located on the fixture through a 4-way pin P1 and a 2-way pin P2. To monitor the side panel’s quality, ten points are measured and their nominal positions are shown in Figure 5.3. In this case, only three potential sources of variation are considered: a. The x direction variation of P1 b. The z direction variation of P1 c. The z direction variation of P2
FIGURE 5.3 Product and process description.
2151_book.fm Page 113 Friday, October 20, 2006 5:04 PM
Principal Component Analysis and Factory Analysis
113
The process is simulated by deliberately introducing process variation sources. The measurement data of points M1, M2, M5, and M6 are collected and given in the attached data file (Problem 4 dataset.xls). Use the factor analysis method to identify the introduced process variation sources. Hint: You may follow the following procedures to solve this diagnosis problem: i. Derive the data covariance S or correlation R matrix. Indicate the reason why you have chosen one (either S or R) rather than the other. ii. Derive the eigenvalue/eigenvector pairs from the matrix you selected in (i). iii. Determine the number of common factors. Indicate the criteria for your decision. iv. Derive the factor loadings for the common factors and try to interpret and link them with potential process variation sources considered. v. Are the factor loadings in (iv) interpretable? If not, try to improve the interpretability of the factor loading by deriving the new loading vectors. Indicate the method and criteria for achieving your new factor loadings. Interpret the new factor loadings, and figure out the variation sources introduced.
REFERENCES 1. Hotelling, H., Analysis of a complex of statistical variables into principal components, Journal of Educational Psychology, 24, 417–441,498–520, 1933. 2. Rao, C.R., The use and interpretation of principal component analysis in applied research, Sankhya A, 26, 329–358, 1964. 3. Lawley, D.N., Tests of significance for the latent roots of covariance and correlation matrices, Biometrika, 43, 128, 1956. 4. Johnson, R.A. and Wichern, D.W., Applied Multivariate Statistical Analysis, 4th ed., Prentice Hall, Upper Saddle River, NJ, 1998. 5. Morrison, D.F., Multivariate Statistical Methods, McGraw-Hill, New York, 1990. 6. Stevens, J., Applied Multivariate Statistics for the Social Sciences, 4th ed., Lawrence Erlbaum Associates, Mahwah, NJ, 2001. 7. Thomson, G.H., The Factor Analysis of Human Ability, Houghton Mifflin Company, New York, 1939. 8. Darlington, R.B., Weinberg, S., and Walberg, H., Canonical variate analysis and related techniques, Review of Educational Research, 453–454, 1973. 9. Rubenstein, D.I., Ecological Aspects of Social Evolution: Birds and Mammals, Princeton University Press, Princeton, NJ, 1986.
ADDITIONAL READING Darlington, R.B., Weinberg, S., and Walberg, H., Canonical variate analysis and related techniques, Review of Educational Research, 453–454, 1973. Rubenstein, D.I., Ecological Aspects of Social Evolution: Birds and Mammals, Princeton University Press, Princeton, NJ, 1986.
2151_book.fm Page 114 Friday, October 20, 2006 5:04 PM
2151_book.fm Page 115 Friday, October 20, 2006 5:04 PM
Part II Variation Propagation Modeling in MMP
2151_book.fm Page 116 Friday, October 20, 2006 5:04 PM
2151_book.fm Page 117 Friday, October 20, 2006 5:04 PM
6
State Space Modeling for Assembly Processes
This chapter develops a state space model characterizing variation propagation in a multistage rigid-part assembly process. First, the background of a multistage assembly process is briefly described using automotive body assembly processes. Major variation sources are then identified and modeling assumptions are summarized, on which a station-indexed state space model is subsequently developed. The state space model for depicting stream of dimensional variation is actually a result of recursively applying standard kinematic analysis to individual operation stages. However, the state space model is not a simple summation of multiple single-stage models, because the across-stage transitions will affect how variation propagates and complicate the dynamics of the entire system. The model is validated through a comparison with commercially available variation simulation software.
6.1 INTRODUCTION OF MULTISTAGE ASSEMBLY PROCESSES A multistage assembly process involves multiple stations or operations to assemble individual components into a sophisticated product. Typical multistage assembly processes include automotive assembly, printed circuit board (PCB) assembly, and aircraft fuselage assembly, to name a few. We will use the car body assembly process to highlight the features of this method. The major part of a car body is the body-in-white (BIW), namely, the structural skeleton of a car. Figure 6.1 shows an example of the BIW. The BIW provides a physical frame on which closure panels, including doors, hood, fenders, and lift gate or trunk, will be mounted. The finished BIW will be measured in a coordinate system as shown in Figure 6.1. This coordinate system is known as the body coordinate system in the automotive industry. A BIW is usually made of 100 to 150 sheet metal panels, which are assembled by a process involving 55 to 75 assembly stations [1]. Figure 6.2 shows the corresponding process layout for producing the BIW in Figure 6.1. The BIW assembly process starts with the component subassembly process, in which a sequence of assembly stations builds up a set of large subassemblies, including the left-hand side aperture (LH-APT), the right-hand side aperture (RH-APT), and the underbody. These subassemblies will then be fed into a framing station, where industrial robot welders will joint them together to form the BIW. Figure 6.2 also marks the optical coordinate measuring machine (OCMM) at a few different locations. OCMM stations are where the key dimensional features of a finished or semifinished car are measured in the production line. In a modern assembly system, OCMM stations are 117
2151_book.fm Page 118 Friday, October 20, 2006 5:04 PM
118
Stream of Variation Modeling and Analysis for MMPs
FIGURE 6.1 Example of an automotive body (body-in-white) and inline sensing using OCMM.
OCMM
Station #2
Left Hand Aperture Assembly
Station #1 Framing
OCMM Right Hand Aperture Assembly Station #5
Station #4
Station #3
Station #6 Underbody Assembly
OCMM
OCMM: Optical Coordinate Measuring Machine
Body-In-White
OCMM
FIGURE 6.2 BIW assembly layout (arrows represent a part input/output at each assembly station).
commonly distributed along the assembly line, instead of being implemented only at the end of a production line. The taxonomy of this treelike representation is the exact graphical equivalent of the physical process. More details can be found in [1]. On each assembly station, part assembly is initiated in two or more sets of fixture locators that aim to constrain the degrees of freedom of a part during a joining operation to ensure a repeatable build. Figure 6.3 illustrates a typical 3-2-1 fixture set used in car body assembly processes. It consists of two locating pins, LP4-way and LP2-way , and three net contact blocks, NCi, i = 1, 2, 3. The two locating pins constrain three degrees of freedom in the x-z plane, where the 4-way pin controls part motion in both the x and z directions and the 2-way pin controls part motion in the z direction. Three NC blocks constrain other degrees of freedom of the workpiece. When a workpiece is nonrigid, more than three NC blocks may be needed to reduce part deformation. An n-2-1 fixture layout, denoted by {LP4-way , LP2-way , NCi , i = 1, 2, .., n}, is a more generic setting in multistage assembly processes. For notation simplicity, unless otherwise indicated, we will also use LP1 to denote a 4-way locator and LP2 for a 2-way locator on a part. Hence, for a 2-D part, we can use the simplified notation {LP1, LP2} to denote the fixture locator pair used to position the part.
2151_book.fm Page 119 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
119
NC 3
LP 2way NC 2
LP 4way NC1
workpiece
y x z FIGURE 6.3 Illustration of a 3-2-1 fixture.
6.2 VARIATION FACTORS AND ASSUMPTIONS Dimensional variation is a major problem affecting product quality in multistage assembly processes. Previous studies have shown that dimensional problems contribute to roughly two thirds of all quality-related problems during new product launches [2,3]. In a multistage system, the major variation factors can be generally classified into those on individual stations and those induced by across-station transitions.
6.2.1 STATION-LEVEL VARIATION FACTORS One of the major factors for dimensional quality of the finished product at the station level is the repeatability of fixture locators used to hold the parts, generally referred to as fixture variation. Fixture locators are used extensively in processes such as car body assembly to provide physical support and dimensional reference within the body coordinate system, thereby determining the dimensional accuracy of the final assembly [3,4]. Fixture locators may fail to provide the desired positioning repeatability (relative to tolerances) during production owing to gradual deterioration of locators and catastrophic events, such as broken or deformed locators. Figure 6.4 shows an example of fixture fault manifestation, where δLP2(z) is a z-directional fixturing deviation caused by the malfunctioning pinhole at LP2. This deviation will affect the positions of the measurement points SPi, i = 1, 2, 3. Another variation factor is the manufacturing imperfection of individual parts, such as deviations in part dimensions or geometrical features from their design,
sensor andSP i : sensor position fixture locators
SP 2
SP 1
z
LP 2 x
workpiece: a panel
LP 1
LP 2(z)
LP 1(z) LP 1(x)
4-way locator, positioning variability in two directions
LP 2(z)
SP 3
FIGURE 6.4 Fixture-locator-induced dimensional variation.
2-way locator, positioning variability in one direction
2151_book.fm Page 120 Friday, October 20, 2006 5:04 PM
120
Stream of Variation Modeling and Analysis for MMPs mating feature Part 1
(a) Lap joint
Part 2
Part 1
Part 2
(b) Butt joint
Part 1
Part 2
(c) Mixed joint
Defined direction
FIGURE 6.5 Cross-sectional views of joint geometries.
generally referred to as part variation. Part variation is actually a result of the processes fabricating the parts: as the manufacturing process is inherently imperfect, part variation is unavoidable. Nevertheless, whether part variation affects the downstream process or not depends on what type of part-to-part joint is involved. In car body assembly processes, Ceglarek and Shi [5] reported that there are two basic important types of joints: one is the lap joint (Figure 6.5a) and the other is the butt joint (Figure 6.5b). The lap joint can absorb variation in the direction of the slip plane and the butt joint can be used as a datum to locate another part. Therefore, variation will propagate in the defined direction through the mating feature only when using butt joints. Figure 6.5c shows a generic joint geometry that can partially allow motion in the defined direction. The third variation factor is related to the flexibility of a part. If a part is flexible, such as the metal panels of a car body, it can deform during the assembly operation owing to incorrect locator positions, uneven forces, or part-to-part interferences. This deformation will cause additional dimensional variability in the finished product. In this chapter, we will assume that rigid body parts are used in assembly operations.
6.2.2 ACROSS-STATION VARIATION FACTORS Shiu et al. [6] discovered a phenomenon called reorientation in a multioperation process, when a change in fixture layouts occurs as the subassembly proceeds to a new station. Figure 6.6 illustrates the reorientation phenomenon, in which the dashed line represents the nominal position of parts and the solid line indicates the actual position. Assume that positioning errors are associated with fixture locators on station k, but not with those on station k + 1. Then, when part 1 and part 2 are assembled on station k, there is some relative position deviation between these two components; i.e., if part 1 is positioned according to its nominal position, part 2 will deviate from its nominal position, and vice versa. When the subassembly of part 1 and part 2 is transferred to station k + 1, it will be repositioned on station k + 1 in order to be assembled with a new part 3. Suppose that the pin-holes on part 2 are used to hold subassembly “1+2.” Even if fixture locators that hold part 2 on station k + 1 are free of error, as shown in Figure 6.6, part 1 may deviate from its nominal position in subassembly “1+2+3.” This deviation of part 1 is not a result of locator deviation at station k + 1. Rather, it is a combined result of deviation of locators on the previous
2151_book.fm Page 121 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
Part 2
Part 1
Fixture deviation
121
Station k
Deviation of part 1 comes from re-orientation of subassembly 1+2 from station k to station k+1 Part 1
Part 2
Station k+1 Part 3
FIGURE 6.6 Across-station variation factor: reorientation.
station and the subassembly repositioning at the current station. The significance of this reorientation-induced deviation is that it actually couples the variations originating from individual stations, as we will see in Subsection 6.3.2. This reorientation-induced error would be almost unavoidable for a multistation assembly process when a different set of fixture locators or a different datum scheme is used to reposition a subassembly on a downstream station. When the parts are considered flexible, the reorientation effect is influenced by flexibility-related effects (such as spring back), and they will further complicate the way variation propagates across stations. The modeling of variation and its propagation taking part flexibility into account can be found in Camelio et al. [7].
6.2.3 SUMMARY
AND
ASSUMPTIONS
FOR
MODELING
The key variation factors can be summarized as: • • • •
Dimensional errors associated with fixture locators at each assembly station Dimensional errors due to part fabrication Across-station reorientation-induced error Flexibility effects of individual components
In this chapter, we will focus on a simplified rigid-part assembly process. That is, we will limit ourselves to considering only 2-D panel-like parts positioned by a 3-2-1 fixture layout. Moreover, we will concentrate on the modeling and analysis of fixture-locator-related errors and assume that only the lap joint is present between parts. Owing to the variation-absorbing nature of a lap joint, part fabrication errors will therefore not affect the stream of variation in the whole production line. The assumptions are summarized as follows: • • •
2-D rigid body part 3-2-1 Fixture layout for rigid part Lap joint between parts
2151_book.fm Page 122 Friday, October 20, 2006 5:04 PM
122
Stream of Variation Modeling and Analysis for MMPs
The resulting model will also be applicable to an n-2-1 fixture layout if the fixture variations being considered cause panel motion largely in the plane of rigidity. Shiu et al. [6] indicated that this simplified process is applicable to 60% of situations in a typical automotive body assembly process. The part fabrication errors will be explicitly considered and modeled using a differential motion vector in Chapter 7 for a machining process.
6.3 STATE SPACE MODELING 6.3.1 REPRESENTATION
OF
PART POSITION
AND ITS
DEVIATION STATE
In dimensional quality control, each part is characterized by its deviation from the nominal position. Let x, y, and z be the translation coordinate variables and, α, β, and φ be the corresponding rotation coordinate variables (please refer to Figure 6.7 for the six coordinate variables [or degree of freedoms]). As we will only explicitly model a 2-D assembly process in this chapter, the position of each single part or a multipart subassembly is represented by three coordinate variables, x, z, and β. In Figure 6.7, the subscripts (i, k) of the coordinate variables indicate that they are for part i on station k, where i = 1, …, np, k = 1, …, N, np is the total number of parts, and N is the total number of stations. To define the position and orientation of a part, we need to specify reference points. In this chapter, we will assign two reference points to each part, and make them coincide with the locating points denoted by LP1,i,k and LP2,i,k. On station k, the position of a part will be represented by the instantaneous position of LP1,i,k, (LP1,i,k (x), LP1,i,k (z)), in the body coordinate system, and its orientation represented by the angle between the line connecting LP1,i,k and LP2,i,k and the x-axis of the body coordinate system. For a subassembly that consists of more than one part, its position and orientation is similarly defined. Each 2-D rigid body subassembly will be positioned by a 4-way locator and a 2-way locator, which are chosen from the pool of 4-way and 2-way locators on the subassembly. The 4-way/2-way locator pair used to position the subassembly is treated as the reference points for the subassembly, and they are
parti on station k
LP 2,i,k LP 1,i,k
z
(LP1,i,k(x), LP 1,i,k(z))
y x
FIGURE 6.7 Representation of part deviation.
i,k
2151_book.fm Page 123 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
123
denoted by LS1,s,k and LS2,s,k, respectively, where the subscripts have a similar meaning but s is the index for subassembly here. The part position indicated in a design blueprint is known as the nominal position, which is the ideal situation when neither fixture variation nor part variation exists. As locating tools and part fabrications are not perfect in reality, the part will deviate from its nominal position. Given that one knows part nominal positions from a design blueprint, people often use deviations from such positions to represent its actual position. Denote the random deviations associated with each of the six degrees of freedom (three translations and three rotations) of part i at station k by xi,k ≡ [δxi,k δyi,k δzi,k δαi,k δβi,k δφi,k]T, where δ in front of a coordinate variable represents a small deviation. For a 2-D assembly, the deviation state of each xi,k can be simplified as [δxi,k δzi,k δβi,k]T. For all the np parts in the final assembly, the deviation state of the whole assembly on station k can be expressed as x k ≡ [x1T,k x Tn p ,k ]T. In a multistation process, part i may not yet have appeared on station k. If not, the corresponding xi,k = 0. We denote the index of the first station where part i appears by ξi. For the deviation state of a multipart subassembly s at station k, we use the notation qs,k to distinguish it from an individual part, but the notation’s physical meaning is the same. We will use uk to denote the random deviation associated with locator pairs on station k. For example, for the locator pair that supports subassembly s on station k, its deviation will be us,k = [δLS1,s,k (x) δLS1,s,k (z) δLS2,s,k (x) δLS2,s,k (z)]T. For all the nk locator pairs used on station k, the fixture deviation vector is xk ≡ [u1T,k u Tnk ,k ]T.
6.3.2 SOME PRELIMINARY RESULTS Two lemmas are presented here before we derive the state space model, which will be used intensively during later derivations. They are based on rigid part kinematics. The detailed proofs of both lemmas can be found in the appendix 6.1 and 6.2, and also in Reference 8 or Reference 9. Lemma 6.1: When a rigid body undergoes a translation and a rotation away from its nominal position, if the rotation angle (denoted as δβ) is small, then deviations of any two points on the same part (denoted as [ δxa, δza ]T and [δxb, δzb ]T) have the relationship: δxb δx a a ,b δ z = R 1 δza b δβ δβ
(6.1)
1 = 0 0
(6.2)
where
R
a ,b 1
0 1 0
−( zb − za ) xb − x a 1
and (xa, za), (xb, zb) are the nominal coordinates of those two points.
2151_book.fm Page 124 Friday, October 20, 2006 5:04 PM
124
Stream of Variation Modeling and Analysis for MMPs
Lemma 6.2: The part deviation state xi,k can be related to the deviations of its reference points using a linear approximation, provided that the deviations at these points are much smaller than the distance between them; that is:
x i,k = R 2i,k
δLP1,i,k ( x ) δLP1,i,k ( z) δLP2,i,k ( x ) δLP2,i,k ( z)
(6.3)
where
R
i, k 2
1 = 0 sin βi,k LDi,k
0 1 cos βi,k − LDi,k
0 0 sin βi,k − LDi,k
0 0 cos βi,k LDi,k 3× 4
(6.4)
and LDi,k is the Euclidean distance between LP1,i,k and LP2,i,k . Lemma 6.2 can be easily extended to a multipart subassembly, as stated in the following two corollaries. Corollary 6.1: If subassembly s at station k is located by points LS1,s,k and LS2,s,k, then its deviation state due to small deviations at the locating points is:
q s ,k = R s3,k
δLS1,s ,k ( x ) δLS1,s ,k ( z) = R s ,k ⋅ u 3 s ,k δLS2,s ,k ( x ) δLS2,s ,k ( z)
(6.5)
where
R
s ,k 3
1 = 0 sin βs ,k SDs ,k
0 1 cos βs ,k − SDs ,k
0 0 0 0 sin βs ,k cos βs ,k − SDs ,k SDs ,k
(6.6)
and SDs,k is the Euclidian distance between LS1,s,k and LS2,s,k. The next corollary characterizes the reorientation when a subassembly is transferred to the next station. As explained in Subsection 6.2.2, reorientation will induce deviation in a part even if the current locating points are free of error.
2151_book.fm Page 125 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
125
Corollary 6.2: Suppose that subassembly s is supported by LS1,s,k and LS2,s,k at station k and also assume that these two locating points are free of error at the current station k. The deviation state of subassembly s on station k when moving from station k – 1 to station k can be expressed as a linear combination of deviations accumulated in its locating points LS1 and LS2 at the previous station k – 1.
q s ,k = R k4−1,k
δLS1,s ,k −1 ( x ) δLS1,s ,k −1 ( z) δLS2,s ,k −1 ( x ) δLS2,s ,k −1 ( z)
(6.7)
where
R
k −1, k 4
−1 = 0 sin βs ,k − SDs ,k
0 −1 cos βs ,k SDs ,k
0 0 0 0 sin βs ,k cos βs ,k − SDs ,k SDs ,k
(6.8)
6.3.3 STATE SPACE REPRESENTATION Based on the previous discussions, as well as the analysis done by Jin and Shi [10], the deviation state of part i on station k can be expressed as a summation: x i,k = x i,k −1 + d if,k + d ir,k
(6.9)
where d if,k and d ir,k are the fixture-error-induced deviation and reorientation-induced deviation, respectively, and the superscripts have the same meaning as the subscript of x. Equation 6.9 is illustrated in Figure 6.8, in which x i,k −1 is the intermediate state of part i during the operations on station k. The two deviation terms d if,k and d ir,k in Equation 6.9 are given by the following theorems.
Deviated part from the previous station
On station k dr
xi,k-1
Re-orientation
~ x i,k 1
df Fixture error
FIGURE 6.8 Deviation accumulation of part i on station k.
x i,k
Input to the next station
2151_book.fm Page 126 Friday, October 20, 2006 5:04 PM
126
Stream of Variation Modeling and Analysis for MMPs
Part i
Subassembly s
LP 2,i,k.
LP 1,i,k
z
LS1,s,k
LS2,s,k
x
FIGURE 6.9 Fixture error-induced deviation.
Theorem 6.1: Suppose part i is in subassembly s on station k. As illustrated in Figure 6.9, its fixture error-induced deviation d if,k can be expressed as d if,k = B1i,k u k
(6.10)
where R 1,s ,k B1i ,k = 1 0 3× 4nk LS
W1 (s) = ∆1s I4× 4
,LP1,i ,k
R 3s ,k W1 (s) if k ≥ ξ i if k < ξ i
∆ 2 s I 4× 4
∆ hs I4× 4
(6.11)
4 × 4 nk
(6.12)
The Kronecker Delta is 1 if h = s ∆ hs = 0 if h ≠ s
(6.13)
Proof: According to Corollary 6.1, the deviation of subassembly s due to fixture errors at LS1,s,k and LS2,s,k is the qs,k expressed in Equation 6.5. Because part i is on subassembly s, Lemma 6.1 can be employed to obtain the deviation state of part i, which is d if,k = R 1
LS1,s , k , LP1,i , k
q s ,k
(6.14)
Substituting Equation 6.5 into Equation 6.14 yields d if,k = R 1
LS1,s , k , LP1,i , k
R s3,k ⋅ u s ,k
(6.15)
2151_book.fm Page 127 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
127
The W1’s defined in Equation 6.12 is a selecting matrix to determine which set of locator pairs is the one affecting the deviation of part i, i.e., u s ,k = W1 (s ) ⋅ u k
(6.16)
Combining Equation 6.15 and Equation 6.16, we end up with the expression of the upper part in Equation 6.11. The lower part in Equation 6.11 implies that fixture error will not be reflected in the deviation of part i before this part appears in the assembly stream. Theorem 6.2: If part i is on subassembly s on station k, its reorientation-induced deviation d ir,k can be expressed as d ir,k = B2i,k F1x J ,k −1 + B2i,k F2 x G ,k −1
(6.17)
where LS , LP R 1,s ,k 1,i,k R k4−1,k Bi2,k = 1 0
1 0 F1 = 0 0 0 0 F2 = 1 0
0 0 0 1
iff k > ξ i if k ≤ ξ i
(6.18)
0 0 0 0
(6.19)
0 0 LP1,G ,k ,LP2 ,G ,k R1 0 0
(6.20)
0 1 0 0
and J and G are the indices of parts on which the points LS1,s,k and LS2,s,k are located, respectively. Proof: The d ir,k is actually caused by the locating deviation accumulated up to station k – 1; therefore, it can be expressed as:
d ir,k = R 1
LS1,s , k , LP1,i , k
R k4−1,k
δLS1,s ,k −1 ( x ) δLS1,s ,k −1 ( z) δLS2,s ,k −1 ( x ) δLS2,s ,k −1 ( z)
(6.21)
2151_book.fm Page 128 Friday, October 20, 2006 5:04 PM
128
Stream of Variation Modeling and Analysis for MMPs
where δLS h,s ,k −1 ( x ) and δLS h,s ,k −1 ( z) (h = 1, 2) have the same meaning as those in Corollary 6.2. Here, R k4−1,k converts fixture deviations at the previous station into the deviation state of the associated subassembly during part transferring across stations (Corollary 6.2), whereas R 1LS1,LP1 further connects the deviation of subassembly s to that of part i using Lemma 6.1. Provided that LS1,s,k is on part J and LS2,s,k on part G, respectively, the deviation information of these two points at the previous station k–1 is actually preserved in xJ,k-1 and xG,k-1, that is, δLS1,s ,k −1 ( x ) 1 = δLS1,s ,k −1 ( z) 0
0 x J ,k −1 0
(6.22)
0 LP1,G ,k ,LP2 ,G ,k x G ,k −1 R1 0
(6.23)
0 1
and δLS2,s ,k −1 ( x ) 1 = δLS2,s ,k −1 ( z) 0
0 1
Putting them together, Equation 6.22 and Equation 6.23 yield δLS1,s ,k −1 ( x ) 1 δLS1,s ,k −1 ( z ) 0 = δLS2,s ,k −1 ( x ) 0 0 δLS2,s ,k −1 ( z ) 0 0 + 1 0
0 1 1 0 0 0 0 0 1 0 0 1
0 x J ,k −1 0
0 1
(6.24) 0 1
0 LP1,G ,k ,LP2 ,G ,k xG ,k −1 R1 0
Substituting Equation 6.24 into Equation 6.21, and also using definitions in Equation 6.18 to Equation 6.20, Equation 6.17 can be obtained. Again, the lower part of Equation 6.18 is 0 because the reorientation-induced deviation only begins to contribute from station k + 1 if part i first appears on station k. Utilizing Theorem 6.1 and Theorem 6.2, Equation 6.9 can be written as, x i,k = x i,k −1 + B1i,k u k + Bi2,k F1x J ,k −1 + Bi2,k F2 x G ,k−1
(6.25)
This equation implies that the deviation state of part i is coupled with that of other parts. It is then impossible to write a decoupled deviation propagation equation
2151_book.fm Page 129 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
129
for each individual part. Alternatively, we can write a state space equation using the deviation state vector of all np parts, which is x k = [x1T,k x Tn p ,k ]T. First, we need to replace x J ,k −1 and x G ,k −1 in Equation 6.25 with the aggregated state vector xk. Define W2, another selecting matrix, as W11 W2 (s, k ) = W21
W12
W22
W1n p W2 n p 6 ×3 n p
(6.26)
where I W jg = 3×3 03×3
if (j, g ) = (1, J ) or (2, G ) otherwise
(6.27)
W2 is used to determine which parts on subassembly s contribute to the reorientation-induced error at station k. Then, define Hi,k as Hi,k = [Bi2,k F1
Bi2,k F2 ]
(6.28)
Using Equation 6.26 and Equation 6.28, Equation 6.25 turns out to be x i,k = x i,k −1 + B1i,k u k + Hi,k W2 (s, k ) x k
(6.29)
Furthermore, define Bk and Hk–1 as B k = [(B11,k )T H k −1 = [(H1,k W2 (s, k ))T
(B12,k )T
...
(B11,nk )T ]T
(H2,k W2 (s, k ))T
...
(H n p ,k W2 (s, k ))T ]T for k ≥ 2,
(6.30)
and H0 = 03 n p ×3 n p. Then, Equation 6.29 can be written as x k = (I3 n p ×3 n p + H k −1 ) x k −1 + B k u k , or one step further, x k = A k −1x k −1 + B k u k where A k −1 = I3 n p ×3 n p + H k −1 .
(6.31)
2151_book.fm Page 130 Friday, October 20, 2006 5:04 PM
130
Stream of Variation Modeling and Analysis for MMPs
Equation 6.31 is the state equation in our station-indexed state space model, governing the deviation propagation in an assembly process under assumptions listed in Subsection 6.2.3. If coordinate sensors are placed on assembly stations along the assembly line, an observation equation should be employed to describe the relationship between the dimensional measurements and the deviation state. Denote mi,k as the number of measurement points on part i at station k and mk as the total number of measurement points at station k. Then the deviation of each point on part i is: y ij,k = R 0
LP1,i , k , SPj ,i
x i, k
j = 1, 2, , mi,k
(6.32)
0 LP1,i,k ,SPj ,i R1 0
(6.33)
where 1 = 0
LP1,i , k , SPj ,i
R0
0 1
and SPj,i is the location of the jth sensor on part i. Thus, the measurement vector of part i on station k is y i,k = [( y1i,k )T
( y i2,k )T
m
( y i,ki,k )T ]T
(6.34)
and it can be written as y i, k = C i, k x i, k
(6.35)
where
C i, k
R 0LP1,i,k ,SP1,i LP ,SP R 1,i,k 2 ,i = 0 LP1,i,k ,SPmi,k ,i R 0 2 mi , k × 3
(6.36)
Then, the deviation vector of all the measurement points on station k is y k ≡ [ y1T where
y Tmk ]T = C k x k
(6.37)
2151_book.fm Page 131 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
C1,k 0 C k = 0
0 C 2,k 0
131
0 0 C n p ,k
(6.38)
If process background disturbances, higher-order terms due to linearization, and measurement noises are considered together with Equation 6.31 and Equation 6.32, the state space model governing dimensional deviation propagation in an N-station process will be: x k = A k −1x k −1 + B k u k + w k , k = 1, 2, …, N
(6.39)
y k = C k x k + v k {k} ⊂ {1, 2, …, N}
(6.40)
where wk and vk are the random vectors representing process disturbances or higherorder terms and measurement noises, respectively.
6.4 MODEL VALIDATION To validate the state space model developed in Section 6.3, consider the assembly process of the side frame of an SUV. The final product, the inner-panel-complete, comprises four components: A-pillar, B-pillar, rail roof side panel, and rear quarter panel, which are shown in Figure 6.10. The four parts are assembled on three stations (Stations I, II, and III). Then, the final assembly is inspected at Station IV, where SP1 to SP 10 marked in Figure 6.10d are the key dimensional features. In such a multistation process, the aforementioned 3-2-1 fixture is used on every station to ensure product dimensional accuracy. In Figure 6.10, LP1 to LP8 are locating points, often called principal locating points (PLPs) in the automotive industry. A fixture layout in a multistation process can be represented using these PLPs as follows: {{LP1, LP2}, {LP3, LP4}}I → {{LP1, LP4}, { LP5, LP6}}II → {{ LP1, LP6}, {{LP7, LP8}}III → {{ LP1, LP8}}IV , where the assembly process starts from Station I (indicated by the subscript) and the arrow represents a transition from one station to the next. As an example, {{LP1, LP4},{LP5, LP6}}II means that at Station II, the first workpiece, the subassembly “A-pillar+B-pillar,” is located by LP1 and LP4 and the second workpiece, the rail roof side panel, is located by LP5 and LP6. The design nominal x and z coordinates for fixture locators and measurement points are given in Table 6.1 and Table 6.2, respectively.
2151_book.fm Page 132 Friday, October 20, 2006 5:04 PM
132
Stream of Variation Modeling and Analysis for MMPs
Rail roof side panel
LP 2 A-Pillar
A-pillar
LP 5 LP 6
LP 3 LP 1 B-pillar
LP 4
z
LP 1 y
(a) Station I: pillar and pillar assembly
(b) Station II: addition of rail roof side panel
SP 7
SP 6
SP 2
LP 4
B-pillar
x
LP 8
A-pillar
SP 1
LP 6
SP 10 SP 3 SP 4
LP 7
Rail roof side panel
SP 9
SP 5
LP 1
SP 8
Rear quarter panel B-pillar
(d) Station IV: key product features
(c) Station III: final assembly of inner-panel
FIGURE 6.10 Assembly process of an SUV side frame.
TABLE 6.1 Coordinates of Fixture Locators (in Figure 6.10a, Figure 6.10b, and Figure 6.10c; units: mm) (x,z)
LP1 (367.8,906.05)
LP2 (667.47,1295.35)
LP3 (1301,1368.89)
LP4 (1272.73,537.37)
(x,z)
LP5 (1470.71,1640.40)
LP6 (1770.50,1702.62)
LP7 (2941.42,1691.31)
LP8 (2120.32,1402.83)
TABLE 6.2 Coordinates of Measurement Points (in Figure 6.10d; units: mm) (x,z)
SP1 (271.50,905)
SP2 (565.7, 1634.7)
SP3 (1289.7,1227.5)
SP4 (1306.5,633.5)
SP5 (1244.5,85)
(x,z)
SP6 (1604.5,1781.8)
SP7 (2884.8, 1951.5)
SP8 (2743.5, 475.2)
SP9 (1838.4,226.3)
SP10 (1979.8,1459.4)
A state space model is established, following the general modeling procedure, as outlined in Section 6.2, to represent the variation propagation in such a fourstation assembly process. Suppose that there are tolerances of ±1.0 mm associated with all the locating points on each of the three assembly stations and we assume
2151_book.fm Page 133 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
133
TABLE 6.3 Comparison between the State Space Model and VSA (unit: mm)
Mean
σ) Std(σ
Mean
σ) Std(σ
Discrepancy between Std’s (%)
271.5002 905.0007 565.7050 1634.6972 1289.6876 1227.4772 1306.5058 633.4785 1244.4226 84.9773 1640.5004 1781.7971 2884.8178 1951.5027 2743.5147 475.2045 1838.4151 226.3067 1979.8176 1459.4052
0.0020 0.1862 1.4103 0.3833 0.8361 0.9885 1.0123 0.9999 1.7122 0.9608 0.9545 0.8902 1.1852 1.1136 1.6114 0.9210 1.9107 0.4698 0.9165 0.3229
271.5000 905.0007 565.7053 1634.6986 1289.6876 1227.4778 1306.5059 633.4783 1244.5228 84.9764 1640.5007 1781.7985 2884.8065 1951.5037 2743.5094 475.2034 1838.4099 226.3016 1979.8075 1459.4019
0.0020 0.1861 1.4102 0.3833 0.8361 0.9885 1.0123 0.9999 1.7122 0.9608 0.9545 0.8900 1.1853 1.1136 1.6115 0.9211 1.9107 0.4698 0.9165 0.3229
0.0000 0.0537 0.0071 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0225 0.0084 0.0000 0.0062 0.0109 0.0000 0.0000 0.0000 0.0000
VSA
SP1(x) SP1(z) SP2(x) SP2(z) SP3(x) SP3(z) SP4(x) SP4(z) SP5(x) SP5(z) SP6(x) SP6(z) SP7(x) SP7(z) SP8(x) SP8(z) SP9(x) SP9(z) SP10(x) SP10(z)
State Space Model
that fixture locators at the measurement station (Station IV) are free of error. Also, it is assumed that deviations at the locating points are normally distributed and the tolerance limits (±1.0) correspond to the 6σ value of a corresponding normal distribution. Variations observed at the sensor locations are calculated using the state space model and a simulation software package, Variation System Analysis (VSA), based on 5000 simulations. The mean value and the standard deviation (std) of all measurement points from both models are compared in Table 6.3. The discrepancy between the std values from the two models is less than 0.054% for this four-station example. We feel that the state space model can provide an adequate linear variation propagation model that can predict dimensional variation as accurately as VSA does under some circumstances.
6.5 EXERCISES This problem aims to provide an exercise to help get familiar with the state space model and perform assembly process simulations. The process analyzed corresponds to the assembly of an automobile side frame, as presented in Figure 6.11. The side frame has three parts: the fender, the side ring, and the rear quarter.
2151_book.fm Page 134 Friday, October 20, 2006 5:04 PM
134
Stream of Variation Modeling and Analysis for MMPs
FIGURE 6.11 Description of the components of a sedan side frame.
FIGURE 6.12 Assembly sequence of the sedan side frame.
The assembly sequence is presented in Figure 6.12. In station 1, the fender and the side ring are assembled together. The subassembly is then moved to Station 2, where the rear quarter is added. Finally, the whole assembly is transferred to the measurement station to measure the KPCs given as the measurement points (M1 to M8). The dots on each part represent the locating points used to locate the parts in each station. The odd numbers correspond to the 4-way locators and the even numbers to the 2-way locators. It is important to note that in station 2, the subassembly coming from station 1 is held by locators P1 and P4. In the measurement station, the locators used are P1 and P6. The position of the locators and measurement points are presented in Table 6.4. Problem 6.1 (Single-part model): In this problem, consider the complete side frame as one part, as shown in Figure 6.13. 1. Derive the relationships that relate the deviations at the measurement points M4 and M6 with respect to the deviations of the locating points P1 and P6. (Hint: Obtain matrix T (T = R1 R2) in this case.) 2. From the result in (1), determine the amount of deviations in the locators (P1 and P6) that causes the deviations of the measurement points for different parts, as presented in Table 6.5. Provide a table with the faults that you identified and their associated deviations.
2151_book.fm Page 135 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
135
TABLE 6.4 Location of the Locators and the Measurement Points X (mm)
P1 P2 P3 P4 P5 P6
(4-way) (2-way) (4-way) (2-way) (4-way) (2-way)
M1 M2 M3 M4 M5 M6 M7 M8
Locators 250 700 1078 2992 3437 3905
Z(mm)
360 365 770 812 495 503
Measurement Points 456 401.7 770 555.3 1078 770 1740 1146.4 1807 35 2456 1260 4050 800 4250 402
FIGURE 6.13 Simplified model of a sedan side frame (single-part model).
Note that: • •
It is recommended to use MatlabTM (Matlab is a registered trademark of The MathWorks, Inc.) to calculate matrices R1 and R2. To determine the deviation of the locators in (b), you should use the command pinv (pseudoinverse) in Matlab instead of inv to calculate the inverse of the matrix.
Problem 6.2 (Multiple-parts model): In this problem, consider the complete assembly process of the three parts that form the side frame presented in Figure 6.12, and the position of the locators and the measurement points presented in Table 6.4.
2151_book.fm Page 136 Friday, October 20, 2006 5:04 PM
136
Stream of Variation Modeling and Analysis for MMPs
TABLE 6.5 Observed Measurements (unit: mm) Part #
M4 (x)
M4 (z)
M6 (x)
M6 (z)
1 2 3 4 5 6 7
0.992 0.215 0 0.43 1.206 0.215 0.562
0.016 0.407 1 0.186 0.609 0.593 0.17
0.990 0.246 0 0.492 1.236 0.246 0.499
0.024 0.603 1 0.205 0.421 0.397 0.229
1. Obtain the state space model matrices (A, B for each station and matrix C for the measurement station) for the process presented in Figure 6.12. The locators used at station 1 are P1 and P2 for the fender, and P3 and P4 for the side ring. At station 2, the locators used are P1 and P4 for the subassembly and P5 and P6 for the rear quarter. At the measurement station, the locators P1 and P6 are used to hold the complete side frame. Note that in order to verify the results, consider that A0(1,1) = 1; A1(3,3) = 1; B1(3,2) = 0.0022; B2(3,2) = 0.0004; C3(16,9) = 813. Remember that the matrices A are index-shifted to 1 because no index can be 0 in Matlab. 2. Compare the following two mounting strategies presented in terms of their impact on quality (the simulation conditions are stated at the end of the problem): • First strategy: Locators P1 and P6 are used to hold the frame at the measurement station. • Second strategy: Locators P1 and P4 are used (i.e., P4 instead of P6) to hold the frame at the measurement station. Do each of the following: a. Compare the standard deviation of both strategies for each individual measurement point M1 to M8 in x and z directions (use the Matlab command std to calculate the standard deviation). b. Compare the 2-norm (sum of squared elements of a vector) of the vectors containing the standard deviations of the eight measurement points for both strategies (use the command norm in Matlab to calculate the vector norms). c. Which mounting strategy do you recommend? When solving this problem, please consider that the only source of variation in the whole process is the fixture variations at each station, which are independent and normally distributed with zero mean and standard deviation 0.0416 mm.
2151_book.fm Page 137 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
137
Deviated position B’ δZ B
Nominal position δX B B
A’
δZ A
A Z
δβ
δX A
β X
FIGURE 6.14 Part deviating from its nominal position.
APPENDIX 6.1: DETERMINATION OF THE DEVIATIONS OF A PART (LEMMA 6.1) Figure 6.14 presents a view of a part deviating from its nominal position. As the deviation is in the 2-D plane (x-z) and the part is assumed to be rigid, three parameters are required to completely characterize the deviations of the part. Those are the deviations in the x and z directions and the rotation along the y- axis (going into the plane, and henceforth considered positive in the y direction). The position of any arbitrary point B on the part can be determined based on the known position of any other arbitrary point A and the part rotation. In Figure 6.14, point A coincides with the location of the part hole. However, the position of point A can be any point on the part. Then the position of point B can be determined as: xb = x a + x ab zb = za + zab
(6.41)
where xa, za, xb, and zb are the positions of points A and B in the x- and z-axis directions (absolute position), respectively. The terms xab and zab stand for the distance between points A and B in the x and z directions and are calculated as xab = xb – xa and zab = zb – za . If the part deviates in the plane and the deviations of the reference point A are known (including part rotation), then it is possible to determine the final position of point B xb′ and zb′ as x b′ = x a′ + x ab ⋅ Cos(δβ) − zab ⋅ Sin(δβ) zb′ = za′ + x ab ⋅ Sin(δβ) + zab ⋅ Cos(δβ)
(6.42)
2151_book.fm Page 138 Friday, October 20, 2006 5:04 PM
138
Stream of Variation Modeling and Analysis for MMPs
where xa′ and za′ are the positions of point A after the deviation of the part in the x and z directions, and δβ is the part rotation. Now considering that the rotations of the part are small (less than 10˚), Equation 6.42 can be simplified (based on the linearization of the trigonometric functions) as x b′ = x a′ + x ab − zab ⋅ δβ zb′ = za′ + x ab ⋅ δβ + zab
(6.43)
By subtracting Equation 6.41 from Equation 6.43, it is possible to determine the deviations of point B with respect to its nominal position as δx b = x b′ − x b = δx a − zab ⋅ δβ δzb = zb′ − zb = δza + x ab ⋅ δβ
(6.44)
Then Equation 6.44 can be rewritten as a matrix multiplication: δxb 1 δzb = 0 δβ 0
0 1 0
δx a −( zb − za ) δx a a ,b xb − x a ⋅ δza = R 1 ⋅ δza δβ 1 δβ
where matrix R1a,b is defined as
R
a ,b 1
1 = 0 0
0 1 0
−( zb − za ) xb − x a 1
APPENDIX 6.2: EFFECT OF FIXTURE DEVIATION ON PART DEVIATION (LEMMA 6.2) Lemma 6.2 relates the part deviations or state (xi,k) as a function of the deviations of the hole and slot, which is caused by the deviations of the pins that fit into those orifices. In the current analysis, the reference point in the part where the deviations are calculated, coincides with the location of the hole. The relations between fixture or pin deviations and the part, deviations are obtained through the decomposition of effects that single deviations have on the deviation state. The case presented in Figure 6.15a corresponds to the general case where the locators of part i in station k (LP1,i,k and LP2,i,k) are not necessarily aligned along the x or z directions. Figure 6.15b presents the effects when only a deviation in the z direction of the pin that fits into the hole (LP1,i,k), has on the deviations of the part.
2151_book.fm Page 139 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
139
LP 2
LP1 ( z)
Z X
LP 1 (a)
(b)
FIGURE 6.15 Effects of the deviation of LP1 on the part (Note: In the figure, the subscripts i,k have been omitted for simplicity). (a) Case where locators are not aligned with X or Z axis. (b) Impact of LP2(z) on part orientation.
Those effects are a translation of the reference point LP1,i,k in the z direction and the rotation of the part. Figure 6.16 presents the detail of the effects that the displacement of LP1,i,k (δLP1,i,k (z)) has on the part. The part rotation δβi,k can be determined as a function of the displacement using geometric relations, as presented in Equation 6.45. The rotation is negative according to the axis defined in Figure 6.15. δβi,k ≈ −Cos (βi,k ) ⋅
δLP1,i,k ( z) LDi,k
(6.45)
The same type of analysis helps to determine the effects that deviations δLP1,i,k (x), δLP2,i,k (x), and δLP2,i,k (z) have on the rotation of the part, as presented in Equation 6.46. In this case, a deviation in the x direction of LP1,i,k and in the z direction of LP2,i,k will generate a positive rotation; deviation in the x direction of LP2,i,k will generate a negative one as presented in Equation 6.46. δβi ,k ≈ sin(β) ⋅
(δLP
1,i ,k
( x ) − δLP2,i ,k ( x ) LDi ,k
) + cos(β) ⋅ (δLP
2 ,i ,k
( z ) − δLP1,i ,k ( z ) LDi ,k
)
(6.46)
LD
LP1 ( z )
LP1 ( z)
LP1 ( z ) Cos ( )
FIGURE 6.16 Details of the effect that the deviation of LP1,i,k (z) has on the deviations of the rotation (Note: In the figure, the subscripts i,k have been omitted for simplicity).
2151_book.fm Page 140 Friday, October 20, 2006 5:04 PM
140
Stream of Variation Modeling and Analysis for MMPs
Finally, the total effect of the deviations of the reference or locating points on the deviations of the part xi,k (state) can be represented as:
x i, k
δxi,k = δzi,k = R i2,k δβi,k
δLP1,i,k ( x ) δLP1,i,k ( z) δLP2,i,k ( x ) δLP2,i,k ( z)
where matrix R2i,k is defined as:
R
i, k 2
1 = 0 sin βi,k LDi,k
0 1 cos βi,k − LDi,k
0 0 sin βi,k − LDi,k
0 0 cos βi,k LDi,k 3× 4
The first two rows of R2i,k account for the translations, and the last row accounts for the rotation. The same analysis is used to derive Corollary 6.1, which explains the effect of locating point deviations of subassemblies on the deviations of the subassembly. Corollary 6.2 explains the effect of reorientation due to errors in the previous station. This is why the terms in the matrix in Equation 6.8 are the negation of the elements in the matrix in Equation 6.6; because in this case it is a locator error (inherited from station k – 1) and not a pin error in station k.
References 1. Ceglarek, D., Shi, J., and Wu, S.M., A Knowledge-based diagnostic approach for the launch of the auto-body assembly process, ASME Transactions, Journal of Engineering for Industry, 116, pp. 491–499, 1994. 2. Shalon, D., Gossard, D., Ulrich, K., and Fitzpatrick, D., Representing geometric variations in complex structural assemblies on CAD systems, Proceedings of the 19th Annual ASME Advances in Design Automation Conference, 44, 121, 1992. 3. Ceglarek, D. and Shi, J., Dimensional variation reduction for automotive body assembly, Manufacturing Review, 8, 139, 1995. 4. Cunningham, T.W. et al., Definition, analysis, and planning of a flexible assembly process, Proceedings of the 1996 Japan/USA Symposium on Flexible Automation, 2, 767, 1996. 5. Ceglarek, D. and Shi, J., Design evaluation of sheet metal joints for dimensional integrity, ASME Transactions, Journal of Manufacturing Science and Engineering, 120, 452, 1998. 6. Shiu, B.W., Ceglarek, D., and Shi, J., Multi-Station sheet metal assembly modeling and diagnostics, NAMRI/SME Transactions, XXIV, 199, 1996.
2151_book.fm Page 141 Friday, October 20, 2006 5:04 PM
State Space Modeling for Assembly Processes
141
7. Camelio, A.J., Hu, S.J., and Ceglarek, D.J., Modeling variation propagation of multistation assembly systems with compliant parts, Transactions of the ASME, Journal of Mechanical Design, 125, 673, 2003. 8. Ding, Y., Ceglarek, D., and Shi, J., Modeling and diagnosis of multistage manufacturing processes, Proceedings of the 2000 Japan/USA Symposium on Flexible Automation, July 23–26, Ann Arbor, MI, 2000JUSFA-13146. 9. Ding, Y., Modeling and Analysis of Stream-of-Variation in Multistage Manufacturing Processes, Ph.D. dissertation, The University of Michigan, Ann Arbor, MI, 2001. 10. Jin, J. and Shi, J., State space modeling of sheet metal assembly for dimensional control, ASME Transactions, Journal of Manufacturing Science and Engineering, 121, 756, 1999.
2151_book.fm Page 142 Friday, October 20, 2006 5:04 PM
2151_book.fm Page 143 Friday, October 20, 2006 5:04 PM
7
State Space Modeling for Machining Processes*
7.1 INTRODUCTION Chapter 6 presented the variation propagation modeling for multistage assembly processes. In this chapter, stream of variation (SoV) modeling for multistage machining processes will be investigated.
7.1.1 INTRODUCTION TO MACHINING PROCESSES AND DIMENSIONAL VARIATION SOURCES Machining is a very important manufacturing process that removes materials from a workpiece to obtain design dimensional accuracy and better surface finishing, or to obtain complicated surface forms that cannot be obtained by other processes. The basic steps of machining are illustrated in Figure 7.1. The first step in a machining process is to set up the workpiece in a fixture system as illustrated in Figure 7.1b. A fixture is a device that holds a workpiece position in a particular setup and provides a means to reference and align the cutting tool to the workpiece. Proper location of the workpiece is essential to ensure accuracy and repeatability of the machining process. Figure 7.2 depicts the most popular 3-2-1 fixture system. Figure 7.2a presents the 6 degrees of freedom (d.o.f.’s) of a workpiece in the space (3 translational and 3 rotational d.o.f.’s). By requiring the workpiece to touch the 6 locators as illustrated by Figure 7.2b, the workpiece is fully constrained and the required positional and orientational relationship between the workpiece and the locators and, hence the machine tool, can be attained. The feature of the workpiece is defined as a physically identifiable portion of a workpiece, such as a plane, hole, slot, and chamfer. The features that touch the fixture locators are called datum or datum feature in machining literature. For example, the bottom plane of the cube in Figure 7.2b is the primary datum. The second step in a machining process is to clamp the workpiece on the fixture system and then mount the fixture system on the machine working table (for some dedicated machining systems, the fixture is an integrated part of the machine). One point worth mentioning is that the purpose of clamping is to fasten the workpiece to the fixture, and it does not change the location of the workpiece in space. In the last step, the cutting tool cuts the workpiece that is located on the fixture. It is worth noting that a cutting-tool path is programmed according to the working table.
* Part of chapter material is based on Reference 22 (pp. 296–309).
143
2151_book.fm Page 144 Friday, October 20, 2006 5:04 PM
144
Stream of Variation Modeling and Analysis for MMPs
(c) Machining Operation
(b) Fixturing System
(a) Workpiece
FIGURE 7.1 Basic steps in machining.
3
z axis
z axis Locator {6}
6
Locators {4,5} 5
y axis y axis 2
4 x axis 1
x axis
(a) Degree of Freedom of a Workpiece
Locators {1 2 3 }
(b) 3-2-1 Fixture System
FIGURE 7.2 Illustration of a fixture system.
A machining operation often produces multiple features on a workpiece. The dimensional accuracy of the workpiece is determined by the geometrical shape of the features and the relative positions and orientations of different features. Because of the complexity of the machining operation, the dimensional accuracy of the final product of a machining process is affected by many interrelated factors. To understand the influential factors of the dimensional accuracy, the relationships among the key elements of a machining process are illustrated in Figure 7.3. From Figure 7.3, it is clear that if the cutting-tool path is at the nominal location of the geometric feature with respect to the workpiece, then the dimensions of the resulting feature will be perfect and there will be no error. However, because it is unrealistic to measure precisely the location of each workpiece after it is mounted in the fixture, it is very difficult, if not impossible, to program the cutting-tool path with respect to the workpiece location. Therefore, in most cases it is programmed with respect to the working table of the machine. Clearly, the errors of the fixture location with respect to the working table (called fixture error) and the errors of the workpiece location with respect to the fixture (called datum error) will result in dimensional error of the features in this case. These two types of errors are often referred to as setup errors in machining processes. Besides the setup error, machine error is another important error source: the true cutting path is different from the programmed path because of various errors in the machine itself, such as geometric and dimensional errors of the kinematic links of the machine, thermal errors, wear of the cuttingtool, etc. The deviation of cutting tool path will certainly cause dimensional errors
2151_book.fm Page 145 Friday, October 20, 2006 5:04 PM
State Space Modeling for Machining Processes
145
Cutting tool Workpiece Fixture Machine Table
FIGURE 7.3 The key elements and their relationships in machining.
in the workpiece. These errors are defined as machine errors in this book. Some error sources will induce both machine error and setup error. For example, the cutting force will cause deformation in both machine and fixture. Therefore, cutting-forceinduced error is a combined error source. These aforementioned errors involved in machining operation often cause large variations in product dimensional quality and, thus, they are also called variation sources in this book. We will use “process fault,” “process error,” and “variation source” interchangeably in this book. Instead of classifying the error sources in machining processes into setup error, machine error, and combined error, these errors are also classified as quasi-static errors and dynamic errors [1]. Quasi-static errors are the static or slow-varying errors between the cutting tool and the workpiece. They include geometric and kinematic errors, thermal errors, cutting-force-induced errors, tool-wear-induced error, fixturing error, etc. Quasi-static errors account for about 70% of overall errors. Dynamic errors are fast-changing errors, caused by sources such as spindle error, machine structure vibration, controller error, etc. They are more dependent on the particular operating conditions of the machine.
7.1.2 MODELING OF VARIATION PROPAGATION MACHINING PROCESSES
IN
MULTISTAGE
The analysis and compensation of a specific type of error on a single-stage operation have been intensively studied. A huge body of literature can be found on the modeling and compensation of machine geometric errors [2–6], thermal errors [7–10], fixtureinduced errors [11–14], and force-induced errors [15–17]. However, the research on building a comprehensive model that includes multiple error sources and multiple operation stages is limited. For a multistage machining process, the product variation at a certain stage consists of two components: the variation brought by the volumetric error of current machine stage and the variation brought by the datum feature error introduced in previous stages. The second component exists because we have to use features produced by previous stages as the reference in the current operation. The variation
2151_book.fm Page 146 Friday, October 20, 2006 5:04 PM
146
Stream of Variation Modeling and Analysis for MMPs
Datum Error D
D
D
C
C
C
(a)
(b)
(c)
FIGURE 7.4 A two-step machining example.
from previous stages will be accumulated onto current operation. The variation propagation can be illustrated in a simple two-step machining example, as shown in Figure 7.4. The workpiece is a cube of metal (the front view is shown). Surface C of the workpiece is milled in the first step (Figure 7.4a). Because of an error in the milling operation, surface C is not perpendicular to surface D. A hole is drilled on surface D using surface C as one of the datum features (Figure 7.4b) in the second step. The location and orientation of the workpiece are determined by both the fixture and the datum features. Clearly, the resulting hole is not perpendicular to surface D (Figure 7.4c). The geometric error of the hole is not caused by the drilling operation; rather, it is caused by the milling operation in the first step. From this example, we have the following key observations: • •
The dimensional variation propagation and the interactions among multiple machining stages exist in machining processes in general. The key reason of variation propagation in machining processes is that the previously machined features are used as the datum in the subsequent machining operations.
From the preceding observations, we will develop a quantitative state space model to describe the variation propagation and the relationship between process error sources and product dimensional quality in multistage machining processes. It is unrealistic to include all the error sources in the model, because of the complexity of a machining process. Instead, we will focus on modeling the following error sources, which mainly account for the variation propagation: 1. Datum error refers to the dimensional and geometric error of datum features. It causes deviation of workpiece in the fixture. The effect of datum error can be obtained through a kinematics analysis based on product/process design information, such as datum feature selection, the nominal locations of datum features and new features, and fixture layout information. Datum error captures the variation propagated from previous stages, because datum features are introduced at previous stages. 2. Fixture error is caused by the errors in a fixture system, such as setup errors, wear of locating pin, clamping errors, etc. Similar to datuminduced error, the contribution of this error can be obtained by a kinematics analysis based on the product/process design information.
2151_book.fm Page 147 Friday, October 20, 2006 5:04 PM
State Space Modeling for Machining Processes
147
3. Machine error is the most complicated contributor. It involves multiple error sources, and the impact of some of those sources, such as geometric error and thermal error, can only be identified empirically. Fortunately, many of these error sources have been well studied, and effective analysis and compensation schemes have been developed [1,18]. Because the focus of the modeling is on variation propagation, the detailed modeling of certain machine errors is not included. Instead, the machine error in the model is represented as a deviation of the tool path from its nominal path. The mapping of certain machining error components to the tool path deviation can be done but is not included in this chapter for the sake of simplicity. In this chapter, a quantitative state space model will be developed to capture the combined impact of the process error sources and the dimensional variation propagation among different stages. The formulation of this model is further introduced in the following subsection.
7.1.3 MODEL FORMULATION The most challenging issue in variation analysis and reduction for complicated multistage manufacturing processes is the modeling of variation accumulation and propagation along different stages. To attack this problem, a quantitative state space model is proposed (Figure 7.5). For an N-stage process, the state space model is x k = A k −1x k −1 + B k u k + w k and y k = C k x k + v k
(7.1)
where k = 1, 2, …, N, is the stage index. The key quality characteristics of the product (e.g., dimensional deviations of key features) after each stage are represented by the state vector xk. The process faults (e.g., fixturing errors and tooling errors) are denoted by uk. Natural variation and unmodeled errors in the process are represented by a noise input to the system, wk. The product quality measurements are collected in yk. Output yk is not necessarily available at every stage. The measurement noise is denoted by vector vk. The coefficient matrices Ak, Bk, and Ck are determined by the system layout and specific characteristics of the process. Particularly, Ak represents the impact of datum error, i.e., propagated variation source, on the product quality, Bk represents the impact of the local variation sources such as fixture error on the product quality, and Ck is the measurement matrix that maps xk to yk . yk
yN
vk x0
Stage 1 x1 u1
w1
…
x k-2
Stage k-1 xk-1 uk-1
wk-1
Stage k xk uk
wk
vN … xN-1
Stage N uN
FIGURE 7.5 Diagram o