Speech Spectrum Analysis

Signals and Communication Technology

For further volumes: http://www.springer.com/series/4748

Sean A. Fulop

Speech Spectrum Analysis

123

Dr. Sean A. Fulop Department of Linguistics California State University Fresno N. Backer Ave. 5245 Fresno CA 93740-8001 USA e-mail: [email protected]

ISSN 1860-4862 ISBN 978-3-642-17477-3

e-ISBN 978-3-642-17478-0

DOI 10.1007/978-3-642-17478-0 Springer Heidelberg Dordrecht London New York Ó Springer-Verlag Berlin Heidelberg 2011 This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. The use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Cover design: eStudio Calamar, Berlin/Figueres Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

For Billy, who helped me write this.

Preface

The analysis and measurement of the spectrum of a speech signal is one of the most important areas of sound signal processing for a number of fields, yet it is not an area to which a book has been specifically devoted. The accurate determination of the speech spectrum is commonly pursued in diverse areas including speech processing, recognition, and acoustic phonetics. With this book I hope to make the subject of spectrum analysis understandable to a wide audience, which I imagine could include those with a solid background in general signal processing (but not necessarily in speech), and also speech scientists and students with some acoustic phonetics experience who have limited knowledge of signal processing. In keeping with these goals, this is not a book that replaces or attempts to cover the material found in a general signal processing textbook. Some essential signal processing concepts are presented in Chap. 2, but even there the concepts are presented in a generally understandable fashion as far as is possible. Throughout the book, the focus will be on applications to speech analysis and the measurement of important descriptive speech parameters. No attention is paid to parametrizing speech purely for coding or decorrelation for further processing. Mathematical theory will be provided for completeness, but many of these developments are set off in boxes for the benefit of those readers with sufficient background. Other readers may proceed through the main text, where the key results and applications will be presented in plain language as far as possible, and illustrated with software routines and practical ‘‘show-and-tell’’ discussions of the results. At some points, the book refers to and uses the implementations in the Praat speech analysis software package, which has the advantages that it is used by many scientists around the world, and it is free and open source software, obtainable on the internet from the Praat homepage. At other points, special software routines have been developed and made available to complement the book, and these are provided in the Matlab programming language. If the reader has the basic Matlab package, he/she will be able to immediately implement most of the programs in that platform—only Chap. 7 requires the extra Signal Processing toolbox. A few other freely available toolboxes are also needed, and all the Matlab code is made available for download at the Springer website for additional materials. vii

viii

Preface

And finally, as was written by Lord Kelvin and Professor Tait in their Treatise on Natural Philosophy (1912), ‘‘I confidently hope that few erratums of serious note will now be found in the work.’’ Fresno, October 2010

Sean Fulop

Acknowledgments

I should definitely thank Kelly Fitz for collaborating with me on developing reassignment algorithms; Doug Nelson for sharing some of his knowledge (and code) on that subject; Paul Boersma for always answering Praat-related questions; François Auger for telling me a few things about the Time–Frequency Toolbox; and Sandy Disner for helping to perfect my reassigned spectrograms. I should also thank Steven Lulich and Stefanie Shattuck–Hufnagel for trying to figure out what a voice bar is. The writing of this book was carried out over a two-year period which included summer visits to the Department of Linguistics, University of Calgary in 2009 and 2010; thanks to John Archibald, Betsy Ritter, and the other members of the department who tolerate my perennial intrusion (and get me a library card). The book was finished while I had a sabbatical at Fresno State, but was never a funded project. Thanks to Christoph Baumann at Springer for facilitating its publication.

ix

Contents

1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2

Phonetics and Signal Processing . . . . . . . . . . . . . . 2.1 Essentials of Phonetics . . . . . . . . . . . . . . . . . . 2.1.1 Speech Production Fundamentals . . . . . 2.1.2 Syllables and Speech Sounds . . . . . . . . 2.1.3 Vowels and Consonants . . . . . . . . . . . . 2.1.4 Uses of Vocal Pitch . . . . . . . . . . . . . . 2.2 Essentials of Digital Signal Processing. . . . . . . 2.2.1 Periodic and Aperiodic Signals . . . . . . . 2.2.2 Sampling of Analog Signals . . . . . . . . . 2.2.3 Autocorrelation . . . . . . . . . . . . . . . . . . 2.2.4 Fourier’s Series and Transform Spectra . 2.2.5 Practical Computing of Fourier Spectra . 2.2.6 Filters . . . . . . . . . . . . . . . . . . . . . . . . 2.2.7 Analytic Signals . . . . . . . . . . . . . . . . . 2.2.8 Concepts of Frequency . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

5 5 5 7 8 12 12 13 15 15 18 27 34 35 37 39

3

History of Speech Spectrum Analysis . . . . . . . . . . . . . 3.1 Fourier Analysis of Speech . . . . . . . . . . . . . . . . . . 3.1.1 Early History . . . . . . . . . . . . . . . . . . . . . . 3.1.2 The Physical Reality of Fourier Components 3.1.3 Recording Sound Signals . . . . . . . . . . . . . . 3.1.4 Early Methods of Fourier Analysis . . . . . . . 3.2 History of Speech Spectra . . . . . . . . . . . . . . . . . . 3.2.1 Vowels and Formants: Early Years . . . . . . . 3.2.2 Vowel Spectra: 1915–1960. . . . . . . . . . . . . 3.2.3 Spectrographic Analysis. . . . . . . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

. . . . . . . . . .

41 41 41 43 45 49 52 52 57 63

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

1 4

xi

xii

Contents

3.3 Parametric Spectral Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

65 66

4

The Fourier Power Spectrum and Spectrogram . . . . . . 4.1 The Power Spectrum in Speech Analysis . . . . . . . . . 4.1.1 Vowel Spectra . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Obstruent Spectra and Averaging Techniques . 4.1.3 Phonation Types . . . . . . . . . . . . . . . . . . . . . 4.2 Principles of the Spectrogram . . . . . . . . . . . . . . . . . 4.2.1 Definitions of the Spectrogram . . . . . . . . . . . 4.2.2 Development of Spectrogram Theory . . . . . . 4.2.3 Uncertainty Principle . . . . . . . . . . . . . . . . . . 4.3 Spectrographic Analysis of Speech . . . . . . . . . . . . . 4.3.1 General Guidelines . . . . . . . . . . . . . . . . . . . 4.3.2 Short Window (Wideband) Analysis . . . . . . . 4.3.3 Long Window (Narrowband) Analysis. . . . . . 4.4 Appendix: Praat and Matlab Techniques . . . . . . . . . 4.4.1 Praat Functions . . . . . . . . . . . . . . . . . . . . . . 4.4.2 Matlab Code. . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . .

69 69 70 75 78 80 80 87 87 88 89 92 99 102 102 104 105

5

Alternative Time–Frequency Representations . . . . . 5.1 Wigner–Ville Distribution. . . . . . . . . . . . . . . . . 5.1.1 Definition and Theory . . . . . . . . . . . . . . 5.1.2 Discrete Implementation . . . . . . . . . . . . 5.1.3 Features of the Wigner–Ville Distribution 5.2 Zhao-Atlas-Marks Distribution . . . . . . . . . . . . . 5.2.1 Quadratic Distributions . . . . . . . . . . . . . 5.2.2 Discrete Implementation . . . . . . . . . . . . 5.2.3 Speech Analysis with ZAM . . . . . . . . . . 5.3 Appendix: Matlab Routines . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

107 108 108 109 111 113 114 115 118 122 124

6

The Reassigned Spectrogram . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Reassignment: History and Definitions. . . . . . . . . . . . . . . 6.2 Reassigning the Spectrogram . . . . . . . . . . . . . . . . . . . . . 6.2.1 Nelson’s Algorithm . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Reassigned Power Spectrum . . . . . . . . . . . . . . . . . 6.3 Pruning the Reassigned Spectrogram . . . . . . . . . . . . . . . . 6.3.1 General Definitions . . . . . . . . . . . . . . . . . . . . . . . 6.3.2 Cross-Spectral Method. . . . . . . . . . . . . . . . . . . . . 6.3.3 Justifying the Interpretation of the Phase Derivative 6.3.4 Separation of Formants from Glottal Impulses . . . . 6.4 Analyzing Phonation . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

127 128 131 131 135 136 136 138 139 139 140

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

Contents

6.4.1 Beyond Source-Filter Theory . . . . 6.4.2 Observations on Phonation Types . 6.4.3 Phonation as a Biometric . . . . . . . 6.5 Dynamics of Formants . . . . . . . . . . . . . . 6.6 Nasals and Nasalization . . . . . . . . . . . . . 6.7 Stops and Fricatives. . . . . . . . . . . . . . . . 6.8 Pitch Tracking. . . . . . . . . . . . . . . . . . . . 6.9 Appendix: Matlab Techniques. . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . .

xiii

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

140 145 149 149 156 158 160 162 164

Linear Prediction and ARMA Spectrum Estimation . . . 7.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.1 Linear Filters and the z-Transform . . . . . . . . 7.1.2 Source-Filter Model of Speech . . . . . . . . . . . 7.2 Speech Spectra from Linear Prediction. . . . . . . . . . . 7.2.1 Computing the LP Coefficients. . . . . . . . . . . 7.2.2 Computing the Gain and Power Spectrum . . . 7.3 Interpretation as Filter Spectrum . . . . . . . . . . . . . . . 7.3.1 Poles and Resonances . . . . . . . . . . . . . . . . . 7.3.2 Picking Peaks Versus Solving Roots . . . . . . . 7.4 Practical Spectrum Analysis and Formant Extraction . 7.4.1 Linear Prediction Accuracy Studies. . . . . . . . 7.4.2 Analysis Windows . . . . . . . . . . . . . . . . . . . 7.4.3 Filter Order and Pre-Emphasis . . . . . . . . . . . 7.4.4 Pitch-Asynchronous Versus Closed Phase . . . 7.5 Application to Real Speech. . . . . . . . . . . . . . . . . . . 7.6 Autoregressive Moving Average Modeling . . . . . . . . 7.6.1 A Little ARMA Theory . . . . . . . . . . . . . . . . 7.6.2 ARMA Computation . . . . . . . . . . . . . . . . . . 7.6.3 Applications to Speech . . . . . . . . . . . . . . . . 7.7 Appendix: Praat and Matlab Techniques . . . . . . . . . 7.7.1 Praat Functions . . . . . . . . . . . . . . . . . . . . . . 7.7.2 Matlab Functions . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

167 168 169 171 173 173 174 175 176 179 181 181 182 184 186 188 192 192 193 194 197 197 198 200

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

203

7

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

Chapter 1

Introduction

The vibration of sound itself contains melody, harmony, and rhythm Vangelis

The quote from the composer and musician Vangelis sounds at first like an artist’s mystical musing, but really it couldn’t be more true, particularly of speech sound. When the vocal cords vibrate (as they do during ‘‘voiced’’ speech sounds), they contact each other and produce air pressure pulses in a repeating rhythm between 70–250 times each second. This rhythm is so rapid that it yields a sound in the surrounding air having the vibration rate as its fundamental frequency, which is heard as a pitched tone (melody). Moreover, the complicated mechanical nature of the vocal cord vibration gives rise to a series of harmonic frequencies in the sound, which are integer multiples of the lowest frequency. Spectrum analysis is essentially a tool for separating the melody, harmony and rhythm of a complicated sound. Its generalization to time–frequency analysis provides a view of how the sound spectrum changes through time, and such analyses are sometimes described as akin to a ‘‘musical score.’’ The spectrum of frequencies within a speech sound is the primary means by which the human auditory system can distinguish different sounds, since it is the spectrum which characterizes the distinct sound timbres or ‘‘qualities of tone.’’ The features of the spectrum also permit us to infer properties of the speech mechanism, so accurate spectrum analysis will always be an invaluable tool in speech science and linguistics. Indeed, numerous current problems in speech science can be addressed through improvements in spectrum analysis. The measurement of speech formant frequencies is the most obvious example, for which no adequate solution has yet been developed. A spectrum, in the narrowest sense, is a pure frequency representation of a sound that ignores any changes that may occur over time. Since speech is a dynamically changing signal, the analysis of the frequency spectrum as such has immediate limitations. The development of methods to show how the spectrum of a sound varies through time is commonly known as time–frequency analysis, and new techniques in this area have significant potential to improve upon the older methods which most practitioners use. Time–frequency representations with ‘‘high resolution’’ compared to the standard can be used to investigate less understood aspects of speech sound, such as the finely detailed and rapid S. A. Fulop, Speech Spectrum Analysis, Signals and Communication Technology, DOI: 10.1007/978-3-642-17478-0_1, Ó Springer-Verlag Berlin Heidelberg 2011

1

2

1 Introduction

variations of the formants and vocal cord impulses within the confines of individual glottal periods. The term spectrum has an interesting history; it was first used by scientists in the 17th century to refer to the range of colors observed from light passed through a prism. By the 19th century, light was understood as a wave phenomenon, and the spectrum of colors was known to result from a spectrum of frequencies of the waves. In the early 19th century, J. Fourier showed how a complicated function, such as a sound signal, could be represented as a number of superposed functions each with a simple frequency. During the 20th century, the range of such functions (represented by their frequencies) present within a signal, showing their relative amplitudes, came to be called the ‘‘spectrum’’ of frequencies, although I have not determined precisely when this term was first applied to sound. Early practitioners in speech science used other terminology to refer to what was essentially a frequency spectrum of a speech sound, like ‘‘harmonic analysis’’ [4] and ‘‘composite frequency analysis’’ [2], but then Lewis [3] can be found using ‘‘spectrum’’ in a matter-of-fact way in his 1936 study of vocal resonance. The present book treats speech spectrum analysis like any other complete subject, with its own foundation, history, established practice, and new frontiers. The next two chapters deal with the fundamentals and history of the subject, respectively. These chapters go beyond the usual nutshell treatments, and I hope that even highly experienced readers will find something of interest in them. Chapter 2 is specifically designed to permit readers with less mathematical experience to learn the theory behind signal processing, Fourier analysis, and digital implementation. A quick overview of phonetics is also provided, chiefly for readers from an engineering background. Chapter 4 discusses mostly time-worn techniques in spectrum analysis involving the Fourier power spectrum (a pure frequency analysis) and short-time Fourier spectrogram (a time–frequency analysis). I have found that, in spite of the age and prevalence of these methods, there is no literature which coherently and systematically outlines the many parameters which govern the analyses generated. While I expect that the treatment in Chap. 4 will be of greatest value to beginning researchers and students, I again believe that even highly experienced readers will find some tidbits of useful information about these well-known methods. Chapter 5 is the first which deals with ‘‘non-standard’’ methods of time– frequency analysis. The so-called quadratic or bilinear time–frequency representations are similar to a spectrogram but use different computational methods to obtain considerably different images; they have been introduced and studied in the signal processing literature since the 1980s. Applied researchers would benefit from understanding something about this area, but the literature tends to be inscrutable, offering little beyond theoretical equations and proofs of mathematical properties. There is no ‘‘how-to’’ literature. I digested some of the theoretical literature and scoured around for implementations by experts in the field; the fruits of my hunting are distilled and provided here tailored to the speech researcher. Some of these techniques have the potential to outperform

1 Introduction

3

conventional spectrograms, or perhaps provide a complementary view of speech sounds, and I have chosen one such bilinear representation to demonstrate. Chapter 6 introduces the method of reassigning a spectrogram, which yields a new sort of time–frequency image that has unparalleled precision in its representation of components and impulses. It achieves this in part by discarding information about the bandwidth of components, but much of this information in a spectrogram is tainted by the short-time Fourier procedure in any case. A discussion of the concept of the instantaneous frequency is included in Chap. 2, and it is especially relevant to reassigned spectrograms. Reassignment acts upon the information found in a spectrogram, changing the location of points in the time– frequency plane so that they line up more closely to the locations of the instantaneous frequencies of components, as well as to the precise time instants at which events have been recorded. It will be shown how the increased precision of the resulting time–frequency analysis of speech reveals a considerable amount of previously obscured information, leading to a variety of avenues for the investigation of hitherto unobservable properties of speech spectra. Both the theory and practice of so-called parametric spectrum estimation is described in the final chapter, which focuses largely on the method of linear prediction. This class of techniques is a double-edged sword; it yields simplified spectral analyses which are easy to mine for measurements automatically, but there is no guarantee of accurate results. Once again, my goal is to enable readers to understand how the various inputs to a linear prediction analysis affect the spectrum estimate which results, and to therefore select the best possible procedure for their purposes. The chapter concludes by introducing a promising extension of the linear prediction framework known as autoregressive moving average modeling. While I have tried to discuss all methods of speech spectrum analysis which I feel deserve detailed understanding by applied researchers in many fields, this book was never intended to be completely comprehensive like the typical book on ‘‘speech signal processing.’’ At least two major topics related to our subject have not been treated at all here, namely cepstral processing and automated tracking of either formants or pitch. The former of these, while widely used in speech engineering, has little to offer the practicing speech scientist in my opinion. Cepstral smoothing of a speech spectrum cannot show any advantages over the other methods presented, so it seemed that including a chapter on it would chiefly serve tradition rather than expedience, and would risk bloating a fairly concise book. On the other hand, a discussion of tracking algorithms has been left out because these are essentially heuristic ad hoc methods for extracting maximally consistent and reasonable results from spectral analyses. Tracking is a post hoc procedure that has no theory governing it, and which relies for its effectiveness on a high-quality spectrum analysis in the first place. In writing this book I have tried to straddle a deadly fence, which has tripped up many who have gone before. The presentation is intended to be especially useful for applied speech scientists and linguistic phoneticians, but it is also more mathematical than probably any other book with this audience in view. On the

4

1 Introduction

other hand, while it is essential to include a few of the most important equations in the discussion, most things are explained in English, and I have also tried to segregate the ‘‘heavier’’ mathematics that can be safely ignored into areas with a gray background which I refer to as ‘‘math boxes.’’ In this way, I have made an attempt at providing ‘‘everything a speech scientist wanted to know about signal processing, but was afraid to ask.’’ At the same time, I have also tried to make the book useful for engineers and scientists from a richer mathematical background. I believe it can serve this second purpose because all the relevant mathematics is presented in some way, each chapter has an extensive bibliography of primary sources, and there are many discussions of methodology specific to speech analysis which are not found in other literature. In order to make the book as practical and inviting as possible, Matlab code has been written for it which implements nearly all procedures discussed, and particularly all that cannot be performed using Praat analysis software [1]. Chapters 4, 5, 6, 7 which deal with practical methodology each conclude with an appendix detailing how Praat can be used and manipulated to perform the procedures when possible, and also detailing how the accompanying Matlab code should be used.

References 1. P. Boersma, D. Weenink, Praat: doing phonetics by computer. Computer software (2009) 2. I.B. Crandall, C.F. Sacia, A dynamical study of the vowel sounds. Bell Syst. Tech. J. III, 232–237 (1924) 3. D. Lewis, Vocal resonance. J. Acoust. Soc. Am. 8, 91–99 (1936) 4. E.W. Scripture, The Elements of Experimental Phonetics (Charles Scribner’s Sons, New York, 1902)

Chapter 2

Phonetics and Signal Processing

2.1 Essentials of Phonetics As this book is only about speech spectrum analysis, it is not intended to give a serious overview of phonetics or speech science. Most readers are probably acquainted with this subject, and so the purpose of this section is merely to present the essential concepts of phonetics as I interpret them. This will allow the definition of all the phonetic terminology to be employed throughout the book, rather than trying to rely on other authors’ definitions found throughout the literature. After all, even well-informed readers may not have read all the same books that I have, or believed all the same controversial phonetic theories, so the goal of this section is to put everyone on the same page, as it were. When the need arises, speech examples throughout the book may be presented in the standard transcription of the International Phonetic Alphabet, although knowledge of this system will rarely be critical for understanding the text. Readers who desire a reference on the IPA are invited to either visit the internet address of the International Phonetic Association, or consult a recent phonetics textbook (e.g. [26]).

2.1.1 Speech Production Fundamentals 2.1.1.1 Phonation The process of human speech production relies foremost on breathing out. The lungs expel air during speech at a carefully controlled rate (often called ‘‘speech breathing’’ [47]). The air passes through the larynx, which contains the vocal folds (often called ‘‘vocal cords,’’ with the term glottis technically referring to the space between them), whose positioning can be finely tuned by a panoply of laryngeal muscles [30]. When the vocal folds are adducted (closed) somewhat gently, then a

S. A. Fulop, Speech Spectrum Analysis, Signals and Communication Technology, DOI: 10.1007/978-3-642-17478-0_2,  Springer-Verlag Berlin Heidelberg 2011

5

6

2 Phonetics and Signal Processing

certain amount of air pressure from the lungs below (subglottal pressure) can set the folds into a self-sustaining oscillation. This condition is called phonation or voicing; and it involves a delicate balance of tissue coupling forces and aerodynamic forces whose overall description is still a current research topic; a relatively recent state of our understanding has been called the ‘‘myomucoviscoelasticaerodynamic theory’’ of phonation [30]. The length of time taken by one complete phonation cycle we may call the fundamental period of phonation; the reciprocal of this period generally determines the perceived pitch of the voice, and is called the fundamental frequency of phonation. The laryngeal muscles are able to position the vocal folds in such a wide variety of postures that a number of distinct phonation types are important in speech, and are indeed more important in some languages than in others. The most common type of phonation generally involves a vocal fold oscillation from a completely closed position (called the closed phase of phonation) to a more open state (called the open phase) and back again, and is usually termed modal phonation. It is also possible to maintain phonation with a larger adductive force acting to close the glottis. At its most extreme, such phonation is called creaky, and is marked by a very low fundamental frequency and a very small amount of air flow, both of which result from an extremely long closed phase within the phonation period. Phonation with considerably higher frequency and airflow than in creaky voice, but still with greater adductive force than modal phonation, has often been termed stiff phonation [25]. At the other end of this ‘‘scale’’ of phonation types, one may point to many instances of phonation which do not in fact exhibit any closed phase. Such phonation is termed breathy, and is caused by there being only a very small adductive force acting to close the vocal folds. In such phonation, the vocal folds vibrate more in the fashion of the double reed in a woodwind instrument such as the oboe. As one might imagine, such phonation is also characterized by a very large amount of air flow, which often creates additional aeroacoustic noise. One may also point to instances of phonation which involve considerably less adductive force than is typical of modal phonation, but which do not yet allow the vocal folds to remain open throughout the phonation period. Such phonation has often been termed slack phonation [25], and is usually characterized by a long open phase within the phonation period, together with a relatively lower fundamental frequency. In the study of languages which involve an important opposition between stiff and slack or breathy phonation, these states of the larynx have very often been called registers: It is also possible to position the vocal folds so that they are closed through approximately 60% of their length anteriorly, while being abducted at the posterior points where they attach to the arytenoid cartilages, thus producing a small triangular opening. It is this posture which is naturally used in whispering, and so it is called whisper when there is no phonation. It is also possible, however, for the anterior portion of the vocal folds to undergo a more or less modal phonation in the posture of whisper, and the superposition of this phonation upon the aeroacoustic noise of the whisper is usually called whispery phonation [8, 27].

2.1 Essentials of Phonetics

7

2.1.1.2 Sources and Filters In modern phonetics and speech science, speech production is usually conceived in the ‘‘source-filter’’ paradigm that grew out of early twentieth century acoustic speech studies. The paradigm is commonly credited to the work of Fant [11], but the idea was essentially introduced to the phonetics community by Joos [22], and had earlier precursors [28]. A speech sound can generally be described as created from a sound source whose output is modified by the vocal tract, viewed as a resonating chamber, or resonant filter (see the section below on filters). Vowels and many other voiced sounds chiefly rely on the vocal cords as a source, and the phonation output is then filtered through the prominent vocal tract resonances called formants. Many consonant sounds have their main source at some place of articulation in the vocal tract, where the release or passage of air past that point generates an aeroacoustic noise that may subsequently be filtered by the vocal tract. The filtering will be neglible in the case of consonants whose primary sound source involves articulation with the lips, since there is no need for the sound to pass through the vocal tract before it emerges. The source-filter model of speech has been very fruitful, and facilitated a greater understanding of how vocal tract resonance shapes speech sounds such as vowels. On the other hand, speech in fact involves complex and dynamic aeroacoustic sound production in which the flowing air is very important. The sourcefilter theory effectively ignores any specific effects from the airflow, and so it is not a perfect model of speech production by any means. In later chapters, we will have several opportunities to test the limits of the source-filter model, and to present findings that can lead to a more accurate understanding of speech production.

2.1.2 Syllables and Speech Sounds Although fluent speech clearly presents a continuous stream of sound, it has often been said that the smallest ‘‘units’’ of speech are syllables: That is to say, syllables are postulated as the smallest pieces of speech which are relevant for the human prodution and perception mechanisms [47, 23]. To quote a simple truism from Ladefoged, ‘‘nobody, not even a baby, can utter anything less than a syllable; he certainly cannot make a [p] or a [b] by itself’’ [23]. I present this view here because I currently subscribe to it, but it is by no means universally held. Fortunately, for the purposes of this book, the absolute truth of this idea is not particularly important, and we can safely adopt it as our methodological viewpoint. Whatever one thinks about the syllable, it is clear that the notion has never been clearly and completely defined—a fact that will not be rectified here. A syllable can be related to speech production by noting that speech inevitably involves moving the jaw, tongue, and lips from more open to more closed postures. A syllable is then approximately a single speech gesture of this kind, a movement of or within the mouth from a more closed to a more open posture (and often includes a subsequent return to a more closed posture).

8

2 Phonetics and Signal Processing

Of course, even upon beginning to define a syllable, one must immediately note that syllables can potentially be further subdivided. This is apparent from a variety of simple facts, such as the English speaker’s ability to distinguish the syllables constituting the words bee and pea; which differ only in their intial speech gestures. A speech gesture, i.e. a component of a syllable, which can be contrasted or swapped in this way with other speech gestures at particular linear positions in the speech output is what is commonly called a segment: Other roughly equivalent terms are ‘‘speech sound’’ and ‘‘phone.’’ I will at no point (except in this very sentence) make use of the theoretically loaded term ‘‘phoneme,’’ since there is no need to engage that debate in this book. Linguists commonly consider the ‘‘structure’’ of syllables, viewing them as containers for segments. The central position of a syllable, containing the main segment that gives a speaker or listener the sensation that there is a syllable, is known as the nucleus. Any segments preceding the nucleus are together called the onset; while segments following the nucleus are called the coda. For example, the English word pea includes an onset as well as the nucleus (which is obligatory for a syllable), while the word please includes a (somewhat larger) onset as well as a coda in its single syllable. Unfortunately, a proper definition of a segment is as elusive as that of a syllable. It is common to assert that a segment is some kind of subunit of syllables, but how big is it, and which gestures of speech production are included in it? For example, standard treatments of English phonetics would transcribe the one-syllable word using three segments. The notation ½ph  indicates a single plea as either ½ph li or segment whose final phase involves a sort of ‘‘h’’-sound called aspiration in the context of a stop consonant. In this sense the segment is ‘‘complex.’’ The alternative includes a symbol for a ‘‘devoiced’’ ‘l’-sound. Is the voiceless portion part of the ‘p’, or part of the ‘l’? For a further example, in the Southeast Asian language Hmong, a very similar syllable would be transcribed by linguists as ½plh i using just two segments, where the first ‘‘laterally released and aspirated’’ ‘p’ is more complex than any one of the English segments. There is, however, no way to demonstrate a purely phonetic difference between a complex segment and a sequence of simpler segments; the reason for prefering one analysis over another in a given case is a methodological one only.

2.1.3 Vowels and Consonants Phonetics has traditionally classified the segments of speech into two basic varieties which are called vowels and consonants: Once again, there has never been a straightforward definition of these terms. Early linguists in India also grappled with the concepts of vowel, consonant, and syllable around 800 BC, and they recognized that the three notions are hopelessly intertwined [1]. The definitions used here will be similar to those of the ancient Sanskrit scholars, and in fact, the development of modern phonetics in the West owes much to the transmission of knowledge in translation from the Sanskrit sources.

2.1 Essentials of Phonetics

9

A vowel is defined as a ‘‘vowel-like segment’’ (what Pike [32] termed a vocoid) that occupies the nucleus of a syllable. A segment is considered to be a vocoid when its articulation permits the relatively free passage of air through the center of the mouth. This definition is also rather loose, but in roughly familiar terms, most segments that are at least as open as an English w or y-sound (the latter is transcribed [j] in IPA) are vocoids, all others being non-vocoids. A consonant is then defined simply as a non-vocoid, no matter what syllable position it occupies. This imperfect dichotomy leaves room for a middle category, that of the semivowel, which is defined as a vocoid located outside the nucleus of a syllable. Semivowels, in spite of being vocoids, are usually regarded as a special sort of consonant (often called a ‘‘glide’’) in the interests of preserving the consonant–vowel dichotomy. The interplay of consonants, vowels, and syllables in the speech stream is given a slightly different (more acoustic) view by Orlikoff and Kahane: Consonants differ from vowels primarily by the amount of vocal tract constriction employed in their production … Speech can be considered to be an overlay of consonants on the vocal signal. The dispersion of consonants results in an amplitude modulation of the acoustic energy that, for the most part, gives rise to our perception of syllables. [30] p. 158

As so many of the speech examples shown in this book are drawn from American or Canadian English, it will be illustrative to give an inventory of the segments found in most major ‘‘standard’’ English dialects of North America, to provide a specific example while the classification of speech sounds is described. Table 2.1 lists the vowels and shows a one-syllable word using each. The one vowel not found in the list is the ‘‘schwa’’ transcribed [E], as this vowel is found only in unstressed syllables of English (e.g. about; [Ebaut]) and so is not normally present in isolated one-syllable words. Vowels are traditionally classified using a number of phonetic features which have more recently been determined to have a largely auditory basis. The feature of height (see Table 2.2) is chiefly measured by the frequency of the lowest Table 2.1 The vowels of North American English

Example word

IPA symbol

Beat Bit Bait Bet Bat Bought Bot Boat Book Boot But Bird Bite Bout Boy

[i] [ I] [eI] [e] [æ] [OE] or [A] [a] or [A] [ou] [u] [u] [V] [g] [a I] [au] [OI]

10

2 Phonetics and Signal Processing

Table 2.2 Vowels of English in the traditional chart i E

eI e æ

V a

front

Table 2.3 The stops of American English

u

high

u

I

Example word

ou OE A back

low

Purr

IPA symbol ½ph 

Rubber Upper Too

[b] [p] ½th 

Stew Patty/paddy Redo Cool

[t] [|] [d] ½kh 

School Lagoon

[k] [g]

characteristic resonance of the vowel, known as F1 ; the first formant frequency. The feature of front-back or ‘‘backness’’ is chiefly measured from the frequency of F2 ; the second formant. At this point, the discussion gets tricky, since F2 is also the chief auditory determinant of the degree of lip rounding in a vowel, which is the last of the three major vowel features. Many authors also suggest that the most important backness metric is in fact the difference F2  F1 ; but in any case the second formant is crucial. The role of F3 is not fully understood in this connection, but it is known to play at most a secondary role except in determining the rhoticity or ‘‘r’’-ness of a vowel as in bird: Because the measurement of formants is so important for characterizing vowels, we will find in later chapters that formant measurement is, and indeed has always been, one of the chief reasons for undertaking speech spectrum analysis. The consonants of a language are traditionally classified into a number of different varieties using articulatory criteria—these are the manners of articulation. Table 2.3 shows the stops found in English, which are defined as those consonants whose oral airflow is completely occluded for a brief period (and which are not nasals). A number of these segments are restricted to occur in certain environments, as is reflected in the choice of exemplifying words; for example, the voiced stops [b, d, g] normally do not initiate a word (although they may). The uniquely brief tap sound [|] is included with the other stops because of its overall similarity to them. The fricatives are defined as those consonants whose oral airflow passes through a constriction which is sufficiently narrow to yield an aeroacoustic noise

2.1 Essentials of Phonetics Table 2.4 The fricatives of American English

11 Example word

IPA symbol

Fat Vat Thin This Sip Zip Pressure Pleasure hot

[f] [v] [h] [ð] [s] [z] R ½  [Z] [h]

source. Table 2.4 gives the nine fricatives of English together with exemplifying words. It is useful to note that most treatments of English phonetics also recognize two special complex single segments [T, F], found at the beginning and end of the respective words church, judge. These combinations of a stop and a fricative are known as ‘‘affricates,’’ but in truth there is no special phonetic evidence to determine that these sound sequences are indeed single segments. It should also be pointed out that the example words in Table 2.4 illustrating the fricatives [S, Z] show them in the word-medial position. Having come this far in the description of consonants, it must be pointed out that, in addition to the place and manner of articulation, consonants are usually identified as either voiced or voiceless: A voiceless consonant is one during which the vocal cords do not phonate, while a voiced consonant is one during which phonation does take place. We often encounter pairs of consonants in a language which are distinguished only by this means, such as [f, v]. The other manners of articulation have smaller inventories in English, and so do not require tabulation. The nasals [m, n, n] are defined as consonants whose airflow is completely diverted through the nasal sinus. The words meat, neat, king are sufficient to exemplify. It is noteworthy that the third one of these cannot initiate an English word. An approximant is defined as a consonant whose articulatory constriction is slightly more open than that of a fricative, to the point that aeroacoustic noise is not produced. The only examples of such a segment in English are provided by the voiced [1], as in less, or its voiceless counterpart exemplified above in please: The remaining segments of English are the semivowels [j, w, ¤], as in yet, wet, right.1

1 It might be noted that most treatments (including the IPA itself) classify English [¤] of right as an approximant rather than a semivowel, but I believe this is not consistent with the definitions of the terms, and is mostly done because of tradition. I have to thank my colleague Chris Golston for convincing me that [¤] is a semivowel.

12

2 Phonetics and Signal Processing

2.1.4 Uses of Vocal Pitch As nearly all words in the languages of the world include a voiced vowel or approximant (though there are noteworthy exceptions [20]), a word must be spoken with a particular choice of vocal cord vibrational frequency. The fundamental frequency of phonation is usually named by the perceptual term pitch in phonetics, since the scale of perceived pitch is very nearly equal to the physical frequency scale in the range of normal speech (50–350 Hz). There are, however, sometimes reasons for saying that the perceived voice pitch is not equal to the fundamental frequency, usually because there is some kind of ‘‘abnormal’’ phonation process that makes the pitch ambiguous. In any case, it is clear that the voice pitch is very important to the linguistic phonetic pronunciation of language, acting to manifest many prosodic ‘‘above the segment’’ aspects of the speech. The overall pitch melody of an utterance is called its intonation. Different intonations are frequently used for the purpose of questions, emphasis, and many forms of speaker attitude expression. On a different note, many languages require words to have specified pitch melodies applied to each syllable; this phenomenon is called tone; with such languages being called tone languages. Finally, pitch is of considerable importance in other prosodic aspects of language besides tone; English syllable stress provides but one example, where we normally find that stressed syllables differ substantially in pitch from neighboring syllables.

2.2 Essentials of Digital Signal Processing The term signal is used to mean any quantity y that varies over time. In general, a signal could be multidimensional, but in a book about speech sound our concern is entirely with the acoustic output of the voice. This is usually measured as the varying voltage output from a microphone responding to the acoustic pressure wave of the voice, a single scalar quantity which varies as a function of time. Signals can be classified in a number of fundamental ways. A natural signal is generally continuous-time (or simply continuous), meaning that yðtÞ has values for every real number time value in the time interval through which the signal exists. This is true of the microphone voltage signal, for example. It is possible to use an analog device to perform analog signal processing on such a signal, but these approaches will only be considered in the historical review chapter. Normally the first thing that happens in modern signal processing is the sampling of a continuous signal at regular time intervals to yield a discrete or digital version of the signal that is also discrete in the y-axis showing the signal (voltage) values. In this case, the signal is represented as a sequence sðnÞ of the sample values for each whole number time sample point n; which in Matlab software is a one-dimensional matrix or vector: Another important dichotomy is the distinction between deterministic and stochastic or random signals. A random signal is formally a single realization of a

2.2 Essentials of Digital Signal Processing

13

stochastic process that generates it; each time the signal is realized, it is somewhat different owing to the randomness of the process. A deterministic signal can then simply be defined as resulting from a process that is not at all random. As the reader can already imagine, there is often sufficient randomness in human speech production to allow the treatment of speech signals as random. On the other hand, speech is never totally random. In practice, speech signal analysis treats speech as random or deterministic, whichever is most convenient for a given approach. This chapter will focus almost entirely on defining concepts for deterministic signals; spectrum analysis procedures which treat speech as a random signal will be dealt with in other chapters.

2.2.1 Periodic and Aperiodic Signals A periodic signal is one that repeats a pattern at regular time intervals. The time taken for one repetition or cycle of the pattern is called the period; and the reciprocal of the period is called the fundamental frequency of the signal. With time measured in seconds, the unit of frequency then becomes the ‘‘cycle per second’’ or Hertz, written Hz. A signal that does not have a recurrent period is an aperiodic signal. We will see that even this sort of signal can be represented as a sum or integral of periodic functions, however. Periodic signals are of great importance, not least because the most fundamental kind of signal is periodic, and is described by either of the fundamental functions of trigonometry. When a point Q moves uniformly in a circle, the perpendicular QP drawn from its position at any instant to a fixed diameter AA0 of the circle, intersects the diameter in a point P; whose position changes by a simple harmonic motion. ([36] p. 38)

A Q

P O

A’

14

2 Phonetics and Signal Processing

Then, thanks to trigonometry, the displacement s of such a point P performing a simple harmonic motion back and forth across a zero position is a trigonometric function of time: sðtÞ ¼ A cosð2pft þ /Þ;

ð2:1Þ

in which A is the maximum displacement (amplitude), f is the fundamental frequency, and / is a time offset called the phase which is included for generality (so it can be specified where we start the motion at t ¼ 0). The angle value inside parentheses is always expressed in radians rather than degrees; recall that a complete circle subtends an angle of 2p radians. Those familiar with signal processing will be well aware of its many conventions, including the convention of using the angular frequency x (radians/s) to stand for 2pf : Let us remind ourselves of Euler’s relations between sinusoidal and complex exponential functions involving exponents of the number e, also written exp½: eixt ¼ cosð2pftÞ þ i sinð2pftÞ

ð2:2Þ

A A A cosð2pftÞ ¼ eixt þ eixt 2 2

ð2:3Þ

A ixt A ixt e  e : 2i 2i

ð2:4Þ

A sinð2pftÞ ¼

It will usually prove convenient to use the complex exponential functions instead of sin and cos; since the mathematics of spectrum analysis is much easier to deal with by involving complex numbers. In the discrete-time regime, a sinusoid is expressed as a function of a set of integer time indices marking points at which the signal samples occur; the variable ranging over the time indices (or points) is normally symbolized as n: sðnÞ ¼ A cosð2pfn þ /Þ;

ð2:5Þ

where now f is a frequency in cycles per sample rather than Hz. In fact, the highest frequency that can exist in the discrete-time regime is 1=2 cycle per sample. Let us note that a periodic discrete-time signal with period N meets the condition that sðn þ NÞ ¼ sðnÞ: The total energy in a discrete-time signal is defined as the sum of the squared absolute signal values: def

E¼

1 X

j sðnÞ j2

ð2:6Þ

n¼1

Any signal whose total energy is finite is usually called an energy signal. A periodic signal represented for the entire time domain ð1; 1Þ then has to have infinite total energy.

2.2 Essentials of Digital Signal Processing

15

2.2.2 Sampling of Analog Signals In order to perform digital signal processing on a real signal it has to be sampled and quantized. This is often called analog-to-digital conversion. Sampling is the process by which values of the signal (what used to be called ordinates) are recorded at equally spaced time intervals separated by the sampling period. The reciprocal of the sampling period is the sampling frequency or sampling rate. If the signal values could be recorded with arbitrary precision when sampling was performed, then one would have a discrete-time signal with continuous ordinates. This type of signal is theoretically of importance, but in practice the signal values are also assigned to a discrete set of values. This was true even when sampling was (a long time ago) performed by hand from some kind of analog record like an oscillogram. When a digital computer is used, the signal values are assigned to a value within a discrete set by a process known as quantizing. Such a fully discrete signal is also called a digital signal. When a digital signal results from sampling an analog signal at a frequency fs (e.g. 44.1 kHz in the case of standard digital audio), the highest frequency that can be represented in the digital signal is the Nyquist frequency fmax ¼ fs =2; or 22.05 kHz in this case. This is because the highest frequency for a digital signal must be 1=2 cycle per sample. For speech, a very useful sampling rate available on most soundcards is 22,050 Hz, since this preserves the sound frequencies up to 11,025 Hz, and very little important energy is present in speech at higher frequencies than this. As a result of the above ‘‘Nyquist theorem’’ governing sampling rates and the frequencies of digital signals, it is critical to somehow filter or cut off the frequency range of an analog signal when it is sampled, so that it is bandlimited below fmax : In modern practice, the hardware A-to-D converters found in computer sound cards and digital recorders do all the hard work of bandlimiting, sampling and quantizing, so all the user generally needs to worry about is the sampling rate. In modern consumer-grade equipment, the quantizing is not user-adjustable, but it is fixed to such a high fidelity that almost no distortion will be introduced that way. As a result, no one besides a hardware engineer needs to be concerned with the particulars of amplitude quantizing; the typical computer sound card digitizes a sound using 65,536 ð216 Þ discrete amplitude steps, and professional hardware often provides more than this.

2.2.3 Autocorrelation A useful quantity that is computable from a signal is its autocorrelation, which is, roughly speaking, a function obtained by multiplying the signal sðnÞ by a timeshifted copy of itself sðn  ‘Þ; the time shift ‘ is standardly called the lag; and the autocorrelation is really a function of the lag. The subject of autocorrelation brings

16

2 Phonetics and Signal Processing

us to a terminology problem (highlighted in [3]), because it is one of the subjects where the signal processing literature intersects with literature on statistics of sampled signals (usually called time series), and these two disciplines have distinct terminological fashions and traditions. In the time series literature, what signal engineers call the autocorrelation is instead called the autocovariance, and then the autocorrelation is usually defined as the autocovariance normalized by (i.e. divided by) the particular autocovariance value for zero lag—which in turn is just the statistical variance of the signal. In this book I will feel free to follow signal processing authors in using the term ‘‘autocorrelation’’ to refer to what is strictly a form of the autocovariance, since as a practical matter it is not too important which of the various defining equations is used (see box for details). This is because we do not usually care about the actual value of the autocorrelation function, we chiefly care about the lag times where it has large peaks. Any of these quantities allows us to probe the same properties of a signal; in particular these quantities are highly sensitive to periodicity which may be present in an otherwise adulterated or noisy signal. A lag time showing a large peak in the autocorrelation function is indicative of a period in the signal equal to that lag. Intuitively, the autocorrelation takes a large value whenever the timeshifted copy of the signal matches the original well—the signal is highly correlated with itself at those lags.

In literature on time series statistics such as [33], there are two common estimates (from a finite sample) of the theoretical autocovariance of a random process, expressed as a function of lag time ‘: The first is: def

Rss ð‘Þ ¼

N 1 1 X ðsðnÞ  sÞðsðn  ‘Þ  sÞ N  ‘ n¼‘

ð2:7Þ

in which s is the sample mean of the signal, used as an estimate of the overall process mean. The above ‘‘sample autocovariance’’ is asymptotically unbiased. An alternative definition is given by: def

Rss ð‘Þ ¼

N 1 1X ðsðnÞ  sÞðsðn  ‘Þ  sÞ: N n¼‘

ð2:8Þ

This sample autocovariance is often referred to as the ‘‘periodogram-based estimate’’ (the meaning of this will be made clear in a later section). It is biased, but has smaller mean square error than the first alternative. It is this quantity that is computed by Matlab time series tools, as shown in Fig. 2.1. A related sample autocorrelation estimate can be obtained from either of the above, by simply dividing by the signal variance, which is the same thing as the autocovariance at ‘ ¼ 0:

2.2 Essentials of Digital Signal Processing

17

In the signal processing literature on the other hand, we must be aware of a number of differences in the definitions [39]. First of all, what is above called the autocovariance is generally called the autocorrelation. Moreover, the signal mean is usually not subtracted, and further there is usually no factor of N or N  ‘ put into the denominator. Finally, the resulting quantity is applied willy–nilly to random signals or deterministic ones. The ‘‘autocorrelation’’ of an energy signal sðnÞ is nowadays standardly defined in the signal processing literature as [34]: def

rss ð‘Þ ¼

1 X

sðnÞsðn  ‘Þ

ð2:9Þ

n¼1

For a finite-duration signal consisting of N samples, the definition becomes: def

rss ð‘Þ ¼

Njkj1 X

sðnÞsðn  ‘Þ;

ð2:10Þ

n¼j

in which for non-negative lags one sets j ¼ ‘; k ¼ 0; and for negative lags one sets j ¼ 0; k ¼ ‘: In turns out that this function is symmetric about zero, so it is sufficient to compute it only for non-negative lags. Example 2.1 Figure 2.1 shows a 45 ms snippet of the English vowel in the word ‘‘how’d.’’ The obvious periodicity results from the vocal cord vibration. Also shown in the figure is a plot of the autocorrelation, also known as the correlogram; produced in Matlab. This example shows how the correlogram can be used to find the main period in a periodic signal—it is equal to the first positive lag where there is a high autocorrelation value. The highest positive peak occurs at a lag of 417 samples, which Autocorrelation: a

1.2

Normalized voltage

1

0.5 0.4 0.3 0.2 0.1 0 -0.1 -0.2 -0.3 -0.4 -0.5 -0.6 0

0.8

0.6

0.4

0.2

0

5

10

15

20 25 30 Time (ms)

35

40

45

−0.2 −500

−400

−300

−200

−100

0

100

200

300

400

500

Lags

Fig. 2.1 Left panel a brief portion of a natural vowel, the first part of the English diphthong [au] in ‘‘how’d.’’ Right panel a plot of the autocorrelation of the signal at left, produced by Matlab time series tools. The lag time is shown in units of samples. Note how the function is symmetric about zero

18

2 Phonetics and Signal Processing

equals 8.14 ms for the signal’s sampling rate (51.2 kHz). This is the fundamental period of the vocal cord vibration, on average, during the 45 ms displayed.

2.2.4 Fourier’s Series and Transform Spectra Drawing on the work of predecessors Leonhard Euler and Daniel Bernoulli the elder, Jean–Baptiste Joseph Fourier devised the first proof [12] that a periodic function can be represented as a sum of elementary periodic functions (i.e. sinusoidal or their equivalent exponential functions). In this section I will present the fundamentals of Fourier’s results, which are at the heart of spectrum analysis. The treatment is drawn chiefly from [33] and [34].

2.2.4.1 Fourier’s Series Spectrum Consider a periodic signal sðtÞ in continuous time, having period T; over the entire time domain 1\t\1: Fourier’s series representation of the signal is then given by the following formula: 1 X

def

sðtÞ ¼

ck exp½2pikf0 t

ð2:11Þ

k¼1

in which f0 ¼ 1=T is the fundamental frequency of the periodic signal sðtÞ: The content of this historic equation is that a periodic function (one which also meets certain broad conditions) can be expressed as a sum of exponential (i.e. sine and cosine) ‘‘component’’ functions, and each of these has a frequency which is an integer multiple of the fundamental frequency. Components with such frequencies are called ‘‘harmonics’’ of the function. This formula is only valid for representing a signal with infinite support in the time domain, and thus Fourier’s series representation is strictly a theoretical object only. The values fck g for all integers k; called the Fourier series coefficients, represent both the amplitudes and phases of the harmonics; as such they are in general complex numbers, and are specified by the following integral formula: Z ð2:12Þ ck ¼ f0 sðtÞ exp½2pikf0 tdt T

where the interval of integration covers the signal over one period T exactly. One useful fact is that when the signal is real, then the pair of coefficients ck and ck are complex conjugates2 for all k; as a result, for real signals it is sufficient to compute

2

Recall that for complex number a þ bi; its conjugate is a  bi: For complex number Aeih expressed in polar form, its conjugate is Aeih :

2.2 Essentials of Digital Signal Processing

19

the Fourier series coefficients for non-negative integers k; meaning we do not have to bother with the negative frequencies for physical signals. The total energy in an infinite-time periodic signal sðtÞ is also infinite; the energy over a span of one period ðT=2; T=2Þ; however, is given by the integral expression: ZT=2

s2 ðtÞdt;

T=2

and a consequence of ‘‘Parseval’s relation’’ (another historic result from Fourier’s era) is that ZT=2

s2 ðtÞdt ¼ T

1 X

j ck j2 :

ð2:13Þ

k¼0

T=2

The above is an extremely important equation at the foundation of spectrum analysis; it shows that all the information about energy in a signal is present in the (squared) magnitudes of the Fourier series coefficients. The energy in a certain number of periods is usually thought of in terms of energy per unit time, or power: From the above Eq. 2.13, dividing by the time span T yields P 2 2 the total power in the signal as 1 k¼0 jck j : Then each quantity jck j (which is now real) measures the contribution to the total power from the particular term in Fourier’s series for sðtÞ with frequency k=T Hz. If we plot the y-axis values jck j2 against the xaxis values k=T; we obtain a discrete power spectrum describing how the power is distributed over the harmonic components of sðtÞ: This kind of spectrum is sometimes called a power density spectrum, but this terminology is inaccurate because the spectrum is discrete, and so is formally not a density function. In graphing the spectra of acoustic signals, it is typical to show the energy/power on a logarithmic axis that is defined using the decibel; a relative unit of power that is formally dimensionless. Representing the squared signal amplitude by jck j2 for a single Fourier series component, which would have units of squared normalized voltage in the case of a signal from a microphone, the decibel value of this amplitude is calculated relative to a reference level p2ref using the following definition: def

AdB ¼ 10 log10

jck j2 : p2ref

ð2:14Þ

When graphing a spectrum in practice, the reference level is somewhat arbitrary, and quite irrelevant when one considers that negative dB values can just as easily be shown. This means that the reference level in practice serves to determine the position of the 0 dB point on the magnitude axis, and does not at all affect the overall shape of the spectrum graph, or the relative dB amplitudes of the signal components.

20

2 Phonetics and Signal Processing

Fig. 2.2 Portion of a function (supposed to exist across the entire real line) composed of sine waves at 250 and 500 Hz frequencies

3

2

1

0

−1

−2

−3 0

0.005

0.01

0.015

0.02

0.025

Once again, let me emphasize that an infinite-time signal and its discrete spectrum are purely theoretical objects. No physical signal has a precisely discrete Fourier series spectrum. It is therefore not possible to illustrate the foregoing considerations with any real physical example or numerical computation. Let us instead concoct a simple theoretical example, in the form of an infinite-time real signal composed of two sinusoids, one at the fundamental frequency f ¼ 250 Hz and another of twice the amplitude at the next harmonic frequency 2f ¼ 500 Hz. A snippet of this function is shown in Fig. 2.2. Euler’s relation tells us that such a signal comprises four exponential components in two complex-conjugate pairs: 1 2pift 1 2pift e þ e þ ie4pift þ ie4pift 2i 2i

ð2:15Þ

c1 ¼ c1 ¼ 1=2i

ð2:16Þ

jc1 j2 ¼ jc1 j2 ¼ 1=4

ð2:17Þ

c2 ¼ c2 ¼ i

ð2:18Þ

jc2 j2 ¼ jc2 j2 ¼ 1

ð2:19Þ

sinð2pftÞ þ 2 sinð4pftÞ ¼

The end result of the math in the box is to find that the fundamental frequency component, which has half the amplitude of the double frequency component in the signal, ends up contributing 1=4 as much to the total power. On a decibel scale, we find that a component with half the amplitude is 6 dB less (v. Fig. 2.3). One also finds that using the complex exponential form of the Fourier series introduces

2.2 Essentials of Digital Signal Processing

21

1.5

10 8 6

Power

Power (dB)

1

0.5

4 2 0 −2 −4 −6 −8

0 −600

−400

−200 0 200 Frequency (Hz)

400

600

−10 −600

−400

−200

0 200 Frequency (Hz)

400

600

Fig. 2.3 Discrete power spectrum of the (complex) Fourier coefficients computed from the function shown in the previous figure. Left panel shows the raw power, right panel shows the components using a decibel scale, where the highest amplitude components have the reference level

a component with a negative frequency for every positive frequency component. Normally the negative components are ignored in applications, since they have no physical interpretation and are basically like mathematical ‘‘echoes’’ which result from working in the complex number plane. 2.2.4.2 Fourier’s Series in Discrete Time For a periodic signal in discrete-time, Fourier’s series representation can have at most N terms, or frequency components, for a signal of period N (v. [34]). The first equation below represents a discrete-time signal as a Fourier series, while the second gives the formula for the Fourier coefficients in the series. sðnÞ ¼

N1 X k¼0

ck ¼

ck exp

  2pikn N

  N 1 1X 2pikn sðnÞ exp N n¼0 N

ð2:20Þ

ð2:21Þ

Owing to the complex conjugate pair ck ¼ ck (asterisk indicates complex conjugation) for real signal sðnÞ; a further identity follows, namely jck j ¼ jcNk j: The important result of this is that now the coefficients for 0  k  N=2 (for even period N) form a complete spectrum, since the ‘‘upper half’’ of the coefficient sequence simply repeats the same information. For odd period N; one uses the coefficients for 0  k  N1 2 : So, a Fourier spectrum of a discrete-time signal is invariably redundant, and only the lower half of the coefficients are needed to represent the complete information.

22

2 Phonetics and Signal Processing

2.2.4.3 Fourier Transform and Spectrum The Fourier integral representation of a signal is, in essence, the mathematical form taken by the Fourier series representation when the signal is aperiodic. Fourier showed this when he realized that an aperiodic function could be viewed as a periodic function taken to the limit of an infinite period (see box for a derivation). Recalling Eqs. (2.11) and (2.12), one can represent a signal as its Fourier series in the following way: 0 T 1     Z 1 X 1 2pikx A 2pikt @ dx exp ð2:22Þ sðtÞ ¼ sðxÞ exp T k¼1 T T T

Define ð2pkÞ=T as an angular harmonic frequency variable x: In the limit as T ! 1; ð2pÞ=T can be taken as an infinitesimal dx: This is about the point where mathematicians squirm because of the lack of rigor in making such a manoeuver, but this derivation is found in essentially this form in many textbooks, and pretty well represents Fourier’s original derivation. It is possible to make it a more solid mathematical derivation. In any event, this ‘‘step’’ results in the following equation: 0 1 Z1 Z1 1 @ sðxÞeixx dxAeixt dx ð2:23Þ sðtÞ ¼ 2p 1

1

The above was derived by Fourier in 1811, and is now called the Fourier Integral Theorem. The inner integral defines what is now called the Fourier transform(ation) of the signal function, discussed below. For a continuous function of time (i.e. a signal) sðtÞ; the Fourier transform (rewritten in the first equation below) defined by the Fourier Integral Theorem above is a function of frequency obtained from an infinite integral over the time domain, and is often taken to provide a mathematical definition of the premathematical physical concept of ‘‘frequency.’’ def

Sðf Þ ¼

Z1

sðtÞe2pift dt

ð2:24Þ

Sðf Þe2pift df

ð2:25Þ

1

sðtÞ ¼

Z1 1

The second equation shows that the signal can in turn be expressed as a similar integral (over the frequency domain) of its own Fourier transform, which is then called the inverse Fourier transform. A function and its Fourier transform are

2.2 Essentials of Digital Signal Processing

23

commonly called a Fourier transform pair, since they are in essence transforms of each other. The Fourier transform is often expressed as a function of angular frequency x ¼ 2pf ; but this induces slight changes in the form of the transform definitions (see box for details). The Fourier transform and its inverse are commonly expressed using the def

angular frequency x ¼ 2pf ; for which there are two or three different conventions [44]. Here is one pair of equations of this sort: Z1

def

SðxÞ ¼

sðtÞeixt dt

ð2:26Þ

SðxÞeixt dx

ð2:27Þ

1

sðtÞ ¼

1 2p

Z1 1

In any such definition using x in place of f for a frequency variable, one of the integrals (it makes no difference which) must be preceded by the factor 1=2p because of the substitution of angular frequency. A third way is to pffiffiffiffiffiffi restore the symmetry of the transform pair by placing a factor of 1= 2p in front of both integrals. An aperiodic signal has a well-defined Fourier transform only if it vanishes at infinity (unlike a periodic signal which is assumed to have infinite support), and therefore has a finite total energy. Just as with R 1 a periodic signal, the total energy in the continuous-time real signal sðtÞ is 1 s2 ðtÞdt; but now the analog of Parseval’s relation leads to the following equation involving the integral transform in place of the Fourier series coefficients (cf. Eq. 2.13): Z1 1

s2 ðtÞdt ¼

Z1

jSðf Þj2 df :

ð2:28Þ

1

Although often credited to Parseval or Plancherel for the originating concepts, the above equation was derived in this particular form by Wiener [38]. The content of the equation is to say that, for aperiodic signals, the analog to the discrete spectrum of Fourier series coefficient magnitudes is now a continuous-frequency function, commonly called an energy density spectrum, which is determined by the squared magnitude of the Fourier transform. It is this equivalence which leads to the commonly expressed notion that the Fourier transform of a signal gives us its spectrum. As before, it is common to graph the spectrum as a function of frequency using a decibel-scaled axis for the magnitude. The energy density spectrum is often called the periodogram in the context of random signals and other time series data. It can also be connected directly to the

24

2 Phonetics and Signal Processing

autocovariance, since it is in fact the Fourier transform of the latter (e.g. [33]); the intrepid reader can try to verfy this by writing out the discrete Fourier transform (see below) of Eq. 2.8 (the periodogram-based estimate of the sample autocovariance). Once again, let me emphasize that the preceding treatment relates a continuoustime signal sðtÞ to its continuous-frequency Fourier transform Sðf Þ; and so none of the above facts can be computed or illustrated numerically with a computer. In order to progress to the kind of Fourier spectrum which can be, and commonly is, computed, we must move from functions which are continuous in time and frequency to functions which are discrete in time and frequency. 2.2.4.4 Discrete-Time Fourier Transform For a discrete-time function sðnÞ, the discrete-time Fourier transform is now defined as an infinite sum over the domain of time integer indices, instead of an integral over continuous time: 1 X

def

SðxÞ ¼

sðnÞeixn

ð2:29Þ

n¼1

where now x is an angular frequency value in radians/sample. If sðnÞ is sampled from a continuous signal sðtÞ; the discrete-time Fourier transform does approximate the Fourier transform of sðtÞ: The sampling causes the discrete-time Fourier transform to be periodic in the frequency domain, with period equal to the sampling frequency, which is 2p radians/sample [42]. The necessity of this periodicity in angular frequency derives from the periodicity of the exponential function, which is easily expressed [33]: exp½iðx þ 2pkÞn ¼ exp½ixn

for all integers k:

ð2:30Þ

The inverse discrete-time Fourier transform is now the following: 1 sðnÞ ¼ 2p

Zp

SðxÞeixn dx

ð2:31Þ

p

This is the same as in the continuous case, except that now the perodicity of the transform function in angular frequency allows the interval of integration to be limited to ðp; pÞ: Now that we have taken care of the time sampling, we can take the final step to define a Fourier transform which is discrete in the frequency domain as well. 2.2.4.5 Discrete Fourier Transform In applications, one cannot perform a discrete-time Fourier transform literally because it would require calculating over the entire time domain. In practice, one

2.2 Essentials of Digital Signal Processing

25

is limited to a sampled sequence having a finite length L; which is often called the window length. Then Eq. 2.29 for the discrete-time Fourier transform becomes: SðxÞ ¼

L1 X

sðnÞeixn

ð2:32Þ

n¼0

The above transform function is still impossible in digital computer applications because it is specified over a continuous range of frequencies. To represent the function digitally, we make it discrete-frequency by evaluating SðxÞ at a finite set fxk g of N equally spaced frequencies across one interval of length 2p: def 2pk

fxk g ¼

N

;

k ¼ 0; . . .; N  1:

ð2:33Þ

This makes the transform into the discrete Fourier transform (DFT), also called the finite Fourier transform [5], which is discrete in both time and frequency [42]: def

Sðxk Þ ¼

L1 X

 sðnÞ exp

n¼0

 2pikn : N

ð2:34Þ

In applications it is typical to make certain that the number of discrete frequency points N in the transform (also called the frame size) is greater than the number of signal samples L in the window, and when that is the case the transform takes the following standard form:  2pikn sðnÞ exp : Sðxk Þ ¼ N n¼0 def

N 1 X



ð2:35Þ

This form of the DFT no longer refers to the window length, but only the frame size, and so in order to ensure that the signal is not used beyond the intended window length, the signal value sðnÞ is set equal to zero for all sample points n  L: This common procedure is called ‘‘zero-padding’’ beyond the analysis window to yield an analysis frame which is longer than the window but which does not include any extra information. By doing this, the frequency sampling resolution need not be small when we analyze a short window of the signal. The downside of this standard definition is that the frequency sampling resolution has to be large when we analyze a long window, which can be inconvenient because of computation time. The DFT evaluates enough frequency components to allow its inverse transform to reconstruct the segment of the signal that was analyzed. It is sometimes interpreted as reconstructing one period of a periodic signal, assuming that exactly one period was analyzed in the first place. The following gives the inverse DFT formula: sðnÞ ¼

  N 1 1X 2pikn Sðxk Þ exp : N k¼0 N

ð2:36Þ

26

2 Phonetics and Signal Processing

The magnitude spectrum induced by the N-point DFT is just given by jSðxk Þj2 analogous to the continuous Fourier transform, and is commonly graphed as a function of frequency with a decibel-scaled magnitude axis. Now, however, the spectrum is once again in a discrete form, which is usually taken to approximate the ‘‘ideal’’ continuous form. Each discrete frequency xk or fk is referred to as a frequency bin. The reader should at this point note the similarity between the above Eq. 2.35 for the discrete Fourier transform and the earlier Eq. 2.21 for the discrete Fourier series coefficients. Modulo a multiplying factor, they are in fact the same. This shows that, for a digital approximation to a signal (resulting from time-sampling), the digital approximation to the Fourier transform (the DFT, which is also discrete in frequency) is essentially the digital version of the Fourier series [5]. Fourier’s series and transform are thus unified in the digital realm. Owing to the periodicity of the transform that was induced by the passage to discrete-time, the highest frequency that can be found in the DFT is equal to the signal’s sampling frequency fs : In addition, for a real signal the DFT exhibits a symmetry, so that Sðfk Þ ¼ S ðfNk Þmod N [41]. This means that the DFT spectrum is half-redundant, so that the upper half of the frequency bins have the same magnitude spectrum (in mirror image) as the lower half. We already obtained this result earlier, from deriving the Fourier series of a discrete-time (sampled) signal. In fact, then, the highest frequency that can be non-redundantly represented in a DFT spectrum is equal to the signal’s Nyquist frequency fs =2: It is a consequence of this redundancy that a 256-point DFT, for example, would yield a spectrum plot showing magnitudes for just the first 128 frequency bins. The other 128 simply repeat the information in reverse order, and so are not standardly shown.

2.2.4.6 Fast Fourier Transform The above formula for computing the discrete Fourier transform has been known, in essence, since the 1870s [21], and it was also recognized at an early point that it is not a tractable formula—i.e. its computational complexity is too high. In particular, for a signal frame of N samples, the simple DFT formula requires N 2 multiplications of complex numbers and NðN  1Þ complex addition operations [43]. Improving on the complexity of the basic DFT formula, a class of algorithms exists known as Fast Fourier transform algorithms. After extensive research and reinvention of these algorithms over a period of 100 years, it is now firmly established that the upper bound on the computational complexity of these methods is always on the order of N log2 N when N is a power of 2. There is, interestingly, no extant proof that this upper bound is the lowest achievable upper bound on the complexity of an exact DFT computation. Most of the algorithms are fastest when N is a power of 2, and the algorithms known as ‘‘radix 2’’ (e.g. [9]) reduce the number of complex multiplications to the order of ðN=2Þ log2 N; although these methods require that N be a power of 2. Because of the popularity of the radix 2 FFT algorithms, it has often been

2.2 Essentials of Digital Signal Processing

27

erroneously written (e.g. [24, 17]) that a fast Fourier transform can only be performed on a data frame consisting of some power of 2 samples, but this is not at all true of the other varieties of FFT algorithms [10].

2.2.5 Practical Computing of Fourier Spectra 2.2.5.1 DFT and Discrete Fourier Series The earliest computations of the discrete Fourier transform of speech signals (e.g. [21, 35]) would invariably analyze a periodic signal one period at a time. The computation over a signal window of exactly one period makes the DFT analysis provide a discrete Fourier series of a (fictitious) signal that repeats the one period over all time. Moreover, these early analysts saw no reason to alter the recorded sound signal for analysis. They just simply ‘‘clipped out’’ a signal window equal to one period from an analog record, manually measured the signal values at equally spaced sample points, and computed the discrete Fourier series. In later years, it was realized that the discrete Fourier series was constrained to find only those frequencies which are integer multiples of the fundamental, being the reciprocal of the period analyzed. The DFT can also be understood from a different perspective, since it also provides a digital approximation to the Fourier transform of a possibly aperiodic function. The DFT itself construes every function provided to it as periodic, having a period equal to the frame length. The only frequencies that can be correctly found by the DFT to have a significant amplitude are the reciprocal of the analysis frame length (which equals the frequency of the first DFT bin), and integer multiples of that—the DFT just is a discrete Fourier series, after all. This entails that a DFT energy spectrum, plotted as a graph, will look smoother for longer analysis frames—the discrete Fourier series frequencies will better ‘‘sample’’ the continuous frequency dimension. Example 2.2 This example is computed using Praat speech analysis software. Consider the following signal that was examined theoretically before: 1 1 sðtÞ ¼ sinð250  2ptÞ þ sinð500  2ptÞ; 4 2

ð2:37Þ

being a superposition of two sine functions of frequencies 250 and 500 Hz, in which the higher frequency is double the amplitude of the lower. These two are harmonics, so this signal would have a very simple exact Fourier series if it were extended for all time—only two frequencies would have amplitudes in the spectrum. The squared magnitude of the DFT (its lower-frequency half, expressed in decibels) is plotted against frequency, providing a graph of the power spectrum—it is technically power and not energy, since the DFT inherently presumes the function extends for all time although we provide only one period, and so it is assumed to have infinite total energy.

28 60 50 Magnitude (dB)

Fig. 2.4 The straight line segments connect the frequencies of just three components discovered from a DFT of one period of the sines signal. The smoother curve results from a DFT of eight periods of the same signal

2 Phonetics and Signal Processing

40 30 20 10

0

100 200 300 400 500 600 700 800 900 1000 Frequency (Hz)

Figure 2.4 shows two graphs; the one made up of straight line segments is the result of computing the DFT the ‘‘classical’’ way, using exactly one period of the signal as the analysis frame. The smoother line results from computing the DFT from an analysis frame encompassing eight periods. The first graph demonstrates what the Fourier series demands; the lowest frequency is the fundamental of the signal, and the next frequency is its harmonic, and above that there is one more frequency with negligible amplitude resulting from sampling error. The second graph is the result of computing a Fourier series using eight real signal periods as a new fundamental period. From this, the ‘‘fundamental’’ frequency will be 31.25 Hz (one eighth of the original), and all integer multiples of that will possibly have non-zero amplitude. It can be seen in the graph that, because there are nonetheless only two sinusoids, only those two ‘‘harmonics’’ of 31.25 Hz actually have large amplitudes; the small magnitudes in some other frequencies are largely a result of sampling error. What all this means is that, whenever one computes a DFT, one is in fact computing a Fourier series of a fictitious infinite-time signal that is infinitely repeating the content of the analysis frame. The analysis frame length will then automatically determine the fundamental frequency of the Fourier series (regardless of whether this is the fundamental of the signal), and this will be the frequency of the first bin in the DFT. Simply ‘‘cutting out’’ the analysis frame as I have done so far is generally not going to give pleasing results, however. It works well here because the analysis frames were equal to signal periods to within sampling error. But what if we used an analysis frame of some other length, which might cut off the signal after, say, 2.5 periods, or 8.893 periods? 2.2.5.2 Window Functions Example 2.3 Figure 2.5 shows the power spectrum that results from computing the DFT of the same signal as before, using a 34 ms analysis frame—2 ms longer than eight periods. The two component sinusoids are still the prominent peaks in

2.2 Essentials of Digital Signal Processing 70 60 Magnitude (dB)

Fig. 2.5 The power spectrum computed by a DFT of the sines signal, using a window 2 ms longer than 8 periods

29

50 40 30 20 0

100 200 300 400 500 600 700 800 900 1000 Frequency (Hz)

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

0

−0.2

−0.2

−0.4

−0.4

−0.6 −0.8

−0.6 0

200

400

600

800

1000

1200

1400

1600

−0.8 0

50

100

150

200

250

300

350

400

450

Fig. 2.6 The left panel shows the 34 ms frame clipped out of our manufactured signal consisting of two sinusoids (this contains 1,506 samples). The right panel is a close-up of the region where one instance of the frame is appended to another, as is implicitly done by the DFT computation. Note the obvious jump discontinuity, which will induce spectral leakage

the spectrum, but now they are obscured by an unwanted and uninformative broadening of the peaks at their bases. This phenomenon is often called ‘‘spectral leakage,’’ because it appears that the power at the true component frequencies has ‘‘leaked out’’ into adjacent frequencies. The cause of spectral leakage is not sampling error, it is the abrupt truncation and juxtaposition of signal frames in the fictitious periodic signal whose Fourier series we are computing, which introduces jump discontinuities at the boundaries between frames that are akin to impulses which have broad frequency content (v. Fig. 2.6). Leakage will come about whenever the frame length (34 ms, in this case) is not commensurate with the natural period of the signal. Another way to see the problem in the above example is to notice that the actual frequencies of the two signal components now lie between center frequencies of DFT bins, whereas in the preceding example the harmonic frequencies equalled (modulo sampling error) two of the bin frequencies. These two facts (one is a time-domain fact, the other a frequency-domain fact) go hand in hand, inducing leakage effects. The way to mitigate leakage is to apply a window function to the analysis window on the signal which is tapered at the ends, in a more-or-less Gaussian bell

30

2 Phonetics and Signal Processing

1 0.8 0.6 0.4 0.2 0 −0.2 −0.4 −0.6 −0.8

1 0.8 0.6 0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 0

200

400

600

800 1000 1200 1400 1600

0

200

400

600

800 1000 1200 1400 1600

Fig. 2.7 The left panel shows a 1,506-point Hann window function, and also the result of multiplying this window by the 1,506-point frame of the sines signal. The right panel shows a similar figure using a Hamming window Fig. 2.8 A 1,506-point Gaussian window, a ¼ 2:5; and the result of multiplying by the sines signal

1 0.8 0.6 0.4 0.2 0 −0.2 −0.4 −0.6 −0.8 0

200

400

600

800

1000

1200

1400

1600

shape. Normally, a window function the same length as the analysis window is created using a formula, and then the signal window and window function are multiplied to yield a tapered or ‘‘windowed’’ signal window [46] (v. Fig. 2.7). Then, the fictitious signal which endlessly repeats the ‘‘windowed’’ window has much smaller jump discontinuities at the boundaries. There are many different window functions and families of functions which have been proposed over the years; we can look at just a few examples. Two popular and effective windows use versions of a formula which derives the window from a cosine function. The Hann and Hamming windows were so named after their respective developers, although for reasons unknown to me, the Hann window came to be called by the made-up moniker ‘‘Hanning’’ as some sort of jokey allusion to ‘‘Hamming,’’ and now the funny made-up name is more widely known. The Hann and Hamming windows are demonstrated in Fig. 2.7; their formulas are given in the math box below. Another important class of windows is derived directly from the Gaussian function (see box for details). Figure 2.8 demonstrates. The reason there are so many different window functions proposed in the literature is chiefly due to the trade-off between a window’s effect on leakage reduction and its deterioration of the component precision in the power spectrum, but there are a number of other minor factors which should be included in an assessment of a window’s effectiveness in a DFT spectrum application. The reader is referred to Harris [18] for the most complete treatment of window functions I am aware of. By Harris’

31

60

60

50

50

Magnitude (dB)

Magnitude (dB)

2.2 Essentials of Digital Signal Processing

40 30 20

40 30 20 10

10 0 100 200 300 400 500 600 700 800 9001000 Frequency (Hz)

0 100 200 300 400 500 600 700 800 900 1000 Frequency (Hz)

Fig. 2.9 The left panel compares the DFT power spectra of the 1,506-point sines signal frame as computed by Praat software, windowed using the Hann function (dotted) and a Gaussian. The right panel shows a similar comparison between the Hamming (dotted) and Gaussian windowed power spectra

criteria, the Hann and Hamming windows are quite effective, but the Gaussian windows (with e.g. a ¼ 2:5) are often considerably better at leakage reduction (also known as ‘‘sidelobe rejection’’) while maintaining comparable component precision. Figure 2.9 demonstrates the differences between the Hann, Hamming, and Gaussian window effects on a DFT spectrum, where it can be seen that the Hann and Hamming spectra generally have broader peaks (which is not what is wanted in this instance). As yet another alternative, Kaiser derived a very effective window family from the zero-order modified Bessel function I0 well-known in physics [2, 40]; frequently called the Kaiser–Bessel window in homage to its source, it has been judged the best-performing window function overall [18]. Figure 2.10

50

Magnitude (dB)

40 30 20 10 0 0

100 200 300 400 500 600 700 800 900 1000 Frequency (Hz)

Fig. 2.10 Compares the power spectra of the 1506-point sines signal frame resulting from applying each of two Kaiser windows currently available in Praat. Observe how one of these (with a high value of the parameter b in the formula) fails to precisely locate the signal components. The other more moderate choice appears to be the best-performing window we have tried on this example function

32

2 Phonetics and Signal Processing

demonstrates the performance of a couple of different Kaiser window functions. Detailed formulae are given in the math box below. The Hann or ‘‘Hanning’’ window is commonly defined by the first equation below (as a function of discrete sample sequence fng, containing N samples), while the Hamming window is given by the second equation [18].   2pn def wðnÞ ¼ 0:5  0:5 cos ð2:38Þ Nþ1   2pn def ð2:39Þ wðnÞ ¼ 0:54  0:46 cos Nþ1 The family of Gaussian windows is defined using the following equation which includes a parameter a; the reciprocal standard deviation of the normal density function defined by the Gaussian. "   # 1 n 2 def ð2:40Þ wðnÞ ¼ exp  a 2 N=2 The following equation defines the Kaiser family of window functions, parameterized by b:  qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi  2 I0 pb 1  2n N 1 def wðnÞ ¼ ; ð2:41Þ I0 ðpbÞ in which I0 denotes the standard zero-order modified Bessel function [2].

Because the tapering action of any window function effectively decreases the DFT spectrum resolution (in relation to the original unwindowed signal), it is better to use a rectangular window (in other words, don’t use a window function, just clip the signal) in the event the analysis frame length can be made commensurate with the natural fundamental period of the signal. This fact is demonstrated in Fig. 2.11.

2.2.5.3 Zero-Padding the Analysis Frame In our earlier description of the discrete Fourier transform, the procedure of zero-padding the analysis frame was briefly mentioned, but it will be useful to expand on those remarks now. When a discrete-time signal is submitted to a DFT, the number of samples in the DFT frame determines two important things: the frequency sampling resolution of the resulting spectrum, and (in most

2.2 Essentials of Digital Signal Processing

33

60

Magnitude (dB)

50 40 30 20 10

0

100

200

300

400

500

600

700

800

900 1000

Frequency (Hz)

Fig. 2.11 A signal frame exactly eight periods long was cut from the sines signal, and the figure compares resulting DFT power spectra computed in Praat using a Gaussian and a rectangular window (dashes). In this case, the rectangular window is best because the signal frame length is commensurate with the natural signal period, obviating any need for a tapering window

implementations) the speed of the FFT algorithm. There can be some trade-offs involved, since oftentimes a very short analysis window on the signal is desired, but clearly the best sampling resolution in frequency will be achieved by a long DFT frame. For example, suppose I have a speech signal which is sampled at only 10 kHz, which old-timers will remember as once being quite commonly done to save on storage space. Suppose next that, for reasons which will become clear in the upcoming chapters, I wish to compute a spectrum for an 8 ms slice of the signal. These 8 ms will be represented by just 80 sample points. Suppose I simply run an FFT using this 80-point frame. Because the DFT is redundant for its upper half, only 40 frequency bins will be available for sampling the 0–5,000 Hz frequency range that is represented in the discrete signal. This is, you may easily imagine, a pretty rough frequency sampling. Next considering the FFT speed issue, which can still be quite important when spectrograms are computed (as will become apparent later), 80 is not a power of 2, so the FFT algorithm’s speed will not be optimized. It is therefore desired to be able to compute an FFT of the 80-point window using, say, a 1,024-point frame. There is an easy way to do this; simply place the (windowed) discrete signal values into the first 80 points of a 1,024-point frame, and put zeros in for the remaining 944 points. Because the zeros amount to nothing, they have a zero spectrum, so their presence does not adulterate the spectrum of the signal window. But now, we are able to compute an FFT using a frame length which is a power of 2, and moreover, we will now have 512 frequency bins sampling the 0–5,000 Hz range, which will yield a much nicer-looking discrete spectrum than 40 bins. This procedure seems like a cheap trick which is too easy to be free of defects, but in practice a zero-padded long frame can actually yield an improved look at the spectrum versus a shorter frame which exactly matches the signal window (v. Fig. 2.12), because the discontinuities that cause spectral leakage are even

34

2 Phonetics and Signal Processing

60

Magnitude (dB)

50 40 30 20 10 0

100

200

300

400

500

600

700

800

900 1000

Frequency (Hz)

Fig. 2.12 Compares the Kaiser-windowed power spectra of the 1,506-point sines signal frame (dotted) and the same windowed signal zero-padded to 2,048 points. Note that zero-padding the frame provides almost the same spectral peak shapes, but the leakage between peaks is even further ameliorated. An added advantage here is that the FFT computation using 2,048 points should be faster than that using 1,506 points

further reduced by this method.3 There are no disadvantages that I am aware of, except perhaps in the artificial cases when a window can be precisely tailored to the major periods in the signal. In spite of this, writers such as Hamming [16] can still be found who speak out against zero-padding. But Hamming’s objections seem to be concerned purely with the precise coefficients of the discrete Fourier transform, and these will of course be different with different frame lengths, zero-padded or not. Here, our concern is solely with the power spectrum which these coefficients determine.

2.2.6 Filters In signal processing, a filter is any type of system that acts on an input signal to yield an output signal, whose action changes the frequency content of the input. The action of a filter in the frequency domain is usually put in the following way [19]. Given an input signal xðtÞ and an output signal yðtÞ; let us write their respective Fourier transforms as XðxÞ, YðxÞ: The filter action, called the frequency response, can be written HðxÞ; and then the input and output spectra are related by the equation: YðxÞ ¼ HðxÞXðxÞ:

ð2:42Þ

3 The reader has probably noticed that my intentional choice of terminology in discussing these matters uses the term window to refer to the possibly tapered (windowed) slice of the signal being analyzed, while the term frame is used to refer to the discrete Fourier transform analysis length. This terminology will be used throughout the book.

2.2 Essentials of Digital Signal Processing

35

Filters are often classified according to broad aspects of their frequency response. For example, a low-pass filter is one which attenuates frequencies above a certain value. A band-pass filter is one which attenuates frequencies below and above a certain range. Filters can also be found to have more complicated responses, such as attenuating a number of different frequency ranges for the effect of a multiple band-pass. An especially important type of filter from a speech perspective is often called a resonant filter or a tuned system. Such a filter has a frequency response that has a peak at some frequency x0 ; with a pass band surrounding the peak so that the transmitted power rolls off at increasingly distant frequencies. The filter bandwidth 2Dx is specified by the half-power points [19], so that jHðx0 þ Dxj2 ¼ jHðx0 Þj2 =2 and jHðx0  Dxj2 ¼ jHðx0 Þj2 =2: The bandwidth is often cited as a def

quality factor Q which is frequency-dependent, so that Q ¼ x0 =ð2DxÞ: In speech science, a resonance frequency of the vocal tract may be characterized as a resonant filter which acts on the input sound provided by the vocal cords.

2.2.7 Analytic Signals In speech signal processing, it is only natural that we wish to deal with signals that are real-valued only, since these are the only signals that have an obvious physical meaning. In signal processing, however, it is common to deal with complex-valued signals, and we will find some use for these for the reasons to be explained now. For any real signal sðtÞ; a particular complex signal zðtÞ associated to it was originally defined by Gabor [14]: def

zðtÞ ¼ sðtÞ þ izi ðtÞ:

ð2:43Þ

Observe that the real part of zðtÞ is just sðtÞ; so what is the imaginary part for and how should it be specified? It was a matter of some importance at that time to figure out the best way of mathematically defining the phase of a signal, so that the derivative of the signal phase would correspond to the instantaneous frequency (see the next section). Gabor also noted that the spectrum of a real signal, as given by its Fourier transform, formally contains negative frequencies whose power spectrum mirrors the positive frequencies (recall Fig. 2.3). It is possible to define a complex signal which has no negative frequencies in its spectrum, and whose complex phase derivative is indeed the average frequency at each instant, thereby making it a good value to call the instantaneous frequency. Gabor defined zðtÞ so that the Fourier transform ZðxÞ ¼ 0 for all negative frequencies, while ZðxÞ ¼ 2SðxÞ for all positive frequencies, SðxÞ being the Fourier transform of the original real signal. This associated complex signal effectively takes the power from the negative frequencies and ‘‘moves it’’ to the positive side where the values are doubled (thus preserving the energy).

36

2 Phonetics and Signal Processing

It was Ville [37] who introduced the term analytic signal to refer to Gabor’s complex zðtÞ constructed from a real sðtÞ; because zðtÞ meets the mathematical definition of being an analytic complex-valued function. Gabor had also shown that the analytic signal’s imaginary part is guaranteed to be the Hilbert transform of the real part, and vice versa (see box for details). A very important raison d’être for the analytic signal, which was highlighted by Ville, is that this is the particular kind of signal for which the concept of instantaneous frequency can be provided a sensible mathematical definition as the phase derivative (see the next section). The two main properties defining the analytic signal—viz. that it has no negative frequencies, and its real and imaginary parts form a Hilbert transform pair—provide us with two means of actually defining it, which ideally come to the same end, but in the discrete-time realm they are always slightly different. This leaves us with a choice of two definitions for the ‘‘discrete analytic’’ signal (which is no longer formally analytic) that is associated to a real discrete signal sðnÞ: We could compute a second real signal zi ðnÞ by a discrete Hilbert transform of sðnÞ; def

and then simply set zðnÞ ¼ sðnÞ þ izi ðnÞ: This has been advocated as the best technique [4], but Marple [29] demonstrated that the frequency-domain method is superior. For this second method, I have implemented some recent improvements [31]; we first compute the N-point DFT of our real signal sðnÞ (of length N samples), to yield the discrete complex sequence X½m: Then we form the DFT Z½m of the associated ‘‘analytic’’ signal by setting it equal to 2X½m in the range 1  m  N=2; and zero elsewhere (with a couple of adjusted points; see the paper [31] or the Matlab file newhilbert.m for the specific algorithm). The complex ‘‘analytic’’ signal z½n is then computed from Z½m by an inverse DFT. We will have several occasions in this book to consider analytic signals instead of real ones. In the continuous-time regime, the Hilbert transform of a signal sðtÞ (which may in general be complex-valued) has the following definition [19]: def

H½sðtÞ ¼

1 p

Z1 1

2

1 H½sðtÞ ¼ lim4 p e!0

Zte 1

sðsÞ ds ts

sðsÞ ds þ ts

Z1

ð2:44Þ 3 sðsÞ 5 ds ts

ð2:45Þ

tþe

Because the integral in the defining expression is improper (the denominator has a zero), it is evaluated using its Cauchy principal value, which is a sum of two limit integrals that excludes the zero point. The Hilbert transform produces a new function that is ‘‘in quadrature’’ with the original; formally, the two are orthogonal in their function space, having an inner product of zero. Resulting from this, the Hilbert transform of

2.2 Essentials of Digital Signal Processing

37

a Hilbert transform gets us back to the original function with a minus sign: H½H½sðtÞ ¼ sðtÞ: For example, the Hilbert transform of sinðxt þ /Þ is  cosðxt þ /Þ; and the Hilbert transform of cosðxt þ /Þ is sinðxt þ /Þ [19]. Another important mathematical fact concerns the relation between the Fourier transform (spectrum) of a signal and the Fourier transform of its Hilbert transform. Let us denote the Hilbert transform of sðtÞ as ^sðtÞ; and then the Fourier transform of it into a frequency function can be written ^Sðf Þ: The following facts are presented from [13]. ^ Sðf Þ ¼ i sgnðf ÞSðf Þ

ð2:46Þ

Sðf Þ Sðf Þ ¼ i sgnðf Þ^

ð2:47Þ

in which sgnðf Þ denotes the signum function that simply returns a value 1, 1 or 0 depending upon the sign of f : This relationship explains why the analytic signal created from a real signal ends up with a one-sided Fourier transform (zero for negative frequencies). Originally, the Hilbert transform was conceived by David Hilbert in 1905, during work on a problem concerning analytic functions that is now called the Riemann–Hilbert problem [45].

2.2.8 Concepts of Frequency In the end, we perform a spectrum analysis of a signal in order to find out about the frequencies that it contains, presumably because this information can in turn lead to useful inferences about the source of the signal. The trouble is, the notion of ‘‘frequencies in a signal’’ is not rigorously defined. Indeed, the notion of ‘‘frequency’’ is somewhat colloquial, referring as it does in simple language to the rate at which some process of interest repeats or recurs. The mathematical treatment of signals, however, demands a mathematical definition of ‘‘frequency,’’ and here there are two distinct approaches that accord with the colloquial notion in differing ways. The first definition emerges directly from the Fourier series or transform representation of a signal. The ‘‘frequencies’’ in a signal are represented therein by the trigonometric (or complex exponential) component signals, each of which exists only at one frequency. This definition has served well in many applications, and there is no doubt about its mathematical correctness. The question is, does this definition suitably model the colloquial notion of frequency? A problem arises when we consider the related colloquial notion of a changing frequency. Gabor [14] reminds us that ‘‘if the term frequency is used in the strict mathematical sense which applies only to infinite wave-trains, a ‘changing frequency’ becomes a

38

2 Phonetics and Signal Processing

contradiction in terms.’’ Therein lies the rub: the Fourier components of a signal are infinite in time, meaning that strictly speaking, only an infinitely long sine wave can have just a single frequency component. But surely, we all realize that a pure sine wave of just one frequency can be started and stopped, or its frequency can be changed (modulated) in time. Do these actions somehow give it more frequencies, or can we agree there is still only one frequency in a sine wave of a changing nature? This idea provides the intuitive concept of an ‘‘instantaneous frequency’’ of a modulated sine wave. An arbitrary real signal sðtÞ can be written in terms of an instantaneous amplitude and frequency modulation: sðtÞ ¼ AðtÞ cos /ðtÞ;

ð2:48Þ

and one might imagine that by doing so, the derivative of the frequency function /ðtÞ could provide us with an instantaneous frequency for the signal. However, it turns out that the above signal decomposition is not unique [5], so that no coherent definition may be derived from it. This is a moment in the sun for the associated analytic signal, because for any (continuous-time) signal sðtÞ its analytic associate zðtÞ (see Eq. 2.43) is unique, and when expressed in complex polar form: zðtÞ ¼ As ðtÞ ei/s ðtÞ

ð2:49Þ

the pair of functions As ðtÞ; /s ðtÞ giving respectively the amplitude and complex phase modulations is the unique canonical pair associated to sðtÞ [5]. As we already saw, the real part of zðtÞ is just sðtÞ; which may now be decomposed as sðtÞ ¼ As ðtÞ cos /s ðtÞ;

ð2:50Þ

which defines the canonical representation of sðtÞ [5]. So now, the instantaneous frequency of sðtÞ can be properly defined as the derivative of the canonical frequency modulation /s ðtÞ; a definition that dates back to Carson and Fry [7] based upon Carson’s [6] earlier conception of the ‘‘generalized frequency.’’ It is important to note that while we require an analytic signal in order to define the instantaneous frequency, this is shown above to be the instantaneous frequency of the associated real signal also. Ville [37] showed, moreover, that the average frequency at each instant of a changing sinusoidal signal equals the complex phase derivative of its analytic associate. In plain language, this means that the mathematical instantaneous frequency just defined accords precisely with another intuitively defined ‘‘instantaneous’’ frequency, viz. the average sinusoidal frequency at an instant. In the end we have two useful coexisting models of the notion ‘‘frequency’’: the Fourier model, in which frequencies are formally a property only of infinite unchanging signals; and the model owing to Gabor, Ville, and Carson and Fry, in which instantaneous frequencies are defined by means of analytic signals.

References

39

References 1. W.S. Allen, Phonetics in Ancient India (Oxford University Press, Oxford, 1953) 2. G. Arfken, Mathematical Methods for Physicists, 3rd edn. (Academic Press, New York, 1985) 3. R.B. Blackman, J.W. Tukey, The Measurement of Power Spectra from the Point of View of Communications Engineering (Dover Publications, New York, 1958) 4. B. Boashash, A. Reilly, Algorithms for time–frequency signal analysis, in Time–frequency Signal Analysis: Methods and Applications, chap. 7, ed. by B. Boashash (Halsted Press, New York, 1992) 5. R. Carmona, W. L. Hwang, B. Torrésani, Practical Time–Frequency Analysis: Gabor and Wavelet Transforms, with an Implementation in S. (Academic Press, San Diego, 1998) 6. J.R. Carson, Notes on the theory of modulation. Proc. Instit. Radio Eng. 10(1), 57–64 (1922) 7. J.R. Carson, T.C. Fry, Variable frequency electric circuit theory with application to the theory of frequency-modulation. Bell Syst. Tech. J. 16(4), 513–540 (1937) 8. J.C. Catford, Fundamental Problems in Phonetics (Edinburgh University Press, Edinburgh, 1977) 9. J.W. Cooley, J.W. Tukey, An algorithm for the machine calculation of complex Fourier series. Math. Comput. 19, 297–301 (1965) 10. D.F. Elliott, K.R. Rao, Fast Transforms: Algorithms, Analyses, Applications (Academic Press, New York, 1982) 11. G. Fant, Acoustic Theory of Speech Production (Mouton, The Hague, 1960, Reissued 1970) 12. J.B.J. Fourier, Théorie de la propagation de la chaleur dans les solides. Manuscript first published in [15] (1807) 13. L.E. Franks, Signal Theory (Prentice-Hall, Englewood Cliffs, 1969) 14. D. Gabor, Theory of communication. J. IEE Part III 93(26), 429–457 (1946) 15. I. Grattan-Guinness, Joseph Fourier 1768–1830. (The MIT Press, Cambridge, 1972) 16. R.W. Hamming, Digital Filters, 3rd edn. (Prentice-Hall, Englewood Cliffs, 1989) 17. J. Harrington, S. Cassidy, Techniques in Speech Acoustics. (Kluwer, Dordrecht, 1999) 18. F.J. Harris, On the use of windows for harmonic analysis with the discrete Fourier transform. Proc. IEEE 66(1), 51–83 (1978) 19. W.M. Hartmann, Acoustic signal processing. in Springer Handbook of Acoustics, ed. by T.D. Rossing (Springer, New York, 2007) pp. 503–530 20. J.E. Hoard, Syllabication in Northwest Indian languages, with remarks on the nature of syllabic stops and affricates. in Syllables and Segments, ed. by A. Bell, J.B. Hooper (NorthHolland, Amsterdam, 1978) 21. F. Jenkin, J.A. Ewing, On the harmonic analysis of certain vowel sounds. Trans. R. Soc. Edinb. 28, 745–777 (1878) 22. M. Joos, Acoustic Phonetics (No. 23 in Language Monographs, Linguistic Society of America, Baltimore, 1948) 23. P. Ladefoged, Three Areas of Experimental Phonetics (Oxford University Press, Oxford, 1967) 24. P. Ladefoged, Elements of Acoustic Phonetics, 2nd edn. (University of Chicago Press, Chicago, 1996) 25. P. Ladefoged, Linguistic phonetic features. in The Handbook of Phonetic Sciences, ed. by W.J. Hardcastle, J. Laver (Blackwell, London, 1997) 26. P. Ladefoged, A Course in Phonetics, 5th edn. (Thomson, Boston, 2006) 27. J. Laver, The Phonetic Description of Voice Quality (Cambridge University Press, Cambridge, 1980) 28. D. Lewis, Vocal resonance. J. Acoust. Soc. Am. 8, 91–99 (1936) 29. S.L. Marple, Computing the discrete-time ‘‘analytic’’ signal via FFT. IEEE Trans. Sig. Proc. 47(9), 2600–2603 (1999)

40

2 Phonetics and Signal Processing

30. R.F. Orlikoff, J.C. Kahane, Structure and function of the larynx. in Principles of Experimental Phonetics, ed. N.J. Lass (Mosby, St. Louis, 1996) pp. 112–181 31. J.M. O’Toole, M. Mesbah, B. Boashash, A new discrete analytic signal for reducing aliasing in the discrete Wigner–Ville distribution. IEEE Trans. Sig. Proc. 56(11), 5427–5434 (2008) 32. K.L. Pike, Phonetics (The University of Michigan Press, Ann Arbor, 1943) 33. M.B. Priestley, Spectral Analysis and Time Series, vol. 1. (Academic Press, London, 1981) 34. J.G. Proakis, D.G. Manolakis, Digital Signal Processing: Principles, Algorithms, and Applications, 2nd edn. (Macmillan, New York, 1992) 35. E.W. Scripture, The Elements of Experimental Phonetics (Charles Scribner’s Sons, New York, 1902) 36. W. Thomson (Lord Kelvin), P.G. Tait, Treatise on Natural Philosophy. vol. 1, 2nd edn. (Cambridge University Press, Cambridge, 1912) 37. J. Ville, Thèorie et applications de la notion de signal analytique. Cables et Transmission 2A(1), 61–77 (1948) 38. N. Wiener, Generalized harmonic analysis. Acta Math. 55(1), 117–258 (1930) 39. Wikipedia: Autocorrelation. http://www.wikipedia.org (2009) 40. Wikipedia: Bessel function. http://www.wikipedia.org (2009) 41. Wikipedia: Discrete Fourier transform. http://www.wikipedia.org (2009) 42. Wikipedia: Discrete-time Fourier transform. http://www.wikipedia.org (2009) 43. Wikipedia: Fast Fourier transform. http://www.wikipedia.org (2009) 44. Wikipedia: Fourier transform. http://www.wikipedia.org (2009) 45. Wikipedia: Hilbert transform. http://www.wikipedia.org (2009) 46. Wikipedia: Window function. http://www.wikipedia.org (2009) 47. N.I. Zˇinkin, Mechanisms of Speech. (Mouton, The Hague, 1968)

Chapter 3

History of Speech Spectrum Analysis

This chapter traces the history of sound (and in particular, speech) spectrum analysis from its very beginnings in the theory developed by Fourier in the early 1800s. A particular goal of this historical outline is to describe not just the events and developments through the years, but also the beliefs and attitudes of scientists as these changed with the development of a better understanding. Some of the scientists whose work is discussed here are still widely known and cited, while others’ contributions have been unjustly forgotten. With this chapter, I also hope to straighten out the historical record in this respect, giving all due credit to those pioneers who uncovered many facts about speech spectra that are now taken for granted.

3.1 Fourier Analysis of Speech 3.1.1 Early History The concept of spectrum analysis is rooted in the idea of expressing one function (the signal) as a combination of other functions, each of which can be interpreted physically as some kind of frequency. The constituent functions are the basic functions of trigonometry, the sine and cosine, as shown in the previous chapter. The first mathematical work investigating this sort of decomposition of a function was probably that of Leonhard Euler, which is hardly a surprise when we remember that Euler’s work is so often important to the history of modern mathematics. In a number of publications beginning around 1748, Euler explored how some functions could be rewritten as infinite series, such as infinite series of sine or cosine functions. Euler did not seem to have frequency spectrum analysis in mind as an application, however he did seek to solve differential equations with the aid of such series. It was also Euler who related the sine and cosine, which are real functions of a real variable, to the more general complex exponential function eix which underlies our fullest understanding of spectrum analysis. S. A. Fulop, Speech Spectrum Analysis, Signals and Communication Technology, DOI: 10.1007/978-3-642-17478-0_3,  Springer-Verlag Berlin Heidelberg 2011

41

42

3 History of Speech Spectrum Analysis

Perhaps the first application of Euler’s ideas to the physics of vibration was due to Daniel Bernoulli (the elder), who published a paper in 1753 proposing a novel solution to the vibrating string equation. He founded his solution on the physical theory, known from Pythagorean times, wherein a vibrating string of length l is modeled as a superposition of all its characteristic modes of vibration, which is naturally described as a sum of simple harmonic (trigonometric) functions: f ðxÞ ¼ a sin

px 2px þ b sin þ  l l

This suggested form of the solution does indeed satisfy the differential equation for the vibrating string, the so-called wave equation which is foundational to linear acoustics. In a manuscript dated 1807, Joseph Fourier showed how to derive both a sine and a cosine series representation of an ‘‘arbitrary’’ function, which he pursued as a means of solving the differential equation governing heat transfer [23]. Fourier was also aware that the vibrating string was governed by a similar differential equation, and he remarked in the manuscript that his methods would also work to represent functional solutions to that problem, thereby providing solutions to the wave equation. The following quote is my translation of Fourier. These objections make clear how much it was necessary to show that an unspecified function can always be developed in a series of sines or cosines of multiple arcs, and of all the proofs of this proposition, the most complete is that which consists of solving effectively an arbitrary function in such a series, by assigning the values of the coefficients. The preceding theorems satisfy this condition, and I am convinced, indeed, that the movement of the vibrating string is also exactly represented in all possible cases by trigonometric developments using the integral which contains arbitrary functions [23] art. 75.

Fourier’s key contribution in this realm was not exactly the invention of the trigonometric series representation, but rather the work to establish the generality of such representation, and also his work to extend the discrete series form to an integral form capable of representing aperiodic functions. Indeed, most of the details of Fourier’s series and integral were invented independently by another giant of history, Carl Gauss. Gauss’s manuscript on what are now called Fourier’s series and integral was not published until his collected works appeared in 1866, but historians have agreed that this particular manuscript was most likely written in 1805 [28]. In spite of contemporaneous work by Gauss, and despite Fourier’s drawing upon previous publications for inspiration, his claims of such complete generality for his functional representations were greeted with skepticism. Fourier may have been convinced that his series representation was general enough to represent a very wide class of reasonable functions, but most of his contemporaries were not. Lagrange, although Fourier’s doctoral advisor, famously remained opposed to Fourier’s series until his death. The initial poor reception afforded Fourier’s work seems in part to be due to academic politics, but was also due to the low level of mathematical rigor in many of Fourier’s proofs. The wide applicability of

3.1 Fourier Analysis of Speech

43

Fourier’s series representation finally became a firmly proven result thanks to the labors of Johann P. G. L. Dirichlet, whose results on the issue were largely presented in two papers published in 1829 and 1837. In the end, the results have passed down to us as the ‘‘Dirichlet conditions’’ for the convergence of Fourier’s series. Further work on solving differential equations led Fourier from his series to what is now called the Fourier transform, by means of the Fourier Integral Theorem [26], a result he obtained in 1811 but which was not widely circulated until the publication of his book in 1822. He derived his integral representation of a function by considering the behavior of his earlier series expansions in the limit as the period approaches infinity, as was derived in the previous chapter. The Fourier transform relation and its reciprocal nature was discovered independently by yet another giant of the era, Augustin Cauchy (v. Note 19 in [9]).

3.1.2 The Physical Reality of Fourier Components The application of Fourier’s series to physical problems in which a natural signal was to be analyzed into its spectrum of harmonic sinusoids began to be explored in the late 1800s, after the initial skepticism surrounding Fourier’s results had softened. Such analyses were performed on a variety of signals, including sunspot records, tidal height records, optical (light) signals,1 and sound recordings. The results of these studies were not always universally accepted, and the ‘‘physical reality’’ of Fourier spectra was the subject of considerable debate for some time. A representative quote here comes from a paper by Godfrey published in 1900. [T]he equations of optics find their simplest solution in circular [trigonometric] functions. It is desirable to inquire how far we may resolve a natural luminous motion with a sum of simple wave-trains by means of Fourier’s ‘‘Theorem of Double Integrals.’’ This procedure was first suggested by Gouy [1886]. Doubts have often been entertained as to the permissibility of this process. Writers have been sceptical as to the physical meaning and independence of the simple waves thus introduced. ([24], p. 331)

Godfrey discussed Gouy’s [25] description of analyzing ‘‘any disturbance’’ using Fourier’s series. Gouy had considered a function defined on some interval of time; such may be analyzed into a sum of trigonometric functions of time, the periods of the terms being the interval itself and all integer ‘‘submultiples’’ (fractions 12 ; 13 ; 14 ; . . .) of it, as we have already seen. To this point, Gouy was talking of the Fourier series analysis of a periodic signal. By 1886, this procedure was already being carried out both in the analog continuous regime (using analog calculating devices) and the discrete-time regime (using hand calculation).

1 Until the early 1900s, light was not recognized as an electromagnetic particle/wave phenomenon, but rather was erroneously thought to involve waves in an otherwise undetectable substrate known as the ether:

44

3 History of Speech Spectrum Analysis

Gouy went on to suggest that it was unnecessary to restrict the analysis to periodic functions. This suggestion invokes the passage to the limit (shown in the previous chapter), by which the Fourier integral transform is understood as the limit of Fourier’s series when the fundamental period of oscillation goes toward infinity. Godfrey considered the question, prominent at the time, ‘‘have the simple elements meaning in the limit, when their number is infinite, and the sum becomes an integral?’’ [24]. He went on to mention Poincaré’s objection to Gouy’s idea, which highlighted an apparent paradox arising from Fourier’s integral representation of a signal: Each of the component vibrations exists unchanged through all time. This is true whatever be the nature of the disturbance we are analysing. But this disturbance may, for instance, be zero, except within a certain definite interval of time [24] p. 333.

In other words, there was thought to be a problem with general Fourier analysis because the aperiodic signal at issue will generally have finite support, while the sinusoidal components have to have infinite support. Godfrey attempted in his paper to explain away this paradox by some incorrect reasoning, but he was trying his best to argue that Fourier’s integral transform can provide a spectrum associated with a signal that has some physical meaning, even if ‘‘the different simple elements of the Fourier integral cannot in general be said to have any independent physical existence’’ [24]. Concerns about the physical meaning of a Fourier spectrum were aired somewhat later by Miller: If a curve representing some physical phenomenon is periodic, then each separate term of the Fourier [series] equation of the curve may be presumed to correspond to something which has a physical existence; it is the belief in this statement, amply supported by investigation, which leads one to analyze sound waves by this method; … each term is presumed to correspond to a simple partial tone which actually exists. If the curve representing the physical phenomenon is non-periodic, any portion of the curve may be analyzed, and it will be completely represented as to form by the Fourier equation, within the limits analyzed, but not beyond these limits. In this case, the separate terms of the Fourier series may not correspond to anything having a separate physical existence; … There is no general method for analyzing non-periodic curves, that is, for curves containing [inharmonic] or variable components; such a method is very much desired for the study of … all speech sounds except the simple vowels. ([40] pp. 140–141)

Note that Miller, who was a physicist investigating musical and speech acoustics, sees every possible spectrum as being the discrete kind computed from the Fourier series coefficients. While Godfrey was trying to argue in favor of physically interpreting the continuous spectrum implicit in the Fourier transform representation of an aperiodic signal, Miller seems unsympathetic to the prospect. The desire to attribute an independent physical existence to the components found by Fourier analysis has often been a powerful influence on the interpretation of spectral representations. Nevertheless, it is now recognized that the Fourier analysis of a function provides us with a different representation which is mathematically correct overall, but whose individual components may not correspond to anything physical in an obvious way. For example:

3.1 Fourier Analysis of Speech

45

If we consider a quantity which can be represented in a Fourier manner, i.e., by a superposition of monochromatic components, then it is the superposition which has a physical meaning, but not the isolated Fourier components. If we deal, for example, with a swinging chord whose motion can be described by a series of harmonics, a movie of this motion would reveal that the chord has a very complicated form at each moment, and that it varies incessantly according to a complex rule. Nothing in this motion allows us to distinguish the various monochromatic components: these components exist only in the minds of theorists who endeavor an abstract analysis of this motion ([5] as translated in [19]).

Finally, Bouasse tartly reminds us that ‘‘… unless one has lost the most elementary common sense, it is impossible to attribute an objective existence to the harmonic oscillations which emerge in the Fourier series’’ ([4], as translated in [19]).

3.1.3 Recording Sound Signals Before applying Fourier’s theorem to periodic sound signals, it was first necessary to somehow display a sound signal as a function of time. A number of

Fig. 3.1 A diagram of the phonautograph, together with waveform tracings obtained using it [40]

46

3 History of Speech Spectrum Analysis

mechanisms for graphing a ‘‘wave-form’’ (a term perhaps due to Jenkin and Ewing, see below) visualizing sound as a vertical undulation against a horizontal time axis were developed beginning in the late 1800s. The first of these was the phonautograph of Leon Scott, which was a precursor of the earliest Edison phonograph. It did not make a sound recording, but used a rudimentary transducer to drive a small lever which would trace back and forth against a turning smoked cylinder (v. Fig. 3.1). The phonograph of Thomas Edison, in its original configuration, was apparently inspired by the phonautograph since it, too, used a simple transducer to drive a lever whose stylus traced an impression on tinfoil wrapped around a turning cylinder. The phonograph record had the advantage that it could be replayed by a reading stylus that drove a simple loudspeaker, thus providing the earliest means of sound recording. The ‘‘groove’’ traced on the phonograph foil was very small, far too small to be useful as a graphical device. As a result, techniques for reproducing and magnifying the groove were devised, the first of which seems to be that of Jenkin and Ewing [30] (v. Fig. 3.2). The tin foil record on the cylinder at the upper right was read by a stylus which drove a system of levers ending in ‘‘one of Sir W. Thompson’s electrical squirting recorder tubes, which magnified the depth of the indentations 400 times,’’ [17] and thereby transmitted the waveform with ink droplets squirted onto a strip of telegraph paper wound around the large wheel W. The speed of the wheel was such as to magnify the length of the waveform seven times.

Fig. 3.2 An original figure from Jenkin and Ewing [30] diagramming their mechanism for visualizing the sound waveform recorded on a phonograph foil

3.1 Fourier Analysis of Speech

47

A particularly interesting method of sound visualization known as the manometric flame method was first described in 1872 by Koenig [32]. It does not provide a waveform, but is worth discussing in our historical chapter if only because it is so peculiar and impractical, and thus became an historical curiosity after a few decades. In spite of this, it was quite an important tool during the last

Fig. 3.3 At top, a photograph of the equipment set-up for creating manometric flames [40]. At bottom are Koenig’s meticulous drawings of the flames generated by five vowel sounds sung on each of 15 musical notes

48

3 History of Speech Spectrum Analysis

decades of the nineteenth century. The best concise description comes from Edward Scripture: The vowel is sung or spoken into a trumpet leading to a small box known as the ‘manometric capsule.’ This box is divided in two parts by a thin rubber membrane … One part is a tight chamber through which illuminating gas is flowing; the gas is lighted at the end of a small jet. As the sound waves descend they strike the rubber membrane, set it in vibration and thus produce movements of the gas analogous to those of the air in the sound waves [51] p. 26.

The rapidly changing flames were best viewed with the aid of revolving mirrors, and at first could only be recorded by hand-drawing. Figure 3.3 shows an old picture of the apparatus, together with flames produced from five French vowels sung on 15 musical notes. An important development in the recording of sound was the invention of the microphone, which according to Miller [40] was due to Hughes in 1878. The original microphone transduced sound vibrations into electromagnetic waves, which could be received by the oscillograph developed by Blondel (published in 1893 by the Paris Academy of Science). The oscillograph responded to the electromagnetic waves as a galvanometer which vibrated a tiny mirror, which in turn was used together with a light source to produce a waveform of excellent quality recorded on photographic film. Another means for recording a waveform photographically involved a device called the phonodeik, which was developed and described by Dayton Miller [40] (v. Figs. 3.4, 3.5). The sensitive receiver of the phonodeik is a diaphragm, d, of thin glass placed at the end of a resonator horn h; behind the diaphragm is a minute steel spindle mounted in jeweled bearings, to which is attached a tiny mirror m; one part of the spindle is fashioned into a small pulley; a string of silk fibers, or a platinum wire 0.0005 in. in diameter, is attached to the center of the diaphragm and being wrapped once around the pulley is fastened to a spring tension piece; light from a pinhole l is focused by a lens and reflected by the mirror to a moving film f in a special camera. If the diaphragm moves under the action of a sound wave, the mirror is rotated by an amount proportional to the motion, and the spot of light traces the record of the sound wave on the film [40] p. 79.

The resulting 5-in. wide photographs of waveforms were of the highest fidelity obtainable at the time; the apparatus was sensitive to sound frequencies up to 10 kHz. Fig. 3.4 Miller’s [40] schematic of the phonodeik

3.1 Fourier Analysis of Speech

49

Fig. 3.5 Photograph of a phonodeik [40]

3.1.4 Early Methods of Fourier Analysis In the earliest days of practical Fourier analysis, beginning in the late 1870s, investigators sought chiefly to compute the coefficients in a Fourier series that would serve to analyze a given naturally occurring periodic signal into a number of

50

3 History of Speech Spectrum Analysis

harmonics. There quickly arose two types of methods for carrying out this task: using a machine of special design to perform what was in essence an ‘‘analog computation’’ of the harmonics and their amplitudes, or manually sampling a period of the signal and computing the discrete-time Fourier series coefficients by hand calculation. The special-purpose analog computers for performing the analysis were generally called harmonic analyzers. The first such instrument was designed by Thomson (Lord Kelvin) [55] for the computation of tidal frequencies, and was described in some detail in [56]. Harmonic analyzers worked by harnessing the natural relationship between trignometric simple harmonic motion and circular motion. The given natural periodic signal had to be presented as a waveform graph and carefully traced by a stylus, and the stylus in turn transferred components of its motion to a number of cylinders or spheres, each of which would output the amplitude of its particular harmonic. A harmonic analyzer sensitive enough to analyze a traced sound waveform was devised by Henrici in 1894, and thereafter perfected (v. Fig. 3.6) and frequently used for sound spectrum analysis. The mechanisms and operating principles behind Henrici’s analyzer, as well as six other harmonic analyzers, were described in considerable detail by Carse and Urquhart [7]. Concerning the interpretation of the results provided by harmonic analyzers, which relates to the topic (discussed above) of the ‘‘reality’’ of Fourier transform components, Harry Hall [27] had this to say: Transient sounds represent continuous spectra whose amplitude and phase distributions are determined by the sound. These spectra are considered to exist throughout time, but

Fig. 3.6 Photograph of a Henrici analyzer used in Miller’s investigations [40]

3.1 Fourier Analysis of Speech

51

may be observed only during the lifetime of the original sound and are not fully indicated until the full sound has passed … If a Henrici analyzer is used to analyze selected waves taken from different parts of such a short sound, it will not give the true spectrum, but will give the line spectrum which would be produced if the particular wave in question were being repeated in a steady state.

These remarks represent an early attempt to define the notion of a ‘‘short-time’’ Fourier spectrum, meaning one which is valid only for a brief segment of a signal. We will later see how this notion was more firmly codified, so that it forms the foundation of the spectrogram. The first investigators to compute the Fourier series coefficients (and thus a discrete spectrum) for a vowel sound were Schneebeli [50], who used a phonautograph to record speech waveforms, and Jenkin and Ewing [30], who used an Edison phonograph to trace the waveforms on tin foil (see above for some details). In the latter case the waveform was naturally bandlimited by the apparatus to frequencies below approximately 1,200 Hz, and was manually sampled 12 times per period, with the vertical (ordinate) value being measured to an accuracy of 0.005 in. The periodic motion was approximated as being composed of six harmonics. Jenkin and Ewing missed their opportunity to publish the first power spectrum graphs in a simple format, and instead presented their computed spectra as rather confusing tables of numbers. In an appendix on ‘‘Practical Harmonic Analysis,’’ Carslaw [8] summarizes the established methods of the early twentieth century for performing what is, in essence, a hand calculation of the discrete-time Fourier series coefficients using samples of a periodic signal. He then notes that ‘‘Runge gave a convenient scheme for evaluating these constants in the case of 12 equidistant points’’ in a single period, which was first published in 1903 and 1905. Standard forms which facilitated the implementation of Runge’s method using either 12 or 24 sample points were made available by Whittaker and Robinson in the many editions of their book beginning in 1924 [57]. A different method of quick computation of the Fourier series coefficients, which improved upon the speed of Runge’s techniques by using an approximation, was described by Thompson [54], and was thereafter used for speech spectrum analysis by Steinberg [52]. Although Runge’s main method for computing the discrete-time Fourier series coefficients was quite efficient in exploiting trigonometric symmetries, cutting the number of required computations considerably, he had apparently given some hints which, when implemented, could further improve the efficiency of the scheme. Danielson and Lanczos [15] were seemingly the first to apply Runge’s suggestions, implementing a matrix transformation which further improved the efficiency of Runge’s more well-known scheme without resorting to approximation, effectively making the number of calculations proportional to N log N for N sample points or ‘‘ordinates’’. Their contribution was independently reinvented during the computer age, whereupon it was christened the Fast Fourier Transform [11]. So, the Fast Fourier Transform methods which render Fourier analysis relatively tractable for modern computers have surprisingly distant historical

52

3 History of Speech Spectrum Analysis

origins—the general approach arose as a means of making ‘‘practical Fourier analysis’’ more tractable for calculation by hand!

3.2 History of Speech Spectra 3.2.1 Vowels and Formants Early Years For many decades, speech science debated the question of which model of vowel production was more accurate, that of Robert Willis or that of Hermann Helmholtz. Willis [59] maintains two theses: 1. that a vowel consists of [at least] two tones, a cord tone and a mouth tone; 2. that the mouth tone is independent of the cord tone in regard to pitch. The theory of Willis was later taken up by Ludimar Hermann in the 1890s, but was criticised by Wheatstone, who supposed that the vowels arose from the vibrations of the vocal cords through the strengthening of certain overtones by the resonances of the mouth. Wheatstone’s view was expounded as a general hypothesis by Grassmann and developed into a theory by Helmholtz [51] p. 407.

This debate, in the light of present understanding, has in essence been settled in favor of both positions, with each explaining a different aspect of a vowel sound. Indeed, the debate should have settled a long time earlier thanks to an authoritative treatment of the matter by Lord Rayleigh [53]; more will be said about that below. At any rate, the earlier view propounded by Willis reflects that the vocal cords indeed produce their own ‘‘tone,’’ which is what is now called the voice source, by means of their periodic vibration. Then, as Willis had stated, each ‘‘puff’’ of air from the vocal cords excites the resonances of the mouth. We now know that the chief acoustic excitement comes from the pressure change that occurs when the cords close, rather than from the puff, which is more of an aerodynamic event than an acoustic one. A vowel sound can also be profitably understood according to Helmholtz’s view [29] that the periodic vocal cord vibration gives rise to a set of harmonics of the fundamental which are filtered by the vocal tract, thereby emphasizing certain harmonics over others. Helmholtz [29] measured what he called the ‘‘proper tones’’ of German vowels physically, by using tuning forks held up to the mouth configured for a particular vowel. He was able to find only one formant for the vowels [a, o, u], which was presumably the first formant, but he found that the front vowels [e, e, i] ‘‘have each a higher and a deeper tone.’’ It is reasonable to interpret his measurements as indicating he found the first and second formants of these vowels, which are much more widely separated than for the other three vowels. Helmholtz credits Donders [16] with the first empirical finding that ‘‘the cavity of the mouth for different vowels is tuned to different pitches’’ [29]. Donders had measured by ear the main pitch of the noise produced by whispering vowels. Lloyd later refined an ‘‘analysis-by-synthesis’’ technique in which he drove a resonance bottle using a ‘‘hissing-tube’’ that produced a noise similar to a glottal whisper.

3.2 History of Speech Spectra

53

‘‘On listening thus to the sounds issuing from the bottle it is soon found that they have often, and indeed generally, a more or less striking resemblance to whispered vowels …’’ [37]. Lloyd perfected the art of using such a device to determine formant frequencies, since the known dimensions of the adjustable resonators permitted these to be calculated, while the character of the artificial vowel could be altered to suit the experimenter’s audition of a desired sound. One of the first thorough investigations of vowels to include spectrum analysis was due to Jenkin and Ewing [30], who introduced the term wave-form for the tracing of the phonograph groove recording the sound. This term (without the hyphen) is used to this day in speech science to refer to the standard graph showing the output voltage of the microphone as a function of time. Their experiments focused on the English vowels [o] and [u] sung by six speakers. The vowels were sung on a wide range of notes of the musical scale, in order to see whether the relative amplitudes of the six partials characterizing a particular vowel would remain the same for different musical pitches. Jenkin and Ewing found that the vowel spectra did not remain the same for different pitches at all. For the vowel [o], they noted a ‘‘specially strong reinforcement’’ of whatever upper partial was nearest to a particular frequency (representing the first formant of the vowel in this case), over a large range of the fundamental frequency. They struggled with strange data from the singing of [u], and from the nature of their lengthy discussion it can reasonably be inferred that they observed many instances in which the low first formant of the vowel was in close proximity to the fundamental. Additionally they found ‘‘that the experiments have given thorough confirmation of Helmholtz’s discovery that vowel quality is not dependent on phase relation, so long as the constituent tones are unchanged.’’ Their concluding remarks include the following remarkably prescient passage: In distinguishing vowels the ear is guided by two factors, one depending on the harmony of a group of relative partials, and the other on the absolute pitch of the reinforced constituents … We are forced to the conclusion … that the ear recognises the kind of oral cavity by which the reinforcement is produced. . . The vowel-producing resonance cavities are clearly distinguished in virtue of two properties—first, the absolute pitch at which they produce a maximum reinforcement; and, second, the area of pitch over which reinforcement acts. The latter property, when it is extensive, is very probably due to the existence of subordinate proper tones not far from each other in pitch [30] pp. 772–773.

Merritt [39] developed a method for photographing manometric flames, and was able to successfully measure the fundamental frequency of a sung vowel [a] across a wide range. The fundamental and harmonics appear to correspond to the individual small flames in the photographs (v. Fig. 3.7), with the harmonic series (up to around 1,000 Hz or so) visible numerous times as a repeating pattern from left to right. He measured the first formant of vowels by noting which harmonics were emphasized by more prominent flames in the photographs. He was thus able to measure the first formant frequency of [a] as averaging 736 Hz, a value in good agreement with modern measurements. Scripture observed, in all recordings of vowels studied, at least one ‘‘tone of constant pitch—independent of the tone on which it is sung.’’ According to

54

3 History of Speech Spectrum Analysis

Fig. 3.7 Manometric flames photographed by Merritt [39], showing four vowels each sung on four notes

Scripture, this characteristic tone of a vowel was christened with the term formant by Ludimar Hermann.2 The following quote is representative of Scripture’s general understanding of speech production:

2

I have been unable to ascertain which of Hermann’s many papers first uses this term, but it was certainly published around 1890.

3.2 History of Speech Spectra

55

It seems evident that in most cases the cords act by emitting a series of more or less sudden explosions that set the air in the resonance cavities in free oscillation. The periodical changes from strong to weak in these oscillations produce the cord tone as heard, just as a series of sharp noises from a card held against a toothed wheel or puffs from a siren will produce a note. The groups of similar vibrations indicate separate puffs from the cords. . . The cavity tones in a vocal sound probably always include more than two tones. These tones change with the constantly changing shapes of the cavities and probably never remain constant.

Scripture noted that Hermann had measured only one formant for most German vowels, but had found two formants for three of the vowels. The question of how many formants were needed to characterize a vowel sound, and the manner in which they do so, was among the most important in acoustic phonetics during the late 1800s. It was Lloyd [36] who first championed the idea that more than one formant was essential for a correct account of vowel acoustics, noting that the different resonances which would be produced by speakers of differing sizes can then be understood to determine the same vowel by the prime importance of their relation, rather than their absolute frequency values. Scripture appeared to agree with Lloyd’s general ideas about vowel formants, but it took some decades before the ideas were universally held. For example, a news article which appeared in Nature in 1901 [41] virtually ignored Lloyd’s publications on vowels, while discussing at length the various theories of vowels and formants under the assumption that there is only one formant for a vowel. Scripture made his own measurements of vowel formants, but did not seem to find the values worth publishing much of the time. His materials often consisted of waveforms traced from the grooves of grammophone records (i.e. discs rather than phonograph cylinders) of natural speech. He computed the harmonic series of naturally occurring vowels and other speech sounds using the above-mentioned manually calculated discrete-time Fourier series method initiated by Jenkin and Ewing, though he employed Hermann’s specific methodology published in 1890. In his book [51], Scripture did report finding a lower formant for [a] at around 675 Hz, and also an upper formant around 1,150 Hz. Scripture was also among the first to confront the conundrum of the lowest resonance found in voiced speech, or what is now called the voice bar. He suggested that he measured some low resonances which ‘‘may be trachea tones,’’ and relied on suggestions made earlier by Pipping [45]. Pipping considered as chest tones the low ones found in the neighborhood of the note 250 [Hz] in a series of Finnish vowels. The lower resonance tone of constant pitch found in a number of cases of [a] in [ai] (‘I, eye,’ etc.) may possibly arise from the chest instead of the mouth and pharynx. . . [51] p. 294.

We will see later that the voice bar remains something of an enduring mystery that was ‘‘swept under the rug’’ for a long time. Scripture spilled some ink criticizing the Helmholtz vowel theory, and his concerns largely stemmed from a misunderstanding that seems to have been endemic in the era. Firstly, Helmholtz already recognized that if a vowel’s

56

3 History of Speech Spectrum Analysis

resonances act to reinforce harmonics emanating from the vocal cords, then the particular number of the harmonic which is emphasized must vary as the fundamental frequency varies. In his own (translated) words, ‘‘when I sing the vowel [a] on the note e[1 , the reinforced tone is. . . the 12th partial, and when I sing the same vowel on the note b2 it is the second one.’’ Secondly, Helmholtz mistakenly believed that the above fact necessitated that the mouth shape must accommodate itself to one harmonic, and when this changes to a certain extent due to changes in the fundamental, then the mouth must readjust itself to some other harmonic. This idea came to be called the ‘‘accommodation theory’’ of vowel production, and the need for such a strange aspect of speech gestures caused Scripture and others great concern. We now know that no such deliberate accommodation is necessary, because it does not matter if the formant exactly matches any particular harmonic. This is a key reason why both the Willis and Helmholtz theories of vowels are correct, and this fact was not grasped by most practitioners early on. The root of the misunderstanding was the mistaken view that a resonator had to be driven precisely at its resonance frequency in order to emphasize a harmonic component. Insofar as Scripture favored the Willis–Hermann vowel theory completely, many of his proposed modifications to the theory are essentially correct. ‘‘I would amend the Willis–Hermann view by saying that the cords emit a series of puffs whose nature may vary from the sharpest of explosions to a perfectly smooth sinusoid. I would also add … that the character of the sound emitted by the cords depends essentially on the nature of the puff.’’ Yet it is surprising that Scripture and others were still debating these matters (and would still for decades more), when any conflict between the Willis and Helmholtz vowel theories had already been shown to be illusory by Lord Rayleigh [53] in his typical correct and conclusive fashion: [Helmholtz’s] view of the action of a resonator is of course perfectly correct; but at first sight it may appear essentially different from, or even inconsistent with, the account of the matter given by Willis. For example, according to the latter the mouthtone may be, and generally will be, inharmonic as regards the larynx-tone. … Although the natural vibrations of the oral cavity may be inharmonic, the forced vibrations can include only harmonic partials of the larynx note … From these considerations it will be seen that both ways of regarding the subject are legitimate and not inconsistent with one another [53].

That the treatment of vowels in the second edition of Rayleigh’s magnum opus Theory of Sound,3 now probably the most currently read physics book from the nineteenth century, could have been so perfectly overlooked by an entire community of speech investigators from all backgrounds is surely one of the most nagging historical puzzles connected with speech analysis. Alas, the resolution of this puzzle today must be left for historians of science.

3

The treatment of vowels was apparently not in the first edition of 1878, though I have not seen this version firsthand.

3.2 History of Speech Spectra

57

3.2.2 Vowel Spectra 1915–1960 Miller [43] obtained vowel waveforms using his phonodeik described above, and then obtained harmonic spectra from the waveforms with a specially constructed Henrici analyzer. His results were originally published in 1916. By drawing smooth curves over the harmonic spectra (v. Fig. 3.8) he was able to locate the first two formants of most of the English front vowels, and the values he obtained for F1 at least are generally plausible. Owing to deficiencies of his equipment, Miller could find only the first formant of English back vowels, thereby propagating the earlier conclusions of Helmholtz and Scripture, to the effect that back vowels generally have one important formant and front vowels have two. Miller disagreed with Scripture’s position on the vowel theories, and advocated the Helmholtz view as the most in accord with his experimental results. In contrast to many precursors, Miller apparently understood that the application of this theory required no special ‘‘accommodation’’ of the mouth to match the formant to a harmonic, but still the seeming ignorance of Lord Rayleigh’s resolution of the matter is puzzling in the light of Miller’s physics background. Significant advances in speech spectrum analysis were made by Crandall and Sacia at Bell Labs, and published in a series of papers [12–14]. They used an improved oscillograph to obtain photographic records of speech waveforms, and then obtained Fourier spectra using a photomechanical harmonic analyzer designed by Sacia [48]. Although Crandall and Sacia began with the assumption (taken from Miller and also Fletcher [20]) that English back vowels were characterized by a single formant, they obtained more accurate results than any previous, and ultimately Crandall realized that all the vowels they recorded showed at least two important formants [13]. In fact, they also discovered the importance of the low F3 in the English rhotic vowel [g], and related its appearance to the retroflex tongue posture. They interpreted their results as favoring a ‘‘harmonic’’ theory of vowel production (i.e. the Helmholtz view), but also suggested that the inharmonic theory (i.e. Willis–Hermann) may also be needed as a complement to it, pointing to different decay properties of formants which were transiently excited by each vocal cord cycle (as Willis had originally asserted). The discussion by Crandall and Sacia is the first demonstrated understanding in the literature that, not only are the two models of vowel production equally correct, but that each model is needed to understand the disparate aspects of speech production. They also hold the distinction of being the first to publish power spectrum graphs of speech sounds in the format commonly used to the present day (v. Fig. 3.9). Steinberg [52] analyzed vowels and other speech sounds by a discrete ‘‘fast’’ approximate Fourier transform computation owing to Thompson [54], and described the vowel spectra as consisting of harmonic partials which showed at least two regions of reinforced amplitude, one below 1,000 Hz and one between 1,000 and 2,000 Hz. Clearly he was able to locate the first and second formants by examining the spectra of harmonics. To record speech, Steinberg used a condenser microphone and an oscillograph which had a flat response up to 10 kHz to create an

58

3 History of Speech Spectrum Analysis

Fig. 3.8 Vowel spectra from Miller’s book [40]. Characteristics curves for the distribution of the energy in vowels of Class I, having a single region of resonance. Characteristics curves for the distribution of the energy in vowels of Class II, having two regions of resonance

3.2 History of Speech Spectra

59

Fig. 3.9 Power spectrum graphs from Crandall [12] for 13 English vowels

oscillograph record of the waveform of the sentence ‘‘Joe took father’s shoe bench out.’’ Individual periods of the waveform were analyzed by a (hand-calculated) discrete-time Fourier series using up to 60 sample points over a single period. The power spectra thus calculated did reveal formants, and were laid out in a sequence to show the changes in the spectrum over time—Steinberg was the first to publish timevarying spectra, and his display technique is truly a forerunner of the spectrogram (v. Fig. 3.10). Steinberg presented his formant measurements for all the vowels in the utterance, and compared his values to those obtained previously by Crandall and Sacia, and also to those obtained by Paget [43] through analysis of synthetic vowels produced with a physical vocal tract model. His analysis method did not have a frequency resolution sufficient to resolve closely spaced formants of [u], [u], which had been successfully resolved by Paget using synthetic speech.

60

3 History of Speech Spectrum Analysis

Fig. 3.10 A figure from Steinberg [52] showing a time sequence of power spectra. The formant peaks are hand-drawn over the computed discrete spectra

The next big advance in speech spectrum analysis was made by Lewis [35], who made oscillographic records of about a half-second long from a variety of vowels in sustained sung notes. He then traced the oscillographs over selected periods with a 40-component Henrici analyzer, yielding, approximately, the

3.2 History of Speech Spectra

61

Fourier series spectrum of the first 40 harmonics. The singer had been instructed to sing the vowels using his normal vibrato technique, with the result that different periods of the oscillograph had slightly different fundamental frequencies. For each vowel, a number of periods at different frequencies were selected for analysis, with the goal of checking whether the voice source and ‘‘filter’’ were independent. Indeed, Lewis published graphs showing the resulting vowel spectra which were created by overlaying the harmonic analyses from a number of different phonation periods at different fundamental frequencies (v. Fig. 3.11). No matter the

Fig. 3.11 A figure from Lewis [35] showing vowel power spectra drawn with a harmonic analyzer. Formant peaks are hand-drawn over the computed spectra

62

3 History of Speech Spectrum Analysis

particular frequencies of the harmonics, their spectral amplitudes followed the same overall outline, which Lewis took to be the resonance spectrum of the vocal tract. His spectra showed frequency components up to 4 kHz, and four or five formants were clearly visible for each vowel. F1 and F2 for [a] were measured to be 675 and 1,150 Hz, respectively. Other vowels’ formant values are equally plausible by modern standards. Lewis’s landmark study was the first to demonstrate that there are generally five formants for any vowel, and also that the amplitude differences between formants play an important role in the vowel quality. He imagined that each formant was the output of a simple resonator represented in the vocal tract somehow. He also proposed the prescient conjecture that ‘‘the typical quality of a vowel is determined by the two resonators of lowest frequency, with individual voice differences resulting from the action of the other resonators.’’ Because he studied vowels in a singing voice, Lewis also discovered (without realizing it) what is now often called the singer’s formant, which manifested as a 3,200 Hz formant present in most every vowel in the analyses. Chiba and Kajiyama [10] presented power spectra of the harmonics in a variety of Japanese vowels, though they did not describe their method of computing the spectra. As far as interpreting their results, the nineteenth century debate over the best theory of vowel production was regarded as still unresolved by Chiba and Kajiyama [10]. They were led to follow Scripture in favoring the ‘‘inharmonic theory’’ of Willis and Hermann because they frequently found formant frequencies which did not coincide with any harmonic in their calculated spectra. The nature of this finding and the way it is interpreted reflects the very same misunderstanding of the action of a resonator that had befuddled Scripture before them. Harvey Fletcher [21, 22] was, to his credit, possibly the first prominent speech scientist to understand that these two competing models of vowel production are not incompatible, and are both correct. It was seemingly Fletcher who first called the Willis theory ‘‘inharmonic,’’ since it postulates that the formant frequencies are excited directly by each glottal pulse in turn, and so need have no relation to the series of harmonics of the glottal fundamental frequency. The Helmholtz theory (which had originated with Wheatstone) was then dubbed the ‘‘harmonic’’ theory, since it postulates that the series of harmonics of the glottal fundamental are selectively emphasized, possibly in groups, by the formant frequencies which are in fact resonances. Fletcher made it clear that a formant resonance can act to emphasize harmonics emanating from the voice source no matter whether the formant frequency coincides with any one of them, thus obviating the need for any previously supposed ‘‘accommodation’’ of the vocal tract to the harmonics. This new understanding of the physical basis of vowel acoustics would eventually be called the source-filter theory. Vowel sounds had already been described, in essence, in terms of a source-filter theory by Lewis. This theory, which might well be called the cord-tone-resonance theory, states in effect that the vocal cords, during phonation, set up in air immediately adjacent to them a

3.2 History of Speech Spectra

63

complex motion which consists of a fundamental component and a large number of its overtones. This complex motion constitutes the so-called cord-tone. The theory further states that the vocal cavities, on which the cord-tone acts as a force, have the properties of simple resonators and thus serve to modify the spectrum of the energy flowing from the cords. In terms of this theory, a vowel sound, as emitted from the mouth, is due to both selective generation and selective transmission … and it is composed mainly of a harmonic series of simple motions, each of which has a determinable magnitude [35].

Lewis sounds like he is summarizing someone else’s theory, but he gives no citation; as far as I can tell his discussion is a synthesis of his own devise. The ideas trace back to Wheatstone, but Lewis’s summary gives an excellent (albeit early) account of what would come to be accepted as the source-filter theory of speech production, particularly after it was presented to the linguistic phonetics community by Joos [31].

3.2.3 Spectrographic Analysis Just as speech spectrum analysis was coming to be better developed in both the analog and discrete-time regimes, providing a picture of the distribution of speech energy across the frequency range, the spectrograph device was developed as a means of providing ‘‘a permanent visual record showing the distribution of energy in both frequency and time’’ [33]. While the spectrogram image output by the spectrograph is now usually described as a logarithmic plot (in dB) of the squared magnitude of the short-time Fourier transform (to be presented in detail in the next chapter), in fact the analog spectrogram bears a closer affinity to other analog signal processors like the Henrici analyzer, and predated the explicit specification of this transform by more than 20 years. The first ‘‘sound spectrograph’’ device was developed at Bell Labs shortly before World War II, was closely held following the outbreak of war, and was finally described in the open literature following the war’s end [33]. At that point, the short-time spectra it output were only understood from the perspective of electrical filters, since the original spectrograph was an entirely analog electrical signal processing device. The spectrogram would ultimately prove to be the most universally valuable representation for the analysis of speech spectra, because speech is such an inherently time-varying signal. It was shown in some of the earliest papers [34] that the time course of vowel formants could be readily imaged by setting the spectrograph filter to a ‘‘wide’’ bandwidth. Kopp and Green called the resulting images of formants ‘‘resonance bars,’’ owing to their appearance as thick bands of dark grey on the spectrogram, an image now familiar to thousands of speech scientists around the world (v. Fig. 3.12). Such spectrograms effectively performed a very short-time spectral analysis due to the short impulse response of the filter, and it was possible to observe the impulses generated by the vocal cords as ‘‘vertical striations’’ through the frequency range. It was also possible to perform a ‘‘narrowband’’ analysis using the spectrograph, which yielded a spectrogram with

64

3 History of Speech Spectrum Analysis

Fig. 3.12 Some of the first speech spectrograms ever published [34]

sufficient frequency resolution to show the fundamental frequency of voicing together with the harmonics; the longer impulse response of the narrowband filter made it impossible to simultaneously resolve the vocal cord impulses, however. An important publication by Joos [31] helped bring spectrographic analysis of speech to the attention of the phonetics community, as did the book by Potter et al. [46]. In the ensuing years, the spectrographic analysis of speech became a standard approach in phonetics. An archetypical publication by Potter and Steinberg [47] delved into many issues of speech features in spectrograms, and was very instructive as to measurement methods. This paper also demonstrated how to exploit the spectrograph’s capability of computing a single power spectrum analysis at a given time point, in conjunction with a spectrogram. Through the 1950s, the spectrograph’s analyses became quite frequently used to measure speech features such as vowel formants (e.g. [18, 44]), but there remained debate about the best ways of making the desired measurements. Near the end of the 1960s, thanks to the re-invention of the fast Fourier transform algorithm (discussed above), digital computers had become powerful enough to be able to compute spectrograms which could simulate the analog output of the spectrograph device. The general algorithm for a digital spectrogram (presented in the next chapter) was widely disseminated by several proponents, including Oppenheim [42]. In recent years, the procedure of making measurements ‘‘manually’’ from a spectrogram has fallen somewhat out of favor, and has been

3.2 History of Speech Spectra

65

gradually replaced by parametric spectral analysis, to be discussed in the next section. Current developments in spectrographic, or more generally time–frequency analysis, are nevertheless very promising, and it is not unreasonable to suggest that the new methods presented in the upcoming chapters may yet revive the perceived utility of spectrogram reading.

3.3 Parametric Spectral Analysis By the 1960s, the ‘‘source-filter theory’’ of speech production was coming to the fore, and as a result the vocal tract began to be viewed as a resonant filter which was describable by a few parameters, in particular its resonance frequencies and their bandwidths and relative amplitudes. These in turn are compactly represented using the transfer function of the filter, which will be treated more thoroughly in Chap. 7. It is the modeling of the vocal tract as a linear filter which led to a number of related approaches to speech spectrum analysis that have been termed ‘‘parametric.’’ The analysis-by-synthesis scheme was outlined as follows by Bell et al. [2]. The speech was first sent through a ‘‘filter set,’’ and the outputs were sampled digitally and stored. This yielded a basic spectrum estimate which was like a spectrogram. A ‘‘spectrum generator’’ was then used to synthesize speech according to a linear source-filter model, governed by parameters of a voice source and vocal-tract transfer function (i.e. formants) input by the user. A ‘‘comparator’’ then computed a measure of the difference between the input speech spectra (which are nonparametric but rather imprecise) and those generated by the spectrum generator. When a synthesized spectrum providing minimum error was obtained, the parameters of its transfer function and source characteristics were used to produce the parametric spectral estimate. Since this was a rather involved and cumbersome procedure, it was a priority to develop a means of estimating a vocal tract filter model from some input speech without actually performing analysis by synthesis, and thus a number of approaches to spectral estimation by analysis of the speech ‘‘time series’’ data (i.e. the waveform) were developed during the late 1960s. In the field of time-series statistics that I have alluded to previously, many decades of the twentieth century and beyond have been spent on the development of models for time series data, and for statistically estimating the parameters of such models. One of the earliest models for certain kinds of time series was christened linear prediction by Wiener [58], but this idea appears to have its roots in work by Carl Gauss in 1795 [38]. With the advent of digital speech processing and the analysis-by-synthesis techniques just described, the notion arose that a digital speech waveform can be viewed as time series data, since it is in essence just a sequence of ordinate values at equally spaced time points. The application of linear prediction models to speech ‘‘time series’’ was initiated by Saito and Itakura [49], and also by Atal and Schroeder [1]. The method was developed into a standard approach during the 1970s, particularly through the seminal book by

66

3 History of Speech Spectrum Analysis

Markel and Gray [38], who proved many mathematical facts about the various possible means of working with such models, while also teaching the community various algorithms for the method. A different statistical time series model known as the maximum entropy method was introduced by Burg [6], but this was soon recognized as yet another algorithm for linear prediction analysis [3]. It is left to the modern era to refine and further improve upon speech spectrum analysis, some of the methods for which are to be described in the upcoming chapters.

References 1. B.S. Atal, M.R. Schroeder, Predictive coding of speech signals. in Proc. 1967 Conf. Commun. and Process (1967) pp. 360–361 2. C.G. Bell, H. Fujisaki, J.M. Heinz, K.N. Stevens, A.S. House, Reduction of speech spectra by analysis-by-synthesis techniques. J. Acoust. Soc. Am. 33(12), 1725–1736 (1961) 3. A. van den Bos, Alternative interpretation of maximum entropy spectral analysis. IEEE Trans. Inform. Theory 17, 493–494 (1971) 4. H. Bouasse, Acoustique Générale: Ondes Aériennes (Delagrave, 1926) 5. L. de Broglie, Certitudes et Incertitudes de la Science. (Albin Michel, Paris, 1966) 6. J. P. Burg, A new analysis technique for time series data. in Modern Spectrum Analysis, ed. by D.G. Childers (IEEE Press, New York 1978), pp. 42–49. Reprint of a paper presented at the NATO Advanced Study Institute on Signal Processing with Emphasis on Underwater Acoustics, 1968 7. G.A. Carse, J. Urquhart, Harmonic analysis. in Modern Instruments and Methods of Calculation, ed. by E.M. Horsburgh (Tomash, Los Angeles, 1914) pp. 220–248 (Reprinted 1982) 8. H.S. Carslaw, Introduction to the Theory of Fourier’s Series and Integrals, 3rd edn (Dover Publications, New York, 1930) 9. A. Cauchy, Mémoire sur la théorie de la propagation des ondes à la surface d’un fluide pesant. Mémoires des Savans Étranger, pp. 3–313 (1827). Reprinted in Cauchy’s Oevres Complètes 1re série tome 1 10. T. Chiba, M. Kajiyama, The Vowel: Its Nature and Structure (Phonetic Society of Japan, Tokyo, 1958) 11. J.W. Cooley, J.W. Tukey, An algorithm for the machine calculation of complex Fourier series. Math. Comput. 19, 297–301 (1965) 12. I.B. Crandall, The sounds of speech. Bell. Sys. Tech. J. IV, 586–626 (1925) 13. I.B. Crandall, Dynamical study of the vowel sounds, part II. Bell. Sys. Tech. J. VI, 100–116 (1927) 14. I.B. Crandall, C.F. Sacia, A dynamical study of the vowel sounds. Bell. Sys. Tech. J. III, 232–237 (1924) 15. G.C. Danielson, C. Lanczos, Some improvements in practical Fourier analysis and their application to X-ray scattering from liquids. J. Franklin Inst. 233, 365–380, 435–452 (1942) 16. F.C. Donders, Über die Natur der Vokale. Archiv f. d. holländische Beiträge z. Natur- u. Heilkunde I (1858) 17. A.J. Ellis, Analysis and synthesis of vowel sounds. In: On the Sensations of Tone (Helmholtz) [29], pp. 538–543 18. G. Fant, Acoustic Theory of Speech Production (The Hague, Mouton, 1960). Reissued 1970 19. P. Flandrin, Time-Frequency/Time-Scale Analysis. English edn (Academic Press, San Diego, 1999) 20. H. Fletcher, The nature of speech and its interpretation. J. Franklin Inst. 193(6), 729–747 (1922)

References

67

21. H. Fletcher, Speech and Hearing (D. van Nostrand, Princeton, 1929) 22. H. Fletcher, Speech and Hearing in Communication. (D. van Nostrand, Princeton, 1953) 23. J.B.J. Fourier, Théorie de la propagation de la chaleur dans les solides (1807). Manuscript first published in [26] 24. C. Godfrey, On the application of Fourier’s double integrals to optical problems. Phil. Trans. R. Soc. Lond. Ser. A 195, 329–362 (1900) 25. M. Gouy, Sur le mouvement lumineux. J. Phys. 5 (1886) 26. I. Grattan-Guinness, Joseph Fourier 1768–1830 (The MIT Press, Cambridge, 1972) 27. H.H. Hall, Sound analysis. J. Acoust. Soc. Am. 8, 257–262 (1937) 28. M.T. Heideman, D.H. Johnson, C.S. Burrus, Gauss and the history of the fast Fourier transform. IEEE Acoust. Speech Sig. Proc. Mag. 1, 14–21 (1984) 29. H. Helmholtz, On the Sensations of Tone, 2nd English edn. (Longmans & Co., Oxford, 1885) 30. F. Jenkin, J.A. Ewing, On the harmonic analysis of certain vowel sounds. Trans. R. Soc. Edinb. 28, 745–777 (1878) 31. M. Joos, Acoustic Phonetics. (No. 23 in Language Monographs. Linguistic Society of America, Baltimore, 1948) 32. R. Koenig, Die manometrischen Flammen. Ann. der Phys. Chem. 222(6), 161–199 (1872) 33. W. Koenig, H.K. Dunn, L.Y. Lacy, The sound spectrograph. J. Acoust. Soc. Am. 18(1), 19–49 (1946) 34. G.A. Kopp, H.C. Green, Basic phonetic principles of visible speech. J. Acoust. Soc. Am. 18(1), 74–89 (1946) 35. D. Lewis, Vocal resonance. J. Acoust. Soc. Am. 8, 91–99 (1936) 36. R.J. Lloyd, Some researches into the nature of vowel-sound. Ph.D. thesis (University of London, 1890) 37. R.J. Lloyd, Speech sounds: their nature and causation. Phonet. Stud. IV, 37–67 (1891) 38. J.D. Markel, A.H. Gray Jr, Linear Prediction of Speech. (Springer, Berlin, 1976) 39. E. Merritt, On a method of photographing the manometric flame, with applications to the study of the vowel A. Phys. Rev. I (1893) 40. D.C. Miller, The Science of Musical Sounds, 2nd edn. (MacMillan, New York, 1926) 41. J.G. M’Kendrick, Experimental phonetics. Nature 65(1678), 182–189 (1901) 42. A. V. Oppenheim, Speech spectrograms using the fast Fourier transform. IEEE Spectrum (8), 57–62 (1970) 43. R.A.S. Paget, The production of artificial vowel sounds. Proc. R. Soc. Lond. A 102(719), 752–765 (1923) 44. G.E. Peterson, H.L. Barney, Control methods used in a study of the vowels. J. Acoust. Soc. Am. 24(2), 175–184 (1952) 45. H. Pipping, Zur Phonetik der finnischen Sprache. (Helsingfors, 1899) 46. R.K. Potter, G.A. Kopp, H.C. Green, Visible Speech. (D. van Nostrand, New York, 1947) 47. R.K. Potter, J.C. Steinberg, Toward the specification of speech. J. Acoust. Soc. Am. 22(6), 807–820 (1950) 48. C.F. Sacia, Photomechanical wave analyzer applied to inharmonic analysis. J. Opt. Soc. Am. Rev. Sci. Instrum. 9, 487–494 (1924) 49. S. Saito, F. Itakura, The theoretical consideration of statistically optimum methods for speech spectral density. (Technical Report 3107, Electrical Communication Laboratory, N. T. T., Tokyo, 1966) (in Japanese) 50. Schneebeli: Sur la théorie du timbre et particulièrement des voyelles. Archives des Sciences Physiques et Naturelles de Genève (1879) 51. E.W. Scripture, The Elements of Experimental Phonetics. (Charles Scribner’s Sons, New York, 1902) 52. J.C. Steinberg, Application of sound measuring instruments to the study of phonetic problems. J Acoustical Soc Am 6, 16–24 (1934) 53. J.W. Strutt (Baron Rayleigh), The Theory of Sound, vol. II, 2nd edn (Macmillan, London, 1896)

68

3 History of Speech Spectrum Analysis

54. S.P. Thompson, A new method of approximate harmonic analysis by selected ordinates. Proc. Phys. Soc. Lond. 23, 334–343 (1911) 55. W. Thomson (Lord Kelvin), Harmonic analyzer. Proc. R. Soc. Lond. 27, 371–373 (1878) 56. W. Thomson (Lord Kelvin), P.G. Tait, Treatise on Natural Philosophy, vol. 1, 2nd edn. (Cambridge University Press, Cambridge, 1912) 57. E. Whittaker, G. Robinson, The Calculus of Observations. 4th edn (D. van Nostrand, New York, 1944) 58. N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series, with Engineering Applications. (Technology Press of M.I.T., Cambridge, 1949) 59. R. Willis, On the vowel sounds, and on reed organ-pipes. Trans. Camb. Phil. Soc III, 231–268 (1830)

Chapter 4

The Fourier Power Spectrum and Spectrogram

This chapter covers the traditional speech analysis methods which rely on the discrete Fourier transform and its extension to the ubiquitous time–frequency representation known as the spectrogram. The first topic is the power spectrum of a signal window, which is derived from the magnitude of the Fourier transform in the manner explained in Chap. 2. Here, I discuss some of the methods for making power spectra of speech sounds, in an effort to show the best ways of accomplishing the desired imaging. Power spectra may be used to examine the formants of vowels and other resonant sounds, and when treated statistically they may also illuminate aspects of the noise produced during voiceless consonants. A third important application of power spectra is in the analysis and detection of different phonation types such as creaky and breathy voicing. Numerous figures provide examples of power spectra illustrating the points discussed in the text. The second topic is the spectrogram; since this has not been treated properly here until the present chapter, the discussion is divided into first theoretical and then practical matters. The mathematical definition of the spectrogram is presented with an eye toward both its computational and historical aspects, a simplified algorithm for spectrogram computation is provided, and the ever-present problem of the uncertainty principle is discussed. Turning to more practical concerns of the speech scientist, guidelines are given for setting the various user-defined parameters of the spectrogram in order to obtain the best possible images for showing attributes of speech such as formants, transient consonant events and noise, and finally the fundamental frequency of voicing. The effects of the various parameter settings are illustrated in spectrograms of a variety of synthetic and natural speech sounds.

4.1 The Power Spectrum in Speech Analysis What speech scientists normally call the power spectrum of a digital signal is simply the squared magnitude of the discrete Fourier transform defined in Eq. 2.35, graphed showing a decibel-scaled amplitude as a function of frequency S. A. Fulop, Speech Spectrum Analysis, Signals and Communication Technology, DOI: 10.1007/978-3-642-17478-0_4,  Springer-Verlag Berlin Heidelberg 2011

69

70

4 The Fourier Power Spectrum and Spectrogram

(v. Figs. 2.7, 2.8). Furthermore, it is important to note that in practice the frequency range of the graph is limited to show only the lower half of the positive frequencies computed by the DFT. As was explained in the previous chapter, neither the negative frequencies nor the upper half-range of the positive frequencies add any new information. This standard form of the graph is displayed automatically by the typical speech software (e.g. Praat) functions for obtaining and showing a power spectrum.

4.1.1 Vowel Spectra In the previous chapter, the nature of vowels was discussed from an historical perspective on spectrum analysis. We concluded that according to current understanding, a vowel sound can be analyzed in the frequency domain as comprising harmonics of a fundamental (which is the frequency of the phonation) passed through resonant filters called formants. On the other hand, vowels and other voiced speech sounds have a second valid model, which is to simply say that the formants are excited by each glottal impulse during phonation. This is what we may call the ‘‘harmonic-inharmonic duality’’ of voiced speech, in analogy to the famous wave-particle duality of light that came to be recognized also during the early 20th century. Just as light can be detected as particles or waves, depending on the type of phenomenon examined, so voiced speech can be measured as having either a harmonic spectrum or an inharmonic spectrum, depending on the type of analysis performed. A good benchmark from which to test the performance of many spectrum analysis methods is to examine synthesized vowels whose formants and other parameters can be known in advance. An excellent simple vowel synthesizer is included with recent versions of the Praat software [5], and this has been used to create the synthesized vowels examined in this book. The Praat synthesizer operates by invoking the sourcefilter model; beginning with a fairly simple glottal source vibrating with a user-input fundamental frequency, the algorithm computes the output from a virtual vocal tract having user-input values of four formants. The scheme is a typical implementation of what has come to be known as a ‘‘formant synthesizer.’’ To create a small corpus of synthesized vowel tokens, a number of vowels were synthesized using Praat to have English-like characteristics. These vowels have already been used for a study of formant measurement accuracy [12], but here they are recruited for a variety of tests and illustrations through the remainder of the book. Figure 4.1 shows two kinds of power spectra overlaid for each of three English-style vowels. The vowel [æ] is synthesized with formants F1 ¼ 731 Hz and F2 ¼ 1; 768 Hz; [i] has F1 ¼ 306 Hz and F2 ¼ 2; 241 Hz; [O] has F1 ¼ 602 Hz and F2 ¼ 884 Hz: Perceptual studies have generally shown that these two formants are by far the most important for determining the perceived vowel quality, so the remaining formants were set to F3 ¼ 2; 500 Hz and F4 ¼ 3; 500 Hz in all vowel tokens. The standard glottal source spectral amplitude rolloff employed by Praat

4.1 The Power Spectrum in Speech Analysis

30

Magnitude (dB)

Fig. 4.1 Short (12 ms) and long (40 ms) window power spectra of synthetic vowels [æ] (top) [i] (middle) [O], using a Gaussian window

71

20 10 0 -10 -20 0

1

2

3

4

5

4

5

4

5

Frequency (kHz)

Magnitude (dB)

40 30 20 10 0 -10 0

1

2

3

Frequency (kHz)

Magnitude (dB)

40 30 20 10 0 -10 0

1

2

3

Frequency (kHz)

makes the fourth formant almost absent from the sounds. All tokens were generated for approximately 300 ms using a glottal source having a frequency beginning at 120 Hz and descending at a rate of 2 octaves per second. This results in a fairly natural baritone falling intonation for English.

72

4 The Fourier Power Spectrum and Spectrogram

The ‘‘squiggly’’ spectra in Fig. 4.1 were computed in Praat with a 40 ms Gaussian window.1 This window length encompasses several glottal impulses in the synthesized phonation, and as a result the Fourier spectrum shows the fundamental frequency of phonation and its many harmonics as the numerous little peaks in the graph. The harmonics are not well resolved here because the fundamental frequency changed slightly during the course of the analysis window, which normally happens during real vowels. This type of long-window spectrum is for historical reasons called ‘‘narrowband,’’ and it is the right sort of frequency analysis to highlight the harmonic nature of vowels. The formants can be seen, just as Helmholtz had stated [18], as regions in the frequency range within which harmonics are emphasized. The smoother spectra in Fig. 4.1 were computed using a 12 ms Gaussian window, meaning about 6 ms from the signal was effectively used, and so the window encompasses a domain smaller than one complete cycle of the glottal source. This makes it impossible to detect the frequency of phonation, so the single glottal impulse is analyzed as a transient which excites the formants. The window was centered on the portion of the signal waveform lying between glottal impulses, in order to focus on the resonance of the formants and deemphasize the spectrum of the transient impulse which is very broadband. This kind of short window spectrum is for historical reasons called ‘‘wideband,’’ and it is the right sort of frequency analysis to highlight the inharmonic nature of vowels. The three synthesized vowels of Fig. 4.1 were chosen because they represent three commonly encountered extremes of vowel quality. The vowel [æ] has its three major formants well separated in frequency, while [i] has F2 and F3 in close proximity, and [O] has F1 and F2 in close proximity. Vowel formants can in principle be measured from either the short or longwindow power spectrum. A commonly used procedure at one time [26] was to hand-draw a smooth line around the harmonic peaks in a long-window spectrum, and in this way the actual peaks of the formants can be estimated. It is not wise to select a harmonic and measure its peak as that of a formant, since the two kinds of peaks will not coincide in general (this is a reminder that the Helmholtz ‘‘accommodation theory’’ is not correct). On the other hand, in Fig. 4.1 it is quite plain that the closely spaced F2 and F3 of the vowel [i] cannot be discerned at all in the long-window spectrum, while they are able to be measured from the shortwindow spectrum. This brief example demonstrates that, for synthesized vowels, a short-window power spectrum gives a better look at the formants and enables their estimation from the peaks with relative ease. Turning our attention to vowels occurring in natural speech, it quickly becomes apparent that each different spectral analysis seems to provide conflicting

1 It should be noted that a Gaussian window function is strongly tapered, and so the computed spectrum is mostly derived from the center half of the window length. Praat has a number of features intended to compensate for this, to be described in the appendix.

4.1 The Power Spectrum in Speech Analysis

73

information about the vowel. In particular, the problem of formant estimation cannot so easily be solved by a single power spectrum. Figure 4.2 shows two power spectra from the [i] of heed. Both spectra are computed from 12 ms Gaussian windows; the key difference is the position of the window with respect to a phonation cycle. The dark line uses a window centered between glottal impulses, and appears to give a much more usable result, although there nevertheless appears to be a low peak at around 140 Hz that cannot be attributed to a formant. The first peak above this is located at 400 Hz, and a weak peak is visible at 2,170 Hz on the left shoulder of a larger peak located at 2,440 Hz. The speaker is male, so these are at least plausible values of the first three formants for this vowel. The lowest peak is probably a glimpse of the fundamental frequency resulting from the window barely encompassing two glottal impulses. The lighter line in Fig. 4.2 uses a window centered on a glottal impulse. The formant frequencies are greatly obscured in this spectrum, resulting chiefly from the tremendous amount of broadband energy present in the impulse itself. It takes a millisecond or more for the resonances to appear following the glottal transient excitation, and the spectrum of the excitation itself does not contain well-resolved formants. Figure 4.3 uses a 41 ms Gaussian window, yielding a look at the harmonics of the voicing. It is plain to see that formants cannot readily be measured or easily separated in this spectrum. The second harmonic is very loud, and so one might guess that the first formant is nearly coincident with it, but its frequency of 287 Hz is not a very plausible formant value. The reinforcement of this harmonic probably results from the lower resonance band known as the voice bar, which will be the subject of a later discussion. A comparison of the short and long window spectra reveals some disagreement; there is no evidence in the long frame spectrum of Fig. 4.3 to support the presence of the 400 Hz formant that appears in Fig. 4.2. How could this be? We will see in Chap. 6 that the chief reason for this is the nonstationary nature of the speech signal even at the time scale of a single glottal

40 30

Power (dB)

Fig. 4.2 Praat spectral slices with 12 ms Gaussian windows during steady [i] of heed. Phonation period is 7 ms. Dark line uses a window centered on the time between glottal impulses; light line uses a window centered on a glottal impulse

20 10 0 -10 0

1000

2000

3000

Frequency (Hz)

4000

5000

74

4 The Fourier Power Spectrum and Spectrogram

Fig. 4.3 41 ms spectral slice during steady [i] of heed. Phonation period is 7 ms

50

Power (dB)

40 30 20 10 0

0

1000

2000

3000

4000

5000

4000

5000

Frequency (Hz)

Fig. 4.4 Power spectra from 12 ms (black) and 41 ms slices in the English vowel [O] of hawed

40

Power (dB)

30 20 10 0 -10 0

1000

2000

3000

Frequency (Hz)

cycle. The long-window analysis provides a long-term spectrum in which the frequencies resulting from excitation of the formants has been obscured by the glottal impulse spectral information. Figure 4.4 demonstrates that real speech can present seemingly insurmountable difficulties for measuring formants by simple spectrum analysis. The figure shows two power spectra computed from a 12 ms and a 41 ms Gaussian window during the vowel [O] of hawed. With an average male speaker pronouncing this vowel, phonetic theory informs us that the values of F1 and F2 are in proximity and must both lie between 500 and 1,000 Hz. Sadly, neither type of power spectrum permits us to locate two formants in this expected range. The short-window spectrum does display an additional peak at 250 Hz, but this cannot be one of the defining vowel formants and is most likely a manifestation of the voice bar. The fundamental frequency of this vowel at the point of analysis is about 104 Hz.

4.1 The Power Spectrum in Speech Analysis

75

4.1.2 Obstruent Spectra and Averaging Techniques Obstruent sounds such as fricatives are chiefly characterized by noise, which is by definition a random signal. Normally the long-frame spectrum of such noise will betray the randomness by being completely ‘‘filled in’’ without any fundamental or harmonics apparent, although the noise spectrum of a speech sound is nevertheless shaped by the vocal tract. Example spectra of three English fricatives are shown in Fig. 4.5. That these spectra have different shapes partly accounts for our perception of these as different speech sounds in English, although consonants require being uttered in the context of a syllable in order to be most reliably perceived, and in that context other transitional acoustic features become important as well.

4.1.2.1 Spectral Moments It has often proven difficult for phoneticians to describe such noisy obstruent spectra in a useful way. This is a result of the randomness; obstruent spectra are not as consistent from utterance to utterance as vowel spectra, and the features that are observable are rather vague, in the vein of noticing that the spectrum of [s] in Fig. 4.5 ‘‘has a broad peak around 8 kHz,’’ or that the spectrum of [h] ‘‘is very spread out’’ (e.g. [16]). These descriptions are not easy to quantify, so phoneticians have sought ‘‘metrics’’ of obstruent noise spectra that are more quantitative. One frequently used set of metrics for speech noise spectra are the spectral moments. Since the power spectrum of a noise is statistically random, it is not unreasonable to treat it literally as a probability density function, which is a mathematical object characterizable by an infinite set of numbers called its moments. The first moment is generally equivalent to the simple mean of the spectrum, thereby providing a basic measure of location for the spectrum along the frequency axis. The second moment is equivalent to the variance (square of the standard deviation), thereby providing a rough measure inverse to how tightly the spectrum is crowded around its mean. Higher moments, if used, should simply be thought of as numbers which further quantify the power spectrum. I do not wish to discuss the mathematical statistics behind the moments (for this see [33]) since the computation of spectral moments is really a data reduction procedure and not a spectrum analysis method per se; it is sufficient for our purpose here to note that Praat software, among others, has the ability to compute several moments characterizing any power spectrum. The idea of computing statistical moments to quantify noisy speech spectra dates back at least to Forrest et al. [11]. The method was used with some success in later studies of fricatives. Most of the studies which have used spectral moments concluded that the first two are by far the most important for characterizing obstruent power spectra (e.g. [21]). Fulop et al. [14] used moments to classify the burst spectra of click sounds in Yeyi, and also discussed a number of statistical issues with the interpretation of spectral moments. Figure 4.6 shows example spectra of a dental click burst, which

76

30

Magnitude (dB)

Fig. 4.5 Power spectra of English fricatives, entire length (150–180 ms). [s] (top); [S] (middle); [h]

4 The Fourier Power Spectrum and Spectrogram

20 10 0 -10 -20 0

2

4

6

8 10 12 14 Frequency (kHz)

16

18

20

2

4

6

8 10 12 14 Frequency (kHz)

16

18

20

2

4

6

8 10 12 14 Frequency (kHz)

16

18

20

40

Magnitude (dB)

30 20 10 0 -10 0

Magnitude (dB)

20 10 0 -10 -20 -30 0

also illustrates the importance of a good window function. The original study [14] used rectangular windows, which was likely not such a good procedure as it may have fatally affected the data to the point of altering the results. An alveolar click burst spectrum is shown in Fig. 4.7, which shows apparent differences from the dental. These differences were quantifiable for good classification using the moments alone, in spite of the rectangular windows.

77 Magnitude (dB)

Magnitude (dB)

4.1 The Power Spectrum in Speech Analysis

20 10 0 -10 -20 -30

30 20 10 0 -10 -20

0

1

2

3

4

5

6

7

8

9 10 11

0

1

2

3

Frequency (kHz)

4

5

6

7

8

9 10 11

Frequency (kHz)

Fig. 4.6 Power spectra computed from a 26 ms dental click burst in Yeyi. Left panel uses a Gaussian window (which is the recommended procedure), right panel uses a rectangular window

30

Magnitude (dB)

Fig. 4.7 Power spectrum computed from an 18 ms alveolar click burst in Yeyi, using a Gaussian window. Note the difference from the preceding figure showing the dental click

20 10 0 -10 -20 0

1

2

3

4

5

6

7

8

9

10

11

Frequency (kHz)

4.1.2.2 Time and Ensemble Average Spectra Since obstruent noise is inherently random, one way to increase the resolution of pertinent information out of the randomness involves averaging the spectra of a number of windows on the same speech sound. It can be proven that applying this technique (which was first promoted in [35]) to a statistically random process will generally decrease the variance of the resulting average, and provide a more highly resolved picture of the spectrum. To give a conceptual analogy, if you wanted to check that 100 coin tosses yields an expected 50 heads, it is a good idea to perform the 100-toss experiment 10 times and average the results, since this average is much more likely to be very near to 50 than any single run of the experiment. Two fairly simple ways of implementing this concept have been described for fricatives in the literature [30, 31]. One method is to clip out a number of (possibly overlapping) analysis frames from different time points within a single fricative token, and compute an average of their power spectra. The result is called a time average spectrum. An alternative approach requires a number of tokens of the same fricative repeated by the same speaker. We compute a spectrum for one analysis window per token and then compute an average spectrum from these. This is called

78

4 The Fourier Power Spectrum and Spectrogram 20

Magnitude (dB)

Magnitude (dB)

10 0 −10 −20 −30 −40 −50 −60 −70

0

5000

10000

15000

20 10 0 −10 −20 −30 −40 −50 −60 −70 −80

0

10

10

0

0

−10 −20 −30 −40 −50

5000

10000

15000

Frequency (Hz)

Magnitude (dB)

Magnitude (dB)

Frequency (Hz)

−10 −20 −30 −40

0

5000

10000

Frequency (Hz)

15000

−50

0

5000

10000

15000

Frequency (Hz)

Fig. 4.8 Top line: power spectra of 55 ms windows from English fricatives [S, s]; Bottom line: ensemble average power spectra of the same two fricatives, each using 10 repetitions from one speaker

an ensemble average spectrum, and is illustrated in Fig. 4.8. Average spectra will depend upon the stage at which the averaging operation is performed, and this is not always explicitly stated in the literature. The ‘‘raw’’ spectrum, being the result of a Fourier transform, is a vector of complex numbers, and it would not be meaningful to average these. In the linked Matlab code here, the averaging is performed on the squared magnitude of the Fourier transform, before conversion to a logarithmic (dB) scale. The results would look different if averaging was performed on the magnitude spectrum without first squaring, or after conversion to dB.

4.1.3 Phonation Types Long frame power spectra showing voicing harmonics have proven to be very useful tools for detecting a key acoustic difference between different phonation types. A number of studies [24, 34] have demonstrated that, all else being equal (meaning the speaker and the other speech sound features), the relative amplitudes of the fundamental harmonic H1 and the next two harmonics H2 ; H3 in the series usually distinguish two or more different phonation types when they occur in any

4.1 The Power Spectrum in Speech Analysis

79

given language. Normally, creaky phonation is typified by the lowest amplitude of H1 relative to H2 and H3 ; modal phonation displays a somewhat higher H1 amplitude or somewhat lower H2 or H3 than in creaky phonation, while breathy phonation is characterized by the highest relative H1 amplitude overall. These descriptions are illustrated in Fig. 4.9, showing creaky, modal, and breathy voice

40

Power (dB)

30 20 10 0 -10 0

100

200

300

400

600

800

600

800

Frequency (Hz)

Power (dB)

40 30 20 10 0 -10 0

200

400

Frequency (Hz)

40

Power (dB)

Fig. 4.9 Power spectra computed from 30–40 ms (approx.) Gaussian windows of the vowel [e], pronounced by the author in three different phonation types. Top to bottom: creaky, modal, and breathy voice

30 20 10 0 -10 0

200

400

Frequency (Hz)

80

4 The Fourier Power Spectrum and Spectrogram

for the steady vowel [e]. The creaky example is characterized by high amplitudes of several harmonics above the fundamental; the modal example has a relatively high amplitude of H2 alone, while the breathy example has H1 as the loudest harmonic. A recent small study [13] of the Hmong language has demonstrated that breathy and whispery phonation can also be distinguished by the relative harmonic amplitudes in similar fashion to the above. Example power spectra from that study are shown in Fig. 4.10, in which it can be observed that the modal vowel has a smaller relative H1 amplitude than the breathy vowel, which in turn is less than that of the vowel following a whispery voiced stop release. These differences were found to be so robust in the Hmong words studied that they did not require statistical validation. Another means of distinguishing phonation types uses a measure of the amplitude of all the harmonics relative to the noise in the speech. There have been a number of methods for computing this harmonicity metric (sometimes called the harmonics-to-noise ratio) presented in the literature; an excellent metric of this kind is computable using Praat software. The above mentioned study [13] of Hmong phonation types also demonstrated that the modal, breathy, and whispery phonations of Hmong can be distinguished by means of the harmonicity. It was found that the modal vowels usually had less harmonicity than the breathy vowels (which was surprising), while the whispery phonation had substantially less harmonicity than the modal (which was expected).

4.2 Principles of the Spectrogram A spectrogram is a particular time–frequency representation, which is a function of both time and frequency that represents the energy distribution in a signal over the time–frequency plane. It is derived similarly to a power spectrum, from an extension of the Fourier transform to a joint time–frequency setting known as a short-time Fourier transform (sometimes also called the Gabor transform, which is strictly speaking not quite the same thing).

4.2.1 Definitions of the Spectrogram The spectrogram will be defined intuitively here, with the details relegated to a math box below. 4.2.1.1 Sequence of Fourier Transforms Before defining the spectrogram it is necessary to define the short-time Fourier transform (STFT). This can be understood as a time series of Fourier transforms,

4.2 Principles of the Spectrogram

81

where for each time point the Fourier transform is computed for a time-limited analysis window on the signal. The purpose of the window is to localize the Fourier transform’s view of the signal to the vicinity of a particular point in time. The STFT Sw ðt; xÞ is then parameterized by the window function w (so there is

Intensity (dB)

40 30 20 10 0 –10 0

100 200 300 400 500 600 700 800 900 1000

Frequency (Hz) 40

Intensity (dB)

30 20 10 0 –10

0

100 200 300 400 500 600 700 800 900 1000

Frequency (Hz) 40

Intensity (dB)

Fig. 4.10 Power spectra from the first 50 ms (approx.) of the vowel following release of stops in three distinct Hmong syllables. Top: (modal); Middle: (whispery stop release); Bottom: (breathy)

30 20 10 0 –10 0

100 200 300 400 500 600 700 800 900 1000

Frequency (Hz)

82

4 The Fourier Power Spectrum and Spectrogram

really an infinite family of such functions), and is a function of both time and frequency. Since the STFT is derived from the Fourier transform, like the latter it is generally complex-valued and invertible, meaning that the signal itself results from an inverse transform of its STFT. Recall that the (time-independent) energy density spectrum was defined as the squared magnitude of the Fourier transform. The time–frequency spectrogram Spgmw is then defined as the squared magnitude def

of the STFT, so that Spgmw ðt; xÞ ¼ jSðt; xÞj2 : The picture it provides is threedimensional, a sort of ‘‘running power spectrum’’ of successive windowed slices of the signal. It is not difficult to imagine how important the window function is to the particular properties of a spectrogram. To actually compute a short-time Fourier transform, as usual we need to work in a digital setting; the particulars are relegated to the math box, but a simplified algorithm will also be presented below.

In a continuous-time setting, the short-time Fourier transform of a signal sðsÞ is usually defined by the following equation [9]: Z1 def sðsÞwðt  sÞeixs ds; ð4:1Þ Sw ðt; xÞ ¼ 1

in which the function wðtÞ is a real-valued window function having finite support (normally this will be one of the window functions considered in Chap. 2). Comparing with Eq. 2.25, it can be seen that at each time point t0 the function Sw ðt0 ; xÞ is simply the Fourier transform of a portion of the signal windowed by w around t0 : This definition is equivalent to what much literature calls the Gabor transform in the analog regime, but the digital version of the STFT described presently uses a different method than the digital Gabor transform, so it is inaccurate to simply identify the two. Another form of the STFT which is sometimes used is given by: def S0w ðt; xÞ ¼

Z1

sðs þ tÞwðsÞeixs ds

ð4:2Þ

1

This alternate form is also called the moving window transform [22], and its result is equivalent to that from the previous definition except for a linear frequency term added to the phase [9]. The spectrogram can be equivalently defined as the squared magnitude of the STFT (Eq. 4.1) or of the moving window transform (Eq. 4.2), because the two transforms do not differ in magnitude. The STFT in either of the above forms is invertible using the same window function in the analog regime; the following inversion formula corresponds to Eq. 4.1:

4.2 Principles of the Spectrogram

1 sðsÞ ¼ 2p

83

Z Z1

Sw ðt; xÞwðt  sÞeixs dt dx:

ð4:3Þ

1

The inverse STFT can be thought of as providing an ‘‘expansion’’ of the signal using a continuum of ‘‘elementary signals’’ representing the time- and frequency-shifted windows [3]. It is really a signal synthesis formula, while the STFT itself is an analysis formula. To present the mathematics behind the digital spectrogram I will rely on [28]. The digital STFT method starts from the analysis formula (Eq. 4.1) and samples it in time and frequency. First, let us define the discrete-time STFT at particular time point n0 : 1 X

def

Sðn0 ; xÞ ¼

sðmÞwðn0  mÞeixm

ð4:4Þ

m¼1

where m takes integer values. Comparing with Eq. 2.29, this is just the discrete-time Fourier transform of the product of the signal with the analysis window. Next, make the time point a variable, thus defining the discrete-time STFT as a function of time point n and frequency x: 1 X

def

Sðn; xÞ ¼

sðmÞwðn  mÞeixm

ð4:5Þ

m¼1

So much for the discrete-time part; the last step is to move to discretefrequency as well. Just as the discrete Fourier transform of Eq. 2.35 yields frequency samples of the discrete-time transform, and is thus suitable for digital implementation, there is a similar equation for the discrete (digital) STFT which frequency-samples the discrete-time version above: N1 X



2pikm sðmÞwðn  mÞ exp Sðn; xk Þ ¼ N m¼0 def

 ð4:6Þ

Computing the result of this equation with a digital signal and specific window function yields a matrix of complex numbers, one for each ‘‘cell’’ in the time– frequency grid. To get a digital spectrogram from this, one should simply compute the magnitude of each number in the STFT matrix and square it. Equation 4.6 is now the digital version of the analysis formula; one might think that a corresponding inversion formula could likewise be produced by analogy with the continuous regime, but the discretization process has rendered the situation mathematically more complicated, so if we tried to use the sampled inversion formula to invert the sampled analysis formula, it would be improper [1]. Gabor originally approached the problem from the synthesis angle, and his signal expansion yields a discrete version of an inversion

84

4 The Fourier Power Spectrum and Spectrogram

formula rather than an analysis formula. It is possible to derive both a synthesis (now called a Gabor expansion) and a corresponding analysis formula (the Gabor transform) in the digital regime, with the snag that each now must utilize different window functions which are mathematically dual in an interesting way that has inspired quite a bit of research on the topic (e.g. [8]).

Comparing the final definition of the digital STFT above with the DFT equation (2.35), it is plain that the digital STFT is simply a sequence of DFTs of signalwindow products, in which the window is ‘‘sliding’’ along the signal. The defining equation above assumes that the window is moved by one time sample for each new DFT. It is usual practice to decimate the above definition, so that the signalwindow products sðmÞwðnL  mÞ are taken only for integer multiples of some integer L [ 1—i.e. the window is customarily slid along by more than one sample point. The practical STFT parameter L is variously called the ‘‘hop size’’ or ‘‘time step’’ or ‘‘frame advance.’’ It is important to keep the frame advance less than the window length, and preferably less than the effective length (about half of the window), or the spectrogram will be decimated too much and the successive windowed frames will not overlap enough to yield a good representation.2 On the other hand, given the way most digital spectrograms are displayed (to be described below) it is not usually worthwhile to have a very small frame advance, and this does lead to greatly increased computation times. The importance of the proper frame advance is illustrated in Fig. 4.11 showing three spectrograms of the word dad. The similarity of the first two spectrograms there is notable given that the second one required far more FFTs to be calculated; both of these are fairly optimal in performance for the parameters employed, but the top example is far more efficient. The lower image is, by contrast, over-decimated and totally undesirable.

4.2.1.2 STFT from a Filter Bank The earliest spectrograms were computed using analog electronics, without any Fourier transforms being directly performed. The intuition behind this process begins with considering a single fixed frequency within a short-time Fourier transform of a signal. Such a frequency band of an STFT is equal to the signal first modulated by the fixed frequency and passed through a filter whose impulse response is the analysis window. In the digital realm, the STFT has just a certain number of frequency bins, and each frequency row of the STFT matrix can be similarly computed as the signal passed through a bandpass filter whose impulse 2 From the mathematical theory of time–frequency analysis, a deep result called the Balian–Low theorem [17] establishes that a digital STFT must have overlapping windows in order to completely represent the signal (see also [2] for discussion).

4.2 Principles of the Spectrogram

85

5

Frequency (kHz)

4 3 2 1 0 0

200

400

600

800

600

800

600

800

Time (ms) 5

Frequency (kHz)

4 3 2 1 0 0

200

400

Time (ms) 5

Frequency (kHz)

4 3 2 1 0 0

200

400

Time (ms)

Fig. 4.11 Spectrograms of the English word dad [dæd], computed with 12 ms Gaussian window using Praat. Top panel uses the Praat standard frame advance of 2 ms; middle panel uses a frame advance of 0.1 ms; lower panel uses a frame advance of 6 ms

86

4 The Fourier Power Spectrum and Spectrogram

response equals the analysis window. The reader is referred to [28] for details concerning this ‘‘filter bank’’ method of computing the STFT. It will be significant in later chapters that when the STFT is viewed in this fashion, each frequency row in the matrix is itself an analytic signal whose instantaneous frequency can be calculated.

4.2.1.3 Algorithm for the Spectrogram We can assume that a digital signal is represented as a vector (basically just a list) of ordinate values, one for each time sample. To compute a spectrogram in a highlevel environment such as Matlab, the following rough algorithm can be implemented: 1. Divide the signal into ‘‘slices’’ which overlap, and which are each the length of the desired analysis window. 2. Index each slice by the time point at its center. The signal is now represented redundantly as a vector of slices. 3. Multiply each slice by a tapering function (or if you want rectangular windows, skip this step). 4. Compute the FFT of each slice, keeping in mind that the FFT frame length can always be longer than the slice if it is zero-padded properly. Matlab automatically takes care of this. 5. Discard the negative frequency range and the upper half of the positive frequencies. You now have a vector of the discrete Fourier transforms of the signal slices. This is referred to as the STFT matrix; it is a two-dimensional matrix of complex numbers, in which each cell lies at an intersection of time and frequency. The time points are the center points of the signal slices; the frequencies are the values in the FFT. The number of frequencies in the matrix depends on the length of the FFT frame—e.g. a 1,024-sample FFT frame will yield 512 frequency bins. 6. Take the absolute value of each number in the STFT matrix, then square the results of that. You now have the spectrogram matrix. When applied to sound signals, it is customary to scale the spectrogram magnitude logarithmically, as a decibel scale. When a spectrogram is plotted for a screen view or printable image, it is standard to show the decibel magnitude by linking the values to a colormap. In this way, the color plot can be used to show the magnitudes over the two-dimensional matrix. While full-color spectrograms can be useful with a good choice of colormap, it is a long-standing tradition that spectrograms are shown using a grayscale colormap, thus mimicking the appearance of the old analog spectrograms that were singed electrically onto Teledeltos paper. The highest amplitudes are then plotted in the darkest gray or black, and the gray gets lighter as the amplitudes get smaller. To facilitate readability of a digital spectrogram plot, it is important to use a graphical plotting routine that does ‘‘interpolated shading’’ from cell to cell of the spectrogram matrix. This makes the

4.2 Principles of the Spectrogram

87

digital spectrogram look ‘‘analog,’’ which is what humans need to be able to read it. It is possible to plot the digital spectrogram matrix literally as a matrix of blocks of different colors (this is, inexplicably, the way some spectrogram routines work), but the result is almost unreadable.

4.2.2 Development of Spectrogram Theory I already mentioned the spectrogram’s origins, and its ensuing influence on speech science, in the previous chapter. But the success of the spectrogram as an analysis tool in the early going relied solely on a device that produced spectrograms. A complete theoretical understanding of the spectrogram, which would ultimately facilitate the switch to digital computation, took about twenty more years to congeal. The first mathematical attack on this sort of time–frequency analysis was carried out by Gabor [15], in apparent ignorance of the existence of the spectrograph device (although Gabor did mention it in a footnote that was added following acceptance of his paper). Gabor developed a digital form of the inverse STFT formula mentioned above, which is usually called the Gabor signal expansion, but its relationship to the STFT analysis formula was not understood until many others had studied the idea decades later. Gabor did not himself characterize his representation as a short-time power spectrum. The short-time power spectrum was given its first rigorous mathematical treatment by Fano [7], but this was limited to a particular impractical form of the time window. The first short-time power spectrum allowing an arbitrary (continuous-time) window function was derived by Schroeder and Atal [29], although these authors did not identify their function (which was defined in continuous time essentially as in Eq. 4.1) as an invertible short-time Fourier transform—as Gabor had only developed the synthesis formula in the discrete regime, these authors developed only the analysis formula in the analog regime, so the relationship between the two was still muddled for a while. A further treatment was completed by Montgomery and Reed [27], who generalized Helstrom’s [19] work on Gabor’s signal expansions and examined both the synthesis and analysis formulae, showing for the first time that an STFT in the analog regime is generally invertible like a plain Fourier transform. By the 1970s, the analysis formula was standardly called the short-time Fourier transform, and in its digital ‘‘sampled’’ form this is most frequently said to be equivalent to the Gabor transform, although the two have disparate algorithmic aspects.

4.2.3 Uncertainty Principle The uncertainty principle, often named after the physicist Heisenberg, can be stated in the most general terms as in [10]: A nonzero function and its Fourier transform cannot both be sharply localized. Heisenberg introduced this as a

88

4 The Fourier Power Spectrum and Spectrogram

general constraint on observational precision in quantum physics. In that regime, the principle implies, among other things, that the position and momentum in a quantum state cannot both be precisely determined. In the regime of signal analysis, to which Gabor [15] introduced the uncertainty principle, it implies that a signal sðtÞ and its Fourier spectrum cannot both have a small domain. This is often stated in the literature by saying that a signal cannot be both highly timelimited and highly band-limited in frequency [10]. It is important to emphasize that, owing to the uncertainty principle, ‘‘obtaining an intrinsic and infinitely precise description of the ‘time–frequency content’ of a signal is out of the question’’ [6]. The above general uncertainty principle is a global constraint on an entire signal; when performing time–frequency analysis, however, we normally are interested in viewing some kind of breakdown of a complicated signal into ‘‘events’’ in time and ‘‘components’’ in frequency—it is local time–frequency content that interests us. Moreover, the usual discussion of uncertainty in signal processing mentions only the global signal uncertainty as above; when a signal is analyzed with a spectrogram, additional uncertainty is introduced by the windowing operation to yield what has been termed the spectrographic uncertainty [25]. Speaking in general terms, a spectrogram can be thought of as displaying a time–frequency distribution which has a statistical variance in both the time and frequency directions—this is why the image is smeared. Operating locally, at each time–frequency cell there will be a conditional time variance for that frequency and a conditional frequency variance at that time. Each of these will be partly determined from the window function used to compute the spectrogram. The product of these two conditional variances must be locally greater than a certain quantity; this is the local spectrographic uncertainty principle [25], which provides a lower bound on the possible precision of a spectrogram at each time–frequency cell. The spectrographic uncertainty is often described in speech science as the ‘‘resolution trade-off’’ between time and frequency. We will see in the next chapter that it is possible to work with other time–frequency representations that do not have nearly as much uncertainty as the spectrogram.

4.3 Spectrographic Analysis of Speech ‘‘There is no uniqueness of a time–frequency representation: there are many different ways of describing the ‘time–frequency content’ of a signal’’ [6]. Owing to the quoted fact, computing a spectrogram of a bit of speech is not something that can be done effectively without taking care to understand and set the values of a number of parameters which have a dramatic effect on the analysis and its appearance in a display. The most important of these are: • the length of the analysis window; • the particular window function to be used;

4.3 Spectrographic Analysis of Speech

89

• the number of points used to compute the Fourier transform of each windowed segment; • the frame advance; • the dynamic range of the amplitude plot; • whether to use pre-emphasis for the display. Let us now discuss each of these in turn.

4.3.1 General Guidelines A spectrogram provides a time–frequency representation of a signal that by nature assumes the signal is ‘‘short-time stationary,’’ meaning that the spectrogram is unable to detect any changes in the signal during the span of one analysis window. Accordingly, the most important parameter to be set when computing a speech spectrogram is the window length. A window whose effective length is shorter than one glottal cycle during voiced segments will provide the most generally useful and informative type of speech spectrogram, often called a ‘‘wideband’’ spectrogram in reference to the original filterbank computation method. With such a short analysis window, the spectrographic uncertainty principle causes the frequency resolution and precision to be fairly poor, with the consequence that the individual component frequencies such as formants are ‘‘smeared’’ in the frequency axis. Nevertheless, a wideband spectrogram is essential for observing the formant frequencies which are excited with each glottal impulse, and also for observing the brief events associated with consonants such as stop burst noises. By contrast, a window whose effective length encompasses more than two glottal cycles will show the fundamental frequency of voicing and the harmonics associated to it, and will no longer show formant frequencies as being directly excited. In this type of ‘‘narrowband’’ spectrogram, formants can be observed only indirectly as groups of harmonics having a higher intensity. The spectrographic uncertainty principle then causes brief events to be less resolved, and smeared in the time axis. The spectrogram with longer window is unable to detect any changes, such as glottal impulses, that occur within the window’s span. From the discussion of window functions for power spectra in the previous chapter, it was concluded that the optimal choices are drawn from either the Gaussian or Kaiser families of functions. It has been noted, however, that window performance for a single spectrum computation does not necessarily translate to the time–frequency regime for spectrograms. One study specifically examining spectrogram performance concluded that a Gaussian window provides optimal time–frequency localization (i.e. minimal smearing) [20]. For reasons of tradition, the Hamming and ‘‘Hanning’’ windows have been perennial favorites in speech analysis, but more recently the Gaussian window has begun to get promoted in our field as well. I would recommend that readers choose either Gaussian or Kaiser windows for creating spectrograms. Figure 4.12 compares spectrograms computed

90 5

4

Frequency (kHz)

Fig. 4.12 Spectrograms of the English word hide [haId], computed with 6 ms rectangular window (top), Hann window, and Gaussian window (bottom) using Praat

4 The Fourier Power Spectrum and Spectrogram

3

2 1

0 0

200

400

600

800

600

800

600

800

Time (ms) 5

Frequency (kHz)

4

3

2

1

0 0

200

400

Time (ms) 5

Frequency (kHz)

4

3

2

1

0 0

200

400

Time (ms)

with rectangular, Hann, and Gaussian windows. Decreased separation of F1 from the voice bar is observable in the Hann windowed example versus the Gaussian, as well as slightly broader formants.

4.3 Spectrographic Analysis of Speech

91

The number of points used to compute the Fourier transforms for the spectrogram will normally be at least as long as the analysis window, but beyond this it affects the frequency sampling of the resulting image (see the discussion of zeropadding in the preceding chapter). A 256-point frame, for example, will provide only 128 frequency bins to sample the entire frequency range whether it is plotted or not. Supposing we have a sound whose highest recorded frequency is 22.5 kHz, this is not a very rich frequency sampling. In general, I recommend using at least 1,024 points for the Fourier transforms, and 2,048 points is often useful as well. Unfortunately, many popular programs such as Praat do not allow the user to set the FFT frame size independently of the window length, and will automatically set the frame size to some value like the first power of 2 greater than the window length. Recall that a spectrogram is plotted by linking the decibel amplitude values over the time–frequency matrix to a colormap, usually a grayscale. We do not normally want to show the lowest amplitudes in the STFT matrix computed from a signal, since these are frequently just unwanted noise. The spectrographic parameter commonly called the dynamic range has to be set by the user; this is the value that sets how large an amplitude value needs to be to ‘‘make the cut’’ and get plotted in the spectrogram. It is standard for the dynamic range to be a decibel value that refers to the difference between the loudest and quietest amplitudes to be plotted. A good value for speech is usually around 50 dB; this means that time– frequency cells with amplitude more than 50 dB down from the loudest cells will simply be ‘‘clipped’’ out of the plot. A value of 30 dB will usually show too little of the time–frequency content of the signal, losing a lot of valuable information, while a value of 70 dB or greater would only be suitable for high-quality sound recordings made with a soundbooth and an expensive microphone, since otherwise a lot of noise will be shown as light gray in the plot. In the old days of analog spectrograms, the best dynamic range that could be obtained using the Teledeltos paper was around 35 dB. This presented a problem for speech applications, since the average person’s glottal source sound decreases in amplitude as the frequencies get higher, at a rate of -6 dB/octave. As a result, everyone’s F1 is generally their loudest formant, with F2 somewhat quieter, and so on up the frequency scale. A spectrogram of a vowel that is plotted using 35 dB of dynamic range will only show F3 very faintly, and higher formants may be missing from the plot. A solution to this that was employed in the early spectrograph devices was called pre-emphasis, and it involves artificially changing the spectral amplitude slope by some amount to counteract the natural roll-off of the voice source. A common standard pre-emphasis was 6 dB/octave, the precise opposite of the average person’s natural roll-off. The intended result was to even out the amplitudes of the formants across the frequency range. Pre-emphasis is generally effective for this purpose, but I find that in the modern world of spectrograms with much better dynamic range, it can also be detrimental to an accurate picture of the time–frequency content of speech. There is no longer quite as much reason to even out the amplitudes of speech formants artificially; a spectrogram with at least 50 dB dynamic range is usually good enough to show all the formants reasonably

92

4 The Fourier Power Spectrum and Spectrogram

Fig. 4.13 Spectrograms of the author’s name Sean [SAn], computed with 12 ms Gaussian window using Praat. Upper panel uses 40 dB of dynamic range with 6 dB/octave pre-emphasis, while the lower panel uses 55 dB of dynamic range with no pre-emphasis

well. The down side of pre-emphasis is chiefly that it increases the amplitude of high-frequency consonant noise far too much, preventing a realistic picture of the speech time–frequency content. For the most realistic spectrograms, I recommend against pre-emphasis, but anyone following this advice should turn up the dynamic range slightly so that high formants are not too quiet. Figure 4.13 shows two spectrograms of the same utterance, with and without pre-emphasis.

4.3.2 Short Window (Wideband) Analysis Wideband (short window) spectrograms have often been described in the literature as displaying poor frequency resolution and precision, which is true. When applied in speech science, however, it has often been written that such spectrograms are unable to resolve the harmonics of the glottal source, and hence the formants look like fat dark bars. That is to say, formants are supposedly so fat because there are a number of harmonics resonating within each one, and these are all mushed

4.3 Spectrographic Analysis of Speech

93

together in the spectrogram [23]. This way of understanding a wideband spectrogram of voiced speech is not correct. A better way of understanding wideband versus narrowband speech spectrograms is to think in terms of the ‘‘harmonic–inharmonic duality’’ of speech that was discussed earlier. A wideband spectrogram is the right sort of time–frequency analysis for showing the inharmonic aspect of voiced speech. Each formant is shown as it is excited by the glottal impulses, albeit as a fat smeared bar due to the spectrographic uncertainty principle. An important consequence of this view is that an often-stated ‘‘problem’’ stemming from higher fundamental voicing frequency is not necessarily a problem at all: The higher the fundamental frequency, the fewer [are] the harmonics which define the formant and the greater [is] the probability that the most prominent harmonic will be distant from the center frequency of the formant. [26]

The above quoted statement of the problem surely does apply if one elects to examine formants using a narrowband analysis showing the harmonics, but it need not apply to a wideband analysis because the latter does not literally portray a smeared version of the former. From the inharmonic model of voiced speech, we can recognize that each formant is excited by the glottal impulses nonetheless, whether the fundamental frequency is low or high. It is worth commenting that the supposed ‘‘problem’’ resurrects the forgotten debate over the accommodation theory that was chronicled in Chap. 3.

4.3.2.1 Vowels Wideband spectrograms of vowels are chiefly useful for showing the formants. Such images were also widely used at one time to measure the formant frequencies, but this is no mean feat because of the smearing. As a result, ‘‘manual’’ measurement of formants from spectrograms has fallen somewhat into disuse in recent years, at least for research studies where a large number of formants have to be measured.3 For the best possible localization of formants in a spectrogram, a window should be used whose effective length is only about 1 or 2 ms shorter than one glottal cycle. Shorter windows yield fatter-looking formants, while longer windows will begin to show harmonics. Figure 4.14 shows wideband spectrograms of two synthesized vowels, computed using 14 ms Gaussian windows (the effective length is about 7 ms). It is essentially impossible to distinguish F2 from F3 in [i], and F1 is difficult to discern from F2 in [O]. The low F1 of [i] is also very poorly localized. Since formants are excited with each glottal closure, one might think to try zooming in the analysis to show only a few glottal cycles. An attempt at this is shown for [O] in the figure, and

3

One anonymous grant reviewer once wrote to me that nobody uses spectrograms anymore, except for showing speech sounds to phonetics students.

94

4 The Fourier Power Spectrum and Spectrogram

Fig. 4.14 Spectrograms of synthetic vowels [i] (top), [O] (middle), [O] 40 ms segment, computed with 14 ms Gaussian window using Praat

it is somewhat surprising how much it does help in discerning the two lowest formants, in spite of the poor frequency localization. We have just seen that wideband spectrograms, while they do show formants, are not much good for measuring them precisely even in clean synthesized vowels which exactly obey the source-filter theory of speech production. Knowing this, one can only dread the prospect of measuring formants from spectrograms of real

4.3 Spectrographic Analysis of Speech

95

speech, and indeed the values are even less clear for in vivo vowels. The reasons for this are chiefly these two: first, real vowels generally have a voice bar, which is a low resonance around 200–250 Hz (higher for smaller vocal tracts) that is difficult to resolve from F1 in high vowels4; second, real phonation and speech does not precisely obey the source-filter theory because the air flow makes aeroacoustic processes relevant, and this in turn introduces complicating features into the spectrum. Figure 4.15 shows spectrograms of English words containing the vowels [æ, i, O]. The formants of [æ] are maximally separated from each other and from the voice bar, so in this case the challenge is minimal. The real trouble is evident in the other two spectrograms. F1 of [i] is difficult or impossible to separate from the voice bar, while F1 and F2 of [O] cannot be discerned from each other.

4.3.2.2 Obstruents Stop consonants have a number of attributes that are visible and measurable with the aid of a wideband spectrogram. These include the burst event, prevoicing during the closure, aspiration following release, breathy voicing following release, as well as vowel formant transitions going in to the closure and emerging from the release. To best observe these attributes, the wideband spectrogram should be set to have similar parameters to those most useful for vowels. For a given speaker, the window duration should be set to slightly less than one glottal cycle. Preemphasis is likely to be a very bad option for any obstruent spectrogram, since it will artifically increase the amplitude of all the high-frequency noise, making the stop burst and aspiration appear louder than it actually is. Measuring the formant transitions going in to and coming out of a stop is at least as problematic as measuring vowel formants in the first place, but a spectrogram can at least give some impression of the transitional values, and these are generally indicative of the place of articulation. A spectrogram is not the best thing for analyzing stop burst spectra; for this, a carefully windowed power spectrum (shown in the previous section) is probably the best tool. Figure 4.16 shows spectrograms of English words pronounced ½kh At] and [gAt], in which many of the attributes mentioned above can be observed. The initial ½kh  appears to begin with two closely spaced burst events, not an uncommon occurrence for velar stops. The bursts are followed by a period of broadband aspiration noise, during which it is possible to detect formants in the noise. The initial [g] begins with a voiced closure, which is visible as a voice bar. The vowel begins immediately after the stop burst, and so in this case the formant transitions out of the stop are easier to measure. This example shows a common characteristic of

4

The controversial notion that the voice bar is a separate resonance will be discussed in Chap. 6.

96

4 The Fourier Power Spectrum and Spectrogram

Fig. 4.15 Spectrograms of English words had [hæd], heed [hid], hawed [hOEd] spoken by the author in an American English dialect. Computed with 12 ms Gaussian window using Praat

velar stops, with F2 and F3 appearing to emerge from origin points very close to each other. Formant transitions can also be observed heading into the final [t] of both these words.

4.3 Spectrographic Analysis of Speech

97

Fig. 4.16 Spectrograms of English words caught [kh At], got [gAt] spoken by the author in an American English dialect. Computed with 12 ms Gaussian window using Praat

While fricative noise spectra are best quantified using the power spectrum techniques discussed previously, a wideband spectrogram can provide a general look at a fricative that allows one to check whether it is voiced, whether the noise spectrum changes appreciably through the duration (this will not be detected with a single power spectrum), and also permits examination of the formant transitions in surrounding vowels, which can help confirm a place of articulation. When examining fricatives, it is useful to use a frequency range with good extension in to the high values. The examples in Fig. 4.17 show that English [f] has noise extending over 14 kHz, and [s] extends above 11 kHz. It is also easy to observe the large differences between [f] and [v], resulting from the voicing diminishing the airflow power that is available to generate a noise with labiodental origin.

4.3.2.3 Sonorants Approximants and semivowels are chiefly characterized by formants, in similar fashion to vowels, although they may be very brief sounds. Accordingly, they can be observed using the same spectrographic parameters normally employed for

98

4 The Fourier Power Spectrum and Spectrogram

Fig. 4.17 Spectrograms of English word pairs fan [fæn], van [væn] (upper) and sip [sIp], shone [San], spoken by the author in an American English dialect. Computed with 12 ms Gaussian window using Praat

vowels. Figure 4.18 shows spectrograms illustrating word-initial [¤] and [l] in English. This variety of [l] is characterized by a very large frequency spread between its lower and upper formants. The English [¤] sound is quite uncommon in the world’s languages; it is characterized by a very low F3 (below 2,000 Hz), which can be seen rising rapidly out into the vowel in this example. The nasals have formants which can appear quite different from those of oral sounds. They are generated by the coupled oral and nasal cavities, and are generally more damped as a result, increasing their bandwidth; this can make nasal formants appear even more smeared in frequency than oral formants. The coupled cavities also cause the spectrum to contain at least one zero, or frequency at which energy is absent. It is impossible to directly observe a zero in a spectrogram, although it may be evident in a single power spectrum of a nasal. Figure 4.19 shows spectrograms illustrating word-initial [m] and [n] in English. There it can be observed that the different nasals do have slightly different upper formants (which are invariably very faint); nasals in general always have a strong resonance around 250 Hz, however [32], which is often called the ‘‘nasal murmur.’’ The different nasals are distinguished spectrally by their distinctive upper formants, but also by formant transitions in nearby vowels in similar fashion to the oral stops.

4.3 Spectrographic Analysis of Speech

99

Fig. 4.18 Spectrograms of English words light [laIt], right [¤aIt] spoken by the author in an American English dialect. Computed with 12 ms Gaussian window using Praat

4.3.3 Long Window (Narrowband) Analysis So far, only wideband speech spectrograms have been closely examined; these are generally the most useful for observing the acoustic correlates of phonetic articulations during speech. The term ‘‘narrowband’’ is traditionally given to spectrograms computed using an analysis window that is at least two glottal periods in duration. This increased time of assumed signal stationarity now ‘‘observes’’ the glottal periodicity in each window, and then the Fourier transform of each window contains the fundamental voicing frequency and its co-occurring harmonics. The spectrographic uncertainty induces the ‘‘time–frequency resolution tradeoff,’’ whereby the longer analysis window prevents good time localization of any brief events or transitions. Figure 4.20 compares a wideband (12 ms Gaussian window) with a narrowband (120 ms Gaussian) spectrogram of the English word syllable. The wideband image as usual shows the glottal impulses as vertical lines, the time precision is good, and the frequency precision is not good. In contrast, the narrowband image cannot resolve the glottal impulses, but the fundamental and its

100

4 The Fourier Power Spectrum and Spectrogram

Fig. 4.19 Spectrograms of English words map [mæp], nap [næp] spoken by the author in an American English dialect. Computed with 12 ms Gaussian window using Praat

harmonics are shown. Here the time precision is poor, however, and formants are shown only indirectly as groups of louder harmonics. It is evident from any example such as Fig. 4.20 that a narrowband spectrogram is not preferable to a wideband one for measuring formants. However, since the fundamental frequency and harmonics are shown relatively well, one can observe in this example that the pitch of the voice goes up and down over the course of the word. The narrowband spectrogram is, in fact, a reasonably good way to track and measure the pitch of the voice (usually equated with the fundamental frequency, but I will discuss this more at a later point), which is useful for studies of stress or intonation. A standard type of tool known as a pitch-tracking algorithm is most commonly employed for this purpose, but such algorithms are partly probabilistic, have to be heuristically guided, and have many parameters that must be tinkered with to achieve even modest performance. Figure 4.21 shows just the low-frequency range of a sentence portion in which the speaker says a stressed syllable is usually, viewed as a narrowband spectrogram with the pitch track provided from Praat’s algorithm overlaid as a white line. While the fundamental frequency is

4.3 Spectrographic Analysis of Speech

101

Fig. 4.20 Spectrograms of the English word syllable spoken by the author in a Canadian English sentence context. Top with 12 ms Gaussian window; Bottom with 120 ms Gaussian window

Fig. 4.21 Low-frequency range of narrowband spectrogram (120 ms Gaussian windows), showing the annotated phrase uttered in a sentence context. Praat pitch track is overlaid in white

observable in the spectrogram during voiced segments, it is not very well localized, which makes careful manual measurement of the center of the gray line a required procedure for measuring the voice pitch from this image. The pitch tracking

102

4 The Fourier Power Spectrum and Spectrogram

algorithm is semi-automated and the data can be stored as soon as it is calculated; nevertheless, anyone would agree that the results in this case (an unremarkable example) are far from excellent. In fact, the computed pitch track (which I tried to make as good as possible by tinkering with the settings) is only occasionally in agreement with the fundamental in the spectrogram; indeed, it is only occasionally able to be computed at all in this example. The spectrogram must be viewed as the gold standard here because it is just a time–frequency analysis and not a heuristic method, so this shows that a narrowband spectrogram is a more accurate and reliable pitch-tracking method than a state of the art algorithm.

4.4 Appendix: Praat and Matlab Techniques 4.4.1 Praat Functions To create vowel sounds using a formant synthesizer, select the following from the Object view: New   [ Sound   [ Create sound from Vowel Editor The vowel editor allows one to set four formants manually; the first two can also be set by positioning the cursor in a vowel space and listening for the result. The user also sets the duration and intonation of the synthesized sound. There are a few ways to get a power spectrum in Praat. The simplest is from the Edit sound view, where under the Spectrum menu there is a function to View spectral slice from a selected area. The window function will be whatever the spectrogram viewing windows are set to, under the spectrogram settings described below. Often it will be useful to obtain a spectrum object rather than just a view. First, the target sound segment may need to be extracted from a longer sound, and for this the function Extract selected sound (windowed) should be used from the File menu in the sound editing view. Upon selecting this function, a dialog box appears allowing the window function to be set; one of the Gaussian or Kaiser options is always recommended. There are two options for a spectrum object to be computed from a sound, using the Spectrum menu in the Object view. If a raw complex Fourier spectrum is desired or measurements are to be taken, then To spectrum should be selected. The resulting spectrum object can be exported to use the raw values, it can be viewed and measured using an edit view (in which it is shown as a power spectrum on a dB scale), or it can be drawn as a power spectrum in the Picture area. If only a power spectrum picture will be needed, a useful selection is To ltas, which stands for long-term average spectrum. As far as I can tell, the resulting power spectrum is the same as the one computed from a Spectrum object, but the ltas object can only be drawn in the picture area and cannot be measured using an edit view.

4.4 Appendix: Praat and Matlab Techniques

103

Spectral moments can be computed from a Spectrum object under the Query menu. Any of the central moments (which are normally used) can be precisely obtained from the selection Get central moment; alternatively, various statistical quantities related to the moments can be obtained individually, such as the mean (center of gravity, equal to the first ‘‘central’’ moment),5 skewness (a sort of normalized 3rd moment), and kurtosis (a sort of normalized 4th moment). The harmonicity of a target sound can be obtained directly from the sound object under the Periodicity menu, by selecting any of the harmonicity functions which compute a harmonicity object. The harmonicity object is a vector of dB values computed at a sequence of time points from the beginning to the end of the target sound, which may be queried at particular points, drawn as a graph, or exported as text. The values measure the harmonics-to-noise ratio using Boersma’s published algorithm [4]. A spectrogram is optionally displayed in Praat’s edit view of a sound, under the waveform. The appearance of this spectrogram is set by the spectrogram settings under the Spectrum menu of the edit view. The regular settings allow the user to set the frequency range shown, the analysis window length, and the dynamic range of the view. The window function is set to a Gaussian by default for this view; users should be aware that the Gaussian window length is automatically doubled for the computation, so that the length setting corresponds more closely to the effective length for the analysis. If 10 ms windows are really wanted, then 5 ms should be used for the setting. The advanced settings allow the user to make further adjustments including the maximum number of time points and frequency points for which the spectrogram is computed, but these settings should not normally require adjustment (v. the Praat manual for more information). The window tapering function can be changed here if desired, and the pre-emphasis function can be set to a specific amount of roll-off. Setting this to zero turns off preemphasis. It is also possible, under the menu obtained by pressing Spectrum from the object list, to compute a Spectrogram object from any sound object. The spectrogram object can then be optimally displayed in the picture section. When the object is first created, the user sets the window length and tapering function, as well as the highest frequency computed in the spectrogram. The default time and frequency steps are generally adequate here. Once the spectrogram object exists, then one selects Draw --[ Paint to display an image in the Praat picture section. It is at this point that a number of important spectrogram display options are set, including the time and frequency range to be displayed, and the dynamic range and pre-emphasis settings. All of the spectrograms presented in this chapter were created using this procedure.

5 Although commonly phrased this way, strictly speaking there is no such thing as a first central moment; the quantity commonly so called is technically known as the first moment about zero, with the nth central moment being generally defined in terms of the n  1th moment [33].

104

4 The Fourier Power Spectrum and Spectrogram

4.4.2 Matlab Code I have provided several m-files (a term for Matlab programs) for computing and displaying a traditional spectrum and spectrogram. The m-files discussed here also depend upon the presence of other ancillary m-files which are provided as part of the code package. A user should first learn to use the Matlab built-in function wavread to read sound files in the .wav format into a Matlab vector variable. ½signal; Fs; bits ¼ wavreadð‘yourfile:wav’Þ; The above Matlab command will read the contents of the named file into the vector signal, and will also extract the sampling rate as Fs and the quantization bit depth as bits. The bit depth is usually irrelevant for our purposes, but the sampling rate is usually of critical importance. The file powerspec.m provides a basic Fourier power spectrum from an FFT, and is invoked using ½PS; f ¼ powerspecðsignal; Fs; low; highÞ; where the spectrum vector can be stored as PS and the frequency axis vector as f. Using these variables is optional, so the functions here can always be called using only the command to the right of the equals sign if an image is all that is desired. In the above command, signal should be replaced with the name of the target signal vector stored in the workspace, Fs should be the correct sampling rate, and low, high should be the low and high limits of the frequency range to be shown. The file powerspec_ensemble.m provides an average power spectrum from a target matrix of signals, one in each column (so the signals must all be the same length). It is invoked using ½PS; f ¼ powerspec ensembleðsignals; Fs; low; highÞ; The file specgram_2010.m provides a conventional spectrogram of the sort provided by Praat, and is invoked using ½STFTpos; f; tforspgm ¼ specgram 2010ðsignal; Fs; window; overlap; fftn; low; high; clipÞ; in which the analysis window length (in samples) is set in the 3rd argument, the number of samples by which successive windows overlap is set in the 4th argument, the FFT frame size is set in the 5th argument, and the dynamic range is set as a negative dB value in the 8th argument. For example assuming a 44.1 kHz sampling rate, to compute a spectrogram with 1,024 frequency bins using 10 ms windows overlapped by 7 ms, use 441 for window, 309 for overlap, and 2,048 for fftn. The window function is set to Kaiser by default, but this can be changed by altering one line of the code. The optional storage variables can keep the displayed portion of the complex STFT, the frequency axis vector, and the time axis vector.

References

105

References 1. L. Auslander, C. Buffalano, R. Orr, R. Tolimieri, A comparison of the Gabor and short-time Fourier transforms for signal detection and feature extraction in noisy environments, in Proceedings of the SPIE Advanced Signal Processing: Algorithms, Architectures and Implementations, vol. 1348, pp. 230–247 (1990) 2. L. Auslander, I.C. Gertner, R. Tolimieri, The discrete Zak transform application to time– frequency analysis and synthesis of nonstationary signals. IEEE Trans. Signal Process. 39(4), 825–835 (1991) 3. M.J. Bastiaans, A sampling theorem for the complex spectrogram, and Gabor’s expansion of a signal in Gaussian elementary signals. Opt. Eng. 20(4), 594–598 (1981) 4. P. Boersma, Accurate short-term analysis of the fundamental frequency and the harmonicsto-noise ratio of a sampled sound, in Proceedings of the Institute of Phonetic Sciences, vol. 17, pp. 97–110, University of Amsterdam (1993) 5. P. Boersma, D. Weenink, Praat: doing phonetics by computer. Computer software (2009) 6. R. Carmona, W.L. Hwang, B. Torrésani, Practical Time–Frequency Analysis: Gabor and Wavelet Transforms, with an Implementation in S (Academic Press, San Diego, 1998) 7. R.M. Fano, Short-time autocorrelation functions and power spectra. J. Acoust. Soc. Am. 22(5), 546–550 (1950) 8. H.G. Feichtinger, T. Strohmer (eds.), Gabor Analysis and Algorithms (Birkhäuser, Boston, 1998) 9. K.R. Fitz, S.A. Fulop, A unified theory of time–frequency reassignment. Preprint posted on arXiv.org (2005) 10. G.B. Folland, A. Sitaram, The uncertainty principle: a mathematical survey. J. Fourier Anal. Appl. 3(3), 207–238 (1997) 11. K. Forrest, G. Weismer, P. Milenkovic, R.N. Dougall, Statistical analysis of word-initial voiceless obstruents: preliminary data. J. Acoust. Soc. Am. 84(1), 115–123 (1988) 12. S.A. Fulop, Accuracy of formant measurement for synthesized vowels using the reassigned spectrogram and comparison with linear prediction. J. Acoust. Soc. Am. 127(4), 2114–2117 (2010) 13. S.A. Fulop, C. Golston, Breathy and whispery voicing in White Hmong, in Proceedings of Meetings on Acoustics, vol. 4. Acoustical Society of America (2008) 14. S.A. Fulop, P. Ladefoged, F. Liu, R. Vossen, Yeyi clicks: acoustic description and analysis. Phonetica 60(4), 231–260 (2003) 15. D. Gabor, Theory of communication. J. IEE Part III 93(26), 429–457 (1946) 16. M. Gordon, P. Barthmaier, K. Sands, A cross-linguistic acoustic study of voiceless fricatives. J. Int. Phonetic Assoc. 32(2), 141–174 (2002) 17. K. Gröchenig, Foundations of Time–Frequency Analysis (Birkhäuser, Boston, 2001) 18. H. Helmholtz, On the Sensations of Tone, 2nd English edn. (Longmans & Co., London, 1885) 19. C.W. Helstrom, An expansion of a signal in Gaussian elementary signals. IEEE Trans. Inf. Theory IT-12, 81–82 (1966) 20. A.J.E.M. Janssen, Optimality property of the Gaussian window spectrogram. IEEE Trans. Signal Process. 39(1), 202–204 (1991) 21. A. Jongman, R. Wayland, S. Wong, Acoustic characteristics of English fricatives. J. Acoust. Soc. Am. 108(3), 1252–1263 (2000) 22. K. Kodera, R. Gendrin, C. de Villedary, Analysis of time-varying signals with small BT values. IEEE Trans. Acoust. Speech Signal Process. ASSP-26(1), 64–76 (1978) 23. P. Ladefoged, A Course in Phonetics, 5th edn. (Thomson, Boston, 2006) 24. P. Ladefoged, I. Maddieson, M. Jackson, Investigating phonation types in different languages, in Vocal Physiology: Voice Production, Mechanisms, and Functions ed. by O. Fujimura (Raven Press, New York, 1988) 25. P.J. Loughlin, L. Cohen, The uncertainty principle: global, local, or both? IEEE Trans. Signal Process. 52(5), 1218–1227 (2004)

106

4 The Fourier Power Spectrum and Spectrogram

26. R.B. Monsen, A.M. Engebretson, The accuracy of formant frequency measurements: a comparison of spectrographic analysis and linear prediction. J. Speech Hearing Res. 26(3), 89–97 (1983) 27. L.K. Montgomery, I.S. Reed, A generalization of the Gabor–Helstrom transform. IEEE Trans. Inf. Theory IT-13, 344–345 (1967) 28. S.H. Nawab, T.F. Quatieri, Short-time Fourier transform, in Advanced Topics in Signal Processing, Chap. 6, ed. by J.S. Lim, A.V. Oppenheim (Prentice-Hall, Upper Saddle River, 1988) 29. M.R. Schroeder, B.S. Atal, Generalized short-time power spectra and autocorrelation functions. J. Acoust. Soc. Am. 34(11), 1679–1683 (1962) 30. C.H. Shadle, Phonetics, acoustic, in Encyclopedia of Language and Linguistics, vol. 9, 2nd edn., pp. 442–460, ed. by K. Brown (Elsevier, Amsterdam, 2006) 31. C.H. Shadle, C.U. Dobelke, C. Scully, Spectral analysis of fricatives in vowel context. J. Phys. IV 2(Colloque C1), 295–298 (1992) 32. K.N. Stevens, Acoustic Phonetics (The MIT Press, Cambridge, 1998) 33. A. Stuart, J.K. Ord, Distribution Theory, Kendall’s Advanced Theory of Statistics, vol. 1 (Edward Arnold, London, 1994) 34. R. Wayland, A. Jongman, Acoustic correlates of breathy and clear vowels: the case of Khmer. J. Phonetics 31, 181–201 (2003) 35. P.D. Welch, The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans. Audio Electroacoust. 15(2), 70–73 (1967)

Chapter 5

Alternative Time–Frequency Representations

The spectrogram is a well-studied time–frequency representation, but there are numerous others. There has been a rich literature on this subject, and many different time–frequency representations have been devised, studied, and applied to various signal analysis problems (e.g. [1]). Unfortunately, the subject has never to my knowledge been made accessible to speech scientists, with the result that we have rarely availed ourselves of any such representations other than the spectrogram. This chapter is an attempt to rectify this situation somewhat, although the presentation takes on a more advanced mathematical character at certain points. Any time–frequency representation can be viewed pragmatically in two ways. Firstly, it can be thought of as an attempt to show how the energy in a signal is distributed in the time–frequency plane; certainly, a spectrogram can be viewed in this way. This view gave rise to the term distribution being used in place of representation much of the time. Quite often, however, a subject signal is in some physical sense known to consist of a number of distinct ‘‘components,’’ each having its own frequency that may be changing through time. Thus, just as a Fourier spectrum is often viewed as a ‘‘decomposition’’ of a stationary signal showing its frequency components, a time–frequency representation of a nonstationary signal may often be viewed as a decomposition showing how the various component signals change in frequency with the passage of time. In this context, the distinct elements of a multicomponent signal are called line components. What is really sought in many cases is then the instantaneous frequencies of the line components in a signal, although this is not a notion that is rigorously definable [13]. This chapter introduces the general theory of quadratic, or bilinear, time– frequency representations in the most gentle way I could come up with. In order to do this, it is necessary to discuss the Wigner–Ville distribution, which plays a central part in defining the class of quadratic distributions. My ultimate goal, however, is to promote a particular quadratic time–frequency representation called the Zhao–Atlas–Marks distribution, as a possible alternative to the spectrogram for speech analysis. The idea here is to describe the representation, show a few

S. A. Fulop, Speech Spectrum Analysis, Signals and Communication Technology, DOI: 10.1007/978-3-642-17478-0_5, Ó Springer-Verlag Berlin Heidelberg 2011

107

108

5 Alternative Time–Frequency Representations

examples comparing it to spectrograms, and enable the reader to be able to go ahead and try the ZAM distribution in various situations where a spectrogram might be applied. It is certainly not necessary to acquire a deep understanding of the theoretical points, any more than most speech scientists have ever understood the spectrograms which they have employed nevertheless.

5.1 Wigner–Ville Distribution The Wigner–Ville distribution (WVD) is of tremendous theoretical importance in time–frequency analysis [6]. It has many interesting and contradictory properties which make it at once appealing for applied signal analysis, but also virtually useless for this purpose. Understanding the WVD is important as an entrée into the realm of the ‘‘quadratic’’ time–frequency distributions, all of which are definable and understandable in terms of the WVD. It is useful to note at first that, while the STFT is a linear transform and is not a member of the quadratic class, the spectrogram derived from it is a quadratic distribution which can be defined using the WVD instead of the STFT. Considering the good points, the WVD of a linear frequency modulated sinusoid (i.e. a linear FM chirp) exactly shows the instantaneous frequency of the chirp, which is enticing for those of us interested in determining the instantaneous frequencies of signal components. Secondly, the theoretical WVD does not depend on a choice of window; it is in this sense the ‘‘canonical’’ time–frequency analysis of a signal, depending only upon the signal itself. On the minus side, unlike a spectrogram the WVD’s real values are quite often negative, which spoils the physical interpretability, and generally the images of a WVD are displayed to show only the positive part. Secondly, while the WVD can precisely follow the instantaneous frequency of certain simple test signals, for more complicated modulations or multicomponent signals, the bilinearity of the transform introduces interferences and ‘‘cross-terms’’ in the time–frequency distribution which make it virtually unreadable as an analysis of speech. Nevertheless, we will observe later in the chapter that a close relative of the WVD is in many respects an improvement upon the spectrogram for speech spectrum analysis.

5.1.1 Definition and Theory The Wigner–Ville distribution of a signal zðtÞ is defined using the following integral in continuous time [5]: Z1  s  s z t þ z t  eixs ds: Wz ðt; xÞ ¼ 2 2 def

1

ð5:1Þ

5.1 Wigner–Ville Distribution

109

It was first derived by the physicist Eugene Wigner for use in quantum mechanics, where he attempted to represent a probability distribution over paired physical properties which are not simultaneously measurable [21]. The distribution was thereafter derived in a signal analysis setting by Ville [20], who provided no evidence that he was aware of Wigner’s quasi-probability distribution of a similar nature. In the context of signals, time and frequency play the roles of the physical properties which are not simultaneously measurable. This distribution, like most other relatives including the Zhao–Atlas–Marks representation, works best for signal analysis when zðtÞ is the analytic signal associated to a real signal of interest. The core of the expression for the WVD clearly resembles the definition of autocorrelation, except that there are two time variables, s for the lag and t for the signal time. This function:  s  s ð5:2Þ Kz ðt; sÞ ¼ z t þ z t  2 2 is also known as the instantaneous autocorrelation, and we can now see that the WVD is its Fourier transform from lag to frequency. Since I already mentioned how a simple power spectrum is the Fourier transform of a non-normalized autocorrelation (i.e., an autocovariance), it is at least plausible that the WVD will serve as a kind of ‘‘instantaneous’’ spectrum, being as it is the Fourier transform of the instantaneous autocorrelation.

5.1.2 Discrete Implementation While the STFT was in some sense originally conceived as a discrete transform by Gabor, a fair amount of ingenuity was involved in the initial formulation of a discrete Wigner–Ville distribution by Claasen and Mecklenbräuker [10]. The discrete version of the WVD can be obtained by sampling the continuous distribution, but some snags have to be dealt with. The method for N samples of a digital signal zðnÞ with sampling rate fs involves sampling the instantaneous autocorrelation function at the core of the transform, using integer index m to represent the lag samples and integer index k to represent the discrete frequency bins as in a DFT. The correctness of the following expression is proven in [8]:  Wz

n kfs ; fs 2N

 ¼2

      nþm  nm i2pkm z z exp fs fs N jmj\N=2 X

ð5:3Þ

Changing variables to get rid of the sampling rate yields the following expression for the discrete Wigner–Ville distribution of zðnÞ:

110

5 Alternative Time–Frequency Representations

X



i2pkm Wz ðn; kÞ ¼ 2 zðn þ mÞz ðn  mÞ exp N jmj\N=2 def



 ð5:4Þ

By comparing with Eq. 2.35 for the DFT, it can be seen that the above expression involves a DFT of the sampled instantaneous autocorrelation (assuming the necessary periodicity of the DFT), and so the above definition can be slightly recast in a format friendly to computation [8]: Wz ðn; kÞ ¼ 2 DFTfzðn þ mÞz ðn  mÞg: m!k

ð5:5Þ

An improvement upon the above specification of a discrete WVD has recently been published [19], but I hope I am correct in my judgement that the practical differences for our purposes will be slight. In order to implement the above equation a number of finer algorithmic points need to be dealt with, which unfortunately seem never to have been discussed at length in the signal processing literature. To expound on this I am relying on a few tips scattered around [2, 9], together with my own analysis of some public domain Matlab code released by Auger and colleagues [4]. Given a digital signal sðnÞ; the following steps can be taken to compute its discrete WVD. The reader is referred to the linked Matlab code for exact details. 1. Compute the associated analytic signal zðnÞ; 2. Initialize a vector t of numbers from 1 to the length of the signal (in samples), stepping uniformly by some number step. 3. The length N of the lag window must be set; it is normally input by the user. 4. Initialize the discrete Wigner–Ville matrix with N frequency bins and length(t) time columns. 5. For each time column ti positioned at a sample point in time vector t, set up the lag window vector tau to be centered around 0 with length N, or some smaller length as needed to prevent the lag window from running beyond the ends of the signal. 6. Optionally, initialize a tapered window function (e.g. Kaiser, Gaussian) the same length as the lag window. 7. Using the current lag window, compute the current time column of the instantaneous autocorrelation matrix as zðti þ tauÞz ðti  tauÞ; optionally multiplying pointwise by the tapered window function; using a tapered window will produce a discrete pseudo-Wigner–Ville distribution that is smoothed in frequency. 8. When the autocorrelation matrix has been constructed column-by-column in this fashion, the discrete WVD is computed as the real part of the FFT of the matrix. The above outlined procedure contains a number of parameters not present in the ‘‘pure’’ Wigner–Ville distribution. First of all, we set a time step of a certain number of samples; this amounts to a time frame of samples over which the signal is assumed to be unchanging. In practice, this ‘‘time window’’ decimates the

5.1 Wigner–Ville Distribution

111

transform to reduce computational requirements, and should be ‘‘smaller than the local time of stationarity’’ [9]; to approximate the pure WVD, the value of step should be set to 1 (i.e. not decimated). Secondly, the procedure sets up a ‘‘lag window’’ that is possibly shorter than the entire signal; again this is to facilitate computation, and to approximate the pure WVD the value N should be set to the entire signal. Strictly, the WVD with a lag window is called the pseudo-Wigner– Ville distribution [17]; as with the spectrogram, a rectangular window is not the best choice, and it is conventional to multiply by a tapered window function wðsÞ in the lag domain. A window in the lag domain has the effect of smoothing in the frequency dimension, and is often referred to as a frequency-smoothing window. The action of a lag window as a frequency smoothing window derives from the following identity for a signal zðtÞ [12]: Z1 1

Z1  s   s i2pms hðsÞz t þ z t  e ds ¼ Hðm  f ÞWz ðt; f Þ df ; 2 2

ð5:6Þ

1

in which hðsÞ is a lag window, Hðm  f Þ is its Fourier transform involving a second frequency variable m known as the Doppler (it is formally dual to the lag as frequency is dual to the time), and Wz ðt; f Þ is the Wigner–Ville distribution of the signal.

5.1.3 Features of the Wigner–Ville Distribution The Wigner–Ville distribution is of interest here chiefly to allow the introduction of other quadratic time–frequency distributions, but let me illustrate some of its features in contrast with the spectrogram. The spectrogram is fundamentally a short-time analysis, meaning the choice of time window is its most important parameter. The WVD, by contrast, does not necessarily involve a window as such, except where needed to make computation tractable. Fundamentally, in its unvarnished form the WVD provides a joint time–frequency representation of an entire signal at once, by making use of the instantaneous autocorrelation function. These factors are illustrated in Fig. 5.1, in which spectrograms are compared with a WVD of a simple double sine wave. The upper panel, with rectangular windows, shows the extreme importance of a tapering function for a spectrogram; there is a great deal of spectral leakage everywhere. The middle panel, with 100 ms Gaussian windows, shows how even a long window like this does not produce a spectrogram with fabulous frequency localization; the sine waves are rendered as fat bands. The Wigner–Ville distribution in the bottom panel, by contrast, has remarkable frequency localization on the sine waves, but it is contaminated by the presence of a ‘‘false component’’ at a frequency midway between the true components. Such artifacts are called crossterms, and their presence in every WVD is what makes the representation all but

112

5 Alternative Time–Frequency Representations

Fig. 5.1 The top two panels show spectrograms of a signal consisting of two sine waves (50 and 150 Hz); both are computed using 100 ms analysis windows, only the middle panel uses a Gaussian taper function. The lower panel shows a Wigner–Ville distribution of the same signal, computed with 0.512 s rectangular lag windows, and 1.56 ms time steps. This somewhat approximates the ‘‘pure’’ WVD

useless for speech analysis. Technically, a spectrogram still has cross-terms, but they only arise between components or events whose STFTs overlap [15]—i.e., when the components (or events) are too close in frequency (or time) to be fully resolved because of the spectrographic uncertainty principle. Figure 5.2 shows four Wigner–Ville distributions of a sine wave with an up and down frequency modulation in the middle. The top left is a ‘‘pure’’ WVD (approximated as closely as possible by the discrete algorithm here). This representation is highly localized on the signal itself, but is once again contaminated by interference.

5.1 Wigner–Ville Distribution

113

Fig. 5.2 Four Wigner–Ville distributions showing a sine wave with up and down linear frequency modulation. Top row: 1 ms time step, full signal lag window (approximates pure WVD); 100 ms time step, 410 ms rectangular lag window. Bottom row: 1 ms time step, 5 ms rectangular lag window; 1 ms time step, 5 ms Kaiser lag window

These are not cross-terms since there is only one component here, but are instead known as inner interferences [12] resulting from a modulating component interfering with itself; either way, they are a most undesirable feature of the WVD. The top right panel uses 100 ms time steps, which as in a spectrogram has the effect of assuming the signal is unchanging over 100 ms increments. You can see that this assumption is totally unwarranted for the signal at hand, so that the linear frequency modulation is no longer properly located. The bottom left goes back to the fine 1 ms time steps, only this time a 5 ms rectangular window in the lag domain is employed, thus making a pseudo-Wigner–Ville distribution. This has a smoothing effect in the frequency domain, which eliminates a large amount of the interference. The bottom right shows a pseudo-Wigner–Ville distribution which is equivalent to the left panel, but with a Kaiser window function applied to the 5 ms lag frames. It compares very favorably to the two spectrograms of the same signal shown in Fig. 5.3. You can see that the pseudo-WVD seems to be quite a useful representation because of the tapering function. In general, however, this simple approach to smoothing the distribution is not very effective at eliminating crossterms, so we next move to the more successful and more complicated approach involving generalized quadratic distributions.

5.2 Zhao–Atlas–Marks Distribution We have so far seen that the Wigner–Ville distribution has excellent theoretical time–frequency resolution, but even in ‘‘pseudo-’’ form with a tapered lag window, it displays too much interference and cross terms to be a useful analytical tool.

114

5 Alternative Time–Frequency Representations

Fig. 5.3 Two spectrograms of the same signal from the previous figure. Top: 200 ms Gaussian window; bottom: 20 ms Gaussian window

The spectrogram displays very little interference and cross terms, but its time– frequency resolution is not so good. The Zhao–Atlas–Marks distribution family [25] (also known as the ‘‘cone-shaped kernel’’ distributions) is one example of a good compromise; its resolution is ‘‘quite superior to the spectrogram and the interference is quite insignificant’’ [3]. The ZAM has properties that are wellsuited to examining speech signals [3], and which can improve upon spectrograms in many cases. ‘‘Heuristically, we can say that the time–frequency smoothing by a spectrogram is based on only one ‘degree of freedom,’ as it employs a unique short-time window’’ [12]. With the ZAM, however, it is possible to take both the time and frequency dimensions into consideration, and proceed to develop smoothing that has two degrees of freedom.

5.2.1 Quadratic Distributions I showed in the previous section that the Wigner–Ville distribution derives directly from the instantaneous autocorrelation function by a Fourier transform in the lag

5.2 Zhao–Atlas–Marks Distribution

115

variable. It was also shown that the pseudo-Wigner–Ville distribution in turn derives from the pure WVD through multiplying by a lag window wðsÞ inside the Fourier transform, and that this can smooth the frequency dimension of the WVD. It was first shown by Cohen [11] in a quantum mechanics context that a more general approach to smoothing the WVD is both possible and desirable; instead of just multiplying by a lag window, one can perform a time convolution1 of the instantaneous autocorrelation with a two-dimensional function Gðt; sÞ called the time-lag kernel, and this has the effect of smoothing both the time and frequency dimensions of the final distribution. The following expression defines the general class of quadratic time– frequency distributions which can be computed in this fashion: n h s  s io ð5:7Þ qz ðt; f Þ ¼ F Gðt; sÞ z t þ z t  t s!f 2 2 where the symbol F is the Fourier transform operation and  indicates a time t

convolution. A great deal of research in time–frequency analysis has been devoted to the development of useful kernels within this so-called ‘‘Cohen class.’’ It is also important to remember that the spectrogram itself can be defined and computed in a similar fashion to the WVD, thus showing that the spectrogram is a special member of Cohen’s class—in particular it is a type of quadratic distribution which is always positive and has maximally suppressed interference and cross-terms.

5.2.2 Discrete Implementation Converting the expression (5.7) for a general quadratic time–frequency distribution into discrete time–frequency requires a number of steps similar to the derivation of the discrete WVD; the reader is referred to [6] for an account of the steps which lead from the above equation to the discrete time–frequency version: qz ðn; kÞ ¼ 2 DFTfGðn; mÞ½zðn þ mÞz ðn  mÞg m!k

n

ð5:8Þ

where n is the sampled time, m is the sampled lag, and k is the sampled frequency. The above definition is intended to be friendly to computation; a rough algorithm for computing any particular quadratic time–frequency distribution from a digital signal sðnÞ is equal to the earlier algorithm for the discrete WVD, with the addition of one step following (or during) computation of the instantaneous autocorrelation. That step is to perform a discrete convolution in n (time) with the discrete form of the smoothing kernel Gðn; mÞ: In some cases, an approximation to the discrete convolution may be the best option, owing to difficulties with the discrete-time version of the smoothing function. Detailed considerations on this point are presented in the math box below. 1

The process of convolution will be defined below in the discrete-time context.

116

5 Alternative Time–Frequency Representations

When the smoothing kernel for the Zhao–Atlas–Marks distribution family is converted to its discrete counterpart, it takes the following form parameterized by a [8]: h  an i Gðn; mÞ ¼ wðmÞ rect  sincðnÞsincðmÞ; ð5:9Þ 4m in which wðmÞ is a typical kind of tapering function (Kaiser, Gaussian), rect(x) is the standard rectangular step function [23], sincðxÞ ¼ sinðpxÞ px is the standard (normalized) cardinal sine function [24], and where the double convolution is required in continuous-time before sampling the lag vector. This is computationally impossible, and can only be approximated using digital oversampling. A less computationally intensive way is to devise a different discrete time-lag kernel which approximates the effects of the convolutions in the above expression. Two possible approximate ZAM kernels are [7]:  def wðmÞ if janj  j2mj ð5:10Þ Gðn; mÞ ¼ 0 otherwise; def wðmÞ

Gðn; mÞ ¼

2

½1 þ tanhðj4mj  j2anjÞ

ð5:11Þ

The algorithm due to Auger et al. [4] employed in the linked Matlab code computes the ZAM distribution using discrete convolution in time only (see below for a definition) of the instantaneous autocorrelation with the first approximate kernel above. This method also adds a separate time window using a convolution operation, so the implementation is really a timesmoothed ZAM. The discrete convolution of two complex-valued functions f ; g on the integers (e.g. sampling indices) is given by [22]: def

ðf  gÞðnÞ ¼

1 X m¼1

f ðmÞgðn  mÞ ¼

1 X

f ðn  mÞgðmÞ

ð5:12Þ

m¼1

The algorithm employed here for the ZAM distribution employs both a time window and a lag window, in contrast to the original ZAM distribution which has a lag window only. The two windows contribute to smoothing the interferences in the time and frequency dimensions respectively; the user can potentially change the tapering function that is applied in each window, a flexibility that generalizes the original Zhao–Atlas–Marks distribution into what we might call the smoothed ZAM. In addition to changing the smoothing window, a user also has independent control over the time frame used to smooth in time versus frequency; this is in contrast to the spectrogram where one always suffers from the trade-off between time and frequency smoothing/resolution. Since there is only one window for us to

5.2 Zhao–Atlas–Marks Distribution

117

set when computing a spectrogram, it affects both time and frequency localization. When the spectrogram time window is long, one sees good frequency localization but poor time localization; when the window is short, the opposite is true. With the ZAM, the time window does not affect the apparent localization of frequencies, and the lag window does not affect the time localization. Figure 5.4 illustrates four ZAM images of an artificial signal composed of two sine waves, one of which includes a linear frequency modulation up and down. The different window values illustrate the effects of varying degrees of smoothing/ resolution in each dimension. A shorter time window yields better time resolution in the ZAM, without any trade-off in the frequency dimension. A longer lag window yields better frequency resolution, with no loss of time resolution. It should be noted that, as with the WVD images presented above, these ZAM images discard negative amplitudes which may be present in the raw transform. The top left panel of Fig. 5.4 shows the ZAM of the signal with a