Dictionary Use in Foreign Language Writing Exams: Impact and implications (Language Learning & Language Teaching)

Dictionary Use in Foreign Language Writing Exams

Language Learning & Language Teaching (LL<) The LL< monograph series publishes monographs, edited volumes and text books on applied and methodological issues in the field of language pedagogy. The focus of the series is on subjects such as classroom discourse and interaction; language diversity in educational settings; bilingual education; language testing and language assessment; teaching methods and teaching performance; learning trajectories in second language acquisition; and written language learning in educational settings.

Editors Jan H. Hulstijn

Department of Second Language Acquisition, University of Amsterdam

Nina Spada

Ontario Institute for Studies in Education, University of Toronto

Volume 22 Dictionary Use in Foreign Language Writing Exams. Impact and implications by Martin East

Dictionary Use in Foreign Language Writing Exams Impact and implications

Martin East Unitec New Zealand / The University of Auckland

John Benjamins Publishing Company Amsterdam / Philadelphia

8

TM

The paper used in this publication meets the minimum requirements of American National Standard for Information Sciences – Permanence of Paper for Printed Library Materials, ansi z39.48-1984.

Library of Congress Cataloging-in-Publication Data East, Martin.   Dictionary use in foreign language writing exams : impact and implications / Martin East.        p. cm. (Language Learning & Language Teaching, issn 1569-9471 ; v. 22) Includes bibliographical references and index. 1.  Composition (Language arts)--Ability testing. 2.  Rhetoric--Ability testing. 3.  Encyclopedias and dictionaries--Use studies.  I. Title. P53.27.E17    2008 418.0071--dc22 isbn 978 90 272 1983 1 (Hb; alk. paper)

2008010429

© 2008 – John Benjamins B.V. No part of this book may be reproduced in any form, by print, photoprint, microfilm, or any other means, without written permission from the publisher. John Benjamins Publishing Co. · P.O. Box 36224 · 1020 me Amsterdam · The Netherlands John Benjamins North America · P.O. Box 27519 · Philadelphia pa 19118-0519 · usa

Table of contents

Preface Acknowledgments List of key acronyms chapter 1 What is the problem with dictionaries?

vii xi xiii 1

chapter 2 On dictionaries and writing

13

chapter 3 Does the dictionary really make a difference?

37

chapter 4 How do test takers use dictionaries?

65

chapter 5 When the dictionary becomes a liability

95

chapter 6 What do the test takers think of having a dictionary?

125

chapter 7 Some more test taker perspectives

147

chapter 8 Having a dictionary in writing exams – is it useful and is it fair?

169

chapter 9 Maximizing the opportunity and minimizing the liability

187

vi

Dictionary Use in Foreign Language Writing Exams

References

203

appendix 1 A note on intermediate level students

211

appendix 2 A note on data and procedures to establish reliability

213

appendix 3 The writing tasks used in the first two studies

217

appendix 4 A note on inferential statistics

221

Index

227

Preface

Dictionaries – friend or foe? On my latest visit to Berlin, back in 1998, I was asked to make a speech, in German, to the mayor and other dignitaries of the district of Wilmersdorf. A colleague and I from the UK had been regularly accompanying groups of 18-year-old high school students of German who were undertaking two weeks of work experience throughout the region – an excellent opportunity to practice their German in the real world. This was part of an exchange program that also saw groups of German students come to England for similar purposes. German is a foreign language to me, although I consider myself communicatively competent in it. Nevertheless I wanted to make sure that my speech was as accurate and meaningful as possible, and so in my preparation I turned to my bilingual English/German dictionary. I had been used to using a dictionary since I first started learning German, way back in school. I had come to rely on it as a trusted ally and effective tool – one that, in the words of Canale (1983), would help me to make up for gaps in my knowledge and enhance the rhetorical effect of what I wanted to say. With the help of the dictionary I was able to make the speech what I wanted it to be, and, so far as I could tell, it was appropriate and well received. Back home in England, the whole issue of allowing students to use bilingual dictionaries was very much at the forefront of my thinking. As the head of a large modern languages department where almost every student in my school was expected to study a foreign language up to the General Certificate of Education – the first high stakes examination for 15- to 16-year-olds – I was grappling, with my team, with the knowledge that, for the first time, these students were going to be allowed to take a bilingual dictionary with them into their written examinations. We were also preparing to allow our students at Advanced (A) level – the higherlevel high stakes examination for 17- to 18-year-olds – to use dictionaries. At A level we believed we could recommend a more comprehensive and sophisticated dictionary. After all, these were more advanced students who already had a fair degree of language proficiency. For our GCSE candidates, many of

viii Dictionary Use in Foreign Language Writing Exams

whom were quite weak in the language they were studying, it was a more complex decision. This was a totally new experience for us, but one that I, as a frequent user of bilingual dictionaries, believed in. We put a lot of thought into the type of dictionary we would recommend, and into how we were going to help our students to get the best out of it. We decided on a dictionary and we asked all the students to buy the same type so that we could train them in its use. So far so good. But I remember what happened when it came to final examination time. I was scheduled to supervise the reading test. All the students turned up with their shiny new dictionaries and, when the announcement to start the exam was made, set down to work. I remember my heart sinking as I witnessed several students with their noses seemingly buried in the dictionary, apparently oblivious to the passing of time and either unaware of, or unable to deal effectively with, the need to use their dictionaries sparingly and judiciously. Oh, how we had warned them to take care. How we thought we had prepared them for this eventuality. But in the heat of the examination context several of the students seemed to forget our advice. It is impossible to know how much this experience may have impacted on their performance, whether for good or ill, but I remember it to this day. I regarded the dictionary as a friend. Could it be that, for some students of a foreign language, it was more like a foe? But I was soon to leave England for New Zealand, and I put the experience behind me. Dictionaries were not allowed in New Zealand school exams. I was also teaching German at a tertiary level. My students, almost without exception mature students with high levels of motivation to learn German, were keen to find ways to improve their learning. And I was keen to share with them my belief that, used effectively, the bilingual dictionary was a valid and useful tool of the trade. I made sure that my students had ample opportunity to use dictionaries in class. But I had to draw the line at allowing them in exams. At the end of 2000 I took a trip back to England. I was chatting one day with an ex-colleague. In the course of our conversation on the comparative state of foreign languages teaching in the UK and in New Zealand, she remarked to me “by the way, did you know that they’ve changed their minds about dictionaries? Very soon now they’re going to be banned from the exams.” I must admit to being surprised and a little disappointed by what she had to say. My colleague also admitted to her own bewilderment about the decision. Her remark got me thinking. Why the sudden change of heart? On what basis had the policy makers made this decision? To my mind using a dictionary was quite a reasonable thing to do. Why take away from the students a legitimate resource? What signal were we giving to the students about where dictionaries should fit in as they attempted to use a foreign language? On the other hand, I had witnessed first hand a potential problem with allowing the dictionary in



Preface

an ­examination. Could it be that similar experiences had been reported, and the testing authorities had genuine concerns? Did they look back in hindsight and conclude that the experiment wasn’t working? I wondered whether anybody had carried out any research to find out what students actually did when they had a dictionary in an exam. In other words, was there any evidence to back up the policy change of heart? I tracked down several studies into what teachers and students thought about using dictionaries in the exams, but I wanted something that would show me what actually happened in a testing context. I managed to locate one study into bilingual dictionaries in GCSE writing. Two researchers had worked with groups of French students who were being prepared for the new ‘with dictionary’ exams. They asked them to take two GCSE-type writing tests, one with and one without a dictionary. Then they compared their scores. The researchers concluded that using a dictionary made an improvement. From my reading it seemed that dictionaries were subsequently banned on the basis of this conclusion. But there was also a concern about the mistakes the test takers made when using dictionaries, and the amount of time it seemed to take the test takers to use them – something I had witnessed myself. My biggest concern as I read the research and as I looked at what was happening in the UK was this: the research had focused only on the GCSE – and yet, when the announcement was made to remove dictionaries from the GCSE, they were removed from the A level at the same time. I thought about the very first question of the GCSE writing examination which required students to make a list of ten items – things you might take on a picnic, for example, or things you might need on holiday. Surely if you had a dictionary it would not be surprising if you could improve your score on this sort of question. On the other hand, I knew that many GCSE candidates were weak in their language of study. Perhaps dictionaries at GCSE level were more of a liability than a supportive tool for them, and in this case outlawing the dictionary made sense. But the research had not taken account of more proficient learners, those who might be working towards the A level, and who perhaps needed a dictionary all the more because of the higher levels of authentic language they were dealing with. I wondered whether any of these more proficient students, faced with having to write a speech for a set of dignitaries in Germany, and told ‘no, you can’t use a dictionary’, would be able to accomplish this task with the sense of confidence I had enjoyed. Was there any justification for banning the dictionary at A level? This background formed the impetus for the studies I describe in this book. I wanted to investigate what would happen if you allowed higher level more proficient users of a foreign language to take a dictionary with them into a writing exam. I made a deliberate choice to investigate German, in contrast to the study into French I had read about. I also made a deliberate choice to investigate

ix



Dictionary Use in Foreign Language Writing Exams

those who had reached at least the intermediate level in German and who possibly needed a dictionary all the more, or who should theoretically be able to use a dictionary more successfully. The three studies I describe were designed as case studies that would enable me to look at dictionary use from a variety of angles. In the first two studies I worked with my own students, all of whom were taking intermediate (or in some cases advanced) level courses in German language in a New Zealand tertiary institution. For the third study I recruited high school students from a number of local schools, all of whom were studying German for an A level equivalent examination. One focus of my studies was to compare students’ performance in two writing tests, one with and one without a dictionary, to find out whether they did better or worse in the ‘with dictionary’ test, or whether there was no difference. I looked at the marks they were given. I looked at the quality of their writing and the types of words they had used. Another focus was to find out what the test takers had to say about the two tests. What did they like? What did they dislike? Which test did they prefer? Which did they think was a fairer test of their writing skills? The multi-faceted nature of the studies meant that I was able to get a qualitative picture of what a whole range of students thought and did. I was also able to quantify what I found out, so that I could look at trends. One aim of this book is to describe the studies in as clear a way as possible. I hope you will enjoy reading about the test takers’ experiences and what all of this means for language testing. I am also aware that the writing of many students is now assessed through coursework options that allow dictionaries. Another aim of this book is to help language students, at whatever level, and whatever language they are studying, to get the best out of bilingual dictionaries when writing. It is also to help language teachers to help their students to use this resource to full effect. Ultimately, my goals are that what I present here will contribute to ongoing debates around using or not using dictionaries in writing, whatever the context, and that what I have to say will keep these debates going. Martin East Auckland, July 2008

Acknowledgments

No research project occurs in isolation; on the contrary, a project arises out of engagement with the academic community. Many individuals have helped to shape the work that I have presented in this book. Three key people first set me on the research road – Cathie Elder, Melbourne University, Roger Peddie, Auckland University, and Shawn Loewen, Michigan State University. I am indebted to them for their input as research mentors. Three representatives of the Applied Linguistics Association of New Zealand (ALANZ) – John Bitchener, AUT University, Gary Barkhuizen, Auckland University, and Cynthia White, Massey University – judged the research on which this book is based to be of sufficient quality for an ALANZ award. As a consequence I was inspired to consider releasing the research in book form. Ute Knoch, Melbourne University, and Lynn-Elizabeth Hill and Anna Beltowski, Unitec New Zealand, kindly read the first draft of the manuscript and offered me invaluable suggestions for improvement. I am particularly grateful to Ute for giving me her input on the many examples of German I use. On behalf of John Benjamins Publishers, Kees Vaes, Jan Hulstijn, and Nina Spada provided positive and enthusiastic support for and comments on the manuscript which has led to it becoming part of the Language Learning and Language Teaching series. While all of those mentioned have contributed to an improved final manuscript, none is responsible for any remaining shortcomings, errors, or omissions, for which I accept responsibility. It would be impossible to name all the other individuals who supported me in one way or another as I carried out the research, but I would particularly like to mention the following: Nick Shackleford, Jürg Brönnimann, Sue Walter, Janet von Randow, Sheena von Bassewitz, Elena Kolesova, Alan Kirkness, Judith Geare, and Margaret Hardiman. My particular thanks go to the research participants, my own students and other students, for consenting to take part in the project in the first place, and without whom the research would definitely not have been possible. Several individual facets of the studies have been published in other fora, and I would like to thank those who have given me permission to include aspects of previously published material, each of which is acknowledged in the reference list. A number of these publications were primarily aimed at local audiences and local contexts, particularly in New Zealand and Australia. I am, however, also ­grateful

xii Dictionary Use in Foreign Language Writing Exams

for permission to include material published more widely, including extracts from Elsevier Publications (Assessing Writing, 2006) and SAGE Publications Limited (Language Testing, 2007). I acknowledge HarperCollins Publishers for allowing me to reproduce dictionary entries from the Collins German Dictionary (Terrell, Schnorr, Morris, & ­Breitsprecher, 1997), in Chapters 5 and 9, and the Collins Pocket German Dictionary (Collins, 2001), in Chapter 5, and Edexcel Limited for permission to reproduce, in some cases in adapted form, writing examination questions previously used in the United Kingdom’s Advanced level German examinations (Appendix 3). Finally, and on a more personal note, I would like to acknowledge the significant contribution to my research endeavors which my wife, Julie, has made over the years – as mentor, critic, and supporter. My grateful thanks go to her.

List of key acronyms

A level ARG AS level BA CEF CLT GCSE L1 L2 LEP LFP OU PED QCA TLU

Advanced level Assessment Reform Group Advanced Supplementary level (up to 2000); Advanced Subsidiary level (from 2001) Bachelor of Arts Common European Framework Communicative Language Teaching General Certificate of Secondary Education First language Second / foreign language Limited English Proficient Lexical Frequency Profile Open University Portable electronic dictionary Qualifications and Curriculum Authority Target Language Use

chapter 1

What is the problem with dictionaries?

The dictionary, according to Kirkness (2004), is well established as “an essential source, if not indeed the principal source, of information on language for all members of literate societies who might have questions on any aspect of the form, meaning, and/or use of a word or words in their own or in another language” (p. 54). Kirkness goes on to talk of second or foreign language (L2) learners, and claims that for these learners bilingual dictionaries continue to be the most-used reference book at all levels. Ilson (1985) describes the dictionary as “the most successful and significant book about language.” For Ilson, its significance is “shown by the fact that … its authority is invoked, rightly or wrongly, to settle disputes” (p. 1). If this is so, Carter and McCarthy (1988) make a curious observation when they suggest that dictionaries for language learning have been largely ignored in the plethora of books and articles that have been published by those who have a stake in language learning and teaching. But recent events in language testing practices, particularly in the United Kingdom, highlight the more central position that bilingual dictionaries are now taking, especially in debates about their use or non-use by L2 students. As Horsfall (1997) puts it, it might almost have seemed that a dictionary was redundant in the foreign languages classroom. That is, however, until fairly recently. In the late 1990’s the UK’s Qualifications and Curriculum Authority (QCA) – the governing body that develops and maintains the National Curriculum for UK schools and regulates its associated examinations and assessments – made a bold and radical decision with regard to secondary school L2 examinations. It was decided that, from June 1998, students taking the General Certificate of Secondary Education (GCSE) in an L2 would be allowed to take bilingual dictionaries into the examination with them. (The GCSE is the first major ‘high stakes’ examination in UK schools, taken by 15- to 16-year-olds after five years of secondary schooling.) The different examining boards made slightly different arrangements about exactly which parts of the exam would allow dictionaries, but there was general consensus that they should be allowed in the two separate components testing reading comprehension and writing skills. A new option for assessing writing – coursework – was also introduced. By its very nature the coursework option (which could be taken instead of the writing examination) allowed the use of support resources such as dictionaries. These



Dictionary Use in Foreign Language Writing Exams

radical changes mirrored what had already become established in some Advanced Supplementary (AS) and Advanced (A) level L2 examinations (high stakes examinations taken by 17- to 18-year-olds after one or two years of study post-GCSE). In fact, access to a bilingual dictionary had already been introduced into one A level syllabus as early as 1978 (Bishop, 2000). The changes also reflected an early expectation of the UK’s National Curriculum – that students’ independent learning should be developed through the effective use of reference materials like dictionaries (DES, 1990). In 1998, therefore, the first groups of 16-year-old GCSE candidates made their way into the examination rooms armed with bilingual dictionaries to support them in their L2 tests. But less than five years later, in what might have appeared to many as a fin de siècle fit of returning to more ‘traditional’ testing practices, the decision was reversed. From 2003, dictionaries would no longer be allowed in the examination. This decision impacted not only on the GCSE, but also on AS and A level examinations, and a component that had become established in several of these higher level examinations was forcibly removed. At the same time, the coursework option for GCSE remained (as it did for A level), and with it the continued opportunity to use dictionaries. What had happened? Why was it that an initial policy decision to introduce dictionaries into the examination room was so swiftly and decisively reversed? And what effect did all of this have on the teachers and students?

Teacher and student perspectives When the initial decision to introduce bilingual dictionaries into the GCSE was made there was a flurry of research activity which took place before the first cohort of students had sat any examinations with a dictionary. The research arose out of an attempt to find out what teachers and students thought about the initiative, and what impact it was likely to have on the students as the principle stakeholders. Barnes, Hunt, and Powell (1999) sought the opinions of a range of language teachers on issues surrounding the use of dictionaries. Asher (1999) undertook two school surveys. A 1997 survey looked at dictionary policy in 27 schools. A 1998 survey, with 36 schools, focused on teachers’ classroom practice, their attitudes towards dictionary use and some of the effects teachers perceived ­dictionary ­access was having on their students’ patterns of learning. Chambers (1999) aimed to uncover students’ views about dictionary availability in the GCSE. In total, 267 students from three schools filled in a questionnaire relating to their experience of bilingual dictionary use and their perceptions of the experience. Just under half the participants were subsequently interviewed.



Chapter 1.  What is the problem with dictionaries?

If we consider what arose from this range of research we find evidence that, on the positive side, many teachers seemed to support having dictionaries in the examination: they were authentic and helpful, and contributed to the development of learner autonomy, greater self-reliance, and increased learner confidence. One typical comment supporting the positive views was “long overdue, very helpful in independent study” (Barnes et al., 1999, p. 25). Three quarters of respondents in the Barnes et al. study agreed that using dictionaries in an examination was an authentic task. Using a dictionary was true to life; its use was also seen as a transferable skill which could encourage more detailed awareness of language and how it fits together. Asher’s (1999) research also showed that a good number of teachers valued the authenticity that using dictionaries brought to language learning and language use. They saw this as a valuable life-skill which mirrored the situation of those who, when visiting the target language country, might occasionally need to use a dictionary. In both research projects, the observation was made that teachers also needed to refer to dictionaries from time to time. Why not the students? On the negative side, there was concern about students’ inability to use dictionaries appropri­ately and comments highlighted the need for training in dictionary skills. Teachers were worried that students might overuse the dictionary, thereby wasting time, or look up items word by word, thereby either misunderstanding or producing inaccurate language. They were also concerned that students might be less likely to commit words to memory if they knew they had access to a dictionary in the examination. For one teacher in the Barnes et al. (1999) study, allowing a dictionary in the exam room was “making it increasingly difficult to get lazy pupils to do any learning homework properly! I feel sure that some pupils will never finish an exam – they’ll be spending so much time looking up words in a dictionary” (p. 26). Another considered that the majority would “use them as an excuse not to learn vocabulary and spend too much time looking up words they ought to know” (p. 24). The students themselves thought the dictionary would be helpful with understanding rubrics and translating ideas into the target language more easily. They thought they would be helped to face the challenge of the examination with greater confidence. Like the teachers, they were worried that they might spend too long looking for words or become dependent on the dictionary. They were concerned about not being able to find the word in the first place, or being unable to choose the appropriate word for the context. Interestingly, Asher and Barnes et al. appeared to come to different conclusions about the principal teacher perspective on ‘with dictionary’ exams. Asher (1999) observed that “[v]iews of the dictionary reform amongst languages teachers seem predominantly, though not overwhelmingly, negative” (my emphasis).





Dictionary Use in Foreign Language Writing Exams

He conceded that in some cases this may have been “due less to the principle of dictionary use itself than to … the need to include another element of teaching [i.e. dictionary skills] into an already crowded syllabus” (p. 65). Barnes et al. (1999) by contrast concluded that “teachers are on the whole positive about the use of dictionaries … [and appear] to welcome the introduction of dictionaries and to have considered the surrounding issues very carefully” (p. 27, my emphasis). At the same time that the survey research was happening, Hurman and Tall (1998) undertook a very different kind of investigation. They compared actual test taker performances in GCSE French writing examinations, one taken with, and one without, a bilingual dictionary. The major findings and implications of this crucial study will be discussed later. For now, I present some of the comments teachers provided in a questionnaire Hurman and Tall used, which serve to underline the perspectives of the more concerned teachers: I am very much of the belief that in the written exam using a dictionary can often do more harm than good. Pupil use of a dictionary is not sophisticated enough (other than maybe perhaps with the brightest pupils) to stop them making considerable errors… (pp. 27–28) At KS4 [Key Stage Four, or school years 10 and 11] we find particularly less able pupils unable to find their way around a dictionary. (p. 28)

Whatever the views of the teachers and students at the time of its introduction, the dictionary use policy was put into place in 1998, only to be reversed some five years later. Such a swift policy volte-face would surely have seemed bewildering and bizarre. What did teachers and students think about this u-turn? At the end of 2000, at the time the policy reversal had been announced, I undertook a series of interviews among interested stakeholders from which I concluded that there might well be a case for retaining the dictionary in the examination (East, 2006a), or at least for seeing that there is more than one valid way of assessing students’ language ability (East, 2008). Admittedly this was a very small-scale survey, but its findings, summarized below, were telling in terms of highlighting stakeholder opinion at this crucial juncture. (Here and elsewhere I use pseudonyms when referring to any research participants.) I interviewed twelve people individually, in interviews ranging from about half an hour to (in one case) over two hours. There were three ‘language ­professionals’ (Mark – a chief examiner for GCSE and A level for a major examining board, Sue – the head of a foreign languages department in a large and popular ­suburban girls’ school, and Tim – a junior teacher of French and German in a large, successful suburban boys’ school). I also talked to nine students who were being prepared for, or who had recently taken, the A level in German. The nine students



Chapter 1.  What is the problem with dictionaries?

had had different experiences with dictionaries in examinations in a variety of ‘with dictionary’/‘without dictionary’ combinations: • • • •

Two had taken A level in 1996. They had therefore sat both GCSE and A level examinations under the ‘old’ system – no dictionaries in either exam. Two had taken A Level in 1999. They had taken GCSE without a dictionary, but A level with a dictionary. Two were planning to take A level in 2001. They would have taken both GCSE and A level with a dictionary. Three were working towards A level in 2002. They had taken GCSE with a dictionary, but would take A (and AS) level without.

With regard to QCA’s dictionary u-turn, all three language professionals expressed a sense of bewilderment and frustration. Tim argued: To give something two or three years and then say ‘well, it’s not working’ or ‘let’s change it now’ is just typical of the problems within education … you know, ‘move the goalposts’, ‘no, move them again’, ‘oh no, let’s move them back’ …

Sue’s perspective revealed a similar concern. The dictionary appeared to have been withdrawn “for the same reason it was introduced in the first place” – someone in government saying “oh, I could have passed all these exams … if I’d had a dictionary to look up the odd word, so I think they ought to have dictionaries” and then somebody else in government saying “oh, well, I mean, if they want dictionaries, it’s all done for them, isn’t it?” She concluded that “it seems to me completely arbitrary.” Mark was concerned that a resource had been taken away without careful thought into its legitimate value: I don’t understand the government, or QCA’s, reasoning for [the removal of dictionaries], because dictionary skills, to me, are very important. … Ministers have been saying … it’s non-linguists I think … ‘if you have an exam with a dictionary, oh, you can do it all’.

For Mark this appeared to demonstrate on the part of QCA “a lack of understanding of what the subject is, and I think also the lack of understanding it from the pupils’ perspective …” Tim, Sue, and Mark all noted, however, that having a dictionary would certainly help the test takers to feel more secure and confident. The removal of dictionaries at A level was, said Tim, “a rude shock” for those who had got used to it at GCSE – something which, asserted Sue, had “put the wind up them quite a lot.” According to Tim it had been “quite a relief ” for the students to have the ­dictionary “on tap.” There was, said Mark “an enormous amount of value” in





Dictionary Use in Foreign Language Writing Exams

­ aving the dictionary in the examination because it “gave students an enormous h sense of security.” Furthermore, surely allowing a dictionary in the examination was authentic. Sue argued, “when you and I go abroad, we take a dictionary with us, it’s not a big deal … and people do do that.” Tim concurred: “It’s realistic. We would not consider that using a dictionary was a taboo for us … why should we not be allowed to use a dictionary in whatever circumstances are necessary?” There was, according to Mark “an enormous amount of worry” among teachers about the lack of dictionaries. Mark was also concerned about what he saw as negative washback into classrooms – a consequent ‘decontextualization’ or ‘disauthentication’ of language: “I know some teachers are resorting to the old algebraic equivalent, you know, giving a vocab list, twenty words a lesson, and they’re having to learn off by heart.” He perceived that the policy reversal was “bound to have an effect on the teaching, without any doubt … I think we might find (and this is pure conjecture) that kids have been taught lists, and learnt stereotypes, instead of being immersed in language, where they can actually use their own resources.” Among the students themselves, there appeared to be a very clear understanding both of the potential benefits of having a dictionary in the examination, and of the potential pitfalls and problems, both with and without a dictionary. The rote-learning of decontextualized vocabulary had indeed become a problem, and something of a disincentive. Three students who would face the A level without a dictionary all independently agreed about this: There’s a hell of a lot of words to learn every week … I hate it – but it’s necessary. We have a lot of vocab to learn each week, which I know is useful, but it’s just the amount of words we’ve had to learn … and we have a test each week. There’s, like, loads of words you’re meant to learn, and I just can’t do all of them.

For one student, the prospect of not having a dictionary in the A level examination “seems really, really daunting right now … it is a big help to have the dictionary, it’s a burden not to.” This student believed that having the dictionary was useful “because otherwise some [students] might get stressed up totally, and it puts you at a bit of a dead end if you have no inspiration or you think ‘oh, what’s that word?’ and you can never remember it in German … if the dictionary’s there you can just sort of quickly flick through it …” Not having the dictionary was “going to make it more difficult, definitely.” Certainly those who had been allowed the dictionary in their examinations were generally positive about it. One student saw it as a “life-saver”, a useful resource to check up a few key words without which he “would have been lost.” It might slow you down a bit, but “that’s outweighed really by the advantages.”



Chapter 1.  What is the problem with dictionaries?

Another commented that the exam would have been “really hard” without the dictionary because “if there’s a word you don’t know, and you don’t know it, then there’s not a lot really you can do about it.” Certainly for the essay questions “it makes it so much better to have a dictionary … I wouldn’t be able to express myself in the depth that they want without having a dictionary.” On the other hand, some students were aware of the limitations of dictionary use. One who had been allowed a dictionary at A level commented, “you think ‘okay, I’ll look up that word and I’ll put it in’, but at the same time … you actually don’t really know what you’re writing down, you’re just picking everything up from the dictionary [and it might not] make sense to a German person.” A similar perspective was reiterated by another student who had not been allowed a dictionary at either level of examination. He argued that you might be able to “use more complicated words” when writing, but you “often have four or five different German words meaning the one English word you’re looking for, and it’s using the right ones … [and] you might … waste too much time.” Where, then, did things stand in the minds of teachers and students at the time of the dictionary volte-face? Similar issues were raised to those uncovered before dictionary use became established in the examination. Several themes emerge from all the stakeholder perspectives I have described. Dictionaries were helpful for checking individual words that might otherwise cause problems or for broadening vocabulary in writing. Their use was authentic, and they helped with the continued use of authentic texts, both in class and in the examinations. Not allowing them appeared to provoke some stress and to lead to a much greater emphasis on the decontextualized rote-learning of lists, something that students definitely appeared to dislike. Nevertheless, the learning of vocabulary was seen as important, and students could see the drawback of having a dictionary if they did not know how to use it successfully. On the plus side the stakeholders appeared to be supporting the use of dictionaries in exams because they promoted: • • • •

authenticity; flexibility to respond to the questions; increased learner independence; potentially increased test taker confidence.

On the minus side there was concern over: • • • •

taking too long to use the dictionary in the examination; over-reliance on the dictionary; not taking sufficient time to learn vocabulary; misuse of the dictionary.





Dictionary Use in Foreign Language Writing Exams

On this last point, Asher (1999) was particularly emphatic: In terms of enhancing pupils’ performance in reading and writing, the areas of the GCSE where candidates have access to dictionaries during the examination, a far from reassuring picture emerges. It is apparent that many pupils are ill-equipped with some of the most rudimentary skills required to make effective use of their dictionary, resulting in wasted time and inaccurate language production, especially in the target language. In such cases, dictionary use is at present counter­productive, diminishing rather than improving pupils’ performance. (p. 65)

Indeed, the prevalence of so-called ‘dictionary howlers’ would seem to be one of the strongest arguments against their use. Asher himself (p. 61) points out the example of the candidate who rendered ‘I will not’ in German as Ich Testament nicht. One example recently anecdotally reported to me was the candidate who wanted to say ‘I’m a tall boy’ and ended up writing Ich bin eine Kommode. The British Broadcasting Corporation has an interesting website (BBC, 2007) which reveals the more amusing consequences of inappropriate word choices in a range of languages. In an examination context such howlers may raise a chuckle from examiners, but the consequence for the test takers may be disastrous – in a real sense ‘diminishing rather than improving their performance.’ On this basis a ban on dictionaries in writing examinations may be warranted. But outlawing dictionaries from the exam room, and yet continuing to allow students to use them in coursework, still leaves these students open to making errors. If dictionaries continue to be available to students in coursework elements, they are surely potentially just as much a liability here as in an examination and may also contribute to diminished performance. Nevertheless, and perhaps most importantly in terms of the interviews I carried out in 2000, both teachers and students appeared to be generally in favor of allowing dictionaries in the examinations. This was particularly so for those who had had experience with using a dictionary, either at GCSE or at A level. And there was a definite sense that the ‘dictionaries are in’ policy had not been given sufficient time, with a consequent lack of any real evaluation of its effectiveness, strengths, and limitations as perceived by key stakeholders. We therefore need to be very clear about exactly how using a dictionary is affecting students in ­assessments so that we can at the very least help our students to minimize the liability of their use. Why, then, was there a change in policy?



Chapter 1.  What is the problem with dictionaries?

The bigger picture The use of dictionaries in L2 assessment is not just an issue for the UK. Australia and New Zealand, for example, have both faced the question of allowing support resources in L2 assessments, and have come to different conclusions about where they might best fit (East, 2005a, 2008). Beyond language learning and teaching, debate in the United States over allowing access to support resources and accommodations in standardized tests – for example for test takers whose first language is not English – has provoked considerable research activity (Sireci, Li, & Scarpati, 2003). At a more fundamental level the debates surrounding use or non-use of support resources like dictionaries in exams comes down to different understandings of the purposes of testing and assessment. Influential in these debates have been on-going discussions beyond the language learning context about what constitutes ‘best practice’ in assessment. The discussions illustrate an apparent conflict between two opposing value systems. Gipps and Murphy (1994) bring out this tension when they suggest that broadly speaking assessment is for ‘professional and learning’ purposes or for ‘managerial and accountability’ purposes. The first type of assessment focuses on the learning and the second type emphasizes the measurement. The UK’s Assessment Reform Group (ARG) distinguishes between these two intentions or types by using the descriptions ‘assessment for learning’ and the ‘assessment of learning’ (ARG, 2002a, 2002b, 2002c, my emphasis). Assessment for learning sits within a constructivist process-oriented approach to the curriculum which favors ‘dynamic’ or on-going assessment. This type of assessment is concerned with bringing out the test takers’ best performance by using testing procedures that ‘bias for best’ and ‘work for washback’ (Swain, 1985). L2 coursework options fit within this understanding of assessment. The assessment of learning, by contrast, is traditionally carried out through ‘static’ one-time tests. It is rooted in a traditional behaviorist product-oriented and knowledge-based approach. This approach focuses on the discriminatory power of tests to identify different levels of test taker ability and also to predict future academic performance. The big dilemma is that the two assessment paradigms are not mutually exclusive. We cannot say that either one is ‘right’ or ‘wrong’, ‘better’ or ‘worse’. They are just different, and based on different assumptions about what we want to ­measure. As a result, there is tension between them and often an attempt to ‘mix and match’, with assessment for learning sometimes taking the dominant position in the arguments, and with the assessment of learning staking its claim when there is a feeling that its influence is being watered down. The UK’s ‘dictionaries



10

Dictionary Use in Foreign Language Writing Exams

are in / ­dictionaries are out’ policy decisions are symptomatic of this tension. The tension gives rise to a situation in which evaluation and assessment policies have bounced back and forth between an emphasis on more traditional testing practices and a focus on skills development. Taking an historical view of the UK’s GCSE as an example, Gipps (1994) argues that the more traditional type of summative school examinations had had a negative impact on the secondary school system, an effect she describes as “stultifying” (p. 3). This had led to a move towards a more progressive form of assessment, exemplified in the GCSE, that emphasized a broader range of skills and placed less focus on timed examinations. Gipps notes, however, that these moves were introduced and supported by a government that subsequently returned to a system that placed more emphasis on a more traditional formal assessment system. In 2004, however, proposals were made (forming part of “the biggest planned shake-up of English education for 60 years” (Ross, Attwood, & Moynihan, 2004)) to dismantle the dominance of external examinations at GCSE level and place more focus on internal school-based elements. This bouncing back and forth between more traditional testing practices and skills development is driven by conflicting beliefs among those who devise or advise on the assessment policies about what assessment should be about – the assessment of learning or assessment for learning. As Gipps (1994) puts it: Government intervention in the UK has sometimes initiated, sometimes reinforced the move towards a more practical and vocationally oriented curriculum and thus the move towards more practical, school-based assessment. But government has also been concerned with issues of accountability and with what it sees as the maintenance of traditional academic standards through the use of externally set tests. (p. viii)

Gipps concludes that the situation that arises is one of ‘complexity and confusion’. The use or non-use of dictionaries in components of L2 assessment is a case in point, complicated by the two types of assessment available – the ‘traditional’ examination in which (currently at least) “[t]he use of dictionaries will not be permitted in any external assessment” (QCA, 2000), and the writing coursework option in which, as one examination board puts it, candidates may “have access to the task stimulus and a dictionary (which may be on-line)” (Edexcel, 2001, p. 7, my emphasis). There is, however, another concern in the UK which has implications for the use or non-use of dictionaries. Recent research (Coe, 2006) has indicated that GCSEs in foreign languages are considered to be among the hardest subjects, with students being awarded grades relative to other subjects that are sometimes at least one lower. The research, together with the finding of a government inquiry into take-up of L2 study at GCSE level, received considerable media attention.



Chapter 1.  What is the problem with dictionaries?

The government inquiry revealed that the number of students entered for GCSE L2 examinations dropped from 80% in 2000, when studying an L2 at GCSE level was compulsory, to 51% since 2004, when a GCSE in an L2 was made optional (Blair, 2007). One report noted the view that students were opting out “because the exams are harder and are seen to be harder” (Garner, 2007). A subsequent report (Cassidy, 2007) highlighted a continued “dramatic decline” in the face of not only perceived difficulty but also perceived lack of value. None of these reports suggests that no longer having dictionaries is to blame (and indeed the lack of compulsion at GCSE level is a strong contributing factor). Nevertheless in my interview with Mark (the chief examiner), Mark expressed the view that, given the dictionary ban, students would no longer opt to study an L2 – they would compare the subject with something like mathematics, which allows the use of a calculator in certain examination components, and would “vote with their feet” because they “don’t understand the logic.” Bearing in mind the issues I have raised in this chapter – the advantages and disadvantages of dictionary use, the wider issues around assessment practices, the use or otherwise of dictionaries in assessment, and the perception that GCSEs in foreign languages are harder than other subjects – several questions come to mind, specifically with regard to writing in an L2 (the focus of this book): • •

•

Is there really a case for banning dictionaries from examinations? What difference, if any, do dictionaries actually make to test takers, not only in terms of their performance, but also in terms of their perceptions of the test? What pedagogical and training issues arise, especially when, in other assessment contexts (like coursework), students can still use dictionaries?

This book explores each of these questions in depth in an attempt to draw some conclusions about the dictionary use question, and to offer practical advice to teachers and students about how to get the best out of a support resource like a dictionary. The illustrations used in this book are predominantly German, but the principles I draw have relevance for L2 teaching and testing, whatever the language.

11

chapter 2

On dictionaries and writing

In Chapter 1 I suggested that two essential tensions are inherent in the conflict over the use or non-use of bilingual dictionaries in assessments. Firstly, there are conflicting opinions, particularly among teachers, about the added value that dictionaries give. On the one hand, they are seen as authentic; on the other, there is genuine worry about test takers’ ability to use them effectively. Secondly, there is conflict and debate, particularly at a theoretical level among the assessment experts, over two opposing value systems that underpin different assessment practices. Perspectives concerning what makes a fair test differ according to whether the underlying value system favors process-oriented assessment for learning or product-focused assessment of learning. Assessments that fit into the first paradigm focus on the learning, whereas those that are part of the second emphasize the measurement. The two assessment paradigms are not mutually exclusive, however. Neither one is ‘right’ or ‘wrong’, ‘better’ or ‘worse’. They are simply different, and based on different assumptions about what we want to find out. The ‘dictionaries are in / dictionaries are out’ policy decisions are symptomatic of this tension, with dictionaries arguably acceptable in the first paradigm and not acceptable in the second. The situation is compounded by a perception that, at least in the UK, foreign languages are among the hardest of school subjects, and consequently students are ‘voting with their feet’ and abandoning their study of them. It is important to consider whether allowing or disallowing a dictionary might make a difference to students’ subject choice. However, student use and misuse of the dictionary and different assessment paradigms make the question of bilingual dictionaries in assessment, particularly of the skill of writing, a far from straightforward one to address. This chapter presents some of the background to the difficulty. I begin with an overview of what is currently accepted as the predominant way of teaching languages – the communicative approach – and explain how dictionaries naturally fit within this way of teaching. I go on to consider the two main types of dictionary available to the L2 learner – monolingual and bilingual dictionaries – and weigh up their ­relative merits and demerits. The chapter proceeds with a consideration of the two most common forms of assessing L2 students’ writing proficiency: coursework options and timed tests. These two methods have been chosen because they

14

Dictionary Use in Foreign Language Writing Exams

r­ epresent contrasting conceptualizations of writing assessment. I consider the issue of where dictionaries fit into these two types of assessment.

Teaching our students within a framework of communicative competence If we were to ask a group of language teachers what they thought the main purpose of teaching languages was they would almost certainly reply with something along these lines: ‘we teach our students a language so that they can communicate effectively in that language.’ Current understanding about language teaching is bound up with the concept of ‘communication’. The teaching approach that has come to be known as Communicative Language Teaching, or CLT, is now firmly embedded in the culture of language teaching and learning practices in many contexts and is underpinned by theories of communicative competence. Various theoretical frameworks of communicative competence have been expounded (Bachman, 1990; Canale, 1983; Canale & Swain, 1980; Hymes, 1971, 1972, 1982; Savignon, 1983, 1997, 2002; Widdowson, 1978), with each model building on, adapting, and developing the framework over time. The distinguishing feature of CLT approaches is that they have led to a distinct move away from artificiality of language with its emphasis on frequently decontextualized grammatical structures and vocabulary learning common within earlier frameworks such as ‘grammar-translation’. Instead there has been a move towards an understanding that language exists for purposes of real communication with real individuals in real contexts. According to Bachman (2000) the communicative paradigm regards language use as “the creation of discourse, or the situated negotiation of meaning” and language ability as “multicomponential and dynamic” (p. 3). The ‘negotiation of meaning’ has been central to L2 teaching since the 1970s (Kramsch & Thorne, 2002). In addition, notions such as authenticity and learner-centeredness are now well established as central concepts of language teaching, which “cluster around a ‘communicative’ core” (Benson & Voller, 1997, p. 10). These understandings of teaching approaches drive the view that dictionaries may be seen as legitimate ‘tools of the trade’. This is because dictionaries are frequently used for purposes of communication, discourse creation, and negotiation of meaning in real-world contexts. Also, if bilingual dictionaries are the most frequently owned and most used reference book consulted by L2 learners (­Kirkness, 2004; Midlane, 2005), then the bilingual dictionary has, more often than not, ­become the principal tool for which language learners reach, whether their teachers like it or not. There are some good reasons why language teachers do not like it: if, for example, students do not seem able to use bilingual ­dictionaries



Chapter 2.  On dictionaries and writing

e­ ffectively, encouraging their use to support the ‘situated negotiation of meaning’ is surely, as Asher (1999) puts it, counterproductive. This leads to several important questions. The bilingual dictionary may be the most used reference tool, but does that make it the best choice of dictionary for L2 learners, given the number of errors students seem to make? Or would students be better served with a different type of dictionary, or no dictionary at all? If we can answer these questions, we shall be part of the way towards understanding how, if at all, bilingual dictionaries fit into the bigger picture of L2 learning and assessment. I turn now to this first set of questions.

Do dictionaries have value in foreign language learning? Dictionaries as reference tools can be divided into two broad groups – print dictionaries and electronic dictionaries. These groups of dictionaries are also available in three types – monolingual, bilingual, and bilingualized. (The bilingualized dictionary, as I explain in more detail later, is a kind of hybrid of monolingual and bilingual dictionaries.) The monolingual dictionary may be further sub-divided into two: the standard monolingual designed for the native (L1) speaker, and the monolingual learner’s dictionary, designed for the L2 learner (Bogaards, 1999). Of these different types, survey research (Hartmann, 1999) has suggested that the most commonly owned and consulted dictionaries are monolingual and bilingual print dictionaries. These two types of dictionary are therefore the well established tools of the trade, both in general and in L2 language learning contexts (so much so that Bogaards refers to them as ‘traditional’). Ownership of an electronic dictionary, whether on a computer or in portable form, was (at least at the time and in the context Hartmann’s survey was taken) considerably limited. The availability and use of portable electronic dictionaries (PEDs) has grown substantially over time, especially as their cost has fallen, even though to an extent they are still something of a novelty (Midlane, 2005, p. 28). Whether available in print or electronically, it might seem unnecessary to ask which of the two main types of dictionary – monolingual or bilingual – is better for the L2 learner. After all, these dictionary types are quite distinctive and it would surely be easy to conclude that the better tool is the one that fits the particular purpose. But we face a dilemma when it comes to deciding which of these two types L2 learners should be encouraged to use in particular circumstances, and this dilemma is underpinned by various questions. What is the language level of the L2 learner? What information does that learner need to know? What is the best means of providing that information?

15

16

Dictionary Use in Foreign Language Writing Exams

Most significantly, one of the major drivers in the decision-making has to do with how teachers interpret CLT approaches. One major interpretation is that CLT should emphasize the virtually exclusive use of the target language. As a consequence it is often perceived that monolingual dictionaries are the more appropriate pedagogical tools regardless of the circumstances. The argument among language teachers runs something like this: ‘we want you to use a monolingual dictionary because by doing so you are being exposed to the target language and you will learn it more effectively.’ Monolingual dictionary use by L2 learners therefore appears to have become the “accepted orthodoxy” (Thompson, 1987, p. 282). As such, it is important to consider in some detail exactly why this is the case. Underhill (1985) suggests that an advantage of the monolingual dictionary is that it forces its user to use the target language in order to understand it. This apparently helps with the internalization of L2 without the barrier of L1 (Atkins, 1985). In acknowledging (although not necessarily agreeing with) the established viewpoint, Thompson (1987) notes the apparently accepted drawbacks of bilingual dictionaries which would appear to make teachers shy away from them as useful pedagogical tools: • • • •

They strengthen users’ inclination to translate from their L1 rather than helping them to think directly in the L2. They strengthen users’ belief in a one-to-one association between two languages at an individual word level. They do not adequately describe how individual words behave in a sentence. They may order meanings in a way that is not necessarily based on frequency or common usage, which would perhaps be more helpful for users.

Thompson (1987) acknowledges the claims that monolingual dictionaries avoid these drawbacks, particularly with regard to how words fit into sentences in contextually appropriate ways. To these objections we might add the following: •

•

The tiny pocket bilingual dictionaries that students often buy commonly provide only one-word equivalents with one or two ‘frequent’ meanings (Rossner, 1985). This might be sufficient when students use the dictionary to access the meanings of unknown words when reading (which is, after all, what they are often used for). When writing, however, these types of dictionary might give dictionary users insufficient or inadequate information from which to choose an appropriate word. What are apparently nearest translations in a bilingual dictionary may in fact have quite different meaning associations in the two languages. Being able to



Chapter 2.  On dictionaries and writing

find suitable translations in a bilingual dictionary therefore depends on how closely related the two languages are (Laufer & Hadar, 1997; Underhill, 1985). Each of these constraints theoretically leads L2 learners to make a myriad of mistakes when using a bilingual dictionary which could be avoided. Teachers’ practical experience is that this is often the case. Indeed, Tomaszczyk (1983) warns of the inadequacy of some bilingual dictionaries because in their attempt to meet the diverse needs of a broad range of prospective users they try to do too much. These needs are, in his view, so wide-ranging that they are effectively irreconcilable. He perceives the danger of L2 learners being led astray by entries in the bilingual dictionary because these learners are not always proficient enough to realize the limitations and take on board the information given to them without question. As a result Tomaszczyk concludes that what they produce can often reveal the limitations of bilingual dictionaries quite dramatically. Baxter (1980) suggests that the difficulty is made worse by the narrow focus on lexis (rather than a broader focus on discourse). The bilingual dictionary user gets used to matching lexical items and does not necessarily learn about the wider uses of words in context. Those in favor of the monolingual dictionary may argue on this basis that, when aiming to produce language, there is no opportunity to look at definitions that might help the learner to choose the appropriate word for the context, and to see how words work in contexts. These are all very convincing arguments in favor of promoting monolingual dictionaries: they fit into an accepted language teaching approach (CLT); they promote the internalization of L2; and they coax students away from a belief in a oneto-one lexical relationship between words in different languages that often leads to errors. But these perspectives fail to recognize several benefits of the bilingual dictionary. Thompson (1987) acknowledges two essential differences between monolingual and bilingual dictionaries: monolingual dictionaries have headwords in the target language, and monolingual dictionaries define words in the target language. Apart from this, Thompson asserts that the other features of monolingual dictionaries can equally be found in bilingual ones. Furthermore, the two differentiating aspects of monolingual dictionaries can cause problems in the language-learning process, especially for students below the most advanced levels. If, for example, learners do not know the target language word for the concept they wish to communicate, they will not know where to locate the word in a monolingual dictionary and will be obliged to look it up in a bilingual dictionary, at least as a starting point (Nesi, 1992). It may well be that after the initial enquiry students will need a monolingual dictionary to see how to use, or not to use, the word – but in practice students may simply take one of the choices given in the bilingual dictionary

17

18

Dictionary Use in Foreign Language Writing Exams

­ ithout making the effort to check in the monolingual one. Teachers may not see w this as a good thing, and it may lead to error, but it is a reality. A second problem lies with understanding the monolingual dictionary definitions, given that these are provided in the target language. If L2 students do know the word they are looking up the definition is not required (although the grammatical and other information may be useful). If they want a definition, the grammatical structures used in defining examples may be complex, and may be beyond the competence of anybody other than an advanced learner. Amritavalli (1999) suggests that although learners go to a great deal of trouble to decode complex dictionary explanations in monolingual learners’ dictionaries of English, their efforts still often lead to inaccuracy. Entries may involve decoding complex structures such as conditionals, indirect questions, relative clauses, and passives. Furthermore, illustrative examples can be idiomatic, or specific to a particular domain or culture. Students may simply not have learnt enough to be able to understand the definitions they find. Thompson (1987) puts it like this: One claim implicit in monolingual dictionaries is that the users will benefit from being exposed to the foreign language via the definitions. But one has only to look at the majority of the definitions to see that they are, of necessity, ungainly and complicated, and that they employ a special register which is not necessarily the most useful or rewarding for learners to be exposed to. (p. 284)

With regard to the use of English monolingual dictionaries, Amritavalli (1999) claims: In my experience, complex structures in explanations are more likely to be understood if they are spoken by a proficient user of English, who in effect ‘parses’ them for the learner by using appropriate pause and intonation patterns, and by drawing attention to the more complex parts of the definition. Typically, however, dictionary reference is a solitary and silent activity, in which students faced with unduly complex or otherwise unsatisfactory explanations too often lose their way. (p. 264)

The situation remains similar whether the monolingual dictionary is designed for L1 or L2 use. Worsch (1999) notes that monolingual learners’ dictionaries can be just as off-putting as monolingual L1 dictionaries due to users’ frustration over not understanding the definition. These drawbacks highlight several potential benefits of bilingual dictionaries. As has already been pointed out, monolingual dictionary users are particularly disadvantaged if they want to use the L2 productively, because the dictionary is only beneficial if L2 users know or can recall the words they want to use. A strong practical argument in favor of bilingual dictionaries appears to be that they overcome the difficulty inherent in not knowing the



Chapter 2.  On dictionaries and writing

word in the first place, and, secondly, not fully understanding the definition when the word is looked up. When learners are given a word in their own language they can understand its scope and the contextual limitations. For example, the English word ‘scale’ carries several different meanings depending on the context. Direct translations into contextually appropriate equivalents may be more successfully understood by an L2 learner than providing illustrative definitions in the target language. Furthermore, one major perceived drawback of bilingual dictionaries – locating an inappropriately matched lexical item – is not necessarily worse than not using a dictionary at all. Ard (1982) argues, “the fact that students often fail to find acceptable words is no reason to condemn the use of bilingual dictionaries. In most cases, the students would presumably fail to find an acceptable word in the absence of a bilingual dictionary also” (p. 206). It appears, then, that methodological purists prefer the monolingual dictionary because it seems to fit better with an understanding of what CLT is all about. But monolinguals can be hard to use. On a practical level, therefore, the bilingual dictionary may prove to be the more effective tool, particularly for developing L2 learners’ more active expressive abilities (Ard, 1982). Are there, however, any methodological arguments that might support the use of bilingual dictionaries? We might argue that encouraging the use of bilingual dictionaries contributes to an important dimension of CLT – that of helping learners to become more autonomous and independent of the teacher by developing effective communicative strategies (East, 2005b). Indeed, Canale and Swain (1980), whose work has been influential in framing contemporary understandings of communicative competence, speak of one important dimension of communication which they label ‘strategic competence’. By their definition of the construct this competence is the ability to use strategies effectively to make up for gaps in knowledge and thereby to compensate for breakdowns in communication. Canale (1983) went on to develop this dimension by claiming that it might also be useful to “enhance the rhetorical effect of utterances” (p. 339). Surely the bilingual dictionary is useful in both respects. Oxford (1990) notes that language learning strategies are used “because there is a problem to solve, a task to accomplish, an objective to meet, or a goal to obtain” (p. 11). Bilingual dictionary use may help L2 learners to make up for what Oxford calls “an inadequate repertoire of vocabulary” (p. 47) as they aim, for example, to complete a written task. Learners can do this in a way that cannot be achieved with a monolingual dictionary, especially if they do not know the word they are looking for in the first place. Horsfall (1997) suggests that ­students who are able to use a bilingual dictionary confidently are “well on the way to becoming much more independent learners” who are able to problem-solve. The

19

20 Dictionary Use in Foreign Language Writing Exams

dictionary provides the opportunity for students to check up on particular words and clarify misunderstandings. In turn, this may become “a positive motivator and confidence-builder, showing the learner that he/she can proceed without the teacher” (p. 7). This, he claims, makes the bilingual dictionary an aid to both teaching and learning. In summary, confident and effective use of a bilingual dictionary promotes learner autonomy. It also means that language learners can operate from their L1 understanding of a word to find its direct L2 equivalent (L1 → L2), or can check the L1 meaning of a word when presented with its L2 equivalent (L2 → L1). When considering the production of language, for example when writing, it may be that such use of the bilingual dictionary will be frequently advantageous. Furthermore, if authenticity is considered as another important dimension of CLT there is further argument in favor of the use of bilingual dictionaries by L2 learners. Certainly the use of authentic texts, rather than made-up and structured language, carries with it the additional responsibility to make their meaning accessible. The language used is no longer regulated by the teacher or the textbook and will contain vocabulary that may need to be looked up in a bilingual dictionary (Horsfall, 1997). Bilingual dictionaries may be necessary to check items in a task that is being used to generate a written response, and also necessary to complete the response adequately. After all, is this not what users of an L2 often do in the real world? They read something and respond to it with the help of a dictionary. Using a bilingual dictionary is “an authentic activity” (Chambers, 1999, p. 78). There would therefore seem to be strong and convincing methodological arguments in favor of the bilingual dictionary, particularly when writing. These arguments may not, however, be enough to convince the methodological purists that many L2 learners (apart from possibly the most advanced) probably derive more benefit from bilingual dictionaries than monolingual ones. Part of the response to reservations around bilingual dictionaries may be to encourage the use of a different kind of resource such as a thesaurus. The benefit of thesauri is that L2 users can continue to operate in the target language, but are not required to understand definitions of words. They can, for example, establish the meaning of an unknown word via a synonym which they may understand, or use the information available to broaden or extend their writing. There are, however, two major limitations. The first of these is that users are provided with neither L1 support nor definitions. Secondly, synonyms of words do not necessarily convey exactly the same meaning, and L2 users may be led astray by choosing a synonym that is not necessarily appropriate for a particular context (although the Collins School Thesaurus (Collins, 2006) is an example of an English thesaurus that aims to help with this by providing short examples of appropriate use).



Chapter 2.  On dictionaries and writing

­ roviding synonymous information may prove to be insufficient for many users P of a foreign language without the additional benefit of an L1 equivalent. Another response to reservations around bilingual dictionaries may be to develop a new kind of dictionary – one that enables learners to find the L2 equivalent of the L1 word, and then provides contextually appropriate definitions. Indeed, in recent years there has been a move towards developing such a ‘hybrid’ – the bilingualized dictionary. A bilingualized dictionary – “at the intersection of monolingual, bilingual and pedagogical lexicography” (Hartmann, 1994, p. 206) – is defined by Laufer and Hadar (1997) as a combination of a learner’s monolingual dictionary, with the same number of entries and meanings for each entry, with a translation of the entry into the target language. If the target word has several meanings, each meaning is translated. Worsch (1999) argues that the bilingualized dictionary can be regarded as “a kind of spin-off of existing monolingual dictionaries” (p. 103). Laufer and Hadar (1997) suggest that these types of resource are “a step in the right direction” with regard to the creation of user-friendly dictionaries, because they “provide more information than monolingual or bilingual dictionaries and allow the user to choose explanations in the one language with which he or she is more comfortable, or in both languages for reassurance and reinforcement” (p. 196). When writing, L2 students may find that this type of additional information helps them to use the word they have looked up even more effectively than if they had just been presented with word equivalents. Bilingualized dictionaries may be ‘the way to go’ in the future. Their availability and use are not yet widespread, however. In the meantime the majority of learners are faced with the choice between monolingual and bilingual, whether in print or electronic form. And as far as these two are concerned, Thompson (1987) argues: … monolingual learners’ dictionaries have a very important role to play at the most advanced levels. Basically, though, for learners below this level the bilingual dictionary can do all the useful things that the monolingual dictionary can do; and it can do several of the things in a more efficient and more motivating way. All that is needed is the recognition that this is so. (p. 286)

What about learner choice? Although there are constraints, there are some convincing theoretical, practical, and methodological arguments in favor of bilingual dictionaries. Further evidence in support of bilinguals can be found from the results of surveys into dictionary

21

22

Dictionary Use in Foreign Language Writing Exams

preferences. Piotrowski (1989) concludes from a survey of studies of dictionary use that L2 learners and dictionary users turn to their bilingual dictionaries as long as they use dictionaries at all, regardless of their level of competence. Tomaszczyk (1983) concludes from 449 respondents to a questionnaire survey that beginning and intermediate L2 learners rely on bilingual dictionaries almost exclusively. More surprisingly, language teachers in secondary schools and universities used bilingual dictionaries more than L2 and other monolingual dictionaries, even though they had access to these. This finding mirrors Laufer and Hadar’s (1997) observation that even L2 learners who have reached a good level of L2 proficiency and have been trained in skills such as dictionary use still favor a bilingual dictionary, although some may use a monolingual and a bilingual dictionary together. Tomaszczyk argues that it can surely be assumed that the teachers were able to use the L2 quite successfully and had all been “brought up in the tradition that discouraged excessive reliance on the mother tongue” (p. 46). Nevertheless, practically all of them preferred to use bilingual dictionaries, even though they regarded them as considerably less valuable than monolingual ones – an interesting reflection of the disjunction between ideological ­perspectives and actual use. Even those who did not find monolingual dictionaries too difficult to use felt more comfortable with bilingual dictionaries and would prefer them when given the choice. Thompson (1987) makes his own informal observation that only very few students in his classes in several countries claimed regularly to use a monolingual dictionary for production of language. Most admitted that they hardly ever used them for production or comprehension, even after having been trained in their use. Thompson concedes that when learning his students’ languages, he has made far more use of bilingual than monolingual dictionaries. He therefore argues that monolingual dictionaries are “simply not cost-effective for many learners in terms of rewards (correct choice of word) versus effort” (p. 284). Tomaszczyk (1983) puts it like this: “[p]eople use bilingual dictionaries whether language teaching methodologists like it or not. Indeed, to many people the word ‘dictionary’ means ‘a bilingual dictionary’” (p. 45). Ownership of bilingual dictionaries by learners far outweighs that of monolingual dictionaries (Midlane, 2005). In the absence, at present, of widespread bilingualized dictionaries this is therefore the “consumer reality” (Laufer & Hadar, 1997, p. 190) that reflects what students “notoriously prefer” (Atkins, 1985, p. 19). Atkins puts the dilemma well when she reframes the traditional belief in the superior value of monolingual dictionaries in these terms: “Monolinguals are good for you (like wholemeal bread and green vegetables); bilinguals (like alcohol, sugar and fatty foods) are not, though you may like them better” (p. 22).



Chapter 2.  On dictionaries and writing

Do dictionaries have a place in foreign language testing? The evidence so far presented reveals some convincing arguments for the inclusion of bilingual dictionaries in the L2 classroom in a way that is fully commensurate with CLT approaches – their use is authentic, their use promotes learner autonomy, and, most importantly, their use allows students to make up for gaps in their knowledge and potentially enhance the quality of what they produce. Furthermore, if using bilingual dictionaries is what the students ‘notoriously prefer’, and if they are going to use them regardless of what their teachers might say, there would surely be little point in insisting on monolingual dictionaries, and it would surely be beneficial to help language learners to get the best out of bilingual dictionaries. It is also certainly beyond question that L2 learners’ language proficiency has to be assessed in some way if we are to gain a measure of these learners’ communicative competence. This leads to the second major issue we need to consider: given the two assessment paradigms articulated by the ARG – assessment for learning and the assessment of learning – and the central importance of CLT as a teaching approach, where might the bilingual dictionary fit into assessment? Without doubt CLT has compelled language testers to move away from an emphasis on the testing of language ability as an isolated ‘trait’, and has required them to “take into consideration the discoursal and socio-linguistic aspects of language use, as well as the context in which it takes place” (Bachman, 2000, p. 3). In other words, in response to a view of language ability that moves beyond the limitations associated with discrete point or integrative tests (Morrow, 1981; Oller, 1979), language testers have been required to make efforts to develop tests that not only measure linguistic ability considered more widely, for example in terms of the aspects of communicative competence identified by Canale and Swain (1980) and Canale (1983), but are also seen as ‘authentic’ or related to how language is used in the real world. This type of testing (where the underlying construct is based on a theoretical framework of communicative competence) may be considered more valid as a mirror of language in actual use than earlier testing models. The quality of authenticity arguably has central importance (Morrow, 1991; Wood, 1993). With regard to authenticity there is, suggest Bachman and Palmer (1996), the need for a correspondence between language test performance and language use. For a given test to be useful for its intended purposes, performance on the test needs to correspond in demonstrable ways to language use in non-test situations or capture in a representative way the complexities and demands found in the real world (Wiggins, 1989). Bachman and Palmer refer to these as ‘target language use’ (TLU) domains – the actual real-world situations that the test tries to mirror. Indeed, for Bachman and Palmer authenticity is “a critical quality of

23

24

Dictionary Use in Foreign Language Writing Exams

language tests” because “it relates the test task to the domain of generalization to which we want our score interpretations to generalize” (pp. 23–24). In other words, if we want to find out how our students are likely to perform in real world language use tasks beyond the classroom, we need to create assessment opportunities that allow them to use the type of language they are likely to encounter beyond the classroom. This would surely be a persuasive reason, in theory at least, to allow bilingual dictionaries, whatever the type of assessment. Furthermore, Bachman (1990) argues that communicative language tests should provide “the greatest opportunity for test takers to exhibit their ‘best’ performance” so that they are “better and fairer measures of the language abilities of interest” (p. 156). It might be that bilingual dictionaries might be helpful to some test takers. They may therefore have a supportive role to play. It is, however, harder to justify the availability of dictionaries in tests if language learners cannot necessarily use them successfully. There is also a tension between what Weir (1990) notes as two genuine concerns of communicative language testing – that of eliciting the test taker’s best performance (for which a dictionary may or may not be useful) and that of ensuring that the measurement properties of tests, such as reliability, are upheld (for which a dictionary might be problematic). This tension leads to the challenges around assessment practices outlined in Chapter 1 and the conclusion that the role of bilingual dictionaries in tests is controversial (East, 2005c, 2007). We therefore face a problem when we consider where dictionaries fit into assessments. Are they beneficial? Or do they distort the information about language proficiency available from the assessment? We need a framework whereby we can evaluate a given assessment or testing procedure in a meaningful way and draw constructive conclusions about dictionaries. One way of doing this is to consider different assessment opportunities in the light of two questions: is the assessment opportunity useful? And is it fair?

Designing useful tests Bachman and Palmer (1996) argue that “[t]he most important consideration in designing and developing a language test is the use for which it is intended, so that the most important quality of a test is its usefulness” (p. 17). Test usefulness is, they suggest, an overriding consideration for quality control throughout the process of designing, developing, and using a particular language test. After all, what use is a language test if we cannot have confidence that the information it gives us about the test takers’ language proficiency is meaningful? An important



Chapter 2.  On dictionaries and writing

c­ onsideration when thinking about dictionaries in assessment is therefore whether the test or the assessment gives us this kind of useful information. Bachman and Palmer (1996) drew up a kind of ‘checklist’ which they suggest would help us when thinking about the usefulness of tests. They argue that evaluating a test against the six qualities on the list would help test-setters to weigh up the relative usefulness of the test. The qualities are also useful in evaluating alternatives such as coursework. These qualities are:

1. 2. 3. 4. 5. 6.

Reliability Construct validity Interactiveness Impact Practicality Authenticity.

Reliability and construct validity are the two fundamental measurement characteristics of tests. They are concerned with the meaningfulness and accuracy of the scores awarded, in relation to the measurement of an underlying construct, where the scores are indicators of the ability or construct. Construct validity has to do with the assessment task: it relates to whether the task adequately and fairly reflects the construct that the test is trying to measure (and therefore the extent to which the scores are adequate reflections of test takers’ abilities). Reliability has to do with the scores: it relates to the way scores are awarded (and therefore the extent to which the process of awarding the scores is adequate) or to whether the test (or a different version of the test) yields comparable results across different administrations. Thus validity focuses on the tasks themselves and reliability is concerned with how consistently performances on those tasks are measured. The four qualities of interactiveness, impact, practicality, and authenticity are distinct from the measurement qualities. Interactiveness is a quality of the interface between the test taker and the test – how effectively the test taker is able to engage with the task. Impact at the microlevel is concerned with the effects of taking a particular test on the test takers. Impact at the macrolevel is concerned with the wider implications of the test for the stakeholders, including test takers, teachers, parents, educational programs, employers, and gatekeepers. Practicality is a quality of how the test is run – whether the test can be administered efficiently. Authenticity is, as has already been stated, a quality of the test and its relationship to a TLU domain. Bachman and Palmer (1996, p. 18) do not suggest that all six qualities must be traceable in equal measure if we are to conclude that a test is useful. Three principles guide the application of the six qualities to any given test:

25

26 Dictionary Use in Foreign Language Writing Exams

1. It is important to maximize the overall usefulness of the test, rather than the individual qualities. 2. The qualities should not be considered in isolation, but rather in terms of how each quality contributes to the overall usefulness of the test. 3. The appropriate balance among the qualities should be determined for each specific testing situation. It is therefore inevitable, and acceptable, that some tests will regard some qualities of test usefulness as vital that other tests might consider relatively minor. A formative ‘in class’ test, for example, may be highly interactive in the sense that the task is engaging and stimulating for the test takers, drawing on their experience and knowledge of the world. At the same time it may have low reliability. A summative ‘high stakes’ test may be considered to be highly reliable, but it may lead to a low level of interactiveness for test takers who are unable to engage fully with the task due to the pressurized demands of time constraint. Timed tests and coursework options have different balances of the qualities that contribute to their ‘usefulness’. This differential balance gives rise to questions about dictionaries in these assessments. It also leads to a second consideration which needs to be laid alongside questions of usefulness – the extent to which a given testing or assessment procedure is fair.

Designing fair tests Establishing test usefulness is not enough. We also need to establish test fairness. As Kunnan (2000) argues, there is little or no value in a test being useful if it is not fair. Fairness is an important consideration. But what exactly do we mean by fair? Hamp-Lyons (2000) suggests that fairness is a difficult concept because “there is no one standpoint from which a test can be viewed as ‘fair’ or ‘not fair’” (p. 32). There are several implications of this for dictionary use. Using a dictionary might distort the measurement of test takers’ language proficiency. Some test takers may do better with a dictionary, and some may do worse. Some may do better with one type of dictionary rather than another. These factors may lead to unfairness for a whole range of test takers. Taking into consideration Bachman’s (1990) comments about communicative language testing – that tests should provide adequate opportunity for all test takers to demonstrate their best performance – one way of looking at fairness is this: did the test takers have the greatest opportunity to display in the test what they know and can do? Was there part of the test procedure that may have hindered this? What may be the consequences, for the test takers, of this?



Chapter 2.  On dictionaries and writing

In the light of considerations of both usefulness and fairness different ways of assessing writing have been developed. Also in this light questions around using or disallowing dictionaries in writing assessment have arisen. Where, then, might dictionaries fit into these different ways of assessing writing?

Dictionaries and coursework Coursework and portfolio options, whereby students are able to produce an extended piece of work or collect samples of writing over a period of time, are well suited to an understanding of assessing communicative competence in a way that provides opportunity for those being assessed to demonstrate the full extent of their proficiency. This is one reason why this assessment method, which is quite distinct from a ‘test’, is gaining in popularity. Hamp-Lyons and Condon (2000) suggest that the portfolio assessment method arguably gives us a better representation of writers’ efforts and range of writing than a one-off time-constrained summative test. The coursework approach views writing as process, and this has enabled us to rethink our responses to students’ writing. There certainly seems to be a lot to commend coursework and portfolio options for assessing writing. In the view of Grabe and Kaplan (1996), the portfolio system has particular strengths: 1. A range of writing samples, drawing on different topics and task types, can be assessed. 2. Students are able to reflect on their writing and the progress they are making. 3. Students are given responsibility for choosing the writing on which they want to be assessed. 4. A more realistic audience for students’ writing is created. 5. There is a stronger link between teaching and assessment. Perhaps most importantly Grabe and Kaplan (1996) assert that writing assessment should match its assessment criteria to the type of writing that students will be expected to carry out beyond the learning context. Portfolios and coursework options arguably contribute positively in this respect. When dictionary use is being considered it is definitely the case that the assessment available through coursework or the writing portfolio reflects the ways in which those being assessed will carry out writing outside the classroom. Portfolio assessment may therefore be seen as highly authentic in that it mirrors the process of writing through which

27

28

Dictionary Use in Foreign Language Writing Exams

people go as they write in the real world, it is not subject to such rigorous time constraints as the timed test, and it factors in the use of resources. Also, coursework or portfolio assessment may promote greater positive interactiveness on the part of, and greater positive impact on, those being assessed. This is because it is removed from the artificial constraints of the timed test and because it often relies on multiple samples of writing covering different genres and themes from which students can choose their best work. In the words of Swain (1985) it ‘biases for best’ and ‘works for washback’ in ways that the timed test apparently cannot. Elbow (1991), for example, argues: Testing is a process of “checking up on” students to see who is smart and who is dumb, who has been working hard and who has been goofing off. … Portfolio assessment takes the stance of an invitation: “Can you show us your best work, so that we can see what you know and can do – not just what you do not know and cannot do?” (pp. xv–xvi)

Allowing bilingual dictionaries for portfolio work authentically mirrors real-life writing activity and such dictionaries therefore arguably have a legitimate role to play in the writing process. Furthermore, portfolio or coursework assessment of writing is one means of providing a broader range of evidence about what learners can do, commensurate with an understanding that assessment for learning is “far more than just a summative collection of information about learners’ achievements” but rather “a vital part of the learning process [that] impacts on the course of pupils’ learning” (Harlen & Winter, 2004, p. 391). Dictionaries make their contribution here as mediators of a valid communicative strategy which L2 students are learning to use. As a result, it would appear to be beyond dispute that access to resources like dictionaries is considered to be quite acceptable and useful in coursework and portfolio options, and it is relatively easy to justify the use of dictionaries in such options. Also, the positive benefits of these types of assessment outlined above surely make them more useful and fairer as measures of writing proficiency than time-constrained writing examinations.

Dictionaries and timed tests When it comes to questions of usefulness and fairness, however, portfolio and coursework options are not without their challenges. Allowing dictionaries in coursework begs an important fundamental issue. More ‘traditional’ or ‘direct’ tests of writing, variously referred to as the “timed impromptu writing test” (­Weigle, 2002, p. 59) or the “snapshot approach” to writing (Hamp-Lyons & Kroll, 1997, p. 18), remain a popular method of assessing writing proficiency and they



Chapter 2.  On dictionaries and writing

continue to be found in many large-scale testing situations internationally. Given the centrality of timed tests to the assessment of writing proficiency, we need to take their usefulness and fairness as assessment tools seriously, and it is very important to think about whether the bilingual dictionary also has a role to play in these types of test. More importantly, we need to consider why allowing dictionaries in such writing tests is more controversial. The first question to ask, of course, is why the timed test format continues to be so popular and wide-spread given what appear to be the distinct advantages of coursework. There are benefits to the timed test method without doubt. Wolcott and Legg (1998) suggest that particularly from the perspective of measurement experts timed tests offer several advantages: •

• •

A number of variables that may influence the test takers’ writing performance, such as the topic, availability of resources, time allowed to complete the task, and scoring methods, are controlled and kept as identical as possible. Using controlled conditions means that the writing performance of any one test taker can be evaluated relative to the performances of all other test takers. Using controlled conditions means that there is no opportunity for the test takers to pass off somebody else’s work as their own, and questions over the extent of feedback or support do not arise. The certainty that we are assessing the test takers’ own work is particularly important when individual competences are being considered.

The problem of plagiarism has been identified as a major weakness of coursework (Smith, 1991): how can we know for certain that students’ work in portfolios is their own? One solution to the plagiarism problem might be to allow those being assessed only to complete coursework in class, under supervision, and with some time constraints (Green, 1985). If this limitation is placed on coursework, however, why not just use a timed test? Indeed, the benefits of timed tests highlight an important aspect of their usefulness. They are reliable in so far as they control for variations in test task, test condition, and scoring method, standardizing these across all test takers. This also contributes to the construct validity of timed tests when the construct being measured relates to writing performance which is being tested uniformly and in a controlled way. Certainly, Grabe and Kaplan (1996) admit that reliability can be seriously challenged in portfolio assessments. Additional benefits of timed tests are that they are practical and relatively costeffective to administer and score (Wolcott & Legg, 1998). They require a set time period for completion and the deployment of test supervisors for this set, relatively short, period. They therefore enable short turnaround times, and they can be used in almost any setting (White, 1995). Finally, although timed tests place the focus on

29

30

Dictionary Use in Foreign Language Writing Exams

the writing product rather than the writing process, it has been suggested (­Gorrell, 1988; Lederman, 1986; Wolcott & Legg, 1998) that products are ultimately what we get to see, and therefore what we are concerned with evaluating. Where, then, might dictionaries fit into timed writing tests? On the face of it there would appear to be no reason why dictionaries should not be allowed in timed tests of writing proficiency. After all, the construct of writing proficiency based on a framework of communicative competence aims to allow language learners to use language to create meaningful communication in ‘authentic’ and real-life contexts. Timed tests based on this framework frequently aim to reflect such real-world contexts by asking the test takers to write a postcard or a letter, possibly in response to some culturally authentic stimulus material (a picture or another letter in the target language). If we are asking the test takers to compose a letter we must at least acknowledge that when they respond to and write letters in an L2 in real life they might very well use a bilingual dictionary to help them with this task. If we disallow dictionaries from timed tests there is a problem about the extent to which performance on the test can be generalized to performance in the real world and can therefore be ‘authentic’. One way round the problem of lack of authenticity might be to test L2 learners through tasks that mirror TLU domains that are not so dependent on time or resources. An e-mail, for example, might be potentially highly authentic as a writing task set within time constraints and without resources because it is often a product “written … in haste, without much collaboration, and without much chance to “go back in” to erase or correct” (Wolcott & Legg, 1998, p. 19). But anything beyond the limited confines of a short e-mail or a postcard, and certainly something requiring extended discourse, can often not be replicated authentically in the timed test (especially if no support resources are allowed), and we frequently require longer responses so that we have a reasonable sample of language on which we can make judgments about writing proficiency. According to Bachman and Palmer (1996), authenticity is also important for the test takers, because it contributes to test takers’ perceived relevance. This perceived relevance helps to promote a positive affective response to the test task, thereby helping test takers to perform at their best. The test takers’ interaction with the task may therefore be negatively affected if there is perceived lack of authenticity in the task. If using a bilingual dictionary is an authentic part of writing in the real world – what L2 users do when, for example, writing a letter in a real context – and if its use potentially supports the test takers, why should the dictionary be excluded from a timed test that measures test takers’ ability to carry out an authentic task like writing a letter? The answer to this question reveals the complexity of the issue where time-constrained tests are concerned.



Chapter 2.  On dictionaries and writing

The problem of dictionaries in timed tests In Chapter 1 I argued that policy decisions with regard to allowing or outlawing dictionaries in time-constrained examinations are symptomatic of a tension between two opposing value systems and two different understandings of constructs. This tension gives rise to the essential difficulty. Where the emphasis of the test is on the measurement of test takers’ prior learning it is important to establish that the test can measure this adequately. If “the primary function performed by tests is that of a request for information about the test taker’s language ability” (Bachman, 1990, p. 321), tests are required to measure such ability against particular benchmarks, and this measurement function relies, for the accuracy of information it provides, on not allowing anything that will get in the way of test takers’ performances. By this argument, allowing a bilingual dictionary may potentially compromise the fairness of the test as a measure of the underlying construct by ‘giving the answers away’. Indeed, this is how dictionaries in writing tests have often been viewed. As Bishop (1998) explains, “[d]ictionaries were put in the same category as cheating or were accused of making the assessment either impossible to set or the task too easy to achieve. This attitude is deeply rooted even today” (p. 3). However, one counter-argument to the ‘dictionaries make it all too easy’ perspective is that some test takers may simply not be able to use a dictionary effectively. In these cases including it in the test may make the test more difficult. Some test takers may actually do worse than we might have expected, and any ‘advantage’ the dictionary was supposed to give is negated. Another counter-argument is that if dictionary use is a valid part of the construct of writing proficiency which the timed test should aim to measure (namely because it enhances authenticity and provides legitimate support) outlawing its use in the test may mean that an important part of the construct has been excluded. In this case some students may do worse than they otherwise would have done and we do not get a true measure of their writing ability. These different arguments also impact on considerations of fairness. Cumming (2002) argues that in one key respect the timed test is fair: The assessment of writing in high stakes, large scale, international settings is obliged to provide a uniform context for examinees’ performance … so that examinees all start from a basis of equal opportunity. The constructs to be assessed cannot be biased for or against any particular population, nor can they seem to privilege any people with specific abilities or knowledge. (p. 74)

Indeed, for Cumming (2002) the fundamental requirement for fairness in such tests is that the test takers have an equal opportunity to perform on the test.

31

32

Dictionary Use in Foreign Language Writing Exams

­ umming goes on to suggest that fairness is also built into the requirement of C comparability of performance “from one place to another place, from one time to another time, or from one version of the test to another version” (p. 74). In his view, such ‘equitability’ considerations present a challenge for alternative assessment procedures, such as portfolios. One equity challenge for portfolios is that they are highly localized to specific contexts. There is also often no way of controlling the amount of help the writers are getting or the range of support resources on which they can draw. This is surely unfair on those who receive less help, who have less access to resources, who have access to different types of resource, or who cannot use these resources very well. If we were to allow dictionaries in timed tests, we would also have to face these ‘unfair’ considerations. Whatever way we look at it, it would seem that dictionaries are problematic in timed tests. It is perhaps easy to see why Bishop (1998) concludes his assessment on dictionaries by asserting that “[a]rguments still rage over the role of the dictionary and what effect it has on assessment in particular” (p. 3). Depending on the stance taken in the ‘dictionaries are in / dictionaries are out’ debates we would have to conclude that tests are, in the words of Messick (1989), “imperfect measures of constructs because they either leave out something that should be included according to the construct theory or else include something that should be left out, or both” (p. 34). Messick (1989) develops this perspective by articulating the two major problems that, in his view, might threaten the validity and fairness of a test: 1. Construct under-representation: the test tasks exclude important dimensions or facets of the construct we are aiming to measure. Test results may therefore not reveal the test takers’ true abilities in relation to the construct. 2. Construct irrelevant variance: the test measures variables that are not relevant to the construct: a. ‘Construct irrelevant easiness’ occurs when additional information, unrelated to the construct we are aiming to measure, is provided in or with the task, and this information makes the task easier for some test takers. They thereby score higher than they might otherwise have done. b. ‘Construct irrelevant difficulty’ occurs when additional unrelated information is provided, and this information makes the task more difficult for some test takers. They thereby score lower than they might otherwise have done. What does this have to do with dictionaries? Regardless of how the construct is understood, it is important for us to eliminate anything that may threaten a test’s



Chapter 2.  On dictionaries and writing

validity. And depending on how the construct is understood, a valid score on the test is one that has not been unduly affected by either construct under-representation (where something has been left out and the scores may be invalidly low or high), or by construct irrelevant difficulty (where something extraneous has been included and the scores may be invalidly low), or by construct irrelevant easiness (where something extraneous has been included and the scores may be invalidly high). If we are concerned that a timed writing test should measure students’ prior learning and should help us to discriminate between test takers of different proficiency levels on the basis of their test scores, we need to make sure that we take account of anything that might interfere with this measurement – otherwise the interpretations we make on the basis of the scores may be spurious. This might mean saying ‘no’ to dictionaries. If, however, we are concerned that test takers should be given the best opportunity to display what they can do in a timed test we need to make sure that we take account of anything that might support this – otherwise score interpretations may also be spurious. This might mean saying ‘yes’ to dictionaries. And so, when it comes to considering the timed writing test, the most popular and most widespread means of assessing L2 student writing, we come up against huge challenges when we consider allowing dictionaries. On the one hand, the enormous benefits of timed writing tests – practicality, cost-effectiveness, ease of administration – have contributed to their popularity. Furthermore, central to the concept of the timed test are the measurement properties of reliability and validity. Where the construct is perceived more narrowly as ‘writing performance’ or the ‘assessment of learning’, and where discriminating among test takers is key, the timed writing test arguably has high construct validity. On the other hand, the lack of authenticity found in the unnatural time-constraints and the outlawing of support resources works against them. However, if construct validity is central to the usefulness of a timed writing test, and if a major consideration has to be reducing threats to the test’s validity, we must consider where, if at all, dictionaries fit in. We require evidence on the basis of which we can make sound decisions.

Where to from here? This chapter has dealt with a number of complex but important issues surrounding language learning, language use, and language testing. It would be useful, in concluding, to relate these issues back to the central theme of this book – dictionary use in writing. I have suggested the following:

33

34

Dictionary Use in Foreign Language Writing Exams

• • •

Bilingual dictionaries are potentially supportive tools for L2 learners – they help them to make up for gaps in their knowledge. Bilingual dictionary use is authentic – L2 users use them in the real world if they do not know or cannot understand something. The ideal assessment of writing proficiency will aim to mirror real-world practice as authentically as possible.

Relating these to issues of construct validity I also suggest the following: • •

•

•

•

If writing is assessed through coursework or portfolios it is arguably easier to justify including dictionaries as legitimate tools. Allowing dictionaries in timed writing tests is more problematic. If, however, we outlaw the use of dictionaries in such tests there is an immediate disjunction between language testing and language use in the classroom and beyond. If dictionaries are used in the classroom and beyond, students will have got used to them and it is likely that they will see the relevance of them. If students are denied them in the test this may have negative consequences for them (they may not understand why a resource they have come to see as relevant has been taken away from them, and this may impact on their performance). On the other hand, there is clearly a need to preserve the measurement qualities of the test (its ability to measure writing proficiency and to discriminate between different levels of test taker ability). We also need to be mindful of fairness. Having a dictionary in a writing test may be beneficial for some and not for others. Some test takers may use it well; others may find that their performance is compromised. This consideration also impacts on allowing dictionaries in coursework.

A whole range of issues influences decisions on the usefulness of dictionaries in writing assessments and the consequent fairness of the assessment for all stakeholders. Given the centrality of timed writing tests in many contexts, there would appear to be sufficient grounds on the basis of how they are currently carried out – lack of authenticity, time constraints, impact on the test takers – to argue that we need to look closely at the timed test as a testing procedure and to consider its construct validity. Rather than simply accepting the ‘dictionaries drive up marks and make the test too easy’ argument, or the ‘students do not know how to use them and this undermines their performance’ argument, we need to take a proper look at the difference that bilingual dictionaries actually do make to writing performance.



Chapter 2.  On dictionaries and writing

However, the evidence provided by performance alone is insufficient to claim that a particular writing test is construct valid. We need, as Bachman (1990, 2000) and Messick (1989) suggest, to move beyond a conceptualization of construct validity that focuses solely on the test and interpretations of scores. Arguments surrounding what does and does not constitute a fair test have consequences for the stakeholders, whether good or bad, that have implications for that test’s validity. Construct validity must also consider the value implications and social consequences of the test – the impact of the scores on the test takers – and therefore whether the scores were affected by facets of the test procedure that biased the test against some test takers. We need to consider the differential impact of dictionaries on a range of test takers. Also, we need to find out what the test takers themselves think. Weigle (2002) suggests that an evaluation of test usefulness should not be separated from a consideration of fairness and equity. She relates issues of fairness to a canvassing of the views of stakeholders. Rea-Dickins (1997) suggests that perhaps the most important stakeholders who might be consulted are the test takers themselves. This view is in accord with Bachman and Palmer’s (1996) assertion that “one way to promote the potential for positive impact is through involving test takers in the design and development of the test, as well as collecting information from them about their perceptions of the test and test tasks” (p. 32). It also lines up with Messick’s (1989) recommendation that test-taker perceptions should be included as a crucial source of evidence for construct validity. It may be argued (East, 2005c) that such an investigation may just result in face validity – that is, the test is perceived to be reasonable, fair, and appropriate, but the perception may be of little value in evaluating test validity (Savignon, 1983). Test-taker perceptions are therefore often not seen as central to the test validation process because of this (Elder, Iwashita, & McNamara, 2002). But if test taker perceptions are laid alongside all the other evidence in a broader construct validation enquiry, such an enquiry may help to establish whether a given testing procedure is both useful and fair, particularly with regard to the test takers as major stakeholders. The questions we need to look at are these. 1. Does bilingual dictionary use affect test performance? There are two dimensions here. Do test takers actually get higher or lower scores when using a dictionary? Or does using a dictionary make no difference to test scores? Secondarily, does use of a bilingual dictionary enhance or diminish writing quality? There would seem to be little point in allowing a dictionary if it did not actually help students to improve the quality of their work (or, in the words of Ard (1982), to develop more active expressive abilities), or if it had a detrimental

35

36

Dictionary Use in Foreign Language Writing Exams

effect on the quality of their work (or, in Asher’s (1999) words, if inaccurate language production diminished rather than improved students’ performance). 2. What about the test takers? How do they use dictionaries? Do they think they can use them effectively to improve their writing? What do they think about having dictionaries in the test? How can we help the test takers to improve their use of dictionaries? Answers to these questions have relevance not only for timed tests, but also for coursework and portfolio options. In Chapter 3 I begin to explore how we might answer these questions effectively and in a way that will provide us with a comprehensive and multi-faceted picture. I look at the type of research studies we could set up to make a fair evaluation, and I include a summary of the research work of Hurman and Tall (1998), which was pivotal in the UK decision to remove dictionaries from L2 examinations.

chapter 3

Does the dictionary really make a difference?

There are several very good reasons why we should consider allowing students to take bilingual dictionaries with them into assessments of their L2 writing proficiency. Bilingual dictionaries are potentially supportive tools for L2 learners. Their use is authentic, that is, consistent with what L2 users do in the real world if they do not know or understand something. In Chapter 2 I suggested that the ­ ideal assessment of writing proficiency would surely aim to mirror such realworld practice as authentically as possible. Coursework and portfolio options would appear to do this quite well. The issue of allowing dictionaries in timed writing tests is less straightforward. Based on the need to preserve the measurement qualities of the test (the ability of that test to measure writing proficiency and to give us information about differential levels of test taker ability), there has been pressure to disallow dictionaries. They might, for example, make the test too easy. If test takers were allowed to use them we might get an inaccurate picture of their writing ability. Alternatively, dictionaries could potentially make the test more difficult for those who did not know how to use them effectively. Weigle (2002, p. 106) brings out several of the tensions around dictionaries in timed tests quite well. She argues that in traditional language tests vocabulary knowledge is regarded as part of the construct being tested, and on this basis dictionaries would usually be outlawed. She suggests, however, that if writing ability were defined more broadly in a way that included the use of any available resources, using dictionaries might become a possibility. Indeed, it could be argued that good writers know how to use dictionaries efficiently and in a way that enables them to choose the appropriate words for the concepts they wish to convey. As anecdotal evidence demonstrates, however, students often do not know how to use dictionaries effectively. Also, using dictionaries may impact negatively on the time available to complete a written task. If test takers cannot use dictionaries well this might cause them to underperform in writing tests. A range of issues therefore impacts on the use of dictionaries in writing tests and the consequent fairness of the test for everybody. The notion of what constitutes a ‘fair’ test lies at the heart of discussions around different types of testing. Certainly in virtually all types of testing students receive some kind of grade or mark which differentiates their performance

38

Dictionary Use in Foreign Language Writing Exams

from that of others. Tests clearly differ in the extent to which scores are used to provide evidence of ­performance, but wherever they are used a central concern of all the stakeholders is that those scores need to be reliable and fair, or construct valid. As explained in Chapter 2, reliability and construct validity have to do with the meaningfulness and accuracy of the scores awarded, in relation to the measurement of an underlying construct. The scores provide an indication of the ability or construct. Both reliability and validity are connected with the concept of fairness, with validity focusing on the task itself and reliability concerned with the consistency with which task performances are measured. Weir (2005) sees the two qualities as so inter-dependent that he describes reliability as ‘scoring validity’. The central importance of test score evidence means that if we are genuinely concerned with the difference a dictionary might make, it would be useful to find out whether test takers get higher or lower scores when using a dictionary, or whether using a dictionary actually makes no difference to scores. Test score evidence is crucial in helping us to begin to make some meaningful decisions around the usefulness and fairness of having a dictionary in writing tests. If, however, we want to carry out a thorough investigation into the difference, if any, that having a bilingual dictionary in writing is actually going to make, we ideally need to set up a study or studies which can draw on several sources of evidence – not just test scores. This chapter looks at the type of investigation we could set up if we wanted to get the fullest picture possible of dictionary use in writing tests. I explore the fundamental concept of a ‘repeated measures’ study, bringing out the types of consideration that are important to make this type of study effective. I go on to give an overview of Hurman and Tall’s 1998 repeated measures study into French dictionary use – one which was crucial in the recent debates around dictionaries in the United Kingdom. The studies which form the focus of the remainder of this book are then described, and I conclude with my own findings with regard to test scores and what we can learn from these. The first two sections of this chapter will be helpful for those who would like to get a fuller understanding of the methodological rationale for the studies I carried out. Readers who are experienced in the research process may want to skip these two sections and pick things up again on p. 41.

Setting up a comparative study If we are interested in seeing whether a class of L2 students actually performs differently in a writing examination when the students have access to a dictionary we



Chapter 3.  Does the dictionary really make a difference?

could set up a comparative study. We could divide the class into two groups, and then give the groups the same or a similar writing task, but in two different conditions – one group would take the test with a dictionary, and one would take it without. If we subsequently compared students’ performance on the two tests (as measured by their writing scores) we would be able to judge whether there were any differences, and, if so, the extent of those differences. We would look not only at the average performance in the groups, but could employ statistical measures to determine whether any observed score differences were most likely random and down to chance, or whether they were indicating the existence of some real factor (like the dictionary) which was helping to make the difference. The smaller the probability that the differences were down to chance, the more confident we could be that some intervening factor was having an effect. We could argue for a real (statistically significant) difference if the probability that score variations were random was once in at least every twenty (p ≤ .05). This type of independent groups study can provide us with adequate and valid comparative evidence, particularly when dealing with larger sample sizes. However, if sample sizes are small it is important to make sure that the two groups are comparable. Otherwise it would be difficult to isolate the intervention of the variable under consideration from any differences inherent in the groups.

Setting up a repeated measures study Another useful approach would be to work with only one group of students. In a repeated measures dictionary study we could give this one group of students two similar writing tasks in two different conditions – once with the dictionary, and once without. We could then compare each student’s performance in both testing conditions. We would also need to consider ways of isolating the effect of the dictionary from any other potentially impacting factors. In the actual test administration, three considerations are important. Firstly, if you allowed, say, a week’s gap between testing you would not know whether students’ performances improved or became worse due to some factors outside dictionary use – the students might have studied extra hard in that intervening week and thereby improved their performance in a way that was irrelevant to having a dictionary. We would need to control for this potential practice effect. A second type of practice effect may become apparent if the participants always took tasks in a particular order – for example, the ‘without dictionary’ task followed by the ‘with dictionary’ task. Participants might perform better on the second task because it was second rather than because they had a dictionary. We would need to take account of this order effect. Thirdly, one test task might

39

40 Dictionary Use in Foreign Language Writing Exams

be somewhat easier or more difficult for a number of participants in a way that was completely unrelated to dictionary use, and this might influence their performance. We would need to take account of this task effect. Practice effect would be minimized if the participants took the two tests one after the other with a minimal time-lag between each test administration. Order effect and task effect could be mitigated through counterbalancing. That is, some participants would take Task 1 followed by Task 2, and others would take them in the reverse order (two groups). Also, some would take Task 1 with a dictionary and Task 2 without, and others would take them the other way round (two more groups). This type of counterbalancing would thereby contribute to the elimination of biases which might invalidate any conclusions we draw (Keppel, 1991). Two further considerations are important. Firstly, we would need to ensure that the scoring procedures could be shown to be highly reliable and construct valid. As Weigle (2002) suggests, “scoring procedures are critical because the score is ultimately what will be used in making decisions and inferences about writers” (p. 108). This would include: •

• • •

drawing up sufficiently detailed scoring criteria which adequately reflected the construct being measured and which would be sensitive enough to differentiate between different levels of test taker performance; having at least two independent raters who received training in the use of the scoring criteria; putting measures in place to determine how consistently the raters used the scoring relative to each other; and making sure that the raters did not know which test script had been written in which test condition – with or without a dictionary – because knowing this might somehow influence their marking, albeit at a subconscious level.

In a study to determine the effect of using a dictionary this last point is critical. Spolsky (2001), for example, suggests that “it is believed that dictionary use improves performance on the examination” (p. 4, my emphasis) – a perception also mentioned by Bensoussan, Sim, and Weiss (1984). If, for example, the raters think that ‘with dictionary’ answers should be better they might be tempted to be harsher in assigning marks to ‘with dictionary’ scripts. Alternatively raters might be a little more generous in the belief that the test scores should indicate an improvement in writing. A second important consideration is to employ measures other than test scores to see if we could locate and explain more thoroughly what might be accounting for any differences. We could analyze the quality of the writing in other ways. We could use questionnaires, interviews, or focus groups as means of ­homing in on



Chapter 3.  Does the dictionary really make a difference?

participants’ use or perceptions of the dictionary. This type of ‘mixed method’ approach – drawing on both quantitative and qualitative data – provides greater breadth and depth, and arguably adds to the reliability and validity of any empirical investigation, because it is able to take into account multiple and distinctive forms of evidence (Greene, Caracelli, & Graham, 1989; Schutz, Chambless, & ­DeCuir, 2004). Comparative test score data are crucial, but other data add significantly to the available evidence.

Findings of an independent groups study Rivera and Stansfield (1998) argue for the importance of generating comparative test score evidence. These two researchers have been particularly concerned with allowing the use of test accommodations – dictionaries, glossaries, extra time allowance – when students with limited proficiency in English (so-called LEP students) are required to take mainstream tests in English alongside English L1 speakers. Rivera and Stansfield assert that accommodations aim to ‘level the playing field’ by eliminating irrelevant obstacles that may affect some test takers’ performance. Such accommodations are therefore designed to provide equality of opportunity for all students. They support weaker LEP students in assessments so that they are not hindered in displaying what they know and can do by a linguistic barrier – not being able to understand fully what the question is asking. There are, however, recognized problems with accommodations (Butler & Stevens, 1997): • •

•

It may be difficult to identify the subset requiring the accommodation and the needs of the subset. Unfair advantage or disadvantage may be given to test takers who either have or do not have access to a particular accommodation (indeed, test takers with the accommodation may be advantaged or disadvantaged by it, just as test takers without the accommodation may be). This may bring into question whether all test takers do approach the test with a ‘level playing field’, and whether the test is therefore valid.

Several of Butler and Stevens’ concerns are of direct relevance to the dictionary use question. It may be that some dictionaries would provide extra help or support that others do not. If students are allowed a free choice in the type of dictionary they take into the examination with them, some may have an unfair advantage. Also, test takers who are more proficient at dictionary use might be at an unfair advantage over those who cannot use a dictionary well.

41

42

Dictionary Use in Foreign Language Writing Exams

Unfair advantage or disadvantage is a threat to the construct validity of the test. Rivera and Stansfield (1998) go on to argue, however, that if score comparability between tests with and without an accommodation has been established (no significant difference in scores) the accommodation can be rightfully endorsed. By their argument, no significant difference in scores would mean that the accommodation is fair in that it does not compromise the measurement of the construct. To investigate whether this was likely to be true for LEP test takers, Rivera and Stansfield (2001) looked at the performance of over 11,000 non-LEP students on two sets of tests, some presented in standard non-simplified English, and some that had been linguistically simplified. Participants took one of the two types of test. The researchers found that there was no significant difference in scores between the two tests. Stansfield (2002) describes this finding as important because it demonstrates that linguistic simplification can be used for non-LEP test takers without concerns that it provides unfair advantage and thereby affects the comparability of scores across all test takers. This is because, from the point of view of non-LEP test takers (that is, those who did not require the accommodation), linguistic simplification did not make the test ‘too easy’ and did not therefore distort the scores as a measure of ability. Bearing in mind that this finding relates to a construct other than the L2 ability of L2 learners, the finding does have implications for the wholesale use of dictionaries in L2 tests. If test takers took tests in two conditions, with a dictionary and without a dictionary, and a non-significant result were to be found when test scores were compared, this would indicate that there was no crucial threat to the construct validity of the test as a snapshot measure of test takers’ writing performance – at least as far as we could tell from score evidence. That is, the L2 test takers did not ‘require’ the dictionary to demonstrate their ability and its inclusion was not a confounding variable that distorted the ability being tested – making the test seem ‘too easy’ or ‘too hard’.

Findings of a repeated measures study Hurman and Tall (1998) were also interested in obtaining comparative test score evidence. These researchers were commissioned by the UK’s Qualifications and Curriculum Authority to investigate the difference that bilingual dictionaries made in tests. In a study conducted with over 1000 participants they utilized a repeated measures design to look at the effect of using a bilingual English/French dictionary on scores in GCSE French writing at the two levels of the examination – Foundation and Higher Tiers. They were also interested in the comparative impact of different dictionaries on the scores awarded.



Chapter 3.  Does the dictionary really make a difference?

The study incorporated a variety of quantitative and qualitative elements so that it could go beyond test score evidence and paint a broader brush-stroke picture. These included two examination papers at both Foundation and Higher Tiers, designed to be as similar as possible, and taken by each participant (one with, the other without a dictionary), a participant questionnaire after writing the ‘with dictionary’ paper, a qualitative analysis of dictionary look-ups made by participants of different ability ranges across a subset of papers, and an observation schedule to record frequency and duration of the look-ups. Three basic types of dictionary were selected, based on the discovery that these three were in common use in schools (Tall & Hurman, 2000): a. Collins Pocket French Dictionary (Collins, 1995a) – a more ‘traditional’ type. b. Collins Easy Learning French Dictionary (Collins, 1996) – a newer type that aimed to make student use as easy as possible. c. Your French Dictionary (Malvern, 1996) – tailor-made for the GCSE. Some other dictionaries were allowed to some test takers, although the majority used one of the three specified dictionaries. It was considered that a useful comparison could be made between the three because they differed in content and presentation. The test papers used contained the types of task illustrated in Figure 3.1. Hurman and Tall discovered that Foundation Tier candidates gained an average increase of two marks (9%) on the ‘with dictionary’ paper in comparison to their scores on the ‘without dictionary’ paper. Higher candidates gained an average increase of between two and three marks (9%). A couple of marks difference might not seem very much, but the differences in scores were found to be statistically significant (Tall & Hurman, 2002). The use of different dictionaries appeared Foundation Tier

Higher Tier

Form-filling (ten one-word answers) Postcard writing (maximum 40 words) Letter writing (approx. 70 words) Extended writing (approx. 150 words)

Figure 3.1  Test task types across the two tiers of the GCSE.

43

44 Dictionary Use in Foreign Language Writing Exams

to have different effects on different types of question. It was demonstrated, for example, that at Foundation Tier the Collins Pocket might give a slight advantage, whereas at Higher Tier test takers were disadvantaged – the overlapping question was answered more effectively by those using Your French Dictionary and the extended piece of writing was answered more effectively by those using the Collins Easy Learning dictionary. In the light of Butler and Stevens’ (1997) argument around unfair advantage and disadvantage, these differences reveal important considerations: it is unfair if one group of test takers, using one type of dictionary, can thereby do better than another group with a different dictionary. Hurman and Tall also found that a good deal of examination time was spent using the dictionary. On average, participants accessed a dictionary at least once in every 2 minute period, with the majority of look-ups taking between 20 and 40 seconds. Although only 10% of look-ups took more than 60 seconds, there was clearly frequent and constant dictionary use throughout the tests which would have distracted somewhat from time available for task completion. Several instances of actual dictionary use by candidates were recorded (Hurman & Tall, 1998, p. 17). There were examples of phrases being taken directly, and successfully, from the dictionary:

French response la semaine dernière c’est tellement gênant

Meaning last week that’s really embarrassing

On the other hand, there was evidence of considerable confusion about the differences between, for example, verbs and nouns, and lack of understanding about the dictionary’s metalanguage. This resulted in many English expressions being directly transliterated into meaningless French: Error J’ai propré / propu ma chambre Je s’éteindre à 8h.

Intended meaning I cleaned my room I went out at 8 o’clock

Puis semaine

Next week

Je gauche

I left

Nature of error An inappropriate word was selected (‘clean’ as an adjective rather than a verb). An inappropriate verb was selected (s’éteindre = to extinguish) and was used in its infinitive form. An inappropriate word was selected (‘next’ as an adverb when an adjective was required). An inappropriate word was selected (‘left’ as an adjective and not as the past participle of ‘leave’).



Chapter 3.  Does the dictionary really make a difference?

Je suis deborder des I spilt some on a déborder (spill) and des (some) were sur un madame lady incorrect in the context, and the gender of madame was also incorrect. Une fille abattre A girl fell down Inappropriate words (‘fell’ as in ‘chop’ and duvet ‘down’ as in ‘duvet’).

In addition to errors such as these a large majority of participants failed to use the dictionary to check genders or spellings of words. These types of inaccurate dictionary use pointed clearly to a distinct weakness of dictionary availability in writing tests, and one which would bring into question whether they should be allowed. The questionnaires revealed that 62% of participants had been trained to use a dictionary. This is a high number in comparison with findings of other research (Atkins & Varantola, 1998a; Hartmann, 1983, 1999) although in setting up the project teachers had been asked to ensure that participants had received some instruction in dictionary use. Only 42%, however, claimed to have been trained to use the dictionary they had in the examination. Nevertheless 95% felt able to find the French equivalent of an English word and 94% said they could do the converse. Given the instances of error, this is a self-confidence not necessarily matched with proficiency. Some test takers may have thought that they could use a dictionary better than they in fact could (as has already been stated, if students do not know how to use dic­tionaries effectively use of a dictionary during a language test may be counterproductive). Nevertheless, 88% of participants commented that being able to use a dictionary in an examination made them feel more confident, and 3% admitted to using it more frequently than necessary “because it was there” (Hurman & Tall, 1998, p. 26). This extra boost to confidence in the examination is an important consideration because it potentially leads to a more positive feeling about the test and more positive engagement with the task. In view of the finding that participants scored higher marks when using a dictionary, Hurman and Tall (1998) recommended that GCSE boards should take potential increases in scores into account when setting questions and establishing grade boundaries. Their recommendations underscored the steps they perceived as necessary to ensure that ‘with dictionary’ tests were as useful and fair as possible, particularly noting the following (p. 48): • • •

The content of what was written about could be expanded. There could be less limitation on the language used in the test task or expected in the response. The mark scheme could put more emphasis on range and quality of language.

45

46 Dictionary Use in Foreign Language Writing Exams

• •

The mark scheme could require a higher level of accuracy. Time allowed to complete the paper may need to be extended to allow more time for dictionary use.

Consequences of the Hurman and Tall study There is a good deal that we can learn from Hurman and Tall’s study, at least for beginner and pre-intermediate test takers. Their findings, especially the uses to which the dictionaries were put, raise legitimate questions of concern about the dictionary as a potential liability. This research stands us in good stead as we consider the whole issue of dictionaries in writing, whether in testing or non-testing contexts. Hurman and Tall’s study also had huge ramifications in the UK. A bulletin from Birmingham University (2000) reported what were seen as the undeniable consequences of their findings. Under the heading ‘Dictionaries Fail the Test’, it was stated: Some may regard John Hurman and Graham Tall in the School of Education as spoilsports. The Government’s Qualifications and Curriculum Authority, however, fully endorses their research into the effect of taking dictionaries into modern languages examinations. Dictionaries are now banned.

The bulletin went on to underscore the particular place that the test score evidence – that test takers seemed to do better on ‘with dictionary’ tests – appeared to play in this decision: Their findings unequivocally showed that using dictionaries, especially the newer types of language dictionaries now on the market, drives up marks. … The researchers did not go on to expressly recommend that dictionaries should not be allowed in the exam room, but the QCA had no hesitation in outlawing them. It makes languages examinations harder, of course, but it also makes them a better test of ability.

These somewhat emphatic observations appeared to support a perspective that the standard of examinations was being compromised by dictionary access. The observations are reminiscent of Bishop’s (1998) commentary that having ­dictionaries was somehow like cheating or that they made the assessment impossible to set or the task too easy to achieve. Hence, a decision to remove dictionaries from the examinations would make these examinations fairer and more useful measures of test taker ability.



Chapter 3.  Does the dictionary really make a difference?

Building on past research No doubt Hurman and Tall’s test score evidence was one of several factors influencing the policy decision of the QCA (see, for example, Poulet, 1999). Nevertheless, given that they were part of the evidence available, Hurman and Tall’s test score findings are a concern. Firstly, there were several limitations to their scoregathering process (East, 2007). Also, Hurman and Tall had focused only on the GCSE, and yet the QCA dictionary ban also included higher level examinations. There is a world of difference between the type of language required for what Barnes et al. (1999) refer to as ‘single item recall’ tests – test takers giving, for example, a list of ten food items – and the extended discourse required in a higherlevel discursive essay. As Barnes et al. conclude, “[t]he degree of ease or difficulty of some tasks is altered considerably with the use of a dictionary” (p. 24). This may well make it more appropriate to exclude a dictionary from some types of writing task, but it does not necessarily follow that dictionaries should be outlawed at all levels. Finally, in the light of Rivera and Stansfield’s (1998) argument that it is only when no significant difference in test scores is found that we can begin to consider seriously whether to allow a particular support resource, there is a need to gather further evidence, especially with higher levels of writing. In the light of all this I carried out the studies which are explored in the remainder of this book. The three studies I present were formulated so that they could build on Hurman and Tall’s research. For comparative purposes the studies investigated a different foreign language (German in contrast to French) and higher levels of test taker ability (intermediate and beyond in contrast to beginners and pre-intermediate). Given the unequivocal stance of the University of Birmingham concerning Hurman and Tall’s test score evidence, it was important to take all the necessary steps to ensure the reliability of my own scoring evidence. Particular attention was therefore paid to test scores. I also wanted, as Hurman and Tall had done, to go beyond test score evidence to several other dimensions of the test taking experience – including a qualitative analysis of test taker performance and a focus on test taker perspectives. The first two investigations were small-scale exploratory case studies of specific students taking German courses at the intermediate level or above in a New Zealand tertiary institution, and who had been my own students (see Appendix 1 for an explanation of how I understand the term ‘intermediate’). At the time of the studies the institution offered six 14-week courses (two per year), forming part of a three-year Bachelor of Arts degree. The courses were also open to non-degree students on an ad hoc basis. Both studies drew on a small sample (n = 6 and n = 5). The data provided a rich picture of dictionary use from a variety of angles.

47

48 Dictionary Use in Foreign Language Writing Exams

The third study built on the findings and conclusions of the first two, but was broader in scope. It involved 47 high-school students, all 17 to 18 years of age, taking German courses in schools in the Auckland and Northland regions of New Zealand in the academic year 2003. These students were preparing for an external high stakes examination in German (equivalent to A level) known as ‘Bursary’, which they were due to take two months after taking part in the study. This is an intermediate level examination which can be benchmarked to levels B1/ B2 on the Common European Framework or CEF (Appendix 1). A range of different schools was targeted, including state and independent schools, and single sex and co-educational schools, so that a good cross-section of students could be included. The students were recruited via a letter of invitation, sent to German teachers, who were asked to discuss the project with their students and enlist the students’ involvement. The final number of students who took part (39 girls and 8 boys from 11 schools) was quite small. Statistics from New Zealand’s Ministry of Education indicated, however, that on a national level 276 girls and 90 boys in 88 schools were preparing for the Bursary in German in 2003. The sample for the third study therefore represented 14% of girls taking Bursary German, 9% of boys, and overall just under 13% of the total cohort in 12.5% of schools. It was considered that the sample could be regarded as largely representative of the population to which any results might be generalized. It also allowed for a more quantitative investigation using statistical analyses. Generalizability of findings is an important consideration. That is, it is important that the lessons we learn from a particular study can be applied more generally beyond the study to other situations. Johnson and Christensen (2004) suggest that “[p]erhaps the most reasonable stance toward the issue of generalizing is that we can generalize to other people, settings, times, and treatments to the degree to which they are similar to the people, settings, times, and treatments in the original study” (p. 256). Bearing this in mind I leave the extent of the transferability of the findings to other contexts to the reader’s judgment. A similar design was used in all three studies: 1. All participants took a placement test (Oxford University Language Centre, 2003). This test was used to provide a measure of the participants’ abilities for benchmarking purposes. The test categorizes students into one of six bands, from beginner to advanced. 2. They took two timed writing tests, one after the other. In the ‘with dictionary’ test they were asked to indicate, by underlining or highlighting, the items they had used as a result of dictionary look-up. The test scripts were subsequently analyzed in a variety of ways, both quantitative (for example, using lexical



Chapter 3.  Does the dictionary really make a difference?

f­ requency profiling (Laufer & Nation, 1995)) and qualitative (for example, looking at actual instances of dictionary use). 3. They completed two short questionnaires which compared test takers’ preferences and strategies across the two test conditions. Attitudinal questions also focused on the participants’ personal opinions about the relative ease or difficulty of the tasks across the two conditions, and the impact of this on their sense of confidence. 4. They filled in a longer questionnaire with both open- and closed-ended questions, with the open-ended questions providing opportunity for exploration, probing, and elaboration (Peterson, 2000). Questions explored such aspects as how the dictionary was used, and perceived advantages and disadvantages of having a dictionary in the examination. All three studies followed a similar schedule. Participants were allowed 50 minutes to complete the first timed writing test, whether with or without a dictionary. They then filled in a short questionnaire on that test task (5–10 minutes). This procedure was repeated for the second test. After both tests had been completed, participants were asked to fill in the longer questionnaire. Throughout the process of data gathering and analysis, procedures were put in place to ensure that the results derived from data were reliable (see Appendix 2). The small-scale case study nature of the first two studies also enabled two additional sources of data collection that were not used in the final study – participant observation and interviews. The particular focus of the observations was on participants’ use of time – that is, time spent accessing the dictionary and number of dictionary look-ups. After the completion of the tasks, the participants were interviewed in groups, with the interviews designed to provide opportunity for the participants to explore any aspects of the study that particularly interested them. Following on from the longer questionnaire, questions included how test takers had used the dictionary, and what they thought about using the dictionary. The interviews were audio-taped and subsequently transcribed so that I could revisit them later with a view to making a closer analysis.

The first two studies The six participants in the first investigation were drawn from a range of BA classes. The placement test indicated that three could be classified as ‘intermediate’ (Simon, Patricia, and Jenny), and three as ‘advanced’ (Sharon, Rachel, and Janet). Of the five participants in the second study, three were ‘intermediate’ (Mary, Jessica, and Peter) and two ‘upper intermediate’ (John and Sandra). These participants

49

50

Dictionary Use in Foreign Language Writing Exams

could be differentiated from those in the first study because they were members of a particular class, following a 14-week German course (Deutsch 3). This was the third of the six available courses, and was set at a level of language proficiency broadly commensurate with the intermediate level B1 of the CEF (see Appendix 1). All the participants in both these small-scale studies brought with them significant background knowledge and prior learning in German. The majority were ‘mature’ students. In the first study no level of prior specific learning in writing skills or use of a dictionary was assumed. In the second study, however, the development of writing and dictionary skills was a major component of the course (East, 2005b, 2006c). To help with this participants were issued with two study guides at the start of the course: 1. A ‘Writing Study Guide’ dealt with key issues relevant to writing communicative texts. The guide provided practice in using devices such as adjectives, adverbs, conjunctions, and formulaic sequences (Wray, 2002) that would help students to develop both coherence and cohesion and enhance the quality of their writing. Participants were encouraged to learn a good number of the sequences suggested in the guide. Also included were examples of test tasks similar to those the participants would be carrying out at the end of their course, data from which would be used as part of the study. The guide also suggested ways of answering the tasks. 2. A ‘Dictionary Skills Guide’ contained 14 exercises designed to help participants to find their way around a bilingual dictionary. Most exercises were available for the participants to do in their own time, and answers were provided for self-checking. The participants were given ownership of practicing the skills themselves, thereby developing their own autonomy in the learning process. In both studies, students were required in the tests to use the same one-volume bilingual dictionary – the Collins German Dictionary (Terrell, Schnorr, Morris, & Breitsprecher, 1997), and participants were issued with a copy to take with them into the examination room. I selected this dictionary because it is among the most comprehensive published by HarperCollins. Also, one early survey into bilingual English/German dictionaries (Hartmann, 1983) had established that the original Collins German Dictionary, on which this dictionary was based, was considered the most popular out of the more than 25 dictionaries represented: of those who knew the Collins German Dictionary, 96% saw it as ‘excellent’ or ‘good’, 17% ahead of the next most popular. As such, Hartmann observes that this dictionary did “surprisingly well” as a “relative newcomer” (p. 197), first published in 1980. There was therefore some evidence to suggest that the 1997 edition of the



Chapter 3.  Does the dictionary really make a difference?

Collins ­German Dictionary would serve its users well. The dictionary contained over 280,000 references and 460,000 translations. Participants in the second study were also given practice opportunity in class time with using this dictionary. The edition of the dictionary used was not fully compliant with the neue Recht­ schreibung – the new spelling rules introduced in 1996 which were designed to simplify and rationalize German spellings (Heller, 2000). The dictionary did alert its users to the new rules through a detailed appendix section, although the rules only became fully mandatory in schools in Germany in 2005, and participants in the studies were not penalized if they did not follow them. In addition to its extensive vocabulary, the dictionary also has a large central bilingual section containing some 70 pages of formulaic sequences and other items – Language in Use / Sprache Aktiv. This is a distinguishing feature of this dictionary. The intended aim of the section is “to help non-native speakers find fluent, natural ways of expressing themselves in the foreign language, without risk of the native-language distortion that sometimes results from literal translation” (Terrell et al., 1997, p. 819). I drew my students’ attention to this section, and explained to them that the phrases and examples, used effectively, would enable them to enhance the quality of what they were writing. In the first two studies three types of writing examination task were set, which were labeled A, B and C. Two papers were provided for each task type (Task 1 and Task 2), so that one could be completed with a dictionary, and the other without. In the first study each participant carried out one of these tasks, and in the second each participant took all three tasks over a three-week period at the end of the course. The tasks have been reproduced in Appendix 3. The first two sets of tasks required test takers to write a letter in response to a stimulus presented in German. In the first task type (A1 and A2) test takers were given a series of bullet points in English to guide their responses. In the second task type (B1 and B2) the bullet points were in German. In the third set of tasks (C1 and C2) students were presented with three discursive essay titles, in German, and were asked to respond to one of the tasks. Counterbalancing of task was partial (due to the small sample size), and focused on test condition rather than task order. That is, although all participants took Task 1 followed by Task 2, some took Task 1 with a dictionary, and some without.

The third study The third study was distinct from the first two in three respects: number and type of participants (47 participants, 17 to 18 years of age); type of test task used (two discursive essays); and types of dictionary used (free choice). Participants were

51

52

Dictionary Use in Foreign Language Writing Exams

Figure 3.2  Fit of placement test scores to normal distribution. Note. The Anderson-Darling test revealed no reason to reject the null hypothesis of normality (AD = .405, p = .34).

issued with a version of the Writing Study Guide about six months prior to taking the test tasks, and they and their teachers were encouraged to make use of it in class and at home. In practice many students did use it, although it was not mandated as a requirement to take part in the study, and did not form such a central part of test preparation as it had done in the second study. The range of placement test scores (15/50 to 47/50) indicated a broadly normal distribution (Figure 3.2). Participants were placed into four ability levels based on the suggested bands: lower intermediate (n = 5), intermediate (n = 22), upper intermediate (n = 16) and advanced (n = 4). The two writing tasks used in this study were similar to the types of essay these high school students would be expected to complete, without a dictionary, in their final Bursary examination. The main difference was that in the Bursary examination students would have a choice of titles. In the study no choice of title was given so as to contribute to controlling for task-related differentials in performance: Task 1 Sprachen in Neuseeland: “Alle Schüler sollen heutzutage mindestens EINE Fremdsprache in der Schule lernen”. Sind Sie auch dieser Meinung?

Translation Languages in New Zealand: “These days all school students should learn at least ONE foreign language in school”. Are you also of this opinion?



Chapter 3.  Does the dictionary really make a difference?

Task 2 Der Massentourismus: “Viele Touristen aus vielen Ländern besuchen heutzutage Neuseeland, und das ist gut für das Land”. Was meinen Sie?

Translation Mass Tourism: “These days many tourists from many countries visit New Zealand, and that is good for the country”. What do you think?

Because participants were free to select their own bilingual dictionary they were encouraged to use one that they had used before. In all, five different types of dictionary were used (Table 3.1). Among the most popular was the Collins Pocket German Dictionary (Collins, 2001). If participants did not have access to a dictionary on the day of the test they were provided with a copy of this dictionary. This enabled an investigation of the Collins Pocket users as a subset of the whole sample, with the subset using only one dictionary type. (In Hurman and Tall’s (1998) study the Collins Pocket French Dictionary had fared badly in comparison with the other two they had investigated, particularly at the higher GCSE level. Using the Collins Pocket German in this study allowed for some small-scale comparison between the Collins Pocket and the other dictionaries.) I was also interested in knowing how often the participants had used the dictionary before taking the tests. On the longer questionnaire I asked them about this. Participants’ responses to this question were translated into four levels of prior experience with a dictionary: 1. 2. 3. 4.

I have never used [this dictionary] before today → very inexperienced I have used it once or twice in class or at home → quite inexperienced I have used it fairly often in class or at home → quite experienced I have used it quite regularly in class or at home → very experienced

It was therefore possible to place the participants in this study into four ability levels and four levels of prior experience with the dictionary, and this information was taken into account in subsequent statistical analyses. To help to take account of task effect and order effect participants in the third study were placed into one of four groups with roughly equal numbers. The distribution of participants resulted in the following: • • •

Just under half the participants took Task 1 followed by Task 2, and the remainder took these tasks in the reverse order. Just under half the participants wrote Task 1 with a dictionary and Task 2 without, and the remainder wrote Task 1 without and Task 2 with. Roughly half the participants were able to use a dictionary in the first task, and the other half in the second task.

53

54

Dictionary Use in Foreign Language Writing Exams

Table 3.1  Dictionary types used in the third study. Title

Number of references

Distinguishing features

No. of participants using this dictionary

Collins Pocket German Dictionary (Collins, 2001)

Information not available

All headwords are printed in blue.

31

Collins German College Dictionary (Collins, 1995b)

Over 83,000 references and 120,000 translations

Some references include cultural information.

 5

Collins German Dictionary plus Grammar (Collins, 1995c)

Over 83,000 references and 120,000 translations

An additional grammar section of 255 pages. (Reference section identical to the Collins German College Dictionary.)

 5

Over 50,000 referOxford Duden Paperback Dictionary ences and (OUP, 1997) 75,000 translations BBC German 42,000 references Learners Dictionary (BBC/Larousse, 1997)

Covers “the essential vocabulary   3 of everyday life” and is “specially designed to meet the needs of students, tourists, and travelers.”  3 Includes cultural notes and detailed coverage of GCSE word lists.

The test score evidence from the three studies In order to ensure the accuracy of scoring I designed a thorough and detailed diagnostic scoring rubric. The rubric drew on Canale and Swain’s (1980) ­theoretical framework of communicative competence. It was also influenced by two other rubrics that were already in use in different contexts (Edexcel, 2000; Jacobs, Zinkgraf, Wormuth, Hartfiel, & Hughey, 1981). The scoring rubric enabled raters to differentiate between eight levels of performance (from 0 to 7) across five separate facets of the writing (Figure 3.3). Each facet was given equal weight, meaning that test takers could receive a potential maximum total score of 35 (5 × 7). Although it was recognized that applying such a thorough scoring procedure would take the raters some time, I considered that this extra time was worth the effort required to provide accurate marks that could differentiate clearly across different aspects of writing and different levels of performance. In the first study I marked the scripts myself so that I could see how the rubric might work in practice. To provide some measure of reliability, however, I asked a colleague who did not know any of the participants to moderate my scores. In



Chapter 3.  Does the dictionary really make a difference?

the second and third studies the scoring was carried out by two raters who were asked to rate the scripts independently. Two different sets of raters were used in the second and third studies. The raters were trained in the process of scoring and were then given some practice in applying the rubric using examples from the first study (or, in the case of the third study, from the second study). The raters marked anonymous word-processed copies of the essays so that they would not know in which test condition essays had been written (because the test takers had been asked to underline instances of dictionary use in their ‘with dictionary’ answers the hand-written versions could not be used). Considerable effort was therefore made to ensure that the final scores awarded captured, as reliably as possible, the actual performance of the writers under consideration compared against the rubric (East, 2007), and high levels of both inter- and intra-rater reliability were reached (see Appendix 2). Table 3.2 presents the final mean scores awarded in all three studies, together with the range of scores awarded, and the standard deviations from the means. In all cases the mean scores across the two test conditions were roughly comparable. Average scores between the two testing conditions, considered on a study by study basis, differed by a maximum of 1 or 2 marks. A closer look at the scores awarded across all three studies indicates the following: • • •

In the ‘with dictionary’ tests scores ranged from 13/35 to 35/35 with a modal (most common) score of 21/35 (awarded 10% of the time). In the ‘without dictionary’ tests the range was 13.5 / 35 to 34.5 / 35, with a modal score of 20 (again awarded 10% of the time). Although some test takers did do better on the ‘with dictionary’ tests, and some did worse, the average difference in scores between the two conditions, when calculated from the raw scores across all studies, was only ±0.25 of a mark. There was a whole range of scores, but no meaningful difference at any level.

I also took the opportunity to look more closely at different sets of results to determine if any particular factors – like language proficiency, prior experience with using a dictionary, and type of dictionary – appeared to have any effect on test scores. In the first study I was able to compare three intermediate level students with three advanced students, all using the Collins German Dictionary. The three intermediate students’ scores differed across the test conditions by less than half a mark (‘with dictionary’ M = 24.7, SD = 3.2; ‘without dictionary’ M = 24.3, SD = 3.8). The scores for the ‘advanced’ students’ likewise showed minimal difference (‘with

55

Figure 3.3  The scoring rubric.

task is well understood ideas are very clearly stated very well-organized logical sequencing cohesive fluent expression the writing generally ‘flows’

• • • • • 6 • •

• task is understood • loosely organized but main ideas stand out • mainly logical but some incomplete sequencing • mainly cohesive • mainly fluent expression 5 • the writing is a little bit ‘choppy’

implications of task fully understood ideas are extremely clearly stated extremely well-organized very logical sequencing very cohesive very fluent expression the writing ‘flows’

• • • • • 7 • •

Cohesion, coherence and rhetorical organization Grammatical competence: Syntax, sentence-grammar semantics

• rich and complex language • errors are only of a very minor nature • language is very varied and • very few errors of appropriate to the task •   agreement •   tense • very effective choice and usage •   number of words / idioms / functions •   gender • there is solid mastery of word •   word order form •   function • the meaning is not obscured •   articles •   pronouns •   prepositions • language is varied, wide• errors are generally of a minor nature ranging, and appropriate to • a few errors of the task •   agreement •   tense • effective choice and usage of •   number words / idioms / functions •   gender • there is mastery of word form •   word order • the meaning is not obscured •   function •   articles •   pronouns •   prepositions • there are occasional errors in • simple constructions are used accurately choice and usage of words / • minor problems with complex constructions idioms / functions • several errors of • the meaning is very rarely •   agreement •   tense obscured •   number •   gender •   word order •   function •   articles •   pronouns •   prepositions • these are mainly of a less serious nature • the meaning is very rarely obscured

Knowledge of lexis, idiomatic expressions. Functional knowledge

• letter correctly set out, with appropriate and good opening and closing • essay has a very good introduction and conclusion • essay argues the for and against of the topic and provides opinion as appropriate • there is almost always appropriate register • generally appropriate use of cultural references if needed

• letter correctly set out, with appropriate opening and closing • essay has a good introduction and conclusion • essay argues the for and against of the topic and provides opinion as appropriate • there is mainly appropriate register • some cultural references if needed

• demonstrates good mastery of spelling and punctuation conventions • occasional errors of spelling, punctuation, capitalization, and paragraphing • the meaning is very rarely obscured

• •

•

•

•

Knowledge of register and varieties of language; knowledge of cultural references (where appropriate) letter correctly set out, with very appropriate opening and closing essay has a clear and effective introduction and conclusion essay effectively argues the for and against of the topic and provides opinion as appropriate there is always appropriate register appropriate use of cultural references if needed

• demonstrates very good mastery of spelling and punctuation conventions • few errors of spelling, punctuation, capitalization, and paragraphing • the meaning is not obscured

• demonstrates excellent mastery of spelling and punctuation conventions • very few errors of spelling, punctuation, capitalization, and paragraphing • the meaning is not obscured

‘Mechanics’: Spelling and punctuation

56 Dictionary Use in Foreign Language Writing Exams

•

•

•

•

0

no rewardable response

• very limited understanding of the task • there is little knowledge of • ideas are very confused or disconnected German vocabulary, idioms, • virtually no logical sequencing and functions development • the meaning is very confused 1 • virtually no cohesion or obscured • virtually no fluency • the writing is extremely ‘choppy’

• limited understanding of the task • ideas are confused or disconnected • very minimal logical sequencing and development 2 • very weak cohesion • hardly fluent • the writing is very ‘choppy’

• some understanding of the task • ideas are somewhat confused or disconnected • lacks logical sequencing and development • weak cohesion 3 • not very fluent • the writing is quite ‘choppy’

• letter generally correctly set out, with an attempt at an opening and closing • essay has some introduction and conclusion • essay argues to some extent the for and against of the topic and provides opinion as appropriate • there is some appropriate register • minor cultural references if needed

• demonstrates some mastery of spelling and punctuation conventions • frequent errors of spelling, punctuation, capitalization, and paragraphing • the meaning is somewhat confused or obscured

• demonstrates limited mastery of spelling and punctuation conventions • very frequent errors of spelling, punctuation, capitalization, and paragraphing • the meaning is confused or obscured

• letter correctly set out, with satisfactory opening and closing • essay has a satisfactory introduction and conclusion • essay satisfactorily argues the for and against of the topic and provides opinion as appropriate • there is generally appropriate register • some cultural references if needed

• demonstrates satisfactory mastery of spelling and punctuation conventions • more frequent errors of spelling, punctuation, capitalization, and paragraphing • the meaning is seldom obscured

• letter barely correctly set out, with little attempt at an opening and closing • essay has a limited introduction and/or conclusion • essay argues in a limited way the for and against of the topic and provides limited opinion as appropriate • there is limited appropriate register • very minor cultural references if needed • virtually no mastery of sentence construction rules • demonstrates very limited • letter barely correctly set out, with no attempt at • almost exclusively dominated by error mastery of conventions an opening and closing • the meaning is very confused or obscured • dominated by errors of spell- • essay has no introduction and/or conclusion ing, punctuation, capitaliza- • essay argues in a very limited way the for and tion, and paragraphing against of the topic and provides very limited • the meaning is very confused opinion as appropriate or obscured • there is very inadequate appropriate register • minimal cultural references if needed

• simple constructions are mostly used accurately • more noticeable problems with complex constructions • several errors of •  agreement •  tense •  number •  gender •  word order •  function •  articles •  pronouns •  prepositions • these are of a major or more serious nature • the meaning is seldom obscured there are frequent errors in • major problems with both simple and complex choice and usage of words / constructions idioms / functions • frequent errors of the meaning is somewhat •  agreement •  tense confused or obscured •  number •  gender •  word order •  function •  articles •  pronouns •  prepositions • the meaning is somewhat confused or obscured there are very frequent errors • some very minor mastery of sentence construcin choice and usage of words / tion rules idioms / functions • mostly dominated by error communicates, but the meaning • communicates, but the meaning is confused or is confused or obscured obscured

• task is mostly understood • there are more frequent errors • loosely organized but main ideas in choice and usage of words / stand out idioms / functions • generally logical but several instances of • the meaning is seldom incomplete sequencing obscured • satisfactory cohesion • some fluent expression 4 • the writing is somewhat ‘choppy’

Chapter 3.  Does the dictionary really make a difference? 57

58

Dictionary Use in Foreign Language Writing Exams

Table 3.2  Final mean scores across all studies. Study

1 (n = 6) 2 task A (n = 5) 2 task B (n = 5) 2 task C (n = 5) 3 (n = 47)

M SD M SD M SD M SD M SD

Average test score ‘with ‘without dictionary’ dictionary’

Range of scores ‘with ‘without dictionary’ dictionary’

28.5   5.01 26.6   5.5 25.6   3.8 23.2  4 22.7   6.1

26–34

26–34

18–33

19–30

20–28

22–28

20–28

20–31

13–35

13.5–34.5

28   5.33 24.6  5 24.8   2.8 24.4   4.9 22.6   6.1

dictionary’ M = 32.3, SD = 2.9; ‘without dictionary’ M = 31.7, SD = 4). The comparison did show, however, that the advanced students performed better on average regardless of test condition, even though the dictionary made no difference at either level. In the second study I was able to take a look at differences in performance by the same set of test takers, each using the same dictionary, across three different tasks. Once more no meaningful differences were noted. The type of test task did not appear to be a factor in influencing the extent to which having or not having a dictionary might make a difference. The third study provided the opportunity to consider several factors – level of ability, level of prior experience with a dictionary, and the number of look-ups made. Table 3.3 presents score differences, first for all test takers, and then for the Collins Pocket dictionary users, considered against the four levels of ability (as determined by the placement test). These descriptive statistics indicate the following: •

•

For all dictionary users the dictionary appeared to help the lower intermediate participants to improve their overall score by just over one mark when they had access to the dictionary, whereas advanced participants appeared to do slightly worse by just under one mark. There was no apparent difference to overall test scores for the other levels of ability. For the Collins Pocket users there was a slight improvement in the ‘with dictionary’ test (but in all cases by no more than around one mark) for lower and upper intermediate participants, and a slight improvement in the ‘without dictionary’ test for intermediate and advanced participants.



Chapter 3.  Does the dictionary really make a difference?

Table 3.3  Overall scores by level of ability – third study. Levela

‘With’

‘Without’

‘With’

All users LI (n = 5) I (n = 22) UI (n = 16) A (n = 4)

M SD M SD M SD M SD

‘Without’ Collins Pocket users

Total Score

Total Score

Total Score

Total Score

16.2   2.2 20.1   3.1 25.9   5.8 32.3   5.2

14.9   1.8 20   3.1 25.8   5.4   33   1.1

16.3   2.9 19.9   3.5 27.4   5.9 31.3   5.9

15.5   2.3 20.3   3.2 26.6   5.1 32.5   0.5

Note. aLI: lower intermediate, I: intermediate; UI: upper intermediate; A: advanced. Adapted from East (2007, p. 344).

It also seemed to be the case that score and ability level were related. As ability level went up, so too did the scores awarded. It certainly seemed that the scoring rubric was able to distinguish clearly and accurately between levels of ­performance commensurate with test taker proficiency levels. Nevertheless there was no evidence that the dictionary either enhanced or diminished performance at any ability level. Table 3.4 records the mean scores according to the four levels of prior experience with the dictionary (as determined by the ‘prior experience’ question on the longer questionnaire). There appeared to be no evidence to suggest that experience with the dictionary, viewed independently of level of ability, made any positive difference to participants’ performance in the ‘with dictionary’ condition. That is, very ­inexperienced and quite experienced participants improved scores when writing with the dictionary, whereas quite inexperienced and very experienced dictionary users’ scores were less high. These two distinct trends (that is, the differential impact of level of ability and prior experience with the dictionary on the mean test scores) are illustrated for the entire sample in Figure 3.4. The relationship between the number of look-ups participants self-reported that they had used in their response (that is, the number of words they underlined) and final ‘with dictionary’ scores is illustrated in Figure 3.5. Evidence from the scatterplot suggests that there was no relationship between the number of look-ups made and scores on the ‘with dictionary’ test.

59

60 Dictionary Use in Foreign Language Writing Exams

Table 3.4  Overall scores by level of experience – third study. Experience

‘With’

‘Without’

‘With’

Collins Pocket users

All users Very inexperienced (n = 10) Quite inexperienced (n = 15) Quite experienced (n = 13) Very experienced (n = 9)

M SD M SD M SD M SD

‘Without’

Total Score

Total Score

Total Score

Total Score

23.8   5.4 22   5.4 23.5   7.5 21.5   6.1

22.9   5.1 23.8   5.4 21.7   7.4 21.4   6.7

24.1   6.1 22.3   6.5 25.3   8.4 20.6   3.7

22.8   5.7 24.6   5.5 23.8   7.8 21.3   4.7

Adapted from East (2007, p. 345).

The crucial question to ask with regard to the test score evidence was whether any observed differences were significant. Various inferential tests were carried out (see Appendix 4). In all three studies paired samples t-tests indicated that none of the observed differences in scores was statistically significant. In other words, according to the test score evidence from these sets of test takers, having a dictionary in the writing test was apparently making no difference whatsoever. What little variation there was between scores was most likely down to chance or random factors. In the third study I was also able to take account of prior experience with the dictionary, level of ability, and the number of times the test takers reported that they had used a dictionary. It was found that neither prior experience with a dictionary nor the number of look-ups made were significant factors in predicting ‘with dictionary’ scores. It did not seem to matter how ‘dictionary-wise’ the participants were, nor how often they used a dictionary – their ­performances as judged by their scores were not influenced by either of these ­factors. Level of ability did make a significant difference to performance regardless of test condition – the higher the ability level the higher the score. Although these differences did not tell me anything about the difference that the dictionary made, they did confirm the evidence that the test scores were able to differentiate quite sensitively between different levels of ability.



Chapter 3.  Does the dictionary really make a difference?

Figure 3.4  Impact of ability and prior experience on average total scores. Note. Trends for both ‘with dictionary’ and ‘without dictionary’ scores are recorded for the purpose of comparison, although prior dictionary experience is not relevant to scores in the latter test condition.

Figure 3.5  Relationship between ‘with dictionary’ scores and number of look-ups made.

What does the test score evidence mean? All of this test score evidence raises several interesting questions. Spolsky (2001) has suggested that it is believed that dictionary use improves performance on

61

62

Dictionary Use in Foreign Language Writing Exams

examinations, and Hurman and Tall (1998) found a significant improvement in students’ writing (as measured by scores) in ‘with dictionary’ tests. I had found no differences. When I asked myself whether the steps I had taken in these three studies were sufficiently adequate to provide meaningful scoring evidence I ­concluded that they were. I was careful to develop comprehensive scoring criteria that could differentiate sensitively between different levels of test taker performance across several individual facets of writing. In all cases the raters would not have known which scripts had been written in which test condition, and counterbalancing (with some students taking a given test with a dictionary, and some taking the same test without) would have ensured that they would not be able to guess the test condition with any accuracy. Two potential subjective rating factors were thereby reduced. When reliability checks were carried out on the scores (see Appendix 2) it was found not only that inter-rater and intra-rater reliability more than adequately reached acceptable thresholds but also that the scoring rubric was able to differentiate across a range of levels of performance with a high degree of precision. I was left with what appeared to be a clear and unambiguous result. Each study had its own unique features. Across all three studies there was a range of student type (school or university level, high school age or mature learner), student ability (lower intermediate to advanced), course type (one that focused specifically on developing writing and dictionary skills, and ones that did not), test task type (letter writing presented in different ways and a range of discursive essays), dictionary type (large comprehensive dictionary, smaller compact dictionaries), and level of prior experience with the dictionary (modest to extensive). There was, however, no evidence to suggest, across any of the studies, that the dictionary made any difference to test takers’ performance, either positive or negative, as measured by test scores. If we were looking for a straightforward answer to the question ‘does the dictionary really make a difference?’ we could conclude on the basis of this evidence that it does not. For those who believe that the dictionary is extraneous to the test construct, this finding of no significant differences in performance across the two test conditions might be reassuring because it could be interpreted as meaning that the dictionary was not a source of construct irrelevant variance – the use of the dictionary was not masking the measurement of the test takers’ abilities as reflected in the scores. This is not, however, the end of the story. No difference in scores does raise questions for those who see the dictionary as a supportive part of the construct being tested, and who therefore believe that dictionary use should improve performance. It is certainly curious that no differences in scores which can be attributable to dictionary use are evident anywhere in the data, despite the very



Chapter 3.  Does the dictionary really make a difference?

real differences between the three studies. Are we to conclude that prior training with using a dictionary, or prior experience with a dictionary, really make no difference? Are we to assume that no students, at any level of ability, can exploit the dictionary to real advantage and find it to be a supportive tool? Or that task type is irrelevant? Test score evidence alone is insufficient to make a judgment. We need to consider other sources of evidence. How did the students use the dictionary? How successful was this use? The next two chapters consider these questions.

63

chapter 4

How do test takers use dictionaries?

In Chapter 3 I argued that test score evidence is crucial in determining the construct validity of a particular test. If we are concerned about issues of fairness, and especially the difference that any modification to a test procedure might make, comparing test scores across tests delivered in different testing conditions will help us to look at differences. If there is no statistically significant difference in scores between two different testing conditions, we can, as Rivera and Stansfield (1998) suggest, draw a conclusion that the different conditions do not compromise the ability of the test to measure test taker performance fairly. With regard to the three studies that are the focus of this book, the test score evidence presented in Chapter 3 (that is, the lack of significant difference between ‘with dictionary’ and ‘without dictionary’ scores) appeared to suggest that having or using a bilingual dictionary simply did not make any difference at all to writing performance. This seemed to be so regardless of all other factors – test taker type or level of ability, task type, dictionary type, level of prior experience with the dictionary, or extent of prior training in writing and dictionary skills. In terms of the relative construct validity of tests in the different test conditions this might be seen as a reassuring finding. If we believe that dictionary use is to be encouraged, we could, on this basis, argue that students should be allowed dictionaries in writing tests. We could argue that there is no need to worry that having a dictionary will make the task invalidly easy. Perhaps just as importantly, we would also not need to worry too much about tests that did not allow dictionaries – because it would seem on the test score evidence that not having a dictionary will not make the task invalidly difficult. It is, however, a little concerning if it seems (from the test scores) that no other factors are making a difference. If, for example, advanced students with a good deal of prior experience with using a dictionary cannot perform any better in ‘with dictionary’ tests than other tests, when compared with intermediate level students with minimal dictionary experience, it would seem that nobody benefits, in terms of improved or more sophisticated writing. For those who see the dictionary as a supportive tool, this finding is at best disappointing and at worst alarming. It would therefore be advantageous to dig a little deeper – to move beyond test score evidence to other evidence available from the studies. In this chapter

66 Dictionary Use in Foreign Language Writing Exams

I focus on an additional strand of evidence – how the test takers across all three studies used the dictionaries, and whether this dictionary use led to any improvement in the writing quality of these test takers, even if this was not showing up in test scores. This chapter presents findings related to two additional sources of evidence – the Lexical Frequency Profiles (LFPs) of test takers’ writing samples and an analysis of how dictionary entries were actually used to enhance writing quality.

The Lexical Frequency Profiles Previous research into the uses to which L2 learners put dictionaries has suggested, perhaps not surprisingly, that learners want to access the meanings of individual items of lexis more than any other information. According to two small-scale studies involving translation between languages and undertaken by Atkins and Varantola (1998b), the primary use to which students put their dictionaries was to locate the meaning of words, firstly from L1 to L2, and secondly from L2 to L1. This finding is supported by survey research into bilingual dictionary use which concluded that accessing meanings came well ahead of any other uses in both French and German as an L2 (Bishop, 1998; Hartmann, 1983). Given that one of the essential purposes of a bilingual dictionary is to give L2 students access to vocabulary they might otherwise not know or use, or, as Oxford (1990) puts it, to make up for “an inadequate repertoire of vocabulary” (p. 47), I considered that it would be useful to find a way to objectify whether students could use a dictionary in a way that broadened their choice of lexis. This is a particularly important consideration in assessing the quality of L2 students’ writing. Grobe (1981) suggests that a judgment on the quality of writing is “closely associated with vocabulary diversity” (p. 85), and Nation (2001) asserts that “vocabulary plays a significant role in the assessment of the quality of written work” (p. 178). A well-written composition will, suggest Laufer and Nation (1995), make effective use of vocabulary. I decided to take a look at lexical range by calculating these test takers’ LFPs across the different writing samples. This is because LFPs are very useful means of quantifying the extent to which writers are using a varied and large vocabulary when writing with or without a dictionary (East, 2004, 2006b). Calculating the LFPs, and comparing results across essays written in both test conditions, will help to determine whether the availability of the dictionary makes a difference (positive or negative) to test takers’ lexical sophistication, and therefore the quality of the text and the extent to which word choice is effective. The calculation is done by a computer program called Range (Heatley, Nation, & Coxhead, 2002) that compares texts prepared in ASCII format in terms of the range of vocabulary



Chapter 4.  How do test takers use dictionaries?

writers use. The program benchmarks individual items of lexis against pre-determined frequency lists, and generates a report giving the number of words used at each level of the lists. The LFPs provide information about range of lexis in three ways, which Laufer (2005) describes as follows: 1. Tokens: that is, all words in the composition. 2. Types: that is, different words in the composition. 3. Families: that is, the base word, its inflections, and its most common derivations. For these studies into German dictionary use the English benchmark lists originally used by Laufer and Nation were replaced with two German lists which were prepared following Nation’s guidelines (Nation, 2002). The first list contained what might be described as ‘basic’ vocabulary – the kind of vocabulary all students at an intermediate level should be actively familiar with, and the vocabulary prescribed in New Zealand for the ‘School Certificate’ examination (equivalent to the GCSE examination taken in the UK by 15- to 16-year-olds). The second list contained higher level vocabulary of the type specifically prescribed for New Zealand’s ‘Bursary’ examination – what students might reasonably be expected to know at the end of study for that or a similar intermediate-level examination, such as the A level (taken two years post-GCSE). Any words the LFPs showed as being outside these two lists could reasonably be considered to be words students at the intermediate level might not be expected to know – words that perhaps only a dictionary would give them. Despite its claim to quantify vocabulary use in written texts, the LFP is in fact an imprecise measure. It cannot, for example, measure correct lexical use. It only measures lexical range. Errors in word choice are handled by removing items from the calculation. Errors in spelling are corrected before the analysis. Errors in word form are ignored if the word itself is correct. In the case, for example, of an incorrect verb ending the software would recognize the word as belonging to a particular family. The word would then be considered as known to the writer even if its form, in the context, was not. This is a particular problem for German (East, 2006b): the wide use and variation in derived and inflected forms in the German language will increase the likelihood that errors may occur with a particular item of lexis that will not be picked up in the LFP (which regards inflected forms as part of the same family as the base form) – a student may have used a particular word wrongly which the LFP will still record. Take, for example, the English verb ‘sleep’ and its German equivalent schlafen. In English there are only three derived forms: sleeping, sleeps, slept. For schlafen there are at least eleven, not counting subjunctive forms:

67

68 Dictionary Use in Foreign Language Writing Exams



1 2 3 4 5 6

schlafend   7 schlafe   8 schläfst   9 schläft 10 schlaft 11 schlaf

schlief schliefst schliefen schlieft geschlafen

This wider variation in derived forms means that a writer in German is potentially more likely to choose a wrong form of schlafen than if that writer were writing in English and using ‘sleep’. The LFP would not reveal this error in word use. Nevertheless, the LFP is a useful measure to determine lexical range, provided that profile results are interpreted against the limitations. Laufer and Nation (1998) argue that it is also a valid measure. They go on to assert that one way of determining the validity of the LFP as a measure of lexical sophistication is to see whether it can reveal differences between proficiency levels. Since it was evident from my first study that there were two distinct groups of students (intermediate and advanced), whose placement test and writing test scores indicated clear differences, I anticipated that, if the LFP were working well, the profiles would reveal a similar differential picture with regard to lexical sophistication. The data available from the first study enabled me to pilot the LFP so that the effectiveness of the measure could be determined (East, 2004).

Piloting the LFP – the first study As I reported in the previous chapter, the intermediate test takers in the first study achieved average ‘with dictionary’ and ‘without dictionary’ scores of 24.7 (SD = 3.2) and 24.3 (SD = 3.8) respectively. For the advanced test takers, by contrast, the average ‘with dictionary’ and ‘without dictionary’ scores were 32.3 (SD = 2.9) and 31.7 (SD = 4). In both cases the advanced scores were some 7 marks higher, out of a total of 35 (a positive difference of 20%). Also, the scoring rubric (reproduced in Chapter 3) enabled a specific consideration of the test takers’ range of lexis, which included its sophistication – the measure the LFP was designed to record. Scores awarded in the individual facet of lexical range were around 1½ marks higher out of 7 (again a 20% difference). These differences were not necessarily unanticipated. As Laufer and Nation (1998) argue, it is only to be expected that more advanced writers will use richer lexis. Both the overall scores and the scores for lexical range indicated a clearly higher level of performance, both in general and, specifically, in terms of lexical



Chapter 4.  How do test takers use dictionaries?

Table 4.1  Lexical Frequency Profiles from the first study. Level of Profile ability Int. Tokens M (n = 3) SD Families M SD Adv. Tokens M (n = 3) SD Families M SD

First list

Second list

Not in the lists

With Without With Without With Without dictionary dictionary dictionary dictionary dictionary dictionary 92.3   3.2 86.7   3.8 83.2   2.6 72   1.7

93.7   0.6 90.3   1.2 79.6  4 66.3   2.5

3.3 0.6 4 1.7 7.1 1 12 1.7

  2.7   0.6   3.7   1.5   8.8   2.6 13.3   2.1

5 2.6 9.7 4.9 9.7 1.2 16 1.2

 3  1  6   1.7 11.7   2.1 20.6   2.5

quality. The question then became, what would a calculation of the LFPs reveal with regard to differential lexical sophistication? I surmised that if the LFPs were working satisfactorily they should reveal clear differences. The results of the calculations, for tokens and families, are given in Table 4.1. It was found that, regardless of test condition, the intermediate participants drew on considerably more basic lexis than the advanced participants. They used noticeably more tokens and families from the first list (‘basic’ vocabulary), and noticeably fewer tokens and families from outside the lists (words not known), than their advanced counterparts. That is, the intermediate test takers used about 90 words in every 100 from the basic list, whereas the advanced participants used around 80 – a reasonably small difference, but a conspicuous one nonetheless. The advanced test takers therefore appeared to use a broader and more sophisticated range of vocabulary overall than the intermediate test takers. The basic question with regard to the validity of the LFP as a measure of lexical richness – differentiation between ability levels – had been sufficiently answered (East, 2004). It was therefore considered viable to use the LFPs more comprehensively with the data available from the two subsequent studies, and to carry out more thorough analyses to determine the difference, if any, that the bilingual dictionary was making.

Using the LFP – the second study In the first of these analyses LFPs were calculated for tokens, types, and families across all three tasks completed by the five participants in the second study. Results of these calculations, for each individual task, are given in Table 4.2.

69

70 Dictionary Use in Foreign Language Writing Exams

Table 4.2  Average LFP results across all three tasks – second study. Task Profile

First list

Second list

Not in the lists

With Without With Without With Without dictionary dictionary dictionary dictionary dictionary dictionary A

B

C

Tokens

M SD Types M SD Families M SD Tokens M SD Types M SD Families M SD Tokens M SD Types M SD Families M SD

92.6

93.2

  1.7

  3.6

88.4

89

  3.3

  4.2

86.2

86.6

  4.1

  4.6

88.2

89.4

  2.2

  3.1

84.8

85.2

  2.2

  5.6

82.4

82.8

  2.3

  6.5

  3.2   0.8   5.4   1.5   6.2   1.9   6.4   1.1   8.2   1.3   9.4   1.3

82.8

89.6

10.6

  4.5

  4.5

  3.6

79.4

84.6

10.4

  6.5

  4.9

  2.6

 4   3.0   6.2   3.7   7.2   4.4   5.8   2.6  8   4.7  9   5.4   6.6   3.6   9.2   2.8

76.8

83

12

10.4

  7.3

  5.5

  3.0

  3.0

 4   1.6   6.6   2.3   7.4   3.0  5   1.6  7   1.9  8   1.6   6.6   4.5

10.2   6.3 11.6   7.4

3 1.2 5.4 1.7 6.2 2.4 4.8 1.1 7 1.6 8.2 1.8 4.2 1.8 6.4 2.9 6.8 3.4

The profiles suggest that, considered on a task-by-task basis, the writing (whether with or without the dictionary) appeared to become more sophisticated as participants progressed, week by week, through each task. Also, for Task C, there was a greater apparent difference between ‘with dictionary’ and ‘without dictionary’ profiles than was evident in Tasks A and B, with participants moving somewhat away from basic lexis and correspondingly experimenting a little more with more sophisticated lexis (whether from the second list or outside the lists) when the dictionary was available (‘basic’ tokens – 90% (Tasks A and B) → 83% (Task C), ‘unknown’ tokens – 4% → 7%). This was suggesting that test takers required somewhat more sophisticated lexis in the discursive essay task than in the letter-writing tasks – not necessarily surprising given the nature of the tasks set – and that the dictionary was able to help them out. It was also important to find out whether the dictionary look-ups themselves were making this (albeit very slight) improvement, and whether the differences were statistically significant. Since the participants had been asked to highlight all instances of dictionary use, it was possible to identify the look-ups. Although a limitation to this approach is that such self-reporting can be unreliable (test ­takers



Chapter 4.  How do test takers use dictionaries?

Table 4.3  Average LFP results (including look-ups removed) – second study. Profile Tokens Types Families

M SD M SD M SD

First list + +*

–

Second list + +*

–

Not in the lists + +* –

87.9   5.0 84.2   5.6 81.8   6.1

90.7   3.9 86.3   5.0 84.1   5.5

6.7 3.8 8 2.8 9.2 3.2

5.5 3.1 7.8 3.8 8.9 4.3

5.2 2.9 7.9 4.1 9 4.8

89.8   4.8 86.7   5.6 84.6   6.2

6.2 4.0 7.1 2.7 8.1 2.9

4.3 2.7 6.2 4.1 7.3 4.6

4 1.5 6.3 2.1 7.1 2.6

Note. +: with dictionary (look-ups included); +*: with dictionary (look-ups removed); –: without dictionary.

might under-report instances of dictionary use for a variety of reasons such as forgetfulness or not wishing to acknowledge that they used a dictionary), it was really the most feasible method of gathering specific data on actual dictionary use, and the evidence was considered the best that was available. To investigate whether the look-ups were making the difference to the quality of lexis all words accessed from the dictionary were removed and the adapted texts were re-analyzed. Table 4.3 shows the results of this analysis, considered overall across all three tasks, and comparing ‘with dictionary’ profiles (with look-ups included and then removed) and ‘without dictionary’ profiles. This analysis reveals several interesting trends: •

•

•

Considered overall, there appeared to be a slight improvement in participants’ lexical sophistication in the ‘with dictionary’ condition with regard to varying the tokens, types, and families. This was judged by an average decrease in, for example, tokens taken from the first list (91% → 88%), and an average increase in tokens from the second list (5.5% → 7%), and outside the lists (4% → 5%). In real terms, however, this represented small shifts in lexis of around 3 words in every 100. The mean figures between ‘with’ and ‘without’ texts when look-ups were included differed from each other by as much as 2.9%, with an average variation of ±1.5%. The mean figures between ‘with’ and ‘without’ texts when look-ups were taken out differed from each other by, on average, ±0.5%.

When dictionary look-ups were included in the calculation, there appeared to be greater lexical sophistication in the writing. When they were taken out, there was, on average, negligible differentiation in the profiles – the test takers wrote with comparable levels of lexical sophistication if words located in the dictionary were

71

72

Dictionary Use in Foreign Language Writing Exams

not factored into the analysis. There was some evidence to suggest that the ­­lookups themselves were making a positive difference. However, as has already been pointed out, the difference between the profiles was very slight (3 words in every 100), and the sample size (2 × 15 pieces of work) was small, and it is hard to go beyond mere speculation that this was the case. The results of several paired samples t-tests (see Appendix 4) indicated no significant differences between the profiles at any level. Taking the limitation of sample size into account, these results suggest that any improvement in writing with the dictionary available was just as likely due to chance or random factors than to the intervention of the dictionary. The findings from the second study were intriguing, nonetheless. The observed differences may well have been down to chance, but slight improvement in lexis in the ‘with dictionary’ texts was observed which disappeared when the dictionary look-ups were removed. There was enough inconclusive evidence in these data to justify further investigation of the LFP with the larger set of test takers available from the third study.

Using the LFP – the third study In the third study LFPs were calculated on a subset of 39 test takers. This subset was extracted from the data available from all 47 participants according to two criteria: 1. Essays across both test conditions contained more than 100 words. 2. Students had indicated their use of the dictionary through underlining. In cases where any participants did not meet either or both these criteria, both their essays were removed from the calculation. A similar procedure was followed with this data set as with that from the second study. ‘Without dictionary’ profiles were compared with ‘with dictionary’ ones, first with look-ups included, and then with look-ups removed. Results of the profiles – ‘with dictionary’ look-ups included, ‘with dictionary’ look-ups removed and ‘without dictionary’ – are presented in Table 4.4. The profiles indicate the following trends: •

When look-ups were included there was an improvement in lexical sophistication in the ‘with dictionary’ condition, compared to ‘without dictionary’ texts. This was indicated by less reliance on words from the first list, and greater use of words from the second and outside the lists. This trend was noticeable across all levels of word type, token, and family. It was most ­noticeable with



Chapter 4.  How do test takers use dictionaries?

Table 4.4  Impact of look-ups on the LFPs – third study. Profile Tokens Types Families

First List M SD M SD M SD

Second List

Not in the lists

+

+*

–

+

+*

–

+

+*

–

89.2   4.5 82.5   5.1 80.1   5.7

92.1   3.9 87   4.8 85   5.5

91.9   4.4 86.9   5.2 85.1   5.7

  6.5   2.8

  5.21   2.5   8.6   3.4   9.9   3.9

  5.4   2.5  9   3.5

4.4 2.3 7.2 3.2 8.3 3.7

2.7 2.0 4.5 2.9 5.1 3.3

2.7 2.2 4.2 2.9 4.7 3.2

10.3   3.2 11.6   3.5

10.2   3.8

Note. +: with dictionary (look-ups included); +*: with dictionary (look-ups removed); –: without dictionary Taken from East (2006b, p. 189).

•

word families, where 5% fewer word families were drawn on from the first list when writing with the dictionary available, and 3.6% more word families were drawn on from outside the lists. When look-ups were taken out, and comparison was made with the ‘without dictionary’ profiles, there was considerably less variability, in all cases not more than ±0.4%. This indicated a far closer parallel between the profiles.

These differences can be illustrated diagrammatically (Figure 4.1). This bar graph brings out more clearly three major findings from these LFP data: 1. The ‘with dictionary’ texts used more sophisticated lexis than the ‘without dictionary’ texts. 2. When the look-ups were removed the difference between ‘with’ and ‘without dictionary’ texts disappeared. When the two sets of texts were treated as if they had been written in similar circumstances (that is, both as if no dictionary had been available) the samples of writing showed similar and comparable levels of lexical quality. 3. The improvement in lexical sophistication in ‘with dictionary’ texts appeared to be down to the use of words located in the dictionary. When the test takers used a dictionary they increased the richness of their lexis. To determine whether the look-ups were making a statistically significant difference to profiles two further comparisons were made. The first was between ‘with dictionary’ and ‘without dictionary’ profiles. The second was between ‘with dictionary’ profiles with the look-ups removed, and ‘without dictionary’ profiles.

73

74

Dictionary Use in Foreign Language Writing Exams

Figure 4.1  A comparison of profiles.

When the look-ups were removed and the ‘with dictionary’ and ‘without dictionary’ profiles were compared, it was found that the LFPs were not significantly different from each other at any level. When, however, the same comparison was made, this time including the look-ups, the LFPs for words taken from the first list and words outside the lists (whether tokens, types, or families) were significantly different from each other across the conditions at p ≤ .005 (although words taken from the second list were not). That is, the probability of this difference happening by chance alone was equal to or less than 1 in 200 (Appendix 4). When the test takers used a dictionary they significantly increased the richness of their lexis. They used significantly fewer words from the first list, and correspondingly significantly more words from outside the lists. I also decided to take a look at whether the test takers’ level of ability made any difference to the quality of their lexis across the two conditions. In other words, was there any evidence to suggest that one group of test takers could use the dictionary more effectively than another to enhance the quality of their writing? The profiles were placed into two groups: lower ability (n = 20) – that is, the pre-intermediate and intermediate participants; and upper ability (n = 19) – that is, the upper-intermediate and advanced participants. Results are presented in Table 4.5. It was apparent that, when writing with the dictionary, the participants did not differ greatly in terms of the sophistication of their lexis, regardless of ability.



Chapter 4.  How do test takers use dictionaries?

Table 4.5  Impact of look-ups according to ability – third study. Ability level

Profile

Lower

Tokens Types Families

Upper

Tokens Types Families

First list M SD M SD M SD M SD M SD M SD

Second list

Not in the lists

+

–

+

–

+

–

89.7

93.7   2.5 89.0   4.0 87.5   4.3 90.1   5.2 84.6   5.6 82.6   6.0

  6.1   2.7

  4.4   1.7   7.8   3.0   8.8   3.3   6.4   2.8

4.2 2.2 7.4 5.0 8.4 3.4 4.5 2.5 7.1 3.5 8.1 4.1

1.9 1.6 3.2 2.7 3.7 3.0 3.5 2.5 5.1 2.8 5.8 3.0

  4.0

82.5   3.4 80.1   5.0 88.6   5.0 82.5   5.8 80.1   6.5

10.1   3.0 11.5   3.8   6.9   2.9 10.4   3.0 11.8   3.2

10.3   3.6 11.6   3.8

Note. +: with dictionary; –: without dictionary Adapted from East (2006b, p. 187).

The profiles were more or less identical. When writing without the dictionary two trends emerged: Firstly, it appeared that both ability groups wrote with marginally less lexical sophistication; secondly, it seemed that the upper ability participants were able to write with more sophisticated lexis in comparison with the lower ability participants. Again, these trends can be illustrated diagrammatically (Figures 4.2 and 4.3). Once more the question was whether any of the observed differences in the profiles were significant. Independent samples t-tests revealed no significant difference between the two groups across any of the ‘with dictionary’ profiles. However, the ‘without dictionary’ profiles were significantly different from each other at p < .05 across all levels – first list, second list, and outside the lists (see Appendix 4). This difference suggests that when the two groups did not have the dictionary the upper ability participants wrote with significantly more lexical ­sophistication than the lower ability participants. However, ‘with dictionary’ performances were not significantly differentiated between the two ability groups: the lower ability participants were able to increase their range of more sophisticated lexis in the ‘with dictionary’ tasks proportionately more than the upper ability participants so that they wrote more comparably, and no significant differences were found between the groups. The dictionary appeared to ‘level the playing field’ in terms of ability to access and use lexis at a variety of levels. Both groups wrote with more lexical ­sophistication with a dictionary. The previous analysis of profiles had indicated that this increase in sophistication was due to dictionary look-ups.

75

Dictionary Use in Foreign Language Writing Exams

100%

upper ability

lower ability

upper ability

lower ability

40%

lower ability

60%

upper ability

80%

20%

0%

types

tokens

words from 1st list

words from 2nd list

families

words outside the lists

Figure 4.2  A comparison of ‘with dictionary’ profiles according to ability.

100%

upper ability

lower ability

upper ability

40%

lower ability

60%

upper ability

80%

lower ability

76

20%

0%

tokens

words from 1st list

types

words from 2nd list

families

words outside the lists

Figure 4.3  A comparison of ‘without dictionary’ profiles according to ability.



Chapter 4.  How do test takers use dictionaries?

Implications for test scores In the light of this improvement in lexical quality I found it curious that the test scores did not reveal any improvement in performance in ‘with dictionary’ tests. This raised a fascinating question for me. If the dictionary did make a positive difference to test takers (especially, it would seem, lower ability – pre-intermediate to intermediate level – test takers) in terms of an increased range of more sophisticated lexis, why was this difference not showing up in scores? There are several possibilities. This interesting anomaly is similar to that described by Wigglesworth (1997), who explored the effects of planning time on the discourse produced by test candidates in an oral examination, comparing results across two conditions (‘with planning time’ and ‘without planning time’). Her findings suggest that where planning time was provided complexity and accu­racy (at least on some measures) may improve for high-proficiency candidates on some tasks. The test scores revealed, however, no significant differences across the groups as a result of the planning time variable. Wigglesworth argues that it is curious why raters do not perceive differences manifest at the discourse level when assigning test scores. She concludes: It may be because raters assess the discourse at a macrolevel in terms of communicative effectiveness while the differences are manifest at the microlevel. … It may be that there is a point at which there needs to be a larger increase in sophisti­cation of language use by the candidate in order to obtain a better mark from the raters. (p. 103)

If we were to apply this argument to writing we might suggest that raters, not unsurprisingly, will assess a piece of written work and the sophistication of its lexis more globally, and are less likely to notice nuances and subtleties of levels of lexical sophistication (which is where the LFP comes in as a useful measure). By this argument the raters in these studies would have assessed the discourse at a macrolevel in terms of communicative effectiveness and range of lexis, whereas differences in lexical sophistication are only manifest at the microlevel available through the LFP. A more dramatic gain in lexical sophisti­cation would probably be required before this would lead to better scores from raters. (The average increase in lexical sophistication was, as has already been stated, quite small and would presumably have made little impact on lexical quality considered more globally.) There is another possibility, which relates to the particular weakness of the LFP (East, 2006b). That is, the LFP is a limited calculation. It focuses on lexical range, but not lexical accuracy. It may be a very good tool for measuring word

77

78

Dictionary Use in Foreign Language Writing Exams

choice, but the LFP data tell us very little, if anything, about the accurate use of lexis, and accuracy of lexis may have had an influence (consciously or unconsciously) when it came to the raters assigning scores. Even in cases where there was a marginal increase in lexical sophistication instances of lexical error may have influenced the raters, leading to no improvement in scores. (The issue of lexical accuracy is the focus of Chapter 5.) Should the limitations therefore lead us to discount the LFP data? Not really. If we are interested in lexical range and lexical quality (the two dimensions of writing which are most likely to be affected by dictionary use), the LFPs allow us to dig a little deeper than test scores. Provided that the limitations of the measure are taken into account, it is possible to draw two important conclusions. On one measure (the LFPs) the evidence suggests that, on the whole, participants did improve their lexical sophistication when using a dictionary, albeit marginally, and this difference (in the third study) was statistically significant. On another measure (test scores) it was found that using a dictionary did not make any significant difference. For the reasons just outlined neither of these findings is incompatible with the other. The difference in the findings between the LFPs and test scores is intriguing nonetheless. It indicated the need to continue digging even deeper into the data. It was also important to take a closer look at the types of words the test takers actually looked up in the dictionary, and to scrutinize their levels of success with using them appropriately. This would enable a clearer picture to emerge not only of the extent of lexical sophistication achieved with the dictionary but also of the extent of dictionary misuse. In the remainder of this chapter I explore the first of these aspects – the extent of positive dictionary use.

How did the test takers use the dictionary? I have already indicated that previous research has established that L2 learners want to access meanings from their dictionaries more than any other information. Hartmann (1983) investigated the uses to which teachers (n = 67) and students (n = 118) put bilingual English/German dictionaries. Just under two thirds of the students were studying German in school. Bishop’s (1998) survey into bilingual English/French dictionary use spanned both ­ secondary and tertiary level, and involved two cohorts of participants – school students in the first year of studying for the A level in French (n = 25), and second level French students (n = 25) at the UK’s Open University (OU). The school students were at an earlier stage in learning French than many of the OU students. The two surveys provided evidence that students used their dictionaries first and foremost to locate meanings.



Chapter 4.  How do test takers use dictionaries?

Table 4.6  Rank order of dictionary uses. Hartmann (1983) (German) meaning grammar use in context spelling synonyms pronunciation etymology other

Bishop (1998) (French) A level students

OU students

meaning gender spelling feminine forms verb forms synonyms plurals pronunciation register

meaning spelling gender verb forms feminine forms plurals register synonyms pronunciation

Adapted from East (2006c, p. 3).

Table 4.6 illustrates, in rank order from most to least, how dictionary users most often used their dictionary. Hartmann (1983) discovered that 97% of the German dictionary users frequently accessed meaning. Grammar, the next most frequent use, was 15% behind this, at 82%. Bishop (1998) observed that 68% of the A level students and 80% of the OU students most often used the French dictionaries to find meanings. The next most frequent use for the A level students was to check gender (52%), and for the OU students to check spelling (64%). Thus, in both cases, checking meanings ranked as 16% above the next most popular option. I utilized these rank-ordered lists to draw up a series of ‘dictionary use’ statements that were then placed on the longer questionnaire that the participants in my studies would complete. Participants in the first study were asked to indicate from the list of statements the uses to which they put the dictionary (no ranking was required). There was also a space for participants to list any other uses to which they put the dictionary. Their responses were used to confirm the statements that were included in the questionnaires for the two subsequent studies. One additional use was added based on an additional comment received from the first study. In this way a ‘dictionary use checklist’ was presented to the test takers in the second and third studies, on which participants were asked to rank their uses of the dictionary (Figure 4.4). Although the rankings in the second and third studies differed somewhat, a comparison of responses from all three studies helps to paint a picture of the uses to which these test takers put the dictionary, and their importance for the test takers relative to each other. This overall ranking, in three bands, is given in Figure 4.5.

79

80 Dictionary Use in Foreign Language Writing Exams

For each statement please tick the box that most accurately reflects the ways in which you USED THE DICTIONARY for this task. It is important to tick ONE box per statement. When I used the dictionary, I used it . . . to find out the meanings of individual GERMAN words I DEFINITELY did not know to check the meanings of individual GERMAN words I WASN’T SURE about to find out the meanings of GERMAN PHRASES/EXPRESSIONS I did not know to find out or check the genders of individual GERMAN NOUNS to find out or check the plural endings of individual GERMAN NOUNS to find out or check the spelling of individual GERMAN WORDS to find out or check if a GERMAN VERB was irregular to find out or check the correct preposition to use with a GERMAN VERB to find the German equivalent of individual ENGLISH words to find the German equivalent of ENGLISH PHRASES/EXPRESSIONS to find GERMAN PHRASES that I could use in my essay

every most some- rarely never time times times

☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐

Figure 4.4  Dictionary use checklist.

This composite picture supports the findings of previous research and indicates that these test takers most wanted to use the dictionary to access the meanings of individual lexical items, first from L1 to L2, and then from L2 to L1. Using the dictionary to find phrases tended to be of less importance, although checking spelling and gender also featured quite highly. The principal use to which the dictionary was clearly put by all, regardless of the mode of writing, was to find the German equivalent of English words. This raised two questions for which I hoped the comparative shorter questionnaires would provide answers: what did the test takers do when they did not have the dictionary with them, and was using the dictionary the most preferred strategy when it was available? As I argue in Chapter 2, language learning strategies are used because, in the words of Oxford (1990), there is “a problem to solve, a task to accomplish, an objective to meet, or a goal to obtain” (p. 11). Strategy use is also an important dimension of the theoretical framework of communicative competence outlined by Canale and Swain (1980) and Canale (1983). Within their outline strategic competence is called into play “to compensate for breakdowns in communication due to performance variables or to insufficient competence” (p. 30). When the dictionary is not available test takers need to draw on other communicative strategies. The shorter questionnaires presented participants with several statements adapted from Part C of Oxford’s (1990) ‘Strategy Inventory for Language ­Learning’



Chapter 4.  How do test takers use dictionaries?

Information sought

Strategy Primary uses

Meaning, L1 → L2 Meaning, L2 → L1 Meaning, L2 → L1

To find the German equivalent of individual English words To find out the meanings of individual German words I definitely did not know To check the meanings of individual German words I wasn’t sure about Secondary uses

Spelling Gender Meaning, L1 → L2

To find out or check the spelling of individual German words To find out or check the genders of individual German nouns To find the German equivalent of English phrases or expressions Other uses

Meaning, L2 → L1 Plurals Meaning Grammar Grammar

To find out the meanings of German phrases or expressions I did not know To find out or check the plural endings of individual German nouns To find German phrases that I could use in my essay To find out or check the correct preposition to use with a German verb To find out or check if a German verb was irregular

Figure 4.5  Overall ranking of dictionary use strategies.

(SILL) which focuses on ‘compensating for missing knowledge’. In the second study the most popular strategies were as follows: 1. Preferred strategy in the ‘with dictionary’ tests: • In the input: • I guessed the general meaning by using clues. • In the response: • I looked up the word or expression in the dictionary. 2. Preferred strategy in the ‘without dictionary’ tests: • In the input: • I guessed the general meaning by using clues. • In the response: • I kept to the same idea, but found another way to say it.

81

82

Dictionary Use in Foreign Language Writing Exams

In the third study, the larger number of participants made it possible to provide a rank order of a number of strategies. 1. Rank order of strategies used in the ‘with dictionary’ test: • In the input: 1. I looked up the word I did not know in the dictionary. 2. I guessed the general meaning by using clues. 3. I chose to ignore the word. • In the response: 1. I looked up the word or expression in the dictionary. 2. I kept to the same idea, but found another way to say it. 3. I made a guess based on the words I know. 4. I dropped the idea. 5. I invented a word. 2. Rank order of strategies used in the ‘without dictionary’ test: • In the input: 1. I chose to ignore the word. 2. I guessed the general meaning by using clues. • In the response: 1. I kept to the same idea, but found another way to say it. 2. I made a guess based on the words I know. 3. I dropped the idea. 4. I invented a word. The evidence across all three studies suggests that when it came to accessing unfamiliar words from the test tasks guessing was often used regardless of whether the dictionary was available or not. When writing a response to the tasks set, however, using the dictionary, when available, was preferred over any other strategies to make up for gaps in knowledge.

The look-ups themselves Given that accessing the meanings of individual items of lexis was the predominant use to which the participants claimed that they put the dictionary, the types of words the test takers looked up were analyzed closely. Data on this were drawn from the highlighted or underlined instances of dictionary use. Figure 4.6 records the available information on the categories of item participants in each of the studies looked up, together with the relative popularity of these different categories.



Chapter 4.  How do test takers use dictionaries?

70%

Percentage of look-ups

60% 50% 40% 30% 20% 10% 0%

Nouns

Verbs

Adjectives

Adverbs

Phrases

Other

First study

60%

13.3%

17.8%

2.2%

0

6.7%

Second study

50%

19.8%

17%

5.7%

3.8%

3.8%

46.1%

21.9%

17.4%

7.1%

3.5%

4%

Third Study

Type of word

Figure 4.6  Types of word looked up. Adapted from East (2006c, p. 9)

These data reveal a level of similarity across the three studies in the types of items test takers wanted to locate from the dictionary. It was evident that the self-reporting of dictionary uses confirmed the questionnaire responses: individual items of lexis constituted the vast majority of look-ups, with the location of phrases accounting for less than 4% of look-up activity. In total the look-up of nouns made up around half the number of look-ups, with verbs (at one in five of all look-ups) coming second.

The sophistication of the look-ups The exercise in lexical frequency profiling had revealed that, at least in the third study, the dictionary look-ups were making a positive difference to the lexical sophistication of the students’ writing. This was suggesting that the types of look-up the test takers were making were of items they might not reasonably have known, items they were subsequently able to use to improve the quality of their lexis. An important question to ask was therefore: how many of the items looked up by the test takers could be placed into this category of ‘unknown’ words? In other words, to what extent was the dictionary helping the students to find new or more sophisticated ways of expressing themselves, and to what extent was the dictionary being used to check up on words the test takers might reasonably be expected to know? These are important questions because they help to identify

83

84

Dictionary Use in Foreign Language Writing Exams

the extent to which the bilingual dictionary was a useful tool to facilitate more sophisticated language use, and the extent to which it was being used when perhaps it did not need to be used – that is, as a check for items which the test takers really ought to have been actively familiar with. This is not to suggest that checking items which should be known is not a legitimate use to which a dictionary can be put – although those that might argue against allowing dictionaries in writing tests might suggest on this basis that using a dictionary to do this leads to laziness on the part of the test takers (not taking the time or effort to learn vocabulary) and subsequent reliance on the dictionary at the expense of other strategies. It is to suggest that if a dictionary is being used to extend knowledge and to help test takers to become more adventurous in their choice of words, it is fulfilling a useful function and is being used as a valid tool – which would support the arguments of those who are in favor of dictionary use in tests. I considered that calculating an LFP of the look-ups as individual items of lexis would help to differentiate between different levels of look-up. One of the limitations of this type of comparison is that it aims to draw a distinction between words that the test takers might realistically be expected to know (words in the lists) and words they might not know (words not in the lists). In reality, language acquisition and knowledge of lexis are somewhat fluid, and test takers cannot necessarily be expected to be actively familiar with every word on a given list (which is why using a dictionary to check an item may be seen as a legitimate activity). Despite this limitation, however, an analysis of the words looked up provides some picture of the range of items. When I analyzed the individual items of lexis using Range I found that around half the look-ups were of items not found in either of the two benchmark lists. In other words, half the look-ups were of words that (as judged against the benchmark lists) the test takers could not reasonably be expected to know at the intermediate level. A further one in four items could be located in the second list (more sophisticated lexis). These figures indicate that only around a quarter of the look-ups were of ‘basic’ lexis (items from the first list). Based on this evidence, and taken across the data available from all three studies, the following conclusion can be drawn: three out of four occasions of dictionary use were to locate ‘higher order’ items of vocabulary, and half of all look-ups were of items the test takers were not expected to know. Just as the LFPs of the entire texts had indicated, the test takers were using the dictionaries to increase their range of lexis. Examples of the range of vocabulary accessed successfully from dictionaries at each of the three levels measured by the LFP are given on the next page.



Chapter 4.  How do test takers use dictionaries?

First List Type of word

Example

Meaning

Noun Noun Noun Noun Noun Verb Verb Verb Adjective Adjective Adjective Adverb Adverb Adverb

Beruf Fehler Grund Landschaft Mannschaft bauen bezahlen reisen einfach höflich nützlich besonders vielleicht wirklich

profession mistake reason countryside team to build to pay to travel simple polite useful especially perhaps really

Second List Type of word

Example

Meaning

Noun Noun Noun Noun Noun Verb Verb Verb Adjective Adjective Adjective Adverb Adverb Adverb

Einfluss Erinnerung Erlebnis Kenntnis Verständnis beschliessen entwickeln schätzen einheimisch neugierig zweisprachig eindrucksvoll sorgfältig überraschend

influence memory / reminder / memento experience knowledge understanding to decide to develop to treasure / value indigenous curious bilingual impressive carefully surprising

Not in the lists Type of word

Example

Meaning

Noun Noun Noun Noun

Aufmerksamkeit Bereitschaft Entwicklung Tätigkeit

attentiveness readiness development activity / occupation

85

86 Dictionary Use in Foreign Language Writing Exams

Noun Verb Verb Verb Adjective Adjective Adjective Adverb Adverb Adverb

Zunahme beeindrucken befreien erfordern einmalig geistig herausfordernd offensichtlich unerlässlich unflätig

increase to impress to set free to require unique spiritual provocative / inviting obvious / blatant essential / imperative offensive

These lists provide evidence that the test takers were often able to use the dictionary successfully to locate a wide assortment of words, at different levels of sophistication, to meet their needs in a variety of contexts.

Phrasal use The location of phrases (or what Wray (2002) calls formulaic sequences) constituted only a very minor part of the look-ups. It was evident, however, that several test takers were able to locate from the dictionary, and then use successfully, fixed phrases and expressions that enhanced the quality of their response. Examples located in the Collins German Dictionary included: German phrase

Meaning

Ohne Rücksicht auf Ich möchte Sie bitten ... ... was Sie beabichtigen zu tun

Without consideration for I would like to ask you ... ... what you intend to do

Other phrases taken from different dictionaries included the following: German phrase

Meaning

Bis zu einem gewissen Grade … gegenüber im Vorteil sein [Wir müssen auch] dafür sorgen, dass Auf der anderen Seite … Hinzu kommt noch …

To a certain extent To have the advantage over … [We must also] be concerned that ... On the other hand … There is also the fact that …

On the other hand, a qualitative analysis of texts from the second study revealed the successful use of formulaic sequences that had been committed to memory rather than items that had been looked up in the dictionary, at least according to



Chapter 4.  How do test takers use dictionaries?

the test takers’ self-reporting (East, 2005b, 2006c). Across all three tasks, these sequences included: German phrase

Meaning

Ich bin der Meinung, dass (7 times) Ich glaube, dass (4 times) Es tut mir sehr Leid, dass (3 times) Ich bin sicher, dass (twice) Ich schlage vor, dass Man kann mit Sicherheit sagen, dass Es kommt darauf an, ob Es wird viel gesagt, dass Ich habe verstanden, dass Für mich hat es die Bedeutung, dass Zuerst muss ich mich fragen, was

I am of the opinion that I believe that I am really sorry that I am certain that I suggest that It can be said with certainty that It depends whether It is often said that I understood that For me it has the significance that First of all I must ask what

In the discursive essays (Task C), several sequences were drawn on (again not located in the dictionary), although their use indicated some instances of error: German phrase

Meaning

Wenn man das Thema Mütter und ihre When you consider the theme of kleine Kinder betrachtet ... mothers and their small children ... Wenn man das Thema, Drogen betrachtet, When you consider the theme of kann man sicher sein, dass ... drugs you can be certain that ... Über das Thema „Sollten Mütter arbeiten A lot has been written, and perhaps oder [sich um] ihre Kinder kümmern“ even more spoken, about the theme wird sehr viel geschreiben und vielleicht “should mothers work or look after eben mehr gesprochen their children?” This sometimes relatively sophisticated use of sequences without recourse to the dictionary hints at the extent to which test takers were able to enhance the quality of their writing without having to look things up, and the extent to which thorough preparation was likely to pay dividends (an issue which I explore more fully later).

Looking at lexis in context Bearing in mind the extensive use of dictionary look-ups, I also considered it important to look at how individual items of lexis were used in specific contexts

87

88

Dictionary Use in Foreign Language Writing Exams

to add to the quality of the writing. A qualitative analysis of the texts of a number of Collins Pocket users in the third study revealed ways in which quality was improved by correct application of dictionary look-ups. This is illustrated in the following examples. One upper intermediate participant used the expression ‘benefits from’ effectively and correctly when she wrote:

Obwohl einige Schüler jetzt keine Interesse für die Fremdsprachen haben, werden sie in Zukunft aus Fremdsprachen Nutzen ziehen.



Although a few students now have no interest for foreign languages, they will benefit in the future from foreign languages.

One advanced participant indicated that she only used the dictionary twice, but on both occasions the entry was used successfully and correctly when she wrote:

Das ist kaum überraschend That is hardly surprising

and

Für diese Leute gibt es auch viel mehr Arbeitsgelegenheiten. For these people there are also more work opportunities.

Two other advanced participants wrote essays that contained some errors of grammar and spelling, but dictionary look-ups were often correctly applied (relevant sections of these essays are presented verbatim and uncorrected below, although for ease of readability paragraph breaks have been removed). In the first example look-ups were used to good effect on 11 out of 13 occasions. German text

Jedes Jahr kommen viele Touristen, um Neuseelands Abenteuer-, Entspannungs- und Kulturmoglichkeiten zu erleben. Aber wie wirken sie aufs Land? Erstens überlegen wir die wirtschaftlichen Wirkungen. Für neuseeländischen Gesellschaften sind also ausländische Gäste sehr wichtig. Zweitens hat der Zustrom der Touristen positive und negative Konsequenze für die Umwelt. Wenn zu viele Touristen in ein Gebiet reisen dürfen, wird es ohne Zweifel beinflusst. Zum Beispiel in der Sudinsel werden viele abgelegene Einlassen wegen der Emissionen der Reiseboote verschmutzt. ... es könnte für die Neuseeländer vorteilhaft sein, so viele andere Kulture kennen zu lernen können. Alles in allem bin ich überzeugt, dass der Massentourismus viele Nutzen für Neuseeland haben kann, vorausgesetzt dass er sorgfältig kontrolliert wird.



Chapter 4.  How do test takers use dictionaries?

Direct translation

Each year many tourists come in order to experience New Zealand’s adventure, relaxation and culture possibilities. But how do they affect the land? First of all let us think about the economic effects. For New Zealand businesses foreign visitors are therefore very important. Secondly the influx of tourists has positive and negative consequences for the environment. When too many tourists are allowed to travel into a region it is influenced without doubt. For example in the South Island many remote inlets are polluted because of the emissions of the travel boats. … it might be advantageous for the New Zealanders to be able to get to know so many other cultures. All in all I am convinced that mass tourism can have many uses for New Zealand, provided that it is carefully controlled.

As for the two occasions of error, the word Einlassen (inlets) was inappropriate in the context, and also (if it had been the correct word) was misspelt – it should have been Einlässe. The contextually appropriate word was Meeresarme. Also Reiseboote (travel boats) was a good attempt to form an appropriate compound, but a more fitting word might have been Touristenschiffe (tourist ships). In the second example another advanced participant used the Collins Pocket successfully four times. With the exception of the basic word Geld (money), I considered that the other words added to the discourse quality of the essay. German text

Touristen bringen nämlich sehr viel Geld in Neuseeland rein. ... Ohne dieses Geld würde Neuseeland sich nicht weiter entwickeln. Es ist schlecht für die Umwelt weil sie mehr verschmutzt wird mit Mühl, Luftverschmutug und Lärmbelästigung. Die Strände überall in Neuseeland werden überbevölkert mit Touristen ...

Direct translation

Indeed tourists bring a lot of money into New Zealand. … Without this money New Zealand would not develop itself further. It is bad for the environment because it becomes more polluted with rubbish, air pollution and noise pollution. The beaches everywhere in New Zealand are becoming over-populated with tourists …

. In the case of überbevölkert, a native German-speaking colleague who looked at the data had expressed a preference for überhäuft. I considered that überbevölkert communicated the concept of ‘overcrowded beaches’ sufficiently accurately to be considered as effective use of the dictionary.

89

90 Dictionary Use in Foreign Language Writing Exams

The adequacy of the dictionaries used One important remaining question in which I was interested related to the extent to which the test takers found the dictionaries useful for what they had to do. That is, they may often have been able to locate lexis appropriate to their contexts, but to what extent was the dictionary they used always helpful with this? Participants were asked about this in the longer questionnaires (Figure 4.7). Did you find the dictionary you used adequate for the task?

☐ yes

☐ no

☐ sometimes

Why do you say this?

Figure 4.7  The ‘adequacy’ question on the longer questionnaire.

The evidence from the questionnaires suggests that the Collins German Dictionary users (those in the first and second studies) were, on the whole, satisfied with the adequacy of this dictionary. In the first study all six participants recorded an overall high level of satisfaction. They commented that they were mostly or always able to find the meaning of a German word, and all but one said that they were always able to find the German equivalent of an English word. In the second study, all participants noted that they found the dictionary adequate when used, either sometimes or always. Those who found it adequate, at least on one occasion, commented on the following: • • • •

A wide range of words and definitions for a wide variety of contexts was available. Important grammatical information, most particularly genders and plural forms, could be found. Occasionally important expressions or phrases could be located. The dictionary was also useful for checking spelling.

Participants in the third study were also asked about ease of dictionary use (Figure 4.8)



Chapter 4.  How do test takers use dictionaries?

How EASY did you find this dictionary to use? It is important to tick ONE box ONLY. I found it very easy I found it quite easy I found it quite difficult I found it very difficult

☐ ☐ ☐ ☐

Figure 4.8  The ‘ease of use’ question on the longer questionnaire.

Figure 4.9 presents the results of the questions focusing on adequacy of the dictionary and ease of use in the third study. Participants’ responses indicate the following: •

•

•

Virtually all the participants using the Collins Pocket (97%) found it at least sometimes adequate, with a similar result for the remaining participants using the other dictionaries (88%). A good number expressed a high level of satisfaction with the dictionary, judged by the number of responses that indicated ‘yes’ to the adequacy question (42% of the Collins Pocket users, and 44% of the other dictionary users). Virtually all the participants found the dictionary they had at least quite easy to use (94% of the Collins Pocket users and 97% of the users of other dictionaries).

Evidence on the adequacy of the various dictionaries used across the three studies appears to reveal, therefore, that no dictionary stood out as being any more or less adequate than any other. Nevertheless, these findings do hint at some potential problems with the dictionaries. For example, a good number of users in the third study recorded that their own expectations of adequacy were only sometimes met – a finding that was suggesting that the dictionaries they had access to had some definite limitations. This was going to require some investigation.

Where to next? Putting together all of the dictionary use evidence presented in this chapter, a number of important strands emerge:

91

92

Dictionary Use in Foreign Language Writing Exams

Figure 4.9  Opinions about the dictionary used – third study. Note. No participants reported that they found the dictionary they had very difficult to use.

•

• • •

• •

•

When the dictionary was available to the test takers, its use was the most preferred strategy when it came to tackling the response to the questions – the writing of their essays. The dictionary was primarily used to locate meanings, firstly from L1 to L2, and then from L2 to L1. Individual items of lexis made up 97% of look-ups, and half of these were nouns. The test takers were able to increase the sophistication of their lexis when using a dictionary when compared to range of lexis in the ‘without dictionary’ essays. In the third study this increase was statistically significant. Around 75% of all look-ups could be classed as ‘more sophisticated’ lexis when benchmarked against the vocabulary lists used by Range. According to evidence from the second study, test takers were often able to use formulaic sequences successfully in their writing without having to use a dictionary. Most participants were satisfied that the dictionary they had with them was at least sometimes adequate for the uses to which they wanted to put it.



Chapter 4.  How do test takers use dictionaries?

All in all, this evidence paints quite a positive picture of the benefits of having a dictionary in a writing examination. There was some indication, however, that the dictionaries were not always adequate for the task, and the two instances of dictionary error I recorded hint at potential problems with using a dictionary. Chapter 5 explores some of the definite limitations to having dictionaries in writing tests which these studies revealed.

93

chapter 5

When the dictionary becomes a liability

In Chapter 3 I presented the evidence available from the three reported studies about the impact of bilingual dictionaries on writing test scores. If we were to look solely at this evidence we could conclude that utilizing a dictionary makes no difference to test taker performance. In Chapter 4 I began to delve a little deeper than test scores and considered the positive uses to which the test takers put the dictionaries. On the basis of the evidence presented there it might be concluded that dictionary use actually makes a positive difference. It certainly seemed as if the test takers were able to draw on a broader and more sophisticated range of lexis when they wrote with the dictionary, and this more sophisticated lexis seemed to be down to their use of the dictionary. Nevertheless, it cannot necessarily be assumed from this positive increase in lexical sophistication that the test takers were always able to exploit their dictionaries to maximum effect. There was also some indication that the test takers were not entirely satisfied with the dictionaries they had and could not always use them successfully. Another important strand of evidence which must be taken into consideration concerns those occasions when the dictionary became a liability. I turn to the liability dimension of dictionary use in this chapter. As pointed out in Chapter 4, Weigle (2002) suggests that two potential liabilities to having a dictionary in a writing examination are the time taken to use a dictionary and some test takers’ inability to use the dictionary effectively. In this chapter I consider first of all the observational evidence from the first two studies. This evidence is used to explore the influence of dictionary use on time. I go on to look at the number of look-ups test takers reported that they made and examine how this potentially impacts on task completion. Finally I explore the extent to which the test takers made errors with dictionary look-ups and look closely at some of these errors.

The number of look-ups test takers made In the first two studies I gathered observational evidence on the test takers’ use of the dictionary. Participants consented to being video-taped so that the evidence could be scrutinized after the completion of the tasks. All the participants took

96 Dictionary Use in Foreign Language Writing Exams

the tests in the same classroom. The video camera was placed at the front of the room approximately 4 meters from the participants, as illustrated in Figure 5.1 (not drawn to scale). In the first study, up to three participants took the tests at any one time. In the second study participants numbered ‘1’ in Figure 5.1 used the dictionary at the same time for one task (whether first or second), and participants numbered ‘2’ used it at the same time for the other. The recorder was switched on before the participants began the first task, and was allowed to run until both tasks were completed so that all participants were recorded unobtrusively. In practice, it was noted that once participants got underway with the tasks, the video-camera was largely ignored. Observation sheets (Figure 5.2) were completed retrospectively from the video evidence. Look-ups were noted for all 10-minute blocks in the entire 50-minute examination, and stop-watch timings were recorded for each of these. Each look-up was treated as an individual event. That is, if it was apparent that a participant returned to the same dictionary entry on a separate, later occasion this was recorded as a separate look-up. An exception was made whenever it appeared that participants were moving from a particular look-up to their writing and back again in a very short space of time. These occasions were deemed to be single look-ups. On a few occasions it was unclear whether participants were looking at a dictionary entry or re-reading aspects of the test task or their response. Periods of time that were ambiguous were omitted from the schedule of observations because they could not be regarded as definite look-ups. In both studies I also compared the observational evidence with look-ups identified through participants’ self-reporting in the texts. This was done to determine the extent of reliability of the test takers’ self-reports of dictionary use.

1









1

2

2

2



Figure 5.1  Placement of video camera for the first and second studies.



Chapter 5.  When the dictionary becomes a liability

Name _____________________________ 1

2

3

4

5

6

7

8

9

10

1st 2nd 3rd 4th 5th

Figure 5.2  Observation sheets for the first and second studies.

The first study Figure 5.3 illustrates the number of look-ups each participant in the first study made in each 10-minute block. With the exception of Simon, who only chose to use the dictionary on three occasions, all the participants made use of the dictionary at different points throughout most of the examination. Indeed, it was evident that three participants (one intermediate and two advanced) accessed the dictionary more or less evenly throughout the whole examination (that is, in all five 10-minute segments), thereby indicating an on-going reliance on the dictionary to support their writing. A summary of the lengths of time taken to use the dictionary is given in Table 5.1. It was evident from these data that the advanced participants consulted the dictionary more quickly than their intermediate counterparts (average look-ups were 30 seconds long compared to 47 seconds). Also, although the advanced participants consulted the dictionary more often than the intermediate writers (on average, 18 times rather than 13), as a consequence of speed they spent less time overall in dictionary look-ups (8 minutes rather than 10½ minutes). These data suggest that level of test taker ability might be a factor in the length of time it takes to use a dictionary, although both groups in this study appeared to want to use the dictionary, on average, a similar number of times. These test takers were also choosing, on average, to spend around 20% of the available examination time on dictionary consultation. I considered this quite a considerable amount of time and, if typical, indicated a potential distraction when it came to completing a time-constrained examination.

97

98 Dictionary Use in Foreign Language Writing Exams





*OUFSNFEJBUF
1BSUJDJQBOUT

1BSUJDJQBOU

1BUSJDJB


1BSUJDJQBOU

4JNPO































NJOVUF
CMPDL







          















NJOVUF
CMPDL


"EWBODFE
1BSUJDJQBOUT
1BSUJDJQBOU

4IBSPO




           























1BSUJDJQBOU

3BDIFM

          




NJOVUF
CMPDL




NJOVUF
CMPDL

           




           




1BSUJDJQBOU

+BOFU

1BSUJDJQBOU

+FOOZ












NJOVUF
CMPDL













NJOVUF
CMPDL













Figure 5.3  Number of look-ups made – first study.

Table 5.1  Summary of dictionary use trends – first study. Participant Level of ability

Number of look-ups

Average dura- Total dura- Percentage of total tion of each tion of look- time available look-up ups (50 minutes)

1 2 3

Intermediate Intermediate Intermediate M SD

 3 11 24 13 10.6

00:42 00:50 00:51 00:47 00:05

  2:32   7:51 21:35 10:39   9:49

  5% 16% 43% 21% 19.55

4 5 6

Advanced Advanced Advanced M SD

10 26 17 18   8.02

00:31 00:24 00:26 00:30 00:06

  5:44 10:25   7:34   7:54   2:21

12% 21% 15% 16%   4.58



Chapter 5.  When the dictionary becomes a liability

Table 5.2  Summary of look-up timings by task – second study. Task

A

M SD B M SD C M SD All M tasks SD

Block of Time 1st 10 minutes 2nd 10 minutes 3rd 10 minutes 4th 10 minutes 5th 10 minutes L

T

L

T

L

T

L

T

L

T

3.6 1.8 1.6 1.5 2.6 2.2 2.6 1.9

2.25 1:01 1:17 1:06 1:31 1:09 1:51 1:16

3.6 2.5 2 2.0 2.6 1.3 2.7 2.0

2.34 1:08 1:17 1:06 2:11 2:03 2:00 1:29

2.8 1.3 2.4 1.8 2 1.9 2.4 1.6

1.45 0:46 1:51 1:06 1:39 1.18 1:45 1:00

2.6 2.2 3 3.1 3 1.6 2.9 2.2

1.34 1:11 1:38 1:42 1:39 1:16 1:37 1:18

1.6 1.2 2.6 2.1 1.6 1.1 1.9 1.5

0.55 0:51 3:00 2:42 1:00 0:48 1:38 1:52

Note. L = number of look-ups; T = total time.

A closer inspection of the look-up time of the two participants who used the dictionary the most (intermediate participant 3 (Jenny) and advanced participant 5 (Sharon)) makes the potential liability of the dictionary more apparent. Sharon made 26 look-ups, and spent just over 10 minutes (21% of the examination time) on this. Jenny made 24 look-ups, but the longer time each took meant that she spent over 20 minutes, or 43% of her available time, using the dictionary – a serious distraction from the time available to complete the task.

The second study The second study provided an opportunity to further investigate frequency and duration of look-ups, but this time across three different tasks. Table 5.2 records the average number and duration of look-ups in each 10-minute block, for each individual task and then overall. Findings reveal the following dictionary use trends: •

•

In Task A, the dictionary was used most in the first 20 minutes (three to four times in each 10-minute block). In Tasks B and C average look-ups generally ranged between one and three in each 10-minute block. More reliance was therefore placed on consulting the dictionary in the first task than in the next two. On average participants spent between 1½ and 2 minutes in each 10-minute block consulting the dictionary, and on those occasions made between two and three look-ups, with a tendency to make more look-ups in the first 40 minutes.

99

100 Dictionary Use in Foreign Language Writing Exams

•

Overall, the dictionary was used pretty consistently throughout the examination, indicating its usefulness not only to access the task but also to support the response through the whole process of writing. This was so across all three tasks.

Table 5.3 records the total number and duration of all look-ups for each individual task. It was found that, on average, 14 look-ups were made in Task A, over a period of 9½ minutes. This had reduced to between 11 and 12 look-ups, over a period of between 8 and 9 minutes, in Tasks B and C. Look-ups took participants around 45 seconds each. The findings of the second study regarding the total amount of time used to access the dictionary mirrored those of the first – that on average test takers used the dictionary for around 20% of the available time. Using the dictionary over three consecutive weeks did seem, however, to help the participants to use the dictionary marginally more efficiently. It was not possible to determine, from this evidence at least, if this was due to a kind of ‘practice effect’, or was a reflection of the types of task the test takers had been asked to do (although the test taker perceptions explored in the next chapter help to throw some light on which one of these may have made the difference). Nevertheless, if practice with the dictionary was making the difference, however slight, there was some indication that giving students opportunity to practice dictionary use in time-constrained contexts might be helpful in alleviating somewhat the potential distraction of using a dictionary in such contexts – an issue that therefore warranted further reflection.

Table 5.3  Summary of look-up timings by participant – second study. Look-ups ParticiTotal number Total duration pant Task A Task B Task C Task A Task B

Task C

Average duration Task A Task B Task C

1 2 3 4 5 M SD

04:53 11:47 10:10 10:00   3:36   8:05   3:36

00:34 00:47 00:38 00:44 00:37 00:40 00:05

21 12 11 20  7 14.2   6.1

15 14  8 18  3 11.6   6.0

 5 12 18 16  8 11.8   5.4

12:06   9:25   7:03 14:51   4:23   9:33   4:06

  7:49 10:55   9:16 13:49   3:33   9:04   3:48

00:31 00:46 01:09 00:46 01:11 00:52 00:17

00:58 00:58 00:33 00:37 00:27 00:43 00:14



Chapter 5.  When the dictionary becomes a liability 101

An additional question which I wanted to answer was the extent to which the observational evidence from the first two studies matched with the self-reporting evidence of the participants – their highlighting or underlining in the texts. This was important in establishing the extent to which the self-reports could be considered to be giving reliable information about dictionary use. Participants in the first study self-reported that on average they made eight look-ups, whereas the observational evidence indicated an average of 15 look-ups, or about double the number reported. In the second study participants again self-reported an average of around eight look-ups, whereas on average 13 were observed. On both occasions, therefore, the observational evidence appeared to suggest that the dictionary was being used more often than the participants were claiming. Was this indicating that self-reporting of look-ups was an unreliable measure? As mentioned in Chapter 4, there is indeed some unreliability in it – for a variety of reasons the test takers may have under-reported their actual dictionary use. One important factor needed to be taken into account: although the observational data did not discriminate between test taker use of the two different main sections of the dictionary (German/English and English/German) it was clear that on several occasions test takers were moving between both sections before settling on a word. On these occasions the observational evidence would record two look-ups, whereas the self-reporting would indicate one – the dictionary was used twice to solve one look-up problem. Sharon (first study) illustrates this trend. The video evidence indicated that, of a total of 26 observed look-ups, 14 were from the German/English section and 12 were from the English/ German section – Sharon was moving back and forth between entries. Sharon self-reported nine look-ups, and it is feasible to conclude that she may only have used nine items as a result of her look-up activity. Although the self-reporting evidence may need to be treated with some caution, the case of Sharon indicates that the differential between observed and selfreported look-ups was not necessarily as great as it first appeared. This was quite important when it came to interpreting the look-up evidence in the third study, and the extent to which self-reporting gave an informative picture of how participants used their dictionaries.

The third study In the third study participants’ underlining of look-ups, together with a question on the longer questionnaire about frequency of dictionary use, enabled some comparative data on dictionary use trends to be gathered. The questionnaire

102 Dictionary Use in Foreign Language Writing Exams

Figure 5.4  Frequency of dictionary use – third study.

asked participants to indicate on a four-point scale how often they judged that they had used the dictionary. Results are given in Figure 5.4. It appeared that the vast majority of participants claimed that they made up to ten look-ups, indicating fewer look-ups than those in the first two studies had actually made, but similar to the number of times participants in the first two studies had self-reported. When answers to this question were compared, on a case-by-case basis, with the number of occasions participants indicated dictionary use through underlining in the texts, it was noted that the vast majority (31/44 or 70%) reported accurately the number of times they had used the dictionary. All but one of the remainder under-reported in response to the question, indicating that they thought they had used the dictionary less often than the underlining would indicate. Of three participants who had not underlined anything in the texts, questionnaire responses revealed that two (lower intermediate) participants claimed to have used the dictionary more than 11 times, and one (intermediate) participant 3–10 times. It was possible to conclude that the self-report evidence of look-ups (that is, the underlining) gave a fairly accurate picture of look-up activity. Table 5.4 presents the number of look-ups made by participants at each ability level, together with the average number of times participants at each level, and overall, self-reported dictionary use. Participants’ underlining indicated that on average nine look-ups were made, but there did seem to be a difference between ability groups. It appeared that the higher the ability level, the more sparing the use of the dictionary.



Chapter 5.  When the dictionary becomes a liability 103

Table 5.4  Range of look-ups based on underlining in the response – third study. Level 3)a

Lower intermediate (n = Intermediate (n = 21)b Upper intermediate (n = 16) Advanced (n = 4) Overall (n = 44)

Range of look-ups

M

SD

4–15 0–32 2–20 2–13 0–32

9 9.9 8.4 6.8 9

5.6 6.7 4.7 4.9 5.7

Note. a Two lower intermediate participants did not follow the instruction to underline, although it was clear from the questionnaire that they had used the dictionary. These participants were not included in the figures. b One intermediate participant also did not follow the instruction to underline and was not included in the figures. One intermediate participant did not use the dictionary at all in the response, and this is included.

Dictionary look-ups – what do all three studies tell us? A comparison of the duration of look-ups between the first two studies and Hurman and Tall’s study (1998, p. 15), in which the researchers timed 27 test takers, is represented diagrammatically in Figure 5.5. In both contexts look-ups most commonly took between 20 and 40 seconds. However, there did appear to be a tendency for look-ups to take a little longer in the two reported studies (over a third were above 40 seconds) than in Hurman and Tall’s study (about a quarter were above 40 seconds). Another interesting comparison (Figure 5.6) arises from placing the total of look-ups (noted from all three studies) against the number for writing recorded by Bishop (2000), who used questionnaires to investigate the actual experiences

Figure 5.5  A comparison of look-ups by duration.

104 Dictionary Use in Foreign Language Writing Exams

Figure 5.6  A comparison of look-ups by frequency. Note. The percentages for all three studies are derived from the sum of participants’ underlining or highlighting of words used in the responses.

and post hoc perceptions of OU students who had taken examinations in French in two conditions – with and without dictionaries. Bishop (2000) found that two out of three test takers reported that they used the dictionary no more than five times to locate words when writing. Just over one in 10 used the dictionary more than eleven times. There appeared to be somewhat more judicious use of the dictionary in Bishop’s (2000) study than in those reported here, where two out of three participants made at least six look-ups. With regard to look-ups and ability level, I compared the third study data with those generated from Atkins and Varantola (1998a). These two researchers had undertaken a detailed large-scale investigation with groups of learners of English as a foreign language (EFL) drawn from a range of educational institutions across Europe. They wanted to discover what dictionary users actually did when they used a dictionary to complete a set of test tasks. Although, strictly speaking, findings between their study and the one reported here are not directly comparable, essentially because the nature of the tasks in Atkins and Varantola’s study were different, there is an interesting point of comparison (Table 5.5). Lower ability participants (groups D and C in Atkins’ and Varantola’s study, and the lower intermediates and intermediates in the third study) made more use of the dictionary than higher ability participants. The most able participants (group A and the advanced participants) used the dictionary the least. Comparison with previous research raises a number of questions. Why did the participants seem to take longer with look-ups than those observed by ­Hurman



Chapter 5.  When the dictionary becomes a liability 105

Table 5.5  Number of look-ups – a comparison by ability. Atkins and Varantola’s (1998a) study

The third study

Level of ability Number of Percentage of Level of (least to highest) look-ups look-upsa abilityb

Range of Average look-ups in number of the response look-ups

D C B A

4–15 0–32 2–20 2–13

a

634 743 600 301

17.5 19 14 11.5

LI I UI A

9 9.9 8.4 6.8

The percentage of look-ups is a ratio of the total number of possible look-ups. lower intermediate, I: intermediate; UI: upper intermediate; A: advanced.

b LI:

and Tall (1998)? Why did the participants seem to make more look-ups in these tasks than had been identified by Bishop (2000)? Why was there a discrepancy between the first and third studies with regard to the advanced participants’ dictionary use? It may be speculated that the duration and range of look-ups made by participants in these studies, in comparison to evidence derived from elsewhere, is affected by a number of factors.

Duration of look-ups In the first two studies: 1. Higher level writing tasks may demand more sophisticated vocabulary than lower level tasks, meaning that participants took slightly longer to locate words in the first two studies than in Hurman and Tall’s (1998) research, which had focused on writing at less complex levels. 2. The size of the dictionary (around six times larger than those used in the Hurman and Tall study) may have made it somewhat more difficult to locate the required information than was the case for Hurman and Tall’s participants. 3. The maturity and higher proficiency level of the participants may have meant that they were more confident with exploring dictionary entries than the younger and less proficient participants in Hurman and Tall’s study, and were willing to invest slightly more time in doing so. (As Bachman and Palmer (1996) suggest, age may be influential in how test takers perform on a given task.)

106 Dictionary Use in Foreign Language Writing Exams

Look-ups and ability level It is difficult to account for the differential findings between the first and third studies with regard to dictionary use by advanced learners (although the small sample size in the first study must be taken into consideration). Again, one possible explanation is the difference in maturity between the two groups, with those in the first study being more mature, and also more experienced language users. The advanced participants in the first study may have chosen to use the dictionary more because they felt more comfortable doing so in comparison to the younger upper ability participants in the third study, who were also less experienced language users.

Number of look-ups In terms of the number of look-ups made across all three studies, Bishop (2000) suggests the following: “If we define overuse as more than 10 times in the 45 minutes allocated to [writing] then it is clear that students in the main did not overuse their dictionaries [in the OU study] and thereby lose time” (p. 61). Although setting the benchmark for overuse at ten look-ups in 45 minutes appears to be somewhat arbitrary, the tendency for a number of participants across all three studies, in comparison to Bishop’s finding, was to ‘overuse’ the dictionary, with a consequent loss of time. The marginally longer time taken to make the look-ups in comparison with the observed evidence in Hurman and Tall’s (1998) study would have accounted for an even greater negative impact on time. It is not possible to draw any conclusions about why the dictionaries were used more often in the studies reported here. However, Bishop relied on retrospective questionnaire evidence some time after tests had been completed, and this may have influenced the participants’ recall of their actual dictionary use. The important conclusion that can be drawn from the observational and selfreport evidence is this: the dictionary is potentially a serious liability when it comes to its use in time-constrained tests, and the impact on time is one that test setters and test takers must consider seriously. (As to the perceived level of seriousness of this liability, we need to take into account the voices of the test takers themselves – but this will be the subject of the next two chapters.)

The number of errors the test takers made The second major source of dictionary liability which Weigle (2002) identifies is the inability of some test takers to use dictionaries effectively. Their ­inappropriate



Chapter 5.  When the dictionary becomes a liability 107

Figure 5.7  Comparison of percentages of look-up errors across all three studies.

use can lead to the types of error exemplified in Chapters 1 and 4. It is this potential liability of dictionary availability that arguably causes the most concern among test setters and teachers (and if we add to this the time-constrained nature of a writing task it might well be that the problem is being compounded). It is important to take a look at some of the errors the test takers in these three studies made and thereby determine how serious dictionary misuse is as a potential liability. I turn now to the evidence regarding dictionary misuse. In Chapter 4 I indicated that, taken across all three studies, there was a level of symmetry in the types of look-up participants made. The major use to which dictionaries were put was to find individual items of lexis, with nouns making up around half of all look-ups. Participants overall had made 548 look-ups. Of these, 247 items (45%) were found to be in error. This did not appear to be a particularly high level of success, although it should be noted that the majority of look-ups were made in the third study, and this influences the overall impression. Indeed, when these figures were examined on a study-by-study basis an interesting anomaly became apparent. Considered in percentage terms, participants in the first and second studies made errors with look-ups around 15% of the time, and were therefore able to use look-ups correctly on 85% of all occasions when look-ups were used. This indicated a very high success rate with the dictionary. By contrast, those in the third study appeared to make errors with look-ups 50% of the time – one error for every two look-ups. Figure 5.7 illustrates the percentages of errors made in relation to the percentages of items used in each of the six look-up categories. The areas above the lines indicate correct look-ups, with the areas below showing look-up errors.

108 Dictionary Use in Foreign Language Writing Exams

Noticeably more errors were made in all categories of look-up in the third study compared to the first two. These differential findings raise several questions. Why did there appear to be such a difference in look-up errors? Could it be related to the type of task? Or the type of student? Or the type of dictionary? Before considering these questions, it is helpful to take a closer look at the nature of the errors participants made, and then to go on to see if these important questions can be answered satisfactorily.

The first two studies It was evident from the first study that three of the six participants made no errors at all with look-ups, and a further two made errors based on dictionary look-ups on two occasions only. Patricia, however, was led astray with three out of the five look-ups she used. In the second study look-up errors were quite evenly distributed among the test takers, with no one test taker making considerably more or less errors than any other. Examples of look-up error from the first two studies include the following: Error

Intended meaning Should be

Nature of error

kennen

know

wissen

an inappropriate word was selected (‘be acquainted with’ rather than ‘know [a fact]’)

Hafens

ports

Häfen

an incorrect plural was used

hilfsbereit

helpful

hilfreich/nutzlich

an inappropriate word was selected (‘ready to help’ rather than ‘useful’)

Kriegsgefangener prisoner of war

Kriegsgefangene

the correct word was located, but was taken directly from the dictionary, using a contextually incorrect declension

Abschluss

conclusion

Schlussfolgerung

a contextually incorrect word was chosen

gebäude

building

Gebäude

wrong capitalization



Chapter 5.  When the dictionary becomes a liability 109

Geschichte Romane

history [historical] Geschichtsromane failure to form a correct novel compound

In die Schweiz

In Switzerland

In der Schweiz

failure to recognize that a change in case was needed

informieren

information

Informationen

using a verb when a noun would have been more appropriate

vollig

completely

völlig

misspelling

Materiell

material

aus Kunststoff

using an adjective when a noun was required

Two types of error are indicated: an inability to choose the appropriate word from the list of options provided in the dictionary – word choice error; and an inability to decline the word correctly (that is, apply the relevant grammar) – word form error. Although it is not possible to determine whether participants accessed information from which they went astray from the German-English or the English-German section of the dictionary, a look at the relevant English-German entries for a number of other misapplied words in the second study would suggest that these entries (all located in the Collins German Dictionary) were the starting points. It would seem that the participants did not have, or did not apply, knowledge about how to use the entries correctly. Peter, writing a letter about his experiences in a German language course, wanted to speak about having more conversation practice (or chats) in class. He had presumably looked up ‘chat’ in the dictionary:

He had chosen the first entry and had gone on to use the wrong plural form – Unterhaltungs rather than Unterhaltungen. No plural was suggested in the entry. Should an intermediate level student of German know that all nouns ending in -ung are feminine, and make their plural with -en? Or should the dictionary have provided this information? Also, arguably the more appropriate entry for his context was illustrated in the example ‘could we have a ~ about it?’ – that is, he should perhaps have aimed to write ‘[the opportunity to] have chats with each other in class’ rather than the more simple ‘more chats in class’. If he had chosen this phrase he would have needed to restructure his whole sentence, which may

110 Dictionary Use in Foreign Language Writing Exams

have been beyond his linguistic capability. It would seem that the dictionary entry did not necessarily contain sufficient information for Peter to know exactly how to use it. John wanted to talk about the reasons for writing a letter. He had most likely looked up ‘reason’ in the dictionary:

Like Peter, he had used an incorrect plural form – Grunden rather than Gründe. Again, no plural was suggested in the entry. Should John have taken the time to check the entry in the German-English section (where the plural was provided)? Or should the dictionary have provided the appropriate plural ending in the English/German section? Sandra wanted to speak of food poisoning. It is likely she located the following entry:

She needed to use the past participle form, ‘poisoned’. She located the correct entry – the transitive verb form given in entry (2) – but she did not conjugate it correctly, choosing simply to write the infinitive form (vergiften) as given in the dictionary entry. Mary wanted to write ‘I would like to reply to your letter’ and most likely looked up this entry:

She located the entry ‘in ~ to your letter/remarks’, in Beantwortung Ihres Briefes. She successfully switched the formal Ihres into an informal deines, but did not recognize that Beantwortung was a noun, and therefore inappropriate for her context. The appropriate entry for her context was (3) – antworten (auf). Jessica spoke of misusing information:

She located the noun entry Mißbrauch (1). The appropriate entry for her context was the transitive verb mißbrauchen, entry (2).



Chapter 5.  When the dictionary becomes a liability

Elsewhere in the same essay Jessica wanted to talk about how drugs can affect health:

She needed the entry ‘affect’ in the sense of ‘has an effect on’. She had used the first entry under (a), and had applied the information regarding prepositional use (auf + accusative) correctly. Her choice was logical in the circumstances, but the more contextually appropriate verb would have been schaden (last entry under (a)). The dictionary entry did indicate that schaden was an appropriate word to use when talking about health, but Jessica’s use of both groups of entries (‘misuse’ and ‘affect’) suggests that she may simply have gone for the first entry listed in both cases and not looked through the remaining information. In the discursive essay task, John spoke of the state of mind of ‘stay at home’ mums – they might suffer as a result of lack of social contact:

He also selected the first entry – the transitive verb erleiden. In the context he actually needed the intransitive form – leiden (unter) in entry (2). These types of error raise a particularly serious problem with dictionaries. Tomaszczyk (1983), as pointed out in Chapter 1, is inclined to lay the blame at the feet of the dictionaries themselves. He argues that learners of an L2 are “not sophisticated enough to realise that they are poorly served and take the information supplied at face value” with the result that what they produce reveals “the deficiencies of such dictionaries in a most dramatic way” (p. 47). We might conclude that in each of the cases above the dictionary entry was deficient. Perhaps more examples or more contextual and usage information could have been provided. However, the types of errors illustrated here also possibly indicate a more complex interaction. It is not just the dictionary (as Tomaszczyk suggests) and not just the user, but a combination of the two. That is, some users may have coped differently with these entries, or the same users may have coped differently if additional

111

112 Dictionary Use in Foreign Language Writing Exams

or other information had been provided. In fact, Nesi and Meara (1991), who investigated the use of dictionaries in reading tests, speak of a three-way interaction. They suggest that it is reasonable to assume that, out of the test, the dictionary or the user, at least one factor is failing in some way if dictionary use cannot improve performance. In their view, the responsibility lies with all three factors. If we believe that dictionary use should make a supportive difference to students when writing, Nesi and Meara’s conclusion raises the possibility that errors arise from this three-way interaction. If there were items relevant to task completion that the test takers did not know and wanted to use in their writing, either the dictionary did not provide sufficient information, or the dictionary users lacked sufficient skill in using the dictionary and could not locate the relevant information, or they did not know how to use the information correctly when found. The dictionary becomes a liability if this three-way interaction interferes with an improved performance. All of this leads Spolsky (2001), for example, to speculate that although using dictionaries may have beneficial washback into the teaching situation, the use of dictionaries in examinations is a more complex matter. In his view, allowing dictionary use introduces an additional source of testing error into the process, leading to lower reliability and weakened validity. The usefulness of the test is therefore diminished in these respects. Whether we consider that dictionaries are supportive tools or that dictionaries should not be allowed in examinations, evidence of misuse is a considerable concern, especially if we cannot identify the source of error. Is it the dictionary? The user? The task? Or all three? Despite these problems, however, the reported success rate with using the dictionary in the first two studies (at least 85%) is very high.

The third study In the third study, the test takers reported 397 instances of dictionary look-up. On 200 of these occasions (50% of look-ups) the entry was applied incorrectly. The following errors were noted: • • • •

Wrong word – 97 out of 200 occasions (24% of all look-ups) Wrong form – 80 occasions (20% of all look-ups) Wrong gender – 6 occasions (1.5%) Misspelling – 17 occasions (4%).

The 200 errors were analyzed for each of the main categories of look-up. Results of this analysis are recorded in Table 5.6. Principal errors for the different types of words were noted as follows:



Chapter 5.  When the dictionary becomes a liability 113

Table 5.6  Errors according to types of word – third study. Type of word

Nouns Verbs Adjectives Adverbs Phrases Other items

Total

183   87   69   28   14   16 397

Error

  95   49   35    9    9    3 200

Type of error wrong word

wrong form

wrong gender

misspelling

49 21 12  8  5  2 97

28 28 19  0  4  1 80

6 0 0 0 0 0 6

12  0  4  1  0  0 17

Adapted from East (2006b, p. 191).

•

•

•

•

•

Nouns: wrong forms were caused mainly by errors of plural (16 occasions). The most common error of spelling was failure to capitalize the noun (eight occasions). On two occasions errors with failure to form a correct compound were also noted. Verbs: main errors of form were use of an inappropriate conjugation (seven occasions), failure to apply the appropriate grammatical rules with regard to the use of separable verbs (eight occasions), and use of the infinitive when a participle was required (five occasions). Adjectives: most errors of form were accounted for by failure to add the correct adjective ending (13 occasions). There were four spelling errors, which included three occasions when the adjective was capitalized (as if it were a noun). In addition, failure to form the comparative correctly was observed on two occasions. Adverbs: apart from one spelling error, all errors were accounted for by the choice of a wrong or inappropriate adverb. There was a high incidence of success with using adverbs (possibly because adverbs do not require any inflection and can therefore be used ‘as is’ from the dictionary). Adverbs were used correctly or appropriately 67% of the time. Phrases: apart from incorrectly chosen phrases, errors in the use of phrases focused on problems with word order (three occasions) – that is, using the phrase exactly as found in the dictionary without reference to relevant word order rules – or problems with case (one occasion) – that is, using the phrase correctly but failing to take account of a case requirement that was dependent on the phrase.

Several errors mirrored the types observed in the first two studies. Examples include the following:

114 Dictionary Use in Foreign Language Writing Exams

Word(s) looked up

Should be

Meaning

Category of Type of error error

schiessen

schliessen

shut

wrong word may have been a spelling error

stagnate

wrong word noun instead of verb; word choice error

[es wird] Verfall stagnieren

Verschwendung verschwenden disappear

wrong word noun instead of verb

Tourist aktivität Touristenakti- tourist activi- wrong form no compound (also vitäten ties spelling error) Orts

Orte

places

wrong form incorrect plural

viele Grunden

Gründe

many reasons wrong form incorrect plural

wir will nicht

wir wollen nicht

we do not want to

wrong form wrong conjugation

mehr informiert informierter

more informed wrong form wrong form of comparative

es kann beschä- beschädigt digen werden

it can become wrong form infinitive instead of damaged past participle

zu ausgeben

auszugeben

give out / spend

ankommen jeden Tag

kommen jeden arrive each day wrong form incorrect separable Tag an verb use

Ich einverstanden mit das

Ich bin damit I agree with einverstanden that

wrong form incorrect syntax

das Struktur

die Struktur

the structure

wrong gender

die Naturliches Attraktionen

natürliche

the natural attractions

misspelling also wrong form – adjective treated as noun

wirtschaft

Wirtschaft

economy

misspelling no capitalization

wrong form incorrect separable verb use

A further analysis of entries found in the Collins Pocket was also made. This analysis revealed a number of ways in which entries were inappropriately applied. There were, for example, several instances where participants attempted to ­ translate



Chapter 5.  When the dictionary becomes a liability 115

­ irectly, and on a word-for-word basis, from English into German, and as a result d came up with virtually meaningless German (East, 2005a). On other occasions, the incorrect word for the context was chosen. One lower intermediate participant wanted to communicate the notion that people might ‘take advantage’ of tourists in New Zealand. It seems likely that this candidate looked up ‘take’ and ‘advantage’ as two independent words, and as a consequence wrote nehmen Vorteil. In fact the expression she should have used was andere Leute ausnutzen. The dictionary entry for ‘advantage’ provided some clue to this, but may not have supplied sufficient information:

An upper intermediate participant wrote about the ‘double standards’ of people in New Zealand. She suggested [wir haben] Verdoppeln Massstäbe, not realizing that the correct German expression is wir messen mit zweierlei Mass. The dictionary entries for neither ‘double’ nor ‘standards’ would have helped here. An intermediate participant, presumably having looked up ‘way’, translated the expression ‘the best way to’ directly into German by writing Die beste Weg um. It appeared she had simply gone for the first entry, assuming that it would be correct:

She had not only used the wrong gender for Weg (even though the dictionary signaled the correct gender), but had also not realized that Weg carries the sense of ‘pathway’. The dictionary entry did not make this explicit. In fact the concept this participant was trying to communicate would have been expressed better in German by using an adverb (am besten), although the second entry (Art und Weise) could have been utilized in some way. Another intermediate participant successfully located the correct word to render the expression ‘headstart’ – Vorsprung – but had located this as ‘start’ and had added the word Kopf (head) to it, thereby coming up with eine gute Kopf Vorsprung im Leben (a good head headstart in life). This also involved a gender error. There was no information in the dictionary to help this writer to know that Vorsprung itself would have communicated adequately. The most serious error by an intermediate participant led to a completely meaningless German sentence. It seems most likely that this participant had accessed two words in the English-German section of the dictionary, but had

116 Dictionary Use in Foreign Language Writing Exams

c­ ompletely failed to apply the words correctly. Attempting to write ‘that can lead to the possibility’ she had written ... das Büchse Vorbild zu moglichkeit ...

when she should have written … das kann zur Möglichkeit führen ...

This participant had probably looked up the entry ‘can1’, and chosen the noun Büchse (tin can) instead of the modal verb können, which was in the second separate entry:

She had also presumably looked up the word ‘lead’, and chosen the noun Vorbild (‘lead’ in the sense of ‘example’) instead of the verb führen:

The dictionary entries would not have helped this participant to locate the more appropriate words without prior knowledge of what she was looking for (two verbs). The entry ‘can2’ did supply information that should have been useful to this participant, but it would appear that she went for the first listed entry and did not take the trouble to check other entries. It appeared she had even overlooked the ‘keyword’ alert – the “special status” given to some frequently occurring or otherwise significant words (Collins, 2001, p. viii). On two occasions the word ‘around’ was looked up and the entry was applied incorrectly:

One lower intermediate participant, talking of ‘around the world’, wrote ungefähr die Welt. The dictionary entry did indicate that ungefähr means ‘around’ in the sense of ‘approximately’ or ‘almost’, but this participant did not realize that this



Chapter 5.  When the dictionary becomes a liability 117

was the inappropriate word for her context. An intermediate participant similarly wanted to use ‘around’ in the expression ‘around New Zealand’. The dictionary entry did not give sufficient contextual information, and this participant thereby wrote ringsherum Neuseeland (indicating ‘around’ in the sense of circumnavigation) whereas in correct German in the context this should have been durch Neuseeland (‘around’ in the sense of ‘throughout’). Another instance of wrong or inappropriate word choice was ‘economy’ rendered as Sparsamkeit (means ‘economy’ in the sense of ‘thrift’) instead of the contextually more appropriate Wirtschaft. This occurred on two separate occasions, and the error was most likely attributable to the following entry:

The phrase Nutzen ziehen aus (‘to benefit from’) proved a popular expression from the Collins Pocket, used by four participants. The dictionary entry gave the following information:

Three participants (two intermediate and one upper intermediate) located the right part of the entry, but went on to use the expression exactly as found in the dictionary, resulting in incorrect syntax. In summary, errors with regard to look-ups in this study were of the two main types identified in the first two studies. The wrong word or phrase was taken from the dictionary. Alternatively, the word or phrase, in and of itself correct, was applied incorrectly. Similarly, dictionary entries may have led participants astray in two ways. There was insufficient information in the entry about how to use the word or which word was contextually the more appropriate. Also, the participants did not know how to use the word. Herein lies further evidence of what I have earlier described as the complex interaction between the dictionary, the user, and the task, and it is not possible to say with any definitiveness where among these three variables and interactions the source of an error most lies. Given that the test takers in the third study could be classified according to level of ability and level of prior experience with the dictionary, there was also the opportunity to investigate whether either of these factors made any difference to success rate with the dictionary. For this analysis the four participants who either did not underline any words, or (in one case) did not use any words from the dictionary in the response, were omitted, and the group size was reduced from 47 to 43. The levels were collapsed into two: lower ability (lower-intermediate /

118 Dictionary Use in Foreign Language Writing Exams

Figure 5.8  Success with look-ups according to ability and experience – third study.

intermediate, n = 23) and upper ability (upper intermediate / advanced, n = 20), and inexperienced (no experience / quite inexperienced, n = 24) and experienced (quite / very experienced, n = 19). Figure 5.8 records the level of success with look-ups, according to level of ability and level of experience with the dictionary, against two factors: firstly, how often the look-ups were correct (that is, no error of word choice); secondly, if correct, how often look-ups were subsequently used correctly (that is, no error of word form). The upper ability participants were able, on average, to select words correctly from the dictionary eight times out of ten (M = 81.9%, SD = 16.3), compared to seven out of ten for the lower ability group (M = 71.8%, SD = 16). Furthermore, the upper ability group was able to use words correctly two times out of three (M = 61.5%, SD = 25.6), compared to less than half the time for the lower group (M = 45.1%, SD = 21.1). The wide standard deviations indicate a very broad range of success with look-ups. Independent samples t-tests reveal that, as far as ability was concerned, the two groups were significantly different from each other with regard to both correct look-ups (t(41) = –2.044, p = .047) and correct use of look-ups (t(41) = –2.287, p = .027). The upper ability participants were significantly more successful with both correctly locating and correctly using look-ups. It seemed that ability was a factor in determining the success with which test takers could locate and use look-ups correctly.



Chapter 5.  When the dictionary becomes a liability 119

Experience with the dictionary appeared to make some difference to whether participants could correctly access the word. The more experienced group was able to locate the correct word almost three times out of every four (M = 71.8%, SD = 20.3), whereas those with less experience seemed to fare better (M = 80.1%, SD = 12.5). Experience did not appear to make a difference to success with using look-ups, however (experienced users, M = 53.5%, SD = 22.1; inexperienced users M = 52.1%, SD = 26.5). The wide standard deviations again indicated the broad range of success. It is curious that those with less prior dictionary experience seemed to be more successful with correctly locating look-ups. However, no significant differences between the groups were found (correct look ups – t(41) = 1.554, p = .109; correct use of look-ups – t(41) = –.173, p = .