E-Learning Technologies and Evidence-Based Assessment Approaches Christine Spratt Royal Australian and New Zealand College of Psychiatrists, Australia Paul Lajbcygier Monash University, Australia
Information science reference Hershey • New York
Director of Editorial Content: Senior Managing Editor: Managing Editor: Assistant Managing Editor: Typesetter: Cover Design: Printed at:
Kristin Klinger Jamie Snavely Jeff Ash Carole Coulson Amanda Appicello Lisa Tosheff Yurchak Printing Inc.
Published in the United States of America by Information Science Reference (an imprint of IGI Global) 701 E. Chocolate Avenue Hershey PA 17033 Tel: 717-533-8845 Fax: 717-533-8661 E-mail:
[email protected] Web site: http://www.igi-global.com and in the United Kingdom by Information Science Reference (an imprint of IGI Global) 3 Henrietta Street Covent Garden London WC2E 8LU Tel: 44 20 7240 0856 Fax: 44 20 7379 0609 Web site: http://www.eurospanbookstore.com Copyright © 2009 by IGI Global. All rights reserved. No part of this publication may be reproduced, stored or distributed in any form or by any means, electronic or mechanical, including photocopying, without written permission from the publisher. Product or company names used in this set are for identi.cation purposes only. Inclusion of the names of the products or companies does not indicate a claim of ownership by IGI Global of the trademark or registered trademark. Library of Congress Cataloging-in-Publication Data E-learning technologies and evidence-based assessment approaches / Christine Spratt and Paul Lajbcygier, editors. p. cm. Includes bibliographical references and index. Summary: "This book aims to provide readers with a variety of contemporary solutions to identified educational problems of practice related to the assessment of student learning in e-learning environments"--Provided by publisher. ISBN 978-1-60566-410-1 (hardcover) -- ISBN 978-1-60566-411-8 (ebook) 1. Computer-assisted instruction--Evaluation. 2. Web-based instruction-- Evaluation. 3. Educational technology--Evaluation. I. Spratt, Christine. II. Lajbcygier, Paul. LB1028.5.E19 2009 371.33'4--dc22
2008047734
British Cataloguing in Publication Data A Cataloguing in Publication record for this book is available from the British Library. All work contributed to this book set is original material. The views expressed in this book are those of the authors, but not necessarily of the publisher.
Advances in Information and Communication Technology Education (AICTE) Series Editor-in-Chief: Lawrence Tomei, Robert Morris University, USA and Mary Hricko, Kent State University, USA ISBN: 1935-3340
E-Learning Technologies and Evidence-Based Assessment Approaches Edited By: Christine Spratt, Monash University, Australia; Paul Lajbcygier, Monash University, Australia Information Science Reference 2009 Copyright Pages: 339 H/C (ISBN: 978-1-60566-410-1) Our Price: $195.00 E-Learning Technologies and Evidence-Based Assessment Approaches provides a variety of contemporary solutions to identified educational problems related to the assessment of student learning in e-learning environments. This book draws on research and evaluation expertise of academicians engaged in the day-to-day challenges of using e-learning technologies and presents key issues in peer assessment using advanced technologies. Information Communication Technologies for Enhanced Education and Learning: Advanced Applications and Developments Edited By: Lawrence A. Tomei, Robert Morris University, USA H/C (ISBN: 978-1-60566-150-6) Information Science Reference 2008 Copyright Pages: 394 Our Price: $195.00 Information Communication Technologies for Enhanced Education and Learning: Advanced Applications and Developments represents a unique examination of technology-based design, development, and collaborative tools for the classroom. Covering advanced topics in e-pedagogy, online learning, and virtual instruction, this book contributes high quality research for addressing technological integration in the classroom – a must-have for 21st century academicians, students, educational researchers, and practicing teachers. Adapting Information and Communication Technologies for Effective Education Edited By: Lawrence A. Tomei, Robert Morris University, USA H/C (ISBN: 978-1-59904-922-9) Information Science Reference 2008 Copyright Pages: 334 Our Price: $180.00 Adapting Information and Communication Technologies for Effective Education addresses ICT assessment in universities, student satisfaction in management information system programs, factors that impact the successful implementation of a laptop program, student learning and electronic portfolios, and strategic planning for e-learning. Providing innovative research on several fundamental technology-based initiatives, this book will make a valuable addition to every reference library. The Advances in Information and Communication Technology Education (AICTE) Book Series serves as a medium for introducing, collaborating, analyzing, synthesizing, and evaluating new and innovative contributions to the theory, practice, and research of technology education applicable to K-12 education, higher education, and corporate and proprietary education. The series aims to provide cross-disciplinary findings and studies that emphasize the engagement of technology and its influence on bettering the learning process. This series seeks to address the pitfalls of the discipline in its inadequate quantifiable and qualitative validation of successful learning outcomes.
Order Online at www.igi-global.com or call 717-533-8845 x100 -- Mon-Fri 8:30 AM - 5:00 PM (EST) or Fax 24 Hours a Day 717-533-9661
To Madeleine and Benjamin and MS
Editorial Advisory Board Di Challis, Challis Consulting, Australia Dale Holt, Deakin University, Australia Jennifer Weir, Murdoch University, Australia
List of Reviewers Christine Armatas, Victoria University, Australia Robyn Benson, Monash University, Australia Charlotte Brack, Monash University, Australia Rodney Carr, Deakin University, Australia Geoffrey Chow, Swinburne University, Australia Bernard Colbert, Telstra Corporation Ltd., Australia Greg Duncan, Monash University, Australia Gerald Farrell, La Trobe University, Australia Jan Fermelis, Deakin University, Australia Patrick Flanagan, Deakin University, Australia John Hurst, Monash University, Australia Kelvin Jackson, The University of Tasmania, Australia Piet Kommers, University of Twente, The Netherlands Paul Lajbcygier, Monash University, Australia Selby Markham, Monash University, Australia Stuart Rohan Palmer, Deakin University, Australia Anthony Saliba, Charles Sturt University, Australia Andrew Sanford, Monash University, Australia Stephen Segrave, Deakin University, Australia Mark Smithers, La Trobe University, Australia Christine Spratt, Royal Australian and New Zealand College of Psychiatrists, Australia Svetoslav Stoyanov, University of Twente, The Netherlands Debbi Weaver, Swinburne University, Australia Jennifer Weir, Murdoch University, Australia Joachim Wetterling, University of Twente, The Netherlands Paul White, Monash University, Australia
Table of Contents
Foreword . .......................................................................................................................................... xvii Preface . ............................................................................................................................................... xix Acknowledgment . ............................................................................................................................ xxix
Chapter I Re-Assessing Validity and Reliability in the E-Learning Environment.................................................. 1 Selby Markham, Monash University, Australia John Hurt, Monash University, Australia Chapter II Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course............. 20 Päivi Hakkarainen, University of Lapland, Finland Tarja Saarelainen, University of Lapland, Finland Heli Ruokamo, University of Lapland, Finland Chapter III Collaborative E-Learning Using Wikis: A Case Report........................................................................ 37 Charlotte Brack, Monash University, Australia Chapter IV Learning and Assessment with Virtual Worlds...................................................................................... 55 Mike Hobbs, Anglia Ruskin University, UK Elaine Brown, Anglia Ruskin University, UK Marie Gordon, Anglia Ruskin University, UK Chapter V A Faculty Approach to Implementing Advanced, E-Learning Dependent, Formative and Summative Assessment Practices.......................................................................................................... 76 Paul White, Monash University, Australia Greg Duncan, Monash University, Australia
Chapter VI Ensuring Security and Integrity of Data for Online Assessment........................................................... 97 Christine Armatas, Victoria University, Australia Bernard Colbert, Telstra Corporation Ltd., Australia Chapter VII Issues in Peer Assessment and E-Learning.......................................................................................... 117 Robyn Benson, Monash University, Australia Chapter VIII The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings................. 136 Paul Lajbcygier, Monash University, Australia Christine Spratt, Royal Australian and New Zealand College of Psychiatrists, Australia Chapter IX Validation of E-Learning Courses in Computer Science and Humanities: A Matter of Context......... 151 Robert S. Friedman, New Jersey Institute of Technology, USA Fadi P. Deek, New Jersey Institute of Technology, USA Norbert Elliot, New Jersey Institute of Technology, USA Chapter X Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments....................................................................................................................................... 170 Richard Tucker, Deakin University, Australia Jan Fermelis, Deakin University, Australia Stuart Palmer, Deakin University, Australia Chapter XI Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students................................................................................................................................................ 195 Andrew Sanford, Monash University, Australia Paul Lajbcygier, Monash University, Australia Christine Spratt, Royal Australian and New Zealand College of Psychiatrists, Australia Chapter XII Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods? ..... 218 Christine Armatas, Victoria University, Australia Anthony Saliba, Charles Sturt University, Australia Chapter XIII Evaluation Strategies for Open and Distributed Learning Environments........................................... 234 Thomas C. Reeves, University of Georgia, USA John G. Hedberg, Macquarie University, Australia
Chapter XIV Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program in Distance and Online Education................................................................................................................................. 243 Madhumita Bhattacharya, Massey University, New Zealand Chapter XV Practical Strategies for Assessing the Quality of Collaborative Learner Engagement........................ 254 John LeBaron, Western Carolina University, USA Carol Bennett, WRESA Elementary & Middle Grades Curriculum Coordinator, USA Chapter XVI Afterword: Learning-Centred Focus to Assessment Practices............................................................. 270 Compilation of References................................................................................................................ 272 About the Contributors..................................................................................................................... 301 Index.................................................................................................................................................... 307
Detailed Table of Contents
Foreword . .......................................................................................................................................... xvii Preface . ............................................................................................................................................... xix Acknowledgment . ............................................................................................................................ xxix
Chapter I Re-Assessing Validity and Reliability in the E-Learning Environment.................................................. 1 Selby Markham, Monash University, Australia John Hurt, Monash University, Australia In this opening chapter Selby Markham and John Hurst draw on their extensive experience in educational psychology and their many years teaching, researching and collaborating with the “Computers in Education Research Group” at one of Australia’s most influential universities. The approach they have taken in the chapter is also informed by a number of interviews which they undertook with practicing university teachers to assist them in designing and developing the arguments of the chapter. In acknowledging the central role of validity and reliability in any assessment system, they first remind us of the principles underpinning these key concepts and take us on a brief historical journey through the most influential literature in the field—it is no surprise that this draws heavily on work in school education and the psychometrics of educational measurement. The chapter suggests that because the e-learning environment creates new ways of being and interacting for teachers and learners (in what they call a socio-technical pedagogical environment) it ought to allow us to re-assess notions of validity and reliability in e-assessment. They introduce the idea of “knowledge validity” and argue that educational systems may need to create ways to educate students about acceptable standards of engaging with the extensive information sources available to them. They do not argue that our traditional notions of validity and reliability are outmoded, rather what they are suggesting is that the e-learning technologies and tools that are informing how we create e-learning environments necessarily calls on us as “good teachers” to be “good learners”; that is to be self-reflective and critical about our own assessment practices.
Chapter II Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course............. 20 Päivi Hakkarainen, University of Lapland, Finland Tarja Saarelainen, University of Lapland, Finland Heli Ruokamo, University of Lapland, Finland Päivi Hakkarainen and her colleagues from the University of Lapland have developed an empirically-based framework of “meaningful learning” (the model for teaching and meaningful learning: TML) based on several years’ collaborative research. In this chapter they explore the use of the assessment framework for a particular subject in the social sciences at their home institution. In keeping with a commitment to authentic, active learning, the devised model while unique draws on familiar theories from well known educational intellectuals in particular Ausbel, Dewey and more recently Jonassen. For the specific course investigated to inform the chapter, the authors used a range of e-learning supported media in particular digital videos within a case-based pedagogical and authentic assessment approach. The TML model was used as the theoretical assessment framework. While the chapter does not describe the impact of the e-learning environment on student learning outcomes or related assessment specifically, we believe it presents us with convincing evidence that well designed e-learning strategies, including authentic assessment implemented in the context of a holistic approach to course or subject design, promotes effective learning. Furthermore, it emphasises the value that students place on learning that reflects or simulates authentic real-world experiences that they may anticipate in their working lives. Chapter III Collaborative E-Learning Using Wikis: A Case Report........................................................................ 37 Charlotte Brack, Monash University, Australia Charlotte Brack writes this chapter in the context of a large medical school in a major Australian university. Students typically are well motivated and highly intelligent yet heterogeneous. As is the case internationally, many students come to their undergraduate medical studies in Australia from abroad and a significant number from a school experience where English is not their first language and where they may not have undertaken any secondary school level science subjects. The chapter presents an innovative program of study, conducted over an intensive three week period. It was devised using Web 2.0 technologies (Wikis) to prepare students who have not completed year 12 biology, for their first year of medical studies. The program is voluntary and the authors have made significant attempts to engage and motivate students who have no compelling requirements to attend aside form their own interest in being well prepared for demands of the formal program which occurs later. For us the chapter presents a case study of engaging educational design and innovative assessment, albeit formative and informal. Importantly she argues that the use of social software assisted in transition issues for these students who were new to the socio-cultural and political setting in which they were to study in Australia. Certainly one can see in this case useful potential applications in formal university programs. Chapter IV Learning and Assessment with Virtual Worlds...................................................................................... 55 Mike Hobbs, Anglia Ruskin University, UK Elaine Brown, Anglia Ruskin University, UK Marie Gordon, Anglia Ruskin University, UK
Mike Hobbs and his colleagues Elaine Brown and Marie Gordon have been experimenting with the educational potential of virtual worlds in particular “Second Life” for some years. In this chapter they introduce us to the nature of the environment and the constructivist cognitive approach to learning that it supports. They draw extensively on several case studies of work-in-progress in their undergraduate program at Anglia Ruskin University in the UK. They argue that the virtual world is particularly suitable to collaborative and peer directed learning and assessment opportunities. As such it seems an extremely “authentic” environment for students to engage in learning and assessment activities. Moreover, the work that they present was structured to enable increased ownership, indeed, design, of the learning and assessment experiences by the learners themselves. The chapter presents evidence that that students’ engagement in the virtual world has supported the development of important generic skills in group work, project management and problem solving which of course ought to be readily transferable to across other learning environments students will be involved in their studies and working lives. They suggest that loosely specified assessments with suitable scaffolding, within the rich environment of “Second Life”, can be used to help students develop as independent, self motivated learners. While the chapter reports promising findings and postulates future trends, one can see that the design and development of longitudinal studies of student learning and assessment in these virtual worlds would be valuable. Chapter V A Faculty Approach to Implementing Advanced, E-Learning Dependent, Formative and Summative Assessment Practices.......................................................................................................... 76 Paul White, Monash University, Australia Greg Duncan, Monash University, Australia Paul White and Greg Duncan and a number of their colleagues have spent the past five years using a Faculty-based learning and technology committee to drive quality improvement approaches in teaching, learning and assessment at what is one of Australia’s largest and most diverse pharmacy departments; there are over 1,000 undergraduate students, a large cohort of who are international students. Using the Faculty Teaching and Learning Technologies Committee as an organisational impetus for change, they have effectively created considerable transformation in a Faculty with hitherto quite traditional approaches to teaching and assessment. Most recently, the authors and their colleagues have used an audience response system to increase the level of formative assessment that occurs during lectures to large cohorts. The audience response system sends a radiofrequency signal via USB receivers to the lecture theatre computers, with the proprietary software allowing computation of input data. This data is then recorded within the software, and instantaneously produces a summary histogram in the PowerPoint file being used to show the questions. The chapter also presents an overview of the use of new technologies in a blended learning approach where the use of an institutional learning management system has been complemented by technologies and software such as Skype and Web-based video-conferencing to support distributed learning and assessment in postgraduate education. In recognizing that the range of new technologies available to universities is substantial; they argue that the best results are achieved by selecting options that meet teaching needs. The challenge for the future in terms of implementation is to encourage diversity and at the same time deploy those technologies that have been trialed successfully in as many suitable contexts as possible.
Chapter VI Ensuring Security and Integrity of Data for Online Assessment........................................................... 97 Christine Armatas, Victoria University, Australia Bernard Colbert, Telstra Corporation Ltd., Australia While there are fairly general processes for establishing student identification for examination purposes in face-to-face settings, Christine Armatas and Bernard Colbert argue that identification and verification matters remain one the biggest challenges to the widespread adoption of e-learning assessment strategies especially for high stakes summative assessment. Their chapter pursues the latest technologies and research advances in the field. Usefully, they discuss these often complex technologies in the milieu of a large e-learning unit taught at a major distance education university. When one is confronted with over 1,000 learners dispersed geographically and temporally, and who are studying in a “fully online” environment, then the assessment challenges demand innovative and critical thinking. Currently, as the authors argue, there are considerable limitations on assurances for identification and verification of learners who may be undertaking online assessment in such a setting. Consequently, the authors are eager for newer and developing technologies such as public key cryptography and a “network in a box” which Armatas and Colbert describe, so that we can continue to innovate in the field. Chapter VII Issues in Peer Assessment and E-Learning.......................................................................................... 117 Robyn Benson, Monash University, Australia The increasing interest in collaborative learning and assessment that the new technologies encourage was notable in the literature as the authors prepared the text. The authors decided that there was enough interest in the field to warrant the inclusion of a more theoretical chapter addressing the implications of peer assessment in e-learning environments. Robyn Benson has an extensive background as an educational designer in open and distance learning and brings to the chapter her insights from many years preparing off-campus learners to be both independent and collaborative learners. Her chapter takes a pragmatic and evidence-based approach and addresses a number of key issues in the use of e-learning tools and environments for implementing peer assessment. She begins by differentiating peer assessment for learning and peer assessment of learning and considers that the singular challenge for successful design of peer assessment focuses on the characteristics and requirements of teachers and students as users. Importantly she highlights as Markham and Hurst have in Chapter I, that the capacities offered by advanced assessment technologies may force us to reconceptualise the way in which evidence used for peer assessment of learning is presented and judged. Chapter VIII The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings................. 136 Paul Lajbcygier, Monash University, Australia Christine Spratt, Royal Australian and New Zealand College of Psychiatrists, Australia The central concern of this chapter is group assessment in an e-learning environment. The chapter provides a pragmatic example of using a research study as an avenue to debate some of the issues raised by Markham & Hurst and Benson earlier. While the chapter’s underpinning pedagogy was not about peer assessment
per se, the research did investigate the way in which learners in an e-learning environment collaborated on a group project, part of the formal assessment requirements for a particular unit of study in financial computation. The underpinning research measured individual students’ contributions to group processes, individual students’ influence on their peers’ topic understanding of the related curriculum content, and the influence of the overall group experience on personal learning in an e-learning environment designed to act as a catalyst for the group learning. As well, the learning objectives fundamental to the project work were tested individually as part of the final examination. The chapter comments on the relationship that may exist between students’ perceptions of the e-learning environment, the group project work and e-learning group dynamics. The authors conclude that e-learning environments of themselves won’t be successful in the absence of excellent and innovative educational design and this view is evident across several other chapters. The authors also wonder based on their findings, whether more energy ought to be spent on designing effective and efficient group learning opportunities rather than necessarily assessing them. Chapter IX Validation of E-Learning Courses in Computer Science and Humanities: A Matter of Context......... 151 Robert S. Friedman, New Jersey Institute of Technology, USA Fadi P. Deek, New Jersey Institute of Technology, USA Norbert Elliot, New Jersey Institute of Technology, USA Friedman, Deek, and Elliot explore the evaluation of student performance in e-learning settings. The authors believe the chapter offers a potentially useful frame of reference for us as we think more holistically about the assessment of student learning outcomes as one part of the teaching and learning puzzle. Hakkarainen and her colleagues in Chapter II provide a similar broader perspective on assessment and pedagogy. Friedman and his colleagues here want to know why students at the New Jersey Institute of Technology often do not “persist” in their e-learning programs; the authors recognised that seeing the “why” demanded a research gaze through multiple lenses. An interesting aspect of their work in this investigation of persistence is that the data is derived from two quite different disciplines, computer science and humanities. Moreover, the “why” seems to have presented them with some interesting and perhaps unanticipated findings. The chapter does many things; for example it prompts us to think critically about pedagogical research design; it forces us to rethink ideas about how the “variables” that effect learning (e.g. learning styles, instructor teaching style, interaction, course structure and assessment design) might be better integrated to assist in learning design strategies. Moreover, it provides some compelling evidence to take the development of information literacy skills seriously for as they suggest how can E-Learning and assessment benefit students if they are lacking the essential skills in the first place? Chapter X Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments....................................................................................................................................... 170 Richard Tucker, Deakin University, Australia Jan Fermelis, Deakin University, Australia Stuart Palmer, Deakin University, Australia
The authors know from the extensive literature in team-based or group assessment that students and indeed teachers are often skeptical regarding the purpose of team assignments and indeed their reliability and validity over time. Teachers, for example, often find it difficult to reconcile their interest in giving learners experiences in group learning and peer assessment with their worry that such approaches are not perceived by students as “fair”. Like Lajbcygier and Spratt in this chapter, many teachers are concerned whether the group marks they award in such settings truly reflect individual learning outcomes. Learners on the other hand don’t trust their peers to “pull their weight” and resent what the literature often calls “freeloaders” who may do little to contribute meaningfully to the group task but seem to be “rewarded” with an undifferentiated group mark. In this chapter Tucker, Fermelis & Palmer present work based on four years of research, testing and development of an online self-and-peer continuous assessment tool originally developed for small classes of architecture students. The authors argue that the e-learning tool promotes independent, reflective, critical learning, to enhance in students the motivation for participation and to encourage students to take responsibility for their learning. The findings of their pilot studies support the positive contribution of online self-and-peer assessment within student groupbased assignments. Chapter XI Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students................................................................................................................................................ 195 Andrew Sanford, Monash University, Australia Paul Lajbcygier, Monash University, Australia Christine Spratt, Royal Australian and New Zealand College of Psychiatrists, Australia Sanford and his colleagues have considerable experience in teaching courses in accounting and finance which many students often see as complex and difficult. The student cohort at their institution, like all of ours, is heterogeneous, including the cultural background of the students. They report a case that used differential item functioning analysis based on attributes of the student cohort that are unobserved. The investigation revealed that the bias associated with the differential item functioning was related to the a priori background knowledge that students bring to the unit. This is extremely interesting work especially given the diversity on our campuses. While the nature of the research and analysis is quite specialised the implications of the work for the design of multiple choice examination test items for instance, is valuable, and the implications for course design, remediation processes or indeed identifying learning needs prior to the commencement of formal studies in e-learning contexts seems promising. Chapter XII Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods? ..... 218 Christine Armatas, Victoria University, Australia Anthony Saliba, Charles Sturt University, Australia In his foreword, Gary Poole reminds us of the early debates around the “no significant difference phenomenon”— did the use of educational technologies have any impact (positive or otherwise) on student learning outcomes? Here in this final chapter, Armatas and Saliba present us with empirical evidence from a laboratory-based research study that compared the attainment of learning outcomes from four sources, “smart” mobile phones, print-based learning resources, a traditional lecture format, and a com-
puter. Those of us with a background in traditional print-based distance education will smile wryly that print seemed to have the upper hand! Their work demonstrates that learning outcomes are similar when students study by using a computer, mobile phone or lecture format, but that studying with print material yields slightly superior test results. Like all good researchers, the authors recognise the limitations of the research design and the impact of experimental artifacts, in particular whether their self-reported computer “savvy” participants needed more practice in using the mobile phone prior to undertaking the project. They do argue that mobile or m-learning is becoming of increasing interest across the higher education sector as like other technologies, changes and advances in the field are progressing at a rapid rate. For those of us prepared to innovate and take pedagogical risks in learning and assessment design, it is easy to anticipate the potential of m-learning for workplace-based learning and assessment specifically. Chapter XIII Evaluation Strategies for Open and Distributed Learning Environments........................................... 234 Thomas C. Reeves, University of Georgia, USA John G. Hedberg, Macquarie University, Australia This chapter is focused on recommending a set of practical strategies for evaluating open and distributed learning environments. Chapter XIV Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program in Distance and Online Education................................................................................................................................. 243 Madhumita Bhattacharya, Massey University, New Zealand This chapter presents a description and analysis of salient issues related to the development of an integrated e-portfolio application implemented at Massey University to help students track and accumulate evidence of skills developed over their period of study, particularly associated with the three core papers in the program. The Web-based e-portfolio project was initiated to help students provide evidence required by employers and research supervisors in a progressive and reflective manner by identifying the links across different papers and demonstrating their own conceptual understanding. Administrative issues are discussed, as well as considerations for future developments based on the experiences of this study. Chapter XV Practical Strategies for Assessing the Quality of Collaborative Learner Engagement........................ 254 John LeBaron, Western Carolina University, USA Carol Bennett, WRESA Elementary & Middle Grades Curriculum Coordinator, USA Teachers and designers of computer-networked settings increasingly acknowledge that active learner engagement poses unique challenges, especially for instructors weaned on traditional site-based teaching, and that such engagement is essential to the progressive construction of learner knowledge. “Learner engagement” can mean several things: engagement with material, engagement with instructors, and, perhaps most important, peer engagement. Many teachers of computer-networked courses, who are quite diligent about incorporating activities and procedures to promote human interactivity, are confronted with the challenge of assessing the efficacy of their efforts. How do they discern whether the strategies
and tactics woven into their “e-settings” are achieving the desired ends? This chapter outlines issues of self-assessment, including ethical questions. It lays out recommendations for self-assessment in a manner that respects student trust and confidentiality, distinguishing the demands of practical self-assessment from scholarly course research. The institutional pressures from which such assessment emerges are also examined. Chapter XVI Afterword: Learning-Centred Focus to Assessment Practices............................................................. 270 Compilation of References................................................................................................................ 272 About the Contributors..................................................................................................................... 301 Index.................................................................................................................................................... 307
xvii
Foreword
One pervasive challenge in educational research is that learning is often hard to measure. In research meetings with colleagues in which we try to articulate measurable learning outcomes, we have at times wished we were teaching students to tie their shoes. Now there is a learning outcome you can see. You can assess the integrity of the knot. You can time the knot-tying process. You can even undo the knot to better analyse it. Higher education features some learning outcomes that have the overt properties of shoe tying, but many outcomes are far more difficult to operationalize. This is particularly true of e-learning environments. Even if such environments were designed to teach shoe tying, and if students could provide virtual examples of tied shoes, how can we know that the student did the tying and didn’t simply use www.tieupyourshoes.com or Google “shoe tying?” These are thorny issues, but they must be addressed because we can no longer teach subjects like the fundamentals of biology or nursing skills in purely small group, face-to-face environments. Furthermore, there are other skills, such as those associated with managing human networks, that may be taught more effectively using online environments and we need to be able to assess this possibility. Herein lays the value of E-Learning Technologies and Evidence-Based Assessment Approaches. Here, we are introduced to research intended to identify important learning outcomes in e-learning environments and assess them validly and reliably. We desperately need the methods presented in this book. Early research in online learning revealed what has come to be known as “the no significant differences phenomenon.” When comparing online learning to face-to-face learning of the same material, neither environment proved superior. Critics argued that these studies were assessing the wrong things. For example, if an online learning environment claimed to facilitate social construction of knowledge (as is the case with the use of Second Life), it made little sense to use multiple choice tests to measure that which has been socially constructed. Rather, it is necessary to measure the processes of social construction as much as the outcomes of it. Thus, there are chapters in E-Learning Technologies and Evidence-Based Assessment Approaches presenting much-improved methods for assessing learning processes, both individual and social, as much as for assessing learning outcomes. See, for example, Hakkarainen’s chapter on “meaningful learning,” and Brack’s on the biology bridging program for incoming medical students. In the bridging program, wiki spaces provide the virtual equivalent of a well-tied pair of shoes. Process is important, but there are programs for which an assessment of outcomes is essential. For these, the use of technologies such as audience response systems is very promising. Again, the issues of validity and reliability introduced by Markham and Hurst in Chapter I are very relevant, in that the value of such response systems is only realized when the right questions are asked and good tasks are assigned.
xviii
By “the right questions,” I mean those that are valid probes into what students come into a learning environment with (knowledge and mis-conceptions), and what they develop over the course of time. This process of moving from misconception to reconception is crucial for students in higher education. In E-Learning Technologies and Evidence-Based Assessment Approaches, the important point is made that assessment drives learning. The message that “We evaluate that which we value” is not lost on our students. Thus, their work as learners is shaped, in large part, by the assessment strategies a course offers, and our work as educators is informed by the data that such strategies yield. We simply must get this right, and E-Learning Technologies and Evidence-Based Assessment Approaches, helps us do just that. Gary Poole University of British Columbia, Canada
Dr. Gary Poole is the Director of the Centre for Teaching and Academic Growth at the University of British Columbia. In addition to his work at UBC, he facilitates workshops around the world. Gary has won a 3M Teaching Fellowship, which is a Canadian national teaching award, an Excellence in Teaching award from Simon Fraser University, and a Queen’s Golden Jubilee Medal for contributions to Higher Education. Currently, Dr. Poole is the President of the Society for Teaching and Learning in Higher Education, Canada’s national organization dedicated to university teaching. He is also on the council of the International Consortium for Educational Development.
xix
Preface
INTRODUCTION The international quality agenda in higher education has created extensive interest in all aspects of teaching and learning in the post-secondary sector, especially in tertiary and higher education. Most OECD countries now have some form of quality agency responsible for accrediting the quality of higher education.a.While the scope of such quality agencies is broad, the assessment of student learning outcomes which contributes to the certification of institutional degrees lies at the heart of any quality system.b Academics and others engaged in post-secondary education are under considerable pressure to respond proactively to the scrutiny of quality agencies. Assessment is therefore a key concern of all stakeholders in the sector, not least teachers and learners. Along with the quality agenda, the revolution in information and communication technologies (ICT) and the exponential growth in e-learning is another factor which has increased interest in assessment systems; this has occurred concurrently with the globalization of education and the subsequent expansion of cross-border student enrolments. Rapid changes and advances in technology therefore see “emerging technologies” harnessed for educational purposes as rapidly as they appear, often in the absence of convincing empirical evidence of their efficacy. In this book we see e-learning as a generic term that refers to the use of various electronic media to deliver flexible education. It presupposes a more learner oriented approach to teaching and learning. e-learning approaches might include the use of the Web; static computer-based learning resources in the traditional classroom, or perhaps in the workplace; it also includes technologies that support learners learning at home away from their campus of enrolment. We know intuitively and we have growing research evidence that thoughtful e-learning design incorporating ICTs has the potential to enhance the student learning experience. We know less about how ICTs might also enhance related assessment systems so that they develop as transparent, valid and reliable measures of student learning outcomes and performance capability. Academic best practice also demands evidence-based (in other words research-led) learning and teaching practices. Currently dissatisfaction with aspects of the assessment of student learning outcomes is evident in both the school and post-secondary sectors. In Australia this is evidenced by a recent Australian Council for Educational Research (ACER) Reportc and a national study carried out by the University of Melbourne in 2002.d The assessment of student learning is widely recognized as an area that needs “renewal” as part of the broader interest in improving the quality of tertiary teaching and learning. While there is much activity across the sector in e-learning and assessment, there are few texts specifically related to the field. This of course reflects the fact that the field is nascent. It is our contention that assessment drives learning. Moreover that the promotion of an aligned system of learning demands that learning outcomes, teaching strategies and assessment are coherent and complementary thereby making expected learning outcomes explicit (Biggs, 1999). As such assessment approaches ought to reflect learning as multidimensional, integrated and performative and thus central to
xx
pedagogy: not peripheral and not additional. We argue that assessment is beneficial when it is continuous rather than intermittent and when it allows opportunities for timely feedback (Carless, 2007). Contemporary views of assessment also suggest that it ought to be relevant, authentic and adaptive (Gulikers, Bastiaens, & Kirschner, 2004; Gulikers, Bastiaens, & Kirschner, 2005; Herrington & Herrington, 2006; Challis, 2005), valid and reliable (Nicol, 2007), prepare learners for life-long learning (Boud & Falchicov, 2006), and importantly offer learners choice and diversity in approach. There is an extensive literature available to assist us to determine the best way to create meaningful interactive and rewarding learning experiences for learners; e-learning and blended learning environments are now well accepted as integrative strategies in the creation of such environments (Laurillard, 2002; Herrington, Reeves & Oliver, 2006; Scott, 2006). However, ICTs and the e-learning opportunities that arise from them do not in and of themselves create opportunities for innovation in assessment; rather this occurs when teachers think innovatively about the purpose of assessment and how ICTs might assist their educational goals. In light of this, the book’s concern is at the nexus of the ICT revolution, the requirement for ensuring quality in higher education and the globalization of education through various forms of e-learning as described above. The chapters within it are all concerned with an important question: How can information and communication technologies be used to improve assessment and hence the quality of educational outcomes for students? Furthermore the text aims to assist practitioners and researchers design strategies to investigate broader research questions in assessment, e-learning and pedagogy, for example: 1. 2. 3. 4.
What is the impact of social software technologies on assessment practices? Is there a need to reconsider issues of validity and reliability in e-learning and assessment? How will advanced technologies enable us to assess the readiness of students for the workplace? How will e-learning enable increased opportunities for learners to design and judge their own learning and that of their peers? 5. How will technology influence new assessment approaches? 6. What are the most efficient and effective styles of assessment in e-learning environments? 7. How should e-learning be “blended” most effectively with conventional forms of education in higher education? 8. In what ways can e-learning and developing mobile learning technologies inform pedagogical practice and research designs? 9. Does e-learning design affect the experience of student learning? 10. What formative assessment design creates meaningful learning in e-learning environments?
In designing and developing this text we were guided by our own beliefs and educational values about the purposes of assessment in higher education. We have been influenced in our thinking by a number of key researchers and policy makers in Australia and internationally; in particular the “Assessing Learning in Australian Universities Project: 2002”e headed by Professor Richard James and his colleagues at the University of Melbourne, a large project which investigated assessment practices across the Australian sector. The subsequent report and Web site has proven a valuable adjunct to quality improvement initiatives in the assessment of student learning in Australian Universities as it was “designed to support Australian universities and academic staff in maintaining high quality assessment practices, in particular in responding effectively to new issues in student assessment”. Other Australian initiatives have included various
xxi
funded projects from the Australian Learning and Teaching Council (formerly The Carrick Institute for Learning and Teaching in Higher Education)f which has supported innovation in assessment generally, in specific discipline areas, and more recently in e-learning and e-assessment. Recently in the United Kingdom, the REAP Projectg an initiative of the Scottish Funding Council (2005-2007) under its e-learning transformation programme supported three University partners, the Universities of Strathclyde (as the project leader), Glasgow and Glasgow Caledonian University to establish pilot projects to support the redesign of formative assessment and feedback practices in largeenrolment first-year modules across these three institutions. Furthermore the project aimed to design and develop useful approaches to embed creative thinking about assessment into institutional policies and quality improvement processes. The REAP website remains an open, active repository of data and resources for the international assessment community. Research and evaluation material from the project outcomes is also evident in the more recent literature in assessment and e-learning (Nicol 2008, Nicol 2007[a], Nicol 2007[b], Nicol 2007[c], Draper & Nicol 2006). These initiatives provide support for our approach which has informed the structure and content of the text; that to be efficient and effective, assessment systems have to present to the learner clear learning goals and objectives; the identification of transparent standards of expected work or performances; timely and appropriate feedback; opportunities to learn from each other and the prospect of remediation. Furthermore, as outlined earlier, assessment systems ought to reflect thoughtful pedagogical innovation as well as evidence-based or research-led approaches. In higher education, we have seen calls for more qualitative and innovative approaches to assessment beside increased managerialism and quantitative measures of performance proposed by the quality agenda described earlier. These ideas seem at odds with one another surely highlighting that assessment is one of the most challenging aspects of academic work. Consequently, the chapters in this book present novel practices and contribute to the development of an evidence-based approach to e-learning and assessment. In light of this, the book aims to provide practitioners with evidence-based examples to assist them integrate e-learning assessment practices in their pedagogical frameworks as well advance future research and development trends in the broad field of e-learning and the assessment of student learning outcomes. It is evident from the chapters in the text that we have taken an eclectic view of assessment as we attempt to present a range of exciting and interesting techniques in the contemporary applications of assessment technologies and approaches. There are chapters that also describe the strategies practitioners are using to appraise, judge and evaluate their approaches as well as the design and development of research strategies to evaluate student performance in e-learning settings. Moreover, the text also presents various approaches to finding out from students themselves what they think about the courses they are engaged in and the assessment tasks they are expected to undertake to demonstrate they have ‘learnt’. In the opening chapter Selby Markham and John Hurst draw on their extensive experience in educational psychology and their many years teaching, researching and collaborating with the “Computers in Education Research Group” at one of Australia’s most influential universities. The approach they have taken in the chapter is also informed by a number of interviews which they undertook with practicing university teachers to assist them in designing and developing the arguments of the chapter. In acknowledging the central role of validity and reliability in any assessment system, they first remind us of the principles underpinning these key concepts and take us on a brief historical journey through the most influential literature in the field—it is no surprise that this draws heavily on work in school education and the psychometrics of educational measurement. The chapter suggests that because the elearning environment creates new ways of being and interacting for teachers and learners (in what they call a socio-technical pedagogical environment) it ought to allow us to re-assess notions of validity and reliability in e-assessment. They introduce the idea of “knowledge validity” and argue that educational
xxii
systems may need to create ways to educate students about acceptable standards of engaging with the extensive information sources available to them. They do not argue that our traditional notions of validity and reliability are outmoded, rather what they are suggesting is that the e-learning technologies and tools that are informing how we create e-learning environments necessarily calls on us as “good teachers” to be “good learners”; that is to be self-reflective and critical about our own assessment practices. In Chapter II, Päivi Hakkarainen and her colleagues from the University of Lapland have developed an empirically-based framework of “meaningful learning” (the model for teaching and meaningful learning: TML) based on several years’ collaborative research. In this chapter they explore the use of the assessment framework for a particular subject in the social sciences at their home institution. In keeping with a commitment to authentic, active learning, the devised model while unique draws on familiar theories from well known educational intellectuals in particular Ausbel, Dewey and more recently Jonassen. For the specific course investigated to inform the chapter, the authors used a range of e-learning supported media in particular digital videos within a case-based pedagogical and authentic assessment approach. The TML model was used as the theoretical assessment framework. While the chapter does not describe the impact of the e-learning environment on student learning outcomes or related assessment specifically, we believe it presents us with convincing evidence that well designed e-learning strategies, including authentic assessment implemented in the context of a holistic approach to course or subject design, promotes effective learning. Furthermore, it emphasises the value that students place on learning that reflects or simulates authentic real-world experiences that they may anticipate in their working lives. Charlotte Brack writes this chapter in the context of a large medical school in a major Australian university. Students typically are well motivated and highly intelligent yet heterogeneous. As is the case internationally, many students come to their undergraduate medical studies in Australia from abroad and a significant number from a school experience where English is not their first language and where they may not have undertaken any secondary school level science subjects. The chapter presents an innovative program of study, conducted over an intensive three week period. It was devised using Web 2.0 technologies (Wikis) to prepare students who have not completed year 12 biology, for their first year of medical studies. The program is voluntary and the authors have made significant attempts to engage and motivate students who have no compelling requirements to attend aside form their own interest in being well prepared for demands of the formal program which occurs later. For us the chapter presents a case study of engaging educational design and innovative assessment, albeit formative and informal. Importantly she argues that the use of social software assisted in transition issues for these students who were new to the socio-cultural and political setting in which they were to study in Australia. Certainly one can see in this case useful potential applications in formal university programs. Mike Hobbs and his colleagues Elaine Brown and Marie Gordon in Chapter IV, have been experimenting with the educational potential of virtual worlds in particular “Second Life” for some years. In this chapter they introduce us to the nature of the environment and the constructivist cognitive approach to learning that it supports. They draw extensively on several case studies of work-in-progress in their undergraduate program at Anglia Ruskin University in the UK. They argue that the virtual world is particularly suitable to collaborative and peer directed learning and assessment opportunities. As such it seems an extremely “authentic” environment for students to engage in learning and assessment activities. Moreover, the work that they present was structured to enable increased ownership, indeed, design, of the learning and assessment experiences by the learners themselves. The chapter presents evidence that that students’ engagement in the virtual world has supported the development of important generic skills in group work, project management and problem solving which of course ought to be readily transferable to across other learning environments students will be involved in their studies and working lives. They suggest that loosely specified assessments with suitable scaffolding, within the rich environment of “Second Life”, can be used to help students develop as independent, self motivated learners. While
xxiii
the chapter reports promising findings and postulates future trends, one can see that the design and development of longitudinal studies of student learning and assessment in these virtual worlds would be valuable. Paul White and Greg Duncan and a number of their colleagues have spent the past five years using a Faculty-based learning and technology committee to drive quality improvement approaches in teaching, learning and assessment at what is one of Australia’s largest and most diverse pharmacy departments; there are over 1,000 undergraduate students, a large cohort of who are international students. In Chapter V, using the Faculty Teaching and Learning Technologies Committee as an organisational impetus for change, they have effectively created considerable transformation in a Faculty with hitherto quite traditional approaches to teaching and assessment. Most recently, the authors and their colleagues have used an audience response system to increase the level of formative assessment that occurs during lectures to large cohorts. The audience response system sends a radiofrequency signal via USB receivers to the lecture theatre computers, with the proprietary software allowing computation of input data. This data is then recorded within the software, and instantaneously produces a summary histogram in the PowerPoint file being used to show the questions. The chapter also presents an overview of the use of new technologies in a blended learning approach where the use of an institutional learning management system has been complemented by technologies and software such as Skype and Web-based video-conferencing to support distributed learning and assessment in postgraduate education. In recognizing that the range of new technologies available to universities is substantial; they argue that the best results are achieved by selecting options that meet teaching needs. The challenge for the future in terms of implementation is to encourage diversity and at the same time deploy those technologies that have been trialed successfully in as many suitable contexts as possible. While there are fairly general processes for establishing student identification for examination purposes in face-to-face settings, in Chapter VI, Christine Armatas and Bernard Colbert argue that identification and verification matters remain one the biggest challenges to the widespread adoption of e-learning assessment strategies especially for high stakes summative assessment. Their chapter pursues the latest technologies and research advances in the field. Usefully, they discuss these often complex technologies in the milieu of a large e-learning unit taught at a major distance education university. When one is confronted with over 1,000 learners dispersed geographically and temporally, and who are studying in a “fully online” environment, then the assessment challenges demand innovative and critical thinking. Currently, as the authors argue, there are considerable limitations on assurances for identification and verification of learners who may be undertaking online assessment in such a setting. Consequently, the authors are eager for newer and developing technologies such as public key cryptography and a “network in a box” which Armatas and Colbert describe, so that we can continue to innovate in the field. In Chapter VII the increasing interest in collaborative learning and assessment that the new technologies encourage was notable in the literature as the authors prepared the text. The authors decided that there was enough interest in the field to warrant the inclusion of a more theoretical chapter addressing the implications of peer assessment in e-learning environments. Robyn Benson has an extensive background as an educational designer in open and distance learning and brings to the chapter her insights from many years preparing off-campus learners to be both independent and collaborative learners. Her chapter takes a pragmatic and evidence-based approach and addresses a number of key issues in the use of e-learning tools and environments for implementing peer assessment. She begins by differentiating peer assessment for learning and peer assessment of learning and considers that the singular challenge for successful design of peer assessment focuses on the characteristics and requirements of teachers and students as users. Importantly she highlights as Markham and Hurst have in Chapter I, that the capacities offered by advanced assessment technologies may force us to reconceptualise the way in which evidence used for peer assessment of learning is presented and judged.
xxiv
The central concern of this chapter is group assessment in an e-learning environment. Chapter VIII provides a pragmatic example of using a research study as an avenue to debate some of the issues raised by Markham & Hurst and Benson earlier. While the chapter’s underpinning pedagogy was not about peer assessment per se, the research did investigate the way in which learners in an e-learning environment collaborated on a group project, part of the formal assessment requirements for a particular unit of study in financial computation. The underpinning research measured individual students’ contributions to group processes, individual students’ influence on their peers’ topic understanding of the related curriculum content, and the influence of the overall group experience on personal learning in an e-learning environment designed to act as a catalyst for the group learning. As well, the learning objectives fundamental to the project work were tested individually as part of the final examination. The chapter comments on the relationship that may exist between students’ perceptions of the e-learning environment, the group project work and e-learning group dynamics. The authors conclude that e-learning environments of themselves won’t be successful in the absence of excellent and innovative educational design and this view is evident across several other chapters. The authors also wonder based on their findings, whether more energy ought to be spent on designing effective and efficient group learning opportunities rather than necessarily assessing them. Friedman, Deek, and Elliot explore the evaluation of student performance in e-learning settings in Chapter IX. The authors believe the chapter offers a potentially useful frame of reference for us as we think more holistically about the assessment of student learning outcomes as one part of the teaching and learning puzzle. Hakkarainen and her colleagues in Chapter II provide a similar broader perspective on assessment and pedagogy. Friedman and his colleagues here want to know why students at the New Jersey Institute of Technology often do not “persist” in their e-learning programs; the authors recognised that seeing the “why” demanded a research gaze through multiple lenses. An interesting aspect of their work in this investigation of persistence is that the data is derived from two quite different disciplines, computer science and humanities. Moreover, the “why” seems to have presented them with some interesting and perhaps unanticipated findings. The chapter does many things; for example it prompts us to think critically about pedagogical research design; it forces us to rethink ideas about how the “variables” that effect learning (e.g. learning styles, instructor teaching style, interaction, course structure and assessment design) might be better integrated to assist in learning design strategies. Moreover, it provides some compelling evidence to take the development of information literacy skills seriously for as they suggest how can e-learning and assessment benefit students if they are lacking the essential skills in the first place? The authors of Chapter X know from the extensive literature in team-based or group assessment that students and indeed teachers are often skeptical regarding the purpose of team assignments and indeed their reliability and validity over time. Teachers, for example, often find it difficult to reconcile their interest in giving learners experiences in group learning and peer assessment with their worry that such approaches are not perceived by students as “fair”. Like Lajbcygier and Spratt in this chapter, many teachers are concerned whether the group marks they award in such settings truly reflect individual learning outcomes. Learners on the other hand don’t trust their peers to “pull their weight” and resent what the literature often calls “freeloaders” who may do little to contribute meaningfully to the group task but seem to be “rewarded” with an undifferentiated group mark. In this chapter Tucker, Fermelis & Palmer present work based on four years of research, testing and development of an online self-and-peer continuous assessment tool originally developed for small classes of architecture students. The authors argue that the e-learning tool promotes independent, reflective, critical learning, to enhance in students the motivation for participation and to encourage students to take responsibility for their learning. The findings of their pilot studies support the positive contribution of online self-and-peer assessment within student group-based assignments.
xxv
Sanford and his colleagues have considerable experience in teaching courses in accounting and finance which many students often see as complex and difficult. The student cohort at their institution, like all of ours, is heterogeneous, including the cultural background of the students. In Chapter XI, they report a case that used differential item functioning analysis based on attributes of the student cohort that are unobserved. The investigation revealed that the bias associated with the differential item functioning was related to the a priori background knowledge that students bring to the unit. This is extremely interesting work especially given the diversity on our campuses. While the nature of the research and analysis is quite specialised the implications of the work for the design of multiple choice examination test items for instance, is valuable, and the implications for course design, remediation processes or indeed identifying learning needs prior to the commencement of formal studies in e-learning contexts seems promising. In his foreword, Gary Poole reminds us of the early debates around the “no significant difference phenomenon”— did the use of educational technologies have any impact (positive or otherwise) on student learning outcomes? Here in this final chapter, Armatas and Saliba present us with empirical evidence from a laboratory-based research study that compared the attainment of learning outcomes from four sources, “smart” mobile phones, print-based learning resources, a traditional lecture format, and a computer. Those of us with a background in traditional print-based distance education will smile wryly that print seemed to have the upper hand! Their work demonstrates that learning outcomes are similar when students study by using a computer, mobile phone or lecture format, but that studying with print material yields slightly superior test results. Like all good researchers, the authors recognise the limitations of the research design and the impact of experimental artifacts, in particular whether their self-reported computer “savvy” participants needed more practice in using the mobile phone prior to undertaking the project. They do argue that mobile or m-learning is becoming of increasing interest across the higher education sector as like other technologies, changes and advances in the field are progressing at a rapid rate. For those of us prepared to innovate and take pedagogical risks in learning and assessment design, it is easy to anticipate the potential of m-learning for workplace-based learning and assessment specifically.
Additiia In light of the focus of the text, we have solicited several other brief works that reflect the key themes; the assessment of student learning outcomes, using e-learning approaches innovatively and the importance of designing rationale evaluation strategies to measure the success or otherwise of e-learning assessment environments.
Reading 1: Evaluation Strategies for Open and Distributed Learning Evironments Tom Reeves and John Hedberg are well known internationally for their work in e-learning and multimedia pedagogies. Their chapter offers a very pragmatic approach to the important matter of evaluation. They are relatively critical of evaluation “models” that are over complicated or make claims about outcomes the model could never deliver; this is often the case they argue, in respect of “impact evaluation” models. Reeves and Hedberg are pragmatists; their pyramid framework is directed at assisting e-learning designers provide appropriate information to all stakeholders so that evidence-based decisions can be made during development and for improvement. The model accommodates the increasing interest in
xxvi
impact evaluation but recognizes its limitations. The two explanatory case-studies, while brief, serve to illustrate the model in action. For us, one can see the principles of their model reflected in Brack’s work (Chapter III) with undergraduate medical students and the more ‘organizational evaluation’ aspects of the work described by White & Duncan (Chapter V) in this volume.
Reading 2: Introducing Integrated E -Portfolio Across Courses in a Postgraduate Program in Distance and Online Education In his exploration of e-portfolios, Bhattacharya commences with a brief review of the historical educational uses of traditional paper-based portfolios; in particular in allowing students to collect, store and retrieve evidence of their achievement of identified learning outcomes in various settings in professional development. He links the growth in the use of e-portfolios to the continuing interest in finding ways to enable learners to take more control of their own learning and to have the capacity to store evidence in various digital formats. He draws on a case report from Massey University in New Zealand, a major university with a long and respected history in open and distance education. The e-portfolio he describes is used by students in a postgraduate program in education. The chapter highlights the benefits and pitfalls of e-portfolios and illustrates that key principles of educational design ought to inform the purpose and structure of e-portfolios and the way in which they are assessed. While the chapter reports a pilot project, the review and analysis of way his project aimed to integrate the e-portfolio across a program offers valuable insights for those who may begin to investigate the pedagogical applications.
Reading 3: Practical Strategies for Assessing the Quality of Collaborative Larner Engagement This reading explores aspects of a number of themes articulated in various chapters in this text. The recognized value of peer collaboration in learning is explored in a number of ways in the case studies presented to illustrate the benefits and challenges of self-assessment in research and evaluation of e-learning environments. In doing so, the cases illustrate the principles of research to investigate the assessment of student learning in e-learning environments and the evaluation of the efficacy of those teaching and learning environments. The authors offer useful advice about the design of e-learning environments to foster learner engagement and the measurement of the efficacy of those designs—the chapter also presents various models of research pertinent to each. Le Baron & Bennett’s work is usefully read alongside Brack’s and White & Duncan’s work in this volume (Chapter III and Chapter V respectively). Robyn Benson’s Chapter VII in this volume is also relevant given its more theoretical discussion of peer assessment.
REFERENCES Biggs (1999). Teaching for Quality Learning at University. Buckingham: SRHE and Open University Press. Boud, D., & Falchikov, N. (2006). Aligning assessment with long-term learning. Assessment and Evaluation in Higher Education, 31(4), 399-413. Carless, D. (2007). Learning-oriented assessment: Conceptual bases and practical implications. Innovations in Education and Teaching International, 44(1), 57-66.
xxvii
Challis, D. (2005). Committing to Quality Learning through Adaptive Online Assessment. Assessment and Evaluation in Higher Education, 30(5), 519-527. Gulikers, J., Bastiaens, T., & Kirschner, P. (2004). A five-dimensional framework for authentic assessment. Educational Technology Research and Development, 52(3), 67-85. Gulikers, J., Bastiaens, T., & Kirschner, P. (2005). Perceptions of authentic assessment and the impact on student learning. Paper presented at The First International Conference on Enhancing Teaching and Learning Through Assessment, Hong Kong Polytechnic University, Hong Kong, June 2005. Herrington, A., & Herrington, J. (2006). Authentic Learning Environments in Higher Education. London: Idea Group. Herrington, J., Reeves, T., & Oliver, R. (2006). Authentic tasks online: A synergy among learner, task, and technology. Distance Education, 27(2), 233–247. Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective use of learning technologies (2nd ed.). London: Routledge Falmer. Nicol, D. (2008). Transforming assessment and feedback: enhancing integration and empowerment in the first year. Scottish Quality Assurance Agency (QAA) for Higher Education. Retrieved October 7, 2008, from http://www.reap.ac.uk/resources.html Nicol, D. (2007). Laying a foundation for lifelong learning: Case studies of e-assessment in large 1styear classes. British Journal of Educational Technology, 38(4), 668–678. Nicol, D. (2007b). E-assessment by design: using multiple-choice tests to good effect. Journal of Further and Higher Education, 31(1), 53–64. Nicol, D. (2007c). Principles of good assessment and feedback: Theory and practice. From the REAP International Online Conference on Assessment Design for Learner Responsibility, 29th-31st May, 2007. Retrieved October 7, 2008, from http://www.reap.ac.uk/resources.html Draper, S., & Nicol, D. J. (2006). Transformation in E-Learning. Paper presented to the ALT-C, Conference, Edinburgh, September 5-9. Retrieved October 7, 2008, from http://www.reap.ac.uk/resources.html Scott, G. (2006). Accessing the Student Voice: Using CEQuery to identify what retains students and promotes engagement in productive learning in Australian higher education. Department of Education Science and Training, Commonwealth of Australia.
E
a
b
For example in Australia the Australian University Quality Agency (AUQA) (http://ww.auqa. edu.au); throughout Europe the European Association for Quality in Higher Education (EAQHE (http://www.enqa.eu/), in the UK in particular the Quality Assurance Agency (http://www.qaa. ac.uk/); and in the USA, the Council for Higher Education Accreditation (http://www.chea.org) which has a broad brief nationally across higher education quality and accreditation. See for example the CHEA Special Report on Accountability and Accreditation http://www.chea. org/pdf/Accreditation_and_Accountability.pdf, which has as its starting point the centrality of student learning outcomes, that is, assessment as evidence of performance for accreditation.
xxviii
c
f g d e
Reported in The Age (http://tinyurl.com/ywkeq7). The Australian Council for Education al Research (www.acer.edu.au) http://www.cshe.unimelb.edu.au/assessinglearning/index.html http://www.cshe.unimelb.edu.au/assessinglearning/index.html http://www.altc.edu.au/carrick/go/home/about http://www.reap.ac.uk/
xxix
Acknowledgment
Without the able editorial assistance of Steve Gardner, the book would be diminished although of course the final responsibility is ours. We would like to acknowledge the contribution of our many colleagues who gave their time freely to participate in the peer review process. We would also like to thank Professor Lawrence Tomei for his support of our work over the past several years. Our small Editorial Advisory Group, Dr. Di Challis, Dr. Dale Holt and Dr. Jennifer Weir also provided ad hoc advice on a number of matters at short notice. In particular we acknowledge the contribution of our friend and colleague, the late Associate Professor Malcolm Eley, who helped us see that the design of assessment practices was ultimately in the service of our students. Christine Spratt and Paul Lajbcygier Melbourne October 2008
Chapter I
Re-Assessing Validity and Reliability in the E-Learning Environment Selby Markham Monash University, Australia John Hurst Monash University, Australia
ABSTRACT Reliability and validity have a well-established place in the development and implementation of educational assessment devices. With the advent of electronic delivery and assessment some of the factors that in.uence reliability and validity have changed. In order to understand the process involved the authors have suggested that a socio-technical approach to these educational issues gives an economical explanatory system. Within this socio-technical system, the authors show that the way the students extract information from sources is changing to an extent where it is difficult to distinguish between cheating and poor quoting behavior. This has led them to postulate a new classification within validity and reliability – knowledge validity and reliability. They argue that electronic delivery and assessment have not changed their core structures, but rather require revised education and training for both staff and students.
INTRODUCTION What is validity and reliability in e-learning and assessment? To answer this question, we shall first provide an introduction to the broad concep-
tual structure that defines reliability and validity in educational assessment. We will then be in a position to begin the process of mapping this structure into the evolving world of e-learning. We shall attempt to illustrate key points through
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Re-Assessing Validity and Reliability in the E-Learning Environment
case studies from real world teaching issues so that the ideas being presented do not stand in theoretical isolation. We shall also introduce the ideas presented by a small sample of teachers who were interviewed in developing the chapter. At the core of what we shall present is the idea that e-learning, in its many forms and definitions, has created an environment where reliability and validity need to be updated from the traditional thinking. Very few Higher Education (HE) institutions have any formal requirement that staff establish the reliability and validity of their assessments. With the advent of the electronic environment, there has been a pragmatic concern for validity through the plagiarism issue. We shall explore this. Validity and reliability need to have a context within a technology enhanced assessment environment. We argue that this is best done through basic socio-technical thinking, for the simple reason that this emphasizes the importance of the interaction between the human and technical components in the educational system. Additionally, it reinforces the need to understand the transactions that take place between system elements. We will show that this helps us develop a future oriented approach to reliability and validity in e-learning systems.
Staff Reflection
Definitions The definitional framework underlying validity and reliability is important in order to provide a clear picture of what will be dealt with in this chapter. Preparatory discussion with staff suggested that most had little if any understanding of the ideas of reliability and validity, at least in education assessment terms. Below are some definitions of the major areas that are included, although there is some disagreement through differences in emphasis in published assessment and testing texts (Brennan, 2006; Carmines & Zeller, 1991; Hogan, 2007; Payne, 2003).
Reliability The extent to which an assessment device is stable over time. The extent to which the results from an assessment device are reproducible. •
a
One particularly revealing comment about the effect that e-learning has had upon assessment validity is that “conventional take away assignments bleed too much” (in the internet environment). The respondent was concerned that validity almost completely disappears once you let a student walk away with an assignment. Not only do students seek help from on-line articles (“googling the answer” is a widely understood phrase amongst e-learning students), but they also seek the services of third party “assignment sub-contractors”, who often advertise on the web, and who are almost untraceable in terms of identifying the real authors.
Ba
•
Test-retest Reliability can be measured by looking at the agreement between assessment scores at two different times, taking into account the effect of repeated exposure. The agreement is measured by a specialized correlation coefficient. Example: You extract an objective test from the item bank supplied with your text book. The students do the test after their class this week. Next week you repeat exactly the same test. You calculate a reliability coefficient to look at the test-retest reliability. Parallel forms reliability is used where there are two equivalent forms of an assessment and both forms are given to the same group of students. It is reliable if students perform similarly on both. Example: Using the item test bank from your text book, you randomly extract two sets of items that cover the same teaching
Re-Assessing Validity and Reliability in the E-Learning Environment
•
material. You give the students each test as a separate task, possibly a day apart. You calculate a reliability coefficient on the two sets of results. Split-half reliability assumes that the assessment can be split into sections with equivalent content and the performance of the students on the sections can be compared. Specialized correlation coefficients are used to measure this agreement. Example: You devise an objective test that has a good spread of subject materials evenly distributed through the test. When the students have completed the test, you turn the even numbered items into one test and the odd numbered items into another test. You then calculate a reliability coefficient on the two created tests.
•
Validity The extent to which an assessment device is assessing what it purports to assess. The extent to which the results from an assessment device represent what the learner has achieved given the objectives and outcomes of the unit/course.
•
Validity comes in a number of forms: face, content, construct, consequential and predictive.
•
•
The face validity of an assessment is simply the extent to which it looks like it is talking about what is in the curriculum. Example: You set a question on the history of the computer and ask about changes in operating systems but you use the more technical language of the test books rather than the non-technical language of your lectures. Your question may lack face validity. An assessment can have content validity where the assessment is actually measuring the content that it was assumed to be measuring in designing that assessment.
•
Example: You have asked a question about the way Freud used case studies in developing his theory of neurosis. A colleague points out that the question might not measure a student’s knowledge about the way Freud uses case studies in different stages of his work (his later work relied on case studies to a lesser extent than his earlier work). A test can have construct validity if it can be shown to be associated with a measure on a related set of material that is integral to the general construct or conceptual framework that is supposed to be being measured. Example: If you were teaching problem solving strategies, you would want to test the extent to which the students were able to select the most effective strategy. You might want to see if the results from your assessment were commensurate with the broader work on problem solving that students were doing. An assessment device can have outcome or predictive validity if it is associated with some outcome measure. An example would be that an assessment device would be related to the next assessment in the same subject area. This is similar to criterion-related validity. Example: Programming 101 is a prerequisite subject for Programming 201. You would expect that there would be a relationship between the results on assessment tasks in 101 and, at least, the initial assessments within 102. An alternative to predictive validity is consequential validity where we are concerned with both positive and negative outcomes from the assessment. An assessment may be a good predictor of subsequent performance but it may not be related to student feeling of involvement in the educational activities. Example: In presenting your lectures you have made serious comment upon the low quality of a commercial product. Your as-
Re-Assessing Validity and Reliability in the E-Learning Environment
sessment task contains nothing about this product. A student can do well on the assessment but may be naïve about the consequences of your comments and use them against himself. Figure 1 shows the structural relationship between the assessment and various measures of validity. On the left hand side is the assessment development phase where the behavior of the teacher is about the intention to assess. The right hand side addresses the outcomes, particularly the effectiveness of the assessment against the intentions. What should be noted is that this system includes feedback from the actual results of the assessment on the development of future assessment tasks. To this extent, the model assumes that this is also a learning model for the teacher. This is because the process of undertaking reliability and validity studies provides the teacher with information that can influence how they best develop teaching and learning materials. Reliability is placed in a box under all of the other components as a simple one-stage process.
All reliability measures are based upon outcome data even if that data is collected more than once – as is the case with test-retest reliability. There is no structure related to reliability that parallels validity. Reliability and validity are related in quite complex ways. In particular, a reliable assessment may not be a valid assessment. Because you are measuring something consistently over time does not imply that you are measuring what you intend to measure. Conversely, you may have a very valid assessment but it could be relatively unreliable. An example of this would be an assessment of applied skills that is measured through some experiential exercise. It is not easy to replicate such an assessment as it will be influenced by situational factors. It should not be assumed that reliability and validity are without controversy. But much of this controversy is within educational and psychological measurement rather than within the area of this book – educational performance assessment. Validity has been a topic of debate in measurement theory for many years and some of this debate is well summarized by Kane (2008) and Cizek (2008).
Figure 1. The structural relationship between reliability and validity Intention
Outcome Construct Validity
Face Validity
Design assessment
Content Validity
Do assessment
Assessment data
Outcome Validity
Consequential Validity
Reliability (all forms)
Re-Assessing Validity and Reliability in the E-Learning Environment
Hi Reliability and validity have been the touchstones of well-developed assessment tools in education and psychology. The conceptual structure for the traditional approach comes from psychometrics where psychological measurement tools had to be both valid and reliable if their results were to be taken seriously. Educational assessment analysis adopted the psychometric approach, particularly where the assessment was for formal external examinations, such as the US Scholastic Aptitude Tests (Carmines & Zeller, 1991). In these cases it was important that the tests be both reliable and valid. The testing authorities had to know that from testing session to testing session the same academic knowledge was being measured and they had to know that there was content validity. For example, what was the point of running a Mathematics test if it was not measuring mathematical ability? These were large scale objective tests that were designed to be delivered en masse. During the 1960’s, textbooks in most discipline areas would provide workbooks with multiple choice tests and/or banks of test items. Running parallel to this was the growth in computing where the key statistical packages developed specialized tools for analyzing educational tests. Also the development of test theory took place around the objective test. The best known example is probably Item Response Theory and its derivative the Rasch model, both of which are discussed in standard texts on formal assessment (e.g. Payne, 2003). The importance of objectively evaluated testing had an upsurge with equal opportunity and anti-discrimination rulings, particularly in the USA. Under these circumstances, it is important that tests results be valid across cultural groups as well as being traditionally reliable and valid (Lee, Wehmeyer, Palmer, Soukup & Little, 2008). Within higher education, the assessment tradition has varied considerably between discipline areas. For example, in an Arts Faculty much of
the assessment has been through essay-type examinations plus practical work. In Fine Arts the assessment can be 100 percent qualitative with neither essay nor multiple choice question tests used – only the submission of actual art work. In an Engineering Faculty there could be a more complex mix of practical assessment, essays and objective testing. Much of this took place with due regard to the importance of academic freedom. Consequently, in the HE sector, less emphasis has been placed on formal analysis methods for establishing reliability and validity. Given the low interest in formal educational training in appointing academic staff, we might draw the inference that it has not been assumed that the teacher has the expertise to know whether their assessment is reliable and valid. Gipps (1994) provides an overview of thinking about reliability and validity within an assessment framework but this does not include e-learning. Most other books on assessment also give a somewhat idiosyncratic view on reliability and validity but the majority utilizes the basic concepts outlined above. For example, the origins of reliability and validity in psychometrics have been quite stable but terms have tended to be slightly rewritten to give relatively minor shift in emphasis.
Assessment Tpes The various forms of assessment are obviously related to reliability and validity. This is partly based upon how reliability and validity can be ‘measured’. We shall look at objective and open assessment as well as formative and summative assessment. Those forms of validity that come at the Intention stage in Figure 1 are not subject to any of the following discussion. The face and content validity of any device is independent of the type of assessment being devised. The following discussion is about the outcome-based measures of validity and about reliability.
Re-Assessing Validity and Reliability in the E-Learning Environment
Objective Assessment and Open Assessments
when looking at qualitative evaluations in the e-learning environment.
The distinction between objective assessments and more open, essay-type assessments is obviously important in the way that we deal with reliability and validity. Principally, the use of objective assessments lets us readily use statistical tools while the more open assessment limits the type of quantified formal reporting that we can do. Any assessment that is based upon objective items that have a simple item response structure is more easily dealt within formal reliability and validity in that the standard statistical tools for assessing reliability and validity were developed to handle measurable outcomes – principally correlation coefficients. All statistical tools for assessing reliability and validity obviously require outcomes to be quantified and that this quantification fits into the requirements for parametric statistics. When we grade open or written questions the numbers we give can be suspect as metrics for the statistical analysis of reliability and validity. For example, the distribution of grading tends to be skewed rather than being symmetrically distributed across all possible grades. That is, we usually hope to fail no more than about 10% of students, making the bottom half of the possible distribution rather sparse. Technically, we can use statistics (non-parametric) that do not require strong metrics but these are not included in the usual analysis packages for reliability and validity. If we are knowledgeable enough, we might transform the data to allow it to utilize the formal analyses. Clearly we are moving outside the expertise of many lecturers. These issues arise whether we are in a traditional learning environment or in some version of an e-learning environment. Open assessments may be qualitatively evaluated through other methods. These include interviews with learners and expert ratings. We will explore this area further under Process Oriented. Suffice to say here that issues do arise
Formative and Summative
The distinction between formative and summative assessment is important in defining distinct approaches to the way student performance is evaluated. Within either type of approach various forms of assessment can be used. For example, both formative and summative assessment can use objective tests. In spite of this, some believe that formative assessment will be qualitative, so Harlen (2005) reviewed the general validity of reports on summative assessment when he was looking at qualitative work. We suggest that there is nothing in the background to formative and summative assessment that creates peculiar issues for evaluating validity and reliability. This simplifies the following discussion in this chapter.
Th E -Leaaivironment Attempts to develop pedagogies that take into account the changing educational environment are uncommon in the professional literature. Newson (1999) introduced the term ‘techno-pedagogy’ to cover the interaction between information technologies, educational technologies and the teaching environment. In the period since technopedagogy was introduced there has been little published that references it and its implications. Spodark (2005) took up the basic concept and then talked in terms of technoconstructivism which was simply using technology in a constructivist oriented classroom. Mehanna (2004) collected data from postgraduate students and their tutors in e-learning based courses and found structures in her data that were informative, but failed to specify how these patterns related to e-learning per se.
Re-Assessing Validity and Reliability in the E-Learning Environment
There is also work being done that attempts to take a wider view on this environment. Miller (1998) discusses the use of socio-technical systems theory in developing aspects of distance education. Activity theorists, whose position has some of its roots in similar thinking, have discussed the relationship between higher education and technology (e.g. Issroff & Scanlon, 2002). We would argue that there are many aspects of the changing nature of teaching and learning as it moves to more electronic support that can be seen as changes in the transactions between the teacher, the learner and the environment (Markham, 2008). It is the changing nature of these transactions that constitute the key issues for understanding reliability and validity within electronic environments. Consequently, we will frame the section within socio-technical theory (Emery & Trist, 1969; Rice, 1970). To help develop this position, we will look at three aspects of the skills and behavior of the learner (study behavior, Information and Communication Technologies (ICT) skills and self-managed learning) and the effect of these on assessment. We will also deal with the changes in the teaching environment of the teacher and the effects that this can have on assessment.
The Socio-Technical Environment As we have noted in the section on History, the formal structure of reliability and validity has its origins in the educational world of the mid20th century where paper and pencil assessment and the blackboard-based classroom were used. The socio-technical structure of the educational system was based upon having media as verbal or printed – where printed includes copying from the blackboard. Pedagogical interactions, or transactions, were mainly face-to-face. The classroom may have been a one-to-many faceto-face transaction while supervision may have been one-to-one. If transactions were not faceto-face they were based upon the transfer of a
physical object such as a paper-and-pencil essay or report. The e-learning educational environment is changing the basic system structure. In socio-technical terms, the transaction between the student and the information systems has qualitatively changed with the student being offered the opportunity to become a self-directed and life-long learner. Not only is the interface between the student and the information systems changing, but also the basic interface between the student and the teacher. Under e-learning systems the communication between the teacher and the student may never be face-to-face while the transactions during a teaching-learning episode may be oneto-many-to-many as seen in the use of real-time virtual classrooms. At the practical level, there may never be a physical object created for assessment (Johnson, 2006). The critical shifts in the transactions between the student and information systems can be seen in terms of the shift from the physical library to the various electronic resources. In the pre-e-learning world, a major component in the structure of a student’s study system was having physical resources such as books and a library. The actual use of books could be flexible although in some cases, it was restricted to a library or a similar physical repository. The information processing transactions that the student engaged in involved transferring learning materials via pencil-andpaper from the physical resources to their own physical recording system – apart from the transferring of data into their memory. Much of what is important here can be summarized as three main areas: 1. 2. 3.
The social interactions are now mediated by technology systems. Teacher-student interactions are mediated by technological systems The information access system is technology bound, having both learning and social consequences.
Re-Assessing Validity and Reliability in the E-Learning Environment
Each of these has a potential impact upon reliability and validity. In subsequent sections we shall develop this.
Changes in Student Learning Bhavior One comparison between the students of today and a generation ago will demonstrate this idea. Before the advent of e-learning, a student given the task of writing an essay would begin by reading background material, perhaps initially that supplied by the instructor, but at a more advanced stage of preparation, that discovered by a physical search of library materials. There are two basic levels of exploration that we might identify: ‘depth discovery’ and ‘breadth discovery’. Depth discovery takes place when learning materials make reference to some other learning material, and the student goes off to explore the new reference. He may discover that the new material adds and extends the knowledge already acquired, and does so in a directly relevant way to the task at hand (writing the essay). He consequently invests a significant amount of time in reviewing and assimilating the content, which contributes to his depth of knowledge in the discipline area. On the other hand, he may discover that the relevance of the new material is only incidental, and while potentially broadening his knowledge, does not directly contribute to the essay. Given the effort required in physically retrieving and digesting the material, he may choose to put material to one side in an effort to focus more upon the immediate task. This process we will call breadth discovery, and in the context of task completion, often went no further than this. Now consider the current student with access to broad e-learning resources (particularly the Web), which is usually the first choice of engagement with the content area of the essay. Again, both depth discovery and breadth discovery take place,
but the ease of access no longer forces the same stark choice of time and effort investment. Our modern student ‘surfs’ the web, exploring links as they draw his interest. The ready availability of information may favor breadth of knowledge over depth, particularly as the actual effort of constructing the essay now comes down to choosing which pieces of source material to ‘cut and paste’ into the essay. Of particular concern in this context is the ease at which such information may be superficially used, thus affecting the validity of the purported knowledge presented by the student. The work of Morgan, Brickell & Harper (2008) suggests that cut-and-paste behavior detracts from the effective use of learning materials. This change in work practice has begun to change the way that we look at reliability and, particularly, validity. We have to take into account the knowledge upon which an assessment is assumed to be based. If an assessment is validly tapping what was intended for the student to learn, then the assessment has to be designed in such a way that the student can not easily short-cut this show of knowledge.
The Learner’s ICT Skills The introduction of the computer into the learning process has changed some of the parameters surrounding the skills need by the student and has clearly changed the transactions the learner has with the education system. The change from paper-and-pencil to computer is a critical shift in the student-environment interface. It is not simply a linear change as the ball-point pen was from the Fountain pen was from the steel nib and inkwell. The consequences of this for the assessment interface are profound. For example, a hard disk crash in an electronic exam is qualitatively different from running out of ink in a paper-andpencil exam. Little is lost between running out of ink and finding a pencil to replace it, whereas the loss from a hard-disk failure can be total.
Re-Assessing Validity and Reliability in the E-Learning Environment
Case Study While teaching a recent course on statistics to final year psychology undergraduates, a basic e-learning model was employed whereby all activities were computer-based and all communication outside class was via email. It was assumed that students in 2007 would be computer literate. What became evident was that many could routinely use their regular software without having any ‘feel’ for what the software could do – they were very unsophisticated users. If something happened that did not fit into their routine usage they could not sort out the problem. The range of computer sophistication meant that a revamp of some of the course included eliminating proposed on-line exams. It was felt that assessment performance would be colored by their level of computer use sophistication That is, the computer was potentially adding an element into the assessment that was not a part of the knowledge and skills associated with the subject – and there was no way to correct for this element. The skill component in the new interface is a further complication in that the general ability of the student to use the electronic resources, the level of sophistication in computer skills, can potentially influence performance. There is a small but growing body of knowledge that supports the case study and our general contention that the validity and reliability of assessment can be positively or negatively influenced by technology skills (Horkay, Bennett, Allen, Kaplan & Yan, 2006). The case study is an example of how the use of seemingly ubiquitous technology cannot be taken for granted when designing assessment – and in designing a valid teaching environment. If your assumptions about student control of the technology are not correct then you will be creating a situation where a proportion of students will be disadvantaged thereby potentially invalidating the assessment. This is rarely the case in courses taught in conventional face-toface environments.
From this we conclude that the validity (and possibly the reliability) of an assessment can be influenced by the interface being used to record that assessment.
Staff Reflection Several instructors held some concerns as to whether the students may well know more about the use and application of technology than they might. It is not uncommon in the lecture theatre when e-learning technology displays a certain propensity to misbehave to have one or more students offer advice on how to circumvent the problem. Apart from the undermining effect that this might have upon the instructor’s self-confidence, it poses some questions as to the reliability of assessment devices. What little self-confidence the naive elearning instructor might have can be completely shattered when they realize that students may well have the ability to hack into databases holding exam questions or assessment results. The above staff reflection is important in that it emphasizes the way in which the socio-technical structure of the education process can influence outcome. Student control of an interface could have an effect on the quality of the assessment. For example, the staff member may lack the confidence to design a highly valid assessment because they are wary of the extent to which the students might be able to control the situation.
The Learner as Self-Managed Larner A goal of Western education systems is the development of the individual as a self-managed, life-long learner. An example of how this is being operationalized is seen in the discussion of Advanced Distributed Learning by Fletcher, Tobias & Wisher (2007). The e-learning world has enhanced the possibility of an individual acquiring the expertise to take over the management of their learning. As the gate keeping process is reduced
Re-Assessing Validity and Reliability in the E-Learning Environment
and data becomes accessible, the individual can delve into a seemingly unlimited range of knowledge areas. Importantly, as we encourage people to become self-managed learners we are encouraging them to evaluate the information they acquire against their information search needs and interests. As suggested by the staff reflection does this mean that emphasis upon formal validity and reliability is in need of revision?
Staff Reflection One staff member in our informal research into staff attitudes to reliability and validity suggested that this had become a relatively low level issue as students were now self-managed learners. If we take seriously the idea of the self-managed learner, then we have to question the nature of reliability and validity as it has been traditionally presented – even as we have been developing it here. The changing socio-technical structure of the educational environment has changed the potential set of transactions that can take place between the learner and knowledge resources. It is obvious that self-managed and life-long learning happened before the advent of e-learning. The body of research into this would suggest that those individuals were a particular sub-group of the population: independent thinkers, searchers after knowledge etc. Electronic mediation of the knowledge search process has changed many things including the level of motivation needed to do tasks – for example, the Web search versus the library search. The validity of a study sequence in the selfmanaged context focuses upon whether the learner feels that they have attained the ends that they set out to attain. An appropriate assessment should be within the learner’s frame of reference. The teacher can facilitate this approach by providing guidance of what might be achieved and on resources that might be used and can provide appropriate assessment tasks. The learner will
10
choose their path through the possible set of goals and resources.
Case Study This case was reported by a teacher in a post-experience program in the health sciences where she had set an assessment on alternative treatments for hay fever and related allergic conditions. Students were referred to a variety of sources but it was expected that they would research the topic as if they were operating in a professional environment. Some responses from students created problems for the lecturer because of the nature of the data being presented by students. Through the array of information on the Web they had found many sites that presented research results on the beneficial aspects of many non-standard medications. The assessment was no longer an assessment on what was available but became more focused upon the analytical skills of the students to make sense of the veracity of the research that they were quoting. The lecturer had assumed that they would stay with more ‘respectable’ information, namely peer review material. The issue here is the more information that is available, the greater the skills needed by those who access this material to evaluate and weed out the good from the bad. This influences the nature of assessment and consequent reliability and validity.
The Learner as Information Oganizer Within any teaching and learning environment the student has a great capacity for transforming information and placing it within his or her own context – what we might call a student information transmogrification process. The e-learning framework can make it more difficult for the teacher to monitor how far learner utilization of materials has moved from what they intended when those
Re-Assessing Validity and Reliability in the E-Learning Environment
materials were put together (for discussion around this issue in medical education see Wutoh, Boren, & Balas (2004) and in teacher training, Yao et al. (2008). The Web-based world view, supported by the Google search approach, has opened up the access that individuals have to information. But a web page or a referred document has no predetermined veracity as is assumed for an article or paper that has undergone peer review. There are a number of studies that have investigated the way students view information, particularly about their view of taking data from the Web (Badke, 2007; Warn, 2006). Our understanding of validity has to change through this change in information access. The content we get may be valid but the extent to which we are assessing a student’s actual knowledge is not. We are moving away from concern about the formulation of the question to concern about the veracity of the materials that have been submitted. What we are saying is that we now have to add another type of validity to the list given in the opening pages – Knowledge Validity. We will deal with this in the next section.
The Influence of E -L earning on Reliability and Validity The previous sections have pointed to a number of areas where e-learning, as a socio-technical teaching/learning environment, is influencing reliability and validity. We have noted that the mediating influence on an assessment can be the level of technology skill of the learner and this can potentially question the veracity of the results of the assessment rather than simply the reliability and validity. We have also noted that the changing nature of the study environment is changing the perception that students have of the acceptability of data sources. As we have also noted, the formal structure of reliability and validity has its origins in the educational world of paper and pencil assessment and in the blackboard-based lecture theatre. The
change to online, distributed education changes the contact between lecturer and student (Chang & Smith, 2008) but the implications for reliability and validity appears not to have been subjected to research.
Staff Reflection Many of the staff commented upon the much heavier dependency placed upon lecture notes by students. In the past, where notes were either assembled from classroom discussions by the students directly, or offered as photocopied or spirit-duplicated handwritten notes by the instructor, students took a great deal of responsibility for ensuring that their lecture notes provided them with the learning aids that they needed. Now, when many instructors go to great lengths to develop on-line lecture notes, usually to a high standard of presentational excellence through tools such as PowerPoint, students feel that the responsibility for such learning materials now lies entirely with the instructor. The validity of the material is no longer the responsibility of the learner.
Knowledge Validity Knowledge Validity is firstly about the extent to which the assessment being submitted is the work of the student – that this is a valid sample of the student’s knowledge. It is seen as being distinct from other forms of validity because it is about what the student actually submits as the assessment rather than what the teacher may assume the student might submit - as is the case with Face Validity or Content Validity. Furthermore there are no simple external criteria against which Knowledge Validity can be assessed. Figure 2 shows the subsystem upon which Knowledge validity is based. We have shown the transactions with information sources because this is critical to recognizing the way in which this type of validity operates.
11
Re-Assessing Validity and Reliability in the E-Learning Environment
We are well aware of the increasing extent to which the web-based cut-and-paste has replaced the interpret-and-rephrase approach (Warn, 2006)). There are also indications that some students are not clear in their own mind about the inappropriateness of simply cutting and pasting other people’s material (Badke, 2007). Note that this is at a different level to those who blatantly cheat by having others do their work for them. We would not include cheating qua cheating as a question of knowledge validity. Knowledge Validity has always been a part of qualitative and semi-structured assessment. You, the assessor, would be looking to see whether the student had done the necessary digesting of material before putting pen to paper. Students have always plagiarized but they tend to have been very aware of what that meant – both as an activity and as a consequence of that activity. We would suggest that there is a changing culture where Voithofer’s (2002) knowledge is only a click away concept is being transformed into knowledge that is a click away can be simply absorbed into my knowledge. This is not a malicious approach to knowledge acquisition, rather it is an assumption derived from the belief that knowledge is universal (Badke 2007; Warn, 2006). Knowledge validity operates at the stage when the assessment is being done, which differs from
all other types of validity. It is about the assessing process rather than the grading at the end of the assessing process. The conventional outcomebased validity is the interaction between the summation of whatever has happened in doing the assessment and the defined external criteria. Knowledge validity may influence the final mark but it may also invalidate the submission given by a student. This is exacerbated by the questions that have been raised about the veracity of information taken from the web. The teacher’s task in assessing work is made more complicated because material being retrieved may have to be evaluated. We recognize that students were capable of introducing new and interesting materials in pre-electronic submissions, but these, in our experience, were from the small number of more adventurous and involved students. All students are now accessing new material and that material can be simply false. This extends into the generalizability, if not veracity, of research data reported from web-based research (Gosling, Vazire, Srivastava & John, 2004). The assessor’s task becomes more complicated. This is illustrated in Figure 2 through the transactions shown between the assessor and the information sources.
Figure 2. The assessment process subsystem Design assessment
Students do assessment
Marking of assessment
Information sources Knowledge Validity
12
Re-Assessing Validity and Reliability in the E-Learning Environment
Knowledge Reliability A lesser component in this concept is reliability. Where knowledge reliability would apply is in what is known as inter-rater (or inter-assessor) reliability. This is not necessarily about the assessment tool per se but is about the clarity of the assessment task. For example, inter-assessor reliability is enhanced by having clear marking guidelines for assessment tasks so that each assessor is using very similar criteria. Under the conditions we suggested that create knowledge invalidity, various assessors will probably be dealing with information in different ways. This is likely to reduce the reliability between assessors.
Establishing Reliability and Validity A simplistic approach to overcoming some of the issues that are arising through the use of elearning approaches is to say that we should use only well-structured objective tests. Validity and reliability can be more readily maintained and knowledge validity issues will all but disappear. Assessing student attainment involves more than tapping into basic memorizing and low-level analysis. Furthermore, the world of self-managed learning does not easily encompass objective tests. Consequently, we have to look toward more complex solutions We have broken these up into three areas: Educational, technological and process.
Eucationally Oriented If we accept that Knowledge Validity (as we have defined it) is becoming a part of the student culture, then the educational system needs to explore ways of re-orienting students towards acceptable educational practices. We could adopt the cynical path and say that there are new paths that accept that students will adopt surface approaches to
acquiring knowledge – ‘we no longer need to be experts because we have all the knowledge we need at our finger tips’.
Staff Reflection The overriding issue commented upon by most interviewees was the vexed question of plagiarism. All agreed that it was ‘far too easy’ for students to access material from the web relevant to their assessment, and to submit it as their own work. There are numerous methods available to detect and identify plagiarism, and teachers see this very much as a kind of ‘arms race’ between staff and students. Most commentators on this topic tend to suggest that the educational system has to engage in serious training of students in acceptable academic behavior. Some point to the rather limited effects of the plagiarism detection systems (e.g. Warn, 2006). While one of the few studies on how students see plagiarism software (Dahl, 2007) found that students were generally positive about using such software but there were significant areas where education was needed. The wealth of experience in applying learning theories indicates that punitive models of behavior change do not work while those involving positive reinforcement and an educative approach are the ones that are more likely to produce change. Within the education needs of students in elearning environments is also the need to train them to be better evaluators of the information they are receiving. The work of Morgan and his associates (2008) on cut-and-paste behavior indicates that the tendency to cut-and-paste includes a tendency not to analyze. Anecdotal evidence shows that some teachers are teaching better sifting and analyzing skills but that this is far from universal. We have been unable to find any research that identifies what is needed in a program to achieve this end.
13
Re-Assessing Validity and Reliability in the E-Learning Environment
The technology for e-learning appears to be lagging behind what is happening in software development. For example, the various learning management and delivery systems do not take advantage of components such as agent technologies to implement real-time management of routine tasks such as the validity of links to data and resources. We have found little discussion of the pedagogical implications of technology. Technology can be applied to the issue of student work practices to help reduce the Knowledge Validity problems. Extensive work has been done on interpreting keystroke patterns to the point where a reasonably strong keystroke identity can be established. Another aspect of this would be the mistakes a person makes in spelling and grammar. Software tools could be devised to build a profile using these parameters. From this, we would suggest, an electronic assessment booklet could be built that would be difficult to fake as the identity of the user is created through the use of the computer.
Plagiarism has also distorted the assessment reliability and validity through the ‘arms race’ commented upon in an earlier staff reflection. Because teachers increasingly focus upon avoiding easy opportunities for plagiarism to occur (by not reusing past assessments, by making tasks more open-ended, etc.), they are unable to focus upon improving and developing the intrinsic assessment educational attributes, such as reliability and validity. Institutions also demand additional attention to be paid to meeting various ‘quality’ constraints, such as setting clear deadlines, student workload constraints, and return of marked assessment within a specified time frame. The teacher has a broader range of both educational and logistic demands to be met, and the priority given to the educational aspects may suffer as a result. Plagiarism detection software has an extensive following in Higher Education and many appear to see it is the technological means of enhancing the validity of assessments. But this is a technological after-the-fact approach. It is not a solution but rather a ‘band-aid’.
Staff Reflection
Teaching/Learning Process Oriented
The availability of tools such as electronic submission, where students can submit their work electronically, has led many teachers to explore how automatic processing and marking of the submitted work might be employed. Such tools may have an option of returning the graded work to the student almost instantaneously, allowing them to recognize their errors, rework the material, and resubmit it. While not entirely eliminating the issue of validation of the work, the approach has led to some secondary investigations of the nature of the resubmissions. Careful analysis of the changes made by a student allows the teacher to gain insights into the learning process followed by the student, which does increase the content validity of the assessment process.
Within the teaching and learning process there are changes that can enhance aspects of reliability and validity in e-learning environments. Most of these exist outside e-learning and what we are suggesting are extensions rather than innovations. Our concern is to present some ideas that may be of use even if to some they are ‘old-hat’.
Technology Oriented
14
Peer Rview Peer review is a technique that is widely used in evaluating teaching performance and is often used particularly in establishing face and content validity. A structured approach to doing this involves some effort. It needs to start with preparing information for the colleague that makes clear what is intended in the assessment. The
Re-Assessing Validity and Reliability in the E-Learning Environment
course objectives may presume related student learning outcomes will be assessed but each of us has idiosyncratic views on how this can happen. The person(s) doing the review need to be aware of the personalized twists in thinking. This allows the review to not only deal with the content of the assessment but also its process – it is not always the case that the students have not learned material; for example it can be that the teacher has made wrong assumptions about how the students see that material. A peer review approach to validating assessments should aim at open exchange of information rather than at highly structured data collection. Peer review fails when it is tied down to proforma and check lists, such as those seen on many University web sites, because such methods do not encourage open communication and critique. Crucial to the process of validating assessment is the notion that no preconceptions influence the review being made, and that the assessment itself is judged on its merits, not upon its alignment with any formal criteria. In our experience, when the going gets tough, people hide behind the formal system. The structure of the peer review should begin with the peer asking the teacher to define what they assume the students should know in order the answer the question. From there it would move onto the form of the student response; for example, is there one way the students should approach it or are there a number of ways? From here the discussion would move onto the grading of the assessment. At this point it is sometimes the case that the teacher has failed to make a clear link between what they say the students need to do get a pass/credit/distinction and what they have said in earlier descriptions. This can become a core test of the validity of the assessment tasks. Reliability issues will show up in a peer review when the teacher is unclear on some aspects of the way students can be expected to deal with the task. This can lead to assessor unreliability.
Innovative Assessment Development Teaching experience tells us that it is easy to become locked into the way you think about assessment questions. After a couple of years teaching a subject there can be some intellectual routines that develop. Within the e-learning environment, assessment questions can become very similar making it easier for the uninvolved student to cut-and-paste and plagiarize. There are various ways that staleness in thinking can be reduced and the general validity of assessments improved, particularly for Knowledge Validity. The most common one we have found is the staff-room pedagogical lunch group. This starts by simply raising the topic of ‘how would you assess area X’. Academic staff being renowned as opinionated will offer ideas – some good, some bad. The more academic approach is to go to the journals that cover educational issues in your discipline area as well as those in other areas. In fact, Weimer (2006) in her exploration of how best to enhance your scholarly approach to teaching suggests that many of these journals cover much the same materials, adding particular discipline flavors. She sees this a good source for development. Two examples of assessment innovation we found interesting while researching this chapter inluded Johnson’s (2006) evaluation of design portfolios and the analysis of the use of ePortfolios in teacher education by Yao et al. (2008).
Student Review A student review of the validity and reliability of an assessment is similar to the peer review in that it is looking at how the students perceive the assessment tasks. The basis for a student review can be in taking a random sample of students and interviewing them on how they saw and approached the assessment.
15
Re-Assessing Validity and Reliability in the E-Learning Environment
This has been done for programming courses where the aim is to validate the assessment for all students. For example, every student may be interviewed after submitting the major programming task. Answers given by the student are used to interpret the perceived knowledge validity of the written work. Such an approach is definitely highly effective. Not only can the teacher see how well the student has interpreted the task but they also get an indication of the reliability of the task. By asking students to interpret the assessment question on this second occasion, the teacher has an indication of whether the student perception of the question is stable. A more general approach to student review interviews begins by asking the students to interpret the question(s) in their own words. From this you are able to establish whether the student reads the question in much the same way as you wrote it and gives you a direct indication of likely validity and reliability issues.
Future Directions The future does not include throwing out the traditional structure and methods for reliability and validity. Assessments will still take many different forms and will include objective tests as well as more open assessments that are numerically graded. What the future has to focus upon is the fact that reorientations are needed in the teaching and learning environment to be able to support valid and reliable assessments. The fact that e-learning systems differ from face-to-face systems and have particular effects upon the way students approach learning, means that we need to look at how we can enhance reliability and validity in electronic assessments. We see these as fitting into two categories: technology and process. The technology elements are those that simply involve developing appropriate software and hardware while the process elements
16
are those activities that are associated with the actual assessment tasks. The caveat on this is that much of the standard forms of assessment can be analyzed through conventional methods to determine reliability and validity. If you have an acceptable set of numbers then various statistical tools can be used. A multiple choice quiz is still a multiple choice quiz; marks on a set of written questions are marks that can be analyzed. What has changed is the context within which assessment is taking place. This context needs to be better understood. Most of the current assessment within elearning environments has been derived from traditional assessment approaches. One notable variation has been in the area of virtual simulations to support or replace actual practical work in for instance, medicine. Johnson (2006) has written about a systematic approach to the translation of portfolio assessment into a viable online procedure and has shown how valid and reliable results can be attained. This does not use the current ePortfolio method but rather a basic design portfolio that is handled online. The growth of on-line assessment will develop from work such as this and not from the concerns about plagiarism that seem to be driving assessment in e-learning environments. It is the innovative use of assessment methods that belong in the future, because they are about education rather than control. We would suggest that the way forward can also be defined in producing educational support tools that reflect the capacity of the computer to provide a substantive educational development. At the level of software technology, current learning management systems are lagging a long way behind major software developments. For example, as far as is publicly accessible, none of the current proprietary or Open Source systems uses agent technology to help with real-time management tasks. Data storage and retrieval is still tied to the basic meta-tagging approach derived from information management in libraries, whereas
Re-Assessing Validity and Reliability in the E-Learning Environment
Google has shown that a dynamic indexing and search system can be extremely effective (Krishnaswamy, Markham, Chhetri, Hurst, & Casey, 2004). Both of these technologies can be used to support more effective assessment systems. The literature is hardly bursting at the seams with educationalists who wish to define a changing relationship between e-learning and reliability and validity. In fact, data base searches of peer reviewed journals produced very few hits on relevant key words (e.g. reliability AND (“e-learning” OR elearning OR “virtual learning”), validity AND (“e-learning” OR elearning OR “virtual learning”)). Core assessment texts (e.g. Gipps, 1999; Hogan, 2007; Payne, 2003; Sedlacek, 2004) do little to reflect on this changing world. We have shown that a socio-technical systems approach to the teaching and learning environment helps explicate the changing world of e-learning and we believe that educational theorists must look towards establishing the viability of current thinking in the new world given that their major model, constructivism, has been subjected to stringent criticism even in the conventional educational world (Fox, 2001; Simpson, 2002). A new pedagogy would have to be able to account for and deal with the complex set of transactions that take place between students, teaching staff, administrative staff, and technology. Within this, the structure of assessment has to begin to take account of the life-long learning model that is inherent in the developing system (Markham, 2008). This further points to the need for staff training to help lecturers better understand what can be done within an e-learning framework and to give them skills that help them move forward rather than reacting to external pressure.
CONCLUSION The substantive structure of reliability and validity analysis has not changed with the advent of electronic delivery. What has changed are factors that
affect the learners’ approach to assessment tasks and these factors introduce some new components in defining, in particular, validity. The idea that validity can also be seen in terms of the Knowledge Validity that is based on the student’s effective use of information means that validity moves from being an input process (face and content validity) and an output process (construct, consequential and predictive validity), to being a part of the assessment process. That is, we are now talking about validity that is not simply aggregated across the class being assessed but is about the individual student’s actual performance on the assessment.
R Badke, W. (2007). Give plagiarism the weight it deserves. Online, 31(5), 58-60. Brennan, R. L. (Ed.). (2006). Educational measurement. (4th Edn.) Westport, Conn: Praeger. Carmines, E. G., & Zeller, R. A. (1991). Reliability and validity assessment. Newbury Park: Sage Publications. Chang, S-H., & Smith, R. A. (2008). Effectiveness of personal interaction in a learner-centered paradigm distance education class based on student satisfaction. Journal of Research on Technology in Education, 40(4), 407-426. Cizek, G. J. (2008). Assessing educational measurement: Ovations, omissions, opportunities. Educational Researcher, 37(2), 96. Dahl, S. (2007). The student perspective on using plagiarism detection software. Active Learning In Higher Education, 8(2), 173-191. Emery, F. E., & Trist, E. L. (1969). Socio-technical systems. In F. E. Emery (Ed.), Systems thinking, (pp. 281-296). Harmondsworth: Penguin. Fletcher, J. D., Tobias, S., & Wisher, R. A. (2007). Learning anytime, anywhere: Advanced distrib17
Re-Assessing Validity and Reliability in the E-Learning Environment
uted learning and the changing face of education. Educational Researcher, 36(2), 96-102. Fox, R. (2001). Constructivism examined. Oxford Review of Education, 27(1), 23-35. Gipps, C. (1994). Beyond testing: Towards a theory of educational assessment. London: RoutledgeFalmer. {eBook} Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59(2), 93–104. Harlen, W. (2005). Trusting teachers’ judgement: Research evidence of the reliability and validity of teachers’ assessment used for summative purposes. Research Papers in Education, 20(3), 245-270. Hogan, T. P. (2007). Educational assessment: A practical introduction. John Wiley & Son. Horkay, N., Bennett, R. E., Allen, N., Kaplan, B., & Yan, F. (2006). Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment, 5(2). Retrieved 9th August 2008 from http://www.jtla.org. Issroff, K., & Scanlon, E. (2002). Using technology in higher education: An activity theory perspective. Journal of Computer Assisted Learning, 18, 77-83. Johnson, C. S. (2006) A decade of research: Assessing change in the technical communication classroom using online portfolios. Journal of Technical Writing & Communication, 36(4), 413-431.
pedagogy independent education environment. Proceedings of the 4th IEEE International Conference on Advanced Learning Technologies (ICALT 2004), Joensuu, Finland, IEEE Press. Lee, S-H., Wehmeyer, M. L., Palmer, S. B., Soukup, J. H., & Little, T. D. (2008). Self-determination and access to the general education curriculum. The Journal of Special Education, 42(2), 91-107. Markham, S. (2008). A sociocybernetic approach to pedagogical systems. Technical Report 2008-1 Computing Education Research Group, Monash University Retrieved 12th August 2008 from http://cerg.infotech.monash.edu.au/techreps/tr2008-1.pdf Mehanna, W. N. (2004). E-pedagogy: The pedagogies of e-learning. Research in Learning Technology, 12(3), 279-293. Miller, C. (1998). A socio-technical systems approach to distance education for professional development. Open Learning: The Journal of Open and Distance Learning, 13(2), 23-29. Morgan, M., Brickell, G., & Harper, B. (2008). Applying distributed cognition theory to the redesign of the copy and paste function in order to promote appropriate learning outcomes. Computers & Education, 50(1), 125-147. Newson, J. (1999). Techno-pedagogy and disappearing context. Academe, 85(5), 52-56. Payne, D. A. (2003). Applied educational assessment. Belmont: CA: Wadsworth/ Thompson Learning. Rice, A. K. (1970). The modern university: A model organisation. London: Tavistock.
Kane, M. T. (2008). Terminology, emphasis, and utility in validation. Educational Researcher, 37(2), 76-83.
Sedlacek, W. E. (2004). Beyond the big test: Noncognitive assessment in higher education. San Francisco: Jossey-Bass.
Krishnaswamy, S., Markham, S., Chhetri, M. B., Hurst, A. J., & Casey, D. (2004). PIAVEE - A
Simpson, T. L. (2002). Dare I oppose constructivist theory? The Educational Forum, 66(4), 347-354.
18
Re-Assessing Validity and Reliability in the E-Learning Environment
Spodark, E. (2005). Technoconstructivism for the undergraduate. Foreign Language Classroom Foreign Language Annals, 38(3) 428-435. Voithofer, R. (2002). Nomadic epistemologies and performative pedagogies in online education. Educational Theory, 22(4), 479-494. Warn, J. (2006). Plagiarism software: No magic bullet. Higher Education Research and Development, 25(2), 195-208. Weimer, M. (2006). Enhancing scholarly work on teaching and learning. San Francisco: JosseyBass. Wutoh, R., Boren, S., & Balas, E. A. (2004). Internet continuing education for health care professionals: An integrative review. Journal of Continuing Education in the Health Professions, 24(3), 171-180.
Yao, Y., Thomas, M., Nickens, N., Downing, J. A., Burkett, R. S., & Lamson, S. (2008). Validity evidence of an electronic portfolio for preservice teachers. Educational Measurement, Issues and Practice, 27(1), 10-24.
ENDNOTE
a
The Staff Reflections have been generated from the comments from a small sample of 13 staff who were interviewed for this chapter. All were teaching into various undergraduate and postgraduate programs in the Faculty of IT. They were asked to comment on reliability and validity and eLearning.
19
20
Chapter II
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course Päivi Hakkarainen University of Lapland, Finland Tarja Saarelainen University of Lapland, Finland Heli Ruokamo University of Lapland, Finland
ABSTRACT In this chapter the authors report on the assessment framework and practices that they applied to the e-learning version of the Network Management course at the University of Lapland’s Faculty of Social Sciences. The objective of the assessment was to examine students’ perspective regarding how a digital video-supported, case-based teaching approach supported students’ meaningful learning. The model for teaching and meaningful learning (TML) was used as the theoretical assessment framework. To answer the research questions, the authors gathered data through questionnaires completed by the students. The assessment provided them with evidence concerning the student perspective on teaching and learning processes during the e-learning course. The authors will describe and discuss this evidence in this chapter. In addition, they discuss the strengths and limitations of the assessment framework, and practices that they applied to the Network Management course.
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
INTRODUCTION In this chapter, we report on the assessment framework and practices that we applied to the e-learning version of the Network Management course at the University of Lapland’s Faculty of Social Sciences. As the assessment framework, we used a model for teaching and meaningful learning (TML) that we developed over a series of research projects (see Hakkarainen, 2007a, 2007b; Hakkarainen & Saarelainen, 2005b; Hakkarainen, Saarelainen, & Ruokamo, 2007; Karppinen, 2005). The TML model consists of teaching and meaningful learning, which is defined in terms of 17 process characteristics and expected outcomes. Students enrolled in Network Management completed a questionnaire measuring these components of the TML model. Rather than focusing on assessing students’ learning outcomes, this chapter focuses on assessing student perspectives on the course’s teaching and learning processes. Therefore, we consider the assessment to be a formative assessment which provided us with evidence concerning the student perspective and helped us evaluate the course and understand the need to revise teaching and learning activities (see also Bransford, Brown, & Cocking, 2001; Poikela & Poikela, 2006). Our objective in this chapter is to show how the questionnaire we devised and utilized proved to be a useful assessment tool in gathering evidence of the e-learning students’ perspective on teaching and meaningful learning processes in the Network Management course. Due to the poor availability of nonverbal behaviour in typical e-learning settings (Matuga, 2005), we have experienced a strong need to use assessment practices to find out what students are thinking and feeling, how their learning proceeds and how they experience the course design. The chapter begins with a presentation of the Network Management course and the TML
model used in the assessment. Next, we present the assessment practices and their results. Finally, general conclusions are drawn and the applicability of the TML model as an assessment framework is discussed.
Th E -Leaai of Network Management Network Management was implemented online in spring 2005 for students in the final stages of their master’s degree in the Faculty of Social Sciences. The focus of the course is public administration and management, with the aims being that students learn to 1) define a network as a structural and functional form of inter-organizational cooperation, 2) understand how organizational management and leadership differ from network management and leadership, and 3) distinguish different types of networks and understand their limitations. These course goals can be expressed in terms of more specific objectives corresponding to the cases taken up in the course. The rationale for the online implementation was to allow students to develop the desired skills while working in electronic environments, as these are rapidly becoming the norm for employees in public administration (see Schedler, Summermatter, & Schmidt, 2004). Thirty-three students enrolled for the twomonth course. They ranged in age from 22 to 51 years and were spread throughout the country. Following a four-hour introductory, face-to-face lecture, the students embarked on case-based work in groups of three to five using the Finnish Discendum Optima learning management systema. This environment, similar to WebCT and Blackboard, enabled the teacher to provide guidance and facilitated small group conversations, delivery of course materials and preparation of assignments.
21
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
The Pedagogical Model: Case-Based Taching with Digital Video Support Case-based teaching supported by digital video (DV) was considered to be an especially suitable pedagogical model, as the course content is best conveyed through examples that link actual working practices with theoretical knowledge (see also Shulman, 1992). Case-based teaching, inspired and sustained by Dewey’s ideas of discussion pedagogy, has been formalized and refined over a period of decades, most notably through the work done at the Harvard Business School (Barnes, Christensen, & Hansen, 1994). Its advocates assert that it is a valuable approach for both face-to-face and e-learning across disciplines, and it has found extensive application in law, business, medicine, education, architecture, and engineering (McLellan, 2004). On the other hand, the cases, methods, learning activities and group sizes used vary both between and within fields (see Barnes et al., 1994; McLellan, 2004; Shulman, 1992), and Shulman (1992) has claimed: “The case method of teaching does not exist” (p. 2). The cases used in Network Management fall within McLellan’s (2004) definition of a case as “an open-ended story that calls for complex, subtle information from multiple points of view” (pp. 14−15). The students analysed three real-life cases, each centring on a 13- to 20-minute DV simulating a social situation that might arise in the circumstances portrayed. These materials had been produced by the students’ peers in the faceto-face version of the course (see Hakkarainen & Saarelainen, 2005a). The pedagogical function of the DVs was to act as a “trigger” (Schwartz & Hartman, 2007) and a “hook” (Jonassen, Howland, Moore, & Marra, 2003) that engages the students to discuss and solve the case. The use of the DVs in Network Management draws on Jonassen’s work on using computer applications as cognitive tools – Mindtools – in constructive learning to simulate real-world situations and to represent perspectives and arguments of others
22
(Jonassen, 2000). Other learning materials included scientific articles, a book, Web pages related to the cases and PowerPoint slides from the introductory lecture. Each case illustrated one of the following course topics: (1) wicked problems, (2) networking competence and (3) innovation networks. For example, the second case involved gauging the networking competence of the Finnish Sports Federation (FSF). The DV presented a meeting in which consultants, the representatives of FSF and their partners (all played by the students and the teacher of the face-to-face course) discussed what kind of measurement tool might be used for the purpose. The students’ learning outcomes were assessed on the basis of the collaborative learning tasks (on a scale of 1 to 3) and learning journals (pass/fail). The group tasks consisted of writing future scenarios (case 1), devising measures for a self evaluation of networking competence (case 2) and elaborating a strategy for a local innovation network (case 3). The summative assessment practices applied at the end of Network Management can be said to represent authentic assessment and performance assessment (see Jonassen et al., 2003; Mathur & Murray, 2005). The assessment focused on a learning task designed to resemble real activities that the students may need to perform in their working life. The strengths of case-based teaching are increased student interest and engagement, improved retention, and enhanced problem-solving and critical thinking skills (McLellan, 2004). Crucial to successful learning is not the use of cases per se, but how the teacher organizes and supports the students’ learning activities (Grant, 1992).
Assessment Frama Figure 1 illustrates the assessment framework used in Network Management, the TML model, whose
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
roots lie in the integrated model of network-based education (see Vahtivuori et al., 2003). The TML model can be used in designing, implementing and evaluating courses. Among the previous applications of the model is a face-to-face course centring on problem-based learning (PBL) (see Hakkarainen, 2007a, 2007b). The TML model embraces teaching and meaningful learning, which is defined in terms of 17 process characteristics and their expected outcomes, that is, domain-specific and generic knowledge and skills. An additional component is pedagogical models or approaches (e.g., case-based teaching, PBL), which cover both the teaching and the learning processes. Central to the TML model are the relationships between its components. No direct causal relationships can be demonstrated between them; rather, these relationships are reciprocal and conditional, which is depicted in the Figure 1 using dashed, two-way arrows. A further underpinning of the model is the assertion that each of its components is of critical importance. The model draws on the concept of meaningful learning, which is most often associated with
the work of Ausubel (see, e.g., Ausubel, Novak, & Hanesian, 1978) and, later, Novak (1998), who introduced concept mapping as a tool for facilitating and assessing meaningful learning. The influence of Ausubel (ibid.) is most apparent in the active, constructive, individual, and goaloriented characteristics of meaningful learning. It can be argued that the process characteristics of meaningful learning used in the TML model represent an interpretation of meaningful learning for the 2000s. Anderson, Rourke, Garrison, and Archer (2001) have put forward the concept of a teaching presence for e-learning settings that use computer conferencing. They define it as the “design, facilitation, and direction of cognitive and social processes for the purpose of realizing a personally meaningful and educationally worthwhile learning outcome” (p. 5) and describe it in terms of three categories: (1) design and organization (e.g., designing methods, utilizing media effectively), (2) facilitating discourse (e.g., encouraging student contributions, setting the climate for learning, drawing in participants) and (3) direct instruction
Figure 1. The TML model (Hakkarainen, 2007a, 2007b; Hakkarainen et al., 2007)
23
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
(e.g., presenting content/questions, diagnosing misconceptions). The last category includes assessment and explanatory feedback, which elearning students in particular value highly (see also Chang & Petersen, 2005). Echoing Anderson et al. (ibid.), the TML model incorporates a broad conception of teaching as drawing on a variety of teaching activities. The TML model views teaching and meaningful learning as processes triggered by various pedagogical models or approaches, such as casebased teaching (see, e.g., McLellan, 2004), PBL (see, e.g., Poikela & Nummenmaa, 2006) and anchored instruction (see Cognition and Technol-
ogy Group at Vanderbilt [CTGV], 1993). In this framework, a pedagogical model or approach is understood, following Joyce and Weil (1980), as “a plan or pattern that can be used to shape curriculums (long-term courses of studies), to design instructional materials, and to guide instruction in the classroom and other settings” (p. 1). Rather than being antithetical or mutually exclusive, the pedagogical models applied may amplify one another when integrated (Joyce & Weil, 1992). Central to application of the TML model is that not all of the 17 characteristics of meaningful learning need be present at any given time. Moreover, the characteristics can be intertwined,
Table 1. Process characteristics of meaningful learning.b 1) Active: Active learning means that “learners are engaged by the learning process in a mindful processing of information, where they are responsible for the result” (Jonassen, 1995, p.60). Students are encouraged to ask questions, acquire information, critically evaluate information, express new ideas and models of thinking (Ruokamo et al., 2002), and use different productivity tools and cognitive tools (e.g., videos) in their learning environments (Jonassen, 1995, 2000). 2) Self-directed: Self-direction in learning refers to “a process in which a learner assumes primary responsibility for planning, implementing, and evaluating the learning process” (Brockett & Hiemstra, 1991, p.24). The concept is thus intertwined with the characteristics of activeness, goal-orientedness, and reflection. 3) Constructive: Constructive learning means that learners accommodate new ideas into their prior knowledge in a process of meaning making, not of knowledge reception (Jonassen, 1995, 2002). 4) Individual: Individuality means that learners have individual learning styles and strategies and that learning is always influenced by students’ prior knowledge, conceptions and interests (Ruokamo et al., 2002). 5) Collaborative: Working collaboratively makes it possible that students can exploit each other’s skills and provide social support and modelling for other students (Jonassen, 1995, 2002). Collaboration is collaborative knowledge construction in which the group is the subject of learning (see, e.g., Repo-Kaarento, 2004). 6) Co-operative: Co-operative learning entails using groups as tools for enhancing individual learning. Learners’ individual responsibility is the characteristic that sets the approach most clearly apart from collaborative learning. However, the concepts are sometimes used synonymously. (Repo-Kaarento, 2004) 7) Conversational: Conversational learning is a dialogue, that is, a process of internal and social negotiation (Jonassen, 1995, 2002). 8) Contextual: Contextual learning resorts to learning tasks that are either situated in meaningful, real world tasks or simulated through a case-based or problem-based learning environment (Jonassen, 1995, 2000). For example, in anchored instruction, learning is anchored by a videobased problem (CTGV, 1993; see also Ruokamo, 2001). 9) Emotionally involving: Emotion is intertwined with cognition, motivation and learning by affecting perceptions, theoretical imagination, and logical reasoning (Puolimatka, 2004; Schutz & DeCuir, 2002). According to Soini (1999, p.84), emotional involvement is the most important feature of good learning situations for the students and it emerges from “feelings of personal, emotional connectedness to some subject.” Positively toned emotions experienced during the learning process such as interest, joy, surprise (Puolimatka, 2004), and emotions resulting from mastery experiences (Bandura, 1994; Engeström, 1982) have been shown to be vital for learning. However, a successful learning process may also include occasional negatively toned emotions (Kort & Reilly, 2002; Op’t Eynde, De Corte, & Verschaffel, 2001).
continued on following page 24
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
Table 1. continued 10) Goal-oriented: In a goal-oriented learning process, students work actively to achieve a cognitive goal, and can define learning objectives of their own (Jonassen, 1995; Ruokamo & Pohjolainen, 2000). 11) Reflective: Intertwined with goal-orientation is the process of reflection (Jonassen, 1995, 2000). In a reflective learning process, students express what they have learnt and examine the thinking processes required during the process (Jonassen, 1995; Ruokamo & Pohjolainen, 2000). 12) Abstract: Abstract learning can be defined as the construction of new ideas at an abstract level. The development of theoretical ideas reaches from practical experience to a deeper level (Lehtinen, 1997; Ruokamo et al., 2002). 13) Multiple perspectives-oriented: Learners are presented with information from multiple perspectives and case studies that present diverse examples (Spiro, Feltovich, Jacobson, & Coulson, 1992). Learning situations that lead to an awareness of multiple perspectives are experienced by students as good and real learning situations (Soini, 1999). 14) Critical: Critical thinking is a general critical attitude towards knowledge and knowing, described by Cottrell (2005) as “holding open the possibility that what you know at a given time may be only part of the picture” (p. 2). In addition to being a general attitude, critical thinking is a set of practices aimed at exploring evidence in a critical way. It focuses on messages being conveyed through speech, writing, performance or media. (Cottrell, 2005) 15) Experiential: Experiential characteristics mean that students can use their own experiences as starting points in learning and that they are able to apply their own practical experiences during the course. “Experiences” are understood first as meaning students’ prior practical knowledge and second as the aims of learning. Learning should involve integrating theoretical knowledge and practical knowledge into experiential knowledge (Poikela, 2006). 16) Multi-representational: For effective learning processes the careful combination and integration of multiple representational modes (e.g., texts, still/moving pictures, voice) should be used (see, e.g., Dekeyser, 2000; Mayer, 2003; Spiro et al., 1992). 17) Creative: According to Novak (1998), “the creative person sees how to make the right connections between concepts in two domains of knowledge that were previously regarded as unrelated, or in some cases even contradictory” (p. 78). According to Novak, creativity should be viewed as a search for high levels of meaningful learning.
interdependent, interactive, partly overlapping and synergetic (Jonassen et al., 2003; Ruokamo, Tella, Vahtivuori, Tuovinen, & Tissari, 2002). The characteristics are set out in Table 1 (for a more detailed description, see Hakkarainen, 2007b). The expected outcomes of the meaningful learning processes in the TML model comprise domain-specific knowledge and skills as well as transferable, generic knowledge and skills such as metacognitive skills and higher-order thinking (see Tynjälä, 2001). The literature indicates (see Bransford et al., 2001) that transfer can be promoted by ensuring an adequate level of initial learning, encouraging learning through understanding rather than memorizing, and providing students with sufficient time to learn. While contextual learning is, as a rule, conducive to
transfer, excessively contextualized knowledge – sometimes encountered in case-based teaching – can reduce it; knowledge or skills learned in one context only are less likely to be transferred than those acquired in multiple settings. What is more, contextual learning needs to be supported by abstract representations of the focal knowledge. Also known to facilitate transfer is a metacognitive approach to teaching, in which students learn to monitor and regulate their own understanding. In the light of these research findings, one can argue that it is the self-directed, contextual (but not overly contextualized), abstract and reflective characteristics of meaningful learning processes that are most conducive to transferable knowledge and skills (Hakkarainen, 2007b).
25
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
Assessment Praaces The assessment was conducted as part of an action research case study, in which the traditional lecture-based, face-to-face Network Management course was developed into two different versions using case-based teaching: a face-to-face version and an e-learning version. The research was conducted by a course teacher who worked alongside the researchers (see Hakkarainen, 2007b; Hakkarainen et al., 2007; Hakkarainen & Saarelainen, 2005a, 2005b). The strand of action research that the study most closely resembles is the “teacher as researcher” approach, which is derived from the curriculum development work of Stenhouse (see Stenhouse, 1989; Cohen, Manion, & Morrison, 2003).
Research Objectives and Questions The objectives of the action research case study were to develop the teaching and learning processes realized and outcomes achieved in the Network Management course, in order to better support meaningful learning. The assessment focused on finding out student perspectives on the following research questions: 1. How do the teaching activities performed by the teacher support meaningful learning for the students enrolled in the course, from both the process and outcome point of view? 2. In what way does the video-supported casebased teaching used in the course encourage meaningful learning in terms of both the process and outcomes? 3. What kinds of emotions does the case-based teaching used in the course evoke in students, and why does it evoke such emotions?
Questionnaire The e-learning version of the Network Management was realized during spring 2005. To answer
26
the research questions, we devised a 73-item questionnaire, which was posted by mail to the students at the end of the course. The questionnaire cover letter informed students that they were required to complete and return the questionnaire in order to pass the course. Students completed the questionnaires anonymously. The questionnaire included four items relating to students’ demographic variables, that is, gender, age, previous experience with e-learning, and the year they began their studies at the university. Two questions focused on practical learning activities: what case did they study and how much course literature did they read? Students were asked to assess four questions using a grading scale of 1 to 5: (a) the learning management system used in the course, (b) the pedagogical realisation of the course, (c) the realisation of the course with respect to the content matter, and (d) the whole course. The practical implementation of the TML model was measured using a set of 31 statements, which the students were asked to evaluate on a five-point Likert scale. Nine statements focused on the teaching component of the TML model, that is, on teachers’ support and guidance activities. These statements were formulated on the basis of the coding scheme for teaching presence in e-learning used by Anderson and others (2001). The remaining twenty-two were formulated to operationalize the 12 characteristics of meaningful learning processes included in the TML model at the time of the assessment (see Hakkarainen et al., 2007). Out of these survey statements, twelve have been devised and previously used by Nevgi and Tirri (2003; see Nevgi & Löfström, 2005). Twenty-one of the Likert scale questions focused on students’ emotions. The students were asked to indicate to what extent they had experienced a given emotion during the course, and to state what, in their view, had prompted the emotion. Twelve of the twenty emotions appearing on the questionnaire were chosen from those proposed by Kort and Reilly (2002) as possibly
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
relevant to learning: worry, comfort, boredom, interest, frustration, uncertainty, dispiritedness, disappointment, satisfaction, enthusiasm, tension, and embarrassment. To these we added three social emotions relevant to collaborative learning – trust, sense of community and irritation – as well as joy, stress, relief, feelings of inadequacy, and challenge. In addition, students were asked to visualise the development of their emotional involvement during the course by drawing a line or a curve on a chart with time as the x-axis and negative-positive quality of the emotional experiences as the y-axis. Finally, the questionnaire included 10 closed and open-ended questions focusing on students’ experiences using the DVs as learning material. As an example, students were asked to assess whether watching the DVs brought added value to their learning process and outcomes.
Results Out of the 33 students enrolled in the course, 30 completed both the course and the questionnaire. 77 % of the respondents were female. Questionnaire data were analysed qualitatively and quantitatively. As this was a case study and did not seek statistical significance, quantitative analysis was applied as a tool for describing and interpreting the data. Part of the results presented in the following sections have also been reported elsewhere as part of a larger action research case study (see Hakkarainen, 2007b; Hakkarainen et al., 2007; Hakkarainen & Saarelainen, 2005b). In the following, we will present and discuss the results according to the three research questions.
Student Perspective on Teaching Student perspectives on teaching were measured using nine statements on the questionnaire. We further extracted the mean values, standard deviations, and percentages of the students’ ratings of
the teaching activities performed by the teacher of the Network Management course. Table 2 presents the questionnaire data focusing on the practical realisation of teaching activities. The statements focusing on teaching activities were rated somewhat less favourably (M = 3.50–3.80, SD = 0.66–1.00) than the other statements on the questionnaire. However, 60 to 73% of the respondents agreed, or moderately agreed, with these statements. This indicates that the respondents’ perception of teaching activities was fairly positive. The following statement illustrates an exception to this: “The teacher supported my learning significantly by giving personal feedback” (M = 2.37, SD = 1.00), with which only 40% of the respondents agreed or moderately agreed. However, 73% of the students agreed or moderately agreed that the feedback they received “focused on matters relevant to the topics of the course.” These results indicate a need to increase personal feedback in students’ online discussions. Anderson et al. (2001) included feedback as a central component in their concept of teaching presence in online courses (see also Chang & Petersen, 2006; Nevgi & Löfström, 2005; Nevgi & Tirri, 2003). Bransford et al. (2001) concluded that studies of adaptive expertise, learning, and transfer show that continuous formal or informal feedback is extremely important.
Student Perspective on Meaningful Larning Table 3 presents the questionnaire data focusing on the practical realisation of the process characteristics of meaningful learning. The data indicate that DV-supported e-learning in which students solve cases furthers meaningful learning processes, particularly where the following characteristics are concerned: activeness, constructivity, contextuality, abstractness, reflectivity, and multiple perspectives-orientedness. Eighty to ninety-seven percent of the respondents agreed or
27
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
moderately agreed with the statements focusing on these characteristics. Instruction that relies extensively on collaboration may lessen the individuality and self-directedness of the learning process, and the proportion of students indicating that they were able to study in keeping with their personal style was in fact rather low (52 %). However, the data do not permit us to draw conclusions as to whether the students wanted a more individual-oriented or self-directed learning process. The collaborative nature of the learning process was not regarded as supporting learning to the extent that the case-based approach was. While 57% of the respondents agreed or moderately agreed that collaborative learning in small groups helped them learn, it can be argued that this learning occurred through the positive emotions associated with such learning. Particularly striking among the data is the students’ rating of the role of online discussions using chat and a discussion area: most respondents were not convinced that these online forums actually helped them learn. When looked at in light of the students’ views on teaching activities (Table 2), these results point to a need for improved online guidance.
Three of the statements dealt with the transferability of learning outcomes. 90% of the students agreed or moderately agreed with the following statement: “I can utilise what I learned in the course in other situations” (M = 4.43, SD = 0.77), and 93% agreed with the statement “Cases under study supported the acquisition of knowledge and skills needed in working life” (M = 4.43, SD = 0.63). Yet, there was considerably less agreement about whether the course had improved their problem-solving abilities, for only 56% agreed or moderately agreed (M = 3.57, SD = 1.14) with the relevant statement. On balance, these results do not lend clear-cut support to the contention that a case-based teaching approach improves problemsolving skills (see McLellan, 2004). The students assessed the role of the course videos through 10 closed and open-ended questions. Their responses were analysed quantitatively by calculating the proportion of each rating, and qualitatively in light of the themes that emerged in the responses to the open-ended questions. The DVs appear to have played a supportive role in the learning processes: 60% of the respondents agreed or moderately agreed that viewing the videos helped them learn. All of the respondents
Table 2. Students’ (n=30) ratings (1 = disagree, 2 = moderately disagree, 4 = moderately agree, 5 = agree) of teaching activities performed by the teacher Statement on the questionnaire focusing on teaching activities
Teacher supported my learning process and learning outcomes significantly by: (1) setting positive climate for learning (2) providing individual feedback about my progress (3) giving advice on questions related to the subject matter of the course (4) giving advice on technical questions related to the use of the learning management system (5) designing clear course guidelines (6) organizing the learning management system in a clear manner (7) formulating clear course goals and objectives (8) providing feedback and advice in a sufficiently timely manner (9) providing feedback that focused on matters relevant to the topics of the course
28
Mean value
Standard deviation
Moderately agree /agree%
3.80 2.37
0.96 1.00
66.6 40.0
3.77
0.86
70.0
3.50 3.60 3.70 3.67 3.63 3.80
0.86 1.00 0.75 0.92 0.93 0.66
60.0 60.0 60.0 63.4 60.0 73.3
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
stated that the videos elucidated the cases, and 81% asserted that the videos created added value in the learning process; the nature of the added value was detailed in the open-ended questions. From the viewpoint of meaningful learning processes, we can conclude that the DVs supported the contextual aspects of learning by “illustrating the cases,” as well as by “introducing and clarifying the topics” (see also Wiecha, Gramling, Joachim, & Vanderschmidt, 2003; Jonassen, 1995, 2000). On the other hand, by “illustrating the theories and different perspectives,” the DVs also promoted the abstract (Lehtinen, 1997; Ruokamo et al., 2002) and multiple perspectives-oriented aspects
of learning (Soini, 1999; Spiro et al., 1992). In an additional finding, the videos enhanced the students’ emotional involvement (Soini, 1999; see also Bliss & Reynolds, 2004; Schwartz & Hartman, 2007) by “creating an inspiring effect” and by adding “variety” to the learning process. The respondents also noted that the DVs served “to lower the threshold of getting started with learning.”
Student Perspective on Emotions Gnerated by the Course The emotions experienced by the students were analysed by extracting the means and standard
Moderately agree/agree %
4.40
0.72
86.6
Students’ role is to actively acquire, evaluate, and apply information.
3.47
1.17
53.3
I was able to influence the content of my learning tasks.
3.90 4.17 4.20 4.33 3.41 3.93 3.63 3.43 2.93
0.96 0.87 0.85 0.66 1.38 1.02 1.19 1.22 1.02
70.0 86.6 80.0 90.0 51.7 66.7 60.0 56.7 26.7
Conversational
3.07
1.02
30.0
The case-related discussion groups directed their own learning process. I was able to evaluate my own learning during the course. I was able to utilize my prior knowledge about the course topics. The course deepened my understanding of what I had learned before. It was possible for me to study according to my own personal style. I was able to apply my own practical experiences during the course. The students were committed to collaboration. Working in small study groups helped me to learn. Course discussions on chat helped me to learn. Case-related small-group discussions in the discussion area helped me to learn.
Contextual
4.33
0.55
96.7
Case-based teaching helped me to learn.
Goal-oriented
3.93
0.83
76.6
Learning enabled the achievement of my personal goals.
Reflective
4.17
0.87
86.6
I was able to evaluate my own learning during the course.
Abstract
4.30
0.79
86.7
In the course, practical examples were studied in a theoretical framework.
Multiple perspectives oriented
4.17
0.75
86.6
The course helped me to understand different perspectives related to the topics under study.
Process characteristic of meaningful learning
Active Self-directed
Constructive Individual Collaborative
Mean value
Standard deviation
Table 3. Students’ (n=30) ratings (1 = disagree, 2 = moderately disagree, 4 = moderately agree, 5 = agree) of the practical realization of the meaningful learning processes in the network management course
Statement on the questionnaire focusing on the process characteristics
29
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
deviations of their ratings. The students’ positive emotional involvement in solving the DV-supported cases was evident, supporting the perception that case-based teaching is a pedagogical model capable of enhancing learner interest and engagement (see McLellan, 2004). The mean values of the ratings showing positive emotions (Figure 2) were clearly higher than those showing negative emotions (Figure 3). The students reported that their most intense emotions were challenge (M = 3.37, SD = 0.62), interest (M = 3.27, SD = 0.79) and enthusiasm (M = 3.00, SD = 0.79) and that these were associated with the topics of the course, the learning materials, learning tasks, the locally oriented cases, and the small group work. This is an encouraging result, given the crucial role of these emotions for a successful learning process (Engeström, 1982; Puolimatka, 2004). Completing the cases successfully evoked satisfaction (M = 2.77, SD = 0.77), joy (M = 2.70, SD = 0.84) and relief (M = 2.60, SD = 1.04). In sum, the course succeeded in offering the students “plateaus,” or periods when they could complete the assignments, which yielded experiences of success instrumental to sustaining their motivation (see Engeström, 1982, pp. 22−23). Interestingly, even if the students had doubts whether the online discussions helped them learn,
several cited online collaboration as a principal source of joy. Here the research bears out the argument that social interaction is a powerful generator of emotions (Averill, 1982). Collaboration also evoked feelings of trust (M = 2.70, SD = 0.99) in that the students felt their group could be relied upon to complete the assignments. Of the negative emotions (Figure 3), stress (M = 2.59, SD = 0.98) and uncertainty (M = 2.23, SD = 1.17) exhibited the highest intensity, with the students citing the following reasons for their feelings: 1. Tight course schedule (stress, worry, irritation) 2. Quantity of material to cover (stress, feelings of inadequacy, disappointment) 3. Inadequate directions for learning tasks (uncertainty, disappointment) 4. Group dynamics (uncertainty, worry, irritation, feelings of inadequacy, frustration, and tension) 5. Problems with the learning management system (uncertainty, disappointment, irritation) Future implementations of the course must address these concerns. Although the course schedule was identified as one of the main rea-
Figure 2. Mean values of the students’ (n=30) ratings of positive emotions (0 = not at all, 4 = to a great extent) Feelings of challenge
3,37
Interest
3,27
Enthusiasm
3,00 2,77
Satisfaction Trust
2,70
Joy
2,70 2,60
Relief
2,33
Sense of community
2,07
Comfort 0
30
1
2
3
4
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
Figure 3. Mean values of the students’ (n=30) ratings of the negative emotions (0 = not at all, 4 = to a great extent) 2,59
Stress 2,23
Uncertainty 1,87
Frustration
1,83
Worry
1,77
Irritation
1,73
Tension
1,68
Feelings of inadaquacy Disappointment
1,40 1,17
Boredom Dispiritedness
0,63
Embarrassment
0,60 0
sons for negative emotions, we are not convinced that simply stretching the timetable will help. A more effective solution would be to discuss the particular requirements of e-learning, especially the importance of a regular learning schedule, at the very outset of the course. Collaboration also generated intense negative emotions (see also Averill, 1982). Several of the emotions cited, for example, frustration (M = 1.87, SD = 1.14) and tension (M = 1.73, SD = 1.09), stemmed from group dynamics: lack of commitment on the part of others in the group, a lack of trust, divergent working styles and varying effort in tackling the learning task. These findings indicate that successful group dynamics are essential for students’ emotional involvement, signalling in turn that the teacher should ensure the online groups a sound start as a foundation for group dynamics.
Future Trends Although the assessment provided strong evidence concerning students’ perspectives on teaching and on the learning processes, the assessment practices
1
2
3
4
reported here could have been further enhanced. Collecting data from students’ online discussions could have benefited the assessment, as could using the students’ ratings on the questionnaires as a starting point for collecting more data through online interviews (see also Williams, 2005). In addition, online questionnaires could have been used instead of printed questionnaires. The reliability and validity of the entire research questionnaire has not been statistically tested. Only twelve of the twenty-two statements formulated to operationalize meaningful learning processes have been previously tested (see Nevgi & Löfström, 2005; Nevgi & Tirri, 2003). Future research should address this lack and validate the internal consistency of the sub-scales in the questionnaire. As an example, the questionnaire data do not differentiate between students’ collaborative and co-operative learning processes. As Bransford et al. (2001) state, “a challenge for the learning sciences is to provide a theoretical framework that links assessment practices to learning theory” (p. 142). The assessment framework we used was the TML model. How applicable is the TML model? The broad and general nature of the model can be considered as
31
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
both its strength and its weakness. The strength of the TML model lies in its ability to provide a fairly wide and general framework from which to assess learning processes and outcomes within different subject areas and different pedagogical models or approaches. The model is helpful in a process that can be conceptualized as pedagogical profiling, that is, identifying the general pedagogical characteristics, strengths and weaknesses of a given course (Hakkarainen, 2007b). The model can also be thought of as a general snapshot of timely constructivism: it represents what is generally understood as good instructional processes in the 2000s, but cannot be thought of as all-inclusive. Some argue that the TML model is too complex, suggesting that the number of process characteristics and expected outcomes of meaningful learning might be reduced. The results of the assessment do not provide enough evidence for adjusting the model in this respect. One way to further develop the TML model is to organize a long-term teaching experiment on the basis of experimental design, including experiment and control groups, including attitude pre-, post- and delayed tests. Collected data could be analyzed using quantitative methods. Factor analysis could be carried out to see if the number of characteristics can be reduced by combining them. Future research should also address which teaching and learning activities within different pedagogical models are related to characteristics of meaningful learning processes to produce specific learning outcomes.
CONCLUSION The assessment clearly indicates that solving the DV-supported cases promoted the active and contextual characteristics of students’ meaningful learning, as well as the students’ positive emotional involvement in the learning processes. Solving the cases was also associated with the
32
abstract, reflective, and multiple perspectivesoriented characteristics of meaningful learning. Overall, students’ perception of teaching activities was fairly positive. The results point to the pedagogical possibilities of using locally-oriented, case-based teaching in the Network Management course. The critical decisions made by an e-learning teacher concern the assessment of both students’ learning processes and outcomes. The questionnaire we devised and utilized proved a useful tool in gathering evidence of the student perspective on the course’s teaching and learning processes and thereafter deciding on how to revise the course.
REFERENCES Anderson, T., Rourke, L., Garrison, R., & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. JALN Journal of Asynchronous Learning Networks, 5(2). Retrieved from http://www.aln.org/publications/ jaln/v5n2/pdf/v5n2_anderson.pdf Ausubel, D. P., Novak, J. D., & Hanesian, H. (1978). Educational psychology: A cognitive view (2nd ed.). New York: Holt, Rinehart and Winston. Averill, J. R. (1982). Anger and aggression: An essay on emotion. New York: Springer-Verlag. Bandura, A. (1994). Self-efficacy. In V. S. Ramachandran (Ed.), Encyclopedia of human behaviour (Vol. 4, pp. 71−81). New York: Academic Press. Retrieved from http://www.des.emory.edu/mfp/ BanEncy.html Barnes, L. B., Christensen, C. R., & Hansen, A. J. (1994). Teaching with cases at the Harvard Business School. In L. B. Barnes, C. R. Christensen, & A. J. Hansen (Eds.), Teaching and the case method. Text, cases, and readings (3rd ed.) (pp. 34−68). Boston, MA: Harvard Business School Press.
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
Bliss, T., & Reynolds, A. (2004). Quality visions and focused imagination. In J. Brophy (Ed.), Using video in teacher education: Advances in research on teaching, 10, 29−51. Oxford: Elsevier Ltd. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.) (2001). Committee on Developments in the Science of Learning. M. S. Donovan, J. D. Bransford, & J. W. Pellegrino (Eds.), Committee on Learning Research and Educational Practice. How people learn. Brain, mind, experience, and school (Expanded ed.). Washington, DC: National Academy Press. Brockett, R. G., & Hiemstra, R. (1991). Self-direction in adult learning: Perspectives on theory, research, and practice. London: Routledge. Chang, N., & Petersen, N. J. (2005). Cybercoaching: An emerging model of personalized online assessment. In D. D. Williams (Ed.), Online assessment, measurement and evaluation: Emerging practices (pp. 110−130). Hershey, PA: Information Science Publishing. Cohen, L., Manion, L., & Morrison, K. (2003). Research methods in education (5th ed.). London: RoutledgeFalmer. Cottrell, S. (2005). Critical thinking skills: Developing effective analysis and argument. Basingstoke: Palgrave Macmillan. CTGV (Cognition and Technology Group at Vanderbilt). (1993). Designing learning environments that support thinking: The Jasper series as a case study. In T. M. Duffy, J. Lowyck, & D. H. Jonassen (Eds.), Designing environments for constructive learning (pp. 9−36). Berlin: Springer-Verlag. Dekeyser, H. M. (2000). Student preference for verbal, graphic or symbolic information in an independent learning environment for an applied statistics course. In K. Littleton (Series Ed.) & J. F. Rouet, A. Biardeau, & J. J. Levonen (Eds.), Advances in learning and instruction series: Multimedia learning. Cognitive and instructional issues (pp. 99−109). Amsterdam: Pergamon.
Engeström, Y. (1982). Mielekäs oppiminen ja opetus [Meaningful learning and teaching]. Helsinki, Finland: Valtion koulutuskeskus. Grant, G. E. (1992). Using cases to develop teacher knowledge. A cautionary tale. In J. H. Shulman (Ed.), Case methods in teacher education (pp. 211−226). New York: Teachers College Press. Hakkarainen, P. (2007a). Designing and implementing a PBL course on educational digital video production: Lessons learned from a design-based research. Educational Technology Research & Development (ETR&D), Online first article. Retrieved June 12, 2008, from the SpringerLink database. Hakkarainen, P. (2007b). Promoting meaningful learning through the integrated use of digital videos. Doctoral dissertation, University of Lapland. Acta Universitatis Lapponiensis 121. Rovaniemi, Finland: University of Lapland, Faculty of Education. Hakkarainen, P., & Saarelainen, T. (2005a). Producing digital video-supported cases with students − How to make it happen? In H. Ruokamo, P. Hyvönen, M. Lehtonen, & S. Tella (Eds.), Teaching−studying−learning (TSL) processes and mobile technologies − Multi-, inter-, and transdisciplinary (MIT) research approaches. Proceedings of the 12th International NetworkBased Education (NBE) Conference (Former PEG) 2005 (pp. 183−191). University of Lapland Publications in Education 11. Rovaniemi, Finland: University of Lapland, Faculty of Education, Centre for Media Pedagogy. Hakkarainen, P., & Saarelainen, T. (2005b). Towards meaningful learning through designing, producing and solving digital video-supported cases with students. In G. Richards (Ed.), Electronic versions of papers presented at E–Learn 2005 – World Conference on E–Learning in Corporate, Government, Healthcare, and Higher Education (pp. 2081–2088) [CD-ROM]. Association for the Advancement of Computing in Education. 33
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
Hakkarainen, P., Saarelainen, T., & Ruokamo, H. (2007). Towards meaningful learning through digital video supported case based teaching. Australasian Journal of Educational Technology (AJET), 23(1), 87−109. Jonassen, D. H. (1995). Supporting communities of learners with technology: A vision for integrating technology with learning in schools. Educational Technology, 35(4), 60−63. Jonassen, D. H. (2000). Computers as mindtools for schools. Engaging critical thinking. New Jersey: Prentice-Hall. Jonassen, D. H. (2002). Learning as activity. Educational Technology, 42(2), 45−51. Jonassen, D. H., Howland, J., Moore, J., & Marra, M. (2003). Learning to solve problems with technology. A constructivist perspective (2nd ed.). Upper Saddle River, NJ: Merrill Prentice Hall. Joyce, B., & Weil, M. (1980). Models of teaching (2nd ed.). New Jersey: Prentice Hall International. Joyce, B., & Weil, M. (with Showers, B.) (1992). Models of teaching (4th ed.). Boston: Allyn and Bacon. Karppinen, P. (2005). Meaningful learning with digital and online videos: Theoretical perspectives. AACE Journal, 13(3), 233−250. Kort, B., & Reilly, R. (2002). Analytical models of emotions, learning, and relationships: Towards an affective-sensitive cognitive machine. Proceedings of the ITS 2002 − Intelligent Tutoring Systems Conference (pp. 955−962). June 2002, Biarritz, France. Retrieved from http://web.media. mit.edu/~reilly/its2002.pdf Lehtinen, E. (1997). Tietoyhteiskunnan haasteet ja mahdollisuudet oppimiselle [Challenges and possibilities for learning in the information society]. In E. Lehtinen (Ed.), Verkkopedagogiikka (pp. 12−40). Helsinki, Finland: Edita.
34
Mathur, S., & Murray, S. (2005). Authentic assessment online: A practical and theoretical challenge in higher education. In D. D. Williams (Ed.), Online assessment, measurement and evaluation: Emerging practices (pp. 243−258). Hershey, PA: Information Science Publishing. Matuga, J. M. (2005). The role of assessment and evaluation in context: Pedagogical alignment, constraints, and affordances in online courses. In D. D. Williams (Ed.), Online assessment, measurement and evaluation: Emerging practices (pp. 316−330). Hershey, PA: Information Science Publishing. Mayer, R. E. (2003). Multimedia learning. New York: Cambridge University Press. McLellan, H. (2004). The case for case-based teaching in online classes. Educational Technology, 44(4), 14–18. Nevgi, A., & Löfström, E. (2005). The quality of online learning – Teachers’ and students’ evaluations of meaningful learning experiences in online and blended courses [Electronic version]. In S. Kiefer, J. Michalak, A. Sabanci, & K. Winter (Eds.), Analysis of educational policies in a comparative educational perspective (pp. 96–104). Linz, Austria: Pädagogische Hochschule des Bundes in Öberösterreich. Nevgi, A., & Tirri, K. (2003). Hyvää verkko-opetusta etsimässä. Oppimista edistävät ja estävät tekijät verkko-oppimisympäristöissä – opiskelijoiden kokemukset ja opettajien arviot [In search of good online teaching. Factors promoting and hindering learning in online learning environments – Students’ experiences and teachers’ evaluations]. Kasvatusalan tutkimuksia 15. Turku, Finland: Suomen Kasvatustieteellinen Seura. Novak, J. D. (1998). Learning, creating, and using knowledge: Concept maps as facilitative tools in schools and corporations. Mahwah, NJ: Lawrence Erlbaum Associates.
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
Op’t Eynde, P., De Corte, E., & Verschaffel, L. (2001). “What to learn from what we feel”: The role of students’ emotions in the mathematics classroom. In S. Volet & S. Järvelä (Eds.), Motivation in learning contexts. Theoretical advances and methodological implications (pp. 149–167). Amsterdam: Pergamon. Poikela, E. (2006). Knowledge, knowing and problem-based learning – Some epistemological and ontological remarks. In E. Poikela & A. R. Nummenmaa (Eds.), Understanding problembased learning (pp. 15–31). Tampere, Finland: Tampere University Press. Poikela, E., & Nummenmaa, A. R. (Eds.). (2006). Understanding problem-based learning. Tampere, Finland: Tampere University Press. Poikela, E., & Poikela, S. (2006). Developing context-based assessment within the framework of problem-based learning. In E. Poikela & A. R. Nummenmaa (Eds.), Understanding problembased learning (pp. 227−240). Tampere, Finland: Tampere University Press. Puolimatka, T. (2004). Tunteiden kognitiivisuus ja oppiminen [Cognitivity of emotions and learning]. Aikuiskasvatus [The Finnish Journal of Adult Education], 2/2004, 102−110. Repo-Kaarento, S. (2004). Yhteisöllistä ja yhteistoiminnallista oppimista yliopistoon –Käsitteiden tarkastelua ja sovellutusten kehittelyä [Introducing collaborative and cooperative learning at university – Looking at the concepts and developing applications]. Kasvatus [The Finnish Journal of Education], 35(5), 499−515. Ruokamo, H. (2001). The Solver learning environment and anchored instruction on mathematical word problem-solving. In C. Montgomerie & J. Viteli (Eds.), Proceedings of ED-MEDIA 2001 – World Conference on Educational Multimedia, Hypermedia & Telecommunications (pp. 1595–1600) [CD-ROM]. Association for the Advancement of Computing in Education.
Ruokamo, H., & Pohjolainen, S. (2000). Distance learning in multimedia networks project: Main results. British Journal of Educational Technology, 31(2), 117–125. Ruokamo, H., Tella, S., Vahtivuori, S., Tuovinen, H., & Tissari, V. (2002). Pedagogical models in the design and assessment of network-based education. In P. Barker & S. Rebelsky (Eds.), Proceedings of ED-MEDIA 2002 – World Conference on Educational Multimedia, Hypermedia & Telecommunications (pp. 1676–1681) [CD-ROM]. Association for the Advancement of Computing in Education. Schedler, K., Summermatter, L., & Schmidt, B. (2004). Managing the electronic government. From vision to practice. A volume in research in public management. Greenwich: Information Age Publishing Inc. Schutz, P. A., & DeCuir, J. T. (2002). Inquiry on emotions in education. Educational Psychologist, 37(2), 125–134. Schwartz, D. L., & Hartman, K. (2007). It’s not video anymore: Designing digital video for learning and assessment. In R. Goldman, R. Pea, B. Barron, & S. J. Derry (Eds.), Video research in the learning sciences (pp. 335−348). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers. Shulman, L. S. (1992). Toward a pedagogy of cases. In J. H. Shulman (Ed.), Case methods in teacher education (pp. 1−30). New York: Teachers College Press. Soini, H. (1999). Education students’ experiences of learning and their conceptions about learning disabilities. Towards a comprehensive theory of learning. Doctoral dissertation, University of Oulu. Acta Universitatis Ouluensis Scientiae Rerum Socialium 40. Oulu, Finland: Oulu University Press. Spiro, R. J., Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1992). Cognitive flexibility, constructivism and hypertext: Random access 35
Assessing Teaching and Students’ Meaningful Learning Processes in an E-Learning Course
instruction for advanced knowledge acquisition in ill-structured domains. In T. M. Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction: A conversation (pp. 57–75). Hillsdale, NJ: Lawrence Erlbaum Associates. Stenhouse, L. (1989). The case-study tradition and how case studies apply to practice. In J. Rudduck & D. Hopkins (Eds.), Research as a basis for teaching. Readings from the work of Lawrence Stenhouse (pp. 52–55). Oxford: Heinemann Educational Books. Tynjälä, P. (2001). Writing, learning and the development of expertise in higher education. In P. Tynjälä, L. Mason, & K. Lonka (Eds.), Writing as a learning tool. Integrating theory and practice (pp. 37−56). Dordrecht: Kluwer Academic Publishers. Vahtivuori, S., Tella, S., Tissari, V., Tuovinen, H., Ruokamo, H., & Vaattovaara, V. (2003). First evaluations of network-based education: Preliminary findings of the ICT programs evaluation project. In D. Lassner & C. McNaught (Eds.), Proceedings of ED-MEDIA 2003 Conference – World Conference on Educational Multimedia, Hypermedia &
36
Telecommunications (pp. 3177–3183) [CD-ROM]. Association for the Advancement of Computing in Education. Wiecha, J. M., Gramling, R., Joachim, P., & Vanderschmidt, H. (2003). Collaborative e-learning using streaming video and asynchronous discussion boards to teach the cognitive foundation of medical interviewing: A case study. Journal of Medical Internet Research, 5(2). Retrieved from http://www.jmir.org/2003/2/e13/ Williams, D. D. (2005). Measurement and assessment supporting evaluation in online settings. In D. D. Williams (Ed.), Online assessment, measurement and evaluation: Emerging practices (pp. 1−8). Hershey, PA: Information Science Publishing.
ENDNOTES
a
b
See http://www.discendum.com/english/ For a more detailed description, see (Hakkarainen, 2007a, 2007b) and (Hakkarainen et al., 2007).
37
Chapter III
Collaborative E-Learning Using Wikis: A Case Report Charlotte Brack Monash University, Australia
ABSTRACT Within the notion of Web 2.0, social software has characteristics that make it particularly relevant to ELearning, aligning well with a social constructivist approach to learning encompassing peer production and review of curriculum content. In this chapter examples of the use of social software are described and ways in which they draw on these characteristics are discussed. An intensive three week program of study online has been developed for students who have not completed year 12 biology and who are entering first year medical studies. The program incorporates individual self-directed learning and collaborative projects. The latter include a debate and competition between groups involving the creation of a Wiki. Social software, the Wiki, enabled peer learning activities including peer support, and the shared peer production and review of learning materials. The social software also provided mechanisms for developing generic skills and contributed to transition issues for new students. Incorporation of social bookmarking into the project may extend the potential for connected knowledge with a broader concept of peers and peer assessment.
INTRODUCTION Students entering the undergraduate medical course at Monash University are not required to have completed year 12 Biology. While this may
allow students to have a broad background it can put those who have not studied biology at a disadvantage in their first year of studying medicine at university. Evaluation data in the medical school revealed that these students reported feeling out
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Collaborative E-Learning Using Wikis
of their depth and bewildered by terminology and biological concepts with which other students appeared to be familiar. To support this group of students a bridging program covering relevant parts of year 12 Biology was developed. The timetable for students in first semester does not allow for scheduling of the bridging program during semester so an intensive program was designed for students prior to commencement of their medical studies. The time between acceptance of offers and the start of semester one is in the order of four weeks so the duration of the bridging program had to be limited to this period. Many students are committed to paid employment or personal activities which make access to the university campus prior to the start of semester difficult. Students may also be overseas at this time as is the case for many international students. The program therefore had to be not only intensive and prior to first semester but had to be offered online. The recent expansion of Web 2.0 technologies and social software presented an opportunity to design an engaging online learning environment for students. The design needed to take into account that a large majority of the students taking the program were in transition to university and had not yet had experience of university life, nor had they had an opportunity to meet each other. Issues of transition to higher education for first year students are of concern to higher education institutions. Students often struggle with large classes and feelings of anonymity (Peel, 2000; Krause, Hartley, James & McInnis, 2005). At an institutional level strategies to address this include orientation courses and clubs focused on leisure and academic pursuits. Within courses transition camps are effective, and incorporation of group work both formally in course units of study and through encouraging and facilitating informal study groups are also valuable strategies. Through the use of the social software of Web 2.0 technologies the bridging program offered an
38
opportunity to contribute to resolving transition issues for students, from both academic and social perspectives. The software allows students to work collaboratively in groups as it integrates communication with creation of content by students. Web 2.0 as a term is nebulous. In coining the term O’Reilly (2005) describes it as a “gravitational core”, visualising “… a set of principles and practices that tie together a veritable solar system of sites that demonstrate some or all of those principles, at a varying distance from that core.” In order to deal with the lack of clarity the term ‘affordances’ is often used in relation to Web 2.0 technologies to describe functionality. Among the principles and practices of affordances are those categorised as social software because they provide opportunities for online communication and collaboration. While opportunities for communication and collaboration via the internet have been available for decades new technologies go far beyond email and web pages. Social software includes blogs, wikis, podcasting, vodcasting (video podcasting), and social networking tools such as MySpace and Facebook. It revolves around enabling users to modify content. Within the possibilities of the Web 2.0 sphere, social software facilitates collaborative research and synthesis of ideas in the way in which ‘web pages’ (with content) are constructed and integrated with communication. A trail of interactions is recorded including interactions between people (in our case, students), with media and with content. This allows teachers to observe and assess the collaborative process as well as the product being developed thereby opening up avenues for assessment of collaborative group work. In this chapter a case study of an E-Learning program incorporating a collaborative project using social software for first year students is described. The affordances of the software are discussed in terms of learning at multiple levels, assessment and peer support, including transition issues.
Collaborative E-Learning Using Wikis
Ba Collaborative Learning and Learning to Collaborate Social constructivism is a dominant theory of learning in higher education. Derived from ideas of Vygotsky (1978) it describes learning as a social process for which the context of learning is significant. The social context for such learning is often provided by groups of students working collaboratively. Bower and Richards (2006) considered different approaches to collaborative learning and concluded that its success depended on the ways in which it was implemented. A context for implementation is provided through the concept of a ‘community of practice’ described as an environment in which a group of individuals participate in shared activity around a domain of knowledge, where learners become self-regulated and independent (Lave, 1993). In addition to the social constructivist benefits for learning within a domain of knowledge, the opportunities for development of collaborative skills are significant. Many higher education institutes include excellence in skills of collaboration in their statement of graduate attributes (Barrie, 2004). In addition to being invaluable life skills, the capacity to collaborate is also highly valued in most professions, and institutions want to be able to demonstrate to employers that their graduates have these skills. Most contemporary university programs would include the demonstration of group work and collaboration skills as explicit learning objectives and course outcome goals.
Social Software and Collaborative Learning Social software is particularly adaptable to supporting a community of practice providing an environment in which social interaction and en-
gagement with a common interest can occur with ease. As Leslie and Landon (2008) write: ... social software is extremely well suited to enable learning, that in emphasising users’identities, connecting them with each other and helping network level value emerge out of individual action, …(it) addresses many of the critical stumbling blocks which have plagued earlier E-Learning efforts (p. 3). They consider social software in terms of contemporary theories of learning, describing the relevance of characteristics of: communication; tapping into the user’s motivation; authentic online identity and learning experience; networks of affinity (the power of groups) and the emergence of connected knowledge; and peer production and review. The idea of authenticity of online identity contributes to issues of power and control leading to responsibility and ownership which are significant motivators for group work. Dron (2007) discusses social software for collaborative learning in terms of transactional distance and transactional control and the ways in which the software allows for the emergent control of the group. He suggests that “transactional control is concerned with choices” going on to argue “Structure equates to teacher control, dialogue to negotiated control, and autonomy to learner control. At any point in a learning trajectory, transactional control will vary.” (p. 60). This helps inform educational design based on a constructivist approach with an emphasis on social constructivism (Honebein, Duffy, & Fishman, 1993). Among the possibilities of social software wikis specifically enable group output. In reviewing literature on wiki use in education Parker and Chao (2007) suggest that the most powerful uses have been in supporting cooperative/collaborative and constructivist approaches to learning in particular.
39
Collaborative E-Learning Using Wikis
Collaborative Learning and Peer Assessment
others (See also in this text, Chapter Z: Tucker, Fermelis & Palmer).
Despite its value for learning, students often dislike group work which does not recognise different levels of contributions from individuals (Gatfield, 1999; Falchikov, 2005). Studies looking at the experience of group work among first year students showed an acceptance of the value for learning, but concern over the issue of ‘passengers’ (Bourner, Hughes & Bourner, 2001); that is group participants who do not contribute purposefully to the learning activities but succeed based on the group achievement. Peer assessment where students rate the contributions of individual members of their group has been used successfully to address this with student empowerment and levels of satisfaction increasing (Gatfield, 1999), but it is controversial when used for summative assessment (Kennedy, 2005; Li, 2001). A normalisation procedure to reduce bias in peer assessment was devised to address the controversy, but failed to be the ultimate tool (Li, 2001). In studying peer assessment approaches Kennedy (2005) found evidence of tension within groups resulting from a requirement that students assess each other and further, concluded that it did little to improve discrimination between student grades. (Lajbcygier & Spratt (Chapter 8) and Benson (Chapter 7) discuss some of these issues in this volume.) However, considering peer assessment from the perspective of learning Raban and Litchfield (2007) demonstrated its value at multiple levels. They describe an online strategy where students rate and comment on each other repeatedly, which “creates a formative, diagnostic and summative assessment environment in which students can learn the skills of peer assessing” (p.35). Gibbs and Simpson (2004) looked extensively at the role of assessment in learning, concluding that peer evaluation and assessment of project output (rather than contribution of individuals) contributes to learning as students critique the work of
Social Software and Assessment
40
In considering issues of assessment, a question arises: if social software is based on the identity of the group how can individuals be assessed in a higher education context without compromising the integrity and identity of the group? The software itself responds to this due to its stigmergic nature; like the pattern left by ants from which the term is derived, the trail left by contributors, for instance in wikis, reveals an individual’s input. In assessing group project work using wikis in medical and biomedical units, history and discussion functions were used to track contributions of individuals (Brack, Stauder, Doery & Van Damme, 2007). Monitoring of discussion posts also allowed for assessment of process which has in the past presented difficulties (Kennedy, 2005). The development of the community of practice, with its associated social context, project management and problem solving strategies are observable without the need for students to self report or assess their peers.
Social Software and Transition to Uiversity Concepts of social software also suggest a role in addressing transition issues for new students. Dickey (2004) found that the use of blogs for discourse in small group learning communities afforded opportunities to socialise, interact and enter into dialogue, seek support and assistance and express feelings and emotions, and that this helped bridge or prevent feelings of isolation. In the case presented in this chapter the facility of social software to bring together the centrality of the learner with the social nature of the construction of knowledge is harnessed. The level and nature of discourse are discussed in terms of how student projects using wikis reflect the
Collaborative E-Learning Using Wikis
characteristics of other social software uses in the public domain (e.g., Wikipedia, Flickr), including the degree to which students form communities of enquiry and of practice, the nature of the discourse and the dynamics of the interactions. Students may appreciate the flexibility of the online environment for group work (Brack et al., 2007) but does it result in useful learning? And is there evidence that it addresses issues of isolation and transition to university studies? The educational design informing the work of this chapter draws on social constructivism and incorporates individual self-directed learning and collaborative projects.
Educational Desii for E -Leaai We designed and developed a Biology Bridging Program, which we called Leapfrog Biology. In its first offering the program was conducted online
through the university’s existing learning management system (LMS, Blackboard) over three weeks in February 2006. In subsequent years the program was extended by three days to optimise the opportunities for students to engage in the program. The program was not compulsory but was highly recommended to all students who were entering first year of the medical course but who had not done year 12 Biology. Students ‘signedup’ for the program at the time of the face-to-face information session for new students held in late January. There was no opportunity to brief them at this stage but they received an outline of the bridging program including the schedule, time commitments required and details for access to the online site. With no further opportunities to meet face-to-face, briefing and orientation sessions were held online in ‘getting started’ (Figure 1). This included an overview of the program, objectives, expectations and introductions. Benefits of adhering to the schedule were stressed.
Figure 1. The homepage of Leapfrog Biology
41
Collaborative E-Learning Using Wikis
Modular Structure Leapfrog Biology was developed as three thematic modules and students complete a module each week of the program. The modules are: (1) The cellular basis of life, (2) Human genetics, and (3) Disease and immunity. Students were advised to complete one module per week requiring an estimated commitment of three hours per day. Each module consisted of a series of activities supported by a text resource (as pdf files) written specifically for the program by Monash academics. There are activities for students to complete individually and some to participate in as a
group. The former, designed for learning and self-assessment, include interactive multimedia elements (drag-and-drop, animations), quizzes and summary submissions. The feedback given for these activities done by students individually was automated, quizzes also allowed us to monitor students’ progress. The group activities comprised a debate and a group project. Table 1 indicates the activities available to students. In ‘getting started’ (Figure 1) students were introduced to the first group activity, the debate in which they discuss a topic related to the subject of the second group activity in which students create a wiki. Students were randomly allocated
Table 1. Activities in Leapfrog modules Module 1 - The Cellular Basis of Life
Module 2 - Human Genetics
1. 2. 3.
Features of eukaryote cells Quiz - The nature of cells Cell organelles *
4.
Structure of eukaryotic cells
Structure of DNA and RNA Mitosis Quiz- Chromosomes, genetic cycle and cell cycle Transcription
5.
Quiz - Structure and function of cell organelles Quiz - Chemical components of membranes Movement across cell membrane Diffusion: Passive transport* Diffusion: Active transport* Quiz - Diffusion, osmosis and active transport. Submit summary of Part A Quiz - Cell organelles The active site of an enzyme* Quiz - Aerobic and anaerobic metabolism
6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21.
Submit summary of Part B Quiz - Enzymes and energy in chemical bonds The Nobel Factor
Deciphering the genetic code*
Quiz - Microorganisms and tissue damage Meet the scientist C
Translation
Meet the scientist D
Quiz - Making the protein Meiosis Summary of Part A Quiz - Producing gametes
Meet the scientist E Meet the scientist F Summary of part A Quiz - Disease causing organisms
Mendel’s laws* Mendel’s Laws continued* Summary of part B Quiz - Mendel’s experiments and genes in action The Nobel Factor
Cells of the immune system* Quiz - Cellular components of blood Natural defences against disease* Meet the scientist G
Note.* these activities are linked to The Lifewire (www.thelifewire.com)
42
Module 3 - Infection, Disease and Immunity Meet the scientist A Polymerase Chain Reaction* Meet the scientist B
Immunoglobulins* Components associated with inflammation* Meet the scientist H Cells of the immune system Summary of Part B Quiz - Defence systems in animals The Nobel Factor
Collaborative E-Learning Using Wikis
into groups of twelve. They used a group discussion space to introduce themselves to other group members, identify issues and discuss both sides of the debate topic. The debate was guided by four questions and students were advised to arrive at a consensus view of the topic. The group discussions were not moderated by teachers, but students were given guidelines for working in groups and project management.
The Group Project The group project, called ‘the Nobel Factor’ was a set up as a competition between groups. Students designed and developed a wiki exploring Nobel Prize winning discoveries in medical research related to the subject of each of the modules. The final activity in each module describes the required contribution to the wiki. For example, the final activity in ‘Module 1: The cellular basis of life’ is to contribute to the group wiki an overview of the clinical uses of stem cells with reference to a number of specific issues including those for which a Nobel prize has been awarded. In ‘Module 2: Human Genetics’ students explore inherited diseases treatable with stem cells; and in ‘Module 3: Disease and Immunity’ the contribution to the wiki is on prize winning work on stems cells in immunology. Individual activities in module 3 called ‘Meet the scientist’ asked students to research the work of key workers in the field. Students were encouraged to use the output of those activities for their project.
The Wiki A private wiki was set up for each group in ‘Wikispaces’ (www.wikispaces.com). While Wikispaces is ‘lightweight’ compared to some other implementations of the concept it was chosen because of its simplicity, ease of use , rapidly responding ‘helpdesk’ and availability of private spaces (members only ‘read’ and ‘write’) at no costs for one month. Students controlled
membership of their wikis and could open them for all to read (i.e., make them public) at the end of the project when viewing by the students in other groups was required. The markup language was easy to master, tolerated some use of html formatting and was supported with a good basic WYSIWYG editor; it allowed uploads of images and attachment of files, had adequate data storage capacity, easy file management, accessible version control, and discussion spaces that looked like ‘discussion spaces’ (i.e. distinct from ‘content’ pages, and threaded). It was also important that wikis had minimal corporate branding so students could personalise wikis and gain optimal ownership of their project. Wikis becoming available in existing LMSs offer an advantage of incorporation with other learning materials and functions but are identified with the course rather than the group owners. Instructions for joining, signing up and using the wikis were posted in the LMS site (in the ‘getting started’ section). Two synchronous chat sessions (via the LMS) were scheduled in the first week of the program to support students in using wiki technology. The wikis (the product of the group learning experiences) were assessed by teachers and students against criteria relating to content and presentation (Table 2). The process was also assessed by two teachers for project management and collaboration. Students were advised (a) to conduct all communication relating to the project in the discussion space in the wiki so teachers could observe interactions and project management strategies, and (b) to do all editing within the wikis so development of content and individual contributions were evident through page histories. The appearance of a slab of text, which may have been copied from elsewhere, was viewed less favourably than text developed by more than one author. The results of the competition had no consequences in terms of student grades, its purpose being to provide motivation for the project; however the teachers’ assessments of product and process allowed constructive feedback to be given to students. 43
Collaborative E-Learning Using Wikis
Students voted online for the best wiki (excluding their own wiki), the number of student votes for each wiki was added to the score given by teachers. Modeled on television competition programs, where there is a tie in the marks the adjudicators make the final decision. Members of the winning wiki receive a certificate and a prize awarded in one of the early lectures for first year medical students. The best wikis are published so all first year medical students can use them as resources during their first semester studies.
Upaf the Program In the intake of students to the first year medical degree course typically thirty to forty percent of the students in any one year will not have taken Biology at year 12 level and enroll in Leapfrog Biology (Table 3). Of this cohort approximately a quarter of students notify us that they are unable to participate in group activities due to prior commitments or access issues. These students have access to the resources and are able to complete individual activities at times that suit them. After the program is completed by the enrolled students, resources (texts and individual activities) are made available to all first year medical students. Several other course convenors have requested access for their students and these cohorts of students have also been given access. Students actively engaged in the debate in the first days of the program. While there were no
specific ice-breaking activities students used the debate discussion space to introduce themselves. The majority of students engage in the debate discussion space (Table 4), with a small proportion (22% of those engaging in 2008) only contributing social posts, while the others also contributed substantively to discussion of the set curriculum topic. The connections with their peers that the social discussion enabled them to make were on issues of where they were currently located, where they were going to live once semester started, their current employment and other constraints on their time. They discuss moving from interstate or overseas, often offering each other advice and support, setting up networks for the future. Those who also debated the topic developed a framework for their project and started research for their wiki. The debate was often vigorous, with students offering alternate views (being “… devil’s advocate”) and gathering information from newspapers (national and international) and the Internet. In 2008 of the 105 students enrolled in the program 97 (92%) completed the online individual activities and 85 (81%) also completed the group activities.An informal survey of students indicated that while a large majority of students had used a wiki (e.g. Wikipedia) they had not contributed to one, nor did they understand its functionality beyond that of a website, nor its affordances for group work. Students did not appear to have difficulty accessing their wikis. The synchronous chat session with the teacher on day three of the
Table 2. The criteria used to grade wikis Crietria Content:
equal weight to: accuracy and completeness (investigative skills), clarity and succinctness (no plagiarism!) Presentation: equal weight to: clarity, site layout (navigation), interface design Group work: equal weight to: project management (roles/responsibilities, timelines), communication skills (discussion, presentation), problem solving (in working as a group)
44
Proportion of mark 40%
30% 30%
Collaborative E-Learning Using Wikis
Table 3. Student enrolments Number of students Enrolled in program Enrolled in Leapfrog Biology
2006 264 87
2007 310 126
2008 296 105
Table 4. Participation in social software opportunities In 2008 Engaging in debate discussion space: • total • socially only • substantively Signing up for wiki Engaging in wiki
Number of studentsa
% of enrolled
76 17 59 85 53b
72 16 56 81 50
Note 105 students enrolled for Leapfrog Biology in 2008 b 62% of students who signed up for wikis engaged with them a
program ensured that groups were able to manage the basic editing functions, communications and to use the history to record contributions and retain copies of all edits. The chat sessions also enabled the teacher to get to know some of the students. While only eleven students participated in the first chat session those who did found it very useful and requested further sessions. There was considerable sharing of information and resources among students during the sessions indicating an environment of significant peer support.
Eation In 2008, twenty three students completed the online program evaluation survey. A majority of students found the online activities helpful in understanding the study material. In completing module 1, 74% of students surveyed found the activities very or somewhat helpful; for Module 2, the figure was 70%; and for Module 3, 52% of students found them helpful. Students commented that as the program progressed they had less time to participate in the E-Learning activities. Nine
of the twenty-three students surveyed found the group work valuable. In describing the value they made comments such as: “yes the interaction was very enjoyable. Made the course a lot more fun.” “ …so it was very much a discussed creation.” “I found the group interactions insightful …” “It was an odd feeling because I have never been a part of an online syndicate before, but it was quite reassuring to know that other people were also doing the course, so the idea of working on a computer for 4 weeks was more appealing. Also, being in a group meant that people all actively took part in the majority of activities.” One student commented that it was not as useful for transition issues as the transition camp outlined earlier, which was run the week before the start of semester. Two students commented that the group work was the best aspect of the program. The debate was viewed as helpful in establishing the group and its capacity to develop ideas, for example:
45
Collaborative E-Learning Using Wikis
“The debate started us off well.” ”it was good to discuss different ideas and opinions and share research.” The Nobel Factor project stimulated the interest of most students. “Yes, I found that applying the knowledge learnt, combined with added research made the biology course stimulating and it was interesting to research Nobel prize winners in conjunction with other areas.” Negative comments about group work related mostly to time issues for example where prior commitments made it difficult for some students to participate.
Thoup Product: The Cllaborative Wiki In each of the three years we have offered the program there have been three or four outstanding wikis where accurate information on stem cells has been concisely presented and debated. Figure 2 shows the homepage of the winning wiki in 2008. Analysis of the discussion associated with the homepage of most of the wikis revealed the level to which groups managed their projects. In general they used the space well to negotiate roles and responsibilities and schedule tasks. As advised, groups generally created a structure to their wiki with separate pages for different sections linked via a menu. Wikis had varying degrees of complexity, with most adopting the strategy of allocating a separate page for each ‘Nobel Factor’ activity, with pages linked via a side menu. Site were generally easy to navigate and comprehensive, although the contributions tapered off towards the end of the program as students prepared for semester. Most of the discussion occurred on the home page but was related to the site as a whole.
46
Developing the E -Learnii Cmmunii The formation of communities of enquiry is evident in the contributions to the debate and the wiki discussions. For example, one student acknowledges contributions of the group, illustrating student-group interactions, by writing: “well i dont know too much about this kind of stuff but your posts have taught me a bit so thanks” Another student picks up an argument indicating considerable reflective ability and demonstrating how students are using discussion to negotiate subtleties of meaning, in writing: “I’m glad you mentioned the idea of stem-cell research challenging the meaning of being human, because it made me think of mortality and immortality.” The discussions also contributed to a community of practice when the group engaged in discussion about how they will go about their project. In the 2008 evaluation survey students indicated that the debate helped them learn how to engage online explaining: “It did as our group all contributed to the topic and noone was afraid to voice opposing opinions, so we tackled difficult ethical situations with input from everyone.” And indicating its contribution to process with: “yes, it was a good start for getting used to the format” The ‘community’ facilitated through the discussions served to motivate students, assist in problem solving, and provide peer support (Figure 3).
Collaborative E-Learning Using Wikis
Figure 2. Homepage of a Winning Student Wiki
The discussion is informal and supportive often rapidly moving between content related issues and peer support issues.
Peer Productiind Review of Learaieriences The wikis were initially populated with ideas from the debate. Ideas entered as disjointed pieces of information took shape as students edited and refined their work. Individual students were not solely responsible for any part of the wiki and a large part of their work was based on that of a previous contributor. By editing the work of other group members students were informally assessing the work of their peers. Thus the model of peer production and peer review was facilitated by the software.
Evidence of peer review was found in the content of discussion posts but was also reflected in student responses in the evaluation survey. Responding to “Were the interactions with fellow students in your group valuable to you (explain)?” students wrote: “Quite good – it helped share the work and decide what research we needed to do” “The debate started us off well. A few people had info which related to the various sub-topics, and I think those eventually went to the wiki. Interactions were valuable, as they also allowed some people to discuss the modules.” “Our syndicate group involved anyone placing their ideas on the Wiki and then eveyone adding extra comments or ideas in order to strengthen their argument. People were very polite and wor-
47
Collaborative E-Learning Using Wikis
Figure 3. Discussion in a wiki
ried about overriding other people, so it was very much a discussed creation.”
Motivation and Group Learnii
The comments made in the evaluation indicate that discussions provided an opportunity for peer feedback and assessment of quality of content and understanding of concepts. From a perspective of learning this was valuable for students as they discussed the material each contributed. It was also valuable for teachers as it provided an opportunity to uncover misconceptions. Teachers did not engage in the wiki discussion but monitored them to give support if groups were floundering. As Dron (2007) suggests “While teachers may have a relatively small role in the form that a social software system may take, social software can tell the teacher a great deal about the group: their preferences, their interests, their needs, their weaknesses, their strengths.” (p. 62).
In response to the question “Did the Nobel Factor stimulate your interest in scientific discovery?” students’ comments included:
48
“Yes, I found that applying the knowledge learnt, combined with added research made the biology course stimulating and it was interesting to research Nobel prize winners in conjunction with other areas.” “not really it provided a distraction, and took away from my interest and my motivation [i] did however like the idea of debating in a group” The latter comment is interesting, possibly reflecting the competitiveness of the student who
Collaborative E-Learning Using Wikis
is interested in ideas but not in following through to a phase of production. Some students enjoyed and appreciated the support within the group. In regard to working in a group online one student wrote: “For my small part [i] enjoyed working with fellow students.” “yes the interaction was very enjoyable. Made the course a lot more fun.” “It was an odd feeling because I have never been a part of an online syndicate before, but it was quite reassuring to know that other people were also doing the course, so the idea of working on a computer for 4 weeks was more appealing. Also, being in a group meant that people all actively took part in the majority of activities.” Many students appreciated the flexibility offered by the online environment, although some would have preferred to work face-to-face. Negative responses to group work and the online environment related mostly to the time taken to engage with the technology and the project. Farmer, Yue and Brooks (2008) found variation in student responses to the use of social software (blogs in their study) and attributed it in large part to prior experience with the software. Support around familiarity with software and engagement was seen as addressing the variation. In Leapfrog Biology to date students generally have not had prior experience in contributing to wikis although many have had experience with other forms of social software.
Social Software: Its Affordances for theh Program Goup Identity As an initial step towards formation of a functional group students introduced themselves,
creating individual identities. They used the wikis to varying degrees with some personalising the environment creating an avatar complete with image and profile. The identities of groups emerged as students started working together. Group identities were evident in the wikis, with most displaying a graphic homepage for their wiki giving it a distinct identity and often a catchy group name and icon. The online identities of groups and their functionality developed over the time span of the project through individual input and group output. Initially students expressed anxiety about overriding the input of others, one student comments: “Some members were particularly reluctant to edit the wiki because they didnt want to remove or change others work. That made things slow and confusing.” This aspect of using wikis has been observed by many teachers (Parker & Chao, 2007). It suggests that the transition from identity as individuals to that of a group takes time and possibly requires a conceptual shift. Students became more engaged with the wiki and the group output as they gained confidence in the technology, which included the ability to revert to previous versions. Acknowledgment of this aspect of working with wikis early in the program may facilitate identification with the group and thus enhance the progress of the wikis. Dron (2007) suggests that modes of interaction that have become more significant in the social software environment are those to do with the group, including student-group, teacher-group, content-group and group-group. The group has an identity apart from individuals within the group. This is consistent with views of Shirky (2008) who asserts that the user of social software is the group rather than the individual, and drawing value from it requires developing group identity and functionality. As soon as a contribution is made it no longer belongs to the individual but
49
Collaborative E-Learning Using Wikis
to the collective. Shirky (2008) discusses ideas of collective action. He describes a progression from sharing (an aggregate of participants), to cooperation (which creates group identity), collaborative production (requiring collective decisions) to collective action. Collaborative production increases the tension between individual and group goals, because individuals cannot take credit for what is created. The benefit to the individual must be evident. By the end of the wiki project students had been able to identify with group goals and work together towards them. Some groups devoted much time to editing and refining text and improving the wiki product.
Functioning as a Group Groups tended to manage the tasks required for the project on a volunteer basis, a first-come-firstserve system where students followed up the areas in which they were interested. Some volunteered for research tasks, others for compiling and editing, yet others for graphics. As noted previously once students had declared their interests they used the discussion to negotiate responsibilities and timelines. In the first offering of the program (in 2006) students used blogs to debate the topic. While some useful information was posted, debate and discussion was limited in quantity. In subsequent offerings the discussion functionality of the LMS was used and debate and discussion flourished. The reason for this is not clear (and was not formally investigated as part of any evaluation); however it is possible that the ‘post and comment’ format of the blog appeared to be a more formal commitment and the discussion space in the LMS was a safer place to reveal ideas and challenge others. Indeed, in discussing the design of social software for education Dron (2007) suggests that single purpose software lacks the adaptability required to make changes at a local level which can fit the needs of the group. He is talking about the design of the software itself but it applies
50
in the design of the educational applications of the software, the way in which we put elements together to facilitate group learning. In discussing why individuals contribute to wikis, Shirky (2008) describes how Ward Cunningham, who created the first wiki in 1995 (still accessible from http://c2.com/) made the assumption that groups of people who wanted to collaborate also tend to trust one another, the consequence of this would be that a small group could work on a shared writing effort without needing formal management or process. The control is placed in the hands of the users rather than embedding it in the tool. Through analysis of Wikipedia Shirky suggests that wikis reward those who invest in improving them, describing the “Kilroy was here” feeling of success from contributing (p. 132), as evidence of existence and potency. The ways in which individuals contributed suggests that students readily took control of and invested in their wikis. The wikis produced by students were remarkably well developed considering the program itself was optional. As table 5 indicates that 81% of students enrolled in the program signed up for their group’s wiki and of 62% of those students engaged significantly in the group activity of building the wiki. Analysis of the numbers of contributors and contributions in wikis indicated a general trend which accords with that observed in other social software sites such as Flickr and Wikipedia (Shirky, 2008). The shape of the distribution of contributions versus contributors follows a power law distribution. Such a distribution has long been recognised in the social sciences (in particular in relation to distribution of wealth) where it has been termed “a predictable imbalance” (p.122). If equivalence of contribution is required in student project wikis then this presents a problem. However, Shirky (2008) discusses the evidence that the imbalance drives the social systems. In designing student projects using wikis this is worth considering and asking the question about where the values in using the software lie for any particular application.
Collaborative E-Learning Using Wikis
Assessing the Group Wiki and Collaboration The students in the program were not formally assessed as the program was optional. However, the Nobel Factor was a competition and as such, the wikis themselves were assessed for content (accuracy, completeness, clarity and succinctness) and presentation (clarity, layout, and graphic interface design) by staff and students individually. In each of the three offerings of the program there has been agreement between staff and student assessments. Staff also assessed the collaboration of students within groups by considering the discussions (evidence of project management, problem solving, and communication skills). The stigmergic nature of the wikis allowed for assessment of group work including project management, problem solving and peer support via the discussion and history pages. In graded projects we have used these functionalities to assess individual contributions and adjust the group mark accordingly (Brack et al., 2007). Students reported increased satisfaction with group work as a result. A tension between cooperation and competition arises when considering the characteristics and nature of medical students. They have usually gained entry to a medical course by achieving high scores in previous study in a highly competitive environment. Their competitive nature is a valuable motivator, and finding ways to use and reward it within the social environment of group work helps to resolve the tensions giving space for both cooperation and competition. Staging of phases where students cooperate within a group, then groups compete with each other allows such space. This regards the use of social software in a context which addresses multiple issues without being solely responsive to one issue. Akin to the stability of diversity in biological systems, the strength of the environment is enhanced by multiple, even apparently contradictory ways in
which social software is used, its output or in the design of the software itself (Dron, 2007). An optional program offered out of semester time before students have entered the university did not initially seem to have the ingredients to be attractive to students. However, the medical students for whom Leapfrog Biology was designed are highly motivated and competitive and that provided an opportunity in educational design. Appealing to these strengths we introduced the group project as a competition to give students some incentive to participate and an opportunity to get to know other students. The Nobel Factor started with collaborative writing and resulted in a substantial knowledge base which has been made available as a rich learning resource for other students. The contribution to the resources of the entire first year cohort of students increases the profile and credibility of the student groups undertaking the bridging program, Leapfrog Biology. When the winning wiki is announced in one of the early lectures students respond exuberantly and no doubt the winning group members appreciate the prize as well as the acknowledgment.
Transition In the week following the completion of Leapfrog Biology and before the start of first semester students attend a three day transition camp the purpose of which is to introduce new students to the course, to support systems, to staff and most importantly to each other. There is no doubt the camp is more effective than an online environment for engendering a sense of belonging and addressing transition issues for most students. However, the discussions online suggest that students have already formed useful networks, even to the extent of organising transport and housing, and establishing study groups with the potential to provide ongoing support.
51
Collaborative E-Learning Using Wikis
Future Trends In making wikis private we ignore much of what is useful about social software, that is the finding and making of social and intellectual connections. Social bookmarking is an obvious next step in the evolution of these wikis. Social bookmarking provides a mechanism for storing, describing and sharing bookmarks. One of the earliest sites offering a service for social bookmarking was del.icio. us (http://del.icio.us). Users register and access the service which uses tags (e.g., keywords) to link users’ interests, thus generating links to other sites via annotated tags. In this way information discovery can be facilitated, dealing effectively with the increasing volume and variety of sources (Alexander, 2006). Social bookmarking is not inherent in the Leapfrog Biology project but would be a valuable way in which the first year students could gain benefit from and contribute to the wikis. In order to design ‘tags’ students would learn about the information and concepts and the contexts of the knowledge constructed. The connections made and extensions of the networks made as a consequence would add value to the wikis themselves which would evolve as more complex and richer environments. This approach may extend the potential for connected knowledge with a broader concept of peers, peer learning and peer assessment. While beyond the scale of this program exploring the emergence of niches connected with each other using tags would make greater use of the potential of the software. Termed parcellation (Dron, 2007), this potential contributes to the power of social software to promote evolution of sites without imposition of a hierarchical structure. However, rather than letting them emerge we have exerted control as teachers assigning students to groups and creating wikis. Considered in a larger context the wikis created could be viewed as a seed of a bigger picture were parcellation becomes relevant. When student wikis are made accessible to first year medical students as a resource the
52
potential value shifts from the group to a network (Leslie & Landon, 2008). The wiki could then become part of a wider community of inquiry where future students support its evolution. Through parcellation student project wikis could be connected with other student wikis. This could occur between wikis in a single program or between wikis across programs. Would the use of social software for student learning be scalable in this way? Would meaningful learning occur in designing and implementing the connections via tags? Would students retain interest beyond the period of a single program? In terms of transactional control, the group projects using wikis described in the chapter were initiated with considerable structure but moved rapidly to increased control by students. In this way students moved along a learning trajectory to the end of the project. These students then move on to first year studies and new learning experiences. A question arises as to the ongoing effect of the experience, does it become an isolated incident during which they learnt some biology, or does it have the potential for lasting benefit in terms of collaborative learning skills? Having become familiar with the software would students choose the flexibility of the E-Learning environment for further study, for example, would they use it for informal study groups? Students put time and energy into group formation, could the group formed be useful beyond the program? Could the community that arose within a project be sustained beyond the program? If the software were to be used to form communities on a boarder base than that of a single program, what types of practice and knowledge bases would sustain it? Would communities across year levels provide E-Learning opportunities? The broad learning trajectory of students inevitably involves multiple levels of transactional control at different times. Giving students control through the use of social software is useful within a project, but if the knowledge bases of the wikis are to be used a second cycle of structure
Collaborative E-Learning Using Wikis
involving verification of the information and ideas presented would be needed. Thus group project learning becomes a mature web publishing exercise. The use of student project wikis as learning resources for others raises an issue of verification of the information. In the case of Wikipedia the validity of the knowledge base is peer assessed informally. Formal inclusion of peer assessment of student wikis as part of the project, or as the activity of another project could, at least in part, address issues of reliability and validity of information.
alienation in a web-based distancE-Learning environment. Open Learning, 19(3), 280-291.
REFERENCES
Gatfield, T. (1999). Examining student satisfaction with group projects and peer assessment. Assessment and Evaluation in Higher Education, 24(4) 365-378.
Alexander, B. (2006). Web 2.0: A new wave of innovation for teaching and learning. EDUCAUSE Review, 41(2) 32-44. Barrie, S. C. (2004). A research-based approach to generic graduate attributes policy. Higher Education Research & Development, 23(3) 261-276. Bourner, J., Hughes, M., & Bourner, T. (2001) First-year Undergraduate Experiences of Group Project Work. Assessment and Evaluation in Higher Education, 26(1) 19-39. Bower, M., & Richards, D. (2006). Collaborative learning: Some possibilities and limitations for students and teachers. In L. Markauskaite, P. Goodyear, & P. Reimann (Eds.), Proceedings of the 23rd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education: Who’s Learning? Whose Technology? (pp. 79-89). Sydney: Sydney University Press. Brack, C., Stauder, A., Doery, J., & Van Damme, M. P. (2007). Collaborative learning online: Integrating media in case-based learning for clinical biochemistry. Proceedings of EdMedia, (Vancouver, Canada, July 2007) (pp. 187-192). Dickey, M. D. (2004). The impact of web-logs (blogs) on student perceptions of isolation and
Dron, J. (2007). Designing the undesignable: Social software and control. Educational Technology & Society, 10(3), 60-71. Falchikov, N. (2005). Improving assessment through student involvement. London: RoutledgeFalmer. Farmer, B., Yue, A., & Brooks, C. (2008). Using blogging for higher order learning in large cohort university teaching: A case study. Australian Journal of Educational Technology, 24(2), 123-136.
Gibbs, G., & Simpson, C. (2004). Conditions under which assessment supports students’ learning. Learning & Teaching in Higher Education, 1, 3–31. Honebein, P. C., Duffy, T. M. & Fishman, B. J. (1993). Constructivism and the design of learning environments: Context and authentic activities for learning. In T. M. Duffy, J. Lowyck & D. H. Jonassen (Eds.), Designing environments for constructivist learning, (pp. 87-108). Berlin: Springer-Verlag. Kennedy, G. J. (2005). Peer-assessment in Group Projects: Is It Worth It? In A. Young & D. Tolhurst (Eds.), Conferences in Research in Practice in Information Technology, Vol 42 Retrieved May, 2008, from http://crpit.com/confpapers/CRPITV42Kennedy.pdf Krause, K., Hartley, R., James, R., & McInnis, C. (2005). The first year experience in Australian universities: Findings from a decade of national studies. Canberra: Australian Department of Education, Science and Training. Retrieved May, 2008, from http://www.griffith.edu.au/__data/assets/pdf_file/0006/37491/FYEReport05.pdf
53
Collaborative E-Learning Using Wikis
Lave, J. (1993). Situating learning in communities of practice. In L. B. Resnick, J. M. Levine & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 17-36). Washington, DC: American Psychological Association. Leslie, S., & Landon, B. (2008). Social Software for Learning: What is it, why use it? The Observatory on Borderless Higher Education, Association of Commonwealth Universities and Universities UK. Li, L. K. Y. (2001). Some refinements on peer assessment of group projects. Assessment and Evaluation in Higher Education, 26(1), 5-18. O’Reilly, T. (2005). What Is Web 2.0, Retrieved May, 2008, from http://www.oreillynet.com/ pub/a/oreilly/tim/news/2005/09/30/what-is-web20.html
54
Parker, K. R., & Chao, J. T. (2007). Wiki as a teaching tool. Interdisciplinary Journal of Knowledge and Learning Objects, 3, 57-72. Peel, M. (2000). ‘Nobody cares’: the challenge of isolation in school to university transition. Journal of Institutional Research, 9(1) Retrieved May, 2008, from http://www.aair.org.au/jir/May00/ Peel.pdf Raban, R., & Litchfield, A. (2007). Supporting peer assessment of individual contributions in groupwork. Australasian Journal of Educational Technology, 23(1), 34-47. Shirky, C, (2008). Here comes everybody. The power of organizing without organisations. Penguin Group, Australia. Vygotsky, L. (1978). Mind in society: the development of higher psychological processes. Cambridge, MA: Harvard University Press.
55
Chapter IV
Learning and Assessment with Virtual Worlds Mike Hobbs Anglia Ruskin University, UK Elaine Brown Anglia Ruskin University, UK Marie Gordon Anglia Ruskin University, UK
ABSTRACT This chapter provides an introduction to learning and teaching in the virtual world Second Life (SL). It focuses on the nature of the environment and the constructivist cognitive approach to learning that it supports. The authors present detailed accounts of two case studies and provide preliminary analysis of the way in which the environment helps students to achieve both explicit and implicit learning outcomes. The formal assessment for these studies allowed the content, style, narrative and working pattern to be decided by the students. They believe that this approach provides a useful stepping stone between content driven and problem-based teaching techniques. Initial results seem to indicate that students have brought in learning from other areas with a mature approach that enhances their transferable skills in group work, project management and problem based learning. The authors suggest that loosely specified assessments with suitable scaffolding, within the rich environment of Second Life, can be used to help students develop independent, self motivated learning. To support this they map criteria from problembased learning literature and link the learning experience to types of learner.
INTRODUCTION Current practice in Higher Education is moving away from didactic content delivery and the
transfer of abstract concepts (Goodyear, 2002), towards constructivist, student-centred models with increasing emphasis on the skills that support independent, self-motivated learning. This trend,
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Learning and Assessment with Virtual Worlds
reported on by the influential Tavistock report (Cullen et al., 2002) is increasingly facilitated by use of e-learning technologies such as virtual learning environments (VLEs), social networking applications and virtual worlds. While content remains important, the availability of Google, Wikipedia and many other on-line content repositories allows educators to put more emphasis on, transferable, lifelong and problem solving skills. This shift in emphasis can be facilitated by appropriate assessments that allow a broader range of learning outcomes to be assessed. Ideally we want to create a situation where both content and learning skills are practiced and assessed either directly or indirectly. The key to this is to provide activities that intrinsically embody the desired attributes. In the same way that we can develop physical ability by setting an assessment of ‘playing a game of football’, so we can develop other abilities by setting appropriate targets in a suitable environment. Second Life (SL) (Linden Research, 2008) is a 3-D, online, virtual world using similar technology to the Massively Multi-user On-line Role Playing Games (MMORPGs) such as World of Warcraft (Ducheneaut, Yee, Nickell, & Moore, 2006). Contrary to the typical model where content and activities are devised for users to consume, content is built and owned by its users. Second Life provides tools and guidance for manipulating the environment; allowing action scripting, object construction and an economy that supports the creation of virtual businesses. Students can ground their academic knowledge in meaningful practice and rehearse skills through interaction within a realistic environment (Jonassen, 1997). This also allows for setting tasks and activities in keeping with the problem based philosophy to support exploration and self learning skills. The community aspects of SL provide a rich resource for social constructivism. The focus on user created content means that for any significant project a group of people need to get together, share knowledge and disseminate what they
56
have done to the wider community. Resources in-world are often backed up by web pages, wikis and discussion forums that act as tools to support development and promote the aims of the group. An example of this is the Plymouth Sexual Health (Kamel Boulos & Toth-Cohen, 2008) that supports the in-world activities of the simulation. The International Society of Technology in Education (ISTE, 2008) website provides a good example of a community of practice where Second Life provides an additional aspect to an existing web based resource. Users with experience of strongly-themed role-playing games can find that SL lacks depth and detail but it does allow a closer relationship between the virtual (SL) and real worlds. At its simplest SL can provide a mediated communication mechanism that is little different to a video telephone meeting and no less ‘real’ than a telephone conversation. Although the different forms of avatars can be startling at first it is as well to remember that nearly everything that looks like a person is a real person and a good proportion of outlandish looking things are people too. The activities of in-world commerce are significant enough to be covered by Business Week (Hof, 2006) and are measured in hundreds of thousands of US dollars with the in-world currency, the Linden dollar, freely convertible to US dollars. Real world concerns from media (BBC, Channel 4, Reuters) commercial (Nike, Amazon, IBM), and a growing number of universities have a presence ‘in-world’ as it is called in SL. Some of the educational activities in SL tend to follow a traditional class based approach with Universities such as Harvard and San Diego having their own virtual campus with virtual lectures and demonstrations. While these activities provide advantages for distance learning, this does not fully exploit the intrinsic properties of the virtual world. A better model to use for learning in SL would be a field trip where some tasks may be outlined but the detailed implementation is down to the student, their co-students, the resources they
Learning and Assessment with Virtual Worlds
find and other residents with whom they interact. The New Media Consortium has opened a virtual campus where SL is used to support a range of educational activities (New Media Consortium, 2006). SL provides a campus registration facility to assist Universities to establish virtual classes, campus constructs, and student enrolment. Education UK provides space on its Island for educational institutions to set up classrooms, demonstrations and exhibition space. This provides a helpful stepping stone between independent activities and the cost and responsibility of maintaining a private space, and it also makes a good showcase. Links to these and other SL educational resources can be found on the SimTeach website (SimTeach, 2006). In this chapter we present accounts of our experience of using the virtual world Second Life for a range of educational activities. In the group work case study we provide an assessment where the subject specific knowledge is peripheral to the learning outcomes and in the second case study we develop this idea but also use the environment to deliver a context for student content. In both cases the student assessments have to be carefully judged to encourage the development of learning skills.
Ba Virtual worlds have been used for education since the mid 1990s, (Hughes & Moshell, 1997). However, at that time, the number of potential users was restricted by the hardware and connection requirements. Now there is a much wider potential user base, both in terms of the availability of the supporting technology and the experience / expectation of users (Book, 2006). There are more potential students and, just as importantly, an increasing desire to use these tools by a wider range of staff to support new teaching techniques. The active worlds learning environment ‘AWEDU’ (Active Worlds, 2006) originated in
the mid 1990s and allows educational worlds to be built for a variety of purposes. Most commonly used for distance learning and the exploration of three dimensional constructs they tend to mirror a classroom environment where interaction is principally between the registered participants (Dickey, 2003, 2005). In addition to content added by teachers, a virtual world is perceived as having some intrinsic benefits - helping to expose students to novel applications of technology and practice in mediated communication. The social aspects of learning are also well supported by a virtual world and help to maintain interest (Corbit, 2002). This matches our experience (Brown, 2006) when using the more traditional discussion forum as an e-learning tool. This was used as an intrinsic part of course delivery and the student feedback supported our argument that technology could improve engagement and participation even where there was regular face-to-face contact in a conventional classroom. The research in this field seems to suggest that there are educational advantages in a virtual setting (Benford, Greenhalgh, Rodden, & Pycock, 2001). However, as Livingston points out (Livingston, 2007), there is little point in creating virtual classrooms which require synchronous supervision. The strength of a rich virtual environment is that it can provide exploration activities more akin to field trips than the traditional classroom. Although not exclusively constructivist the affordances of these virtual worlds lend themselves to the process of constructing knowledge from interacting with their environment (Jonassen, 1992). One of these constructivist approaches that seems to have a particularly good fit with these environments is Problem Based Learning (PBL). This is an approach that seeks to promote task-orientated, self directed learning which is typically linked to a real world scenario. This often comprises useful transferable skills such as team-work, problem solving and independent thinking. In conventional teaching, information is provided to students who then demonstrate their
57
Learning and Assessment with Virtual Worlds
understanding by using it to solve assessment tasks. In PBL the task is used to provide a focus for students who then gather the information required to solve it. This system has been used in medical and other professional training systems since the 1960s and there are many related techniques that share the central premise of a task centred approach to learning (Savin-Baden, 2000). In common with a number of educators (Sweller & Clark, 2006; Merill, 2007) we use the PBL term as a generic descriptor for the type of approach rather than following a formal PBL methodology. In the IECO case study subsequently we show that a suitably constructed task can be embedded into a virtual world and deliver many of the benefits of a problem based approach. There are other similar approaches to virtual worlds such as simulation based learning (Kindley, 2002) and serious gaming (de Freitas, 2006). These use similar technology although tend to use a pre-constructed simulation or game which typically requires considerable resources, may be expensive and limits the amount of input a user may have. Using a simulation can be very effective but building your own resources engages a wider range of skills. However, simulations have the advantage of a carefully controlled and often high fidelity simulations which can contain physical feedback mechanisms that are beyond a typical virtual world.
Learning Models for Virtual Worlds A recent detailed survey of blended e-learning by Sharpe, Benfield, Roberts, & Francis (2006) for the Higher Education Academy identified three ways of using technology to support teaching in a blended learning environment. We have linked these to the associative, cognitive, situative framework developed by Mayes and de Freitas (2004) for e-learning development and indicate how these can be supported by Second Life.
58
Traditional The most common use for e-learning technologies is to support the traditional material delivery, practice, repetition and recall cycle of teaching. Lectures and seminars tend to focus on an associative learning mode where concepts are explained and then students apply these to illustrative problems to build up a body of knowledge. Second Life provides many tutorials backed up with exercises that help explain how to do various things such as creating 3D objects. What is less typically associative is that a student would be able to access several learning resources with slightly different content and interact with helpers, mentors, support groups and others who had little or no connection with a particular educational institution. Although SL allows detailed learning, particularly in areas of programming with the LSL scripting language, it is not playing to the strengths of the system. In the group work project case study we discuss a module that did require the application of specific technical knowledge. Here the SL environment provided an excellent example of the technology but provided little more than support for that technology.
Transformative The transformative use of e-learning technology, as described by Sharpe et al. (2006), is still innovative and relatively rare. Here technology is used to radically change course design with an emphasis on interaction, communication and real world problem solving skills. The student applies their existing knowledge and experience to integrate new concepts in a personal way – i.e. using their ‘take’ on the situation. Social constructivism exports this concept to the ability of a group of collaborating individuals to create a shared model of knowledge as well as contributing to individual learning. This also represents a good description of the core user and management model for Second Life. This works for the
Learning and Assessment with Virtual Worlds
fundamental instruction needed to use the tools and development applications and also between users who need to interpret and guide each other on the features of the content that they have created. This may give a somewhat idealised view of human interaction within SL, but even the most commercial operation needs to demonstrate their goods and get people to understand how they work. We would expect that in SL the bias towards learning and discovery would also support the individual’s cognitive abilities and allow them to practice and develop their learning skills. The problem is that it is quite difficult to directly assess improvements in cognitive skills. A suitable assessment strategy is to set a partially scaffolded task such as described in the case studies where the quality of the assessed component reflects the application of learning skills.
Holistic Support for the holistic model is still emerging in higher education e-learning strategies. As with constructivist approaches it is based on reflection, active construction and collaboration (Teo Siew Chin & Williams, 2006). It builds on these with a wider, longer view that is appropriate to lifelong professional development. Practice and learning from experience is viewed as a coherent whole that requires technological support before, during and after enrolment on a particular course of study. A good example of this are communities of practice that cater for professionals who have left institution based training and use a range of communication technologies to provide resources throughout their careers. These provide the context for observation, reflection and opportunities for mentorship, and sharing knowledge as Brack and van Damme have described in Chapter III in this volume for undergraduate medical education. In Second Life the situation can be one which has been created by trainers to simulate a particular aspect, such as an accident and emergency hospital or it can be one which has been
created by the user themselves. In either case the simulation persists externally to the educational institution and can invite comment and interaction from a wider audience than those registered on a particular course. By its very nature the long term aspects of a learning activity are impossible to assess within the time span of a particular module. These are more likely to be supported by extracurricular activities where students, staff and others get involved with groups based on interests rather than locations or institutions. SL provides opportunities for students to create virtual activities based on their interests, which can be for fun, or can develop into virtual businesses. The requirements for a successful venture in SL are similar to any in the real world except that the initial set up costs are typically considerably less. It is also much easier to set up international operations where members can be in a variety of physical locations around the world.
Vi rtual World Educational Issues The key issue for the use of virtual worlds and Second Life in education is to identify the areas where it can extend or improve on exiting provision. A typical pattern for any new technology is for excessive enthusiasm from early adopters to be followed by equally excessive criticism. Second Life in particular has suffered from exaggerated claims for its influence for both good and ill in a number of different areas. Looking at a simple list of features it is hard to point to a particular factor that is not supported in some way by an alternative application. However, it is the breadth, scale and diversity of the simulation that offers unique educational opportunities for discovery, social interaction and creativity. The following are a few of the key affordances: •
Telepresence, where the user projects their point of view into the 3D world and is men-
59
Learning and Assessment with Virtual Worlds
•
•
•
•
tally immersed into the context. This can be used to increase engagement with learning materials. Communication–voice, local chat and instant messaging enable multiple simultaneous communications allowing person to person as well as person to group communication. Learning by doing – the ease of construction within SL makes it possible to help students to build their own learning materials or demonstrations. Learning by becoming – SL provides tools for easily customising avatars. One of the first serious educational activities was an English class that role-played the characters from a Jane Austin novel. Association–SL provides a context and content for reporting and reflection through associated e-learning and social networking portals.
7.
8.
9.
At a recent workshop (MML08) the plenary session brought together many practitioners in using Second Life in education and the following summarises a few of the points that had general agreement: •
The following check-list offers more practical advice when considering the suitability of Second Life for a learning activity:
•
Identify learning objectives–what skill/ knowledge are the students going to learn? Identify assessment criteria – how is the skill/ knowledge going to be demonstrated? Identify technological need – Is SL an appropriate tool for all, or some of the learning outcomes? Assess infrastructure – will SL run in the lab or on students’ home computers? Assess student motivation – who really wants/needs to use SL, is there student demand or indifference? Assess student ability – is this a first time in SL for some? Are there experienced SL students who can serve as mentors/group leaders?
•
1.
2. 3.
4. 5.
6.
60
Assess available time – is there sufficient time for students to spend on getting to know the environment? Plan an assessment strategy that allows for diversity of approaches to a problem but also requires students to meet the key learning outcomes. Establish reporting facilities to capture process and activities as well as final artifacts, using forum, blogs and other social networking software.
•
SL is not MySpace or FaceBook and it is generally not used or even known among teenagers and school leavers, so don’t expect much of an existing user base. On a first introduction at least 50% of students don’t particularly like Second Life, so don’t expect the technology itself to entice students. Some students do like SL, and like other on line experiences, it can become addictive, although probably less so than most other on-line gaming applications. Induction – first impressions really count. Make sure any introductory session is sufficiently supplied with helpers (both on line and in the class room), that the computers used are sufficiently capable and that you have carefully thought through the rationale for using SL.
The next two sections show case studies, based on our practice with Second Life, that illustrate the types of activity and assessments to which the environment is suited.
Learning and Assessment with Virtual Worlds
Th rtual Worlds Group Work Project Case Study Anglia Ruskin University funded this research project from their University Centre for Learning and Teaching (UCLT) in 2006 with the specific brief to extend understanding and inform teaching practice. Previous studies into virtual worlds (Dickey, 2005) have provided important insights into the pedagogical implications of these systems. This project contributes to the on-going evaluation of such systems and seeks to start the process of establishing techniques for their effective use. We chose the area of group work as an initial target as it corresponds to the collaborative nature of Second Life. Within the context of the SL environment the situative model of learning seems particularly appropriate to development of group skills. The shared experience and group targets, set by the project, are designed to help develop independent and cooperative learning within the group. The community nature of the environment supports the learning outcomes of the activity which are: 1. Interact effectively with others 2. Maintain co-operative working relationships 3. Play a useful role in group/ team activities 4. Feel confident in a group setting 5. Take a leadership role when asked to do so
The Group Work Project Methodology Flexible interaction is fundamental to acquiring knowledge (Barker, 1994) and allowing students to discover their own understanding (Jonassen, 2000). However, without extensive observation and monitoring, it is difficult to measure the degree and usefulness of the interaction. Although this would be possible for a research project it is not helpful in establishing a robust assessment
strategy. So the assessment needed to be easily measurable but also encourage and depend on the desired learning outcomes. In this case, the formally assessed component was not the activity itself, nor even how well the group performed, but the quality of a reflective report. Implicitly we assumed that the better groups, who performed well on the tasks, would have the material to produce a good report. The idea was that an unconstrained approach to the tasks would better promote broader learning skills and cognitive development. The research was carried out as an evaluative case study using material recorded by students as they progressed through a set of tasks. These were structured according to Salmon’s five-stage model (2004) for computer-mediated conferencing. The model comprises a framework of five stages, each stage building on the previous, to enable increasing student interaction through structured activities and decreasing levels of tutor support. Each stage is characterised by its identifying mode of interaction. Stage 1 (Access and Motivation) may be likened to induction, where tutors ensure that students can access the system, and provide timely welcome and encouragement. Stage 2 (Socialisation) encourages students to establish their on-line identity by finding others with whom to interact without task-based concern. This stage, an opportunity to practice, is important as the success of group work may be affected by the level of virtual competencies. In contrast, stage 3 has focus on co-operative exchange of information with regard to task. At stage 4 (Knowledge Construction) interaction becomes collaborative (mutual learning). By stage 5 (Development) students become self-critical and reflective, taking responsibility for their own learning, and that of their group.
The Group Work Project Implementation Twelve students from the level 1 Computer Gaming and Animation Technology degree agreed to
61
Learning and Assessment with Virtual Worlds
Table 1. E-learning model levels and tasks Model Levels 1. Access and Motivation 2. Socialisation
In-World Task Registration and orientation Meet and join ‘ARC’
3. Information Exchange 4. Knowledge Construction 5. Development
Choose group identity Treasure hunt Building competition
participate in the group work project. None of them had any experience of Second Life, but all were computer literate and had experience of on-line game environments. Although not normally part of their studies the introduction to a virtual world and the group activities made a useful addition to their core curriculum. Prior to the start of the project a tutorial session was held on techniques for group work as part of their personal development portfolio. An on-line resource for the project was set up in the Moodle Virtual Learning Environment (VLE) with project documents, SL and group work related web links, discussion forum (open to all participants), conversation logging tools and individual student blogs.
Project Briefing Students were told about the aims and activities of the project, provided with copies of the project outline and ethical consent forms. These detailed their responsibilities, the rewards for participation and the constraints on the use of the information gathered by the project.
Registration, Orientation and Basic Sills Once a student chooses an avatar they undertake 1-2 hours of individual orientation using the standard SL resources of ‘Orientation Island’ for basic interaction skills (moving, communicating, personalising avatar etc.) and the ‘Help Island’ for the introductory tutorials for building and editing objects. 62
The Anglia Ruskin Computing Group After initial orientation and basic tutorial sessions on help islands students were directed to the Anglia Ruskin Computing (ARC) group meeting point. All the groups met together and could see each other’s avatars for the first time. By this time some students had already explored and brought examples of things they had found and ‘showed off’ the skills they had learned. Students were invited to join the ARC group which allowed them permission to build on the ARC site. Once they had joined they were able to collect a note card with the treasure hunt task details.
Group Formation and Identity The students organised themselves into three groups of four. Each group had to choose an ‘identity’ and devise a visual cue in their avatar appearance. In addition they had to post a message to the blog to demonstrate that visual identity and send an email to the tutor to identify their avatar.
Group Treasure Hunt The group treasure hunt task was to find objects and locations specified on a note card. The idea was to facilitate development of searching, navigation and group working skills. To show that they had found an item, a snapshot of the group was posted to their blog. Figure 1 shows a team in front of the Second Life Library.
Learning and Assessment with Virtual Worlds
Figure 1. Image of group treasure hunt activity in second life
Building Competition The ‘ARC’ group land was divided into three waterfront plots. The groups were given the brief to build a jetty for a yacht and a lodge for the owner to relax in. These were to be judged and the best construction would win a prize (a small amount of the Second Life ‘linden’ dollars). To support this activity students were directed to in-world tutorial resources and were given a workshop / tutorial session on the building and editing tools.
Feedback and Recording Feedback from students was recorded in three ways – observation in class, individual blogs and through the assessment, which was a reflective report written shortly after the end of the activity. Evidence for activities was recorded by students using the SL snapshot tool and shared with the group by adding them to their blog. The snapshots of in-world activities were linked to the Moodle VLE where the individual blogs were hosted.
These procedures not only helped students practice reflective skills but also introduced them to a wider range of tools.
Skill and Knowledge Differentiation Students quickly showed a marked differentiation in both the expertise and type of skills learned. Within the first few hours it was clear that the majority were gaining wider and deeper skills than was necessary for the basic task completion. However, keeping them focused on the current activity was difficult and a few students did not gain sufficient skills to complete tasks on their own. We believe the differences in skills reflect innate ability but are emphasised by using the in-world learning resources. These resources are diverse and cover a broad range of skills that require a more sophisticated learning style than a closely monitored set of class exercises. The resources make it easy to explore and extend skills but it is also easy for a student to move on before they have learned enough. The highly autonomous
63
Learning and Assessment with Virtual Worlds
nature of the learning allows students to focus on those things that interest them, developing some skills at the expense of others. This was evidenced by some students becoming adept at avatar customisation but failing to pick-up many building skills. If content were more important than process it would be necessary to add better defined exercises.
Real and Virtual Integration Most of the group work was done in a classroom where it was easy to communicate face to face as well as interacting with the environment. It was unsurprising that there was considerable interaction about the project tasks (and other things). However, it was surprising to see how the ability to interact in the virtual world complimented and
enhanced the overall communication. Students interacted seamlessly between the real and the virtual worlds, particularly during the more complex building tasks. Discussion and communication were typically done in the classroom but demonstration, knowledge discovery and sharing were done in-world. This was an entirely un-prompted, emergent behaviour enabled by the environment.
Group Working Although this exercise was a blended learning situation it was apparent that the environment allowed the sort of casual, informal interaction that is often missing in purely on-line distance learning. It is possible that this could alleviate the lack of opportunity to learn ‘vicariously’
Table 2. Group work goals Learning Goal 1. Interact with others
Degree Comprehensive
2. Maintain co-operative working relationships 3. Play a useful role in group/ team activities 4. Demonstrate confidence in a group setting 5. Develop and take on a leadership role
Comprehensive Informal / flexible Partial – depended on engagement One main leader with domain specialists
Evidence Negotiation of group theme, blog accounts (appendix 1) Artefacts created with contributions from all group members. Tasks assigned informally, one or two tended to lead. Personal blogs (appendix 1) Group typically driven by one person but sub tasks lead by others in the group.
Table 3. Level and nature of engagement Criteria 1. Sense of fun and novelty insufficient to motivate use of technology. 2. Assessment (necessity) a strong motivation for using technology. 3. Structured navigation preferred by less confident students. 4. Match between technology and task. 5. Perceived value of activity.
64
Finding True – particularly for students used to interactive gaming technology. True – other work took precedence until linked to assessment. Not proven – unstructured nature of task criticised by some students. True – shared environment facilitated group building activity. True – depends on engagement threshold for activity
Evidence Some students failed to engage beyond assessment guidelines. Reporting and reflection linked to PDP assessment Some students ignored tasks and ‘did their own thing’ Groups worked in both virtual and real environments. Students carried on developing in SL beyond end of project.
Learning and Assessment with Virtual Worlds
through the incidental lab encounters with peers from which face-to-face students benefit (Bower, 2007). Analysis of the reflective reporting showed that most students found the experience useful and interesting and anecdotal evidence seems to suggest that they were able to develop their group work skills. Table 2 shows a summary of some of the evidence that the project provided to support the skills we wish to see developed for group tasks.
Level and Nature of Engagement A key rationale for using educational technology is to enhance engagement between the student and the learning activity or material. However, technology alone cannot be relied on to provide this. Students quickly get used to even the most exotic environments and in order to hold their attention there must be a correspondence between the technology and what they perceive as being useful (Laurillard, Stratfold, Luckin, Plowman, & Taylor, 2000). In our own practice (Brown, 2006) we have identified five key points that influence engagement in the use of on-line discussion groups and Table 3 maps these criteria to the virtual world exercise. In our role as mentors and observers, we discerned a variety of attitudes to the environment from this and other virtual world exercises that we have carried out. These can be classified into four broad categories and we found (noted first below) that they also corresponded well to the classic Honey and Mumford (1982) classification of learning styles (italicized below): •
•
‘Superficial’ – user does not engage and does not find the SL environment interesting. Theorist - not much to interest the theorist style of learner who is good at complex concepts but not very flexible. ‘Realistic’ – user acts and behaves in SL as they would in RL, regards other avatars as other people in social situations. Reflector
•
•
- observant, methodical and typically concerned about the real world implications of their virtual activities. ‘Empowered’ – user acts in SL as they would in RL but feels empowered to be more adventurous in initiating activity and social situations. Pragmatist, practical and experimental in nature, principally interested in creating things and seeing what the environment will allow them to do. ‘Fantastic’ – user regards SL as a game where other avatars have little connection to real people, bold social behaviour, with less social responsibility than RL. Activist, risk taker interested in the novelty of the environment and most likely to interact with other residents and seek out new experiences.
Although far from conclusive this does provide clues as to where assessment tasks may be placed to structure learning towards specific learning styles. It is possible that an in-world experience may be more attractive to activist learners than class or VLE based exercises but this approach may be less attractive to theorist learners. To maximise the transferability of skills learnt in Second Life we need to aim at migrating users towards the ‘empowered’ view where there is a close correspondence between the real and virtual worlds with the possibility that confidence built up in the virtual environment is able to develop into progress in the real world.
Group Worlds Project: Summary Experience, research and preliminary findings all point to the need to devise carefully planned learning activities to produce the desired learning outcomes. Although Second Life is not a magic wand it does have accordance to transformative educational goals by providing a rich environment for individual exploration. Open-ended learning tasks and field trip style activities can be devised. However, the sophistication of the environment
65
Learning and Assessment with Virtual Worlds
makes this a more, rather than less, challenging task, as does the wider range of possible styles of interaction in a virtual world compared to classroom or traditional VLEs. The following list outlines some of the ways in which Second Life enables students to take more responsibility for their learning: 1.
2.
3.
4.
5.
Blending real and virtual – virtual on-line environments lend themselves to distance learning but they can also add a new dimension to group activities which can seamlessly move from face to face to virtual. Task based learning – the skills needed to complete the tasks were principally learned from the virtual environment and perceived as part of the task rather than being external. Varied autonomous learning – students gained skills in different areas by accessing different learning resources and experiences. Peer to peer learning – shared environment made it easy to demonstrate applied knowledge. Students would take on a tutoring role to disseminate skills. Mobile group structures – the core leadership role in groups did not change but group members would lead sub-tasks where they had particular skills.
Thectronic Content Oiition (IECO) Module The encouraging results from the group work project enable us to integrate an element of Second Life into the assessment for a mainstream module. The aim is to use the environment for both cognitive development and to demonstrate subject specific knowledge and skills. This module is designed to introduce students to a variety of different techniques in the creation and manipulation of images, sound and video editing. The idea
66
is to give the level 1 first year computer science and computer gaming students a basic grounding in a range of applications that will stand them in good stead throughout their university career and beyond. The common theme was to use freely available software and teach enough about the underlying theory and techniques for students to be able to get started and direct them to resources that would enable them to go further if required. The thinly hidden agenda, which was explained at the start to the students, was to demonstrate how to go about learning new types of software and give students practice in developing their own strategies. As before, this is a desirable outcome but not one that is easily measurable. The following lists the applications and key operations that were covered by this module: •
•
•
•
•
•
•
GIMP – Open source bit map editor, used to create and manipulate photographic images by selection, removal and insertion operations. Inkscape – Vector graphic editor, particularly useful for text and logos, demonstrates vector representation of image data through SGL text files. Art of Illusion – 3D image editor. Used to create 3D models and introduce the concept of adding textures (2D images) to them. 3D Modeling in Second Life – basic principles of creating and texturing artifacts in Second Life, using a chess piece tutorial. FRAPS – Video frame capture software to create Machinima allowing the creation of video sequences of action from virtual environments such as computer games and virtual worlds. Video Editing – Basic introduction to Windows Movie Maker which allows editing of a video stream at the individual frame level. Audacity – Audio editing, allows insertion of effects, filters, the combination and sequencing of audio material.
Learning and Assessment with Virtual Worlds
Despite the wide range of applications covered there were intrinsic processes that were common to the way these tools were used. Each of the exercises reinforced a problem decomposition approach, where sub elements of the problem may require the student to extend their learning beyond the initial introductory material.
examples for building artifacts in Second Life. Additionally they were shown the basic techniques of machinima and in-world resources for setting camera positions and lighting conditions. The ‘getting started in SL’ text is available on the Anglia virtual worlds education Wiki (Hobbs M., 2008).
Machinima
Te Assessment
Machinima is a portmanteau word linking ‘machine’ with ‘cinema’ to describe the activity of capturing output from real time interactive virtual world environments to create video footage. The output may appear like a computer generated animation but the production is much more cinematic with virtual actors, stages, rehearsals, direction, camera angles, lighting, costume and props. Machinima started in the late 1990 when computer gamers captured clips of action where they added dialog and used their characters as actors to play roles within the game environment (Lowood, 2005). The key feature is that the animation is provided by the game engine rather than a dedicated animation application. This limits the quality and complexity of the genre but game engine and simulated environments provide a quick way to rough out ideas and prototype scenes for films or animation.
Students in groups of 2 or 3 had to create a scene and provide the action for a short video clip shot using machinima techniques in Second Life. They had to use the narrative to bring in things that they had built, textured and used as props in the scene. Each group handed in a joint report as well as an individual reflection on what they did in accordance to the following assessment instructions, set out in Table 4. From this marking scheme it can be seen that the focus of the assessment was on the process, which allowed considerable flexibility on both process and content. Figure 2 shows four separate backdrops for the video clip, one is a replica of the computing lab where students have taken photos of the PCs and applied these as textures to create a more realistic image. Another is a ‘Super Mario’ style side scrolling arcade game simulation. Other props relate to a space based narrative. There were several features of the learning task that had a correspondence with the Problem Based Learning criteria, as proposed by HmeloSilva (2004). Table 5 shows a mapping between the key PBL goals and the features of the assessment in the Second Life environment. An important feature for a problem based, or scaffolded inquiry environment is that it gives students the opportunity to engage in complex tasks beyond their current abilities (Sweller & Clark, 2006). The intrinsic qualities of Second Life make this relatively easy for sufficiently motivated students. An example of this was evidenced by a student creating a whole chess set, rather than the single piece required for the exercise. Other
Getting Started in Second Life Students were encouraged to undertake the standard ‘induction’ activities provided by Second Life for new users. In addition they had to confirm that they had read the terms and conditions for using Second Life and the requirements on behavior for the Anglia Ruskin Island (a larger area than the original ARC site with a wider range of user groups). As well as technical elements students were set exercises that included socialization activities, similar to those developed for the group work project. The main taught content focused on showing students where to find tutorials and
67
Learning and Assessment with Virtual Worlds
Table 4. Assessment criteria Common Elements (40%) Worked on by all in the group, hand in one copy Narrative – the concept and content of what is being shown.
Individual Artifacts (40%) Your contribution to the group Account of the how the Machinima video was made.
Show your machinima video
Appropriate use of images, sound and 3D models to support the narrative.
Description of artifacts made – at least one each of: • Bitmap image or texture • vector graphic • 3D Second Life artifact. Presentation – structure and coherence of document in report format.
Give a brief explanation of what you have done
Story board and asset list indicating who created each asset. Production and editing of video
Group Presentation (20%)
Be prepared to answer questions on your work
Figure 2. Machinima film sets created in second life
students extended the assessment by learning about scripting actions for objects using the Second Life programming language. Suitable scaffolding informs students on both how and why a task should be done in a particular way. In many instances techniques were introduced by demonstrating a finished piece of work, then showing how the various elements were brought together using the available tools. Second Life abounds with suitable examples that can provide inspiration as well as the tutorials
68
that build up from scratch. In this course we used ‘building a chess piece in Second Life’ and an example machinima video as an inspiration of what might be achieved. Another key aspect of PBL is the ‘real world’ nature of the task. A well selected task, or trigger, should encourage students to develop knowledge and skills that direct the abstract skills of graduate learning to solve a problem of a type that they may encounter in a professional capacity. In this respect although the environment is virtual the
Learning and Assessment with Virtual Worlds
Table 5. Mapping problem based learning criteria to evidence from the group work project PBL Goal Flexible Knowledge
Effective problem-solving skills (planning, design and problem decomposition) Self Directed learning Skills
Effective collaboration
Intrinsic motivation
Evidence Each of the student groups had to consider how the limitations of environment and resources affected their creative ideas. Without a single ‘correct’ way to approach the implementation we found a variety of different approaches that suited the group skills. The story line, the action and the supporting props all need to be created by the group so these have to be designed and integrated. The scene has to provide three minutes of coordinated action so every aspect has to be planned and choreographed. Basic exploration of second life provides the context for developing simple research skills and sifting through the various resources, tutorials and knowledge base web pages to find what you need provides a good way of testing these abilities. When shooting video clips from Second Life you need at one, or two other people to provide the action while someone gives directions and works the camera. The group also has to work well together with attention to timing and direction. Peer group learning was also apparent as locations for free resources and various techniques were shared throughout the cohort. Groups were not in competition but the ability to see what others were building and in some instances join in, provided motivation to create something that appealed to peers as well as assessors.
machinima video represents an artifact created through a real design process with the kinds of limitations imposed by a real world – not unlike a brief from a media design company.
Student Response The students were asked to respond to a questionnaire (Appendix 1) which asked them a number of questions about the nature of the assessment. One of the key areas was to focus on how well they thought they had been supported to be enabled to do each of the two assessments for the module. The first assessment was strongly based on taught material and required them to create and document a bit mapped image, a vector image and a 3D image. The results (Appendix 2) show that most students were happy with the level of support for both assessments despite the considerable increase in the amount of student learning required to complete the second task. Most of the comments demonstrated a good level of engagement as many related to weaknesses in the tools used and the desire to extend the capabilities of Second Life, rather than a rejection of the concept.
Future Trends for Vi rtual Worlds in Education At this point it is difficult to say if virtual worlds such as Second Life are going to become generally important in higher education or if they are merely a temporary technological diversion. The important technical limitation for virtual worlds is the number of concurrent users a system can support (for SL this is currently around 60,000) and the number of simultaneous interactions in a contiguous area which can limit the number of people in the same place at the same time (currently for SL this is around 30 or 40). Despite these limitations there are around 130 higher education institutions with a presence in Second Life (SimTeach, 2008), operating a wide variety of activities. These numbers continue to grow and it would seem that there is an ongoing commitment to the environment by educationalists. A related development is the Open Simulator system based around a separate but similar open source virtual world server (Open Sim, 2008). Although it is still very much work in progress there are around 30 open Simulator sites with
69
Learning and Assessment with Virtual Worlds
various resources. The advantage to an educational institution is that they can set up their own server and create an environment to their own specification. While this may be useful for some activities the content and diversity created by the Second Life community is likely to keep it as a core component to these satellite systems. Another open source system that is being developed is ‘Croquet’ (Open Croquet, 2008) which provides a virtual world environment that operates on a peer to peer network on a variety of servers. It is possible that in future the ability to host a small section of an interconnected virtual world, in much the same way as web servers support the Internet, will add significantly to the wider use of these systems. In addition to the open source solutions there are many proprietary MMO systems under development for games and other purposes which are likely to continue to increase awareness and the user base for these technologies.
Assessment in Virtual Worlds Educational activities and assessments in virtual worlds are diverging into a number of different areas. Conklin (2007) is a common starting point for educators and presents a long list of activities. These and many other, educational activities can be placed into four broad, typically overlapping categories of learning outcomes: 1. 2. 3.
4.
Technical: focus on skills such as building or animating artifacts. Informative: using exploration, simulation or role play to convey content. Reflective: using experiences in world for self development in areas of learning skills, reflection and communication. Social: interaction and role play to explore questions such as group dynamics, ethics and individual identity.
Most good assessments have an element of each of these. Early virtual world activities focused on
70
the social and community aspects as there was little else that could easily be achieved. Assessment would be based on reflective writing about the experience rather than attempting to measure the experience directly. As recording tools (snapshots, machinima, blogging etc.) have improved more of the direct experience, such as building artifacts, can be recorded and assessed. The latest developments have been to integrate these tools into VLEs such as the SLOODLE system (SLOODLE, 2008; Livingston, 2007) which combines Moodle (Moodle, 2008) and Second Life so that chat, questions and even primitive objects created in-world can be recorded by the managed learning environment for assessment. The traditional research areas for virtual worlds are to look at how participation and interaction within the environment affects overall learning for students (Steinkuehler, 2008). In particular areas such as exploration and problem based learning are being assessed. The Open University Schome (2007) project used Second Life to support a community of teenage learners and found that it helped to develop creativity, motivation, leadership, problem solving, communication, teamwork and social skills. Although based in the more carefully controlled ‘teen grid’ this study is relevant to any virtual world learning activities. Formal assessment of generic skills and cognitive development is considerably more difficult than measuring the understanding and application of subject knowledge. In a conventional learning and assessment strategy it is easy to focus on the acquisition of knowledge but harder to develop the skills that might help the student make use of this knowledge beyond the educational setting. A suitable assessment strategy in a virtual world requires the development of these skills in order to achieve the measurable outcomes. Both the case studies presented in this chapter illustrate group and peer to peer learning which were supported by the inherently collaborative nature of the environment and the selection of an assessed
Learning and Assessment with Virtual Worlds
task that drove the potentially chaotic interactions towards a measurable goal.
CONCLUSION Although principally based on Second Life many of our conclusions hold for other virtual world systems. Throughout the chapter we have shown where the issues raised have correspondence in other areas of educational theory and e-learning practice. As we found with both the group work project and the IECO module, a virtual world can be used to generate recording, reporting and reflection through other media such as blogs, photo sharing sites, or machinima. It can also serve as a catalyst to bring together techniques and procedures learned in other modules. The range and depth of resources both within the world and through its supporting websites provide increased opportunities for problem based learning and other constructivist educational activities. To fully utilise virtual world environments a careful balance has to be struck between over specified assessments that constrain exploration and those that do not provide sufficient targets to satisfy the intended learning outcomes. The six key ingredients of a virtual world as itemised by Book (2006) are: 1. 2. 3.
4.
5.
Graphical user interface: The world depicts space visually. Immediacy: Interaction takes place in real time. Interactivity: The world allows users to alter, develop, build, or submit customized content. Persistence: The world’s existence continues regardless of whether individual users are logged in. Community: The world allows and encourages the formation of in-world social groups like teams, guilds, clubs, cliques and neighbourhoods.
This list implies that assessment strategies should consider issues such as space, location, geography, creative interactivity, self image, and role play and community interaction from both within the assessed group and from the wider community of users. These elements can be supported by other e-learning environments but are seamlessly integrated as part of the natural make up of a virtual world. Mason’s (1998) observation that “technology is rarely the problem - and equally rarely the solution” is still true, but may be qualified by saying that technology is increasingly important in motivating and guiding students into independent self-learners.
REFERENCES Active Worlds Inc. (2006). Active Worlds and education. Retrieved August 20, 2008, from http://www.activeworlds.com/edu/index.asp Barker, P. (1994). Designing interactive learning. In T. de Jong & L. Sarti (Eds.), Design and production of multimedia and simulation-based learning material. Boston, MA: Kluwer. Benford, S., Greenhalgh, C., Rodden, T., & Pycock, J., (2001). Collaborative virtual environments. Communications of the ACM, 44(7), 79-85. Book, B., (2006). Virtual worlds: Today and in the future. British Computing Society. Retrieved 15 May, 2006 from http://www.bcs.org/server. php?show=ConWebDoc.3336. Brown, E., (2006). Discussion board or discussion bored? Presented at the 7th UCLT Conference, Anglia Ruskin University. Bower, M. (2007). Group work activities in synchronous on-line classroom spaces. In Proceedings of the 38th SIGCSE technical symposium on Computer Science education (pp. 91-95). Covington, Kentucky, USA. 71
Learning and Assessment with Virtual Worlds
Cullen, J., Hadjivassiliou, E., Hamilton, E., Kelleher, J., Sommerlad, E., & Stern, E. (2002). Review of current pedagogic research and practice in the fields of post-compulsory education and lifelong learning. Final report for ESRC by the Tavistock Institute. Retrieved August 20, 2008 from http:// www.tlrp.org/pub/acadpub.html Conklin, M. S. (2007). 101 uses for Second Life in the college classroom. Retrieved from http://facstaff.elon.edu/mconklin/pubs/glshandout.pdf Corbit, M. (2002). Building virtual worlds for informal science learning (SciCentr and SciFair) in the Active Worlds Educational Universe (AWEDU). Presence, 11(1), 55-67. de Freitas, S. I. (2006). Using games and simulations for supporting learning. Learning, Media and Technology, 31(4), 343–358. Dickey, M. D. (2003). Teaching in 3D: Pedagogical affordances and constraints of 3D virtual worlds for synchronous distance learning. Distance Education, 24(1), 105-121. Dickey, M. D. (2005). Three-dimensional virtual worlds and distance learning: Two case studies of Active Worlds as a medium for distance education. British Journal of Educational Technology, 36(3), 439-451. Ducheneaut, D., Yee, N., Nickell, E., & Moore, R. J. (2006). Building an MMO with mass appeal: A look at gameplay in World of Warcraft. Games and Culture, 1(4), 281-317. Goodyear, P. (2002). Psychological foundations of networked learning. In C. Steeples & C. Jones (Eds.), Networked learning: Perspectives and issues (pp. 49-76). London: Springer. Hobbs, M. (2008). Anglia Ruskin virtual worlds wiki. Retrieved on August 20, 2008, from http:// mojo.csd.anglia.ac.uk/wiki Hof, R. (2006, March 28). A virtual world’s real dollars. Business Week. Retrieved on May 25,
72
2006, from http://www.businessweek.com/technology/ Honey, P., & Mumford, A. (1982). The manual of learning styles. Maidenhead: Peter Honey Publications. Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235-266. Hughes, C. E., & Moshell, E. J. (1997). Shared virtual worlds for education: The ExploreNet experiment. ACM Multimedia, 5(2), 145-154. ISTE (2008). Home page. International Society for Technology in Education. Retrieved August 20, 2008, from http://www.iste.org/ Jonassen, D. H. (1992). Evaluating constructivist learning. In T. M. Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction (pp 137-148). Hillsdale: Ealbaum Associates. Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology: Research and Development, 45(1), 65-95. Jonassen, D. H. (2000). Toward a meta-theory of problem solving. Educational Technology: Research and Development, 48(4), 63-85. Kamel Boulos, M. N., & Toth-Cohen, S. (2008). A sexual health SIM in Second Life. Retrieved from http://sl-sexualhealth.org.uk/ Kindley, R. (2002). The power of simulation-based e-learning (SIMBEL). E-Learning Developers Journal. Retrieved from h t t p: // w w w.el e a r n i n g g u i ld .c o m / p d f / 2 / 091702DES-H.pdf Laurillard, D., Stratfold, M., Luckin, R., Plowman, L., & Taylor, J. (2000). Affordances for learning in a non-linear narrative medium. Journal of
Learning and Assessment with Virtual Worlds
Interactive Media in Education. Retrieved from http://www-jime.open.ac.uk/00/2/index.html Linden Research, Inc. (2008). What is Second Life? Retrieved August 20, 2008, from http://secondlife. com/whatis/ Livingstone, D. (2007). Learning support in multiuser virtual environments. In Proceedings of the European Conference on Games-Based Learning: ECGBL 2007. Academic Conferences Ltd. Lowood, H. (2005). Real-time performance: Machinima and game studies. The International Digital Media and Arts Association Journal 2(1), 10-17. Mayes, T., & de Freitas, S. I. (2004). JISC elearning models desk study: Stage 2: Review of e-learning theories, frameworks and models. Retrieved from http://www.essex.ac.uk/chimera/ projects/JISC/Stage 2 Learning Models (Version 1).pdf Mason, R. (1998). Models of online courses. In L. Banks, C. Graebner, & D. McConnell (Eds.), Networked lifelong learning: Innovative approaches to education and training through the internet. University of Sheffield. Merrill, M. D. (2007). A task-centred instructional strategy. Journal of Research on Technology in Education, 40(1), 33-50. Moodle. (2008). Moodle VLE website. Retrieved August 20, 2008, from http://moodle.org/ MML08 (2008). Workshop for Massively Multi Learner 2008. Higher Education Authority: Anglia Ruskin University, Cambridge, UK. New Media Consortium. (2006). The horizon report. Retrieved from http://www.nmc.org/ pdf/2006_Horizon_Report.pdf. OpenCroquet. (2008). Open Croquet Organisation home page. Retrieved August 20, 2008, from http://www.opencroquet.org/
Open Sim. (2008). Open Simulator Organisation wiki. Retrieved August 20, 2008, from http://opensimulator.org/wiki/ Salmon, G. (2004). E-moderating: The key to online teaching and learning. London: Routledge. Savin-Baden, M. (2000). Problem-based learning in higher education: Untold Stories. Buckingham: Open University Press/SRHE. Schome Community. (2007). The schomeNAGTY Teen Second Life pilot final report: A summary of key findings and lessons learnt. Milton Keynes: The Open University. Retrieved from http://kn.open.ac.uk/public/document. cfm?docid=9851 Sharpe, R., Benfield, G., Roberts, G., & Francis, R. (2006). The undergraduate experience of blended e-learning: A review of UK literature and practice. The Higher Education Academy. Retrieved from http://www.heacademy.ac.uk/assets/York/ documents/ourwork/research/literature_reviews/ blended_elearning_full_review.pdf Simteach. (2008). Simteach educational wiki. Retrieved from http://www.simteach.com/wiki SLOODLE. (2008). SLOODLE organisation website. Retrieved August 20, 2008, from http:// www.sloodle.org/ Steinkuehler, C. (2008). Massively multiplayer online games as an educational technology: An outline for research. Educational Technology, 48(1), 10-21. Sweller, J., & Clark, R. (2006). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner. Educational Psychologist, 42(2), 99-107. Teo, S. C. S., & Williams, J. B. (2006). A theoretical framework for effective online course design. MERLOT Journal of Online Learning and Teaching, 2(1).
73
Learning and Assessment with Virtual Worlds
AppENDIX 1: LECO QuestIi Answer the questions by indicating a numerical level or degree of agreement as follows: 1 very little / none : 2 little / few : 3 average / some : 4 much / many : 5 most / all Question 1) To what extent did you feel prepared for the first assessment? 2) To what extent did you have to find your own material for the first assessment? 3) To what extent did you feel prepared for the machinima assessment? 4) To what extent did you have to find your own material for the machinima assessment? 5) How much do you communicate with the Internet using things like Facebook, MSN, and Skype that provide instant messaging or chat facilities? 6) Have you used virtual environments such as Second Life, or games such as World of Warcraft before this module? Select the degree to which you think you developed the following skills during the machinima exercise 7) Planning (outline design) 8) Project management (setting out and achieving tasks) 9) Group work (allocating and completing tasks) 10) Remote working using Communication media (email, skype, chat etc.) 11) Research – finding resources and information 12) Self Learning – identifying and learning new skills 13) How much of the assessment was developed by communicating from different working together in the same room? 14) How much time did you spend on the machinima assessment? 15) Did you find the machinima assessment difficult? 16) Did you enjoy the machinima assessment? 17) Please state one thing you liked about the machinima assessment: 18) Please state one thing you disliked about the machinima assessment: Which of the following motivated you during the machinima assessment? 19) The challenge to get a good result 20) Working with a group 21) Getting a good mark 22) Interesting environment 23) Creating something to show others 24) Anything else:
74
locations, rather than
1
2
3
4
5
Learning and Assessment with Virtual Worlds
APPpp: Student Responses to IEC Module Questionnaire Student Responses to IECO Module Questionnaire Numerically coded responses
Text based questions
Question No.
1 none
2 few
3 some
4 many
5 most
Text responses were grouped into the categories as shown
1
2
0
10
4
3
Q 17, 18 what did you like / dislike?
2
1
3
10
3
0
Response
3
0
1
14
3
1
4
0
2
3
9
5
0
0
4
6
6
1
7
1
8
Liked
Disliked
No response
2
4
3
Second Life*
1
6
5
8
Assessment
7
1
3
2
4
Group Work
3
2
4
8
3
0
Tools
2
3
1
4
11
2
0
End result
2
9
1
5
8
5
1
Nothing
10
2
2
6
8
0
11
0
3
10
0
4
12
1
0
6
7
3
Response
13
6
4
1
3
2
No response
8
14
0
2
10
2
2
Change SL
2
15
0
3
9
3
2
More help
2
16
1
1
9
5
2
Less Machinima
1
19
0
2
4
5
3
More Machinima
2
20
1
1
6
7
1
Change tools
1
21
1
1
7
3
2
22
1
2
5
7
1
23
0
2
6
6
2
1
Q 24 – what did you want to change?
75
76
Chapter V
A Faculty Approach to Implementing Advanced, E-Learning Dependent, Formative and Summative Assessment Practices Paul White Monash University, Australia Greg Duncan Monash University, Australia
ABSTRACT This chapter describes innovative approaches to E-Learning and related assessment, driven by a Faculty Teaching and Learning Technologies Committee within the Faculty of Pharmacy, Monash University, Australia. Using this group as a driver, we have caused institutional change in a Faculty that was previously quite traditional in its approach to teaching and assessment. The authors implemented a strategy for the pilot testing and broad adoption of innovative technologies, using a purpose-driven approach. They have used a range of technologies to increase the level of formative assessment that occurs during lectures to large student cohorts. They have used an audience response system to allow students to test and improve a range of cognitive skills in an “active” lecture environment; they will present an evaluation of this tool. The authors found that student perceptions of the level of feedback rose with the use of the audience response system, as did their perceived use of critical thinking skills. They further discuss the benefits and limitations of the use of audience response systems within the chapter and discuss our use of E-Learning technologies for summative assessment purposes. Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
A Faculty Approach to Implementing Advanced, E-Learning Dependent
Ba Whilst many higher education institutions have developed policies on e-learninga, these are usually broad-ranging in nature and often signify intent rather than practice. In particular, strategic use of e-learning for formative and summative assessment is notably absent in policy documents, and changes to practice occur on an ad-hoc basis. This ‘organic’ approach has the advantage of allowing academics and innovators in information communication technology (ICT) the freedom to try many new technologies to meet specific or individual needs. However, it does result in some cases in the parallel development of equivalent technologies, with consequent inefficiencies of infrastructure, cost and workload. Also, the benefits of such innovations are sometimes not provided to the broader University community. This chapter describes the systematic implementation of technology innovations at the Faculty (School) level within the Faculty of Pharmacy, Monash University, Australia, using both opt-in and mandatory innovation approaches. A need was identified within the Faculty for e-learning alternatives to content-driven, teacher-focused approaches to teaching and learning that were seen to predominate. We describe the nature of the structural changes in Faculty e-learning implementation, and the outcomes of some of the major innovations.
Introductiiior Chage The Faculty of Pharmacy is one of ten Faculties within Monash University, with around 60 fulltime academic staff and over 1000 undergraduate students studying Bachelor of Pharmacy and Bachelor of Pharmaceutical Science degrees. Senior Management in the Faculty has identified high quality, efficient teaching, and independent learning as high priority outcomes for the Faculty.
In recent years, significant funding has been provided to create systems and procedures to support high quality education. From a senior management perspective, the Faculty performance identified in student experience surveys was both a key driver for change, and a key indicator of performance. As an example, scores on survey items related to the adequacy of ‘feedback’ were consistently low within the faculty. While other faculties shared this problem to some degree, our senior Faculty management staff were motivated to improve the teaching within the Faculty to address this issue, among others, in accordance with the desire for high quality, efficient teaching that promoted the independent learning attributes of students. In addition, in our personal view, we saw a Faculty that historically had a deeply embedded history of traditional didactic teaching, with lectures, practical classes and tutorials being the major teaching and learning activities. A typical subject within a course consisted of around 36 lectures, six practical classes of three hours duration, and a number of tutorials. The advantage of this system largely stemmed from the efficiency of content delivery via lectures. Faculty staff, particularly those within basic sciences subjects, were not required to teach large numbers of small group classes, and were free to prepare lectures of high quality. This resulted in students consistently attending lectures given by experts in particular fields of Pharmacy and Pharmaceutical Science. Attainment and understanding of content were generally the major student requirements, and assessment results over time indicated that student attainment of these types of learning objectives was at a high level. The disadvantage of this approach was that there was little in the way of active learning in many of the lectures – few teaching and learning activities gave students ‘time on task’ to develop their critical thinking skills, and few assessment tasks evaluated student capabilities in these areas. Students thus began each learning cycle (i.e. content topic) with lectures which initiated /
77
A Faculty Approach to Implementing Advanced, E-Learning Dependent
Figure 1. Staff and student perceptions of the importance of different cognitive skills/alignment of learning objectives, teaching and learning activities and assessment
encouraged a content attainment approach, and thus they often attended practical classes and tutorials with a view to completing the required content attainment. In summary, the learning was largely staff-driven, that is teacher-centred, and did not stimulate analysis of content or encour-
78
age novel thinking by students often enough. Figure 1 provides some substance to the above observations. In 2006 and 2007, we surveyed two separate groups of academic staff and two separate cohorts of undergraduate students as to their perceptions of
A Faculty Approach to Implementing Advanced, E-Learning Dependent
the way in which the current teaching and learning approaches promoted the development of lower and higher cognitive functions in their courses. Analysis of staff and student survey data reveals a mismatch between the perception of relative importance of lower and higher order cognitive skills, and the number of teaching and learning activities and assessment tasks that addressed these skills (Figure 1). Thus, there were fewer assessment activities that evaluated these higher level skills than those addressing understanding and content attainment objectives. Interestingly, both staff and students had earlier indicated that these higher order skills were very important from a career perspective. Also of note was that the survey data from the students more clearly indicated some lack of alignment between objectives, teaching and learning activities and assessment than the staff survey. Thus, the Faculty has over the last two years strategically planned to develop approaches that developed skills in analysis, critical thinking and synthesis of new ideas. This is in some areas a work in progress, as some of the innovations have been broadly implemented and some are currently being evaluated. A significant body of literature exists on the drivers for change in teaching practices, and the challenges and best practices which have emerged (Chua & Lam, 2007; Davis & Fill, 2007; Hannan, 2005; Hannan, English, & Silver, 1999; Nichols, 2008; White, 2007). A major driver for change in our case, the need or desire to improve student learning, was the most common reason given in a survey by Hannan and colleagues (Hannan et al., 1999). In the same study, an increased student class size is commonly cited as a driver for change (Hannan et al., 1999). The number of undergraduate students within the Faculty has increased from less than 500 to more than 1000 in 2008, and greater student numbers have contributed to the difficulty in embedding critical thinking exercises into our teaching, learning and assessment. Certainly, the use of online, automatedmarking quizzes described in the second part
of the chapter has increased as a direct result of the difficulty of providing feedback to a greater number of students.
The Strategy In 2002, the Faculty established a Teaching and Learning Technologies Committee with a view to develop a policy on the electronic delivery of lecture notes. In the subsequent years, the role of the committee has expanded to include the following objectives: 1.
2.
3. 4. 5.
Facilitate the development of new teaching methods that involve ICT. In particular, to support initiatives that: a. demonstrate quality; b. show evidence of aligning teaching and learning activities with desired learning outcomes; and c. cater for a variety of learning needs and styles. Use information and communication technologies to organise, research, interpret, analyse, communicate and represent knowledge. Create policy regarding the consistent use of ICT across the Faculty where appropriate. Regularly evaluate the use of ICT across the Faculty. Communicate regularly with the Faculty Education Committee regarding ICT use for teaching and learning. This includes an important role in acting as a driver for change in teaching and learning within the Faculty.
The Teaching and Learning Technologies Committee reports to the Faculty Education Committee, and is responsible for Policy development and implementation for innovation in Educational Technologies subject to approval by the Faculty Education Committee. The following is a description of some of the initiatives undertaken
79
A Faculty Approach to Implementing Advanced, E-Learning Dependent
from 2004 to the present under the guidance of the Faculty Teaching and Learning Technologies Committee. Perhaps the most important element of the success of these approaches was the support received from Faculty administration and technical staff. Significant funding was supplied to purchase infrastructure and licences, and equally important was the clear message from the Dean, Associate Dean (Education) and Faculty Manager that teaching quality was a high priority for the Faculty. The chapter is divided into two parts addressing formative assessment and summative assessment. The major exemplar of the innovation used within the Faculty was the use of an audience response system in lectures to stimulate critical thinking via systematized formative assessment. The section on summative assessment describes our experiences with summative online assessments using WebCT/Blackboard as a learning management system.
Frmative Assessment: Enhahaictures Usii a Audiisponse System The limitations of lectures as a teaching and learning activity have been well documented; lectures promote a teacher-focused, content transfer style of teaching and learning, in particular where large student cohorts are concerned (Biggs, 2003; Bligh, 1998; Laurillard, 2002; Ramsden, 2003). The Faculty has introduced a wide range of activities to supplement the lecture / practical class / tutorial paradigm, including problem-based learning sessions, debates, online self-directed learning modules and a series of tasks involving peer assessment. The lecture, however, is still the most common teaching and learning activity. In particular, the ability to promote the development of higher order cognitive skills (e.g., critical thinking, analysis, evaluation, synthesis of new ideas) in lectures has been found to be limited (Biggs,
80
2003; Pickford, 2006). Given that in our context we see these skills as desired graduate attributes and as outcomes that direct specific learning objectives for individual units, there is the potential for a mismatch between outcomes and teaching and learning activities. It is unrealistic to expect students to attain these skills through ‘osmosis’ in a content-driven curriculum; a more strategic approach involves a cycle of learning and practicing these skills in the context of knowledge development attainment, and the provision of peer and lecturer feedback and interaction within each cycle. Pharmacy students face significant challenges when attempting to meet and achieve complex competency standards, including development of higher order cognitive skills that we’ve already identified, during their undergraduate studies in preparation for professional practice. For these reasons, health professional education, especially medical education, has become increasingly interested in problem-based learning (PBL) approaches in undergraduate programs despite the variable evidence that exists of its superiority over other educational methods (Hoffman, Hosokawa, Blake, Headrick, & Johnson, 2006; Hogan & Lundquist, 2006; Rideout et al., 2002; Sanson-Fisher & Lynagh, 2005; Schmidt, Vermeulen, & van der Molen, 2006; Vernon & Blake, 1993). Sanson-Fischer & Lynagh (2005) helpfully reviewed the existing research in problem-based learning especially that which claimed to measure the educational outcomes and impact of PBL in undergraduate medical education. They found that: the most consistently demonstrated advantage of the PBL approach is the personal satisfaction of medical students engaged in this form of learning and their superior interpersonal skills. The importance given to these aspects of the educational process is, perhaps, a matter of social and institutional values. One could speculate that a more enjoyable, formative educational experience
A Faculty Approach to Implementing Advanced, E-Learning Dependent
may translate to a greater resilience when coping with potential difficulties in one’s professional life (Sanson-Fischer & Lynagh, 2005, p. 258). Such findings were reflected in a study of undergraduate nursing education where Rideout et al., (2002) compared the educational outcomes of students enrolled in a problem-based program with those in a more traditional didactic curriculum. In this study there were no significant differences in participants’ abilities to make clinical decisions or their overall assessment scores, but notably students scored higher on scales related to their overall satisfaction with the program and the development of independent learning skills. In an earlier meta-analysis Vernon & Blake (1993, p. 542) investigated problem-based learning and undertook a meta-analyses on 35 studies representing 19 institutions. In 22 studies PBL was found to be significantly superior with respect to students’ program evaluations and measures of students’ clinical performance, however, PBL and traditional methods did not differ on miscellaneous tests of factual knowledge. Colliver & Markwell (2007) more recently argued that the debate about the educational merits and ‘superiority’ of problem-based learning continues. They reviewed four studies from a special edition of the journal Medical Education and argued that the papers would not convince ‘skeptics’ of its merits and they were critical of the design and conclusions of each study. Despite these concerns PBL and its variants such as case-based learning are well-established as the preferred models in health professional curricula. Where alternatives to a lecture-based curriculum are not desired or practical, various innovative methods have been used to increase the value of lectures. The Faculty chose to perform a pilot study of the use of a Keepad Audience Response System in two undergraduate Pharmacy units. From a pedagogical perspective, the aim of the pilot study was to determine whether use of the Audience Response System would alter student
perceptions of ‘the lecture experience’, in particular, their perception of how they learned during lectures (e.g., their development of critical thinking and analysis skills), and the level of interaction and feedback students perceived to occur during the lecture. Issues of feedback were of particular interest, as feedback has been identified, through a number of internal course review and curriculum evaluation processes and the institutional student satisfaction surveys noted earlier, as an area of student dissatisfaction.
The Pilot Project: The Process and Ealuation The main purpose of this pilot project was to provide formative assessment and increase the attainment of higher order cognitive skills for students undertaking the bachelor of Pharmacy and the Bachelor of Pharmaceutical Science. Furthermore we aimed to increase engagement of students during lectures, to increase two-way communication between students and lecturers during lectures and to provide a forum for students to express opinions on matters related to teaching and learning. The hardware and licence for the use of the technology was obtained from Keepad Inc. The purchases required significant capital expenditure. The system consisted of the Keepad “clickers”, which sent a radiofrequency signal via USB receivers to the lecture theatre computers, with the proprietary software allowing computation of input data. This data was then recorded within the software, and instantaneously produced a summary histogram that appeared within the PowerPoint file being used to show the questions. Figure 2 shows some examples of data generated immediately following questions being posed.
Stage 1: Teething Problems There were a number of initial difficulties experienced when installing software on staff and
81
A Faculty Approach to Implementing Advanced, E-Learning Dependent
lecture theatre computers, and consequently in staff training. The receivers for the radio signals coming from the devices required initialising each receiver on each computer, and in earlier versions of the software this involved administrator access (later software versions overcame this difficulty). When the system was demonstrated to potential early adopters (academic staff looking to use the system), there were frequent early difficulties during the training sessions. This had a significantly detrimental effect on uptake of the technology in the first 18 months of use within the Faculty. Only recently, as ICT, training staff and technology champions became more familiar with the technology was this overcome. Once working within a given session, the audience response system was found to be robust. Students were immediately engaged by the technology, and often rushed to the lectern to receive their Keepad device. An issue that immediately emerged was delivering the devices to students – determining
the most sensible way to distribute the devices to students. After some trial and error, it was found that using student volunteers to take the devices to all parts of the lecture theatre, and retrieve the devices at the end of the class, was the most effective practice. In the first few sessions, staff and students were allowed time to become familiar with the devices and the question / answer format. Over the six months of the trial, some good practice principles were established through trial and error. These were: 1.
Figure 2. Examples of questions at the start of each session
82
Use 3-8 questions per lecture. At least 3 questions per lecture appears to be the minimum number of questions posed in order to justify the use of the technology, in particular where some time is taken out of the lecture to distribute the response devices. With regards to the maximum number of questions, the aim with the benefit of expe-
A Faculty Approach to Implementing Advanced, E-Learning Dependent
2.
rience is to spend an ideal amount of time considering content, posing questions and considering the responses and novel ideas. Each Keepad question generates between 0 and 10 minutes of discussion with a median of perhaps 3 minutes, and thus more than 8 questions can take up a majority of the lecture time to pose and discuss. We found that student response rates did not diminish between questions 1-6 posed (see Table 1), and thus there was no evidence of “response fatigue”. In the few instances where we asked more than 6 questions, the response rate did fall and the number of obviously incorrect answers increased (data not shown). We settled on 3 - 8 questions per lecture in order to achieve the best balance between content consideration, discussion and synthesis of new ideas, and in order to avoid student fatigue. Use the devices in every second lecture. This was consistent with student surveys and academic staff perception, as more frequent use was found to result in diminished student engagement. We also have some experience with more frequent use than every second lecture. In the first month of our trial, the Keepads were used in every lecture. This coincided with a response rate below 60% of students enrolled in the Unit for those lectures, which is less than the later %
3.
4.
5.
response where the devices were generally used every second lecture. Start each session with a question unrelated to content. An engaging way to start the session was to use a question which probed student opinion on a matter of teaching and learning activities or assessment procedures (see Figure 2). Use a single question addressing knowledge / understanding objectives, posed every 15 minutes. The study of Pharmacy and Pharmaceutical Science requires students to memorise and recall a large number of facts in order to produce informed critical thinking. In order to achieve an appropriate balance, students were asked questions assessing recall / understanding objectives on carefully selected, crucial matters. (Figure 3). Use a problem-based approach to allow students to consider content at a deeper level. The major objectives of the use of the audience response system were to stimulate student analysis of content and encourage synthesis of new ideas. Once per lecture, a problem was presented to the students, and their immediate responses to a series of questions determined over the next 10 minutes of the class. Figure 4 shows some
Table 1. The percentage of the students enrolled in VCP2042 in 2006 and 2007 responding to questions posed (in order of being asked) for Eight Separate Sessions. Q1 Q2 Q3 Q4 Q5 Q6
Mean 71.74204 68.37328 70.78773 72.05035 75.6136 76.9697
SEM 3.871382 4.326355 3.737704 2.723082 3.665542 3.030304
Note. Data presented are mean + SEM.
83
A Faculty Approach to Implementing Advanced, E-Learning Dependent
Figure 3. Examples of review questions (every 15 minutes)
Figure 4. Examples of problem questions used every 30 minutes
Note. The problem solving required in A was not at a high level – the problem was presented too late in the series.
examples of problems presented to students and their responses. After responses to the questions were shown, the lecturer led a discussion of (i) the reasons for any incorrect responses being so; and (ii) the
84
principles / facts / evidence underlying the “correct” response. On some occasions, further discussion resulted in disagreement as to the correct response. The perception of academic staff and students was that the
A Faculty Approach to Implementing Advanced, E-Learning Dependent
use of these types of questions increased the level of higher order cognitive skills. As a summary of an extended evaluation, more than 90% of students reported that the use of the audience response system increased their analysis and critical thinking in most or all lectures. For a more detailed literature review of best practice in the use of audience response system, see Caldwell (2007).
Impact on Student Learning: Pedagogical Implications Improved feedback has been the major focus of the use of audience response systems in higher education. Hounsell et al. (2005) have argued that feedback is a critical component of learning, and that ‘clear sense of how well we are doing’ increases the speed of learning. Students did report a perceived increase in the level of feedback after the use of the audience response system in our context. In a survey of our Bachelor of Pharmacy students in 2006, 83% of students agreed or strongly agreed with the statement “Keepads are an effective means of providing me with feedback during lectures”. Thus, we see the use of the audience response system as an integral part of our strategy to allow students to effectively reflect on their progress and implement change where required. The nature of feedback given using an audience response system is such that it is immediate and very specific to narrow areas of content, and therefore other assessment strategies are required to allow feedback on areas not covered using the audience response system. We use formative and summative assessments of written responses with associated peer and independent evaluation in order to provide feedback related to construction of argument, written communication skills and synthesis of new ideas. Using peer and tutor evaluation, we attempt to create an assessment dialogue between students
and tutors, as suggested by Carless (2006), and also an assessment dialogue between individual students and their peers. We submit that the use of an audience response system also increases students’ critical thinking skills, by embedding the use of analytical thought processes at the beginning of each teaching and learning cycle, rather than at the end. Over 90% of Bachelor of Pharmacy students in 2006 responded that the use of Keepads ‘increased my analysis and critical thinking’ in most or all lectures. Whilst there are some limitations of student assessments of their own learning (see next section), an increased level of engagement of students with more complex content areas appeared to coincide with the use of the audience response system. Discussion group postings after the lecture increased dramatically after lectures involving Keepad use. Thus, while we do not have direct evidence of enhanced student performance in attaining higher order objectives, there is evidence of student perception of greater performance and this coincides with increased engagement both during and after the lecture.
Limitations of the Audience Response System A number of limitations of the use of an audience response system to achieve the stated objectives were identified during the trial and subsequently. The most important of these was the limitation of responses to those presented within the question. This limitation had the effect of limiting student responses to a framework that was lecturer-driven, perhaps at the expense of student ideas that were of value. While lecturer questions to the group sometimes drew these ideas from students, there was no systematic means to accomplish this. Thus, there was a need to complement the lectures using problem-based approaches with online discussion to further tease out ideas / reasons for responses. In particular, the synthesis of new ideas was problematic. We are currently developing this
85
A Faculty Approach to Implementing Advanced, E-Learning Dependent
part of the strategy, in particular by identifying parts of problems that require novel ideas to be produced by students within the session, and summatively assessing these using online discussion groups and within examinations. A weakness that remains within this strategy is the relative lack of feedback on their ideas for new ways to look at the problems posed that students receive prior to these assessments. A problem that arises from interpretation of student perceptions in the absence of before and after intervention data is that students are not always reliable assessors of their own learning (Kruger & Dunning, 1999, 2002). There was no valid comparison of marks between cohorts to be made in this instance, as we are in the process of changing the curriculum significantly from year to year, and thus there was a major confounding variable that would render analysis of the effect of the audience response system invalid. We are currently planning such an evaluation for components of the revised curriculum that involve simple and complex learning objective attainment, for groups which receive audience response system questions and for groups that do not receive audience response system questions.
Integration of the Audience Response System in the FaF At the completion of the trial in 2006, the Faculty ran a series of training sessions to introduce all academic staff to the audience response system and its use. Initially there was some reluctance to use the system, due to failures that occurred in some of the training sessions. Academic staff were extremely positive as to the benefits of the technology, but very wary of spending time incorporating into their teaching a system that was not robust. As mentioned earlier, these teething problems have since been overcome, and the use of the Keepads has increased significantly, with more than 20% of academic staff having attended training sessions and the majority of those hav-
86
ing used the Keepads in a teaching context. It is difficult to generate ideal usage levels for such an innovation – this is only one of a number of teaching and learning activities that can be used to enhance lectures. A goal that is emerging is the use of the audience response system across all year levels of each degree program taught within the faculty. The level of use at the time of writing, whilst much higher than in 2006, has not reached this goal, and the reasons for this are worth considering. In the context of managing change in e-learning in higher education institutions, Nichols (2008) argues that where sustainable change has not occurred, one or more barriers to progress must remain. Potential barriers cited included lack of senior management support, a culture ‘not ready for innovation’, poor professional development, and a mismatch between an e-learning activity and current systems. None of these appear to fit perfectly to the slower than anticipated uptake of audience response system usage in our context, although none could be said to be completely irrelevant. Without doubt, the failure of the software in a staff training session, such that the session had to be discontinued, was an impediment to use. Another issue that is apparent is the lack of an urgent driver for change for some academic staff. The motivation of early adopters of the audience response system within the Faculty was to increase the priority given to teaching and assessment of critical thinking and other higher level student cognitive functions, as well as to increase engagement within lectures and increase feedback. For academic staff who are not early adopters, these drivers are perhaps weaker motivating factors, and the enculturation process may take longer. In order to increase the use of the audience response system, the Faculty has established Quality Assurance procedures for all units linked to student unit evaluations. Where a lack of feedback has been identified by students for a particular unit, the teaching staff are encouraged to seek advice about the use of Keepads. In combination with
A Faculty Approach to Implementing Advanced, E-Learning Dependent
ongoing support and repeated demonstrations, it is anticipated that this strategy will increase the use of the audience response system to approach the ideal level.
G As Figure 2 shows, the use of the audience response system was not confined to consideration of content. It emerged during the trial that the system could be used to improve the communication between students and academic staff regarding broader issues. Some examples of improved communication include the evaluation of supplementary exams (and a discussion of the Faculty decision to discontinue these exams), and the matter of disruption of lectures by students who were talking during lectures. This latter issue was helped considerably by using the audience response system; making the students who were talking aware that their colleagues were unhappy with their behaviour produced significant improvement in the lecture environment, and the responses to Q1 of Fig 2 have subsequently been used to inform students in other cohorts of the same.
ThVaif Summative Assessment Usingi Asynchh Quizzes Learning management systems have been used widely in higher education since the mid-late 1990s. One of the attractive features of these systems is a tool that allows students to test themselves online, providing both formative and summative assessment opportunities. We have been using the learning management system (LMS), WebCT (now Blackboard) since 1999, and have extensive experience with this system. As part of the Faculty e-learning initiatives discussed earlier, we have used online quizzes in a large number of units over that period. One of the major issues
of contention with asynchronous, unsupervised summative assessment is whether the submitted responses are wholly generated by the student being evaluated, or whether there was a contribution from others. For distance education courses, cheating is a particular concern given that most or all of the assessment can involve unsupervised assessment (Trenholm, 2007). In our context, we were interested to determine how much weighting could be justifiably placed on asynchronous online quizzes, i.e. whether they were valid and reliable means of assessment. The Faculty Education structure described earlier played a significant role in the identification of this issue as important. Discussions within the Teaching and Learning Technologies Committee revealed that unsupervised quizzes were increasingly being used to summatively assess students in Medicinal Chemistry and Pharmacology disciplines. Anecdotal evidence presented to the committee included advice from library staff that summative assessments were being performed collaboratively when the instructions clearly indicated that students should do the tests by themselves (i.e. cheating was occurring). Also, a very low variability in quiz scores for tasks which were felt to be of moderate difficulty was reported, and the use of the ‘time taken to perform quiz’ data generated by the learning management system indicated that a small but significant number of students were taking less than one minute to perform a quiz that consisted of 20-30 questions. The committee structure we had created allowed the collective evidence of a problem to be considered within a body that was motivated to ensure integrity of assessment. Thus we recently investigated the discrimination and validity of the assessment that occurs using unsupervised online quizzes.
The Rationale for Online Quizzes Online quizzes were largely designed to assess student attainment of learning objectives over a
87
A Faculty Approach to Implementing Advanced, E-Learning Dependent
one to two week period of the semester. These quizzes consisted of multiple choice, true / false, fill in the blank and short answer format questions, and used the automated marking facility of WebCT. One of the main objectives of the use of the quizzes was to (efficiently) provide regular feedback to students as to their progress. The second component of the quizzes was to act as a summative assessment, so as to motivate students to spend time on the preparation for the assessment and to provide a significant proportion of the overall assessment for the unit.
The Unsupervised Online Quizzes: Ealuating Validity Students were allotted a 12-24 hour period in which the quiz was made available online. Quiz responses were marked using an automated system, and the feedback provided for most questions
– indicated the reasons for the correct response and for incorrect responses being judged as such. To determine whether the online quizzes were valid assessments, we compared students’ performance in individual quizzes with their performance overall, and then performed the same comparison for other tasks which were supervised, such as invigilated final semester exams. Linear regression analyses were performed using GraphPad Prism software on linked student marks for each task within the semester for two separate cohorts of students, i.e.: student performance in individual online assessments, as compared to overall performance across a number of units. Another assessment task included in the analysis was an essay task which was marked using both peer and non-peer (Faculty) marking. The results of the post-hoc evaluation of the assessments are shown in Figure 5. The regression analysis was performed against the assessment
Figure 5. Linear regression analysis of student performance in an unsupervised and supervised example of the same online quiz (a,b), and two other types of assessment for the same cohort (c,d)
88
A Faculty Approach to Implementing Advanced, E-Learning Dependent
considered most likely to indicate student ‘quality’ – the final mark for the unit. Since each individual task only comprised 2.5-5% of the final mark, it was felt that an external measure of student performance was not required. The results indicate that the peer-marked and independently marked assessments were of moderate to high discrimination – there was a clear relationship between student performance in the individual tasks and student performance overall. For the two unsupervised online assessments, there was no clear relationship between student performance in that task and performance in the semester exam. These data raise a number of issues. An experienced educationalist may not have been at all surprised by these findings, and may have questioned the approach taken across the Faculty to use unsupervised quizzes in the first instance. However, anecdotally we know that unsupervised online quizzes are used extensively within higher education without such analysis – the authors have had a number of discussions with academic staff who felt that while there might be a small amount of cheating inherent to this approach, the significant efficiencies involved in unsupervised online assessment outweigh those minor concerns. For proponents of this view, the data we have gathered should prompt some reconsideration. Cheating on tasks that were clearly specified to be performed individually is unethical and should not be tolerated by the Faculty. Finally, the formative value of the assessment tasks is likely to be compromised where the student is simply entering another students’ response. The solution to this problem can be as simple as supervising the online assessment. Of course this diminishes many of the advantages of using an online assessment, and is thus a poor solution. The other option might be to make the quiz formative only (i.e. for feedback purposes). However, it should not be forgotten that (summative) assessment is a major driver for student engagement and learning.
The Faculty is currently evaluating the best ways to get the benefits of formative and summative assessment using unsupervised online quizzes, using the Teaching and Learning Technologies Committee as the forum for discussion and also to identify appropriate actions. One option used to good effect within the distance learning community is to create a system in which students are required to provide their own “assessment validation”, using a signed statement from an observer that the work submitted is their own and was produced under controlled conditions. Armatas & Colbert (Chapter X) make some suggestions for this dilemma in their discussion of security issues and Markham & Hurst (Chapter I) argue that we need new ways of thinking about validity and reliability in this new and challenging era of e-learning and assessment. The following section of the chapter outlines another strategy we have implemented with the assistance of the Faculty Teaching and Learning Technologies Committee to use various technologies in innovative ways to assist in improving our assessment practices. In this report, the context is a postgraduate unit of study undertaken by working professionals, rather than a large undergraduate program reported in the previous section.
Smmative Assessment Using Asynchronous Di scussii Goups and ‘b lended’ Techh Asynchronous discussion groups form the backbone of a flexibly-delivered graduate unit in evidence based clinical practice. These discussions dovetail with a number of other elements including directed readings, reflective exercises, database searches, e-tutorials and practice-setting observations as well as optional Skype chats to form a blended e-learning approach to learning for this unit. Reporting on the experiences of each of the elements and their outcomes are incorporated
89
A Faculty Approach to Implementing Advanced, E-Learning Dependent
into assessment along with the contributions to the asynchronous discussions. In designing the unit, significant engagement with technology has been included as one of the goals to increase the capacity of students to use electronic resources available to them in the work place and elsewhere. The knowledge and skills students gain from this unit translate directly into important knowledge and skills in contemporary health care provision. The unit is delivered into a suite of programs including the Master of Clinical Pharmacy, the Graduate Certificate in Pharmacy Practice and the Graduate Diploma/Masters in Wound Care within our Faculty. It is important to note that students in this unit are clinical practitioners from a range of disciplines including pharmacy, nursing, podiatry and medicine so far as the development of expertise in evidence-based practice is a generic skill germane to health professional education broadly. Not only do students come from diverse disciplines within health with a variety of levels of practice experience but they also come from geographically diverse places. A face-to-face course would be accessible only to students in the metropolitan area so it was decided to deliver the unit online as potential students were not only based in metropolitan Melbourne but across rural and regional areas of the state, interstate as well as internationally in New Zealand, various parts of Asia and the Middle East. While the student cohorts have been relatively small, (seven to 25), we now have four years experience in the style of pedagogy and the way in which the identified technologies have supported the pedagogical approach. Asynchronous discussions were chosen for this unit as they met a number of logistical and educational needs. The logistical drivers for using asynchronous discussions are quite obvious. As a communication tool they provide the opportunity to connect with a large number of people and provide the flexibility for interaction with offcampus students, who may have previously lacked
90
connection with teachers and other students. The inherent flexibility of asynchronous discussion groups allows both teachers and students to integrate study into their normal daily lives of work, family and other commitments while maintaining on-going communication. Students read and post at a time that sits within the course framework but at a time of day that meets their needs. The frequency of participation will vary but course work designed for flexible delivery traditionally allows for this approach to participation. From an educational perspective, well planned and clearly structured discussions facilitate a number of outcomes. In this unit there were specific goals for using these discussions in terms of what effect it would have on overall delivery and impact of the unit starting form quite basic uses and impacts through to complex and detailed activities. Principally for us some of these were: 1.
2.
Delivery of messages to whole class: Many students come to this unit after several years away from a formal study environment, The use of asynchronous discussions is generally unfamiliar to them for communication let alone education, so there is advantage in starting to use these groups for simple tasks and activities initially. This involves scheduling information, introduction to members of each small discussion group, provision of more detail about the unit or its activities not available else where and initial provision of some learning materials as attachments or included in the text of postings. The goal at this stage is to stimulate some discussion, whether on the topic at hand or in a general social context just so that the process becomes familiar and students become comfortable with the technology at its most basic level. Response to content questions for whole class: Many teaching staff have useful (though sometimes less than useful) and im-
A Faculty Approach to Implementing Advanced, E-Learning Dependent
3.
portant questions asked of them by students either face-to-face at the end of a class or lecture, in their offices and by email about course content, processes, assessments, etc. Students in this unit don’t have the luxury of face-to-face contact so use of a ‘general‘ discussion group is encouraged to post such questions so that the entire class may benefit from the answer, be it logistical or contextual. Discussion groups provide a very useful method of both receiving these questions and providing responses to an entire group. For this to work well, students are encouraged to post their questions to the discussion group (or email it to the moderator for posting if they want to remain anonymous) and the moderator checks this specific general discussion regularly for postings. Social connection between students and staff: As students in this unit are usually physically very distant from each other, the social dimension of a discussion group can allow students to experience some social interaction that has ongoing benefits. Apart from the ‘feel good’ factor, it may allow familiarity to develop between students, which encourages participation in the structured discussions for learning. ‘Knowing’ the other students in their individual groups makes the structured interaction easier especially as students are expected to comment on the postings of others thoughts and experiences. With an increased level of participation though the social dimension, students write more and in different styles, depending on the interaction, which has the potential benefit to improve written communication skills. In most cases, these discussions are necessarily separate from the rest of the structured course discussions, so a separate group is set up for this specific purpose and students encouraged to post to the various groups appropriately. Most LMSs
4.
5.
allow for a number of discussion groups to be established and managed within a course. Course feedback and evaluation: On or near completion of this course of study feedback is requested. Apart from the university course evaluations, students in this unit are invited to feedback on their experience of the unit and reflect particularly on the impact it has made on their professional practice or service delivery. Many courses will survey students on completion but discussion group feedback has the potential to offer more formative evaluative data; it allows for individual expression of strengths and weaknesses, suggestions for future development, etc., as well as reflection on how the learning has impacted on their perceived approach to professional practice. Assessment of learning: When students meet the requirements for posting this is a simple indication that the process requirements have been met and suggests competence with use of the LMS and discussion groups.
The assessment of content is explored in more detail subsequently. 6.
7.
Structured engagement with content (may include formative assessment): The use of a structured approach to content is described further on with examples provided in the key features list. Discussion group-based summative assessment: This form of assessment can range from a relatively simple approach such as the allocation of marks per meaningful (or other) post up to a certain value, through to an evaluation of engagement with set questions or tasks and a relative assessment of demonstration of knowledge acquisition or application through the postings made.
91
A Faculty Approach to Implementing Advanced, E-Learning Dependent
The Unit Assessment Structure Using the E-Learning Group to Inform Learning and Assessment Engagement Students enrolled in this unit engage with the asynchronous discussions several times a week. The nature of both required tasks and recommended activities means that students demonstrate their learning through the significant contributions made to these discussions. In recognition for the participation, the degree of engagement and the amount contributed, a proportion of the final mark for the unit is given for asynchronous discussion participation. This is somewhat subjective but in an ‘assessment drives learning’ context our experience indicates that it encourages participation at an appropriate level from all. Students commonly contribute to the discussion groups at a much higher rate than required as they find the interaction stimulating. To facilitate further student interaction students may have access to webcams and headsets to allow for face to face discussions with colleagues in the class and the staff using interfaces such as Skype or other web chat facilities. This is popular with only some of the cohort so no structured educational aspect or any assessment is attributed o this interaction. The majority of the assessment for the unit is not derived specifically from the asynchronous discussions, but from assignment work and presentations that draw on the experience in these groups. Students use clinical knowledge gaps form their professional practice settings as the basis for the unit’s work and introduce these early to their discussion group. This gives a meaningful context to the work being done. Using this clinical knowledge gap students work through the elements of the unit that allow them to understand the nature of the knowledge gap, learn strategies for searching for relevant information, critically evaluating the information found and then applying this new knowledge to the clinical situation from which the original gap was derived. After
92
each of these steps, a form of written assessment is submitted which is based on and draws from the discussion group’s interaction. So while not directly assessing the discussion group at each point, it is how knowledge attitudes and skills are explored through the groups that they contribute significantly to the assessment tasks.
Skype and Desktop Web Conferencing to Support the E-Learning Journal Club Task The final element of assessment is the presentation of a critical appraisal of a randomised controlled trial in a ‘Journal Club’ style presentation to a small group of peers and assessors, to stimulate discussion around the methods and outcomes described by the paper. The appraisal skills and knowledge base will be derived from the e-learning experiences in the unit. As the majority of the students in the unit live outside the metropolitan area flexible approaches to the journal club presentation are used. Students who can attend the university conveniently often do so for a faceto-face interaction but all the journal clubs are conducted in a multi-modal fashion. Students are able to remotely present using teleconferencing (with local moderator managing slide transitions), or web conference using Marratech, Elluminate or other software. They may use Skype for the face-to-face effect, but Marratech and Elluminate allow for the student to not only be seen on screen while presenting but to also control slide transitions, use highlighting or bring in other files as necessary. Most remote students prefer to teleconference often commenting they are a little intimidated by the technology for web-conferencing. The use of tele- and web-conferencing allows for fairer assessment of remote students as a proportion of the assessment is on presentation style and engagement with peers during the journal club. This is easier when students can see as well as hear each other and less of the subtleties of group interaction are lost.
A Faculty Approach to Implementing Advanced, E-Learning Dependent
The Student Evaluation of ‘Blended’ Technologies to Support Learning and Assessment In evaluating this course of study, feedback from students about the course content and delivery was derived from the university’s standard unit evaluation data (a web based survey). Sample positive and negative responses have been included below but it has been interesting to observe the nature of responses. Most students, while finding this technological engagement challenging due to their lack of familiarity with it generally, were enthusiastic as they recognised the benefit to their professional practice in doing so. A number though found the unit too much driven by technology and the need for them to engage to be able to access all of the required information was not acceptable t them. As in many educational settings, these students wanted all of the information to be presented to them and not for them to have to develop the skills to find it before beginning the appraisal process. The open ended questions on the evaluation ask for the best aspects of the course as well as areas in need of improvement to be identified. The impact of both the moderation and structured approach were significant issues for students. Unit content aside, the feedback on delivery varied and was a reflection on a number of factors – for example, student expectation and previous experience, moderator participation, difficulty in spacing material released so that weekly workloads were similar. The following annotated qualitative comments indicate the students perception of the unit and the blended technologies used. Best Aspects “Clear time allocated to tasks to ensure you • don't get behind” • “Lecturers are interactive but wait for discussion around a topic before adding their response. Moderators are very knowledge-
•
able and very encouraging for people to enhance their skills”“The moderator has had very little input into the discussion and questioning, this is in stark contrast to other online subjects I have completed. With such small group sizes and minimal moderator involvement, the discussion group is not achieving its full potential.” “Online delivery method learning strategies”
Aspects in Need of Improvement: • “Inflexible release of course materials.” • “Time frames for completing course work readings, tutorials and assessments were very tight, need to spread the workload a bit more evenly. First 6-8 weeks very stressful.” No articles or readings were provided. All • had to be accessed online or by research, which increased the amount of time that I needed to spend on this unit, taking away from time in others.” • “Group size needs to facilitate adequate discussion. There has been a total of 4 students in the group, all previous online subjects completed have had 8+ students which allowed issues and topics to be explored. The current group size has resulted in quite strained discussion and only skimming the surface of issues and topics.” In the complementary institutional survey evaluation data that is available for two semesters (n=10 and 7 responses) the statement “The online teaching materials associated with this unit were a useful resource” 70% and 100% either agreed or agreed strongly. The unit also rated very highly on intellectual stimulation (80 and 100% agree/agree strongly) and enabling students to meet the identified learning objectives (70% and 100%). There is considerable anecdotal feedback from students on how this experience has changed their practice has been overwhelmingly positive. On
93
A Faculty Approach to Implementing Advanced, E-Learning Dependent
completion of this unit, students describe moving to a “different level of practice”, being “more highly regard by their peers”, their “opinions being sought by those at higher levels of responsibility” and ’felling valued for the evidence-based contributions they make‘. Most describe a new part of their professional practice role is to educate their colleagues on how to become a structured evidence based practitioner to improve patient care, organisational, efficiency and management of their won activities. While a number struggled with the demands of the unit in terms of content and delivery, their positive outcome has been a driver that encourages further development of teaching and learning in this area and in refining the way it is delivered maintaining the flexibility but continually improving access for those for whom this approach is a foreign land. Overall, in our experience of teaching evidence based clinical practice to postgraduate coursework students in an e-learning environment, the use of asynchronous discussions has proved to be an exciting and stimulating method with clear strengths but as with any method, inherent challenges which require extra effort in order to minimize their impact on students and optimise learning outcomes. Furthermore, this e-learning environment has been complemented by other technologies that were not available to us when the Faculty first began using a standard LMS. These technologies such as real-time desktop web conferencing have allowed us to innovate in assessment to serve the needs of postgraduate students. In particular the way in which our ‘blended’ technologies model (So & Brush, 2008) has evolved has we believe provided an added dimension to our teaching as it has assisted our consideration of the types of assessment strategies that suit our particular context; that is the substantive course content related to evidence-based practice and the geographically dispersed multi-disciplinary student cohort. Clearly for us our experience over several years illustrates that assessment drives
94
learning and in cohorts such as ours, busy professionals who are returning to study, their expect to be engaged with relevant and practical learning experiences to enhance their practice and support them in their learning. In fact the model we have devised for this unit, has served both formative and summative purposes and also contributed to the professional development of students by exposing them to technologies that will be used in their workplaces.
Conclusiiture Diections Over the past six years, the Faculty has slowly embraced systematic change with respect to the use of technology. Subjectively, one of the best features of the implementation of the new technologies has been the clear understanding of the desired learning outcomes prior to the decision as to which technology to implement. In most cases, the horse has come before the metaphorical cart, and future innovations will need to be justified on educational grounds rather than technological. Already, the range of new technologies available to Universities is remarkable; from our perspective the best results are achieved by selecting options that meet teaching needs. The challenge for the future in terms of implementation is to encourage diversity and at the same time deploy those technologies that have been trialled successfully in as many suitable contexts as possible. The technologies that have been implemented in our context have been implemented to achieve specific goals; increasing active learning and feedback within lectures for the audience response system and to use a blended technologies approach for formative and summative assessment. What we are working towards is a well-rounded experience for students within each of our courses, where assessment technologies such as these are employed in combination to improve assessment and feedback both during and after the class.
A Faculty Approach to Implementing Advanced, E-Learning Dependent
To elaborate, a student who can test their early thoughts and ideas within a lecture, and then fully explore issues raised online whilst gaining regular feedback, will be very well placed to achieve the desired learning outcomes. Also, the summative assessment procedures can be built into the teaching and learning activities in this way. From a Faculty strategic viewpoint, the Teaching and Learning Technologies Committee is well placed to drive a move towards such a cohesive approach.
REFERENCES Biggs, J. (2003). Teaching for quality learning at university: What the student does (2nd ed.). Buckingham: Society for Research into Higher Education and Open University Press. Caldwell, J. E. (2007). Clickers in the large classroom: Current research and best-practice tips. CBE Life Sciences Education, 6(1), 9-20. Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31(2), 219-233. Chua, A., & Lam, W. (2007). Quality assurance in online education: The Universitas 21 global approach. British Journal of Educational Technology 38(1), 133-152. Colliver, J., & Markwell, S. (2007). Research on problem-based learning: The need for critical analysis of methods and results. Medical Education, 41, 533-535.
Hannan, A., English, S., & Silver, H. (1999). Why innovate? Some preliminary findings from a research project on “Innovations in teaching and learning in higher education”. Studies in Higher Education, 24(3), 279-289. Hoffman, K., Hosokawa, M., Blake, R., Headrick, L., & Johnson, G. (2006). Problem-based learning outcomes: Ten years of experience at the University of Missouri-Columbia school of medicine. Academic Medicine, 81(7), 617-625. Hogan, S., & Lundquist, L. (2006). The impact of problem-based learning on students’ perceptions of preparedness for advanced pharmacy practice experiences. American Journal of Pharmaceutical Education, 70(4), 1-7. Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated selfassessments. Journal of Personality and Social Psychology, 77(6), 1121-1134. Kruger, J., & Dunning, D. (2002). Unskilled and unaware--but why? A reply to Krueger and Mueller (2002). Journal of Personality and Social Psychology, 82(2), 189-192. Nichols, M. (2008). Institutional perspectives: The challenges of e-learning diffusion British Journal of Educational Technology, 39(4), 598-609. Pickford, R. T. (2006, July). The art of teaching: A model for the lecture in the 21st century. Paper presented at the The Higher Education Academy Annual Conference.
Davis, H. C., & Fill, K. (2007). Embedding blended learning in a university’s teaching culture: Experiences and reflections. British Journal of Educational Technology, 38(5), 817-828.
Rideout, E., England, V., Oxford, B., FothergillBourbonnais, F., Ingram, C., Benson, G., et al. (2002). A comparison of problem-based and conventional curricula in nursing education. Advances in Health Sciences Education, 7.
Hannan, A. (2005). Innovating in higher education: Contexts for change in learning technology. British Journal of Educational Technology, 36 (6), 975-985.
Sanson-Fisher, R. W., & Lynagh, M. C. (2005). Problem-based learning: A dissemination success story? . Medical Journal of Australia, 183(5), 258-260.
95
A Faculty Approach to Implementing Advanced, E-Learning Dependent
Schmidt, H. G., Vermeulen, L., & van der Molen, H. T. (2006). Long term effects of problembased learning: A comparison of competencies acquired by graduates of a problem-based and a conventional medical school. Medical Education, 40, 562-567. So, H. J., & Brush, T. A. (2008).Student Perceptions of Collaborative Learning, Social Presence and Satisfaction in a Blended Learning Environment: Relationships and Critical Factors , Computers & Education, 51(1), 318-336. Trenholm, S. (2007). A review of cheating in fully asynchronous online courses: A math or factbased course perspective. Journal of Educational Technology Systems, 35(3), 281-300.
96
Vernon, D. T., & Blake, R. L. (1993). Does problem-based learning work? A meta-analysis of evaluative research. Academic Medicine, 68(7), 542-554. White, S. (2007). Critical success factors for elearning and institutional change – some organisational perspectives on campus-wide e-learning. British Journal of Educational Technology, 38 (5), 840-850.
ENDNOTE a
For example see University of Queensland (http://elearn.uq.edu.au/policies.html); University of Sydney (http://www.usyd.edu. au/learning/governance/elearning_docs/elearning_model.pdf)
97
Chapter VI
Ensuring Security and Integrity of Data for Online Assessment Christine Armatas Victoria University, Australia Bernard Colbert Telstra Corporation Ltd., Australia
ABSTRACT Two challenges with online assessment are making sure data collected is secure and authenticating the data source. The first challenge relates to issues such as network security, robustness against attack and data management. The second is currently a significant impediment to widespread implementation of formal online assessment due to difficulties ensuring the identity of the person completing the assessment. In this chapter the authors discuss technical aspects associated with keeping data secure and the implications this has for delivering online assessment. The chapter also examines technologies that can assist with the issue of authenticating the identity of individuals completing online assessments and we provide some practical advice for those considering using online assessment tools. To conclude the chapter, the authors look at technologies likely to be available in the future and examine how these could be used to conduct online assessment that ensures data security and integrity without imposing an unreasonable burden on users.
INTRODUCTION While online or e-assessment has many potential benefits for both students and teachers, it is not without its challenges. As with traditional methods
for delivering examinations, the procedures associated with online assessment need to ensure any assessment data is secure. This means ensuring exam materials are only accessed by those who are entitled to access them, and that any access
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Ensuring Security and Integrity of Data for Online Assessment
is appropriate to the user’s role. The second challenge is authenticating the source of the data and being able to ensure the identity of the person completing the assessment. In this chapter we are concerned with issues relating to security and integrity of assessment data transmitted online over any network - fixed or mobile, wired or wireless. We examine the four key functions technology can facilitate and discuss their application in the context of online assessment. Examples from the first author’s experience teaching and assessing online are used to illustrate the practical implications for staff wanting to use online assessment. We also briefly look at future technologies and how they could fundamentally change how we do online assessment.
Ba Conversations about online assessment generally reveal two schools of thought. Supporters will often talk about the flexibility online assessment provides such that students are able to take a test anywhere and at any time (Cann, 2005; Engelbrecht & Harding, 2004). Or they mention the decreased administrative and marking overheads associated with online assessment as a significant benefit (James, McInnis, & Devlin, 2002; Nicol, 2007a). Online assessment tasks can also assist educators to assess a broader range of skills, provide students with different types of assessments, including ones not easily achieved using traditional assessments methods. They can also provide students with benefits such as timely and informative feedback on their progress, as well as teaching students new skills and ways of studying and learning (James, et al., 2002). The second group most frequently point to the difficulties associated with verifying the identity of the person taking an online test, or they express concerns about how to stop cheating and the problem with networks crashing during tests. They might also be understandably concerned with protecting the
98
confidentiality of assessment material, wanting to be sure that assessment material is not accessed by unauthorized persons, that assessment data, including student responses and grades, are not altered. What is often missing in these vigorous and important discussions is an understanding of how technology can be used to address some of the very valid concerns associated with online assessment and what additional benefits it can provide if implemented appropriately. Dependability is the key requirement for eassessment according to Weippl (2007), which encompasses a number of factors including availability, reliability, safety, integrity, and maintainability. He stresses that when e-assessment is used for examinations, all these aspects of dependability become critical. The assessment needs to be available when it is required, so measures should be in place to prevent attacks on availability, which are referred to as Denial of Service (DoS) attacks. Not only must the service be available when required, but there needs to be continuity of the service. Before, during and after the exam the integrity of exam questions and materials, student responses and grades need to be ensured. Any system used to deliver an online exam needs to be safe and maintainable. The essentials of secured networks include having a proper network security policy, enforcing identification, confidentiality and integrity and implementing proper compliance monitoring mechanisms (von Solms & Marais, 2004). While aspects of safety and maintainability seem to be mainly a responsibility of the Information Technology (IT) department in higher education institutions for example, as we will see, responsibility for availability, reliability and integrity belongs to those preparing, administering and marking the exam as well as the IT department. Online exams can fail for reasons such as software bugs as well as because of hardware or other infrastructure failure. As Weippl points out, “[E]-learning has a strong technological component but it is important to keep in mind
Ensuring Security and Integrity of Data for Online Assessment
that it also encompasses a lot of organization issues.” (Weippl, 2007. p. 294). In order to deliver dependable e-assessment, all parties need to work together to ensure the pieces of the puzzle are in place. Academics understand well the issues associated with reliability and integrity as they relate to traditional exam formats. However, this is not always the case when it comes to online assessment, where often technology is “blamed” for unfairly attributed failures when in fact human error is to blame. When this happens the reaction can be to reject technology rather than recognizing the human element to the delivery failure and taking steps to ensure it is rectified and does not occur again. Assessment generally, and exams in particular, are important and sensitive issues with students and staff. Therefore, when online assessment goes astray, there is a tendency to abandon online assessment as a means for assessing students. This is unfortunate for several reasons, including the aforementioned one that often the problem is not technical in origin but rather results from an omission or lack of planning. A second reason is that technology has great potential to enhance the assessment process. In particular, there are four areas in which technology can play an important role in the assessment process. The first is identification, which has two parts – authentication and verification. Authentication is when you provide proof that you are entitled to take the assessment, or access or change assessment materials or data, while verification is providing proof that you are who you claim to be. As will be discussed in more detail later, depending on the type of assessment and the context in which it is being used, sometimes it is critical to know who is accessing or submitting assessment material and data and sometimes it doesn’t matter. A second role for technology is ensuring the integrity of data, so that assessment responses are transmitted properly and data is protected from being altered. Access control is the third key area in which technology can be used effectively and involves encryption,
separating sessions, and version control. What sort of access policy is necessary for a given assessment needs to be considered in the context of the type of assessment and the purposes it is to be used for, and can extend to control over printing, viewing, sharing, editing/modifying, and saving copies of assessment material. Finally, technology can provide tools for logging and auditing data, which academics can use for a range of purposes related to assessment and teaching and learning. Logging and auditing also serve as a non-repudiation mechanism, provided that the security and integrity of data can be assured. Being able to track user actions through activity logs is important as actions can be traced back to users, who are then unable to plausibly deny having performed an action (Weippl, 2005).
Online Assessment for Frmative and Summative Purposes There is a growing body of research that computer assisted assessment (CAA) is successful for assessing learning and providing timely and useful feedback to students (Nicol, 2007a). Furthermore, as computers have become part of everyday life, students find this an acceptable assessment method (Sim, Holifield, & Brown, 2004). CAA can take many forms including analysis of the content of discussion postings, blogs or wikis, and automated essay and short answer question marking. Multiple choice questions (MCQ) are the most straightforward CAA method and offer advantages such as being automated and repeatable, with the opportunity for students to undertake the assessment in their own time and at their own pace, while also receiving immediate, good quality feedback on their performance (see White & Duncan, Chapter 5 in this volume). Drawbacks to this type of assessment include the time and skill it takes to design good quality questions and that students can guess the correct
99
Ensuring Security and Integrity of Data for Online Assessment
answer (Sim et al., 2004). However, well designed multiple response questions can be very effective in assessing for knowledge acquisition as well as comprehension, application, analysis, synthesis and evaluation. At the same time they offer staff efficiencies in marking and administering assessments (Nicol, 2007b). Nicol (2007a) has argued that if MCQs are used as a formative assessment tool, they should be linked to developing learner self-regulation and autonomy. The first author had similar goals in mind when incorporating MCQ and other forms of CAA into core first and second year psychology units she was responsible for while working at Deakin University. For the first year units, CAA was used as a motivational study tool, providing students with opportunities to assess their understanding of concepts and skills being taught in the course (see Armatas, Holt, & Rice, 2003 for a discussion of developments in these first year psychology units). It also provided students the opportunity to familiarise themselves with the format for the end of semester exam, which was MCQ and worth 60 per cent of their final mark in the unit. Through regular online assessment opportunities, students were able to obtain feedback on their progress much earlier and in far greater detail than had previously been possible given the large student enrolment (~1000 students) and high student to staff ratio. In the second year unit, CAA was taken a step further and contributed to the students’ final grade, albeit with only 10 per cent of their final mark being allocated to these assessments. However, the second year unit was taught fully online using a learning management system (LMS), so formative and summative online assessment was a logical inclusion in the course (see Armatas, Holt, & Rice, 2004 for a description of this fully online unit). The reasons why online summative assessment only attracted a small percentage of the final grade are discussed in more detail later. Suffice to say, in 2004 fully online units were not typical and the teaching team responsible for these units proceeded cautiously.
100
But central to concerns about online assessment were issues of authentication and verification of students and security of examination materials, responses and results.
Identification Accurate identification of users serves multiple functions (Raitman, Ngo, Augar, & Zhou, 2005), including providing a mechanism for controlling access to materials and information and allowing the creation of user profiles to customise and personalise the learning environment. The most common role for identification in online assessment is to help ensure that any assessment results which may be subsequently used for certification purposes can be unambiguously attributed to a specific student (Graf, 2002). However there are others besides students who require access to assessment material and data, including teaching and administrative staff. The Deakin units in which the first author used CAA were delivered via a LMS, through which role based access to test materials was possible. This meant that teaching staff were given access to create or modify the tests and view students’ marks. In contrast, students could only take the tests, and their access to tests was restricted and controlled by the unit co-ordinator. An advantage of using the LMS to deliver the test was that students already had identification credentials for the system (their Deakin username and password) and so we did not need to create or use new ones. Accurate user identification is needed to determine and enforce the access privileges a user has. Although user names and passwords are the most common, many techniques exist for authentication of users— that is, to correctly identify a user. The techniques can be classified into four categories: 1.
what you know — e.g., username and password combinations,
Ensuring Security and Integrity of Data for Online Assessment
2.
3. 4.
what you have — e.g., having an authentication token such as SecureID token or mobile phone subscriber identity module (SIM), what you are — biometrics, for example fingerprints, and where you are — e.g., a particular network access.
These techniques have a range of cost, usability, acceptability and strength/effectiveness that need to be taken into consideration when using online assessment.
Usernames and Passwords The username and password or Personal Identification Number (PIN) combination is an identification method that most people are familiar with. However, problems with username and password/PIN combinations are well known and include passwords and PINs that are easy to break, difficult to remember, written down or lost. The overheads associated with retrieving or resetting lost passwords and PINs and ensuring they are sufficiently robust and stored securely can be substantial. However, acceptability of usernames and passwords/PIN combinations is high and, with appropriate usage guidelines and protocols in place, usability and effectiveness can also be high. An advantage with this form of identification is that students and staff will already have a username and password/PIN to access other university systems such as email and web resources. Using an identification system already in place has the advantage of not creating another one with the additional overheads this requires, as well as being familiar and more easily remembered by users. Concerns about security and strength of passwords and PINs can be managed by instituting procedures such as regularly requiring them to be changed and using software that tests the “strength” of a password or PIN. These can be supplemented with additional measures such as the use of secret questions selected by the user,
with answers presumably known only to them. If users comply with usage policies and procedures and don’t actively conspire to compromise the system, username and passwords can be an effective authentication method.
Hardware Tokens If used correctly, hardware tokens, including mobile phone SIMs, are an effective authentication method. Banks and other institutions are already using secure token technology as an additional authentication mechanism along side username and password combinations. These tokens typically work by requiring a randomly generated number to be entered in addition to a username and password, with the number changing regularly and only lasting for a short period of time before expiring. Short Message Service (SMS) is also used as a security measure, with system generated number sets sent to a registered SIM when a user logs on or attempts other operations in the system requiring authentication. Given the high ownership of mobile phones amongst students, a SMS system could be implemented relatively cheaply to bolster the security of username and password combinations. By comparison, the token system involves the cost of the system and the tokens and requires users to have the token with them when accessing the system. On the downside, both the token or SIM and the username and password combination can be stolen, which makes this method vulnerable. When using usernames, passwords and tokens for authentication, it is important to ensure that the process for obtaining the identification credentials is valid and that the user is accurately identified in the first place. This was another reason for delivering the online assessment via Deakin’s LMS – the authentication of students had already been done by the University on enrolment, at which point student identification numbers, usernames and passwords were assigned to each student. Given that students can access a range of services
101
Ensuring Security and Integrity of Data for Online Assessment
and information with this information, including potentially sensitive examination results, we felt that students had compelling reasons to protect these identification credentials and not compromise them by giving them out to others.
Biometric Technologies Even with valid processes for issuing usernames, passwords and security tokens, there is never complete certainty that the person accessing the system is who they say they are, even though they may possess information or artefacts that support their identity claim. In contrast, biometric technologies are marketed as being effective user verification methods because if the user is the key, stealing or duplicating the key becomes extremely difficult, if not impossible. Biometrics use either physical or behavioural characteristics of an individual: for example fingerprints (physical) or voice (behavioural). Every system requires an enrolment procedure with the individual present. The data collected during enrolment is stored. During validation, information is collected from the individual and compared with the stored data. An algorithm is then applied to determine the similarity between the information collected from the individual and the stored data. Given the nature of biometric data, this information is usually not identical and the system must determine whether the individual is the same as the one who gave the original sample. The system will provide a positive response – the sample was taken from the claimed user – or a negative response – the sample was not taken from the claimed user. The effectiveness of biometrics is related to the false positive and false negative rates. A false positive occurs when the system incorrectly identifies an individual as a valid user; a false negative occurs when the system incorrectly rejects a valid individual. For assessment purposes, both false negatives and false positives create problems as the former prevents a valid user from taking an assessment or accessing assessment materials
102
and data, while the latter allows assessments and related information to be accessed by someone claiming to be someone they are not. Speech is another biometric that could be used for identification and verification of students taking an exam, and for providing role-based access to examination materials for staff. However, speech also has its limitations and problems - state of the art speaker verification has a correct verification rate of about 90 percent after training. There are also other difficulties for speaker verification such as changes in the voice caused by illness or other conditions such as an increased fluency in English. Some speaker verification systems can also be defeated by using voice recordings. To counter this, speaker verification systems can include a “liveness” test, whereby the user has to speak a randomly generated set of numbers or phrases which would be difficult, if not impossible to produce using recordings. Biometric systems are not foolproof, as Graf (2002) points out, citing a German Information Security Agency study which found the recognition rates for several of the 12 systems tested were unsatisfactory and that they could be fooled quite easily and without much effort. The second author’s experience with fingerprint readers shows they are reasonably easy to defeat using only a modest amount of materials available from an arts supply store. A second draw back with biometrics is that they do not preclude collusion, which some students have been known to use as a strategy to defeat assessment security measures. Biometrics can be an effective security measure in situations where people do not wish to collude with another to take advantage of a system – e.g., for banking or other financial purposes. However, in the absence of invigilation, biometrics are not a guaranteed authentication and verification technique because they can be defeated with the co-operation of the individual who is the key. A further drawback to biometric authentication is that these systems require secure readers at the user’s end. For online systems, this means that
Ensuring Security and Integrity of Data for Online Assessment
the user needs access to a reader in order to be authenticated - unless a reader is freely available or portable, remote access becomes difficult. The bottom line is that although biometric and liveness tests may demonstrate that the student is present for the assessment; they do not demonstrate that the student is alone or is not being assisted by other people. As such, they may suitable for providing access to students; however, they should not be relied upon for ensuring that the student is not cheating. The hostility and suspicion that could potentially be invoked if these authentication and verification methods are used should also be taken into account.
Smartcards Smartcards are another cryptographic mechanism that can be used for identification purposes. If the user is required to enter a PIN to activate the smartcard, two-factor authentication is required – i.e., the user must have both the smartcard and the PIN to access the assessment. This provides some protection against theft or unauthorised use, but again does not prevent misuse through collusion. Smartcards can be designed to be tamper resistant increasing their security, but like biometric systems they require additional infrastructure in the form of readers and a mechanism to issue the smartcards. They also require that the user has the smartcard with them when they want to access online assessment, and the smartcard itself can be lost or stolen. These problems combined with a lack of standards for smartcards, means that they are currently not a viable security mechanism for online assessment purposes (Graf, 2002). Web cameras have also been suggested as a tool that could be used to allow students to sit an assessment at a remote location while being monitored to ensure nothing untoward is happening. On the surface this seems reasonable, but there are both technical and other drawbacks that need to be taken into account. A major consideration is the bandwidth required for the camera to be
useful or usable. For example, if the video coding is running at 64kb/s, then it will be a low quality video with little resolution. Greater resolution will require higher bandwidths. Other constraints such as the quality of service of the connexion may mean this approach will be infeasible for many students. In addition, although they create the illusion of surveillance, there are many freely available tools that will allow students to sit in front of the camera while sharing the computer desktop, or using other channels such as instant messaging, to communicate with other people. Thus, web cameras are not likely to be effective if a student is planning to cheat and you risk antagonising other students by the imposition of the camera. As Kambourakis and colleagues point out, even when the problems of authentication and encryption are solved, you still do not necessarily solve the problem of one student sitting or passing an exam for another. This is the case even when measures such as biometrics, tokens and monitoring or video surveillance are implemented. Since this issue remains to be resolved, at this point in time, technology does not offer a foolproof solution for the purposes of identification which means that for distance education students, exams still need to be conducted under controlled conditions, such as at an exam centres (Kambourakis, Kontoni, Rouskas, & Gritzalis, 2007).
Considerations with Identification Methods Usability is an important aspect to be considered with all of these methods of authentication and verification. For example, in theory, passwords can be made very strong by making them long and requiring certain characteristics to make them less predictable. However, this can make it difficult for users, who then respond by writing down passwords or storing them in insecure ways. Random number generation via tokens or SMS requires having the token or the registered
103
Ensuring Security and Integrity of Data for Online Assessment
SIM to hand when access is required, which may not always be possible, practical or desirable. Furthermore, there are accessibility issues to be considered, such as usability for those with visual impairments or physical disabilities. Biometric systems can be easy to use, although more complex to set up and maintain. Speaker verification systems, for example, require an enrolment session and training to establish the speech patterns of the user. Since speech changes with a number of factors, including age and current state of health, the system needs to be adaptable enough to correctly identify a speaker over time, while not disallowing them access when they are entitled to it or allowing access to others not entitled. The acceptability of authentication methods is also an important issue. Most users will not object to being required to remember a short password, however, they may object to having to buy a $100 device or to carry their mobile phone or a security token with them always. They may also object to having their fingerprints taken or an iris scan. Apart from issues of consent, there will always be individuals for whom one or other biometric method will be unsuitable. In addition, the storage and management of any biometric data raises concerns about privacy which need to be addressed. Fortunately, universities already have in place policies and procedures relating to privacy of information, which could be expanded to include any biometric information collected about staff and students. In the future, as biometric technologies improve and are more accessible and widely used, it may be possible to collect biometric data from staff and students and incorporate these in the staff and student identification card system. This information could then be used for authentication purposes. The cost of solutions, both to implement and to run, will also need to be considered. Username and password systems are popular because they can be implemented for low cost. However, mechanisms to increase the security of these systems, such as longer passwords and regular changes of
104
the password will increase the operating costs of these systems. At the same time, it has to be remembered that username and password systems are not robust verification methods. When a username and password are assigned to an individual, steps may need to be taken to verify the individual’s identity. Generally, students and staff at universities are required to provide proof of identity before being issued with a username and password. However, depending on how this is done, a staff member or student may be issued with a username for an identity that does not belong to him or is inaccurate. The ramifications of this may be limited to an individual obtaining a degree under an identity that either doesn’t exist or does not belong to them, or could be more far-reaching. The integrity of procedures for identifying individuals for the purpose of obtaining a username and password can be strengthened by adopting a 100-point identification system such as the one mandated by the Australian Government’s Financial Transactions Reports Act (1988). This type of system is used by banks and other financial institutions and requires individuals to produce multiple identification documents, including one with photograph identification. Adoption of a rigorous system for issuing and renewing user access in the first place will go some way towards helping to ensure that only those entitled to gain access to a system can do so. However, the cost in terms of resources such as time, money and effort, needs to be weighed against what individuals will be able to do with their access. Even with good screening processes at the start, the correct entry of a username and password on a given occasion however, does not guarantee that the person is who they say they are, as the identification details may have been given out to another party or may have been stolen. Depending on the purpose and context, authentication may be sufficient to grant access, or verification methods establishing that the person is who they say they are with a high degree of certainty may
Ensuring Security and Integrity of Data for Online Assessment
be required. For assessment purposes, there may be a range of requirements around authentication and verification which will be discussed later.
Integrity Data integrity mechanisms ensure that data can not be modified or changed without being detected. This is important for both the assessment administrator and the student. For the assessment administrator it means that submitted work cannot be changed either by the student or another party. For the student, it provides a record of work submitted and the assurance that it cannot be altered. The integrity of the data needs to be protected in the communication of the assessment, and in storing the assessment. Communications protocols, such as IPsec and TLS/SSL are able to protect the integrity and confidentiality of the data while it is being communicated and therefore should be used in online assessments. The usual mechanism for protecting the integrity of data and documents is digital signatures (Weippl, 2007). Digital signature schemes use public key cryptography (PKC), where the user has a pair of keys: a public key and a private key. Although these keys are mathematically related, it is not feasible to derive one from the other. The public key and private key are also inverses of each other: that is, if something is encrypted using the public key, it is decrypted using the private key - and vice versa. Since the public and private keys are difficult to relate, the public key can be published without revealing or compromising the private key. Thus, users can be publicly bound to the public key. The other element of digital signatures is the digest or hash. This is a method of assigning a unique number to the document using a digest algorithm or hashing function. These functions have properties that ensure that all documents will have a unique digest, and that given a digest it is difficult to generate a document with that digest. However, although uniqueness can not
be guaranteed, the probability of any two nonidentical documents having the same signature is astronomically small. This means the chances of two documents signed by any two parties having the same signature is also very, very small. The most commonly used digest algorithms are MD5 and SHA1.The digital signature is generated by determining the digest for the document and then the user encrypting the digest with his/her private key. This is then appended to the document. This binds the user to the document, since no one else can easily generate the signature for the document. The signature can be verified by calculating the digest, decrypting the signature using the public key and comparing the two values. If they correspond, then the signature is accepted. If they are different the signature is rejected. One of the challenges of PKC is binding users to public keys. Various schemata have been proposed: however, they all have a common element of a declaration from a trusted authority that the public key is bound to a particular known identity. In a teaching setting, this means that there needs to be some way of registering staff and students and their public keys. This may be done through the university setting up a certification authority to issue public key certificates - signed statements from the university that link the students to their public keys - or by a registration process which stores the public keys in an accessible database or directory. It is important to note that since PKC has several functions, e.g., authentication, signature generation and other cryptographic functions, the students will need to have key pairs for each of these functions. Software exists to support the generation of PKC key pairs: and there is a lot of software which supports digital signature generation and checking. For example S/MIME (Secure/Multipurpose Internet Mail Extensions) and PGP (Pretty Good Privacy) have been integrated into most messaging clients. PGP can also be used for digitally signing documents: the standards body OASIS has developed the Digital Signature Service standards for XML documents. 105
Ensuring Security and Integrity of Data for Online Assessment
There are costs associated with implementing and managing a Digital Signature scheme. These include enrolling and issuing students and staff with keys, software to sign documents, and support costs. More over, the submission of the documents will need to be a three step process. 1. 2. 3.
The user signs the document/work and submits it to the submission system. The submission system validates the signature of the document. The system issues a signed receipt to the user with the time and signature of the document, which allows the user to verify successful submission. (This should be a standard format such as: Document with digital signature number was submitted by Person X at Time Y).
Once the digital signature system is in place however, they are easy to use and can be used from any location. Digital signatures can be misused if the details used to sign a document electronically (which are usually a username and password combination) are stolen or appropriated. However, with appropriate policies and procedures for their use, digital signatures are an excellent means of ensuring data integrity, providing accountability for those accessing assessment data and materials. If a document management system is being used, then data integrity can be ensured using permissions in the system. This means that appropriate privileges need to be set up and enforced, such that users are limited in what they have access to and how they can use their access. An example is that only the document’s creator has modification rights, while the assessor can annotate or read the document, and logs of all accesses to the document are kept. Thus, if disputes concerning the integrity of the document arise, then these will provide a means of resolving the dispute. Data integrity is important for assessment purposes as it provides evidence that the student
106
submitted the work, time of submission and assurance for the student that the work has not been changed after submission. This in turn provides a mechanism to resolve disputes about what assessment was submitted, when and by whom. It also means that, in the event that something disastrous happens, students will be able to resubmit assessment with the original receipt. Another aspect of integrity is ensuring that the system delivering the exam, delivers that exam according the correct specifications. An example of this is where students complete an exam where the questions are drawn from a pool of items of comparable difficulty, but each student is given a different exam. The ability to create multiple versions of an assessment from a question or item bank is a feature of many learning management systems and testing software applications. Two issues arise from this. The first is technical, in that the software must reliably compile the tests properly. The second is that when the items for the test bank are developed there needs to be an appropriate mechanism for checking that items are of comparable difficulty. Without this check, even if the software performs its task reliably, the integrity of the test is compromised even before it has been sighted by the students.
Access Cntrol Access Control refers to those technologies, protocols and processes which provide access to data and control what can be done and by whom. For example, the access control system may provide access to lecture notes, but not allow these notes to be copied or modified. The same system may also allow staff to copy the notes and the author to modify the notes. Generally, access to online learning environments and assessment is only granted to registered users, and individual users are usually restricted as to what they can access in a system depending on their roles and responsibilities. This in turn is reliant on accurate authentication and verification of users.
Ensuring Security and Integrity of Data for Online Assessment
There are two parts to access control: policy formulation, and policy enforcement. The policy determines who can do what to data, while the enforcement is how this is achieved. When formulating policy there are several issues that need to be considered. The first relates to the language in which the policy is written. Policies are normally formulated in natural language, but then need to be translated into a formal language or a set of instructions that a machine is able to follow. Fortunately, most policies are straightforward and the systems support this translation. However, problems can arise if requirements are complex and difficult to express. One strategy for managing access control is to formulate generic policies for staff that are written as templates. This eases the burden on staff and provides consistency, but it is likely that there will be circumstances in which particular sets of access control requirements will be needed which the policy does not cover or disallows. These instances will need to be carefully managed by whoever has responsibility for the policy. The second issue is the achievability and enforceability of the policies. According to Weippl (2005), non-compliance is the greatest risk to security and increases with complexity. There may also be practical reasons why particular policies, though desirable, may not be enforceable. For example, consider a policy that requires that all work must be submitted from within Australia and not from abroad. To overcome this, a student who is offshore may access an Australian ISP and submit work through the Australian ISP, and their submission system would accept this as being from within Australia. Alternatively, a student could be using an ISP which has hosts overseas, which means that the policy would deny the student the ability to submit any assessment, even though they are submitting from within Australia. The third issue is consistency. Policies must be consistent so as to ensure that unintended access does not occur. Most access control systems will apply policies in a particular order
to ensure consistency of outcomes. If this were not enforced, then inconsistencies may lead to significant breaches of policy and provide access to unexpected parties. In particular, unintended consequences need to be considered. For example, a student may hold a position as tutor or lecturer in the department, and the policy may allow all staff to review assessment given to students. The unintended consequence of this would be that this student may then have prior access to exams and other assessments. Some access control systems come with tools to assist in determining the consequences. However, with complex rules, these may provide too much information making it tedious to check. Furthermore, many access control systems overlook the usability of the platforms. The task of translating a natural language policy into a formal instruction is often difficult and the systems need to support this activity. To ensure the policy is translated into practice properly requires working closely with technical experts. File management systems come with access control schemata, while operating systems offer a file by file access control: this usually provides limited control options and is difficult to manage. However, software exists to provide centralised policy-based access control, which can provide greater flexibility such as being able to define and enforce embargos on files. The software available for access control includes CA’s SiteMinder, Sun’s Access Manager and IBM’s PolicyDirector. Typically, the centralised software requires a policy engine and software agents on each server to allow or deny access to files on the server. Each of the agents communicates with the policy engine to determine whether a particular access should be permitted or denied. The policy engine will provide a permit or deny response. For example, if a document is embargoed until a particular time, the policy server will respond with a “deny” for all access before the embargo time and a “permit” for access after the embargo has been lifted. Digital rights management (DRM) software can also be used for access control, but this pres107
Ensuring Security and Integrity of Data for Online Assessment
ents its own problems which include expense and accessibility. Furthermore, it is highly likely that some students will attempt and likely succeed in breaking the DRM schemata. Whatever type of access control software is used, the main issue for policy enforcement is to have adequate mechanisms. This includes mechanisms that are able to perform the required function and that do not require significant resources.
Logging and Audit Logging refers to recording of particular network and systems events: for example, when a user logs into a system, when a document is submitted, and the network identity of communications. Logging is necessary for the efficient operation and security of any IT system, since it allows problems to be identified and rectified. Audit is the process of examining and analysing the logs: without audit the logging has no purpose. Auditing can either be a manual process – an administrator reads through the logs looking for anomalies and exceptions, or it can be automated using specialised software to analyse the data in the logs. One of the tasks associated with logging is determining what should be logged. There are several approaches to deciding what to log. One is to log every possible event. However, this approach makes auditing more complex and has the potential to unnecessarily consume resources such as time and storage capacity. A more useful approach is to determine what information the system provides and how this information could be used. The requirements for logging will most likely differ between systems. For example usage logs for general web page access may only consist of counts of visitors, time, date and duration, while for pages relating to course materials, information about the user identification, IP address of the person accessing the page, together with information about what was done with the page (e.g., printed materials, followed links etc.) may be beneficial or necessary to log. This will need to be determined
108
in accordance with policy and users informed of what information is being logged. Processes need to be developed to determine who or what should audit the logs and who will have access to logs and under what circumstances. Clearly, staff should not have arbitrary access to logs due to privacy concerns and other abuses that may occur. Policy, legal and regulatory requirements will also shape the processes relating to storage and retention of logs, and who has responsibility for the ownership and security of the logs will need to be clear. For example, a department would not normally be responsible for network access for staff and students, and therefore would not have responsibility for obtaining, processing and storing logs. On the other hand, they are likely to need to have some access to logs for administrative purposes and may need to oversee submission systems. A recurring issue associated with logging is that it happens across various systems, with log information stored separately by each system. Often, analysis of more than one log is required, which sometimes is not straightforward or is at the least time consuming. As an example, consider a student who is disputing a penalty for late submission of assessment, claiming a network outage during the submission. The assessment submission system will have records of when the student accessed the system and when work was submitted from the student. The network logs will have records of when the student accessed the network and the requests that the student made. Both logs will need to be accessed and analysed to assess the student’s claim. If online assessment is used, it is necessary to have the means to investigate and substantiate students’ claims about submission, network outages etc. in a time efficient but thorough and accurate manner. Students quickly learn the shortcuts and loopholes and how to exploit them and online assessment can fall prey to this unless appropriate policies and procedures are in place and these are carefully adhered to.
Ensuring Security and Integrity of Data for Online Assessment
Onlii Assessment in Psychhit Deaa Ui Issues associated with authentication and verification of students, data integrity, and access control and monitoring were important in shaping how online assessment was incorporated into the two online units the first author was responsible for at Deakin University. As discussed earlier, the assessment was delivered via the LMS, so students used their Deakin username and password to access the assessment. This had the advantage of using an identification method with which students were already familiar. By delivering the assessment via the LMS we were also able to control when the assessment was delivered, how it was delivered and what students could do with the assessment. Online assessment in the first year psychology unit was formative and did not contribute to the final unit grade. The teaching staff wanted students to be able to attempt the assessment as often as they wished and the LMS assisted in meeting this goal. With a very large student cohort of around 1,000 enrolments across three on-campus (i.e., face-to-face) locations and off-campus (i.e., via distance education), the summative unit assessment comprised a MCQ format final exam and a written laboratory report. There were two aims in using online tests. The first was to familiarize students, formatively, with the type of questions they could expect in the final exam and to show them how their knowledge, understanding and application of the material they had covered could be assessed. The second was to provide students with the opportunity to self-assess their progress in the unit. The LMS assisted in achieving these aims in several ways. First, it provided a convenient means for uploading a test bank of questions selected by the teaching staff to reflect the unit curriculum. Second, using the testing tools built into the LMS, it was possible to set up a series of review tests
with a specific structure, such as a fixed number of questions from each topic, with the question stems randomised on each presentation occasion. Teaching staff developed good quality feedback with each question so that students were able to follow up questions they answered incorrectly. Using the LMS enabled the creation of a large number of review tests that students could complete in their own time, at their own pace and as often as they wished. Given that the primary motivation was to provide a review resource for students, on balance, issues associated with identification of the person taking the test were secondary to wanting students to benefit from opportunities for formative assessment (White & Duncan discuss a similar approach in a Pharmacy program in Chapter 5 of this volume). To encourage students to take advantage of the online assessment, questions that appeared on the final exam were included in the test bank and students were told that some of the questions in the online test would be on the final exam. If students chose to take the tests together this was not considered a problem, and since there were no marks attached to the tests, staff felt that there was no motive to get someone else to take the tests for you. The test bank was sufficiently large that even if a student wrote down every question from every version of the online exam and memorised the answers, they probably still learnt something. However, this was seen as a very time consuming and ineffective strategy. The second year unit, which had enrolments of approximately 300 students, extended the online assessment incorporated in the first year unit to include online testing for mastery of basic mathematics and two review tests that contributed 10 percent in total to the final grade for the unit. The purpose of the mastery test was to ensure that students had the basic mathematical skills necessary to manage the formulae they would encounter in the statistical sections of the unit. The mastery tests required short answers – mainly numeric responses – targeted to evaluate specific
109
Ensuring Security and Integrity of Data for Online Assessment
skills such as orders of operation, with students needing to obtain 75 percent to satisfy the mastery requirement. Students could take the test as often as they needed and were given detailed feedback on their weaknesses and how to address them. As was the case with the first year unit, the online review tests contained items that were also on the final, invigilated exam. The teaching staff for this unit wanted students to have an opportunity for formative assessment, but recognised that students needed checkpoints along the course of the semester to ensure they were on track to successfully complete the unit. Two tests contributing a total of 10 percent to the final grade were considered appropriate for several reasons. First, this percentage was not a large enough proportion of the final grade to warrant excessive invigilation and the level of security and authentication provided by the LMS was considered appropriate. However, 10 percent of the final grade was sufficient to make doing the assessment worthwhile if the motivation of checking your progress was not enough. A test bank of MCQ items was used and several review tests created so that students did not see the same questions on each test, but the tests were comparable with respect to what they assessed. Being delivered via the LMS prevented the tests from being printed. Furthermore there was a time limit for completing the assessment and students only had one attempt, making it impractical to copy down the questions and answers. The only circumstances under which students would be allowed to take the test again was if there was a documented and substantiated interruption to the test, however this was not an issue for any student during the course of the semester. Although the teaching staff recognised that some students could and would get someone else to do the test for them, they agreed that most students would not do this and would want to benefit from the assessment opportunity provided them. Access control and logging for both units were managed by the LMS through role-based permissions. The way this was implemented for
110
the second year unit was that two senior teaching staff had permissions that allowed them to create and modify assessments, as well as view and edit student marks. Other teaching staff could view the tests and student results for the tests. Students could take the test once and could view their own test results. In addition to being able to record student marks, it was also possible for the senior staff to analyse student responses across questions for each topic to determine whether there were problem areas that needed to be addressed by the teaching staff. Furthermore, staff were able to track whether students had completed the online assessment and send them reminder emails if they hadn’t. This additional information helped staff monitor student progress and assisted them in keeping on track with assessment in the unit. Neither of the units had online summative assessment that contributed a significant proportion of the final grade for the unit. The reasons for this included: •
•
•
Concerns about how to invigilate the exams, acknowledging that the greater the proportion of the final grade the online assessment counted for, the greater the need for ensuring accurate identification of the test taker, and managing security of the examination material and integrity of recorded responses; Lacking institutional support for large scale online testing, such that computer laboratories could not be used during the advertised university examination period for the purposes of invigilated exams. This meant that if the teaching staff wanted to conduct online assessment for the unit, they had to schedule time at the end of semester to do so and manage all the organisation associated with conducting the exams; At a local level, students and other staff were not convinced that the benefits of using online assessment were worth the potential problems that could arise, such as technical failures.
Ensuring Security and Integrity of Data for Online Assessment
Although the teaching teams would have liked to extend the amount of CAA that was included as summative assessment, at the time the obstacles were such that it was not considered viable. However, there was a level of optimism that in the future it would become easier to conduct online assessments and that there would be institutional support for using online tests for summative assessment.
Solutii Recommendations What Technology Can Offer So far we have discussed the four ways in which technology can be used to facilitate online assessment and presented case studies from units using online assessment. It should be clear that there is no technological “silver bullet” for addressing the challenges associated with online assessment. In particular, adherence to and enforcement of appropriate policies and procedures is a critical partner to the implementation of any technology system. On the surface it seems that online assessment creates a new set of problems for assessors already concerned about plagiarism and cheating (see Markham & Hurst, Chapter 1 in this volume). However, we would argue that many of the issues associated with online learning are also present with other traditional forms of assessment. Furthermore, there are several aspects to online assessment that offer advantages over and above traditional assessment tasks, and these advantages make this form of assessment very attractive. These can be broadly classified as either enhancements to assessment quality or administrative/technical efficiencies or benefits, although there are examples where the two interact to provide benefits not easily achieved without the use of technology. If online assessment is implemented properly, the educational benefits can potentially outweigh the disadvantages.
When deciding whether to use online assessment, it is important to remember why assessment is used and to match learning outcomes to appropriate online assessment methods, being cognisant of learner characteristics. Any assessment needs to guide and encourage effective approaches to learning and to measure expected learning outcomes reliably and validly. The assessment and how it is graded also needs to be consistent with and reflective of the required academic standards. (James et al., 2002). Ideally, online assessment should be designed to exploit the technological capabilities in a way that helps to achieve the desired goals. It also needs to be implemented efficiently and effectively so that any educational benefits are not lost due to technical or administrative failures. The strategies and checklists provided by James et al. (2002) are a useful resource for those wanting to use online assessment as they provide prompts to help ensure that access and usage, quality of teaching and learning, and technical and administrative concerns are considered when planning online assessment. Technology also offers the opportunity to provide detailed and personalized feedback to the person taking the test. This can include feedback on strengths, weaknesses and knowledge gaps, as well as recommendations about how to address problem areas. When used as a formative assessment tool, this type of online assessment can be valuable in motivating and directing student learning and encouraging learner independence (Cann, 2005). As discussed previously, the security and data integrity requirements for formative online assessment of this nature would be less stringent than for summative assessment which contributes to the end of course grade – while the need for data integrity still exists, there may be less concern about authentication and verification of the user taking the assessment as the formative assessment is mainly for the user’s benefit. However, there are other uses for formative assessment where accurate identification of the user is important. For
111
Ensuring Security and Integrity of Data for Online Assessment
example, Challis (2005) discusses the advantages of adaptive testing techniques whereby the learner undertakes a testing procedure that sharpens areas of weakness as identified via the pattern of correct responses to questions in the test bank. One of the advantages of adaptive testing is that it decreases the amount of time it takes to complete the assessment. As a diagnostic assessment tool, adaptive testing is very valuable as the results can be used to tailor learning activities suitable for each student. Therefore, it is important that the learner, and not their friend who has already done the course, does the adaptive testing so that the learning activities are appropriate. Since it seems counterproductive for a learner to collude in this assessment situation, it may be sufficient to require a username and password for identification in this case. In the case of an online exam for summative purposes however, this may not be acceptable and students may be required to attend on-campus to take the online assessment under invigilation. There is a continuum of informal to formal assessment situations which vary with respect to requirements for authentication/verification, data integrity and security. For informal assessment there could be limited or no need for verification of the identity of the user, although there may still be security and data integrity requirements. At the other end of the spectrum for formal assessment, authentication and verification are essential, with data integrity and security of high importance. Issues of access control will also likely differ and need to be considered and dealt with. For example consider a formal assessment via an online exam, where students can only have one attempt and must complete the assessment in a fixed timeframe; we would anticipate that the assessment administrator will want to authenticate and verify the identity of the person taking the test. A further requirement is that the assessment administrator does not want the assessment copied, printed or otherwise distributed and they also don’t want other programs to be accessed during
112
the test and so need to disable access to other applications. After the exam has been taken, the assessment results need to be accessed by other academic staff for the purposes of marking and quality control. These other staff need to be able to view assessment materials and data, but should not be able to modify them. In contrast, consider a formal assessment in the form of a task (e.g., essay) that is completed over a period of time. Again the assessment administrator will want to authenticate and verify the identity of the person submitting the material. The student will most likely have a number of versions of the work, and the assessor may want to track different versions. They may also want to use plagiarism detection software on the essay’s content, and some sharing of submitted data will be required for marking purposes. These examples illustrate the importance of understanding the purpose of the assessment and the implications this has for the level of security and data integrity required. Just as assessors need to make judgements when using traditional assessment methods, online assessment requires consideration of the purpose the assessment is being used and the implications for administering the assessment that flow from this. Technology offers a range of tools that collect and manage information. Thus, teaching staff can potentially collect and analyse more data in order to make more accurate assessments of students. One application for this is group work, which can be a source of student disputes and angst—Benson in Chapter 7 in this volume and Tucker et al Chapter 10 in this volume include detailed discussions of these matters. Often a member of the group will put very little or no effort into the assessment, expecting the other members of the group to make up the deficit. The ability to log student activity and submission will allow staff to assess the validity of the claims of what is often called “social loafing”. This can be useful, particularly in the case where there are disputes amongst students working in groups on a formal
Ensuring Security and Integrity of Data for Online Assessment
assessment task that counts towards their final grades. However, usage logs are only one part of the story, and patterns of engagement discerned from these logs need to be considered in context of other possibilities, such as the student didn’t contribute online but did contribute in other ways not reflected in the logs. Usage logs also provide a non-repudiation mechanism which is useful in a variety of contexts and particularly important in the case of assessment. Audits often reveal a range of information and can be used to confirm who did what, and when they did it. This can be very reassuring for users. For example, it provides confidence that what actually happened can be reconstructed from the logs (e.g., claims of problems logging onto the system or a system crash can be verified by the log data). It can also work against them if claims cannot be substantiated through the data logged by the system. However, in order to be able to make use of log information for these purposes, there needs to be a mechanism for obtaining the information in a timely manner. This needs to be managed through appropriate policies and procedures that are implemented and to which stakeholders adhere. Security and integrity of data, particularly for assessment purposes, is an issue that needs to be attended to at all levels. At an institutional level, policies and procedures should be formulated to assist the organisation to meet its mission, objectives and legal requirements. IT infrastructure needs to be put in place and managed in accordance with these policies and procedures. In addition, staff need to be made aware of policies and procedures and ensure they are followed. It is also important that policies are effectively structured and organised, and aligned with the institutional culture (von Solms & von Solms, 2004a). People need to share and have ownership of policies and procedures in order for them to be effective, as noncompliance is a significant threat to security and integrity in assessment procedures. The deadly sins of information security management include a
lack of recognition that security is a business issue rather than a technical one, underestimating the complexity of information security governance, failing to base security on identified risks, and failing to have proper policies and compliance enforcement and monitoring. Related to this are the “sins” of not realizing the importance of information security awareness amongst users and not empowering people in the organisation with appropriate tools to ensure policy adherence (von Solms & von Solms, 2004b). Policies and procedures however well formulated and enacted, are no guarantee that online assessment will be efficient, effective and safe or valid and reliable. It seems that, in spite of everyone’s best efforts, there are always a small number wanting to thwart the system. The suggestions for assessing risk made by Kritzinger and von Solms (2006) can be used at both an institutional and local level. These include conducting a proper risk analysis for all IT and business-related risks involved in using a system and determining the likelihood and potential impact of each risk. Once the risks are understood, countermeasures to manage risk can be put in place. At the institutional level identified risks may include DoS attacks, interception of data or unauthorised access (Weippl, 2005). At the local level, the academic or instructor using online assessment may be concerned with whether students or others can copy or otherwise distribute assessment materials, who should have access to assessment data and what should they be allowed to do with it, and how likely is it that students will collude, cheat or plagiarise and to what extent this is problematic given the nature and purpose of the assessment. There is interdependency between the dimensions of policy, measuring and monitoring, awareness and organisational governance (von Solms, 2001); for example the technical dimension of information security consists of such things as firewalls, which need to be customised to the organisation according to their policy requirements. The measuring and monitoring
113
Ensuring Security and Integrity of Data for Online Assessment
dimension is a means of determining if the policy is being carried out, but this will only happen if the awareness dimension is attended to through proper dissemination of the policy to members of the organisation. The final dimension is one of governance, whereby issues or incidents can be handled and gaps in policy, procedure or both can be addressed. If individuals decide to use online assessment it is important that the institutional policies and infrastructure support them in their activities. It is critical that there are also mechanisms for updating policies and procedures to deal with situations or circumstances arising from the use of online assessment that are not already covered. Upper management needs to listen to, acknowledge and respond to feedback from those involved in the online assessment process, including students and teaching and technical staff, who in turn need to communicate constructively and co-operatively with others at all levels of the organisation.
FutureTrends Sim et al., (2004b) make the point that institutional support is needed for successful implementation of online assessment. In our experience in Australian universities there seems to be good support for CAA for formative assessment, with institutions using a LMS routinely providing staff with access to software applications that support multiple choice testing and which work with test banks such as those provided by a text book publishers. However, there does not seem to be any move towards widespread adoption of online assessment for summative assessment through invigilated exams. While using online testing for the purposes of formative assessment has much to recommend it, many rightly point out that summative assessment is what students’ grades are based on. At this point in time large scale examination of students online under invigilated conditions seems difficult to envisage
114
and impractical. A number of factors contribute to this, but most significant is having to provide sufficient, securely networked computers in one place to cater for the thousands of students who typically sit exams during university examination periods. The current model for conducting exams, where venues are converted into examination centres for a specified period and casual staff are employed as exam officials, does not lend itself well to online examination. Just the setting up of the computers and associated networks would be very time consuming. However, there is also an additional staffing overhead that comes with online examination. Technical staff need to be available at all times to respond to any issues that arise and invigilators would require training to ensure they are familiar with online testing conditions and how to manage incidents that will inevitably arise. Technology tends to evolve rapidly and on the horizon are a number of developments that could make online assessment in an examination centre feasible. These include “network in a box” which can be used in remote areas to establish phone and internet access quickly and efficiently. As this technology matures and becomes widely available, it could be used to provide a dedicated, secure network for an exam period in a designated venue. Firewalls could be put in place to block access to unauthorised sources, such as mobile phone signals, essentially locking down the network to the examination room. While this deals with the issue of being able to easily set up a network, it doesn’t easily address the provision of computers. However, a network in a box could support a thin client model, where students only require a monitor and keyboard and the exam material is delivered from the network. Alternately, students could bring their own laptops and secure browsers used to deliver examination material while locking them out of applications other than those approved for the exam. Personal computing is also evolving, with devices becoming more powerful, portable and
Ensuring Security and Integrity of Data for Online Assessment
usable. While the focus is currently on handheld devices with keypad, touch or stylus data entry, alternate forms are being developed. The most notable of these is electronic paper, which is being hailed as the second paper revolution (Genuth, 2007). E-papers of the future will be inexpensive, thin and flexible, able to be rolled up and put in your pocket, while holding incredibly large amounts of data. By 2020, students could be able to attend a supervised examination centre where the exam is displayed on electronic paper and students complete the exam using a digital pen. Information stored on the digital pen would serve to record students’ responses, as well as providing a means of biometric authentication, either by handwriting or by other means. Invigilators would authenticate students, possibly using biometrics such as iris scans. The room itself would have a range of technologies managing data access and exchange to those sources permitted for the exam. Devices that students might think of using to gain an unfair advantage in the exam, such as mobile phones, could be quarantined, but secure browsers would allow access to information and applications. Under these conditions, many of the current security issues are removed – authentication methods will ensure that the person sitting the exam is the person who should be, while technology is used to manage access to authorized exam materials. Students would not be able to remove the exam paper from the venue – even if they took the e-paper the exam would be deleted automatically once the student left the examination room. Of course, even with these measures in place, some students will come up with ways of cheating which will need to be addressed.
CONCLUSION For the average teacher trying to create effective teaching and learning experiences for their students, issues relating to policy, procedures
and infrastructure to support online assessment may seem somewhat removed or beyond their control. Hopefully we have raised awareness that at all levels of an educational institution there are issues involved in ensuring security and integrity of data in online assessment. At the same time we hope we have provided some guidance on how technology can be beneficial and pointed the way to future developments that could revolutionise online assessment. As “front-line troops”, teaching staff or instructors are the key people who deal with the consequences of inadequate attention to what is needed to implement and deliver effective online learning environments. The importance students place on assessment underscores the need to ensure that appropriate systems and structures are in place so that assessment materials and data delivered online are protected. It is likely that in the future technology will be able to assist in addressing many of the security and integrity issues we have discussed. However, socially engineered threats will continue to be an issue. Rather than dismissing online assessment as being too difficult or too problematic, we hope that we have provided an overview of some of the issues that need to be attended to in order to realise the benefits that online assessment can provide to both teachers and students.
REFERENCES Armatas, C., Holt, D., & Rice, M. (2003). Impacts of an online-supported resource-based learning environment: Does one size fit all? Distance Education, 24(2), 141-158. Armatas, C., Holt, D., & Rice, M. (2004). From online-enhanced to wholly online: reflections on e-learning developments in psychology in higher education. In R. Atkinson, C. McBeath, D. JonasDwyer, & R. Phillips (Eds.). Beyond the Comfort Zone: Proceedings of the 21st ASCILITE Conference (pp. 78-87), Perth, 5-8 December.
115
Ensuring Security and Integrity of Data for Online Assessment
Australian Government. (1988). Financial Transaction Reports Act. Retrieved April 10, 2008 from http://www.austrac.gov.au/rg_7.html Cann, A. J. (2005). Extended matching sets questions for online numeracy assessments: A case study. Assessment & Evaluation in Higher Education, 30(6), 633-640. Challis, D. (2005). Committing to quality learning through adaptive online assessment. Assessment & Evaluation in Higher Education, 30(6), 519-527. Engelbrecht, J., & Harding, A. (2004). Combining online and paper assessment in a web-based course in undergraduate mathematics. Journal of Computers in Mathematics and Science Teaching, 23(3), 217-231. Genuth, I. (2007). The future of electronic paper. Retrieved September 8, 2008, from http://thefutureofthings.com/articles/1000/the-future-ofelectronic-paper.html Graf, F. (2002). Providing security for e-learning. Computers and Graphics, 26, 355-365. James, R., McInnis, C., & Devlin, M. (2002). Assessing learning in Australian universities. Canberra, ACT: Centre for the Study of Higher Education, The University of Melbourne & The Australian Universities Teaching Committee. Kambourakis, G., Kontoni, D. N., Rouskas, A. & Gritzalis, S. (2007). A PKI approach for deploying modern secure distributed e-learning and m-learning environments. Computers and Education, 48, 1-16. Kritzinger, E., & von Solms, S. H. (2006). Elearning: Incorporating information security governance. Issues in Informing Science and Information Technology, 3, 319-325. Nicol, D. (2007a). Laying a foundation for lifelong learning: Case studies of e-assessment in large
116
1st-year classes. British Journal of Educational Technology, 38(4), 668-678. Nicol, D. (2007b). E-assessment by design: Using multiple-choice tests to good effect. Journal of Further and Higher Education, 31(1), 53-64. Raitman, R., Ngo, L., Augar, N., & Zhou, W. (2005). Security in the online e-learning environment. In P. Goodyear, D. G. Sampson, D. J. Yang, T. Kinshuk, R. Okamoto, Hartley & N. Chen, (Eds.), The proceedings of the 5th IEEE International Conference on Advanced Learning Technologies (ICALT’05) (pp. 702-706). IEEE Computer Society: United States. Sim, G., Holifield, P., & Brown, M. (2004). Implementation of computer assisted assessment: lessons from the literature. Association for Learning Technology Journal, Research in Learning Technology, 12(3), 215-229. von Solms, B. (2001). Information security – a multidimensional discipline. Computers & Society, 20, 504-508. von Solms, B., & Marais, E. (2004). From secure wired networks to secure wireless networks – what are the extra risks? Computers & Security, 23, 633-637. von Solms, B., & von Solms, R. (2004a). The 10 deadly signs of information security management. Computers & Security, 23, 371-376. von Solms, B., & von Solms, R. (2004b). From policies to culture. Computers & Security, 23, 275-279. Weippl, E. R. (2005). In-depth tutorials: Security in e-learning. eLearn Magazine, 3(3). Retrieved from http://www.elearnmag.org/subpage.cfm?se ction=tutorials&article=19-1 Weippl, E. R. (2007). Dependability in e-assessment. International Journal on E-Learning, 6(2), 293-302.
117
Chapter VII
Issues in Peer Assessment and E-Learning Robyn Benson Monash University, Australia
ABSTRACT This chapter addresses some issues relating to the use of e-learning tools and environments for implementing peer assessment. It aims to weigh up the opportunities and the challenges that are offered by considering peer assessment for learning and peer assessment of learning. In doing this, reference is made to some of the general issues that arise in implementing peer assessment in higher education, as well as to the functionalities of e-learning tools and environments, and the characteristics of those who use them in this context (teachers and students). Discussion of opportunities focuses on strategies for peer assessment available from tools and environments that are categorized as pre-Web 2.0 (and continuing) technologies, Web 2.0 technologies, and ‘other tools’. Consideration of challenges focuses on the characteristics and requirements of teachers and students as users. It is concluded that opportunities outweigh challenges, particularly in relation to peer assessment for learning, but that peer assessment of learning is more challenging and likely to be more limited in uptake because of the expectations that are placed on users. It is also noted that the capacities offered by Web 2.0 technologies for peer-based relationships and interaction with content present both an opportunity and a challenge which may have future implications for the role of the teacher and for supporting a reconceptualization of how evidence used for peer assessment of learning is presented and judged.
INTRODUCTION The use of peer assessment in e-learning environments raises a number of issues, not all of them
related to reliability and validity. Unpacking these issues requires consideration of the nature and purposes of assessment, examination of the nature and functionalities of particular e-learn-
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Issues in Peer Assessment and E-Learning
ing tools and environments, and consideration of factors affecting the teachers and students who use them. In relation to the nature and purposes of assessment, the statement by Rowntree (1977) that assessment is an interaction which is aimed, to some extent, at knowing another person, foreshadowed a view that has become influential in recent years, suggesting that assessment is about more than the measurement of performance, which in turn raises issues beyond those associated with reliability and validity. Serafini (2004) identifies three paradigms of assessment: assessment as measurement, followed historically by assessment as procedure, and thirdly, assessment as inquiry. It is in relation to the last of these that students may become involved in taking an active role in assessing their own learning and that of their peers in a wide variety of ways. This may also require a change in teachers’ perceptions about their own roles as they collect information about students’ learning in order to inform subsequent teaching and learning activities. Students’ involvement in this process is frequently related to the formative function of assessment in improving learning, rather than to its summative function in establishing the quality of learning by making judgments based on the application and verification of standards. However, students may also be involved in summative assessment of the performance of their peers. The chapter addresses these functions in relation to peer assessment and e-learning in order to consider some of the associated opportunities and challenges, and their implications for assessment practice, including their implications for reliability and validity. This requires exploration of the functionalities of different kinds of e-learning tools and environments and of some of the factors impacting on their use by teachers and students. In this chapter, the implementation of peer assessment for e-learning includes use of the networked environment as a medium for undertaking assessment tasks and/or for peer communication about assessment, as well as the use of electronic tools
118
which may or may not make use of a networked environment, where peer communication does not necessarily occur electronically. In particular, it will focus on whether the opportunities outweigh the challenges for one or both of these functions of assessment. The chapter concludes with an assessment of the value of e-learning tools and environments in relation to peer assessment for learning and of learning, commenting on how this compares to peer assessment in a face-to-face environment and noting the potential implications that Web 2.0 technologies offer for supporting a reconceptualization of how evidence for assessment is presented and judged.
Ba Peer Assessment Roberts (2006, p.6) defines peer assessment as ‘the process of having the learners critically reflect upon, and perhaps suggest grades for, the learning of their peers’, distinguishing it from group assessment in that ‘students assess each other’s learning, even though the learning may have occurred individually, or at least outside of any formal collaborative groups.’ However, this does not exclude assessment by or of students working in groups. The above definition accommodates the use of peer assessment for learning (by the peer assessor through critical reflection, by the peer who is assessed through the feedback provided, and by both through the communication involved) and assessment of learning (through its use in grading). Many teachers may encourage the former use of peer assessment more readily than the latter because of concerns about reliability, validity and practicality when peer assessment is used summatively. In doing this they would also be reflecting recent views about assessment which focus on its value for learning (e.g., Boud, 1995a, 2007; Carless, 2007), involving the idea of
Issues in Peer Assessment and E-Learning
assessment as inquiry, rather than being ‘primarily about the allocation of rewards and punishments’ (Ramsden, 2003, p.180). However, strategies for the summative use of peer assessment are not new (e.g., Boud, 1995b; Nightingale et al, 1996). Falchikov (2005), in a major contribution to improving assessment through student involvement, addresses both the formative and summative functions of peer assessment. In relation to the former, she discusses the value of peer assessment in terms of learning together. In discussing summative assessment, she draws on previous contributions in this area to consider some of the common problems associated with peer assessment and to suggest some strategies for overcoming them. For example, she suggests that problems associated with student reluctance to participate or student discomfort during and after participation can be addressed by preparation and training. Concerns about peers deferring to the teacher can be dealt with by structuring activities to prevent this occurring. Bias from friendship effects can be reduced by requiring students to justify ratings with the possibility of scaling marks if the overall standard is high compared to the teacher’s grade. She suggests that bias related to gender, age, ability or ethnicity can be approached by discussing potential sources of bias openly and making expectations and practices explicit. Requiring students to justify ratings and explicitly linking criteria with ratings can be used to address a number of these problems, including potential collusion between students. She notes as a central issue the need for a standard against which to judge students’ marking (which is usually the teacher’s grade) and questions whether, by comparing student and teacher marks, we are investigating reliability or validity – leaving it up to individuals to decide on this issue. However, she comments that ‘[g]iven the lack of reliability of much teacher assessment, it is surprising that many formal comparisons indicate close resemblance between teacher and student marking’ (p.251). Based on a meta-analysis of peer assess-
ment studies she recommends that it is better to use an overall global mark with well-understood criteria (and to involve students in discussions about criteria) than to expect student assessors to rate many individual dimensions. Efforts have been made by some staff to help students prepare for peer assessment and manage the application of assessment criteria (e.g., Beaman, 1998; Bloxham & West, 2004). When students are involved in the assessment of group work, additional guidelines may be needed to address the frequently-raised concern of assessing individual contributions fairly (James, McInnis & Devlin, 2002). Falchikov (2005) draws on the work of Lejk, Wyvill and Farrow (1996) to discuss eight commonly used strategies for differentiating group and individual marks: multiplying the group mark by an individual weighting factor; distribution of a pool of marks; use of a contribution mark; separation of product and process; equally sharing a mark but with exceptional tutor intervention; splitting group tasks and individual tasks; use of yellow (warning) and red (zero grade) cards; and calculation of deviation from the norm. Engagement of students in discussion about the selected strategies provides an avenue to enhance peer assessment of group work with potential benefits for reliability and validity. Other issues to consider in relation to the implementation of peer assessment include its value in preparing students for the kinds of highly contextualized learning that are part of life and work. This is addressed by Boud and Falchikov (2006) who note the socially constructed, highly situated nature of learning in work and life settings which frequently involves working collaboratively with others. There are clear implications for peer assessment in these contexts, particularly in its support for ‘real world’ learning through authentic assessment, the value of which is currently emphasized in higher education (Falchikov, 2005). However, the contextual nature of assessment in these settings presents challenges for validity and reliability since the criteria against
119
Issues in Peer Assessment and E-Learning
which judgments are made must be determined in each situation. Knight (2006) explores this point, commenting on the essentially local nature of assessment and how statements of general achievement based on local practices should not be treated as valid and reliable. Knight and Yorke (2004) suggest that sensibly promoting employability through assessment and helping students to represent their achievements to employers can only occur by adopting differentiated approaches to assessment through approaches that are not all high stakes, and not all direct, and by planning assessment across programs. The above factors are indicative of the numerous issues to be considered when implementing peer assessment, whether or not it is undertaken in relation to e-learning. However, the use of elearning tools and environments adds a further range of factors to be addressed.
Functionalities of E -Learning Tools and Environments For convenience, in this chapter the capacities of e-learning tools and environments are divided into pre-Web 2.0 (and continuing) technologies, Web 2.0 technologies, and ‘other tools’. Web 2.0 technologies or ‘social software’, including wikis, blogs, social bookmarking, social networking services (such as MySpace and Facebook), and access to virtual worlds, enable unprecedented online interactivity between users, and between users and content. Definitions of Web 2.0 are still evolving, but the focus is on software that supports group interaction (Shirky, 2003) as opposed to individual activity, with a key emphasis on democratic engagement. This draws on the concept of the wisdom of crowds (Surowiecki, 2004) which suggests that the collective wisdom of a group can exceed that of its individual members. Dron (2007, p.233) notes that ‘one of the most distinctive features of social software is that control and structure can arise through a process of communication, not as a result of design, but as
120
an emergent feature of group interaction.’ Wikipedia (http://en.wikipedia.org/wiki/) is probably one of the most widely recognized examples of this process, although contributions are subject to editorial control as explained on the website. Definitions of Web 1.0 depend on the definition of Web 2.0, with the business-related origins of the latter term used to identify new applications of the web which were emerging following the dot-com collapse in 2001, in contrast to the static, non-interactive websites and proprietary rather than open source applications associated with Web 1.0 (O’Reilly, 2005). Strickland (2008) states that ‘if Web 2.0 is a collection of approaches that are the most effective on the World Wide Web, then Web 1.0 includes everything else.’ In relation to peer assessment, the ability of the online environment to facilitate communication is a key attribute. This includes computer mediated communication which occurred both during and before Web 1.0 developments, with Web 2.0 functionality being the most recent manifestation of this evolution (Allen, 2004). These pre-Web 2.0 social tools include email, chat and asynchronous online discussion. Learning management systems also emerged during the Web 1.0 era, accommodating these communication tools and additional tools that can be used for peer assessment. Because many of the tools that are useful for peer assessment are not those most readily associated with Web 1.0 (which assumes a primarily read-only capacity), they are considered in this chapter as pre-Web 2.0 (and continuing) technologies to distinguish them from the particular developments associated with Web 2.0. The ‘other tools’ considered in this chapter include further possibilities which are available as standalone electronic options (though they may be incorporated into first or second generation web-based environments), ranging from use of Word, Adobe Acrobat and Excel files, to audio and video files which may be available on CDROM or DVD.
Issues in Peer Assessment and E-Learning
In considering the functionalities of the above range of tools for peer assessment, attention needs to be given to infrastructure, accessibility and security issues, and issues related to authentication, as these have particular implications for summative assessment. Opportunities and challenges may also be considered in terms of support for authentic assessment, particularly in the context of constructivist perspectives which have underpinned much of the pedagogical theorizing about e-learning. This relates to the social aspects of e-learning, drawing on the work of Vygotsky (1978), given the capacity for authenticity which is offered by online, asynchronous environments (Mathur & Murray, 2006), and for conceptualizing the situated nature of individual learning (e.g., Jonassen, Howland, Moore, & Marra, 2003) which suggests a need to design for cognitive flexibility and contextualized assessment to allow students to deal with ‘the real world complexity and ill-structuredness of many knowledge domains’ (Spiro, Feltovich, Jacobson, & Coulson, 1991). The growth of problem-based learning in higher education (Boud & Feletti, 1997) offers an effective means of combining learning and assessment because it requires the learner to grapple with a real task in context. It is therefore a useful approach which is reflected in the development of electronically-based real world environments, including games, simulations and virtual worlds, with the most recent of these belonging to the Web 2.0 era.
Tachers and Students as Users In e-learning environments, the characteristics of teachers and students as users have the potential to assume as much or more importance for peer assessment than the functionalities of the tools themselves. Hence, staff development issues need to be addressed, as do factors affecting students, including orientation, support and access. If teaching staff are not comfortable in using learning technologies, they are unlikely to
be proactive about peer assessment and e-learning, particularly where high stakes summative assessment is involved. In their review of strategies to address differences in uptake of new technologies by teaching staff, Wilson and Stacey (2004) note how much of the literature relating to the professional development of staff for online teaching has been dominated by Rogers’ (2003) diffusion of innovation theory which classifies individuals’ adoption rates into five categories (innovators, early adopters, early majority, later majority and laggards). Using this perspective, the readiness of staff to introduce peer assessment arrangements for their students may be seen in terms of factors such as their position on this continuum, the availability of staff development opportunities in relation to teaching online, and their openness to using them. The characteristics of students as users may add further factors that affect the implementation of peer assessment for e-learning. Prensky’s (2001) distinction between ‘digital natives’ who have grown up in the digital age and older ‘digital immigrants’ who adapt less readily to digital technologies suggests that a digital divide between ‘net generation’ students and staff who are digital immigrants may be a major issue that influences the use of peer assessment for e-learning. As ‘digital natives’, Generation X and Y learners and Millennials (Hoerr, 2007; McCrindle, 2006; Oblinger, 2003), who have been familiar with networked environments from an early age and are accustomed to using them, may adapt more readily to peer assessment online than the ‘digital immigrants’ who preceded them. Their digital connectedness potentially offers opportunities for a range of innovative approaches which support learning (Oliver & Goerke, 2007). As Barnes and Tynan (2007, p.198) note, ‘[w]hereas universities are still struggling to assimilate the first wave of Web-based teaching tools, an increasing number of students have already arrived in a Web 2.0 world.’ However, there is evidence emerging that this oversimplifies students’ uptake of technology
121
Issues in Peer Assessment and E-Learning
and the readiness of many of them to use it for learning purposes (Bennett, Maton & Kervin, 2008; Kennedy, Judd, Churchward, Gray, & Krause, 2008), suggesting that there is a need for caution in making assumptions about the predisposition of learners to engage with digital technologies for activities associated with peer assessment. Hence, the characteristics of both teaching staff and students are likely to have a major impact on the implementation of learning technologies for peer assessment. Another factor relating to teachers and students as users which can affect the potential for peer assessment and e-learning is the workload that may be involved, including the administrative workload. Although some technologies associated with e-learning reduce the administrative workload, there are also online peer assessment activities (such as managing the assessment of online discussion groups) which may increase it. There are further potential implications for support staff (including technical staff).
Opprtunities of e -Learning for Peer Assessment One way of viewing the central benefit that elearning tools and environments offer for peer assessment is to conceptualize it in terms of improvements in communication between learners and/or between teacher and learners. This includes: •
122
The quality of what is communicated (e.g., authentic forms of assessment responses available in a recorded form including date and time of submission or posting which can assist in establishing validity, or automatically calculated assessment results based on judgments made against specified criteria which may also calculate inter-rater reliability);
•
•
Speed of communication (involving nearinstantaneous transfer of information which may include automated processes); and Flexibility of communication (including time, place and pace for both the assessors and those being assessed) which supports the development of collaborative processes. Flexibility in relation to time includes the benefits of both synchronous and asynchronous communication, with the latter providing an opportunity to facilitate reflection, which as well as providing advantages for students who are being assessed, may also be seen as helpful for self assessment or assessment of peers (Topping, Smith, Swanson & Elliott, 2000).
Underpinning the communication benefits are the advantages which technologies may offer for supporting approaches to learning and assessment which are currently seen as valuable, and for efficient and effective management of some of the administrative tasks involved. Most students who undertake assessment tasks in e-learning environments are asked to use the environment in one or more of the followings ways: •
•
• •
For submission of assessment items (e.g., essays, reports, reviews, media-based presentations); For automated assessment where items are automatically delivered to students, frequently with scoring and feedback automatically offered (e.g., quizzes with a range of question types such as multiple choice, short answer, matching or calculated, or multimedia items using features such as drag and drop, matching and simulations); For assessable online discussion (e.g., forums, debates, role plays); and For web ‘publishing’ of assessment items (e.g., html pages, PowerPoint presentations,
Issues in Peer Assessment and E-Learning
blogs, wikis, podcasts, e-portfolios) (Benson & Brack, 2007). It is aspects of the last two of these which may be seen as lending themselves most readily to peer assessment because of the opportunities for collaborative engagement which are intrinsic to the functionality of the tools. These can be used both for undertaking assessment tasks as well as engaging in peer (individual or group) assessment. Online submission capabilities can also be used to facilitate peer assessment of tasks which may be undertaken individually or in teams, particularly by speeding up assignment submission and return, although within learning management systems grading and feedback are under the control of the teacher. Automated assessment is usually undertaken individually by students and has benefits for both self assessment and teacher assessment. However, automated assessment systems which are specifically designed to support peer assessment by assessors through an interface that allows them to enter feedback and/or grades for automatic compilation by the system, probably offer the best opportunities currently available for reliable and valid assessment of learning by peers. Opportunities for peer assessment associated with the above uses of the e-learning environment are outlined below in relation to pre-Web 2.0 technologies, Web.2.0 technologies and other tools. There is inevitably some blurring across these categories, particularly where pre-Web 2.0 developments incorporate ‘other tools’, while Web 2.0 may incorporate aspects of both.
Pre-W eb 2.0 (and Cntinuing) Tchnologies The most ubiquitous form of online, asynchronous communication which can be used as a tool for peer assessment is that of email. Email pre-dates Web 1.0 as well as Web 2.0 and continues to be central to electronic communication. With ability to sup-
port peer-to-peer or peer-to-group communication and to convey information via attachments or links to websites, as well as within the body of the message, email can be used to facilitate presentation of assessment items, and assessment of them. However, when the content of online messages forms the material to be assessed, this is more likely to occur through threaded discussion groups (readily available using the tools in learning management systems, which also offer the ability to divide students into groups). The availability of a record of individual or socially constructed knowledge offers clear potential benefits for assessment, including peer assessment. While these benefits are probably utilized most frequently for formative assessment, providing for a balance of prompt but reflective feedback when students comment on each others’ contributions, their use for summative assessment has been reflected in the development of rubrics to assess online communication (e.g., Baron & Keller, 2003; Edelstein & Edwards, 2002), which may be used for self, peer, group or teacher assessment. The potential for authentic experiences such as debates, case analyses and anonymous role plays provides further benefits which may also be used in the context of peer assessment. Online discussion groups also play a valuable role in supporting the increasing emphasis which is currently being placed on communities of practice as a means by which groups of people deepen their knowledge and expertise by interacting on an ongoing basis (Wenger, McDermott & Snyder, 2002). Extending this notion, Garrison and Anderson (2003) define a community of inquiry in an educational setting as a critical community of teachers and students who engage in transactions that will further their learning, simultaneously encouraging both cognitive independence and social interdependence. These concepts suggest clear opportunities for underpinning collaborative peer assessment while the technology offers a means of managing it. McConnell (2006) describes a model developed over a number of years
123
Issues in Peer Assessment and E-Learning
as an approach to validly assessing collaborative learning in e-learning contexts which can be used for self, peer and tutor assessment of collaborative learning. It has four components (product achievement, communication skill, social relationships and reflective skill) with indicators of achievement in each area. For example, indicators for product achievement include contributing to project ideas, building on comments and on help received from others, helping to produce the report, essay or other product, and meeting deadlines. Communication skill indicators include initiating dialogue and discussion, seeking information from and giving information to others in the group. Social relationship indicators include being sympathetic, encouraging members of the group, showing interest in the members of the group and praising others, while reflective skill indicators include analyzing the group’s behaviour, noting reaction to comments, learning about oneself and learning about others. Models such as this are valuable in establishing criteria for assessing collaborative work in e-learning environments (by students or others), which contribute to the valid and reliable assessment of individual and group contributions and processes, as well as the product itself. While the teacher-centred nature of learning management systems (Dron, 2007) and their management focus (Dalsgaard, 2006) inhibit their ability to lend themselves easily to peer assessment, some of the other tools within them can be used or adapted for this purpose. For example, students as well as the lecturer may be allowed to publish submissions, or the quiz (survey) tool may be adapted so that students use it as a marking or polling form with pre-designed closed response questions provided for grading and open ended questions for offering comments on published pieces of work. Mann (2006) describes a ‘post and vote’ system of web-based peer assessment using a combination of tools available in learning management systems (the discussion tool, the submission tool, the survey tool, and the mail tool) which was found to be valid. Polling
124
or voting by students may also be undertaken in lecture theatres and facilitated by the use of audience response systems. While these systems are often used primarily to increase interactivity in lectures (e.g., Sharma, Khachan, Chan & O’Byrne, 2005), they also provide a means for immediate peer assessment of activities such as class-based presentations which offer evidence of inter-rater reliability. In addition to laptop and handheld computers, other mobile devices such as cameras, telephones and portable digital voice recorders can also serve assessment purposes, either by capturing and communicating information that is used in presenting assessment items, or in assessment of them. For example, digital files can be included in electronic portfolios (Mathur & Murray, 2006) while mobile telephones and portable voice recorders provide opportunities that support synchronous and asynchronous peer communication for both undertaking assessment tasks or assessing others. Further opportunities for synchronous communication are available via online chat (which also offers the availability of a record of conversation through the transcripts generated by the chat tool in a learning management system). Edwards (2005) used online chat to support group assessment tasks related to a problem-based learning scenario, uploading the transcript of the chat after the session so that participating students could read it again and those who were unable to participate had access to what had been discussed. Instant messaging (including SMS mobile phone messaging and MSN internet messaging), and Skype software (http://www.skype.com) which offers free internet-based synchronous two-way communication that can be used by up to four people simultaneously, offer additional opportunities for assessment-related communication. Online videoconferencing and web conferencing also allow for real-time communication which is enhanced by other tools, including file sharing capabilities, whiteboards, chat programs and polling. These technologies offer a range of pos-
Issues in Peer Assessment and E-Learning
sibilities which might be included in the design of peer assessment. However, software programs specifically designed to implement structured peer assessment online have been developed over a number of years to support the provision of feedback and grades by students (e.g., Fermelis, Tucker & Palmer, 2007; Freeman & McKenzie, 2001; Liu, Lin, Chiu & Yuan, 2001; Ngu, Shepherd & Magin, 1995; Raban & Litchfield, 2007; Sung, Chang, Chiou & Hou, 2005; Rushton, Ramsey & Rada, 1993). Features may include providing (and perhaps critiquing) feedback and grades, viewing (and perhaps modifying) assessment criteria, calculation of grades for individuals and groups, and calculation of inter-rater reliability (which may involve comparison of teacher and student grades). Submission of assignments may occur via these programs. They offer a number of benefits relating to the quality, flexibility and speed of peer assessment, including reduction of manual workload (and of errors), and opportunities for monitoring reliability, though the opportunities are usually limited to the particular institutional contexts in which the tools are developed and used. Fermelis, Tucker and Palmer (2007) note that while most appear to make use of open source software they are currently incompatible with the centrally supported Oracle-based online environments commonly used in many Australian universities. At a simpler level, the availability of customizable grading forms within learning management systems provides another means of establishing criteria which may be used to support validity and reliability in the context of peer assessment. Also relevant to the opportunities for assessment provided by web-based technologies are those related to the management of plagiarism. Assessment design strategies such as personalizing and contextualizing assignment tasks, and structuring them so that each assessment task is linked to the previous one, help to reduce opportunities for intentional or unintentional pla-
giarism, whether or not assessment is undertaken electronically. Bhattacharya and Jorgensen (2008) consider that the development of approaches such as these involve design of the learning environment as well as assessment design. Oliver and Herrington (2001) present a model of online learning design that emphasizes the initial design of authentic learning tasks complemented by the design of resources and supports to allow students to complete the tasks. They suggest that the most successful forms of assessment occur when the learning tasks and the assessment tasks merge, and recommend forms of assessment including casebased activities, problem-solving, portfolio-based activities, product submissions, peer assessment and collaborative elements. Approaches of this kind lend themselves to the online environment and offer the pedagogical advantages of authentic assessment through situated learning rather than the reproduction of information that is often associated with plagiarism. When assessment tasks do involve reproducible information, it is certainly the case that the online environment presents a number of challenges in relation to plagiarism, particularly related to the ready accessibility of information, opportunities for collusion, and issues relating to identification of who completes the assessment task. However there are a range of strategies for minimizing plagiarism whether or not assessment is online. James, McInnis and Devlin (2002) present a number of these which include advantages which result from the teacher’s own access to electronic resources for monitoring material which students may misuse, as well as the ability to archive electronically previous students’ work on the same topic. In circumstances where the potential for plagiarism is unavoidable, one of the key opportunities that online technologies currently offer is the availability of plagiarism detection software. Its potential for verifying that an assessment submission represents a student’s own work has implications for validity. While it may be more appropriately used in a peer assessment context as a formative tool rather than
125
Issues in Peer Assessment and E-Learning
a judgmental one, Roberts (2008) notes that the future development of plagiarism detection software is likely to be a burgeoning area.
Web 2.0 Technologies A key advantage of Web 2.0 technologies for peer assessment is that not only do they offer the capacity for developing and presenting an individual or group output for assessment, but they also offer provision for commentary, either while the output is in development or as a summative option. In addition, the output may be in authentic forms, utilizing the features of web publishing, with mobile devices such as cameras and telephones providing easily accessible resources for enhancing the visual or auditory authenticity of the assessable material. Hence, a blog offers the potential of an authentic journalling experience (with each contribution recorded, dated and archived) which may be enhanced by audiovisual components. The provision for commentary by others (and for polling) provides a natural capacity for peer review and assessment as part of the environment. E-portfolio tools which may provide access to others for peer assessment offer similar advantages. In a wiki, the combination of individual discussion postings and a record of the history of individual contributions, together with evidence of a group’s output, offers a rich resource for both individual and group assessment by either teacher or peers. Given that group interactivity and outputs are fundamental features of the environment, it appears particularly appropriate that peer assessment should be involved when the software is used for teaching and learning. Through the development of rubrics with criteria relating to group and individual contributions students are able to participate in summative assessment. Podcasting can add further advantages associated with mobility and peer assessment. While it is probably most frequently associated with transmission of content to students (e.g., Kamel Boulos, Maramba & Wheeler, 2006), when the
126
content is designed by students and transmitted as an authentic assessment item, peer review and assessment can be included as part of the process (Thompson, 2007). Web 2.0 technologies which underpin current developments in games, simulations and virtual worlds provide highly sophisticated opportunities for immersive collaborative engagement. Drawing on the widely recognized benefits of experiencebased learning (Ruben, 1999), the opportunities that they offer for designed experiences (Squire, 2006) that present authentic challenges through narrative social spaces which require the collective action of the players (Amory, 2007), create major benefits for facilitating learning based on individual and social construction of meaning in solving situated, ill-structured problems. While these environments do not lend themselves easily to traditional concepts of peer assessment, they offer opportunities to explore new ways of considering how the interdependencies that are created between peers could be used for assessment.
Other Tools Given the range of software that is currently available, the possible uses of existing tools for peer assessment are probably only limited by the imagination of teachers. However, some obvious uses of tools that may exist independently of a networked environment, or may be utilized within learning environments involving both pre-Web 2.0 and Web 2.0 technologies, include: •
•
•
Use of the Track Changes or Commenting functions in Microsoft Word to provide feedback on work by peers; Use of the Commenting and Markup tools in Acrobat Professional for providing feedback on work presented in pdf files to ensure that changes to the document or the commentary are avoided; and Use of Excel spreadsheets for marking against criteria and compiling grades.
Issues in Peer Assessment and E-Learning
In addition, while audio and video are increasingly available as digital files within networked environments, the possibility remains for continuing use of pre-Web 2.0 technologies such as CD-ROMs or DVDs for delivering audio and video files, particularly where large file sizes are involved. These may be helpful for communicating an assessment task to students (e.g., presentation of an interview for analysis of interview skills), recording students’ responses to an assessment item (e.g., in providing evidence of performance or work-based competence which is undertaken at a site remote from the university), or for presentation of feedback and possibly a final judgment by assessors. The last option offers the advantages of accessibility, detail and the affective characteristics of oral feedback, along with a record of the assessment.
Chalenges of e-Learning for Peer Assessment This section outlines a number of specific challenges faced by teachers and students as users in supporting or undertaking peer assessment in relation to e-learning. These exclude general assessment design challenges of implementing peer assessment in any environment, such as those addressed by Falchikov (2005), except where the e-learning context adds further complexity. There are some fundamental challenges that need to be met in universities before peer assessment activities which depend on new learning technologies can be considered. These include the availability and technical reliability of information technology systems and programs which, in turn, depend on the existence of appropriate institutional policies, resources and infrastructure that support elearning if widespread implementation is to be possible. The time lag between the availability of new technologies and their uptake by universities has already been noted (Barnes & Tynan, 2007). Nichols (2008, p.598) comments that [u]nless a
state of institutional sustainability is achieved, it is likely that e-learning activity will in the long term be limited to enthusiasts.’ Sustainability is unlikely to occur through a ‘Lone Ranger and Tonto’ approach to educational innovation where small scale innovations often ‘end up as a costly supplement to conventional teaching’ and ‘usually lack quality in the final product’ (Bates, 1997). In the following sub-sections the availability of appropriate institutional infrastructure is assumed.
Pedagogical Callenges Informal peer assessment activities which support learning may occur readily between students, with or without the teacher’s involvement. However, peer assessment that is used for formal judgment of performance in award courses usually requires organization by the teacher. Teachers who use elearning tools or environments for this purpose require both familiarity with the issues involved in implementing peer processes to assess learning, and sufficient information literacy skills and familiarity with e-learning options to either plan use of existing tools or environments to meet the specific assessment requirements of the context concerned, or develop new options to meet those requirements. Even the introduction of simple, formative peer assessment strategies which utilize e-learning tools as a formal component of teaching and learning requires teaching staff to have a level of familiarity with the options available and awareness of the advantages and disadvantages of different ways of using them. If this does not exist, then the availability of appropriate staff development opportunities is a further requirement, which may involve addressing the particular issues raised by reconceptualizing teaching and assessment to draw on the functionalities of Web 2.0 technologies. Current developments in incorporating social software into learning management systems themselves create a pedagogical challenge: there appears
127
Issues in Peer Assessment and E-Learning
to be a conceptual contradiction in entering a user-controlled Web 2.0 environment through the teacher-controlled gateway of a learning management system. This contradiction seems to undermine the concept of user control. Consequently, there are considerable deterrents to the wide adoption of peer assessment for e-learning, suggesting a need for corresponding pedagogical (or other) imperatives and support systems if these are to be overcome. As Northcote (2002, p.624) notes, ‘it is the driving force of the pedagogical beliefs of the users … that will ultimately reflect the quality of online assessment.’ Even if teachers are comfortable in using the technology options available to them, there are further challenges associated with orienting students to undertake peer assessment in e-learning contexts. This may require careful communication with them (and the opportunity to practice) involving use or negotiation of assessment criteria as well as the tools themselves. If communication with students is only available online, this may add a further level of complexity in introducing peer assessment strategies. This is a situation where two-way synchronous communication may be beneficial in conveying expectations but the lack of face-to-face contact may be alienating for some students, while alternative options, such as online chat or desktop videoconferencing, potentially create additional barriers because both can be difficult in managing group communication. If possible, ‘the thoughtful integration of classroom face-to-face learning experiences with online learning experiences’ in a ‘blended’ learning environment (Garrison & Kanuka, 2004, p. 96) may be the best way of preparing students for online peer assessment because the face-to-face environment provides an opportunity to clarify expectations and requirements about the assessment activities which are to occur online. In addition to the pedagogical learning curve which may be required of students in applying assessment criteria and giving feedback to their peers, the form of the material to be assessed
128
can itself be challenging. For example, assessment of online discussion postings can be quite cumbersome, requiring the ability to handle and judge a potentially large amount of text-based material within multiple discussion postings by following contributions in discussion threads. This is made even more complex if students do not thread their contributions correctly. While simple approaches to assessing online discussion contributions focus on the quantity of messages, this provides no indication of the quality of the content. Development of rubrics for assessment of online communication provides a means of addressing this, as discussed earlier, but the use of rubrics themselves may be problematic in normalizing postings and reducing scoring to a ‘box-ticking exercise’ (Panko, 2006). In general, the pedagogical challenges which many teaching staff may face in conceptualizing peer assessment in an e-learning environment may be even more challenging for potential student assessors. The degree of preparation required may outweigh the potential advantages.
Management and Administration Challenges In addition to the requirement that both staff and students are familiar with the pedagogical aspects of peer assessment online, there are also a considerable number of management and administration challenges, some of which affect students as well as staff. They include the need to: •
•
Develop contingency plans for unexpected technical problems, particularly if peer assessment activities need to be undertaken within limited time frames; Ensure that students are able to access the sites or systems required (issues may include problems with usernames and passwords, browser configuration, or lack of broadband access if students are using dial-up modems from remote sites);
Issues in Peer Assessment and E-Learning
•
•
•
•
•
•
Ensure security of both students’ assessable work and the output of peer assessors when assessment involves a system that is not university-supported; Ensure that procedures are in place to protect privacy relating both to the material included in assessable work and to the results of the assessment; Develop or enforce procedures to reduce cheating and plagiarism and to deal with it appropriately if it is identified by peer assessors; Ensure that appropriate support systems are in place for students and that they know how to access them, taking into consideration issues such as access and equity, variations in competence with the technology, the needs of international students and any costs or ethical issues; Consider the workload involved for staff and students when assessment practices are time consuming (e.g., analyzing online discussions or wiki contributions and determining the extent to which criteria are met); and Managing the reporting and communication of feedback and grades (Benson & Brack, 2007).
The assistance of administrative and technical staff may be required to meet the above challenges.
Antiiting the Future for E -Learniieer Assessment The emphasis on assessing for learning in higher education, with the related implications for peer assessment, shows no signs of abating. It is further supported by a focus on teamwork which frequently appears in university graduate attributes as a means of ensuring preparation of students for employment and for life more
generally. While it is difficult to predict specific functionalities of e-learning tools that will emerge in the future, the democratic values which have pervaded the field of online learning (Goodyear, Salmon, Spector, Steeples & Tickner, 2001) are now firmly entrenched as a fundamental aspect of Web 2.0 technologies. The egalitarian nature of these environments supports the concept of peer review but the novice-expert implications of making reliable and valid judgments about performance seems alien to them. Hence, harnessing the intrinsic advantages of these environments to support peer assessment of learning, appears to require some adjustment to the way evidence is judged, drawing on egalitarian dialogue and consensus to identify standards and the extent to which they are met in any given situation. While this approach, which reflects the way judgments are made in many professional contexts, may seem cumbersome and a long way from the reality of most university teaching environments, the current pervasiveness of Web 2.0 technologies is undeniable and a need to ultimately make use of the opportunities that they offer for teaching and assessment seems inevitable. The current support for the notions of communities of practice and communities of enquiry in higher education provides further impetus in this direction, as do the ideas considered earlier about the importance of local, contextualized assessment, particularly for valid and reliable assessment of complex levels of achievement. Developments in games, simulations and virtual worlds represent another dimension of contextualized learning experiences where assessment for and of learning merge, with evidence of validity and reliability transparently identified by the extent to which challenges are met. The related merging of real and virtual worlds potentially offers major benefits for life and work where, for example, ‘online games can be informal but realistic simulators for contemporary leadership training (Reeves, Malone & O’Driscoll, 2008, p. 64) that is immediately applied in business. There appear to be unprecedented opportuni129
Issues in Peer Assessment and E-Learning
ties and challenges for reconceptualizing peer assessment in universities to take advantage of advances in Web 2.0 technologies, and identify the related implications for reliability and validity, suggesting a need for research and development to explore them. In the shorter term, the open source movement associated with Web 2.0 technologies is already gathering pace and challenging the ‘proprietary lock-in’ (Dron, 2007) of established learning management systems, which themselves replaced the ‘home grown’ institutional systems that preceded them. If e-learning environments are to become increasingly open source, what is the future of purpose-built peer assessment systems which currently offer one of the most efficient methods for summative peer assessment online? Perhaps it is a matter of time until one or more of these systems, built from open source software and with the capacity to meet peer assessment needs in a wide variety of situations becomes dominant, and institutional systems meet the end of their life cycle. This would reduce the duplication of systems with similar functions which have been developed at a number of universities, though it may be at the cost of tailoring applications to meet specific needs. However, this assumes a continuation of traditional methods of judging performance unless challenges to these methods are accommodated within the systems which emerge.
CONCLUSION E-learning tools and environments clearly offer benefits in relation to peer assessment that go beyond those that are available in face-to-face environments and they offer opportunities to enhance peer assessment even when it is conducted face-to-face. In fact, it is probably difficult to imagine a peer assessment situation in Western universities which does not make use of technology, even if in a very rudimentary way.
130
The opportunities of e-learning tools and environments relate to the quality, speed and flexibility of communication benefits, the administrative benefits that are offered, and the ability to support aspects of assessment for and of learning that are increasingly seen as valuable, involving teambased contextualized learning. Many of the tools available (both pre- and post-Web 2.0) might be seen as supporting assessment for learning more readily than assessment of learning, but their use for the latter purpose, and their appropriateness for supporting reliable and valid judgments, depends on the specific tools or environments used and, importantly, the way that they are used. This, in turn, raises a number of challenges related to the characteristics of the teachers and students as users. Leaving aside the inherent challenges of implementing peer assessment in any context, and assuming the availability of robust hardware and software, there may be a considerable learning curve involved for both teachers and students, which needs to be addressed before summative peer assessment can be implemented. There are then a range of management and administration factors which will need to be considered by the teacher, suggesting the requirement of a high level of commitment to and awareness of pedagogical issues if a satisfactory outcome is to be achieved. Management and administrative tasks may be minimized if purpose-built automated peer assessment tools are used, but development of these tools may be seen as requiring even higher levels of commitment and expertise by teachers, including a large upfront investment in time for planning, design and development, and then ongoing evaluation. Technical support and funding are likely to be further prerequisites. While appropriate design and implementation of these solutions probably offer the best opportunity for conducting reliable and valid peer assessment in e-learning environments, there is unlikely to be broad uptake unless one or more of these tools becomes dominant as open source software.
Issues in Peer Assessment and E-Learning
Beyond this, the functionalities of emerging Web 2.0 technologies offer both an opportunity and a challenge. They could be used to implement more sophisticated ways of presenting valid and reliable evidence and forming collegial, professional judgments that better prepare students for life and work. Overall, it would seem that in relation to peer assessment the benefits of e-learning tools and environments outweigh the challenges, offering clear advantages even if they are used in quite rudimentary ways. This is particularly the case in the use of strategies to support peer assessment for learning, though there may be circumstances where assessment of learning is more easily conducted without, or with only limited, use of these tools. However, the ultimate opportunity that e-learning offers is the ability to begin to conceptualize new ways that students might interact with each other and with information, which may in turn lead to new ways of considering evidence for assessment.
REFERENCES Allen, C. (2004). Tracing the evolution of social software. Retrieved April 21, 2008, from http://www.lifewithalacrity.com/2004/10/tracing_the_evo.html Amory, A. (2007). Game object model version II: A theoretical framework for educational game development. Educational Technology Research and Development, 55(1), 51-77. Barnes, C., & Tynan, B. (2007). The adventures of Miranda in the brave new world: Learning in a Web 2.0 millennium. ALT-J, 15(3), 189-200. Baron, J., & Keller, M. (2003, November). Use of rubrics in online assessment. Paper presented at the second ATN Evaluations and Assessment Conference, University of South Australia, Adelaide. Retrieved May 5, 2008, from http://www.unisa. edu.au/evaluations/Full-papers/BaronFull.doc
Bates, A. W. (1997). Restructuring the university for technological change. Retrieved August 18, 2008, from http://bates.cstudies.ubc.ca/carnegie/ carnegie.html Beaman, R. (1998). The unquiet … even loud, andragogy! Alternative assessments for adult learners. Innovative Higher Education, 23(1), 47-59. Bennett, S., Maton, K., & Kervin, L. (2008). The ‘digital natives’ debate: A critical review of the evidence. British Journal of Educational Technology. 39(5), 775-786. Benson, R., & Brack, C. (2007). Educational design and online assessment: A workbook. Melbourne: Faculty of Medicine, Nursing and Health Sciences, Monash University. Bhattacharya, M., & Jorgensen, L. (2008). Minimizing plagiarism by redesigning the learning environment and assessment. In T. S. Roberts (Ed.), Student plagiarism in an online world: Problems and solutions (pp.194-207). Hershey, PA: Information Science Reference. Bloxham, S., & West, A. (2004). Understanding the rules of the game: Marking peer assessment as a medium for developing students’ conceptions of assessment. Assessment & Evaluation in Higher Education, 29(6), 721-733. Boud, D. (1995a). Assessment and learning: Contradictory or complementary? In P. Knight (Ed.), Assessment for learning in higher education (pp.35-48). London: Kogan Page. Boud, D. (1995b). Enhancing learning through self assessment. London: Kogan Page. Boud, D. (2007). Reframing assessment as if learning were important. In D. Boud & N. Falchikov (Eds.), Rethinking assessment in higher education: Learning for the longer term (pp. 14-25). London: Routledge. Boud, D., & Falchikov, N. (2006). Aligning assessment with long-term learning. Assessment 131
Issues in Peer Assessment and E-Learning
& Evaluation in Higher Education, 31(4), 399413. Boud, D., & Feletti, G. I. (Eds.). (1997). The challenge of problem-based learning (2nd ed.). London: Kogan Page. Carless, D. (2007). Learning-oriented assessment: Conceptual bases and practical implications. Innovations in Education and Teaching International, 44(1), 57-66. Dalsgaard, C. (2006). Social software: E-learning beyond learning management systems. European Journal of Open, Distance and E-learning, 2. Retrieved August 15, 2008, from http://www.eurodl. org/materials/contrib/2006/Christian_Dalsgaard. htm Dron, J. (2007). Control and constraint in elearning: Choosing when to choose. Hershey, PA: Information Science Publishing. Edelstein, S., & Edwards, J. (2002). If you build it, they will come: Building learning communities through threaded discussions. Online Journal of Distance Learning Administration, 5(1). Retrieved August 15, 2008, from http:// www.westga.edu/~distance/ojdla/search_results_id.php?id=152 Edwards, S. (2005). Higher education in the twenty-first century: Examining the interface between graduate attributes, online and problembased learning at Monash University. Technology, Pedagogy and Education, 14(3), 329-351. Falchikov, N. (2005). Improving assessment through student involvement. London: RoutledgeFalmer. Fermelis, J., Tucker, R., & Palmer, S. (2007). Online self and peer assessment in large, multicampus, multi-cohort contexts. In R. J. Atkinson, C. McBeath, S. K. A. Soong, & C. Cheers (Eds.), ICT: Providing choices for learners and learning. Proceedings Ascilite Singapore 2007 (pp. 271-281). Singapore: Centre for Educational
132
Development, Nanyang Technological University. Retrieved August 15, 2008, from http://www. ascilite.org.au/conferences/singapore07/procs/ fermelis.pdf Freeman, M., & McKenzie, J. (2002). SPARK, a confidential web-based template for self and peer assessment of student teamwork: Benefits of evaluating across different subjects. British Journal of Educational Technology, 33(5), 551-569. Garrison, R., & Anderson, T. (2003). E-Learning in the 21st Century: A framework for research and practice. London: RoutledgeFalmer. Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7, 95-105. Goodyear, P., Salmon, G., Spector, J. M., Steeples, C., & Tickner, S. (2001). Competencies for online teaching: A special report. Educational Technology, Research and Development, 49(1), 65-72. Hoerr, T. R. (2007). Supervising Generation X. Educational Leadership, October, 85-86. James, R., McInnis, C., & Devlin, M. (2002). Assessing learning in Australian universities. Melbourne: Centre for the Study of Higher Education and The Australian Universities Teaching Committee. Jonassen, D. H., Howland, J., Moore, J., & Marra, R. M. (2003). Learning to solve problems with technology: A constructivist approach. Upper Saddle River, NJ: Merrill Prentice Hall. Kamel Boulos, M. N., Maramba, I., & Wheeler, S. (2006). Wikis, blogs and podcasts: A new generation of Web-based tools for virtual collaborative clinical practice and education. BMC Medical Education, 6(41), 1-8. Kennedy, G. E., Judd, T. S., Churchward, A., Gray, K., & Krause, K. (2008). First year students’ experiences with technology: Are they really digital
Issues in Peer Assessment and E-Learning
natives? Australasian Journal of Educational Technology, 24(1), 108-122. Knight, P. (2006). The local practices of assessment. Assessment & Evaluation in Higher Education, 31(4), 435-452. Knight, P., & Yorke, M. (2004). Learning, curriculum and employability in higher education. London: RoutledgeFalmer. Lejk, M., Wyvill, M., & Farrow, S. (1996). A survey of methods of deriving individual grades from group assessments. Assessment & Evaluation in Higher Education, 21(3), 267-280. Liu, E. Z. -F., Lin, S. S. -J., Chiu, C. -H., & Yuan, S. -M. (2001). Web-based peer review: The learner as both adapter and reviewer. IEEE Transactions on Education, 44(3), 246-251. McConnell, D. (2006). E-learning groups and communities. Maidenhead: Society for Research in Higher Education and the Open University Press. McCrindle, M. (2006). New generations at work: Attracting, recruiting, retaining and training generation Y. Retrieved May 1, 2008, from http:// www.mccrindle.com.au/wp_pdf/NewGenerationsAtWork.pdf Mann, B. L. (2006). Testing the validity of post and vote web-based peer assessment. In D. D. Williams, S. L. Howell, & M. Hricko (Eds.), Online assessment, measurement and evaluation: Emerging practices (pp. 131-152). Hershey, PA: Information Science Publishing. Mathur, S., & Murray, T. (2006). Authentic assessment online: A practical and theoretical challenge in higher education. In D. D. Williams, S. L. Howell, & M. Hricko (Eds.), Online assessment, measurement and evaluation: Emerging practices (pp.238-258). Hershey, PA: Information Science Publishing.
Ngu, A. H. H., Shepherd, J., & Magin, D. (1995). Engineering the ‘Peers’ system: The development of a computer-assisted approach to peer assessment. Research and Development in Higher Education, 18, 582-587. Nichols, M. (2008). Institutional perspectives: The challenges of e-learning diffusion. British Journal of Educational Technology, 39(4), 598-609. Nightingale, P., Te Wiata, I., Toohey, S., Ryan, G., Hughes, C., & Magin, D. (1996). Assessing learning in universities. Sydney: University of New South Wales Press. Northcote, M. (2002). Online assessment: Friend, foe or fix? British Journal of Educational Technology, 33(5), 623-625. Oblinger, D. (2003). Boomers, Gen-Xers and Millennials: Understanding the new students. EDUCAUSE Review, 38(4) 37-47. Retrieved August 15, 2008, from http://www.educause. edu/ir/library/pdf/ERM0342.pdf Oliver, B., & Goerke, V. (2007). Australian undergraduates’ use and ownership of emerging technologies: Implications and opportunities for creating engaging learning experiences for the Net Generation. Australasian Journal of Educational Technology, 23(2), 171-186. Oliver, R., & Herrington, J. (2001). Teaching and learning online: A beginner’s guide to e-learning and e-teaching in higher education. Mt Lawley, Western Australia: Centre for Research in Information Technology and Communications, Edith Cowan University. O’Reilly, T. (2005). What is Web 2.0: Design patterns and business models for the next generation of software. Retrieved May 1, 2008, from http://www.oreillynet.com/pub/a/oreilly/tim/ news/2005/09/30/what-is-web-20.html Panko, M. (2006). Learning and assessment: A case study – going the full Monty. In T. S. Rob-
133
Issues in Peer Assessment and E-Learning
erts (Ed.), Self, peer and group assessment in elearning (pp. 85-100). Hershey, PA: Information Science Publishing. Prensky, M. (2001). Digital natives, digital immigrants. On the Horizon, 9(5). Retrieved May 1, 2008, from http://www.marcprensky.com/writing/Prensky%20-%20Digital%20Natives,%20Di gital%20Immigrants%20-%20Part1.pdf Raban, R., & Litchfield, A. (2007). Supporting peer assessment of individual contributions in groupwork. Australasian Journal of Educational Technology, 23(1), 34-47. Ramsden, P. (2003). Learning to teach in higher education (2nd ed.). London: RoutledgeFalmer. Reeves, B., Malone, T. W., & O’Driscoll, T. (2008). Leadership’s online labs. Harvard Business Review, May, 59-66. Roberts, T. S. (2006). Self, peer and group assessment in e-learning: An introduction. In T. S. Roberts (Ed.), Self, peer and group assessment in e-learning (pp. 1-16). Hershey, PA: Information Science Publishing. Roberts, T. S. (2008). Student plagiarism in an online world. In T. S. Roberts (Ed.), Student plagiarism in an online world: Problems and solutions (pp.1-9). Hershey, PA: Information Science Reference. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: The Free Press. Rowntree, D. (1977). Assessing students: How shall we know them? London: Harper & Row. Ruben, B. D. (1999). Simulations, games and experience-based learning: The quest for a new paradigm for teaching and learning. Simulation & Gaming, 30(4), 498-505. Rushton, C., Ramsey, P., & Rada, R. (1993). Peer assessment in a collaborative hypermedia environment: A case study. Journal of Computer-Based Instruction, 20(3), 73-80.
134
Serafini, F. (2004). Three paradigms of assessment: Measurement, procedure, and inquiry. In International Reading Association (Ed.), Preparing reading professionals (pp. 207-216). Newark, DE: International Reading Association. Sharma, M. D., Khachan, J., Chan, B., & O’Byrne, J. (2005). An investigation of the effectiveness of electronic classroom communication systems in large lecture classes. Australasian Journal of Educational Technology, 21(2), 137-154. Shirky, C. (2003). A group is its own worst enemy. Retrieved May 2, 2008, from http://shirky. com/writings/group_enemy.html Spiro, R. J., Feltovich, P. L., Jacobson, M. J., & Coulson, R. L. (1991). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. Educational Technology, 31(5), 24-33. Squire, K. (2006). From content to context: Video games as designed experience. Educational Researcher, 35(8), 19-29. Strickland, J. (2008). Is there a Web 1.0? Retrieved April 21, 2008, from http://computer.howstuffworks.com/web-101.htm Sung, Y.-T., Chang, K.-E., Chiou, S.-K., & Hou, H.-T. (2005). The design and application of a web-based self- and peer-assessment system. Computers & Education, 45, 187-192. Surowiecki, K. (2004). The wisdom of crowds. New York: Doubleday. Thompson, L. (2007). Podcasting: The ultimate learning experience and authentic assessment. In R. J. Atkinson, C. McBeath, S. K. A. Soong, & C. Cheers (Eds.), ICT: Providing choices for learners and learning. Proceedings ascilite Singapore 2007 (pp.1019-1023). Singapore: Centre for Educational Development, Nanyang Technological University. Retrieved August 15, 2008,
Issues in Peer Assessment and E-Learning
from http://www.ascilite.org.au/conferences/singapore07/procs/thompson-poster.pdf Topping, K. J., Smith, E. F., Swanson, I., & Elliott, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Education, 25(2), 149-66. Vygotsky, L. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.
Wenger, E., McDermott, R., & Snyder, W. M. (2002). Cultivating communities of practice. Boston: Harvard Business School Press. Wilson, G., & Stacey, E. (2004). Online interaction impacts on learning: Teaching the teachers to teach online. Australasian Journal of Educational Technology, 20(1), 33-48.
135
136
Chapter VIII
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings Paul Lajbcygier Monash University, Australia Christine Spratt Royal Australian and New Zealand College of Psychiatrists, Australia
ABSTRACT This chapter presents recent research on group assessment in an e-learning environment as an avenue to debate contemporary issues in the design of assessment strategies. The underpinning research measured individual students’ contributions to group processes, individual students’ influence on their peers’ topic understanding of the related curriculum content, and the influence of the overall group experience on personal learning in an e-learning environment designed to act as a catalyst for the group learning. As well, the learning objectives fundamental to the project work were tested individually as part of the final examination. Further, the authors complemented the quantitative aspects of the research with focus group interviews to determine if students perceived that the e-learning environment helped attain the group learning objectives. The authors found that e-learning does not necessarily enhance deep learning in group assignments. They also found that the attainment of group learning objectives does not translate to the attainment of the same individual learning objectives. The chapter provides comment on the relationship that may exist between students’ perceptions of the e-learning environment, the group project work and e-learning group dynamics.
INTRODUCTION Having students work together in small groups, on some common assignment task is part of most
fields in university teaching (Biggs, 2003; Laurillard, 2002; Lejk, Wyvill & Farrow, 1997; Ramsden, 2003; Wen & Chin-Chung, 2008). Claims in support of using group work have ranged across
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
pedagogical activities such as providing for practice in group skills, preparation for professional life, the fostering of learning related interaction, and even the reduction of teachers’ marking loads (Bourner, Hughes & Bourner; 2001; Gammie & Matson 2007; Goldfinch, Laybourn, MacLeod & Stewart, 1999; Higher Education Academy, 2008; Sharp, 2006; Steensels et al., 2006; Thorpe 2008). Such activities can involve groups from three or four students up to larger teams of perhaps a dozen students. The work undertaken by those groups can range from some small defined task that could take days, through to large scale multi-faceted projects that might take an entire semester (often 13 weeks of full time study), and consume the bulk of the study time that a student has available within a single enrolment unit. With the growth of E-Learning across the higher education and corporate learning sectors globally, there is now a plethora of ‘tools’ available that can act as catalysts to promote group learning opportunities—for example proprietary and open source learning management systems such as Blackboard and Moodle; discussion groups (e.g. bulletin boards) permit group members to post information for the other members of the group to view non-synchronously; online chat forums permit group members to meet virtually and discuss their progress synchronously. More recently online social networking technologies and interactive spaces (e.g. YouTube, MySpace and wikis) as well as virtual real-time environments (e.g. Second Life) have been explored and used by teachers and learners to meet educational goals (Miller & Lu, 2007; Boulos, Hetherington & Wheeler, 2007; Elgort, Smith & Toland, 2008). In this chapter we focus on the alignment, attainment and testing of group and individual learning objectives. While this takes place in an E-Learning setting, we conclude that such an environment is not crucial for this purpose. In our context, the E-Learning that took place was part of a ‘blended’ learning strategy. The E-Learning that took place was in the ‘background’ so to speak,
facilitating weekly discussion and debate required for students’ group assignments using ‘chat’ and bulletin boards. Ultimately, while it aided the attainment of the group learning objectives, we did not investigate the impact of the E-Learning environment on group dynamics. As such, this chapter’s focus is on a comparison of individual versus group assessment in an E-Learning supported environment. Over the last five years we have studied whether group learning assessment is valid and whether E-Learning acts as a catalyst for group learning. This chapter, which is a synthesis of our prior work, provides evidence that E-Learning does not necessarily enhance deep learning in group assignments and that the attainment of group learning objectives does not translate to the attainment of the same individual learning objectives. It suggests the obvious: that an E-Learning environment per se does not automatically lead to the attainment of deep learning objectives. It also raises implications for further work in relation to group dynamics and ELearning supported group work in classrooms. An important caveat is that our study is somewhat limited since we focused on emoderation in forums and chats, which may not be representative of the learning and assessment potential of social networking techniques. It is feasible that the use of different technologies in the same assessment setting may create different group learning opportunities or lead to different conclusions.
Bakground: Stiiting Goup Leara E -Leaavironment The contextual setting that informs this chapter is a postgraduate unit of study undertaken over a 13- week academic semester, in which the majority of the students over time have been full-fee paying international students who are also in
137
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
paid employment in professional fields related to financial computation and investments. In previous years, students in this particular unit claimed that they enjoyed the unit and assessment tasks very much because they were aligned with their practical, applied interest in investments. While students appreciated the link between the assignments in the assessment strategy, the unit coordinator (the first author) considered there was limited critical debate and discussion among the student cohort concerning substantive curriculum content. In other words, while the students appreciated the curriculum structure and assessment approach, their motivation for learning (the assessment tasks) may well have limited their deeper engagement in aspects of the curriculum (in particular collaborative debate and problem solving) that was valued by the unit coordinator who designed the unit. To be a literate investor it is essential that investment decisions be based on a prudent process, including sound reasoning (considering all available resources), ongoing monitoring of investment performance and the incorporation of new information to modify risk exposure. Discourse among student peers, where they are given opportunities to reflect, debate and compare investment performance, is an ideal way to encourage this process. While the students in the unit are forced, weekly, to present their developing investment portfolios to their peers (and hence to monitor their investment performance), the unit coordinator believed that a potentially more valuable, ongoing discourse of critique and analysis was not occurring both inter and intra group as much as he had hoped. In other words, a new strategy was needed as it became apparent that students were not engaged in an ongoing discourse about their investment decisions in their large groups (which comprised up to 10 students). Instead, only those students responsible for investment decisions on any one week were considering the issues involved in changing their portfolios.
138
Consequently, the unit was redesigned to include opportunities for online engagement among students’ existing face-to-face groups. The unit’s redesigned pedagogical approach attempted to create a bridge between the pre-existing group assignment and an e-moderated online discussion group to encourage the kind of communication that reflected critical analysis which the unit coordinator suspected would ultimately contribute to the development of critical investment decision-making skills. He anticipated the potential challenges of e-moderation and chose to base his approach on the work of Salmon (2000, 2002), which he believed would provide a pragmatic structure and useful guiding principles. The university’s corporate learning management system provided the E-Learning infrastructure for the unit. With the addition of the E-Learning environment the unit was ‘flexibly’ delivered; what we mean by this is that we conceived the E-Learning environment as a support to the on-campus campus experience in an integrated manner. In other words, the E-Learning environment was used to enhance learning and provide a degree of choice for students in where and how they studied. This is essentially one form of what is now commonly recognised as ‘blended learning’. To be explicit the ‘face-to-face’ component of the subject included a two hour lecture and a one hour tutorial each week. The online component occurred when students met virtually to chat online prior to their weekly group assignment deadlines. They could also post information online so that the other members of their group could absorb and comment on the various activities when it suited them. The unit’s redesign was also informed by our existing understanding of the value of group learning in higher education, the principles of which were well supported in the various models of discursive E-Learning environments that were evolving and beginning to be reported in the literature. Sweet & Michaelsen (2007) provides an interesting overview of key aspects of group learning research in which they
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
analysed the way in which psychological research into group dynamics has informed the evolution of group learning approaches in higher education settings; in particular the influence of theories of discourse and interaction as well the role of group dynamics. The thrust of their argument is that ‘discourse structures matter to learning’ (p. 45): those structures are central to deciding the efficacy of group learning environments. Furthermore, that group learning outcomes for the group as a whole and the group’s individual members are influenced by the way the group matures over time, that is, like individuals, groups need to “learn how to learn’ (Sweet & Michaelsen, 2007. p. 33). We will return to these issues subsequently.
Assessiioup Learan an e -Leaaivironment The use of group work unavoidably raises some very real and complex assessment issues. How should the learning that has hopefully resulted from the group work experience be best assessed? How can measures of that learning be transformed into individual marks, thereby contributing to individual grades? Commonly the group work’s outcome is used as a proxy for the learning of individual group members (Conway, Kember, Sivan & Wu, 1993; Goldfinch, 1994; Goldfinch & Raeside, 1990; Lejk, Wyvill & Farrow, 1996; Gammie & Matson, 2007). The group’s product is marked as a single entity, and the resulting mark is then translated into marks attributed to the individual group members; the overall product mark becomes the individual group member’s mark, unaltered. When within-group variations are made, moderation of the product mark occurs to reflect the individual’s rated contribution to the group and that group product. There are at least two assumptions implicit in such approaches to marking group work. First, higher levels of individual learning are assumed to manifest as higher quality products or outcomes.
This assumption allows the single mark for the group product to be taken as an indicator of the “centroid” of the members’ individual attainments. Second, higher levels of participation in the group’s work are assumed to result in higher levels of learning by the participating individual. This assumption allows a single member’s rated contribution to be taken as an indicator of that member’s relative learning from the group work experience. But what if these assumptions are false? The primary role of educational assessment is the determination of students’ attainment of course or unit learning objectives or of their attainment of some demonstrable performance or outcome competence depending on the intentions of the particular curriculum framework. From this perspective, a mark is intended to be an indicator of the extent of those learning objectives’ attainment. But if contributions to group activity do not necessarily relate to individuals’ learning and product quality does not relate strongly to the aggregate of members’ learning levels, then our common practice of using moderated product marks as individuals’ marks could well constitute an unrecognized shift in the fundamental meanings of marks and grades. Work by Lejk and Wyvill (2001a; 2002) has demonstrated that when such contribution ratings are used to moderate group marks, the resulting individual marks can vary considerably dependent upon how those ratings are generated. When those ratings are made against separate aspects of contribution, rather than judged holistically, then individual marks are more likely to vary from the overall product mark, and to spread more evenly over a mark range (Lejk & Wyvill, 2001a; 2002). When contribution ratings result from a pooling of individual judgments made in private, rather than from consensus judgments made in public by the group, then the resulting individual marks tend to spread wider (Lejk & Wyvill, 2001b). In summary, a psychometrically desirable greater differentiation amongst group members would
139
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
be encouraged by nominating specific aspects of contribution to be considered, and by having students rate their peers in private. But this will not guarantee that any resultant contribution ratings will yet relate to individual learning attainments; it simply increases the chance that such a relationship will evidence, should it exist. Other work by Lejk and his colleagues has demonstrated that group work outcomes might relate only loosely to individual learning (Lejk, Wyvill & Farrow, 1999). Groups arrived at consensus solutions to discrete computing tasks; separately, students were tested individually on task related material. The quality of group solutions related strongly to the mean test performances of group members only for groups that were homogeneous with respect to those test performances. The starting point for the study underlying this chapter was a concern to test whether group marks based on the quality of group products do reflect the learning attainments of the group members, on knowledge and skills directly relevant to performing the group activity. If independent and individual measures can be found for those learnings, can they be shown to relate positively to group work outcomes? Moreover we wanted to test some alternative approaches to deriving individual marks, based on ratings other than contribution to group functioning. If we accept that group work is a learning experience, rather than an assessment exercise, then perhaps we could rate individuals’ performances as facilitators of their peers’ learning. Might ratings based on perceptions of peer tutoring effectiveness be a better basis for deriving individual marks from product marks? Might more direct measures of individual students’ involvement in learning related discussion be better indicators of those students’ learnings as attained from the group experience? Relationships amongst these measures were expected to test the assumptions outlined previously relating to the use of group outcome marks, and group contribution ratings as mark modifiers.
140
Thoup Project The subjects were the 66 students enrolled in a Business Systems unit in financial computing, versions of which are taken at final year undergraduate or as part of a coursework masters degree. All 66 eventually received a mark for their major group project work, although for the present analyses only 64 sat the final examination—52 completed the various rating instruments that provided data for the analyses. Working in five groups of ten students, one of nine, and one of seven, all students completed a major project activity. Groups made virtual investments across five market sectors, with the aim of maximizing their returns over the period of the project. Each group met at least once per week to review its portfolio, and to decide on any investment changes. The meetings were conducted face-to-face and virtually. The learning objectives associated with the project involved the use of various information sources in the making of investment decisions. These sources included company balance sheets, financial news media, technical price action reports, and macroeconomic indices. The intent was for students to recognize that investment decisions are not uniquely determined by the range of information available. Work on the project began in week four of the semester, and continued through to week twelve, when groups made class presentations and submitted their final project reports. Group marks for the project were decided predominantly on the bases of these final reports. The semester comprised 13 weeks of class activities, followed by a four week period in which examinations were taken. The mark received by an individual student for the group project contributed ten percent of the final grade in the unit.
The Ratings Measures Three instruments were developed with which students could rate their group project experi-
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
ences. On the first, students separately rated each of their group peers on contribution to the functioning of the group. In forming their judgments, students were explicitly prompted to think about a peer’s contribution to decisions on investment sector weightings and on specific firms, to providing extra information from media sources, to debate and discussion of issues, to fulfilling group coordination and allocated responsibilities. A five-point response scale was defined by descriptors at the extreme and mid-scale points; ‘participated consistently and reliably .. usually cooperative .. typically showed genuine interest and enthusiasm’, ‘participated satisfactorily .. at least sufficient that the group’s work could proceed adequately’, and ‘participated negligibly .. either rarely contributing or simply absent’. Point values of five through one were ascribed to the most participative scale response through to the least. The second instrument also required separate ratings of individual peers, but this time on contribution to a student’s present understanding of the project’s related topic material. Students were explicitly reminded of the main topic areas and learning objectives underlying the project. They were prompted to judge how interaction with a peer might have contributed to topic understanding. A five-point response scale was again defined by descriptors at extreme and mid-scale points; ‘.. frequently led directly to me learning new things .. having topic learning clarified .. discovering and correcting errors .. had strong influence on my topic understanding’, ‘.. had an influence on my topic understanding, but overall not strong or frequent or crucial’, and ‘.. rarely had any direct influence on my learning of the topic material .. my understanding predominantly from my own study, or from other members’. Point values of five through one were ascribed to the most influential scale response through to the least. The final instrument required students to make a relative judgment of the influence of the group experience per se to their topic learning. Again,
students were explicitly reminded of those topic areas and learning objectives. They were asked to decide whether the discussions and meetings with the group contributed to their present topic understanding, more or less than their own reading and study. Extreme and mid-scale descriptors defined a five-point response scale; ‘.. discussion with the group often led directly to me learning new things, or to correcting inaccuracies .. learning of the project content was strongly influenced by project oriented discussion and interaction’, ‘..learning of project content was influenced in roughly equivalent proportions by project discussions as by my own personal study and reading’, and ‘.. project discussion rarely resulted in understanding beyond that from my personal study and reading .. my learning of project content only minimally influenced by project oriented discussion and interaction’. Point values of five through one were ascribed to the most influential scale response through to the least. In the summer prior to the semester in which the present study was conducted, the present unit was taught to a summer enrolment cohort. Earlier drafts of these three instruments were used then, in a pilot trialing of their structure, and of their administration to the students. The versions used here were developed out of that piloting.
Procedure The financial computing unit ran over the semester as it would have done were no empirical investigation being conducted. In week 12, students completed their group projects, and submitted their reports for marking. In week 13, after submissions but before marking and examinations, the three instruments were distributed to students within a normal scheduled lecture class. The purposes of the study were explained as being to consider alternative ways in which project marks might be derived. Students were told that their responses to the three instruments would be used to compare different approaches to modifying the marks
141
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
that individual students might receive for group project work. However, students were assured that for the purposes of their actual marks in the unit, the present study was ‘hypothetical’. They would receive marks for their project work in exactly the way that was described in their original unit outlines; that being that all members in a group would receive the mark that the group report and presentation earned. Students were invited to complete the instruments there and then, privately. They were promised that their individual responses would not be made known to any of their classmates. The instrument forms used were individually customized to simplify the practicalities of responding; they were labeled with the respondent student’s name, and for the two peer rating versions, the names of the ‘other’ group members were already entered. During the four-week examination period that followed week 13, students sat a two-hour examination on the unit. That examination comprised 28 true-false questions scored one or zero, 38 multiple choice questions scored three or zero, and 12 extended multiple choice questions scored five or zero. Nine of the three-mark multiple choice questions targeted the learning objectives associated with the group project. Performance on those project related questions thus contributed 27 of the 202 raw marks possible on the examination. The examination contributed 60 percent of the final grade; thus the project related questions contributed eight percent of the final grade. In unit outline materials provided to students, the structure and content of the final examination had been advised.
The Group Project: Study Results In the first analysis the performances of individual group members on the nine project related examination questions were compared to the project marks attained by their groups. A simple perusal of Table 1 indicates no obvious relationship between
142
project outcome quality, as indicated by project marks, and the individual attainment of those learning objectives directly related to the project work, as indicated by performance on related examination questions. There would seem to be considerable spread in individual achievement within each of the groups, except perhaps for the fourth in the table. The simple correlation of group marks against mean number of questions correct was 0.198; but at df=5, this correlation would need to have exceeded 0.75 to be treated as significant at p<.05, 2-tailed. An important caveat is that with a sample size of n=7, we should not draw strong conclusions from this analysis. The present findings then provide no basis for concluding that group marks in any systematic way reflect the learning of individuals from the group experience. If the marks allocated to an individual should reflect the learning demonstrated by that individual, then awarding the mark for a group outcome to the individual group members would seem invalid as an assessment practice. If overall group product marks do not then reflect the relevant learning of group members, at least in some ‘averaging’ sense, it can still be asked whether those group marks could yet be modified to reflect learning variations amongst group members. That is, can the way in which we transform single group product marks into individual marks be such as to ensure that those individual marks yet reflect learning variations within a group? Within each group, the ratings for contribution to group functioning that any single student received separately from each of his or her group peers were averaged to yield a single ‘derived’ contribution rating for that student. Likewise, the separate ratings that a student received from each of his or her peers for influence on those peers’ topic understanding were averaged to yield a single ‘derived’ learning influence rating. As a preliminary, quartile values were calculated for both of these derived rating variables, across the available class enrolment as a single
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
Table 1. Numbers of group members performing at different levels on project related examination questions, compared to group mark for the project Number of examination questions correct
Group project mark
3
4
5
6
7
7.0
1
2
1
2
4
7.0
1
1
3
2
1
3
2
3
5
4
1
4
2
1
1
2
4
7.5 7.5 8.5
1
8.5 9.3
1
1
pooled group. Further, frequencies of ratings of the influence of the overall group experience on personal topic learning were calculated. It can be seen from Table 2 that all three measures spread widely. For the overall group experience rating, responses spread across all possible scale points (lower panel, Table 2). For the two derived rating measures, values spread almost across the possible range of one through 5 (upper panel, Table 2). Given that individual values for both derived ratings represent means, this is quite exceptional, and indicates that students were minimally ‘defaulting to the mid-scale’. It seems reasonable that the scale point definitions for all three measures effectively represented variation in perceived experiences, and that the students genuinely differentiated when making their respective rating judgments. Pearson correlations were calculated using the entire data as a single pooled set, amongst the derived contribution rating, the derived influence on peer understanding rating, the influence of the overall group experience rating, the project mark, the number of related examination questions correct, and total raw score on the final examination (see Table 3). The correlation between individual performances on project related examination questions and project marks is effectively zero (at 0.067), corroborating that the two measures can
8
9
5.6 3
5.8 5.8
1
6.6 5.6
4 1
Mean number of questions correct
1
1
7.4 5.8
be treated as independent. Further, project marks correlated minimally and non-significantly with examination total raw scores (at 0.173), indicating that project outcomes seem also not to relate to student learning in an overall aggregate sense. The correlation between students’ contributions to group functioning, as perceived by their group peers, and those students’ performances on project related questions was effectively zero (at -0.022). The correlation between students’ influences on the project related topic understanding of their group peers, as judged by those peers, and performances on project related questions was effectively zero (at 0.056). Finally, the correlation between students’ own perceptions of the overall group experience’s influence on their project related topic understanding, and their subsequent performances on project related questions was positive, but only minimally so, and not significant statistically (at 0.173). The present findings thus suggest that using contribution ratings to moderate project marks would not yield individual marks that reflected project related learning, but neither would ‘learning facilitator’ ratings nor self-perceptions of the influence of the overall group experience, that is if the benchmark is that individual marks should in some fashion reflect on individual’s learning on relevant learning objectives.
143
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
Table 2. Distributions of point values derived from the three rating measures, across all groups Contribution to group functioning Influence on peers’ topic understanding Influence of overall group experience on topic understanding Count (Percent)
Minimum 1.25 1.38 Pt value 1 (weak) 7 (13.5)
1st quartile 2.38
Median 3.06
3rd quartile 3.67
Maximum 4.67
3.09 Point value 2
3.87 Point value 3
4.43 Point value 4
3 (5.8)
20 (38.5)
13 (25.0)
5.00 Pt value 5 (strong) 9 (17.3)
Table 3. Correlations amongst rating measures, project mark, and examination scores .. peers’ ..overall .. mark .. questions .832 (66) .015 -.100 (52) (52) .348 .375 .290 (66) (66) (52) Number of related questions correct -.022 .056 .173 .067 (.. questions) (64) (64) (51) (64) Examination total raw score .368 .441 -.126 .173 .370 (64) (64) (51) (64) (64) Note. Top figure is correlation, lower figure is number of data pairs. Bold indicates significant at 0.01, 2-tailed; italics indicates significant at 0.05, 2-tailed.
Contribution to group functioning Influence on peers’ topic understanding (.. peers’) Influence of overall group experience on topic understanding (.. overall) Mark for group project (.. mark)
If group project marks do not necessarily reflect the learning of group members on project related topics, might the present findings suggest an interpretation of how project groups operate, and thus what such group marks do reflect? The correlations of each of the contribution and learning influence ratings with project marks proved moderately positive and significant (at 0.348 and 0.375). Project groups that include students whom their peers rate as having contributed well to group functioning, and as having had a positive influence on those peers’ related topic learning, tend to achieve better marks for their project outcomes. The correlation of judged influence from the overall group experience and project marks also proved positive and significant, albeit slightly less so (at 0.290). Students who rate the overall group experience as having positively influenced their
144
own related topic learning tend to have belonged to groups that achieved better project marks. The correlation between the two derived ratings measures, contribution and learning influence, was high, positive, and clearly significant (at 0.832). Students rated well by their peers for contributing to group functioning, and for influencing those peers’ topic learning, seem largely the same individuals. Finally, the correlations of each of the derived ratings measures with examination raw scores were moderately positive and significant (at 0.368 and 0.441). Students rated well for contributing to group functioning, and for influencing peer learning, tend also to be higher achieving students, at least in overall examination result terms. What is a possible picture of group functioning that emerges? The group project mark, or rather the quality of the group project outcome, might be seen as primarily a function of the effective-
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
ness with which a group operates in pursuing the project aims. In the present study’s terms, group effectiveness could be seen as a function of the presence of students who are good contributors, and good ‘learning facilitators’, and who are perhaps also above some threshold in terms of their basic academic learning capabilities. And none of this need have any necessary predictive relationship to the learning attained by individual group members, specifically as a result of the group experience. The group outcome is determined by how well the group operates as a group, in pursuit of the group’s goal of completing the project task. So long as the group ensures that its tasks are properly completed, by whomever, then a higher quality outcome is more likely. But that outcome quality need not depend directly on all or most of the group members achieving a higher level of understanding of the group project and its composite tasks and understandings. To use a sporting analogy, a ‘champion team’ is not necessarily a ‘team of champions’. One might speculate that in group functioning, a ‘critical mass’ model might apply. That is, if some minimum proportion of a group’s members are good ‘group operators’, then the group can function effectively. Until the membership passes that hypothetical proportion, it will remain dysfunctional. As a rough test, the simple multiplication of each student’s contribution rating with his or her learning influence rating was calculated. The possible scale range for each of these derived rat-
ing measures is one through five, with a mid-scale three. A somewhat arbitrary cut-off of 12 was selected for this multiplication, in that 12 would require a product of a ‘mid-scale’ with a ‘point above mid-scale’. This cut-off of 12 was intended to represent a ‘minimally but clearly collaborative’ group member. To re-state, this is a rough test. But from Table 4 it can be seen that the three best performing groups, in terms of project mark, had the three highest proportions of members above this arbitrary definition of ‘collaborator’.
The Group Project: Discussion The purpose of the study was to test two assumptions that underlie the common practice of using project outcome marks as the basis for ascribing marks to individual students, with ratings of contribution to group functioning moderating any differentiation amongst those individual marks. Neither assumption was found to hold. Project outcome marks did not reflect an aggregate of within-group individual attainment. Contributions to group functioning did not relate to within-group variation in individual attainment. The present study also tested two possible alternative bases for mark moderation, namely individual students’ ratings for influence on their peers’ learning, and individual students’ personal perceptions of the influence of the group experience on their related topic learning. Neither was found to relate to individual learning.
Table 4. Comparison of group marks against proportions of members judged ‘collaborative’ Mark for group project 9.3 8.5 8.5 7.5 7.5 7.0 7.0
Proportion members above ‘collaborative’ cut-off 0.60 0.78 0.57 0.50 0.10 0.50 0.30
Mean rating for contribution to group functioning 3.57 3.54 2.96 2.84 2.26 3.21 2.71
Mean rating for influencing peer learning 3.93 4.48 3.91 3.79 3.27 3.51 3.04
145
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
Th Group Project aa Impcations for Group Learning in e -Learang Evironments What the present findings did suggest instead was that group outcomes might be better interpreted as reflecting the effectiveness of a group’s operation, which in turn might be a function of some threshold presence of group members who are both ‘good group operators’ and generally good academically. Such an interpretation is perhaps supported by recent findings that while group project marks can relate strongly to changes in a group’s group functioning skills, there can yet be considerable variability on both group skills and marks within groups (Laybourn, Goldfinch, Graham, MacLeod & Stewart, 2001). How could it be that a group operates effectively, produces a high quality outcome, and yet not all group members benefit in terms of personal learning and understanding? If the mark that individual group members receive for group work is the project mark, or some function of it, then it is to each member’s advantage that that the group outcome be of the highest quality possible. This will be true regardless of how individual marks might be moderated. We might expect then that groups will tend to operate in whatever fashion efficiently maximizes the probabilities of such higher quality outcomes. These issues are important considerations in the current climate in higher education where more interactive web technologies presume greater opportunities for collaborative group learning in E-Learning. Sweet & Michaelsen (2007) provide a theoretical analysis of the most contemporary thinking in educational psychology regarding the dynamics of groups. They draw on several key models in their discussion of the implications of group dynamics for small group learning in higher education. They critique Slavin’s ‘integrative model’ which emphasises the role of motivational learning in small
146
group settings; where he claims that group goals increase three aspects of members motivation, that is: ‘their individual desire to learn, the extent to which they encourage other members’ learning and their willingness to support members’ efforts to learn’ (Sweet & Michaelsen 2007, p. 32). Van Meter & Stevens (cited in Sweet & Michaelsen, 2007, p. 33) suggest that it is ‘the structure of the collaborative discourse, not grouping per se, that has implications for the learning of individual members’—they argue that it is the way in which peers respond to groupmates by extending discussion points and learning comments, that becomes the crucial factor in the attainment of individual learning goals. We couldn’t agree more: much effort is expended by the teaching staff to foster student critique and debate, using the WebCT E-Learning environment. The discussion postings for each group are monitored and regularly comments are posted by the teaching staff in the hope that debate is generated. The ‘feedback’ mechanism is of course how the student groups investments are performing on a week by week basis. The students change their investments at the beginning of the week in accordance to macro-, country specific and stock specific news. They calculate their profit or loss to see if their decisions added value. Next, they reflect, critique and debate their decisions in the hope of making better decisions next week. The students are continuously learning how to learn in the group setting. One issue that has not been addressed is the highly ‘noisy’ investment environment and its effect on group learning. Financial markets are very difficult to predict. Consequently, despite much hard work and group interaction, investment decisions may lead to poor performance. This can be demotivating for students and counter productive for their learning. To counter this, relative investment performance is highlighted, not absolute. In this way students can compare their investment decisions to their peers, rather than to the market as a whole. Nevertheless, as
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
in life, hard work is a pre-condition for success but no guarantee of it! While the quantitative analysis of the group project reported here do not give an holistic indication that this was the case for us, we did see more potential (albeit less determined) evidence of these aspects in the qualitative evaluation of the project as a curriculum innovation discussed subsequently. As we have already identified, Sweet & Michaelsen (2007, p. 34) argue that the models provided by Slavin, Van Meter & Stevens and others provide empirical evidence to support their theories of discourse as central to group learning, however they deny: 50 years of evidence from the group dynamics literature showing that discourse structures in groups develop through clearly distinct and markedly different stages as groups mature over time. At the very least, group maturity merits mention among the most important “contextual factors” in models of small group learning. Further, rectifying this omission may very well require restructuring of our existing small group learning models. In our judgment, any model of small group learning will be incomplete unless it takes into account the fact that groups, like individuals, learn how to learn. (Sweet & Michaelsen 2007, p. 34) What this suggests of course is that curriculum design ought to ‘keep groups together long enough for them to develop into effectiveness, and promote both cognitive efforts and effective group interaction’ (Sweet & Michaelsen 2007, p. 41). Sweet & Michaelsen’s (2007) valuable paper is written in the context of research in face-to-face contexts, however one might presume that the principles of instructional design they indentify based on the empirical research in group dynamics would have utility in E-Learning settings. While we were engaged in the quantitative research of the group project reported above, we also wanted to complement that more empirical work with a holistic, qualitative evaluation of the overall pedagogical
aspects of the unit—and this is reported in detail elsewhere (Lajbcyier & Spratt, 2005). While the evaluation plan was established in this case to investigate the student experience in the unit, it focused on examining the way in which the integrated assessment strategy, especially, achieved the desired learning outcomes and the way in which students engaged online to complete the assessment tasks and meet the unit’s goals of developing skills of critique and analysis through debate and discourse. The evaluation aimed to ascertain if students perceived that the E-Learning discussion forum contributed to their developing problem-solving and analysis skills and their perceptions of the assessment strategy. Data collection was relatively standard and the evaluation employed online surveys and focus group interviews. It is worth noting that students used the institutional LMS as well as a free-ware instant messenger facility to conduct their virtual group planning and discussion sessions and to prepare the weekly reports required of them (The use of instant messaging–chat–was not planned by the unit coordinator but was student initiated.) The survey data indicated that students enjoyed comparing their group’s performance with the others using tools provided in the online site; they did not consider that the online chat forum made the group assignment any easier; they were generally neutral on the value of the online discussion forum. In the focus group interview students recognized that the success of any online discussion (synchronous or asynchronous) lies in their sense of its responsiveness. They were, however, quick to point out that they felt ‘it was not as good as face-to-face’ and importantly that ‘its usefulness depends on the individual and their motivation’. While students used the chat facility extensively, one student explained, in chat ‘you can’t see how people are reacting, if they are arguing with you,’ and this can be a disadvantage in gauging the success of the discussion. Students were well aware of the importance of motivation and social presence (what they have characterized
147
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
as ‘responsiveness’) to successful E-Learning and of finding opportunities themselves and with their peers to integrate the E-Learning environment they created with the assessment tasks to support successful learning outcomes. The students recognized that the unit instilled collaborative discourse, debate and decision making in investment decisions, and that they understood how investment decision processes can ‘incorporate debate and discussion and how that can lead to better investment decisions’. However, they did not attribute the attainment of this knowledge purely to the E-Learning environment or the technology per se. Overall, students felt that the online environment, including the discussion forum and chat forum were not fundamental to the deep learning that was trying to be instilled. Students believed that this kind of approach to learning could have been achieved without the online learning site. In other words, students recognized the online experience as providing a complimentary or supporting role in attaining the deep learning objectives of the unit: it was not essential but somewhat helpful.
Fture Trends Based on the contemporary theories of small group learning noted earlier we should perhaps return to what is probably the central issue, how we assess the learning that occurs in group project work? If project outcomes cannot be trusted to reflect individual learning, and a variety of group process measures cannot be trusted as learning sensitive moderation indices, how can we validly assess that learning? The study we report in this chapter suggests that in the final analysis individual learning should be assessed individually. We caution that our case study is just one within a particular instructional context. But if the findings here prove generalizable, then maybe we should re-define group projects to be not assessment opportunities, but a form of learning
148
experience (see Lejk, Wyvill & Farrow, 1999). Given the possible influence of contingencies discussed above, we might need to find ways to structure them to ensure that students commit and participate. This might mean not modeling group projects on workplace project teams, but rather deliberately designing them to encourage the student learning behaviours that we desire. But the simple conclusion here is that we cannot assume that individual assessments necessarily or directly derive from project outcomes. In light of other work in this text, our experiences have implications for the kinds of learning environments and learning experiences and assessment that may arise from the continuing growth and development in educational and other technologies. How might the use of wikis as described by Brack and Van Damme in Chapter 3 be implemented more formally into university programs of study? Can we expect such collaborative E-Learning environments to provide us with valid and reliable assessment opportunities in what some call ‘high stakes’ summative assessment? Have we made too many assumptions, based on face-to face pedagogies, about the ways in which groups form and function in E-Learning environments?
Ci E-Learning environments of themselves won’t be successful in the absence of excellent and innovative educational design. The curriculum design of the group project reported in this chapter was led by our interest in the potential of group learning and whether project work assignment grades could be considered a proxy for individual learning attainment as well as our interest in fostering the kind of critical discourse we knew intuitively might foster deep learning in our students. We presumed that the addition of a facility for online collaboration may have fostered both intended goals. However, for us in
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
this case, the evidence was not convincing for either intended outcome. We might speculate that high quality learning outcomes will result when groups align whatever relevant expertise they might possess with the tasks and activities required by their projects. This might mean members singly undertaking particular tasks for which they are well suited. It might mean varying combinations of members working on different activities. It would require expertise at group organization and functioning, somewhere in the group. It would require some minimum of content expertise somewhere, to enable substantive critiquing of project work. But none of this would necessarily require that all members develop high or even equivalent levels of content understanding. Such an operational picture of student project groups might not fit with an image of cooperative interaction in which group members support and develop their lesser peers, and we might lament that. But such a picture could be seen as the simple result of contingencies inherent in a project based marking regime. And in hindsight, this picture is probably not too different from how project teams in a commercial, ‘non-educational’ environment might operate.
ACKNOWLEDGMENT The original work informing this chapter was led by our friend and colleague, the late Associate Professor Malcolm Eley; it was presented by him at the ‘Improving Student Learning: Diversity and Inclusivity’ Conference in Birmingham, United Kingdom in 2004. Malcolm provided intellectual and collegial support to both of us and many other colleagues at Monash University over his long career in academic staff development. We are indebted to him not only for his professional advice but his friendship.
REFERENCES Biggs, J. (2003). Teaching for quality learning at university. Buckingham: Society for Research into Higher Education and Open University Press. Boulos, M., Hetherington, L., & Wheeler, S. (2007). Second Life: An overview of the potential of 3-D virtual worlds in medical and health education. Health Information and Libraries Journal, 24(4), 233-245. Bourner, J., Hughes, M., & Bourner, T. (2001) First-year undergraduate experiences of group project work. Assessment and Evaluation in Higher Education, 26, 19-39. Conway, R., Kember, D., Sivan, A., & Wu, M. (1993). Peer assessment of an individual’s contribution to a group project. Assessment and Evaluation in Higher Education, 18, 45-56. Elgort, I., Smith, A., & Toland, J. 2008, Is wiki an effective platform for group course work? Australasian Journal of Educational Technology, 24(2), 195-210. Gammie, E., & Matson, M. (2007). Group assessment at final degree level: An evaluation, Accounting Education: An International Journal, 16(2), 185-206. Goldfinch, J. (1994). Further developments in peer assessment of group projects. Assessment and Evaluation in Higher Education, 19, 29-35. Goldfinch, J., Laybourn, P., MacLeod, L., & Stewart, S. (1999). Improving group working skills in undergraduates through employer involvement. Assessment and Evaluation in Higher Education, 24, 41-49. Goldfinch, J., & Raeside, R. (1990). Development of a peer assessment technique for obtaining individual marks on a group project. Assessment and Evaluation in Higher Education, 15, 210-231.
149
The Validity of Group Marks as a Proxy for Individual Learning in E-Learning Settings
The Higher Education Academy. (2008). Groupwork, Retrieved August 6, 2008 from http://www. heacademy.ac.uk/ourwork/learning/assessment/ Group Lajbcygier, P., & Spratt, C. (2005) Using ‘Blended Learning’ to Develop Tertiary Students’ Skills of Critique, In L. Tomei (Ed.), Integrating information technologies into the classroom. Information Science Publishing, Hershey PA. Laurillard, D. (2002). Rethinking university teaching: a framework for the effective use of learning technologies. RoutledgeFalmer, London. Laybourn, P., Goldfinch, J., Graham, J., MacLeod, L., & Stewart. (2001). Measuring changes in group working skills in undergraduate students after employer involvement in group skill development. Assessment and Evaluation in Higher Education, 26, 364-380. Lejk, M., & Wyvill, M. (2001a). Peer assessment of contributions to a group project: A comparison of holistic and category-based approaches. Assessment and Evaluation in Higher Education, 26, 61-72. Lejk, M., & Wyvill, M. (2001b). The effect of the inclusion of self-assessment with peer assessment of contributions to a group project: A quantitative study of secret and agreed assessments. Assessment and Evaluation in Higher Education, 26, 552-561. Lejk, M.,& Wyvill, M. (2002). Peer assessment of contributions to a group project: Student attitudes to holistic and category-based approaches. Assessment and Evaluation in Higher Education, 27, 569-577. Lejk, M., Wyvill, M., & Farrow, S. (1996). A survey of methods of deriving individual grades from group assessments. Assessment and Evaluation in Higher Education, 21, 267-280. Lejk, M., Wyvill, M., & Farrow, S. (1997). Group learning and group assessment on undergraduate computing courses in higher education in the UK:
150
Results of a survey. Assessment and Evaluation in Higher Education, 22, 81-91. Lejk, M., Wyvill, M., & Farrow, S. (1999). Group assessment in systems analysis and design: A comparison of the performance of streamed and mixed ability groups. Assessment and Evaluation in Higher Education, 24, 5-14. Miller, M., & Lu, M-L. Serving non-traditional students in E-Learning environments: Building successful communities in the virtual campus, Educational Media International, 40(1), 163169. Ramsden, P. (2003), Learning to teach in higher education (2nd edition). London: RoutledgeFalmer. Salmon, G. (2000) E-moderating: The key to online learning. London: Routledge Falmer. Salmon, G. (2002). E-tivities: The key to active online learning. London: Kogan Page. Sharp, S. (2006). Deriving individual student marks from a tutor’s assessment of group work. Assessment & Evaluation in Higher Education, 31(3), 329 – 343. Sweet, M., & Michaelsen, L. (2007). How group dynamics research can inform the theory and practice of postsecondary small group learning. Educational Psychology Review, 19(1), 31-47. Steensels, C., Leemans, L., Buelens, H., Laga, E., Lecoutere, A., Laekeman, G. et al. (2006) Peer assessment: A valuable tool to differentiate between student contributions to group work? Pharmacy Education, 6(2), 111-118. Thorpe, M. (2008). Effective online interaction: Mapping course design to bridge from research to practice. Australasian Journal of Educational Technology, 24(1), 57-72. Wen M. L., & Chin-Chung T. (2008). Online peer assessment in an in service science and mathematics teacher education course. Teaching in Higher Education, 113(1), 55-67.
151
Chapter IX
Validation of E-Learning Courses in Computer Science and Humanities: A Matter of Context Robert S. Friedman New Jersey Institute of Technology, USA Fadi P. Deek New Jersey Institute of Technology, USA Norbert Elliot New Jersey Institute of Technology, USA
ABSTRACT In order to offer a unified framework for the empirical assessment of e-learning (EL), this chapter presents findings from three studies conducted at a comprehensive technological university. The first, an archival study, centers on student performance in undergraduate computer science and humanities courses. The second study, a survey given three times within EL classes, investigates the variables of learning style, general expectation, and interaction in student performance. The third study investigates student performance on computer-mediated information literacy. Taken together, these three studies—focusing on archival, process, and performance-based techniques—suggest that a comprehensive assessment model has the potential to yield a depth of knowledge allowing shareholders to make informed decisions on the complexities of asynchronous learning in post-secondary education.
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Validation of E-Learning Courses in Computer Science and Humanities
INTRODUCTION The aim of this chapter is to present evidence derived from three studies of e-learning (EL) targeting student outcomes. The studies were undertaken to identify a profile for potential success and support policy guidelines for limiting registration and course enrollment. This aim is achieved through discussion of three studies that were conducted at a comprehensive technological university in the United States, where EL has historically been offered as an asynchronous, online alternative to traditional face-to-face classes, with content management systems such as Webboard, WebCT and Moodle providing a “virtual classroom” environment;; the difference in persistence and success rates between the two modes of course delivery, however, are comparatively lower for online sections. The first study is archival and centers on student performance in undergraduate computer science and humanities courses. The second study, a survey distributed three times within humanities EL courses, investigates the variables of learning style, general expectation, and interaction in student performance. The third study investigates student performance on computer-mediated information literacy tasks. Our findings confirm not only that students in EL sections lack the required technical and information literacy skills to succeed. Taken together, these archival, process, and performance-based techniques warrant a unified framework for the empirical assessment of EL. Our findings suggest that a comprehensive assessment model has the potential to yield a depth of knowledge allowing shareholders to make informed decisions on the complexities of asynchronous learning in postsecondary education. New Jersey Institute of Technology (NJIT) has a history in e-learning (EL) beginning in the 1980s when researchers in the university’s Department of Computer and Information Science created and deployed the Electronic Information Exchange System for use in the original Virtual ClassroomTM (Hiltz & Turoff, 1993). In 2002, with computing education at its enrollment height,
152
approximately three-quarters of the 150 EL class sections (15-week semester cohorts) offered were in computer science or information systems at the undergraduate and graduate levels. Today, with 5,585 undergraduate students and 2,822 graduate students enrolled in the fall of 2008, demand for EL remains strong, and the disciplinary distribution of the approximately 60 EL sections offered during the current academic year is balanced among majors in computer science, information systems, information technology, and management; several of the university’s general university requirements in humanities are, as well, offered in an EL format. All class teachers are historically committed to delivering high quality instruction, and all are committed to an empirical base for decision-making regarding the evaluation of these courses to facilitate positive student outcomes in terms of success with courses and degree completion (Foster, Bower, & Watson, 2002). This chapter presents our efforts—through archival studies, surveys, and performance measures—to come to terms with the complexities of offering E-Learning courses. Within an environment that would be, in mission and vision, ideally suited to successful asynchronous instruction, it might be imagined that all measures would report incomparable success, yet such is not the uniform case. While much has been gained in our understanding of the complex variables and validation processes involved in justifying information, much remains to be done (Millwood & Terrell, 2005). Our studies reveal that a triangulated model is promising when a variety of shareholders investigate what really happens in asynchronously offered undergraduate courses.
Archihiudy: Two Diciplii Research on rates of student success in EL classes tends to be drawn from single samples. Yet comparison of two disciplines—one invested in re-
Validation of E-Learning Courses in Computer Science and Humanities
sponding to empirically-oriented tasks in a limited response format (responding to multiple-choice questions or creating computer code), the other invested in responding to verbally-oriented tasks in a open response format (participating in online discussions or submitting essays)—seemed ideal in allowing more to be known about the specifics of EL across disciplinary frameworks (Elliot, Friedman, & Briller, 2005).
Analysis of Undergraduates Enrolled in Computer Science Courses In 2002, administrators and faculty, attempting to address the existence of low student success rates in EL classes—and, subsequently, lower time to completion and graduation rates—learned they were not alone. In the USA, retention in E-Learning is a recurring issue. Low perseverance rates are found to fall in disproportional ranges when compared to traditionally taught classes (Institute for Higher Education Policy, 1999; Zimmerman, 2002). A recent study regarding perseverance in an EL introductory computer science (CS) class, for example, noted that the “dropout rate” (i.e., a student who submitted some work and then either withdrew from the class or failed to take the final examination) was 42%, a rate much higher than the 12% and the 26% for the same class, taught
in prior years by the same instructor in a face-toface format (Mock, 2003). These numbers were not greatly different from those within computing programs at NJIT. During the spring of 2003, administrators in the College of Computing Sciences (the administrative transformation of the then-expanding Department of Computer and Information Science) determined to address the disparity between EL classes’ low passing rates relative to sections offered in a traditional face-to-face (FTF) format. To validate that perception and to warrant changes in enrollment policy, student records were extracted from fall of 1996 to spring of 2002. The total sample of students for EL classes was 2,554, as Table 1 demonstrates, while the total number of students for the FTF classes was 15,468. Academic preparedness, withdrawal rates categorized according to grade point average (GPA), and class repetition were addressed in the archival study. Taking the college admission SAT Reasoning Test, which tests students’ subject matter knowledge in reading, writing, and mathematics (College Board, 2008a) as a standard measure of preparedness, the archival study revealed that the SAT score for the students in EL courses (M = 1097) was comparable to the average FTF score (M = 1106). Similarly, the SAT scores for students
Table 1. Computer science courses: An archival analysis of grade point average (GPA) All CS Courses
Total Seats
Cumulative GPA Less than 2.0
Cumulative GPA from 2.1 to 3.0
Cumulative GPA from 3.1 to 4.0
Total enrollment Number withdrew Percent withdrew Face to Face Enrollment Number withdrew Percent withdrew ELearning Enrollment Number withdrew Percent withdrew
18,022 1,9101 10.5% 15,468 1479 9.6% 2,554 422 16.5%
955 185 19.4% 833 142 17% 122 43 35.2%
9,872 1, 264 12.8% 8,534 1009 11.8% 1,338 255 19.1%
7,195 452 6.28% 7,110 328 4.6% 1,094 124 11.3%
153
Validation of E-Learning Courses in Computer Science and Humanities
who withdrew from the EL courses (M = 1095) and students who completed the EL course (M = 1099) were also comparable. With the acknowledgement of the College Board (2008b) that a range of 30 to 40 points reflects ability, the SAT scores of students who completed the FTF courses (M = 1131) may be understood to be just within the Board’s established range of difference than those who withdrew (M = 1098). If the academic preparation was similar in a comparison of EL and FTF undergraduate classes, the withdrawal rate revealed differences. Overall, the withdrawal rates from the EL classes were higher (16.5%) than from the FTF courses (9.6%). Examination of the GPA of the students revealed striking differences: for 955 students with GPAs below 2.0, 35.2% withdrew from EL classes, while only 17% withdrew from FTF classes. Further study revealed that 66.7% of students with a GPA below 1.0 withdrew from E-Learning classes, while only 16.7% of that same group withdrew from FTF classes; in a similar pattern, 70% of students with a GPA between 1.1 and 2.0 withdrew from E-Learning classes, while only 21.3% withdrew from FTF classes. These were the highest withdrawal rates identified in the study. For the 9,872 students with GPAs between 2.1 and 3.0, 11.8% withdrew from FTF classes, while 19.l% withdrew from EL classes. On the highest end of the spectrum, the withdrawal patterns were retained: for the 7,195 students with GPAs between 3.1 and 4.0, only 4.6% withdrew from FTF classes, while 11.3% withdrew from EL classes. Further analysis revealed that for the very best students—those with GPAs between 3.5 and 4.0, only 2.9% withdrew FTF classes, while 7.6% withdrew from EL classes. Further analysis using a Pearson correlation revealed a statistically significant correlation between student GPA and grades in both EL (r = .542, p < .01) and FTF (.532, p < .01) CS classes. Were students who withdrew or failed the EL classes more likely to pass if they repeated those classes in EL or FTF format? Of the 422
154
EL students who withdrew from their classes, only 30% repeated those classes in that format. Those students then passed at a 57% success rate. Those 70% who took the course in a FTF mode passed at a similar rate of 55%. However, of the 200 students who failed their EL classes, only 60% of those students were successful in their second EL classes attempt; of the 71% who took FTF classes on their next attempt to pass the class, 84% succeeded. Such archival analysis problematizes the traditional academic impulse: admit only students who have higher GPAs to EL courses. Students in both EL and FTF classes were of similar academic ability, and withdrawal rates were similar across GPAs. Taking a second EL classes after failing it resulted in a higher failure rate, while taking the course in a FTF mode increased the rate of success. Indeed, had a policy been designed to admit only students with high GPAs—a conclusion warranted by the data—then 39.6% (n = 1012) of the 2,554 students in EL classes would have been denied enrollment; had the cut off GPA been increased to 3.0, then 57.2% of the students (n = 1460) would have been denied enrollment. Such a policy appears especially problematic when we recognize that the very best students—those with GPAs of 3.5 to 4.0 (n = 2395)—withdrew from FTF courses at a noticeably lower rate (2.9%) than those enrolled in EL classes (7.6%). Clearly, more large scale studies need to be undertaken regarding the potential uniqueness of withdrawal rates, occurring regardless of student preparation and GPA, to the discipline of Computer Science.
Analysis of Undergraduates Enrolled in Humanities Courses A concurrent study extracted the records of all students (n = 384) enrolled in EL humanities courses offered as electives to all undergraduates to fulfill humanities general university requirement, from 1994 to 2001. Developed under funding from the Alfred P. Sloan foundation—a web-based consor-
Validation of E-Learning Courses in Computer Science and Humanities
tium that “encourages the collaborative sharing of knowledge and effective practices to improve online education in learning effectiveness, access, affordability for learners and providers, and student and faculty satisfaction” (SloanC, 2008) —two world literature courses were of special interest. Specifically developed to be offered in an EL environment, the courses feature lectures by the humanities faculty distributed to students (originally on VHS tape; currently streamed via the Internet) and asynchronous conferences in a WebCT format, with full use of well-articulated assignments leading to group projects and researched essays. These courses had reached their maturity in 2001 and 2002, and the 115 enrolled students provided a look at EL at its most advanced levels of delivery and student-centeredness. Unlike the majority of computing classes, offering content that is quantitative in nature and empirical in origin (making possible widespread use of multiple choice exams and other binaryoptioned assessment measures), all humanities classes anticipated highly subjective responses to complex readings, a process requiring significant peer-peer and peer-instructor interaction. The two world literature classes offered heightened interaction with students. While the junior-level students were far removed from any relevance to be gained from SAT comparisons (and many lacked SAT scores due to transfer status from two-year colleges), archival analysis demonstrated that the average GPA was similar between all the EL humanities courses (M = 2.85) and the two EL world literature classes (M = 2.81). Hence, the academic preparation appears to be nearly identical across groups. Nevertheless, as was the case in the CS analysis, retention complexities are apparent. The EL world literature classes began with a total of 115 students. Within the first two weeks of the class, 52 students withdrew. As the deadline for withdrawal approached, 26 students more students withdrew. By the end of the class, only 63 students were awarded final grades. Thus, there was a 42.5% loss
of students in a class taught by those instructors most prepared to work within an EL environment. Ironically, the featured class, incorporating those techniques and values most central to asynchronous learning, produced a higher withdrawal rate than had the CS courses. Taken together, the large CS archival study and the smaller world literature study are evocative; indeed, it may be useful to think of the CS study as the larger population that is representative of EL, with the literature classes part of that larger world. The similarities between student performances in the two worlds are noteworthy. There appears to be little difference in the academic preparedness of those who elect to take EL classes. Nevertheless,, even with similar academic preparation, there are striking differences in the performance of the EL students. In the EL classes, regardless of disciplines, the perseverance rates are unacceptably low. In order to understand this outcome, a student survey was developed, with questions practical and applied in nature, to allow further inquiry (DeTure, 2004).
Survey Study: One Disciplii To learn more about the process of student reaction to EL instruction, a survey was designed and administered to students enrolled in humanities classes over a two-semester period. Humanities classes were selected for survey because they are, at the present writing, designed to follow a best practices model of EL course delivery (Friedman, Elliot, & Haggerty, in press). Specifically, the instructional model currently employed is best described as a socio-technical system. In terms of education, Socio-Technical Systems (STSs) are computer technologies that enable social interaction of on-line learning, whether conversation (email), group discussion (chat), or group writing (wiki). STSs allow social networking, as well as collaborative idea generation and the sharing of knowledge through
155
Validation of E-Learning Courses in Computer Science and Humanities
academic journals (Whitworth, 2006; Whitworth & Friedman, 2008). As well, the humanities classes were selected because they are all writing intensive in nature. Students participate in discussions, draft, and submit writing in which referential (comprising scientific, informative, and exploratory discourse) and persuasive writing (comprising the systematic application of logical models) are both required (Kinneavy, 1971). Furthermore, the courses are informed by traditional and current directions in writing theory (Flower, 1994; MacArthur, Graham, and Fitzgerald, 2006), the ways that writing shapes and is shaped by cognition (Bazerman, 2008), and the state-of-the-art methods by which writing is assessed (Elliot, Briller, & Joshi, 2007). As Langdon Winner (1980) long ago correctly argued, artifacts have politics. While general principles such as the Sloan Consortium’s categories for success in on-line learning (Lorenzo & Moore, 2002) were in mind as the system was emerging, we sought to know the specifics of learning effectiveness, student satisfaction, and access. A model of student learning was therefore designed in the survey around the variables of student demographics, learning style, general expectations, and interaction. During the summer of 2006, Survey 1 (n = 108) yielded a 41.53% return, Survey 2, (n = 89) yielded a 34.23% return, and Survey 3 (administered after the withdrawal date during the closing days of the class, n = 62) yielded a 23.85% return. Surveys were given in classes with the following titles: The Pre-Modern World, the Making of the Modern World, World Literature I (2 sections), World Literature II, Writing about Science, Technology, and Society (two sections), and Esthetics and Modern Technology. During the fall of 2006, Survey 1 (n = 68) yielded a 50.75% return, Survey 2 (n = 58) yielded a 43.28% return, and Survey 3 (administered after the withdrawal date during the closing days of the class, n = 58) yielded a 52.25% return. Surveys were given in classes with the following titles: Technical Writing (two
156
sections), Literature and Medicine (two sections), and Engineering Ethics. Each instructor in each class had previously taught in an E-Learning environment and had volunteered to teach these classes, electives for all students that satisfy the general university requirements.
Demographics Nineteen questions on Survey 1 in our study were designed to obtain information about our students. While questions of gender, age, and ethnicity were standard, questions were also asked about the best language for writing and hours worked per week. To determine if the students had the technology at hand for these digitally intensive E-Learning classes, questions were included regarding the availability of high-speed connection and computers. Information regarding experiences with previous on-line courses was also requested. Broadly viewed as one of the nation’s most diverse campuses, the NJIT undergraduate students enrollment composition for 2005—the time after which all of the students in the survey were enrolled—reveals that 34.8% of the students are white, non-Hispanic, 20.2% are Asian/Pacific Islander, 31.2% are Hispanic, and 10.8% are Black, non-Hispanic. (14.8% declined to state their ethnicity, and 5.9% identified themselves as noncitizens.) 20 % of the 5, 360 undergraduates were women, while 80% of undergraduates are male. During the summer, 84.3% (n = 91) of the students reported that their best language for writing was English (χ2 (1, N = 108) = 50.704, p = .01.) as opposed to another language. When asked about the hours they were working either on or off campus, 47.2% (n = 51) reported that they were working 35 or more hours per week, while only 13.9% (n = 15) reported that they were working less than 5 hours per week. When asked if they used a DSL or cable connection when engaged in online learning, 94.4% (n = 102) reported that they did (χ2 (1, N = 108) = 85.333, p = .01.); asked if a computer was reserved primarily or exclusively
Validation of E-Learning Courses in Computer Science and Humanities
for their use, 89.9% (n = 97) replied that there was (χ2 (1, N = 108) = 156.167, p = .01.). Asked if this was their first on-line learning class, 39.8% of the students replied that it was, while 32.4% (n = 35) replied that they had enrolled in two or fewer classes; only 27.8% (n = 30) of the students were experienced on-line learners, having taken three or more such classes. During the fall, 77.9% of the students (n = 53) reported that their best language for writing was English (χ2 (1, N = 68) = 21.235, p = .01.). When asked about the hours they worked, 27.9% (n = 19) reported that they were working 35 or more hours per week, while 27.9% (n = 19) reported that they were working less than 5 hours per week. When asked if they used high-speed connections, 97.1% (n = 66) replied that they did (χ2 (1, N = 68) = 60.235, p = .01.); and 97.1% (n = 66) replied that they had a computer reserved for their use (χ2 2(1, N = 68) = 60.235, p = .01.). Twenty five percent (n = 17) of the students replied that this was their first experience with on-line learning, while 33.8% (n = 23) had enrolled in two or fewer classes. Forty one percent (n = 28) of the students reported that they had taken three or more on-line classes. Among the most striking results associated with the demographic portion of the survey were the perseverance rates revealed by the WebCT’s “Quizzes/Surveys” tool. The tool captures each student enrolled in the class at the time the survey was taken and maintains the students’ names in the database. Hence, the tool records students who enroll yet withdraw before the registrar’s deadline. These students may be termed “shoppers,” as they take up seats in the E-Learning classes beyond the deadline to drop or add a class, preventing other students from enrolling. During the summer semester 260 names appeared for the first survey; by the end of the semester, only 176 of these names appeared on the roster. Thus, 84 students—47.72 % of the available seats—were taken by those whose names never appeared on the final roster. During the fall semester, 134 names appeared on the first survey;
by the end of the semester, 111 names appeared on the final roster—a 17.16% loss of available seats due to the presence of shoppers. With 47% of students working nearly full-time in the summer and 28% working that same amount in the fall, it is easy to see how the convenience of asynchronous access can be confused with ease of course content (Williams & Hellman, 2004).
Learning Style To provide information about learning styles, the survey included nine questions that asked students to select between viewing the instructor as the class expert or the class facilitator, along with questions on motivation and preferences in deadlines. Analysis of survey 3, the summative summer set of responses, revealed that 62.9% of the responding students felt that they wanted the instructor to serve as an expert, rather than a facilitator, in class. That response lessened during the longer fall term, yet 58.6% remained steadfast in their views that the instructor should serve as expert, not facilitator. Regarding motivation, however, students did not feel that the instructor was the greatest motivator in class. In the summer 61% said that their own self-motivation was most important, followed by the instructor (32.3%), with fellow classmates placing a distant third (6.5%). A similar set of responses was given in the fall: self-motivation (67.2%), the instructor (29.3%), and classmates (3.4%). Overwhelmingly, students preferred hard deadlines over no deadline for work to be submitted. The preference for hard deadlines in the summer (79%) was reflected again in the fall survey (70.7%).
General Expectations While the survey included seventeen questions asking students to identify the degree of difficulty of the class, the amount of time, and the degree of engagement required for success, the survey also asked if the students found the writing de-
157
Validation of E-Learning Courses in Computer Science and Humanities
mands of E-Learning courses different than those encountered in face-to-face classes. During the summer, 91.9% of the students reported that they felt that the EL class was either the same as, or harder than, a FTF class, and in the fall 82.6% of the students reported the same level of difficulty. In the summer, 92.9% of the students reported that the class required either the same or more time than an FTF class, and 86.3% of the fall students reported the same level of time required; indeed, in the spring, 64.4% reported that more time was needed, and in the fall 50% reported the same. Again, the survey proved sensitive to the time of class; in the summer, 54.4% of the students reported that the course materials needed to be engaged four or five days a week, while in the fall, 50% of the students reported that the materials needed to be engaged two or three times a week. When that engagement occurred during the summer, 37.1% of the students reported that the engagement required two to three hours of work, and 61.3% reported that more than 3 hours were required; in the fall, 25.9% of the students reported that between one and two hours were required, 39.9% reported that between two to three hours were required, and 32.8% reported that three or more hours of engagement were required. During the summer, 82.3% of the students reported that their attention was either focused or completely focused, while in the fall 75.9% of the students reported similar levels of attention. Regarding the level of writing time, 66.1% of the summer students reported that the E-Learning class required more writing time, and 62.1% of the fall students reported that more writing time was required.
Interaction Nine questions were designed to gather information about the importance of interaction with the instructor, with classmates, with the digital video material, and with the on-line reading material. Two of the questions employed a six-point Likert
158
scale constructed according to a balanced model in which the labels formed an equal interval continuum bounded by opposite poles with a neutral midpoint (Dunham & Davidson, 1991), 65.5% of the summer students reported that they either very strongly agreed or strongly agreed that the instructor was an important part of the class (M = 4.98, SD = .98). In the fall, 51.7% of the students reached the same conclusion (M = 4.62, SD = 1.14). In the summer, 37.1% of the students agreed with the statement that collaborative classmate interaction was an important part of the class (M = 4.16, SD = 1.27), while in the fall 44.8% of the students agreed with this statement (M = 4.1, SD = 1.22). Overwhelmingly, students reported that the digital video material was either easy or foolproof (89.1% in the summer; 74.1% in the fall). A similar finding was true of the on-line reading material, stored both in the course management system and the university’s digital databases: 95.3% of summer students reported that access was either easy or foolproof, and 89.7% of the fall students reported the same levels of access.
Learning Style and General Epectations Variable Interactions The relationships between learning style and general expectations would, we hoped, reveal more about the variables of E-Learning in undergraduate humanities as they were hosted within our STS framework. Studies on general expectations within an E-Learning environment have also become central to E-Learning research. As enrollments have continued to grow—the National Center for Education Statistics (2003) has reported that during the 12-month 2000–2001 academic year, 56 percent (2,320) of all 2-year and 4-year granting institutions offered distance education courses, with an enrollment of 3,077,000 during that period—topics such as instructor preparedness have come to the forefront of research. Shavelson & Huang (2003), for example, found that many universities lack faculty who are properly
Validation of E-Learning Courses in Computer Science and Humanities
trained for creating and delivering online classes, resulting in lower student success rates. Elements for success identified by Frey, Faul & Yankelov (2003) include online posting of grades, sufficiently detailed and accurate lecture notes, and well-defined guidelines on how to finish the assignments, as well as consistent, constant interaction with the instructor. Such contact and interaction can take the form of virtual office hours and phone availability, lecturer clarity and dependability, according to Memon, Shih & Thinger (2006). In terms of behaviors that foster success, Ley & Young (2001) find that self-regulation is an essential component. This research is complementary to Dabbagh & Kitsantas (2005), who suggest that instructors spend more time building up and designing classes that actually promote self-regulation. In their comparative study, Leners & Sitzman (2006) find that caring in face-to-face classrooms is experienced through voice, body language, facial expressions, and behaviors that translate, according to Dillon & Stines (1996), to the E-Learning classroom through means of an empathetic perspective, timeliness of communications, and a tone of appreciation. This finding is in accord with Simonson (1996), who stresses that the relationship between the teacher and student may also matter in the student’s success. For Cereijo (2006), a similarly optimistic attitude and a strong willpower is required for student success, which is facilitated by self-discipline and enthusiasm reinforced by faculty-student interactions. Generally, as Jamison (2003) has
found, capability beliefs, a responsive environment, goal-oriented curriculum, mutual respect, enthusiasm, and diplomacy are strong factors of student completion. In the context of the success factors above, scale construction proved promising. Survey 3, given at the end of the classes, was used to construct the scale. Questions were grouped and combined to produce the scale and subscale. Questions of interest, including the ones discussed above, were combined to produce The Learning Style Scale. The General Expectations Scale also employed key questions, and two sub-scales were devised, focusing on E-Learning versus face-to-face comparisons and on comparative issues of time. The Interaction Scale was based, as well, on the questions of interest discussed above. If relationships were identified among the scales, then a coherent model could be developed. Overall, as Table 2 demonstrates, the internal consistencies of the scales were essentially strong, ranging from .698 to .787. The relationship of all scales to subscales, as measured by Cronbach’s alpha, was .549 during the summer semester and .515 in the fall semester. In the summer, the General Expectations Scale and its Subscale 1 demonstrated a high Pearson product moment correlation and high significance (summer, r = .759, p < .01, r = .797, p < .01). In the fall, the General Expectations Scale and both Subscales 1 and 2 demonstrated similarly high correlations and high significance (fall, r = .833, p < .01, r = .858, p < .01, r = .431, p < .01). The
Table 2. Cronbach’s Alpha correlations for learning style, general expectations, and interaction scales Summer 2006 (n = 62) / Fall 2006 (n = 58) 1. 2.
Learning Style (variable with 6 questions) General Expectations (variable with 4 questions on E-Learning vs. FTF comparison and 3 questions
.73 .698
.741 .735
3. 4.
on time) General Expectations Subscale 1: E-Learning vs. FTF (variable with 4 questions) General Expectations Subscale 2: Time (variable with 3 questions)
.764 .74
.787 .758
5.
Interaction (variable with 4 questions)
.744
.719
159
Validation of E-Learning Courses in Computer Science and Humanities
key to these strong within-scale relationships is to be found in the writing intensive nature of the E-Learning classes. As the high correlations between the General Expectations Scale and its two Subscales demonstrate, it is justified to think about student writing as being the cohesive vehicle by which the STS framework is mediated. However, while the General Expectations proved cohesive, no similar cohesiveness was demonstrated among the scales. Only a single statistically significant positive correlation (.394, p < .01) was found in the summer semester between the Interaction Scale and the General Expectation Scale; no other statistically significant positive correlations were observed. Of interest is the statistically significant negative correlation observed in the summer semester between the Learning Style Scale and the Interaction Scale (r = -.394, p < .01). A hint as to the cause of this finding is found in comparison of questions regarding the significance of the instructor and the role of motivation. During the summer semester, only 32% of the students held that the instructor was the greatest motivation, and 29% of the fall students answered in the same fashion. Yet 94% of the students agreed, strongly agreed, or very strongly agreed that interaction with the instructor was an important part of the course in the summer, and 89% of the fall students reported the same perception. The Learning Style Scale, thus, appears to be reporting a very different set of beliefs than the Interaction Scale: while learning style may be varied—students were split on the issue of instructor-centered versus learner-centered approaches—interaction with the instructor remains the most important factor in E-Learning classes (Andrysyszn, Cragg, & Humbert, 2001; Swan et al 2001; Paloff & Pratt, 2003; Wallace, 2005).
Gyles as Models Question 21 of Survey 3 asked students to identify their expected grade in the class on a 9-point
160
scale (A = 9, B+ = 8, B = 7, C+ = 6, C = 5, D = 4, W = 3, I = 2, F = 1). While we could not match survey to student, we nevertheless knew who had completed Survey 3 and, thus, could obtain their final grade from the class roster. In the summer semester, these students (n = 62) expected a final grade with a mean of 7.23 (SD = 2.13). The actual final grade mean of those students was 7.58 (SD = 1.4). The expected and final grades were correlated (.286, p < .05), although there was no statistically different difference (t = 42.36(61), p < .01). In the fall semester, students who completed survey 3 (n = 58) expected a final grade with a mean of 7.74 (SD = 12.9). The actual final grade mean of those students was 7.63 (SD = 1.6). Again, the expected and final grades were correlated (.286, p < .05), although there was no statistically different difference (t = 36.45(58), p < .001). Using question 21 as a proxy performance-based dependent variable appeared conceptually warranted because of the traditional use of course grades as a performance measure and because of the correlations between expected and final grades. The regression model, with the Learning Scale, the General Expectations Scale (and its two Subscales), and the Interaction Scale serving as the independent variables, was not statistically significant (R2 = .12 F(4, 56) = 1.97, p = .112), with only 12% of the variability of the expected grade representing the proportion of variance explained by the interaction of the independent variables. Similar results were seen in a regression analysis of the fall semester (R2 = .17, F(5, 51) = 2.092, p = .1). Clearly, there were factors influencing the model that were not explained by the usual criterion variable of the final grade. A dependent variable constructed was based on Type II learning styles—those expressing a preference for structure, cognitive simplicity, and conformity (Zhang & Sternberg, 2005, 2006). Using these questions as a constructed variable, we performed a regression analysis—with the General Expectations Scale (and its two Subscales), and the Interaction Scale serving as the
Validation of E-Learning Courses in Computer Science and Humanities
independent variables—for the fall semester with the following results: R2 = .227, F(4, 52) = 3.816, p < .01. The level of significance was high, with 23% of the variance in the Type II dependent variable represented by the interaction of the independent variables. This finding is among the most important of this third study. We offer three reasons for the significance of this finding. First, an outcome variable capturing a structured type of learning style provides a cohesive model for the STS framework examined in this study. This is not to say that grades, the universal proxy for performance, do not matter, but it appears as if the construct of structure is more important to the present model’s coherence (Felder & Brent, 1996). Second, structure appears to be dependent on factors attributed to the instructor. The syllabus, class materials, assignment content, and grading criteria—all provided by the instructor—are overwhelmingly found by the students to be adhered to, on hand, known, and articulated. As other researchers have found, the role of the instructor is key to E-Learning (Andrysyszn, Cragg, & Humbert, 2001; Swan et al, 2001; Paloff & Pratt, 2003; Wallace, 2005). Third, in that these are humanities classes under investigation, it might be assumed that an enrolled student might prefer reflective observation and active experimentation. In fact, however, the students might be said to prefer concrete experience and abstract conceptualization. As students in a technological university, their learning styles may—or may not—vary according to subject matter and may be mediated by both the STS framework (incorporating structure) and the humanities instructors (fostering divergent thinking) (Sahin, 2008). Hence, with Zhang & Sternberg (2005) we believe that an orientation toward learning styles should be based not on rigid categories but on individual differences as they are found in context. In sum: learning styles matter and must be considered as they emerge within contexts in which structure, provided by the instructor, rests at the center of the E-Learning environment.
Information Literacy : A Laent Variaia Borsboom, Mellenbergh, & van Heerden (2003) correctly argue that a “realist account of the latent variable is required to maintain a consistent connection between the formal and empirical concept” of the third variable (p. 204). Conceptually following these researchers, we seek to answer the relevant research question: Are there latent variable processes that generate the behaviors we are observing in our E-Learning studies? As the above analyses demonstrate, these behaviors may be summarized as follows: a very diverse group of CS undergraduate students with solid SAT Reasoning Test scores appear to be withdrawing from E-Learning courses at unacceptable rates that increase regardless of high or low GPAs. When sub-groups are examined in writing intensive E-Learning courses, these withdrawal rates increase. These students report no difficulty with their English language proficiently, nor do they report difficulties with the technological aspects of course management systems. While they believe that their own motivation is important, they also report, in seeming contradiction, that the instructor’s presence in very important. When outcomes models are developed, course grade does not appear to be as important as learning styles. Yet, although the new media’s intersection with writing is frequently described as part of the process of electracy (Ulmer, 2003)—a term describing rhetorical stances of chora, approbation, juxtaposition, commutation, nonlinearity, and imagery (Rice, 2007)—the student learning styles suggest they are anelectrate in their desire for structure, cognitive simplicity, and conformity. Are there individual processes, we must then ask, that exist as third variables involved in E-Learning that, if identified, will help us to frame the often contradictory relationships we are observing? Is there, as Sanford, Lajbcygier, and Spratt have proposed in this book (Chapter 8), a priori background knowledge that students bring, or do not bring, to the E-Learning tasks at hand? 161
Validation of E-Learning Courses in Computer Science and Humanities
In the fall of 2004, librarians and faculty at NJIT began a formal investigation of the information literacy skills of undergraduate students. Working with specialists in research and information literacy at the university’s Robert Van Houten Library, instructors in the department of humanities worked to design an information literacy model based on standards derived from the Association of College & Research Libraries (ACRL). In that the faculty had been assessing the writing skills of students enrolled in general undergraduate requirements (GUR) in humanities since 1996, a traditional portfolio assessment system had emerged that allowed reliable and valid programmatic information to be gained about student writing (Elliot, Briller, & Joshi, 2007). A new portfolio assessment system launched in spring 2005—termed the NJIT Information Literacy Scale (ILS)—shifted the assessment focus from writing to information literacy assessment (Scharf, Elliot, Huey, Briller, & Joshi, 2007). The NJIT ILS was designed to investigate student ability to identify, find, understand, and use information in drafting, revising, and finalizing researched, persuasive writing. While allowing similarly strong validity evidence to be warranted as the original portfolio system, the information literacy scores were lower than anticipated, documenting marginal to unacceptable levels of student information literacy skills. Instructional and library faculty were interested in learning more about the information literacy skills of their students. In fall 2005, NJIT and the Educational Testing Service (ETS) undertook a collaborative research agreement to investigate more fully—by means of multiple approaches—the variables of information literacy as they were evidenced within student performance at a public comprehensive technological university (Katz, et al., 2008). The collaboration would bring together the portfolio-based assessment approach of NJIT with the performance-based, automatically scored iSkills assessment, which was designed to measure
162
information literacy skills as they appear in technological environments (Katz, 2007b). The information and communication technology skills, complementary to the NJIT Literacy Scales, were designed assess student ability to appropriately use digital technology, communication tools, and/or networks to solve information problems in order to function in an information society. This student ability includes employing information as a tool to research, organize, and communicate information and having a fundamental understanding of the ethical/legal issues surrounding accessing and using information. Key to the ETS assessment is its Internet-delivered, performance-based assessment basis. Assessment administration takes approximately 75 minutes, divided into two sections lasting 35 and 40 minutes, respectively. During this time, students respond to 15 interactive tasks comprising a real-world scenario, such as a class or work assignment, that frames the information task. Students solve the tasks in the context of a simulation (for example, e-mail, Web browser, or library database) having the look and feel of typical applications. Thus, the study embodied two elements central to E-Learning within our specific institutional site: the use of writing as a process to deepen knowledge, and the use of computer-mediated environments to deliver content information. If students could demonstrate proficiency in these information literacy measures, we would be able to rule out the presence of a latent variable in the E-Learning process; that is, students were able to demonstrate, in FTF settings (where traditional instructional means were in place) that they possessed the skills necessary to retrieve and integrate information from divergent sources into their work. As well, if students were able to demonstrate, in the iSkills simulation, that they could research, organize, and communicate such information in a timed setting, then the essential skills necessary for success in E-Learning could be demonstrated.
Validation of E-Learning Courses in Computer Science and Humanities
While first-year students were assessed, these students do not take asynchronously offered E-Learning courses. For the present purposes, information gained from the sophomores, juniors, and seniors is most relevant. A simple random sample of upper-division students was created across each section of two representative humanities writing classes: cultural history and technical writing. These students, along with students from the senior seminar who were selected as described below, were identified for portfolio submission. The senior seminar students consisted of a census of all whose transcripts revealed that they had never taken any class outside of NJIT; hence, these students, while small in number, represented a meaningful population of NJIT students. Overall, students were found in cultural history (n = 95), as well as in technical writing (n = 48) and the senior seminars (n = 33). The sample resulting from the sampling plan closely matched the NJIT student population. Students were tested in a proctored computer lab on the iSkills assessment in late March 2006 and in May 2006 portfolios of targeted students were evaluated according to the NJIT ILS. Table 3 presents the results of the ETS / NJIT collaborative research. The scores for students on the iSkills Advanced Test enrolled in cultural history (M = 548.5, SD = 36.9), technical writing (M = 547.2, SD = 39.5), and the senior seminars (M = 568.3, SD = 28.2) were, at best, average. With a score range from
400 to 700, the students in the study performed at a level similar to that reported on a national level. Of 3,571 students tested on the Advanced iSkills test between January 2006 and May 2007, a score of 555 (SD = 32) was the reported as the average. That is, students were only able to answer 55% of the questions correctly. Nationally, across several tasks, few test takers could adapt material for a new audience. In a web search task, only 40% entered multiple search terms to narrow the results. When asked to organize a large amount of information efficiently, more than half the students failed to sort the information to clarify related material. When searching a large database, only 50% of test takers used a strategy that minimized irrelevant results (Katz, 2007a). Regarding the NJIT scale, it is clear that much remains to be done in helping students integrate sources meaningfully into their essays. While students in cultural history could achieve basic documentation forms—almost universally those advocated by the Modern Language Association (Gibaldi, 2003)—they had problems demonstrating that they had provided sources beyond those noted in the syllabus (evidence of research), identifying sources germane to their topic (integration), and achieving an overall sense of a researched essay (holistic score). In that a score below 7 is taken at NJIT to be a demonstration of weak performance, it appeared that only the citation variable—a mechanistic adherence to
Table 3. Information literacy scores: Advanced level ETS iskills assessment and NJIT information literacy scale (means and standard deviations) Cultural History (n=95) ETS Advanced iSkills Score
Technical Writing (n=48)
Senior Seminar (n=33)
548.5 (36.9)
547.2 (39.5)
568.3 (28.2)
NJIT Citation
7.8 (2.8)
5.3 (2.7)
8.3 (2.1)
NJIT Evidence of Research
6.9 (2.8)
5.8 (2.49)
6.8 (2.5)
NJIT Appropriateness
6.7 (2.7)
n/a
7.2 (2.3)
NJIT Integration
6.5 (2.7)
n/a
6.7 (2.3)
NJIT Holistic Score
6.8 (2.4)
n/a
7.0 (2.2)
163
Validation of E-Learning Courses in Computer Science and Humanities
citation format—could be taken as a measure of competency. The technical writing students’ abilities to cite sources and provide evidence of independent research were profoundly weak. Although only these two variables were assessed among other variables important to technical writing teachers, it is distressing that the lowest performance on the NJIT scales was identified in this group of students. Less distressing, however, were the scores of the senior seminar students. Scharf, et al. (2007), investigating a similar group of students at NJIT (n = 100), found ILS scores below the cut score of 7 on each of variables that make up the component ILS variable used in the current study. One year later, the citation, appropriateness, and holistic scores had met the competency score of 7. Surely, the skills addressed by both the ETS and NJIT measures are integral to successful E-Learning. If, as the ETS research confirms, the vast majority of students are performing poorly on the iSkills measure, then we must wonder about the impact of this lack of skill on E-Learning students. As well, if students in writing-intensive classes are not able to cite sources, extend searches beyond a syllabus, identify appropriate sources, and integrate information into their work—processes concurrent with drafting, revising, and finalizing researched, persuasive writing—then these students will surely either fail in E-Learning classes requiring such skills, or they will withdraw at high levels. It may indeed be the case that the essential E-Learning skills are not there in the first place.
Di As Markham and Hurst have reminded us in Chapter I, new conceptualizations of the process of gathering validity evidence are now emerging (Kane, 2006; Dialogue on Validity, 2007). The present shift from the term “validity” (a noun suggesting a concretized fact) to “validate” (a
164
transitive verb suggesting recurrent process) has been epitomized in Michael T. Kane’s conclusion about the nature of evidence in assessment contexts: “Validation has a contingent character; the evidence required to justify a proposed interpretation or use depends on the proposed interpretation or use” (2006, p. 60). With validity now viewed as a series of integrated judgments (Messick, 1989; AERA, APA, & NCME, 1999; Brennan, 2006), context-specific arguments (Mislevy & Brennan, 2006; Mislevy, 2007) and procedurally-dependent observations resembling science itself (Embretson, 2007), a new world of evaluation is before us. The three kinds of studies performed in this study—archival, process, and outcomes—serve to help us better understand the complexities involved in the assessment of E-Learning. Indeed, any one of the methods may not, in itself, prove sufficient: trend data, student response, and student performance (Lane & Stone, 2006) should each be taken into account when effectiveness studies are to be designed, conducted, and reported (Watkins & Corry, 2005). As our three studies demonstrate, there are complex variables, often contradictory, in play within a specific institutional setting. It is, therefore, difficult indeed to come to conclusions about what must be done regarding the success of any system. The kinds of tasks required for general E-Learning success at NJIT may be, for instance, similar to those found in the computer-mediated iSkills assessment. Yet those skills may not be sufficient to ensure success in specific classes, such as world literature, that are writing intensive. If students in FTF classes can demonstrate, at the very end of a class, only marginal skills, then how can similar students be expected to demonstrate those skills in the already complex asynchronous environment? To end with a case in point regarding the complexities of assessing E-Learning, consideration of the presence of cognitive complexity is in order. Writing assignments are socio-cogni-
Validation of E-Learning Courses in Computer Science and Humanities
tive activities that are congruent with many of the best practice values of E-Learning. As they write, students participate in discussions, draft, and submit assignments in which referential (comprising scientific, informative, and exploratory discourse) and persuasive writing (comprising the systematic application of logical models) are both required (Kinneavy, 1971). At their best, writing tasks are informed by traditional and current directions in writing theory (MacArthur, Graham, & Fitzgerald, 2006), demonstrate both the ways that writing shapes and is shaped by cognition (Bazerman, 2008), and the state-of-theart methods by which writing is assessed (Elliot, Briller & Joshi, 2007). As well, writing serves to lessen transactional distance. As proposed by Moore (1973; Moore, 2007), activities yielding high transactional distance, such as reading a textbook, afford little dialogue and less communication; activities yielding low transactional distance, such as independent studies, yield highly individualized communication. In the writingenhanced E-Learning environment, all transactions are writing-intensive, thus allowing for the desired low transactional distance. Hence, student writing is seen as the vehicle by which the STS framework is mediated (Bolter & Grusin, 2000) into an individualistic, student-centered context allowing low transactional distance—and, thus, greater communication. In that student writing plays such a central role in class design, the concept of lessening transactional distance through writing tasks should be of interest to all E-Learning instructors. Specifically, a student in a writing-intensive E-Learning class would likely be involved in the following potential case: planning a researched essay on, for example, the measurement of human intelligence, searching databases such as Academic Search Premiere and JSTOR to find the very best articles, reading and synthesizing those identified sources, documenting them in a standardized format, drafting and submitting the work for review, and then making final
revisions based on that review. If, however, the recent results of the iSkills assessment are taken into account—less than half the tested students could enter multiple search terms to narrow the results, more than half the students failed to sort the information to clarify related material, and only half of the test takers used a strategy that minimized irrelevant results—then the essential skills are lacking in the first place. Never mind about grammar and mechanics, about essay organization and writer’s aim, about documentation and editing; the assignment breaks down at the initial point of researched writing: the database search itself.
Future Trends Although the three studies presented in this chapter offer evidence obtained from recent experiences with E-Learning at NJIT, it is clear to us that the resulting implications affect all contemporary learning, regardless of its form. The rapid integration of course management tools and asynchronous instruction, originally deemed exclusive to EL, into FTF teaching has blurred the distinction between the two modes of learning. In fact, the amalgamation of the two communities is becoming the norm. Furthermore, ubiquitous technologies, both wearable and wireless, promptly embraced by this generation of learners, are assertively shifting campus interaction from exclusively physical locations to their digital alternatives. It is not difficult to imagine the emergent applications of this evolving “smart campus” to education (Jones & Grandhi, 2005). Thus, protesting EL and issuing calls for limiting its enrollment, which were unsuccessful in the past, are even more unrealistic now. Instead, technological and pedagogical policies need to converge in order to enhance the quality of student learning and ensure that the literacy and skills “needed to fully participate in an increasingly competitive work environment” are learned
165
Validation of E-Learning Courses in Computer Science and Humanities
(Kirsch, Braun, Yamamoto, & Sum, 2007), whether in physical or virtual settings.
C Regardless of SAT score or GPA, students simply lack the fundamental skills to deal with the new media. Alone, this fact helps us come to terms with the stunning withdrawal rates and the conflicted survey information presented in this chapter. Without the critical skills of identifying, finding, understanding, and using information—as well as rapidly and effectively researching and organizing information responsibly—the student will certainly be lost upon point of entry – in class and in the information workplace. Our attempts to validate our processes may thus achieve methodological soundness but fail to provide directions to deal with this emerging context. As our notions of validity are presently being re-imagined to acknowledge process, context, and contingency, it is perhaps helpful to imagine E-Learning itself as an emerging construct, one that mirrors the realities of students’ environments far more than it does the protocols of brick and mortar universities. In this view, there is much yet to learn about this critical socio-technical system and the contexts in which it operates.
REFERENCES American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association. Andrysyszyn, M. A., Cragg, B., & Humbert, J. (2001). Nurse practitioner preferences for distance education methods related to learning style, course content and achievement. Journal of Nursing Education, 40, 163- 170.
166
Aragon, S. R., Johnson, S. D., & Shaik, N. (2002). The influence of learning style preferences on student success in online versus face-to-face environments. American Journal of Distance Education, 16, 227-244. Bazerman, C. (Ed.). (2008). Handbook of research on writing: History, society, school, individual, text. New York: Earlbaum. Bolter, J. D., & Grusin, R. (2000). Remediation: Understanding the new media. Cambridge: MA, MIT Press. Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110, 203-219. Brennan, R. L. (2006). Perspectives on the evolution and future of educational measurement. In R. L. Brennan (Ed.). Educational measurement (4th edition, pp. 1-16). Westport, CT: Praeger. College Board. (2008a). SAT Reasoning Test. Retrieved October 4, 2008, from http://www.collegeboard.com/student/testing/sat/about/SATI. html College Board. (2008b). Score range. Retrieved October 4, 2008, from http://www.collegeboard. com/student/testing/sat/scores/understanding/ scorerange.html. Cereijo, M. V. P. (2006). Attitude as predictor of success in online training. International Journal on E-Learning, 5, 623-639. Dabbagh, N., & Kitsantas, A. (2005). The role of web-based pedagogical tools in supporting student self-regulation in distributed learning environments. Instructional Science, 25, 24-37. DeTure, M. (2004). Cognitive style and self-efficacy: Predicting student success in online distance education. American Journal of Distance Education, 18, 21-38. Dialogue on Validity [Special issue]. 2007. Educational Researcher, 36(8).
Validation of E-Learning Courses in Computer Science and Humanities
Dillon, R. S., & Stines, R. W. (1996). A phenomenological study of faculty-student caring interactions. Journal of Nursing Education, 35, 113-118. Dunham, T. C., & Davidson, M. L. (1991). Effect of scale anchors on student rating of instructors. Applied Measurement in Education, 4, 23-35. Elliot, N., Briller, V., & Joshi, K. (2007). Portfolio assessment: Quantification and community. Journal of Writing Assessment, 3, 5-29. Elliot, N., Friedman, R. S., & Briller, V. (2005, June-July). Irony and asychronicity: Interpreting withdrawal rates in E-Learning courses. Paper presented at the meeting of the Ed-Media World Conference on Educational Multimedia, Hypermedia, and Telecommunications, Montreal, Canada. Elwert, B., & Hitch, L. (Eds.). (2002). Motivating and retaining adult learners online. Essex Junction, Vt.: GetEducated.com. Embretson, S. E. (2007). Construct validity: A universal validity system or just another test evaluation procedure? Educational Researcher, 36, 449-455. Felder, R. M., & Brent, R. (1996). Navigating the bumpy road to student-centered instruction. College Teaching, 44, 43-47. Flower, L. (1994). The construction of negotiated meaning: A social cognitive theory of writing. Carbondale: Southern Illinois University Press. Foster, L., Bower, B. L., & Watson, L. W. (Eds.). (2002). Western cooperative for educational telecommunications. Best practices for electronically offered degree and certificate programs. ASHE Reader: Distance Education: Teaching and Learning in Higher Education. Boston: Pearson. Friedman R., Elliot, N, & Haggerty, B. (in press). E-Learning in undergraduate humanities classes:
Unpacking the variables. International Journal on E-Learning. Frey, A., Faul, A., & Yankelov, P. (2003). Student strategies of web-assisted teaching strategies. Journal of Social Work Education, 39, 443-457. Gibaldi, J. (2003). MLA handbook for writers of research papers. (6th ed.). New York: Modern Language Association. Hiltz, R. S., & Turoff, M. (1993). Network nation - revised edition: Human communication via computer. Cambridge, MA: MIT Press. Institute for Higher Education Policy. (1999). A review of contemporary research on the effectiveness of distance learning in higher education. Washington, DC: The Institute for Higher Education Research. Jamison, T. M. (2003). Ebb from the web: Using motivational systems theory to predict student completion of asynchronous web-based distance education courses. Unpublished doctoral dissertation, George Mason University. Jones, Q., & Grandhi, S. (2005). P3 systems: putting the place back into social networks. IEEE Internet Computing, 9, 38-46. Katz, I. R. (2007a, June). Digital natives but not information fluent: Results from ETS’s iSkills assessment. Presented at the Alliance for a Media Literate American Research Summit, St. Louis, MO. Katz, I. R. (2007b). Testing information literacy in digital environments: ETS’s iSkills™ assessment. Information Technology and Libraries, 26(3), 3–12. Katz, I. R., Elliot, N., Attali, Y., Scharf, D., Powers, D., Huey, H., Joshi, K., & Briller, V. (2008). The assessment of information literacy: A case study (ETS Research Rpt. No 08-33.) Princeton, NJ: Educational Testing Service.
167
Validation of E-Learning Courses in Computer Science and Humanities
Kane, M. T. (2006) Validation. In R. L. Brennan (Ed.), Educational measurement (4th edition, pp. 17-64). Westport, CT: American Council on Education/Praeger.
Millwood, R., & Terrell, I. (2005). Overview: New technology, learning and assessment in higher education. Innovations in Education & Teaching International, 42, 195-204.
Kinneavy, J. L. (1971). A theory of discourse. New York: Norton.
Mislevy, R. J., & Brennan, R. L. (2006).Cognitive psychology and educational assessment. In R. L. Brennan (Ed.). Educational measurement (4th edition, pp. 257-305). Westport, CT: Praeger.
Kirsch, I., Braun, H., Yamamoto, K., & Sum, A. (2007). America’s perfect storm: Three forces changing our nation’s future. New Jersey: Educational Testing Service. Lane, S., & Stone, C. A. (2006). Performance assessment. In R. L. Brennan (Ed.), Educational measurement (4th edition, pp. 387-431). Westport, CT: American Council on Education/Praeger. Leners, D. W., & Sitzman, K. (2006). Graduate student perceptions: Feeling the passion of caring online. Nursing Education Perspectives, 27, 315-319. Ley, K., & Young, D. B. (2001). Instructional principles for self-regulation. Educational Technology Research and Development, 49, 93–103.
Mislevy, R. J. (2007). Validity by design. Educational Researcher 36, 463-469. Mock, K. (2003). The development of a CSO course for distance delivery. The Journal of Computing Sciences in Colleges, 19(2), 30-38. Moore, M. G. (1973). Toward a theory of independent teaching and learning. The Journal of Higher Education, 44, 661-679. Moore, M. G. (2007). The theory of transactional distance. In M. G. Moore (Ed.), Handbook of Distance Education (2nd edition, pp. 89-105). New Jersey: Earlbaum.
Lorenzo, J., & Moore, J. (2002). The Sloan Consortium report to the nation: Five pillars of quality online education. Retrieved from http://www. sloan-c.org/effectivepractices/pillarreport1.pdf
National Center for Education Statistics (2003). Distance learning in higher education institutions: 2000-2001. (Rep. No. NCES 2003-017). Washington, DC: National Center for Education Statistics.
MacArthur, C. A., Graham, S., & Fitzgerald, J. (2006). Handbook of writing research. New York: Guilford Press.
Paloff, R., & Pratt, K. (2003). The virtual student: A profile and guide to working with online learners. San Francisco: Jossey-Bass.
Memon, A., Shih, L., & Thinger, B. (2006, June). Development and delivery of nuclear engineering online courses: The Excelsior College experience. Paper presented at the meeting of the ASEE Annual Conference & Exposition: Excellence in Education, Chicago, IL.
Rice, J. (2007). The rhetoric of cool: Composition studies and the new media. Carbondale: Southern Illinois University Press.
Messick, S. (1989). Validity. In R. L. Linn (Ed.). Educational measurement (3rd edition, pp. 13103). New York: American Council on Education/Macmillan.
168
Sahin, S. (2008). The relationship between student characteristics, including learning styles, and their perceptions and satisfactions in web-based courses in higher education. Turkish Journal of Distance Education, 9, 123-138. Scharf, D., Elliot, N., Huey, H. A., Briller, V., & Joshi, K. (2007). Direct assessment of informa-
Validation of E-Learning Courses in Computer Science and Humanities
tion literacy using writing portfolios. Journal of Academic Librarianship, 33, 462-477. Shavelson, R. J., & Huang, L. (2003, January/ February). Responding responsibly to the frenzy to assess learning in higher education. Change, 11–19. Simonson, C. L. S. (1996). Teaching caring to nursing students. Journal of Nursing Education, 35, 100-104. Sloan-C. (2008). A consortium of institutions and organizations committed to quality online education. Retrieved October 4, 2008, from http://www. sloan-c.org/ Swan, K., Shea, P., Fredericksen, E., Pickett, F., Pelz, W. & Maher, G. (2001). Building Knowledge Building Communities: Consistency, Contact and Communication in the Virtual Classroom, Journal of Educational Computing Research, 23(4/2000), 359 - 383. Ulmer, G. L. (2003). Internet invention: From literacy to electracy. New York: Longman. Wallace, R. M. (2005). Online learning in higher education: A review of research on interactions among teachers and students. Education, Communication, and Information, 3, 241-280.
Watkins, R., & Corry, M. (2005). E-Learning companion: A student’s guide to online success. Boston: Houghton-Mifflin. Whitworth, B. (2006). Social-technical systems. In C. Ghaoui (Ed.), Encyclopedia of Human Computer Interaction (pp. 533-541). London: Idea Group Reference. Whitworth, B., & Friedman, R. (2008). The challenge of modern academic knowledge exchange. SIGITE Newsletter, 5(2), 4-10. Williams, P. E., & Hellman, C. M. (2004). Differences in self-regulation for online learning between first- and second-generation college students. Research in Higher Education, 45, 71-82. Winner, L. (1980). Do artifacts have politics? Daedalus, 101, 121-136. Zhang, L., & Sternberg, R. J. (2005). A threefold model of intellectual styles. Educational Psychology Review, 17(1), 1-53. Zhang, L., & Sternberg, R. J. (2006) The nature of intellectual styles. New Jersey: Erlbaum. Zimmerman, M. C. (2002). Academic self-regulation explains persistence and attrition in webbased courses: A grounded theory. Unpublished doctoral dissertation, Northern Arizona University, Flagstaff.
169
170
Chapter X
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments Richard Tucker Deakin University, Australia Jan Fermelis Deakin University, Australia Stuart Palmer Deakin University, Australia
ABSTRACT There is considerable evidence of student scepticism regarding the purpose of team assignments and high levels of concern for the fairness of assessment when all team members receive the same grade. This chapter considers online self-and-peer assessment (SAPA) as a fair, valid and reliable method of assessing team processes and individualising grades. A pilot study is detailed that evaluated an online self-and-peer continuous assessment (SAPCA–a particular form of SAPA) tool originally developed for small classes of architecture students. The tool was adapted for large classes of up to 1,000 business communication students in a semester. The student sample trialling SAPCA studied on three dispersed campuses, as well as in off-campus and off-shore modes. The chapter proceeds from a literature review of SAPA, to a description of findings from four years of research, testing and development, and finally to a case study of SAPCA implementation with a total of 1,800 students enrolled in a business communication program.
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
INTRODUCTION
Ba
How can students be helped to develop teamwork skills at university and how can instructors assess these skills? During four years of researching these questions at an Australian university, focus has sharpened on the development of an online Self-and-Peer Continuous Assessment (SAPCA—a particular form of self-and-peer assessment) tool allowing for the individualisation of grades in teamwork assignments. Research has examined three interconnected areas; fair assessment, formative assessment, and reflective learning. Studies have involved diverse course cohorts, ranging from eighty to close to 1000 students, drawn from two faculties; a faculty of Science and Technology and one of Business and Law. Three courses offered in three degree programs have tested the SAPCA model. In one, around 1800 students enrol in a course offered in two semesters each year, on three campuses, in off-campus mode and at two offshore partnership campuses. Up to fourteen different members of staff are involved in the delivery of this unit at any one time, with uniform teaching materials and a strict comparability of assessment protocol. Approximately 60% of the cohort comprises full-fee-paying, International students, primarily from South East Asia, China and the Indian Sub-continent. Team compositions in this cohort can range from monocultural teams consisting Caucasian Australian same-gender students with English as their first language, to mixed-sex multicultural teams where the majority of students have English as a second language. It could be said that the SAPCA model has been piloted under the most testing of educational conditions. What follows is a synopsis of our findings; starting with a literature review, moving on to a description of the findings from four years of researching, testing and developing SAPCA, and finally to a case study of the SAPCA model tested in 2007 by way of the 1800 student, two-semester Business and Law course described above.
The reasons for the use of student teamwork in the completion of assessment tasks are many (Fermelis, 2006). It is posited that teamwork can lead to an improvement in student learning (James, McInnis, & Devlin, 2002). This improvement might be due to one or more of the following factors: the development of social behavioural skills and higher order thinking skills as well as promoting inclusive participation (Cohen, 1994); the development of critical thinking skills (Dochy, Segers, & Sluijsmans, 1999; Gokhale, 1995; Sluijsmans, Dochy, & Moerkerke, 1999); moving students from a passive to more active learning role (McGourty, Dominick, & Reilly, 1998); the ability to tackle more substantially-sized assessment projects (Goldfinch & Raeside, 1990); or that students learn from their peers within the team (van den Berg, Admiraal, & Pilot, 2006). It is also commonly identified that teamwork can develop skills that are sought by employers (Clark, Davies, & Skeers, 2005; Goldfinch & Raeside, 1990; Hanrahan & Isaacs, 2001), especially a range of non-technical ‘generic’ skills (James, et al., 2002; McGourty, et al., 1998), including interpersonal skills (Goldfinch & Raeside, 1990) and the capacity for lifelong learning (Hanrahan & Isaacs, 2001). Teamwork is cited as being more representative of the real world of work in a professional practice context, and, for students from the design-based disciplines, ideas and experience can be combined collectively for a superior result (Barber, 2004). Finally, used appropriately, student teamwork is one option for addressing issues related to rising student numbers in higher education (Ballantyne, Hughes, & Mylonas, 2002; Goldfinch & Raeside, 1990; James, et al., 2002), including: the expanding demand for physical resources in assessment (Brown, 1995); increasing student-to-staff ratios (Davies, 2000); and the drive from governments and other funding bodies for increased efficiency in higher education (Hanrahan & Isaacs, 2001).
171
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
However, the use of teamwork is not without challenges that require serious pedagogical considerations. It has been known since the 1900s that teamwork can be subject to the ‘Ringelmann effect’ otherwise known as social loafing, where the combined output of the team is less than would be expected from combining the output of individual team members (Kravitz & Martin, 1986). Teamwork may be subject to problems of team discipline and/or domination of the team by the most assertive members (Brown, 1995). Team members who are academically weaker may become “passengers” gaining a free ride from the efforts of other members (Goldfinch & Raeside, 1990). This ‘free-riding’ poses a question we shall consider in relation to assessment objectives in the section ‘Forms of formative SAPCA feedback’; namely, how to encourage active participation by all team members (Cohen, 1994). If teamwork is employed in assessment activities because it is a desirable skill for students to develop, then we should, as we shall discuss in the section ‘General Principles of SAPCA’, seek to assess the teamwork process itself (Clark, et al., 2005; Freeman & McKenzie, 2002), as well as devise a method for the fair assessment of the contribution of each individual team member (Brown, 1995; James, et al., 2002), so that equitable individual marks/grades can be awarded (see the section ‘Making Assessments and the Individualisation of Team Scores using SAPCA’). Student self- and peer assessment (SAPA) is often proposed as a solution to motivation, process and assessment issues in team-based assessment (Fermelis, 2006; Tucker, 2008; Tucker and Reynolds 2006). SAPA of teamwork may take a range of forms, with the basic premise being that students provide some form of assessment of the contributions to the teamwork process made by both themselves and the other team members. Such assessment can be: qualitative and/or quantitative; formative and/or summative; informal and/or formal; and periodic or one-off (Dochy, et al., 1999). Advocates of SAPA suggest that it
172
can increase student engagement (Michaelson 1992), with a consequent improvement in learning, through a number of mechanisms, including: giving increased responsibility, autonomy and power to students (Falchikov & Goldfinch, 2000; Taras, 2008); if students know that their contribution is to be assessed there will be less ‘free-riding’ (Johnston & Miles, 2004); it encourages students to consider the learning objectives and desired performance levels of assessment (McGourty, Dominick, & Reilly, 1998); encouragement of reflective thinking (Dochy, et al. 1999; Freeman & McKenzie, 2002; McGourty, Dominick, & Reilly, 1998); and the process of providing feedback to others can make one focus on improving one’s own performance (Davies, 2000; Freeman & McKenzie, 2002; McGourty, et al., 1998; Sluijsmans, Dochy, & Moerkerke, 1999). In addition to learning benefits, SAPA is seen as contributing to generic skills development by offering students experience in giving and receiving critical appraisal (Sivan, 2000). Apart from student benefits, for academic staff with the responsibility for assessment, SAPA is a practical mechanism for the individualisation of student results from a teamwork exercise (Freeman & McKenzie, 2002; Goldfinch & Raeside, 1990; Raban & Litchfield, 2006; Walker, 2001), as well as a means for the provision of some additional formative feedback/assessment for students in the face of increasing class sizes (Davies, 2000; Mulder & Pearce, 2007; Topping, 1998). Like teamwork, SAPA itself is not without issues that deserve consideration. SAPA is a broad term that covers a range of assessment possibilities and implementations (Dochy, et al., 1999) – and the most appropriate combination will depend on the requirements of the situation. Peer assessment may lead to ill-will between team members, or, if team members avoid honest criticism of their peers, it may fail to accurately reflect individual efforts or become more of a measure of member personality than contribution (Goldfinch & Raeside, 1990). A key question about SAPA that is used for summa-
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
tive purposes relates to the validity and reliability of assessment made by students. However, a large body of research reports good correlation between assessments made by students and academic staff (Falchikov & Goldfinch, 2000; Sluijsmans, et al., 1999; Stefani, 1994; Topping, 1998). As we touch upon in the next section, in the case of the use of SAPA to assess the teamwork process rather than the absolute merit of the teamwork product, members from within the team are probably better placed than external academic staff to make this assessment (Brown, 1995). Finally, there is no clear evidence that, regardless of the absolute level of reliability of SAPA, it is any less reliable than traditional assessments made by academic staff (Walker, 2001). The literature provides some guidance on the characteristics of an effective SAPA system. SAPA can require students to make their assessment based on specific dimensions/criteria, or to make a simple, holistic assessment of team member contribution. An analysis of published cases of peer assessment found that holistic assessments by students more closely resembled assessments by academic staff than did methods involving rating on multiple dimensions (Falchikov & Goldfinch, 2000). Another study compared a peer assessment scheme incorporating a range of dimensions, each with detailed descriptors, against one that required a more straightforward holistic assessment of contribution based on each team member having ‘100 percent’, which they were free to divide amongst the team members in any manner which they felt reflected each member’s contribution to the group’s work. It was found that the ‘holistic’ group reported a more positive rating of teamwork, suggesting that this SAPA approach might be more supportive of the aims of teamwork, and lead to a greater spread in individual marks than a system based in detailed criteria (Lejk & Wyvill, 2002). Collecting SAPA ratings anonymously can avoid the potential for team conflict arising from negative performance ratings/feedback, and has also been found to lead to
a greater spread in individual marks, perhaps due to team members feeling freer to rate truthfully when their identity and rating is secret. Additionally, in the absence of collusion between raters, anonymous/secret rating is more statistically reliable, as the ratings awarded by one member should not influence any other rater (Sharp, 2006). It is suggested that a SAPA system should include a rating period or window during which a rater can modify their rating based on further reflection of team performance in isolation from possible peer pressure (Willey & Freeman, 2006). A normal aim of employing SAPA is to facilitate the individualisation of team member marks. A range of approaches to individualisation exist, but many involve using student SAPA ratings (whether holistic or multi-criterion) to compute a multiplicative scaling factor (MSF) used as the basis to convert an overall combined team mark into individual student marks (Brown, 1995; Goldfinch & Raeside, 1990; Johnston & Miles, 2004; Raban & Litchfield, 2006; Willey & Freeman, 2006). As with any marking exercise, the slavish use of formulae to individualise teamwork marks should be avoided, as this may lead to anomalous results (Goldfinch & Raeside, 1990; Sharp, 2006). It is observed that the individualisation factors derived through SAPA seldom suggest the extreme modification of individual marks (Johnston & Miles, 2004), and that SAPA results are only one of the factors that should be taken into account when determining final individual marks for teamwork (Luca & McLoughlin, 2002). One approach is to use the presence of ‘extreme’ MSFs (outside of some predetermined upper and lower band) to signal the need for further investigation of team performance and/or possible individualisation of teamwork marks (Willey & Freeman, 2006). While the ultimate aim of SAPA might include the individualisation of teamwork marks, the inclusion of multiple (continuous) SAPA stages during a teamwork activity offers the possibility of valuable formative feedback as well, as students have the opportunity to modify their
173
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
performance and approach to teamwork as the activity proceeds (Walker, 2001). Improvement in student performance has been observed in teamwork contexts incorporating multiple instances of SAPA (McGourty, et al., 1998). While SAPA is often cited as a strategy for reducing assessment workload, the administration of a SAPA system for a large class can become a daunting workload in its own right (Ballantyne, et al., 2002; Mulder & Pearce, 2007). The emergence of computer-assisted SAPA systems was noted in the 1990s (Goldfinch & Raeside, 1990; Topping, 1998), and since that time the use and evaluation of many online SAPA systems providing assistance to both students and academic staff have been reported (Freeman & McKenzie, 2002; Lin, Liu, & Yuan, 2001; McGourty, et al., 1998; Raban & Litchfield, 2006; Sitthiworachart & Joy, 2003; Wen & Tsai, 2008). Computer-supported learning systems, including SAPA, have been shown to foster student engagement with learning activities, and can provide an independent administrative record of student teamwork activity (Resta & Laferrière, 2007). However the successful use of student teamwork involves much more than effective assessment (computer-based, online or otherwise). It includes the design of assessment tasks suited for completion by teams, development of teamwork skills by students and the on-going management of team dynamics during the course of the teamwork. Design of processes and tools for the assessment of teamwork in isolation of consideration of these other factors is unlikely to produce a successful outcome (Freeman & McKenzie, 2002).
Slf-and-Peer Continuous Assessment G Cntinuous Assessment Based on the use and development over three years of online SAPA as a solution to teamwork
174
assessment problems for cohorts of between eighty and 180 students in an Australian school of Architecture, we have previously published the following findings and recommendations (Tucker & Reynolds, 2006): 1.
2.
3.
4.
If teamwork assignments are to reflect the type of willing and productive collaboration demanded by professional practice, then the completed assignment can only be assessed as a product of that collaboration, whereas the assessment of an individual’s contribution to the project must focus rather on the process of arriving at that product. Since tutors are party to only a fraction of that process, then, as Brown has shown (1995), the students themselves are best placed to accurately evaluate contributions to process. Students find peer assessment to be more manageable and a more accurate reflection of individual contribution when it is continuous throughout a teamwork assignment (as multiple periodic assessments of process), rather than a one-off assessment of the product occurring at the end of the assignment. The quality of teamwork as measured in grades increases in problem- and projectbased learning assignments when continuous peer assessment is used to assess individual contributions ahead of other assessment models, such as teacher-only assessment of individual contributions to group work or the awarding of all team members the same grade. Students greatly prefer the individualised assessment of their assignment contributions based on online SAPA rather than all team members being allocated the same mark.
We have found the use of continuous online SAPA throughout the duration of an assessment task to lead to a participatory student-centred assessment forum where reflective learning aids the
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
development of interpersonal, professional, cognitive and conflict management skills needed to filter and synthesise more efficiently the information necessary for working in teams. The mechanics of such a Self-and-Peer Continuous Assessment model (or SAPCA as we shall refer to it from now on) are described in detail below.
Designing Assignments for SAPCA SAPCA requires students to rate each other’s contributions and performance holistically and on a regular basis throughout the duration of a team assignment. In common with other online SAPA systems such as TeCTra (Raban & Litchfield, 2007), SAPCA aims to create a formative and summative assessment environment that encourages students to learn peer-assessing skills using quantitative ratings and qualitative comments. In order to give students a clear indication of what stages of the assessment task they are rating, it is best to specify project progress targets coinciding with rating periods. These targets may or may not be assessed, although, as is the often the case with lengthy assignments, we have found that students are more likely to contribute throughout the duration of a project if progress targets are assessed. Instructors must be careful, however, that the assessment loadings of periodic submissions, compared to the loading of the final submission, accurately reflect the relative importance attached to the process and the products of teamwork. What follows below is a description (Tucker, Fermelis, & Palmer, 2007) of how these periodic quantitative and qualitative assessments can be used to individualise contributions to the collaborative process.
Making Assessments and the Individualisation of Team Scores Uing SAPCA On completion of an assignment and after using SAPCA, each team is awarded a team mark by
instructors. The team mark is then individualised if there is evidence of unequal contributions by team members. The decision to individualise is predicated on the evidence within periodic SAPCA ratings, peer comments and feedback, on tutor feedback and any other information received by the unit co-ordinator. As noted elsewhere (Willey & Freeman, 2006), the recommended approach is to use an ‘extreme’ SAPCA rating range outside of a predetermined upper and lower band to signal the need for further investigation of team performance. Throughout a team assignment, students are required by SAPCA to make regular holistic ratings of their own and their team-mates’ contributions to process. Students are asked to award ratings holistically, taking into consideration whether each member attended meetings and tutorials, actively communicated with team-mates, participated in decision-making, completed work they were designated to the required standard and/or form, met deadlines and shared the workload. Cohorts are informed that making assessments is regarded as one indication of active participation in the assignment. Individual students make assessments by logging on to a password protected web-site accessed via an online study portal. They are given a time entry window to make their assessment. At any time during this window students are able to change their entries. The log-in page asks students to select an appropriate course code - in order to allow for the likelihood that students may be involved in multiple team assignments. Figure 1 below is a screen-shot of the SAPCA tool as students see it after having accessed the appropriate course code. As can be seen from Figure 1, students are asked to make three different types of assessment. The first measure asks them to award a holistic relative contribution score for each team member. This score must add up to the total number of members in the team. Thus, for example, if it is believed that all team members contributed evenly, the student awards everyone (including
175
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
themselves) a rating of 1. The intent of this first measure is to encourage students to consider the question of workload distribution. The first measure is complemented by a second asking students to rate the individual “performance” of all team members, using a drop-down menu, on a five point multiple-response Likert scale ranging from 1 for “Inadequate” to 5 for “Excellent.” Likert evaluation, which is commonly used to rate aspects of the group experience (Ellis & Hafner, 2005), allows for the coding of responses and the subsequent statistical analysis of possible patterns of bias. While the Likert evaluation aims to encourage students to consider the quality, as opposed to the quantity, of each other’s contributions, it is translated into a numeric value that is used in combination with the quantitative relative contribution assessment to produce a holistic rating of each member’s contribution. The combination of two modes of assessments makes less likely peer over-marking, which is a problem common to many peer assessment methods (Falchikov, 1986; Freeman & McKenzie,
Figure 1. SAPCA screenshot
176
2000). The purpose of the third qualitative measure, which elicits comments on the performance of their peers, is twofold; firstly, to elucidate for instructors ratings, anomalies and unexpected final evaluations and, secondly, to develop in students the evaluation, feedback and reflective skills that are key objectives of teamwork learning. As Dominick et al. (1997) have found, students who complete the qualitative feedback section, even if they themselves do not receive feedback in the forms of constructive or informative comments, might become motivated to improve their own performance. Figure 2 below is a screen-shot of the SAPCA model as students see it when accessing feedback. At the end of each periodic assessment, and at the conclusion of the team assignment, an assessment matrix is generated for each team that calculates for every student a multiplicative scaling factor (MSF). Importantly, before the calculation is made, all self assessment marks are removed from the matrix to negate the bias of self over-marking. The MSF is calculated as follows:
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
1.
2. 3. 4.
Individual Total Peer Assessment (ITPA): total of each team member’s Relative Contribution (RC*) plus Individual Performance (IP**) scores Team Total Peer Assessment (TTPA): total of all team members’ ITPA scores Team Mean Peer Assessment (TMPA): TTPA divided by the number of team members Multiplicative Scaling Factor (MSF) for each student: ITPA divided by TMPA
Where: * RC scores are restricted to between 0.5 and 1.5. ** IP scores are between 1 and 5. The MSF a student sees at the end of each periodic assessment, and then at the completion of the assignment, therefore indicates how peers’ ratings of that student compare to the mean rating for the team. As explained to students, if their rating is less than 1 they are considered to be performing below the average team performance. Equivalently, if their rating is greater than 1 they are considered to be performing higher than the average. Students are advised to aim for a rating of close to 1 or greater. Trials have indicated that
SAPCA ratings of between 0.8 and 1.2 are the norm (Tucker, 2008). A wide range in SAPCA student scores within any team triggers investigations into the evenness of student contributions. Thus, the online interface for instructors clearly displays a SAPCA range for each team (Figure 3), which is calculated by subtracting the lowest student rating from the highest and which can be interpreted as a measure of how great is the inequity between team member contributions. We have found that varying the text size of the figure representing the range in proportion to the range helps highlight those teams that require further investigation. Trials have indicated a range of greater than 0.3 to be an appropriate trigger for investigation. To facilitate this investigation, a hyperlink takes instructors from the SAPCA range to a list of team members (Figure 4), from which further hyperlinks lead to the qualitative comments made about each team member (Figure 5). The team list (Figure 4) lists five measures of performance for each team member; 1. 2. 3.
MSF (1) calculated without self ratings, MSF (2) calculated with self-ratings, Average contribution rating,
Figure 2. SAPCA feedback screenshot
177
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
Figure 3. SAPCA instructor feedback screenshot indicating ratings ranges
Figure 4. SAPCA instructor feedback team-list screenshot indicating five measures of performance; self (MSF1), self (MSF2), cont (average contribution), perf (average performance), and self-range (degree of self under or over-rating). Note. Names have been blanked out.
178
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
Figure 5. SAPCA instructor feedback detailed ratings screenshot including qualitative comments. Note. Names have been blanked out.
4. 5.
Average performance rating, The difference between MSF1 and MSF2.
MSF1 is the measure that is used for individualising grades for it disregards self over-rating. The fifth measures indicates the degree to which a student under or over-rates, for this can be important to know when instructors are making further investigations into team dynamics.
Forms of Student Participation We shall consider now two decisions facing instructors about to introduce some form of SAPA into a program. First, should participation in SAPA be compulsory or voluntary, and second, if participation is compulsory, how should that participation be enforced? The consequences of making participation in SAPA optional are best illustrated by the example of two studies. We
reported in 2006 on an early pilot of SAPA in an architectural design course (Tucker & Rollo, 2006). Students were offered three options of mark allocation; either by SAPA conducted via roundthe-table negotiation, by one-off paper-based anonymous SAPA or by opting out of SAPA and choosing instead to simply allocate marks evenly. Most teams selected the third option, but these teams were later identified as experiencing the most team conflicts, generally due to “free riding” or uneven contributions. In our 2007 study, it was possible to consider the consequences of compulsory versus optional participation in SAPCA by comparing participation rates across four cohorts. For three of the cohorts SAPCA was compulsory, but for the fourth, individuals (not teams) were able to opt out. Participation rates for each cohort were calculated by comparing the actual number of SAPCA ratings made to the total if all students had completed all ratings.
179
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
Participation rates for the three cohorts where SAPCA was compulsorily were 82%, 67%, and 57%. For the cohort given the choice of opting out of SAPCA the participation rate was only 25%. Reasons for the wide range of participation in the compulsory SAPCA units are evident when we consider the second question facing instructors; how should we enforce participation? Our experience suggests the only practical way of enforcing SAPCA participation in large cohorts is to allocate marks for the quantity of ratings made by each student rather than for the quality of their SAPCA participation. In small cohorts it may be possible to assess participation by assigning marks according to the quality of qualitative comments made, but this becomes unwieldy in large cohorts where a lengthy assignment may see thousands of qualitative comments recorded. There are two methods of awarding marks; either by rewarding participation or by penalising non-participation. We have compared these two methods across three cohorts. In two cohorts, students were rewarded a bonus equivalent to 2% of the assignment marks each time they completed the ratings. In the other cohort, students were penalised 2% of the assignment marks each time they failed to complete ratings. In the two cohorts that were rewarded for making ratings, the participation rates were 67% and 57%. In the cohort that was penalised for not submitting ratings, the participation rate was 82%. The message of these findings is clear; SAPCA must be compulsory and the compulsion to participate is far stronger when student marks are penalised for non participation.
Forms of Formative SAPCA Fedback It is possible when employing SAPCA to expose students to qualitative peer feedback only (peer comments), to quantitative feedback only (peer ratings), to both forms of feedback or to no peer feedback at all. Instructors should consider and compare the benefits of these different reflec-
180
tive formative feedback components in light of the pedagogic aims of the team assignments they have designed. In the interest of achieving an appropriate balance between formative and summative feedback, of encouraging reflection throughout a project (Dochy, et al., 1999; Freeman & McKenzie, 2002; McGourty, et al., 1998) and of enabling students to respond to feedback (Walker, 2001), we have found it best to expose students to at least one peer feedback component. As might be expected, students are less resistant to receiving, and more readily willing to give, quantitative over qualitative feedback. In the cohorts we have studied, there are a number of reasons for this preference. Firstly, in small teams there are problems of maintaining anonymity when qualitative peer feedback comments are made visible. For, although the orders in which scores and comments appear can and should be randomised, often students are still able to determine through elimination who has written what. As well as leading to conflict, this can discourage students from making accurate and honest assessments. Secondly, in multicultural cohorts, students with English as a second language may have problems making comments and interpreting feedback and they may also feel that their use of English might identify them to team-mates. Thus, we have found that such students make significantly fewer comments in multi-cultural teams than when in monocultural teams.
S Effectiveness S Transparency As the primary focus of assessment should be to encourage, direct and reinforce learning, then a primary aim of team assessment should be to enable the learning of teamwork skills. Assessment tasks devised for a course or unit involving teamwork should therefore not only explicitly reflect in scope and depth the stated objectives of
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
that course; they should also reflect the pedagogic objectives of team assignments in that course. It follows that assessment transparency should not only entail giving students a precise explanation of how a SAPA tool operates, but that students should also be made aware of the pedagogical intent of the tool and, in the case of research, of the research aims of testing that tool. Furthermore, as a result of our early studies into the teaching of group design projects (Tucker, 2008), we recommend that the teaching of teamwork skills be introduced into any curriculum where teamwork assignments are to be used. The importance of assessment transparency to instructors using our SAPCA tool led to the inclusion of presentations to students on the pedagogical intent of the model, as well as tutorial exercises that informed students’ abilities to assess and provide feedback on the work of others. These sub-skills were presented to the students as being prerequisite to effective teamwork, itself a core graduate skill or attribute.
Sftware and Procedural Refinements It is worth here touching upon refinements to our SAPCA tool introduced prior, during and after its testing with large cohorts. These included: 1. 2.
3. 4.
5.
Creation of a model team for student demonstration purposes; Students receiving a screen message indicating successful entry of ratings and comments; Students being permitted to edit ratings and comments within each time window; Instructors being able to quickly download complete listings of all student teams, of ranges of multiplicative scaling factors, of individualised rating scores and of comments by and about any team member; and Students being encouraged to regard their final entry as a global one (i.e., one that as-
sesses the duration of an assignment) that could then account for a weighted proportion of the MSF. This last refinement requires explanation. Many students complained that weekly displayed MSFs did not accurately reflect actual team-mate contributions. It was revealed that many viewed their early ratings as inaccurate because the ratings had been based on misplaced trust. Gearing-up periods for new teams can be lengthy when some team members have a poor work ethic or a lack of motivation. Communication between team members may in such cases be shallow; a problem exacerbated if team members previously did not know each other. In such situations, peers may have little evidence of how much a team member has contributed and therefore can be misled when rating. The likelihood of poorly performing team members deliberately misleading team-mates about contribution is increased when the contribution is being peer assessed. Team members’ misconceptions of team-mates’ contributions may not be revealed until it is too late to accurately reflect uneven contribution within SAPCA ratings. A weighted final global rating allows students the opportunity to redress such inequalities. The comparisons of weekly MSFs to global ratings can also indicate the dynamics of a team for instructors, which is especially useful when there has been conflict, or in the event of student appeals against their individualised assignment scores. Consistent communication between teammates and between students and instructors is essential. We have thus found it important to establish online discussion areas to allow students to ask and answer questions and for instructors to identify and monitor unexpected problems. In addition, it is helpful to add to the SAPCA tool a protocol that automatically emails students weekly reminders to make ratings. Finally, students appreciate being provided with information about early research findings from SAPCA piloting. This
181
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
information might include an indication of the number of students and entries made, the number of individuals whose scores were individualised and an overview of what has been learnt as a result of the students’ efforts. A detailed description of the technical software demands of establishing SAPA tools is beyond the scope of this chapter, for these demands and associated difficulties vary from institution to institution with different system architectures of online study environments. SAPA is used in a wide variety of applications, with the functional requirements of each implementation being determined by the particular combination of factors, including academic discipline, purpose of assessment, formality of assessment and number of assessments (Dochy, et al., 1999). While a wide range of online SAPA systems has been reported in the literature, all are premised on the requirements of their specific application, and are not automatically or even easily reused in a different assessment context. Additional factors which need to be considered include portability of software between differing operating systems, computer hardware setups, network configurations, database environments and institutional software approval processes. It has been observed that the development of an online peer assessment system for a single specific application may be relatively straightforward, but that extending the tool for more general use even in the same institution proves a significantly greater challenge (Mulder & Pearce, 2007). One final procedural recommendation, which clearly applies to all online SAPA tools in light of the large volumes of electronic data that can be generated, is the necessity for regular automated data backup. The implications of data loss are obvious; not the least being loss of student confidence in the tool due to software crashes, no matter how brief, for such confidence is prerequisite to the effective application of online tools.
182
Cae Study: Priies Ariiirom the Use of SAPCA in Large, Multiius, Multi-Cohhntexts Introduction and Nture of the Curse and Cohort Following the successful use and ongoing development of SAPCA with Architecture students over a period of three years, it was trialled with a much larger, multi-campus student cohort in 2007. The aim was to determine the tool’s robustness and adaptability and to continue its refinement within a markedly different and more complex teamwork context. A Business Communication course within a faculty of Business and Law was selected because its course convenor desired a computerised SAPA tool following the forced abandonment of a paper-based SAPA system of individualisation due to the increasing size and diversity of the student cohort. The trial cohort comprised a total of 1,835 students enrolled in semesters one (754 students) and two (1,081 students). Enrolments were dispersed - located at three different campuses within the state of Victoria, in off-campus mode within and beyond Australia, and at two partner campuses in Asia. Approximately 70% of the cohort comprised non-English speaking background (NESB) ‘International students’, mostly from wider Asia and the sub-continent. At any one time, up to fourteen different instructors were directly involved in teaching the course. The teamwork assignment comprised an oral and written report task worth 30 percent of final marks. Students worked in self-selected teams of four, with a total of 443 student teams completing the team assignment across the two semesters. Because of the university’s strict comparability of assessment policy, all students were required to undergo comparable teaching/learning experiences and identical assessment tasks. An added
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
challenge was therefore the need for SAPCA to be introduced and utilised in the same manner by all instructors for all enrolled students.
Preparation of S Documentation Incorporating SAPCA within this new and complex context demonstrated that the tool was sufficiently robust and revealed a number of additional principles for successful implementation. It was found there is a need for careful creation of clear SAPCA documentation with a focus on the student audience rather than on technology. For this trial, preliminary course overview materials contained a concise rationale for use of SAPCA in connection with the team assignment tasks with sufficient introductory information to meet the strategic and planning needs of all students, and the needs of off-campus and part-time students in particular. To facilitate error-free data entry by an instructor and the electronic creation of team sites within the SAPCA database, a team registration proforma was devised for newly formed teams to register their class number, unique team name, team members’ names, student ID numbers and online usernames. A clear and comprehensive document containing easy-to-follow instructions was then created to guide students’ SAPCA registration and the submission of their ratings. This document performed a vital role and required a convincing and motivating rationale for SAPCA and complete but non-excessive information, presenting helpful headings and subheadings, dot points, checklists and Help contact details. The course convenor then drafted one further document - a reminder email which included a précis of instructions, cross-referenced to complete documentation, and an Internet hotlink to the online SAPCA system. All documentation was uploaded onto the course study portal website for ongoing student access. An online discussion area was created in preparation for the team registration and the
first SAPCA rating period. An online frequently asked questions (FAQ) area was not provided but in hindsight this addition could have reduced the time spent by the course convenor in troubleshooting the more predictable problems experienced by individual students. A SAPCA online database was constructed and populated with student names, university identification number and university email prefix, as these were readily available within the greater university database. This was in readiness for the convenor to later key in student team details.
Nature of the Team Assignment and Tam Formation It is important for team assignments to be carefully planned and introduced. Students in the Business Communication course were prepared for their team experience by being made aware of the potential benefits of teamwork and provided with specific strategies and skills likely to facilitate a positive teamwork experience. A multi-channel approach was used to minimise the chance of any students being excluded from this preparation. Supportive content was included in lectures and reading materials, team member behaviours conducive to effective team assignment outcomes were discussed during tutorials, and a speed-meeting exercise was conducted to assist students to form their teams (Fermelis, 2006). Online meeting areas were created to allow absent and off-campus students to find each other and form teams. To create a sense of psychological identification and ownership after team formation, students were set simple team tasks designed to produce enjoyable and successful initial collaborative experiences (Fermelis, 2006), which they completed via face-to-face or in e-meetings. Teams collaborated to choose a team name and report topic, allocate specific team roles and complete their team proforma for the course convenor.
183
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
Introducing SAPCA and Training Saff No matter how strongly committed the course convenor, with large cohorts it is the class instructors who are entrusted with encouraging, motivating and inducting students into their SAPCA experience. It is therefore vital to ensure that all instructors are both committed and co-operative regarding the use of SAPCA as a pedagogic tool and for the individualisation of student assessment. In our trial, instructors were fully briefed on how to motivate students and assist them to register and commence making SAPCA ratings. Most fully embraced the concept of the individualisation of student results for the purpose of ‘fair’ student assessment in a team assignment situation, particularly when they had previously experienced students in some of ‘their’ class teams suffer great distress at the disappointing contributions by team mates who failed to complete tasks on time, failed to attend team meetings or failed to remain in contact. Other instructors required greater persuasion, believing that it was part of the team assignment experience for students to resolve their own team problems, and that SAPCA explanations, demonstrations and registrations simply stole time away from other classroom activities. Whilst no instructor refused to communicate to their students that they were required to submit SAPCA entries, over time it became clear that some instructors encouraged their students more than other instructors. Unsurprisingly, there was evidence in the trial that more enthusiastic instructors seemed to achieve greater success in inspiring students to participate than did less committed and less co-operative instructors.
Introducing SAPCA and Inducting Sudents The introduction of any form of SAPA to a large, dispersed and diverse student cohort of varying English speaking abilities and course participa-
184
tion habits requires careful planning and clear, consistent and effectively presented information. A multi-channel approach is more likely to achieve success than reliance on a single communication channel. Thus, once teams had been formed and SAPCA introduced, the detailed instructions and team registration proforma were explained via individual email messages, online postings, and lecture demonstrations. Because the unit was not available to first year students, induction and training were not complicated by issues of transition into university life. Although timetabling clashes, distances between campuses, and geographic dispersion of off-campus and off-shore students prevented the course convenor from personally inducting all students, the convenor ensured that the SAPCA access, registration and rating process was demonstrated with one on-campus tutorial group for each separate instructor. Instructors on other campuses were emailed documentation and trained by the course convenor by telephone. Off-campus and off-shore students were sent both e-copies and hard copies of SAPCA instructions, which appeared to introduce a more personal and thoughtful dimension for these cohorts. Off-campus students followed these instructions to work through the process themselves with support provided via the SAPCA online discussion site, email and telephone. During the first semester a range of minor problems were encountered, leading to refinements of the tool and documentation which greatly reduced the number of problems experienced in semester two.
Managing SAPCA throughout the Tam Assignment Once students had successfully registered and personalised their password, they were obliged to make a series of seven (semester one) or five (semester two) weekly SAPCA entries in relation to their team experiences. Ongoing instructor enthusiasm, support and encouragement were required throughout this period, not only to main-
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
tain student motivation and compliance but also to assist them with problems and improve teamwork learning outcomes. Although only a very small percentage of students experienced problems, in a large cohort this still meant a significant number of students needed individualised assistance. Some students were physically or electronically ‘absent’ during team formation and/or SAPCA induction. Other students had enrolled late and so were not included within the SAPCA database. Others wanted to join existing teams, form new teams, change teams or discontinue the course. Many students were unable to complete some entries because of absence due to illness, work obligations or holidays; others suffered software or hardware failure. Within one particular period an unexpected server failure prevented a large number of students from submitting their entries. Remembering to submit SAPCA entries was another common problem for many students, so email reminder messages complete with SAPCA hotlink were sent out to all enrolled students immediately prior to the opening of each time entry window. The fast identification, resolution and prevention of problems were key to the management of SAPCA and maintaining student enthusiasm throughout the life of their team assignment.
Using SAPCA During the Team Assignment Notwithstanding the development and refinements which accompanied implementing SAPCA, students appeared to embrace the opportunity to provide feedback via the tool as an ongoing contribution to their own assessment and that of their team mates. They embraced the concept of ‘fair’ assessment, with individualisation as a response to the quantity and quality of their contributions, with only two complaints being received. The first student was quietened when confronted with the consistently low ratings and somewhat negative comments made by his team mates. The other
was from a highly motivated off-campus student who felt that the individualisation had been too conservative. Students found the tool to be both intuitively sound and user friendly. In fact a small number of users found the tool so easy to use, that they were unsure whether or not they had successfully completed an entry. A feature was then added to clarify this that informed students of successful completion. An interesting observation was that off-campus students in particular appeared to relish the opportunity to provide often extensive feedback in relation to under-performing team-mates. This was in contrast to at least some on-campus students who actively self-censored their comments in order to maintain friendly face-to-face relationships.
Using SAPCA Data to Individualise Sudent Results As described above, the use of SAPCA is aimed at the equitable individualisation of student scores for team assignment tasks by including formative and summative assessment of the assignment process whilst maintaining validity and reliability. Although individualisation can be achieved by use of paper-based instruments, large student enrolments render manual individualisation methods unfeasible. In such situations online SAPCA is a tool that can quickly and efficiently identify teams with uneven contributions and enable the calculation of individualised team scores. In SAPCA, the sets of ratings entered by team members produce a numerical MSF for each team member. The tool also provides the instructor with a figure representing the range between highest and lowest MSF within each team. This range figure effectively represented the level of (in)equity within each team. In semester one of the trial, the instructors decided that a range greater than 0.3 was indicative of unequal team member contributions, so 72 out of 185 teams were identified as potentially requiring individualisation of team scores. This first stage of
185
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
the individualisation process was both fast and clear. In the second phase, each student member’s MSF, which reflected the quantity and quality of their contributions to the team task, was to be multiplied by their team score. For example, in a situation where the team assignment had scored 65% and one team member’s MSF was 1.1, their final individualised score would become 71.5%. Their team-mate’s MSF of 0.9 would then individualise that student’s score to 58.5%. The individualisation effectively redistributed scores within each team. In practice, this second phase was more problematic than had been anticipated, but for different reasons in the two semesters. In the first semester, although students had been advised that it was compulsory to submit all seven sets of SAPCA ratings, it became apparent that for some teams there were MSF inconsistencies due to large numbers of ‘missed’ assessment ratings by some team members. Even though a default rating had been devised for missed assessments, which was intended to marginally penalise miscreants, due to a software bug the default was found to actually slightly reward the miscreants whilst slightly penalising those who had diligently submitted ratings. It was deemed impossible to determine from MSFs alone which students had simply neglected to submit their SAPCA ratings and which students had been precluded by circumstances beyond their control. As it was deemed unfair to reward tardy students and penalise precluded students, the course convenor examined the qualitative comments submitted to SAPCA about each student. This qualitative data helped complete a picture of member’s efforts within their team, enabling individualisation to proceed manually but conservatively because of diminished reliability. Redistribution of a team’s marks typically resulted in one or two students’ scores being increased by 5-10% and one or two other students’ scores being reduced by 5-10%. The scores of 29 teams were individualised, however in some
186
teams data was insufficient to use for valid score redistribution. For teams with low SAPCA participation, this second phase was thus both frustrating and time-consuming for the course convenor. In the second trial semester, the default weekly ratings were adjusted to slightly advantage students who conscientiously made SAPA entries each week and disadvantage their less diligent team-mates. Unfortunately, at the beginning of this second semester the course convenor role was unexpectedly transferred to an instructor who insisted that SAPCA not be compulsory for the team assignment. Thus, although all students were inducted into SAPCA, they were instructed that SAPCA was optional. It was encouraging to find that approximately 25% of the 1081 students still submitted SAPCA entries, but this submission rate was deemed insufficient for the convenor to validly complete any individualisation of team scores. A theoretical criticism of SAPA, albeit one which appears to have no empirical support, is that students whose scores have been individualised down will complain or appeal, adding to instructor workloads. It was therefore gratifying to observe the almost universal acceptance by students, during the semester-one trial, of their individualised scores. It is suggested that this indication of high student satisfaction was because SAPCA had been explicitly introduced as a vehicle facilitating fair assessment in team assignments in relation to the quantity and quality of student contributions, rather than as a “punishment” for free riding team members.
Issues for Particular Sub-Groups of Sudents As part of the trial of SAPCA with architecture and business communication undergraduate students, participants were offered the opportunity to complete an entry and/or exit questionnaire. The questionnaires included twenty items and were completed by 490 students aged between 18 and 40
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
years with a mean age of 20. It is beyond the scope of this chapter to delve in great statistical detail into the differential experiences and perceptions of SAPCA by different sub-groupings of students. Nonetheless, it is interesting to consider particular findings in relation to general students’ attitudes towards SAPCA as well as make particular observations in relation to the SAPCA experiences and perceptions of specific sub-cohorts. The literature (Garbett, 2004; Hart & Stone, 2002; Monk-Turner & Payne, 2005; Morris & Hayes, 1997), our survey data and anecdotal evidence from students and instructors involved in this SAPCA trial consistently indicate that what students dislike most about team assignments is their perception of unfairness and inequity when all team members are awarded the same score. Our pilot survey promisingly revealed that students hold more positive views of group work after having used SAPCA. There was a statistically significant increase in positive views towards SAPCA from the beginning of semester (M 2.54 SD .39) to the end of semester (M 2.60 SD .43) (t (287) = 1.91, p<.05). However, the eta squared statistic (.01) indicated a small effect size. This finding is line with the improvement reported by instructors in student satisfaction and class spirit using the online SAPCA tool during team assignments in comparison to that experienced in equivalent past assignments that utilised no or more rudimentary forms of peer assessment. Increased maturity and confidence in many students as the assignments progressed was also apparent. Numerous students reported that SAPCA provided a “pressure valve” throughout assignments that allowed teams to function harmoniously despite unequal levels of skill and contributions. The SAPCA model may be seen to have allowed students to become more tolerant of the differential learning and assessment aspirations of their peers. Consequently our online SAPCA tool seems to have changed for the better the group dynamics seen in teams collaborating in the Architecture and Business Communication courses under study.
Online and face-to-face discussions with all student sub-cohorts consistently indicated that they were enthusiastic about the equity and fairness of individualised scores for their team assignment task, particularly if they aspired to high scores or had previously felt disadvantaged by their team assignment experiences. Nonetheless, many semester-one students asked whether or not SAPCA was compulsory, suggesting that they would not submit SAPCA ratings and comments unless they were to be penalised for non-compliance. During semester two, approximately 25% of the second semester students submitted entries after having received instructions that SAPCA was optional. This suggests that students’ desire to minimise learning and assessment activities may be stronger than their belief in the importance of fair and equitable team assignment scores. The survey results indicate that NESB students may have experienced difficulties with participating in the team assignment task and with composing the qualitative comments about their team-mates’ contributions. There was an approximately even split in the questionnaire data between native English speakers (266, 54.3%) and NESB students (224, 45.7%). Table 1 illustrates the findings. NESB students were less positive about teamwork than those with English as their first language. An independent sample t-test was conducted to compare selfand-peer assessment attitude scores for students with English as their first and second language. There was a trend for NESB students to hold more negative attitudes towards self-and-peer assessment compared to students with English as their first language. Many English speaking students were observed to relish the opportunity to candidly and at great length vent their frustrations about their underperforming team-mates. In contrast, International, NESB and overseas-born students generally made their comments cautious and polite. Some appeared unsure of the notion of individualisation of scores, which could reflect
187
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
a collectivist perception of team loyalty. Many seemed to submit stereotypical and generic remarks, possibly reflecting lack of experience in how to give positive feedback and/or limited vocabulary, and suggesting a need for additional training. Interestingly, a small number of NESB students submitted bland early comments but used their final entries as an opportunity to admonish their recalcitrant and lazy team-mates for having let down their teams. Even though NESB students were less positive about teamwork than those whose first language was English, International students (a sub-group who may or may not have been native English speakers) were found to be more positive than Domestic students about group work (refer to Table 1).This conundrum is in need of further research to better understand the precise nature of the mechanisms at work. Differences in team formation preferences were also found to exist between the Domestic and International cohorts, with the former finding it more important than International students to choose their own team members rather than having them allocated by instructors. Further research might be able to illuminate if the factors behind this are related to collectivist versus individualist attitudes, or
differential attitudes toward authority. Alternatively, the finding may relate to previous experiences in team assignments. Our study revealed that previous experiences of working in groups influenced, as might be expected, overall views of team work. For there was a significant difference between students who perceived previous teams as functional and those who perceived their previous teams as dysfunctional on attitudes towards team work in general. There were some interesting qualitative differences observed with the SAPCA experiences of on-campus students in comparison to off-shore student sub-cohorts or those studying in offcampus or distance mode. On-campus students appeared to experience significant problems with the anonymity and frankness of their SAPCA comments. During induction, Business Communication students had been instructed to provide positive and constructive but frank feedback to their team-mates. Off-campus Business Communication students were so compliant with this instruction that the SAPCA IT support consultant independently commented on the frankness of their entries compared to those of the on-campus Architecture and Building students. In contrast, there was anecdotal evidence that the comments of
Table 1. Participant attitudes towards teamwork using SAPCA Question Attitudes towards teamwork Attitudes towards SAPA Attitudes towards teamwork Importance of choosing team mates
Attitudes towards teamwork
188
English Speaking students
NESB
M 2.44 SD .537
M 2.73 SD .56
M 2.64, SD 0.374 Domestic Students
M 2.71, SD 0.43 International Students
M 2.74, SD .57
M 2.42, SD .52
M 2.01, SD 0.97
M 2.21, SD 0.93
Prior teamwork experience functional
Prior teamwork experience dysfunctional
M 2.49, SD .49
M 3.01, SD .56
Significance F (1, 464) = 1.754, p = 0.001; partial eta squared = 0.065 t(364) =1.903, p =.058.
F (1, 464) = 11.496, p = 0.001; partial eta squared = 0.075 F (1, 464) = 4.831, p = 0.02; partial eta squared = 0.01.
F (1, 410) = 92.70, p = 0.001; partial eta squared = 0.184.
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
on-campus students may not have always reflected students’ true feelings. Individual students from some on-campus teams, for which contribution was rated as “even” by SAPCA, confided in the course instructors that they had been reluctant to be frank in SAPCA because this would have created embarrassment for them in later team meetings, tutorial classes or incidental encounters. Off-campus students apparently were immune to this problem, presumably because they were unlikely to ever meet their team-mates again.
Facilitating Development of Student Sills A major rationale for team assignments within higher education is that they foster reflective student learning and the development of the teamwork skills valued by employers and thereby facilitate lifelong learning. Within this Business Communication case study there is consistent evidence of the first two types of learning, which would imply that lifelong learning may later follow. The team assignment task unquestionably increased students’ awareness of interpersonal communication and teamwork. They were formally introduced into teamwork skills, roles and strategies, but in order to satisfactorily complete their team tasks students were compelled to interact with their team-mates and were unable to avoid practising and therefore developing their teamwork skills throughout the assignment period. Students had been provided with a meta-language and framework for discussing interpersonal and team communication, which they were required to utilise during personal journals maintained throughout this period. On these journal entries they were separately and individually assessed and also examined. Making analytical journal entries for an instructor assessor audience should thus be seen to have complemented SAPCA. The concepts and terminology of interpersonal communication were rarely included within the SAPCA comments, but then again the latter were
created to be read by team-mates rather than by instructors. By making regular SAPCA entries, students were both engaging with the assignment task and reflecting on their own contributions, as well as those of their team-mates, on a regular basis throughout the life of the team. It is unclear at the present time whether and to what extent the MSF scores and published peer comments may have impacted on their learning experiences, although it is undeniable that these assessments provided an impetus for at least some students to improve their contributions. Similarly, it is not possible to claim that students’ exposure to a team assignment and to SAPCA will facilitate lifelong learning. However, it would be difficult to argue that students’ team experiences within this course, in which SAPCA played a major role, failed to assist development of awareness, self-reflection, conceptual frameworks, skills and strategies.
Future Directii An enduring message from the literature is the need to expand the comparatively meagre body of research related to the impact of participation on SAPA activities on students’ final unit results (Dochy, et al., 1999; Hanrahan & Isaacs, 2001; Strauss & Alice, 2007). For instance, it has already been noted that the balance between detailed rating criteria versus global/holistic ratings has an impact on the amount of qualitative feedback provided by raters, and hence, an impact on the balance of formative and summative effects of SAPA; further research into this assessment trade-off effect is suggested (Miller, 2003). While there has been a significant “open source” movement in the development of SAPA software tools, with many systems based on freely available database and web server technologies, for the reasons outlined in the section ‘Software and Procedural Refinements’, the reality is that free access to SAPA system software source code
189
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
is not the same as free access to a working SAPA system that is tailored to the specific needs of a particular application. A more recent development in the open source arena are “social software” applications, such as blogs, wikis and social networking sites, which have as their essence the collaborative collection, ordering and interpretation of user-created content. These software systems provide new tools that are inherently aligned to the collaborative and critically reflective activities underpinning SAPA. They offer an emerging and interesting option for the development of the next generation of SAPA software tools. A number of questions have arisen from our study requiring further investigation because their consideration is beyond the scope of the data generated. These can be conveniently summarised as follows: 1.
2.
3.
4.
What is the impact of ratings and comments on student contributions to team assignments? Can different attitudes towards SAPCA and teamwork in general, between Domestic and International students and between International students of different nationalities, be attributed to the reported communal versus independent learning styles of Western versus Eastern cultures? Are there differences between males and females in their ability and willingness to evaluate their peers? Do NESB students in multilingual cohorts experience language barriers as restrictive to using SAPCA?
What We Would Like to do Differently in the Future The testing of the SAPCA tool with Business Communication students proved it to be sufficiently robust and adaptable to work with a very large and complex cohort and for a very different type of team assignment devised as part of a very dif-
190
ferent course, compared to the architectural design assignments for which the tool was originally devised. As mentioned earlier, several enhancements were made to the SAPCA tool before, during and after the time it was tested. A number of these refinements are worth stressing: 1.
2.
3.
4.
5.
6.
Make SAPCA mandatory for the team assignment with bonus points for compliance, and simultaneously configure a default for missed submissions that slightly disadvantages the miscreants; Make the MSF visible to students throughout the assignment period, enabling under-performing team-members to become aware of how they were being perceived by their team-mates, and thus providing an opportunity for them to self-reflect and improve their participation; Make comments visible to the instructor/assessor only, to be referred to in the case of student complaints or appeals regarding their individualised scores. Introducing clarity to the identity of the reader audience ensures confidentiality and improves comment frankness and accuracy; Automate the email reminders to be sent electronically to students at the commencement of the ratings period; Extend each ratings period to maximise the number of SAPCA entries made and reduce the number of student messages sent to the course convenor in relation to missed submissions; Include a FAQ site within the course SAPCA online discussion site to enable students to quickly self-access solutions to predictable problems.
One final suggestion for improvement relates to curriculum decisions. The institution should have a policy of encouraging all courses with team assignments to make use of a SAPA tool for the individualisation of team assignments where
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
the assignment objectives are reinforced (rather than compromised) with individualisation. This would increase student compliance and reduce any reluctance to make SAPA entries. Such a policy would also gradually create a pool of experienced students who could help induct and train “naïve” SAPA users. It would also guarantee instructor commitment to and support of the SAPA tool, and reduce the burden of instructor induction. In the Architecture and Building school that has trialled and developed SAPCA over three years, the tool has been used repeatedly in multiple courses so that students now see it as a routine and necessary aspect of their teamwork projects.
Ci In line with other studies on peer assessment, we have found SAPCA to promote independent, reflective, critical learning, to enhance in students the motivation for participation and to encourage students to take responsibility for their learning. Moreover, online SAPA systems have been be found to solve problems of confidentiality and improve assessment efficiency. The findings of our pilot studies therefore support the positive contribution of online self-and-peer assessment within student group-based assignments. Despite the size of the pilot classes, team assignment scores were available to students far earlier than in previous years because accessing peer assessment data via online SAPCA required substantially less time than the collation of individual paper-based assessments. It is encouraging that the acceptance of individualised team assignment scores was almost universal within the two experimental cohorts. Indeed, not one student made a complaint about the individualisation of their mark within Architecture. Within the Business Communication cohort, only two teams registered complaints and these were quickly resolved. Of course, with increasing academic workloads leaving little time for either manual
paper-based or more elaborate self-and-peer assessment methods, it is more important than ever that it is easy both for students to make ratings and for instructors to quickly access ratings, MSFs and final individualised team results.
ACKNOWLEDGMENT The authors would like to acknowledge the input of Catherine Reynolds of the School of Behavioural Science at Melbourne University who has been a meticulous and diligent research assistant for the duration of our research. They would also like to acknowledge the invaluable help of their Information Technology consultant Russell Greenwood, who played a very significant role in designing and implementing the SAPCA software.
REFERENCES Ballantyne, R., Hughes, K., & Mylonas, A. (2002). Developing procedures for implementing peer assessment in large classes using an action research process. Assessment & Evaluation in Higher Education, 27(5), 427-441. Barber, P. (2004). Developing and assessing group design work: A case study. Paper presented at the 2nd International Engineering and Product Design Education Conference, Delft. Brown, R. W. (1995, 1-4 November). Autorating: Getting individual marks from team marks and enhancing teamwork. Paper presented at the 25th Annual Frontiers in Education Conference, Atlanta, Georgia. Clark, N., Davies, P., & Skeers, R. (2005). Self and peer assessment in software engineering projects. Paper presented at the Proceedings of the 7th Australasian conference on Computing education - Volume 42, Newcastle, New South Wales, Australia.
191
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups. Review of Educational Research, 64(1), 1-35. Davies, P. (2000). Computerized peer assessment. Innovations in Education and Teaching International, 37(4), 346-355. Dochy, F., Segers, M., & Sluijsmans, D. M. A. (1999). The use of self-, peer and co-assessment in higher education: A review. Studies in Higher Education, 24(3), 331-350. Dominick, P. G., Reilly, R. R., & McGourty, J. (1997). Incorporating student peer review and feedback into the assessment process. Paper presented at the Best Assessment Processes in Engineering Education: A Working Symposium, Terre Haute, Indiana. Ellis, T. J., & Hafner, W. (2005). Peer evaluations of collaborative learning experiences conveyed through an asynchronous learning network. Paper presented at the Conference Proceedings Hawaii International Conference on System Sciences (HICSS-38), Big Island, Hawaii. Falchikov, N. (1986). Product comparisons and process benefits of collaborative peer group and self-assessment. Assessment & Evaluation in Higher Education, 11(2), 146-166. Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287-322. Fermelis, J. (2006). Making teamwork work. Paper presented at the 71st Association for Business Communication Annual Convention. Freeman, M., & McKenzie, J. (2000, 2-5 July). Self and peer assessment of student teamwork: Designing, implementing and evaluating spark, a confidential, web based system. Paper presented at the ASET-HERDSA 2000 Conference, Toowoomba, Qld.
192
Freeman, M., & McKenzie, J. (2002). Spark, a confidential web-based template for self and peer assessment of student teamwork: Benefits of evaluating across different subjects. British Journal of Educational Technology, 33(5), 551-569. Garbett, D. (2004). Researching practice as a teacher educator. Retrieved from http://www. aare.edu.au/04pap/gar04590.pdf. Gokhale, A. (1995). Collaborative learning enhances critical thinking. Journal of Technology Education, 7(1), 22-30. Goldfinch, J., & Raeside, R. (1990). Development of a peer assessment technique for obtaining individual marks on a group project. Assessment and Evaluation in Higher Education, 15(3), 210-231. Hanrahan, S. J., & Isaacs, G. (2001). Assessing self- and peer-assessment: The students’ views. Higher Education Research & Development, 20(1), 53-70. Hart, G., & Stone, T. (2002). Conversations with students: The outcomes of focus groups with QUT students. Retrieved from http://www.ecu.edu. au/conferences/herdsa/main/papers/nonref/title. html James, R., McInnis, C., & Devlin, M. (2002). Assessing learning in Australian universities. Melbourne: Centre for the Study of Higher Education and The Australian Universities Teaching Committee. Johnston, L., & Miles, L. (2004). Assessing contributions to group assignments. Assessment & Evaluation in Higher Education, 29(6), 751768. Kravitz, D. A., & Martin, B. (1986). Ringelmann rediscovered: The original article. Journal of Personality and Social Psychology, 50(5), 936-941. Lejk, M., & Wyvill, M. (2002). Peer assessment of contributions to a group project: Student attitudes to holistic and category-based approaches.
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
Assessment & Evaluation in Higher Education, 27(6), 569-577.
Society for Computers in Learning in Tertiary Education, Sydney.
Lin, S. S. J., Liu, E. Z. F., & Yuan, S. M. (2001). Web-based peer assessment: Feedback for students with various thinking-styles. Journal of Computer Assisted Learning, 17, 420-432.
Raban, R., & Litchfield, A. (2007). Supporting peer assessment of individual contributions in groupwork. Australasian Journal of Educational Technology, 23(1), 34-47.
Luca, J., & McLoughlin, C. (2002, 8-11 December). A question of balance: Using self and peer assessment effectively in teamwork. Paper presented at the 19th Annual Conference of the Australian Society for Computers in Tertiary Education, Auckland.
Resta, P., & Laferrière, T. (2007). Technology in support of collaborative learning. Educational Psychology Review, 19(1), 65-83.
McGourty, J., Dominick, P., & Reilly, R. R. (1998, 4-7 November). Incorporating student peer review and feedback into the assessment process. Paper presented at the 28th Annual Frontiers in Education Conference, Tempe, Arizona. Miller, P. J. (2003). The effect of scoring criteria specificity on peer and self-assessment. Assessment & Evaluation in Higher Education, 28(4), 383-394. Monk-Turner, E., & Payne, B. (2005). Addressing issues in group work in the classroom. Journal of Criminal Justice Education, 16(1), 166-179. Morris, R., & Hayes, C. (1997). Small group work: Are group assignments a legitimate form of assessment? Proceedings of the 6th Annual Teaching and Learning Forum. Retrieved from http://lsn.curtin.edu.au/tlf/tlf1997morris.html
Sharp, S. (2006). Deriving individual student marks from a tutor’s assessment of group work. Assessment & Evaluation in Higher Education, 31(3), 329-343. Sitthiworachart, J., & Joy, M. (2003, 9-11 July). Web-based peer assessment in learning computer programming. Paper presented at the 3rd IEEE International Conference in Advanced Learning Technologies, Athens, Greece. Sivan, A. (2000). The implementation of peer assessment: An action research approach. Assessment in Education, 7(2), 193-213. Sluijsmans, D. M. A., Dochy, F., & Moerkerke, G. (1999). Creating a learning environment by using self-, peer- and co-assessment. Learning Environments Research, 1(3), 293-319. Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19(1), 69-75. Strauss, P., & Alice, U. (2007). Group assessments: Dilemmas facing lecturers in multicultural tertiary classrooms. Higher Education Research & Development, 26(2), 147-161.
Mulder, R. A., & Pearce, J. M. (2007). Praze: Innovating teaching through online peer review. In R. Atkinson, C. McBeath, A. S. S. Kit, & C. Cheers (Eds.), 24th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (pp. 727-736). Singapore: ASCILITE
Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249-276.
Raban, R., & Litchfield, A. (2006, 3-6 December). Supporting peer assessment of individual contributions in groupwork. Paper presented at the 23rd Annual Conference of the Australasian
Tucker, R. (2008). The impact of assessment modes on collaborative group design projects. In S. Frankland (Ed.), Enhancing teaching and learning through assessment: Embedded strate-
193
Designing, Implementing and Evaluating a Self-and-Peer Assessment Tool for E-Learning Environments
gies and their impacts, 2, 72-85. Hong Kong: The Assessment Resource Centre, The Hong Kong Polytechnic University. Tucker, R., Fermelis, J., & Palmer, S. (2007). Online self-and-peer-assessment for teamwork in architecture and business communications. In J. Coulson, D. Schwede & R. Tucker (Eds.), Towards solutions for a liveable future: Progress, practice, performance, people: The 41st annual conference of the Architectural Science Association ANZAScA (pp. 264-271). Geelong: Deakin University. Tucker, R., & Reynolds, C. (2006). The impact of teaching models on collaborative learning in the student design studio. Journal for Education in the Built Environment, 1(2), 39-56. Tucker, R., & Rollo, J. (2006). Teaching and learning in collaborative group design projects. Journal of Architectural Engineering & Design
194
Management, 2 (Teaching and Learning in the Built Environment), 19-30. van den Berg, I., Admiraal, W., & Pilot, A. (2006). Peer assessment in university teaching: Evaluating seven course designs. Assessment & Evaluation in Higher Education, 31(1), 19-36. Walker, A. (2001). British psychology students’ perceptions of group-work and peer. Psychology Learning and Teaching, 1(1), 28-36. Wen, M. L., & Tsai, C.-C. (2008). Online peer assessment in an inservice science and mathematics teacher education course. Teaching in Higher Education, 13(1), 55-67. Willey, K., & Freeman, M. (2006). Improving teamwork and engagement: The case for self and peer assessment. Retrieved 12 December, 2006, from http://www.aaee.com.au/journal/2006/willey0106.pdf
195
Chapter XI
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students Andrew Sanford Monash University, Australia Paul Lajbcygier Monash University, Australia Christine Spratt Royal Australian and New Zealand College of Psychiatrists, Australia
ABSTRACT A differential item functioning analysis is performed on a cohort of E-Learning students undertaking a unit in computational finance. The motivation for this analysis is to identify differential item functioning based on attributes of the student cohort that are unobserved. The authors find evidence that a model containing two distinct latent classes of students is preferred, and identify those examination items that display the greatest level of differential item functioning. On reviewing the attributes of the students in each of the latent classes, and the items and categories that mostly distinguish those classes, the authors conclude that the bias associated with the differential item functioning is related to the a priori background knowledge that students bring to the unit. Based on this analysis, they recommend changes in unit instruction and examination design so as to remove this bias.
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
INTRODUCTION The aim of this chapter is to discuss the identification of latent classes or groups within a cohort of E-Learning students. These latent classes are determined by the existence of differential item functioning, or item bias, experienced by students within these different classes.a In our illustrative case study, we are able to identify and interpret a number of these latent classes. Our thesis is that the differential item functioning (DIF), and the latent class structures identified, are a consequence of the students’ diverse educational backgrounds. The case study looks at a unit in computational finance where the students are taking either a single major in commerce or information technology, or double majors in both. We argue that given the multi-disciplinary nature of the unit, those taking a double major are advantaged in terms of background or a priori knowledge over those taking single degrees, resulting in the identified DIF. DIF analysis seeks to determine the existence of systematic differences in item responses among groups of student, the cause of which is some factor, or factors, other than the innate ability or proficiency of the students. The meaning of ‘innate’ ability is the student trait which the test items have been designed to measure. DIF analysis seeks to identify test items that discriminate amongst students based on factors other than their ability. Item Response Theory (IRT) modeling has a long association with educational and psychometric research, and has proven to be a popular method for detecting DIF.b Usually, in investigating DIF with IRT models, students are placed into groups based on the presence or absence of these observable, non-ability factors. Two IRT models are then estimated for each of the groups and their parameters are checked to see whether they are significantly different. If they are, then DIF is considered to exist. The DIF analysis discussed in this chapter uses examination items and student responses
196
from a unit in computational finance, which has been taught by one of the authors for many years.c Materials in this unit have been designed to suit online E-Learning, and all assessment has been prepared in a multiple choice format, appropriate for automated delivery and scoring. The unit attracts students from a diverse range of educational and cultural backgrounds, and thus provides a ready number of observed factors (e.g. gender, major area of study, years of academic attainment, international status, ethnic background, etc.) which can be used to put students into groups. Although using observed factors is common, it might not be valid in all circumstances. For example, DIF may be due to factors, such as a student’s level of motivation, learning intentions, language deficiencies, anxiety or problem solving strategies, which are not readily observable. An alternative to specifying the student class membership prior to carrying out the DIF analysis is to infer the student membership as an output of the DIF analysis. In our case study, a predefined number of latent classes are specified within the IRT model, and students are allocated to those classes based on their test item responses. Within the case study, DIF analysis is carried out using a polytomous IRT model previously developed by Bolt, Cohen and Wollack (2001).d A valuable output common to most IRT models, and one which provides a useful visual display of DIF, are the Item Response Functions (IRFs) or Item Category Characteristic Curves (ICCCs). These functions display the probabilities associated with the selection of each item and/or category as a function of a student’s ability (or proficiency) level. e The correct category response usually records the highest probability. We reproduce a number of ICCC functions to illustrate the differences in response probabilities for the different latent classes of student. (See Figure 2 and Figure 3.) In the following sections, we review the current IRT and assessment literature; discuss features of the computational finance unit, and the student response data. We then discuss in greater detail
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
the polytomous IRT model and the methodology used to estimate and compare models. Finally, the results of the DIF analysis are presented and discussed, and recommendations made toward E-Learning assessment.
Li There is an expansive assessment and measurement literature in education, educational psychology and psychometrics, and substantial evidence that students in higher education direct their learning to meet the demands of assessment (Gibbs & Simpson, 2004; Struyven, Dochy, & Janssens, 2003). Escalating class sizes and internationalization of the higher education sector has contributed to a broader academic interest in not only assessment in the E-Learning context, but more generally, all aspects of assessment including peer assessment, summative assessment, and the role of feedback in assessment as well as a growing literature in online E-Learning assessment (Boud, Cohen, & Sampson, 1999; Brown, Race, & Bull, 1999; Carless, 2006; Falchikov, 2005; Hounsell, 2003; Huba & Freed, 2000; Knight, 2002; Nicola & Macfarlane-Dick, 2006). There has also been increasing interest in constructivist approaches to learning and assessment, which are inherently qualitative (James & Fleming, 2004; Struyven, Dochy, & Janssens, 2003), whilst at the same time governments and external stakeholders (for example business and industry groups), demand quantitative evidence of validity and reliability in the assessment practices that prepare graduates. From our interest in E-Learning technologies and assessment, this case study provides a demonstration of the identification of latent classes and DIF between those classes in a particular subject, Computational Finance. While multiple choice question examinations may be considered mainstream in some disciplines and are particularly suited to an E-Learning environ-
ment, we believe that post-test analysis is underresearched, and has the potential to contribute to the improvement of teaching and instruction in an E-Learning environment. For teaching and instruction more generally, Shepherd (2001, p. 1) has argued that historically, in educational programs that prepare teachers, ‘few connections were made […] to suggest ways that testing might be used instructionally’ [our italics]. While there is an extensive literature on IRT and psychometrics, there seems limited research that explores latent class analysis in the context of assessment data in E-Learning in the higher education sector. IRT and psychometric methods incorporated into the E-Learning context provide an opportunity for automating the diagnostic analysis of student performance. In this regard, Katz, Martinez, Sheehan, & Tatsuoka (1998, p. 257) were interested in finding strategies that extended ‘beyond raw scores or one-dimensional IRT estimates of overall proficiency’ in considering assessment practices in architecture programs. While they do not use latent class analysis per se, their work with the ‘Rule Space Methodology’, illustrates the expanding interest in testing models that have the potential to be diagnostic and that might suit complex design-based disciplines like architecture insofar as it “yields diagnostic information that could assist the examinee in pinpointing areas of non-mastery. Like latent state models, Rule Space diagnoses examine ability by inferring unobservable mastery states from observable responses” (Katz, Martinez, Sheehan, & Tatsuoka, 1998, p. 274). Brown (2000) used latent class analysis in a project that involved primary school children. He reviewed the literature of assessment and performance standards in school education in the United States and established that the setting of performance standards has generally been done by using ‘judgmental approaches’ in which ‘human raters, presumably experts, make judgments or decisions about where the standard should be set’ (Brown, 2000, p.3). Brown (2000) presents
197
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
latent class analysis as an empirical method that “does not presume that a continuous trait underlies performance but rather that groups or classes differ qualitatively from one another and that these differences account for all the relationships in the data” (p. 8). His paper proceeds to describe the study: a latent class analysis of responses from 191 primary school students on two performance tests related to probability. Further he compares his results to existing ‘judgmental data’ on the same tests and presents comparable results. For us the key finding from Brown’s comprehensive literature review and his subsequent study is the potential of latent class analysis to provide evidence-based input to curriculum and assessment decisions, and to be complementary to other more judgmental approaches. Gitomer & Yamamoto (1991) argued that while IRT is well–researched and well-accepted in the educational measurement community, it is essentially limited to those settings where “representing items and ordering individuals on a one-dimensional continuum of proficiency” (p.173) is required. In many settings for example, when testing is diagnostic, more detailed information about the learner is required. They proceed to suggest that “psychometric models that are sensitive to the qualitative nature of individual performance” (Gitomer & Yamamoto, 1991, p. 174) are required. While they acknowledge latent class modeling as a useful means of developing more qualitative measures of individual performance, their paper describes a HYBRID model which employs latent classes and IRT. The key question for them was to determine “whether the application of the HYBRID model, with classes specified from an empirical and theoretical cognitive analysis, can significantly improve diagnostic assessment with respect to both psychometric and substantive considerations” (Gitomer & Yamamato, 1991, p. 181). It is this principle which informed our study, although the Gitomer & Yamamoto (1991) model was not employed. Like Gitomer & Yamamoto
198
(1991, p.185) however, we anticipate that latent class analysis will assist in providing an evidential basis to redevelop curricula and design remediation that is targeted to identified needs.
Ba In the next five sub-sections, we discuss the computational finance unit, the student cohort, the polytomous item response model, the observation data and the methodology.
The Unit Computational Finance is a multi-disciplinary elective unit offered to advanced undergraduate and postgraduate students in the School of Business Systems, Faculty of Information Technology, Monash University. The unit provides students with the opportunity to study a number of financial concepts and techniques associated with equities, bonds and derivative instruments and markets. Its focus is on the computational aspects of finance, integrating components from both statistics and IT. The statistical area covers basic background theory on random variables, probability distributions, expectations, standard deviations, covariance and correlation measures, whilst the important financial concepts of risk and uncertainty are also covered. The IT coverage focuses on the development of the students’ Excel and Visual Basic skills to a level where they can make operational a number of the financial techniques studied in the unit. The subject is taught in an E-Learning mode, or more accurately, a ‘Blended Learning’ approach. In the contemporary E-Learning literature in both higher education and corporate training settings, ‘blended learning’ has become ‘amorphous’ (Oliver & Trigwell, 2005) in that there is considerable debate about ‘blended learning’ as a unifying descriptor for particular forms of technologically supported teaching. Oliver & Trigwell (2005, p.
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
17) identified three common definitions of the term which they go on to critique; these are: 1. 2. 3.
the integrated combination of traditional with web-based approaches; the combination of media and tools employed in an E-Learning environment; and the combination of a number of pedagogic approaches, irrespective of learning technology.
Oliver & Trigwell (2005) suggest that the first of these is the most common interpretation. So, in this sense the students in this subject are E-Learning students. More precisely, the students were encouraged to use web-based software, known as WebCT. This tool has a number of functions: it acts as a repository for all unit materials (i.e. lecture slides, readings, past exam questions), it provides discussion groups, and it hosts chat-rooms. These E-Learning tools were used to foster deep learning: to build social presence in an online learning environment, to foster debate, and ultimately to use blended learning to develop tertiary students’ skills of critique in investment decision making (Lajbcygier and Spratt, 2007).
The Student Cohort The semester 1, 2004 cohort of students consisted of IT majors, commerce majors and students completing majors in both disciplines. Approximately half were international students, originating predominantly from China, SouthEast Asia and the subcontinent. About two-thirds were undergraduates, with the remainder being post-graduates, mainly completing the Masters of Business Systems by course work. Student motivation is assumed to be high, although we have no formal measures. This assumption is based on the fact that students are warned in their first lecture that the unit is difficult and those without some background in
IT, statistics or finance, are likely to find it very difficult. Following this announcement, it is not unusual to have an initial student withdrawal from the unit of about 10-20 percent.
The Mixture Nominal Response Model For the case study, we apply the mixture nominal response model (MNRM), previously developed by Bolt, Cohen and Wollack (2001), which allows for latent class effects. The model comprises two equations, the linear propensity function shown in equation: zgjk = λjk θ + ζ gjk
(1)
And the response equation expressed in equation:
Pgjk =
exp( z gjk )
∑
K h =1
exp( z gjk )
(2)
The linear propensity function describes the propensity with which a student, in class g, has toward selecting the k category response for an item j. The response equation normalizes this propensity and transforms it into a response probability measure. The response equation is a function of the ability or proficiency θ f It is the function previously described as the IRF or ICCC. Parameters in equation are the slope or “discrimination” parameter, λjk, and the intercept or “difficulty” parameter ξgjk. The λjk is the rate at which a change in ability, θ, affects the propensity of response, and is item/category specific and common across all latent classes.g The intercept parameter, ξgjk, captures non-ability factors that influence the propensity. The intercept is class/ item/category specific, and so varies between the latent classes g. In our DIF analysis, it is the variation in ξgjk parameter between groups that identifies DIF. For a given level of student ability θ, because the slope parameter λjk is common
199
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
across student groups, any difference in the intercept parameter ξgjk for each group translates into a different propensity. Figure 1 represents this graphically where two linear propensity functions have common slopes λjk but different intercepts ξgjk. To further illustrate, assume that we have a group of primary school students, we separate them into two groups where Group 1 contains students with high reading proficiency and Group 2 those with low reading proficiency. If the students were given a test to measure their ability to do simple arithmetic, and the test items used only English prose, then Class 1 may be assessed to have superior arithmetic skills than those in Class 2. If the items were presented using mathematical symbols that by-passed the comparative reading strengths of the two classes, then each group may then be shown to have comparable arithmetic abilities.
The Response Data The semester 1, 2004 final examination for the computational finance unit consisted of 79 mul-
tiple choice items. There were three main sections; a true/false section, a concept section and a problem solving section. Only the concept and problem solving sections comprised items that have five response categories, and it is for this reason that we concentrate on the items contained in just those two sections. This provides us with a data set of 50 responses per student. The starting cohort of students comprised a total set of 65. Of the original 65 students, only those students that responded to all of the 50 items were retained. The remaining students were then split into two approximately equal sets, of 29 and 28 students. The set of 29 we identify as the calibration set, and the remainder as the cross validation set. The purpose of the calibration set was to estimate the parameters of the 1- and 2Class models, whilst the cross validation set was used to assess the out-of-sample performance of the 1- and 2- Class modelsh,i.
Te Methodology The methodology does not deviate from Bolt, Cohen and Wollack (2001), who also use a
Figure 1. A graphical demonstration of the MNRM’s linear propensity equation. Note. The two lines represent the fit of the propensity equation to an arithmetic test item and its correct response category. The higher propensity line represents that for the class of student that has higher reading proficiency (Class 1), whilst the lower propensity line represents the class of students with the lower reading proficiency (Class 2) Propensity (z)
z1 jk = l jkq + z 1 jk
z2 jk = l jkq + z 2 jk
ability (θ)
200
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Bayesian approach.j Our DIF analysis requires the estimation of a 1-Class model and a 2-Class model, where all classes are latent. The 1-Class model is the base model and assumes that all students are members of a single latent class. The 1-Class model parameters are estimated, as well as the proficiency θ’s of each student. The proficiency θ estimates are retained to represent each student’s ability in the examination items, and used as fixed inputs to the 2-Class model, where two latent classes are assumed.k Both the 1-Class and 2-Class model parameters are estimated using the calibration data. To compare the 1-Class and 2-Class models, we use two methods. In the first, we calculate each model’s marginal log-likelihood using the cross-validation data, with the preferred model having the largest marginal log-likelihood. We also estimate the residual correlations between item categories for each model, where the preferred model has the lowest total residual correlations.l To evaluate DIF between the latent classes within the 2-Class model, a discriminator function Djk is used. The discriminator Djk calculates the expected or average difference in response probability between the two latent classes of students. It is effectively the expectation of the difference, taken with respect to the distribution of the proficiencies, θ’s.m
As the discriminator Djk operates at the item j and category k level, we also use a total discriminator TDj, to measure DIF at the item j level only. The total discriminator for item j is defined as the sum of the absolute values of Djk for each category k, TD j = ∑ | D jk |
(3)
k
Dicussii Estimates of the ability parameter values for both models are listed in Table 5 and Table 6 of Appendix B – Ability parameter estimates. The efficacy of these ability estimates is supported by the correspondence between the ordinal positions of each student based on their estimated ability and their overall performance in the computational finance unit. Summary statistics for the mean and precisions of the ability parameters in each of the two classes for the 2-Class model are documented in Table 1.n Class 2 shows a higher average ability level than class 1 students, although this does not indicate a ‘low-ability/high-ability’ distinction between classes. Class distinction, as already discussed, is based on the non ability related parameter ξgjk. In terms of actual membership of each of the classes, 36% of the calibration cohort belongs to the class 1 whilst the remainder below to class 2.
Table 1. Mean and precision parameter estimates for 2-class model Mean and precision of ability parameters for students in class 1 and class 2
Mean
Sd
MC error
0.025
Median
0.975
(µ1) (t1)
1.527 1.178
0.3655 0.5038
0.02302 0.01908
0.6395 0.3814
1.592 1.115
2.068 2.333
(µ2) (t2)
2.074 1.609
0.1981 0.5454
0.004572 0.0206
1.725 0.673
2.067 1.563
2.462 2.803
201
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Table 2. Discriminator values for selected items and categories. Note. The correct response category is marked by an asterisk
Items
1
2
2 13 21 25 39 46 48 50
0.00110 0.00292 0.33718 0.29335 -0.03987 0.00205 0.33712 -0.41086*
0.28712 0.23960 -0.08076 0.03673 -0.09750 0.32883 -0.30799* 0.08533
Djk Categories 3 -0.14601 -0.33786* -0.49188* -0.28533* -0.05672 -0.29868 -0.00445 0.26785
In comparing between the 1-Class and 2Class models, the log-Likelihood and residual correlations for each was calculated using the cross-validation response data. For the log-Likelihood, results were inconclusive with near identical values, 1146 and 1147 for the 1-Class model and 2-Class models respectively. For the correlation residuals calculations, the results were more supportive of a 2-Class model, with this model recording lower residual correlations between item categories. Given the support for the 2-Class model, we focus the remainder of our discussion on that model.
Differential Item Functioning To achieve insights into the potential sources of the latent class distinctions of the 2-Class model, we carry out a content analysis of those items identified as demonstrating DIF between the latent classes. This is achieved by reference to the ICCC outputs generated from the 2-Class model, for those items that displayed the greatest total discriminator measures, TDj. The TDj and Djk values are listed in Table 2. The positive Djk values in Table 2 indicate that, according to the estimates of the 2-Class model, class 1 students have a higher probability
202
TDj 4
5
-0.36893* -0.00143 -0.01781 -0.07445 0.00417 -0.09027 -0.05738 -0.00006
0.23928 -0.03671 0.02572 -0.04269 0.45079* -0.11729* -0.02262 0.25645
1.04243 0.61852 0.95335 0.73255 0.64905 0.83711 0.72957 1.02055
of selecting that category than students in class 2. The opposite is true for negative Djk values. Those categories marked with asterisks, indicating the correct response, are predominantly negative suggesting that class 2 students are more likely to select the correct response. Of all the fifty examination items, those items displaying the greatest differentiation, according to their TDj values, are 2, 50, 21, 46, 25, 48, 39 and 13. Each item is listed in Appendix A – Items and Categories. These items are derived from both the conceptual and problem solving sections of the exam. Items, 2, 13, 21, and 25 are drawn from the conceptual section and represent just over 10% of that section. The remaining items are from the problem solving section and represent 25% of the items in that section. This distinction may suggest that latent class membership is more determined by the ability of students to apply learned concepts, rather than to describe or distinguish between them. We now look at each of the listed items in detail, and identify the differences in category responses between classes 1 and 2 students using the ICCC’s generated. Item 2 tests a student’s knowledge of dividend discount models. It requires a student to respond with a category that best represents the distinction
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
between the constant growth dividend discount model and other discount models. Referring to Figure 2, the Item 2 ICCC for class 1 students shows an increasing preference for the correct response, category 4. We note that this response includes the term ‘factor’ in its description. In teaching the constant growth discount model, as per the title of the model, we distinguish dividends as growing by a constant growth ‘rate’, and not by a constant growth ‘factor’. In the authoring of the item, the conscious decision to avoid the term ‘constant rate’ and replace it with ‘constant factor’ may have introduced student confusion as to the meaning of the word ‘factor’. This may possibly explain why, over a considerable segment of the ability range for class 1 students, two other categories maintain relatively high response probabilities; the category 2 and 5 response. The category 2 response is true for all dividend discount models, but certainly not particular to the constant growth dividend discount model. The category 5 response, ‘none of the above’, is also relatively strong. This type of response category tends to be an efficient distractor for students who cannot make a strong commitment to any alternative category. Item 13 tests the student’s knowledge of the binominal option pricing model. To respond correctly, the student must know the distinction between European and American call and put options, be able to read an Excel spreadsheet setup of the binominal model, understand how to construct the terminal prices of the underlying equity asset, and apply the correct payoff equation for the specified option. Reviewing the ICCC for item 13 in Figure 2 class 1 students over a large range of ability favor the incorrect call option category response 2. Class 2 students on the other hand favor the correct put option category 3 response. The class 1 students have incorrectly specified implicitly the payoff equation for a call option, not a put option. This is a basic error. The call/put distinction is usually covered in most first year introductory finance courses, and is covered
in the computational finance unit for students with a less substantive financial background. An explanation for the error may be as innocuous as a simple lack of diligence on the part of class 1 students in reading the item. However, it may also show evidence of bias. Students who have not had the advantage of several iterations of these concepts in previous units may be more easily confused by them. This deficiency may be further exacerbated given that the unit instruction focuses on call options, effectively programming some students into an incorrect response. Item 21 tests the ability of a student to calculate the net operating working capital of a company. To respond correctly, the student must know the definition for net operating working capital and be able to identify the raw input values from the company balance sheet provided. The class 1 students showed a predominance to select category 1, whereas class 2 students showed the correct category 3 with high probability. This distinction may identify a lack of accounting sophistication on behalf of class 1 students. Apart from using an incorrect definition for net operating working capital, a naïve interpretation of ‘cashat-bank’, as being related to ‘capital’, may have been employed by the class 1 student to warrant a category 1 response. Once again, this class 1 response can be attributed to an inadequate finance background. Item 25 tests a student’s knowledge of financial ratios, particularly the return on equity. To respond correctly a student must be familiar with the concept of the debt to equity ratio, the relationship of basic earning power to earnings before interest and taxes, interest rates applied to debt, tax rates applied to earnings before taxes, and the calculation of net income. The dominant response for class 1 students was to select category 1. This response incorrectly applies total assets to the ratio, when total equity should be applied. Students in class 1 were able to apply the ratios but were incorrect in distinguishing the equity component from the asset component of
203
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
the business. This again is a basic distinction to which most first year accounting students would be familiar. Item 39 tests a student’s ability to calculate the share price of a firm using the constant growth dividend model. The ICCCs for Item 39 in Figure 3 present an anomaly. Unlike the items discussed so far, the class 1 students respond to the correct category 5 with a much higher probability than did the class 2 students. It is interesting to note when reviewing the class 2 ICCC that class 2 students maintain similar probabilities over the dominant ability parameter range for four of the categories, category 1, 2, 3 and 5. This suggests some ambiguity experienced by class 2 students in responding to this item, and may relate to the manner in which the item was expressed. Class 1 students may have responded correctly to category 5 with a higher probability, not because of their greater aptitude, but rather because of their inability to justify and commit to any other alternative response. Item 46 tests a student’s ability in using Excel to calculate the variance/covariance matrix of a set of returns on equities. It also tests their understanding of what constitutes a valid variance/covariance matrix. To have responded correctly to this item, students would have had to have been familiar with concepts of matrix multiplication, Excel’s MMULT functions, the symmetrical properties of the variance/covariance matrix, and the correct formula for calculating the variance/covariance matrix from the matrix of excess returns. The correct response was category 5, ‘None of the above’. Referring to the ICCC for item 46 in Figure 3, we see that both classes had relatively high probabilities for this category. However, for class 1 students the response category 2 was quite dominant over a large ability range. This category 2 selection suggests some familiarity with matrix multiplication and the MMULT function, but not the variance/covariance symmetry, and not the correct form of the calculation using excess return matrices. For class 2 students, the
204
dominant incorrect selection was category 3. This demonstrates that all conceptual problems had been resolved in this selection, except the correct calculation using the excess returns. Item 48 tests a student’s financial knowledge with regard to the binomial option pricing model, a student’s IT knowledge with regard to Visual Basic for Applications programming, and a student’s mathematical and statistical knowledge of complex calculations. This item is one of the more intricate and difficult items in our list. The correct response is category 2. We find that even with class 1 students of high ability, their response is incorrect. Their dominate response is category 1 which suggests that class 1 students are naively reading their response directly from the output of the debug sample code provided in the item. To identify the correct response, students are required to understand the logic of the program, working through the process to identify the correct response. In contrast, class 2 students are responding with the correct response throughout a broad range of ability values. The naïve response of the class 1 students may suggest a lower level of sophistication in their exposure to generic programming skills. Finally, item 50 tests a student’s knowledge of the binominal option pricing model and the manner in which it approximations the BlackScholes option pricing model. To provide a correct response, students must be familiar with the mathematical derivation of a variable that represents the smallest non-negative integer that identifies all asset price paths that result in the option expiring ‘in the money’, or with a positive terminal value. The correct response is category 1. The ICCC in Figure 3 for item 50 shows that for the class 1 students, the category 3 and category 5 responses have high probabilities over a broad and high end range of abilities. Class 2 students tend to show a high probability of selecting the correct response over a very broad ability range. Selection of response category 3 for class 1 students may suggest that these students have not
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Figure 2. CCCs for Items 2 through to 25 and Class 1 and class 2 Item 2 C lass 1
1 0.9 0.8
0.8 0.7
0.5 0.4
0.4
0.2
0.2
0.1
0.1
0
1
θ
2
0 -3
3
Item 13 C lass 1 C ategory C ategory C ategory C ategory C ategory
0.8 0.7
1 2 3 4 5
1
C ategory C ategory C ategory C ategory C ategory
0.7 probability
0
θ
2
3
Item 13 C las s 2
0.8
0.5
1 2 3 4 5
0.6 0.5
0.4
0.4
0.3
0.3 0.2
0.2
0.1
0.1 -2
-1
0
1
θ
2
0 -3
3
Item 21 C lass 1
1
C ategory C ategory C ategory C ategory C ategory
0.9 0.8 0.7
-2
-1
1 2 3 4 5
C ategory C ategory C ategory C ategory C ategory
0.9 0.8 0.7 probability
0.5 0.4
0
θ
1
2
3
1
2
3
Item 21 C las s 2
1
0.6
1 2 3 4 5
0.6 0.5 0.4 0.3
0.3
0.2
0.2
0.1
0.1 0 -3
-1
0.9
0.6
0 -3
-2
1
0.9
probability
0.5
0.3
-1
1 2 3 4 5
0.6
0.3
-2
C ategory C ategory C ategory C ategory C ategory
0.9
0.6
1
probability
1 2 3 4 5 probability
probability
0.7
0 -3
Item 2 C las s 2
1
C ategory C ategory C ategory C ategory C ategory
-2
-1
0
θ
1
2
3
0 -3
-2
-1
0
θ
continued on following page
205
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Figure 2. continued Item 25 C lass 1
1
C ategory C ategory C ategory C ategory C ategory
0.9 0.8
1 2 3 4 5
0.8 0.7
0.6 0.5
1 2 3 4 5
0.6 0.5
0.4
0.4
0.3
0.3 0.2
0.2
0.1
0.1 -2
-1
0
θ
1
2
3
fully understood what is being asked of them. In selecting category 3, they are focusing on the need to find terminals nodes where S*>K, which is when the option has a positive payoff. These payoffs are represented by the equation identified in the incorrect category 3 response. The data analysis allows the identification of each student and their class membership. When comparing the distribution of students and their majors, we find the distribution as set out in Table 4. We observe that the proportion of single majors tended to dominate in class 1, whilst double majors dominated in class 2. Likewise for IT majors versus Commerce majors, we found that 79% of commerce majors were in class 2, compared to 21% in class 1. This suggests that the students with backgrounds in both disciplines have an advantage in terms of avoiding negative DIF within the examination items. However, of the two disciplines separately, it would appear from Table 4 (b) that a student having a commerce background is better at avoiding negative DIF, than a student with a technology major. When we consider from Table 3 that the items identified for analysis are predominantly of a financial
206
C ategory C ategory C ategory C ategory C ategory
0.9
probability
probability
0.7
0 -3
Item 25 C las s 2
1
0 -3
-2
-1
0
θ
1
2
3
nature, then this conclusion would appear to be intuitively reasonable. Having identified a potential source of DIF allows consideration of corrective responses to existing teaching and assessment. Just identifying the possibility of a priori knowledge bias makes one more cognizant of avoiding such bias when developing new examination items. In setting future examination items, greater care can be taken to ensure that the examination items posed are as self-contained, or restricted to material covered in the unit as possible. This would then avoid excessive reliance on a student’s a priori knowledge to respond correctly to an item. This may be especially true for international students. Many international students transfer between institutions and/or countries, receiving credits for ‘equivalent’ units. Such units may only require a proportional level of content to be deemed ‘equivalent’ to local units. It is therefore important that assessment be restricted to content covered within any one unit.o Another strategy available is to allow students the option to select items in an assessment. Optional items could be included that allow students to exercise their own discretion as to the assessment task, and to identify items
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Figure 3. ICCCs for Item 39 to 50 and Class 1 and Class 2 Item 39 C lass 1
1 0.9
0.9
0.8
0.8
0.5 0.4
1 2 3 4 5
probability
probability
C ategory C ategory C ategory C ategory C ategory
0.6
0.2 0.1
-2
-1
0
1
θ
C ategory C ategory C ategory C ategory C ategory
0.9 0.8 0.7
0 -3
3
-1
0
1
θ
C ategory C ategory C ategory C ategory C ategory
0.8 0.7
0.5 0.4
3
1 2 3 4 5
0.6 0.5 0.4
0.3
0.3
0.2
0.2
0.1
2
Item 46 C lass 2
0.9
0.6
0.1
-2
-1
0
θ
1
2
0 -3
3
Item 48 C lass 1
1
C ategory C ategory C ategory C ategory C ategory
0.9 0.8 0.7
1 2 3 4 5 probability
0.2 0.1
2
0 -3
3
-2
-1
Item 50 C lass 1
1
C ategory C ategory C ategory C ategory C ategory
0.9 0.8 0.7
C ategory C ategory C ategory C ategory C ategory
0.9 0.8 0.7 probability
0.6 0.5 0.4
2
3
1
2
3
1 2 3 4 5
0.6 0.5 0.4
0.3
0.3
0.2
0.2
0.1
0
θ
Item 50 C lass 2
1
1 2 3 4 5
1
1 2 3 4 5
0.4
0.1 1
3
0.5
0.3
0
2
0.6
0.2
θ
1
Item 48 C lass 2
0.7
0.3
-1
θ
C ategory C ategory C ategory C ategory C ategory
0.8
0.4
0
-1
0.9
0.5
-2
-2
1
0.6
0 -3
-2
1
1 2 3 4 5
probability
probability
2
Item 46 C lass 1
1
probability
0.4 0.3
0.1
probability
0.5
0.2
0 -3
1 2 3 4 5
0.6
0.3
0 -3
C ategory C ategory C ategory C ategory C ategory
0.7
0.7
0 -3
Item 39 C las s 2
1
0.1
-2
-1
0
θ
1
2
3
0 -3
-2
-1
0
θ
207
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Table 3. Detailed item information Item 2
Topic Dividend Discount Model
IT
13
Binomial Option Pricing Model
21
Financial Ratio Analysis
25
Financial Ratio Analysis
39
Dividend Discount Model
46
Variance/Covariance Binomial Option Pricing Model with Visual Basic for Applications programming Binomial Option Pricing Model
48 50
Finance
Maths/Stats
Table 4. Distribution of student types between classes, (a) single major versus double major and (b) IT major versus commerce major (a) Single Major Double Major (b)
Class 1 44% 27%
Class 2 56% 73%
IT Major
53%
47%
Commerce Major
21%
79%
that best suit their educational backgrounds and strengths. Alternative approaches to teaching may also be employed that attempt to alleviate potential bias associated with educational backgrounds. Foundational teaching material, perhaps assigned to students from different backgrounds, may be introduced that will provide greater support in successfully completing the unit. Quizzes designed to identify background knowledge deficiencies may be used to identify possible problems at an earlier stage of instruction. Extra remedial material could then target areas of weakness. For example, an analysis not unlike the one performed here, could be readily carried out on a mid semester test, to identify those students who classify as experiencing a negative DIF.
208
Future Trends: Impliitions for e -Learai Assessment The details from the case study provided above demonstrate the potential value in including such diagnostic facilities within the E-Learning context. We have shown that the identification of latent classes within the student cohort can support the ongoing development of not only assessment formats and items, but also the syllabus. Such analysis can be readily integrated in E-Learning technologies, providing diagnostic features to instructional staff. The graphical ICCC output from the IRT model, as illustrated in Figure 2 and Figure 3and also provides a user-friendly visual interface for analyzing the performance of multiple choice items and their distractors. The IRT models used and the Bayesian inferential
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
techniques can be readily automated and incorporate into the existing E-Learning technology platforms. This would be more readily achieved, within an E-Learning context, where a larger student cohort than that described in the case study above, and would make inferences more valid and reliable. The use of a Bayesian approach is not incidental to the analysis. Bayesian methods are particularly suited to models that contain a great deal of latent, or unobserved, features such as the IRT models estimated in the case study. Bayesian methods are increasingly utilized given the availability of improved simulation and computational algorithms. Although the computational resources required to estimate the model parameters are too demanding for any ‘real-time’ diagnostics, processing of large datasets of student responses and reporting could most certainly be handled using a batch mode approach.
C The DIF analysis has highlighted the existence of bias in assessment outcomes between students in our cohort. A lower residual correlation measure for the 2-Class model demonstrates that it is preferred over the 1-Class base model, and that the inclusion of latent classes helps explain the observed pattern of student’s item category responses. Given that the residual correlations between item categories in the 2-Class models are not significantly smaller than for the 1-Class model, suggests that although latent classes exist, the levels of DIF introduced by the current examination items and unit instruction is not excessively overt. In our view, the greatest value of the analytical work carried out is in the provision of an objective, empirical measure of bias, and the identification of items and students for which that bias is mostly associated. It is through this identification that teaching can be better informed and corrective actions, where necessary, more effectively directed.
The results do suggest that a pre-testing of students on entering the unit may be of value, given the diversity of backgrounds. Any shortfall in presumed knowledge coming into the unit can be approached with extra remedial work on those students, particularly those coming from either a commerce or information technology background only.
R van Abswoude, A. A. H., van der Ark, L. A., & Sijtsma, K. (2004). A comparative study of test data dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28(1), 3-24. van der Ark, A. (2007). Mokken scale analysis in R. Journal of Statistical Software, 20(11), 1-19. Bolt, D. M, Cohen, A. S., & Wollack, J. A. (2001). A mixture item response model for multiple-choice data. Journal of Educational and Behavioral Statistics, 26, 381-409. Boud, D., Cohen, R., & Sampson, J. (1999). Peer learning and assessment. Assessment and Evaluation in Higher Education, 24(4), 413-426. Brown, R. (2000, April). Using latent class analysis to set academic performance standards. Paper presented to the Annual Conference of the American Association for Educational Research, New Orleans, LA. Brown, S., Race, P., & Bull, J. (1999). Computerassisted assessment of students. London: Kogan Page. Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31(2), 219-233. Congdon, P. (2003). Applied Bayesian modelling. Chichester: John Wiley & Sons Ltd.
209
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Falchikov, N. (2005). Improving assessment through student involvement: Practical solutions for aiding learning in higher and further education. London: Routledge. Gibbs, G., & Simpson C. (2004). Conditions under which assessment supports students. Learning and Teaching in Higher Education, 1, 3-31. Gitomer, D., & Yamamoto, K. (1991). Performance modeling that integrates latent trait and class theory. Journal of Educational Measurement, 28, 173-189. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff Publishing. Hounsell, D. (2003). Student feedback, learning and development. In M. Slowey, and D. Watson (Eds), Higher Education and the Lifecourse (pp. 67-78). Buckingham: SRHE & Open University Press. Huba, M. E., & Freed, J. E. (2000). Learner-centered assessment on college campuses: Shifting the focus from teaching to learning. Boston: Allyn and Bacon.
van der Linden, W. J., & Hambleton, R. K. (Eds.) (1997). Handbook of modern item response theory. New York: Springer-Verlag. Lunn, D. J., Thomas, A., Best, N., & Spielgelhalter, D. (2000). WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing, 10(4), 325-337. Mokken, R. J. (1971). A theory and procedure of scale analysis. Berlin, Germany: De Gruyter. Nicola, D., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. Nunnally, J. C. & Bernstein, I. H. (1994). Psychometric theory (3rd Edition). New York: McGraw-Hill. Oliver, M., & Trigwell, K. (2005). Can ‘blended learning’ be redeemed? E-Learning, 2(1), 17-25.
James, D., & Fleming, S. (2004). Agreement in student performance in assessment. Learning and Teaching in Higher Education, 1, 32-51.
Shepherd, L. (2001). The role of classroom assessment in teaching and learning (CSE Technical Report 517). University of California, Los Angeles: Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies.
Knight, P. (2002). Summative assessment in higher education: Practices in disarray. Studies in Higher Education, 27(3), 275-286.
Sijtsma, K., & Molenaar, I. W. (2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage.
Katz, I., Martinez, M., Sheehan, K., & Tatsuoka, K. (1998). Extending the rule space methodology to a semantically-rich domain: Diagnostic assessment in architecture. Journal of Educational and Behavioral Statistics, 23(3), 254-278.
Struyven, K., Dochy, F., & Janssens, S. (2003). Students’ perceptions about new modes of assessment in higher education: A review. In M. Segers, F. Dochy & E. Cascallar (Eds.), Optimizing new modes of assessment: In search of qualities and standards (pp. 171-224). Boston/Dordrecht: Kluwer Academic.
Lajbcygier, P., & Spratt, C., (2007) Using “blended learning” to develop tertiary students’ skills of critique. In Lawrence Tomei (Ed.), Integrating information & communications technologies into the classroom (pp. 1-18). Hershey PA: Information Science Publishing. 210
Yen, W. (1984). Effects of local item dependence on the fit and equating performance of the threeparameter logistic model. Applied Psychological Measurement, 8, 125-145.
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
APPppi A: Items aaegorii Item 2. The constant growth dividend discount model is more realistic than the other dividend discount models because it assumes:
Category 1. 2. 3. 4. 5.
constant dividends. dividends that are infinite. dividends that grow to a maximum value. dividends that grow each period by a factor. * None of the above.
Item 13. What are the European put option values for the leafs (i.e. G15, G17 and G19)?
Category 1. 2. 3. 4. 5.
7.83; 2.18; 0. 10.5; 3.35; 0. 0; 0; 2.95. * 0; 0; 0. None of the above.
Item 21. What is the company’s net operating working capital?
Category 1. 2. 3. 4. 5.
$10 $20 $30 * $40 $50
Item 25. What is Amer’s ROE?
211
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Category 1. 2. 3. 4. 5.
11.04% 12.31% 16.99% * 28.31% 30.77%
Item 39. An internet stock pays low dividends in the first few years of its life. Then, its dividends grow at a constant rate. Evaluate the share price. 2001 $0.50 2002 $0.75 2003 $1.00 Then the shares grow at a constant 5% per annum rate. Assume that interest rates are 7% per annum. What is the share price?
Category 1. 2. 3. 4. 5.
$51.94 $42.75 $35 $5 None of the above. *
Item 46. In what Excel command provides cells B16:D18 and what number should be in cells E and F.
Category 1. 2. 3. 4. 5.
“=MMULT(B8:D10,B8:D10)”;0.00111;-0.00231 “=MMULT(B8:D10,B12:D14)”;0.00417;-0.00231 “=MMULT(B12:D14,B8:D10)”;0.00111;-0.00231 “=MMULT(B12:D14,B8:D10)”;-0.00417;-0.00231 None of the above. *
Item 48. Given the information above what is the next value of EurCall given the Expressions in the watch list?
212
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Hint: Note that the ‘execution arrow’ (on the left hand margin of the screen) has not gone to te line “Next Index” yet.
Category 1. 2. 3. 4. 5.
0.95025 2.50288* 3.21678 4.00254 None of the above.
Item 50. When using the binomial pricing approach (and the equation in the question above) to show its equivalence to the Black––Scholes approach we need to simplify the equation in the question above. In particular, we need to eliminate the expression max[0, S-K]. To do this we find the point on the binomial tree ‘leafs’ such that S*>K (where S* is the underlying value and K is the strike price). We can then safely ignore all S*
K is: Hint: u is the up value; d is the down value; n is the total number of leaves; K is the strike price; S is the underlying price.
Category 1. 2. 3. 4. 5.
Log(K/Sdn)/log(u/d). * Log(u/d). Max[0,S-K]. S-K None of the above.
AppENDIX B: Ability Paraaites
Table 5. 1-Class calibrating model Mean and precision of 1-Class ability parameter
Mean
Sd
MC error
0.025
Median
0.975
µ
1.905
0.2391
0.007874
1.457
1.896
2.393
t
1.497
0.4614
0.009402
0.7532
1.446
2.538
213
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
Table 6. 1-Class calibrating model Ability parameter for student, i θ1 θ2 θ3 θ4 θ5 θ6 θ7 θ8 θ9 θ10 θ11 θ12 θ13 θ14 θ15 θ16 θ17 θ18 θ19 θ20 θ21 θ22 θ23 θ24 θ25 θ26 θ27 θ28 θ29
Mean 1.145 2.016 1.786 2.077 2.952 1.02 1.916 1.399 2.03 2.673 2.446 2.551 2.139 2.388 1.575 2.475 1.185 1.201 1.987 2.544 0.6863 2.43 2.632 1.753 2.047 1.539 2.095 2.422 1.538
Sd 0.3001 0.427 0.3928 0.4523 0.5899 0.2905 0.4264 0.3289 0.4366 0.6107 0.5167 0.5702 0.4838 0.517 0.3621 0.5181 0.3031 0.3091 0.4256 0.5668 0.2514 0.5452 0.596 0.3882 0.4444 0.3643 0.4619 0.5424 0.3658
MC error 0.008586 0.01253 0.009097 0.01406 0.01733 0.007459 0.01293 0.008481 0.01122 0.02144 0.01406 0.01705 0.01511 0.01463 0.009839 0.01529 0.008458 0.008204 0.01075 0.01756 0.00671 0.01741 0.02028 0.01016 0.01245 0.01021 0.0129 0.01397 0.01057
0.025 0.5987 1.271 1.104 1.285 1.922 0.4998 1.172 0.8199 1.275 1.659 1.555 1.608 1.336 1.516 0.9367 1.599 0.6369 0.6492 1.249 1a.579 0.2334 1.495 1.65 1.065 1.28 0.8947 1.302 1.51 0.9006
Median 1.133 1.984 1.759 2.045 2.908 1.003 1.883 1.376 1.993 2.606 2.406 2.489 2.094 2.347 1.55 2.425 1.167 1.182 1.954 2.489 0.6741 2.384 2.567 1.721 2.01 1.513 2.058 2.37 1.507
0.975 1.782 2.953 2.632 3.071 4.279 1.631 2.86 2.093 2.986 4.047 3.582 3.809 3.226 3.519 2.361 3.589 1.829 1.857 2.918 3.819 1.205 3.619 3.968 2.588 3.021 2.346 3.097 3.631 2.337
AppENDIX C: Testiior Model Assumptions A prerequisite investigation is advisable prior to the estimation of the IRT model to ensure that the implicit assumptions associated with the model are supported. There are three main assumptions underlying most IRT models. According to van Abswoude, van der Ark and Sijtsma (2004), these include (1) unidimensionality, (2) local independence, and (3) latent monotonicity. A fourth assumption is also included by van der Ark (2007), that of non-intersection. We will explain each main assumption in turn. Uni-dimensionality is assumed in most IRT models that have been applied in practice. These models assume that the probability of a student’s response is a function of only a single trait or proficiency. The functional form of the Item Response Function is Pjk (θ) where θ represents a single ability. Models that assume multi-dimensionality have also been researched and applied, however these models present the investigator with a very large set of parameters to be estimated. Hence, applying uni-dimensional models is perhaps preferred, at least at the initial stages of an investigation. This presents a problem with the item responses used in our case study investigation. As specified, the unit and its final examinations
214
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
responses are drawn from items where a correct response may require students to access knowledge related to or combining finance, statistics and/or information technology. This suggests that a model containing at least two proficiency factors, or perhaps three would be preferable. Local independence can be stated as J
P ( X = x | ) = ∏ P ( X j = x j | ) j =1
(4)
This assumption is that the probability of selecting a response for item j is conditional only on the proficiency factor, θ, and not on any of the other item responses. This means that each item response is assumed to be independent of any other item response, given knowledge of a student’s proficiency. This assumption allows the joint probability distribution for all of a student’s responses to a test to be the product of all of the individual response probability distributions as demonstrated in equation (4). According to van Abswoude, van der Ark and Sijtsma (2004), the two assumptions of uni-dimensionality and local independence do not imply falsifiable consequences on the observed data. This is achieved by the inclusion of the final assumption, a restriction of latent monotonicity. This restriction means that the IRF is monotone decreasing in θ, which can be expressed as follows; Pj [ a ] ≤ Pj [ b ]whenever
a
≤
b
, for all j
(5)
To test for the reasonableness of these assumptions in the response data, prior to estimation, one may employ a number of methods. The one we will describe in detail is based on Mokken Scale Analysis, which we employ using the R package ‘mokken’. All student data, both calibration and cross-validation, was loaded to the R environment resulting in a 65 by 50 dimensional matrix of ‘ones’ and ‘zeros’ indicating a correct or incorrect response. We then passed the matrix to the mokken package function ‘coefH()’ which calculates and outputs the test scalability coefficients, Hij, Hj and H, for item pairs, individual items and the overall test item set. For a detailed discussion of these coefficients and their interpretation, refer to Sijtsma and Molenaar (2002, Chapter 4) and van Abswoude, van der Ark and Sijtsma (2004). In brief, Mokken (1971) suggests that the following rules should be used as a rule of thumb when interpreting the H values. A scale is considered weak if 0.3 ≤ H ≤ 0.4, moderate if 0.4 ≤ H ≤ 0.5, and strong if H > 0.5. The assumptions of unidimensionality, local independence and latent monotonicity imply, according to Sijtsma and Molenaar (2002), the following restrictions, 0 ≤ Hij ≤ 1, for all i ≠ j; 0 ≤ Hj ≤ 1 for all j ; and 0 ≤ H ≤ 1. Alternative methods for testing some of the modeling assumptions are described by van Abswoude, van der Ark and Sijtsma (2004), who test and report the comparative performances of these alternatives. The four covered include MSP (Hemker, Sijtsma and Molenaar, 1995) which has been discussed above, under its R implementation. The others include DETECT (Kim, 1994), HCA/CCPROX (Roussos, 1993) and DIMTEST (Nandakumar and Stout, 1993). Commercial copies of DETECT, DIMTEST and HCA/CCRPOX are available from www.assess.com as DIMPACK.
ENDNOTES
a
b
Examination questions are referred to as ‘items’. For an introductory coverage of IRT, refer to Hambleton and Swaminathan (1985), whilst for a more recent and advanced level text we cite van der Linden and Hambleton (1997). Nunnally and 215
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
c
d
e f
g
h
i
Bernstein (1994) also provide an excellent coverage. Monash University refers to course ‘subjects’ as course ‘units’ The advantage of using a polytomous IRT model is that it provides information on the performance of various distractor categories in each examination item, and allows us to identify those distractors that are more likely to be chosen by one class, over another class of student. This information can then be helpful in identifying the sources of DIF and the manner in which class membership is being determined. In a multiple choice item, each choice available is referred to as a category. Computational Finance has items from different domains: statistics, IT and finance. Therefore, it is unclear if there exists one generic ability for Computational Finance or three, one for each of the sub-domains. Incorrectly assuming one ability dimension can lead to spurious classes. The multiplicative relationship of λjk and θ recognizes that a student’s performance is just not a function of their innate ability, θ but also the interaction effects that exist between ability and a test item. A valid criticism of this work is that the sample size is small, which may result in unstable and ungeneralisable parameter estimates and consequently dubious conclusions. To an extent, we control for this potential overfitting by testing our model out-of-sample on the cross-validation data set. Indeed, we find a sensible interpretation for our model estimated on calibrated data. This result alleviates our fears of over-fitting. We represent the response data, denoted Y, as an (N × J) matrix, where N is the number of students, and J is the number of items in the test. As shown below, the student i’s response for item j is denoted as yij. The responses yij can take any integer value between 1 and K inclusive, where K is the number of response categories. For all response data, the number of categories is K=5. items , j y11 y12 y1 j y21 Y = examinees,i yi1 yij
j
k
l
216
For a comprehensive coverage of applied Bayesian modeling refer to Congdon (2003). All of the modeling and estimation is carried out using the automated Bayesian software tool, WinBUGS. For a detailed discussion of WinBUGS; refer to Lunn, Thomas, Best and Spielgelhalter (2000). In the remainder of this section, we provide a brief summary of the main features of the methodology. For a comprehensive discussion refer to Bolt, Cohen and Wollack (2001). Being common, this improves the interpretation of θ across models and is reasonable given that we are identifying the latent classes on the basis of non-ability factors. Bolt, Cohen and Wollack (2001, p390) note that this also removes some identification issues associated with estimating multiple class models. This residual correlation measure is a modification of Yen’s Q3 measure (see Yen (1984)). See Bolt et al. (2001) for further details of the measure as used in comparing latent class models.
Identifying Latent Classes and Differential Item Functioning in a Cohort of E-Learning Students
m
The functional form of this discriminator is given by
D jk = ∫ P1 jk (
This integral is evaluated numerically using the approximation function,
1 D jk = N
n
o
) − P2 jk ( ) f ( )d
N
∑{P ( ) − P ( )} i =1
1 jk
i
2 jk
i
The distribution of the ability levels of the students as represented by the function f(θ).
Precision is the reciprocal of Variance, and is the preferred dispersion measure used by the WinBUGS software The co-authors would like to thank an anonymous referee for pointing out this issue.
217
218
Chapter XII
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods? Christine Armatas Victoria University, Australia Anthony Saliba Charles Sturt University, Australia
ABSTRACT A concern with E-Learning environments is whether students achieve superior or equivalent learning outcomes to those obtained through traditional methods. In this chapter the authors present the results of a research study comparing students’ learning outcomes with four different delivery methods - printed study material, lecture format, computers and “smart” mobile phones. The results of our study show that learning outcomes are similar when students study by using a computer, mobile phone, or lecture format, while studying with print material yields slightly superior test results. These findings are discussed in the context of the type of learning used in the study and the factors that impact on the effectiveness of using mobile phones for learning purposes, such as learning styles and attitudes to computers. The authors conclude the chapter by briefly discussing developments in mobile technologies and the opportunities they present for mobile learning.
INTRODUCTION Contemporary learning environments in higher education are increasingly characterised by the use of E-Learning opportunities designed to sup-
port and extend students’ learning experiences. Recently a shift has occurred in the use of information technology to support education such that educational content and learning opportunities are now being made accessible to a mobile device
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
(e.g., laptop, personal digital assistant (PDA), iPod, mp3Player or mobile phone), over a wireless network (Hoppe, Joiner, Milrad, & Sharples, 2003; Leung & Chan, 2003; Chen, Chang & Wang, 2006 ). Mobile learning or m-learning is the term that has been coined to describe learning using a mobile device and its attraction is that content can be accessed from any location at any point in time with a device that is small, lightweight and easy to carry. Preliminary evidence indicates that m-learning provides a number of opportunities to support traditional and new forms of learning, although there are significant concerns around screen size, device memory and access costs that need to be managed carefully (Chen et al., 2006). There is also concern amongst teachers as to whether students can achieve equivalent or superior learning outcomes from online learning environments generally, and particularly with mobile devices given their compact nature. In this chapter we examine whether learning using a mobile device is as effective as other learning methods in order to establish its efficacy as a viable technology tool to enhance teaching and learning. We report the results of a controlled study comparing the learning outcomes students achieve when studying material using four delivery methods and discuss the implications of our finding that students learn just as well when material is delivered using a mobile phone compared to a computer or lecture-format. Although the type of learning assessed in our study is limited to recall and recognition learning, our study shows that mobile phones can be an effective learning tool. We discuss the implications of our findings broadly and we present other learning and assessment opportunities mobile phones can be used for, both now and in the future.
Ba Most students have a mobile phone or access to one. It was estimated in 2003, that there were over
300 million world-wide users of mobile phones (Leung & Chan 2003). However, a report from the International Telecommunications Union in 2008 estimated there are 3.3 billion users (ITU, 2009). Many also have iPods and mp3Players; ready access to which makes these devices ideal as a mobile learning tool (Stockwell, 2007). In 2006 the Technology Advancement Centre at East Carolina University conducted a survey to assess the mobile needs of distance education and campus learners. Of the 4,000 students who responded to the survey 94% owned cell (mobile) phones and the preferred communication device carried by campus students was a mobile phone (DuVall, Powell & Lucier, 2006). There are several advantages to using a mobile learning device. First, mobile devices are more portable because of their small size and when combined with access to wireless networks, educational activities can occur in locations beyond the classroom, embedding the learning situation within a real-life context that can enhance the relevance of the learning situation for students (Chen, Kao, & Sheu, 2003; Liu, Tao & Nee, 2007; Motiwalla, 2005). Mobile learning can also promote immediacy of learning by allowing learning to operate in real time (any time), so that students can access information as urgently as required (Fallahkhair, Pemberton, & Griffiths, 2007; Triantafillou, Georgiadou, & Econimides, 2006). With a mobile learning device, students no longer need to record a question and later refer to a textbook or wait for an opportunity to access information online (Chen et al., 2003; Leung & Chan, 2003; Liu et al., 2003). Although mobile phones are cheaper and more portable than personal computers (PCs), at present, there are several technical limitations preventing small mobile learning devices from replacing the PC as the principle E-Learning device. With many mobile devices, the screen size limits the amount of information that can be displayed (Shudong & Higgins, 2005). Small font is hard to read and larger text presents the inconvenience of continually scrolling down
219
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
(Csete, Wong & Vogel, 2004). Small screen size further restricts the colour depth and resolution of pictures, resulting in poorer picture quality than that provided over a computer, although this technical limitation is quickly changing with more sophisticated and power-efficient mobile screens (Shudong & Higgins, 2005). Liu and Kao (2007) have demonstrated a methodology whereby students can access shared displays in classrooms while using the more limited mobile screen when necessary outside of the classroom. Small screen size is the most significant physical limitation to the successful adoption of mobile learning, so it is important to note that both technological and procedural solutions are on the horizon. Text input speed is slow on most mobile devices making the transfer of information inefficient using the handset keyboard in comparison to the speeds achievable using a QWERTY keyboard on a laptop or PC (Houser, Thornton & Kludge, 2002). The battery span of mobile devices is limited and can be quickly drained by the processor load demanded by a number of applications running simultaneously, requiring the inconvenience of continually recharging devices to avoid loss of information (Csete et al., 2004). The current connection speed and data transfer rates of many mobile devices is also slow, limiting fast access to web-based information (Shudong & Higgins, 2005). Slow transfer of data is especially noticeable when downloading and retrieving images and consequently most web-based connections are for simple material like e-mail (Csete et al., 2004). Furthermore, limited memory space presents a barrier and restricts the application of many programs, and this is especially true for mobile phones (Csete et al., 2004; Thornton & Houser, 2004). Finally, the operating system of mobile devices is not uniform across devices, which often means that information cannot be shared amongst users with different devices (Csete et al., 2004; Leung & Chan, 2003; Roschelle, 2003; Shudong & Higgins, 2005). Fortunately, with developments in research and technology, many
220
of the technical limitations of mobile devices are being overcome. For example, many new mobile devices now come with expandable memory cards and third-generation networks offer extremely fast rates of data transfer that can be configured for specific applications such as m-learning (Bradlow, Saliba, Ivanovich, & Fitzpatrick, 2005), although at a cost which may be prohibitive for educational purposes. Speech-to-text technologies are also maturing and will be a viable means for data input and will help to overcome some of the limitations of a small keypad (Fan, Saliba, Kendall, & Newmarch 2005). At present, mobile phones are mainly being used by educators to support the administration of teaching and learning (Harley, Winn, Pemberton, & Wilcox, 2007). Short Message Service (SMS) is particularly popular amongst students for receiving exam results and for communicating with students and staff. However, “just-in-time” administrative applications are designed to enhance access to educational or educational-related content as opposed to enhancing learning specifically. Thornton and Houser (2004) demonstrated that SMS can be used effectively as a learning tool. They applied the principles of elaboration and distributive rehearsal to enhance learning for English vocabulary by Japanese University students, sending mini-vocabulary lessons to students via SMS. Elaboration and deep processing for materials was encouraged by placing words to be remembered in an elaborative sentence frame, e.g., the word vision appeared on mobile phones as “Today’s word, vision is the same as eyesight. Do you have good vision or do you have to wear glasses?” Spaced rehearsal was also encouraged by sending words at three fixed time-periods per day. Performance of students who learnt vocabulary via their mobile phones was compared to students who learnt via traditional paper rehearsal method and also to students who learnt the words by accessing definitions over the web using a PC. Results showed that significantly more words were recalled at test following SMS compared to the E-
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
Learning or paper methods. In addition, students also reported that they preferred receiving the vocabulary over their mobile phones compared to the other methods and that the screen size had not inhibited learning. Mobile devices have also been used to encourage greater participation by students during face-to-face teaching sessions and one particular application in this area has been the development of classroom response systems (Bollen, Eimler, & Hoppe, 2004, Csete, et al., 2004; Roschelle, 2003). In a typical scenario, a teacher poses a short answer or multiple choice question to which students respond anonymously via individual hand-held devices. Responses are sent to the teacher’s PC and responses are aggregated and presented via a visual display such as a bar graph projected onto a whiteboard or screen. The graph becomes a point of reference for collaborative discussion. The classroom response system provides an opportunity to enhance collaboration beyond teacher-oriented and E-Learning driven methods. In addition, anonymous responding encourages passive and shy students to provide feedback and second, teachers can monitor and modify their own teaching styles to quickly rectify misunderstandings (Roschelle, 2003). A number of researchers have investigated how mobile learning devices can foster collaborative learning outside of the classroom, eg. field-learning activities (e.g., Csete et al., 2004; Hoppe et al., 2003). Typically, students must wait for an opportunity to participate in face-to-face interactions with educators to discuss experiences that take place outside the classroom, which may be difficult where students are undertaking an extended period of work experience. Seppala and Alamaki (2003) demonstrated the usefulness of mobile devices for managing interactions between educators and teachers in training in a study using SMS and digital pictures to facilitate the supervision process. Students took pictures of teacher training events and learning activities with a digital camera and images were downloaded to
a shared database, and subsequently uploaded to a mobile device. Students and supervisors used SMS to make comments and observations about the various pictures. Ultimately, the interaction that occurred between peers and supervisors was a virtual collaboration process because no face-toface interaction occurred during communications. Research has suggested that technologies like SMS can encourage rich involvement in communication, even where participants have high levels of anxiety about communicating (Moore, Armatas, & Saliba, 2005). Seppala and Alamaki’s (2003) study was limited in evaluation to assessment of student and teacher opinions with reported benefits of convenience, immediacy and increased interactivity of learning. The researchers however provided no assessment of actual learning outcomes, such as final test grades or supervisory reports. Such measures are paramount if teachers are to be convinced of the usefulness of mobile devices in the educational process. Mobile devices have been used as a means of encouraging in-field concept mapping (Silander, Sutinen, & Tarhio, 2004). Concept mapping shows the development of ideas in a hierarchical diagram and becomes a collaborative process when individuals are encouraged to communicate around the concepts and their relations. Silander and colleagues developed an application based on SMS sent using mobile phones to allow students to engage in collaborative concept mapping of tree species. Out in the field students used SMS and script language to add concepts and relations pertaining to tree species which were subsequently relayed to students in the classroom equipped with wireless laptops. Class-based students informed in-field students whether concepts had already been provided by others and in addition worked on class-based resources to seek further information about tree types. Messages were later analysed for content and showed that the concept mapping exercise encouraged spontaneous participation and communication between students both within the field and the classroom. Students also reported
221
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
that they enjoyed the experience and found it easy to add new concepts and their relations. Ease of use and positive affect (or level of enjoyment) have both been shown to be important determinants of technology use (Fan, Kendall, & Saliba, 2004). The importance of context in design is less clear, though some studies have demonstrated that substantial improvements to mobile applications can be made by specifically addressing the user context (Roy, Scheepers, Kendall, & Saliba, 2006). Good application design is paramount for mobile devices so as not to introduce further usability limitations above those which mobile devices can inherently possess (e.g. screen size). The previous discussion illustrates that mobile devices generally and mobile phones more specifically, are capable of supporting teaching and learning. In particular, researchers have demonstrated that through the simple technology of SMS, embedding information in rich and elaborate contexts can improve foreign-language learning by students above traditional paper and pen rehearsal methods. Mobile devices can also provide students with the inherent benefits of learning through visual methods and have been used to support collaborative learning, which encourages interaction, critical thinking and behavioural development with peers and educators as opposed to the traditional passive lecture-style method of delivering educational content. Mobile devices provide the potential for new ways and areas in which collaboration can occur, most noticeably, students can interact with an environment outside of their classroom and collaboration can occur at any point in time (Librero, Juan Ramos, Ranga, Trinona, & Lambert, 2007). However, there is an overwhelming need for researchers to empirically and objectively evaluate the learning that results when using mobile devices. The theoretical benefits of mobile learning notwithstanding, at present the evidence that exists in support of mobile learning is based mainly on subjective comments from students and teachers
222
regarding the quality of the educational experience. Furthermore, many researchers have either failed to evaluate their projects or have written on the hypothetical benefits of mobile learning. Whilst experiences and enjoyment with the educational experience are inherently important, researchers need to employ objective measures of the learning outcomes such as test and exam grades. In addition, whilst mobile devices do offer the potential for taking learning outside of the classroom and for greater convenience by allowing students to study anywhere and anytime, the question is whether this enhances learning above traditional teaching delivery methods such as print and face-to-face teaching. In our review of the literature on mobile learning, we found only one study that directly compared learning outcomes achieved using online and mobile devices and more traditional delivery methods. This study was by Zurita and Nussbaum (2004) who developed a constructivist learning environment to help 6-7 year old children learn to read using mobile devices supported by a wireless network. To assess the learning outcomes achieved using this mobile method and a paper based method, Zurita and Nussbaum randomly assigned children to groups of three. The children then completed a language task using either handheld mobile devices or a paper based method. The learning tasks for the two methods had the same underlying pedagogy and methodology, but the handheld devices were able to offer additional benefits such as feedback and structured decision making. By comparing pre- and post-test scores, Zurita and Nussbaum concluded that those children performing the activity using the mobile devices performed significantly better than the children using the paper-based method. In their view, the learning environment created using the mobile device and wireless network helped the students to work constructively and collaboratively with less teacher support. This in turn led to better learning outcomes, as well as being enjoyable for the students. Whether this
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
result generalises to other student groups, such as secondary and tertiary students, remains to be empirically established. To address the question of whether equivalent learning outcomes are achievable using online and mobile learning environments, we conducted a study comparing learning outcomes achieved using four different teaching delivery methods – lecture format, print materials, personal computer and mobile phone. Students studied four geography lessons on fictitious countries using each of the delivery methods. After each lesson a short answer test was given to measure lesson learning. The PC and mobile phone delivery methods offered the same material, with both visual and audio modalities and opportunities to review material. The lecture format presented the same material in audio and visual formats, but the only option to review material was through students’ notes. The print materials provided only visual formats. It was predicted that learning outcomes from the PC and mobile phone delivery methods would be similar to those from more traditional methods, such as lecture and print materials, providing other factors, such as anxiety or difficulties using technology did not come into play. The experimental design we used for our study allowed us to control for confounding effects so that differences between delivery methods for test scores could be attributed to how the material was studied, not who studied it, what they studied or when they studied it. We controlled for teacher effects by using the same person to deliver all the lectures and for individual effects by having each participant study using all four study methods. Everyone studied exactly the same material, regardless of the way they studied it. They were also given the same amount of time to study the material. We assessed both recall and recognition memory through lesson tests administered after each study session, and the lesson order was changed across testing sessions to control for differences between lessons with respect to interest or difficulty and to minimise
fatigue and motivational effects. Although our study context was somewhat artificial and certainly only used a specific type of learning (recall and recognition assessed using short answer and multiple choice questions), it does provide a relatively pure assessment of how well students learn using print, lecture, personal computers and mobile devices.
How Well Do Students Learna Whn Using a Mobile Phh To test whether learning outcomes are similar – or at least no worse – when using a mobile phone as the delivery method, we recruited undergraduate students enrolled at a major metropolitan university in Melbourne, Australia. The 81 students who took part found out about the study via an advertisement on their University’s on-line ‘job opportunities’ page. Their ages varied from 18 to 27 years with a mean of 21.43 years (SD = 2.13 years). There were 41 males (50.6%) and 40 females (49.4%) in total and students were paid for their participation. All but one of the participants reported that they used the Internet for their course subjects, mainly for downloading or viewing content. How often they did this varied – some only went online once a week, some over 40 times a week – which suggested a range of experience and interest in online learning amongst the group. Only one student reported using their mobile phone for accessing educational content related to their coursework. The students took part in the study in groups of three. When they arrived they were told that the session should last between one-and-a-half and two hours, and that during this time they would be asked to study four lessons using four study methods: Print; face-to-face lecture; computer and mobile phone. It was explained that the four lessons were geography lessons about fictitious countries and at the end of each lesson a short test would be given. As an incentive, they were
223
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
told that a prize would be given to the participant who scored the highest on these tests. Testing took place in a laboratory which was set up with three rooms - a ‘lounge room’, ‘bedroom’ and ‘study’. In the ‘study’, participants studied using the computer, in the ‘bedroom’ they used the mobile phone, and in the ‘lounge room’, printed material and the lecture were presented. To simulate a more realistic presentation experience for the lesson being studied via lecture format, testing was arranged so that the three participants in a session sat together in the lounge room for the lecture presentation.
Study Material and Tests Four lessons of similar difficulty were created. Each lesson described a fictitious country and provided information regarding its population, capital, flag, landmarks, fauna and flora, governing system and currency. Fictitious countries were used to ensure that none of the participants had previous knowledge that would assist them with learning the study material and answering the test questions. Tests corresponding to each lesson were also designed and consisted of short answer questions about material in the lesson. Each test was scored out of twenty marks and required twenty pieces of information from the lesson. In order to determine that participants found the lessons comparable, after completing each test participants rated the level of interest (1=Not at all Interesting, 2=Not Interesting, 3=Interesting, 4= Very interesting), enjoyment (1=Not at all enjoyable, 2=Not very enjoyable, 3= Enjoyable, 4=Very Enjoyable) and difficulty (1=Very Difficult, 2=Difficult, 3=Not Very Difficult, 4=Not at all Difficult) of the lesson with respect to both the content and the way it was delivered. A fifth test was devised which participants completed after the study sessions. The format for this 10item test was multiple-choice with four possible answers, with questions drawing on material from all four lessons.
224
Encoding: Visual and Auditory All four lessons were created in HTML format for use on the computer. These files were then converted into XHTML for the mobile phone. For the Lecture condition, PowerPoint Presentation files were made by copying and pasting HTML files into a PowerPoint template, which was formatted to look identical to the HTML pages. The hyperlinks for the navigation buttons used in the HTML files (e.g., ‘back’, ‘next’, ‘audio’) were removed for the PowerPoint version, with navigation being controlled by the researcher presenting the lecture. Printouts of the PowerPoint slides were used for the Printed Material condition. Both the mobile phone and computer conditions had audio links for each study session and the participant had the choice of listening to the lesson rather than reading the lesson or doing both. Audio files were recorded in a sound-proof room using a Sony DAT recorder and Sony stereo microphone connected to a Macintosh Power PC via an analogue connection. The AIFF files from the Macintosh PC were converted into MP3 files at 128kbps 16kHz rate and the MP3 files were then used on the computer and mobile phone. For the lecture, the presenter read out the same information that was available on the audio track for the mobile phone and PC conditions.
Technology Devices A Nokia 7610 smart phone was used for the mobile phone condition. This phone was selected as it offered a larger than average screen size and the XHTML pages could be dimensioned to display on the phone screen more easily. The screen resolution was 176 x 206 pixels with a colour depth of 65k and had polyphonic sound capability. For the Computer Condition, a Pentium 4 with 512MB RAM running XP Pro Service Pack 1 was used, with a 20” LCD monitor. The PowerPoint presentation used for the lecture condition was run on an IBM Thinkpad T20, with a resolution of 1024 x
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
768 pixels. The presentation was projected using a Sony VPL-CX5 LCD Data Projector. For each testing session, the order in which participants completed each study mode (i.e., computer, mobile, lecture and print) was changed, and a different lesson assigned to each study mode. We used randomisation and counterbalancing techniques to generate the testing procedure so as to eliminate any order or fatigue effects and to control for any differences in level of difficulty or interest between the lessons being studied. Each testing session was structured so that a lesson was allocated to a delivery method and the participants rotated through the delivery methods in the order prescribed by the testing procedure for that session. Students worked separately for the print, computer and mobile versions in the assigned rooms, but together for the lesson being studied with the lecture format. At the start of each testing session students were given a tour of the laboratory and had the opportunity to practice with the mobile phone and computer to ensure they were familiar with how to navigate using these delivery methods. However, this familiarisation did not include any of the materials students saw during the study period. This familiarisation and practice opportunity was provided to ensure that students were not disadvantaged because of issues associated with the delivery method such as navigation. After familiarising themselves with the laboratory setup and the mobile phone and PC, the students completed a pre-test questionnaire in a group setting. The questionnaire was used to obtain demographic information and to assess patterns of mobile phone ownership and usage. It also included the Computer Attitude Scales (CAS, Loyd & Gressard, 1984) which measured each student’s computer confidence, liking, and anxiety and the extent to which they perceive computers are useful. Higher scores on the four scales for the CAS indicate a more positive attitude towards computers.
Once the questionnaire was completed, each participant was told where they would study the first lesson and that they had eight minutes study time. They were left in the allocated room by themselves to study, except for the lecture condition where participants as a group were presented a PowerPoint Presentation by the researcher. After eight minutes study time, any material relating to the lesson, including participants’ notes, were put away out of sight and participants completed the lesson test, which they had five minutes to complete. At the end of the test, participants were told which lesson and study method they would complete next. After all participants had completed lessons in the four delivery modes, the final, multiple choice test on all of the lessons was administered, with five minutes allowed for participants to complete this test. Following this, a post-test questionnaire was given to participants to complete. The second questionnaire was used to gather data on participants’ study preferences using the Approaches to Learning and Studying (ALS) scale from the Experiences of Teaching and Learning Questionnaire (ETLQ; Entwistle, McCune, & Hounsell, 2002). This 36-item scale consists of five subscales which assess the extent to which students report they use strategies for study that involve a deep approach, surface approach, monitoring activities, being organised or managing effort. Higher scores on each subscale indicate a greater tendency to employ the particular approach when studying. The second questionnaire also contained items assessing students’ use of technology for educational purposes and a series of uses for mobile phones for mobile learning that participants were asked to rate the usefulness of using a 5-point scale. Finally, participants were asked to rate how interested they were in using their mobile phone to facilitate learning and how likely they thought it was that this would happen. When this questionnaire was completed, participants were thanked and paid for their participation.
225
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
on how they studied. For these analyses we used repeated measures analysis of variance (ANOVA), comparing students’ test scores across each study method – see Table 1 for descriptive statistics. A one-way repeated measures ANOVA showed that students’ test scores were better for lessons they studied using print (F(3,228)=6.848, p=.000) compared with the other study methods. However, there was no statistically significant difference between test scores for the lecture, PC and mobile phone study methods, indicating that students performed equally well on tests for the other study method types. Since we found that students’ test scores were better for lessons studied using print, but were equivalent for the other three study methods, we wanted to check that there weren’t any systematic differences in how enjoyable, interesting or difficult students found the lesson they studied with a given delivery method – see Table 2 for descriptive statistics.
What We Found On average participants had owned a mobile phone for 3.6 years (SD=1.8) and reported using their phone 2.7 hours per week (SD=1.8). Just over half the students were on post-paid plans (51%) and the remainder were pre-paid customers. For the question “How interested are you in using your mobile phone to facilitate learning?” over 60 percent of the sample indicated that they were interested or very interested, with 15 percent being not interested and the remaining 25 percent unsure. Just under half the sample (46 percent) indicated that it was likely or very likely that they would use their mobile phone to facilitate learning in the future. A quarter of the sample reported this was unlikely or very unlikely, while the remainder of the sample were unsure. The first analysis we conducted was to find out if students perform differently depending
Table 1. Means and standard deviations for the test scores as a function of study method Study Method (N=77) Print Computer Lecture Mobile Phone
Mean 15.16 14.30 14.19 13.58
Standard Deviation 2.847 3.358 3.141 3.495
Table 2. Means and standard deviations for rating of content and way the lesson was delivered as a function of study method Ratings
Content of lesson Interest in content (N=59) Difficulty of content (N=57) Enjoyment of content (N=57) How lesson was delivered Interest in study method (N=57) Difficulty of study method (N=56) Enjoyment of study method (N=55)
226
Study Method Mobile Phone
Computer
Print
Lecture
M (SD)
M (SD)
M (SD)
M (SD)
3.07 (.64) 2.96 (.63) 2.89 (.62)
2.92 (.62) 3.14 (.64) 2.70 (.63)
2.80 (.74) 2.88 (.68) 2.68 (.63)
2.76 (.69) 2.88 (.68) 2.75 (.74)
3.02 (.67) 3.11 (.65) 2.98 (.59)
2.43 (.68) 3.13 (.69) 2.45 (.77)
2.77 (.87) 2.61 (.73) 2.58 (.71)
2.61 (.76) 2.86 (.72) 2.51 (.74)
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
Repeated measures ANOVA showed that regardless of which lesson was studied by which method, the content of the lesson for all study methods was equally enjoyable. Similarly, there was no significant difference in the perceived difficulty of the content between the study modes. In contrast, one-way ANOVA showed that the study methods differed with respect to how interesting participants found the content F(3,174)=3.957, p=.009, such that students found material studied using the computer significantly more interesting than other study methods, regardless of the lesson assigned to each study method. One-way ANOVA tests conducted on the ratings for the way the lesson was delivered (i.e., the study method used) showed differences between the four study methods for how interesting F(3,142)=7.981, p=.000 , difficult F(3,165)=9.297, p=.000 and enjoyable F(3,162)=8.586, p=.002, participants rated the way the lesson was delivered. Ratings of how interesting the way the lesson was delivered were significantly less for lessons studied using the print version compared with the other study methods. Further more, participants rated enjoyment of the way the lesson was delivered higher for the computer compared to the mobile phone F(1,54)=12.774, p=.001 and both of these more enjoyable on average than the lecture F(1,54)=7.776, p=.007. Lessons studied using print and the computer were perceived as significantly less difficult F(1,55)=17.589, p=.000, than the other methods on average.
Te Relationship Between Test Sores and Computer Attitudes and Sudy Preferences Descriptive statistics for the four scales measuring computer anxiety, confidence, liking and usefulness (CAS, Loyd & Gressard, 1984) and the subscales of the ALS (Entwistle et al., 2002) are detailed in Table 3. To investigate whether there is any relationship between test scores for each delivery method and how students score on the
subscales for the computer attitudes and study preferences measures, Pearson correlations were calculated. The only significant correlation found was between test scores for lessons studied by mobile phone and scores on the scale for surface learning (r=-.36, p<.001). The negative correlation indicated that as the tendency to use a surface approach to learning increased (as reflected in a higher score on the surface scale from the ALS) the poorer the learning outcomes for studying using a mobile phone were (as reflected by lower test scores for this study method). However, this relationship was not strong, with approximately 13 percent of the variability in scores on the test for the mobile phone being explained by scores on the surface learning strategy scale. Several correlations approached significance. The first was between test scores for the mobile phone study condition and scores on the computer anxiety scale (r= .31, p=.006) suggesting a tendency for better learning outcomes for material studied using the mobile phone when computer anxiety was less. There was also a tendency for scores obtained for lessons studied using the print method to be positively correlated with computer anxiety (r= .26, p>.001).
Wean The purpose of our study was to establish whether studying using a mobile phone is as effective as studying when material is delivered via print, lecture or a computer. It was predicted that, provided other factors such as anxiety using computers and other technologies did not come into play, students should learn equally well using mobile phones, computers, print or lecture presentation. By comparing students’ test scores for the four modes of delivery of content, this study has demonstrated that the learning outcomes when material is delivered via a computer, mobile phone or lecture are equivalent. However, students appear to learn more when studying printed material, as evidenced by their higher test scores when
227
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
Table 3. Means and standard deviations for responses on computer attitudes and learning styles scales Scale Computer Attitudes (N=80) Confidence Liking Anxiety Usefulness Learning & Studying Deep (N=80) Surface (N=81) Monitor (N=81) Organise (N=81) Effort (N=81)
studying content delivered in print format. The superiority of learning outcomes with print cannot be attributed to interest, enjoyment or perceived difficulty of either the content or the way it was delivered. Participants reported finding lessons studied using printed materials as significantly less interesting than lessons studied by the other methods, while material delivered via the mobile phone was as interesting, enjoyable and difficult/easy as the same content delivered via other modes. Similarly, the way content was delivered was as interesting, enjoyable and difficult/easy via a mobile phone as it was for other delivery methods. There did appear to be some effect of study strategy on the quality of learning outcomes when learning material using a mobile phone as evidenced by the significant negative correlation between the extent to which a surface approach to learning is favoured and scores on the mobile phone learning condition. This suggests that where students tend to adopt a surface approach to learning, using a mobile phone may have a detrimental effect on their learning, as indicated by poor test scores for material studied using a mobile phone. However, this effect is only small and may be related to the need to navigate using the mobile phone keys in order to view or listen
228
M (SD) 39.51 (6.74) 36.40 (6.46) 35.50 (3.90) 42.28 (3.62) 4.00 (.46) 2.72 (.62) 3.79 (.56) 3.28 (.80) 3.72 (.72)
to content in the lesson. This effect did not occur for the computer condition, which also required navigating using the mouse or cursor, suggesting that there may be something about navigation with the mobile phone coming into play. During our observation of students when familiarising themselves with the mobile phone navigation, they did not require assistance using the mobile phone and seemed to be able to translate their knowledge of how to use web pages to navigating using the phone. It may, however, have taken participants a bit more time to familiarise themselves with the functioning of the mobile phone and this in turn may have reduced the amount of time available for them to study the material. That there was a correlation between computer anxiety and scores on the lesson test for the mobile phone is consistent with this. Lower scores on the computer anxiety scale indicate greater discomfort with technology and showed a non-significant trend for those with higher computer anxiety to do worse on the test of material learning using the mobile phone. The level of familiarity and practice with mobile phones is something that should be more tightly controlled in future work of this nature as it may have reduced the effectiveness of this method of delivery compared with methods more familiar to students.
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
Results from this study show that when the same material is presented in the same modes (i.e. audio and visual) using lecture, computer or mobile phone as the delivery method, learning outcomes are similar. That lessons studied using computers and mobile phones have similar learning outcomes is not unexpected as they both offer similar quality audio and visual content and, for this study, had the same way of navigating the lesson. Both offered opportunities to review material or to study the content as the learner preferred – students could choose to listen to the audio, read the text or do both and could review the screens making up the lesson in the order they wished. In this respect, compared with the lecture format, the mobile phone and computer conditions offered greater flexibility in reviewing materials. Although students could review any notes they took during the lecture, the effectiveness of the review of material was reliant on the accuracy and completeness of their notes from the lecture. In the lecture condition, students could not review the audio or visual aspects of the presentation, as they could in the mobile phone and computer conditions. Despite this, learning outcomes were similar for the three study methods. One explanation for the similarity in learning outcomes between these three conditions may be the experience our university sample had with the lecture format as a study method. Students are exposed to lectures on a regular basis at university and so to be successful need to be efficient and effective when learning via this method. Their experience with the lecture format may have been an advantage which made up for other shortcomings of this delivery method (Alavi; 1994; Jeffries; 2003), which in this study included a lack of opportunity to review audio and visual aspects of the presentation. Familiarity might also explain the superior learning outcomes for the print condition, as evidenced by the significantly higher score for tests of lessons studied using print. For the print condition students had the same text and visual content as the other conditions, but there was
no audio available. However, the printed pages could be viewed in any order and as often as the student wished, making navigation through the lesson very flexible and review of material easy and quick. It seems that, although for the print condition the additional level of coding in the form of auditory information was not available, visual and text processing were sufficient to facilitate deep processing of the material, which in turn led to better recall and recognition performance on the test. In terms of information workload, McCracken and Aldrich (1984) describe processing as a division of resources into four distinct categories: visual, auditory, cognitive and psychomotor (VACP). The VACP model predicts that when one modality is in use, it can effect processing in other modalities. The lack of auditory stimuli in the printed condition in our study may have allowed more processing resources available in the visual channel to result in better encoding of material. The other conditions required processing resources across multiple channels, and this may have led to less optimal encoding. This raises the question of whether three levels of encoding – audio, visual and text – are too much for students to learn effectively. In the lecture condition there were two encoding levels (audio and visual) and for the mobile phone and computer conditions students could choose between some combination of audio, visual and text. No data was collected on what combinations students used for the mobile phone and computer conditions, although it is possible that students elected to read the text and view the pictures, or to hear the text read out and view the pictures. This could be investigated in the future to determine whether two processing modes are optimal for learning to occur. Participants in this study all owned a mobile phone and reported using it frequently. Over half of the participants were studying full-time and employed part-time. Although the participants reported that they did not use their mobile phones for educational purposes, they did express inter-
229
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
est in doing so. Overall, our sample had very positive attitudes towards computers, seeing them as useful and tending to enjoy using them. They were also not anxious using computers. If this attitude towards computers extends to other, similar technologies such as PDAs and mobile phone applications, this could explain the similar learning outcomes and reported interest, enjoyment and level of difficulty for the computer and mobile phone learning conditions found in this study. Computers are now an important tool university students need to use in their studies and so it is not surprising that the level of comfort and acceptance with them was high with this sample. Since the learning application used on the mobile phone condition functioned in exactly the same way as the computer condition, it seems likely that the students generalised their computer skills to the mobile phone. Consistent with this, only one student asked for assistance using the mobile phone and navigating through the material. However, students may have lost some study time familiarising themselves or experimenting with the mobile phone’s navigation, which may have negatively impacted on their learning. Providing students with a structured practice session with both the computer and mobile phone conditions would have removed this factor. At the time of our study, the mobile phone used was considered ‘high end’ and as such, not all students would have been familiar with such a device. However, it is likely that familiarity would be much higher should a similar study be conducted again, as the capabilities of even entry level mobile phones have become similar to the ‘high end’ phone we used in our study. The sample in this study was drawn from a range of study areas and year levels and so represents a diverse student group, making the findings generalisable to other student populations. However, in this study only immediate recall and recognition of material was tested. Whether retention of material over a longer time period is the same for the conditions used in this study remains
230
to be established. A similar study using testing at strategic intervals e.g., immediately following the study session and one week later, would be valuable in answering this question. It also needs to be acknowledged that the type of learning used in this study was limited to recall and recognition learning. While our study is valuable because it addresses in a controlled manner the question of whether students learn as well when using mobile phones as they do with other methods, the way learning was measured and the type of learning investigated was limited. In this respect the design of our study does not do m-learning justice as it ignores other types of learning that mobile devices can facilitate. These include the opportunities for collaborative and constructivist learning that mobile devices can support which have been used by researchers to investigate whether m-learning is effective (e.g., Csete et al., 2004; Hoppe et al., 2003; Seppala & Alamaki, 2003).
Fture Trends As the storage and memory capacity of the average mobile phone increases, and the cost of mobile data decreases, the more attractive m-learning will become. It already provides a simple and affordable means for students to interact with one another and teaching staff using voice, SMS and mobile instant messaging. However, authoring learning material for mobile phones is not well supported and currently there is little integration of mobile phones with learning management systems. This will change in time, particularly as the efficacy of m-learning is confirmed by researchers and practitioners and new and innovative uses for mobiles in learning contexts are developed and implemented. Developments in mobile technologies will open up new possibilities for teachers wanting to stretch the boundaries of the learning environment for students. As speech recognition technologies
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
mature, it will soon be possible to control mobile phone applications with voice commands and to use speech-to-text to enter data. This will help address some of the problems with entering data on a mobile phone that currently limit its uses. In the future when unified communications are truly realised, voice, video and data will come together and the boundaries between fixed and mobile networks will be blurred. It will be possible to easily and seamlessly move from a fixed phone to a mobile phone and back, maintaining your video call with high quality audio and video. For students this means they can start watching a live lecture stream from their home computer and then switch it to their mobile phone when they have to leave to catch the bus to work. It also means that they can participate in online tutorials with both video and audio wherever they are, without needing access to a PC with an internet connection. Charging models will also become more flexible, so that it will be possible for students to take advantage of wireless hotspots and local area networks when they are available, only needing to incur mobile data costs when they are not. The current problem many mobile phones have with limited screen real estate will also be addressed through wirelessly connected display modules. These could include glasses or head sets that display screen content in full size, screens projected on to a convenient surface or even three dimensional holograms. While we wait for these innovations to hit the market, other mobile devices beside the mobile phone will emerge to capture the imagination of educators. Already ruggedized, mobile data enabled tablets and laptops are appearing on the market, with the cost of these devices likely to decrease as the technology becomes mainstream. These devices are bigger than mobile phones, and allow the functions of desktop PCs to be taken anywhere that mobile data is accessible. With the increasing focus on work-based learning these devices will become valuable learning tools which help students and teachers manage the logistical issues associated
with students being away from the campus for extended periods. Parallel developments, such as battery extenders for mobile phones which can be plugged in to your phone when the battery is flat to get you operational again and solar-chargers for laptops, will also help to make mobile devices reliable and attractive learning tools.
C In summary, the results of our study have shown that learning outcomes are similar when students study using a computer, mobile phone or lecture format, but that studying with print material yields slightly superior test results. Whether the results of this study are due to experimental artefacts such as not providing sufficient practice for the mobile phone condition or the number of modalities students used to study the material remains to be investigated. However, the results of this study indicate that despite its small screen size, the mobile phone can be used as an effective learning tool. Although our study only used a specific type of learning, it confirms that mobile phones are an effective learning tool. However, as the growing body of research on the use of mobile devices in educational contexts shows, m-learning can take many forms and can support learning activities beyond the classroom. What seems certain is that m-learning will continue to flourish and that future technology developments in the mobile phone and other industries will greatly assist in this.
R Alavi, M. (1994). Computer-mediated collaborative learning: An empirical evaluation. MIS Quarterly, (pp. 159-174). Bollen, L., Eimler S., & Hoppe, H. (2004). SMS based discussions- Technology enhanced collaboration for a literature course. In J. Roschelle, T. Chan, Kinshuk & S. Yang (Eds.), Proceedings of
231
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
the 2nd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2004), Mobile Support for Learning Communities (pp. 209-210) Los Alamitos, USA: IEEE Computer Society. Bradlow, H., Saliba, A., Ivanovich, M., & Fitzpatrick, P. (2005). Recognising user perception in the design of telecommunications networks. Annual Review of Communications, 58, 7-14. Chen, G. D., Chang, C. K., & Wang, C. Y. (2006). Ubiquitous learning website: Scaffold learners by mobile devices with information-aware techniques. Computers and Education, 50(1), 77-90. Chen, Y., Kao, T., & Sheu, J. (2003). A mobile learning system for scaffolding bird watching learning. Journal of Computer Assisted Learning, 19, 347-359. Csete, J., Wong, Y. H., & Vogel, D. (2004). Mobile devices in and out of the classroom. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of the 16th World Conference on Educational Multimedia and Hypermedia & World Conference on Educational Telecommunications (pp. 4729-4736) Norfolk VA: Association for the Advancement of Computing in Education. DuVall, J. B., Powell, M., & Lucier, A. (2006) Online survey of current student use of mobile phones for enhanced personal safety and communication. East Carolina University, Technology Advancement Centre, (pp. 24-28). Entwistle, N., McCune, V., & Hounsell, J. (2002). Approaches to studying and perceptions of university teaching-learning environments: Concepts, measures and preliminary findings. Retrieved October 12, 2008 from ELT Project Web site, http://www.ed.ac.uk/etl/docs/ETLreport1.pdf Fallahkhair, S., Pemberton, L., & Griffiths, R. (2007). Development of a cross-platform ubiquitous language learning service via mobile phone and interactive television. Journal of computerassisted learning, 23(4), 312-325. 232
Fan, Υ., Saliba, Α., Kendall, Ε., & Newmarch, J. (2005). Speech interface: An enhancer to the acceptance of m-commerce applications. In W. Brookes, E. Lawrence, R. Steele & E. Chang (Eds.), Proceedings of the IEEE International Conference on Mobile Business (pp. 445-451) Los Alamitos, USA: IEEE Computer Society. Fan, Y., Kendall, E., & Saliba, A. (2004). Acceptance of technology: A meta-analytic approach. Paper presented at the Honolulu International Conference on Social Science, Honolulu. Harley, D., Winn, S., Pemberton, S., & Wilcox, P. (2007). Using texting to support students’ transition to university. Innovations in Education and Teaching International, 44(3), 229-241. Hoppe, H., Joiner, R., Milrad, M., & Sharples, M. (2003). Guest editorial: Wireless and mobile technologies in education. Journal of Computer Assisted Learning, 19, 255-259. Houser, C., Thornton, P., & Kludge, D. (2002). Mobile learning: Cell phones and PDA’s for education. In Proceedings of the International Conference on Computers in Education (pp. 1149-1150) Massey University, New Zealand: ICCE. International Telecommunications Union (2009). Global Mobile Phone Users Top 3.3 Billion by End-2007. Retrieved January 27, 2009 from http://www.itu.int/ITU-D/ict/newslog/ Jeffries, P. (2003). ICT in supporting collaborative learning: Pedagogy and practice. Journal of Educational Media, 28(1), 35-48. Leung, C. H., & Chan, Y. Y. (2003). Mobile learning: A new paradigm in electronic learning. In Proceedings of the 3rd IEEE International Conference on Advanced Learning Technologies, (pp. 76-80) Los Alamitos, USA: IEEE. Librero, F., Juan Ramos, A., Ranga, A. I., Trinona, J., & Lambert, D. (2007). Uses of the cell phone for education in the Philippines and Mongolia. Distance Education, 28(2), 231-244.
Is Learning as Effective When Studying Using a Mobile Device Compared to Other Methods?
Liu, C. C., & Kao, L. C. (2007). Do handheld devices facilitate face-to-face collaboration? Journal of Computer Assisted Learning, 23(4). 285-299. Liu, C. C., Tao, S. Y., & Nee, J. N. (2007) Bridging the gap between students and computers: Supporting activity awareness for network collaborative learning with GSM network. Behaviour and Information Technology, 27(2), 127-137. Liu, T. C., Wang, H. Y., Liang, J. K., Chan, T. W., Ko, H. W., & Yang, J. C. (2003). Wireless and mobile technologies to enhance teaching and learning. Journal of Computer Assisted Learning, 19, 371-382. Loyd, B. H., & Gressard, C. (1984). Reliability and factorial validity of computer attitude scales. Educational and Psychological Measurement, 44, 501-505. McCracken, J. H., & Aldrich, T. B. (1984). Analysis of selected LHX mission functions: Implications for operator workload and system automation goals (Technical note ASI 479-024-84(b)). Fort Rucker, AL: Anacapa Sciences, Inc. Moore, V., Armatas, C., & Saliba, A. (2005). The role of trait anxiety and impression management in adolescents’ use of the Short Message Service on mobile telephones. In M. Katsikitis (Ed.), Proceedings of the 40th annual conference of the Australian Psychological Society:Past reflections, future directions (pp. 213-217). Melbourne: The Australian Psychological Society Ltd. Motiwalla, L. F. (2005). Mobile learning: A framework and evaluation. Computer and Education, 49(3), 581-596. Roschelle, J. (2003). Keynote paper: Unlocking the learning value of wireless mobile devices. Journal of Computer Assisted Learning, 19, 260-272. Roy, N. L. S., Scheepers, H., Kendall, E., & Saliba, A. (2006). A comprehensive model incorporating mobile context to design for mobile use. In Proceedings of the 5th Conference on Human
Computer Interaction (pp. 22-30) New York: ACM SIGCHI. Seppala, P., & Alamaki, H. (2003). Mobile learning in teacher training. Journal of Computer Assisted Learning, 19, 330-335. Shudong, W., & Higgins, M. (2005). Limitations of mobile phone learning. In H. Ogara, M. Sharples, Kinshuk, Y. Yano (Eds.), Proceedings of the 3rd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2005 (pp. 179-181) Los Alamitos, USA: IEEE Computer Society. Silander, P., Sutinen, E., Tarhio, J. (2004). Mobile collaborative concept mapping: Combining classroom activity with simultaneous field exploration. In J. Roschelle, T. Chan, Kinshuk & S. Yang (Eds.), Proceedings of the 2nd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2004), Mobile Support for Learning Communities (pp. 114-118) Los Alamitos, USA: IEEE Computer Society Stockwell, G. (2007). Vocabulary on the move: Investigating an intelligent mobile phone-based vocabulary tutor. Computer Assisted Language Learning, 20(4), 365-383. Thornton, P., & Houser, C. (2004). Using mobile phones in education. In J. Roschelle, T. Chan, Kinshuk, & S. Yang (Eds.), Proceedings of the 2nd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2004), Mobile Support for Learning Communities (pp. 3-10) Los Alamitos, USA: IEEE Computer Society. Triantafillou, E., Georgiadou, D., & Econimides, A. A. (2006) The design and evaluation of a computerized adaptive test on mobile devices, Computers and Education, 50(4), 1319-1330. Zurita, G., & Nussbaum, M. (2004). A constructivist mobile learning environment supported by a wireless handheld network. Journal of Computer Assisted Learning, 20, 235-243.
233
234
Chapter XIII
Evaluation Strategies for Open and Distributed Learning Environments Thomas C. Reeves University of Georgia, USA John G. Hedberg Macquarie University, Australia
INTRODUCTION Evaluation falls into the category of those often neglected human practices such as exercise and eating right. All of us involved in education or training know that we should engage in systematic evaluation when designing or implementing any type of learning environment, but we rarely get around to it. Perhaps this lapse stems from the fact that most instructional design models such as the ubiquitous ADDIE (Analysis, Design, Development, Implementation, and Evaluation) model (Molenda, 2003) appear to suggest that we can postpone evaluation until the end of the process. Whatever the reason, evaluation often remains in the realm of promises made, but not kept, such as “I’ll eat better tomorrow.” Even when we do evaluate interactive instructional products or programs such as open
and distributed learning environments, we often do so in an ill-conceived manner, the evaluative equivalent of thinking that if we eat a salad with a burger and fries, we have somehow engaged in healthy eating. For example, quasi-experimental comparisons of open and flexible learning environments with traditional classroom learning environments continue to dominate studies published in refereed research journals or presented at research conferences. Did we really need another large-scale meta-analysis such as the one recently reported by Bernard et al. (2004) to tell us that such comparisons are “of low quality” (p. 416) and that the final outcome is almost always of “no significant difference.” Bernard et al. (2004) produced an excellent piece of scholarship, but as with most such meta-analyses of educational technologies (e.g., Dillon & Gabbard, 1998; Fabos & Young, 1999), their analytical synthesis provides
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Evaluation Strategies for Open and Distributed Learning Environments
precious little to guide designers or practitioners in their efforts. With these failings in mind, this chapter is focused on recommending a set of practical strategies for evaluating open and distributed learning environments. A much more extensive treatment of this important topic can be found in a book titled Interactive Learning Systems Evaluation (Reeves & Hedberg, 2003). A point of clarification about terminology is necessary. In his innovative Framework for Web-Based Learning, Khan (2001) includes “Evaluation” as one of the eight key dimensions. Under Evaluation, Khan lists both “assessment of learners” and “evaluation of the instruction and learning environment” (p. 78). In our work (Reeves & Hedberg, 2003), we have preferred to separate these two factors, reserving the term “assessment” to refer to activities focused on measuring characteristics of human learners (their learning, motivation, attitudes, etc.), a process that we see as a sub-dimension of pedagogy. We use the term “evaluation” solely to refer to activities focused on estimating the outcomes and worth of products, programs, and projects. In short, we assess people and evaluate things. The remainder of this chapter is exclusively focused on evaluation issues.
Why EEVLUTE We recommend a primarily pragmatic philosophy of evaluation that maintains that you should evaluate in order to provide the information that you and other decision makers need to make better decisions about the design and implementation of open and distributed learning environments. We view this as analogous to the conclusion that you should exercise and eat right to provide the necessary ingredients for a long and healthy lifespan. Exercise and evaluation are not ends in themselves in most contexts, but means to longer life on the one hand and better decision making on the other.
As a developer, manager, or implementer of open and distributed learning environments, you must make decisions, similar to those made by other professionals. For example, before rendering a diagnosis, a physician usually questions a patient to ascertain the patient’s presenting complaint and medical history, conducts a thorough examination, and runs various tests. In fact, the quality and reputation of a physician is determined largely by how skillful he or she is in conducting “evaluative” acts such as interviewing, examining, and testing. The same is true of an evaluator. Years of experiences as designers and evaluators of interactive learning environments have convinced us that decisions informed by sound evaluation are better than those based on habit, ignorance, intuition, prejudice, or guesswork. This may seem painfully obvious, and yet, far too often, we have seen people make poor decisions about the design and implementation of open and distributed learning environments simply because they failed to seek pertinent information that would be relatively easy to obtain. The e-learning field is replete with horror stories of bad choices related to factors such as course management systems, pedagogical design, and graphical user interface (Reeves, 2003).
EVluaTFUNCTons From time to time in the United States, the federal government’s Department of Agriculture issues recommendations for healthy eating, usually presented in the format of a food pyramid. Recent versions of these dietary pyramids suggest that the broad base of the pyramid should encourage us to exercise and consume plentiful helpings of whole grains, fruits, and vegetables, whereas the narrow pinnacle of the pyramid should limit us to the relatively infrequent consumption of red meat, sweets, and butter. We suggest a similar pyramidic metaphor for evaluation as illustrated in Figure 1. There are six types of evaluation functions represented in this 235
Evaluation Strategies for Open and Distributed Learning Environments
evaluation pyramid: (1) Review, (2) Needs Assessment, (3) Formative Evaluation, (4) Effectiveness Evaluation, (5) Impact Evaluation, and (6) Maintenance Evaluation. The pyramid signals that the largest investments in time, money, and resources should go into the first three of these functions, especially formative evaluation. These first three aspects are often omitted from popular models of evaluation which focus more on Effectiveness and Impact and little else. From the pyramid in Figure 1, smaller “servings” of effective, impact, and maintenance evaluation are suggested because, although these are “good” for you, the returnon-investment from these types of evaluation is lower than for the baseline functions. The function of Review as an evaluation strategy is most important during the initial conceptualization of an open and distributed learning environment. Two key activities of the Review function are studying the published literature related to flexible learning and examining similar learning environments. In short, you should carry out Review activities to find out what is already known about the type of interactive learning environment you plan to develop and to understand the state of the art of design of
learning environments of that kind. The failure to engage in Review can be seen at almost any large e-learning trade show where products with remarkably similar objectives and designs are marketed by multiple vendors. The function of Needs Assessment as an evaluation strategy is to identify the critical needs that the proposed learning environment will address. A need is any significant gap between desired levels of performance and current levels of performance. Most needs stem from deficiencies (the lack of required knowledge, skills, and attitudes) or discrepancies (differences between what people can do and what they normally do). For example, there may be a need to expand professional development opportunities for teachers with respect to technology integration, in which case an open and distributed learning environment may be a viable solution. Alternatively, there may be a need to enhance the quality of the outcomes of educational programs concerning responsible alcohol consumption by young adults, and again, the provision of an e-learning environment may be the best way of meeting such a need. The primary activities carried out during Needs Assessment are task analysis, job analysis, and learner analysis (Rossett, 1987).
Figure 1. Evaluation pyramid
Maintenance Evaluation
Impact Evaluation Effectiveness Evaluation
Review - Needs Assessment – Formative Evaluation
236
Evaluation Strategies for Open and Distributed Learning Environments
The function of Formative Evaluation as an evaluation strategy is to provide the information required to guide decisions about creating, debugging, and enhancing an open and distributed learning environment at various stages of its development. Formative evaluation should drive the instructional design and development process regardless of whether a traditional instructional systems design (ISD) or rapid prototyping model is pursued. Two of the primary activities carried out during Formative Evaluation include expert review and usability testing (Flagg, 1990; George & Cowan, 1999; Rubin, 1994). The function of Effectiveness Evaluation as an evaluation strategy is to determine whether an open and distributed learning environment accomplishes its objectives within the immediate or short-term context of its implementation. It is essential to evaluate the implementation of a program with the same rigor as outcomes are evaluated. Some of the primary Effectiveness Evaluation activities include field tests, observations, interviews with different stakeholders, and performance assessment (Horton, 2001; Reeves & Hedberg, 2003). The function of Impact Evaluation as an evaluation strategy is to determine whether the knowledge, skills, and attitudes (or more broadly “outcomes”) learned in the context of an open and distributed learning environment transfer to the intended context of use, for example, the workplace or into further education. Inevitably, practical impact evaluations, including return-oninvestment studies, entail considerable degrees of inference from results to decisions. Some of the primary Impact Evaluation activities include document analysis, interviews, and observations, as well as experimental methods. The latter are quite expensive in most cases, impractical in others, and yield results that are inevitably subject to multiple interpretations. The function of Maintenance Evaluation as an evaluation strategy is to examine the continuing viability of an open and distributed learning
environment over time. This is perhaps the most infrequently applied evaluation function, but the importance of it is growing, as the size and scope of e-learning enterprises rapidly expand. Some of the primary activities carried out in the name of Maintenance Evaluation are document analysis, interviews, observations, and automated data collection. In this function, the role of critical incidents is becoming one of the most common indicators of a need to change. For example, most managers of large implementations of learning management systems can tell nightmarish tales about how a database was corrupted or some other catastrophic event occurred which disrupted learners’ access to an online learning system at critical times. Another way of envisioning the interrelationships of these six evaluation functions is to illustrate them in relationship to the major development functions carried out when an open and distributed learning environment is designed, produced, and put into operation. Figure 2 demonstrates how the major development functions (conceptualize, design, develop, implement, institutionalize, and re-conceptualize) should be informed by the major functions of evaluation (review, needs assessment, formative evaluation, effectiveness evaluation, impact evaluation, and maintenance evaluation).
Case Studies To illustrate how evaluations of open and distributed learning environments can be carried out, two case studies are described below, one related to formative evaluation issues and the other related to impact evaluation issues. The two case studies are hypothetical, although we have actually conducted related evaluations with similar results. The case studies are organized around the following key steps that are required for the planning, implementation, and reporting of any evaluation:
237
Evaluation Strategies for Open and Distributed Learning Environments
Figure 2. Relationships between development and evaluation functions DEVELOPMENT FUNCTIONS
EVALUATION FUNCTIONS Should be informed by
Project Conceptualization
Review
Design
Needs Assessment
Development
Formative Evaluation
Implementation
Effectiveness Evaluation
Institutionalization
Impact Evaluation
Project Re-Conceptualization
Maintenance Evaluation
1. Identify the decisions that the evaluation should inform. 2. Specify the questions that must be answered to provide the information needed to inform the identified decisions. 3. Select reliable, valid, and feasible evaluation methods. 4. Implement the evaluation methods in a rigorous and professional manner. 5. Report the findings in an accurate and timely manner so that decisions can be informed as intended.
Formative Example Scenario: A large pharmaceutical company commissioned the development of e-learning materials related to GMP (Good Manufacturing Practice) by an entrepreneurial e-learning firm. Decisions: Decisions needed to be made about the graphical user interface (GUI) for the e-learning materials. The final programs provide more than eight hours of interactive training for pharmaceutical workers who range in education
238
levels from two-year associate degrees to doctorates. The GUI had to be designed to provide userfriendly navigation, and allow multiple forms of interactions and clear readability of text. Questions: Based on graphical design principles, learner analysis, and other instructional design techniques, the e-learning firm developed a prototype GUI for the GMP programs. To inform decisions related to GUI enhancement, the developers require answers to the following questions: • •
What enhancements do e-learning experts recommend for the prototype GUI? What enhancements are required on the basis of user testing of the prototype GUI?
Methods: Heuristic review and usability testing are the two most commonly used evaluation strategies carried out to improve the navigation and user-friendliness of interactive computer programs (Nielsen, 1993). The developers first subjected the GUI to heuristic review (Nielsen, 1994) using a panel of five experts. Heuristic review involves the following procedures:
Evaluation Strategies for Open and Distributed Learning Environments
a.
Each member of the panel of experts reviews the prototype e-learning screens using a set of heuristics such as the ones provided by Nielsen (1994) or ones specifically developed for open and distributed learning environments. A typical heuristic would be: “Minimalist Design: Screen displays should not contain information that is irrelevant, and media features such as animation and sound should not be gratuitously added to the e-learning program.” b. After identifying violations of the design heuristics, each expert independently exams each violation to rate its severity (How seriously will the violation affect the learner’s experience?) and extent (How pervasive is the violation within the overall program?). Sample severity and extensiveness scales are illustrated in Figures 3 and 4. c. The expert reviews are synthesized and shared at a group meeting conducted face to face or online. Based upon the severity and extensiveness ratings, opportunities for improvements are identified and specific recommendations about how to redesign the GUI are made. Some of the advantages of heuristic review are: (1) it is relatively quick because you do not need to find or schedule users to review the program, (2) it is relatively easy to review problem areas many times, and (3) it is relatively inexpensive because no fancy usability testing facilities are needed. Some of the disadvantages of heuristic review are: *1) its validity is weaker than usability testing because no actual learners are involved, (2) it generally finds fewer problems than usability testing, (3) recruiting good experts can be challenging, and (4) building consensus among experts can be difficult sometimes. In most cases, heuristic review should be conducted as a precursor to usability testing. Implementation: The heuristic review takes place over a period of five days and costs almost
$6,000. The costs include consulting fees paid to each of the five experts and the expenses associated with a half-day review meeting, including the time of the two primary instructional designers from the e-learning firm. Reporting: Significant changes are made to the e-learning screen designs based upon the analysis of the heuristic review. However, the developers know that heuristic review, while valuable, is no substitute for user testing, and they subsequently engage in formal usability testing. Reeves and Carter (2001) provide an introduction to usability testing, and detailed guides to usability testing have been written by Nielsen (1993) and Rubin (1994).
Impact E Scenario: An international wireless telecommunications company wishes to increase sales of its products and services when customers call for support of products they already own. The company develops an e-learning program to help its customer service representatives (CSRs) recognize and take advantage of additional sales opportunities when engaged in customer support. Decisions: The e-learning program is ready for rollout at one of the company’s regional customer support centers. Decisions must be made about two key issues: (1) should the e-learning program be disseminated to other regional customer support centers, and (2) should a generic version of the e-learning program be developed as a commercial product for marketing to other companies providing telephone customer support? Questions: To inform decisions related to internal and external dissemination of the CSR sales e-learning program, the corporate decision makers require answers to the following questions: •
How does CSR call performance change as a result of completing the e-learning program?
239
Evaluation Strategies for Open and Distributed Learning Environments
Figure 3. Severity scale for rating violations of e-learning screen design heuristics Severity Scale for Rating Violations of Screen Design Heuristics 1. Cosmetic: Fix if possible. 2.
Minor: Fixing this should be given low priority.
3.
Medium: Fixing this should be given medium priority.
4.
Major: Fixing this should be mandatory before the system is launched. If the problem cannot be fixed before launch, ensure that the documentation clearly shows the user a workaround.
5.
Catastrophic: Fixing this is mandatory; no workaround possible.
Figure 4. Extensiveness scale for rating violations of e-learning screen design heuristics Extensiveness Scale for Rating Violations of Screen Design Heuristics 1. Single case 2.
Several places
3.
Widespread
•
How are the sales of related wireless products affected as a result of the rollout of the e-learning program?
Methods: To collect the data needed to answer the aforementioned questions, three types of evaluation methods are planned. First, a random sample of transcripts of CSR call performance before and after completion of the e-learning program are analyzed by an independent specialist in qualitative data analysis to determine the rate at which opportunities for additional sales have been recognized and acted upon by the CSR. The qualitative specialist is unaware of which transcripts were recorded before or after the training. Second, Web surveys are distributed to all the CSRs at the call center where the program was originally implemented as well as to their immediate supervisors. The survey for the CSRs seeks to determine the extent to which the CSRs performed each of these sales promotion tasks after the e-learning course in comparison with
240
how they performed these tasks prior to the course (much better, better, same, worse, much worse). The survey for the supervisors seeks to determine whether the supervisors had observed the changes in their CSR staff after the e-learning program related to behaviors such as: • • •
•
The CSRs seem to know more about our products and services than they did before the e-learning. The CSRs seem to have more confidence dealing with customers. The CSRs seem to be able to recognize sales opportunities when speaking to customers. The CSRs seem to be able to increase their sales of related products and services.
Third, the sales figures for wireless products generated at this call center are compared with the sales figures of the previous 12 months of the fiscal year, as well as with the sales figures generated at each of the other call centers.
Evaluation Strategies for Open and Distributed Learning Environments
Implementation: The discourse analysis takes two weeks to complete and costs $8,000. The data analyst charges $800 per day for her services. The surveys are designed, distributed, and analyzed in house at a cost of $5,000 over a period of three months. Sales figures are tracked for two quarters after the e-learning program is released at the one regional call center. Analysis of the sales data is done in house at a cost of $3,000. Reporting: Results are positive on all counts. Although the average length of a customer’s call increases by two minutes, this time is totally focused on additional sales of wireless products and services. Survey results indicate that CSRs feel more confident in responding to client calls, recognize sales opportunities, and act upon them. Sales increase by 40% during the two quarters after the initial rollout at the participating call center, whereas sales at the other sales center report no comparable increases. As a result of the evaluation, the e-learning program is immediately rolled out to all other call centers and design of a generic e-learning product commences.
CONCLUS Can we guarantee that you will receive the types of positive results noted above if you invest more of your time, money, and people resources in evaluation? No. Your results may be less than compelling, but what we can promise is that your decision making will be enhanced. You will have the critical information you and others need to make informed decisions about the design, use, and outcomes of e-learning. Returning to exercise and diet with which this chapter began, no one can guarantee that you will not be hit by a car while jogging or that you will not suffer food poisoning from dining at the salad bar rather than the hamburger joint. But, on average, exercise and eating right lead to longer, healthier, and happier lives. Similarly, on average, appropriate evaluation of open and distributed learning environments
at key times in their development and use will enhance the decision making required to make these environments the best they can be.
REFERENCES Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74(3), 379-439. Dillon, A., & Gabbard, R. (1998). Hypermedia as an educational technology: A review of the quantitative research literature on learner comprehension, control and style. Review of Educational Research, 68(3), 322-349. Fabos, B., & Young, M.D. (1999). Telecommunications in the classroom: Rhetoric versus reality. Review of Educational Research, 69(3), 217-259. Flagg, B.N. (1990). Formative evaluation for educational technologies. Hillsdale, NJ: Lawrence Erlbaum. George, J., & Cowan, J. (1999). A handbook of techniques for formative evaluation: Mapping the student’s learning experience. London: Taylor & Francis Group. Horton, W. (2001). Evaluating e-learning. Alexandria, VA: American Society for Training and Development. Khan, B.H. (2001). A framework for Web-based learning. In B.H. Khan (Ed.), Web-based training (pp. 75-98). Englewood Cliffs, NJ: Educational Technology Publications. Molenda, M. (2003). In search of the elusive ADDIE model. Performance Improvement, 42(5), 34-36.
241
Evaluation Strategies for Open and Distributed Learning Environments
Nielsen, J. (1993). Usability engineering. Boston: Academic Press. Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen & R.L. Mack (Eds.), Usability inspection methods (pp. 25-64). New York: John Wiley & Sons.
B. Khan (Ed.), Web-based training (pp. 547-557). Englewood Cliffs, NJ: Educational Technology Publications. Reeves, T.C., & Hedberg, J.G. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology Publications.
Reeves, T.C. (2003). Storm clouds on the digital education horizon. Journal of Computing in Higher Education, 15(1), 3-26.
Rossett, A. (1987). Training needs assessment. Englewood Cliffs, NJ: Educational Technology Publications.
Reeves, T.C., & Carter, B.J. (2001). Usability testing and return-on-investment studies: Key evaluation strategies for Web-based training. In
Rubin, J. (1994). Handbook of usability testing. New York: John Wiley & Sons.
This work was previously published in Flexible Learning in an Information Society, edited by B. Khan, pp. 226-235, copyright 2007 by Information Science Publishing (an imprint of IGI Global).
242
243
Chapter XIV
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program in Distance and Online Education Madhumita Bhattacharya Massey University, New Zealand
ABSTRACT This chapter presents a description and analysis of salient issues related to the development of an integrated e-portfolio application implemented at Massey University to help students track and accumulate evidence of skills developed over their period of study, particularly associated with the three core papers in the program. The Web-based e-portfolio project was initiated to help students provide evidence required by employers and research supervisors in a progressive and reflective manner by identifying the links across different papers and demonstrating their own conceptual understanding. Administrative issues are discussed, as well as considerations for future developments based on the experiences of this study.
INTRODUCTION This chapter reports on the conceptualization and implementation of the integrated e-portfolio project initiated at the College of Education, Massey University, New Zealand. The conceptual model of the integrated e-portfolio is based on the assessment related tasks and the graduate profile
of students in the distance and online education postgraduate program in the College of Education at Massey University. The portfolio framework was designed to allow students to display a range of their work completed in three core papers/ courses in the program. The e-portfolio became a means for students in the area of distance and online education to display key aspects of their
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
expertise. The making of integrated portfolio supports a programmatic approach to assessment and knowledge development. Initially the digital portfolio platform of John Hopkins University has been used. It is a Web browser-based framework for e-portfolio for its use across different papers and subject areas. The framework for e-portfolio enables students to edit multimedia objects. The task of developing e-portfolio has been introduced initially for the three core papers only. The coordinators of the three core papers have designed the assignments, keeping objectives of the papers and graduate profile into consideration, interrelating and overlapping tasks so that students are able to visualize the links among different courses and understand the ways in which learning in one paper is complemented in another paper. The portfolio is a part of student’s assessment. On completion of their study students will have a portfolio of assessed work that will provide a presentation of key knowledge and skills developed throughout the program of the postgraduate studies. The author envisages that the employers will be able to see students work in an electronic form that demonstrates their skills and knowledge. Through this process of learning students were led to reflect on their strengths and weaknesses in a manner that gave direction for future study. Creation and development of integrated e-portfolio also enabled students to learn and develop research skills useful for project/thesis work in the later part of their study.
Eairogram E-learning is defined by the New Zealand Ministry of Education (2004, p. 3) as “learning that is enabled or supported by the use of digital tools and content. It typically involves some form of interactivity, which may include online interaction between the learner and their teacher or peers. Elearning opportunities are usually accessed via the Internet, though other technologies such as
244
CD-ROM are also used in e-learning.” It would be an extremely rare tertiary institution that does not have a learning management system (LMS) for online delivery, and a body of staff already using it in their courses (Nichols & Anderson, 2005). At Massey University we have acquired WebCT as the LMS since 2002 for delivery of courses at a distance. At Massey University we also use CMC platforms such as Horizon Wimba and Macromedia Breeze for conducting real-time meeting sessions with the distance students. We have about 20,000 extramural (distance) students which is double the number of internal (on campus) students. Massey University postgraduate qualifications staircase (Figure 1), in education, permits students to tailor courses and programs in ways that are suitable and which accommodate a wide range of circumstances and needs. On the staircase, the qualifications commence with two-paper postgraduate certificates (commonly for individuals just starting postgraduate study). Thereafter, the qualifications include general and endorsed 100point postgraduate diplomas and a wide range of 200-point masterates, and they culminate with doctoral study, either the EdD or the PhD. Since 2004 we have introduced a completely Web-based postgraduate program in distance and online education focusing on different aspects of e-learning. It is a new endorsement under the postgraduate qualifications in education offered by the College of Education. The program has three core papers (“Instructional Design and Learning Technologies in Distance and Online Education,” “Teaching for E-learning,” and “Policy, Practice and Trends in Distance and Online Education”). These papers are compulsory for students who wish to earn a postgraduate diploma or masterate in “distance and online education.” The integrated e-portfolio project presented in this chapter has been implemented across the three core papers as mentioned previously. It is essential to discuss about electronic portfolios, its purpose, and its types before discussing about the concept of integrated e-portfolio.
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
Figure 1. Massey University postgraduate qualification staircase
S et U p
S et U p
S et U p Postgraduate Certificate 2 papers (50 points)
Postgraduate D iploma 2 cert papers + 2 more papers (100) points
Whaiortfolioi The ever-advancing capabilities of computer technology and the increased need for portability of evidence related to qualifications, knowledge, and attributes mean that the “shoe-box” approach to storage is no longer adequate. An electronic version offers a different type of storage and a more flexible means of presentation—be it a PowerPoint, hyperlinked text, or an Acrobat PDF presentation. Also, as a career management tool to help write job applications, students can quickly and effectively store and access large amounts of information that are easy to update, reflect upon, and improve (Rogers & Williams, 1999). Electronic portfolios focus on “growth and development over time, implemented through selection, reflection and inspection of skills, attributes and achievements, along with goal setting and self-evaluation” (Barrett, 2001, p. 2). Additionally the e-portfolios provide the capability of directly linking students’ portfolio evidence to the standards for which they may need to demonstrate achievement. These standards include the recently introduced Massey University graduate attributes, employment or graduate studies selection criteria, or practicum and/or course assessment outcomes.
Master’s D egree 4 diploma papers + project and 2 or more papers or thesis only (200 points)
Professional D octorate by coursework and thesis or PhD by thesis only (300 points)
The electronic portfolio is a multimedia portfolio approach that allows the creator to present teaching, learning, and reflective artifacts in a variety of media formats (audio, video, and text). E-portfolios are sometimes referred to as e-folios, electronic portfolios, digital portfolios, or multimedia portfolios (Montgomery, 2004). Wheeler (2005) stated that e-portfolio is best defined by its purpose. Integrating many varied, published definitions, e-portfolio can be understood as a collection of purposefully organized artifacts that support backward and forward reflection to augment and assess growth over time. Artifacts could include any digital work, such as, a class-assigned report, transcript, diploma, video of a performance, audio of a speech, images of fieldwork, links to Web sites, and so forth. Early e-portfolio work has established key principles for e-portfolio practice: •
•
Individuals (students, faculty, etc.) rather than institutions have life-long ownership and control of their e-portfolio. Individuals retain the right and ability to grant limited access to portions of the e-portfolio and to move their portfolio among institutions. E-portfolios should ultimately support the reflective practices needed for life-long 245
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
•
•
•
learning. This implies e-portfolio applicability and portability within K-12, to higher education, and on to career. E-portfolio users benefit from system- and faculty-based guidance in choosing learning artifacts and arranging them in views. Assessment against curricular rubrics is essential in establishing meaning for eportfolio artifacts. Standards for portability of e-portfolios are essential for lifelong individual and institutional value.
Individual ownership of his or her life-long learning information is revolutionary. It shifts ownership of an educational record from passive management among many disparate organizational systems (e.g., K-12 schools, universities, professional career development in corporations, etc.) to active management by the individual. The individual assembles collections of artifacts from his or her e-portfolio into a view and grants access to that specific view to specific individuals/organizations for a period of time. In the widely discussed quest for a society engaged in life-long learning, e-portfolios provide an essential tool to enable integration, meaning, and portability among the many educational experiences of an individual’s life. At Massey University we are engaged in a number of e-portfolio projects. Based on our research findings we affirm that e-portfolio is one of the essential tools for transforming higher education.
2. The structured e-portfolio: A predefined organization exists for work that is yet to be created. 3. The learning e-portfolio: Organization of the work evolves as the work is created. There are a variety of purposes for developing electronic portfolios: as an assessment tool, for marketing or employment, and to document the learning process and growth for learners of all ages, from pre-school through graduate school and into the professions. The purposes and goals for the portfolio determine the content. •
Types of E -Portfolios It can be helpful to think about e-portfolios in terms of when the work is organized relative to when the work is created. This results in three types of e-portfolios: 1. The showcase e-portfolio: Organization occurs after the work has been created.
246
•
Showcase e-portfolios: With so much material in digital form, a common starting point for e-portfolio thinking is to organize and present work that has already been created. A showcase e-portfolio enables the author (student) to share specific examples of work and to control who can see these collections, most simply by setting and then distributing passwords for different audiences. Although a showcase e-portfolio may look like a personal Web page, it is much more than that. The e-portfolio author should be able to organize and manage documents stored on the Internet and to control access even without knowing how to use HTML or build Web pages. Ideally, showcase e-portfolios should go beyond simply sharing work that has been completed. They should provide a stimulating context for reflecting on a body of work in order to make new connections, personalize learning experiences, and gain insights that will influence future activities. Without supporting reflection, a showcase e-portfolio can be reduced to merely a collection of artifacts. Structured e-portfolio: Another approach is to use a structured e-portfolio to establish a predefined organization in anticipation of work that will be completed. In a structured
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
e-portfolio, demonstrating accomplishments for certification or fulfillments of specific requirements is a common goal. By clearly articulating requirements, a structured e-portfolio can effectively focus a student’s time and attention. Furthermore, the predefined organization of a structured e-portfolio can make it easier for work to be systematically reviewed, evaluated, and compared. Because meeting a requirement or demonstrating a skill is not necessarily the same as taking a specific course, structured e-portfolios provide opportunities for developing new approaches to assessment. Although some institutions are beginning to use a structured e-portfolio approach to assist with student advising and career planning, others are developing a “learning matrix” of formal learning objectives and student outcomes as a way to ensure that an institution’s commitment to learning is being achieved by all students. Each objective has a descriptor of what the work should demonstrate, clarifying for the student what is expected and providing a common framework for advising on and assessing the competencies being demonstrated. Outside reviewers can play an important role in this approach by sampling student work to confirm that institutional goals are being achieved and by identifying curricular strengths and weaknesses. Some professions like elementary and secondary teaching have formal standards and certification requirements that candidates must meet regardless of the institution they are attending. The Center for Technology in Education (CTE) at Johns Hopkins University has developed a standards-based e-portfolio for teacher education as a replacement for the paper portfolios used in the Master of Arts in Teaching program. In the CTE electronic portfolio (EP), prospective teachers demonstrate their evolving skills in the context of established standards, local or state certification requirements, or standards required for their field. Participants can share and discuss work with peers,
request feedback from advisors, and use an online journal to reflect on their progress and growth as a teacher. At the end of a program, participants can submit their EPs for formal review and use them to showcase accomplishments for possible teaching positions. Supported mentoring can significantly enhance structured e-portfolios. Guiding and encouraging students through a sequence of experiences will better enable them to develop the skills they need to demonstrate required competencies. Without supported mentoring, a structured e-portfolio can be reduced to a set of directions that students follow to meet seemingly arbitrary requirements. •
Learning e-portfolios: Whereas a showcase e-portfolio is used to organize and present accomplishments and a structured e-portfolio can ensure that specified work will be done, the organization of a learning -portfolio is dynamic.
The organization of work evolves over time as tasks are identified, worked on, and completed in response to the student’s changing interests, requirements, and understanding. Students in the process of developing an e-portfolio can reach back in time across different activities to make new connections. This ongoing reorganization of work can be well thought out and clear, or it can be spontaneous and messy. Barrett (2005) described that learning or process portfolios involve the focus on Plato’s directive, “know thyself,” which can lead to a lifetime of investigation. Self-knowledge becomes an outcome of learning. In a portfolio study conducted with adult learners who were developing portfolios to document prior learning, Brown (2002) found the following outcomes: increased students’ understanding of what, why, and how they learned throughout their careers, and enhanced their communication and organization skills. The results of this study reinforce the importance of reflection in learning.
247
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
Figure 2. The learning portfolio (Zubizaretta, 2004, p. 20) Reflection
Learning Portfolio Documentation
Primary motive of a learning portfolio: “to improve student learning by providing a structure for students to reflect systematically over time on the learning process and to develop the aptitudes, skills and habits that come from critical reflection” (Zubizaretta, 2004, p. 15). Zubizaretta borrows from Peter Seldin’s work on teaching portfolios, and identifies three fundamental components of learning portfolios, as shown in the following diagram (Figure 2).
Integrated e -Portfolii An Innovative Approach to E -Learai Integrated e-portfolio can be described as a hybrid form of all the different types of portfolios described previously, having additional features that demonstrate learning in all the three domains of learning (cognitive, psychomotor, and affective) including interpersonal domain. Integrated e-portfolio special emphasis is on one’s own integration and conceptualization of the process of learning through linking of learning in different papers to the graduate profile (Figure 3). The basic concept of integrated e-portfolio is to allow students to understand the links between the different courses they study and its association with the concept of integrative learning. Integra-
248
Collaboration
tive learning comes in many varieties: connecting skills and knowledge from multiple sources and experiences; applying theory to practice in various settings; utilizing diverse and even contradictory points of view; and, understanding issues and positions contextually. Significant knowledge within individual disciplines serves as the foundation, but integrative learning goes beyond academic boundaries. Indeed, integrative experiences often occur as learners address real-world problems, unscripted and sufficiently broad to require multiple areas of knowledge and multiple modes of inquiry, offering multiple solutions and benefiting from multiple perspectives. The development of integrated e-portfolios allows students to self-evaluate their learning. This is a platform for “assessment for learning” as opposed to “assessment of learning.” Students while creating their e-portfolios demonstrate their own understanding of the links between different assessment tasks of each paper and across the core papers. Through their thinking log and self-reflection students demonstrate the process of learning by providing evidences as per the rubric given in Figure 4 (Bhattacharya, 2001). The present project has been initiated across three core papers in the new postgraduate program in distance and online education as mentioned. Assignment tasks for each of these papers were designed in such way that the students while
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
Figure 3. Framework for integrated e-portfolio
Figure 4. The rubric for identifying different phases of learning Metacognition & Progression Creativity & Innovation Conceptualization & Implementation Indices of Reflection
Evaluation & Modification Cooperation & Collaboration Critical Thinking & Decision Making
working through these tasks would be able to visualize the connections between the learning in each of these papers. Students are required to provide evidence (artifacts) for linking them to the graduate profile of the program they are enrolled in. Students enrolled in one paper/course only may link the artifacts to the achievement objectives/learning outcomes of the individual paper/course.
Admitive Issues Despite the many advantages of portfolios, there are also disadvantages, many related to their implementation: “Portfolios are messy to construct, cumbersome to store, difficult to score and vulnerable to misrepresentation” (Wolfe, 1999, p. 129), and there is always “the possibility of (portfolios) becoming a useless paper chase and a futile exercise” (Wheeler, 1996, p. 89).
249
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
Also, a lack of technical support and assistance (at both micro and macro levels) is seen as a major area of concern with Bloom and Bacon (1995, p. 2) “highlighting that especially new students may have difficulty with the lack of structure in the process.” In 2004 the integrated e-portfolio project received the grant under the Fund for Innovations and Excellence in Teaching from Massey University competitive internal research funds. In order to avoid some of the problems as mentioned previously, we decided to host the students’ portfolio accounts through an external service provider. After some survey we decided to host students’ portfolios at the John Hopkins University. In the future we intend to develop our own platform or create an open source platform for electronic portfolios. The choice of a software tool can have profound effects on the nature of the interaction within a networked learning community. As theorists and researchers on situated learning have argued, cognition is a social activity that emerges out of a system comprising individuals, a context of intentions, and the tools available (Greeno, Collins, & Resnick, 1966). Tools shape and structure action (Wertsch, Del Rio, & Alvarez, 1995) by empowering or constraining particular activities of the individuals who use those devices—that is, tools have distinctive profiles of affordances and constraints (Gibson, 1979). We will need to conduct further research studies before deciding on a generic e-portfolio platform for the number of different ongoing projects. In order to conduct any research where we need to quote students work on portfolios we will have to go through the process of securing ethics approval from the Massey University ethics committee. Copyright of the project remains with Massey University. Intellectual Property Rights remains with the authors of the products, for example, research publications, students’ assignments, and so forth. Implementation of
250
integrated e-portfolio across different disciplines and courses would definitely require organizing workshops, seminars, and demonstration of cases for faculty and administrative staff.
Evaluation of the Integrated E -Portfolio Project The integrated e-portfolio project described in this chapter is a work in progress. We have conducted an evaluation study of a mini integrated e-portfolio project based on an intensive summer school program for postgraduate students at the University of Tartu in Estonia (Bhattacharya & Mimirinis, 2005). It was a real eye opener for us to see how different the individual perception of the same course is. Following are some of the conceptual maps developed by the participants of the summer school as part of their integrated e-portfolio (Figures 5 and 6). This pilot study demonstrates the importance of individual’s knowledge structure for making any decision or passing any judgments about individuals learning outcomes. We perceive and understand any new learning experience by relating and accommodating it with our previous knowledge structure, our motivation, and our interest. Therefore, each of us constructs individual meaning from the same course material or instruction.
Students Reflection on the Process of Developing e -Portfolio Following are excerpts from two students’ responses to the usefulness of e-portfolio (posted online in a web-based course). Students have found developing e-portfolio and integrating the learning across different papers/courses in the degree program made it very useful for showcasing not only their work but also the progression in understanding and knowledge.
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
Student 1: Until this paper—and in fact the arrival of our “digital portfolio guide” by The John Hopkins University—I had never heard of “e-portfolios” period. When I did, my mind immediately arrived at the “showcase” idea where I could use an eportfolio to showcase myself as an individual from
an array of dimensions far beyond the constrict of a traditional CV, including: • • • •
Personal philosophy of education Artifacts of actual work I have done Supporting reflective documents that describe the artifact mechanism Personal goals short term and long
Figure 5. Concept map 1
Figure 6. Concept map 2
251
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
The biggest challenge for me was in actually sitting down—taking a look at myself and drawing up a personal philosophy in education—in writing, something I had never done before, something that felt personally “empowering” once completed. The development of the e-portfolio thus far has given me a greater sense of direction in my research and work—and something to look back on, assess myself against, challenge myself against, and develop myself against. So now it seems so much more than just a “showcase” of my work. Student 2: Developing the e-portfolio provided an opportunity to reflect on studies to-date and the achievement of objectives at a paper and program level. Engagement with the task has highlighted and clarified the links between the core papers, motivated me to go beyond one assignment and develop my e-portfolio to include completed papers (PGDipEd), and provided a potential focus for future Masters studies.
Concludiimarks and Fture Plaa It will take at least another year to determine the usefulness of the integrated e-portfolios for students enrolled in the distance and online education core papers at Massey University. Expected outcomes of this project include that students will on completion of their study in the core papers of the MEd (DistOnEd) have a portfolio of assessed work that will provide a presentation of key knowledge and skills developed through their study. Employers will be able to see in an electronic form student work that demonstrates their skills and knowledge. Students will through this process be led to reflect on their strengths and weaknesses in a manner that gives direction for future study. Creation and development of integrated e-portfolio will enable students to learn and develop research skills useful for project/thesis work in the later part of their study.
252
Working through this project made it clear that it is important for the faculty to work as a team, to take initiative, and share each other findings in order to be successful in any innovative endeavor. Integrated e-portfolio development allowed both students and the faculty to visualize individual student’s understanding of the different concepts and their links toward the formation of individual cognition. Finally the e-portfolios created through continuous activities and reflections demonstrated the process and the product of learning, which should be an integral part of any online course to minimize plagiarism. At Massey University we are keen on introducing the e-portfolio as an integral part of all the undergraduate and postgraduate courses. The policy development toward this initiative is already underway. Some of the pedagogical, administrative, and technical issues remain to be addressed before we can introduce e-portfolios as a compulsory component across all the programs in the university. Evaluation studies showed positive remarks by the students about the portfolio approach in their learning. Interactions and collaborations during the process of learning provided opportunities to distance students to develop interpersonal skills and build communities of practice.
ACKNOWLEDGMENT The design and development of integrated eportfolio work has been undertaken with the grant recieved from the Fund for Innovations and Excellence in Teaching- 2004, Massey University, New Zealand.
REFERENCES Barrett, H. (2005). Researching electronic portfolios and learner engagement (White Paper). In The REFLECT Initiative: Researching Electronic portFolios: Learning, Engagement and Collabora-
Introducing Integrated E-Portfolio Across Courses in a Postgraduate Program
tion through Technology. Retrieved November 30, 2005, from http:// www.taskstream.com/reflect/ whitepaper.pdf Barrett, H. C. (2001). ICT support for electronic portfolios and alternative assessment: The state of the art. Paper presented at the World Conference on Computers and Education (WCCE). Retrieved November 30, 2005, from http://transition.alaska. edu/www/portfolios/wccepaper.pdf Bhattacharya, M. (2001). Electronic portfolios, students reflective practices, and the evaluation of effective learning. AARE2001, Fremantle, Australia. Retrieved November 30, 2005, from http://www.aare.edu.au/01pap/bha01333.htm Bhattacharya, M., & Mimirinis, M. (2005). Australian Association for Educational Research, SIG report (p. 9). AARE News no. 52. Retrieved November 30, 2005, from http://www.aare.edu. au/news/newsplus/news52.pdf Bloom, B., & Bacon, E. (1995). Using portfolios for individual learning and assessment. Teacher Education and Special Education, 18(1), 1-9. Brown, J. O. (2002). Know thyself: The impact of portfolio development on adult learning. Adult Education Quarterly, 52(3), 228-245. Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton. Greeno, J. G., Collins, A. M., & Resnick, L. B. (1996). Cognition and learning. In D. Berliner & R. Calfee (Eds.), Handbook of educational psychology (pp. 15-46.) New York: Macmillan. Horizon Wimba: http://www.horizonwimba. com/
Macromedia Breeze: http://www.macromedia. com/software/breeze/ Massey University Elearning: http://elearning. massey.ac.nz/ Massey University GSE, CoE: http://education. massey.ac.nz/gse/study/postgraduate.htm Ministry of Education (2004). Interim tertiary e-learning framework. Retrieved November 26, 2005, from http://www.minedu.govt.nz Montgomery, K., & Wiley, D. (2004). Creating e-portfolios using PowerPoint. Thousand Oaks, CA: Sage Publications. ISBN 0-7619-2880-4. Nichols, M., & Anderson, W. G. (2005). Strategic e-learning implementation. Distance Education Association of New Zealand (DEANZ) Electronic discussions. Retrieved November 30, 2005, from http://deanz-discuss.massey.ac.nz/july2005. html Rogers, G., & Williams, J. (1999). Building a better portfolio. ASEE Prism, 8(5), 30-32. Wertsch, J. V., Del Rio, A. & Alvarez, A. (Eds.). (1995). Sociocultural studies of mind. Cambridge: Cambridge University Press. Wheeler, P. (1996). Using portfolios to assess teacher performance. In K. Burke (Ed.), Professional portfolios: A collection of articles. (pp. 74-94). Victoria: Hawker Brownlow Education. Wolfe, E. W. (1999). How can administrators facilitate portfolio implementation. High School Magazine, 6(5), 29-33. Zubizarreta, J. (2004). The learning portfolio. Bolton, MA: Anker Publishing.
John Hopkins University: http://cte.jhu.edu/epweb/ This work was previously published in Cases on Global E-Learning Practices: Successes and Pitfalls, edited by R. Sharma and S. Mishra, pp. 95-107, copyright 2007 by Information Science Publishing (an imprint of IGI Global).
253
254
Chapter XV
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement John LeBaron Western Carolina University, USA Carol Bennett WRESA Elementary & Middle Grades Curriculum Coordinator, USA
ABSTRACT Teachers and designers of computer-networked settings increasingly acknowledge that active learner engagement poses unique challenges, especially for instructors weaned on traditional site-based teaching, and that such engagement is essential to the progressive construction of learner knowledge. “Learner engagement” can mean several things: engagement with material, engagement with instructors, and, perhaps most important, peer engagement. Many teachers of computer-networked courses, who are quite diligent about incorporating activities and procedures to promote human interactivity, are confronted with the challenge of assessing the efficacy of their efforts. How do they discern whether the strategies and tactics woven into their “e-settings” are achieving the desired ends? This chapter outlines issues of self-assessment, including ethical questions. It lays out recommendations for self-assessment in a manner that respects student trust and confidentiality, distinguishing the demands of practical self-assessment from scholarly course research. The institutional pressures from which such assessment emerges are also examined.
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
INTRODUCTION Computer-supported collaborative learning (CSCL) outlined by Orvis and Lassiter (2006) makes a case for the active engagement of students in their own learning. These authors introduce certain challenges unique to computer-networked vs. face-to-face settings. Their commentary begs the question, “How do we know if our intentions work?” If we are truly committed to active student engagement and peer collaboration, then how do we gauge the achievement of our intentions? Orvis and Lassiter suggest that cognitive growth depends on a successful social construction of knowledge. If this is true, online instructors and designers need to devise techniques to discern the effectiveness of tactics and strategies incorporated into their course settings. Personal interaction is crucial to the success of all forms of teaching and learning (Laurillard, 2000; Swan, 2002; Vrasidas & McIsaac, 1999). Computer-supported learning allows for many kinds of interactions: one-to-one, one-to-many, or many-to-many. By itself, however, technology does not promote interaction. Technology requires human intervention in design and instruction to assure strong student engagement in networked settings (Harasim, 1993; Harasim, Hiltz, Teles, & Turroff, 1995; Kearsley & Schneiderman, 1999). Roblyer and Wiencke (2003) add that specific, deliberate activities are necessary to promote and support interaction among course participants. Inquiry into the questions of self-assessment in computer-networked learning environments has progressed little since the day when research concentrated on direct efficacy comparisons between computer-mediated and traditional classroom teaching. As computer-networked education was just emerging, Verduin and Clark (1991) reviewed 56 studies comparing the academic achievement of students in conventional classrooms to “distance learning” students. While focusing on student performance measured by grades, they found little or no distinction. Continuing this
“no significant difference” stream of research, Russell’s growing compendium of studies (2001) revealed no significant difference in student performance between learners in conventional classrooms and those enrolled in various types of “distance learning” courses. Based on such “no significant difference” research, findings to date have indicated that distance learning, in a variety of modalities, typically matches or exceeds teaching, at least when effectiveness is gauged by student perceptions or performance measured by, say, their course grades. These studies, however, provide little insight beyond that indicated by survey results or student transcripts. They fail to reveal much about the qualitative nature of the compared learning environments, and leave unanswered such other questions as: Do different teaching modalities transform traditional instructional media into significantly different learning experiences? What tactics and strategies do particular teaching and learning settings enable to promote the kinds of student growth sought by the course designers and instructors? Several scholars have decried the persistent failure of scholarly research to analyze academic practice deeply or to improve it (Brown & JohnsonShull, 2000; Phipps & Merisotis, 1999). Ehrmann (1997) suggests that most research comparing technology-infused with traditional teaching fails to address important substantive questions about distance education. Indeed, comparisons between alternative modes of instruction are meaningless because they typically fail to account for the innumerable and complex variables that distinguish different types of learning environment. As Ramage (2002) points out, the research deficiencies on effective higher education teaching are by no means limited to the analysis of education-at-a-distance. Research on classroom practice is similarly weak. Comparative “modality research” assumes that technology does little more than assume a simple replication of what is occurring in
255
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
conventional classrooms. Rather than thinking about technology as the electronic equivalent of the conventional classroom, researchers might instead ask how technology could transform, say, a 100-student introductory physics lecture into a rich multidisciplinary “conversation” that links the humanities and the arts with related concepts in physics where students participate actively and collectively in the distribution of knowledge? In this way, technology becomes transformative rather than adaptive. Self-assessment attempts, therefore, become multifaceted, formative, and naturalistic as instructors struggle to assess process, at least as much as products represented in transcripts of student work. Aside from the common-sense proposition that the quality of teaching and course design is advanced by deliberate attempts to gauge efficacy, the practice of self-assessment is theoretically supported by a more critical application of the rich “reflective practitioner” literature (Bengtsson, 1995, 2003; Schön, 1987a, b). The “reflective practitioner” (RP) concept has informed the education field for several decades. It has its roots in the progressive schooling movement of Europe and America in the 1930s and 1940s, but more recently has been championed by the late Donald Schön. Schön’s work is a response to rigidly positivistic accountability pressures so globally prevalent in contemporary educational policy. He urges higher education teaching to focus significantly on “reflective practica,” where teacher intuition interplays with “received knowledge” (i.e., formal research) in close coaching environments where the coach assumes more the Socratic role of “critical discussant” than of “knowledge dispenser.” Moving from theory back to common sense for a moment, the argument that thoughtful reflection about one’s own practice somehow impedes the objective assessment of efficacy defies logic, so long as such reflection is supported by data and by theory. Rigorous RP is challenging. As Schön has noted, “…The introduction of a reflective
256
practicum ... is an uphill business.... If you think about introducing a reflective practicum into ... education you must work against the view that practice is a second-class activity” (1987a). A discussion about self-assessment carries implications for evaluative technique, for ethical research behavior, and for the institutional context where the outcomes of these discussions often carry rather high career stakes, especially for instructors. This chapter attempts to advance this discussion and to suggest future directions that it might take. Between them, the authors have been engaged in designing computer-networked learning environments since 1999. Several online graduate education courses taught between 2003 and 2006 serve as case histories for the ensuing discussion. The first author designed and taught two of these courses through several iterations at two universities. The second author is a former student in two of these courses. She has gone on to design her own online learning projects for the professional development of school teachers. Her particular student perspectives appear in narrative boxes throughout this chapter.
Assessiih Efficacy of Collaborative Lagagement Hill, Han, and Raven (2001) suggest that the use of group work contributes positively to a sense of belonging and connection within Web-based courses. Other pioneers such as Haythornthwaite, Kazmer, and Robins (2000), and Lock (2002) challenge CSCL designers, therefore, to assess the processes of interaction and engagement among learners. Online course instructors and designers need to distinguish between the demands of formative self-assessment and summative public research as they plan to ascertain the effectiveness of the processes they use to promote student collaboration.
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
Self-assessment is formative in nature. Although clear ethical procedures require much care in the collection and analysis of data, the purpose of self-assessment is to determine what works in order to improve future instruction. Research, on the other hand, is by definition a public act, therefore requiring greater rigor and more attention to ethics, not only in reporting results, but also in collecting and analyzing information. With the growing acceptance in higher education of formal inquiry into teaching, (e.g., the Scholarship of Teaching and Learning (SoTL) as legitimate research for career advancement, online course evaluation might assume that today’s private self assessment may become tomorrow’s published research (Boyer, 1990; Hutchings, Babb, & Bjork, 2002). We shall return to these issues later, but published research demands more rigorous human subject safety review than does the data collected and analyzed for the private assessment of an individual teacher. Having said this, important distinctions exist between self-assessment and scholarly research. Research expects to advance disciplinary knowledge in a rigorous manner that observes certain rules of inquiry within a community of critical peers. Self-assessment, on the other hand, is dedicated to the discovery of efficacy and achievement related to more narrowly-conceived instructional purposes. Although such discovery may ultimately become part of a research agenda, observing the relatively harsher rules of research may constrain many useful assessment techniques. For example, self-assessment typically addresses very small subject samples. In most cases, it need not undergo the data coding and analysis required of formal research. There is neither the possibility nor the need to generalize the assessed findings to other settings. The assessor is attempting to find out what is effective in this course, with this population of students at this particular time. When and where to start? As with most endeavors, knowing how well one does something depends on a clearly-articulated awareness of what
was intended in the first place. As banal as this notion may seem, the principle is often overlooked and difficult to sustain. So, if course activities are structured to promote peer collaboration, it makes sense, as Orvis and Lassiter (2006) advise, to embed such activity early in the course and to assess it in several ways as soon after completion of the exercise as possible. Among the course activities analyzed for this discussion, was an initial “Icebreaker” assignment. Woods and Ebersole (2003) researched the usefulness of personal information-sharing in nonsubstantive discussion boards in an online course. They discovered that the use of autobiographies helped foster positive relationships within the course community for students and instructor alike. Many learning management platforms (LMS) offer little or no default information to students about their classmates, a glaring deficiency if the instructor means to promote frequent, deep, purposeful dialogue throughout the study period. Therefore, upon enrollment in several of the authors’ courses, an early assignment required students to submit more detailed biographical data via a hot-linked browser-based form. Students were asked to list their primary professional and recreational interests and provide more detailed contact information than a simple e-mail address. Most students also capitalized on the option to file-attach a personal photo. This information was then shared on a separate course page, shown in Figure 1. Based on these online biographies, students were required to undertake a simple “get to know you” assignment, called the “Icebreaker,” within the first three weeks of the course. John Cowan, a professor well known for his work on student-centered online learning at the British Open University, has advised that everything an instructor truly values should be assessed as part of the overall calculation of student performance (personal communication, June 10, 2001). Therefore, 10% of the course grade was attributed to the
257
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
Figure 1. The instructor provides an online form through which students submit contact information and biographical data. Additionally, they send photos of themselves as e-mail attachments. From these submissions, the instructor created a “student mini-biography” Web page to post as a course resource for several collaborative group assignments.
completion of this assignment, even though it was not directly associated with the course content. It should be noted here that this aspect of the grade depended only on assignment completion; in no way was the content of the student contributions judged—it was only reported. Students were asked to review the online student biographies and begin a dialogue within private small teams created by the instructor within the asynchronous discussion boards of the LMS.
Former Student Perspective on the Curse Icebreaker Assignment The Icebreaker activity was a comfortable way to quickly get involved in the course. Right from the start students had an assignment deadline; but one with which we could almost certainly be successful. It was an authentic way to get us actively engaged from the get-go. An additional benefit was that I found I learned more about my classmates than I had in other classes where we met face-to-face on a weekly basis. As our
258
coursework projects progressed throughout the semester, I frequently referred back to the student biographies to get a clearer understanding of my classmates’ perspectives. The major purpose of this activity was to build an immediate sense of community within the course as a foundation for subsequent, substantive knowledge construction. To ascertain whether this purpose was met, the instructor surveyed the students anonymously through an online form that asked on a four-point Likert scale, among other questions about this exercise, if the activity promoted the intended sense of community. The scaled questions were supplemented by an openended textbox question requesting additional comments about the Icebreaker. Over several course iterations, on the scaled questions, 96% of respondents answered affirmatively that, yes, the assignment promoted a sense of community. Zero percent responded negatively, and 4% selected “N/A” for reasons that are not clear. Prominently mentioned comments
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
offered in response to the open-ended textbox question were, “The icebreaker was a good way to launch the course”; “The Icebreaker offered a good method to meet classmates”; “The Icebreaker encouraged longer-term dialogue among students”; and “The expanded student list, with photos, was useful.” A helpful way to obtain information on student perceptions about various aspects of teaching and design is through the use of online forms that ask questions keyed explicitly to instructional purpose. In this case, students were asked directly if the intent of the exercise was achieved. Such forms need not be long. They may be used repeatedly for the instructor to build knowledge about the relative success of their intentions formatively and incrementally. Provided that the forms are neither onerous nor excessively time-consuming, students appreciate being asked about their perceptions, especially if they see mid-course instructor corrections made in response to them.
Former Student Perspective on Multiple Polling about Student Perception As a student, I especially appreciated the fact that the instructor cared enough to elicit student feedback during the course, rather than waiting until the end, when any changes would have little effect on current students. It was a highly effective strategy for initiating student-teacher trust. Online forms allow for a wide variety of question types to be asked of students in a single sitting, from scaled ratings to open-ended textbox narratives. By encouraging students to comment in their own words in addition to rating preformatted statements about course design and teaching, instructors can obtain a relatively deep, personalized sense of their student’s authentic feelings about their experiences. The usefulness of different kinds of information-gathering in online course evaluation is stressed by a host of researchers,
including Hammond and Winiyapinit (2005), Hara, Bonk, and Angeli (2000), and Naidu and Järvelä (2005). Our assessment of the Icebreaker, however, could have employed even more robust data sources than it did. For example, transcripts of the students’ conversations in this activity could have been qualitatively analyzed according to objectives established in advance for the exercise by the instructor or the course designer. Like, we were saying; yes, really saying... Student learning is enhanced when diverse media of expression are made available in online course settings. As a result of infusing media streaming into her undergraduate biology courses, Michelich (2002) reports anecdotally on improved student class participation and performance on assignments and exams. Literally giving an audible “voice” to students provides a naturalistic environment for discussion and collaboration. By themselves, most learning management systems fail to offer students such “voice,” but after-market conferencing tools such as Centra™ or Elluminate™ may be power-linked to most learning management systems to provide diverse channels of dialogue in a variety of media. In this case, the instructor established team and whole class dialogues in the conventional textbased asynchronous discussion boards in a manner that challenged students to respond not only to the weekly questions posed by the instructor but also to one another in progressive threads. There is nothing unusual about this kind of activity, but through an asynchronous voice tool, powered by Horizon/Wimba™, students were led directly from the text discussion to a threaded voice environment where students were encouraged to talk directly together in their real voices. Because this voice communication capacity is not seamlessly integrated with other LMS communications tools, navigation between the text and the voice utilities is awkward, requiring students to back out of one communication protocol and redirect their steps into a different one. The challenge for instructors, therefore, is to make this
259
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
Figure 2. The instructor indicates in the standard text discussion board (lower left corner of the figure) that he has responded by voice. On opening his message to this effect, the student opens a message containing a hot-link taking her or him directly to the voice board where the message resides.
cumbersome procedure more seamless so that students would actually use it and benefit from doing so. With the help of University technical staff, an HTML script was created that could be inserted into any text discussion message with the result that a simple click would instantly open a new Wimba window side-by-side with the original text message. Students then had the choice of pursuing the conversation by voice or in text, with no prejudice attached to their choice. Figure 2, below, shows the screen appearance of this procedure.
Fudent Perspective on Linking Text Discussions to Voice Treads This was an effective strategy to encourage student engagement and collaborative peer interactions. The asynchronous discussion boards facilitated our getting to know one another at an even deeper level and made our discussions much richer.
260
Additionally, the daily postings encouraged our frequent participation, partly through curiosity about what our peers were saying and additionally, from our innate desire to excel in the class. Two assessment techniques were used to assess whether this technique met its major purpose: one formative and one summative. Shortly after Wimba was launched for the purpose outlined above (midway through the course), a threaded discussion group was established for anonymous postings. The instructor asked students to respond anonymously to the following question: “I am using voice messaging … for my responses to your discussion postings.... I’d like to know how it’s working for you. I have added two discussion topics explicitly requesting your feedback. These topics enable anonymous postings. I will not know the originator of any message unless you choose to sign your posting…. Please use your own words to respond.”
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
Former Student Perspective on the U se of Voice Messaging I enjoyed the voice messages because it felt like personal feedback from the instructor. Often in face-to-face classes, there is little time to speak to the instructor one-on-one. The voice messages felt more like I was getting individual attention. When students started adding their voice messages, it added another dimension to the dialogue and strengthened the lines of cooperative communication. If another classmate left a voice message for me, I felt compelled to answer in a more effectual manner than I did with just the text messages. Supplementing this request for anonymous student narratives were one open-ended and two scaled questions about the use of Wimba during the course. Thus, three data sources were employed for this one procedure that the instructor had used for the first time. The results reflected the “rookie” status of this undertaking. In their scaled end-of-course evaluative responses, six students agreed that this use of Wimba “helped create a personal dimension in [their] dialogue with the instructor”; five disagreed. Five agreed that Wimba “was a useful medium for [them] to make good use of instructor feedback”; an equal number disagreed. Two representative comments in the open-ended summative student evaluation were, “Make Wimba required for discussions and chats (otherwise folks will default to the traditional text)” and “Either use Wimba throughout course or not at all.”
Former Student Perspective on Mandated use of Voice Cmmunication I actually appreciated the choice the instructor provided for us to use text or Wimba. I personally would not have liked Wimba to have been mandated throughout the course because I enjoy the reflective time that text messaging allows.
However, I did enjoy using the voice boards occasionally as a nice change of pace from the text. In my opinion, using both voice and text messaging helped to meet the diverse needs and learning accommodations of different students. Narrative student responses in the midterm anonymous discussion board comments reinforced the judgments conveyed in the final online student evaluation. Excerpts from several randomly selected comments appear below: •
•
•
•
“I appreciate the fact that you are extremely active in the course.... The only thing I find a little frustrating is that there are so many places [within the course setting] to look.” “I like receiving voice messages. However, I am a visual learner so when I ask specific questions, I like to have an answer written out for me that I can print and place in my notebook for this class.” “I do like the Wimba option. However, I must agree with some of the other responses that with email, and several discussion boards going at one time that I often get overwhelmed with all the things going on in this class.” “[Text] discussion threads and e-mail are currently better for me.”
One student chose to identify herself in her response. She wrote, in part, “I am getting used to the Wimba chat feedback, although I am much more familiar with the text chat. I like the fact that I can print, review and keep the text responses. I also feel more comfortable responding to a text chat.” The salient points here are not so much what the students indicated as how the instructor interprets the information and what he does with it. Assessing instructional efficacy can seem harsh when the instructors fail to remove their egos from the information conveyed. In this case, the message seemed clear. Although students appeared to appreciate the effort put into this innovation,
261
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
it was not promoting student satisfaction or the peer dialogue intended. The varied data sources pointed to the highly plausible interpretation of a collective student perception, namely a lukewarm endorsement. As a result, the exercise either had to be improved or abandoned. Not yet convinced that abandonment was indicated, for his next online course, the instructor decided to follow one specific direction suggested in the end-of-course student evaluations, and to reject another. He enabled the Wimba asynchronous tool immediately as the course was launched, and maintained its use throughout the semester. However, because a major purpose of using it was to give students communicative choices according to preference and style, he did not require it. Thus, the revised Wimba discussion seemed more confident and less tentative. Students accepted it from the outset, and their end-of-course evaluations reflected substantially higher levels of satisfaction. From a total of 11, nine students agreed that the use of Wimba “helped created a personal dimension in [their] dialogue with the instructor” (four strongly); two disagreed (none strongly). Eight agreed that Wimba “was a useful medium for [them] to make good use of instructor feedback” (three strongly); three disagreed (none strongly). In this way, an end-of-course evaluative summation became formative for improving the next course iteration. Assessment results of these and other strategies embedded in the authors’ courses are found in several other publications (LeBaron & Miller, 2004; LeBaron & Miller, 2005; LeBaron, Pulkkinen, & Scollin, 2000; LeBaron and Santos, 2005).
Broader Discussion of CSCL Efficacy The examples provided above prompt a more general dialogue about quality assessment of instructional strategies to promote student collaboration. In the foregoing discussion, we have touched upon issues of purpose, ethics, and the
262
use of multiple data sources and application of assessment results. The importance of keeping keenly mindful of the process intentions was stressed, so that queries about efficacy could be keyed specifically to these intentions. Put another way, if one wishes to know what students think about something; find different, safe ways to ask them about that particular thing (Henri, 1991). Transcript analysis. To deepen the authenticity of the assessments described above, more can be done. For example, assessment of the Icebreaker and the asynchronous Wimba discussions depended largely on various expressions of student perception gleaned through ratings and comments submitted via Web-based interactive forms provided by the instructor either at a mid-point or at the end of a course. Assessment depth, however, would be enhanced by a focused analysis of the transcripted conversations actually undertaken during the Icebreaker. Such transcripts might be supported by performance rubrics describing various levels of depth, responsiveness and relevance embedded in the student postings. Rubrics need not be used for grading student performance (although they may be). In this scenario, they would be used to assess how well the instructor has designed an exercise to achieve the goals originally established for it. In depth, qualitative transcript analysis has been promoted for instructional self-assessment by much commentary in the field (Gerbic & Stacey, 2005; Mason, 1991; Naidu & Järvelä, 2006). Successful transcript analysis depends on knowing what information is desired. For example, is the assessor seeking evidence of the collective construction of knowledge, evidence of critical thinking, social engagement, depth of dialogue, substantive analysis of course content, or some combination of the above? For the purpose of this discussion, the choice does not matter much. What matters is that assessors know exactly what they’re looking for so that they might develop a coding system that accurately produces the answers geared to informational categories determined in advance.
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
Third-party interviews. Although transcript analysis offers a potentially rich leg on the triangulation tripod, an equally valuable strategy is the third-party interview. Such interviews are conducted under the overall guidance of the instructor or course designer seeking to secure “richer, thicker” insights into student perceptions about particular instructional purposes. Such interviews are best conducted within the month after a course has ended. Conducting them in-progress can seem threatening to students, no matter how convincing the assurances of confidentiality and anonymity. The third party may be an unbiased faculty colleague, a teaching assistant, or a peer student—someone who understands the course and its intentions but has no vested interest in the responses or the identity of those making them. Interviews may be conducted in-person, by telephone or through e-mail. Typically, such follow-up is undertaken only with a relatively small sample of a whole course enrollment. A randomly-selected “A list” of subjects is identified, with “B list” substitutes if students initially selected are unable or unwilling to participate. This assisting assessor reveals only the raw aggregated results of the interview. It is then up to the instructor to code and interpret the results according as rigorously as deemed necessary for the assessment purpose. LMS “log counts.” Campos (2004) and Mason (1991) have pointed out that simple “log counts” of discussions provided by most LMS tracking tools are of limited value because they indicate nothing about the content, relevance, scholarship, or responsiveness to instructional purpose of the student postings. This is not to say that logs should not be used, just that if they are used, it should be understood that they offer little more than unqualified counts. They may, however, serve to reinforce other, richer sources of information. As Garrison, Anderson, and Archer (2001) suggest, any single data source is by definition unacceptably limiting for deep understanding.
Outside the formal assessment box. There are many ways outside a formal program of assessment for online course designer-instructors to gain information about the degree to which their intentions are being achieved. Here, the assessor must be particularly attentive to ethical issues. For example, private e-mail provides a very rich vein of information; because it is private, however, it is not the business of public research; indeed, it may concern private assessment in only a limited way. E-mail among students should not even be viewed by the instructor, and a message directed to the instructor in a moment of frustration is best forgotten as a deep indicator of anything beyond the momentary frustration. The same principle applies to student chatter in private chat rooms, or “virtual cafés” established explicitly for their socialization, but visible to the instructor. More appropriate, though more challenging, are longitudinal assessment techniques, undertaken, again through third-party interview or anonymous survey, a significant period of time after a course has ended. In some of the applied sciences, for example, many graduate course syllabi contain certain goals of workplace application (Boyer, 1990). A curriculum course typically proposes that the theoretical scholarship tackled in class will be applied in subsequent professional curriculum work. One or more years after the course, it is useful to return to the course syllabus, to extract the goals of scholarly application, and to ask the former enrollees incisive questions about their particular applications to practice. Ethics. How can assessors assess, and how should they not? Is what they can do, what they ought to do? Clearly, the success of any online instructional assessment, especially when it comes to CSCL, depends on faculty-student trust. Deep trust emerges from course to course over time. It is difficult to build, but it can be lost in a split second. Trust is related to a respect for regulation and policy, but it is much more than that. As discussed above, rules of human subjects’ research apply in greater measure to evaluations
263
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
destined for broad publication, than for the private application to course improvement. If research is intended, or even remotely anticipated, Institutional Review Board (IRB) human subjects’ rules should be observed. Even here, however, the rules are less important than trust, laboriously earned and scrupulously maintained. Elements of trust include confidentiality, anonymity, and intellectual honesty in interpreting the information gleaned. As a rule of thumb, all evaluative data should be aggregated for any public discourse about the efficacy of course activity. There should be no possibility, directly or by implication, of connecting any individual student to any particular sentiment. This is especially true when student scholarly performance is being used to assess the quality of course activity. To greater or lesser degrees, the same may be said of student content analysis and postings in open or closed discussion forums. The literature on computer-networked assessment ethics is quite rich. Browne (2003), King (1996), and Anderson and Kanuka (2003) suggest the student permission is sine qua non for any published research related to course activity. If such permission is withheld for any reason, the transcripts of those students must be excluded from analysis whether or not it compromises the substantive integrity of the conversational thread. The problem here, of course, is that the ethical “bathwater” might jettison the informational “baby” (Rourke, Anderson, Garrison, & Archer, 2001). A more flexible approach may be taken toward unpublished private self-assessment, but assessors must protect themselves by anticipating the possibility of future research. As King points out, all self-assessment using the content of student work as a measure must be accompanied by informed consent, outlining the assurances of anonymity and confidentiality, the potential harm that the students might endure, even in the remotest circumstances, and procedures for the student to opt out. In order for assessed student information to carry legitimacy, it should not, in most cases, be 264
traced to individual students. Lack of confidence in anonymity may corrupt the integrity of any student’s comment, even after a course terminates because the same student may be enrolled in a future course, or the course of a friendly colleague. As Anderson and Kanuka (2003) affirm, the debate on ethics in assessing course attributes based on student work is still emerging. Human subjects’ research guidelines offer some guidance in this respect. It may be best to assure safety of students and assessors by erring on the side of caution. Closing the conversational circle. The foregoing discussion presents a workload potential capable of filling an instructor’s entire professional time card. As Hara, Bonk, and Angeli (2000) suggest, in-depth online instructional assessment can be so exceedingly time-consuming that it fails to get done. Fortunately, strategies exist for managing time without unduly compromising assessment results. For example, “mountain” of data can be reduced to more manageable “hills” through various sampling techniques. Transcripts may be selected from representative segments of a course, for example, every third week of a fifteen-week course (Pena-Shaff & Nicholls, 2004; Santos & LeBaron, 2006). Interviews may be conducted with a random sample of enrolled students. Qualitative information may be coded for efficient analysis. The question of personal ego was raised earlier, with the suggestion to strip it away from the analysis of assessed student perceptions. This is not entirely true, however. Professional educators must trust their own deep intuitive sense about the meaning of the information they collect. For example, Virginia Michelich’s sense (2002) about the efficacy of streamed media in her undergraduate biology course is based largely on evidence she calls anecdotal. By “anecdotal,” she means her experienced, expert, professional analysis of student work quality. For the purposes of formative self-assessment, such conduct is perfectly acceptable. Michelich appears confident enough to know (and report publicly) what works well in her courses when she sees it.
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
The Institutional Stakes The practical implications of the foregoing discussion occur in the real world of higher education policy. The usefulness of assessing CSCL or even attempting to integrate it into learning environments in the first place, depends on the prevailing scholarly values of its sponsoring institutions. Typically, but not always, these institutions are colleges or universities. Universities pose special challenges. Alex Wright (2005) recently penned a discussion about online learning’s metamorphosis from a speculative entrepreneurial bubble in the 1990s into mainstream higher education practice today. Parts of Wright’s article, “From Ivory Tower to Academic Sweatshop,” are troubling to faculty who are passionate about teaching quality. Reporting on one particular university’s business model for online learning, Wright declared that this institution “has built its business through economics of scale, developing a course once and then replicating it, so that many teachers can administer the course to the school’s 200,000plus student body.” Such an approach to online education has been described as a “package and deliver” approach where course components are bundled into deliverable products requiring only token human facilitation to service student access and keep instruction moving within defined timeframes. Recognizing the threat of commercial competition from nontraditional newcomers to higher education, institutions developing computer networked learning programs need to engage in deep developmental thinking, guided by their educational missions, to compete effectively with the lower-cost “package and deliver” strategies of learning distribution so common in certain marketplace sectors (Pulkkinen, 2005). For institutions committed to personalized teaching excellence, viable markets will continue to exist for high-quality post-secondary teaching (Twigg, 1998). Designers and instructors in such institu-
tions require support for assessing the collaborative processes they build into their courses. Universities following the lure of scholarly commoditization embrace a model reminiscent of that voiced by Arizona State University’s Steve Salik who opined to Wright (2005), “Our professors are content experts. That’s all they are” (emphasis ours). Such a sentiment can come only from someone who knows exponentially less about teaching and learning than about marketing. While universities can ill-afford to ignore market conditions, doing so at the expense of their core educational missions embodied in highly skilled teaching faculty members harms students and demeans the scholarly integrity of educational institutions. In such a setting, the professor is divorced, among other things, from the pedagogical creation of processes to promote learner collaboration. Such a setting renders discussion about collaborative learning, and the assessment of it, entirely moot in a world that may indeed be new but not particularly brave.
CONCLUSION AND FUTURE TRENDS While we do not deny the value of rigorous research in the process of efficacy evaluation in online teaching and course design, we point out that useful self-assessment procedures are available which may not meet the most rigorous standards of scientific research. Nonetheless, these may give online instructors valuable information about how they are doing with their efforts to promote computer-supported learner engagement. We encourage teachers and designers judiciously to undertake such measures, because if we hold all self-assessment to the strictest rigors of pure research, then we may end up with very little of it, much to the detriment of teaching quality. This chapter has advocated the use of multiple data sources collected over time, formatively and summatively. We have particularly urged the use
265
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
of various data-gathering techniques to assess the efficacy of designs to promote learner engagement. We have focused on student perception polling. As a sole data source, the value of student perception may be limited. Actual student performance must also be taken into consideration. In doing so, however, assessors and designers need to think about what they mean by “performance,” distinguishing in their own minds the relative virtues of process vs. product outcomes. In some courses, process is very important. Indeed some courses are primarily about process. Strategies for the effective assessment of learner engagement apply to all instructional formats, whether computer-enabled or not. Fortunately for conscientious online teachers and developers, networked computing not only provides ever-richer potential for promoting learner engagement, it also offers more robust tools for assessing the efficacy of such engagement. Synchronous and asynchronous discussion and conferencing tools enable powerfully-mediated channels for communication during any course, at its termination, and retrospectively after the ostensible benefits of study have “sunken in.” Qualitative software support for the analysis of discussion transcripts, interviews, and open-ended narratives encourages increasingly sophisticated synthesis of assessment results (di Gregorio, 2000). In short, the future for the enrichment of online learner engagement and for the assessment of it appears bright indeed.
REFERENCES Anderson, T., & Kanuka, H. (2003). E-research: Methods, strategies and issues. London: Allyn and Bacon. Bengtsson, J. (1995). What is reflection: On reflection in the teaching profession and teacher education. Teachers and Teaching, 1(1), 23-32. Bengtsson, J. (2003). Possibilities and limits of self-reflection in the teaching profession. Studies in Philosophy and Education, 22(3/4), 295-316. 266
Boyer, E. (1990). Scholarship reconsidered: Priorities of the professoriate. Princeton, NJ: The Carnegie Foundation for the Advancement of Teaching. Brown, G., & Johnson-Shull, L. (2000, May/June). Teaching online: Now we’re talking. The Technology Source. Retrieved November 10, 2007, from http://technologysource.org/article/teaching_online/ Browne, E. (2003). Conversation in cyberspace: A study of online learning. Open Learning, 18(3), 245-259. Campos, M. (2004). A constructivist method for the analysis of networked cognitive communication and the assessment of collaborative learning and knowledge building. JALN, 8(2). Retrieved November 10, 2007, from http://www.sloan-c.org/ publications/jaln/v8n2/index.asp di Gregario, S. (2000). Using NVivo for your literature review. Paper presented at the Strategies in Qualitative Research: Issues and Results from Analysis Using QSR NVivo and NUD*IST Conference, London. Retrieved November 10, 2007, from http://www.sdgassociates.com/downloads/literature_review.pdf Ehrmann, S. (1997). Asking the right questions: What does research tell us about technology and higher learning? In Engines of inquiry: A practical guide for using technology to teach American culture. Retrieved November 10, 2007, from http://www.georgetown.edu/crossroads/guide/ ehrmann.html Garrison, R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence and computer conference in distance education. The American Journal of Distance Education, 15(1), 7-23. Gerbic, P., & Stacey, E. (2005). A purposive approach to content analysis: Designing analytical frameworks. The Internet and Higher Education, 8(1), 45-59.
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
Hammond, M., & Wiriyapinit, M. (2005). Learning through online discussion: A case of triangulation in research. Australasian Journal of Educational Technology, 21(3), 283-302. Hara, N., Bonk, C. J., & Angeli, C. (2000). Content analysis of online discussion in an applied educational psychology course. Instructional Science, 28(2), 115-152. Harasim, L. (1993). Networlds: Networks as social space. In L. Harasim (Ed.), Global networks: Computers and international communication (pp. 15-34). MIT Press: Cambridge, MA. Harasim, L., Hiltz, S. R., Teles, L., & Turroff, M. (1995). Learning networks: A field guide to teaching and learning online. Cambridge, MA: MIT Press. Haythornthwaite, C., Kazmer, M., & Robins, J. (2000). Community development among distance learners: Temporal and technological dimensions. Journal of Computer Mediated Communication, 6(1). Retrieved November 10, 2007, from http:// www.ascusc.org/jcmc/vol6/issue1/haythornthwaite.html Henri, F. (1991). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collaborative learning through computer conferencing. The Najaden papers (pp. 117-136). New York: Springer-Verlag. Hill, J. R., Han, S., & Raven, A. (2001, November). Build it and they will stay: A research based model for creating community in Web based learning environments. In Proceedings of the National Convention of the Association for Educational Communication and Technology, Atlanta, Georgia (pp. 192-199). Hutchings, P., Babb, M., & Bjork, C. (2002). The scholarship of teaching and learning in higher education: An annotated bibliography. The Carnegie for the Advancement of Teaching. Retrieved November 10, 2007, from http://www.
carnegiefoundation.org/dynamic/downloads/ file_1_196.pdf Kearsley, G., & Schneiderman, B. (1999). Engagement theory: A framework for technology-based teaching and learning. Retrieved November 10, 2007, from http://home.sprynet.com/~gkearsley/ engage.htm King, S. A. (1996). Researching internet communities: Proposed ethical guidelines for the reporting of results. The Information Society, 12(2), 119-127. Laurillard, D. (2000). New technologies, students and the curriculum: The impact of communication and information technology on higher education. In P. Scott (Ed.), Higher education re-formed (pp. 133-153). London: Falmer Press. LeBaron, J., & Miller, D. (2004). The teacher as “agent provocateur”: Strategies to promote community in online course settings. In T. Latomaa, J. Pohjonen, J. Pulkkinen, & M. Ruotsalainen (Eds.), eReflections: Ten years of educational technology studies at the University of Oulu (pp. 109-125). Oulu, Finland: Oulun yliopiston kasvatustieteiden tiedekunnan. Retrieved November 10, 2007, from http://gse.uml.edu/lebaron/Oulu-TAPfinal_ 030922.pdf LeBaron, J., & Miller, D. (2005). The potential of jigsaw role-playing to promote the social construction of knowledge in an online graduate education course. Teachers College Record, 107(8), 1652-1674. LeBaron, J., Pulkkinen, J., & Scollin, P. (2000). Promoting cross-border communication in an international Web-based graduate course. Interactive Electronic Multimedia Journal of Computer-Enhanced Learning, 2(2). Retrieved November 10, 2007, from http://imej.wfu.edu/articles/2000/2/01/index.asp LeBaron, J., & Santos, I. (2005). Authentic engagement of adult learners in online settings.
267
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
MountainRise, 2(1). Retrieved November 10, 2007, from http://facctr.wcu.edu/mountainrise/archive/ vol2no1/html/authentic_engagement.html Lock, J. V. (2002). Laying the groundwork for the development of learning communities within online courses. Quarterly Review of Distance Education, 3(4), 395-408. Mason, R. (1991). Evaluation methodologies for computer conferencing applications. In A. R. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp. 105-116). New York: Springer-Verlag. Michelich, V. (2002, January/February). Streaming media to enhance teaching and improve learning. The Technology Source. Retrieved November 10, 2007, from http://technologysource. org/article/streaming_media_to_enhance_teaching_and_improve_learning/ Naidu, S., & Järvelä, S. (2006). Analyzing CMC content for what? Computers & Education, 46(1), 96-103. Orvis, K. L., & Lassiter, A. R. L. (2005). Computer-supported collaborative learning: The role of the instructor. In S. P. Ferris & S. H. Godar (Eds.), Teaching and learning with virtual teams (pp. 158-179). Hershey, PA: Information Science Publishing.
Pulkkinen, J. (2005, March 14). Online learning: Enablers and barriers for educational technology. Paper presented as visiting scholar at Western Carolina University, Cullowhee, NC. Ramage, T. R. (2002). The “no significant difference” phenomenon: A literature review. Retrieved November 10, 2007, from http://www.usq.edu.au/electpub/e-jist/docs/ html2002/ramage.html Roblyer, M. D., & Wiencke, W. R. (2003). Design and use of a rubric to assess and encourage interactive qualities in distance courses. The American Journal of Distance Education, 17(2), 77-98. Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education, 12, 8-22. Russell, T. L. (2001). The no significant difference phenomenon: A comparative research annotated bibliography on technology for distance education (5th ed.). Montgomery, AL: The International Distance Education Certification Center (IDECC). Santos, I., & LeBaron, J. (2006). Ethical constraints in the valid interpretation of transcribed communication in online study. The Internet and Higher Education, 9, 191-199.
Pena-Shaff, J. B., & Nicholls, N. (2004). Analyzing student interactions and meaning construction in computer bulletin board discussions. Computers & Education, 42(3), 243-265.
Schön, D. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco: Jossey-Bass.
Phipps, R., & Merisotis, J. (1999). What’s the difference? A review of contemporary research on the effectiveness of distance learning in higher education. Institute for Higher Education Policy for the National Education Association and the American Federation of Teachers. Retrieved November 10, 2007, from http://www2.nea.org/ he/abouthe/diseddif.pdf
Schön, D. (1987a). Educating the reflective practitioner. Paper presented to the American Educational Research Association, Washington, DC. Retrieved November 10, 2007, from http://edu. queensu.ca/~russellt/howteach/schon87.htm
268
Schön, D. (1987b). Educating the reflective practitioner. Paper presented at the American Educational Research Association annual confer-
Practical Strategies for Assessing the Quality of Collaborative Learner Engagement
ence, Washington, DC. Retrieved November 10, 2007, from http://edu.queensu.ca/~russellt/howteach/schon87.htm Swan, K. (2002). Building learning communities in online courses: The importance of interaction. Education, Communication and Information, 2(1), 23-49. Twigg, C. (1998, May 2). A regulated monopoly enterprise gives way to competition. AAHE Bulletin. Retrieved November 12, 2007, from http://www. aahe.org/bulletin/bull_2may98.htm#Twigg Verduin, J. R., & Clark, T. (1991). Distance education: The foundations of effective practice. San Francisco: Jossey-Bass.
Vrasidas, C., & McIsaac, M. S. (1999). Factors influencing interaction in an online course. The American Journal of Distance Education, 13(3), 22-35. Woods, R., & Ebersole, S. (2003). Using nonsubject matter specific discussion boards to build connectedness in online learning. The American Journal of Distance Education, 17(2), 99-118. Wright, A. (2005, January 26). From ivory tower to academic sweatshop. Salon.com. Retrieved Novembet 10, 2007, from http://www.salon. com/tech/feature/2005/01/26/distance_learning/print.html
This work was previously published in Computer-Supported Collaborative Learning: Best Practices and Principles for Instructors, edited by K. Orvis and A. Lassiter, pp. 300-323, copyright 2008 by Information Science Publishing (an imprint of IGI Global).
269
270
Chapter XVI
Afterword:
Learning-Centred Focus to Assessment Practices Som Naidu Charles Stuart University, Australia
Many teachers commonly use assessment as the starting point of their teaching activities because they believe that assessment drives learning and teaching activities. Hence students tend to organise their learning activities around these prescribed assessment tasks. These beliefs and practices have the potential to detract from promoting effective, efficient, and engaging learning. Teachers, in using assessment tasks to orchestrate their teaching activities send out a message to their learners which minimises the importance of the learning experience. Not only does this constrain learners from taking full advantage of the designed learning experience, but with an explicit focus on assessment tasks by teachers, learners tend to adopt coping mechanisms that focus on the assessment task itself, and little else. This approach to teaching leads learners to adopt a surface approach to their learning
activities, when a deeper approach is recognised as the more desirable thing to do. Learners are not intrinsically deep or surface processors (Laurillard, 1979). Their approach to learning is dependent upon the teaching strategies and assessment tasks that teachers employ. If the teaching strategies and assessment tasks do not require students to engage deeply with the subject matter or the learning activities, chances are they will skip it or gloss over it. Teaching strategies and assessment tasks can influence what parts of the course content gets studied and how it gets studied (Kirkwood & Price, 2008). In a model of teaching where assessment drives learning, students tend to pay little attention to teaching activities such as lectures, tutorials, and group discussions. They tend to cruise along throughout the term only to ramp up their effort and momentum close to assessment due dates. A more effective and efficient approach to teaching would be to focus greater attention
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Afterword
on careful design of the learning experiences. After all, it is only the learning experiences of learners that one can design, not the learning itself. Learning is a personal and also a social process. It can be influenced by the design of learning experiences in light of the prescribed learning outcomes for a course or program of study. This view of learning and teaching suggests that the careful design of learning experiences, and not assessment, should be the key focus of any learning and teaching transaction. It begins with the identification of the intended learning outcomes of the course and what learners would do to be able to attain those learning outcomes. These essential learning activities may include reading some text, collecting data of some sort, carrying out experiments, and/or doing something with their peers online or offline within an authentic learning context, problem or situation. These learning activities will lead to products of learning which may be in the form of reports, results obtained, critical reflections, and so forth. A selection of these products, but not all, will be assessable. These are the items that will and should form the assessment items, and they are determined after the design of the learning experiences and not before it (Naidu, 2007; Naidu, Menon, Gunawardena, Lekamge, & Karunanayaka, 2007). In focussing on the learning experience, it is the learning activity and not the assessment task that drives the learning and teaching transaction. As such the assessment task is bound to be meaningful, motivating, and closely integrated with the learning and teaching activities. In this approach there is no opportunity for the learners to take short cuts. An assessment task can not be completed without completing the learning activity as expected. Moreover, it can not be left to the last minute. This ensures that all parts of the selected content get studied thoroughly. It also ensures that students are fully engaged in their learning right from the start, and that they remain engaged throughout the duration of the course
unable to lay low until an assignment due date approaches. This focus on the learning experience has a much greater chance for promoting a deeper approach to learning. The case studies and examples in this book demonstrate how such a learning-centred focus can be used to drive the assessment activity and learning itself. The other unique strength of the contributions in this book lies in its references to learning-centred assessment practices in technology enhanced learning environments which is increasingly the norm, at least in more developed educational settings. I hope that you found these contributions and their insights, as well as the commentaries of the editors interesting and engaging to read, and that they have inspired you to critically examine your own approaches towards promoting a learning-centred focus to the design of assessment practices.
REFERENCES Kirkwood, A., & Price, L. (2008). Assessment and student learning: A fundamental relationship and the role of information and communications technologies. Open Learning, 23(1), 5-16. Laurillard, D. (1979). The process of student learning, Higher Education, 8(4), 395-409. Naidu, S. (2007). Instructional designs for distance education. In M. G. Moore (Ed.), Handbook of Distance Education (2nd Edition), (pp. 247-258), Mahwah, NJ: Erlbaum. Naidu, S., Menon, M., Gunawardena, C., Lekamge, D., & Karunanayaka, S, (2007). How can scenario-based learning engender and promote reflective practice in online and distance education. In M. Spector (Eds.), Finding Your Online Voice: Stories Told by Experienced Online Educators (pp. 53-72), NJ: Lawrence Erlbaum.
271
272
Compilation of References
Active Worlds Inc. (2006). Active Worlds and education. Retrieved August 20, 2008, from http://www.activeworlds.com/edu/index.asp Alavi, M. (1994). Computer-mediated collaborative learning: An empirical evaluation. MIS Quarterly, (pp. 159-174). Alexander, B. (2006). Web 2.0: A new wave of innovation for teaching and learning. EDUCAUSE Review, 41(2) 32-44. Allen, C. (2004). Tracing the evolution of social software. Retrieved April 21, 2008, from http://www.lifewithalacrity.com/2004/10/tracing_the_evo.html American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association. Amory, A. (2007). Game object model version II: A theoretical framework for educational game development. Educational Technology Research and Development, 15, 51-77. Anderson, T., & Kanuka, H. (2003). E-research: Methods, strategies and issues. London: Allyn and Bacon. Anderson, T., Rourke, L., Garrison, R., & Archer, W. (2001). Assessing teaching presence in a computer conferencing context. JALN Journal of Asynchronous Learning Networks, 5(2). Retrieved from http://www.aln.org/publications/jaln/ v5n2/pdf/v5n2_anderson.pdf
Andrysyszyn, M. A., Cragg, B., & Humbert, J. (2001). Nurse practitioner preferences for distance education methods related to learning style, course content and achievement. Journal of Nursing Education, 40, 163170. Aragon, S. R., Johnson, S. D., & Shaik, N. (2002). The influence of learning style preferences on student success in online versus face-to-face environments. American Journal of Distance Education, 16, 227-244. Armatas, C., Holt, D., & Rice, M. (2003). Impacts of an online-supported resource-based learning environment: Does one size fit all? Distance Education, 24(2), 141-158. Armatas, C., Holt, D., & Rice, M. (2004). From onlineenhanced to wholly online: reflections on e-learning developments in psychology in higher education. In R. Atkinson, C. McBeath, D. Jonas-Dwyer, & R. Phillips (Eds.). Beyond the Comfort Zone: Proceedings of the 21st ASCILITE Conference (pp. 78-87), Perth, 5-8 December. Australian Government. (1988). Financial Transaction Reports Act. Retrieved April 10, 2008 from http://www. austrac.gov.au/rg_7.html Ausubel, D. P., Novak, J. D., & Hanesian, H. (1978). Educational psychology: A cognitive view (2nd ed.). New York: Holt, Rinehart and Winston. Averill, J. R. (1982). Anger and aggression: An essay on emotion. New York: Springer-Verlag. Badke, W. (2007). Give plagiarism the weight it deserves. Online, 31(5), 58-60.
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Compilation of References
Ballantyne, R., Hughes, K., & Mylonas, A. (2002). Developing procedures for implementing peer assessment in large classes using an action research process. Assessment & Evaluation in Higher Education, 27(5), 427-441.
Paper presented at the World Conference on Computers and Education (WCCE). Retrieved November 30, 2005, from http://transition.alaska.edu/www/portfolios/wccepaper.pdf
Bandura, A. (1994). Self-efficacy. In V. S. Ramachandran (Ed.), Encyclopedia of human behaviour (Vol. 4, pp. 71−81). New York: Academic Press. Retrieved from http://www.des.emory.edu/mfp/BanEncy.html
Barrie, S. C. (2004). A research-based approach to generic graduate attributes policy. Higher Education Research & Development, 23(3) 261-276.
Barber, P. (2004). Developing and assessing group design work: A case study. Paper presented at the 2nd International Engineering and Product Design Education Conference, Delft. Barker, P. (1994). Designing interactive learning. In T. de Jong & L. Sarti (Eds.), Design and production of multimedia and simulation-based learning material. Boston, MA: Kluwer. Barnes, C., & Tynan, B. (2007). The adventures of Miranda in the brave new world: Learning in a Web 2.0 millennium. ALT-J, 15(3), 189-200. Barnes, L. B., Christensen, C. R., & Hansen, A. J. (1994). Teaching with cases at the Harvard Business School. In L. B. Barnes, C. R. Christensen, & A. J. Hansen (Eds.), Teaching and the case method. Text, cases, and readings (3rd ed.) (pp. 34−68). Boston, MA: Harvard Business School Press. Baron, J., & Keller, M. (2003, November). Use of rubrics in online assessment. Paper presented at the second ATN Evaluations and Assessment Conference, University of South Australia, Adelaide. Retrieved May 5, 2008, from http://www.unisa.edu.au/evaluations/Full-papers/BaronFull.doc Barrett, H. (2005). Researching electronic portfolios and learner engagement (White Paper). In The REFLECT Initiative: Researching Electronic portFolios: Learning, Engagement and Collaboration through Technology. Retrieved November 30, 2005, from http:// www.taskstream.com/reflect/whitepaper.pdf Barrett, H. C. (2001). ICT support for electronic portfolios and alternative assessment: The state of the art.
Bates, A. W. (1997). Restructuring the university for technological change. Retrieved August 18, 2008, from http://bates.cstudies.ubc.ca/carnegie/carnegie.html Bazerman, C. (Ed.). (2008). Handbook of research on writing: History, society, school, individual, text. New York: Earlbaum. Beaman, R. (1998). The unquiet … even loud, andragogy! Alternative assessments for adult learners. Innovative Higher Education, 23(1), 47-59. Benford, S., Greenhalgh, C., Rodden, T., & Pycock, J., (2001). Collaborative virtual environments. Communications of the ACM, 44(7), 79-85. Bengtsson, J. (1995). What is reflection: On reflection in the teaching profession and teacher education. Teachers and Teaching, 1(1), 23-32. Bengtsson, J. (2003). Possibilities and limits of self-reflection in the teaching profession. Studies in Philosophy and Education, 22(3/4), 295-316. Bennett, S., Maton, K., & Kervin, L. (2008). The ‘digital natives’ debate: A critical review of the evidence. British Journal of Educational Technology. 39(5), 775-786. Benson, R., & Brack, C. (2007). Educational design and online assessment: A workbook. Melbourne: Faculty of Medicine, Nursing and Health Sciences, Monash University. Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M., & Huang, B. (2004). How does distance education compare to classroom instruction? A meta-analysis of the empirical literature. Review of Educational Research, 74(3), 379-439.
273
Compilation of References
Bhattacharya, M. (2001). Electronic portfolios, students reflective practices, and the evaluation of effective learning. AARE2001, Fremantle, Australia. Retrieved November 30, 2005, from http://www.aare.edu.au/01pap/ bha01333.htm Bhattacharya, M., & Jorgensen, L. (2008). Minimizing plagiarism by redesigning the learning environment and assessment. In T. S. Roberts (Ed.), Student plagiarism in an online world: Problems and solutions (pp.194-207). Hershey, PA: Information Science Reference. Bhattacharya, M., & Mimirinis, M. (2005). Australian Association for Educational Research, SIG report (p. 9). AARE News no. 52. Retrieved November 30, 2005, from http://www.aare.edu.au/news/newsplus/news52.pdf Biggs, J. (2003). Teaching for quality learning at university. Buckingham: Society for Research into Higher Education and Open University Press. Biggs, J. (2003). Teaching for quality learning at university: What the student does (2nd ed.). Buckingham: Society for Research into Higher Education and Open University Press. Bliss, T., & Reynolds, A. (2004). Quality visions and focused imagination. In J. Brophy (Ed.), Using video in teacher education: Advances in research on teaching, 10, 29−51. Oxford: Elsevier Ltd. Bloom, B., & Bacon, E. (1995). Using portfolios for individual learning and assessment. Teacher Education and Special Education, 18(1), 1-9. Bloxham, S., & West, A. (2004). Understanding the rules of the game: Marking peer assessment as a medium for developing students’ conceptions of assessment. Assessment & Evaluation in Higher Education, 29(6), 721-733. Bollen, L., Eimler S., & Hoppe, H. (2004). SMS based discussions- Technology enhanced collaboration for a literature course. In J. Roschelle, T. Chan, Kinshuk & S. Yang (Eds.), Proceedings of the 2nd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2004), Mobile Support for Learning
274
Communities (pp. 209-210) Los Alamitos, USA: IEEE Computer Society. Bolt, D. M, Cohen, A. S., & Wollack, J. A. (2001). A mixture item response model for multiple-choice data. Journal of Educational and Behavioral Statistics, 26, 381-409. Bolter, J. D., & Grusin, R. (2000). Remediation: Understanding the new media. Cambridge: MA, MIT Press. Book, B., (2006). Virtual worlds: Today and in the future. British Computing Society. Retrieved 15 May, 2006 from http://www.bcs.org/server.php?show=Con WebDoc.3336. Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2003). The theoretical status of latent variables. Psychological Review, 110, 203-219. Boud, D. (1995a). Assessment and learning: Contradictory or complementary? In P. Knight (Ed.), Assessment for learning in higher education (pp.35-48). London: Kogan Page. Boud, D. (1995b). Enhancing learning through self assessment. London: Kogan Page. Boud, D. (2007). Reframing assessment as if learning were important. In D. Boud & N. Falchikov (Eds.), Rethinking assessment in higher education: Learning for the longer term (pp. 14-25). London: Routledge. Boud, D., & Falchikov, N. (2006). Aligning assessment with long-term learning. Assessment & Evaluation in Higher Education, 31(4), 399-413. Boud, D., & Feletti, G. I. (Eds.). (1997). The challenge of problem-based learning (2nd ed.). London: Kogan Page. Boud, D., Cohen, R., & Sampson, J. (1999). Peer learning and assessment. Assessment and Evaluation in Higher Education, 24(4), 413-426. Boulos, M., Hetherington, L., & Wheeler, S. (2007). Second Life: An overview of the potential of 3-D virtual worlds in medical and health education. Health Information and Libraries Journal, 24(4), 233-245.
Compilation of References
Bourner, J., Hughes, M., & Bourner, T. (2001) First-year Undergraduate Experiences of Group Project Work. Assessment and Evaluation in Higher Education, 26(1) 19-39. Bourner, J., Hughes, M., & Bourner, T. (2001) First-year undergraduate experiences of group project work. Assessment and Evaluation in Higher Education, 26, 19-39. Bower, M. (2007). Group work activities in synchronous on-line classroom spaces. In Proceedings of the 38th SIGCSE technical symposium on Computer Science education (pp. 91-95). Covington, Kentucky, USA. Bower, M., & Richards, D. (2006). Collaborative learning: Some possibilities and limitations for students and teachers. In L. Markauskaite, P. Goodyear, & P. Reimann (Eds.), Proceedings of the 23rd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education: Who’s Learning? Whose Technology? (pp. 79-89). Sydney: Sydney University Press. Boyer, E. (1990). Scholarship reconsidered: Priorities of the professoriate. Princeton, NJ: The Carnegie Foundation for the Advancement of Teaching. Brack, C., Stauder, A., Doery, J., & Van Damme, M. P. (2007). Collaborative learning online: Integrating media in case-based learning for clinical biochemistry. Proceedings of EdMedia, (Vancouver, Canada, July 2007) (pp. 187-192). Bradlow, H., Saliba, A., Ivanovich, M., & Fitzpatrick, P. (2005). Recognising user perception in the design of telecommunications networks. Annual Review of Communications, 58, 7-14. Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.) (2001). Committee on Developments in the Science of Learning. M. S. Donovan, J. D. Bransford, & J. W. Pellegrino (Eds.), Committee on Learning Research and Educational Practice. How people learn. Brain, mind, experience, and school (Expanded ed.). Washington, DC: National Academy Press. Brennan, R. L. (2006). Perspectives on the evolution and future of educational measurement. In R. L. Brennan
(Ed.). Educational measurement (4th edition, pp. 1-16). Westport, CT: Praeger. Brennan, R. L. (Ed.). (2006). Educational measurement. (4th Edn.) Westport, Conn: Praeger. Brockett, R. G., & Hiemstra, R. (1991). Self-direction in adult learning: Perspectives on theory, research, and practice. London: Routledge. Brown, E., (2006). Discussion board or discussion bored? Presented at the 7th UCLT Conference, Anglia Ruskin University. Brown, G., & Johnson-Shull, L. (2000, May/June). Teaching online: Now we’re talking. The Technology Source. Retrieved November 10, 2007, from http://technologysource.org/article/teaching_online/ Brown, J. O. (2002). Know thyself: The impact of portfolio development on adult learning. Adult Education Quarterly, 52(3), 228-245. Brown, R. (2000, April). Using latent class analysis to set academic performance standards. Paper presented to the Annual Conference of the American Association for Educational Research, New Orleans, LA. Brown, R. W. (1995, 1-4 November). Autorating: Getting individual marks from team marks and enhancing teamwork. Paper presented at the 25th Annual Frontiers in Education Conference, Atlanta, Georgia. Brown, S., Race, P., & Bull, J. (1999). Computer-assisted assessment of students. London: Kogan Page. Browne, E. (2003). Conversation in cyberspace: A study of online learning. Open Learning, 18(3), 245-259. Caldwell, J. E. (2007). Clickers in the large classroom: Current research and best-practice tips. CBE Life Sciences Education, 6(1), 9-20. Campos, M. (2004). A constructivist method for the analysis of networked cognitive communication and the assessment of collaborative learning and knowledge building. JALN, 8(2). Retrieved November 10, 2007, from http://www.sloan-c.org/publications/jaln/v8n2/index. asp
275
Compilation of References
Cann, A. J. (2005). Extended matching sets questions for online numeracy assessments: A case study. Assessment & Evaluation in Higher Education, 30(6), 633-640.
Cizek, G. J. (2008). Assessing educational measurement: Ovations, omissions, opportunities. Educational Researcher, 37(2), 96.
Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31(2), 219-233.
Clark, N., Davies, P., & Skeers, R. (2005). Self and peer assessment in software engineering projects. Paper presented at the Proceedings of the 7th Australasian conference on Computing education - Volume 42, Newcastle, New South Wales, Australia.
Carless, D. (2006). Differing perceptions in the feedback process. Studies in Higher Education, 31(2), 219-233. Carless, D. (2007). Learning-oriented assessment: Conceptual bases and practical implications. Innovations in Education and Teaching International, 44(1), 57-66. Carmines, E. G., & Zeller, R. A. (1991). Reliability and validity assessment. Newbury Park: Sage Publications. Cereijo, M. V. P. (2006). Attitude as predictor of success in online training. International Journal on E-Learning, 5, 623-639. Challis, D. (2005). Committing to quality learning through adaptive online assessment. Assessment & Evaluation in Higher Education, 30(6), 519-527. Chang, N., & Petersen, N. J. (2005). Cybercoaching: An emerging model of personalized online assessment. In D. D. Williams (Ed.), Online assessment, measurement and evaluation: Emerging practices (pp. 110−130). Hershey, PA: Information Science Publishing. Chang, S-H., & Smith, R. A. (2008). Effectiveness of personal interaction in a learner-centered paradigm distance education class based on student satisfaction. Journal of Research on Technology in Education, 40(4), 407-426.
Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups. Review of Educational Research, 64(1), 1-35. Cohen, L., Manion, L., & Morrison, K. (2003). Research methods in education (5th ed.). London: RoutledgeFalmer. College Board. (2008a). SAT Reasoning Test. Retrieved October 4, 2008, from http://www.collegeboard.com/student/testing/sat/about/SATI.html College Board. (2008b). Score range. Retrieved October 4, 2008, from http://www.collegeboard.com/student/testing/sat/scores/understanding/scorerange.html. Colliver, J., & Markwell, S. (2007). Research on problembased learning: The need for critical analysis of methods and results. Medical Education, 41, 533-535. Congdon, P. (2003). Applied Bayesian modelling. Chichester: John Wiley & Sons Ltd. Conklin, M. S. (2007). 101 uses for Second Life in the college classroom. Retrieved from http://facstaff.elon. edu/mconklin/pubs/glshandout.pdf
Chen, G. D., Chang, C. K., & Wang, C. Y. (2006). Ubiquitous learning website: Scaffold learners by mobile devices with information-aware techniques. Computers and Education, 50(1), 77-90.
Conway, R., Kember, D., Sivan, A., & Wu, M. (1993). Peer assessment of an individual’s contribution to a group project. Assessment and Evaluation in Higher Education, 18, 45-56.
Chen, Y., Kao, T., & Sheu, J. (2003). A mobile learning system for scaffolding bird watching learning. Journal of Computer Assisted Learning, 19, 347-359.
Corbit, M. (2002). Building virtual worlds for informal science learning (SciCentr and SciFair) in the Active Worlds Educational Universe (AWEDU). Presence, 11(1), 55-67.
Chua, A., & Lam, W. (2007). Quality assurance in online education: The Universitas 21 global approach. British Journal of Educational Technology 38(1), 133-152.
276
Compilation of References
Cottrell, S. (2005). Critical thinking skills: Developing effective analysis and argument. Basingstoke: Palgrave Macmillan. Csete, J., Wong, Y. H., & Vogel, D. (2004). Mobile devices in and out of the classroom. In L. Cantoni & C. McLoughlin (Eds.), Proceedings of the 16th World Conference on Educational Multimedia and Hypermedia & World Conference on Educational Telecommunications (pp. 4729-4736) Norfolk VA: Association for the Advancement of Computing in Education. CTGV (Cognition and Technology Group at Vanderbilt). (1993). Designing learning environments that support thinking: The Jasper series as a case study. In T. M. Duffy, J. Lowyck, & D. H. Jonassen (Eds.), Designing environments for constructive learning (pp. 9−36). Berlin: Springer-Verlag. Cullen, J., Hadjivassiliou, E., Hamilton, E., Kelleher, J., Sommerlad, E., & Stern, E. (2002). Review of current pedagogic research and practice in the fields of postcompulsory education and lifelong learning. Final report for ESRC by the Tavistock Institute. Retrieved August 20, 2008 from http://www.tlrp.org/pub/acadpub.html Dabbagh, N., & Kitsantas, A. (2005). The role of webbased pedagogical tools in supporting student self-regulation in distributed learning environments. Instructional Science, 25, 24-37. Dahl, S. (2007). The student perspective on using plagiarism detection software. Active Learning In Higher Education, 8(2), 173-191. Dalsgaard, C. (2006). Social software: E-learning beyond learning management systems. European Journal of Open, Distance and E-learning, 2. Retrieved August 15, 2008, from http://www.eurodl.org/materials/contrib/2006/Christian_Dalsgaard.htm Davies, P. (2000). Computerized peer assessment. Innovations in Education and Teaching International, 37(4), 346-355. Davis, H. C., & Fill, K. (2007). Embedding blended learning in a university’s teaching culture: Experiences
and reflections. British Journal of Educational Technology, 38(5), 817-828. de Freitas, S. I. (2006). Using games and simulations for supporting learning. Learning, Media and Technology, 31(4), 343–358. Dekeyser, H. M. (2000). Student preference for verbal, graphic or symbolic information in an independent learning environment for an applied statistics course. In K. Littleton (Series Ed.) & J. F. Rouet, A. Biardeau, & J. J. Levonen (Eds.), Advances in learning and instruction series: Multimedia learning. Cognitive and instructional issues (pp. 99−109). Amsterdam: Pergamon. DeTure, M. (2004). Cognitive style and self-efficacy: Predicting student success in online distance education. American Journal of Distance Education, 18, 21-38. di Gregario, S. (2000). Using NVivo for your literature review. Paper presented at the Strategies in Qualitative Research: Issues and Results from Analysis Using QSR NVivo and NUD*IST Conference, London. Retrieved November 10, 2007, from http://www.sdgassociates. com/downloads/literature_review.pdf Dialogue on Validity [Special issue]. 2007. Educational Researcher, 36(8). Dickey, M. D. (2003). Teaching in 3D: Pedagogical affordances and constraints of 3D virtual worlds for synchronous distance learning. Distance Education, 24(1), 105-121. Dickey, M. D. (2004). The impact of web-logs (blogs) on student perceptions of isolation and alienation in a webbased distancE-Learning environment. Open Learning, 19(3), 280-291. Dickey, M. D. (2005). Three-dimensional virtual worlds and distance learning: Two case studies of Active Worlds as a medium for distance education. British Journal of Educational Technology, 36(3), 439-451. Dillon, A., & Gabbard, R. (1998). Hypermedia as an educational technology: A review of the quantitative research literature on learner comprehension, control and style. Review of Educational Research, 68(3), 322-349.
277
Compilation of References
Dillon, R. S., & Stines, R. W. (1996). A phenomenological study of faculty-student caring interactions. Journal of Nursing Education, 35, 113-118. Dochy, F., Segers, M., & Sluijsmans, D. M. A. (1999). The use of self-, peer and co-assessment in higher education: A review. Studies in Higher Education, 24(3), 331-350. Dominick, P. G., Reilly, R. R., & McGourty, J. (1997). Incorporating student peer review and feedback into the assessment process. Paper presented at the Best Assessment Processes in Engineering Education: A Working Symposium, Terre Haute, Indiana.
University. Technology, Pedagogy and Education, 14(3), 329-351. Ehrmann, S. (1997). Asking the right questions: What does research tell us about technology and higher learning? In Engines of inquiry: A practical guide for using technology to teach American culture. Retrieved November 10, 2007, from http://www.georgetown.edu/ crossroads/guide/ehrmann.html Elgort, I., Smith, A., & Toland, J. 2008, Is wiki an effective platform for group course work? Australasian Journal of Educational Technology, 24(2), 195-210.
Dron, J. (2007). Control and constraint in e-learning: Choosing when to choose. Hershey, PA: Information Science Publishing.
Elliot, N., Briller, V., & Joshi, K. (2007). Portfolio assessment: Quantification and community. Journal of Writing Assessment, 3, 5-29.
Dron, J. (2007). Designing the undesignable: Social software and control. Educational Technology & Society, 10(3), 60-71.
Elliot, N., Friedman, R. S., & Briller, V. (2005, June-July). Irony and asychronicity: Interpreting withdrawal rates in E-Learning courses. Paper presented at the meeting of the Ed-Media World Conference on Educational Multimedia, Hypermedia, and Telecommunications, Montreal, Canada.
Ducheneaut, D., Yee, N., Nickell, E., & Moore, R. J. (2006). Building an MMO with mass appeal: A look at gameplay in World of Warcraft. Games and Culture, 1(4), 281-317. Dunham, T. C., & Davidson, M. L. (1991). Effect of scale anchors on student rating of instructors. Applied Measurement in Education, 4, 23-35. DuVall, J. B., Powell, M., & Lucier, A. (2006) Online survey of current student use of mobile phones for enhanced personal safety and communication. East Carolina University, Technology Advancement Centre, (pp. 24-28). Edelstein, S., & Edwards, J. (2002). If you build it, they will come: Building learning communities through threaded discussions. Online Journal of Distance Learning Administration, 5(1). Retrieved August 15, 2008, from http://www.westga.edu/~distance/ojdla/search_results_id.php?id=152 Edwards, S. (2005). Higher education in the twenty-first century: Examining the interface between graduate attributes, online and problem-based learning at Monash
278
Ellis, T. J., & Hafner, W. (2005). Peer evaluations of collaborative learning experiences conveyed through an asynchronous learning network. Paper presented at the Conference Proceedings Hawaii International Conference on System Sciences (HICSS-38), Big Island, Hawaii. Elwert, B., & Hitch, L. (Eds.). (2002). Motivating and retaining adult learners online. Essex Junction, Vt.: GetEducated.com. Embretson, S. E. (2007). Construct validity: A universal validity system or just another test evaluation procedure? Educational Researcher, 36, 449-455. Emery, F. E., & Trist, E. L. (1969). Socio-technical systems. In F. E. Emery (Ed.), Systems thinking, (pp. 281-296). Harmondsworth: Penguin. Engelbrecht, J., & Harding, A. (2004). Combining online and paper assessment in a web-based course in undergraduate mathematics. Journal of Computers in Mathematics and Science Teaching, 23(3), 217-231.
Compilation of References
Engeström, Y. (1982). Mielekäs oppiminen ja opetus [Meaningful learning and teaching]. Helsinki, Finland: Valtion koulutuskeskus.
R. Steele & E. Chang (Eds.), Proceedings of the IEEE International Conference on Mobile Business (pp. 445451) Los Alamitos, USA: IEEE Computer Society.
Entwistle, N., McCune, V., & Hounsell, J. (2002). Approaches to studying and perceptions of university teaching-learning environments: Concepts, measures and preliminary findings. Retrieved October 12, 2008 from ELT Project Web site, http://www.ed.ac.uk/etl/ docs/ETLreport1.pdf
Farmer, B., Yue, A., & Brooks, C. (2008). Using blogging for higher order learning in large cohort university teaching: A case study. Australian Journal of Educational Technology, 24(2), 123-136.
Fabos, B., & Young, M.D. (1999). Telecommunications in the classroom: Rhetoric versus reality. Review of Educational Research, 69(3), 217-259. Falchikov, N. (1986). Product comparisons and process benefits of collaborative peer group and self-assessment. Assessment & Evaluation in Higher Education, 11(2), 146-166. Falchikov, N. (2005). Improving assessment through student involvement. London: RoutledgeFalmer. Falchikov, N. (2005). Improving assessment through student involvement. London: RoutledgeFalmer. Falchikov, N. (2005). Improving assessment through student involvement: Practical solutions for aiding learning in higher and further education. London: Routledge. Falchikov, N., & Goldfinch, J. (2000). Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287-322. Fallahkhair, S., Pemberton, L., & Griffiths, R. (2007). Development of a cross-platform ubiquitous language learning service via mobile phone and interactive television. Journal of computer-assisted learning, 23(4), 312-325. Fan, Y., Kendall, E., & Saliba, A. (2004). Acceptance of technology: A meta-analytic approach. Paper presented at the Honolulu International Conference on Social Science, Honolulu. Fan, Υ., Saliba, Α., Kendall, Ε., & Newmarch, J. (2005). Speech interface: An enhancer to the acceptance of mcommerce applications. In W. Brookes, E. Lawrence,
Felder, R. M., & Brent, R. (1996). Navigating the bumpy road to student-centered instruction. College Teaching, 44, 43-47. Fermelis, J. (2006). Making teamwork work. Paper presented at the 71st Association for Business Communication Annual Convention. Fermelis, J., Tucker, R., & Palmer, S. (2007). Online self and peer assessment in large, multi-campus, multicohort contexts. In R. J. Atkinson, C. McBeath, S. K. A. Soong, & C. Cheers (Eds.), ICT: Providing choices for learners and learning. Proceedings Ascilite Singapore 2007 (pp. 271-281). Singapore: Centre for Educational Development, Nanyang Technological University. Retrieved August 15, 2008, from http://www.ascilite.org. au/conferences/singapore07/procs/fermelis.pdf Flagg, B.N. (1990). Formative evaluation for educational technologies. Hillsdale, NJ: Lawrence Erlbaum. Fletcher, J. D., Tobias, S., & Wisher, R. A. (2007). Learning anytime, anywhere: Advanced distributed learning and the changing face of education. Educational Researcher, 36(2), 96-102. Flower, L. (1994). The construction of negotiated meaning: A social cognitive theory of writing. Carbondale: Southern Illinois University Press. Foster, L., Bower, B. L., & Watson, L. W. (Eds.). (2002). Western cooperative for educational telecommunications. Best practices for electronically offered degree and certificate programs. ASHE Reader: Distance Education: Teaching and Learning in Higher Education. Boston: Pearson. Fox, R. (2001). Constructivism examined. Oxford Review of Education, 27(1), 23-35.
279
Compilation of References
Freeman, M., & McKenzie, J. (2000, 2-5 July). Self and peer assessment of student teamwork: Designing, implementing and evaluating spark, a confidential, web based system. Paper presented at the ASET-HERDSA 2000 Conference, Toowoomba, Qld. Freeman, M., & McKenzie, J. (2002). SPARK, a confidential web-based template for self and peer assessment of student teamwork: Benefits of evaluating across different subjects. British Journal of Educational Technology, 33(5), 551-569. Freeman, M., & McKenzie, J. (2002). Spark, a confidential web-based template for self and peer assessment of student teamwork: Benefits of evaluating across different subjects. British Journal of Educational Technology, 33(5), 551-569. Frey, A., Faul, A., & Yankelov, P. (2003). Student strategies of web-assisted teaching strategies. Journal of Social Work Education, 39, 443-457. Friedman R., Elliot, N, & Haggerty, B. (in press). E-Learning in undergraduate humanities classes: Unpacking the variables. International Journal on E-Learning. Gammie, E., & Matson, M. (2007). Group assessment at final degree level: An evaluation, Accounting Education: An International Journal, 16(2), 185-206.
Gatfield, T. (1999). Examining student satisfaction with group projects and peer assessment. Assessment and Evaluation in Higher Education, 24(4) 365-378. Genuth, I. (2007). The future of electronic paper. Retrieved September 8, 2008, from http://thefutureofthings. com/articles/1000/the-future-of-electronic-paper.html George, J., & Cowan, J. (1999). A handbook of techniques for formative evaluation: Mapping the student’s learning experience. London: Taylor & Francis Group. Gerbic, P., & Stacey, E. (2005). A purposive approach to content analysis: Designing analytical frameworks. The Internet and Higher Education, 8(1), 45-59. Gibaldi, J. (2003). MLA handbook for writers of research papers. (6th ed.). New York: Modern Language Association. Gibbs, G., & Simpson C. (2004). Conditions under which assessment supports students. Learning and Teaching in Higher Education, 1, 3-31. Gibbs, G., & Simpson, C. (2004). Conditions under which assessment supports students’ learning. Learning & Teaching in Higher Education, 1, 3–31. Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton.
Garbett, D. (2004). Researching practice as a teacher educator. Retrieved from http://www.aare.edu.au/04pap/ gar04590.pdf.
Gipps, C. (1994). Beyond testing: Towards a theory of educational assessment. London: RoutledgeFalmer. {eBook}
Garrison, D. R., & Kanuka, H. (2004). Blended learning: Uncovering its transformative potential in higher education. The Internet and Higher Education, 7, 95-105.
Gitomer, D., & Yamamoto, K. (1991). Performance modeling that integrates latent trait and class theory. Journal of Educational Measurement, 28, 173-189.
Garrison, R., & Anderson, T. (2003). E-Learning in the 21st Century: A framework for research and practice. London: RoutledgeFalmer.
Gokhale, A. (1995). Collaborative learning enhances critical thinking. Journal of Technology Education, 7(1), 22-30.
Garrison, R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence and computer conference in distance education. The American Journal of Distance Education, 15(1), 7-23.
Goldfinch, J. (1994). Further developments in peer assessment of group projects. Assessment and Evaluation in Higher Education, 19, 29-35.
280
Goldfinch, J., & Raeside, R. (1990). Development of a peer assessment technique for obtaining individual
Compilation of References
marks on a group project. Assessment and Evaluation in Higher Education, 15, 210-231. Goldfinch, J., & Raeside, R. (1990). Development of a peer assessment technique for obtaining individual marks on a group project. Assessment and Evaluation in Higher Education, 15(3), 210-231. Goldfinch, J., Laybourn, P., MacLeod, L., & Stewart, S. (1999). Improving group working skills in undergraduates through employer involvement. Assessment and Evaluation in Higher Education, 24, 41-49. Goodyear, P. (2002). Psychological foundations of networked learning. In C. Steeples & C. Jones (Eds.), Networked learning: Perspectives and issues (pp. 49-76). London: Springer. Goodyear, P., Salmon, G., Spector, J. M., Steeples, C., & Tickner, S. (2001). Competencies for online teaching: A special report. Educational Technology, Research and Development, 49(1), 65-72. Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59(2), 93–104. Graf, F. (2002). Providing security for e-learning. Computers and Graphics, 26, 355-365. Grant, G. E. (1992). Using cases to develop teacher knowledge. A cautionary tale. In J. H. Shulman (Ed.), Case methods in teacher education (pp. 211−226). New York: Teachers College Press. Greeno, J. G., Collins, A. M., & Resnick, L. B. (1996). Cognition and learning. In D. Berliner & R. Calfee (Eds.), Handbook of educational psychology (pp. 15-46.) New York: Macmillan. Hakkarainen, P. (2007). Designing and implementing a PBL course on educational digital video production: Lessons learned from a design-based research. Educational Technology Research & Development (ETR&D), Online first article. Retrieved June 12, 2008, from the SpringerLink database.
Hakkarainen, P. (2007). Promoting meaningful learning through the integrated use of digital videos. Doctoral dissertation, University of Lapland. Acta Universitatis Lapponiensis 121. Rovaniemi, Finland: University of Lapland, Faculty of Education. Hakkarainen, P., & Saarelainen, T. (2005). Producing digital video-supported cases with students − How to make it happen? In H. Ruokamo, P. Hyvönen, M. Lehtonen, & S. Tella (Eds.), Teaching−studying−learning (TSL) processes and mobile technologies − Multi-, inter-, and transdisciplinary (MIT) research approaches. Proceedings of the 12th International Network-Based Education (NBE) Conference (Former PEG) 2005 (pp. 183−191). University of Lapland Publications in Education 11. Rovaniemi, Finland: University of Lapland, Faculty of Education, Centre for Media Pedagogy. Hakkarainen, P., & Saarelainen, T. (2005). Towards meaningful learning through designing, producing and solving digital video-supported cases with students. In G. Richards (Ed.), Electronic versions of papers presented at E–Learn 2005 – World Conference on E–Learning in Corporate, Government, Healthcare, and Higher Education (pp. 2081–2088) [CD-ROM]. Association for the Advancement of Computing in Education. Hakkarainen, P., Saarelainen, T., & Ruokamo, H. (2007). Towards meaningful learning through digital video supported case based teaching. Australasian Journal of Educational Technology (AJET), 23(1), 87−109. Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff Publishing. Hammond, M., & Wiriyapinit, M. (2005). Learning through online discussion: A case of triangulation in research. Australasian Journal of Educational Technology, 21(3), 283-302. Hannan, A. (2005). Innovating in higher education: Contexts for change in learning technology. British Journal of Educational Technology, 36 (6), 975-985. Hannan, A., English, S., & Silver, H. (1999). Why innovate? Some preliminary findings from a research project
281
Compilation of References
on “Innovations in teaching and learning in higher education”. Studies in Higher Education, 24(3), 279-289. Hanrahan, S. J., & Isaacs, G. (2001). Assessing self- and peer-assessment: The students’ views. Higher Education Research & Development, 20(1), 53-70. Hara, N., Bonk, C. J., & Angeli, C. (2000). Content analysis of online discussion in an applied educational psychology course. Instructional Science, 28(2), 115-152. Harasim, L. (1993). Networlds: Networks as social space. In L. Harasim (Ed.), Global networks: Computers and international communication (pp. 15-34). MIT Press: Cambridge, MA. Harasim, L., Hiltz, S. R., Teles, L., & Turroff, M. (1995). Learning networks: A field guide to teaching and learning online. Cambridge, MA: MIT Press. Harlen, W. (2005). Trusting teachers’ judgement: Research evidence of the reliability and validity of teachers’ assessment used for summative purposes. Research Papers in Education, 20(3), 245-270.
ing community in Web based learning environments. In Proceedings of the National Convention of the Association for Educational Communication and Technology, Atlanta, Georgia (pp. 192-199). Hiltz, R. S., & Turoff, M. (1993). Network nation - revised edition: Human communication via computer. Cambridge, MA: MIT Press. Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235-266. Hobbs, M. (2008). Anglia Ruskin virtual worlds wiki. Retrieved on August 20, 2008, from http://mojo.csd. anglia.ac.uk/wiki Hoerr, T. R. (2007). Supervising Generation X. Educational Leadership, October, 85-86. Hof, R. (2006, March 28). A virtual world’s real dollars. Business Week. Retrieved on May 25, 2006, from http://www.businessweek.com/technology/
Harley, D., Winn, S., Pemberton, S., & Wilcox, P. (2007). Using texting to support students’ transition to university. Innovations in Education and Teaching International, 44(3), 229-241.
Hoffman, K., Hosokawa, M., Blake, R., Headrick, L., & Johnson, G. (2006). Problem-based learning outcomes: Ten years of experience at the University of MissouriColumbia school of medicine. Academic Medicine, 81(7), 617-625.
Hart, G., & Stone, T. (2002). Conversations with students: The outcomes of focus groups with QUT students. Retrieved from http://www.ecu.edu.au/conferences/herdsa/ main/papers/nonref/title.html
Hogan, S., & Lundquist, L. (2006). The impact of problembased learning on students’ perceptions of preparedness for advanced pharmacy practice experiences. American Journal of Pharmaceutical Education, 70(4), 1-7.
Haythornthwaite, C., Kazmer, M., & Robins, J. (2000). Community development among distance learners: Temporal and technological dimensions. Journal of Computer Mediated Communication, 6(1). Retrieved November 10, 2007, from http://www.ascusc.org/jcmc/ vol6/issue1/haythornthwaite.html
Hogan, T. P. (2007). Educational assessment: A practical introduction. John Wiley & Son.
Henri, F. (1991). Computer conferencing and content analysis. In A. R. Kaye (Ed.), Collaborative learning through computer conferencing. The Najaden papers (pp. 117-136). New York: Springer-Verlag. Hill, J. R., Han, S., & Raven, A. (2001, November). Build it and they will stay: A research based model for creat-
282
Honebein, P. C., Duffy, T. M. & Fishman, B. J. (1993). Constructivism and the design of learning environments: Context and authentic activities for learning. In T. M. Duffy, J. Lowyck & D. H. Jonassen (Eds.), Designing environments for constructivist learning, (pp. 87-108). Berlin: Springer-Verlag. Honey, P., & Mumford, A. (1982). The manual of learning styles. Maidenhead: Peter Honey Publications.
Compilation of References
Hoppe, H., Joiner, R., Milrad, M., & Sharples, M. (2003). Guest editorial: Wireless and mobile technologies in education. Journal of Computer Assisted Learning, 19, 255-259. Horizon Wimba: http://www.horizonwimba.com/ Horkay, N., Bennett, R. E., Allen, N., Kaplan, B., & Yan, F. (2006). Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment, 5(2). Retrieved 9th August 2008 from http://www.jtla.org. Horton, W. (2001). Evaluating e-learning. Alexandria, VA: American Society for Training and Development. Hounsell, D. (2003). Student feedback, learning and development. In M. Slowey, and D. Watson (Eds), Higher Education and the Lifecourse (pp. 67-78). Buckingham: SRHE & Open University Press. Houser, C., Thornton, P., & Kludge, D. (2002). Mobile learning: Cell phones and PDA’s for education. In Proceedings of the International Conference on Computers in Education (pp. 1149-1150) Massey University, New Zealand: ICCE. Huba, M. E., & Freed, J. E. (2000). Learner-centered assessment on college campuses: Shifting the focus from teaching to learning. Boston: Allyn and Bacon. Hughes, C. E., & Moshell, E. J. (1997). Shared virtual worlds for education: The ExploreNet experiment. ACM Multimedia, 5(2), 145-154. Hutchings, P., Babb, M., & Bjork, C. (2002). The scholarship of teaching and learning in higher education: An annotated bibliography. The Carnegie for the Advancement of Teaching. Retrieved November 10, 2007, from http://www.carnegiefoundation.org/dynamic/downloads/file_1_196.pdf Institute for Higher Education Policy. (1999). A review of contemporary research on the effectiveness of distance learning in higher education. Washington, DC: The Institute for Higher Education Research.
Issroff, K., & Scanlon, E. (2002). Using technology in higher education: An activity theory perspective. Journal of Computer Assisted Learning, 18, 77-83. ISTE (2008). Home page. International Society for Technology in Education. Retrieved August 20, 2008, from http://www.iste.org/ James, D., & Fleming, S. (2004). Agreement in student performance in assessment. Learning and Teaching in Higher Education, 1, 32-51. James, R., McInnis, C., & Devlin, M. (2002). Assessing learning in Australian universities. Canberra, ACT: Centre for the Study of Higher Education, The University of Melbourne & The Australian Universities Teaching Committee. James, R., McInnis, C., & Devlin, M. (2002). Assessing learning in Australian universities. Melbourne: Centre for the Study of Higher Education and The Australian Universities Teaching Committee. James, R., McInnis, C., & Devlin, M. (2002). Assessing learning in Australian universities. Melbourne: Centre for the Study of Higher Education and The Australian Universities Teaching Committee. Jamison, T. M. (2003). Ebb from the web: Using motivational systems theory to predict student completion of asynchronous web-based distance education courses. Unpublished doctoral dissertation, George Mason University. Jeffries, P. (2003). ICT in supporting collaborative learning: Pedagogy and practice. Journal of Educational Media, 28(1), 35-48. John Hopkins University: http://cte.jhu.edu/epweb/ Johnson, C. S. (2006) A decade of research: Assessing change in the technical communication classroom using online portfolios. Journal of Technical Writing & Communication, 36(4), 413-431. Johnston, L., & Miles, L. (2004). Assessing contributions to group assignments. Assessment & Evaluation in Higher Education, 29(6), 751-768.
283
Compilation of References
Jonassen, D. H. (1992). Evaluating constructivist learning. In T. M. Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction (pp 137-148). Hillsdale: Ealbaum Associates. Jonassen, D. H. (1995). Supporting communities of learners with technology: A vision for integrating technology with learning in schools. Educational Technology, 35(4), 60−63. Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology: Research and Development, 45(1), 65-95. Jonassen, D. H. (2000). Computers as mindtools for schools. Engaging critical thinking. New Jersey: Prentice-Hall. Jonassen, D. H. (2000). Toward a meta-theory of problem solving. Educational Technology: Research and Development, 48(4), 63-85. Jonassen, D. H. (2002). Learning as activity. Educational Technology, 42(2), 45−51. Jonassen, D. H., Howland, J., Moore, J., & Marra, M. (2003). Learning to solve problems with technology. A constructivist perspective (2nd ed.). Upper Saddle River, NJ: Merrill Prentice Hall. Jonassen, D. H., Howland, J., Moore, J., & Marra, R. M. (2003). Learning to solve problems with technology: A constructivist approach. Upper Saddle River, NJ: Merrill Prentice Hall. Jones, Q., & Grandhi, S. (2005). P3 systems: putting the place back into social networks. IEEE Internet Computing, 9, 38-46. Joyce, B., & Weil, M. (1980). Models of teaching (2nd ed.). New Jersey: Prentice Hall International. Joyce, B., & Weil, M. (with Showers, B.) (1992). Models of teaching (4th ed.). Boston: Allyn and Bacon. Kambourakis, G., Kontoni, D. N., Rouskas, A. & Gritzalis, S. (2007). A PKI approach for deploying modern
284
secure distributed e-learning and m-learning environments. Computers and Education, 48, 1-16. Kamel Boulos, M. N., & Toth-Cohen, S. (2008). A sexual health SIM in Second Life. Retrieved from http://slsexualhealth.org.uk/ Kamel Boulos, M. N., Maramba, I., & Wheeler, S. (2006). Wikis, blogs and podcasts: A new generation of Webbased tools for virtual collaborative clinical practice and education. BMC Medical Education, 6(41), 1-18. Kane, M. T. (2006) Validation. In R. L. Brennan (Ed.), Educational measurement (4th edition, pp. 17-64). Westport, CT: American Council on Education/Praeger. Kane, M. T. (2008). Terminology, emphasis, and utility in validation. Educational Researcher, 37(2), 76-83. Karppinen, P. (2005). Meaningful learning with digital and online videos: Theoretical perspectives. AACE Journal, 13(3), 233−250. Katz, I. R. (2007a, June). Digital natives but not information fluent: Results from ETS’s iSkills assessment. Presented at the Alliance for a Media Literate American Research Summit, St. Louis, MO. Katz, I. R. (2007b). Testing information literacy in digital environments: ETS’s iSkills™ assessment. Information Technology and Libraries, 26(3), 3–12. Katz, I. R., Elliot, N., Attali, Y., Scharf, D., Powers, D., Huey, H., Joshi, K., & Briller, V. (2008). The assessment of information literacy: A case study (ETS Research Rpt. No 08-33.) Princeton, NJ: Educational Testing Service. Katz, I., Martinez, M., Sheehan, K., & Tatsuoka, K. (1998). Extending the rule space methodology to a semantically-rich domain: Diagnostic assessment in architecture. Journal of Educational and Behavioral Statistics, 23(3), 254-278. Kearsley, G., & Schneiderman, B. (1999). Engagement theory: A framework for technology-based teaching and learning. Retrieved November 10, 2007, from http://home. sprynet.com/~gkearsley/engage.htm
Compilation of References
Kennedy, G. E., Judd, T. S., Churchward, A., Gray, K., & Krause, K. (2008) First year students’ experiences with technology: Are they really digital natives? Australasian Journal of Educational Technology, 24(1), 108-122. Kennedy, G. J. (2005). Peer-assessment in Group Projects: Is It Worth It? In A. Young & D. Tolhurst (Eds.), Conferences in Research in Practice in Information Technology, Vol 42 Retrieved May, 2008, from http://crpit. com/confpapers/CRPITV42Kennedy.pdf Khan, B.H. (2001). A framework for Web-based learning. In B.H. Khan (Ed.), Web-based training (pp. 7598). Englewood Cliffs, NJ: Educational Technology Publications. Kindley, R. (2002). The power of simulation-based elearning (SIMBEL). E-Learning Developers Journal. Retrieved from http://www.elearningguild.com/pdf/2/ 091702DES-H.pdf King, S. A. (1996). Researching internet communities: Proposed ethical guidelines for the reporting of results. The Information Society, 12(2), 119-127. Kinneavy, J. L. (1971). A theory of discourse. New York: Norton. Kirkwood, A., & Price, L. (2008). Assessment and student learning: A fundamental relationship and the role of information and communications technologies. Open Learning, 23(1), 5-16.
Kort, B., & Reilly, R. (2002). Analytical models of emotions, learning, and relationships: Towards an affective-sensitive cognitive machine. Proceedings of the ITS 2002 − Intelligent Tutoring Systems Conference (pp. 955−962). June 2002, Biarritz, France. Retrieved from http://web.media.mit.edu/~reilly/its2002.pdf Krause, K., Hartley, R., James, R., & McInnis, C. (2005). The first year experience in Australian universities: Findings from a decade of national studies. Canberra: Australian Department of Education, Science and Training. Retrieved May, 2008, from http://www.griffith.edu. au/__data/assets/pdf_file/0006/37491/FYEReport05. pdf Kravitz, D. A., & Martin, B. (1986). Ringelmann rediscovered: The original article. Journal of Personality and Social Psychology, 50(5), 936-941. Krishnaswamy, S., Markham, S., Chhetri, M. B., Hurst, A. J., & Casey, D. (2004). PIAVEE - A pedagogy independent education environment. Proceedings of the 4th IEEE International Conference on Advanced Learning Technologies (ICALT 2004), Joensuu, Finland, IEEE Press. Kritzinger, E., & von Solms, S. H. (2006). E-learning: Incorporating information security governance. Issues in Informing Science and Information Technology, 3, 319-325.
Kirsch, I., Braun, H., Yamamoto, K., & Sum, A. (2007). America’s perfect storm: Three forces changing our nation’s future. New Jersey: Educational Testing Service.
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121-1134.
Knight, P. (2002). Summative assessment in higher education: Practices in disarray. Studies in Higher Education, 27(3), 275-286.
Kruger, J., & Dunning, D. (2002). Unskilled and unaware-but why? A reply to Krueger and Mueller (2002). Journal of Personality and Social Psychology, 82(2), 189-192.
Knight, P. (2006). The local practices of assessment. Assessment & Evaluation in Higher Education, 31(4), 435-452.
Lajbcygier, P., & Spratt, C. (2005) Using ‘Blended Learning’ to Develop Tertiary Students’ Skills of Critique, In L. Tomei (Ed.), Integrating information technologies into the classroom. Information Science Publishing, Hershey PA.
Knight, P., & Yorke, M. (2004). Learning, curriculum and employability in higher education. London: RoutledgeFalmer.
Lajbcygier, P., & Spratt, C., (2007) Using “blended learning” to develop tertiary students’ skills of critique. In
285
Compilation of References
Lawrence Tomei (Ed.), Integrating information & communications technologies into the classroom (pp. 1-18). Hershey PA: Information Science Publishing. Lane, S., & Stone, C. A. (2006). Performance assessment. In R. L. Brennan (Ed.), Educational measurement (4th edition, pp. 387-431). Westport, CT: American Council on Education/Praeger. Laurillard, D. (1979). The process of student learning, Higher Education, 8(4), 395-409. Laurillard, D. (2000). New technologies, students and the curriculum: The impact of communication and information technology on higher education. In P. Scott (Ed.), Higher education re-formed (pp. 133-153). London: Falmer Press. Laurillard, D. (2002). Rethinking university teaching: a framework for the effective use of learning technologies. RoutledgeFalmer, London. Laurillard, D., Stratfold, M., Luckin, R., Plowman, L., & Taylor, J. (2000). Affordances for learning in a nonlinear narrative medium. Journal of Interactive Media in Education. Retrieved from http://www-jime.open. ac.uk/00/2/index.html Lave, J. (1993). Situating learning in communities of practice. In L. B. Resnick, J. M. Levine & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 17-36). Washington, DC: American Psychological Association. Laybourn, P., Goldfinch, J., Graham, J., MacLeod, L., & Stewart. (2001). Measuring changes in group working skills in undergraduate students after employer involvement in group skill development. Assessment and Evaluation in Higher Education, 26, 364-380. LeBaron, J., & Miller, D. (2004). The teacher as “agent provocateur”: Strategies to promote community in online course settings. In T. Latomaa, J. Pohjonen, J. Pulkkinen, & M. Ruotsalainen (Eds.), eReflections: Ten years of educational technology studies at the University of Oulu (pp. 109-125). Oulu, Finland: Oulun yliopiston kasvatustieteiden tiedekunnan. Retrieved November 10, 2007, from http://gse.uml.edu/lebaron/Oulu-TAPfinal_030922.pdf
286
LeBaron, J., & Miller, D. (2005). The potential of jigsaw role-playing to promote the social construction of knowledge in an online graduate education course. Teachers College Record, 107(8), 1652-1674. LeBaron, J., & Santos, I. (2005). Authentic engagement of adult learners in online settings. MountainRise, 2(1). Retrieved November 10, 2007, from http://facctr.wcu. edu/mountainrise/archive/vol2no1/html/authentic_engagement.html LeBaron, J., Pulkkinen, J., & Scollin, P. (2000). Promoting cross-border communication in an international Web-based graduate course. Interactive Electronic Multimedia Journal of Computer-Enhanced Learning, 2(2). Retrieved November 10, 2007, from http://imej.wfu. edu/articles/2000/2/01/index.asp Lee, S-H., Wehmeyer, M. L., Palmer, S. B., Soukup, J. H., & Little, T. D. (2008). Self-determination and access to the general education curriculum. The Journal of Special Education, 42(2), 91-107. Lehtinen, E. (1997). Tietoyhteiskunnan haasteet ja mahdollisuudet oppimiselle [Challenges and possibilities for learning in the information society]. In E. Lehtinen (Ed.), Verkkopedagogiikka (pp. 12−40). Helsinki, Finland: Edita. Lejk, M., & Wyvill, M. (2001a). Peer assessment of contributions to a group project: A comparison of holistic and category-based approaches. Assessment and Evaluation in Higher Education, 26, 61-72. Lejk, M., & Wyvill, M. (2001b). The effect of the inclusion of self-assessment with peer assessment of contributions to a group project: A quantitative study of secret and agreed assessments. Assessment and Evaluation in Higher Education, 26, 552-561. Lejk, M., & Wyvill, M. (2002). Peer assessment of contributions to a group project: Student attitudes to holistic and category-based approaches. Assessment & Evaluation in Higher Education, 27(6), 569-577. Lejk, M., Wyvill, M., & Farrow, S. (1996). A survey of methods of deriving individual grades from group assess-
Compilation of References
ments. Assessment & Evaluation in Higher Education, 21(3), 267-280. Lejk, M., Wyvill, M., & Farrow, S. (1996). A survey of methods of deriving individual grades from group assessments. Assessment and Evaluation in Higher Education, 21, 267-280. Lejk, M., Wyvill, M., & Farrow, S. (1997). Group learning and group assessment on undergraduate computing courses in higher education in the UK: Results of a survey. Assessment and Evaluation in Higher Education, 22, 81-91. Lejk, M., Wyvill, M., & Farrow, S. (1999). Group assessment in systems analysis and design: A comparison of the performance of streamed and mixed ability groups. Assessment and Evaluation in Higher Education, 24, 5-14. Lejk, M.,& Wyvill, M. (2002). Peer assessment of contributions to a group project: Student attitudes to holistic and category-based approaches. Assessment and Evaluation in Higher Education, 27, 569-577.
Librero, F., Juan Ramos, A., Ranga, A. I., Trinona, J., & Lambert, D. (2007). Uses of the cell phone for education in the Philippines and Mongolia. Distance Education, 28(2), 231-244. Lin, S. S. J., Liu, E. Z. F., & Yuan, S. M. (2001). Webbased peer assessment: Feedback for students with various thinking-styles. Journal of Computer Assisted Learning, 17, 420-432. Linden Research, Inc. (2008). What is Second Life? Retrieved August 20, 2008, from http://secondlife. com/whatis/ Liu, C. C., & Kao, L. C. (2007). Do handheld devices facilitate face-to-face collaboration? Journal of Computer Assisted Learning, 23(4). 285-299. Liu, C. C., Tao, S. Y., & Nee, J. N. (2007) Bridging the gap between students and computers: Supporting activity awareness for network collaborative learning with GSM network. Behaviour and Information Technology, 27(2), 127-137.
Leners, D. W., & Sitzman, K. (2006). Graduate student perceptions: Feeling the passion of caring online. Nursing Education Perspectives, 27, 315-319.
Liu, E. Z. -F., Lin, S. S. -J., Chiu, C. -H., & Yuan, S. M. (2001). Web-based peer review: The learner as both adapter and reviewer. IEEE Transactions on Education, 44(3), 246-251.
Leslie, S., & Landon, B. (2008). Social Software for Learning: What is it, why use it? The Observatory on Borderless Higher Education, Association of Commonwealth Universities and Universities UK.
Liu, T. C., Wang, H. Y., Liang, J. K., Chan, T. W., Ko, H. W., & Yang, J. C. (2003). Wireless and mobile technologies to enhance teaching and learning. Journal of Computer Assisted Learning, 19, 371-382.
Leung, C. H., & Chan, Y. Y. (2003). Mobile learning: A new paradigm in electronic learning. In Proceedings of the 3rd IEEE International Conference on Advanced Learning Technologies, (pp. 76-80) Los Alamitos, USA: IEEE.
Livingstone, D. (2007). Learning support in multi-user virtual environments. In Proceedings of the European Conference on Games-Based Learning: ECGBL 2007. Academic Conferences Ltd.
Ley, K., & Young, D. B. (2001). Instructional principles for self-regulation. Educational Technology Research and Development, 49, 93–103. Li, L. K. Y. (2001). Some refinements on peer assessment of group projects. Assessment and Evaluation in Higher Education, 26(1), 5-18.
Lock, J. V. (2002). Laying the groundwork for the development of learning communities within online courses. Quarterly Review of Distance Education, 3(4), 395-408. Lorenzo, J., & Moore, J. (2002). The Sloan Consortium report to the nation: Five pillars of quality online education. Retrieved from http://www.sloan-c.org/effectivepractices/pillarreport1.pdf
287
Compilation of References
Lowood, H. (2005). Real-time performance: Machinima and game studies. The International Digital Media and Arts Association Journal 2(1), 10-17. Loyd, B. H., & Gressard, C. (1984). Reliability and factorial validity of computer attitude scales. Educational and Psychological Measurement, 44, 501-505. Luca, J., & McLoughlin, C. (2002, 8-11 December). A question of balance: Using self and peer assessment effectively in teamwork. Paper presented at the 19th Annual Conference of the Australian Society for Computers in Tertiary Education, Auckland. Lunn, D. J., Thomas, A., Best, N., & Spielgelhalter, D. (2000). WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility. Statistics and Computing, 10(4), 325-337. MacArthur, C. A., Graham, S., & Fitzgerald, J. (2006). Handbook of writing research. New York: Guilford Press. Macromedia Breeze: http://www.macromedia.com/software/breeze/ Mann, B. L. (2006). Testing the validity of post and vote web-based peer assessment. In D. D. Williams, S. L. Howell, & M. Hricko (Eds.), Online assessment, measurement and evaluation: Emerging practices (pp. 131-152). Hershey, PA: Information Science Publishing. Markham, S. (2008). A sociocybernetic approach to pedagogical systems. Technical Report 2008-1 Computing Education Research Group, Monash University Retrieved 12th August 2008 from http://cerg.infotech. monash.edu.au/techreps/tr-2008-1.pdf Mason, R. (1991). Evaluation methodologies for computer conferencing applications. In A. R. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp. 105-116). New York: Springer-Verlag. Mason, R. (1998). Models of online courses. In L. Banks, C. Graebner, & D. McConnell (Eds.), Networked lifelong learning: Innovative approaches to education and training through the internet. University of Sheffield.
288
Massey University Elearning: http://elearning.massey. ac.nz/ Massey University GSE, CoE: http://education.massey. ac.nz/gse/study/postgraduate.htm Mathur, S., & Murray, S. (2005). Authentic assessment online: A practical and theoretical challenge in higher education. In D. D. Williams (Ed.), Online assessment, measurement and evaluation: Emerging practices (pp. 243−258). Hershey, PA: Information Science Publishing. Mathur, S., & Murray, T. (2006). Authentic assessment online: A practical and theoretical challenge in higher education. In D. D. Williams, S. L. Howell, & M. Hricko (Eds.), Online assessment, measurement and evaluation: Emerging practices (pp.238-258). Hershey, PA: Information Science Publishing. Matuga, J. M. (2005). The role of assessment and evaluation in context: Pedagogical alignment, constraints, and affordances in online courses. In D. D. Williams (Ed.), Online assessment, measurement and evaluation: Emerging practices (pp. 316−330). Hershey, PA: Information Science Publishing. Mayer, R. E. (2003). Multimedia learning. New York: Cambridge University Press. Mayes, T., & de Freitas, S. I. (2004). JISC e-learning models desk study: Stage 2: Review of e-learning theories, frameworks and models. Retrieved from http://www. essex.ac.uk/chimera/projects/JISC/Stage 2 Learning Models (Version 1).pdf McConnell, D. (2006). E-learning groups and communities. Maidenhead: Society for Research in Higher Education and the Open University Press. McCracken, J. H., & Aldrich, T. B. (1984). Analysis of selected LHX mission functions: Implications for operator workload and system automation goals (Technical note ASI 479-024-84(b)). Fort Rucker, AL: Anacapa Sciences, Inc. McCrindle, M. (2006). New generations at work: Attracting, recruiting, retaining and training generation
Compilation of References
Y. Retrieved May 1, 2008, from http://www.mccrindle. com.au/wp_pdf/NewGenerationsAtWork.pdf McGourty, J., Dominick, P., & Reilly, R. R. (1998, 4-7 November). Incorporating student peer review and feedback into the assessment process. Paper presented at the 28th Annual Frontiers in Education Conference, Tempe, Arizona.
Millwood, R., & Terrell, I. (2005). Overview: New technology, learning and assessment in higher education. Innovations in Education & Teaching International, 42, 195-204. Ministry of Education (2004). Interim tertiary e-learning framework. Retrieved November 26, 2005, from http://www.minedu.govt.nz
McLellan, H. (2004). The case for case-based teaching in online classes. Educational Technology, 44(4), 14–18.
Mislevy, R. J. (2007). Validity by design. Educational Researcher 36, 463-469.
Mehanna, W. N. (2004). E-pedagogy: The pedagogies of e-learning. Research in Learning Technology, 12(3), 279-293.
Mislevy, R. J., & Brennan, R. L. (2006).Cognitive psychology and educational assessment. In R. L. Brennan (Ed.). Educational measurement (4th edition, pp. 257305). Westport, CT: Praeger.
Memon, A., Shih, L., & Thinger, B. (2006, June). Development and delivery of nuclear engineering online courses: The Excelsior College experience. Paper presented at the meeting of the ASEE Annual Conference & Exposition: Excellence in Education, Chicago, IL. Merrill, M. D. (2007). A task-centred instructional strategy. Journal of Research on Technology in Education, 40(1), 33-50. Messick, S. (1989). Validity. In R. L. Linn (Ed.). Educational measurement (3rd edition, pp. 13-103). New York: American Council on Education/Macmillan. Michelich, V. (2002, January/February). Streaming media to enhance teaching and improve learning. The Technology Source. Retrieved November 10, 2007, from http://technologysource.org/article/streaming_media_ to_enhance_teaching_and_improve_learning/ Miller, C. (1998). A socio-technical systems approach to distance education for professional development. Open Learning: The Journal of Open and Distance Learning, 13(2), 23-29. Miller, M., & Lu, M-L. Serving non-traditional students in E-Learning environments: Building successful communities in the virtual campus, Educational Media International, 40(1), 163-169. Miller, P. J. (2003). The effect of scoring criteria specificity on peer and self-assessment. Assessment & Evaluation in Higher Education, 28(4), 383-394.
MML08 (2008). Workshop for Massively Multi Learner 2008. Higher Education Authority: Anglia Ruskin University, Cambridge, UK. Mock, K. (2003). The development of a CSO course for distance delivery. The Journal of Computing Sciences in Colleges, 19(2), 30-38. Mokken, R. J. (1971). A theory and procedure of scale analysis. Berlin, Germany: De Gruyter. Molenda, M. (2003). In search of the elusive ADDIE model. Performance Improvement, 42(5), 34-36. Monk-Turner, E., & Payne, B. (2005). Addressing issues in group work in the classroom. Journal of Criminal Justice Education, 16(1), 166-179. Montgomery, K., & Wiley, D. (2004). Creating e-portfolios using PowerPoint. Thousand Oaks, CA: Sage Publications. ISBN 0-7619-2880-4. Moodle. (2008). Moodle VLE website. Retrieved August 20, 2008, from http://moodle.org/ Moore, M. G. (1973). Toward a theory of independent teaching and learning. The Journal of Higher Education, 44, 661-679. Moore, M. G. (2007). The theory of transactional distance. In M. G. Moore (Ed.), Handbook of Distance Education (2nd edition, pp. 89-105). New Jersey: Earlbaum.
289
Compilation of References
Moore, V., Armatas, C., & Saliba, A. (2005). The role of trait anxiety and impression management in adolescents’ use of the Short Message Service on mobile telephones. In M. Katsikitis (Ed.), Proceedings of the 40th annual conference of the Australian Psychological Society:Past reflections, future directions (pp. 213-217). Melbourne: The Australian Psychological Society Ltd. Morgan, M., Brickell, G., & Harper, B. (2008). Applying distributed cognition theory to the redesign of the copy and paste function in order to promote appropriate learning outcomes. Computers & Education, 50(1), 125-147. Morris, R., & Hayes, C. (1997). Small group work: Are group assignments a legitimate form of assessment? Proceedings of the 6th Annual Teaching and Learning Forum. Retrieved from http://lsn.curtin.edu.au/tlf/tlf1997morris.html Motiwalla, L. F. (2005). Mobile learning: A framework and evaluation. Computer and Education, 49(3), 581596. Mulder, R. A., & Pearce, J. M. (2007). Praze: Innovating teaching through online peer review. In R. Atkinson, C. McBeath, A. S. S. Kit, & C. Cheers (Eds.), 24th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (pp. 727-736). Singapore: ASCILITE Naidu, S. (2007). Instructional designs for distance education. In M. G. Moore (Ed.), Handbook of Distance Education (2nd Edition), (pp. 247-258), Mahwah, NJ: Erlbaum. Naidu, S., & Järvelä, S. (2006). Analyzing CMC content for what? Computers & Education, 46(1), 96-103. Naidu, S., Menon, M., Gunawardena, C., Lekamge, D., & Karunanayaka, S, (2007). How can scenario-based learning engender and promote reflective practice in online and distance education. In M. Spector (Eds.), Finding Your Online Voice: Stories Told by Experienced Online Educators (pp. 53-72), NJ: Lawrence Erlbaum. National Center for Education Statistics (2003). Distance learning in higher education institutions: 2000-2001.
290
(Rep. No. NCES 2003-017). Washington, DC: National Center for Education Statistics. Nevgi, A., & Löfström, E. (2005). The quality of online learning – Teachers’ and students’ evaluations of meaningful learning experiences in online and blended courses [Electronic version]. In S. Kiefer, J. Michalak, A. Sabanci, & K. Winter (Eds.), Analysis of educational policies in a comparative educational perspective (pp. 96–104). Linz, Austria: Pädagogische Hochschule des Bundes in Öberösterreich. Nevgi, A., & Tirri, K. (2003). Hyvää verkko-opetusta etsimässä. Oppimista edistävät ja estävät tekijät verkkooppimisympäristöissä – opiskelijoiden kokemukset ja opettajien arviot [In search of good online teaching. Factors promoting and hindering learning in online learning environments – Students’ experiences and teachers’ evaluations]. Kasvatusalan tutkimuksia 15. Turku, Finland: Suomen Kasvatustieteellinen Seura. New Media Consortium. (2006). The horizon report. Retrieved from http://www.nmc.org/pdf/2006_Horizon_Report.pdf. Newson, J. (1999). Techno-pedagogy and disappearing context. Academe, 85(5), 52-56. Ngu, A. H. H., Shepherd, J., & Magin, D. (1995). Engineering the ‘Peers’ system: The development of a computer-assisted approach to peer assessment. Research and Development in Higher Education, 18, 582-587. Nichols, M. (2008). Institutional perspectives: The challenges of e-learning diffusion British Journal of Educational Technology, 39(4), 598-609. Nichols, M. (2008). Institutional perspectives: The challenges of e-learning diffusion. British Journal of Educational Technology, 39(4), 598-609. Nichols, M., & Anderson, W. G. (2005). Strategic e-learning implementation. Distance Education Association of New Zealand (DEANZ) Electronic discussions. Retrieved November 30, 2005, from http://deanz-discuss.massey. ac.nz/july2005.html
Compilation of References
Nicol, D. (2007a). Laying a foundation for lifelong learning: Case studies of e-assessment in large 1st-year classes. British Journal of Educational Technology, 38(4), 668-678. Nicol, D. (2007b). E-assessment by design: Using multiple-choice tests to good effect. Journal of Further and Higher Education, 31(1), 53-64. Nicola, D., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. Nielsen, J. (1993). Usability engineering. Boston: Academic Press. Nielsen, J. (1994). Heuristic evaluation. In J. Nielsen & R.L. Mack (Eds.), Usability inspection methods (pp. 25-64). New York: John Wiley & Sons. Nightingale, P., Te Wiata, I., Toohey, S., Ryan, G., Hughes, C., & Magin, D. (1996). Assessing learning in universities. Sydney: University of New South Wales Press. Northcote, M. (2002). Online assessment: Friend, foe or fix? British Journal of Educational Technology, 33(5), 623-625. Novak, J. D. (1998). Learning, creating, and using knowledge: Concept maps as facilitative tools in schools and corporations. Mahwah, NJ: Lawrence Erlbaum Associates. Nunnally, J. C. & Bernstein, I. H. (1994). Psychometric theory (3rd Edition). New York: McGraw-Hill. O’Reilly, T. (2005). What Is Web 2.0, Retrieved May, 2008, from http://www.oreillynet.com/pub/a/oreilly/tim/ news/2005/09/30/what-is-web-20.html O’Reilly, T. (2005). What is Web 2.0: Design patterns and business models for the next generation of software. Retrieved May 1, 2008, from http://www.oreillynet. com/pub/a/oreilly/tim/news/2005/09/30/what-is-web20.html Oblinger, D. (2003). Boomers, Gen-Xers and Millennials: Understanding the new students. EDUCAUSE Review,
38(4) 37-47. Retrieved August 15, 2008, from http://www. educause.edu/ir/library/pdf/ERM0342.pdf Oliver, B., & Goerke, V. (2007). Australian undergraduates’ use and ownership of emerging technologies: Implications and opportunities for creating engaging learning experiences for the Net Generation. Australasian Journal of Educational Technology, 23(2), 171-186. Oliver, M., & Trigwell, K. (2005). Can ‘blended learning’ be redeemed? E-Learning, 2(1), 17-25. Oliver, R., & Herrington, J. (2001). Teaching and learning online: A beginner’s guide to e-learning and e-teaching in higher education. Mt Lawley, Western Australia: Centre for Research in Information Technology and Communications, Edith Cowan University. Op’t Eynde, P., De Corte, E., & Verschaffel, L. (2001). “What to learn from what we feel”: The role of students’ emotions in the mathematics classroom. In S. Volet & S. Järvelä (Eds.), Motivation in learning contexts. Theoretical advances and methodological implications (pp. 149–167). Amsterdam: Pergamon. Open Sim. (2008). Open Simulator Organisation wiki. Retrieved August 20, 2008, from http://opensimulator. org/wiki/ OpenCroquet. (2008). Open Croquet Organisation home page. Retrieved August 20, 2008, from http://www. opencroquet.org/ Orvis, K. L., & Lassiter, A. R. L. (2005). Computer-supported collaborative learning: The role of the instructor. In S. P. Ferris & S. H. Godar (Eds.), Teaching and learning with virtual teams (pp. 158-179). Hershey, PA: Information Science Publishing. Paloff, R., & Pratt, K. (2003). The virtual student: A profile and guide to working with online learners. San Francisco: Jossey-Bass. Panko, M. (2006). Learning and assessment: A case study – going the full Monty. In T. S. Roberts (Ed.), Self, peer and group assessment in e-learning (pp. 85-100). Hershey, PA: Information Science Publishing.
291
Compilation of References
Parker, K. R., & Chao, J. T. (2007). Wiki as a teaching tool. Interdisciplinary Journal of Knowledge and Learning Objects, 3, 57-72.
http://www.marcprensky.com/writing/Prensky%20-% 20Digital%20Natives,%20Digital%20Immigrants%20%20Part1.pdf
P a y n e , D . A . ( 2 0 0 3) . A p p l i e d e d u c a tional assessment. Belmont: CA: Wadswor th / Thompson Learning.
Pulkkinen, J. (2005, March 14). Online learning: Enablers and barriers for educational technology. Paper presented as visiting scholar at Western Carolina University, Cullowhee, NC.
Peel, M. (2000). ‘Nobody cares’: the challenge of isolation in school to university transition. Journal of Institutional Research, 9(1) Retrieved May, 2008, from http://www. aair.org.au/jir/May00/Peel.pdf Pena-Shaff, J. B., & Nicholls, N. (2004). Analyzing student interactions and meaning construction in computer bulletin board discussions. Computers & Education, 42(3), 243-265. Phipps, R., & Merisotis, J. (1999). What’s the difference? A review of contemporary research on the effectiveness of distance learning in higher education. Institute for Higher Education Policy for the National Education Association and the American Federation of Teachers. Retrieved November 10, 2007, from http://www2.nea. org/he/abouthe/diseddif.pdf Pickford, R. T. (2006, July). The art of teaching: A model for the lecture in the 21st century. Paper presented at the The Higher Education Academy Annual Conference. Poikela, E. (2006). Knowledge, knowing and problembased learning – Some epistemological and ontological remarks. In E. Poikela & A. R. Nummenmaa (Eds.), Understanding problem-based learning (pp. 15–31). Tampere, Finland: Tampere University Press. Poikela, E., & Nummenmaa, A. R. (Eds.). (2006). Understanding problem-based learning. Tampere, Finland: Tampere University Press. Poikela, E., & Poikela, S. (2006). Developing contextbased assessment within the framework of problem-based learning. In E. Poikela & A. R. Nummenmaa (Eds.), Understanding problem-based learning (pp. 227−240). Tampere, Finland: Tampere University Press. Prensky, M. (2001). Digital natives, digital immigrants. On the Horizon, 9(5). Retrieved May 1, 2008, from
292
Puolimatka, T. (2004). Tunteiden kognitiivisuus ja oppiminen [Cognitivity of emotions and learning]. Aikuiskasvatus [The Finnish Journal of Adult Education], 2/2004, 102−110. Raban, R., & Litchfield, A. (2006, 3-6 December). Supporting peer assessment of individual contributions in groupwork. Paper presented at the 23rd Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Sydney. Raban, R., & Litchfield, A. (2007). Supporting peer assessment of individual contributions in groupwork. Australasian Journal of Educational Technology, 23(1), 34-47. Raban, R., & Litchfield, A. (2007). Supporting peer assessment of individual contributions in groupwork. Australasian Journal of Educational Technology, 23(1), 34-47. Raban, R., & Litchfield, A. (2007). Supporting peer assessment of individual contributions in groupwork. Australasian Journal of Educational Technology, 23(1), 34-47. Raitman, R., Ngo, L., Augar, N., & Zhou, W. (2005). Security in the online e-learning environment. In P. Goodyear, D. G. Sampson, D. J. Yang, T. Kinshuk, R. Okamoto, Hartley & N. Chen, (Eds.), The proceedings of the 5th IEEE International Conference on Advanced Learning Technologies (ICALT’05) (pp. 702-706). IEEE Computer Society: United States. R a m a g e , T. R . (2 0 0 2). T h e “n o s i g n i f i c a n t d i f f e r e n c e” p h e n o m e n o n : A l i t e r a ture review. Retrieved November 10, 2007, from http://www.usq.edu.au/electpub/e-jist/docs/html2002/ ramage.html
Compilation of References
Ramsden, P. (2003), Learning to teach in higher education (2nd edition). London: RoutledgeFalmer. Ramsden, P. (2003). Learning to teach in higher education (2nd ed.). London: RoutledgeFalmer. Reeves, B., Malone, T. W., & O’Driscoll, T. (2008). Leadership’s online labs. Harvard Business Review, May, 59-66. Reeves, T.C. (2003). Storm clouds on the digital education horizon. Journal of Computing in Higher Education, 15(1), 3-26. Reeves, T.C., & Carter, B.J. (2001). Usability testing and return-on-investment studies: Key evaluation strategies for Web-based training. In B. Khan (Ed.), Web-based training (pp. 547-557). Englewood Cliffs, NJ: Educational Technology Publications. Reeves, T.C., & Hedberg, J.G. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology Publications. Repo-Kaarento, S. (2004). Yhteisöllistä ja yhteistoiminnallista oppimista yliopistoon –Käsitteiden tarkastelua ja sovellutusten kehittelyä [Introducing collaborative and cooperative learning at university – Looking at the concepts and developing applications]. Kasvatus [The Finnish Journal of Education], 35(5), 499−515. Resta, P., & Laferrière, T. (2007). Technology in support of collaborative learning. Educational Psychology Review, 19(1), 65-83. Rice, A. K. (1970). The modern university: A model organisation. London: Tavistock. Rice, J. (2007). The rhetoric of cool: Composition studies and the new media. Carbondale: Southern Illinois University Press. Rideout, E., England, V., Oxford, B., Fothergill-Bourbonnais, F., Ingram, C., Benson, G., et al. (2002). A comparison of problem-based and conventional curricula in nursing education. Advances in Health Sciences Education, 7.
Roberts, T. S. (2006). Self, peer and group assessment in e-learning: An introduction. In T. S. Roberts (Ed.), Self, peer and group assessment in e-learning (pp. 1-16). Hershey, PA: Information Science Publishing. Roberts, T. S. (2008). Student plagiarism in an online world. In T. S. Roberts (Ed.), Student plagiarism in an online world: Problems and solutions (pp.1-9). Hershey, PA: Information Science Reference. Roblyer, M. D., & Wiencke, W. R. (2003). Design and use of a rubric to assess and encourage interactive qualities in distance courses. The American Journal of Distance Education, 17(2), 77-98. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: The Free Press. Rogers, G., & Williams, J. (1999). Building a better portfolio. ASEE Prism, 8(5), 30-32. Roschelle, J. (2003). Keynote paper: Unlocking the learning value of wireless mobile devices. Journal of Computer Assisted Learning, 19, 260-272. Rossett, A. (1987). Training needs assessment. Englewood Cliffs, NJ: Educational Technology Publications. Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education, 12, 8-22. Rowntree, D. (1977). Assessing students: How shall we know them? London: Harper & Row. Roy, N. L. S., Scheepers, H., Kendall, E., & Saliba, A. (2006). A comprehensive model incorporating mobile context to design for mobile use. In Proceedings of the 5th Conference on Human Computer Interaction (pp. 22-30) New York: ACM SIGCHI. Ruben, B. D. (1999). Simulations, games and experiencebased learning: The quest for a new paradigm for teaching and learning. Simulation & Gaming, 30(4), 498-505. Rubin, J. (1994). Handbook of usability testing. New York: John Wiley & Sons.
293
Compilation of References
Ruokamo, H. (2001). The Solver learning environment and anchored instruction on mathematical word problemsolving. In C. Montgomerie & J. Viteli (Eds.), Proceedings of ED-MEDIA 2001 – World Conference on Educational Multimedia, Hypermedia & Telecommunications (pp. 1595–1600) [CD-ROM]. Association for the Advancement of Computing in Education. Ruokamo, H., & Pohjolainen, S. (2000). Distance learning in multimedia networks project: Main results. British Journal of Educational Technology, 31(2), 117–125. Ruokamo, H., Tella, S., Vahtivuori, S., Tuovinen, H., & Tissari, V. (2002). Pedagogical models in the design and assessment of network-based education. In P. Barker & S. Rebelsky (Eds.), Proceedings of ED-MEDIA 2002 – World Conference on Educational Multimedia, Hypermedia & Telecommunications (pp. 1676–1681) [CDROM]. Association for the Advancement of Computing in Education. Rushton, C., Ramsey, P., & Rada, R. (1993). Peer assessment in a collaborative hypermedia environment: A case study. Journal of Computer-Based Instruction, 20(3), 73-80. Russell, T. L. (2001). The no significant difference phenomenon: A comparative research annotated bibliography on technology for distance education (5th ed.). Montgomery, AL: The International Distance Education Certification Center (IDECC). Sahin, S. (2008). The relationship between student characteristics, including learning styles, and their perceptions and satisfactions in web-based courses in higher education. Turkish Journal of Distance Education, 9, 123-138. Salmon, G. (2000) E-moderating: The key to online learning. London: Routledge Falmer. Salmon, G. (2002). E-tivities: The key to active online learning. London: Kogan Page. Salmon, G. (2004). E-moderating: The key to online teaching and learning. London: Routledge.
294
Sanson-Fisher, R. W., & Lynagh, M. C. (2005). Problembased learning: A dissemination success story? . Medical Journal of Australia, 183(5), 258-260. Santos, I., & LeBaron, J. (2006). Ethical constraints in the valid interpretation of transcribed communication in online study. The Internet and Higher Education, 9, 191-199. Savin-Baden, M. (2000). Problem-based learning in higher education: Untold Stories. Buckingham: Open University Press/SRHE. Scharf, D., Elliot, N., Huey, H. A., Briller, V., & Joshi, K. (2007). Direct assessment of information literacy using writing portfolios. Journal of Academic Librarianship, 33, 462-477. Schedler, K., Summermatter, L., & Schmidt, B. (2004). Managing the electronic government. From vision to practice. A volume in research in public management. Greenwich: Information Age Publishing Inc. Schmidt, H. G., Vermeulen, L., & van der Molen, H. T. (2006). Long term effects of problem-based learning: A comparison of competencies acquired by graduates of a problem-based and a conventional medical school. Medical Education, 40, 562-567. Schome Community. (2007). The schome-NAGTY Teen Second Life pilot final report: A summary of key findings and lessons learnt. Milton Keynes: The Open University. Retrieved from http://kn.open.ac.uk/public/document. cfm?docid=9851 Schön, D. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco: Jossey-Bass. Schön, D. (1987a). Educating the reflective practitioner. Paper presented to the American Educational Research Association, Washington, DC. Retrieved November 10, 2007, from http://edu.queensu.ca/~russellt/howteach/ schon87.htm Schön, D. (1987b). Educating the reflective practitioner. Paper presented at the American Educational Research
Compilation of References
Association annual conference, Washington, DC. Retrieved November 10, 2007, from http://edu.queensu. ca/~russellt/howteach/schon87.htm Schutz, P. A., & DeCuir, J. T. (2002). Inquiry on emotions in education. Educational Psychologist, 37(2), 125–134. Schwartz, D. L., & Hartman, K. (2007). It’s not video anymore: Designing digital video for learning and assessment. In R. Goldman, R. Pea, B. Barron, & S. J. Derry (Eds.), Video research in the learning sciences (pp. 335−348). Mahwah, NJ: Lawrence Erlbaum Associates, Publishers. Sedlacek, W. E. (2004). Beyond the big test: Noncognitive assessment in higher education. San Francisco: Jossey-Bass. Seppala, P., & Alamaki, H. (2003). Mobile learning in teacher training. Journal of Computer Assisted Learning, 19, 330-335. Serafini, F. (2004). Three paradigms of assessment: Measurement, procedure, and inquiry. In International Reading Association (Ed.), Preparing reading professionals (pp. 207-216). Newark, DE: International Reading Association. Sharma, M. D., Khachan, J., Chan, B., & O’Byrne (2005). An investigation of the effectiveness of electronic classroom communication systems in large lecture classes. Australasian Journal of Educational Technology, 21(2), 137-154. Sharp, S. (2006). Deriving individual student marks from a tutor’s assessment of group work. Assessment & Evaluation in Higher Education, 31(3), 329 – 343. Sharp, S. (2006). Deriving individual student marks from a tutor’s assessment of group work. Assessment & Evaluation in Higher Education, 31(3), 329-343. Sharpe, R., Benfield, G., Roberts, G., & Francis, R. (2006). The undergraduate experience of blended e-learning: A review of UK literature and practice. The Higher Education Academy. Retrieved from http://www.heacademy.
ac.uk/assets/York/documents/ourwork/research/literature_reviews/blended_elearning_full_review.pdf Shavelson, R. J., & Huang, L. (2003, January/February). Responding responsibly to the frenzy to assess learning in higher education. Change, 11–19. Shepherd, L. (2001). The role of classroom assessment in teaching and learning (CSE Technical Report 517). University of California, Los Angeles: Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing, Graduate School of Education & Information Studies. Shirky, C, (2008). Here comes everybody. The power of organizing without organisations. Penguin Group, Australia. Shirky, C. (2003). A group is its own worst enemy. Retrieved May 2, 2008, from http://shirky.com/writings/group_enemy.html Shudong, W., & Higgins, M. (2005). Limitations of mobile phone learning. In H. Ogara, M. Sharples, Kinshuk, Y. Yano (Eds.), Proceedings of the 3rd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2005 (pp. 179-181) Los Alamitos, USA: IEEE Computer Society. Shulman, L. S. (1992). Toward a pedagogy of cases. In J. H. Shulman (Ed.), Case methods in teacher education (pp. 1−30). New York: Teachers College Press. Sijtsma, K., & Molenaar, I. W. (2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage. Silander, P., Sutinen, E., Tarhio, J. (2004). Mobile collaborative concept mapping: Combining classroom activity with simultaneous field exploration. In J. Roschelle, T. Chan, Kinshuk & S. Yang (Eds.), Proceedings of the 2nd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2004), Mobile Support for Learning Communities (pp. 114-118) Los Alamitos, USA: IEEE Computer Society Sim, G., Holifield, P., & Brown, M. (2004). Implementation of computer assisted assessment: lessons from the
295
Compilation of References
literature. Association for Learning Technology Journal, Research in Learning Technology, 12(3), 215-229. Simonson, C. L. S. (1996). Teaching caring to nursing students. Journal of Nursing Education, 35, 100-104. Simpson, T. L. (2002). Dare I oppose constructivist theory? The Educational Forum, 66(4), 347-354. Simteach. (2008). Simteach educational wiki. Retrieved from http://www.simteach.com/wiki Sitthiworachart, J., & Joy, M. (2003, 9-11 July). Web-based peer assessment in learning computer programming. Paper presented at the 3rd IEEE International Conference in Advanced Learning Technologies, Athens, Greece.
hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. In T. M. Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction: A conversation (pp. 57–75). Hillsdale, NJ: Lawrence Erlbaum Associates. Spiro, R. J., Feltovich, P. L., Jacobson, M. J., & Coulson, R. L. (1991). Cognitive flexibility, constructivism, and hypertext: Random access instruction for advanced knowledge acquisition in ill-structured domains. Educational Technology, 31(5), 24-33. Spodark, E. (2005). Technoconstructivism for the undergraduate. Foreign Language Classroom Foreign Language Annals, 38(3) 428-435.
Sivan, A. (2000). The implementation of peer assessment: An action research approach. Assessment in Education, 7(2), 193-213.
Squire, K. (2006). From content to context: Video games as designed experience. Educational Researcher, 35(8), 19-29.
Sloan-C. (2008). A consortium of institutions and organizations committed to quality online education. Retrieved October 4, 2008, from http://www.sloan-c.org/
Steensels, C., Leemans, L., Buelens, H., Laga, E., Lecoutere, A., Laekeman, G. et al. (2006) Peer assessment: A valuable tool to differentiate between student contributions to group work? Pharmacy Education, 6(2), 111-118.
SLOODLE. (2008). SLOODLE organisation website. Retrieved August 20, 2008, from http://www.sloodle. org/ Sluijsmans, D. M. A., Dochy, F., & Moerkerke, G. (1999). Creating a learning environment by using self-, peerand co-assessment. Learning Environments Research, 1(3), 293-319. So, H. J., & Brush, T. A. (2008).Student Perceptions of Collaborative Learning, Social Presence and Satisfaction in a Blended Learning Environment: Relationships and Critical Factors , Computers & Education, 51(1), 318-336. Soini, H. (1999). Education students’ experiences of learning and their conceptions about learning disabilities. Towards a comprehensive theory of learning. Doctoral dissertation, University of Oulu. Acta Universitatis Ouluensis Scientiae Rerum Socialium 40. Oulu, Finland: Oulu University Press. Spiro, R. J., Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1992). Cognitive flexibility, constructivism and
296
Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19(1), 69-75. Steinkuehler, C. (2008). Massively multiplayer online games as an educational technology: An outline for research. Educational Technology, 48(1), 10-21. Stenhouse, L. (1989). The case-study tradition and how case studies apply to practice. In J. Rudduck & D. Hopkins (Eds.), Research as a basis for teaching. Readings from the work of Lawrence Stenhouse (pp. 52–55). Oxford: Heinemann Educational Books. Stockwell, G. (2007). Vocabulary on the move: Investigating an intelligent mobile phone-based vocabulary tutor. Computer Assisted Language Learning, 20(4), 365-383. Strauss, P., & Alice, U. (2007). Group assessments: Dilemmas facing lecturers in multicultural tertiary
Compilation of References
classrooms. Higher Education Research & Development, 26(2), 147-161. Strickland, J. (2008). Is there a Web 1.0? Retrieved April 21, 2008, from http://computer.howstuffworks. com/web-101.htm Struyven, K., Dochy, F., & Janssens, S. (2003). Students’ perceptions about new modes of assessment in higher education: A review. In M. Segers, F. Dochy & E. Cascallar (Eds.), Optimizing new modes of assessment: In search of qualities and standards (pp. 171-224). Boston/Dordrecht: Kluwer Academic. Sung, Y.-T., Chang, K.-E., Chiou, S.-K., & Hou, H.-T. (2005). The design and application of a web-based selfand peer-assessment system. Computers & Education, 45, 187-192. Surowiecki, K. (2004). The wisdom of crowds. New York: Doubleday. Swan, K. (2002). Building learning communities in online courses: The importance of interaction. Education, Communication and Information, 2(1), 23-49. Swan, K., Shea, P., Fredericksen, E., Pickett, F., Pelz, W. & Maher, G. (2001). Building Knowledge Building Communities: Consistency, Contact and Communication in the Virtual Classroom, Journal of Educational Computing Research, 23(4/2000), 359 - 383. Sweet, M., & Michaelsen, L. (2007). How group dynamics research can inform the theory and practice of postsecondary small group learning. Educational Psychology Review, 19(1), 31-47. Sweller, J., & Clark, R. (2006). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner. Educational Psychologist, 42(2), 99-107. Teo, S. C. S., & Williams, J. B. (2006). A theoretical framework for effective online course design. MERLOT Journal of Online Learning and Teaching, 2(1). The Higher Education Academy. (2008). Groupwork, Retrieved August 6, 2008 from http://www.heacademy. ac.uk/ourwork/learning/assessment/Group
Thompson, L. (2007). Podcasting: The ultimate learning experience and authentic assessment. In R. J. Atkinson, C. McBeath, S. K. A. Soong, & C. Cheers (Eds.), ICT: Providing choices for learners and learning. Proceedings ascilite Singapore 2007 (pp.1019-1023). Singapore: Centre for Educational Development, Nanyang Technological University. Retrieved August 15, 2008, from http://www. ascilite.org.au/conferences/singapore07/procs/thompson-poster.pdf Thornton, P., & Houser, C. (2004). Using mobile phones in education. In J. Roschelle, T. Chan, Kinshuk, & S. Yang (Eds.), Proceedings of the 2nd IEEE International Workshop on Wireless and Mobile Technologies in Education (WMTE 2004), Mobile Support for Learning Communities (pp. 3-10) Los Alamitos, USA: IEEE Computer Society. Thorpe, M. (2008). Effective online interaction: Mapping course design to bridge from research to practice. Australasian Journal of Educational Technology, 24(1), 57-72. Topping, K. (1998). Peer assessment between students in colleges and universities. Review of Educational Research, 68(3), 249-276. Topping, K. J., Smith, E. F., Swanson, I., & Elliott, A. (2000). Formative peer assessment of academic writing between postgraduate students. Assessment & Evaluation in Education, 25(2), 149-66. Trenholm, S. (2007). A review of cheating in fully asynchronous online courses: A math or fact-based course perspective. Journal of Educational Technology Systems, 35(3), 281-300. Triantafillou, E., Georgiadou, D., & Econimides, A. A. (2006) The design and evaluation of a computerized adaptive test on mobile devices, Computers and Education, 50(4), 1319-1330. Tucker, R. (2008). The impact of assessment modes on collaborative group design projects. In S. Frankland (Ed.), Enhancing teaching and learning through assessment: Embedded strategies and their impacts, 2, 72-85. Hong
297
Compilation of References
Kong: The Assessment Resource Centre, The Hong Kong Polytechnic University.
assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28(1), 3-24.
Tucker, R., & Reynolds, C. (2006). The impact of teaching models on collaborative learning in the student design studio. Journal for Education in the Built Environment, 1(2), 39-56.
van den Berg, I., Admiraal, W., & Pilot, A. (2006). Peer assessment in university teaching: Evaluating seven course designs. Assessment & Evaluation in Higher Education, 31(1), 19-36.
Tucker, R., & Rollo, J. (2006). Teaching and learning in collaborative group design projects. Journal of Architectural Engineering & Design Management, 2 (Teaching and Learning in the Built Environment), 19-30.
van der Ark, A. (2007). Mokken scale analysis in R. Journal of Statistical Software, 20(11), 1-19.
Tucker, R., Fermelis, J., & Palmer, S. (2007). Online self-and-peer-assessment for teamwork in architecture and business communications. In J. Coulson, D. Schwede & R. Tucker (Eds.), Towards solutions for a liveable future: Progress, practice, performance, people: The 41st annual conference of the Architectural Science Association ANZAScA (pp. 264-271). Geelong: Deakin University. Twigg, C. (1998, May 2). A regulated monopoly enterprise gives way to competition. AAHE Bulletin. Retrieved November 12, 2007, from http://www.aahe.org/bulletin/bull_2may98.htm#Twigg Tynjälä, P. (2001). Writing, learning and the development of expertise in higher education. In P. Tynjälä, L. Mason, & K. Lonka (Eds.), Writing as a learning tool. Integrating theory and practice (pp. 37−56). Dordrecht: Kluwer Academic Publishers. Ulmer, G. L. (2003). Internet invention: From literacy to electracy. New York: Longman. Vahtivuori, S., Tella, S., Tissari, V., Tuovinen, H., Ruokamo, H., & Vaattovaara, V. (2003). First evaluations of network-based education: Preliminary findings of the ICT programs evaluation project. In D. Lassner & C. McNaught (Eds.), Proceedings of ED-MEDIA 2003 Conference – World Conference on Educational Multimedia, Hypermedia & Telecommunications (pp. 3177–3183) [CD-ROM]. Association for the Advancement of Computing in Education. van Abswoude, A. A. H., van der Ark, L. A., & Sijtsma, K. (2004). A comparative study of test data dimensionality
298
van der Linden, W. J., & Hambleton, R. K. (Eds.) (1997). Handbook of modern item response theory. New York: Springer-Verlag. Verduin, J. R., & Clark, T. (1991). Distance education: The foundations of effective practice. San Francisco: Jossey-Bass. Vernon, D. T., & Blake, R. L. (1993). Does problem-based learning work? A meta-analysis of evaluative research. Academic Medicine, 68(7), 542-554. Voithofer, R. (2002). Nomadic epistemologies and performative pedagogies in online education. Educational Theory, 22(4), 479-494. von Solms, B. (2001). Information security – a multidimensional discipline. Computers & Society, 20, 504-508. von Solms, B., & Marais, E. (2004). From secure wired networks to secure wireless networks – what are the extra risks? Computers & Security, 23, 633-637. von Solms, B., & von Solms, R. (2004a). The 10 deadly signs of information security management. Computers & Security, 23, 371-376. von Solms, B., & von Solms, R. (2004b). From policies to culture. Computers & Security, 23, 275-279. Vrasidas, C., & McIsaac, M. S. (1999). Factors influencing interaction in an online course. The American Journal of Distance Education, 13(3), 22-35. Vygotsky, L. (1978). Mind in society: the development of higher psychological processes. Cambridge, MA: Harvard University Press.
Compilation of References
Vygotsky, L. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Walker, A. (2001). British psychology students’ perceptions of group-work and peer. Psychology Learning and Teaching, 1(1), 28-36. Wallace, R. M. (2005). Online learning in higher education: A review of research on interactions among teachers and students. Education, Communication, and Information, 3, 241-280. Warn, J. (2006). Plagiarism software: No magic bullet. Higher Education Research and Development, 25(2), 195-208. Watkins, R., & Corry, M. (2005). E-Learning companion: A student’s guide to online success. Boston: Houghton-Mifflin. Weimer, M. (2006). Enhancing scholarly work on teaching and learning. San Francisco: Jossey-Bass. Weippl, E. R. (2005). In-depth tutorials: Security in e-learning. eLearn Magazine, 3(3). Retrieved from http://www.elearnmag.org/subpage.cfm?section=tutor ials&article=19-1 Weippl, E. R. (2007). Dependability in e-assessment. International Journal on E-Learning, 6(2), 293-302. Wen M. L., & Chin-Chung T. (2008). Online peer assessment in an in service science and mathematics teacher education course. Teaching in Higher Education, 113(1), 55-67. Wen, M. L., & Tsai, C.-C. (2008). Online peer assessment in an inservice science and mathematics teacher education course. Teaching in Higher Education, 13(1), 55-67. Wenger, E., McDermott, R., & Snyder, W. M. (2002). Cultivating communities of practice. Boston: Harvard Business School Press. Wertsch, J. V., Del Rio, A. & Alvarez, A. (Eds.). (1995). Sociocultural studies of mind. Cambridge: Cambridge University Press.
Wheeler, P. (1996). Using portfolios to assess teacher performance. In K. Burke (Ed.), Professional portfolios: A collection of articles. (pp. 74-94). Victoria: Hawker Brownlow Education. White, S. (2007). Critical success factors for e-learning and institutional change – some organisational perspectives on campus-wide e-learning. British Journal of Educational Technology, 38 (5), 840-850. Whitworth, B. (2006). Social-technical systems. In C. Ghaoui (Ed.), Encyclopedia of Human Computer Interaction (pp. 533-541). London: Idea Group Reference. Whitworth, B., & Friedman, R. (2008). The challenge of modern academic knowledge exchange. SIGITE Newsletter, 5(2), 4-10. Wiecha, J. M., Gramling, R., Joachim, P., & Vanderschmidt, H. (2003). Collaborative e-learning using streaming video and asynchronous discussion boards to teach the cognitive foundation of medical interviewing: A case study. Journal of Medical Internet Research, 5(2). Retrieved from http://www.jmir.org/2003/2/e13/ Willey, K., & Freeman, M. (2006). Improving teamwork and engagement: The case for self and peer assessment. Retrieved 12 December, 2006, from http://www.aaee. com.au/journal/2006/willey0106.pdf Williams, D. D. (2005). Measurement and assessment supporting evaluation in online settings. In D. D. Williams (Ed.), Online assessment, measurement and evaluation: Emerging practices (pp. 1−8). Hershey, PA: Information Science Publishing. Williams, P. E., & Hellman, C. M. (2004). Differences in self-regulation for online learning between first- and second-generation college students. Research in Higher Education, 45, 71-82. Wilson, G., & Stacey, E. (2004). Online interaction impacts on learning: Teaching the teachers to teach online. Australasian Journal of Educational Technology, 20(1), 33-48. Winner, L. (1980). Do artifacts have politics? Daedalus, 101, 121-136.
299
Compilation of References
Wolfe, E. W. (1999). How can administrators facilitate portfolio implementation. High School Magazine, 6(5), 29-33. Woods, R., & Ebersole, S. (2003). Using non-subject matter specific discussion boards to build connectedness in online learning. The American Journal of Distance Education, 17(2), 99-118. Wright, A. (2005, January 26). From ivory tower to academic sweatshop. Salon.com. Retrieved Novembet 10, 2007, from http://www.salon.com/tech/feature/2005/01/26/distance_learning/print.html Wutoh, R., Boren, S., & Balas, E. A. (2004). Internet continuing education for health care professionals: An integrative review. Journal of Continuing Education in the Health Professions, 24(3), 171-180. Yao, Y., Thomas, M., Nickens, N., Downing, J. A., Burkett, R. S., & Lamson, S. (2008). Validity evidence of an electronic portfolio for preservice teachers. Educational Measurement, Issues and Practice, 27(1), 10-24.
300
Yen, W. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125-145. Zhang, L., & Sternberg, R. J. (2005). A threefold model of intellectual styles. Educational Psychology Review, 17(1), 1-53. Zhang, L., & Sternberg, R. J. (2006) The nature of intellectual styles. New Jersey: Erlbaum. Zimmerman, M. C. (2002). Academic self-regulation explains persistence and attrition in web-based courses: A grounded theory. Unpublished doctoral dissertation, Northern Arizona University, Flagstaff. Zubizarreta, J. (2004). The learning portfolio. Bolton, MA: Anker Publishing. Zurita, G., & Nussbaum, M. (2004). A constructivist mobile learning environment supported by a wireless handheld network. Journal of Computer Assisted Learning, 20, 235-243.
301
About the Contributors
Christine Spratt has a Bachelor of Education from the University of New England (Armidale), a Master’s Degree in Distance Education and a PhD (Education), both from Deakin University (Geelong). She and has worked in Australia and Singapore in a variety of academic teaching, research and senior management positions in the university, corporate and non-profit educational sectors in particular in health (Nursing, Health Sciences and Medicine).). Dr Spratt has particular research and pedagogical interests in curriculum design and development and e-learning, open and distance education. She has extensive experience in educational leadership in professional education managing a variety of education and curriculum projects at universities such as Monash, Deakin and the University of Tasmania in Australia. Paul Lajbcygier combines extensive industry and academic experience in investments. Since 1990, Paul has provided investment advice for various prominent domestic and international: funds managers, banks and hedge funds. Since 1995, Paul has published over 50 academic papers and generated over $3.1 million in government grants and payments in-kind. He has sat on over 10 journal editorial boards and conference program committees. He has also worked and researched at amongst the best business schools in the world, including: London Business School and the Stern School of Business, New York University. *** Dr Christine Armatas has a PhD in Psychology from the University of Melbourne and over twelve years experience working as an academic, including eight years at Deakin University teaching in the School of Psychology. In this role she was involved in promoting innovation in teaching and learning through innovative curriculum design for blended and fully online units. At Deakin she was an ‘Online Teaching and Learning Fellow’ and led the team which was awarded a grant from the Australian University Teaching and Learning Committee which funded an innovative multi-media learning resource for the teaching of research methodologies. She has recently returned to academia after working for three years in a commercial role in the telecommunications industry as a human factors and interaction design expert, researching innovative applications for emerging technologies to support teaching and learning. Her current role is as an Educational Developer at Victoria University.
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
About the Contributors
Dr Benson is a Senior Lecturer in Educational Design and e-Learning in the Faculty of Medicine, Nursing and Health Sciences at Monash University, Australia. She has an MEd (Hons) in adult education and her PhD is in distance education. She has worked in educational design, development and research for over twenty years and during this time has undertaken and coordinated evaluation activities and programs, developed and offered academic professional development programs, and worked with teaching staff to increase the flexibility of their learning and teaching activities and resources. In recent years a particular focus of her work has been on the support of good teaching practice through the appropriate use of new learning technologies. Charlotte Brack joined Monash University in 2003. She has an academic background in Biochemistry. She pursued research and teaching in the discipline and became increasingly interested in the challenges and theories of teaching and learning. She developed early computer-facilitated modules for teaching molecular mechanisms, moving to problem-based and online methods within the discipline. Charlotte is interested in exploring educational design in online environments, in particular: situated learning, case and problem-based learning, experiential, authentic learning. She is also interested in using new technologies to support learning communities. Working in the Faculty of Medicine, Nursing and Health Sciences at Monash, Charlotte has developed online, face-to-face and blended learning opportunities within interdisciplinary contexts. Elaine Brown is a Senior Lecturer in Computing and Technology at Anglia Ruskin University. She is interested in the affordances of representation (designs to programs, avatars to people), in particular the affordances of virtual worlds for learning and teaching. Her avatar name in Second Life is Kur Ash. Dr Bernard Colbert has been actively involved in the design and analysis of the security of information systems and networks for over 15 years. He worked as a security consultant to assess the security of IT assets of major banks and government organisations. His research is in the fields of Cryptography and Robust Network Design. He is currently working in Telstra’s Chief Technology Office in Melbourne Australia and is a Visiting Researcher at Deakin University in Melbourne. Fadi P. Deek is Dean of the College of Science and Liberal Arts, Professor of Information Systems, Information Technology, and Mathematical Sciences at NJIT. He is also a member of the Graduate Faculty - Rutgers University Business School. His research interests include Learning/Collaborative Systems, Software Engineering, Open Source Development, Computer Science Education. Dr. Deek has published over 140 articles in journals and conference proceedings and he has given over 40 invited and professional presentations. He is also the co-author of ten book chapters and three books: Open Source – Technology and Policy, Strategic Software Engineering – An Interdisciplinary Approach, and Computer-Supported Collaboration with Applications to Software Development. Greg Duncan is a Lecturer in Pharmacy Practice at Monash University. Greg has postgraduate qualifications in public health and is involved internationally with practice development and education. He teaches in the Faculty programs in pharmacy and medicine. His particular expertise is in Evidence-Based Practice (EBP) and Public Health. He designs and conducts graduate and continuing professional development programs in evidence-based clinical practice and plays a key role in the development of wound care and EBP skills for health professionals through professional development
302
About the Contributors
programs in Australia and internationally. He is heavily involved in the development and evaluation of a range of innovations in teaching and learning in the health care area and has presented at international conferences. He recently presented at an international symposium on assessment at the Prato campus of Monash University. Norbert Elliot is Professor of English at New Jersey Institute of Technology. He is author, most recently, of ‘On a Scale: A Social History of Writing Assessment’, a volume that was awarded the 2007 Outstanding Book Award from the Conference on College Composition and Communication. His most recent collaborative assessment work has been published in the ETS Research Report series, Journal of Writing Assessment, Technical Communication, International Journal on E-Learning, and IEEE Transactions on Professional Communication. His research and teaching interests cluster in the following areas: History, Practice, and Theory of Writing Assessment; Theory and Practice of Technical Communication; Biography; Theory and History of Environmental Rhetoric; Empirical Research Methods; Technology and Education. Jan Fermelis is a Senior Lecturer with the School of Management and Marketing at Deakin University. She has made significant contributions to the strategic priorities of Deakin in relation to graduate attributes and international education, having received university and national awards for her passionate teaching of business and academic communication, and her efforts to improve the learning outcomes of international students. Following a secondment as a Fellow of the Institute of Teaching and Learning, she was recently invited to become a founding member of the College of Distinguished Deakin Educators. In recent years Jan has worked to improve her research profile, publishing on topics related to pedagogy within higher education, international education, family business and business communication. She is currently conducting research for a doctoral award in the area of Australian intercultural business communication in Shanghai. Robert S. Friedman is Associate Professor of Humanities and Information Technology at New Jersey Institute of Technology, where he directs the B.S. in Science, Technology and Society program. Recent publications include “On Heuristics, Narrative and Knowledge Management” for Technovation, Principal Concepts of Technology and Innovation Management: Critical Research Models, and Collaborative Learning Systems: A Case Study. Current research includes explication of the confluence of science, technology, and environmental expression in 19th-century American texts, and the development of an open electronic archiving system for information and computing scholarship based on an open source ethos and building on successful social computing methodologies. Marie Gordon is a Senior Lecturer in Computing and Technology, at Anglia Ruskin University, specialing in networking Linux,and Open Source applications. She has set up and supports departmental VLEs such as Moodle, wordpress and MediaWiki. She Introduced Second Life into the main curriculum, and has a great deal of practical experience with a combination of in-world / off-world skills from deep immersion in SL. Her avatar name in Second Life is Sandry Logan. Dr Päivi Hakkarainen received her PhD from the University of Lapland, Finland, in 2007. Currently she is lecturer in media education at the University of Lapland. Her research interests are educational uses of digital videos, meaningful learning, development of teaching, pedagogical models, and problem-based learning. 303
About the Contributors
Mike Hobbs is a Senior Lecturer in Computing and Technology and Pathway Leader for the Computer Gaming and Animation Technology degree at Anglia Ruskin University. Teaching courses in programming languages, artificial intelligence, game design and development. Researching into applications of Artificial Intelligence, knowledge management and more recently involved in learning and teaching with virtual worlds and helped to host the Higher Education Academy’s Massively Multi Learner workshop in 2008 His Second Life avatar name is Austin Thatcher. Professor Hurst’s research interests include programming languages and software engineering, particularly in the areas of persistence, compiler construction, literate programming, document technologies, and, most recently, computing education. He is co-leader of a research group called the Computing Education Research Group (CERG), which has undertaken several significant research projects. The most significant of these was an investigation into best practice and innovation in IT and CT, funded by the Australian Universities Teaching Commission for $300,000 over two years. Currently he leads the IntelliVEE project, which is investigating “pedagogically agnostic” e-learning systems. He holds BSc and BE degrees from The University of Adelaide, and a PhD in Computer Science from the University of New South Wales. He also completed a Graduate Certificate in Higher Education at Monash UNiversity, during his term as Associate Dean (Teaching). More recently, he served a term as President of the Academic Board at Monash University. Dr. Markham is a psychologist with extensive experience in education and in consulting. His core professional area is vocational behavior and careers counseling. He was a comprehensive background in educational and psychological measurement. This has ranged from standard educational assessment within teaching environments through to the development of psychological assessment tools in traditional as well as cross-cultural contexts, including developing careers assessment tools for the Indonesian technical education system. He has also carried out evaluation projects on assessment systems for both educational and commercial organizations. Dr.Markham has also been working with computing-based systems for most of his career. This involvement began with the educational use of an early HP minicomputer at Queensland Institute of Technology. Some years later he wrote a careers support tool for the Apple IIe. In 1997 he developed an internet-based career development system. More recently he has supported the Computing Education Research Group at Monash University in the development of various projects on computers in education. Dr. Som Naidu is Director, Teaching and Learning Quality Enhancement and Evaluation Services, Charles Sturt University, Division of Teaching and Learning Services, and Adjunct Associate Professor in the Graduate School of Management & Technology, University of Maryland University College, USA. Dr. Naidu possesses undergraduate qualifications in Education from New Zealand and graduate qualifications in Educational Technology from Montreal, Canada. His career in the field of educational technology dates back to the early 1980s during which period he has worked in various roles in the fields of educational technology, e-learning and also distance education in Canada, Australia, United Kingdom and the USA. More recently, he has been working on several educational development projects in India, Sri Lanka, Malaysia and Singapore. Dr. Naidu’s research interest is on learning experience design in online educational environments. His publications in this area include several books, book chapters, journal articles and conference papers. Dr. Naidu has been Executive Editor (since 1997) of the international peer-reviewed journal “Distance Education” [http://www.tandf.co.uk/journals/car-
304
About the Contributors
fax/01587919.html], the journal of the Open and Distance Learning Association of Australia which is published by the Taylor & Francis Group. He is Co-series Editor of the RoutledgeFalmer book series on “Open and Flexible Learning” also published by the Taylor and Francis Group [http://www.routledge. com/education]. Dr. Stuart Palmer is a Senior Lecturer in the Institute of Teaching and Learning at Deakin University. He is a chartered professional engineer, having practiced in consulting engineering for a decade before joining the School of Engineering at Deakin University. He lectured in the management of technology for twelve years, working with engineering students studying in on-campus, off-campus, off-shore and online modes. In 1999 he was awarded the Australasian Association for Engineering Education McGraw-Hill New Engineering Educator Award. His research interests include frequency domain image analysis and the effective use of digital/online technologies in teaching and learning. More recently, he has joined the Deakin University Institute of Teaching and Learning where he contributes to institutional research and academic professional development. Proifessor Heli Ruokamo is currently professor and the director of the Centre for Media Pedagogy at the University of Lapland, Finland. She received her PhD from the University of Helsinki, Finland, in 2000. Her research interests include network-based education, pedagogical models, playful learning environments, virtual learning environments, and mobile learning. Dr. Tarja Saarelainen received her PhD from the University of Lapland, Finland, in 2003. She teaches in the areas of public management. Her current research focuses on strategic partnerships and network management practices in purchaser-provider models. Dr. Saliba has a BSc(Hons) in Biochemistry majoring in Psychology, with a PhD in Perceptual Psychology. Anthony’s previous role was as Director of Consumer Behaviour Research at Telstra Research Laboratories; the use of mobile phones as educational tools was one of the major enabling research programs. His research seeks to understand why people do things in a particular contexts’. He has worked in diverse fields such as telecommunications and aviation, and is currently working for the National Wine and Grape Industry Centre as a Sensory Scientist. His current research aims to better understand the wine consumer. Dr. Andrew Sanford is currently a Lecturer in the Department of Accounting and Finance where he teaches undergraduate units in finance. He previously worked for over twelve years in the Banking, Finance and IT industries before joining academia to complete his PhD. His research focuses on the application of Bayesian analysis to empirical finance and educational research. Dr. Richard Tucker is an internationally recognised specialist in the pedagogy of collaborative and sustainable design. His work has involved substantial grant-funded projects that have enhanced learning satisfaction and outcomes for his students. He has over 25 publications and submitted eight grant applications in a previously neglected area of education. His acknowledged teaching excellence lies in the area of teaching innovation through his research and related scholarship to enable effective collaborative, multidisciplinary, studio-based learning in design and sustainable building. Richard’s research has led to improvements in collaborative learning across Deakin University, the promise of improvements in
305
About the Contributors
this area in other higher education institutions, and has been recognised through a Carrick citation and the WJC Banks Award as the most outstanding teacher at Deakin University in 2007. Richard’s teaching leadership was recently recognised by his appointment as Associate Head of School (Teaching and Learning) and by his admission as a Fellow of the College of Distinguished Deakin Educators. Dr. Paul J. White is a Senior Lecturer and Alternate Associate Dean (Education) in the Faculty of Pharmacy and Pharmaceutical Sciences at Monash University in Melbourne, Australia. He completed a PhD in 1996, and was a post-doctoral scientist at the Murdoch Children’s Research Institute prior to his academic appointment at Monash University in 1998. Dr White is heavily involved in innovative teaching and learning approaches, including leadership and ‘coalface worker’ roles at the Faculty and University level. Dr White has been an invited participant in Australian Government funded projects in 2001 and 2007 aimed at improving teaching and learning outcomes in Higher Education. His areas of interest within higher education include active learning in lectures, formative and summative assessment online, and peer assessment.
306
307
Index
Symbols
B
3D Modeling in Second Life 66
‘blended’ technologies 89 Biology Bridging Program 41 biometric systems 104 biometric technologies 102 blended e-learning 58 breadth discovery 8
A “active” lecture environment 76 academic knowledge 56 access control 99, 106 access control systems 107 accurate user identification 100 active learner engagement 254 active learning 24 active worlds learning environment 57 ADDIE (Analysis, Design, Development, Implementation, and Evaluation) model 234 Anglia Ruskin Computing (ARC) 62 Art of Illusion 66 assessment 1 assessment, types of 5 assessment of learning 91 assessment practices, formative 76 assessment practices, summative 76 assessment tasks 79 association 60 asynchronous online quizzes 87 audacity 66 audience response system 76, 80, 82 audience response system, general pedagogical issues 87 audience response system, integration of 86 audience response system, limitations of 85 audit 108 AWEDU 57
C case-based teaching 22 case-based teaching approach 20 Center for Technology in Education (CTE), The 247 change, drivers for 77 co-operative learning 24 cognitive skills/alignment 78 collaboration 51 collaborative e-learning 37 collaborative idea generation 155 collaborative knowledge 24 collaborative learner engagement, assessing the efficacy 256 collaborative learner engagement, practical strategies for 254 collaborative learning 39 collaborative learning, peer assessment 40 collaborative learning tasks 22 collaborative projects 37 collaborative wiki 46 communication 60 communication, flexibility of 122 communication, speed of 122 community 71
Copyright © 2009, IGI Global, distributing in print or electronic forms without written permission of IGI Global is prohibited.
Index
computer-mediated 255 computer-mediated conferencing 61 computer-networked 254 computer-networked education 255 computer-networked learning environments 255, 256 computer-networked settings 254 computer-supported collaborative learning (CSCL) 255 computer assisted assessment (CAA) 99 computer science courses 153 computer science courses, analysis of undergraduates 153 computer technologies 155 consequential validity 3 constructive learning 24 constructivist 55 construct validity 3 contemporary learning environments 218 content attainment approach 78 content delivery 55 content validity 3 conversational learning 24 conversation logging tools 62 corporate learning 137 creativity 59 criterion-related validity 3 critical thinking skills 22, 76 Cronbach’s Alpha correlations for learning style 159 CSCL efficacy 262 CSRs 240
D “distance learning” courses. 255 “distance learning” students 255 data integrity mechanisms 105 Deakin University psychiatry units 109 denial of service (DoS) attacks 98 depth discovery 8 DIF analysis 196 differential item functioning 195 digital immigrants 121 digital natives 121 digital portfolio 244, 245 digital rights management (DRM) 107
308
digital signature 106 digital signature generation 105 digital video (DV) 22 discussion forum 62 distance education, postgraduate program 243 document management system 106
E e-assessment 97 e-learning 1 e-learning (EL), empirical assessment of 151 e-learning, collaborative 37 e-learning, educational design for 41 e-learning, influence on validity and reliability 11 e-learning, innovative approaches to 76 e-learning, issues in peer assessment 117 e-learning, management and administration challenges 128 e-learning, pedagogical challenges 127 e-learning and assessment, implications for 208 e-learning and peer assessment, future of 129 e-learning community, developing the 46 e-learning course 20 e-learning courses, computer science and humanities 151 e-learning courses, validation of 151 e-learning environment 1, 6 e-learning environment, assessing group learning in 139 e-learning environment, reassessing validity and reliability 1 e-learning environment, stimulating group learning 137 e-learning environments 117, 218 e-learning environments, functionalities 120 e-learning environments, implications for group learning 146 e-learning environments, self-and-peer assessment tool 170 e-learning environments, users 121 e-learning for formative and summative assessment 77 e-learning for peer assessment, challenges of 127
Index
e-learning for peer assessment, opportunities 122 e-learning group dynamics 136 e-learning in higher education institutions 86 e-learning model, levels and tasks 62 e-learning program 38, 244 e-learning settings 23, 136 e-learning settings, proxy for individual learning 136 e-learning strategies, holistic 59 e-learning students, cohort of 195 e-learning technologies 56, 76 e-learning technologies, traditional use of 58 e-learning technology, transformative use of 58 e-learning tools 117 e-learning tools, functionalities 120 e-portfolio xv, 243 e-portfolio, definition 245 e-portfolios, integrated 248 e-portfolios, learning 246 e-portfolios, showcase 246 e-portfolios, structured 246 e-portfolios, types of 246 e-settings 254 educational assessment devices 1 educationally oriented 13 effectiveness evaluation 236 electronic content origination (IECO) module 66 electronic delivery 1 electronic information exchange system 152 electronic mediation 10 electronic portfolio (EP) 247 emotions generated by the course, student perspective 29 encoding 224 engagement, level and nature of 65 ethics 250 evaluation functions 235 evaluation pyramid 236
F face-to-face (FTF) format 153 face-to-face course 23 face validity 3
file management systems 107 Finnish Sports Federation (FSF) 22 flexible interaction 61 formative assessment 6, 76, 80 formative evaluation 236 formative SAPCA feedback, forms of 180 FRAPS 66
G GIMP 66 good practice principles 82 Google 56 grade point average (GPA) 153 graphical user interface 71 group, functioning as a 50 group discussion 155 group dynamics 137 group functioning skills 146 group identity 49 group learning 48, 146 group learning assessment 137 group marks, validity of 136 group operators 145, 146 group project 43, 140 group project, discussion 145 group project, implications for group learning 146 group project, study results 142 group project work 136 group skills 137 group treasure hunt task 62 group wiki 51 group work goals 64 group working 64 Group Work Project Implementation 61 Group Work Project Methodology 61 group writing 155
H hardware tokens 101 higher education 55 humanities courses, analysis of undergraduates 154 HYBRID model 198
309
Index
I ICT Skills, learner 8 identification 99 identification methods, considerations 103 identification of users 100 immediacy 71 impact evaluation 236 independent thinkers 10 individuality 24 individual learning 136 individual student blogs 62 individual total peer assessment (ITPA) 177 information access system 7 information and communication technologies 79 information and communication technologies (ICT) 7 information communication technology (ICT) 77 information literacy 161 information literacy scores 163 information organizer 10 information technology (IT) 98 Inkscape 66 innovation networks 22 innovative assessment development 15 integrated e-portfolio project 250 integrative learning 248 integrity of data 99 inter-organizational co-operation 21 interactivity 71 International Society of Technology in Education 56 item category characteristic curves (ICCCs) 196 item response theory (IRT) modeling 196
K knowledge differentiation 63 knowledge domains 121 knowledge reliability 13 knowledge validity 1, 11
L latent classes, identifying 195 Leapfrog Biology homepage 41
310
Leapfrog modules, activities in 42 learner’s ICT skills 8 learner knowledge 254 learning activities 79 learning by becoming 60 learning by doing 60 learning e-portfolio 246, 247 learning experiences 47 learning journals 22 learning management platforms (LMS) 257 learning management system (LMS) 80, 87, 100, 244 learning management systems, teacher-centred nature of 124 learning models for virtual worlds 58 learning objectives, types of 77 learning outcomes, informative 70 learning outcomes, reflective 70 learning outcomes, social 70 learning outcomes, technical 70 learning portfolio 248 learning process 28 learning style 157, 160 learning styles, empowered 65 learning styles, realistic 65 learning styles, superficial 65 linear regression analysis 88 logging 108 logging and auditing data 99
M machinima 67 maintenance evaluation 236 Massey University 243 Massively Multi-user On-line Role Playing Games (MMORPGs) 56 meaningful learning, process characteristics 24 meaningful learning, student perspective 27 meaningful learning processes 20 mobile device, is learning effective 218 mobile devices 221 mobile group structures 66 mobile learning 218 modular structure 42 monitoring mechanisms 98 motivation 48 multimedia portfolio 245
Index
multiple choice questions (MCQ) 99 multiplicative scaling factor (MSF) 173, 176
N needs assessment 236 net generation 121 network-based education 23 networking competence 22 network management 22 network management, e-learning version 21 network management course 20 New Jersey Institute of Technology (NJIT) 152 Nobel Factor project 46
O objective assessment 6 on-line content repositories 56 on-line learning 155 online assessment, Deakin University psychiatry units 109 online assessment, ensuring security 97 online assessment, formative and summative purposes 99 online assessment, integrity of data 97 online education, postgraduate program 243 online identity 39 online quizzes, rationale for 87 online quizzes, unsupervised 88 open and distributed learning environments 234 open and distributed learning environments, evaluation strategies 234 open assessment 6 organizational management 21 osmosis 80 ownership 246
P parallel forms reliability 2 passwords 101 peer-based relationships 117 peer assessment 37, 40, 118 peer assessment, issues in 117 peer assessment of learning 117 peer learning 37
peer production 37, 47 peers 37 peer support 37 peer to peer learning 66 persistence 71 personal identification number (PIN) 101 PGP (Pretty Good Privacy) 105 plagiarism 14 Plymouth Sexual Health 56 Pre-Web 2.0 technologies 123 predictive validity 3 problem-based learning (PBL) 23, 81 problem-based learning literature 55 problem-solving 22 problem based learning (PBL) 57 problem based learning criteria 69 public administration and management 21 public key cryptography (PKC) 105
Q QWERTY keyboard 220
R ‘real world’ learning 119 reflective practitioner (RP) 256 reliability 1 reliability, definition 2 reliability and validity, establishing 13 reliability and validity, history 5 reliability and validity, structural relationship 4 retention 22
S “smart” mobile phones 218 S/MIME (Secure/Multipurpose Internet Mail Extensions) 105 SAPA approach 173 SAPA ratings 173 SAPA system 173 SAPCA, designing assignments for 175 SAPCA, during team assignment 185 SAPCA, individualisation of team scores 175 SAPCA, inducting students 184 SAPCA, making assessments 175 SAPCA, managing team assignment 184
311
Index
SAPCA, software and procedural refinements 181 SAPCA, training staff 184 SAPCA data, individualise student results 185 SAPCA documentation, preparation of 183 SAPCA effectiveness 180 SAPCA feedback screenshot 177 SAPCA instructor feedback detailed ratings screenshot 179 SAPCA instructor feedback screenshot 178 SAPCA instructor feedback team-list screenshot 178 SAPCA piloting 181 SAPCA screenshot 176 SAPCA transparency 180 Scholarship of Teaching and Learning (SoTL) 257 searchers after knowledge 10 Second Life (SL) 55, 56 Second Life, environment of 55 Second Life, getting started in 67 Second Life, Machinima film sets 68 self-and-peer assessment (SAPA) 170 self-and-peer continuous assessment 174 self-and-peer continuous assessment (SAPCA) 170 self-and-peer continuous assessment, general principles of 174 self-and-peer sssessment tool 170 self-and-peer sssessment tool, designing 170 self-and-peer sssessment tool, evaluating 170 self-and-peer sssessment tool, implementing 170 self-assessment 254 self-directed learning 37 self-direction in learning 24 self-managed learner 9 self learning skills 56 self motivated learning 55 short message service (SMS) 101 showcase e-portfolio 246 showcase e-portfolios 246 SL, community aspects of 56 SL environment 58 smartcards 103 social bookmarking 37
312
social constructivism 39, 56 social interaction 59 social interactions 7 social networking applications 56 social software 37, 39, 49 social software, assessment 40 social software, collaborative learning 39 social software, concepts of 40 social software, transition to university 40 socio-technical approach 1 socio-technical environment 7 socio-technical systems (STSs) 155 split-half reliability 3 structured e-portfolio 246 student-centred models 55 student cohort 199 student interest and engagement 22 student learning, pedagogical implications 85 student learning behavior, changes in 8 student participation, forms of 179 students’ learning activities 22 students’ meaningful learning 20 student skills, facilitating development of 189 subscriber identity module (SIM) 101 summative assessment 6 summative assessment, using asynchronous discussion 89 summative assessment, validity of 87 summative assessment purposes 76
T task based learning 66 teacher-focused approaches 77 teacher-student interactions 7 teaching, assessing 20 teaching, student perspective on 27 teaching/learning process oriented 14 teaching and meaningful learning (TML) 20 team assignment 175, 183 team formation 183 team mean peer assessment (TMPA) 177 team performance 175 team total peer assessment (TTPA) 177 technology devices 224 technology oriented 14 telepresence 59
Index
test-retest reliability 2 theoretical knowledge 22 TML model 21, 23 traditional classroom teaching 255 traditional site-based teaching 254 transition 51
U unit assessment structure 92 University Centre for Learning and Teaching (UCLT) 61 usernames 101 users, teachers and students 121
V validity 1 validity, definition 3 varied autonomous learning 66 video editing 66 virtual businesses 56 virtual classroom 152 virtual integration 64 virtual learning environment (VLE) 56, 62 virtual on-line environments 66 virtual world 55
virtual world educational issues 59 virtual worlds 55, 56 virtual worlds, assessment in 70 virtual worlds, learning and assessment with 55 virtual worlds, learning models for 58 Virtual Worlds Group Work Project 61 virtual worlds in education 69
W Web 2.0 37 Web 2.0 technologies 38, 117, 126 WebCT 87, 244 WebCT/Blackboard 80 web links 62 wicked problems 22 Wiki 37, 43 wiki, discussion in 48 Wikipedia 56 wikis 37 wikis, criteria used to grade 44 Wikispaces 43 Winning Student Wiki homepage 47 wireless technology 98 World of Warcraft 56
313
