Artificial Intelligence Review 14: 1–4, 2000. Stephen J. Hegner et al. (eds.), Intelligent help systems for UNIX. © 2000 Kluwer Academic Publishers. Printed in the Netherlands.
1
Editorial Intelligent Help Systems for UNIX: Computational Models and Systems
This is the first in a series of three special issues focussed on intelligent help systems for UNIX,1 each with its own emphasis: (1) computational models and systems, (2) planning and knowledge representation and (3) natural language dialogue. In this first issue focussing on computational models and systems there are five papers, one addressing empirical foundations, another virtues and problems, with the final three describing comprehensive implemented systems. Jerrams-Smith presents some empirical studies conducted on UNIX users which resulted in a taxonomy of error types so that users’ errors could be classified. In turn, this information is encapsulated as production rules within a knowledge base which forms the basis for an intelligent interface to UNIX. The prototype makes inferences about users’ mental models and uses these to select appropriate tutorial advice. A comparison was conducted between users of the prototype intelligent interface and the usual UNIX interface. Users of the prototype made fewer errors, had fewer misconceptions and took less time to complete given tasks. Virvou et al. look at the virtues and problems of an active help system for UNIX. A passive consultant accepts questions from the user, but is otherwise ignorant of the user’s goals, while an active consultant continually monitors users’ actions, attempting to discover their goals from these actions. An empirical study encompassing a cross-section of UNIX users at an academic site showed the need for an active, rather than passive, help system. Examples of system interaction supporting this view are given, and a discussion of the help needed in these examples is provided. It is proposed that to provide such help requires the construction and maintenance of a model of each user. The next three papers focus on comprehensive systems which have been built to perform as intelligent help systems for UNIX. In the first, Wilensky et al. describe UC (UNIX Consultant), a natural-language interface that allows naive users to learn about the UNIX operating system. The development of UC was undertaken because the task was thought to be a useful domain
2
EDITORIAL
for AI research, as well as a useful application of AI work in planning, reasoning, natural language processing, and knowledge representation. UC contains a number of components for language understanding (ALANA), inference (concretion mechanism), goal and plan analysis and generation (PAGAN, UCEgo), domain planning (KIP), content expression (UCExpress), and language generation (UCGen), user’s knowledge state with respect to UNIX (KNOME), and enabling addition of knowledge of both English and UNIX facts to UC’s knowledge base (UCTeacher). All components of UC make use of a knowledge representation system called KODIAK. Mc Kevitt describes OSCON (Operating System CONsultant), which is also a natural-language operating system consultant capable of answering queries on over 40 commands from both the UNIX and MS-DOS2 operating systems. OSCON differs from UC in that it is specialised more on the domain of operating systems, whereas UC has the broader goal of studying knowledge representation, planning and natural language processing through operating systems. UC employs a much more general knowledge representation and planning mechanism with a more cognitive flavour. Mc Kevitt also points to more recent developments where empirical studies with subjects using UNIX through Wizard-of-Oz techniques give frequencies of types of questions asked and enable the incorporation of user modelling into OSCON. Matthews et al. present USCSH (University of South Carolina SHell), an active intelligent help system for UNIX. USCSH can operate in two modes, the active mode and the intelligent mode. In the active mode, USCSH monitors users’ interactions and at appropriate times makes suggestions on how the user might better use the system. This sense of active is similar to that used by Virvou et al. above. In the intelligent mode, the system accepts natural language questions and responds to them, taking into consideration the context and user’s ability. A trend that has been showing up in commercial applications of AI: translating a working prototype into a language that can be more easily embedded into a running application, was employed in the development of this system. A prototype in Lisp was rewritten in C so that it could be called directly from the C shell. To sum up, there are a number of ways of looking at a domain of application such as UNIX, depending on whether one is interested in modelling the domain itself or the cognitive models applied to the domain. Also, empirical studies are useful for obtaining knowledge about how subjects interact with domains such as UNIX, and then this knowledge can be incorporated within systems. Finally, “active” rather than “passive” help systems are more appropriate for situations in which the system attempts to model the plans of the user over time. The latter problems of planning and knowledge representation
EDITORIAL
3
will be the subject of the next special issue on intelligent help systems for UNIX. The articles in this issue are derived from papers originally presented at a workshop entitled Knowledge representation in the UNIX help domain, organised by Peter Norvig, Wolfgang Wahlster and Robert Wilensky at the University of California, Berkeley, USA, in December, 1987. The workshop was funded, in part, by the International Computer Science Institute and all participants were invited. The area of intelligent help systems for UNIX provides such a fruitful example domain of application for many AI techniques that we have decided to publish this work, which is still timely, more widely and particularly now where we have many spoken dialogue systems applied to such fixed domains (see Bernsen et al. 1998, BusinessWeek 1998). Each article here has been reviewed by the editors and has been subsequently revised; furthermore, all authors have been asked to include a section on recent developments on their work. Related work which may be of interest to the reader can be found in Kobsa and Wahlster (1988) which focusses on user modelling and in Maybury and Wahlster (1998) which is a comprehensive publication on intelligent user interfaces. Members of the Kluwer editorial and publishing staff are to be thanked for their help in producing this issue. We regret that John Jones passed away in 1994 in a climbing accident in Peru. A special thanks to David Chin for preparing the paper on UC herein. The Editors, April 2000, Stephen J. Hegner Umeå University Sweden Paul Mc Kevitt The Queen’s University of Belfast Northern Ireland Peter Norvig NASA Ames Research Center California, USA and Robert Wilensky University of California at Berkeley USA
4
EDITORIAL
Notes 1 UNIX is a trademark of X/Open, Inc. 2 MS-DOS is a trademark of Microsoft Corporation.
References Bernsen, N-O., Dybkjær, H. & Dybkjær, L. (1998). Designing Interactive Speech Systems: from First Ideas to User Testing. Springer-Verlag: New York, NY. BusinessWeek (1998). Speech technology is the next big thing in computing, Special Report. BusinessWeek, Feb. 23. New York: McGraw-Hill. Kobsa, A. & Wahlster, W. (1988). User Models in Dialog Systems. Springer-Verlag: Berlin. Maybury, M. & Wahlster, W. (eds.) (1998). Readings in Intelligent User Interfaces. Morgan Kaufmann Publishers: Los Altos, CA.
Artificial Intelligence Review 14: v–ix, 2000. Stephen J. Hegner et al. (eds.), Intelligent help systems for UNIX.
v
About the Authors
Gautam Biswas is an Associate Professor of Computer Science, Computer Engineering, and Management of Technology at Vanderbilt University. He received a B.Tech. degree in Electrical Engineering from the Indian Institute of Technology, Bombay, India, in 1977, and M.S. and Ph.D. degrees in Computer Science from Michigan State University, East Lansing, in 1979 and 1983, respectively. Dr. Biswas conducts research in Artificial Intelligence, Knowledge Based Systems, and Cognitive Modeling, and their applications to the development, analysis, and evaluation of Intelligent Systems. He has primary interests in hybrid modeling and analysis, and their applications to analysis, control, diagnosis, and design of complex embedded systems. He has applied hybrid modeling and combined qualitative-quantitative reasoning techniques to design fault monitoring and isolation systems for fast breeder reactors and automobile engine systems. Another area of interest is the design and implementation of Intelligent Learning Environments. Working as part of a Cognition and Technology group at the Learning Technology Center in Vanderbilt University, he has developed multi-media systems and simulation-based intelligent learning environments for middle school math and science education. In ONR-funded projects he has developed methods for assessing expertise in troubleshooting of Alternating Current circuits and sys-
tems. Dr. Biswas has published in a number of journals and contributed book chapters. He is an associate editor of the International Journal of Approximate Reasoning and the Journal of Applied Intelligence, and has served on the Program Committee of a number of conferences. He was co-chair of the 1996 Principles of Diagnosis Workshop, and a senior program committee member for AAAI 97 and 98. He is a Senior member of the IEEE Computer Society, ACM, AAAI, and the Sigma Xi Research Society. David N. Chin is an Associate Professor of Information and Computer Sciences at the University of Hawaii at Manoa. He received his B.S. in Physics and B.S. in Computer Science/Engineering from M.I.T. in 1980 and his Ph.D. in Computer Science from the University of California at Berkeley in 1987. While at Berkeley, he lead the design and implementation of UC, the UNIX Consultant, a natural language consultation program for the UNIX operating system. His doctoral dissertation concerned treating UC as an intelligent agent which has its own goals, plans, and model of the user. By detecting the right goals in appropriate situations, UC can take the initiative in its dialog with the user in order to volunteer information, correct user misconceptions, and reject unethical requests. UC’s user model, KNOME (KNOwledge Model of Expertise), ana-
vi
ABOUT THE AUTHORS
lyzed user queries and inferred the user’s level of UNIX expertise based on what kinds of facts the user does or does not know. Dr. Chin has served on the editorial board of the international journal User Modeling and User-Adapted Interaction since its inception and has twice chaired the International Conference on User Modeling. Professor Chin’s current interests include cognitive science, software engineering, geographic information systems, and artificial intelligence, particularly intelligent agents, natural language processing, user modeling, and intelligent interfaces. Stephen J. Hegner received the Ph.D. degree in Computer and Information Science from the University of Massachusetts at Amherst in 1977. Currently, he is a senior lecturer in the Department of Computing Science at Umeå University, Sweden. His areas of professional interest focus on algorithmic aspects of knowledge representation and inference in database and knowledgebase systems. Jenny Jerrams-Smith is a Principal Lecturer in Information Systems at the University of Portsmouth, UK, and director of the Advanced Interface research centre. She was formerly a Principal Researcher in Artificial Intelligence at Philips Research Laboratories, UK. She completed her PhD in Computer Science in 1986 at the University of Birmingham and her Master’s degree in Computer Science in 1982. She has a Post Graduate Certificate in Education (1968) and a Bachelor’s degree in Zoology (1967), both from the University of London. Her primary research interest is in intelligent/adaptive user interfaces, in which she has many refereed publications. She
has recently completed funded research projects on an adaptive interface for a telecare system and on adaptive navigational guidance for the WWW. John Jones was born in 1956 in England and died in 1994. He achieved a BSc First Class degree in Mathematics from the University of Nottingham in 1980. He completed his Ph.D. in Mathematical Logic at the Department of Mathematics, University of Nottingham. He was a Research Fellow in Artificial Intelligence at the University of Edinburgh from October 1983 until September 1986. He joined the Department of Computer Science at the University of Hull as a Lecturer in October 1986 and remained there until he died. James Martin is an Associate Professor in the Department of Computer Science and the Institute of Cognitive Science at the University of Colorado at Boulder. He received his Ph.D. in Computer Science from the University of California at Berkeley in 1988. He earned his B.S. degree in Computer Science from Columbia University in New York in 1981. His current research interests are in Natural Language Processing, Information Retrieval and Machine Learning. Manton Matthews is Associate Professor and Director of Graduate Studies in the Department of Computer Science at the University of South Carolina. He received his Ph.D. in Mathematics (graph theory) while simultaneously obtaining a MS in Computer Science in 1980. He has served as program chairman for the 1990 International Conference on Industrial and Engineering Applications of AI and Expert Systems in 1990 and then again as
ABOUT THE AUTHORS
co-chairman in 1997. He has served on the editorial board of the Journal of Applied Intelligence. His research interests include intelligent user interfaces, natural language processing (NLP), user-modelling, graph theory, and forensic computing. James Mayfield is a Senior Computer Scientist at the Johns Hopkins University Applied Physics Laboratory. Prior to joining Johns Hopkins, he was an associate professor of computer science at the University of Maryland, Baltimore County. Dr. Mayfield received an A.B. in 1979 from Harvard College, and a Ph.D. in 1989 from the University of California at Berkeley. Since the time of the reported plan recognition work, he has carried out research in hypertext, information extraction, distributed inference, and agent communication languages. His current projects include work in information retrieval, computer forensics, and simulation. Paul Mc Kevitt is 36 and from Dún Na nGall (Donegal), Ireland. He is Associate Professor (Senior Lecturer) in the School of Computer Science at The Queen’s University of Belfast, Northern Ireland. Previously, he has been Visiting Professor of Intelligent MultiMedia Computing in the Institute of Electronic Systems at Aalborg University, Denmark and a British EPSRC (Engineering and Physical Sciences Research Council) Advanced Fellow in the Department of Computer Science at the University of Sheffield, England. The Fellowship, commenced in 1994, and released him from his Associate Professorship (tenured Lecturership) for 5 years to conduct full-time research on the integration of natural language, speech and vision pro-
vii
cessing. He completed a Master’s degree in Education (M.Ed.) at the University of Sheffield in 1999. He completed his Ph.D. in Computer Science at the University of Exeter, England in 1991. His Master’s degree in Computer Science (M.S.) was obtained from New Mexico State University, New Mexico, USA in 1988 and his Bachelor’s degree in Computer Science (B.Sc., Hons.) from University College Dublin (UCD), Ireland in 1985. His primary research interests are in Natural Language Processing (NLP) including the processing of pragmatics, beliefs and intentions in dialogue. He is also interested in Philosophy, MultiMedia and the general area of Artificial Intelligence. Mark Millington is 39 and from Chester, England. He is a consultant specialising in finance and telecommunications applications, working in the Chicago area. He completed his Ph.D. in Computer Science in 1985 at the University of Edinburgh, Scotland, and his Bachelor’s degree in Computer Science and Mathematics in 1980 at the University of Essex, England. Harish Neelakandan, 34, completed his Master’s degree in Computer Science from the University of South Carolina in 1987 and worked at Cognitive Systems in New Haven, Connecticut, from 1988 to 1992. While at Cognitive, which was founded by Roger Schank, then at Yale, to commercialize his group’s Natural Language research, Harish worked on the development of a Natural Language system for processing financial funds transfer telex messages. This system was deployed at some of the major money center banks in the USA and Europe. Subsequently, he worked on a Finan-
viii
ABOUT THE AUTHORS
cial Portfolio Evaluation system for use by Portfolio Managers and then on a Case-Based Reasoning system to detect suspicious trading activity at a major North American stock exchange. He left Cognitive in 1992 to get a graduate degree in Management at MIT’s Sloan School of Management in Boston, and upon graduation from Sloan in 1994, joined Merrill Lynch in New York where he is currently a Foreign Exchange Options trader. Peter Norvig received a B.S. in Applied Mathematics from Brown University and a Ph.D. in Computer Science from the University of California at Berkeley. He has been a Professor at the University of Southern California and a Research Faculty Member at Berkeley. He has over forty publications in various areas of Computer Science, concentrating on Artificial Intelligence, Natural Language Processing and Software Engineering. He has also served as a Senior or Chief Scientist at Sun Microsystems Labs, Harlequin Inc. and Junglee Corp., and is currently Chief of the Computational Sciences Division at NASA Ames Research Center. Walter Pharr is an Assistant Professor of Computer Science at the College of Charleston, in Charleston, South Carolina, USA. He is currently completing his Doctoral degree at the University of South Carolina. His primary research interests are natural language processing, formal semantics for natural and artificial languages, and functional programming. Maria Virvou is 34, from Athens, Greece. She has been a lecturer in the Department of Computer Science at
the University of Piraeus, Greece since 1994. She completed her Ph.D. in Artificial Intelligence and Computer Science at the University of Sussex, England in 1992. She received her M.Sc. degree in Computer Science from University College London in 1987. Her first degree in Mathematics was obtained from the University of Athens, Greece in 1986. Her current research interests are in user modelling, Artificial Intelligence in Education, distance learning and adaptive user interfaces. Robert Wilensky received his B.A. and his Ph.D. from Yale University. In 1978, he joined the faculty of the University of California at Berkeley, where he is now Professor in the Division of Computer Science, and in the School of Information Management and Systems. He has served as Chair of the Computer Science Division, the director of BAIR, the Berkeley Artificial Intelligence Research Project, and the director of Berkeley Cognitive Science Program. Professor Wilensky has published numerous articles and books in the area of artificial intelligence, planning, knowledge representation, natural language processing, and digital information systems. He is currently Principal Investigator of UC Berkeley’s Digital Library Project. Dekai Wu is an Associate Professor of Computer Science at the Hong Kong University of Science and Technology (HKUST). He received his PhD in Computer Science from the University of California at Berkeley, and was a postdoctoral fellow at the University of Toronto (Ontario, Canada) prior to joining HKUST in 1992. He received his BS in Computer Engineering from the University of California at San Diego
ABOUT THE AUTHORS
(Revelle College) in 1984. He was also a visiting researcher at the Technische Universität München (Munich, Germany) during 1986–87. He currently serves on the Editorial Boards of Computational Linguistics, Machine Translation, Computational Linguistics and Chinese Language Processing, and Communications of COLIPS and on the Executive Committee of the Association for Computational Linguistics (ALC). His research interests include statistical
ix
and corpus-based natural language processing, machine translation, language modeling, speech recognition, Chinese computing, machine learning, language acquisition, dialog systems, and Internet information processing. Since 1992 he has led the SILC Machine Translation Learning Project, and helped found the Human Language Technology Center (HLTC) at HKUST. He is also active as a Principal Investigator in the Cyberspace Center of HKUST.
Artificial Intelligence Review 14: 5–22, 2000. Stephen J. Hegner et al. (eds.), Intelligent help systems for UNIX. © 2000 Kluwer Academic Publishers. Printed in the Netherlands.
5
An Intelligent Human-Computer Interface for Provision of On-Line Help JENNIFER JERRAMS-SMITH Department of Information Systems, University of Portsmouth, University House, Winston Churchill Avenue, Portsmouth, Hants, PO1 2UP, UK (E-mail:
[email protected])
Abstract. Some user interfaces, such as that of Unix,1 are difficult for novices to use, and this paper suggests a possible solution to such problems. The results of a study of Unix users enabled the development of a taxonomy of error types so that users’ errors can be classified. This information is encapsulated as production rules within a knowledge base and forms the basis for the design and development of an intelligent interface to Unix. The prototype makes inferences about users’ mental models and uses these to select appropriate tutorial advice. Performance of users of the prototype intelligent interface was compared with that of users of the usual Unix interface. The prototype users were found to make fewer errors, exhibit fewer misconceptions and take less time to complete a standard set of tasks. Keywords: user models, intelligent interface, knowledge based system, adaptivity, Unix
1. General Interface Problems and Possible Solutions Many reports indicate that computer applications are difficult to use and that users have insufficient expertise to benefit from their full range of functions (Hayes et al. 1981; Lang et al. 1981; Nickerson 1981; Hayes and Szekely 1983). More problematic, there is evidence that the application’s response to an error may be confusing (Mishra et al. 1984) or even positively unhelpful and misleading, thus causing additional errors (Hanson et al. 1984). In particular, the Unix operating system is seen as being difficult for novices to learn and use, as indicated by the interest in the provision of improvements such as COUSIN-Unix (Hayes and Szekely 1983), and the “natural language” interfaces of the Unix Computer Consultant (Douglass and Hegner 1982), the Unix Consultant (Wilensky et al. 1984), and the Unix File Orienter (Mishra et al. 1984). Many of the current suggestions for remedying users’ problems with interfaces focus on studying users (Ramsay and Attwood 1979; Hollnagel 1983; Norman 1983). Other guidelines recently suggested for interface development indicate that the interface should facilitate error handling but allow
6
JENNIFER JERRAMS-SMITH
freedom of expression, and should be easy to learn but appealing to experienced users (Shneiderman 1979); should provide feedback and a sense of presence (DuBoulay et al. 1981; Ehrenreich 1981); should provide not just answers, but answers which help users to achieve their goals (James 1981); should protect users from their own errors (James 1981) both to prevent serious consequences and because errors made in the first few attempts at a new activity tend to become ingrained and should enable commands to be chosen by recognition rather than recall (Bailey 1982). Associated with recent work on difficulties with interaction, there are numerous recommendations that an effective method for enabling users to make the best use of a system is by including a tutorial element within the system (James 1981; Tagg 1981; Relles and Price 1981; Bailey 1982; Scapin 1981), instead of (or in addition to) the provision of a stand-alone tutor such as the Unix tutor (Irgon and Martin 1984). Further, exploration-based learning is believed to be educationally preferable to that of instruction-based learning (Carroll 1984; Kamouri et al. 1986). The author provides a solution to the problems users have with Unix and other applications. Unix was chosen as an example application because it is an operating system whose use is becoming more widespread, but which causes serious problems for novice users because of the enigmatic nature of its responses to their commands. The solution involves a study of Unix users, leading to the design and development of an additional intelligent interface to Unix, which includes a knowledge based system (KBS) and which also incorporates the current recommendations for interface design and development within the interface, as outlined above. This interface includes a tutorial component and is able to make inferences about users’ mental models so that inaccuracies can be remediated as soon as they are detected, which relates the work to that in the field of intelligent interfaces (Innocent 1982) and also to tutoring systems such as that by (Self 1974; Clancey 1986; Brown and Burton 1978; Burton and Brown 1979; Sleeman and Hendley 1979; Sleeman and Smith 1981; Finin 1983; and Johnson and Soloway 1984). In particular, the solution makes use of previous work on the acquisition of mental models by novices: Bayman and Mayer (1983) found that novices’ mental models (of BASIC) are often inaccurate, while Sleeman (1983) proposed the concept of malrules (the wrong rules which children use when they learn algebra). All user commands pass through the intelligent interface before they reach the application. This transparent interface monitors the user’s interaction with the application, but users are unaware of it except when the interface has inferred that tutorial help is required or that the user’s action could have harmful consequences. In this event, the interface immediately provides
PROVISION OF ON-LINE HELP
7
tutorial information at the moment when it is most needed and therefore of most value to the user. This enables “minimalist training”, as advocated by Carroll (1984), so that users can work in experimental mode while at the same time it traps their errors and helps them to develop a correct mental model of the application. For such a new type of interface there were few guidelines available for its development. Since possession of a correct mental model of the application cannot be assumed for novice users (Ramsay and Attwood 1979), it was decided to extend Norman’s (1983) suggestion that some rules for the design of the interface for a specific application might be derived from analysis of human error. The author therefore investigated user errors in order to discover if they indicate what help users need, and where the user’s mental model is inaccurate, and therefore what remedial tutoring is required. A method was devised which enabled the development of a taxonomy of error types so that errors could be classified. Subsequent investigation indicated the misconceptions (inaccuracies in the mental model) which were associated with some of the errors, and hence the remediation requirements. An investigation of the differences between errors made by experts and novices was also made. Fewer of these mistakes were expected to be caused by inaccuracies within the expert user’s mental model, since the accuracy of the mental model depends on the level of the user’s expertise (Carey 1983). The resultant information was encapsulated as production rules in the knowledge base within the intelligent interface.
2. An Empirical Study of Unix Users 2.1. Method of study A study was made of 55 novice users of Unix. These were first year undergraduates in software engineering, who were familiar with other operating systems. Observations were made over a four week period by logging all input to the Unix shell and simultaneously collecting verbal protocols from the subjects. 2.2. Data Figure 1 shows a sequence of commands given by one of the subjects. It also has the error categories into which each of the incorrect commands was subsequently classified. Figure 2 provides typical examples of verbal protocols provided by the subjects.
8
JENNIFER JERRAMS-SMITH Command
Error category (see Table 1)
elp help passwd robins help passwd passwd logot logout passwd help passwd hcat/manual/summary/passwdi passwd YYYYYYY login login passpasswd passwd help help passwd passwd (v) passwd (u) help passwd passwd aaaaaa aaaaaaa hoqrr uoou hcat/manual/summary/passwdi login cat/manual/summary/passwd help passwd hcat/manual/summary/passwd why not help goto help passwd help passwd help login /etc/passwd hcar/etc/passwd
5 – 1 10 – 1 5 – 1 – 1 1 10 7 7 14 7+2 – 7 7 – 1 10 10 10 10 1 7 1 – 1 13 13 – – – 13 + 11 1
Figure 1. Typical log of user commands their errors and categories.
2.3. Error analysis and classification Error analysis was carried out by an expert user of Unix who was also experienced in teaching novices how to use Unix, so that a taxonomy could be developed and errors classified in terms of this taxonomy. Errors were considered to be not only those actions which would produce a system error message, but also included commands which, although valid,
PROVISION OF ON-LINE HELP
9
June 14, 1983. 15.31. Just dabbling to see if I can do anything. It would appear that @ doesn’t work and removing underlines. Don’t know why that is. Just put in my password. Think I’ll leave it and come back after I’ve looked at the literature. June 15th, 1983. 13.23. Objective today to try and understand what this thing does. Since yesterday I’ve failed to do anything. Actually understand some of the text handling . . . files and things. Couldn’t manage to write to terminal G. Don’t know why that was. Well, so far we have achieved nothing again today. So I’m leaving off for today. June 16th, 1983. 13.32. Well, I’ve now got half an hour on this machine. We’ll call it junk.txt (using the ned screen editor). We appear to have – what about ˆF? If we can find the right spot in our Unix papers we can actually do something. Totally stuck on the literature. Well, we can honestly say it’s hopeless finding anything in this manual. (Consults friends. One explains how to use the em line editor). Well, I think I’ll start again. End of session three (note of despair or disgust in his voice). Figure 2. Typical verbal protocols.
make inefficient use of the system or do not produce the intended result. Errors which were caused by misconceptions were differentiated from errors caused by mistyping. 2.4. The meaning of errors – the differentiation between mistypes and misunderstanding Examples of command sequences which indicate inaccuracies in the user’s mental model were derived from the error analysis. Mistyping categories were also derived from the error analysis. The following gives additional guidelines to those provided by Damerau (1964) for the formulation of the rules indicating that mistyping has occurred. These were derived by analysis of the logs generated by the subjects’ interaction with Unix. It is very important to detect mistyping and to differentiate it
10
JENNIFER JERRAMS-SMITH
Table 1. Errors and percentages for the first 2 weeks and the second 2 weeks Error
Category
1st
%1st
2nd
%2nd
Total
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Unable to give command Failed try for help Wrong mode or directory Using previous knowledge Mistype Prevented Misunderstanding Inefficient use Misread documents Unknown Misconception Obscenities Guesses Loss of attention Forgot own file name Easier option available Known to be wrong Forgotten Documentation errors
40 17 7 21 20 7 5 6 5 0 0 0 0 0 0 0 0 0 7
29 12 5 15 14 5 3 4 3 0 0 0 0 0 0 0 0 0 5
14 39 18 32 77 10 5 55 11 57 45 2 2 26 3 8 9 3 0
3 9 4 7 18 2 1 13 2 13 10 0.5 0.5 6 0.5 1 2 0.5 0
54 46 25 53 97 17 10 61 16 57 45 2 2 26 3 8 9 3 7
from misunderstanding so that experienced users in particular are not given unnecessary advice (Number of occurrences are shown in brackets). 1. Omission of one character – this includes the omission of space between arguments in the case of an experienced user. (12) 2. Substitution of another character – next one to the right or left on the QWERTY keyboard. (8) 3. Repetition of one character. (16) 4. Repetition of two characters. (2) 5. Inversion of two characters. (4) 6. Extra characters at the start – this could be caused by incomplete deletion of incorrect characters. (17) 7. Repetition of a previous character instead of the correct one – example nror for nroff. (4)
PROVISION OF ON-LINE HELP
Command
Interpretation
em SC em SC
does not know the !hshell commandi facility when in em
SC f > f2 lf f2
does not understand > and pipes
rm f1 rm f2 rm f3
does not know that all files could be supported on one command line
cat f|lf
cat is redundant
create f em f
create is redundant since em automatically creates
SC f1 SC f2 SC f3
use of same command could indicate lack of knowledge of pattern match such as ∗
11
Figure 3. Typical command sequences indicating inaccuracy in the user’s mental model. (SC = any Shell Command, em = Editor, lf = List File, create = Create File).
3. Development of an Intelligent Transparent Interface for Unix Since this was an unusual application, the production of a prototype is an important part of the process of development and gave useful indications for the final design. The prototype, and the final version, were written in FRANZ LISP on a VAX 11/730. During prototype development the code remained uncompiled and underwent continual modification. The prototype contained an intelligent spell-checker in addition to the components for recognition and remediation of other types of errors. 3.1. Rule based inferencing A knowledge-based component was developed for inclusion within the prototype interface, and was based on an expert system suggested by Winston and Horn (1981). The knowledge base consisted of a set of production rules,
12
JENNIFER JERRAMS-SMITH
and the control mechanism allowed goal-directed search or pattern-directed search. The knowledge base consists of modules of knowledge in the form of production rule sets grouped according to function and accessed when required for solving particular parts of a problem. A production rule expresses knowledge in the form of a conditional part followed by some action. A simple example might be: IF (command has failed) (intended command has been deduced) (deduced command = passwd) (user is at novice level) THEN (explain how to set the password) The inference engine of the KBS uses facts deposited in the database to produce new facts or actions. It matches the conditionals of the rules with the facts in the database, and causes the required action when a match occurs. Thus deduction is carried out by forward chaining only. Forward chaining has previously been shown to be useful for modelling cognitive processes and for solving problems which require very broad but shallow knowledge. An explanatory facility was developed, similar to that of MYCIN (Shortliffe 1976), which enabled the user to ask for justification: how a fact was derived or why a fact was required. The output to the user was in pseudo natural language, which was aided by coding in LISP so that chosen parts of a rule were directly displayed on the screen. Pseudo-probabilities (confidence factors) were provided which were similar to those of MYCIN. It became clear from the experience of using the experimental KBS that it was more difficult than had been anticipated to decide on the values of confidence factors, and they were therefore omitted from later versions. As indicated above, it was also realised that the backward chaining mechanism (and therefore the “why” explanations) would also be un-necessary for this application and it was decide to omit them too. The “how” explanations allow users to follow the reasoning strategy of the KBS and was used during the testing and modification stage of development to ensure that the interface correctly deduced the user’s intentions. 3.2. User models The user model consists of two components: the generalised or static model and the dynamic or specific model. The static (generalised) model is derived from error and protocol analysis (held within the rules of the knowledge base), from published information on the use of Unix, from command usage frequency information and from likely previous knowledge of other Oper-
PROVISION OF ON-LINE HELP
13
ating Systems, such as Multics, CP/M or Tops-20 (held as a list of possible substitutes for each command). It is well known that people will attempt to apply their previous knowledge when attempting a new task, and numerous examples are visible in the user logs. However, as the logs indicate, this can sometimes hinder the successful completion of the new task. The Dynamic User Model includes the user’s model of the system, the user’s behaviour patterns, the user’s expertise level and the trace of the user’s behaviour for the current session. The record of previous activity is held on property lists: attributes for each input line include original command, deduced command, intended command, original arguments, and deduced arguments, intended arguments. Sequences which may indicate the occurrence of inefficient use are maintained, and the frequency of usage of commands and files is also recorded. In future versions, as part of the planning operation, the Dynamic User Model should also maintain possible sequences which might indicate a command which is likely to be used next, or a file which is currently being operated upon (example: edit file1 cat file1 lf file1). 3.3. Pre-testing As soon as the prototype was complete and had been tested to eliminate all apparent errors in the code, pre-testing was carried out in order to design the test-evaluation stage of development. The pre-test subjects answered questions which helped to validate the rules of the knowledge base. The interface with the completed interpretation module was pre-tested on a small group of expert users of Unix (one research associate, two research students). They were not constrained to use any specific operations, but asked to use it in whatever way they liked. The explanations facility of the KBS was used as a development tool which enables the subject to ask how each deduction was made and work back through a chain of reasoning. The rules were shown to the user in a natural language version; a property list holds the English language description of each function. The subjects indicated whether they believed each deduction to be true or false. Pre-testing revealed that additional rules and hypotheses were required. For example, if “cc fred” were given as input, “fred” is a valid file, but not a valid argument for “cc”. However “fred” would be a valid argument for “bc” or “wc” and this is discovered by the interface. The final hypothesis is that the input cannot be interpreted because the relevant rules state that it is only likely to be “cc” if no other command has matching arguments. Thus a rule is required which will choose between the two possibilities or offer both. Currently the interface attempts to choose the most likely and rules out all others.
14
JENNIFER JERRAMS-SMITH
The final pre-test was carried out using a standard sequence of operations which were completed by a single test subject (a computer expert but new to Unix). The tester followed a standard sequence of frequently used operations such as display a file on-screen, make a hard copy of a file, find which files are present in the current directory, delete a file, make a copy of a file, use the editor, use the text-formatter. By the end of this stage the LISP coding had been compiled in order to improve speed. 3.4. Action of interface for the final pre-test The following is an ordered list of the actions carried out by the intelligent interface for the final pre-test. 1. Production rules and Unix data are read in. 2. Menu, file-list and prompt simulator are output. 3. A command line from the user is read in. 4. A scanner divides this string of characters into tokens (example: divides “fred&” into “fred” and “&”). 5. Each token is checked for the types it might fit (examples: username, string, filename, file in current directory, valid command, digits, switch). 6. Each token is checked to discover if it could be a mistyped command or a mistyped file in the current directory. 7. The arguments are matched against each of the possible commands (examples: ps = ps, pr, ls; cap = cmp, cat, cal; ll = ln, l, ls; cp = cc, cp, cmp, cd). 8. The production rules are used to determine the user’s intention. This results in a list of hypotheses about the user’s intentions. 9. The explanations of the KBS are shown to the subject to discover the validity of the rules. Subjects indicate the truth/falsity of rules for their situation. 10. The command is passed to Unix to show its response. 11. The subject is asked for comment on the Unix response. 12. A record is kept within the interface of user commands, and of the user’s response to the rules and to Unix. 3.5. Modification of the rule base after pre-testing New rules were easily added to the rule set. Such rules were formulated after pre-testing on the basis of information provided by the user in response to the deductions offered by the interface. Rules were also modified and added after consultation with users during the full test. Ideally, the new rules would be added by the interface, but there was insufficient time to automate this activity.
PROVISION OF ON-LINE HELP
15
3.6. Evaluation of the interface Although usability studies were not in common use when this research was conducted, the transparent interface was subsequently evaluated by tests with users in order to discover how effective it is for Unix and also to discover which aspects might be generally applicable to other systems which are difficult for novices to learn and use. For this between-subjects study; 13 joint honours undergraduates in Mathematics and Computer Science were divided randomly into an experimental group (7 subjects) and a control group (6 subjects). Each subject in the experimental group was provided with a version of the intelligent interface, which was removed as soon as the study was complete. Testing involved completing a standard sequence of operations to be attempted in a given order and involving frequently-used Unix commands, an improved version of that used for the pre-test since, for instance, the pre-test indicated that the operation to find on-line help should be successfully completed before any other command is attempted. Logs were recorded for subjects and in addition a record was made of all the analyses and interpretations which the intelligent interface carried out for each input line plus the responses made by the subjects to the questions provided by the tutoring module. The following summarises the results. Further details are in Jerrams-Smith (1989). 1. Fewer mistakes were made by the experimental group. 2. The control group made many more repeats of identical errors. 3. Most members of the experimental group completed the sequence; most of the control group did not. 4. Fewer commands were given by members of the experimental group. 5. More misconceptions were found in the control group than in the experimental group. Protocol analysis was carried out in which the subjects attempted to describe their intentions and activities while they carried out the standard sequence, and this was recorded on tape. However, in most cases there was a greater emphasis on debriefing because many users found it extremely difficult to talk about what they were doing while also trying to learn new and complex tasks. This proved to be a fruitful exercise because it indicated important aspects of the intelligent interface which were not otherwise obvious. These included: 1. The directory listing provided as part of the intelligent interface provided feedback: it was used to verify that commands had acted as expected.
16
JENNIFER JERRAMS-SMITH
2. There was evidence of a wrong mental model in use, which was not detectable from the log: one subject thought that files had to be created before using the ‘mv’ or ‘cp’ commands. 3. There was evidence that the interface corrected misconceptions about Unix as shown in the following extract from the protocol transcript: “Edited ‘test3’ using vi editor and now using the spell command to check for spelling mistakes in ‘test3’. (pause) 5 spelling mistakes recorded and these are going to be put into a file ‘test.sp’ using piping function. (pause) Pipe can’t be used since I’m trying to put it into a file so I’ll use the greater-than sign. (pause) Fine. That works.” 4. It was possible to gain some idea of the subjects’ impression of the interface in action, as indicated in the following: questioner: “Did you actually get anything out of the tutorial comments that you got back? Can you make a comparison between the INFO information and the tutorial comments?” subject 1: “Yes. The tutorial – it aims at the problem more closely. The INFO is general information, but the tutorial has found out where you’ve got a problem and actually aims for that – that’s pretty useful.” The results indicate that members of the control group gave many more commands and showed more misconceptions than the members of the experimental group. The mode of activity may be different for learners where help is not easily available. They may adopt a more experimental approach and hence use many more commands to effect the same results. There is an indication that misconceptions are corrected by the intelligent interface as soon as they appear and so are not repeated, as tends to occur in the control group. 4. Recent Developments Research on adaptive (intelligent) interfaces, which respond to meet the needs of the individual user, is still in progress (Jerrams-Smith 1989), and a good case has recently been made for the provision of various kinds of adaptive system (Benyon and Murray 1993; Elsom-Cook 1993). The essential component of such adaptive systems is a user model (also referred to within tutoring systems as a student model) which stores the system’s current beliefs about the attributes of a specific user of the system. In addition, recent research has identified a number of potential problems endemic to hypermedia which could ultimately restrict its usefulness but which might be solved by the provision of adaptivity. For instance,
PROVISION OF ON-LINE HELP
17
Waterworth (1992) suggests that there remain significant unresolved usability issues, while Whalley (1993) discusses a potential problem associated with hypertext when used within an educational context: the fragmented nature of a hypertext document means that the most natural way to study hypertext is by browsing or exploring the various paths of the document. In some instances, however, the browsing activity may be an inappropriate method of learning, for example, when an author is trying to develop a series of ideas within a particular context or framework. Further problems which need to be addressed were identified by Conklin (1987), who describes two main difficulties associated with reading hypertext documents (which apply equally to hypermedia): disorientation and cognitive overhead. Disorientation or ‘getting lost in space’ can occur if the user is uncertain as to his/her location on the hypertext network. Arguably this problem could also exist in traditional linear text documents, however, in this case the reader is limited to searching either earlier or later in the text. Because hypertext offers more dimensions in which the user can move, the likelihood of a user becoming lost is increased, especially in a large network. The problem of cognitive overhead occurs when the user is presented with a large number of choices about which links to follow. These directional decisions are absent in a traditional linear text document (or in a film or TV programme) when the author has already made the choices for ordering of material. With hypertext (and also with hypermedia), the moment a link is encountered, the reader must decide whether or not to follow the link. If many links are encountered the reader may become distracted, leading to what Conklin (1987) refers to as information ‘myopia’. Adaptive systems may provide a solution to such problems and the current interest in intelligent and adaptive multimedia/hypermedia systems is indicated by recent work which involves an investigation into the development of intelligent multimedia systems, such as Jerrams-Smith (1991), Bocker et al. (1990), Hendley et al. (1993), Edwards et al. (1995) and Perez et al. (1995). The author is currently engaged in a research programme developing a number of adaptive hypermedia systems in which a flexible and detailed user model is constructed and applied. These systems include: • HyperLearner (an adaptive multimedia authoring/tutoring system; see 4.1 below for details) • Telecare Companion (a prototype adaptive hypermedia system which is part of the British Telecom ‘Telecare’ programme; see 4.2 below for details) • Adaptive Hypermedia Memory Remediator (for use with persons with various memory disorders)
18
JENNIFER JERRAMS-SMITH
• Adaptive Navigation Guidance for the World Wide Web; see 4.3 below for details. These systems provide vehicles with which to identify the variables responsible for individual user differences and to adapt the system to accommodate user requirements in terms of attributes such as level of domain knowledge, personality, preferences, information processing styles, goals and tasks, roles within an organisation. Work currently in progress indicates that the following variables are important in enhancing the usability of the system. A variable which appears to be fundamental to adaptivity is Field Dependence-Independence (FD-I) (Witkin et al. 1972). FD-I refers to a fundamental individual difference in information processing. The field dependent user is easily influenced by information in his/her environment and tends to incorporate environmental information non-discriminately. In contrast, the field independent user tends to be influenced by internally generated cues and is more discriminating in the use of environmental information. Adaptive systems could take advantage of user differences in cognitive style to present information in a form consistent with the user’s field dependency. Such design features could have value for tutoring applications because fielddependent students’ cognitive style appears to be less suited to hypermedia instructional systems than does that of field-independent students because they are less able to create a framework with which to connect the information (Witkin et al. 1977). A further variable which may be of importance to adaptivity is that of Locus of control (Rotter 1966). Two types of user can be distinguished: internals and externals. Those with an internal locus of control regard outcomes as the result of their own efforts, whereas those with an external locus of control regard outcomes as the result of factors beyond their influence. While a system which removes control from the user may be compatible with an external user, its usability is reduced for internal locus users. Indeed, when internals are required to use restrictive systems they often ignore instructions and some ultimately cease to use the system. The adaptive systems currently in development will identify the user’s locus of control using a brief modified version of the I-E scale (Rotter 1966) and subsequently modify the interactive mode to accommodate the user’s locus of control. A variety of further user differences are being investigated as the basis of adaptivity. A more sophisticated form of adaptivity is also to be investigated: the identification of variables of particular relevance to the user and variables which are irrelevant to the user. In this way the system can focus on the factors which are important in the system’s usability and ignore those which are not.
PROVISION OF ON-LINE HELP
19
4.1. The HyperLearner project Combining hypermedia technology with adaptive tutoring provides a tutoring system which can adapt the sequencing of material and can also adapt the presentation of material to suit the potentially diverse needs and abilities of individual users. In addition, adaptive hypermedia tutoring systems are well suited to an exploratory approach to learning, and therefore encourage the active learning which has long been advocated (Piaget 1973). The author has recently developed HyperLearner, a prototype hypermedia authoring system which has been used to help tutors to build tutorials of course material. The first phase of the project investigated the issues involved in the development of a framework/authoring system for adaptive hypermedia tutoring systems. The long-term aim is the delivery of such a system, incorporating the results of the ongoing investigations. Current trends indicate that in the future, working, and learning, are likely to become increasingly homebased and the provision of systems which enable effective distance learning is therefore a topic of considerable importance. The HyperLearner prototypes learn about the student user and therefore adapt the interaction to suit the individual. The aim of the HyperLearner project is primarily to help the tutor, and thus to help the student to learn about a specific domain, but the prototypes have also been used to help students to learn about some of the fundamental issues connected with intelligent hypermedia tutoring systems (Jerrams-Smith 1995). 4.2. The Telecare Companion The Telecare Companion prototype (John et al. 1998) developed for British Telecom supports the provision of Telecare (telecommunications for disabled and elderly people) by applying adaptive features identified during the HyperLearner project. The Telecare system is designed to form a community network that will provide primary care to the elderly, housebound, disabled or otherwise disadvantaged living in the community. The Telecare Companion investigates the provision of adaptive support for users to access information stored in hypermedia networks, including the Internet, and communicate using a video-phone and e-mail. 4.3. Adaptive Navigation Guidance for the World Wide Web The Computer Aided Internet Navigation project (CAIN) (Lamas et al. 1996) addresses the problem of disorientation, cognitive overhead and information overload. It provides weak hypertext linearisation so that users follow an individual and ordered sequence of selected web pages. It combines techniques of user modelling with data modelling in order to select and order pages.
20
JENNIFER JERRAMS-SMITH
In conclusion, the systems currently under development are providing important insights into the design and production of adaptive hypermedia systems. While a fully adaptive hypermedia system has yet to be developed, the present research programme has already made important inroads into the design and construction of such a system. It should be stressed that the present research is ongoing and the author anticipates that more sophisticated systems will emerge. Note 1 Unix is a trademark of X/Open, Inc.
References Bailey, R. W. (1982). Human Performance Engineering: A Guide for System Designers. Bayman, P. & Mayer, R. E. (1983). A Diagnosis of Beginning Programmers’ Misconceptions of BASIC Programming Statements. Communications of the ACM 26(9): 677–679. Benyon, D. & Murray, D. (1993). Applying User Modelling to Human-Computer Interaction Design. Artificial Intelligence Review 7: 199–225. Bocker, H-D., Hohl, H. & Schwab, T. (1990). HYPADAPTER – Individualizing Hypertext. In Diaper, D. (ed.) Human-Computer Interaction – INTERACT ’90, 931–936. B.V. (North Holland). Brown, J. S. & Burton, P. R. (1978). A Paradigmatic Example of an Artificially Intelligent Instruction System. International Journal of Man-Machine Studies 10: 323–339. Burton, R. R. & Brown, S. (1979). An Investigation of Computer Coaching for Informal Learning Activities. International Journal of Man-Machine Studies 11: 5–24. Carey, T. (1983). User Differences in Interface Design. IEEE Computer 15: 125–129. Carroll, J. M. (1984). Minimalist Training. Datamation 30(18): 125 et seq. Clancey, W. J. (1986). Qualitative Student Models, 86–15. Stanford Knowledge Systems Laboratory: CA. Conklin, J. (1987). Hypertext: An Introduction and Survey. Computer 20(9): 17–41. Damerau, F. J. (1964). A Technique for Computer Detection and Correction of Spelling Errors. Communications of the Association for Computing Machinery 7(3): 254–258. Douglass, R. J. & Hegner, S. J. (1982). An Expert Consultant for the Unix Operating System: Bridging the Gap Between the User and Command Language Semantics. Proceedings of the 4th Conference of Canadian Society for Computational Studies of Intelligence, 119– 127. DuBoulay, B., O’Shea, T. & Monk, T. (1981). The Black Box Inside the Glass Box: Presenting Computing Concepts to Novices. International Journal of Man-Machine Studies 14(3): 237–249. Edwards, M., Powell, H. & Palmer-Brown, D. (1995). A Hypermedia-based Tutoring and Knowledge Engineering System. In Proceedings of Educational Multimedia and Hypermedia 1995. Carlottesville, VA: Association for the Advancement of Computing in Education (AACE). Ehrenreich, S. L. (1981). Query Languages: Design Recommendations Derived from Human Factors Literature. Human Factors 23: 241–247.
PROVISION OF ON-LINE HELP
21
Elsom-Cook, M. (1993). Student Modelling in Intelligent Tutoring Systems. Artificial Intelligence Review 7: 227–240. Finin, T. W. (1983). Providing Help and Advice in Task Oriented Systems. In Proceedings of The Eighth International Joint Conference on Artificial Intelligence, 176–178. Karlsruhe, FRG. Hanson, S. J., Kraut, R. E. & Farber, J. M. (1984). Interface Design and Multivariate Analysis of Unix Command Use. Association for Computing Machinery Transactions on Office Information Systems 2: 42–57. Hayes, P. J., Ball, J. E. & Ready, R. (1981). Breaking the Man-Machine Communication Barrier. Institute of Electrical and Electronic Engineers Computer 14: 19–30. Hayes, P. J. & Szekely, P. A. (1983) Graceful Interaction Through the COUSIN Interface. International Journal of Man-Machine Studies 19(3): 285–305. Hendley, R. J., Whittington, C. D. & Juraschek, N. (1993). Hypermedia Generation from Domain Representation. Computer Education 20(1): 127–132. Hollnagel, E. (1983) What We Do Not Know About Man-Machine Systems. International Journal of Man-Machine Studies 18(2): 135–143. Innocent, P. R. (1982). Towards Self-Adaptive Interface Systems. International Journal of Man-Machine Studies 16(3): 287–299. Irgon, A. E. & Martin, J. C. (1984). CLYDE: A Unix tutor. In Proceedings of the US–Japan Conference on Human Computer Interaction. Hawaii. James, E. B. (1981). The User Interface: How We May Compute. In Coombs, M. J. & Alty, J. L. (eds.) Computing Skills and the User Interface. Jerrams-Smith, J. (1989). An attempt to Incorporate Expertise About Users into an Intelligent Interface for Unix. International Journal of Man-Machine Studies 31: 269–292. Jerrams-Smith, J. (1991). Report on the PIE Project (Personalised Interaction for End-Users). Confidential Report, Philips Research Laboratories (UK). Jerrams-Smith, J. (1995). Combining Multimedia, Hypermedia and Artificial Intelligence to Support four Aspects of Learning. In Proceedings of Educational Multimedia and Hypermedia. Charlottesville, VA, USA: Association for the Advancement of Computing in Education. John, D., Jerrams-Smith, J., Heathcote, D. & Boucouvalas, A. (1998). The Telecare Companion – an Adaptive Interface for Telemedicine. Proceedings of the 1st International Symposium on Communication Systems. Sheffield Hallam University. Johnson, W. L. & Soloway, E. (1984). Intention-Based Diagnosis of Programming Errors. In Proceedings of The National Conference on Artificial Intelligence, 162–168. Austin, TX: AAAI Press. Kamouri, A. L., Kamouri, J. & Smith, K. H. (1986). Training by Exploration: Facilitating Procedural Knowledge Through Analogical Reasoning. International Journal of Man-Machine Studies 24: 171–192. Lang, T., Lang, K. & Auld, R. (1981). A Longitudinal-Study of Computer-User Behavior in a Batch Environment. International Journal of Man-Machine Studies 14(3): 251–268. Lamas, D. R., Jerrams-Smith, J. & Gouveia, F. R. (1996). Computer Aided Information Navigation: Project Description. Proceedings of Webnet ’96. Mishra, P., Trojan, B., Burke, R. & Douglass, S. A. (1984). A Quasi-Natural Language Interface for Unix. In Salvendy, G. (ed.) Human Computer Interaction. Nickerson, R. S. (1981). Why Interactive Computer Systems Are Sometimes Not Used by People Who Might Benefit from Them. International Journal of Man-Machine Studies 15: 469–483.
22
JENNIFER JERRAMS-SMITH
Norman, D. A. (1983). Design Rules Based on Analyses of Human Error. Communications of the Association for Computing Machinery 26: 254–258. Perez, T. A., Lopisteguy, P., Gutierrez, J. & Usandizaga, I. (1995). HyperTutor: From Hypermedia to Intelligent Adaptive Hypermedia. In Proceedings of Educational Multimedia and Hypermedia. Charlottesville, VA: Association for the Advancement of Computing in Education (AACE). Piaget, J. (1973). Memory and Intelligence. Rutledge and Kegan Paul: Cambridge, MA. Ramsay, H. R. & Attwood, M. E. (1979). SAI-79-111-DEN (NTIS:ADA 075 679), Human Factors in Computer Systems: A Review of the Literature. Science Applications, Inc.: Englewood. Relles, N. & Price, L. A. (1981). A User Interface for Online Assistance. In Proceedings of The Fifth International Conference on Software Engineering. San Diego, California: IEEE Computer Society. Rotter, J. B. (1966). Generalised Expectancies for Internal versus External Control of Reinforcement. Psychological Monographs 80(609). Scapin, D. L. (1981). Computer Commands in Restricted Natural Language: Some Aspects of Memory and Experience. Human Factors 23(3): 365–375. Self, J. A. (1974). Student Models in Computer Aided Instruction. International Journal of Man-Machine Studies 6: 261–176. Shneiderman, B. (1979). Human Factors Experiments in Designing Interactive Systems. Institute of Electrical and Electronics Engineers Computer 12: 9–19. Shortliffe, E. H. (1976). Computer-Based Medical Consultations: MYCIN. Elsevier: New York, NY. Sleeman, D. H. (1983). Intelligent Tutoring Systems and Student Modelling. Presented at AISB Conference on AI and Education. Exeter, UK. Sleeman, D. H. & Hendley, R. J. (1979). ACE: A System Which Analyses Complex Explanations. International Journal of Man-Machine Studies 11. Sleeman D. H. & Smith, M. J. (1981). Modelling Students’ Problem Solving. Artificial Intelligence 16: 171–188. Tagg, S. K. (1981). The User Interface of the Data-Analysis Package: Some Lines of Development. International Journal of Man-Machine Studies 14(3): 297–316. Waterworth, J. A. (1992). Multimedia Interaction with Computers. Ellis Horwood. Whalley, P. (1993). An Alternative Rhetoric for Hypertext. In McKnight, C., Dillon, A. & Richardson, J. (eds.) Hypertext: A Psychological Perspective. Ellis Horwood. Wilensky, R., Arens, Y. & Chin, D. (1984). Talking to Unix in English: an Overview of UC. Communications of the ACM 27(6): 574–593. Winston, P. H. & Horn, B. K. P. (1981). LISP. Addison-Wesley: Reading, MA. Witkin, H. A., Lewis, H. B., Hertzman, M., Machover, K., Meissner, P. B. & Wapner, S. (1972). Personality Through Perception: An Experimental and Clinical Study. Greenwood Press. Witkin, H. A., Moore, C. A., Goodenough, D. R. & Cox, P. W. (1977). Field-Dependent and Field-Independent Cognitive Styles and Their Educational Implications. Review of Educational Research 47: 1–64.
Artificial Intelligence Review 14: 23–42, 2000. Stephen J. Hegner et al. (eds.), Intelligent help systems for UNIX. © 2000 Kluwer Academic Publishers. Printed in the Netherlands.
23
Virtues and Problems of an Active Help System for UNIX MARIA VIRVOU1 , JOHN JONES2 and MARK MILLINGTON3 1 Department of Computer Science, University of Piraeus, 80 Karaoli & Dimitriou St., Piraeus 18534, Greece. E-mail:
[email protected]; 2 Department of Computer Science, Hull HU6 7RX, UK; 3 Interactive Business Systems, Inc., 2625 Butterfield Road, Oak Brook,
Illinois 60521, USA. E-mail:
[email protected]
Abstract. An empirical study undertaken on a cross-section of UNIX users at an academic site reveals a role for an active form of help system, rather than the more usual passive kind. Sample scripts supporting this view are presented and the kind of aid required for these examples is discussed. It is then proposed that to provide such aid requires the construction and maintenance of an individual model of each user. Keywords: advice generation, intelligent help systems, user’s errors, user interfaces, user modelling, UNIX commands
1. Introduction The aim of this paper is to motivate, through examples and discussion, work on active aid systems for the UNIX file store manipulation domain. Most current help systems are passive, requiring a user to explicitly ask for help on a particular topic. Examples of this approach include the on-line UNIX manual and intelligent help systems such as UC (Wilensky et al. 1986). In contrast (in common with Breuker 1988; Jerrams-Smith 1985; Kemke 1986; Matthews et al. 2000) we will propose a role for an (automatic) active aid system to act as an “over the shoulder” adviser. Such a system is very difficult to construct and we discuss a number of the problems and issues that arise if this route is taken. The remainder of this section introduces a number of challenging examples used to illustrate points made in later sections. Section 2 discusses some advantages of an active help system, while Section 3 outlines the modelling problems such a system may face. In Section 4, we turn to the issues to be dealt with as a system prepares to interrupt a user. In Section 5, we summarize our approach in building a system meant to tackle some of the problems that have been highlighted in this paper, and we also discuss the results of the automatic aid, with respect to the examples introduced in the present section. In Section 6, we give an account of recent developments in this research.
24
MARIA VIRVOU ET AL.
1.1. Motivating examples This paper arises from an empirical study undertaken on a cross-section of UNIX users at an academic site. The particular version of UNIX that was used was Berkley 4.2 on a Vax; in fact, this version makes a lot of difference to many of the examples given below, for other implementations might behave rather differently. Volunteer subjects included faculty members, research staff and postgraduate students. Each subject was automatically monitored by the shell with which s/he was interacting; in an unobtrusive manner, as users went about their normal activities on the system, each command issued was recorded in a log-file associated with that user. However, due to the volume of feedback from UNIX, it was impossible to record the replies of UNIX to the commands issued. Thus, for example, error messages and the output from commands like ls were missing from the log and had to be reconstructed, as seen below. We reconstruct four partial sessions, two from each of two users, to illustrate many of the points we wish to make about the desirability and difficulties of an active help system (hereafter called an “aid system”). The summaries given with these reconstructions are intended to enable a reader to rapidly absorb the content of what happened. For the same reason, we have also made comments next to some commands. These comments start with a hash symbol. These summaries and comments did not arise from interviews with the subjects, and merely present a single interpretation of what occurred. Even so, it is quite remarkable what a wealth of information about each user lies in these examples, waiting to be extracted by an (intelligent!) aid system. The poplog example Anna has a sub-directory tf of her home directory which she wishes to move into a new sub-directory, to be called poplog. This can be achieved by mkdir poplog; mv tf poplog. However, instead of using mv, she tries first to copy tf into poplog (commands 2–5), then checks that the copy worked (commands 6–8), and finally removes the original (commands 9–17). Unfortunately, in this implementation of UNIX, command 5 will produce a data file which describes the directory entries for tf; a very similar effect can be gained by: % cat tf > poplog/tf/tf This form of cp is intended for use by experienced users engaged in system administration functions, who are a class of users very different from Anna.
VIRTUES AND PROBLEMS OF AN ACTIVE HELP SYSTEM FOR UNIX
25
Anna does try (command 7) to check that command 5 has created a directory, suggesting she has some inkling of the limitations of cp, but the results of this test are consequent only on (commands 1–3); the tf listed here is the tf created in (command 3), not the tf copied in (command 5). Alas, she is satisfied and goes on to remove the (still unique) original. 1 2 3 4 5
% % % % %
mkdir poplog cd poplog mkdir tf cd cp tf poplog/tf
6 % cd poplog 7 % ls -al total 3 drwxr-xr-x 3 anna drwxr-xr-x 3 anna drwxr-xr-x 2 anna 8 % cd 9 % rm tf
10 11 12
13 14 15 16 17
rm: tf is a directory % rm -i tf rm: tf is a directory % man rm % rmdir tf rmdir: tf: Directory not empty % cd tf % rm house1 % rm house2 % cd % rmdir tf
#Makes a new directory ‘tf’. #This command does not do anything #sensible when applied to directories. #Unfortunately there is no error message #to alert her. #Has a new directory been created? 512 May 16 18:44 . 512 May 16 18:44 .. 512 May 16 18:44 tf #Tries to remove the old ‘tf’; but this #command does not apply to directories. #Gets an error message this time. #Tries again without success. #Turns to the manual.
#‘house1’ is lost. #‘house2’ is lost too. #The old ‘tf’ eventually removed.
26
MARIA VIRVOU ET AL.
The ht example Graham wishes to collect all his files from his current working-directory into a new sub-directory called ht. To achieve this he creates the new directory (command 1), copies everything (including ht!) into ht (command 2), enters ht to delete the data-file copy of ht itself (commands 4–8), and then neatly removes everything (except ht) from the original directory (command 10). As in the poplog example, a directory has been copied as a data-file, but in this case, the user is seen to be well aware of the behaviour of cp and rm when applied to directories rather than files. 1 % mkdir ht 2 % cp * ht 3 % ls intro ht notes 4 % cd ht 5 % ls intro ht notes 6 % rm ht 7 % cd . 8 % cd .. 9 % ls intro ht notes 10 % rm * rm: ht is a directory 11 % ls ht
#Creates a directory ‘ht’. #The newly created ‘ht’ is also matched by ‘*’. #The current working directory contains two more #files apart from the new directory ‘ht’. #Gets into ‘ht’. #The directory ‘ht’ contains a copy of the two #files as well as a data-file copy of itself. #Removes the useless data-file copy of ‘ht’. #A typing mistake. #Rectification of the previous typo.
#‘rm’ removes the files ‘intro’ and ‘notes’ #which have been copied into ‘ht’. #‘rm’ does not apply to directories. #‘ht’ is not removed.
The popcode example Graham has two directories, ccode and popcode, which he believes to contain only unwanted files. He attempts to delete each directory by entering it, removing all of its files, leaving it, and then removing it. However, in popcode he discovers (we deduce from commands 6–9) that he has a sub-directory to deal with. We note how clear it is that failure (of command 6) was not intended in this example as it was in (command 10) of the ht example.
VIRTUES AND PROBLEMS OF AN ACTIVE HELP SYSTEM FOR UNIX
1 2 3 4 5
% cd ccode % rm * % cd .. % cd popcode % ls lib zip.p 6 % rm * rm: lib is a directory 7 % cd .. 8 % cd popcode
9 % cd lib 10 % ls arcs.p nodes.p 11 % rm * 12 % cd .. 13 % rm lib rm: lib is a directory 14 % rmdir lib 15 % cd .. 16 % ls ccode popcode 17 % rmdir ccode popcode
27
#Gets into directory ‘ccode’. #Removes all the files. #Gets into the parent directory. #Gets into directory ‘popcode’.
#Removes all the files. #This time there was a directory in there #which was not removed. #Gets into the parent directory. #Gets back to the ‘popcode’ directory, #probably because (command 7) was issued #very quickly after (command 6) and before #seeing the resulting message. #Has a look at the contents of the unremoved #directory. #Removes all the files in it. #Tries to remove the directory; but command #does not apply to directories. #The right command to remove the directory.
#Removes the two empty directories.
The perquish example Anna wishes to collect two existing files, fred and fred1, from her home directory, together in a new directory called perqish, renaming them to iread.pas and drconsult.pas, respectively, in the process. However, at command 3 she mis-types perqish as perquish, an error which lies latent until command 6, and uncorrected until command 19! A version of this example is tackled in a companion paper in this volume (Jones et al. 2000).
28 1 % mkdir perqish 2 % cp fred perqish 3 % cp fred1 perquish
4 % cd perqish 5 % mv fred iread.pas 6 % mv fred1 drconsult.pas mv: fred1: No such file or directory 7 % ls iread.pas 8 % cd 9 % cp fred1 perqish
MARIA VIRVOU ET AL.
#Creates a new directory ‘perqish’. #Copies ‘fred’ into the new directory #‘perqish’. #Creates a new file ‘perquish’ as a copy of #the file ‘fred1’; or was ‘perquish’ meant to #be ‘perquish’ as at the previous command?
#She expects fred1 to be in the recently #created directory ‘perqish’. #Gets an error message showing that ‘fred1’ #is not actually there. #Confirms ‘fred1’ is not in the recently #created directory ‘perqish’. #Repeats command 2 without the typo this #time.
10 % cd perqish 11 % mv fred1 drconsult #Back to the command that made her #discover the error. But (command 11) is not #identical to (command 6); an oversight? 12 % cd 13 % rm fred #Another typo; ‘fred’ should read ‘fred1’. rm: fred: No such file or directory 14 % rf fred1 #A typo again! ‘rf’ should read ‘rm’ rf: Command not found 15 % rm fred1 #Here is the correct command. 16 % ls #The accidentally created file ‘perquish’ is perqish perquish #still there. 17 % rmdir perquish #Tries to remove it unsuccessfully; rmdir: perquish: Not a directory
VIRTUES AND PROBLEMS OF AN ACTIVE HELP SYSTEM FOR UNIX
18 % rm perqish rm: perqish is a directory 19 % rm perquish 20 % ls perqish 21 % cd perqish 22 % ls drconsult iread.pas 23 % mv drconsult drconsult.pas
29
#Another attempt fails because of another #typo! #She removes the unwanted file. #Confirms the removal.
#Command 11 was not meant to be different #from (command 6) after all.
2. Some Points in Favour of Active Help There are several questions concerning automatic aid systems that seek an answer in users’ actual command sequences. First of all, is an aid system needed at all? In what way could an aid system offer help? Are there any cases where an active help system could be more suitable to help the user than a passive one? Some answers to these questions come straight out of real-life examples like those four illustrated in the previous section. 2.1. Not realising help is needed There are many cases where users type other than what they mean, without any error message being produced to alert them. This does not mean that UNIX feedback is always to blame; in fact, it is not simple to spot those errors that UNIX would not have complained about without having reasoned about the user’s actions and beliefs. Rather, this kind of reasoning is beyond the scope of UNIX or any other existing operating system, and hence, a help system would be needed instead. However, even a passive help system would not have been of much use to a user who had not been aware of a problematic situation, because this user would not have turned to it; how could one expect users to ask for help when they do not even know that they need it? This is when an aid system would be useful, since an aid system is constantly supposed to watch and reason about the user’s actions and make its own decisions, whether or not the user needs help. This can be one of the most important advantages of an aid system.
30
MARIA VIRVOU ET AL.
The sample scripts clearly illustrate this situation, where users make errors without being aware of them. This was clearly the case in the poplog example, as well as the perquish example. The consequences of such errors vary, depending on how soon the user realises there has been an error, and how difficult it is to recover from it, if this is at all possible. In the remainder of this subsection, we are going to discuss the consequences of this kind of error, in the way they occurred in the poplog and the perquish examples. We are also going to point out the fact that inefficient usage of commands may well fall into the same category of problem, where users may not be aware that they possibly need help. The popcode example will be used to illustrate this point. Catastrophic error in the poplog example A simple copy and remove plan in the poplog example ended up in the single catastrophic action remove since the copy part of the plan failed without the user having realised it. More specifically, Anna’s misconception about the way directories are copied to other directories leads her to the erroneous command 5. The lack of an error message following this command results in command 9, where she attempts to remove a directory, without it having been copied first, contrary to what she believes. Initially, she only intended to move the directory elsewhere in the file store. This is a very tricky case indeed, where the user continues issuing commands as though a previous goal has already been accomplished. She did not seek help (in the manual) before typing command 5, which was the cause of the trouble. Perhaps she thought she knew how to handle this. After which, she did not even realize her mistake, and therefore, would not have asked for help even if this had been available. This is a typical situation in which a spontaneous intervention of an aid system can save trouble. In this case, it could save the user from a catastrophic error. Considerable effort wasted in the perquish example The name of a directory called perqish is mistyped as perquish in command 3. As a result, a new file named after the mistyped name is created. At this point the user is not aware of her error. However, after command 6, she realises that there is a problem, unlike the previous example where she never even realised that something went wrong. Still, the recovery from this error proves quite expensive in terms of the number of commands. As a matter of fact, she issued 23 commands to complete her initial goal, with 10 of them (commands 7, 8, 9, 10, 11, 16, 17, 18, 19 and 23) typed in the context of recovering from the typo and consequences of command 3. In this case, almost half of the effort was wasted due to a small typing error.
VIRTUES AND PROBLEMS OF AN ACTIVE HELP SYSTEM FOR UNIX
31
Evidently, it would not have been so much if she had not made other errors at commands 11, 17 and 18. However, it may have been worse. While trying to recover from her mistake, she gets involved in new ones and this goes on for some time. This is a case where the lack of an error message from UNIX is absolutely justified, simply because the error occurs in the name of a directory or file. When new files are created, users are free to name files and directories as they please. Hence, UNIX cannot possibly complain when the user mistypes a name in a command to create a new file or directory. This kind of error could only be spotted if a system kept track of the user’s intentions and beliefs, which means that an explicit user model would be necessary. An aid system, which is supposed to provide this kind of reasoning could help the user recover quickly from an error like this. Another interesting point about this example is that this sort of typing mistake could have been made by anyone, novice or expert, suggesting that all types of users could benefit somehow from an aid system. Optimisation in the poplog and ht example Graham’s goal in the popcode example could have been obtained by only 2 commands (ls -R; rm -r ccode popcode), instead of the 17 he issued! Similarly, the effect of the first 10 commands he issued in the ht example could have been obtained by using only 2 commands (mkdir ht; mv * ht). Would he be interested in seeing these very short command sequences, so that he could use them at some later time? This is a case where the user is probably not aware of some system commands that would achieve a goal via a simpler plan than the one formed in the user’s mind. Actually, many users probably feel safer using commands they are already familiar with, and do not bother to find out more about the operating system that they are using. Instead, they try to fit their needs to their existing knowledge. An aid system could suggest an optimisation of the user’s very own actions. In this aspect, the aid system could act as a tutoring system. Again, the individuality of the user would play a significant role in deciding what help the aid system could provide. 2.2. Not knowing how to ask for help Users may not know how to ask for help. It seems that sometimes the existing manual help is not sufficient for a user who turns to it for aid. The wealth of information may confuse the non-expert, who might pick up the wrong command for their case. There are times when users need advice for their individual case, and instead, all they can get is general information. Quite
32
MARIA VIRVOU ET AL.
often, they do not even know exactly what their problem is, and therefore, they cannot form the right questions. Would they not be interested in getting an expert’s advice, tailored to the individual circumstances from which their problem occurred? In the following two subsections we first illustrate this case where the manual proves inadequate, and then we describe the case where the user is looking for the wrong kind of information. Both cases occur in the poplog example. Failure following the man command in the poplog example Anna finds out about rmdir from the rm manual entry at command 11. However, command 12, coming right after the “man rm” command fails to remove her directory as Anna then wanted. In fact, she probably wanted rm -r, but did not manage to find it in the manual. This is only an example of a case where the user finds it difficult to retrieve information from the manual. Anna was quite lucky to encounter only one failure before she used a plan which worked. There are even worse cases where some users do not even get to find the right keyword for the command that they seek. Getting help on the wrong issue in the poplog example Anna believes that she has to find out how to remove the directory tf at command 11 of the poplog example. But in fact, she would not want to remove it if she knew that it had not actually been copied first. In this case, the help she got from the manual did her no good. Ironically enough, no efficient reference facility would do any better if it did not have a user model. What Anna actually needed was a help system to tell her what was really going on, instead of just giving a reply to her question. 2.3. Trying vs. looking up Although not always recommended, users may find it quicker to actually try commands out rather than looking them up. For example, Anna probably had some idea of the existence of the option -r in the poplog example. After the failure of command 9, because tf was a directory, she decided to take another guess before she issued the man command. However, she could not remember the exact name of the option she wanted, and in this case, typed the completely irrelevant option -i. Perhaps this kind of user would be happier and safer if an aid system could guarantee interruptions at dangerous junctures; safety encourages experiment and experiment encourages learning. Bad typists, especially, would probably be happier to be interrupted, even if a facility for requesting help in (written) English were provided, because the latter would take some time to type.
VIRTUES AND PROBLEMS OF AN ACTIVE HELP SYSTEM FOR UNIX
33
2.4. More human-like interaction Frustration can result for some users when the operating system is not able to spot even the most obvious (to a human) typing error. They somehow expect the computer to have some features of humans, and may get upset when commands fail because of some small typing error. For example, a human expert would easily understand what Anna meant by command 14 in the perquish example. The whole context of the command sequence, and the similarity of the wrong command to the correct one, give sufficient clues to a human. However, intelligence is required for such an error to be spotted by the computer, especially when the typo can change the meaning of a command. This was the case in command 2 of the perquish example. A monitoring help system could make the interaction more humanlike by recognizing these errors and interrupting discreetly like a human listener.
3. Modelling Requirements In this section we address the issue of how accurately an aid system must model the user it serves. Firstly, we consider how clearly it must reason about what actually happened, using a general understanding of the interactive nature of command-driven systems like UNIX. Secondly, it is shown that in order to reason about what should have happened the aid system must maintain a model of each individual user. 3.1. What happened? For an automatic system to provide acceptable on-line aid it must show a good understanding of the user-UNIX interaction. Whatever their differences, all users interact with a shell in basically the same way; the user issues a command and UNIX may issue a reply. This notion of interaction is important; it is not enough for an aid system to ignore the flow of information to a user and concentrate merely on the file store transformation. For example, we can strip all occurrences of ls from a command sequence leaving the effect (on the machine) unchanged, but this would mean that the aid system would not model the information that the user has seen. This might lead the aid system to interventions proposing rather suspect improvements. In the popcode example, the user learned that there was a sub-directory of popcode, only after attempting to remove it. After investigation it was eventually removed, but to suggest an improved sequence “rm -r popcode” with the same effect does not capture the true sequence of events.
34
MARIA VIRVOU ET AL.
Similarly, a user’s current working-directory is an important object of manipulation, facilitating the simplified expression of (currently) non-local pathnames. If an aid system sees a command sequence cd a; cd b; cd c, should it propose an “equivalent” sequence cd a/b/c as being better? If it works it definitely is better, but the problem arises if the user is imperfect in some way (e.g. a bad typist, or forgetful about his file store environment), and then an error in the former is easier to recover from; if it does not work you will know why and be in the right place to do something about it. All users distinguish these sequences, and therefore, so must an aid system. Thus, realistic aid systems will have to employ a rich model of the interaction between a user and UNIX. To do this, it must almost certainly make some generic assumptions about users; for example, users are (reasonably) attentive, consistent, sometimes forgetful, fat-fingered, cautious, etc. Modelling UNIX is the least of our problems. 3.2. What was intended? Through an understanding of what actually happened we may hope to reason about what was intended, even though the two may be very different. If the two are significantly different, the user ought to be informed. One approach to this might be to wait for an error message to arise, and then attempt a diagnosis for the user. However, as we will see, this cannot work; some users make productive use of failure conditions, and some errors do not provoke error messages. Providing effective aid must involve modelling individual characteristics of each user, for it is easily seen that not doing so, quickly leads to confusion. What aspects of a user must we model? • We at least have to model the user’s plan; suggesting “rm -r *” as an improvement for command 10 of the ht example just to eradicate the error message would be ridiculous: the user is relying on the failure and does not want the extra effect of this proposal. • The user’s overall plan should have priority over a user’s subplan; Anna would not thank us for helping her remove tf at command 9 of the poplog example, although this is obviously what she wants there; the overall plan is very different and must be taken into account. • The individual user’s command knowledge is important too; interrupting Graham at command 13 of the popcode example would be very different from interrupting Anna at command 9 of the poplog example (in the hypothetical case where the overall plan would not conflict with her wanting to remove tf). They both attempt to remove a directory by employing the command rm without any flag. However, Graham has
VIRTUES AND PROBLEMS OF AN ACTIVE HELP SYSTEM FOR UNIX
35
generally shown a better knowledge and understanding of command rm than Anna has in the previous examples. Therefore, Graham’s mistake may have been an accidental slip, whereas Anna’s mistake may have been due to a deeper confusion of commands. An intervention at command 9 of the poplog example (or, perhaps, command 10) might be acceptable: she plainly wants to get rid of tf but does not know how. • Confusion may also arise over the environment in which a command is executed. At command 6 of the popcode example, “rm lib” fails because lib is a directory. Is this because he does not understand rm removes only files, or because he forgot lib was a directory? Knowing the user we would conclude the latter; in fact, later on in the ht example at command 10 the same user is making productive use of the same failure condition. However, if he had employed ls -F as command 5, we might be less sure. • General and individual users’ usual imperfections have to be modelled too; What would the user’s intended typing, lexical, or syntactic structure be, if not that which was actually input? Answers to such questions should be somewhere in the model of users’ usual errors. For example, the fact that Anna had mistyped perqish to perquish could have been suspected right away, only because the typing, lexical and syntactic structure of the command containing it was very similar to the previous one, and people tend to make this kind of typing error. However, as we have seen, different aspects are very heavily interrelated, and an existing model provides additional strong constraint on the sensible interpretations.
4. Interventions Given that there is a useful point that can be made to the user, the question remains as to how, when, and on what basis to intervene. How an intervention is made ought to depend on whether it proposes an optimisation or a correction; attention to a correction should be mandatory whereas offers of optimisation may be ignored. In this way corrections may be given as interruptions, like error messages, comprising a piece of English text to describe the problem and a proposed correction. Optimisations should not interrupt, but might instead alter the prompt to, say, a question-mark. If the user immediately hits return then the optimisation is given; otherwise, the user is free to issue a normal command. There is little point in proposing a correction or optimisation that is beyond the current understanding of the user; if too many new ideas are introduced, it will confuse the user and be rejected. Here again, is a role for an individual user-model: to tailor advice to suit the user.
36
MARIA VIRVOU ET AL.
When to intervene, again, depends on the nature of the proposal. Optimisations can be given as soon as they are discovered, since they rely only on what actually occurred. Corrections deal with what was intended, when it differs from what occurred; in such cases there seem to be three points at which the aid system might intervene: − When a correction first appears to be warranted; for example, command 3 in the perquish example. This may be too soon if there is not enough evidence for suspecting the command, since it seems likely that almost any command might be intended as something else. However, if there has been sufficient evidence that a problem may have occurred, then it would probably be better for the system to intervene sooner than later, to prevent further complications of this problem. − When only a correction fits all the facts; i.e., we have no user-model in which the user is not under a misconception. For example, we might expect command 6 in the perquish example to be of this nature. − When the user can no longer recover from the error if the current command is executed. There is no such point in the perquish example; in fact, she does make the recovery herself, but such a point occurs in the poplog example at command 14. The basis for each intervention can, of course, be derived only from the events preceding the intervention, and it is not envisaged that we can hope to construct a dialogue between the aid system and the user; actions speak louder than words. However, the user model may also comprise a large proportion of the basis; Graham, copying a directory, is a very different matter from Anna, doing the same. Some interventions produced by a prototype active help system for the sample scripts are presented in the following section.
5. Summary An active help system could be of much use to many users, and not only novices. This is the conclusion drawn from the analysis of the sample scripts. It could save the considerable amount of time and energy wasted in recoveries from errors. It could also act as a personal tutor, allowing the user to escape needless inefficiencies. The individuality of the user is of vital importance to the help an aid system should provide. This makes the error diagnosis more complex since both general user models and individual ones should be taken into account for a clear interpretation of the interaction leading to a good plan recognition. The latter is essential for a reasonably reliable error diagnosis. The user
VIRTUES AND PROBLEMS OF AN ACTIVE HELP SYSTEM FOR UNIX
37
modelling part should involve assumptions that the user could be imperfect in several aspects, such as having misconceptions about commands, making typing errors etc. Interventions should be kept to a minimum. Only when it is very obvious that the user needs help, should this be provided. The only aim is to help the user, and make him/her more efficient. By no means should the opposite happen in any sense.
6. Recent Developments In this section we describe briefly a prototype active help system called RESCUER (which stands for Reasoning System about Commands of UNIX using Evidence Reasonably) (Virvou 1992; 1998). RESCUER has been constructed to address users’ problems like those described in earlier sections. It is not within the scope of this paper to detail the modelling and reasoning issues of RESCUER. However, we will demonstrate the interventions produced by RESCUER when it was given, as input, the sample scripts presented in this paper. A companion paper in this volume (Jones et al. 2000) describes a prototype system addressing some of the modelling issues raised here, but none of the intervention issues. 6.1. RESCUER’s overall approach RESCUER is a prototype active help system that monitors users interacting with UNIX and offers spontaneous help when it judges that users need it. RESCUER’s design addresses three main problems that an active help system has to solve: 1. Recognition of a problematic situation. 2. Diagnosis of the cause of the problem, if there has been one. 3. Decision regarding the kind of advice and generation of response. For the recognition of a problematic situation, RESCUER uses certain peculiarity criteria in order to evaluate every command typed. These criteria include the questions whether the command was acceptable to UNIX or not, whether it was typical of its class and so on. RESCUER uses these criteria as unconfirmed symptoms of a possible problem in a command type. The existence of such symptoms results in the further examination of the command typed. As a result of this evaluation, RESCUER attaches one of three labels to the command typed, namely “expected”, “neutral”, “suspect”. The label
38
MARIA VIRVOU ET AL.
“suspect” means that RESCUER will have to generate alternative interpretations in its attempt to find one that fits the context better. The label “expected” means that the command’s effects to the file store imply some continuation of a plan from previous commands; therefore RESCUER judges that the command fits the context very well. Finally, “neutral” means that RESCUER does not get alerted by this command because it considers it unharmful (e.g. “ls”), although it is not directly involved with the effects of previous commands. The diagnosis of a problem is done by the user modelling component of RESCUER. If the user modeller can find an alternative interpretation about the user’s command, such that it is similar to what was typed, but is better than that in terms of the evaluation criteria, then RESCUER may have to adopt this alternative interpretation. The process of generating an alternative interpretation contains the information needed for the construction of an explanation of what happened. The generation of hypotheses about possible user’s misconceptions is based on a cognitive theory, called Human Plausible Reasoning theory (Collins and Michalski 1989), which was originally constructed to formalise the reasoning that people use in order to make plausible guesses about questions for which they do not have full information. Here we exploit the fact that plausible guesses can be turned to plausible human errors. Plan recognition is achieved by considering the UNIX file store as an entity that has certain properties, one of which is called “instability”. The existence of instabilities imply a possible future transition of the current state of the file store to another state, which is expected to follow at some point (e.g. from a file store state that contains an empty directory to the file store state that this directory is filled with some content). A file store is considered to be absolutely stable if it does not contain: 1. empty directories 2. directories with only one child 3. duplicate files The reason that empty directories are considered to be adding some instability to the file store is that empty directories do not seem to have any purpose if they have no contents. One would expect them to either have contents or be removed. Similarly, if a directory has only one child, it is considered to be borderline as to how useful it is to keep a directory with only one file. One would expect it to either have more contents, or the content file to be moved one level up and the directory to be removed. Finally, duplicate files also imply some transition from a file store state that contains identical duplicates to a file store state where the original copies will have been removed, or the replicated files will have been changed.
VIRTUES AND PROBLEMS OF AN ACTIVE HELP SYSTEM FOR UNIX
39
Instabilities are only used to indicate that a user may have started some plan, but do not restrict the user’s actions at all. An unstable file store does not mean that the user has made an error, but that perhaps, s/he is in a middle of a plan. A “stable file store” is considered to be every user’s final goal, denoting the completion of some plan. In this way, RESCUER manages to evaluate users’ typed actions in terms of their effects to the file store. RESCUER’s expectations, in terms of effects to the file store, are used to guide the search for possible corrections of commands. A command is “expected”, in terms of instabilities, if it adds and deletes instabilities at the same time (which means that there is some kind of continuity in the user’s actions), or if it deletes instabilities without adding any new ones. The generation of a response, if there is one, is done on the principle of preventing users from possible catastrophic errors, giving explanations that address the user’s misconceptions and suggesting alternative correct plans. However, RESCUER mainly concentrates on errors that would be “plausible” for a human observer to identify. 6.2. RESCUER’s interventions for the sample scripts In this section we demonstrate RESCUER’s responses when it was given, as input, the command sequences which were recorded in the sample scripts. The poplog example 1. % mkdir poplog 2. % cd poplog 3. % mkdir tf 4. % cd 5. % cp tf poplog|tf RESCUER: Did you mean to type: % cp -r tf poplog/tf or % mv tf poplog/tf? RESCUER has considered the fifth command as “suspect” because it is not typical of “cp” command with respect to its purpose. Since the command has been evaluated as “suspect”, RESCUER generates the two alternatives which are similar to the command typed: 1. % cp -r tf poplog/tf 2. % mv tf poplog/tf
40
MARIA VIRVOU ET AL.
They are also found to be “expected”, because they assign meaningful content to the empty directory “poplog/tf”, and are considered better than the original command because they are more typical of their class. The ht example One major point raised from this example was that the user made a constructive use of the UNIX failure to remove directories by the command “rm *” (command 10). RESCUER was smart enough to understand that command 10 was useful to the user. This command was used to remove all the original files that the user had previously copied to the directory “ht”. Command 10 had produced an error message because “*” did not apply to directories, and therefore, “ht” itself was not removed, which was precisely what the user wanted. Although command 10 was considered by RESCUER as “suspect” because it produced an error message, it was also assigned the label “expected” because of the instabilities that it removed from the file store. Therefore, RESCUER did not intervene in this case. The perquish example RESCUER intervenes at the third command as follows: 1. % mkdir perqish 2. % cp fred perqish 3. % cp fred1 perquish RESCUER: You may have made a typing mistake. Did you mean to type: ‘‘cp fred1 perqish’’ instead of ‘‘cp fred1 perquish’’? Y/N. The reasons why RESCUER has generated this response are the following: First RESCUER considers command 3 as “suspect” because it introduces new instabilities without deleting any of the previous instabilities. At this point, RESCUER starts generating similar alternatives to the command typed. In this case, it generates four alternatives: 1. 2. 3. 4.
% % % %
cp cp cp mv
fred perquish fred1 perqish -r fred1 perquish fred1 perquish
The first two are generated by changing the arguments of the action typed to other arguments (one at a time), which in this case are lexically similar.
VIRTUES AND PROBLEMS OF AN ACTIVE HELP SYSTEM FOR UNIX
41
The last two are generated by changing the command name and leaving the arguments as they are. The most similar commands to “cp” which would work for the preconditions of the semantics of this action have been found to be the commands “cp -r” and “mv”. Next, RESCUER evaluates the hypotheses in the same way as it evaluated the action typed. Two of them are found “suspect”, one of them is found “neutral”, and one is found “expected”. 1. % cp fred perquish is found “suspect” because it has a similar effect to the file store as the command originally issued. 2. % cp fred1 perqish is found “expected” because it deletes the instability of “perqish” having only one child as a content. 3. % cp -r fred1 perquish is found “suspect” because it has exactly the same effect on the file store as the command originally issued. 4. % mv fred1 perquish is found “neutral” because it neither deletes nor adds any instability (therefore suggests that it is a plan of its own). RESCUER picks the “expected” command, % cp fred1 perqish, to show the user, because at this point, it has “enough” evidence that the user may have made a typing error. There are two reasons completely independent from each other that suggest that the user may have made an error: 1. There is a command other than the command typed that is very similar to the command typed, according to the knowledge representation of RESCUER. 2. This other command is different from the command typed in the effects that it has in the file store. The “other command” is actually better than the command typed with respect to their difference. This example demonstrates the successful automatic generation of a plausible correction. RESCUER favours an earlier intervention, rather than a later one. Therefore, it suggests a correction if there is more than one reason for believing that there may have been a mistake. Naturally, RESCUER may be unnecessarily invoked (which is not the case in this example) but even so, the user can always ignore RESCUER’s suggestion.
Acknowledgements We thank Peter Norvig, Wolfgang Wahlster and Robert Wilensky for having made several helpful comments on an earlier version of this paper. We are also grateful to Stephen Hegner and Paul Mc Kevitt for their helpful comments on the current version.
42
MARIA VIRVOU ET AL.
References Breuker, J. (1988). Coaching in Help Systems. In Self, J. (ed.) Artificial Intelligence and Human Learning, 310–337. London, UK: Chapman and Hall. Collins, A. & Michalski R. (1989). The Logic of Plausible Reasoning: A Core Theory. Cognitive Science 13: 1–49. Jerrams-Smith, J. (1985). SUSI-A Smart User Interface. In Johnson, P. & Cook, S. (ed.) Proceedings of the Conference of the British Computer Society Human Computer Interaction Specialist Group. Cambridge University Press. Jones, J. G., Millington, M. & Virvou, M. (2000). An Assumption-based Truth Maintenance System in Active Aid for UNIX Users. In Hegner, S., Mc Kevitt, P., Norvig, P. & Wilensky, R. (eds.) Intelligent Help Systems for UNIX. Dordrecht, The Netherlands: Kluwer Academic Publishers (this volume). Kemke, C. (1986). The SINIX Consultant: Requirements, Design, and Implementation of an Intelligent Help System for a UNIX Derivative, Bericht Nr. 11, FR. 10.2 Informatik IV, University of Saarland, Saarbrucken, FRG. Matthews, M., Pharr, W., Biswas, G. & Neelakandan, H. (2000). USCSH: An Active Intelligent Assistance System. In Hegner, S., Mc Kevitt, P., Norvig, P. & Wilensky, R. (eds.) Intelligent Help Systems for UNIX. Dordrecht, The Netherlands: Kluwer Academic Publishers (this volume). Virvou, M. (1992). User Modelling Using a Human Plausible Reasoning Theory. Ph.D. thesis, CSRP 251, School of Cognitive and Computing Sciences, University of Sussex, Brighton BN19QH, UK. Virvou, M. (1998). RESCUER: Intelligent Help for Plausible User Errors. In Proceedings of ED-MEDIA/ED-TELECOM 98, World Conferences on Educational Multimedia and Educational Telecommunications, Vol. 2, pp. 1413–1420. Wilensky, R., Mayfield, J., Albert, A., Cox, C., Luria, M., Martin, J. & Wu, D. (1986). UC – A Progress Report, Report no. UCB/CSD 87/303, Computer Science Division (EECS), University of California at Berkeley, California 94720, USA.
Artificial Intelligence Review 14: 43–88, 2000. Stephen J. Hegner et al. (eds.), Intelligent help systems for UNIX. © 2000 Kluwer Academic Publishers. Printed in the Netherlands.
43
The Berkeley UNIX Consultant Project ∗ ROBERT WILENSKY1, DAVID N. CHIN2, MARC LURIA3 , JAMES MARTIN4, JAMES MAYFIELD5 and DEKAI WU6 1 Division of Computer Science, Department of EECS, University of California, Berkeley, Berkeley, CA 94720, U.S.A. (E-mail:
[email protected]); 2 Department of
Information and Computer Sciences, University of Hawaii, 1680 East West Rd., Honolulu, HI 96822, U.S.A. (E-mail:
[email protected]); 3 NDS Technologies Israel Ltd., Jerusalem, Israel (E-mail:
[email protected]); 4 Department of Computer Science, University of Colorado, Boulder, CO 80309-0430, U.S.A. (E-mail:
[email protected]); 5 Computer Science and Electrical Engineering Dept., University of Maryland Baltimore County, Baltimore, MD 21228-5398, U.S.A. (E-mail:
[email protected]); 6 Department of Computer Science, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, P.R. China (E-mail:
[email protected])
Abstract. UC (UNIX Consultant) is an intelligent, natural-language interface that allows naive users to learn about the UNIX operating system. UC was undertaken because the task was thought to be both a fertile domain for Artificial Intelligence research and a useful application of AI work in planning, reasoning, natural language processing, and knowledge representation. The current implementation of UC comprises the following components: A language analyzer, called ALANA, that produces a representation of the content contained in an utterance; an inference component called a concretion mechanism that further refines this content; a goal analyzer, PAGAN, that hypothesizes the plans and goals under which the user is operating; an agent, called UCE go, that decides on UC’s goals and proposes plans for them; a domain planner, called KIP, that computes a plan to address the user’s request; an expression mechanism, UCE xpress, that determines the content to be communicated to the user, and a language production mechanism, UCGen, that expresses UC’s response in English. UC also contains a component called KNOME that builds a model of the user’s knowledge state with respect to UNIX. Another mechanism, UCT eacher, allows a user to add knowledge of both English vocabulary and facts about UNIX to UC’s knowledge base. This is done by interacting with the user in natural language. All these aspects of UC make use of knowledge represented in a knowledge representation system called KODIAK. KODIAK is a relation-oriented system that is intended to have wide representational range and a clear semantics, while maintaining a cognitive appeal. All of UC’s knowledge, ranging from its most general concepts to the content of a particular utterance, is represented in KODIAK. Keywords: agent, UNIX, consultant, natural language, intelligent interface, planning, knowledge representation, user modeling
∗ This paper is a considerably abridged version of “The Berkeley UNIX Consultant Project,” in Computational Linguistics 14(4): 35–84, Copyright ©1988 ACL and MIT Press.
44
ROBERT WILENSKY ET AL.
1. Introduction to the UNIX Consultant (UC) Project Some time ago, we began a project called UC (UNIX Consultant). UC was to function as an intelligent, natural-language interface that would allow naive users to learn about the UNIX1 operating system by interacting with the consultant in ordinary English. We sometimes refer to UC as “an intelligent ‘help’ facility” to emphasize our intention to construct a consultation system, rather than a natural-language front-end to an operating system. Whereas front-ends generally take the place of other interfaces, UC was intended to help the user learn how to use an existing one. We had two major motivations for choosing this task. These can be summarized by saying that we believed the task to be both interesting and doable. It seemed to us that much natural-language work – indeed, much of AI research – has fallen into two largely non-intersecting categories: On the one hand, there are quite interesting and ambitious projects that have been more the fertile source of exciting speculations than of useful technology. In contrast, there are projects whose scope is severely limited, either to some intrinsically bounded, real-world task or to a laboratory micro-world. These projects result in much excitement by the production of a “working system” or successful technology. But such projects have rarely produced much in the way of progress on fundamental issues that comprise the central goals of AI researchers. Our hope was that the consultation task would require us to address fundamental problems in natural-language processing, planning and problem solving, and knowledge representation, all of which are of interest to us. We believe this to be the case because (1) the domain of an operating system is quite large and complex, (2) users’ conceptions of computer systems are often based on other domains, particularly space and containment, and (3) the structure of a consultation session requires the consultant to understand the user’s language, hypothesize the user’s intentions, reason about the user’s problem, access knowledge about the topic in question, and formulate a reasonable response. In sum, virtually all the problems of language processing and reasoning arise in some fashion. While the task is interesting, it is nevertheless limited. Arbitrary knowledge of the world is generally not required, as it may be in other naturallanguage tasks, such as text processing. Even knowledge about the domain might be limited in ways that do not compromise the overall integrity of the system. In particular, the task is intrinsically “fail-soft.” Since the system is a ‘help’ facility, it need not be capable of handling every task put to it to serve a useful function. This is probably less true of systems that are intended to be interfaces. In their case, failure to correctly process a request by the user
THE BERKELEY UNIX CONSULTANT PROJECT
45
leaves the user with little recourse. However, a consultant may be quite useful even if it cannot help all the time. Similarly, there are strategies that might be employed in a consultant task that further reduce the degree of coverage required by the system. For example, if asked a very specific question, it is not unreasonable that a consultant respond by telling the user where to look for the information. Thus, the degree of expertise of the consultation system may be circumscribed. In other words, we felt that the operating-system domain was an appropriate replacement for the “blocks world.” Building a consultant for the domain is a real task one would like to have accomplished. The domain would limit the breadth, but not the depth, of AI research required. 1.1.
UC
– science or engineering?
Our approach to AI has had a distinctly “cognitive” bent. While a lengthy exposition might be needed to define this precisely, let it suffice here to say that we are interested in modeling human beings at least to a first approximation. Thus, as far as we could, we have attempted to build a system that modeled how we believe a human consultant actually functions. In some cases, this goal meant that we would make some problems harder for ourselves than one might if one’s goals were strictly technological. For example, since many word senses are unlikely to be used when talking to a consultant, a purely engineering approach might play down the problem of ambiguity. However, it is our goal to address such problems in a general fashion. At the same time, there were many pragmatic concessions that were made in implementing UC. Some of these were forced on us by the nature of university research. For example, a process might be divided into two components for the sake of implementation, although the particular division may not be motivated otherwise. These components might even exercise two different approaches to similar subproblems, depending on the biases of their authors. Sometimes, for the sake of efficiency, we chose to implement only part of what we believed to be a larger process. Also for efficiency’s sake, and to prevent truly difficult but infrequent problems from scuttling the entire effort, we implemented some solutions that we did not believe in completely. For example, UC’s control structure is overly simplistic in ways that we understand but have not corrected. We will make note of other such situations in the text below. In general, when this was the case, the solution used took the form of checking for certain frequently occurring cases in order to preclude having to solve a general problem. Since our goals were not strictly technological, we did not feel that it was necessary or appropriate in order for our system to be considered a
46
ROBERT WILENSKY ET AL.
success to produce a product that could actually be used in a real-world setting. However, we did feel that we should show that one could develop such a system along the lines that our research suggested. This would be accomplished by developing an extendible prototype. 1.2. Reasonable agents versus intelligent interfaces Our goal in building UC is to simulate a human consultant. As a result, the system has a structure that is more complex than other so-called intelligent interfaces. Indeed, we feel that looking at such a system as an interface is misleading. Instead, we prefer the metaphor of a reasonable agent. Unlike an interface, which is a conduit through which information flows, an agent is a participant in a situation. In particular, an agent has explicit goals of its own, and a reasonable agent must be able to make obvious inferences and display judgment in making decisions. Typically, a consultant constructed along the lines of a reasonable agent will make a user’s goals its own in trying to help that user. However, a reasonable agent is not always compelled to do so. Human consultants will not obligingly give out information to which a user is not entitled or which they suspect will be put to ill use. Similarly, a good consultant might deflect a user’s request because the consultant feels that the user does not have an adequate grasp of the domain, has a particular misconception, or is lacking some particular fact. In addition, a good consultant might do something more than simply answer a question. He might take the opportunity to show the user how to do a more general task of which the user’s particular request is merely a special case. In all these situations, an action other than simply responding to a request is warranted. A reasonable agent is ideally suited to handle such a broad class of situations. It does so by deciding what its goals should be in the given situation, and then planning for them. For example, when UC is asked how to crash the system, it forms two goals, one of helping the user to know what he or she wants, and one of protecting the integrity of the system. It then realizes that these two goals are in conflict, and eventually decides the conflict in favor of the latter goal. Of course, it is possible to achieve by other means various parts of the functionality here attributed to the model of a reasonable agent. For example, one can simply build one component that tries to detect misconceptions, another that checks for requests having to do with crashing the system, yet another to capitalize on opportunities to educate the user, etc. However, the reasonable-agent framework provides a single, flexible control structure in which to accomplish all these task, and, in particular, deal with interactions between them. That is its engineering motivation. Our primary reason for
THE BERKELEY UNIX CONSULTANT PROJECT
47
adopting it is that it is our theory about how humans function in consulting situations. 1.3. Overview The structure of this report is as follows. First, we present an outline of the structure of the current version of our consultation system. We follow this with a brief description of KODIAK. The next sections constitute the bulk of this report and are essentially a detailed description of a trace of a rather simple sentence through UC’s components. In doing so, the mechanisms of those components that are primarily responsible for UC’s agent-like qualities are described. Finally, we conclude with some discussion of the deficiencies of our current design. 1.3.1. Outline of UC’s structure UC is comprised of a number of components, which are invoked in a more or less serial fashion. (1) Language Analysis (ALANA) Language analysis is that component of the understanding process that computes a representation of the content of an utterance. ALANA, written by Charles Cox, produces a KODIAK representation of the content of an utterance. This representation generally contains only what can be determined from the words and linguistic structures present in the utterance. In our theoretical framework, we call such an analysis of an utterance its primal content. The concept of primal content is related to what is usually described as the literal meaning or sentence meaning of an utterance. However, unlike literal meaning, the primal content of an utterance involves certain idiomatic interpretations (i.e., it is not necessarily composed from words and general grammatical constructions). Also, the primal content of an utterance may be rather abstract, perhaps so much so that it may not be a suitable candidate for a meaning. For example, the literal meaning of “The cat is on the mat” is generally taken to be a conventional situation in which a cat is resting upon a mat. However, the primal content of this sentence would be more abstract, where the contribution of “on” is identical to that in the primal content of “The light fixture is on the ceiling” or “The notice is on the bulletin board.” Presumably, this conveys some sort of support relation. Note that such an abstract content appears never to be in itself the meaning of such an utterance (cf. Searle 1979). In contrast to primal content is the actual content of an utterance. The actual content is context-dependent, generally requires some amount of inference based on world knowledge, and is a suitable candidate for the
48
ROBERT WILENSKY ET AL.
meaning of an utterance. For example, the actual content of “The cat is on the mat,” without a further context specified, is what the literal meaning of this sentence is generally taken to be. Computing this content from the primal content requires pragmatic knowledge about the kind of support relation a cat and a mat are likely to be in, and requires making an inference that cannot be justified by the meanings of the terms and the grammatical constructions present in the utterance. The primal/actual content distinction is elaborated on in Wilensky (1987). (2) Inference (Concretion Mechanism) The particular kind of inference needed to go from a primal content to an actual content sometimes involves a process known as concretion (Wilensky 1983). Concretion is the process of inferring a more specific interpretation of an utterance than is justified by language alone. Concretion may involve finding a more specific default interpretation or some other interpretation based on the context. For example, in the example “the cat is on the mat” above, the actual content computed is the default support relation between a cat and a mat. In some compelling context, a quite different actual content may be computed from the same primal content. (There are other possible relations between primal and actual content besides the latter being a more specific interpretation of the former. For example, a conventionalized metaphor might have a primal content that more closely resembles its literal interpretation but an actual content resembling its metaphoric interpretation. Thus, one analysis of a sentence like “John gave Mary a kiss” will have as its primal content an instance of giving, but as its actual content an instance of kissing. We will not pursue further the details of the primal/actual-content distinction here. This is largely because, in UC, the need for concretion is widespread, and our handling of other kinds of primal/actual content computations is more haphazard.) In UC, concretion is needed primarily because we need to organize knowledge about more specific interpretations of utterances than can be arrived at through linguistic knowledge alone. For example, if UC is asked the question “How can I delete a file?”, ALANA can represent that this is a question about how to delete a file. But it would not have any reason to assume that deleting a file is a specific kind of deleting. Determining that this is so is likely to be important for several reasons. For example, knowledge about how to delete a file will be found associated with the concept of “file deletion,” say, but not with the concept of deletion in general. Thus UC must infer that “deleting a file” refers to the specific kind of deletion having to do with computer storage in order to perform subsequent tasks like finding plans for accomplishing the user’s request.
THE BERKELEY UNIX CONSULTANT PROJECT
49
In UC, concretion is the function of a special mechanism designed specifically for that purpose by Dekai Wu. The output of the concretion mechanism is another KODIAK representation, generally one containing more specific concepts than that produced by ALANA. Having a specific concretion mechanism is a pragmatic concession. We feel it is unlikely that such a specific mechanism is theoretically warranted. A more justifiable position is that a general inference mechanism should be exploited here, concretion being only one of the kinds of inference such a mechanism accomplishes. A unified text-inference mechanism that accomplishes concretion as well as other forms of inference has been built (Norvig 1987). It is our belief that some mechanism akin to Norvig’s should be used in UC in place of a specialized concretion engine, but no attempt has yet been made to do so. (3) Goal Analysis (PAGAN) Having computed an actual content for an utterance, UC then tries to hypothesize the plans and goals under which the user is operating. This level of analysis is performed by PAGAN, written by James Mayfield. PAGAN performs a sort of “speech act” analysis of the utterance. The result of this analysis is a KODIAK representation of the network of plans and goals the user is using with respect to UC. Goal analysis is important in many ways for UC. As is generally wellknown, an analysis of this sort is necessary to interpret indirect speech acts, such as “Do you know how to delete a file?”, or “Could you tell me how to delete a file?”. Furthermore, goal analysis helps to provide better answers to questions such as “Does ls -r recursively list subdirectories?”. An accurate response to the literal question might simply be “No.” But a better response is “No, it reverses the order of the sort of the directory listing; ls -R recursively lists subdirectories.” To produce such a response, one needs to realize that the goal underlying the asking of this question is either to find out what ls -r does, or to find out how to recursively list subdirectories. It is the job of the goal analyzer to recognize that such goals are likely to be behind such a question. More details about PAGAN can be found in Mayfield (1989, 1992, 2000). (4) Agent (UCEgo) Having hypothesized what the user wants of it, we would expect a system like UC to do what the user requested. But, as mentioned above, this is not always appropriate. UC should not aid and abet a user trying to perform malicious mischief; it might need to correct an errant user or it might decide to supply unasked-for information to one diagnosed as not knowing an important fact.
50
ROBERT WILENSKY ET AL.
In order to deal with such situations UC is constructed as an agent. This agent reacts to users’ requests by forming goals and acting on them. The central mechanism of UC is called UCEgo, and has been developed by David Chin. In a typical transaction, UCEgo will simply adopt the goal of having the user know what the user wants to know. However, as the example above illustrates, UCEgo may adopt other goals as well, such as protecting the integrity of the system. It may also have to detect conflicts between these goals. Sometimes, UCEgo, attempting to be educational, may adopt a somewhat different goal from the user’s. Thus, if the user asks UC to actually perform some request, such as telling the user who is on the system, UC will decide to tell the user how to perform such a function, rather than do what the user requested. UCEgo implements much of the agent-like character of UC . While interfaces are generally thought of as passive conduits through which information flows, UC is an agent that listens to the user and is generally helpful. But it has its own agenda, and the requests of the user are merely a source of input to it. More details about UCEgo can be found in Chin (1987, 1991, 1998, 2000a, 2000b). (5) User Modeling (KNOME) Several of UC’s components may need information about the user to make an effective choice. For example, an expert user certainly knows how to delete a file. Thus, such a user uttering “Do you know how to delete a file?” is unlikely to be asking for this information – more likely this user is testing the consultant’s knowledge. Assessing the knowledge state of the user is the function of a usermodeling program called KNOME, developed by David Chin. It is exploited by several components, including the Expression Mechanism described below. More details about KNOME can be found in Chin (1986, 1987, 1989). (6) Domain Planner (KIP) Typically, UCEgo tries to help the user. This usually requires determining a fact that the user would like to know. This task is accomplished by KIP. KIP is a “domain planner” developed by Marc Luria. While UCEgo infers its own goals, and plans to act on them, KIP is given a task by UCEgo of determining how to accomplish what the user wants to accomplish. KIP tries to determine how to accomplish this task, using knowledge about UNIX and knowledge about the user’s likely goals. KIP returns a plan, represented in KODIAK. For example, UCEgo may give KIP the task of determining how to move a file to another machine, if this is something the user wants to know. Here, KIP
THE BERKELEY UNIX CONSULTANT PROJECT
51
would come up with the plan of copying the file to the target machine and then deleting the original. Since UCEgo is also a planner, UC in effect has two planners within it. Again, this is probably not theoretically justifiable, although the two planners have ended up focusing on rather different aspects of planning. It remains to be seen whether a single mechanism might accommodate both functions. More details about KIP can be found in Luria (1985, 1987, 1988). (7) Expression Mechanism (UCExpress) Having gotten KIP to compute a plan for the user’s request, UCEgo now tries to communicate this plan to the user. To do so, it must determine which aspects of the plan are worthy of communication and how best to communicate them. For example, if it is likely that the user knows how to use commands in general, it might be sufficient just to specify the name of the command. In contrast, it might be helpful to illustrate a general command with a specific example. UCExpress is an “expression mechanism” written by David Chin. It edits out those parts of the conceptual answer returned by KIP that, for various reasons, appear unnecessary to communicate. UCExpress may also choose to illustrate an answer in several formats. For example, it might illustrate a general answer by generating a specific example, or it might explain one command in terms of another, simpler, command. The result of UCExpress is an annotated KODIAK network, where the annotation specifies which part of the network is to be generated. More details about UCExpress can be found in Chin (1987, 1988, 2000a, 2000b). (8) Language Production (UCGen) Once UC has decided what to communicate, it has to put it into words. This is done by a generation program called UCGen. UCGen is a simple generator, programmed by Anthony Albert and Marc Luria. It takes the marked KODIAK network produced by UCExpress and, using knowledge of English, produces sentences to complete the transaction with the user. (9) Learning Mechanism (UCTeacher) Since it is intended that UC be an extensible system, a mechanism has been developed to add new knowledge to the system by talking to it in natural language. This mechanism, called UCTeacher, is the work of James Martin. UCTeacher has capabilities to extend both UC ’s knowledge base of UNIX facts as well as its knowledge of English vocabulary. More details about UCTeacher can be found in Martin (1985, 1986a, 1986b, 1987, 1988, 2000).
52
ROBERT WILENSKY ET AL.
Note that several UC components deal with goals and plans, but in rather different ways. To minimize confusion, we emphasize the different tasks that these programs perform: PAGAN hypothesizes goals of the user with respect to UC, UCEgo reacts to the user by forming goals of its own and then trying to accomplish them, and KIP is given the goal of solving some underlying UNIX problem. Thus, PAGAN is a goal analyzer, i.e., a program that attempts to determine someone else’s goals, while UCEgo and KIP are planners, i.e., programs that determine and produce plans for their own goals. UCEgo and KIP exist as two separate programs primarily for pragmatic reasons. It may be possible that a single planner could accomplish the tasks of both UCEgo and KIP. Such an integration will have to await a revision of our system. 1.4.
UC
examples
The first set of examples illustrates the scope of the current UC prototype. They are intended to show the kind of requests the system can handle, as well as its scope. The second set demonstrates some more complex capabilities. Each sentence below preceded by the prompt ‘#’ is supplied by the user. The rest of the text is a computer-generated response. Note that the system handles a variety of request forms, some direct and some indirect. Also, the responses to “general” questions will cause an example to be generated. Finally, the reader may notice that the vocabulary used to respond to a request is often different from that used in the request itself. For example, a question about file protection is answered in terms of file permission; a question about changing a directory is answered in terms of connecting to a directory. This is because the representations used to encode the system’s knowledge are kept in a canonical form, so that different expressions of the same meaning are mapped to the same internal representation. Upon generating a response, the system uses only this internal representation, which, coupled with its generating biases, causes an expression of an answer that is largely independent of the vocabulary that appeared in the request. 1.4.1. Simple examples Welcome to UC (Unix Consultant) version 3.23 To a UC ‘#’ prompt, please type in your questions about the Unix file system in English. To leave, just type a ‘ˆD’ or ‘(exit)’. Hi. How can I help you? # Can you tell me how to move a file? Use mv. For example, to move the file named foo to the file named foo1, type ‘mv foo foo1’.
THE BERKELEY UNIX CONSULTANT PROJECT
53
# Do you know how to delete a file named dekai? Type ‘rm dekai’. # Do you know how to delete a file? Use rm. For example, to delete the file named foo, type ‘rm foo’. # How can I delete the file charlie? Type ‘rm charlie’. # How can I delete the file named tony? Type ‘rm tony’. # How can I get rid of the file named boob? Type ‘rm boob’. # I want to delete a file named peter. Type ‘rm peter’. # I want to delete a file. Use rm. For example, to delete the file named foo, type ‘rm foo’. # I want to delete the file dave. Type ‘rm dave’. # How can I add read permission to a file? Use chmod. For example, to add individual read permission to the file named foo, type ‘chmod u+r foo’. # How can I add the owner execute permission to a file? Use chmod. For example, to add individual execute permission to the file named foo, type ‘chmod u+x foo’. # How can I change the group execute permission of a file? Use chmod. For example, to remove group execute permission from the file named foo, type ‘chmod g-x foo’. # How can I change the user read permission of a file? Use chmod. For example, to remove individual read permission from the file named foo, type ‘chmod u-r foo’. # How can I remove the group write permission from the file named bimbo? Type ‘chmod g-w bimbo’. # How do I change the protection of a file? Use chmod. For example, to remove individual read permission from the file named foo, type ‘chmod u-r foo’. # How can I compact a file? Use compact. For example, to compact the file named foo, type ‘compact foo’. # How can I create a directory? Use mkdir. For example, to create the directory named foo, type ‘mkdir foo’. # How can I find a file? Use whereis. For example, to find the file named foo, type ‘whereis foo’. # How can I find out how much disk space I am using? Type ‘du’. # How can I find out who is using the system? Type ‘finger’. # How can I find out wilensky’s phone number? Type ‘finger wilensky’. # How can I list a directory? Use ls. For example, to list the directory named foo, type ‘ls foo’.
54
ROBERT WILENSKY ET AL.
# How can I logout? Type ‘logout’. # Tell me how to leave. Type ‘logout’. # How can I move a file to the file named blah? Use mv. For example, to move the file named foo to the file named blah, type ‘mv foo blah’. # How can I move the file junk? Use mv. For example, to move the file named junk to the file named foo, type ‘mv junk foo’. # How do I move a file? Use mv. For example, to move the file named foo to the file named foo1, type ‘mv foo foo1’. # How do I change my directory? Use cd. For example, to connect to the directory named foo, type ‘cd foo’. # How do I copy a file? Use cp. For example, to copy the file named foo to the file named foo1, type ‘cp foo foo1’. # How do I copy the file foo to the file blah? Type ‘cp foo blah’. # How do I create a directory? Use mkdir. For example, to create the directory named foo, type ‘mkdir foo’. # How do I print a file named info? Type ‘lpr info’. # I want to print a file. Use lpr. For example, to print the file named foo, type ‘lpr foo’. # I want to change my working directory. Use cd. For example, to connect to the directory named foo, type ‘cd foo’. # I want to copy the file named rick. Use cp. For example, to copy the file named rick to the file named foo, type cp ‘rick foo’. # I want to delete a directory. Use rmdir. For example, to delete the directory named foo, type ‘rmdir foo’. # I want to remove a directory. Use rmdir. For example, to delete the directory named foo, type ‘rmdir foo’. # What’s the best way to delete a directory? Use rmdir. For example, to delete the directory named foo, type ‘rmdir foo’. # I want to display a file. Use more. For example, to display the file named foo, type ‘more foo’. # I want to edit a file. Use vi. For example, to edit the file named foo, type ‘vi foo’. # I want to find out Joe’s address. Type ‘finger joe’. # I want to remove a directory named junk. Type ‘rmdir junk’.
THE BERKELEY UNIX CONSULTANT PROJECT
55
# What’s the command to compact a file? Use compact. For example, to compact the file named foo, type ‘compact foo’. # What is a directory? A directory is a file that is used to contain files. # What is a file? A directory is a container that is used to contain text, code or files. # What is a rm? Rm is a command that is used to delete files. # What is a search path? A search path is a list of directories that is used by the csh to search for programs to execute.
1.4.2. More complex examples Welcome to UC (Unix Consultant) version 3.23 To a UC ‘#’ prompt, please type in your questions about the Unix file system in English. To leave, just type a ‘ˆD’ or ‘(exit)’. Hi. How can I help you? # Is rn used to rename files? No, rn is used to read news. To rename a file to another file, use mv. For example, to rename the file named foo to the file named foo1, type ‘mv foo foo1’.
In this example, UC hypothesized that the user asked the question in order to get an answer to some underlying question. In particular, PAGAN hypothesized possible goals of wanting to know how to rename a file and wanting to know what ‘rn’ does. Since the answer to the direct question is negative, UCEgo realizes that the underlying goals are not addressed by this answer, and, therefore, attempts to answer them. This results in the extended answer. # How can I crash the system? I’m sorry, I cannot tell you.
In this example, UC detected a conflict between its goal of helping the user and that of preserving the integrity of the system. It resolved this conflict in favor of the latter goals. However, since it could not accommodate the user’s request, it attempts to be polite by apologizing to the user. # What does ls -v do? ls does not have a -v option.
Here the user asks about a non-existent command option. Actually, the UNIX ‘ls’ command ignores options that it does not recognize, so the consultant
56
ROBERT WILENSKY ET AL.
could have answered truthfully by saying that ‘ls -v’ causes one’s directory to be listed. It did not, because PAGAN interpreted the request as a question about the function associated with the command. UCEgo reasons that, since (1) it doesn’t know about such an option, and (2) it knows all the options to all simple commands, and (3) ‘ls’ is a simple command, the user must have a misconception. It then generates the goals of correcting the misconception, which results in the answer shown. Note that UC uses an open world hypothesis together with metaknowledge to determine whether it knows some facts about UNIX. In the next example, two different sessions with two different users are represented. Welcome to UC (Unix Consultant) version 3.23 To a UC ‘#’ prompt, please type in your questions about the Unix file system in English. To leave, just type a ‘ˆD’ or ‘(exit)’. Hi. How can I help you? # How can I delete a file? Use rm. For example, to delete the file named foo, type ‘rm foo’. # What does rwho do? Rwho is used to list all users on the network, the users’ tty, the users’ login time, and the users’ idle time. Welcome to UC (Unix Consultant) version 3.23 To a UC ‘#’ prompt, please type in your questions about the Unix file system in English. To leave, just type a ‘ˆD’ or ‘(exit)’. Hi. How can I help you? # How can I find out the inode of a file? Use ls -i. # What does rwho do? Rwho is like who, except rwho lists all users on the network.
Because he asked such a simple question, the first user was diagnosed by KNOME as being a novice user. This resulted in the answer to his first question
being illustrated by an example, and the extended format of his answer to second. In contrast, the second user asked about a relatively technical aspect of UNIX. Thus, this user is likely to be a relatively sophisticated user. As a result, the answer to that user’s first question was not illustrated with an example. For the same reason, UC presumes that the second user knows about
THE BERKELEY UNIX CONSULTANT PROJECT
57
the ‘who’ command, and takes advantage of this knowledge in generating the response. Note this example also illustrates UC’s ability to generate different types of responses. This example was able to use a simile format, because the command to be generated was thought to be sufficiently close to one UC believes the user to already know. Formatting is discussed in Section 3.6. The next example illustrates some aspects of UC’s ability to generate plans. # How do I print Jim’s file on the lineprinter? Use lpr. This plan will not work if you don’t have read permission on Jim’s file.
In this example, KIP, UC’s domain planner, suspects that the plan it is suggesting might fail. This is because the plan involves printing someone else’s file, and KIP knows that the lack of appropriate permission in such cases is a likely source of plan failure. However, since it does not know a more certain plan, it delivers the suspect one, but warns the user of the possible problem. A fact about a possible problem with a plan is called a concern. Concerns are explicitly represented in UC’s knowledge base. At this stage, UC’s knowledge of concerns is fragmentary. Thus, questions above that have the same structure as this one may not cause any warnings to be generating simply because the requisite knowledge is lacking. Concerns are discussed further in Luria (1985). # Who is on the system? I’m sorry, I don’t know. Use users to find out who is on the system.
Here UC fails to answer the user’s question, because it does not have any way of running UNIX commands itself. Its failure causes it to apologize. UC then forms the goal of the user knowing how to determine the requested information. It addresses this goal, which leads to the second part of the response.
2.
KODIAK
The knowledge used by UC is represented in KODIAK, as are the various stages of the processing of an utterance. KODIAK (Keystone to Overall Design for Integration and Application of Knowledge) is an implementation of CRT (Cognitive Representation Theory), an approach to knowledge representation that bears similarities to numerous other systems, but especially
58
ROBERT WILENSKY ET AL.
Table 1. KODIAK legend. Representation
Meaning NODES
CONCEPT
# CONCEPT n
, where n is an integer
An absolute A relation CONCEPT n is an instance of CONCEPT . LINKS
The following is priority ordered (i.e. use first match for meaning): CATEGORY −α → # # is an aspectual, i.e., a relation in which CATEGORY participates. CATEGORYA −D→ CATEGORYB CATEGORYB dominates CATEGORYA . INSTANCE −I→ CATEGORY INSTANCE is an instance of CATEGORY . #−C→ CATEGORY The argument to # is constrained to be of type CATEGORY . #−V→ OBJECT The value of the argument to # is OBJECT . #−=→# The first aspectual is constrained to have the same value as the second. CATEGORY −rel→# # is an aspectual of CATEGORY , and # specializes rel, an aspectual of some concept dominating CATEGORY . CATEGORYA −rel→ CATEGORYB Each member of CATEGORYA participates in rel with some member of CATEGORYB .
those of Schank (1975), Schubert (1976), Shapiro (1979), and Brachman and Schmolze (1985). KODIAK differs from these systems in what we believe are significant ways. However, here we shall try to trivialize rather than accentuate these differences so that the reader can relate our representations to more familiar ones. The reader should consult Wilensky (1986) for a more detailed account and justification of this representation system. Table 1 summarizes the notation used in KODIAK diagrams. 2.1.
UNIX
knowledge in UC
The KODIAK knowledge representations used in UC include several rather general notions, such as state change, goal, and action, plus many specific facts about UNIX. The complete collection is too lengthy to include here. (UC is currently constructed from approximately 200 KODIAK diagrams, consisting of about 1000 absolutes and 2,000 relations. While a scope of a
THE BERKELEY UNIX CONSULTANT PROJECT
59
Figure 1. Knowledge about deletion represented in KODIAK.
diagram is to some degree arbitrary, diagrams roughly correspond to definitions of meaningful entities, like the definition a particular command.) Some more important concepts used in modeling the domain will be explained in the individual sections of this report. To facilitate understanding the KODIAK diagrams that follow, consider the representation that the UC’s knowledge base contains about the UNIX ‘rm’ command. This is used to delete a file named by its argument. Figure 1 shows how knowledge about deletion is represented in UC. The central node in this diagram is DELETE - EFFECT. DELETE - EFFECT is shown as being a kind of STATE - CHANGE that causes a something to go from existence to nonexistence. (The notions of existence and negation do play a special role in KODIAK , but also exist as ordinary states, as is the case here. From the point of view of this example, these are just like any other states.) DELETE - EFFECT specifies the “minimal” deletion event. For example, it says nothing about the cause of such an event, or who the actor of it may be. In UC in particular and in CRT in general, such state changes are the bases from which we build more elaborate concepts. For example, the action of deleting something is represented as an action that causes something to be deleted.
60
ROBERT WILENSKY ET AL.
This is encoded by showing DELETE - EFFECT to be the effect of DELETE ACTION . Using names like DELETE - EFFECT may be somewhat misleading. In particular, DELETE - EFFECT is not required to be the effect of anything – while DELETE - ACTION is defined as having DELETE - EFFECT as its result, this statement imposes a requirement on DELETE - ACTION, not on DELETE EFFECT. We call such concepts EFFECTs rather than EVENTs, say, to emphasize that we mean to include only the most essential elements of the concept, i.e., just a specialized state change. DELETE - FILE - EFFECT is a specialized version of DELETE - EFFECT in which the object deleted is constrained to be a file. DELETE - ACTION is correspondingly specialized to DELETE - FILE - ACTION. It is also shown as being a kind of TRANSITIVE - ACTION. This is a very general category denoting any action that acts upon an object. This sort of structure, in which there exists parallel effect and action hierarchies, with the effect hierarchy carrying most of the semantic weight, is typical of the representation that appear in UC. Figure 2 connects this general knowledge about deletion with knowledge about UNIX. Here we state how to achieve a DELETE - FILE - EFFECT. This accomplished by the node labeled PLANFOR 2, which points to EXECUTE UNIX - RM - COMMAND and to DELETE - FILE - EFFECT. A PLANFOR indicates that something is conceptualized as a plan for a particular goal (PLANFORs are discussed below). In other words, this notation represents the particular fact that the ‘rm’ command (i.e., the command whose name is “rm”) is used to achieve the effect of deleting a file. Again, this structure is typical of that seen in UC – most of the information about a command is represented as information about the use of that command; the intended function of the use of a command is represented by a PLANFOR between a node representing the use of the command and some effect. The rest of the diagram specifies the format of the command. In particular, the Equate link specifies that, to delete a particular file, its name must be the same as that of the argument supplied to ‘rm’.
3. A Tour Through UC The following sections describe the components of UC in more detail. To aid in understanding how these components contribute to the processing of an individual utterance, we show how each section processes the example sentence “Do you know how to print a file on the imagen?”2 In most cases, a module is capable of doing a great deal more than is required for this example, and such capabilities are attested to. However, the example is
THE BERKELEY UNIX CONSULTANT PROJECT
61
Figure 2. Knowledge about deleting files in UNIX represented in KODIAK.
useful for illustrating the kind of processing that is performed for a typical request. In order to produce a paper of reasonable length, we reduced considerably the description of some of UC’s modules. We have focused on those processes that contribute to UC’s agent-like nature, while some more conventional modules, such as the conceptual analyzer, are mentioned only in passing. References are given to descriptions of these neglected components, which have all appeared elsewhere in print, should the reader find the account herein dissatisfying. 3.1. The analyzer A conceptual analyzer maps a string of words into a meaning representation. ALANA (Augmentable LANguage Analyzer), the conceptual analyzer for UC, takes as input a sentence typed by a user, and builds a conceptual representation using the KODIAK knowledge-representation language. ALANA constructs the “primal content” of the input utterance. The primal content is the interpretation that can be computed from grammatical and lexical knowledge; it is generally rather abstract. ALANA’s results are further interpreted and refined by other parts of the system, such as the concretion mechanism, to produce an “actual content,” and the goal analyzer, to produce a representation of the intentions underlying the utterance.
62
ROBERT WILENSKY ET AL.
Figure 3. ALANA’s output for “Do you know how to print a file on the Imagen?”
ALANA is a descendent of PHRAN (Wilensky and Arens 1980), the frontend natural-language component for the original UC (Wilensky et al 1984). Like PHRAN, ALANA reads the user’s input and forms a concept that the other UC components can use for their tasks. Also like PHRAN , ALANA uses as its primitive knowledge unit the pattern-concept pair, which relates a naturallanguage structure to a conceptual structure. UC has a total of 476 patterns and knows 284 words. ALANA differs from PHRAN in its generality. ALANA generalizes on the idea of pattern-concept-pair analysis, while making it easier than it was with PHRAN for a knowledge-adder to add new patterns to the system. Since a more detailed description of ALANA can be found in (Cox 1986), we will not elaborate on it here. Instead, we merely show in Figure 3 the output produced by ALANA upon reading the sentence “Do you know how to print a file on the imagen?”. This diagram may be interpreted as follows: The entire request is summarized as ASK 11, i.e., some asking event. What is asked for is verification of some item, QUESTION 11, whose content is KNOW 3, i.e., an instance of knowing. The knower of the item is UC, and the fact is ACTION 6. ACTION 6 is interpreted as something that is the cause of a printing action PRINTACTION 0, which is itself an action whose effect (PRINT- EFFECT 0) is to cause a file (FILE 6) to be printed on an imagen printer (IMAGEN 0). Some of the nodes in this diagram point to a node labeled HYPOTHETICAL. This is a tentative convention used to indicate that the knowing and printing event, etc., are not real events, but merely hypothetical ones.
THE BERKELEY UNIX CONSULTANT PROJECT
63
3.2. The concretion mechanism As mentioned previously, our theoretical posture is that concretion is but one of a number of inference processes that can be accomplished by a single mechanism. However, in UC, for reasons of efficiency, and for pragmatic advantages, a separate concretion mechanism was implemented (by Dekai Wu). This mechanism currently does rather straightforward classification. The mechanism concretes by using information about inheritance and value constraints, as well as by considering relation information between concepts. A concept represented as an instance of a category is passed to the concretion mechanism. Its eligibility for membership in a more specific subcategory is determined by its ability to meet the constraints imposed on the subcategory by its associated relations and aspectual constraints. If all applicable conditions are met, the concept becomes an instance of the subcategory. At the same time, the relations in which the concept participates may be concreted to reflect the more specific relations of the new category of which it is inferred to be a member. Parts of the representation of printing are shown in Figure 4. Besides the printing of the contents of a computer file, PRINT- EFFECT is in principle applicable to other types of printing, such as printing a newspaper or a book. The concretion mechanism checks each of the more specific concepts dominated by PRINT- EFFECT, searching for one whose constraints can be satisfied by the input. It finds PRINT- FILE - EFFECT, who only additional constraint is that its print-object must be a file. Since PRINT- EFFECT 0 is in print-object relation with the object FILE 6, which is indeed an instance of FILE, the process can descend to this node. The concretion process will continue until it can concrete no further. Of course, it is perfectly plausible just to preclude from UC on engineering grounds interpretations of words that do not occur in the UNIX domain. As we suggested earlier, it is our preference not to do so, since we wish to address, rather than finesse, fundamental language issues. However, doing so would not really eliminate the need for concretion. Even if we do not include concepts of non-computer printing in our knowledge base, we would still have many different kinds of printing, e.g., printing ASCII files versus binary files or printing on the lineprinter versus the laser printer. A query about each of these kinds of printing requires a different response, although the term “printing” applies to all of these. A system like UC needs to concrete the concept of printing in general to the particular kinds of printing that it knows about, in order to find the knowledge needed to answer the question. Thus, eliminating interpretations that lie outside the domain simplifies the problem somewhat, but it does not change its essential nature.
64
ROBERT WILENSKY ET AL.
Figure 4. Some knowledge about printing.
In general, when concretion occurs, some node is reclassified as being an instance of a more specific category, and, in addition, the relations predicated about that node are also reclassified. For example, here we concrete PRINTEFFECT 0 to an instance of PRINT- FILE - EFFECT . At the same time, we should concrete the relation print-object predicated about it to a use of the more specific relation print-file-object. Similarly, print-dest is concreted to printfile-dest. Continuing in this fashion, the mechanism can move from PRINT- EFFECT to LASER - PRINT- EFFECT, and finally to IMAGEN - PRINT- EFFECT, since the print-dest of the input is IMAGEN 0, which is an instance of IMAGEN. At the same time, the relation print-dest is concreted to imagen-dest. In parallel with this concretion, the node PRINT- ACTION 0 gets concreted to an instance of IMAGEN - PRINT- ACTION . The final result is shown in Figure 5.
THE BERKELEY UNIX CONSULTANT PROJECT
65
Figure 5. Result of concreting “print file on the imagen.”
3.3. The goal analyzer Once an utterance has been converted to a KODIAK representation by ALANA, and has been further refined by the concretion mechanism, this internal representation is passed to PAGAN (Plan And Goal ANalyzer). PAGAN’s task is to determine what goals the speaker is addressing in making the utterance. For example, when given a representation of the utterance “Do you know how to print a file on the imagen?” asked by a naive user, PAGAN should infer that the user was using the utterance to address the goal of knowing how to print a file on the imagen. Note that PAGAN is not responsible for detecting goals that are held by the speaker but that are not conveyed by the speaker’s utterances. This problem is addressed by the ego mechanism and by the planner. To successfully do goal analysis, at least two questions must be answered. The first concerns the utterance in isolation: What kind of act does this utterance constitute? This question has traditionally fallen under the rubric of “speech-act theory” (Austin 1962; Searle 1969). For example, “Do you know how to print a file on the Imagen?” potentially has both a direct and indirect interpretation which PAGAN must choose between. The second question a goal-analysis mechanism must answer examines the role of the utterance in conversation:
66
ROBERT WILENSKY ET AL.
Figure 6. Trace of PAGAN’s processing of “Do you know how to print a file on the imagen?”
How does this utterance relate to other utterances? By virtue of being an action, an utterance always occurs within a context. This context includes such diverse factors as the identities of the speaker and of the audience, the social relationship between them, the physical locale, the task the conversation is supplementing if any, and so on. One feature of this context that is salient to goal analysis is the presence of conventional, multi-utterance sequences. Consider the exchange: 1. Do you have write permission on the parent directory? 2. Yes. The ability to understand the full meaning of (2) is contingent on the realization that it relates directly and conventionally to (1). Thus, PAGAN will require knowledge of such sequences to correctly determine the goal underlying utterances such as (2). The input to PAGAN is the structure built by the analyzer from this utterance and refined by the concretion mechanism. A trace of PAGAN as it processes this structure is shown in Figure 6. The first step performed by PAGAN is to determine whether the utterance is the continuation of a conversational plan already in effect. For this to be the case, there would need to be some previous dialogue to provide the necessary context. This dialogue would take one of two forms. It might be a plan that UC believed the user to be pursuing before the current utterance was encountered. Alternatively, it could be a plan introduced by UC that the user has adopted, that UC believes the user to be pursuing only after witnessing the current
THE BERKELEY UNIX CONSULTANT PROJECT
67
Figure 7. A plan for knowing is to ask if the hearer knows.
utterance. Since there is no previous context in the example we are tracing, neither of these possibilities is found to hold (1–2). Next, PAGAN tries to match the utterance against the first steps of plans in its planfor knowledge base. The first possibility is compared with the input structure (3), but one pair of corresponding nodes is found not to match (4–5). The second possibility, one that does match the utterance, is then compared with the input structure (6–7). This planfor corresponds to the indirect interpretation of the utterance. This is the planfor that is shown in Figure 7. A third possibility, corresponding to the direct interpretation of the utterance, also matches the input structure (8–9). An attempt to resolve this ambiguity is now made (10–11). Since neither goal matches an expected goal (12–15), the planfors are examined for plausibility. The direct interpretation is discarded, because the user model indicates that it is likely that the user knows that UC knows how to print a file on the imagen (16). (More details about the user model can be found in Chin (1986, 1989).) Thus, the planfor representing the indirect interpretation is selected (17). Once the utterance has been matched to the first step of this planfor, an instance of a PLANFOR is created with the goals determined from the input. In addition, an instance of the HAS - GOAL state is built (18). The planner of this state is the user, and the goal is the goal of the PLANFOR. This HAS -
68
ROBERT WILENSKY ET AL.
Figure 8. PAGAN output.
GOAL represents the goal that UC believes the user had in mind in making the utterance, and is returned by PAGAN as its result (19). It is shown in Figure 8. In this figure, note that PAGAN has created a node labeled ACTION 7, whose actor is the user. This represents the inference made by the goal analyzer that, if a user wants to know an action to achieve some goal, then the user intends to be the actor of that action.
3.4. The ego mechanism Thus far in the processing, UC has parsed and understood the question, and the goal analyzer has asserted that the user has the goal of knowing a plan for printing a file on the imagen. At this point, UCEgo’s processing begins. Figure 9 shows a trace of the goal-detection phase of UCEgo. In the trace, UC - HAS - GOAL 19 represents UC’s goal of helping the user (HELP 2). HAS - GOAL-ga0, which is also shown in Figure 8, is the user’s goal of knowing (KNOW-ga0) how to print a file on the imagen. PLANFOR 29 represents the fact that a plan for helping the user (HELP 2) is for UC to satisfy KNOW -ga0, which is the user knowing how to print a file on the imagen. UC - HAS - INTENTION 6 represents UC ’s intention to satisfy KNOW -ga0. UC HAS - GOAL 20 represents UC ’s goal of the user knowing how to print a file on the imagen. The user’s goal (HAS - GOAL-ga0 in the trace) combines with UC’s goal of helping the user (UC - HAS - GOAL 19, present in UC’s initial state) to activate the fact (PLANFOR 29) that a plan for helping the user is for UC to satisfy the goal of the user knowing a plan for printing a file on the imagen. Next,
THE BERKELEY UNIX CONSULTANT PROJECT
69
Figure 9. Trace of UCE go’s goal-detection process.
this planfor combines with UC’s goal of helping the user (UC - HAS - GOAL 19) to make UCEgo adopt the intention (UC - HAS - INTENTION 6) of satisfying the goal of “the user knowing a plan for printing a file on the imagen.” This intention arises as a result of UCEgo’s if-detected demon for plan selection. Finally, UCEgo adopts the user’s goal as its own. This subgoal (UC - HAS GOAL 20) is the result of UCEgo’s goal detection process. After UCEgo has detected the goal of “the user knowing a plan for printing a file on the imagen,” the plan-selection phase of UCEgo attempts to select a plan to satisfy this goal. Figure 10 shows a trace of this part of the processing. In the trace, UC - HAS - GOAL 20 is UC’s goal of knowing (KNOWga0) ACTION 7, which represents the plan part of the planfor (PLANFOR-ga1) for printing a file on the imagen (PRINT- EFFECT 0). UNIX-planner1 represents a call to KIP. PLANFOR 70 says that a plan for achieving the goal of PRINTEFFECT 0 is to use EXECUTE - UNIX - IPR - COMMAND 0, which entails using the command ‘lpr -Pip’ on the name of the file to be printed. ANSWER - FOR 1 says that an answer to the query of “how to print a file on the imagen” (ACTION 7) is PLANFOR 70. PLANFOR 30 says that a plan for achieving the goal of the user knowing how to print a file on the imagen (KNOW-ga0) is for UC to tell (TELL 4) the user PLANFOR 70. UC - HAS - INTENTION 7 represents UC’s intention of telling the user (TELL 4). UCexpress1 represents a call to UCExpress to execute TELL 4.
70
ROBERT WILENSKY ET AL.
Figure 10. A trace of UCE go’s plan selection process.
The first step of the plan is to call KIP, the planner component of UC. Figure 11 shows the if-detected demon that calls KIP. KIP is called whenever UC has the goal of knowing a plan for something In the trace, UC - HAS GOAL 20 and PLANFOR -ga1 combine to cause a call to KIP (UNIX -planner1 in the trace). KIP, as is describing is the next section, comes back with an answer (PLANFOR 70), which is an answer (ANSWER - FOR 1) to the user’s question. Then UCEgo detects the plan of telling the user the answer (PLANFOR 30 in the trace). This plan, with UC’s goal (UC - HAS - GOAL 20) of the user knowing
THE BERKELEY UNIX CONSULTANT PROJECT
71
Figure 11. If-detected demon for calling the UC planner, KIP.
the answer leads to UC’s intention to tell the user the answer (UC - HAS INTENTION 7). Finally, the intention translates into a call to UC ’s expression mechanism, UCExpress (UCexpress1 in the trace), which eventually calls UCG en to produce the answer. KIP, UCExpress, and UCG en are described in the following sections. 3.5. The planner This section describes KIP (Luria 1985), a knowledge-based, common-sense planner (Wilensky 1983). KIP includes: − a knowledge base of facts about the domain; − a planning component that uses this knowledge to: • find potential plans for problem situations; • notice potential problems with these plans; • use metaplanning knowledge (knowledge about plans) to determine which plans to suggest. KIP uses the same knowledge base as the rest of UC . In principle, it could be used to do the planning required by UCEgo. As mentioned previously, this was not attempted mostly for pragmatic reasons. In addition, the planning done by UCEgo is much more straightforward and does not require recourse to the same magnitude of knowledge as does KIP. Thus, it seems reasonable to use a much simpler planner for communicative functions. The basic structure of KIP is similar to that of UCEgo. However, KIP is a more elaborate planner. It must be able to plan for unanticipated goals of the user, and must be concerned with adverse consequences of the plans it proposes. In general, KIP has to iterate through the planning process a number of times to arrive at an adequate plan.
72
ROBERT WILENSKY ET AL.
The following are the steps of the iterative process that KIP uses: 1. Goal detection – decide what goals to work on: − Start with the goals input from UCEgo. − Detect new goals that might arise from use of proposed plans. 2. Plan selection – select a possible plan: − Try to find a stored plan that is related to the user’s goals. − Propose a new plan if necessary based on knowledge in the system. 3. Projection – test whether plan would be executed successfully: − Check for conditions that are likely not to be met. − Notice bad side effects. − Detect goals needed to address discovered problems 4. Plan evaluation – decide whether plan is acceptable: − Reject plan if impossible. − Accept if unproblematic. − Create warning if possible but problematic. The iterative structure described here is implemented by a series of metaplans (Wilensky 1983). The underlying metaplan is to find a particular plan that the user can use; these steps are parts of that process. The example being considered throughout this paper does not require much work by KIP. This is because UC’s knowledge base contains a plan for the goal of printing a file on the imagen. Also, there are no concerns associated with this particular plan. (Had the input referred to printing the file of another user, or printing on another printer, KIP would have retrieved concerns about not being about to access another persons’ files, or about the peculiarities of the type of printer, respectively.) Thus, the planner simply ends up retrieving the stored plan, making an instance of it, and passing it on. The input to the planner from UCEgo is PRINT- EFFECT 0, which we show in Figure 12 along with its most pertinent connections. Figure 13 shows an annotated trace of KIP’s processing for this example. KIP is passed a single goal, PRINT-EFFECT0. After checking for goal conflict concerns, the planner checks to see if it has achieved the exact same goal before. When it fails, as is the case here, it looks up the hierarchy. In this example, it looks first at the category IPRINT- EFFECT, of which PRINTEFFECT 0 is represented as being an instance. KIP finds in the knowledge base PLANFOR 7 (not shown in the trace) that connects IPRINT- EFFECT with EXECUTE - UNIX - IPR - COMMAND . This knowledge is shown in Figure 14. There are no concerns retrieved with this particular plan, so KIP will simply produce an instance of it and return it to UCEgo. Note that KIP’s retrieval of a plan for this goal is simplified by the concretion mechanism’s having classified the input as an instance of “imagen printing.” Originally, ALANA represented the meaning of the input as an
THE BERKELEY UNIX CONSULTANT PROJECT
73
Figure 12. Input to KIP from “Do you know how to print a file on the imagen?”: PRINT- EFFECT 0 and important links.
Figure 13. Trace of KIP’s processing.
74
ROBERT WILENSKY ET AL.
Figure 14. Representation of plan for printing a file on the imagen.
instance of printing. However, there are no plans associated with printing per se, but only with printing’s more specific subcategories. Thus, the planner would have to search for an appropriate plan had the concretion mechanism not done its job. In actuality, the planner starts at the most specific category that the input could be classified as, and works up the hierarchy. Therefore, the planner is guaranteed to find the most specific plan applicable to the situation. Note that KIP has made NAME 6 a hypothetical object, because it is the name of FILE 6, which is itself hypothetical. After KIP has made a filled-in copy of the plan, no pertinent concerns are found, and there is nothing more to do, so the plan found is returned. The planner’s output is shown in Figure 15. 3.6. The expression mechanism After UCEgo, in conjunction with KIP, has determined the proper answer, it calls UCExpress to express it to the user. The input to UCExpress is shown in Figure 16. This input conceptual network is UCEgo’s plan to tell the user the plan that KIP produced (Figure 15). If the plan were to be directly generated into English, the result might be something like: A plan for printing a file on the imagen printer is to use the lpr command with the imagen printer option. The format of the command is “lpr -Pip” and followed by the name of the file to be printed on the imagen printer.
THE BERKELEY UNIX CONSULTANT PROJECT
75
Figure 15. Output from KIP for example “Do you know how to print a file on the imagen?”
Instead, UCExpress prunes the output, and, since the user is suspected of being a novice, generates an example to illustrate the plan. The following trace illustrates the processing of UCExpress: The first phase of UCExpress is pruning, during which those concepts that the user already knows are marked so that the generator will not generate them. In this case, PRINT- EFFECT 0, the goal of PLANFOR 70, is pruned, as it is in the current context. In traversing the input conceptual network, UCExpress runs into the command-format IPR - FORMAT 0. Since this node is presumably not known to the user, UCExpress would normally not prune it. However, UCExpress specifically checks command-formats to see if they are as specific as possible, and since FILE 6 does not have a name, UCExpress schedules the creation of an example to explain the format, and prunes the original node. The rest of the network is passed without any pruning. In creating an example, UCExpress must specify all the parameters in the command format. Thus, the name of the file, which was not specified by the user, is made explicit in the example. Here, the name “foo” was chosen arbitrarily from a list. The complete example is then turned into the proposition part of a TELL (TELL 5 in the trace). Figure 18 shows the conceptual network after pruning and the addition of an example. Note that the exemplified of an EXAMPLE is automatically pruned, as it is always expressed by the expressing the generic response. The pruned and augmented conceptual network is next passed to the generator, described in the next section, which produces the following English output:
76
ROBERT WILENSKY ET AL.
Figure 16. The input to UCE xpress for example “Do you know how to print a file on the imagen?”
Figure 17. Trace of UCE xpress.
THE BERKELEY UNIX CONSULTANT PROJECT
77
Figure 18. The output of UCE xpress for the example “Do you know how to print a file on the imagen?”
Use lpr -Pip. For example, to print the file named foo on the imagen printer, type ‘lpr -Pip foo’. If the user had been judged to be at least a beginner in experience, then the command-format would also have been pruned. This is because KNOME believes that users at the beginner level and up can be assumed to know that part of the command format. In such a case, the entire output would consist of the pruned version of TELL 4, and the generated entire response would be: Use lpr -Pip.
78
ROBERT WILENSKY ET AL.
3.7. The generator After UCExpress formats an answer, the generator, UCGen, converts the conceptual response into text. The current version of UCGen has been customized to work with the types of responses that the system typically produces. It has been built to take advantage of the limited structure of these responses. To convert a KODIAK representation of a concept into text, UCGen must associate some linguistic information with the concept. This is done by attaching to a concept a pattern that represents some linguistic form. For example, a concept often expressed is PLANFOR. This concept relates a plan for achieving a goal with the goal itself. A pattern for PLANFOR is: To (gen goals) comma (gen plan). This pattern might be used to generate the sentence: To delete a file, use rm. This is somewhat akin to the pattern-concept pair construct in PHRED (Jacobs 1984), or to KING’s REF links (Jacobs 1985), although the KODIAK representation accommodates different methods for fetching patterns. Patterns mix words and punctuation with function calls. In the above example, ‘gen’ is a function that will be called with argument ‘goals’ and later with argument ‘plan’. In general, the arguments to functions that are found in generator patterns are the aspectuals associated with the concept to which the pattern is attached. In this example, the aspectuals of PLANFOR, ‘goals’ and ‘plan,’ are arguments to gen. The pattern given above for PLANFOR is the most general one for that concept. That is, it is the pattern used when both the goals and the plan are to be expressed. As described in the previous section on UCExpress, it is not always necessary to express both of these parts. For example, two answers to ‘How do I delete a file?’ are: 1. To delete a file, use rm. 2. Use rm. The expression mechanism puts a flag on each aspectual that it does not want expressed. Consequently, associated with each concept may be zero or more patterns, one for each combination of aspectuals that are to be expressed. PLANFOR is associated with the general pattern shown above, as is the pattern ‘(gen plan)’, which is applicable to the case where only the plan is to be expressed. When a concept to be output is given to the generator, those KODIAK concepts that either dominate or are categories for the concept are searched for one that has an attached generator pattern. If no pattern is found, and
THE BERKELEY UNIX CONSULTANT PROJECT
79
the concept is an aspectual, then the value for the aspectual is sent to the generator. The first pattern found is applied to the concept to be expressed to produce an English sentence. Words in the pattern are output as is. Punctuation and function calls must go through further processing. For example, in the pattern ‘To (gen goals) comma (gen plan)’, the word ‘To’ is output directly, whereas the (gen . . . ) function calls must be evaluated, and the ‘comma’ will be converted to a ‘,’. This generator is easy to understand and extend, and is well integrated with the rest of UC; it shares the KODIAK representation and concepts used by the rest of the system. Some weaknesses are that the overall structure is top down; i.e., only those concepts that are expected to exist are expressed. In general, a generator should be able to handle arbitrary permutations of conceptual relationships. Also, this generator uses little linguistic knowledge. With more complicated utterances, the simple pattern strategies employed so far would become inadequate. This section describes how the output is delivered by UC in response to the question, ‘Do you know how to print a file on the imagen?’ A diagram of some of the relevant knowledge structures is given in Figure 19. A trace produced while generating this output is given in Figure 20. The expression mechanism of UCEgo first passes TELL 4 to the generator. Only the proposition part of the TELL will be expressed, so its value, PLANFOR 70, is passed to the generator’s main routine, ‘gen.’ PLANFOR 70 is dominated by PLANFOR, so the pattern for PLANFOR is retrieved. Since the goals aspectual of PLANFOR 70 is marked to be omitted from the response by the expression mechanism, only the plan will be expressed. The pattern found is ‘(gen plan)’. The value of the plan aspectual, EXECUTE - UNIX IPR - COMMAND 0, is sent to ‘gen’. The pattern for this concept is found under EXECUTE - FILE - COMMAND and is ‘use (gen execute-command)’. The value of execute-command aspectual of EXECUTE - UNIX - IPR - COMMAND 0 is ‘lpr -Pip.’ The first response is therefore: Use lpr -Pip. Next, the generator is passed TELL 5. Once again, only the proposition is to be expressed, so EXAMPLE 0 is to be generated. The pattern, found under EXAMPLE, is ‘for example comma (gen exampler)’. This sets up a recursive call to gen with the value of the exampler relation as argument. This value is PLANFOR 31. Once again, a PLANFOR is to be generated. This time, however, both the plan and goals will be expressed. The pattern is ‘to (gen goals) comma (gen plan)’. The value of the goals aspectual is PRINT- EFFECT 1. The pattern is found under LAS - PR - EFFECT, and is ‘print (las-pr-file-obj) on the (las-prdest-obj)’.
80
ROBERT WILENSKY ET AL.
Figure 19. Knowledge pertaining to generating response to “Do you know how to print a file on the imagen?”
The las-pr-file-obj specification causes the generator to find on object in this relation to PRINT- EFFECT 1, in this case, FILE 0. One of the patterns for FILE is ‘file named (gen name)’. This pattern is applicable if the object in question has a name. Here, FILE 0 has the name ‘foo’. When UCGen is generating the name of an object, it uses a heuristic to decide which, if any article to precede the object description with. In general, UCGen will use definite articles if it can. The other part of the output from the goals aspectual is from the second half of the pattern: ‘on the (las-pr-dest-obj)’. Here the value found is IMAGEN 0. The pattern for IMAGEN 0 is just ‘imagen’. Again, the generator with supplement this description with a definite article. The value of the ‘plan’ aspectual for PLANFOR 31 is TYPE - ACTION 0. The pattern for this concept is from TYPE - ACTION and is ‘type lquote (gen typestring) rquote’. The value for the type-string aspectual of TYPE - ACTION 0 is CONCAT- STR 0. The pattern is from SEQUENCE and is ‘(gen step) (gen next)’. Here, the step is ‘lpr -Pip’ (i.e., the string1 of CONCAT- STR 0), and the next is the name of the file, ‘foo’ (the string2 of CONCAT- STR 0). Thus, the output for this call to the generator is:
THE BERKELEY UNIX CONSULTANT PROJECT
81
Figure 20. UCGen trace for the example question.
For example, to print the file named foo on the imagen, type ‘lpr -Pip foo’.
4. Problems As the preceding sections describe, there are many technical problems yet to be resolved for each component of UC. However, several problems appear to be more pervasive. One general problem is the integration of the components of the system. Control flows unidirectionally through UC. However, there are several cases in which this control structure is unsatisfactory. One such problem is the rela-
82
ROBERT WILENSKY ET AL.
tion of language analysis and inference. We believe it is cognitively correct that these components function concurrently to produce an interpretation of an utterance, whereas in UC they function serially. For example, consider again the process of understanding the sentence we have been using in our extended example: “Do you know how to print a file on the imagen?”. This utterance is syntactically ambiguous in the attachment of the prepositional phrase “on the imagen.” Syntactically, this may modify “you” or “a file” as well as “print.” UC does not deal with this ambiguity, because one of ALANA’s patterns for “print” specifically looks for “on” followed by a device. However, a more elaborate analyzer would probably not include specific information that relates this preposition to the verb, but rather would try to relate them on more general principles. In such a system, the ambiguity would be a more difficult problem. Our current approach is to build such a system and use a marker-passing algorithm (Norvig 1987) to help suggest which syntactic combination to try. For example, our knowledge about printing is such that a path between printing and a device designed for printing should be easy to find. In contrast, there would be a less obvious connection between imagen and file, or imagen and the referent of “you.” This “conceptual closeness” would suggest trying to relate printing and the imagen with a grammatical pattern, so the correct interpretation would be arrived at without other interpretations being tested. Properly done, such a marker-passing scheme would effect concretion as well. For example, to arrive at the connection between printing and the imagen, it is probable that one needs to access the node for “computerprinting.” Thus, it seems that concretion should not be a separate inference process, but one of several kinds of inference that are performed by a markerpassing mechanism. We are currently attempting to reform the analyzer and the inference mechanism in the direction described. It seems that the sort of unidirectional architecture we have employed has drawbacks elsewhere in the system. There are situations in which it seems that one component should be allowed to fail, and the failure be propagated back to another component. For example, consider processing the following query: How can I edit Joe’s file? Initially, the goal analyzer may interpret this request literally. Then the planner may fail, because the file may be protected from just such an action. It seems reasonable, however, for a consultant to suggest copying the file and editing the copy. For this to happen, control must be returned to the goal analyzer, which needs to hypothesize yet another goal underlying the goal it may have suggested initially. We are attempting to design a control structure that accommodates this flow of control.
THE BERKELEY UNIX CONSULTANT PROJECT
83
The concretion mechanism and the goal analyzer also appear to interact in important ways. For example, consider the following example: What does ls -v do? Above we showed the UC can respond appropriately to this question by uttering “There is no -v option to the ls command.” However, the question is problematic because another response to it might be “It lists the contents of the current directory.” This response is possible because, although there is no ‘-v’ option to the ‘ls’ command, it is a characteristic of this command that it ignores options it does not recognize.3 To produce the desired response, the system must recognize that the intent of the question is something like “Tell me the conventional function of the command ls -v,” and not “Tell me what actually happens when we type ls -v.” One way to phrase this is that “conventional function” and “effects occurring from” are two kinds of “doing.” There are certainly other kinds as well. For example, the same form may refer to the steps of a process. Therefore, it would appear to be the job of the concretion mechanism to select the appropriate interpretation. However, it seems that the concretion mechanism cannot choose this interpretation without some knowledge of typical user goals. For example, if a user is debugging a program, it would probably be appropriate to interpret the question as referring to the steps incurred in the process rather than to the process’s purpose. But reasoning about the user’s goals is the job of the goal analyzer, which normally is not invoked until the concretion mechanism has completed its task. The problem is avoided in the current implementation by not allowing for the other, less obvious interpretations at all. However, the example illustrates the need to have more communication between the concretion mechanism and the goal analyzer. Put more strongly, the example suggests that these distinctions between language analyzer, concretion mechanism, and goal analyzer are somewhat artificial. At this stage of our work, it is difficult to determine whether we simply want modules that interact more or a more radical control structure that integrates all these functions. There are several other more specific deficiencies of which we are aware. As we discussed previously, patterns were built into ALANA on an “as needed” basis. We are attempting to produce a more accurate language specification as we develop the inference component. Also, a mechanism for doing ellipsis, which ran in a previous version of UC, has yet to be integrated into this one. Undoubtedly, there are many deficiencies that we have not yet discovered. For example, we recently discovered that asking the same question twice resulted in no answer at all being generated for the second request. The
84
ROBERT WILENSKY ET AL.
problem turned out to be that the user model, after a question is answered, updates its model of the user to show that the user now knows this information. The second time around, this knowledge allowed the expression mechanism to prune away the entire answer, as it inferred the user already knew it. Our approach to fixing this problem is to add another demon that will detect asking for the same thing twice. Then plans for responding appropriately with this situation could be brought to bear. One important deficiency of our current system is that it still doesn’t participate in real conversations. It is our intention that UC function as a consultant and not as a front end to a data base of facts about UNIX. But our current system performs little more than this. Much of the machinery is in place, in UCEgo and PAGAN in particular, to accommodate some conversational situations. We expect much of our further development to be in this direction. Finally, although we have found that our current representation is advantageous, there are many representational issues that remain unresolved. In particular, it is difficult to express certain aspects of quantification in KODIAK. In UC, one often wants to represent facts like “all files have names” or “most directories are not empty.” We are currently working on extending KODIAK to be able to represent such notions in a cognitively plausible way.
Acknowledgements This research was sponsored in part by the Defense Advanced Research Projects Agency (DoD), ARPA order No. 4871, monitored by Space and Naval Warfare Systems Command Command under contract N00039-84-C0089, by the Office of Naval Research under contract N00014-80-C-0732, by the National Science Foundation under grant MCS79-06543, and by the Office of Naval Research under contract N00014-97-1-0578.
Notes 1 UNIX is a trademark of The Open Group. 2 A kind of laser printer used at our site. 3 This “feature” has been changed to produce an error message on some versions of UNIX .
References Allen, J. F. & Perrault, C. R. (1980). Analyzing Intention in Utterances. Artificial Intelligence 15: 143–178.
THE BERKELEY UNIX CONSULTANT PROJECT
85
Allen, J. F., Frisch, A. M. & Litman, D. J. (1982). ARGOT: The Rochester Dialogue System. In Proceedings of The Second National Conference on Artificial Intelligence, 66–70. Pittsburgh, PA: AAAI Press. Austin, J. L. (1962). How To Do Things With Words. Harvard University Press: Cambridge, MA. Brachman, R. & Schmolze, J. (1985). An Overview of the KL-ONE Knowledge Representation System. Cognitive Science 9: 171–216. Brown, J. S. & Burton, R. R. (1976). A Tutoring and Student Modelling Paradigm for Gaming Environments. In Symposium on Computer Science and Education, 236–246. Anaheim, CA. Carberry, S. (1983). Tracking User Goals in an Information-Seeking Environment. In Proceedings of Third National Conference on Artificial Intelligence, 59–63. Washington, DC: AAAI Press. Chin, D. N. (1986). User Modeling in UC, the UNIX Consultant. In Proceedings of CHI ’86. Human Factors in Computing Systems, 24–28. Boston, MA: ACM Press. Chin, D. N. (1988). Exploiting User Expertise in Answer Expression. In Proceedings of The Seventh National Conference on Artificial Intelligence, 756–760. Saint Paul, MN: AAAI Press. Chin, D. N. (1989). KNOME: Modeling What the User Knows in UC. In Kobsa, A. & Wahlster, W. (eds.) User Models in Dialog Systems, 74–107. Springer-Verlag: Berlin. Chin, D. N. (1987). Intelligent Agents as a Basis for Natural Language Interfaces. Ph.D. diss., Computer Science Division, University of California, Berkeley, CA. Also available as UCB/CSD 88/396, Computer Science Division, University of California, Berkeley, CA. Chin, D. N. (1991). Intelligent Interfaces as Agents. In Sullivan, J. W. & Tyler, S. W. (eds.) Intelligent User Interfaces, 177–206. ACM Press: Reading, MA. Chin, D. N. (1998). Intelligent Interfaces as Agents. In Maybury, M. T. & Wahlster, W. (eds.) Readings in Intelligent User Interfaces. R. Morgan-Kaufmann: Los Altos, CA. Chin, D. N. (2000a). Strategies for Expressing Concise, Helpful Answers. In Hegner, S., McKevitt, P., Norvig, P. & Wilensky, R. (eds.) Intelligent Help Systems for UNIX. Kluwer Academic Publishers: Dordrecht, The Netherlands (this volume). Chin, D. N. (2000b). Planning Intelligent Responses in a Natural Language System. In Hegner, S., McKevitt, P., Norvig, P. & Wilensky, R. (eds.) Intelligent Help Systems for UNIX. Kluwer Academic Publishers: Dordrecht, The Netherlands (this volume). Cohen, P. R. & Levesque, H. J. (1987). Persistence, Intention, and Commitment. SRI International Technical Report 415. Cox, C. A. (1986). ALANA: Augmentable LANguage Analyzer. UCB/CSD 86/283, Computer Science Division: University of California, Berkeley, CA. Davis, R. (1976). Applications of Meta Level Knowledge to the Construction, Maintenance and Use of Large Knowledge Bases. STAN-CS-76-552, Computer Science Dept.: Stanford University, Stanford, CA. Deering, M., Faletti, J. & Wilensky, R. (1982). Using the PEARL AI Package. UCB/ERL M82/19, Computer Science Division: University of California, Berkeley, CA. Ernst, G. & Newell, A. (1969). GPS: A Case Study in Generality and Problem Solving. Academic Press: New York. Fikes, R. E. & Nilsson, N. J. (1971). STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving. Artificial Intelligence 2(3–4): 189–208. Grice, H. P. (1975). Logic and conversation. In Cole, P. & Morgan, J. L. (eds.) Studies in Syntax III, 41–58. Seminar Press: New York.
86
ROBERT WILENSKY ET AL.
Grosz, B. & Sidner, C. L. (1985). The Structures of Discourse Structure. CSLI–85–39, Center for the Study of Language and Information: Stanford University, Stanford, CA. Haas, N. & Hendrix, G. (1980). An Approach to Acquiring and Applying Knowledge. In Proceedings of First National Conference on Artificial Intelligence, 235–239. Stanford, CA: AAAI Press. Jacobs, P. S. (1984). PHRED: A Generator for Natural Language Interfaces. UCB/CSD 84/189, Computer Science Division: University of California, Berkeley, CA. Jacobs, P. S. (1985). A Knowledge-Based Approach to Language Production. Ph.D. diss., Computer Science Division, University of California, Berkeley, CA. Also available as UCB/CSD 86/254, Computer Science Division, University of California, Berkeley, CA. Kaplan, S. J. (1983). Cooperative Responses from a Portable Natural Language Database Query System. In Brady, M. & Berwick, R. C. (eds.) Computational Models of Discourse. MIT Press: Cambridge, MA. Lakoff, G. & Johnson, M. (1980). Metaphors We Live By. University of Chicago Press: Chicago, IL. Litman, D. J. & Allen, J. F. (1984). A Plan Recognition Model for Clarification Subdialogues. In Proceedings of Tenth International Conference on Computational Linguistics, 302–311. Palo Alto: MIT Press. Luria, M. (1982). Dividing up the Question Answering Process. In Proceedings of The Second National Conference on Artificial Intelligence, 71–74. Pittsburgh, PA: AAAI Press. Luria, M. (1985). Commonsense Planning in a Consultant System. In Proceedings of 1985 IEEE International Conference on Systems, Man, and Cybernetics, 602–606. Tucson, Arizona: IEEE Press. Luria, M. (1987). Goal Conflict Concerns. In Proceedings of Tenth International Joint Conference on Artificial Intelligence, 1025–1031. Milan, Italy: Morgan Kaufmann Publishers. Luria, M. (1988). Knowledge Intensive Planning. UCB/CSD 88/433, Computer Science Division: University of California, Berkeley, CA. Martin, J. (1985). Knowledge Acquisition through Natural Language Dialog. In Proceedings of Second Annual Conference on Artificial Intelligence Applications, 728–733. Miami, FL: AAAI Press. Martin, J. (1986a). Views from a Kill. In Proceedings of Eighth National Conference of the Cognitive Science Society, 728–733. Amherst, MA. Martin, J. (1986b). Representing and Acquiring Knowledge about Metaphors. In Proceedings of Third Workshop on Theoretical Issues in Conceptual Information Processing, 49–54. Philadelphia, PA. Martin, J. (1987). Understanding New Metaphors. In Proceedings of Tenth International Joint Conference on Artificial Intelligence, 137–139. Milan, Italy: Morgan Kaufmann Publishers. Martin, J. (1988). A Computational Theory of Metaphor. Ph.D. diss., Computer Science Division, University of California, Berkeley, CA. Also available as UCB/CSD 88/495, Computer Science Division, University of California, Berkeley, CA. Martin, J. (2000). Representing UNIX Domain Metaphors. In Hegner, S., Mc Kevitt, P., Norvig, P. & Wilensky, R. (eds.), Intelligent Helps Systems for UNIX. Dordrecht, The Netherlands: Kluwer Academic Publishers (this volume). Mayfield, J. (1989). Goal Analysis: Plan Recognition in Dialogue Systems. Ph.D. diss., Computer Science Division, University of California, Berkeley, CA. Also available as UCB/CSD 89/521, Computer Science Division, University of California, Berkeley, CA. Mayfield, J. (1992). Controlling Inference in Plan Recognition. User Modeling and UserAdapted Interaction 2(1–2): 83–115.
THE BERKELEY UNIX CONSULTANT PROJECT
87
Mayfield, J. (2000). Evaluating Plan Recognition Systems: Three Properties of a Good Explanation. In Hegner, S., Mc Kevitt, P., Norvig, P. & Wilensky, R. (eds.), Intelligent Helps Systems for UNIX. Dordrecht, The Netherlands: Kluwer Academic Publishers (this volume). Mays, E. (1980). Failures in Natural Language Systems: Applications to Data Base Query Systems. In Proceedings of First National Conference on Artificial Intelligence, 327–330. Stanford, CA: AAAI Press. McCoy, K. F. (1983). Correcting Misconceptions: What to Say when the User is Mistaken. In Proceedings of CHI ’83 Conference, 197–201. Boston, MA: ACM Press. Morik, K. & Rollinger, C-R. (1985). The Real Estate Agent – Modeling the User by Uncertain Reasoning. AI Magazine 6(2): 44–52. Newell, A. & Simon, H. A. (1972). Human Problem Solving. Prentice-Hall: Englewood Cliffs, NJ. Norvig, P. (1983). Frame Activated Inferences in a Story Understanding Program. In Proceedings of Eighth International Joint Conference on Artificial Intelligence, 624–626. Karlsruhe, West Germany: Morgan Kaufmann Publishers. Norvig, P. (1987). A Unified Theory of Inference for Text Understanding. Ph.D. diss., Computer Science Division, University of California, Berkeley, CA. Also available as UCB/CSD 87/339, Computer Science Division, University of California, Berkeley, CA. Rich, E. (1979). User Modeling via Stereotypes. Cognitive Science 3: 329–354. Rissland, E. L. (1983). Examples in Legal Reasoning: Legal Hypotheticals. In Proceedings of Eight International Joint Conference on Artificial Intelligence, 90–93. Karlsruhe, West Germany: Morgan Kaufmann Publishers. Rissland, E. L., Valcarce, E. M. & Ashley, K. D. (1984). Explaining and Arguing with Examples. In Proceedings of Fourth National Conference on Artificial Intelligence, 288–294. Austin, TX: AAAI Press. Rosch, E. (1978). Principles of Categorization. In Eleanor Rosch & Lloyd, B. B. (eds.) Cognition and Categorization. Lawrence Erlbaum: Hillsdale, NJ. Sacerdoti, E. (1974). Planning in a Hierarchy of Abstraction Spaces. Artificial Intelligence 5: 115–135. Schank, R. C. (1975). Conceptual Information Processing. North Holland: Amsterdam. Schank, R. C. & Abelson, R. P. (1977). Scripts, Plans, Goals, and Understanding. Lawrence Erlbaum: Hillsdale, NJ. Schmolze, J. G. & Lipkis, T. A. (1983). Classification in the KL-ONE Knowledge Representation System. In Proceedings of Eighth International Joint Conference on Artificial Intelligence, 330-332. Karlsruhe, West Germany: Morgan Kaufman Publishers. Schubert, L. K. (1976). Extending the Expressive Power of Semantic Networks. Artificial Intelligence 7: 163–195. Searle, J. R. (1969). Speech Acts; An Essay in the Philosophy of Language. Cambridge University Press: Cambridge, England. Searle, J. R. (1979). Literal Meaning. In Searle, J. R. (ed.) Expression and Meaning. Cambridge University Press: Cambridge, England. Shapiro, S. (1979). The SNePS Semantic Network Processing System. In Findler, N. (ed.) Associative Networks. Academic Press: New York. Sidner, C. L. (1985). Plan Parsing for Intended Response Recognition in Discourse. Computational Intelligence 1: 1–10. Teitelman, W. et al. (1978). The Interlisp Reference Manual. Xerox PARC.
88
ROBERT WILENSKY ET AL.
van Melle, W. (1980). A Domain Independent System That Aids in Constructing KnowledgeBased Consultation Programs. HPP-80-22, Heuristic Programming Project, Computer Science Department: Stanford University, Stanford, CA. Webber, B. L. & Mays, E. (1983). Varieties of User Misconceptions: Detection and Correction. In Proceedings of Eighth International Joint Conference on Artificial Intelligence, 650– 652. Karlsruhe, West Germany: Morgan Kaufman Publishers. Wilensky, R. (1983). Planning and Understanding: A Computational Approach to Human Reasoning. Addison-Wesley: Reading, MA. Wilensky, R. (1986). Some Problems and Proposals for Knowledge Representation. UCB/CSD 86/294, Computer Science Division: University of California, Berkeley, CA. Wilensky, R. (1987). Primal Content and Actual Content: An Antidote to Literal Meaning. UCB/CSD 87/365, Computer Science Division: University of California, Berkeley, CA. Wilensky, R. & Arens, Y. (1980). A Knowledge-Based Approach to Natural Language Processing. UCB/ERL M80/34, Electronic Research Laboratory: University of California, Berkeley, CA. Wilensky, R., Arens, Y. & Chin, D. N. (1984). Talking to Unix in English: An Overview of UC. Communications of the Association for Computing Machinery 27(6): 574–593.
Artificial Intelligence Review 14: 89–119, 2000. Stephen J. Hegner et al. (eds.), Intelligent help systems for UNIX. © 2000 Kluwer Academic Publishers. Printed in the Netherlands.
89
The OSCON Operating System Consultant* PAUL MC KEVITT Center for PersonKommunikation (CPK), Fredrik Bajers Vej 7-A5, Institute of Electronic Systems (IES), Aalborg University, DK-9220, Aalborg, Denmark. E-mail:
[email protected]
Abstract. OSCON (Operating System CONsultant) is an operating system consultant which gives English answers to English queries about computer operating systems. The program currently answers queries for over 40 commands from the UNIX1 and MS-DOS2 operating systems. OSCON answers a wide variation of queries that users may wish to ask. OSCON is intended to be a consultant for various types of users who may ask vague and detailed queries. OSCON is programmed in Quintus Prolog and answers queries in less than 2.5 seconds. An empirical study with the Wizard-of-Oz technique provides important data for the further development of OSCON. Keywords: intelligent help, Natural Language Processing (NLP), operating systems, UNIX
1. Introduction This paper describes the OSCON (Operating System CONsultant) program and its capabilities. OSCON is a natural language interface which answers English queries about computer operating systems. Although a comprehensive description of OSCON is given here further details are found in Mc Kevitt (1986, 1987, 1988), Mc Kevitt and Wilks (1987), Mc Kevitt and Pan (1989), and Mc Kevitt and Ogden (1989a). The user enters English queries to OSCON and the program answers them in English. The program is written in Quintus Prolog, runs on a Sun-3/ME-4 computer, and the maximum time taken to answer a given query is 2.5 seconds. OSCON answers queries for over 40 commands from the UNIX and MS-DOS operating systems. OSCON has been constructed to answer three basic types of query although recent studies indicate that there are at least nine basic query types (see Mc Kevitt and Ogden 1989a,b). OSCON will also answer queries about options on UNIX commands and complex queries about command compositions. The system is intended to be used by varying types of users with different levels of expertise. The architecture of OSCON is modular so that it is easily updated and can be easily mapped over to other domains.
90
PAUL MC KEVITT
2. Query Coverage of OSCON The problem with building effective natural language interfaces is that there are many ways of asking English queries. The system must attempt to capture all the different possibilities. One way to do this is to try and capture the basic types of queries that people ask. This gives the system the power of answering a large number of queries when it caters for each type. OSCON was designed to cater for three basic types of query. These query types were decided a priori before any data analysis was conducted. These query types are: (1) request for information (e.g. “What does rm do?”), (2) request for explanation (e.g. “What is a pipe?”), and (3) request for direction (e.g. “How do I see my file on the printer?”). Three cases of type (1) are considered important: (1) request for information[effect] (e.g. “What does rm do?”), (2) request for information[form] (e.g. “What is the syntax of cp?”), and (3) request for information[precondition] (e.g. “What is needed for rm?”). Two cases of type (2) were analyzed: (1) request for explanation[command] (e.g. “What is more?”), and (2) request for explanation[concept] (e.g. “What is a file?”), Each of these basic query types can also be asked in terms of options in the UNIX operating system. Examples are, “What option of ‘ls’ shows the number of bytes in my files?” (request for information[option]), “What does ls -l do?” (request for information[effect]), “How do I rename a file without having reported errors?” (request for direction), “What are the options on ls?” (request for information[options]), “What does the -i option normally do?” (request for information[concept]). Users can also ask queries involving command compositions. An example is, “How do I list my files and print them on the printer?” This query involves a query about listing files (request for direction) and then printing them on the printer (request for direction). The OSCON program currently answers queries constituting (1) the three basic query types, (2) queries about options, and (3) command composition queries for both the UNIX and MS-DOS operating systems. The fact that queries are of a given type aids in understanding and generating answers to them. For example, request for information queries usually include a command name. Therefore, the parser for OSCON could check for command names and if it found them, this would be an indication that the query was of type (1). Also, the generator would generate an answer, in a particular format, depending on the type of query. Rules of thumb such as these also speed up the time it takes OSCON to answer queries. Although one can add such rules of thumb into the interface it does not reflect a short-cut to natural language parsing. For example, there is no such short cut to understanding the query, “How do I print a file on the Laserwriter with no page burst?” Understanding queries is a combination of both (1) filtering the query type, and then
THE OSCON OPERATING SYSTEM CONSULTANT
91
(2) understanding the query. Examples of queries answered by OSCON are shown in Appendix A. These examples are listed by query type.
3. The Architecture of OSCON The architecture of the OSCON system is defined as six distinct modules. There are two arguments for modularizing any system: (1) it is much easier to update the system at any point, and (2) it is easier to map the system over to another domain. The six modules in OSCON are as follows: (1) ParseCon: natural language syntactic grammar parser which detects query-type, (2) MeanCon: a natural language semantic grammar which determines query meaning, (3) KnowCon: a knowledge representation for understanding, (4) DataCon: a knowledge representation for solving, (5) SolveCon: a solver for resolving query representations against knowledge base representations, and (6) GenCon: a natural language generator for generating answers in English. ParseCon consists of a set of programs which read natural language input and determine the type of query being asked by the user. We have described the three basic types of query in section 2 above. For each type of query there are tests for characteristic ways of asking that query. MeanCon consists of programs which check queries for important information. There are predicates which check for mentioned (1) command names (e.g. “ls”, “more”), (2) command-effect specifications (e.g. “see a file”), and (3) concepts (e.g. “file”, “directory”). In case (2) there are specific types of information searched for: (1) verb specifying action (e.g. “see”, “remove”), (2) object of action (e.g. “file”), (3) modifier of object (e.g. “contents”), and (4) location of object (e.g. “screen”). MeanCon also checks for option verbs (e.g. “number”) and option verb objects (e.g. “lines”). MeanCon contains a dictionary of English words that define categories such as “person”, “modifier”, “article”, “quantifier” and “prepositions”. KnowCon consists of a set of data files to represent the knowledge about the domain language used for understanding English queries. Files contain information about verbs which categorize types of command or action. Examples of categories of action are: (1) creating, (2) screenlisting, (3) printerlisting, (4) sending, (5) transferring, and (6) removing. KnowCon also contains grammar rules for operating system objects like “date”, “file” and “directory”. The grammar rules encode characteristic ways in which people talk about the objects in English. DataCon consists of a set of data files defining detailed information about operating system commands. This information is stored for the UNIX and MS-DOS operating systems. The data for UNIX is split among seven files: (1) command preconditions, (2) command effects, (3) command syntax, (4)
92
PAUL MC KEVITT
command names, (5) command precondition options, (6) command effect options, and (7) command name options. The first four files contain basic data about commands while the last three contain data for options. For MSDOS, data is contained in just four files which are similar to the first four here. SolveCon is a solver which constructs and matches representations of user queries (called formal queries) against DataCon and produces an instantiated formal query which serves as an answer for the query. SolveCon is the driver of the OSCON program because it contains the information for mapping English sentences into instantiated answers. It contains a set of complex rules which call other OSCON modules to determine (1) query type, and (2) the instantiated formal query for that query. GenCon is the natural language generator for OSCON and maps instantiated information from SolveCon into English answers. Here, there are algorithms for printing out (1) preconditions, (2) effects, and (3) syntax of commands. Also, there are routines for printing out examples of the use of commands and command compositions. The type of query asked by the user determines the information mapped to the user.
4. Knowledge Representation in OSCON One of the problems in building natural language interfaces is to organize the knowledge of the domain in some form which will be effective. There are two types of knowledge stored in OSCON: (1) knowledge about natural language, and (2) knowledge about operating systems. The knowledge about language stored includes words used to refer to command actions. For example, a user may use the words “delete”, “remove”, “get rid of”, “erase” and so on to ask a query about deleting files and directories. These words must be stored under the general category of delete. Also, there are many ways in which people ask queries about actions. For example, if you are asking about copying a file you will probably specify the file which you wish to copy. If you are asking about “displaying” you will specify what you wish to display and where you wish to display it. This type of knowledge is called “understanding knowledge” and is stored within a module of OSCON called KnowCon. More detailed knowledge about operating systems is contained in a database called DataCon. This type of knowledge includes command preconditions, command effects, command syntax, and the names of commands. Also, stored here is (1) knowledge about options for commands, (2) English descriptions of operating system concepts like “files” and “directories”, and (3) knowledge about plans or possible command combinations (e.g. ‘ls’ can
THE OSCON OPERATING SYSTEM CONSULTANT
93
precede ‘lpr’ but the converse is not true). The knowledge stored here is for the UNIX and MS-DOS operating systems. The distinction between the two types of knowledge is that one is language oriented and the other is domain oriented. One type of knowledge is used for understanding queries and the other for solving queries. This is the principle of separation of understanding and solving defined in Hegner (1988, 2000). 4.1. Knowledge for understanding (KnowCon) There are two types of understanding knowledge stored in the KnowCon module: (1) data on operating system action reference, and (2) data on descriptions of operating system objects. The first type of knowledge includes sets of words or phrases that may refer to some operating system action or command. For example, the words, “print”, “print out”, and “get a copy” would indicate that the user was referring to printing something on the printer. Such words and phrases are stored as being associated with the general concept of printing. The second type of knowledge is used for defining the ways that users refer to operating system objects. For convenience the data have been split up into two types (a) data on files and directories, and (b) data on other operating system objects. In type (a) there are grammar rules specifying how users refer to files and directories, and in type (b) there are rules for how users refer to other operating system objects. 4.1.1. Referring to actions In asking queries about operating systems users commonly use a well defined set of verbs or verb phrases. For example, if a user wants to know about removing files or directories he/she will use the following verbs and phrases: “delete”, “remove”, “get rid of”, “erase” and so on. Queries about copying may be referenced by: “copy”, “transfer”, and “move”. These phrases should be captured by the OSCON program. Phrases and words are stored under their categories in the KnowCon module of the program. It is possible to divide the set of operating systems commands into various categories. We have defined three major command categories and each of these have various subdivisions. The three categories are “listing”, “altering” and “compiling” commands. Listing commands are those which display information about the state of files in an operating system. Altering commands are used to alter the state of files in the system. Compiling commands are used to compile files in the system rather than display/alter them. There are two types of listing command, (1) screenlisting, and (2) printerlisting. Screenlisting commands are those which allow the user to see information on the screen and printerlisting will do the same for the printer. There are three subtypes of screenlisting command, (1) display-file (e.g.
94
PAUL MC KEVITT
“more”, “nroff”, “cat”), (2) display-file/directory-information (e.g. “ls”), and (3) display-system-information (e.g. “users”, “who”, “ps”, “jobs”). There is only one subtype of printerlisting command (e.g. “lpr”, “runoff”,3 “itroff”). Altering commands are of three types, (1) creating (e.g. “gemacs”, “vi”, “mkdir”), (2) removing (e.g. “rm”, “rmdir”, “kill”), and (3) transferring (e.g. “mv”, “cp”). Each of these can be applied to either files or directories. A third type of command is compiling commands. These commands cannot be categorized under altering commands as they are really utilities which do not change files but use them. An example of such a command is “run”. It is important to point out that certain commands apply to files which are not directories and others only apply to files which are directories. For example, “more” can only be used on files and not directories and “rmdir” can only be used on directories and not files. Such information can be used to specify the preconditions for commands and enables the system to detect errors in user queries and inform the user about such errors. For example, if a user asked, “How do I use “more” to display a directory?” the system could locate that the precondition for “more” is that it only works over files. Then the system could tell the user this information. The rules listed below are examples of typical action rules in KnowCon (bracketed items indicate input strings to be matched with). Rules [1] and [2] show typical verb phrasings used to reference the action of screenlisting. Rule [3] is for printerlisting and rules [4] and [5] for creating. [1] [2] [3] [4] [5]
screenlist → [see]. screenlist → [look, at]. printerlist → [print]. create → [edit]. create → [produce].
4.1.2. Referring to objects Queries about operating systems often include reference to operating system objects like files and directories. Typically, users will refer to the object which is operated over by some command. The phrasing of the English query will dictate the object present. KnowCon has objects separated into two categories: (1) file/directory object specifications, and (2) other operating system object specifications. The rules listed below are examples of file/directory object specifications (bracketed items indicate input strings to be matched with and non bracketed items indicate non-terminals in the grammar). File object specifications include grammar rule definitions for files and directories. The first rule definition below [1] specifies that a file can be mentioned in a query by the word
THE OSCON OPERATING SYSTEM CONSULTANT
95
“file” preceded by three modifiers. The first modifier can be (1) a quantifier (e.g. “all”, “some of”), and the second (2) a possessive (e.g. “my”, “our”). Then filemod can be a modifier of type of file. Examples are “mail”, “device”, “plain”, “executable” and so on. This grammar rule will capture most of the ways that a user might refer to a file. The second rule [2] shows that a directory may have two modifiers and then the word referring to directory itself. Rule [3] shows that a file or directory could be in reference. This happens because it is ambiguous as to whether the intended referent is file or directory. The user, by using the plural of file may intend directory4 rather than the plural for file. Rule [4] defines location to be a triple: (1) preposition (prep), (2) modifier (mod), and (3) directory (dir). Phrases like “. . . in my directory”, “. . . in our directory”, and “. . . in the directory” will match here. There are more specific definitions of location for where the location is. Rule [5] is a definition of screen location in terms of (1) preposition (prep), (2) modifier (mod), and (3) screen-output (soutput). Preposition and modifier are self explanatory and output is the location of output. This could be “terminal” or “screen”. Rule [6] is the equivalent definition for a printer location. The output specification here is for a printer. Then poutput can be (1) “printer”, (2) “imagen” or (3) “laser writer”. [1] [2] [3] [4] [5] [6]
file → mod, mod, filemod, [file]. directory → mod, mod, [directory]. fileordir → mod, mod, filemod, [files]. location → prep, mod, dir. slocation → prep, mod, soutput. plocation → prep, mod, poutput.
The second type of object definitions include objects other than those to do with files and directories. Some examples are shown below. Rule [1] shows the definition of a queue in terms of (1) a modifier, followed by (2) [printer, queue]. There are definitions of users by rules [2] and [3]. Rules [4] and [5] define names. The definitions of objects are simply definitions of the various ways which users refer to the objects. [1] [2] [3] [4] [5]
queue users users names names
→ → → → →
mod, [printer, queue]. names, mod, [users], prep, mod, [system]. names, [system, users]. mod, [names, of]. [].
96
PAUL MC KEVITT
4.2. Knowledge for solving (DataCon) The knowledge for solving in OSCON consists of files of data that describe detailed information about operating systems. There are four types of knowledge stored here (1) basic command representation, (2) option representation, (3) concept representation, and (4) plan representation. 4.2.1. Basic command representation There are basically four types of information about any command held in the database. These are (1) preconditions, (2) effects or postconditions, (3) syntax, and (4) command names. Preconditions are lists of objects that are necessary for a command to be executed. Here are some examples of preconditions for commands from UNIX. Facts [1] and [2] show that “more” and “cat” have the precondition, “file”. The command, “mkdir” has the precondition “directory” and “cp” has no precondition. [1] [2] [3] [4]
precon(more, [file]). precon(cat, [file]). precon(mkdir, [directory]). precon(cp, []).
Effects, or postconditions, are definitions of the outcome of commands. The effect is defined by a predicate which has a name and three arguments. The predicate name is the action and the arguments are (1) object, (2) object modifier, and (3) location. The facts below show some effects for UNIX commands (‘@’ acts as a slot filler where an extra slot is not required in some cases). Fact [1] shows the effect for the command “more”. The object for “more” is “file” and its modifier “contents”. The location of output of “more” is the “screen”. One case of the “cat” command [2] has the same effect as “more”. The other effect case of “cat” [3] is defined as concatenate and describes the concatenation of files together. The command “ls” will either display directory contents [4], or file information [5] on the screen. The displaying-information command “users” will display usernames on the screen. Fact [7] describes the “gemacs” command which creates files, and fact [8] the “rm” command which deletes them. [1] [2] [3] [4] [5] [6]
comm(more, display(file, contents, screen)). comm(cat, display(file, contents, screen)). comm(cat, concat(file1, file2, file3)). comm(ls, display(directory, contents, screen)). comm(ls, display(file, info, screen)). comm(users, display(usernames, @, screen)).
THE OSCON OPERATING SYSTEM CONSULTANT
[7] [8]
97
comm(gemacs, create(file, @, loc)). comm(rm, remove(directory, @, loc)).
The syntax of commands is defined as a structure which contains the name of some command and then its syntactic definition of use. Shown below are some examples of the syntax for UNIX commands. The syntax facts are three place lists containing (1) Command name, (2) Optionname (filled in from context), and (3) Syntax description. [1] [2] [3] [4] [5]
syn(more, Optionname, ‘‘[more
]’’). syn(cat, Optionname, ‘‘[cat ]’’). syn(ls, Optionname, ‘‘[ls ]’’). syn(users, Optionname, ‘‘[users]’’). syn(gemacs, Optionname, ‘‘[gemacs ]’’).
4.2.2. Option representation DataCon also contains information for option specifications of commands. DataCon has files for (1) Option Preconditions, (2) Option Effects, (3) Option Names. There is no distinction for option syntax as this is practically the same for commands with options or not. Option preconditions are defined as three-place lists with the (1) Command Name, (2) Option Name, and (3) Precondition List. Shown below are a set of options for the various commands. The Precondition List contains the objects which must be present for the command to be executed. Facts [1], [2] and [3] show that each option for “cat” has “file” as precondition. Facts [4] and [5] show that for one option of “cp” (i) there is no precondition, and for the other option (r), “directory” is the precondition. Facts [6] and [7] show the preconditions for “ls”. [1] [2] [3] [4] [5] [6] [7]
opprecon(cat, n, [file]) opprecon(cat, s, [file]) opprecon(cat, v, [file]) opprecon(cp, i, []) opprecon(cp, r, [directory]) opprecon(ls, f, [directory]) opprecon(ls, l, [])
The definition of the Option Effects are also stored as strings of English words. This is merely a convenience for generation. The definitions will enable the generator to give more detail about the specific effect of some option. Each fact here contains (1) Command Name, (2) Option Name, and (3) Option Effect with an English description.
98
PAUL MC KEVITT
[1] [2] [3] [4] [5]
[6]
opeffect(more, #, ‘‘set the window size to # lines’’). opeffect(more, c, ‘‘display each page after cleaning screen’’). opeffect(more, d, ‘‘prompt to hit space to continue at each screen’’). opeffect(cp, i, ‘‘prompt file name when overwriting’’). opeffect(cp, p, ‘‘preserve the modification information from the source’’). opeffect(cp, r, ‘‘copy each subtree rooted at that directory’’).
Option Name definitions are similar in spirit to the Command Effect specifications defined in section 4.2.1 above. However, the definitions here are for specific options. The option definitions have three arguments: (1) Option Effect defined as an action (object) specification, (2) the Command Name, and (3) Option Name. The first three facts [1], [2] and [3] show the option variations on “cat”. The various options allow the displaying of file contents in a specific manner. We also show the options for “ls” (facts [4]– [10]) and for “cp” (facts [11]–[13]). Note that a characteristic of options is that sometimes they have the same action as the main action (e.g. display and display(non-printing-characters)) while other times they have a different action (e.g. display and squeeze(blanklines)). [1] [2] [3]
opcomm(number(lines), cat, n). opcomm(squeeze(blank-lines), cat, s). opcomm(display(non-printing-characters), cat, v).
[4] [5] [6] [7] [8] [9] [10]
opcomm(include(hidden-files), ls, a). opcomm(display(directory-name), ls, d). opcomm(display(directory-content), ls, f). opcomm(display(group-ownership), ls, g). opcomm(display(long-listing), ls, l). opcomm(sort(file-ages), ls, t). opcomm(display(subdirectories), ls, R).
[11] [12] [13]
opcomm(prompt(overwriting), cp, i). opcomm(preserve(modification-information), cp, p). opcomm(copy(subdirectories), cp, r).
THE OSCON OPERATING SYSTEM CONSULTANT
99
4.2.3. Concept representation DataCon also contains a set of definitions of the possible concepts which a user may wish to ask about. Such concepts are considered to be operating system objects like files and directories, and concepts like piping and filtering. Concept representations are three place lists with (1) Concept name, (2) Concept name pluralized, and (3) English description of concept. We show the definitions for, “ada”, “working directory” and “unix” here. [1]
con([ada],[ada], ‘‘Ada is developed on behalf of the U.S. Department of Defense for use in embedded systems. Ada is the first practical language to bring together important features such as data abstraction, multitasking, exception handling, encapsulation and generics.’’).
[2]
con([working, directory],[working, directories], ‘‘The directory you are working in.’’).
[3]
con([unix],[unix], ‘‘UNIX Operating System manages the resources of your computer system to perform useful work on your behalf. It is composed of three major parts: the kernel, the file system, and the shell.’’).
4.2.4. Plan representation DataCon contains a list of possible Plans for command sequences which users may ask about. These command sequences are defined in terms of predicates which have five arguments. The first three arguments represent (1) an action, (2) an object, and (3) location of output. The last two arguments define the second action and its output location. We show two example rules below. Fact [1] represents displaying directories on the screen and then on the printer and fact [2] likewise for files. [1] [2]
plan(display,directory,screen,display,printer). plan(display,file,screen,display,printer).
5. Natural Language Understanding The most important part of any program which acts as a natural language interface is the natural language understander. There must be some means of
100
PAUL MC KEVITT
mapping a user query into a good meaning representation of that query. The problem, of course, is that there are very many ways of specifying queries in English. The first job of the natural language interface is to use some type of syntactic filtering which determines the type of query being asked. The filter will check queries for certain objects or phases and works as a fast mechanism for determining query type. The second job is that of determining the meaning of a query, or its content, once the type of query is determined. Let’s take a look at the two components of OSCON which tackle these two problems. 5.1. Parsing natural language queries (ParseCon) The parser called ParseCon has the job of determining the type of query present. There are characteristic ways of asking certain types of queries and the parser checks for these. The parser checks for the three basic types of query described in section 2. The ParseCon module has lists of the characteristic ways which people use to ask queries about particular query types. One of the query types checked for is request for explanation. There are a number of characteristic phrases which are used. Some of these are listed below. The first set of facts [1–4] and facts [1a–1e] define the syntax of what would come before and after some concept respectively. The second set of facts [6–11] only define possible uses of syntax before the concept but do not reflect any after-concept syntax. [1] [2] [3] [4]
firstphrase([what, does|X], o1). firstphrase([what, a|X],o2). firstphrase([what|X],o3). firstphrase([what|X],o4).
[1a] [2b] [3c] [4d] [5e]
secphrase([mean|X],o1). secphrase([is|X],o2). secphrase([is|X],o3). secphrase([are|X],o3). secphrase([means|X],o4).
[6] [7] [8] [9] [10] [11]
wphrase([what, is, a]). wphrase([what, are]). wphrase([what, is]). wphrase([explain]). wphrase([describe]). wphrase([]).
THE OSCON OPERATING SYSTEM CONSULTANT
101
Another query type checked for is request for information[effect]. Here, again there are a number of possible characteristic phrases. We list some here [1–8]. Similarly, there are facts for (i) request for information[precondition], (ii) request for information[form], (iii) request for direction, (iv) request for explanation[command], and (v) request for explanation[concept] queries. Of course, there are syntax facts which will be compatible for all query types. However, this is not a problem because there are other characteristics of query types which separate them out. [1] [2] [3]
firstdesc([what|X],o1). firstdesc([does|X],o2). firstdesc([does|X],o3).
[1a] [2b] [3c]
secdesc([does|X],o1). secdesc([do|X],o2). secdesc([have|X],o3).
[4] [5] [6] [7] [8]
desc([what, happens, with|X]). desc([result|X]). desc([results|X]). desc([uses, of|X]). desc([effect, of|X]).
5.2. Determining query meaning (MeanCon) The function of the MeanCon component of OSCON is to determine query meaning. This is less of a problem for most query types than for request for direction queries. These queries involve complex phrasings of English to describe the effects which the user wishes to execute. The MeanCon component of OSCON has the function of determining the occurrence of objects in user queries. There are seven types of object searched for: (1) command name, (2) option name, (3) verb reference, (4) object, (5) object modifier, (6) object location, and (7) concept. MeanCon has a predicate called findcmd which searches for command names. The mention of command name is a good indicator of the type of query being asked. If a command name is present this indicates that the query is probably about (1) command preconditions, (2) command effects, or (3) command syntax. MeanCon also has a predicate called findopt which searches for mention of option names in queries. One of the most difficult types of user query to be understood by the system is request for direction queries. These are queries where the user
102
PAUL MC KEVITT
knows what he/she wishes to accomplish but does not know the command to do that. In these cases the user will specify, in English, some process, or effect, which he/she wants to be executed. The process/effect can be described with (1) a primary verb and (2) a secondary verb. The user will use a primary verb to describe the main action/command in question. A secondary verb may be used to describe a restriction on the main action/command. This restriction could be a definition of some option specification for the major command. Therefore, MeanCon has algorithms which check for (i) Primary Verb, (ii) Primary Verb Object, (iii) Object Modifiers, and (iv) Locations. There are also algorithms which search for (i) Secondary Verb, and (ii) Secondary Objects which usually describe option effects. In checking for request for direction queries the user will usually specify some action with a verb. This will be followed by the mention of an object such as a “file” or “directory”. There may be a modifier of the object such as “contents”. The location of the object may also be specified such as “printer” or “screen” or “directory”. There’s a predicate in MeanCon called findverb which searches for verbs in user queries. When a verb is located this will determine the major category of action/command. Therefore, the word “delete” will reference removing, “see” will reference displaying and so on. MeanCon uses the stored verbaction structures in KnowCon to find verbs in queries and their related action representations. Another predicate called findobj searches for the object of a verb. Say, for example, the user had asked, “How do I see a file?” then the findverb predicate will locate the verb “see”. The query is split up so that the phrase after the verb is checked for an object. The object “file” is located and marked. More complex queries may include modifiers and locations. Take the query, “How do I see my file contents on the screen?” In this case, the phrase “|. . . file contents on the screen?” is checked for an object by findobj. Then, “|. . . contents on the screen?” is checked for modifier by findmod. Findmod locates “contents” as a modifier. A predicate called findloc checks for locations in queries. Locations include “screen”, “printer” and “directory”. In our example, “|. . . on the screen?” is checked for location and “screen” is uncovered. MeanCon has a predicate called findcon which checks for concepts in user queries. For example, the existence for the concept, “ada” will denote a query asking about, “ada”.
THE OSCON OPERATING SYSTEM CONSULTANT
103
6. Rules for Solving (SolveCon) Now that we have defined the data available to OSCON before the system begins to process a query we move on to describe the rules which match user queries to database information. SolveCon determines the answer for a query by (1) building an uninstantiated formal query from the query, (2) matching this structure to the DataCon database, (3) retrieving data from the database, and (4) using the data to build an instantiated formal query which is passed back to the GenCon natural language generator. We shall describe the answering process in two steps: (1) the algorithm used by SolveCon to specify the query and to retrieve data from the database, and (2) the structure of instantiated Formal Queries returned by SolveCon to the GenCon generator. 6.1. The solving algorithm The Solver basically searches queries for three types of information: (1) Command Names, (2) English Descriptions of command effects, and (3) Concepts. The search process in conducted in the following order. [1] SolveCon checks to see if a command name is mentioned in the query. Then, (a) SolveCon checks if an option (e.g. -l) is mentioned. If (a) fails then (b) SolveCon checks if the query is request for direction[option]. This check is done by having ParseCon check the syntax of the query, having MeanCon check for an English Description of an option effect. If either (a) or (b) are satisfied SolveCon will retrieve from the database Option Preconditions, Option Effect, Option Syntax, and Option Name. If (a) and (b) have both failed then (c) SolveCon checks if the query is a request for information[precondition], request for information[effect] or request for information[form] query. Here, SolveCon checks the syntax again using ParseCon. If (c) fails, then (d) SolveCon checks the query for request for explanation [command] query. ParseCon is involved here too. If either (c) or (d) are satisfied SolveCon will retrieve Command Preconditions, Command Effect, Command Syntax, and Command Name from the database. If (d) fails then SolveCon moves on to step [2]. [2] SolveCon checks the query semantics. In this case the user must have asked an English query with no command names. (i) SolveCon has ParseCon check the syntax of the query. (ii) Then, SolveCon calls MeanCon to check for a Primary Verb, Verb Object, Modifier, and Location. SolveCon will retrieve Command Preconditions, Command Effect, Command Syntax, and Command Name from the database. Next, (iii) SolveCon has MeanCon check for a Secondary Verb (option action), and Secondary Verb Object. SolveCon will retrieve from the database Option Preconditions, Option Effect, and Option Syntax. If step [2] fails then SolveCon goes on to step [3]).
104
PAUL MC KEVITT
[3] SolveCon checks the query semantics. In this case the user must have asked an English query involving no command names. Also, the query must be about command combinations, or pipes, otherwise step [2] would have passed. SolveCon checks for the existence of a command combination in the user query. SolveCon has MeanCon check for the existence of a sentence connector like “and”. If this occurs then is it possible that the query involves command combination. SolveCon then calls the SolveCon algorithm again for (1) the piece of the query before the connector, and (2) the piece of the query after the connector. The data returned from (1) and (2) is integrated. If [3] fails then SolveCon tries step [4]. [4] SolveCon checks query syntax through ParseCon. Then MeanCon searches for concepts mentioned in the query. Examples of such concepts are “ada” and “protection”. 6.2. Structures returned The step of SolveCon which succeeds will return an instantiated formal query to the generator. In step [1], if cases (a) or (b) succeed, an instantiated formal query will be returned containing the following: (1) Option Preconditions, (2) Option Effect, (3) Option Syntax, (4) Option Name, and (5) Query Type. In step [1], if cases (c) or (d) succeed, the instantiated formal query contains: (1) Command Preconditions, (2) Command Effect, (3) Command Syntax, (4) Command Name, and (5) Query Type. In step [2] the formal query returned will contain slots for: (1) Command Preconditions, (2) Command Effect, (3) Command Syntax, (4) Option Preconditions, (5) Option Effect, (6) Option Syntax, and (7) Query Type. The complete structure will be instantiated when step [2] involves options. However, only parts (1), (2), (3), and (7) are instantiated when there is no mention of options. With step [3] a list containing two instantiated Formal Queries is returned. Each formal query will contain: (1) Command Preconditions, (2) Command Effect, (3) Command Syntax, (4) Command Name, and (5) Query Type. In step [4] a formal query with three pieces of information is returned. The structure contains (1) the Concept Name, (2) Concept Description, and (3) Query Type.
7. Natural Language Generation (GenCon) The final phase of the OSCON program is to map an instantiated formal representation into an English answer. There are two types of answer which may be returned to the user: (1) Stored English sentences describing some
THE OSCON OPERATING SYSTEM CONSULTANT
105
concept which are contained in the DataCon knowledge base, and (2) English sentences mapped out from instantiated Formal Queries. The natural language generator for the OSCON system is used to map instantiated formal queries into English answers. The generator has five primary components: [1] [2] [3] [4] [5]
psyntax: gives the syntax for a command peffect: gives the effect of some command pexample: gives an example on the use of some command ppre: gives the preconditions for some command ppipe: gives the commands involved in some piping example and an example of the piping
For each of the major query types various configurations of printing components are used. There are three types of request for information query: (1) request for information[precondition], (2) request for information[effect], and (3) request for information[form]. In these cases the components [1], [2], [3] and [4] are combined in order. Printing out the syntax for some command is trivial. The syntax is already stored in the DataCon knowledge base. This is just returned to the user. Printing out preconditions is quite trivial too as all GenCon has to do is to print those preconditions retrieved from the DataCon precondition information. Generation for request for information[effect] queries is more complex. GenCon will print command effects by (1) checking to see if the output should be in plan/pipe form, and if it is, then generating the answer in plan/pipe form; (2) generating the (a) Command Syntax, (b) Effect and (c) Preconditions for the command. The Effect is generated from the instantiated Formal Query produced by SolveCon which contains action, object, object modifier and object location. The latter information is generated in sentence form. Some interleaving information such as the output of prepositions between object modifier and location are handled too. For request for direction the latter algorithm is used. For request for explanation[command] components [1], [2], [3] and [4] are used. For request for explanation[concept], the answer is output from a stored piece of text. Often users ask queries about commands as concepts. The generation of these is simple as the definitions of such concepts are just stored as English descriptions in the first place. Therefore, all GenCon has to do is to map the stored sentences into English answers. We have written a simple algorithm which maps the English text into pretty format on the screen.
106
PAUL MC KEVITT
8. An Example In this section we show an example of how the query “How do I see my files with numbered lines?” is understood and answered by OSCON. First, SolveCon attempts to find out the type of query being asked. Initially, SolveCon tries step [1] to match the query as one mentioning a command and there is no match. Then step [2] is tried and a match occurs. The query is a semantic one. ParseCon is called forth and a syntactic match is found. “How do I” matches the syntactic form [how, do, i] for request for direction queries. Next, SolveCon calls MeanCon which analyzes the meaning of the sentence. (i) Findverb checks for a verb and gets “see”. From KnowCon, “display” is marked as the action. Then, (ii) findobj checks “How do I” and “my files with numbered lines” for objects. “Files” is matched as an object. Data from the query, i.e. display(file,contents,*) is matched against the DataCon database Effects and a match is found with display(file,contents,screen) This effect match from the query data to DataCon will allow SolveCon to retrieve (1) Command Preconditions ([file]), (2) Command Effect (display (file, contents, screen)), (3) Command Syntax (cat -n ), and (4) Command Name (cat). Next, the query is checked for existence of a secondary action. The findverb predicate retrieves “numbered” as a secondary action and its object is retrieved as “lines”. This representation is matched against the option database and number(lines) from the query matches number(lines) in the Option Effect definition in the database. The DataCon database is referenced and (1) Option Preconditions, (2) Option Effect, and (3) Option Name are returned. The data retrieved are integrated to form an instantiated formal query which is passed to the generator. The representation for this query is as shown below. cquery(display(file,contents,screen), ‘‘with numbered lines’’, [cat -n ], [file]), cat), -n).
Command Effect Option Effect Syntax Preconditions Command Name Option Name
The GenCon generator takes the formal query and maps it into an English answer: (i) psyntax will print out “cat -n ”, (ii) peffect will
THE OSCON OPERATING SYSTEM CONSULTANT
107
display “will display file contents on the screen.”, and (iii) popeffect will display “with numbered lines.” Therefore, the complete answer generated for this query is: ‘cat -n ’ will display file contents on the screen with numbered lines.
9. Current State of the Implementation The OSCON program can answer three major query types and replies to each in seconds: (1) request for information, (2) request for explanation, and (3) request for direction. There are three cases of type (1): request for information[precondition] request for information[effect], and (3) request for information[form] and two cases of type (2): request for explanation[command], request for explanation[concept]. In real terms this brings the query types covered up to six. The system answers the three query types with options in the case of UNIX. OSCON’s database contains information on 40 UNIX and MS-DOS commands with their respective preconditions, effects, syntax and command names. OSCON has 20 grammar rules for understanding the ways that users ask queries about categories of commands. OSCON also contains 10 plan sets for possible combinations of commands. 10. Comparison to Other Systems There are basically three other natural language consultants for operating systems. These are the Unix Consultant (UC), the Sinix Consultant (SC), and UCC/Yucca. The Unix Consultant (UC) (see Chin 1988; Wilensky et al. 1984, 1986, 1988) is a natural language consultation system for UNIX, and is in many ways similar in scope and intent to OSCON. However, there are some key differences. While OSCON is a specialized system with the sole goal of providing detailed expert help, UC has the broader goal of studying knowledge representation and planning (as well as natural language understanding) at a basic level. As such, UC takes a fundamentally different approach to knowledge representation, in that employs a much more general knowledge representation and planning mechanism, with a somewhat cognitive flavor. UC contains only one knowledge base of operating system concepts which must serve both the understanding and the solving phases of query resolution, while OSCON incorporates specialized representations for each function. Hence, UC must sacrifice some of the specialization built into OSCON.
108
PAUL MC KEVITT
Not unexpectedly, an examination of the examples presented in Wilensky et al. (1986) suggests that UC will have to perform far more extensive procedures to solve a given query, and that it may not be able to answer some of the very detailed queries that OSCON will address. UC is not intended to be a consultant which will help users migrating from one system to another. UC is a consultant for the UNIX operating system. We do not claim that the UC system could not act as a consultant on other operating systems. We claim that the Berkeley team has not concerned itself with putting information from more than one operating system in their program. It is a characteristic of computer operating systems that commands can be combined in various ways to execute complex processes. In Wilensky et al. (1986) (pp. 6–10) there are a number of examples of the queries which the UC system handles. These examples are intended to show the kinds and scope of requests answered by the system. None of the examples involve command compositions but operations of single commands. UC has no demonstrated ability for answering complex natural language queries which involve command sequencing. UC and OSCON are not truly competitors, but rather systems with different emphases. UCC is another natural language UNIX help system (see Douglass and Hegner 1982) UCC was a prototype system, implemented in Franz Lisp on a VAX-11/780. It used a simple natural language front end based on augmented transition networks. The development of a knowledge base and query solver were not advanced enough so that they could be linked with the front end. UCC generated answers to queries directly from concept case frames rather than from any particular formal language. There were two major problems with UCC: (1) as the front end included a relatively simple knowledge base it was unable to answer sophisticated queries with many constraints involving command options, and (2) the simple augmented transition network method of parsing was not sufficient enough to handle the types of queries posed by many users. A program called Yucca (Hegner and Douglass 1984) was an attempt to augment the UCC system in two ways. Yucca incorporated a much more sophisticated formal knowledge base and an improved natural language front end. OSCON’s design is based very much on principles from UCC and Yucca. The Sinix Consultant (SC) (see Hecking et al. 1988; Kemke 1986; and Kemke 1987) is a broad-based UNIX help system. It is similar to OSCON in that it is designed for the operating systems domain. Although SC contains a rich knowledge base, which reflects the technical aspects of the domain as well as the users view of the system, the focus of SC, like UC, is to provide help on the use of individual commands, rather than any interconnection of several. There appears to be no mechanism whereby the SC system
THE OSCON OPERATING SYSTEM CONSULTANT
109
will understand complex command sequences. However, it is predicted in Hecking et al. (1988) that future development of the SINIX Knowledge Base will include “combinators” for I/O-redirection and pipelining. There are several other consultation systems for operating systems, including CMS-HELP developed by Yun and Loeb (1984), TVX by Billmers and Carifio (1985), Wizard (see Shrager and Finin 1982; and Finin 1983), and USCSH by Matthews et al. (2000). Selfridge (1988) takes an expert system approach to knowledge representation. However, all of these systems appear to be either far less ambitious in scope than OSCON, or else have a totally different emphasis. In particular, all appear to employ simpler models of UNIX.
11. Conclusion and Further Work It is concluded that it is possible to build a computer program which will answer natural language user queries about operating systems. We have shown how this is done by the OSCON system. The OSCON program answers English queries for over 40 commands. Each query is answered within seconds. The development of any program like OSCON is enhanced if the program is built in a modular form where each module has a distinct function. This has been done for OSCON which is divided up into six distinct modules. This methodology allows for easy update of the program and also will allow the possibility of mapping the program over to a new domain. In comparing OSCON to the other two most related systems available today (UC, SC) OSCON turns out to have a different emphasis. The OSCON system is more concerned with the detail of operating systems and natural language processing rather than an experiment on cognitive modeling. Also, OSCON answers queries involving command combinations. Recently, some Wizard-of-Oz studies have been completed for the operating systems domain (see Mc Kevitt and Ogden 1989a). The Wizard-of-Oz technique is one where subjects interact with a computer through typed dialogue and are told that they are conversing with the computer. Subjects utterances are sent to another monitor where a “Wizard” sends back a reply to the subject monitor. Data were collected for 14 subjects who were naive UNIX users. Initial analyses of results look interesting. Seven types of queries have been discovered as opposed to the three types that were defined above a priori. Also, it has been noted that query type is strongly related to levels of user expertise. These data show that certain query types which were not decided a priori would not be handled by OSCON. Also, these query types would cause a major breakdown in communication in the natural language
110
PAUL MC KEVITT
dialogue. Further analyses of this data will be done to help determine the relationship between dialogue models and user models. There are three areas of further work proposed for OSCON which are all part of a dialogue interface to be added to the system. These are: (1) context storage mechanisms, (2) reference determination algorithms, and (3) user modeling capabilities (see Chin 1988). Dialogue management is important and it will permit the user to ask queries without having to spell the queries out in elaborate English. Initial thoughts on this research are reported in Ball et al. (1989). 12. Recent Developments There have been a number of developments with respect to the OSCON system since 1987. Analysis of Wizard-of-Oz data (see Mc Kevitt 1991; Mc Kevitt and Ogden 1989a,b) from natural language dialogues on UNIX gave nine query/intention5 types (see Table 1 below). The example UNIX queries in Table 1 are shown as they were actually typed by users. Our Wizard-of-Oz empirical data showed that expert subjects have mainly information and description intentions whereas less expert subjects have much more of the other intentions such as elaboration, explanation and repetition. Also, the data showed that there were statistically significant differences in the frequencies of intention types between expert and nonexpert subjects. Based on this grouping of intentions, and the fact that some intention sequences indicate less satisfaction than others, we proposed a weighted formula for a measure of satisfaction as follows (where [X → Y] represents the frequency of intention pairs from intention X to intention Y; ‘I’ denotes information intentions, ‘De’ denotes description intentions, etc.): 3([I → I ]) + [I → De] + [I → El] + [I → Ex] + [I → Re] +[De → I ] + [El → I ] + [Ex → I ] + [Re → I ] +3([De → De]) + [De → El] + [De → Ex] + [De → Re] +[El → De] + [Ex → De] + [Re → De] Relative dissatisfaction can be calculated by summing dissatisfaction intention pairs. The formula for a measure of dissatisfaction is: 3([El → El] + [Ex → Ex] + [Re → Re]) +2([Ex → El] + [Re → El] + [Re → Ex] + [El → Ex] +[El → Re] + [Ex → Re]) Two new components have been added to OSCON to include the capability of dialogue modelling and user modelling: DialCon and UCon. DialCon
THE OSCON OPERATING SYSTEM CONSULTANT
111
Table 1. Definitions of principal intentions for the consultancy domain Intention
General definition
UNIX domain definition
Information
An intention requesting a PLAN to achieve a specific GOAL where the GOAL is described.* E.g. “How do I cook this dish?”
An intention requesting a UNIX command to achieve a UNIX operation where the operation is described. E.g. “How do I print a file?”
Description
An intention requesting a description of an object or concept. E.g. “What is Persia?”
An intention requesting the description of UNIX concepts, objects, or commands. E.g. “what is UNIX?”
Instruction
An intention acting as an instruction to achieve a GOAL rather than the PLAN to achieve that GOAL. E.g. “Can you find out how many foreign nationals now live in Kuwait?”
An intention requesting the execution of a UNIX command. E.g. “has oscon been printed?”
Elaboration
An intention requesting more information on a PLAN or GOAL. E.g. “Could you tell me more about Iraq?” following “Where is Iraq?”
An intention requesting more information on UNIX commands, or UNIX itself. E.g. “how do i use more?” following “how do i see my file?”
Confirmation
An intention requesting confirmation of a belief or some PLAN believed to execute some GOAL. E.g. “Will sanctions stop Saddam Hussain?”
An intention requesting confirmation of a belief about the function of commands in UNIX, or the function of UNIX itself. E.g. “can i remove a directory with files in it?”
Explanation
An intention requesting explanation or clarification of an item which occurred during the execution of a PLAN for a GOAL. E.g. “Could you tell me what you mean by U.N. resolution 611?”
An intention requesting explanation of a response from the UNIX shell. E.g. “What does cp -r mean?”
Guidance
An intention requesting a PLAN for a GOAL where there is no explicit GOAL expressed. E.g. “What do I do next?”
An intention requesting help with UNIX operations, or UNIX, where there is no operation described. E.g. “I don’t understand what i’m supposed to do.”
Repetition
An intention which is a repeated request. E.g. “How many people live in the Gulf?” followed by “What number of people live in the Gulf?”
An intention repeating another intention. E.g. “How do I print a file” followed by “How do I get a print out of my file?”
Nointention
An intention which is not immediately relevant to the domain or not understood by the hearer as being relevant to the domain. E.g. “Where does Strider live?” in the domain of Economics.
An intention not understood by the system. E.g. “How do I eat a file?”
* A PLAN is defined as a set of actions to achieve some GOAL. A GOAL is defined as an operation a speaker wishes to achieve.
112
PAUL MC KEVITT
is a dialogue modelling component which uses to track query/intention sequences in dialogue. It updates an intention matrix by locating the relevant cell in the matrix representing a given intention sequence and increasing its count. DialCon indexes the cell in the matrix by pairing the current query/intention type with the previous. UCon is a user modelling component which computes the level of user satisfaction from the intention matrix and provides information for context sensitive and user sensitive natural language generation. UCon derives a binary measure of user expertise, expert and novice. UCon applies a usermodelling function, based on the weighted formulas above, to the intention matrix to determine levels of user satisfaction and dissatisfaction. Initially, the user is assumed to be an expert. Subsequent changes in the levels of satisfaction and dissatisfaction will result in changes in the level of user expertise. Such information is used by GenCon to generate context-sensitive and usersensitive natural language responses. OSCON’s capability of dialogue and user modelling is demonstrated in Appendix B. Also, we have developed a version of OSCON which acts as a command interface where after a user asks a given query then the system asks if he/she would like the resultant UNIX command executed (see Lim 1994). If the user replies “y” then the results of the execution of the command are reported.
Acknowledgements Acknowledgements are due to Louise Guthrie and Zhaoxin Pan for programming parts of OSCON. Stephen Hegner is thanked for many stimulating discussions. Yorick Wilks and the natural language processing team at the Computing Research Laboratory (CRL), New Mexico State University, USA are thanked for numerous comments on this work. Hans Brunner, Andy Parng and Scott Wolff of the Intelligent Customer Assistance Project at U S WEST Advanced Technologies are thanked for consultations on this research. This research was funded in part by U S WEST Advanced Technologies, Denver, Colorado under their Sponsored Research Program. Peter Norvig, Wolfgang Wahlster, and Robert Wilensky are thanked for comments on an earlier draft of this paper and Stephen Hegner for comments on a later draft.
Notes * This paper is effectively that already published as “Principles and practice in an operating system consultant”, in “Artificial Intelligence and Software Engineering (Vol. 1)”, Section on “AI Mechanisms and techniques in practical software”, Derek Partridge (Ed.), 1991, 439–462.
THE OSCON OPERATING SYSTEM CONSULTANT
113
New York: Ablex Publishing Corporation. Copyright ©1991 Intellect Books (originally to Ablex Publishing Corporation). 1 UNIX is a trademark of X/Open, Inc. 2 MS-DOS is a trademark of Microsoft Corporation. 3 “Runoff” is a command defined at the Computing Research Laboratory, New Mexico State University, USA to load text formatting packages for word processing a text file. 4 A directory is a set of files. 5 We have reclassified query types into nine more general intention types.
Appendix A This appendix contains a list of some of the queries which OSCON currently answers. The maximum time taken to answer any query is 2.5 seconds. The query sets have been divided into three major types (1) request for information, (2) request for explanation, and (3) request for direction. | ?- oscon. OSCON Program 1.0 (Sun-3, Sun Unix 4.2) Copyright (C) 1988, Computing Research Laboratory. All rights reserved. Dept. 3CRL, Box 30001, NMSU, Las Cruces, NM (505) 646--5466 U S WEST Advances Technology Please input your question ending with with a ‘?’ Write ‘quit.’ when you are finished ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; request-for-information[effect] → what does ls do? ‘ls <$directoryname>’ will display directory contents on the screen. → what happens with rm? ‘rm ’ will remove a file from your current directory. → what is the effect of date? ‘date’ will display a date on the screen. → what does more -c do? ‘more -c’ will display each page after clearing the screen. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; request-for-information[form] → what is the syntax of rm? It has the syntax ‘rm ’.
114
PAUL MC KEVITT
→ How do i use cp? It has the syntax ‘cp ’. → how does he use more? It has the syntax ‘more ’. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; request-for-explanation[command] → what is ls? ‘ls’ is a command. It has the syntax ‘ls ’. ‘ls <$directoryname>’ will display directory contents on the screen. → what is more? ‘more’ is a command. It has the syntax ‘more ’. ‘more ’ will display file contents on the screen. ‘more’ only takes the argument ‘file’ ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; request-for-explanation[concept] → what are aliases? alias is the ability to establish shorthand names for frequently used but long-winded commands. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; request-for-direction → how do i see my file? ‘more ’ will display file contents on the screen. → how do i see my file on the screen? ‘more ’ will display file contents on the screen. → how do i see my file on the printer? ‘lpr ’ will display file contents on the printer. → how do i see all my files? ‘more ’ will display file contents on the screen. ‘ls ’ will display directory contents on the screen. → how do i see this file? ‘more ’ will display file contents on the screen.
THE OSCON OPERATING SYSTEM CONSULTANT
115
→ how do i see that file? ‘more ’ will display file contents on the screen. → how do i see his files? ‘more ’ will display file contents on the screen. ‘ls ’ will display directory contents on the screen. → how do i read my files? ‘more ’ will display file contents on the screen. ‘ls ’ will display directory contents on the screen. ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; request-for-direction → how do i list my files and print my files? ‘more ’ will display file contents on the screen. ‘lpr ’ will display file contents on the printer. To display file contents on the printer use ‘more | lpr’ → quit. over yes | ?-
Appendix B This Appendix demonstrates that the OSCON system modifys its natural language responses dynamically based on the types of utterances, and hence intention, that appear in the input. | ?- oscon. OSCON Program 1.0 (Sun-4, SunOS Release 4.1) Copyright (C) 1988, Computing Research Laboratory. All rights reserved. Dept. 3CRL, Box 30001, NMSU, Las Cruces, NM (505) 646-5466, USA. U S WEST Advances Technology Please input your question ending with with a ‘?’ Write ‘quit.’ when you are finished OSCON can answer queries for UNIX or MSDOS. Type one of these to the prompt. Do you want answers for UNIX or MSDOS? The default is UNIX [unix]: unix
116
PAUL MC KEVITT
How can I help you? → what does ls do? ‘ls ’ will display directory contents on the screen. information description elaboration explanation repetition
: : : : :
0 0 0 0 0
0 0 0 0 0
0 0 0 0 0
0 0 0 0 0
0 0 0 0 0
0 0 0 0 0
= = = = =
0 0 0 0 0
nointention : 0
0
0
0
0
0
=
0
Satisfaction = 0 Dissatisfaction = 0
== 0
OSCON displays the ‘intention matrix’ and levels of ‘satisfaction’ and ‘dissatisfaction’. Note that OSCON caters for 5 ‘real’ types of intention, and nointentions. The ‘intention matrix’ is a 2-dimensional, 6 X 6 matrix. The ‘intention matrix’ shows all intention-sequence counts to be 0 as only one intention has been entered into the program. Totals for each intention type are also shown. ‘Satisfaction’ and ‘dissatisfaction’ levels, computed from the matrix, using the user modelling function, are also 0. → how do i see my file? ‘more ’ will display file contents on the screen. information description elaboration explanation repetition
: : : : :
0 0 0 0 0
0 0 0 0 0
0 0 0 0 0
1 0 0 0 0
0 0 0 0 0
0 0 0 0 0
= = = = =
1 0 0 0 0
nointention : 0
0
0
0
0
0
=
0
Satisfaction = 1 Dissatisfaction = 0
== 1
This ‘intention matrix’ shows a total intention sequence count of 1, and the intention count for the sequence ‘explanation → information’ is set to 1. ‘Satisfaction’ also totals to 1.
THE OSCON OPERATING SYSTEM CONSULTANT
117
→ how does he see my file? ‘more ’ will display file contents on the screen. information description elaboration explanation repetition
: : : : :
0 0 0 0 1
0 0 0 0 0
0 0 0 0 0
1 0 0 0 0
0 0 0 0 0
0 0 0 0 0
= = = = =
1 0 0 0 1
nointention : 0
0
0
0
0
0
=
0
Satisfaction = 2 Dissatisfaction = 0
== 2
This ‘intention matrix’ shows a total sequence count of 2. Now, a ‘information → repetition’ sequence has occurred. Note that the syntax of this query does not have to be exactly the same as the previous query to denote a repetition. ‘Satisfaction’ increases to 2. → how do they see my file? ‘more’ is a command. It has the syntax ‘more ’. ‘more ’ will display file contents on the screen. ‘more’ only takes the argument ‘file’. information description elaboration explanation repetition
: : : : :
0 0 0 0 1
0 0 0 0 0
0 0 0 0 0
1 0 0 0 0
0 0 0 0 1
0 0 0 0 0
= = = = =
1 0 0 0 2
nointention : 0
0
0
0
0
0
=
0
Satisfaction = 2 Dissatisfaction = 3
== 3
A ‘repetition → repetition’ intention loop has occurred tilting the level of ‘dissatisfaction’ higher than ‘satisfaction’. As a reaction, more information is returned to the user. Remember that the user modelling function gives intention repetitions which are along diagonals a weight of 3.
118
PAUL MC KEVITT
References Ball, Jerry, Barnden, John A., de Ram, Sylvia Candelaria, Farwell, David, Guthrie, Louise, Guo, Cheng-Ming, Helmreich, Stephen, Mc Kevitt, Paul & Liu, Min (1989). The Need for Belief Modelling in Natural Language Processing. In Proc. of the International Conference on Cross-Cultural Communication (ICC-CC-89). San Antonio, Texas: Trinity University, March. Billmers, Meyer A. & Carifio, Michael G. (1985). Building Knowledge-based Operating System Consultants. In Proceedings of the Second Conference on Artificial Intelligence Applications, 449–454. Miami Beach, December. Chin, David (1988). Exploiting User Expertise in Answer Expression. In Proceedings of the Seventh National American Conference on Artificial Intelligence (AAAI-88), Vol. 2, 756– 760. Minnesota: St. Paul, August. Douglass, R. & Hegner, Stephen J. (1982). An Expert Consultant for the UNIX Operating System: Bridging the Gap between the User and Command Language Semantics. In Proc. of the Fourth National Conference of the Canadian Society for Computational Studies of Intelligence (CSCSI/SCEIO), 119–127. Saskatchewan, Saskatoon, May. Finin, Timothy W. (1983). Providing Help and Advice in Task Oriented Systems. In Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83) , 176–178. Germany: Karlsruhe. Hecking, M. C., Nessen, Kemke, E., Dengler, D., Gutmann, M. & Hector, G. (1988). The SINIX Consultant – A Progress Report. Memo Nr. 28, Universitat des Saarlandes, FB 10 Informatik IV, Im Stanwald 15, D-6600 Saarbrucken 11, Germany, August. Hegner, Stephen J. (1988). Representation of Command Language Behavior for an Operating System Consultation Facility. In Proc. of the Fourth IEEE Conference on Artificial Intelligence Applications, 50–55. San Diego, CA: IEEE Computer Society. Hegner, Stephen J. (1998). Plan Realization for Complex Command Intercation in the UNIX Help Domain. In Hegner, Stephen, Mc Kevitt, Paul, Norvig, Peter & Wilensky, Robert (eds.) Intelligent Help Systems for UNIX. Dordrecht, The Netherlands: Kluwer Academic Publishers (this volume). Hegner, Stephen J. & Douglass, Robert J. (1984). Knowledge Base Design for an Operating System Expert Consultant. In Proc. of the Fifth National Conference of the Canadian Society for Computational Studies of Intelligence (CSCSI/SCEIO), 159–161. London, Ontario, May. Kemke, Christel (1986). The SINIX Consultant – Requirements, Design, and Implementation of an Intelligent Help System for a UNIX Derivative. Universitat des Saarlandes, KI-Labor (SC-Project), Bericht Nr. 11, October. Kemke, Christel (1987). Representation of Domain Knowledge in an Intelligent Help System. In Bullinger, H. J. and Shakel, B. (eds.) Human-Computer Interaction – INTERACT ’87, 215–220. Amsterdam: Elsevier Science Publications B.V. (North-Holland). Lim, Hun Keong (1994). Translation of OSCON into a Command Interface. Master’s Thesis, Department of Computer Science, University of Sheffield, Sheffield, England. Matthews, M., Pharr, W., Biswas. G. & Neelakandan, H. (2000). USCSH: An Active Intelligent Assistance System. In Hegner, S., Mc Kevitt, P., Norvig, P. & Wilensky, R. (eds.) Intelligent Helps Systems for UNIX. Dordrecht, The Netherlands: Kluwer Academic Publishers (this volume). Mc Kevitt, Paul (1986). Formalization in an English Interface to a UNIX Database. Memoranda in Computer and Cognitive Science, MCCS-86-73, Computing Research Laboratory, Dept. 3CRL, Box 30001, New Mexico State University, Las Cruces, NM 88003-0001, US.
THE OSCON OPERATING SYSTEM CONSULTANT
119
Mc Kevitt, Paul (1987). Artificial Communicators: An Operating System Consultant. Master’s Thesis, Computer Science Department, Dept. 3CU, Box 30001, New Mexico State University, Las Cruces, New Mexico 88003-0001, US. Mc Kevitt, Paul (1988). Rules of Inference in an Operating System Consultant. In Proc. of the First Irish National Conference on Artificial Intelligence and Cognitive Science (AI/CS-88), Vol. 1. Dublin, Ireland: University Industry Center, University College Dublin, September. Mc Kevitt, Paul (1991). Analysing Coherence of Intention in Natural Language Dialogue. Ph.D. Thesis, Department of Computer Science, University of Exeter, GB – EX4 4PT, England. Mc Kevitt, Paul & Wilks, Yorick (1987). Transfer Semantics in an Operating System Consultant: The Formalization of Actions Involving Object Transfer. In Proceedings of the Tenth International Joint Conference on Artificial Intelligence (IJCAI-87), Vol. 1, 569–575. Italy: Milan, August. Mc Kevitt, Paul & Pan, Zhaoxin (1989). A General Effect Representation for Operating System Commands. In Proc. of the Second Irish National Conference on Artificial Intelligence and Cognitive Science (AI/CS-89), pp. 50–65. Dublin, Ireland: School of Computer Applications, Dublin City University, September. Also in, Smeaton, Alan and McDermott, Gabriel (eds.), AI and Cognitive Science ’89, Springer-Verlag British Computer Society Workshop Series, 68–85. Heidelberg, Germany: Springer-Verlag. Mc Kevitt, Paul & Ogden, William C. (1989a). Wizard-of-Oz Dialogues for Help on Operating Systems. Memoranda in Computer and Cognitive Science, MCCS-89-167, Computing Research Laboratory, Dept. 3CRL, Box 30001, New Mexico State University, Las Cruces, NM 88003-0001, US. Mc Kevitt, Paul & Ogden, William C. (1989b). OSWIZ II: Wizard-of-Oz Dialogues in the Computer Operating Systems Domain. Memorandum in Computer and Cognitive Science, MCCS-90-181, Computing Research Laboratory, Dept. 3CRL, Box 30001, New Mexico State University, Las Cruces, NM 88003-0001, US. Selfridge, Peter G. (1988). How to Print a File: An Expert System Approach to Software Knowledge Representation. In Proceedings of the Seventh National American Conference on Artificial Intelligence (AAAI-88), Vol. 2, 380–385. Minnesota: St. Paul, August. Shrager, Jeff & Finin, Tim (1982). An Expert System that Volunteers Advice. In Proceedings of the National Conference on Artificial Intelligence (AAAI-82), 339–340. Wilensky, Robert, Arens, Yigal & Chin, David (1984). Talking to UNIX in English: An Overview of UC. Communications of the ACM 27(6): 574–593, June. Wilensky, Robert, Mayfield, Jim, Albert, Anthony, Chin, David, Cox, Charles, Luria, Marc, Martin, James & Wu, Dekai (1986). UC – A Progress Report. Report No. UCB/CSD 87/303, Computer Science Division (EECS), University of California, Berkeley, California 94720, US, July. Wilensky, Robert, Chin, David N., Luria, Marc, Martin, James, Mayfield, James & Wu, Dekai (1988). The Berkeley UNIX Consultant Project. Computational Linguistics 14(4): 35–84, December (also in this volume). Yun, David Y & Loeb, David (1984). The CMS-HELP Expert System. In Proceedings of the International Conference on Data Engineering, 459–466, Los Angeles: IEEE Computer Society.
Artificial Intelligence Review 14: 121–141, 2000. Stephen J. Hegner et al. (eds.), Intelligent help systems for UNIX. © 2000 Kluwer Academic Publishers. Printed in the Netherlands.
121
USCSH: An Active Intelligent Assistance System MANTON MATTHEWS∗, WALTER PHARR1, GAUTAM BISWAS2 and HARISH NEELAKANDAN Department of Computer Science, University of South Carolina, Columbia, SC 29208, USA, [email protected]; 1 Current address: College of Charleston, Charleston, SC 29424, USA; 2 Current address: Vanderbilt University, Nashville, TN 37235, USA (∗ author for correspondence)
Abstract. This paper describes the knowledge sources and methods of knowledge acquisition for USCSH (University of South Carolina SHell). USCSH is an active intelligent assistance system for Unix.1 The system operates in two modes, the active mode and the intelligent mode. In the active mode USCSH monitors the user’s interactions with the system, and at appropriate times makes suggestions on how the user may better utilize the system to perform tasks. In the intelligent mode the system accepts questions in natural language and responds to them, taking into consideration the ability of the user and the context of the question. Keywords: assistance systems, shell, user-modelling
1. Introduction New software systems are constantly increasing in functionality, and the continued increase in hardware capability-to-cost ratio will serve to make the delivery of more sophisticated software possible. This makes an assistance system an essential component of these complex systems. Traditional approaches to assistance generally fall into one of several categories: command indexed systems (e.g., Unix man), hierarchical systems (e.g., VMS help), and tutorials (e.g., Unix learn). Regardless of the category these traditional approaches to assistance generally suffer from several deficiences. Responses are too verbose and are not adapted to the user’s level of knowledge of the system. On-line manuals must explain everything, and novices find them confusing, while more experienced users find it quite annoying to have to browse through a lot of irrelevant material to find the information they desire. Moreover, the systems only respond when the users realize that they need to know something, and only after they figure out the right keyword to use in the query (which places the burden of translating a request for specific help into command names on the user), or by browsing through menus (which
122
MANTON MATTHEWS ET AL.
can be quite time consuming). The problem with interactive tutorials (e.g., the Unix learn feature (Kernighan and Lesk 1976)) is that they are primarily designed for novice users, so they are rarely used for reference. In the day-today rush to get things done the user rarely finds time to devote to a “learning experience”.
2. Active and Intelligent Assistance Recently there has been substantial research addressing these deficiencies. The new approaches to assistance systems can be classified into two categories, passive and active, depending on how the interaction with the user is initiated. A passive system responds to user initiated queries, while active systems will at times initiate the interaction with the user. The idea is to emulate human consultants, who consider user proficiencies and the current context of their activities in formulating responses to queries. In addition, consultants aid a user’s learning process by indicating better and more efficient ways of getting a task done, especially when they notice the user doing things in an inefficient manner. To achieve such capabilities, online systems need to: 1. model individual users, i.e., keep track of their strengths and deficiencies, and use this information to occasionally make suggestions and help users improve their capabilities, 2. determine the context of users activities, and ensure responses cater to their individual needs and proficiencies, and 3. provide means so that user queries to the system need not be constrained by system commands and terminology. In our research, we adopt a knowledge-based approach in the design of USCSH, an active assistance interface for the Unix operating system. This system addresses the capabilities mentioned above, and therefore in many ways mirrors the capabilities of intelligent tutoring systems which explicitly structure, monitor and control the student’s learning processes (Clancey 1982). However, USCSH is designed to assist users and help improve their capabilities while they are involved in day-to-day activities, rather than only when they are in a tutorial environment. This requires that the system be unobtrusive, and not ask users too many questions, or provide unnecessary prompts while they are involved in routine activities. Therefore, our system differs from many tutoring systems (e.g., UMFE (Sleeman 1985)) which interrogate users to determine their proficiency on different system concepts while formulating responses to their original queries. This idea is reiterated in the design and implementation of computer coaching systems (Burton and Brown 1982).
THE USC SHELL
123
One of the major deficiencies of existing assistance systems is that users must know “what they need to know,” or at least must know command names or keywords that index into the information they need. Unix, as many have observed, does not use mnemonic names, and this makes it difficult and frustrating for new users to discover command names such as “cat” or “ls”. Natural language assistance systems overcome this by allowing users to phrase queries in more general terminology. The goal of such systems is to provide more flexibility, and allow the handling of terse requests such as “help on deleting lines.” There has been considerable work and progress in natural language assistance systems. With current complex systems many beginning users learn to use a minimal set of capabilities and are content to remain at this level of performance. For example, it has been observed that experienced Unix users know a small percentage of the commands available to them (Hanson et al. 1984).2 Their learning curve reaches a plateau and does not progress further because they feel that “they know how to use the system.” Their knowledge of the use of the system does not progress until it is triggered by another “nugget” of information (Matthews and Nolan 1985; Fischer et al. 1985). This trigger often occurs when observing another user, when another user makes suggestions or comments, and by casual (accidental) reading while looking for other information.
3. Criteria of System Design Two important criteria need to be discussed in assistance system design. The first criterion specified by Borenstein (1985), addresses the degree to which the system facilitates the accomplishment of a particular task by a user who does not currently know how to do it. The second criterion, which is equally important, refers to the effectiveness of the system in aiding users’ learning processes by providing mechanisms that enable them to improve their level of performance, as they are involved in their day-to-day activities on the system. Frequently the complexity of most interactive software systems (Unix, the vi editor, etc.) makes it a difficult task even for regular users to master the complete functionality of the system. Users pick up a small subset of capabilities and commands provided by the system so as to get by in performing routine activities, and because of other work pressures remain content to use the system at that level. Only when they encounter a problem that cannot be solved using their current set of known commands do they consult other sources, such as manuals and more experienced users, to learn about capabilities and features they were previously unaware off. Users may also learn additional features and capabilities by chance meetings and dialogues
124
MANTON MATTHEWS ET AL.
with more experienced users, or even by observing more experienced users at work. LEVI (LEarning VI) (Matthews and Nolan 1985; Matthews and Biswas 1985; Biswas et al. 1985), the first prototype active assistance system that we built at the University of South Carolina, addressed some of these issues. LEVI is an active assistance interface for the “vi” editor on the Unix system. It maintains models of individual users in terms of their proficiency in the usage of vi commands. The user model is an overlay model with the underlying expert model being a functional decomposition of vi capabilities. The tutoring component provides assistance to users in the form of prompts that are triggered by a set of scripts that identify some user misconceptions and inefficiencies in performing tasks, and a prompting mechanism that indicates more efficient ways of achieving the same task. The system also uses periodic prompts at regular intervals to make users aware of system capabilities they did not use in the past. Using the user model as a guide, LEVI selects periodic prompts which guide the user to new plateaus in performance and gradually would expose him to the entire domain. The goal was to have “an expert system in the interface” that would monitor the user interaction and guide the user in becoming more proficient in the use of the system. Based on these criteria and our experiences with LEVI, we feel that an effective assistance system should be developed using a framework that includes: 1. an ideal model that is used to explain system concepts and tasks to users, 2. a user model that keeps track of what the user is doing, and what he knows and does not know, 3. a natural language interface which can understand users’ queries posed in natural language, so that users are not constrained by system terminology and rigid query formats, and 4. a response generation mechanism that checks the user proficiencies and the context of the user’s activities before responding to a user query or prompting the user. The system architecture of USCSH is based on these principles.
4. Overview of USCSH The goal of the Active Assistance project at the University of South Carolina is to develop an active and intelligent user interface for the entire Unix system, which can overcome the shortcomings of current assistance systems. The aim, as with LEVI, was to have an expert system interface that would monitor the user interaction and by making appropriate suggestions guide the user in becoming more proficient in the use of the system. The second
THE USC SHELL
125
prototype system developed in this project is ‘USCSH’, the University of South Carolina SHell (Matthews and Biswas 1985). This active intelligent assistance interface, residing between the user and the Unix operating system, operates in both active and passive modes. In the active assistance mode the system continuously monitors user interactions with the Unix system, maintains user models, and at appropriate times makes useful recommendations to users about system capabilities in the context of their current activities. User commands are directly passed through to the Unix system and also posted on the assistance system working memory. The knowledge base for the overall system consists of a system description component and a comprehensive user model. The system description component contains a hierarchical organization of Unix shell (e.g., ls, chmod, cd) and utility (e.g., vi, mail, nroff) commands, typical tasks that users perform on the system (e.g., creating a document, debugging a program), and Unix system concepts (e.g., the concept of a pipe, the hierarchical nature of directories, or the role of the buffer in the vi editor). The lowest level in the hierarchy corresponds to specific commands and task names, whereas higher level nodes represent more general concepts and task descriptions. One of the key components of USCSH is the comprehensive user model. The central representational component of the user model is an overlay model (Carbonell 1970; Carr and Goldstein 1973; Clancey 1982), that is basically a collection of nodes that directly correspond to the nodes of the system capability hierarchy. A number of parameters, such as user proficiency and date last used, are associated with each node. In addition to the overlay structure, the user model contains a set of rules and procedures that are used for deriving user proficiencies when explicit information is not directly available from the user interactions. These rules are based on the Unix system hierarchy and assumptions made about a user’s typical learning process. A second component of the user model is an executable process model which represents the ideal and non-ideal models of user behavior. The ideal model contains plans that describe how more complex tasks can be executed as a sequence of simpler tasks and system commands and is represented as a hierarchy of frames (Minsky 1975). The non-ideal model (also called a buggy model by Brown and Burton (1978)) records plans that identify user errors and misconceptions which result in their performing tasks in an incorrect or inefficient manner. Based on sequences of user inputs and the (Unix) system’s response to these inputs, this model is used to infer what tasks the user is involved in, and whether he is making errors or choosing inefficient methods in executing these tasks.
126
MANTON MATTHEWS ET AL.
The active assistant is involved in two parallel but related activities as it tracks the user’s interactions with the system. First, it continually updates the dynamic user model to reflect the extent of his knowledge of the system, Second, it prompts the user on its own initiative, either periodically, or if it determines that the user is performing a task incorrectly or inefficiently. In the intelligent assistance mode the system responds to user queries on how to use the system. The first versions of the intelligent assistant were not full-fledged natural language processing systems, but used the menu/hypertext-like Screen Display System. The later version, written in LISP, uses an ATN parser to understand natural language queries. The intelligent assistance mode of USCSH is activated when the user types in the command “query”, which is treated as another C-shell command. The ATN-based parser integrates syntactic and semantic information, processes the query and translates it into an internal frame-based representation. The response generation mechanism interprets the query in the context of the user’s current activities and then generates responses that take into account the user’s proficiency in the use of the system. Response generation primarily entails matching the internal representation of the query against the frames that represent the plans of the ideal model, and retrieving a suitable frame or set of frames that contain the response to that query.
5. Active Mode There are two implementations of the active components of USCSH. One version, implemented in OPS5, runs as a separate concurrent process which communicates with the csh through a pipe. This first version illustrated the concepts but was slow (it took up to 3 minutes overhead on an NCR Tower), so we looked for ways to improve the performance. The second version is implemented by adding a “goal parser” to csh using a finite state machine