Margit Pohl - Peter Prenner - Peter Purgathofer
University of Technology Vienna
Dep. Design and Assessment of Technology
A - 1040 Möllwaldpl. 5 Vienna Austria
email:
margit@iguwnext.tuwien.ac.at -
purg@iguwnext.tuwien.ac.at
Fax: +43 1 504 11 86
Analysis of the Students´ Documents
Hypertext and Hypermedia systems are used increasingly in educational environments. The active participation of students is generally supposed to be one of the major advantages of hypermedia systems. Nevertheless, inexperienced users of such systems still have many problems with them, especially when authoring. Our experience shows that these problems can be alleviated by the introduction of an authoring tool tailored to the specific needs of the user. To develop such a tool, a systematic analysis of the authoring process is necessary. The methods used in the study described in the following text were a content analysis and an automatic monitoring system which yields a dynamic view of the authoring process. The results of our study indicate that inexperienced authors of hypermedia systems need special support to create non-hierarchical relationships and an adequate layout in their hypermedia document.
Hypertext and Hypermedia systems are used increasingly in educational environments even if their efficiency is by no means clear (Jonassen & Grabinger 1990). The active participation of readers and authors is generally supposed to be one of the major advantages of hypertext systems (e.g. Landow 1992). There is some evidence from cognitive psychology supporting this assumption. Deep processing and elaboration usually leads to better comprehension than information analysis at higher levels of processing. One way of reaching these deep processing levels is "doing things" (Mayes 1992). Active involvement of learners does not mean, however, letting students browse in a hypertext base aimlessly. Students must be encouraged to actively seek out information This can only be done by giving students well-defined tasks (Hammond 1992). Even if there is some evidence substantiating this approach in educational hypertext, it is necessary to study the impact of such an approach in day-to-day teaching and learning practice at universities. The following text describes the experiences gained in such an experiment.
Our research on the problems of hypertext authoring is part of a larger project. The aim of this project is the development and evaluation of a hypertext-based learning and information system. In the course of this project we asked students to write their seminar papers in the form of hypertext instead of traditional linear essays. The students study computer science in their third or fourth year. They are therefore acquainted with the use of computers and interested in new developments in the field of computer science. The topic of the seminars held at our department can be roughly described as "Social aspects of computer technology and technology assessment". In the course of these seminars students usually have classes for two hours per week for one term. The task in such seminars is to read a textbook or some other scientific publication, to present the content of the text in the seminar, and to prepare a written assignment. The written assignments are about 20 to 30 pages long. Students usually work alone and occasionally in small groups of two or three. The goal of a lecture form like this is to enable students to discuss the content of a scientific text from various points of view and to teach students how to communicate their ideas in oral and written form. The problem of traditional written assignments is that students have to write a certain number of pages. They therefore tend to fill these pages with very detailed information and very often fail to grasp the basic ideas of the text they read. Our assumption is that hypermedia can help overcome this problem because such systems rather encourage the creation of structure than the creation of large amounts of text. Nevertheless, our practical experience showed that it is necessary to develop an authoring tool specifically tailored to the needs of untrained authors if they have to fulfill the task described above. Such an authoring tool has to contain certain constraints to support students in their learning activity.
Therefore, we decided to develop our own authoring tool, a generic system running on top of Hypercard for the Macintosh. The most basic aim of this authoring tool was to solve the students´ technical problems. In this context, the interface (ease of use, ease of learning) plays an important role (Marchionini et al 1994). We tried to make the system as clear and understandable as possible. To write their hypermedia documents students need only a small part of Hypercard´s functionality. The authoring tool is restricted to these few features. Linking and writing is made much easier. To make a link authors do not have to learn a sequence of several steps, they just click on a button on the screen and then fill in a dialog box. Writing became much easier because of the introduction of a "default" layout. Figure 1 shows a screenshot of an empty text window. All the necessary commands are represented as icons at the bottom of the screen. There are five symbols for typed links, buttons which enable students to integrate sound, pictures and video into their document, and several buttons for navigation. Besides, there are two predefined text windows. Students are supposed to write only short pieces of text. They are allowed to organise their documents as they think is appropriate. Once they are finished the students' documents are integrated into the hypertext-based information system developed at our department. Sometimes, they serve as a starting point for subsequent seminars.
There are several reasons for conducting an accurate analysis of the students´ documents. Firstly, it is necessary to know the students´ problems quite thoroughly to be able to improve the authoring tool. For this we cannot rely exclusively on the opinion of students and staff. Secondly, you can get some evidence about the usability of hypermedia systems, and the nature of authoring processes in general.
Hierarchy
Our first rather circumstantial experience after having introduced hypertext in our lectures was that students did not explore the new potentials offered to them but stuck to the structure of their linear seminar papers. A more detailed investigation of the hypertext documents showed that students did not exactly write linear text, but wrote text structured in a hierarchical manner. Traditional seminar papers are usually composed of several sections and sub-sections. It must be mentioned that linear text and hierarchically structured text are very similar because one can be easily transformed into the other (Rada 1991). Part of the analysis of hypertext documents deals with the question of hierarchy and the conditions for the creation of non-hierarchical structure. The creation of non-hierarchical structures is supported specifically by some hypertext systems. Our assumption is that students tend to use hierarchical structures because they are educated in a tradition of writing hierarchical essays. They only change their behaviour if they use a tool that restricts the development of hierarchical structure and supports the use of non-hierarchical links.
One of the first things we noticed when we started to investigate the use of hierarchical structures was that the concept of hierarchy tends to become meaningless through the use of hypertext. Hypertext apparently induces at least some of the users to describe the main topics of their document as a kind of "information landscape" or "information map" where the distance between two concepts and their relative position on the screen conveys more information about the underlying concepts than any hierarchical structure. Such an image transcends rather than contradicts the idea of hierarchy, and it becomes more and more difficult to assess if a link is hierarchical or not. We tried to overcome this problem by a fairly strict definition of hierarchy: hierarchy is constituted either by a part-whole relationship, by class-inclusion or by a relationship between a more general and a more specific concept. Nevertheless, the inherent fuzziness of the concept of hierarchy has to be kept in mind when interpreting the results.
Link Types
Another subhypothesis deals with the nature of non-hierarchical links. In the beginning, we were interested in what types of non-hierarchical links untrained authors would create without ample assistance from teachers or the system. Many different link taxonomies are described in hypertext literature (see e.g. DeRose 1989, Parsaye et al 1989). We found two systems especially useful because of their empirical value (Rada 1991, Bloomfield & Johnson 1993). Both studies tried to define link types by a bottom-up approach. Subjects were asked to define links, and these links were subsequently categorized by the authors. In our research project we applied a similar approach to find out what link types students preferred. It must be mentioned, however, that the link types students preferred are not always the most useful ones from a cognitive point of view. This agrees with Fisher (1992) who reports that students tend to make very general links. She posits that this "is not desirable when the goal is to learn about a complex domain of knowledge." (p. 71) More specific relations on the other hand incorporate more information and therefore make learning more effective.
Rada (1991) distinguishes between sequence links, outline links, reference/citation links and embedded or cross-reference links. Outline links are apparently hierarchical whereas reference and embedded links are supposed to convey a non-hierarchical relationship. Nevertheless, in an empirical study, Rada found out that embedded links were also used in a hierarchical sense. Bloomfield and Johnson (1993) argue that navigation problems could be alleviated by sematically "rich" links. Semantically "rich" in this context means that the link type already helps users anticipating the content of the link target. Bloomfield and Johnson conducted an empirical study to find out what types of relationships subjects would find beween different paragraphs of a certain piece of text. They distinguish between 15 link types: compare, contrast, advice, describe, explain, elaborate, expand, general -> specific, continue, proc (temporal flow), (same) subject, subject (different aspects), subject (general), subject (minor topics), domain specific links.
Visualization
Visual factors play an important role in hypermedia design. The visual appearance should support the structure of the document and convey a sense of order and clarity (Kahn et al 1990). It is important to mention that visualization in this context means more than making a pleasing layout or making a good user interface design. It is quite plausible to assume that graphical design plays an even more important role in electronic text than in the printing technology because structure and information retrieval are more sophisticated in hypermedia than in traditional books. Table of contents, paragraph structure, pagination, section titles or indices are different possibilities for organising a linear text. Partly, these elements use graphical cues to do this. In electronic text there is a much wider variety of such possibilities. Again, many of them take on a topological form and use a space metaphor to make contents more understandable. The value of topological information (information about proximity, spatial inclusion or distance between different concepts) is increasingly obvious (Bolter 1991). Cognitive maps and "information landscapes" not only give users an overview about the contents, they also comprise information about the nature of the relationships between the different topics. In Hypermedia, graphics and pictures can be integrated into the text in a way which was not possible in the age of print (Bolter 1991).
On a theoretical level the importance of visual elements seems to be clear. On the other hand, there is still not enough empirical evidence to give a comprehensive account of the effectiveness of different devices. It is still an open question as to which general principles of user interface design can be applied to hypermedia. As a consequence, the formulation of hypotheses in this area can only be tentative. It is certainly interesting to try to find out which new methods of presenting and organising information in the sense described above are used by inexperienced authors. On the other hand, many of the traditional principles of user interface design have to be applied to make hypermedia documents understandable. Again, our research goals were twofold. Firstly, we wanted to find out if hypermedia really offers new opportunities to integrate image and text and if authors use this opportunity. Secondly, we tried to get feedback about the usability of various features of the authoring tool we used. If our analysis showed us that authors had problems with some elements of graphical design we tried to change our authoring tool to help the students overcome this problem.
Improvement of Hypermedia Documents
Many different hypermedia learning systems are used in university education. The most basic question in this context is whether these systems improve learning. In this abstract form the question cannot be answered because hypermedia systems and the situations in which they are used vary considerably. For example, students participating in lectures at our department had difficulties when they used Hypercard to write their essays. Therefore, we decided to develop our own authoring tool first before comparing hypermedia and traditional means of education. The basic assumption about the authoring tool we use is that the use of this tool makes authoring easier and improves hypermedia documents compared to hypertext systems which are available on the market. This question is very difficult to answer as there is no catalogue of criteria for hypermedia quality, and it is questionable if there ever will be one. To overcome this problem, Nielsen (1990) proposed to inspect existing hypertext documents. As we have to tell authors explicitly how they should design their documents, this was not a viable solution for us. In this sense, the analysis is influenced by a certain conception of hypermedia quality. This conception is based on empirical evidence described in literature. In this sense, it is still fairly general.
To analyse the students´ documents we conducted a Content Analysis. Content Analysis (CA) is a methodology which is used in many disciplines, especially in social sciences. CA is a systematic analysis of mainly verbal material based on a set of relevant categories. The notion that verbal material conveys some sort of meaningful message is central for many kinds of CA. One of the reasons for the decision to use CA was to analyse this very aspect: hypermedia as a medium of communication. In this context, we concentrated on formal aspects of hypermedia, that is, its current and potential structural characteristics and graphical design elements.
The categories of CA are derived either from theoretical considerations or practical experience. In our case, both elements played an important role. On the one hand, general principles of hypertext development (see e.g. Jonassen & Mandl 1990, Kuhlen 1991, Nielsen 1990, Rada 1991) and of user interface design (see e.g. Mayhew 1992, Preece 1993, Tullis 1991) were the basis for category formation. We also included the category "hierarchical vs. non-hierarchical structure" in the CA to find out if it was possible to support the creation of alternative ways of organising information. On the other hand, we discovered a few of the students´ problems through practical experience. Some students did not realize that an understandable hypertext should offer the opportunity to get from one level in the hierarchy to another quite easily. We included this category to find out if that was a general problem or not. It must be mentioned, however, that the categories for the CA are not meant as a checklist for evaluation of hypertexts.
There are two main areas of analysis: structure and visualization. The "structure" part is made up of questions concerning the prevalence of hierarchical structure, the types of non-hierarchical links which were used, and common inconsistencies of structure. To categorize link types we used six different categories:
The "Visualization" part of the analysis consisted of three sub-sections: typography, screen design, and media. The section on typography includes elements like fonts, highlighting, and line length, the section on screen design includes aspects like "amount of information presented" and "grouping of information", and the section of media tries to analyse the use of different media (graphics, animation, video, audio). Generally, it is very difficult to assess these features independently. If information in a document is not well organised, by using features like highlighting or grouping of information then just a few sentences can give the user the impression of a cluttered screen. Some of the principles we found in the literature do not apply to hypermedia. The "catch the user´s eye"-principle (important information should be displayed in a prominent place: Preece 1993) applies to information in tabular form rather than to hypermedia assignments. Hypermedia systems very often force authors to write only one or two paragraphs of text for one node, therefore the distinction between more and less important information is quite difficult to make. On the other hand, some principles might be more important for hypermedia than for other applications. A clear underlying visual structure and the creative use of white space are part of the idea of spatial organisation of information in hypermedia.
The analysis of hypermedia documents is also supported by an automated monitoring system. While authors are working with the authoring system they are constantly monitored. All structure generating actions are logged. Generally, most of these actions take place in the Overview Editor. They include: generating, moving and renaming cards, defining and deleting links, generating and defining envelopes, and deleting cards and envelopes. All monitored actions are written to a log field in the document and can be accessed later to review the structuring process. The best way to view data like this is in an animation. In a visualization tool built as part of the project the log data can be loaded and run as an animated reconstruction (Fig. 3). The advantage of such a device is that it presents a dynamic and direct view of the authoring process whereas the analysis of the finished documents only permits a static and indirect view. There is not much literature on the interpretation of such dynamic protocols. So far, we have used the animation for case studies of single documents.
So far, we have analysed 33 documents. They consist of documents created by first and third generation tools. The students who wrote first generation documents used Hypercard with a few small additional tools. The second generation authoring tool was an intermediate stage with only a few documents. Third generation documents are based on a prototype of the authoring tool we use now, an authoring tool which is quite different from Hypercard.
Fifteen of the 33 documents are first generation documents. Most of the documents (9) have a very dominant hierarchical structure. There is no document with a weak hierarchical structure and with a dominance of non-hierarchical links. Five documents do not contain any non-hierarchical link at all.
There are some fundamental differences between first generation documents and third generation documents. There are generally more non-hierarchical links in third generation documents than in the first generation documents (Table 1). The maximum number of non-hierarchical links in one document and the mean number of non-hierarchical links per document are significantly higher for third generation documents. The number of documents with a dominant hierarchical structure decreased (in brackets: percentage of the total number of documents of this generation). This results in a reduction of hierarchy in the documents. Two of the students who used the third generation authoring tool even ventured to create a completely non-hierarchical structure. One of these documents is very clear and understandable which points to the fact that hierarchy as such does not lead to clarity and a non-hierarchical structure to confusion.
first generation ( =13)
| third generation (=18)
| |
|---|---|---|
| max. no. non-hierarchical links | 17,0 | 33,00 |
| mean non-hierarchical links | 3,4 | 12,06 |
| dominant hierarchical structure | 9,0 (69%) | 7,0 (39%) |
Table 1: Structural differences between first generation and third generation documents
Our assumption is that it is much easier for inexperienced authors to create links using the third generation authoring tool than when using Hypercard. This would indicate that some of the problems students had with hypertext structure were due to technical reasons rather than to a lack of understanding. On the other hand, the vast majority of links still were defined from a concept on a card to another occurrence of the same concept on another card (66% of all non-hierarchical links in third generation documents). One exception to this rule was the fact that students used Post-Its quite heavily (47 times) to explain concepts mentioned in the text in more detail or to make comments on the contents of a card. Post-Its are small pop-up windows where authors can enter comments or short explanations about a concept. Similar links were only made 10 times in first generation documents. Apparently, students are willing to use tools to improve the structure of their documents and their links if these tools are easy enough to learn. This is the reason why we decided to introduce "Typed Links" in the fourth generation of our authoring tool.
There were similar developments in regard to layout and graphics even if the evidence is sometimes contradictory (Table 2). As an example, we will describe the feature "amount of information presented". This feature is composed of three variables: line length, proportion of black and white space, and overall impression (in brackets: percentage of the total number of documents of this generation). "Line length " measures how many documents contain cards with text lines which are too long (=longer than 50 characters). "Proportion of black and white space" refers to the fact that it is difficult to read text on a screen which does not use white space to structure this text. The numbers in the table below show how many documents of one generation have one or more cards with too much text (=black space) on them. "Overall impression" reflects researchers' subjective evaluation of the document layout irrespective of the physical number of characters on the screen. The respective numbers in Table 2 mean that 9 documents out of a total number of 15 (=60%) have cluttered screens in first generation documents and 8 out of 18 (=44%) in third generation documents. Despite the layout constraints in the third generation tool students apparently still put too much text into text fields.
| first generation (=15)
| third generation (=18)
| |
|---|---|---|
| lines of text too long | 11,0 (73%) | |
| proportion of black and white space | 8,0 (53%) | 2,0 (11%) |
| overall impression | 9,0 (60%) | 8,0 (44%) |
Table 2: Layout differences between first generation and third generation documents
In third generation documents the screen is partitioned into one text field for running text and one for subtitles and comments. Line length is therefore a variable which is not valid any more for third generation documents because the line length is defined by the "default" layout. Similar constraints in the third generation authoring tool are responsible for the marked decrease in the amount of black space. Apparently, it is possible to solve these problems by introducing certain constraints into the authoring tool. On the other hand, the overall impression that the screens in many documents are still cluttered remains for third generation documents. It is obvious, therefore, that there are still problems to be solved.
It is quite difficult to assess the educational effects of the usage of hypertext-based learning systems. Firstly, the results of hypertext usage depend very much upon the educational context and the students' task. Presumably, hypertext is more useful for some purposes than it is for others. Secondly, it can be argued that the goals of various methods of teaching are different. Therefore, the effects of using these different methods cannot be compared easily. Rote learning might, by the very nature of the method, lead to a different learning outcome than concept mapping. Thirdly, the teaching methods to be compared should be chosen carefully. Hypertext is still a very young technology. Therefore, many of the systems are still in an experimental state. It does not seem plausible to assume that comparing such a system to a conventional textbook of good quality can be the basis for a conclusive decision about the usefulness of hypertext learning systems. We encountered this problem when we started to use HyperCard as an authoring tool. HyperCard or any other more complex hypertext system is apparently not an appropriate solution for inexperienced authors who cannot spend very much time to get acquainted with the system. Before comparing hypertext and traditional methods of teaching and learning we therefore had to develop an adequate authoring tool to avoid getting biased results.
It seems fairly obvious that naive users will become more involved with hypertext systems in the near future, both as readers and as authors. This development will be especially important for university education. Systems must be tailored according to the specific needs of these users and their tasks. To be able to do that, a careful analysis of the transition process from linear text to hypertext is necessary. There are both technical and conceptual problems in the transition process. They must be analysed separately. Untrained authors of hypertext documents usually have difficulties with the hypertext concept of structure, with the design of an appropriate layout and with the creation of understandable text. We think, that structure is the most basic concept of the three because text as well as layout can be defined in a way to obscure or elucidate an existing strucure. Our experience shows that these problems can, at least partly, be overcome by the introduction of an appropriate authoring tool.
So far, we have only analysed systematically the usability of the authoring tool. But we have some circumstantial evidence which indicates that hypertext learning systems can be a valuable addition to traditional methods of teaching and learning. At the beginning of our project we developed a short questionnaire to assess students' attitude towards hypertext. The detailed results of this questionnaire have been discussed elsewhere (Pohl & Purgathofer 1993). Generally, it can be said that students preferred hypertext to traditional methods of writing essays. We also compared hypertext documents to traditional linear documents written by the same students about the same topics. Apparently, some of the students are specifically supported by hypertext. They do not seem to feel at ease with traditional linear text but develop very creative hypertext documents. Interpretation of the maps (especially the animated version, see Fig. 3) produced by the students using the Overview Editor shows that students spend very much time organising and structuring their documents. In contrast to the experience of Reader & Hammond (1994) their overview maps are usually very elaborate and are revised again and again. These differences seem to be due to a different institutional setting. In addition, students told us in informal conversation, that the Overview Editor helped them to develop a more concise interpretation of the text they had read. We intend to investigate these questions more systematically in the future.
Our research is part of a project funded by the Austrian "Fonds zur Förderung der wissenschaftlichen Forschung", project no.: P8565-TEC. This project is directed by Prof. Peter Fleissner.
Bloomfield, H., and Johnson, P. (1993) Towards Cognitively Salient Relations for hypertext navigation. In: Alty, J.L., Diaper, D., Guest, S.: People and Computers VIII. Proceedings of the HCI ´93 Conference. Cambridge: Cambridge University Press, 463-477
Bolter, J.D. (1991) Writing Space. The Computer, Hypertext, and the History of Writing. Hillsdale, New Jersey: Lawrence Erlbaum Associates
DeRose, S.J. (1989) Expanding the notion of links. In: ACM Hypertext ´89 (Proceedings) 1989, 249-258
Eysenck, M.W., and Keane, M.T. (1992) Cognitive Psychology. Hove & London: Lawrence Erlbaum Associates
Fisher, K. (1992) SemNet: A Tool for Personal Knowledge Construction. In: P.A.M. Kommers, D.H. Jonassen, J.T. Mayes Cognitive Tools for Learning. Berlin, Heidelberg, New York: Springer, 63-75
Hammond, N. (1992) Tailoring Hypertext for the Learner. In: P.A.M. Kommers, D.H. Jonassen, J.T. Mayes Cognitive Tools for Learning. Berlin, Heidelberg, New York: Springer, 149-160
Jonassen, D.H. (1993) Effects of Semantically Structured Hypertext Knowledge Bases on Users' Knowledge Structures. In: C. McKnight, A. Dillong and J. Richardson Hypertext - A Psychological Perspective. New York, London, Toronto: Ellis Horwood
Jonassen, D.H., and Grabinger, S. (1990) Problems and Issues in Designing Hypertext/Hypermedia. In: Jonassen, D.H., and Mandl, H. Designing Hypermedia for Learning. NATO ASI Series, Series F, Vol.67, Berlin, Heidelberg: Springer, 3-25
Jonassen, D.H., and Mandl H. (1990) Designing Hypermedia for Learning. NATO ASI Series, Series F, Vol.67, Berlin, Heidelberg: Springer
Kahn, P., Launhardt, J., Lenk, K., Peters, R. (1990) Design of Hypermedia Publications: Issues and Solutions. In: Furuta, R. (ed.) Proceedings of the International Conference on Electronic Publishing, Document Manipulation, and Typography (EP 90). Cambridge University Press, 107-124
Kommers, P.A.M., Jonassen, D.H., Mayes, J.T. (1992) Cognitive Tools for Learning. Berlin, Heidelberg, New York: Springer
Kuhlen, R. (1991) Hypertext - Ein nicht-lineares Medium zwischen Buch und Wissensbank. Berlin, Heidelberg: Springer
Landow, G.P. (1992) Hypertext - The Convergence of Contemporary Critical Theory and Technology. Baltimore and London: The John Hopkins University Press
Mayes, T. (1992) Cognitive Tools: A Suitable Case for Learning. In: P.A.M. Kommers, D.H. Jonassen, J.T. Mayes: Cognitive Tools for Learning. Berlin, Heidelberg, New York: Springer, 7-18
Mayhew, D. (1992) Principles and Guidelines in Software User Interface Design. New Jersey: Prentice Hall
Marchionini, G., Neuman, D., Morrell, K. (1994): Directed and Undirected Tasks in Hypermdia: Is Variety the Spice of Learning? In: T. Ottmann, and I. Tomek: Eudcational Multimedia and Hypermedia, 1994. Proceedings of ED-Media 94. Charlottesville, USA: AACE, 373-378
Nielsen, J. (1990) Hypertext & Hypermedia. Boston: Academic Press
Parsaye, K., Chignell, M., Khoshafian, S., Wong, H. (1989) Intelligent databases. Object-oriented, deductive hypermedia technologies. New York: Wiley
Pohl, M., Purgathofer, P. (1993) The Problems of Untrained Authors Creating Hypertext Documents. In: T. Grechenig, M. Tscheligi (eds.): Vienna Conference on Human Computer Interaction ´93. Fin de Siècle, Vienna, Austria, September 1993. Berlin, Heidelberg, New York: Springer, 338-349
Preece, J. (ed.) (1993) A Guide to Usability. Wokingham, England: Addison-Wesley
Rada, R. (1991) Hypertext. From Text to Expertext. London, New York: McGraw-Hill
Reader, W., Hammond, N. (1994) Computer-Based Tools to Support Learning from Hypertext: Concept Mapping Tools and Beyond. In: Computers Educ., 22, 99-106
Tullis, T.S. (1991) Screen Design. In: M. Helander (ed.) Handbook of Human-Computer Interaction. Amsterdam: North Holland (2nd edition), 377-411