The Dynamics of Collaborative Design

Lorraine Sherry
Research Associate
RMC Research Corporation, Denver

and

Karen Madsen Myers
Director
HealthONE, Denver


Full citation:

Sherry, L., & Myers, K.M. (1998). The dynamics of collaborative design. IEEE Transactions on Professional Communication, 41 (2), 123-139.

Copyright © 1998 IEEE. Permission has been granted to post this document on this website.

Portions of this paper appear in the Proceedings of the 43rd Annual Conference of the Society for Technical Communication.


Abstract

The University of Colorado at Denver's Internet Task Force has been conducting developmental research on the collaborative learning and participatory design process while creating and implementing the School of Education's Web page and mediated learning environment over the past three years. We developed a design and learning process model that is appropriate for designers working in groups in academic or corporate environments. Critical features are authentic tasks, knowledge development, generating research questions, and reflection.

There is a great deal of interest in using the Internet as a powerful constructivist environment for learning. The University of Colorado at Denver's (UCD) Internet Task Force—a team of graduate students together with our faculty advisor—wanted to enable our academic community to join the Internet culture; develop an on-line knowledge-building and learner-support environment; and conduct developmental research on computer-supported collaborative learning, participatory design, and technology adoption.

We realized that the World Wide Web (or just "the Web") is a library, a resource database, and a vastly powerful research tool that excites, motivates, and empowers students (Martinez, Billig, Sherry, & Hoffman, 1997; Lawyer-Brook & Sherry, 1997). We wanted to build a home page that would be a resource of depth rather than one which emphasized "bells and whistles". Our intent was to lend support to our academic community in their attempts to access information, build knowledge, and share their findings with others.

This type of effort requires close cooperation among members of a team who are attempting to design a learning environment by doing three things simultaneously: building a shared base of knowledge and relevant skills; designing and developing a software product; and making sure that product is usable by the intended audience. It also requires close communication between our design team and the rest of our learning community, since the design cycle involves empirically testing a prototype on typical end users, then iteratively adjusting and redesigning the product and distributing it appropriately (Gould & Lewis, 1985; Brockmann & Sinatra, 1995).

Besides being interested in the design process, we had some specific research questions we wanted to explore:

1. What processes are occurring in a collaborative learning and design group that is using a mediated environment?

2. What are the group dynamics that support the design process?

3. What are the differences between face-to-face and electronically mediated collaboration?

We studied, reflected upon, and documented our collaborative process as we participated in both face-to-face meetings and e-mail discussions in an attempt to carry out an authentic task. This process is known as developmental research (Seels & Richey, 1994).

Background

In the spring of 1994, two of our members created Instructional Technology (IT) Connections, a hypertext system that links students with a variety of Internet resources and tools relating to information and learning technologies and cognitive science. IT Connections is linked to the On-line Helpdesk, an all-purpose hypertext-based help page for both beginning and experienced users of UCD computers (Ryder, 1994).

The following autumn, an ad hoc group of graduate students from the Information, Learning, and Technology (ILT) master's degree program and the Curriculum, Learning, and Technology (CLT) doctoral program joined the pair and founded the UCD Internet Task Force. Based on an informal needs assessment of a few members of our academic division, we expanded the original hypertext documents to include a diverse set of trifold "help" brochures, electronic job aids, and three on-line tutorials. We hoped that these would enable faculty and students to share scholarly products via e-mail, file sharing, and the UCD gopher server. Each member was individually responsible for contributions to the joint effort of the Task Force (Sherry, 1996). To be honest, our activities were rather scattered, even though we shared the same overarching goals.

In January 1995, three important things happened: The Internet Task Force conducted a formal needs assessment regarding present and potential reasons for use, barriers to use, and perceived usefulness of aids and supports for new Internet users; the university unveiled its new Web server; and students in the graduate seminars began to construct home pages.

In spite of our enthusiasm for the potential of Internet resources, we were sensitive to the very real obstacles facing students and faculty alike. In particular, we were wary of mandating policy and enforcing changes in the School's culture without careful assessment of people's values, perceptions and needs (Sarason, 1988). Based on our previous research (Wilson, Hamilton, Teslow, & Cyr, 1995; Sherry & Morse, 1995), and Rossett's (1991) needs assessment model, we conducted a division-wide survey in January 1995 to assess the level of Internet training and support needed by our students, faculty, and staff. We were particularly interested in their reasons for use, barriers to use, and their perceived usefulness of the aids and supports that the Internet Task Force was considering implementing.

When asked to rate the usefulness of a variety of reasons for using e-mail, the responses tended to break into three factors: communicate locally and share information, find and organize information, and observe/collaborate with others within and outside the university. Our results are reported in Wilson, Ryder, McCahan, Sherry, and Myers (1995) and Sherry (1995).

When the university implemented its Web server, we were finally able to progress from piecemeal interventions to a dynamic, systemic approach toward information delivery and performance support. This also provided us with an opportunity to explore the potential of the network to support communication among the distributed members of our team to carry out a collaborative design process.

At this point, the Internet Task Force began to create and link a set of newly created pages into a coherent superstructure—the School of Education Home Page. As the Home Page grew into its current form, it expanded to include the School's vision, mission statement, and instructional goals; a database of scholarly products generated by the School's faculty and students; full descriptions of all academic programs; research projects and products from the doctoral laboratories; K-12 and research connections; templates, tutorials, and reference guides about on-line publishing; links to national and international professional societies; information about labs and other facilities; and a suggestion box for feedback from users.

Conceptual Framework

There were two parallel processes occurring during our project: (a) developmental (or action) research and (b) collaborative learning and design. Though the team as a whole was primarily interested in the development of the Web page, both of us were also concerned with studying the collaborative design process, and the learning that supported it, as it was occurring.

Developmental Research

In its simplest form, developmental research could be described as a situation in which "someone is performing instructional design, development, or evaluation activities and studying the process at the same time" (Richey & Nelson, 1996, p. 1213). In each case, a clear distinction is made between performing a process and studying that process. Since we were learning new skills and content knowledge, performing instructional design, development and evaluation, and studying the process at the same time, we took a Type 2 developmental research approach to describe our group's activities (Richey & Nelson, 1996).

Developmental research is defined as "the systematic study of designing, developing and evaluating instructional programs, processes and products that must meet the criteria of internal consistency and effectiveness" (Seels & Richey, p. 127). Its emphasis is on "the production of knowledge with the ultimate aim of improving the processes of instructional design, development, and evaluation" (Richey & Nelson, 1996, p. 1213). Type 2 research typically involves an analysis of design, development or evaluation processes which is generalizable even though the instances studied may be context specific.

A similar approach is used by other researchers in studies of practitioners at work. For example, Schon (1983) examined the knowing-in-action of architects, psychotherapists, and managers as they work out problems of practice; Argyris (1985) called this process action research. Schon captures the way in which a professional designer reflects in action. The practitioner is not confined by structured theory; rather, he/she is constructing a dynamic conceptual framework that evolves as the design process progresses. "The practitioner’s inquiry is not limited to a deliberation about means which depend on a prior agreement about ends. He does not keep means and ends separate, but defines them interactively as he frames a problematic situation. He does not separate thinking from doing...because his experimenting is a kind of action, implementation is built into his inquiry" (Schon, p. 68). This is exactly the process we tried to capture in our own developmental research.

Collaborative Learning and Design

Our approach melded the processes of collaborative learning and design. We also examined the kinds of communication exchanges which support such collaboration and the media through which they may occur.

Collaborative Learning

Models of the collaborative learning process proliferate throughout the literature (Brown, 1994; Crook, 1994a, 1994b; Koschmann, Myers, Feltovich, & Barrow, 1994; Feltovich, Spiro, Coulson, & Feltovich, 1996; Brown & Campione, in press)—these focus on negotiating and building a shared knowledge base. In these studies, the shared knowledge base need not be mediated electronically. More recently, groupware and distributed hypertext systems for educational and research (Scardamalia & Bereiter, 1991; Edelson, Pea, & Gomez, 1996; Feltovich, Spiro, Coulson, & Feltovich, 1996; Harel & Papert, 1993; Collis, 1997; Wolcott & Robertson, 1997) have become popular—these studies describe the effective use of shared electronic tools for communication and knowledge-building among a community of learners.

Pea (1994) describes two modes of communication that foster collaborative learning. In information transmission, knowledge is imparted to the learner by some form of instruction; in ritual communication, learners come to share common knowledge and values through participation and socialization with other group members. These are not mutually exclusive. When learners participate in inquiries at a field's frontiers of knowledge with mature communities of practitioners, the communication process transcends both modes and gives rise to a process whereby both sender and receiver are transformed.

By capitalizing on varieties of talent, the group setting forces learning, understanding, and conceptual change. Ideas migrate through the group, and new concepts are further refined. Hanson and Gladfelter (36), like Brown (4) found that discourse takes place within a networked learning community joined by common values and an esprit de corps.

Collaborative Design

Instructional design models abound in the instructional technology literature (Tennyson, 1995)—these focus on the analysis, design, development, implementation, and evaluation of learning environments. There is also a rich source of information about design in the human performance literature and its associated disciplines (see Sherry & Wilson, 1996). In examining the design process, we found Gould and Lewis' (1985) three principles for user-centered design particularly helpful: early focus on typical users and tasks; empirical measurement that focuses on intended users; and an iterative design process.

In Gould and Lewis' model, the design team should be brought into direct contact with typical users right from the start. In our case, our design team was comprised of a group of typical users, not all of whom had advanced skills in either Web authoring or UNIX. This brought to the foreground various issues for our own team, not just of design, but also of the learning that takes place as members of the team strive to share a common knowledge base. Secondly, Gould and Lewis stress the development of prototypes that are not "cast in concrete", but are easy to revise, based on participatory feedback. Design guidelines are not algorithms; they are based on informed opinion rather than established principles. As such, they should be flexible, rather than "freezing" parts of the design that should be open to change, such as a user interface or the linking structure among the component Web pages. In our endeavors, we wanted a way that individual design decisions could be viewed by the group and open to discussion and feedback. Thirdly, iterative design is promoted by modular implementation and early usability testing.

In discussing environments for design, Winograd (1995) comments that the design cycle does not start and end with the product. It co-evolves in an environment in which new tools lead to new practices, creating problems and possibilities for new innovations. Some bugs do not show up in the initial code; they only emerge when the user is in the natural, "non-ideal" context in which the system is to be employed. Winograd's solution is to de-emphasize "generate, then test", and to emphasize interaction and dialogue between designers and users through all phases of the design. The debugging starts with the ideas, not with the code.

To share their ideas and put them into practice, cutting-edge designers often have the opportunity to use a seamless, shared, electronic workspace such as that described by Ishii, Kobayashi, and Arita (1994). We did not have such a sophisticated infrastructure. Instead, our members had e-mail accounts on different systems and were working within their own student accounts on the university's network server. Only later on did we have a class account that could be shared by our whole team, thereby enabling each of us to work jointly on shared files. At the beginning of our project, our problem was to gather all the pieces of the puzzle and assemble them—the learning and design environment, together with all the necessary tools for design and communication.

The media with which we were working, namely, e-mail communication and HTML authoring within a shared design space, supported an iterative process of learning and design among our team members. Our design process was guided by our own input as users, learners, and designers, and was facilitated by the types of communication exchanges that the e-mail system supported.

Method

Our research efforts took the form of a case study with all data analysis performed after the initial design activities were competed, using converging lines of inquiry (Yin, 1994). The study began with the formulation of the UCD SOE Home Page design and development team and ended when the page was made public on the Internet.

The Design Process

To provide a user-friendly interface, we had Netscape installed on all the computers in the Instructional Technology laboratory. We also developed a template for automatically creating and customizing student home pages, titled "Makepage". Our next step was to design the School of Education Home Page itself. Our initial face-to-face meetings were spent negotiating the conceptual framework and content outlines for the Home Page. We agreed upon tasks, timelines, responsibilities, and deliverables. Each member took ownership of a piece of the project and began to create sections of the product in his/her own student directory. Up to this point, this was a fairly linear, individualized process. Then things changed.

As the creation of the Home Page progressed, we began to engage in an iterative, nearly simultaneous process of design and revision. The rapid revision process that ensued was a direct result of our use of a mediated environment consisting of e-mail messaging and a shared design and learning space on the Web. As one team member designed a piece of the product, other team members would comment on it and provide feedback; sometimes they would make changes, if appropriate. Thus, the product that resulted was emergent—the result of the entire efforts of the team, viewed from multiple perspectives—it was more than the sum of the inputs of individual designers. Moreover, our group began to cohere as everyone on the team monitored all the on-line exchanges and looked for ways to bring ideas together, a process described by Scardamalia (1997).

At the beginning of the project, none of us had expertise in HTML (the hypertext mark-up language), and there were no HTML editors in existence. Most of us had only a rudimentary knowledge of the UNIX operating system and hypermedia-specific design principles. Thus, our developmental research, carried out over a six-month period, focused on identifying processes underlying group collaboration within a team whose members had multiple identities. Rather than creating a group that consisted of developers, learning facilitators, designer-researchers, presenters, and end users, as is being done in the corporate sector today (ref. Sherry & Wilson, 1996), our group members each wore many hats.

Where skills were lacking, team members with greater expertise coached novices. Where tasks were intertwined, team members worked as pairs, either sharing or linking related elements. Our advisor settled into a pattern of providing consultative leadership and support, while leaving the day-to-day development to others, based on the needs, expertise, and progress of group members.

We designed collaboratively by asking questions, requesting technical information, and seeking opinions about handling problems. Often, alternative views resulted in major differences of opinion. Rather than stifle conflict, we acknowledged opposing viewpoints, debated alternatives, and negotiated differences of opinion. Like Brown and Palincsar (1989), we arrived at a mutually agreeable solution by re-thinking our original premises, restructuring our ideas, and developing a unique synthesis of two alternate points of view.

What made this happen? Bereiter (1994) notes that in progressive discourse of this type, there has to be a commitment to frame questions and propositions so that evidence can be brought to bear against them. Jenlink and Carr (1996) refer to this type of conversation as dialogue that transforms the individual thinking and thought processes, creating collective thought. It requires that individuals first examine their personal assumptions or opinions and then suspend these assumptions before the entire group. Isaacs (1996), studying the same phenomenon, noted that the process of dialogue seems to involve shifts in the very ground on which people stand, transforming and expanding their sense of self, and deepening their capacity to hear and inquire into perspectives vastly different from their own.

Socially constructed knowledge of this sort allowed us to develop totally new understandings of the tasks at hand, as well as the negotiation process itself. The collaborative structure of our project led to a cognitive processing model that encompassed learning as well as design outcomes. Moreover, by using mediated communication to construct knowledge, our team and the electronic network that we were using were effectively combined into a single system—two parts of one whole—more than an information processing system (Tikhomirov, 1979, p. 261).

Data Collection

We captured the collaborative design process from our own team members by saving and extracting and triangulating information from three sources (see Table 1).




Table 1

Triangulation Sources.

Key Source
E E-mail messages among Task Force members
M Task Force Meeting notes
Q Member responses to a written Questionnaire

 

(E) All e-mail messages were saved.

(M) A record of all face-to-face meetings was kept.

(Q) At the end of the project, all participants completed a written questionnaire that asked about both their individual role and the group's role in the design process. Participants were also asked to reflect on the design process. All participants were asked the same questions; answers were open-ended. Typical questions included:



Data Analysis

Besides formal and informal face-to-face meetings, we made use of two important computer-based tools, namely, e-mail for communication and the HTML authoring language for designing all the Web pages. Our electronic discussions kept the learning component of our creative efforts on track between our monthly meetings. This created what Crook describes as a narrative state—it furnished "a recognized platform for the next set of explorations" (1994a, p. 107) and established a longitudinal continuity within the dynamically constructed, common knowledge base of an electronically-linked learning community. Since our mediated conversations stemmed from common resources and shared prior experiences, comments were often directed toward groups of participants rather than to individuals.

We began by analyzing the content of the collected e-mail messages, following the coding strategies set forth in Miles and Huberman (1994). The coding process was framed by our original research questions regarding the processes that occur in a collaborative design group using a mediated environment, the group dynamics that support the design process, and the differences between face-to-face and mediated communication. Our intention was to develop a model that focused on the simultaneous processes of learning, design and communication supported by such environments. Four themes emerged from these messages that we identified as critical processes in effective mediated collaborative design:

1. participating in the collaborative design process by taking responsibility for assigned tasks and helping others do likewise;

2. developing a shared knowledge base;

3. tracing the emergence of new research questions; and

4. making explicit use of reflective strategies.

We analyzed the meeting notes and questionnaire responses using the same coding scheme that we had used for the e-mail messages. These other two data sources supported the existence of the four themes listed above. We did find that the communicative patterns in the two collaborative environments, namely, e-mail and face-to-face meetings, were clearly different.

Interpretation of Results

We found that the collaborative structure of our design process, with its associated learning component, enabled us to engage in the type of interpersonal communication that Pea (1994) terms transformative. In Pea's model, knowledge is restructured and all participants are transformed as learners by means of their communicative activities. We felt that we were also engaging in the type of on-line conversation that Bereiter (1994) describes as progressive discourse. Specifically, we had a commitment to work toward a common understanding that was satisfactory to all, a willingness to frame questions in such a way that evidence might be brought to bear on them, a commitment to expand the common body of knowledge as well as considering possible alternative solutions to problems we encountered, and a willingness to allow any belief to be subjected to criticism if it would advance the discourse.

Moreover, in order to move the design process forward without wandering aimlessly, we found that it was important for team members to co-construct a common conceptual ground, to establish a shared understanding and vision of the final product, and to share visual representations of the partially developed structure of the product as it evolved. In our case, we were working in a text based environment. We did not have the benefit of a shared visual workspace that could enable all members to simultaneously construct and edit graphic representations, nor did we have visual social cues that would allow members to "read" one another and see if they were really going to plunge into their agreed-upon tasks. As a result, this conceptualization and negotiation had to be done face to face. Records of our meeting notes and e-mail messages clearly portray this difference.

Whereas the face-to-face meetings enabled us to create the "big picture", the e-mail environment enabled us to carry on discussions about the rapidly evolving product. The pattern in the face-to-face conversations was one of negotiating shared meaning and understandings, assessing the diverse skills of individual team members, conceptualizing the design process, and agreeing upon who would do which task. In contrast, the e-mail conversations consisted primarily of question-and-answer sessions or conveying factual information such as:

Later, as the development phase proceeded, messages such as "Look at this...what do you think of my latest revision?" and responses to such probes became more common. Since we used the PINE mailing list feature so that we could keep track of the numerous messages, the team's e-mail messages were all public. Thus, each team member could learn from the efforts of all of the others.

We propose that the four key activities that comprise the design and learning process model interact and provide a framework for analysis of successful collaboration based on shared goals, shared understandings, and a common ground of knowledge. The components of our design and learning model, which are presented in Figure 1, include:

1. authentic task—identifying and taking ownership of a task whose cognitive challenges are consistent with the cognitive demands of the design environment for which we are preparing ourselves and our team members;

2. knowledge development—building a relevant, common base of knowledge, shared experiences and understandings, and associated skills;

3. research—formulating questions concerning the design process and group dynamics that emerge as a result of working on our task; and

4. reflection—consciously reflecting, monitoring, and making explicit our own cognitive orientations and processes in relationship to the group goal.




Figure 1

A Model of the Collaborative Design Process.




 

Using our model as an organizing structure for our research, we are presenting a summary of our findings and their implications for other designers anchored to the literature on collaborative learning. Our findings are organized as a set of principles that are very much in keeping with the collaborative learning model of Brown (1994). We include a few notable quotes from team members, along with their sources. (See Table 1 for the key to the three sources of data).

The Team Takes Ownership of the Task

In a distributed, collaborative environment, designers as learners take ownership of, and responsibility for, authentic tasks. This activity is foundational for the emerging design.

Group members begin designing as soon as they assess where they are and who is capable of taking responsibility for specific aspects of the emerging design. The design process doesn't just occur at the idea generation and group deliberation point. It begins when the actual design activities start to occur. "It links the intent and activities of the designers to the results that are produced when a designed object is experienced in practice" (Winograd, 1966). For a team member, this begins when that individual takes ownership of an authentic task.

Designing Involves Learning.

Designers decide not only what tasks they will take responsibility but but also what new skills and knowledge are necessary to support the emerging design.

Our findings demonstrated the strong link between designing and learning. Whenever a product emerges from the design and development process, as opposed to being constructed from a pre-defined blueprint, designing will require development of new skills and concurrent knowledge with the carrying out of design tasks.

We also saw a strong parallel between our team's efforts and problem-based learning (PBL). Savery and Duffy note, "When we are in a learning environment, there is some stimulus or goal for learning—the learner has a purpose for being there." (1995, p. 31). Bereiter and Scardamalia (1994) refer to this as intentional learning or goal-directed behavior. Moreover, the task must be authentic, that is, the cognitive demands of the task must be consistent with the cognitive demands of the environment for which the learners are preparing themselves.

In a traditional learning environment, instructors decide just what learners need to know, the behaviors that are appropriate for achieving their objectives, and the context in which their activities will be evaluated. Such artificial situations foster a dependence on the instructor, and as a result, learners are not prepared to function independently in complex situations with ill-defined problems. In learner-centered education, of which problem based learning (PBL) is one example, participants are charged with the responsibility for much of their own learning. It is up to them to know what knowledge they bring to the problem-solving situation, and to decide what they need to know, what resources are needed, and the most effective and efficient use of such resources (1996, pp. 115-16). Moreover, when the problems to be solved are relevant to the participants, and are subject only to the scope of the project, then their learning has a clear purpose, and they begin to take ownership of and responsibility for their work.

This concept of intentionality can be illustrated by the following responses from our team members:

We have an interesting group of very independent thinkers with a lot of personal ownership. That's positive because we have a high level of commitment to seeing this through right. We need to begin working with one another collaboratively on the product, and it's time to negotiate who will do what, and how. (E)

My goals were entirely collaborative. I knew we needed Home Pages for our School of Education, our doctoral program, and the instructional technology program. (Q)

I knew up front what my goals were: to learn the tools and to have a defined piece within which to explore and express that learning—the Ph.D. page met that need so I took it. (Q)

The Distributive, Collaborative Environment Supports the Authentic Task.

The distributive, collaborative environment supported knowledge construction and sharing of ideas while enabling individual members to continue work on their own tasks.

We also found, like Crook (1994b), that the use of computer-mediated communication and our shared space on the university's Web server evolved from tools for simply transferring information and building an on-line product to a full-fledged learning environment. This enabled our team to construct and share knowledge, to take individual ownership of specific sections of finished product and the resources used to create it, to engage in an ongoing dialogue, and to reflect upon our individual and group activities. The following quote illustrates how one team member used these shared tools deliberately and effectively:

After a couple of days I began to construct my own job aid to refer to. I started out with a written aid which I referred to for a brief amount of time. Soon I started an e-mail file called HP Tools. I dumped all sorts of items into that toolbox—hints from others, reminders to myself. It was interesting because I rarely had to go back to the resource after a few cycles of using a new process, but it was there if I needed it. (Q)

Our experiences with the varied use of the "Makepage" template and the directory structure created by each team member paralleled the observations of Wolcott and Robertson (1996)—each member of the team approached their work and the organization of their files in a distinctly different manner, personalizing it and projecting onto it their own degree of need for order and method.

Knowledge-building

A common knowledge base emerges from active dialogue and participation among those who seek to understand and apply concepts and techniques.

A shared language and set of representations (for example, how to structure a directory or lay out a page) is critical to developing a framework for group coherence. Unless each member of the team shares the same vocabulary and negotiated meanings of individual aspects of the product there is no way to communicate about the development of the product.

Tikhomirov (1981) states that information is essentially a system of signs or symbols, and that the processing of information deals with the various types of processing of these symbols according to given rules (1981 p. 260). If one simply looks at the manipulation of symbols, one is purely at the level of information theory. In contrast, activity theory (Vygotsky, 1978), of which Tikhomirov is a proponent, deals with the thought processes that occur among group members as they carry out activities that involve the mutual manipulation of these symbols and the mutual negotiation of their meaning.

Interpreting our findings according to Tikhomirov's framework, the type of learning that occurred in our group was mediated by our electronic network. Specifically, our symbols were our shared vocabulary, and our tool for manipulating them was the university e-mail system that supported our electronic conversations. Team members were able to immediately apply to the Internet design environment what was shared over e-mail. As Leont'ev argues, tools have a central role in activity theory since "the tool mediates activity and thus connects humans not only with the world of objects but also with other people" (1981, p. 55). This is the essence of computer-mediated communication.

Thus, we needed to initially formulate our own language conventions in order to support the collaborative design process. The group shared a language that described the situations they encountered and allowed them to ask questions, using a domain-specific vocabulary:

Everything moved really fast in our first two sessions together as a team. As a novice I needed hands-on experience to develop a framework for actual development because the terminology was like a foreign language and the design process strange and new. Two of the other team members were a big help here, clarifying common terms and commands, and showing me how to translate those into actual practice. (Q)

A couple of the experts used esoteric language. I remember going to Barnes and Noble and just sitting in the aisle of the computer section reading the books on the 'Net. It was part of my concept building. One of the team members with a computer background drove me to distraction at first with his terminology. I'd e-mail him for explanations and he'd respond, and sometimes I'd still not understand. There was another team member, also with a technical background. I could turn to him for a layman's explanation. He's a great teacher. Language can be a barrier to learning if you don't know the terminology. (Q)

Expertise Is Not Uniformly Distributed.

Expertise is irregularly distributed and develops at different rates for each individual in the community.

This process allows relatively novice designers to take advantage of the distributed expertise of the team and function in a manner more analogous to expert designers. Because roles are blurred in this type of environment, novices are able to participate in all aspects of the design process.

Hiltz (1994) sees the collaborative learning process as stemming from "co-labor", i.e., working together. Computer-mediated communication is particularly suited to the implementation of this process. This is especially important in building the "knowledge about knowledge" metacognitive skills—applying effort to meet a goal, contending with obstacles, and allocating mental resources properly—as illustrated in these extracts from e-mail messages:

I'm down at school now and I've copied the faculty disk for you. I went into the "talk" function at the system prompt, thinking you might be on and could tell me a way to get this to you electronically. It's the next skill I want to learn. (E)

I set up a structure on my Home Page that I was going to link the text files to so folks could look at what I wrote prior to next Wednesday's meeting. While I've mastered the upload/download to myself, I still need assistance uploading/downloading to you all. (E)

Citing Vygotsky (1978), Brown (1994) notes that learning occurs in a "zone of proximal development" between one's current level of learning and the level one can reach with the help of people, tools, and aids. Some members are already experts in certain specific areas; others are exercising cognitive activities and skills that are just emerging. Our members remarked that the design process was strengthened because of the distributed expertise of the group.

As a community of participants in the newly formed Home Page Task Force, we must take advantage of the1 individual talents of the participants and glean the absolute best from each member. (E)

Your initial role will be to keep us on track with realistic goals and pragmatic approaches to the initial steps in this process. (E)

As one of the early developers of a home page, I helped others get started on simple page development. As others became proficient, I identified a need: the existence of a page where people in the educational technology community could find up-to-date information about calls for participation and upcoming conferences. I then focused on developing and maintaining this capability. (Q)

That's what's nice about a collaborative effort. Because of the division of labor, nobody is swamped with the whole project, and everyone can let his or her own talents shine. We have a variety of talents and lots of energy. That's what makes a team function really well. (E)

Experts in the group coached novices and provided scaffolding. Novices began to take on more independent activities, and fading occurred as the experts had to answer fewer and fewer elementary questions. Evidence exists that assumption of roles was an active process.

Once I started exploring, two other team members were always there to give me, as well as the other team members, up front information so I could just get into the process. I asked lots of questions over e-mail. They were always incredible in the speed with which they responded—sometimes within minutes of my on-line questions. Someone always responded within a 24 hour period. Nothing was ever considered insignificant and all questions were answered with respect. As I was building my own cognitive tools, it was sometimes helpful to have clarification from more than one source. (Q)

This process is similar to Aronson's (1978) Jigsaw method, in which learners are divided into small groups, and each group is assigned an area in which to develop expertise. Members within the same group meet regularly to discuss their common responsibility. Having mastered their subject area, they then return to the cooperative learning group and pass on their information to the others. In our case, each "group" consisted of one or two individuals who had already developed expertise in some specific areas of interest to the team as a whole, such as uploading/downloading files to the server, UNIX commands, Web page design, HTML authoring, technical writing, and so forth. A major advantage of this distributed expertise, shared in jigsaw fashion, is that the group as a whole benefited from the increased range of expertise and the combined knowledge of the team members.

Skills and strengths were dispersed irregularly among different team members. Somehow the project achieved a focus and people's talents were put to very good use. (Q)

Hewitt and Scardamalia (1997) state that a computer-supported learning environment can serve not only as an on-line conferencing facility but also as a true learning environment if it enables participants to represent a problem from multiple perspectives, to build knowledge communally, and to examine knowledge and refine design elements at different levels of abstraction.

In retrospect, our learning environment did exactly that. We synthesized information by using our distributed expertise; our combined knowledge base became greater than the sum of the individual knowledge contributions. We refined design elements collaboratively by having several people work on the same page, in an iterative fashion. We also represented problems from different perspectives—in particular, the use of a template for creating home pages automatically vs. doing step-by-step HTML coding and using UNIX commands—and then discussed the relative merits and drawbacks of each method. Moreover, though not each person participated in all phases of the on-line discussion, team members valued the "polish" that was apparent in the finished product after an extensive on-line conversation and subsequent revisions.

There is no doubt in my mind that we're able to accomplish more collectively than individually. More gets done, and it gets done better. Feedback from team members can be difficult to take, if you're being told to start over or redo a section. But the undoing and redoing results in an improved product. (Q)

Confrontation Helps Restructure Knowledge.

The process of confrontation and conflict among group members, rather than being viewed as a disadvantage, brings about fundamental cognitive restructuring.

Our findings demonstrate the critical nature of context in resolving differences. We found that in negotiating conflicts, a face-to-face meeting is generally more suitable than e-mail conferencing due to the presence of non-verbal social context cues among participants and our ability to work on a shared representation or an evolving idea at the same time. Once we had negotiated issues regarding the "big picture" and learned more about one another's' communication styles, we were able to accomplish most of the design tasks via mediated communications. We began to view challenges as opportunities to improve our design.

Brown and Palincsar note the important role of the skeptic in group discussions: "by forcing the group to defend or elaborate solutions, a more mature resolution will emerge." (1989, p. 403) Areas of conflict brought about negotiation and restructuring of our knowledge about the task.

I emphasize that front-end planning is not as important as just "doing it". This is why I suggested that a group effort might get in the way of individual creativity. (E)

Well, I'd like to know who is going to maintain the page? We just can't keep letting that issue slide by.(Q)

You're a "Devil's Advocate" who constantly reminds us to look at all facets of things as we charge on ahead! (E)

Brown and Palincsar also argue that situations which encourage dissatisfaction with the existing state of knowledge tend to foster conceptual change. Environments that encourage questioning, evaluating, and criticizing the status quo are considered to be fruitful breeding grounds for restructuring knowledge (1989, p. 395), as illustrated in the following response to the questionnaire:

Restructuring generally occurred after E-mail discussion identified changes that were necessary. As a general rule, changes occurred only following collaboration with other team members. (Q)

Through group planning, the team determined the requisite conditions for the design. The individual designer then determined if the affordances of the technology and the look and function of the newly created page matched the constraints of the domain, i.e., how the group agreed the page should look and act. If the individual product failed to meet the group's criteria, individuals began to make adjustments without having to be told to do so by the group.

There must be understanding and agreement of the requirements and objectives that guide each member's effort. There must be a least a general understanding of the task flow and the information flow from member to member. There must be a clearly identifiable product to be created by each member and a time for its delivery, with the time being consistent with available resources. (E)

The Team Becomes Cohesive.

A disparate group self-organizes into an interdependent community of learners, with common discourse based on shared expertise, strategies, values, and representational tools.

Our team self-organized in a way that capitalized on the strengths of individuals with dispersed knowledge. Members began to share leadership and discourse activities which are normally reserved for a mentor or manager, while the mentor became a full participant in the design process. At various times, different members of the group took the lead in helping the team manage time and resources. The following excerpts from e-mail messages illustrate this process:

[Two of us] met this afternoon and developed a conceptual outline for our Home Pages. We can use this as a template for the division of labor. (E)

I'd like to propose some target dates or milestones for the next two weeks in order that we end up as a group having produced a Home Page product in a specific amount of time. It's time we make some of our efforts into a concrete draft product. Let's get something developed so we can try it out and then start evaluating the results. (E)

This shared leadership contributed to group coherence, as reported by two team members in the questionnaire:

I learned a lot as I went along—mostly about the incredible energy and initiative brought to projects by the students themselves. I learned that groups have a way of self-organizing in a way that capitalizes on the strengths of individuals. (Q)

I gained a renewed appreciation for the total interdependency of the network. You can't have total consistency; you need so many pieces to be up and running, and you have to depend on those pieces being there for you. (Q)

Collis notes that problems may arise in maintaining coherence and momentum in group projects, especially in formal university courses, as students become immersed in their respective projects. Her solution is to integrate all aspects of the course, theoretical and project-related, in a single course Web site, accessible to all, and making group progress visible to all (1997, p. 112). By integrating all of our team efforts in a single Web site, with "class" (i.e. read/write/execute) permission for all members, and by using the PINE address feature of the e-mail system to keep the whole group connected electronically, we effectively implemented the solution that Collis suggests.

What helped to keep our team on track? Using data from surveys, interviews, and e-mail messages, Herrmann (1995) found that there were three communication functions taking place on listservs: academic, administrative, and community-building. Using our own triangulated data, we found three similar patterns in the communicative activities of our design team: informative messages—posing and answering questions and conveying important information; administrative messages—keeping individual team members on task and the project as a whole on track; and supportive messages—giving positive feedback and moral support to people who were really working hard and doing a good job.

If we had limited our communication strictly to administrative communications such as keeping individual members on task, or to informative communications such as posing and answering questions and conveying factual information, without adding supportive communications such as complimenting individuals on a job well done or acknowledging their stress level and helping them through a difficult situation, it is unlikely that our team's energies would have been sustained as long as they were.

Research Questions Emerge

Research occurs within the group and extends to resources outside the group.

It is in the process of reconceptualizing the design, updating its structure, and iteratively observing and testing it, that leads to the emergence of new ideas and questions. That is what we mean when we say that new research questions emerge from the process.

Research Questions Emerge from the Iterative Design Process.

Research questions emerge from, as opposed to driving, the iterative design process.

Research may be either deductive or inductive. Brown and Campione (in press) describe a deductive knowledge-building process in their Fostering Communities of Learners (FCL) project. Students start by engaging in research in order to share knowledge and expertise with their group. Their efforts culminate in an authentic task which demands that all participants have learned about all aspects of a joint topic. Problem-based learning (PBL), in contrast, is an inductive knowledge-building process often used by pre-professionals in medical or business school. The facilitator poses a practical problem; the learners must work collaboratively to find the resources to solve it. This is a cognitive apprenticeship environment in which novices are supported while they develop the metacognitive skills to solve typical problems that they will encounter later in actual practice (Savery & Duffy, 1995).

Though our design and learning process model uses similar building blocks as Brown and Campione, we found that we applied ours in quite a different manner. Our activities were more like those in an open-ended, iterative PBL scenario than a closed FCL process. Our interactions, both in the design component and in the learning component, reflected the same overall aims and goals of PBL, namely, to enable team members to learn skills that could sustain problem-solving throughout professional life. Foundational to sustaining the design process is the ability to ask pertinent questions regarding the issues at hand.

The group's knowledge was socially negotiated. Conflict, questioning, and requesting clarification played a part as the group began to articulate new research questions:

Computer networks may be the best solution to support collaboration when collocation is not possible, but are they as effective as collocation of participants? (E)

The template lessens the cognitive load, but it's beginning to stifle creativity. At what point do we remove the scaffolding? (M)

It is at this stage that we see a distinct difference between our group's collaborative design/learning process and that of other models in the literature. Whereas other models tend to begin with student-generated research questions, in the model presented here, research questions emerged as part of the design and learning process. This spontaneous generation of questions facilitates the sort of participatory, iterative design that Gould and Lewis (1985) aspired to and thus represents an important outcome of the collaborative design process.

Expertise can be Viewed as Performing Expertly within a Context.

The process of generating questions within a design environment that supports learning, helps move novice designers toward performing expertly within a context.

Over the months of our design/development phase, our group members slowly moved from a novice perspective to that of an expert within the design context. Rowland (1992) studied the differences between novice and expert designers. Novices tend to think linearly, breaking the design process into parts, considering local factors one at a time, and then jumping into the problem solution phase as quickly as they can. Experts see the design problem more globally and contextually. Since the problem definition is more systemic and dynamic, experts tend to spend more time on this phase of design before moving to the problem solution.

Novices tend to view problems as well-defined, because that is the way they have been taught—to use prescribed formal design techniques to produce a good design. Nanard and Nanard (1995) observed the behavior of good designers and found that they move back and forth between a set of design processes and implementation/testing processes in a complex and unpredictable fashion. The activities within the design space are determined by the form that the design is taking at any point in time—representing the epitome of flexible thinking and design. Traditional bottom-up and top-down design processes are bridged systematically by recursive cycles of iterative conceptualizing, prototyping, testing and revising. The observation and evaluation usually leads to reconceptualization of the design and to the emergence of new ideas and questions.

Reflection

Consciously monitoring and making explicit our own cognitive orientations and processes in relationship to the goals of the group is an integral part of the design and learning process.

Reflection is critical to the dialogue process. It means monitoring where each individual is at all times with regard to the activities of the group. Suspending beliefs and assumptions in a collaborative environment supports the process of dialogue because it makes possible the exploration of underlying, tacit thinking that results in a particular conclusion (Isaacs, 1996).

Reflection Supports Iterative Design.

Reflection on questions generated by the team as well as the skills and knowledge needed to move the process forward is key to a new design cycle.

Collis notes that when groups work asynchronously, without direct supervision, and from different locations, maintaining a balance in the efforts of each member can become a serious problem (1997, p. 118). The success in carrying out a task depends on the group's commitment to reflective practice and making tacit cognitive processes explicit. Hence, for us, reflection —which we define as the conscious monitoring of our own design decisions—became the starting point for the iterative cycles of both the design and learning components of our collaborative process.

We began with several brainstorming sessions to determine what we knew, what we didn't know, and what we needed to know so that we could begin to locate sources of information and skills which would be needed later on. This process continued throughout our dialogue and design conversations, both mediated and face-to-face. Our continuing conversations enabled us to identify individual strengths and weaknesses and to create a good match between individual skills and necessary tasks. Then, we proceeded to develop the requisite skills and common knowledge base for us to take ownership of our respective tasks.

A Reciprocal Relationship exists between Individual and Group Activity.

It is within group activities supported by dialogue and purposeful shared reflection that the design process moves forward.

[Two of us] began developing the Ph.D. page by identifying our individual strengths and setting realistic timelines. At this point my questions were directed to two other group members and my partner utilized a third member as her primary technical resource. (Q)

We also found a reciprocal relationship between individual and group activities. Learning is socially mediated. Group discussions, goals, and plans influence individual action and construction of knowledge (Vygotsky, 1978).

Everything moved really fast in our first two sessions together. Novices need hands-on experience to develop a framework for actual design and development because the terminology is foreign and the process strange and new. Next I just started exploring. Two of the members gave us enough up front information to get into the process. I began asking lots of questions over E-mail. I got use to the rhythm of working on my own but knowing when others would be available so I could ask questions that would extend my understanding. With time I turned to other team members for help because of the direction the e-mail discussion was taking us. The group's input was not only essential to but influenced the direction of my own work. (Q)

In this reciprocal group-individual interaction which we term "shared reflection", group discussions, including negotiation and conflict resolution, goal setting, and planning influence individual action within parameters set by the group. When this could not be accomplished via e-mail, one member remarked:

In the light of the diverse remarks presented via last week's e-mail regarding the purpose and intent of the Home Page project, I think a face-to-face meeting to establish a common ground is warranted. (E)

By comparing the types of discussions carried on via e-mail and in face-to-face meetings, we found that design conversation occurred more frequently in face-to-face meetings than in electronic discourse. Berge (1997) notes that nonverbal cues, body language, and other social context cues associated with face-to-face conversations are missing in text-based e-mail conversations. This allows participants to concentrate on the content of the message rather than the presenter. However, in negotiation of meaning and conflict resolution, it is often important to judge the general mental state of the participants by carefully noting their nonverbal, as well as their verbal, conversation. One team member brought up this very issue in an e-mail message:

What should be the proper mix of face-to-face communications to optimize communications and knowledge-building within a collaborative group of designers? (E)

Koschmann, Kelson, Feltovich and Barrows (1996) identified two phases that occur in group discussions in PBL: idea generation and group deliberation. Group deliberation can be best conducted through spoken discourse and free-wheeling interpersonal interactions rather than mediated conversations. In contrast, Berge (1997) notes that written communication tends to be more reflective than spoken interaction. This may account for the fact that most conflicts among our team members were resolved face-to-face, whereas most sharing of knowledge was done via e-mail.

Through design conversation, if an individual can begin to think like a group—with multiple perspectives, honoring complexity—learners can develop advantageous skills for thinking about complex subject matter—whether a group is present or not. This is the essence of Cognitive Flexibility Theory (Feltovich, Spiro, Coulson, & Feltovich, 1996). It enables an individual to contemplate a given situation from the general world perspective of other individuals—to see the world through another person's eyes. Both Piaget (1926) and Vygotsky (1978) consider that mature thinkers who have internalized and individualized the roles of the group members will tend to carry on their own internal dialogue, which consists of planning, guiding, and monitoring their thoughts and actions.

In looking at the actions of expert designers, Rowland notes that they are able to design flexibly by establishing "weak links" between the design problem and its solutions. In contrast, novices tend to form "hard links" that they are not inclined or not able to weaken. (1992, p. 78). While we were novice designers, as a group we functioned more as an expert team because of the distributed expertise and multiple perspectives of individual team members. The importance of "weak links" to effective team membership was illustrated in our own group. One of our members, who was very involved in the initial conceptualization of the project, continued to work in his own narrowly defined way even at those times when the consensus of the group involved revising the nature and scope of the project. By maintaining his "hard links", and by not suspending his assumptions and preconceived notions during the face-to-face meetings where differences of opinion were being negotiated, he never really moved into the dialogue. Thus, he was unable to participate fully in the design conversation and eventually dropped out of the team.

Yakimovicz & Murphy (1995) found that participants in an electronic discussion group worked together, shared adversities and successes, increased their self-concept, and expanded existing awareness. As they reflected upon their experiences with problems, they coupled them with their individual and collaborative solutions to make meaning out of their cooperative venture. This is similar to the reflective component that was central to our own design process model.

Whether or not the Home Page becomes a reality, the community of discourse is valuable. Let's continue the discussion further before we move on to a specific design. (E)

Our goal of completing what we considered to be an authentic task was enhanced by our commitment to reflective practice. One of the goals of cognitive apprenticeship, as described by Collins, Brown, and Newman (1989) is for members of the group to monitor the relationship between their outputs and the target performance they aim for. Our group had a conceptual model of what they wanted the final product to look like, and they continually updated it. Initiation came from the group through individuals, as well as from individuals to the group.

I think the question that is facing our design team is, how can we, as individuals within an academic division at a recognized research university, best reveal our work and interests to the outside world? (E)

If philosophical support for the Home Page exists, we need to be asking if this tool best represents our collaborative vision or if other modes of expression do that better. (E)

It might be helpful to have a Home Page that serves as a threshold to environments that are held together by themes where uniqueness of individual thought is represented as well as our collaborative efforts. (E)

As we worked, there were issues to be dealt with, differences of opinion to be negotiated, and more questions to be answered. Thus, it was in the process of sharing knowledge and skills and purposeful shared reflection while carrying out the task that we generated our research questions, not the reverse. Constructing our knowledge base roughly paralleled Brown's (1994) description of knowledge construction among a community of learners.

In Retrospect

Nearly three years have passed since the team first started designing the Home Page. Since then, old members have graduated or dropped out, and new members have joined. The mentoring and collaborative knowledge-building process continues as new members are enculturated into the group and are beginning to design additional pages for professional societies, doctoral portfolios and dissertations, scholarly products from doctoral research laboratories, schoolwide grants, and other activities that were never envisioned at the start of the project.

[A new member] demonstrated her K-12 resource page, and we were all impressed with the scope and number of resources available. She got some suggestions and encouragement for continuing her work this semester. (M)

Professors in disciplines outside the original ILT/CLT cohort are now becoming involved in the participatory design process as members of the Internet Task Force guide them through the initial process of learning HTML and designing pages. This role reversal is reminiscent of the original merging of roles that occurred in the team's original design efforts. Here are some comments from a professor who has recently begun to collaborate with the design team:

When [a new member] gets his Doctoral Program Page completed, I suggest that we undo the link you made, and instead insert a link to the Doctoral Program, more conspicuous than the current one nested under the Information for Researchers, and then link through that to the new research laboratory Home Page. Meanwhile, [an original member] and I are changing the initial screen of the research laboratory home page to match [the new member's] format. It is an attractive format, and there is perhaps something to be said for each lab having a similar format for their initial page. By the way, I think the Doctoral Committee would like to know about [the new member's] work on the Doctoral Program Home Page. I suggest that he contact [the chair of the program] to get on the committee's agenda for the first spring meeting. (E)

Since the faculty are beginning to network with the change agents on our team and within the School of Education, both interest and support are being generated at the highest levels.

This semester, twenty new students have applied to our programs from out of state. We found out that they discovered our School through the Web Page. We are interested in expanding our program pages, and we will even take over the effort if you will teach us the knowledge and skills we need to do this. (M)

Originally, the administration was not involved in the project at all, nor did they outwardly support it. However, with the School's current presence on the World Wide Web and the influx of new funding as a result (an unintended but very positive side effect!) they are beginning to take notice and are requesting staff development from the design team.

[A new member] , with the help of [an old member], will assume the position of the point person on our collective assignment to support the School of Education staff. She will work 8-20 hours per week and will work closely with [the Dean's Assistant] in coordinating the overall effort. Please contact her if you would like to provide Web development support to staff members, mostly in the establishment of Web pages for programs and services within the School. (M)

As a result, some pages are now being redesigned and expanded, and new pages are being developed to meet the needs of the staff. The scope of our work has also changed as our academic community has become a integral part of the larger Internet culture. This has taken place primarily through dissemination of our magazine and journal articles, proceedings papers, book chapters, and other scholarly products. These on-line documents serve as excellent models for the various types of portfolio products that UCD doctoral students are expected to create, and, as such, are invaluable to students who have recently entered the program. They also inform corporate trainers, staff developers, and graduate students from other organizations about our ongoing research and participation in the global instructional design community.

Though the creation, sharing, and global dissemination of scholarly products was part of the original vision of the design team, it was the last phase of the project to be adopted by the School. Interestingly, it was the most important from the global community's point of view. The K-12 And Beyond page alone is accessed through links from about fifty different sites on the Web. With the proliferation of student-generated pages, the new interest shown by our professors in showcasing their research laboratories, and the support by the design team for the HTML authoring classes and staff development workshops, it is simply a matter of time until the information dissemination aspect of the project attains the same level of success as the use of the Home Page for recruiting students, accessing information, and building knowledge.

Conclusion

What have we learned from our efforts that can be generalized to other design teams and environments, academic or corporate? While the case presented and the model explicating the collaborative experience were specific to our own group, we believe our skills as collaborative designers have been enhanced. The strength of the model is in making designers aware of information that is often implicit and processes that are often covert .

As team members began to develop expertise in their chosen areas, their initial roles began to evolve. Whereas in Scardamalia and Bereiter's research (1994), the moderator stays in command, our own advisor evolved, learned, and became a participant. Participants took on leadership roles such as keeping up the team's energy, moving us toward our goal, providing resources, and giving feedback. This process is continuing as faculty, staff, and students in other programs begin to collaborate with the team.

No one member possessed the expertise of the entire group. The group became a self-organizing system with members contributing their unique expertise and, in turn, learning new skills and extending the group's common knowledge base. New students are being enculturated into the group, and are rapidly becoming productive members of the design team. Research questions dealing with adoption/diffusion issues are now beginning to arise, as the use of the Web Page is spreading throughout the School as a whole. (See Rogers, 1995.)

Our interactions were supported by, but not limited to, electronic communication. In moving beyond using computers as tools or objects to be programmed, we have begun to view them as a shared representational space that can support knowledge building and design. The interactive processes described by our model—working together on an authentic task, developing a shared knowledge base, allowing research questions to emerge from the design and development process as the product is constructed, and intentionally sharpening our individual reflections through group interactions —supported the development of our own expertise. As the project evolved, the technology became more and more transparent to us, and instead of fighting the technology, we began to mold the tools to our needs. Eventually the computer mediated environment became the interface between the minds of the team and the design environment. Our model and the principles supporting it can assist others undertaking collaborative design projects.

References

Argyris, C. (1985). Action Science. San Francisco: Jossey-Bass.

Aronson, E. (1978). The Jigsaw Classroom. Beverly Hills, CA: Sage Publications.

Bereiter, C. (1994). Implications of postmodernism for science, or, science as progressive discourse. Educational Psychologist , 29 (1), 3-12.

Bereiter, C., & Scardamalia, M. (1989). Intentional learning as a goal of instruction. In L.B. Resnick (Ed.), Knowing, learning, and instruction, Hillsdale, NJ: Erlbaum.

Berge, Z. (1997). Computer conferencing and the on-line classroom. International Journal of Educational Telecommunications, 3 (1), 3-21.

Brockmann, R.J., & Sinatra, S. (1995). How the iterative process helped the Allies win the Persian Gulf War. STC Intercom , 42 (9), 1, 44.

Brown, A. L. (1994). The advancement of learning. Educational Researcher 23 (8), 4-12.

Brown, A.L., & Campione, J.C. In press. Psychological theory and the design of innovative learning environments: On procedures, principles, and systems. In Contributions of instructional innovation to understanding learning, edited by L. Schauble & R. Glaser. Hillsdale, NJ: Erlbaum.

Brown, A.L., and Palincsar, A.S. (1989). Guided, cooperative learning and individual knowledge acquisition. In L.B. Resnick (Ed.), Knowing, learning, and instruction. Hillsdale, N.J: Erlbaum.

Collins, A., Brown, J.S., & Newman, S.E. (1989). Cognitive apprenticeship: Teaching the crafts of reading, writing, and mathematics. In L.B. Resnick (Ed.), Knowing, learning, and instruction, Hillsdale, N.J: Erlbaum.

Collis, B., Andernach, T., & Van Diepen, N. (1997). Web Environments for group-based project work in higher education. International Journal of Educational Telecommunications, 3 (2/3), 109-130.

Crook, C. (1994a). Computers and the collaborative experience of learning. London: Routledge.

Crook, C. (1994b). Computer networking and collaborative learning within a departmentally focused undergraduate course. In H.C. Foot, C.J. Howe, A. Anderson, A.K. Tolmie, & D.A. Warden (Eds.), Group and Interactive Learning. Southampton, UK: Computational Mechanics Publications.

Edelson, D.C., Pea, R.D., and Gomez, L. (1996). Constructivism in the collaboratory. In B.G. Wilson (Ed.), Constructivist learning environments: Case studies in instructional design,. Englewood Cliffs NJ: Educational Technology Publications.

Feltovich, P.J., Spiro, R.J., Coulson, R.L., & Feltovich, J. (1996). Collaboration within and among minds: Mastering complexity, individually and in groups. In T. Koschmann (Ed.), CSCL: Theory and Practice of an Emerging Paradigm. Mahwah, NJ: Erlbaum.

Gagne, E., Yekovich, C., & Yekovich, F. (1993). The Cognitive Psychology of School Learning, Second Edition. NY: Harper Collins.

Gelman, R., & Greeno, J.G. (1989). On the nature of competence: Principles for understanding in a domain. In L.B. Resnick (Ed.), Knowing, learning, and instruction, Hillsdale, N.J: Erlbaum.

Gould, J.D., & Lewis, C. (1985). Designing for usability: Key principles and what designers think. Communications of the ACM, 29 (3), 300-311.

Hanson, N., & Gladfelter, J. (1995, January). Teaching graduate-level seminars using electronic mail: Creative distance education. DEOSNEWS, 5 (1). Retrieved February 1, 1995 from the World Wide Web: Send e-mail to LISTSERV@PSUVM.PSU.EDU with the message GET DEOSNEWS LOG9105

Harel, I., & Papert, S. (1993). Software design as a learning environment. Interactive Learning Environments , 1 (1), 1-32.

Herrmann, F. (1995). Listserver communication: The discourse of community-building. Indianapolis: Proceedings of the CSCL'95 Conference. Retrieved August 16, 1997 from the World Wide Web: http://www-cscl95.indiana.edu/cscl95/herrmann.html

Hewitt, J., Scardamalia, M., & Webb, J. (1997). Situative design issues for interactive learning environments: The problem of group coherence. Paper presented at the Annual Meeting of the AERA, 1997, Chicago.

Hiltz, S.R. (1994). The virtual classroom. Norwood, NJ: Ablex.

Isaacs, W.N. (1996, January-February). The process and potential of dialogue in social change. Educational Technology, 20-30.

Ishii, H., Kobayashi, M., & Arita, K. (1994). Iterative design of seamless collaboration media. Communications of the ACM, 37(8), 83-97.

Jenlink, P., & Carr, A.A. (1996, January-February). Conversation as a medium for change in education. Educational Technology, 31-38.

Koschmann, T., Kelson, A.C., Feltovich, P.J., & Barrows, H.S. (1996). Computer-supported problem-based learning: A principled approach to the use of computers in collaborative learning. In T. Koschmann (Ed.), CSCL: Theory and Practice of an Emerging Paradigm, pp. 83-124. Mahwah, NJ: Erlbaum.

Koschmann, T.D., A.C. Myers, P.J. Feltovich, and H.S. Barrows. (1994). Using technology to assist in realizing effective learning and instruction: A principled approach to the use of computers in collaborative learning. The Journal of the Learning Sciences 3 (3), 227-264.

Lawyer-Brook, D., & Sherry, L. (1997). Boulder Valley Internet Project: Final Report. Denver: RMC Research Corporation.

Leont'ev, A.N. (1981). The problem of activity in psychology. In J.V. Wertsch (Ed.), The Concept of Activity in Soviet Psychology., pp 37-71. Armonk, NY: M.E. Sharpe, Inc.

Martinez, J., Billig, S., Sherry, L., & Hoffman, D. (1997). The WEB Project: Progress Report. Manuscript in preparation: RMC Research Corporation.

Miles, M.B., & Huberman, A.M. (1994). Qualitative Data Analysis. Thousand Oaks, CA: Sage Publications.

Nanard, J., & Nanard, M. (1995). Hypertext design environments and the hypertext design process. Communications of the ACM, 38 (8), 49-56.

Patton, M.Q. 1990. Qualitative Evaluation and Research Methods. Newbury Park, CA: Sage Publications.

Pea, R.D. (1994). Seeing what we build together: Distributed multimedia learning environments for transformative communications. The Journal of the Learning Sciences , 13 (3), 285-299.

Piaget, J. (1926). The Language and Thought of the Child. London: Routledge & Kegan Paul.

Richey, R.C., and W. Nelson. (1996). Developmental research. In D. Jonassen (Ed.), Handbook for Research in Educational Communications and Technology,. NY: Simon & Schuster Macmillan.

Rogers, E.M. (1995). Diffusion of Innovations, Fourth Edition. NY: The Free Press.

Rossett, A. (1991). Needs assessment. In G.A. Anglin (Ed.), Instructional Technology, pp. 156-169. Englewood, CO: Libraries Unlimited.

Rowland, G. (1993). Designing and instructional design. Educational Technology Research and Development , 41 (1), 79-91.

Rowland, G. (1992). What do instructional designers actually do? An initial investigation of expert practice. Performance Improvement Quarterly, 5 (2) pp. 65-86.

Ryder, M. (1994). Augmentation of the Intellect. Retrieved August 16, 1997 from the World Wide Web: http://www.cudenver.edu/~mryder/augment.html

Sarason, S.B. (1988). The creation of settings and the future societies. Cambridge, MA: Brookline Books.

Savery, J.R., and Duffy, T.M. (1995). Problem based learning: An instructional model and its constructivist framework. Educational Technology, 35 (5), 31-38.

Scardamalia, M., & Bereiter, C. (1991). Higher levels of agency for children in knowledge building: A challenge for the design of new knowledge media. Journal of the Learning Sciences, 1 (1), 7-68.

Scardamalia, M., & Bereiter, C. (1994). Computer support for knowledge-building communities. The Journal of the Learning Sciences , 13, (3), 265-283.

Schon, D.A. (1987). Educating the reflective practitioner: Toward a new design for teaching and learning in the professions. San Francisco: Jossey-Bass.

Seels, B.B., & Richey, R.C. (1994). Instructional technology: The definition and domains of the field. Washington, D.C.: Association for Educational Communications and Technology.

Sherry, L. (1995). Results of the TSS survey on e-mail usage. Retrieved August 16, 1997 from the World Wide Web: http://www.cudenver.edu/public/education/edschool/survey_results.html

Sherry, L. (1997). A Re-Analysis of the 1995 TSS E-mail Survey. Unpublished manuscript: University of Colorado at Denver.

Sherry, L., and Morse, R. (1995). An assessment of training needs in the use of distance education for instruction. International Journal of Educational Telecommunications , 1 (1), 5-22.

Sherry, L. & the UCD Internet Task Force. (1996). Supporting a networked community of learners. TechTrends , 1 (4), 28-32.

Sherry, L., & Wilson, B. (1996). Supporting human performance across disciplines: A converging of roles and tools. Performance Improvement Quarterly, 9 (4), 19-36.

Tennyson, R.D. (1995). The impact of the cognitive science movement on instructional design fundamentals. In Instructional design fundamentals: A reconsideration, edited by B.B. Seels. Englewood Cliffs, NJ: Educational Technology Publications.

Tikhomirov, O.K. (1981). The psychological consequences of computerization. In J.V. Wertsch (Ed.), The Concept of Activity in Soviet Psychology., pp. 256-278. Armonk, NY: M.E. Sharpe, Inc.

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.

Wilson, B.G., Hamilton, R., Teslow, J.L., & Cyr, T.A. (1995). Technology making a difference: The Peakview Elementary School Study. Syracuse, NY: ERIC Clearinghouse on Information and Technology.

Wilson, B.G., Ryder, M., McCahan, J., Sherry, L.,& Myers, K.M. (1996). Cultural Assimilation of the Internet: A Case Study. Paper presented at the AECT In-CITE '96 Conference, Indianapolis, Indiana.

Winograd, T. (1995). From programming environments to environments for designing. Communications of the ACM, 38 (6), 65-74.

Winograd, T. (1996). Bringing Design to Software. NY: ACM Press.

Wolcott, J.R., & Robertson, J.E. (1997). The World Wide Web as an Environment for Collaborative Research. International Journal of Educational Telecommunications, 3 (2/3), 219-236.

Yakimovicz, A.D., & Murphy, K.L. (1995). Constructivism and collaboration on the Internet: Case study of a graduate class experience. Computers in Education , 24 (3), 203-209.

Yin, R.K. (1994). Case study research: Design and methods, Second edition. Thousand Oaks, CA: Sage Publications.