Social Considerations in the Development, Deployment and Adoption of
Web-Based Organizational Memories
Andrew Gorman
Center for LifeLong Learning and Design
Department of Computer Science and
Institute of Cognitive Science
University of Colorado
Boulder, CO 80309-0430
Abstract
This paper begins with the premise that people want t o
participate in designing their future. While an individual
unaided human mind is powerful, real power is derived from
humans working in conjunction with tools and other humans. Developing computational support for collaboration among groups of individuals is a difficult task, but its
potential benefit is tremendous. This paper will describe
three broad models of collaborative systems along with
examples of systems that typify each model. The sociotechnical aspects of collaboration and participation will be
discussed as they relate to the development, deployment,
and adoption of an organizational memory system intended
to support the collaborative design of a new building.
Introduction
The Discovery Learning Center (DLC) is a new building
being constructed on the campus of the University of Colorado at Boulder. This process gathers groups of people who
have varying backgrounds and interests. The goal that is
common to all is the construction of a new building. However, each has his or her own unique agenda. In order to support such a process, it is desirable to create an information
space that can be useful to all stakeholders as they participate in the design and construction of the new building. The
DLC information space began as a static repository of design documents and background information. It has since
grown into a more dynamic system for disseminating design
alternatives and gathering feedback from stakeholders. The
adoption of such a system by stakeholders who have little
prior exposure to one another can be a very complicated
process, which is affected by both technical and social factors.
Information Access is Necessary but not
Sufficient
In today's workplace, people need to know how to access
information. However, simply knowing how to access information falls short of what is truly needed in today’s
workplace. For example, in the President’s Information
Technology Advisory Committee report (PITAC, 1999),
there is a call for ubiquitous information access. This call
for accessibility needs to be extended to include the understanding that the key to the future lies not only in greater
access to information, but in greater support for knowledge
construction (Scardamalia & Bereiter, 1994). Although there
is value in such an access model, its focus is incomplete. It
is based on an impoverished view that relevant knowledge
already exists, waiting to be accessed. In order to truly gain
the benefits of information technology, what is needed is not
simply greater access to information, but a greater ability for
average people to construct and distribute new knowledge
(Arias, 1999 (in press)).
People Want to Participate
How can more than 261 million individual Americans define
and reconcile their needs and aspirations with community values and the needs of the future? Our most important finding is
the potential power of and growing desire for decision processes that promote direct and meaningful interaction involving
people in decisions that affect them. Americans want to take
control of their lives (PCSD, 1996, p.7).
This finding of the President's Council on Sustainable
Development (PCSD) supports the claim that something
more is needed than access alone. Therefore, an important
challenge for future information technology is to enable
stakeholders of problems to become involved informed participants (Brown, Duguid, & Haviland, 1994).
To make informed participation a reality, we need support
for new forms of knowledge creation, integration, and dissemination. People seldom explore large repositories of information in the abstract (Fischer, Lemke, McCall, &
Morch, 1996; Moran & Carroll, 1996). Instead, information
is typically sought in response to breakdowns encountered
during meaningful, real-life activities (Fischer, 1994; Popper, 1965). By overcoming such breakdowns, new knowledge is created, which then must be integrated with any
knowledge that may have been generated during prior breakdowns. This cycle leads to the creation and evolution of rich
information spaces that can empower interested community
members as they “take control of their lives.”
Exam #3
2
M1-M3 Models of Participation
A typical model of information sharing (e.g., (Ackerman
& Malone, 1990; Ackerman & McDonald, 1996)) focuses
on experts sharing information with non-experts. This is
through a single person or small committee. This is typical
of information generated in an open source model of software development (Raymond, 1999). Here there are many
contributors, but only a few (or one) that integrates feedback
from the community back into a coherent structure. There
has been much success using this model in open source
movements (Fielding, 1999; Torvalds, 1999) and while this
is an advantage over the M1 model, in terms of collaborative
construction, it can lead to problems of scalability. Furthermore, there needs to be a highly dedicated person that has
the full-time responsibility of analyzing and structuring all
of the feedback.
The M3 model (figure 3) can theoretically support distrib-
Figure 1 - The M1 Model of Collaboration
represented by the M1 model (seen in figure 1) in which a
class of experts controls the production of information and
individuals act as consumers whose only need is that of access.
In the M2 model (figure 2), all information is funneled
Figure 3 - The M3 Model of Collaboration
uted collaboration in a more direct way. In this model, there
is no “gatekeeper.” All contributors have the ability to add
content directly. Developing systems that support this type
of collaboration can be extremely difficult. One way of providing structure to collaboratively constructed information is
by codifying the knowledge and expertise of the gatekeeper
described in the M2 model. Another approach is to distribute this responsibility among the community members by
establishing policies (Edwards, 1996) to govern the construction, organization and use of information.
Human Cognition and Tasks
The memory of an individual can be roughly divided into
two categories: short-term memory (STM) (also described as
Exam #3
3
working memory) and long-term memory (LTM).1 STM is
relatively small, typically thought to have a capacity of 7 ±
2 chunks of information (Miller, 1956). In contrast, LTM is
virtually unlimited (Matlin, 1998). In an information processing theory of human cognition (Pinker, 1997), data
needed for a given task is typically activated and retrieved
from LTM and then held in STM while it is actively used.
In this process, information is constantly being swapped in
and out of STM as new information is constructed and encoded into LTM. This model of human memory is analogous to register and disk storage used in modern-day computers.
Expert behavior is often based on a well-developed technique, or pneumonic, for encoding and retrieving information. Intelligent behavior, therefore, is often attributed in
large part to being able to effectively transfer information
into LTM so that it can later be activated for future use in
STM. Because of the limitations of STM and human attention, cognition can be viewed as a limited scarce resource
that needs to be allocated as properly during task performance. This view is analogous to operating systems that
allocate computer resources during the execution of a process
or sub-process.
Given that human cognition is limited, it makes sense
that the nature of a task (i.e., the cognitive resources it demands) affects our ability to perform the task. Over the
course of human history, cultures have invented tasks that
not only push, but also transcend the limits of human cognitive capabilities. Because of this, humans have needed to
develop cognitive artifacts (Norman, 1991) and systems to
aid in their artificial tasks. For example, long division is an
artificial system that produces such a cognitive load that it is
difficult to solve even a moderately advanced problem without the aid of cognitive artifacts such as memory aids like
paper and pencil. With the advent of computers, some of
these artificial tasks have been codified so that they may
now be performed with computational devices such as handheld calculators.
Distributed Cognition
Distributed cognition (Brown, et al., 1993; Fischer, 1995;
Hewitt & Scardamalia, 1996; Hutchins, 1993; Norman,
1993; Salomon, 1993) emphasizes that the heart of intelligent human performance is not the individual human mind
in isolation but the interaction of the mind with tools and
artifacts as well as groups of minds in interaction with each
other. It is important to understand the fundamental difference between these two forms of distributed cognition.
When distributed cognition is at work between the individual
human mind and cognitive artifacts, it often functions well
because the knowledge an individual needs is distributed between her/his head and the world (e.g., calculators, address
books, e-mail messages, filing cabinets). On the other hand,
when cognition is distributed among groups of minds, a
group has no head, no single mind to store the information
about this distribution of knowledge, which is available to
all members of the group. In this case, externalizations are
critically more important. Externalizations (1) create a record
of our mental efforts, one that is “outside us” rather than
vaguely in memory and (2) represent artifacts that can talk
back to us (Schön, 1992) and form the basis for critique and
negotiation. These can be thought of as cognitive artifacts
for groups.
Although creative individuals are often thought of as
working in isolation, the role of interaction and collaboration with other individuals is critical (Engelbart, 1995).
Creative activity grows out of the relationship between an
individual and the world of his or her work, and out of the
ties between an individual and other human beings. The predominant activity in complex problem solving is that participants teach and instruct each other (Greenbaum & Kyng,
1991). Because complex problems require more knowledge
than any single person possesses, it is necessary that all
involved stakeholders participate, communicate, and collaborate with each other. For example, during the design and
construction of a new building on a college campus, there
are building architects, tenants, collage administrators, and
government legislators, all of whom have different agendas
and background knowledge. Furthermore, communication
breakdowns are often experienced because stakeholders belonging to different cultures use different norms, symbols,
and representations (Snow, 1993). Such a setting is governed by a symmetry of ignorance (Rittel, 1984) in which
all stakeholders are aware that even though they each possess
relevant knowledge, none of them has all the relevant
knowledge. Each participant must act as a reflective practitioner rather than as an all-knowing expert (Schön, 1983).
Organizational Memories for Supporting
Long and Short Term Collaboration
The term “organizational memory” has no clear or agreed
upon definition within the computer science literature
(Ackerman & Halverson, 1998). In this paper, the following operational definition will be used:
Organizational memory provides a shared information space
that supports a group of people (an organization) to do work.
The information space should be "living" in the sense that it is
an evolving product of the work done by the members of the
organization as opposed to simply being a static storage of information (Fischer, 1998a)
In this definition, an organizational memory can be
viewed as a cognitive artifact that provides an externalization
for groups of minds interacting with each other. One could
argue that an organization or group’s memory exists independent of any explicit externalization. Every group, organization, or culture has tacit knowledge that helps govern or
define acceptable behavior, standard operating procedures, and
social norms. It can be very beneficial to identify and make
explicit a groups tacit knowledge, but this not the concern
of this paper. Instead, this paper will consider the use of
organizational memory systems supporting the collaboration
of a heterogeneous group of stakeholders during the design
of an artifact.
1
Sensory memory is also described in the cognitive psychology literature, but this is outside the scope of this paper.
For more information on sensory memory see (Matlin, 1998)
Exam #3
4
Media and Systems Supporting Organizational
Memory and Collaboration
A challenge for supporting collaboration is in providing a
mechanism that allows various participants to integrate their
perspectives (Stahl, 1993) in a meaningful way. Supporting
informed participation requires processes that integrate the
individual and the group knowledge through collaborative
constructions. Information spaces need to be constructed
collaboratively (Scardamalia & Bereiter, 1994) and integrated
into the work and social practices of the community (Lave,
1988). These collaborative constructions result in work
products that are enriched by the multiple perspectives
emerging through community discourse.
Ackerman (Ackerman, 1998) describes the development of
Answer Garden, a system for capturing and managing an
organizations knowledge and expertise. One of the primary
motivating forces behind such a system is an organization
wishing to share expertise and avoid reinventing the proverbial wheel. The most common scenario for using a system
like Answer Garden is when a worker is faced with a problem to which they have no answer. Answer Garden supports
this dilemma in two possible ways: it makes a previously
recorded solution available for retrieval or, in the absence of
such a solution, it provides access to the person who is
likely to have the appropriate expertise. This approach focuses on recording and structuring knowledge so that it may
be available for later use. However, this model makes a
strict distinction between those who possess knowledge and
those who do not. There are two separate roles in this scenario and there is no intersection seen between these two
groups. This resembles the M1 model of collaboration
where a class of experts provides access to information for a
class of non-experts. While systems such as Answer Garden
may be useful in many situations, it falls short of supporting groups of collaborators where each person is simultaneously an expert and a non-expert (e.g., an expert building
architect, but a non-expert campus planner). What is needed
in this case is a system that is open to all users.
The DynaSites system (Ostwald, 1997) is another type of
system for the creation of dynamic and user-extensible webbased information spaces. This system is designed to support M3 types of collaboration where all users can directly
contribute to the construction of new knowledge. One of the
fundamental problems of sustaining a useful open information space is that of maintaining and organizing the information. One approach to this problem is the use of a gatekeeper (Raymond, 1999), but as mentioned earlier, this has
problems of scalability. Another approach is to view an
information space as a seed (Fischer, 1998b). In this approach, an information space begins as a seed which evolves
over time through use. At some point, the information
space grows to be so unwieldy that it is no longer useful
(i.e., it is too difficult to find relevant information). At such
a time, the information needs to be restructured, or re-seeded.
However, in practice, this is difficult to accomplish. Questions of what information is pruned from the space can be
difficult to answer. Usage data can help answer questions
like these, but still, any major restructuring of critical information source can potentially be very disruptive.
Participation Revisited
All of the discussion thus far has been based on the premise that people want to participate. After all, according to the
President's Council on Sustainable Development (PCSD,
1996, p.7), ”Americans want to take control of their lives.”
However, just because people want to take control, does that
mean that they actually do? What evidence exists to support
this conclusion? According to the Institute for Democracy
and Electoral Assistance, voter turnout in US presidential
elections shows a downward trend in participation (IDEA,
1999). In fact, 1996 has had the lowest percentage of voter
turnout (47.2%) in over 50 years2. However, is voter turnout in US presidential elections an indicator of people’s desire and willingness to participate in designing their future at
a local, more intimate level? Not necessarily, but it does
raise some questions.
One factor affecting the use of group information spaces
is that of perceived utility. The utility of a system can be
viewed as a function of perceived benefit and required effort.
Two fundamental question that must be asked (and answered)
are, “Who does the work?” and “Who receives the benefit?”
(Grudin, 1994). A classic example is found in the practice of
software documentation. The software developer who is required to document his or her code may perceive no direct
benefit in doing so. For this reason, software documentation
often goes undone.
One way of increasing utility is to decrease the required effort. Another is to increase the perceived benefit. Traveen, et
al. (Terveen, Hill, Amento, McDonald, & Creter, 1997)
have sought to do both. They have developed PHOAKS
(People Helping One Another Know Stuff), which is a recommended system, or social filter. This system was able to
automatically search Usenet groups and find positive references, or “recommendations” of Web sources. In this case,
user effort is negligible: no effort was required to input or
organize information. What about it’s perceived benefit,
though? As it turns out, users perceived considerable benefit,
but the programmatic identification and categorization of
sources was imperfect. At times, PHOAKS’ recommendations were seen to be misleading or the sources that were
recommended became outdated (dead links). The mechanism
for correcting these problems was a manual process. In an
M2-like fashion, users would email their feedback to the
PHOAKS administrators, who would then evaluate and respond as they saw fit. While this method was somewhat
successful, the issue of scalability and sustainability became
obvious: as the use of the system grew, could they keep up
with user demand? By analyzing usage patterns and types of
email requests, the PHOAKS developers were able to create
mechanisms that allowed users to directly modify the content of their system (Hill & Terveen, 1997). One interesting
point that was made by these researchers was that there was
a marked distinction in consumer / producer roles for users
of their system. Specifically, they found that only a small
minority of users expended the effort of contributing modification.
2
Statistics were only available dating back to 1948.
Exam #3
5
The Design and Construction of the Discovery Learning Center
The Discovery Learning Center (DLC) is a building being
constructed on the campus of the University of Colorado,
Boulder. It is being built to support a broad-reaching vision
for the College of Engineering and Applied Science known
as the Discovery Learning Initiative. (DLI) In an effort to
embody the spirit of the initiative, a web-based information
space was created (Gorman, 1999) in May of 1999 to support the informed participation of all stakeholders involved
in the project. Stakeholders in the project include the Dean
and associate Deans of the college, perspective tenants, the
university’s facility management group, state legislators,
local industry and private donors, and the building architects
along with their consultants.
This information space was originally maintained in an
M2-like fashion as a collection of static letters, memos,
design documents, space requests from prospective tenants,
meeting minutes, and other resources related to the design
and construction of the building. It was designed as an organizational memory for the project that would support the
informed participation of interested stakeholders. Using the
M2 model, content was contributed by stakeholders and then
organized and made statically available by a gatekeeper, or
chief designer (Raymond, 1999).
Several problems exist in trying to design such an information space:
1. There was only a weak or ephemeral coalescing of work
groups. As Landauer (Landauer, 1995) points out, the
trouble with so many computer systems stems from a
lack of user-, or in this case, group-centered design.
2. With such diverse groups of target users, it is difficult
to organize the information so that each group of users
can find the information they need. Norman (Norman,
1993) suggests fitting the artifact to the person. What is
to be done about designing an artifact for an extremely
heterogeneous work group?
3. There were no well-established processes in the project
– processes were being defined on the fly. Without established processes, it is impossible to perform task
analysis (Lewis & Rieman, 1993) so that information
could be structured a priori.
Through reflection on these problems, it seems clear that
static nature of this information space made it difficult to
work with the unknowns that were present.
Participation in the Discovery Learning Center
In September of 1999, a more dynamic component was
added to the predominantly static DLC site. During this
time, the architects and their consultants were designing the
lab spaces for the initial tenant groups. This was to be an
iterative process where alternate design were created each
tenant group’s lab. These designs were then to be reviewed
by the tenant groups who were then responsible for providing feedback to the architects.
In order to provide organizational memory support for this
process, a mechanism disseminating design alternatives and
collecting feedback was added to the site. A typical design
feedback forum consisted of one or more proposed design
artifacts attached to a discussion forum. This was intended
to encourage discussion and collect feedback regarding a particular design artifact (see Appendix A). A user could select
one of the design artifacts by clicking on its thumbnail image. This would generate an enlargement of the artifact in a
separate window, which could then be inspected and compared with other competing designs. (see Appendix C). User
comments were collected and structured as a threaded discussion. Each entry would show the subject, time and the name
of the contributing user. In order to provide some awareness
of who was contributing what, users could add profile information describing their stake in the project along with
contact information such as email address and home page
address (see Appendix B).
A cursory examination of the site’s design feedback forums revealed that they generated very little discussion
among the stakeholders. At the time of writing this paper, it
is unclear exactly why this tool was not used -- results from
a survey (see appendix D) have just begun to return.
Speculation on Reasons for Lack of Participation
While it is still a little early to identify a conclusive reason for the system’s lack of use, some patterns and warning
signs have begun to emerge that point to possible causes.
Grudin (Grudin, 1994) in an evaluation of the social dynamics of groupware systems, identified factors affecting the
adoption of group-oriented systems. Some of these factors
may very well have been an issue for the adoption of this
system by its targeted users. There may have been little perceived benefit for using the feedback system. Because of
media competition (Whittaker, 1996) and ill-defined procedures, many tenants may have simply found it easier to just
use email, phone messages, or impromptu face-to-face meeting to deliver their feedback. The system may have felt too
formal so the added work of entering a well-developed critique of their design options may have been too great
(Erickson, et al., 1999).
Critical mass may also have been an issue affecting the
adoption of this system. Most group systems develop a user
base through a grass roots effort (Palen, 1999) and/or a substantial ramp up period. Given the fast-paced nature of this
project and ephemeral quality of collaborative groups, there
may not time to establish enough critical mass to crate a
sustainable user group.
The use of this system may have created too much disruption in the normal social processes that existed in the college. Tenant faculty members may have been use to communicating more directly with the Dean’s office and the use
of a information system that would result in a persistent
record of their communications may have been too formal
and threatening.
Finally, groupware systems require a more careful introduction into the workplace than do typical software systems.
Unless there is enough time to allow a grassroots adoption
to take place, any deployment of such a system will require
a significant top-down approach. In other words, with a fastpaced project there is not enough time for a more natural
adoption of group memory system. In such a case, the role
of the system needs to be more explicitly built into the
process.
Exam #3
6
Conclusion
There is a variety of models that can be useful in the construction of organizational memory systems. The M1 model
can be useful for recording and disseminating an organization’s knowledge, but it is not very useful in supporting
collaboration. An M2 model can help provide structure and
openness to a system, but scalability issues arise, as the
feedback generated becomes too great. The M3 model provides the greatest openness and support for direct collaboration. However, it can be very challenging to develop systems with the ability to govern the creation and modification
of information as well as the structuring and retrieval of that
information.
These models can be viewed as points along a continuum
rather than as discrete choices. As seen with PHOAKS and
the DLC, systems can be incrementally enhanced to support
greater levels of collaboration. Usage data can be very helpful in identifying opportunities for developing greater levels
of collaborative functionality in a system.
One of the most important considerations in the use of an
organizational memory system is the social environment in
which it will be used. These types of systems generally take
time to develop roots and gain critical mass. Therefore, if
time for deployment is scarce, other methods must be enforced. Having a clear work process defined can help in the
deployment phase, but whether or not a system gets used
may ultimately depend on other factors, such as media competition, perceived benefit, and minimizing extra effort by
providing smooth integration into existing work practices.
References
Brown, J.S., Duguid, P., & Haviland, S. (1994). Toward Informed Participation: Six Scenarios in Search of
Democracy in the Information Age. The Aspen Institute
Quarterly, 6(4), 49-73.
Edwards, K.W. (1996). Policies and roles in collaborative applications, Proceedings of the ACM 1996 conference
on on Computer supported cooperative work (pp. 11 - 20).
Engelbart, D.C. (1995). Toward Augmenting the Human
Intellect and Boosting our Collective IQ, Communications
of the ACM, Vol. 38 (pp. 30-33).
Erickson, T., Smith, D.N., Kellogg, W.A., Laff, M.,
Richards, J.T., & Bradner, E. (1999). Socially translucent
systems: social proxies, persistent conversation, and the
design of "babble", Proceeding of the CHI 99 conference on
Human factors in computing systems: the CHI is the limit
(pp. 72 - 79).
Fielding, R.T. (1999). Shared leadership in the Apache
project. Communications of the ACM, 42(4), 42 - 43.
Fischer, G. (1994). Turning Breakdowns into Opportunities for Creativity. Knowledge-Based Systems, Special Issue
on Creativity and Cognition, 7(4), 221-232.
Fischer, G. (1995). Distributed Cognition, Learning
Webs, and Domain-Oriented Design Environments. In J.L.
Schnase & E.L. Cunnius (Eds.), Proceedings of CSCL'95:
The First International Conference on Computer Support for
Collaborative Learning (pp. 125-129). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Ackerman, M.S. (1998). Augmenting Organizational
Memory: A Field Study of Answer garden. ACM Transactions on Information Systems, 16(3), 203 - 224.
Fischer, G. (1998a). Organizational Memory. Web Page
at
http://Seed.cs.colorado.edu/Dynagloss.makeGlossPage.fcgi$
Organizational_Memory
Ackerman, M.S., & Halverson, C. (1998). Considering
an organization's memory, Proceedings of the ACM 1998
conference on Computer supported cooperative work (pp. 39
- 48).
Fischer, G. (1998b). Seeding, Evolutionary Growth and
Reseeding: Constructing, Capturing and Evolving Knowledge in Domain-Oriented Design Environments. Automated
Software Engineering, 5(4), 447-464.
Ackerman, M.S., & Malone, T.W. (1990). Answer Garden: a tool for growing organizational memory, Proceedings
of the conference on Office information systems (pp. 31-39),
Cambridge, MA USA.
Fischer, G., Lemke, A.C., McCall, R., & Morch, A.
(1996). Making Argumentation Serve Design. In T. Moran
& J. Carrol (Eds.), Design Rationale: Concepts, Techniques,
and Use (pp. 267-293). Mahwah, NJ: Lawrence Erlbaum and
Associates.
Ackerman, M.S., & McDonald, D.W. (1996). Answer
Garden 2: Merging Organizational Memory with Collaborative Help. In M.S. Ackerman (Ed.), Proceedings of ACM
CSCW'96 Conference on Computer-Supported Cooperative
Work (pp. 97-105). New York: ACM Press.
Arias, E., Eden H., Fischer, G., Gorman, A., and
Scharff, E. (1999 (in press)). Beyond Access: Informed Participation and Empowerment, Proceedings of the conference
on Computer Supported Collaborative Learning (CSCL
'99).
Brown, A.L., Ash, D., Rutherford, M., Nakagawa, K.,
Gordon, A., & Campione, J.C. (1993). Distributed Expertise in the Classroom. In G. Salomon (Ed.), Distributed
Cognitions: Psychological and Educational Considerations.
Cambridge, England: Cambridge University Press.
Gorman, A. (1999). The Discovery Learning Center.
Web Page at http://DiscoveryLearning.Coloardo.Edu/
Greenbaum, J., & Kyng, M. (1991). Design at Work:
Cooperative Design of Computer Systems. Hillsdale, NJ:
Lawrence Erlbaum Associates, Inc.
Grudin, J. (1994). Groupware and social dynamics: Eight
challenges for developers. Communications of the ACM,
37(1), 92-105.
Hewitt, J., & Scardamalia, M. (1996). Design Principles
for the Support of Distributed Processes. Paper Presented in
a Symposium Distributed Cognition: Theoretical and Practical Contributions, at the Annual Meeting of the American
Exam #3
7
Educational Research Association, New York City. at
http://csile.oise.on.ca/abstracts/distributed/
Hill, W.C., & Terveen, L.G. (1997). Involving remote
users in continuous design of web content, Proceedings of
the conference on Designing interactive systems: processes,
practices, methods, and techniques (pp. 137 - 145).
Rittel, H. (1984). Second-Generation Design Methods.
In N. Cross (Ed.), Developments in Design Methodology
(pp. 317-327). New York: John Wiley & Sons.
Salomon, G. (1993). Distributed Cognitions: Psychological and Educational Considerations. Cambridge, United
Kingdom: Cambridge University Press.
Hutchins, E.L. (1993). Learning to Navigate. In S.
Chaiklin & J. Lave (Eds.), Understanding Practice (pp. 3563). Cambridge, UK: Cambridge University Press.
Scardamalia, M., & Bereiter, C. (1994). Computer Support for Knowledge-Building Communities. Journal of the
Learning Sciences, 3(3), 265-283.
IDEA (1999). Voter Turnout from 1945 to 1997: A
Global
Report
on
Political
Participation.
at
http://www.idea.int/Voter_turnout/index.html
Schön, D. (1992). Designing as reflective conversation
with the materials of a design situation. Knowledge-Based
Systems Journal, Special Issue on AI in Design, 5(1), 3-14.
Landauer, T.K. (1995). The Trouble with Computers.
Cambridge, MA: MIT Press.
Schön, D.A. (1983). The Reflective Practitioner: How
Professionals Think in Action. New York: Basic Books.
Lave, J. (1988). Cognition in Practice. Cambridge, UK:
Cambridge University Press.
Snow, C.P. (1993). The Two Cultures. Cambridge, UK:
Cambridge University Press.
Lewis, C., & Rieman, J. (1993). Task-Centered User Interface Design: A Practical Introduction. Boulder, Colorado:
University of Colorado, Boulder.
Stahl, G. (1993). Interpretation in Design: The Problem
of Tacit and Explicit Understanding in Computer Support of
Cooperative Design. Ph.D. Dissertation., University of
Colorado at Boulder.
Matlin, M.W. (1998). Cognition (4th ed.). New York:
Harcourt Brace College Publishers.
Miller, G. (1956). The Magic Number Seven, Plus or
Minus Two" Some Limits on Our Capicity for Processing
Information. Psychology Review, 63, 81-97.
Moran, T.P., & Carroll, J.M. (1996). Design Rationale:
Concepts, Techniques, and Use. Hillsdale, NJ: Lawrence
Erlbaum Associates, Inc.
Norman, D. (1991). Cognitive Artifacts. In J.M. Carroll
(Ed.), Designing Interaction (pp. 17-38). Cambridge: Cambridge University Press.
Terveen, L., Hill, W., Amento, B., McDonald, D., &
Creter, J. (1997). PHOAKS: A System for Sharing Recommendation. Communications of the ACM, 40(3), 59-62.
Torvalds, L. (1999). The linux edge. Communications
of the ACM, 42(4), 38 - 39.
Whittaker, S. (1996). Talking to Strangers: An Evaluation of the Factors Affecting Electronic Collaboration, Proceedings of the ACM 1996 conference on on Computer supported cooperative work (pp. 409 - 418).
Norman, D.A. (1993). Things That Make Us Smart.
Reading, MA: Addison-Wesley Publishing Company.
Ostwald,
J.
(1997).
Dynasites.
at
http://Seed.cs.colorado.edu/dynasites.documentation.fcgi$nod
e=dynasites.doc.home
Palen, L. (1999). Social, individual and technological issues for groupware calendar systems, Proceeding of the CHI
99 conference on Human factors in computing systems: the
CHI is the limit (pp. 17 - 24).
PCSD (1996). Sustainable America: A New Consensus
for Prosperity, Oppertunity and a Healthy Environment for
the Future. Washington, D.C.: President's Council on Sustainable Development.
Pinker, S. (1997). How the Mind Works. New York: W.
W. Norton & Company.
Popper, K.R. (1965). Conjectures and Refutations. New
York, Hagerstown, San Francisco, London: Harper & Row.
Raymond, E.S. (1999). The Cathedral & The Bazaar:
Musings on Linux and Open Source by an Accidental Revolutionary. Sebastopol, CA: O'Reilly.
Exam #3
Appendix A
DLC Design Feedback Forum Centered Around Design Artifacts
Exam #3
Appendix B
DLC User Profile
Exam #3
Appendix C
DLC Design Artifact
Exam #3
Appendix D
DLC Survey
For the following questions rate your answers on a scale from 1 to 5, where
1 is very little and 5 is very much.
1) How often did you use the following modes of communication throughout
your involvement in this project.
a)
b)
c)
d)
e)
f)
Face-to-face formal/scheduled meetings
Face-to-face informal/impromptu meetings
Telephone (including voice mail)
Inter-office mail
Email
Discussion forum on the Discovery Learning Web Site
2) How valuable were the following modes of communication throughout your
involvement in this project.
a)
b)
c)
d)
e)
f)
Face-to-face formal/scheduled meetings
Face-to-face informal/impromptu meetings
Telephone (including voice mail)
Inter-office mail
Email
Discussion forum on the Discovery Learning Web Site
3a) During your involvement in this project, have you used the DLC Web site
(y or n)?
If you answered "yes" to #3a
3b) how many times did you visit the site?
a)
b)
c)
d)
1 - 5 times
6 - 10 times
11 - 20 times
> 20 times
3c) what did you fine most useful at the site?
3d) what did you fine least useful at the site?
3e) what what would you say is the main purpose of the site.?
4) if you answered "no" to questions #3, were you aware of the DLC Web
Site's existence?
5) On a scale of 1 to 5 please rate your level of involvement during the
design of YOUR lab space.
5b) How often did you use the following modes of communication during the
design of YOUR lab space.
a)
b)
c)
d)
e)
f)
Face-to-face formal/scheduled meetings
Face-to-face informal/impromptu meetings
Telephone (including voice mail)
Inter-office mail
Email
Discussion forum on the Discovery Learning Web Site
6) Were you aware that proposed floor plans for YOUR lab were available on
the DLC web site (y or n)?
7) Please describe in your own words your involvement in the design of YOUR
lab space.
Exam #3
8) Please add any additional comments that you may have about the project.
Exam #3