A Media Space Field Study
Garry Beirne, Gale Moore, Bill Buxton, Marilyn Mantei
Computer Systems Research Institute
University of Toronto
Toronto, Ontario, Canada M5S 1A1
KEYWORDS: Multimedia, media space, desktop videoconferencing, computer supported collaborative work, groupware, social interaction, user interface, field studies.
As researchers apply user-centred design techniques towards the development of the social interface, testing the systems in the field becomes a critical research component.
It is only in the field that we can observe the social effects of media space deployment and usage. The problems and logistical issues of undertaking media space field studies are becoming more important for media space researchers.
The studies we perform outside of the laboratory are social studies around the technology rather than a trial specifically of the technology. We are as concerned with the social effects of the use of these technologies as we are with trials of our own technologies. We refer to this work as field studies rather than field trials. We have completed one wrist's length field study and are in the planning and preparation stages of an arm's length field study.
For the past two years we have been increasingly involved in the study of media spaces outside of the laboratory. In this paper we report on some of our practice and experiences in order to benefit others who may be about to embark on this type of research. After a brief review of previous work, we outline a four-tiered framework within which our research is organized. These tiers move the research progressively from contained local tests and experiments to remote arm's length field studies. We discuss the structure in terms of both its practical and methodological implications - in particular how one can better incorporate user-centred design in this class of system.
While we argue that testing systems in the field is important, our experience thus far has made it abundantly clear that doing so is fraught with problems, especially when resources are limited and time is short. Consequently, a part of our discussion reports on the roles assumed by the various players so as to minimize overhead, and improve the quality of the study.
We have developed a strategy for deployment which avoids a common pitfall of technological field trials. If the technology has no real life expectancy users are not motivated to modify their work habits to use the technology, and any results obtained from the study are suspect. Our strategy insures that the technological infrastructure has a continued life after the duration of our studies.
Our technology has been deployed at a number of sites outside of our laboratory, several of which are engaged in research similar to The Ontario Telepresence Project. We have completed one wrist's length field study using our own and commercial technologies. At this time we are several months into the planning and preparation stages for an arm's length field study.
Investigations of media spaces have traditionally fallen into two categories: relatively short experiments and laboratory studies [12,13,23,24], or longer term self-investigations [2,9]. All of these studies have been internal to the respective organizations and have provided valuable insights into the impact of media spaces on work behaviour.
Telepresence personnel are distributed between two cities, with one site in Toronto and in Ottawa. The team is deliberately distributed geographically in order for team members to experience working in the media space we are implementing. We also have formal working relationships with several European research facilities. Our media space software is being used by many organizations (Telepresence, Xerox PARC, Bell Canada, and Eurecom). In this way we are able to continue our investigation of media spaces through our own practice as well as via controlled experimentation.
This initial in-house testing is fundamental to the development of the technologies, but unlike single user systems, media spaces need to be tested in the social milieu for which they were intended. Much of their design is directed at supporting social interchange. This environment is not possible to duplicate in laboratory settings. Thus, it is impossible to get appropriate feedback data for redesign. This creates a problem. We must dedicate resources to create a prototype with which to conduct a comprehensive study, but it must behave with the robustness of a commercial product.
Click here for Picture
Figure 1: Four tiers of media space study
Part of our strategy for dealing with this problem has been to adopt a four-tiered model of evaluation: Experiments and Laboratory Study, In-House Study, Wrist's Length Study, and Arm's Length Study. (See Figure 1). This model allows us to gain some feedback on development before too much investment has occurred.
In-house study is, by far, the most common type of study in media space research [2,15,20]. For these studies the researchers use systems that they are themselves developing. In some cases the systems are used by other colleagues in the same organization .
There is still much to be learned within a research environment, even from users familiar with the system and tolerant of change. This user group is willing to use and test application software and interface designs that are still under development. Our facilities are more complex than could be used in controlled experiments, and we can easily explore new feature sets. Our offices are bursting with media space technology, and little attempt is made to make the technology unobtrusive. We keep the devices and wires accessible for rapid reconfiguration, resulting in devices and wires intruding on our workspaces.
Most research in media spaces falls into one of the above two categories. The implementation and study of media spaces outside a laboratory is radically different, and it is this feature that distinguishes our research from previous work.
A site at wrist's length is not likely to have the same technical support as a research lab. The main interest is simply to provide a new method to support communication between their geographically separated employees. Their familiarity with media spaces and their communication requirements are very different than ours, so customized applications are needed. These are usually a subset of the applications we use in-house.
In a wrist's length site, changes to the system occur less frequently than in-house, but we still have the opportunity to evaluate new concepts, make modifications to the system and generally have a greater presence in their environment than would be appropriate for sites not sympathetic to our work. We undertake longitudinal studies with these sites and acquire useful data about acceptance, usage, and social impacts of the media space. At the same time we test installation and maintenance procedures and the usability of our various manuals.
At the beginning of a study at wrist's length, we work closely with the site to provide technical support, counseling on installation, training on the system, and user group management. By the end of the study the software is stable, Telepresence withdraws our technical management and support, and the system continues to be operated and maintained by the field study site.
The research methods and design we use will vary from study to study, but can be broadly characterized as follows: experimental method, surveys, user and usability surveys, field research and network analysis. More details can be found in .
* the technology must be more robust, compact, less functional and more constrained.
* the cost of doing the study and the planning time increases.
* the users are less "friendly" to the scientific goals of the project and more concerned with the way the system affects their workplace and daily tasks.
* the control over users and their environment diminishes and the risk of scientific failure from confounding causes increases.
* the value of the information to our industrial partners increases.
Some additional characteristics of field studies are:
* the participants in the studies must be able to continue in their normal job functions with minimal personal disruption.
* issues of cost, support, aesthetics and ethics become factors in selecting and installing equipment.
Cost. The study site finances the purchase of equipment. The same economic pressures that guide all of their purchases affect their media space decisions.
Aesthetics. The equipment must "fit in" at least as well as any other technological device in the users' personal environments. One of our goals is to make the appliances invisible and ubiquitous. We select equipment that matches furniture and office decor.
Features. Cost takes precedent over features, but the following elements are important to consider. We look for equipment with the fewest dials and knobs. This reduces the intimidation factor and discourages users from making unnecessary, often problematic, adjustments. Fewer controls means there are fewer opportunities to push the wrong button.
Theft. Until recently the least expensive colour video cameras available to us were small consumer camcorders, which are very attractive to would-be thieves. We now use small, inexpensive, commercially available cameras designed for desktop video conferencing.
Audio quality. It has been suggested that the audio channel makes more of a difference than the visual channel . Our own anecdotal evidence is in line with these observations . We find it entirely unacceptable to use the speakers in most television monitors. Their frequency response and tendency to send the sound in all directions make it difficult to prevent feedback. We use professional audio equipment to keep the quality of the audio channels high, and we are quite satisfied with the quality of consumer grade PZM microphones (Pressure Zone Microphone).
Wires. One unfortunate characteristic of using analog technologies is the abundance of wires that are required . When installing a media space at a field site, the wires must be installed in a manner coordinated with the overall image being presented by the environment. Wires must be neatly hidden. We anticipate the arrival of a digital architecture  which will do away with the spaghetti conditions we now must manage.
Prior literature has indicated some of the technological obstacles to using off-the-shelf audio and video technology in media spaces [3,20]. We have found that users are very concerned with the way they appear on the monitor, and have witnessed significant office space rearrangement to accommodate a more attractive view from the camera's perspective.
In another experience an individual refused a media space node for their office, but they accepted our offer of a video monitor, and the electronic window. The individual became comfortable enough with the presence of the monitor that they soon requested a full media space node. A different user likened a 'black' monitor (one with no image on the screen) to a "black hole." It's mere presence was intimidating. Another user, who had a large picture window in their office, would frequently notice changes in the weather on their electronic window (which was more likely to be in their field of view) rather than use the wall-sized 'real' window.
These anecdotes suggest that using the monitor to provide a more pleasant work environment may decrease anxiety and encourage adoption.
Click here for Picture
Figure 2: Relationships among players in field studies.
Figure 2 illustrates a simplified view of the relationships among the participants in Telepresence field studies. The three main players are illustrated as ovals. They are: the Telepresence researchers, the field study users, and the industrial/commercial partners.
Users are observed at work by the Telepresence social scientists. The media space technology foundation used by the users was designed and tested by the Telepresence researchers in conjunction with the industrial/commercial affiliates. However, the actual deployment and maintenance of the system is undertaken by Telepresence industrial/commercial affiliates (in consultation with the Telepresence researchers, with diminishing involvement over time). The technology foundation is financed by the user/clients, and commissioned directly from the industrial/commercial affiliates.
Central to this arrangement is the view that the deployed technology must have a life in the client organization after the field studies. Telepresence has a central role in the initiation, evaluation and revision of the foundation technology and deployment. At the end of the study, however, Telepresence withdraws their involvement, and the system continues to be maintained by the study site and the industrial/commercial affiliates. Everything to the left of the vertical dotted line is self-contained and no longer requires the attention of Telepresence.
By following this strategy, we avoid the common trap of field studies where the technology works only for the duration of the study. If the technology has no real life expectancy users are not motivated to modify their work habits to use the technology, and any results obtained from the study are suspect. Our strategy avoids this common pit-fall, and also means that studies can be undertaken with only minimal resources from Telepresence. The primary resources for the study come from the main stake-holders in the private sector and the user/clients.
The system architects/developers, who were accustomed to having the media space running under their watchful eyes, were suddenly charged with the task of ensuring that the system would run effectively and reliably in an environment over which they had no control. Thus, reporting and handling error conditions suddenly became more important, as did performance, security, and reliability. As well, having made commitments of deliverables to outside organizations to install and operate systems by certain dates, we found our management and programming practices had to change to reflect more of a product development cycle. This was often at odds with the traditional research practice in our environment.
Even in our own in-house environment the successful adoption of the media spaces by approximately 24 people has been a double edged success. On the one hand we have demonstrated the usefulness of the tools. On the other hand we have created a demand for the system that requires that it be fully operational at all times, thus decreasing our ability to explore, test, debug and otherwise research new media space concepts. We are trying to manage apparently opposing objectives of research and exploration on the one hand, and production and dependability of the system on the other.
To achieve the transition from research software to solid production software we have added several members to our engineering team. Part of their efforts have been to make the technology robust enough to be supported by third party industrial partners. This has allowed the Telepresence technology foundation to remain in place and operational after our initial field studies, thus motivating subjects to modify their work habits to use the technology in the course of the studies.
We have documented anecdotally the major issues that arose for our group in trying to conduct our field study design research. In summary these are: (1) Field studies need technical support. Either the research staff has to provide this support or it needs to be contracted for elsewhere. The latter is difficult because noone knows how to support an entirely new system. (2) The technology in field studies needs to work at all times. Without this reliability, the users won't use the system and the field study will be worthless. This means that the research staff has to have sufficient resources and expertise to make a nearly commercially acceptable product. (3) The technology for the field study needs to have a reasonable life expectancy if users are to change their work habits to accommodate the technology. This requires support for the maintenance and update of the technology once the field study is over. Vendors are reluctant to do this for novel systems, but we have found that industrial affiliates can be a good substitute for this continuing support.
2. Bly, S.A., Harrison, S.R. and Irwin, S. Media Spaces: Bringing People Together in a Video, Audio, and Computing Environment. Communications of the ACM , Vol. 26, No. 1 (January 1993), pp. 28-47.
3. Buxton, W. and Moran, T. EuroPARC's Integrated Interactive Intermedia Facility (iiif): Early Experiences. In Proceedings of the IFIP WG8.4 Conference on Multi-user Interface and Applications (Heraklion, Crete, 1990), Elsevier Science Publishers, The Netherlands, pp. 11-34.
4. Buxton, W. Telepresence: Integrating Shared Task and Shared Person Spaces. In Proceedings of Graphics Interface '92 (Vancouver, May 11-15, 1992), pp. 123-129.
5. Dourish, P. and Bly, S. Portholes: Supporting Awareness in a Distributed Work Group. In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, New York , pp. 541-547.
6. Dourish, P. The Affordances of Media Spaces for Collaboration. In Proceedings of CSCW'92 (Toronto, Nov 1-4, 1992), ACM Press, New York , pp. 17-24.
7. Egido, C. Videoconferencing as a Technology to Support Group Work: A Review of its Failures. In Proceedings of CSCW'88 (Portland, Sept 26-28, 1988), ACM Press, New York , pp. 13-24.
8. Elliott, C. High-Quality Multimedia Conferencing Through a Long-Haul Packet Network. In Proceedings of ACM Multimedia'93 (Anaheim, Aug 1-6, 1993), ACM Press, New York, pp. 91-98.
9. Fish, R.S., Kraut, R.E. and Chalfonte, B.L. The Video Window System in Informal Communication. In Proceedings of CSCW'90 (Los Angeles, Oct 7-10, 1990), ACM Press, New York , pp. 1-11.
10. Galloway, K. and Rabinowitz, S. Hole-in-Space. Mobile image videotape, Santa Monica, California, 1980.
11. Gaver, W., Moran, T., MacLean, A, Lövstrand, L., Dourish, P., Carter, K. and Buxton, W. Realizing a Video Environment: EuroPARC's RAVE System. In Proceedings of CHI'92 (Monterey, May 3-7, 1992), ACM Press, New York , pp. 27-35.
12. Gaver, W., Sellen, A., Heath, C. and Luff, P. One is Not Enough: Multiple Views in a Media Space. In Proceedings of CHI'93 (Amsterdam, April 24-29, 1993), ACM Press, New York , pp. 335-341.
14. Heath, C. and Luff, P. Disembodied Conduct: Communication Through Video in a Multi-Media Office Environment. In Proceedings of CHI'91 (New Orleans, April 27-May 1, 1991), ACM Press, New York , pp. 99-103.
15. Hopper, A., et al. The Pandora Multimedia System (a videotape), Olivetti Corporation, Cambridge, England, 1991.
16. Ishii, H. and Kobayashi, M. ClearBoard: A Seamless Medium for Shared Drawing and Conversation With Eye Contact. In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, New York , pp. 525-532.
17. Li, J. and Mantei, M. Working Together, Virtually. In Proceedings of Graphics Interface '92 (Vancouver, May 11-15, 1992), pp. 115-122.
18. Lu, I. and Mantei, M. Idea Management in a Shared Drawing Tool. In Proceedings of ECSCW'91, Kluwer Academic, The Netherlands, pp. 97-112.
19. Lyles, J.B. and Swinehard, D.C. The Emerging Gigabit Environment and the Role of Local ATM, IEEE Communications, Vol. 30, No. 4, (April 1992), pp. 52-58.
20. Mantei, M.M., Baecker, R.M., Sellen, A.J., Buxton, W., Milligan, T. and Wellman, B. Experiences in the User of a Media Space. In Proceedings of CHI'91 (New Orleans, April 27-May 1, 1991), ACM Press, New York , pp. 203-208.
21. Moore, G. Social Science Models and Strategies for Telepresence User Studies, Ontario Telepresence Technical Report OTP-93-01, February, 1993, University of Toronto, Toronto, Canada.
22. Pye, R. and Williams, E. Teleconferencing: Is Video Valuable or is Audio Adequate?, Telecommunications Policy (June 1977), pp. 230-241.
23. Sellen, A.J. Remote Conversations: The Effects of Mediating Talk with Technology. In the Journal of Human-Computer Interaction, forthcoming 1993.
24. Sellen, A.J. Speech Patterns in Video-Mediated Conversations. In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, New York , pp. 49-59.
25. Stults, R. Media Space, Xerox PARC technical report, 1986, Xerox Palo Alto Research Center, Palo Alto, Calif.