ABSTRACT

The Ontario Telepresence Project is investigating the effectiveness of media spaces in the work environment. An important part of the research involves prototyping new media space applications, and evaluating them through experimentation and field studies outside of the laboratory. As researchers apply user-centred design techniques towards the development of the social interface, testing the systems in the field becomes a critical research component. The logistics of moving media space technologies to organizations outside the laboratory environment presents new challenges. Subjects are concerned not only with user interface issues, but also with aesthetics, surveillance and quality of the sound and image. For the past two years we have been increasingly involved in the study of media spaces. We have installed and operated media spaces outside of our laboratory, and we have undertaken sociological analysis of their use. In this paper we report on some of the methods, techniques and experiences we have had in deploying these technologies.

KEYWORDS: Multimedia, media space, desktop videoconferencing, computer supported collaborative work, groupware, social interaction, user interface, field studies.

INTRODUCTION

The Ontario Telepresence Project is a three-year joint government, university and industry research project investigating the effectiveness of media spaces (enhanced desktop video conferencing [2,25]) in the work environment. We design, build, test and use technologies in our own facility which explore new concepts in media space design. Part of the research involves the evaluating them through experimentation and field studies outside of the laboratory environment.

As researchers apply user-centred design techniques towards the development of the social interface, testing the systems in the field becomes a critical research component.

It is only in the field that we can observe the social effects of media space deployment and usage. The problems and logistical issues of undertaking media space field studies are becoming more important for media space researchers.

The studies we perform outside of the laboratory are social studies around the technology rather than a trial specifically of the technology. We are as concerned with the social effects of the use of these technologies as we are with trials of our own technologies. We refer to this work as field studies rather than field trials. We have completed one wrist's length field study and are in the planning and preparation stages of an arm's length field study.

For the past two years we have been increasingly involved in the study of media spaces outside of the laboratory. In this paper we report on some of our practice and experiences in order to benefit others who may be about to embark on this type of research. After a brief review of previous work, we outline a four-tiered framework within which our research is organized. These tiers move the research progressively from contained local tests and experiments to remote arm's length field studies. We discuss the structure in terms of both its practical and methodological implications - in particular how one can better incorporate user-centred design in this class of system.

While we argue that testing systems in the field is important, our experience thus far has made it abundantly clear that doing so is fraught with problems, especially when resources are limited and time is short. Consequently, a part of our discussion reports on the roles assumed by the various players so as to minimize overhead, and improve the quality of the study.

We have developed a strategy for deployment which avoids a common pitfall of technological field trials. If the technology has no real life expectancy users are not motivated to modify their work habits to use the technology, and any results obtained from the study are suspect. Our strategy insures that the technological infrastructure has a continued life after the duration of our studies.

Our technology has been deployed at a number of sites outside of our laboratory, several of which are engaged in research similar to The Ontario Telepresence Project. We have completed one wrist's length field study using our own and commercial technologies. At this time we are several months into the planning and preparation stages for an arm's length field study.

PREVIOUS MEDIA SPACE RESEARCH

The current round of media space research began in 1985 at Xerox PARC [25], following on earlier exploration by artists Kit Galloway and Sherri Rabinowitz [10]. Recent years have seen a marked increase in research efforts on media spaces and their application in support of collaborative work. Some of the research facilities involved in media space research include Bellcore [9], Xerox PARC [2], Rank Xerox EuroPARC [11], Olivetti [15], and the University of Toronto [20].

Investigations of media spaces have traditionally fallen into two categories: relatively short experiments and laboratory studies [12,13,23,24], or longer term self-investigations [2,9]. All of these studies have been internal to the respective organizations and have provided valuable insights into the impact of media spaces on work behaviour.

ONTARIO TELEPRESENCE PROJECT

In 1992 the locally-based University of Toronto CAVECAT project evolved into the Ontario Telepresence Project. A large part of Telepresence research involves prototyping new media space applications, evaluating them through experimentation and examining their use and adoption in field studies. The primary focus of the project is on "shared interpersonal space", with secondary attention to "shared task spaces" [4,16].

Telepresence personnel are distributed between two cities, with one site in Toronto and in Ottawa. The team is deliberately distributed geographically in order for team members to experience working in the media space we are implementing. We also have formal working relationships with several European research facilities. Our media space software is being used by many organizations (Telepresence, Xerox PARC, Bell Canada, and Eurecom). In this way we are able to continue our investigation of media spaces through our own practice as well as via controlled experimentation.

LEVELS OF STUDY OF MEDIA SPACES

Earlier work evaluated the effectiveness of a media space within our own organization. Other research projects, such as those previously cited, have similarly focused on in-house use. In many cases the subjects have had a vested interest in the use of the systems, often because either they or a colleague have been involved in developing the technologies. The subjects are invariably proficient with computer technology.

This initial in-house testing is fundamental to the development of the technologies, but unlike single user systems, media spaces need to be tested in the social milieu for which they were intended. Much of their design is directed at supporting social interchange. This environment is not possible to duplicate in laboratory settings. Thus, it is impossible to get appropriate feedback data for redesign. This creates a problem. We must dedicate resources to create a prototype with which to conduct a comprehensive study, but it must behave with the robustness of a commercial product.

Click here for Picture

Figure 1: Four tiers of media space study

Part of our strategy for dealing with this problem has been to adopt a four-tiered model of evaluation: Experiments and Laboratory Study, In-House Study, Wrist's Length Study, and Arm's Length Study. (See Figure 1). This model allows us to gain some feedback on development before too much investment has occurred.

Experiments and Laboratory Study

In the experiments and laboratory studies we place subjects in a controlled environment and assign them specific tasks [23,24]. We use groups of two to four people. Subjects do not perform ordinary work tasks and are not in their work environment. Frequently the subjects have no previous exposure to media spaces and spend less than an hour in the environment. Because time is so short, simple tasks are used as well as simple interfaces. Any user interface must be trivial to learn. Laboratory studies allow the researchers to hold other variables constant to measure the effect of the media space usage. We use formal experiments and informal usability studies at this level. Other media space research projects also engage in this type of study [12,13]. The hardware and software installations in our laboratory are complex, but flexible. We can rapidly reconfigure resources to run new experiments.

In-House Studies

At the second level, we investigate the impact of media spaces on ourselves. These investigations differ from laboratory studies in several ways. They allow a larger population to be studied over longer periods of time, and the technology is incorporated into regular work activity. The best designs from the experimental level migrate to in-house practice. As users we exhibit advanced technological competency, and are familiar with the technical details of the system. Thus, applications often have sophisticated feature sets. We are able to make changes to the system regularly, and it is not uncommon to have system features change weekly during heavy development cycles. The users are tolerant of change, don't require training, and have little use for manuals. Technical installation and support is conducted by the system developers. Users frequently take responsibility for enhancing the system.

In-house study is, by far, the most common type of study in media space research [2,15,20]. For these studies the researchers use systems that they are themselves developing. In some cases the systems are used by other colleagues in the same organization [9].

There is still much to be learned within a research environment, even from users familiar with the system and tolerant of change. This user group is willing to use and test application software and interface designs that are still under development. Our facilities are more complex than could be used in controlled experiments, and we can easily explore new feature sets. Our offices are bursting with media space technology, and little attempt is made to make the technology unobtrusive. We keep the devices and wires accessible for rapid reconfiguration, resulting in devices and wires intruding on our workspaces.

Most research in media spaces falls into one of the above two categories. The implementation and study of media spaces outside a laboratory is radically different, and it is this feature that distinguishes our research from previous work.

Wrist's Length Studies

The field studies at wrist's length place media spaces in a workplace outside our laboratory, in an organization that has a working relationship with our researchers. The site is sympathetic to our research, but not directly involved with it, and they do not have a vested interest in it. The sites considered to date have had offices in two or more locations, where there was a requirement for close collaboration among employees across sites. These sites are small, with approximately 10 to 30 people employees.

A site at wrist's length is not likely to have the same technical support as a research lab. The main interest is simply to provide a new method to support communication between their geographically separated employees. Their familiarity with media spaces and their communication requirements are very different than ours, so customized applications are needed. These are usually a subset of the applications we use in-house.

In a wrist's length site, changes to the system occur less frequently than in-house, but we still have the opportunity to evaluate new concepts, make modifications to the system and generally have a greater presence in their environment than would be appropriate for sites not sympathetic to our work. We undertake longitudinal studies with these sites and acquire useful data about acceptance, usage, and social impacts of the media space. At the same time we test installation and maintenance procedures and the usability of our various manuals.

At the beginning of a study at wrist's length, we work closely with the site to provide technical support, counseling on installation, training on the system, and user group management. By the end of the study the software is stable, Telepresence withdraws our technical management and support, and the system continues to be operated and maintained by the field study site.

The research methods and design we use will vary from study to study, but can be broadly characterized as follows: experimental method, surveys, user and usability surveys, field research and network analysis. More details can be found in [21].

Arm's Length Studies

The studies at arm's length are full-fledged field studies in workplaces that are completely independent of us. An example might be financial institutions or government agencies. These sites will have between 30 and 100 users connected to the media space. heir own technical staff will install and support the installation, and Telepresence will function in a consulting role. The deployed technology must be more robust, compact, and efficient, and the user interface easier to use than in previous levels. These field sites make significant capital investment in the study, so full documentation of the study and the system is required by the site before the study begins. We use the longitudinal study techniques that were refined in the earlier studies. These sites are permanent installations. The system will not disappear at the end of the study, and will continue to be used, operated, and maintained by the field study site. We are currently in the planning stages for an arm's length field study.

Why the Distinction Between Wrist's and Arm's ?

Media spaces are new technologies which provide new capabilities that haven't been researched before. They also bring together researchers form many different disciplines who have never worked together. The studies at wrist's length provide an opportunity to work out the design of the research methods and to build a research team from the varied personnel. They also afford the incremental acquisition of skills and confidence to support the more difficult arm's length studies. Studies at wrists' length provide us with a vehicle to incrementally learn the techniques of field trial deployment and execution. Therefore, it is the strategy of Telepresence to start studies locally and, as the underlying test-bed, applications and social science techniques mature, move toward sites that are at arm's length.

BEYOND THE LABORATORY - INSTALLATION ISSUES

The deployment of a media space outside the laboratory environment is entirely different from in-house use. In Figure 1 the four levels of study were represented as concentric circles. As the circles get larger the following becomes truer:

* the technology must be more robust, compact, less functional and more constrained.

* the cost of doing the study and the planning time increases.

* the users are less "friendly" to the scientific goals of the project and more concerned with the way the system affects their workplace and daily tasks.

* the control over users and their environment diminishes and the risk of scientific failure from confounding causes increases.

* the value of the information to our industrial partners increases.

Some additional characteristics of field studies are:

* the participants in the studies must be able to continue in their normal job functions with minimal personal disruption.

* issues of cost, support, aesthetics and ethics become factors in selecting and installing equipment.

Equipment Selection

Several factors affect the selection of equipment for use in offices.

Cost. The study site finances the purchase of equipment. The same economic pressures that guide all of their purchases affect their media space decisions.

Aesthetics. The equipment must "fit in" at least as well as any other technological device in the users' personal environments. One of our goals is to make the appliances invisible and ubiquitous. We select equipment that matches furniture and office decor.

Features. Cost takes precedent over features, but the following elements are important to consider. We look for equipment with the fewest dials and knobs. This reduces the intimidation factor and discourages users from making unnecessary, often problematic, adjustments. Fewer controls means there are fewer opportunities to push the wrong button.

Theft. Until recently the least expensive colour video cameras available to us were small consumer camcorders, which are very attractive to would-be thieves. We now use small, inexpensive, commercially available cameras designed for desktop video conferencing.

Audio quality. It has been suggested that the audio channel makes more of a difference than the visual channel [22]. Our own anecdotal evidence is in line with these observations [20]. We find it entirely unacceptable to use the speakers in most television monitors. Their frequency response and tendency to send the sound in all directions make it difficult to prevent feedback. We use professional audio equipment to keep the quality of the audio channels high, and we are quite satisfied with the quality of consumer grade PZM microphones (Pressure Zone Microphone).

Wires. One unfortunate characteristic of using analog technologies is the abundance of wires that are required [3]. When installing a media space at a field site, the wires must be installed in a manner coordinated with the overall image being presented by the environment. Wires must be neatly hidden. We anticipate the arrival of a digital architecture [19] which will do away with the spaghetti conditions we now must manage.

Prior literature has indicated some of the technological obstacles to using off-the-shelf audio and video technology in media spaces [3,20]. We have found that users are very concerned with the way they appear on the monitor, and have witnessed significant office space rearrangement to accommodate a more attractive view from the camera's perspective.

A Room with a View

Several media space installations have cameras in public areas which increase awareness and accommodate informal interactions [2,11]. Through questionnaires and interviews we found that many users, before the installation of the media space, were concerned that the media space would invade their privacy,. They had no desire to have cameras placed in public spaces. At the beginning of our study, users were inclined to leave their equipment turned off when they arrived in the morning until they wanted to contact someone. Users, when notified of a communication attempt on their workstation, had to hastily turn on their equipment. (We attempted to hide the power switches for each of the devices, but this was not possible for the monitors and speakers we selected.) In an attempt to remedy this problem we pointed a camera out a window that had a lively street scene, providing every user with a virtual window. This had the immediate effect of encouraging people to turn on their equipment the moment they arrived in the morning.

In another experience an individual refused a media space node for their office, but they accepted our offer of a video monitor, and the electronic window. The individual became comfortable enough with the presence of the monitor that they soon requested a full media space node. A different user likened a 'black' monitor (one with no image on the screen) to a "black hole." It's mere presence was intimidating. Another user, who had a large picture window in their office, would frequently notice changes in the weather on their electronic window (which was more likely to be in their field of view) rather than use the wall-sized 'real' window.

These anecdotes suggest that using the monitor to provide a more pleasant work environment may decrease anxiety and encourage adoption.

Technical Support for a Field Study and Beyond

Telepresence is a small team of people with a full palette of research activities on our agendas. We have few remaining resources to install and support media spaces. The enabling technologies which we deploy in field studies are assembled in non-standard ways, making it difficult to find hardware vendors with the expertise to support media space installations. The software that we provide is complex and complete installation and system manuals are still under development. There is no comprehensive source of information that gives instructions on how to setup, operate and maintain a media space. All of these add up to a potential technical support nightmare.

Click here for Picture

Figure 2: Relationships among players in field studies.

Figure 2 illustrates a simplified view of the relationships among the participants in Telepresence field studies. The three main players are illustrated as ovals. They are: the Telepresence researchers, the field study users, and the industrial/commercial partners.

Users are observed at work by the Telepresence social scientists. The media space technology foundation used by the users was designed and tested by the Telepresence researchers in conjunction with the industrial/commercial affiliates. However, the actual deployment and maintenance of the system is undertaken by Telepresence industrial/commercial affiliates (in consultation with the Telepresence researchers, with diminishing involvement over time). The technology foundation is financed by the user/clients, and commissioned directly from the industrial/commercial affiliates.

Central to this arrangement is the view that the deployed technology must have a life in the client organization after the field studies. Telepresence has a central role in the initiation, evaluation and revision of the foundation technology and deployment. At the end of the study, however, Telepresence withdraws their involvement, and the system continues to be maintained by the study site and the industrial/commercial affiliates. Everything to the left of the vertical dotted line is self-contained and no longer requires the attention of Telepresence.

By following this strategy, we avoid the common trap of field studies where the technology works only for the duration of the study. If the technology has no real life expectancy users are not motivated to modify their work habits to use the technology, and any results obtained from the study are suspect. Our strategy avoids this common pit-fall, and also means that studies can be undertaken with only minimal resources from Telepresence. The primary resources for the study come from the main stake-holders in the private sector and the user/clients.

Development Cycle: Research versus Production

Our first field study at wrist's length has given us insights into the process needed to move experimental technologies out of the lab. Moving the technology outside the lab requires a degree of robustness and documentation not normally required in a community of technically advanced users. For the technical staff, who are experienced in conducting design in a research environment, these requirements required a significant shift of priorities.

The system architects/developers, who were accustomed to having the media space running under their watchful eyes, were suddenly charged with the task of ensuring that the system would run effectively and reliably in an environment over which they had no control. Thus, reporting and handling error conditions suddenly became more important, as did performance, security, and reliability. As well, having made commitments of deliverables to outside organizations to install and operate systems by certain dates, we found our management and programming practices had to change to reflect more of a product development cycle. This was often at odds with the traditional research practice in our environment.

Even in our own in-house environment the successful adoption of the media spaces by approximately 24 people has been a double edged success. On the one hand we have demonstrated the usefulness of the tools. On the other hand we have created a demand for the system that requires that it be fully operational at all times, thus decreasing our ability to explore, test, debug and otherwise research new media space concepts. We are trying to manage apparently opposing objectives of research and exploration on the one hand, and production and dependability of the system on the other.

To achieve the transition from research software to solid production software we have added several members to our engineering team. Part of their efforts have been to make the technology robust enough to be supported by third party industrial partners. This has allowed the Telepresence technology foundation to remain in place and operational after our initial field studies, thus motivating subjects to modify their work habits to use the technology in the course of the studies.

CONCLUSIONS

A media space is a different kind of technology from that normally deployed in the workplace. The evaluation of a media space requires observation and analysis of personal and interpersonal behaviour. In order to effectively study a media space we have to build and install one in a real world situation. Subjects have to have confidence in the reliability and longevity of the media space if they are to change their behaviour to take advantage of the new modes of communication afforded by the technology. Only then can we evaluate the user interface and user features appropriately. But this means that we must design and install a media space in a real world situation before we fully understand how to design and install a media space. We developed a staged approach to the problem of migrating our systems from the laboratory to the real world. We gathered and assimilated user information from early lab versions and the wrist's length versions of the technology, and used these results to affect the design, implementation and deployment of the commercial quality arm's length version of the system. This is not a typical product development environment in which we are refining a product idea. Nor is it the type of environment where the researchers are following the product through its development cycle. The field studies are research, not product introduction. As such, the problems we faced were unique and are likely to be faced by others as more researchers build cooperative work style systems.

We have documented anecdotally the major issues that arose for our group in trying to conduct our field study design research. In summary these are: (1) Field studies need technical support. Either the research staff has to provide this support or it needs to be contracted for elsewhere. The latter is difficult because noone knows how to support an entirely new system. (2) The technology in field studies needs to work at all times. Without this reliability, the users won't use the system and the field study will be worthless. This means that the research staff has to have sufficient resources and expertise to make a nearly commercially acceptable product. (3) The technology for the field study needs to have a reasonable life expectancy if users are to change their work habits to accommodate the technology. This requires support for the maintenance and update of the technology once the field study is over. Vendors are reluctant to do this for novel systems, but we have found that industrial affiliates can be a good substitute for this continuing support.

ACKNOWLEDGMENTS

For research support the authors are indebted to Technology Ontario, the Information Technology Research Centre and the Telecommunications Research Institute of Ontario, and to our industrial affiliates Xerox, Bell Canada, The Arnott Design Group, Hewlett Packard, and Sun Microsystems Inc. We acknowledge the contributions of Tracy Narine, Beverly Harrison, Tom Milligan, Chris Passier, Barry Wellman, Janet Salaff, Dimitrina Dimitrova, Laura Garton, Dick Dillon, Jo Tombaugh and Barbara Whitmer for their invaluable efforts in our first field study. Thank you. Special thanks to Barbara Whitmer, Russell Owen and Hiroshi Ishii for comments and suggestions on this paper.

REFERENCES

1. Baecker, R.M., Nastos, D., Posner, I.R. and Mawby, K.L. The User-Centred Iterative Design of Collaborative Writing Software. In Proceedings of CHI'93 (Amsterdam, April 24-29, 1993), ACM Press, New York , pp. 399-405.

2. Bly, S.A., Harrison, S.R. and Irwin, S. Media Spaces: Bringing People Together in a Video, Audio, and Computing Environment. Communications of the ACM , Vol. 26, No. 1 (January 1993), pp. 28-47.

3. Buxton, W. and Moran, T. EuroPARC's Integrated Interactive Intermedia Facility (iiif): Early Experiences. In Proceedings of the IFIP WG8.4 Conference on Multi-user Interface and Applications (Heraklion, Crete, 1990), Elsevier Science Publishers, The Netherlands, pp. 11-34.

4. Buxton, W. Telepresence: Integrating Shared Task and Shared Person Spaces. In Proceedings of Graphics Interface '92 (Vancouver, May 11-15, 1992), pp. 123-129.

5. Dourish, P. and Bly, S. Portholes: Supporting Awareness in a Distributed Work Group. In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, New York , pp. 541-547.

6. Dourish, P. The Affordances of Media Spaces for Collaboration. In Proceedings of CSCW'92 (Toronto, Nov 1-4, 1992), ACM Press, New York , pp. 17-24.

7. Egido, C. Videoconferencing as a Technology to Support Group Work: A Review of its Failures. In Proceedings of CSCW'88 (Portland, Sept 26-28, 1988), ACM Press, New York , pp. 13-24.

8. Elliott, C. High-Quality Multimedia Conferencing Through a Long-Haul Packet Network. In Proceedings of ACM Multimedia'93 (Anaheim, Aug 1-6, 1993), ACM Press, New York, pp. 91-98.

9. Fish, R.S., Kraut, R.E. and Chalfonte, B.L. The Video Window System in Informal Communication. In Proceedings of CSCW'90 (Los Angeles, Oct 7-10, 1990), ACM Press, New York , pp. 1-11.

10. Galloway, K. and Rabinowitz, S. Hole-in-Space. Mobile image videotape, Santa Monica, California, 1980.

11. Gaver, W., Moran, T., MacLean, A, Lövstrand, L., Dourish, P., Carter, K. and Buxton, W. Realizing a Video Environment: EuroPARC's RAVE System. In Proceedings of CHI'92 (Monterey, May 3-7, 1992), ACM Press, New York , pp. 27-35.

12. Gaver, W., Sellen, A., Heath, C. and Luff, P. One is Not Enough: Multiple Views in a Media Space. In Proceedings of CHI'93 (Amsterdam, April 24-29, 1993), ACM Press, New York , pp. 335-341.

13. Gaver, W., Smith, R. and O'Shea, T. Effective Sounds in Complex Systems: The Arkola Simulation. In Proceedings of CHI'91 (New Orleans, April 27-May 1, 1991), ACM Press, New York , pp. 85-90.

14. Heath, C. and Luff, P. Disembodied Conduct: Communication Through Video in a Multi-Media Office Environment. In Proceedings of CHI'91 (New Orleans, April 27-May 1, 1991), ACM Press, New York , pp. 99-103.

15. Hopper, A., et al. The Pandora Multimedia System (a videotape), Olivetti Corporation, Cambridge, England, 1991.

16. Ishii, H. and Kobayashi, M. ClearBoard: A Seamless Medium for Shared Drawing and Conversation With Eye Contact. In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, New York , pp. 525-532.

17. Li, J. and Mantei, M. Working Together, Virtually. In Proceedings of Graphics Interface '92 (Vancouver, May 11-15, 1992), pp. 115-122.

18. Lu, I. and Mantei, M. Idea Management in a Shared Drawing Tool. In Proceedings of ECSCW'91, Kluwer Academic, The Netherlands, pp. 97-112.

19. Lyles, J.B. and Swinehard, D.C. The Emerging Gigabit Environment and the Role of Local ATM, IEEE Communications, Vol. 30, No. 4, (April 1992), pp. 52-58.

20. Mantei, M.M., Baecker, R.M., Sellen, A.J., Buxton, W., Milligan, T. and Wellman, B. Experiences in the User of a Media Space. In Proceedings of CHI'91 (New Orleans, April 27-May 1, 1991), ACM Press, New York , pp. 203-208.

21. Moore, G. Social Science Models and Strategies for Telepresence User Studies, Ontario Telepresence Technical Report OTP-93-01, February, 1993, University of Toronto, Toronto, Canada.

22. Pye, R. and Williams, E. Teleconferencing: Is Video Valuable or is Audio Adequate?, Telecommunications Policy (June 1977), pp. 230-241.

23. Sellen, A.J. Remote Conversations: The Effects of Mediating Talk with Technology. In the Journal of Human-Computer Interaction, forthcoming 1993.

24. Sellen, A.J. Speech Patterns in Video-Mediated Conversations. In Proceedings of CHI'92 (Monterey, May 3-7,1992), ACM Press, New York , pp. 49-59.

25. Stults, R. Media Space, Xerox PARC technical report, 1986, Xerox Palo Alto Research Center, Palo Alto, Calif.