Prof. Markus Gross, Stephan Würmlin

blue-c: A Spatially Immersive Display and 3D Video Portal for Telepresence

blue-c [1] combines the qualities of total immersion experienced in CAVE™-like immersive projection environments with simultaneous, real-time 3D video acquisition and rendering from multiple cameras. This concept enables users to interact and collaborate inside an immersive, virtual world, while perceiving the photorealistic three-dimensional representations of their collaboration partners in real time. The blue-c project addresses a variety of challenging research issues, including immersive projection using active screens, advanced 3D video processing, network communication architectures, and user interface technology. In addition, we developed a variety of applications comprising information visualization, shopping experiences, and product development.

By bringing together competence from different departments of ETH Zurich, the blue-c project exploits the synergies arising from interdisciplinary research and harvests the knowledge and engineering expertise from different research areas. The blue-c project is pursued by a team of twenty researchers from four different ETH departments:

• Computer Graphics (Department of Computer Science),
• Computer Vision (Department of Information Technology and Electrical Engineering),
• Mechanical Systems (Department of Mechanical and Process Engineering), and
• Computer Aided Architectural Design (Department of Architecture).

blue-c is currently implemented as two portals with complementary characteristics, networked through Gigabit Ethernet. One portal is located in a restricted area of the Computing Center (RZ) at ETH Zentrum, the second one is situated in a public space in the School of Architecture (HIL) at Hönggerberg where it can also be used as multimedia room.

The blue-c project was motivated by the increasing interest in intuitive interaction with digital content, where content can be either computer-generated or a real representation of a user and his environment. This leverages the concept of telepresence by mixing virtual worlds with remotely located people and places. Telepresence can be understood as an extension of synchronous, real-time technologies such as telephone or video conferencing and enables the sensation of "being" in a remote space. Thus, by introducing a solution for telepresence, blue-c addresses the needs of our information society for high-end communication and collaboration. As a central feature of its design, blue-c integrates a real-time 3D video acquisition and processing system to display humans in three dimensions.

The blue-c system concept

The portal at ETH Zentrum consists of three twin LCD projectors with additional LC shutters which are utilized to generate a CAVE™-like immersive display empowered with active stereo. One of the technical core novelties of blue-c is the use of an actively shuttered projection screen allowing the video cameras to “see through the wall” during frame acquisition. This solves a variety of problems regarding 3D reconstruction and projection. Additional active LED illumination produces calibrated lighting conditions during the acquisition phase, which greatly improves the texture and color quality of the 3D video. Video frame capturing is triggered at a fixed rate in a small time slot between left and right eye projection. During acquisition, the shutter glasses are switched to opaque for both eyes, as are the projector’s LC shutters. These components demand precise synchronization by special-purpose microcontroller hardware. For image generation, we currently use an SGI Onyx 3200. Additional hard- and software components accomplish head tracking, spatial audio, voice communication, and 3D user interaction. Although blue-c constitutes a high-end SID, our design allows connections to remote portals with much less technical sophistication.

The second portal at ETH Hönggerberg is an example of such a simplified design. It uses a single DLP projector with active-stereo capabilities. Acquisition of the user is accomplished by 16 cameras which are placed around a single projection wall.

Software sub-system

We designed a novel 3D video engine to process multiple video streams on a Linux PC cluster. After background subtraction and silhouette extraction, we compute a so-called 3D video fragment representation of the user in real time [2, 4]. This format progressively encodes the 3D stream and facilitates efficient rendering and 3D compositing. The blue-c communication layer handles the transmission of 3D video streams, along with the events from the collaborative application. A high level, distributed scene graph API gives users full access and control over all blue-c-specific features in the displayed virtual worlds.

video: windows media, mpeg-4

Applications

To demonstrate the functionality and practical usefulness of the blue-c system, various applications have been developed in the fields of information visualization, shopping experiences and design. The blue-c API was developed and evaluated in collaboration with the application designers. These applications demonstrate the unique characteristics of spatial awareness and stereoscopic depth offered by immersive environments. Currently, video-based gesture recognition modules are in place that enable participants to use gestures for navigation or interaction.

Outlook

On May 1st, the follow-up project blue-c-II officially started with the same participants. The focus of blue-c-II is to investigate and develop fundamental methods for interactive, view-independent 2D and 3D video and display technology with the goal to acquire ordinary, open, and complex physical environments. The acquisition of such complex spaces poses great technical challenges. In addition, blue-c-II focuses on flexible and mobile setups. Many simple, off-the-shelf hardware components will be dispersed throughout the scene, creating "seas of cameras, displays, and compute nodes".

On June 23, an official demo day will be held to demonstrate the blue-c project and technology to a larger audience. Interested ETH employees can see blue-c in action both at ETH Zentrum and at ETH Hönggerberg. Registration is required and can be done through the blue-c web page [3]. Requests will be served on a first-come, first-served basis.

Literature

[1] Gross M., Würmlin S., Naef M., Lamboray E., Spagno Ch., Kunz A., Koller-Meier E., Svoboda T., Van Gool L., Lang S., Strehlke K., Vande Moere A., Staadt O., "blue-c: A Spatially Immersive Display and 3D Video Portal for Telepresence". In Proceedings of ACM SIGGRAPH 2003 (Transactions on Graphics), pp. 819-827, ACM Press, 2003.

[2] Würmlin S., Lamboray E., Gross M., "3D Video Fragments: Dynamic Point Samples for Real-time Free-Viewpoint Video". In Computers & Graphics 28(1), Special Issue on Coding, Compression and Streaming Techniques for 3D and Multimedia Data, pp. 3-14, Elsevier Ltd, 2004.

[3] blue-c project web page: http://blue-c.ethz.ch/

[4] 3D video project web page: http://graphics.ethz.ch/3dvideo/