TY - JOUR
AU1 - Kim, Minju
AU2 - Wohn, Kwangyun
AB - Abstract While visualization alone can play a meaningful role, visualized images today often fail to invoke active engagement from the audience. In numerous cases, a human presenter’s interactive involvement can drastically enhance the engagement with visualization. In this paper, we propose and further develop a novel concept called augmented presentation, in which a human presenter occupies the same physical space as the visualization, thereby presenting and interacting with the visualized information seamlessly. We focus mainly on investigating the potential of the presenter by referring to the interaction framework for visualization and the proposed presenter’s roles. Depending on the type of interaction and level of engagement, the presenter’s role may vary from a narrator to an augmenter who may be regarded as part of the visualization. Further developing the idea of augmented presentation, we designed and implemented a presentation system named HoloBox. Then, we conducted two user studies to investigate the possibilities of HoloBox as a new presentation channel. The results suggest that our new form of augmented presentation has the potential to enhance the quality of presentations and enrich the audience experience with regard to visualization. In terms of the presenter, five out of six participants chose HoloBox as their preferred presentation method despite the extra effort required to make a presentation. We believe that our augmented presentation can support new forms of visualization and information delivery. RESEARCH HIGHLIGHTS Presents and develops augmented presentation concept, in which the human presenter intervenes as a part of 3D visualization interactively and enhances the visualization delivery. Designs elements of augmented presentation concept and implements it through a HoloBox system. Conducts two user studies to investigate the feasibility of augmented presentation. 1. INTRODUCTION Visualization aids in presenting complex and abstract data using visual metaphors to help people better understand and gain insight into raw quantities. Rather than just showing a visualization, presenting visual content to people is more effective in fields such as education, entertainment and advertising. In many cases, visualization is transported by the human presenter. At this moment, the behavior of the presenter is as crucial as his or her message. The presenter’s involvement in visualization can provide cognitive and learning benefits to the audience’s data perception. For example, the presenter’s facial expressions, gestures and verbal explanations may help the audience focus on relevant visual information and build a connection between multiple representations (Fiorella and Mayer, 2016; Krekhov et al., 2017). In addition, through the presenter’s behavior, complex visualizations can be delivered in a manageable form, thus guiding the audience to access the information more deeply and facilitating the information perception process (de Koning and Tabbers, 2011; Mayer, 2002). To go beyond the traditional presentation method where a presenter shows and explains information, there have been recent attempts to tightly combine the characteristics of presenters and visualization. These attempts can be divided into direct interaction and visual integration issues. Regarding the interaction issues, there have been attempts to complement visual communication by raising connectivity through the presenter’s direct input and control. In particular, gestures, one of the important parts of the presenter’s behavior, have been developed in many areas because they can help the audience to understand data by easily linking what is currently occurring in the visualization (de Koning and Tabbers, 2011; Fourney et al., 2010; Krekhov et al., 2017; Zeng et al., 2012). Other researchers have explored visual integration that combines the presenter and visualization by enabling the audience to see multiple representations in the same visual field (Ainsworth, 2014; Liston et al., 2000). To support this, various display techniques (e.g. using half-mirror displays) have been developed. This kind of presentation recently has gained popularity in many areas such as showcases and performances (Bimber and Raskar, 2005; Geng, 2013). However, previous research on developing interaction and visualization techniques does not sufficiently consider the integration of the digital information and the presenter in a single integrated visualization framework. Even when the presenter and the visualization were dealt with together in the same context, they were not successfully merged spatially. In addition, owing to a lack of analysis of integrated visualization, especially regarding the audience and presenter, the presenter’s roles and application scenarios could not be expanded. In this paper, we propose and further develop a novel form of representing and delivering information called augmented presentation. Rather than just displaying information, augmented presentation is designed to enable the human presenter to express and interact with the visualization by occupying the same physical space as the visualized information. For this, we design a presentation space and focus mainly on discovering the potential of the presenter by referring to the interaction framework for visualization (Card et al., 1999). Depending on the type of interaction and degree of involvement in the visualization, the presenter can perform roles as a narrator, manipulator and augmenter. Thus, the presenter can further enrich the information delivery process in various forms by interacting with 3D visualized information that floats around him or her. In this way, the presenter can provide a context for visualization while encouraging the audience to experience a more engaging style of presentation and elicit deeper participation. To manifest and analyze the proposed concept of augmented presentation, we have designed and further developed HoloBox. The presenter stands between two screens: a half-mirror film on the front and a wall projection screen on the rear. The 3D stereoscopic images are projected on both screens, which reproduce a continuous 3D visualization space that is filled with the digital information and the presenter simultaneously. We describe a spatial visualization method and develop interaction technique that allows the presenter to perceive 3D visualization space and manipulate digital information based on gesture interaction. We then conducted two user studies to investigate the feasibility of HoloBox. First, we conducted a formative user study with six presenter participants. This helped us understand the characteristics of HoloBox for delivering augmented visualization, especially in terms of visualization representation and the presenter’s interaction facilities. Second, we conducted an experiment to examine the subjective level of engagement of the audience with HoloBox as compared with the traditional presentation with 18 participants. Our preliminary study suggests that most presenters favored HoloBox because they can combine and interact with visualized information physically and cognitively, even though extra effort is required to present with HoloBox. Furthermore, augmented presentation has the potential to enhance the quality of information presentation and enrich audience engagement with visualizations. This paper is structured as follows. We begin with the related study in Section 2, and describe the concept of augmented presentation, mainly focusing on the opportunities of the presenter in Section 3. Then, in Section 4, the spatial visualization technique and interaction method are described. In Section 5, we describe HoloBox implementation, including the hardware configuration and software. Then, in Section 6, we evaluate the feasibility of augmented presentation concept and HoloBox both for presenters and audiences and discuss its results. Lastly, we discuss findings, limitations and future work in Section 7, and conclusions in Section 8. 2. RELATED WORK 2.1. Information visualization with human presenter Visualization aids in presenting abstract and concrete data through graphical means to help people better understand visual content. Rather than simply showing the visualized information, communicating visual content to the audience is more effective in many areas. In many cases, a human presenter’s active involvement is one of the main factors that make a presentation more powerful and compelling (Gale, 1990; Krekhov et al., 2017). During the presentation, the presenter utilizes a variety of means such as gestures and verbal explanations to help the audience better make sense of the visual material (Mayer, 2002). For instance, it has been shown that the presenter’s hand can serve as a visual cue to direct the audience’s attention toward related parts of the visualization and that verbal explanation allows the audience to integrate multiple contents in a coherent manner (Curtis et al., 2016; de Koning and Tabbers, 2011; Haider et al., 2016). Especially, Fiorella and Mayer (2016) suggested that the presence of the instructor’s hand can motivate learners to make sense of the visualized information by providing a significant social cue. Likewise, many researchers have explored and verified the impact of incorporating the human presenter in the visualization for enhancing the audience’s perception of data. According to these characteristics of the presenter, interactive presentation systems and interfaces have been developed to support the presenters’ access to information and their direct control (Craig and Amernic, 2006; Cruz-Neira et al., 1992; Edge et al., 2013; Isaacs et al., 1995). In some cases, by intervening as a virtual or digital conversational agent, the presenter delivers visualization and interacts with the audience in real time (Buisine and Martin, 2007; Noma et al., 2000; Prendinger et al., 2007; Van Welbergen et al., 2005). Thus, the human presenter participates in the visualization in various forms for richer visualization delivery and better data perception for the audience. 2.2. Visual integration of visualization and presenter With regard to information visualization, it is effective to visually incorporate associated information because this enables the viewer to quickly grasp the relationships between pieces of information, compare different forms of information and to get engaged in the information (Liston et al., 2000; Moreno and Mayer, 1999). Similarly, in the context of presentations, there have been many attempts to optically integrate visualization with the presenter. Figure 1 shows forms of composition of the visualization and the presenter according to the degree of spatial integration between the two. ‘Detached’ is a style where the presenter delivers the visualization by standing in front of, beside or away from it, as in the traditional form. With recent advances in display technology, it has become possible to present naturally blended views of digital information and the human presenter in the form of ‘frontal immersion’ (Bimber and Raskar, 2005; Geng, 2013). Optically combining the presenter and visualization helps the viewer perceive an integrated view and breaks the boundaries in between. In particular, by intervening directly as a part of visualization, the human presenter can control digital content directly and express the information more realistically. For example, Rosling (2010) used half-mirror film and floating visualizations on the presentation stage. Then, he explained it by standing directly behind the screen. Rakkolainen et al. (2009) and Lee et al. (2009) also proposed a presentation system where digital information floats in front of the presenter and is merged with the presenter using a fog screen. However, most research studies have focused on developing visualization and interaction techniques to present and control information in an effective and efficient way. In addition, previous research studies have not dealt with the integration of the presenter and the visualization in a single integrated visualization framework. Even though the presenter’s intervention in the visualization was considered, the information and the presenter were typically not combined spatially or seamlessly. This limits the presenter to traditional forms of presentation and does not permit an expansion of application scenarios. In this paper, we propose augmented presentation in the last column of Fig. 1, which is capable of ‘spatial immersion,’ and analyze the characteristics and advantages of such an integration. Furthermore, we introduce a visualization and interaction technique that seamlessly merges digital information and the presenter. We also introduce the possibility of the human presenter’s role in the 3D visualization space in detail. FIGURE 1. View largeDownload slide Forms of the presentation with a human presenter according to the degree of spatial integration between visualization and the presenter. FIGURE 1. View largeDownload slide Forms of the presentation with a human presenter according to the degree of spatial integration between visualization and the presenter. 2.3. Uses of half-mirror displays for visualization Various display technologies have been developed to create seamless visual integration between physical and virtual information. Among them, the use of half-mirror film is one of the most realistic ways to produce a seamless image of synthetic imagery and physical objects in the space behind it. According to such characteristic, the technique has been widely used in various fields such as performances (Gingrich et al., 2013, Steinmeyer, 1999), marketing, showcases (Bimber and Raskar, 2005) and telepresence (Ogi et al., 2001). Also, it has recently been personalized and used in desktop workspaces where direct user input is required (Hachet et al., 2011; Hilliges et al., 2012). In addition, visual technologies have been developed to reproduce the digital contents realistically by expanding the virtual space through the half-mirror film. For example, several researchers have expanded the visualization space by placing additional displays that are physically separated from each other behind the half-mirror display in parallel, typically in the form of a multi-layered display (Akeley et al., 2004; Dünser et al., 2008; Prema et al., 2006). Also, Kim et al. (2014) and Olwal et al. (2005) applied 3D stereoscopic image on a half-mirror film to extend the visualization in 3D space and integrate it into the real environment naturally instead of just fixing visual forms on a flat screen. However, previous research findings have focused on the registration issues in order to integrate the physical object into the 3D information space flawlessly. Expanding the scope of perspective, in this paper, we explore the characteristics and new possibilities of information representation and delivery when the human presenter occupies the same physical space as the visualization does, thereby presenting and interacting with the digital information seamlessly. Kim et al. (2016) presented augmented presentation concept and discussed issues associated with presenting the human presenter and virtual information via a half-mirror display. They have specifically focused on depth visualization issue, which aims to merge the presenter and visualized information in 3D space flawlessly. Moreover, though they discussed the opportunities of the presenter in an augmented presentation, the presenter’s possibilities were not assessed systematically and analyzed concretely. We take this exploration further and focus on discovering the potential of the presenter by referring to the reference model for visualization. Furthermore, after improving the presentation system, we conduct feasibility studies both for the presenter and audience to investigate the potential of augmented presentation as a new form of visualization delivery method. 3. AUGMENTED PRESENTATION 3.1. Concept In augmented presentation, rather than just focusing on showing the visualization to the audience, the human presenter intervenes as a part of the visualization interactively and enhances the visualization delivery process. The presenter occupies the same physical space as the virtual information, presenting and interacting with the 3D digital information in a more realistic way (Kim et al., 2016; Fig. 2). Specifically, the presenter can directly encounter and reach into the virtual information that is presented in front of, beside, behind and above him/her by moving within the presentation space himself. Beyond the traditional roles as a slide navigator and explainer, the presenter can further enhance the visualization by explaining and realistically manipulating the data or visualized forms that are floating around him/her. In addition, by intervening as part of the information via the presenter himself or by physical objects that he/she handles, the presenter can supplement the visualized data. FIGURE 2. View largeDownload slide Augmented presentation concept with a spatially integrated human presenter. FIGURE 2. View largeDownload slide Augmented presentation concept with a spatially integrated human presenter. According to the level of engagement and type of interaction in the visualization, the presenter may play several different roles including narrator, manipulator and augmenter, who may be considered as part of the digital information (Fig. 4). Likewise, the audience can observe 3D visualization by integrating the presenter with the structure of various information sources, both of which occupy the same physical space perceptually. Furthermore, the presenter’s active intervention enables the audience to watch the visualization in a more engaging and immersive way. 3.2. Augmented presentation space The presentation space in augmented presentation is the space where the 3D visualization is shown to the audience and the human presenter intervenes directly. These are then optically combined. In addition, the presentation space is where the same visualization shown to the audience is displayed for the presenter to provide visual feedback. The half-mirror front screen and the wall projection back screen that are arranged in parallel compose a 3D visualization space, and the human presenter stands between these two screens. At this time, the audience stands outside the presentation space and can watch the overlapping images of 3D visualization on the two screens and the presenter simultaneously (Fig. 3, left). The half-mirror film enables the realistic display of 3D visualized images in mid-air. At the same time, the rear screen behind the presenter expands the presentation space physically, allowing coherent 3D visualization that is connected to the front screen. The rear screen enhances the audience’s spatial impression through an increased number of displays (Lee et al., 2009). In addition, by increasing the amount of visualization space, the rear screen allows the presenter to express supplemental or contextual data simultaneously with the main information. In addition, instead of just overlapping two screens and the presenter, we project a 3D stereoscopic display on both screens. This creates a continuous visualization space filled with the visualized information and completely includes the human presenter in the 3D presentation space. Therefore, we can place digital information at any depth in the presentation space by adjusting the binocular disparity of the virtual information shown on either of the two focus planes. At this point, dividing the stereoscopic display on the front and rear screens can help the audience to recognize 3D content with less visual discomfort. For this reason, previous studies on displaying stereo images showed that vergence–accommodation conflicts could cause visual discomfort and poor stereo performance. As one way to solve this problem, several researchers suggested that rather than presenting images on a single focal plane, using multiple focal planes and placing images near the focal plane could eliminate vergence–accommodation conflicts and reduce visual fatigue (Kim et al., 2014; MacKenzie et al., 2012). In addition, after separating the 3D visualization space into front and back spaces in accordance with the depth position of the presenter, and displaying information accordingly on each screen, we can present accurate occlusion cues without a complicated rendering process (Fig. 3, right). Furthermore, visualization for the presenter is displayed on the opposite side of the audience’s view (back side of the film mirror), so the presenter can show and control the visualization more naturally by simultaneously perceiving the 3D visualization space and focusing on the audience’s reaction (Fig. 8). We discuss how we realize this in the design and implementation section. FIGURE 3. View largeDownload slide Audience’s view of augmented presentation (left), and application of two stereoscopic 3D screens and the presenter in 3D visualization space (right). FIGURE 3. View largeDownload slide Audience’s view of augmented presentation (left), and application of two stereoscopic 3D screens and the presenter in 3D visualization space (right). 3.3. Opportunities of the human presenter One of the primary roles of the human presenter is to support and enhance communication between the visualization and the audience. In this paper, to discover the new possibilities of the presenter’s roles in augmented presentation, we investigated the existing interaction classifications for information visualization. A number of researchers have published taxonomies of interaction techniques for data exploration and interpretation. In addition, several researchers have discussed the place of interaction throughout the process of mapping raw data to the visual form. Although most of the past research studies focused on the interaction between visualization and users who observe and manipulate the information directly, we were able to explore (from the perspective of a presenter who handles and delivers the information instead of the user) the possibilities of the presenter working with the visualization (Shneiderman, 1996; Ward et al., 2010; Yi et al., 2007). In particular, we refer to the ‘reference model for visualization’ that indicates interactions (operations) that can be applied to the four distinct stages of delivering data to the audience at a high level (Card et al., 1999). This model shows a basic interaction process in which raw data is transformed into visual forms through data transformation, visual mapping and view transformation. In this pipeline, we investigated the possible interaction operations that a human presenter in an augmented presentation can execute. In augmented presentation, the presenter can assume the roles of narrator, manipulator or augmenter depending on the level of engagement and type of interaction in visualization (Fig. 4). FIGURE 4. View largeDownload slide Presenter’s roles in augmented presentation; narrator, manipulator, augmenter. FIGURE 4. View largeDownload slide Presenter’s roles in augmented presentation; narrator, manipulator, augmenter. 3.3.1. Narrator The narrator plays a significant role in the view transformation stage, which is the closest interaction process with the audience. Although the presenter is not directly involved in controlling the visualized entities, he/she turns data insights into a narrative and communicates this story to the audience. For example, by approaching an analyzed and visualized representation and gazing at it, the presenter can add narration to deliver information in a more descriptive and compelling manner. When offering the same data features in several different views, the presenter supplements the verbal description to make it easier for the audience to understand. In particular, by placing the visualization (which is floating in front of him/her) and the audience in the same line of sight, he/she can explain the information by focusing on both the information and the audience’s reaction. At the same time, the presenter can create an appropriate context for the presentation with the help of supplementary information such as background images and 3D models, which are located in the extended space behind him/her (Fig. 5a). At this time, the audience may participate and be engaged in the visualization more actively by shifting its attention with the presenter’s explanation and by looking at the rich visual content simultaneously. FIGURE 5. View largeDownload slide Application scenarios of augmented presentation with a spatially combined human presenter. Depending on the level of engagement and type of interaction in visualization, the presenter’s role may vary from a narrator (a), and a manipulator (b, c, d), and to an augmenter (e, f). (b), (c), and (e) were captured in mono for visual clarity and the others were captured in stereo. FIGURE 5. View largeDownload slide Application scenarios of augmented presentation with a spatially combined human presenter. Depending on the level of engagement and type of interaction in visualization, the presenter’s role may vary from a narrator (a), and a manipulator (b, c, d), and to an augmenter (e, f). (b), (c), and (e) were captured in mono for visual clarity and the others were captured in stereo. 3.3.2. Manipulator A manipulator interacts with the visualization more actively than a narrator does. While the narrator shares the information by speaking out loud, a manipulator can use gestures to help him/her control the presentation in a more tangible and realistic way. From a small unit of information to the overall visualization view, the manipulator controls a wide range of factors in the presentation and interacts with the visualization by intervening in all three interaction stages: data transformation, visual mapping and view transformation. First, in the data transformation stage, the presenter can navigate raw data surrounding him/her, then grasp and select visual content directly by stretching out his/her hand. Similar to dealing with a physical object in the real world, the presenter can contain the chosen information in his/her hand and analyze it in detail (Fig. 5b). In addition, the presenter can assemble the retrieved data into more compelling forms. At this time, he/she can move other supplemental data to the rear area to highlight specific target information. Second, in the visual mapping stage, the presenter can show different arrangements or representations by changing the visualization type, or show different visual structures by directly controlling the visual elements and variables (Fig. 5c). Alternatively, he/she can arrange similar data structures to overlap at different depths through gestures, or establish logical connections in depth space to help the audience identify the information more easily. Third, in the view transformation stage, the presenter can show various aspects of visualization by changing the view attributes of the visualization in real time, e.g. rotating part of the visualization, adjusting the ratio or changing the viewpoint (Fig. 5d). 3.3.3. Augmenter The augmenter plays the role of mapping the visualization to a physical form by completely intervening as part of the visualization (visual mapping stage). The shape, size and characteristics of the human presenter become another set of physical information and are used to make existing visual content more realistic and practically useful. Sometimes, the presenter can augment the visualization through a physical object (information) that he/she carries and handles. For example, the presenter’s body can be used as a visualization structure or interface, while physical cues throughout parts of his/her body may present the information more intuitively and at spatially appropriate locations. Information can be mapped to the presenter’s height and the distance between his/her arms, and physical sense of the digital information can be added by allowing the audience to perceive the size and form of the abstract visualization intuitively (Fig. 5e). Furthermore, after inputting certain digital information structures into the physical object (information) that the presenter handles, the presenter can integrate that object with the digital information in the presentation space (Fig. 5f). For example, after assigning a particular data structure according to the properties of the physical object operated by the presenter, he/she can supplement and explain the information by integrating it with other virtual information floating in the space. At this stage, we set the order of the three roles of the presenter according to the type of his/her participation in visualization enhancement. On the one hand, as a narrator, he/she just adds verbal explanation but is not directly involved in controlling the visualization. On the other hand, as a controller, he/she manipulates visual forms or data by using existing virtual information in the presentation space. Finally, as an augmenter, he/she uses other physical objects, such as even himself/herself, for augmented visualization delivery. Even though we divided the presenter’s roles into three forms, in practice the roles of the presenter can be seamlessly combined with each other, and also the roles can take another different dimension to define themselves. 4. HOLOBOX DESIGN To implement our augmented presentation concept, we created and further developed a prototype system named HoloBox. For this setup, we need to consider two important issues, one each for the audience and the presenter. First, for the audience, we need to offer a visualization technique that integrates the presenter and two parallel displays seamlessly, allowing the audience to experience a continuous and immersive presentation space. Second, for the presenter, we need to provide an interactive presentation environment that can support the diverse roles of the presenter while being fully involved in the 3D presentation space. 4.1. Spatial integration To enable the audience to observe augmented presentation effectively, we focused on constructing a visualization space to allow the audience to perceive 3D integrated visualization in more efficient and stable ways. We first discuss a method to organize the 3D presentation space using both displays and the presenter to provide an effective visualization experience to the audience. Second, we discuss a visualization technique to express digital information effectively even at the depth boundary between the 3D displays generated by the two screens. A depth boundary inevitably occurs when two parallel displays are optically overlapped. For this, we referred to previous research that approached this subject technically and empirically for similar visualization issues (Lee et al., 2009; MacKenzie et al., 2012). 4.1.1. Utilization of depth visualization space Previous research looked into the possibility of enabling the viewer to perceive an integrated 3D visualization space through two layered stereoscopic displays (Kim et al., 2014). The researchers found that the 87.5% of participants recognized the depth accurately, but large disparities can disturb the participants’ correct depth perception. Therefore, Kim et al. (2016) conducted an additional user study to explore the depth visualization capabilities of each screen and the physical space between to provide the audience with more precise 3D presentation space. At first, the researchers anticipated that a front display that renders positive parallax would be beneficial for a larger depth range compared with a back display as asserted in the comfort zone of stereoscopic theory (Sun and Holliman, 2009). However, the result suggested that it is more effective to let the back display cover more of the depth range. To be specific, the results showed that when presenting digital information at ~33.3% of the front area on the front screen and at ~66.7% of the back area on the back screen, the participants perceived the most stable and correct 3D information volume. For this reason, the researchers assumed that both the presence of a physical object and the transparent nature of the half-mirror display affect the results. In addition to Kim’s exploration, there are several approaches to optically overlap two stereoscopic displays with the presenter at the same time. Taking the intuitive approach in this paper, in the case where the presenter occludes the screen, we divide the depth space according to the depth position of the presenter and assign the front and back displays to manage the relative area, respectively. Then, the depth area of the remaining range is divided according to the results of Kim et al. (Fig. 6, left). FIGURE 6. View largeDownload slide Divide the presentation space according to the presenter’s depth position and results of Kim et al. (left) and transit between two planes (right). FIGURE 6. View largeDownload slide Divide the presentation space according to the presenter’s depth position and results of Kim et al. (left) and transit between two planes (right). 4.1.2. Depth boundary filtering Since two layered displays optically overlap and operate at the same time in HoloBox, the occurrence of a depth boundary between two 3D screens is difficult to avoid. Therefore, when 3D digital information is presented in the boundary area or transit between the two displays, a discontinuity phenomenon can be observed. To solve this problem, in a multiplane display system, previous research studies explored presenting digital information flawlessly at the depth boundary without disturbing the entire context. They used a spaced stack of more than two displays or set the space between the layers to be very narrow to generate coherent 3D volumetric images (Akeley et al., 2004; MacKenzie et al., 2012). However, unlike the physical setups of previous systems, HoloBox has two focus planes arranged in parallel at a distance of 1.5 m. To create a seamless continuity around the depth boundary between the two displays of HoloBox, we apply a depth filtering approach for two-layer displays (Lee et al., 2009; MacKenzie et al., 2012). To be specific, when expressing information in a volumetric display that is comprised of a number of widely spaced image planes, depth filtering distributes the image intensity at each display plane according to the depth position. This allows for spatial antialiasing within the digital information plane and a continuous 3D representation. In addition, when visualized information is moving into the depth boundary area, the content of the visualized information is replaced with gradual intensity gradients. According to the depth position, the front and back layers of the visualized information are distributed in a linear proportion. Then, when the depth-filtered voxels of visualized information are summed optically, the gradual intensity of the voxel images changes, thus ensuring the audience perceives a coherent volume visualization (Fig. 6, right). 4.2. Spatial interaction With regard to spatial interaction, there are several issues to consider to enable the presenter, who is fully immersed in the 3D presentation space and actively interacts with the 3D visualization. 4.2.1. Spatial visual feedback It is important to offer proper spatial visual feedback to the presenter who is shown in the information space and who augments and is combined with the 3D digital information allowing him/her to recognize the visualized information. As a half-mirror-based system reflects the virtual images to the direction of the audience’s viewpoint, in conventional methods, digital information is invisible to the presenter who is standing in the system. Only the audience can see the digital information. Therefore, up to now, presenters need to thoroughly acquaint the digital information in advance in live presentations, performances and many other systems that have used a similar half-mirror system (Gingrich et al., 2013). However, there are limits to the abilities of the presenters to fully understand the forms and the positions of all the digital information beforehand. Also, in these cases, there are limitations that the presenter is only able to perform the predefined scenarios. Above all, it is hard for the presenter to make eye contact with the audience making the presentation less attractive. Recently, Kim et al. (2016) proposed a method of displaying the same visualization in mid-air as a guide image, but it was shown in 2D flat screen disturbing the presenter’s accurate depth perception. In this paper, we display the same 3D digital information shown to the audience also on the back side of the film mirror to provide spatial visual feedback to the presenter. Wearing the stereoscopic glasses, the presenter can proceed the presentation smoothly by perceiving the 3D position of digital information floating around him/her and the presenter’s position through the back side of the half-mirror film (Fig. 8). We discuss how we realized this in Section 5. 4.2.2. Gesture-based spatial interactions It is crucial that the human presenter has an immersive and interactive environment available that lets her manipulate the 3D visualization entities directly and naturally in the presentation space. Then, the presenter can access the presented information easily, and the system can facilitate interaction with the information. In this paper, we provide for interaction within the HoloBox prototype by recognizing the spatial position and type of gestures of the presenter, then making it fully connected with the visualization interactively. Once the presenter enters the visualization space, the digital information displayed on each screen composes 3D presentation interactively according to the presenter’s depth position (Fig. 6). Then, the presenter interacts with digital information and plays proposed roles via gestures or with a physical object that he/she handles. In the current stage regarding manipulation through gestures, the presenter can manipulate information in three ways: selection/translation, rotating and scaling. Selection and translation. The presenter can select digital information as if he/she is to grab a physical object through a left or right one-handed grabbing gesture. If digital information is grabbed and the hand intersects the information, it becomes attached to the presenter’s hand, and digital information can be moved freely within the 3D data volume. Afterward when the presenter opens his/her hand, the information remains unselected at that position. Rotating. Digital information can be rotated when both hands are placed around the certain visualization entity and the orientation of presenter’s both hands is changed. Scaling. Digital information can be uniformly scaled using a two-handed compression gesture. After placing his/her hands, the scaling factor of digital information is a function of their distance. We implemented relatively, when the presenter first assumes the compression posture, a distance of his/her hands is identified as 100% scale. Changing the distance between both hands then scales the digital information. Also, through the physical object tracking, the system can recognize basic objects, for instance, box and sphere, and they can be merged and displayed with the digital information in the 3D presentation space. At current stage, we support only the basic interactions. However, further studies that design presenter-centered interaction will enable him/her to easily access the information and provide opportunities for richer augmented presentation to the audience. 5. HOLOBOX IMPLEMENTATION 5.1. Hardware HoloBox uses two large displays that are large enough to cover the presenter’s upper body. The size of the front screen is 1280 mm × 720 mm, and the back screen is 1920 mm × 1080 mm. We installed two projectors (120 Hz) on top of the system to project 3D stereoscopic images onto a bottom, respectively, rear screen. Each Digital Light Processing projector displays 1280 × 720 pixels images at 3000 ANSI LUMEN. We applied an active shutter 3D system, as polarization based 3D display cannot be used with our diffusion screens and active shutter 3D systems generally have higher contrast. The front display of HoloBox uses the image projected onto the bottom screen and shows it to the audience via a half-mirror film with ~50% light transmission. The half-mirror film is installed at a 45-degree to the direction of the audience and reflects digital content, manifesting it as if it was floated in the mid-air. In addition, to provide the same digital information to the presenter in real time, we installed a total reflection mirror that has high reflectivity on the ceiling. A standard wall projection screen, placed in the back, serves as the back display. The two displays are placed in parallel and 1.5 m apart from each other physically, and the space in between is used to place the human presenter. In addition, two Kinect depth cameras are installed at the top frame of the system, facing both the presenter and the audience, respectively, to track their movements in real time. We installed controllable lights through an ARDUINO at the top of the system to improve the visibility of the human presenter. Our physical setup of our current HoloBox is illustrated in Fig. 7. The complete operation of HoloBox can be viewed in the accompanying video. FIGURE 7. View largeDownload slide Physical setup of our current HoloBox prototype with main components labeled. FIGURE 7. View largeDownload slide Physical setup of our current HoloBox prototype with main components labeled. In addition, to provide the same digital information to the presenter in real time as spatial visual feedback, we use a total reflection mirror and place it on the ceiling parallel to the bottom screen (Fig. 8). The mirror reflects the 3D virtual images from the bottom projection screen and then reflects it back via the (another side of the) half-mirror film to the presenter. Through this process, both 3D digital information projected on the back screen and the presenter itself can be reflected in the mirror and the film at the same time and also be provided to the presenter as spatial visual feedback. Although the presenter cannot yet directly feel as if 3D visualization is floating around him/her in the current HoloBox prototype, he/she can interact with digital information by perceiving the digital presentation space including the presenter himself through the images floating in front of him/her as if looking into a mirror. FIGURE 8. View largeDownload slide Light path of the front/back stereoscopic displays and presenter for the 3D visual feedback. To express the visual feedback image more clearly, we manipulated contrast of the image. FIGURE 8. View largeDownload slide Light path of the front/back stereoscopic displays and presenter for the 3D visual feedback. To express the visual feedback image more clearly, we manipulated contrast of the image. 5.2. Calibration and sensing Even though the presenter appears in the presentation space, it is important to express integrated visualized information naturally and accurately without disturbing the presentation of the existing information. Also, it is crucial to allow the audience to see the digital information as their attention is correctly registered on the presenter’s manipulation consistently. To support this, our prototype uses two Kinects to track the position of both the presenter and the audience. At first, to guarantee the tight spatial coupling between the Kinect and the virtual visualization space, we spatially calibrated the system. We used a standard checkerboard-based method to retrieve the calibration of intrinsic and extrinsic parameters of the Kinect (Hilliges et al., 2012; Zeng et al., 2012). Then, we can track the position of the presenter’s body and hand joints in real time without any body worn device. We recognize and track the 3D pose of the body and both hands of the moving presenter by using skeletal tracking of the Kinect Software Development Kit. Then, we identify the presenter’s hand gesture based through a gesture recognition algorithm. In our current prototype, we borrow well-established interaction techniques from the 3D user interface community and implement basic interactions such as selection and translation, rotating and scaling 3D digital information. In addition, we track 3D pose of the physical object that the human presenter handles with Vuforia (2013). As a result, the digital information can be aligned according to the acquired position of the presenter or physical object, and the visualization can be accurately manipulated by the presenter’s gestures. Furthermore, to support the audience’s coherent viewing experience, one depth camera (Kinect) faces the audience and tracks the position to enable the motion parallax. It tracks the position of the audience’s head and links it with the cameras which render graphical elements in the virtual 3D space. This enables the audience to see the digital information as their attention is correctly registered on the presenter or the physical information consistently. Thus, it ensures the continuity of the spatial experience, increasing the interactivity between the information and the audience. 5.3. Optical calibration HoloBox is composed of stacked displays using a half-mirror film and a wall projection screen. As the perceived brightness and color of projection onto a wall are higher than projection onto a half-mirror film, it is essential to adjust the color levels and brightness on the two screen (Lee et al., 2009). In order to overcome this problem, we calibrated the projectors to accomplish intensity consistency between the two final images. In addition, in our software, we adjusted the color level of the final images rendered, allowing the audience to perceive equivalent images. Furthermore, we leverage the depth data of the presenter from Kinect in real time to handle the boundary between the depth ranges covered by the two displays. After mapping the presenter’s depth values to the virtual space, we adjust the depth range that each display occupies adaptively in accordance with the tracked presenter’s position. We implemented our system in Unity3D. In the Unity3D virtual environment, we set two virtual cameras that account for the front and rear screen, respectively. We wrote a custom shader that discards the pixels by comparing their Z-values with the corresponding value from the presenter’s depth map. In the case where the presenter does not occlude the screen, we applied our results for the suitability of depth ranges from Section 4.1. This enables us not only to compose the presentation space effectively by using two screens simultaneously, but also to present correct occlusion without a complex rendering process (Fig. 6, left). Furthermore, to handle visualized information passing through the depth boundary, we created front and back material layers for each item of digital information. Thus, whenever digital information crosses the depth boundary, we activate both layers applying appropriate intensity gradients (Fig. 6, right). Therefore, the audience can perceive consistent and continuous volume information through the two optically blended material layers. 6. FEASIBILITY STUDIES OF HOLOBOX We conducted two user studies to investigate the possibilities of HoloBox as a new visualization and presentation medium. We first ran a preliminary feasibility study to probe the unique characteristics of HoloBox for presenters. We then conducted an experiment to compare HoloBox with the traditional presentation type in terms of a subjective level of engagement of the audience. 6.1. HoloBox as a new presentation channel 6.1.1. Participants and study design We conducted semistructured interviews with six visualization and presentation professionals (two females). Participants were experienced in expressing and communicating different kinds of information in different environments. The participants included two visualization experts who were familiar with displaying information through holographic displays (P1 and P2), two experts specializing in delivering information in special environments such as museums, medical and biological areas (P3 and P4), and a teacher and a student who express and deliver information in ordinary settings (P5 and P6). Each interview session was carried out for ~90–120 min, and each participant received a $30 gift card for his or her time. 6.1.2. Procedure At the start of the interview, we asked the participants to fill out a questionnaire to collect background information about their degree of experience with holographic displays, stereoscopic images and presentation skills. Then, we gave the participants a brief introduction regarding the concept of augmented presentation and HoloBox. In the first stage, to help them understand the system, the participants observed a presentation conducted through HoloBox as an audience. Afterward, the participants entered the system as presenters. Each participant engaged his/her entire body as part of the 3D visualization and controlled different types of visual elements directly through gesture interactions. This stage familiarized the participants with the features and 3D interfaces of HoloBox. In the second stage, participants had time to familiarize themselves with two types of simple datasets (InfoVis and SciVis) that were prepared in advance, and made presentations using this data via the HoloBox system. At this time, one of the experimenters watched the presentation as one of the audiences, and where necessary, the experimenter responded by answering the presenter’s questions, nodding, or making eye contact. We informed the participants about the possible ways and roles in which they could interact with our system. However, the experimenter did not intervene unless the participants had trouble using the system. After the presentation, we interviewed the participants. During the interview, we asked them about the feasibility of augmented visualization and presentation as a new means of presentation. In particular, we focused on two aspects: visualization representation and the presenter’s interaction facilities for information delivery. These are essential elements in the visualization and presentation fields (Lanir et al., 2008). At this stage, the discussion was mainly focused on evaluating the potential of HoloBox rather than its usability or accessibility. 6.1.3. Datasets and visualization scenarios In this user study, we prepared two visualization types, information visualization and scientific visualization scenarios, depending whether the data is abstract or physically existing (Card et al., 1999). One is 3D network visualization that illustrates the relationship between movies, directors and actors interactively (Fig. 5b, c and e), and the other is 3D scientific visualization that projects solar system by reproducing space and planets (Fig. 5a, d and f). Each visualization scenario had comparable length, structure and complexity, and all visualized contents such as 3D objects, images and texts were controllable by presenter’s direct manipulation. Also, each scenario sufficiently reflected characteristics of HoloBox system and adequately used all three different roles of the presenter. The presenter needed to convey the following contents by using the presented visualized information. (i) After navigating the visualization space, select certain information, then show and explain the relevant information. (ii) Change the attributes (size, position, angle, etc.) of visual contents or provide information on various aspects. (iii) Compare or reorganize (or reconstruct) visualization elements to provide them in new forms. We suggested possible presenter’s interactions but did not force them. 6.1.4. Results and discussion According to the preliminary questionnaire, all participants’ presentation skills were high, and most of them were familiar with watching stereoscopic images. In addition, four out of six presenters had experience with half-mirror film. Specifically, two participants were familiar with using half-mirror films and two had prior experience viewing contents through half-mirror films at museums or performance stages. The other two had never experienced a half-mirror film before. Overall, five out of six presenter participants chose HoloBox as their preferred presentation method even though all six presenters indicated that HoloBox required more effort. Regarding the visual aspect of representation, participants mentioned that constructing immersive and interactive 3D visualization space by expressing information on the back or in front of the presenter continuously, and by using 3D physical space with a presenter inside, was impressive. P4 and P5 pointed out that ‘it is interesting to make the presenter be superimposed in 3D space by allowing him/her to act as visual materials. For example, after reproducing 3D universe or planets that are hard to see in the real world in HoloBox, the presenter could navigate the contents more realistically’. Several participants commented that as the depth value was added, the possibility of expressing the ranges, types and amount of information was expanded. In particular, P6 emphasized that ‘when presenting challenging information, rather than showing information in a sequential manner, it could help the presenter display related information in visualization space simultaneously’. P3 also mentioned that ‘depending on the situation, by displaying information in 2D or 3D form, the presenter could selectively use the visualization space surrounding him/her’. On the other hand, P1 and P2 stated that considering the characteristics of the half-mirror film in HoloBox, it would be better to avoid expressing complex information such as text directly on the presenter’s body or on physical objects for better readability because the lights projected on them hinder legibility. In addition, some participants commented that it would be important to determine what type of information would maximize the effectiveness of the system. In the visualization scenarios used in this interview, five of six participants preferred visual information including 3D objects and images rather than containing complex text in terms of expression and delivery. Second, concerning the presenter’s interaction facilities, most of the participants stated that the range of data expression and the information delivery process could be enriched by the interactive intervention of the presenter. They mentioned that they felt more connected physically and cognitively to the information they wanted to describe. P6 pointed out that like the Minority Report, he could manipulate the information and highlight it more intensively. Furthermore, he felt he could more closely relate to the audience because he was able to have a discussion with them while placing the visual content between the audience and himself. In addition, P3 stated that ‘it is important to show the prepared information well, but it is even more important to show visualization following the progress of the presentation. In HoloBox, when audience asked a question or additional explanation was needed, I could bring the information scattered in the 3D visualization space forward or reconstruct the displayed information in another form for the audience instantly.’ Furthermore, P5 mentioned that ‘I usually use physical information (e.g. real modules, real objects) to complement the existing digital information. Since the physical and digital information is shown together interactively in the augmented presentation, I felt that I could highlight and deliver the content more intensively.’ However, several participants mentioned that although the system provided visual feedback in real time, it is difficult to recognize the exact depth position of each piece of visualized information. In particular, they stated that when information at different depth positions overlapped in the same field of view, it was difficult to select them accurately. In addition, several participants pointed out that the interaction ability of the presenter who uses the HoloBox system is crucial. If the presenter’s showmanship is good, he/she can take advantage of the benefits of the system, but a passive speaker will be hesitant or unfamiliar. Some participants also commented on the perceived usability issue and said that it would take time for the presenter to become fully accustomed to the system. 6.2. Exploring the engagement from HoloBox We conducted a second user study to investigate how the audience reacts to data presentation through HoloBox. We examined the subjective level of the perceived engagement of the audience to explore different types of presentation. HoloBox was compared with the ‘detached form’ of the traditional presentation system, as mentioned earlier in Fig. 1. 6.2.1. Study design and participants We recruited 18 participants (nine females) to evaluate the audience experience. The ages ranged from 24 to 53 years old, and the average age was 32.4. All participants passed a stereopsis test beforehand. We conducted the feasibility study as a within-subjects design. After one presenter was thoroughly familiar with both systems and the visualized content, he gave a presentation on both systems to each audience member. We used visualization scenarios from the first user study: a 3D network visualization and a 3D universe visualization (Fig. 9). However, to avoid the visual complexity caused by the overlaid information in 3D space as mentioned in the first user study, we modified and simplified the composition of visualized information. In addition, traditional presentation conditions do not support reconstructing information or changing properties in real time, so we prepared visualization slides in advance. To measure the level of engagement, we surveyed the participants’ subjective ratings for each type. Engagement is an essential factor that indicates a user’s interest in putting effort into investigating and exploring visualization, and there are a number of ways to measure it (Lam et al., 2012; O’Brien and Toms, 2010; Saket et al., 2016). Among them, we referred to a questionnaire that identifies the audience’s engagement during multimedia presentations (Webster and Ho, 1997). This questionnaire classified the factors affecting the engagement of the listener into four types: challenge, feedback, control and variety when the speaker transmits information to the viewer. In addition, the questionnaire was designed to measure the engagement of the overall presentation flow that the viewer perceives during the presentation through attention focus, curiosity and intrinsic interest items. All questions were modified to fit our system while maintaining the intention (context) of each question. We excluded questions concerning individuals’ perceptions of their own control since the presenter rather than the audience controlled the presentation. The questionnaire was composed of the following seven questions. Q1 (Challenges): The presentation challenges me to see. Q2 (Feedback): The presentation provides direct/clear feedback. Q3 (Control): The presentation gives the presenter control over the delivery and direction of the visualization. Q4 (Variety): The presentation uses a variety of presentation formats and styles. Q5 (Attention focus): The presentation holds my attention. Q6 (Curiosity): The presentation excites my curiosity. Q7 (Interest): The presentation is intrinsically interesting. Each question was measured on a seven-point scale ranging from strongly disagree to strongly agree. FIGURE 9. View largeDownload slide InfoVis scenario for user study: traditional type (top) and HoloBox (bottom). FIGURE 9. View largeDownload slide InfoVis scenario for user study: traditional type (top) and HoloBox (bottom). 6.2.2. Procedure We first explained the goal of the feasibility study and the characteristics of each type of presentation. Before starting the user test, we showed the audience several visualization examples to familiarize them with the HoloBox. Then, the presenters, for 6 min each, presented two scenarios in two different styles. Each subject participated in each test set in the order of InfoVis and SciVis. We counterbalanced the order of the two types of presentation to avoid ordering effects. After that, the subjects were asked to complete a questionnaire on the two types of presentation they watched. At this time, to neutralize the inconvenience of wearing glasses in HoloBox, stereoscopic glasses were worn during the conventional presentation type as well. In addition, we asked the students to rate the two types of presentation, not the lecture content. After that, we showed the visualization scenes, which represented each presenter’s role, to the audience again. In addition, we explained the characteristics of the three roles. Then, we conducted interviews to investigate their thoughts about the proposed presenter’s roles and to derive feedback for the augmented presentation concept. The experiment took ~60 min for each participant. 6.2.3. Results and discussion Figure 10 shows the average subject responses from the audience participants for both types of presentation. We analyzed these subjective responses using a Friedman Chi-square test and found significant differences in the answers to all seven questions. FIGURE 10. View largeDownload slide Average subjective responses from the audience participants on 7-point Likert scale. 1 = strongly disagree and 7 = strongly agree. FIGURE 10. View largeDownload slide Average subjective responses from the audience participants on 7-point Likert scale. 1 = strongly disagree and 7 = strongly agree. As presented in Table 1, the audience indicated that HoloBox helped them gain a clearer understanding than the traditional presentation (Q1-challenges: x2(1, N = 18) = 12.25, P < 0.001), and they could receive more direct feedback from a HoloBox presentation (Q2-feedback: x2(1, N = 18) = 17.00, P < 0.001). Six participants mentioned that ‘the presenter gave immediate feedback by selecting visualized information directly and adding or modifying data at the request of the audience’. Q3 and Q4 showed the largest difference in response results between the two presentation types (Q3-control: x2(1, N = 18) = 13.235, P < 0.001, Q4-variety: x2(1, N = 18) = 18.00, P < 0.001). P9 and P15 stated that ‘having the various presentation materials for a single 3D visualization space helped us have greater context for understanding the visualization. Moreover, it was very impressive that the presenter could expand the scope of visualization by creating the visualization form according to the situational needs rather than displaying prearranged information’. In addition, P7 said, ‘I was able to observe the information in various aspects when 3D information was rotating just above the presenter’s hand, and when the presenter’s body became connected with the digital information, serving as a part of visualization.’ However, six participants commented that sometimes the presenter’s gestures seemed unnatural. In addition, when there was a mismatch between the presenter’s hand and his target visual aids, they became distracted from the presentation. TABLE 1. Frequency table of the responses from the audience participants on 7-point Likert scale. 1 = strongly disagree and 7 = strongly agree.     TABLE 1. Frequency table of the responses from the audience participants on 7-point Likert scale. 1 = strongly disagree and 7 = strongly agree.     Concerning the questions that investigated the overall engagement of the presentation, Q5(attention focus: x2(1, N = 18) = 9.00, P < 0.003), Q6(curiosity: x2(1, N = 18) = 17.00, P < 0.001), Q7(intrinsic interest: x2(1, N = 18) = 18.00, P < 0.001), results show that the audience participants were more engaged by the presentations done with HoloBox. Ten participants said they could stay more focused than they did during static traditional presentations. However, with regard to attention focus, participants mentioned, ‘Although the entire 3D visualization space is used to represent information, when many types of content are mixed or overlapped in the space, it interfered with my immersion into the information’. In particular, P10 pointed out that visualization techniques such as adjusting the brightness when physical information overlaps with digital content or lowering the transparency of the supplementary information can be applied to allow the viewer to be absorbed in the presentation more deeply. In addition, four participants commented that there might be differences in engagement depending on the purpose of the presentation or the type of content. Among the InfoVis and SciVis content used in the experiment, ~66.7% (12 out of 18) participants preferred SciVis. They added that HoloBox seems to be more specialized in showing complex 3D structures, simulations and objects that are difficult to see in the real world. Three audience participants preferred the traditional presentation system because the visualized information was cleaner and the presenter’s actions were clearer and more stable. Nevertheless, four participants said, ‘I would be eager to give a presentation using the HoloBox system even if it may require more effort’. Overall, 14 out of 18 participants chose HoloBox as their favored presentation method, and one participant said it is difficult to choose between the two. Although extra effort and attention were required for the audience to observe presentations through HoloBox, rich visual materials and the variety of presenter’s roles made them prefer augmented presentation. Nevertheless, three participants mentioned that the visual complexity and dynamics provided by HoloBox often made it difficult to focus on the visualization itself. In particular, as P10 commented previously, eight participants were confused when various pieces of information were shown in the 3D visualization space at the same time. Therefore, it is necessary to develop visualization methods to construct the HoloBox interface efficiently by considering the visual complexity and cognitive load of the audience. Furthermore, 10 participants said that observing the realistic and spatial visualization space of HoloBox was attractive enough to forget that they were wearing stereoscopic glasses. However, two participants said that wearing stereoscopic glasses disturbed their information perception, especially when viewing textual or detailed information. In addition, they mentioned that if they had to look at the text without the presenter’s additional explanations, it would have been more difficult to perceive. Thus, it is necessary to discuss appropriate data types that can reflect the characteristics of HoloBox when information is rendered in stereoscopic 3D. We discuss this issue in Section 7 in more detail. 6.2.4. Discussion of the proposed presenter roles Based on our user study, we analyzed how the presenter’s behavior affects the audience’s presentation perception, mainly focusing on the proposed three roles. In the case of a narrator, most of the audience stated that the presenter supplemented visualization by providing a connection between the various contents through a verbal explanation. Especially, explaining the visualization by placing content between the presenter and audience could further strengthen the connection between the two. However, in our current system, both the presenter and the audience need to wear stereoscopic glasses to view the 3D information. As mentioned earlier, nonverbal signals from the presenter, especially the eye contact with the audience and the presence of a hand are important factors during visualization delivery (Aziz, 1998; Krekhov et al., 2017; Mayer, 2002). The lack of sufficient interaction between the presenter and the audience due to the apparatus is a limitation of our work that needs to be overcome. In the case of a manipulator, the audience mentioned that the presenter’s hands and movements acted as visual cues enabling them to focus more on the information. In particular, seeing the visualization, especially the 3D content, in various forms and angles helped them gain cognitive benefits in data perception. However, several audience members commented that the perceived engagement might vary depending on the controlling ability of the presenter. During a feasibility study, the presenter’s inexperienced gestures seemed to distract the audience from observing the content. Furthermore, three participants mentioned that the quality of the visualization could differ depending on the scope of the presenter’s involvement because the visualization is not static but transformed dynamically during the presentation. Since the control range of the manipulator is the largest and most complicated among the proposed roles, it is crucial to provide presentation environments that allow the presenter to access and manipulate data easily. Additionally, although we provide three basic interaction methods in the current stage, further exploration is required to allow presenters to extend the range of their capability of representing data. In the case of an augmenter, most of the audience stated that utilizing familiar information such as parts of the body or physical objects helped them to better understand abstract or complex visualization. Nevertheless, some of them mentioned that it would be more effective to provide a physical metaphor that can be easily linked to the properties of the digital information. Several audience members pointed out that they felt confused when the additional physical information was less intuitive. While we utilized attributes such as shape and size of physical information in the current stage, it would be interesting to use physical characteristics we have not yet found. In the same vein, there are opportunities to incorporate different visualization techniques that reflect physical characteristics into virtual information. Referring to the ‘physical visualization’ may be one of the possible ways of approaching this issue (Jansen et al., 2015). Furthermore, because the augmenter presents virtual and physical information in a spatially linked manner, four participants mentioned that it would be useful in education or advertisements where digital information and supplemental physical information (e.g. advertised goods, real objects, etc.) are shown together. Additionally, they stated that the augmenter could further emphasize their features when visualizing, for instance, biological or sports data but less so for complex formulas or texts. Further exploration of visualization scenarios that reflect the nature of physical information is required. 7. DISCUSSION AND FUTURE WORK Owing to the powerful impact of presenters on visualization delivery, there has been an increasing demand to engage presentation methods that utilize the characteristics of the presenter. However, how presenters augment the process of delivering visualizations by ameliorating the connectivity between visualized images and themselves is a subject that remains unexplored. Most existing presentation methods focus on developing visualization and interaction techniques. However, further research is needed to approach this issue from a more integrated perspective. In this paper, we presented augmented presentation, in which the presenter shows and interacts with virtual information by occupying the same physical space as the visualization. Within the integrated visualization framework, we enable the presenter to have more opportunities to enhance the visualization by playing various roles. Through our feasibility study, we found that presenting information diversely with the presenter tightly connected to it is an important factor in enhancing the audience’s visualization viewing experience and data perception. In addition, presenters can expand their capacity in delivering messages by accessing visualization directly and manipulating visualized content in various ways. However, to realize our concept more clearly, there are remaining issues to consider, as described Section 7.1. 7.1. Richness vs. cognitive load of HoloBox In HoloBox, although visualized content becomes richer by expanding the visualization space, which contains the presenter, in 3D, the range of information the audience perceives is enlarged. Most of the audience mentioned that the action of the presenter was not cognitively demanding, but they sometimes felt confused when various pieces of information (e.g. complex nodes and text) were displayed simultaneously and overlapped with the presenter. Although adding visual diversity and context to enhance data representation is a good approach, reducing data loss and the audience’s cognitive load is another priority. In the current stage, our concept provides the presenter with enough 3D visualization space to place the information at a distance. However, to realize our intention more clearly, further investigation is required to create an effective visualization space by considering the audience’s cognitive load in data perception (Sweller, 2008). In particular, we need to develop visualization methods that do not hinder the audience’s learning or create misconceptions, especially when the subject is further subdivided into learners beyond the audience. To accomplish this, applying a ‘focus + context technique’ to our HoloBox visualization space is a possible approach (Card et al., 1999; Dursteler, 2002). For example, we can change the visual attributes (transparency, brightness, contrast, etc.) of content depending on the level of focus in the visualization delivery process. Thus, further investigation is required to balance the visualization design between data richness and cognitive efficiency. 7.2. Appropriate visualization types for HoloBox This initial research suggests that there might be suitable data types that maximize the effectiveness of our HoloBox system. HoloBox has been designed to support data presentation and delivery by interactively involving a human presenter as a part of the 3D visualization. Based on our concept design and the insights gained from our feasibility study, we suggest the appropriate data types for HoloBox visualization. First, augmented presentation would be better suited for information to be provided with storytelling rather than just showing the visualization itself. The HoloBox allows the presenter to stand near the visualization and to turn data insights into a narrative and communicate this story to the audience. Adding the presenter’s narrative and storytelling to the visualization could help the audience better understand and recall the information. In particular, we confirmed that a characteristic of HoloBox, which allows both the presenter and audience to face and talk to each other by placing the content between them, could work effectively as a communication medium for storytelling. Second, our concept would be more effective for visualization expressed in the 3D space rather than on a 2D screen. Through the expanded visualization space of our HoloBox system, the presenter can show complicated information that cannot be adequately or appropriately displayed on a 2D screen, more naturally. Additionally, it would be more useful for presenting inherently 3D content such as architectural and scientific data. In this case, the presenter’s manipulation showing various views and forms of 3D visualization could help the audience interpret structure and perceive spatial correlations of data easily. Moreover, 3D information such as medical, biological and sports data, which could include the physical movement of the presenters located in the 3D visualization space, would be appropriate. Third, rather than displaying only a piece of information, heterogeneous information, in which various types of visualization are presented together, would be better suited for HoloBox visualization. When presenting different views of the same data, the presenter can place various types of graphs (e.g. time-series, bar chart, scatter plot, etc.) simultaneously and combine them by fully using the 3D presentation space. Moreover, rather than displaying static visualization, the HoloBox would be suitable for showing information that changes according to the condition (e.g. time, values) through the presenter’s dynamic manipulation. Furthermore, offering context and other forms of enhancements with important data could help the audience to find relationships between data and underlying patterns more easily. 7.2.1. Challenges of the presenter in HoloBox Interestingly, most presenter participants chose HoloBox as their preferred presentation method even though it required more work. This does not mean that reducing the presenter’s burden inevitably leads to an optimal experience. Although the presenter needs to put in more effort when using HoloBox, given the richer interaction and high involvement in visualization of the system, the presenters were more immersed and satisfied. In fact, several presenter participants stated that they liked having dynamic control and believed that it helped them keep the audience more interested and engaged. However, there are remaining issues to be discussed regarding the challenges of the interactive intervention of the presenter. First, with regard to visual feedback, the current HoloBox prototype does not completely support spatial feedback for the presenter. The presenter can only perceive the visualized content as if he/she is looking into a mirror image through the 3D images floating in front of him/her. As P1 commented, it is crucial for the presenter to provide a virtual environment that sufficiently supports real time monitoring of visualization progress. Therefore, further developments in the hardware system and visual feedback method are required to allow the presenter to recognize realistic depth cues for better control within HoloBox. For example, we can provide additional sensors or project shadows of each visualized content on the presentation floor. Second, three participants mentioned that interactions through HoloBox were nice, but they were concerned that they had only gestures to use during presentations, which sometimes could occupy more than a few minutes. Similar issues were raised in the field of natural user interfaces and in reducing the cognitive and physical burden of the presenter, and these are important matters to consider (Nielsen et al., 2003). Likewise, in HoloBox, it is necessary to investigate effective interaction methods that can ease the burden on the presenter but increase expressivity. When the presenter is not directly involved in manipulating visualization, such as when performing basic functions (e.g. moving to the next/previous visual form), offering separate interfaces or devices may be useful. Third, since the 3D visualized content is located between the audience and the presenter, some presenters attempted to transfer the information to the direction of the audience. Similarly, during the user study, three audience participants wanted to have visual content around them and interact with it directly. As Sundén et al. (2014) mentioned, it is important to explore interaction techniques that increase the involvement of both the audience and the presenter during knowledge transfer and communication. Thus, it would be of interesting to investigate the possibility of data interchange between the audience and presenter, and its effect on visualization delivery. Likewise, further exploration is needed to encourage the presenter to deliver more than just the visualization shown on the screen, and to help both the presenter and audience stay more engaged. 7.2.2. Limitation of user studies Although we conducted a feasibility study to confirm the potential capacity of the presenter and the audience’s level of engagement for HoloBox in the early stage, a full-scale user study is required to verify the proposed concept and system. As an initial stage, although our user study was conducted for a short time, further evaluations of the audience’s attention and data perception when using HoloBox for a long presentation or continuing usage are required. In particular, it would be important to examine whether our concept can enhance the audience’s learning process and lead to better understanding and comprehension. In addition, a deeper user study with more presenter participants concerning the semantics, efficiency, attractiveness and stress of the proposed roles will help us better understand the feasibility of the HoloBox approach. Discussing the possibility of allowing the audience participants to become presenters might be a possible solution to this issue. At this time, it is essential to carefully design the experimental process and content so that participants can objectively evaluate our concept and system without being influenced by individual differences. In addition, our user study compared HoloBox only with the traditional presentation method that includes a presenter. Further comparisons to various types of presenters such as visually integrated, using gesture interaction, or a disembodied presenter will help us better identify the strengths and weaknesses of the HoloBox. In addition, our user study could not consider the novelty effect. The participants might have preferred HoloBox because it is different and new. In addition, the presenters spent more time practicing the HoloBox since they were already familiar with the traditional method. This can affect the quality of information delivery. Future research is needed to examine these factors. Regarding the system design, additional experiments are required to investigate the effects of system characteristics on the audience’s data perception. In particular, a rear screen used in our system has advantages in constructing a 3D visualization space, but it could make the audience’s data perception more complicated. It would be interesting to discuss the suitability of the existence of a back screen depending on the type and complexity of data. In addition, when dealing with system design problems including the above issues, it is necessary to approach a user-centered design by considering the user’s requirements, reaction, context of use and cognitive load rather than focusing on technical aspects. 8. CONCLUSION Rather than just displaying the content of a presentation, we presented and further developed a novel concept called augmented presentation. Augmented presentation enables a presenter to enhance the process of visualization delivery by providing context and other forms of enhancements. We summarize our contributions as follows. First, we further developed the concept of augmented presentation and focused mainly on discovering the possibilities of the presenters in visualization delivery by using an interaction framework for visualization. Second, we presented and improved a prototype system called HoloBox, and proposed interaction methods where the presenter offered interaction with the augmented presentation to support the three roles of the presenter. Third, we conducted two user studies one for the presenter and the other for the audience to evaluate the feasibility of HoloBox as a new presentation system. The results showed that most presenters favored HoloBox because they can integrate themselves with the visualized information more physically and cognitively even if extra effort is required to present with HoloBox. In addition, in terms of the audience, augmented presentation was preferred due to its potential to enhance the quality of the presented information and be engaged more through enriched visualization. We believe our study illustrates the potential of augmented presentation to support new forms of visualization and information delivery. We expect our concept to be useful in diverse fields such as exhibition, education, interactive presentations and performances, and medical and biological applications. At this time, if users of HoloBox configure the 3D visualization space by adjusting the diversity and context of the information while considering the audience’s cognitive load, the characteristics of augmented presentation can be utilized more efficiently. SUPPLEMENTARY MATERIAL Supplementary data is available at Interacting with Computers online. ACKNOWLEDGEMENTS We thank the anonymous reviewers for their constructive comments. We wish to pay gratitude towards all of the participants for valuable feedback in user studies. This research was partially supported by NRF and the BK21 Plus (Postgraduate Organization for Content Science) Framework. REFERENCES Ainsworth, S. ( 2014) 20—The Multiple Representation Principle in Multimedia Learning. In Mayer, R. (ed.), The Cambridge Handbook of Multimedia Learning . 464. Cambridge University Press, Cambridge. Google Scholar CrossRef Search ADS   Akeley, K., Watt, S.J., Girshick, A.R. and Banks, M.S. ( 2004) A stereo display prototype with multiple focal distances. ACM Trans. Graph. , 23, 804– 813. ACM. Google Scholar CrossRef Search ADS   Aziz, K. ( 1998) The key to perfect presentations. Ind. Comm. Train. , 30, 214– 217. Google Scholar CrossRef Search ADS   Bimber, O. and Raskar, R. ( 2005) Spatial Augmented Reality: Merging Real and Virtual Worlds . CRC press. Google Scholar CrossRef Search ADS   Buisine, S. and Martin, J.C. ( 2007) The effects of speech-gesture cooperation in animated agents’ behavior in multimedia presentations. Interact. Comput. , 19, 484– 493. Google Scholar CrossRef Search ADS   Card, S.K., Mackinlay, J.D. and Shneiderman, B. ( 1999) Readings in Information Visualization: Using Vision to Think . Morgan Kaufmann, Burlington, MA. Craig, R.J. and Amernic, J.H. ( 2006) PowerPoint presentation technology and the dynamics of teaching. Innov. Higher Educ. , 31, 147– 160. Google Scholar CrossRef Search ADS   Cruz-Neira, C., Sandin, D.J., DeFanti, T.A., Kenyon, R.V. and Hart, J.C. ( 1992) The CAVE: audio visual experience automatic virtual environment. Commun. ACM , 35, 64– 73. Google Scholar CrossRef Search ADS   Curtis, K., Jones, G.J. and Campbell, N. ( 2016). Speaker impact on audience comprehension for academic presentations. In Proc. 18th ACM Int. Conf. Multimodal Interaction, ACM, pp. 129–136. de Koning, B.B. and Tabbers, H.K. ( 2011) Facilitating understanding of movements in dynamic visualizations: An embodied perspective. Educ. Psychol. Rev. , 23, 501– 521. Google Scholar CrossRef Search ADS   Dünser, A., Billinghurst, M. and Mancero, G. ( 2008). Evaluating visual search performance with a multi-layer display. In Proc. 20th Australasian Conf. Computer-Human Interaction: Designing for Habitus and Habitat, ACM, pp. 307–310. Dursteler, J.C. ( 2002). The digital magazine of InfoVis. net. Edge, D., Savage, J. and Yatani, K. ( 2013). HyperSlides: dynamic presentation prototyping. In Proc. SIGCHI Conf. Human Factors in Computing Systems, ACM, pp. 671–680. Fiorella, L. and Mayer, R.E. ( 2016) Effects of observing the instructor draw diagrams on learning from multimedia messages. J. Educ. Psychol. , 108, 528. Google Scholar CrossRef Search ADS   Fourney, A., Terry, M. and Mann, R. ( 2010). Gesturing in the wild: understanding the effects and implications of gesturebased interaction for dynamic presentations. In Proc. 24th BCS Interaction Specialist Group Conf. British Computer Society, pp. 230– 240. Gale, S. ( 1990) Human aspects of interactive multimedia communication. Interact. Comput. , 2, 175– 189. Google Scholar CrossRef Search ADS   Geng, J. ( 2013) Three-dimensional display technologies. Adv. Opt. Photon. , 5, 456– 535. Google Scholar CrossRef Search ADS   Gingrich, O., Renaud, A. and Emets, E. ( 2013) KIMA: a holographic telepresence environment based on cymatic principles. Leonardo , 46, 332– 343. Google Scholar CrossRef Search ADS   Hachet, M., Bossavit, B., Cohe, A. and de la Riviere, J.B. ( 2011). Toucheo: multitouch and stereo combined in a seamless workspace. In Proc. 24th Annual ACM Symposium on User Interface Software and Technology, ACM, pp. 587–592. Haider, F., Cerrato, L., Campbell, N. and Luz, S. ( 2016). Presentation quality assessment using acoustic information and hand movements. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE Int. Conf. IEEE, pp. 2812– 2816. Hilliges, O., Kim, D., Izadi, S., Weiss, M. and Wilson, A. ( 2012). HoloDesk: direct 3d interactions with a situated seethrough display. In Proc. SIGCHI Conf. Human Factors in Computing Systems, ACM, pp. 2421–2430. Isaacs, E.A., Morris, T., Rodriguez, T.K. and Tang, J.C. ( 1995). A comparison of face-to-face and distributed presentations. In Proc. SIGCHI Conf. Human Factors in Computing Systems, ACM Press/Addison-Wesley Publishing Co., pp. 354–361. Jansen, Y., Dragicevic, P., Isenberg, P., Alexander, J., Karnik, A., Kildal, J. and Hornbæk, K. ( 2015). Opportunities and challenges for data physicalization. In Proc. 33rd Annual ACM Conf. Human Factors in Computing Systems, ACM, pp. 3227– 3236. Kim, M., Lee, J., Stuerzlinger, W. and Wohn, K. ( 2016). HoloStation: augmented visualization and presentation. In SIGGRAPH ASIA 2016 Symposium on Visualization, ACM, p. 12. Kim, M., Lee, J. and Whon, K. ( 2014). SPAROGRAM: the spatial augmented reality holographic display for 3D visualization and exhibition. In 3DVis (3DVis), 2014 IEEE VIS Int. Workshop on, IEEE, pp. 81–86. Krekhov, A., Emmerich, K., Babinski, M. and Krüger, J. ( 2017). Gestures from the point of view of an audience: towards anticipatable interaction of presenters with 3D content. In Proc of the 2017 CHI Conference on Human Factors in Computing Systems, ACM, pp. 5284– 5294. Lam, H., Bertini, E., Isenberg, P., Plaisant, C. and Carpendale, S. ( 2012) Empirical studies in information visualization: seven scenarios. IEEE Trans. Visual. Comput. Graph. , 18, 1520– 1536. Google Scholar CrossRef Search ADS   Lanir, J., Booth, K.S. and Findlater, L. ( 2008). Observing presenters’ use of visual aids to inform the design of classroom presentation software. In Proc. SIGCHI Conf. Human Factors in Computing Systems, ACM, pp. 695–704. Lee, C., DiVerdi, S. and Hollerer, T. ( 2009) Depth-fused 3D imagery on an immaterial display. IEEE Trans. Visual. Comput. Graph. , 15, 20– 33. Google Scholar CrossRef Search ADS   Liston, K., Fischer, M. and Kunz, J. ( 2000) Designing and evaluating visualization techniques for construction planning. Comput. Civil. Build. Eng. , 2, 1293– 1300. MacKenzie, K.J., Dickson, R.A. and Watt, S.J. ( 2012) Vergence and accommodation to multiple-image-plane stereoscopic displays: ‘real world’ responses with practical image-plane separations? J. Electron. Imag. , 21, 011002– 1. Google Scholar CrossRef Search ADS   Mayer, R.E. ( 2002) Multimedia learning. Psychol. Learn. Motivat. , 41, 85– 139. Google Scholar CrossRef Search ADS   Moreno, R. and Mayer, R.E. ( 1999) Cognitive principles of multimedia learning: the role of modality and contiguity. J. Educ. Psychol. , 91, 358. Google Scholar CrossRef Search ADS   Nielsen, M., Störring, M., Moeslund, T.B. and Granum, E. ( 2003). A procedure for developing intuitive and ergonomic gesture interfaces for HCI. In International Gesture Workshop. Springer Berlin Heidelberg, pp. 409–420. Noma, T., Badler, N.I. and Zhao, L. ( 2000). Design of a virtual human presenter. Ogi, T., Yamada, T., Yamamoto, K. and Hirose, M. ( 2001) Invisible Interface for the Immersive Virtual World. In Immersive Projection Technology and Virtual Environments 2001 . pp. 237– 246. Springer, Vienna. Google Scholar CrossRef Search ADS   Olwal, A., Lindfors, C., Gustafsson, J., Kjellberg, T. and Mattsson, L. ( 2005). ASTOR: an autostereoscopic optical see-through augmented reality system. In Fourth IEEE and ACM Int. Symposium on Mixed and Augmented Reality (ISMAR’05), IEEE, pp. 24–27. O’Brien, H.L. and Toms, E.G. ( 2010) The development and evaluation of a survey to measure user engagement. J. Am. Soc. Inf. Sci , 61, 50– 69. Google Scholar CrossRef Search ADS   Prema, V., Roberts, G. and Wuensche, B.C. ( 2006) 3D visualisation techniques for multi-layer display technology. IVCNZ , 6, 27– 29. Prendinger, H., Ma, C. and Ishizuka, M. ( 2007) Eye movements as indices for the utility of life-like interface agents: a pilot study. Interact. Comput. , 19, 281– 292. Google Scholar CrossRef Search ADS   Rakkolainen, I., HÃűllerer, T., DiVerdi, S. and Olwal, A. ( 2009) Mid-air display experiments to create novel user interfaces. Multimedia Tools Appl. , 44, 389– 405. Google Scholar CrossRef Search ADS   Rosling, H. ( 2010) The Joy of Stats . 9. BBC4, London. Saket, B., Endert, A. and Stasko, J. ( 2016). Beyond usability and performance: a review of user experience-focused evaluations in visualization. In Proc. Beyond Time and Errors on Novel Evaluation Methods for Visualization, ACM, pp. 133–142. Shneiderman, B. ( 1996). The eyes have it: a task by data type taxonomy for information visualizations. In Visual Languages, 1996. Proc., IEEE Symposium on, IEEE, 336–343. Steinmeyer, J. ( 1999). The Science behind the Ghost: A Brief History of Pepper’s Ghost. Sun, G. and Holliman, N. ( 2009). Evaluating methods for controlling depth perception in stereoscopic cinematography. In IS-T/SPIE Electronic Imaging International Society for Optics and Photonics, 72370I–72370I. Sundén, E., Bock, A., Jönsson, D., Ynnerman, A. and Ropinski, T. ( 2014). Interaction techniques as a communication channel when presenting 3D visualizations. In 3DVis (3DVis), 2014 IEEE VIS Int. Workshop on, IEEE, pp. 61–65. Sweller, J. ( 2008). Evolutionary bases of human cognitive architecture: implications for computing education. In Proc. 4th Int. workshop on computing education research, ACM, pp. 1–2. Van Welbergen, H., Nijholt, A., Reidsma, D. and Zwiers, J. ( 2005). Presenting in virtual worlds: Towards an architecture for a 3D presenter explaining 2D-presented information. In Int. Conf. Intelligent Technologies for Interactive Entertainment, Springer Berlin Heidelberg, pp. 203–212. Vuforia, S.D.K. ( 2013). Vuforia Developer Portal. Ward, M.O., Grinstein, G. and Keim, D. ( 2010) Interactive Data Visualization: Foundations, Techniques, and Applications . CRC Press, Boca Raton, Florida. Webster, J. and Ho, H. ( 1997) Audience engagement in multimedia presentations. ACM SIGMIS. Database , 28, 63– 77. Yi, J.S., ah Kang, Y., Stasko, J. and Jacko, J. ( 2007) Toward a deeper understanding of the role of interaction in information visualization. IEEE Trans. Visual. Comput. Graph. , 13, 1224– 1231. Google Scholar CrossRef Search ADS   Zeng, B., Wang, G. and Lin, X. ( 2012) A hand gesture based interactive presentation system utilizing heterogeneous cameras. Tsinghua Sci. Technol. , 17, 329– 336. Google Scholar CrossRef Search ADS   Author notes Editorial Board Member: Dr. Sharon Tettegah © The Author(s) 2018. Published by Oxford University Press on behalf of The British Computer Society. All rights reserved. For Permissions, please email: journals.permissions@oup.com This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/about_us/legal/notices)
TI - HoloBox: Augmented Visualization and Presentation with Spatially Integrated Presenter
JF - Interacting with Computers
DO - 10.1093/iwc/iwy007
DA - 2018-04-11
UR - https://www.deepdyve.com/lp/oxford-university-press/holobox-augmented-visualization-and-presentation-with-spatially-rAYdJMmdts
SP - 1
EP - 242
VL - Advance Article
IS - 3
DP - DeepDyve
ER -