Abstract
Digital technology offers numerous advantages, such as preserving the authenticity, replicating reality, and facilitating dissemination. It enables the preservation of intangible cultural heritage (ICH) in its original form and allows for the creation of comprehensive graphic, audio, and visual databases. Among these technologies, holographic technology holds promise for protecting ICH and promoting its dissemination. This paper focuses on interactive holographic technology and presents the design and implementation of a dynamic holographic display system that combines digital hologram (DH) and computer-generated hologram (CGH) to showcase 3D images consisting of both virtual and real objects. Real-time loading of DH into a spatial light modulator enables the optical reproduction of real objects, while the loading of two CGHs into other spatial light modulators facilitates the optical reproduction of virtual objects. Computational holography allows for the addition of virtual information, such as coordinate text, and the fusion of the three reconstructed images in space, resulting in an augmented reality experience and enhanced 3D display of real objects. An experimental setup employing three liquid crystal on silicon (LCOS) devices confirms the validity of the proposed method. Compared to other techniques, this approach demonstrates improved image signal-to-noise ratio, reduced alignment errors, and wider coverage of light traversal for laser 3D reconstruction images. The holographic technology presented in this paper enables the fusion display of real and virtual scenes and real-time two-way interaction between the audience and virtual images. This research holds significant practical value in promoting the effective dissemination and protection of ICH.
1. Introduction
As time progresses and technology and the economy continue to develop, most of the ICH is dispersed from its original cultural context, so the existence and transmission of ICH is becoming more and more dissipated. Because of their “tangible” and “intangible” characteristics, the diversity, complexity, and dynamism of the means of transmission, ICH is facing more difficulties in their preservation [1]. Today’s legal policies add to the legal rights and interests of those who preserve ICH and provide certain financial support, but this does not completely solve the problem of preserving and passing on ICH.
Traditional ICH artifacts are usually displayed in a static, simple, repetitive visual effect. Traditional displays are focused on the conservation and research of cultural heritage exhibits, but they do not allow visitors to get close to them, and they can only understand, recognize, and judge through visual and audio [2, 3]. Most of the visitors are not satisfied with this situation; they want to experience and interact with the space and resonate with the inner culture.
The advent of the digital era has broken the traditional viewing-based approach to visiting, and the application of digital media technology has entered an era of comprehensive innovation in the preservation and dissemination of intangible culture [4]. The use of digital display technology can realize the expansion of computer virtual space to real space. The use of digital media dynamic display to convey the information behind the cultural relics “alive” has become a form of intangible cultural communication [5], such as the use of audio-visual technology, touchscreen, digital sand table, film and television recording, interactive cinema, holographic projection, phantom imaging, and other digital media or virtual image technology and other high-tech means to vividly and intuitively reproduce the material and immaterial culture and cultural landscape and natural landscape [6]. These methods can effectively promote the narrative of the exhibition, realize the interaction between people and exhibits, and greatly enrich the visitor’s experience, making the exhibition have more viewing value.
The integration of technology, especially touchscreen and digital media, in modern society has created both opportunities and challenges across various sectors. Taherian Kalati and Kim delved into the effects of touchscreen technology on the cognitive development and learning processes of young children, finding a range of outcomes influenced by the implementation and context of this technology [7]. Their systematic review emphasized the nuances of touchscreen technology’s educational impact. Concurrently, Østerlie and Monteiro explored the ever-evolving nature of digital representations, terming them as “digital sand.” Their study focused on the fluidity and transformative nature of digital representations, emphasizing their nonstatic, ever-changing characteristics in the face of modern technological advancements [8]. On the psychological front, Twenge investigated the relationship between digital media use and well-being, unearthing potential negative associations between prolonged digital engagement and overall happiness [9]. This study underscores the need for balanced digital media consumption in maintaining psychological well-being. In an educational setting, Vidal-Hall et al. studied the beliefs of early childhood practitioners regarding the integration of digital media in classrooms. Their findings underscored a dichotomy: while practitioners acknowledged the potential benefits, they also voiced concerns about technology’s appropriateness in child-centered learning environments [10]. Venturing into the political domain, Govil and Baishya critically analyzed the intersection of digital social media with autocracy, particularly in the propagation of right-wing populist narratives. Their research sheds light on the instrumentalization of digital platforms for political gains and the consequent shaping of public discourse [11]. Lastly, in a more technical realm, Lennartz et al. conducted a comparative study on the qualitative and quantitative differences among various dual-energy CT scanners, particularly focusing on virtual unenhanced images. Their rigorous assessment offers valuable insights into the performance metrics of these medical imaging devices and their implications for patient diagnostics [12].
The important form of holographic technology is based on images, which can make the images more vivid and richer, allowing the viewer to get a more realistic experience and a more vivid visual experience. Holographic images are different from the previous 2D and 3D; although both are able to project images, holographic images are more three-dimensional and can make the viewer get a comprehensive viewing experience.
Holographic technology contains a wide variety of technologies, and digital holography is one of them. Digital holography can provide new ideas for the preservation of ICH and promote the preservation of ICH [13]. The main use of digital holography is to analyze the morphology of objects and to improve virtual images of ICH through digital construction. For cultural communication, its role will be more prominent and more advantageous.
Digital holography and computational holography can be used for holographic 3D display, respectively. Li et al. [14] reconstructed a full parallax holographic video with multiple SLMs (space light modulators) based on computational holography. Zhang et al. [15] used digital holography to achieve dynamic 3D image reconstruction. Blinder et al. [16] used DH to produce CGH by fusing numerically reproduced object light waves with virtual object light waves. This method achieves the reconstruction of static holographic reproduction images with large viewing angles and enhances the display of static objects. Park and Lee [17] discussed the great advantages of holographic optics in the virtual reality and augmented reality industries, which provides a new direction for the development of AR/VR technology in holographic optics.
In this paper, digital holography and computational holography are simultaneously input to a spatial light modulator for optical reproduction to achieve the fusion of the image of the actual 3D object and the reproduced image of the virtual object for the purpose of reality enhancement. Through experiments, the feasibility of the proposed technology to achieve the fusion of the reproduced image is confirmed, and a good optical reproduction image is obtained. This study is a useful attempt to apply digital holographic technology in reality enhancement, which has an important inspiration and reference value for the development of holographic 3D display applications on NRMs.
2. Related Works
2.1. Principle and Application of Holographic Projection Technology
Holographic projection technology is a kind of 3D technology, which refers to the use of interference and diffraction principles to record and reproduce the real three-dimensional image of the object [18]. With the continuous development of holographic projection and other related technologies, some scenarios that originally appeared only in science fiction movies have slowly come into our lives [19]. The most important feature of holographic projection is that the audience can see the three-dimensional image of the object without wearing any auxiliary equipment, thus improving the visual sense. Therefore, the holographic projection has a wide scope of development in any scenario where images need to be displayed, including education, merchandising, and visual arts.
Caggianese et al. explore the evolving role of information and communication technologies (ICT) [20]. The authors highlight the merger of tangible and digital cultural artifacts and emphasize the importance of interactivity for a comprehensive understanding of cultural heritage. They present an analysis of a holographic projection system that uses a gesture-based interface. Through both quantitative and qualitative user studies, the researchers delve into users’ preferences regarding interaction techniques in museum settings. Their empirical findings advocate for task-specific design patterns for touchless user interfaces when showcasing digital heritage content. This paper provides invaluable insights into the burgeoning field of virtual heritage applications and the intricacies of user interaction therein.
The principle of holographic imaging is to use the reflected light waves of an illuminated object to carry the information propagation of the object form, record the light waves carrying the information with a recording medium, and then realize the reproduction of the light waves through the principles of reflection and diffraction [21]. Essentially, the image is formed through the air or on a special stereoscopic lens. While the projection of a flat screen is only a two-dimensional surface object, holographic projection technology is a true 360° viewing with the naked eye. The imaging principle is shown in Figure 1.
Holographic projection technology uses three-dimensional laser scanning and digital technology to achieve the original 1 : 1 size of the real three-dimensional visualization of cultural relics. It uses three-dimensional virtual projection technology to hover the three-dimensional image in midair of the real scene, with a high-tech sense and high visual sense, which can create the effect of both fantasy and reality. For example, the Yunnan Museum of Ethnicity has chosen 360° holographic projection exhibits at the end of the Hall of Ethnic Instruments, projecting different ethnic instruments in all directions and from multiple angles. From viewing to experiencing, visitors can feel the real three-dimensional image to attract more visitors (see Figure 2).
Holographic projection was first proposed by physicist Dennis Gerber in 1947 [22], and after 1960, the appearance of laser made holographic projection technology enter a brand new stage. Since then, holographic projection technology entered a bottleneck period of relatively slow development due to the limitations of display devices and projection carriers. In 2002, Chad Dyne, a graduate student at the Massachusetts Institute of Technology (MIT), proposed air projection technology [23] (see Figure 3(a)). This technology draws air into the machine and reemits it, producing a 3D image with interactive features in the air stream. Around 2007, the Japanese company Science and Technology proposed the laser beam projection technology, in which a small blast in the air using a mixture of nitrogen and oxygen can form an ephemeral 3D image. Since then, the laser rotational projection technique [24] (see Figure 3(b)), which uses the principle of the visual transient to create a holographic effect by reflecting laser light from a high-speed rotating mirror, was proposed. The changing water curtain combined with laser animation projection at different angles to form a holographic three-dimensional effect of water curtain projection [25] (see Figure 3(c)) has also been fully developed and applied in the cultural tourism industry. With the development of commercial application of holographic technology, people’s visual enjoyment is gradually changing from traditional flat 2D to naked-eye 3D, and holographic projection technology has been widely used in various fields of science and life.
(a) Air projection
(b) Laser rotational projection
(c) Water curtain projection
In addition, in the application of holographic projection technology, sound can also be fully utilized as an aid to giving the audience a feeling of being in the story of cultural relics while visiting, bringing them a realistic and rich experience [26]. For example, the old street of Chongqing Bouncy Stone brings a new masterpiece for the national visitors, a large 3D live holographic projection show “Buddha’s Light as Illusion.” The performance uses the principle of interference and diffraction to record and reproduce the real three-dimensional images of objects and then combines original music and stone carvings as the backbone. The show is a magnificent legend of Buddha’s light, showing a pair of stone carvings with “life,” a very shocking visual and auditory enjoyment. The reverberating Sanskrit sounds in the exhibition hall make the audience feel as if they are entering the mysterious world in a dreamlike mood and realize the highest realm of Tibetan Tantric practice (see Figure 4). This breakthrough in the traditional physical display method has given a new lease of life to the nontraditional culture. Both the form and colour are rendered with striking realism and clarity, exuding a strong three-dimensional presence that is visually stunning. It genuinely allows visitors to transition from mere observation to a feeling of personal immersion.
2.2. Realization of Interactive Holography
Holographic projection technology can transform the flat display of public space on multimedia screens into a three-dimensional display, and at the same time, holographic projection technology is also interactive and can be used to manipulate the projected content through gesture commands.
Interactive holographic technology combines holographic technology with gesture control technology to achieve interaction between viewers and stereoscopic holograms as well as human-computer interaction, which is widely used in museum exhibit displays, merchandise exhibitions, immersive art exhibitions, and other related fields. Take museums as an example; the artifacts in museums are old, precious, and fragile and have high conservation requirements. Visitors can also only view these artifacts in fixed locations. Some of the artifacts cannot be moved to display in all directions due to their size or weight. Sometimes, due to exhibition space constraints or temporary restoration of venues, some cannot be visited by the audience in person. Therefore, it is necessary to introduce diversified technical means to increase the variety and number of artifacts that can be displayed and enhance the effect of artifact display. Some scholars applied spatial interaction technology to museum exhibitions by gesturing movements to grasp, swipe, zoom, and rotate holographic stereoscopic images in a contactless manner. For example, the three-sided holographic pyramid in Figure 5 allows users to manipulate digital content by using spatial interaction techniques in a noncontact manner. Visitors can visualize 3D objects from three different angles. The system is equipped with a noncontact user interface based on leap motion sensor. It tracks the user’s hand movements through infrared and camera technology and is able to capture the position, orientation, and movement of fingers and palms with great precision. Since direct sunlight is often present in museums, leap motion is the least disturbed by solar radiation of all short-range infrared sensors. This sensor (capable of tracking hands) and Kinect (capable of tracking the whole body) are the most widely used infrared hardware devices to track and process natural gestures used within the dissemination of cultural heritage assets. Applying interactive holography to museum exhibitions, visitors can use one-handed motions to grasp, swipe, zoom, and rotate objects in a noncontact manner. Figure 6 shows the interactive holographic rotation technique.
(a) Clutching
(b) Swipe
Figure 6(a) shows the clutching technique in the rotation task. In the clutching technique, to rotate the 3D object, the user has to arrange the virtual hand next to it, assume the grasp position, and, while maintaining this hand posture, move her/his hand in the direction in which she/he wants to rotate the hologram. To conclude the rotation, the user has to assume again the open hand posture. If the users want to carry out large rotations, they have to grab, move, and release several times. Figure 6(b) shows the swipe technique in the rotation task. In the swipe technique, the object rotates according to the user’s control hand position with respect to the sensor: when the hand is to the right (or left) of the sensor, the hologram continuously rotates to the right (or left) with a speed of rotation that changes according to the distance of the hand position from the center of the interaction area. To stop the rotation, the user has to move her/his control hand either toward the center or outside of the interaction area.
Interactive holographic technology uses infrared technology for body capture and voice control to achieve real-time two-way interaction between the audience and the virtual image. In this process, not only can we know whether the communication is clear and accurate but also can make the audience actively participate in it. Compared with the interactivity of cell phone touchscreen, holographic technology raises interactivity to a new level and brings more accurate and efficient cultural communication.
3. Methodology
Since DHs and computational holograms are obtained in different ways, the relevant parameters of DHs and computational holograms must be matched in order to make their reproduced images achieve the expected fusion effect.
3.1. Digital Holographic Recording Conditions and Their Parameters
It is well known that the DH can be recorded using a lensless Fourier transform holographic optical path to obtain the most information about the target. Figure 7 shows the schematic diagram of the lens-free Fourier transform holographic recording optical path, where the line connecting the center of the sample and the center of the recording device is the optical axis, i.e., -axis, and the point coordinates of the reference light source are and .
Let the complex amplitude reflectivity of the object be , and when the object wave propagates to the recording plane, the complex amplitude distribution can be expressed as
In Equation (1), denotes the Fourier transform operator. To simplify the problem, we only discuss the one-dimensional case and approximate the object as a plane. In the one-dimensional case, Equation (1) can be simplified as follows:
According to the characteristics of lensless Fourier transform holography, let the reference light be , and the transmission of the hologram obtained by interfering with is proportional to
Let the hologram be illuminated with illuminating light waves; then, the complex amplitude of the light waves emitted from the hologram can be expressed as three terms:
For the zero-level term light waves,
For the original image light wave,
For the conjugate image light wave:
The following is an illustration of the properties of the Fourier transform of the original image light wave in terms of its imaging. The light distribution in the plane at a distance from the hologram is examined.
Clearly, the core of the above result is the Fourier transform of . According to the Fourier transform property, the reproduced image is obtained as follows:
It can be seen that the reason why it is called lensless Fourier transform holography is that there is a Fourier transform relationship between the reproduced image of this recording method and its hologram. Similarly, the distribution of the zero-level term light wave and the conjugate image light wave on the image plane can be obtained as
From Equations (8) and (10), it can be seen that the 2 mutually conjugate images are separated on both sides of the zero level and are shifted by and , respectively, with respect to the point . The zero-level term reproduces the core part of the light wave (Equation (9)) as the Fourier transform of the original light wave mode squared. According to the convolution property of the Fourier transform, Equation (9) represents a light spot with a width twice the width of the object at the center of the image plane.
Let the maximum width of the object in thej-direction be . Obviously, to make the reproduced image not overlap with the zero-level spot, it is required that
Consider further the limitation of the object size by the resolution of the DH recording device itself. The hologram is essentially an interferogram of the recorded object light wave and the reference light wave. For a lens-free Fourier transform hologram, if the maximum width of the object is , the interval between adjacent interference fringes is
According to the sampling theorem, must be greater than two times the sampling interval of the hologram recording device, and let the pixel interval of the recording device be ; then, it is required that
This leads to
Assuming a limiting case, i.e., let the point source coordinates of the reference light be , then it would be
The hologram reproduces the object-image relationship as follows:
In Equation (16), is the magnification of the hologram; are the point coordinates of the reproduced light source. For a lensless Fourier transform hologram , the hologram is incident vertically with a point source whose coordinates are , , and . Equation (16) can be simplified as
Let the spatial light modulator pixel spacing be , and the hologram will be magnified by times after the DH is loaded into the spatial light modulator. Let and ; then,
Equation (18) shows that the reproduced image is magnified by a factor of and the center of the image is shifted in the -direction by .
3.2. Parameter Setting for Calculating Holography
To simplify the problem, the lens-free Fourier transform holographic optical path is still used to calculate the hologram. The sampling interval of the designed computational hologram is . If the spatial light modulator with pixel interval is used to reproduce, the magnification of the reproduced image is . Suppose the simulated object light wave distribution of the design calculation hologram is and the simulated reference light is ; in principle, the computerized hologram is the distribution of Equation (3). There are various methods to calculate the hologram, one of the most direct methods according to the following equation: where is the data of object light wave potential phase distribution and is the data of reference light potential phase distribution. Compared with Equation (3), Equation (21) replaces with a constant . In optical holograms, is unavoidable, which not only does not contribute to the reproduction of the image but also increases the computational bandwidth and introduces noise.
If and and normalize, then Equation (7) can be written as follows:
The second term in Equation (22) contributes all the information of the object light wave, and this encoding method is called a Burch-type computational hologram. When the hologram is computed using Equation (22), since there is no zero-level convolution scatter in its reproduction image plane, the relationship between the point source coordinates of the reference light and the object width is satisfied as long as
The 2 reconstructed conjugate images will not overlap. In order to satisfy the hologram sampling interval for the display of the spatial frequency of the interferometric fringe, the condition of Equation (14) must also be satisfied at this point. In the case where the point source coordinates of the reference light are chosen in the limit, that is, when , we have
Compared with Equation (15), the width of the object that can be recorded by computational holography can reach twice the width of digital holography. It should be noted that when the size of the object is just chosen as the maximum of Equations (15) and (24), there are exactly 2 pixels in one cycle of the hologram interference fringe. Due to the inhomogeneity of the target surface of the recording device and the influence of electronic noise, the quality of the hologram obtained at this time is difficult to guarantee. In practical applications, in order to ensure a better DH, the interval between adjacent interference fringes can be set to 4 pixels, and then, Equations (15) and (24) can be modified as follows:
4. Result Analysis and Discussion
In order to verify the proposed method, experiments are conducted for the fused reproduction of holograms, and their effectiveness and superiority are verified by comparing them with other comparison algorithms.
4.1. Optical Reproduction System
The schematic diagram of our optical reproduction system is shown in Figure 8, which can be divided into three channels for reproduction in total. We use a computer to output the hologram of each channel into three LCOS and use a 632 nm red laser, a 532 nm green laser, and a 473 nm blue laser to collimate and expand the beam to form a point source to illuminate the LCOS of each channel for optical reproduction. Each LCOS has a resolution of and a pixel spacing of 4.06 μm. The point light source is conjugated with the reference light for hologram generation, and the LCOS is illuminated through two lenses with a focal length of 30 cm. In the experiments conducted in this study, the monochromatic reproduction was achieved in the red laser channel, and the colour reproduction was achieved using the R, G, and B channels.
4.2. Reproduced Image Results and Analysis of the Fused Hologram
Before fusing the holograms, the DHs are first filtered to remove the zero-level term which has a great impact on the diffraction efficiency. Then, the DH is resampled to make the sampling interval of the DH also 4.06 μm. Finally, we superimpose and fuse the preprocessed DH with the computer-made hologram to improve the diffraction efficiency of the fused hologram. Figure 9 shows the optical reproduction results after preprocessing. Where Figure 9(a) is the target physical image, Figure 8(b) is the result of digital hologram reproduction after preprocessing, and Figure 9(c) is the result of hologram reproduction after fusion by preprocessing. It can be found that the diffraction efficiency of the hologram is greatly improved at this time, which verifies the feasibility of the proposed method.
(a) Target physical image
(b) Digital hologram reproduction after preprocessing
(c) Hologram reproduction after fusion by preprocessing
4.3. Performance Comparison and Analysis
To verify the effectiveness of the method in this study, the method of literature [27], the method of literature [28], and the method of literature [29] were compared with the method of this study to make the experimental results more illustrative. The signal-to-noise ratio and alignment error of the fused images of the four methods are compared, and the detailed comparison results are shown below.
The holographic 3D display method of this paper was used to compare and analyze the image effects with the other three comparison methods, respectively, and the results of the image signal-to-noise ratio comparison of the four methods are shown in Table 1.
Analyzing Table 1, it can be seen that the average signal-to-noise ratio of the output images of the comparison method does not exceed 35 dB, while the signal-to-noise ratio of the output images of this paper is higher, averaging around 50 dB, and the signal-to-noise ratio of the output images is more stable than that of the traditional method. It indicates that the image reconstructed with this method has better visibility and stronger visual expression.
The alignment error is compared with the other three comparison methods, and the comparison results are shown in Table 2.
In order to further quantitatively analyze the performance of this method, the fiber traversal coverage of the four methods for image 3D reconstruction was calculated, and the results are shown in Figure 10. It can be seen from Figure 10 that the proposed holographic 3D display method has more comprehensive light traversal coverage than the other comparison methods.
In summary, the method in this paper has a higher signal-to-noise ratio, lower alignment error, and more comprehensive light traversal coverage than other comparison methods. The reason is that the holographic 3D display method based on the combination of DH and CGH has greatly improved the accuracy of image feature point alignment and thus reduced the alignment error. Moreover, the image is preprocessed in advance by image filtering and other steps, thus improving the signal-to-noise ratio of the image, and thus, the application performance is better than other methods.
5. Conclusion
The use of dynamic digital media displays to convey the underlying information of cultural relics has become an innovative approach to preserving and disseminating intangible cultural heritage. These technologies effectively enhance the narrative of exhibitions, enable interaction between visitors and exhibits, and provide a more comprehensive understanding of exhibit information during visits, as well as richer displays and dissemination. In this study, a digital dynamic holographic 3D display system is developed by combining digital holography and computational holography. The hologram of the real object is recorded using digital holography, while the hologram of the virtual object is generated through computational holography. These two holograms are then input into a spatial light modulator. The experimental results demonstrate that the reproduced image, obtained through the diffraction of the spatial light modulator, combines virtual and actual information in space. This method achieves the fusion of digital hologram and computational hologram reproductions, enhancing the 3D display effect of real objects. The application of holographic technology proposed in this study effectively achieves the fusion of virtual and real scenes. This method addresses the limitations of traditional intangible cultural heritage display and dissemination, bridging the gap between people and intangible cultural heritage and promoting awareness of its protection.
Data Availability
The labeled datasets used to support the findings of this study are available from the corresponding author upon request.
Conflicts of Interest
The authors and contributors of this article independently conducted their research, guided and supported by their respective institutions, but without any financial assistance. No conflicts of interest exist with these institutions or any other entities. The research was exclusively supported through academic guidance, with no funding from external projects.
Acknowledgments
This article received academic guidance and assistance from the 2021 Academic Program for Top Talents in Anhui University of Business (Project ID: Smbjrc202105), without financial support and benefits.