3D Vision and Visualization
Editor's Note
In recent years, the concept of "Metaverse" has become popular again, and Facebook has even changed its name to "Meta" to embrace the metaverse. Metaverse is a concept created by the famous American science fiction writer Neal Stephenson in the novel "Snow Crash" published in 1992. Its core is to build a virtual digital world parallel to the real physical world.
But how to construct a virtual mirror of the real physical world is a key technical issue in realizing the metaverse. Vision is one of the most important ways for humans to perceive the world. With the continuous progress of 3D sensing technology, the rapid development of deep learning and the explosive growth of 3D visual data, the acquisition, analysis, understanding, expression, modeling, presentation, interaction and visualization of 3D visual data have become the core research issues in the construction of virtual images to the real world.
Our journal is honored to invite Professor Chen Baoquan, Executive Director of the Frontier Computing Research Center of Peking University, and Researcher Cheng Zhanglin of Shenzhen Institutes of Advanced Technology, Chinese Academy of Science to serve as guest editors of the special issue to share the research advancements of Chinese scholars on 3D vision and visualization.
Guest Editor
Baoquan Chen，Professor
Executive Director of the Center on Frontiers of Computing Studies, School of Electronics Engineering and Computer Science, Peking University.
His research interests generally lie in computer graphics, visualization, and human-computer interaction, focusing specifically on large-scale city modeling, simulation and visualization.
Zhanglin Cheng, Professor
Shenzhen Institutes of Advanced Technology, Chinese Academy of Science, Shenzhen, China
His research interests lie in computer graphics, visualization, and virtual reality.
Article List
2021, 10(6):3-19.
DOI: 10.12146/j.issn.2095-3135.20210630003
Abstract:
Since industry 4.0 was introduced in 2013, industries across the globe have been rushing towards the era of intelligent manufacturing. The advances in data sensing technologies have further helped the collection of massive industrial data, providing an excellent opportunity for innovations in industrial informatization. However, it remains a major challenge to analyze industrial data because of its large scale, high dimensionality, heterogeneity and complexity. Constantly changing application scenarios also lead to strict requirements in the flexibility of analyses, which demand placing domain experts in the analysis loop. Therefore, visualization has been widely applied to analyzing industrial data. This review article first summarizes the data types commonly used in the industrial scenarios based on the production stages and properties. Then, based on the data properties, this paper introduces visualization methods for the temporal, spatial and spatio-temporal types. Further, this paper overviews the applications of visual analytics in the industrial scenarios and discusses the integration of automated analysis methods in the visual analytics systems. Finally, this paper prospects the development of industrial data visual analytics and possible research directions for the future.
Since industry 4.0 was introduced in 2013, industries across the globe have been rushing towards the era of intelligent manufacturing. The advances in data sensing technologies have further helped the collection of massive industrial data, providing an excellent opportunity for innovations in industrial informatization. However, it remains a major challenge to analyze industrial data because of its large scale, high dimensionality, heterogeneity and complexity. Constantly changing application scenarios also lead to strict requirements in the flexibility of analyses, which demand placing domain experts in the analysis loop. Therefore, visualization has been widely applied to analyzing industrial data. This review article first summarizes the data types commonly used in the industrial scenarios based on the production stages and properties. Then, based on the data properties, this paper introduces visualization methods for the temporal, spatial and spatio-temporal types. Further, this paper overviews the applications of visual analytics in the industrial scenarios and discusses the integration of automated analysis methods in the visual analytics systems. Finally, this paper prospects the development of industrial data visual analytics and possible research directions for the future.
2021, 10(6):20-33.
DOI: 10.12146/j.issn.2095-3135.20210311001
Abstract:
As a common design type and visual art form, collage is widely used in poster design, logo production, and other applications. However, traditional manual process requires a lot of repetitive attempts and it is difficult for designers to achieve the optimal layout. When using computers to solve problems, there are many different ways. This paper summarizes different collage ideas which are roughly concluded as two ideas of top-down and bottom-up, and their algorithm routes are analyzed and compared. Finally, future development trend of the problem is presented.
As a common design type and visual art form, collage is widely used in poster design, logo production, and other applications. However, traditional manual process requires a lot of repetitive attempts and it is difficult for designers to achieve the optimal layout. When using computers to solve problems, there are many different ways. This paper summarizes different collage ideas which are roughly concluded as two ideas of top-down and bottom-up, and their algorithm routes are analyzed and compared. Finally, future development trend of the problem is presented.
2021, 10(6):34-57.
DOI: 10.12146/j.issn.2095-3135.20210618001
Abstract:
Single image based three dimensional structure reconstruction is a classical and important topic in computer vision domain. This survey focus on image acquisition, such as surface material, surface shape and the information loss of target, and classified the single image based 3D reconstruction methods into three categories, i.e. illumination model, geometric element distribution, and deep learning. By analyzing and concluding the advantage and disadvantage of different methods, potential research direction is also suggested.
Single image based three dimensional structure reconstruction is a classical and important topic in computer vision domain. This survey focus on image acquisition, such as surface material, surface shape and the information loss of target, and classified the single image based 3D reconstruction methods into three categories, i.e. illumination model, geometric element distribution, and deep learning. By analyzing and concluding the advantage and disadvantage of different methods, potential research direction is also suggested.
2021, 10(6):58-73.
DOI: 10.12146/j.issn.2095-3135.20210630001
Abstract:
Traditional interactive modeling of plants can get accurate plants models, but this process is extremely time-consuming and laborious. Automatic modeling methods like L-system-based methods can generate complex models quickly, but it is difficult for non-expert users because of its high learning cost and poor control over plant morphology. As an intuitive and efficient interaction means, the 2D sketch has a strong descriptive ability for plant morphology. In order to improve the plant modeling efficiency, a plant modeling method based on the 2D sketch is investigated. Firstly, the sketch drawn by the creator is preprocessed and analyzed to infer the creator’s intention. Then, based on the botany knowledge, the depth recovery algorithm of flowers, branches and leaves is proposed to recover the missing depth information in the 2D sketch. Finally, the target plant model is constructed according to the depth and plant characteristics. The proposed method is efficient and can support plant models with branch and leaf structures, such as flowers, potted plants, and trees.
Traditional interactive modeling of plants can get accurate plants models, but this process is extremely time-consuming and laborious. Automatic modeling methods like L-system-based methods can generate complex models quickly, but it is difficult for non-expert users because of its high learning cost and poor control over plant morphology. As an intuitive and efficient interaction means, the 2D sketch has a strong descriptive ability for plant morphology. In order to improve the plant modeling efficiency, a plant modeling method based on the 2D sketch is investigated. Firstly, the sketch drawn by the creator is preprocessed and analyzed to infer the creator’s intention. Then, based on the botany knowledge, the depth recovery algorithm of flowers, branches and leaves is proposed to recover the missing depth information in the 2D sketch. Finally, the target plant model is constructed according to the depth and plant characteristics. The proposed method is efficient and can support plant models with branch and leaf structures, such as flowers, potted plants, and trees.
2021, 10(6):74-85.
DOI: 10.12146/j.issn.2095-3135.20210630002
Abstract:
This paper presents a novel plane extraction algorithm based on 3D line cloud model. The algorithm first maps each line segment in the three-dimensional line cloud model to a point on the Gaussian sphere, and simplifies the plane extraction problem in the 3D space as a fitting problem of the plane through the sphere center. Then, the approximate uniform sampling is applied to the Gaussian sphere, and a plane passing through the sphere center is fitted. Finally, the plane can be extracted by separating parallel planes according to the intercept of the plane equation. The experimental results show that, in terms of the completeness of the plane extraction and the quality of plane extraction, the proposed algorithm has a significant improvement compared with classical plane extraction algorithms.
This paper presents a novel plane extraction algorithm based on 3D line cloud model. The algorithm first maps each line segment in the three-dimensional line cloud model to a point on the Gaussian sphere, and simplifies the plane extraction problem in the 3D space as a fitting problem of the plane through the sphere center. Then, the approximate uniform sampling is applied to the Gaussian sphere, and a plane passing through the sphere center is fitted. Finally, the plane can be extracted by separating parallel planes according to the intercept of the plane equation. The experimental results show that, in terms of the completeness of the plane extraction and the quality of plane extraction, the proposed algorithm has a significant improvement compared with classical plane extraction algorithms.
2022, 11(1):40-51.
DOI: 10.12146/j.issn.2095-3135.20210928001
Abstract:
The indoor scene generation task is an important research topic in recent years. It can not only provide a natural annotated indoor scene dataset for computer vision tasks to help better understand the scene, but also can be applied to many real scenes such as robot navigation. The diversity of indoor scene layouts makes scene generation a very challenging task. This paper reviews the recent research progress in the field of indoor scene generation, summarizes and classifies the generation algorithms in terms of scene input, scene generation method, scene representation, scene generation order, and scene context relationship. The three categories of the generation algorithms including sample-free generation method based on object relationship, sample-free generation method based on human activities, and sample-based object relationship based on object relationship are analyzed with advantages and disadvantages. In addition, this article also summarizes the limitations of the existing algorithms and points out the direction that can be explored in the field of indoor scene generation in the future.
The indoor scene generation task is an important research topic in recent years. It can not only provide a natural annotated indoor scene dataset for computer vision tasks to help better understand the scene, but also can be applied to many real scenes such as robot navigation. The diversity of indoor scene layouts makes scene generation a very challenging task. This paper reviews the recent research progress in the field of indoor scene generation, summarizes and classifies the generation algorithms in terms of scene input, scene generation method, scene representation, scene generation order, and scene context relationship. The three categories of the generation algorithms including sample-free generation method based on object relationship, sample-free generation method based on human activities, and sample-based object relationship based on object relationship are analyzed with advantages and disadvantages. In addition, this article also summarizes the limitations of the existing algorithms and points out the direction that can be explored in the field of indoor scene generation in the future.
2022, 11(1):52-65.
DOI: 10.12146/j.issn.2095-3135.20211001001
Abstract:
The information industry is developing rapidly and the mainstream digital media has produced an evolution from text to pictures to videos. How to quickly and effectively extract the key points of interest of characters in videos has become a hot topic in the field of Internet entertainment and big data analysis. However, existing methods for acquiring character information usually have significant limitations in obtaining information directly from the video interface. To address this problem, this paper proposes a novel “coarse to fine” intelligent face search framework based on feature hybrid clustering and key point detection. The real-time search of face data under big data is subdivided. First, the face detection algorithm based on multi-scale depth feature hybrid clustering uses the Softmax function to achieve data classification, and then uses the central loss function to form clustering centers that are modified by the regression of centroids to achieve coarse screening of faces. Then, based on the face key point detection algorithm, 68 individual face key feature points are extracted to generate standardized features that are easy to calculate and process to realize the fine search of faces under big data. This enables real-time and highly robust intelligent face search from Internet video data. Notably, this paper also constructs two film and television face datasets to provide big data analysis for subsequent related Internet industry and entertainment multimedia. System’s overall experimental results prove that this paper has a certain improvement in recognition accuracy and efficiency compared with existing mainstream face detection methods, including a 31.2% improvement in recognition efficiency and 3 times improvement in discrimination of false-positive samples, and the overall operation efficiency meets the standard and has certain practical value.
The information industry is developing rapidly and the mainstream digital media has produced an evolution from text to pictures to videos. How to quickly and effectively extract the key points of interest of characters in videos has become a hot topic in the field of Internet entertainment and big data analysis. However, existing methods for acquiring character information usually have significant limitations in obtaining information directly from the video interface. To address this problem, this paper proposes a novel “coarse to fine” intelligent face search framework based on feature hybrid clustering and key point detection. The real-time search of face data under big data is subdivided. First, the face detection algorithm based on multi-scale depth feature hybrid clustering uses the Softmax function to achieve data classification, and then uses the central loss function to form clustering centers that are modified by the regression of centroids to achieve coarse screening of faces. Then, based on the face key point detection algorithm, 68 individual face key feature points are extracted to generate standardized features that are easy to calculate and process to realize the fine search of faces under big data. This enables real-time and highly robust intelligent face search from Internet video data. Notably, this paper also constructs two film and television face datasets to provide big data analysis for subsequent related Internet industry and entertainment multimedia. System’s overall experimental results prove that this paper has a certain improvement in recognition accuracy and efficiency compared with existing mainstream face detection methods, including a 31.2% improvement in recognition efficiency and 3 times improvement in discrimination of false-positive samples, and the overall operation efficiency meets the standard and has certain practical value.
2022, 11(1):66-76.
DOI: 10.12146/j.issn.2095-3135.20210629001
Abstract:
There are certain limitations in the information display dimension of traditional interactive devices. Augmented reality technology can expand the interactive space and information dimension, but it is slightly insufficient in the overall information interaction and decision-making. This paper combined the traditional mobile interactive device iPad and the augmented reality device HoloLens: the overall characteristic information is displayed on the iPad, and the details are explored in HoloLens. Based on this idea, the paper designed and implemented a new interactive environment. A brand-new interactive environment, designed to achieve multi-level, multiplayer, multi-terminal spatial information display and interaction methods, takes the sensor log data of the venue as an example to demonstrate the advantages of the system in analyzing and processing problems.
There are certain limitations in the information display dimension of traditional interactive devices. Augmented reality technology can expand the interactive space and information dimension, but it is slightly insufficient in the overall information interaction and decision-making. This paper combined the traditional mobile interactive device iPad and the augmented reality device HoloLens: the overall characteristic information is displayed on the iPad, and the details are explored in HoloLens. Based on this idea, the paper designed and implemented a new interactive environment. A brand-new interactive environment, designed to achieve multi-level, multiplayer, multi-terminal spatial information display and interaction methods, takes the sensor log data of the venue as an example to demonstrate the advantages of the system in analyzing and processing problems.
2022, 11(1):77-87.
DOI: 10.12146/j.issn.2095-3135.20210701001
Abstract:
With the development of computer multimedia, visual analytic system goes beyond the traditional screen-mouse-keyboard. Digital sandbox projection system has attracted more and more attention in recent years. In the context of smart campus, this work proposes an interactive campus digital sandbox, that supports multi-user cooperation and multi-item interaction by constructing a visual analytic pipeline of sensor input, intelligent computing terminal, visual output driven by deep learning. Taking the heatmap simulation of campus activities as an application scenario, the feasibility of the system is verified.
With the development of computer multimedia, visual analytic system goes beyond the traditional screen-mouse-keyboard. Digital sandbox projection system has attracted more and more attention in recent years. In the context of smart campus, this work proposes an interactive campus digital sandbox, that supports multi-user cooperation and multi-item interaction by constructing a visual analytic pipeline of sensor input, intelligent computing terminal, visual output driven by deep learning. Taking the heatmap simulation of campus activities as an application scenario, the feasibility of the system is verified.
2022, 11(1):88-96.
DOI: 10.12146/j.issn.2095-3135.20210930001
Abstract:
In continuous mask projection 3D printing, due to the limited resin filling distance in a limited time, continuous forming is suitable for hollow structure or sheet structure. For the solid structures it is easy to fail in printing due to the insufficient resin filling in the process of printing. In addition, continuous printing based on ploydimethylsiloxane (PDMS) film soaked in lubricating fluid has limited printable height due to the loss of lubricating fluid in the printing process. To solve the above problems, a model guided 3D printing scheme combining layer-wise and continuous printing is proposed. The printing plane is formed by a certain thickness of fluorinated oil (also known as lubricating fluid) with a density greater than the resin and immiscible with the resin, which greatly reduces the adhesion between the printed object and the forming plane. Furthermore, experiments are designed to estimate the Maximum Filling Distance (MFD) and the Optimal Lifting Height (OLH) under the liquid-liquid interface. Based on MFD, combined with the Minimum-Maximum Distance (DisMax-Min) of model slices, the continuous printing and layer-wise printing are effectively connected. Based on OLH, with model analysis, the optimal lifting height of the platform could be estimated for layer-wise printing. The experimental results demonstrate that the proposed scheme can print the model height without restriction by lubrication fluid , and it can print the models of both solid and hollow structures. Compared with the traditional layer-wise printing scheme, the proposed method is more efficient.
In continuous mask projection 3D printing, due to the limited resin filling distance in a limited time, continuous forming is suitable for hollow structure or sheet structure. For the solid structures it is easy to fail in printing due to the insufficient resin filling in the process of printing. In addition, continuous printing based on ploydimethylsiloxane (PDMS) film soaked in lubricating fluid has limited printable height due to the loss of lubricating fluid in the printing process. To solve the above problems, a model guided 3D printing scheme combining layer-wise and continuous printing is proposed. The printing plane is formed by a certain thickness of fluorinated oil (also known as lubricating fluid) with a density greater than the resin and immiscible with the resin, which greatly reduces the adhesion between the printed object and the forming plane. Furthermore, experiments are designed to estimate the Maximum Filling Distance (MFD) and the Optimal Lifting Height (OLH) under the liquid-liquid interface. Based on MFD, combined with the Minimum-Maximum Distance (DisMax-Min) of model slices, the continuous printing and layer-wise printing are effectively connected. Based on OLH, with model analysis, the optimal lifting height of the platform could be estimated for layer-wise printing. The experimental results demonstrate that the proposed scheme can print the model height without restriction by lubrication fluid , and it can print the models of both solid and hollow structures. Compared with the traditional layer-wise printing scheme, the proposed method is more efficient.
