To see the other types of publications on this topic, follow the link: Gaussian splatting.
Journal articles on the topic 'Gaussian splatting'
Create a spot-on reference in APA, MLA, Chicago, Harvard, and other styles
Consult the top 50 journal articles for your research on the topic 'Gaussian splatting.'
Next to every source in the list of references, there is an 'Add to bibliography' button. Press on it, and we will generate automatically the bibliographic reference to the chosen work in the citation style you need: APA, MLA, Harvard, Chicago, Vancouver, etc.
You can also download the full text of the academic publication as pdf and read online its abstract whenever available in the metadata.
Browse journal articles on a wide variety of disciplines and organise your bibliography correctly.
1
Radl, Lukas, Michael Steiner, Mathias Parger, Alexander Weinrauch, Bernhard Kerbl, and Markus Steinberger. "StopThePop: Sorted Gaussian Splatting for View-Consistent Real-time Rendering." ACM Transactions on Graphics 43, no. 4 (2024): 1–17. http://dx.doi.org/10.1145/3658187.
Full textAbstract:
Gaussian Splatting has emerged as a prominent model for constructing 3D representations from images across diverse domains. However, the efficiency of the 3D Gaussian Splatting rendering pipeline relies on several simplifications. Notably, reducing Gaussian to 2D splats with a single viewspace depth introduces popping and blending artifacts during view rotation. Addressing this issue requires accurate per-pixel depth computation, yet a full per-pixel sort proves excessively costly compared to a global sort operation. In this paper, we present a novel hierarchical rasterization approach that systematically resorts and culls splats with minimal processing overhead. Our software rasterizer effectively eliminates popping artifacts and view inconsistencies, as demonstrated through both quantitative and qualitative measurements. Simultaneously, our method mitigates the potential for cheating view-dependent effects with popping, ensuring a more authentic representation. Despite the elimination of cheating, our approach achieves comparable quantitative results for test images, while increasing the consistency for novel view synthesis in motion. Due to its design, our hierarchical approach is only 4% slower on average than the original Gaussian Splatting. Notably, enforcing consistency enables a reduction in the number of Gaussians by approximately half with nearly identical quality and view-consistency. Consequently, rendering performance is nearly doubled, making our approach 1.6x faster than the original Gaussian Splatting, with a 50% reduction in memory requirements. Our renderer is publicly available at https://github.com/r4dl/StopThePop.
2
SMIRNOV, A. O. "Camera Pose Estimation Using a 3D Gaussian Splatting Radiance Field." Kibernetika i vyčislitelʹnaâ tehnika 216, no. 2(216) (2024): 15–25. http://dx.doi.org/10.15407/kvt216.02.015.
Full textAbstract:
Introduction. Accurate camera pose estimation is crucial for many applications ranging from robotics to virtual and augmented reality. The process of determining agents pose from a set of observations is called odometry. This work focuses on visual odometry, which utilizes only images from camera as the input data. The purpose of the paper is to demonstrate an approach for small-scale camera pose estimation using 3D Gaussians as the environment representation. Methods. Given the rise of neural volumetric representations for the environment reconstruction, this work relies on Gaussian Splatting algorithm for high-fidelity volumetric representation. Results. For a trained Gaussian Splatting model and the target image, unseen during training, we estimate its camera pose using differentiable rendering and gradient-based optimization methods. Gradients with respect to camera pose are computed directly from image-space per-pixel loss function via backpropagation. The choice of Gaussian Splatting as representation is particularly appealing because it allows for end-to-end estimation and removes several stages that are common for more classical algorithms. And differentiable rasterization as the image formation algorithm provides real-time performance which facilitates its use in real-world applications. Conclusions. This end-to-end approach greatly simplifies camera pose estimation, avoiding compounding errors that are common for multi-stage algorithms and provides a high-quality camera pose estimation. Keywords: radiance fields, scientific computing, odometry, slam, pose estimation, Gaussian Splatting, differentiable rendering.
3
Gao, Lin, Jie Yang, Bo-Tao Zhang, et al. "Real-time Large-scale Deformation of Gaussian Splatting." ACM Transactions on Graphics 43, no. 6 (2024): 1–17. http://dx.doi.org/10.1145/3687756.
Full textAbstract:
Neural implicit representations, including Neural Distance Fields and Neural Radiance Fields, have demonstrated significant capabilities for reconstructing surfaces with complicated geometry and topology, and generating novel views of a scene. Nevertheless, it is challenging for users to directly deform or manipulate these implicit representations with large deformations in a real-time fashion. Gaussian Splatting (GS) has recently become a promising method with explicit geometry for representing static scenes and facilitating high-quality and real-time synthesis of novel views. However, it cannot be easily deformed due to the use of discrete Gaussians and the lack of explicit topology. To address this, we develop a novel GS-based method (GaussianMesh) that enables interactive deformation. Our key idea is to design an innovative mesh-based GS representation, which is integrated into Gaussian learning and manipulation. 3D Gaussians are defined over an explicit mesh, and they are bound with each other: the rendering of 3D Gaussians guides the mesh face split for adaptive refinement, and the mesh face split directs the splitting of 3D Gaussians. Moreover, the explicit mesh constraints help regularize the Gaussian distribution, suppressing poor-quality Gaussians ( e.g. , misaligned Gaussians, long-narrow shaped Gaussians), thus enhancing visual quality and reducing artifacts during deformation. Based on this representation, we further introduce a large-scale Gaussian deformation technique to enable deformable GS, which alters the parameters of 3D Gaussians according to the manipulation of the associated mesh. Our method benefits from existing mesh deformation datasets for more realistic data-driven Gaussian deformation. Extensive experiments show that our approach achieves high-quality reconstruction and effective deformation, while maintaining the promising rendering results at a high frame rate (65 FPS on average on a single commodity GPU).
4
Jäger, Miriam, Theodor Kapler, Michael Feßenbecker, Felix Birkelbach, Markus Hillemann, and Boris Jutzi. "HoloGS: Instant Depth-based 3D Gaussian Splatting with Microsoft HoloLens 2." International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLVIII-2-2024 (June 11, 2024): 159–66. http://dx.doi.org/10.5194/isprs-archives-xlviii-2-2024-159-2024.
Full textAbstract:
Abstract. In the fields of photogrammetry, computer vision and computer graphics, the task of neural 3D scene reconstruction has led to the exploration of various techniques. Among these, 3D Gaussian Splatting stands out for its explicit representation of scenes using 3D Gaussians, making it appealing for tasks like 3D point cloud extraction and surface reconstruction. Motivated by its potential, we address the domain of 3D scene reconstruction, aiming to leverage the capabilities of the Microsoft HoloLens 2 for instant 3D Gaussian Splatting. We present HoloGS, a novel workflow utilizing HoloLens sensor data, which bypasses the need for pre-processing steps like Structure from Motion by instantly accessing the required input data i.e. the images, camera poses and the point cloud from depth sensing. We provide comprehensive investigations, including the training process and the rendering quality, assessed through the Peak Signal-to-Noise Ratio, and the geometric 3D accuracy of the densified point cloud from Gaussian centers, measured by Chamfer Distance. We evaluate our approach on two self-captured scenes: An outdoor scene of a cultural heritage statue and an indoor scene of a fine-structured plant. Our results show that the HoloLens data, including RGB images, corresponding camera poses, and depth sensing based point clouds to initialize the Gaussians, are suitable as input for 3D Gaussian Splatting.
5
Chen, Meida, Devashish Lal, Zifan Yu, et al. "Large-Scale 3D Terrain Reconstruction Using 3D Gaussian Splatting for Visualization and Simulation." International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLVIII-2-2024 (June 11, 2024): 49–54. http://dx.doi.org/10.5194/isprs-archives-xlviii-2-2024-49-2024.
Full textAbstract:
Abstract. The fusion of low-cost unmanned aerial systems (UAS) with advanced photogrammetric techniques has revolutionized 3D terrain reconstruction, enabling the automated creation of detailed models. Concurrently, the advent of 3D Gaussian Splatting has introduced a paradigm shift in 3D data representation, offering visually realistic renditions distinct from traditional polygon-based models. Our research builds upon this foundation, aiming to integrate Gaussian Splatting into interactive simulations for immersive virtual environments. We address challenges such as collision detection by adopting a hybrid approach, combining Gaussian Splatting with photogrammetry-derived meshes. Through comprehensive experimentation covering varying terrain sizes and Gaussian densities, we evaluate scalability, performance, and limitations. Our findings contribute to advancing the use of advanced computer graphics techniques for enhanced 3D terrain visualization and simulation.
6
Du, Yu, Zhisheng Zhang, Peng Zhang, Fuchun Sun, and Xiao Lv. "UDR-GS: Enhancing Underwater Dynamic Scene Reconstruction with Depth Regularization." Symmetry 16, no. 8 (2024): 1010. http://dx.doi.org/10.3390/sym16081010.
Full textAbstract:
Representing and rendering dynamic underwater scenes present significant challenges due to the medium’s inherent properties, which result in image blurring and information ambiguity. To overcome these challenges and accomplish real-time rendering of dynamic underwater environments while maintaining efficient training and storage, we propose Underwater Dynamic Scene Reconstruction Gaussian Splatting (UDR-GS), a method based on Gaussian Splatting. By leveraging prior information from a pre-trained depth estimation model and smoothness constraints between adjacent images, our approach uses the estimated depth as a geometric prior to aid in color-based optimization, significantly reducing artifacts and improving geometric accuracy. By integrating depth guidance into the Gaussian Splatting (GS) optimization process, we achieve more precise geometric estimations. To ensure higher stability, smoothness constraints are applied between adjacent images, maintaining consistent depth for neighboring 3D points in the absence of boundary conditions. The symmetry concept is inherently applied in our method by maintaining uniform depth and color information across multiple viewpoints, which enhances the reconstruction quality and visual coherence. Using 4D Gaussian Splatting (4DGS) as a baseline, our strategy demonstrates superior performance in both RGB novel view synthesis and 3D geometric reconstruction. On average, across multiple datasets, our method shows an improvement of approximately 1.41% in PSNR and a 0.75% increase in SSIM compared with the baseline 4DGS method, significantly enhancing the visual quality and geometric fidelity of dynamic underwater scenes.
7
Lyu, Xiaoyang, Yang-Tian Sun, Yi-Hua Huang, et al. "3DGSR: Implicit Surface Reconstruction with 3D Gaussian Splatting." ACM Transactions on Graphics 43, no. 6 (2024): 1–12. http://dx.doi.org/10.1145/3687952.
Full textAbstract:
In this paper, we present an implicit surface reconstruction method with 3D Gaussian Splatting (3DGS), namely 3DGSR, that allows for accurate 3D reconstruction with intricate details while inheriting the high efficiency and rendering quality of 3DGS. The key insight is to incorporate an implicit signed distance field (SDF) within 3D Gaussians for surface modeling, and to enable the alignment and joint optimization of both SDF and 3D Gaussians. To achieve this, we design coupling strategies that align and associate the SDF with 3D Gaussians, allowing for unified optimization and enforcing surface constraints on the 3D Gaussians. With alignment, optimizing the 3D Gaussians provides supervisory signals for SDF learning, enabling the reconstruction of intricate details. However, this only offers sparse supervisory signals to the SDF at locations occupied by Gaussians, which is insufficient for learning a continuous SDF. Then, to address this limitation, we incorporate volumetric rendering and align the rendered geometric attributes (depth, normal) with that derived from 3DGS. In sum, these two designs allow SDF and 3DGS to be aligned, jointly optimized, and mutually boosted. Our extensive experimental results demonstrate that our 3DGSR enables high-quality 3D surface reconstruction while preserving the efficiency and rendering quality of 3DGS. Besides, our method competes favorably with leading surface reconstruction techniques while offering a more efficient learning process and much better rendering qualities.
8
Smirnov, Anton О. "Dynamic map management for Gaussian Splatting SLAM." Control Systems and Computers, no. 2 (306) (July 2024): 3–9. http://dx.doi.org/10.15407/csc.2024.02.003.
Full textAbstract:
Map representation and management for Simultaneous Localization and Mapping (SLAM) systems is at the core of such algorithms. Being able to efficiently construct new KeyFrames (KF), remove redundant ones, constructing covisibility graphs has direct impact on the performance and accuracy of SLAM. In this work we outline the algorithm for maintaining dynamic map and its management for SLAM algorithm based on Gaussian Splatting as the environment representation. Gaussian Splatting allows for high-fidelity photorealistic environment reconstruction using differentiable rasterization and is able to perform in real-time making it a great candidate for map representation in SLAM. Its end-to-end nature and gradient-based optimization significantly simplifies map optimization, camera pose estimation and KeyFrame management.
9
Kerbl, Bernhard, Andreas Meuleman, Georgios Kopanas, Michael Wimmer, Alexandre Lanvin, and George Drettakis. "A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets." ACM Transactions on Graphics 43, no. 4 (2024): 1–15. http://dx.doi.org/10.1145/3658160.
Full textAbstract:
Novel view synthesis has seen major advances in recent years, with 3D Gaussian splatting offering an excellent level of visual quality, fast training and real-time rendering. However, the resources needed for training and rendering inevitably limit the size of the captured scenes that can be represented with good visual quality. We introduce a hierarchy of 3D Gaussians that preserves visual quality for very large scenes, while offering an efficient Level-of-Detail (LOD) solution for efficient rendering of distant content with effective level selection and smooth transitions between levels. We introduce a divide-and-conquer approach that allows us to train very large scenes in independent chunks. We consolidate the chunks into a hierarchy that can be optimized to further improve visual quality of Gaussians merged into intermediate nodes. Very large captures typically have sparse coverage of the scene, presenting many challenges to the original 3D Gaussian splatting training method; we adapt and regularize training to account for these issues. We present a complete solution, that enables real-time rendering of very large scenes and can adapt to available resources thanks to our LOD method. We show results for captured scenes with up to tens of thousands of images with a simple and affordable rig, covering trajectories of up to several kilometers and lasting up to one hour.
10
Dong, Zheng, Ke Xu, Yaoan Gao, Hujun Bao, Weiwei Xu, and Rynson W. H. Lau. "Gaussian Surfel Splatting for Live Human Performance Capture." ACM Transactions on Graphics 43, no. 6 (2024): 1–17. http://dx.doi.org/10.1145/3687993.
Full textAbstract:
High-quality real-time rendering using user-affordable capture rigs is an essential property of human performance capture systems for real-world applications. However, state-of-the-art performance capture methods may not yield satisfactory rendering results under a very sparse (e.g., four) capture setting. Specifically, neural radiance field (NeRF)-based methods and 3D Gaussian Splatting (3DGS)-based methods tend to produce local geometry errors for unseen performers, while occupancy field (PIFu)-based methods often produce unrealistic rendering results. In this paper, we propose a novel generalizable neural approach to reconstruct and render the performers from very sparse RGBD streams in high quality. The core of our method is a novel point-based generalizable human (PGH) representation conditioned on the pixel-aligned RGBD features. The PGH representation learns a surface implicit function for the regression of surface points and a Gaussian implicit function for parameterizing the radiance fields of the regressed surface points with 2D Gaussian surfels, and uses surfel splatting for fast rendering. We learn this hybrid human representation via two novel networks. First, we propose a novel point-regressing network (PRNet) with a depth-guided point cloud initialization (DPI) method to regress an accurate surface point cloud based on the denoised depth information. Second, we propose a novel neural blending-based surfel splatting network (SPNet) to render high-quality geometries and appearances in novel views based on the regressed surface points and high-resolution RGBD features of adjacent views. Our method produces free-view human performance videos of 1K resolution at 12 fps on average. Experiments on two benchmarks show that our method outperforms state-of-the-art human performance capture methods.
11
Papantonakis, Panagiotis, Georgios Kopanas, Bernhard Kerbl, Alexandre Lanvin, and George Drettakis. "Reducing the Memory Footprint of 3D Gaussian Splatting." Proceedings of the ACM on Computer Graphics and Interactive Techniques 7, no. 1 (2024): 1–17. http://dx.doi.org/10.1145/3651282.
Full textAbstract:
3D Gaussian splatting provides excellent visual quality for novel view synthesis, with fast training and realtime rendering; unfortunately, the memory requirements of this method for storing and transmission are unreasonably high. We first analyze the reasons for this, identifying three main areas where storage can be reduced: the number of 3D Gaussian primitives used to represent a scene, the number of coefficients for the spherical harmonics used to represent directional radiance, and the precision required to store Gaussian primitive attributes. We present a solution to each of these issues. First, we propose an efficient, resolution-aware primitive pruning approach, reducing the primitive count by half. Second, we introduce an adaptive adjustment method to choose the number of coefficients used to represent directional radiance for each Gaussian primitive, and finally a codebook-based quantization method, together with a half-float representation for further memory reduction. Taken together, these three components result in a x27 reduction in overall size on disk on the standard datasets we tested, along with a x1.7 speedup in rendering speed. We demonstrate our method on standard datasets and show how our solution results in significantly reduced download times when using the method on a mobile device (see Fig. 1).
12
Jiang, Yuheng, Zhehao Shen, Yu Hong, et al. "Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos." ACM Transactions on Graphics 43, no. 6 (2024): 1–15. http://dx.doi.org/10.1145/3687926.
Full textAbstract:
Volumetric video represents a transformative advancement in visual media, enabling users to freely navigate immersive virtual experiences and narrowing the gap between digital and real worlds. However, the need for extensive manual intervention to stabilize mesh sequences and the generation of excessively large assets in existing workflows impedes broader adoption. In this paper, we present a novel Gaussian-based approach, dubbed DualGS , for real-time and high-fidelity playback of complex human performance with excellent compression ratios. Our key idea in DualGS is to separately represent motion and appearance using the corresponding skin and joint Gaussians. Such an explicit disentanglement can significantly reduce motion redundancy and enhance temporal coherence. We begin by initializing the DualGS and anchoring skin Gaussians to joint Gaussians at the first frame. Subsequently, we employ a coarse-to-fine training strategy for frame-by-frame human performance modeling. It includes a coarse alignment phase for overall motion prediction as well as a fine-grained optimization for robust tracking and high-fidelity rendering. To integrate volumetric video seamlessly into VR environments, we efficiently compress motion using entropy encoding and appearance using codec compression coupled with a persistent codebook. Our approach achieves a compression ratio of up to 120 times, only requiring approximately 350KB of storage per frame. We demonstrate the efficacy of our representation through photo-realistic, free-view experiences on VR headsets, enabling users to immersively watch musicians in performance and feel the rhythm of the notes at the performers' fingertips. Project page: https://nowheretrix.github.io/DualGS/.
13
Guo, Chaoyang, Chunyan Gao, Yiyang Bai, and Xiaoling Lv. "RD-SLAM: Real-Time Dense SLAM Using Gaussian Splatting." Applied Sciences 14, no. 17 (2024): 7767. http://dx.doi.org/10.3390/app14177767.
Full textAbstract:
Simultaneous localization and mapping (SLAM) is fundamental for intelligent mobile units to perform diverse tasks. Recent work has shown that integrating neural rendering and SLAM showed promising results in photorealistic environment reconstruction. However, existing methods estimate pose by minimizing the error between rendered and input images, which is time-consuming and cannot be run in real-time, deviating from the original intention of SLAM. In this paper, we propose a dense RGB-D SLAM based on 3D Gaussian splatting (3DGS) while employing generalized iterative closest point (G-ICP) for pose estimation. We actively utilize 3D point cloud information to improve the tracking accuracy and operating speed of the system. At the same time, we propose a dual keyframe selection strategy and its corresponding densification method, which can effectively reconstruct new observation scenes and improve the quality of previously constructed maps. In addition, we introduce regularization loss to address the scale explosion of the 3D Gaussians and over-elongate in the camera viewing direction. Experiments on the Replica, TUM-RGBD, and ScanNet datasets show that our method achieves state-of-the-art tracking accuracy and runtime while being competitive in rendering quality.
14
Cui, Jiadi, Junming Cao, Fuqiang Zhao, et al. "LetsGo: Large-Scale Garage Modeling and Rendering via LiDAR-Assisted Gaussian Primitives." ACM Transactions on Graphics 43, no. 6 (2024): 1–18. http://dx.doi.org/10.1145/3687762.
Full textAbstract:
Large garages are ubiquitous yet intricate scenes that present unique challenges due to their monotonous colors, repetitive patterns, reflective surfaces, and transparent vehicle glass. Conventional Structure from Motion (SfM) methods for camera pose estimation and 3D reconstruction often fail in these environments due to poor correspondence construction. To address these challenges, we introduce LetsGo, a LiDAR-assisted Gaussian splatting framework for large-scale garage modeling and rendering. We develop a handheld scanner, Polar, equipped with IMU, LiDAR, and a fisheye camera, to facilitate accurate data acquisition. Using this Polar device, we present the GarageWorld dataset, consisting of eight expansive garage scenes with diverse geometric structures, which will be made publicly available for further research. Our approach demonstrates that LiDAR point clouds collected by the Polar device significantly enhance a suite of 3D Gaussian splatting algorithms for garage scene modeling and rendering. We introduce a novel depth regularizer that effectively eliminates floating artifacts in rendered images. Additionally, we propose a multi-resolution 3D Gaussian representation designed for Level-of-Detail (LOD) rendering. This includes adapted scaling factors for individual levels and a random-resolution-level training scheme to optimize the Gaussians across different resolutions. This representation enables efficient rendering of large-scale garage scenes on lightweight devices via a web-based renderer. Experimental results on our GarageWorld dataset, as well as on ScanNet++ and KITTI-360, demonstrate the superiority of our method in terms of rendering quality and resource efficiency.
15
Nguyen, Van Minh, Emma Sandidge, Trupti Mahendrakar, and Ryan T. White. "Characterizing Satellite Geometry via Accelerated 3D Gaussian Splatting." Aerospace 11, no. 3 (2024): 183. http://dx.doi.org/10.3390/aerospace11030183.
Full textAbstract:
The accelerating deployment of spacecraft in orbit has generated interest in on-orbit servicing (OOS), inspection of spacecraft, and active debris removal (ADR). Such missions require precise rendezvous and proximity operations in the vicinity of non-cooperative, possibly unknown, resident space objects. Safety concerns with manned missions and lag times with ground-based control necessitate complete autonomy. This requires robust characterization of the target’s geometry. In this article, we present an approach for mapping geometries of satellites on orbit based on 3D Gaussian splatting that can run on computing resources available on current spaceflight hardware. We demonstrate model training and 3D rendering performance on a hardware-in-the-loop satellite mock-up under several realistic lighting and motion conditions. Our model is shown to be capable of training on-board and rendering higher quality novel views of an unknown satellite nearly 2 orders of magnitude faster than previous NeRF-based algorithms. Such on-board capabilities are critical to enable downstream machine intelligence tasks necessary for autonomous guidance, navigation, and control tasks.
16
Yu, Zehao, Torsten Sattler, and Andreas Geiger. "Gaussian Opacity Fields: Efficient Adaptive Surface Reconstruction in Unbounded Scenes." ACM Transactions on Graphics 43, no. 6 (2024): 1–13. http://dx.doi.org/10.1145/3687937.
Full textAbstract:
Recently, 3D Gaussian Splatting (3DGS) has demonstrated impressive novel view synthesis results, while allowing the rendering of high-resolution images in real-time. However, leveraging 3D Gaussians for surface reconstruction poses significant challenges due to the explicit and disconnected nature of 3D Gaussians. In this work, we present Gaussian Opacity Fields (GOF), a novel approach for efficient, high-quality, and adaptive surface reconstruction in unbounded scenes. Our GOF is derived from ray-tracing-based volume rendering of 3D Gaussians, enabling direct geometry extraction from 3D Gaussians by identifying its levelset, without resorting to Poisson reconstruction or TSDF fusion as in previous work. We approximate the surface normal of Gaussians as the normal of the ray-Gaussian intersection plane, enabling the application of regularization that significantly enhances geometry. Furthermore, we develop an efficient geometry extraction method utilizing Marching Tetrahedra, where the tetrahedral grids are induced from 3D Gaussians and thus adapt to the scene's complexity. Our evaluations reveal that GOF surpasses existing 3DGS-based methods in surface reconstruction and novel view synthesis. Further, it compares favorably to or even outperforms, neural implicit methods in both quality and speed.
17
Yang, Chen, Sikuang Li, Jiemin Fang, et al. "GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting." ACM Transactions on Graphics 43, no. 6 (2024): 1–13. http://dx.doi.org/10.1145/3687759.
Full textAbstract:
Reconstructing and rendering 3D objects from highly sparse views is of critical importance for promoting applications of 3D vision techniques and improving user experience. However, images from sparse views only contain very limited 3D information, leading to two significant challenges: 1) Difficulty in building multi-view consistency as images for matching are too few; 2) Partially omitted or highly compressed object information as view coverage is insufficient. To tackle these challenges, we propose GaussianObject, a framework to represent and render the 3D object with Gaussian splatting that achieves high rendering quality with only 4 input images. We first introduce techniques of visual hull and floater elimination, which explicitly inject structure priors into the initial optimization process to help build multi-view consistency, yielding a coarse 3D Gaussian representation. Then we construct a Gaussian repair model based on diffusion models to supplement the omitted object information, where Gaussians are further refined. We design a self-generating strategy to obtain image pairs for training the repair model. We further design a COLMAP-free variant, where pre-given accurate camera poses are not required, which achieves competitive quality and facilitates wider applications. GaussianObject is evaluated on several challenging datasets, including MipNeRF360, OmniObject3D, OpenIllumination, and our-collected unposed images, achieving superior performance from only four views and significantly outperforming previous SOTA methods.
18
Abramov, Nikolai, Havana Lankegowda, Shunwei Liu, Luigi Barazzetti, Carlo Beltracchi, and Pierpaolo Ruttico. "Implementing Immersive Worlds for Metaverse-Based Participatory Design through Photogrammetry and Blockchain." ISPRS International Journal of Geo-Information 13, no. 6 (2024): 211. http://dx.doi.org/10.3390/ijgi13060211.
Full textAbstract:
This paper explores participatory design methods for the interconnection of digital recording techniques, like digital photogrammetry and Gaussian splatting, with emerging domains such as the metaverse and blockchain technology. The focus lies in community engagement and the economic growth of urban and rural areas through blockchain integration, utilizing low-cost digitalization methods to create Web3 environments mirroring real settlements. Through a case study of an Italian village, the potential of participatory design and community-led development strategies in revitalizing neglected areas are explored, and the use of low-cost drone-based photogrammetry and Gaussian splatting in digitization are compared, highlighting their advantages and drawbacks considering the aim of this work, i.e., the creation of an interactive metaverse space. Ultimately, the study underscores the transformative role of digital technologies in reshaping design processes and fostering community development through a workflow, stressing collaborative decision-making and blockchain-driven economy, manufacturing, and maintenance through self-ownership models and performance-based smart contracts.
19
Lian, Haojie, Kangle Liu, Ruochen Cao, Ziheng Fei, Xin Wen, and Leilei Chen. "Integration of 3D Gaussian Splatting and Neural Radiance Fields in Virtual Reality Fire Fighting." Remote Sensing 16, no. 13 (2024): 2448. http://dx.doi.org/10.3390/rs16132448.
Full textAbstract:
Neural radiance fields (NeRFs) and 3D Gaussian splatting have emerged as promising 3D reconstruction techniques recently. However, their application in virtual reality (VR), particularly in firefighting training, remains underexplored. We present an innovative VR firefighting simulation system based on 3D Gaussian Splatting technology. Leveraging these techniques, we successfully reconstruct realistic physical environments. By integrating the Unity3D game engine with head-mounted displays (HMDs), we created and presented immersive virtual fire scenes. Our system incorporates NeRF technology to generate highly realistic models of firefighting equipment. Users can freely navigate and interact with fire within the virtual fire scenarios, enhancing immersion and engagement. Moreover, by utilizing the Photon PUN2 networking framework, our system enables multi-user collaboration on firefighting tasks, improving training effectiveness and fostering teamwork and communication skills. Through experiments and surveys, it is demonstrated that the proposed VR framework enhances user experience and holds promises for improving the effectiveness of firefighting training.
20
Kerbl, Bernhard, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." ACM Transactions on Graphics 42, no. 4 (2023): 1–14. http://dx.doi.org/10.1145/3592433.
Full textAbstract:
Radiance Field methods have recently revolutionized novel-view synthesis of scenes captured with multiple photos or videos. However, achieving high visual quality still requires neural networks that are costly to train and render, while recent faster methods inevitably trade off speed for quality. For unbounded and complete scenes (rather than isolated objects) and 1080p resolution rendering, no current method can achieve real-time display rates. We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time (≥ 30 fps) novel-view synthesis at 1080p resolution. First, starting from sparse points produced during camera calibration, we represent the scene with 3D Gaussians that preserve desirable properties of continuous volumetric radiance fields for scene optimization while avoiding unnecessary computation in empty space; Second, we perform interleaved optimization/density control of the 3D Gaussians, notably optimizing anisotropic covariance to achieve an accurate representation of the scene; Third, we develop a fast visibility-aware rendering algorithm that supports anisotropic splatting and both accelerates training and allows realtime rendering. We demonstrate state-of-the-art visual quality and real-time rendering on several established datasets.
21
Cai, Zhenglong, Junxing Yang, Tianjiao Wang, He Huang, and Yue Guo. "3D Reconstruction of Buildings Based on 3D Gaussian Splatting." International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLVIII-4/W10-2024 (May 31, 2024): 37–43. http://dx.doi.org/10.5194/isprs-archives-xlviii-4-w10-2024-37-2024.
Full textAbstract:
Abstract. In the current era of urban construction, smart city management, and digital twinning, three-dimensional reconstruction of urban buildings is particularly important. Traditional methods have limitations in reconstructing complex geometric scenes, while new methods such as Nerf focus on using implicit MLP to represent the geometric space of the model, but suffer from slow training and rendering speeds. To address this issue, this paper proposes the use of 3D Gaussian scatter points for three-dimensional reconstruction of urban buildings, improving training speed and reconstruction quality through optimized and accelerated rendering algorithms. This method demonstrates high efficiency and editability, providing a new solution for urban building reconstruction.
22
Chen, Wenbo, and Ligang Liu. "Deblur-GS: 3D Gaussian Splatting from Camera Motion Blurred Images." Proceedings of the ACM on Computer Graphics and Interactive Techniques 7, no. 1 (2024): 1–15. http://dx.doi.org/10.1145/3651301.
Full textAbstract:
Novel view synthesis has undergone a revolution thanks to the radiance field method. The introduction of 3D Gaussian splatting (3DGS) has successfully addressed the issues of prolonged training times and slow rendering speeds associated with the Neural Radiance Field (NeRF), all while preserving the quality of reconstructions. However, 3DGS remains heavily reliant on the quality of input images and their initial camera pose initialization. In cases where input images are blurred, the reconstruction results suffer from blurriness and artifacts. In this paper, we propose the Deblur-GS method for reconstructing 3D Gaussian points to create a sharp radiance field from a camera motion blurred image set. We model the problem of motion blur as a joint optimization challenge involving camera trajectory estimation and time sampling. We cohesively optimize the parameters of the Gaussian points and the camera trajectory during the shutter time. Deblur-GS consistently achieves superior performance and rendering quality when compared to previous methods, as demonstrated in evaluations conducted on both synthetic and real datasets.
23
Haitz, Dennis, Max Hermann, Aglaja Solana Roth, Michael Weinmann, and Martin Weinmann. "The Potential of Neural Radiance Fields and 3D Gaussian Splatting for 3D Reconstruction from Aerial Imagery." ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences X-2-2024 (June 10, 2024): 97–104. http://dx.doi.org/10.5194/isprs-annals-x-2-2024-97-2024.
Full textAbstract:
Abstract. In this paper, we focus on investigating the potential of advanced Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting for 3D scene reconstruction from aerial imagery obtained via sensor platforms with an almost nadir-looking camera. Such a setting for image acquisition is convenient for capturing large-scale urban scenes, yet it poses particular challenges arising from imagery with large overlap, very short baselines, similar viewing direction and almost the same but large distance to the scene, and it therefore differs from the usual object-centric scene capture. We apply a traditional approach for image-based 3D reconstruction (COLMAP), a modern NeRF-based approach (Nerfacto) and a representative for the recently introduced 3D Gaussian Splatting approaches (Splatfacto), where the latter two are provided in the Nerfstudio framework. We analyze results achieved on the recently released UseGeo dataset both quantitatively and qualitatively. The achieved results reveal that the traditional COLMAP approach still outperforms Nerfacto and Splatfacto approaches for various scene characteristics, such as less-textured areas, areas with high vegetation, shadowed areas and areas observed from only very few views.
24
Basso, A., F. Condorelli, A. Giordano, S. Morena, and M. Perticarini. "EVOLUTION OF RENDERING BASED ON RADIANCE FIELDS. THE PALERMO CASE STUDY FOR A COMPARISON BETWEEN NERF AND GAUSSIAN SPLATTING." International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLVIII-2/W4-2024 (February 14, 2024): 57–64. http://dx.doi.org/10.5194/isprs-archives-xlviii-2-w4-2024-57-2024.
Full textAbstract:
Abstract. In recent years there has been a rapid diffusion of new digitization methodologies based on radiance fields and the implementation of new rendering processes and learning systems based on neural networks. The article focuses on these new tools and how they can be used for the knowledge and dissemination of Cultural Heritage. A case study is then described regarding the video acquisition of a noble chapel of the Cemetery of Santa Maria dei Rotoli in Palermo to promote knowledge of ‘fragile’ artefacts, exposed to the risk of radical transformation or degradation, and thus protecting their conservation. The research aims to compare the first results obtained through the NeRF and Gaussian Splatting methodology which constitute the current state of the art of this type of processing; both the source algorithms (Nerfacto and 3D Gaussian Splatting) and the Luma AI web app were used, and data management was studied using third-party software such as Blender 3D, Unreal Engine 5.0 and the playcanvas game engine. The results obtained with this case study are of particular interest, above all for the processing of data useful for the visualization of heritage starting from unconventional acquisitions.
25
Gao Jian, 高建, 陈林卓 Chen Linzhuo, 沈秋 Shen Qiu, 曹汛 Cao Xun та 姚遥 Yao Yao. "基于三维高斯溅射技术的可微分渲染研究进展". Laser & Optoelectronics Progress 61, № 16 (2024): 1611010. http://dx.doi.org/10.3788/lop241369.
Full text
26
Luo, Jie, Tianlun Huang, Weijun Wang, and Wei Feng. "A review of recent advances in 3D Gaussian Splatting for optimization and reconstruction." Image and Vision Computing 151 (November 2024): 105304. http://dx.doi.org/10.1016/j.imavis.2024.105304.
Full text
27
Moenne-Loccoz, Nicolas, Ashkan Mirzaei, Or Perel, et al. "3D Gaussian Ray Tracing: Fast Tracing of Particle Scenes." ACM Transactions on Graphics 43, no. 6 (2024): 1–19. http://dx.doi.org/10.1145/3687934.
Full textAbstract:
Particle-based representations of radiance fields such as 3D Gaussian Splatting have found great success for reconstructing and re-rendering of complex scenes. Most existing methods render particles via rasterization, projecting them to screen space tiles for processing in a sorted order. This work instead considers ray tracing the particles, building a bounding volume hierarchy and casting a ray for each pixel using high-performance GPU ray tracing hardware. To efficiently handle large numbers of semi-transparent particles, we describe a specialized rendering algorithm which encapsulates particles with bounding meshes to leverage fast ray-triangle intersections, and shades batches of intersections in depth-order. The benefits of ray tracing are well-known in computer graphics: processing incoherent rays for secondary lighting effects such as shadows and reflections, rendering from highly-distorted cameras common in robotics, stochastically sampling rays, and more. With our renderer, this flexibility comes at little cost compared to rasterization. Experiments demonstrate the speed and accuracy of our approach, as well as several applications in computer graphics and vision. We further propose related improvements to the basic Gaussian representation, including a simple use of generalized kernel functions which significantly reduces particle hit counts.
28
Hu, Qichun, Xiaolong Wei, Ronghui Cheng, et al. "Visual localization of robotic end effector via fusion of 3D Gaussian Splatting and heuristic optimization algorithm." Measurement 242 (January 2025): 116195. http://dx.doi.org/10.1016/j.measurement.2024.116195.
Full text
29
Gentet, Philippe, Matteo Coffin, Byung Hoon Choi, et al. "Outdoor Content Creation for Holographic Stereograms with iPhone." Applied Sciences 14, no. 14 (2024): 6306. http://dx.doi.org/10.3390/app14146306.
Full textAbstract:
Digital holographic stereograms have met expectations in various fields since their introduction. Traditionally, recording large outdoor physical models has required time-consuming and complex processes involving professional tools and technical expertise. This study, however, aims to streamline the process by utilizing simple equipment, such as an iPhone, basic tools, free phone applications, and free software. Four successful experiments were conducted and evaluated using the digital CHIMERA holographic stereogram-printing technique combined with photogrammetry, Gaussian splatting, light detection and ranging (LiDAR), and image interpolation. This approach records large-scale outdoor content more efficiently and effectively. The selected method allows the development and large-scale dissemination of realistic outdoor content holograms to the public. This study demonstrates the feasibility of creating ultra-realistic outdoor holograms using accessible tools and methods, offering potential applications in various fields such as art, education, and entertainment.
30
Henriques, Horácio, Alan de Oliveira, Eder Oliveira, Daniela Trevisan, and Esteban Clua. "Foveated Path Culling: A mixed path tracing and radiance field approach for optimizing rendering in XR Displays." Journal on Interactive Systems 15, no. 1 (2024): 576–90. http://dx.doi.org/10.5753/jis.2024.4352.
Full textAbstract:
Real-time effects achieved by path tracing are essential for creating highly accurate illumination effects in interactive environments. However, due to its computational complexity, it is essential to explore optimization techniques like Foveated Rendering when considering Head Mounted Displays. In this paper we combine traditional Foveated Rendering approaches with recent advancements in the field of radiance fields, extending a previous work and including recent advancements based on Gaussian Splatting. The present paper proposes the usage of mixing real time path tracing at the fovea region of an HMD while replacing the images at the peripheral by pre-computed radiance fields, inferred by neural networks or rendered in real time due to Gaussian splats. We name our approach as Foveated Path Culling (FPC) due to the process of culling raycasts, diminishing the workload by replacing most of the screen raytracing tasks by a less costly approach. FPC allowed us for better frame rates when compared to purely path tracing while rendering scenes in real time, increasing the frame rate speedup proportionally to the display resolution. Our work contributes to the development of rendering techniques for XR experiences that demand low latency, high resolution and high visual quality through global illumination effects.
31
Kang, Hee Won, and Jee Youn Kim. "Performance Analysis and Applicability Study of 3D Modeling Using 3D Gaussian Splatting in The 3D Model Production Process." CONTENTS PLUS 22, no. 6 (2024): 23–39. http://dx.doi.org/10.14728/kcp.2024.22.06.023.
Full text
32
Ress, Vincent, Wei Zhang, David Skuddis, Norbert Haala, and Uwe Soergel. "SLAM for Indoor Mapping of Wide Area Construction Environments." ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences X-2-2024 (June 10, 2024): 209–16. http://dx.doi.org/10.5194/isprs-annals-x-2-2024-209-2024.
Full textAbstract:
Abstract. Simultaneous localization and mapping (SLAM), i.e., the reconstruction of the environment represented by a (3D) map and the concurrent pose estimation, has made astonishing progress. Meanwhile, large scale applications aiming at the data collection in complex environments like factory halls or construction sites are becoming feasible. However, in contrast to small scale scenarios with building interiors separated to single rooms, shop floors or construction areas require measures at larger distances in potentially texture less areas under difficult illumination. Pose estimation is further aggravated since no GNSS measures are available as it is usual for such indoor applications. In our work, we realize data collection in a large factory hall by a robot system equipped with four stereo cameras as well as a 3D laser scanner. We apply our state-of-the-art LiDAR and visual SLAM approaches and discuss the respective pros and cons of the different sensor types for trajectory estimation and dense map generation in such an environment. Additionally, dense and accurate depth maps are generated by 3D Gaussian splatting, which we plan to use in the context of our project aiming on the automatic construction and site monitoring.
33
Zhuang, Jingyu, Di Kang, Yan-Pei Cao, Guanbin Li, Liang Lin, and Ying Shan. "TIP-Editor: An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts." ACM Transactions on Graphics 43, no. 4 (2024): 1–12. http://dx.doi.org/10.1145/3658205.
Full textAbstract:
Text-driven 3D scene editing has gained significant attention owing to its convenience and user-friendliness. However, existing methods still lack accurate control of the specified appearance and location of the editing result due to the inherent limitations of the text description. To this end, we propose a 3D scene editing framework, TIP-Editor, that accepts both text and image prompts and a 3D bounding box to specify the editing region. With the image prompt, users can conveniently specify the detailed appearance/style of the target content in complement to the text description, enabling accurate control of the appearance. Specifically, TIP-Editor employs a stepwise 2D personalization strategy to better learn the representation of the existing scene and the reference image, in which a localization loss is proposed to encourage correct object placement as specified by the bounding box. Additionally, TIP-Editor utilizes explicit and flexible 3D Gaussian splatting (GS) as the 3D representation to facilitate local editing while keeping the background unchanged. Extensive experiments have demonstrated that TIP-Editor conducts accurate editing following the text and image prompts in the specified bounding box region, consistently outperforming the baselines in editing quality, and the alignment to the prompts, qualitatively and quantitatively.
34
Cai, Youcheng, Runshi Li, and Ligang Liu. "MV2MV: Multi-View Image Translation via View-Consistent Diffusion Models." ACM Transactions on Graphics 43, no. 6 (2024): 1–12. http://dx.doi.org/10.1145/3687977.
Full textAbstract:
Image translation has various applications in computer graphics and computer vision, aiming to transfer images from one domain to another. Thanks to the excellent generation capability of diffusion models, recent single-view image translation methods achieve realistic results. However, directly applying diffusion models for multi-view image translation remains challenging for two major obstacles: the need for paired training data and the limited view consistency. To overcome the obstacles, we present a first unified multi-view image to multi-view image translation framework based on diffusion models, called MV2MV. Firstly, we propose a novel self-supervised training strategy that exploits the success of off-the-shelf single-view image translators and the 3D Gaussian Splatting (3DGS) technique to generate pseudo ground truths as supervisory signals, leading to enhanced consistency and fine details. Additionally, we propose a latent multi-view consistency block, which utilizes the latent-3DGS as the underlying 3D representation to facilitate information exchange across multi-view images and inject 3D prior into the diffusion model to enforce consistency. Finally, our approach simultaneously optimizes the diffusion model and 3DGS to achieve a better trade-off between consistency and realism. Extensive experiments across various translation tasks demonstrate that MV2MV outperforms task-specific specialists in both quantitative and qualitative.
35
Comte, F., A. Pamart, K. Réby, and L. De Luca. "STRATEGIES AND EXPERIMENTS FOR MASSIVE 3D DIGITALIZATION OF THE REMAINS AFTER THE NOTRE DAME DE PARIS’ FIRE." International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLVIII-2/W4-2024 (February 14, 2024): 127–34. http://dx.doi.org/10.5194/isprs-archives-xlviii-2-w4-2024-127-2024.
Full textAbstract:
Abstract. After the catastrophic fire at Notre Dame de Paris, a significant challenge was presented by the numerous lead-contaminated remnants. To address this, a detailed digitization strategy was devised and executed, tailored to the unique needs of this extensive and diverse corpus. This strategy involved the development and customization of both hardware and software tools, ensuring their effectiveness throughout the digitization process – from initial data acquisition to data dissemination.Central to our approach was the alignment of our methods with the distinct characteristics of each artifact, facilitating their effective preservation and future utility. Our strategy's adaptability was key, allowing us to incorporate advanced deep learning techniques into various aspects of our workflow. Notably, this included the implementation of the Segment Anything Model for automatic image segmentation, enhancing our image-based modeling capabilities. We also ventured into pioneering methods like 3D Gaussian Splatting and the exploration of radiance field methods for visualization.Moreover, the project has been mindful of data responsibility, aiming to make all digital data openly accessible beyond 2025. We have placed a strong emphasis on harmonizing and managing data, minimizing redundancies, and ensuring efficient storage, all while maintaining transparency about the limitations and errors in our methodologies. This holistic approach to digitization, balancing technological innovation with responsible data management, aims to preserve and make accessible the digital heritage of Notre Dame de Paris for future generations.
36
Gao, Ruicheng, and Yue Qi. "A Brief Review on Differentiable Rendering: Recent Advances and Challenges." Electronics 13, no. 17 (2024): 3546. http://dx.doi.org/10.3390/electronics13173546.
Full textAbstract:
Differentiable rendering techniques have received significant attention from both industry and academia for novel view synthesis or for reconstructing shapes and materials from one or multiple input photographs. These techniques are used to propagate gradients from image pixel colors back to scene parameters. The obtained gradients can then be used in various optimization algorithms to reconstruct the scene representation or can be further propagated into a neural network to learn the scene’s neural representations. In this work, we provide a brief taxonomy of existing popular differentiable rendering methods, categorizing them based on the primary rendering algorithms employed: physics-based differentiable rendering (PBDR), methods based on neural radiance fields (NeRFs), and methods based on 3D Gaussian splatting (3DGS). Since there are already several reviews for NeRF-based or 3DGS-based differentiable rendering methods but almost zero for physics-based differentiable rendering, we place our main focus on PBDR and, for completeness, only review several improvements made for NeRF and 3DGS in this survey. Specifically, we provide introductions to the theories behind all three categories of methods, a benchmark comparison of the performance of influential works across different aspects, and a summary of the current state and open research problems. With this survey, we seek to welcome new researchers to the field of differentiable rendering, offer a useful reference for key influential works, and inspire future research through our concluding section.
37
Yang, Zongyuan, Baolin Liu, Yingde Song, et al. "DirectL: Efficient Radiance Fields Rendering for 3D Light Field Displays." ACM Transactions on Graphics 43, no. 6 (2024): 1–19. http://dx.doi.org/10.1145/3687897.
Full textAbstract:
Autostereoscopic display technology, despite decades of development, has not achieved extensive application, primarily due to the daunting challenge of three-dimensional (3D) content creation for non-specialists. The emergence of Radiance Field as an innovative 3D representation has markedly revolutionized the domains of 3D reconstruction and generation, simplifying 3D content creation for common users and broadening the applicability of Light Field Displays (LFDs). However, the combination of these two technologies remains largely unexplored. The standard paradigm to create optimal content for parallax-based light field displays demands rendering at least 45 slightly shifted views preferably at high resolution per frame, a substantial hurdle for real-time rendering. We introduce DirectL, a novel rendering paradigm for Radiance Fields on autostereoscopic displays with lenticular lens. By thoroughly analyzing the interleaved mapping of spatial rays to screen sub-pixels, we accurately render only the light rays entering the human eye and propose subpixel repurposing to significantly reduce the pixel count required for rendering. Tailored for the two predominant radiance fields---Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting (3DGS), we propose corresponding optimized rendering pipelines that directly render the light field images instead of multi-view images, achieving state-of-the-art rendering speeds on autostereoscopic displays. Extensive experiments across various autostereoscopic displays and user visual perception assessments demonstrate that DirectL accelerates rendering by up to 40 times compared to the standard paradigm without sacrificing visual quality. Its rendering process-only modification allows seamless integration into subsequent radiance field tasks. Finally, we incorporate DirectL into diverse applications, showcasing the stunning visual experiences and the synergy between Light Field Displays and Radiance Fields, which reveals the immense potential for application prospects. DirectL Project Homepage: direct-l.github.io
38
Wu, Tong, Yu-Jie Yuan, Ling-Xiao Zhang, et al. "Recent advances in 3D Gaussian splatting." Computational Visual Media, July 8, 2024. http://dx.doi.org/10.1007/s41095-024-0436-y.
Full textAbstract:
AbstractThe emergence of 3D Gaussian splatting (3DGS) has greatly accelerated rendering in novel view synthesis. Unlike neural implicit representations like neural radiance fields (NeRFs) that represent a 3D scene with position and viewpoint-conditioned neural networks, 3D Gaussian splatting utilizes a set of Gaussian ellipsoids to model the scene so that efficient rendering can be accomplished by rasterizing Gaussian ellipsoids into images. Apart from fast rendering, the explicit representation of 3D Gaussian splatting also facilitates downstream tasks like dynamic reconstruction, geometry editing, and physical simulation. Considering the rapid changes and growing number of works in this field, we present a literature review of recent 3D Gaussian splatting methods, which can be roughly classified by functionality into 3D reconstruction, 3D editing, and other downstream applications. Traditional point-based rendering methods and the rendering formulation of 3D Gaussian splatting are also covered to aid understanding of this technique. This survey aims to help beginners to quickly get started in this field and to provide experienced researchers with a comprehensive overview, aiming to stimulate future development of the 3D Gaussian splatting representation.
39
Kovács, Áron Samuel, Pedro Hermosilla та Renata G. Raidou. "𝒢‐Style: Stylized Gaussian Splatting". Computer Graphics Forum, 8 листопада 2024. http://dx.doi.org/10.1111/cgf.15259.
Full textAbstract:
AbstractWe introduce 𝒢‐Style, a novel algorithm designed to transfer the style of an image onto a 3D scene represented using Gaussian Splatting. Gaussian Splatting is a powerful 3D representation for novel view synthesis, as—compared to other approaches based on Neural Radiance Fields—it provides fast scene renderings and user control over the scene. Recent pre‐prints have demonstrated that the style of Gaussian Splatting scenes can be modified using an image exemplar. However, since the scene geometry remains fixed during the stylization process, current solutions fall short of producing satisfactory results. Our algorithm aims to address these limitations by following a three‐step process: In a pre‐processing step, we remove undesirable Gaussians with large projection areas or highly elongated shapes. Subsequently, we combine several losses carefully designed to preserve different scales of the style in the image, while maintaining as much as possible the integrity of the original scene content. During the stylization process and following the original design of Gaussian Splatting, we split Gaussians where additional detail is necessary within our scene by tracking the gradient of the stylized color. Our experiments demonstrate that 𝒢‐Style generates high‐quality stylizations within just a few minutes, outperforming existing methods both qualitatively and quantitatively.
40
Ye, Sheng, Zhen‐Hui Dong, Yubin Hu, Yu‐Hui Wen, and Yong‐Jin Liu. "Gaussian in the Dark: Real‐Time View Synthesis From Inconsistent Dark Images Using Gaussian Splatting." Computer Graphics Forum, October 24, 2024. http://dx.doi.org/10.1111/cgf.15213.
Full textAbstract:
Abstract3D Gaussian Splatting has recently emerged as a powerful representation that can synthesize remarkable novel views using consistent multi‐view images as input. However, we notice that images captured in dark environments where the scenes are not fully illuminated can exhibit considerable brightness variations and multi‐view inconsistency, which poses great challenges to 3D Gaussian Splatting and severely degrades its performance. To tackle this problem, we propose Gaussian‐DK. Observing that inconsistencies are mainly caused by camera imaging, we represent a consistent radiance field of the physical world using a set of anisotropic 3D Gaussians, and design a camera response module to compensate for multi‐view inconsistencies. We also introduce a step‐based gradient scaling strategy to constrain Gaussians near the camera, which turn out to be floaters, from splitting and cloning. Experiments on our proposed benchmark dataset demonstrate that Gaussian‐DK produces high‐quality renderings without ghosting and floater artifacts and significantly outperforms existing methods. Furthermore, we can also synthesize light‐up images by controlling exposure levels that clearly show details in shadow areas.
41
Li, J., Z. Wen, L. Zhang, et al. "GS‐Octree: Octree‐based 3D Gaussian Splatting for Robust Object‐level 3D Reconstruction Under Strong Lighting." Computer Graphics Forum, October 24, 2024. http://dx.doi.org/10.1111/cgf.15206.
Full textAbstract:
AbstractThe 3D Gaussian Splatting technique has significantly advanced the construction of radiance fields from multi‐view images, enabling real‐time rendering. While point‐based rasterization effectively reduces computational demands for rendering, it often struggles to accurately reconstruct the geometry of the target object, especially under strong lighting conditions. Strong lighting can cause significant color variations on the object's surface when viewed from different directions, complicating the reconstruction process. To address this challenge, we introduce an approach that combines octree‐based implicit surface representations with Gaussian Splatting. Initially, it reconstructs a signed distance field (SDF) and a radiance field through volume rendering, encoding them in a low‐resolution octree. This initial SDF represents the coarse geometry of the target object. Subsequently, it introduces 3D Gaussians as additional degrees of freedom, which are guided by the initial SDF. In the third stage, the optimized Gaussians enhance the accuracy of the SDF, enabling the recovery of finer geometric details compared to the initial SDF. Finally, the refined SDF is used to further optimize the 3D Gaussians via splatting, eliminating those that contribute little to the visual appearance. Experimental results show that our method, which leverages the distribution of 3D Gaussians with SDFs, reconstructs more accurate geometry, particularly in images with specular highlights caused by strong lighting. The source code can be downloaded from https://github.com/LaoChui999/GS-Octree.
42
Zhang, Pingping, Xiangrui Liu, Meng Wang, Shiqi Wang, and Sam Kwong. "2D Gaussian Splatting for Image Compression." APSIPA Transactions on Signal and Information Processing 13, no. 6 (2024). http://dx.doi.org/10.1561/116.20240025.
Full text
43
Sun, Y., R. Tian, X. Han, X. Liu, Y. Zhang, and K. Xu. "GSEditPro: 3D Gaussian Splatting Editing with Attention‐based Progressive Localization." Computer Graphics Forum, November 4, 2024. http://dx.doi.org/10.1111/cgf.15215.
Full textAbstract:
AbstractWith the emergence of large‐scale Text‐to‐Image(T2I) models and implicit 3D representations like Neural Radiance Fields (NeRF), many text‐driven generative editing methods based on NeRF have appeared. However, the implicit encoding of geometric and textural information poses challenges in accurately locating and controlling objects during editing. Recently, significant advancements have been made in the editing methods of 3D Gaussian Splatting, a real‐time rendering technology that relies on explicit representation. However, these methods still suffer from issues including inaccurate localization and limited manipulation over editing. To tackle these challenges, we propose GSEditPro, a novel 3D scene editing framework which allows users to perform various creative and precise editing using text prompts only. Leveraging the explicit nature of the 3D Gaussian distribution, we introduce an attention‐based progressive localization module to add semantic labels to each Gaussian during rendering. This enables precise localization on editing areas by classifying Gaussians based on their relevance to the editing prompts derived from cross‐attention layers of the T2I model. Furthermore, we present an innovative editing optimization method based on 3D Gaussian Splatting, obtaining stable and refined editing results through the guidance of Score Distillation Sampling and pseudo ground truth. We prove the efficacy of our method through extensive experiments.
44
Xin, Zhe, Chengkai Dai, Ying Li, and Chenming Wu. "GauLoc: 3D Gaussian Splatting‐based Camera Relocalization." Computer Graphics Forum, November 7, 2024. http://dx.doi.org/10.1111/cgf.15256.
Full textAbstract:
Abstract3D Gaussian Splatting (3DGS) has emerged as a promising representation for scene reconstruction and novel view synthesis for its explicit representation and real‐time capabilities. This technique thus holds immense potential for use in mapping applications. Consequently, there is a growing need for an efficient and effective camera relocalization method to complement the advantages of 3DGS. This paper presents a camera relocalization method, namely GauLoc, in a scene represented by 3DGS. Unlike previous methods that rely on pose regression or photometric alignment, our proposed method leverages the differential rendering capability provided by 3DGS. The key insight of our work is the proposed implicit featuremetric alignment, which effectively optimizes the alignment between rendered keyframes and the query frames, and leverages the epipolar geometry to facilitate the convergence of camera poses conditioned explicit 3DGS representation. The proposed method significantly improves the relocalization accuracy even in complex scenarios with large initial camera rotation and translation deviations. Extensive experiments validate the effectiveness of our proposed method, showcasing its potential to be applied in many real‐world applications. Source code will be released at https://github.com/xinzhe11/GauLoc.
45
Fei, Ben, Jingyi Xu, Rui Zhang, Qingyuan Zhou, Weidong Yang, and Ying He. "3D Gaussian Splatting as New Era: A Survey." IEEE Transactions on Visualization and Computer Graphics, 2024, 1–20. http://dx.doi.org/10.1109/tvcg.2024.3397828.
Full text
46
Franke, Linus, Darius Rückert, Laura Fink, and Marc Stamminger. "TRIPS: Trilinear Point Splatting for Real‐Time Radiance Field Rendering." Computer Graphics Forum, April 30, 2024. http://dx.doi.org/10.1111/cgf.15012.
Full textAbstract:
AbstractPoint‐based radiance field rendering has demonstrated impressive results for novel view synthesis, offering a compelling blend of rendering quality and computational efficiency. However, also latest approaches in this domain are not without their shortcomings. 3D Gaussian Splatting [KKLD23] struggles when tasked with rendering highly detailed scenes, due to blurring and cloudy artifacts. On the other hand, ADOP [RFS22] can accommodate crisper images, but the neural reconstruction network decreases performance, it grapples with temporal instability and it is unable to effectively address large gaps in the point cloud. In this paper, we present TRIPS (Trilinear Point Splatting), an approach that combines ideas from both Gaussian Splatting and ADOP. The fundamental concept behind our novel technique involves rasterizing points into a screen‐space image pyramid, with the selection of the pyramid layer determined by the projected point size. This approach allows rendering arbitrarily large points using a single trilinear write. A lightweight neural network is then used to reconstruct a hole‐free image including detail beyond splat resolution. Importantly, our render pipeline is entirely differentiable, allowing for automatic optimization of both point sizes and positions.Our evaluation demonstrate that TRIPS surpasses existing state‐of‐the‐art methods in terms of rendering quality while maintaining a real‐time frame rate of 60 frames per second on readily available hardware. This performance extends to challenging scenarios, such as scenes featuring intricate geometry, expansive landscapes, and auto‐exposed footage. The project page is located at: https://lfranke.github.io/trips
47
Dalal, Anurag, Daniel Hagen, Kjell G. Robbersmyr, and Kristian Muri Knausgård. "Gaussian Splatting: 3D Reconstruction and Novel View Synthesis, a Review." IEEE Access, 2024, 1. http://dx.doi.org/10.1109/access.2024.3408318.
Full text
48
Zheng, Yuhang, Xiangyu Chen, Yupeng Zheng, et al. "GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Grasping." IEEE Robotics and Automation Letters, 2024, 1–8. http://dx.doi.org/10.1109/lra.2024.3432348.
Full text
49
Qu, Ziyuan, Omkar Vengurlekar, Mohamad Qadri, et al. "Z-Splat: Z-Axis Gaussian Splatting for Camera-Sonar Fusion." IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 1–12. http://dx.doi.org/10.1109/tpami.2024.3462290.
Full text
50
Guo, Zhiyang, Wengang Zhou, Li Li, Min Wang, and Houqiang Li. "Motion-aware 3D Gaussian Splatting for Efficient Dynamic Scene Reconstruction." IEEE Transactions on Circuits and Systems for Video Technology, 2024, 1. http://dx.doi.org/10.1109/tcsvt.2024.3502257.
Full text