From: hu-po
The field of 3D content creation and representation is rapidly evolving, with new techniques challenging established norms. Historically, mesh-based models dominated virtual environments, but recent advancements, particularly with Gaussian Splats, suggest a significant shift in how 3D objects and scenes will be represented and rendered.
Traditional 3D Representation: Meshes and Textures
For a very long time, and still predominantly today, 3D objects in video games and CGI are represented using a texture and mesh format [05:23:00]. This format consists of two primary components:
- Mesh: The 3D model itself, which is a collection of vertices forming polygons (often triangles) that define the object’s surface [05:35:00].
- Texture: A 2D image that is “wrapped” onto the mesh to provide color and surface detail [06:02:00].
This approach has been the standard due to its established pipelines in game engines and CGI tools [06:38:00]. However, it presents challenges: meshes involve complexities like handling polygon cleanliness and connectivity, and textures are inherently 2D, meaning they need re-creation if the mesh changes significantly [05:50:00], [05:08:50].
Neural Radiance Fields (NeRFs)
A relatively recent development before Gaussian Splats were Neural Radiance Fields (NeRFs), which gained significant popularity [06:46:00]. NeRFs represent 3D objects implicitly by training a neural network [07:03:00]. This network stores the 3D object within its weights and can generate the color of every pixel from any given camera position and viewpoint [07:23:00].
While exciting, NeRFs suffer from a key limitation: they are “per-sample” [11:57:00]. This means a new neural network must be trained for each object, leading to slow optimization times and limiting practical usage [12:15:00]. Furthermore, their implicit nature makes it difficult to directly understand or modify individual components of the 3D object [41:47:00].
Gaussian Splats: A New Paradigm
Gaussian Splats represent a new 3D representation technique that has garnered significant attention due to its efficiency and quality [04:47:00], [16:09:00]. They represent a 3D scene using millions of small particles, similar to a point cloud [08:08:00]. Each of these particles is a 3D Gaussian, defined by:
- A 3D position [08:14:00]
- An orientation [08:16:00]
- A scale [08:16:00]
- An opacity [08:18:00]
- A view-dependent color [08:18:00]
When rendering, these 3D Gaussians are projected onto the camera’s image plane, appearing as 2D Gaussians [38:23:00]. The final color at each pixel is calculated by summing the contributions of Gaussians within a specific tiled area, sorted by depth [39:12:00], [40:41:00].
Advantages of Gaussian Splats
Gaussian Splats offer several key advantages over previous methods:
- Speed: They are significantly faster than NeRFs, enabling real-time rendering [08:54:00], [16:17:00].
- Quality: They can achieve very clean and detailed 3D scene reconstructions [08:41:00], [16:21:00].
- Simplicity: They are conceptually simpler than NeRFs [04:59:00].
- Explicit Nature: Unlike NeRFs, Gaussian Splats explicitly store the 3D object’s properties. This allows for direct manipulation, such as deleting parts of a scene or copying and pasting objects, which is extremely difficult or impossible with implicit representations like NeRFs [42:23:00]. This compositionality is a crucial advantage for 3D content generation and editing [42:40:00].
The Future: A Splat-Dominated Virtual Landscape
The explicit and efficient nature of Gaussian Splats leads to a strong prediction about their future dominance:
- Superseding Meshes and Textures: Meshes and textures are predicted to “disappear” or be “superseded” by Gaussian Splats [14:17:00], [51:03:00]. This is because Splats offer better detail capture and avoid the “smoothness” issues seen in mesh-based methods [01:30:08].
- Direct Integration into Game Engines: Instead of converting Splats to meshes for use in game engines, the expectation is that game engines will directly support Splats, making entire virtual worlds composed of them [01:57:50], [01:58:20].
- Addressing the “Janus Problem”: Older text-to-3D methods suffered from the “Janus problem,” where objects generated from 2D diffusion models would have inconsistent views (e.g., multiple faces) due to the models’ bias towards frontal views [23:05:00], [01:34:57]. Gaussian Splatting approaches, especially when combined with 3D point cloud priors, mitigate this problem by better shaping the 3D geometry from the outset [01:09:50], [01:33:15].
- Evolution of 3D Printing: While currently 3D printing often relies on mesh formats like STL, the explicit nature of Gaussian Splats could lead to new additive manufacturing processes that directly utilize Splat data, potentially offering a more natural representation of 3D volume than infinitely thin surfaces [01:12:53], [01:25:01].
While current text-to-3D Gaussian Splatting methods might still have limitations like “baked lighting” (lack of view-dependent lighting effects) or blurriness in certain views, these are considered solvable problems with further advancements in 2D diffusion models and other techniques [01:43:35], [01:45:53]. The underlying architecture and explicit representation of Gaussian Splats position them as a strong candidate for the future standard of 3D content in virtual environments.