AI & Machine Learning for CGEdit

Generative AI and ML tools across image, video, 3D, texture, audio. AI motion capture is also under Animation & Rigging.

Machine Learning for CG

ML fundamentals and courses for CG artists.

• 3D Machine Learning. Resource repository for 3D machine learning.
• Awesome 3D Human. Curated list of papers and resources on 3D human research, including face, body, hand and motion.
• DINOv2. Meta self-supervised vision features. Strong general-purpose image embeddings. Useful as backbone for CG ML tasks.
• Generative AI Genius 2024. Free generative AI course material by Aishwarya Naresh Reganti.
• IEEE Transactions on Neural Networks and Learning Systems. Publishes technical articles on the theory, design, and applications of neural networks and related learning systems.
• Introduction to Generative AI Community Course. Free community course on generative AI fundamentals by iNeuron.
• Meta 3D Gen. Meta research pipeline for text-to-3D. Combines AssetGen geometry with TextureGen materials. Paper and results.
• PyTorch3D. Meta FAIR library for 3D deep learning. Differentiable rendering, mesh ops, point cloud utils on PyTorch.

Image Generation

text-to-image models and platforms.

Related:

• Canva AI Text-to-Image Generator Free tier. Canva's free built-in text-to-image generator.
• RenderNet Free tier. AI tool for generating images and videos with control over character design, composition, and style.
• Stable Diffusion Frivolous. Community response to the Stable Diffusion litigation.
• Word-As-Image for Semantic Typography. Semantically transforming fonts into illustrations.

Video Generation

AI video generation platforms and models.

Related:

• Deforum Notebook v0.5 (Stable Diffusion animation). Deforum v0.5 notebook for Stable Diffusion animation with math automation, perspective flips, prompt weights, and video masking.
• Deforum Stable Diffusion Animation (Artificial Selections). YouTube channel covering Deforum Stable Diffusion animation techniques.
• Emu Video (Meta). Meta's text-to-video research model. Factorized generation, image-then-video. Demos and paper.
• Stable Diffusion KLMC2 Animation. Colab notebook by @RiversHaveWings generating animation from scripted prompts via KLMC2 discretization of underdamped Langevin dynamics.
• Stable Diffusion KLMC2 Animation (forked). Colab notebook for Stable Diffusion animation, forked by @DigThatData.

3D Generation

AI tools for generating 3D models from text or images.

Related:

• NativeBlend CLI. AI CLI to generate and edit low-poly 3D game assets in Blender from text prompts.
• Pixal3D. Single-image to 3D asset generation via pixel-feature back-projection. Outputs geometry and PBR textures.
• Point-E (OpenAI). OpenAI point-cloud diffusion for text and image to 3D synthesis.
• Threestudio. A unified framework for 3D content generation.
• Transforming 2D Images into 3D with AdaMPI. Guide to using the AdaMPI model to create 3D photos from 2D images.

Texture & Material Generation

AI-generated PBR textures and materials.

Related:

• StableGen. Blender add-on that textures scenes by projecting multi-view diffusion images through a ComfyUI backend.
  See also: Software Reference → AI Image & Texture Generation Software

ComfyUI Ecosystem

Node-based AI generation. ComfyUI and its ecosystem.

Related:

• Awesome ComfyUI. Curated custom nodes collection.
• Comflowy. ComfyUI tutorials for ControlNet, SDXL, FLUX workflows.
• Comfy UI Manager. Install and manage custom nodes and dependencies.
• ComfyUI Registry. Community workflow and subgraph sharing.
• ControlNet SD3.5. Blur/Canny/Depth models, free commercial use.

AI-Assisted CG Tools

AI tools that augment traditional CG workflows.

Related:

• AI Game Developer. Unity Editor and Unity Runtime AI integration. Unit Test, Coding, C# Roslyn, Reflection, Assets. Helps to create games with AI. And helps to run AI logic during gameplay.
• AI Render (Stable Diffusion in Blender). Blender addon that renders an AI image from a text prompt and your scene via Stable Diffusion.
• ClipDrop: Image Upscaler Free tier. AI image upscaler and enhancer.
• Comfy UI Blender AI. Run ComfyUI workflows inside Blender.
• ControlNet. Depth/pose/edge control for precision CG workflows.
• Coplay Free tier. AI assistant for Unity that helps build and edit game projects from inside the editor.
• CorridorKey. Corridor Crew's open keying tool. AI-driven chroma key for VFX work.
• CoTracker. Meta FAIR point tracker for video. Tracks arbitrary points jointly across frames. Useful for matchmove and roto prep.
• DEVA (Tracking Anything). Decoupled video segmentation. Long-form, multi-object roto with text or click prompts.
• Dream Textures (Blender). Generate textures in Blender via Stable Diffusion, locally.
• Genetic Drawing. Python library that generates a stylized rendering from an image.
• GPT 4. Prompt engineering techniques for GPT-4, including tips, applications, limitations, and additional reading materials.
• MCP for Blender. MCP server that lets Claude and other LLM clients control Blender: build scenes, edit objects, run Python.
  See also: Software Reference → AI-Assisted CG Software
• RIFE for Nuke. Real-Time Intermediate Flow Estimation for video frame interpolation (ML framerate upscaling) for Nuke.
• Robust Video Matting (RVM). Real-time human video matting. Strong alpha mattes with no greenscreen.
• Rotobot Paid. Paid AI roto plugin for Nuke and After Effects. Auto-mattes people and objects.
• SAM 2 (Segment Anything Model 2). Meta's promptable segmentation for images and video. Masks objects across frames from a single click. Useful for rotoscope and matte work.
• Sammie Roto 2. SAM-based rotoscope app. Click-to-mask, propagates across frames.
• SolidUI. AI-generated visualization prototyping and editing platform, support 2D, 3D models, combined with LLM(Large Language Model) for quick editing.
• Track Anything. SAM plus XMem for interactive video object tracking and segmentation. Useful for roto and matte propagation.
• Wonder Studio (Autodesk Flow Studio) Free tier. AI character replacement with auto roto, body tracking, and re-lighting from a single video. Now part of Autodesk Flow Studio.

AI Audio & Music

AI music generation, voice, TTS, and SFX.

Related:

• AudioCraft (MusicGen / AudioGen / EnCodec). Meta's open audio generation stack. MusicGen for music, AudioGen for SFX, EnCodec for neural audio compression.
• COVAL. Guide to building, scaling, and evaluating voice AI, from speech recognition to emotional intelligence.

Open-Source Models (HuggingFace)

Notable open-source generative models.

• Hugging Face Transformers. NLP models and pipelines.
• Hunyuan Video I2V. Image-to-video, multilingual.
• sd-concepts-library (Stable Diffusion Concepts Library). Community library of Stable Diffusion textual-inversion concepts to browse and use in prompts.
• SV3D (Stable Video 3D). Orbital video from single image.

Research Papers

Research papers on ML for CG, generative 3D, neural rendering, and related topics. Each entry: title (the plain-English summary), year, and venue/links in the description.

• 3D Neural Scene Representations for Visuomotor Control. Li et al., CoRL 2021 Oral.
• 3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows. ArXiv 2022 paper on integrating text-to-image AI in 3D design workflows.
• A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields. Park et al., SIGGRAPH Asia 2021. HyperNeRF: lifts NeRF into a higher-dimensional space to handle scenes whose topology changes.
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis. Pan et al., NeurIPS 2021. ShadeGAN: 3D-aware GAN with explicit lighting and shading for more accurate generated shapes.
• Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models. Ling et al., CVPR 2024. Align Your Gaussians generates animated 4D objects from text with dynamic 3D Gaussians.
• AligNeRF: High-Fidelity Neural Radiance Fields via Alignment-Aware Training. Jiang et al., CVPR 2023. Alignment-aware training that reduces misalignment blur so NeRF renders high-resolution detail.
• Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies. Peng et al., ICCV 2021. Builds animatable NeRF human models from multi-view video using skeleton-driven deformation fields.
• Animatable Neural Radiance Fields from Monocular RGB Videos. Chen et al., arXiv 2021. Builds animatable NeRF human avatars from monocular RGB video using a parametric body model.
• AutoInt: Automatic Integration for Fast Neural Volume Rendering. Lindell et al., CVPR 2021. Learns closed-form integrals of volume rendering to speed up neural rendering.
• BeyondPixels: A Review of the Evolution of Neural Radiance Fields. AKM Shahariar Azad Rabby and Chengcui Zhang, Arxiv 2023.
• Block-NeRF: Scalable Large Scene Neural View Synthesis. Tancik et al., Arxiv 2022.
• BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects. Wen et al., CVPR 2023. Jointly tracks the 6-DoF pose of an unknown object and reconstructs its 3D shape from RGB-D video.
• CADOps-Net: Jointly Learning CAD Operation Types and Steps from Boundary-Representations. Dupont et al., 3DV 2022. Network that predicts CAD operation types and their construction steps from B-rep models.
• CLA-NeRF: Category-Level Articulated Neural Radiance Field. Tseng et al., ICRA 2022. Category-level NeRF that renders articulated objects and estimates their part poses.
• CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields. Wang et al., CVPR 2022. Edits the shape and color of NeRF objects using CLIP text or image prompts.
• CodeNeRF: Disentangled Neural Radiance Fields for Object Categories. Jang et al., ICCV 2021. Category-level NeRF with separate shape and texture codes for editing and novel view synthesis.
• Color-NeRF. NeRF research paper on color and appearance modelling.
• Consolidating Attention Features for Multi-view Image Editing. Patashnik et al., SIGGRAPH 2024. Keeps diffusion image edits consistent across multi-view images of a scene.
• Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion. Bhalgat et al., NeurIPS 2023 (Spotlight)
• CROSSFIRE: Camera Relocalization On Self-Supervised Features from an Implicit Representation. Moreau et al., ICCV 2023. CROSSFIRE relocalizes a camera in real time using a NeRF as the scene map.
• D-NeRF: Neural Radiance Fields for Dynamic Scenes. Pumarola et al., CVPR 2021. Extends NeRF with a time-conditioned deformation field to render dynamic scenes from monocular video.
• Deep Generative Models on 3D Representations: A Survey. Shi et al., arXiv 2022. Survey of deep generative models across 3D representations such as point clouds, meshes, voxels and neural fields.
• DeepCAD: A Deep Generative Network for Computer-Aided Design Models. Wu et al., ICCV 2021. Generative network that outputs CAD models as sequences of sketch and extrude operations.
• Deformable Neural Radiance Fields. Park et al., ICCV 2021. Nerfies: adds a learned deformation field to NeRF to capture non-rigid scenes from handheld phone video.
• Depth-supervised NeRF: Fewer Views and Faster Training for Free. Deng et al., CVPR 2022. Adds sparse depth from structure-from-motion as supervision so NeRF trains faster with fewer views.
• DeRF: Decomposed Radiance Fields. Rebain et al., Arxiv 2020.
• DFNet: Enhance Absolute Pose Regression with Direct Feature Matching. Chen et al., ECCV 2022. Improves camera pose regression by matching features against novel views rendered from a NeRF.
• Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. Sun et al., CVPR 2022. Reconstructs radiance fields on explicit voxel grids, converging in minutes instead of hours.
• DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images. Wang et al., ICLR 2023. Decomposes NeRF scenes into individual 3D objects from 2D images so they can be moved, rotated or removed.
• DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model. Xu et al., ICLR 2024. DMV3D denoises multi-view diffusion with a transformer 3D reconstruction model.
• DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. Neff et al., CGF 2021. Uses a depth oracle network to cut samples per ray for near real-time NeRF rendering.
• Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models. Xu et al., CVPR 2023. Zero-shot text-to-3D that uses a 3D shape prior and text-to-image diffusion to guide NeRF optimization.
• DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior. Sun et al., ICLR 2024. DreamCraft3D generates 3D objects from a reference image with bootstrapped diffusion.
• DreamFusion: Text-to-3D using 2D Diffusion. Poole et al., ICLR 2023. Generates 3D NeRF models from text by optimizing them with a pretrained 2D diffusion model via score distillation.
• DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation. Tang et al., ICLR 2024. DreamGaussian generates textured 3D meshes from text or images in minutes with Gaussian splatting.
• DreamGaussian4D: Generative 4D Gaussian Splatting. Ren et al., arXiv 2023. DreamGaussian4D generates animated 4D content quickly with Gaussian splatting.
• DreamWaltz: Make a Scene with Complex 3D Animatable Avatars. Huang et al., NeurIPS 2023. DreamWaltz generates animatable 3D avatars from text with a human body prior.
• DyLiN: Making Light Field Networks Dynamic. Yu et al., CVPR 2023. Extends light field networks to dynamic scenes for fast rendering of deforming content.
• Dynamic Neural Radiance Fields for Monocular 4D Facial Avatar Reconstruction. Gafni et al., CVPR 2021. Dynamic NeRF conditioned on expression parameters to build 4D facial avatars from monocular video.
• DynIBaR: Neural Dynamic Image-Based Rendering. Li et al., CVPR 2023. Image-based rendering method for novel view synthesis from long monocular videos of dynamic scenes.
• Editable Free-viewpoint Video Using a Layered Neural Representation. Zhang et al., SIGGRAPH 2021. Layered neural representation of free-viewpoint video that lets performers be moved, scaled or removed.
• Editing Conditional Radiance Fields. Liu et al., ICCV 2021. Conditional radiance fields that let users edit shape and color of object categories with a few scribbles.
• En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data. Men et al., CVPR 2024. En3D generates realistic 3D human avatars from synthetic 2D data.
• EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction. Liu et al., arXiv 2024. EndoGaussian reconstructs dynamic endoscopic surgical scenes in real time with Gaussian splatting.
• ENeRF: Efficient Neural Radiance Fields for Interactive Free-viewpoint Video. Lin et al., SIGGRAPH Asia 2022. Generalizable NeRF using cascaded cost volumes for interactive free-viewpoint video of dynamic scenes.
• FastNeRF: High-Fidelity Neural Rendering at 200FPS. Garbin et al., Arxiv 2021.
• FiG-NeRF: Figure Ground Neural Radiance Fields for 3D Object Category Modelling. Xie et al., 3DV 2021. Separates foreground objects from background with two NeRFs to model 3D object categories.
• From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approach. Wang et al., AAAI 2025. CAD2Program reconstructs 3D parametric models from 2D CAD drawings with a vision-language model.
• GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting. Zhou et al., ICML 2024. GALA3D generates complex text-to-3D scenes from LLM layouts with Gaussian splatting.
• GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimation. Chng et al., ECCV 2022. Radiance field with Gaussian activations that jointly reconstructs the scene and estimates camera poses.
• GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians. Qian et al., CVPR 2024. GaussianAvatars builds photoreal head avatars from 3D Gaussians rigged to a face model.
• GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting. Yang et al., SIGGRAPH Asia 2024. GaussianObject reconstructs 3D objects from four views with Gaussian splatting.
• Gemini Robotics 1.5. Embodied reasoning, thinking, and motion transfer.
• Generative Occupancy Fields for 3D Surface-Aware Image Synthesis. Xu et al., NeurIPS 2021. Combines occupancy and radiance fields in a 3D-aware GAN for more accurate surfaces.
• GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. Niemeyer et al., CVPR 2021.
• GNeRF: GAN-based Neural Radiance Field without Posed Camera. Meng et al., ICCV 2021. GAN-based NeRF that trains on images with unknown or random camera poses.
• GO-NeRF: Generating Objects in Neural Radiance Fields for Virtual Reality Content Creation. Dai et al., arXiv 2024. GO-NeRF generates 3D objects placed inside existing NeRF scenes for VR content.
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. Schwarz et al., NeurIPS 2020. Generative radiance field model that synthesizes 3D-consistent images from unposed 2D image collections.
• GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering. Trevithick and Yang, ICCV 2021. Learns a general radiance field from 2D image features that renders novel views of unseen scenes.
• Ha-NeRF: Hallucinated Neural Radiance Fields in the Wild. Chen et al., CVPR 2022. Builds NeRFs from in-the-wild photo collections, handling appearance changes and transient occluders.
• HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video. Weng et al., CVPR 2022. Renders moving people from new viewpoints using a NeRF trained on a single monocular video.
• IBRNet: Learning Multi-View Image-Based Rendering. Wang et al., CVPR 2021. Learns to blend nearby source views for image-based rendering that generalizes to unseen scenes.
• IDE-3D: Interactive Disentangled Editing for High-Resolution 3D-aware Portrait Synthesis. Sun et al., SIGGRAPH Asia 2022. 3D-aware GAN for high-resolution portraits with interactive, disentangled editing of shape, texture and pose.
• Image Sculpting: Precise Object Editing with 3D Geometry Control. Yenphraphai et al., CVPR 2024. Image Sculpting edits 2D objects by lifting them to 3D geometry.
• Importance Sampling for Production Rendering. SIGGRAPH course notes on importance sampling for production rendering.
• In-Place Scene Labelling and Understanding with Implicit Scene Representation. Zhi et al., ICCV 2021. Semantic-NeRF: adds semantic labels to a radiance field to propagate sparse labels across a scene.
• iNeRF: Inverting Neural Radiance Fields for Pose Estimation. Yen-Chen et al. IROS 2021.
• InfNeRF: Towards Infinite Scale NeRF Rendering with O(log n) Space Complexity. Liang et al., SIGGRAPH Asia 2024. Octree-based level-of-detail NeRF that renders very large scenes with O(log n) memory.
• Instant3D: Instant Text-to-3D Generation. Li et al., arXiv 2023. Instant3D generates 3D objects from text in one feed-forward pass, without per-prompt optimization.
• KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. Reiser et al., ICCV 2021. Splits a NeRF into thousands of tiny MLPs to render orders of magnitude faster.
• KiloNeuS: Implicit Neural Representations with Real-Time Global Illumination. Esposito et al., Arxiv 2022.
• L2G-NeRF: Local-to-Global Registration for Bundle-Adjusting Neural Radiance Fields. Chen et al., CVPR 2023. Bundle-adjusting NeRF that registers cameras locally per pixel before global alignment.
• Learned Initializations for Optimizing Coordinate-Based Neural Representations. Tancik et al., CVPR 2021. Meta-learns initial weights for coordinate-based networks so they fit new signals in fewer steps.
• Learning Compositional Radiance Fields of Dynamic Human Heads. Wang et al., CVPR 2021. Hybrid voxel and MLP radiance field for rendering dynamic human heads with fine detail.
• Learning Neural Transmittance for Efficient Rendering of Reflectance Fields. Mohammad Shafiei et al., BMVC 2021.
• Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering. Yang et al., ICCV 2021. Object-compositional NeRF that lets individual objects in a scene be moved, duplicated or removed.
• Learning the 3D Fauna of the Web. Li et al., CVPR 2024. 3D-Fauna learns a deformable 3D model of 100+ animal species from web images.
• Lightning NeRF: Efficient Hybrid Scene Representation for Autonomous Driving. Cao et al., ICRA 2024. Lightning NeRF is a hybrid scene representation for fast NeRF training in driving scenes.
• Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields. Maggio et al., ICRA 2023. Robot localization method that runs a Monte Carlo particle filter against a pretrained NeRF map.
• Local 3D Editing via 3D Distillation of CLIP Knowledge. Hyung et al., CVPR 2023. Edits local regions of 3D-aware GAN outputs by distilling CLIP knowledge.
• M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts. Li et al., arXiv 2023. M3DBench is a benchmark for instructing large models with multi-modal 3D prompts.
• Magic3D: High-Resolution Text-to-3D Content Creation. Lin et al., CVPR 2023. Two-stage coarse-to-fine text-to-3D method that produces high-resolution textured meshes.
• Make-A-Character: High Quality Text-to-3D Character Generation within Minutes. Ren et al., arXiv 2023. Make-A-Character creates lifelike 3D character avatars from text descriptions.
• Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior. Tang et al., ICCV 2023. Make-It-3D creates textured 3D objects from a single image using a 2D diffusion prior.
• MinD-3D: Reconstruct High-quality 3D objects in Human Brain. Gao et al., ECCV 2024. MinD-3D reconstructs 3D objects from fMRI brain signals.
• MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis. Li et al., ICCV 2021. Combines multiplane images with NeRF to predict continuous-depth MPIs from a single image.
• Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. Barron et al., Arxiv 2022.
• Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. Barron et al., ICCV 2021. Renders anti-aliased cones instead of rays so NeRF handles multiple scales without aliasing.
• Mixture of Volumetric Primitives for Efficient Neural Rendering. Lombardi et al., SIGGRAPH 2021.
• MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo. Chen et al., ICCV 2021. Uses multi-view stereo cost volumes so radiance fields generalize to new scenes from three input views.
• NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes. Irshad et al., ICCV 2023. NeO 360 synthesizes novel views of outdoor scenes from a few input images.
• NeRF (paper site). Mildenhall et al., ECCV 2020. The original NeRF: an MLP that maps 5D coordinates to color and density for novel view synthesis.
• NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images. Ben Mildenhall et al, arXiv 2021.
• NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. Martin-Brualla et al., CVPR 2021.
• NeRF--: Neural Radiance Fields Without Known Camera Parameters. Wang et al., arXiv 2021. Trains NeRF while jointly estimating camera intrinsics and extrinsics from forward-facing images.
• NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors. Liu et al., arXiv 2022. Removes user-selected objects from a NeRF scene using RGB-D inpainting priors.
• NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fields. Irshad et al., ECCV 2024. NeRF-MAE pretrains masked autoencoders on NeRF grids for 3D representation learning.
• NeRF-SOS: Any-view Self-supervised Object Segmentation on Complex Real-world Scenes. Fan et al., ICLR 2023. Self-supervised object segmentation that works from any view of complex real-world scenes using NeRF.
• NeRF-VAE: A Geometry Aware 3D Scene Generative Model. Kosiorek et al., Arxiv 2021.
• NeRF: Neural Radiance Field in 3D Vision, Introduction and Review. Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, Jonathan Li.
• NeRF++: Analyzing and Improving Neural Radiance Fields. Zhang et al., arXiv 2020. Analyzes NeRF and adds an inverted-sphere background model for unbounded 360-degree scenes.
• NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo. Wei et al., ICCV 2021. Guides NeRF optimization with monocular depth priors for indoor multi-view stereo reconstruction.
• NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. Srinivasan et al. CVPR 2021.
• Neural 3D Video Synthesis from Multi-view Video. Li et al., CVPR 2022. Dynamic NeRF conditioned on latent codes that renders novel views of multi-view video.
• Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans. Peng et al., CVPR 2021. Anchors latent codes to a deformable body mesh to render dynamic humans from sparse multi-view video.
• Neural Radiance Flow for 4D View Synthesis and Video Processing. Du et al., ICCV 2021. Learns a 4D radiance and flow field for view synthesis and video processing such as denoising and super-resolution.
• Neural Rays for Occlusion-aware Image-based Rendering. Liu et al., CVPR 2022. Image-based rendering that predicts visibility per ray to handle occlusions when synthesizing novel views.
• Neural Refinement for Absolute Pose Regression with Feature Synthesis. Chen et al., CVPR 2024. Refines camera pose regression at test time using features rendered from a neural feature field.
• Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes. Li et al., CVPR 2021. Models dynamic scenes as time-varying scene flow fields for space-time view synthesis from monocular video.
• Neural Scene Graphs for Dynamic Scenes. Ost et al., CVPR 2021. Decomposes dynamic scenes like driving footage into a learned scene graph of moving objects.
• Neural Sparse Voxel Fields. Liu et al., NeurIPS 2020. Organizes a radiance field into a sparse voxel octree to skip empty space and render faster.
• Neural Volume Rendering: NeRF. Dellaert and Yen-Chen, arXiv 2020. Survey of the NeRF paper and the wave of neural volume rendering work that followed it.
• NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. Wang et al., NeurIPS 2021. Represents surfaces as signed distance functions trained with volume rendering for multi-view reconstruction.
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video. Tretschk et al., ICCV 2021. Reconstructs deforming scenes from monocular video with a NeRF plus a learned ray-bending deformation field.
• NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior. Bian et al., CVPR 2023. Trains NeRF without pose priors by using monocular depth to constrain relative camera poses.
• One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion. Liu et al., CVPR 2024. One-2-3-45++ turns a single image into a textured 3D mesh in about a minute.
• ParSeNet: A Parametric Surface Fitting Network for 3D Point Clouds. Sharma et al., ECCV 2020. Network that segments point clouds and fits parametric surfaces such as B-splines and primitives.
• PC2WF: 3D Wireframe Reconstruction from Raw Point Clouds. Liu et al., ICLR 2021. Network that extracts 3D wireframes of vertices and edges directly from raw point clouds.
• Photo tourism: Exploring photo collections in 3D. Snavely, Seitz, Szeliski. SIGGRAPH 2006.
• pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis. Chan et al., CVPR 2021. 3D-aware GAN using SIREN periodic activations in a radiance field to generate view-consistent images.
• pixelNeRF: Neural Radiance Fields from One or Few Images. Yu et al., CVPR 2021. Conditions NeRF on image features so it predicts radiance fields from one or a few input images.
• PlankAssembly: 3D Reconstruction from Three Orthographic Views with Learnt Shape Programs. Hu et al., ICCV 2023. PlankAssembly turns three orthographic line drawings into 3D CAD models.
• PlenOctrees for Real-time Rendering of Neural Radiance Fields. Yu et al., ICCV 2021. Bakes NeRFs into octrees of spherical harmonics for real-time rendering.
• Point-NeRF: Point-based Neural Radiance Fields. Xu et al., CVPR 2022. Uses neural point clouds to model radiance fields, initialized from multi-view stereo for fast training.
• Point2Cyl: Reverse Engineering 3D Objects from Point Clouds to Extrusion Cylinders. Uy et al., CVPR 2022. Reverse-engineers point clouds into editable CAD by fitting sketch-and-extrude cylinders.
• PolyGen: An Autoregressive Generative Model of 3D Meshes. Nash et al., ICML 2020. Transformer that generates 3D meshes directly as sequences of vertices and faces.
• Ponder: Point Cloud Pre-training via Neural Rendering. Huang et al., ICCV 2023. Pre-trains point cloud encoders by rendering color and depth images through differentiable neural rendering.
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm. Zhu et al., arXiv 2023. PonderV2 is a universal pretraining method for 3D foundation models.
• PoRF: Pose Residual Field for Accurate Neural Surface Reconstruction. Bian et al., ICLR 2024. Learns a pose residual field to refine camera poses for more accurate neural surface reconstruction.
• Portrait Neural Radiance Fields from a Single Image. Gao et al., arXiv 2020. Meta-learned NeRF that reconstructs a portrait radiance field from a single headshot.
• Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts. Cheng et al., ICLR 2024. Progressive3D edits text-to-3D content region by region for complex prompts.
• PSAvatar: A Point-based Shape Model for Real-Time Head Avatar Animation with 3D Gaussian Splatting. Zhao et al., arXiv 2024. PSAvatar animates real-time head avatars with a point-based model and 3D Gaussians.
• Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis. Matthew Tancik et al., Arxiv 2021.
• PVA: Pixel-aligned Volumetric Avatars. Raj et al., CVPR 2021. Builds volumetric head avatars from a few input views using pixel-aligned features.
• PVDeconv: Point-voxel deconvolution for autoencoding cad construction in 3D. Cherenkova et al., ICIP 2020. Point-voxel deconvolution autoencoder that reconstructs CAD models from 3D scans.
• Reconstructing Editable Prismatic CAD from Rounded Voxel Models. Lambourne et al., SIGGRAPH Asia 2022. Converts rounded voxel shapes into editable prismatic CAD models built from extruded sketches.
• Recovering HDR Radiance Maps (Debevec & Malik, 1997). SIGGRAPH 1997 paper. Original HDR-from-bracketed-exposures recovery method.
• Rendering Synthetic Objects into Real Scenes (Debevec, 1998). SIGGRAPH 1998 paper founding image-based lighting.
• RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D. Qiu et al., CVPR 2024. RichDreamer uses a normal-depth diffusion model for detailed text-to-3D.
• Rig3DGS: Creating Controllable Portraits from Casual Monocular Videos. Rivero et al., arXiv 2024. Rig3DGS builds controllable 3D Gaussian portraits from casual phone videos.
• Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion. Wang et al., CVPR 2023. Diffusion model that generates 3D digital avatars as neural radiance fields, including from text or a photo.
• RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering. Li et al., ICCAD 2022. Algorithm and hardware co-design for real-time NeRF rendering on edge devices aimed at AR and VR.
• SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance. Xu et al., arXiv 2023. SEEAvatar generates photoreal 3D avatars from text with constrained geometry and appearance.
• Self-Calibrating Neural Radiance Fields. Jeong et al., ICCV 2021. Jointly learns scene geometry and camera calibration, including lens distortion, without a calibration target.
• SIGNeRF: Scene Integrated Generation for Neural Radiance Fields. Dihlmann et al., CVPR 2024. SIGNeRF generates and edits objects inside existing NeRF scenes.
• SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field. Bao et al., CVPR 2023. Edits a NeRF from a single edited image using semantic and geometric priors.
• SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image. Xu et al., ECCV 2022. Trains a NeRF on complex scenes from a single image using semi-supervised geometry and semantic losses.
• Space-time Neural Irradiance Fields for Free-Viewpoint Video. Xian et al., CVPR 2021. Space-time irradiance field that renders free-viewpoint video from a single monocular video plus depth.
• Streaming Radiance Fields for 3D Video Synthesis. Li et al., NeurIPS 2022. Streams radiance fields frame by frame to reconstruct and render free-viewpoint 3D video.
• Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting. Yan et al., ECCV 2024. Street Gaussians models dynamic urban driving scenes with Gaussian splatting.
• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis. Gu et al., ICLR 2022. Style-based 3D-aware generator combining NeRF and StyleGAN to synthesize high-resolution, view-consistent images.
• Supervised Fitting of Geometric Primitives to 3D Point Clouds. Li et al., CVPR 2019. SPFN: supervised network that fits geometric primitives such as planes, spheres and cylinders to point clouds.
• SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes. Gao et al., CVPR 2023. SurfelNeRF reconstructs and renders indoor scenes online with neural surfels.
• Switch-NeRF: Learning Scene Decomposition with Mixture of Experts for Large-scale Neural Radiance Fields. Mi et al., ICLR 2023. Mixture-of-experts NeRF that learns to split large-scale scenes into sub-networks.
• Taming Mode Collapse in Score Distillation for Text-to-3D Generation. Wang et al., arXiv 2023. Reduces multi-face Janus artifacts in score distillation text-to-3D generation.
• Text-To-4D Dynamic Scene Generation. Singer et al., ICML 2023. MAV3D generates dynamic 3D scenes from text with a 4D NeRF and text-to-video diffusion.
• Text2CAD: Text to 3D CAD Generation via Technical Drawings. Yavartanoo et al., arXiv 2024. Text2CAD generates 3D CAD models from text by first producing technical drawings.
• TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields. Huang et al., ICLR 2024. TextField3D adds noisy text fields for open-vocabulary 3D generation.
• TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion. Yeh et al., CVPR 2024. TextureDreamer transfers relightable textures from 3 to 5 photos onto 3D shapes.
• TiNeuVox: Fast Dynamic Radiance Fields with Time-Aware Neural Voxels. Fang et al., SIGGRAPH Asia 2022. Time-aware voxel features that train dynamic radiance fields in minutes.
• TOSS:High-quality Text-guided Novel View Synthesis from a Single Image. Shi et al., arXiv 2023. TOSS adds text guidance to novel view synthesis from a single image.
• Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers. Zou et al., CVPR 2024. Reconstructs 3D Gaussians from a single image with a triplane transformer.
• UV Volumes for Real-time Rendering of Editable Free-view Human Performance. Chen et al., CVPR 2023. Encodes human performances in UV volumes so free-view video renders in real time and can be retextured.
• ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields. Dong et al., NeurIPS 2023. ViCA-NeRF edits NeRF scenes from text instructions with multi-view consistency.
• Volume Rendering of Neural Implicit Surfaces. Yariv et al., NeurIPS 2021. VolSDF: models density as a function of signed distance for better surface reconstruction with volume rendering.
• Wonder3D: Single Image to 3D using Cross-Domain Diffusion. Long et al., CVPR 2024. Wonder3D generates textured meshes from a single image with cross-domain diffusion.
• X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360° Insufficient RGB-D Views. Zhu et al., WACV 2023. Explicit radiance field that reconstructs multiple 360-degree scenes from sparse, insufficient RGB-D views.
• Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model. Shi et al., arXiv 2023. Zero123++ generates consistent multi-view images from one input image for 3D creation.

AI Image & Texture Generation Software

Also in Software Reference → AI Image & Texture Generation Software

AI Video Generation Software

Also in Software Reference → AI Video Generation Software

AI 3D Generation Software

Also in Software Reference → AI 3D Generation Software

AI Audio & Music Generation Software

Also in Software Reference → AI Audio & Music Generation Software

AI-Assisted CG Software

Also in Software Reference → AI-Assisted CG Software

AI Creative Canvas Software

Also in Software Reference → AI Creative Canvas Software

AI Design Viz Software

Also in Software Reference → AI Design Viz Software

AI Product Photography Software

Also in Software Reference → AI Product Photography Software