AI & Machine Learning for CGEdit

Generative AI and ML tools across image, video, 3D, texture, audio. AI mocap lives in §3 (dual-listed).

Machine Learning for CG

ML fundamentals and courses for CG artists.

• DINOv2. Meta self-supervised vision features. Strong general-purpose image embeddings. Useful as backbone for CG ML tasks.
• IEEE Transactions on Neural Networks and Learning Systems. Publishes technical articles on the theory, design, and applications of neural networks and related learning systems.
• Meta 3D Gen. Meta research pipeline for text-to-3D. Combines AssetGen geometry with TextureGen materials. Paper and results.
• PyTorch3D. Meta FAIR library for 3D deep learning. Differentiable rendering, mesh ops, point cloud utils on PyTorch.

Image Generation

text-to-image models and platforms.

Related:

• (5) Deforum notebook v0.5 for Stable Diffusion animations is out! Now with math automation, perspective flips, prompt weights, video masking and waifus! : StableDiffusion. (5) Deforum notebook v0.5 for Stable Diffusion animations is out! Now with math automation, perspective flips, prompt weights, video masking and waifus! : StableDiffusion.
• 3DALL-E: Integrating Text-to-Image AI in 3D Design Workflows. ArXiv 2022 paper on integrating text-to-image AI in 3D design workflows.
• AI Render. Stable Diffusion in Blender. This add-on renders an AI generated image based on a text prompt and your scene.
• Canva Adds a Free and Unlimited AI Text-to-Image Generator | PetaPixel. Canva Adds a Free and Unlimited AI Text-to-Image Generator | PetaPixel.
• COVAL. Architecture of voice AI, from speech recognition to emotional intelligence, and learn how to build, scale, and evaluate them
• Deforum Stable Diffusion Animation - v5 Math Functions - Demo and Test - YouTube. Deforum Stable Diffusion Animation - v5 Math Functions - Demo and Test - YouTube.
• Generative AI Genius 2024 (created by Aishwarya Naresh Reganti) course material. Generative AI Genius 2024 (created by Aishwarya Naresh Reganti) course material.
• Introduction to Generative AI Community Course. By ineuron
• Openai/point E. OpenAI's point cloud diffusion for 3D model synthesis
• sd-concepts-library (Stable Diffusion concepts library). Stable Diffusion Textual Inversion Concepts Library - browse through objects and styles taught by the community to Stable Diffusion and use them in your prompts!
• Stable Diffusion Frivolous · Because lawsuits based on ignorance deserve a response. Community response for the "Stable Diffusion litigation"
• Stable Diffusion KLMC2 Animation.ipynb. Notebook by @RiversHaveWings to generate animation based on scripted prompts using a technique called KLMC2 discretization of underdamped Langevin dynamics
• Stable Diffusion KLMC2 Animation.ipynb forked. Fork by @DigThatData
• Transforming 2D Images into 3D with the AdaMPI AI Model. Guide on how to use the AdaMPI AI model for creating 3D photos from 2D images
• Word-As-Image for Semantic Typography. Semantically transforming fonts into illustrations

Video Generation

AI video generation platforms and models.

Related:

• Emu Video (Meta). Meta's text-to-video research model. Factorized generation, image-then-video. Demos and paper.

3D Generation

AI tools for generating 3D models from text or images.

Related:

• Pixal3D. Single-image to 3D asset generation via pixel-feature back-projection. Outputs geometry and PBR textures.
• Threestudio. A unified framework for 3D content generation.

Texture & Material Generation

AI-generated PBR textures and materials.

ComfyUI Ecosystem

Node-based AI generation. ComfyUI and its ecosystem.

Related:

• Awesome ComfyUI. Curated custom nodes collection.
• Comflowy. ComfyUI tutorials for ControlNet, SDXL, FLUX workflows.
• Comfy UI Manager. Install and manage custom nodes and dependencies.
• ComfyUI Registry. Community workflow and subgraph sharing.
• ControlNet SD3.5. Blur/Canny/Depth models, free commercial use.

AI-Assisted CG Tools

AI tools that augment traditional CG workflows.

Related:

• AI Game Developer. Unity Editor and Unity Runtime AI integration. Unit Test, Coding, C# Roslyn, Reflection, Assets. Helps to create games with AI. And helps to run AI logic during gameplay.
• Comfy UI Blender AI. Run ComfyUI workflows inside Blender.
• ControlNet. Depth/pose/edge control for precision CG workflows.
• Coplay Freemium. AI Copilot for Unity
• CorridorKey. Corridor Crew's open keying tool. AI-driven chroma key for VFX work.
• CoTracker. Meta FAIR point tracker for video. Tracks arbitrary points jointly across frames. Useful for matchmove and roto prep.
• DEVA (Tracking Anything). Decoupled video segmentation. Long-form, multi-object roto with text or click prompts.
• Dream Textures (Blender). Generate textures in Blender via Stable Diffusion, locally.
• GPT 4. Prompt engineering techniques for GPT-4, including tips, applications, limitations, and additional reading materials.
• RIFE for Nuke. Real-Time Intermediate Flow Estimation for Video Frame Interpolation (ie machine learning for framerate upscaling) for Nuke.
• Robust Video Matting (RVM). Real-time human video matting. Strong alpha mattes with no greenscreen.
• Rotobot Paid. Paid AI roto plugin for Nuke and After Effects. Auto-mattes people and objects.
• SAM 2 (Segment Anything Model 2). Meta's promptable segmentation for images and video. Masks objects across frames from a single click. Useful for rotoscope and matte work.
• Sammie Roto 2. SAM-based rotoscope app. Click-to-mask, propagates across frames.
• SolidUI. AI-generated visualization prototyping and editing platform, support 2D, 3D models, combined with LLM(Large Language Model) for quick editing.
• Track Anything. SAM plus XMem for interactive video object tracking and segmentation. Useful for roto and matte propagation.
• Wonder Studio (Autodesk Flow Studio) Freemium. AI character replacement with auto roto, body tracking, and re-lighting from a single video. Now part of Autodesk Flow Studio.

AI Audio & Music

AI music generation, voice, TTS, and SFX.

Related:

• AudioCraft (MusicGen / AudioGen / EnCodec). Meta's open audio generation stack. MusicGen for music, AudioGen for SFX, EnCodec for neural audio compression.

Open-Source Models (HuggingFace)

Notable open-source generative models.

• Hugging Face Transformers. NLP models and pipelines.
• Hunyuan Video I2V. Image-to-video, multilingual.
• SV3D (Stable Video 3D). Orbital video from single image.

Research Papers

Research papers on ML for CG, generative 3D, neural rendering, and related topics. Each entry: title (the plain-English summary), year, and venue/links in the description.

• Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models. 🌐 Project Page
• Animatable Neural Radiance Fields from Monocular RGB Videos. Chen et al., Arxiv 2021 | github | bibtex
• BeyondPixels: A Review of the Evolution of Neural Radiance Fields. AKM Shahariar Azad Rabby and Chengcui Zhang, Arxiv 2023 | bibtex
• CADOps-Net: Jointly Learning CAD Operation Types and Steps from Boundary-Representations. 3DV 2022 | [project] |
• CLA-NeRF: Category-Level Articulated Neural Radiance Field. Tseng et al., ICRA 2022 | bibtex
• Consolidating Attention Features for Multi-view Image Editing. 🌐 Project Page
• CROSSFIRE: Camera Relocalization On Self-Supervised Features from an Implicit Representation. Moreau et al., ICCV 2023 | bibtex
• Deep Generative Models on 3D Representations: A Survey. 🌐 Project Page
• DeepCAD: A Deep Generative Network for Computer-Aided Design Models. ICCV 2021 | [project] [code] |
• Depth-supervised NeRF: Fewer Views and Faster Training for Free. Deng et al., Arxiv 2021 | github | bibtex
• DeRF: Decomposed Radiance Fields. Rebain et al. Arxiv 2020 | bibtex
• Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. Sun et al., CVPR 2022 | github | bibtex
• DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images. 💻 Code
• DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Model. 🌐 Project Page
• Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models. 🌐 Project Page | [💻 Code][Coming soon.]
• DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior. 🌐 Project Page | 💻 Code
• DreamFusion: Text-to-3D using 2D Diffusion. 🌐 Project Page | 💻 Code
• DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation. 🌐 Project Page | 💻 Code
• DreamGaussian4D: Generative 4D Gaussian Splatting. 🌐 Project Page | 💻 Code
• DreamWaltz: Make a Scene with Complex 3D Animatable Avatars. 🌐 Project Page | 💻 Code
• En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data. 🌐 Project Page | 💻 Code
• EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction. 🌐 Project Page | 💻 Code
• FastNeRF: High-Fidelity Neural Rendering at 200FPS. Garbin et al., Arxiv 2021 | bibtex
• From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approach. AAAI 2025 | [project] |
• GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting. 🌐 Project Page | 💻 Code
• GARF: Gaussian Activated Radiance Fields for High Fidelity Reconstruction and Pose Estimation. Chng et al., ECCV 2022 | bibtex
• GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians. 🌐 Project Page
• GaussianObject: High-Quality 3D Object Reconstruction from Four Views with Gaussian Splatting. 🌐 Project Page | 💻 Code
• Gemini Robotics 1.5. Embodied reasoning, thinking, and motion transfer.
• GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields. Niemeyer et al., CVPR 2021, bibtex
• GNeRF: GAN-based Neural Radiance Field without Posed Camera. Meng et al., Arxiv 2021 | bibtex
• GO-NeRF: Generating Objects in Neural Radiance Fields for Virtual Reality Content Creation. 🌐 Project Page
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. Schwarz et al., NeurIPS 2020 | github | bibtex
• GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering. Trevithick and Yang, Arxiv 2020 | github | bibtex
• IDE-3D: Interactive Disentangled Editing for High-Resolution 3D-aware Portrait Synthesis. 🌐 Project Page | 💻 Code
• Image Sculpting: Precise Object Editing with 3D Geometry Control. 🌐 Project Page | 💻 Code
• Instant3D: Instant Text-to-3D Generation. 🌐 Project Page | 💻 Code
• KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs. Reiser et al., ICCV 2021 | github | bibtex
• KiloNeuS: Implicit Neural Representations with Real-Time Global Illumination. Esposito et al., Arxiv 2022 | bibtex
• Learned Initializations for Optimizing Coordinate-Based Neural Representations. Tancik et al., CVPR 2021 | github | bibtex
• Learning the 3D Fauna of the Web. 🌐 Project Page | 💻 Code
• Lightning NeRF: Efficient Hybrid Scene Representation for Autonomous Driving. Cao et al. ICRA 2024 | github | bibtex
• Loc-NeRF: Monte Carlo Localization using Neural Radiance Fields. Maggio et al., ICRA 2023 | github | bibtex
• Local 3D Editing via 3D Distillation of CLIP Knowledge. Hyung et al., CVPR 2023| github | bibtex
• M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts. 🌐 Project Page | 💻 Code
• Magic3D: High-Resolution Text-to-3D Content Creation. 🌐 Project Page | [💻 Code][Coming soon.]
• Make-A-Character: High Quality Text-to-3D Character Generation within Minutes. 🌐 Project Page | 💻 Code
• Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior. 🌐 Project Page | 💻 Code
• MinD-3D: Reconstruct High-quality 3D objects in Human Brain. 🌐 Project Page | 💻 Code
• Mixture of Volumetric Primitives for Efficient Neural Rendering. Lombardi et al., SIGGRAPH 2021 | bibtex
• NeO 360: Neural Fields for Sparse View Synthesis of Outdoor Scenes. Irshad et al., ICCV 2023 | github | bibtex
• NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation Learning for Neural Radiance Fields. Irshad et al., ECCV 2024 | bibtex
• NeRF-VAE: A Geometry Aware 3D Scene Generative Model. Kosiorek et al., Arxiv 2021 | bibtex
• NeRF: Neural Radiance Field in 3D Vision, Introduction and Review. Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, Jonathan Li
• NeRF++: Analyzing and Improving Neural Radiance Fields. Zhang et al., Arxiv 2020 | github | bibtex
• Neural Volume Rendering: NeRF. Dellaert and Yen-Chen, Arxiv 2020 | blog | github |
• NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. Wang et al., NeurIPS 2021 | github | bibtex
• One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion. 🌐 Project Page | 💻 Code
• ParSeNet: A Parametric Surface Fitting Network for 3D Point Clouds. ECCV 2020 | [project] [code] |
• PC2WF: 3D Wireframe Reconstruction from Raw Point Clouds. ICLR 2021 | [code] |
• pixelNeRF: Neural Radiance Fields from One or Few Images. Yu et al., CVPR 2021 | github | bibtex
• PlankAssembly: 3D Reconstruction from Three Orthographic Views with Learnt Shape Programs. ICCV 2023 | [project] [code] |
• Point2Cyl: Reverse Engineering 3D Objects from Point Clouds to Extrusion Cylinders. CVPR 2022 | [project] [code] |
• PolyGen: An Autoregressive Generative Model of 3D Meshes. ICML 2020 | [code] |
• PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm. Zhu et al., Arxiv 2023 | github | bibtex
• Progressive3D: Progressively Local Editing for Text-to-3D Content Creation with Complex Semantic Prompts. 🌐 Project Page | 💻 Code
• PSAvatar: A Point-based Shape Model for Real-Time Head Avatar Animation with 3D Gaussian Splatting. 💻 Code
• PVDeconv: Point-voxel deconvolution for autoencoding cad construction in 3D. ICIP 2020 | [project] |
• Reconstructing Editable Prismatic CAD from Rounded Voxel Models. SIGGRAPH Asia 2022 | |
• RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D. 🌐 Project Page | 💻 Code
• Rig3DGS: Creating Controllable Portraits from Casual Monocular Videos. 🌐 Project Page
• RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Rendering. Li et al., ICCAD 2022 | bibtex
• SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance. 🌐 Project Page
• SIGNeRF: Scene Integrated Generation for Neural Radiance Fields. 🌐 Project Page | 💻 Code
• Streaming Radiance Fields for 3D Video Synthesis. Li et al. NeurIPS 2022 | github | bibtex
• Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting. 🌐 Project Page | 💻 Code
• Supervised Fitting of Geometric Primitives to 3D Point Clouds. CVPR 2019 | [code] |
• SurfelNeRF: Neural Surfel Radiance Fields for Online Photorealistic Reconstruction of Indoor Scenes. Gao et al., CVPR 2023 | github | bibtex
• Taming Mode Collapse in Score Distillation for Text-to-3D Generation. 🌐 Project Page | 💻 Code
• Text-To-4D Dynamic Scene Generation. 🌐 Project Page
• Text2CAD: Text to 3D CAD Generation via Technical Drawings. NeurIPS 2024 | [project] |
• TextField3D: Towards Enhancing Open-Vocabulary 3D Generation with Noisy Text Fields. 🌐 Project Page
• TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion. 🌐 Project Page
• TOSS:High-quality Text-guided Novel View Synthesis from a Single Image. 🌐 Project Page
• Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformers. 🌐 Project Page
• ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields. 💻 Code
• Volume Rendering of Neural Implicit Surfaces. Yariv et al., NeurIPS 2021 | github | bibtex
• Wonder3D: Single Image to 3D using Cross-Domain Diffusion. 🌐 Project Page | 💻 Code
• X-NeRF: Explicit Neural Radiance Field for Multi-Scene 360° Insufficient RGB-D Views. Zhu et al., WACV 2023 | github | bibtex
• Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model. 🌐 Project Page | 💻 Code | 🤗 Hugging Face

AI Image & Texture Generation Software

Also in Software Reference → AI Image & Texture Generation Software

AI Video Generation Software

Also in Software Reference → AI Video Generation Software

AI 3D Generation Software

Also in Software Reference → AI 3D Generation Software

AI Audio & Music Generation Software

Also in Software Reference → AI Audio & Music Generation Software

AI-Assisted CG Software

Also in Software Reference → AI-Assisted CG Software