Skip to content

A colab friendly toolkit to generate 3D mesh model / video / nerf instance / multiview images of colourful 3D objects by text and image prompts input, based on dreamfields.

License

Notifications You must be signed in to change notification settings

shengyu-meng/dreamfields-3D

 
 

Repository files navigation

dreamfields-3D

A colab friendly toolkit to generate 3D mesh model / video / NeRF instance / multiview images of colourful 3D objects by text and image prompts input. Edited by Shengyu Meng (Simon)

Check the colab notebook for usage. Open In Colab

Dreamfields-3D is modified from dreamfields-torch and dreamfields, please check the Credits.md for details.

2

Update logs

Beta v0.65 :add options for apply image prompts only in assigned direction to avoid overfitting; could skip saving depth maps in training and testing; output sequence images in validation process. (2022-10-03)
Beta v0.60 : apply random fovy (view angle) in training; update image augmentation before feed into CLIP; improve training stability and performance. (2022-09-25)

ngp_200_rgb-.mp4

Example generated by text prompt: "a cyborg organic biological pavilion could breathe with building skin containing algae, in the style of dezeen, trending on artstation, surreal", with CLIP ViT-L/14 model, training for 200 epochs with clip_aug and random_fovy_training mode enabled.

Preview.mp4

Example generated by text prompt: "a beautiful painting of a flower tree, by Chiho Aoshima, Long shot, surreal", with CLIP ViT-L/14 model, training for 200 epochs.

Main Contributions:

  • Export obj & ply model with vertex colour.
  • Export 360° Video of final model.
  • Visualizing the training progress and preview the output video in colab.
  • Improve the generation quality.
    • Allow to use different CLIP models.
    • Improve the pre-process of the renderings before feeding into CLIP.
    • Apply random view angle in training.
  • Add more useful augments.
  • Organize the colab notebook.

Future update plan:

  • Use different CLIP models simultaneously.
  • Convert existing mesh to NeRF instance then modify by text / image prompts.
  • Reduce GPU RAM occupation in training.

Becuase I am a coding beginner, cannot promise for the progress. Welcome to contact me to contribute to this repository.

Compatibility:

  • About system:
    • Colab: Pass on google Colab (tested on A100/v100/P100 GPU at 08/09/2022, GPU less than 16G RAM need to lower the clip_w_h and clip_aug_copy volume)
    • Ubuntu: The previous version (dreamfields-torch) has successfully ran on Ubuntu 18.04 with RTX 3090. No test has been done for the dreamfields-3D yet, but it should be fined.
    • Windows: It should work in windows with proper environment, but I failed to build the raymarching extension in different windows machine. More test will be required.
    • For locally running, please refer to the following installation guide from dreamfields-torch.
  • About Exporting 3D model:
    • The 3D models will be exported as obj and ply format with vertex colour.
    • Mesh with vertex colour could be directly viewed in meshLab and Rhino3D.
    • For viewing colour in Blender, please import the ply format model first (obj supported in latest 3.3 version), then create a new material, and plug a Color Attribute node into the base color in shader editor, then you should see the vertex colour.
  • About GUI:
    • GUI is supported only when running locally. However, some new features maybe not available in GUI model yet. I would recommend to train without GUI first, then run test on the pretrained model in GUI, to browse the final 3D NeRF instance.

👇 Bellow readme was modified from the dreamfields-torch repository.


dreamfields-torch (WIP)

A pytorch implementation of dreamfields as described in Zero-Shot Text-Guided Object Generation with Dream Fields.

An example of a generated neural field by prompt "cthulhu" viewed in real-time:

cthulhu.mp4

Install

The code framework is based on torch-ngp.

git clone https://github.com/shengyu-meng/dreamfields-3D.git
cd dreamfields-3D

Install with pip

pip install -r requirements.txt

install customized verion of pymarchingcubes

bash scripts/install_PyMarchingCubes.sh

Build extension

# install all extension modules
bash scripts/install_ext.sh
# if you want to install manually, here is an example:
cd raymarching
python setup.py build_ext --inplace # build ext only, do not install (only can be used in the parent directory)
pip install . # install to python path (you still need the raymarching/ folder, since this only install the built extension.)

Tested environments

(For dreamfields-torch, not for dreamfileds-3D)

  • Ubuntu 20 with torch 1.10 & CUDA 11.3 on a TITAN RTX.
  • Windows 10 with torch 1.11 & CUDA 11.3 on a RTX 3070.

Currently, --ff only supports GPUs with CUDA architecture >= 70. For GPUs with lower architecture, --tcnn can still be used, but the speed will be slower compared to more recent GPUs.

Usage

First time running will take some time to compile the CUDA extensions.

# text-guided generation
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16

# use the GUI
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16 --gui

# [experimental] image-guided generation (also use the CLIP loss)
python main_nerf.py --image /path/to/image --workspace trial --cuda_ray --fp16

check the scripts directory for more provided examples.

Difference from the original implementation

  • Mip-nerf is not implemented, currently only the original nerf is supported.
  • Sampling poses with an elevation range in [-30, 30] degrees, instead of fixed at 30 degree.
  • Use the origin loss.

Update Logs

  • 5.18: major update.
  • 3.16: basic reproduction.

Acknowledgement

  • The great paper and official JAX implementation of dreamfields:

    @article{jain2021dreamfields,
        author = {Jain, Ajay and Mildenhall, Ben and Barron, Jonathan T. and Abbeel, Pieter and Poole, Ben},
        title = {Zero-Shot Text-Guided Object Generation with Dream Fields},
        journal = {arXiv},
        month = {December},
        year = {2021},
    }   
    
  • The GUI is developed with DearPyGui.

About

A colab friendly toolkit to generate 3D mesh model / video / nerf instance / multiview images of colourful 3D objects by text and image prompts input, based on dreamfields.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 56.7%
  • Cuda 31.3%
  • Jupyter Notebook 6.4%
  • C 3.9%
  • Shell 1.2%
  • C++ 0.5%