nerf_plus_plus/FORK.md

Why this fork

This is a fork from the original NeRF++ implementation. The original code has some isses I needed to fix in order to get the algorithm working and to be able to reproduce the authors work. Hopefully this version should be better explained, will the necessary scripts and without bugs to enable someone to train a dataset from start to finish without having delve into the code if do not want to.

How to install

Create a virtual env if you want to

pip3 -m venv env
source env/bin/activate

Install the needed dependencies

pip3 install -r requirements.txt

You will also need COLMAP

How to run

Dataset

First you need to create or find a dataset. A large set of images (at least 30, more if you want a 360 degree reconstruction). In order to maximise the quality of the reconstuction it is recommended to take pictures with the same illumination, from differents angles from the same subject (take a step between each picture, do not only rotate). Remember that higher quality pictures can always be resized later if needed with tools like ImageMagick (mogrify or convert)

mogrify -resize 800 *.jpg

Camera pose estimation

Then use the wrapper colmap colmap_runner/run_colmap.py

First change the two lines corresponding to input and output at the end of the script img_dir is your dataset and output_dir is the ouput. If your COLMAP binary is not located in /usr/local/bin/colmap also change it.

cd colmap_runner
python3 run_colmap.py

Then you will need to use the format_dataset.py script to transform the wrapper COLMAP binary format data into the datastructure required by NeRF++.

You again need to change the input_path and output_path.

python3 format_dataset.py

Before training you can visualise your camera poses using the camera_visualizer/visualize_cameras.py script.

Note: the cam_dict_norm.json file is the kai_cameras_normalized.json created by the colmap wrapper. Or you can use the vizu.py script.

Training the model

You then need to create the configuration file, copying the example in configs and tweaking the values to your need is recommended. Refer to the help inside ddp_train_nerf.py if you need to understand a parameter.

python3 ddp_train_nerf.py --config configs/lupo/training_lupo.txt

Your training should be running, if you want to visualise the training in real time using tensorboard you can use:

tensorboard --logdir logs --host localhost

And opening the given socket (ip:port) in your browser. It should be 0.0.0.0:6006