SMERF baking wrong results and missing js files in Webviewer

[Yochengliu]

@duckworthd Great work 👍
After baking process on our own dataset, the render results are somewhat wrong, with many messy black regions (simple meeting room). But the original_render results are quite good. Any debug suggestions :)

The Webviewer can't work without all js files available (listed in index.html). When will these missing files be open-source? We are looking forward to experience real-time smooth and good rendering provided by your excellent SMERF.

Many thanks.

[duckworthd]

Hi @Yochengliu,

Thanks for pointing out the missing Javascript files. You can find them here. I'll update the repo here soon.

As for why baked renders look poor while original renders look good: this suggests that an insufficient number of views were used for baking. Unless configured otherwise, the set of all training views will be passed to the teacher's proposal network to determine which voxels contain content for baking. I recommend verifying that these Config parameters are set to their default values in $CHECKPOINT_DIR/config.gin.

If you have verified that these parameters are left to their default values and quality is still poor, I recommend loading the alive_voxels.npy file stored under $CHECKPOINT_DIR/temp/sm_$SUBMODEL_ID/alive_voxels.npy in Jupyter and visualizing the high-resolution voxel grid. It is likely that this grid is incomplete.

NOTE: The temp/ directory is used as a cache at baking time. This folder should be manually deleted if any hyperparameters related to baking are changed!

Best,
Daniel

[Yochengliu]

@duckworthd Hi, thanks for your detailed suggestions, but the issue still exists. Below I upload debug information as much as possible. Please check them :)

To remove the influence of customized datasets, I reproduce SMERF on the Room scene of mipNeRF 360, with default setting (only modify some print setting and batch size). You can check config.gin. ZipNeRF is trained by the this repo. Below the results are from my reproduction. There are the same issues (wrong depth and baking results) as on our own dataset.

Issue 1: Wrong depth from SMERF

ZipNeRF depth	SMERF depth	RGB

ZipNeRF depth is obtained from teacher_rendering['distance_median'][1] in train.py. You can see that SMERF has some regions with wrong depths. They seem to happen at dark regions. The situation becomes much worse on our own dataset, i.e., the depths at ribbonlike and regular black regions are almost all wrong (depth below). I speculate that this may be because some backgound color masking tricks in SMERF, but I didn't find the codes's location. Or is it because that I use the ZipNeRF without camp? Please give some suggestions :)

Issue 2: Incomplete baking results.

rgb.render_path.mp4

You can see the Room baking results also have many black regions, just like the cases happen on our own dataset. I save training and baking log for your check. train & bake.log
BTW, on our own dataset, I make sure that all images are used for baking and the config is with default setting as the config block you referred to. I also checked alive_voxels.npy, the proportion of alive voxels is only 2%, too few. Please give some suggestions :)

Best,
Yocheng

[Yochengliu]

@duckworthd Hi, the baking issues are solved. The reason is that triplane_resolution and sparse_grid_resolution must meet a certain proportional relationship (4:1). Yet I set (1024, 384) on the above experiments.

I still have no idea why SMERF depths have some regular errors. Any suggestions :)

[duckworthd]

I'm glad to hear you found a solution! I'll add a check to ensure that triplane_resolution and sparse_grid_resolution are aligned.