- 2023.03.20 Init repository.
- 2023.03.24 Release the arXiv version.
- 2023.04.05 🚀 FishDreamer has been accepted to 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW OmniCV2023).
- 2023.09.05 Release code.
This paper raises the new task of Fisheye Semantic Completion (FSC), where dense texture, structure, and semantics of a fisheye image are inferred even beyond the sensor field-of-view (FoV). Fisheye cameras have larger FoV than ordinary pinhole cameras, yet its unique special imaging model naturally leads to a blind area at the edge of the image plane. This is suboptimal for safety-critical applications since important perception tasks, such as semantic segmentation, become very challenging within the blind zone. Previous works considered the out-FoV outpainting and in-FoV segmentation separately. However, we observe that these two tasks are actually closely coupled. To jointly estimate the tightly intertwined complete fisheye image and scene semantics, we introduce the new FishDreamer which relies on successful ViTs enhanced with a novel Polar-aware Cross Attention module (PCA) to leverage dense context and guide semantically-consistent content generation while considering different polar distributions. In addition to the contribution of the novel task and architecture, we also derive Cityscapes-BF and KITTI360-BF datasets to facilitate training and evaluation of this new track. Our experiments demonstrate that the proposed FishDreamer outperforms methods solving each task in isolation and surpasses alternative approaches on the Fisheye Semantic Completion.
(Overview)
After code reorganization, we retrained FishDreamer on Cityscapes-BF:
| Method | PSNR | mIoU |
|---|---|---|
| FishDreamer (Paper) | 25.05 | 54.54 |
| FishDreamer (This Repo) | 25.21 | 54.88 |
This repo has been tested in the following environment:
torch == 1.9.0
pytorch-lightning == 1.8.6
mmcv-full == 1.5.2
To train FishDreamer, first set environment variable:
export USER=$(whoami)
export TORCH_HOME=$(pwd) && export PYTHONPATH=$(pwd)
Then start training:
python3 bin/train.py \
-cn FishDreamer \
data.batch_size=3 \
trainer.kwargs.max_epochs=70 \
data.train.transform_variant=resize
To output visual result, run:
python3 bin/predict.py \
model.path=$ModelPath \
indir=$DataPath \
outdir=$OutPath
You can eval your result by:
python3 bin/evaluate_predicts.py \
config=$ConfigPath \
datadir=$DataPath \
predictdir=$OutPath \
outpath=$OutMetricPath
The pretrained model and Cityscapes-BF dataset can be found there:
https://share.weiyun.com/7ShuPa2Y
For KITTI360-BF, please follow the instruction of FisheyeEX.
If you find our paper or repo useful, please consider citing our paper:
@inproceedings{shi2023fishdreamer,
title={FishDreamer: Towards Fisheye Semantic Completion via Unified Image Outpainting and Segmentation},
author={Shi, Hao and Li, Yu and Yang, Kailun and Zhang, Jiaming and Peng, Kunyu and Roitberg, Alina and Ye, Yaozu and Ni, Huajian and Wang, Kaiwei and Stiefelhagen, Rainer},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={6433--6443},
year={2023}
}This project would not have been possible without the following outstanding repositories:
Hao Shi, Yu Li
Feel free to contact me if you have additional questions or have interests in collaboration. Please drop me an email at haoshi@zju.edu.cn. =)