codestella
commited on
Commit
•
31baa93
1
Parent(s):
40c15ce
change README
Browse files
README.md
CHANGED
@@ -1,14 +1,34 @@
|
|
1 |
# Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation
|
|
|
2 |
[![Open in Streamlit](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://huggingface.co/spaces/flax-community/DietNerf-Demo) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing)
|
3 |
|
4 |
<p align="center"><img width="450" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126361638-4aad58e8-4efb-4fc5-bf78-f53d03799e1e.png"></p>
|
5 |
|
6 |
-
|
7 |
-
|
8 |
-
|
9 |
-
|
10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
11 |
|
|
|
|
|
|
|
12 |
|
13 |
## 🤗 Hugging Face Hub Repo URL:
|
14 |
We will also upload our project on the Hugging Face Hub Repository Also.
|
@@ -49,23 +69,6 @@ Our JAX/Flax implementation currently supports:
|
|
49 |
</tbody>
|
50 |
</table>
|
51 |
|
52 |
-
## 🤩 Demo
|
53 |
-
|
54 |
-
- Streamlit Space Demo
|
55 |
-
|
56 |
-
You can check our Streamlit Space demo on following site !
|
57 |
-
With any input camera pose, we can render the novel view synthesis.
|
58 |
-
[https://huggingface.co/spaces/flax-community/DietNerf-Demo](https://huggingface.co/spaces/flax-community/DietNerf-Demo)
|
59 |
-
|
60 |
-
- Colab Demo
|
61 |
-
|
62 |
-
Moreover, we prapare the colab ipython notebook for you.
|
63 |
-
You need colab pro account for running our model on the colab(For memory issue)
|
64 |
-
[https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing
|
65 |
-
](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing
|
66 |
-
)
|
67 |
-
|
68 |
-
|
69 |
## 💻 Installation
|
70 |
|
71 |
```bash
|
|
|
1 |
# Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation
|
2 |
+
|
3 |
[![Open in Streamlit](https://static.streamlit.io/badges/streamlit_badge_black_white.svg)](https://huggingface.co/spaces/flax-community/DietNerf-Demo) [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1etYeMTntw5mh3FvJv4Ubb7XUoTtt5J9G?usp=sharing)
|
4 |
|
5 |
<p align="center"><img width="450" alt="스크린샷 2021-07-04 오후 4 11 51" src="https://user-images.githubusercontent.com/77657524/126361638-4aad58e8-4efb-4fc5-bf78-f53d03799e1e.png"></p>
|
6 |
|
7 |
+
This project attempted to implement the paper **[Putting NeRF on a Diet](https://arxiv.org/abs/2104.00677)** (DietNeRF) in JAX/Flax.
|
8 |
+
DietNeRF is designed for rendering quality novel views in few-shot learning scheme, a task that vanilla NeRF (Neural Radiance Field) struggles.
|
9 |
+
To achieve this, the author coins **Semantic Consistency Loss** to supervise DietNeRF by prior knowledge from CLIP Vision Transformer. Such supervision enables DietNeRF to learn 3D scene reconstruction with CLIP's prior knowledge on 2D views.
|
10 |
+
|
11 |
+
Besides this repo, you can check our write-up and demo here:
|
12 |
+
- ✍️ **[Write-up in Notion](https://steep-cycle-f6b.notion.site/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745)**: more details of DietNeRF and our experiments
|
13 |
+
- ✨ **[Demo in Hugging Face Space](https://huggingface.co/spaces/flax-community/DietNerf-Demo)**: showcase our trained DietNeRFs by Streamlit
|
14 |
+
|
15 |
+
## 🤩 Demo
|
16 |
+
1. You can check out [our demo in Hugging Face Space](https://huggingface.co/spaces/flax-community/DietNerf-Demo)
|
17 |
+
2. Or you can set up our Streamlit demo locally (model checkpoints will be fetched automatically upon startup)
|
18 |
+
```shell
|
19 |
+
pip install -r requirements_demo.txt
|
20 |
+
streamlit run app.py
|
21 |
+
```
|
22 |
+
|
23 |
+
<p align="center"><img width="600" height="400" alt="Streamlit Demo" src="assets/space_demo.png"></p>
|
24 |
+
|
25 |
+
## ✨ Implementation
|
26 |
+
|
27 |
+
Our code is written in JAX/ Flax and mainly based upon [jaxnerf](https://github.com/google-research/google-research/tree/master/jaxnerf) from Google Research. The base code is highly optimized in GPU & TPU. For semantic consistency loss, we utilize pretrained CLIP Vision Transformer from [transformers](https://github.com/huggingface/transformers) library.
|
28 |
|
29 |
+
To learn more about DietNeRF, our experiments and implementation, you are highly recommended to check out our very detailed **[Notion write-up](https://www.notion.so/DietNeRF-Putting-NeRF-on-a-Diet-4aeddae95d054f1d91686f02bdb74745)**!
|
30 |
+
|
31 |
+
<p align="center"><img width="500" height="600" alt="스크린샷 2021-07-04 오후 4 11 51" src="assets/report_thumbnail.png"></p>
|
32 |
|
33 |
## 🤗 Hugging Face Hub Repo URL:
|
34 |
We will also upload our project on the Hugging Face Hub Repository Also.
|
|
|
69 |
</tbody>
|
70 |
</table>
|
71 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
72 |
## 💻 Installation
|
73 |
|
74 |
```bash
|