wanghaofan
commited on
Commit
β’
e45bee7
1
Parent(s):
8bef12f
Update README.md
Browse files
README.md
CHANGED
@@ -1,278 +1,282 @@
|
|
1 |
-
---
|
2 |
-
license: apache-2.0
|
3 |
-
|
4 |
-
|
5 |
-
|
6 |
-
|
7 |
-
|
8 |
-
|
9 |
-
|
10 |
-
|
11 |
-
|
12 |
-
[//]: # (
|
13 |
-
|
14 |
-
|
15 |
-
|
16 |
-
<
|
17 |
-
|
18 |
-
[
|
19 |
-
|
20 |
-
|
21 |
-
|
22 |
-
|
23 |
-
|
24 |
-
|
25 |
-
|
26 |
-
|
27 |
-
|
28 |
-
|
29 |
-
[//]: # ()
|
30 |
-
|
31 |
-
[//]: # ()
|
32 |
-
[//]: # ([//]: # (- **`2024/07/
|
33 |
-
[//]: # ()
|
34 |
-
[//]: # ([//]: # (- **`2024/07/
|
35 |
-
[//]: # (
|
36 |
-
[//]: # (
|
37 |
-
|
38 |
-
[//]: # (
|
39 |
-
|
40 |
-
[//]: # (- **`2024/
|
41 |
-
|
42 |
-
[//]: # ()
|
43 |
-
|
44 |
-
|
45 |
-
|
46 |
-
|
47 |
-
[//]: # (
|
48 |
-
|
49 |
-
[//]: # (- [
|
50 |
-
|
51 |
-
[//]: # (- [
|
52 |
-
|
53 |
-
[//]: # (- [ ]
|
54 |
-
|
55 |
-
|
56 |
-
|
57 |
-
|
58 |
-
|
59 |
-
##
|
60 |
-
|
61 |
-
|
62 |
-
|
63 |
-
|
64 |
-
|
65 |
-
|
66 |
-
|
|
67 |
-
|
68 |
-
|
69 |
-
|
70 |
-
|
71 |
-
|
72 |
-
|
73 |
-
|
74 |
-
|
75 |
-
|
76 |
-
|
77 |
-
|
78 |
-
|
79 |
-
|
80 |
-
|
81 |
-
|
82 |
-
</
|
83 |
-
|
84 |
-
|
85 |
-
|
86 |
-
|
87 |
-
|
88 |
-
|
89 |
-
|
90 |
-
|
91 |
-
|
92 |
-
|
93 |
-
|
94 |
-
|
95 |
-
#
|
96 |
-
|
97 |
-
|
98 |
-
|
99 |
-
|
100 |
-
|
101 |
-
|
102 |
-
|
103 |
-
|
104 |
-
|
105 |
-
|
106 |
-
|
107 |
-
|
108 |
-
|
109 |
-
|
110 |
-
|
111 |
-
|
112 |
-
|
113 |
-
|
114 |
-
|
115 |
-
|
116 |
-
|
117 |
-
|
118 |
-
|
119 |
-
|
120 |
-
|
121 |
-
|
122 |
-
|
123 |
-
|
124 |
-
|
125 |
-
from
|
126 |
-
|
127 |
-
|
128 |
-
|
129 |
-
|
130 |
-
|
131 |
-
|
132 |
-
|
133 |
-
|
134 |
-
|
135 |
-
|
136 |
-
|
137 |
-
|
138 |
-
|
139 |
-
|
140 |
-
|
141 |
-
|
142 |
-
|
143 |
-
|
144 |
-
|
145 |
-
|
146 |
-
|
147 |
-
|
148 |
-
|
149 |
-
|
150 |
-
|
151 |
-
|
152 |
-
|
153 |
-
|
154 |
-
|
155 |
-
|
156 |
-
|
157 |
-
|
158 |
-
|
159 |
-
|
160 |
-
|
161 |
-
|
162 |
-
|
163 |
-
|
164 |
-
|
165 |
-
|
166 |
-
|
167 |
-
|
168 |
-
|
169 |
-
|
170 |
-
|
171 |
-
|
172 |
-
|
173 |
-
|
174 |
-
|
175 |
-
|
176 |
-
|
177 |
-
|
178 |
-
|
179 |
-
|
180 |
-
|
181 |
-
|
182 |
-
|
183 |
-
|
184 |
-
|
185 |
-
|
186 |
-
|
187 |
-
|
188 |
-
|
189 |
-
|
190 |
-
|
191 |
-
|
192 |
-
|
193 |
-
|
194 |
-
|
195 |
-
|
196 |
-
|
197 |
-
|
198 |
-
|
199 |
-
|
200 |
-
|
201 |
-
|
202 |
-
|
203 |
-
|
204 |
-
|
205 |
-
|
206 |
-
|
207 |
-
|
208 |
-
|
209 |
-
|
210 |
-
|
211 |
-
|
212 |
-
|
213 |
-
|
214 |
-
|
215 |
-
|
216 |
-
|
217 |
-
|
218 |
-
|
219 |
-
|
220 |
-
|
221 |
-
|
222 |
-
|
223 |
-
|
224 |
-
|
225 |
-
|
226 |
-
|
227 |
-
|
228 |
-
|
229 |
-
|
230 |
-
|
231 |
-
|
232 |
-
|
233 |
-
|
234 |
-
|
235 |
-
|
236 |
-
|
237 |
-
|
238 |
-
<
|
239 |
-
|
240 |
-
|
241 |
-
|
242 |
-
<p align="center">
|
243 |
-
<img src="assets/
|
244 |
-
</p>
|
245 |
-
|
246 |
-
|
247 |
-
<
|
248 |
-
|
249 |
-
|
250 |
-
|
251 |
-
|
252 |
-
<
|
253 |
-
|
254 |
-
|
255 |
-
|
256 |
-
|
257 |
-
<
|
258 |
-
|
259 |
-
|
260 |
-
|
261 |
-
|
262 |
-
|
263 |
-
|
264 |
-
|
265 |
-
|
266 |
-
|
267 |
-
|
268 |
-
|
269 |
-
|
270 |
-
|
271 |
-
|
272 |
-
|
273 |
-
|
274 |
-
|
275 |
-
|
276 |
-
|
277 |
-
}
|
|
|
|
|
|
|
|
|
278 |
```
|
|
|
1 |
+
---
|
2 |
+
license: apache-2.0
|
3 |
+
language:
|
4 |
+
- en
|
5 |
+
library_name: diffusers
|
6 |
+
pipeline_tag: text-to-image
|
7 |
+
---
|
8 |
+
<div align="center">
|
9 |
+
|
10 |
+
[//]: # (<h1>CSGO: Content-Style Composition in Text-to-Image Generation</h1>)
|
11 |
+
|
12 |
+
[//]: # ()
|
13 |
+
[//]: # ([**Peng Xing**](https://github.com/xingp-ng)<sup>12*</sup> Β· [**Haofan Wang**](https://haofanwang.github.io/)<sup>1*</sup> Β· [**Yanpeng Sun**](https://scholar.google.com.hk/citations?user=a3FI8c4AAAAJ&hl=zh-CN&oi=ao/)<sup>2</sup> Β· [**Qixun Wang**](https://github.com/wangqixun)<sup>1</sup> Β· [**Xu Bai**](https://huggingface.co/baymin0220)<sup>1</sup> Β· [**Hao Ai**](https://github.com/aihao2000)<sup>13</sup> Β· [**Renyuan Huang**](https://github.com/DannHuang)<sup>14</sup> Β· [**Zechao Li**](https://zechao-li.github.io/)<sup>2β</sup>)
|
14 |
+
|
15 |
+
[//]: # ()
|
16 |
+
[//]: # (<sup>1</sup>InstantX Team Β· <sup>2</sup>Nanjing University of Science and Technology Β· <sup>3</sup>Beihang University Β· <sup>4</sup>Peking University)
|
17 |
+
|
18 |
+
[//]: # (<sup>*</sup>equal contributions, <sup>β</sup>corresponding authors)
|
19 |
+
|
20 |
+
<a href='https://csgo-gen.github.io/'><img src='https://img.shields.io/badge/Project-Page-green'></a>
|
21 |
+
<a href='https://arxiv.org/abs/2408.16766'><img src='https://img.shields.io/badge/Technique-Report-red'></a>
|
22 |
+
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-App-red)](https://huggingface.co/spaces/xingpng/CSGO/)
|
23 |
+
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue)](https://huggingface.co/spaces/InstantX/CSGO)
|
24 |
+
|
25 |
+
|
26 |
+
</div>
|
27 |
+
|
28 |
+
|
29 |
+
[//]: # (## Updates π₯)
|
30 |
+
|
31 |
+
[//]: # ()
|
32 |
+
[//]: # ([//]: # (- **`2024/07/19`**: β¨ We support ποΈ portrait video editing (aka v2v)! More to see [here](assets/docs/changelog/2024-07-19.md).))
|
33 |
+
[//]: # ()
|
34 |
+
[//]: # ([//]: # (- **`2024/07/17`**: π We support macOS with Apple Silicon, modified from [jeethu](https://github.com/jeethu)'s PR [#143](https://github.com/KwaiVGI/LivePortrait/pull/143).))
|
35 |
+
[//]: # ()
|
36 |
+
[//]: # ([//]: # (- **`2024/07/10`**: πͺ We support audio and video concatenating, driving video auto-cropping, and template making to protect privacy. More to see [here](assets/docs/changelog/2024-07-10.md).))
|
37 |
+
[//]: # ()
|
38 |
+
[//]: # ([//]: # (- **`2024/07/09`**: π€ We released the [HuggingFace Space](https://huggingface.co/spaces/KwaiVGI/liveportrait), thanks to the HF team and [Gradio](https://github.com/gradio-app/gradio)!))
|
39 |
+
[//]: # ([//]: # (Continuous updates, stay tuned!))
|
40 |
+
[//]: # (- **`2024/08/30`**: π We released the initial version of the inference code.)
|
41 |
+
|
42 |
+
[//]: # (- **`2024/08/30`**: π We released the technical report on [arXiv](https://arxiv.org/pdf/2408.16766))
|
43 |
+
|
44 |
+
[//]: # (- **`2024/07/15`**: π₯ We released the [homepage](https://csgo-gen.github.io).)
|
45 |
+
|
46 |
+
[//]: # ()
|
47 |
+
[//]: # (## Plan πͺ)
|
48 |
+
|
49 |
+
[//]: # (- [x] technical report)
|
50 |
+
|
51 |
+
[//]: # (- [x] inference code)
|
52 |
+
|
53 |
+
[//]: # (- [ ] pre-trained weight)
|
54 |
+
|
55 |
+
[//]: # (- [ ] IMAGStyle dataset)
|
56 |
+
|
57 |
+
[//]: # (- [ ] training code)
|
58 |
+
|
59 |
+
## Introduction π
|
60 |
+
This repo, named **CSGO**, contains the official PyTorch implementation of our paper [CSGO: Content-Style Composition in Text-to-Image Generation](https://arxiv.org/pdf/).
|
61 |
+
We are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) π.
|
62 |
+
|
63 |
+
## Detail β¨
|
64 |
+
We currently release two model weights.
|
65 |
+
|
66 |
+
| Mode | content token | style token | Other |
|
67 |
+
|:----------------:|:-----------:|:-----------:|:---------------------------------:|
|
68 |
+
| csgo.bin |4|16| - |
|
69 |
+
| csgo_4_32.bin |4|32| Deepspeed zero2 |
|
70 |
+
| csgo_4_32_v2.bin |4|32| Deepspeed zero2+more(coming soon) |
|
71 |
+
|
72 |
+
|
73 |
+
## Pipeline π»
|
74 |
+
<p align="center">
|
75 |
+
<img src="assets/image3_1.jpg">
|
76 |
+
</p>
|
77 |
+
|
78 |
+
## Capabilities π
|
79 |
+
|
80 |
+
π₯ Our CSGO achieves **image-driven style transfer, text-driven stylized synthesis, and text editing-driven stylized synthesis**.
|
81 |
+
|
82 |
+
π₯ For more results, visit our <a href="https://csgo-gen.github.io"><strong>homepage</strong></a> π₯
|
83 |
+
|
84 |
+
<p align="center">
|
85 |
+
<img src="assets/vis.jpg">
|
86 |
+
</p>
|
87 |
+
|
88 |
+
|
89 |
+
## Getting Started π
|
90 |
+
### 1. Clone the code and prepare the environment
|
91 |
+
```bash
|
92 |
+
git clone https://github.com/instantX-research/CSGO
|
93 |
+
cd CSGO
|
94 |
+
|
95 |
+
# create env using conda
|
96 |
+
conda create -n CSGO python=3.9
|
97 |
+
conda activate CSGO
|
98 |
+
|
99 |
+
# install dependencies with pip
|
100 |
+
# for Linux and Windows users
|
101 |
+
pip install -r requirements.txt
|
102 |
+
```
|
103 |
+
|
104 |
+
### 2. Download pretrained weights(coming soon)
|
105 |
+
|
106 |
+
The easiest way to download the pretrained weights is from HuggingFace:
|
107 |
+
```bash
|
108 |
+
# first, ensure git-lfs is installed, see: https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage
|
109 |
+
git lfs install
|
110 |
+
# clone and move the weights
|
111 |
+
git clone https://huggingface.co/InstantX/CSGO
|
112 |
+
```
|
113 |
+
Our method is fully compatible with [SDXL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0), [VAE](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix), [ControlNet](https://huggingface.co/TTPlanet/TTPLanet_SDXL_Controlnet_Tile_Realistic), and [Image Encoder](https://huggingface.co/h94/IP-Adapter/tree/main/sdxl_models/image_encoder).
|
114 |
+
Please download them and place them in the ./base_models folder.
|
115 |
+
|
116 |
+
tips:If you expect to load Controlnet directly using ControlNetPipeline as in CSGO, do the following:
|
117 |
+
```bash
|
118 |
+
git clone https://huggingface.co/TTPlanet/TTPLanet_SDXL_Controlnet_Tile_Realistic
|
119 |
+
mv TTPLanet_SDXL_Controlnet_Tile_Realistic/TTPLANET_Controlnet_Tile_realistic_v2_fp16.safetensors TTPLanet_SDXL_Controlnet_Tile_Realistic/diffusion_pytorch_model.safetensors
|
120 |
+
```
|
121 |
+
### 3. Inference π
|
122 |
+
|
123 |
+
```python
|
124 |
+
import torch
|
125 |
+
from ip_adapter.utils import resize_content
|
126 |
+
import numpy as np
|
127 |
+
from ip_adapter.utils import BLOCKS as BLOCKS
|
128 |
+
from ip_adapter.utils import controlnet_BLOCKS as controlnet_BLOCKS
|
129 |
+
from PIL import Image
|
130 |
+
from diffusers import (
|
131 |
+
AutoencoderKL,
|
132 |
+
ControlNetModel,
|
133 |
+
StableDiffusionXLControlNetPipeline,
|
134 |
+
|
135 |
+
)
|
136 |
+
from ip_adapter import CSGO
|
137 |
+
|
138 |
+
|
139 |
+
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
|
140 |
+
|
141 |
+
base_model_path = "./base_models/stable-diffusion-xl-base-1.0"
|
142 |
+
image_encoder_path = "./base_models/IP-Adapter/sdxl_models/image_encoder"
|
143 |
+
csgo_ckpt = "./CSGO/csgo.bin"
|
144 |
+
pretrained_vae_name_or_path ='./base_models/sdxl-vae-fp16-fix'
|
145 |
+
controlnet_path = "./base_models/TTPLanet_SDXL_Controlnet_Tile_Realistic"
|
146 |
+
weight_dtype = torch.float16
|
147 |
+
|
148 |
+
|
149 |
+
vae = AutoencoderKL.from_pretrained(pretrained_vae_name_or_path,torch_dtype=torch.float16)
|
150 |
+
controlnet = ControlNetModel.from_pretrained(controlnet_path, torch_dtype=torch.float16,use_safetensors=True)
|
151 |
+
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
|
152 |
+
base_model_path,
|
153 |
+
controlnet=controlnet,
|
154 |
+
torch_dtype=torch.float16,
|
155 |
+
add_watermarker=False,
|
156 |
+
vae=vae
|
157 |
+
)
|
158 |
+
pipe.enable_vae_tiling()
|
159 |
+
|
160 |
+
|
161 |
+
target_content_blocks = BLOCKS['content']
|
162 |
+
target_style_blocks = BLOCKS['style']
|
163 |
+
controlnet_target_content_blocks = controlnet_BLOCKS['content']
|
164 |
+
controlnet_target_style_blocks = controlnet_BLOCKS['style']
|
165 |
+
|
166 |
+
csgo = CSGO(pipe, image_encoder_path, csgo_ckpt, device, num_content_tokens=4,num_style_tokens=32,
|
167 |
+
target_content_blocks=target_content_blocks, target_style_blocks=target_style_blocks,controlnet_adapter=True,
|
168 |
+
controlnet_target_content_blocks=controlnet_target_content_blocks,
|
169 |
+
controlnet_target_style_blocks=controlnet_target_style_blocks,
|
170 |
+
content_model_resampler=True,
|
171 |
+
style_model_resampler=True,
|
172 |
+
|
173 |
+
)
|
174 |
+
|
175 |
+
style_name = 'img_1.png'
|
176 |
+
content_name = 'img_0.png'
|
177 |
+
style_image = Image.open("../assets/{}".format(style_name)).convert('RGB')
|
178 |
+
content_image = Image.open('../assets/{}'.format(content_name)).convert('RGB')
|
179 |
+
|
180 |
+
caption ='a small house with a sheep statue on top of it'
|
181 |
+
|
182 |
+
num_sample=4
|
183 |
+
|
184 |
+
#image-driven style transfer
|
185 |
+
images = csgo.generate(pil_content_image= content_image, pil_style_image=style_image,
|
186 |
+
prompt=caption,
|
187 |
+
negative_prompt= "text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry",
|
188 |
+
content_scale=1.0,
|
189 |
+
style_scale=1.0,
|
190 |
+
guidance_scale=10,
|
191 |
+
num_images_per_prompt=num_sample,
|
192 |
+
num_samples=1,
|
193 |
+
num_inference_steps=50,
|
194 |
+
seed=42,
|
195 |
+
image=content_image.convert('RGB'),
|
196 |
+
controlnet_conditioning_scale=0.6,
|
197 |
+
)
|
198 |
+
|
199 |
+
#text editing-driven stylized synthesis
|
200 |
+
caption='a small house'
|
201 |
+
images = csgo.generate(pil_content_image= content_image, pil_style_image=style_image,
|
202 |
+
prompt=caption,
|
203 |
+
negative_prompt= "text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry",
|
204 |
+
content_scale=1.0,
|
205 |
+
style_scale=1.0,
|
206 |
+
guidance_scale=10,
|
207 |
+
num_images_per_prompt=num_sample,
|
208 |
+
num_samples=1,
|
209 |
+
num_inference_steps=50,
|
210 |
+
seed=42,
|
211 |
+
image=content_image.convert('RGB'),
|
212 |
+
controlnet_conditioning_scale=0.4,
|
213 |
+
)
|
214 |
+
|
215 |
+
#text-driven stylized synthesis
|
216 |
+
caption='a cat'
|
217 |
+
#If the content image still interferes with the generated results, set the content image to an empty image.
|
218 |
+
# content_image =Image.fromarray(np.zeros((content_image.size[0],content_image.size[1], 3), dtype=np.uint8)).convert('RGB')
|
219 |
+
|
220 |
+
images = csgo.generate(pil_content_image= content_image, pil_style_image=style_image,
|
221 |
+
prompt=caption,
|
222 |
+
negative_prompt= "text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry",
|
223 |
+
content_scale=1.0,
|
224 |
+
style_scale=1.0,
|
225 |
+
guidance_scale=10,
|
226 |
+
num_images_per_prompt=num_sample,
|
227 |
+
num_samples=1,
|
228 |
+
num_inference_steps=50,
|
229 |
+
seed=42,
|
230 |
+
image=content_image.convert('RGB'),
|
231 |
+
controlnet_conditioning_scale=0.01,
|
232 |
+
)
|
233 |
+
```
|
234 |
+
|
235 |
+
## Demos
|
236 |
+
<p align="center">
|
237 |
+
<br>
|
238 |
+
π₯ For more results, visit our <a href="https://csgo-gen.github.io"><strong>homepage</strong></a> π₯
|
239 |
+
</p>
|
240 |
+
|
241 |
+
### Content-Style Composition
|
242 |
+
<p align="center">
|
243 |
+
<img src="assets/page1.png">
|
244 |
+
</p>
|
245 |
+
|
246 |
+
<p align="center">
|
247 |
+
<img src="assets/page4.png">
|
248 |
+
</p>
|
249 |
+
|
250 |
+
### Cycle Translation
|
251 |
+
<p align="center">
|
252 |
+
<img src="assets/page8.png">
|
253 |
+
</p>
|
254 |
+
|
255 |
+
### Text-Driven Style Synthesis
|
256 |
+
<p align="center">
|
257 |
+
<img src="assets/page10.png">
|
258 |
+
</p>
|
259 |
+
|
260 |
+
### Text Editing-Driven Style Synthesis
|
261 |
+
<p align="center">
|
262 |
+
<img src="assets/page11.jpg">
|
263 |
+
</p>
|
264 |
+
|
265 |
+
## Star History
|
266 |
+
[![Star History Chart](https://api.star-history.com/svg?repos=instantX-research/CSGO&type=Date)](https://star-history.com/#instantX-research/CSGO&Date)
|
267 |
+
|
268 |
+
|
269 |
+
|
270 |
+
## Acknowledgements
|
271 |
+
This project is developed by InstantX Team, all copyright reserved.
|
272 |
+
|
273 |
+
## Citation π
|
274 |
+
If you find CSGO useful for your research, welcome to π this repo and cite our work using the following BibTeX:
|
275 |
+
```bibtex
|
276 |
+
@article{xing2024csgo,
|
277 |
+
title={CSGO: Content-Style Composition in Text-to-Image Generation},
|
278 |
+
author={Peng Xing and Haofan Wang and Yanpeng Sun and Qixun Wang and Xu Bai and Hao Ai and Renyuan Huang and Zechao Li},
|
279 |
+
year={2024},
|
280 |
+
journal = {arXiv 2408.16766},
|
281 |
+
}
|
282 |
```
|