|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
tags: |
|
- Kolors |
|
- text-to-image |
|
- stable-diffusion |
|
library_name: diffusers |
|
--- |
|
|
|
# Kolors-IP-Adapter-FaceID-Plus weights and inference code |
|
|
|
<div align="center" style="display: flex; justify-content: center; flex-wrap: wrap;"> |
|
<a href="https://github.com/Kwai-Kolors/Kolors"><img src="https://img.shields.io/static/v1?label=Kolors Code&message=Github&color=blue&logo=github-pages"></a>   |
|
<a href="https://kwai-kolors.github.io/"><img src="https://img.shields.io/static/v1?label=Team%20Page&message=Page&color=green"></a>   |
|
<a href="https://github.com/Kwai-Kolors/Kolors/blob/master/imgs/Kolors_paper.pdf"><img src="https://img.shields.io/static/v1?label=Tech Report&message=Arxiv:Kolors&color=red&logo=arxiv"></a>   |
|
<a href="https://kolors.kuaishou.com/"><img src="https://img.shields.io/static/v1?label=Official Website&message=Page&color=green"></a> |
|
</div> |
|
|
|
## <a name="Introduction"></a>📖 Introduction |
|
|
|
We provide Kolors-IP-Adapter-FaceID-Plus module weights and inference code based on [Kolors-Basemodel](https://huggingface.co/Kwai-Kolors/Kolors). Examples of Kolors-IP-Adapter-FaceID-Plus results are as follows: |
|
<img src="demo.png"> |
|
|
|
- Our Kolors-IP-Adapter-FaceID-Plus module is trained on a large-scale and high-quality face dataset. We use the face ID embeddings generated by [insightface](https://github.com/deepinsight/insightface) and the CLIP features of face area to keep the face ID and structure information. |
|
|
|
## <a name="Evaluation"></a>📊 Evaluation |
|
For evaluation, we constructed a test set consisting of over 200 reference images and text prompts. We invited several image experts to provide fair ratings for the generated results of different models. The experts assessed the generated images based on five criteria: visual appeal, text faithfulness, face similarity, facial aesthetics and overall satisfaction. Visual appeal and text faithfulness are used to measure the text-to-image generation capability, adhering to the evaluation standards of BaseModel. Meanwhile, face similarity and facial aesthetics are used to evaluate the performance of the proposed Kolors-IP-Adapter-FaceID-Plus. The results are summarized in the table below, where Kolors-IP-Adapter-FaceID-Plus outperforms SDXL-IP-Adapter-FaceID-Plus across all metrics. |
|
|
|
|
|
| Model | Average Text Faithfulness | Average Visual Appeal | Average Face Similarity | Average Facial Aesthetics | Average Overall Satisfaction | |
|
| :--------------: | :--------: | :--------: | :--------: | :--------: | :--------: | |
|
| SDXL-IP-Adapter-FaceID-Plus | 4.014 | 3.455 | 3.05 | 2.584 | 2.448 | |
|
| **Kolors-IP-Adapter-FaceID-Plus** | **4.235** | **4.374** | **4.415** | **3.887** | **3.561** | |
|
------ |
|
|
|
<img src="comparison.png"> |
|
|
|
<font color=gray style="font-size:12px"> *Kolors-IP-Adapter-FaceID-Plus employs chinese prompts, while SDXL-IP-Adapter-FaceID-Plus use english prompts.*</font> |
|
|
|
|
|
## <a name="Usage"></a>🛠️ Usage |
|
|
|
### Requirements |
|
|
|
The dependencies and installation are basically the same as the [Kolors-BaseModel](https://huggingface.co/Kwai-Kolors/Kolors). |
|
|
|
<br> |
|
|
|
1. Repository Cloning and Dependency Installation |
|
|
|
```bash |
|
apt-get install git-lfs |
|
git clone https://github.com/Kwai-Kolors/Kolors |
|
cd Kolors |
|
conda create --name kolors python=3.8 |
|
conda activate kolors |
|
pip install -r requirements.txt |
|
pip install insightface onnxruntime-gpu |
|
python3 setup.py install |
|
``` |
|
|
|
2. Weights download [link](https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus)： |
|
```bash |
|
huggingface-cli download --resume-download Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus --local-dir weights/Kolors-IP-Adapter-FaceID-Plus |
|
``` |
|
or |
|
```bash |
|
git lfs clone https://huggingface.co/Kwai-Kolors/Kolors-IP-Adapter-FaceID-Plus weights/Kolors-IP-Adapter-FaceID-Plus |
|
``` |
|
|
|
3. Inference： |
|
```bash |
|
python ipadapter_FaceID/sample_ipadapter_faceid_plus.py ./ipadapter_FaceID/assets/image1.png "穿着晚礼服，在星光下的晚宴场景中，烛光闪闪，整个场景洋溢着浪漫而奢华的氛围" |
|
|
|
python ipadapter_FaceID/sample_ipadapter_faceid_plus.py ./ipadapter_FaceID/assets/image2.png "西部牛仔，牛仔帽，荒野大镖客，背景是西部小镇，仙人掌，,日落余晖, 暖色调, 使用XT4胶片拍摄, 噪点, 晕影, 柯达胶卷，复古" |
|
|
|
``` |
|
|
|
### Acknowledgments |
|
- Thanks to [insightface](https://github.com/deepinsight/insightface) for the face representations. |
|
- Thanks to [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter) for the codebase. |
|
<br> |
