mohammad-shirkhani commited on
Commit
ccd4a42
1 Parent(s): 2b0afb0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +76 -6
README.md CHANGED
@@ -2,28 +2,97 @@
2
 
3
  ## Model Overview
4
 
5
- This model pipeline is designed to generate images from Persian text descriptions by translating the Persian text into English and then using a fine-tuned Stable Diffusion model to generate the corresponding image. The pipeline combines two models: a translation model (`mohammad-shirkhani/finetune_persian_to_english_mt5_base_summarize_on_celeba_hq`) and an image generation model (`ebrahim-k/Stable-Diffusion-1_5-FT-celeba_HQ_en`).
6
 
7
  ## Model Details
8
 
9
  ### Translation Model
10
  - **Model Name**: `mohammad-shirkhani/finetune_persian_to_english_mt5_base_summarize_on_celeba_hq`
11
  - **Architecture**: mT5
12
- - **Purpose**: This model is used to translate Persian text into English. It has been fine-tuned specifically on the CelebA-HQ dataset for summarization tasks, making it well-suited for translating descriptions of facial features.
13
 
14
  ### Image Generation Model
15
  - **Model Name**: `ebrahim-k/Stable-Diffusion-1_5-FT-celeba_HQ_en`
16
  - **Architecture**: Stable Diffusion 1.5
17
- - **Purpose**: This model generates high-quality images from the English text produced by the translation model. It has been fine-tuned on the CelebA-HQ dataset, making it particularly effective for generating realistic human faces based on text descriptions.
18
 
19
  ## Pipeline Description
20
 
21
- The pipeline works as follows:
22
 
23
  1. **Text Translation**: The Persian input text is translated into English using the mT5-based translation model.
24
- 2. **Image Generation**: The translated English text is then fed into the Stable Diffusion model to generate the corresponding image.
25
 
26
- ### Example Usage
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
27
 
28
  ```python
29
  from IPython.display import display
@@ -39,3 +108,4 @@ display(image)
39
  persian_text2 = "این مرد جذاب دارای موهای قهوه ای ، سوزش های جانبی ، دهان کمی باز و کیسه های زیر چشم است.این فرد جذاب دارای کیسه های زیر چشم ، سوزش های جانبی و دهان کمی باز است."
40
  image2 = persian_to_image_model(persian_text2)
41
  display(image2)
 
 
2
 
3
  ## Model Overview
4
 
5
+ This model pipeline is designed to generate images from Persian text descriptions. It works by first translating the Persian text into English and then using a fine-tuned Stable Diffusion model to generate the corresponding image. The pipeline combines two models: a translation model (`mohammad-shirkhani/finetune_persian_to_english_mt5_base_summarize_on_celeba_hq`) and an image generation model (`ebrahim-k/Stable-Diffusion-1_5-FT-celeba_HQ_en`).
6
 
7
  ## Model Details
8
 
9
  ### Translation Model
10
  - **Model Name**: `mohammad-shirkhani/finetune_persian_to_english_mt5_base_summarize_on_celeba_hq`
11
  - **Architecture**: mT5
12
+ - **Purpose**: This model translates Persian text into English. It has been fine-tuned on the CelebA-HQ dataset for summarization tasks, making it effective for translating descriptions of facial features.
13
 
14
  ### Image Generation Model
15
  - **Model Name**: `ebrahim-k/Stable-Diffusion-1_5-FT-celeba_HQ_en`
16
  - **Architecture**: Stable Diffusion 1.5
17
+ - **Purpose**: This model generates high-quality images from English text produced by the translation model. It has been fine-tuned on the CelebA-HQ dataset, which makes it particularly effective for generating realistic human faces based on text descriptions.
18
 
19
  ## Pipeline Description
20
 
21
+ The pipeline operates through the following steps:
22
 
23
  1. **Text Translation**: The Persian input text is translated into English using the mT5-based translation model.
24
+ 2. **Image Generation**: The translated English text is then used to generate the corresponding image with the Stable Diffusion model.
25
 
26
+ ### Code Implementation
27
+
28
+ #### 1. Install Required Libraries
29
+
30
+ ```python
31
+ !pip install transformers diffusers accelerate torch
32
+ ```
33
+ #### 2. Import Necessary Libraries
34
+
35
+ ```python
36
+ import torch
37
+ from transformers import MT5ForConditionalGeneration, T5Tokenizer
38
+ from diffusers import StableDiffusionPipeline
39
+ ```
40
+
41
+ #### 3. Set Device (GPU or CPU)
42
+ This code determines whether the pipeline should use a GPU (if available) or fallback to a CPU.
43
+
44
+ ```python
45
+ # Determine the device: GPU if available, otherwise CPU
46
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
47
+ print(f"Using device: {device}")
48
+ ```
49
+
50
+ #### 4. Define and Load the Persian-to-Image Model Class
51
+ The following class handles both translation and image generation tasks.
52
+
53
+ ```python
54
+ # Define the model class
55
+ class PersianToImageModel:
56
+ def __init__(self, translation_model_name, image_model_name, device):
57
+ self.device = device
58
+
59
+ # Load translation model
60
+ self.translation_model = MT5ForConditionalGeneration.from_pretrained(translation_model_name).to(device)
61
+ self.translation_tokenizer = T5Tokenizer.from_pretrained(translation_model_name)
62
+
63
+ # Load image generation model
64
+ self.image_model = StableDiffusionPipeline.from_pretrained(image_model_name).to(device)
65
+
66
+ def translate_text(self, persian_text):
67
+ input_ids = self.translation_tokenizer.encode(persian_text, return_tensors="pt").to(self.device)
68
+ translated_ids = self.translation_model.generate(input_ids, max_length=512, num_beams=4, early_stopping=True)
69
+ translated_text = self.translation_tokenizer.decode(translated_ids[0], skip_special_tokens=True)
70
+ return translated_text
71
+
72
+ def generate_image(self, english_text):
73
+ image = self.image_model(english_text).images[0]
74
+ return image
75
+
76
+ def __call__(self, persian_text):
77
+ # Translate Persian text to English
78
+ english_text = self.translate_text(persian_text)
79
+ print(f"Translated Text: {english_text}")
80
+
81
+ # Generate and return image
82
+ return self.generate_image(english_text)
83
+ ```
84
+ #### 5. Instantiate the Model
85
+ The following code snippet demonstrates how to instantiate the combined model.
86
+
87
+ ```python
88
+ # Instantiate the combined model
89
+ translation_model_name = 'mohammad-shirkhani/finetune_persian_to_english_mt5_base_summarize_on_celeba_hq'
90
+ image_model_name = 'ebrahim-k/Stable-Diffusion-1_5-FT-celeba_HQ_en'
91
+
92
+ persian_to_image_model = PersianToImageModel(translation_model_name, image_model_name, device)
93
+ ```
94
+ #### 6. Example Usage of the Model
95
+ Below are examples of how to use the model to generate images from Persian text.
96
 
97
  ```python
98
  from IPython.display import display
 
108
  persian_text2 = "این مرد جذاب دارای موهای قهوه ای ، سوزش های جانبی ، دهان کمی باز و کیسه های زیر چشم است.این فرد جذاب دارای کیسه های زیر چشم ، سوزش های جانبی و دهان کمی باز است."
109
  image2 = persian_to_image_model(persian_text2)
110
  display(image2)
111
+ ```