{ "cells": [ { "cell_type": "markdown", "id": "cd47645b-3a64-433e-89a0-25fa30217a2c", "metadata": {}, "source": [ "## 説明" ] }, { "cell_type": "markdown", "id": "06077106-1f0b-406e-8c82-fb127574bebe", "metadata": {}, "source": [ "Dreambooth-Loraの学習をPeperspaceで動かす為のNotebook \n", "本家sd-scripts(https://github.com/kohya-ss/sd-scripts) \n", "\n", "以下ソースを参考に作成してるで。 \n", "sd-scripts(https://github.com/kohya-ss/sd-scripts) \n", "colab用kohya-trainer(https://github.com/Linaqruf/kohya-trainer) \n", "Peperspace用webui(https://github.com/Engineer-of-Stuff/stable-diffusion-paperspace) \n", "\n", "学習素材と正規化画像はあらかじめstorageかtmpにアップしてな。 \n", "永続Storageがある事と一部ターミナル使う前提になってるから無課金では動かんかもしれんで " ] }, { "cell_type": "markdown", "id": "b07c14b6-b67f-41f3-9a1b-02730b32becf", "metadata": {}, "source": [ "既知の不具合 \n", "学習実行時に以下の警告メッセージが表示されるで \n", "解決策わかったら教えてください \n", "- 「--use_8bit_adam 」を有効にすると別パッケージから参照の警告メッセージが表示される。(多分bitsandbytesのパスがおかしい) \n", "- 「Could not load dynamic library 'libnvinfer_plugin.so.7';」の警告メッセージが表示される。(libnvinfer_plugin.so.7がpython3.9に無い？) \n", "- 「Unable to register cuBLAS factory 」の警告メッセージが表示される。(xpaformer入れる為にcudnnのバージョン下げてるのが怪しい） \n" ] }, { "cell_type": "markdown", "id": "4eb1d725-e55c-41e9-8574-da6dfb641ff0", "metadata": { "tags": [] }, "source": [ "## 1.SETTING" ] }, { "cell_type": "markdown", "id": "d33d8e53-af14-4033-9ba2-0c4044541763", "metadata": { "tags": [] }, "source": [ "# 1-0 設定値保存\n", "仮想マシン起動時毎回実行する" ] }, { "cell_type": "code", "execution_count": null, "id": "e90d2a8f-f497-421d-9a7e-3921caff41c4", "metadata": { "tags": [] }, "outputs": [], "source": [ "#リポジトリ　永続ストレー、一時領域ジシンボリックリンク作成\n", "repo_dir = '/notebooks' \n", "!ln -s /storage/ /notebooks/\n", "!ln -s /tmp/ /notebooks/\n", "\n", "#その他設定値\n", "activate_xformers = True # Enables the xformers optimizations using pre-built wheels.\n", "\n", "%store repo_dir activate_xformers" ] }, { "cell_type": "markdown", "id": "8f1b1a2f-d1ab-4b84-87dc-c83190bf506d", "metadata": { "tags": [] }, "source": [ "# 1-1.Git Clone\n", "導入時 更新時" ] }, { "cell_type": "code", "execution_count": null, "id": "114fc353-213a-4d91-afce-f733ba5a9de2", "metadata": { "tags": [] }, "outputs": [], "source": [ "%cd {repo_dir}\n", "\n", "import os\n", "\n", "def clone_kohya_sd_scripts():\n", " # Check if the directory already exists\n", " if os.path.isdir('/notebooks/sd-scripts'):\n", " %cd /notebooks/sd-scripts\n", " print(\"This folder already exists, will do a !git pull instead\\n\")\n", " !git pull\n", " else:\n", " !git clone https://github.com/kohya-ss/sd-scripts\n", "\n", "# Clone or update the Kohya Trainer repository\n", "clone_kohya_sd_scripts()" ] }, { "cell_type": "markdown", "id": "b52dd301-0ed2-4ae4-911e-643d39c0f1bf", "metadata": { "tags": [] }, "source": [ "# 1-2.Install and Setting\n", "仮想マシン起動時毎回実行する" ] }, { "cell_type": "code", "execution_count": null, "id": "85954927-9497-4a2c-995a-dc4e6ba4b16c", "metadata": {}, "outputs": [], "source": [ "%store -r repo_dir activate_xformers\n", "\n", "appDir = f'{repo_dir}/sd-scripts'\n", "%cd {appDir}\n", "\n", "!pip install --upgrade pip\n", "!pip install --upgrade -r requirements.txt\n", "!pip uninstall -y torch torchvision torchaudio # Remove existing pytorch install.\n", "!pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113 # Install pytorch for cuda 11.3\n", "\n", "import os\n", "if activate_xformers:\n", " print('Installing xformers...')\n", " import subprocess\n", " def download_release(url):\n", " binary = 'xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl' # have to save the binary as a specific name that pip likes\n", " tmp_dir = subprocess.check_output(['mktemp', '-d']).decode('ascii').strip('\\n')\n", " !wget \"{url}\" -O \"{tmp_dir}/{binary}\"\n", " return os.path.join(tmp_dir, binary)\n", "\n", " # Set up pip packages\n", " s = subprocess.getoutput('nvidia-smi')\n", " if 'A4000' in s:\n", " xformers_whl = download_release('https://github.com/Cyberes/xformers-compiled/releases/download/A4000-Oct-28-2022/a4000-xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl')\n", " elif 'A5000' in s:\n", " xformers_whl = download_release('https://github.com/Cyberes/xformers-compiled/releases/download/A5000-Nov-1-2022/a5000-xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl')\n", " elif 'A6000' in s:\n", " xformers_whl = download_release('https://github.com/Cyberes/xformers-compiled/releases/download/A6000-Nov-1-2022/a6000-xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl')\n", " elif 'P5000' in s:\n", " xformers_whl = download_release('https://github.com/Cyberes/xformers-compiled/releases/download/P5000-Nov-1-2022/p5000-xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl')\n", " elif 'RTX 4000' in s:\n", " xformers_whl = download_release('https://github.com/Cyberes/xformers-compiled/releases/download/RTX-4000-Nov-1-2022/rtx4000-xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl')\n", " elif 'RTX 5000' in s:\n", " xformers_whl = download_release('https://github.com/Cyberes/xformers-compiled/releases/download/RTX-5000-Nov-1-2022/rtx5000-xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl')\n", " elif 'A100' in s:\n", " xformers_whl = download_release('https://github.com/Cyberes/xformers-compiled/releases/download/A100-Nov-1-2022/a100-xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl')\n", " elif 'M4000' in s:\n", " print('xformers for M4000 hasn\\'t been built yet.')\n", " # xformers_whl = download_release('https://github.com/Cyberes/xformers-compiled/releases/download/A100-Nov-1-2022/a100-xformers-0.0.14.dev0-cp39-cp39-linux_x86_64.whl')\n", " else:\n", " print('GPU not matched to xformers binary so a one-size-fits-all binary was installed. If you have any issues, please build xformers using the Tools block below.')\n", " xformers_whl = download_release('https://raw.githubusercontent.com/Cyberes/xformers-compiled/main/various/xformers-0.0.14.dev0-cp37-cp37m-linux_x86_64.whl')\n", " !pip install --force-reinstall \"{xformers_whl}\"" ] }, { "cell_type": "markdown", "id": "7076d849-dd45-491d-9e6c-473ed1bdbc6e", "metadata": { "tags": [] }, "source": [ "# 1-3.Accelerate config作成 \n", "導入時初回のみターミナルから実行する。 \n", "対話型で選択肢に回答する形式なのでターミナルから実行 \n", " cd /notebooks/sd-scripts \n", " accelerate config \n", "質問回答後下記メッセージが出たら完了 \n", "accelerate configuration saved at /root/.cache/huggingface/accelerate/default_config.yaml " ] }, { "cell_type": "markdown", "id": "bf57b0ce-882c-415c-9d80-a923a7026124", "metadata": { "tags": [] }, "source": [ "# 1-4.accelerate configファイルをsd-scriptsディレクトリにコピーする\n", "導入時初回のみ実行する \n", "1.3で作ったコンフィグファイルを永続ストレージにコピーする" ] }, { "cell_type": "code", "execution_count": null, "id": "9bbd243b-75d3-4492-9bac-196faf55ee97", "metadata": {}, "outputs": [], "source": [ "!cp -r /root/.cache/huggingface/accelerate/ /notebooks/sd-scripts/accelerate/" ] }, { "cell_type": "markdown", "id": "648e5671-b153-4549-96c8-88afb204b3e4", "metadata": {}, "source": [ "## RUNNING" ] }, { "cell_type": "markdown", "id": "730c4e1b-308f-438c-95a8-e26c671055f5", "metadata": { "tags": [] }, "source": [ "# 2-0.Dataset Setting" ] }, { "cell_type": "code", "execution_count": null, "id": "569d34de-15db-46f4-9af4-7acbfc98e5c8", "metadata": {}, "outputs": [], "source": [ "#起動時。学習素材変更時実行する\n", "#Learning checkpointName .ckpt\n", "model_file_name = \"wd-1-4-anime_e1.ckpt\" #@param {'type' : 'string'} \n", "\n", "model_storage_dir =\"/notebooks/storage/models\"\n", "\n", "model_file_path = f\"{model_storage_dir}/{model_file_name}\"\n", "\n", "# ===================================================================================================\n", "# 正規化データ クラス名\n", "reg_count = 1 #@param {type: \"integer\"}\n", "reg_class =\"girl\" #@param {type: \"string\"}\n", "\n", "#学習元データ トークン(インスタンス)名、クラス名\n", "train_count = 20 #@param {type: \"integer\"} 1epoch=学習素材 × カウント数のステップを回す（webui版で10の部分）\n", "train_token = \"nahida\" #@param {type: \"string\"}\n", "train_class = \"girl\" #@param {type: \"string\"}\n", "\n", "storage_train_dir = \"/notebooks/storage/atelier/dataset/1024_nahidav3\" #@param {type: \"string\"}\n", "storage_class_dir = \"/notebooks/storage/atelier/dataset/Classification\" #@param {type: \"string\"}\n", "\n", "# ===================================================================================================\n", "# Save variables to Jupiter's temp storage so we can access it even if the kernel restarts.\n", "%store model_storage_dir model_file_path reg_count reg_class train_count train_token train_class storage_train_dir storage_class_dir" ] }, { "cell_type": "markdown", "id": "b017ace2-cd08-427a-89d5-28ad8f7dbfc5", "metadata": {}, "source": [ "# 2-1 Dreambooth フォルダ削除 \n", "学習結果を消すので注意 \n", "※学習画像データは消さない " ] }, { "cell_type": "code", "execution_count": null, "id": "b5aa4b00-b54d-4f5f-a0da-5153a86c1f87", "metadata": {}, "outputs": [], "source": [ "# 学習結果を消すので注意\n", "%cd /notebooks/\n", "\n", "import os\n", "\n", "def delete_dreambooth_folder():\n", " # Check if the directory already exists\n", " if os.path.isdir('/notebooks/dreambooth'):\n", " %rm -r /notebooks/dreambooth\n", " print(\"dreambooth dataset folder deleted done!!\")\n", " else:\n", " print(\"dreambooth dataset folder none\")\n", "\n", "# Delete Dreamboothe Dataset folder\n", "delete_dreambooth_folder()" ] }, { "cell_type": "code", "execution_count": null, "id": "5a27b535-ccd3-40cc-86b0-cab5cd3d63b7", "metadata": {}, "outputs": [], "source": [ "# 起動時、学習素材変更時実行する\n", "#@title Create train and reg folder based on description above\n", "%store -r model_storage_dir model_file_path reg_count reg_class train_count train_token train_class storage_train_dir storage_class_dir\n", "\n", "# Import the os and shutil modules\n", "import os\n", "import shutil\n", "\n", "# Change the current working directory to /content\n", "%cd /notebooks\n", "\n", "# Define the dreambooth_directory variable\n", "dreambooth_directory = \"/notebooks/dreambooth\"\n", "\n", "# Check if the dreambooth directory already exists\n", "if os.path.isdir(dreambooth_directory):\n", " # If the directory exists, do nothing\n", " pass\n", "else:\n", " # If the directory does not exist, create it\n", " os.mkdir(dreambooth_directory)\n", "\n", "#@markdown ### Define the reg_folder variable\n", "#reg_count = 1 #@param {type: \"integer\"}\n", "#reg_class =\"kasakai_hikaru\" #@param {type: \"string\"}\n", "reg_folder = str(reg_count) + \"_\" + reg_class\n", "\n", "# Define the reg_directory variable\n", "reg_directory = f\"{dreambooth_directory}/reg_{reg_class}\"\n", "\n", "# Check if the reg directory already exists\n", "if os.path.isdir(reg_directory):\n", " # If the directory exists, do nothing\n", " pass\n", "else:\n", " # If the directory does not exist, create it\n", " os.mkdir(reg_directory)\n", "\n", "# Define the reg_folder_directory variable\n", "reg_folder_directory = f\"{reg_directory}/{reg_folder}\"\n", "\n", "# Check if the reg_folder directory already exists\n", "if os.path.isdir(reg_folder_directory):\n", " # If the directory exists, do nothing\n", " pass\n", "else:\n", " # If the directory does not exist, create it\n", " #os.mkdir(reg_folder_directory)\n", " os.symlink(storage_class_dir, reg_folder_directory)\n", "\n", "#@markdown ### Define the train_folder variable\n", "#train_count = 3300 #@param {type: \"integer\"}\n", "#train_token = \"sls\" #@param {type: \"string\"}\n", "#train_class = \"kasakai_hikaru\" #@param {type: \"string\"}\n", "train_folder = str(train_count) + \"_\" + train_token + \"_\" + train_class\n", "\n", "# Define the train_directory variable\n", "train_directory = f\"{dreambooth_directory}/train_{train_class}\"\n", "\n", "# Check if the train directory already exists\n", "if os.path.isdir(train_directory):\n", " # If the directory exists, do nothing\n", " pass\n", "else:\n", " # If the directory does not exist, create it\n", " os.mkdir(train_directory)\n", " \n", "# Define the train_folder_directory variable\n", "train_folder_directory = f\"{train_directory}/{train_folder}\"\n", "\n", "# Check if the train_folder directory already exists\n", "if os.path.isdir(train_folder_directory):\n", " # If the directory exists, do nothing\n", " pass\n", "else:\n", " # If the directory does not exist, create it\n", " #os.mkdir(train_folder_directory)\n", " os.symlink(storage_train_dir, train_folder_directory)\n", " \n", "%store train_directory train_folder_directory reg_directory reg_folder_directory" ] }, { "cell_type": "markdown", "id": "1bb54f2f-66a8-4a4f-80c3-a38f6eacd9fe", "metadata": {}, "source": [ "# Lora Train Start\n", "Dreambooth-Loraの学習を実行する \n", "引数の詳細情報は「sd-scripts/train_network.py」のソースを参照　　" ] }, { "cell_type": "code", "execution_count": null, "id": "e0b13039-fec7-4002-998a-64429599baca", "metadata": {}, "outputs": [], "source": [ "#@title Training begin Lora\n", "%store -r model_storage_dir model_file_path train_directory reg_directory \n", "accelerate_config = \"/notebooks/sd-scripts/accelerate/default_config.yaml\"\n", "num_cpu_threads_per_process = 8 #@param {'type':'integer'}\n", "pre_trained_model_path =model_file_path #@param {'type':'string'}\n", "train_data_dir = train_directory #@param {'type':'string'}\n", "reg_data_dir = reg_directory #@param {'type':'string'}\n", "\n", "output_dir =\"/notebooks/dreambooth\" #@param {'type':'string'}\n", "train_batch_size = 6 #@param {type: \"slider\", min: 1, max: 10}\n", "resolution = \"768,768\" #@param [\"512,512\", \"768,768\"] {allow-input: false}\n", "learning_rate =\"1e-4\" #@param {'type':'string'}\n", "mixed_precision = \"bf16\" #@param [\"fp16\", \"bf16\"] {allow-input: false}\n", "max_train_steps = 3200 #@param {'type':'integer'}\n", "save_precision = \"fp16\" #@param [\"float\", \"fp16\", \"bf16\"] {allow-input: false}\n", "save_every_n_epochs = 5 #@param {'type':'integer'}\n", "use_network_module = \"networks.lora\" #@param {'type':'string'}\n", "caption_extension =\".txt\" #@param {'type':'string'}\n", "#resme_path ='/notebooks/dreambooth/last-state' #学習再開する場合フォルダを指定する\n", "resme_path ='' #学習再開する場合フォルダを指定する\n", "resume = f'--resume={resme_path}' if resme_path else '' #@param {'type':'string'}\n", "max_token_length = 225 #@param {'type':'integer'}\n", "\n", "%cd /notebooks/sd-scripts/\n", "!accelerate launch --config_file {accelerate_config} --num_cpu_threads_per_process {num_cpu_threads_per_process} train_network.py \\\n", " --v2 \\\n", " --max_token_length={max_token_length} \\\n", " --pretrained_model_name_or_path={pre_trained_model_path} \\\n", " --train_data_dir={train_data_dir} \\\n", " --reg_data_dir={reg_data_dir} \\\n", " --output_dir={output_dir} \\\n", " --prior_loss_weight=1.0 \\\n", " --resolution={resolution} \\\n", " --train_batch_size={train_batch_size}\\\n", " --learning_rate={learning_rate}\\\n", " --max_train_steps={max_train_steps} \\\n", " --use_8bit_adam \\\n", " --xformers \\\n", " --cache_latents \\\n", " --mixed_precision={mixed_precision} \\\n", " --gradient_checkpointing \\\n", " --save_every_n_epochs={save_every_n_epochs} \\\n", " --enable_bucket \\\n", " --network_module={use_network_module} \\\n", " --caption_extension={caption_extension} \\\n", " --save_state {resume}" ] }, { "cell_type": "markdown", "id": "371b43fe-9293-4f1e-a026-72b3f94df6e2", "metadata": { "jp-MarkdownHeadingCollapsed": true, "tags": [] }, "source": [ "# 3.Dataset Labeling （おまけ）\n", "FineTune用 Lora学習には使わない。WD14taggerは使うかも" ] }, { "cell_type": "code", "execution_count": null, "id": "05528d73-883e-4365-a1e7-d82cf61eee6e", "metadata": {}, "outputs": [], "source": [ "# 3-1.BLIPでキャプションファイル(.caption)を学習素材と同じ場所に作成する\n", "%store -r storage_train_dir\n", "%cd /notebooks/sd-scripts/\n", "batch_size = 8 #@param {'type':'integer'}\n", "\n", "!python finetune/make_captions.py --batch_size {batch_size} {storage_train_dir}" ] }, { "cell_type": "code", "execution_count": null, "id": "7c58ed68-fc08-4d1a-9630-d14c6b0b3db8", "metadata": {}, "outputs": [], "source": [ "# 3-2 WD1.4 taggerでタグテキスト(.txt)を学習素材と同じ場所に作成する\n", "#@title Start WD 1.4 Tagger\n", "%store -r storage_train_dir\n", "%cd /notebooks/sd-scripts/\n", "\n", "batch_size = 8 #@param {'type':'integer'}\n", "caption_extension = \".txt\" #@param [\".txt\",\".caption\"]\n", "\n", "!python finetune/tag_images_by_wd14_tagger.py \\\n", " {storage_train_dir} \\\n", " --batch_size {batch_size} \\\n", " --caption_extension {caption_extension}" ] }, { "cell_type": "code", "execution_count": null, "id": "b4aef070-b6f2-4346-a553-888bb4404e83", "metadata": {}, "outputs": [], "source": [ "# 3-3 キャプションとタグを結合して1つのファイルにまとめる(meta_clean.json作成)\n", "#@title Create meta_clean.json \n", "# Change the working directory\n", "%store -r storage_train_dir\n", "%cd /notebooks/sd-scripts/\n", "\n", "#@markdown ### Define Parameters\n", "meta_cap_dd = \"/notebooks/dreambooth/meta_cap_dd.json\" \n", "meta_cap = \"/notebooks/dreambooth/meta_cap.json\" \n", "meta_clean = \"/notebooks/dreambooth/meta_clean.json\" #@param {'type':'string'}\n", "\n", "# Check if the train_data_dir exists and is a directory\n", "if os.path.isdir(storage_train_dir):\n", " # Check if there are any .caption files in the train_data_dir\n", " if any(file.endswith('.caption') for file in os.listdir(storage_train_dir)):\n", " # Create meta_cap.json from captions\n", " !python finetune/merge_captions_to_metadata.py \\\n", " {storage_train_dir} \\\n", " {meta_cap}\n", "\n", " # Check if there are any .txtn files in the train_data_dir\n", " if any(file.endswith('.txt') for file in os.listdir(storage_train_dir)):\n", " # Create meta_cap_dd.json from tags\n", " !python finetune/merge_dd_tags_to_metadata.py \\\n", " {storage_train_dir} \\\n", " {meta_cap_dd}\n", "else:\n", " print(\"train_data_dir does not exist or is not a directory.\")\n", "\n", "# Merge meta_cap.json to meta_cap_dd.json\n", "if os.path.exists(meta_cap) and os.path.exists(meta_cap_dd):\n", " !python finetune/merge_dd_tags_to_metadata.py \\\n", " {storage_train_dir} \\\n", " --in_json {meta_cap} \\\n", " {meta_cap_dd}\n", "\n", "# Clean meta_cap_dd.json and store it to meta_clean.json\n", "if os.path.exists(meta_cap_dd):\n", " # Clean captions and tags in meta_cap_dd.json and store the result in meta_clean.json\n", " !python finetune/clean_captions_and_tags.py \\\n", " {meta_cap_dd} \\\n", " {meta_clean}\n", "elif os.path.exists(meta_cap):\n", " # If meta_cap_dd.json does not exist, clean meta_cap.json and store the result in meta_clean.json\n", " !python finetune/clean_captions_and_tags.py \\\n", " {meta_cap} \\\n", " {meta_clean}\n" ] }, { "cell_type": "code", "execution_count": null, "id": "067d56ab-cc17-4acf-af5c-3913c56c722a", "metadata": {}, "outputs": [], "source": [ "# 3-4 latentsの事前取得\n", "#@title Aspect Ratio Bucketing\n", "%store -r storage_train_dir model_file_path\n", "\n", "# Change working directory\n", "%cd /notebooks/sd-scripts/\n", "\n", "#@markdown ### Define parameters\n", "\n", "#model_dir = \"runwayml/stable-diffusion-v1-5\" #@param {'type' : 'string'} \n", "model_dir = model_file_path #@param {'type' : 'string'} \n", "batch_size = 4 #@param {'type':'integer'}\n", "max_resolution = \"768,768\" #@param [\"512,512\", \"768,768\"] {allow-input: false}\n", "mixed_precision = \"bf16\" #@param [\"no\", \"fp16\", \"bf16\"] {allow-input: false}\n", "meta_clean = \"/notebooks/dreambooth/meta_clean.json\"\n", "meta_lat = \"/notebooks/dreambooth/meta_lat.json\"\n", "\n", "\n", "# Run script to prepare buckets and latents\n", "!python finetune/prepare_buckets_latents.py \\\n", " {storage_train_dir} \\\n", " {meta_clean} \\\n", " {meta_lat} \\\n", " {model_dir} \\\n", " --batch_size {batch_size} \\\n", " --max_resolution {max_resolution} \\\n", " --mixed_precision {mixed_precision}\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 5 }