Sygil-Dev/sygil-webui
1
1
Fork 0
mirror of https://github.com/Sygil-Dev/sygil-webui.git synced 2024-12-14 05:58:18 +03:00
Code Issues Projects Releases Wiki Activity

Merge branch 'dev' into master

Browse Source
This commit is contained in:
cmdr2 2022-11-03 09:46:25 +05:30 committed by GitHub
commit 5e725e036e
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
40 changed files with 3105 additions and 2605 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@ -21,7 +21,7 @@ Come to our [Discord Server](https://discord.gg/gyXNe4NySY) or use [Discussions]
Check the [Contribution Guide](CONTRIBUTING.md) Check the [Contribution Guide](CONTRIBUTING.md)
[sygil-dev](https://github.com/sygil-dev) main devs: [Sygil-Dev](https://github.com/Sygil-Dev) main devs:
* ![hlky's avatar](https://avatars.githubusercontent.com/u/106811348?s=40&v=4) [hlky](https://github.com/hlky) * ![hlky's avatar](https://avatars.githubusercontent.com/u/106811348?s=40&v=4) [hlky](https://github.com/hlky)
* ![ZeroCool940711's avatar](https://avatars.githubusercontent.com/u/5977640?s=40&v=4)[ZeroCool940711](https://github.com/ZeroCool940711) * ![ZeroCool940711's avatar](https://avatars.githubusercontent.com/u/5977640?s=40&v=4)[ZeroCool940711](https://github.com/ZeroCool940711)

241
Web_based_UI_for_Stable_Diffusion_colab.ipynb
View File

@ -129,7 +129,7 @@
"\n", "\n",
"## Streamlit\n", "## Streamlit\n",
"\n", "\n",
"![](images/streamlit/streamlit-t2i.png)\n", "![](https://github.com/aedhcarrick/sygil-webui/blob/patch-2/images/streamlit/streamlit-t2i.png?raw=1)\n",
"\n", "\n",
"**Features:**\n", "**Features:**\n",
"\n", "\n",
@ -148,7 +148,7 @@
"\n", "\n",
"## Gradio\n", "## Gradio\n",
"\n", "\n",
"![](images/gradio/gradio-t2i.png)\n", "![](https://github.com/aedhcarrick/sygil-webui/blob/patch-2/images/gradio/gradio-t2i.png?raw=1)\n",
"\n", "\n",
"**Features:**\n", "**Features:**\n",
"\n", "\n",
@ -166,7 +166,7 @@
"\n", "\n",
"### GFPGAN\n", "### GFPGAN\n",
"\n", "\n",
"![](images/GFPGAN.png)\n", "![](https://github.com/aedhcarrick/sygil-webui/blob/patch-2/images/GFPGAN.png?raw=1)\n",
"\n", "\n",
"Lets you improve faces in pictures using the GFPGAN model. There is a checkbox in every tab to use GFPGAN at 100%, and also a separate tab that just allows you to use GFPGAN on any picture, with a slider that controls how strong the effect is.\n", "Lets you improve faces in pictures using the GFPGAN model. There is a checkbox in every tab to use GFPGAN at 100%, and also a separate tab that just allows you to use GFPGAN on any picture, with a slider that controls how strong the effect is.\n",
"\n", "\n",
@ -176,7 +176,7 @@
"\n", "\n",
"### RealESRGAN\n", "### RealESRGAN\n",
"\n", "\n",
"![](images/RealESRGAN.png)\n", "![](https://github.com/aedhcarrick/sygil-webui/blob/patch-2/images/RealESRGAN.png?raw=1)\n",
"\n", "\n",
"Lets you double the resolution of generated images. There is a checkbox in every tab to use RealESRGAN, and you can choose between the regular upscaler and the anime version.\n", "Lets you double the resolution of generated images. There is a checkbox in every tab to use RealESRGAN, and you can choose between the regular upscaler and the anime version.\n",
"There is also a separate tab for using RealESRGAN on any picture.\n", "There is also a separate tab for using RealESRGAN on any picture.\n",
@ -265,7 +265,56 @@
{ {
"cell_type": "markdown", "cell_type": "markdown",
"source": [ "source": [
"# Setup" "# Config options for Colab instance\n",
"> Before running, make sure GPU backend is enabled. (Unless you plan on generating with Stable Horde)\n",
">> Runtime -> Change runtime type -> Hardware Accelerator -> GPU (Make sure to save)"
],
"metadata": {
"id": "iegma7yteERV"
}
},
{
"cell_type": "code",
"source": [
"#@markdown WebUI repo (and branch)\n",
"repo_name = \"Sygil-Dev/sygil-webui\" #@param {type:\"string\"}\n",
"repo_branch = \"dev\" #@param {type:\"string\"}\n",
"\n",
"#@markdown Mount Google Drive\n",
"mount_google_drive = True #@param {type:\"boolean\"}\n",
"save_outputs_to_drive = True #@param {type:\"boolean\"}\n",
"#@markdown Folder in Google Drive to search for custom models\n",
"MODEL_DIR = \"\" #@param {type:\"string\"}\n",
"\n",
"#@markdown Enter auth token from Huggingface.co\n",
"#@markdown >(required for downloading stable diffusion model.)\n",
"HF_TOKEN = \"\" #@param {type:\"string\"}\n",
"\n",
"#@markdown Select which models to prefetch\n",
"STABLE_DIFFUSION = True #@param {type:\"boolean\"}\n",
"WAIFU_DIFFUSION = False #@param {type:\"boolean\"}\n",
"TRINART_SD = False #@param {type:\"boolean\"}\n",
"SD_WD_LD_TRINART_MERGED = False #@param {type:\"boolean\"}\n",
"GFPGAN = True #@param {type:\"boolean\"}\n",
"REALESRGAN = True #@param {type:\"boolean\"}\n",
"LDSR = True #@param {type:\"boolean\"}\n",
"BLIP_MODEL = False #@param {type:\"boolean\"}\n",
"\n"
],
"metadata": {
"id": "OXn96M9deVtF"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"# Setup\n",
"\n",
">Runtime will crash when installing conda. This is normal as we are forcing a restart of the runtime from code.\n",
"\n",
">Just hit \"Run All\" again. 😑"
], ],
"metadata": { "metadata": {
"id": "IZjJSr-WPNxB" "id": "IZjJSr-WPNxB"
@ -277,6 +326,7 @@
"id": "eq0-E5mjSpmP" "id": "eq0-E5mjSpmP"
}, },
"source": [ "source": [
"#@title Make sure we have access to GPU backend\n",
"!nvidia-smi -L" "!nvidia-smi -L"
], ],
"execution_count": null, "execution_count": null,
@ -285,14 +335,14 @@
{ {
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"#@title Install miniConda (mamba)\n",
"!pip install condacolab\n", "!pip install condacolab\n",
"import condacolab\n", "import condacolab\n",
"condacolab.install_from_url(\"https://github.com/conda-forge/miniforge/releases/download/4.14.0-0/Mambaforge-4.14.0-0-Linux-x86_64.sh\")\n", "condacolab.install_from_url(\"https://github.com/conda-forge/miniforge/releases/download/4.14.0-0/Mambaforge-4.14.0-0-Linux-x86_64.sh\")\n",
"\n", "\n",
"import condacolab\n", "import condacolab\n",
"condacolab.check()\n", "condacolab.check()\n",
"\n", "# The runtime will crash here!!! Don't panic! We planned for this remember?"
"# The runtime will crash after this, its normal as we are forcing a restart of the runtime from code. Just hit \"Run All\" again."
], ],
"metadata": { "metadata": {
"id": "cDu33xkdJ5mD" "id": "cDu33xkdJ5mD"
@ -303,9 +353,13 @@
{ {
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"!git clone https://github.com/Sygil-Dev/sygil-webui.git\n", "#@title Clone webUI repo and download font\n",
"%cd /content/sygil-webui/\n", "import os\n",
"!git checkout dev\n", "REPO_URL = os.path.join('https://github.com', repo_name)\n",
"PATH_TO_REPO = os.path.join('/content', repo_name.split('/')[1])\n",
"!git clone {REPO_URL}\n",
"%cd {PATH_TO_REPO}\n",
"!git checkout {repo_branch}\n",
"!git pull\n", "!git pull\n",
"!wget -O arial.ttf https://github.com/matomo-org/travis-scripts/blob/master/fonts/Arial.ttf?raw=true" "!wget -O arial.ttf https://github.com/matomo-org/travis-scripts/blob/master/fonts/Arial.ttf?raw=true"
], ],
@ -318,7 +372,10 @@
{ {
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"!mamba install cudatoolkit=11.3 git numpy=1.22.3 pip=20.3 python=3.8.5 pytorch=1.11.0 scikit-image=0.19.2 torchvision=0.12.0 -y" "#@title Install dependencies\n",
"!mamba install cudatoolkit=11.3 git numpy=1.22.3 pip=20.3 python=3.8.5 pytorch=1.11.0 scikit-image=0.19.2 torchvision=0.12.0 -y\n",
"!python --version\n",
"!pip install -r requirements.txt"
], ],
"metadata": { "metadata": {
"id": "dmN2igp5Yk3z" "id": "dmN2igp5Yk3z"
@ -329,52 +386,29 @@
{ {
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"#@title Install dependencies.\n", "#@title Install localtunnel to openGoogle's ports\n",
"!python --version\n",
"!pip install -r requirements.txt"
],
"metadata": {
"id": "vXX0OaR8KyLQ"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"!npm install localtunnel" "!npm install localtunnel"
], ],
"metadata": { "metadata": {
"id": "FHyVuT5aSM2G" "id": "Nxaxfgo_F8Am"
}, },
"execution_count": null, "execution_count": null,
"outputs": [] "outputs": []
}, },
{
"cell_type": "markdown",
"source": [
"#Launch the WebUI"
],
"metadata": {
"id": "csi6cj6gQZmC"
}
},
{ {
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"#@title Mount Google Drive\n", "#@title Mount Google Drive (if selected)\n",
"import os\n",
"mount_google_drive = True #@param {type:\"boolean\"}\n",
"save_outputs_to_drive = True #@param {type:\"boolean\"}\n",
"\n",
"if mount_google_drive:\n", "if mount_google_drive:\n",
" # Mount google drive to store your outputs.\n", " # Mount google drive to store outputs.\n",
" from google.colab import drive\n", " from google.colab import drive\n",
" drive.mount('/content/drive/', force_remount=True)\n", " drive.mount('/content/drive/', force_remount=True)\n",
"\n", "\n",
"if save_outputs_to_drive:\n", "if save_outputs_to_drive:\n",
" os.makedirs(\"/content/drive/MyDrive/sygil-webui/outputs\", exist_ok=True)\n", " # Make symlink to redirect downloads\n",
" os.symlink(\"/content/drive/MyDrive/sygil-webui/outputs\", \"/content/sygil-webui/outputs\", target_is_directory=True)\n" " OUTPUT_PATH = os.path.join('/content/drive/MyDrive', repo_name.split('/')[1], 'outputs')\n",
" os.makedirs(OUTPUT_PATH, exist_ok=True)\n",
" os.symlink(OUTPUT_PATH, os.path.join(PATH_TO_REPO, 'outputs'), target_is_directory=True)\n"
], ],
"metadata": { "metadata": {
"id": "pcSWo9Zkzbsf" "id": "pcSWo9Zkzbsf"
@ -385,20 +419,114 @@
{ {
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"#@title Enter Huggingface token\n", "#@title Pre-fetch models\n",
"!git config --global credential.helper store\n", "%cd {PATH_TO_REPO}\n",
"!huggingface-cli login" "# make list of models we want to download\n",
"model_list = {\n",
" 'stable_diffusion': f'{STABLE_DIFFUSION}',\n",
" 'waifu_diffusion': f'{WAIFU_DIFFUSION}',\n",
" 'trinart_stable_diffusion': f'{TRINART_SD}',\n",
" 'sd_wd_ld_trinart_merged': f'{SD_WD_LD_TRINART_MERGED}',\n",
" 'gfpgan': f'{GFPGAN}',\n",
" 'realesrgan': f'{REALESRGAN}',\n",
" 'ldsr': f'{LDSR}',\n",
" 'blip_model': f'{BLIP_MODEL}'}\n",
"download_list = {k for (k,v) in model_list.items() if v == 'True'}\n",
"\n",
"# get model info (file name, download link, save location)\n",
"import yaml\n",
"from pprint import pprint\n",
"with open('configs/webui/webui_streamlit.yaml') as f:\n",
" dataMap = yaml.safe_load(f)\n",
"models = dataMap['model_manager']['models']\n",
"\n",
"# copy script from model manager\n",
"import requests, time\n",
"from requests.auth import HTTPBasicAuth\n",
"\n",
"def download_file(file_name, file_path, file_url):\n",
" os.makedirs(file_path, exist_ok=True)\n",
" if os.path.exists(os.path.join(MODEL_DIR , file_name)):\n",
" print( file_name + \"found in Google Drive\")\n",
" print( \"Creating symlink...\")\n",
" os.symlink(os.path.join(MODEL_DIR , file_name), os.path.join(file_path, file_name))\n",
" elif not os.path.exists(os.path.join(file_path , file_name)):\n",
" print( \"Downloading \" + file_name + \"...\", end=\"\" )\n",
" token = None\n",
" if \"huggingface.co\" in file_url:\n",
" token = HTTPBasicAuth('token', HF_TOKEN)\n",
" try:\n",
" with requests.get(file_url, auth = token, stream=True) as r:\n",
" starttime = time.time()\n",
" r.raise_for_status()\n",
" with open(os.path.join(file_path, file_name), 'wb') as f:\n",
" for chunk in r.iter_content(chunk_size=8192):\n",
" f.write(chunk)\n",
" if ((time.time() - starttime) % 60.0) > 2 :\n",
" starttime = time.time()\n",
" print( \".\", end=\"\" )\n",
" print( \"done\" )\n",
" print( \" \" + file_name + \" downloaded to \\'\" + file_path + \"\\'\" )\n",
" except:\n",
" print( \"Failed to download \" + file_name + \".\" )\n",
" else:\n",
" print( file_name + \" already exists.\" )\n",
"\n",
"# download models in list\n",
"for model in download_list:\n",
" model_name = models[model]['model_name']\n",
" file_info = models[model]['files']\n",
" for file in file_info:\n",
" file_name = file_info[file]['file_name']\n",
" file_url = file_info[file]['download_link']\n",
" if 'save_location' in file_info[file]:\n",
" file_path = file_info[file]['save_location']\n",
" else: \n",
" file_path = models[model]['save_location']\n",
" download_file(file_name, file_path, file_url)\n",
"\n",
"# add custom models not in list\n",
"CUSTOM_MODEL_DIR = os.path.join(PATH_TO_REPO, 'models/custom')\n",
"if MODEL_DIR != \"\":\n",
" MODEL_DIR = os.path.join('/content/drive/MyDrive', MODEL_DIR)\n",
" if os.path.exists(MODEL_DIR):\n",
" custom_models = os.listdir(MODEL_DIR)\n",
" custom_models = [m for m in custom_models if os.path.isfile(MODEL_DIR + '/' + m)]\n",
" os.makedirs(CUSTOM_MODEL_DIR, exist_ok=True)\n",
" print( \"Custom model(s) found: \" )\n",
" for m in custom_models:\n",
" print( \" \" + m )\n",
" os.symlink(os.path.join(MODEL_DIR , m), os.path.join(CUSTOM_MODEL_DIR, m))\n",
"\n"
], ],
"metadata": { "metadata": {
"id": "IsbG7fvIrKwg" "id": "vMdmh81J70yA"
}, },
"execution_count": null, "execution_count": null,
"outputs": [] "outputs": []
}, },
{
"cell_type": "markdown",
"source": [
"# Launch the web ui server\n",
"### (optional) JS to prevent idle timeout:\n",
"Press 'F12' OR ('CTRL' + 'SHIFT' + 'I') OR right click on this website -> inspect. Then click on the console tab and paste in the following code.\n",
"```js,\n",
"function ClickConnect(){\n",
"console.log(\"Working\");\n",
"document.querySelector(\"colab-toolbar-button#connect\").click()\n",
"}\n",
"setInterval(ClickConnect,60000)\n",
"```"
],
"metadata": {
"id": "pjIjiCuJysJI"
}
},
{ {
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"#@title <-- Press play on the music player to keep the tab alive (Uses only 13MB of data)\n", "#@title Press play on the music player to keep the tab alive (Uses only 13MB of data)\n",
"%%html\n", "%%html\n",
"<b>Press play on the music player to keep the tab alive, then start your generation below (Uses only 13MB of data)</b><br/>\n", "<b>Press play on the music player to keep the tab alive, then start your generation below (Uses only 13MB of data)</b><br/>\n",
"<audio src=\"https://henk.tech/colabkobold/silence.m4a\" controls>" "<audio src=\"https://henk.tech/colabkobold/silence.m4a\" controls>"
@ -409,27 +537,10 @@
"execution_count": null, "execution_count": null,
"outputs": [] "outputs": []
}, },
{
"cell_type": "markdown",
"source": [
"JS to prevent idle timeout:\n",
"\n",
"Press F12 OR CTRL + SHIFT + I OR right click on this website -> inspect. Then click on the console tab and paste in the following code.\n",
"\n",
"function ClickConnect(){\n",
"console.log(\"Working\");\n",
"document.querySelector(\"colab-toolbar-button#connect\").click()\n",
"}\n",
"setInterval(ClickConnect,60000)"
],
"metadata": {
"id": "pjIjiCuJysJI"
}
},
{ {
"cell_type": "code", "cell_type": "code",
"source": [ "source": [
"#@title Open port 8501 and start Streamlit server. Open link in 'link.txt' file in file pane on left.\n", "#@title Run localtunnel and start Streamlit server. ('Ctrl' + 'left click') on link in the 'link.txt' file. (/content/link.txt)\n",
"!npx localtunnel --port 8501 &>/content/link.txt &\n", "!npx localtunnel --port 8501 &>/content/link.txt &\n",
"!streamlit run scripts/webui_streamlit.py --theme.base dark --server.headless true 2>&1 | tee -a /content/log.txt" "!streamlit run scripts/webui_streamlit.py --theme.base dark --server.headless true 2>&1 | tee -a /content/log.txt"
], ],

10
configs/webui/webui_streamlit.yaml
View File

@ -23,6 +23,7 @@ general:
streamlit_telemetry: False streamlit_telemetry: False
default_theme: dark default_theme: dark
huggingface_token: '' huggingface_token: ''
stable_horde_api: '0000000000'
gpu: 0 gpu: 0
outdir: outputs outdir: outputs
default_model: "Stable Diffusion v1.5" default_model: "Stable Diffusion v1.5"
@ -65,6 +66,10 @@ general:
update_preview: True update_preview: True
update_preview_frequency: 10 update_preview_frequency: 10
admin:
hide_server_setting: False
hide_browser_setting: False
debug: debug:
enable_hydralit: False enable_hydralit: False
@ -230,7 +235,8 @@ img2img:
step: 0.01 step: 0.01
# 0: Keep masked area # 0: Keep masked area
# 1: Regenerate only masked area # 1: Regenerate only masked area
mask_mode: 0 mask_mode: 1
noise_mode: "Matched Noise"
mask_restore: False mask_restore: False
# 0: Just resize # 0: Just resize
# 1: Crop and resize # 1: Crop and resize
@ -315,7 +321,7 @@ gfpgan:
strength: 100 strength: 100
textual_inversion: textual_inversion:
pretrained_model_name_or_path: "models/diffusers/stable-diffusion-v1-4" pretrained_model_name_or_path: "models/diffusers/stable-diffusion-v1-5"
tokenizer_name: "models/clip-vit-large-patch14" tokenizer_name: "models/clip-vit-large-patch14"

467
db.json Normal file
View File

@ -0,0 +1,467 @@
{
"stable_diffusion": {
"name": "stable_diffusion",
"type": "ckpt",
"description": "Generalist AI image generating model. The baseline for all finetuned models.",
"version": "1.5",
"style": "generalist",
"nsfw": false,
"download_all": true,
"requires": [
"clip-vit-large-patch14"
],
"config": {
"files": [
{
"path": "models/ldm/stable-diffusion-v1/model_1_5.ckpt"
},
{
"path": "configs/stable-diffusion/v1-inference.yaml"
}
],
"download": [
{
"file_name": "model_1_5.ckpt",
"file_path": "models/ldm/stable-diffusion-v1",
"file_url": "https://{username}:{password}@huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.ckpt",
"hf_auth": true
}
]
},
"available": false
},
"stable_diffusion_1.4": {
"name": "stable_diffusion",
"type": "ckpt",
"description": "Generalist AI image generating model. The baseline for all finetuned models.",
"version": "1.4",
"style": "generalist",
"nsfw": false,
"download_all": true,
"requires": [
"clip-vit-large-patch14"
],
"config": {
"files": [
{
"path": "models/ldm/stable-diffusion-v1/model.ckpt",
"md5sum": "c01059060130b8242849d86e97212c84"
},
{
"path": "configs/stable-diffusion/v1-inference.yaml"
}
],
"download": [
{
"file_name": "model.ckpt",
"file_path": "models/ldm/stable-diffusion-v1",
"file_url": "https://www.googleapis.com/storage/v1/b/aai-blog-files/o/sd-v1-4.ckpt?alt=media"
}
],
"alt_download": [
{
"file_name": "model.ckpt",
"file_path": "models/ldm/stable-diffusion-v1",
"file_url": "https://huggingface.co/CompVis/stable-diffusion-v-1-4-original/resolve/main/sd-v1-4.ckpt",
"hf_auth": true
}
]
},
"available": false
},
"waifu_diffusion": {
"name": "waifu_diffusion",
"type": "ckpt",
"description": "Anime styled generations.",
"version": "1.3",
"style": "anime",
"nsfw": false,
"download_all": true,
"requires": [
"clip-vit-large-patch14"
],
"config": {
"files": [
{
"path": "models/custom/waifu-diffusion.ckpt",
"md5sum": "a2aa170e3f513b32a3fd8841656e0123"
},
{
"path": "configs/stable-diffusion/v1-inference.yaml"
}
],
"download": [
{
"file_name": "waifu-diffusion.ckpt",
"file_path": "models/custom",
"file_url": "https://huggingface.co/hakurei/waifu-diffusion-v1-3/resolve/main/wd-v1-3-full.ckpt"
}
]
},
"available": false
},
"Furry Epoch": {
"name": "Furry Epoch",
"type": "ckpt",
"description": "Furry styled generations.",
"version": "4",
"style": "furry",
"nsfw": false,
"download_all": false,
"requires": [
"clip-vit-large-patch14"
],
"config": {
"files": [
{
"path": "models/custom/furry-diffusion.ckpt",
"md5sum": "f8ef45a295ef4966682f6e8fc2c6830d"
},
{
"path": "configs/stable-diffusion/v1-inference.yaml"
}
],
"download": [
{
"file_name": "furry-diffusion.ckpt",
"file_path": "models/custom",
"file_url": "https://sexy.canine.wf/file/furry-ckpt/furry_epoch4.ckpt"
}
]
},
"available": false
},
"Yiffy": {
"name": "Yiffy",
"type": "ckpt",
"description": "Furry styled generations.",
"version": "18",
"style": "furry",
"nsfw": false,
"download_all": true,
"requires": [
"clip-vit-large-patch14"
],
"config": {
"files": [
{
"path": "models/custom/yiffy.ckpt",
"md5sum": "dbe25794e24af183565dc45e9ec99713"
},
{
"path": "configs/stable-diffusion/v1-inference.yaml"
}
],
"download": [
{
"file_name": "yiffy.ckpt",
"file_path": "models/custom",
"file_url": "https://sexy.canine.wf/file/yiffy-ckpt/yiffy-e18.ckpt"
}
]
},
"available": false
},
"Zack3D": {
"name": "Zack3D",
"type": "ckpt",
"description": "Kink/NSFW oriented furry styled generations.",
"version": "1",
"style": "furry",
"nsfw": true,
"download_all": true,
"requires": [
"clip-vit-large-patch14"
],
"config": {
"files": [
{
"path": "models/custom/Zack3D.ckpt",
"md5sum": "aa944b1ecdaac60113027a0fdcda4f1b"
},
{
"path": "configs/stable-diffusion/v1-inference.yaml"
}
],
"download": [
{
"file_name": "Zack3D.ckpt",
"file_path": "models/custom",
"file_url": "https://sexy.canine.wf/file/furry-ckpt/Zack3D_Kinky-v1.ckpt"
}
]
},
"available": false
},
"trinart": {
"name": "trinart",
"type": "ckpt",
"description": "Manga styled generations.",
"version": "1",
"style": "anime",
"nsfw": false,
"download_all": true,
"requires": [
"clip-vit-large-patch14"
],
"config": {
"files": [
{
"path": "models/custom/trinart.ckpt"
},
{
"path": "configs/stable-diffusion/v1-inference.yaml"
}
],
"download": [
{
"file_name": "trinart.ckpt",
"file_path": "models/custom",
"file_url": "https://huggingface.co/naclbit/trinart_stable_diffusion_v2/resolve/main/trinart2_step95000.ckpt"
}
]
},
"available": false
},
"RealESRGAN_x4plus": {
"name": "RealESRGAN_x4plus",
"type": "realesrgan",
"description": "Upscaler.",
"version": "0.1.0",
"style": "generalist",
"config": {
"files": [
{
"path": "models/realesrgan/RealESRGAN_x4plus.pth"
}
],
"download": [
{
"file_name": "RealESRGAN_x4plus.pth",
"file_path": "models/realesrgan",
"file_url": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth"
}
]
},
"available": false
},
"RealESRGAN_x4plus_anime_6B": {
"name": "RealESRGAN_x4plus_anime_6B",
"type": "realesrgan",
"description": "Anime focused upscaler.",
"version": "0.2.2.4",
"style": "anime",
"config": {
"files": [
{
"path": "models/realesrgan/RealESRGAN_x4plus_anime_6B.pth"
}
],
"download": [
{
"file_name": "RealESRGAN_x4plus_anime_6B.pth",
"file_path": "models/realesrgan",
"file_url": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth"
}
]
},
"available": false
},
"GFPGAN": {
"name": "GFPGAN",
"type": "gfpgan",
"description": "Face correction.",
"version": "1.4",
"style": "generalist",
"config": {
"files": [
{
"path": "models/gfpgan/GFPGANv1.4.pth"
},
{
"path": "gfpgan/weights/detection_Resnet50_Final.pth"
},
{
"path": "gfpgan/weights/parsing_parsenet.pth"
}
],
"download": [
{
"file_name": "GFPGANv1.4.pth",
"file_path": "models/gfpgan",
"file_url": "https://github.com/TencentARC/GFPGAN/releases/download/v1.3.4/GFPGANv1.4.pth"
},
{
"file_name": "detection_Resnet50_Final.pth",
"file_path": "./gfpgan/weights",
"file_url": "https://github.com/xinntao/facexlib/releases/download/v0.1.0/detection_Resnet50_Final.pth"
},
{
"file_name": "parsing_parsenet.pth",
"file_path": "./gfpgan/weights",
"file_url": "https://github.com/xinntao/facexlib/releases/download/v0.2.2/parsing_parsenet.pth"
}
]
},
"available": false
},
"LDSR": {
"name": "LDSR",
"type": "ckpt",
"description": "Upscaler.",
"version": "1",
"style": "generalist",
"nsfw": false,
"download_all": true,
"config": {
"files": [
{
"path": "models/ldsr/model.ckpt"
},
{
"path": "models/ldsr/project.yaml"
}
],
"download": [
{
"file_name": "model.ckpt",
"file_path": "models/ldsr",
"file_url": "https://heibox.uni-heidelberg.de/f/578df07c8fc04ffbadf3/?dl=1"
},
{
"file_name": "project.yaml",
"file_path": "models/ldsr",
"file_url": "https://heibox.uni-heidelberg.de/f/31a76b13ea27482981b4/?dl=1"
}
]
},
"available": false
},
"BLIP": {
"name": "BLIP",
"type": "blip",
"config": {
"files": [
{
"path": "models/blip/model__base_caption.pth"
}
],
"download": [
{
"file_name": "model__base_caption.pth",
"file_path": "models/blip",
"file_url": "https://storage.googleapis.com/sfr-vision-language-research/BLIP/models/model*_base_caption.pth"
}
]
},
"available": false
},
"ViT-L/14": {
"name": "ViT-L/14",
"type": "clip",
"config": {
"files": [
{
"path": "models/clip/ViT-L-14.pt"
}
],
"download": [
{
"file_name": "ViT-L-14.pt",
"file_path": "./models/clip",
"file_url": "https://openaipublic.azureedge.net/clip/models/b8cca3fd41ae0c99ba7e8951adf17d267cdb84cd88be6f7c2e0eca1737a03836/ViT-L-14.pt"
}
]
},
"available": false
},
"ViT-g-14": {
"name": "ViT-g-14",
"pretrained_name": "laion2b_s12b_b42k",
"type": "open_clip",
"config": {
"files": [
{
"path": "models/clip/models--laion--CLIP-ViT-g-14-laion2B-s12B-b42K/"
}
],
"download": [
{
"file_name": "main",
"file_path": "./models/clip/models--laion--CLIP-ViT-g-14-laion2B-s12B-b42K/refs",
"file_content": "b36bdd32483debcf4ed2f918bdae1d4a46ee44b8"
},
{
"file_name": "6aac683f899159946bc4ca15228bb7016f3cbb1a2c51f365cba0b23923f344da",
"file_path": "./models/clip/models--laion--CLIP-ViT-g-14-laion2B-s12B-b42K/blobs",
"file_url": "https://huggingface.co/laion/CLIP-ViT-g-14-laion2B-s12B-b42K/resolve/main/open_clip_pytorch_model.bin"
},
{
"file_name": "open_clip_pytorch_model.bin",
"file_path": "./models/clip/models--laion--CLIP-ViT-g-14-laion2B-s12B-b42K/snapshots/b36bdd32483debcf4ed2f918bdae1d4a46ee44b8",
"symlink": "./models/clip/models--laion--CLIP-ViT-g-14-laion2B-s12B-b42K/blobs/6aac683f899159946bc4ca15228bb7016f3cbb1a2c51f365cba0b23923f344da"
}
]
},
"available": false
},
"ViT-H-14": {
"name": "ViT-H-14",
"pretrained_name": "laion2b_s32b_b79k",
"type": "open_clip",
"config": {
"files": [
{
"path": "models/clip/models--laion--CLIP-ViT-H-14-laion2B-s32B-b79K/"
}
],
"download": [
{
"file_name": "main",
"file_path": "./models/clip/models--laion--CLIP-ViT-H-14-laion2B-s32B-b79K/refs",
"file_content": "58a1e03a7acfacbe6b95ebc24ae0394eda6a14fc"
},
{
"file_name": "9a78ef8e8c73fd0df621682e7a8e8eb36c6916cb3c16b291a082ecd52ab79cc4",
"file_path": "./models/clip/models--laion--CLIP-ViT-H-14-laion2B-s32B-b79K/blobs",
"file_url": "https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/resolve/main/open_clip_pytorch_model.bin"
},
{
"file_name": "open_clip_pytorch_model.bin",
"file_path": "./models/clip/models--laion--CLIP-ViT-H-14-laion2B-s32B-b79K/snapshots/58a1e03a7acfacbe6b95ebc24ae0394eda6a14fc",
"symlink": "./models/clip/models--laion--CLIP-ViT-H-14-laion2B-s32B-b79K/blobs/9a78ef8e8c73fd0df621682e7a8e8eb36c6916cb3c16b291a082ecd52ab79cc4"
}
]
},
"available": false
},
"diffusers_stable_diffusion": {
"name": "diffusers_stable_diffusion",
"type": "diffusers",
"requires": [
"clip-vit-large-patch14"
],
"config": {
"files": [
{
"path": "models/diffusers/"
}
],
"download": [
{
"file_name": "diffusers_stable_diffusion",
"file_url": "https://{username}:{password}@huggingface.co/CompVis/stable-diffusion-v1-4.git",
"git": true,
"hf_auth": true,
"post_process": [
{
"delete": "models/diffusers/stable-diffusion-v1-4/.git"
}
]
}
]
},
"available": false
}
}

98
db_dep.json Normal file
View File

@ -0,0 +1,98 @@
{
"sd-concepts-library": {
"type": "dependency",
"optional": true,
"config": {
"files": [
{
"path": "models/custom/sd-concepts-library/"
}
],
"download": [
{
"file_name": "sd-concepts-library",
"file_path": "./models/custom/sd-concepts-library/",
"file_url": "https://github.com/sd-webui/sd-concepts-library/archive/refs/heads/main.zip",
"unzip": true,
"move_subfolder": "sd-concepts-library"
}
]
},
"available": false
},
"clip-vit-large-patch14": {
"type": "dependency",
"optional": false,
"config": {
"files": [
{
"path": "models/clip-vit-large-patch14/config.json"
},
{
"path": "models/clip-vit-large-patch14/merges.txt"
},
{
"path": "models/clip-vit-large-patch14/preprocessor_config.json"
},
{
"path": "models/clip-vit-large-patch14/pytorch_model.bin"
},
{
"path": "models/clip-vit-large-patch14/special_tokens_map.json"
},
{
"path": "models/clip-vit-large-patch14/tokenizer.json"
},
{
"path": "models/clip-vit-large-patch14/tokenizer_config.json"
},
{
"path": "models/clip-vit-large-patch14/vocab.json"
}
],
"download": [
{
"file_name": "config.json",
"file_path": "models/clip-vit-large-patch14",
"file_url": "https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/config.json"
},
{
"file_name": "merges.txt",
"file_path": "models/clip-vit-large-patch14",
"file_url": "https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/merges.txt"
},
{
"file_name": "preprocessor_config.json",
"file_path": "models/clip-vit-large-patch14",
"file_url": "https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/preprocessor_config.json"
},
{
"file_name": "pytorch_model.bin",
"file_path": "models/clip-vit-large-patch14",
"file_url": "https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/pytorch_model.bin"
},
{
"file_name": "special_tokens_map.json",
"file_path": "models/clip-vit-large-patch14",
"file_url": "https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/special_tokens_map.json"
},
{
"file_name": "tokenizer.json",
"file_path": "models/clip-vit-large-patch14",
"file_url": "https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/tokenizer.json"
},
{
"file_name": "tokenizer_config.json",
"file_path": "models/clip-vit-large-patch14",
"file_url": "https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/tokenizer_config.json"
},
{
"file_name": "vocab.json",
"file_path": "models/clip-vit-large-patch14",
"file_url": "https://huggingface.co/openai/clip-vit-large-patch14/resolve/main/vocab.json"
}
]
},
"available": false
}
}

92
docs/44.competition.md Normal file
View File

@ -0,0 +1,92 @@
# Textual inversion usage competition
We are hosting a competition where the community can showcase their most inventive use of textual inversion concepts in text-to-image or text-to-video.
Our compute cluster; `Nataili`, currently comprises of 3 nodes, two have 3090, the other has 2 x A5000.
We estimate `Nataili` can handle 12 concepts per hour, and we can add more workers if there is high demand.
Hopefully demand will be high, we want to train **hundreds** of new concepts!
# Schedule
2022/10/20 - Stage 1 begins, train concept command opened for usage
2022/10/22 12AM UTC - Stage 2 begins, text to image command opened for usage
2022/10/22 12PM UTC - Stage 1 ends, train concept command closed
2022/10/24 12PM UTC - Stage 2 ends, no more entries will be accepted
2022/10/24 6-12PM UTC - Winners announced
# What does `most inventive use` mean?
Whatever you want it to mean! be creative! experiment!
There are several categories we will look at:
* anything that's particularly creative, ~ artistic ~ or a e s t h e t i c
![20221019203426_00000](https://user-images.githubusercontent.com/106811348/197045193-d6f9c56b-9989-4f1c-b42a-bb02d62d77cd.png)
* composition; meaning anything related to how big things are, their position, the angle, etc
* styling;
![image](https://user-images.githubusercontent.com/106811348/197045629-029ba6f5-1f79-475c-9ce7-969aaf3d253b.png)
* `The Sims(TM): Stable Diffusion edition`
## So I can trai-
* Yes, as long as it's sfw
## `The Sims(TM): Stable Diffusion edition` ?
For this event the theme is “The Sims: Stable Diffusion edition”.
So we have selected a subset of [products from Amazon Berkely Objects dataset](https://github.com/sd-webui/abo).
Any other object is welcome too these are just a good source of data for this part of the competition.
Each product has images from multiple angles, the train concept command accepts up to 10 images, so choose the angles and modify backgrounds, experiment!
The goal with this category is to generate an image using the trained object, and the other categories apply, your imagination is the only limit! style a couch, try to make a BIG couch, try to make a couch on top of a mountain, try to make a vaporwave couch, anything!
# How do I train a concept using the discord bot?
Type `/trainconcept` then press tab to go through the fields
`Concept name` is just a name for your concept, it doesn't have to be a single word
`Placeholder` is what you will use in prompts to represent your concept
Add `<` and `>` so it is unique, multiple words should be hyphenated
`Initializer` is used as the starting point for training your concept, so this should be a single word that represents your concept
Minimum 2 images. Squareish aspect ratios work best
![Untitled-2](https://user-images.githubusercontent.com/106811348/197035834-cc973e29-31f8-48de-be2d-788fbe938b2e.png)
![image](https://user-images.githubusercontent.com/106811348/197035870-b91ef2a8-0ffd-47e1-a8df-9600df26cd6b.png)
# How do I use the trained concept?
## Prompting with concepts
When your concept is trained you can use it in prompts.
`a cute <nvidiafu> as an astronaut`:
![image](https://user-images.githubusercontent.com/106811348/197037250-044ea241-72a5-4caa-b772-35034245b4b6.png)
or `a green <green-couch> sitting on top of a floor, a 3D render, trending on polycount, minimalism, rendered in cinema4d`:
![image](https://user-images.githubusercontent.com/106811348/197037344-7ce72188-9129-4ba2-8a28-cba5fd664a9c.png)
## Using concepts in the webui
The discord bot will give you a link to a `.zip` file, download this, extract it, and put the folder in `stable-diffusion-webui/models/custom/sd-concepts-library`
![image](https://user-images.githubusercontent.com/106811348/197037892-ce53bea4-d1db-4b25-bb7c-7dfe4d71b2b1.png)

4
frontend/css/streamlit.main.css
View File

@ -160,7 +160,7 @@ div.gallery:hover {
/* Remove some empty spaces to make the UI more compact. */ /* Remove some empty spaces to make the UI more compact. */
.css-18e3th9{ .css-18e3th9{
padding-left: 10px; padding-left: 10px;
padding-right: 10px; padding-right: 30px;
position: unset !important; /* Fixes the layout/page going up when an expander or another item is expanded and then collapsed */ position: unset !important; /* Fixes the layout/page going up when an expander or another item is expanded and then collapsed */
} }
.css-k1vhr4{ .css-k1vhr4{
@ -168,7 +168,7 @@ div.gallery:hover {
} }
.css-ret2ud{ .css-ret2ud{
padding-left: 10px; padding-left: 10px;
padding-right: 25px; padding-right: 30px;
gap: initial; gap: initial;
display: initial; display: initial;
} }

2
requirements.txt
View File

@ -31,6 +31,8 @@ streamlit-option-menu==0.3.2
streamlit_nested_layout==0.1.1 streamlit_nested_layout==0.1.1
streamlit-server-state==0.14.2 streamlit-server-state==0.14.2
streamlit-tensorboard==0.0.2 streamlit-tensorboard==0.0.2
streamlit-elements==0.1.* # used for the draggable dashboard and new UI design (WIP)
streamlit-ace==0.1.1 # used to replace the text area on the prompt and also for the code editor tool.
hydralit==1.0.14 hydralit==1.0.14
hydralit_components==1.0.10 hydralit_components==1.0.10
stqdm==0.0.4 stqdm==0.0.4

69
scripts/Settings.py
View File

@ -50,7 +50,7 @@ def layout():
with col1: with col1:
st.title("General") st.title("General")
st.session_state['defaults'].general.gpu = int(st.selectbox("GPU", device_list, st.session_state['defaults'].general.gpu = int(st.selectbox("GPU", device_list, index=st.session_state['defaults'].general.gpu,
help=f"Select which GPU to use. Default: {device_list[0]}").split(":")[0]) help=f"Select which GPU to use. Default: {device_list[0]}").split(":")[0])
st.session_state['defaults'].general.outdir = str(st.text_input("Output directory", value=st.session_state['defaults'].general.outdir, st.session_state['defaults'].general.outdir = str(st.text_input("Output directory", value=st.session_state['defaults'].general.outdir,
@ -208,15 +208,65 @@ def layout():
with col4: with col4:
st.title("Streamlit Config") st.title("Streamlit Config")
st.session_state["defaults"].general.streamlit_telemetry = st.checkbox("Enable Telemetry", value=st.session_state['defaults'].general.streamlit_telemetry,
help="Enables or Disables streamlit telemetry. Default: False")
st.session_state["streamlit_config"]["browser"]["gatherUsageStats"] = st.session_state["defaults"].general.streamlit_telemetry
default_theme_list = ["light", "dark"] default_theme_list = ["light", "dark"]
st.session_state["defaults"].general.default_theme = st.selectbox("Default Theme", default_theme_list, index=default_theme_list.index(st.session_state['defaults'].general.default_theme), st.session_state["defaults"].general.default_theme = st.selectbox("Default Theme", default_theme_list, index=default_theme_list.index(st.session_state['defaults'].general.default_theme),
help="Defaut theme to use as base for streamlit. Default: dark") help="Defaut theme to use as base for streamlit. Default: dark")
st.session_state["streamlit_config"]["theme"]["base"] = st.session_state["defaults"].general.default_theme st.session_state["streamlit_config"]["theme"]["base"] = st.session_state["defaults"].general.default_theme
if not st.session_state['defaults'].admin.hide_server_setting:
with st.expander("Server", True):
st.session_state["streamlit_config"]['server']['headless'] = st.checkbox("Run Headless", help="If false, will attempt to open a browser window on start. \
Default: false unless (1) we are on a Linux box where DISPLAY is unset, \
or (2) we are running in the Streamlit Atom plugin.")
st.session_state["streamlit_config"]['server']['port'] = st.number_input("Port", value=st.session_state["streamlit_config"]['server']['port'],
help="The port where the server will listen for browser connections. Default: 8501")
st.session_state["streamlit_config"]['server']['baseUrlPath'] = st.text_input("Base Url Path", value=st.session_state["streamlit_config"]['server']['baseUrlPath'],
help="The base path for the URL where Streamlit should be served from. Default: '' ")
st.session_state["streamlit_config"]['server']['enableCORS'] = st.checkbox("Enable CORS", value=st.session_state['streamlit_config']['server']['enableCORS'],
help="Enables support for Cross-Origin Request Sharing (CORS) protection, for added security. \
Due to conflicts between CORS and XSRF, if `server.enableXsrfProtection` is on and `server.enableCORS` \
is off at the same time, we will prioritize `server.enableXsrfProtection`. Default: true")
st.session_state["streamlit_config"]['server']['enableXsrfProtection'] = st.checkbox("Enable Xsrf Protection",
value=st.session_state['streamlit_config']['server']['enableXsrfProtection'],
help="Enables support for Cross-Site Request Forgery (XSRF) protection, \
for added security. Due to conflicts between CORS and XSRF, \
if `server.enableXsrfProtection` is on and `server.enableCORS` is off at \
the same time, we will prioritize `server.enableXsrfProtection`. Default: true")
st.session_state["streamlit_config"]['server']['maxUploadSize'] = st.number_input("Max Upload Size", value=st.session_state["streamlit_config"]['server']['maxUploadSize'],
help="Max size, in megabytes, for files uploaded with the file_uploader. Default: 200")
st.session_state["streamlit_config"]['server']['maxMessageSize'] = st.number_input("Max Message Size", value=st.session_state["streamlit_config"]['server']['maxUploadSize'],
help="Max size, in megabytes, of messages that can be sent via the WebSocket connection. Default: 200")
st.session_state["streamlit_config"]['server']['enableWebsocketCompression'] = st.checkbox("Enable Websocket Compression",
value=st.session_state["streamlit_config"]['server']['enableWebsocketCompression'],
help=" Enables support for websocket compression. Default: false")
if not st.session_state['defaults'].admin.hide_browser_setting:
with st.expander("Browser", expanded=True):
st.session_state["streamlit_config"]['browser']['serverAddress'] = st.text_input("Server Address",
value=st.session_state["streamlit_config"]['browser']['serverAddress'] if "serverAddress" in st.session_state["streamlit_config"] else "localhost",
help="Internet address where users should point their browsers in order \
to connect to the app. Can be IP address or DNS name and path.\
This is used to: - Set the correct URL for CORS and XSRF protection purposes. \
- Show the URL on the terminal - Open the browser. Default: 'localhost'")
st.session_state["defaults"].general.streamlit_telemetry = st.checkbox("Enable Telemetry", value=st.session_state['defaults'].general.streamlit_telemetry,
help="Enables or Disables streamlit telemetry. Default: False")
st.session_state["streamlit_config"]["browser"]["gatherUsageStats"] = st.session_state["defaults"].general.streamlit_telemetry
st.session_state["streamlit_config"]['browser']['serverPort'] = st.number_input("Server Port", value=st.session_state["streamlit_config"]['browser']['serverPort'],
help="Port where users should point their browsers in order to connect to the app. \
This is used to: - Set the correct URL for CORS and XSRF protection purposes. \
- Show the URL on the terminal - Open the browser \
Default: whatever value is set in server.port.")
with col5: with col5:
st.title("Huggingface") st.title("Huggingface")
st.session_state["defaults"].general.huggingface_token = st.text_input("Huggingface Token", value=st.session_state['defaults'].general.huggingface_token, type="password", st.session_state["defaults"].general.huggingface_token = st.text_input("Huggingface Token", value=st.session_state['defaults'].general.huggingface_token, type="password",
@ -225,6 +275,15 @@ def layout():
and WILL NOT be share with us or anyone. You can get your access token \ and WILL NOT be share with us or anyone. You can get your access token \
at https://huggingface.co/settings/tokens. Default: None") at https://huggingface.co/settings/tokens. Default: None")
st.title("Stable Horde")
st.session_state["defaults"].general.stable_horde_api = st.text_input("Stable Horde Api", value=st.session_state["defaults"].general.stable_horde_api, type="password",
help="First Register an account at https://stablehorde.net/register which will generate for you \
an API key. Store that key somewhere safe. \n \
If you do not want to register, you can use `0000000000` as api_key to connect anonymously.\
However anonymous accounts have the lowest priority when there's too many concurrent requests! \
To increase your priority you will need a unique API key and then to increase your Kudos \
read more about them at https://dbzer0.com/blog/the-kudos-based-economy-for-the-koboldai-horde/.")
with txt2img_tab: with txt2img_tab:
col1, col2, col3, col4, col5 = st.columns(5, gap='medium') col1, col2, col3, col4, col5 = st.columns(5, gap='medium')

97
scripts/barfi_baklavajs.py Normal file
View File

@ -0,0 +1,97 @@
# This file is part of sygil-webui (https://github.com/Sygil-Dev/sandbox-webui/).
# Copyright 2022 Sygil-Dev team.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# base webui import and utils.
#from sd_utils import *
from sd_utils import *
# streamlit imports
#streamlit components section
#other imports
import os, time, requests
import sys
from barfi import st_barfi, barfi_schemas, Block
# Temp imports
# end of imports
#---------------------------------------------------------------------------------------------------------------
def layout():
#st.info("Under Construction. :construction_worker:")
#from barfi import st_barfi, Block
#add = Block(name='Addition')
#sub = Block(name='Subtraction')
#mul = Block(name='Multiplication')
#div = Block(name='Division')
#barfi_result = st_barfi(base_blocks= [add, sub, mul, div])
# or if you want to use a category to organise them in the frontend sub-menu
#barfi_result = st_barfi(base_blocks= {'Op 1': [add, sub], 'Op 2': [mul, div]})
col1, col2, col3 = st.columns([1, 8, 1])
from barfi import st_barfi, barfi_schemas, Block
with col2:
feed = Block(name='Feed')
feed.add_output()
def feed_func(self):
self.set_interface(name='Output 1', value=4)
feed.add_compute(feed_func)
splitter = Block(name='Splitter')
splitter.add_input()
splitter.add_output()
splitter.add_output()
def splitter_func(self):
in_1 = self.get_interface(name='Input 1')
value = (in_1/2)
self.set_interface(name='Output 1', value=value)
self.set_interface(name='Output 2', value=value)
splitter.add_compute(splitter_func)
mixer = Block(name='Mixer')
mixer.add_input()
mixer.add_input()
mixer.add_output()
def mixer_func(self):
in_1 = self.get_interface(name='Input 1')
in_2 = self.get_interface(name='Input 2')
value = (in_1 + in_2)
self.set_interface(name='Output 1', value=value)
mixer.add_compute(mixer_func)
result = Block(name='Result')
result.add_input()
def result_func(self):
in_1 = self.get_interface(name='Input 1')
result.add_compute(result_func)
load_schema = st.selectbox('Select a saved schema:', barfi_schemas())
compute_engine = st.checkbox('Activate barfi compute engine', value=False)
barfi_result = st_barfi(base_blocks=[feed, result, mixer, splitter],
compute_engine=compute_engine, load_schema=load_schema)
if barfi_result:
st.write(barfi_result)

126
scripts/img2img.py
View File

@ -33,6 +33,7 @@ from ldm.models.diffusion.plms import PLMSSampler
# streamlit components # streamlit components
from custom_components import sygil_suggestions from custom_components import sygil_suggestions
from streamlit_drawable_canvas import st_canvas
# Temp imports # Temp imports
@ -381,7 +382,7 @@ def layout():
# creating the page layout using columns # creating the page layout using columns
col1_img2img_layout, col2_img2img_layout, col3_img2img_layout = st.columns([1,2,2], gap="medium") col1_img2img_layout, col2_img2img_layout, col3_img2img_layout = st.columns([2,4,4], gap="medium")
with col1_img2img_layout: with col1_img2img_layout:
# If we have custom models available on the "models/custom" # If we have custom models available on the "models/custom"
@ -426,7 +427,7 @@ def layout():
mask_expander = st.empty() mask_expander = st.empty()
with mask_expander.expander("Mask"): with mask_expander.expander("Mask"):
mask_mode_list = ["Mask", "Inverted mask", "Image alpha"] mask_mode_list = ["Mask", "Inverted mask", "Image alpha"]
mask_mode = st.selectbox("Mask Mode", mask_mode_list, mask_mode = st.selectbox("Mask Mode", mask_mode_list, index=st.session_state["defaults"].img2img.mask_mode,
help="Select how you want your image to be masked.\"Mask\" modifies the image where the mask is white.\n\ help="Select how you want your image to be masked.\"Mask\" modifies the image where the mask is white.\n\
\"Inverted mask\" modifies the image where the mask is black. \"Image alpha\" modifies the image where the image is transparent." \"Inverted mask\" modifies the image where the mask is black. \"Image alpha\" modifies the image where the image is transparent."
) )
@ -434,15 +435,32 @@ def layout():
noise_mode_list = ["Seed", "Find Noise", "Matched Noise", "Find+Matched Noise"] noise_mode_list = ["Seed", "Find Noise", "Matched Noise", "Find+Matched Noise"]
noise_mode = st.selectbox( noise_mode = st.selectbox("Noise Mode", noise_mode_list, index=noise_mode_list.index(st.session_state['defaults'].img2img.noise_mode), help="")
"Noise Mode", noise_mode_list, #noise_mode = noise_mode_list.index(noise_mode)
help=""
)
noise_mode = noise_mode_list.index(noise_mode)
find_noise_steps = st.number_input("Find Noise Steps", value=st.session_state['defaults'].img2img.find_noise_steps.value, find_noise_steps = st.number_input("Find Noise Steps", value=st.session_state['defaults'].img2img.find_noise_steps.value,
min_value=st.session_state['defaults'].img2img.find_noise_steps.min_value, min_value=st.session_state['defaults'].img2img.find_noise_steps.min_value,
step=st.session_state['defaults'].img2img.find_noise_steps.step) step=st.session_state['defaults'].img2img.find_noise_steps.step)
# Specify canvas parameters in application
drawing_mode = st.selectbox(
"Drawing tool:",
(
"freedraw",
"transform",
#"line",
"rect",
"circle",
#"polygon",
),
)
stroke_width = st.slider("Stroke width: ", 1, 100, 50)
stroke_color = st.color_picker("Stroke color hex: ", value="#EEEEEE")
bg_color = st.color_picker("Background color hex: ", "#7B6E6E")
display_toolbar = st.checkbox("Display toolbar", True)
#realtime_update = st.checkbox("Update in realtime", True)
with st.expander("Batch Options"): with st.expander("Batch Options"):
st.session_state["batch_count"] = st.number_input("Batch count.", value=st.session_state['defaults'].img2img.batch_count.value, st.session_state["batch_count"] = st.number_input("Batch count.", value=st.session_state['defaults'].img2img.batch_count.value,
help="How many iterations or batches of images to generate in total.") help="How many iterations or batches of images to generate in total.")
@ -527,7 +545,9 @@ def layout():
upscaling_method_list.append("LDSR") upscaling_method_list.append("LDSR")
st.session_state["upscaling_method"] = st.selectbox("Upscaling Method", upscaling_method_list, st.session_state["upscaling_method"] = st.selectbox("Upscaling Method", upscaling_method_list,
index=upscaling_method_list.index(st.session_state['defaults'].general.upscaling_method)) index=upscaling_method_list.index(st.session_state['defaults'].general.upscaling_method)
if st.session_state['defaults'].general.upscaling_method in upscaling_method_list
else 0)
if st.session_state["RealESRGAN_available"]: if st.session_state["RealESRGAN_available"]:
with st.expander("RealESRGAN"): with st.expander("RealESRGAN"):
@ -583,55 +603,63 @@ def layout():
editor_image = st.empty() editor_image = st.empty()
st.session_state["editor_image"] = editor_image st.session_state["editor_image"] = editor_image
st.form_submit_button("Refresh")
#if "canvas" not in st.session_state:
st.session_state["canvas"] = st.empty()
masked_image_holder = st.empty() masked_image_holder = st.empty()
image_holder = st.empty() image_holder = st.empty()
st.form_submit_button("Refresh")
uploaded_images = st.file_uploader( uploaded_images = st.file_uploader(
"Upload Image", accept_multiple_files=False, type=["png", "jpg", "jpeg", "webp", 'jfif'], "Upload Image", accept_multiple_files=False, type=["png", "jpg", "jpeg", "webp", 'jfif'],
help="Upload an image which will be used for the image to image generation.", help="Upload an image which will be used for the image to image generation.",
) )
if uploaded_images: if uploaded_images:
image = Image.open(uploaded_images).convert('RGBA') image = Image.open(uploaded_images).convert('RGB')
new_img = image.resize((width, height)) new_img = image.resize((width, height))
image_holder.image(new_img) #image_holder.image(new_img)
mask_holder = st.empty() #mask_holder = st.empty()
uploaded_masks = st.file_uploader( #uploaded_masks = st.file_uploader(
"Upload Mask", accept_multiple_files=False, type=["png", "jpg", "jpeg", "webp", 'jfif'], #"Upload Mask", accept_multiple_files=False, type=["png", "jpg", "jpeg", "webp", 'jfif'],
help="Upload an mask image which will be used for masking the image to image generation.", #help="Upload an mask image which will be used for masking the image to image generation.",
#)
#
# Create a canvas component
with st.session_state["canvas"]:
st.session_state["uploaded_masks"] = st_canvas(
fill_color="rgba(255, 165, 0, 0.3)", # Fixed fill color with some opacity
stroke_width=stroke_width,
stroke_color=stroke_color,
background_color=bg_color,
background_image=image if uploaded_images else None,
update_streamlit=True,
width=width,
height=height,
drawing_mode=drawing_mode,
initial_drawing=st.session_state["uploaded_masks"].json_data if "uploaded_masks" in st.session_state else None,
display_toolbar= display_toolbar,
key="full_app",
) )
if uploaded_masks:
mask_expander.expander("Mask", expanded=True)
mask = Image.open(uploaded_masks)
if mask.mode == "RGBA":
mask = mask.convert('RGBA')
background = Image.new('RGBA', mask.size, (0, 0, 0))
mask = Image.alpha_composite(background, mask)
mask = mask.resize((width, height))
mask_holder.image(mask)
if uploaded_images and uploaded_masks: #try:
if mask_mode != 2: ##print (type(st.session_state["uploaded_masks"]))
final_img = new_img.copy() #if st.session_state["uploaded_masks"] != None:
alpha_layer = mask.convert('L') #mask_expander.expander("Mask", expanded=True)
strength = st.session_state["denoising_strength"] #mask = Image.fromarray(st.session_state["uploaded_masks"].image_data)
if mask_mode == 0:
alpha_layer = ImageOps.invert(alpha_layer)
alpha_layer = alpha_layer.point(lambda a: a * strength)
alpha_layer = ImageOps.invert(alpha_layer)
elif mask_mode == 1:
alpha_layer = alpha_layer.point(lambda a: a * strength)
alpha_layer = ImageOps.invert(alpha_layer)
final_img.putalpha(alpha_layer) #st.image(mask)
with masked_image_holder.container():
st.text("Masked Image Preview")
st.image(final_img)
#if mask.mode == "RGBA":
#mask = mask.convert('RGBA')
#background = Image.new('RGBA', mask.size, (0, 0, 0))
#mask = Image.alpha_composite(background, mask)
#mask = mask.resize((width, height))
#except AttributeError:
#pass
with col3_img2img_layout: with col3_img2img_layout:
result_tab = st.tabs(["Result"]) result_tab = st.tabs(["Result"])
@ -645,7 +673,6 @@ def layout():
st.session_state["progress_bar_text"] = st.empty() st.session_state["progress_bar_text"] = st.empty()
st.session_state["progress_bar"] = st.empty() st.session_state["progress_bar"] = st.empty()
message = st.empty() message = st.empty()
#if uploaded_images: #if uploaded_images:
@ -666,14 +693,17 @@ def layout():
CustomModel_available=server_state["CustomModel_available"], custom_model=st.session_state["custom_model"]) CustomModel_available=server_state["CustomModel_available"], custom_model=st.session_state["custom_model"])
if uploaded_images: if uploaded_images:
image = Image.open(uploaded_images).convert('RGBA') #image = Image.fromarray(image).convert('RGBA')
new_img = image.resize((width, height)) #new_img = image.resize((width, height))
#img_array = np.array(image) # if you want to pass it to OpenCV ###img_array = np.array(image) # if you want to pass it to OpenCV
#image_holder.image(new_img)
new_mask = None new_mask = None
if uploaded_masks:
mask = Image.open(uploaded_masks).convert('RGBA') if st.session_state["uploaded_masks"]:
mask = Image.fromarray(st.session_state["uploaded_masks"].image_data)
new_mask = mask.resize((width, height)) new_mask = mask.resize((width, height))
#masked_image_holder.image(new_mask)
try: try:
output_images, seed, info, stats = img2img(prompt=prompt, init_info=new_img, init_info_mask=new_mask, mask_mode=mask_mode, output_images, seed, info, stats = img2img(prompt=prompt, init_info=new_img, init_info_mask=new_mask, mask_mode=mask_mode,
mask_restore=img2img_mask_restore, ddim_steps=st.session_state["sampling_steps"], mask_restore=img2img_mask_restore, ddim_steps=st.session_state["sampling_steps"],

5
scripts/img2txt.py
View File

@ -367,10 +367,7 @@ def layout():
col1, col2 = st.columns([1, 4], gap="large") col1, col2 = st.columns([1, 4], gap="large")
with col1: with col1:
#url = st.text_area("Input Text","") st.session_state["uploaded_image"] = st.file_uploader('Input Image', type=['png', 'jpg', 'jpeg', 'jfif', 'webp'], accept_multiple_files=True)
#url = st.text_input("Input Text","", placeholder="A corgi wearing a top hat as an oil painting.")
#st.subheader("Input Image")
st.session_state["uploaded_image"] = st.file_uploader('Input Image', type=['png', 'jpg', 'jpeg', 'jfif'], accept_multiple_files=True)
with st.expander("CLIP models", expanded=True): with st.expander("CLIP models", expanded=True):
st.session_state["ViT-L/14"] = st.checkbox("ViT-L/14", value=True, help="ViT-L/14 model.") st.session_state["ViT-L/14"] = st.checkbox("ViT-L/14", value=True, help="ViT-L/14 model.")

0
scripts/nataili/__init__.py
View File

0
scripts/nataili/inference/__init__.py
View File

0
scripts/nataili/inference/compvis/__init__.py
View File

551
scripts/nataili/inference/compvis/img2img.py
View File

@ -1,551 +0,0 @@
import os
import re
import sys
import k_diffusion as K
import tqdm
from contextlib import contextmanager, nullcontext
import skimage
import numpy as np
import PIL
import torch
from einops import rearrange
from ldm.models.diffusion.ddim import DDIMSampler
from ldm.models.diffusion.kdiffusion import CFGMaskedDenoiser, KDiffusionSampler
from ldm.models.diffusion.plms import PLMSSampler
from nataili.util.cache import torch_gc
from nataili.util.check_prompt_length import check_prompt_length
from nataili.util.get_next_sequence_number import get_next_sequence_number
from nataili.util.image_grid import image_grid
from nataili.util.load_learned_embed_in_clip import load_learned_embed_in_clip
from nataili.util.save_sample import save_sample
from nataili.util.seed_to_int import seed_to_int
from slugify import slugify
import PIL
class img2img:
def __init__(self, model, device, output_dir, save_extension='jpg',
output_file_path=False, load_concepts=False, concepts_dir=None,
verify_input=True, auto_cast=True):
self.model = model
self.output_dir = output_dir
self.output_file_path = output_file_path
self.save_extension = save_extension
self.load_concepts = load_concepts
self.concepts_dir = concepts_dir
self.verify_input = verify_input
self.auto_cast = auto_cast
self.device = device
self.comments = []
self.output_images = []
self.info = ''
self.stats = ''
self.images = []
def create_random_tensors(self, shape, seeds):
xs = []
for seed in seeds:
torch.manual_seed(seed)
# randn results depend on device; gpu and cpu get different results for same seed;
# the way I see it, it's better to do this on CPU, so that everyone gets same result;
# but the original script had it like this so i do not dare change it for now because
# it will break everyone's seeds.
xs.append(torch.randn(shape, device=self.device))
x = torch.stack(xs)
return x
def process_prompt_tokens(self, prompt_tokens):
# compviz codebase
tokenizer = self.model.cond_stage_model.tokenizer
text_encoder = self.model.cond_stage_model.transformer
# diffusers codebase
#tokenizer = pipe.tokenizer
#text_encoder = pipe.text_encoder
ext = ('.pt', '.bin')
for token_name in prompt_tokens:
embedding_path = os.path.join(self.concepts_dir, token_name)
if os.path.exists(embedding_path):
for files in os.listdir(embedding_path):
if files.endswith(ext):
load_learned_embed_in_clip(f"{os.path.join(embedding_path, files)}", text_encoder, tokenizer, f"<{token_name}>")
else:
print(f"Concept {token_name} not found in {self.concepts_dir}")
del tokenizer, text_encoder
return
del tokenizer, text_encoder
def resize_image(self, resize_mode, im, width, height):
LANCZOS = (PIL.Image.Resampling.LANCZOS if hasattr(PIL.Image, 'Resampling') else PIL.Image.LANCZOS)
if resize_mode == "resize":
res = im.resize((width, height), resample=LANCZOS)
elif resize_mode == "crop":
ratio = width / height
src_ratio = im.width / im.height
src_w = width if ratio > src_ratio else im.width * height // im.height
src_h = height if ratio <= src_ratio else im.height * width // im.width
resized = im.resize((src_w, src_h), resample=LANCZOS)
res = PIL.Image.new("RGBA", (width, height))
res.paste(resized, box=(width // 2 - src_w // 2, height // 2 - src_h // 2))
else:
ratio = width / height
src_ratio = im.width / im.height
src_w = width if ratio < src_ratio else im.width * height // im.height
src_h = height if ratio >= src_ratio else im.height * width // im.width
resized = im.resize((src_w, src_h), resample=LANCZOS)
res = PIL.Image.new("RGBA", (width, height))
res.paste(resized, box=(width // 2 - src_w // 2, height // 2 - src_h // 2))
if ratio < src_ratio:
fill_height = height // 2 - src_h // 2
res.paste(resized.resize((width, fill_height), box=(0, 0, width, 0)), box=(0, 0))
res.paste(resized.resize((width, fill_height), box=(0, resized.height, width, resized.height)), box=(0, fill_height + src_h))
elif ratio > src_ratio:
fill_width = width // 2 - src_w // 2
res.paste(resized.resize((fill_width, height), box=(0, 0, 0, height)), box=(0, 0))
res.paste(resized.resize((fill_width, height), box=(resized.width, 0, resized.width, height)), box=(fill_width + src_w, 0))
return res
#
# helper fft routines that keep ortho normalization and auto-shift before and after fft
def _fft2(self, data):
if data.ndim > 2: # has channels
out_fft = np.zeros((data.shape[0], data.shape[1], data.shape[2]), dtype=np.complex128)
for c in range(data.shape[2]):
c_data = data[:,:,c]
out_fft[:,:,c] = np.fft.fft2(np.fft.fftshift(c_data),norm="ortho")
out_fft[:,:,c] = np.fft.ifftshift(out_fft[:,:,c])
else: # one channel
out_fft = np.zeros((data.shape[0], data.shape[1]), dtype=np.complex128)
out_fft[:,:] = np.fft.fft2(np.fft.fftshift(data),norm="ortho")
out_fft[:,:] = np.fft.ifftshift(out_fft[:,:])
return out_fft
def _ifft2(self, data):
if data.ndim > 2: # has channels
out_ifft = np.zeros((data.shape[0], data.shape[1], data.shape[2]), dtype=np.complex128)
for c in range(data.shape[2]):
c_data = data[:,:,c]
out_ifft[:,:,c] = np.fft.ifft2(np.fft.fftshift(c_data),norm="ortho")
out_ifft[:,:,c] = np.fft.ifftshift(out_ifft[:,:,c])
else: # one channel
out_ifft = np.zeros((data.shape[0], data.shape[1]), dtype=np.complex128)
out_ifft[:,:] = np.fft.ifft2(np.fft.fftshift(data),norm="ortho")
out_ifft[:,:] = np.fft.ifftshift(out_ifft[:,:])
return out_ifft
def _get_gaussian_window(self, width, height, std=3.14, mode=0):
window_scale_x = float(width / min(width, height))
window_scale_y = float(height / min(width, height))
window = np.zeros((width, height))
x = (np.arange(width) / width * 2. - 1.) * window_scale_x
for y in range(height):
fy = (y / height * 2. - 1.) * window_scale_y
if mode == 0:
window[:, y] = np.exp(-(x**2+fy**2) * std)
else:
window[:, y] = (1/((x**2+1.) * (fy**2+1.))) ** (std/3.14) # hey wait a minute that's not gaussian
return window
def _get_masked_window_rgb(self, np_mask_grey, hardness=1.):
np_mask_rgb = np.zeros((np_mask_grey.shape[0], np_mask_grey.shape[1], 3))
if hardness != 1.:
hardened = np_mask_grey[:] ** hardness
else:
hardened = np_mask_grey[:]
for c in range(3):
np_mask_rgb[:,:,c] = hardened[:]
return np_mask_rgb
def get_matched_noise(self, _np_src_image, np_mask_rgb, noise_q, color_variation):
"""
Explanation:
Getting good results in/out-painting with stable diffusion can be challenging.
Although there are simpler effective solutions for in-painting, out-painting can be especially challenging because there is no color data
in the masked area to help prompt the generator. Ideally, even for in-painting we'd like work effectively without that data as well.
Provided here is my take on a potential solution to this problem.
By taking a fourier transform of the masked src img we get a function that tells us the presence and orientation of each feature scale in the unmasked src.
Shaping the init/seed noise for in/outpainting to the same distribution of feature scales, orientations, and positions increases output coherence
by helping keep features aligned. This technique is applicable to any continuous generation task such as audio or video, each of which can
be conceptualized as a series of out-painting steps where the last half of the input "frame" is erased. For multi-channel data such as color
or stereo sound the "color tone" or histogram of the seed noise can be matched to improve quality (using scikit-image currently)
This method is quite robust and has the added benefit of being fast independently of the size of the out-painted area.
The effects of this method include things like helping the generator integrate the pre-existing view distance and camera angle.
Carefully managing color and brightness with histogram matching is also essential to achieving good coherence.
noise_q controls the exponent in the fall-off of the distribution can be any positive number, lower values means higher detail (range > 0, default 1.)
color_variation controls how much freedom is allowed for the colors/palette of the out-painted area (range 0..1, default 0.01)
This code is provided as is under the Unlicense (https://unlicense.org/)
Although you have no obligation to do so, if you found this code helpful please find it in your heart to credit me [parlance-zz].
Questions or comments can be sent to parlance@fifth-harmonic.com (https://github.com/parlance-zz/)
This code is part of a new branch of a discord bot I am working on integrating with diffusers (https://github.com/parlance-zz/g-diffuser-bot)
"""
global DEBUG_MODE
global TMP_ROOT_PATH
width = _np_src_image.shape[0]
height = _np_src_image.shape[1]
num_channels = _np_src_image.shape[2]
np_src_image = _np_src_image[:] * (1. - np_mask_rgb)
np_mask_grey = (np.sum(np_mask_rgb, axis=2)/3.)
np_src_grey = (np.sum(np_src_image, axis=2)/3.)
all_mask = np.ones((width, height), dtype=bool)
img_mask = np_mask_grey > 1e-6
ref_mask = np_mask_grey < 1e-3
windowed_image = _np_src_image * (1.-self._get_masked_window_rgb(np_mask_grey))
windowed_image /= np.max(windowed_image)
windowed_image += np.average(_np_src_image) * np_mask_rgb# / (1.-np.average(np_mask_rgb)) # rather than leave the masked area black, we get better results from fft by filling the average unmasked color
#windowed_image += np.average(_np_src_image) * (np_mask_rgb * (1.- np_mask_rgb)) / (1.-np.average(np_mask_rgb)) # compensate for darkening across the mask transition area
#_save_debug_img(windowed_image, "windowed_src_img")
src_fft = self._fft2(windowed_image) # get feature statistics from masked src img
src_dist = np.absolute(src_fft)
src_phase = src_fft / src_dist
#_save_debug_img(src_dist, "windowed_src_dist")
noise_window = self._get_gaussian_window(width, height, mode=1) # start with simple gaussian noise
noise_rgb = np.random.random_sample((width, height, num_channels))
noise_grey = (np.sum(noise_rgb, axis=2)/3.)
noise_rgb *= color_variation # the colorfulness of the starting noise is blended to greyscale with a parameter
for c in range(num_channels):
noise_rgb[:,:,c] += (1. - color_variation) * noise_grey
noise_fft = self._fft2(noise_rgb)
for c in range(num_channels):
noise_fft[:,:,c] *= noise_window
noise_rgb = np.real(self._ifft2(noise_fft))
shaped_noise_fft = self._fft2(noise_rgb)
shaped_noise_fft[:,:,:] = np.absolute(shaped_noise_fft[:,:,:])**2 * (src_dist ** noise_q) * src_phase # perform the actual shaping
brightness_variation = 0.#color_variation # todo: temporarily tieing brightness variation to color variation for now
contrast_adjusted_np_src = _np_src_image[:] * (brightness_variation + 1.) - brightness_variation * 2.
# scikit-image is used for histogram matching, very convenient!
shaped_noise = np.real(self._ifft2(shaped_noise_fft))
shaped_noise -= np.min(shaped_noise)
shaped_noise /= np.max(shaped_noise)
shaped_noise[img_mask,:] = skimage.exposure.match_histograms(shaped_noise[img_mask,:]**1., contrast_adjusted_np_src[ref_mask,:], channel_axis=1)
shaped_noise = _np_src_image[:] * (1. - np_mask_rgb) + shaped_noise * np_mask_rgb
#_save_debug_img(shaped_noise, "shaped_noise")
matched_noise = np.zeros((width, height, num_channels))
matched_noise = shaped_noise[:]
#matched_noise[all_mask,:] = skimage.exposure.match_histograms(shaped_noise[all_mask,:], _np_src_image[ref_mask,:], channel_axis=1)
#matched_noise = _np_src_image[:] * (1. - np_mask_rgb) + matched_noise * np_mask_rgb
#_save_debug_img(matched_noise, "matched_noise")
"""
todo:
color_variation doesnt have to be a single number, the overall color tone of the out-painted area could be param controlled
"""
return np.clip(matched_noise, 0., 1.)
def find_noise_for_image(self, model, device, init_image, prompt, steps=200, cond_scale=2.0, verbose=False, normalize=False, generation_callback=None):
image = np.array(init_image).astype(np.float32) / 255.0
image = image[None].transpose(0, 3, 1, 2)
image = torch.from_numpy(image)
image = 2. * image - 1.
image = image.to(device)
x = model.get_first_stage_encoding(model.encode_first_stage(image))
uncond = model.get_learned_conditioning([''])
cond = model.get_learned_conditioning([prompt])
s_in = x.new_ones([x.shape[0]])
dnw = K.external.CompVisDenoiser(model)
sigmas = dnw.get_sigmas(steps).flip(0)
if verbose:
print(sigmas)
for i in tqdm.trange(1, len(sigmas)):
x_in = torch.cat([x] * 2)
sigma_in = torch.cat([sigmas[i - 1] * s_in] * 2)
cond_in = torch.cat([uncond, cond])
c_out, c_in = [K.utils.append_dims(k, x_in.ndim) for k in dnw.get_scalings(sigma_in)]
if i == 1:
t = dnw.sigma_to_t(torch.cat([sigmas[i] * s_in] * 2))
else:
t = dnw.sigma_to_t(sigma_in)
eps = model.apply_model(x_in * c_in, t, cond=cond_in)
denoised_uncond, denoised_cond = (x_in + eps * c_out).chunk(2)
denoised = denoised_uncond + (denoised_cond - denoised_uncond) * cond_scale
if i == 1:
d = (x - denoised) / (2 * sigmas[i])
else:
d = (x - denoised) / sigmas[i - 1]
dt = sigmas[i] - sigmas[i - 1]
x = x + d * dt
return x / sigmas[-1]
def generate(self, prompt: str, init_img=None, init_mask=None, mask_mode='mask', resize_mode='resize', noise_mode='seed',
denoising_strength:float=0.8, ddim_steps=50, sampler_name='k_lms', n_iter=1, batch_size=1, cfg_scale=7.5, seed=None,
height=512, width=512, save_individual_images: bool = True, save_grid: bool = True, ddim_eta:float = 0.0):
seed = seed_to_int(seed)
image_dict = {
"seed": seed
}
# Init image is assumed to be a PIL image
init_img = self.resize_image('resize', init_img, width, height)
if sampler_name == 'PLMS':
sampler = PLMSSampler(self.model)
elif sampler_name == 'DDIM':
sampler = DDIMSampler(self.model)
elif sampler_name == 'k_dpm_2_a':
sampler = KDiffusionSampler(self.model,'dpm_2_ancestral')
elif sampler_name == 'k_dpm_2':
sampler = KDiffusionSampler(self.model,'dpm_2')
elif sampler_name == 'k_euler_a':
sampler = KDiffusionSampler(self.model,'euler_ancestral')
elif sampler_name == 'k_euler':
sampler = KDiffusionSampler(self.model,'euler')
elif sampler_name == 'k_heun':
sampler = KDiffusionSampler(self.model,'heun')
elif sampler_name == 'k_lms':
sampler = KDiffusionSampler(self.model,'lms')
else:
raise Exception("Unknown sampler: " + sampler_name)
torch_gc()
def process_init_mask(init_mask: PIL.Image):
if init_mask.mode == "RGBA":
init_mask = init_mask.convert('RGBA')
background = PIL.Image.new('RGBA', init_mask.size, (0, 0, 0))
init_mask = PIL.Image.alpha_composite(background, init_mask)
init_mask = init_mask.convert('RGB')
return init_mask
if mask_mode == "mask":
if init_mask:
init_mask = process_init_mask(init_mask)
elif mask_mode == "invert":
if init_mask:
init_mask = process_init_mask(init_mask)
init_mask = PIL.ImageOps.invert(init_mask)
elif mask_mode == "alpha":
init_img_transparency = init_img.split()[-1].convert('L')#.point(lambda x: 255 if x > 0 else 0, mode='1')
init_mask = init_img_transparency
init_mask = init_mask.convert("RGB")
init_mask = self.resize_image(resize_mode, init_mask, width, height)
init_mask = init_mask.convert("RGB")
assert 0. <= denoising_strength <= 1., 'can only work with strength in [0.0, 1.0]'
t_enc = int(denoising_strength * ddim_steps)
if init_mask is not None and (noise_mode == "matched" or noise_mode == "find_and_matched") and init_img is not None:
noise_q = 0.99
color_variation = 0.0
mask_blend_factor = 1.0
np_init = (np.asarray(init_img.convert("RGB"))/255.0).astype(np.float64) # annoyingly complex mask fixing
np_mask_rgb = 1. - (np.asarray(PIL.ImageOps.invert(init_mask).convert("RGB"))/255.0).astype(np.float64)
np_mask_rgb -= np.min(np_mask_rgb)
np_mask_rgb /= np.max(np_mask_rgb)
np_mask_rgb = 1. - np_mask_rgb
np_mask_rgb_hardened = 1. - (np_mask_rgb < 0.99).astype(np.float64)
blurred = skimage.filters.gaussian(np_mask_rgb_hardened[:], sigma=16., channel_axis=2, truncate=32.)
blurred2 = skimage.filters.gaussian(np_mask_rgb_hardened[:], sigma=16., channel_axis=2, truncate=32.)
#np_mask_rgb_dilated = np_mask_rgb + blurred # fixup mask todo: derive magic constants
#np_mask_rgb = np_mask_rgb + blurred
np_mask_rgb_dilated = np.clip((np_mask_rgb + blurred2) * 0.7071, 0., 1.)
np_mask_rgb = np.clip((np_mask_rgb + blurred) * 0.7071, 0., 1.)
noise_rgb = self.get_matched_noise(np_init, np_mask_rgb, noise_q, color_variation)
blend_mask_rgb = np.clip(np_mask_rgb_dilated,0.,1.) ** (mask_blend_factor)
noised = noise_rgb[:]
blend_mask_rgb **= (2.)
noised = np_init[:] * (1. - blend_mask_rgb) + noised * blend_mask_rgb
np_mask_grey = np.sum(np_mask_rgb, axis=2)/3.
ref_mask = np_mask_grey < 1e-3
all_mask = np.ones((height, width), dtype=bool)
noised[all_mask,:] = skimage.exposure.match_histograms(noised[all_mask,:]**1., noised[ref_mask,:], channel_axis=1)
init_img = PIL.Image.fromarray(np.clip(noised * 255., 0., 255.).astype(np.uint8), mode="RGB")
def init():
image = init_img.convert('RGB')
image = np.array(image).astype(np.float32) / 255.0
image = image[None].transpose(0, 3, 1, 2)
image = torch.from_numpy(image)
mask_channel = None
if init_mask:
alpha = self.resize_image(resize_mode, init_mask, width // 8, height // 8)
mask_channel = alpha.split()[-1]
mask = None
if mask_channel is not None:
mask = np.array(mask_channel).astype(np.float32) / 255.0
mask = (1 - mask)
mask = np.tile(mask, (4, 1, 1))
mask = mask[None].transpose(0, 1, 2, 3)
mask = torch.from_numpy(mask).to(self.model.device)
init_image = 2. * image - 1.
init_image = init_image.to(self.model.device)
init_latent = self.model.get_first_stage_encoding(self.model.encode_first_stage(init_image)) # move to latent space
return init_latent, mask,
def sample(init_data, x, conditioning, unconditional_conditioning, sampler_name):
t_enc_steps = t_enc
obliterate = False
if ddim_steps == t_enc_steps:
t_enc_steps = t_enc_steps - 1
obliterate = True
if sampler_name != 'DDIM':
x0, z_mask = init_data
sigmas = sampler.model_wrap.get_sigmas(ddim_steps)
noise = x * sigmas[ddim_steps - t_enc_steps - 1]
xi = x0 + noise
# Obliterate masked image
if z_mask is not None and obliterate:
random = torch.randn(z_mask.shape, device=xi.device)
xi = (z_mask * noise) + ((1-z_mask) * xi)
sigma_sched = sigmas[ddim_steps - t_enc_steps - 1:]
model_wrap_cfg = CFGMaskedDenoiser(sampler.model_wrap)
samples_ddim = K.sampling.__dict__[f'sample_{sampler.get_sampler_name()}'](model_wrap_cfg, xi, sigma_sched,
extra_args={'cond': conditioning, 'uncond': unconditional_conditioning,
'cond_scale': cfg_scale, 'mask': z_mask, 'x0': x0, 'xi': xi}, disable=False)
else:
x0, z_mask = init_data
sampler.make_schedule(ddim_num_steps=ddim_steps, ddim_eta=0.0, verbose=False)
z_enc = sampler.stochastic_encode(x0, torch.tensor([t_enc_steps]*batch_size).to(self.model.device))
# Obliterate masked image
if z_mask is not None and obliterate:
random = torch.randn(z_mask.shape, device=z_enc.device)
z_enc = (z_mask * random) + ((1-z_mask) * z_enc)
# decode it
samples_ddim = sampler.decode(z_enc, conditioning, t_enc_steps,
unconditional_guidance_scale=cfg_scale,
unconditional_conditioning=unconditional_conditioning,
z_mask=z_mask, x0=x0)
return samples_ddim
torch_gc()
if self.load_concepts and self.concepts_dir is not None:
prompt_tokens = re.findall('<([a-zA-Z0-9-]+)>', prompt)
if prompt_tokens:
self.process_prompt_tokens(prompt_tokens)
os.makedirs(self.output_dir, exist_ok=True)
sample_path = os.path.join(self.output_dir, "samples")
os.makedirs(sample_path, exist_ok=True)
if self.verify_input:
try:
check_prompt_length(self.model, prompt, self.comments)
except:
import traceback
print("Error verifying input:", file=sys.stderr)
print(traceback.format_exc(), file=sys.stderr)
all_prompts = batch_size * n_iter * [prompt]
all_seeds = [seed + x for x in range(len(all_prompts))]
precision_scope = torch.autocast if self.auto_cast else nullcontext
with torch.no_grad(), precision_scope("cuda"):
for n in range(n_iter):
print(f"Iteration: {n+1}/{n_iter}")
prompts = all_prompts[n * batch_size:(n + 1) * batch_size]
seeds = all_seeds[n * batch_size:(n + 1) * batch_size]
uc = self.model.get_learned_conditioning(len(prompts) * [''])
if isinstance(prompts, tuple):
prompts = list(prompts)
c = self.model.get_learned_conditioning(prompts)
opt_C = 4
opt_f = 8
shape = [opt_C, height // opt_f, width // opt_f]
x = self.create_random_tensors(shape, seeds=seeds)
init_data = init()
samples_ddim = sample(init_data=init_data, x=x, conditioning=c, unconditional_conditioning=uc, sampler_name=sampler_name)
x_samples_ddim = self.model.decode_first_stage(samples_ddim)
x_samples_ddim = torch.clamp((x_samples_ddim + 1.0) / 2.0, min=0.0, max=1.0)
for i, x_sample in enumerate(x_samples_ddim):
sanitized_prompt = slugify(prompts[i])
full_path = os.path.join(os.getcwd(), sample_path)
sample_path_i = sample_path
base_count = get_next_sequence_number(sample_path_i)
filename = f"{base_count:05}-{ddim_steps}_{sampler_name}_{seeds[i]}_{sanitized_prompt}"[:200-len(full_path)]
x_sample = 255. * rearrange(x_sample.cpu().numpy(), 'c h w -> h w c')
x_sample = x_sample.astype(np.uint8)
image = PIL.Image.fromarray(x_sample)
image_dict['image'] = image
self.images.append(image_dict)
if save_individual_images:
path = os.path.join(sample_path, filename + '.' + self.save_extension)
success = save_sample(image, filename, sample_path_i, self.save_extension)
if success:
if self.output_file_path:
self.output_images.append(path)
else:
self.output_images.append(image)
else:
return
self.info = f"""
{prompt}
Steps: {ddim_steps}, Sampler: {sampler_name}, CFG scale: {cfg_scale}, Seed: {seed}
""".strip()
self.stats = f'''
'''
for comment in self.comments:
self.info += "\n\n" + comment
torch_gc()
del sampler
return

201
scripts/nataili/inference/compvis/txt2img.py
View File

@ -1,201 +0,0 @@
import os
import re
import sys
from contextlib import contextmanager, nullcontext
import numpy as np
import PIL
import torch
from einops import rearrange
from ldm.models.diffusion.ddim import DDIMSampler
from ldm.models.diffusion.kdiffusion import KDiffusionSampler
from ldm.models.diffusion.plms import PLMSSampler
from nataili.util.cache import torch_gc
from nataili.util.check_prompt_length import check_prompt_length
from nataili.util.get_next_sequence_number import get_next_sequence_number
from nataili.util.image_grid import image_grid
from nataili.util.load_learned_embed_in_clip import load_learned_embed_in_clip
from nataili.util.save_sample import save_sample
from nataili.util.seed_to_int import seed_to_int
from slugify import slugify
class txt2img:
def __init__(self, model, device, output_dir, save_extension='jpg',
output_file_path=False, load_concepts=False, concepts_dir=None,
verify_input=True, auto_cast=True):
self.model = model
self.output_dir = output_dir
self.output_file_path = output_file_path
self.save_extension = save_extension
self.load_concepts = load_concepts
self.concepts_dir = concepts_dir
self.verify_input = verify_input
self.auto_cast = auto_cast
self.device = device
self.comments = []
self.output_images = []
self.info = ''
self.stats = ''
self.images = []
def create_random_tensors(self, shape, seeds):
xs = []
for seed in seeds:
torch.manual_seed(seed)
# randn results depend on device; gpu and cpu get different results for same seed;
# the way I see it, it's better to do this on CPU, so that everyone gets same result;
# but the original script had it like this so i do not dare change it for now because
# it will break everyone's seeds.
xs.append(torch.randn(shape, device=self.device))
x = torch.stack(xs)
return x
def process_prompt_tokens(self, prompt_tokens):
# compviz codebase
tokenizer = self.model.cond_stage_model.tokenizer
text_encoder = self.model.cond_stage_model.transformer
# diffusers codebase
#tokenizer = pipe.tokenizer
#text_encoder = pipe.text_encoder
ext = ('.pt', '.bin')
for token_name in prompt_tokens:
embedding_path = os.path.join(self.concepts_dir, token_name)
if os.path.exists(embedding_path):
for files in os.listdir(embedding_path):
if files.endswith(ext):
load_learned_embed_in_clip(f"{os.path.join(embedding_path, files)}", text_encoder, tokenizer, f"<{token_name}>")
else:
print(f"Concept {token_name} not found in {self.concepts_dir}")
del tokenizer, text_encoder
return
del tokenizer, text_encoder
def generate(self, prompt: str, ddim_steps=50, sampler_name='k_lms', n_iter=1, batch_size=1, cfg_scale=7.5, seed=None,
height=512, width=512, save_individual_images: bool = True, save_grid: bool = True, ddim_eta:float = 0.0):
seed = seed_to_int(seed)
image_dict = {
"seed": seed
}
negprompt = ''
if '###' in prompt:
prompt, negprompt = prompt.split('###', 1)
prompt = prompt.strip()
negprompt = negprompt.strip()
if sampler_name == 'PLMS':
sampler = PLMSSampler(self.model)
elif sampler_name == 'DDIM':
sampler = DDIMSampler(self.model)
elif sampler_name == 'k_dpm_2_a':
sampler = KDiffusionSampler(self.model,'dpm_2_ancestral')
elif sampler_name == 'k_dpm_2':
sampler = KDiffusionSampler(self.model,'dpm_2')
elif sampler_name == 'k_euler_a':
sampler = KDiffusionSampler(self.model,'euler_ancestral')
elif sampler_name == 'k_euler':
sampler = KDiffusionSampler(self.model,'euler')
elif sampler_name == 'k_heun':
sampler = KDiffusionSampler(self.model,'heun')
elif sampler_name == 'k_lms':
sampler = KDiffusionSampler(self.model,'lms')
else:
raise Exception("Unknown sampler: " + sampler_name)
def sample(init_data, x, conditioning, unconditional_conditioning, sampler_name):
samples_ddim, _ = sampler.sample(S=ddim_steps, conditioning=conditioning, unconditional_guidance_scale=cfg_scale,
unconditional_conditioning=unconditional_conditioning, x_T=x)
return samples_ddim
torch_gc()
if self.load_concepts and self.concepts_dir is not None:
prompt_tokens = re.findall('<([a-zA-Z0-9-]+)>', prompt)
if prompt_tokens:
self.process_prompt_tokens(prompt_tokens)
os.makedirs(self.output_dir, exist_ok=True)
sample_path = os.path.join(self.output_dir, "samples")
os.makedirs(sample_path, exist_ok=True)
if self.verify_input:
try:
check_prompt_length(self.model, prompt, self.comments)
except:
import traceback
print("Error verifying input:", file=sys.stderr)
print(traceback.format_exc(), file=sys.stderr)
all_prompts = batch_size * n_iter * [prompt]
all_seeds = [seed + x for x in range(len(all_prompts))]
precision_scope = torch.autocast if self.auto_cast else nullcontext
with torch.no_grad(), precision_scope("cuda"):
for n in range(n_iter):
print(f"Iteration: {n+1}/{n_iter}")
prompts = all_prompts[n * batch_size:(n + 1) * batch_size]
seeds = all_seeds[n * batch_size:(n + 1) * batch_size]
uc = self.model.get_learned_conditioning(len(prompts) * [negprompt])
if isinstance(prompts, tuple):
prompts = list(prompts)
c = self.model.get_learned_conditioning(prompts)
opt_C = 4
opt_f = 8
shape = [opt_C, height // opt_f, width // opt_f]
x = self.create_random_tensors(shape, seeds=seeds)
samples_ddim = sample(init_data=None, x=x, conditioning=c, unconditional_conditioning=uc, sampler_name=sampler_name)
x_samples_ddim = self.model.decode_first_stage(samples_ddim)
x_samples_ddim = torch.clamp((x_samples_ddim + 1.0) / 2.0, min=0.0, max=1.0)
for i, x_sample in enumerate(x_samples_ddim):
sanitized_prompt = slugify(prompts[i])
full_path = os.path.join(os.getcwd(), sample_path)
sample_path_i = sample_path
base_count = get_next_sequence_number(sample_path_i)
filename = f"{base_count:05}-{ddim_steps}_{sampler_name}_{seeds[i]}_{sanitized_prompt}"[:200-len(full_path)]
x_sample = 255. * rearrange(x_sample.cpu().numpy(), 'c h w -> h w c')
x_sample = x_sample.astype(np.uint8)
image = PIL.Image.fromarray(x_sample)
image_dict['image'] = image
self.images.append(image_dict)
if save_individual_images:
path = os.path.join(sample_path, filename + '.' + self.save_extension)
success = save_sample(image, filename, sample_path_i, self.save_extension)
if success:
if self.output_file_path:
self.output_images.append(path)
else:
self.output_images.append(image)
else:
return
self.info = f"""
{prompt}
Steps: {ddim_steps}, Sampler: {sampler_name}, CFG scale: {cfg_scale}, Seed: {seed}
""".strip()
self.stats = f'''
'''
for comment in self.comments:
self.info += "\n\n" + comment
torch_gc()
del sampler
return

0
scripts/nataili/inference/diffusers/__init__.py
View File

458
scripts/nataili/model_manager.py
View File

@ -1,458 +0,0 @@
import os
import json
import shutil
import zipfile
import requests
import git
import torch
import hashlib
from ldm.util import instantiate_from_config
from omegaconf import OmegaConf
from transformers import logging
from basicsr.archs.rrdbnet_arch import RRDBNet
from gfpgan import GFPGANer
from realesrgan import RealESRGANer
from ldm.models.blip import blip_decoder
from tqdm import tqdm
import open_clip
import clip
from nataili.util.cache import torch_gc
from nataili.util import logger
logging.set_verbosity_error()
models = json.load(open('./db.json'))
dependencies = json.load(open('./db_dep.json'))
remote_models = "https://raw.githubusercontent.com/Sygil-Dev/nataili-model-reference/main/db.json"
remote_dependencies = "https://raw.githubusercontent.com/Sygil-Dev/nataili-model-reference/main/db_dep.json"
class ModelManager():
def __init__(self, hf_auth=None, download=True):
if download:
try:
logger.init("Model Reference", status="Downloading")
r = requests.get(remote_models)
self.models = r.json()
r = requests.get(remote_dependencies)
self.dependencies = json.load(open('./db_dep.json'))
logger.init_ok("Model Reference", status="OK")
except:
logger.init_err("Model Reference", status="Download Error")
self.models = json.load(open('./db.json'))
self.dependencies = json.load(open('./db_dep.json'))
logger.init_warn("Model Reference", status="Local")
self.available_models = []
self.tainted_models = []
self.available_dependencies = []
self.loaded_models = {}
self.hf_auth = None
self.set_authentication(hf_auth)
def init(self):
dependencies_available = []
for dependency in self.dependencies:
if self.check_available(self.get_dependency_files(dependency)):
dependencies_available.append(dependency)
self.available_dependencies = dependencies_available
models_available = []
for model in self.models:
if self.check_available(self.get_model_files(model)):
models_available.append(model)
self.available_models = models_available
if self.hf_auth is not None:
if 'username' not in self.hf_auth and 'password' not in self.hf_auth:
raise ValueError('hf_auth must contain username and password')
else:
if self.hf_auth['username'] == '' or self.hf_auth['password'] == '':
raise ValueError('hf_auth must contain username and password')
return True
def set_authentication(self, hf_auth=None):
# We do not let No authentication override previously set auth
if not hf_auth and self.hf_auth:
return
self.hf_auth = hf_auth
def get_model(self, model_name):
return self.models.get(model_name)
def get_filtered_models(self, **kwargs):
'''Get all model names.
Can filter based on metadata of the model reference db
'''
filtered_models = self.models
for keyword in kwargs:
iterating_models = filtered_models.copy()
filtered_models = {}
for model in iterating_models:
# logger.debug([keyword,iterating_models[model].get(keyword),kwargs[keyword]])
if iterating_models[model].get(keyword) == kwargs[keyword]:
filtered_models[model] = iterating_models[model]
return filtered_models
def get_filtered_model_names(self, **kwargs):
filtered_models = self.get_filtered_models(**kwargs)
return list(filtered_models.keys())
def get_dependency(self, dependency_name):
return self.dependencies[dependency_name]
def get_model_files(self, model_name):
return self.models[model_name]['config']['files']
def get_dependency_files(self, dependency_name):
return self.dependencies[dependency_name]['config']['files']
def get_model_download(self, model_name):
return self.models[model_name]['config']['download']
def get_dependency_download(self, dependency_name):
return self.dependencies[dependency_name]['config']['download']
def get_available_models(self):
return self.available_models
def get_available_dependencies(self):
return self.available_dependencies
def get_loaded_models(self):
return self.loaded_models
def get_loaded_models_names(self):
return list(self.loaded_models.keys())
def get_loaded_model(self, model_name):
return self.loaded_models[model_name]
def unload_model(self, model_name):
if model_name in self.loaded_models:
del self.loaded_models[model_name]
return True
return False
def unload_all_models(self):
for model in self.loaded_models:
del self.loaded_models[model]
return True
def taint_model(self,model_name):
'''Marks a model as not valid by remiving it from available_models'''
if model_name in self.available_models:
self.available_models.remove(model_name)
self.tainted_models.append(model_name)
def taint_models(self, models):
for model in models:
self.taint_model(model)
def load_model_from_config(self, model_path='', config_path='', map_location="cpu"):
config = OmegaConf.load(config_path)
pl_sd = torch.load(model_path, map_location=map_location)
if "global_step" in pl_sd:
logger.info(f"Global Step: {pl_sd['global_step']}")
sd = pl_sd["state_dict"]
model = instantiate_from_config(config.model)
m, u = model.load_state_dict(sd, strict=False)
model = model.eval()
del pl_sd, sd, m, u
return model
def load_ckpt(self, model_name='', precision='half', gpu_id=0):
ckpt_path = self.get_model_files(model_name)[0]['path']
config_path = self.get_model_files(model_name)[1]['path']
model = self.load_model_from_config(model_path=ckpt_path, config_path=config_path)
device = torch.device(f"cuda:{gpu_id}")
model = (model if precision=='full' else model.half()).to(device)
torch_gc()
return {'model': model, 'device': device}
def load_realesrgan(self, model_name='', precision='half', gpu_id=0):
RealESRGAN_models = {
'RealESRGAN_x4plus': RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4),
'RealESRGAN_x4plus_anime_6B': RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=6, num_grow_ch=32, scale=4)
}
model_path = self.get_model_files(model_name)[0]['path']
device = torch.device(f"cuda:{gpu_id}")
model = RealESRGANer(scale=2, model_path=model_path, model=RealESRGAN_models[models[model_name]['name']],
pre_pad=0, half=True if precision == 'half' else False, device=device)
return {'model': model, 'device': device}
def load_gfpgan(self, model_name='', gpu_id=0):
model_path = self.get_model_files(model_name)[0]['path']
device = torch.device(f"cuda:{gpu_id}")
model = GFPGANer(model_path=model_path, upscale=1, arch='clean',
channel_multiplier=2, bg_upsampler=None, device=device)
return {'model': model, 'device': device}
def load_blip(self, model_name='', precision='half', gpu_id=0, blip_image_eval_size=512, vit='base'):
# vit = 'base' or 'large'
model_path = self.get_model_files(model_name)[0]['path']
device = torch.device(f"cuda:{gpu_id}")
model = blip_decoder(pretrained=model_path,
med_config="configs/blip/med_config.json",
image_size=blip_image_eval_size, vit=vit)
model = model.eval()
model = (model if precision=='full' else model.half()).to(device)
return {'model': model, 'device': device}
def load_open_clip(self, model_name='', precision='half', gpu_id=0):
pretrained = self.get_model(model_name)['pretrained_name']
device = torch.device(f"cuda:{gpu_id}")
model, _, preprocesses = open_clip.create_model_and_transforms(model_name, pretrained=pretrained, cache_dir='models/clip')
model = model.eval()
model = (model if precision=='full' else model.half()).to(device)
return {'model': model, 'device': device, 'preprocesses': preprocesses}
def load_clip(self, model_name='', precision='half', gpu_id=0):
device = torch.device(f"cuda:{gpu_id}")
model, preprocesses = clip.load(model_name, device=device, download_root='models/clip')
model = model.eval()
model = (model if precision=='full' else model.half()).to(device)
return {'model': model, 'device': device, 'preprocesses': preprocesses}
def load_model(self, model_name='', precision='half', gpu_id=0):
if model_name not in self.available_models:
return False
if self.models[model_name]['type'] == 'ckpt':
self.loaded_models[model_name] = self.load_ckpt(model_name, precision, gpu_id)
return True
elif self.models[model_name]['type'] == 'realesrgan':
self.loaded_models[model_name] = self.load_realesrgan(model_name, precision, gpu_id)
return True
elif self.models[model_name]['type'] == 'gfpgan':
self.loaded_models[model_name] = self.load_gfpgan(model_name, gpu_id)
return True
elif self.models[model_name]['type'] == 'blip':
self.loaded_models[model_name] = self.load_blip(model_name, precision, gpu_id, 512, 'base')
return True
elif self.models[model_name]['type'] == 'open_clip':
self.loaded_models[model_name] = self.load_open_clip(model_name, precision, gpu_id)
return True
elif self.models[model_name]['type'] == 'clip':
self.loaded_models[model_name] = self.load_clip(model_name, precision, gpu_id)
return True
else:
return False
def validate_model(self, model_name):
files = self.get_model_files(model_name)
all_ok = True
for file_details in files:
if not self.check_file_available(file_details['path']):
return False
if not self.validate_file(file_details):
return False
return True
def validate_file(self, file_details):
if 'md5sum' in file_details:
file_name = file_details['path']
logger.debug(f"Getting md5sum of {file_name}")
with open(file_name, 'rb') as file_to_check:
file_hash = hashlib.md5()
while chunk := file_to_check.read(8192):
file_hash.update(chunk)
if file_details['md5sum'] != file_hash.hexdigest():
return False
return True
def check_file_available(self, file_path):
return os.path.exists(file_path)
def check_available(self, files):
available = True
for file in files:
if not self.check_file_available(file['path']):
available = False
return available
def download_file(self, url, file_path):
# make directory
os.makedirs(os.path.dirname(file_path), exist_ok=True)
pbar_desc = file_path.split('/')[-1]
r = requests.get(url, stream=True)
with open(file_path, 'wb') as f:
with tqdm(
# all optional kwargs
unit='B', unit_scale=True, unit_divisor=1024, miniters=1,
desc=pbar_desc, total=int(r.headers.get('content-length', 0))
) as pbar:
for chunk in r.iter_content(chunk_size=16*1024):
if chunk:
f.write(chunk)
pbar.update(len(chunk))
def download_model(self, model_name):
if model_name in self.available_models:
logger.info(f"{model_name} is already available.")
return True
download = self.get_model_download(model_name)
files = self.get_model_files(model_name)
for i in range(len(download)):
file_path = f"{download[i]['file_path']}/{download[i]['file_name']}" if 'file_path' in download[i] else files[i]['path']
if 'file_url' in download[i]:
download_url = download[i]['file_url']
if 'hf_auth' in download[i]:
username = self.hf_auth['username']
password = self.hf_auth['password']
download_url = download_url.format(username=username, password=password)
if 'file_name' in download[i]:
download_name = download[i]['file_name']
if 'file_path' in download[i]:
download_path = download[i]['file_path']
if 'manual' in download[i]:
logger.warning(f"The model {model_name} requires manual download from {download_url}. Please place it in {download_path}/{download_name} then press ENTER to continue...")
input('')
continue
# TODO: simplify
if "file_content" in download[i]:
file_content = download[i]['file_content']
logger.info(f"writing {file_content} to {file_path}")
# make directory download_path
os.makedirs(download_path, exist_ok=True)
# write file_content to download_path/download_name
with open(os.path.join(download_path, download_name), 'w') as f:
f.write(file_content)
elif 'symlink' in download[i]:
logger.info(f"symlink {file_path} to {download[i]['symlink']}")
symlink = download[i]['symlink']
# make directory symlink
os.makedirs(download_path, exist_ok=True)
# make symlink from download_path/download_name to symlink
os.symlink(symlink, os.path.join(download_path, download_name))
elif 'git' in download[i]:
logger.info(f"git clone {download_url} to {file_path}")
# make directory download_path
os.makedirs(file_path, exist_ok=True)
git.Git(file_path).clone(download_url)
if 'post_process' in download[i]:
for post_process in download[i]['post_process']:
if 'delete' in post_process:
# delete folder post_process['delete']
logger.info(f"delete {post_process['delete']}")
try:
shutil.rmtree(post_process['delete'])
except PermissionError as e:
logger.error(f"[!] Something went wrong while deleting the `{post_process['delete']}`. Please delete it manually.")
logger.error("PermissionError: ", e)
else:
if not self.check_file_available(file_path) or model_name in self.tainted_models:
logger.debug(f'Downloading {download_url} to {file_path}')
self.download_file(download_url, file_path)
if not self.validate_model(model_name):
return False
if model_name in self.tainted_models:
self.tainted_models.remove(model_name)
self.init()
return True
def download_dependency(self, dependency_name):
if dependency_name in self.available_dependencies:
logger.info(f"{dependency_name} is already installed.")
return True
download = self.get_dependency_download(dependency_name)
files = self.get_dependency_files(dependency_name)
for i in range(len(download)):
if "git" in download[i]:
logger.warning("git download not implemented yet")
break
file_path = files[i]['path']
if 'file_url' in download[i]:
download_url = download[i]['file_url']
if 'file_name' in download[i]:
download_name = download[i]['file_name']
if 'file_path' in download[i]:
download_path = download[i]['file_path']
logger.debug(download_name)
if "unzip" in download[i]:
zip_path = f'temp/{download_name}.zip'
# os dirname zip_path
# mkdir temp
os.makedirs("temp", exist_ok=True)
self.download_file(download_url, zip_path)
logger.info(f"unzip {zip_path}")
with zipfile.ZipFile(zip_path, 'r') as zip_ref:
zip_ref.extractall('temp/')
# move temp/sd-concepts-library-main/sd-concepts-library to download_path
logger.info(f"move temp/{download_name}-main/{download_name} to {download_path}")
shutil.move(f"temp/{download_name}-main/{download_name}", download_path)
logger.info(f"delete {zip_path}")
os.remove(zip_path)
logger.info(f"delete temp/{download_name}-main/")
shutil.rmtree(f"temp/{download_name}-main")
else:
if not self.check_file_available(file_path):
logger.init(f'{file_path}', status="Downloading")
self.download_file(download_url, file_path)
self.init()
return True
def download_all_models(self):
for model in self.get_filtered_model_names(download_all = True):
if not self.check_model_available(model):
logger.init(f"{model}", status="Downloading")
self.download_model(model)
else:
logger.info(f"{model} is already downloaded.")
return True
def download_all_dependencies(self):
for dependency in self.dependencies:
if not self.check_dependency_available(dependency):
logger.init(f"{dependency}",status="Downloading")
self.download_dependency(dependency)
else:
logger.info(f"{dependency} is already installed.")
return True
def download_all(self):
self.download_all_dependencies()
self.download_all_models()
return True
def check_all_available(self):
for model in self.models:
if not self.check_available(self.get_model_files(model)):
return False
for dependency in self.dependencies:
if not self.check_available(self.get_dependency_files(dependency)):
return False
return True
def check_model_available(self, model_name):
if model_name not in self.models:
return False
return self.check_available(self.get_model_files(model_name))
def check_dependency_available(self, dependency_name):
if dependency_name not in self.dependencies:
return False
return self.check_available(self.get_dependency_files(dependency_name))
def check_all_available(self):
for model in self.models:
if not self.check_model_available(model):
return False
for dependency in self.dependencies:
if not self.check_dependency_available(dependency):
return False
return True

0
scripts/nataili/postprocess/__init__.py
View File

48
scripts/nataili/postprocess/upscaler.py
View File

@ -1,48 +0,0 @@
# Class realesrgan
# Inputs:
# - model
# - device
# - output_dir
# - output_ext
# outupts:
# - output_images
import PIL
from torchvision import transforms
import numpy as np
import os
import cv2
from nataili.util.save_sample import save_sample
class realesrgan:
def __init__(self, model, device, output_dir, output_ext='jpg'):
self.model = model
self.device = device
self.output_dir = output_dir
self.output_ext = output_ext
self.output_images = []
def generate(self, input_image):
# load image
img = cv2.imread(input_image, cv2.IMREAD_UNCHANGED)
if len(img.shape) == 3 and img.shape[2] == 4:
img_mode = 'RGBA'
else:
img_mode = None
# upscale
output, _ = self.model.enhance(img)
if img_mode == 'RGBA': # RGBA images should be saved in png format
self.output_ext = 'png'
esrgan_sample = output[:,:,::-1]
esrgan_image = PIL.Image.fromarray(esrgan_sample)
# append model name to output image name
filename = os.path.basename(input_image)
filename = os.path.splitext(filename)[0]
filename = f'{filename}_esrgan'
filename_with_ext = f'{filename}.{self.output_ext}'
output_image = os.path.join(self.output_dir, filename_with_ext)
save_sample(esrgan_image, filename, self.output_dir, self.output_ext)
self.output_images.append(output_image)
return

0
scripts/nataili/upscalers/__init__.py
View File

48
scripts/nataili/upscalers/realesrgan.py
View File

@ -1,48 +0,0 @@
# Class realesrgan
# Inputs:
# - model
# - device
# - output_dir
# - output_ext
# outupts:
# - output_images
import PIL
from torchvision import transforms
import numpy as np
import os
import cv2
from nataili.util.save_sample import save_sample
class realesrgan:
def __init__(self, model, device, output_dir, output_ext='jpg'):
self.model = model
self.device = device
self.output_dir = output_dir
self.output_ext = output_ext
self.output_images = []
def generate(self, input_image):
# load image
img = cv2.imread(input_image, cv2.IMREAD_UNCHANGED)
if len(img.shape) == 3 and img.shape[2] == 4:
img_mode = 'RGBA'
else:
img_mode = None
# upscale
output, _ = self.model.enhance(img)
if img_mode == 'RGBA': # RGBA images should be saved in png format
self.output_ext = 'png'
esrgan_sample = output[:,:,::-1]
esrgan_image = PIL.Image.fromarray(esrgan_sample)
# append model name to output image name
filename = os.path.basename(input_image)
filename = os.path.splitext(filename)[0]
filename = f'{filename}_esrgan'
filename_with_ext = f'{filename}.{self.output_ext}'
output_image = os.path.join(self.output_dir, filename_with_ext)
save_sample(esrgan_image, filename, self.output_dir, self.output_ext)
self.output_images.append(output_image)
return

1
scripts/nataili/util/__init__.py
View File

@ -1 +0,0 @@
from nataili.util.logger import logger,set_logger_verbosity, quiesce_logger, test_logger

16
scripts/nataili/util/cache.py
View File

@ -1,16 +0,0 @@
import gc
import torch
import threading
import pynvml
import time
with torch.no_grad():
def torch_gc():
for _ in range(2):
gc.collect()
torch.cuda.empty_cache()
torch.cuda.ipc_collect()
torch.cuda.synchronize()
torch.cuda.reset_peak_memory_stats()
torch.cuda.reset_accumulated_memory_stats()

18
scripts/nataili/util/check_prompt_length.py
View File

@ -1,18 +0,0 @@
def check_prompt_length(model, prompt, comments):
"""this function tests if prompt is too long, and if so, adds a message to comments"""
tokenizer = model.cond_stage_model.tokenizer
max_length = model.cond_stage_model.max_length
info = model.cond_stage_model.tokenizer([prompt], truncation=True, max_length=max_length,
return_overflowing_tokens=True, padding="max_length", return_tensors="pt")
ovf = info['overflowing_tokens'][0]
overflowing_count = ovf.shape[0]
if overflowing_count == 0:
return
vocab = {v: k for k, v in tokenizer.get_vocab().items()}
overflowing_words = [vocab.get(int(x), "") for x in ovf]
overflowing_text = tokenizer.convert_tokens_to_string(''.join(overflowing_words))
comments.append(f"Warning: too many input tokens; some ({len(overflowing_words)}) have been truncated:\n{overflowing_text}\n")
del tokenizer

22
scripts/nataili/util/get_next_sequence_number.py
View File

@ -1,22 +0,0 @@
from pathlib import Path
def get_next_sequence_number(path, prefix=''):
"""
Determines and returns the next sequence number to use when saving an
image in the specified directory.
If a prefix is given, only consider files whose names start with that
prefix, and strip the prefix from filenames before extracting their
sequence number.
The sequence starts at 0.
"""
result = -1
for p in Path(path).iterdir():
if p.name.endswith(('.png', '.jpg')) and p.name.startswith(prefix):
tmp = p.name[len(prefix):]
try:
result = max(int(tmp.split('-')[0]), result)
except ValueError:
pass
return result + 1

21
scripts/nataili/util/image_grid.py
View File

@ -1,21 +0,0 @@
import math
import PIL
def image_grid(imgs, n_rows=None):
if n_rows is not None:
rows = n_rows
else:
rows = math.sqrt(len(imgs))
rows = round(rows)
cols = math.ceil(len(imgs) / rows)
w, h = imgs[0].size
grid = PIL.Image.new('RGB', size=(cols * w, rows * h), color='black')
for i, img in enumerate(imgs):
grid.paste(img, box=(i % cols * w, i // cols * h))
return grid

40
scripts/nataili/util/load_learned_embed_in_clip.py
View File

@ -1,40 +0,0 @@
import os
import torch
def load_learned_embed_in_clip(learned_embeds_path, text_encoder, tokenizer, token=None):
loaded_learned_embeds = torch.load(learned_embeds_path, map_location="cpu")
# separate token and the embeds
if learned_embeds_path.endswith('.pt'):
# old format
# token = * so replace with file directory name when converting
trained_token = os.path.basename(learned_embeds_path)
params_dict = {
trained_token: torch.tensor(list(loaded_learned_embeds['string_to_param'].items())[0][1])
}
learned_embeds_path = os.path.splitext(learned_embeds_path)[0] + '.bin'
torch.save(params_dict, learned_embeds_path)
loaded_learned_embeds = torch.load(learned_embeds_path, map_location="cpu")
trained_token = list(loaded_learned_embeds.keys())[0]
embeds = loaded_learned_embeds[trained_token]
elif learned_embeds_path.endswith('.bin'):
trained_token = list(loaded_learned_embeds.keys())[0]
embeds = loaded_learned_embeds[trained_token]
embeds = loaded_learned_embeds[trained_token]
# cast to dtype of text_encoder
dtype = text_encoder.get_input_embeddings().weight.dtype
embeds.to(dtype)
# add the token in tokenizer
token = token if token is not None else trained_token
num_added_tokens = tokenizer.add_tokens(token)
# resize the token embeddings
text_encoder.resize_token_embeddings(len(tokenizer))
# get the id for the token and assign the embeds
token_id = tokenizer.convert_tokens_to_ids(token)
text_encoder.get_input_embeddings().weight.data[token_id] = embeds
return token

102
scripts/nataili/util/logger.py
View File

@ -1,102 +0,0 @@
import sys
from functools import partialmethod
from loguru import logger
STDOUT_LEVELS = ["GENERATION", "PROMPT"]
INIT_LEVELS = ["INIT", "INIT_OK", "INIT_WARN", "INIT_ERR"]
MESSAGE_LEVELS = ["MESSAGE"]
# By default we're at error level or higher
verbosity = 20
quiet = 0
def set_logger_verbosity(count):
global verbosity
# The count comes reversed. So count = 0 means minimum verbosity
# While count 5 means maximum verbosity
# So the more count we have, the lowe we drop the versbosity maximum
verbosity = 20 - (count * 10)
def quiesce_logger(count):
global quiet
# The bigger the count, the more silent we want our logger
quiet = count * 10
def is_stdout_log(record):
if record["level"].name not in STDOUT_LEVELS:
return(False)
if record["level"].no < verbosity + quiet:
return(False)
return(True)
def is_init_log(record):
if record["level"].name not in INIT_LEVELS:
return(False)
if record["level"].no < verbosity + quiet:
return(False)
return(True)
def is_msg_log(record):
if record["level"].name not in MESSAGE_LEVELS:
return(False)
if record["level"].no < verbosity + quiet:
return(False)
return(True)
def is_stderr_log(record):
if record["level"].name in STDOUT_LEVELS + INIT_LEVELS + MESSAGE_LEVELS:
return(False)
if record["level"].no < verbosity + quiet:
return(False)
return(True)
def test_logger():
logger.generation("This is a generation message\nIt is typically multiline\nThee Lines".encode("unicode_escape").decode("utf-8"))
logger.prompt("This is a prompt message")
logger.debug("Debug Message")
logger.info("Info Message")
logger.warning("Info Warning")
logger.error("Error Message")
logger.critical("Critical Message")
logger.init("This is an init message", status="Starting")
logger.init_ok("This is an init message", status="OK")
logger.init_warn("This is an init message", status="Warning")
logger.init_err("This is an init message", status="Error")
logger.message("This is user message")
sys.exit()
logfmt = "<level>{level: <10}</level> | <green>{time:YYYY-MM-DD HH:mm:ss}</green> | <green>{name}</green>:<green>{function}</green>:<green>{line}</green> - <level>{message}</level>"
genfmt = "<level>{level: <10}</level> @ <green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{message}</level>"
initfmt = "<magenta>INIT </magenta> | <level>{extra[status]: <11}</level> | <magenta>{message}</magenta>"
msgfmt = "<level>{level: <10}</level> | <level>{message}</level>"
try:
logger.level("GENERATION", no=24, color="<cyan>")
logger.level("PROMPT", no=23, color="<yellow>")
logger.level("INIT", no=31, color="<white>")
logger.level("INIT_OK", no=31, color="<green>")
logger.level("INIT_WARN", no=31, color="<yellow>")
logger.level("INIT_ERR", no=31, color="<red>")
# Messages contain important information without which this application might not be able to be used
# As such, they have the highest priority
logger.level("MESSAGE", no=61, color="<green>")
except TypeError:
pass
logger.__class__.generation = partialmethod(logger.__class__.log, "GENERATION")
logger.__class__.prompt = partialmethod(logger.__class__.log, "PROMPT")
logger.__class__.init = partialmethod(logger.__class__.log, "INIT")
logger.__class__.init_ok = partialmethod(logger.__class__.log, "INIT_OK")
logger.__class__.init_warn = partialmethod(logger.__class__.log, "INIT_WARN")
logger.__class__.init_err = partialmethod(logger.__class__.log, "INIT_ERR")
logger.__class__.message = partialmethod(logger.__class__.log, "MESSAGE")
config = {
"handlers": [
{"sink": sys.stderr, "format": logfmt, "colorize":True, "filter": is_stderr_log},
{"sink": sys.stdout, "format": genfmt, "level": "PROMPT", "colorize":True, "filter": is_stdout_log},
{"sink": sys.stdout, "format": initfmt, "level": "INIT", "colorize":True, "filter": is_init_log},
{"sink": sys.stdout, "format": msgfmt, "level": "MESSAGE", "colorize":True, "filter": is_msg_log}
],
}
logger.configure(**config)

20
scripts/nataili/util/save_sample.py
View File

@ -1,20 +0,0 @@
import os
def save_sample(image, filename, sample_path, extension='png', jpg_quality=95, webp_quality=95, webp_lossless=True, png_compression=9):
path = os.path.join(sample_path, filename + '.' + extension)
if os.path.exists(path):
return False
if not os.path.exists(sample_path):
os.makedirs(sample_path)
if extension == 'png':
image.save(path, format='PNG', compress_level=png_compression)
elif extension == 'jpg':
image.save(path, quality=jpg_quality, optimize=True)
elif extension == 'webp':
image.save(path, quality=webp_quality, lossless=webp_lossless)
else:
return False
if os.path.exists(path):
return True
else:
return False

22
scripts/nataili/util/seed_to_int.py
View File

@ -1,22 +0,0 @@
import random
def seed_to_int(s):
if type(s) is int:
return s
if s is None or s == '':
return random.randint(0, 2**32 - 1)
if type(s) is list:
seed_list = []
for seed in s:
if seed is None or seed == '':
seed_list.append(random.randint(0, 2**32 - 1))
else:
seed_list = s
return seed_list
n = abs(int(s) if s.isdigit() else random.Random(s).randint(0, 2**32 - 1))
while n >= 2**32:
n = n >> 32
return n

34
scripts/post_processing.py Normal file
View File

@ -0,0 +1,34 @@
# This file is part of sygil-webui (https://github.com/Sygil-Dev/sandbox-webui/).
# Copyright 2022 Sygil-Dev team.
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU Affero General Public License for more details.
# You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
# base webui import and utils.
#from sd_utils import *
from sd_utils import *
# streamlit imports
#streamlit components section
#other imports
import os, time, requests
import sys
# Temp imports
# end of imports
#---------------------------------------------------------------------------------------------------------------
def layout():
st.info("Under Construction. :construction_worker:")

141
scripts/sd_utils.py
View File

@ -14,15 +14,13 @@
# You should have received a copy of the GNU Affero General Public License # You should have received a copy of the GNU Affero General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>. # along with this program. If not, see <http://www.gnu.org/licenses/>.
# base webui import and utils. # base webui import and utils.
import collections.abc
#from webui_streamlit import st #from webui_streamlit import st
import gfpgan import gfpgan
import hydralit as st import hydralit as st
# streamlit imports # streamlit imports
from streamlit import StopException, StreamlitAPIException from streamlit import StopException, StreamlitAPIException
from streamlit.runtime.scriptrunner import script_run_context #from streamlit.runtime.scriptrunner import script_run_context
#streamlit components section #streamlit components section
from streamlit_server_state import server_state, server_state_lock from streamlit_server_state import server_state, server_state_lock
@ -35,7 +33,7 @@ import streamlit_nested_layout
import warnings import warnings
import json import json
import base64 import base64, cv2
import os, sys, re, random, datetime, time, math, glob, toml import os, sys, re, random, datetime, time, math, glob, toml
import gc import gc
from PIL import Image, ImageFont, ImageDraw, ImageFilter from PIL import Image, ImageFont, ImageDraw, ImageFilter
@ -68,15 +66,25 @@ import piexif.helper
from tqdm import trange from tqdm import trange
from ldm.models.diffusion.ddim import DDIMSampler from ldm.models.diffusion.ddim import DDIMSampler
from ldm.util import ismap from ldm.util import ismap
from abc import ABC, abstractmethod #from abc import ABC, abstractmethod
from typing import Dict, Union from typing import Dict, Union
from io import BytesIO from io import BytesIO
from packaging import version from packaging import version
from uuid import uuid4 from uuid import uuid4
from pathlib import Path
from huggingface_hub import hf_hub_download
#import librosa #import librosa
from logger import logger, set_logger_verbosity, quiesce_logger from logger import logger, set_logger_verbosity, quiesce_logger
#from loguru import logger #from loguru import logger
try:
from realesrgan import RealESRGANer
from basicsr.archs.rrdbnet_arch import RRDBNet
except ImportError as e:
logger.error("You tried to import realesrgan without having it installed properly. To install Real-ESRGAN, run:\n\n"
"pip install realesrgan")
# Temp imports # Temp imports
#from basicsr.utils.registry import ARCH_REGISTRY #from basicsr.utils.registry import ARCH_REGISTRY
@ -84,14 +92,6 @@ from logger import logger, set_logger_verbosity, quiesce_logger
# end of imports # end of imports
#--------------------------------------------------------------------------------------------------------------- #---------------------------------------------------------------------------------------------------------------
# we make a log file where we store the logs
logger.add("logs/log_{time:MM-DD-YYYY!UTC}.log", rotation="8 MB", compression="zip", level='INFO') # Once the file is too old, it's rotated
logger.add(sys.stderr, diagnose=True)
logger.add(sys.stdout)
logger.enable("")
#
try: try:
# this silences the annoying "Some weights of the model checkpoint were not used when initializing..." message at start. # this silences the annoying "Some weights of the model checkpoint were not used when initializing..." message at start.
from transformers import logging from transformers import logging
@ -112,6 +112,8 @@ mimetypes.add_type('application/javascript', '.js')
opt_C = 4 opt_C = 4
opt_f = 8 opt_f = 8
# The model manager loads and unloads the SD models and has features to download them or find their location
#model_manager = ModelManager()
def load_configs(): def load_configs():
if not "defaults" in st.session_state: if not "defaults" in st.session_state:
@ -269,6 +271,33 @@ def make_grid(n_items=5, n_cols=5):
return cols return cols
def merge(file1, file2, out, weight):
alpha = (weight)/100
if not(file1.endswith(".ckpt")):
file1 += ".ckpt"
if not(file2.endswith(".ckpt")):
file2 += ".ckpt"
if not(out.endswith(".ckpt")):
out += ".ckpt"
#Load Models
model_0 = torch.load(file1)
model_1 = torch.load(file2)
theta_0 = model_0['state_dict']
theta_1 = model_1['state_dict']
for key in theta_0.keys():
if 'model' in key and key in theta_1:
theta_0[key] = (alpha) * theta_0[key] + (1-alpha) * theta_1[key]
logger.info("RUNNING...\n(STAGE 2)")
for key in theta_1.keys():
if 'model' in key and key not in theta_0:
theta_0[key] = theta_1[key]
torch.save(model_0, out)
def human_readable_size(size, decimal_places=3): def human_readable_size(size, decimal_places=3):
"""Return a human readable size from bytes.""" """Return a human readable size from bytes."""
for unit in ['B','KB','MB','GB','TB']: for unit in ['B','KB','MB','GB','TB']:
@ -282,6 +311,8 @@ def load_models(use_LDSR = False, LDSR_model='model', use_GFPGAN=False, GFPGAN_m
CustomModel_available=False, custom_model="Stable Diffusion v1.5"): CustomModel_available=False, custom_model="Stable Diffusion v1.5"):
"""Load the different models. We also reuse the models that are already in memory to speed things up instead of loading them again. """ """Load the different models. We also reuse the models that are already in memory to speed things up instead of loading them again. """
#model_manager.init()
logger.info("Loading models.") logger.info("Loading models.")
if "progress_bar_text" in st.session_state: if "progress_bar_text" in st.session_state:
@ -431,7 +462,6 @@ def load_models(use_LDSR = False, LDSR_model='model', use_GFPGAN=False, GFPGAN_m
try: try:
server_state["model"].args.use_multiprocessing_for_evaluation = False server_state["model"].args.use_multiprocessing_for_evaluation = False
except AttributeError as e: except AttributeError as e:
logger.error(e)
pass pass
if st.session_state.defaults.general.enable_attention_slicing: if st.session_state.defaults.general.enable_attention_slicing:
@ -1350,6 +1380,77 @@ def load_RealESRGAN(model_name: str):
return server_state['RealESRGAN'] return server_state['RealESRGAN']
#
class RealESRGANModel(nn.Module):
def __init__(self, model_path, tile=0, tile_pad=10, pre_pad=0, fp32=False):
super().__init__()
try:
from basicsr.archs.rrdbnet_arch import RRDBNet
from realesrgan import RealESRGANer
except ImportError as e:
logger.error(
"You tried to import realesrgan without having it installed properly. To install Real-ESRGAN, run:\n\n"
"pip install realesrgan"
)
model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32, scale=4)
self.upsampler = RealESRGANer(
scale=4, model_path=model_path, model=model, tile=tile, tile_pad=tile_pad, pre_pad=pre_pad, half=not fp32
)
def forward(self, image, outscale=4, convert_to_pil=True):
"""Upsample an image array or path.
Args:
image (Union[np.ndarray, str]): Either a np array or an image path. np array is assumed to be in RGB format,
and we convert it to BGR.
outscale (int, optional): Amount to upscale the image. Defaults to 4.
convert_to_pil (bool, optional): If True, return PIL image. Otherwise, return numpy array (BGR). Defaults to True.
Returns:
Union[np.ndarray, PIL.Image.Image]: An upsampled version of the input image.
"""
if isinstance(image, (str, Path)):
img = cv2.imread(image, cv2.IMREAD_UNCHANGED)
else:
img = image
img = (img * 255).round().astype("uint8")
img = img[:, :, ::-1]
image, _ = self.upsampler.enhance(img, outscale=outscale)
if convert_to_pil:
image = Image.fromarray(image[:, :, ::-1])
return image
@classmethod
def from_pretrained(cls, model_name_or_path="nateraw/real-esrgan"):
"""Initialize a pretrained Real-ESRGAN upsampler.
Args:
model_name_or_path (str, optional): The Hugging Face repo ID or path to local model. Defaults to 'nateraw/real-esrgan'.
Returns:
PipelineRealESRGAN: An instance of `PipelineRealESRGAN` instantiated from pretrained model.
"""
# reuploaded form official ones mentioned here:
# https://github.com/xinntao/Real-ESRGAN
if Path(model_name_or_path).exists():
file = model_name_or_path
else:
file = hf_hub_download(model_name_or_path, "RealESRGAN_x4plus.pth")
return cls(file)
def upsample_imagefolder(self, in_dir, out_dir, suffix="out", outfile_ext=".png"):
in_dir, out_dir = Path(in_dir), Path(out_dir)
if not in_dir.exists():
raise FileNotFoundError(f"Provided input directory {in_dir} does not exist")
out_dir.mkdir(exist_ok=True, parents=True)
image_paths = [x for x in in_dir.glob("*") if x.suffix.lower() in [".png", ".jpg", ".jpeg"]]
for image in image_paths:
im = self(str(image))
out_filepath = out_dir / (image.stem + suffix + outfile_ext)
im.save(out_filepath)
# #
@retry(tries=5) @retry(tries=5)
def load_LDSR(model_name="model", config="project", checking=False): def load_LDSR(model_name="model", config="project", checking=False):
@ -1744,6 +1845,9 @@ def seed_to_int(s):
if s is None or s == '': if s is None or s == '':
return random.randint(0, 2**32 - 1) return random.randint(0, 2**32 - 1)
if ',' in s:
s = s.split(',')
if type(s) is list: if type(s) is list:
seed_list = [] seed_list = []
for seed in s: for seed in s:
@ -1906,6 +2010,8 @@ def GFPGAN_available():
st.session_state["GFPGAN_available"] = True st.session_state["GFPGAN_available"] = True
else: else:
st.session_state["GFPGAN_available"] = False st.session_state["GFPGAN_available"] = False
st.session_state["use_GFPGAN"] = False
st.session_state["GFPGAN_model"] = "GFPGANv1.4"
# #
def RealESRGAN_available(): def RealESRGAN_available():
@ -1924,6 +2030,8 @@ def RealESRGAN_available():
st.session_state["RealESRGAN_available"] = True st.session_state["RealESRGAN_available"] = True
else: else:
st.session_state["RealESRGAN_available"] = False st.session_state["RealESRGAN_available"] = False
st.session_state["use_RealESRGAN"] = False
st.session_state["RealESRGAN_model"] = "RealESRGAN_x4plus"
# #
def LDSR_available(): def LDSR_available():
#with server_state_lock["RealESRGAN_models"]: #with server_state_lock["RealESRGAN_models"]:
@ -1944,6 +2052,8 @@ def LDSR_available():
st.session_state["LDSR_available"] = True st.session_state["LDSR_available"] = True
else: else:
st.session_state["LDSR_available"] = False st.session_state["LDSR_available"] = False
st.session_state["use_LDSR"] = False
st.session_state["LDSR_model"] = "model"
@ -1955,6 +2065,7 @@ def save_sample(image, sample_path_i, filename, jpg_sample, prompts, seeds, widt
filename_i = os.path.join(sample_path_i, filename) filename_i = os.path.join(sample_path_i, filename)
if "defaults" in st.session_state:
if st.session_state['defaults'].general.save_metadata or write_info_files: if st.session_state['defaults'].general.save_metadata or write_info_files:
# toggles differ for txt2img vs. img2img: # toggles differ for txt2img vs. img2img:
offset = 0 if init_img is None else 2 offset = 0 if init_img is None else 2
@ -2563,7 +2674,7 @@ def process_images(
#output_images.append(image) #output_images.append(image)
#if simple_templating: #if simple_templating:
#grid_captions.append( captions[i] ) #grid_captions.append( captions[i] )
if "defaults" in st.session_state:
if st.session_state['defaults'].general.optimized: if st.session_state['defaults'].general.optimized:
mem = torch.cuda.memory_allocated()/1e6 mem = torch.cuda.memory_allocated()/1e6
server_state["modelFS"].to("cpu") server_state["modelFS"].to("cpu")

222
scripts/stable_diffusion_pipeline.py
View File

@ -10,7 +10,6 @@ import time
import json import json
import torch import torch
from diffusers import ModelMixin
from diffusers.configuration_utils import FrozenDict from diffusers.configuration_utils import FrozenDict
from diffusers.models import AutoencoderKL, UNet2DConditionModel from diffusers.models import AutoencoderKL, UNet2DConditionModel
from diffusers.pipeline_utils import DiffusionPipeline from diffusers.pipeline_utils import DiffusionPipeline
@ -22,59 +21,39 @@ from diffusers.pipelines.stable_diffusion import StableDiffusionPipelineOutput
from transformers import CLIPFeatureExtractor, CLIPTextModel, CLIPTokenizer from transformers import CLIPFeatureExtractor, CLIPTextModel, CLIPTokenizer
from torch import nn from torch import nn
from .upsampling import RealESRGANModel from sd_utils import RealESRGANModel
logger = logging.get_logger(__name__) # pylint: disable=invalid-name logger = logging.get_logger(__name__) # pylint: disable=invalid-name
def get_spec_norm(wav, sr, n_mels=512, hop_length=704): def get_timesteps_arr(audio_filepath, offset, duration, fps=30, margin=1.0, smooth=0.0):
"""Obtain maximum value for each time-frame in Mel Spectrogram, y, sr = librosa.load(audio_filepath, offset=offset, duration=duration)
and normalize between 0 and 1
Borrowed from lucid sonic dreams repo. In there, they programatically determine hop length # librosa.stft hardcoded defaults...
but I really didn't understand what was going on so I removed it and hard coded the output. # n_fft defaults to 2048
""" # hop length is win_length // 4
# win_length defaults to n_fft
D = librosa.stft(y, n_fft=2048, hop_length=2048 // 4, win_length=2048)
# Generate Mel Spectrogram # Extract percussive elements
spec_raw = librosa.feature.melspectrogram(y=wav, sr=sr, n_mels=n_mels, hop_length=hop_length) D_harmonic, D_percussive = librosa.decompose.hpss(D, margin=margin)
y_percussive = librosa.istft(D_percussive, length=len(y))
# Obtain maximum value per time-frame # Get normalized melspectrogram
spec_raw = librosa.feature.melspectrogram(y=y_percussive, sr=sr)
spec_max = np.amax(spec_raw, axis=0) spec_max = np.amax(spec_raw, axis=0)
# Normalize all values between 0 and 1
spec_norm = (spec_max - np.min(spec_max)) / np.ptp(spec_max) spec_norm = (spec_max - np.min(spec_max)) / np.ptp(spec_max)
return spec_norm # Resize cumsum of spec norm to our desired number of interpolation frames
x_norm = np.linspace(0, spec_norm.shape[-1], spec_norm.shape[-1])
y_norm = np.cumsum(spec_norm)
y_norm /= y_norm[-1]
x_resize = np.linspace(0, y_norm.shape[-1], int(duration*fps))
T = np.interp(x_resize, x_norm, y_norm)
def get_timesteps_arr(audio_filepath, offset, duration, fps=30, margin=(1.0, 5.0)): # Apply smoothing
"""Get the array that will be used to determine how much to interpolate between images. return T * (1 - smooth) + np.linspace(0.0, 1.0, T.shape[0]) * smooth
Normally, this is just a linspace between 0 and 1 for the number of frames to generate. In this case,
we want to use the amplitude of the audio to determine how much to interpolate between images.
So, here we:
1. Load the audio file
2. Split the audio into harmonic and percussive components
3. Get the normalized amplitude of the percussive component, resized to the number of frames
4. Get the cumulative sum of the amplitude array
5. Normalize the cumulative sum between 0 and 1
6. Return the array
I honestly have no clue what I'm doing here. Suggestions welcome.
"""
y, sr = librosa.load(audio_filepath, offset=offset, duration=duration)
wav_harmonic, wav_percussive = librosa.effects.hpss(y, margin=margin)
# Apparently n_mels is supposed to be input shape but I don't think it matters here?
frame_duration = int(sr / fps)
wav_norm = get_spec_norm(wav_percussive, sr, n_mels=512, hop_length=frame_duration)
amplitude_arr = np.resize(wav_norm, int(duration * fps))
T = np.cumsum(amplitude_arr)
T /= T[-1]
T[0] = 0.0
return T
def slerp(t, v0, v1, DOT_THRESHOLD=0.9995): def slerp(t, v0, v1, DOT_THRESHOLD=0.9995):
@ -130,7 +109,6 @@ def make_video_pyav(
frame = pil_to_tensor(Image.open(img)).unsqueeze(0) frame = pil_to_tensor(Image.open(img)).unsqueeze(0)
frames = frame if frames is None else torch.cat([frames, frame]) frames = frame if frames is None else torch.cat([frames, frame])
else: else:
frames = frames_or_frame_dir frames = frames_or_frame_dir
# TCHW -> THWC # TCHW -> THWC
@ -208,6 +186,16 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
new_config["steps_offset"] = 1 new_config["steps_offset"] = 1
scheduler._internal_dict = FrozenDict(new_config) scheduler._internal_dict = FrozenDict(new_config)
if safety_checker is None:
logger.warn(
f"You have disabled the safety checker for {self.__class__} by passing `safety_checker=None`. Ensure"
" that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered"
" results in services or applications open to the public. Both the diffusers team and Hugging Face"
" strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling"
" it only for use-cases that involve analyzing network behavior or auditing its results. For more"
" information, please have a look at https://github.com/huggingface/diffusers/pull/254 ."
)
self.register_modules( self.register_modules(
vae=vae, vae=vae,
text_encoder=text_encoder, text_encoder=text_encoder,
@ -251,6 +239,8 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
width: int = 512, width: int = 512,
num_inference_steps: int = 50, num_inference_steps: int = 50,
guidance_scale: float = 7.5, guidance_scale: float = 7.5,
negative_prompt: Optional[Union[str, List[str]]] = None,
num_images_per_prompt: Optional[int] = 1,
eta: float = 0.0, eta: float = 0.0,
generator: Optional[torch.Generator] = None, generator: Optional[torch.Generator] = None,
latents: Optional[torch.FloatTensor] = None, latents: Optional[torch.FloatTensor] = None,
@ -259,12 +249,13 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
callback: Optional[Callable[[int, int, torch.FloatTensor], None]] = None, callback: Optional[Callable[[int, int, torch.FloatTensor], None]] = None,
callback_steps: Optional[int] = 1, callback_steps: Optional[int] = 1,
text_embeddings: Optional[torch.FloatTensor] = None, text_embeddings: Optional[torch.FloatTensor] = None,
**kwargs,
): ):
r""" r"""
Function invoked when calling the pipeline for generation. Function invoked when calling the pipeline for generation.
Args: Args:
prompt (`str` or `List[str]`): prompt (`str` or `List[str]`, *optional*, defaults to `None`):
The prompt or prompts to guide the image generation. The prompt or prompts to guide the image generation. If not provided, `text_embeddings` is required.
height (`int`, *optional*, defaults to 512): height (`int`, *optional*, defaults to 512):
The height in pixels of the generated image. The height in pixels of the generated image.
width (`int`, *optional*, defaults to 512): width (`int`, *optional*, defaults to 512):
@ -278,6 +269,11 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale > Paper](https://arxiv.org/pdf/2205.11487.pdf). Guidance scale is enabled by setting `guidance_scale >
1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`, 1`. Higher guidance scale encourages to generate images that are closely linked to the text `prompt`,
usually at the expense of lower image quality. usually at the expense of lower image quality.
negative_prompt (`str` or `List[str]`, *optional*):
The prompt or prompts not to guide the image generation. Ignored when not using guidance (i.e., ignored
if `guidance_scale` is less than `1`).
num_images_per_prompt (`int`, *optional*, defaults to 1):
The number of images to generate per prompt.
eta (`float`, *optional*, defaults to 0.0): eta (`float`, *optional*, defaults to 0.0):
Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to Corresponds to parameter eta (η) in the DDIM paper: https://arxiv.org/abs/2010.02502. Only applies to
[`schedulers.DDIMScheduler`], will be ignored for others. [`schedulers.DDIMScheduler`], will be ignored for others.
@ -300,8 +296,10 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
callback_steps (`int`, *optional*, defaults to 1): callback_steps (`int`, *optional*, defaults to 1):
The frequency at which the `callback` function will be called. If not specified, the callback will be The frequency at which the `callback` function will be called. If not specified, the callback will be
called at every step. called at every step.
text_embeddings(`torch.FloatTensor`, *optional*): text_embeddings (`torch.FloatTensor`, *optional*, defaults to `None`):
Pre-generated text embeddings. Pre-generated text embeddings to be used as inputs for image generation. Can be used in place of
`prompt` to avoid re-computing the embeddings. If not provided, the embeddings will be generated from
the supplied `prompt`.
Returns: Returns:
[`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] or `tuple`: [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] or `tuple`:
[`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] if `return_dict` is True, otherwise a `tuple. [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] if `return_dict` is True, otherwise a `tuple.
@ -340,7 +338,7 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
if text_input_ids.shape[-1] > self.tokenizer.model_max_length: if text_input_ids.shape[-1] > self.tokenizer.model_max_length:
removed_text = self.tokenizer.batch_decode(text_input_ids[:, self.tokenizer.model_max_length :]) removed_text = self.tokenizer.batch_decode(text_input_ids[:, self.tokenizer.model_max_length :])
logger.warning( print(
"The following part of your input was truncated because CLIP can only handle sequences up to" "The following part of your input was truncated because CLIP can only handle sequences up to"
f" {self.tokenizer.model_max_length} tokens: {removed_text}" f" {self.tokenizer.model_max_length} tokens: {removed_text}"
) )
@ -349,21 +347,51 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
else: else:
batch_size = text_embeddings.shape[0] batch_size = text_embeddings.shape[0]
# duplicate text embeddings for each generation per prompt, using mps friendly method
bs_embed, seq_len, _ = text_embeddings.shape
text_embeddings = text_embeddings.repeat(1, num_images_per_prompt, 1)
text_embeddings = text_embeddings.view(bs_embed * num_images_per_prompt, seq_len, -1)
# here `guidance_scale` is defined analog to the guidance weight `w` of equation (2) # here `guidance_scale` is defined analog to the guidance weight `w` of equation (2)
# of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1` # of the Imagen paper: https://arxiv.org/pdf/2205.11487.pdf . `guidance_scale = 1`
# corresponds to doing no classifier free guidance. # corresponds to doing no classifier free guidance.
do_classifier_free_guidance = guidance_scale > 1.0 do_classifier_free_guidance = guidance_scale > 1.0
# get unconditional embeddings for classifier free guidance # get unconditional embeddings for classifier free guidance
if do_classifier_free_guidance: if do_classifier_free_guidance:
# HACK - Not setting text_input_ids here when walking, so hard coding to max length of tokenizer uncond_tokens: List[str]
# TODO - Determine if this is OK to do if negative_prompt is None:
# max_length = text_input_ids.shape[-1] uncond_tokens = [""]
elif type(prompt) is not type(negative_prompt):
raise TypeError(
f"`negative_prompt` should be the same type to `prompt`, but got {type(negative_prompt)} !="
f" {type(prompt)}."
)
elif isinstance(negative_prompt, str):
uncond_tokens = [negative_prompt]
elif batch_size != len(negative_prompt):
raise ValueError(
f"`negative_prompt`: {negative_prompt} has batch size {len(negative_prompt)}, but `prompt`:"
f" {prompt} has batch size {batch_size}. Please make sure that passed `negative_prompt` matches"
" the batch size of `prompt`."
)
else:
uncond_tokens = negative_prompt
max_length = self.tokenizer.model_max_length max_length = self.tokenizer.model_max_length
uncond_input = self.tokenizer( uncond_input = self.tokenizer(
[""] * batch_size, padding="max_length", max_length=max_length, return_tensors="pt" uncond_tokens,
padding="max_length",
max_length=max_length,
truncation=True,
return_tensors="pt",
) )
uncond_embeddings = self.text_encoder(uncond_input.input_ids.to(self.device))[0] uncond_embeddings = self.text_encoder(uncond_input.input_ids.to(self.device))[0]
# duplicate unconditional embeddings for each generation per prompt, using mps friendly method
seq_len = uncond_embeddings.shape[1]
uncond_embeddings = uncond_embeddings.repeat(batch_size, num_images_per_prompt, 1)
uncond_embeddings = uncond_embeddings.view(batch_size * num_images_per_prompt, seq_len, -1)
# For classifier free guidance, we need to do two forward passes. # For classifier free guidance, we need to do two forward passes.
# Here we concatenate the unconditional and text embeddings into a single batch # Here we concatenate the unconditional and text embeddings into a single batch
# to avoid doing two forward passes # to avoid doing two forward passes
@ -374,19 +402,20 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
# Unlike in other pipelines, latents need to be generated in the target device # Unlike in other pipelines, latents need to be generated in the target device
# for 1-to-1 results reproducibility with the CompVis implementation. # for 1-to-1 results reproducibility with the CompVis implementation.
# However this currently doesn't work in `mps`. # However this currently doesn't work in `mps`.
latents_device = "cpu" if self.device.type == "mps" else self.device latents_shape = (batch_size * num_images_per_prompt, self.unet.in_channels, height // 8, width // 8)
latents_shape = (batch_size, self.unet.in_channels, height // 8, width // 8) latents_dtype = text_embeddings.dtype
if latents is None: if latents is None:
latents = torch.randn( if self.device.type == "mps":
latents_shape, # randn does not exist on mps
generator=generator, latents = torch.randn(latents_shape, generator=generator, device="cpu", dtype=latents_dtype).to(
device=latents_device, self.device
dtype=text_embeddings.dtype,
) )
else:
latents = torch.randn(latents_shape, generator=generator, device=self.device, dtype=latents_dtype)
else: else:
if latents.shape != latents_shape: if latents.shape != latents_shape:
raise ValueError(f"Unexpected latents shape, got {latents.shape}, expected {latents_shape}") raise ValueError(f"Unexpected latents shape, got {latents.shape}, expected {latents_shape}")
latents = latents.to(latents_device) latents = latents.to(self.device)
# set timesteps # set timesteps
self.scheduler.set_timesteps(num_inference_steps) self.scheduler.set_timesteps(num_inference_steps)
@ -431,12 +460,19 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
image = self.vae.decode(latents).sample image = self.vae.decode(latents).sample
image = (image / 2 + 0.5).clamp(0, 1) image = (image / 2 + 0.5).clamp(0, 1)
image = image.cpu().permute(0, 2, 3, 1).numpy()
safety_checker_input = self.feature_extractor(self.numpy_to_pil(image), return_tensors="pt").to(self.device) # we always cast to float32 as this does not cause significant overhead and is compatible with bfloa16
image = image.cpu().permute(0, 2, 3, 1).float().numpy()
if self.safety_checker is not None:
safety_checker_input = self.feature_extractor(self.numpy_to_pil(image), return_tensors="pt").to(
self.device
)
image, has_nsfw_concept = self.safety_checker( image, has_nsfw_concept = self.safety_checker(
images=image, clip_input=safety_checker_input.pixel_values.to(text_embeddings.dtype) images=image, clip_input=safety_checker_input.pixel_values.to(text_embeddings.dtype)
) )
else:
has_nsfw_concept = None
if output_type == "pil": if output_type == "pil":
image = self.numpy_to_pil(image) image = self.numpy_to_pil(image)
@ -449,16 +485,9 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
def generate_inputs(self, prompt_a, prompt_b, seed_a, seed_b, noise_shape, T, batch_size): def generate_inputs(self, prompt_a, prompt_b, seed_a, seed_b, noise_shape, T, batch_size):
embeds_a = self.embed_text(prompt_a) embeds_a = self.embed_text(prompt_a)
embeds_b = self.embed_text(prompt_b) embeds_b = self.embed_text(prompt_b)
latents_a = torch.randn(
noise_shape, latents_a = self.init_noise(seed_a, noise_shape)
device=self.device, latents_b = self.init_noise(seed_b, noise_shape)
generator=torch.Generator(device=self.device).manual_seed(seed_a),
)
latents_b = torch.randn(
noise_shape,
device=self.device,
generator=torch.Generator(device=self.device).manual_seed(seed_b),
)
batch_idx = 0 batch_idx = 0
embeds_batch, noise_batch = None, None embeds_batch, noise_batch = None, None
@ -477,7 +506,7 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
torch.cuda.empty_cache() torch.cuda.empty_cache()
embeds_batch, noise_batch = None, None embeds_batch, noise_batch = None, None
def generate_interpolation_clip( def make_clip_frames(
self, self,
prompt_a: str, prompt_a: str,
prompt_b: str, prompt_b: str,
@ -530,7 +559,7 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
eta=eta, eta=eta,
num_inference_steps=num_inference_steps, num_inference_steps=num_inference_steps,
output_type="pil" if not upsample else "numpy", output_type="pil" if not upsample else "numpy",
)["sample"] )["images"]
for image in outputs: for image in outputs:
frame_filepath = save_path / (f"frame%06d{image_file_ext}" % frame_index) frame_filepath = save_path / (f"frame%06d{image_file_ext}" % frame_index)
@ -557,6 +586,8 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
resume: Optional[bool] = False, resume: Optional[bool] = False,
audio_filepath: str = None, audio_filepath: str = None,
audio_start_sec: Optional[Union[int, float]] = None, audio_start_sec: Optional[Union[int, float]] = None,
margin: Optional[float] = 1.0,
smooth: Optional[float] = 0.0,
): ):
"""Generate a video from a sequence of prompts and seeds. Optionally, add audio to the """Generate a video from a sequence of prompts and seeds. Optionally, add audio to the
video to interpolate to the intensity of the audio. video to interpolate to the intensity of the audio.
@ -603,13 +634,17 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
Optional path to an audio file to influence the interpolation rate. Optional path to an audio file to influence the interpolation rate.
audio_start_sec (Optional[Union[int, float]], *optional*, defaults to 0): audio_start_sec (Optional[Union[int, float]], *optional*, defaults to 0):
Global start time of the provided audio_filepath. Global start time of the provided audio_filepath.
margin (Optional[float], *optional*, defaults to 1.0):
Margin from librosa hpss to use for audio interpolation.
smooth (Optional[float], *optional*, defaults to 0.0):
Smoothness of the audio interpolation. 1.0 means linear interpolation.
This function will create sub directories for each prompt and seed pair. This function will create sub directories for each prompt and seed pair.
For example, if you provide the following prompts and seeds: For example, if you provide the following prompts and seeds:
``` ```
prompts = ['a', 'b', 'c'] prompts = ['a dog', 'a cat', 'a bird']
seeds = [1, 2, 3] seeds = [1, 2, 3]
num_interpolation_steps = 5 num_interpolation_steps = 5
output_dir = 'output_dir' output_dir = 'output_dir'
@ -722,7 +757,7 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
audio_offset = audio_start_sec + sum(num_interpolation_steps[:i]) / fps audio_offset = audio_start_sec + sum(num_interpolation_steps[:i]) / fps
audio_duration = num_step / fps audio_duration = num_step / fps
self.generate_interpolation_clip( self.make_clip_frames(
prompt_a, prompt_a,
prompt_b, prompt_b,
seed_a, seed_a,
@ -742,7 +777,8 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
offset=audio_offset, offset=audio_offset,
duration=audio_duration, duration=audio_duration,
fps=fps, fps=fps,
margin=(1.0, 5.0), margin=margin,
smooth=smooth,
) )
if audio_filepath if audio_filepath
else None, else None,
@ -783,6 +819,23 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
embed = self.text_encoder(text_input.input_ids.to(self.device))[0] embed = self.text_encoder(text_input.input_ids.to(self.device))[0]
return embed return embed
def init_noise(self, seed, noise_shape):
"""Helper to initialize noise"""
# randn does not exist on mps, so we create noise on CPU here and move it to the device after initialization
if self.device.type == "mps":
noise = torch.randn(
noise_shape,
device='cpu',
generator=torch.Generator(device='cpu').manual_seed(seed),
).to(self.device)
else:
noise = torch.randn(
noise_shape,
device=self.device,
generator=torch.Generator(device=self.device).manual_seed(seed),
)
return noise
@classmethod @classmethod
def from_pretrained(cls, *args, tiled=False, **kwargs): def from_pretrained(cls, *args, tiled=False, **kwargs):
"""Same as diffusers `from_pretrained` but with tiled option, which makes images tilable""" """Same as diffusers `from_pretrained` but with tiled option, which makes images tilable"""
@ -799,15 +852,6 @@ class StableDiffusionWalkPipeline(DiffusionPipeline):
patch_conv(padding_mode="circular") patch_conv(padding_mode="circular")
return super().from_pretrained(*args, **kwargs) pipeline = super().from_pretrained(*args, **kwargs)
pipeline.tiled = tiled
return pipeline
class NoCheck(ModelMixin):
"""Can be used in place of safety checker. Use responsibly and at your own risk."""
def __init__(self):
super().__init__()
self.register_parameter(name="asdf", param=torch.nn.Parameter(torch.randn(3)))
def forward(self, images=None, **kwargs):
return images, [False]

51
scripts/txt2img.py
View File

@ -106,7 +106,7 @@ def stable_horde(outpath, prompt, seed, sampler_name, save_grid, batch_size,
log.append("Generating image with Stable Horde.") log.append("Generating image with Stable Horde.")
st.session_state["progress_bar_text"].code('\n'.join(str(log)), language='') st.session_state["progress_bar_text"].code('\n'.join(log), language='')
# start time after garbage collection (or before?) # start time after garbage collection (or before?)
start_time = time.time() start_time = time.time()
@ -157,7 +157,7 @@ def stable_horde(outpath, prompt, seed, sampler_name, save_grid, batch_size,
logger.debug(submit_results) logger.debug(submit_results)
log.append(submit_results) log.append(submit_results)
st.session_state["progress_bar_text"].code('\n'.join(str(log)), language='') st.session_state["progress_bar_text"].code(''.join(str(log)), language='')
req_id = submit_results['id'] req_id = submit_results['id']
is_done = False is_done = False
@ -282,7 +282,7 @@ def txt2img(prompt: str, ddim_steps: int, sampler_name: str, n_iter: int, batch_
RealESRGAN_model: str = "RealESRGAN_x4plus_anime_6B", use_LDSR: bool = True, LDSR_model: str = "model", RealESRGAN_model: str = "RealESRGAN_x4plus_anime_6B", use_LDSR: bool = True, LDSR_model: str = "model",
fp = None, variant_amount: float = 0.0, fp = None, variant_amount: float = 0.0,
variant_seed: int = None, ddim_eta:float = 0.0, write_info_files:bool = True, variant_seed: int = None, ddim_eta:float = 0.0, write_info_files:bool = True,
use_stable_horde: bool = False, stable_horde_key:str = ''): use_stable_horde: bool = False, stable_horde_key:str = "0000000000"):
outpath = st.session_state['defaults'].general.outdir_txt2img outpath = st.session_state['defaults'].general.outdir_txt2img
@ -410,7 +410,7 @@ def layout():
sygil_suggestions.suggestion_area(placeholder) sygil_suggestions.suggestion_area(placeholder)
# creating the page layout using columns # creating the page layout using columns
col1, col2, col3 = st.columns([1,2,1], gap="large") col1, col2, col3 = st.columns([2,5,2], gap="large")
with col1: with col1:
width = st.slider("Width:", min_value=st.session_state['defaults'].txt2img.width.min_value, max_value=st.session_state['defaults'].txt2img.width.max_value, width = st.slider("Width:", min_value=st.session_state['defaults'].txt2img.width.min_value, max_value=st.session_state['defaults'].txt2img.width.max_value,
@ -475,7 +475,7 @@ def layout():
with gallery_tab: with gallery_tab:
st.session_state["gallery"] = st.empty() st.session_state["gallery"] = st.empty()
st.session_state["gallery"].info("Nothing but crickets here, try generating something first.") #st.session_state["gallery"].info("Nothing but crickets here, try generating something first.")
with col3: with col3:
# If we have custom models available on the "models/custom" # If we have custom models available on the "models/custom"
@ -502,7 +502,7 @@ def layout():
with st.expander("Advanced"): with st.expander("Advanced"):
with st.expander("Stable Horde"): with st.expander("Stable Horde"):
use_stable_horde = st.checkbox("Use Stable Horde", value=False, help="Use the Stable Horde to generate images. More info can be found at https://stablehorde.net/") use_stable_horde = st.checkbox("Use Stable Horde", value=False, help="Use the Stable Horde to generate images. More info can be found at https://stablehorde.net/")
stable_horde_key = st.text_input("Stable Horde Api Key", value='', type="password", stable_horde_key = st.text_input("Stable Horde Api Key", value=st.session_state['defaults'].general.stable_horde_api, type="password",
help="Optional Api Key used for the Stable Horde Bridge, if no api key is added the horde will be used anonymously.") help="Optional Api Key used for the Stable Horde Bridge, if no api key is added the horde will be used anonymously.")
with st.expander("Output Settings"): with st.expander("Output Settings"):
@ -570,7 +570,9 @@ def layout():
#print (st.session_state["RealESRGAN_available"]) #print (st.session_state["RealESRGAN_available"])
st.session_state["upscaling_method"] = st.selectbox("Upscaling Method", upscaling_method_list, st.session_state["upscaling_method"] = st.selectbox("Upscaling Method", upscaling_method_list,
index=upscaling_method_list.index(str(st.session_state['defaults'].general.upscaling_method))) index=upscaling_method_list.index(st.session_state['defaults'].general.upscaling_method)
if st.session_state['defaults'].general.upscaling_method in upscaling_method_list
else 0)
if st.session_state["RealESRGAN_available"]: if st.session_state["RealESRGAN_available"]:
with st.expander("RealESRGAN"): with st.expander("RealESRGAN"):
@ -654,42 +656,9 @@ def layout():
message.success('Render Complete: ' + info + '; Stats: ' + stats, icon="✅") message.success('Render Complete: ' + info + '; Stats: ' + stats, icon="✅")
#history_tab,col1,col2,col3,PlaceHolder,col1_cont,col2_cont,col3_cont = st.session_state['historyTab']
#if 'latestImages' in st.session_state:
#for i in output_images:
##push the new image to the list of latest images and remove the oldest one
##remove the last index from the list\
#st.session_state['latestImages'].pop()
##add the new image to the start of the list
#st.session_state['latestImages'].insert(0, i)
#PlaceHolder.empty()
#with PlaceHolder.container():
#col1, col2, col3 = st.columns(3)
#col1_cont = st.container()
#col2_cont = st.container()
#col3_cont = st.container()
#images = st.session_state['latestImages']
#with col1_cont:
#with col1:
#[st.image(images[index]) for index in [0, 3, 6] if index < len(images)]
#with col2_cont:
#with col2:
#[st.image(images[index]) for index in [1, 4, 7] if index < len(images)]
#with col3_cont:
#with col3:
#[st.image(images[index]) for index in [2, 5, 8] if index < len(images)]
#historyGallery = st.empty()
## check if output_images length is the same as seeds length
#with gallery_tab:
#st.markdown(createHTMLGallery(output_images,seeds), unsafe_allow_html=True)
#st.session_state['historyTab'] = [history_tab,col1,col2,col3,PlaceHolder,col1_cont,col2_cont,col3_cont]
with gallery_tab: with gallery_tab:
logger.info(seeds) logger.info(seeds)
st.session_state["gallery"].text = ""
sdGallery(output_images) sdGallery(output_images)

966
scripts/txt2vid.py
View File

File diff suppressed because it is too large Load Diff

31
scripts/webui_streamlit.py
View File

@ -123,10 +123,16 @@ def layout():
# specify the primary menu definition # specify the primary menu definition
menu_data = [ menu_data = [
{'id': 'Stable Diffusion', 'label': 'Stable Diffusion', 'icon': 'bi bi-grid-1x2-fill'}, {'id': 'Stable Diffusion', 'label': 'Stable Diffusion', 'icon': 'bi bi-grid-1x2-fill'},
{'id': 'Train','label':"Train", 'icon': "bi bi-lightbulb-fill", 'submenu':[
{'id': 'Textual Inversion', 'label': 'Textual Inversion', 'icon': 'bi bi-lightbulb-fill'}, {'id': 'Textual Inversion', 'label': 'Textual Inversion', 'icon': 'bi bi-lightbulb-fill'},
{'id': 'Fine Tunning', 'label': 'Fine Tunning', 'icon': 'bi bi-lightbulb-fill'},
]},
{'id': 'Model Manager', 'label': 'Model Manager', 'icon': 'bi bi-cloud-arrow-down-fill'}, {'id': 'Model Manager', 'label': 'Model Manager', 'icon': 'bi bi-cloud-arrow-down-fill'},
#{'id': 'Tools','label':"Tools", 'icon': "bi bi-tools", 'submenu':[ {'id': 'Tools','label':"Tools", 'icon': "bi bi-tools", 'submenu':[
{'id': 'API Server', 'label': 'API Server', 'icon': 'bi bi-server'}, {'id': 'API Server', 'label': 'API Server', 'icon': 'bi bi-server'},
{'id': 'Barfi/BaklavaJS', 'label': 'Barfi/BaklavaJS', 'icon': 'bi bi-diagram-3-fill'},
#{'id': 'API Server', 'label': 'API Server', 'icon': 'bi bi-server'},
]},
{'id': 'Settings', 'label': 'Settings', 'icon': 'bi bi-gear-fill'}, {'id': 'Settings', 'label': 'Settings', 'icon': 'bi bi-gear-fill'},
#{'icon': "fa-solid fa-radar",'label':"Dropdown1", 'submenu':[ #{'icon': "fa-solid fa-radar",'label':"Dropdown1", 'submenu':[
# {'id':' subid11','icon': "fa fa-paperclip", 'label':"Sub-item 1"},{'id':'subid12','icon': "💀", 'label':"Sub-item 2"},{'id':'subid13','icon': "fa fa-database", 'label':"Sub-item 3"}]}, # {'id':' subid11','icon': "fa fa-paperclip", 'label':"Sub-item 1"},{'id':'subid12','icon': "💀", 'label':"Sub-item 2"},{'id':'subid13','icon': "fa fa-database", 'label':"Sub-item 3"}]},
@ -172,6 +178,10 @@ def layout():
#horizontal_orientation=False, #horizontal_orientation=False,
#override_theme={'txc_inactive': 'white','menu_background':'#111', 'stVerticalBlock': '#111','txc_active':'yellow','option_active':'blue'}) #override_theme={'txc_inactive': 'white','menu_background':'#111', 'stVerticalBlock': '#111','txc_active':'yellow','option_active':'blue'})
#
#if menu_id == "Home":
#st.info("Under Construction. :construction_worker:")
if menu_id == "Stable Diffusion": if menu_id == "Stable Diffusion":
# set the page url and title # set the page url and title
#st.experimental_set_query_params(page='stable-diffusion') #st.experimental_set_query_params(page='stable-diffusion')
@ -180,9 +190,10 @@ def layout():
except NameError: except NameError:
st.experimental_rerun() st.experimental_rerun()
txt2img_tab, img2img_tab, txt2vid_tab, img2txt_tab, concept_library_tab = st.tabs(["Text-to-Image", "Image-to-Image", txt2img_tab, img2img_tab, txt2vid_tab, img2txt_tab, post_processing_tab, concept_library_tab = st.tabs(["Text-to-Image", "Image-to-Image",
#"Inpainting",
"Text-to-Video", "Image-To-Text", "Text-to-Video", "Image-To-Text",
"Concept Library"]) "Post-Processing","Concept Library"])
#with home_tab: #with home_tab:
#from home import layout #from home import layout
#layout() #layout()
@ -207,6 +218,10 @@ def layout():
from img2txt import layout from img2txt import layout
layout() layout()
with post_processing_tab:
from post_processing import layout
layout()
with concept_library_tab: with concept_library_tab:
from sd_concept_library import layout from sd_concept_library import layout
layout() layout()
@ -222,11 +237,21 @@ def layout():
from textual_inversion import layout from textual_inversion import layout
layout() layout()
elif menu_id == 'Fine Tunning':
#from textual_inversion import layout
#layout()
st.info("Under Construction. :construction_worker:")
elif menu_id == 'API Server': elif menu_id == 'API Server':
set_page_title("API Server - Stable Diffusion Playground") set_page_title("API Server - Stable Diffusion Playground")
from APIServer import layout from APIServer import layout
layout() layout()
elif menu_id == 'Barfi/BaklavaJS':
set_page_title("Barfi/BaklavaJS - Stable Diffusion Playground")
from barfi_baklavajs import layout
layout()
elif menu_id == 'Settings': elif menu_id == 'Settings':
set_page_title("Settings - Stable Diffusion Playground") set_page_title("Settings - Stable Diffusion Playground")

4
webui.cmd
View File

@ -103,10 +103,10 @@ call "%v_conda_path%\Scripts\activate.bat" "%v_conda_env_name%"
:PROMPT :PROMPT
set SETUPTOOLS_USE_DISTUTILS=stdlib set SETUPTOOLS_USE_DISTUTILS=stdlib
IF EXIST "models\ldm\stable-diffusion-v1\Stable Diffusion v1.5.ckpt" ( IF EXIST "models\ldm\stable-diffusion-v1\Stable Diffusion v1.5.ckpt" (
python -m streamlit run scripts\webui_streamlit.py --theme.base dark --server.address localhost python -m streamlit run scripts\webui_streamlit.py --theme.base dark
) ELSE ( ) ELSE (
echo Your model file does not exist! Once the WebUI launches please visit the Model Manager page and download the models by using the Download button for each model. echo Your model file does not exist! Once the WebUI launches please visit the Model Manager page and download the models by using the Download button for each model.
python -m streamlit run scripts\webui_streamlit.py --theme.base dark --server.address localhost python -m streamlit run scripts\webui_streamlit.py --theme.base dark
) )
::cmd /k ::cmd /k

4
webui.sh
View File

@ -156,12 +156,12 @@ launch_webui () {
done done
printf "\n\n########## LAUNCH USING STREAMLIT OR GRADIO? ##########\n\n" printf "\n\n########## LAUNCH USING STREAMLIT OR GRADIO? ##########\n\n"
printf "Do you wish to run the WebUI using the Gradio or StreamLit Interface?\n\n" printf "Do you wish to run the WebUI using the Gradio or StreamLit Interface?\n\n"
printf "Streamlit: \nHas A More Modern UI \nMore Features Planned \nWill Be The Main UI Going Forward \nCurrently In Active Development \nMissing Some Gradio Features\n\n" printf "Streamlit: \nHas A More Modern UI \nMore Features Planned \nWill Be The Main UI Going Forward \nCurrently In Active Development \n\n"
printf "Gradio: \nCurrently Feature Complete \nUses An Older Interface Style \nWill Not Receive Major Updates\n\n" printf "Gradio: \nCurrently Feature Complete \nUses An Older Interface Style \nWill Not Receive Major Updates\n\n"
printf "Which Version of the WebUI Interface do you wish to use?\n" printf "Which Version of the WebUI Interface do you wish to use?\n"
select yn in "Streamlit" "Gradio"; do select yn in "Streamlit" "Gradio"; do
case $yn in case $yn in
Streamlit ) printf "\nStarting Stable Diffusion WebUI: Streamlit Interface. Please Wait...\n"; python -m streamlit run scripts/webui_streamlit.py --theme.base dark --server.address localhost; break;; Streamlit ) printf "\nStarting Stable Diffusion WebUI: Streamlit Interface. Please Wait...\n"; python -m streamlit run scripts/webui_streamlit.py; break;;
Gradio ) printf "\nStarting Stable Diffusion WebUI: Gradio Interface. Please Wait...\n"; python scripts/relauncher.py "$@"; break;; Gradio ) printf "\nStarting Stable Diffusion WebUI: Gradio Interface. Please Wait...\n"; python scripts/relauncher.py "$@"; break;;
esac esac
done done