diffusers-Load pipelines,models,and schedulers

這篇具有很好參考價(jià)值的文章主要介紹了diffusers-Load pipelines,models,and schedulers。希望對(duì)大家有所幫助。如果存在錯(cuò)誤或未考慮完全的地方，請(qǐng)大家不吝賜教，您也可以點(diǎn)擊"舉報(bào)違法"按鈕提交疑問。

https://huggingface.co/docs/diffusers/using-diffusers/loadinghttps://huggingface.co/docs/diffusers/using-diffusers/loading

有一種簡(jiǎn)便的方法用于推理是至關(guān)重要的。擴(kuò)散系統(tǒng)通常由多個(gè)組件組成，如parameterized model、tokenizers和schedulers，它們以復(fù)雜的方式進(jìn)行交互。這就是為什么我們?cè)O(shè)計(jì)了DiffusionPipeline，將整個(gè)擴(kuò)散系統(tǒng)的復(fù)雜性包裝成易于使用的API，同時(shí)保持足夠的靈活性，以適應(yīng)其他用例，例如將每個(gè)組件單獨(dú)加載作為構(gòu)建塊來組裝自己的擴(kuò)散系統(tǒng)。

1.Diffusion Pipeline

DiffusionPipeline是擴(kuò)散模型最簡(jiǎn)單最通用的方法。

from diffusers import DiffusionPipeline

repo_id = "runwayml/stable-diffusion-v1-5"
pipe = DiffusionPipeline.from_pretrained(repo_id, use_safetensors=True)

也可以使用特定的pipeline

from diffusers import StableDiffusionPipeline

repo_id = "runwayml/stable-diffusion-v1-5"
pipe = StableDiffusionPipeline.from_pretrained(repo_id, use_safetensors=True)

Community pipelines是原始實(shí)現(xiàn)不同于DiffusionPipeline，例如StableDiffusionControlNetPipeline.

1.1 local pipeline

from diffusers import DiffusionPipeline

repo_id = "./stable-diffusion-v1-5" # local path
stable_diffusion = DiffusionPipeline.from_pretrained(repo_id, use_safetensors=True)

from_pretrained()方法在檢測(cè)到本地路徑時(shí)不會(huì)下載。

1.2 swap components in a pipeline

可以使用另一個(gè)兼容的組件來自定義任何流程的默認(rèn)組件。定制非常重要，因?yàn)椋?/p>

更改調(diào)度器對(duì)于探索生成速度和質(zhì)量之間的權(quán)衡是重要的。
模型的不同組件通常是獨(dú)立訓(xùn)練的，您可以用性能更好的組件替換掉現(xiàn)有組件。
在微調(diào)過程中，通常只有一些組件（如UNet或文本編碼器）進(jìn)行訓(xùn)練。

from diffusers import DiffusionPipeline

repo_id = "runwayml/stable-diffusion-v1-5"
stable_diffusion = DiffusionPipeline.from_pretrained(repo_id, use_safetensors=True)
stable_diffusion.scheduler.compatibles

from diffusers import DiffusionPipeline, EulerDiscreteScheduler, DPMSolverMultistepScheduler

repo_id = "runwayml/stable-diffusion-v1-5"
scheduler = EulerDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler")
stable_diffusion = DiffusionPipeline.from_pretrained(repo_id, scheduler=scheduler, use_safetensors=True)

可以將PNDMScheduler更換為EulerDiscreteScheduler，在回傳到DiffusionPipeline中。

1.3 safety checker

safety checker可以根據(jù)已知的NSFW內(nèi)容檢查生成的輸出，?

from diffusers import DiffusionPipeline

repo_id = "runwayml/stable-diffusion-v1-5"
stable_diffusion = DiffusionPipeline.from_pretrained(repo_id, safety_checker=None, use_safetensors=True)

1.4 reuse components across pipelines

可以在多個(gè)pipeline中可以重復(fù)使用相同的組件，以避免將權(quán)重加載到RAM中2次

from diffusers import StableDiffusionPipeline, StableDiffusionImg2ImgPipeline

model_id = "runwayml/stable-diffusion-v1-5"
stable_diffusion_txt2img = StableDiffusionPipeline.from_pretrained(model_id, use_safetensors=True)

components = stable_diffusion_txt2img.components

可以將components傳遞到另一個(gè)pipeline中，無需將權(quán)重重新加載到RAM中：

stable_diffusion_img2img = StableDiffusionImg2ImgPipeline(**components)

下面的方式更加靈活：

from diffusers import StableDiffusionPipeline, StableDiffusionImg2ImgPipeline

model_id = "runwayml/stable-diffusion-v1-5"
stable_diffusion_txt2img = StableDiffusionPipeline.from_pretrained(model_id, use_safetensors=True)
stable_diffusion_img2img = StableDiffusionImg2ImgPipeline(
    vae=stable_diffusion_txt2img.vae,
    text_encoder=stable_diffusion_txt2img.text_encoder,
    tokenizer=stable_diffusion_txt2img.tokenizer,
    unet=stable_diffusion_txt2img.unet,
    scheduler=stable_diffusion_txt2img.scheduler,
    safety_checker=None,
    feature_extractor=None,
    requires_safety_checker=False,
)

1.5 checkpoint variants

以torch.float16保存，節(jié)省一半的內(nèi)存，但是無法訓(xùn)練，EMA不用于推理，用于微調(diào)模型。

2. models

from diffusers import UNet2DConditionModel

repo_id = "runwayml/stable-diffusion-v1-5"
model = UNet2DConditionModel.from_pretrained(repo_id, subfolder="unet", use_safetensors=True)

所有的權(quán)重都存儲(chǔ)在一個(gè)safetensors中，可以用.from_single_file()來加載模型。safetensors安全且加載速度快。

2.1 load different stable diffusion formats

.ckpt也可以用from_single_file()，但最好轉(zhuǎn)成hf格式，可以使用diffusers官方提供的服務(wù)轉(zhuǎn)：https://huggingface.co/spaces/diffusers/sd-to-diffusers

也可以使用腳本轉(zhuǎn)：https://github.com/huggingface/diffusers/blob/main/scripts/convert_original_stable_diffusion_to_diffusers.py

python ../diffusers/scripts/convert_original_stable_diffusion_to_diffusers.py --checkpoint_path temporalnetv3.ckpt --original_config_file cldm_v15.yaml --dump_path ./ --controlnet

?A1111 Lora文件，diffusers可以使用load_lora_weights()加載lora模型：

from diffusers import DiffusionPipeline, UniPCMultistepScheduler
import torch

pipeline = DiffusionPipeline.from_pretrained(
    "andite/anything-v4.0", torch_dtype=torch.float16, safety_checker=None
).to("cuda")
pipeline.scheduler = UniPCMultistepScheduler.from_config(pipeline.scheduler.config)

# uncomment to download the safetensor weights
#!wget https://civitai.com/api/download/models/19998 -O howls_moving_castle.safetensors

pipeline.load_lora_weights(".", weight_name="howls_moving_castle.safetensors")

prompt = "masterpiece, illustration, ultra-detailed, cityscape, san francisco, golden gate bridge, california, bay area, in the snow, beautiful detailed starry sky"
negative_prompt = "lowres, cropped, worst quality, low quality, normal quality, artifacts, signature, watermark, username, blurry, more than one bridge, bad architecture"

images = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=512,
    height=512,
    num_inference_steps=25,
    num_images_per_prompt=4,
    generator=torch.manual_seed(0),
).images

from diffusers.utils import make_image_grid

make_image_grid(images, 2, 2)

3.scheduler

scheduler沒有參數(shù)化或訓(xùn)練；由配置文件定義。加載scheduler不會(huì)消耗大的內(nèi)存，并且相同的配置文件可以用于各種不同的scheduler，比如下面的scheduler均可與StableDiffusionPipline兼容。

Diffusion流程本質(zhì)上是由擴(kuò)散模型和scheduler組成的集合，它們?cè)谝欢ǔ潭壬媳舜霜?dú)立。這意味著可以替換流程的某些部分，其中最好的例子就是scheduler。擴(kuò)散模型通常只定義從噪聲到較少噪聲樣本的前向傳遞過程，而調(diào)度器定義了整個(gè)去噪過程，包括：

去噪步驟是多少？隨機(jī)的還是確定性的？用什么算法找到去噪樣本？調(diào)度器可以非常復(fù)雜，并且經(jīng)常在去噪速度和去噪質(zhì)量之間進(jìn)行權(quán)衡。

from diffusers import StableDiffusionPipeline
from diffusers import (
    DDPMScheduler,
    DDIMScheduler,
    PNDMScheduler,
    LMSDiscreteScheduler,
    EulerDiscreteScheduler,
    EulerAncestralDiscreteScheduler,
    DPMSolverMultistepScheduler,
)

repo_id = "runwayml/stable-diffusion-v1-5"

ddpm = DDPMScheduler.from_pretrained(repo_id, subfolder="scheduler")
ddim = DDIMScheduler.from_pretrained(repo_id, subfolder="scheduler")
pndm = PNDMScheduler.from_pretrained(repo_id, subfolder="scheduler")
lms = LMSDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler")
euler_anc = EulerAncestralDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler")
euler = EulerDiscreteScheduler.from_pretrained(repo_id, subfolder="scheduler")
dpm = DPMSolverMultistepScheduler.from_pretrained(repo_id, subfolder="scheduler")

# replace `dpm` with any of `ddpm`, `ddim`, `pndm`, `lms`, `euler_anc`, `euler`
pipeline = StableDiffusionPipeline.from_pretrained(repo_id, scheduler=dpm, use_safetensors=True)

4.DiffusionPipline explained

作為一個(gè)類方法，DiffusionPipeline.from_pretrained()做兩件事，1.下載推理所需的權(quán)重并緩存，一般存在在.cache文件中，2.將緩存文件中的model_index.json進(jìn)行實(shí)例化。

feature_extractor--CLIPFeatureExtractor(transformers)；scheduler--PNDMScheduler；text_encoder--CLIPTextModel(transformers)；tokenizer--CLIPTokenizer(transformers)；unet--UNet2DConditionModel；vae--AutoencoderKL

{
  "_class_name": "StableDiffusionPipeline",
  "_diffusers_version": "0.6.0",
  "feature_extractor": [
    "transformers",
    "CLIPImageProcessor"
  ],
  "safety_checker": [
    "stable_diffusion",
    "StableDiffusionSafetyChecker"
  ],
  "scheduler": [
    "diffusers",
    "PNDMScheduler"
  ],
  "text_encoder": [
    "transformers",
    "CLIPTextModel"
  ],
  "tokenizer": [
    "transformers",
    "CLIPTokenizer"
  ],
  "unet": [
    "diffusers",
    "UNet2DConditionModel"
  ],
  "vae": [
    "diffusers",
    "AutoencoderKL"
  ]
}

下面是runway/stable-diffusion-v1-5的文件夾結(jié)構(gòu)：

.
├── feature_extractor
│?? └── preprocessor_config.json
├── model_index.json
├── safety_checker
│?? ├── config.json
│?? └── pytorch_model.bin
├── scheduler
│?? └── scheduler_config.json
├── text_encoder
│?? ├── config.json
│?? └── pytorch_model.bin
├── tokenizer
│?? ├── merges.txt
│?? ├── special_tokens_map.json
│?? ├── tokenizer_config.json
│?? └── vocab.json
├── unet
│?? ├── config.json
│?? ├── diffusion_pytorch_model.bin
└── vae
    ├── config.json
    ├── diffusion_pytorch_model.bin

可以查看組件的屬性和配置：文章來源地址http://www.zghlxwxcb.cn/news/detail-740809.html

pipeline.tokenizer
CLIPTokenizer(
    name_or_path="/root/.cache/huggingface/hub/models--runwayml--stable-diffusion-v1-5/snapshots/39593d5650112b4cc580433f6b0435385882d819/tokenizer",
    vocab_size=49408,
    model_max_length=77,
    is_fast=False,
    padding_side="right",
    truncation_side="right",
    special_tokens={
        "bos_token": AddedToken("<|startoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=True),
        "eos_token": AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=True),
        "unk_token": AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=True),
        "pad_token": "<|endoftext|>",
    },
)

到了這里，關(guān)于diffusers-Load pipelines,models,and schedulers的文章就介紹完了。如果您還想了解更多內(nèi)容，請(qǐng)?jiān)谟疑辖撬阉鱐OY模板網(wǎng)以前的文章或繼續(xù)瀏覽下面的相關(guān)文章，希望大家以后多多支持TOY模板網(wǎng)！