Hugging Face's logo

jimchoi
/
simpletuner-lora

Text-to-Image
Diffusers
sd3
sd3-diffusers
simpletuner
lora
template:sd-lora
lycoris
Model card Files Community
simpletuner-lora / README.md
jimchoi's picture
jimchoi
Model card auto-generated by SimpleTuner
625c781 verified
preview code
|
raw
history blame
8.66 kB
metadata
license: other
base_model: stabilityai/stable-diffusion-3-medium-diffusers
tags:
  - sd3
  - sd3-diffusers
  - text-to-image
  - diffusers
  - simpletuner
  - not-for-all-audiences
  - lora
  - template:sd-lora
  - lycoris
inference: true
widget:
  - text: unconditional (blank prompt)
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_0_0.png
  - text: A swift and agile elven archer perched in a tree, nocking an arrow.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_1_0.png
  - text: A cyberpunk hunter in neon-lit city alleys, armed with a high-tech rifle.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_2_0.png
  - text: A mighty fantasy knight in gleaming armor, wielding a sword and shield.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_3_0.png
  - text: >-
      A space pirate captain standing on the bridge of a starship, ready for
      adventure.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_4_0.png
  - text: A powerful demonic sorcerer casting a spell in a dark, mysterious chamber.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_5_0.png
  - text: >-
      A friendly robotic assistant with a sleek design, helping a player
      navigate a game.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_6_0.png
  - text: A stealthy ninja warrior crouching in the shadows, ready to strike.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_7_0.png
  - text: >-
      A group of survivors in a post-apocalyptic world, fending off a zombie
      horde.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_8_0.png
  - text: >-
      A brave dragon tamer soaring through the sky on the back of a majestic
      dragon.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_9_0.png
  - text: >-
      A wise medieval wizard in a towering castle, studying ancient tomes of
      magic.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_10_0.png
  - text: >-
      Sun Wukong, the Monkey King from 'Black Myth: Wukong', clad in
      battle-damaged golden armor, wearing a phoenix-feathered purple golden
      crown. His eyes gleam with an indomitable spirit of battle. He wields the
      Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an
      ancient battlefield shrouded in demonic fog, surrounded by shattered
      battle flags and scattered weapons. Wukong's fur billows in the wild wind
      of the battlefield, his muscular physique clearly defined, showcasing his
      unparalleled strength and agility.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_11_0.png
  - text: >-
      Sun Wukong, the Monkey King from 'Black Myth: Wukong', clad in
      battle-damaged golden armor, wearing a phoenix-feathered purple golden
      crown. His eyes gleam with an indomitable spirit of battle. He wields the
      Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an
      ancient battlefield shrouded in demonic fog, surrounded by shattered
      battle flags and scattered weapons. Wukong's fur billows in the wild wind
      of the battlefield, his muscular physique clearly defined, showcasing his
      unparalleled strength and agility.
    parameters:
      negative_prompt: blurry, cropped, ugly
    output:
      url: ./assets/image_12_0.png

simpletuner-lora

This is a LyCORIS adapter derived from stabilityai/stable-diffusion-3-medium-diffusers.

The main validation prompt used during training was:

Sun Wukong, the Monkey King from 'Black Myth: Wukong', clad in battle-damaged golden armor, wearing a phoenix-feathered purple golden crown. His eyes gleam with an indomitable spirit of battle. He wields the Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an ancient battlefield shrouded in demonic fog, surrounded by shattered battle flags and scattered weapons. Wukong's fur billows in the wild wind of the battlefield, his muscular physique clearly defined, showcasing his unparalleled strength and agility.

Validation settings

  • CFG: 3.0
  • CFG Rescale: 0.0
  • Steps: 20
  • Sampler: FlowMatchEulerDiscreteScheduler
  • Seed: 42
  • Resolution: 1024x1024
  • Skip-layer guidance:

Note: The validation settings are not necessarily the same as the training settings.

You can find some example images in the following gallery:

Prompt
A swift and agile elven archer perched in a tree, nocking an arrow.
Negative Prompt
blurry, cropped, ugly
Prompt
A cyberpunk hunter in neon-lit city alleys, armed with a high-tech rifle.
Negative Prompt
blurry, cropped, ugly
Prompt
A mighty fantasy knight in gleaming armor, wielding a sword and shield.
Negative Prompt
blurry, cropped, ugly
Prompt
A space pirate captain standing on the bridge of a starship, ready for adventure.
Negative Prompt
blurry, cropped, ugly
Prompt
A powerful demonic sorcerer casting a spell in a dark, mysterious chamber.
Negative Prompt
blurry, cropped, ugly
Prompt
A friendly robotic assistant with a sleek design, helping a player navigate a game.
Negative Prompt
blurry, cropped, ugly
Prompt
A stealthy ninja warrior crouching in the shadows, ready to strike.
Negative Prompt
blurry, cropped, ugly
Prompt
A group of survivors in a post-apocalyptic world, fending off a zombie horde.
Negative Prompt
blurry, cropped, ugly
Prompt
A brave dragon tamer soaring through the sky on the back of a majestic dragon.
Negative Prompt
blurry, cropped, ugly
Prompt
A wise medieval wizard in a towering castle, studying ancient tomes of magic.
Negative Prompt
blurry, cropped, ugly

The text encoder was not trained. You may reuse the base model text encoder for inference.

Training settings

  • Training epochs: 10
  • Training steps: 10000
  • Learning rate: 0.0001
    • Learning rate schedule: polynomial
    • Warmup steps: 100
  • Max grad norm: 0.01
  • Effective batch size: 1
    • Micro-batch size: 1
    • Gradient accumulation steps: 1
    • Number of GPUs: 1
  • Gradient checkpointing: True
  • Prediction type: flow-matching (extra parameters=['shift=3'])
  • Optimizer: adamw_bf16
  • Trainable parameter precision: Pure BF16
  • Caption dropout probability: 10.0%

LyCORIS Config: 

{
    "algo": "lokr",
    "multiplier": 1.0,
    "linear_dim": 10000,
    "linear_alpha": 1,
    "factor": 16,
    "apply_preset": {
        "target_module": [
            "Attention",
            "FeedForward"
        ],
        "module_algo_map": {
            "Attention": {
                "factor": 16
            },
            "FeedForward": {
                "factor": 8
            }
        }
    }
}

Datasets

wikiart_sargent

  • Repeats: 0
  • Total number of images: 920
  • Total number of aspect buckets: 4
  • Resolution: 1.048576 megapixels
  • Cropped: False
  • Crop style: None
  • Crop aspect: None
  • Used for regularisation data: No

Inference 

import torch
from diffusers import DiffusionPipeline
from lycoris import create_lycoris_from_weights


def download_adapter(repo_id: str):
    import os
    from huggingface_hub import hf_hub_download
    adapter_filename = "pytorch_lora_weights.safetensors"
    cache_dir = os.environ.get('HF_PATH', os.path.expanduser('~/.cache/huggingface/hub/models'))
    cleaned_adapter_path = repo_id.replace("/", "_").replace("\\", "_").replace(":", "_")
    path_to_adapter = os.path.join(cache_dir, cleaned_adapter_path)
    path_to_adapter_file = os.path.join(path_to_adapter, adapter_filename)
    os.makedirs(path_to_adapter, exist_ok=True)
    hf_hub_download(
        repo_id=repo_id, filename=adapter_filename, local_dir=path_to_adapter
    )

    return path_to_adapter_file
    
model_id = 'stabilityai/stable-diffusion-3-medium-diffusers'
adapter_repo_id = 'jimchoi/simpletuner-lora'
adapter_filename = 'pytorch_lora_weights.safetensors'
adapter_file_path = download_adapter(repo_id=adapter_repo_id)
pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.bfloat16) # loading directly in bf16
lora_scale = 1.0
wrapper, _ = create_lycoris_from_weights(lora_scale, adapter_file_path, pipeline.transformer)
wrapper.merge_to()

prompt = "Sun Wukong, the Monkey King from 'Black Myth: Wukong', clad in battle-damaged golden armor, wearing a phoenix-feathered purple golden crown. His eyes gleam with an indomitable spirit of battle. He wields the Ruyi Jingu Bang, its golden body shimmering with light, standing amidst an ancient battlefield shrouded in demonic fog, surrounded by shattered battle flags and scattered weapons. Wukong's fur billows in the wild wind of the battlefield, his muscular physique clearly defined, showcasing his unparalleled strength and agility."
negative_prompt = 'blurry, cropped, ugly'

## Optional: quantise the model to save on vram.
## Note: The model was quantised during training, and so it is recommended to do the same during inference time.
from optimum.quanto import quantize, freeze, qint8
quantize(pipeline.transformer, weights=qint8)
freeze(pipeline.transformer)
    
pipeline.to('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu') # the pipeline is already in its target precision level
image = pipeline(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=20,
    generator=torch.Generator(device='cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu').manual_seed(42),
    width=1024,
    height=1024,
    guidance_scale=3.0,
).images[0]
image.save("output.png", format="PNG")