Staffusion

Overview

Introduced: 2022 in Germany (HH - Those who get it, get it)
Purpose: Converts text prompts into images using a diffusion model.

Diffusion Model

A generative model that learns the probability distribution from data to sample and create new, non-existent images.

Why Use Distributions?

• Example: Creating a fake identity for a criminal (age and height)
  • Gather statistics for age and height
  • Draw probability distributions for both
  • Sampling Approaches:
    • Approach 1: Random sampling from age and height
      • Can produce nonsensical combinations
    • Approach 2: Joint Distribution
      • Consider age and height as dependent variables
      • Use joint distribution to model the relationship
      • Each pixel is a distribution, and the entirety of all the pixels forms a Large Joint Distribution

Goal

• Treat data as a large Giant Distribution
• Use neural networks to learn the complex distribution
• Sample from the complex distribution to generate fake data

Latent Variable

• Forward/Diffusion Process: Transform the original image into a noise image
  • Start with the initial image
  • Add noise iteratively to create a progressively noisier image
  • Continue for 1000+ steps until the image is fully noisy
  • This process is fixed and standard
• Reverse Process:
  • Train a neural network to perform the reverse process
  • Neural network learns to denoise the image

Maths for Staffusion

• The next state of the noise image is generated by a Bayesian relationship between states
• We can sample from the distribution to go from the original image to an image at a timestamp t

ELBO (Evidence Lower Bound)

• To find the optimal fake identity, we use marginalization
• Marginalizing over all latent features is complex
• ELBO is the lower bound for the likelihood (for p(x))
• Maximizing the lower bound also maximizes the likelihood
• Loss function: Create a model to predict the amount of noise at a particular step t
• Use gradient descent in the training loop to learn the loss

Learning Context of the Data

• Approach 1: Combine image and context during training
  • Joint distribution over images and conditioning signals (P(X,C))
  • Not ideal as it complicates marginalization (P(X))
• Approach 2: Conditioning during training
  • Build the model using U-net
  • U-net identifies noise at a particular timestamp
  • Pass the prompt with the noisified input for better context
  • Model learns to remove noise based on the prompt
  • Model becomes both conditioned and un-conditioned

Inference Flow - Classifier Free Guidance

1. Step 1:
   • Start from pure noise at timestamp T = 1000
   • Provide the prompt to the U-net
   • Predict and remove the noise to move towards the desired output
2. Step 2:
   • Start from pure noise at timestamp T = 1000 without prompt
   • Predict and remove the noise to move towards any possible output
3. Step 3:
   • Combine results from step 1 and step 2
   • Determine the alignment with the prompt
   • Train the model by alternating between giving and not giving the prompt

CLIP - Contrastive Language - Image Pre-training

• Purpose: Understand the prompt using embeddings
• Embeddings: Vectors representing the words
• CLIP Text Encoder: Extract embeddings
  • Connects text with images
• Use the trained encoder's embeddings as a Conditioning Signal for U-net to denoise as desired

Limitations of U-net

• Large images (e.g., 512x512) require substantial computational power
• Solution: Compress the image using Variational Auto Encoder

Latent Diffusion Model

• Stable Diffusion: Also known as Latent Diffusion Model
• Learn the latent representation of the data
• Compress data and learn the compressed version
• Rebuild from the compressed version to generate the required output
  • Similar to sending a zipped file and unzipping it on the receiving end
• Use latent space and decoder to reconstruct the image
• More noise allows more change in the U-net model
• Less noise allows less change in the model