Running PixArt-Σ/Flux.1 Image Generation on Lower VRAM GPUs: A Short Tutorial in Python | by Youness Mansar | Aug, 2024

Diffusers and Quanto giving hope to the GPU-challenged

Picture era instruments are hotter than ever, they usually’ve by no means been extra highly effective. Fashions like PixArt Sigma and Flux.1 are main the cost, due to their open weight fashions and permissive licenses. This setup permits for artistic tinkering, together with coaching LoRAs with out sharing knowledge outdoors your laptop.

Nonetheless, working with these fashions might be difficult when you’re utilizing older or much less VRAM-rich GPUs. Sometimes, there’s a trade-off between high quality, pace, and VRAM utilization. On this weblog put up, we’ll deal with optimizing for pace and decrease VRAM utilization whereas sustaining as a lot high quality as potential. This method works exceptionally properly for PixArt on account of its smaller measurement, however outcomes may differ with Flux.1. I’ll share some different options for Flux.1 on the finish of this put up.

Each PixArt Sigma and Flux.1 are transformer-based, which implies they profit from the identical quantization methods utilized by giant language fashions (LLMs). Quantization entails compressing the mannequin’s parts to make use of much less reminiscence. It lets you maintain all mannequin parts in GPU VRAM concurrently, resulting in sooner era speeds in comparison with strategies that transfer weights between the GPU and CPU, which might gradual issues down.

Let’s dive into the setup!

Setting Up Your Native Atmosphere

First, guarantee you have got Nvidia drivers and Anaconda put in.

Subsequent, create a python setting and set up all the principle necessities:

conda set up pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

Then the Diffusers and Quanto libs:

pip set up pillow==10.3.0 loguru~=0.7.2 optimum-quanto==0.2.4 diffusers==0.30.0 transformers==4.44.2 speed up==0.33.0 sentencepiece==0.2.0

Quantization Code

Right here’s a easy script to get you began for PixArt-Sigma:

from optimum.quanto import qint8, qint4, quantize, freeze
from diffusers import PixArtSigmaPipeline
import torch
pipeline = PixArtSigmaPipeline.from_pretrained(
"PixArt-alpha/PixArt-Sigma-XL-2-1024-MS", torch_dtype=torch.float16
)
quantize(pipeline.transformer, weights=qint8)
freeze(pipeline.transformer)
quantize(pipeline.text_encoder, weights=qint4, exclude="proj_out")
freeze(pipeline.text_encoder)
pipe = pipeline.to("cuda")
for i in vary(2):
generator = torch.Generator(gadget="cpu").manual_seed(i)
immediate = "Cyberpunk cityscape, small black crow, neon lights, darkish alleys, skyscrapers, futuristic, vibrant colours, excessive distinction, extremely detailed"
picture = pipe(immediate, peak=512, width=768, guidance_scale=3.5, generator=generator).pictures[0]
picture.save(f"Sigma_{i}.png")

Understanding the Script: Listed below are the foremost steps of the implementation

1. Import Needed Libraries: We import libraries for quantization, mannequin loading, and GPU dealing with.
2. Load the Mannequin: We load the PixArt Sigma mannequin in half-precision (float16) to CPU first.
3. Quantize the Mannequin: We apply quantization to the transformer and textual content encoder parts of the mannequin. Right here we apply totally different ranges of quantizations: The Textual content encoder half is quantized at qint4 on condition that it’s fairly giant. The imaginative and prescient half, if quantized at qint8, would make the total pipeline burn up 7.5 G VRAM, if not quantized in any respect would use round 8.5 G VRAM.
4. Transfer to GPU: We transfer the pipeline to the GPU .to("cuda")for sooner processing.
5. Generate Photographs: We use the pipe to generate pictures based mostly on a given immediate and save the output.

Operating the Script

Save the script and run it in your setting. You need to see a picture generated based mostly on the immediate “Cyberpunk cityscape, small black crow, neon lights, darkish alleys, skyscrapers, futuristic, vibrant colours, excessive distinction, extremely detailed” saved as sigma_1.png. Technology takes 6 seconds on a RTX 3080 GPU.

You may obtain related outcomes with Flux.1 Schnell, regardless of its extra parts, however it will necessitate extra aggressive quantization, which might negatively decrease high quality (Until you have got entry to extra VRAM, say 16 or 25 Gigs)

import torch
from optimum.quanto import qint2, qint4, quantize, freeze
from diffusers.pipelines.flux.pipeline_flux import FluxPipeline
pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16)
quantize(pipe.text_encoder, weights=qint4, exclude="proj_out")
freeze(pipe.text_encoder)
quantize(pipe.text_encoder_2, weights=qint2, exclude="proj_out")
freeze(pipe.text_encoder_2)
quantize(pipe.transformer, weights=qint4, exclude="proj_out")
freeze(pipe.transformer)
pipe = pipe.to("cuda")
for i in vary(10):
generator = torch.Generator(gadget="cpu").manual_seed(i)
immediate = "Cyberpunk cityscape, small black crow, neon lights, darkish alleys, skyscrapers, futuristic, vibrant colours, excessive distinction, extremely detailed"
picture = pipe(immediate, peak=512, width=768, guidance_scale=3.5, generator=generator, num_inference_steps=4).pictures[0]
picture.save(f"Schnell_{i}.png")

We will see that quantization of the textual content encoder to qint2 and imaginative and prescient transformer to qint8 is likely to be too aggressive, which had a major influence on the standard for Flux.1 Schnell

Listed below are some options for operating Flux.1 Schnell:

If PixArt-Sigma will not be adequate on your wants and also you don’t have sufficient VRAM to run Flux.1 at adequate high quality you have got two fundamental choices:

• ComfyUI or Forge: These are GUI instruments that fanatics use, they principally sacrifice pace for high quality.
• Replicate API: It prices 0.003 per picture era for Schnell.

Deployment

I had a little bit enjoyable deploying PixArt Sigma on an older machine I’ve. Here’s a transient abstract of how I went about it:

First the checklist of part:

1. HTMX and Tailwind: These are just like the face of the mission. HTMX helps make the web site interactive with out lots of further code, and Tailwind provides it a pleasant look.
2. FastAPI: It takes requests from the web site and decides what to do with them.
3. Celery Employee: Consider this because the arduous employee. It takes the orders from FastAPI and truly creates the photographs.
4. Redis Cache/Pub-Sub: That is just like the communication middle. It helps totally different elements of the mission discuss to one another and bear in mind necessary stuff.
5. GCS (Google Cloud Storage): That is the place we maintain the completed pictures.

Now, how do all of them work collectively? Right here’s a easy rundown:

• While you go to the web site and make a request, HTMX and Tailwind make certain it appears good.
• FastAPI will get the request and tells the Celery Employee what sort of picture to make by Redis.
• The Celery Employee goes to work, creating the picture.
• As soon as the picture is prepared, it will get saved in GCS, so it’s simple to entry.

Service URL: https://image-generation-app-340387183829.europe-west1.run.app

Conclusion

By quantizing the mannequin parts, we will considerably cut back VRAM utilization whereas sustaining good picture high quality and enhancing era pace. This technique is especially efficient for fashions like PixArt Sigma. For Flux.1, whereas the outcomes is likely to be blended, the ideas of quantization stay relevant.

References: