Diffusion models

Diffusion Models are generative models, which means that they are used to make data that looks like the data they were trained on. Diffusion Models work by destroying training data by adding Gaussian noise to it over and over again, and then learning how to get the data back by reversing this process of adding noise.

What is Stable Diffusion?

Stable Diffusion is an algorithm developed by Compvis (the Computer Vision research group at Ludwig Maximilian University of Munich) and sponsored primarily by Stability AI, a startup that aims to be the driving-force behind a grass-roots, open-source AI revolution. The algorithm itself builds on ideas from Open AI’s DALL-E 2, Google’s Imagen and other image generation models, with a lot of optimisations on top.

Stability AI employ Katherine Crowson as their lead coder. If you haven’t heard the name, Katherine is one of the main driving forces behind the AI art explosion in the last 1.5 years. She was the first to combine VQGAN with OpenAI’s CLIP, and then she went on to develop the CLIP-Guided diffusion method underpinning Disco Diffusion, NightCafe and various other AI image generation websites.

Stable Diffusion is a bit different to those algorithms in that it is not “CLIP-Guided”. Instead, a version of CLIP is “Frozen” and embedded into the generation algorithm itself. This is an idea borrowed from Imagen, and makes stable diffusion a LOT faster than it’s CLIP-guided ancestors.

Incredibly, compared with DALL-E 2 and Imagen, the Stable Diffusion model is a lot smaller. While DALL-E 2 has around 3.5 Billion parameters, and Imagen has 4.6 Billion, the first Stable Diffusion model has just 890 million parameters, which means it uses a lot less VRAM and can actually be run on consumer-grade graphics cards.