# Blog · 16 October 2026 · 2 min read

DiffLab: a video studio for generated video on one graphics card

An open-source tool that turns an idea and a few photographs into a chapter of shots, on a single 24 GB card.

I built DiffLab, an open-source tool for making AI video from an idea, photographs and text descriptions.

It is more than a form where you type a prompt and get a clip. DiffLab tries to put something like a full production process around the generation.

How it works

As input you can give it:

  • an idea and a set of photographs;
  • a text split into chapters;
  • an existing video that needs to be rebuilt scene by scene.

Then the system:

  • turns the idea into the structure of a film;
  • works out the characters, places, objects and visual style;
  • splits the story into scenes of roughly 5 to 15 seconds;
  • creates references and a first frame for every shot;
  • generates a video clip with sound;
  • judges the result;
  • lets you regenerate or edit a single bad scene;
  • assembles the shots into chapters and the final video.

Every shot stays an object of its own, with its references, first frame, prompt and finished clip. So one failed scene does not mean regenerating the whole film.

What you can do separately

  • generate a short clip from a photograph and a description;
  • edit an image, including a sequence of changes;
  • assemble full productions from chapters and scenes.

Image editing has its own checking loop: the model compares the result with the original photograph, and failed steps can be rewritten and repeated.

What is inside

The pipeline is built around a few models:

  • Qwen3.8-27B: director, planner, prompt writer and judge of the result;
  • Qwen-Image-Edit: images, key frames and editing;
  • MiniMax H3: video with sound;
  • SeedVR2: frame restoration;
  • RealESRGAN / 4xLSDIR: upscaling;
  • RIFE: frame interpolation.

DiffLab does not use ComfyUI as a hidden layer. It is an ordinary Python project on Diffusers and Transformers, with a web interface in Vue.

Practical details

The project is designed for one NVIDIA card with 24 GB of VRAM. I developed it on an RTX 3090.

You can run it on your own computer, on RunPod, or through a desktop app for Windows, macOS and Linux. The desktop app has a setup wizard, starts the GPU pod, keeps the RunPod key in the system keychain, shows logs and costs, and stops the instance on a limit or when the work is done.

I also tried to do caching properly. Results are tied to their inputs, so a crashed process can be resumed, and changing one shot does not rebuild the whole video.

For now this is not a universal "make a film" button. It is a fairly technical tool for people who want to control the whole process, from script and references to single frames and the final assembly.

github.com/DiverOfDark/difflab

I would be glad to get feedback, especially from anyone who has already built video generation pipelines on local hardware or through RunPod.

← All postsRSS