It is hard to tell what a diffusion model has actually learned. Metrics like FID and IS score a whole set of images at once, so they do not show which skills a model gets wrong or how. Benchmarks like GenEval and T2I-CompBench test skills such as counting, but only on pre-trained text-to-image models. Nobody knows exactly what those models saw during training, so a good score could mean the model generalizes, or that it has seen something similar before.
To answer it, we built the Visual Generative Lab (VGL). We draw simple images from rules and train diffusion models on them from scratch. Because we generate the data, we decide exactly which values of each skill, and which combinations of skills, the model sees in training. Each panel above changes one skill and holds the others fixed. The model receives each value as a number or a label, so asking for an unseen value is a clean test of generalization.
We use size as an example. We draw circles on a 64×64 pixel canvas and train only on the 16 whole-number radii from 5 to 20 pixels. Then we test three things: interpolation, a radius between training values such as 12.5; extrapolation, a radius outside the range such as 25; and composition, size combined with other skills.
| Skill | Type | Domain | Extrapolation values |
| size | continuous | radius ∈ [5, 20] px | {1, 2, 3, 4} ∪ {21, 22, …, 30} px |
| position | continuous (2D) | (x, y) ∈ [−18, 18]2 px | L∞ ≥ 4 px outside [−18, 18]2 (320 positions) |
| rotation | continuous | angle ∈ [45°, 315°] | {0°, …, 44°} ∪ {316°, …, 359°} |
| count | discrete | {2, 3, …, 7} objects | {0, 1, 8, 9} |
| shape | categorical | 4 classes | n/a |
| color | categorical | 8 classes | n/a |
A short program checks every generated image, with no learned or perceptual metric involved. For size, we compare the mask of the generated circle with the target mask and count the sample as correct when their intersection-over-union (IoU) is at least 0.90. For the other skills we use the center of the mask (position), template matching (rotation), connected components (count), shape templates (shape) and the nearest palette color (color).