Do Diffusion Models Learn to Generalize
Basic Visual Skills?

Princeton University

NeurIPS 2026


Each model is trained from scratch on one skill. We move the requested value through the training range (gray) and past it. The outline shows the requested size, position or angle, and the border turns from green to red when the sample stops matching. The farther we go beyond the training range, the more often the models miss the requested value.

Abstract

Diffusion models generate high-quality images, but how well they generalize is hard to measure. Current evaluations rely on human preference and prompt-following benchmarks for text-to-image models and on distribution-level metrics such as Fréchet Inception Distance (FID) and Inception Score (IS) for class-conditional models, but these cannot tell whether models learn basic visual skills or largely match training patterns. To address this gap, we introduce the Visual Generative Lab (VGL), a controlled experimental framework for studying how diffusion models learn and generalize basic visual skills, including size, position, rotation, count, color, shape, and their compositions. By generating synthetic training data for each skill from explicit rules, VGL allows precise measurement of generalization with task-specific, rule-based metrics. Training models from scratch on these data reveals three patterns in this controlled setting: (1) with our default design, extrapolation is limited: size and position degrade within a few pixels of the training range, count reaches only about 50% accuracy on out-of-range counts, and rotation extrapolation accuracy varies widely across seeds; (2) for compositional generalization, the number of distinct basic visual skill combinations matters more than dataset size; (3) skills mostly fail independently, except position.

Visual Generative Lab (VGL)

It is hard to tell what a diffusion model has actually learned. Metrics like FID and IS score a whole set of images at once, so they do not show which skills a model gets wrong or how. Benchmarks like GenEval and T2I-CompBench test skills such as counting, but only on pre-trained text-to-image models. Nobody knows exactly what those models saw during training, so a good score could mean the model generalizes, or that it has seen something similar before.


This leads to our question:
do diffusion models generalize basic visual skills
beyond their training data, or do they mostly match training patterns?


The six VGL skills: size, position, rotation, count, shape and color.

To answer it, we built the Visual Generative Lab (VGL). We draw simple images from rules and train diffusion models on them from scratch. Because we generate the data, we decide exactly which values of each skill, and which combinations of skills, the model sees in training. Each panel above changes one skill and holds the others fixed. The model receives each value as a number or a label, so asking for an unseen value is a clean test of generalization.

We use size as an example. We draw circles on a 64×64 pixel canvas and train only on the 16 whole-number radii from 5 to 20 pixels. Then we test three things: interpolation, a radius between training values such as 12.5; extrapolation, a radius outside the range such as 25; and composition, size combined with other skills.

SkillTypeDomainExtrapolation values
sizecontinuousradius ∈ [5, 20] px{1, 2, 3, 4} ∪ {21, 22, …, 30} px
positioncontinuous (2D)(x, y) ∈ [−18, 18]2 pxL∞ ≥ 4 px outside [−18, 18]2 (320 positions)
rotationcontinuousangle ∈ [45°, 315°]{0°, …, 44°} ∪ {316°, …, 359°}
countdiscrete{2, 3, …, 7} objects{0, 1, 8, 9}
shapecategorical4 classesn/a
colorcategorical8 classesn/a

A short program checks every generated image, with no learned or perceptual metric involved. For size, we compare the mask of the generated circle with the target mask and count the sample as correct when their intersection-over-union (IoU) is at least 0.90. For the other skills we use the center of the mask (position), template matching (rotation), connected components (count), shape templates (shape) and the nearest palette color (color).


Extrapolation Is Limited

We first train our default model, a DiT-S/2, on one skill at a time. For size and position, it gets training and in-between values right but fails a few pixels outside the training range. For count, it is right on only 52.5% of counts just outside the range. Rotation is different: some training runs extrapolate perfectly and others do not.

IoU per radius and angular error per angle, inside and outside the training range.

Left: IoU at each radius. Right: angular error at each angle. Lines are means over seeds (three for size, ten for rotation), and bands show ±1 standard deviation. The dashed line is what a model would score if it always drew the nearest training value.

Size

Samples from one size model as the requested radius moves from 1 to 30 px. The ring is the requested circle, and the plot traces the radius of the generated circle for three samples.

We train on radii from 5 to 20 pixels and test on radii 1 to 4 and 21 to 30. The model gets every training radius right and 96.8% of in-between radii, but only 51.5±7.5% of radii outside the range. Every seed still gets at least half its samples right up to radius 24 and falls below half from radius 27 on. Below the range, every seed works at radius 3 and fails at radius 1. The model extrapolates about 4 px above the range and 2 px below it.

Position

Samples from one position model as the requested center moves along the x-axis, up to 6 px outside the training square. The ring is the requested circle, and the plot traces the detected x-coordinate for three samples.

We train on a 32×32 grid of positions in [−18, 18]2 pixels, then ask for the 320 positions at least 4 pixels outside that square (L∞ distance). With a 2-pixel tolerance, the model places every training position and 97.4±3.2% of in-between positions correctly, but only 29.7±16.8% of the far positions. Accuracy falls from 49% at 4.6 px outside the square to 11% at 5.8 px, and the missed circles land closer to the training square than we asked.

Rotation

One sample from each of ten seeds trained on the same data with the same settings, as the requested angle moves from 300° through 0° to 60° and back, crossing the out-of-range arc around 0°. The outline is the requested arrow.

We draw asymmetric arrows and train on every whole degree from 45° to 315°, then ask for the angles outside that interval. Whether the model extrapolates depends on the training seed. With the same data and settings, four of ten seeds get the out-of-range angles right to within 2° on average, three are off by 8° to 15°, and three fail near 0°, the angle farthest from the training range, with errors above 30°. Mean accuracy at the 5° threshold is 68.0%. Rotation is the only skill where some seeds get every out-of-range value right.

Count

Four samples from the count model shown in the paper's qualitative figure, for requested counts 0 to 9.

We train on images with 2 to 7 circles and ask for 0, 1, 8 and 9. The model counts every training value correctly, but gets only 52.5% of the counts just outside the range right.

Design choices

Table of single-skill accuracy under different design choices.

Entries are mean±std accuracy (%) over three seeds, or ten for the rotation baseline, and bold rows are our defaults. To check whether these limits come from our setup, we change one design choice at a time: the embedding, the conditioning, pixel or latent space, the training objective, the architecture and the model size. A linear embedding extrapolates better than sinusoidal or rotary embeddings on size, position and rotation, and pixel-space models extrapolate better on those three skills than models trained in the latent space of a pre-trained VAE. Part of that gap may come from sequence length: the latent model sees 16 tokens per image and the pixel model sees 1024. Beyond that, no choice extrapolates better on every skill. AdaLN helps position and count, while concatenation is better on size and rotation. The U-Net loses most of the rotation extrapolation but does comparatively well on count. DiT-L improves count and position but extrapolates no further on size or rotation.


Coverage Matters More Than Dataset Size

Most prompts ask for several skills at once, such as a red triangle. To test this, we list every combination of skill values, train on only some of them, and check whether the model can draw the rest. We vary two things: coverage, the share of combinations seen in training, and density, the number of images per seen combination.

Compositional evaluation categories and accuracy on held-out combinations by coverage and dataset size.

(a) Test categories on a two-skill grid: A is training, B is interpolation (each value seen, combination unseen), C moves one skill out of range, and D moves both. (b) Color + shape models trained on four shapes and eight colors with 5k, 10k or 15k images in total. Bars show mean accuracy on held-out combinations (both skills correct) over three seeds, and error bars show the range across seeds. For a fixed number of images, seeing more combinations helps more than seeing the same combinations more often. Higher coverage raises held-out accuracy even though each seen combination then gets fewer images. More images per combination at fixed coverage gives no consistent gain.

Per-combination shape and color accuracy at 25, 50 and 75 percent coverage.

Results for the 10k-image models, with one cell per combination. The top row scores shape and the bottom row scores color. Black cells are training combinations, and held-out cells show the fraction of correct samples, averaged over three seeds. Held-out shape accuracy grows with coverage, and at 75% coverage both skills are right on every held-out combination.


Skills Mostly Fail Independently, Except Position

Next, we push one skill out of its training range and keep the others in range. If a model learned each skill separately, only the out-of-range skill should break, and the others should stay correct.

Splitsizepositionshapecolor
In range100.0100.0100.0100.0
size OOD54.6100.0100.0100.0
position OOD79.50.032.0100.0

Accuracy (%) of each skill for one model trained on size, position, shape and color together at 75% coverage. Gray marks accuracy below 90%. An out-of-range size leaves position and shape at 100%. An out-of-range position also breaks skills that are still in range: shape accuracy falls from 100% to 32.0%, even though we asked for shapes the model knows. The two three-skill models show the same pattern (20.4% and 59.9% shape accuracy), and in two-skill models an out-of-range position lowers in-range size to 8.9% and rotation to 54.4%. Color stays at 100% on every split, so some skills are tied to position and others are not.


Discussion

In our setting, diffusion models handle unseen values and combinations inside their training data well, but rarely carry a skill far beyond it. If large text-to-image models work the same way, much of what looks like generalization may be interpolation within their training data. Data providers and model trainers would then need data that covers the values and combinations users ask for, rather than just more data, and evaluations should test outside that coverage. Our data are synthetic and we mainly study a DiT backbone, so how far this carries over to natural images is still open. We hope VGL is a useful place to study these questions in richer settings.

BibTeX

@inproceedings{sethi2026vgl,
  title     = {Do Diffusion Models Learn to Generalize Basic Visual Skills?},
  author    = {Sethi, Amish and Zeng, Boya and Chai, Wenhao and Liu, Zhuang},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}