Before the Caption

Why are image generators tied so closely to language models? They arrived together: first DALL-E 2 in April 2022 and ChatGPT seven months later, out of the same company, and every jump in one has come with a jump in the other ever since.

Part of that tie is obvious: to generate an image you have to ask for one, and asking means words the model already knows. The surprise is that the words are needed at the other end too. So the tie was never to the language model. It was to the captions, written by people, one photograph at a time.

Three weeks ago, a team including Fei-Fei Li tested that assumption directly. They built a family of generators called Halcyon from scratch with no language model inside them at all, and the models beat considerably larger systems. The language model proved unnecessary. The captions did not: Halcyon still needed a hundred million photographs with words already attached, because its only language supervision comes from caption tokens paired with images.

These machines name before they see, and cannot start until somebody has written down what is in the picture. A person looking at a photograph works the other way, and the gap is measurable: recognition is finished in under 150 milliseconds, well before any word is available. We see and then name.

Photography sits on our side of that line. It is the least discussed property of the medium and, with generated images now arriving by the million, the one that matters most.

What a machine needs from a photograph

A photograph presents a machine with a difficulty unrelated to resolution. A sensor records every variation in front of it, and the variations themselves give no indication of which ones matter. Light, angle, and surface all change at once, and the pixels carry no opinion about which differences should be treated as the same thing and which make two things genuinely different. Settling that requires categories, and the categories a machine can be steered by are the ones we have already named.

For four years the field believed those categories were the property of language models. Building a generator called Imagen in 2022, a team at Google asked which half of the system was worth enlarging, the half that reads the words or the half that draws, and reported that increasing the size of the language model boosts both sample fidelity and image-text alignment much more than increasing the size of the image diffusion model. The half handling the words had never seen a picture, having been trained on text alone, and the team locked it so that it could learn nothing further. Enlarging that locked, sightless half improved the pictures more than enlarging the half that actually draws them.

DALL-E 3 pointed the same way a year later, its advance coming from the captions rather than the drawing. OpenAI trained a model to write long descriptions of its own training pictures, then trained the generator on those, with 95% synthetic captions and 5% ground-truth captions. Words written about pictures are what made the pictures better.

So Halcyon sharpens the account rather than demolishing it. The lever was never the size of the language model, and the categories cannot be acquired secondhand from something that has read the internet, which means image generation depends on captioning, the task our industry has always paid and respected the least.

What the viewer throws away

The uncomfortable part is that a human viewer does the same reduction to a photograph, and does it faster.

Follow what happens when someone looks at a picture: Neurons in the primary visual cortex respond to edges at particular angles, and that is all they respond to. The areas above them assemble those edges into contours, then into the curves and simple shapes that make up a nose or a wheel, and by the inferior temporal cortex, single neurons are firing for a whole face wherever it appears in the frame and whatever the light happens to be doing to it. Each stage gains that reliability by discarding the specifics underneath, so what arrives at the top is the identity of a face, while the fall of shadow across the cheek, the thing you waited all afternoon for, is gone.

All of this happens faster than thinking. Thorpe and his colleagues flashed photographs in front of people for twenty milliseconds and found the recognition finished in under 150 milliseconds. Three weeks of training on the same images did not make it any quicker. It is involuntary, and practice does not improve it.

The sequence does not stop at the face. From the inferior temporal cortex, the signal carries on into the medial temporal lobe, and along that run the representation grows steadily less attached to how the thing was delivered. First it stops mattering how the subject was lit or where it sat in the frame. Then it stops mattering whether it arrived as a picture at all.

In 2005, recording from electrodes implanted in epilepsy patients, Rodrigo Quian Quiroga’s team found single neurons that fired for photographs of one person, drawings of that person, and that person’s name spelled out in text. Later work added the spoken name and recall with no stimulus present at all. “What matters for the human is the concept, not how you trigger the concept,” as Quiroga put it. When OpenAI looked inside CLIP in 2021, it found artificial neurons behaving the same way, and said so, citing him.

Note where this happens, because the medial temporal lobe is the memory system. Your photograph and its caption become one item in memory rather than in perception. Which is to say the viewer keeps roughly what the caption keeps — a man on a bench — and the file keeps everything the two of them discarded.

Our order runs that way for a reason, and the dates are lopsided enough to make it plain. The first fossils of eyes appear in the lower Cambrian, about 540 million years ago, and about a third of the human cerebral cortex is still given over to vision. Language is far younger. A genomic survey published last year concludes that the capacity for it was in place at least 135,000 years ago. Against 540 million, that is a rounding error. Fedorenko, Piantadosi and Gibson have argued from imaging and brain injury that language is a tool for communication, contrary to a prominent view that we use language for thinking. If right, the system that builds our picture of the world and the system that passes it to someone else are two different systems, and the second one is recent.

The near side of language

Photography is the only recording technology that operates before the naming.

The obvious objection is commercial. Most photography is commissioned, briefed, art-directed, and conceived entirely in words, so how can it be pre-verbal? Because a brief constrains a photograph without specifying it. The commission can ask for the CEO looking approachable in the new office, and everything that makes the resulting frame that frame — where the hands settled, the light at four o’clock, an expression that lasted an eighth of a second — was absent from the brief and could not have been put there. Words draw a boundary, and something unnamed (creativity?) fills it. In a generated image, the description doesn’t draw a boundary around the content; it supplies the content, so nothing unnamed fills the file. That holds for the most heavily directed assignment ever shot.

Reception generalizes the point past the making. Every photograph, however it was commissioned, is read in under 150 milliseconds by a system with no words available yet. So words come before the photograph in the brief, and after it in the caption, and at the two moments that matter, when the shutter opens and when someone looks, they are not there at all.

The deeper objection is that there is no pre-conceptual seeing at all, that we only ever see through trained categories. That objection is correct, and it leaves the argument standing, because concept-laden is a different thing from language-laden. The distinctions the visual system draws are learned; they go unnamed, and they are vastly finer than any vocabulary — you can tell thousands of faces apart and name a handful of them. The machines show the same limit from the other side. A benchmark published last year found that vision-language models handle negation at close to chance level, and that scaling them up does not help; they read a negated caption as though it were affirmative. A system assembled out of captions has little grip on what is absent, because absence is rarely what a caption records. That is what a photograph records and what a caption cannot.

The surplus

This is what separates a photograph from a generated image, and the separation is not about accuracy or trust. A generated image holds what somebody named: what you asked for, what a language model added to your prompt before the generator saw it, or what sat in the captions it trained on. Ask for a car, and it arrives on a road, because that is what accompanies cars in those captions.

It will not spontaneously put the car in space. It is missing the creative spark. For now.

A photograph was made before any description of it existed, which is why it routinely holds things that were never requested, never noticed at the time, and never written down, and why those things stay in the file whether or not a viewer ever climbs down to find them.

A photograph lives in perception and ends in description. A generated image starts in description and stays there.

Every photographer and every agency already knows this from the ledger. A picture can be licensed a second and a third time for uses that did not exist when it was taken, and the value of an archive has never been the sum of what its catalog says it contains. That unrequested remainder is what you are actually selling the second time.

This recasts a job the industry has done since it first began sending pictures out, and has always treated as overhead. The caption was written by hand on the back of the print long before there was a field to type it into. Keywording, controlled vocabularies, taxonomies, metadata schemas, and now training captions do the same job, mechanized: naming after the fact what the camera caught before naming, and it taught the machines to see. It was never clerical. It was the conversion of the pre-verbal into the verbal, done by hand, at scale, for almost no money, and the industry is still the only place where that conversion happens.

The machines will keep needing it, because they have no other way in. And photography will keep putting into the record what no caption written afterward can carry, which is why we keep making photographs at all instead of writing descriptions and having them rendered.