Sora Patches Explained Through Film

Imagine a film print of The Dark Knight. You know what I’m talking about: the reel of cellophane wound around a metal disk that a young man in a red blazer hooks up to a projector in an old-style movie theater. You unroll the film from its scroll and chop off the first 100 frames of cellophane. You take each frame—here the Joker laughing maniacally, there the Batman grimacing —and perform the following strange ritual: You use an X-acto knife to cut a gash in the shape of an amoeba in the first frame. You excise this cellophane amoeba with tweezers as carefully as a watchmaker and put it in a safe place. Then you move on to the next frame: You cut the same amoeba-shaped hole from the same part of the next cellophane frame. You remove this new amoeba—shaped precisely the same as the last one—with tweezers and stack it carefully on top of the first one. You keep going, until you’ve done this to all 100 frames. You now have a multicolored amoeba extruded along its Y-axis. A tower of cellophane that could be run through a projector to show a small area of The Dark Knight, as if someone stuck their hand in a loose fist in front of the projector, letting only a little bit of the movie through. This tower is what’s called a “patch”—a smear of color changing through time. The patch is the basic unit of Sora in the same way the “token” is the basic unit of GPT-4. Tokens are bits of words, while patches are bits of movies. GPT-4 has been trained to take in a sequence of tokens and output the next token in the sequence. Sora has been trained to do the same thing: It takes in a sequence of patches and outputs the next “patch” in the sequence.

A screenshot contains a dense explanatory passage describing how Sora represents videos as temporal “patches,” comparing them with GPT-4 tokens. It uses a painstaking Dark Knight film-cell metaphor to explain how small changing regions of a movie can be stacked and predicted over time, making the technical concept intentionally elaborate and absurd. EN: A screenshot contains a dense explanatory passage describing how Sora represents videos as temporal “patches,” comparing them with GPT-4 tokens. It uses a painstaking Dark Knight film-cell metaphor to explain how small changing regions of a movie can be stacked and predicted over time, making the technical concept intentionally elaborate and absurd.

• @ithueti / 11865 2024-02-17