Visual media has come a long way since the first prehistoric cave dwellers used the flickering light of a torch to revive hand-drawn art on their walls. Today, the pixel — despite its modest low-resolution origins — represents the current pinnacle of digital display technology. In his new book, Biography of Pixel, Pixar co-founder Alvy Ray Smith examines the fascinating history and development of image elements (hence “pix” – “el”) from their often disputed beginnings in the laboratories of pioneering computer researchers like Alan Turing to their ubiquitous presence in modern life. In the paragraph below, Smith looks back to the old bad times before digital imaging to explain the science behind our brain’s ability to perceive movement through the rapid flash of static images.
Taken from ‘Biography of Pixel‘by Alvy Ray Smith (MIT Press, 2021)
How the movies were really over
What did the film’s inventors do (or didn’t) make the system they gave us so imperfect? First, they did not give us the current samples as needed by sampling. The frames of the film are thick. They have a duration. The camera shutter is open for a short exposure time. The moving subject moves during that short interval and is thus lightly smeared on the frame during the exposure time of the film. It’s like what happens when you try to take a photo with a long exposure where your child throws a ball and his hand is just blurred. It turned out that this was the saving grace of cinema as it was actually practiced.
Second, they made each frame project the projector twice (at least). Ouch! This is not sampling. Why did the inventors do that? The simple economy required it: 24 frames per second costs half as much film as 48 frames per second. But the eye needs to be refreshed about 50 times per second, or the image on the retina fades between frames. In fact, 48 is close enough to 50 to work in dark theater. How to get 48 out of 24? You display each frame twice! If you display only 24 frames per second, the screen will flash. Hence the “jerks” from the first days of cinema before higher frame rates were adopted.
The third thing the original inventors did was to turn off the light between the projected frames. This meant that 48 times per second nothing (blackness) was projected into the eye – inside the pupil, onto the retina. It’s suitable for film machines – both the camera and part of the projector – to “shut up” on ink like this between frames. Allows time for the next frame of the film to mechanically advance into position. In-camera, it prevents the film from capturing the real world during the film’s physical progress. It keeps moving film out of sight in the projector because it is physically advanced.
When you ask how a movie projector works, some people say something like this: There is an upper roll of film that is the source of the film, and a lower roll for recording. The film moves from the roll to the reel and passes between the projector’s light source and its lens, which magnifies the frame size image to the screen size. In other words, the film is constantly moving past the light source. But it doesn’t work. The eye sees exactly what is there, and with this scheme the eye would see how one frame slides while the next frame slides on the opposite side. He would see skating. And it won’t work.
The projector actually does just that: it brings each frame into a fixed position with the light source blocked. It is a shutter function. Then the shutter opens and the illuminated frame is projected on the screen. Then the shutter closes. It then reopens and the illuminated frame is projected on the screen a second time. Then the shutter closes and the next frame slides into position and so on.
We have just described the discrete or intermittent motion of a film through a projector, as opposed to the unfeasible continuous motion. The same idea applies to the camera. The physical device that performs this action is called, in fact, an intermittent movement. This is a key term in cinema history that can be compared to a conditional branch in computer history. The crazy rush to the movie machine involved who got the correct operation of the projector first, and that depended on who got the occasional movement that works properly. It is a term that defines.
To summarize: an actual film-based film projector does not reconstruct a continuous visual flow from frame samples and does not present it to the eye. Instead, it sends “fat samples” – thick with time and smeared movements – directly into the retina of the eye. Each frame sends twice, and in between it sends black. It is up to the brain to reconstruct the movement from these inputs. How does it work?
In a way, the eye-brain system “reconstructs the visual flow” that is represented by the visual patterns of fat it receives. Of course, it doesn’t really do that. The intensity of light comes through the pupil as input. But the output from the eye to the brain, through the optic nerve, is an electrochemical impulse sequence. Neural impulse trains are not visual currents. It is possible that the retina actually reconstructs the visual flow, and then converts it into impulse trains for brain consumption. The reactions of some neurons in the eye certainly indicate an expansion function, along with a high positive crest and negative lobes. But brain activity goes beyond the scope of this book. Instead, let’s concentrate on the usual explanations of motion perception from a series of photographs.
Perception of movement
The classic explanation is old persistence of vision. This is a true characteristic of human vision: once the image stimulus on the retina ceases, we continue to observe that image briefly. But the persistence of the vision only explains why you don’t see the blackness between the frames in the case of film-based films. If an actor or animated character moves to a new position between shots, then – with the persistence of vision – you should see him in both positions: two Humphrey Bogart, two Buzz Lightyears. In fact, your retinas see both, one fading as the other enters – each frame is projected long enough to ensure that. It is the persistence of vision. But that doesn’t explain why you perceive one moving object rather than two objects in different positions. What your brain does with information from the retina determines whether you perceive two Bogart in two different positions or one Bogart moving between them.
Psychophysicists performed experiments to determine the characteristics of another real phenomenon in the brain, the so-called apparent movement. The experiments do not explain how the brain perceives movement, but describe the limitations of the phenomenon. A small white dot on a black background is represented on the retina of the subject. Then that point is removed and the other point is displayed in a different position. Experimenters can change two things, the spatial separation of two points, and the time delay between position changes. The brain perceives one point here and another there, but only if distance and delays are sufficient. If the distance and delays are short, the brain notices that the point is moving from one position to another. It is apparent movement because the eye is not shown actual movement. The brain perceives what it does not see.
The consistency of the vision is such that we continue to perceive the first image when the second arrives. That sounds a lot like expanding the frame. The short duration frame expands over time and adds to the next frame also expands over time. It is as if the retina is expanding the image and adding successive frames of expansion. Something like this has to happen because we perceive a continuous visual field even though the film projector does not display it. You can consider the shape of the persistent function of the eye to be the shape of the frame expander built into us human observers. Another reason we can assume that the eye-brain system has to do reconstruction, one that implicitly uses the sampling theorem, is that we observe exactly the errors we would expect to be a real mechanism — such as the wheels of a wheel turning backwards.
Classic target animation — old varieties of celluloid ink — relies on the phenomenon of apparent motion. Old animators intuitively knew how to keep successive frames of movement within its “not too far, nor too slow” boundaries. If they were to cross those boundaries, they had tricks to help us spot the movement. They drew real lines of speed, which showed the brain the direction of movement and implied that it was fast, like blurring. Or they provided a POOF the dust that marked the rapid descent of Wile E. Coyote as he unexpectedly descended from the flesh in search of that truly cunning Road Runner. They provided visual language that the brain could interpret.
Exceed the apparent limits of movement – without the tricks of these animators – and the results are ugly. You may have seen old school stop-motion animations — like classic swordfighting skeletons by Ray Harryhausen in Jason and the Argonauts (1963) – tormented by the awkward twitching of characters. You see at least twice – several edges of the skeleton at the same time – and correctly interpret it as movement, but painful. The edges stutter or “tremble” or “strobos” across the screen. These words reflect the pain inflicted by the staccato movement.
Live movies are sequences of discrete shots, just like animations. Why don’t these movies stutter? (Imagine directing Mind Thurman to stay within the boundaries of “not too far, not too slow.”) There is a general explanation that works. It’s called motion blur, and it is simply beautiful. The shot taken by a real film camera is thick with duration. It’s not a pattern at one point like Road Runner or Harryhausen’s frame. Motion blur is what you see in a photo when the subject moves and the shutter button is not fast enough to stop the movement. In still photos, this is often an unwanted result, but it has been shown to be a feature in movies. Without blur, all the films would look as jerky as Harryhausen’s skeletons – unless Uma miraculously stayed within bounds. Blurring the movement of moving objects in a fat frame gives the brain clues as to what is moving and what is not. The direction of blur gives the direction of motion, and its length indicates speed. In a mysterious way, the brain converts this spatial information – blur – into temporal, and then perceives movement with the help of the apparent phenomenon of movement.
All products recommended by Engadget have been selected by our editorial team, independent of our parent company. Some of our stories involve partnerships. If you purchase something through one of these links, we can earn a commission for affiliates.