Bit Video

In this experiment I provided a cursor's eye view of the live screen using a purpose built system that recorded mouse motion and then rendered frames in non-realtime.

I was frustrated that Smalltalk in the '80s didn't have the horsepower to generate live video. I discovered it could bitblt smallish bitmaps at 15Hz. So I set out to generate interesting movies in this format.

I sampled the mouse position and button state at 15Hz. I set up some simple means to start and stop recording.

I operated the browser while recording. Scroll bars moved, selections highlighted, menus popped, windows opened.

I computed camera orientation from samples as if it were an airplane flying over the terrain. I pointed the way would it have to point to get to the next sample.

I computed camera elevation from the distance between samples. The faster the mouse was moving, the higher the point of view.

I wrote a double-nested loop to render each frame. I computed where on the real display I should find each pixel of the display.

I ran the script recreating all of the mouse moves so that the display showed exactly the same terrain as I had seen when recording.

The recording must have taken hours to produce seconds. I don't actually remember. The computer was in my home and had no other application.

I displayed the results with an appropriately timed loop.

The visual effect was stunning. I did appear to be flying as I imagined the mouse flew from click to click. The terrain appeared to be a Smalltalk screen stretched out like a table top.

When the mouse moved fast, the resulting elevation allowed me to see into the blackness beyond the screen's bounds.

When the mouse moved slowly, I could see individual pixels become blocks. Individual menu items would swell to fill the screen.

The dynamics were perfect. The recorded ballistics of my hand motion transfered to believable motion. This was the premise of my 2d work Animation with Captured Gestures.

I stored the frames in Smalltalk's object memory. Somehow that version got misplaced or broken. I showed the video multiple times over a few days and then no more.

I tried to repeat the experiment. I found that my second attempts had none of the grace and flair of my original version. I think this is because I thought too much about what I was doing while I recorded samples.

I tried to repeat this work only once. Having to wait many hours to see disappointing results led me to put off a third try indefinitely.

I suspect this can be done realtime now with the graphic resources available on modern computers. However the mouse recording and especially the slow playback has never been an objective of modern window manager designs. I had an opportunity available only when all parts were integrated and intentionally simple.

Smooth and Efficient Zooming and Panning by Jarke J. van Wijk Wim A.A. Nuij Technische Universiteit Eindhoven

Interpolate-Zoom d3 plugin demonstrated.