I like waking up in the dark. It feels like a I have a secret few hours all to myself where no-one knows I’m awake, just laying out the seating. At about 0731 every morning I can hear the squeak of my neighbours pulling up their blinds and from then on the whole orchestra of the city begins to warm up: I can hear the muffled mechanical noises of kettles and toasters being tuned, windows glint as they are swung back and forth to get the pitch of light and heat just right and (somewhere, presumably) the conductor shakes out their arms for another performance.

Hardware Lottery

The lamentable failure and forgetting of the sodium vapour process that I mentioned last week has stuck with me as a metaphor. As a reminder, sodium vapour lighting was a doomed method of post-production that was better than blue or green screening for many decades. The exact colour of the sodium light is easily isolatable in post-production making it easy to matte on special effects. It was most famously used in Mary Poppins (1964). I don’t know if cinema would have been radically different had Disney not held on to it so tightly and instead allowed it to be used, experimented and iterated on. But it’s worth entertaining the idea that the processes, pipelines and even words we take for granted today in visual image production might have been changed had an entirely different process taken root.

‘What if things had been different?’ is a well known sub-genre of speculation but one that’s not as explored as future-oriented work. Sascha Pohflepp’s Golden Institute or Tim Clarke’s High Speed Horizons are both great examples of where a different social or political decision results in different hardware. In Pohflepp’s case Jimmy Carter wins the 1980 US presidential election and begins an agenda of innovation of environmental technologies. In Clarke’s alternative present, commercial flight hasn’t evolved from military-industrial innovations but from other places. Phillip Ronnenburg’s Post-Cyberwar Series, another great project, explores an Internet over TV radio waves and a GPS system based on seismic sensors.

Often what is total serendipity is post-fact chalked up to some sort of inevitability. The story of sodium vapour lighting isn’t told any more because it didn’t take hold, but for decades it was the easiest and most accurate matte process. So instead, in stories about plucky underdogs like Industrial Light and Magic, blue screening is narrated as a sort of inevitable innovation that we now take for granted everywhere while at the time it was a niche and unlikely successor. Katrin Fritsch has written about this tendency to post-rationalise serendipitous advances as inevitable or part of a myth of progress in machine learning research. In her interpretation it’s a way of blending the often unachievable hype with technical reality – a sort of ‘yeah, that was what I intended all along‘ – for catching the falling glass of innovation at the last minute.

This idea of chance and rationalisation came up a lot this week. A morbid conversation on opportunistic technical relations with Mrs Revell’s elder and younger the other day turned to suicide and we discussed about how the change from coal gas to natural gas in the late sixties in Britain led to a sudden drop in suicide rates: Coal gas, once used in domestic ovens and heating, is significantly more lethal than natural gas because of its large carbon monoxide quantity. Researchers exploring the almost halving of the suicide rate (from 5714 to 3693 a year between 1963 and 1975) concluded that ‘means reduction saves lives.’ In other words, opportunity was as much a driver as determination. Mrs Revell the younger cemented this idea with the Dorothy Parker poem Resumé: 

Razors pain you;
Rivers are damp;
Acids stain you;
And drugs cause cramp.
Guns aren’t lawful;
Nooses give;
Gas smells awful;
You might as well live.

Dorothy Parker – Resumé

All of this wraps up nicely with a paper I found from Sara Hooker (I can’t remember where I saw it, sorry) called the Hardware Lottery which examines the outsize role that hardware plays in artificial intelligence research. Hooker proposes that the serendipitous nature of working with whatever is easily to hand is as much responsible for advances in AI as anything else. Importantly for Hooker’s argument, research directions are not pursued because they are superior or more promising but because they are more technically feasible with the tools to hand.

For example, she writes of how the Graphics Processing Unit (GPU), a niche tool developed for games and 3D graphics just happened to conform to the specifications of computation that AI researchers had been clamouring for. Machine learning in the late 20th century was a small niche within the niche of AI research but the serendipitous arrival of the GPU catapulted this subfield to the top of the research agenda. Until that point, researchers were chaining Central Processing Units (CPUs) to get the level of parallel processing needed. Over the coming decades, the GPU increasingly took the lead in innovation. Hooker describes how in 2012 it took 16000 CPUs to do what four GPUs could do in 2013.

Of course the advances in machine learning aided by GPUs had a similarly serendipitous affect on GPU manufacturers. NVIDIA, the prime mover in this space, serving games (the world’s largest media industry), crypto-currencies (a hip new fad the teenagers are into) and AI research dominates commercial computation, particularly with its recent acquisition of ARM even if Wikipedia still refers to them as a ‘video game company.’ This total domination has seen journalists call for the replacement of Moore’s Law – the notion named for Gordon Moore, founder of Fairchild Semiconductor that the number of transistors on a chip would double every two years – with Huang’s Law (named after NVIDIA CEO Jensen Huang). In Huang’s Law, the power of GPUs double every two years, although there’s more nuance to it than that.

I want to highlight that Hooker’s paper also acknowledges that though these advances are exciting, they are running into limits. The software architectures designed for GPUs are still incredibly energy inefficient and expensive compared to the human brain which ‘runs on the equivalent of an electric shaver.’ Hooker suggests that though technical advances are heavily influenced by the convenience of available hardware, when this hardware landscape is too homogenous it prohibits new serendipitous advances: The biggest discussion out there is machine learning, GPUs and AI, what else are we precluding? Are we hitting the limits of what machine learning can actually usefully do? Huang’s Law can continue to accelerate GPU power but what if there are radical and untouched methods to be explored similar to the position machine learning was in 30 years ago. GPT-3, which has people all in a tizz cost 12 million dollars just to train and it’s still racist and a bit rubbish. We’ve had semi-convincing nonsense generative text machines since the Oulipo and it doens’t really do more than that. As Dan Hon twittered, GPT-3 is “…like my kids running to me and saying LOOK AT ALL THE STICKS WE FOUND and carefully telling me about how each one is interesting and… they’re not wrong.”

Short Stuff

• About the only time I ever credit Elon Musk with something is on his insistence that people not use acronyms – that they exclude people from conversations. Returning to work and even I’m struggling to remember all the acronyms a large organisation develops. They’re counter-revolutionary. There must be some sort of law that governs the ration of acronyms to the scale of an organisation.
• I have a new job advert out, let me know if you want to talk about it. It’s a really exciting time for this course so think about it.
• Semisopochnoi Island, off the Alaskan island chain is both the most eastern point in the United States and one of its most western points. Here’s an XKCD as a clue.
• I was one of like three people to watch this talk from Sheldon Brown live at MAAT the other week which is lucky because it was a stage-by-stage deconstruction of what my PhD research is about.

That was a really short short stuff to make up for the longer short stuffs of recent weeks. I am now out of Content. Love you, speak to you next week.