From: veritasium
For centuries, analog computers were considered the most powerful computing devices globally, capable of predicting eclipses and tides, and guiding anti-aircraft guns [00:00:00]. However, the advent of solid-state transistors ushered in the era of digital computers, making them the dominant form of computing today [00:00:09]. Despite this, a combination of factors is currently paving the way for a resurgence of analog technology [00:00:18].
How Analog Computers Work
An analog computer operates without zeros and ones [00:00:57]. Instead, it uses physical phenomena, such as voltages, to represent and solve problems [00:01:03]. By connecting wires in specific ways, it can be programmed to solve a range of differential equations [00:00:27].
For example, a setup can simulate a damped mass oscillating on a spring, where a voltage oscillates exactly like the physical mass [00:00:34]. The electrical circuitry acts as an analog for the physical problem, executing calculations much faster [00:01:10].
Another example is simulating the Lorenz system, a basic model of convection in the atmosphere, known as one of the first discovered examples of chaos [00:01:16]. Parameters can be changed on an analog computer to observe their effects in real-time [00:01:38].
Advantages and Disadvantages
Advantages
Analog computers are powerful computing devices capable of completing many computations quickly and with low power consumption [00:01:46].
- Addition: Adding two currents can be done by simply connecting two wires [00:02:08], whereas a digital computer requires around 50 transistors for two eight-bit numbers [00:02:01].
- Multiplication: Passing a current through a resistor allows the voltage across it (I times R) to effectively multiply two numbers [00:02:23]. A digital computer needs about 1,000 transistors for multiplication [00:02:15].
Disadvantages
Despite their strengths, analog computers have significant drawbacks that led to their decline [00:02:40].
- Single-Purpose: They are not general-purpose devices, meaning they cannot run diverse software like Microsoft Word [00:02:42].
- Inexact and Non-Repeatable: Inputs and outputs are continuous, preventing exact value input and consistent results for repeated calculations [00:02:49]. Manufacturing variations in components (like resistors or capacitors) lead to an approximate 1% error margin [00:03:01].
These limitations—being single-purpose, non-repeatable, and inexact—were the primary reasons analog computers fell out of favor once digital computers became viable [00:03:10].
The Resurgence: AI and Neural Networks
The resurgence of analog computers is closely linked to artificial intelligence (AI) [00:03:43].
Early AI: The Perceptron
The term “AI” was coined in 1956 [00:03:51]. In 1958, Frank Rosenblatt built the perceptron, designed to mimic how neurons fire in the brain [00:03:55].
A basic model of a neuron works as follows:
- An individual neuron either fires (activation of 1) or doesn’t (activation of 0) [00:04:08].
- Inputs come from other neurons, with varying connection strengths represented by weights (positive for excitatory, negative for inhibitory) [00:04:16].
- To determine if a neuron fires, the activation of each input neuron is multiplied by its weight, and these products are summed [00:04:34].
- If the sum exceeds a bias, the neuron fires; otherwise, it doesn’t [00:04:44].
Rosenblatt’s perceptron used 400 photocells (pixels) as input neurons for 20x20-pixel images, with activations between zero and one [00:04:53]. These inputs connected to a single output neuron, each with an adjustable weight [00:05:15]. The process of multiplying activations by weights and summing them is essentially a vector dot product [00:05:25].
The perceptron’s goal was to distinguish between two images, like a rectangle and a circle [00:05:38]. Training involved showing it images and adjusting weights:
- If the output was correct, no weight change occurred [00:06:12].
- If incorrect (e.g., didn’t fire for a circle), input activations were added to weights [00:06:21].
- If incorrect (e.g., fired for a rectangle), input activations were subtracted from weights [00:06:38]. This algorithm was proven to converge if the categories were separable [00:06:52].
The perceptron could distinguish shapes and letters [00:07:02]. Rosenblatt claimed it could tell cats from dogs and exhibit “original thought,” leading to exaggerated media claims of self-reproducing, conscious machines [00:07:09].
In reality, the perceptron’s capabilities were limited; it could not differentiate cats from dogs [00:07:45]. Critiques by Minsky and Papert in 1969 led to the first AI winter, a period of decline for artificial neural networks and AI [00:07:52].
The Rise of Deep Learning
The 1980s saw an AI resurgence, exemplified by Carnegie Mellon’s ALVINN, one of the first self-driving cars steered by an artificial neural network [00:08:25]. ALVINN was an advancement on the perceptron, featuring a hidden layer of artificial neurons between input (30x32-pixel road images) and output layers [00:08:36]. Going from one layer to the next involved matrix multiplication of input activations and weights [00:09:01]. Training involved a human driver providing correct steering angles, with weights adjusted by backpropagation [00:09:15]. As training progressed, the network learned to identify road markings, and its steering angle matched the human driver’s [00:09:46]. The vehicle’s speed (1-2 km/h) was limited by the computer’s matrix multiplication speed [00:10:00].
Despite these advancements, neural networks still struggled with tasks like distinguishing cats and dogs, leading to a second AI lull in the 1990s [00:10:12].
ImageNet and the AI Boom
By the mid-2000s, most AI researchers focused on improving algorithms [00:10:38]. However, Fei-Fei Li posited that more training data was needed [00:10:43]. From 2006 to 2009, she created ImageNet, a database of 1.2 million human-labeled images, the largest of its kind [00:10:52]. From 2010 to 2017, the ImageNet Large Scale Visual Recognition Challenge had software programs compete to detect and classify images into 1,000 categories, including 90 dog breeds [00:11:06].
In 2012, AlexNet, a neural network from the University of Toronto, revolutionized the competition [00:12:12]. It achieved a top-5 error rate of 16.4%, significantly better than previous bests of 28.2% (2010) and 25.8% (2011) [00:11:53]. AlexNet’s success stemmed from its size and depth: eight layers and 500,000 neurons, with 60 million weights and biases adjusted during training [00:12:22]. This required 700 million math operations per image [00:12:40]. The team pioneered the use of Graphics Processing Units (GPUs) for fast parallel computations to handle this computational intensity [00:12:48].
AlexNet’s research paper, cited over 100,000 times, highlighted the importance of network scale [00:13:00]. Following this lead, the ImageNet top-5 error rate plummeted to 3.6% by 2015, surpassing human performance [00:13:19]. This network had 100 layers of neurons [00:13:31].
Why Analog Computers are Making a Comeback for AI
The increasing demand for larger neural networks presents several challenges for traditional digital computers [00:13:35]:
- Energy Consumption: Training a neural network can consume as much electricity as three households annually [00:13:43].
- Von Neumann Bottleneck: Most modern digital computers store data in memory and access it via a bus [00:13:50]. For deep neural networks, most time and energy are spent fetching weight values rather than computing [00:14:00].
- Limitations of Moore’s Law: The doubling of transistors on a chip every two years is slowing as transistor size approaches atomic scale, presenting fundamental physical challenges to further miniaturization [00:14:10].
This “perfect storm” creates an opening for analog computers [00:14:26]. Digital computers are nearing their limits, while neural networks are exploding in popularity [00:14:30]. A core operation for neural networks is matrix multiplication [00:14:35]. Crucially, neural networks do not require the extreme precision of digital computers; a slight variability in components can be tolerated [00:14:41]. This makes analog computing well-suited for AI workloads [00:14:52].
Modern Analog Computing for AI (Mythic AI)
Companies like Mythic AI are developing analog chips to run neural networks [00:14:58]. They repurpose digital flash storage cells, traditionally used to store binary data, as variable resistors [00:15:54].
- By placing a specific number of electrons on each floating gate, they control the resistance of the channel [00:16:33].
- The more electrons, the higher the resistance [00:16:45].
- Applying a small voltage causes current (V/R or Voltage x Conductance) to flow, effectively multiplying two values [00:16:49].
To run a neural network, Mythic AI writes all network weights as the conductance of flash cells [00:17:09]. Activation values are then input as voltage, and the resulting current (activation times weight) is summed as cells are wired together, completing the matrix multiplication [00:17:16].
Mythic AI’s first product can perform 25 trillion math operations per second on a small chip, consuming about three watts of power [00:17:39]. While newer digital systems can do 25-100 trillion operations per second, they are large, expensive systems consuming 50-100 watts [00:17:54]. These analog chips are designed for deploying AI workloads, not training them, and have applications in security cameras, autonomous systems, and manufacturing inspection equipment [00:18:10].
One proposed use for analog circuitry is in smart home speakers, solely to listen for wake words like “Alexa” or “Siri” [00:18:40]. This would use less power and quickly activate the device’s digital circuitry [00:18:47].
A challenge for analog AI is managing signal distortion over many computational layers [00:18:53]. To address this, the signal is converted from analog to digital, sent to the next processing block, and then converted back to analog, preserving the signal [00:19:10].
Conclusion
Historically, Rosenblatt initially used a digital IBM computer for his perceptron but switched to a custom analog computer with variable resistors due to speed limitations [00:19:20]. His idea of neural networks proved correct, and perhaps his belief in analog computing will too [00:19:35].
While it’s uncertain if analog computers will take off as digital ones did, they appear better suited for many current computing tasks [00:19:43]. Though digital has dominated information processing for decades (music, pictures, video), it might be seen as a starting point rather than an endpoint in 100 years [00:20:01]. Human brains exhibit both digital characteristics (neurons firing or not) and analog (thinking occurring everywhere simultaneously) [00:20:17]. Perhaps the power of analog will be essential for achieving true artificial intelligence that thinks like humans [00:20:28].