Autoencoders, Explained

Imagine handing someone a detailed photograph, asking them to describe it in a single tweet, and then demanding they redraw the original from that tweet alone. Do this a few million times, punishing every mistake, and they’ll get eerily good at distilling what actually matters in an image. Congratulations: you’ve just understood the autoencoder, one of deep learning’s most elegant and underrated ideas.

The Shape of the Thing

An autoencoder is a neural network trained to copy its input to its output. That sounds gloriously pointless — why build a network whose job is to return what you gave it? The trick is in the middle. The network is hourglass-shaped: it has three parts.

• The encoder squeezes the input down into a small, dense vector.
• The bottleneck (also called the latent representation) is that small vector — the narrow waist of the hourglass.
• The decoder expands the vector back out, trying to reconstruct the original.

Because the bottleneck is deliberately too small to hold everything, the network can’t cheat by memorising. As IBM puts it, a sufficiently tight bottleneck stops the decoder from simply copying the input, which would technically satisfy the task while learning nothing. Forced to economise, the encoder learns to keep only the signal — the genuinely informative structure — and discard the noise. That compressed code is the prize. The reconstruction is just the exam that proves the network learned to study.

The training loop is refreshingly simple. There are no labels; the input is its own target. You measure reconstruction error (how far the output drifts from the input) and use backpropagation to shrink it. This makes autoencoders a form of self-supervised learning — they manufacture their own homework.

A Tiny One in Keras

Here’s a minimal autoencoder for 784-pixel (28×28) images, the classic MNIST setup:

from tensorflow import keras
from tensorflow.keras import layers

# 784 -> 32 -> 784: a 24x squeeze through the bottleneck
inputs = keras.Input(shape=(784,))
encoded = layers.Dense(32, activation="relu")(inputs)      # encoder + bottleneck
decoded = layers.Dense(784, activation="sigmoid")(encoded) # decoder

autoencoder = keras.Model(inputs, decoded)
autoencoder.compile(optimizer="adam", loss="mse")

# Note: x is both the input AND the target. No labels needed.
autoencoder.fit(x_train, x_train, epochs=20, batch_size=256)

That 32 is the whole story. Widen it and reconstructions get crisper but the network learns less; narrow it and you force harsher, more meaningful compression. Tuning the bottleneck is the central knob of the entire genre.

What They’re Actually Good For

Copying images is a party trick. The useful behaviour falls out sideways.

Dimensionality reduction. That 32-number bottleneck is a compressed feature vector. Think of it as PCA’s more flexible cousin — it can capture curved, nonlinear structure that linear methods flatten out. Feed those codes into a clustering or classification step and you’ve cut your dimensions without throwing away the good stuff.

Anomaly detection. Train an autoencoder only on normal data, and it becomes fluent in normal. Show it something weird — a fraudulent transaction, a defective part on the line, a strange heartbeat — and the reconstruction error spikes because the network has never learned to rebuild that. High reconstruction error becomes your alarm bell. No labelled fraud examples required, which is handy because fraudsters rarely fill in the training set politely.

Denoising. Denoising autoencoders are fed deliberately corrupted inputs — speckled, blurred, half-erased — and trained to reconstruct the clean original. The network learns to tell signal from grit, which is why the same idea shows up in medical-image cleanup and old-photo restoration.

The Plot Twist: VAEs

Vanilla autoencoders learn a messy latent space — points scattered with gaps, so picking a random spot and decoding it usually yields garbage. Variational autoencoders (VAEs) fix this with a clever change. Instead of encoding each input to a single point, a VAE encodes it to a probability distribution — a mean and a standard deviation — and samples from it. That nudges the latent space into something smooth and continuous, where neighbouring points decode into sensibly similar outputs.

The payoff is generation. Sample a fresh point from that well-behaved space, run it through the decoder, and out comes a brand-new face, molecule, or melody that never existed. VAEs were among the first workhorses of generative AI, and the encoder-bottleneck-decoder skeleton still underpins much of today’s generative tooling.

The Takeaway

Reach for an autoencoder when you have a pile of unlabelled data and one of these three needs: compress it (dimensionality reduction), flag the weird stuff (anomaly detection — train on normal, watch the reconstruction error), or clean it up (denoising). Want to generate new samples rather than just reconstruct old ones? Upgrade to a VAE so your latent space is continuous enough to sample from.

Start small: take MNIST, set the bottleneck to 32, and watch what survives the squeeze. The pixels that make it through the hourglass are the ones that mattered all along.