TL;DR
Combine statistical outlier removal (Mahalanobis distance) with autoencoder reconstruction to create anomaly detection systems. Train with clean normal data, score with MSE, explain with per-feature attribution. Download notebook
Most anomaly detection systems fail—and surprisingly, model choice is rarely the culprit.
The most common problems are:
- Learning from contaminated “normal” data
- Producing anomaly scores without a clear explanation
- Diluting multiple failure signals into a single score
- Difficulty adapting to regime changes or concept drift
This post outlines a minimal, end-to-end anomaly detection pattern that addresses those issues using a combination of statistics and autoencoder reconstruction. The emphasis is not on production complexity, but on clarity and practical guidance.
The goal is to present a reusable pattern that mirrors how production systems work conceptually, just at smaller scale for learning purposes.
Why Are Autoencoders Used As Anomaly Detectors?
Purely statistical methods (like z-scores or Mahalanobis distance) struggle with complex multivariate patterns. Purely learned models lack interpretability and can be fragile. The sweet spot lies in between.
In real systems, anomaly detection benefits from combining the following:
- A clear definition of “normal”
- A learned model of that normal structure
- A simple and consistent anomaly score
- A method for explaining why something looks anomalous
This pattern separates these concerns instead of forcing a single model to do everything.
Conceptual Overview
At a high level, the pipeline consists of four stages:
- Define ‘normal’ statistically
- Learn the structure of normal
- Score deviations from that structure
- Attribute anomalies back to features
Interpretable Autoencoder-Based Anomaly Detection Pipeline
How Autoencoders Work and Can Be Used
Autoencoders are frequently described as anomaly detectors, but that is not their originally intended use.
An autoencoder does one thing:
> It learns a compressed representation of normal data and reconstructs it.
The anomaly signal does not come from the model itself. It comes from reconstruction error when the input no longer matches the learned structure.
💡 Key Insight: The anomaly signal comes from reconstruction error, not the autoencoder itself. The model learns normal patterns; anomalies reveal themselves by failing to reconstruct.
In this pattern, the autoencoder is used specifically to:
- Capture relationships between features
- Learn correlated behavior
- Provide a reconstruction-based error signal
It is not a classifier.
When to Use This Pattern
Good fit for:
- Multivariate time series (stock prices, sensor readings)
- Financial ratios and portfolio metrics
- Correlated technical indicators
- System health metrics with multiple dimensions
Not appropriate for:
- Single-feature outlier detection (use z-scores instead)
- Categorical data (use different methods)
- Data with fewer than 3 correlated features
- Cases requiring real-time (<100ms) responses
The common thread: multivariate signals where relationships matter more than individual values.
Quick Preview
Here’s the core pattern in ~10 lines of code:
from sklearn.covariance import MinCovDet
import numpy as np
# 1. Define normal (remove outliers)
mcd = MinCovDet().fit(X_train)
X_clean = X_train[mcd.mahalanobis(X_train) 0.95 on well-structured synthetic data
Everything beyond this in production is primarily about **scale, stability, and governance**.
---
## Key Formulas Reference
**Mean Squared Error (MSE):**
MSE = (1/n_features) × Σ(xᵢ – x̂ᵢ)²
**Feature Attribution:**
Feature_percentage = 100 × (xⱼ – x̂ⱼ)² / Σ(xₖ – x̂ₖ)²
**Anomaly Detection:**
is_anomaly = (MSE > threshold) where threshold = percentile(training_MSE, 99)
**Mahalanobis Distance:**
D² = (x – μ)ᵀ Σ⁻¹ (x – μ) where Σ is robust covariance estimate
## Next Steps
### 1. Try It Yourself
[Download the notebook](https://github.com/timpinard/portfolio-anomaly-detection/blob/main/notebooks/autoencoder_anomaly_detection_starter.ipynb) and run it locally. The synthetic data generation makes it easy to experiment.
### 2. Adapt to Your Data
Replace synthetic data with your own multivariate time series. Start with 3-5 correlated features.
### 3. Experiment with Architecture
Try different encoder/decoder sizes. Deeper networks work better with more features.
### 4. Add Temporal Patterns
Extend to LSTM autoencoders for sequence modeling. Use sliding windows of 30-60 time steps.
### 5. Share Your Results
Comment below with your use case and results. What worked? What didn't?
---
## Additional Resources
**Further Reading:**
- Deep Learning for Anomaly Detection (Ian Goodfellow)
- Robust Statistics and Outlier Detection (Rousseeuw & Leroy)
---
*Have questions or suggestions? Leave a comment below or reach out on [LinkedIn].*
*Found this helpful? Share it with your team.*
Tim Pinard 