Mean-field Boltzmann machines

Denoise a binary image with one belief: a pixel probably looks like its neighbors.

Write that as an energy, read the energy as a probability, and solve it with the cheapest approximate inference there is — mean field. No training set, no labels.

The tug of war

Every pixel is a spin $x_i \in \{-1, +1\}$ : ink or background.
Two forces on each pixel:
- agree with your neighbors (smoothness prior, weight $J$ )
- stay loyal to the noisy pixel you observed (data field, weight $h$ )
A lone flipped pixel feels 4 votes to switch and 1 to stay — the noise gets outvoted.
The balance point of the two forces is a clean image.

The model

Ising / Boltzmann energy, then a Boltzmann distribution over configurations:

E(x) = -J \sum_{(i,j)} x_i x_j \; - \; h \sum_i x_i y_i

p(x) = \frac{1}{Z} \exp\!\big(-E(x)\big)

$Z$ sums over $2^{576}$ configurations of a 24×24 grid — intractable. So we don't compute it.

Mean field

Approximate the posterior with independent per-pixel coins, one mean $\mu_i$ each, and update each by the field pressing on it:

\mu_i \leftarrow \tanh\!\Big( J \sum_{j \in \mathcal{N}(i)} \mu_j \; + \; h\, y_i \Big)

Sweep it one pixel at a time to convergence. That minimizes the free energy:

F(\mu) = \mathbb{E}_q[E(x)] - H(q) = -\log Z + \mathrm{KL}(q \,\|\, p)

Driving $F$ down pushes the approximation toward the true posterior.

Watch it denoise

Top: the current mean-field estimate $\mu$ , sweep by sweep. Bottom: the free energy, sliding downhill and never rising.

The guarantee

Sequential (Gauss-Seidel) sweeps are coordinate descent on $F$ — so it only ever goes down. Cousin to EM's monotone climb, pointing the other way.

−1009.6 → −1351.7, converged in 13 sweeps.

Scratch vs. baseline

18% salt-and-pepper noise on a 24×24 "3". No sklearn twin — the honest baseline is a 3×3 median filter (majority vote). BernoulliRBM is a cousin, not a substitute (it learns its couplings).

Noisy: 83.3% accuracy, 7.78 dB PSNR (96 wrong pixels)
Median filter: 92.0%, 10.98 dB (46 wrong)
Mean-field MRF: 96.7%, 14.82 dB (19 wrong)

Iteration over the whole grid beats a one-shot local vote by ~4 dB.

Takeaways

Energy-based model + mean field: local pairwise prior + noisy evidence, cheaply.
It's an approximation — assumes pixels independent, so it over-smooths fine detail.
Coordinate descent to a local optimum, like EM — monotone, not global.
This lineage is everywhere now: CRFs, dense-CRF segmentation, VAEs, diffusion models.