Deepity 发布手写 C++/CUDA 预测编码库，MNIST 上超过 jpc 与 ngc-learn

I've been building a predictive coding (PC) engine from scratch in C++20, with a thin Python layer on top, and just cut the first public release. Posting here for feedback and because I haven't seen many from-scratch PC implementations with real kernel-level optimization work behind them.

What predictive coding is: instead of backprop's single global error signal flowing backward through a stored computation graph, each layer in a PCN holds a local belief about its own state and compares it against a top-down prediction from the layer above. Training alternates between "settling" (every layer locally reduces its own prediction error over several steps) and a weight update that only looks at that layer's own error. No layer needs the transpose of another layer's weights, and there's no backward graph to keep in memory.

What Deepity actually is:

- A C++20 core with hand-written AVX2/AVX-512 SIMD kernels (via SLEEF for the transcendental activations), OpenBLAS/MKL for GEMM, and a CUDA backend (cuBLAS + CUTLASS + CUDA graph capture for the settling loop) that's still being tuned for large-scale training (not available yet).

- Python bindings via nanobind. pip install pydeepity, import deepity, define a network the same way you would in PyTorch/Keras.

- Several PCN variants: a plain classic-settling network, Direct Kolen-Pollack feedback (far fewer settling steps, some accuracy cost), and a "kitchen sink" network with µPC scaling, residual connections, incremental PC, error-based PC and momentum settling all available as independent toggles, off by default.

The headline benchmark (matched config across all three PC implementations: 784-512-512-10 sigmoid network, T=20 settling steps, batch 250, 100 epochs, AdamW):

Deepity: 98.21% in 487.5s

jpc (JAX): 97.55% in 531.4s

ngc-learn: 98% in 1707.7s

Testing was done on a single machine (12th Gen i7-12700H laptop CPU), reproducible from the repo, with raw logs in the repository. A plain PyTorch backprop run on the same architecture gets 97.51% in 627.8s included as a reference point (not a fair comparison, since it's one pass per batch where PC does 20 settling passes).

Kernel-level numbers, if you care about where the time actually goes: a Google Benchmark harness measuring a full PC train step sustains ~690 GFLOPS/s, and the same FLOP accounting applied to the real end-to-end MNIST run above works out to ~675 GFLOPS/s.

What's not done yet, to be upfront about it: this is alpha software (0.1.0a3), solo-maintained so far. The CPU path is the mature, benchmarked one. The CUDA backend builds and runs, but large-scale ImageNet training on it is still being optimized, and the convolution path is currently im2col+cuBLAS rather than cuDNN (TF32 is on, but it still trails cuDNN's specialized small-kernel conv algorithms). No CUDA-vs-CPU numbers on purpose, as I don't want to publish a number I haven't actually validated.

Install is deliberately light, just numpy and rich, no bundled ML framework:

pip install pydeepity

from deepity import SimplePCN, Linear, TanH import numpy as np net = SimplePCN(Linear(2, 8), TanH(), Linear(8, 1), batch_size=1) net.configure(learning_rate=0.05, inference_rate=0.3, lmbda=0.0001, optimizer="SGD") x = np.array([1.0,-1.0], dtype=np.float32) y = np.array([1.0], dtype=np.float32) energy = net.train_step(x, y, steps=50) # returns total pred. error 

Repo (MIT licensed): https://github.com/Deepity-HQ/Deepity

PyPI: https://pypi.org/project/pydeepity

Happy to answer questions about the algorithm, the kernel fusion work, or why PC is or isn't worth your time compared to backprop. Looking for contributors too, especially anyone with C++/CUDA performance or build-systems/DevOps background.

submitted by /u/Important-Home4431
[link] [留言]