Brief #19 · Machine Learning
Neural Network from Scratch
Easy⏱ 10-14 hSoon
Write a neural network with nothing but NumPy: forward pass, backpropagation, gradient checking and training, then rebuild it in PyTorch and compare.
Outlines are free · building with tracking needs Pro
NThe briefFrom the team
Our new ML hires can call model.fit() but freeze when a loss suddenly turns into NaN or stops going down. Before they touch production models, we want each of them to build one network by hand and understand every number inside it. This is that exercise, done properly.
What you'll build
- 1Load MNIST, normalise it, and set aside a validation split.
- 2Implement a two-layer network in NumPy: linear layers, ReLU, softmax and cross-entropy loss.
- 3Derive and code backpropagation, and verify every gradient against a numerical gradient check.
- 4Train with mini-batch SGD, plot loss and accuracy, and break it on purpose: a learning rate too high, bad weight initialisation.
- 5Rebuild the same network in PyTorch and check that both give the same results with the same seed.
Step-by-step walkthrough
Walkthrough coming soon
This project's step-by-step walkthrough, with real outputs at every step, is still being written. You can read the brief and try it on your own meanwhile; tracking opens when the walkthrough is ready.
Done when
- Gradient check passes for every parameter, with relative error under 1e-6.
- The NumPy network reaches at least 97% test accuracy.
- Plots showing what a too-high learning rate and bad initialisation do to the loss.
- A short write-up explaining backpropagation for one weight, in your own words.
Stretch goals
- Add momentum and Adam, and compare how fast each one trains.
- Add dropout and show its effect on the gap between training and validation accuracy.