Project · Build micrograd — in your language of choice

Hints

• Start with addition — it's the simplest backward: gradient flows unchanged to both parents.
• Multiplication's backward is where the chain rule shows up most clearly: a.grad += b.data * out.grad. Write this out on paper first.
• Topological sort: process a node only after all its descendants are processed. DFS post-order is the easiest implementation.
• Don't try to add division, exp, or neural-net ops — this project is strictly about + and * to prove the autograd skeleton works.

Expected output

step   0  loss=18.2411  w=0.182  b=0.058
step  40  loss=0.1438  w=1.902  b=2.804
step  80  loss=0.0938  w=1.984  b=2.962
step 120  loss=0.0911  w=1.995  b=2.988
step 160  loss=0.0910  w=1.996  b=2.991
final: w=1.996, b=2.991   (truth: w=2, b=3)

Stretch goals

• Add a tanh op with its backward 1 - tanh^2(x). Use it to fit a non-linear function like y = tanh(x).
• Add power x**n with backward n * x**(n-1) * out.grad.
• Visualize the computation graph: for a 3-op expression, print the DAG as a text tree showing nodes and gradients after backward.
• Benchmark your autograd against PyTorch's for a 100-step fit. Expect to be 1000× slower — that's normal, and tells you what C++ GPU kernels buy you.
• Port your autograd to a second language and confirm the fit converges identically given the same seed.