Toy Code - My First Neural Network

As part of my annual tradition of avoiding filing taxes by starting new side projects, I decided to write my own neural network in C code. A quick Web search turned up several beautiful neural-network libraries written in C. But I wanted something more like a worked-example than a full-fledged library. Github turned up tinn - the Tiny Neural Network (library). Tinn is pretty spiffy and a great tutorial for working out the mechanics of a simple feed-foward neural net. While it could be re-purposed to work on other datasets, it ships with the Semeion dataset.


As spiffy as tinn is, I still needed to rewrite it "in my own C" in order to be sure I understood what is happening internally. So, I wrote basic_nn (Github).


I like to include a general-purpose test harness in my one-off C projects that acts like a built-in debugger. This is the dev_prompt() function. Consult the README on Github for instructions on using the command codes at the prompt.


I had some trouble getting the network to train properly, at first, and this turned out to be an issue with not mapping the weights correctly during the forward and backward propagation phases. The update rules are very straightforward and this dataset is very small, so once I got the array indices correct (getting the right weights mapped to the right parts of the update equations), everything just snapped into place. I was able to use the debug prompt to view the internal weights and get a visual idea of what is happening internally. The mathematics of neural nets are copiously documented throughout the web so I won’t recapitulate them, here. I will try to show diagrammatically what is happening inside basic_nn, however.

When the NN is not training correctly, the entire weight space will tend to change like random static during training; or, if the array indices get transposed (oops, yes, I did that), then you can see “bands” appear in the hidden weight space when training many iterations, especially on a small training set. The image below shows the hidden states (256x32) with the erroneous bands exaggerated so they show up better visually:

But when the NN is training properly, you will see an initial change across the entire weight space and then the weights will tend to “settle”, with most weights staying the same and only a few scattered weights changing with additional training iterations. The image below shows the hidden states. The first box shows the hidden states after 20 epochs of training (one epoch passes through the entire dataset once; note that basic_nn kind of cheats in order to avoid the need to shuffle the dataset, see command-code 8 in the source-code). The second box shows the hidden states after another 10 epochs. Because you can’t visually see the difference between them, the third box shows both snapshots of the hidden state overlaid and visually subtracted so you can see the points where the two snapshots differ:

So, that’s how to write and train your own neural net in C with the Semeion dataset. Stay tuned for more episodes of Toy Code.