Simple Deep Neural Network Code

During my final days at UCR, I developed an artificial neural net code in C++ in order to get a more complete understanding of deep learning. I used that code to make the following animation, which shows a very small artificial neural network learning a function.

The function that the network is learning in this demo is:

The input nodes consist of the first 13 bits of the integer that results from casting to an integer (i.e., int(1000.0f * x)). The output is the single floating point value, . Weights and biases are color-coded; red indicates positive values while blue indicates negative values. The transparency of the nodes and weights correspond to the magnitudes of the weights and biases; higher opacity indicates a larger magnitude. The network was trained using stochastic gradient descent, using the backpropagation algorithm to compute the gradient. Each frame of the video corresponds to one training epoch. All layers except for the output layer used sigmoidal activations. The output layer used a linear activation (which is sort of no activation function at all). For this network, I used the quadratic cost function, which is currently the only cost function that is implemented in the code.

The source code can be found on Bitbucket. It's not highly optimized and it lacks many features, so it's more valuable as a learning tool for studying small networks (up to ~50,000 weights and biases) at the moment. The example in main.cpp should be clear enough to start using the Network class in situations other than the example shown in main.cpp. If I have time, I'll at least modify the Network class so that an arbitrary number of output values can be specified; it is currently limited to a single output value.

To compile the example code, you simply run the provided build script, build.sh, assuming that you have a modern version of g++ installed. If you want to modify the number of input nodes, change the value of N_INPUT in the build script. To run the example program, you simply do:

[bash] $ ./approx-net number_of_nodes_in_first_layer number_of_nodes_in_second_layer ...
[/bash]

For example, to create a network with 7 artificial neurons in the first layer, 5 in the second layer, and 3 in the third layer, you would do this:

[bash] $ ./approx-net 7 5 3
[/bash]

When the training is complete, the network will be saved to a file called graph.dot, which is a Graphviz dot file. You can render this file to an image using the dot command. For example, to render the output of the above example to an SVG image:

[bash] $ dot -Tsvg graph.dot > graph.svg
[/bash]

Here's what that looks like:

Of course, visualization with dot is only practical for very small networks. The nework class should eventually have a way of outputting a JSON file, describing the network, so that the network can be viewed and interactively zoomed with the help of a JavaScript graph visualzation library.