The overall structure of the demo program, with a few minor edits to save space, is presented in Listing 1.
To create the demo, I launched Visual Studio and created a new project
named DeepNeuralNetwork. The demo has no significant Microsoft .NET
Framework version dependencies, so any relatively recent version of
Visual Studio should work. After the template-generated code loaded into
the editor, I removed all using statements except the one that
references the top-level System namespace. In the Solution Explorer
window I renamed the file Program.cs to the slightly more descriptive
DeepNetProgram and Visual Studio automatically renamed class Program for
me.
Listing 1: Overall Demo Program Structure
using System; namespace DeepNeuralNetwork { class DeepNetProgram { static void Main(string[] args) { Console.WriteLine("Begin Deep Neural Network demo"); Console.WriteLine("Creating a 3-4-5-2 network"); int numInput = 3; int numHiddenA = 4; int numHiddenB = 5; int numOutput = 2; DeepNeuralNetwork dnn = new DeepNeuralNetwork(numInput, numHiddenA, numHiddenB, numOutput); double[] weights = new double[] { 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.30, 0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.40, 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.50, 0.51, 0.52, 0.53 }; dnn.SetWeights(weights); double[] xValues = new double[] { 1.0, 2.0, 3.0 }; Console.WriteLine("Dummy weights and bias values are:"); ShowVector(weights, 10, 2, true); Console.WriteLine("Dummy inputs are:"); ShowVector(xValues, 3, 1, true); double[] yValues = dnn.ComputeOutputs(xValues); Console.WriteLine("Computed outputs are:"); ShowVector(yValues, 2, 4, true); Console.WriteLine("End deep neural network demo"); Console.ReadLine(); } static public void ShowVector(double[] vector, int valsPerRow, int decimals, bool newLine) { for (int i = 0; i < vector.Length; ++i) { if (i % valsPerRow == 0) Console.WriteLine(""); Console.Write(vector[i].ToString("F" + decimals) + " "); } if (newLine == true) Console.WriteLine(""); } } // Program public class DeepNeuralNetwork { . . } }
The program class consists of the Main entry point method and a ShowVector helper method. The deep neural network is encapsulated in a program-defined class named DeepNeuralNetwork. The Main method instantiates a 3-4-5-2 fully connected feed-forward neural network and assigns 53 dummy values for the network's weights and bias values using method SetWeights. After dummy inputs of 1.0, 2.0 and 3.0 are set up in array xValues, those inputs are fed to the network via method ComputeOutputs, which returns the outputs into array yValues. Notice that the demo illustrates only the deep neural network feed-forward mechanism, and doesn't perform any training.
The Deep Neural Network Class
The structure of the deep neural network class is presented in Listing 2.
The network is hard-coded for two hidden layers. Neural networks with
three or more hidden layers are rare, but can be easily created using
the design pattern in this article. A challenge when working with deep
neural networks is keeping the names of the many weights, biases, inputs
and outputs straight. The input-to-layer-A weights are stored in matrix
iaWeights, the layer-A-to-layer-B weights are stored in matrix
abWeights, and the layer-B-to-output weights are stored in matrix
boWeights.
Listing 2: Deep Neural Network Class Structure
public class DeepNeuralNetwork { private int numInput; private int numHiddenA; private int numHiddenB; private int numOutput; private double[] inputs; private double[][] iaWeights; private double[][] abWeights; private double[][] boWeights; private double[] aBiases; private double[] bBiases; private double[] oBiases; private double[] aOutputs; private double[] bOutputs; private double[] outputs; private static Random rnd; public DeepNeuralNetwork(int numInput, int numHiddenA, int numHiddenB, int numOutput) { . . } private static double[][] MakeMatrix(int rows, int cols) { . . } private void InitializeWeights() { . . } public void SetWeights(double[] weights) { . . } public double[] ComputeOutputs(double[] xValues) { . . } private static double HyperTanFunction(double x) { . . } private static double[] Softmax(double[] oSums) { . . } }
The two hidden layers and the single output layer each have an array of associated bias values, named aBiases, bBiases, and oBiases respectively. The local outputs for the hidden layers are stored in class-scope arrays named aOutputs and bOutputs. These two arrays could have been defined locally to the ComputeOutputs method.
Most forms of neural networks use some type of randomization during initialization and training. Static class member rnd is used by the demo network to initialize the weights and bias values. The class exposes three public methods: a constructor, method SetWeights and method ComputeOutputs. Private methods MakeMatrix and InitializeWeights are helpers used by the constructor. Private methods HyperTanFunction and Softmax are the activation functions used by method ComputeOutputs.