Deep Neural Networks: A Getting Started Tutorial, Part #2

Deep Neural Networks: A Getting Started Tutorial, Part #2

30 June 2014, 12:28
Sergey Golubev
0
867
Overall Program Structure

The overall structure of the demo program, with a few minor edits to save space, is presented in Listing 1. To create the demo, I launched Visual Studio and created a new project named DeepNeuralNetwork. The demo has no significant Microsoft .NET Framework version dependencies, so any relatively recent version of Visual Studio should work. After the template-generated code loaded into the editor, I removed all using statements except the one that references the top-level System namespace. In the Solution Explorer window I renamed the file Program.cs to the slightly more descriptive DeepNetProgram and Visual Studio automatically renamed class Program for me.

Listing 1: Overall Demo Program Structure

using System;
namespace DeepNeuralNetwork
{
  class DeepNetProgram
  {
    static void Main(string[] args)
    {
      Console.WriteLine("Begin Deep Neural Network demo");

      Console.WriteLine("Creating a 3-4-5-2 network");
      int numInput = 3;
      int numHiddenA = 4;
      int numHiddenB = 5;
      int numOutput = 2;

      DeepNeuralNetwork dnn =
        new DeepNeuralNetwork(numInput,
          numHiddenA, numHiddenB, numOutput);

      double[] weights = new double[] {
        0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10,
        0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.20,
        0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28, 0.29, 0.30,
        0.31, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37, 0.38, 0.39, 0.40,
        0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.50,
        0.51, 0.52, 0.53 };

      dnn.SetWeights(weights);

      double[] xValues = new double[] { 1.0, 2.0, 3.0 };

      Console.WriteLine("Dummy weights and bias values are:");
      ShowVector(weights, 10, 2, true);

      Console.WriteLine("Dummy inputs are:");
      ShowVector(xValues, 3, 1, true);

      double[] yValues = dnn.ComputeOutputs(xValues);

      Console.WriteLine("Computed outputs are:");
      ShowVector(yValues, 2, 4, true);


      Console.WriteLine("End deep neural network demo");
      Console.ReadLine();
    }

    static public void ShowVector(double[] vector, int valsPerRow,
      int decimals, bool newLine)
    {
      for (int i = 0; i < vector.Length; ++i)
      {
        if (i % valsPerRow == 0) Console.WriteLine("");
        Console.Write(vector[i].ToString("F" + decimals) + " ");
      }
      if (newLine == true) Console.WriteLine("");
    }
  } // Program


  public class DeepNeuralNetwork { . . }
  
}

The program class consists of the Main entry point method and a ShowVector helper method. The deep neural network is encapsulated in a program-defined class named DeepNeuralNetwork. The Main method instantiates a 3-4-5-2 fully connected feed-forward neural network and assigns 53 dummy values for the network's weights and bias values using method SetWeights. After dummy inputs of 1.0, 2.0 and 3.0 are set up in array xValues, those inputs are fed to the network via method ComputeOutputs, which returns the outputs into array yValues. Notice that the demo illustrates only the deep neural network feed-forward mechanism, and doesn't perform any training.

The Deep Neural Network Class

The structure of the deep neural network class is presented in Listing 2. The network is hard-coded for two hidden layers. Neural networks with three or more hidden layers are rare, but can be easily created using the design pattern in this article. A challenge when working with deep neural networks is keeping the names of the many weights, biases, inputs and outputs straight. The input-to-layer-A weights are stored in matrix iaWeights, the layer-A-to-layer-B weights are stored in matrix abWeights, and the layer-B-to-output weights are stored in matrix boWeights.

Listing 2: Deep Neural Network Class Structure

public class DeepNeuralNetwork
{
  private int numInput;
  private int numHiddenA;
  private int numHiddenB;
  private int numOutput;

  private double[] inputs;

  private double[][] iaWeights;
  private double[][] abWeights;
  private double[][] boWeights;

  private double[] aBiases;
  private double[] bBiases;
  private double[] oBiases;

  private double[] aOutputs;
  private double[] bOutputs;
  private double[] outputs;

  private static Random rnd;

  public DeepNeuralNetwork(int numInput, int numHiddenA,
    int numHiddenB, int numOutput) { . . }
  private static double[][] MakeMatrix(int rows, int cols) { . . }
  private void InitializeWeights() { . . }
  public void SetWeights(double[] weights) { . . }
  public double[] ComputeOutputs(double[] xValues) { . . }
  private static double HyperTanFunction(double x) { . . }
  private static double[] Softmax(double[] oSums) { . . }
}

The two hidden layers and the single output layer each have an array of associated bias values, named aBiases, bBiases, and oBiases respectively. The local outputs for the hidden layers are stored in class-scope arrays named aOutputs and bOutputs. These two arrays could have been defined locally to the ComputeOutputs method.

Most forms of neural networks use some type of randomization during initialization and training. Static class member rnd is used by the demo network to initialize the weights and bias values. The class exposes three public methods: a constructor, method SetWeights and method ComputeOutputs. Private methods MakeMatrix and InitializeWeights are helpers used by the constructor. Private methods HyperTanFunction and Softmax are the activation functions used by method ComputeOutputs.


Share it with friends: