Introduction To Neural Networks – Part 1: The Neuron

Neural Networks is an important subject in artificial intelligence to handle things in the way our brain does. Besides its importance in artificial intelligence, it provides a base for biological modelling of the  human (or animal) brain organization. Though it’s not the final conclusion about how the brain works, the model that neural networks suggest is a reasonable, practical, functional and working reduction of the brain.

The brain is a complex and huge connection of nerve cells called neuron, each neuron collects information (signals) from other neurons with dendrites, and after a simple calculation inside its soma it generates its own information (signal) and distributes it to other neurons through its axon. So from this sentence you can easily understand that the axon terminals connects with other dendrites. This connection is called a synapse.

The neuron as it is in the brain – Source : Carlson NA (1992) Foundations of Physiological Psychology. Needham Heights, Massachusetts: Simon & Schuster.

A synapse is the gate to transfer electro-chemical material from one cell to the other. This material can either downgrade (-) the soma load or upgrade (+) it. The synapses vary from each other by their effect which is called in AI terminology the weight. So each input signal enters to the soma is summed after it is multiplied by its weight. Now the cell has a load in itself. If this load is more than the cell can keep (greater than its threshold), a signal (1) will be generated through the axon independent of how much the load is greater than the threshold.

So if we want to simulate this neuron, we will just use this simple math. Take the inputs, multiply each of them with its weight, sum up all, check if the total is bigger than the threshold or not. If yes fire the signal (1), if No keep silence (0). It is simple as illustrated in below (Figure 1), but man could make the history real with this simple math in his brain.

Figure 1: Simple Math inside a Neuron

To make the simulation I will code an Object Pascal class in Delphi (Firemonkey) and then make a simple application to test it. Note that a single neuron is not so useful for doing complex things, for making real AI, you should connect many neurons each other and arrange the weights and thresholds. A code model for connecting neurons will be also published in this blog as Part 2.

Our class name is TNeuron. It has got an array property of type Single for inputs, an array property of type Single for weights, a Single property for threshold, a Single property for output. Note that even the input and output property may be either 0 or 1 (since it represents a digital signal), we will make their types as single (not Byte, not Bool or not Integer) to be able to use analog signals in future posts.

TNeuron = class(TObject)
private
  FThreshold: Single;
  FInputCount: Integer;
  FInputs:TSingleDynArray;
  FWeights:TSingleDynArray;
  FAnalogOutput:Boolean;
  function GetInput(Index: Integer): Single;
  function GetOutput: Single;
  function GetWeight(Index: Integer): Single;
  procedure SetInput(Index: Integer; const Value: Single);
  procedure SetWeight(Index: Integer; const Value: Single);
  procedure SetInputCount(const Value: Integer);
  function GetLoad: Single;
public
  Constructor Create(aInputCount:Integer);
  procedure RandomInitialize;
  Property Load:Single read GetLoad;
  property Output:Single read GetOutput;
  property Threshold:Single read FThreshold write FThreshold;
  property Inputs[Index: Integer]: Single read GetInput write SetInput;
  property Weights[Index: Integer]: Single read GetWeight write SetWeight;
  property InputCount:Integer read FInputCount write SetInputCount;
  property AnalogOutput read FAnalogOutput write FAnalogOutput;
end;

The readonly Output property will be calculated each time it is called in GetOuput, so this function is the main execution of the neuron process in which it is decided if the neuron will fire or not. Normally the inputs will be provided from outside to the class, the source may be either the date to be recognized or other neuron signals. We will see the signal transmission in the next part of the subject. In this article we will focus on the single cell and its simple execution logic. The InputCount represents the receiving synapse count of the cell which is done by connecting the dendrites to other neurons’ axon terminals. So this count must be a variable which ve can arrange later according to our model. Each input is evaluated with its weight, so we will have a weight array which is equal to inputs array in size. In the constructor of the class, the dynamic arrays are created through the InputCount property setter and they are populated with initial random values.

constructor TNeuron.Create(aInputCount:Integer);
begin
  inherited Create;
  FAnalogOutput := False;
  InputCount := aInputCount;
  RandomInitialize;
end;

procedure TNeuron.SetInputCount(const Value: Integer);
begin
  FInputCount := Value;
  SetLength(FInputs,Value);
  SetLength(FWeights,Value);
end;

procedure TNeuron.RandomInitialize;
var i:Integer;
begin
  Randomize;
  for i := 0 to FInputCount-1 do
  begin
    FInputs[i] := RandomRange(0,1);
    FWeights[i] := Random*RandomFrom([-1,1]);
  end;
  FThreshold := Random;
end;

To be able to calculate the output value of the Neuron we will use the logic explained in above Figure 1. First we will calculate the cell load with the below summation method and after calculating the cell load, we will fire the signal if the load is greater than the inside threshold. Otherwise the result will be 0, which means no signal is fired.

function TNeuron.GetLoad: Single;
var i: Integer;
begin
  Result := 0;
  for i := 0 to FInputCount-1 do
  begin
    Result := Result+ FInputs[i]*FWeights[i];
  end;
end;

function TNeuron.GetOutput: Single;
var i: Integer;
begin
  Result := 0;
  if Load > FThreshold  then Result := 1;
end;

Please note that, in this function GetOutput, the result may be either 0 or 1. Normally neurons make digital transfer between each other which we usually represent with 1 or 0. However, on network models, sometimes we will use an analog signal which represents the relation with the load of the cell and the threshold. This analog value which is mostly arranged as to be at the final output layer of the network may be used for two reasons. The first is to calculate the network error between the expected and the actual results. The second is , we sometimes really need analog values as output such as the velocity of an item.

So to be able to use analog outputs on demand, we should apply a lit bit more math on the load and threshold. You can fire the difference, the ratio or any relation between load and threshold. But the best one proved in many experimental practices is using the below exponential relation between them.

If the AnalogOutput property of the Neuron is set to True, the output will be fired according to the above equation. So here is our new GetOutput function.

function TNeuron.GetOutput: Single;
var i: Integer;
begin
  Result := 0;
  if FAnalogOutput then 
    Result := 1/1+Exp(-Load+FThreshold)
  else if Load > FThreshold  then Result := 1;
end;

Now we are ready to use our TNeuron class. Remember that Neurons do great things only if they work together. However for testing the usage of the class I will introduce you a simple demo project on Delphi-Firemonkey which you can see how this TNeuron is used as a single identity. The demo also includes some single usages to mimic the behaviour of some binary logical gates (like and,or,xor); however please never forget that our brain doesn’t have this kind of logical gates as we have in electronics.

For downloading the full source code of the project with the AI.NN.Neuron unit you can use this SVN link. For non-programmers the compiled Win32, Win64, MacOSX (Thanks to Firemonkey) applications are also available to download.