Artificial neural network
Definition
An artificial neural network is a collection of nodes, each of which is an artificial neuron, as well as directed edges between the nodes, called connections, each of which has a weight.
The output of each artificial neuron flows out from it on each of the connections from that node and become an input to the node the connection goes to. The weight applied to that input is the weight on the connection. Each node also has an associated bias term that is the weight applied to the input 1 (not corresponding to any connection).
Nodes that have no connections going to them represent the inputs to the neural network; these correspond to the inputs or features for the problem being modeled through the neural network. Nodes that have no connections going out from them represent the outputs of the neural network, these correspond to the labels we are trying to predict.
Things to decide in a neural network
Let's say we want to design a neural network with inputs and outputs being predicted. There are a few main things to decide about the neural network.
Activation function of the neural network
We need to select the activation function that will be used for the artificial neurons. In principle, a different activation function could be used for each neuron; in practice neural networks choose a single activation function across all neurons.
Topology of the neural network
The topology of the neural network is the diagram of the nodes and their connections -- but without information on the weights and the activation function used. This topology might include connections for which we ultimately decide to set a weight of zero (and that we can therefore remove).
| Type of network | Description of topology | Implications for learning |
|---|---|---|
| Single-layer neural network (no hidden layers) | There are no nodes other than the input and output nodes. The input nodes feed directly into the output nodes. | This basically involves just doing machine learning on the activation function. For instance, if the activation function is a logistic function, it means running a logistic regression separately on each of the outputs. |
| Single-hidden-layer (feedforward) neural network | There is no direct connection between input and output nodes. There is a single hidden layer of neurons; all connections are either from inputs to the hidden layer or from the hidden layer to the outputs. In the "fully connected" version, all the permitted connections exist. |
Information about the problem domain may go into the design of the topology of the neural network.
Weights of the neural network (including bias terms)
The weights of the neural network (including weights on connections as well as bias terms) are the most detailed and specific part of the description of the neural network.
These weights are generally learned by running a gradient descent / backpropagation algorithm on training data.