Artificial neuron - Revision history

Vipul: /* Common choices of activation function */

2021-06-12T02:34:02Z

Common choices of activation function

← Older revision		Revision as of 02:34, 12 June 2021
Line 32:		Line 32:
	\| Linear threshold unit or McCulloch-Pitts neuron \|\| Heaviside step function (zero if less than a threshold, one if above the threshold) \|\| for threshold <math>\theta</math>: 0 if <math>\sum_{j=0}^m w_j x_j < \theta</math>, 1 if <math>\sum_{j=0}^m w_j x_j < \theta</math>, 1/2 if <math>\sum_{j=0}^m w_j x_j = \theta</math> \|\| This is not continuous at the threshold <math>\theta</math>; geometrically the region of discontinuity is a hyperplane. Linear threshold units are good for implementing boolean functions.		\| Linear threshold unit or McCulloch-Pitts neuron \|\| Heaviside step function (zero if less than a threshold, one if above the threshold) \|\| for threshold <math>\theta</math>: 0 if <math>\sum_{j=0}^m w_j x_j < \theta</math>, 1 if <math>\sum_{j=0}^m w_j x_j < \theta</math>, 1/2 if <math>\sum_{j=0}^m w_j x_j = \theta</math> \|\| This is not continuous at the threshold <math>\theta</math>; geometrically the region of discontinuity is a hyperplane. Linear threshold units are good for implementing boolean functions.
	\|-		\|-
	\| Logistic neuron \|\| [[calculus:Logistic function\|logistic function]] \|\| <math>g\left(\sum_{j=0}^m w_j x_j = \theta \right)</math> where <math>g</math> is the logistic function \|\| An artificial neural network with just one logistic neuron is equivalent to [[logistic regression]]. The continuity and in fact infinite differentiability of the logistic function makes it amenable to gradient descent / backpropagation methods. Artificial neural networks where all neurons are logistic neurons are commonly used in practice.		\| Logistic neuron \|\| [[calculus:Logistic function\|logistic function]] \|\| <math>g\left(\sum_{j=0}^m w_j x_j = \theta \right)</math> where <math>g</math> is the logistic function <math>g(t) = \frac{1}{1 + e^{-t}}</math> \|\| An artificial neural network with just one logistic neuron is equivalent to [[logistic regression]]. The continuity and in fact infinite differentiability of the logistic function makes it amenable to gradient descent / backpropagation methods. Artificial neural networks where all neurons are logistic neurons are commonly used in practice.
	\|}		\|}

Vipul: /* Purpose of the activation function */

2021-06-12T02:33:26Z

Purpose of the activation function

← Older revision		Revision as of 02:33, 12 June 2021
Line 23:		Line 23:
	For instance, suppose a self-driving car is trying to determine whether a particular segment of the picture frame represents paved road or a sidewalk. The degree of certainty that the picture is of paved road can be described by a probability that can range from 0 to 1. We may compute this probability using a [[logistic regression]] problem: we combine a lot of different pieces of information about the picture frame to compute a real number describing the log-odds of it being paved road, then apply the logistic function to compute the probability. Here, the logistic function plays the role of the activation function.		For instance, suppose a self-driving car is trying to determine whether a particular segment of the picture frame represents paved road or a sidewalk. The degree of certainty that the picture is of paved road can be described by a probability that can range from 0 to 1. We may compute this probability using a [[logistic regression]] problem: we combine a lot of different pieces of information about the picture frame to compute a real number describing the log-odds of it being paved road, then apply the logistic function to compute the probability. Here, the logistic function plays the role of the activation function.

	~~A few other remarks:~~		For an artificial neural network to have some power beyond a single artificial neuron, we ''must'' have a nonlinear activation function, because composing linear functions just gives us a linear function.

	* The logistic function ~~is a fairly common choice~~ of activation function, ~~and~~ the ~~default~~ [[~~artificial neural network~~]] ~~architecture uses~~ logistic ~~functions at all artificial neurons, so we can view~~ artificial neural ~~networks as generalizations of~~ [[logistic regression]].		=== Common choices of activation function ===
	* Activation functions such as the logistic function~~, and most others that~~ are ~~typically chosen, have the property that for generally nice inputs, they~~ are ~~likely to simulate some form of almost-binary logic, and the artificial neural network can be viewed as a slight fuzzification of what is essentially a Boolean circuit~~.
	* For an artificial neural network to have some power beyond a single artificial neuron, we ''must'' have a nonlinear activation function, because composing linear functions just gives us a linear function.		{\| class="sortable" border="1"
			! Name of artificial neuron type !! Choice of activation function !! Mathematical form !! More information
			\|-
			\| Linear threshold unit or McCulloch-Pitts neuron \|\| Heaviside step function (zero if less than a threshold, one if above the threshold) \|\| for threshold <math>\theta</math>: 0 if <math>\sum_{j=0}^m w_j x_j < \theta</math>, 1 if <math>\sum_{j=0}^m w_j x_j < \theta</math>, 1/2 if <math>\sum_{j=0}^m w_j x_j = \theta</math> \|\| This is not continuous at the threshold <math>\theta</math>; geometrically the region of discontinuity is a hyperplane. Linear threshold units are good for implementing boolean functions.
			\|-
			\| Logistic neuron \|\| [[calculus:Logistic function\|logistic function]] \|\| <math>g\left(\sum_{j=0}^m w_j x_j = \theta \right)</math> where <math>g</math> is the logistic function \|\| An artificial neural network with just one logistic neuron is equivalent to [[logistic regression]]. The continuity and in fact infinite differentiability of the logistic function makes it amenable to gradient descent / backpropagation methods. Artificial neural networks where all neurons are logistic neurons are commonly used in practice.
			\|}

Vipul at 01:56, 12 June 2021

2021-06-12T01:56:51Z

← Older revision		Revision as of 01:56, 12 June 2021
Line 1:		Line 1:
	==Definition==		==Definition==

	An '''artificial neuron'''~~, also known as a '''semi-linear unit''', '''Nv neuron''', '''binary neuron''', '''linear threshold function''', or '''McCulloch–Pitts (MCP) neuron''',~~ is a function of the form:		An '''artificial neuron''' is a function of the form:

	<math>y = \varphi\left(\sum_{j=0}^m w_j x_j\right)</math>		<math>y = \varphi\left(\sum_{j=0}^m w_j x_j\right)</math>

Vipul: Created page with "==Definition== An '''artificial neuron''', also known as a '''semi-linear unit''', '''Nv neuron''', '''binary neuron''', '''linear threshold function''', or '''McCulloch–Pi..."

2014-06-22T17:56:35Z

Created page with "==Definition== An '''artificial neuron''', also known as a '''semi-linear unit''', '''Nv neuron''', '''binary neuron''', '''linear threshold function''', or '''McCulloch–Pi..."

New page

==Definition==

An '''artificial neuron''', also known as a '''semi-linear unit''', '''Nv neuron''', '''binary neuron''', '''linear threshold function''', or '''McCulloch–Pitts (MCP) neuron''', is a function of the form:

<math>y = \varphi\left(\sum_{j=0}^m w_j x_j\right)</math>

where <math>w_j</math> are the weights on the neuron and <math>\varphi</math> is the activation function. Artificial neurons form components of [[artificial neural network]]s: an artificial neural network is obtained by composing and combining artificial neurons (i.e., using the outputs of some neurons as inputs for other neurons).

Generally, in [[machine learning]] problems, the topology of the artificial neural network, as well as the choice of activation function for each neuron, are fixed in advance. The values of the weights are discovered using the [[training set]] by minimizing an appropriately chosen cost function.

===Bias term===

Generally, the variable <math>x_0</math> is always taken to be <math>+1</math>, and called the ''bias term''. The weight <math>w_0</math> is the bias weight.

===Purpose of the weights===

The purpose of the weights is to combine the inputs in a way that extracts some information from all of them.

===Purpose of the activation function===

The purpose of the activation function is to rescale in a manner that extracts the relevant valuable information from the linear combination. In general, the activation function tends to squish the domain down to a smaller subset. The idea is that the goal of the neuron is closer to a classification problem than a problem of finding an exact magnitude, so very large values should get squished down to the same value as intermediate values.

For instance, suppose a self-driving car is trying to determine whether a particular segment of the picture frame represents paved road or a sidewalk. The degree of certainty that the picture is of paved road can be described by a probability that can range from 0 to 1. We may compute this probability using a [[logistic regression]] problem: we combine a lot of different pieces of information about the picture frame to compute a real number describing the log-odds of it being paved road, then apply the logistic function to compute the probability. Here, the logistic function plays the role of the activation function.

A few other remarks:

* The logistic function is a fairly common choice of activation function, and the default [[artificial neural network]] architecture uses logistic functions at all artificial neurons, so we can view artificial neural networks as generalizations of [[logistic regression]].
* Activation functions such as the logistic function, and most others that are typically chosen, have the property that for generally nice inputs, they are likely to simulate some form of almost-binary logic, and the artificial neural network can be viewed as a slight fuzzification of what is essentially a Boolean circuit.
* For an artificial neural network to have some power beyond a single artificial neuron, we ''must'' have a nonlinear activation function, because composing linear functions just gives us a linear function.