Inference vs Training

The inference is when we call a neural network with an input and get an output. The most common example is LLMs: we type a prompt and get a message in response.
The training is when we find the best weights and biases for our neural network, generally using backpropagation. To train a neural network, we use a dataset (a massive one for LLMs), pass an input to the neural network, and check the output. If the result is wrong, we adjust the weights and biases accordingly. Then we pass another input, and so on.
We also keep some testing data, to test the inference after training.

Neural network

A neural network has:

In reality a neural network is just a function.

Input Hidden 1 Hidden 2 Output

// example

// explain non linearity

Neural networks are meant to answer non-linear questions. For example, let's consider dose and effectiveness. The effectiveness does not linearly follow the increase of the dose; past a certain point, the dose becomes less effective. So a neural network can answer questions even when the data points are a mess.

Effectiveness must be fit by whatever shape the data takes: a squiggle 0 0.5 1 Low Medium High Drug Dose a bent shape 0 0.5 1 Low Medium High Drug Dose or far more complicated 0 0.5 1 Low Medium High Drug Dose

Single input/output neural network

Here is a simple example of a single input and output neural network

× 1.43 + -0.61 × 2.63 + -0.27 × -3.89 × 1.35 Dose (Input) Sum Effectiveness (Output)

In the schema we can see that the hidden layer uses the ReLU activation function. An activation function is just a function where we plug the input on the X axis and get the output on the Y axis.
There are multiple activation functions, but ReLU is commonly used. It basically sets all negative values to 0 and keeps positive ones as-is.

In this example the math is simple:

ReLU(input × 1.43 − 0.61) × (−3.89)
+ ReLU(input × 2.63 − 0.27) × 1.32
= output

If we input enough values we will be able to draw the curve and see that it follows the data points.

1. Each node bends once 0 0.5 1 Dose ReLU(2.63x − 0.27) ReLU(1.43x − 0.61) 2. Scaled by output weights 0 0.5 1 Dose × 1.35 × −3.89 3. Summed: fits the data 0 0.5 1 Dose

Explaining the different activation function

Training aka: Backpropagation

like stated before training is the process of finding the write weight and biases of a neural network for it to answer the problem. For that we have a training data. In our previous case a training data would contain the dose and their known effectiveness. Then we would input the doese and assert if the neiural network find the right output. If not we punish the network and update the weight and bias and input new value and so on.

That process is called backpropagation

Optimizing a single weight

× 1.43 + -0.61 × 2.63 + -0.27 × ??? × 1.35 Dose (Input) Sum Effectiveness (Output)

Lets assume that one weight is unknow, we put a random number 0.0. If we do so the top node no longer contributes anything to the sum, and the curve stops matching the data points:

1. Each node bends once 0 0.5 1 Dose ReLU(2.63x − 0.27) ReLU(1.43x − 0.61) 2. Scaled by output weights (top = 0.0) 0 0.5 1 Dose × 1.35 × 0.0 3. Summed: no longer fits 0 0.5 1 Dose

Our goal is to bend the green curve to fit the data points. For that we compute a loss function, for instance the SSR (sum of squared residuals). The idea is simple: we measure the distance between each data point and the curve, square it (to avoid negative), and sum them all. Then, if we try another weight value and find a lower SSR, it means we follow the data points better.

weight = 0.0 → SSR = 10.32 0 1 2 3 0 0.5 1 Dose residuals weight = −3.89 → SSR ≈ 0.00 0 1 2 3 0 0.5 1 Dose residuals ≈ 0

In theory we could try an infinite number of weights and compare their SSR, but that would be far too costly and slow. So instead we try a handful of weight values, just enough to sketch the SSR curve.

Once we have that curve, we compute its derivative, which measures the steepness of the curve. The idea is to find the bottom of the curve, where the derivative is 0. For that we use an iterative process called gradient descent, which repeatedly computes the derivative and steps downhill until it gets close to 0.

Here are the first three steps, starting from weight = 0.0 and moving by half the slope each time:

Gradient descent: 3 steps toward the best weight 0 5 10 −8 −6 −2 0 Weight SSR tangent, slope 5.31 1.68 0.53 0.17 min ≈ −3.89 step 0 step 1 2 3 step weight slope SSR 0 0.00 5.31 10.32 1 −2.66 1.68 1.03 2 −3.50 0.53 0.10 3 −3.76 0.17 0.01

Each step lands closer to the bottom, the tangent gets flatter, and the SSR shrinks from 10.32 to 0.01. A few more steps and the weight settles on −3.89, the value we started the article with.

Normalizing output: Softmax

To interpret the result neural network apply SoftMax in output to normalize the value between 0 and 1, making it a probability distribution much more easy to interpret.

// output after softmax

Published: July 30, 2026