November 4, 2023
class Neuron contains parameter
self.nonlin = nonlin
The function
def __call__(self, x):
will first calculate and create a new Value act:
act = sum((wi * xi for wi, xi in zip(self.w, x)), self.b)
Depending on the whether nonlin is False or True:
- If nonlin is False:
__call__ will return this Value if nonlin is False.
- If nonlin is True:
__call__ will create another new Value act.relu() and return that new Value
return act.relu() if self.nonlin else act
nonlin=False:
nonlin=True:
November 5, 2023
Multiplication and Gradient
The gradient of each operand is the data of the other operand times the gradient of the following Value
Here, the total gradient of the weight is
7*7=49
When a Value has multiple operations, then the contributions of each gradient are added.
For example, if the operations are multiplications:
def _backward():
self.grad += other.data * out.grad
other.grad += self.data * out.grad
Here, the total gradient of the input is the sum of the products of
the weights with the gradients of the following Values
0.3*1 + 0.5*1 = 0.8
Here, the total gradient of the input is the sum of the products of
the weights with the gradients of the following Values
0.2 * 0.8 + 0.3 * 0.4 = 0.16 + 0.12 = 0.28
