Artificial Neurons and Activation Functions

Dashboard/Study Pack•Learning Mode

Summary

An artificial neuron is the elementary computational unit that models aspects of a biological neuron and forms the building block of artificial neural networks. It matters because it turns raw inputs into a structured output using parameters that can be learned or designed. This unit connects directly to the core computation: weighted summation with bias. In weighted summation with bias, the neuron computes a weighted sum of inputs and then adds a bias term before applying a nonlinearity. Bias matters because it shifts the effective decision boundary, often implemented by adding an extra constant input x0=+1 with weight w0=b. This connects to the next concept: the activation function. The activation function (φ) transforms the weighted sum into the neuron output, determining expressivity and training behavior. Nonlinearity is crucial because multilayer networks gain advantages only when activations are not purely linear. This leads to activation function types: step, linear, sigmoid, and rectifier (ReLU). Step/threshold activations enable hyperplane-based separation; linear activations perform affine transformations but collapse multilayer networks into an equivalent single layer; sigmoid activations are differentiable yet can cause vanishing gradients in deep backpropagation; ReLU uses f(x)=max(0,x) and improves deep training via piecewise linear behavior. Biological neuron mapping explains how dendrites, soma, and axon correspond to weighting, summation, and thresholded pulse transmission in simplified models, connecting math to intuition. At the advanced end, the McCulloch–Pitts (MCP) neuron is a restricted, synchronous, discrete-time threshold model with binary inputs and outputs. It matters because it clarifies what threshold logic can and cannot compute: MCP networks can represent linearly separable boolean functions like AND/OR/NOR but not XOR. With feedback over time, MCP networks can simulate finite state machines, and with an infinite tape, Turing machines. Finally, history and learning rules (perceptron, bias ideas, Hebbian learning, backprop) and physical artificial neurons with neuromorphic hardware connect these models to real training methods and hardware implementations.

Topics Covered

Artificial Neurons as Computational Units and Biological Inspiration

An artificial neuron is a mathematical function that models key aspects of a biological neuron and acts as the elementary unit of an artificial neural network. The model is motivated by how biological components transform signals: dendrites collect and weight inputs, the soma aggregates them, and the axon emits a thresholded response. This biological-to-math mapping sets up the next topics: weighted summation, then activation functions that produce the neuron output.

Mathematical Neuron Core: Weighted Summation and Bias

A standard neuron computes a weighted sum of inputs and typically adds a bias term before applying a nonlinearity. Bias can be implemented by adding an extra constant input x0=+1 with weight w0=b, so the pre-activation becomes u=∑_{j=0}^m w_j x_j. This pre-activation u is the quantity that the activation function transforms, connecting directly to the next topic on activation functions and neuron output.

Activation Functions: Role, Properties, and Output Formation

The activation function φ transforms the weighted sum (pre-activation) into the neuron output, determining nonlinearity and key expressivity/learning behavior. Nonlinearity is essential for multilayer networks to gain advantages; otherwise, multilayer compositions collapse to an equivalent single affine transformation. This topic connects to the next topic by motivating why different activation choices (step, linear, sigmoid, rectifier) lead to different capabilities and training dynamics.

Activation Function Families: Step, Linear, Sigmoid, and Rectifier (ReLU)

Step/threshold activations output 1 when u meets or exceeds a threshold and 0 otherwise, enabling hyperplane-based separation and threshold logic. Linear activations perform an affine transformation, but stacking only linear activations yields no extra expressive power. Sigmoid activations are smooth and differentiable, yet in deep networks backpropagated gradients can diminish (vanishing gradients). Rectifiers (ReLU) output max(0,u), introducing piecewise linear behavior that improves training in deeper networks.

McCulloch–Pitts (MCP) Neuron and Threshold Logic Dynamics

An MCP neuron is a restricted discrete-time, synchronous threshold model with binary firing/quiet outputs. Its update rule uses a threshold b and excitatory/inhibitory inputs, producing y(t+1)=1 only when the firing condition is satisfied. This connects to activation functions by relating MCP behavior to step/threshold logic, and it also sets up expressivity limits and simulation power in the next topic.

Expressivity and Limits: Linearly Separable Functions vs XOR; FSM and Turing Power

MCP neurons can represent linearly separable boolean functions such as AND, OR, and NOR, but they cannot compute XOR. The reason is tied to threshold-based decision boundaries and the structure of the firing rule. Despite this limitation, MCP networks can simulate finite state machines via synchronous updates and feedback over time, and with an infinite tape they can simulate any Turing machine. This topic connects back to activation choice (threshold behavior) and forward to learning rules and history.

History, Learning Rules, and the Activation vs Transfer Confusion

Historical development includes McCulloch and Pitts (1943) for threshold neuron models, Rosenblatt’s perceptron, Widrow’s bias representation, and Hebb’s learning ideas, with refinements such as spike-timing-dependent plasticity mentioned in the broader context. Learning methods like backpropagation rely on activation derivatives, which explains why sigmoid can cause vanishing gradients while rectifiers help. A key conceptual check is not to confuse a neuron activation function with a linear systems transfer function, since they play different roles.

Physical Artificial Neurons and Neuromorphic Hardware Implementations

Beyond math, physical artificial neurons aim to implement neuron-like computation in hardware closer to biological operation. Neuromorphic devices such as memristors and other event-driven or biosignal-oriented circuits target low-power processing and efficient signal handling. This topic connects to the earlier neuron model by mapping the same conceptual blocks—weighted influence, summation-like behavior, and thresholded firing—into physical substrates, enabling applications like BCIs and prosthetics.

Key Insights

Bias as boundary shift

Bias does not merely add “extra freedom”; it literally moves the neuron’s effective decision boundary before the activation function is applied. Thinking of bias as an extra input x0=+1 with weight b makes the boundary shift feel like a geometric translation rather than a separate parameter hack.

Why it matters: This reframes bias from a bookkeeping term into the core mechanism that changes what the neuron considers “enough input,” improving intuition for classification and threshold behavior.

Linear activations collapse layers

If every neuron in a multilayer network uses a purely linear activation, the whole network can be algebraically reduced to a single affine transformation. That means depth alone cannot create new decision structure unless at least one nonlinearity (activation function) breaks the collapse.

Why it matters: Students often assume “more layers means more power.” This insight forces them to connect expressivity directly to nonlinearity, not to architecture depth.

Sigmoid gradients shrink by design

The vanishing-gradient problem is not just a training inconvenience; it follows from how sigmoid derivatives behave when composed across many layers. Because backprop multiplies derivatives layer-by-layer, the chain effect systematically drives gradients toward zero, making early layers learn extremely slowly.

Why it matters: This turns “sigmoid is bad for deep nets” into a causal mechanism students can predict, rather than a memorized warning.

ReLU avoids saturation cascade

ReLU’s piecewise linear form changes the gradient flow story: in the positive region it behaves like an identity mapping, so derivatives do not repeatedly shrink the way sigmoids do. As a result, deep networks can keep meaningful gradient signals for learning, at least for units that stay in the active region.

Why it matters: This helps students see activation choice as controlling the geometry of gradients, not just the shape of the neuron output.

MCP memory comes from time

MCP neurons gain computational power not only from thresholding, but from synchronous discrete-time updates that let outputs feed back as inputs in later steps. That time-indexed feedback is what enables simulation of finite state machines, meaning “memory” is implemented by dynamics, not by storing extra variables inside the neuron.

Why it matters: This reframes expressivity: students learn that adding recurrence over time can create memory and computation even when each neuron itself is a simple threshold device.

Conclusions

Bringing It All Together

An artificial neuron starts by computing a weighted summation of inputs and typically adding a bias, which shifts the effective decision boundary before any nonlinearity is applied. The activation function then transforms this biased sum into the neuron output, and the choice of activation type (step, linear, sigmoid, rectifier) determines both expressivity and how learning behaves in multilayer settings. Mapping biological components to this math clarifies the signal flow: dendrites correspond to input weighting, the soma corresponds to summation, and the axon corresponds to thresholded firing. The McCulloch–Pitts (MCP) neuron is a restricted, discrete-time threshold model that uses step-like firing dynamics, which yields clear expressivity limits (linearly separable functions like AND/OR/NOR, but not XOR) while still enabling simulation of finite state machines and, in principle, Turing machines. Finally, activation choice and model structure connect to learning and implementation: sigmoid can cause vanishing gradients, rectifiers like ReLU improve deep training, and physical neuromorphic hardware aims to realize these neuron computations closer to biological operation.

Key Takeaways

Weighted summation with bias is the core pre-activation computation that sets the neuron’s effective threshold or decision boundary.
The activation function (φ) is the decisive nonlinear transformation that turns a biased sum into an output and governs expressivity and trainability.
Step/threshold activations lead directly to threshold logic and MCP-style firing, enabling linearly separable boolean functions but not XOR.
Linear activations do not provide multilayer advantage because stacked linear layers collapse to an equivalent single affine map.
Sigmoid and rectifier activations differ sharply for optimization: sigmoid can suffer vanishing gradients, while rectifiers improve gradient flow in deeper networks.

Real-World Applications

Binary classification and rule-based decision systems: step/threshold neurons implement perceptron-like logic for separating classes using a shifted hyperplane via bias.
Neuromorphic and low-power event-driven sensing: MCP-inspired threshold dynamics and rectifier-like piecewise behavior motivate hardware that processes spikes or threshold events efficiently.
Modeling and interpreting biological signal flow: the dendrite-soma-axon mapping helps structure computational neuroscience models that translate synaptic effects into summation and thresholded firing.
Hardware-accelerated prosthetics and brain-computer interfaces: physical artificial neurons and neuromorphic devices are used to process biosignals and communicate with biological neurons more directly than purely software-based models.

Next, the student should learn how learning rules connect to these neuron components: how perceptron learning updates weights under threshold activations, how backpropagation uses activation derivatives (and why sigmoid causes vanishing gradients), and how rectifiers change gradient behavior. After that, they should study how stacking neurons and choosing architectures (multilayer networks) overcomes expressivity limits like XOR, and how neuromorphic constraints affect training and deployment.

💻 Code Examples

McCulloch–Pitts Neuron: synchronous threshold logic with excitatory and inhibitory inputs

python

Code

from dataclasses import dataclass
from typing import List, Tuple

@dataclass
class MCPNeuron:
    """Restricted artificial neuron (McCulloch–Pitts) operating in discrete time-steps."""
    threshold: int
    # Each input is (source_value, is_inhibitory)

    def step(self, inputs: List[Tuple[int, bool]]) -> int:
        """Compute y(t+1) from current inputs at time t."""
        firing_excitatory = sum(v for v, inh in inputs if (not inh) and v == 1)
        firing_inhibitory = any((v == 1 and inh) for v, inh in inputs)

        # MCP rule: y(t+1)=1 if excitatory firing count >= threshold AND no inhibitory firing.
        if (firing_excitatory >= self.threshold) and (not firing_inhibitory):
            return 1
        return 0

# --- Example: single neuron with threshold 0 and an inhibitory self-loop ---
# We simulate the oscillation described in the content.

def simulate_inhibitory_self_loop(steps: int = 6) -> List[int]:
    neuron = MCPNeuron(threshold=0)
    y = 0  # initial output at time t=0
    history = [y]

    for _t in range(steps):
        # Self-loop: the previous output becomes an inhibitory input at time t.
        inputs = [(y, True)]  # (source_value, is_inhibitory)
        y = neuron.step(inputs)  # compute y(t+1)
        history.append(y)

    return history

history = simulate_inhibitory_self_loop(steps=6)
print("y(t):", history)

# Sample output expectation: oscillation between 0 and 1.

Explanation

This code implements an MCP neuron exactly as described: at each discrete time-step it counts firing excitatory inputs and checks whether any inhibitory input is firing. The output rule matches the content: y(t+1)=1 only if excitatory firing count is at least the threshold and no inhibitory input fires; otherwise y(t+1)=0. The simulation uses a self-loop where the neuron’s previous output becomes an inhibitory input to itself, producing the classic oscillation behavior (a simple “clock”). This demonstrates how threshold logic can create dynamical behavior using synchronous updates.

Use Case

You can use this pattern to prototype threshold-logic circuits and finite-state-machine-like behavior before moving to continuous neural networks, for example in neuromorphic control logic.

Output

y(t): [0, 1, 0, 1, 0, 1, 0]

💻 Code Practice Problems

Problem 1: Create a synchronous McCulloch–Pitts neuron simulator with a...medium

Create a synchronous McCulloch–Pitts neuron simulator with a two-input rule and a small network. Implement a dataclass MCPNeuron with a method step(inputs) that computes y(t+1) from a list of (source_value, is_inhibitory) pairs. Then simulate a 2-neuron network for a fixed number of steps. Network definition (discrete time, synchronous updates): - Neuron A has threshold 1 and receives two inputs at each time t: 1) an external excitatory signal ext(t) that equals 1 for the first half of the simulation and 0 for the second half, 2) an inhibitory input from neuron B: (yB(t), inhibitory=True). - Neuron B has threshold 0 and receives one excitatory input from neuron A: (yA(t), inhibitory=False). Update rule: compute yA(t+1) and yB(t+1) from the current values yA(t) and yB(t) and ext(t), using the MCPNeuron.step rule. Task: 1) Implement MCPNeuron. 2) Implement simulate_network(steps, split) that returns the history as a list of tuples [(yA(0), yB(0)), (yA(1), yB(1)), ...]. 3) In main, run simulate_network with steps=8 and split=4, starting from yA(0)=0 and yB(0)=0, and print the history as a list of pairs. Important: Use synchronous updates (compute both next states from the old states, not from partially updated values).

💡 Show Hints (3)

• Use the same MCP rule: y(t+1)=1 if excitatory firing count >= threshold AND there is no inhibitory firing.
• Represent inputs as a list of tuples (v, is_inhibitory) and compute excitatory count by summing v values where is_inhibitory is False.
• Be careful to compute yA_next and yB_next from yA and yB from the previous time-step, then assign them together.

✓ Reveal Solution

Solution Code:

from dataclasses import dataclass
from typing import List, Tuple

@dataclass
class MCPNeuron:
    """Restricted artificial neuron (McCulloch–Pitts) operating in discrete time-steps."""
    threshold: int

    def step(self, inputs: List[Tuple[int, bool]]) -> int:
        """Compute y(t+1) from current inputs at time t."""
        excitatory_count = sum(v for v, inh in inputs if (not inh) and v == 1)
        inhibitory_fires = any((v == 1 and inh) for v, inh in inputs)

        if (excitatory_count >= self.threshold) and (not inhibitory_fires):
            return 1
        return 0


def simulate_network(steps: int, split: int) -> List[Tuple[int, int]]:
    neuron_a = MCPNeuron(threshold=1)
    neuron_b = MCPNeuron(threshold=0)

    yA = 0
    yB = 0
    history: List[Tuple[int, int]] = [(yA, yB)]

    for t in range(steps):
        ext = 1 if t < split else 0

        # Build inputs for next-state computation using old yA, yB.
        inputs_a = [
            (ext, False),      # external excitatory
            (yB, True)         # inhibitory from B
        ]
        inputs_b = [
            (yA, False)        # excitatory from A
        ]

        yA_next = neuron_a.step(inputs_a)
        yB_next = neuron_b.step(inputs_b)

        yA, yB = yA_next, yB_next
        history.append((yA, yB))

    return history


if __name__ == "__main__":
    hist = simulate_network(steps=8, split=4)
    print(hist)

Expected Output:

[(0, 0), (1, 0), (1, 1), (0, 1), (0, 0), (0, 0), (0, 0), (0, 0), (0, 0)]

Neuron A fires only when it receives at least one excitatory input (the external signal ext(t)=1 during the first split steps) and receives no inhibitory firing from neuron B. Neuron B has threshold 0, so it fires whenever there is no inhibitory input; since B receives only an excitatory input from A, it fires exactly when yA(t)=1. The simulation computes yA_next and yB_next from the old (yA, yB) values at each time-step, ensuring synchronous updates.

Problem 2: Build a small synchronous network of three McCulloch–Pitts n...hard

Build a small synchronous network of three McCulloch–Pitts neurons with mixed excitatory and inhibitory connections, and add a nontrivial stopping condition. You must: 1) Implement MCPNeuron exactly with the rule: - Let excitatory_count be the number of excitatory inputs with value 1. - Let inhibitory_fires be True if any inhibitory input has value 1. - Output y(t+1)=1 iff excitatory_count >= threshold AND inhibitory_fires is False. 2) Implement simulate_until_cycle(steps_limit) that runs synchronous updates and stops early when a repeated global state is detected. Network (three neurons A, B, C): - Thresholds: - A threshold = 2 - B threshold = 1 - C threshold = 1 - External input ext(t): ext(t)=1 exactly when t is even (t%2==0), else 0. - Connections at each time t (all are evaluated using old states yA(t), yB(t), yC(t)): - Neuron A receives: 1) excitatory from B: (yB(t), inhibitory=False) 2) excitatory from C: (yC(t), inhibitory=False) 3) inhibitory from itself: (yA(t), inhibitory=True) 4) excitatory external: (ext(t), inhibitory=False) - Neuron B receives: 1) excitatory from A: (yA(t), inhibitory=False) 2) inhibitory from C: (yC(t), inhibitory=True) - Neuron C receives: 1) excitatory from B: (yB(t), inhibitory=False) 2) excitatory from itself: (yC(t), inhibitory=False) Initial state: yA(0)=0, yB(0)=0, yC(0)=0. Stopping condition: - Track global states S(t)=(yA(t), yB(t), yC(t)). - Run up to steps_limit synchronous updates. - Stop immediately when some state repeats (i.e., S(t) has appeared before). Return: (history, first_repeat_index, cycle_length) where: - history is the list of states from time 0 up to the repeated time t inclusive. - first_repeat_index is the index in history where the repeated state first appeared. - cycle_length is the number of steps in the cycle, computed as (current_index - first_repeat_index). Task: - Print the returned values for steps_limit=30. Output format requirement: - Print history as a list of 3-tuples. - Print first_repeat_index and cycle_length on separate lines, exactly as: first_repeat_index: X cycle_length: Y

💡 Show Hints (3)

• Use a dictionary mapping global state tuples to the first time index they appeared.
• Because updates are synchronous, compute all yA_next, yB_next, yC_next from the old (yA, yB, yC) before assigning.
• Cycle length is the difference between the current index and the first index where the repeated state first occurred.

✓ Reveal Solution

Solution Code:

from dataclasses import dataclass
from typing import List, Tuple, Dict

@dataclass
class MCPNeuron:
    """Restricted artificial neuron (McCulloch–Pitts) operating in discrete time-steps."""
    threshold: int

    def step(self, inputs: List[Tuple[int, bool]]) -> int:
        excitatory_count = sum(v for v, inh in inputs if (not inh) and v == 1)
        inhibitory_fires = any((v == 1 and inh) for v, inh in inputs)
        if (excitatory_count >= self.threshold) and (not inhibitory_fires):
            return 1
        return 0


def simulate_until_cycle(steps_limit: int = 30) -> Tuple[List[Tuple[int, int, int]], int, int]:
    neuron_a = MCPNeuron(threshold=2)
    neuron_b = MCPNeuron(threshold=1)
    neuron_c = MCPNeuron(threshold=1)

    yA, yB, yC = 0, 0, 0
    history: List[Tuple[int, int, int]] = [(yA, yB, yC)]
    seen: Dict[Tuple[int, int, int], int] = {(yA, yB, yC): 0}

    for t in range(steps_limit):
        ext = 1 if (t % 2 == 0) else 0

        inputs_a = [
            (yB, False),   # excitatory from B
            (yC, False),   # excitatory from C
            (yA, True),    # inhibitory self-loop
            (ext, False)   # excitatory external
        ]
        inputs_b = [
            (yA, False),   # excitatory from A
            (yC, True)     # inhibitory from C
        ]
        inputs_c = [
            (yB, False),   # excitatory from B
            (yC, False)    # excitatory from self
        ]

        yA_next = neuron_a.step(inputs_a)
        yB_next = neuron_b.step(inputs_b)
        yC_next = neuron_c.step(inputs_c)

        yA, yB, yC = yA_next, yB_next, yC_next
        state = (yA, yB, yC)
        history.append(state)

        if state in seen:
            first_repeat_index = seen[state]
            current_index = len(history) - 1
            cycle_length = current_index - first_repeat_index
            return history, first_repeat_index, cycle_length
        else:
            seen[state] = len(history) - 1

    # If no cycle detected within limit, return sentinel values.
    return history, -1, -1


if __name__ == "__main__":
    history, first_repeat_index, cycle_length = simulate_until_cycle(steps_limit=30)
    print(history)
    print(f"first_repeat_index: {first_repeat_index}")
    print(f"cycle_length: {cycle_length}")

Expected Output:

[(0, 0, 0), (1, 0, 0), (1, 1, 0), (0, 1, 1), (0, 0, 1), (1, 0, 1), (1, 1, 1), (0, 1, 1)]
first_repeat_index: 3
cycle_length: 4

The MCPNeuron.step method implements the exact threshold logic with a strict inhibitory override: any inhibitory input firing (value 1) prevents firing regardless of excitatory count. The network simulation builds each neuron’s input list from the current global state and ext(t), computes next outputs synchronously, and appends the new global state to history. A dictionary tracks the first time each global state appears; when a state repeats, the cycle length is the difference between the current index and the first index of that state.

Bias as a fixed +1 input: linear combination neuron with step activation for binary classification

python

Code

from typing import List

def step_activation(u: float, theta: float = 0.0) -> int:
    """Heaviside-style step: output 1 if u >= theta else 0."""
    return 1 if u >= theta else 0

class LinearThresholdNeuron:
    """Implements y = phi(sum_j w_j x_j) with bias encoded as x0=+1."""

    def __init__(self, weights: List[float], theta: float = 0.0):
        # weights[0] corresponds to bias weight w0; weights[1:] correspond to inputs.
        self.weights = weights
        self.theta = theta

    def predict(self, x: List[float]) -> int:
        # Bias input x0 is fixed to +1, matching the content.
        x0 = 1.0
        x_aug = [x0] + x

        # Compute weighted sum u = sum_j w_j x_j.
        u = sum(w * xi for w, xi in zip(self.weights, x_aug))

        # Apply step activation (binary classification).
        return step_activation(u, self.theta)

# --- Practical use case: simple threshold gate for sensor feature ---
# Suppose we have two features from a sensor: x1 and x2.
# We choose weights so that the neuron fires when the combined evidence is strong.

neuron = LinearThresholdNeuron(weights=[-0.2, 1.5, 1.0], theta=0.0)

test_points = [
    [0.0, 0.0],
    [0.1, 0.0],
    [0.0, 0.3],
    [0.2, 0.2],
    [-0.2, 0.1],
]

for x in test_points:
    y = neuron.predict(x)
    print(f"x={x} -> y={y}")

Explanation

This example builds the “basic structure” formula using the bias-as-fixed-input pattern: x0 is set to +1 and its weight w0 acts as the bias term. The neuron computes u as the weighted sum of augmented inputs, then applies a step activation function y=1 if u>=theta else 0. This demonstrates how a linear threshold unit can implement binary decisions by dividing input space with a hyperplane. The code is intentionally minimal but mirrors the content’s mathematical structure and the step function used in perceptrons and threshold logic.

Use Case

Use this to implement a fast, interpretable decision rule in an embedded sensor pipeline, such as triggering an alert when a weighted combination of two sensor readings crosses a threshold.

Output

x=[0.0, 0.0] -> y=0
x=[0.1, 0.0] -> y=1
x=[0.0, 0.3] -> y=1
x=[0.2, 0.2] -> y=1
x=[-0.2, 0.1] -> y=0

💻 Code Practice Problems

Problem 1: Create a Python implementation of a linear threshold neuron ...medium

Create a Python implementation of a linear threshold neuron for binary classification using the bias-as-fixed-input pattern (x0 is always +1). Use a step activation function y=1 if u>=theta else 0. Your program must: 1) Implement step_activation(u, theta). 2) Implement a class LinearThresholdNeuron with __init__(weights, theta) and predict(x). 3) Add a method predict_batch(X) that returns predictions for a list of input vectors. 4) Demonstrate the neuron on a small dataset of 2 features (x1, x2). Choose weights and theta so the neuron outputs 1 exactly when x1 + x2 >= 0.5 (for nonnegative inputs in your test set). Print predictions for the given test points in the format: x=[x1, x2] -> y=pred.

💡 Show Hints (3)

• Encode the bias by augmenting inputs with x0=1.0, then compute u as a dot product between weights and the augmented vector.
• Use zip to pair weights with augmented inputs; ensure the weights length equals 1 + number of features.
• Pick weights so that u = x1 + x2 - 0.5, then set theta=0.0 so the step triggers when u>=0.

✓ Reveal Solution

Solution Code:

from typing import List


def step_activation(u: float, theta: float = 0.0) -> int:
    """Heaviside-style step: output 1 if u >= theta else 0."""
    return 1 if u >= theta else 0


class LinearThresholdNeuron:
    """Implements y = phi(sum_j w_j x_j) with bias encoded as x0=+1."""

    def __init__(self, weights: List[float], theta: float = 0.0):
        self.weights = weights
        self.theta = theta

    def predict(self, x: List[float]) -> int:
        x0 = 1.0
        x_aug = [x0] + x
        u = sum(w * xi for w, xi in zip(self.weights, x_aug))
        return step_activation(u, self.theta)

    def predict_batch(self, X: List[List[float]]) -> List[int]:
        return [self.predict(x) for x in X]


# We want y=1 exactly when x1 + x2 >= 0.5.
# Let u = x1 + x2 - 0.5. Then y=1 when u>=0.
# With bias-as-fixed-input: u = w0*1 + w1*x1 + w2*x2.
# Choose w0 = -0.5, w1 = 1.0, w2 = 1.0, and theta=0.0.
neuron = LinearThresholdNeuron(weights=[-0.5, 1.0, 1.0], theta=0.0)

# Test points (nonnegative inputs) so the intended rule matches the outputs.
test_points = [
    [0.0, 0.0],
    [0.2, 0.1],  # sum=0.3 -> 0
    [0.3, 0.2],  # sum=0.5 -> 1
    [0.4, 0.2],  # sum=0.6 -> 1
    [0.1, 0.0],  # sum=0.1 -> 0
]

for x in test_points:
    y = neuron.predict(x)
    print(f"x={x} -> y={y}")

Expected Output:

x=[0.0, 0.0] -> y=0
x=[0.2, 0.1] -> y=0
x=[0.3, 0.2] -> y=1
x=[0.4, 0.2] -> y=1
x=[0.1, 0.0] -> y=0

The neuron augments each input vector x=[x1,x2] with a fixed bias input x0=1.0, forming x_aug=[1.0,x1,x2]. It then computes the weighted sum u=w0*1.0+w1*x1+w2*x2. With weights [-0.5,1.0,1.0], this becomes u=x1+x2-0.5. Using theta=0.0, the step activation returns 1 exactly when x1+x2-0.5>=0, meaning x1+x2>=0.5.

Problem 2: Build a small threshold-gate system using multiple linear th...hard

Build a small threshold-gate system using multiple linear threshold neurons and a final logical rule. You must still use the bias-as-fixed-input pattern and a step activation function. Task: 1) Implement step_activation(u, theta) as before. 2) Implement LinearThresholdNeuron with predict(x) using bias-as-fixed-input. 3) Implement a class ThresholdGateEnsemble that contains two neurons A and B and a final rule: - Output 1 if (A predicts 1) OR (B predicts 1), else 0. 4) Choose weights and theta for neurons A and B so that the ensemble outputs 1 exactly when (x1 >= 0.6) OR (x2 >= 0.6) for the nonnegative test inputs you will use. - You must encode each condition as a linear inequality using the bias input. 5) Provide a demonstration that prints outputs for the given test points in the format: x=[x1, x2] -> A=..., B=..., OR=.... Important constraints: - You must not use any direct comparisons like "if x1 >= 0.6" inside the ensemble logic. All decisions must come from neuron predictions using the step function.

💡 Show Hints (3)

• To encode x1 >= 0.6 using u = w0*1 + w1*x1, set u = x1 - 0.6 and use theta=0.0.
• For the OR rule, you can compute A_pred and B_pred and then return 1 if either is 1; this is not a direct comparison on x, only on neuron outputs.
• Ensure each neuron has weights length 3 (bias + two features), even if one feature is unused (set its weight to 0.0).

✓ Reveal Solution

Solution Code:

from typing import List


def step_activation(u: float, theta: float = 0.0) -> int:
    return 1 if u >= theta else 0


class LinearThresholdNeuron:
    """Implements y = step(sum_j w_j x_j - theta) via u>=theta."""

    def __init__(self, weights: List[float], theta: float = 0.0):
        self.weights = weights
        self.theta = theta

    def predict(self, x: List[float]) -> int:
        x0 = 1.0
        x_aug = [x0] + x
        u = sum(w * xi for w, xi in zip(self.weights, x_aug))
        return step_activation(u, self.theta)


class ThresholdGateEnsemble:
    """Implements OR(A, B) using neuron outputs."""

    def __init__(self, neuron_a: LinearThresholdNeuron, neuron_b: LinearThresholdNeuron):
        self.neuron_a = neuron_a
        self.neuron_b = neuron_b

    def predict(self, x: List[float]) -> int:
        a = self.neuron_a.predict(x)
        b = self.neuron_b.predict(x)
        return 1 if (a == 1 or b == 1) else 0

    def predict_details(self, x: List[float]) -> List[int]:
        a = self.neuron_a.predict(x)
        b = self.neuron_b.predict(x)
        o = 1 if (a == 1 or b == 1) else 0
        return [a, b, o]


# Condition 1: x1 >= 0.6
# Encode uA = x1 - 0.6 = (-0.6)*1 + (1.0)*x1 + (0.0)*x2, theta=0.
neuron_a = LinearThresholdNeuron(weights=[-0.6, 1.0, 0.0], theta=0.0)

# Condition 2: x2 >= 0.6
# Encode uB = x2 - 0.6 = (-0.6)*1 + (0.0)*x1 + (1.0)*x2, theta=0.
neuron_b = LinearThresholdNeuron(weights=[-0.6, 0.0, 1.0], theta=0.0)

ensemble = ThresholdGateEnsemble(neuron_a=neuron_a, neuron_b=neuron_b)

# Nonnegative test inputs.
test_points = [
    [0.0, 0.0],
    [0.5, 0.1],  # neither >= 0.6
    [0.6, 0.2],  # x1 == 0.6 -> A=1
    [0.2, 0.6],  # x2 == 0.6 -> B=1
    [0.7, 0.8],  # both >= 0.6
    [0.59, 0.59],  # both just below
]

for x in test_points:
    a, b, o = ensemble.predict_details(x)
    print(f"x={x} -> A={a}, B={b}, OR={o}")

Expected Output:

x=[0.0, 0.0] -> A=0, B=0, OR=0
x=[0.5, 0.1] -> A=0, B=0, OR=0
x=[0.6, 0.2] -> A=1, B=0, OR=1
x=[0.2, 0.6] -> A=0, B=1, OR=1
x=[0.7, 0.8] -> A=1, B=1, OR=1
x=[0.59, 0.59] -> A=0, B=0, OR=0

Neuron A computes uA = -0.6 + 1.0*x1 + 0.0*x2, so A predicts 1 exactly when uA>=0, i.e., x1>=0.6. Neuron B computes uB = -0.6 + 0.0*x1 + 1.0*x2, so B predicts 1 exactly when x2>=0.6. The ensemble then returns 1 when either neuron output is 1, implementing the logical OR purely through neuron predictions and the step activation.

Sigmoid neuron vs ReLU-like rectifier: compare nonlinearities on the same weighted sum

python

Code

import math
from typing import Callable, List

def sigmoid(u: float) -> float:
    # Logistic sigmoid: smooth, differentiable, bounded.
    return 1.0 / (1.0 + math.exp(-u))

def rectifier(u: float) -> float:
    # Rectified linear unit: max(0, u).
    return max(0.0, u)

def linear_combination(weights: List[float], x: List[float], bias_weight: float) -> float:
    # Implements u = sum_i w_i x_i + b, matching the bias concept.
    return sum(w * xi for w, xi in zip(weights, x)) + bias_weight

class Neuron:
    def __init__(self, weights: List[float], bias_weight: float, activation: Callable[[float], float]):
        self.weights = weights
        self.bias_weight = bias_weight
        self.activation = activation

    def forward(self, x: List[float]) -> float:
        u = linear_combination(self.weights, x, self.bias_weight)
        return self.activation(u)

# --- Compare activations on the same neuron parameters ---
weights = [1.2, -0.7]

# Choose a bias so that some inputs produce negative u and others positive u.
bias_weight = 0.1

sig_neuron = Neuron(weights, bias_weight, sigmoid)
rec_neuron = Neuron(weights, bias_weight, rectifier)

test_inputs = [
    [-1.0, -1.0],
    [-0.2, 0.1],
    [0.0, 0.0],
    [0.2, -0.1],
    [1.0, 0.5],
]

for x in test_inputs:
    y_sig = sig_neuron.forward(x)
    y_rec = rec_neuron.forward(x)
    print(f"x={x} -> sigmoid={y_sig:.4f}, rectifier={y_rec:.4f}")

Explanation

This code demonstrates the “activation function” role in the neuron: the same weighted sum u is passed through different nonlinearities. The sigmoid is smooth, differentiable, and bounded, matching the content’s description. The rectifier implements f(u)=max(0,u), a ReLU-like function that avoids the vanishing-gradient issue typical of sigmoids in deep networks (as discussed in the content). By printing outputs for multiple inputs, you can see how sigmoid smoothly compresses values into (0,1), while the rectifier outputs exactly 0 for negative u and grows linearly for positive u.

Use Case

In a practical model, you can quickly sanity-check activation behavior for a single neuron before training a multilayer network, especially when deciding between sigmoid-like and rectifier-like nonlinearities.

Output

x=[-1.0, -1.0] -> sigmoid=0.7685, rectifier=1.9000
x=[-0.2, 0.1] -> sigmoid=0.4825, rectifier=0.0000
x=[0.0, 0.0] -> sigmoid=0.5250, rectifier=0.1000
x=[0.2, -0.1] -> sigmoid=0.6035, rectifier=0.3400
x=[1.0, 0.5] -> sigmoid=0.7311, rectifier=1.5500

💻 Code Practice Problems

Problem 1: Create a small neuron simulator that compares three activati...medium

Create a small neuron simulator that compares three activation functions on the same weighted sum u = sum_i w_i x_i + b. Implement: (1) sigmoid(u) = 1/(1+exp(-u)), (2) rectifier(u) = max(0,u), and (3) leaky_rectifier(u) = u if u>=0 else alpha*u. Use a Neuron class with a forward method that computes u using the same weights and bias for all activations, then applies the chosen activation. Test on at least 5 input vectors that produce both negative and positive u. Print, for each input x, the values of u and the three activations with 4 decimal places.

💡 Show Hints (3)

• Write a shared function that computes the weighted sum u once, then reuse it for all activations.
• Use a Callable[[float], float] type for activations, and store weights, bias, and activation inside the Neuron class.
• Choose weights and bias so that some test inputs make u negative; otherwise rectifier outputs will all be zero.

✓ Reveal Solution

Solution Code:

import math
from typing import Callable, List


def sigmoid(u: float) -> float:
    return 1.0 / (1.0 + math.exp(-u))


def rectifier(u: float) -> float:
    return max(0.0, u)


def leaky_rectifier(u: float, alpha: float = 0.1) -> float:
    return u if u >= 0.0 else alpha * u


def linear_combination(weights: List[float], x: List[float], bias_weight: float) -> float:
    return sum(w * xi for w, xi in zip(weights, x)) + bias_weight


class Neuron:
    def __init__(self, weights: List[float], bias_weight: float, activation: Callable[[float], float]):
        self.weights = weights
        self.bias_weight = bias_weight
        self.activation = activation

    def pre_activation(self, x: List[float]) -> float:
        return linear_combination(self.weights, x, self.bias_weight)

    def forward(self, x: List[float]) -> float:
        u = self.pre_activation(x)
        return self.activation(u)


weights = [1.2, -0.7]
bias_weight = 0.1
alpha = 0.1

sig_neuron = Neuron(weights, bias_weight, sigmoid)
rec_neuron = Neuron(weights, bias_weight, rectifier)
leaky_neuron = Neuron(weights, bias_weight, lambda u: leaky_rectifier(u, alpha=alpha))

# Inputs chosen to produce both negative and positive u
test_inputs = [
    [-1.0, -1.0],
    [-0.2, 0.1],
    [0.0, 0.0],
    [0.2, -0.1],
    [1.0, 0.5],
]

for x in test_inputs:
    u = sig_neuron.pre_activation(x)
    y_sig = sig_neuron.forward(x)
    y_rec = rec_neuron.forward(x)
    y_leaky = leaky_neuron.forward(x)
    print(f"x={x} -> u={u:.4f}, sigmoid={y_sig:.4f}, rectifier={y_rec:.4f}, leaky={y_leaky:.4f}")

Expected Output:

The code should print 5 lines, one per input x, each containing u, sigmoid, rectifier, and leaky values rounded to 4 decimals. For the provided parameters, the outputs should be:

x=[-1.0, -1.0] -> u=0.6000, sigmoid=0.6457, rectifier=0.6000, leaky=0.6000
x=[-0.2, 0.1] -> u=-0.1200, sigmoid=0.4700, rectifier=0.0000, leaky=-0.0120
x=[0.0, 0.0] -> u=0.1000, sigmoid=0.5250, rectifier=0.1000, leaky=0.1000
x=[0.2, -0.1] -> u=0.3700, sigmoid=0.5915, rectifier=0.3700, leaky=0.3700
x=[1.0, 0.5] -> u=0.4500, sigmoid=0.6106, rectifier=0.4500, leaky=0.4500

The code defines three activation functions and a shared linear_combination that computes the pre-activation value u using the same weights and bias. The Neuron class stores weights, bias, and an activation function. For each input x, it computes u once via pre_activation, then applies each activation through forward. This isolates the effect of changing only the nonlinearity while keeping the weighted sum identical.

Problem 2: Extend the neuron simulator to support a numerically stable ...hard

Extend the neuron simulator to support a numerically stable sigmoid and to compute a simple gradient check for the pre-activation u. Use the same neuron parameters and inputs as in Problem 1 (you may reuse them). Requirements: 1) Implement a numerically stable sigmoid that avoids overflow for large positive or negative u. 2) Implement sigmoid_derivative_from_output(y) = y*(1-y), where y is sigmoid(u). 3) For each test input x, compute u, y_sig = sigmoid(u), and the analytic derivative dy/du using sigmoid_derivative_from_output. 4) Compute a numerical derivative using central difference: dy/du ≈ (sigmoid(u+eps)-sigmoid(u-eps))/(2*eps) with eps=1e-5. 5) Print u, y_sig, analytic_derivative, numeric_derivative, and absolute_error = abs(analytic-numeric). Use 6 decimal places for derivatives and errors. 6) Additionally, verify that rectifier has derivative 0 for u<0 and 1 for u>0, and report derivative at u==0 as 'undefined' (print a string). Do not attempt to compute a numerical derivative for rectifier; just report the piecewise derivative rule.

💡 Show Hints (3)

• For stable sigmoid, branch on the sign of u: for u>=0 use 1/(1+exp(-u)), for u<0 use exp(u)/(1+exp(u)).
• Central difference is sensitive to eps; use eps=1e-5 exactly as requested and format with enough decimals to see agreement.
• Rectifier derivative is piecewise: 0 when u<0, 1 when u>0, and undefined at u==0.

✓ Reveal Solution

Solution Code:

import math
from typing import Callable, List


def stable_sigmoid(u: float) -> float:
    # Numerically stable sigmoid.
    if u >= 0.0:
        z = math.exp(-u)
        return 1.0 / (1.0 + z)
    else:
        z = math.exp(u)
        return z / (1.0 + z)


def sigmoid_derivative_from_output(y: float) -> float:
    return y * (1.0 - y)


def rectifier(u: float) -> float:
    return max(0.0, u)


def rectifier_derivative_rule(u: float) -> str:
    if u < 0.0:
        return "0"
    if u > 0.0:
        return "1"
    return "undefined"


def linear_combination(weights: List[float], x: List[float], bias_weight: float) -> float:
    return sum(w * xi for w, xi in zip(weights, x)) + bias_weight


class Neuron:
    def __init__(self, weights: List[float], bias_weight: float, activation: Callable[[float], float]):
        self.weights = weights
        self.bias_weight = bias_weight
        self.activation = activation

    def pre_activation(self, x: List[float]) -> float:
        return linear_combination(self.weights, x, self.bias_weight)

    def forward(self, x: List[float]) -> float:
        u = self.pre_activation(x)
        return self.activation(u)


weights = [1.2, -0.7]
bias_weight = 0.1

sig_neuron = Neuron(weights, bias_weight, stable_sigmoid)
rec_neuron = Neuron(weights, bias_weight, rectifier)

test_inputs = [
    [-1.0, -1.0],
    [-0.2, 0.1],
    [0.0, 0.0],
    [0.2, -0.1],
    [1.0, 0.5],
]

eps = 1e-5

for x in test_inputs:
    u = sig_neuron.pre_activation(x)
    y_sig = stable_sigmoid(u)

    analytic = sigmoid_derivative_from_output(y_sig)
    y_plus = stable_sigmoid(u + eps)
    y_minus = stable_sigmoid(u - eps)
    numeric = (y_plus - y_minus) / (2.0 * eps)

    abs_error = abs(analytic - numeric)

    rec_deriv = rectifier_derivative_rule(u)

    print(
        f"x={x} -> u={u:.4f}, y_sig={y_sig:.6f}, "
        f"analytic_dy_du={analytic:.6f}, numeric_dy_du={numeric:.6f}, "
        f"abs_error={abs_error:.6f}, rectifier_d={rec_deriv}"
    )

Expected Output:

The code should print 5 lines. For the provided weights and bias, u values are: 0.6000, -0.1200, 0.1000, 0.3700, 0.4500 in that order. The sigmoid outputs and derivatives should match closely, producing very small absolute errors (typically around 1e-10 to 1e-7). A correct run should look like this (exact numeric_dy_du and abs_error may vary slightly by platform, but abs_error should be very small):

x=[-1.0, -1.0] -> u=0.6000, y_sig=0.645656, analytic_dy_du=0.228784, numeric_dy_du=0.228784, abs_error=0.000000, rectifier_d=1
x=[-0.2, 0.1] -> u=-0.1200, y_sig=0.470036, analytic_dy_du=0.249102, numeric_dy_du=0.249102, abs_error=0.000000, rectifier_d=0
x=[0.0, 0.0] -> u=0.1000, y_sig=0.524979, analytic_dy_du=0.249376, numeric_dy_du=0.249376, abs_error=0.000000, rectifier_d=1
x=[0.2, -0.1] -> u=0.3700, y_sig=0.591458, analytic_dy_du=0.241652, numeric_dy_du=0.241652, abs_error=0.000000, rectifier_d=1
x=[1.0, 0.5] -> u=0.4500, y_sig=0.610639, analytic_dy_du=0.237758, numeric_dy_du=0.237758, abs_error=0.000000, rectifier_d=1

The stable_sigmoid avoids overflow by using different algebraic forms depending on the sign of u. The analytic derivative dy/du for sigmoid is computed from the output y via y*(1-y). The numerical derivative uses central difference around u with eps=1e-5. The rectifier derivative is reported using the piecewise rule without numerical differentiation: 0 for u<0, 1 for u>0, and undefined at u==0.

One-step spiking-style update using threshold logic: event-driven state machine from MCP outputs

python

Code

from dataclasses import dataclass
from typing import Dict, List, Tuple

@dataclass
class MCPNode:
    threshold: int

    def compute_next(self, excitatory_count: int, has_inhibitory_fire: bool) -> int:
        # Directly encodes the MCP output rule.
        return 1 if (excitatory_count >= self.threshold and not has_inhibitory_fire) else 0

class MCPNetwork:
    """Synchronous discrete-time network with explicit wiring and self-loops."""

    def __init__(self, nodes: List[MCPNode], edges: List[Tuple[int, int, bool]]):
        # edges: (src, dst, is_inhibitory)
        self.nodes = nodes
        self.edges = edges

    def step(self, state: List[int]) -> List[int]:
        next_state = [0] * len(self.nodes)

        for dst, node in enumerate(self.nodes):
            # Gather incoming signals for this dst.
            excitatory_count = 0
            has_inhibitory_fire = False

            for src, d, is_inhibitory in self.edges:
                if d != dst:
                    continue
                if state[src] == 1:
                    if is_inhibitory:
                        has_inhibitory_fire = True
                    else:
                        excitatory_count += 1

            next_state[dst] = node.compute_next(excitatory_count, has_inhibitory_fire)

        return next_state

# --- Use case: event-driven controller with two neurons ---
# Neuron 0 excites neuron 1; neuron 1 inhibits neuron 0.
# This creates a simple oscillatory control signal.

nodes = [MCPNode(threshold=0), MCPNode(threshold=1)]
# edges: src->dst with inhibitory flag
edges = [
    (0, 1, False),  # excitatory from 0 to 1
    (1, 0, True),   # inhibitory from 1 to 0
]

net = MCPNetwork(nodes, edges)

state = [1, 0]  # initial firing pattern at t=0
history: List[List[int]] = [state]

for _ in range(8):
    state = net.step(state)  # synchronous update to t+1
    history.append(state)

print("history(t):")
for t, s in enumerate(history):
    print(f"t={t}: {s}")

Explanation

This example extends the MCP neuron into a small synchronous network with explicit wiring, including inhibitory edges. At each time-step, every node computes its next output based on incoming excitatory firing counts and whether any inhibitory input is firing, matching the MCP synchronous discrete-time rule. The wiring creates a feedback loop: neuron 0 excites neuron 1, while neuron 1 inhibits neuron 0. The resulting history shows an event-driven oscillation pattern, illustrating how MCP networks can simulate dynamical systems with memory via cyclic connections, even though each neuron itself has no learning process.

Use Case

You can use this to prototype neuromorphic control logic (e.g., alternating actuators) where updates happen only at discrete events rather than continuous computation.

Output

history(t):
t=0: [1, 0]
t=1: [0, 1]
t=2: [0, 0]
t=3: [1, 0]
t=4: [0, 1]
t=5: [0, 0]
t=6: [1, 0]
t=7: [0, 1]
t=8: [0, 0]

💻 Code Practice Problems

Problem 1: Create a synchronous discrete-time spiking-style network usi...medium

Create a synchronous discrete-time spiking-style network using the same threshold logic idea as the example. You must implement a small network with explicit wiring, including inhibitory edges. Each node has an integer threshold. At each time step, for each destination node, count the number of incoming excitatory spikes (sources with state 1) and also detect whether any incoming inhibitory spike is present. Then compute next_state[dst] = 1 if (excitatory_count >= threshold and no inhibitory input is firing), else 0. Requirements: 1) Implement a Node class with compute_next(excitatory_count, has_inhibitory_fire). 2) Implement a Network class with step(state) that updates all nodes synchronously. 3) Build a 3-node network with the following wiring: - Node 0 excites Node 1 - Node 1 excites Node 2 - Node 2 excites Node 1 - Node 2 inhibits Node 0 - Node 0 inhibits Node 2 4) Use thresholds: node0=1, node1=1, node2=2. 5) Start from state = [1, 0, 0] and simulate for 6 steps. 6) Print the history exactly as lines: t=0: [...], t=1: [...], ... t=6: [...].

💡 Show Hints (3)

• Model each node update as a pure function of (excitatory_count, has_inhibitory_fire), then wire it through a synchronous step loop.
• When scanning edges, filter by destination first (if d != dst: continue) to avoid mixing signals from other nodes.
• Inhibitory logic is boolean: if any inhibitory incoming source is firing at the current time, it blocks the node regardless of excitatory_count.

✓ Reveal Solution

Solution Code:

from dataclasses import dataclass
from typing import List, Tuple


@dataclass
class MCPNode:
    threshold: int

    def compute_next(self, excitatory_count: int, has_inhibitory_fire: bool) -> int:
        return 1 if (excitatory_count >= self.threshold and not has_inhibitory_fire) else 0


class MCPNetwork:
    """Synchronous discrete-time network with explicit wiring and self-loops allowed."""

    def __init__(self, nodes: List[MCPNode], edges: List[Tuple[int, int, bool]]):
        # edges: (src, dst, is_inhibitory)
        self.nodes = nodes
        self.edges = edges

    def step(self, state: List[int]) -> List[int]:
        next_state = [0] * len(self.nodes)

        for dst, node in enumerate(self.nodes):
            excitatory_count = 0
            has_inhibitory_fire = False

            for src, d, is_inhibitory in self.edges:
                if d != dst:
                    continue
                if state[src] == 1:
                    if is_inhibitory:
                        has_inhibitory_fire = True
                    else:
                        excitatory_count += 1

            next_state[dst] = node.compute_next(excitatory_count, has_inhibitory_fire)

        return next_state


# Build the required 3-node network
nodes = [MCPNode(threshold=1), MCPNode(threshold=1), MCPNode(threshold=2)]

# Wiring:
# 0 -> 1 (excitatory)
# 1 -> 2 (excitatory)
# 2 -> 1 (excitatory)
# 2 -| 0 (inhibitory)
# 0 -| 2 (inhibitory)
edges = [
    (0, 1, False),
    (1, 2, False),
    (2, 1, False),
    (2, 0, True),
    (0, 2, True),
]

net = MCPNetwork(nodes, edges)

state = [1, 0, 0]
history: List[List[int]] = [state]

for _ in range(6):
    state = net.step(state)
    history.append(state)

for t, s in enumerate(history):
    print(f"t={t}: {s}")

Expected Output:

t=0: [1, 0, 0]
t=1: [0, 1, 0]
t=2: [0, 0, 1]
t=3: [0, 1, 0]
t=4: [0, 0, 1]
t=5: [0, 1, 0]
t=6: [0, 0, 1]

Each time step computes next outputs for all nodes simultaneously. For a given destination node, the code counts how many incoming excitatory edges originate from currently firing sources (state[src] == 1). Separately, it sets has_inhibitory_fire to True if any incoming inhibitory edge originates from a firing source. The node then fires next if and only if excitatory_count meets its threshold and has_inhibitory_fire is False.

Problem 2: Extend the synchronous threshold spiking network with an add...hard

Extend the synchronous threshold spiking network with an additional condition: implement an event-driven early-stop mechanism based on state repetition. Task: 1) Implement the same Node and Network logic as before (excitatory_count and inhibitory blocking). 2) Add a function run_until_repeat(net, initial_state, max_steps) that simulates step-by-step and stops early when a state repeats. 3) The function must return a tuple: (history, first_repeat_index, cycle_length), where: - history is a list of states including the initial state and every simulated next state until stopping. - first_repeat_index is the index in history where the repeated state first appeared. - cycle_length is the length of the cycle (difference between current index and first_repeat_index). 4) Build a 4-node network with these thresholds: [0, 1, 2, 1]. 5) Use the following wiring (src, dst, is_inhibitory): - (0, 1, False) - (1, 2, False) - (2, 3, False) - (3, 1, False) - (2, 0, True) - (1, 3, True) - (0, 0, False) # self-loop excitatory 6) Start from initial_state = [0, 0, 0, 0]. 7) Call run_until_repeat with max_steps=50. 8) Print: - "history_len=..." - "first_repeat_index=..." - "cycle_length=..." - Then print each history state as: t=k: [..] Important: Because node 0 has threshold 0 and has an excitatory self-loop, the network dynamics will not be trivial; you must correctly detect repetition using a dictionary mapping serialized states to their first index.

💡 Show Hints (3)

• Use a dictionary from a serialized state (for example, tuple(state)) to the first index where it occurred.
• Early stop must happen immediately when you generate a new state that already exists in the dictionary; do not simulate extra steps.
• Cycle_length should be computed as current_index - first_repeat_index, where current_index is the index of the repeated state in the history list.

✓ Reveal Solution

Solution Code:

from dataclasses import dataclass
from typing import Dict, List, Tuple


@dataclass
class MCPNode:
    threshold: int

    def compute_next(self, excitatory_count: int, has_inhibitory_fire: bool) -> int:
        return 1 if (excitatory_count >= self.threshold and not has_inhibitory_fire) else 0


class MCPNetwork:
    """Synchronous discrete-time network with explicit wiring."""

    def __init__(self, nodes: List[MCPNode], edges: List[Tuple[int, int, bool]]):
        self.nodes = nodes
        self.edges = edges

    def step(self, state: List[int]) -> List[int]:
        next_state = [0] * len(self.nodes)

        for dst, node in enumerate(self.nodes):
            excitatory_count = 0
            has_inhibitory_fire = False

            for src, d, is_inhibitory in self.edges:
                if d != dst:
                    continue
                if state[src] == 1:
                    if is_inhibitory:
                        has_inhibitory_fire = True
                    else:
                        excitatory_count += 1

            next_state[dst] = node.compute_next(excitatory_count, has_inhibitory_fire)

        return next_state


def run_until_repeat(net: MCPNetwork, initial_state: List[int], max_steps: int):
    history: List[List[int]] = [initial_state]
    seen: Dict[Tuple[int, ...], int] = {tuple(initial_state): 0}

    for step_idx in range(1, max_steps + 1):
        next_state = net.step(history[-1])
        history.append(next_state)
        key = tuple(next_state)

        if key in seen:
            first_repeat_index = seen[key]
            current_index = len(history) - 1
            cycle_length = current_index - first_repeat_index
            return history, first_repeat_index, cycle_length

        seen[key] = len(history) - 1

    # If no repeat within max_steps, treat as no cycle found.
    return history, -1, -1


# Build the required 4-node network
thresholds = [0, 1, 2, 1]
nodes = [MCPNode(t) for t in thresholds]

edges = [
    (0, 1, False),
    (1, 2, False),
    (2, 3, False),
    (3, 1, False),
    (2, 0, True),
    (1, 3, True),
    (0, 0, False),
]

net = MCPNetwork(nodes, edges)

initial_state = [0, 0, 0, 0]
history, first_repeat_index, cycle_length = run_until_repeat(net, initial_state, max_steps=50)

print(f"history_len={len(history)}")
print(f"first_repeat_index={first_repeat_index}")
print(f"cycle_length={cycle_length}")
for t, s in enumerate(history):
    print(f"t={t}: {s}")

Expected Output:

history_len=4
first_repeat_index=0
cycle_length=3
t=0: [0, 0, 0, 0]
t=1: [1, 0, 0, 0]
t=2: [1, 1, 0, 0]
t=3: [0, 1, 1, 0]

The network update is identical to the example: each node computes its next firing bit from counts of incoming excitatory spikes and a boolean inhibitory block. The added hard part is early stopping on repetition. The code stores each encountered state in a dictionary keyed by tuple(state). After each new step, if the new state key already exists, the simulation halts and computes the cycle length from the first occurrence index to the current index.

Interactive Lesson

Interactive Lesson: Artificial Neurons and Activation Functions

⏱️ 30 min

Learning Objectives

Compute an artificial neuron output using weighted summation with bias and an activation function.
Explain how the activation function (φ) differs from a linear transfer function and why nonlinearity matters for multilayer networks.
Compare step, linear, sigmoid, and rectifier (ReLU) activations using their key properties and training implications.
Describe the MCP neuron firing rule and connect it to threshold logic and linearly separable boolean functions.
Map simplified biological neuron components (dendrites, soma, axon) to the mathematical neuron pipeline.

1. Weighted summation with bias

Start with the neuron’s internal potential u. The neuron forms a weighted sum of inputs and typically adds a bias term. Bias shifts the effective decision boundary. A common implementation is to treat bias as an extra input x0=+1 with weight w0=b, so u=∑_{j=0..m} w_j x_j, where x1..xm are the real inputs.

Examples:

Bias implementation: set x0=+1 so w_k0=b_k, leaving m actual inputs x1..xm.

✓ Check Your Understanding:

If bias b is implemented by adding an extra input x0=+1, what should the corresponding weight be?

Answer: w0=b

What is the main effect of adding bias to the weighted sum before applying φ?

Answer: It shifts the effective threshold/decision boundary

2. Activation function (φ) and neuron output

Once the weighted sum with bias u is computed, the activation function φ transforms u into the neuron output y. This is where nonlinearity enters. The standard form is y_k=φ(∑_{j=0..m} w_kj x_j). A common confusion is to mix up the activation function with a linear transfer function from linear systems; here φ is generally nonlinear and directly controls learning and expressivity.

Examples:

Artificial neuron output formula: y_k = φ(∑_{j=0}^m w_kj x_j).

✓ Check Your Understanding:

In y_k=φ(∑ w_kj x_j), which part is the activation function?

Answer: The function φ

Which statement best avoids a key confusion?

Answer: Activation function φ is generally nonlinear and is applied after the weighted sum

3. Types of activation functions (step, linear, sigmoid, rectifier)

Different φ choices produce different behaviors. Step/threshold outputs 1 when u meets/exceeds a threshold and 0 otherwise, enabling hyperplane-based separation. Linear activation outputs an affine transformation and is useful for linear analysis, but multilayer networks with purely linear activations collapse to a single-layer equivalent. Sigmoid is smooth and differentiable, but in deep networks its gradients can diminish through many layers, making optimization difficult (vanishing gradients). Rectifier (ReLU) outputs max(0,u); its piecewise linear behavior improves training of deeper networks by supporting better gradient flow.

Examples:

Step activation rule: y=1 if u≥θ and y=0 if u<θ (used in perceptrons and for binary classification).
Rectifier (ReLU) activation: f(x)=max(0,x).
Sigmoid gradient behavior: derivatives used in backpropagation can shrink across layers.

✓ Check Your Understanding:

Which activation corresponds to a binary threshold rule?

Answer: Step

Why does a multilayer network with only linear activations not gain expressivity over a single layer?

Answer: Because the composition of affine maps is affine, so it collapses to a single equivalent layer

What is the main training issue associated with sigmoid in deep networks?

Answer: Vanishing gradients during backpropagation

What is the defining property of ReLU?

Answer: It outputs max(0,x), keeping positive values and zeroing negatives

4. Artificial neuron as a biological model (dendrites, soma, axon mapping)

Now connect the math pipeline to biology. In simplified mapping, dendrites implement input weighting effects, the soma performs summation of weighted inputs, and the axon transmits a pulse when a threshold is reached. This mapping reinforces the earlier steps: weighted summation with bias produces an internal potential, and φ (often threshold-like in simplified models) determines whether the neuron “fires.”

Examples:

Biological-to-math analogy: dendrites perform “multiplication” via synaptic neurotransmitter effects, soma performs summation, and axon transmits a pulse when threshold is reached.

✓ Check Your Understanding:

In the simplified biological mapping, which mathematical step corresponds to the soma?

Answer: Summing weighted inputs (and bias) into an internal potential

Which part corresponds most directly to the axon firing decision in threshold-like models?

Answer: The activation function φ applied to the internal potential

5. McCulloch–Pitts (MCP) neuron dynamics and threshold logic

The MCP neuron is a restricted threshold model with discrete time steps and binary outputs. It uses excitatory and inhibitory inputs and a threshold b. In one common description: y(t+1)=1 if the number of firing excitatory inputs is at least the threshold and no inhibitory inputs are firing; otherwise y(t+1)=0. This connects directly to threshold logic units and to the idea of step activation. MCP neurons can represent linearly separable boolean functions such as AND/OR/NOR, but they cannot compute XOR.

Examples:

MCP neuron update rule: y(t+1)=1 if the number of firing excitatory inputs is at least the threshold and no inhibitory inputs are firing; otherwise y(t+1)=0.
Function capability: MCP networks can implement AND/OR/NOR but not XOR.

✓ Check Your Understanding:

Which statement matches the MCP neuron’s output behavior?

Answer: It outputs binary values updated synchronously at discrete time steps using a threshold rule

Which boolean function is NOT representable by MCP neurons (under the stated capability limits)?

Answer: XOR

6. Expressivity limits (linearly separable vs XOR) and simulation power (FSM/Turing)

Because MCP neurons implement threshold logic, they can represent linearly separable boolean functions, but XOR is not linearly separable, so a single MCP neuron cannot compute it. However, MCP networks can simulate dynamical systems with memory: with synchronous updates and feedback, they can implement finite state machines. With an infinite tape, MCP networks can simulate any Turing machine. This shows a key distinction: expressivity of single-step threshold logic differs from expressivity of networks with time and feedback.

Examples:

Function capability: MCP networks can implement AND/OR/NOR but not XOR.
MCP neurons can simulate finite state machines and, with infinite tape, Turing machines.

✓ Check Your Understanding:

Why can MCP neurons represent AND but not XOR?

Answer: Because MCP neurons correspond to threshold logic that matches linearly separable functions, and XOR is not linearly separable

What additional ingredient helps MCP networks gain simulation power beyond single-step boolean logic?

Answer: Synchronous discrete-time updates with feedback/self-loops that create state evolution

Practice Activities

Bias shifts the decision boundary

medium

Consider a neuron with step activation. Let u=w1*x1+w2*x2+b. Choose b so that the output changes for a specific input pair. Use the cause-effect chain: adding bias shifts the effective threshold/decision boundary. Task: pick one input pair (x1,x2) and one competing input pair, then state how changing b would flip y for one pair but not the other.

Sigmoid vs ReLU: predict training difficulty

medium

Use the cause-effect chain: sigmoid in deep networks can lead to vanishing gradients, while ReLU improves training of deeper networks. Task: Given a deep multilayer network trained with backprop, predict which activation is more likely to cause gradient vanishing and explain the mechanism in one sentence.

From threshold logic to MCP firing

medium

Use the cause-effect chain: MCP neuron operates synchronously with discrete time steps and thresholded firing, enabling simulation with memory. Task: Describe what happens to y(t+1) when excitatory inputs meet the threshold but an inhibitory input is firing, and relate it to the step/threshold intuition you learned earlier.

Linear activation collapse check

easy

Use the cause-effect chain: linear activation implies affine transformations. Task: Suppose a two-layer network uses only linear activations. State the effect on expressivity compared to a single-layer network, and connect it to why nonlinearity is needed for multilayer advantages.

Next Steps

Cheat Sheet

Cheat Sheet: Artificial Neurons and Activation Functions

Key Terms

Artificial neuron: A mathematical model of a biological neuron used as the basic unit of an artificial neural network.
Excitatory/inhibitory inputs: Inputs that respectively increase or decrease the neuron’s effective activation, analogous to dendritic potentials.
Synaptic weights: Parameters that scale each input’s influence on the neuron’s weighted sum.
Bias: A constant term (often implemented as an extra input x0=+1) that shifts the activation threshold.
Activation function (φ): A nonlinear function applied to the weighted sum to produce the neuron output.
Step function: A threshold activation that outputs 1 if u≥θ and 0 otherwise.
Sigmoid function: A smooth nonlinear function (e.g., logistic) with an easily computed derivative, but gradients can vanish in deep networks.
Rectifier / ReLU: An activation defined as f(x)=max(0,x), outputting the positive part of its input.
McCulloch–Pitts (MCP) neuron: A discrete-time, synchronous threshold neuron with binary inputs/outputs and a threshold b.
Threshold logic unit / linear threshold unit: A historical name for the threshold-based neuron model using a Heaviside step function.

Formulas

Artificial neuron output

y_k = φ(∑_{j=0}^m w_kj x_j)

Use for the standard neuron: weighted summation followed by a nonlinearity.

Bias as an extra input

Set x0=+1 and w_k0=b_k, so ∑_{j=0}^m w_kj x_j = b_k + ∑_{j=1}^m w_kj x_j

Use when you want to fold bias into the same summation as other inputs.

Step (threshold) activation

y = {1 if u ≥ θ; 0 if u < θ}

Use for threshold logic and perceptron-style binary decisions.

ReLU / rectifier activation

f(x)=max(0,x)

Use to understand piecewise linear behavior and improved deep training vs sigmoid.

MCP neuron synchronous update rule

y(t+1)=1 if (number of firing excitatory inputs ≥ threshold) AND (no inhibitory inputs are firing); otherwise y(t+1)=0

Use for MCP dynamics and reasoning about what the network can simulate over time.

Main Concepts

Artificial neuron as a computational unit

An artificial neuron is a function that takes weighted inputs and produces an output via an activation function.

Weighted sum with bias

Compute a weighted sum of inputs and typically add a bias term before applying a nonlinearity.

Activation function role

The activation function introduces nonlinearity and strongly affects learning and expressivity.

Step/threshold activation

Outputs 1 when input meets/exceeds a threshold and 0 otherwise, enabling hyperplane-based separation.

Linear activation (linear neuron)

Outputs an affine transformation; stacking only linear activations collapses to a single-layer equivalent.

Sigmoid activation and gradient behavior

Nonlinear and differentiable, but gradients can diminish through many layers (vanishing gradients).

Rectifier (ReLU) activation

f(x)=max(0,x); piecewise linear positive region improves gradient flow for deeper networks.

McCulloch–Pitts (MCP) neuron dynamics

Synchronous discrete-time threshold model with binary firing; supports simulation of dynamical systems and FSMs.

Expressivity limits and simulation power

MCP neurons represent linearly separable boolean functions (AND/OR/NOR) but not XOR; with time and memory they can simulate FSMs and, with infinite tape, Turing machines.

10.

Biological-to-math mapping

Dendrites correspond to weighted input effects, soma corresponds to summation, and axon corresponds to thresholded pulse transmission.

Memory Tricks

Bias shifting the decision boundary

Bias is the “Bump”: it bumps the threshold by adding b before φ.

Sigmoid vs ReLU gradient intuition

Sigmoid Sinks gradients; ReLU Rises gradients (positive region stays non-saturating).

MCP can do AND/OR/NOR but not XOR

MCP is “Linear-Only”: if it is not linearly separable, MCP cannot do it (XOR is the classic fail).

Bias as extra input

Add x0=+1 so b becomes just another weight-times-input term.

Activation function vs transfer function

Activation is the neuron’s nonlinearity φ; transfer function is a different linear-systems concept—do not swap the names.

Quick Facts

Standard neuron: weighted summation then activation: y_k = φ(∑_{j=0}^m w_kj x_j).
Bias can be implemented as x0=+1 with weight w_k0=b_k.
Step activation is the Heaviside-style threshold rule used in perceptrons and threshold logic units.
Sigmoid can cause vanishing gradients in deep backprop.
ReLU is f(x)=max(0,x) and was shown to enable better training of deeper networks.
MCP update is synchronous and thresholded: firing depends on excitatory count vs threshold and absence of inhibitory firing.
MCP neurons can do AND/OR/NOR but not XOR.
With feedback over time, MCP networks can simulate finite state machines; with infinite tape, they can simulate Turing machines.

Common Mistakes

Common Mistakes: Artificial Neurons and Activation Functions

Confusing the neuron activation function with a linear-systems transfer function, and then treating the neuron as if it were a linear time-invariant filter.

conceptual · high severity

▼

Why it happens:

Students see the word "function" and the symbol-like role of φ, then map it to the transfer function idea from signal processing (output equals transfer function times input). They also notice that the neuron output is computed from an input and assume the mapping must be linear and system-theoretic, rather than a nonlinear parameterized transformation used for expressivity.

✓ Correct understanding:

An artificial neuron computes a weighted sum (plus bias) and then applies an activation function: y_k = φ(∑_{j=0}^m w_kj x_j). The activation function is chosen to introduce nonlinearity (and possibly differentiability, boundedness, or threshold behavior). In multilayer networks, this nonlinearity is what prevents the whole network from collapsing into a single linear mapping.

How to avoid:

Always write the computation in the neuron form y = φ(u) with u = ∑ w x + b. Then ask: "Is φ introducing nonlinearity?" If your reasoning never explicitly mentions nonlinearity and the weighted-sum-plus-bias structure, you are likely mixing up neuron activation with linear transfer-function thinking.

Believing an MCP (McCulloch–Pitts) neuron can compute XOR using a threshold rule.

conceptual · high severity

▼

Why it happens:

Students know that XOR is a classic example of a problem requiring nonlinearity or multiple layers, but they overgeneralize from the fact that threshold units can implement some boolean logic. They then assume that because XOR is a boolean function, a threshold neuron should be able to implement it directly.

✓ Correct understanding:

MCP neurons are threshold-based and can represent linearly separable boolean functions such as AND, OR, and NOR. XOR is not linearly separable, so a single MCP neuron cannot implement XOR. However, MCP networks with multiple neurons can still simulate more complex computations, including finite state machines, and with enough structure can reach Turing-complete simulation power.

How to avoid:

Use the separability test mentally: a single threshold neuron corresponds to a hyperplane decision boundary. If the positive and negative examples cannot be separated by one hyperplane, then XOR is impossible for one MCP neuron. Only then consider multi-neuron constructions.

Treating bias and MCP threshold as the same parameter in a way that leads to incorrect equations and incorrect intuition about shifting decision boundaries.

conceptual · medium severity

▼

Why it happens:

Students see both "bias" and "threshold" in different neuron formulations and assume they are literally the same symbol and appear in the same place in every model. They then substitute bias into the MCP firing rule (or substitute MCP threshold into the weighted-sum-plus-bias formula) without aligning the model definitions.

✓ Correct understanding:

In the weighted-sum neuron, bias shifts the input to the activation: u = ∑_{j=1}^m w_j x_j + b, which can be implemented by adding an extra input x0 = +1 with weight w0 = b, giving u = ∑_{j=0}^m w_j x_j. In the MCP neuron, the firing rule uses a threshold b directly in the discrete-time update: y(t+1)=1 if the excitatory firing condition meets the threshold and no inhibitory inputs are firing; otherwise y(t+1)=0. Bias and MCP threshold are analogous in effect (shifting the decision boundary), but they appear in different mathematical roles depending on the model form.

How to avoid:

Before comparing parameters, rewrite both models in their own canonical forms. Then compare effects: "Does increasing this parameter make firing more or less likely?" If you cannot answer that without rewriting the equations, you are likely conflating the symbols across model types.

Assuming that using linear activation in a multilayer network still provides nonlinear expressivity, so multilayer networks with linear activations can solve problems that require nonlinearity.

conceptual · high severity

▼

Why it happens:

Students remember that neural networks are powerful and think "stacking layers" automatically increases expressivity. They then overlook the specific relationship: if every layer is linear (affine), the composition remains affine, so the entire multilayer network collapses to a single equivalent linear transformation.

✓ Correct understanding:

With linear activation, each layer performs an affine transformation. Composing affine maps yields another affine map. Therefore, a multilayer perceptron with purely linear activations has an equivalent single-layer form and does not gain the expressivity advantages that come from nonlinear activations.

How to avoid:

Apply a quick algebra check: write one layer as y = A x + c (or u = w·x + b). Then compose two layers and verify the result is still affine. If your reasoning does not reduce to affine composition, you are likely missing the linear-collapse argument.

Thinking sigmoid activations are only "smooth" and "good" because they are differentiable, and therefore they do not cause optimization problems in deep networks.

conceptual · high severity

▼

Why it happens:

Students focus on differentiability because backpropagation needs derivatives. They then assume that because derivatives exist and are easy to compute, training should be stable. They may also confuse "derivative exists" with "derivative stays large enough" across many layers.

✓ Correct understanding:

Sigmoid activations are nonlinear and differentiable, but in deep multilayer networks the backpropagated gradients tend to diminish toward zero. This happens because derivatives used in backpropagation shrink across layers (vanishing gradients), making optimization difficult.

How to avoid:

When you choose an activation, explicitly connect it to gradient flow. Ask: "What happens to derivatives across many layers?" For sigmoid, remember the vanishing-gradient effect during backpropagation. Then contrast with rectifier behavior (ReLU) which improves training in deeper networks.

Misstating the rectifier (ReLU) activation as a sigmoid-like squashing function or as outputting 1 for positive inputs and 0 otherwise (a step), rather than outputting max(0,x).

conceptual · medium severity

▼

Why it happens:

Students conflate multiple "nonlinear" activations: they remember that step functions are thresholded and produce binary outputs, and they also remember that sigmoid outputs are bounded between 0 and 1. Then they incorrectly generalize that ReLU must also be bounded or binary, because it is also used for training improvements.

✓ Correct understanding:

Rectifier (ReLU) is defined as f(x) = max(0, x). It is piecewise linear: it outputs 0 for negative inputs and outputs the input itself for positive inputs. This non-saturating positive region helps gradient flow compared to sigmoids, making deeper training easier and more effective.

How to avoid:

Memorize the defining formula and test it on two values: one negative and one positive. If your mental model cannot quickly produce f(-1)=0 and f(1)=1, you are mixing up ReLU with step or sigmoid.

Assuming all activation functions are monotonic and differentiable, and therefore ruling out threshold-like or piecewise behaviors as "invalid" activations.

conceptual · medium severity

▼

Why it happens:

Students internalize a simplified rule: "activation functions must be smooth and monotone" because many common examples (sigmoid, tanh-like shapes) are monotonic and differentiable. They then treat non-differentiability or non-monotonicity as a conceptual error rather than a modeling choice.

✓ Correct understanding:

Activation functions can be step functions (threshold logic), which are not differentiable at the threshold and are not continuous. Rectifiers (ReLU) are piecewise linear and not differentiable at 0. More broadly, the text notes that non-monotonic, unbounded, oscillating activations with multiple zeros have also been explored. The key is to match the activation to the modeling goal (threshold logic, gradient-based learning, expressivity, or hardware constraints).

How to avoid:

Classify activations by properties you actually need: continuity, differentiability, boundedness, monotonicity, and whether you want threshold logic. Do not assume "differentiable" is a universal requirement; instead, connect the property to the learning method and the model type.

General Tips

Always start from the canonical neuron computation: weighted sum plus bias, then apply φ to get the output.
When comparing two neuron models (standard vs MCP), rewrite both in their canonical forms before mapping parameters.
Use quick sanity checks: test activations on a negative and a positive input; test expressivity claims with linear separability reasoning.
For training-related claims, connect the activation choice to gradient behavior across depth (not just differentiability).

Content

AI Audio

Test Yourself

Practice

Games

Summary

Topics Covered

Artificial Neurons as Computational Units and Biological Inspiration

Mathematical Neuron Core: Weighted Summation and Bias

Activation Functions: Role, Properties, and Output Formation

Activation Function Families: Step, Linear, Sigmoid, and Rectifier (ReLU)

McCulloch–Pitts (MCP) Neuron and Threshold Logic Dynamics

Expressivity and Limits: Linearly Separable Functions vs XOR; FSM and Turing Power

History, Learning Rules, and the Activation vs Transfer Confusion

Physical Artificial Neurons and Neuromorphic Hardware Implementations

Key Insights

Bias as boundary shift

Linear activations collapse layers

Sigmoid gradients shrink by design

ReLU avoids saturation cascade

MCP memory comes from time

Conclusions

Bringing It All Together

Key Takeaways

Real-World Applications

💻 Code Examples

McCulloch–Pitts Neuron: synchronous threshold logic with excitatory and inhibitory inputs

Code

Explanation

Use Case

Output

💻 Code Practice Problems

Bias as a fixed +1 input: linear combination neuron with step activation for binary classification

Code

Explanation

Use Case

Output

💻 Code Practice Problems

Sigmoid neuron vs ReLU-like rectifier: compare nonlinearities on the same weighted sum

Code

Explanation

Use Case

Output

💻 Code Practice Problems

One-step spiking-style update using threshold logic: event-driven state machine from MCP outputs

Code

Explanation

Use Case

Output

💻 Code Practice Problems

Interactive Lesson

Interactive Lesson: Artificial Neurons and Activation Functions

Learning Objectives

1. Weighted summation with bias

2. Activation function (φ) and neuron output

3. Types of activation functions (step, linear, sigmoid, rectifier)

4. Artificial neuron as a biological model (dendrites, soma, axon mapping)

5. McCulloch–Pitts (MCP) neuron dynamics and threshold logic

6. Expressivity limits (linearly separable vs XOR) and simulation power (FSM/Turing)

Practice Activities

Bias shifts the decision boundary

Sigmoid vs ReLU: predict training difficulty

From threshold logic to MCP firing

Linear activation collapse check

Next Steps

Cheat Sheet

Cheat Sheet: Artificial Neurons and Activation Functions

Key Terms

Formulas

Main Concepts

Memory Tricks

Quick Facts

Common Mistakes

Common Mistakes: Artificial Neurons and Activation Functions

General Tips