Deep Learning Deep Reinforcement Learning

Most networks learn from examples with correct answers. Reinforcement learning works differently. An agent learns by trying actions, seeing the results, and collecting rewards. Deep reinforcement learning adds a neural network to help the agent handle complex situations such as video games, robot control, and resource planning.

Learning Like a Puppy

Imagine training a puppy to sit. The puppy tries behaviours at random. A treat follows the right behaviour, and nothing follows the others. Over time the puppy repeats what earns treats. A reinforcement learning agent follows the same pattern with numeric rewards.

The Main Parts

• Agent - the learner that makes decisions
• Environment - the world the agent acts in
• State - a description of the current situation
• Action - a choice the agent can make
• Reward - a score that tells the agent how good the last action was

        action
Agent ------------> Environment
  ^                      |
  |   new state          |
  +----------------------+
  |   reward             |
  +----------------------+

A Simple Example: Maze Game

State:   the agent's position in the maze
Actions: up, down, left, right
Rewards: +10 for reaching the exit
         -1 for every step (encourages short paths)
         -5 for hitting a wall

Goal: collect the highest total reward

Policy and Value

A policy is the agent's strategy. It maps each state to an action. A value estimate predicts how much future reward a state or action will bring. Good agents learn one or both.

Policy: "In this state, move right."
Value:  "Moving right from here will probably earn 7 points in total."

Exploration and Exploitation

The agent faces a constant choice. Exploitation repeats the best known action. Exploration tries something new that might work better. Picture choosing a restaurant: you can return to a favourite or test a new place. Agents often explore heavily at first and reduce exploration over time.

Where Deep Learning Fits

Tables of values work for tiny problems. A video game has millions of possible screens, so a table cannot hold them all. A neural network takes the state as input and estimates values or actions, even for situations the agent never saw before.

Deep Q-Network (DQN)

A DQN estimates the value of every action for a given state. The agent picks the action with the highest estimate. Experience replay stores past moves and trains on random samples from that memory, which stabilizes learning.

State (game screen) -> Neural network -> Action values
                                         left  = 2.1
                                         right = 5.4  <- choose
                                         jump  = 3.0

Policy Gradient Methods

These methods train the network to output action probabilities directly. Actions that led to high rewards become more likely.

Actor-Critic Methods

Two networks cooperate. The actor chooses actions, and the critic scores them. Algorithms such as PPO belong to this family and enjoy wide use because of their steady training.

Training Loop

1. Observe the current state
2. Choose an action with the network (sometimes explore)
3. Receive a reward and a new state
4. Store the experience
5. Update the network from stored experiences
6. Repeat for thousands of episodes

Applications

• Game-playing agents
• Robot arms that learn to grasp objects
• Energy management in data centres and buildings
• Traffic signal timing
• Fine-tuning language models with human feedback

Challenges

• Training needs a very large number of trials
• Poorly designed rewards teach unintended behaviour
• Real-world mistakes cost money, so engineers train in simulators first
• Results vary between runs, so test with several random seeds

Key Terms

• Reward - the score that guides learning
• Policy - the agent's strategy for choosing actions
• Exploration - trying new actions
• Exploitation - repeating the best known action
• Episode - one complete run from start to finish