The Maze Runner
Explore reward shaping and policy gaps in a DQN maze agent to reach a forbidden state.
You think running a maze is hard? How about finding the exit inside you.
Overview
You’re given a pre-trained reinforcement learning (RL) agent (a DQN) that navigates a 12×12 maze. Under normal play, it moves from the start (0,0) to the goal (11,11) while avoiding a forbidden state at (6,6).
Your mission: exploit flaws in the agent and/or environment to force the agent into (6,6), which reveals the flag.
- Challenge Type: RL exploitation / reward hacking
- Difficulty: Hard
- Flag:
IEEE{rl_reward_hacking_forbidden_6_6}
Phase 1 — Understand the Maze Environment
What to do (explanation only)
- State space: 12×12 grid (144 positions).
- Action space: 4 moves (Up=0, Right=1, Down=2, Left=3).
- Rewards:
0.1per step+100when reaching the goal(11,11)- Special quirks: “bonus” rewards when certain hidden sequences occur.
- Forbidden state:
(6,6)contains the flag, but the agent avoids it under normal policy.
Key observations
- Reward shaping flaws → Circular movement patterns yield unintended cumulative rewards.
- Policy gaps → The agent was undertrained near
(6,6). - Environment exploits → Specific action sequences let you phase through walls.
Phase 2 — Analyze Agent Behaviour
What to do (explanation only)
- Run the demo (
python maze_agent_demo.py) and watch normal navigation. - Use policy visualization to see action preferences at each state.
- Notice how the agent consistently avoids the central forbidden zone.
- Look for states where Q-values are unstable → training blind spots.
Phase 3 — Develop Exploits
1. Reward Hacking (circular moves)
- Repeated sequences (e.g.,
Up, Right, Down, Left) trick the reward function. - The agent accrues enough “neutral” reward to push into risky states.
2. Phase-through sequence
- Some sequences (e.g., rapid alternating Left/Right at walls) bypass obstacles.
- This lets you guide the agent toward
(6,6)even if the direct path is blocked.
3. Direct navigation into forbidden state
- Once close to
(6,6), craft a precise sequence that exploits policy gaps. - Because the agent lacks experience here, adversarial inputs can steer it inside.
Code snippet — guided sequence execution
from forbidden_maze_env import ForbiddenMazeEnv
from dqn_agent import DQNAgent
import torch
# Load environment and agent
env = ForbiddenMazeEnv()
agent = DQNAgent(state_size=144, action_size=4, hidden_size=128)
agent.qnetwork_local.load_state_dict(torch.load("vulnerable_agent.pth"))
agent.qnetwork_local.eval()
# Force agent with crafted sequence
actions = [1,1,2,2,3,0,0,2] # Example: Right, Right, Down, Down, ...
state, _ = env.reset()
for a in actions:
state, reward, done, _, _ = env.step(a)
env.render()
if done:
break
Phase 4 — Reach the Forbidden State
Success condition
When the crafted exploit guides the agent into (6,6), the environment prints:
🚩 FORBIDDEN STATE REACHED! 🚩
Flag: IEEE{rl_reward_hacking_forbidden_6_6}