Snake Game + DQN Reinforcement Learning Agent

Course: CSCI 495, Deep Learning, Spring 2025

Assignment: Snake_DQN

Assignment Intent

An iterative, multi-week group project ("investigate a project topic, chosen from the Deep Learning domain") culminating in a final report covering Introduction, Background/literature search, Experiment Design, Experiment Procedure, Results and Discussion, Conclusions, and References. Per the author, the chosen topic was a Deep Q-Network (DQN) reinforcement-learning agent, with this Snake game serving as its training environment -- the game itself was a means to that end, not the graded focus. The final report is not currently available to verify the literature review/experiment design/results against; this section will be updated once it's located.

Update: the DQN agent has since been added. An earlier pass over this repository found only the game environment, with get_state()/ get_danger_state() built but never consumed by any training code -- a real, confirmed gap between the project's stated focus and what the repository held at the time. notebooks/CSCI495_Project_DQN.ipynb has since been added and closes that gap: it implements the DQN agent (model, replay buffer, target network, training loop) and consumes get_danger_state() as its state representation, exactly matching what the earlier scaffolding was built for. See Overview and Known Issues below for what's now confirmed working versus what's still open.

Overview

This repository has two parts: the game environment, and the DQN agent that trains against it.

src/snake.py (279 lines, pygame) is the environment: a grid-based game board, snake movement, wall and self-collision detection, random food placement, on-screen rendering (snake, food, score), and a keyboard-driven game loop for human play (SnakeGame.play_human()). It also exposes a programmatic interface the DQN notebook drives directly: update(direction) (step the simulation without going through pygame's event loop), is_valid_direction(), get_danger_state() (an 11-element feature vector: danger straight/right/left, current-direction one-hot, food-direction flags), and reset(). SnakeGame(..., display_game=False) skips pygame.init()/window creation entirely, so the environment can run headless for training.

• SnakeGame(width=20, height=20, cell_size=20, fps=10, display_game=True, seed=None) configures the board.
• Running the file directly (python src/snake.py) starts a 30x30-cell game window controlled with the arrow keys.

notebooks/CSCI495_Project_DQN.ipynb (22 cells, TensorFlow/Keras) is the DQN agent: a build_dqn_model() MLP (Dense 128 -> Dense 64 -> Dense 4, taking the 11-element get_danger_state() vector as input), a DQN class implementing epsilon-greedy action selection, an experience replay buffer, a separate target network (synced every 50 training steps), and Huber loss with gradient clipping; a reward function (RewardConfig) that rewards eating food and closing the distance to it, penalizes dying (more heavily for an early death), and adds small survival bonuses at 100/200/300 steps; a training loop (up to 50 episodes, capped at 600 steps each, with early stopping once the agent consistently reaches a fruit-count goal); and an evaluation pass (evaluate_agent/analyze_evaluation, 25 greedy-policy runs) reporting average fruits eaten, average steps survived, and a success-rate breakdown against fruit/step thresholds.

Dependencies

requirements.txt covers both the game and the notebook (pip install -r requirements.txt):

Package	Used for	Used by
pygame	Window, rendering, input handling, clock	src/snake.py
numpy	Array-based state representations, evaluation statistics	both
tensorflow	keras model/training (build_dqn_model, DQN)	notebook
sys, random, enum, collections.deque, time	standard library	both

Running the notebook itself also requires Jupyter/nbconvert (pip install jupyter nbconvert ipykernel), which is not in requirements.txt since it's a way to run the notebook rather than a dependency of the code itself -- see Continuous Integration below for how CI installs it.

No version constraints are pinned in requirements.txt.

Environment Setup

python -m venv venv
# Windows
venv\Scripts\activate
# macOS/Linux
source venv/bin/activate

pip install -r requirements.txt

Run the game (human play):

python src/snake.py

Controls: arrow keys to steer, close the window (or press any key after game over) to exit.

Run the DQN notebook:

jupyter notebook notebooks/CSCI495_Project_DQN.ipynb

Run top to bottom. The notebook's setup cell adds ../src (relative to the notebook's own directory, which is where Jupyter/nbconvert set the kernel's working directory) to sys.path so src/snake.py can be imported directly -- no separate setup is needed. (The notebook also contains a fallback that clones this repository if src/snake.py isn't found locally, for the case of opening just the notebook standalone in Google Colab; that path is skipped entirely when running from a full checkout, including CI.) Training runs headless (display_game=False), so no display or SDL driver configuration is needed either locally or in CI.

Continuous Integration

A GitHub Actions workflow (.github/workflows/ci.yml) runs on every push and can also be triggered manually via workflow_dispatch, with two jobs:

1. build -- installs requirements.txt, compile-checks src/snake.py, then runs it for 5 seconds under SDL_VIDEODRIVER=dummy as a bounded smoke test (see the inline comment in ci.yml for why a full run isn't possible: play_human() blocks on real keyboard input with no non-interactive mode).
2. train-dqn-notebook -- installs requirements.txt plus jupyter/nbconvert, then executes notebooks/CSCI495_Project_DQN.ipynb end to end with nbconvert: the full DQN training run (up to 50 episodes) and the 25-run evaluation, not a reduced or mocked version. This is a real training job, not a syntax check, so it takes noticeably longer than the build job; a 120-minute timeout-minutes is a safety net against a genuinely stuck run, not an indication anything is wrong if it takes a while. The executed notebook (with all outputs) is uploaded as a downloadable build artifact; output is written to a fresh, git-ignored path rather than --inplace, so this job never modifies the committed notebook.

Known Issues

Dead Code -- mostly resolved now that the DQN notebook consumes it

An earlier pass over this repository (before notebooks/CSCI495_Project_DQN.ipynb was added) found the following functions defined but never exercised by the playable game loop, reading as leftover scaffolding for an RL agent that hadn't been built yet. Five of the six are now genuinely used by the DQN notebook; one remains dead.

1. get_danger_state() -- src/snake.py:124-169 -- Now used. The notebook's get_state(game) wrapper calls this directly as the DQN's input representation (build_dqn_model(input_dim=11, ...) matches its 11-element output exactly).

2. check_danger(direction) -- src/snake.py:171-185 -- Now used (indirectly, via get_danger_state()).

3. get_right_direction() -- src/snake.py:187-195 -- Now used (indirectly, via get_danger_state()).

4. get_left_direction() -- src/snake.py:197-205 -- Now used (indirectly, via get_danger_state()).

5. get_state() -- src/snake.py:106-122 -- Still dead. Builds a (height, width, 3) grid-shaped state representation -- a second, different encoding than get_danger_state(). The DQN notebook uses only get_danger_state(); this method is still only ever called once, from reset(), where the return value is discarded. If the grid-shaped representation isn't going to back a different model architecture (e.g. a CNN), it can be removed along with the return in reset().

6. self.steps -- src/snake.py:31,56,94,98 -- Now used. The notebook's calculate_reward() reads game.steps to award survival bonuses at 100/200/300 steps.

Fixed while wiring up CI for the notebook

• Self-cloning cell assumed a Colab environment. The notebook's setup cell unconditionally ran !git clone https://github.com/dagron27/Snake_DQN.git (the repository's old, pre-rename name) and appended a mismatched path (./Snake_DQN, not the ./snake-game-dqn-simulation folder name an actual clone of that URL would produce) to sys.path -- harmless in Colab where the notebook is opened standalone, but unnecessary (and pointed at a stale URL) when running from within a full checkout of this repository, including CI, where src/snake.py is already reachable via ../src relative to the notebook. The cell now checks whether ../src/snake.py is already present locally and skips cloning entirely in that case, only falling back to cloning (from the correct, current repository name) if it isn't.

Worth a second look, not changed

• replay(batch_size) is called twice per environment step in the training loop (dqn.replay(BATCH_SIZE) appears back-to-back). This doubles the number of gradient updates per step relative to a more typical one-replay-per-step DQN loop. It isn't broken -- training converges regardless -- but it's not commented, so it's unclear whether it's a deliberate choice to train more aggressively per step or a copy-paste duplication. Left as-is since changing training dynamics wasn't part of making the notebook run correctly; worth clarifying with whoever wrote it.

Security

src/snake.py itself has no findings: no network calls, no file I/O, no eval/exec/pickle/subprocess, no credentials, no input surface beyond pygame keyboard events.

The notebook's setup cell does shell out to git clone (via a Jupyter ! magic) when src/snake.py isn't already present locally -- this only fires in the Colab-standalone fallback path (see Environment Setup above), never when running from a full checkout including CI, and the URL is a hardcoded literal pointing at this repository, not user-controllable input. Training and evaluation involve no file I/O either -- the notebook doesn't call model.save() anywhere, so no trained model artifact is persisted or reloaded, and consequently there's no model-deserialization trust boundary to assess here (unlike the portfolio's other Keras-based projects, which do save .h5 files). No pickle/joblib usage of any kind.

No meaningful attack surface beyond what's noted above.

Status

The Snake game itself is playable as-is. The DQN agent (notebooks/CSCI495_Project_DQN.ipynb) is implemented and confirmed working: after fixing the notebook's setup cell (see Known Issues), a reduced verification run (3 training episodes, 3 evaluation runs) executed end to end with no errors, confirming the game environment, the model, the training loop, and the evaluation pass all function correctly together.

The full 50-episode training run's exact results are not yet available. A full local run was attempted and did not complete within 30 minutes -- based on the timing observed in the reduced run (roughly 150ms per environment step, and each step makes up to five model predict/fit calls -- one for action selection plus two full replay passes), the full run likely takes something in the neighborhood of 60-90+ minutes locally. CI's train-dqn-notebook job now allows up to 120 minutes and uploads the executed notebook (with full training/ evaluation output) as a downloadable artifact once it completes -- that artifact is the source for real fruit/step/success-rate numbers, not this section, until it's been checked.

Contributions

This was a group project for CSCI 495/595. Per the author, the other group members' names and specific roles were not preserved, so this section can't give the file-by-file breakdown this portfolio's other group-project repositories do. This repository is hosted and maintained by Daniel Leone as his own record of the project -- the code here (src/snake.py, notebooks/CSCI495_Project_DQN.ipynb, and this documentation) reflects his own work on the assignment. See LICENSE for how this affects licensing.