Fraud Detection RL Environment

Multi-turn reinforcement learning environment for investigating fraudulent sessions. Agents gather evidence across signals (IP patterns, device fingerprints, login frequency, geo anomalies) and issue verdicts before investigation costs accumulate.

Problem

Fraud detection requires reasoning under uncertainty. Investigators don't make snap judgments—they gather evidence, weigh signals, and balance investigation cost against confidence. This environment trains agents to:

• Reason about suspicious patterns in app logs
• Handle adversarial fraud that mimics legitimate behavior
• Make decisions with incomplete information
• Optimize investigation efficiency (max 3-8 steps per session)

How It Works

Session Logs → Agent Investigates Signals → Agent Issues Verdict → Reward

Agent can:

• Investigate signals: IP velocity, device fingerprint, login frequency, geo anomaly, request patterns
• Issue verdict: Fraud or legitimate

Each investigation reveals ground-truth data. Correct verdicts earn high reward. Missed fraud heavily penalized.

Setup

Prerequisites: Python 3.10+, PyTorch

git clone https://github.com/pulkit300405/fraud-detection-env
cd fraud-detection-env
pip install -r requirements.txt
python server.py

Server runs at http://localhost:8000

API Usage

Reset Episode

curl -X POST http://localhost:8000/reset \
  -H "Content-Type: application/json" \
  -d '{"task": "explain-subtle", "difficulty": "medium"}'

Returns session logs and available investigation actions.

Take Action

curl -X POST http://localhost:8000/step \
  -H "Content-Type: application/json" \
  -d '{
    "action": "investigate:ip_velocity",
    "reasoning": "Session shows rapid logins from different IPs"
  }'

Tasks

Task	Difficulty	Max Steps	Description
flag-obvious	Easy	3	Clear fraud signals (multiple IPs, countries, rapid logins)
explain-subtle	Medium	5	Mixed signals requiring multi-step reasoning
adversarial-hunt	Hard	8	Sophisticated fraud mimicking legitimate patterns

Action & Observation

Agent Actions:

• investigate:ip_velocity — Check unique IPs
• investigate:device_fingerprint — Check device variety
• investigate:login_frequency — Check login attempt count
• investigate:geo_anomaly — Check country spread
• investigate:request_pattern — Check suspicious endpoints
• verdict:fraud — Issue fraud verdict
• verdict:real — Issue legitimate verdict

Agent Observes:

• Session ID and logs
• Signals already investigated
• Available actions
• Current step count
• Task difficulty
• Reward for last action

Reward Function

Event	Reward
Investigate new signal	+0.1
Correct verdict (fraud)	+2.0
Correct verdict (legitimate)	+2.0
Missed fraud	-3.0
False alarm	-0.5
Duplicate investigation	-0.05
Invalid action	-0.1

Strategy: Maximize reward by investigating smartly, then issuing correct verdicts.

Baseline Results

Using Qwen-72B at temperature 0.0:

Task	Score
flag-obvious	1.85
explain-subtle	1.10
adversarial-hunt	0.75

Running Agent Inference

export OPENAI_API_KEY=your_key
export MODEL_NAME=gpt-4
python inference.py

Agent uses ReAct loop:

1. Observe session logs
2. Reason about fraud signals
3. Decide: investigate or verdict
4. Receive reward + next observation
5. Repeat until verdict issued

Project Structure

fraud-detection-env/
├── server.py              # FastAPI environment
├── models.py              # Signal generation + grading
├── inference.py           # Agent inference loop
├── client.py              # Test client
├── requirements.txt
├── pyproject.toml
├── openenv.yaml
└── tests/
    ├── test_env.py
    ├── test_signals.py
    └── test_inference.py

Signal Generation

Fraud signals generated synthetically:

• IP velocity — Count of unique IPs per session
• Device fingerprint — Count of unique devices
• Login frequency — Login attempts in time window
• Geo anomaly — Countries accessed from
• Request pattern — Suspicious endpoint sequences

Each signal designed to detect specific fraud types.

Key Implementation

Multi-turn reasoning: Agents investigate iteratively. Each investigation reveals ground-truth signal values. Agents learn to combine signals for verdicts.

Difficulty scaling: Easy tasks have obvious patterns. Hard tasks require adversarial reasoning against sophisticated fraud.

Deterministic grading: All verdicts evaluated via heuristic matching (regex + logic) not LLM-as-judge. Ensures reproducibility and speed.

Tech Stack

• Backend: Python 3.10+, FastAPI
• Framework: PyTorch
• API: OpenEnv spec
• Testing: pytest

Author

Pulkit Singh — Environment design, signal generation, server API, agent inference, Docker setup.

GitHub: @pulkit300405