This project is actively in development! If you have any questions or want to help out, you can reach out via issues or discussions, or send me an email.
PromptForest Ranger (or PFRanger) is a CLI tool for auditing LLM prompts for injection attacks and jailbreaks. It leverages the PromptForest ensemble engine to scan datasets and generate comprehensive and accurate HTML reports, mitigating the problem of high-certainty false positives.
We intend this tool to be a swiss army knife: it should be used for multiple situations regarding unsafe prompts. Here are some, but not all, of the ways to use PFRanger
-
Processing Unsafe Datasets Huge datasets may contain prompts that weaken a model, making it more vulnerable to jailbreak attempts. PFRanger can mitigate this issue before you feed it to your model.
-
Auditing Datasets Feeding potentially dangerous datasets into AI model with highly privileged abilities may produce catastrophic results. PFRanger helps by giving you an overview of the dataset's security.
All data stays on your computer.
PFRanger utilises the PromptForest engine, which provides responses that are not only accurate, but also calibrated and reliable. PromptForest uses a strategically selected ensemble of small models using soft weighted model voting. Read more about PromptForest here
Startup latency of ~3s and an average request rate of 27 prompts/s on a consumer GPU with optimal parallelization.
Supports a large number of input and output file types. Also outputs a shareable HTML file to help you analyse the data.
PromptForest automatically handles all model downloading on first run. Downloading takes around 3GB disk space and around a minute to download on consumer network download speeds.
A simple quick start guide is provided below. You can view the official wiki in GitHub Pages, and the DeepWiki version here for those interested in how Ranger works under-the-hood.
pip install pfrangerpfranger [INPUT_FILE] [OPTIONS]Scan a CSV file:
pfranger validation_prompts.csv -p textScan with custom configuration:
pfranger prompts.jsonl -c my_config.yaml| Argument | Description | Default |
|---|---|---|
input |
Path to the file containing prompts | Required |
--output, -o |
Output path for HTML report | report.html |
--input-format, -f |
Force input format (csv, json, jsonl, txt) | Auto-detect |
--output-format |
Add additional output format alongside HTML | None |
--prompt-col, -p |
Column name for prompts (CSV/JSON) | prompt |
--timestamp-col, -t |
Column name for timestamp | None |
--workers, -w |
Number of parallel worker threads | 4 |
--config, -c |
Path to PromptForest YAML configuration file | Benchmark Defaults |
See LICENSE.