Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 17 additions & 5 deletions OPERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ Small-account BTC paper rehearsal:
python -m bot.profile_runner paper research btc
```

Both `spy` (QQQ) and `btc` research runs now use $150 starting equity to match the real account.

Outputs:

- `reports/research_latest.md`
Expand Down Expand Up @@ -111,26 +113,36 @@ Useful optional env controls:

## Normal Bot Run

Run one BTC bot cycle:
Run one bot cycle (equity profile, the default live deployment target):

```powershell
docker compose run --rm paper
docker compose run --rm trade
```

BTC variants:

```powershell
docker compose run --rm paper-btc
docker compose run --rm trade-btc
```

Direct profile runner equivalents:

```powershell
python -m bot.profile_runner paper trade spy
python -m bot.profile_runner live trade spy
python -m bot.profile_runner paper trade btc
python -m bot.profile_runner live trade btc
```

Profile-specific validation:

```powershell
python -m bot.profile_runner paper validate spy
python -m bot.profile_runner live validate spy
python -m bot.profile_runner paper validate btc
python -m bot.profile_runner live validate btc
python -m bot.profile_runner live validate spy
```

Current runtime defaults:
Expand All @@ -141,15 +153,15 @@ Current runtime defaults:
- end-of-day flattening starts `5` minutes before the close by default; override with `FLATTEN_BEFORE_CLOSE_MINUTES`
- `paper` and `trade` select separate Alpaca key pairs from `.env` when `ALPACA_PAPER_*` and `ALPACA_LIVE_*` variables are set
- BTC paper writes runtime artifacts under `runtime/paper_btc`, BTC live under `runtime/live_btc`
- The BTC paper profile mirrors the live BTC signal filters, with larger paper sizing for rehearsal
- `config/paper_btc.env` is an exact mirror of `config/live_btc.env` (same sizing, same filters) so paper fills validate the strategy that actually runs live
- `reports/monitor_latest.md` includes 24h/7d rejection counts, near-miss entry bars, and latest filter metrics for diagnosing quiet BTC live periods
- `config/live_spy.env` enables fractional, long-only SPY sizing for small live accounts; replay it before relying on it live.
- `config/live_spy.env` trades QQQ on hourly bars (fractional, long-only, ~90% notional) and is the recommended default live profile for a small account; see `docs/strategy_revamp_2026-07.md` for the replay evidence. Replay any change to it before relying on it live.

## Notes

- Start Docker Desktop before using the commands above.
- Keep real Alpaca keys only in the local untracked `.env`.
- Alpaca's paper account balance still needs to be adjusted in the dashboard. The repo now mirrors that target by using roughly `$250` as the paper research starting equity.
- Alpaca's paper account balance still needs to be adjusted in the dashboard. The repo now mirrors that target by using `$150` as the paper research starting equity (see `RESEARCH_STARTING_EQUITY` in `config/paper_spy.env` and `config/paper_btc.env`).
- Review `reports/monitor_latest.md` after each trading session if you want a concise explanation of what the bot did and why.

## EC2 Deploy
Expand Down
104 changes: 27 additions & 77 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Trading Bot

Trend-following trading bot for Alpaca, supporting both intraday equities (SPY) and 24/7 crypto (BTC/USD). Uses a trend-confirmed moving-average strategy with ADX, ATR, volume, momentum, and regime filters, layered risk controls, and multiple exit mechanisms.
Trend-following trading bot for Alpaca. The default live/paper deployment is an hourly QQQ trend strategy sized for a small account; a defensive BTC/USD profile is also available. Uses a trend-confirmed moving-average strategy with ADX, ATR, volume, momentum, and regime filters, layered risk controls, and multiple exit mechanisms.

It currently:

Expand Down Expand Up @@ -133,7 +133,7 @@ Run the walk-forward optimizer:
docker compose run --rm optimize
```

Direct BTC live tuning equivalent:
BTC equivalent:

```powershell
python -m bot.profile_runner live optimize btc
Expand All @@ -156,88 +156,35 @@ Main outputs:
- `reports/optimize_latest.md`
- `reports/optimize_latest.json`

For BTC live tuning, the optimizer compares candidates against the loaded live baseline over the same bars. A candidate is marked accepted only if it clears the full replay, walk-forward, baseline-improvement, and 2x-slippage checks in the report.
The optimizer compares candidates against the loaded live baseline over the same bars, for either market. A candidate is marked accepted only if it clears the full replay, walk-forward, baseline-improvement, and 2x-slippage checks in the report.

## Deployment

The bot can be deployed to AWS EC2 using GitHub Actions.

### Prerequisites

1. AWS EC2 instance with Docker installed
2. AWS ECR repository for the Docker images
3. IAM user with ECR and EC2 access
4. SSH key pair for EC2 access

### Setup

1. **Create AWS ECR Repository:**
```bash
aws ecr create-repository --repository-name trading-bot --region your-region
```

2. **Configure GitHub Secrets:**
Add the following secrets to your GitHub repository:
- `AWS_ACCESS_KEY_ID`: Your AWS access key
- `AWS_SECRET_ACCESS_KEY`: Your AWS secret key
- `AWS_ACCOUNT_ID`: Your AWS account ID
- `AWS_REGION`: AWS region (e.g., us-east-1)
- `EC2_HOST`: EC2 instance public IP or DNS
- `EC2_USER`: SSH username (usually 'ec2-user' or 'ubuntu')
- `EC2_SSH_KEY`: Private SSH key for EC2 access

4. **Setup EC2 Instance:**
- Install Docker and AWS CLI
- Configure AWS CLI with credentials that have ECR pull permissions
- Clone your repository
- Copy `.env` file with your Alpaca credentials to the project directory
- Ensure the data, logs, and reports directories exist and are writable
- Make sure Docker daemon is running
- **Set up cron job for periodic execution:**
```bash
# Edit crontab
crontab -e

# Add line to run every 5 minutes during market hours (9:30 AM - 4:00 PM ET, Monday-Friday)
# Note: Adjust timezone as needed
*/5 9-15 * * 1-5 docker run --rm --env-file /path/to/trading-bot/.env -v /path/to/trading-bot/data:/app/data -v /path/to/trading-bot/logs:/app/logs -v /path/to/trading-bot/reports:/app/reports trading-bot:latest
```

Or create a simple run script:
```bash
#!/bin/bash
docker run --rm \
--env-file .env \
-v $(pwd)/data:/app/data \
-v $(pwd)/logs:/app/logs \
-v $(pwd)/reports:/app/reports \
trading-bot:latest
```

Use the provided `run.sh` script for this purpose.

5. **Manual Deployment:**
```bash
# On EC2 instance
./deploy.sh
```

### Automatic Deployment

Push to the `main` branch to trigger automatic deployment via GitHub Actions.
The bot deploys to a plain EC2 instance via GitHub Actions: the workflow at
`.github/workflows/deploy-ec2.yml` rsyncs the repo over SSH, builds the
Docker image on the instance, and installs a cron job — no container
registry or AWS IAM setup involved. Full setup steps, required secrets, and
verification commands are in `docs/github_actions_ec2.md`.

It triggers automatically on pushes to `master`, or manually from the
Actions tab with a chosen profile (`live`/`paper`) and market
(`spy`/`btc`, defaults to `spy`).

### Security Notes

- Use IAM roles on EC2 instead of access keys when possible
- Restrict EC2 security groups to only allow SSH from your IP
- Generate a dedicated SSH key pair for deployment
- Store sensitive credentials only in GitHub secrets and the .env file on EC2
- Store sensitive credentials only in GitHub secrets and the `.env` file on EC2
- Regularly rotate API keys and SSH keys

## Project Structure

- `bot/` - trading logic, broker integration, storage, reporting
- `config/` - per-profile env files (`live_spy`, `paper_spy`, `live_btc`, `paper_btc`)
- `data/` - SQLite database
- `logs/` - runtime logs and CSV snapshots
- `reports/` - generated reports
- `docs/` - strategy audits, build log, and deployment guides

## Risk Controls

Expand All @@ -250,7 +197,7 @@ The bot has multiple independent safety layers:
| Breakeven stop | `ENABLE_BREAKEVEN_STOP` | false |
| Profit lock | `ENABLE_PROFIT_LOCK` | false |
| Time-based stop | `MAX_BARS_IN_TRADE` | 12 bars |
| Trend reversal exit | `REVERSAL_SIGNAL_STRENGTH_MIN` | 35 |
| Regime-invalidation exit | `EXIT_ON_REGIME_INVALIDATION` | true |
| Daily drawdown halt | `MAX_DAILY_DRAWDOWN_PCT` | 1% |
| Dollar loss cap | `MAX_DAILY_LOSS` | disabled |
| Consecutive loss halt | `MAX_CONSECUTIVE_LOSSES` | 3 |
Expand All @@ -266,7 +213,7 @@ Set `IS_CRYPTO=true` (or use `config/paper_btc.env` / `config/live_btc.env`) to
- Bypasses NYSE market-hours check — trades 24/7
- Orders use `TimeInForce.GTC` instead of `DAY`
- Position sizing returns fractional quantities (e.g. `0.0005 BTC`)
- Cron schedule should run every 5 minutes around the clock
- Trades hourly bars like the equity profile; cron runs once an hour around the clock (see `docs/github_actions_ec2.md`)

Important for small accounts: Alpaca crypto costs ~0.25% taker fee + spread per
side (~0.6% per round trip). Replay evidence in `docs/strategy_revamp_2026-07.md`
Expand All @@ -279,19 +226,22 @@ profile for growth.

## Small Equity Accounts

For a roughly `$150` SPY account, one whole SPY share is too large. Use `config/live_spy.env` or set:
For a roughly `$150` account, one whole share of most ETFs is too large a
chunk of the account to size or diversify sensibly. `config/live_spy.env`
(despite the filename, it trades `QQQ`) already sets this up:

- `ALLOW_FRACTIONAL_EQUITIES=true`
- `ALLOW_SHORTS=false`
- `POSITION_SIZING_MODE=atr_risk`
- `MAX_POSITION_NOTIONAL_PCT` and `TARGET_POSITION_NOTIONAL_PCT` high enough to create a real fractional order, but low enough to cap exposure
- `POSITION_SIZING_MODE=notional_cap` with `TARGET_POSITION_NOTIONAL_PCT=0.90` — near-full-notional single positions, since a $150 account can't usefully diversify anyway
- `MAX_POSITION_NOTIONAL_PCT` above the target as a hard ceiling

## Notes

- Keep real API keys only in your local `.env` — never commit them.
- By default, the bot behaves as an intraday system: it flattens inherited overnight positions on the next session and exits before market close. Set `ALLOW_OVERNIGHT_HOLDING=true` and `FLATTEN_BEFORE_CLOSE_MINUTES=0` for crypto.
- The SPY runners write to `runtime/paper` and `runtime/live`; BTC runners write to `runtime/paper_btc` and `runtime/live_btc`.
- Without a profile, the raw `bot.main` default is an intraday equity system: it flattens inherited overnight positions on the next session and exits before market close. Both shipped profiles override this — `config/live_spy.env` and `config/live_btc.env` both set `ALLOW_OVERNIGHT_HOLDING=true` and `FLATTEN_BEFORE_CLOSE_MINUTES=0`, since the current strategy is a multi-day trend hold, not an intraday one.
- The `spy`-market runners write to `runtime/paper` and `runtime/live`; `btc`-market runners write to `runtime/paper_btc` and `runtime/live_btc`.
- Use the optimizer to rank parameter sets on walk-forward windows before going live.
- `OPERATIONS.md` has day-to-day commands.
- `docs/github_actions_ec2.md` covers the GitHub Actions EC2 deployment path.
- `docs/BUILD_LOG.md` tracks strategy and infrastructure changes over time.
- Run `scripts/validate.ps1` (Windows) or `scripts/validate.sh` (Unix) for a full local validation pass.
2 changes: 1 addition & 1 deletion deploy/ec2/install_cron.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ set -Eeuo pipefail
profile="${1:-live}"
app_dir="${2:-${APP_DIR:-/opt/trading-bot/app}}"
cron_tz="${CRON_TZ:-America/New_York}"
schedule="${CRON_SCHEDULE:-*/5 * * * *}"
schedule="${CRON_SCHEDULE:-5 * * * *}"
docker_bin="${DOCKER_BIN:-$(command -v docker || true)}"
job_marker="# trading-bot-${profile}"

Expand Down
124 changes: 124 additions & 0 deletions docs/BUILD_LOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# Build Log

Running log of strategy and infrastructure changes to this bot. Newest entry
first. Each entry should say what changed, why (with evidence where
possible), and what to watch after deploying it.

---

## 2026-07-12 — Repo alignment checkover

**What:** Swept the repo for leftover references to the old BTC-5m-default
setup after the strategy revamp below, since several docs/scripts still
assumed BTC was the primary deployed strategy and that bars were 5 minutes.

**Found and fixed:**
- Cron schedule (`deploy/ec2/install_cron.sh`) still defaulted to
`*/5 * * * *`, i.e. every 5 minutes, but every profile now trades hourly
bars (`TIMEFRAME_MINUTES=60` in all four `config/*.env` files). Left as-is,
a fresh EC2 deploy would have invoked the bot ~12x per hour for one hourly
decision — harmless (cooldown/pending-order guards prevent duplicate
orders) but wasteful API calls and log noise. Changed the default to
`5 * * * *` (once per hour, 5 minutes after each bar closes).
- `docs/github_actions_ec2.md` explicitly said "the deployed profile is BTC,
24/7 scheduling is the correct default" — no longer true now that `spy`
(QQQ) is the default deploy market. Updated.
- `OPERATIONS.md`: paper research equity doc said `$250`; both paper profiles
now use `$150` to match the real account. Also removed a line claiming the
BTC paper profile uses "larger sizing" than live — it's an exact mirror of
`live_btc.env` now, by design, so paper fills validate the same strategy
that runs live.
- `README.md`:
- Opening description and Crypto section still framed BTC as the always-on
24/7 default with 5-minute cron; corrected.
- The "Deployment" section described AWS ECR, IAM roles, `run.sh`,
`deploy.sh`, and pushing to a `main` branch — none of which exist in this
repo. The actual deploy path is GitHub Actions → SSH/rsync → EC2 cron,
already documented in `docs/github_actions_ec2.md`. Replaced the stale
section with a pointer to that doc.
- Removed the `REVERSAL_SIGNAL_STRENGTH_MIN` row from the risk-controls
table — grepping `bot/` shows no code reads that variable (only test
fixtures set it). It's vestigial from an earlier strategy iteration;
flagged separately rather than touched here since removing it fully
means editing `tests/test_research_replay.py`, which is out of scope for
a docs pass.
- Added "superseded" banners to `docs/strategy_audit_current.md` and
`docs/live_account_path_100usd.md` (both dated 2026-04-14, both centered on
an intraday SPY strategy that no longer exists) pointing at
`docs/strategy_revamp_2026-07.md`. Kept their original content intact as a
historical record rather than rewriting them.

**Not changed:** `bot/main.py`'s hardcoded `SYMBOL` default of `SPY` and
`TIMEFRAME_MINUTES` default of `5` — these are fallbacks for running the bot
without a profile at all (e.g. `python -m bot.main` with a bare `.env`).
Every shipped profile overrides both, so this doesn't affect deployed
behavior, but it means an operator who skips the profile runner entirely
still gets the old defaults. Worth a follow-up if that path is ever used for
real.

**Verification:** `pytest` (66 passed), `bot.validate_runtime`,
`bot.validate_profile_env` for all four profiles — all still pass; this
entry was docs/config/deploy-script only, no strategy logic touched.

---

## 2026-07-12 — Strategy revamp: QQQ hourly trend replaces BTC 5m scalp

**What:** Investigated why the live BTC bot ($150 account, 5-minute bars) was
making no valuable trades, then replaced the live deployment strategy.

**Root cause (full writeup: `docs/strategy_revamp_2026-07.md`):**
- The live filter stack required ~14 conditions to align at once; replayed
over 120 days of real bars it produced 2 trades, both losses.
- Position sizing capped trades at ~$45; Alpaca crypto's ~0.6% round-trip
friction exceeds the gross P&L of most 5-minute trades, so even a trade
that fired couldn't clear its own costs.
- BTC fell ~46% over the trailing year and Alpaca doesn't support shorting
crypto, so a long-only bot had no tailwind.
- Directly tested the "more, smaller trades" hypothesis: a loosened
high-frequency BTC config made 184 trades/year and lost $95, ~$99 of it
pure fee friction. Activity was the cost, not the fix, on this venue/size.

**Changes:**
- `config/live_spy.env` + `config/paper_spy.env`: switched to `SYMBOL=QQQ`,
hourly bars, long-only trend-following, ~90% notional fractional sizing,
daily-EMA(20) regime gate, wide trailing exits, multi-day holds. Replay
2023-08→2026-07 at $150 start: net +$55.3 (+37%), profit factor 1.76, win
rate 45%, max drawdown 6.7%, positive every calendar year, stable under 3x
slippage stress.
- `config/live_btc.env` + `config/paper_btc.env`: kept BTC live but made it
defensive — hourly bars, strict 4h-EMA(120) uptrend gate
(`REGIME_MIN_SLOPE_PCT=0.008`). Zero trades in the 2025-26 bear-year replay
(capital preserved through a 46% market decline); mildly positive
(+$3, PF 1.14) in the 2024-25 bull-year replay.
- `bot/trade_controls.py::sync_replay_day`: fixed the replay harness to reset
`consecutive_losses` on ET day rollover, matching `bot/store.py`'s live
behavior. Previously the replay never reset this counter, so any backtest
that hit `MAX_CONSECUTIVE_LOSSES` stopped trading for the rest of the
replay — every prior research report in this repo undercounted trades and
is unreliable.
- `bot/broker_alpaca.py::get_recent_bars`: lookback window now scales with
`timeframe_minutes` and asset session hours instead of a fixed 7 days.
The old fixed window could never warm up hourly/daily-regime indicators
for equities (only ~5 session-days per 7 calendar days).
- `bot/profile.py`: profile env file values now take precedence over market
defaults (`SYMBOL`, `ALLOW_OVERNIGHT_HOLDING`, `FLATTEN_BEFORE_CLOSE_MINUTES`)
instead of being silently overwritten by them — this is what let
`config/live_spy.env` actually set `SYMBOL=QQQ` and hold overnight.
- `docker-compose.yml`, `deploy/ec2/deploy_remote.sh`,
`.github/workflows/deploy-ec2.yml`: default deploy market switched from
`btc` to `spy`. BTC remains reachable via `trade-btc` / `paper-btc` /
`research-btc` compose services.
- Tests: updated profile-contract assertions in `tests/test_profile.py` to
match the new defaults, added regression coverage for the loss-streak
reset and profile-env precedence.

**Verification:** `pytest` (66 passed), `bot.validate_runtime`,
`bot.validate_profile_env` for all four profiles, and a replay of the exact
committed `config/live_spy.env` / `config/live_btc.env` files against real
historical bars (QQQ, SPY, BTC bull year, BTC bear year) to confirm the
numbers in `docs/strategy_revamp_2026-07.md` match what's actually deployed.

**What to watch after deploy:** run `docker compose run --rm paper` for a
week or two before trusting live fills; compare paper fills against the
replay assumptions (slippage, fill price) before increasing size.
Loading
Loading