Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ docker build --target test ./backtest_be_fast

**DB schema:** `database/schema.sql` is the first-boot initdb script; schema changes go through Alembic (`backtest_be_fast/alembic/`). There are **no physical foreign keys** — cross-table references are logical, integrity is owned by the app (A-10). A DB created from schema.sql **before** revision `7b2e9c4f1a30` still has the old FK: baseline it with `alembic stamp d5c3763b29e6` then `alembic upgrade head` (plain `stamp head` would skip the FK drop). A DB created from the current schema.sql is already at head (`alembic stamp head`). Migrations are NOT run by the GitHub Actions pipeline or the runtime image — apply them manually. schema.sql and the Alembic head must stay identical (COMMENTs included): change both together and run `scripts/check-schema-parity.sh`. Keep the `SET NAMES utf8mb4;` at the top of schema.sql — the official image runs initdb with a latin1 client and would double-encode Korean COMMENTs. Upgrade/baseline procedures and the MySQL 8.0 → 8.4 notes live in `database/README.md`.

**API:** POST `/api/v1/backtest` calculates results; GET `/api/v1/market-data` independently loads reference data. The console disables all POST `include_*` flags, renders the calculation first, then loads/retries reference data without rerunning it. Legacy POST defaults remain unchanged. See `docs/architecture/api-boundaries.md`. Errors: `@handle_portfolio_errors` decorator on the calculation path.
**API:** POST `/api/v1/backtest` calculates results; four GETs (`/prices`, `/exchange-rates`, `/benchmarks`, `/news` under `/api/v1`) independently load reference data. The console disables all POST `include_*` flags, renders the calculation first, then loads/retries each resource without rerunning it. The old `/market-data` GET is replaced; legacy POST defaults remain unchanged. All reads share one bounded runner, not four independent capacity pools. See `docs/architecture/api-boundaries.md`. Errors: `@handle_portfolio_errors` decorator on the calculation path.

## Critical Constraints

Expand Down Expand Up @@ -84,15 +84,15 @@ docker build --target test ./backtest_be_fast
- **Verify in Docker** (`docker compose exec`, or `docker build --target test`) before declaring work complete.
- **BE markers:** `@pytest.mark.unit` (no DB), `@pytest.mark.integration` (DB), `@pytest.mark.external` (real API)
- **FE:** Vitest + React Testing Library. Playwright E2E exists in `backtest-console` (`playwright.config.ts`, one smoke spec) and needs the dev stack running — it is deliberately NOT in the Docker CI test stage (no browser, no live backend there).
- **Current baseline (2026-09-27):** BE 492 unit tests + 12 integration + 1 e2e golden master (`pytest tests/e2e -m e2e`, no network/DB), FE 344 tests — all green. Any failure is a regression, not pre-existing noise.
- **Current Docker baseline (2026-10-25):** BE 529 unit tests + 1 e2e golden master (`pytest tests/e2e -m e2e`, no network/DB); the separate console has 356 tests. The older 12 DB integration tests are a historical baseline, not part of the Docker test target. Runtime HTTP verification for the four reads is `tests/integration/verify_reference_reads_http.py` with isolated synthetic fixtures. Any failure is a regression, not pre-existing noise.
- **Test files are type-checked** via `tsconfig.test.json` / `npm run type-check:test`. `tsconfig.build.json` deliberately excludes them.
- **Coverage (BE 2026-08-03 / FE 2026-09-27):** BE 71.6%, FE about 74% statements. Core financial modules are 82-98%; the remaining gaps are `data_fetcher` (41%), the `app/validators/` package (23-32%), and `currency_converter` (57%). The validators look dead but are reached via `backtest_engine.py` → `validation_service` — do not delete them.

## CI

GitHub Actions lives in `.github/workflows/cicd.yml`. PRs to main/dev and pushes to either branch run Docker tests, dependency audit, runtime build, and schema parity checks. Only main publishes an ARM64 `ghcr.io/kyj0503/backtest` image and deploys its immutable digest through Tailscale and pinned SSH. dev has no OCI deployment. Runtime credentials stay in `/opt/backtest/production/.env` and are never committed.

The dependency audit retains the documented bokeh exception in `scripts/audit-deps.sh`; do not weaken it. Deployment checks `/health/ready` after Docker liveness succeeds and restores the previous release on failure. CI does not automatically migrate a deployed DB. Shared gateway configuration is managed outside this repository at `/opt/gateway`.
The dependency audit has no exceptions in `scripts/audit-deps.sh`; do not weaken it. Deployment checks `/health/ready` after Docker liveness succeeds and restores the previous release on failure. CI does not automatically migrate a deployed DB. Shared gateway configuration is managed outside this repository at `/opt/gateway`.

## Commit Convention

Expand Down
22 changes: 22 additions & 0 deletions HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,28 @@

---

## 2026-10-25 네 종류의 독립 조회

- **A-39**: 통합 GET `/api/v1/market-data`를 `/prices`, `/exchange-rates`,
`/benchmarks`, `/news` 네 GET으로 교체했다. 각 응답은 소유한 자료와 수집 상태만
반환한다. 기존 POST 요청·기본값·계산 필드는 유지한다.
- 기존 수집기를 재사용하되 뉴스·환율·벤치마크의 불필요한 종목 메타데이터 조회를
생략한다. 주가와 급등락은 동일 시세를 공유한다. 뉴스에는 기간을 받지 않고,
취소 후 다음 종목을 조회하거나 캐시에 쓰지 않도록 취소 확인을 추가했다.
- 네 GET은 하나의 파일 슬롯 실행기를 공유하며 기본 전체·IP 상한을 각각 4로 맞췄다.
API마다 독립 풀을 만들어 총 조회량이 네 배로 늘어나는 구성을 피한다.
- 콘솔의 로딩·오류·빈 결과·재시도를 종류별로 분리했다. 함수형 상태 병합과 필드
허용 목록으로 병렬 응답끼리 덮어쓰지 않으며 재시도는 다른 요청을 취소하지 않는다.
새 계산·화면 이탈 때 모든 이전 요청을 취소한다. 뉴스 설명도 최신 뉴스로 바로잡았다.
- Docker BE 529개·골든 E2E 1개, 콘솔 356개·lint·두 TypeScript 검사,
예외 없는 두 의존성 감사와 runtime 빌드를 통과했다. 배포용 이미지·격리 MySQL의
합성 시세·뉴스로 네 병렬 조회, 기존 계산값·현금 1,500 보존을 확인했다.
- 실제 브라우저에서 네 GET에 503을 주고 하나씩 복구했다. 각 재시도는 그 GET만
1회 증가시키며 계산 POST는 네 번의 재시도 전후 4건으로 같았다. 운영 자격증명과
운영 DB는 사용하지 않았다. 이 검증은 계산 정확성 전반이나 운영 부하 보장이 아니다.
- 옛 GET을 호출하는 콘솔은 새 API와 호환되지 않으므로 운영 릴리스는 두 저장소를
함께 계획하고 이전 탭을 새로고침해야 한다. 현재 계약은 [API 경계](docs/architecture/api-boundaries.md)에 있다.

## 2026-10-25 API 경계·게이트웨이 보완

- **A-38**: 콘솔의 계산 POST와 참고 자료 GET을 분리했다. 기존 POST 본문·기본값·응답
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ CA 파일은 `/opt/backtest/production/tls/oracle-mysql-ca.pem`에 두고 컨테

`POST /api/v1/backtest`가 주요 API입니다. 최소 백테스트 기간은 30일입니다.

콘솔은 `include_*`를 끄고 계산 결과를 먼저 받은 뒤 `GET /api/v1/market-data`로
참고 데이터를 조회합니다. 기존 통합 응답은 기본 옵션으로 계속 지원합니다.
콘솔은 `include_*`를 끄고 계산 결과를 먼저 받은 뒤 `/api/v1/prices`, `/exchange-rates`,
`/benchmarks`, `/news`를 각각 조회·재시도합니다. 기존 POST 통합 응답은 기본 옵션으로 계속 지원합니다.
조회·취소·실행 한도와 이전 절차는 [API 경계](docs/architecture/api-boundaries.md)를 참고하세요.
프론트엔드의 `MIN_BACKTEST_PERIOD_DAYS`와 백엔드 설정을 함께 유지합니다.
CORS는 공통 Nginx에서 처리합니다. 서버 비밀값을 프론트엔드 `VITE_*` 변수에 넣지 않습니다.
Expand Down
4 changes: 2 additions & 2 deletions backtest_be_fast/app/api/v1/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

**엔드포인트**:
- POST /api/v1/backtest: 계산 실행 (기존 include_* 통합 응답도 지원)
- GET /api/v1/market-data: 계산과 독립적인 주가·환율·벤치마크·뉴스 조회
- GET /api/v1/prices, /exchange-rates, /benchmarks, /news: 독립 조회·재시도
- 전략 목록은 프론트엔드에서 관리
- 콘솔은 계산을 먼저 표시하고 참고 데이터를 별도로 조회한다.
"""
Expand All @@ -23,5 +23,5 @@
tags=["백테스팅"]
)

api_router.include_router(market_data.router, prefix="/market-data", tags=["시장 데이터"])
api_router.include_router(market_data.router, tags=["시장 데이터"])

111 changes: 78 additions & 33 deletions backtest_be_fast/app/api/v1/endpoints/market_data.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Reference market data, independently retriable without another simulation."""
"""Independently retriable reads sharing one bounded reference-data capacity."""
import asyncio
from datetime import date
from functools import lru_cache
Expand All @@ -19,23 +19,36 @@
_running = Gauge('market_data_jobs_running', 'Active reference-data jobs', multiprocess_mode='livesum')
_waiting = Gauge('market_data_jobs_waiting', 'Queued reference-data jobs', multiprocess_mode='livesum')

# Both price views share one fetch. A response can only update its own fields.
READ_FIELDS = {
'prices': ('ticker_info', 'stock_data', 'volatility_events'),
'exchange-rates': ('exchange_rates', 'exchange_stats'),
'benchmarks': ('sp500_benchmark', 'nasdaq_benchmark'),
'news': ('latest_news',),
}
READ_STATUSES = {
'prices': ('ticker_info', 'stock_data', 'volatility_events'),
'exchange-rates': ('exchange_rates',),
'benchmarks': ('benchmarks',),
'news': ('news',),
}

class MarketDataQuery(BaseModel):

class SymbolsQuery(BaseModel):
model_config = {'extra': 'forbid'}
symbols: list[str] = Field(default_factory=list, max_length=settings.max_portfolio_items)
start_date: date
end_date: date
include_stock_data: bool = True
include_volatility_events: bool = True
include_exchange_rates: bool = True
include_benchmarks: bool = True
include_news: bool = True
symbols: list[str] = Field(min_length=1, max_length=settings.max_portfolio_items)

@field_validator('symbols')
@classmethod
def validate_symbols(cls, values):
return list(dict.fromkeys(PortfolioStock(symbol=value).symbol for value in values))


class PeriodQuery(BaseModel):
model_config = {'extra': 'forbid'}
start_date: date
end_date: date

@model_validator(mode='after')
def validate_dates(self):
if self.end_date < self.start_date:
Expand All @@ -47,48 +60,80 @@ def validate_dates(self):
return self


class PricesQuery(PeriodQuery, SymbolsQuery):
pass


class NewsQuery(SymbolsQuery):
limit: int = Field(default=15, ge=1, le=20)


@lru_cache(maxsize=1)
def get_market_data_runner():
return BacktestJobRunner(
slot_dir=settings.backtest_slot_dir + '-market-data',
max_concurrent=settings.max_concurrent_market_data,
per_client=1,
per_client=4,
queue_timeout=settings.market_data_queue_timeout_seconds,
exec_timeout=settings.market_data_timeout_seconds,
cancel_grace=settings.backtest_cancel_grace_seconds,
running_metric=_running,
waiting_metric=_waiting,
job_name='참고 데이터 조회',
running_metric=_running, waiting_metric=_waiting, job_name='참고 데이터 조회',
)


@router.get('', summary='주가·환율·벤치마크·뉴스 조회')
async def get_market_data(query: Annotated[MarketDataQuery, Query()], request: Request):
async def _read(section: str, request: Request, *, symbols=None, start_date=None, end_date=None, limit=15):
async def collect():
data = await asyncio.to_thread(
unified_data_service.collect_all_unified_data,
symbols=query.symbols,
start_date=query.start_date.isoformat(),
end_date=query.end_date.isoformat(),
include_news=query.include_news,
news_display_count=15,
include_stock_data=query.include_stock_data,
include_volatility_events=query.include_volatility_events,
include_exchange_rates=query.include_exchange_rates,
include_benchmarks=query.include_benchmarks,
symbols=symbols or [],
start_date=start_date.isoformat() if start_date else None,
end_date=end_date.isoformat() if end_date else None,
include_ticker_info=section == 'prices',
include_stock_data=section == 'prices',
include_volatility_events=section == 'prices',
include_exchange_rates=section == 'exchange-rates',
include_benchmarks=section == 'benchmarks',
include_news=section == 'news',
news_display_count=limit,
timeout_seconds=settings.supplemental_data_timeout_seconds,
)
benchmark = data.get('sp500_benchmark', [])
data['sp500_total_return_pct'] = (
unified_data_service.calculate_benchmark_return(benchmark)
if len(benchmark) >= 2 else None
)
return {'status': 'success', 'data': recursive_serialize(data)}
result = {field: data[field] for field in READ_FIELDS[section]}
result['supplemental_status'] = {
name: data['supplemental_status'][name] for name in READ_STATUSES[section]
}
if section == 'benchmarks':
benchmark = result['sp500_benchmark']
result['sp500_total_return_pct'] = (
unified_data_service.calculate_benchmark_return(benchmark)
if len(benchmark) >= 2 else None
)
return {'status': 'success', 'data': recursive_serialize(result)}

try:
return await get_market_data_runner().run(
collect, client_key=_client_key(request),
is_disconnected=request.is_disconnected, label='market-data',
collect, client_key=_client_key(request), is_disconnected=request.is_disconnected,
label=section,
)
except BacktestRejected as exc:
return _rejection_response(exc)


@router.get('/prices', summary='종목 정보·주가·급등락 이벤트 조회')
async def get_prices(query: Annotated[PricesQuery, Query()], request: Request):
return await _read('prices', request, symbols=query.symbols,
start_date=query.start_date, end_date=query.end_date)


@router.get('/exchange-rates', summary='기간별 원달러 환율·통계 조회')
async def get_exchange_rates(query: Annotated[PeriodQuery, Query()], request: Request):
return await _read('exchange-rates', request, start_date=query.start_date, end_date=query.end_date)


@router.get('/benchmarks', summary='기간별 S&P 500·NASDAQ 조회')
async def get_benchmarks(query: Annotated[PeriodQuery, Query()], request: Request):
return await _read('benchmarks', request, start_date=query.start_date, end_date=query.end_date)


@router.get('/news', summary='종목별 최신 뉴스 조회 — 백테스트 기간과 무관')
async def get_news(query: Annotated[NewsQuery, Query()], request: Request):
return await _read('news', request, symbols=query.symbols, limit=query.limit)
2 changes: 1 addition & 1 deletion backtest_be_fast/app/core/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ def backend_cors_origins(self) -> list[str]:

# Read-only market data has its own capacity; slow feeds must not occupy
# the simulation slots or keep a completed calculation waiting.
max_concurrent_market_data: int = 2
max_concurrent_market_data: int = 4
market_data_queue_timeout_seconds: float = 2.0
market_data_timeout_seconds: float = 20.0

Expand Down
19 changes: 13 additions & 6 deletions backtest_be_fast/app/services/unified_data_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
)
from app.repositories.stock_repository import get_stock_repository
from ..core.config import settings
from ..core.cancellation import CancelToken, bind_token, current_token, submit_with_context
from ..core.cancellation import CancelToken, bind_token, current_token, submit_with_context, check_cancelled

logger = logging.getLogger(__name__)

Expand Down Expand Up @@ -273,6 +273,7 @@ def collect_latest_news(

latest_news = {}
for symbol in symbols:
check_cancelled()
try:
# 1. DB에서 먼저 조회 (3시간 이내)
cached_news = self.stock_repo.load_ticker_news(symbol, max_age_hours=max_cache_hours)
Expand All @@ -286,6 +287,7 @@ def collect_latest_news(
logger.info(f"{symbol} 뉴스 캐시가 없거나 오래됨 - API 호출")
search_query = self.news_service.get_ticker_query(symbol)
news_list = self.news_service.search_news(search_query, display=display)
check_cancelled()

# 3. API 결과를 DB에 저장
if news_list:
Expand Down Expand Up @@ -350,15 +352,16 @@ def _fetch_price_histories(
def collect_all_unified_data(
self,
symbols: List[str],
start_date: str,
end_date: str,
start_date: Optional[str] = None,
end_date: Optional[str] = None,
include_news: bool = True,
news_display_count: int = 20,
include_stock_data: bool = True,
include_volatility_events: bool = True,
include_exchange_rates: bool = True,
include_benchmarks: bool = True,
timeout_seconds: Optional[float] = None,
include_ticker_info: bool = True,
) -> Dict[str, Any]:
"""
모든 통합 데이터를 한 번에 수집
Expand All @@ -374,6 +377,7 @@ def collect_all_unified_data(
include_exchange_rates: 환율·환율 통계 포함 여부 (A-08)
include_benchmarks: S&P 500/NASDAQ 포함 여부 (A-08)
timeout_seconds: 병렬 수집 구간의 시간 예산. None이면 무제한 (A-08)
include_ticker_info: 종목 메타데이터 포함. 독립 뉴스·환율·벤치마크 조회는 끈다.

Returns:
모든 통합 데이터를 포함하는 딕셔너리. 끈 섹션·시간 초과 섹션도 키는
Expand Down Expand Up @@ -405,6 +409,8 @@ def collect_all_unified_data(
타임아웃까지 응답을 기다린 뒤 끝난다(CPU를 쓰지 않는 대기라 동시 계산
상한의 목적은 유지된다).
"""
if (include_stock_data or include_volatility_events or include_exchange_rates or include_benchmarks) and not (start_date and end_date):
raise ValueError('가격·환율·벤치마크 조회에는 시작일과 종료일이 필요합니다.')
total_start = time.perf_counter()
timings: Dict[str, float] = {}

Expand All @@ -431,9 +437,10 @@ def timed(stage: str, fn: Callable, *args, **kwargs):
try:
# submit_with_context가 제출 시점 컨텍스트(하위 토큰 바인딩)를 스레드로 넘긴다
with bind_token(supplemental_token):
futures['ticker_info'] = submit_with_context(
executor, timed, "ticker_info", self.collect_ticker_info, symbols
)
if include_ticker_info:
futures['ticker_info'] = submit_with_context(
executor, timed, "ticker_info", self.collect_ticker_info, symbols
)
if need_prices:
futures['price_history'] = submit_with_context(
executor, timed, "price_history", self._fetch_price_histories,
Expand Down
Loading