Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions docs/FINANCIAL_FOUNDATIONS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Financial foundations v1 — 01–06 / 08–10

这是一组证据驱动的离线金融数据契约,不是全市场历史数据库,也不认证输入来源完整性。所有时间戳必须带时区;effective_at 是经济生效时间,available_at 是研究者获知时间。修订不能提前可见。不可将今天的证券名单或修订值直接回填过去。

## 01 单位与因子输入

`financial.units.normalize_trading_units(frame, volume_unit=..., amount_unit=..., currency=..., share_basis="raw", lot_size=...)` 显式把手转换为股、千元/万元转换为币种金额。手必须提供 lot_size;不允许复权股数、不推断流通股数。输出 free_float_shares 必须来自当时已知的原始股本口径。历史交易量均值不是换手率。

## 02 生命周期与历史证券池

`financial.lifecycle.COLUMNS` 是完整事件列清单。事件含稳定 instrument_id、symbol、venue、kind、effective_at、available_at、source、evidence_id、universe_id、successor_id;不适用字段保留空值。支持上市、退市、更名、转板、合并、入池、出池。
`select_universe(events, effective_at, known_at, universe_id=None, holdings=())` 返回 eligible、retained_holdings、unknown_holdings。出池不等于退市;存量持仓必须继续估值/清算。冲突事件和同时重复的活跃代码/交易所映射拒绝处理。它是 instrument_master 的伴随历史,不替代交易规则主表。

## 03 公司行动与总收益

`financial.actions.ActionTerms` 统一拆并股、现金分红权益/支付、碎股现金、换股合并、分拆、供股权分派与行权、终止现金。换股/分拆/权利分派必须提供成本分配比例和目标估值,行权必须显式指定 election_quantity;跨币种、做空权益和税务自动推断不支持。
`financial.returns.total_return_panel(prices, actions, timezone=..., price_basis="raw")` 仅接受原始报价,构造拆股/分红权益的信号指数。假设除息收盘理论再投资,不代表现金到账。复杂或同时多项行动拒绝进入此简化指数,须使用 QExec 账户回放。账本不得再次应用复权价格中的同一行动。

## 04 可交易状态

`financial.status.permission_asof(records, instrument_id, effective_at, known_at)` 返回买卖两侧许可。字段包含 effective_from、effective_to(右开)、available_at、buy_status/sell_status、reason、source、evidence_id。状态为 tradable/blocked/unknown;缺失、过期、冲突一律 unknown。`no_restriction` 不能作为 tradable 的证据。QExec 和港股适配负责将 unknown 变成订单阻断。

## 05 用途日历

`PurposeCalendar(calendar_id, purpose, version, available_at, valid_from, valid_to, open_days, source, evidence_kind)` 不隐式生成工作日。purpose 为 trading/settlement/banking/connect/dealing/confirmation;有效区间、版本获知时间和开日都必须显式。
`CalendarBook([...]).asof(id, purpose, at, on)` 选择当时可见版本,`calendar.advance(on, lag, at=..., purpose=...)` 计算到期日。缺少覆盖直接失败。正 lag 从 on 之后计开日;lag=0 要求 on 本身开放。HK 与基金适配已消费该契约。

## 06 来源冲突与新快照

1. `discrepancies(observations, at=...)` 比较同一证券/字段/生效时间,先处理金额倍数及百分比;币种、复权口径不一致标为不可直接比较。
2. `freeze_snapshot(chosen_records)` 创建每个经济键唯一的治理快照。
3. `adjudicate(case, selected_observation_id=..., resolved_at=..., reviewer=..., rationale=..., evidence_uri=..., prior_snapshot=..., dependencies=...)` 要求人工选择已有证据,计算受影响节点的传递闭包。
4. `publish_decision(decision, new_path)` 独占创建裁决文件;`apply_decision_snapshot(parent, decision, new_path)` 实际生成修正数据快照,不改旧文件,修正值从裁决时间起可见。
5. 重算受影响的因子/订单/报告是单独动作;发布裁决不等于重算完成。quant-report-hub 可核验并展示待重算清单。

快照和裁决有内容哈希,不等于数字签名/审批权限控制。调用者仍须保管完整来源、旧版本和运行依赖图。

## 08 SEC 独立季度、TTM、重述

`us_research.sec.quarterly_facts(facts, at)` 按实际 fiscal period 边界识别直接季度、同年度累计差分;不按自然季度硬切,不把累计数字当独立季度。保留 accession 与 vintage_consistent。
`ttm_facts(..., allow_mixed_vintages=False, max_age_days=200)` 要求四个连续季度,兼容 53 周年,拒绝缺口及未确认可比的跨文件差分。允许混合版本必须显式 opt-in;这不构成重述可比性认证。
`ttm_quality` 要求同一截止期 USD 净利润/经营现金流和期末资产;ROA 分母是期末资产,不是平均资产。美股 quality_ttm 单独启用,旧 annual 不更名。
真实小样本位于 tests/fixtures/financial/apple_cash_flow_extract.json:Apple 三份 2024 财年 10-Q 的公开现金流与申报索引事实摘录。它验证累计差分及 acceptance+5min 边界,不是下载快照或全市场回测。SEC 下载仍需用户自己的真实联系身份,未写入凭据。

## 09 宏观 vintage

`financial.macro.from_alfred(observations, series_id=..., unit=..., release_times=..., source=...)` 接受存档 ALFRED JSON。realtime_start 只有日期精度,不能猜成午夜或默认 08:30;release_times 必须显式提供有证据的带时区发布时间。同一系列不能隐式改变单位。`macro_asof` 按当时已知版本选择,缺失点不填充。quant-regime 输出上下文、缺失和陈旧状态,不暗中改变原状态分类器。

## 10 基金/ETF 穿透

`financial.holdings.COLUMNS` 定义 disclosure_id、fund_id、holding_date、available_at、instrument_id、asset_type、currency、weight、source、evidence_id。披露是完整快照,不把后来缺失的持仓与旧披露拼接。
`look_through(disclosures, {fund_id: "portfolio_weight"}, at, max_depth=8, max_age_days=180)` 使用 Decimal 并保留每条路径。未披露余额、陈旧、缺失、循环和深度上限都变为 UNKNOWN,不丢弃权重;exposure_summary 聚合重叠证券与币种。现金必须使用 CASH:<currency>。
仅支持 long-only 权重;不自动推断衍生品 delta、杠杆、汇率或债券发行人集团敞口。披露穿透不是实时持仓。金额输入须先用已知 FX 转为统一组合权重。

## 使用与回归

安装本分支后执行:
```powershell
python -m quant_data_kit.financial.cli --input examples/financial/units.json --output units-result.json
python -m pytest tests/test_financial_foundations.py tests/test_financial_evidence.py
```
CLI schema 为 puresaber.financial-foundations/1,operation 支持 units/lifecycle/universe/status/calendar/reconcile/sec-ttm/macro/lookthrough。结果明确标注 supplied_facts_not_market_coverage,拒绝覆盖输出文件。

## 参考与边界

- [SEC 官方 API 与 XBRL](https://www.sec.gov/search-filings/edgar-application-programming-interfaces):申报版本、单位、期间与 acceptance 来源。
- [ALFRED realtime period](https://fred.stlouisfed.org/docs/api/fred/realtime_period.html):知识时点与修订,不等于日内发布时间。
- [HKEX 结算](https://www.hkex.com.hk/Services/Settlement-and-Depository/Settlement?sc_lang=en):交易和交收用途分离。
- [LEAN corporate actions](https://www.quantconnect.com/docs/v2/writing-algorithms/securities/asset-classes/us-equity/corporate-actions):行动事件与复权口径分离,未复制第三方实现。
- [Apple IR dividend/split history](https://investor.apple.com/dividend-history/):4-for-1 拆股条款;测试价格/持仓及保守获知时间为场景值,不冒充真实逐笔证据。

这些模块不提供收益承诺、投资建议或实盘下单能力;真实数据授权、完整覆盖及历史可获得性仍需使用者验证。
43 changes: 38 additions & 5 deletions docs/RESEARCH_DATASETS.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ The Python API is available directly from the module:

```python
from quant_data_kit.research_dataset import (
bind_instrument_master,
build_dataset,
inspect_dataset,
load_research_snapshot,
Expand Down Expand Up @@ -56,11 +57,43 @@ hash-bound `source.json` and Parquet history, so
`quant_data_kit.research_coverage.load_history(snapshot / "history")` works
without adapting a loose file.

The catalog's `available_at` is the real snapshot capture time. QDK does not
backdate instrument-master availability to the research start. Until a separate
historical exchange/security-master source proves earlier publication, a strict
historical replay must remain blocked; a structurally loadable catalog is not
evidence that the rules were known at each historical decision time.
The default catalog's `available_at` is the real snapshot capture time. QDK
does not backdate instrument-master availability to the research start. A
structurally loadable catalog is not evidence that the rules were known at each
historical decision time.

## Bind a historical instrument master

`quant_data_kit.instrument_master` imports an explicit source declaration,
catalog and official documents into one immutable bundle. Import rejects a
changed source hash, an unknown evidence ID, a rule published after its claimed
effective time, a coverage gap and catalog availability before its initial
evidence. Every source needs an HTTPS identity, publication time, effective
interval and exact SHA-256.

```powershell
python -m quant_data_kit.instrument_master import `
--source ..\artifacts\pit-sources\source `
--output ..\artifacts\pit-sources\bundle

python -m quant_data_kit.research_dataset bind-master `
--root ..\artifacts\etf-research `
--instrument-master ..\artifacts\pit-sources\bundle
```

Binding publishes a child snapshot. It copies already verified market data and
actions without a provider request, replaces the current-capture catalog with
the evidence-bound catalog, and retains the complete master bundle below
`instrument_master/`. A later incremental update automatically carries that
bundle forward and rechecks its symbol set, validity interval and availability.

The bundle proves only its declared fields and intervals. A retrospective
listing document becomes usable at the document publication time; it is not
backdated to the listing day. An empty present-day suspension query is not
expanded into daily `tradable=true`, and volume does not prove all-day
tradability. If the source declares a fixed retrospective pool, missing daily
status or missing dynamic-universe history, consumers must preserve those
limits rather than label the result a complete dynamic PIT backtest.

Every live manifest records the AKShare package version, QDK version, endpoint
names, normalized request and capture time. Every Parquet file has an exact
Expand Down
15 changes: 15 additions & 0 deletions docs/us-research.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# US research data v1

`quant_data_kit.us_research` is an explicit USD research namespace. It does not change the legacy six-digit domestic symbol adapters.

- `calendar.schedule`: XNYS sessions with UTC opens/closes, DST and half-days.
- `calendar.settlement_session`: standard T+2/T+1 date regimes from 2018.
- `prices.validate_prices/write_bundle/load_bundle`: OHLC/action validation and immutable hashed CSV snapshots.
- `prices.fetch_yahoo`: opt-in Yahoo retrospective split-normalized prices, not raw as-traded prices or historical constituent certification. Dividend payment dates remain unknown.
- `universe.members_asof`: source-backed membership events selected by effective session and knowledge time.
- `sec.download_sec/read_sec_snapshot`: explicit SEC contact identity, rate-limited snapshots, full submissions pagination and hash checks.
- `sec.filing_table/fact_table/facts_asof/annual_quality`: accession acceptance plus modeled processing lag, retained restatements, period-matched annual quality metrics. Facts with unknown acceptance are excluded and audited.

Install `.[us-research]` for optional calendar/Yahoo/SEC clients. Unit tests never require live data or contact credentials. `historical_pit` is source evidence supplied by the importer, not automatic certification by the package.

Settlement uses regular exchange holidays plus Columbus/Veterans bank holidays, independently of exchange-only ad-hoc closures. Saturday Veterans Day is not observed on Friday by Federal Reserve Banks. The daily research account models settled funds as available on the settlement date, without intraday clearing or broker-specific holds. Sources: [Federal Reserve bank holidays](https://www.federalreserve.gov/releases/k8/default.htm), [DTCC Veterans Day 2024](https://www.dtcc.com/-/media/Files/pdf/2024/10/11/a9503.pdf), [SIFMA January 2025 closure matrix](https://www.sifma.org/resources/guides-playbooks/unscheduled-close-market-matrix). Future or exceptional clearing changes require calendar review.
18 changes: 18 additions & 0 deletions examples/financial/units.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"schema": "puresaber.financial-foundations/1",
"operation": "units",
"records": [
{
"volume": 12,
"amount": 36,
"free_float_shares": 1000000
}
],
"options": {
"volume_unit": "lots",
"lot_size": 100,
"amount_unit": "thousand_currency",
"currency": "CNY",
"share_basis": "raw"
}
}
4 changes: 3 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ dependencies = [
]

[project.optional-dependencies]
us-research = ["exchange-calendars>=4.5,<5", "yfinance>=0.2.65,<1", "requests>=2.32,<3"]
hong-kong = ["akshare>=1.12", "exchange-calendars>=4.10,<5"]
akshare = ["akshare>=1.12"]
baostock = ["baostock>=0.8.9"]
Expand All @@ -36,7 +37,7 @@ all-equity-providers = [
"yfinance>=0.2.65",
"requests>=2.32,<3",
]
dev = ["coverage>=7.6", "psutil>=6.0,<8.0", "pytest>=7.4", "ruff>=0.9.0"]
dev = ["coverage>=7.6", "psutil>=6.0,<8.0", "pytest>=7.4", "ruff>=0.9.0", "exchange-calendars>=4.5,<5"]

[project.scripts]
qdk-validate = "quant_data_kit.cli:main_validate"
Expand All @@ -46,6 +47,7 @@ qdk-capture = "quant_data_kit.capture_v2.cli:main_capture"
qdk-providers = "quant_data_kit.cli:main_providers"
qdk-research-inputs = "quant_data_kit.research_inputs:main"
qdk-research-dataset = "quant_data_kit.research_dataset:main"
qdk-financial = "quant_data_kit.financial.cli:main"

[tool.setuptools.packages.find]
where = ["src"]
Expand Down
19 changes: 16 additions & 3 deletions requirements.lock
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --allow-unsafe --build-deps-for=editable --constraint=requirements-constraints.txt --extra=dev --index-url=https://pypi.org/simple --output-file=requirements.lock --strip-extras pyproject.toml
# pip-compile --allow-unsafe --build-deps-for=editable --constraint=requirements-constraints.txt --extra=dev --index-url=https://pypi.org/simple --no-index --output-file=requirements.lock --strip-extras pyproject.toml
#
attrs==26.1.0
# via
Expand All @@ -14,6 +14,8 @@ coverage==7.16.0
# via quant-data-kit (pyproject.toml)
duckdb==1.5.5
# via quant-data-kit (pyproject.toml)
exchange-calendars==4.13.2
# via quant-data-kit (pyproject.toml)
iniconfig==2.3.0
# via pytest
jsonschema==4.26.0
Expand All @@ -22,16 +24,21 @@ jsonschema-rs==0.52.0
# via quant-data-kit (pyproject.toml)
jsonschema-specifications==2025.9.1
# via jsonschema
korean-lunar-calendar==0.4.0
# via exchange-calendars
numpy==2.2.6
# via
# exchange-calendars
# pandas
# quant-data-kit (pyproject.toml)
orjson==3.12.0
# via quant-data-kit (pyproject.toml)
packaging==26.3
# via pytest
pandas==2.3.3
# via quant-data-kit (pyproject.toml)
# via
# exchange-calendars
# quant-data-kit (pyproject.toml)
pluggy==1.6.0
# via pytest
psutil==7.2.2
Expand All @@ -40,6 +47,8 @@ pyarrow==25.0.1
# via quant-data-kit (pyproject.toml)
pygments==2.21.0
# via pytest
pyluach==2.3.0
# via exchange-calendars
pytest==9.1.1
# via quant-data-kit (pyproject.toml)
python-dateutil==2.9.0.post0
Expand All @@ -61,12 +70,16 @@ ruff==0.16.5
# via quant-data-kit (pyproject.toml)
six==1.17.0
# via python-dateutil
toolz==1.1.0
# via exchange-calendars
typing-extensions==4.16.0
# via
# quant-data-kit (pyproject.toml)
# referencing
tzdata==2026.3
# via pandas
# via
# exchange-calendars
# pandas
websockets==15.0.1
# via quant-data-kit (pyproject.toml)

Expand Down
7 changes: 7 additions & 0 deletions src/quant_data_kit/financial/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
"""Versioned financial facts; explicit knowledge time, provenance and coverage.

These contracts validate supplied evidence, not completeness of market history.
No provider is contacted implicitly and missing facts are never guessed.
"""

SCHEMA_VERSION = "puresaber.financial-foundations/1"
Loading
Loading