Skip to content

Commit c866670

Browse files
committed
Merge branch 'develop' of https://github.com/makr-code/ThemisDB into tmp/push-fix3
2 parents bd33a5f + f3f5d0e commit c866670

115 files changed

Lines changed: 9629 additions & 513 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/actions/setup-cpp-build/action.yml

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -26,22 +26,34 @@ inputs:
2626
runs:
2727
using: 'composite'
2828
steps:
29+
# Pre-configure sccache environment variables BEFORE sccache-action starts
30+
# the server. Variables written to GITHUB_ENV in step N are visible in
31+
# step N+1, including to the sccache daemon launched by sccache-action.
32+
# SCCACHE_IGNORE_SERVER_IO_ERROR and SCCACHE_ERROR_WHEN_DISCONNECTED must
33+
# be set *before* the daemon starts so that transient GHA Cache storage
34+
# outages (HTTP 503) cause sccache to fall back to direct compilation
35+
# instead of aborting with exit code 2.
36+
- name: Pre-configure sccache environment
37+
shell: bash
38+
run: |
39+
echo "SCCACHE_GHA_ENABLED=true" >> "$GITHUB_ENV"
40+
# Ignore transient cache I/O errors so sccache falls back to direct
41+
# compilation instead of aborting with exit code 2.
42+
echo "SCCACHE_IGNORE_SERVER_IO_ERROR=1" >> "$GITHUB_ENV"
43+
# When the GHA cache storage backend is fully unavailable (e.g. returns
44+
# an HTML error page on startup), sccache would otherwise fail the
45+
# server-start and cause every compilation to exit with code 2.
46+
# Setting this to false lets sccache fall back to direct compilation
47+
# transparently instead of aborting the build.
48+
echo "SCCACHE_ERROR_WHEN_DISCONNECTED=false" >> "$GITHUB_ENV"
49+
2950
# sccache replaces ccache as the compiler-output cache.
3051
# SHA pinned to mozilla-actions/sccache-action v0.0.6 (2024-11-01).
3152
- name: Setup sccache
3253
uses: mozilla-actions/sccache-action@9e326ebed976843c9932b3aa0e021c6f50310eb4 # v0.0.6
3354
with:
3455
version: "v0.8.1"
3556

36-
- name: Enable sccache GHA backend
37-
shell: bash
38-
run: |
39-
echo "SCCACHE_GHA_ENABLED=true" >> "$GITHUB_ENV"
40-
# Ignore transient cache I/O errors (e.g. temporary GHA Cache storage
41-
# outages) so sccache falls back to direct compilation instead of
42-
# aborting with exit code 2.
43-
echo "SCCACHE_IGNORE_SERVER_IO_ERROR=1" >> "$GITHUB_ENV"
44-
4557
- name: Install build dependencies
4658
shell: bash
4759
run: |

.github/workflows/gate-pr-version-targeting.yml

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,13 +38,14 @@ jobs:
3838
)
3939
4040
echo "Target Version extracted: '$TARGET_VERSION'"
41-
echo "target_version=$TARGET_VERSION" >> "$GITHUB_OUTPUT"
42-
41+
4342
if [[ -z "$TARGET_VERSION" ]]; then
44-
echo "::error::PR does not have a Target Version specified. Fill in the 'Target Version' field in the PR template."
45-
exit 1
43+
echo "::warning::PR does not have a Target Version specified. Defaulting to '[Unreleased]'. Fill in the 'Target Version' field in the PR template before merge."
44+
TARGET_VERSION="[Unreleased]"
4645
fi
4746
47+
echo "target_version=$TARGET_VERSION" >> "$GITHUB_OUTPUT"
48+
4849
- name: Validate Target Version format
4950
id: validate_format
5051
run: |

ROADMAP.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1269,6 +1269,21 @@ Audit method:
12691269
- Improved scanner pipeline: Phase 1–6 vollständig implementiert, alte Scanner bereinigt
12701270
- Delta vs Pre-Improvement-Baseline (2026-05-27): Methodik geändert (dedupliziert vs raw); Trend: FP-Anteil signifikant reduziert
12711271

1272+
**Wave 3 Source-Code Gap Remediation — COMPLETE (2026-08-25):**
1273+
> Subagent-verifizierte Triage der rohen Scanner-CRITICAL-Counts für die vier Core-Module mit den meisten gemeldeten Gaps. Echte Gaps deutlich geringer als Scanner suggeriert (Inflationsfaktor 7–34×).
1274+
1275+
| Modul | Raw CRITICAL | Echte CRITICAL | Inflationsfaktor | Status |
1276+
|-------|-------------|----------------|-----------------|--------|
1277+
| storage | 69 | 2 | 34× | ✅ Wave 3-A COMPLETE — columnar decode, backup fail-closed, diagnostics emit, ggml guard |
1278+
| query | 52 | 3 | 17× | ✅ Wave 3-B COMPLETE — watchdog bounded-wait (×3), sequential null guard, JIT corruption sentinel |
1279+
| index | 29 | 1 | 29× | ✅ Wave 3-C COMPLETE — VectorAutoBuffer `noexcept` dtor; 28 restliche pre-existing fixed bestätigt |
1280+
| network | 29 | 4 || ✅ Wave 3-D COMPLETE — command_injection RCE→posix_spawn, TCP health probe, FD-RAII, SO_SNDTIMEO POSIX |
1281+
| **Gesamt** | **179** | **10** | **18×** |**10 CRITICAL + 9 HIGH gefixt** |
1282+
1283+
Haupt-FP-Ursachen: `scope_mismatch` (anon-ns in `namespace themis`), `braces_imbalance@line:1`, `db_connection_leak` auf `shared_ptr`-verwaltete Verbindungen, `no_transit_encryption` bei SDK-TLS.
1284+
Closure-Evidence: `src/storage/WAVE_3A_CLOSURE_EVIDENCE.md`, `src/query/WAVE_3B_CLOSURE_EVIDENCE.md`, `src/index/WAVE_3C_CLOSURE_EVIDENCE.md`, `src/network/WAVE_3D_CLOSURE_EVIDENCE.md`.
1285+
Plan: `src/MODULE_GAP_ANALYSIS_WAVE2.md` (Wave 2 + Wave 3 konsolidiert).
1286+
12721287
**Scanner Roadmap — Next Steps (Phase 7+):**
12731288
- Weitere FP-Reduktion bei dominierenden Regeln: `missing_doxygen_*`, `circular_lock_ordering`
12741289
- Delta-Messung nach jeder Regelwelle (Fast-Scan mit Baseline-Vergleich)

SETUP.md

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -147,7 +147,8 @@ sudo apt-get install -y \
147147
librocksdb-dev \
148148
libzstd-dev \
149149
libfmt-dev \
150-
libspdlog-dev
150+
libspdlog-dev \
151+
libcpp-httplib-dev
151152
```
152153

153154
#### Fedora/RHEL:
@@ -299,6 +300,15 @@ not deterministic.
299300

300301
Alternatively, use the `linux-release` preset with vcpkg, which includes all dependencies.
301302

303+
### Error: "httplib.h: No such file or directory"
304+
305+
**Cause**: Using `community-release` on a system where cpp-httplib is installed only via system packages and the required development package is missing.
306+
307+
**Solution**: Install the cpp-httplib development package:
308+
- Debian/Ubuntu: `sudo apt-get install libcpp-httplib-dev`
309+
- Fedora/RHEL: install the distro package that provides `httplib.h` / `cpp-httplib`
310+
- macOS: use the `linux-release` preset with vcpkg or provide a compatible `cpp-httplib` install
311+
302312
### Build Reproducibility Issues on `linux-release` or `community-release`
303313

304314
**Batch A Gate Status**: [~] In Progress (active Phase-0 gate blockers)
Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
{
2+
"module": "llm",
3+
"scan_timestamp": "2026-08-25T16:12:53Z",
4+
"verifier_version": "wave3-v1.0",
5+
"source_files_scanned": 113,
6+
"summary": {
7+
"total_raw": 155,
8+
"verified_gaps": 5,
9+
"false_positives_removed": 150,
10+
"downgrades": 3,
11+
"severity_distribution": {
12+
"CRITICAL": 2,
13+
"HIGH": 3,
14+
"MEDIUM": 0,
15+
"INFO": 0
16+
},
17+
"fp_breakdown": {
18+
"braces_imbalance_all": 37,
19+
"circular_lock_ordering_all": 108,
20+
"data_race_unconfirmed": 11,
21+
"sql_injection_mock_strings": 7,
22+
"stub_compile_gated": 1,
23+
"other": 0
24+
}
25+
},
26+
"findings": [
27+
{
28+
"id": "LLM-CRIT-001",
29+
"file": "src/llm/docs_assistant.cpp",
30+
"lines": [678, 683],
31+
"pattern": "prompt_injection",
32+
"original_severity": "CRITICAL",
33+
"verified_severity": "CRITICAL",
34+
"classification": "Real Gap",
35+
"rationale": "User-supplied 'topic' and 'error_description' parameters are concatenated directly into LLM prompt strings without passing through PromptPolicy::evaluate() or any other sanitisation. Attacker-controlled input can redirect LLM behaviour, exfiltrate context, or bypass guardrails.",
36+
"code_snippet": "std::string query = \"How do I configure \" + topic + \" in ThemisDB?...\";\nstd::string query = \"I'm experiencing this issue: \" + error_description + \"...\";"
37+
},
38+
{
39+
"id": "LLM-CRIT-002",
40+
"file": "src/llm/ai_orchestrator.cpp",
41+
"lines": [264, 269, 280, 289],
42+
"pattern": "deadlock_risk",
43+
"original_severity": "CRITICAL",
44+
"verified_severity": "CRITICAL",
45+
"classification": "Real Gap",
46+
"rationale": "PluginAdapterApplyService::applyAdapter() acquires non-reentrant mutex_ at line 264 and then calls plugin->unloadLoRA(), path_resolver_(), and plugin->loadLoRA() while holding it. Any of these callbacks may re-enter AIOrchestrator::currentAdapter() or ::isModelLoaded() which also acquire mutex_, producing a non-reentrant deadlock. The trigger path is confirmed: currentAdapter() (line 299) acquires mutex_ unconditionally.",
47+
"code_snippet": "std::lock_guard<std::mutex> lock(mutex_); // line 264 — locked\n...\nconst bool unload_ok = plugin->unloadLoRA(...); // line 269 — external call under lock\nconst auto resolved = path_resolver_(...); // line 280 — callback under lock\nconst bool ok = plugin->loadLoRA(...); // line 289 — external call under lock"
48+
},
49+
{
50+
"id": "LLM-HIGH-001",
51+
"file": "src/llm/model_downloader.cpp",
52+
"lines": [150, 239],
53+
"pattern": "path_traversal",
54+
"original_severity": "CRITICAL",
55+
"verified_severity": "HIGH",
56+
"classification": "Real Gap",
57+
"rationale": "config.model_name is not validated for '../', '/', '\\', or null-byte sequences before constructing filesystem paths. model_name can be supplied via YAML config (line 626) or as an API parameter. Downgraded from CRITICAL because model_name is typically operator-configured rather than directly user-supplied at runtime.",
58+
"code_snippet": "std::string expected_path = config.download_dir + \"/\" + config.model_name + \".gguf\";"
59+
},
60+
{
61+
"id": "LLM-HIGH-002",
62+
"file": "src/llm/model_downloader.cpp",
63+
"lines": [595, 108, 135],
64+
"pattern": "insecure_model_url",
65+
"original_severity": "CRITICAL",
66+
"verified_severity": "HIGH",
67+
"classification": "Real Gap",
68+
"rationale": "Default Ollama URL is http://localhost:11434 (plaintext). validateOllamaUrl() issues only a WARN for non-local HTTP targets and returns true (permitting the connection). Production deployments pointing at a remote Ollama endpoint can transmit model weights over an unencrypted channel. Downgraded from CRITICAL because localhost deployments (the common case) are unaffected.",
69+
"code_snippet": ": \"http://localhost:11434\"; // line 595 — plaintext default\n...\nTHEMIS_WARN(\"plain HTTP used for non-local endpoint\"); // line 135 — warn only, not reject\nreturn true; // always accepts"
70+
},
71+
{
72+
"id": "LLM-HIGH-003",
73+
"file": "src/llm/llm_prefix_cache.cpp",
74+
"lines": [46],
75+
"pattern": "hardcoded_path",
76+
"original_severity": "CRITICAL",
77+
"verified_severity": "HIGH",
78+
"classification": "Real Gap",
79+
"rationale": "Cache directory hardcoded to /tmp/themis_llm_prefix_cache with no configurable override. In multi-instance deployments, all processes share the same /tmp path, causing race conditions on cache files. World-readable /tmp also exposes cached embeddings. Downgraded from CRITICAL because exploitation requires multi-tenant shared-host deployments.",
80+
"code_snippet": "embed_config.cache_dir = \"/tmp/themis_llm_prefix_cache\";"
81+
}
82+
],
83+
"false_positives": [
84+
{
85+
"group": "braces_imbalance",
86+
"count": 37,
87+
"pattern": "braces_imbalance + braces_imbalance_midfile",
88+
"original_severity": "CRITICAL",
89+
"verified_severity": null,
90+
"classification": "False-Positive",
91+
"rationale": "State-machine brace counter confirmed all 113 .cpp files terminate at depth 0. Scanner FP causes: (1) C++14 digit separators like 10'000 parsed as char literals '000' by simple regex; (2) Multi-line raw string literals R\"(...)\" containing unescaped { and } characters; (3) char literals like '\"' and '\\\\' causing string-stripping regex to misidentify string boundaries. All flagged files end correctly with } // namespace llm / } // namespace themis."
92+
},
93+
{
94+
"group": "circular_lock_ordering",
95+
"count": 108,
96+
"pattern": "circular_lock_ordering",
97+
"original_severity": "CRITICAL",
98+
"verified_severity": null,
99+
"classification": "False-Positive",
100+
"rationale": "Heuristic fires on any file using more than one mutex name. Manual inspection of all top mutex-heavy files (inference_engine_enhanced.cpp 70 uses, multi_lora_manager.cpp 65 uses, gpu_memory_manager.cpp 47 uses, async_inference_engine.cpp 45 uses, ml_model_manager.cpp 28 uses) found zero ABBA lock-ordering inversions. Most files use a single mutex_ throughout; those with multiple mutexes acquire them in consistent documented order or in non-overlapping scopes."
101+
},
102+
{
103+
"group": "data_race",
104+
"count": 11,
105+
"pattern": "data_race",
106+
"original_severity": "CRITICAL",
107+
"verified_severity": null,
108+
"classification": "False-Positive",
109+
"rationale": "All shared mutable state in inspected files is protected: counters use std::atomic<>, collections are guarded by std::mutex or std::shared_mutex. gpu_available_ in GPUMemoryManager is written only during constructor (single-threaded) and read-only thereafter. Scanner FP fires on non-const member variables in classes that happen to contain a std::thread member."
110+
},
111+
{
112+
"group": "sql_injection",
113+
"count": 7,
114+
"pattern": "sql_injection",
115+
"original_severity": "CRITICAL",
116+
"verified_severity": null,
117+
"classification": "False-Positive",
118+
"rationale": "All 7 instances are either: (a) hardcoded mock AQL strings in llm_client_default.cpp (test/stub code, no user input concatenated); (b) RPC protocol envelope strings in distributed_training_coordinator.cpp named 'rpc_query' that are not SQL; (c) internal audit log fields in moral_analyzer.cpp. No parameterised SQL construction from user input was found."
119+
},
120+
{
121+
"group": "embedded_llm_stub",
122+
"count": 1,
123+
"pattern": "simulation_stub_marker",
124+
"original_severity": "CRITICAL",
125+
"verified_severity": null,
126+
"classification": "False-Positive",
127+
"rationale": "embedded_llm_stub.cpp stub path (success=true with hardcoded text) is permanently gated by #ifdef THEMIS_LLM_STUB_MODE. Documentation confirms this flag is never set in release build presets. The production else-branch correctly returns success=false with a diagnostic error message."
128+
}
129+
]
130+
}

0 commit comments

Comments
 (0)