Skip to content

Commit 04b82ed

Browse files
Merge pull request #480 from Ratio1/develop
Develop
2 parents 4a7743e + e031bf1 commit 04b82ed

39 files changed

Lines changed: 3494 additions & 246 deletions

AGENTS.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -695,3 +695,30 @@ Entry format:
695695
- Details: `ThHfModelBase` keeps Transformers/PT as the default GPU and fallback path, but CPU-only `HF_RUNTIME=auto` now loads `artifact_manifest.json`, selects a declared ONNX Runtime artifact, downloads only safe allow-patterns, loads schema and contract decoder from HF artifacts, and exposes the decoded artifact contract through the existing text-classifier flow. Business API response shaping now passes through generic model/runtime metadata emitted by serving.
696696
- Verification: `python3 -m unittest extensions.serving.test_th_hf_model_base extensions.serving.test_th_text_classifier extensions.serving.test_th_privacy_filter extensions.business.edge_inference_api.test_text_classifier_inference_api extensions.business.edge_inference_api.test_privacy_filter_inference_api`; `python3 -m py_compile extensions/serving/default_inference/nlp/th_hf_model_base.py extensions/business/edge_inference_api/text_classifier_inference_api.py`; required serving gate `python3 -m unittest extensions.serving.model_testing.test_llm_servings` currently fails at import with `ImportError: cannot import name 'Logger' from 'naeural_core'`.
697697
- Links: `extensions/serving/default_inference/nlp/th_hf_model_base.py`, `extensions/business/edge_inference_api/text_classifier_inference_api.py`, `extensions/serving/test_th_hf_model_base.py`
698+
699+
- ID: `ML-20260723-001`
700+
- Timestamp: `2026-07-23T13:45:20Z`
701+
- Type: `change`
702+
- Summary: dAuth job-secret requests now require signed 120-second timestamp nonces, and GET responses encrypt secret bundles to the authorized runner.
703+
- Criticality: Security protocol change preventing indefinite signed-request/response replay and removing plaintext job secrets from HTTP responses.
704+
- Details: `/add_secrets` and `/get_secrets` validate signed hex-millisecond timestamp nonces and echo them in successful signed responses. `/get_secrets` encrypts the serialized bundle to the signed requester address; clients must verify the response signer and echoed nonce before decrypting.
705+
- Verification: `python -m unittest discover -s extensions/business/dauth -p 'test_*.py'`; cross-repo SDK dAuth client tests.
706+
- Links: `extensions/business/dauth/dauth_mixin.py`, `extensions/business/dauth/dauth_manager.py`
707+
708+
- ID: `ML-20260731-001`
709+
- Timestamp: `2026-07-31T16:48:11Z`
710+
- Type: `change`
711+
- Summary: dAuth job-secret ChainStore writes and minute syncs now target only startup-cached dAuth registry peers.
712+
- Criticality: Secret-replication boundary and recovery behavior across every dAuth server.
713+
- Details: The dAuth manager reads registry ETH addresses once at startup, keeps local service eligibility fixed until restart, and refreshes only ETH-to-internal mappings from local NetMon state. `DAUTH_JOB_SECRETS` writes and 60-second hsync calls disable default/configured ChainStore peers. Known deferred risks: generic ChainStore does not authorize inbound operations by hash namespace, and first-response hsync has no freshness arbitration; production hardening requires an inbound ACL or dedicated authenticated replication protocol plus version-aware merges.
714+
- Verification: `python3 -m unittest discover -s extensions/business/dauth -p 'test_*.py'`; `python3 -m py_compile extensions/business/dauth/dauth_registry.py extensions/business/dauth/dauth_manager.py extensions/business/dauth/dauth_mixin.py extensions/business/dauth/test_dauth_registry_gating.py extensions/business/dauth/test_dauth_secret_routing.py`; `git diff --check`
715+
- Links: `extensions/business/dauth/dauth_registry.py`, `extensions/business/dauth/dauth_manager.py`, `extensions/business/dauth/dauth_mixin.py`
716+
717+
- ID: `ML-20260803-001`
718+
- Timestamp: `2026-08-03T16:09:26Z`
719+
- Type: `change`
720+
- Summary: dAuth server eligibility and secret-replication peers now refresh from the on-chain registry every hour; secret hsync runs every 10 minutes.
721+
- Criticality: Authorization revocation and secret-replication routing across every dAuth server.
722+
- Details: Lifecycle pause/resume predicates perform the rate-limited registry refresh without adding RPC calls to endpoint request paths. Successful reads remain cached for one hour; failed reads clear cached peers, fail closed, and retry after one minute. Registry reads are synchronous and rely on the SDK Web3 provider to return or time out. A removed local node causes the web app to pause and become unready; readiness returns only after a resumed Uvicorn process reports startup. Remaining dAuth nodes replace their cached peer set on their next hourly refresh. The inbound namespace authorization and version-aware hsync limitations from `ML-20260731-001` remain open.
723+
- Verification: `python3 -m unittest discover -s extensions/business/dauth -p 'test_*.py'`; `python3 -m py_compile extensions/business/dauth/dauth_registry.py extensions/business/dauth/dauth_manager.py extensions/business/dauth/dauth_mixin.py extensions/business/dauth/test_dauth_registry_gating.py extensions/business/dauth/test_dauth_secret_routing.py`; `git diff --check`
724+
- Links: `extensions/business/dauth/dauth_manager.py`, `extensions/business/dauth/test_dauth_registry_gating.py`

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -223,7 +223,7 @@ For further information, visit our website at [https://ratio1.ai](https://ratio1
223223

224224
## Project Financing Disclaimer
225225

226-
This project incorporates open-source components developed with the support of financing grants **SMIS 143488** and **SMIS 156084**, provided by the Romanian Competitiveness Operational Programme. We extend our gratitude for this support, which has been instrumental in advancing our work and enabling us to share these resources with the community.
226+
This project incorporates open-source components developed with the support of financing grants **SOLIS SMIS 143488** and **ReDeN SMIS 156084**, provided by the Romanian Competitiveness Operational Programme. We extend our gratitude for this support, which has been instrumental in advancing our work and enabling us to share these resources with the community.
227227

228228
The content and information within this repository reflect the authors' views and do not necessarily represent those of the funding agencies. The grants have specifically supported certain aspects of this open-source project, facilitating broader dissemination and collaborative development.
229229

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
"""Connection-window aggregation and signal semantics shared across scan levels."""
2+
3+
RESPONSIVE_CONNECTION_OUTCOMES = frozenset(("connected", "refused", "reset"))
4+
MIN_QUALIFIED_WINDOW_ATTEMPTS = 5
5+
BLOCKING_BASELINE_RATE = 0.8
6+
BLOCKING_RESPONSE_RATE = 0.2
7+
THROTTLING_DROP_RATIO = 0.7
8+
9+
10+
def detect_connection_signals(windows: list | None) -> dict:
11+
"""Derive blocking/throttling signals from sufficiently sampled windows."""
12+
qualified = []
13+
for window in windows or []:
14+
attempts = window.get("attempts")
15+
if not isinstance(attempts, (int, float)) or attempts < MIN_QUALIFIED_WINDOW_ATTEMPTS:
16+
continue
17+
responsive_count = window.get("responsive_count")
18+
response_rate = window.get("response_rate")
19+
if response_rate is None and responsive_count is not None:
20+
response_rate = responsive_count / attempts
21+
if responsive_count is None and response_rate is not None:
22+
responsive_count = response_rate * attempts
23+
if response_rate is None or responsive_count is None:
24+
continue
25+
qualified.append({
26+
"attempts": attempts,
27+
"responsive_count": responsive_count,
28+
"response_rate": response_rate,
29+
})
30+
31+
blocking = any(
32+
previous["response_rate"] >= BLOCKING_BASELINE_RATE
33+
and current["response_rate"] <= BLOCKING_RESPONSE_RATE
34+
for previous, current in zip(qualified, qualified[1:])
35+
)
36+
37+
throttling = False
38+
if len(qualified) >= 4:
39+
first_attempts = sum(window["attempts"] for window in qualified[:2])
40+
last_attempts = sum(window["attempts"] for window in qualified[-2:])
41+
first_responsive = sum(window["responsive_count"] for window in qualified[:2])
42+
last_responsive = sum(window["responsive_count"] for window in qualified[-2:])
43+
baseline_rate = first_responsive / first_attempts
44+
later_rate = last_responsive / last_attempts
45+
throttling = (
46+
later_rate > BLOCKING_RESPONSE_RATE
47+
and later_rate < baseline_rate * THROTTLING_DROP_RATIO
48+
)
49+
50+
return {
51+
"rate_limiting_detected": throttling,
52+
"blocking_detected": blocking,
53+
}
54+
55+
56+
def merge_connection_windows(metrics_list: list) -> list | None:
57+
"""Merge aligned count-bearing windows, excluding unverifiable legacy samples."""
58+
grouped = {}
59+
legacy_fallback = None
60+
for metrics in metrics_list:
61+
windows = metrics.get("success_rate_over_time") or []
62+
if legacy_fallback is None or len(windows) > len(legacy_fallback):
63+
legacy_fallback = windows
64+
for window in windows:
65+
attempts = window.get("attempts")
66+
responsive_count = window.get("responsive_count")
67+
if attempts is None or attempts <= 0:
68+
continue
69+
if responsive_count is None:
70+
response_rate = window.get("response_rate")
71+
if response_rate is None:
72+
continue
73+
responsive_count = round(response_rate * attempts)
74+
key = (window.get("window_start", 0), window.get("window_end", 0))
75+
bucket = grouped.setdefault(key, {
76+
"attempts": 0,
77+
"responsive_count": 0,
78+
"connected_weight": 0.0,
79+
})
80+
bucket["attempts"] += attempts
81+
bucket["responsive_count"] += responsive_count
82+
bucket["connected_weight"] += window.get("success_rate", 0) * attempts
83+
84+
if not grouped:
85+
return legacy_fallback or None
86+
87+
merged = []
88+
for (window_start, window_end), counts in sorted(grouped.items()):
89+
attempts = counts["attempts"]
90+
merged.append({
91+
"window_start": window_start,
92+
"window_end": window_end,
93+
"success_rate": round(counts["connected_weight"] / attempts, 3),
94+
"attempts": attempts,
95+
"responsive_count": counts["responsive_count"],
96+
"response_rate": round(counts["responsive_count"] / attempts, 3),
97+
})
98+
return merged

extensions/business/cybersec/red_mesh/constants.py

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -182,6 +182,25 @@ class ScanType(str, Enum):
182182
PORT_ORDER_SHUFFLE = "SHUFFLE"
183183
PORT_ORDER_SEQUENTIAL = "SEQUENTIAL"
184184

185+
# Network target-response timeout profiles. Standard preserves every existing
186+
# call-site timeout; Thorough expands ordinary waits without changing probe
187+
# breadth, pacing, or timing-sensitive detection thresholds.
188+
TIMEOUT_PROFILE_STANDARD = "STANDARD"
189+
TIMEOUT_PROFILE_THOROUGH = "THOROUGH"
190+
TIMEOUT_PROFILES = frozenset({TIMEOUT_PROFILE_STANDARD, TIMEOUT_PROFILE_THOROUGH})
191+
192+
193+
def normalize_timeout_profile(value):
194+
normalized = str(value or TIMEOUT_PROFILE_STANDARD).strip().upper()
195+
return normalized if normalized in TIMEOUT_PROFILES else TIMEOUT_PROFILE_STANDARD
196+
197+
198+
def resolve_target_response_timeout(timeout_profile, standard_timeout):
199+
"""Resolve an ordinary network target-response maximum wait in seconds."""
200+
if normalize_timeout_profile(timeout_profile) != TIMEOUT_PROFILE_THOROUGH:
201+
return standard_timeout
202+
return round(min(float(standard_timeout) * 3, 15.0), 3)
203+
185204
# LLM Agent API status constants
186205
LLM_API_STATUS_OK = "ok"
187206
LLM_API_STATUS_ERROR = "error"
@@ -298,6 +317,28 @@ class ScanType(str, Enum):
298317

299318
ALL_PORTS = list(range(1, 65536))
300319

320+
# =====================================================================
321+
# Geographic vantage-point comparison mode
322+
# =====================================================================
323+
# When comparison mode is enabled every selected node runs the SAME "comparison
324+
# tier" of ports (so results can be compared across countries). The distribution
325+
# choice controls what the tier is:
326+
# - SLICE (default): the tier is the standard COMMON_PORTS bundle; the
327+
# operator's chosen range is split across nodes for coverage (not compared).
328+
# - MIRROR: the tier is the whole chosen range (plus COMMON_PORTS), so every
329+
# port is compared across countries, at N x the work.
330+
331+
# Standard webapp/graybox feature bundle always run (mirrored to every node) in
332+
# comparison mode so cross-country response divergence is meaningful even if the
333+
# operator narrowed the selection. These are safe, unauthenticated checks; their
334+
# methods are force-enabled (removed from excluded_features) when comparison
335+
# mode is on. Referenced by feature id in FEATURE_CATALOG.
336+
COMPARISON_GRAYBOX_BUNDLE_FEATURE_IDS = [
337+
"web_discovery",
338+
"web_hardening",
339+
"web_api_exposure",
340+
]
341+
301342
# =====================================================================
302343
# Risk score computation
303344
# =====================================================================

extensions/business/cybersec/red_mesh/mixins/live_progress.py

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
from ..graybox.models import GrayboxCredentialSet
99
from ..models import WorkerProgress
1010
from ..constants import PHASE_ORDER, GRAYBOX_PHASE_ORDER
11+
from ..connection_metrics import detect_connection_signals, merge_connection_windows
1112

1213
DEFAULT_PROGRESS_PUBLISH_INTERVAL = 30.0
1314

@@ -175,7 +176,6 @@ def _status_rank(v):
175176
all_phases[phase] = max(all_phases.get(phase, 0), dur)
176177
if all_phases:
177178
merged["phase_durations"] = all_phases
178-
longest = max(metrics_list, key=lambda m: m.get("total_duration", 0))
179179
# Merge stats distributions (response_times, port_scan_delays)
180180
# Use weighted mean, global min/max, approximate p95/p99 from max of per-thread values
181181
for stats_field in ("response_times", "port_scan_delays"):
@@ -193,12 +193,12 @@ def _status_rank(v):
193193
"p99": round(max(s.get("p99", 0) for s in stats_list), 4),
194194
"count": total_count,
195195
}
196-
# Success rate over time: take from the longest-running thread
197-
if longest.get("success_rate_over_time"):
198-
merged["success_rate_over_time"] = longest["success_rate_over_time"]
199-
# Detection flags (any thread detecting = True)
200-
merged["rate_limiting_detected"] = any(m.get("rate_limiting_detected") for m in metrics_list)
201-
merged["blocking_detected"] = any(m.get("blocking_detected") for m in metrics_list)
196+
# Merge aligned traffic evidence, then derive node signals from the combined
197+
# sample counts. Legacy windows without counts remain visible but unverified.
198+
connection_windows = merge_connection_windows(metrics_list)
199+
if connection_windows:
200+
merged["success_rate_over_time"] = connection_windows
201+
merged.update(detect_connection_signals(connection_windows))
202202
# Open port details: union, deduplicate by port
203203
all_details = []
204204
seen_ports = set()

0 commit comments

Comments
 (0)