Skip to content

PROJECT_EXPLANATION_GUARD_AI.md

Latest commit

 

History

History
1437 lines (1152 loc) · 61.7 KB

PROJECT_EXPLANATION_GUARD_AI.md

File metadata and controls

1437 lines (1152 loc) · 61.7 KB
Raw

Inceptrix / Guard AI — Complete Project Explanation

Tagline: "Defeating AI using AI" A unified deepfake detection, media protection, and blockchain authenticity platform.

Table of Contents

  1. Project Overview
  2. Repository Layout
  3. Backend — Core Architecture
  4. Backend — Routes (All API Endpoints)
  5. Backend — Services
  6. Backend — Middleware
  7. ML / v2 — Detection Pipeline
  8. ML / v2 — XAI & Forensics Modules
  9. ML / Protection Pipeline
  10. Video Detection Module
  11. Mint / NFT Pipeline
  12. Frontend — Next.js Application
  13. Frontend — Pages
  14. Frontend — Components
  15. Frontend — API Client (lib/api.ts)
  16. Chrome Extension (guard-ai-extension)
  17. Parallax Sub-project
  18. Database Schema (MongoDB)
  19. Configuration & Environment Variables
  20. Celery Task Queue
  21. WebSocket / SocketIO Real-Time Events
  22. Utility Scripts & Tools
  23. External Integrations
  24. Data Flow Diagrams
  25. Technology Stack Summary

1. Project Overview

Inceptrix / Guard AI is a full-stack AI security platform with three pillars:

Pillar What it does
Detection Analyzes images, videos, and audio for deepfake manipulation using a multi-layer pipeline: EfficientNet-B4 CNN + 6 XAI methods + FFT frequency analysis + PRNU forensic fingerprint + Gemini 2.5 Pro VLM analysis
Protection Applies imperceptible adversarial perturbations to images that disrupt AI-based manipulation (Stable Diffusion, Qwen Image Edit, etc.) using three protection algorithms: MSAP, BlurGuard, and AberrationEngine
Authentication Mints ERC-721 NFTs on the Polygon blockchain with IPFS-hosted metadata, creating tamper-proof provenance certificates for verified authentic media

Secondary features include:

  • Real-time video call deepfake monitoring via WebRTC with server-side signaling
  • Chrome Extension v2 for browser-based scanning on any website
  • Dashboard with scan history, protection records, and NFT certificates
  • Diffusion compare tool to visually demonstrate protection effectiveness

2. Repository Layout

inceptrix/
│
├── backend/                   ← Flask REST API + WebSocket server
│   ├── app.py                 ← Entry point; registers all blueprints
│   ├── config.py              ← Dataclass-based centralized config
│   ├── database.py            ← MongoDB sync + async operations (motor/pymongo)
│   ├── models.py              ← MongoDB document schemas (dataclasses)
│   ├── celery_config.py       ← Celery + Redis task queue setup
│   ├── middleware/
│   │   ├── auth.py            ← JWT + API key decorators (demo-mode override)
│   │   ├── logging.py         ← Per-request logging with request IDs
│   │   └── rate_limit.py      ← Redis-based rate limiting
│   ├── routes/
│   │   ├── detection.py       ← /api/detection/* endpoints
│   │   ├── protection.py      ← /api/protection/* endpoints
│   │   ├── nft.py             ← /api/nft/* endpoints
│   │   ├── auth.py            ← /api/auth/* endpoints
│   │   ├── analytics.py       ← /api/analytics/* endpoints
│   │   ├── api_keys.py        ← /api/keys/* endpoints
│   │   ├── video_call.py      ← /api/video-call/* endpoints (WebRTC signaling)
│   │   └── extension_api.py   ← /api/extension/* Chrome extension endpoints
│   ├── services/
│   │   ├── detection_service.py     ← Full 8-stage image detection pipeline
│   │   ├── video_detection_service.py ← Video detection wrapper
│   │   ├── audio_detection_service.py ← Dual-model audio detection
│   │   ├── protection_service.py    ← MSAP/BlurGuard/Aberration protection
│   │   ├── nft_service.py           ← IPFS/Pinata + NFT minting
│   │   ├── auth_service.py          ← JWT issuance, user login/register
│   │   └── diffusion_compare_service.py ← img2img Stable Diffusion comparison
│   ├── plugins/
│   │   └── ensemble_service.py     ← Ensemble voting across multiple models
│   ├── sockets/
│   │   └── __init__.py             ← SocketIO event handlers
│   ├── tasks/
│   │   ├── detection_tasks.py      ← Celery async detection tasks
│   │   ├── protection_tasks.py     ← Celery async protection tasks
│   │   └── nft_tasks.py            ← Celery async NFT tasks
│   └── utils/
│       ├── crypto.py               ← Password hashing, token generation
│       ├── image.py                ← Image resize/format utilities
│       ├── validators.py           ← Input validation helpers
│       └── video.py                ← Video frame extraction helpers
│
├── v2/                        ← Core ML detection engine (EfficientNet-B4)
│   ├── model.py               ← DeepfakeDetector class + transforms
│   ├── pipeline.py            ← UnifiedPipeline (FAST/STANDARD/FULL modes)
│   ├── gradcam.py             ← GradCAM++ implementation
│   ├── explainer.py           ← Multi-method XAI runner (IG, SHAP, LIME, etc.)
│   ├── frequency.py           ← FFT power spectrum analysis
│   ├── forensics.py           ← PRNU fingerprint, gradient saliency, counterfactual
│   ├── metadata.py            ← EXIF/XMP/C2PA metadata extractor
│   ├── gemini_client.py       ← Gemini 2.5 Pro client (round-robin key rotation)
│   ├── xai_gemini.py          ← Gemini forensic analysis + bounding boxes
│   ├── dataset.py             ← Dataset loader for training
│   ├── train.py               ← Training loop
│   ├── inference.py           ← CLI inference
│   ├── evaluation.py          ← Evaluation metrics
│   └── video_analyzer.py      ← Frame-level video analysis
│
├── ml/                        ← Protection algorithms
│   ├── protection/
│   │   ├── __init__.py        ← MediaProtector + ProtectionConfig (MSAP)
│   │   ├── blurguard.py       ← BlurGuard protector (PGD + adaptive blur)
│   │   ├── aberration_engine.py ← AberrationEngine (4 novel formulae)
│   │   └── unified_protector.py ← UnifiedProtector (all 3 combined)
│   └── detection/
│       └── audio-deepfake-detection/
│           └── audio_classifier.h5  ← TF/Keras CNN (ASVspoof 2019 trained)
│
├── video_detection/           ← Video deepfake detection module
│   ├── __init__.py            ← create_video_detector() factory
│   ├── video_detector.py      ← VideoDetector class with aggregation logic
│   ├── frame_extractor.py     ← OpenCV-based frame extraction
│   └── frame_sampler.py       ← Statistical sampling strategies
│
├── mint/                      ← NFT minting pipeline
│   ├── __init__.py
│   ├── pipeline.py            ← MintPipeline orchestrator
│   ├── pinata.py              ← Pinata IPFS client
│   ├── metadata.py            ← ERC-721 metadata builder
│   └── blockchain.py          ← Web3.py Polygon blockchain minter
│
├── frontend/                  ← Next.js 14 frontend (App Router)
│   ├── app/
│   │   ├── layout.tsx         ← Root layout with ThemeProvider
│   │   ├── page.tsx           ← Landing/home page
│   │   ├── detection/page.tsx ← Detection page (image/video/audio)
│   │   ├── protection/page.tsx ← Protection page + diffusion compare
│   │   ├── dashboard/page.tsx ← User dashboard
│   │   └── video-call/page.tsx ← WebRTC video call with deepfake overlay
│   ├── components/
│   │   ├── navbar.tsx         ← Top navigation bar
│   │   ├── hero-section.tsx   ← Landing hero with particle animation
│   │   ├── stats-section.tsx  ← Stats cards
│   │   ├── features-section.tsx ← Feature highlights
│   │   ├── developer-experience.tsx ← Developer API showcase
│   │   ├── code-section.tsx   ← Code example display
│   │   ├── docs-section.tsx   ← Documentation links
│   │   ├── pricing-section.tsx ← Pricing tiers
│   │   ├── enterprise-section.tsx ← Enterprise features
│   │   ├── cta-section.tsx    ← Call to action
│   │   ├── footer.tsx         ← Site footer
│   │   ├── theme-provider.tsx ← Dark/light mode provider
│   │   └── ui/                ← 40+ shadcn/ui components
│   ├── lib/
│   │   ├── api.ts             ← Full backend API client (typed)
│   │   └── utils.ts           ← Tailwind utility helpers
│   └── hooks/
│       ├── use-mobile.ts      ← Mobile breakpoint hook
│       └── use-toast.ts       ← Toast notification hook
│
├── guard-ai-extension/        ← Chrome Extension v2 (MV3)
│   ├── manifest.json          ← Extension manifest
│   ├── background/
│   │   └── service-worker.js  ← Background service worker
│   ├── content/
│   │   ├── content.js         ← Content script injected on all pages
│   │   └── content.css        ← Content script styles
│   └── popup/
│       └── popup.html         ← Extension popup UI
│
├── Parallax/                  ← Older multi-modal detection sub-project
│   ├── Backend/
│   │   ├── app.py             ← Flask API on port 5002
│   │   ├── controllers/       ← Route controllers per modality
│   │   └── processors/        ← AI model processors per modality
│   ├── Frontend/              ← Static HTML/JS frontend
│   └── Web-Extension/         ← MV3 Chrome extension (older)
│
├── checkpoints_v2/            ← Model weights
│   ├── best.pth               ← EfficientNet-B4 trained weights
│   ├── evaluation.json        ← Evaluation metrics + optimal threshold
│   └── history.json           ← Training history
│
├── backend/protected_images/  ← Saved protected images (output)
├── approach3/                 ← Research config (protection approach 3)
├── approch2/                  ← Research config (protection approach 2)
├── benchmarks/                ← Protection benchmark scripts
├── examples/                  ← Usage examples
└── tests/                     ← Test suite

3. Backend — Core Architecture

backend/app.py

The Flask application entry point. It:

  1. Creates the Flask app with MAX_CONTENT_LENGTH = 50MB.
  2. Loads BlurGuard protection defaults from environment variables:
    • PROTECTION_DEFAULT_METHOD (default: blurguard)
    • BLURGUARD_DEFAULT_STRENGTH (default: high)
    • BLURGUARD_DEFAULT_EOT_SAMPLES (default: 5)
    • BLURGUARD_DEFAULT_LAMBDA_REPULSION (default: 2.5)
    • BLURGUARD_DEFAULT_LAMBDA_INSTABILITY (default: 1.0)
    • BLURGUARD_DEFAULT_LAMBDA_TV (default: 0.06)
  3. Enables CORS for all /api/* routes accepting Content-Type, Authorization, and X-API-Key.
  4. Creates SocketIO instance with threading async mode, 60s ping timeout.
  5. Registers 7 Flask Blueprints:
    • /api/detection — Detection routes
    • /api/protection — Protection routes
    • /api/nft — NFT routes
    • /api/auth — Auth routes
    • /api/video-call — Video call routes
    • /api/analytics — Analytics routes
    • /api/keys — API key routes
    • /api/extension — Chrome extension routes
  6. Instantiates all services (DetectionService, ProtectionService, NFTService, AuthService).
  7. Initializes SocketIO handlers.
  8. Exposes health check at GET /health and API docs at GET /api/docs.
  9. Provides CLI commands: flask init-db, flask create-admin.

backend/config.py

Dataclass-based configuration system with sub-configs:

Config Class Key Fields
DatabaseConfig mongodb_uri, mongodb_db_name, redis_url
AuthConfig jwt_secret, jwt_algorithm, access_token_expiry (3600s), refresh_token_expiry (604800s)
MLConfig device (cuda/cpu), detection_model_path, vlm_model_path, batch_size=4
BlockchainConfig polygon_rpc_url, polygon_testnet_rpc_url, wallet_private_key, nft_contract_address
IPFSConfig api_url, gateway_url, pinata_jwt
RateLimitConfig default_limit=100/min, detection_limit=20/min, protection_limit=10/min
AppConfig debug, host=0.0.0.0, port=5000, secret_key, max_content_length=50MB

backend/celery_config.py

Configures Celery with Redis as both broker and backend. Defines tasks for async processing of detection, protection, and NFT operations.

4. Backend — Routes (All API Endpoints)

Detection Routes (/api/detection)

Method Path Description
POST /api/detection/analyze Analyze image for deepfakes. Accepts: multipart/form-data (file), JSON {image: base64}, JSON {url: string}
POST /api/detection/analyze/batch Analyze multiple images. Accepts multiple file uploads or array of base64
POST /api/detection/analyze/video Analyze video file (mp4/avi/mov/mkv/webm). Returns per-frame details with temporal analysis
POST /api/detection/analyze/audio Analyze audio for voice deepfakes (wav/mp3/flac/ogg/m4a/aac/wma). Returns dual-model results + spectrogram
GET /api/detection/explain/<analysis_id> Retrieve explanation for a previous analysis
GET /api/detection/heatmap/<analysis_id> Retrieve GradCAM heatmap for a previous analysis

Protection Routes (/api/protection)

Method Path Description
POST /api/protection/protect Apply protection to image. Methods: unified, blurguard, aberration, msap. All 20+ parameters configurable
GET /api/protection/protect/download/<protection_id> Download protected image by ID
POST /api/protection/protect/batch Protect multiple images
POST /api/protection/fingerprint Generate unique perceptual fingerprint for image authentication
POST /api/protection/verify Verify image against stored fingerprint
POST /api/protection/quality-check Compute SSIM/LPIPS/MSE/PSNR between original and protected images
POST /api/protection/diffusion-compare Run img2img Stable Diffusion on original and protected, return both outputs for comparison

NFT Routes (/api/nft)

Method Path Description
POST /api/nft/mint Mint ERC-721 authenticity NFT. Uploads to IPFS/Pinata, simulates on-chain mint on Polygon
GET /api/nft/verify/<token_id> Verify NFT authenticity certificate
GET /api/nft/certificates/<wallet_address> Get all certificates for a wallet
POST /api/nft/ipfs/upload Upload file to IPFS (Pinata)
GET /api/nft/ipfs/<cid> Retrieve content from IPFS by CID
GET /api/nft/metadata/<token_id> Get NFT metadata JSON

Auth Routes (/api/auth)

Method Path Description
POST /api/auth/register Register new user (email + password + optional name, wallet)
POST /api/auth/login Login → returns access_token + refresh_token
POST /api/auth/refresh Refresh access token using refresh token
POST /api/auth/logout Invalidate tokens (requires Bearer token)
GET /api/auth/me Get current user info (requires token)
PUT /api/auth/me Update user info (requires token)
POST /api/auth/wallet/connect Connect Ethereum wallet to account
POST /api/auth/api-key Generate API key for programmatic access

Analytics Routes (/api/analytics)

Method Path Description
GET /api/analytics/user/stats User statistics (total scans, fake detections, protections, certificates). Falls back to demo mode when MongoDB unavailable
GET /api/analytics/user/history Paginated scan + protection history. Params: page, limit, type=scans/protections/all
GET /api/analytics/user/certificates User's NFT certificates. Falls back to in-memory cache
GET /api/analytics/detection/<analysis_id> Detailed detection record
GET /api/analytics/protection/<protection_id> Detailed protection record
GET /api/analytics/verify?hash=<sha256> Verify image by hash/fingerprint across all sources (DB + memory cache)
GET /api/analytics/verify/fingerprint/<fingerprint_id> Verify by fingerprint ID

Video Call Routes (/api/video-call)

Method Path Description
POST /api/video-call/private/create Create ephemeral private call room. Returns 6-char room_code + 8-char safety_code + host peer_id
POST /api/video-call/private/join Join private room (requires both codes). Returns guest peer_id
POST /api/video-call/private/signal Push WebRTC signaling packet (offer/answer/ice-candidate/hangup) to other participant
POST /api/video-call/private/poll Poll queued signaling packets for this participant
POST /api/video-call/private/leave Leave room, notifies other participant
POST /api/video-call/session/start Start deepfake monitoring session (configurable interval/sensitivity/threshold)
POST /api/video-call/session/<id>/analyze Analyze single base64 frame within session
GET /api/video-call/session/<id>/status Get session stats + threat level
POST /api/video-call/session/<id>/stop End session, returns summary
POST /api/video-call/quick-check Single frame check without session

Extension Routes (/api/extension)

Method Path Description
POST /api/extension/analyze Optimized detection for Chrome extension. Accepts file upload or base64 JSON
POST /api/extension/protect Optimized protection for Chrome extension

API Key Routes (/api/keys)

CRUD operations for API key management: list, create, revoke, test.

5. Backend — Services

DetectionService (services/detection_service.py)

The core 8-stage analysis pipeline for images:

Stage 1: EfficientNet-B4 Classification
         → real_prob, fake_prob, threshold comparison
Stage 2: Grad-CAM++
         → Heatmap overlay, raw CAM for bounding box extraction
Stage 3: Multi-method XAI (5 methods)
         → Integrated Gradients, Kernel SHAP, LIME, Guided Backprop, GradCAM++
         → SuspiciousRegionDetector on each heatmap
Stage 4: Gradient Saliency
         → Mean/max activation scores
Stage 5: Counterfactual Explanation
         → Masks suspicious regions, re-predicts → measures causal delta
Stage 6: FFT Frequency Analysis
         → Power spectrum slope deviation from natural -2.0 slope
         → Spectral peaks count
Stage 7: PRNU Forensic Fingerprint
         → Noise energy, spectral flatness, PRNU score
Stage 8½: Gemini 2.5 Pro VLM Analysis
         → Sends image to Gemini, receives structured JSON with:
           verdict, confidence, manipulation_type, explanation
           regions (bbox + label + description), attributes
         → Draws annotated image with bounding boxes
Stage 8: Metadata / C2PA Extraction
         → EXIF fields, XMP metadata, C2PA markers, AI tool signatures
         → Software fingerprints (Stable Diffusion, DALL-E, etc.)

Combined Verdict: Weighted score
  local_model: 30%   | fft: 15%  | prnu: 15%
  region_density: 15% | gemini: 25%

Key methods:

  • analyze_image(image) → Full 8-stage analysis, returns rich dict
  • quick_analyze(image) → EfficientNet only, for real-time/WebSocket use
  • analyze_video(video_path, sample_rate) → Frame-by-frame sampling
  • analyze_url(url) → Downloads image, runs full analysis
  • analyze_frame(frame_b64) → Wrapper for WebSocket frame analysis
  • get_explanation(analysis_id) → Retrieve cached explanation
  • get_heatmap(analysis_id) → Retrieve cached GradCAM heatmap

VideoDetectionService (services/video_detection_service.py)

Wraps video_detection.VideoDetector. Lazy-loads on first request.

Config:

  • sampling_strategy = "stratified" (stratified/uniform/scene-change)
  • num_sample_frames = 16
  • run_gemini = True
  • gemini_max_frames = 4

Returns: aggregated verdict, per-frame details, temporal segments, XAI/PRNU/FFT aggregates, Gemini explanation.

AudioDetectionService (services/audio_detection_service.py)

Dual-model audio deepfake detection pipeline:

Model 1 — TF/Keras CNN (ASVspoof 2019):

  • Loads audio_classifier.h5 (128 Mel bins × 109 time steps)
  • Labels: spoof (0) / bonafide (1)
  • 60% weight in combined verdict

Model 2 — HuggingFace Pipeline:

  • Model: motheecreator/Deepfake-audio-detection
  • 40% weight in combined verdict

Additional outputs:

  • Mel-spectrogram visualization (base64 PNG)
  • GradCAM-style activation heatmap (4-panel: spectrogram, heatmap, overlay, thresholded)
  • Audio waveform visualization
  • Spectral features: centroid, bandwidth, rolloff, ZCR, 13 MFCCs
  • Audio metadata: duration, sample rate, channels, RMS energy, dynamic range

ProtectionService (services/protection_service.py)

Exposes 4 protection methods:

1. protect_unified(image, strength, options)

  • Combines all 3 algorithms: MSAP + BlurGuard + AberrationEngine
  • Strongest protection available
  • On CPU: downscales to 256px max dimension

2. protect_against_qwen_edit(image, strength, options) (BlurGuard)

  • PGD adversarial attack with adaptive per-region Gaussian blur
  • Power spectrum regularization (lambda_freq)
  • EOT (Expectation over Transformations) sampling
  • Latent repulsion loss to push features away from real distribution
  • Instability loss to increase temporal variance
  • Optionally appends AberrationEngine perturbations

3. protect_with_aberration(image, strength, options) (AberrationEngine)

  • F1: HVS-Weighted Anisotropic Frequency Perturbation
  • F2: Orthogonal Null-Space Semantic Projection (ONP)
  • F3: Jacobian Singular-Value Subversion (JSVS)
  • F4: Psycho-Visual Adaptive Step Decay Gate (PVASD)

4. protect_image(image, options) (MSAP)

  • Frequency Domain Cloaking (epsilon_freq)
  • Semantic Disruption (epsilon_sem)
  • Latent Space Poisoning (tau_latent)
  • Quality gating with SSIM

Cache: All protected images stored in backend/protected_images/ with in-memory _protected_cache dict keyed by protection_id.

Quality metrics: SSIM, LPIPS, MSE, PSNR, perturbation L∞ norm.

NFTService (services/nft_service.py)

Manages IPFS uploads and NFT lifecycle:

  • upload_to_ipfs(content, filename) — Tries Pinata first (JWT or API key), falls back to mock CID (SHA-256 based)
  • mint_authenticity_nft(image_data, wallet_address, metadata) — Uploads image + JSON metadata to IPFS, creates ERC-721 metadata structure with Guard AI attributes, simulates transaction hash
  • verify_nft(token_id) — Returns NFT data + IPFS availability check
  • get_user_certificates(wallet_address) — Lists certificates per wallet
  • transfer_certificate(token_id, from, to) — Transfers NFT ownership
  • In-memory caches: _ipfs_cache, _nft_cache, _certificates

AuthService (services/auth_service.py)

  • register_user(email, password, name, wallet_address) — Hashes password with bcrypt, stores in MongoDB
  • login_user(email, password) — Verifies password, issues JWT access + refresh tokens
  • verify_token(token) — Decodes JWT, returns payload
  • refresh_access_token(refresh_token) — Issues new access token
  • connect_wallet(user_id, wallet_address, signature) — Links wallet to account
  • generate_api_key(user_id) — Creates random API key, stores SHA-256 hash

DiffusionCompareService (services/diffusion_compare_service.py)

Runs Stable Diffusion img2img on both original and protected images with the same prompt + seed. Returns base64-encoded outputs for side-by-side comparison showing how protection disrupts AI manipulation.

6. Backend — Middleware

middleware/auth.py

Decorators:

Decorator Behavior
@auth_required Demo mode active: Sets g.user_id = "demo_developer_id" without checking tokens. Production: verify JWT
@api_key_required Extracts X-API-Key header, SHA-256 hashes it, looks up in MongoDB
@optional_auth Demo mode: Same as auth_required. Production: sets None if no token
@require_permission(perm) Checks API key permissions list for specific permission

Note: auth_required and optional_auth are currently in demo mode (bypass real auth) to allow local development without login.

middleware/logging.py

Assigns request_id UUID to each request. Logs method, path, status, duration in ms. Provides request_logger with log_error(error, context) method.

middleware/rate_limit.py

Redis-based sliding window rate limiter. Limits:

  • Default: 100 req/min
  • Detection: 20 req/min
  • Protection: 10 req/min

7. ML / v2 — Detection Pipeline

v2/model.py — DeepfakeDetector

Architecture:
  Backbone: EfficientNet-B4 (tf_efficientnet_b4.ns_jft_in1k — Noisy Student)
  Features: 1792-dimensional global average pool
  Head: Dropout(0.35) → Linear(1792→512) → ReLU → Dropout(0.21) → Linear(512→2)
  Total params: ~20M
  Input: 380×380 RGB, ImageNet normalization
  Output: logits + softmax → {real_prob, fake_prob}

Training transforms: RandomResizedCrop, RandomHorizontalFlip, RandomRotation(15°), ColorJitter, RandomGrayscale, RandomErasing.

Inference transforms: Resize(380, 380), ToTensor, Normalize(ImageNet).

Methods:

  • forward(x) → raw logits
  • predict_proba(x) → {logits, probs, real_prob, fake_prob}
  • extract_features(x) → 1792-d feature vector
  • save(path) / load(path, device) → checkpoint management
  • freeze_backbone_except_last(n_blocks) → partial fine-tuning

Checkpoint: checkpoints_v2/best.pth. Optimal threshold loaded from checkpoints_v2/evaluation.json.

v2/pipeline.py — UnifiedPipeline

Three processing modes:

Mode Stages Speed
FAST EfficientNet only <1s
STANDARD + GradCAM++ + FFT ~3s
FULL + Gemini + Multi-XAI + Metadata ~15-30s

Combined verdict weights:

  • local_model: 40%, gemini: 45%, fft: 15%

Features: image hash caching (LRU, 100 entries), batch analysis, video frame analysis with EMA temporal smoothing (α=0.7).

v2/gemini_client.py — GeminiClient

Thread-safe Gemini 2.5 Pro client with round-robin load balancing across up to 3 API keys (GEMINI_API_KEY_1/2/3).

  • Automatic retry with exponential backoff (base 2s, max 30s)
  • Key failover on error
  • analyze_image(image, prompt) → raw text
  • analyze_image_json(image, prompt) → parsed JSON dict
  • JSON repair: strips markdown fences, fixes trailing commas, salvages partial JSON

8. ML / v2 — XAI & Forensics Modules

v2/gradcam.py — GradCAMPlusPlus

Grad-CAM++ implementation using PyTorch hooks on model.backbone.conv_head. Generates:

  • Raw CAM heatmap (normalized 0-1)
  • Overlay image (alpha-blended with original)

v2/explainer.py — ExplainableAI

Runs multiple XAI techniques on the same input:

Technique Library What it produces
GRADCAM_PLUS Custom hooks Spatial activation heatmap
INTEGRATED_GRADIENTS Captum Attribution map with noise tunnel
KERNEL_SHAP Captum Superpixel Shapley values
LIME lime library Local surrogate model explanation
GUIDED_BACKPROP Captum Positive-gradient saliency

SuspiciousRegionDetector: Thresholds each heatmap, finds contours (OpenCV), scores regions by area + activation intensity. Classifies severity: low/medium/high/critical.

ExplanationOutput fields: heatmap, overlay_image, suspicious_regions (list of SuspiciousRegion with bbox/confidence/severity/reason/method), importance_scores, reasoning.

v2/forensics.py

Class What it does
ForensicFingerprint PRNU (Photo Response Non-Uniformity) analysis. Extracts camera sensor noise, computes noise_energy, spectral_flatness, prnu_score. High score = authentic sensor pattern
GradientSaliency Computes gradient × input saliency map for target class
CounterfactualExplainer Fills suspicious regions with blur/inpaint, re-predicts. Measures delta (change in fake_prob) and causal_score

v2/frequency.py — FFT Analysis

Computes 2D FFT of grayscale image, converts to power spectrum, fits log-log regression.

  • Natural image slope: ~-2.0 (1/f² noise)
  • Anomaly score: deviation from expected slope
  • Spectral peaks: count of abnormal high-frequency peaks
  • Verdict: SUSPICIOUS if score > 0.3

v2/metadata.py — MetadataExtractor

Extracts from image files:

  • EXIF data (GPS, camera model, software, datetime)
  • XMP metadata snippets
  • C2PA (Content Credentials) markers by binary scan
  • AI generation markers: stable diffusion, midjourney, dall-e, runway, pika, sora, gemini, gpt, adobe firefly, etc.
  • SHA-256 hash, file size

v2/xai_gemini.py — GeminiXAI

Sends image to Gemini with a structured forensic prompt requesting:

{
  "verdict": "real|fake",
  "confidence": 0.0-1.0,
  "manipulation_type": "string",
  "explanation": "string",
  "regions": [
    {
      "bbox": [x1, y1, x2, y2],  // normalized 0-1
      "label": "string",
      "confidence": 0.0-1.0,
      "description": "string"
    }
  ],
  "attributes": {
    "attribute_name": {
      "score": 0.0-1.0,
      "description": "string"
    }
  }
}

draw_annotations(image_path, gemini_data, output_path) — Draws colored bounding boxes and labels on the image using PIL/OpenCV. Color coding: red=high confidence fake, yellow=medium.

9. ML / Protection Pipeline

ml/protection/__init__.py — MSAP (Multi-layer Semantic Adversarial Perturbation)

The original 3-layer protection algorithm:

Layer 1: Frequency Domain Cloaking (epsilon_freq=0.03)

  • DCT/FFT-based perturbations in frequency domain
  • Disrupts spectral signatures AI models rely on
  • Invisible to human eye (high-frequency perturbation)

Layer 2: Semantic Disruption (epsilon_sem=0.05)

  • Attacks semantic feature representations
  • Pushes feature vectors away from manipulation-friendly directions
  • Uses gradient-based optimization against a surrogate semantic encoder

Layer 3: Latent Space Poisoning (tau_latent=0.1)

  • Targets latent space of generative models (VAE encoder)
  • Embeds adversarial signals that cause reconstruction artifacts
  • Quality gating: rejects if SSIM drops below min_ssim=0.95

ProtectionOutput: protected_image, ssim_score, lpips_score, perturbation_norm, info dict with per-layer diagnostics.

ml/protection/blurguard.py — BlurGuardProtector

Specifically designed to defeat Qwen Image Edit and similar diffusion-based editing models.

Algorithm:

  1. Adaptive per-region Gaussian blur warmup — Applies variable sigma blur to different image regions, then reverses to use as adversarial seed
  2. PGD attack (Projected Gradient Descent) — Optimizes perturbation with:
    • lambda_freq: Power spectrum regularization (forces perturbation to look like natural frequency noise)
    • lambda_repulsion: Latent repulsion loss (pushes latent code away from the real distribution)
    • lambda_instability: Temporal instability loss (maximizes frame-to-frame variance for video stability)
    • lambda_tv: Total variation smoothness penalty
  3. EOT sampling — Averages gradients over eot_samples random augmentations for robustness
  4. Optional AberrationEngine appended for extra coverage

BlurGuardConfig fields:

epsilon             # L∞ perturbation bound (default: 16/255)
num_pgd_steps       # PGD iterations (GPU: 100-200, CPU: 10-20)
num_blur_warmup_steps
lambda_freq, eot_samples
lambda_repulsion, lambda_instability, lambda_tv
use_aberration_engine
aberration_* (14 parameters for AberrationEngine sub-config)

ml/protection/aberration_engine.py — AberrationEngine

4 novel mathematical formulae for adversarial protection:

F1: HVS-Weighted Anisotropic Frequency Perturbation

  • Applies frequency perturbations weighted by Human Visual System contrast sensitivity function
  • Anisotropic — different strength in horizontal vs. vertical frequency bands
  • Parameter: aberration_epsilon_freq, aberration_phase_mix

F2: Orthogonal Null-Space Semantic Projection (ONP)

  • Computes semantic feature gradient direction
  • Projects perturbation onto orthogonal null-space
  • Disrupts semantic encoding without changing perceptual quality
  • Parameters: aberration_epsilon_sem, onp_num_iterations

F3: Jacobian Singular-Value Subversion (JSVS)

  • Estimates Jacobian matrix of the model's latent encoder via finite differences
  • Attacks singular values to collapse information in latent subspace
  • Parameters: aberration_tau_latent, aberration_latent_jacobian_samples, aberration_latent_components, aberration_latent_tail_start

F4: Psycho-Visual Adaptive Step Decay Gate (PVASD)

  • Monitors loss progress and applies exponential decay when convergence is detected
  • Psycho-visual masking: larger steps in visually busy regions, smaller in flat regions
  • Parameters: aberration_pv_loss_threshold, aberration_pv_kappa

ml/protection/unified_protector.py — UnifiedProtector

Combines all three approaches sequentially:

  1. Apply MSAP (Frequency + Semantic + Latent)
  2. Apply BlurGuard on top
  3. Apply AberrationEngine on the result

The strongest protection — all perturbations are additive and quality-gated.

10. Video Detection Module

video_detection/frame_extractor.py — FrameExtractor

OpenCV-based video frame extraction:

  • VideoMeta: fps, total_frames, duration_sec, width, height, codec
  • FrameRecord: frame_index, timestamp, pil_image, is_scene_change
  • Scene change detection using histogram correlation

video_detection/frame_sampler.py — FrameSampler

Sampling strategies:

Strategy Description
UNIFORM Evenly spaced frames
STRATIFIED Uniform + extra coverage at scene changes
SCENE_BASED One frame per detected scene change

Returns SamplingReport with: strategy, num_sampled, temporal_coverage %.

video_detection/video_detector.py — VideoDetector

Main orchestrator:

  1. Extracts frames via FrameExtractor
  2. Samples via FrameSampler
  3. For each sampled frame:
    • Runs DetectionService.analyze_image(frame) (full 8-stage)
    • Collects: Grad-CAM overlay, XAI regions, FFT, PRNU, Gemini annotations
  4. Aggregates results:
    • Weighted majority vote for video-level verdict
    • Temporal consistency score (confidence standard deviation)
    • Segment verdicts (every N frames)
    • Aggregated XAI regions across frames
    • Aggregated Gemini explanation from representative frames

VideoDetectionResult.to_dict() returns the full serializable payload.

11. Mint / NFT Pipeline

mint/pinata.py — PinataClient

Wraps Pinata IPFS pinning API:

  • pin_file(file_path, name, metadata) — Upload file to Pinata
  • pin_file_bytes(data, filename, name, metadata) — Upload bytes
  • pin_json(json_content, name, metadata) — Upload JSON metadata
  • Supports JWT auth or legacy API key + secret
  • Returns PinataUploadResult with cid, ipfs_url, pinata_url, success/error

mint/metadata.py — MetadataBuilder + NFTMetadata

Builds ERC-721 OpenSea-compatible metadata JSON:

{
  "name": "Guard AI Verified Media",
  "description": "...",
  "image": "ipfs://<CID>",
  "attributes": [
    {"trait_type": "Protection Level", "value": "High"},
    {"trait_type": "SSIM Score", "display_type": "number", "value": 0.97},
    {"trait_type": "Verified", "value": "True"},
    {"trait_type": "Guard AI Version", "value": "2.0"},
    ...
  ],
  "guard_ai": {
    "image_hash_sha256": "...",
    "verification_timestamp": "...",
    "protection_metrics": {...},
    "xai_summary": {...},
    "ipfs_cid": "..."
  }
}

mint/blockchain.py — BlockchainMinter

Web3.py-based Polygon blockchain interaction:

  • Connects to Polygon mainnet or testnet RPC
  • Calls AuthenticityNFT.safeMint(to, tokenId, imageHash, metadataCid, protectionLevel) on the deployed smart contract
  • Returns MintResult with token_id, transaction_hash, block_number, gas_used, explorer_url

mint/pipeline.py — MintPipeline

Orchestrates the full mint flow in sequence:

  1. SHA-256 hash of image
  2. Upload image to Pinata (IPFS)
  3. Build ERC-721 metadata JSON
  4. Upload metadata JSON to Pinata (IPFS)
  5. (FULL mode only) Call blockchain minter

Two entry points:

  • mint(image_path, title, description, protection_level, creator, wallet_address, protection_metrics, xai_summary)
  • mint_from_bytes(image_bytes, filename, **kwargs) — For in-memory images

mint_cli.py

Command-line interface for the mint pipeline. Supports --mode IPFS_ONLY/FULL, --title, --description, --wallet, --level.

12. Frontend — Next.js Application

Framework: Next.js 14 with App Router Styling: Tailwind CSS UI Library: shadcn/ui (40+ components built on Radix UI) Animations: Framer Motion Icons: Lucide React State: React useState/useRef/useCallback (no Redux) API: Custom typed fetch client in lib/api.ts Theme: next-themes (dark mode default)

frontend/app/layout.tsx

Root layout wrapping all pages with ThemeProvider (dark mode forced). Sets metadata: title "Guard AI", description, viewport.

13. Frontend — Pages

app/page.tsx — Landing Page (Home)

Marketing landing page composed of:

  • Navbar — Top navigation
  • HeroSection — Animated typing effect + particle canvas + CTA buttons
  • StatsSection — Key metrics (accuracy, speed, etc.)
  • FeaturesSection — Feature cards
  • DeveloperExperience — API usage showcase
  • CodeSection — Code example display
  • DocsSection — Documentation links
  • PricingSection — Free/Pro/Enterprise tiers
  • EnterpriseSection — Enterprise feature highlights
  • CTASection — Sign up call to action
  • Footer — Site footer

app/detection/page.tsx — Detection Page

File/media type selector: Image | Video | Audio tabs

State machine: idle → uploading → model → xai → forensics → complete | error

Drag-and-drop upload with preview.

For images — Result tabs:

  1. Overview — Verdict badge (REAL/FAKE), confidence gauge, probabilities, bounding boxes on image, reasoning text, breakdown table
  2. Gemini — VLM verdict, explanation, annotated image with bounding boxes, attributes table
  3. XAI — 6 technique overlays (Grad-CAM++, IG, SHAP, LIME, Guided BP, Gradient Saliency) with suspicious regions list
  4. FFT — Frequency score, slope, spectral peaks, verdict badge
  5. Forensic — PRNU noise energy, spectral flatness, score, interpretation
  6. Metadata — File info, EXIF field count, C2PA markers, XMP snippets, AI tool markers
  7. Counterfactual — Original vs patched fake probability, causal score, message

For videos — Result tabs:

  1. Overview — Video-level verdict, confidence, frame statistics, temporal consistency
  2. Timeline — Per-segment verdicts with fake ratios
  3. Gemini — Aggregated explanation + regions table
  4. Frames — Per-frame gallery with GradCAM overlays and Gemini annotations

For audio — Result tabs:

  1. Overview — Verdict, dual-model results (CNN + HF), fake/real probabilities
  2. Spectrogram — Mel spectrogram + GradCAM-style activation heatmap + waveform
  3. Metadata — Audio file metadata + spectral features (centroid, bandwidth, MFCCs)

app/protection/page.tsx — Protection Page

Method selector: Unified | MSAP | BlurGuard

State machine: idle → uploading → freq_cloaking → semantic_disruption → latent_poisoning → blurguard_attack → aberration_engine → quality_gate → fingerprinting → complete | error

Each step shown with progress animation.

Advanced settings panel (collapsible):

  • Epsilon controls (freq, sem, latent)
  • BlurGuard parameters (eot_samples, lambda_repulsion, instability, tv)
  • AberrationEngine toggle + all 9 aberration parameters

Results display:

  • Side-by-side original vs protected preview
  • SSIM / LPIPS metrics
  • Fingerprint ID display
  • Download button
  • NFT minting button

Diffusion Compare section:

  • Text prompt input
  • Strength slider (0-1)
  • Runs img2img on both original and protected
  • Shows 4 images: original input → original output, protected input → protected output
  • Demonstrates how protection breaks AI manipulation

app/dashboard/page.tsx — Dashboard

Overview stats cards: Total Scans, Fake Detections, Protected Images, NFT Certificates

4 tabs:

  1. Scan History — Table of past detection results (filename, prediction, confidence, date, type)
  2. Protected Images — Table of protected files (filename, SSIM, fingerprint ID, date, has certificate)
  3. Certificates — NFT certificates list (title, token ID, IPFS hash, date, protection level) with Pinata link
  4. Verify — Hash search input: enter SHA-256 or fingerprint to verify authenticity + see certificate details + Pinata URL

Auto-refreshes on load. Demo mode fallback if database unavailable.

app/video-call/page.tsx — Video Call Page

WebRTC private video call with deepfake detection overlay.

Flow:

  1. Host: Click "New Private Call" → backend creates room with 6-char room_code + 8-char safety_code → Share codes with guest
  2. Guest: Enter room_code + safety_code → Joins room
  3. Both: WebRTC peer connection established via polling-based signaling
  4. Detection overlay: Periodic frame captures analyzed for deepfakes, badge shown on video

DRM-style screen capture prevention (useCaptureShield hook):

  • Intercepts all known screenshot keyboard shortcuts (PrtScn, Ctrl+Shift+S, Cmd+Shift+3/4/5, etc.)
  • Displays full-screen blackout overlay for 800ms when capture is detected
  • CSS content-visibility and hardware-layer isolation
  • Blocks right-click, drag, picture-in-picture on video element
  • Pauses video on tab visibility change

WebRTC implementation:

  • ICE server: Google STUN (stun:stun.l.google.com:19302)
  • Polling-based signaling (no WebSocket dependency): polls /api/video-call/private/poll every 500ms
  • Supports: offer/answer/ice-candidate/hangup signal types
  • Auto-hangup on leave

14. Frontend — Components

Marketing Components

Component Description
Navbar Logo + nav links (Features, Docs, Pricing) + "Get Started" button
HeroSection Animated typewriter "Defeating AI using AI", particle field canvas animation, stats badges, CTA buttons
StatsSection 4 metric cards with animated counters
FeaturesSection Grid of feature cards with icons (Detection, Protection, NFT, Video Call, Extension, API)
DeveloperExperience API usage highlights with code snippets
CodeSection Syntax-highlighted code example carousel
DocsSection Documentation links grid
PricingSection Free/Pro/Enterprise tier cards with feature lists
EnterpriseSection Enterprise feature highlights
CTASection Sign-up call to action with background gradient
Footer Links + social + copyright
ThemeProvider next-themes provider (forced dark)

UI Components (shadcn/ui based, components/ui/)

40+ pre-built components: accordion, alert, alert-dialog, aspect-ratio, avatar, badge, breadcrumb, button, button-group, calendar, card, carousel, chart, checkbox, collapsible, command, context-menu, dialog, drawer, dropdown-menu, empty, field, form, hover-card, input, input-group, input-otp, item, kbd, label, menubar, navigation-menu, pagination, popover, progress, radio-group, resizable, scroll-area, select, separator, sheet, sidebar, skeleton, slider, sonner, spinner, switch, table, tabs, textarea, toast, toaster, toggle, toggle-group, tooltip

15. Frontend — API Client (lib/api.ts)

Typed TypeScript client for all backend routes. Key patterns:

  • Base URL: NEXT_PUBLIC_API_URL env var or http://localhost:5000
  • Auth: reads localStorage.get("auth_token"), adds Authorization: Bearer header
  • Error handling: throws with backend error message

Exported functions:

Function Backend Route Returns
detectMedia(file) POST /api/detection/analyze DetectionResult
detectBase64(base64) POST /api/detection/analyze DetectionResult
detectVideo(file) POST /api/detection/analyze/video VideoDetectionResult
detectAudio(file) POST /api/detection/analyze/audio AudioDetectionResult
protectMedia(file, options) POST /api/protection/protect ProtectionResult
getProtectedDownloadUrl(id) — Download URL string
diffusionCompare(original, protectedB64, prompt, strength, seed) POST /api/protection/diffusion-compare DiffusionCompareResult
getUserStats() GET /api/analytics/user/stats UserStats
getScanHistory() GET /api/analytics/user/history?type=scans ScanRecord[]
getProtectionHistory() GET /api/analytics/user/history?type=protections ProtectionRecord[]
getCertificates() GET /api/analytics/user/certificates CertificateRecord[]
verifyByHash(hash) GET /api/analytics/verify?hash= VerifyHashResult
startVideoSession(options) POST /api/video-call/session/start VideoSessionConfig
analyzeVideoFrame(sessionId, frame, frameNumber) POST /api/video-call/session/:id/analyze FrameAnalysisResult
stopVideoSession(sessionId) POST /api/video-call/session/:id/stop SessionSummary
createPrivateCallRoom() POST /api/video-call/private/create PrivateRoomSession
joinPrivateCallRoom({roomCode, safetyCode}) POST /api/video-call/private/join PrivateRoomSession
sendPrivateCallSignal({...}) POST /api/video-call/private/signal {success, delivered}
pollPrivateCallSignals({...}) POST /api/video-call/private/poll PrivateSignalPollResult
leavePrivateCallRoom({...}) POST /api/video-call/private/leave {success}

Key TypeScript interfaces:

  • DetectionResult — Full analysis: analysis_id, prediction, confidence, probabilities, bounding_boxes, explanation, vlm_analysis, gradcam_overlay, xai_techniques[], counterfactual, fft_analysis, forensic_fingerprint, metadata, combined_score, gemini_analysis, gemini_annotated
  • VideoDetectionResult — verdict, confidence, frames, temporal, sampling, video, xai, gemini, frame_details[]
  • AudioDetectionResult — verdict, confidence, cnn_classification, hf_classification, metadata, spectral_features, spectrogram, gradcam_visualization, waveform
  • ProtectionResult — protection_id, protected_image, metrics, fingerprint
  • DiffusionCompareResult — original_output, protected_output, original_input, protected_input

16. Chrome Extension (guard-ai-extension)

Manifest Version: 3 Version: 2.0.0

Permissions:

  • activeTab, scripting, storage, contextMenus, notifications, downloads
  • host_permissions: <all_urls> (analyze any image on any website)

Architecture:

File Role
background/service-worker.js Background service worker. Handles context menu events, coordinates analysis requests to backend, manages notification sending
content/content.js Content script injected on every page. Scans images on the page, adds Guard AI badge overlays on analyzed images, intercepts right-click on images to offer "Analyze with Guard AI"
content/content.css Badge overlay styles
popup/popup.html Extension popup UI (shows current page scan status, quick scan button, settings)
icons/ Extension icons at 16, 32, 48, 128px

API used: /api/extension/analyze and /api/extension/protect (lightweight endpoints reusing same DetectionService and ProtectionService instances).

17. Parallax Sub-project

A separate, older multi-modal deepfake detection sub-project at Parallax/.

Backend (Parallax/Backend/app.py — Flask on port 5002):

Blueprint Route Prefix Controller
video_bp /api/video controllers/video_controller.py
image_bp /api/image controllers/image_controller.py
audio_bp /api/audio controllers/audio_controller.py
text_bp /api/text controllers/text_controller.py

Processors:

Processor Model Technique
processors/video/rppg_model.py rPPG (Remote Photo-Plethysmography) Detects fake pulse signals in video
processors/video/lipsync_model.py Lip sync model Checks audio-visual lip synchronization
processors/audio/audio_model.py Audio deepfake classifier Pre-trained audio classifier (joblib)
processors/text/text_model.py Text classifier AI-generated text detection

Video processor (video_processor.py): Runs rPPG and LipSync in parallel threads, majority vote for final label.

Audio processor (audio_processor.py): Wraps audio model prediction.

Pre-trained models:

  • processors/audio/models/audio_label_encoder_optimized.joblib
  • processors/audio/models/audio_scaler_optimizer.joblib

Frontend (Parallax/Frontend/): Static HTML/CSS/JS page using vanilla JS.

Parallax Web Extension (Parallax/Web-Extension/):

  • Manifest V3 Chrome extension
  • background.js — Service worker
  • content.js — Content script
  • popup.js — Popup logic
  • utils/analysis.js — Analysis utilities

18. Database Schema (MongoDB)

Database name: guardai (configurable via MONGODB_DB_NAME)

Collections

users

_id: string (UUID)          email: string (unique)
password_hash: string       name: string
wallet_address: string      api_key: string
is_active: bool             is_verified: bool
scan_count: int             protection_count: int
created_at: datetime        updated_at: datetime

Indexes: email (unique), wallet_address (sparse), api_key (sparse)

analyses

_id: string (UUID)          user_id: string
image_hash: string          prediction: string
confidence: float           manipulation_type: string
ml_results: dict            explanation: dict
heatmap_url: string         vlm_analysis: dict
regions: list               attributes: dict
created_at: datetime

Indexes: user_id, image_hash, created_at

protections

_id: string (UUID)          user_id: string
original_hash: string       protected_hash: string
protection_level: int       settings: dict
ssim: float                 lpips: float
frequency_strength: float   semantic_strength: float
latent_strength: float      protected_image_path: string
fingerprint_id: string      nft_token_id: string
created_at: datetime

Indexes: user_id, original_hash, fingerprint_id (sparse)

certificates

_id: string (UUID)          user_id: string
wallet_address: string      token_id: string (unique)
image_hash: string          ipfs_cid: string
metadata: dict              contract_address: string
transaction_hash: string    block_number: int
protection_id: string       protection_level: int
created_at: datetime

Indexes: user_id, wallet_address, token_id (unique), image_hash

api_keys

_id: string (UUID)          user_id: string
key_hash: string (unique)   name: string
is_active: bool             permissions: list
rate_limit: int             usage_count: int
last_used: datetime         created_at: datetime

Indexes: user_id, key_hash (unique)

scan_history

_id: string (UUID)          user_id: string
analysis_id: string         source: string  # web/extension/api
result: string              ip_address: string
user_agent: string          created_at: datetime

Indexes: user_id, created_at

19. Configuration & Environment Variables

File: backend/.env

# Application
SECRET_KEY=guard-ai-secret-key
FLASK_DEBUG=True
HOST=0.0.0.0
PORT=5000

# Database
MONGODB_URI=mongodb://localhost:27017/guardai
MONGODB_DB_NAME=guardai
REDIS_URL=redis://localhost:6379/0
MONGO_SERVER_SELECTION_TIMEOUT_MS=2000

# Authentication
JWT_SECRET_KEY=your-jwt-secret
JWT_ACCESS_TOKEN_EXPIRES=3600
JWT_REFRESH_TOKEN_EXPIRES=604800

# ML
DEVICE=cuda                           # or cpu
DETECTION_MODEL_PATH=checkpoints_v2/best.pth

# Gemini API (round-robin across 3 keys)
GEMINI_API_KEY_1=AIza...
GEMINI_API_KEY_2=AIza...
GEMINI_API_KEY_3=AIza...
GEMINI_MODEL=gemini-2.5-pro

# Blockchain
POLYGON_RPC_URL=https://polygon-rpc.com
POLYGON_TESTNET_RPC_URL=https://rpc-amoy.polygon.technology/
WALLET_PRIVATE_KEY=0x...
NFT_CONTRACT_ADDRESS=0x...

# IPFS / Pinata
PINATA_JWT=eyJ...
PINATA_API_KEY=...
PINATA_API_SECRET=...
IPFS_GATEWAY_URL=https://gateway.pinata.cloud/ipfs/

# Rate Limits
RATE_LIMIT_DEFAULT=100
RATE_LIMIT_DETECTION=20
RATE_LIMIT_PROTECTION=10

# BlurGuard Defaults
PROTECTION_DEFAULT_METHOD=blurguard
BLURGUARD_DEFAULT_STRENGTH=high
BLURGUARD_DEFAULT_EOT_SAMPLES=5
BLURGUARD_DEFAULT_LAMBDA_REPULSION=2.5
BLURGUARD_DEFAULT_LAMBDA_INSTABILITY=1.0
BLURGUARD_DEFAULT_LAMBDA_TV=0.06

Frontend: frontend/.env.local

NEXT_PUBLIC_API_URL=http://localhost:5000

20. Celery Task Queue

File: backend/celery_config.py

Uses Redis as broker and result backend. Async tasks:

tasks/detection_tasks.py:

  • detect_image_task(image_path, user_id) — Run full detection pipeline asynchronously
  • detect_batch_task(image_paths, user_id) — Batch detection

tasks/protection_tasks.py:

  • protect_image_task(image_path, options, user_id) — Run protection asynchronously

tasks/nft_tasks.py:

  • mint_nft_task(image_data, wallet_address, metadata) — Mint NFT asynchronously

21. WebSocket / SocketIO Real-Time Events

File: backend/sockets/__init__.py

init_socket_handlers(socketio, detection_service) registers these events:

Event Direction Description
analyze_frame Client → Server Send base64 frame for quick analysis. Server emits frame_result
start_stream_analysis Client → Server Begin continuous frame detection
stop_stream_analysis Client → Server End continuous detection
protection_progress Server → Client Broadcast protection step updates
detection_progress Server → Client Broadcast detection stage updates
connect — Connection established
disconnect — Connection closed

Real-time video call uses polling (not WebSocket) via /api/video-call/private/poll every 500ms.

22. Utility Scripts & Tools

Script Description
infer.py CLI inference on single image
infer_ensemble.py Ensemble inference across multiple models
infer_vit.py ViT-based inference (alternative backbone)
infer_xai.py Full XAI inference + save visualizations
detect_video.py CLI video deepfake detection
prepare_dataset.py Dataset preparation for training
prepare_hf_dataset.py HuggingFace dataset preparation
mint_cli.py CLI for NFT minting
check.py Model/pipeline sanity check
_check_syntax.py Syntax checker for backend files
patch_frontend.py Frontend patch utility
patch_route.py Route patch utility
test_aberration_engine.py Unit test for AberrationEngine
test_blurguard_cpu.py CPU performance test for BlurGuard
test_api.py API endpoint integration tests
benchmarks/protection_benchmarks.py Protection quality benchmarks
examples/protection_examples.py Protection usage examples
tests/test_protection.py Protection algorithm unit tests
v2/train.py Model training script
v2/evaluation.py Model evaluation metrics

ML research directories:

  • approach3/ — Research config for protection approach 3
  • approch2/ — Research config for protection approach 2
  • checkpoints_v2/evaluation.json — Saved evaluation metrics including optimal threshold (0.84)
  • checkpoints_v2/history.json — Training loss/accuracy history

23. External Integrations

Service Usage Config
Google Gemini 2.5 Pro VLM forensic analysis of images, structured JSON with bounding boxes GEMINI_API_KEY_1/2/3
Pinata IPFS Store protected images + NFT metadata on IPFS PINATA_JWT / PINATA_API_KEY
Polygon Blockchain Mint ERC-721 AuthenticityNFT tokens WALLET_PRIVATE_KEY, NFT_CONTRACT_ADDRESS, POLYGON_RPC_URL
MongoDB Primary database for users, analyses, protections, NFTs MONGODB_URI
Redis Rate limiting + Celery task queue REDIS_URL
HuggingFace motheecreator/Deepfake-audio-detection audio classification model Downloaded via transformers
timm EfficientNet-B4 tf_efficientnet_b4.ns_jft_in1k pretrained weights Auto-downloaded
Google STUN WebRTC ICE negotiation for video calls stun:stun.l.google.com:19302

24. Data Flow Diagrams

Image Detection Flow

User uploads image
        ↓
POST /api/detection/analyze
        ↓
DetectionService.analyze_image()
        ↓
  Stage 1: EfficientNet-B4 → fake_prob (0-1)
  Stage 2: GradCAM++ → heatmap + bounding boxes
  Stage 3: IG + SHAP + LIME + Guided BP + Saliency → 5 overlays + suspicious regions
  Stage 5: Counterfactual → causal delta
  Stage 6: FFT → slope + anomaly score
  Stage 7: PRNU → sensor noise fingerprint
  Stage 8½: Gemini 2.5 Pro → VLM forensic analysis + bbox JSON
  Stage 8: Metadata → EXIF + C2PA + AI markers
        ↓
  Weighted combination:
    local_model(30%) + fft(15%) + prnu(15%) + region_density(15%) + gemini(25%)
        ↓
  Final verdict: REAL | FAKE + confidence
        ↓
Return JSON with all stages, overlays, and Gemini annotated image

Protection Flow

User uploads image + method choice
        ↓
POST /api/protection/protect
        ↓
ProtectionService.protect_unified() / protect_against_qwen_edit() / protect_with_aberration()
        ↓
  (BlurGuard example)
  Adaptive blur warmup
        ↓
  PGD attack (100 steps):
    loss = adversarial_loss + λ_freq*spectrum_loss + λ_repulsion*repulsion_loss
           + λ_instability*instability_loss + λ_tv*tv_loss
    gradient averaged over eot_samples augmentations
        ↓
  AberrationEngine (optional):
    F1: HVS-weighted freq perturbation
    F2: ONP semantic projection
    F3: JSVS latent subversion
    F4: PVASD adaptive gate
        ↓
  Quality check: SSIM + LPIPS
        ↓
  Save to protected_images/<id>.png
  Generate fingerprint (SHA-256)
        ↓
Return protected_image (base64) + metrics + fingerprint

NFT Minting Flow

Protected image + wallet address
        ↓
POST /api/nft/mint
        ↓
NFTService.mint_authenticity_nft()
        ↓
  Upload image to Pinata → get image CID
        ↓
  Build ERC-721 metadata JSON (name, image, attributes, guard_ai provenance)
        ↓
  Upload metadata to Pinata → get metadata CID
        ↓
  Simulate on-chain mint (or real Web3.py call if credentials present)
        ↓
Return token_id + transaction_hash + ipfs_cid + opensea_url

25. Technology Stack Summary

Backend

Layer Technology
Web Framework Flask + Flask-CORS + Flask-SocketIO
Database MongoDB (motor async + pymongo sync)
Cache/Queue Redis
Task Queue Celery
Auth JWT (PyJWT) + bcrypt
ML Framework PyTorch + timm + torchvision
DL Models EfficientNet-B4, TF/Keras CNN, HuggingFace transformers
XAI Captum (IG, SHAP, Guided BP), LIME, custom Grad-CAM++
Computer Vision OpenCV, Pillow
Audio librosa, soundfile, TensorFlow/Keras
Blockchain Web3.py
IPFS Pinata API
VLM Google Gemini 2.5 Pro (google-generativeai)
Diffusion Stable Diffusion (via diffusers)

Frontend

Layer Technology
Framework Next.js 14 (App Router)
Language TypeScript
Styling Tailwind CSS
UI Components shadcn/ui (Radix UI)
Animations Framer Motion
Icons Lucide React
Real-time WebRTC (native browser API) + polling signaling
Theme next-themes

Infrastructure

Component Technology
Database MongoDB
Cache Redis
IPFS Storage Pinata Cloud
Blockchain Polygon (mainnet + Amoy testnet)
AI Models Local (EfficientNet-B4) + Cloud (Gemini API)

This document covers every module, feature, API endpoint, data model, and integration in the Inceptrix / Guard AI platform as of March 2026.