v1.2.0 - Core Engine Optimization & Semantic Analytics | (Sprints 1-3) + Review Feedback/Regression Fixes (#11)

* feat: Enhance analytics with TF-IDF calculation and CSV report generation

- Added TF-IDF analysis to diagnostic analytics for keyword extraction.
- Implemented CSV report generation in the delivery module.
- Improved file ingestion with caching and fingerprinting for efficiency.
- Enhanced predictive analytics with weighted moving average forecasting.
- Updated prescriptive analytics to handle missing metadata more gracefully.
- Introduced GitHub Actions CI pipeline for automated testing across multiple Node.js versions.

* feat: complete v1.2.0 pipeline (concurrency, memoization, tf-idf, and cross-linking)

* docs: update readme usage flags and architectural pipeline notes

- Added advanced CLI flags (--workers, --clear-cache, --format=csv) to the root README.md usage scope.
- Updated docs/architecture.md to detail v1.2.0 pipeline enhancements, including multithreaded worker pool mechanics and semantic vector cross-linking via TF-IDF / Cosine Similarity.
- Verified all documentation structures and ran local test runner pipelines cleanly.

* addition of usage section and formatting readme

* readme formatting

* chore: remove analytics cache file

* chore: add .analytics_cache.json to .gitignore

* fix: validate workers argument and handle symlinks in file ingestion

Co-authored-by: Copilot <copilot@github.com>

* fix: address PR review security and concurrency feedback

* chore: add explicit workflow token permissions

* fix(analytics): base forecast on doc dates and optimize worker IPC memory

- Fixes domain logic in predictive analytics by switching the timeline basis from the OS file modification time to actual parsed document dates. This prevents modern download timestamps from invalidating historical UAP forecasting.
- Resolves worker IPC performance bottlenecks by calculating word frequencies directly inside the worker thread pool rather than passing massive raw string arrays across the boundary.
- Mitigates main-thread blocking in diagnostic analytics by capping the O(N²) TF-IDF cosine similarity matrix calculations to a maximum of 500 files.
- Adds backwards-compatibility layers in descriptive and diagnostic modules to gracefully handle legacy cache formats without crashing.
- Refines watch mode path exclusions in the index file to use a strictly scoped regex for the data exports directory.

* fix review feedback gaps

* fix workflow token permissions

* fix pipeline/watch/cache regressions and modernize test workflow

* Address latest PR review thread regressions

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

Author: 3 people

.github/workflows/test.yml

@@ -0,0 +1,32 @@
+name: Node.js CI Pipeline
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        node-version: [20.x, 22.x]
+
+    steps:
+    - uses: actions/checkout@v3
+    
+    - name: Use Node.js ${{ matrix.node-version }}
+      uses: actions/setup-node@v3
+      with:
+        node-version: ${{ matrix.node-version }}
+        cache: 'npm'
+        
+    - name: Clean Install and Test
+      run: |
+        npm ci
+        npm test
+        npm run docs:check

.gitignore

@@ -32,6 +32,7 @@ Thumbs.db
 .env.local
 copilot-chat-history.json
 *.traineddata
+.analytics_cache.json
 
 # =========================
 # Bot Specific: Data & Media

.husky/pre-commit

@@ -0,0 +1,6 @@
+#!/usr/bin/env sh
+
+npm run docs:generate || exit 1
+git add docs/ || exit 1
+npm test || exit 1
+npm run docs:check || exit 1

README.md

@@ -80,14 +80,17 @@ The current Node ingestion pipeline only analyzes text-oriented files.
 | `.csv` | Ingested by the active Node pipeline |
 | `.log` | Ingested by the active Node pipeline |
 | `.pdf` | Ingested by the active Node pipeline |
+| `.png` | Ingested by the active Node pipeline |
+| `.jpg` | Ingested by the active Node pipeline |
+| `.jpeg` | Ingested by the active Node pipeline |
 <!-- GENERATED:supported-file-types:END -->
 
 ## Repository Layout
 
 <!-- GENERATED:repo-layout:START -->
 - `src/index.js` — Node CLI entry point.
 - `src/pipeline.js` — Pipeline coordinator that assembles all analytics tiers.
-- `src/ingestion/file-ingestion.js` — Read-only recursive file ingestion for supported text files.
+- `src/ingestion/file-ingestion.js` — Read-only recursive file ingestion for supported files.
 - `src/analytics/` — Descriptive, diagnostic, predictive, and prescriptive analytics modules.
 - `test/pipeline.test.js` — Node test coverage for core pipeline behavior.
 - `docs/architecture.md` — Hand-authored architecture overview for current and planned system design.
@@ -125,6 +128,99 @@ The bot must never modify, move, or delete ingested source files. Ingestion is r
 - When adding analytics, classify behavior under one of the four analytics tiers.
 - Update [docs/architecture.md](docs/architecture.md) when implementation changes affect current-vs-planned system boundaries.
 
+
+<br>
+
+
+
+## ⚙️ Installation & Setup
+
+**Prerequisites:** Ensure you have [Node.js](https://nodejs.org/) installed (version 18, 20, or 22+ recommended).
+
+1. **Clone the repository:**
+```bash
+git clone https://github.com/aj1126/uap_analyticsbot.git
+cd uap_analyticsbot
+
+```
+
+
+2. **Install dependencies:**
+This project installs as a standard Node.js CLI package, so there are no extra native build steps required for the current worker-thread ingestion flow. Simply run:
+```bash
+npm install
+
+```
+
+
+3. **Verify the installation:**
+Run the local test suite to ensure the multithreaded worker pool and caching engine are functioning correctly on your machine:
+```bash
+npm test
+
+```
+
+
+*(If all tests pass green, you are ready to start analyzing documents!)*
+
+
+---
+
+<br>
+
+
+
+
+## Usage
+
+
+To run the AnalyticsBot, simply pass the target directory containing your text files as the first argument:
+
+```bash
+node src/index.js ./my_folder/
+
+```
+
+By default, this will parse the documents and output a formatted JSON report directly to your console.
+
+### 👀 Watch Mode
+
+Keep the pipeline running in the background. It will automatically re-analyze the documents and recalculate the math whenever you add, edit, or delete a file in the target directory:
+
+```bash
+node src/index.js ./my_folder/ --watch
+
+```
+
+### 🖨️ Report Generation
+
+Instead of dumping JSON directly to the console, you can generate formatted report files that are automatically saved to the `/data_exports/` directory:
+
+```bash
+node src/index.js ./my_folder/ --format=md
+
+```
+
+*(Supports `md` for Markdown or `csv` for spreadsheet datasets).*
+
+
+---
+<br>
+
+### 🚀 Advanced Usage
+
+The v1.2.0 AnalyticsBot engine supports multithreading and memoization caching. You can control these via CLI arguments:
+
+* `node src/index.js ./my_folder --workers=4` : Manually set the number of Node.js worker threads (defaults to max CPU cores).
+* `node src/index.js ./my_folder --clear-cache` : Bypasses the `.analytics_cache.json` file and forces a fresh read of all documents.
+* `node src/index.js ./my_folder --format=csv` : Exports the final report as a spreadsheet-compatible `.csv` file.
+
+<br>
+<br>
+<br>
+
+
+
 ## 🚀 Planned Technical Optimizations
 
 ### 1. Performance & Infrastructure

docs/USER_GUIDE.md

@@ -56,14 +56,16 @@ npm start -- "C:\Path\To\Folder" > analytics_report.json
 
 ## Supported File Types
 
-Currently, the ingestion engine natively parses the following text-based extensions:
+Currently, the ingestion engine natively parses the following extensions:
 * `.txt`
 * `.md`
 * `.json`
 * `.csv`
 * `.log`
-
-*(Note: Binary and multimedia extraction, such as PDF parsing and Image OCR, are tracked for a future development stage).*
+* `.pdf`
+* `.png`
+* `.jpg`
+* `.jpeg`
 
 ## Testing & Validation
 

docs/architecture.md

@@ -4,26 +4,31 @@
 
 The repository currently ships a Node.js CLI-centered analytics flow:
 
-1. **CLI Orchestrator (`src/index.js`)** resolves the source directory and writes the final report to stdout.
-2. **Read-Only Ingestion (`src/ingestion/file-ingestion.js`)** recursively scans supported text files, streams file content, and extracts words, dates, locations, and filesystem metadata.
+1. **CLI Orchestrator (`src/index.js`)** resolves the source directory, supports watch mode, and routes report output to stdout or export files.
+2. **Read-Only Ingestion (`src/ingestion/file-ingestion.js`)** recursively scans supported text files, dispatches parsing work to Node.js worker threads, memoizes compatible results in `.analytics_cache.json`, and extracts words, dates, locations, and filesystem metadata.
 3. **Analytics Pipeline (`src/pipeline.js`)** builds the descriptive, diagnostic, predictive, and prescriptive tiers from the ingested file set.
-4. **Output Layer** returns a single structured JSON report for the requested directory.
+4. **Output Layer** returns structured JSON or saves Markdown / CSV exports for the requested directory.
+
+### v1.2.0 Pipeline Architecture
+* **Ingestion (Multithreaded):** Utilizes Node.js `worker_threads` and file-stat fingerprinting (`.analytics_cache.json`) to bypass redundant processing and drastically speed up execution.
+* **Semantic Analytics:** Employs a TF-IDF weighting engine to filter generic stop-words and a Cosine Similarity math engine to automatically cluster related UAP documents based on vector distance.
 
 ## Current Runtime Boundaries
 
 Implemented today:
 
 - recursive read-only ingestion for `.txt`, `.md`, `.json`, `.csv`, and `.log`
+- multithreaded parsing with fingerprint-based cache reuse for compatible ingestions
 - tokenization plus lightweight date/location extraction
 - descriptive, diagnostic, predictive, and prescriptive analytics modules
-- JSON report delivery through the Node CLI
+- JSON, Markdown, and CSV report delivery through the Node CLI
+- directory watch mode that re-runs the pipeline after file changes
 
 Not yet implemented in the active system:
 
 - binary or multimedia extraction
 - Named Entity Recognition (NER)
-- dashboard or alternate export formats
-- background scheduling or directory watching
+- dashboard or background scheduling
 
 ## Planned Expansion
 

docs/docs-source.json

@@ -26,7 +26,7 @@
       "description": "Auto-generate CHANGELOG.md, bump the semantic version, and create a Git release tag based on conventional commit history."
     }
   ],
-  "supportedFileTypes": [".txt", ".md", ".json", ".csv", ".log", ".pdf"],
+  "supportedFileTypes": [".txt", ".md", ".json", ".csv", ".log", ".pdf", ".png", ".jpg", ".jpeg"],
   "repoLayout": [
     {
       "path": "src/index.js",
@@ -38,7 +38,7 @@
     },
     {
       "path": "src/ingestion/file-ingestion.js",
-      "description": "Read-only recursive file ingestion for supported text files."
+      "description": "Read-only recursive file ingestion for supported files."
     },
     {
       "path": "src/analytics/",

package-lock.json

@@ -6,9 +6,10 @@
   "main": "src/index.js",
   "scripts": {
     "start": "node src/index.js",
-    "test": "node --test",
+    "test": "node --test --experimental-test-coverage",
     "docs:generate": "node scripts/generate-docs.js",
     "docs:check": "node scripts/generate-docs.js --check && node scripts/validate-docs.js",
+    "prepare": "husky",
     "release": "commit-and-tag-version",
     "postrelease": "git push --follow-tags && gh release create v%npm_package_version% --notes-file CHANGELOG.md --title \"Release v%npm_package_version%\""
   },
@@ -29,6 +30,7 @@
     "tesseract.js": "^7.0.0"
   },
   "devDependencies": {
-    "commit-and-tag-version": "^12.7.3"
+    "commit-and-tag-version": "^12.7.3",
+    "husky": "^9.1.7"
   }
 }

package.json

@@ -1,34 +1,43 @@
-function countBy(items) {
-    return items.reduce((counts, item) => {
-        counts[item] = (counts[item] ?? 0) + 1;
-        return counts;
-    }, {});
-}
-
 function sortEntriesDescending(record) {
     return Object.entries(record).sort((left, right) => right[1] - left[1] || left[0].localeCompare(right[0]));
 }
 
 function buildDescriptiveAnalytics(files) {
-    const allWords = files.flatMap((file) => file.words);
-    const allDates = files.flatMap((file) => file.dates);
-    const allLocations = files.flatMap((file) => file.locations);
+    const allDates = files.flatMap((file) => file.dates || []);
+    const allLocations = files.flatMap((file) => file.locations || []);
+
+    const globalWordFrequency = {};
+    const glossarySet = new Set();
 
-    const wordFrequency = countBy(allWords);
+    // Iterate through files using the new memory-efficient object format
+    files.forEach((file) => {
+        if (file.wordFrequency) {
+            for (const [word, count] of Object.entries(file.wordFrequency)) {
+                globalWordFrequency[word] = (globalWordFrequency[word] || 0) + count;
+                glossarySet.add(word);
+            }
+        } else if (file.words) { 
+            // Backwards compatibility layer
+            for (const word of file.words) {
+                globalWordFrequency[word] = (globalWordFrequency[word] || 0) + 1;
+                glossarySet.add(word);
+            }
+        }
+    });
 
     return {
         fileCount: files.length,
-        glossary: [...new Set(allWords)].sort(),
-        wordFrequency,
-        topWords: sortEntriesDescending(wordFrequency).slice(0, 10).map(([word, count]) => ({ word, count })),
+        glossary: [...glossarySet].sort(),
+        wordFrequency: globalWordFrequency,
+        topWords: sortEntriesDescending(globalWordFrequency).slice(0, 10).map(([word, count]) => ({ word, count })),
         dates: [...new Set(allDates)].sort(),
         locations: [...new Set(allLocations)].sort(),
         files: files.map((file) => ({
             path: file.relativePath,
-            extension: file.extension, // <-- FIX: Added extension propagation
+            extension: file.extension, 
             size: file.size,
             modifiedAt: file.modifiedAt,
-            wordCount: file.words.length,
+            wordCount: file.totalWords || (file.words ? file.words.length : 0),
             dates: file.dates,
             locations: file.locations,
             metadata: file.metadata || {}