Morphogen-X

Morphogen-X is an autonomous feature engineering engine designed for high-performance dimensionality expansion and optimal signal isolation. Built entirely upon an out-of-core architecture, it utilizes zero-copy Apache Arrow memory mappings to synthesize and filter computational features without triggering memory bloat (OOM).

Disclaimer: This project is a personal experimental endeavor created by idkBsy strictly for educational purposes and architectural exploration. It is not intended as a commercial, production-ready product, but rather a rigorous exploration into out-of-core data engineering.

---

🔬 Core Architecture

Morphogen-X operates on a bifurcated architecture consisting of three primary layers, functioning seamlessly to generate, evaluate, and filter features automatically.

1. The Cortex Layer (src/morphogen_x/cortex)

The orchestration pipeline. It manages the lifecycle of the optimization process, binding the abstract mathematical graphs into contiguous execution states. It orchestrates the flow of data from ingestion, through evolution and evaluation, to final export.

2. The Nucleus Layer (src/morphogen_x/nucleus)

The computational engine containing discrete Information Theory kernels:

• Evolutionary Synthesizer (evolution.py): Stochastically breeds novel feature geometries utilizing domain-protected mathematical primitives (SafeMath). It creates abstract syntax trees (ASTs) of operations (add, sub, mul, div, log, inv, sq) without directly computing them immediately, saving memory.
• Entropy Scanner (entropy_scanner.py): Calculates Information Density and discrete Shannon Entropy bounded by $H(X) = -\sum(p \cdot \log_2(p))$. Used to eliminate features with zero variance or minimal informational value.
• MIC Engine (mic_engine.py): An Adaptive Binned Mutual Information engine evaluating non-linear predictive power (Uncertainty Coefficients). It captures relationships that standard Pearson correlation misses (e.g., U-shapes, waves).
• Information Bottleneck / MRMR (bottleneck.py): The Natural Selection engine enforcing Minimum Redundancy Maximum Relevance. It greedily prunes collinear vectors via a lazy correlation matrix execution, ensuring the final feature set is both highly predictive of the target and mutually distinct.

3. The Synapse Layer (src/morphogen_x/synapse)

The I/O abstraction layer enforcing strict Arrow-based buffer mapping (polars.LazyFrame) to guarantee zero-copy ingestion and parquet serialization. It acts as the bridge between disk and memory.

---

🗂️ Project Structure

morphogen-x/
├── Makefile                    # Automation routines (install, test, clean)
├── pyproject.toml              # Project metadata and dependencies definition
├── README.md                   # Project documentation
├── src/
│   └── morphogen_x/
│       ├── cli.py              # Primary CLI execution harness
│       ├── cortex/             # Engine Orchestration pipeline
│       ├── lib/                # Domain-protected mathematical primitives
│       ├── nucleus/            # Information Theory kernels and MRMR selection
│       └── synapse/            # Zero-copy I/O storage operations (ArrowIO)
├── tests/                      # Pytest unit testing suite
└── tools/                      # Diagnostic and synthetic data generation scripts

---

⚖️ Objective Capabilities

Strengths (Pros)

• Extreme Memory Efficiency: The entire pipeline operates in the Abstract Syntax Tree (AST) space via Polars. Data is only materialized exactly when necessary, allowing single machines to process datasets (e.g., 10+ million rows) that would otherwise crash standard Pandas/NumPy pipelines.
• Non-Linear Dependency Detection: Unlike standard Pearson correlation engines, the Mutual Information (MIC) scanner accurately isolates predictive signals regardless of geometric distribution (e.g., U-shapes, sinusoidal variance).
• Mathematical Safety: Features strict domain protection preventing asymptotic divergence (e.g., division by zero, logarithmic evaluations of negative integers) during stochastic feature breeding.
• Zero-Copy Serialization: Leverages PyArrow and Polars to drastically reduce I/O bottlenecks.

Limitations (Cons)

• Limited Transformation Primitives: The evolutionary synthesizer is currently restricted to basic unary and binary mathematical operations. It lacks temporal, lag, or windowed functions.
• Deterministic Discretization Loss: The continuous variable discretization algorithm within the Entropy/MIC scanners utilizes uniform width-binning. This may obscure highly localized dense signals compared to quantile-based or kernel-density estimations.
• Strict Ecosystem Dependency: Deeply tied to the Polars/Apache Arrow ecosystem. Integrating with traditional NumPy arrays or Pandas DataFrames necessitates a costly serialization overhead.

---

🛠️ Installation & Setup

Prerequisites

• Operating System: Linux / macOS (Windows natively supported, WSL2 recommended)
• Python: Strictly requires Python >= 3.12 to support native type aliasing and modern language features.
• Build Tools: make installed on the host machine.

Installation Protocol

1. Clone the Repository:

   git clone https://github.com/idkbsy/morphogen-x.git
   cd morphogen-x

2. Establish the Virtual Environment:

   python3 -m venv .venv
   source .venv/bin/activate

3. Install Dependencies: Execute the Makefile installation directive which parses pyproject.toml and installs the package in editable mode along with dev dependencies.

   make install

---

🚀 Execution Guide

1. Generating Diagnostic Stress Data (Optional)

To verify the system's capacity, generate a synthetic Arrow buffer. This uses the embedded tool to produce a parquet file with complex non-linear relationships.

make generate-data

Outputs: stress_test_10m.parquet (or similar depending on script config)

2. Running the Morphogen-X Engine

Execute the primary orchestration CLI directly from your terminal using the globally mapped command.

Parameters:

• --input : Path to source buffer (.parquet or .csv).
• --target: The objective dependent variable column name (the column you want to predict).
• --generations: Total cycles of stochastic synthesis to execute (controls how deep the math AST goes).
• --threshold: (Optional) The Minimum Redundancy Maximum Relevance (MRMR) MIC score threshold. Defaults to 0.30. Lower this (e.g., 0.10) for noisy real-world datasets.
• --output: Path for the optimized dataset serialization (.parquet).

Example:

morphogen-x \
    --input stress_test_10m.parquet \
    --target TARGET_Y \
    --generations 2 \
    --threshold 0.30 \
    --output optimized_vectors.parquet

3. System Verification Diagnostics

To run the internal unit tests, rigorous mathematical assertions, and structural diagnostics:

make test

This will execute the comprehensive Pytest suite validating the Cortex pipeline, Entropy bounds, MIC scoring, Evolutionary AST generation, and the MRMR bottleneck.

---

Complete Uninstallation

To entirely remove the project and its cached environments from your system:

1. Clean project caches:

   make clean

2. Deactivate the virtual environment (if active):

   deactivate

3. Navigate outside the project directory:

   cd ..

4. Purge the project directory forcefully:

   rm -rf morphogen-x

(Note: Ensure no sensitive user data is stored within the directory before execution).

---

Architected and Engineered by idkBsy