Categorical & Performance Preprocessor

[RigelAlgebar]

🧑‍💻 Issue 2 – Subset 2 Categorical & Performance Preprocessor

** Build a functional + OOP categorical/performance cleaning & encoding block (subset 2) **

You will work with these columns:

SUBSET2_COLS = [
    "id",
    "member_id",
    # numeric – performance / payments
    "out_prncp_inv",
    "total_pymnt",
    "total_pymnt_inv",
    "total_rec_prncp",
    "total_rec_int",
    "total_rec_late_fee",
    "recoveries",
    "collection_recovery_fee",
    "last_pymnt_amnt",
    "collections_12_mths_ex_med",
    "mths_since_last_major_derog",
    "policy_code",
    "annual_inc_joint",
    "dti_joint",
    "acc_now_delinq",
    "tot_coll_amt",
    # categorical / text / target-ish
    "loan_status",      # target (do NOT use as input feature)
    "pymnt_plan",
    "url",
    "desc",
    "purpose",
    "title",
    "zip_code",
    "addr_state",
]

Use loan_status only as y, not as a feature.

---

🎯 Goal

Create a categorical + text-based transformer for subset 2 that:

• Cleans & normalizes categorical string fields
• Handles missing categories and groups rare ones into "Other"
• Derives simple numeric features from text (desc, title)
• Is a sklearn-compatible transformer (OOP) built on pure functions (FP)

---

✅ Tasks

• In src/cleaning_categorical.py, implement pure categorical cleaners:

def normalize_cat_strings(df: pd.DataFrame, cols: list[str]) -> pd.DataFrame:
    # Lowercase + strip whitespace in the given categorical columns.
    ...

def fill_cat_missing_with_sentinel(
    df: pd.DataFrame,
    cols: list[str],
    label: str = "Missing",
) -> pd.DataFrame:
    # Replace NaNs in categorical columns with a sentinel category.
    ...

def group_rare_categories(
    df: pd.DataFrame,
    col: str,
    min_count: int = 100,
) -> pd.DataFrame:
    # Replace rare categories in a single column by "Other".
    ...

• In src/cleaning_text.py, implement simple text feature functions:

def add_desc_length(df: pd.DataFrame) -> pd.DataFrame:
    # Create "desc_len" as character length of "desc" (or 0 if NaN).
    ...

def add_title_word_count(df: pd.DataFrame) -> pd.DataFrame:
    # Create "title_word_count" as word count of "title".
    ...

• Implement a functional composition helper:

def apply_cat_steps(
    df: pd.DataFrame,
    cols: list[str],
    steps: list,
) -> pd.DataFrame:
    # Apply a sequence of functions(df, cols) -> df to categorical columns.
    for step in steps:
        df = step(df, cols)
    return df

• In src/transformers.py, implement Subset2CategoricalPerformanceTransformer:

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.preprocessing import OneHotEncoder
import pandas as pd
import numpy as np

class Subset2CategoricalPerformanceTransformer(BaseEstimator, TransformerMixin):
    def __init__(self, cat_cols: list[str], min_count: int = 100):
        self.cat_cols = cat_cols
        self.min_count = min_count
        self.encoder = OneHotEncoder(handle_unknown="ignore", sparse=False)
        self.rare_maps_ = {}  # mapping per column for rare categories

    def fit(self, X, y=None):
        df = pd.DataFrame(X, columns[self.cat_cols]).copy()
        # 1) normalize strings
        df = normalize_cat_strings(df, self.cat_cols)
        # 2) fill missing with sentinel
        df = fill_cat_missing_with_sentinel(df, self.cat_cols)
        # 3) detect rare categories per column and store mapping
        for col in self.cat_cols:
            counts = df[col].value_counts()
            rare = counts[counts < self.min_count].index
            self.rare_maps_[col] = set(rare)
            df[col] = df[col].where(~df[col].isin(rare), "Other")
        # 4) fit encoder
        self.encoder.fit(df[self.cat_cols])
        return self

    def transform(self, X):
        df = pd.DataFrame(X, columns=self.cat_cols).copy()
        # apply normalization, missing fill, and rare grouping using self.rare_maps_
        df = normalize_cat_strings(df, self.cat_cols)
        df = fill_cat_missing_with_sentinel(df, self.cat_cols)
        for col in self.cat_cols:
            rare = self.rare_maps_.get(col, set())
            df[col] = df[col].where(~df[col].isin(rare), "Other")
        encoded = self.encoder.transform(df[self.cat_cols])
        return encoded

• Integrate text numeric features (desc_len, title_word_count) inside fit/transform (or via a helper) and concatenate them with the one-hot encoded output (you can extend the class to np.hstack encoded + text features).

• Add an example usage:

cat_cols_subset2 = ["pymnt_plan", "purpose", "title", "zip_code", "addr_state"]

transformer = Subset2CategoricalPerformanceTransformer(cat_cols=cat_cols_subset2)
transformer.fit(X_train[cat_cols_subset2])
X_train_cat = transformer.transform(X_train[cat_cols_subset2])

• Add tests in tests/test_subset2_transformer.py:

  • After transform, there are no NaNs in the output
  • Rare categories in purpose or addr_state are mapped to "Other"
  • Text features (if you include them) are numeric

---

✅ Acceptance Criteria

• Subset-2 categorical features are normalized, filled, and rare categories grouped
• If implemented, simple numeric features can be derived from text (desc, title)
• Implementation uses pure functions + Subset2CategoricalPerformanceTransformer OOP
• Tests verify missing + rare category handling (and text-derived features if present)