BrentLab · cmatKhan · Mar 18, 2026 · Jun 2, 2025 · Jun 4, 2025 · Jun 4, 2025
diff --git a/.vscode/settings.json b/.vscode/settings.json
@@ -6,6 +6,7 @@
         "astype",
         "dtype",
         "interactor",
+        "interactors",
         "issubdtype",
         "ndarray",
         "ndim",
@@ -14,8 +15,8 @@
         "pvalues",
         "randperm",
         "rankdata",
-        "topn",
-        "tfbpmodeling"
+        "tfbpmodeling",
+        "topn"
     ],
     "[markdown]": {
         "editor.formatOnSave": false

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -26,8 +26,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 - `poetry run mkdocs gh-deploy` - Deploy documentation to GitHub Pages
 
 ### Running the Application
-- `poetry run python -m tfbpmodeling --help` - Show main help
-- `poetry run python -m tfbpmodeling linear_perturbation_binding_modeling --help` - Show modeling command help
+- `poetry run python -m tfbpmodeling --help` - Show full help and all options
 
 ## Project Architecture
 
@@ -36,33 +35,38 @@ This package provides tools for transcription factor binding and perturbation (T
 
 ### Main Components
 
-#### Entry Point (`__main__.py`)
-- Primary CLI interface using argparse
-- Main command: `linear_perturbation_binding_modeling`
-- Configurable logging with console/file handlers
-- Extensible subcommand structure
-
-#### Core Workflow (`interface.py`)
-The main modeling workflow consists of 4 stages:
-1. **Preprocessing**: Data validation and preparation
-2. **Bootstrap Modeling**: LassoCV modeling with bootstrap resampling on all data
-3. **Top-N Modeling**: Secondary modeling on significant predictors from top-performing data
-4. **Interactor Significance**: Evaluation of interaction terms vs main effects
+#### Entry Point and Workflow (`__main__.py`)
+All CLI argument definitions and the modeling workflow live in a single file. The
+`main()` function builds the argparse parser and dispatches to `tfbpmodeling(args)`,
+which runs the full sequential workflow:
+
+- **Stage 0**: Preprocessing — input validation and output directory setup
+- **Stage 1**: Bootstrap LassoCV on all data with the full interactor model; fits a
+  best all-data model on the significant predictors
+- **Stage 2**: Bootstrap LassoCV on the top-N data subset using Stage 1's significant
+  predictors
+- **Stage 3 - LassoCV Bootstrap** (optional, `--stage3_lassocv_bootstrap`): Refits surviving interactors
+  with their independent main effects on all data using the Stage 1 protocol
+- **Stage 3 - Lasso** (always runs): Tests significance of each surviving interactor
+  term against its corresponding main effect
 
 #### Key Modules
 - `modeling_input_data.py` - Core data structures and preprocessing
 - `bootstrapped_input_data.py` - Bootstrap resampling functionality
 - `bootstrap_stratified_cv.py` - Cross-validation with stratification
-- `evaluate_interactor_significance_*.py` - Statistical significance testing (LassoCV and linear methods)
+- `bootstrap_stratified_cv_loop.py` - Iterative dropout variant of bootstrap CV
+- `evaluate_interactor_significance_lassocv.py` - LassoCV-based significance testing
+- `evaluate_interactor_significance_linear.py` - Linear regression significance testing
 - `stratification_classification.py` - Data stratification logic
+- `configure_logger.py` - Logger configuration utilities
 - `utils/` - Utility functions for data manipulation
 
 #### Data Flow
 1. Input files: response data (gene expression) and predictors (binding data)
 2. Data preprocessing with optional feature selection and binning
 3. Bootstrap resampling with stratified cross-validation
-4. Model fitting using LassoCV or linear regression
-5. Significance testing of interaction terms
+4. Model fitting using LassoCV
+5. Significance testing of surviving interaction terms against main effects
 6. Output generation with confidence intervals and model statistics
 
 ### Development Patterns

diff --git a/README.md b/README.md
@@ -1,211 +1,19 @@
 # tfbpmodeling
 
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-[![style](https://img.shields.io/badge/%20style-sphinx-0a507a.svg)](https://www.sphinx-doc.org/en/master/usage/index.html)
 [![Pytest](https://github.com/BrentLab/tfbpmodeling/actions/workflows/ci.yml/badge.svg)](https://github.com/BrentLab/tfbpmodeling/actions/workflows/ci.yml)
 [![codecov](https://codecov.io/gh/BrentLab/tfbpmodeling/graph/badge.svg?token=7zBsImRmjC)](https://codecov.io/gh/BrentLab/tfbpmodeling)
 
 A Python package for transcription factor binding and perturbation (TFBP) modeling
 that analyzes relationships between transcription factor binding and gene expression
 perturbations using LASSO.
 
-NOTE: this documentation is produced by AI and hasn't had significant human revision.
-Please, if you find problems or there is anything confusing or missing, open an issue
-and just explain where the docs stopped being helpful.
-
-## Quick Start
-
-### Installation
-
-```python
-python -m pip install git+https://github.com/BrentLab/tfbpmodeling.git
-```
-
-or, for the development branch `dev`:
-
-```python
-python -m pip install git+https://github.com/BrentLab/tfbpmodeling.git@dev
-```
-
-### Basic Usage
-
-Run the main modeling command:
-
-```bash
-python -m tfbpmodeling linear_perturbation_binding_modeling \
-    --response_file response_data.csv \
-    --predictors_file binding_data.csv \
-    --perturbed_tf YourTF
-```
-
-## Command Line Interface
-
-The package provides a single main command with comprehensive options for modeling
-transcription factor binding and perturbation data.
-
-### Main Command
+## Installation
 
 ```bash
-python -m tfbpmodeling linear_perturbation_binding_modeling [OPTIONS]
-```
-
-This command executes a sequential 4-stage workflow:
-
-1. **All Data Modeling**: Models `perturbation ~ binding` on complete dataset using
-  bootstrap resampling
-1. **Top-N Modeling**: Extracts significant predictors and models on top-performing
-  data subset
-1. **Interactor Significance**: Evaluates surviving interaction terms against
-  corresponding main effects
-1. **Output Generation**: Produces comprehensive results with confidence intervals
-  and statistics
-
-### Required Arguments
-
-| Argument | Description |
-|----------|-------------|
-| `--response_file` | Path to response CSV file (gene expression data) |
-| `--predictors_file` | Path to predictors CSV file (binding data) |
-| `--perturbed_tf` | Name of perturbed TF (must match response file column) |
-
-### Key Options
-
-#### Input Control
-
-- `--blacklist_file`: Exclude specific features from analysis
-- `--n_bootstraps`: Number of bootstrap samples (default: 1000)
-- `--random_state`: Set seed for reproducible results
-- `--top_n`: Features to retain in second modeling round (default: 600)
-
-#### Feature Engineering
-
-- `--row_max`: Include row maximum as predictor
-- `--squared_pTF`: Include squared perturbed TF term
-- `--cubic_pTF`: Include cubic perturbed TF term
-- `--ptf_main_effect`: Include perturbed TF main effect
-- `--exclude_model_variables`: comma separated list of column names from the input
-  predictor data to exclude from the automatic generation of model variables. Eg,
-  if you add a column "base_expression" to the predictor data, you can exclude it
-  from the automatic generation of interaction terms by including
-  `--exclude_model_variables base_expression`. To add "base_expression" as a model
-  variable, you would include `--add_model_variables base_expression`.
-- `--add_model_variables`: A comma separated list of variables to add to the
-  model. These must be valid methods of specifying model variables from the
-  input predictor data. See
-  [python's patsy formula documentation](https://patsy.readthedocs.io/en/latest/overview.html)
-
-#### Model Parameters
-
-- `--all_data_ci_level`: Confidence interval for first round (default: 98.0%)
-- `--topn_ci_level`: Confidence interval for second round (default: 90.0%)
-- `--max_iter`: Maximum LassoCV iterations (default: 10000)
-- `--iterative_dropout`: Enable iterative variable dropout
-- `--stage4_lasso`: Use LassoCV for Stage 4 significance testing
-
-#### Data Processing
-
-- `--normalize_sample_weights`: Normalize bootstrap weights to sum to 1
-- `--scale_by_std`: Scale model matrix by standard deviation (without centering)
-- `--bins`: Bin edges for data stratification (default: "0,8,12,np.inf")
-
-#### Output Control
-
-- `--output_dir`: Results directory (default: "./linear_perturbation_binding_modeling_results")
-- `--output_suffix`: Suffix for output subdirectory naming
-
-#### System Options
-
-- `--n_cpus`: CPU cores for parallel processing (default: 4)
-- `--log-level`: Logging verbosity (DEBUG, INFO, WARNING, ERROR, CRITICAL)
-- `--log-handler`: Log output destination (console, file)
-
-### Example Commands
-
-#### Basic Analysis
-
-```bash
-python -m tfbpmodeling linear_perturbation_binding_modeling \
-    --response_file data/expression.csv \
-    --predictors_file data/binding.csv \
-    --perturbed_tf YPD1
-```
-
-#### Advanced Analysis with Custom Parameters
-
-```bash
-python -m tfbpmodeling linear_perturbation_binding_modeling \
-    --response_file data/expression.csv \
-    --predictors_file data/binding.csv \
-    --perturbed_tf YPD1 \
-    --n_bootstraps 2000 \
-    --top_n 500 \
-    --all_data_ci_level 95.0 \
-    --topn_ci_level 85.0 \
-    --squared_pTF \
-    --ptf_main_effect \
-    --iterative_dropout \
-    --stage4_lasso \
-    --output_dir ./results \
-    --output_suffix _custom_run \
-    --n_cpus 8 \
-    --random_state 42
-```
-
-#### Reproducible Analysis with Feature Engineering
-
-```bash
-poetry run python -m tfbpmodeling linear_perturbation_binding_modeling \
-    --response_file data/expression.csv \
-    --predictors_file data/binding.csv \
-    --perturbed_tf YPD1 \
-    --blacklist_file data/exclude_genes.txt \
-    --random_state 12345 \
-    --row_max \
-    --cubic_pTF \
-    --exclude_model_variables "red_median" \
-    --add_model_variables "red_median" \
-    --bins "0,5,10,15,np.inf" \
-    --normalize_sample_weights \
-    --scale_by_std
+python -m pip install git+https://github.com/BrentLab/tfbpmodeling.git
 ```
 
-### Input File Formats
-
-#### Response File (expression data)
-
-- CSV format with genes/features as rows
-- First column: gene identifiers (matching predictor file)
-- Subsequent columns: sample expression values
-- Must contain column matching `--perturbed_tf` argument
-
-#### Predictors File (binding data)
-
-- CSV format with genes/features as rows
-- First column: gene identifiers (matching response file)
-- Subsequent columns: binding measurements for different TFs
-
-#### Blacklist File (optional)
-
-- Plain text file with one feature identifier per line
-- Features listed will be excluded from analysis
-
-### Output
-
-Results are saved in the specified output directory with subdirectories for each
-analysis run. Output includes:
-
-- Model coefficients and confidence intervals
-- Bootstrap statistics and distributions
-- Significance testing results
-- Diagnostic plots and summaries
-- Log files with detailed execution information
-
 ## Documentation
 
-For detailed documentation and API reference, see
 [https://brentlab.github.io/tfbpmodeling/](https://brentlab.github.io/tfbpmodeling/)
-
-## Development
-
-See [CLAUDE.md](CLAUDE.md) for development setup, testing commands, and
-contribution guidelines.
diff --git a/docs/api/bootstrap_model_results.md b/docs/api/bootstrap_model_results.md
@@ -6,7 +6,9 @@ Results aggregation and statistical analysis for bootstrap modeling.
 
 ## Overview
 
-The `bootstrap_model_results` module provides classes and functions for aggregating and analyzing results from bootstrap modeling. It handles the statistical analysis of coefficient distributions, confidence intervals, and significance testing.
+The `bootstrap_model_results` module provides classes and functions for aggregating and
+analyzing results from bootstrap modeling. It handles the statistical analysis of
+coefficient distributions, confidence intervals, and significance testing.
 
 ## Key Features
 
@@ -53,4 +55,5 @@ results.plot_coefficient_distributions()
 
 - **[bootstrapped_input_data](bootstrapped_input_data.md)**: Bootstrap data generation
 - **[interface](interface.md)**: Main workflow integration
-- **[interactor_significance_results](interactor_significance_results.md)**: Interaction analysis
+- **[interactor_significance_results](interactor_significance_results.md)**:
+  Interaction analysis
diff --git a/docs/api/bootstrapped_input_data.md b/docs/api/bootstrapped_input_data.md
@@ -6,12 +6,16 @@ Bootstrap resampling functionality for tfbpmodeling input data.
 
 ## Overview
 
-The `bootstrapped_input_data` module provides the `BootstrappedModelingInputData` class that extends the base `ModelingInputData` with bootstrap resampling capabilities. This is essential for the statistical inference approach used in tfbpmodeling.
+The `bootstrapped_input_data` module provides the `BootstrappedModelingInputData` class
+that extends the base `ModelingInputData` with bootstrap resampling capabilities. This
+is essential for the statistical inference approach used in tfbpmodeling.
 
 ## Key Features
 
-- **Bootstrap Sample Generation**: Creates multiple resampled datasets from the original data
-- **Stratified Sampling**: Maintains data distribution characteristics across bootstrap samples
+- **Bootstrap Sample Generation**: Creates multiple resampled datasets from the original
+  data
+- **Stratified Sampling**: Maintains data distribution characteristics across bootstrap
+  samples
 - **Reproducible Results**: Supports random seed setting for consistent results
 - **Memory Efficient**: Optimized storage and access patterns for large bootstrap sets
 
@@ -27,7 +31,7 @@ from tfbpmodeling.bootstrapped_input_data import BootstrappedModelingInputData
 base_data = ModelingInputData(
     response_file='expression.csv',
     predictors_file='binding.csv',
-    perturbed_tf='YPD1'
+    perturbed_tf='pTF1'
 )
 
 # Create bootstrap version

diff --git a/docs/api/evaluate_interactor_significance_lassocv.md b/docs/api/evaluate_interactor_significance_lassocv.md
@@ -6,7 +6,10 @@ LassoCV-based interactor significance testing for tfbpmodeling.
 
 ## Overview
 
-The `evaluate_interactor_significance_lassocv` module provides functions for evaluating the significance of interaction terms using LassoCV regularization. This approach uses regularized regression to compare models with and without interaction terms, providing a conservative approach to interaction significance testing.
+The `evaluate_interactor_significance_lassocv` module provides functions for evaluating
+the significance of interaction terms using LassoCV regularization. This approach uses
+regularized regression to compare models with and without interaction terms, providing
+a conservative approach to interaction significance testing.
 
 ## Key Features
 
@@ -77,6 +80,8 @@ results = evaluate_interactor_significance_lassocv(
 
 ## Related Modules
 
-- **[evaluate_interactor_significance_linear](evaluate_interactor_significance_linear.md)**: Linear regression alternative
-- **[interactor_significance_results](interactor_significance_results.md)**: Results handling
+- **[evaluate_interactor_significance_linear](evaluate_interactor_significance_linear.md)**:
+  Linear regression alternative
+- **[interactor_significance_results](interactor_significance_results.md)**:
+  Results handling
 - **[interface](interface.md)**: Workflow integration
diff --git a/docs/api/evaluate_interactor_significance_linear.md b/docs/api/evaluate_interactor_significance_linear.md
@@ -6,7 +6,10 @@ Linear regression-based interactor significance testing for tfbpmodeling.
 
 ## Overview
 
-The `evaluate_interactor_significance_linear` module provides functions for evaluating the significance of interaction terms using standard linear regression methods. This approach uses classical statistical tests to compare models with and without interaction terms.
+The `evaluate_interactor_significance_linear` module provides functions for evaluating
+the significance of interaction terms using standard linear regression methods. This
+approach uses classical statistical tests to compare models with and without interaction
+terms.
 
 ## Key Features
 
@@ -92,6 +95,8 @@ results = evaluate_interactor_significance_linear(
 
 ## Related Modules
 
-- **[evaluate_interactor_significance_lassocv](evaluate_interactor_significance_lassocv.md)**: Regularized alternative
-- **[interactor_significance_results](interactor_significance_results.md)**: Results handling
+- **[evaluate_interactor_significance_lassocv](evaluate_interactor_significance_lassocv.md)**:
+  Regularized alternative
+- **[interactor_significance_results](interactor_significance_results.md)**:
+  Results handling
 - **[interface](interface.md)**: Workflow integration