Skip to content

Monorepo Restructuring Plan for pinjected using packages directory #103

Open
Open
Monorepo Restructuring Plan for pinjected using packages directory#103
Labels
enhancementNew feature or request

Description

@proboscis
proboscis
opened on Jun 27, 2025

---\nMigrated from CyberAgentAILab/pinjected#17\nOriginally created by @app/devin-ai-integration on null\n\n---\n\n# Pinjected Monorepo Migration Plan

Overview

This plan outlines the steps to migrate the pinjected repository to a monorepo structure with the core pinjected package and the pinjected-openai extension.

Target Directory Structure

pinjected/                      # Root directory
├── packages/                       # Libraries directory
│   ├── core/                   # Core pinjected package
│   │   ├── src/                # Source code
│   │   │   ├── pinjected/      # Main package code
│   │   │   └── test/           # Test support packages
│   │   ├── tests/              # Test files
│   │   └── pyproject.toml      # Package configuration
│   └── openai_support/         # OpenAI integration package
│       ├── src/                # Source code
│       │   └── pinjected_openai/ # Main package code
│       ├── tests/              # Test files
│       └── pyproject.toml      # Package configuration
├── pyproject.toml              # Root configuration with workspace definition
├── Makefile                    # Build and test targets
└── uv.lock                     # Workspace lock file

Step-by-Step Migration Plan

Step 1: Clone Fresh Repository

  1. Clone a fresh copy of the repository:

    git clone https://github.com/CyberAgentAILab/pinjected.git
cd pinjected

  2. Create a new branch for the monorepo restructuring:

    git checkout -b feature/monorepo-restructuring

Step 2: Clone pinjected-openai Repository

  1. Clone the pinjected-openai repository: 
    git clone https://github.com/proboscis/pinjected-openai.git /tmp/pinjected-openai

Step 3: Create Directory Structure

  1. Create directory structure for core package:

    mkdir -p packages/core/src/pinjected
mkdir -p packages/core/src/test
mkdir -p packages/core/tests

  2. Create directory structure for OpenAI integration:

    mkdir -p packages/openai_support/src/pinjected_openai
mkdir -p packages/openai_support/tests

Step 4: Move Files to New Structure

  1. Move core package files:

    # Move source files
cp -r pinjected/* packages/core/src/pinjected/

# Move test support packages
cp -r pinjected/test/* packages/core/src/test/

# Move test files
cp -r test/* packages/core/tests/

# Move documentation
mkdir -p packages/core/docs
cp -r docs_md/* packages/core/docs/
cp -r docs_ja packages/core/
cp -r docs-for-ai packages/core/

  2. Move OpenAI integration files:

    # Move source files
cp -r /tmp/pinjected-openai/pinjected_openai/* packages/openai_support/src/pinjected_openai/

# Move test files
cp -r /tmp/pinjected-openai/tests/* packages/openai_support/tests/

# Move documentation
cp /tmp/pinjected-openai/README.md packages/openai_support/
cp /tmp/pinjected-openai/CHANGELOG.md packages/openai_support/

Step 5: Update Configuration Files

  1. Create root pyproject.toml:

    [build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.uv.workspace]
members = ["packages/*"]

  2. Update core package pyproject.toml:

    [build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "pinjected"
version = "0.2.246"
description = "Immutable Dependency Injection for Python."
authors = [
    {name = "proboscis", email = "nameissoap@gmail.com"}
]
license = {text = "MIT"}
requires-python = ">=3.10"
dependencies = [
    "returns>=0.25.0",
    "expression",
    "cloudpickle",
    "tabulate",
    "cytoolz",
    "frozendict",
    "fire",
    "pyyaml",
    "pyvis",
    "makefun",
    "loguru",
    "rich",
    "pydantic",
    "pytest",
    "beartype",
    "pytest-asyncio",
    "pytest-xdist",
]

[project.optional-dependencies]
dev = [
    "pytest",
    "tqdm>=4.66.5",
    "click>=8.1.7",
    "wandb>=0.17.8",
    "pytest-xdist>=3.6.1",
]

[project.scripts]
pinjected = "pinjected.main_impl:main"

[tool.pytest.ini_options]
asyncio_mode = "strict"
asyncio_default_fixture_loop_scope = "function"

[tool.hatch.build.targets.wheel]
packages = ["src"]

  3. Create OpenAI integration pyproject.toml:

    [build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "pinjected-openai"
version = "0.1.0"
description = "OpenAI integration for pinjected"
authors = [
    {name = "proboscis", email = "nameissoap@gmail.com"}
]
license = {text = "MIT"}
requires-python = ">=3.10"
dependencies = [
    "pinjected",
    "openai>=1.0.0",
    "tiktoken",
    "tenacity",
    "pydantic"
]

[tool.hatch.build.targets.wheel]
packages = ["src"]

[tool.uv.sources]
pinjected = { workspace = true }

  4. Update Makefile:

    .PHONY: test test-cov publish tag-version release sync setup-all

sync:
	uv venv
	uv sync

test:
	cd packages/core && uv run pytest tests
	cd packages/openai_support && uv run pytest tests

test-cov:
	cd packages/core && uv run pytest tests --cov=pinjected --cov-report=xml
	cd packages/openai_support && uv run pytest tests --cov=pinjected_openai --cov-report=xml

publish-core:
	cd packages/core && uv build
	cd packages/core && uv publish dist/*.whl dist/*.tar.gz

publish-openai:
	cd packages/openai_support && uv build
	cd packages/openai_support && uv publish dist/*.whl dist/*.tar.gz

tag-version-core:
	git tag pinjected-v$(shell grep -m 1 version packages/core/pyproject.toml | cut -d'"' -f2)
	git push --tags

tag-version-openai:
	git tag pinjected-openai-v$(shell grep -m 1 version packages/openai_support/pyproject.toml | cut -d'"' -f2)
	git push --tags

setup-all:
	uv sync ./packages/core
	uv sync ./packages/openai_support

release-core: tag-version-core publish-core
release-openai: tag-version-openai publish-openai

Step 6: Update Import Paths

  1. Update import paths in Python files: 
    # Update import paths in core package
find packages/core -name "*.py" -exec sed -i 's/from test\./from pinjected.test./g' {} \;
find packages/core -name "*.py" -exec sed -i 's/import test\./import pinjected.test./g' {} \;

# Update import paths in OpenAI integration
find packages/openai_support -name "*.py" -exec sed -i 's/from pinjected import/from pinjected import/g' {} \;
find packages/openai_support -name "*.py" -exec sed -i 's/import pinjected/import pinjected/g' {} \;

Step 7: Create Necessary init.py Files

  1. Create init.py files:

    touch packages/core/src/__init__.py
touch packages/core/src/pinjected/__init__.py
touch packages/core/src/test/__init__.py
touch packages/openai_support/src/__init__.py
touch packages/openai_support/src/pinjected_openai/__init__.py

  2. Copy the content of the original init.py files:

    cp pinjected/__init__.py packages/core/src/pinjected/__init__.py

Step 8: Install Dependencies and Test

  1. Install dependencies:

    uv venv
uv sync
uv sync ./packages/core
uv sync ./packages/openai_support

  2. Run tests:

    make test

Step 9: Update CI Configuration

  1. Update .github/workflows/ci.yml:

    name: CI

on:
  push:
    branches: [ main, feature/* ]
  pull_request:
    branches: [ main, feature/* ]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - name: Check out code
        uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.10'

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install uv
          make sync
          make setup-all

      - name: Run tests
        run: make test

      - name: Run test coverage
        run: make test-cov

      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v3
        with:
          file: ./packages/core/coverage.xml
          flags: unittests
          name: codecov-umbrella
          fail_ci_if_error: true

  2. Update .github/workflows/pytest.yml if it exists.

Step 10: Update Documentation

  1. Update README.md with monorepo information: 
    ## Monorepo Structure

This repository is organized as a monorepo with the following packages:
- [pinjected](packages/core): Core dependency injection framework
- [pinjected-openai](packages/openai_support): OpenAI API bindings for pinjected

### Working with Monorepo Packages
To use the packages in development mode:
```bash
# From the repository root
make setup-all
# Or individually
uv sync ./packages/core
uv sync ./packages/openai_support
  2. Update other documentation as needed.

Key Challenges and Solutions

Challenge 1: Directory Structure Inconsistency

Problem: The current repository has a mix of directory structures with some files in pinjected/ at the root and others in src/pinjected/. This inconsistency makes it difficult to maintain a clean monorepo structure.

Solution:

  • Adopt a consistent directory structure with packages/core/src/pinjected/ for source code
  • Move test support packages to packages/core/src/test/
  • Keep test files in packages/core/tests/
  • This structure follows Python packaging best practices and makes the package structure clear

Challenge 2: Import Path Resolution

Problem: The codebase uses various import patterns including direct imports (from pinjected import ...), module-specific imports (from pinjected.di.design import ...), test package imports (from test.test_package import ...), and relative imports (from .module import ...). Changing the directory structure will break these imports.

Solution:

  • Update import paths systematically using find/sed commands
  • Convert from test.test_package to from pinjected.test.test_package
  • Ensure __init__.py files are created in all necessary directories
  • Test imports thoroughly after migration to catch any missed imports

Challenge 3: Workspace Configuration

Problem: The uv workspace configuration is not properly set up, causing dependency resolution issues between packages. The pinjected-openai package depends on the core pinjected package, but the workspace configuration is not correctly handling this dependency.

Solution:

  • Configure root pyproject.toml with proper workspace members: members = ["packages/*"]
  • Set up tool.uv.sources in the OpenAI package to reference the core package: pinjected = { workspace = true }
  • Use uv sync at the root level to ensure consistent dependency versions across packages
  • Install packages in development mode with uv sync ./packages/core

Challenge 4: Test Discovery and Execution

Problem: Tests are spread across multiple directories with inconsistent naming and import patterns. The test discovery mechanism relies on specific directory structures and import paths that will change with the monorepo restructuring.

Solution:

  • Organize tests consistently in packages/core/tests/ and packages/openai_support/tests/
  • Update test discovery paths in pytest configuration
  • Update the Makefile to run tests from the correct directories
  • Ensure test support packages are properly imported from pinjected.test

Challenge 5: CI/CD Integration

Problem: CI workflows are configured to run tests and build packages based on the current directory structure. Changes to the directory structure will break these workflows.

Solution:

  • Update CI workflows to use the Makefile targets for building and testing
  • Ensure CI workflows install dependencies correctly using uv sync
  • Test CI workflows locally before pushing changes
  • Add appropriate CI checks to verify the monorepo structure works correctly

Verification Steps

After completing the migration, verify the following:

  1. Package Installation: Ensure both packages can be installed with uv sync ./packages/core and uv sync ./packages/openai_support
  2. Import Paths: Verify that import paths work correctly in both packages
  3. Tests: Ensure all tests pass with make test
  4. CI Checks: Verify that CI checks pass
  5. Documentation: Ensure documentation is up-to-date with the new structure

Rollback Plan

If issues are encountered during the migration, the following rollback steps can be taken:

  1. Revert to the previous commit: git reset --hard HEAD~1
  2. Push the revert: git push --force
  3. Restore any deleted files from the previous commit

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions