Monorepo Restructuring Plan for pinjected

[proboscis]

---\nMigrated from CyberAgentAILab/pinjected#16\nOriginally created by @app/devin-ai-integration on null\n\n---\n\n# Monorepo Restructuring Plan for pinjected

Current State

The pinjected repository is currently in a partially restructured state with many deleted files and an incomplete monorepo structure. The repository was intended to be restructured as a monorepo with the core pinjected package and the pinjected-openai extension, but the restructuring was not completed successfully.

Current Directory Structure

• Root Directory: /home/ubuntu/repos/pinjected
• Core Package: libs/pinjected/
  • Contains the main pinjected dependency injection framework
  • Published as the pinjected package
• OpenAI Integration: libs/pinjected_openai/
  • Contains OpenAI API bindings for pinjected
  • Published as the pinjected-openai package
• Tests: Located in various directories
  • libs/pinjected/tests/
  • libs/pinjected/test/
• Documentation:
  • libs/pinjected/docs/
  • libs/pinjected/docs_ja/ (Japanese documentation)
  • libs/pinjected/docs-for-ai/

Current Issues

1. Duplicate Code: The same code exists in multiple locations (e.g., libs/pinjected/ and libs/pinjected/pinjected/)
2. Inconsistent Import Paths: Import paths vary across the codebase
3. Workspace Configuration: The workspace configuration is not properly set up
4. Directory Structure: The directory structure does not follow a clear monorepo pattern
5. Test Organization: Tests are spread across multiple directories with inconsistent naming
6. Many Deleted Files: Many files have been deleted without proper relocation
7. CI Failures: CI checks are failing due to the incomplete restructuring

Target Directory Structure

The pinjected repository should be structured as a monorepo with the following organization:

pinjected/                      # Root directory
├── libs/                       # Libraries directory
│   ├── pinjected/              # Core pinjected package
│   │   ├── src/                # Source code
│   │   │   ├── pinjected/      # Main package code
│   │   │   └── test/           # Test support packages
│   │   ├── tests/              # Test files
│   │   └── pyproject.toml      # Package configuration
│   └── pinjected_openai/       # OpenAI integration package
│       ├── src/                # Source code
│       │   └── pinjected_openai/ # Main package code
│       ├── tests/              # Test files
│       └── pyproject.toml      # Package configuration
├── pyproject.toml              # Root configuration with workspace definition
└── uv.lock                     # Workspace lock file

Migration Plan

Step 1: Clone Fresh Repository

1. Clone a fresh copy of the repository:

   git clone https://github.com/CyberAgentAILab/pinjected.git
   cd pinjected

Step 2: Create Directory Structure

1. Restore deleted files from git:

   git checkout HEAD -- libs/pinjected/ libs/pinjected_openai/

Step 3: Create Directory Structure

1. Create directory structure for core package:

   mkdir -p libs/pinjected/src/pinjected
   mkdir -p libs/pinjected/src/test
   mkdir -p libs/pinjected/tests

2. Create directory structure for OpenAI integration:

   mkdir -p libs/pinjected_openai/src/pinjected_openai
   mkdir -p libs/pinjected_openai/tests

Step 4: Move Files to New Structure

1. Move core package files:

   # Move source files
   cp -r libs/pinjected/pinjected/* libs/pinjected/src/pinjected/
   cp -r libs/pinjected/test/* libs/pinjected/src/test/
   
   # Move test files
   cp -r libs/pinjected/tests/* libs/pinjected/tests/
   
   # Move documentation
   mkdir -p libs/pinjected/docs
   cp -r libs/pinjected/docs/* libs/pinjected/docs/
   cp -r libs/pinjected/docs_ja libs/pinjected/
   cp -r libs/pinjected/docs-for-ai libs/pinjected/

2. Move OpenAI integration files:

   # Move source files
   cp -r libs/pinjected_openai/pinjected_openai/* libs/pinjected_openai/src/pinjected_openai/
   
   # Move test files
   cp -r libs/pinjected_openai/tests/* libs/pinjected_openai/tests/
   
   # Move documentation
   cp libs/pinjected_openai/README.md libs/pinjected_openai/
   cp libs/pinjected_openai/CHANGELOG.md libs/pinjected_openai/

Step 5: Update Configuration Files

1. Create root pyproject.toml:

   [build-system]
   requires = ["hatchling"]
   build-backend = "hatchling.build"
   
   [tool.uv.workspace]
   members = ["libs/*"]

2. Update core package pyproject.toml:

   [build-system]
   requires = ["hatchling"]
   build-backend = "hatchling.build"
   
   [project]
   name = "pinjected"
   version = "0.2.246"
   description = "Immutable Dependency Injection for Python."
   authors = [
       {name = "proboscis", email = "nameissoap@gmail.com"}
   ]
   license = {text = "MIT"}
   requires-python = ">=3.10"
   dependencies = [
       "returns>=0.25.0",
       "cloudpickle",
       "frozendict",
       "makefun",
       "loguru",
       "cytoolz",
       "beartype",
       "rich",
       "networkx",
       "pyvis",
       "fire",
       "tqdm",
       "pydantic",
       "expression",
       "pytest-asyncio"
   ]
   
   [tool.hatch.build.targets.wheel]
   packages = ["src"]

3. Update OpenAI integration pyproject.toml:

   [build-system]
   requires = ["hatchling"]
   build-backend = "hatchling.build"
   
   [project]
   name = "pinjected-openai"
   version = "0.1.0"
   description = "OpenAI integration for pinjected"
   authors = [
       {name = "proboscis", email = "nameissoap@gmail.com"}
   ]
   license = {text = "MIT"}
   requires-python = ">=3.10"
   dependencies = [
       "pinjected",
       "openai>=1.0.0",
       "tiktoken",
       "tenacity",
       "pydantic"
   ]
   
   [tool.hatch.build.targets.wheel]
   packages = ["src"]
   
   [tool.uv.sources]
   pinjected = { workspace = true }

4. Update Makefile:

   .PHONY: test test-cov publish tag-version release sync setup-all
   
   sync:
   	uv venv
   	uv sync
   
   test:
   	cd libs/pinjected && uv run pytest tests
   	cd libs/pinjected_openai && uv run pytest tests
   
   test-cov:
   	cd libs/pinjected && uv run pytest tests --cov=pinjected --cov-report=xml
   	cd libs/pinjected_openai && uv run pytest tests --cov=pinjected_openai --cov-report=xml
   
   publish-core:
   	cd libs/pinjected && uv build
   	cd libs/pinjected && uv publish dist/*.whl dist/*.tar.gz
   
   publish-openai:
   	cd libs/pinjected_openai && uv build
   	cd libs/pinjected_openai && uv publish dist/*.whl dist/*.tar.gz
   
   tag-version-core:
   	git tag pinjected-v$(shell grep -m 1 version libs/pinjected/pyproject.toml | cut -d'"' -f2)
   	git push --tags
   
   tag-version-openai:
   	git tag pinjected-openai-v$(shell grep -m 1 version libs/pinjected_openai/pyproject.toml | cut -d'"' -f2)
   	git push --tags
   
   setup-all:
   	uv pip install -e ./libs/pinjected
   	uv pip install -e ./libs/pinjected_openai
   
   release-core: tag-version-core publish-core
   release-openai: tag-version-openai publish-openai

Step 6: Update Import Paths

1. Update import paths in Python files:

   # Update import paths
   find libs -name "*.py" -exec sed -i 's/from libs.pinjected/from pinjected/g' {} \;
   find libs -name "*.py" -exec sed -i 's/import libs.pinjected/import pinjected/g' {} \;
   find libs -name "*.py" -exec sed -i 's/from test.test_package/from pinjected.test.test_package/g' {} \;
   find libs -name "*.py" -exec sed -i 's/import test.test_package/import pinjected.test.test_package/g' {} \;

Step 7: Create Necessary init.py Files

1. Create init.py files:

   touch libs/pinjected/src/__init__.py
   touch libs/pinjected/src/pinjected/__init__.py
   touch libs/pinjected/src/test/__init__.py
   touch libs/pinjected_openai/src/__init__.py
   touch libs/pinjected_openai/src/pinjected_openai/__init__.py

Step 8: Install Dependencies and Test

1. Install dependencies:

   uv venv
   uv sync
   uv pip install -e ./libs/pinjected
   uv pip install -e ./libs/pinjected_openai

2. Run tests:

   make test

Step 9: Update CI Configuration

1. Update CI configuration to use the Makefile targets:
   • Update .github/workflows/ci.yml
   • Update .github/workflows/pytest.yml

Step 10: Update Documentation

1. Update README.md with monorepo information
2. Update other documentation as needed

Key Challenges and Solutions

Challenge 1: Directory Structure Inconsistency

Problem: The current repository has a mix of directory structures with some files in libs/pinjected/pinjected/ (duplicated directory) and others directly in libs/pinjected/. This inconsistency makes it difficult to maintain a clean monorepo structure.

Solution:

• Adopt a consistent directory structure with libs/pinjected/src/pinjected/ for source code
• Move test support packages to libs/pinjected/src/test/
• Keep test files in libs/pinjected/tests/
• This structure follows Python packaging best practices and makes the package structure clear

Challenge 2: Import Path Resolution

Problem: The codebase uses various import patterns including direct imports (from pinjected import ...), module-specific imports (from pinjected.di.design import ...), test package imports (from test.test_package import ...), and relative imports (from .module import ...). Changing the directory structure will break these imports.

Solution:

• Update import paths systematically using find/sed commands
• Convert from test.test_package to from pinjected.test.test_package
• Ensure __init__.py files are created in all necessary directories
• Test imports thoroughly after migration to catch any missed imports

Challenge 3: Workspace Configuration

Problem: The uv workspace configuration is not properly set up, causing dependency resolution issues between packages. The pinjected-openai package depends on the core pinjected package, but the workspace configuration is not correctly handling this dependency.

Solution:

• Configure root pyproject.toml with proper workspace members: members = ["libs/*"]
• Set up tool.uv.sources in the OpenAI package to reference the core package: pinjected = { workspace = true }
• Use uv sync at the root level to ensure consistent dependency versions across packages
• Install packages in development mode with uv pip install -e ./libs/pinjected

Challenge 4: Test Discovery and Execution

Problem: Tests are spread across multiple directories with inconsistent naming and import patterns. The test discovery mechanism relies on specific directory structures and import paths that will change with the monorepo restructuring.

Solution:

• Organize tests consistently in libs/pinjected/tests/ and libs/pinjected_openai/tests/
• Update test discovery paths in pytest configuration
• Update the Makefile to run tests from the correct directories
• Ensure test support packages are properly imported from pinjected.test

Challenge 5: CI/CD Integration

Problem: CI workflows are configured to run tests and build packages based on the current directory structure. Changes to the directory structure will break these workflows.

Solution:

• Update CI workflows to use the Makefile targets for building and testing
• Ensure CI workflows install dependencies correctly using uv sync
• Test CI workflows locally before pushing changes
• Add appropriate CI checks to verify the monorepo structure works correctly

Verification Steps

After completing the migration, verify the following:

1. Package Installation: Ensure both packages can be installed with uv pip install -e ./libs/pinjected and uv pip install -e ./libs/pinjected_openai
2. Import Paths: Verify that import paths work correctly in both packages
3. Tests: Ensure all tests pass with make test
4. CI Checks: Verify that CI checks pass
5. Documentation: Ensure documentation is up-to-date with the new structure

Rollback Plan

If issues are encountered during the migration, the following rollback steps can be taken:

1. Revert to the previous commit: git reset --hard HEAD~1
2. Push the revert: git push --force
3. Restore any deleted files from the previous commit