Reorganization and cleanup

docs/REORGANIZATION_PLAN.md

@@ -0,0 +1,123 @@
+# Repository Reorganization and Cleanup Plan
+
+## Overview
+
+This document outlines the plan for reorganizing and cleaning up the SynthPlayground repository. Two experimental approaches are being developed as candidates for replacing the current main workflow:
+
+1. **CoPilot_Experiment_0**: An AI-assisted development workflow experiment
+2. **CatCleanup**: A categorical cleanup and organization system
+
+## Current State Analysis
+
+### Main Entry Points
+
+The repository currently has multiple entry points:
+- `run.py`: Main interpreter for APPL language
+- `main.aura`: Aura-zero witness program
+- `self_improvement_project/main.py`: Self-improvement process
+- `tooling/agent_shell.py`: Interactive agent shell
+- `tooling/master_control_cli.py`: Master control CLI
+
+### Repository Structure Issues
+
+1. **Multiple overlapping systems**: The repository contains several language implementations (Aura, APPL, LFI-ILL, pLLLU) with unclear relationships
+2. **Scattered documentation**: Documentation exists in multiple formats and locations
+3. **Unclear entry points**: Multiple "main" files create confusion about the primary workflow
+4. **Archive accumulation**: The `archive/` directory contains unorganized historical code
+
+## Experimental Approaches
+
+### CoPilot_Experiment_0
+
+**Status**: Planning Phase
+
+**Objective**: Create an AI-assisted development workflow that integrates with the existing protocol system while providing a more intuitive interface for human developers.
+
+**Key Features to Develop**:
+- Unified command-line interface
+- Integration with existing tooling
+- Simplified protocol management
+- Enhanced error reporting and recovery
+- Better integration with version control
+
+**Prerequisites**:
+- [ ] Audit current agent_shell.py and master_control.py
+- [ ] Design unified CLI interface
+- [ ] Create prototype implementation
+- [ ] Develop test suite
+- [ ] Document migration path from current system
+
+### CatCleanup
+
+**Status**: Planning Phase
+
+**Objective**: Implement a categorical approach to code organization based on formal category theory principles, ensuring clear separation of concerns and well-defined interfaces.
+
+**Key Features to Develop**:
+- Category-based module organization
+- Formal interface definitions
+- Dependency graph visualization
+- Automated refactoring tools
+- Consistency verification
+
+**Prerequisites**:
+- [ ] Map current codebase to categorical structure
+- [ ] Define category boundaries and functors
+- [ ] Create migration tooling
+- [ ] Implement verification system
+- [ ] Document categorical architecture
+
+## Cleanup Tasks
+
+### Phase 1: Documentation and Inventory
+
+- [x] Create this reorganization plan
+- [ ] Document all current entry points
+- [ ] Map dependencies between modules
+- [ ] Identify deprecated code
+- [ ] Create migration checklist
+
+### Phase 2: Archive Organization
+
+- [ ] Review and categorize archive/ contents
+- [ ] Move truly obsolete code to archive/deprecated/
+- [ ] Extract reusable components from archive
+- [ ] Document archive structure
+
+### Phase 3: Module Consolidation
+
+- [ ] Consolidate language implementations
+- [ ] Merge overlapping tooling
+- [ ] Standardize naming conventions
+- [ ] Update import paths
+
+### Phase 4: Testing and Validation
+
+- [ ] Ensure all tests pass
+- [ ] Add missing test coverage
+- [ ] Validate protocol compliance
+- [ ] Run full system audit
+
+### Phase 5: Migration
+
+- [ ] Implement chosen experimental approach
+- [ ] Create migration scripts
+- [ ] Update documentation
+- [ ] Deprecate old entry points
+- [ ] Update CI/CD pipelines
+
+## Success Criteria
+
+The reorganization will be considered successful when:
+
+1. There is a single, clear entry point for the primary workflow
+2. All modules have well-defined responsibilities
+3. Dependencies are explicit and minimal
+4. Documentation is comprehensive and up-to-date
+5. All tests pass
+6. The archive is organized and documented
+7. Migration path is clear and documented
+
+## Timeline
+
+To be determined based on resource availability and priority assessment.

experiments/CatCleanup/README.md

@@ -0,0 +1,79 @@
+# CatCleanup Experiment
+
+## Overview
+
+CatCleanup is an experimental approach to reorganizing and cleaning up the SynthPlayground repository. This experiment focuses on categorizing, consolidating, and archiving files to create a more maintainable and understandable codebase.
+
+## Status
+
+**EXPERIMENTAL - NOT READY FOR PRODUCTION**
+
+This experiment is in the planning and development phase. It is being considered as a candidate for replacing the current main workflow but requires significant work before it can be deployed.
+
+## Goals
+
+1. **Categorization**: Organize files into logical categories based on their purpose and functionality
+2. **Consolidation**: Merge duplicate or similar functionality into unified modules
+3. **Archival**: Move obsolete or deprecated code to the archive directory
+4. **Documentation**: Ensure all active code is properly documented
+5. **Testing**: Verify that all functionality is covered by tests
+
+## Approach
+
+### Phase 1: Analysis
+- Scan the repository to identify all files and their purposes
+- Categorize files by type, functionality, and usage
+- Identify duplicates, obsolete code, and missing documentation
+- Generate a comprehensive inventory report
+
+### Phase 2: Planning
+- Create a detailed reorganization plan
+- Define new directory structure
+- Map old file locations to new locations
+- Identify files for archival or deletion
+
+### Phase 3: Execution
+- Move files to their new locations
+- Update import statements and references
+- Archive obsolete code
+- Update documentation
+
+### Phase 4: Validation
+- Run all tests to ensure nothing is broken
+- Verify all imports and references are correct
+- Update build system and tooling
+- Generate updated documentation
+
+## Work Required
+
+### High Priority
+- [ ] Complete repository analysis and inventory
+- [ ] Define new directory structure
+- [ ] Create file migration plan
+- [ ] Identify obsolete code for archival
+
+### Medium Priority
+- [ ] Update import statements across codebase
+- [ ] Consolidate duplicate functionality
+- [ ] Update documentation to reflect new structure
+- [ ] Update build system configuration
+
+### Low Priority
+- [ ] Optimize file organization for discoverability
+- [ ] Create migration guide for contributors
+- [ ] Update CI/CD pipelines
+
+## Dependencies
+
+- Repository analysis tools
+- Automated refactoring tools
+- Comprehensive test suite
+- Documentation generation system
+
+## Next Steps
+
+1. Run comprehensive repository analysis
+2. Generate detailed inventory report
+3. Propose new directory structure
+4. Create detailed migration plan
+5. Seek approval before execution

experiments/CoPilot_Experiment_0/README.md

@@ -0,0 +1,80 @@
+# CoPilot Experiment 0: AI-Assisted Development Workflow
+
+## Status: In Development
+
+## Overview
+
+This experiment explores an enhanced AI-assisted development workflow that could potentially replace or augment the current main workflow. The goal is to create a more intuitive, efficient, and powerful interface for agent-driven development.
+
+## Objectives
+
+1. **Streamlined Agent Interaction**: Simplify the interface between human developers and AI agents
+2. **Enhanced Context Awareness**: Improve the agent's ability to understand and maintain context across tasks
+3. **Intelligent Code Suggestions**: Implement smarter code completion and refactoring suggestions
+4. **Automated Testing Integration**: Seamlessly integrate test generation and execution
+5. **Protocol Compliance**: Ensure all operations comply with existing protocols
+
+## Key Features (Planned)
+
+### 1. Conversational Interface
+- Natural language task specification
+- Interactive clarification and refinement
+- Real-time feedback and progress updates
+
+### 2. Context Management
+- Automatic context gathering from repository
+- Intelligent file and symbol discovery
+- Dependency-aware code modifications
+
+### 3. Code Intelligence
+- Advanced code analysis and understanding
+- Refactoring suggestions based on best practices
+- Automated code review and quality checks
+
+### 4. Test-Driven Development
+- Automatic test generation for new code
+- Test-first development workflow support
+- Continuous test execution and reporting
+
+## Work Required Before Production
+
+### Phase 1: Foundation (Current)
+- [ ] Define core architecture and interfaces
+- [ ] Create prototype conversational interface
+- [ ] Implement basic context gathering
+- [ ] Establish protocol compliance framework
+
+### Phase 2: Core Features
+- [ ] Develop natural language processing for task specification
+- [ ] Implement intelligent code analysis engine
+- [ ] Create automated test generation system
+- [ ] Build context-aware code modification tools
+
+### Phase 3: Integration
+- [ ] Integrate with existing tooling system
+- [ ] Ensure compatibility with protocol system
+- [ ] Implement safety and validation checks
+- [ ] Create migration path from current main workflow
+
+### Phase 4: Testing and Validation
+- [ ] Comprehensive unit and integration testing
+- [ ] Performance benchmarking
+- [ ] Security audit
+- [ ] User acceptance testing
+
+### Phase 5: Documentation and Training
+- [ ] Complete user documentation
+- [ ] Create migration guide
+- [ ] Develop training materials
+- [ ] Update protocol documentation
+
+## Dependencies
+
+- Python 3.8+
+- Existing tooling system
+- Protocol system
+- Knowledge core
+
+## Related Issues
+
+- Issue #263: Reorganization and cleanup

plans/reorganization_and_cleanup.txt

@@ -0,0 +1,47 @@
+# Plan: Repository Reorganization and Cleanup
+# Issue #263: Prepare CoPilot_Experiment_0 and CatCleanup as candidates for replacing main
+# This plan outlines the work needed to reorganize the repository and prepare experimental approaches
+
+# Phase 1: Assessment and Documentation
+# Step 1: Analyze current main entry points and workflows
+run_in_bash_session
+python tooling/context_awareness_scanner.py run.py > /tmp/main_analysis.json
+
+# Step 2: Analyze the self_improvement_project/main.py
+run_in_bash_session
+python tooling/context_awareness_scanner.py self_improvement_project/main.py > /tmp/self_improvement_analysis.json
+
+# Step 3: Document current architecture
+message_user
+Analyzing current main entry points and documenting architecture...
+
+# Phase 2: Create Experimental Approaches
+# Step 4: Create CoPilot_Experiment_0 directory structure
+run_in_bash_session
+mkdir -p experiments/copilot_experiment_0
+
+# Step 5: Create CatCleanup directory structure
+run_in_bash_session
+mkdir -p experiments/cat_cleanup
+
+# Step 6: Generate initial experimental files
+message_user
+Creating experimental approach directories and initial files...
+
+# Phase 3: Cleanup and Organization
+# Step 7: Run full system audit to identify cleanup targets
+call_plan
+full-system-audit
+
+# Step 8: Identify duplicate and obsolete files
+run_in_bash_session
+python tooling/filesystem_lister.py --output /tmp/file_inventory.txt
+
+# Step 9: Create cleanup checklist
+message_user
+Generating cleanup checklist based on audit results...
+
+# Phase 4: Documentation
+# Step 10: Update documentation with reorganization plan
+message_user
+Reorganization and cleanup plan execution complete. Review experimental directories and cleanup checklist.