Reorganization

[metavacua]

@RepoBirdBot plan
There needs to be a reorganization of the repository and its directories, and there are expected to be a number of subsequent issues created by this re-arrangement due to absolute vs relative paths and Google's coding assistant, Jules, specific idiosyncracies.

[gitauto-ai]

Click the checkbox below to generate a PR!

• Generate PR

[repobird]

Status: ✅ Completed @RepoBirdBot plan

Timeline: ✅ Queued → ✅ Initializing → ✅ Processing → ✅ Finalizing → ✅ Completed

🔗 Run Details: https://repobird.ai/repos/issue-runs/1320

📊 Remaining Runs: 10 Basic, 3 Pro, 4 Plan.

📝 Note: The prompt includes the original issue title, body, and any additional text provided in the triggering command comment.

[RepoBirdBot]

Repository Reorganization Plan

Problem Summary

The repository currently suffers from significant organizational issues that impact maintainability, discoverability, and development workflow:

1. Root Directory Pollution: 96+ files in the root directory, including Python scripts, JavaScript/JSX files, Lisp files, test files, configuration files, and documentation
2. Mixed Concerns: Production code, test files, demo scripts, experiment artifacts, and documentation are intermixed without clear separation
3. Unclear Module Boundaries: Multiple related files scattered across different locations (e.g., logic system components in root vs. logic_system/ directory)
4. Path Complexity: The reorganization will expose absolute vs. relative path issues, particularly affecting:
   • Import statements in Python modules
   • Tooling that references file paths
   • Build configuration paths
   • Test discovery and execution paths
   • Jules (Google's coding assistant) which may have specific path handling idiosyncrasies

Proposed Solution

Implement a hierarchical, purpose-driven directory structure that follows industry best practices for polyglot repositories, with careful migration of path references to prevent breakage.

Proposed Directory Structure

/
├── src/                          # All production source code
│   ├── core/                     # Core language implementations
│   │   ├── aura/                 # Aura language (from aura_lang/)
│   │   ├── lfi_ill/              # LFI-ILL logic system (from lfi_ill/)
│   │   └── logic_system/         # Logic system (from logic_system/)
│   ├── interpreters/             # Language interpreters and parsers
│   │   ├── aura.py
│   │   ├── parser.py
│   │   ├── interpreter.py
│   │   └── type_checker.py
│   ├── witnesses/                # Decidability witnesses
│   │   ├── classical_logic_witness.py
│   │   ├── presburger_arithmetic_witness.py
│   │   ├── skolem_arithmetic_witness.py
│   │   └── witness_d.py
│   ├── agents/                   # Agent-related utilities
│   │   ├── planning.py
│   │   └── dbpedia_client.py
│   └── web/                      # Web applications and components
│       ├── gemini/               # Gemini-related web apps
│       │   ├── GeminiAppCanvasAgent.jsx
│       │   ├── GeminiAppCanvasCLI.jsx
│       │   ├── GeminiAppJavascriptIntrospector.jsx
│       │   ├── GeminiAppProbeReactApp.jsx
│       │   ├── GeminiCDNCanary.jsx
│       │   ├── GeminiIsoGitTest.jsx
│       │   ├── GeminiLibraryTester.jsx
│       │   ├── GeminiOSProfiler.jsx
│       │   └── GeminiResourceProfiler.jsx
│       ├── activity_tools/       # Activity analysis tools
│       │   ├── MyActivityAnalysisTool.jsx
│       │   └── MyActivityReductionTool.jsx
│       ├── rd_consumer/          # RD Consumer components
│       │   ├── RDConsumerAIKernel.jsx
│       │   ├── RDConsumerAIKernelAlt.jsx
│       │   └── DataSourceRDAIKernel.js
│       ├── sequoia/
│       │   └── SequoiaReactApp.jsx
│       ├── index.html
│       ├── script.js
│       ├── style.css
│       └── gemini_logic.js
├── tests/                        # All test files (existing + moved)
│   ├── unit/
│   │   ├── test_interpreter.py
│   │   ├── test_parser.py
│   │   ├── test_type_checker.py
│   │   ├── test_planning.py
│   │   ├── test_classical_logic_witness.py
│   │   ├── test_presburger_arithmetic_witness.py
│   │   ├── test_skolem_arithmetic_witness.py
│   │   └── test_plan_generator.py
│   ├── integration/
│   │   ├── test-absolute.html
│   │   └── test-relative.html
│   ├── complexity/
│   │   ├── high_complexity_test.py
│   │   └── high_complexity_test.udc
│   ├── appl/                     # Keep existing
│   ├── data/                     # Keep existing
│   ├── protocols/                # Keep existing
│   └── fixtures/
│       ├── test.appl.lfi_ill
│       ├── test.appl.py
│       ├── paradox.lfi_ill
│       ├── integration_demo.aura
│       ├── integration_demo.lfi_ill
│       ├── main.aura
│       └── test_executor.aura
├── examples/                     # Keep existing + add demos
│   └── DangerousCodeTest.jsx    # Move demo files here
├── experiments/                  # Keep existing structure
├── tooling/                      # Keep existing structure (164 Python files)
├── protocols/                    # Keep existing structure (131 files)
├── docs/                         # All documentation
│   ├── spec/                     # Keep existing
│   ├── theory/                   # Move from language_theory/
│   │   ├── ontology/
│   │   ├── toolchain/
│   │   ├── witnesses/
│   │   └── *.md files
│   ├── research/                 # Move from research/
│   ├── reports/                  # Move from reports/
│   ├── postmortems/              # Move from postmortems/
│   ├── architecture/             # New: for architecture docs
│   │   ├── Agentic_Toolchain.md
│   │   ├── AIACHC.md
│   │   └── architectural_review.md
│   ├── research_artifacts/       # Research documents
│   │   ├── research_plan.md
│   │   ├── research_report.md
│   │   ├── RESEARCH_FINDINGS.md
│   │   ├── RESEARCH_REPORT.md
│   │   └── various .txt research files
│   └── manuals/                  # Jules-specific and setup docs
│       ├── jules-environment-limitations-analysis.txt
│       ├── jules-repository-setup-and-self-improvement.txt
│       ├── github-repository-setup-for-jules.txt
│       └── github-repository-special-files-explained.txt
├── knowledge_core/               # Keep existing structure
├── plans/                        # Keep existing structure
├── archive/                      # Keep existing structure
├── utils/                        # Keep existing structure
├── hdl/                          # New: HDL-related files
│   ├── HDL.LSP
│   ├── HDL_alts.LSP
│   └── HDLProvev*.lsp (all versions)
├── scripts/                      # New: utility scripts
│   ├── archive_script.sh
│   └── enrich_protocols.py
├── config/                       # New: configuration files
│   ├── build_config.yaml
│   ├── language_classification.yaml
│   ├── language_classification_schema.json
│   ├── protocol.context.jsonld
│   └── resource_map.json
├── .github/                      # Keep existing
├── .git/                         # Keep existing
├── AGENTS.md                     # Keep in root (generated)
├── AGENTS.md.bak                 # Keep in root
├── protocols.yaml-ld             # Keep in root (generated)
├── LICENSE                       # Keep in root
├── CONTRIBUTING.md               # Keep in root
├── SECURITY.md                   # Keep in root
├── LOGGING_SCHEMA.md             # Keep in root
├── README.md.template            # Keep in root
├── README.v2.md.template         # Keep in root
├── Makefile                      # Keep in root
├── requirements.txt              # Keep in root
├── .gitignore                    # Keep in root
├── .flake8                       # Keep in root
├── .pre-commit-config.yaml       # Keep in root
└── run.py                        # Keep in root (entry point)

Implementation Steps

Phase 1: Preparation and Analysis (Non-Destructive)

1. Create directory structure - Create all new directories without moving files
2. Audit path references - Scan all code for path references:
   • Python imports (import, from)
   • File path strings (relative and absolute)
   • Build system paths (build_config.yaml)
   • Tool invocations in tooling/
   • Test discovery patterns
   • .gitignore patterns
3. Document current paths - Create a mapping file: migration_map.json with old → new paths
4. Identify high-risk files - Flag files that will require complex path updates

Phase 2: Core Source Migration

5. Move language implementations:
   • aura_lang/ → src/core/aura/
   • lfi_ill/ → src/core/lfi_ill/
   • logic_system/ → src/core/logic_system/
6. Move interpreter files:
   • aura.py, parser.py, interpreter.py, type_checker.py → src/interpreters/
7. Move witness files → src/witnesses/
8. Move agent utilities:
   • planning.py, dbpedia_client.py → src/agents/
9. Update imports in moved files - Fix relative imports to work from new locations

Phase 3: Web Assets Migration

10. Move Gemini web apps → src/web/gemini/
11. Move activity tools → src/web/activity_tools/
12. Move RD consumer components → src/web/rd_consumer/
13. Move Sequoia app → src/web/sequoia/
14. Move core web files → src/web/
15. Update web asset paths - Fix script/style references in HTML/JSX files

Phase 4: Test Migration

16. Move unit tests → tests/unit/
17. Move integration tests → tests/integration/
18. Move complexity tests → tests/complexity/
19. Move test fixtures → tests/fixtures/
20. Update test imports - Fix imports to reference new source locations
21. Update pytest/test discovery - Ensure test runners find tests in new locations

Phase 5: Documentation Reorganization

22. Move language theory docs - language_theory/ → docs/theory/
23. Move research docs - research/ → docs/research/
24. Move reports - reports/ → docs/reports/
25. Move postmortems - postmortems/ → docs/postmortems/
26. Create architecture docs - Move architecture-related files → docs/architecture/
27. Create research artifacts - Move research documents → docs/research_artifacts/
28. Create manuals - Move Jules-specific docs → docs/manuals/

Phase 6: Configuration and Utility Files

29. Move HDL files → hdl/
30. Move utility scripts → scripts/
31. Move configuration files → config/
32. Move demo files → examples/

Phase 7: Build System Updates

33. Update build_config.yaml - Fix all paths in build targets
34. Update tooling paths - Scan all 164 Python files in tooling/ for hardcoded paths:

• Protocol compiler source paths
• Knowledge integrator paths
• Documentation builder paths
• Test runner paths
• Tool references to moved modules

35. Update requirements.txt - If any paths reference local modules
36. Update .gitignore - Adjust patterns for new structure

Phase 8: Path Reference Updates

37. Update absolute paths - Convert absolute paths to relative where appropriate
38. Update relative paths - Adjust relative paths for new directory depths
39. Update import paths - Comprehensive import statement updates:

• Direct imports: from aura import X → from src.core.aura import X
• Tool imports in tooling/ that reference moved modules
• Test imports that reference source modules

40. Update protocol references - Fix paths in protocol YAML files
41. Update plan references - Fix paths in plan files

Phase 9: Testing and Validation

42. Run build system - Execute python3 tooling/builder.py --target all
43. Run full test suite - Execute python3 -m pytest and validate all tests pass
44. Validate tooling - Test each tool in tooling/ individually
45. Check git status - Ensure no broken symlinks or orphaned files
46. Validate documentation generation - Ensure docs build correctly
47. Test protocol compilation - Ensure AGENTS.md generates correctly

Phase 10: Jules-Specific Validation

48. Test Jules path handling - Verify Jules can navigate new structure
49. Check Jules-specific patterns - Review Jules documentation for path handling quirks
50. Update Jules configuration - If Jules has configuration files, update paths
51. Test Jules workflows - Run common Jules operations to catch path issues

Phase 11: Cleanup

52. Remove empty directories - Clean up old directory structure
53. Update documentation - Update README and contributing guides with new structure
54. Archive migration artifacts - Move migration_map.json to archive/
55. Create migration postmortem - Document issues encountered and solutions

Files to Modify

High-Priority (Core Build System)

• build_config.yaml - All build target paths
• tooling/builder.py - May contain hardcoded paths
• tooling/protocol_compiler.py - Protocol source discovery
• tooling/knowledge_integrator.py - Knowledge source paths
• tooling/doc_builder.py - Documentation source paths
• tooling/master_agents_md_generator.py - Protocol compilation paths
• .gitignore - Update path patterns

Medium-Priority (Tooling with Path References)

All 164 files in tooling/ need review, particularly:

• tooling/symbol_map_generator.py - Code scanning paths
• tooling/dependency_graph_generator.py - Dependency scanning paths
• tooling/context_awareness_scanner.py - File scanning
• tooling/filesystem_lister.py - Directory listing
• tooling/auditor.py - Protocol and plan validation paths
• tooling/plan_manager.py - Plan registry paths
• Test runners and validators

Medium-Priority (Source Files with Imports)

Approximately 294 code files with import statements:

• All moved Python files in src/ - Update internal imports
• All files in utils/ - May import moved modules
• All test files - Import moved source modules
• Web components - If they reference Python modules

Low-Priority (Documentation and Configuration)

• All documentation files that reference paths
• Protocol YAML files with path references
• Plan files with path references
• Configuration files in config/

Testing Approach

Pre-Migration Testing

1. Baseline test run - Run full test suite and document current pass/fail state
2. Build system verification - Build all targets and verify outputs
3. Path inventory - Generate comprehensive list of all path references

During Migration Testing

4. Incremental validation - After each phase, run affected tests
5. Import validation - Use python -m py_compile to check syntax after import changes
6. Module resolution testing - Test that all imports resolve correctly

Post-Migration Testing

7. Full test suite - Run entire test suite and compare to baseline
8. Build verification - Build all targets and diff outputs with baseline
9. Tool validation - Execute each tooling script with test inputs
10. Documentation build - Verify all documentation generates correctly
11. End-to-end workflow test - Run complete FDC workflow
12. Jules compatibility test - Test Jules operations in new structure

Regression Testing

13. Path resolution audit - Verify no broken path references remain
14. Import audit - Verify no import errors
15. Protocol compilation test - Verify AGENTS.md is identical to baseline
16. Knowledge integration test - Verify knowledge artifacts generate correctly

Success Criteria

1. Zero broken imports - All Python modules import successfully
2. Zero broken paths - All file path references resolve correctly
3. All tests pass - Test suite passes with same or better results than baseline
4. Build system works - All build targets execute successfully
5. Tools functional - All 164+ tooling scripts execute without path errors
6. Documentation builds - All documentation generates correctly
7. Improved discoverability - Clear separation of concerns visible in directory structure
8. Jules compatible - Jules can navigate and operate in new structure
9. No absolute paths - All cross-repository references use relative paths (where appropriate)
10. Git clean state - No orphaned files, broken symlinks, or unexpected changes

Potential Challenges

1. Import Path Complexity

Challenge: Python import statements may break when files move, especially with relative imports.
Mitigation:

• Use search-and-replace tools systematically
• Test each module after moving with python -m py_compile
• Consider using absolute imports from repository root
• May need to add src/ to PYTHONPATH or create __init__.py files

2. Circular Dependencies

Challenge: Moving files may expose hidden circular dependencies.
Mitigation:

• Use import dependency analysis tools before moving
• Break circular dependencies before migration
• Consider refactoring if dependencies are too complex

3. Build System Path Hardcoding

Challenge: Build tools may have hardcoded assumptions about file locations.
Mitigation:

• Thoroughly audit build_config.yaml and all tooling
• Use configuration-driven paths where possible
• Test build system incrementally after each phase

4. Test Discovery Breakage

Challenge: pytest and other test runners may not discover tests in new locations.
Mitigation:

• Update pytest.ini or pyproject.toml with new test paths
• Ensure __init__.py files exist in test directories if needed
• Verify test discovery before and after migration

5. Jules-Specific Path Handling

Challenge: Jules may have undocumented assumptions about repository structure.
Mitigation:

• Review Jules documentation thoroughly
• Test Jules operations after each phase
• Document any Jules-specific workarounds
• Keep Jules-specific docs in easily discoverable location

6. Generated File Dependencies

Challenge: Generated files (AGENTS.md, protocols.yaml-ld, README.md) may have path dependencies.
Mitigation:

• Regenerate all artifacts after migration
• Diff generated files with baseline to catch issues
• Update generators before moving source files

7. Absolute vs Relative Path Confusion

Challenge: Mixed use of absolute and relative paths may cause issues.
Mitigation:

• Establish convention: relative paths for intra-repo references
• Convert absolute paths to relative where appropriate
• Document path conventions in CONTRIBUTING.md

8. Large-Scale Search-Replace Errors

Challenge: Automated path updates may over-match or miss edge cases.
Mitigation:

• Use targeted, file-type-specific replacements
• Test after each batch of replacements
• Manual review of critical files (build system, entry points)

9. Git History and Blame Preservation

Challenge: Moving files may complicate git blame and history tracking.
Mitigation:

• Use git mv instead of manual move + add/delete
• Document migration in commit message with file mapping
• Consider using git log --follow flag in documentation

10. Partial Migration State Risks

Challenge: Repository may be broken during multi-phase migration.
Mitigation:

• Work in a feature branch
• Keep phases small and testable
• Have rollback plan for each phase
• Consider using feature flags for gradual cutover

Risk Assessment

• High Risk: Build system changes, import path updates, test discovery
• Medium Risk: Jules compatibility, circular dependencies, generated files
• Low Risk: Documentation moves, configuration consolidation, HDL file moves

Rollback Plan

If critical issues arise:

1. Revert to previous commit using git reset --hard
2. Document specific failure in migration postmortem
3. Address root cause before retrying
4. Consider smaller migration phases

Post-Migration Activities

1. Update CONTRIBUTING.md with new structure and conventions
2. Create migration postmortem in docs/postmortems/
3. Update onboarding documentation
4. Notify team/users of structural changes
5. Monitor for path-related issues in subsequent development
6. Create follow-up issues for any deferred cleanup

---

To execute this plan, copy the plan text above into a new comment, make any desired changes, and start the comment with @repobird run <description> where <description> includes the quoted reply + any other changes/context edits needed.

Plan Generated by Repobird 📦🐦