Skip to content

PHASE3_ROADMAP.md

Latest commit

 

History

History
635 lines (497 loc) · 16.7 KB

PHASE3_ROADMAP.md

File metadata and controls

635 lines (497 loc) · 16.7 KB
Raw

Phase 3 Implementation Roadmap - ThemisDB Kompendium

Status: 📋 PLANNING
Version: v1.4.0-Phase3
Date: 10. Januar 2026
Prerequisites: Phase 1 & 2 Complete ✅

Executive Summary

Phase 1 und 2 haben alle kritischen Features implementiert. Das Kompendium ist bereits produktionsbereit mit:

  • ✅ 1.7 MB HTML mit vollständiger YAML-Struktur
  • ✅ 6.9 MB PDF (natives Format)
  • ✅ 101 Diagramme mit Abbildungsverzeichnis
  • ✅ TOC, Header/Footer, interne Links

Phase 3 ist optional und fügt Advanced Features hinzu für:

  • PDF Navigation (Bookmarks/Outline)
  • Stichwortverzeichnis
  • Syntax Highlighting
  • Content Validation & QA

Phase 3 Feature Priority Matrix

Feature Business Value Technical Complexity Effort Priority
PDF Bookmarks ⭐⭐⭐⭐⭐ 🔧🔧🔧 4h HIGH
Content QA ⭐⭐⭐⭐⭐ 🔧 3h HIGH
Syntax Highlighting ⭐⭐⭐ 🔧🔧 3h MEDIUM
Stichwortverzeichnis ⭐⭐⭐ 🔧🔧 4h MEDIUM
TOC Page Numbers ⭐⭐ 🔧🔧🔧🔧🔧 12h LOW
EPUB Export ⭐⭐ 🔧🔧🔧 6h LOW

Recommendation: Implement HIGH priority items (PDF Bookmarks + Content QA)

Phase 3A: PDF Bookmarks Implementation

Objective

Add PDF navigation outline/bookmarks for improved user experience in PDF readers.

Business Value

  • ⭐⭐⭐⭐⭐ VERY HIGH: Dramatically improves navigation in 1000+ page PDF
  • Users can jump to sections/chapters instantly
  • Standard feature in professional PDFs
  • Low effort, high impact

Technical Approach

Option 1: PyPDF2 (Recommended)

from PyPDF2 import PdfReader, PdfWriter

def add_bookmarks_to_pdf(pdf_path, toc_structure, output_path):
    """
    Add navigation bookmarks to existing PDF.
    
    Args:
        pdf_path: Input PDF file
        toc_structure: Navigation structure from YAML
        output_path: Output PDF with bookmarks
    """
    reader = PdfReader(pdf_path)
    writer = PdfWriter()
    
    # Copy all pages
    for page in reader.pages:
        writer.add_page(page)
    
    # Add bookmarks hierarchically
    # Part I → parent bookmark
    #   ├─ Kapitel 1 → child bookmark
    #   ├─ Kapitel 2 → child bookmark
    #   └─ ...
    
    page_offset = 3  # Cover, TOC, Figure Index
    
    for section in toc_structure:
        if section['type'] == 'section':
            # Add part bookmark
            parent = writer.add_bookmark(
                section['title'], 
                page_offset,
                parent=None
            )
            page_offset += 1  # Section page
        elif section['type'] == 'page':
            # Add chapter bookmark
            writer.add_bookmark(
                section['title'],
                page_offset,
                parent=parent
            )
            # Calculate page count for this chapter
            # (approximation: 10 pages per chapter)
            page_offset += estimate_page_count(section['file'])
    
    # Write output
    with open(output_path, 'wb') as f:
        writer.write(f)
    
    print(f"✅ Bookmarks added: {output_path}")

def estimate_page_count(md_file):
    """
    Estimate page count from markdown file size.
    Average: ~3000 chars per page.
    """
    with open(md_file, 'r', encoding='utf-8') as f:
        content = f.read()
    
    char_count = len(content)
    return max(1, char_count // 3000)

Option 2: wkhtmltopdf TOC (Experimental)

# wkhtmltopdf has built-in TOC support
wkhtmltopdf \
  --enable-local-file-access \
  toc \
  --xsl-style-sheet toc.xsl \
  input.html \
  output.pdf

Option 3: pdftk (Command-line)

# Generate bookmark file
cat > bookmarks.txt <<EOF
BookmarkBegin
BookmarkTitle: Teil I - Grundlagen
BookmarkLevel: 1
BookmarkPageNumber: 10
BookmarkBegin
BookmarkTitle: Kapitel 1 - Einführung
BookmarkLevel: 2
BookmarkPageNumber: 11
EOF

# Add bookmarks to PDF
pdftk input.pdf update_info bookmarks.txt output output.pdf

Recommended: PyPDF2 (Option 1) - Best Python integration

Implementation Steps

Step 1: Page Number Extraction (NEW)

# step4_add_bookmarks.py
import PyPDF2
from pathlib import Path

def extract_page_mapping(html_path):
    """
    Extract which pages correspond to which chapters.
    Uses HTML anchor IDs to map chapters to page numbers.
    """
    # This is complex - alternative: use approximation
    pass

def generate_page_mapping_approx(flat_nav):
    """
    Approximate page mapping without PDF parsing.
    """
    page_map = {}
    current_page = 3  # Cover + TOC + Figure Index
    
    for item in flat_nav:
        if item['type'] == 'section':
            page_map[item['title']] = current_page
            current_page += 1  # Section page
        elif item['type'] == 'page':
            page_map[item['file']] = current_page
            # Estimate pages (can be improved with word count)
            current_page += max(1, estimate_page_count(item['file']))
    
    return page_map

Step 2: Bookmark Generation

def generate_bookmarks(pdf_path, output_path, flat_nav):
    """
    Add hierarchical bookmarks to PDF.
    """
    reader = PyPDF2.PdfReader(pdf_path)
    writer = PyPDF2.PdfWriter()
    
    # Copy pages
    for page in reader.pages:
        writer.add_page(page)
    
    # Generate page mapping
    page_map = generate_page_mapping_approx(flat_nav)
    
    # Add bookmarks
    parent_bookmark = None
    
    for item in flat_nav:
        if item['type'] == 'section':
            # New section - top-level bookmark
            parent_bookmark = writer.add_bookmark(
                item['title'],
                page_map[item['title']],
                parent=None
            )
        elif item['type'] == 'page':
            # Chapter - child bookmark
            writer.add_bookmark(
                item['title'],
                page_map[item['file']],
                parent=parent_bookmark
            )
    
    # Save
    with open(output_path, 'wb') as f:
        writer.write(f)
    
    print(f"✅ PDF with bookmarks: {output_path}")

Step 3: Integration in build_all.sh

# Step 4: Add bookmarks to PDF
echo "Step 4: Add PDF bookmarks..."
python3 step4_add_bookmarks.py

Testing & Validation

  • Open PDF in Adobe Acrobat - check bookmarks panel
  • Open PDF in Firefox - check outline/bookmarks
  • Open PDF in Chrome - check navigation sidebar
  • Verify bookmark hierarchy (Parts → Chapters)
  • Verify page numbers correct

Estimated Effort

  • Implementation: 3 hours
  • Testing: 1 hour
  • Total: 4 hours

Risks & Mitigations

Risk: Page number approximation inaccurate
Mitigation: Use actual PDF page count extraction (PyPDF2.PdfReader.pages)

Risk: wkhtmltopdf doesn't preserve bookmarks
Mitigation: Add bookmarks as post-processing step

Phase 3B: Content QA & Validation

Objective

Ensure all content is correctly formatted and complete in the final PDF.

Business Value

  • ⭐⭐⭐⭐⭐ VERY HIGH: Critical for release quality
  • Prevents errors in production
  • Builds user trust

QA Checklist

1. Structural Validation

# qa_validator.py
import yaml
from pathlib import Path

def validate_structure():
    """Validate all files referenced in YAML exist."""
    with open('mkdocs-nav.yml', 'r') as f:
        nav = yaml.safe_load(f)['nav']
    
    errors = []
    
    # Check all files exist
    for item in flatten_nav(nav):
        if item['type'] == 'page':
            file_path = Path(item['file'])
            if not file_path.exists():
                errors.append(f"Missing file: {item['file']}")
    
    # Check expected file counts
    expected_chapters = 43
    expected_appendices = 7
    
    chapter_files = list(Path('.').glob('chapter_*.md'))
    appendix_files = list(Path('.').glob('appendix_*.md'))
    
    if len(chapter_files) != expected_chapters:
        errors.append(f"Expected {expected_chapters} chapters, found {len(chapter_files)}")
    
    if len(appendix_files) != expected_appendices:
        errors.append(f"Expected {expected_appendices} appendices, found {len(appendix_files)}")
    
    return errors

2. Link Validation

def validate_links():
    """Check all internal links are valid."""
    import re
    
    errors = []
    
    for md_file in Path('.').glob('*.md'):
        with open(md_file, 'r', encoding='utf-8') as f:
            content = f.read()
        
        # Find all markdown links
        links = re.findall(r'\[([^\]]+)\]\(([^)]+)\)', content)
        
        for text, link in links:
            # Check internal links
            if link.endswith('.md'):
                target = Path(link)
                if not target.exists():
                    errors.append(f"{md_file}: Broken link to {link}")
            
            # Check anchors
            if '#' in link:
                # TODO: Validate anchor exists
                pass
    
    return errors

3. Visual QA (Manual)

PDF Visual Inspection:

  • Open output/ThemisDB-Kompendium-v1.4.0.pdf
  • Page 1: Cover page renders correctly
  • Page 2-3: TOC complete with all parts/chapters
  • Page 4-5: Figure Index with all 101 diagrams
  • Random check 10 chapters: Headers/footers visible
  • Random check 10 diagrams: Sharp, not pixelated
  • Last pages: All 7 appendices present
  • Footer: Page numbers incrementing correctly

HTML Visual Inspection:

  • Open output/ThemisDB-Kompendium-v1.4.0.html in browser
  • All diagrams render (no broken images)
  • CSS styles applied correctly
  • Links functional (click TOC entries)
  • No layout overflow/breaks

4. Content Correctness

def validate_diagram_count():
    """Ensure all diagrams are captured."""
    total_diagrams = 0
    
    for md_file in Path('.').glob('chapter_*.md'):
        with open(md_file, 'r', encoding='utf-8') as f:
            content = f.read()
        
        # Count mermaid blocks
        diagrams = content.count('```mermaid')
        total_diagrams += diagrams
    
    expected = 101
    if total_diagrams != expected:
        return f"Diagram count mismatch: expected {expected}, found {total_diagrams}"
    
    return None

5. PDF Properties

# Check PDF metadata
pdfinfo output/ThemisDB-Kompendium-v1.4.0.pdf

# Expected:
# Pages: ~1000
# File size: 5-10 MB
# PDF version: 1.4+
# Encrypted: no

Automation Script

# qa_all.py
def run_all_qa_checks():
    """Run all automated QA checks."""
    print("=" * 70)
    print("ThemisDB Kompendium - QA Validation")
    print("=" * 70)
    
    errors = []
    
    print("\n[1/4] Structural validation...")
    errors.extend(validate_structure())
    
    print("[2/4] Link validation...")
    errors.extend(validate_links())
    
    print("[3/4] Diagram count validation...")
    diagram_error = validate_diagram_count()
    if diagram_error:
        errors.append(diagram_error)
    
    print("[4/4] File size check...")
    pdf_path = Path('output/ThemisDB-Kompendium-v1.4.0.pdf')
    if pdf_path.exists():
        size_mb = pdf_path.stat().st_size / (1024 * 1024)
        print(f"  PDF size: {size_mb:.2f} MB")
        if size_mb > 15:
            errors.append(f"PDF too large: {size_mb:.2f} MB > 15 MB")
    else:
        errors.append("PDF file not found")
    
    # Results
    print("\n" + "=" * 70)
    if errors:
        print(f"❌ QA FAILED - {len(errors)} errors found:")
        for error in errors:
            print(f"  - {error}")
        return False
    else:
        print("✅ QA PASSED - All checks successful")
        return True

if __name__ == "__main__":
    success = run_all_qa_checks()
    exit(0 if success else 1)

Estimated Effort

  • Script development: 2 hours
  • Manual QA: 1 hour
  • Total: 3 hours

Phase 3C: Syntax Highlighting (Optional)

Objective

Add color syntax highlighting to code blocks for better readability.

Business Value

  • ⭐⭐⭐ MEDIUM: Nice-to-have, improves developer experience
  • Makes code examples more readable
  • Professional appearance

Implementation

# In step2_generate_html.py
from pygments import highlight
from pygments.lexers import get_lexer_by_name, guess_lexer
from pygments.formatters import HtmlFormatter

def process_code_with_highlighting(html_content):
    """
    Add syntax highlighting to code blocks.
    """
    import re
    
    # Pattern: <pre><code class="language-python">...</code></pre>
    pattern = r'<pre><code class="language-(\w+)">(.*?)</code></pre>'
    
    def highlight_code(match):
        lang = match.group(1)
        code = match.group(2)
        
        try:
            lexer = get_lexer_by_name(lang, stripall=True)
        except:
            # Fallback to plain text
            return match.group(0)
        
        formatter = HtmlFormatter(
            style='monokai',
            noclasses=False,
            cssclass='highlight'
        )
        
        highlighted = highlight(code, lexer, formatter)
        return highlighted
    
    return re.sub(pattern, highlight_code, html_content, flags=re.DOTALL)

# CSS for Pygments
PYGMENTS_CSS = HtmlFormatter(style='monokai').get_style_defs('.highlight')

Estimated Effort

  • Implementation: 2 hours
  • Testing: 1 hour
  • Total: 3 hours

Phase 3D: Stichwortverzeichnis (Optional)

Objective

Generate alphabetical keyword index from glossary.

Business Value

  • ⭐⭐⭐ MEDIUM: Useful for reference, but not critical
  • Helps users find definitions quickly

Implementation

def generate_keyword_index():
    """
    Generate keyword index from appendix_h_glossary.md
    """
    glossary_path = Path('appendix_h_glossary.md')
    
    with open(glossary_path, 'r', encoding='utf-8') as f:
        content = f.read()
    
    # Extract terms (format: **Term**: Definition)
    pattern = r'\*\*([^*]+)\*\*:\s*(.+?)(?=\n\*\*|\Z)'
    terms = re.findall(pattern, content, re.DOTALL)
    
    # Sort alphabetically
    terms.sort(key=lambda x: x[0].lower())
    
    # Generate HTML
    html = '<div class="keyword-index">'
    html += '<h1>Stichwortverzeichnis</h1>'
    html += '<dl class="index-list">'
    
    for term, definition in terms:
        html += f'<dt>{term}</dt>'
        html += f'<dd>{definition.strip()}</dd>'
    
    html += '</dl></div>'
    
    return html

Estimated Effort

  • Implementation: 3 hours
  • Testing: 1 hour
  • Total: 4 hours

Phase 3E: NOT RECOMMENDED

TOC with Page Numbers

Reason: Very complex (requires two-pass generation)
Effort: 12+ hours
Value: Low (bookmarks are better alternative)

EPUB Export

Reason: Different use case, limited demand
Effort: 6+ hours
Value: Low

Implementation Timeline

Week 1: High Priority (7 hours)

  • Day 1-2: PDF Bookmarks (4h)
    • Implement PyPDF2 integration
    • Test in multiple PDF readers
  • Day 3: Content QA (3h)
    • Develop QA automation scripts
    • Manual visual inspection

Week 2: Medium Priority (Optional - 7 hours)

  • Day 4: Syntax Highlighting (3h)
    • Integrate Pygments
    • Test with AQL code samples
  • Day 5: Stichwortverzeichnis (4h)
    • Parse glossary
    • Generate index page

Success Metrics

Phase 3A (PDF Bookmarks)

  • Bookmarks visible in Adobe Acrobat
  • Correct hierarchy (Parts → Chapters)
  • Page numbers accurate (±2 pages tolerance)
  • All 11 parts + 53 chapters bookmarked

Phase 3B (Content QA)

  • Zero structural errors
  • Zero broken links
  • All 101 diagrams present
  • All 43 chapters + 7 appendices complete
  • PDF size 5-10 MB

Phase 3C (Syntax Highlighting)

  • Code blocks colorized
  • All languages supported (Python, AQL, bash, etc.)
  • Readable in both HTML and PDF

Phase 3D (Stichwortverzeichnis)

  • All glossary terms indexed
  • Alphabetical order correct
  • Definitions complete

Recommendation

IMPLEMENT NOW:

  1. ✅ PDF Bookmarks (4h) - High value, medium effort
  2. ✅ Content QA (3h) - Critical for release

IMPLEMENT LATER (Optional): 3. ⏳ Syntax Highlighting (3h) - Nice-to-have 4. ⏳ Stichwortverzeichnis (4h) - Nice-to-have

DO NOT IMPLEMENT: 5. ❌ TOC Page Numbers - Too complex 6. ❌ EPUB Export - Low demand

Conclusion

Phase 3 adds professional polish to an already production-ready system. Focus on high-value features (PDF Bookmarks + QA) for maximum impact with minimal effort.

Total Recommended Effort: 7 hours (Bookmarks + QA)
Total Optional Effort: +7 hours (Syntax + Index)

Document Version: v1.0
Last Updated: 10. Januar 2026
Status: 📋 Planning Phase
Next: Approve Phase 3A implementation