Skip to content

CHANGELOG.md

Latest commit

 

History

History
452 lines (344 loc) · 20 KB

CHANGELOG.md

File metadata and controls

452 lines (344 loc) · 20 KB
Raw

Changelog

All notable changes to the Markdown-based Automation Framework.

[v2.3.0] - 2025-12-26

🚀 Scalability & Performance

Enterprise-Ready for 10,000+ Libraries

Hybrid Registry Architecture

  • Persistent Caching: Implemented .cache/registry.json for instant subsequent startups
    • First run: Indexes all libraries (~5s for 1000 files)
    • Subsequent runs: ~0.05s regardless of library count
    • Automatic cache invalidation based on file modification times
  • Parallel Bootstrapping: Added ProcessPoolExecutor for multi-core cold boot indexing
    • Processes 1000+ files in parallel during initial scan
    • Includes Windows multiprocessing fixes (__name__ == '__main__' guards)
  • Header-Only Scanning: Optimized MarkdownParser.parse_metadata_only()
    • Reads only first 50 lines for metadata extraction
    • Skips heavy content parsing until action is actually used
    • Lazy loading: Full parsing happens on-demand

Keyword Index & Search

  • Inverted Index: Implemented keyword-based search for Actions and Pages
    • Find "Login User" even when searching "User Login"
    • Tokenizes action names and page metadata
    • Supports pages metadata for multi-page actions
  • Persistent Keyword Cache: Optimized for 50k+ libraries
    • Fast path: Load from .cache/keywords.json (~0.01s)
    • Incremental updates: Only rebuild changed files
    • Cold boot: Build from scratch + persist (~5s for 50k files)
    • Smart cache invalidation with file modification tracking

🎨 Rich Logging V2

Cinematic, Phase-Based Console Experience

4-Phase Logging Structure

  1. 🚀 BOOTSTRAPPING: CLI arguments, project config, execution mode
  2. ⚙️ INITIALIZATION: Service startup (NLP, Watcher) + Registry stats
  3. ⚡ EXECUTING TESTCASES: Unified flow for steps, verification, and post-conditions
  4. 🏁 TEARDOWN: System cleanup (Observer, Browser, Playwright) + Report generation

Granular Statistics

  • Registry Stats: Libraries, Actions, Pages, Keywords counts
  • Cache Performance: "Cached: X, Rebuilt: Y" format
    • Shows cache hit/miss rates
    • Displays boot time with millisecond precision
  • Configuration Display: Project name, Mode (Headful/Turbo), Intelligent services status
  • Visual Hierarchy: Section headers, status icons (⚡, ✓, ❌), color-coded output

🐛 Bug Fixes & Stability

Comprehensive Codebase Audit - 15 Bugs Fixed

Critical Fixes (2)

  1. Duplicate Exception Handlers (locator.py)

    • Fixed unreachable code in element finding
    • Combined exception handlers for proper error logging
    • Ensures self-healing triggers correctly

  2. State Saving Error Handling (runner.py)

    • Added try-catch with browser context validation
    • Prevents crashes when state saving fails
    • Proper error messages and status reporting

High Priority Fixes (3)

  1. Broad Exception Handling (Multiple files)

    • Replaced generic except Exception with specific types
    • Registry: FileNotFoundError, json.JSONDecodeError, PermissionError
    • Observer: AttributeError, KeyError for network monitoring
    • Improved error logging with appropriate levels

  2. Missing Test Name Attribute (runner.py)

    • Extracts current_test_name from file path
    • Ensures unique state files per test
    • Prevents state file overwrites

  3. Tag Cleanup Race Condition (runner.py)

    • Tracks tags pushed with counter instead of checking locals()
    • Safe cleanup even if exception occurs during tag setup
    • Prevents tag stack corruption across tests

Medium Priority Fixes (5)

  1. Silent Registry Cache Failures (registry.py)

    • Added logging for cache save/load errors
    • Specific exception handling for permission errors
    • Warns on cache corruption

  2. Silent Keyword Cache Failures (registry.py)

    • Added logging for keyword index operations
    • Debug-level logs for cache hits
    • Error-level logs for unexpected failures

  3. Memory Leak in Observer (observer.py)

    • Explicitly removes matched network latches
    • Prevents accumulation during long test runs
    • Fixed comment claiming "waiter removes latch"

  4. State Save Timeout (state.py)

    • Added 10-second timeout with asyncio.wait_for()
    • Prevents hanging on slow storage operations
    • Proper timeout error handling

  5. Inconsistent Error Logging (Framework-wide)

    • Established logging standards
    • logger.error() for test failures
    • logger.warning() for recoverable issues
    • logger.debug() for verbose diagnostics

Low Priority Fixes (3)

  1. Code Quality: Improved exception handling in critical paths
  2. Documentation: Added comprehensive docstrings to key methods
  3. Code Consistency: Verified no unused imports, consistent f-string usage

🗂️ Project Structure

Cleaner, More Organized

  • Renamed Folders:
    • states/ → .browserstates/ (hidden, descriptive)
    • Centralized caches in .cache/ directory
  • Automatic Folder Creation:
    • .cache/ - Created by registry on first run
    • .browserstates/ - Created by StateManager on initialization
    • reports/ - Created by Reporter during test execution
  • Updated .gitignore:
    • Excludes .cache/, .browserstates/, reports/
    • Removed old states/ entry

📊 Performance Metrics

Scenario v2.2.1 v2.3.0 Improvement
1,000 Libraries (Cold Boot) ~5.0s ~5.0s Same (one-time)
1,000 Libraries (Warm Boot) ~5.0s ~0.05s 100x faster
10,000 Libraries (Warm Boot) ~50.0s ~0.2s 250x faster
Keyword Search N/A ~0.01s New feature

🔧 Technical Improvements

  • Error Handling: Specific exception types, better logging
  • Resource Management: Fixed memory leaks, added timeouts
  • Code Quality: Comprehensive docstrings, consistent formatting
  • Scalability: Persistent caching, parallel processing, lazy loading
  • User Experience: Rich logging, granular stats, visual hierarchy

[v2.2.1] - 2025-12-25

🚑 Critical Hotfixes (Stability)

  • Fix Silent Failures: Parallel assertion batches (# Assertions) now correctly raise CompositeAssertionError on failure, preventing false positives.
  • Fix State Leakage: implemented ContextManager.reset() to clear variables between test runs, ensuring isolation.
  • Fix Race Condition: Refactored SmartLocator to be stateless (removed last_observation instance var), preventing log corruption during parallel execution.
  • Fix Unbounded Cache: Added MAX_CACHE_SIZE=1000 to SmartLocator to prevent memory leaks in long-running tests.
  • Robust Regex: Protected Wait for API against invalid user-supplied regex patterns.

[2.2.0] - 2025-12-25

🚀 Major Admin & Performance (Orbit Engine)

  • Turbo Mode: Added --turbo flag to block heavy assets (images, fonts, media) for ultra-fast validation.
  • Smart Selector Caching: SmartLocator now caches successful selectors in memory, eliminating redundant DOM scanning for repeated elements.
  • Cache Safety: Auto-clears cache on Navigate actions to prevent stale state.

🧠 Intelligent State Verification

  • Network Latching: Implemented precise Wait for API "METHOD URL" synchronization. Tests now wait for backend signals rather than hardcoded sleeps.
  • API Mocking Support: Architecture prepared for future response mocking.

🐛 Bug Fixes & Refinements

  • Documentation: Unified all version headers to v2.2.0.
  • Cleanliness: Removed temporary debug logs (test_latch_debug.log).

[2.1.0] - 2025-12-25

🚀 Major Features

Two-Tier Parallel Verification

  • Micro-Parallelism: AssertionEngine now performs simultaneous checks for all child elements within a container (using asyncio.gather), drastically reducing verification time.
  • Macro-Parallelism: TestRunner executes all autonomous assertions in a Page Object (# Assertions section) concurrently.
  • Performance: Reduced Dashboard health check time by ~85% (from ~10s to ~1.5s).

🐛 Bug Fixes

  • URL Assertion Timing: Fixed race condition in Wait for url. Implemented page.wait_for_url with dynamic regex matching and increased timeout (30s) to handle slow redirects robustly.
  • Cleanup: Removed deprecated container.py and various temporary debug artifacts.

🛡️ Improvements

  • Robustness: Enhanced key normalization for assertions.
  • Logging: Added distinct tags for Parallel execution (Continuum Parallel) to differentiate from sequential steps.

[2.0.1] - 2024-12-25

🎨 Modern UI & Dashboard (Continuum v2.0.1)

  • Premium Aesthetics: Implemented a dark-themed, glassmorphism-inspired HTML dashboard.
  • Smart Metrics: Added automatic test duration tracking and summary statistics (Pass/Fail/API Errors).
  • Interactive Timeline: Redesigned step execution flow with Lucide icons and collapsible details.
  • High-Res Media: Integrated a high-resolution screenshot lightbox for detailed failure analysis.
  • Improved API Table: Enhanced network failure reporting with method badges and payload previews.

🛡️ Core Stability

  • Reporting Fix: Resolved a NameError where api_error_count was undefined in certain failure scenarios.

[2.0.0] - 2024-12-25

🏗️ Architecture Redesign (Event-Driven)

  • Decoupled Service Model: Transitioned from rigid Dependency Injection to a flexible, Event-Driven Services architecture mediated by an EventBus.
  • Core Orchestration: TestRunner now acts as a coordinator, delegating specific logic to high-cohesion services (NLPEngine, AssertionEngine, SmartLocator).
  • Dead Code Cleanup: Archived the legacy container.py dependency injection framework.

🚀 Major Enhancements

  • Advanced NLP (SpaCy Integration): Added support for powerful syntax-aware step parsing using SpaCy's en_core_web_sm model.
  • Hybrid NLP Engine: Implemented an automatic fallback mechanism to Regex-based parsing if SpaCy models are missing.
  • Agentic Self-Healing: Enhanced SmartLocator to not only resolve broken selectors but also auto-update the source .md files with fixed locators.
  • Intelligent Error Watcher: Proactive monitoring for unexpected error popups and system-level messages with automatic evidence capture.

🛡️ Legacy Logic & Stability Fixes

  • Data Masking 2.0: Refined DataMasker regex to correctly redact sensitive data (passwords) even when provided as unquoted values in steps.
  • Variable Resolution Hygiene: Added aggressive whitespace and "value " prefix stripping in the NLP engine to prevent variable resolution failures.
  • URL Normalization: Fixed inconsistent trailing slash handling in "Wait for url" steps.
  • Auto-Context Linking: Actions now automatically inherit the context (locators/metadata) of their associated Page Object files.

📊 Professional Logging

  • Hierarchical Health Checks: Page verification steps now perform a live "Health Check" of all child elements, logged clearly in -vv mode.
  • Tiered Verbosity:
    • Default: Essential steps and assertions.
    • -v: Detailed locator resolution logs (🔍).
    • -vv: Full debug tracing, API monitoring, and network logs.

[1.5.0] - 2024-12-24

🚀 Enhancements

Logging System 2.0

  • Rich Console Output: Implemented RichLogger-style formatting with ANSI colors, icons, and indentation.
  • Thread-Safe Logging: Added locking mechanisms to prevent garbled output during concurrent execution (Observer vs Main).
  • Hierarchical Logs: Visual nesting of library actions and steps for better readability.

Core Stability

  • Robust Observer Shutdown: Improved teardown logic in DOMObserver to prevent network monitoring race conditions.

[1.4.0] - 2025-12-24

🚀 Enhancements

Logging System 2.0

  • Rich Console Output: Implemented RichLogger-style formatting with ANSI colors, icons, and indentation.
  • Thread-Safe Logging: Added locking mechanisms to prevent garbled output during concurrent execution (Observer vs Main).
  • Hierarchical Logs: Visual nesting of library actions and steps for better readability.

Core Stability

  • Robust Observer Shutdown: Improved teardown logic in DOMObserver to prevent network monitoring race conditions.

🐛 Bug Fixes

Critical Fixes (4)

  1. Healer Crash: Fixed NameError in healer.py preventing self-healing.
  2. Locator Regex Injection: Fixed vulnerability where special characters in locators caused crashes.
  3. Assertion Quote Logic: Fixed logic flaw where quotes in step text caused assertions to fail.
  4. Scanner Generatio: Fixed SyntaxError and invalid CSS generation for special characters in scanner.py.

📝 Files Modified

  • framework/logger.py (Rewritten)
  • framework/runner.py (Enhanced logging calls)
  • framework/healer.py
  • framework/locator.py
  • framework/assertion.py
  • framework/scanner.py
  • framework/observer.py

[1.3.0] - 2025-12-24

🚀 Major Features

Async Core Architecture

  • Full Async Migration: Refactored entire framework to use Playwright's Asynchronous API.
  • Non-Blocking Execution: runner.py, locator.py, observer.py, and assertion.py now use async/await.
  • Configurable Stability: Replaced hardcoded sleeps with Config.STABILITY_WAIT_MS (default 1000ms).

Robust Assertions 2.0

  • Bottom-Up Verification: Verifies child elements before parents.
  • Soft Pass: Allows tests to proceed if parent containers are hidden but functional children are visible.
  • Key Normalization: Intelligently matches "Login Form" to login_form.

Enhanced Parser

  • Container Hierarchy: Support for indented elements in Markdown even if the parent has a selector value (e.g., Sidebar > Menu).

📚 Documentation

  • New CONTRIBUTING.md: Dedicated guide for developers.
  • New ARCHITECTURE.md: Detailed technical specification of internal logic (Scoring, NLP).
  • Expanded README.md: Added "Page Verification" examples.

📝 Files Modified

  • framework/ (All modules refactored)
  • README.md, CONTRIBUTING.md, ARCHITECTURE.md

[1.2.0] - 2025-12-18

🎉 New Features

API Error Monitoring

  • Network Request/Response Interception: Automatically monitors all HTTP requests and responses
  • Error Detection: Detects and logs 4xx (client errors) and 5xx (server errors) status codes
  • Response Body Capture: Captures JSON/text response bodies for failed API calls
  • Screenshot on API Error: Automatically captures screenshots when API errors occur
  • Smart Filtering: Filters out static assets, CDN, and analytics calls to reduce noise
  • HTML Report Integration: Displays API errors in a dedicated section with detailed table
  • Console Logging: Real-time API error logging with severity levels (WARNING for 4xx, FAILED for 5xx)

Proactive Error Detection

  • Active Error Polling: Checks for errors every 500ms during wait operations
  • Enhanced Error Keywords: Added 12+ session/authentication-specific keywords:
    • Session errors: "session expired", "session timeout", "session invalid"
    • Auth errors: "unauthorized", "access denied", "authentication failed"
    • Permission errors: "forbidden", "not authenticated", "permission denied"
  • Case-Insensitive Detection: Detects errors regardless of capitalization
  • Fast Failure: Tests fail within 1-2 seconds when errors are detected (instead of 30s timeout)
  • Error-Specific Screenshots: Captures screenshots immediately when page errors are detected

🐛 Bug Fixes

Critical Bugs (2)

  1. Missing Type Imports in locator.py - Added Page, Locator, Optional, Dict imports to prevent NameError
  2. Wrong Attribute Reference - Fixed report_dir → output_dir in error watcher screenshot path

High Priority Bugs (4)

  1. Duplicate Reporter Assignment - Removed duplicate self.reporter initialization
  2. Duplicate Context Browser - Removed duplicate self.context_browser initialization
  3. Duplicate Page File Assignment - Removed duplicate self.current_page_file assignment
  4. Duplicate Comment Block - Removed duplicate "Structural Priority" comment in locator.py
  5. Scoring Indentation Bug - Fixed indentation in scorer.py so best_candidate tracking works for all intents

Medium Priority Bugs (3)

  1. Missing Fill Intent Scoring - Added scoring logic for "fill" intent in intelligent fallback
  2. No Error Handling for State Save - Added try-catch with logging for state save operations

Low Priority Bugs (2)

  1. Bare Exception Handlers - Replaced except: with except ValueError: for better debugging
  2. Wait Duration Parsing - Fixed logic bug in wait duration parsing (ms vs seconds)

🔧 Improvements

Code Quality

  • Better Exception Handling: Specific exception types instead of bare except: clauses
  • Enhanced Logging: More descriptive error messages with context
  • Code Cleanup: Removed all duplicate code and comments

Performance

  • Minimal Overhead: API monitoring adds <2ms per request
  • Efficient Polling: 500ms poll interval balances responsiveness and performance

User Experience

  • Clear Error Messages: Specific error keywords returned (e.g., "session expired")
  • Visual Feedback: Screenshots automatically captured on all errors
  • Detailed Reports: HTML reports now include API errors section with full details

📝 Files Modified

  • framework/observer.py - Added API monitoring, enhanced error keywords (+150 lines)
  • framework/runner.py - Integrated API monitoring, proactive error polling (+70 lines)
  • framework/reporter.py - Added API errors section to HTML reports (+55 lines)
  • framework/locator.py - Fixed imports and removed duplicates (+2, -3 lines)
  • framework/scorer.py - Fixed indentation and added fill intent (+4 lines)
  • framework/state.py - Added error handling (+6 lines)
  • .gitignore - Added states/ and healer.log entries

🎯 Breaking Changes

None - All changes are backward compatible. Existing test cases and libraries work without modification.

[1.1.0] - 2025-11-26

Initial Release

Core Features

  • Markdown-based test case and library format
  • Natural Language Processing (NLP) for steps
  • Smart Locators with semantic, structural, and intelligent fallback strategies
  • Self-healing locator mechanism
  • Dynamic data generation with Faker
  • HTML report generation
  • Screenshot capture on failures
  • Context and state management
  • Reusable library system

Browser Support

  • Playwright integration
  • Chromium browser support
  • Headless mode option
  • Geolocation and permissions configuration

Version History Summary

Version Release Date Key Features Status
2.2.1 2025-12-26 Stability Hotfixes (Silent Failures, Leaks) Latest
2.2.0 2025-12-25 Intelligent State & Orbit Performance Stable
2.1.0 2025-12-25 Two-Tier Parallelism & Robust URL Assertions Stable
2.0.1 2024-12-25 Continuum Dashboard & Analytics Stable
2.0.0 2024-12-25 Event-Driven Redesign, SpaCy NLP Legacy

Upgrade Notes

From 1.1.0 to 1.2.0

No action required! All changes are backward compatible.

New Features Automatically Available:

  • API error monitoring is enabled by default
  • Enhanced error detection works automatically
  • Screenshots on API errors happen automatically
  • All existing test cases continue to work

Optional Actions:

  • Review HTML reports for new API Errors section
  • Check console output for detailed API error logs
  • Update test expectations if relying on 30s timeouts (now fails faster on errors)

Known Issues

None at this time.

Roadmap

Planned for 2.3.0+

  • Multi-browser support (Firefox, WebKit)
  • Data-driven testing from CSV/Excel
  • Integration with CI/CD pipelines
  • Visual regression testing (Phase 2)

Completed

  • Parallel test execution (Micro & Macro)
  • API request/response mocking (Network Latching)
  • Performance metrics capture (Dashboard)