diff --git a/common/wuanalysis/GITHUB_COPILOT_PROMPT.md b/common/wuanalysis/GITHUB_COPILOT_PROMPT.md new file mode 100644 index 00000000000..fd0e4cd37de --- /dev/null +++ b/common/wuanalysis/GITHUB_COPILOT_PROMPT.md @@ -0,0 +1,442 @@ +# GitHub Copilot Implementation Prompt: Thor Queue Recommendation Analysis + +## Context + +You are working on the HPCC Platform, a big data processing system. You need to implement a post-execution analysis feature that recommends which Thor queue (cluster configuration) would have been most efficient for a completed workunit. + +## Location +- **Directory**: `/common/wuanalysis/` +- **New files to create**: `queuerecommender.hpp` and `queuerecommender.cpp` +- **Existing files to modify**: `anawu.hpp` and `anawu.cpp` (add new API functions) + +## Background + +### Existing Infrastructure +The directory `common/wuanalysis/` contains: +- `anacommon.hpp/cpp`: Common interfaces (`IWuScope`, `IWuActivity`, `IWuSubGraph`, `PerformanceIssue`) +- `anawu.hpp/cpp`: Main analysis classes (`WorkunitAnalyserBase`, `WorkunitRuleAnalyser`, `WorkunitStatsAnalyser`, `WuScope`) +- `anarule.hpp/cpp`: Rule-based analysis framework + +### Key Existing Classes +- `WorkunitAnalyserBase`: Base class for workunit analysis with methods like `analyse()`, `collateWorkunitStats()` +- `WuScope`: Represents a scope in workunit execution tree, has methods `getStatRaw()`, `queryScopeType()` +- Available helper functions: + - `calcCostNs(double ratePerHour, stat_type ns)` from `jstats.h` - calculates cost from hourly rate and nanoseconds + - `money2cost_type(double money)` - converts dollars to internal cost representation + - `cost_type2money(cost_type cost)` - converts internal cost to dollars + +## Requirements + +Implement a system that: + +1. **Analyzes a completed Thor workunit** to extract resource usage from each subgraph +2. **Estimates execution time and cost** for running the same workunit on different Thor queue configurations +3. **Recommends the optimal queue** based on cost efficiency + +### Thor Queue Configuration +Each queue has: +- Maximum row memory (bytes) +- Maximum temp disk memory (bytes) +- Number of CPUs +- Number of workers (parallelism level) +- VM cost per hour (dollars) + +### Subgraph Statistics to Extract + +For each subgraph, gather these statistics (you'll need to identify the correct `StatisticKind` enum values): +- **SizePeakRowMemory**: Maximum memory for row processing +- **SizePeakTempDisk**: Memory spilled to disk (0 if no spill) +- **SizePeakEphemeralDisk**: Temporary file disk usage +- **TimeUser**: User-space CPU time (nanoseconds) +- **TimeSystem**: Kernel CPU time (nanoseconds) +- **TimeElapsed**: Wall-clock time (nanoseconds) + +**TODO**: Research and identify the actual `StatisticKind` enum values for these statistics. They may be named differently in the codebase. + +### Time Estimation Algorithm + +For each subgraph on each candidate queue: + +``` +ScaledMemory = PeakRowMemory * (CandidateWorkers / ActualWorkers) + +IF ScaledMemory <= QueueMaxMemory AND !OriginallySpilled THEN + // Memory fits and didn't spill - use actual time + EstimatedElapsed = ActualElapsed +ELSE IF OriginallySpilled THEN + // Already spilled - time stays same + EstimatedElapsed = ActualElapsed +ELSE + // Will spill on candidate but didn't originally - apply penalty + EstimatedElapsed = ActualElapsed * SpillPenaltyFactor +END IF + +// Scale based on CPU utilization and workers +CpuTime = TimeUser + TimeSystem +BlockedTime = MAX(0, TimeElapsed - CpuTime) +EstimatedCpuTime = CpuTime * (CandidateWorkers / ActualWorkers) +FinalEstimatedTime = EstimatedCpuTime + BlockedTime +``` + +**Default spill penalty factor**: 10.0 (configurable) + +### Cost Calculation + +``` +ElapsedTimeNs = SUM(all subgraph estimated times) +CostInDollars = calcCostNs(QueueVMCostPerHour, ElapsedTimeNs) * NumWorkers +TotalCost = money2cost_type(CostInDollars) +``` + +## Implementation Tasks + +### 1. Create `queuerecommender.hpp` + +Define these classes with proper HPCC Platform conventions: + +#### QueueConfiguration +```cpp +class WUANALYSIS_API QueueConfiguration : public CInterface +{ + // Stores queue configuration: name, memory limits, workers, cost + // Constructor: QueueConfiguration(name, maxRowMemory, maxTempMemory, numCpus, numWorkers, costPerHour) + // Add getters for all properties + // Add setters for configuration +}; +``` + +#### SubgraphResourceUsage +```cpp +class SubgraphResourceUsage +{ + // Extracts and stores subgraph statistics + // Constructor: SubgraphResourceUsage(WuScope * subgraph, unsigned actualWorkers) + // Method: extractStatistics() - use WuScope::getStatRaw() to extract stats + // Getters: queryPeakRowMemory(), queryPeakTempDisk(), queryTimeElapsed(), etc. + // Calculated: queryCpuTime(), queryBlockedTime(), didSpill() +}; +``` + +#### QueueEstimation +```cpp +class WUANALYSIS_API QueueEstimation : public CInterface +{ + // Stores estimation results for one queue + // Accumulates subgraph estimates + // Calculates total time and cost + // Tracks if workunit will fit (memory constraints) + // Comparison methods for sorting by cost/time +}; +``` + +#### WorkunitQueueAnalyser +```cpp +class WUANALYSIS_API WorkunitQueueAnalyser : public WorkunitAnalyserBase +{ + // Main analysis class + // Methods: + // - addQueueConfiguration(QueueConfiguration * queue) + // - setSpillPenaltyFactor(double factor) + // - analyseQueues(IConstWorkUnit * wu) + // - queryBestQueue() - returns lowest cost queue that fits + // - printReport() - outputs formatted analysis + // - addReportToWorkunit(IWorkUnit * wu) - stores results in WU + + // Protected helper methods: + // - gatherSubgraphUsages() - collect from all subgraphs + // - estimateForQueue(queue, estimation) - run estimation algorithm + // - estimateSubgraphTime(usage, queue) - per-subgraph estimation +}; +``` + +### 2. Implement `queuerecommender.cpp` + +Implement all methods following HPCC Platform coding style: +- Use `LINK()`/`Release()` for reference counting +- Use `StringBuffer` for string building +- Use `CIArrayOf<>` for collections of CInterface objects +- Follow error handling patterns from existing code +- Add comprehensive logging via DBGLOG + +#### Key Implementation Details + +**Finding Subgraphs**: +```cpp +void WorkunitQueueAnalyser::gatherSubgraphUsages() +{ + // Iterate through root->scopes recursively + // Filter for queryScopeType() == SSTsubgraph + // For each subgraph, create SubgraphResourceUsage and extract stats +} +``` + +**Extracting Statistics**: +```cpp +void SubgraphResourceUsage::extractStatistics() +{ + // Use subgraph->getStatRaw(StatisticKind) for each needed stat + // TODO: Identify correct StatisticKind enum values + // Example: peakRowMemory = subgraph->getStatRaw(StSizePeakMemory); +} +``` + +**Time Estimation** (implement algorithm from above): +```cpp +stat_type WorkunitQueueAnalyser::estimateSubgraphTime( + const SubgraphResourceUsage & usage, + const QueueConfiguration & queue, + bool & willSpill) const +{ + // Implement the algorithm described in requirements + // Scale memory by worker ratio + // Determine if spilling will occur + // Apply penalty if newly spilling + // Scale CPU time, add blocked time + // Return estimated elapsed time +} +``` + +**Report Generation**: +```cpp +void WorkunitQueueAnalyser::printReport() const +{ + // Print formatted table with: + // - Queue name, worker count + // - Estimated time and cost + // - Spill indication + // - Recommendation + // Include comparison to actual execution + // Suggest alternatives (faster but more expensive, etc.) +} +``` + +### 3. Add API Functions to `anawu.hpp/cpp` + +```cpp +// In anawu.hpp (add at end with other API functions) +void WUANALYSIS_API analyseQueueRecommendation( + IConstWorkUnit * wu, + IPropertyTree * queueConfig, + IPropertyTree * options); + +// In anawu.cpp +void WUANALYSIS_API analyseQueueRecommendation( + IConstWorkUnit * wu, + IPropertyTree * queueConfig, + IPropertyTree * options) +{ + WorkunitQueueAnalyser analyser; + + // Load queue configurations from queueConfig IPropertyTree + // Expected format: ... + Owned queues = queueConfig->getElements("Queue"); + ForEach(*queues) + { + IPropertyTree & queue = queues->query(); + // Parse configuration and create QueueConfiguration + // Add to analyser + } + + // Get spill penalty from options (default 10.0) + double spillPenalty = options ? options->getPropReal("spillPenaltyFactor", 10.0) : 10.0; + analyser.setSpillPenaltyFactor(spillPenalty); + + // Run analysis + analyser.analyse(wu, nullptr); + analyser.analyseQueues(wu); + + // Output results + analyser.printReport(); +} +``` + +### 4. Configuration Loading + +Implement loading queue configurations from IPropertyTree with this expected XML structure: + +```xml + + + 4294967296 + 10737418240 + 8 + 10 + 2.50 + + + 17179869184 + 53687091200 + 16 + 40 + 10.00 + + +``` + +Handle parsing with proper error checking and defaults. + +### 5. Error Handling + +- Validate queue configurations (positive values, reasonable ranges) +- Handle missing statistics gracefully +- Warn if critical statistics are unavailable +- Don't fail if some subgraphs lack complete stats +- Log warnings for queues that can't handle the workload + +### 6. Output Format + +Generate a report similar to: + +``` +Queue Recommendation Analysis +Workunit: W20231115-120000 +Actual: queue=medium, workers=40, time=1234.56s, cost=$12.34 + +Candidate Queues: +Queue Workers Est.Time(s) Est.Cost($) Spills Status +------------------------------------------------------------ +small 10 2000.00 5.56 Yes Too small +medium 40 1234.56 12.34 No Best match +large 100 800.00 32.00 No Faster/costlier + +Recommendation: medium (current queue) +Cost-effective: Estimated $12.34 vs actual $12.34 (0% difference) + +Alternative: 'large' queue would be 35% faster but cost 159% more +``` + +## Critical TODOs / Information Gaps + +Before full implementation, you MUST research and document: + +1. **Statistic Kind Enums**: Find the actual enum values in the codebase for: + - Peak row memory (search for "Memory", "Peak" in jstats.h or workunit headers) + - Peak temp disk (search for "Disk", "Temp", "Spill") + - Peak ephemeral disk + - System CPU time (search for "System", "Kernel") + - Verify TimeUser and TimeElapsed enum names + +2. **Worker Count**: How to get actual worker count from workunit? + - Search for statistics about cluster size or workers + - May be in workunit attributes rather than statistics + +3. **Statistics Availability**: + - Are these stats at subgraph or activity level? + - How to aggregate if at activity level? + - What to do if statistics are missing? + +4. **Configuration Source**: + - Where should default queue configs come from? + - Hard-code for now or require external config? + - Add placeholder comments for configuration loading + +## Implementation Guidance + +### Phase 1: Skeleton (Do This First) +1. Create header file with all class declarations +2. Add placeholder comments: `// TODO: [Description of what's needed]` +3. Implement constructors and basic getters/setters +4. Get code to compile (empty methods are OK) + +### Phase 2: Statistics (Critical) +1. Research actual StatisticKind enum values +2. Implement SubgraphResourceUsage::extractStatistics() +3. Test with a real workunit to verify stats are extracted +4. Document which statistics are available and which are not + +### Phase 3: Algorithm +1. Implement estimateSubgraphTime() with full algorithm +2. Add unit tests for time estimation +3. Implement estimateForQueue() +4. Test with various scenarios (spill/no-spill, different worker counts) + +### Phase 4: Integration +1. Implement gatherSubgraphUsages() +2. Implement analyseQueues() +3. Add API function +4. Test end-to-end with real workunit + +### Phase 5: Reporting +1. Implement printReport() +2. Add formatting and tables +3. Include comparisons and recommendations +4. Test output readability + +## Code Style Requirements + +Follow HPCC Platform conventions: +- Use `WUANALYSIS_API` macro for exported functions +- Use `CInterface` base class with reference counting +- Use `Linked` and `Owned` for automatic reference management +- Use `CIArrayOf` for collections +- Use `StringBuffer` for string building +- Use `ForEach()` macros for iteration +- Add DBGLOG for debugging output +- Use `assertex()` for invariant checks +- Follow existing naming conventions (camelCase for methods, member variables) + +## Testing Approach + +Create test cases in `testing/wuanalysis/` directory: +1. Test with synthetic workunit data +2. Test estimation accuracy vs. real executions +3. Test edge cases (zero workers, no spills, all spills, missing stats) +4. Validate cost calculations +5. Test configuration parsing + +## Example Usage + +After implementation, usage would look like: + +```cpp +// In wutool or other analysis tool +Owned wu = getWorkunit("W20231115-120000"); +Owned queueConfig = createPTreeFromXMLFile("queues.xml"); +Owned options = createPTree(); +options->setPropReal("spillPenaltyFactor", 10.0); + +analyseQueueRecommendation(wu, queueConfig, options); +``` + +## Documentation Requirements + +Add comprehensive documentation: +1. Class and method doc comments (doxygen style) +2. Algorithm explanation in code comments +3. TODO comments for information gaps +4. Example usage in header file comments +5. README.md in common/wuanalysis explaining the feature + +## Success Criteria + +Implementation is complete when: +- [ ] Code compiles without errors +- [ ] All classes implemented with documented methods +- [ ] Can extract statistics from a real workunit +- [ ] Time estimation algorithm works correctly +- [ ] Cost calculation produces reasonable results +- [ ] Report generation produces readable output +- [ ] TODOs are documented for missing information +- [ ] Code follows HPCC Platform style guidelines +- [ ] Basic testing passes + +## Notes for Implementer + +- **Start simple**: Get basic structure working first, add sophistication later +- **Document unknowns**: Use TODO comments extensively where information is missing +- **Test incrementally**: Test each component as you build it +- **Ask for clarification**: If StatisticKind enums can't be found, document what you tried +- **Placeholder values**: Use reasonable defaults where configuration is unclear +- **Error handling**: Fail gracefully when statistics are missing + +This is a complex feature. Focus on getting a working skeleton with clear TODOs, then progressively fill in the implementation as information becomes available. + +## References + +Study these existing files for patterns: +- `common/wuanalysis/anawu.cpp`: WorkunitRuleAnalyser, WorkunitStatsAnalyser +- `common/wuanalysis/anarule.cpp`: PerformanceIssue, rule checking patterns +- `system/jlib/jstats.h`: Statistics types and helper functions +- `common/workunit/workunit.hpp`: Workunit interfaces + +Good luck with the implementation! diff --git a/common/wuanalysis/IMPLEMENTATION_SUMMARY.md b/common/wuanalysis/IMPLEMENTATION_SUMMARY.md new file mode 100644 index 00000000000..83d4bdd90af --- /dev/null +++ b/common/wuanalysis/IMPLEMENTATION_SUMMARY.md @@ -0,0 +1,232 @@ +# Thor Queue Recommendation - Implementation Summary + +## What Was Delivered + +This deliverable provides comprehensive planning documentation for implementing a Thor queue recommendation feature in the HPCC Platform. As requested in the problem statement: **"Do not implement the code. Instead create a detailed plan of how you would implement this functionality, and generate a prompt as the output that would be suitable for supplying to the github copilot to implement the functionality."** + +## Delivered Artifacts + +### 1. QUEUE_RECOMMENDATION_PLAN.md +**Comprehensive implementation plan including:** + +- **Architecture Overview**: How the feature fits into existing `common/wuanalysis/` infrastructure +- **Detailed Design**: Complete class specifications for all components + - `QueueConfiguration` - Stores queue resource limits and costs + - `SubgraphResourceUsage` - Extracts and represents actual resource usage + - `QueueEstimation` - Holds estimated performance for a candidate queue + - `WorkunitQueueAnalyser` - Main analysis orchestrator + +- **Algorithm Details**: Step-by-step time estimation and cost calculation algorithms with pseudocode +- **Configuration Loading**: Proposed XML format for queue definitions +- **API Design**: New public functions to integrate with existing analysis framework +- **Output Format**: Detailed report format with example output +- **Implementation Phases**: Breaking down the work into manageable chunks +- **Testing Strategy**: Unit, integration, and validation testing approaches +- **Open Questions**: 7 critical information gaps that need resolution before implementation + +### 2. GITHUB_COPILOT_PROMPT.md +**Comprehensive prompt for GitHub Copilot including:** + +- **Context Setting**: Background on HPCC Platform and existing codebase +- **Detailed Requirements**: Precise specification of what to build +- **Implementation Tasks**: 6 major task areas with specific guidance for each +- **Code Templates**: Class skeletons with method signatures and usage patterns +- **Algorithm Pseudocode**: Step-by-step time estimation logic +- **Critical TODOs**: Clear placeholders for missing information (StatisticKind enums, worker counts, etc.) +- **Code Style Requirements**: HPCC Platform conventions to follow +- **Testing Approach**: How to validate the implementation +- **Success Criteria**: Checklist of completion requirements +- **Example Usage**: How the feature will be used once implemented +- **Implementation Phases**: Suggested build order from skeleton to complete feature + +### 3. Key Features of the Design + +#### Smart Resource Scaling +The algorithm accounts for different worker counts across queues: +- Memory usage scales proportionally with workers +- CPU time scales inversely with workers +- Blocked time (I/O, network) remains constant +- Spill behavior changes based on available memory + +#### Cost-Aware Recommendations +- Uses existing `calcCostNs()` cost calculation functions +- Factors in worker count and VM costs per hour +- Provides cost vs. performance trade-off analysis +- Identifies "best value" vs "fastest" options + +#### Robust Handling of Unknown Information +The design explicitly identifies 7 critical information gaps: +1. Exact StatisticKind enum values for Thor statistics +2. Queue configuration source and format +3. How to determine actual worker count +4. Memory unit conventions +5. Spill detection mechanisms +6. Integration points and triggers +7. Result storage location + +Each gap is documented with: +- What information is needed +- Where to look for it +- Placeholder approaches to use until resolved + +#### Integration with Existing Infrastructure +- Extends `WorkunitAnalyserBase` class +- Uses existing `WuScope` for statistics extraction +- Follows patterns from `WorkunitRuleAnalyser` and `WorkunitStatsAnalyser` +- Integrates with existing API functions in `anawu.hpp/cpp` +- Compatible with wutool command-line interface + +## How to Use These Documents + +### For Developers Implementing the Feature + +1. **Read QUEUE_RECOMMENDATION_PLAN.md first** to understand the complete design +2. **Review the open questions section** and resolve information gaps before coding +3. **Use GITHUB_COPILOT_PROMPT.md** as input to GitHub Copilot or as implementation guide +4. **Follow the phased approach** - build skeleton first, then fill in details +5. **Document additional TODOs** discovered during implementation + +### For Code Reviewers + +1. Check implementation against the design specifications +2. Verify all critical TODOs are addressed or documented +3. Ensure algorithm matches the pseudocode +4. Validate cost calculations use correct formulas +5. Check adherence to HPCC Platform coding conventions + +### For Project Managers + +1. Use implementation phases for sprint planning +2. Reference open questions for dependency tracking +3. Use success criteria for acceptance testing +4. Plan for configuration management (queue definitions) +5. Consider user documentation needs + +## Next Steps + +### Before Implementation Begins + +1. **Resolve Information Gaps**: Research and document answers to the 7 open questions + - Most critical: Identify correct StatisticKind enum values + - Determine queue configuration source + - Find how to get actual worker count + +2. **Set Up Configuration**: Decide on queue configuration approach + - Configuration file format and location + - Default queue definitions for testing + - Configuration validation requirements + +3. **Plan Testing**: Identify test workunits + - Need workunits with known execution characteristics + - Various resource usage patterns (memory-heavy, CPU-heavy, I/O-bound) + - Both with and without spilling + +### During Implementation + +1. **Phase 1 (Week 1)**: Build skeleton with all classes and placeholder TODOs +2. **Phase 2 (Week 1-2)**: Research and implement statistics extraction +3. **Phase 3 (Week 2-3)**: Implement core estimation algorithm +4. **Phase 4 (Week 3)**: Add configuration loading and API integration +5. **Phase 5 (Week 4)**: Implement reporting and polish + +### After Implementation + +1. **Validation**: Test estimates against real workunit executions +2. **Calibration**: Tune spill penalty factor based on real data +3. **Documentation**: Create user guide and API documentation +4. **Integration**: Add to workunit completion workflow (optional) +5. **UI Enhancement**: Consider ECLWatch integration (future work) + +## Design Decisions and Rationale + +### Why Subgraph-Level Analysis? +- Problem statement specifies subgraphs as the unit of analysis +- Subgraphs capture meaningful resource usage patterns +- Aggregation to job level is straightforward +- Aligns with Thor's execution model + +### Why Configurable Spill Penalty? +- Actual penalty varies by workload type +- Default of 10x is conservative but reasonable +- Allows tuning based on empirical data +- Documented as configuration option + +### Why Cost-Based Recommendation? +- Primary goal is efficiency (cost-effectiveness) +- Time vs. cost trade-off is valuable information +- Aligns with cloud economics +- Supports budget-conscious decision making + +### Why Separate Classes for Each Concept? +- Clear separation of concerns +- Easier to test individual components +- Follows existing patterns in wuanalysis +- Supports future enhancements + +## Potential Challenges and Mitigations + +### Challenge 1: Missing Statistics +**Risk**: Required statistics may not be recorded by Thor +**Mitigation**: Design includes graceful degradation; document what's available + +### Challenge 2: Estimation Accuracy +**Risk**: Simple linear scaling may not match reality +**Mitigation**: Make algorithm tunable; validate against real data; iterate + +### Challenge 3: Configuration Management +**Risk**: Queue definitions may be hard to maintain +**Mitigation**: Flexible configuration format; validation; defaults + +### Challenge 4: Integration Complexity +**Risk**: Integration with existing analysis may be challenging +**Mitigation**: Design extends existing classes; follows established patterns + +## Success Metrics + +The implementation will be successful if: + +1. **Functional**: Can analyze any completed Thor workunit +2. **Accurate**: Estimates within 20% of actual for typical jobs +3. **Useful**: Identifies genuinely better queue choices when they exist +4. **Performant**: Analysis completes in < 5 seconds for typical workunits +5. **Maintainable**: Code is clear, well-documented, and testable +6. **Integrated**: Works seamlessly with existing wuanalysis tools + +## References for Implementation + +### Existing Code to Study +- `common/wuanalysis/anawu.cpp` - Pattern for analyzer classes +- `common/wuanalysis/anarule.cpp` - Pattern for issue detection and reporting +- `system/jlib/jstats.h` - Statistics types and cost calculation +- `common/workunit/workunit.cpp` - Workunit interface and Thor cost calculation + +### Key Interfaces +- `IConstWorkUnit` - Accessing workunit data +- `IPropertyTree` - Configuration and result storage +- `WuScope` - Extracting statistics via `getStatRaw()` +- `IStatisticGatherer` - Collecting statistics + +### Helpful Functions +- `calcCostNs(ratePerHour, nanoseconds)` - Cost calculation +- `money2cost_type()` / `cost_type2money()` - Currency conversion +- `formatStatistic()` - Formatting statistics for display + +## Conclusion + +This planning package provides everything needed to implement the Thor queue recommendation feature: + +✅ Complete architectural design +✅ Detailed class specifications +✅ Step-by-step algorithms +✅ Integration strategy +✅ GitHub Copilot prompt +✅ Testing approach +✅ Clear identification of unknowns + +The design is practical, follows HPCC Platform conventions, and can be implemented incrementally. With the information gaps resolved, implementation can proceed with confidence. + +--- + +**Document Version**: 1.0 +**Date**: 2024 +**Status**: Ready for Implementation (pending information gap resolution) diff --git a/common/wuanalysis/QUEUE_RECOMMENDATION_PLAN.md b/common/wuanalysis/QUEUE_RECOMMENDATION_PLAN.md new file mode 100644 index 00000000000..a328a2c2bee --- /dev/null +++ b/common/wuanalysis/QUEUE_RECOMMENDATION_PLAN.md @@ -0,0 +1,534 @@ +# Thor Queue Recommendation Analysis - Implementation Plan + +## Overview +This document provides a detailed implementation plan for adding queue recommendation analysis functionality to the HPCC Platform's workunit analysis system. The goal is to analyze completed workunits and determine which Thor queue configuration would have been most efficient for execution. + +## Problem Statement + +When a workunit executes in Thor, the costs of running the job are closely linked to the resources that the underlying processes have available. The system allows multiple different Thor configurations (or queues), and ideally a job should target the queue that uses the minimum resources that can efficiently process a job. + +### Queue Configuration Constraints + +Each Thor queue has the following constraints: +- **Maximum memory available for rows** - memory limit for processing data +- **Maximum memory available for temp disk** - memory before spilling to disk +- **Number of CPUs** - processing power available +- **Number of workers** - parallelism level + +### Subgraph Statistics Available + +Each subgraph in a Thor job records the following statistics: +- **SizePeakRowMemory** - Maximum memory required to run the subgraph +- **SizePeakTempDisk** - Amount of data spilled to disk (0 if no spilling occurred) +- **SizePeakEphemeralDisk** - Maximum local disk space consumed by temporary files +- **TimeUser** - Amount of userspace CPU time consumed +- **TimeSystem** - Amount of kernel/system CPU time consumed +- **TimeElapsed** - Total wall-clock time for the subgraph + +### Key Calculations + +Average CPU utilization: `(TimeSystem + TimeUser) / TimeElapsed` + +## Architecture Overview + +### Location in Codebase +- **Directory**: `common/wuanalysis/` +- **New Files to Create**: + - `queuerecommender.hpp` - Header file with class definitions and interfaces + - `queuerecommender.cpp` - Implementation of queue recommendation logic + - Unit tests (if test infrastructure exists) + +### Integration Points +- Extend `WorkunitAnalyserBase` or create a new analyzer class +- Integrate with existing analysis functions in `anawu.hpp/cpp` +- Add new API function: `analyseQueueRecommendation()` + +## Detailed Design + +### 1. Data Structures + +#### QueueConfiguration Class +Represents a single Thor queue configuration with its resource constraints. + +```cpp +class QueueConfiguration +{ +public: + QueueConfiguration(const char * name, + stat_type maxRowMemory, + stat_type maxTempDiskMemory, + unsigned numCpus, + unsigned numWorkers, + double vmCostPerHour); + + // Getters + const char * queryName() const; + stat_type queryMaxRowMemory() const; + stat_type queryMaxTempDiskMemory() const; + unsigned queryNumCpus() const; + unsigned queryNumWorkers() const; + double queryCostPerHour() const; + + // Setters (for configuration loading) + void setMaxRowMemory(stat_type memory); + void setMaxTempDiskMemory(stat_type memory); + void setNumCpus(unsigned cpus); + void setNumWorkers(unsigned workers); + void setVMCostPerHour(double cost); + +private: + StringAttr name; + stat_type maxRowMemory; + stat_type maxTempDiskMemory; + unsigned numCpus; + unsigned numWorkers; + double vmCostPerHour; // Cost per hour for this queue configuration +}; +``` + +**Implementation Notes**: +- Store memory values in bytes (stat_type) +- Cost should be in dollars per hour (double) +- Name should identify the queue uniquely + +#### SubgraphResourceUsage Class +Captures the resource usage statistics from a completed subgraph execution. + +```cpp +class SubgraphResourceUsage +{ +public: + SubgraphResourceUsage(WuScope * subgraph, unsigned actualWorkers); + + // Extract statistics from the WuScope + void extractStatistics(); + + // Getters for actual resource usage + stat_type queryPeakRowMemory() const; + stat_type queryPeakTempDisk() const; + stat_type queryPeakEphemeralDisk() const; + stat_type queryTimeUser() const; + stat_type queryTimeSystem() const; + stat_type queryTimeElapsed() const; + bool didSpill() const; + + // Calculated values + double queryCpuUtilization() const; + stat_type queryCpuTime() const; // TimeUser + TimeSystem + stat_type queryBlockedTime() const; // TimeElapsed - CpuTime (if positive, else 0) + +private: + Linked subgraph; + unsigned actualWorkers; + + // Extracted statistics (TODO: Determine the exact StatisticKind enums to use) + // These need to be mapped to the actual statistics available in Thor + stat_type peakRowMemory; // StatisticKind: StSizePeakRowMemory (?) + stat_type peakTempDisk; // StatisticKind: StSizePeakTempDisk (?) + stat_type peakEphemeralDisk; // StatisticKind: StSizePeakEphemeralDisk (?) + stat_type timeUser; // StatisticKind: StTimeUser + stat_type timeSystem; // StatisticKind: StTimeSystem (?) + stat_type timeElapsed; // StatisticKind: StTimeElapsed +}; +``` + +**TODO - Information Needed**: +1. What are the exact `StatisticKind` enum values for these statistics? + - `StSizePeakRowMemory` - does this exist or use different name? + - `StSizePeakTempDisk` - does this exist or use different name? + - `StSizePeakEphemeralDisk` - does this exist or use different name? + - `StTimeSystem` - does this exist or use different name? +2. Are these statistics recorded at the subgraph level or activity level? +3. How to aggregate if statistics are at activity level? + +#### QueueEstimation Class +Represents the estimated performance and cost for running on a specific queue. + +```cpp +class QueueEstimation +{ +public: + QueueEstimation(const QueueConfiguration * queue); + + // Add a subgraph estimation + void addSubgraphEstimate(const SubgraphResourceUsage & usage, + stat_type estimatedElapsed, + bool willSpill); + + // Calculate final metrics + void finalize(); + + // Getters + const char * queryQueueName() const; + stat_type queryEstimatedElapsedTime() const; + cost_type queryEstimatedCost() const; + bool queryWillFit() const; // Can this queue handle the job? + unsigned queryNumSpills() const; // How many subgraphs will spill? + + // Comparison for sorting + int compareCost(const QueueEstimation & other) const; + int compareTime(const QueueEstimation & other) const; + +private: + Linked queue; + stat_type totalEstimatedElapsed; + cost_type estimatedCost; + bool willFit; + unsigned numSpills; + std::vector subgraphEstimates; +}; +``` + +**Implementation Notes**: +- Use `calcCost()` from `jstats.h` for cost calculation +- Formula: `cost = calcCostNs(vmCostPerHour, elapsedTimeNs) * numWorkers` +- Mark as "will not fit" if peak memory exceeds queue limits + +### 2. Core Algorithm Class + +#### WorkunitQueueAnalyser Class +Main class that performs the queue recommendation analysis. + +```cpp +class WorkunitQueueAnalyser : public WorkunitAnalyserBase +{ +public: + WorkunitQueueAnalyser(); + + // Configuration + void addQueueConfiguration(QueueConfiguration * queue); + void setSpillPenaltyFactor(double factor); // Default: 10.0 + void setActualWorkerCount(unsigned workers); + + // Analysis execution + void analyseQueues(IConstWorkUnit * wu); + + // Results + const QueueEstimation * queryBestQueue() const; // Lowest cost that fits + void getRecommendations(CIArrayOf & results) const; + void printReport() const; + void addReportToWorkunit(IWorkUnit * wu) const; + +protected: + // Internal methods + void gatherSubgraphUsages(); + void estimateForQueue(const QueueConfiguration & queue, QueueEstimation & estimation); + stat_type estimateSubgraphTime(const SubgraphResourceUsage & usage, + const QueueConfiguration & queue, + bool & willSpill) const; + +private: + CIArrayOf queueConfigurations; + CIArrayOf subgraphUsages; + CIArrayOf estimations; + double spillPenaltyFactor; + unsigned actualWorkers; +}; +``` + +### 3. Estimation Algorithm Details + +#### Time Estimation for Each Subgraph + +For each subgraph, estimate the execution time on a candidate queue: + +```cpp +stat_type estimateSubgraphTime( + const SubgraphResourceUsage & usage, + const QueueConfiguration & queue, + bool & willSpill) const +{ + unsigned actualWorkers = usage.queryActualWorkers(); + unsigned candidateWorkers = queue.queryNumWorkers(); + + // Scale memory usage based on worker count + stat_type scaledPeakMemory = usage.queryPeakRowMemory() * candidateWorkers / actualWorkers; + + // Determine if this subgraph will spill on the candidate queue + bool originallySpilled = usage.didSpill(); + bool willSpillOnCandidate = (scaledPeakMemory > queue.queryMaxRowMemory()); + + stat_type baseElapsed = usage.queryTimeElapsed(); + stat_type estimatedElapsed; + + if (scaledPeakMemory <= queue.queryMaxRowMemory() && !originallySpilled) { + // Memory fits and didn't spill originally - time stays the same + estimatedElapsed = baseElapsed; + } + else if (originallySpilled) { + // Already spilled in original run - time stays the same + estimatedElapsed = baseElapsed; + } + else { + // Will spill on candidate queue but didn't originally + // Apply penalty factor + estimatedElapsed = baseElapsed * spillPenaltyFactor; + willSpillOnCandidate = true; + } + + // Scale elapsed time based on CPU utilization and worker count + stat_type cpuTime = usage.queryCpuTime(); + stat_type blockedTime = usage.queryBlockedTime(); + + // EstimatedCpuTime = cpuTime * candidateWorkers / actualWorkers + stat_type estimatedCpuTime = cpuTime * candidateWorkers / actualWorkers; + + // Final estimated time = EstimatedCpuTime + BlockedTime + stat_type finalEstimated = estimatedCpuTime + blockedTime; + + // If we had to apply spill penalty, use the larger of scaled time or penalty time + if (willSpillOnCandidate && !originallySpilled) { + finalEstimated = std::max(finalEstimated, estimatedElapsed); + } + else { + finalEstimated = estimatedElapsed; // Use the CPU-scaled time + } + + willSpill = willSpillOnCandidate; + return finalEstimated; +} +``` + +**Algorithm Summary**: +1. Scale memory usage by worker ratio +2. Check if subgraph will spill +3. Apply spill penalty if newly spilling +4. Scale CPU time by worker ratio +5. Add blocked time (unchanged) +6. Return maximum of scaled times + +#### Cost Calculation + +```cpp +cost_type calculateQueueCost(const QueueConfiguration & queue, stat_type elapsedTimeNs) +{ + // Use existing calcCostNs function from jstats.h + double costInDollars = calcCostNs(queue.queryCostPerHour(), elapsedTimeNs) * queue.queryNumWorkers(); + return money2cost_type(costInDollars); +} +``` + +### 4. Configuration Loading + +#### QueueConfiguration Source + +**TODO - Information Needed**: +Where should queue configurations come from? +- Option 1: Configuration file (e.g., `queues.yaml` or `queues.xml`) +- Option 2: Dali/Environment configuration +- Option 3: Passed as parameter to analysis function +- Option 4: Hard-coded defaults for testing + +Suggested structure for configuration file: +```xml + + + 4GB + 10GB + 8 + 10 + 2.50 + + + 16GB + 50GB + 16 + 40 + 10.00 + + + 64GB + 200GB + 32 + 100 + 40.00 + + +``` + +### 5. API Functions + +Add to `anawu.hpp`: + +```cpp +// Analyze queue recommendations for a completed workunit +void WUANALYSIS_API analyseQueueRecommendation( + IConstWorkUnit * wu, + IPropertyTree * queueConfig, + IPropertyTree * options); + +// Get queue recommendations and return structured results +void WUANALYSIS_API getQueueRecommendations( + QueueRecommendationResults & results, + IConstWorkUnit * wu, + IPropertyTree * queueConfig, + IPropertyTree * options); +``` + +### 6. Output Report Format + +The analysis should produce a report showing: + +``` +Queue Recommendation Analysis for Workunit: W20231115-120000 +Actual Execution: queue=medium, workers=40, elapsed=1234.56s, cost=$12.34 + +Estimated Performance on Available Queues: ++---------+---------+---------------+----------+---------+----------+ +| Queue | Workers | Estimated | Est. | Will | Recommend| +| | | Time (s) | Cost ($) | Spill | | ++---------+---------+---------------+----------+---------+----------+ +| small | 10 | 2000.00 | 5.56 | Yes | | +| medium | 40 | 1234.56 | 12.34 | No | Best Cost| +| large | 100 | 800.00 | 32.00 | No | | ++---------+---------+---------------+----------+---------+----------+ + +Recommendation: Run on 'medium' queue (current queue) +- Estimated cost: $12.34 (actual: $12.34) +- Estimated time: 1234.56s (actual: 1234.56s) +- No spilling expected + +Alternative: 'large' queue would complete 35% faster but cost 159% more + +Subgraph Analysis: + sg1: 456.78s on medium (40 workers) -> 182.71s on large (100 workers) + sg2: 777.78s on medium (40 workers) -> 617.29s on large (100 workers) + ... +``` + +### 7. Integration with Existing Analysis + +The queue recommendation analysis should be callable: +1. Standalone via wutool command +2. Automatically after workunit completion (optional) +3. Via ECLWatch UI (future enhancement) + +Example wutool usage: +```bash +wutool queuerecommend W20231115-120000 daliserver=192.168.1.100 queueconfig=/etc/HPCCSystems/queues.xml +``` + +## Implementation Steps + +### Phase 1: Core Data Structures +1. Create `queuerecommender.hpp` with all class definitions +2. Implement `QueueConfiguration` class +3. Implement `SubgraphResourceUsage` class +4. Add comprehensive documentation comments + +### Phase 2: Algorithm Implementation +1. Implement `WorkunitQueueAnalyser` constructor and configuration methods +2. Implement `gatherSubgraphUsages()` method +3. Implement `estimateSubgraphTime()` algorithm +4. Implement `estimateForQueue()` method +5. Add unit tests for time estimation logic + +### Phase 3: Configuration and Integration +1. Implement configuration file loading +2. Add API functions to `anawu.hpp/cpp` +3. Integrate with existing workunit analysis infrastructure +4. Add command-line support via wutool + +### Phase 4: Reporting and Output +1. Implement `printReport()` method +2. Implement `addReportToWorkunit()` method +3. Add structured output support (JSON/XML) +4. Document usage examples + +### Phase 5: Testing and Validation +1. Create test cases with known workunit statistics +2. Validate estimation accuracy against real workunit data +3. Test with various queue configurations +4. Performance testing for large workunits + +## Open Questions / Information Needed + +### Critical Information Gaps + +1. **Statistics Naming**: + - What are the exact `StatisticKind` enum values for: + - Peak row memory usage + - Peak temp disk usage + - Peak ephemeral disk usage + - System/kernel CPU time + - Are these recorded at subgraph or activity level? + +2. **Queue Configuration Source**: + - Where should queue configurations be stored? + - How should they be accessed at runtime? + - Who maintains the configuration? + +3. **Worker Count Information**: + - How to determine the actual number of workers used? + - Is this stored in workunit statistics? + - StatisticKind enum value? + +4. **Cost Calculation**: + - Should we use existing `calculateThorCost()` function? + - Does it need modification for this use case? + - Are there separate manager/worker costs to consider? + +5. **Memory Units**: + - Are memory statistics stored in bytes? + - Any conversion needed? + +6. **Spill Detection**: + - How to detect if spilling occurred? + - Is `SizePeakTempDisk > 0` sufficient? + - Are there other indicators? + +7. **Integration Points**: + - Should this run automatically after workunit completion? + - Should it be opt-in via configuration? + - Where should results be stored (workunit attributes, separate DB, etc.)? + +## Testing Strategy + +### Unit Tests +- Test time estimation algorithm with various scenarios +- Test memory scaling logic +- Test spill detection +- Test cost calculation + +### Integration Tests +- Test with real workunit data +- Validate against known execution times +- Test configuration loading +- Test report generation + +### Validation Tests +- Compare estimates vs actual for various workunits +- Measure estimation accuracy +- Identify edge cases where estimation fails + +## Documentation Requirements + +1. **Code Documentation**: + - Comprehensive class and method documentation + - Algorithm explanation comments + - Example usage in code comments + +2. **User Documentation**: + - How to configure queue definitions + - How to run queue recommendation analysis + - How to interpret results + - Wutool command reference + +3. **Developer Documentation**: + - Architecture overview + - Extension points for future enhancements + - Testing guidelines + +## Future Enhancements (Out of Scope) + +1. Machine learning model for better time prediction +2. Historical analysis to improve estimates +3. Real-time queue recommendation during submission +4. Automatic queue selection based on ECL code analysis +5. Integration with workunit scheduler for automatic routing +6. Cost optimization suggestions +7. Multi-dimensional optimization (cost vs time vs resource usage) + +## Summary + +This plan provides a comprehensive approach to implementing queue recommendation analysis in the HPCC Platform. The implementation should be modular, well-documented, and integrate cleanly with existing workunit analysis infrastructure. The main challenges are identifying the correct statistics to use and determining the source of queue configuration data. diff --git a/common/wuanalysis/README_QUEUE_RECOMMENDATION.md b/common/wuanalysis/README_QUEUE_RECOMMENDATION.md new file mode 100644 index 00000000000..0955afac7fb --- /dev/null +++ b/common/wuanalysis/README_QUEUE_RECOMMENDATION.md @@ -0,0 +1,298 @@ +# Thor Queue Recommendation Feature - Documentation Index + +## Overview + +This directory contains comprehensive planning documentation for implementing a Thor queue recommendation feature in the HPCC Platform. The feature will analyze completed workunits to determine which Thor queue configuration would have been most cost-effective. + +## Purpose + +As stated in the requirements: **"Do not implement the code. Instead create a detailed plan of how you would implement this functionality, and generate a prompt as the output that would be suitable for supplying to the github copilot to implement the functionality."** + +This documentation fulfills that requirement by providing: +- Complete architectural design +- Detailed implementation plan +- GitHub Copilot-ready prompt +- Clear identification of information gaps + +## Document Index + +### 📋 [IMPLEMENTATION_SUMMARY.md](./IMPLEMENTATION_SUMMARY.md) +**Start here** - Executive summary and quick reference + +**Contents:** +- What was delivered +- How to use these documents +- Next steps and timeline +- Design decisions and rationale +- Success metrics + +**Audience:** Project managers, technical leads, code reviewers + +--- + +### 📐 [QUEUE_RECOMMENDATION_PLAN.md](./QUEUE_RECOMMENDATION_PLAN.md) +**Complete technical design** - Deep dive into architecture and algorithms + +**Contents:** +- Architecture overview and integration points +- Detailed class specifications with method signatures +- Time estimation algorithm (pseudocode and explanation) +- Cost calculation formulas +- Configuration format (XML examples) +- API design +- Testing strategy +- Implementation phases +- 7 documented information gaps with resolution guidance + +**Audience:** Software engineers, architects, implementers + +**Key Sections:** +- Section 1: Data Structures (4 classes) +- Section 2: Core Algorithm Class (WorkunitQueueAnalyser) +- Section 3: Estimation Algorithm Details +- Section 4: Configuration Loading +- Section 5: API Functions +- Section 6: Output Report Format +- Section 7: Integration + +--- + +### 🤖 [GITHUB_COPILOT_PROMPT.md](./GITHUB_COPILOT_PROMPT.md) +**Ready-to-use implementation guide** - Paste this into GitHub Copilot + +**Contents:** +- Context setting and background +- Detailed requirements +- Implementation tasks (6 phases) +- Code templates and class skeletons +- Algorithm implementation guidance +- Critical TODOs with placeholders +- Code style requirements +- Testing approach +- Success criteria checklist + +**Audience:** Developers implementing the feature, GitHub Copilot + +**How to Use:** +1. Resolve critical TODOs (StatisticKind enums, worker counts) +2. Copy relevant sections to GitHub Copilot +3. Follow the phased implementation approach +4. Use code templates as starting point +5. Refer back to QUEUE_RECOMMENDATION_PLAN.md for details + +--- + +## Quick Start Guide + +### For Implementers + +1. **Read in order:** + - IMPLEMENTATION_SUMMARY.md (15 minutes) + - QUEUE_RECOMMENDATION_PLAN.md (45 minutes) + - GITHUB_COPILOT_PROMPT.md (30 minutes) + +2. **Before coding:** + - Resolve the 7 critical information gaps + - Set up test environment with sample workunits + - Create skeleton branch + +3. **During implementation:** + - Follow phased approach from GITHUB_COPILOT_PROMPT.md + - Use code templates as starting points + - Document additional TODOs discovered + - Test incrementally after each phase + +### For Reviewers + +1. **Design Review:** + - Review QUEUE_RECOMMENDATION_PLAN.md architecture section + - Validate algorithm correctness + - Check integration points + +2. **Code Review:** + - Compare implementation against class specifications + - Verify algorithm matches pseudocode + - Check TODO resolution + - Validate test coverage + +### For Project Managers + +1. **Planning:** + - Use implementation phases for sprint planning + - Track information gap resolution + - Plan configuration management approach + +2. **Tracking:** + - Use success criteria from IMPLEMENTATION_SUMMARY.md + - Monitor validation against real workunits + - Track estimation accuracy improvements + +## Feature Summary + +### What It Does + +Analyzes a completed Thor workunit and: +1. Extracts resource usage from each subgraph (memory, CPU, disk) +2. Estimates execution time and cost on different queue configurations +3. Recommends the most cost-effective queue +4. Provides trade-off analysis (faster vs. cheaper options) + +### How It Works + +``` +Workunit (completed) + ↓ +Extract subgraph statistics + ↓ +For each queue configuration: + - Scale memory by worker ratio + - Detect if spilling will occur + - Estimate execution time + - Calculate cost + ↓ +Sort by cost (ascending) + ↓ +Recommend best fit +``` + +### Key Algorithm + +For each subgraph on each candidate queue: + +1. **Scale memory:** `ScaledMemory = ActualMemory × (CandidateWorkers / ActualWorkers)` +2. **Detect spilling:** `WillSpill = (ScaledMemory > QueueMaxMemory)` +3. **Apply penalty if newly spilling:** `Time = Time × 10` (configurable) +4. **Scale CPU time:** `CpuTime = CpuTime × (CandidateWorkers / ActualWorkers)` +5. **Calculate cost:** `Cost = calcCostNs(hourlyRate, time) × workers` + +### Example Output + +``` +Queue Recommendation Analysis +Workunit: W20231115-120000 + +Candidate Queues: +Queue Workers Est.Time(s) Est.Cost($) Spills Status +------------------------------------------------------------ +small 10 2000.00 5.56 Yes Too small +medium 40 1234.56 12.34 No ✓ Best match +large 100 800.00 32.00 No Faster/costlier + +Recommendation: medium queue +- Most cost-effective option +- Estimated: $12.34 vs actual: $12.34 +- Alternative: 'large' is 35% faster but costs 159% more +``` + +## Critical Information Gaps + +Before implementation, these must be resolved: + +| # | Gap | Impact | Resolution Strategy | +|---|-----|--------|---------------------| +| 1 | StatisticKind enum values | **HIGH** | Search jstats.h, workunit headers | +| 2 | Queue configuration source | **HIGH** | Decide: file, Dali, or hardcoded | +| 3 | Actual worker count | **HIGH** | Find in workunit statistics | +| 4 | Memory units | Medium | Verify bytes vs other units | +| 5 | Spill detection | Medium | Confirm `PeakTempDisk > 0` | +| 6 | Integration points | Medium | Decide: automatic vs. on-demand | +| 7 | Result storage | Low | Document in workunit or separate | + +Each gap is detailed in QUEUE_RECOMMENDATION_PLAN.md with guidance on resolution. + +## Dependencies + +### Code Dependencies +- `WorkunitAnalyserBase` (existing) - Base class for analyzers +- `WuScope` (existing) - Access to statistics +- `calcCostNs()` (existing) - Cost calculation +- `IPropertyTree` (existing) - Configuration and results + +### Information Dependencies +- Thor statistics recording (must include memory, CPU, disk stats) +- Queue configuration data (must be available somewhere) +- Worker count information (must be recorded in workunit) + +## Testing Strategy + +### Unit Tests +- Time estimation with various scenarios +- Memory scaling calculations +- Cost calculations +- Spill detection logic + +### Integration Tests +- Statistics extraction from real workunits +- Configuration loading +- End-to-end analysis +- Report generation + +### Validation Tests +- Estimate accuracy vs. actual executions +- Various workload types (CPU-bound, memory-bound, I/O-bound) +- Edge cases (no spills, all spills, missing stats) + +## Implementation Timeline + +Estimated 4-week effort: + +- **Week 1:** Skeleton + Statistics (Phases 1-2) +- **Week 2:** Algorithm Implementation (Phase 3) +- **Week 3:** Integration (Phase 4) +- **Week 4:** Reporting + Testing (Phase 5) + +*Note: Timeline assumes information gaps are resolved before starting.* + +## Success Criteria + +✅ Code compiles and runs +✅ Can extract statistics from workunits +✅ Time estimation works correctly +✅ Cost calculation is accurate +✅ Report generation is readable +✅ Estimates within 20% of actual +✅ Identifies better queues when available +✅ Analysis completes in < 5 seconds + +## Future Enhancements + +- Machine learning for better predictions +- Historical analysis database +- Real-time recommendation during submission +- ECLWatch UI integration +- Automatic queue selection +- Multi-objective optimization + +## Related Files + +### In This Directory +- `anacommon.hpp/cpp` - Common interfaces +- `anawu.hpp/cpp` - Main workunit analysis +- `anarule.hpp/cpp` - Rule-based analysis + +### External References +- `system/jlib/jstats.h` - Statistics types +- `common/workunit/workunit.cpp` - Thor cost calculation +- `common/thorhelper/commonext.hpp` - Helper functions + +## Questions or Issues? + +For questions about: +- **Design decisions:** See QUEUE_RECOMMENDATION_PLAN.md design rationale +- **Implementation details:** See GITHUB_COPILOT_PROMPT.md implementation tasks +- **Information gaps:** See QUEUE_RECOMMENDATION_PLAN.md open questions section +- **Testing:** See all three documents' testing sections + +## Document Maintenance + +These documents should be updated: +- When information gaps are resolved (mark as resolved, add details) +- When design decisions change (update rationale) +- During implementation (add discovered TODOs) +- After completion (add lessons learned) + +--- + +**Last Updated:** 2024 +**Status:** Planning Complete, Ready for Implementation +**Next Step:** Resolve critical information gaps, then begin Phase 1