setup improvements

Author: arsatyants

.github/copilot-instructions.md

@@ -1,13 +1,31 @@
 # Trading Analytics - AI Agent Instructions
 
+## Quick Reference
+
+**Start Here:**
+- Run analysis: `python web_server_parallel.py` → http://localhost:8080
+- Single script: `python gpu_wavelet_gpu_plot.py BTC 5m`
+- Setup: `./setup_env.sh` (creates .venv, installs deps, detects GPU)
+
+**Key Constraints:**
+- LSTM scaling: Always use `MinMaxScaler(feature_range=(-1, 1))` - never [0,1]
+- Data loading: Adaptive date ranges by timeframe (see pattern below)
+- OpenCL: Auto-detect best platform with scoring (NVIDIA>AMD>ARM)
+- Pi Zero: Use `--demo --no-plots` flag (ccxt causes bus errors)
+
+**File Organization:**
+- Notebooks: Monolithic, state-dependent (must run sequentially)
+- GPU scripts: Accept `<CURRENCY> <TIMEFRAME>` CLI args
+- Parallel system: `prepare_data.py` → 6 plot processes (3.5x speedup)
+
 ## Project Overview
-Cryptocurrency trading toolkit targeting **Orange Pi and embedded systems**. Performs BTC/ETH/SOL price prediction and anomaly detection using LSTM networks and GPU-accelerated wavelet transforms. Real-time data from Binance via CCXT.
+Cryptocurrency trading toolkit for **Orange Pi and embedded systems**. Performs BTC/ETH/SOL price prediction and anomaly detection using LSTM networks and GPU-accelerated wavelet transforms. Real-time data from Binance via CCXT.
 
 **Key Design Goals:**
-- Multi-platform GPU support (OpenCL → CUDA → Vulkan → OpenGL → CPU)
-- Graceful degradation (demo mode, lazy imports)
-- Parallel processing optimization for multi-core ARM devices
-- Publication-quality plot generation (300 DPI PNG)
+- Multi-platform GPU support (OpenCL → CUDA → Vulkan → OpenGL → CPU fallback)
+- Graceful degradation (demo mode, lazy imports, error handling)
+- Parallel processing (3.5x speedup via OS-level subprocess spawning)
+- Publication-quality plot generation (300 DPI PNG, 6 plots per analysis)
 
 ## Architecture
 
@@ -17,14 +35,14 @@ Jupyter Notebooks (btc-prediction.ipynb, wave_nada.ipynb)
     ↓ LSTM training, interactive analysis
     
 GPU Wavelet Scripts (5 variants: opencl/cuda/vulkan/opengl/cpu)
-    ↓ Batch processing with command-line args
+    ↓ Batch processing with CLI args: python <script>.py <CURRENCY> <TIMEFRAME>
     
 Web Servers (3 variants: standard/parallel/vulkan)
-    ↓ HTTP interface with async job tracking
+    ↓ HTTP interface (port 8080) with async job tracking
     
-Parallel Plot System (NEW)
+Parallel Plot System (PRODUCTION)
     prepare_data.py → [plot_01.py, plot_template.py × 5] (subprocess.Popen)
-    ↓ 3.5x speedup via OS-level parallelism
+    ↓ 3.5x speedup via OS-level parallelism (13s vs 45s)
 ```
 
 ### Core Files
@@ -45,9 +63,9 @@ Parallel Plot System (NEW)
 - `web_server_parallel.py`: **PREFERRED** - parallel plot generation (3.5x faster)
 - `web_server_vulkan.py`: Vulkan backend variant
 
-**Parallel System** (NEW):
+**Parallel System** (PRODUCTION):
 - `prepare_data.py`: Data load + GPU wavelet → `wavelet_data.pkl` cache
-- `plot_common.py`: Shared utilities (`gpu_convolve`, `load_wavelet_data`)
+- `plot_common.py`: Shared utilities (`gpu_convolve`, `load_wavelet_data`, `save_wavelet_data`)
 - `plot_01.py`: Main 4-panel overview (full implementation)
 - `plot_template.py`: Simplified plots 02a-05 (accepts plot ID arg)
 
@@ -60,7 +78,7 @@ python gpu_wavelet_gpu_plot.py BTC 5m    # Generates 6 PNG plots
 ./run_all_currencies.sh                  # 18 plots (BTC/ETH/SOL × 6)
 ./run_all_currencies_console.sh          # Console output with ASCII
 
-# Web interface
+# Web interface (RECOMMENDED)
 python web_server_parallel.py            # http://localhost:8080 (parallel)
 python web_server.py                     # http://localhost:8080 (sequential)
 
@@ -75,7 +93,7 @@ python gpu_wavelet_cpu_plot.py --demo --no-plots
 
 ## Critical Code Patterns
 
-### 1. Data Loading (MANDATORY for all scripts)
+### 1. Data Loading (Adaptive Date Ranges)
 **Pattern:** Adaptive date range based on timeframe to get ~700-1000 candles consistently.
 
 ```python

setup_env.sh

@@ -3,7 +3,8 @@
 # ===========================================
 # Automatic setup: Creates .venv and installs all dependencies + GPU support
 
-set -e  # Exit on error
+# Don't exit on error for optional components
+set +e
 
 # Colors for output
 GREEN='\033[0;32m'
@@ -70,44 +71,63 @@ echo -e "${GREEN}✓${NC} Activated"
 
 # Upgrade pip
 echo ""
-echo -e "${BLUE}[3/6]${NC} Upgrading pip..."
+echo -e "${BLUE}[3/7]${NC} Upgrading pip..."
 pip install --upgrade pip -q
 echo -e "${GREEN}✓${NC} pip upgraded"
 
-# Install all dependencies
+# Install system dependencies for Python packages
 echo ""
-echo -e "${BLUE}[4/6]${NC} Installing all dependencies from requirements.txt..."
-echo "This may take several minutes..."
-pip install -r requirements.txt
+echo -e "${BLUE}[4/7]${NC} Installing system dependencies..."
+echo "This requires sudo privileges..."
+
+# Python development headers (required for pyopencl)
+sudo apt update -qq
+sudo apt install -y python3-dev python3-pip build-essential cmake ocl-icd-opencl-dev clinfo >/dev/null 2>&1 || echo -e "${YELLOW}⚠${NC} Some system packages may not be available"
+echo -e "${GREEN}✓${NC} System dependencies installed"
+
+# Install core dependencies (without pyopencl)
 echo ""
-echo -e "${GREEN}✓${NC} All dependencies installed
+echo -e "${BLUE}[5/7]${NC} Installing core Python dependencies..."
+echo "This may take several minutes..."
 
-# GPU Setup (OpenCL & Vulkan)
+# Create temporary requirements without pyopencl
+grep -v "^pyopencl" requirements.txt > /tmp/requirements_core.txt
+
+pip install -r /tmp/requirements_core.txt
 echo ""
-echo -e "${BLUE}[5/6]${NC} Setting up GPU support (OpenCL & Vulkan)..."
+echo -e "${GREEN}✓${NC} Core dependencies installed"
+
+# Try to install pyopencl (optional)
 echo ""
+echo -e "${BLUE}[6/7]${NC} Installing GPU support (pyopencl - optional)..."
+if pip install pyopencl==2025.2.7 2>&1 | tee /tmp/pyopencl_install.log; then
+    echo -e "${GREEN}✓${NC} pyopencl installed successfully"
+else
+    echo -e "${YELLOW}⚠${NC} pyopencl installation failed (GPU acceleration disabled)"
+    echo "    CPU fallback will be used. Check /tmp/pyopencl_install.log for details."
+    echo "    The project will still work with gpu_wavelet_cpu_plot.py"
+fi
 
-# Install OpenCL support
-echo "Installing OpenCL runtime..."
-sudo apt update -qq
-sudo apt install -y ocl-icd-libopencl1 mesa-opencl-icd clinfo >/dev/null 2>&1
-echo -e "${GREEN}✓${NC} OpenCL installed"
+# GPU Runtime Setup (Vulkan & Mesa)
+echo ""
+echo -e "${BLUE}[7/7]${NC} Setting up GPU runtime (Vulkan & Mesa)..."
+echo ""
 
-# Install Vulkan support
-echo "Installing Vulkan runtime..."
-sudo apt install -y vulkan-tools mesa-vulkan-drivers libvulkan1 >/dev/null 2>&1
-echo -e "${GREEN}✓${NC} Vulkan installed"
+# Install Vulkan support (optional)
+echo "Installing Vulkan runtime (optional)..."
+sudo apt install -y vulkan-tools mesa-vulkan-drivers libvulkan1 >/dev/null 2>&1 || echo -e "${YELLOW}⚠${NC} Vulkan packages not available"
+echo -e "${GREEN}✓${NC} Vulkan setup complete"
 
-# Install Mesa utilities
-echo "Installing Mesa GPU utilities..."
-sudo apt install -y mesa-utils libgl1-mesa-dri >/dev/null 2>&1
+# Install Mesa utilities (optional)
+echo "Installing Mesa GPU utilities (optional)..."
+sudo apt install -y mesa-utils libgl1-mesa-dri mesa-opencl-icd >/dev/null 2>&1 || echo -e "${YELLOW}⚠${NC} Mesa packages not available"
 echo -e "${GREEN}✓${NC} Mesa utilities installed"
 
-# Load panfrost GPU driver
-echo "Loading panfrost GPU driver..."
-sudo modprobe panfrost 2>/dev/null || echo -e "${YELLOW}⚠${NC} Panfrost module not loaded (enable GPU in orangepi-config)"
-echo "panfrost" | sudo tee /etc/modules-load.d/panfrost.conf >/dev/null
-echo -e "${GREEN}✓${NC} Panfrost configured to load on boot"
+# Load panfrost GPU driver (Orange Pi specific)
+echo "Loading panfrost GPU driver (Orange Pi specific)..."
+sudo modprobe panfrost 2>/dev/null || echo -e "${YELLOW}⚠${NC} Panfrost module not loaded (enable GPU in orangepi-config if on Orange Pi)"
+echo "panfrost" | sudo tee /etc/modules-load.d/panfrost.conf >/dev/null 2>&1 || true
+echo -e "${GREEN}✓${NC} Panfrost configured"
 
 # Set GPU permissions
 echo "Configuring GPU permissions..."
@@ -117,7 +137,7 @@ echo -e "${GREEN}✓${NC} User added to render and video groups"
 
 # Configure environment
 echo ""
-echo -e "${BLUE}[6/6]${NC} Configuring GPU environment..."
+echo "Configuring GPU environment variables..."
 
 # Add to venv activation
 echo 'export RUSTICL_ENABLE=panfrost' >> "$VENV_DIR/bin/activate"
@@ -147,6 +167,10 @@ else
     echo -e "${YELLOW}⚠${NC} GPU device not found - enable in orangepi-config and reboot"
 fi"
 
+# Check what was installed
+PYOPENCL_INSTALLED=false
+$VENV_DIR/bin/python -c "import pyopencl" 2>/dev/null && PYOPENCL_INSTALLED=true
+
 # Summary
 echo ""
 echo "======================================================================"
@@ -155,13 +179,30 @@ echo "======================================================================"
 echo ""
 echo "Virtual environment: $VENV_DIR/"
 echo ""
+echo "Python Packages:"
+echo -e "  ${GREEN}✓${NC} Core dependencies (numpy, pandas, torch, etc.)"
+if [ "$PYOPENCL_INSTALLED" = true ]; then
+    echo -e "  ${GREEN}✓${NC} pyopencl (GPU acceleration via OpenCL)"
+else
+    echo -e "  ${YELLOW}⚠${NC} pyopencl not installed (CPU fallback available)"
+fi
+echo ""
 echo "GPU Status:"
 if [ -e /dev/dri/card0 ]; then
-    echo -e "  ${GREEN}✓${NC} GPU device detected"
-    echo "  To use GPU acceleration: RUSTICL_ENABLE=panfrost python script.py"
-    echo "  (automatically set when activating venv)"
+    echo -e "  ${GREEN}✓${NC} GPU device detected: /dev/dri/card0"
+    if [ "$PYOPENCL_INSTALLED" = true ]; then
+        echo -e "  ${GREEN}✓${NC} OpenCL GPU acceleration available"
+        echo "  To use GPU: RUSTICL_ENABLE=panfrost python script.py"
+        echo "  (automatically set when activating venv)"
+    else
+        echo -e "  ${YELLOW}⚠${NC} GPU detected but pyopencl not installed"
+        echo "  Run: source .venv/bin/activate && pip install pyopencl"
+    fi
 else
-    echo -e "  ${YELLOW}⚠${NC} GPU not detected - run 'sudo orangepi-config' to enable"
+    echo -e "  ${YELLOW}⚠${NC} GPU device not detected"
+    if [[ "$MODEL" == *"OrangePi"* ]]; then
+        echo "  Enable GPU in orangepi-config and reboot"
+    fi
 fi
 echo ""
 echo "To activate the environment manually:"