Skip to content

Commit 00ed574

Browse files
FreaxMATEFreaxMATE
committed
update readme
1 parent bed0614 commit 00ed574

1 file changed

Lines changed: 129 additions & 119 deletions

File tree

‎README.md‎

Lines changed: 129 additions & 119 deletions
Original file line numberDiff line numberDiff line change
@@ -1,164 +1,174 @@
1-
# Energy Trading Day-Ahead Price Analysis
1+
# Energy Trading Analysis
22

3-
A Python-based tool for exploring and optimizing energy procurement strategies using Danish day-ahead electricity prices (DK2). The centerpiece is an **interactive dashboard** for visualizing price trends, moving averages, and predictions for multiple countries.
3+
![Dashboard Screenshot](screenshot.png)
44

5-
## 🖼️ Interactive Dashboard
5+
Download, analyze, and publish ENTSO-E day-ahead spot prices (and generation mix) per bidding zone/country, with computed features (moving average + Holt‑Winters forecasts) and static HTML dashboards.
66

7-
![Dashboard Screenshot](screenshot.png)
7+
Core entry points:
8+
- CLI: [`main.main`](src/main.py) in [src/main.py](src/main.py)
9+
- Data pipeline: [`datamanager.DataManager`](src/datamanager.py) in [src/datamanager.py](src/datamanager.py)
10+
- Analyses: [`analysis.MovingAverageAnalyzer`](src/analysis.py), [`analysis.ForecastAnalyzer`](src/analysis.py), orchestrated by [`analysis.AnalysisRunner`](src/analysis.py) in [src/analysis.py](src/analysis.py)
11+
- Static reporting: [`plot_bokeh.create_static_dashboard`](src/plot_bokeh.py) + [`plot_bokeh.generate_index_html`](src/plot_bokeh.py) in [src/plot_bokeh.py](src/plot_bokeh.py)
812

9-
*Example: Dash web app showing price and moving average for selected country. Explore historical trends, zoom into the last 24 hours, and compare predictive features interactively.*
13+
---
1014

11-
## 📊 Features
15+
## What it does
1216

13-
- **Interactive Dashboard**: Explore price and moving average trends for any country using a Dash web app (`src/plot.py`).
14-
- **SciChart.js Stack**: High-performance alternative visualization using Flask + SciChart.js (`src/plot_scichart.py`).
15-
- **Static HTML Reports**:
16-
- **Bokeh (Recommended)**: High-performance, standalone HTML reports suitable for VCS hosting (`src/plot_bokeh.py`).
17-
- **ECharts**: Legacy static reports (`src/plot_echarts.py`).
18-
- **Historical Price Analysis**: Processes 3 years of hourly electricity price data (2023-2025).
19-
- **Procurement Optimization**: Implements adaptive procurement algorithm with configurable parameters.
20-
- **Cost Analysis**: Compares total costs across different procurement frequencies (1-24 times per year).
21-
- **Visualization**: Generates comprehensive charts showing price trends and optimal purchase points.
17+
1. **Downloads data from ENTSO-E**
18+
- Spot prices per country code via `EntsoePandasClient` inside [`datamanager.DataManager.download`](src/datamanager.py) → [`datamanager.DataManager.download_by_country_code`](src/datamanager.py).
19+
- Generation mix per country code via [`datamanager.DataManager.download_generation_by_country_code`](src/datamanager.py).
2220

23-
## 🚀 Quick Start
21+
2. **Stores data on disk**
22+
- Price data is loaded with CSV schema `time, price` by [`datamanager.DataManager.__read_data`](src/datamanager.py).
23+
- Features are saved as `{country}_{feature}.csv` by [`datamanager.DataManager.save_analysis`](src/datamanager.py) and discovered via [`datamanager.DataManager.__read_features`](src/datamanager.py).
24+
- `data/features.csv` is (re)generated with `ma` and `forecast` columns by [`datamanager.DataManager.__update_features_file`](src/datamanager.py).
2425

25-
### Prerequisites
26-
- Python 3.8+ (Windows, macOS, Linux)
27-
- Git (optional)
26+
3. **Runs analyses**
27+
- Moving average feature `ma` via [`analysis.MovingAverageAnalyzer.analyze`](src/analysis.py).
28+
- Forecast feature(s) via Holt‑Winters exponential smoothing in [`analysis.ForecastAnalyzer.analyze`](src/analysis.py).
2829

29-
### Installation
30+
4. **Generates static HTML reports**
31+
- Per-country report: [`plot_bokeh.create_static_dashboard`](src/plot_bokeh.py) (Bokeh, WebGL, downsampling).
32+
- Dashboard index that links all reports: [`plot_bokeh.generate_index_html`](src/plot_bokeh.py).
3033

31-
**Simple Setup:**
32-
```bash
33-
git clone <repository-url>
34-
cd EnergyTradingAnalysis
35-
pip install -r requirements.txt
36-
```
34+
---
3735

38-
**Alternative Methods:**
39-
- **Automated**: Run `./scripts/setup.sh` (Linux/macOS) or `scripts\setup.bat` (Windows)
40-
- **Conda**: `conda env create -f environment.yml && conda activate energy-trading-analysis`
41-
- **Docker**: `docker-compose up --build`
42-
- **Nix**: `nix develop` (Linux/macOS)
36+
## Repository layout (high-level)
4337

44-
### Running the Dashboard & Analysis
38+
- Source code: [src/](src/)
39+
- Data pipeline: [src/datamanager.py](src/datamanager.py)
40+
- Analysis: [src/analysis.py](src/analysis.py)
41+
- Static reporting: [src/plot_bokeh.py](src/plot_bokeh.py)
42+
- Config: [src/config.py](src/config.py)
43+
- Utilities: [src/utils.py](src/utils.py)
44+
- Logging: [src/logger.py](src/logger.py)
45+
- Exceptions: [src/exceptions.py](src/exceptions.py)
46+
- Data (CSV): [data/](data/)
47+
- Generated static site output directory (configured): [doc/](doc/) via `OUTPUT_DIR` in [src/config.py](src/config.py)
48+
- Prebuilt/served static site directory (also present): [docs/](docs/) and [Dockerfile](Dockerfile)
4549

46-
From the `src/` directory:
50+
Note: both [doc/](doc/) and [docs/](docs/) exist. `OUTPUT_DIR` is set to `doc/` in [src/config.py](src/config.py), while the [Dockerfile](Dockerfile) copies `docs/` into nginx. If you want Docker to serve newly generated reports, ensure the generated output ends up in the directory being served.
4751

48-
```bash
49-
cd src
50-
python main.py plot # launch interactive dashboard (Dash)
51-
python plot_scichart.py # launch high-performance dashboard (SciChart.js)
52-
python plot_bokeh.py [COUNTRY] # generate high-performance static report (e.g., DK_2)
53-
python plot_echarts.py [COUNTRY] # generate legacy static HTML report
54-
python scheduled_procurement.py # scheduling analysis + day-ahead trend plot
55-
python day_prices.py # hourly profile (price by hour) plot
56-
```
52+
---
53+
54+
## Requirements
55+
56+
- Python **3.8+** (setup script enforces this) — see [scripts/setup.sh](scripts/setup.sh) and [scripts/setup.bat](scripts/setup.bat)
57+
- An **ENTSOE API key** (see configuration below)
58+
- Python dependencies in [requirements.txt](requirements.txt) (also Nix dev shell in [flake.nix](flake.nix))
59+
60+
---
61+
62+
## Configuration
63+
64+
This project expects an ENTSO-E API key via environment variables.
65+
66+
- `.env.example` shows the expected shape: [.env.example](.env.example)
67+
- Local overrides: `.env` (not committed) — [.env](.env)
68+
- Config validation happens in [src/config.py](src/config.py). Missing key raises [`exceptions.ConfigException`](src/exceptions.py).
69+
70+
Required:
71+
- `ENTSOE_API_KEY`: ENTSO-E Transparency Platform API key (used by `EntsoePandasClient` inside [`datamanager.DataManager.download`](src/datamanager.py)).
5772

58-
**Generated Output:**
59-
- `output/dash_screenshot.png`: Dashboard screenshot (add manually)
60-
- `output/report_DK_2.html`: Interactive HTML report generated by Bokeh (or ECharts)
61-
- `output/dayaheadprices.png`: Price trends with optimal purchase points (produced by `scheduled_procurement.py`)
62-
- `output/total_cost_vs_nproc.png`: Total cost vs number of procurements (produced by `scheduled_procurement.py`)
63-
- `output/price_by_hour.png`: Average price by hour with error bars (produced by `day_prices.py`)
73+
---
6474

65-
## 📁 Project Structure
75+
## Quickstart (venv)
6676

77+
### 1) Create and install
78+
Use the provided scripts:
79+
80+
```sh
81+
./scripts/setup.sh
6782
```
68-
├── src/ # Source code and analysis scripts
69-
│ ├── main.py # CLI entry point (download, analyze, plot)
70-
│ ├── plot.py # Dash dashboard for interactive exploration
71-
│ ├── plot_scichart.py # High-performance dashboard using SciChart.js
72-
│ ├── plot_bokeh.py # Generate static HTML reports with Bokeh
73-
│ ├── plot_echarts.py # Generate static HTML reports with ECharts
74-
│ ├── datamanager.py # Data loading and feature extraction
75-
│ ├── dataanalysis.py # Analysis functions (moving average, prediction)
76-
│ ├── scheduled_procurement.py # Procurement optimization analysis
77-
│ ├── day_prices.py # Hourly price profile and plot
78-
│ └── utils.py # Utility functions
79-
├── data/ # CSV price data files (2023-2025)
80-
├── output/ # Generated visualizations (PNG files)
81-
├── scripts/ # Setup scripts for different platforms
82-
├── requirements.txt # Python dependencies
83-
├── environment.yml # Conda environment
84-
├── Dockerfile # Docker support
85-
├── docker-compose.yml # Docker orchestration
86-
├── flake.nix # Nix environment
87-
└── README.md # Project documentation
83+
84+
Windows:
85+
86+
```bat
87+
scripts\setup.bat
8888
```
8989

90-
## 🔬 Algorithm Overview
90+
(They create `venv/` and install from [requirements.txt](requirements.txt).)
9191

92-
**Data Source**: [ENTSO-E Transparency Platform](https://newtransparency.entsoe.eu/)
93-
- Market: DK2 (Denmark Eastern) day-ahead prices
94-
- Resolution: Hourly data, converted to daily averages
95-
- Period: 2023-2025
92+
### 2) Set your ENTSO-E key
93+
Create `.env` at repo root (or export env var):
9694

97-
**Procurement Strategy (`sched_proc` function):**
98-
1. **Time Partitioning**: Divides the year into `n_parts` equal segments
99-
2. **Reference Tracking**: Maintains reference price that updates to lower values
100-
3. **Purchase Trigger**: Buys energy when price exceeds `reference + limit` (default: €10/MWh)
101-
4. **Cost Calculation**: Computes total cost for specified energy volume (default: 1000 MWh)
95+
```sh
96+
export ENTSOE_API_KEY="your_key_here"
97+
```
10298

103-
**Parameters:**
104-
- `mwhs`: Total energy to procure (default: 1000 MWh)
105-
- `n_parts`: Number of procurement periods (tested: 1, 2, 3, 4, 6, 12, 24)
106-
- `limit`: Price increase threshold (default: €10/MWh)
99+
### 3) Download data
100+
Runs ENTSO-E pulls for all configured country codes:
107101

108-
## 📈 Key Results
102+
```sh
103+
python src/main.py download
104+
```
109105

110-
- **Optimal Frequency**: Usually 3-6 procurements per year minimize total costs
111-
- **Cost Savings**: Significant reduction compared to single annual purchase
112-
- **Seasonal Patterns**: Purchase timing typically aligns with seasonal price cycles
113-
- **Diminishing Returns**: Increased procurement frequency shows diminishing cost benefits
106+
This calls [`datamanager.DataManager.download`](src/datamanager.py), which iterates `country_codes.csv` (see `COUNTRY_CODES_FILE` in [src/config.py](src/config.py)).
114107

115-
## 🛠️ Usage Examples
108+
### 4) Analyze data
109+
Computes moving average + forecasts per country:
116110

117-
**Basic Analysis:**
118-
```python
119-
buy_indices, total_cost = sched_proc(price_avg)
111+
```sh
112+
python src/main.py analyze
120113
```
121114

122-
**Custom Parameters:**
123-
```python
124-
buy_indices, total_cost = sched_proc(
125-
price=price_avg,
126-
mwhs=2000, # 2000 MWh total
127-
n_parts=6, # 6 procurements per year
128-
limit=15 # €15/MWh threshold
129-
)
115+
This calls [`datamanager.DataManager.analysis`](src/datamanager.py) → [`datamanager.DataManager.analysis_by_country_code`](src/datamanager.py), using [`analysis.AnalysisRunner`](src/analysis.py) with [`analysis.MovingAverageAnalyzer`](src/analysis.py) and [`analysis.ForecastAnalyzer`](src/analysis.py).
116+
117+
### 5) Generate static dashboards
118+
Generate per-country reports and the index dashboard:
119+
120+
```sh
121+
python src/plot_bokeh.py
130122
```
131123

132-
## 🔧 Troubleshooting
124+
Outputs:
125+
- `report_{COUNTRY}.html` files into `OUTPUT_DIR` (configured as `doc/`) in [src/config.py](src/config.py)
126+
- `index.html` dashboard via [`plot_bokeh.generate_index_html`](src/plot_bokeh.py)
127+
128+
Open in a browser:
129+
- [doc/index.html](doc/index.html)
130+
131+
---
132+
133+
## Data & file conventions
133134

134-
**Common Issues:**
135-
- Ensure Python 3.8+ is installed
136-
- Activate virtual environment before running
137-
- Use `python3` if `python` points to Python 2.x
138-
- On Linux/macOS: `chmod +x scripts/setup.sh` if permission denied
135+
### Prices
136+
- Loaded by [`datamanager.DataManager.__read_data`](src/datamanager.py) as:
137+
- `time` (UTC parsed), `price`
139138

140-
## 📄 License
139+
### Features
140+
- Written by [`datamanager.DataManager.save_analysis`](src/datamanager.py) to per-country files:
141+
- `{country}_ma.csv` (moving average output includes `ma`)
142+
- `{country}_forecast.csv` (forecast output contains multiple forecast columns)
143+
- Registry file: `data/features.csv` maintained by [`datamanager.DataManager.__update_features_file`](src/datamanager.py)
141144

142-
Educational and research use. Price data from ENTSO-E under their terms of use.
145+
### Utilities used by the pipeline
146+
- Efficient incremental download uses [`utils.read_last_csv_line`](src/utils.py) (reads last CSV line) in [`datamanager.DataManager.download_by_country_code`](src/datamanager.py).
147+
- Time-window extraction helper: [`utils.extract_last`](src/utils.py) in [src/utils.py](src/utils.py).
143148

144-
## 📸 Plots (output/)
149+
---
145150

146-
The repository includes example generated visualizations in the `output/` directory. If you run the scripts above they will be (re)created.
151+
## Development environments
147152

148-
- dayaheadprices.png
153+
### Nix dev shell
154+
A ready dev shell is defined in [flake.nix](flake.nix) (includes `entsoe-py`, pandas, statsmodels, bokeh, etc.).
149155

150-
![Day-ahead prices with procurement points](output/dayaheadprices.png)
156+
### Docker (static site)
157+
The provided [Dockerfile](Dockerfile) serves static files through nginx and copies [docs/](docs/) into the container. If you want to serve the freshly generated content from [doc/](doc/), align the directories (either generate into `docs/` or serve `doc/`).
151158

152-
Caption: Daily-averaged day-ahead prices with procurement markers for different numbers of procurements (N_proc). Each marker shows the time and price where the algorithm decided to buy — useful to inspect timing and clustering of purchase points.
159+
See also: [docker-compose.yml](docker-compose.yml).
153160

154-
- total_cost_vs_nproc.png
161+
---
155162

156-
![Total cost vs number of procurements](output/total_cost_vs_nproc.png)
163+
## Logging & troubleshooting
157164

158-
Caption: Total procurement cost (€) as a function of the number of procurements per year. Use this to identify the procurement frequency that minimizes total cost.
165+
- Logging is configured by [`logger.setup_logger`](src/logger.py) in [src/logger.py](src/logger.py) to both console and rotating files under [logs/](logs/).
166+
- Common failures:
167+
- Missing `ENTSOE_API_KEY` → [`exceptions.ConfigException`](src/exceptions.py) raised by [src/config.py](src/config.py).
168+
- Empty/malformed CSV → [`exceptions.DataException`](src/exceptions.py) used by [`utils.read_last_csv_line`](src/utils.py) and [`utils.extract_last`](src/utils.py).
159169

160-
- price_by_hour.png
170+
---
161171

162-
![Average price by hour](output/price_by_hour.png)
172+
## License
163173

164-
Caption: Average day-ahead price by hour-of-day with error bars (standard deviation). Highlights intraday patterns and hours with highest/lowest average prices.
174+
Licensed under **GNU GPL v3.0** — see [LICENSE](LICENSE).

0 commit comments

Comments
 (0)