Diagnostics

.env.example

@@ -1,5 +1,5 @@
 project_path=""
-data_folder=""
+data_folder="data"
 
 estate_details_folder="estate_details"
 geo_locations_folder="locations"

.gitignore

@@ -15,5 +15,7 @@ notes.md
 /data_*/
 .db
 all_data.db
+price_history_loaded.txt
 # Logs
-scraping.log
+scraping.log
+app.log

README.md

@@ -1,19 +1,32 @@
 # Sreality
 #### Web-scraping Project - costs of houses in the Czech Republic on the biggest real-estate web - "Sreality.cz"
-## OLD INFO:
-* Approximately 12.000 items scraped every week and added to total - "data_prodej_byty_souhrn.xlsx"
-* PowerBI file with some interactive visualizations for 29.3.20-1.3.201 - "Vizualizace.pbix"
-* Four notebooks: Scraper, Cleaning & Dropping, Visualizaton, and All in one.
-* To run scraping, one needs to have an up-to-date chromedriver.exe in the same folder as Jupyter notebook
 
-## NEW INFO:
 ### Structure of this project:
 - MakeFile
 - Requirements: 
 - scraper: responsible for obtaining the data, there are few options: ...
 - utils: ..., Geodata 
 - db_managment: ..
-#### xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+### Good to know about Sreality:
+1) There is no official documentation for the API. Instead I used two main sources:
+    - https://dspace.cvut.cz/bitstream/handle/10467/103384/F8-BP-2021-Malach-Ondrej-thesis.pdf
+    - https://dspace.cvut.cz/bitstream/handle/10467/111141/F8-DP-2023-Drska-Vojtech-thesis.pdf
+
+2) Header is necessary for requests, e.g. {"User-Agent": "Mozilla/5.0"} If you do not use any, data is stil provided by request, BUT it is RANDOMIZED (prices, size of area, GPS) eventhough it might still looks valid, it is probably made up!
+3) There is a limitation to number of displayed estates per page = 999 (using "&per_page=999") - enough to get unique id + price, but not all the details, so the second API is needed to obtain details.
+4) There is a limitation to display estates in general - 60k, meaning one can get data from url ending with "&per_page=999&page=60", but not "&per_page=999&page=61", neither "&per_page=500&page=121" etc. Therefore it is a good practice to focus on categories one by one, instead of the general request, as there are currently around 95k offers to scrape.
+5) There is a way how to redirect to the full url using 'estate_id' a.k.a. 'hash_id'. Full address looks like: https://www.sreality.cz/detail/prodej/komercni/ubytovani/karlstejn-karlstejn-/937702732 but you can be redirected to this one using simplified url: https://www.sreality.cz/detail/x/x/x/x/937702732 where 'x' can be replaced by any value, ID and the structure is what matters.
+
+|       | A    | B    |
+|-------------|---  -|------|
+| 123         | ✅   | ✅  |
+| 456         | ✅   | ❌  |
+| 124         | ❌   | ✅  |
+| 324         | ❌   | ✅  |
+| 225         | ❌   | ✅  |
+
+ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 * 12.3.- 22.4.2020 = First outputs, preparation of the automatized process.
 * 24.-26.4. Visualizations in PowerBI
 * 27.-28.4. Creating representative .ipynb files with comments
@@ -32,4 +45,12 @@ xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 * 29.4. Preparing and inserting estate_details into DB
 * 30.4. Preparing and inserting Price history into DB
 * 1.5. Complete run with scraping missing details and updating DB
-* 3.5. Logging significantly improved
+* 3.5. Logging significantly improved
+* 4.5. basic mailing and scheduler of the scraper
+* 5.5. Diagnostics: new batch summary, price changes as mail
+* 6.5. scraping info to provide URL of the estate
+* 7.5. rework of estate_id, diagnostics provide URL and percentage change of price
+* 10.5. separating scrapers per category to avoid limit of 60k offers
+* 15-17.5. started using FastAPI, main.py + main.html
+* 19.5. main page formatting statistics + refreshing options 
+* 22.5. hacked redirecting or URL - estate_id is all we need

assets/css/main_page.css

@@ -0,0 +1,31 @@
+/* Custom font from Google Fonts */
+@import url('https://fonts.googleapis.com/css2?family=Roboto:wght@400;700&display=swap');
+
+body {
+    font-family: 'Roboto', sans-serif;
+}
+
+.nav-link {
+    color: #ffffff;  /* White text for better contrast */
+    text-decoration: none;
+    margin: 0 10px;  /* Spacing between links */
+    font-weight: 500; /* Slightly bold */
+    transition: color 0.2s ease-in-out; /* Smooth transition */
+}
+
+.nav-link:hover {
+    color: #66b2ff; /* Light blue when hovered */
+}
+
+.nav-link:visited {
+    color: #d3d3d3; /* Light grey when visited */
+}
+
+.nav-link:active {
+    color: #d3d3d3; /* Light grey when active */
+}
+
+.separator {
+    color: #ffffff;  /* Color for the separator between links */
+    font-weight: 300; /* Lighter weight for the separator */
+}

callbacks.py

@@ -0,0 +1,88 @@
+from datetime import datetime, timedelta
+from dash.dependencies import Input, Output, State
+from dash import dcc, html
+import pandas as pd
+import numpy as np
+import plotly.express as px
+import plotly.graph_objects as go
+
+def register_callbacks(app, runner, data):
+
+     @app.callback(
+        Output('output-message-email', 'children'),
+        [
+          Input('send-email-button', 'n_clicks'),
+          Input('input-string-email', 'value')
+        ]
+    )
+
+     def send_email(n_clicks, text_email):
+
+          if n_clicks > 0:
+               current_time = datetime.now()
+               try:
+                    runner.mailing.send_email(subject=f"testing_{current_time}", message_text=f"pokus: {text_email}")
+                    return "Sent sucessfully"
+               except Exception as e:
+                    return f"An error occurred: {e}"
+
+          else:
+               pass
+
+
+     @app.callback(
+        Output('output-message-favorite', 'children'),
+        [
+          Input('save-favorite-button', 'n_clicks'),
+          Input('input-estate_id', 'value'),
+          Input('input-note', 'value')
+        ]
+    )
+
+     def save_favorite(n_clicks, estate_id, notes):
+
+          if n_clicks > 0:
+               current_time = datetime.now()
+               #TODO: i was thinking about this being loaded outside of callbacks,
+               #todo but it cannot, we need to refresh it after each click
+               favs_df = runner.data_manager.get_all_favorites("saved_estates")
+               if estate_id in favs_df['estate_id'].unique():
+                    return f"Estate_id '{estate_id}' already in favorites."
+               else:
+                    try:
+                         runner.data_manager.insert_new_favorite(
+                              "saved_estates", estate_id, notes, current_time
+                         )
+                         return f"New estate_id '{estate_id}' with note '{notes}' saved sucessfully."
+
+                    except Exception as e:
+                         return f"An error occurred while saving estate_id '{estate_id}': {e}"                 
+          else:
+               pass
+
+     @app.callback(
+          Output('favorites-table', 'data'),
+          Input('refresh-favorite-button', 'n_clicks'),
+    )
+
+     def display_favorites(n_clicks):
+
+          if n_clicks > 0:
+               saved_favs = runner.data_manager.get_all_favorites("saved_estates")
+               saved_estates_id = saved_favs['estate_id'].tolist()
+
+               data = runner.data_manager.get_all_records(
+                    "price_history_new2",
+                    offer_ids=saved_estates_id
+                    )
+               data = data.merge(saved_favs[['estate_id', 'notes']], on="estate_id",  how='left')
+
+               data = data.sort_values(by='estate_id')
+
+               return data.to_dict('records')
+
+          else:
+               pass
+
+
+