A user has reported that they're unable to download a set of data that covers the full YNWT area:
Investigating this it seems that there is a subset of stations causing failures in the downloader. Further investigation reveals that the failing stations are part of the ECCC network. Suspect that this a data issue.
As the data is relatively small, wrote a small downloader program to test all the stations for correctness:
import json
from urllib.parse import quote_plus
from urllib.request import urlopen
import os
import time
import logging
from decimal import Decimal
from io import BytesIO
from zipfile import ZipFile
# set up logging
logging.basicConfig(
filemode='a',
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
datefmt='%d-%b-%y %H:%M:%S',
filename='download.log')
histories = json.load(open('histories.json'))
# open all stations json data from url
# histories = json.load(urlopen('https://services.pacificclimate.org/met-data-portal-ynwt/meta/histories'))
# output total stations
print(f'Total stations: {len(histories)}')
# make sure cached zips and output folders exist
os.makedirs('cached-zips', exist_ok=True)
os.makedirs('output', exist_ok=True)
# define downloadurl template:
def downloadUrl(lat, long) :
boxOffset = 0.02
polygon = quote_plus(f'MULTIPOLYGON ((({(long-boxOffset):.4f} {(lat-boxOffset):.4f}, {(long-boxOffset):.4f} {(lat+boxOffset):.4f}, {(long+boxOffset):.4f} {(lat+boxOffset):.4f}, {(long+boxOffset):.4f} {(lat-boxOffset):.4f}, {(long-boxOffset):.4f} {(lat-boxOffset):.4f})))')
return f'https://services.pacificclimate.org/met-data-portal-ynwt/data/pcds/agg/?from-date=&to-date=&network-name=&input-vars=&input-freq=&input-polygon={polygon}&only-with-climatology=&download-timeseries=Timeseries&data-format=nc'
# download each zip file for each station using the template url
# save to the cached zips folder
for history in histories:
id = history['id']
try:
# check if the file is already downloaded in our cached zips
if os.path.isfile(f'cached-zips/{id}.zip'):
logging.info(f'File already downloaded: {id}')
with open(f'cached-zips/{id}.zip', 'rb') as f:
with ZipFile(BytesIO(f.read())) as z:
z.extractall(f'output/{id}/')
# if not, download the file
else:
logging.info(f'Downloading file: {id}')
url = downloadUrl(history['lat'], history['lon'])
logging.info(f'Downloading from: {url}')
with urlopen(url) as stationZip:
content = stationZip.read()
# save the zip file to the cached zips folder
with open(f'cached-zips/{id}.zip', 'wb') as f:
f.write(content)
# unzip the file to the output folder
with ZipFile(BytesIO(content)) as z:
z.extractall(f'output/{id}/')
# sleep for a second to avoid overloading the server
time.sleep(1)
except:
print(f'Error downloading file: {id}')
# output exception message to log file
logging.error(f'Error downloading file: {id}', exc_info=True)
Notes: The download url doesn't allow for a specific station, so we use the history's lat and long to draw a box around each station. This has the limitation that very close together stations may download each other.
The following histories failed:
193,195,199,318,319,320,349,367,368,369,370,371,376,377,378,379,381,382,383,401,433,434,436,437,438,440,441,442,443,445,448,449,451,457,458,459,460,467,468,469,470,472,485,486,534,535,537,539,558,560,561,574,575,577,580,581,582,583,611,613,616,631,632,653,654,655,656,663,665,669,670,671,672,673,677,683,684,696,698,699,700,701,708,709,710,711,712,722,743,744,745,746,747,748,753,754,755,756,787,788,789,812,826,887,888,889,891,892,893,1064
Continuing to investigate.
A user has reported that they're unable to download a set of data that covers the full YNWT area:
Investigating this it seems that there is a subset of stations causing failures in the downloader. Further investigation reveals that the failing stations are part of the ECCC network. Suspect that this a data issue.
As the data is relatively small, wrote a small downloader program to test all the stations for correctness:
Notes: The download url doesn't allow for a specific station, so we use the history's lat and long to draw a box around each station. This has the limitation that very close together stations may download each other.
The following histories failed:
193,195,199,318,319,320,349,367,368,369,370,371,376,377,378,379,381,382,383,401,433,434,436,437,438,440,441,442,443,445,448,449,451,457,458,459,460,467,468,469,470,472,485,486,534,535,537,539,558,560,561,574,575,577,580,581,582,583,611,613,616,631,632,653,654,655,656,663,665,669,670,671,672,673,677,683,684,696,698,699,700,701,708,709,710,711,712,722,743,744,745,746,747,748,753,754,755,756,787,788,789,812,826,887,888,889,891,892,893,1064Continuing to investigate.