Use WSPR database directly instead of web scraping - #7
Conversation
Connect to the WsprDaeamon TimescaleDB directly and pull spots. It's faster, gets more recent spots (bypass wsprnet cache) This also moves the installation of python requirements over venv
domnantas
left a comment
domnantas
left a comment
There was a problem hiding this comment.
Nice! Just a couple of questions since I'm not an expert in Python
| def to_list(tuple): | ||
| return list(tuple) | ||
|
|
There was a problem hiding this comment.
Why is this necessary? Isn't it possible to map(list, sqldata) without this wrapper function?
There was a problem hiding this comment.
Its way more clear and understandable, using this 2 line function. Because for those who don't know how does map works, this 2 line is easy to understand that it basically converts tuples to lists.
There was a problem hiding this comment.
I struggle to understand how
def to_list(tuple):
return list(tuple)
tabledata = map(to_list, sqldata)is more clear than
tabledata = map(list, sqldata)it's just a tiny abstraction and you still need to understand how map works. Don't think this is worth discussing, both approaches work fine :)
There was a problem hiding this comment.
First code that you write is not a usage of functions in python
Normal usage is:
def to_list(tuple):
return list(tuple)
tabledata = to_list(sqldata) And in that way you don't need to use map(). But as you mentioned, yeah both approaches work fine but this one is easier to understand for python developers.
There was a problem hiding this comment.
That's exactly how to_list is used in this PR
https://github.com/sm3ulc/hab-wspr/pull/7/files#diff-d6af0459a37d985953d7040c14f53feb3b9cc9e58b543aa3c2b80256d276c5e0R30
There was a problem hiding this comment.
Oh, noticed that just now. Probably map is taking just the function name, didn't need to use it before, my bad mate.
There was a problem hiding this comment.
In Javascript and functional programming, it is common to see something like this
[1, 2, 3].map(toString); // toString is a function
// which basically translates to
[toString(1), toString(2), toString(3)]Cheers!
| except psycopg2.OperationalError as err: | ||
| logging.error("PostgreSQL connect error: " + err) | ||
| conn = None |
There was a problem hiding this comment.
Not too familiar with Python, but isn't it necessary to return in the exception handler, so the conn.cursor() on the next line would not be executed?
* dry_run: Fix printing of PostgreSQL error Dry run fix
3 participants
Connect to the WsprDaeamon TimescaleDB directly and pull spots.
It's faster, gets more recent spots (bypass wsprnet cache).
This also moves the installation of python requirements over venv.