Fix Brazil fetcher

[kratzert]

The API for Brazil requires login credentials to fetch an API token (which is valid for 15 minutes) from their API. This requires changes to the code and a way to allow the user to provide login credentials (via .env file).

This will only be fixed once I have working credentials.

[kratzert]

https://github.com/kratzert/RivRetrieve-Python/tree/fix-brazil-fetcher contains a potential fix. However, I can't test it because my login credentials are still not working.

If anyone has login credentials, add a .env file to the root directory with

ANA_USERNAME=...
ANA_PASSWORD=...

and replace the dots with your login data. Then try running examples/test_brazil_fetcher.py

[simonmoulds]

@kratzert My credentials are working (checked this morning with an existing script), so I can take a look at this.

[simonmoulds]

I think I've fixed the Brazil fetcher. @kratzert perhaps you could try running examples/test_brazil_fetcher.py from the fix-brazil-fetcher branch with your credentials now?

[kratzert]

Thanks for looking into this and sorry if I wasn't clear enough. The problem is that my auth credentials are not working, also not on their platform https://www.ana.gov.br/hidrowebservice/swagger-ui/index.html#/WSEstacoesTelemetricasController/oAUth For that reason, I (and thus Gemini) was never able to test if the code works and to see potential failures and fix them. The only thing I changed so far was adding the option to save the credentials in the .env file and to read from their. Plus I provided the documentation pdf to Gemini to understand how to call the auth service. But I couldn't (and still can't) test the actual data download.

I already reached out to them and asked for clarification. I will look into your changes once my credentials work.

[simonmoulds]

No problem - you were perfectly clear. In any case the fetcher was not working with my credentials either, hence the changes I've made. Basically I think Gemini had decided to use the endpoint that is optimised for realtime data ("EstacoesTelemetricas/HidroinfoanaSerieTelemetricaAdotada/v1") to download the full timeseries. I've changed it now so that it uses either "EstacoesTelemetricas/HidroSerieVazao/v1" (discharge) or "EstacoesTelemetricas/HidroSerieCotas/v1" (stage), which return daily data in 366 day chunks.

Anyway, let's pause this discussion until your credentials are working.

[kratzert]

They fixed the problem and my login credentials are now working. I will look back into this in the next days.

[kratzert]

Okay, I finally got back to it. I had to do some minor modifications to your code before it ran for me (the spaces in the parameter names of the requests were not encoded as "%20" but as "+". I also added support for fetching metadata from EstacoesTelemetricas/HidroInventarioEstacoes/v1? (there are more than 37k stations returned). I haven't added back the support for the real-time data (15 minute resolution that can only be queried in chunks of 30days.

For precipitation: This is not "catchment precipitation" but precipitation from point measurements. I am undecided if we really want to add this here. I mean it doesn't hurt, but then where do we separate between fetching "river" data and just fetching any data. For NRFA, it was "catchment average precipitation", which I can see how this relates a bit more. But to be honest, I actually would not even include that.

Before I push my changes to this branch, I will wait for #46 to be merged, so that I can merge these changes into this branch.

[kratzert]

I messed up a git merge of the main branch into the fix-brazil-fetcher branch, leading to an annoying git history. To keep a clean history, I copied the brazil related changes to a new branch and merge this in #48

Note from the PR:

• the cached metadata file includes all stations (37k) not only the river related stations. Maybe we should change that?
• I haven't included the instantaneous data yet.
• The metadata function though retrieves metadata for all stations at once.