Fitbit API exporter

The project is a simple script to export data from Fitbit's Web APIs to a custom InfluxDB database, and then graph it via Grafana.
Everything is made easier via docker and docker-compose. Grafana dashboards are available at:

• API export: https://grafana.com/grafana/dashboards/12348
• Archive data export: https://grafana.com/grafana/dashboards/12374

Quick setup (API exporter)

Notes:

• basic knowledge of docker is assumed
• unless otherwise noted, leave fields at default
• the script sources environment variables if missing, otherwise sources the saved config data (this is because the token gets refreshed automatically thanks to the refresh token)
• if you don't run the scripts on a server, you may need to refresh the access token and refresh token at the next run, because the refresh token may have already expired (see below)

Step-by-step guide:

1. Do a test-run of the docker-compose.yml provided, and customize it to your liking, ensure the containers come up, esp. the InfluxDB one
2. Go at Fitbit dev login, login with your account
3. Register an application: click on "Register an app" at the top
   • Application name and description to your liking
   • Application website, Organization website, terms of service, privacy policy, callback URL, you can all set "http://localhost:8080/"
   • OAuth 2.0 Application Type: Personal
   • Default Access Type: Read-Only
4. Get first two parameters: click on "Manage my apps", click on your new application; you will need to note down the following in order to fill corresponding env vars:
   • OAuth 2.0 Client ID: CLIENT_ID
   • Client Secret: CLIENT_SECRET
   • Callback URL: CALLBACK_URL
5. Generate the tokens: click on the small link at the bottom of the page "OAuth 2.0 tutorial page"
   1. Select "Flow type": "Authorization Code Flow"
   2. Select Scopes: activity, heartrate, profile, settings, sleep, weight
   3. Click on the link at the end of section 1 "We've generate the authorization URL for you, all you need to do is just click on the link below:"
   4. Select all scopes and click "Allow"
   5. Copy the code parameter from the URL of the window that opens: code=[.....]#=
   6. Paste it in the "1A Get Code" form
   7. Copy the curl call to a script, remove newlines, execute the script
   8. Copy the JSON output from the script and paste into the "2: Parse response" section
   9. Note down:
      • Access token: ACCESS_TOKEN
      • Refresh Token: REFRESH_TOKEN
6. You can fill the provided docker-compose.yml with the parameters obtained
7. Pull up the containers

Quick setup (Archive export loader)

1. Navigate to www.fitbit.com, access with your credentials
2. Go to "Account settings", then "Data Export"
3. Select "Export account archive" (or similar - the whole of data available)
4. Confirm the export via the export email
5. Wait for the data to be available
6. Download the data
7. Extract the archive
8. Pull up the influxdb container: docker-compose up -d influxdb
9. Run the docker image: docker run -it --network fitbit-exporter -v<folder_containing_extracted_archive>:/dump -eDB_HOST=... -eDB_PORT=... -eDB_USER=... -eDB_PASSWORD=... -eDB_NAME=... registry.gitlab.com/fsvm88/fitbit-api-exporter:latest /fitbit_export_loader.py (recheck the network name, as it depends on docker-compose)
10. Wait for the container to finish

The command can be rerun as required in case of errors, subsequent runs should skip already-written datapoints using the timestamps of the entries.

NOTE: the data folder must be provided at user-site-export parent level (e.g: /path/to/folder/username)

Limitations

Fitbit APIs

1. Fitbit APIs are limited to 150calls/hour, the script will detect this and sleep for 1h 10s
2. The script will sleep for 4h when there's no additional data to fetch, and only fetch the last day at each round
3. Support for older responses of the APIs is limited/non-existing

Intra-day time series

Intra-day time series are not implemented.
I originally wanted to add them, but saw little benefit after finishing the normal time-series.
One additional issue with intra-day series is that they require additional setup on Fitbit's side, and you may need to request access to those via Fitbit support. I have not tried contacting them because I had no need for it.
One last note is that intra-day data is much more granular (1m/30s/1s intervals), so each query will return a lot more data to download and store, which I had no need for.

InfluxDB 2

InfluxDB 2 seems to have evolved in the "we don't need actual users" sense.
Query language is completely different, total stranger to InfluxQL, SQL and readability, and built on a badly designed language (JS) on the grounds of "it was the most popular language of 2018" (sigh).
On top of that, it adds a lot of NIH design decisions, like:

• renaming "databases" -> "buckets", then giving them names just-as-if they were databases
• adding "organizations", adding another layer of complexity most won't require, except now it's mandatory
• using IDs everywhere just for the sake of it without much rationale (tell me again why I name a bucket "fitbit" and still have to manually pass the ID in another command, where I also have to specify org-name or org-id anyway)
• keeping v1 APIs, except they're now hidden behind "influx v1" command, except they are not completely the same, and new authorization has been plumbed into them without much thought for usability (see for instance)

The CLI expects you to juggle between tokens, organization IDs, bucket IDs, user IDs and user names (hint: they're not the same), authorization IDs+Token Names+User names ("influx v1 auth list" - !). Commands require you explicitly pass named options, even though they really only accept IDs - e.g: "cat thisfile.py" vs "cat --file thisfile.py" - so you have to consult the help before every single command you run, because of course it's never as simple as "--id".

The documentation also seems to suck, spent 4h trying to work out all the details, and when I finally got it working, of course Grafana dashboards refused to render anything, even though Datasource test went fine, and data was accessible. Must assume they broke compat for something in the InfluxQL layer...

As such, I do not plan nor intend to upgrade the project to InfluxDB 2.0, ever.

Development

api_poller.py is all there is to it, use docker-compose-dev.yml to pull up a local instance for testing

Deps

Python script, developed with:

• Fitbit API client
• InfluxDB client