sensors.AFRICA API

API to save and access data from deployed sensors in cities all around Africa.

Documentation

The API is documented here.

Development

Gitignore is standardized for this project using gitignore.io to support various development platforms. To get the project up and running:

• Clone this repo

Virtual environment

• Use virtualenv to create your virtual environment; virtualenv venv
• Activate the virtual environment; source venv/bin/activate
• Install feinstaub; pip install git+https://github.com/opendata-stuttgart/feinstaub-api
• Install the requirements; pip install .
• Create a sensorsafrica database with the following sql script:

CREATE DATABASE sensorsafrica;
CREATE USER sensorsafrica WITH ENCRYPTED PASSWORD 'sensorsafrica';
GRANT ALL PRIVILEGES ON DATABASE sensorsafrica TO sensorsafrica;

• Migrate the database; python manage.py migrate

• Run the server; python manage.py runserver

• Create super user for admin login; python manage.py createsuperuser

  username: <username> email: blank password: <password>

Docker

Using docker compose:

• Create a .env file using .env.template . docker-compose has some default values for these variables
• Build the project; docker-compose build or make build
• Run the project; docker-compose up -d or make up

Docker compose make commands:

• make build
• make up - run docker and detach
• make log - tail logs
• make test - run test
• make migrate - migrate database
• make createsuperuser - create a super user for admin
• make compilescss
• make enter - enter docker shell
• make django - enter docker django shell

NOTE: docker-compose is strictly for development and testing purposes. The Dockerfile is written for production since dokku is being used and it will look for Dockerfile.

Tests

Tests use SQLite by default so no external database is needed:

pytest -q

Run with verbose output:

pytest -v

Running tests with PostgreSQL (optional)

For full compatibility, you can run tests against PostgreSQL. Start a PostgreSQL container:

docker run -d --name test-pg \
  -e POSTGRES_USER=sensorsafrica \
  -e POSTGRES_PASSWORD=sensorsafrica \
  -e POSTGRES_DB=sensorsafrica \
  -p 5432:5432 postgres:11

Then run tests pointing to it:

SENSORSAFRICA_TEST_DATABASE_URL=postgres://sensorsafrica:sensorsafrica@localhost:5432/sensorsafrica pytest -q

Running tests with act (GitHub Actions locally)

act -j test

Tests marked with @pytest.mark.postgres_only are automatically skipped on SQLite.

Running tests against PostgreSQL (for full compatibility)

NOTE: If entrypoint and start scripts are changed, make sure they have correct/required permissions since we don't grant permissions to the files using the Dockerfile. Run the commands:

chmod +x contrib/entrypoint.sh
chmod +x contrib/start.sh

Deployment

Dokku

On your local machine run:

git remote add dokku dokku@dokku.me:sensorsafrica-api
git push dokku master

For more information read Deploying to Dokku.

GitHub Actions CI/CD

A GitHub Actions workflow now handles PR staging deploys and production deploys.

• Pull requests targeting master build a Docker image tagged as beta-pr-<PR># and deploy it to staging.
• Pushes to master inspect the latest Docker Hub image tags, bump the next production semantic version, push it to DockerHub, and deploy it to production with Dokku using docker-image:from.

Required repository secrets:

• DOCKERHUB_USERNAME — DockerHub username
• DOCKERHUB_TOKEN — DockerHub access token or password
• DOCKERHUB_REPOSITORY — optional DockerHub repository name, e.g. codeforafrica/sensors-africa-api
• DOKKU_SSH_STAGING_PRIVATE_KEY — SSH private deploy key for the staging Dokku server
• STAGING_DOKKU_HOST — staging Dokku hostname/IP
• STAGING_APP_NAME — staging Dokku app name
• DOKKU_SSH_PRIVATE_KEY — SSH private deploy key for the production Dokku server
• PRODUCTION_DOKKU_HOST — production Dokku hostname
• PRODUCTION_APP_NAME — production Dokku app name

The workflow checks the latest semantic Docker image tags on Docker Hub, bumps the patch version for the next production release, and deploys using tags like v0.1.1.

Staging deploys

Staging deploys use a beta tag derived from the PR number to keep deployments isolated and easy to trace.

Production deploys

Production deploys use the next semantic version derived from the latest Docker Hub tag and deploy the image to Dokku using that versioned Docker tag.

Notes

If your DockerHub repo is private, ensure the Dokku server can pull private images from DockerHub.

Cronjob

This project uses celery to create cronjobs and flower to monitor the cron jobs as a web admin. To create your jobs, add the task to the tasks.py and CELERY_BEAT_SCHEDULE in settings.py.

Everything starts automatically as setup in the contrib/start.sh:

celery -A sensorsafrica beat -l info &> /src/logs/celery.log  &
celery -A sensorsafrica worker -l info &> /src/logs/celery.log  &
celery -A sensorsafrica flower --basic_auth=$SENSORSAFRICA_FLOWER_ADMIN_USERNAME:$SENSORSAFRICA_FLOWER_ADMIN_PASSWORD &> /src/logs/celery.log  &

Note: If you run the project in the virtualenv you will have to start rabbitmq and pass that link to settings by the env variable SENSORSAFRICA_RABBITMQ_URL

Monitoring

Flower

It starts up in in the contrib/start.sh:

...
celery -A sensorsafrica flower --basic_auth=$SENSORSAFRICA_FLOWER_ADMIN_USERNAME:$SENSORSAFRICA_FLOWER_ADMIN_PASSWORD &> /src/logs/celery.log  &

Slack

Provide channel webhook as an enivronment variable SENSORSAFRICA_CELERY_SLACK_WEBHOOK. The default options are used:

DEFAULT_OPTIONS = {
    "slack_beat_init_color": "#FFCC2B",
    "slack_broker_connect_color": "#36A64F",
    "slack_broker_disconnect_color": "#D00001",
    "slack_celery_startup_color": "#FFCC2B",
    "slack_celery_shutdown_color": "#660033",
    "slack_task_prerun_color": "#D3D3D3",
    "slack_task_success_color": "#36A64F",
    "slack_task_failure_color": "#D00001",
    "slack_request_timeout": 1,
    "flower_base_url": None,
    "show_celery_hostname": False,
    "show_task_id": True,
    "show_task_execution_time": True,
    "show_task_args": True,
    "show_task_kwargs": True,
    "show_task_exception_info": True,
    "show_task_return_value": True,
    "show_task_prerun": False,
    "show_startup": True,
    "show_shutdown": True,
    "show_beat": True,
    "show_broker": False,
    "use_fixed_width": True,
    "include_tasks": None,
    "exclude_tasks": None,
    "failures_only": False,
    "webhook": None,
    "beat_schedule": None,
    "beat_show_full_task_path": False,
}

Sentry

Set the enivronment variable SENSORSAFRICA_SENTRY_DSN.

Archiving

Archives are sent to CKAN and require environment configuration:

- CKAN_ARCHIVE_API_KEY=..
- CKAN_ARCHIVE_OWNER_ID=...
- CKAN_ARCHIVE_URL=<url that supports bulk uploads>

Contributing

opendata-stuttgart/feinstaub-api prefer generating and applying migration to the database at the point of deployment (probably to reduce the number of changes to be applied). We, on the other hand, prefer the Django recommended approach of creating and reviewing migration files at the development time, and then applying the same migration files to different environments; dev, staging and eventually production.

Hence, with any contribution, include both sensors.AFRICA-api and opendata-stuttgart/feinstaub-api migration files by running python manage.py makemigrations command before creating a PR.

License

GNU GPLv3

Copyright (C) 2018 Code for Africa

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.