Skip to content

Development.md

Latest commit

 

History

History
110 lines (77 loc) · 4.56 KB

Development.md

File metadata and controls

110 lines (77 loc) · 4.56 KB
Raw

Development Guide

This guide describes how to build, run, test, lint, and statically analyze the first-party code in this repository. InstantX is primarily a deployment-and-integration project: an off-the-shelf broker/observability stack orchestrated with docker-compose, glued together by first-party Python services and scripts plus an Apache NiFi flow.

First-party Python lives in:

Build

The full platform is built and run with Docker Compose. All commands run from deployment/:

# Build images and start the stack
docker-compose up --build -d

# Start without rebuilding
docker-compose up -d

# Tear down
docker-compose down

The NiFi scripts package is a standard Poetry project:

cd deployment/nifi/nifi-scripts
poetry install            # installs runtime + dev dependencies

poetry.lock is committed, so poetry install is reproducible and GitHub's dependency graph can resolve exact versions for this directory. Requires Poetry 2.x (pipx install 'poetry>=2.0,<3.0'). After changing dependencies, run poetry lock and commit the result — CI's poetry-lock job runs poetry check --lock and fails if the two drift apart.

The package supports Python >=3.9, matching the runtime images (eventPublisher and kafka_speed_exporter both build on python:3.9-slim). Two dev tools have since dropped 3.9, so the dev group declares a version per interpreter range and you get different versions depending on your local Python:

Tool Python 3.9 Python 3.10+
pytest 8.4.x 9.1.x
bandit 1.8.x 1.9.x

Both are minor-version differences in the tooling only — they do not affect the code under test. CI runs ruff and bandit on Python 3.12 via a plain pip install, independently of Poetry.

The Event Publisher uses a pinned requirements.txt:

cd deployment/eventPublisher
pip install -r requirements.txt

All build and analysis tools used here are free/libre and open source (Docker, Poetry, pytest, ruff, bandit).

Test

The first-party automated test suite uses pytest and lives in the NiFi scripts package:

cd deployment/nifi/nifi-scripts
poetry run pytest                              # run the suite
poetry run pytest -s --cov=src --cov-report=xml  # with coverage

Testing policy

Major new functionality must be accompanied by automated tests, and bug fixes should add a regression test where practical. This policy is stated in CONTRIBUTING.md and reinforced by a checklist item in the pull request template.

Lint (ruff)

Python code is linted with ruff. The shared configuration lives in deployment/nifi/nifi-scripts/pyproject.toml under [tool.ruff].

Run it across all first-party Python from the repository root:

ruff check \
  deployment/eventPublisher \
  deployment/kafka_speed_exporter \
  deployment/nifi/nifi-scripts \
  --config deployment/nifi/nifi-scripts/pyproject.toml

ruff is available as a Poetry dev dependency in the NiFi scripts package (poetry run ruff ...) or as a standalone tool (pipx install ruff / uvx ruff).

Static analysis (bandit)

Python code is scanned for common security issues with bandit before each major release:

bandit -r \
  deployment/eventPublisher \
  deployment/kafka_speed_exporter \
  deployment/nifi/nifi-scripts/src

bandit is available as a Poetry dev dependency in the NiFi scripts package (poetry run bandit ...) or standalone (pipx install bandit). Findings of medium severity or higher are addressed (fixed or, where a finding is a false positive, suppressed with a justifying comment) before release.

Before opening a pull request

  1. poetry run pytest passes (add/update tests for your change).
  2. ruff check ... --config deployment/nifi/nifi-scripts/pyproject.toml reports no errors.
  3. bandit -r ... reports no unaddressed medium-or-higher findings.
  4. Update CHANGELOG.md for user-facing changes.