anansi

A change data capture (CDC) for MongoDB and RabbitMQ (LavinMQ)

Why

I currently maintain a NodeJS project that uses PNPM to manage a workspace with about 11 services.

The codebase uses HTTP and RabbitMQ to pass message among themselves. As the codebase grows, we catch ourselves doing double write.

Take for example, we want to update an order status and dispatch a notification about the change. Currently, we make the database update and then dispatch the message to RabbitMQ which routes the message to the appropate service, if the database update succeeds.

This flow introduce a point of failure that may lead to message lost; How you may ask, if the database update is successfull but for any reason the RabbitMQ server is down, the message is lost forever.

This is what this project solves.

How

To resolve the double write issue, using our order example. The optimal approach is to use the Outbox pattern.

The outbox approach involves keeping another MongoDB collection order_notification_stream that takes the message that should be sent to RabbitMQ. The order update and the message will be writen to the database in a database transaction.

To catch the Message, Anansi will be listen for any insert operation to the order_notification_stream collection using MongoDB change stream; whenever there is a new insert ops. Anansi picks it up and route it to RabbitMQ.

If for any reason, Anansi is down, when anansi comes back online, Anansi checks the if there any message that has not be proccess and picks off from there. No message(s) are lost.

In SQL and Kafka stack, Debezium is definately the best option to go for.

We use MongoDB and RabbitMQ in our stack.

A detailed write up about this project can be found in the docs/adr/about.md

Nomenclatures

The name Anansi (Ananse) comes from the West-Africa (Ghana) folklore. Anansi is the keeper of stories. See here - Britannica and here - Wikipedia

Architecture Decisions

The decisions that led to some architecture patterns are documented in the ARDs Directory

Lessons and Gotchas

The lessons I learnt and the gotchas I encoutered are documented in the ./docs/lessons/ directory

RoadMap and Task

I maintian a list of task that help guild me on what is been worked on and what has been accomplished RoadMap and Tasks

Enviromental Variables

For development, .env is used to maintain envronmental configuration.

.env for Docker set up and ./config/app.env for app. See the corresponding .env.example for the content of this files

Running the app

for dev enviroment

• Start MongoDb with docker. You don't need this if you have MongoDb install locally

docker compose -f docker-compose.dev.yaml up --build

• Run the app

go run cmd/main.go

• Stop MongoDB after running

docker compose -f docker-compose.dev.yaml down`

  anansi:
    container_name: anansi-dev
    build:
      context: ./
      dockerfile: ./Dockerfile.dev
      args:
        - APP_NAME=anansi
    command: go run cmd/main.go
    environment:
      - MONGO_NAME=${MONGO_NAME}
      - MONGO_USER=${MONGO_USER}
      - MONGO_PASSWORD=${MONGO_PASSWORD}
      - MONGO_HOST=${MONGO_HOST}
    ports:
      - '3012:3012'
    depends_on:
      mongodb:
          condition: service_healthy
    volumes:
      - .:/srv
    restart: unless-stopped
    networks:
      - anansi-network