Add backend documentation and implement SSE authentication via API key query parameters

[Copilot]

This PR addresses the need for comprehensive backend documentation and aligns Server-Sent Events (SSE) authentication between the backend and chat frontend to consistently use API key authentication via query parameters.

Changes Made

1. Backend Documentation (backend/README.md)

Created comprehensive documentation covering:

• Architecture overview: Express, DynamoDB, SSE, BullMQ, and file uploads
• Quickstart guide: Local development and Docker Compose setup
• Environment variables: Required AWS, JWT, Twilio, and Stripe configuration
• Routing map: Complete breakdown of all API endpoints and domain routes
• Data model: DynamoDB table structure and model documentation
• Chat/SSE endpoints: Real-time messaging and streaming event documentation
• Security & authentication: CORS restrictions, JWT middleware, and API key usage
• Development workflow: Testing, useful paths, and contribution guidelines

2. SSE Authentication Implementation (backend/server.js)

Enhanced SSE endpoints to require proper authentication:

• New withSseAuth wrapper: Handles API key authentication for SSE connections
• Query parameter support: Accepts API keys via x-api-key or api_key query parameters (required because EventSource doesn't support custom headers)
• Middleware integration: Injects query parameters into headers for existing auth middleware validation
• Proper error handling: Returns 401 Unauthorized for invalid/missing API keys
• Backward compatibility: Maintains support for existing api_key parameter while preferring x-api-key

3. Frontend Alignment (frontend/chat-ui/src/components/ChatWindow.jsx)

Updated documentation to reflect the new authentication flow:

• Comment updates: Clarified that the component uses the centralized stream() helper which handles authentication
• No functional changes needed: The component was already correctly using the centralized API helper

Authentication Flow

The implementation ensures secure SSE connections while maintaining EventSource compatibility:

// Backend: Accept API key via query param and inject into headers
function withSseAuth(handler) {
  return (req, res) => {
    const qKey = req.query['x-api-key'] || req.query['api_key'];
    if (qKey && !req.headers['x-api-key']) {
      req.headers['x-api-key'] = qKey;
    }
    authMiddleware(req, res, (err) => {
      if (err) {
        res.status(401).end();
        return;
      }
      handler(req, res);
    });
  };
}

// Frontend: Centralized stream() helper automatically appends API key
export function stream(path) {
  let url = buildUrl(path);
  if (API_KEY) {
    const hasQuery = url.includes('?');
    url = `${url}${hasQuery ? '&' : '?'}x-api-key=${encodeURIComponent(API_KEY)}`;
  }
  return new EventSource(url);
}

Testing

Added comprehensive test coverage for the new authentication logic:

• API key injection from x-api-key query parameter
• Backward compatibility with api_key query parameter
• Precedence handling when both parameters are provided
• Existing header preservation

All existing tests continue to pass, ensuring no regressions were introduced.

Security Benefits

• Authenticated SSE connections: Prevents unauthorized access to real-time chat streams
• Consistent authentication: Aligns SSE auth with existing API endpoint patterns
• Backward compatibility: Existing integrations continue to work without modification
• Proper error handling: Clear 401 responses for debugging authentication issues

This implementation provides a secure foundation for real-time chat features while maintaining the flexibility needed for EventSource-based connections.

This pull request was created as a result of the following prompt from Copilot chat.

Summary
Add a comprehensive backend/README.md explaining the backend architecture and setup. Align the Server‑Sent Events (SSE) authentication flow between the backend and the chat frontend so both consistently use an API key passed via query parameter for SSE connections. Ensure backward compatibility with any existing "api_key" usage while standardizing on "x-api-key".

Scope of work

1. Documentation: backend/README.md

• Create a new file backend/README.md with the essentials:
  • Overview of architecture (Express, DynamoDB, SSE, BullMQ, uploads)
  • Quickstart and Docker Compose
  • Environment variables
  • Route map and domain breakdown
  • Data model (DynamoDB tables)
  • Chat/SSE endpoints and events
  • Transactions & notifications
  • Auth and security notes (CORS + JWT/API key usage)
  • Jobs & SMS
  • Useful paths and testing notes

Use the exact content below for backend/README.md:

# Fruitful Backend (Agrinet Platform)

This backend powers Agrinet services using Node.js, Express, AWS DynamoDB, and Server‑Sent Events (SSE) for streaming chat updates.

## Overview

- Runtime: Node.js + Express
- Data: AWS DynamoDB (DocumentClient)
- Streaming: SSE (`/events`, `/stream/:conversationId`)
- Auth: JWT middleware; API Key support for some endpoints
- Queues: BullMQ for SMS and background jobs
- Uploads: Files stored under `backend/uploads` served at `/uploads`

## Quickstart

```bash
cd backend
npm install
node server.js
```

Docker-based local dev with DynamoDB Local:
```bash
docker compose up --build
```

The API typically runs on port 5000 (see docker-compose).

## Environment

Required (or set via `.env`):
- AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, DYNAMODB_ENDPOINT (for local)
- JWT_SECRET
- TWILIO_SID, TWILIO_AUTH_TOKEN, TWILIO_FROM_NUMBER (for SMS; optional TWILIO_STATUS_CALLBACK_URL)
- STRIPE_KEY (if deposits enabled)

## Routing

Main mounts (see `server.js`):
- `/health` – health check
- `/uploads/*` – static uploads
- `/events` – SSE broadcast channel
- `/stream/:conversationId` – SSE per‑conversation stream

Domain routes (when not in minimal mode):
- `/` → `routes/api.js` (composed transaction + notification)
- `/api/auth` → auth routes
- `/api/keys` → key management
- `/api/contracts` → contracts
- `/api/admin` → admin
- `/api/marketplace` → marketplace
- `/users` → user management
- `/products` → products
- `/broadcast` → broadcasts
- `/sms` → SMS webhooks
- `/cart`, `/orders`, `/subscriptions`
- `/conversations`, `/messages`
- `/inventory`, `/api/location`
- `/federation`, `/trends`
- `/deposit` (optional; guarded by auth)

## Data Model (DynamoDB tables)

- `Users`, `Keys`
- `Listings`, `Contracts`, `Transactions`
- `Conversations`, `Messages`
- `Notifications`
- `Inventory`
- `AgriculturalData` (for price/info lookups)

Each model file exports a `*_TABLE_NAME` and an item builder, e.g.:
- `models/user.js` → `USER_TABLE_NAME`, `createUserItem()`
- `models/transaction.js` → `TRANSACTION_TABLE_NAME`, `createTransactionItem()`

## Chat & Streaming

Endpoints (used by the Chat UI):
- `GET /conversations` → list
- `POST /conversations` → create
- `PUT /conversations/:id` → rename
- `POST /conversations/:id/pin` → toggle pin
- `DELETE /conversations/:id` → delete
- `GET /messages/:conversationId` → list messages
- `POST /messages/:conversationId` → send message (optionally with file)
- `GET /stream/:conversationId` (SSE) → events:
  - `token`: `{ id, token }`
  - `message`: `{ message }`

Server emitters (global):
- `emitToken(conversationId, id, token)`
- `emitMessage(conversationId, message)`

## Transactions & Notifications

- `POST /api/transactions`:
  - Writes item to `Transactions`
  - Writes `Notifications` item for buyer
  - Enqueues a “ping” job
  - May broadcast SSE events

## Security & Auth

- CORS: restricted to `https://www.ntari.org`
- JWT: middleware enforces authorization on protected routes
- API Key for SSE: the Chat UI passes an API key as `x-api-key` query parameter for SSE requests; the server accepts `x-api-key` (and also `api_key` for backward compatibility) on `/events` and `/stream/:conversationId`.

## Jobs & SMS

- `bull/smsQueue.js`: BullMQ worker that sends SMS via Twilio
- `routes/smsRoutes.js`: webhooks for incoming/status
- Background workers in Docker Compose:
  - `federation-sync`, `key-expiry-cleaner`

## Useful Paths

- Entry: `backend/server.js`
- Models: `backend/models/*`
- Routes: `backend/routes/*`
- Utils: `backend/utils/*`
- Queues: `backend/bull/*`
- Uploads: `backend/uploads`

## Testing

See `backend/package.json` for Jest config and tests. Add route/model tests under `__tests__/`.

---
Contributions welcome! Please keep `server.js` slim by adding routes and logic in domain folders.

2. Backend code: authenticate SSE via API key query param (align auth)

• File: backend/server.js
• For both SSE endpoints (GET /events and GET /stream/:conversationId):
  • Read API key from either req.query['x-api-key'] or req.query['api_key'].
  • If present, inject it into req.headers['x-api-key'] so existing auth middleware can validate it.
  • Call the authMiddleware for these handlers explicitly before establishing the SSE stream. If unauthorized, respond with 401 and close the connection. Keep health and CORS unchanged.
  • Leave backward compatibility for api_key query parameter.
  • Add inline comments documenting the behavior.

Implementation sketch:

function withSseAuth(handler) {
  return (req, res) => {
    // Accept API key via query param for SSE (no custom headers on EventSource)
    const qKey = req.query['x-api-key'] || req.query['api_key'];
    if (qKey && !req.headers['x-api-key']) req.headers['x-api-key'] = qKey;
    authMiddleware(req, res, (err) => {
      if (err) {
        res.status(401).end();
        return;
      }
      handler(req, res);
    });
  };
}

app.get('/events', withSseAuth((req, res) => { /* existing SSE setup */ }));
app.get('/stream/:conversationId', withSseAuth((req, res) => { /* existing SSE setup */ }));

Keep the existing SSE logic intact otherwise (clients sets, cleanup on close, etc.).

3. Frontend chat: unify SSE client usage and query param

• File: frontend/chat-ui/src/components/ChatWindow.jsx
• Replace the manual EventSource URL construction with the central stream() helper from src/api.js, which appends the x-api-key query parameter. Remove usage of the api_key parameter in this component.
• Import stream from ../api and call const events = stream(/stream/${id});.
• Leave the existing event handlers (token, message) unchanged.

Acceptance criteria

• A new backend/README.md is added with the content above.
• Opening the chat UI with VITE_API_KEY set connects to /stream/:conversationId via SSE and is authorized by the backend reading the x-api-key query param.
• If no API key (or invalid) is provided, SSE endpoints return a 401 (connection fails), without affecting /health.
• ChatWindow.jsx uses the centralized stream() helper and no longer hardcodes api_key.
• Backwards compatibility: server continues to accept api_key query param as an alias.

Notes

• Do not alter CORS config or other route mounts.
• Keep the code style consistent with existing files and add comments where behavior changes.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.