Document Processing Pipeline

A serverless AWS document-processing pipeline that accepts documents through an API, stores them in S3, processes them asynchronously with SQS and Lambda, extracts text, scans for keywords, and sends notifications through SNS.

The entire infrastructure is managed with Terraform.

Architecture

How It Works

1. A user uploads a document through the API.
2. API Gateway sends the request to the UploadProcessor Lambda.
3. UploadProcessor decodes the Base64 file and stores it in S3.
4. S3 sends an event to the SQS processing queue.
5. SQS triggers the DocumentParser Lambda.
6. DocumentParser retrieves the document from S3.
7. The parser extracts the document text and scans for keywords.
8. Processing results and errors are logged in CloudWatch.
9. When keywords are detected, SNS sends an email notification.
10. If processing fails, SQS retries the message.
11. After the maximum number of retries, the message is moved to the DLQ.
12. The ErrorHandler Lambda processes failed messages from the DLQ and sends an alert.

AWS Services

• API Gateway — Receives document upload requests.
• Lambda — Handles uploads, document processing, and failed messages.
• S3 — Stores uploaded documents.
• SQS — Decouples document uploads from processing and provides retries.
• SQS DLQ — Stores messages that repeatedly fail processing.
• SNS — Sends email notifications.
• CloudWatch — Provides logs, metrics, and alarms.
• IAM — Controls permissions between AWS services.

Supported Documents

• PDF
• DOCX

PDF text extraction is currently implemented in the document parser.

Keywords

The parser currently scans documents for:

urgent
fraud
invoice
contract

Deployment

Prerequisites

• AWS account
• AWS CLI configured
• Terraform
• Python 3.12+

Build

Build the DocumentParser Lambda package:

./build.sh

Deploy

terraform init
terraform plan
terraform apply

After deployment, Terraform outputs the API URL, S3 bucket, SQS queues, and CloudWatch log groups needed for testing.

Testing

Testing instructions are available in the test file in the the root folder