glue-utils

A lightweight Python toolkit for developing and testing AWS Glue jobs locally at no cost. It wraps common Glue and Spark utilities so you can iterate quickly in a Docker-based environment — without spinning up real Glue resources or incurring AWS charges.

Key capabilities:

• Run Glue job logic locally using a containerised Spark session
• Reusable helpers for Glue job setup (glue_utils.glue) and Spark transformations (glue_utils.spark)
• Fast feedback loop via local unit tests and an optional Jupyter Notebook interface

How to Use

Module	Purpose
glue_utils.glue	Glue job setup and context helpers
glue_utils.spark	Spark read/write data and transformation utilities

Local Development Guide

Prerequisites

• Docker
• docker-compose

Step 1: Build the Docker service

docker-compose build glue-dev

Step 2: Run options

# lint
docker-compose run glue-dev -c "inv lint"

# Auto fix lint
docker-compose run glue-dev -c "inv lint-fix"

# test
docker-compose run glue-dev -c "inv test"

# run command line in container
docker-compose run glue-dev

Interact with Jupyter Notebook

To interact with Jupyter Notebook and access AWS resources (e.g. read/write data in S3), you may need to log in to AWS locally. We recommend setting up AWS login with the following steps:

Recommended setup for AWS login

1. ~/.aws/config - example

   [sso-session sha-personal]
   sso_start_url = <sso sign-in url>
   sso_region = eu-west-2
   sso_registration_scopes = sso:account:access
   
   [profile sha-personal-etl-user]
   sso_session = sha-personal
   sso_account_id = <aws account id>
   sso_role_name = <sso role>
   region = eu-west-2
   

2. ~/.zshrc - add customised function to assist login

   aws_login_sso() {
       if [ -z "$1" ]; then
           echo "Usage: aws_login_sso <profile>"
           return 1
       fi
   
       aws sso login --sso-session $(aws configure get sso_session --profile $1) 
       export AWS_PROFILE="$1"
   
       echo "successfully login $1, caller identity:"
       aws sts get-caller-identity --query Arn --output text
       eval "$(aws configure export-credentials --profile $1 --format env)"
   }

3. Run AWS login command in terminal

   aws_login_sso sha-personal-etl-user

Start Jupyter Notebook

1. Run command in terminal

docker-compose run glue-dev -c "jupyter notebook"

2. After starting the notebook, copy and paste the URL provided in the terminal output to access the notebook.