Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 0 additions & 34 deletions .circleci/config.yml

This file was deleted.

10 changes: 10 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
.git
.github
.idea
.gradle
build
*.md
.gitignore
.circleci
example
LICENSE.txt
48 changes: 48 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: CI

on:
push:
branches:
- develop
- main
pull_request:
branches:
- develop
- main

jobs:
build:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Set up JDK 8
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '8'
cache: 'gradle'

- name: Grant execute permission for gradlew
run: chmod +x gradlew

- name: Run tests
run: ./gradlew test
env:
TERM: dumb
JVM_OPTS: -Xmx3200m

- name: Build with Gradle
run: ./gradlew publish
env:
TERM: dumb
JVM_OPTS: -Xmx3200m

- name: Upload build artifacts
uses: actions/upload-artifact@v4
with:
name: build-artifacts
path: build/repo
retention-days: 7
22 changes: 22 additions & 0 deletions .github/workflows/git-pr-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: git-pr-release

on:
push:
branches:
- develop

jobs:
git-pr-release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0

- name: git-pr-release
uses: bakunyo/git-pr-release-action@master
env:
GITHUB_TOKEN: ${{ github.token }}
TZ: Asia/Tokyo
GIT_PR_RELEASE_BRANCH_PRODUCTION: main
GIT_PR_RELEASE_BRANCH_STAGING: develop
58 changes: 58 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
name: Release

on:
push:
tags:
- 'v*'

jobs:
build:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Check if tag is on main branch
run: |
if ! git branch -r --contains ${{ github.ref }} | grep -q 'origin/main'; then
echo "Error: Tag must be created from main branch"
exit 1
fi

- name: Set up JDK 8
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '8'
cache: 'gradle'

- name: Grant execute permission for gradlew
run: chmod +x gradlew

- name: Run tests
run: ./gradlew test
env:
TERM: dumb
JVM_OPTS: -Xmx3200m

- name: Build with Gradle
run: ./gradlew build publish
env:
TERM: dumb
JVM_OPTS: -Xmx3200m

- name: Create GitHub Release
uses: softprops/action-gh-release@v1
with:
files: build/libs/*.jar
generate_release_notes: true
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Trigger JitPack build
run: |
VERSION=${GITHUB_REF#refs/tags/}
curl -s "https://jitpack.io/com/github/Gunosy/digdag-operator-athena/${VERSION}/build.log" || echo "JitPack build triggered"
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@
* [New Feature] Add `workgroup` option.
* [Breaking Change - `athena.query`] Remove the `output` option as the deprecation is notified from before.
* [Deprecated - `athena.ctas>`] Make `select_query` deprecated.
* [Note] Introduce `pro.civitaspo.digdag.plugin.athena.aws` package to divide dependencies about aws.
* [Note] Introduce `com.github.Gunosy.digdag.plugin.athena.aws` package to divide dependencies about aws.
* [Note] Use the Intellij formatter instead of spotless, so remove spotless from CI.


Expand Down
60 changes: 60 additions & 0 deletions DOCKER.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# Docker Usage

## Build and Test with Docker Compose

### Build the project
```bash
docker-compose run --rm build
```

### Run tests
```bash
docker-compose run --rm test
```

### Build Docker image only
```bash
docker-compose build
```

### Clean up
```bash
docker-compose down -v
```

## Manual Docker Commands

### Build image
```bash
docker build -t digdag-operator-athena .
```

### Run build
```bash
docker run --rm -v $(pwd)/build:/app/build digdag-operator-athena ./gradlew publish
```

### Run tests
```bash
docker run --rm -v $(pwd)/build:/app/build digdag-operator-athena ./gradlew test
```

## Release to JitPack

### Create and push a new tag
```bash
# Create a new tag (e.g., v0.4.1)
git tag v0.4.1

# Push the tag to GitHub
git push origin v0.4.1
```

This will automatically:
1. Trigger the GitHub Actions release workflow
2. Build the project
3. Create a GitHub Release with artifacts
4. Trigger JitPack to build and publish the package

### Check JitPack build status
Visit: https://jitpack.io/#com.github.Gunosy/digdag-operator-athena
20 changes: 20 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
FROM eclipse-temurin:8-jdk

WORKDIR /app

# Copy Gradle wrapper and configuration files
COPY gradlew gradlew.bat ./
COPY gradle ./gradle
COPY build.gradle settings.gradle ./

# Copy source code
COPY src ./src

# Make gradlew executable
RUN chmod +x gradlew

# Set environment variables
ENV JVM_OPTS="-Xmx3200m"
ENV TERM="dumb"

CMD ["./gradlew", "build"]
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# digdag-operator-athena
[![Jitpack](https://jitpack.io/v/pro.civitaspo/digdag-operator-athena.svg)](https://jitpack.io/#pro.civitaspo/digdag-operator-athena) [![CircleCI](https://circleci.com/gh/civitaspo/digdag-operator-athena.svg?style=shield)](https://circleci.com/gh/civitaspo/digdag-operator-athena) [![Digdag](https://img.shields.io/badge/digdag-v0.9.27-brightgreen.svg)](https://github.com/treasure-data/digdag/releases/tag/v0.9.27)
[![Jitpack](https://jitpack.io/v/com.github.Gunosy/digdag-operator-athena.svg)](https://jitpack.io/#com.github.Gunosy/digdag-operator-athena) [![Digdag](https://img.shields.io/badge/digdag-v0.9.27-brightgreen.svg)](https://github.com/treasure-data/digdag/releases/tag/v0.9.27)

digdag plugin for operating a query on athena.

Expand All @@ -15,7 +15,7 @@ _export:
repositories:
- https://jitpack.io
dependencies:
- pro.civitaspo:digdag-operator-athena:0.4.0
- com.github.Gunosy:digdag-operator-athena:0.4.1
athena:
auth_method: profile

Expand Down
10 changes: 3 additions & 7 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -4,20 +4,16 @@ plugins {
id 'com.github.johnrengelman.shadow' version '5.1.0'
}

group = 'pro.civitaspo'
version = '0.4.0'
group = 'com.github.Gunosy'
version = '0.4.1'

def digdagVersion = '0.9.39'
def digdagVersion = '0.9.42'
def awsSdkVersion = "1.11.653"
def scalaSemanticVersion = "2.13.0"
def depScalaVersion = "2.13"

repositories {
mavenCentral()
jcenter()
maven {
url 'https://dl.bintray.com/digdag/maven'
}
}

dependencies {
Expand Down
28 changes: 28 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@

services:
build:
build:
context: .
dockerfile: Dockerfile
volumes:
- ./build:/app/build
- gradle-cache:/root/.gradle
environment:
- JVM_OPTS=-Xmx3200m
- TERM=dumb
command: ./gradlew publish

test:
build:
context: .
dockerfile: Dockerfile
volumes:
- ./build:/app/build
- gradle-cache:/root/.gradle
environment:
- JVM_OPTS=-Xmx3200m
- TERM=dumb
command: ./gradlew test

volumes:
gradle-cache:
2 changes: 1 addition & 1 deletion example/example.dig
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ _export:
- file://${repos}
# - https://jitpack.io
dependencies:
- pro.civitaspo:digdag-operator-athena:0.4.0
- com.github.Gunosy:digdag-operator-athena:0.4.1
athena:
auth_method: profile
value: 5
Expand Down
2 changes: 1 addition & 1 deletion src/main/resources/META-INF/services/io.digdag.spi.Plugin
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
pro.civitaspo.digdag.plugin.athena.AthenaPlugin
com.github.Gunosy.digdag.plugin.athena.AthenaPlugin

Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
package pro.civitaspo.digdag.plugin.athena
package com.github.Gunosy.digdag.plugin.athena


import com.typesafe.scalalogging.LazyLogging
import io.digdag.client.config.{Config, ConfigFactory}
import io.digdag.spi.{OperatorContext, SecretProvider, TemplateEngine}
import io.digdag.util.{BaseOperator, DurationParam}
import pro.civitaspo.digdag.plugin.athena.aws.{Aws, AwsConf}
import com.github.Gunosy.digdag.plugin.athena.aws.{Aws, AwsConf}


abstract class AbstractAthenaOperator(operatorName: String,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package pro.civitaspo.digdag.plugin.athena
package com.github.Gunosy.digdag.plugin.athena


import java.lang.reflect.Constructor
Expand All @@ -7,17 +7,17 @@ import java.util.{Arrays => JArrays, List => JList}
import io.digdag.client.config.{Config, ConfigException}
import io.digdag.spi.{Operator, OperatorContext, OperatorFactory, OperatorProvider, Plugin, TemplateEngine}
import javax.inject.Inject
import pro.civitaspo.digdag.plugin.athena.add_partition.AthenaAddPartitionOperator
import pro.civitaspo.digdag.plugin.athena.apas.{AthenaApasOperator, AthenaDiffSchemaInternalOperator}
import pro.civitaspo.digdag.plugin.athena.ctas.AthenaCtasOperator
import pro.civitaspo.digdag.plugin.athena.drop_partition.AthenaDropPartitionOperator
import pro.civitaspo.digdag.plugin.athena.drop_table.AthenaDropTableOperator
import pro.civitaspo.digdag.plugin.athena.drop_table_multi.AthenaDropTableMultiOperator
import pro.civitaspo.digdag.plugin.athena.each_database.AthenaEachDatabaseOperator
import pro.civitaspo.digdag.plugin.athena.partition_exists.AthenaPartitionExistsOperator
import pro.civitaspo.digdag.plugin.athena.preview.AthenaPreviewOperator
import pro.civitaspo.digdag.plugin.athena.query.AthenaQueryOperator
import pro.civitaspo.digdag.plugin.athena.table_exists.AthenaTableExistsOperator
import com.github.Gunosy.digdag.plugin.athena.add_partition.AthenaAddPartitionOperator
import com.github.Gunosy.digdag.plugin.athena.apas.{AthenaApasOperator, AthenaDiffSchemaInternalOperator}
import com.github.Gunosy.digdag.plugin.athena.ctas.AthenaCtasOperator
import com.github.Gunosy.digdag.plugin.athena.drop_partition.AthenaDropPartitionOperator
import com.github.Gunosy.digdag.plugin.athena.drop_table.AthenaDropTableOperator
import com.github.Gunosy.digdag.plugin.athena.drop_table_multi.AthenaDropTableMultiOperator
import com.github.Gunosy.digdag.plugin.athena.each_database.AthenaEachDatabaseOperator
import com.github.Gunosy.digdag.plugin.athena.partition_exists.AthenaPartitionExistsOperator
import com.github.Gunosy.digdag.plugin.athena.preview.AthenaPreviewOperator
import com.github.Gunosy.digdag.plugin.athena.query.AthenaQueryOperator
import com.github.Gunosy.digdag.plugin.athena.table_exists.AthenaTableExistsOperator


object AthenaPlugin
Expand Down
Loading
Loading