Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
159 commits
Select commit Hold shift + click to select a range
54581f2
updated config for compatibility with newest ump release
StefanSchuhart Jun 2, 2025
4b79797
bump chart version: 0.6.0 --> 0.7.0
StefanSchuhart Jun 2, 2025
5c7fb29
updated keycloak credentials
StefanSchuhart Jun 2, 2025
5b2a6d5
added geoserver credentials
StefanSchuhart Jun 2, 2025
ef45aa1
removed credentials
StefanSchuhart Jun 2, 2025
10cd414
fixed: comment
StefanSchuhart Jun 3, 2025
7ba1149
removed secret-based settings
StefanSchuhart Jul 2, 2025
9a5bbd2
added env from secrets
StefanSchuhart Jul 2, 2025
12f9f77
fixed env var naemes
StefanSchuhart Jul 2, 2025
59fa98b
fixed env var names
StefanSchuhart Jul 2, 2025
3283119
added scratch folder
StefanSchuhart Jul 3, 2025
816e707
made url prefix customizable
StefanSchuhart Jul 3, 2025
3e9ad61
removed
StefanSchuhart Jul 3, 2025
8476a32
added timeout
StefanSchuhart Jul 3, 2025
a3ea658
fixed missing engine, removed unused imports
StefanSchuhart Jul 3, 2025
6b20910
handle remote server errors gracefully
StefanSchuhart Jul 4, 2025
c535e46
added debug config
StefanSchuhart Jul 4, 2025
ab74638
added an rfc-like exception
StefanSchuhart Jul 4, 2025
25e1821
using the rfc-like exception
StefanSchuhart Jul 4, 2025
ea35856
added detailed log messages, reduced user facing details
StefanSchuhart Jul 4, 2025
e3d201d
renamed process to process_config
StefanSchuhart Jul 4, 2025
5d724e4
reworked adn fixed error handling, added (short) timeout when fetchin…
StefanSchuhart Jul 4, 2025
23cf2ee
fixed some formatting issues and replaced root logger
StefanSchuhart Jul 4, 2025
8241dc8
using ruff formatter instead of black
StefanSchuhart Jul 4, 2025
205f13b
added process details, renamed vars and func to refelct its purpose
StefanSchuhart Jul 4, 2025
ab13762
mainly formatting, added notes on top, renamed two methods
StefanSchuhart Jul 4, 2025
aacf80f
refactored start_process_execution and splitted it into smaller pieces
StefanSchuhart Jul 4, 2025
3cf1d62
creating job and extracting job id fixed, ware working now
StefanSchuhart Jul 4, 2025
8a81e9a
fixed some typing issues
StefanSchuhart Jul 7, 2025
9e774f5
added type hint
StefanSchuhart Jul 7, 2025
90e0d63
fixed job status update
StefanSchuhart Jul 7, 2025
7c9b9bf
added type hint
StefanSchuhart Jul 7, 2025
db1f506
added note
StefanSchuhart Jul 7, 2025
04caff7
splitted _wait_for_results into smaller helpers , improved error hand…
StefanSchuhart Jul 7, 2025
14c3821
fix: wrong config reference for workspace
StefanSchuhart Jul 7, 2025
cac9c03
raise exception with details page, fixed types
StefanSchuhart Jul 7, 2025
a3d1734
handle process not found error
StefanSchuhart Jul 7, 2025
63b1224
check_process_availability -> load_process_config
StefanSchuhart Jul 7, 2025
e3473e5
type added, some formatting issues resolved
StefanSchuhart Jul 7, 2025
dba2c69
fetch_response_content added
StefanSchuhart Jul 7, 2025
66a90d7
handling remote server responses (Bad request and the like)to give us…
StefanSchuhart Jul 7, 2025
65c70ae
rm unused import
StefanSchuhart Jul 7, 2025
ed6f2cb
added notes
StefanSchuhart Jul 7, 2025
ec5c1e4
fix: set status, dont set "successfully" manually, it MUST be fetrche…
StefanSchuhart Jul 7, 2025
aa0cc4a
reduce timeout to reasonable values (from default 300 seconds), use u…
StefanSchuhart Jul 7, 2025
d30a452
using parameter names
StefanSchuhart Jul 7, 2025
5c94394
added optional raise_for_status
StefanSchuhart Jul 7, 2025
cb2a22b
fixed some typing errors
StefanSchuhart Jul 7, 2025
f3c6e42
added new setting
StefanSchuhart Jul 7, 2025
ff0d9c4
using correct prefix setting
StefanSchuhart Jul 7, 2025
af09240
server url has schema now
StefanSchuhart Jul 7, 2025
27232d5
full links
StefanSchuhart Jul 7, 2025
d8d27fb
using url and prefix
StefanSchuhart Jul 7, 2025
78279a1
added clean url joining function
StefanSchuhart Jul 7, 2025
51970ed
creating clean urls
StefanSchuhart Jul 7, 2025
a9286cd
fixed results link
StefanSchuhart Jul 7, 2025
b131e21
fix: links fully working
StefanSchuhart Jul 7, 2025
c8f8daf
fix: missing jab.started datetime added
StefanSchuhart Jul 7, 2025
17331e6
added tag
StefanSchuhart Jul 7, 2025
1c0aacd
updated version: 2.0.0 -> 2.1.0.rc1+fix-error-handling
StefanSchuhart Jul 7, 2025
89a962e
updated entrypoint script
StefanSchuhart Jul 7, 2025
93b2786
bump version: 2.1.0rc2+fix-error-handling"
StefanSchuhart Jul 7, 2025
aac3397
Merge branch 'dev' of https://github.com/citysciencelab/urban-model-p…
StefanSchuhart Jul 8, 2025
cee5d37
added scratch
StefanSchuhart Jul 8, 2025
82b5da8
added bump-my-version for automatic version handling
StefanSchuhart Jul 8, 2025
144f98f
Merge branch 'fix/error-handling' of https://github.com/citysciencela…
StefanSchuhart Jul 8, 2025
0c3e545
removed values.yaml and Dockerfile
StefanSchuhart Jul 8, 2025
6d60c18
using IMAGE_TAG from .env
StefanSchuhart Jul 8, 2025
7797fd0
added version bump commands
StefanSchuhart Jul 8, 2025
c523087
renamed
StefanSchuhart Jul 8, 2025
1a4fcb5
moved to charts/
StefanSchuhart Jul 8, 2025
6bd9db2
updated charts version bump cmd
StefanSchuhart Jul 8, 2025
50d9e93
updated tag
StefanSchuhart Jul 8, 2025
86743bb
settings updated
StefanSchuhart Jul 8, 2025
b78850f
env var names updated
StefanSchuhart Jul 8, 2025
da7d71e
env var names updated
StefanSchuhart Jul 8, 2025
86b173b
added line break
StefanSchuhart Jul 8, 2025
313f03f
fixed command
StefanSchuhart Jul 8, 2025
6c43ebe
fixed current version
StefanSchuhart Jul 8, 2025
fdaf01c
fixed path
StefanSchuhart Jul 8, 2025
ffbde8c
ranmed files
StefanSchuhart Jul 8, 2025
85549f9
bump chart version: 0.7.0 -> 0.8.0
StefanSchuhart Jul 8, 2025
15bfb28
fix: newline
StefanSchuhart Jul 8, 2025
4a15ed7
bump chart version: 0.8.0 -> 0.8.1
StefanSchuhart Jul 8, 2025
7226435
empty dummy secrets
StefanSchuhart Jul 8, 2025
f412617
bump chart version: 0.8.1 -> 0.8.2
StefanSchuhart Jul 8, 2025
94974c5
fix: providersFilePath unified
StefanSchuhart Jul 8, 2025
c206cce
bump chart version: 0.8.2 -> 0.8.3
StefanSchuhart Jul 8, 2025
afb106b
deleted job and job related objects
StefanSchuhart Jul 9, 2025
9b9f0c4
added a dummy adress
StefanSchuhart Jul 9, 2025
165691b
version bump: 0.8.3 -> 0.9.0
StefanSchuhart Jul 9, 2025
4aec5c8
set flask app
StefanSchuhart Jul 9, 2025
0147dc9
bump chart version: 0.9.0 -> 0.9.1
StefanSchuhart Jul 9, 2025
42ce434
fixed: using mount path instead of file
StefanSchuhart Jul 9, 2025
f56d591
added mount path
StefanSchuhart Jul 9, 2025
183cfe8
added dummy values
StefanSchuhart Jul 9, 2025
a3c36c1
bump chart version: 0.9.1 -> 0.9.2
StefanSchuhart Jul 9, 2025
5738493
readyness probe must use dynamic prefix
StefanSchuhart Jul 9, 2025
7c3a41a
added missing backendRefs keys
StefanSchuhart Jul 9, 2025
2f5df09
bump chart version: 0.9.2 -> 0.9.3
StefanSchuhart Jul 9, 2025
5f07f0a
improved error details when job is not successful
StefanSchuhart Jul 10, 2025
83a71e9
updated version: 2.1.0rc2 -> 2.1.0rc3
StefanSchuhart Jul 10, 2025
57322c1
removed read-only from providers mount
StefanSchuhart Jul 14, 2025
6495a4f
updated chart version: 0.9.3 -> 0.9.4
StefanSchuhart Jul 14, 2025
046b4e3
Merge branch 'main' of https://github.com/citysciencelab/urban-model-…
StefanSchuhart Jul 15, 2025
47a6e64
added UMP_API_SERVER_URL_PREFIX
StefanSchuhart Jul 15, 2025
35383df
Merge pull request #99 from citysciencelab/fix/error-handling
StefanSchuhart Jul 15, 2025
5f80c3d
Merge branch 'dev' of https://github.com/citysciencelab/urban-model-p…
StefanSchuhart Jul 15, 2025
5674699
changed providers loader to listen on any event (k8s configmap issue)
StefanSchuhart Jul 15, 2025
4094f56
Merge pull request #102 from citysciencelab/charts
StefanSchuhart Jul 15, 2025
31c7840
raise ValueError when provider or process not found
StefanSchuhart Jul 15, 2025
927a529
"get_" instead of "load_"
StefanSchuhart Jul 15, 2025
ebae632
fix: guarantee that process.extend receives only real results, not ex…
StefanSchuhart Jul 23, 2025
a3a3c7b
added new UMP_API_SERVER_URL_PREFIX to control gunicorn --timeout set…
StefanSchuhart Jul 23, 2025
37d3c7c
fetch_json error handling and flow improved
StefanSchuhart Jul 23, 2025
5c77444
added a log message in case server response does not contain "processes"
StefanSchuhart Jul 23, 2025
ae1f21e
added type hints
StefanSchuhart Jul 23, 2025
bf9a814
raise_for_status must be false on clientsession to get a response in …
StefanSchuhart Jul 23, 2025
f9a3828
using fetch_json utility function here, too to make sure all possible…
StefanSchuhart Jul 23, 2025
314720b
formatting, note added
StefanSchuhart Jul 23, 2025
8eae699
log message for grants improved
StefanSchuhart Jul 23, 2025
53db8fa
renamed is_process_visible to has_user_access_rights
StefanSchuhart Jul 23, 2025
1812459
log the error, too
StefanSchuhart Jul 23, 2025
af0332b
version update: 2.1.0rc4 -> 2.1.0rc5
StefanSchuhart Jul 23, 2025
ae1229c
added missing "processes" endpoint
StefanSchuhart Jul 23, 2025
039ea3f
added field validator to ensure server_url ends with a "/"
StefanSchuhart Jul 23, 2025
72dbefa
samll fix of missing parenthesis, errorhandler deteails adjusted
StefanSchuhart Jul 29, 2025
56ef586
added note
StefanSchuhart Jul 29, 2025
b1b63c8
fixed return type
StefanSchuhart Jul 29, 2025
4242b29
improved error response and distinguish between "Unauthorized" and "F…
StefanSchuhart Jul 29, 2025
f7352aa
updated: 2.1.0rc5 -> 2.1.0rc6
StefanSchuhart Jul 29, 2025
f736da8
added note
StefanSchuhart Jul 29, 2025
1886541
fix: error message was wrongly placed
StefanSchuhart Jul 29, 2025
c3e0fe9
added bug note
StefanSchuhart Jul 29, 2025
00d40e3
version update: 2.1.0rc6 -> 2.1.0rc7
StefanSchuhart Jul 29, 2025
cd1f129
Merge pull request #103 from citysciencelab/feature/improve-provider-…
StefanSchuhart Jul 29, 2025
13d702c
added process.py logger
StefanSchuhart Jul 29, 2025
45b8073
fixed the hash algorithm (it failed when user_id) was actually set
StefanSchuhart Jul 29, 2025
e4d392c
added a note
StefanSchuhart Jul 29, 2025
9ff5e4b
fixed a problem when deterministic is false AND user_id is set
StefanSchuhart Jul 29, 2025
5020a7d
version update: 2.1.0rc8 -> 2.1.0rc9
StefanSchuhart Jul 29, 2025
6391dec
fixed a an order typo
StefanSchuhart Jul 29, 2025
df4b396
removed bug note, because the quick fix handled it
StefanSchuhart Jul 29, 2025
95818ec
Merge pull request #105 from citysciencelab/fix/remote-process-timeout
StefanSchuhart Jul 30, 2025
9bb1e7e
using setting UMP_KEYCLOAK_CLIENT_ID instead of hard-coded "ump-client"
StefanSchuhart Jul 30, 2025
a8c2e41
added notes
StefanSchuhart Jul 30, 2025
e346254
Correct deploy docs update
herzogrh Jul 30, 2025
be00387
Merge pull request #104 from citysciencelab/fix/not-authorized
StefanSchuhart Jul 31, 2025
dfa44a2
Merge pull request #106 from citysciencelab/fix/insert-failure-when-a…
StefanSchuhart Jul 31, 2025
1f576db
Merge pull request #107 from citysciencelab/fix/hard-coded-clientid
StefanSchuhart Jul 31, 2025
22e10b3
changelog updated
StefanSchuhart Jul 31, 2025
e817dd1
added UMP_SERVER_TIMEOUT setting
StefanSchuhart Jul 31, 2025
a60401c
added new UMP_SERVER_TIMEOUT setting to compose
StefanSchuhart Jul 31, 2025
6ba1275
version update: 2.1.0rc8 -> 2.1.0
StefanSchuhart Jul 31, 2025
8919dea
chart version update: 0.9.4 -> 0.9.5
StefanSchuhart Jul 31, 2025
8e906df
Merge pull request #108 from citysciencelab/changelog
herzogrh Aug 4, 2025
2bebc0f
Merge pull request #109 from citysciencelab/charts
herzogrh Aug 4, 2025
2825cee
added note
StefanSchuhart Aug 6, 2025
9f953d2
Merge branch 'dev' of https://github.com/citysciencelab/urban-model-p…
StefanSchuhart Aug 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .bumpversion.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
[bumpversion]
current_version = "2.1.0rc1+fix-error-handling"
commit = true
tag = true
tag_name = "v{new_version}"

[[bumpversion.files]]
filename = "pyproject.toml"
search = 'version = "{current_version}"'
replace = 'version = "{new_version}"'

[[bumpversion.files]]
filename = ".env"
search = 'IMAGE_TAG={current_version}'
replace = 'IMAGE_TAG={new_version}'
3 changes: 2 additions & 1 deletion .env.example
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
#---- App settings ----
# The API_SERVER_URL is only used to return the complete URL in the result of the job details as specified in OGC.
# Should be the base url to the api.

UMP_SERVER_TIMEOUT=30
UMP_LOG_LEVEL=DEBUG
UMP_PROVIDERS_FILE=providers.yaml
UMP_API_SERVER_URL=localhost:5000
UMP_API_SERVER_URL_PREFIX=/api
UMP_REMOTE_JOB_STATUS_REQUEST_INTERVAL=5
UMP_DATABASE_NAME=ump
UMP_DATABASE_HOST=localhost
Expand Down
60 changes: 0 additions & 60 deletions .github/workflows/deploy-docs.yml

This file was deleted.

1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -172,3 +172,4 @@ cython_debug/

geoserver_data
postgresql_data
scratch/*
26 changes: 25 additions & 1 deletion .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,31 @@
"version": "0.2.0",
"configurations": [
{
"name": "Launch api",
"name": "ump + db",
"type": "debugpy",
"request": "launch",
"module": "flask",
"envFile": "${workspaceFolder}/.env",
"env": {
"FLASK_APP": "src/ump/main.py",
"FLASK_DEBUG": "1"
},
"args": [
"--debug",
"run",
"--no-debugger",
"-p",
// "5005",
"${command:pickArgs}"
],
"jinja": true,
"autoStartBrowser": false,
"preLaunchTask": "start-postgis-container",
"postDebugTask": "stop-postgis-container",
"justMyCode": false,
},
{
"name": "ump + db + migrations",
"type": "debugpy",
"request": "launch",
"module": "flask",
Expand Down
3 changes: 2 additions & 1 deletion .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"--preview"
],
"[python]": {
"editor.defaultFormatter": "ms-python.black-formatter",
"editor.defaultFormatter": "charliermarsh.ruff",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.organizeImports": "explicit"
Expand All @@ -20,4 +20,5 @@
],
"ruff.importStrategy": "fromEnvironment",
"black-formatter.importStrategy": "fromEnvironment",
"python.analysis.typeCheckingMode": "basic",
}
37 changes: 37 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,43 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.1.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

# [2.x]
## [2.1.0] - 2025-07-31
### Changed:
- improved error handling when requesting remote servers processes and jobs
- helm chart is up-to-date with current UMP
- provider loader listens on any event to be compatible with configmap-updates in k8s
- provider loader improved: debouncing rapid file changes and atomic updates of providers object
- improved server responses in certain situations, especially when something went wrong showing users json information (as this is a json api) in accordance with OGC api process spec
- improved job starting mechanism

### Added:
- a new setting to control gunicorn worker timout was introduced: UMP_SERVER_TIMEOUT
- a new setting to control ump server path prefix introduced: UMP_API_SERVER_URL_PREFIX
- timeouts for all requests to remote servers

### Fixed:
- using setting UMP_KEYCLOAK_CLIENT_ID instead of hard-coded "ump-client"
- job insert queries failed when logged-in user created a job
- missing job metadata
- fetch correct job status from remote server

## [2.0.0] - 2025-06-25

### Added
- comprehensive documentation added
- unified database connection pool handling

### Changed
- created a providers pydantic class for better type safety and concise handling
- improved provider.yaml loading and provider updateing mechanism
- improved logging
- keycloak coinnection error handling improved

### Fixed
- ump ran out of database connections due to unclosed connections

# [1.x]
## [1.2.0] - 2024-05-25

### Added
Expand Down
6 changes: 5 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,12 @@ ARG USER_UID=1000
ARG USERNAME=pythonuser
ARG USER_GID=2000
ARG SOURCE_COMMIT
ARG IMAGE_TAG=2.0.0

LABEL maintainer="Urban Data Analytics" name="analytics/urban-model-platform" source_commit=$SOURCE_COMMIT
LABEL maintainer="Urban Data Analytics" \
name="analytics/urban-model-platform" \
source_commit=$SOURCE_COMMIT \
version=${IMAGE_TAG}

# add user and group
RUN groupadd --gid $USER_GID $USERNAME && \
Expand Down
29 changes: 28 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,10 @@ initiate-dev:
build-image:
@echo 'Building release ${CONTAINER_REGISTRY}/${CONTAINER_NAMESPACE}/$(IMAGE_NAME):$(IMAGE_TAG)'
# build your image
docker compose -f docker-compose-build.yaml build --build-arg SOURCE_COMMIT=$(GIT_COMMIT) api
docker compose -f docker-compose-build.yaml build \
--build-arg SOURCE_COMMIT=$(GIT_COMMIT) \
--build-arg TAG=$(IMAGE_TAG) \
api

upload-image: build-image
docker compose -f docker-compose-build.yaml push api
Expand Down Expand Up @@ -95,3 +98,27 @@ build-docs:

clean-docs:
jupyter-book clean docs

# Update app version: bump major, minor, or patch
bump-app-version:
@if [ -z "$(part)" ]; then \
echo "Usage: make bump-app part={major|minor|patch}"; \
exit 1; \
fi; \
bump-my-version bump $(part)

# Update app version: set to a specific version
set-app-version:
@if [ -z "$(version)" ]; then \
echo "Usage: make set-app-version version={version}"; \
exit 1; \
fi; \
bump-my-version set $(version)

# Update chart version: bump major, minor, or patch
bump-chart-version:
@if [ -z "$(part)" ]; then \
echo "Usage: make bump-chart part={major|minor|patch}"; \
exit 1; \
fi; \
(cd charts && bump-my-version bump $(part))
10 changes: 10 additions & 0 deletions charts/.bumpversion.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[bumpversion]
current_version = "0.7.0"
commit = true
tag = true
tag_name = "chart-v{new_version}"

[[bumpversion.files]]
filename = "urban-model-platform/Chart.yaml"
search = 'version: {current_version}'
replace = 'version: {new_version}'
4 changes: 2 additions & 2 deletions charts/urban-model-platform/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,13 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.6.0
version: 0.9.5

# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "0.9.2"
appVersion: "2.1.0"
kubeVersion: ">= 1.31.0"
annotations:
category: API
Expand Down
17 changes: 0 additions & 17 deletions charts/urban-model-platform/templates/configmap-job-settings.yaml

This file was deleted.

14 changes: 8 additions & 6 deletions charts/urban-model-platform/templates/configmap-settings.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@ metadata:
labels:
{{- include "ump.labels" . | nindent 4 }}
data:
PROVIDERS_FILE: {{ .Values.config.providersFilePath | quote }}
CORS_URL_REGEX: {{ .Values.config.corsUrlRegex | quote }}
API_SERVER_URL: {{ .Values.config.serverUrl | quote }}
NUMBER_OF_WORKERS: {{ .Values.config.numberOfWorkers | quote }}
POSTGRES_PORT: {{ .Values.config.postgresPort | quote }}
POSTGRES_HOST: {{ .Values.config.postgresHost | quote }}
UMP_LOG_LEVEL: {{ .Values.config.logLevel | quote }}
UMP_PROVIDERS_FILE: {{ .Values.config.providersFilePath | quote }}
UMP_API_SERVER_URL: {{ .Values.config.apiServerUrl | quote }}
UMP_API_SERVER_URL_PREFIX: {{ .Values.config.apiServerUrlPrefix | quote }}
UMP_REMOTE_JOB_STATUS_REQUEST_INTERVAL: {{ .Values.config.remoteJobStatusRequestInterval | quote }}
UMP_JOB_DELETE_INTERVAL: {{ .Values.config.jobDeleteInterval | quote }}
UMP_API_SERVER_WORKERS: {{ .Values.config.apiServerWorkers | quote }}
UMP_SERVER_TIMEOUT: {{ .Values.config.serverWorkerTimeout | quote }}
Loading