doc: document the Configuration system

Author: HectorJM2018Cosmo

docs/tutorials/cosmotech-api.md

@@ -98,6 +98,38 @@ Keycloak authentication requires these environment variables:
 !!! warning "API Client Lifecycle"
     Always close the API client when you're done using it to release resources. The best practice is to use a `try`/`finally` block to ensure the client is closed even if an error occurs.
 
+## Configuration
+
+The CoAL configuration system is based on a centralized data dictionary used to manage platform settings and behaviors dynamically. It allows scripts to run without requiring users to manually define connection or output specifics every single time. Data is primarily sourced from a TOML file loaded into a Kubernetes ConfigMap.
+
+### Core mechanics
+
+- The Configuration singleton: CoAL provides a `ENVIRONMENT_CONFIGURATION` singleton that users can import this into their scripts (`from cosmotech.coal.utils.configuration import ENVIRONMENT_CONFIGURATION as EC`) to access properties using dot-notation, such as `EC.cosmotech.runner_id`.
+
+- Kubernetes (K8s) ConfigMap integration: To supply configuration inside a pod launched via a workflow, CoAL mounts a K8s ConfigMap containing the configuration file directly inside the container.
+
+- Automatic path loading: CoAL automatically attempts to load the TOML file at the specific path `/mnt/coal/coal-config.toml`, making K8s ConfigMap auto-mounts seamless.
+
+### Syntax
+
+The configuration uses the TOML format to support specific features:
+
+- **secrets**: Environment variables (e.g. credentials, `TWIN_CACHE_HOST`, or `IDP_BASE_URL`) that are loaded at startup. At import, they are initialized and then removed from the final configuration dictionary, so variables like `run_template_id` are accessed directly under `EC.cosmotech` rather than a "secrets" sub-dictionary. CosmoTech environment variables provided by the API are always loaded.
+
+- **env.**: Fetches environment variables dynamically at runtime (e.g. `env.POSTGRES_USER_PASSWORD`), unlike "secrets" which are resolved statically at import.
+
+- **Internal References ($)**: Allows configuration keys to reference other values in the same TOML file (e.g. `$postgres.host`).
+
+- **[[outputs]]**: Uses TOML double-bracket list syntax to define a series of output destinations (such as PostgreSQL, S3, or Azure Blob Storage) utilized by the ChannelSplitter to direct simulation results.
+
+- **Error handling**: CoAL handles of internal configuration references (like `$config.path`) with proper error reporting such as the `ReferenceKeyError` exception for missing configuration references.
+
+### Configuration dictionary
+
+```toml title="Configuration TOML file" linenums="1"
+--8<-- 'tutorial/cosmotech-api/coal-config.toml'
+```
+
 ## Working with Workspaces
 
 Workspaces in the CosmoTech platform provide a way to organize and share files. `WorkspaceApi` offers methods for listing, downloading, and uploading files.

tutorial/cosmotech-api/coal-config.toml

@@ -0,0 +1,78 @@
+# This an exemple of a coal-config.toml
+
+# the double [[ section indicate a list. This allows to define multiple outputs
+# Each output define its type (currently supported: s3, az_storage, postgres)
+# For each output, the configuration defines value needed to interact with a storage. Mandatory value are mark with a [M]
+
+[[outputs]]
+type = "s3"         # indicate the output channel to use
+[outputs.conf.s3]
+endpoint_url =      # [M] url of the s3 server
+access_key_id =     # [M] the id used to connect (equivalent to a username)
+secret_access_key = # [M] the key to connect (equivalent to a password)
+bucket_name =       # [M] name of the bucket. s3 bucket is a concept that define a ressource that holds data. 
+bucket_prefix =     # This is a prefix that will be add in front of each upload file 
+                        # (this can allow to store in subfolder by adding a prefix ending with "/")
+outputs_type =      # indicate the type of the data push. Can be .parquet or .csv (default: .csv)
+use_ssl =           # indicate the use of ssl (default: True)
+ssl_cert_bundle =   # in case of a s3 using custom SSL certification; here is where to put the path to the custom pem bundle. 
+                        # (Can also be set to False to not verify SSL certificat)
+
+[[outputs]]
+type = "az_storage" # indicate the output channel to use
+[outputs.conf.azure]
+account_name =      # use to build azure storage URL 
+tenant_id =         # Azure tenant ID
+client_id =         # Azure client ID
+client_secret =     # Azure secret
+container_name =    # Name of the container (equivalent to AWS Bucket name)
+outputs_type =      # indicate the type of the data push. Can be .parquet or .csv (default: .csv)
+file_prefix =       # This is a prefix that will be add in front of each upload file 
+                        # (this can allow to store in subfolder by adding a prefix ending with "/")
+
+[[outputs]]
+type = "postgres"
+[outputs.conf.postgres]
+host =              # Host URL of postgres server
+port =              # Port expose by postgres server
+db_name =           # Postgres db name
+db_schema =         # Postgres db schema
+user_name =         # postgres username
+user_password =     # posrgres password
+table_prefix =      # prefix used on table creation (useful in case of using a centralize DB to differentiate)
+password_encoding = # boolean indicating if the password should be encoder (default: False) (used for password with special characters)
+
+
+# The secrets section contains all values that will be replaced by environement variables
+# The secrets section is transform at initialisation and then merge to root (the secrets sections no longer exist after transformation)
+
+# The cosmotech sub section is added by default (it's all the environment variables given by the API)"
+
+# # # DON'T ADD THIS IN THE FINAL FILE # # #
+
+[secrets.cosmotech]
+dataset_absolute_path = "CSM_DATASET_ABSOLUTE_PATH"
+parameters_absolute_path = "CSM_PARAMETERS_ABSOLUTE_PATH"
+output_absolute_path = "CSM_OUTPUT_ABSOLUTE_PATH"
+tmp_absolute_path = "CSM_TEMP_ABSOLUTE_PATH"
+organization_id = "CSM_ORGANIZATION_ID"
+workspace_id = "CSM_WORKSPACE_ID"
+runner_id = "CSM_RUNNER_ID"
+run_id = "CSM_RUN_ID"
+run_template_id = "CSM_RUN_TEMPLATE_ID"
+
+[secrets.cosmotech.api]
+url = "CSM_API_URL "
+scope = "CSM_API_SCOPE"
+
+[secrets.cosmotech.twin_cache]
+host = "TWIN_CACHE_HOST"
+port = "TWIN_CACHE_PORT"
+password = "TWIN_CACHE_PASSWORD"
+username = "TWIN_CACHE_USERNAME"
+
+[secrets.cosmotech.idp]
+base_url = "IDP_BASE_URL"
+tenant_id = "IDP_TENANT_ID"
+client_id = "IDP_CLIENT_ID"
+client_secret = "IDP_CLIENT_SECRET"