Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,7 @@ __pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# C extensions *.so
# Distribution / packaging
.Python
build/
Expand Down Expand Up @@ -175,3 +173,7 @@ cython_debug/

# IDE
.vscode

# Custom
.orca
orca.json
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [unreleased]

### Added
- A stubbed out `registry` module within the package that the Orca CLI generates type stubs for. This module will contain the algorithms, window types and metadata present within Orca core. The LSP that the user uses must be pointed towards the stubs generated from the CLI for this to be effective.

## [v0.10.0] - 27-09-2025
- Bumped Orca version

Expand Down
57 changes: 54 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,17 +74,68 @@ Replace the contents of `ORCA_CORE` and `PROCESSOR_ADDRESS` with the output of `

Check out more examples [here](./examples/).

## Environment Variables
7. Sync local registry of algorithms to Orca Core

Several environment variables are require to register an Orca Processor:
When building algorithms that depend on the result of algorithms in other proccessors it's neccessary to reference these algorithms as python objects in the `depends_on` argument of the `algorithm` decorator function.

To do this, sync your python project with Orca Core:

```bash
orca sync
```

Run this command at the root level of your python project. This will generate a `.orca` folder that contains type stubs for the algorithms stored in other processors.

Then, configure the settings of your language server and/or linter to include these type stubs. E.g. for pyright, you would include these settings in your `pyrightconfig.json` file at the root of your python project:
```json
{
"stubPath": ".orca",
}
```
Now, when importing modules from `orca_python.registry.algorithms` you will see algorithms configured by remote processors.

For example:
```python
from orca_python.registry.algorithms import <my_remote_algorithm>
```

This stubbed algorithm can be used as an algorithm in the `depends_on` argument of the `algorithm` decorator function.

```python
from orca_python import Processor
from orca_python.registry.window_types import every_30_seconds
from orca_python.registry.algorithms import my_remote_algorithm

proc = Processor("ml")

@proc.algorithm(
"MyAlgo",
"1.0.0",
every_30_seconds,
depends_on=[my_remote_algorithm]
)
def my_algorithm(params: ExecutionParams) -> StructResult:
...
```

## Configuration

Some configuration is required to register a process then start accepting processing requests. If using the internal orca stack for development (via `orca start`), then simply initialise the repository with an `orca.json` file:

```bash
orca init
```

You can additionally provide these environment variables to override, or set, certain configurations:

- `ORCA_CORE` - the address to reach the Orca-core service
- `PROCESSOR_ADDRESS` - the address needed by Orca-core to reach the processor, of format `<address>:<port>`
- `PROCESSOR_EXTERNAL_PORT` - an optional alternative port that should be used by Orca-core to contact the processor. Useful in scenarios like deploying the processor behind a managed service.
- `ENV` - when set to `production` the processor will serve using TLS

## 🧱 Key Concepts

Checkout the Orca [docs](https://app.orc-a.io/docs) for info on how Orca works.
Checkout the Orca [docs](https://orc-a.io/docs) for info on how Orca works.

## 👥 Community

Expand Down
Empty file.
Empty file.
433 changes: 433 additions & 0 deletions examples/simpletrigger/poetry.lock

Large diffs are not rendered by default.

18 changes: 18 additions & 0 deletions examples/simpletrigger/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
[project]
name = "simpletrigger"
version = "0.1.0"
description = ""
authors = [
{name = "Frederick Mannings",email = "fred.mannings@solvicode.com"}
]
readme = "README.md"
requires-python = ">=3.13"
dependencies = [
"orca-python (>=0.10.0,<0.11.0)",
"schedule (>=1.2.2,<2.0.0)"
]


[build-system]
requires = ["poetry-core>=2.0.0,<3.0.0"]
build-backend = "poetry.core.masonry.api"
3 changes: 3 additions & 0 deletions examples/simpletrigger/pyrightconfig.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"stubPath": ".orca",
}
Empty file.
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@

proc = Processor("ml")

# E.g. you have you have a fleet of busses, where every bus has a particular ID and runs a particular route
# E.g. you have you have a fleet of busses, where every bus has a particular
# ID and runs a particular route
route_id = MetadataField(name="route_id", description="The unique ID of the route")
bus_id = MetadataField(name="bus_id", description="The unique ID of the bus")

Expand All @@ -24,6 +25,7 @@

@proc.algorithm("MyAlgo", "1.0.0", Every30Second)
def my_algorithm(params: ExecutionParams) -> StructResult:
"""A simple algorithms that does nothing interesting"""
route_id = params.window.metadata.get("route_id", None)
bus_id = params.window.metadata.get("bus_id", None)
print(route_id, bus_id)
Expand Down
2 changes: 1 addition & 1 deletion orca
Submodule orca updated from b2e84f to 09af32
126 changes: 112 additions & 14 deletions orca_python/envs.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,14 @@
import os
import re
import json
from typing import Tuple, Optional
from logging import getLogger
from pathlib import Path
from dataclasses import dataclass

from orca_python.exceptions import BadEnvVar, MissingEnvVar
from orca_python.exceptions import BadEnvVar, BadConfigFile, MissingEnvVar

LOGGER = getLogger(__name__)


def _parse_connection_string(connection_string: str) -> Optional[Tuple[str, int]]:
Expand Down Expand Up @@ -33,28 +39,74 @@ def _parse_connection_string(connection_string: str) -> Optional[Tuple[str, int]
return (address, port)


def getenvs() -> Tuple[bool, str, str, int, int]:
@dataclass
class ConfigData:
projectName: str
orcaConnectionString: str
processorPort: int
processorConnectionString: str


def parseConfigFile() -> Tuple[bool, str, str, str, int, int]:
currentDir = Path.cwd()
configFile = currentDir / "orca.json"
hasConfig = False

if configFile.exists():
hasConfig = True
try:
with open(configFile, "r") as f:
configData = ConfigData(**json.load(f))
except Exception as e:
LOGGER.error(f"Could not parse config file: {e}")
raise BadConfigFile(f"Could not parse config file: {e}")
else:
return (False, "", "", "", 0, 0)

res = _parse_connection_string(configData.processorConnectionString)
if res is None:
raise BadConfigFile(
"processorConnectionString is not a valid address of the form <ip>:<port>"
)
processor_address, processor_port = res

return (
hasConfig,
configData.projectName,
configData.orcaConnectionString,
processor_address,
processor_port,
configData.processorPort,
)


def getenvs(strict: bool = False) -> Tuple[bool, str, str, int | None, int | None]:
orca_core = os.getenv("ORCA_CORE", "")
if orca_core == "":
if strict and orca_core == "":
raise MissingEnvVar("ORCA_CORE is required")
orca_core = orca_core.lstrip("grpc://")

proc_address = os.getenv("PROCESSOR_ADDRESS", "")
if proc_address == "":
if strict and proc_address == "":
raise MissingEnvVar("PROCESSOR_ADDRESS is required")

res = _parse_connection_string(proc_address)
if res is None:
raise BadEnvVar(
"PROCESSOR_ADDRESS is not a valid address of the form <ip>:<port>"
)
processor_address, processor_port = res
processor_address = ""
processor_port = None
if proc_address != "":
res = _parse_connection_string(proc_address)
if res is None:
raise BadEnvVar(
"PROCESSOR_ADDRESS is not a valid address of the form <ip>:<port>"
)
processor_address, processor_port = res

_processor_external_port = os.getenv("PROCESSOR_EXTERNAL_PORT", "")
if _processor_external_port != "":
if strict and _processor_external_port != "":
if not _processor_external_port.isdigit():
raise BadEnvVar("PROCESSOR_EXTERNAL_PORT is not a valid number")
processor_external_port = int(_processor_external_port)
elif _processor_external_port == "":
processor_external_port = None
else:
processor_external_port = processor_port

Expand All @@ -73,6 +125,52 @@ def getenvs() -> Tuple[bool, str, str, int, int]:
)


is_production, ORCA_CORE, PROCESSOR_HOST, PROCESSOR_PORT, PROCESSOR_EXTERNAL_PORT = (
getenvs()
)
# config file takes priority. Env vars can overwrite. And if config file not
# present, all the env vars have to be there.
(
hasConfig,
PROJECT_NAME,
ORCA_CORE,
PROCESSOR_HOST,
PROCESSOR_PORT,
PROCESSOR_EXTERNAL_PORT,
) = parseConfigFile()

if PROJECT_NAME == "":
LOGGER.warning(
"Project name could not be found in `orca.json` (or the config is not present). When generating stubs with `orca sync` this may cause algorithm definitions that are present in this repository to be duplicated locally.\nRun `orca init` to generate a `orca.json` config file to avoid this."
)
if hasConfig:
(
is_production,
_ORCA_CORE,
_PROCESSOR_HOST,
_PROCESSOR_PORT,
_PROCESSOR_EXTERNAL_PORT,
) = getenvs()
if _ORCA_CORE != "":
ORCA_CORE = ORCA_CORE
if _PROCESSOR_HOST != "":
PROCESSOR_HOST = _PROCESSOR_HOST
if _PROCESSOR_PORT is not None:
PROCESSOR_PORT = _PROCESSOR_PORT
if _PROCESSOR_EXTERNAL_PORT is not None:
PROCESSOR_EXTERNAL_PORT = _PROCESSOR_EXTERNAL_PORT
else:
(
is_production,
ORCA_CORE,
PROCESSOR_HOST,
_PROCESSOR_PORT,
_PROCESSOR_EXTERNAL_PORT,
) = getenvs(strict=True)

if _PROCESSOR_PORT is None:
MissingEnvVar("PROCESSOR_PORT required")
else:
PROCESSOR_PORT = _PROCESSOR_PORT

if _PROCESSOR_EXTERNAL_PORT is None:
MissingEnvVar("PROCESSOR_EXTERNAL_PORT required")
else:
PROCESSOR_EXTERNAL_PORT = _PROCESSOR_EXTERNAL_PORT
8 changes: 8 additions & 0 deletions orca_python/exceptions.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,11 @@ class MissingEnvVar(BaseOrcaException):

class BadEnvVar(BaseOrcaException):
"""Raised when an environment variable is poorly defined"""


class BadConfigFile(BaseOrcaException):
"""Raised when the orca.json config file is poorly defined"""


class BrokenRemoteAlgorithmStubs(BaseOrcaException):
"""Raised when remote algorithm stubs cannot be properly parsed and read"""
Loading