Skip to content

Code Organization

Jump to bottom
joshsh edited this page Jul 19, 2026 · 25 revisions

Code organization

This page describes Hydra's code organization pattern used across multiple language implementations.

The packages/heads/overlay/dist pattern

Hydra uses a consistent separation between hand-written and generated code organized into a small set of top-level directories.

Principle

  • packages/ holds a package's DSL-based module definitions, plus source-language helpers used to write them.
  • heads/ holds per-host runtimes that run those modules after they have been translated to a target language.
  • overlay/ holds hand-written, language-specific source that is overlaid onto the generated distribution packages to make them complete and deployment-ready (the source-package → complete-distribution model). Its shape mirrors dist/: overlay/<lang>/<package>/src/.... It holds each host's hydra-kernel runtime (the native primitive implementations, DSL helpers, and kernel entry points), the Haskell hydra umbrella module, and downstream-package overlays where host-specific integrations live (overlay/java/hydra-pg, overlay/java/hydra-rdf, overlay/python/hydra-pg). Overlay files use the hydra.overlay.<lang>.* namespace (#501), keeping a hard boundary between translingual hydra.* code and host-native overlay code. The sync overlays these onto dist/<lang>/<pkg>/; they are authored, not generated, and are never compiled in place.
  • dist/ holds generated and copied artifacts, never edited by hand.

The test for whether a file belongs in packages/ or heads/: does it describe (or help describe) Hydra modules, or does it run them after translation? Description goes in packages/; running goes in heads/. Hand-written source destined for a published distribution package (the kernel runtime, the Haskell hydra umbrella) — but which is not part of the head's own compile — goes in overlay/.

A package's source-language helpers (such as extra DSLs convenient for specifying that package's module definitions) live alongside the package's DSL sources in packages/, written in the same language as the sources. In some cases these helpers can be exported and reused by other packages written in the same source language.

The Hydra repository is not a place for general-purpose utilities written in a specific host language. Host-specific code that is not part of writing or running Hydra modules belongs elsewhere. The one deliberate exception is host-specific third-party integrations — adapters that connect a Hydra package to an external library (Eclipse rdf4j for hydra-rdf; ANTLR-based Cypher/GQL parsers and the TinkerPop/Gremlin bridge for hydra-pg). These live as overlay source under overlay/<lang>/<pkg>/, each declaring its third-party build configuration in overlay/<lang>/<pkg>/build.json, and ship inside the corresponding distribution package. (They were formerly a separate top-level bindings/ tree of independently-published artifacts; #511 folded them into the overlay system. See docs/overlays.md.)

Two flavors of integration exist:

  1. Third-party adapters — wrap an external library against a Hydra package (e.g., the rdf4j Rio integration in overlay/java/hydra-rdf; the ANTLR-based openCypher/GQL parsers in overlay/java/hydra-pg converting to hydra.pg.query.*).
  2. Per-package host DSL helpers — hand-written host-language code that provides DSL surface for a Hydra package, with no third-party dependency (e.g., the Java fluent builders for hydra.pg.{model,query} in overlay/java/hydra-pg).

If hand-written host-language code wants to depend on a third-party library (rdf4j, ANTLR, Neo4j, Apache Jena, TinkerPop, etc.) or wants to provide Java/Python/etc. DSL surface for a specific Hydra package, that code belongs in the package's overlay, not in a heads/<lang>/ runtime. The runtime stays third-party-free except for host stdlib + minimal build tooling.

Directory structure

  • packages/ - DSL source packages. Most are Haskell-based, but hydra-java and hydra-python are now host-language-native (Java and Python sources respectively); hydra-scala joined them in 0.17 (Scala sources). See the per-package notes below.

    • packages/hydra-kernel/ - Kernel type and term modules (the heart of Hydra, written in the Hydra DSL — Haskell-based)
    • packages/hydra-haskell/ - Haskell coder DSL sources (Haskell-based)
    • packages/hydra-java/ - Java coder DSL sources (Java-based as of 0.15; legacy Haskell sources deleted in 0.16)
    • packages/hydra-python/ - Python coder DSL sources (Python-based as of 0.15; legacy Haskell sources deleted in 0.16)
    • packages/hydra-scala/ - Scala coder DSL sources (Scala-based as of 0.17; legacy Haskell sources deleted, #509)
    • packages/hydra-lisp/ - Per-target coder DSL sources (Haskell-based)
    • packages/hydra-pg/ - Property graph models and coders (Pg, Cypher, Tinkerpop, Graphviz)
    • packages/hydra-rdf/ - RDF, SHACL, OWL, ShEx, and XML schema models
    • packages/hydra-ext/ - Long-tail extension coders
    • packages/hydra-bench/ - Synthetic inference benchmark workloads (opt-in via bin/sync-bench.sh) (Avro, Protobuf, GraphQL, Cpp, Csharp, Go, Rust, TypeScript, Yaml, ...)
    • packages/hydra-coq/, packages/hydra-typescript/, packages/hydra-go/, packages/hydra-wasm/ - Additional targets (Coq complete; TypeScript complete per #126; Go and Wasm are "head buds" with partial runtimes)

  • heads/ - Hand-written runtime + generation machinery per language

    • heads/haskell/ - Haskell DSL helpers, code generation utilities, generation drivers, tests
    • heads/java/ - Java framework classes, generation drivers, tests
    • heads/python/ - Python DSL utilities, generation drivers, tests
    • heads/scala/ - Scala primitives, tests
    • heads/lisp/ - Per-dialect Lisp runtimes (clojure/, scheme/, common-lisp/, emacs-lisp/) sharing the hydra-lisp coder
    • heads/typescript/ - TypeScript runtime (#126)
    • heads/go/, heads/wasm/ - Head buds; partial runtimes pending completion
    • (For Haskell/Java/Python the hand-written hydra-kernel runtime — native primitive implementations and DSL helpers — was relocated to overlay/<lang>/ by #418; what remains in heads/<lang>/ is the generation machinery and head-only drivers.)

  • overlay/ - Hand-written source overlaid onto distribution packages (#418). Overlay files use the hydra.overlay.<lang>.* namespace (#501).

    • overlay/haskell/hydra-kernel/ - Native Haskell primitive impls, DSL helpers, and kernel entry points (under the Hydra.Overlay.Haskell.* namespace)
    • overlay/haskell/hydra/ - The hydra umbrella module
    • overlay/java/hydra-kernel/ - Java kernel runtime (hydra/{util,lib,dsl,tools}, Adapters, Coders, ...)
    • overlay/python/hydra-kernel/ - Python kernel runtime (hydra/{lib,dsl,sources}, tools.py)
    • overlay/scala/hydra-kernel/, overlay/typescript/hydra-kernel/ - Scala and TypeScript kernel runtimes
    • overlay/java/hydra-pg, overlay/java/hydra-rdf, overlay/python/hydra-pg - Downstream-package overlays for host-specific third-party integrations

  • dist/ - Generated code per language

    • dist/haskell/hydra-kernel/ - Generated Haskell kernel
    • dist/java/hydra-kernel/ - Generated Java kernel
    • dist/python/hydra-kernel/ - Generated Python kernel
    • dist/scala/hydra-kernel/ - Generated Scala kernel
    • dist/typescript/hydra-kernel/ - Generated TypeScript kernel
    • dist/go/hydra-kernel/ - Generated Go kernel (head bud — kernel only)
    • dist/clojure/, dist/scheme/, dist/common-lisp/, dist/emacs-lisp/ - Per-dialect Lisp kernels
    • dist/json/ - JSON kernel modules (canonical interchange format; tracked in git)

Only dist/json/ and dist/haskell/ are tracked in git; the rest regenerate from dist/json/ on demand.

Benefits

This separation serves several purposes:

  1. Clear distinction - Easy to identify what is hand-written vs. generated
  2. Multi-language parity - Same DSL sources generate Haskell, Java, Python, Scala, and Lisp implementations
  3. Reproducibility - Generated code can be recreated from sources at any time
  4. Version control - Both source and generated code are checked in, enabling:
    • Tracking changes and reviewing diffs
    • Understanding the impact of DSL changes
    • Bisecting regressions across generations
  5. Separation of concerns - Language-specific runtime code stays in heads/, while the kernel remains pure in dist/

Generated code markers

All generated files include a header comment indicating they should not be manually edited. For example in Java:

// Note: this is an automatically generated file. Do not edit.

Or in Haskell:

-- Note: this is an automatically generated file. Do not edit.

Regenerating code

Generated code should be regenerated whenever DSL sources change. See the specific README files for each package:

Test code

Generated tests follow the same pattern as generated main code:

  • dist/<lang>/hydra-kernel/src/test/ - Generated test code
    • Hydra-kernel test suite ensuring parity across implementations
    • Generated from the same test specifications
    • Validates that all language implementations behave identically

Implementation-specific details

Each Hydra package adapts this pattern to its language and purpose:

Hydra-Haskell

The Haskell implementation serves as the bootstrapping implementation for the entire Hydra project. Hand-written sources are split between packages/hydra-kernel/ (kernel type and term specifications), packages/hydra-haskell/ (Haskell coder DSL sources), and heads/haskell/ (runtime: primitives, DSL helpers, code generation drivers, tests). Generated code lives under dist/haskell/.

  • packages/hydra-kernel/src/main/haskell/ contains:

    • Kernel type and term DSL specifications (Hydra/Sources/Kernel/Types/, Hydra/Sources/Kernel/Terms/)
    • Canonical primitive registry — one PrimitiveDefinition-emitting module per hydra.lib.<sub> namespace (Hydra/Sources/Kernel/Lib/)

  • overlay/haskell/hydra-kernel/src/main/haskell/ contains:

    • Host-side primitive bindings — the hydraLib* library definitions pairing primitives with native impls (Hydra/Dsl/Libraries.hs); primitive names derive from the generated PrimitiveDefinitions (#473)

  • packages/hydra-haskell/src/main/haskell/ contains:

    • Haskell coder DSL sources (Hydra/Sources/Haskell/)

  • overlay/haskell/hydra-kernel/src/main/haskell/ contains (host-native, hydra.overlay.haskell.*, #501/#418):

    • DSL helpers and wrappers (Hydra/Overlay/Haskell/Dsl/)
    • Native primitive implementations (Hydra/Overlay/Haskell/Lib/)

  • heads/haskell/src/main/haskell/ contains:

    • Code generation utilities (Hydra/Generation.hs)

  • dist/haskell/hydra-kernel/src/main/haskell/ contains:

    • Complete generated kernel implementation
    • Generated DSL modules (Hydra/Dsl/) with constructors, accessors, and updaters for all Hydra types
    • Generated from DSL sources via writeHaskell and writeDslHaskell

See Hydra-Haskell README for details.

Hydra-Java

The Java implementation provides a Java API for Hydra with the same kernel semantics. The Java coder DSL sources are themselves written in Java (as of 0.15); hand-written runtime lives under heads/java/; generated code under dist/java/hydra-kernel/.

  • packages/hydra-java/src/main/java/hydra/sources/java/ contains:

    • The Java coder DSL sources: Syntax.java, Language.java, Coder.java, Serde.java, Names.java, Utils.java, Environment.java, Testing.java
    • Support classes (JavaHelpers.java, SourceDsl.java)
    • These are the source of truth for hydra.java.* modules; the self-host entry point is bin/generate-hydra-java-from-java.sh.

    These Java sources are the sole source of truth: the former Haskell-DSL copy under packages/hydra-java/src/main/haskell/ has been deleted (#346). The native driver runs against the published hydra-java host (Maven) by default; --local-host builds from local source for a backward-incompatible kernel change.

  • heads/java/src/main/java/ contains:

    • Hand-written primitive function implementations (hydra/lib/)
    • Core utilities (hydra/util/)
    • Framework classes (hydra/tools/)
    • Core algorithms (Rewriting.java, Reduction.java)
    • The native Java DSL → JSON driver (hydra/UpdateJavaJson.java)
    • Language-specific parsers

  • dist/java/hydra-kernel/src/main/java/ contains:

    • Generated Java code from Hydra DSL sources
    • Core types (hydra/core/)
    • Graph and module structures
    • Type adapters and computational abstractions
    • Generated via writeJava in heads/haskell

Uses the visitor pattern for representing algebraic data types in Java.

See Hydra-Java README for details.

Domain packages (hydra-pg, hydra-rdf, hydra-ext, hydra-bench)

Extension modules are organized into four domain-specific packages:

  • packages/hydra-pg/ - Property graph models, coders, and related tools. See the Hydra-PG README.

    • PG data model (Hydra/Sources/Pg/)
    • GraphSON, GQL, Cypher, TinkerPop syntax models
    • Graphviz support

  • packages/hydra-rdf/ - RDF, SHACL, OWL, ShEx, and XML schema models. See the Hydra-RDF README.

    • RDF syntax model (Hydra/Sources/Rdf/)
    • SHACL model and coder (Hydra/Sources/Shacl/)
    • OWL 2 syntax model (Hydra/Sources/Owl/)

  • packages/hydra-ext/ - Long-tail extension coders

    • Avro, Protobuf, GraphQL, Pegasus
    • Cpp, Csharp, Go, Rust, TypeScript syntax models
    • Kusto, Delta, Datalog, JSON Schema, YAML, and other miscellaneous models

  • packages/hydra-bench/ - Synthetic inference benchmark workloads (hydra.bench.*). See the Hydra-Bench README. Deliberately stress-shaped; not regenerated by the default sync. Use bin/sync-bench.sh to refresh on demand before running bin/run-inference-bench.sh.

Generated code for these packages lives in dist/*/hydra-pg/, dist/*/hydra-rdf/, dist/*/hydra-ext/, and dist/*/hydra-bench/. Demos are in demos/ at the repository root.

Hydra-Python

The Python implementation uses the same pattern as other implementations. The Python coder DSL sources are themselves written in Python (as of 0.15); hand-written runtime lives under heads/python/; generated code under dist/python/hydra-kernel/.

  • packages/hydra-python/src/main/python/hydra/sources/python/ contains:

    • The Python coder DSL sources: syntax.py, language.py, coder.py, serde.py, names.py, utils.py, environment.py, testing.py
    • Support modules (_python_helpers.py, _kernel_refs.py)
    • These are the source of truth for hydra.python.* modules; the self-host entry point is bin/generate-hydra-python-from-python.sh.

    These Python sources are the sole source of truth: the former Haskell-DSL copy under packages/hydra-python/src/main/haskell/ has been deleted (#346). The native driver runs against the published hydra-python host (PyPI) by default; --local-host builds from local source for a backward-incompatible kernel change.

  • heads/python/src/main/python/ contains:

    • Hand-written primitive implementations (hydra/lib/)
    • DSL utilities (hydra/dsl/)
    • Language-specific parsers and extensions

  • dist/python/hydra-kernel/src/main/python/ contains:

    • Generated Python code from Hydra DSL sources
    • Core types (hydra/core.py)
    • Graph and module structures (hydra/graph.py, hydra/module.py)
    • Type inference and checking (hydra/inference.py, hydra/checking.py)
    • Term transformations (hydra/reduction.py, hydra/rewriting.py, hydra/hoisting.py)
    • Generated via writePython in heads/haskell

  • dist/python/hydra-kernel/src/test/python/ contains:

    • Generated test suite ensuring parity with Haskell, Java, Python, Scala, and Lisp
    • Generation tests (terms generated to Python and executed)

See Hydra-Python README for details.

Related topics

  • Implementation - Detailed implementation guide including the bootstrap process
  • Testing - Hydra-kernel test suite and language-specific testing
  • Concepts - Core concepts and design principles

Clone this wiki locally