Ability to distribute Fabric templates inside a plugin

[traut]

Background

With the content libraries, like Fabric Templates, accumulating useful templates for various use cases, and vendors producing integration-specific templates, there should be an easy way for the users to access the templates.

Design

Utilize our existing plugin infrastructure (plugin registry and dependency installation logic) to deliver content by including the templates inside the plugins.

Requirements

• the plugins may include Fabric templates
• it should be possible to add templates to existing plugins without breaking the current installation / upgrade flow. For example, we should be able to add Elastic-specific templates inside blackstork/elastic plugin
• it should be possible to have content-only plugins -- plugins that contain only templates, without any data sources / content provides / data sinks. For example, we might build blackstork/fabric-templates plugin that would represent Fabric Template release.
• the templates / blocks defined in the Fabric configuration files inside the plugin should be accessible for the user -- for example, it should be possible to reference a block with ref syntax:
  • To Estimate: if it is possible to namespace the block id (for example, section.ctid_executive_summary) without breaking the parsing, we should do it. If it would require significant refactoring or breaking changes, we should postpone it.
  • For example, a namespaced section ID for a section in blackstork/fabric_tempaltes plugin would look like this:

    document "my-doc" {
      section ref {
        base = fabric_templates/section.ctid_executive_summary
      }
    }

Full example:

fabric {
  plugin_versions = {
    "blackstork/elastic" = ">= 1.2.3",
    "blackstork/fabric_templates" = ">= 0.1.1",
  }
}

document "my-doc" {

  data ref "my_alerts" {
    base = elastic/data.elasticsearch.alerts_with_aggregations
    query = "@timestamp:[now-7d TO now]"
  }

  section ref {
    base = fabric_templates/section.ctid_executive_summary
  }
}

Implementation

TBD in the comments

Constraints

• storage is cheaper than computing, so it is better to unpack the configs to the FS for use during fabric execution instead of unpacking-on-the-fly

[traut]

an alternative namespacing schema that does not break HCL identifier constraint is to use -- as a separator:

document "my-doc" {
  section ref {
    base = fabric_templates--section.ctid_executive_summary
  }
}

However, it is less "namespacy" -- the prefix is not perceived as a namespace at a glance.

[dobarx]

@traut
What about separating namespace with prefix import.<plugin>.<path>? Usage:

document "my-doc" {
  section ref {
    base = import.fabric_templates.section.ctid_executive_summary
  }

We could also choose different prefix like template.<plugin>.<path> or plugin.<plugin>.<path>

[traut]

@dobarx oh, that's neat! I like import.<plugin-name>.<path> a lot, awesome idea!

It is clear that it is an import while the name is nicely namespaced. Let's go with it!

[traut]

One thing that occurred to me:

We talked about adding a manifest-like to the plugins. Another argument and use case for it, in the context of template distribution in the plugins, is the need to have a list of template dependencies -- what plugins must be installed for the templates to work?