New Relic Integration for HashiCorp Vault

Overview

This integration collects metrics from HashiCorp Vault's /v1/sys/metrics endpoint (Prometheus format) and forwards them to New Relic.

Why Prometheus Format?

This integration uses Vault's Prometheus format (?format=prometheus) instead of the default JSON format. The key difference:

• JSON format: Uses a "flush-and-reset" mechanism where counter values are reset after each collection interval. This can cause data loss when the polling interval doesn't align perfectly with Vault's internal aggregation interval (10s)
• Prometheus format: Uses cumulative counters that never reset. This ensures 100% data fidelity regardless of polling interval alignment.

Metric Types

The SDK supports multiple metric types. This integration maps Prometheus metric types as follows:

Prometheus Type	New Relic Type	Description	NRQL Query Tip
Gauge	gauge	Point-in-time measurements (e.g., vault_runtime_num_goroutines)	Use latest() or average()
Counter	cumulative-count	Cumulative counts with automatic delta calculation	Use sum() for total count, rate(sum(...), 1 minute) for rate
Summary	prometheus-summary	Statistical summaries with _count, _sum, and quantiles	Use average() on the metric directly
Histogram	prometheus-summary	Converted to summary format with _count, _sum, and bucket boundaries	Use average() on the metric directly

Cumulative Count Metrics

The SDK's cumulative-count type automatically calculates the delta between consecutive values, making it easy to query rates directly. Each metric includes a prometheus_type attribute to help identify the original metric type returned from sys/metrics.

For a complete list of available metrics, see the HashiCorp Vault Telemetry Documentation.

Installation

Prerequisites

Vault prerequisites

1. Enable Prometheus metrics retention - This is required for the integration to work. Add or update the telemetry stanza in your Vault configuration:

telemetry {
  prometheus_retention_time = "24h"  # Required - must be longer than polling interval
  disable_hostname          = true   # Recommended - prevents double-tagging by New Relic
}

Important: Without prometheus_retention_time configured, Vault returns an empty response for the Prometheus endpoint.

2. Create a metrics access policy in the root namespace that allows read access to /v1/sys/metrics:

# newrelic.hcl - Allow reading the metrics endpoint
path "sys/metrics" {
  capabilities = ["read", "list"]
}

# Optional: Allow reading the health endpoint for status checks
path "sys/health" {
  capabilities = ["read", "list"]
}

Apply the policy:

vault policy write newrelic newrelic.hcl

3. Configure metrics access via one of:
   • Unauthenticated Access: Enable in Vault listener config (unauthenticated_metrics_access = true)
   • Authenticated Access: A valid Vault token with the newrelic policy. See Secure Authentication for token management.

New Relic prerequisites

• A New Relic account
• New Relic Infrastructure Agent installed and configured with a valid license key

Install Steps

1. Download a pre-generated binary for your OS or build from source (see Building)
2. Copy the integration binary (nri-vault) to /var/db/newrelic-infra/custom-integrations/
3. Copy the sample configuration to /etc/newrelic-infra/integrations.d/
4. Set required environment variables (see Configuration)
5. Restart the Infrastructure Agent:

   sudo systemctl restart newrelic-infra

Configuration

Environment Variables

Variable	Required	Default	Description
VAULT_ADDR	No	http://127.0.0.1:8200	Vault server address
VAULT_TOKEN	Yes*	-	Vault token for authentication
VAULT_TOKEN_FILE	Yes*	-	Path to file containing Vault token (recommended)
UNAUTH_METRICS_ACCESS_ENABLED	No	false	Set to true if Vault allows unauthenticated metrics
TIMEOUT	No	60s	HTTP timeout for Vault requests
METRICS_TO_IGNORE	No	-	Comma-separated metric names to exclude
LABELS_TO_IGNORE	No	-	Comma-separated label names to exclude (see warning below)

*Either VAULT_TOKEN or VAULT_TOKEN_FILE is required unless UNAUTH_METRICS_ACCESS_ENABLED=true

⚠️ Warning - LABELS_TO_IGNORE

This setting removes the specified labels from metric attributes but does NOT aggregate metrics. If you ignore a label that differentiates metric series (e.g., database, cluster, gauge), you'll end up with multiple metrics having identical names and dimensions, which can cause data conflicts in New Relic.

Safe to ignore: Labels that are redundant or don't differentiate metrics (e.g., a hostname label when you're already capturing host info elsewhere).

Unsafe to ignore: Labels like database, type, namespace, or gauge that distinguish different metric series.

Secure Authentication (Recommended)

While this integration does support hardcoding a long-lived Vault token in the configuration, it is recommended to use Vault Agent to manage the token lifecycle. Vault Agent can authenticate with Vault (i.e: using AppRole, or other preferred methods) and write a valid token to a file (sink). The integration then reads this file. Below is an example of how to configure using an AppRole.

NOTE: This is just one example for authentication. See Auth methods for alternatives

1. Enable AppRole

vault auth enable approle

2. Create the Role

2.1 Define the role

vault write auth/approle/role/newrelic-role \
    token_policies="newrelic" \
    token_ttl=1h \
    token_max_ttl=24h \
    secret_id_ttl=0 \
    secret_id_num_uses=0

Note: secret_id_ttl=0 means the SecretID does not expire. In a more mature environment, you might rotate SecretIDs daily using a configuration management tool (Puppet/Ansible). For this example, a static SecretID (protected by file permissions) is the baseline requirement.

2.2 Retrieve Credentials & Generate SecretID

vault read auth/approle/role/newrelic-role/role-id
vault write -f auth/approle/role/newrelic-role/secret-id

3. Configure Vault Agent

Create a Vault Agent configuration file (i.e: agent-config.hcl):

pid_file = "/tmp/pidfile"
exit_after_auth = false # Run as daemon to renew token

vault {
  address = "http://127.0.0.1:8200"
}

auto_auth {
  method "approle" {
    mount_path = "auth/approle"
    config = {
      role_id_file_path = "/etc/vault/role_id"
      secret_id_file_path = "/etc/vault/secret_id"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = {
      path = "/etc/vault/token"
    }
  }
}

4. Run Vault Agent

Run the agent as a background process or sidecar:

vault agent -config=agent-config.hcl

5. Configure the Integration

Update vault-config.yml to point to the token file:

env:
  VAULT_TOKEN_FILE: "/etc/vault/token"

Usage

Once configured, the New Relic Infrastructure agent will automatically run the integration periodically and ingest data. To explore the generated metrics:

1. Open New Relic One.
2. Navigate to Query Your Data or Metrics & Events.
3. Filter by metrics starting with vault_.

NRQL Examples

Below are example NRQL queries to get started with exploring data more. Additionally, if you are more familiar with PromQL, use this doc as a reference for NRQL translation.

-- Active Vault nodes (gauge)
FROM Metric SELECT latest(vault_core_active) FACET cluster

-- Memory usage over time (gauge)
FROM Metric SELECT average(vault_runtime_alloc_bytes) TIMESERIES

-- Cache hit rate (cumulative-count - automatic delta calculation)
FROM Metric SELECT sum(vault_cache_hit) AS 'total hits'
WHERE prometheus_type = 'counter' TIMESERIES

-- Cache hit rate per minute
FROM Metric SELECT rate(sum(vault_cache_hit), 1 minute) AS 'hits/min'
WHERE prometheus_type = 'counter' TIMESERIES

-- Token creation count (cumulative-count)
FROM Metric SELECT sum(vault_token_creation) AS 'total tokens' TIMESERIES

-- Find all counter metrics
FROM Metric SELECT uniques(metricName) WHERE prometheus_type = 'counter' LIMIT MAX

-- Request latency average (from summary metrics)
FROM Metric SELECT average(vault_core_handle_request) WHERE prometheus_type = 'summary' TIMESERIES

-- Raft storage write time by database
FROM Metric SELECT average(vault_raft_storage_bolt_write_time) FACET database TIMESERIES

-- Get the count of observations
FROM Metric SELECT latest(vault_raft_storage_bolt_spill_time_count) FACET database TIMESERIES

-- Get the cumulative sum
FROM Metric SELECT latest(vault_raft_storage_bolt_spill_time_sum) FACET database TIMESERIES

-- Calculate average (sum/count)
FROM Metric SELECT latest(vault_raft_storage_bolt_spill_time_sum) / latest(vault_raft_storage_bolt_spill_time_count) AS 'average'
FACET database TIMESERIES

-- Get a specific percentile (e.g., 99th)
FROM Metric SELECT latest(vault_raft_storage_bolt_spill_time)
WHERE quantile = '0.99' FACET database TIMESERIES

-- Rate of operations per second
FROM Metric SELECT rate(sum(vault_raft_storage_bolt_spill_time_count), 1 second) AS 'ops/sec'
FACET database TIMESERIES

Dashboard

A sample dashboard JSON is available in dashboards/dashboard.json. To import it into New Relic:

1. Edit all occurrences of the accountIds field in the json (ctrl+f -> search for 123 -> replace with your NR accountId that you are importing to)
2. Follow these instructions

Troubleshooting

Empty or No Metrics

1. Check prometheus_retention_time: Ensure your Vault telemetry config includes prometheus_retention_time. Without this, the Prometheus endpoint returns empty data.

2. Verify endpoint access: Test with curl:

   curl -H "X-Vault-Token: $VAULT_TOKEN" "http://127.0.0.1:8200/v1/sys/metrics?format=prometheus"

3. Check logs: Look for errors in the Infrastructure agent logs at /var/log/newrelic-infra/newrelic-infra.log

Authentication Errors

• Ensure the token has the newrelic policy attached
• If using VAULT_TOKEN_FILE, verify the file exists and is readable by the Infrastructure agent:

  # Check file exists and permissions/owner
  ls -la /etc/vault/token
  
  # Verify the Infrastructure agent user can read it (skip if agent runs as root)
  sudo -u nri-agent cat /etc/vault/token
  
  # Test the token is valid
  curl -H "X-Vault-Token: $(cat /etc/vault/token)" "http://127.0.0.1:8200/v1/sys/metrics?format=prometheus" | head -20

• Check if the token has expired

Building

Prerequisites

• Go 1.23 or higher
• Make

Steps

1. Clone this repo and open a terminal in the folder where you cloned it.
2. Run make to build for all platforms, or:
   • make macos-arm for macOS ARM64
   • make macos-intel for macOS AMD64
   • make linux-arm for Linux ARM64
   • make linux-intel for Linux AMD64
   • make windows for Windows AMD64

The generated binaries will be located in the bin/ directory.

Support

This project is actively maintained by the New Relic Labs team. Connect with us directly by creating issues or asking questions in the discussions section of this repo.

We also encourage you to bring your experiences and questions to the Explorers Hub where our community members collaborate on solutions and new ideas.

New Relic has open-sourced this project, which is provided AS-IS WITHOUT WARRANTY OR DEDICATED SUPPORT.

Security

As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.

If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.

Contributing

Contributions are welcome (and if you submit a Enhancement Request, expect to be invited to contribute it yourself 😁). Please review our Contributors Guide.

Keep in mind that when you submit your pull request, you'll need to sign the CLA via the click-through using CLA-Assistant. If you'd like to execute our corporate CLA, or if you have any questions, please drop us an email at opensource@newrelic.com.

License

This project is distributed under the Apache 2 license.