feat(consul): filter nodes in upstream with metadata by jizhuozhi · Pull Request #12448 · apache/apisix

jizhuozhi · 2025-07-20T05:57:24Z

Description

This PR introduces metadata-based node filtering for consul discovery, supporting Consul service discovery based upstreams

Motivation

Currently, APISIX selects upstream nodes based on service name from discovery without additional filtering logic. In real-world scenarios like canary release or swimlane routing, users often tag backend instances with custom metadata (e.g., version, env, lane, dc, and etc) and expect the gateway to route only to specific subsets.

This change allows users to define a metadata_match field in discovery_args configuration, which filters nodes before load balancing based on their metadata values.

Changes

Consul discovery:

Include Service.Meta in the node definition and respect its weight if available (aligned with Eureka).
Filter with discovery_args when fetch upstream nodes.

Tests: Add test cases to cover both:
- discovery-based upstream (Consul) with metadata_match

Example Usage

upstream:
  type: roundrobin
  scheme: http
  discovery_type: eureka
  discovery_args:
    metadata_match:
      lane:
      - prod
      - canary
      dc:
      - us-east-1
      - us-east-2

Only nodes with metadata.lane in [prod, canary] and metadata.dc in [us-east-1, us-east-2] will be used for load balancing.

Fixes

Fixes #12464

Checklist

I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (no breaking changes)

Baoyuantop · 2025-07-21T02:43:23Z

Hi @jizhuozhi, thanks for your contribution.

I think it is useful for node filtering of Consul discovery. But I don't understand static upstream filtering. It seems that I need to mark metadata for each node in the upstream object, and then use metadata_match to configure filtering? Because each node is manually defined, if it is not needed, can I just add or delete the node?

jizhuozhi · 2025-07-21T02:47:08Z

Hi @jizhuozhi, thanks for your contribution.

I think it is useful for node filtering of Consul discovery. But I don't understand static upstream filtering. It seems that I need to mark metadata for each node in the upstream object, and then use metadata_match to configure filtering? Because each node is manually defined, if it is not needed, can I just add or delete the node?

Here is a unified approach: I only determine whether there is a filtering rule, without distinguishing whether it is a service discovery or static list.

However, according to my previous experience as a gateway administrator, there will be corresponding business developers who temporarily add rules for some debugging considerations but do not want to change the original instance list (for quick adjustment)

Baoyuantop · 2025-07-21T03:04:46Z

there will be corresponding business developers who temporarily add rules for some debugging considerations but do not want to change the original instance list

Thanks for your reply. Could you please describe the scenario in detail? Why can't the existing methods solve this problem?

jizhuozhi · 2025-07-21T11:21:47Z

Thanks for your reply. Could you please describe the scenario in detail? Why can't the existing methods solve this problem?

It is not a production environment, but it is common in the testing and verification phase. We need to specify specific instances frequently (for example, to capture flame graphs for performance analysis), but we need to add other instances back after deleting them, so we need to specify instances by filtering.

In fact, we also matched according to the dynamic colored metadata when loading balancing but not predefine the routes, similar to https://github.com/kitex-contrib/loadbalance-tagging (I am also using lua to implement the same capabilities, but this is not within the scope of this discussion).

Baoyuantop · 2025-07-22T05:59:30Z

Thanks for your reply. Could you please describe the scenario in detail? Why can't the existing methods solve this problem?

It is not a production environment, but it is common in the testing and verification phase. We need to specify specific instances frequently (for example, to capture flame graphs for performance analysis), but we need to add other instances back after deleting them, so we need to specify instances by filtering.

In fact, we also matched according to the dynamic colored metadata when loading balancing but not predefine the routes, similar to https://github.com/kitex-contrib/loadbalance-tagging (I am also using lua to implement the same capabilities, but this is not within the scope of this discussion).

I still have doubts about what is in Example Usage. Do you mean that if I need to adjust the nodes used, I don't need to change the content of the nodes list, but adjust metadata_match?
By the way, are you using static nodes or service discovery?

jizhuozhi · 2025-07-22T06:20:00Z

I still have doubts about what is in Example Usage. Do you mean that if I need to adjust the nodes used, I don't need to change the content of the nodes list, but adjust metadata_match?

Yes, just adjust metadata_match (but the discussion of this use case has been separated from this PR). For the runtime, it is a unified filtering rule for the service list that does not need to distinguish the source.

By the way, are you using static nodes or service discovery?

We are currently using Consul on kubernetes. When I was working in another company a few years ago, we were using cloud virtual machines (or EC2). The cloud platform did not provide an API interface, but we used scripts to synchronize static instance lists at regular intervals. At this time, the static list was also a kind of dynamic discovery. (why not filter in the script? Because we were lazy:)

Baoyuantop · 2025-07-25T05:41:34Z

Hi @jizhuozhi, There is currently no modification to the upstream schema, which means that the current modifications in the upstream only serve consul. Is it more appropriate to put all these logics into the consul module?

jizhuozhi · 2025-07-25T05:55:27Z

Hello, @Baoyuantop, thanks for your reply.

Hi @jizhuozhi, There is currently no modification to the upstream schema, which means that the current modifications in the upstream only serve consul. Is it more appropriate to put all these logics into the consul module?

Not only consul, but also Eureka (which has already supported metadata in apisix) will inherit this function. In my forked dashboard has already supported configuring metadata_match for Consul and Eureka
https://github.com/jizhuozhi/apisix-dashboard/blob/9bd72c82e4fcbfa0d2bf34420280028c2ca853c8/web/src/components/Upstream/components/ServiceDiscovery.tsx#L30-L37

(The examples in the PR description are just examples, because this allows testing without the registry, and we don't need to care about service discovery or static nodes.)

And the current discovery package is responsible for pulling all instances, and the filtering in discovery is effective for all service names and upstreams, it means that I can only configure general filtering rules, but cannot configure differentiated matching for different routes and upstreams. This is our current online effect

jizhuozhi · 2025-07-25T06:13:54Z

Hello @Baoyuantop , I see. Currently, upstream has passed the discovery args to nodes, so the loop can be closed in discovery. I will modify it.

        local new_nodes, err = dis.nodes(up_conf.service_name, up_conf.discovery_args)
        if not new_nodes then
            return HTTP_CODE_UPSTREAM_UNAVAILABLE, "no valid upstream node: " .. (err or "nil")
        end

jizhuozhi · 2025-07-25T06:44:10Z

Hello @Baoyuantop , PTAL, thanks :)

We also have Spring Cloud applications with Eureka, but I have no time to write test case now, so I will create a new PR for Eureka later.

Baoyuantop · 2025-07-25T07:08:50Z

Hi @jizhuozhi, we are still discussing whether to accept the feature of this PR, and we need to reach a consensus before we can start the review. Since there is no separate issue to discuss this issue, you need to clearly tell the maintainer what this feature does and why it is needed in the PR description (the current description already exists).

The examples in the PR description are just examples, because this allows testing without the registry, and we don't need to care about service discovery or static nodes.

This is inappropriate and you need to replace it with an example from a real scenario. The current example will confuse other maintainers. In the latest changes, I see that you have cancelled the upstream related code. The current PR seems to focus on the filtering of consul services. Please update the PR description to reflect this. Thanks again for your contribution.

jizhuozhi · 2025-07-25T07:20:51Z

This is inappropriate and you need to replace it with an example from a real scenario. The current example will confuse other maintainers.

Thank you for your reminder, the PR content has been updated

Baoyuantop · 2025-07-28T06:45:56Z

Please fix the failed CI.

SkyeYoung · 2025-07-31T07:49:50Z

 GET /t
 --- response_body
-{"service_a":[{"host":"127.0.0.1","port":30511,"weight":1}],"service_b":[{"host":"127.0.0.1","port":8002,"weight":1}]}
+{"service_a":[{"host":"127.0.0.1","metadata":{"service_a_version":"4.0"},"port":30511,"weight":1}],"service_b":[{"host":"127.0.0.1","metadata":{"service_b_version":"4.1"},"port":8002,"weight":1}]}


Why does it affect this place? I don't think this test should be changed

Since we are now relying on metadata, we need to persist it when persisting.

@jizhuozhi Ok. Another question is, if this is added, do we now lack a test case for the situation where there is "no metadata"?

I will add it :)

Baoyuantop · 2025-11-21T02:52:43Z

Hello @Baoyuantop , the test case t/plugin/mcp-bridge.t seems not effected by this PR, maybe need retry the CI, thanks :)

The main branch has fixed this bug.

jizhuozhi · 2025-11-24T08:56:42Z

I will see the reason and fix the testcases

silence8013 · 2025-12-18T10:21:05Z

This feature is fantastic, I can't wait!

Baoyuantop · 2025-12-19T08:12:35Z

Hi @jizhuozhi, following up on the previous review comments. Let us know if you have an update. Thanks!

jizhuozhi · 2025-12-27T12:06:07Z

I apologize that due to business demands, KPI pressure, and year-end performance reviews, my response to this request was delayed in Q4. I can now dedicate my full attention to this PR. I apologize again for the delay.

jizhuozhi · 2026-01-03T14:54:57Z

The past failed job seems not caused by this PR change, I have merged latest master branch, please retry it, thanks.

github-actions · 2026-03-05T10:16:39Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 8 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-11T09:24:52Z

+APISIX supports filtering service instances based on metadata. When a route is configured with metadata conditions, only service instances whose metadata matched with roles specified in the route's `metadata` configuration will be selected.
+
+Example: If a service instance has metadata `{lane: "a", env: "prod", version: "1.0"}`, it will match routes configured with metadata `{lane: ["a"]}` or `{lane: ["a", "b"], env: "prod"}`, but not routes configured with `{lane: ["c"]}` or `{lane: "a", region: "us"}`.


The docs/examples here use discovery_args.metadata but the wording says “route’s metadata configuration”, which is misleading (this is upstream discovery_args). Also, the example env: "prod" is a string, while the schema expects an array of strings for every key (eg env: ["prod"]). Please align the wording and examples with the actual config shape.

Suggested change

APISIX supports filtering service instances based on metadata. When a route is configured with metadata conditions, only service instances whose metadata matched with roles specified in the route's `metadata` configuration will be selected.

Example: If a service instance has metadata `{lane: "a", env: "prod", version: "1.0"}`, it will match routes configured with metadata `{lane: ["a"]}` or `{lane: ["a", "b"], env: "prod"}`, but not routes configured with `{lane: ["c"]}` or `{lane: "a", region: "us"}`.

APISIX supports filtering service instances based on metadata. When an upstream is configured with metadata conditions via `discovery_args.metadata`, only service instances whose metadata matches the values specified in `discovery_args.metadata` will be selected.

Example: If a service instance has metadata `{lane: "a", env: "prod", version: "1.0"}`, it will match upstreams configured with `discovery_args.metadata` `{lane: ["a"]}` or `{lane: ["a", "b"], env: ["prod"]}`, but not upstreams configured with `{lane: ["c"]}` or `{lane: ["a"], region: ["us"]}`.

Copilot · 2026-03-11T09:24:53Z

+
+This route will only route traffic to service instances that have both `lane: "a"` and `env: "prod"` in their metadata.
+
 You could find more usage in the `apisix/t/discovery/stream/consul.t` file.


The reference path apisix/t/discovery/stream/consul.t doesn’t exist in the repo; the tests live under t/discovery/stream/consul.t. Update this link/path so readers can actually find the example.

Suggested change

You could find more usage in the `apisix/t/discovery/stream/consul.t` file.

You could find more usage in the `t/discovery/stream/consul.t` file.

Copilot · 2026-03-11T09:24:53Z

+| Name           | Type   | Requirement | Default | Valid | Description                                                  |
+|----------------| ------ | ----------- | ------- | ----- | ------------------------------------------------------------ |
+| metadata | object | optional    | {}      |       | Filter service instances by metadata using containment matching |


The markdown table under ### discovery_args starts rows with ||, which renders as an extra empty column in most markdown parsers. It should start with a single leading | for each row (header, separator, and data rows).

Copilot · 2026-03-11T09:24:53Z

+local function do_metadata_match(node, expected_metadata)
+    local metadata = node.metadata
+    -- because metadata has already been checked in nodes_metadata,
+    -- there is at least one role, if there is no metadata in node, it's must not matched


Comment grammar reads awkwardly (“it's must not matched”). Please correct the wording so the intent is clear.

Suggested change

-- there is at least one role, if there is no metadata in node, it's must not matched

-- there is at least one role, so if a node has no metadata, it cannot match

Copilot · 2026-03-11T09:24:53Z

+local function nodes_metadata(nodes, metadata)
+    if not nodes then
+        return nil
+    end
+
+    -- fast path: there is not metadata roles, all nodes are available,
+    -- and make a guarantee for do_metadata_match: at least one role
+    if not metadata then
+        return nodes
+    end


nodes_metadata treats an empty table ({}) as an active filter, which causes an unnecessary allocation + per-request table churn while still matching all nodes (because pairs({}) is empty). Consider treating empty metadata as “no filter” (eg if not metadata or next(metadata) == nil then return nodes end) and adjust the comment that claims “at least one role” accordingly.

Copilot · 2026-03-11T09:24:54Z

+                    type = "object",
+                    additionalProperties = {
+                        type = "array",
+                        items = {
+                            description = "candidate metadata value",
+                            type = "string",
+                        },


The new discovery_args.metadata schema allows empty arrays for a key (eg {"version": []}), which will silently filter out all nodes at runtime. If that’s not intentional, add minItems = 1 (and potentially minProperties = 1 at the object level) so invalid filters are rejected at validation time.

Suggested change

type = "object",

additionalProperties = {

type = "array",

items = {

description = "candidate metadata value",

type = "string",

},

type = "object",

minProperties = 1,

additionalProperties = {

type = "array",

items = {

description = "candidate metadata value",

type = "string",

},

minItems = 1,

Copilot · 2026-03-11T09:24:54Z

                    type = "string",
                },
+                metadata = {
+                    description = "metadata for filtering service instances",


This change introduces discovery_args.metadata on the generic upstream schema, but only Consul appears to apply it in dis.nodes(...) right now. Consider clarifying in the schema description (or docs) that this field is currently only effective for specific discovery types (eg consul) to avoid configs being accepted but ignored for other discovery backends.

Suggested change

description = "metadata for filtering service instances",

description = "metadata for filtering service instances, only effective for specific discovery types (e.g., consul)",

Copilot · 2026-03-11T09:24:54Z

                        core.table.insert(nodes, {
                            host = svc_address,
                            port = tonumber(svc_port),
                            weight = default_weight,
+                            metadata = metadata
                        })


The PR description mentions “respect its weight if available (aligned with Eureka)”, but nodes are still created with weight = default_weight unconditionally. If Consul service meta is expected to carry a weight value (eg metadata.weight), it should be applied here (and removed from metadata like Eureka does) or the description/docs should be updated to avoid implying weight support.

moonming

Hi @jizhuozhi, thank you for the Consul metadata filtering support!

This is a highly requested feature (31 reviews!) for canary/lane routing with Consul-based service discovery. Combined with #12445 (Nacos metadata filtering), this covers the two most popular discovery backends.

Current state: The PR is marked as CONFLICTING. To move forward:

Please rebase on the latest master to resolve conflicts
Confirm all 31 review comments have been addressed
Consider coordinating with #12445 (Nacos) to ensure a consistent metadata filtering API across discovery backends

Once rebased, I'd like to prioritize this for merge. Thank you for the substantial community engagement! 🙏

github-actions · 2026-05-15T10:50:50Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

github-actions · 2026-06-12T11:20:17Z

This pull request/issue has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Jul 20, 2025

jizhuozhi mentioned this pull request Jul 21, 2025

feat(nacos): add metadata filtering support to nacos discovery #12445

Open

5 tasks

moonming requested a review from Copilot July 21, 2025 08:31

This comment was marked as outdated.

Sign in to view

jizhuozhi force-pushed the master branch from fa69d85 to 043128e Compare July 25, 2025 06:38

jizhuozhi changed the title ~~feat(upstream): filter nodes in upstream with metadata~~ feat(consul): filter nodes in upstream with metadata Jul 25, 2025

jizhuozhi mentioned this pull request Jul 26, 2025

feat: I hope we could filter nodes via metadata when config upstream #12464

Closed

dosubot Bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jul 27, 2025

jizhuozhi force-pushed the master branch from dd1fcfd to 70e3a82 Compare July 27, 2025 18:10

Baoyuantop added this to ⚡️ Apache APISIX Roadmap Jul 28, 2025

Baoyuantop moved this to 👀 In review in ⚡️ Apache APISIX Roadmap Jul 28, 2025

Baoyuantop assigned jizhuozhi Jul 28, 2025

SkyeYoung reviewed Jul 31, 2025

View reviewed changes

Baoyuantop added the wait for update wait for the author's response in this issue/PR label Aug 1, 2025

feat(consul): filter nodes in upstream with metadata

0e23e48

Merge branch 'apache:master' into master

788e68d

Baoyuantop reviewed Dec 19, 2025

View reviewed changes

Comment thread apisix/utils/discovery.lua Outdated

Baoyuantop added wait for update wait for the author's response in this issue/PR and removed user responded labels Dec 24, 2025

jizhuozhi added 2 commits December 27, 2025 19:11

Merge branch 'apache:master' into master

1b25fd5

fix variable names

a9e2932

github-actions Bot added the user responded label Dec 27, 2025

Merge branch 'apache:master' into master

a160edb

Baoyuantop added awaiting review and removed wait for update wait for the author's response in this issue/PR user responded labels Jan 4, 2026

github-actions Bot added the stale label Mar 5, 2026

Baoyuantop requested a review from Copilot March 11, 2026 09:15

Copilot started reviewing on behalf of Baoyuantop March 11, 2026 09:16 View session

Copilot AI reviewed Mar 11, 2026

View reviewed changes

github-actions Bot removed the stale label Mar 11, 2026

moonming requested changes Mar 16, 2026

View reviewed changes

Baoyuantop mentioned this pull request Apr 9, 2026

feat: Filter Consul services by metadata or tags #13184

Open

github-actions Bot added the stale label May 15, 2026

github-actions Bot closed this Jun 12, 2026

github-project-automation Bot moved this from 👀 In review to ✅ Done in ⚡️ Apache APISIX Roadmap Jun 12, 2026

		APISIX supports filtering service instances based on metadata. When a route is configured with metadata conditions, only service instances whose metadata matched with roles specified in the route's `metadata` configuration will be selected.

		Example: If a service instance has metadata `{lane: "a", env: "prod", version: "1.0"}`, it will match routes configured with metadata `{lane: ["a"]}` or `{lane: ["a", "b"], env: "prod"}`, but not routes configured with `{lane: ["c"]}` or `{lane: "a", region: "us"}`.


		This route will only route traffic to service instances that have both `lane: "a"` and `env: "prod"` in their metadata.

		You could find more usage in the `apisix/t/discovery/stream/consul.t` file.

	You could find more usage in the `apisix/t/discovery/stream/consul.t` file.
	You could find more usage in the `t/discovery/stream/consul.t` file.

	-- there is at least one role, if there is no metadata in node, it's must not matched
	-- there is at least one role, so if a node has no metadata, it cannot match

	description = "metadata for filtering service instances",
	description = "metadata for filtering service instances, only effective for specific discovery types (e.g., consul)",

Uh oh!

Conversation

jizhuozhi commented Jul 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation

Changes

Example Usage

Fixes

Checklist

Uh oh!

Baoyuantop commented Jul 21, 2025

Uh oh!

jizhuozhi commented Jul 21, 2025

Uh oh!

Baoyuantop commented Jul 21, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

jizhuozhi commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Baoyuantop commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jizhuozhi commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Baoyuantop commented Jul 25, 2025

Uh oh!

jizhuozhi commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jizhuozhi commented Jul 25, 2025

Uh oh!

jizhuozhi commented Jul 25, 2025

Uh oh!

Baoyuantop commented Jul 25, 2025

Uh oh!

jizhuozhi commented Jul 25, 2025

Uh oh!

Baoyuantop commented Jul 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SkyeYoung Jul 31, 2025

Choose a reason for hiding this comment

Uh oh!

jizhuozhi Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

SkyeYoung Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jizhuozhi Aug 4, 2025

Choose a reason for hiding this comment

Uh oh!

Baoyuantop commented Nov 21, 2025

Uh oh!

jizhuozhi commented Nov 24, 2025

Uh oh!

silence8013 commented Dec 18, 2025

Uh oh!

Uh oh!

Baoyuantop commented Dec 19, 2025

Uh oh!

jizhuozhi commented Dec 27, 2025

Uh oh!

jizhuozhi commented Jan 3, 2026

Uh oh!

github-actions Bot commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

jizhuozhi commented Jul 20, 2025 •

edited

Loading

jizhuozhi commented Jul 21, 2025 •

edited

Loading

Baoyuantop commented Jul 22, 2025 •

edited

Loading

jizhuozhi commented Jul 22, 2025 •

edited

Loading

jizhuozhi commented Jul 25, 2025 •

edited

Loading

SkyeYoung Aug 4, 2025 •

edited

Loading