AI Sampling Chain Test

A five-service Azure Functions test app used to validate Application Insights ingestion sampling behaviour: when many spans across multiple services share a single operation_Id (W3C trace id), do they all survive ingestion sampling together, or does sampling drop them independently?

All five services are deployed as separate Azure Function Apps, but share one Application Insights resource, so a single end-to-end request produces spans under one operation_Id across all five cloud_RoleNames.

Each service is a clone of ../otel-esbuild-esm-static-loader-import-azure-external-azure-function-prewarm-without-node-options (esbuild + ESM + --loader=import-in-the-middle/hook.mjs --import ./dist/src/opentelemetry.mjs) with Key Vault and the prewarm logic stripped. The OTel bootstrap (src/opentelemetry.ts) is identical in every service.

Architecture

sequenceDiagram
    autonumber
    participant C as Client
    participant GO as ai-sampling-get-order
    participant GP as ai-sampling-get-product
    participant RO as ai-sampling-reprice-order
    participant CO as ai-sampling-capture-order
    participant SB as Service Bus<br/>queue: orders
    participant PO as ai-sampling-place-order

    C->>GO: GET /api/order/{id}
    GO->>GP: GET /api/product/{pid} (x N)
    GP-->>GO: product
    GO->>RO: POST /api/reprice (order)
    RO-->>GO: order + total
    GO->>CO: POST /api/capture (order)
    CO->>SB: send message (traceparent attached)
    CO-->>GO: 202 captured
    GO-->>C: 200 order + capture
    SB-->>PO: trigger (continues trace)
    PO->>RO: POST /api/reprice (re-price before persist)
    RO-->>PO: order + total
    PO->>PO: mock persist

Loading

Propagation:

• HTTP hops: @opentelemetry/instrumentation-http + instrumentation-undici + axios propagate traceparent automatically.
• Service Bus hop: @azure/opentelemetry-instrumentation-azure-sdk instruments @azure/service-bus so the producer span attaches the W3C trace context to the SB message. The capture-order handler also adds an explicit applicationProperties.traceparent for visibility. @azure/functions-opentelemetry-instrumentation continues the trace on the consumer side when the Service Bus trigger fires place-order.

Services

Folder	Function App name	Trigger	Route	Notes
get-order/	${PREFIX}-get-order	HTTP GET	/api/order/{id}	Entry. Orchestrates the chain.
get-product/	${PREFIX}-get-product	HTTP GET	/api/product/{id}	Mock product.
reprice-order/	${PREFIX}-reprice-order	HTTP POST	/api/reprice	Adds tax, computes total. Called twice.
capture-order/	${PREFIX}-capture-order	HTTP POST	/api/capture	Publishes to Service Bus.
place-order/	${PREFIX}-place-order	Service Bus queue	n/a	Triggered by SB; re-prices then persists (mock).

Deploy

# Optional overrides
export RESOURCE_GROUP_NAME=playground-kamil-ai-sampling
export LOCATION=westeurope
export PREFIX=ai-sampling-abcd      # function app name prefix
./deploy-all.sh

The script provisions:

• Resource group
• Storage account (shared)
• Application Insights (shared) → APPLICATIONINSIGHTS_CONNECTION_STRING
• Service Bus namespace + orders queue → SERVICEBUS_CONNECTION
• Linux Flex Consumption plan per app (auto-created by az functionapp create)
• 5 Function Apps with shared OTel bootstrap (languageWorkers__node__arguments)

…then runs each service's deploy.sh to build (esbuild) and func azure functionapp publish.

Run

export ENTRY_URL="https://${PREFIX}-get-order.azurewebsites.net"
./request.sh

Capture the traceparent header in the response — the middle 32-hex segment is the operation_Id you'll query in App Insights.

To generate load:

for i in $(seq 1 200); do ORDER_ID="load-$i" ./request.sh; done

Validate distributed trace

union requests, dependencies, traces, exceptions
| where operation_Id == "<TRACE_ID>"
| summarize spans = count() by cloud_RoleName, itemType
| order by cloud_RoleName asc

Expected: rows for all five cloud_RoleNames (ai-sampling-…-get-order, …-get-product, …-reprice-order (appears twice in the chain so will have ~2× the spans), …-capture-order, …-place-order).

Ingestion sampling experiment

By default Application Insights enables adaptive sampling at the SDK level. To exercise ingestion sampling (the mechanism the linked docs warn against), set it on the App Insights resource:

az monitor app-insights component update \
  --app "${APP_INSIGHTS_NAME}" \
  --resource-group "${RESOURCE_GROUP_NAME}" \
  --sampling-percentage 20

Re-run the load loop, then ask:

let operationIds = requests
  | where cloud_RoleName == "${PREFIX}-get-order"
  | where timestamp > ago(30m)
  | distinct operation_Id;
union requests, dependencies
| where timestamp > ago(30m)
| where operation_Id in (operationIds)
| summarize spans = count(), services = dcount(cloud_RoleName) by operation_Id
| summarize complete = countif(services == 5), partial = countif(services < 5)

If ingestion sampling truly preserves full traces by operation_Id, partial should be 0. The Microsoft docs say it doesn't — and this experiment shows the gap.

To reset: set --sampling-percentage 100.

Teardown

RESOURCE_GROUP_NAME=playground-kamil-ai-sampling ./teardown.sh

Notes

• Service Bus integration uses the legacy app.serviceBusQueue trigger with the extension bundle, matching the rest of this repo.
• All five services run on Flex Consumption (each app auto-provisions its own plan; Flex billing is per-execution so plan count is cost-neutral). Cold starts are expected; the OTel bootstrap fires before the worker hands the request to user code via the --import arg.
• The shared src/opentelemetry.ts registers HttpInstrumentation, UndiciInstrumentation, the Azure SDK instrumentation (which covers @azure/service-bus), and the Azure Functions OTel instrumentation — together they cover every hop in the chain.