input: add pause/resume callbacks

[edsiper]

This patch implements the missing pause and resume callbacks for in_tcp, in_http and in_opentelemetry plugins. When the plugin is paused, all active client connections are forcefully closed, and new incoming connections are rejected. The
plugin tracks its paused state to ensure correct behavior until it is resumed.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features
  • Added public HTTP server pause and resume controls.
  • Enabled pause/resume lifecycle support for Elasticsearch, HTTP, OpenTelemetry, Prometheus remote write, and Splunk, including worker-based and HTTP/2+TLS scenarios.
• Bug Fixes
  • Improved paused behavior by deferring pending work and ensuring stalled connections are handled safely.
• Tests
  • Added pause/resume integration scenarios and a shared testing utility, plus new Elasticsearch/HTTP/Splunk/OTLP/Prometheus remote write coverage and a unit test for safe session destruction.

[edsiper]

note: this PR needs to be based on top of #10790 (new downstream pause/resume functionality)

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Changes

The HTTP server now supports pause/resume lifecycle control through downstream connection management. Supported HTTP input plugins register lifecycle callbacks, and integration tests cover protocols, worker modes, connection teardown, metrics, shutdown behavior, and Valgrind execution.

Changes

HTTP server pause and resume

Layer / File(s)	Summary
Downstream accept and connection lifecycle include/fluent-bit/flb_connection.h, include/fluent-bit/flb_downstream.h, src/flb_downstream.c, src/flb_network.c	Adds accept callbacks, asynchronous connection coroutines, bulk connection release, event registration, and refined nonblocking accept handling.
HTTP server pause/resume implementation include/fluent-bit/http_server/..., src/http_server/...	Adds public pause/resume APIs, paused connection handling, deferred session teardown, async startup, worker dispatch, and re-entrant destruction protection.
Input plugin lifecycle wiring plugins/in_elasticsearch/..., plugins/in_http/..., plugins/in_opentelemetry/..., plugins/in_prometheus_remote_write/..., plugins/in_splunk/...	Registers pause and resume callbacks for each plugin’s HTTP server context.
Integration validation tests/integration/..., tests/internal/http_server.c	Adds shared helpers and scenarios covering pause state metrics, rejected requests, stalled connection closure, resumed requests, shutdown, and session destruction.
Focused CI validation .github/workflows/input-http-pause-tests.yaml	Builds the project and runs focused pause/lifecycle tests both normally and under strict Valgrind.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant InputPlugin
  participant HTTPServer
  participant Downstream
  participant MetricsAPI
  Client->>InputPlugin: Send pause-triggering request
  InputPlugin->>HTTPServer: Pause HTTP server
  HTTPServer->>Downstream: Pause and release connections
  MetricsAPI-->>Client: Report paused input state
  Client->>InputPlugin: Send request while paused
  InputPlugin-->>Client: Close or reject connection
  Client->>InputPlugin: Send resume request
  InputPlugin->>HTTPServer: Resume HTTP server
  HTTPServer->>Downstream: Reap stale sessions and resume
  MetricsAPI-->>Client: Report resumed input state

Loading

Possibly related PRs

• fluent/fluent-bit#10790: Adds downstream pause/resume interfaces used by this HTTP server lifecycle implementation.
• fluent/fluent-bit#11742: Overlaps with HTTP server session lifecycle and stale-session teardown paths.
• fluent/fluent-bit#11775: Refactors HTTP server worker/runtime handling related to this lifecycle implementation.

Suggested reviewers: cosmo0920, fujimotos

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding pause/resume callbacks for input plugins.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch input-plugins-pause-callbacks

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/src/utils/input_pause_resume.py`:
- Around line 61-84: Update the pause/resume test around
wait_for_input_pause_state to establish and retain a client socket or session
before triggering the pause, then verify that the already-active connection is
closed after pause becomes active. Preserve the existing paused-state request
assertion so both active-session teardown and rejection of new requests remain
covered.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6ecd7cd5-20e2-471b-98c0-f14e44767f70

📥 Commits

Reviewing files that changed from the base of the PR and between dad5356 and c123f0e.

📒 Files selected for processing (14)

• include/fluent-bit/http_server/flb_http_server.h
• plugins/in_elasticsearch/in_elasticsearch.c
• plugins/in_http/http.c
• plugins/in_opentelemetry/opentelemetry.c
• plugins/in_prometheus_remote_write/prom_rw.c
• plugins/in_splunk/splunk.c
• src/http_server/flb_http_server.c
• tests/integration/scenarios/in_elasticsearch/config/in_elasticsearch_pause_resume.yaml
• tests/integration/scenarios/in_elasticsearch/tests/test_in_elasticsearch_001.py
• tests/integration/scenarios/in_http/config/in_http_pause_resume.yaml
• tests/integration/scenarios/in_http/tests/test_in_http_001.py
• tests/integration/scenarios/in_splunk/config/splunk_pause_resume.yaml
• tests/integration/scenarios/in_splunk/tests/test_in_splunk_001.py
• tests/integration/src/utils/input_pause_resume.py

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/integration/scenarios/in_http/tests/test_in_http_001.py`:
- Around line 160-170: Update the pause/resume test around
assert_pause_resume_cycles to pass partial_connection into the helper, then
assert connection closure immediately whenever each paused-state observation
occurs rather than only after all cycles complete. Preserve validation of both
successful cycles and failed/paused teardown behavior, and remove the delayed
standalone assertion if the helper now performs it.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 9b8ed12b-ba84-4afd-81f1-6304b412a76a

📥 Commits

Reviewing files that changed from the base of the PR and between c123f0e and 82f3e6f.

📒 Files selected for processing (14)

• include/fluent-bit/http_server/flb_http_server.h
• plugins/in_elasticsearch/in_elasticsearch.c
• plugins/in_http/http.c
• plugins/in_opentelemetry/opentelemetry.c
• plugins/in_prometheus_remote_write/prom_rw.c
• plugins/in_splunk/splunk.c
• src/http_server/flb_http_server.c
• tests/integration/scenarios/in_elasticsearch/config/in_elasticsearch_pause_resume.yaml
• tests/integration/scenarios/in_elasticsearch/tests/test_in_elasticsearch_001.py
• tests/integration/scenarios/in_http/config/in_http_pause_resume.yaml
• tests/integration/scenarios/in_http/tests/test_in_http_001.py
• tests/integration/scenarios/in_splunk/config/splunk_pause_resume.yaml
• tests/integration/scenarios/in_splunk/tests/test_in_splunk_001.py
• tests/integration/src/utils/input_pause_resume.py