diff --git a/docs/snapshot-serving-plan.md b/docs/snapshot-serving-plan.md index 9b46d556c0..0500a75a5d 100644 --- a/docs/snapshot-serving-plan.md +++ b/docs/snapshot-serving-plan.md @@ -252,7 +252,7 @@ The peers endpoint is a separate feature and was not implemented in this phase. ### Configuration -A single CLI-only option (not config file — single-use bootstrap): +The endpoint is CLI-only because bootstrap is a single-use operation: ``` --snapshot-endpoint URL Fetch snapshot from URL and bootstrap. @@ -261,6 +261,12 @@ A single CLI-only option (not config file — single-use bootstrap): https://snap.example.com/50000 → fetches block 50000 ``` +Snapshot bootstrap is an attended operation against an operator-selected endpoint. It reports connection/request phases +and, every five seconds during transfer, downloaded bytes, percentage, rate, and ETA when `Content-Length` is available. +SIGINT cancels pending resolver or socket work and retains atomic temporary-file cleanup. The existing +`chain-state-db-size-mb` setting bounds both fixed-length and chunked response bodies. No snapshot-specific resource +options are added. + The block number is encoded as a trailing path component of the URL. If the last path segment is a decimal number, it's treated as a specific block request (POST to `/v1/snapshot/by_block`); otherwise POST to `/v1/snapshot/latest`. ### Files @@ -281,7 +287,9 @@ The block number is encoded as a trailing path component of the URL. If the last ``` This works naturally with `--delete-all-blocks` which clears state before snapshot handling. 3. **Fetch metadata:** POST to `/v1/snapshot/latest` or `/v1/snapshot/by_block` depending on URL format. -4. **Download snapshot:** Uses `fc::http_client::post_to_file()` to POST to `/v1/snapshot/download` and save binary response to local snapshots directory. +4. **Download snapshot:** Uses bounded `fc::http_client::post_to_file()` streaming to POST to + `/v1/snapshot/download`, enforce size and disk-headroom bounds, and atomically save the response to the local + snapshots directory. 5. **Root hash verification:** Uses `threaded_snapshot_reader::load_index()` to read the footer and compare the stored root hash against the advertised `root_hash`. This is a fast metadata-only check that catches download corruption. Full integrity verification (re-hashing all sections) happens during snapshot loading, and on-chain attestation verification happens after syncing. 6. **Continue normal loading:** Sets `snapshot_path` to downloaded file and `snapshot_auto_fetched = true`. No `--genesis-json` needed — snapshot contains genesis. diff --git a/libraries/libfc/include/fc/network/http/http_client.hpp b/libraries/libfc/include/fc/network/http/http_client.hpp index e34b9d6727..4f62f7825f 100644 --- a/libraries/libfc/include/fc/network/http/http_client.hpp +++ b/libraries/libfc/include/fc/network/http/http_client.hpp @@ -11,10 +11,44 @@ #include #include +#include #include +#include +#include namespace fc { +/** Current phase of a streamed HTTP file download. */ +enum class http_file_download_phase { + connecting, + sending_request, + waiting_for_response, + downloading, + complete, +}; + +/** Periodic status for a streamed HTTP file download. */ +struct http_file_download_status { + /// Current request or transfer phase. + http_file_download_phase phase; + /// Decoded response-body bytes persisted so far. + uint64_t downloaded_bytes; + /// Expected response-body bytes when the peer supplied Content-Length. + std::optional total_bytes; + /// Time elapsed since the download request started. + microseconds elapsed; +}; + +/** Resource limits and status reporting for a streamed HTTP file download. */ +struct http_file_download_options { + /// Maximum accepted response-body size in bytes. + uint64_t max_response_body_bytes; + /// Retry a failed request once when it used a cached connection. Only enable for idempotent requests. + bool retry_failed_reused_connection = false; + /// Receive phase transitions and periodic transfer status. The callback must not throw. + std::function status_callback; +}; + class http_client { public: http_client(); @@ -29,19 +63,36 @@ class http_client { return post_sync(dest, payload_v, deadline); } - /// Download binary response from a POST request to a file. - /// Uses a temp file during download and renames on completion. - /// Clean up on failure. + /** + * Download a binary POST response using explicit resource limits. + * + * The response is written to a temporary sibling of @p output and renamed only after + * the complete bounded body has been persisted successfully. + */ void post_to_file(const url& dest, const variant& payload, const std::filesystem::path& output, - const time_point& deadline = time_point::maximum()); + const http_file_download_options& options); + + /** + * Set a predicate used to cancel synchronous HTTP operations. + * + * The predicate is polled while resolver and socket operations are pending. Returning true + * cancels the active operation so callers can interrupt otherwise unbounded requests. + */ + void set_cancel_check(std::function cancel_check); void add_cert(const std::string& cert_pem_string); void set_verify_peers(bool enabled); private: + friend struct http_client_test_access; + + /// Override the filesystem free-space query for deterministic transport tests. + void set_space_available_provider_for_testing( + std::function provider); + std::unique_ptr _my; }; -} \ No newline at end of file +} diff --git a/libraries/libfc/src/network/http/http_client.cpp b/libraries/libfc/src/network/http/http_client.cpp index 20d1594ac4..bf68791137 100644 --- a/libraries/libfc/src/network/http/http_client.cpp +++ b/libraries/libfc/src/network/http/http_client.cpp @@ -3,13 +3,20 @@ #include #include +#include +#include #include +#include +#include +#include +#include #include #include #include #include #include +#include #include #include @@ -19,6 +26,20 @@ namespace local = boost::asio::local; namespace fc { +namespace { + +// Beast limits an individual parser read to roughly 64 KiB, so a larger body buffer is unused. +constexpr size_t download_read_buffer_bytes = 64 * 1024; +constexpr size_t download_parser_buffer_bytes = 64 * 1024; +constexpr size_t max_error_response_body_bytes = 200; +constexpr uint64_t disk_space_check_interval_bytes = 64 * 1024 * 1024; +constexpr uint64_t disk_space_concurrency_margin_bytes = 64 * 1024 * 1024; +constexpr auto disk_space_check_interval = std::chrono::seconds(5); +constexpr auto operation_monitor_interval = std::chrono::milliseconds(100); +constexpr auto download_status_interval = std::chrono::seconds(5); + +} // namespace + /** * mapping of protocols to their standard ports */ @@ -36,6 +57,7 @@ class http_client_impl { using unix_url_split_map = std::map; using error_code = boost::system::error_code; using deadline_type = std::chrono::system_clock::time_point; + using operation_monitor = std::function; http_client_impl() :_ioc() @@ -47,113 +69,264 @@ class http_client_impl { void set_verify_peers(bool enabled) {} + void set_cancel_check(std::function cancel_check) { + _cancel_check = std::move(cancel_check); + } + + void set_space_available_provider_for_testing( + std::function provider) { + _space_available_provider = std::move(provider); + } + + /** Return whether the caller requested cancellation of the active HTTP operation. */ + bool is_cancelled() const { + return _cancel_check && _cancel_check(); + } + template - error_code sync_do_with_deadline( SyncReadStream& s, deadline_type deadline, Fn f, CancelFn cf ) { - bool timer_expired = false; - boost::asio::system_timer timer(_ioc); - - timer.expires_at(deadline); - bool timer_cancelled = false; - timer.async_wait([&timer_expired, &timer_cancelled] (const error_code&) { - // the only non-success error_code this is called with is operation_aborted but since - // we could have queued "success" when we cancelled the timer, we set a flag at the - // safer scope and only respect that. - if (!timer_cancelled) { - timer_expired = true; - } - }); + error_code sync_do_with_deadline_impl(SyncReadStream& s, deadline_type deadline, Fn f, CancelFn cf, + const operation_monitor& monitor = {}) { + if (is_cancelled()) { + return boost::asio::error::operation_aborted; + } + const bool has_deadline = deadline != deadline_type::max(); + bool deadline_expired = false; + bool deadline_cancelled = false; + boost::asio::system_timer deadline_timer(_ioc); + if (has_deadline) { + deadline_timer.expires_at(deadline); + deadline_timer.async_wait([&deadline_expired, &deadline_cancelled](const error_code&) { + // The only non-success error_code expected here is operation_aborted. A success + // completion may already be queued when cancellation occurs, so use the explicit + // flag to decide whether the deadline still applies. + if (!deadline_cancelled) { + deadline_expired = true; + } + }); + } + + bool operation_cancelled = false; + boost::asio::steady_timer monitor_timer(_ioc); std::optional f_result; + std::function arm_monitor; + if (_cancel_check || monitor) { + arm_monitor = [&]() { + monitor_timer.expires_after(operation_monitor_interval); + monitor_timer.async_wait([&](const error_code& ec) { + if (ec == boost::asio::error::operation_aborted || f_result) { + return; + } + if (is_cancelled()) { + operation_cancelled = true; + cf(); + return; + } + if (monitor) { + monitor(); + } + arm_monitor(); + }); + }; + arm_monitor(); + } + f(f_result); _ioc.restart(); - while (_ioc.run_one()) - { + while (_ioc.run_one()) { if (f_result) { - timer_cancelled = true; - timer.cancel(); - } else if (timer_expired) { + deadline_cancelled = true; + if (has_deadline) { + deadline_timer.cancel(); + } + monitor_timer.cancel(); + } else if (deadline_expired) { cf(); } } - if (!timer_expired) { - return *f_result; - } else { + if (operation_cancelled) { + return boost::asio::error::operation_aborted; + } + if (deadline_expired) { return error_code(boost::system::errc::timed_out, boost::system::system_category()); } + return *f_result; } template - error_code sync_do_with_deadline( SyncReadStream& s, deadline_type deadline, Fn f) { - return sync_do_with_deadline(s, deadline, f, [&s](){ - s.lowest_layer().cancel(); - }); - }; + error_code sync_do_with_deadline(SyncReadStream& s, deadline_type deadline, Fn f, + const operation_monitor& monitor = {}) { + return sync_do_with_deadline_impl(s, deadline, f, [&s]() { + error_code ec; + s.lowest_layer().cancel(ec); + }, monitor); + } template - error_code sync_connect_with_timeout( SyncReadStream& s, const std::string& host, const std::string& port, const deadline_type& deadline ) { + error_code sync_connect_with_timeout(SyncReadStream& s, const std::string& host, const std::string& port, + const deadline_type& deadline, const operation_monitor& monitor = {}) { tcp::resolver local_resolver(_ioc); bool cancelled = false; - auto res = sync_do_with_deadline(s, deadline, [&local_resolver, &cancelled, &s, &host, &port](std::optional& final_ec){ - local_resolver.async_resolve(host, port, [&cancelled, &s, &final_ec](const error_code& ec, tcp::resolver::results_type resolved ){ - if (ec) { - final_ec.emplace(ec); - return; - } - - if (!cancelled) { - boost::asio::async_connect(s, resolved.begin(), resolved.end(), [&final_ec](const error_code& ec, auto ){ - final_ec.emplace(ec); + auto res = sync_do_with_deadline_impl( + s, deadline, + [&local_resolver, &cancelled, &s, &host, &port](std::optional& final_ec) { + local_resolver.async_resolve( + host, port, + [&cancelled, &s, &final_ec](const error_code& ec, tcp::resolver::results_type resolved) { + if (ec) { + final_ec.emplace(ec); + return; + } + + if (!cancelled) { + boost::asio::async_connect( + s, resolved.begin(), resolved.end(), + [&final_ec](const error_code& ec, auto) { final_ec.emplace(ec); }); + } }); - } - }); - },[&local_resolver, &cancelled](){ - cancelled = true; - local_resolver.cancel(); - }); + }, + [&local_resolver, &cancelled, &s]() { + cancelled = true; + local_resolver.cancel(); + error_code ec; + s.lowest_layer().cancel(ec); + }, + monitor); return res; }; template - error_code sync_write_with_timeout(SyncReadStream& s, http::request& req, const deadline_type& deadline ) { + error_code sync_write_with_timeout(SyncReadStream& s, http::request& req, + const deadline_type& deadline, const operation_monitor& monitor = {}) { return sync_do_with_deadline(s, deadline, [&s, &req](std::optional& final_ec){ http::async_write(s, req, [&final_ec]( const error_code& ec, std::size_t ) { final_ec.emplace(ec); }); - }); + }, monitor); } template - error_code sync_read_with_timeout(SyncReadStream& s, boost::beast::flat_buffer& buffer, http::response& res, const deadline_type& deadline ) { + error_code sync_read_with_timeout(SyncReadStream& s, boost::beast::flat_buffer& buffer, + http::response& res, const deadline_type& deadline, + const operation_monitor& monitor = {}) { return sync_do_with_deadline(s, deadline, [&s, &buffer, &res](std::optional& final_ec){ http::async_read(s, buffer, res, [&final_ec]( const error_code& ec, std::size_t ) { final_ec.emplace(ec); }); - }); + }, monitor); } /// Read only the response header into @p parser, honoring @p deadline. Used by the /// streaming download path to inspect the status line before committing the body to a file. template - error_code sync_read_header_with_timeout(SyncReadStream& s, boost::beast::flat_buffer& buffer, Parser& parser, const deadline_type& deadline ) { + error_code sync_read_header_with_timeout(SyncReadStream& s, boost::beast::flat_buffer& buffer, Parser& parser, + const deadline_type& deadline, + const operation_monitor& monitor = {}) { return sync_do_with_deadline(s, deadline, [&s, &buffer, &parser](std::optional& final_ec){ http::async_read_header(s, buffer, parser, [&final_ec]( const error_code& ec, std::size_t ) { final_ec.emplace(ec); }); - }); + }, monitor); } - /// Read the remainder of the response (the body) into @p parser, honoring @p deadline. + /// Read one increment of a streaming response body into @p parser, honoring @p deadline. template - error_code sync_read_parser_with_timeout(SyncReadStream& s, boost::beast::flat_buffer& buffer, Parser& parser, const deadline_type& deadline ) { - return sync_do_with_deadline(s, deadline, [&s, &buffer, &parser](std::optional& final_ec){ - http::async_read(s, buffer, parser, [&final_ec]( const error_code& ec, std::size_t ) { + error_code sync_read_parser_some_with_timeout(SyncReadStream& s, boost::beast::flat_buffer& buffer, + Parser& parser, const deadline_type& deadline, + const operation_monitor& monitor = {}) { + return sync_do_with_deadline(s, deadline, [&s, &buffer, &parser](std::optional& final_ec) { + http::async_read_some(s, buffer, parser, [&final_ec](const error_code& ec, std::size_t) { final_ec.emplace(ec); }); - }); + }, monitor); + } + + /** + * Read one decoded response-body increment into caller-owned storage. + * + * @p ec retains transport and parser failures for the caller, while need_buffer is normalized + * because it only means the supplied buffer_body storage was filled successfully. + */ + size_t read_body_increment(connection& conn, boost::beast::flat_buffer& buffer, + http::response_parser& parser, char* sink, size_t sink_size, + const operation_monitor& monitor, error_code& ec) { + parser.get().body().data = sink; + parser.get().body().size = sink_size; + ec = std::visit([&](auto& stream) { + return sync_read_parser_some_with_timeout( + *stream, buffer, parser, deadline_type::max(), monitor); + }, conn); + if (ec == http::error::need_buffer) { + ec = {}; + } + return sink_size - parser.get().body().size; + } + + /// Return whether @p ec can indicate that a cached peer connection closed before reuse. + static bool is_stale_connection_error(const error_code& ec) { + return ec == boost::asio::error::eof || + ec == boost::asio::error::connection_reset || + ec == boost::asio::error::connection_aborted || + ec == boost::asio::error::broken_pipe || + ec == boost::asio::error::not_connected || + ec == boost::asio::error::shut_down || + ec == http::error::end_of_stream; + } + + /// Return a path on the destination filesystem that can be queried for free space. + static std::filesystem::path space_query_path(const std::filesystem::path& destination) { + if (destination.has_parent_path()) { + return destination.parent_path(); + } + + std::error_code ec; + auto current = std::filesystem::current_path(ec); + FC_ASSERT(!ec, "Failed to resolve current directory for download disk-space check: {}", ec.message()); + return current; + } + + /// Return available bytes on the destination filesystem, using the test seam when configured. + uint64_t available_disk_space(const std::filesystem::path& query_path) const { + if (_space_available_provider) { + return _space_available_provider(query_path); + } + + std::error_code ec; + const auto space = std::filesystem::space(query_path, ec); + FC_ASSERT(!ec, "Failed to query free disk space at {}: {}", query_path.string(), ec.message()); + return space.available; + } + + /** Require enough free space for @p next_write_bytes. */ + void require_disk_space(const std::filesystem::path& destination, uint64_t next_write_bytes) const { + const auto query_path = space_query_path(destination); + const auto available_bytes = available_disk_space(query_path); + FC_ASSERT(next_write_bytes <= available_bytes, + "Insufficient disk space at {} for HTTP download: {} bytes available and {} bytes required", + query_path.string(), available_bytes, next_write_bytes); + } + + /** + * Verify and return the next cached write budget. + * + * Each budget is capped at 64 MiB and requires a separate 64 MiB concurrency margin. + * The margin bounds headroom exposure if another process consumes space between probes; + * slow transfers also refresh the probe every five seconds. + */ + uint64_t refresh_disk_space_write_budget(const std::filesystem::path& destination, + uint64_t remaining_body_bytes) const { + const auto write_budget = std::min(remaining_body_bytes, disk_space_check_interval_bytes); + if (write_budget == 0) { + require_disk_space(destination, 0); + return 0; + } + + require_disk_space(destination, write_budget + disk_space_concurrency_margin_bytes); + return write_budget; } host_key url_to_host_key( const url& dest ) { @@ -181,11 +354,13 @@ class http_client_impl { return res.first; } - connection_map::iterator create_raw_connection( const url& dest, const deadline_type& deadline ) { + connection_map::iterator create_raw_connection(const url& dest, const deadline_type& deadline, + const operation_monitor& monitor = {}) { auto key = url_to_host_key(dest); auto socket = std::make_unique(_ioc); - error_code ec = sync_connect_with_timeout(*socket, *dest.host(), dest.port() ? std::to_string(*dest.port()) : "80", deadline); + error_code ec = sync_connect_with_timeout( + *socket, *dest.host(), dest.port() ? std::to_string(*dest.port()) : "80", deadline, monitor); FC_ASSERT(!ec, "Failed to connect: {}", ec.message()); auto res = _connections.emplace(std::piecewise_construct, @@ -195,9 +370,10 @@ class http_client_impl { return res.first; } - connection_map::iterator create_connection( const url& dest, const deadline_type& deadline ) { + connection_map::iterator create_connection(const url& dest, const deadline_type& deadline, + const operation_monitor& monitor = {}) { if (dest.proto() == "http") { - return create_raw_connection(dest, deadline); + return create_raw_connection(dest, deadline, monitor); } else if (dest.proto() == "unix") { return create_unix_connection(dest, deadline); } else { @@ -224,50 +400,65 @@ class http_client_impl { } } - connection_map::iterator get_connection( const url& dest, const deadline_type& deadline ) { + connection_map::iterator get_connection(const url& dest, const deadline_type& deadline, + bool* reused_connection = nullptr, + const operation_monitor& monitor = {}) { auto key = url_to_host_key(dest); auto conn_itr = _connections.find(key); if (conn_itr == _connections.end() || check_closed(conn_itr)) { - return create_connection(dest, deadline); + if (reused_connection) { + *reused_connection = false; + } + return create_connection(dest, deadline, monitor); } else { + if (reused_connection) { + *reused_connection = true; + } return conn_itr; } } struct write_request_visitor : visitor { - write_request_visitor(http_client_impl* that, http::request& req, const deadline_type& deadline) + write_request_visitor(http_client_impl* that, http::request& req, + const deadline_type& deadline, const operation_monitor& monitor = {}) :that(that) ,req(req) ,deadline(deadline) + ,monitor(monitor) {} template error_code operator() ( S& stream ) const { - return that->sync_write_with_timeout(*stream, req, deadline); + return that->sync_write_with_timeout(*stream, req, deadline, monitor); } http_client_impl* that; http::request& req; const deadline_type& deadline; + operation_monitor monitor; }; struct read_response_visitor : visitor { - read_response_visitor(http_client_impl* that, boost::beast::flat_buffer& buffer, http::response& res, const deadline_type& deadline) + read_response_visitor(http_client_impl* that, boost::beast::flat_buffer& buffer, + http::response& res, const deadline_type& deadline, + const operation_monitor& monitor = {}) :that(that) ,buffer(buffer) ,res(res) ,deadline(deadline) + ,monitor(monitor) {} template error_code operator() ( S& stream ) const { - return that->sync_read_with_timeout(*stream, buffer, res, deadline); + return that->sync_read_with_timeout(*stream, buffer, res, deadline, monitor); } http_client_impl* that; boost::beast::flat_buffer& buffer; http::response& res; const deadline_type& deadline; + operation_monitor monitor; }; variant post_sync(const url& dest, const variant& payload, const fc::time_point& _deadline) { @@ -388,8 +579,8 @@ class http_client_impl { return _unix_url_paths.emplace(full_url, fc::url("unix", socket_file.string(), ostring(), ostring(), url_path.string(), ostring(), ovariant_object(), std::optional())).first->second; } - void post_to_file(const url& dest, const variant& payload, const std::filesystem::path& final_dest, const fc::time_point& deadline_time) { - auto deadline = deadline_time.to_system_clock(); + void post_to_file(const url& dest, const variant& payload, const std::filesystem::path& final_dest, + const http_file_download_options& options) { FC_ASSERT(dest.host(), "No host set on URL"); std::string path = dest.path() ? dest.path()->generic_string() : "/"; @@ -411,46 +602,134 @@ class http_client_impl { req.set(http::field::user_agent, BOOST_BEAST_VERSION_STRING); req.set(http::field::content_type, "application/json"); req.keep_alive(false); - req.body() = json::to_string(payload, deadline_time); + req.body() = json::to_string(payload, fc::time_point::maximum()); req.prepare_payload(); - auto conn_iter = get_connection(dest, deadline); - auto eraser = make_scoped_exit([this, &conn_iter](){ - _connections.erase(conn_iter); + const auto request_started = std::chrono::steady_clock::now(); + auto next_status_report = request_started; + auto download_phase = http_file_download_phase::connecting; + uint64_t downloaded_bytes = 0; + std::optional response_body_bytes; + auto report_status = [&](bool force) { + if (!options.status_callback) { + return; + } + const auto now = std::chrono::steady_clock::now(); + if (!force && now < next_status_report) { + return; + } + const auto elapsed = std::chrono::duration_cast(now - request_started); + options.status_callback(http_file_download_status{ + .phase = download_phase, + .downloaded_bytes = downloaded_bytes, + .total_bytes = response_body_bytes, + .elapsed = fc::microseconds(elapsed.count()), + }); + next_status_report = now + download_status_interval; + }; + auto transition_to = [&](http_file_download_phase phase) { + download_phase = phase; + report_status(true); + }; + operation_monitor monitor; + if (options.status_callback) { + monitor = [&]() { report_status(false); }; + } + const auto no_deadline = deadline_type::max(); + + bool reused_connection = false; + transition_to(http_file_download_phase::connecting); + auto conn_iter = get_connection(dest, no_deadline, &reused_connection, monitor); + const auto connection_key = url_to_host_key(dest); + auto eraser = make_scoped_exit([this, connection_key](){ + _connections.erase(connection_key); }); - error_code ec = std::visit(write_request_visitor(this, req, deadline), conn_iter->second); - FC_ASSERT(!ec, "Failed to send POST request: {}", ec.message()); + std::unique_ptr buffer; + std::unique_ptr> parser; + error_code ec; + bool retried_stale_connection = false; + while (true) { + transition_to(http_file_download_phase::sending_request); + ec = std::visit( + write_request_visitor(this, req, no_deadline, monitor), conn_iter->second); + if (ec) { + if (options.retry_failed_reused_connection && reused_connection && !retried_stale_connection && + is_stale_connection_error(ec)) { + _connections.erase(conn_iter); + transition_to(http_file_download_phase::connecting); + conn_iter = get_connection(dest, no_deadline, &reused_connection, monitor); + retried_stale_connection = true; + continue; + } + FC_ASSERT(false, "Failed to send POST request: {}", ec.message()); + } - // Stream the response body straight to disk through a file_body parser: snapshot - // downloads are multi-GB, so buffering the payload in a string_body response could - // exhaust memory long before the data reaches its hash check. - boost::beast::flat_buffer buffer; - http::response_parser parser; - // The download exceeds the parser's default body-size limit; the transfer is bounded - // by the caller's deadline and by available disk space instead. - parser.body_limit(boost::none); + // Parse into a bounded caller-owned buffer so every write can enforce the byte ceiling, + // cancellation checks, and filesystem headroom before bytes reach disk. + buffer = std::make_unique(download_parser_buffer_bytes); + parser = std::make_unique>(); + parser->body_limit(options.max_response_body_bytes); - ec = std::visit([&](auto& stream) { - return sync_read_header_with_timeout(*stream, buffer, parser, deadline); - }, conn_iter->second); - FC_ASSERT(!ec, "Failed to read response header: {}", ec.message()); + transition_to(http_file_download_phase::waiting_for_response); + ec = std::visit([&](auto& stream) { + return sync_read_header_with_timeout(*stream, *buffer, *parser, no_deadline, monitor); + }, conn_iter->second); + if (!ec) { + break; + } + if (ec == http::error::body_limit) { + FC_THROW("HTTP response body exceeds configured maximum of {} bytes", options.max_response_body_bytes); + } + if (options.retry_failed_reused_connection && reused_connection && !retried_stale_connection && + is_stale_connection_error(ec)) { + _connections.erase(conn_iter); + transition_to(http_file_download_phase::connecting); + conn_iter = get_connection(dest, no_deadline, &reused_connection, monitor); + retried_stale_connection = true; + continue; + } + FC_ASSERT(false, "Failed to read response header: {}", ec.message()); + } // Require exactly 200 OK: this client never sends Range requests, so 206 Partial Content // can only come from a broken or hostile endpoint/proxy, and renaming a partial body into // the final destination would hand the caller a truncated file. - const auto status = parser.get().result(); + const auto status = parser->get().result(); if (status != http::status::ok) { - // Error responses carry small diagnostic bodies; read into memory for the message. - // Best effort: on a read failure the throw below reports whatever body arrived. - http::response_parser err_parser{std::move(parser)}; - if (!err_parser.is_done()) { - std::visit([&](auto& stream) { - return sync_read_parser_with_timeout(*stream, buffer, err_parser, deadline); - }, conn_iter->second); + // Preserve a useful endpoint diagnostic without buffering an unbounded error response. + // One parser increment is enough for the first 200 decoded bytes and remains cancellable. + std::string error_body(max_error_response_body_bytes, '\0'); + if (!parser->is_done()) { + error_body.resize(read_body_increment(conn_iter->second, *buffer, *parser, error_body.data(), + error_body.size(), monitor, ec)); + } else { + error_body.clear(); + } + if (error_body.empty()) { + FC_THROW("HTTP POST failed with status {}", parser->get().result_int()); } - FC_THROW("HTTP POST failed with status {}: {}", (int)status, err_parser.get().body().substr(0, 200)); + FC_THROW("HTTP POST failed with status {}: {}", parser->get().result_int(), error_body); + } + + const auto content_length = parser->content_length(); + if (content_length) { + response_body_bytes = *content_length; + } + uint64_t disk_space_write_budget = 0; + if (content_length) { + FC_ASSERT(*content_length <= std::numeric_limits::max() - disk_space_concurrency_margin_bytes, + "HTTP response Content-Length {} is too large for the {}-byte disk safety margin", + *content_length, disk_space_concurrency_margin_bytes); + const auto preflight_bytes = *content_length == 0 + ? 0 + : *content_length + disk_space_concurrency_margin_bytes; + require_disk_space(final_dest, preflight_bytes); + disk_space_write_budget = std::min(*content_length, disk_space_check_interval_bytes); + } else { + disk_space_write_budget = refresh_disk_space_write_budget(final_dest, options.max_response_body_bytes); } + auto next_disk_space_check = std::chrono::steady_clock::now() + disk_space_check_interval; // Write to temp file then rename auto temp_path = final_dest; @@ -460,28 +739,59 @@ class http_client_impl { std::filesystem::remove(temp_path, rm_ec); }); - http::file_body::value_type file; - file.open(temp_path.c_str(), boost::beast::file_mode::write, ec); - FC_ASSERT(!ec, "Failed to open temp file {} for writing: {}", temp_path.string(), ec.message()); - parser.get().body() = std::move(file); + std::ofstream file(temp_path, std::ios::binary | std::ios::trunc); + FC_ASSERT(file.is_open(), "Failed to open temp file {} for writing", temp_path.string()); - if (!parser.is_done()) { - ec = std::visit([&](auto& stream) { - return sync_read_parser_with_timeout(*stream, buffer, parser, deadline); - }, conn_iter->second); + std::vector read_buffer(download_read_buffer_bytes); + transition_to(http_file_download_phase::downloading); + while (!parser->is_done()) { + const auto chunk_bytes = read_body_increment(conn_iter->second, *buffer, *parser, read_buffer.data(), + read_buffer.size(), monitor, ec); + if (ec == http::error::body_limit) { + FC_THROW("HTTP response body exceeds configured maximum of {} bytes", options.max_response_body_bytes); + } FC_ASSERT(!ec, "Failed to read response body: {}", ec.message()); + + FC_ASSERT(chunk_bytes <= options.max_response_body_bytes && + downloaded_bytes <= options.max_response_body_bytes - chunk_bytes, + "HTTP response body exceeds configured maximum of {} bytes", options.max_response_body_bytes); + if (chunk_bytes == 0) { + report_status(false); + continue; + } + + // Amortize filesystem-space queries while bounding concurrent-consumption exposure with + // a separate safety margin. Recheck after 64 MiB or five seconds, whichever comes first. + if (disk_space_write_budget < chunk_bytes || + std::chrono::steady_clock::now() >= next_disk_space_check) { + const auto response_size_limit = content_length.value_or(options.max_response_body_bytes); + const auto remaining_body_bytes = response_size_limit - downloaded_bytes; + disk_space_write_budget = refresh_disk_space_write_budget(final_dest, remaining_body_bytes); + next_disk_space_check = std::chrono::steady_clock::now() + disk_space_check_interval; + } + file.write(read_buffer.data(), static_cast(chunk_bytes)); + FC_ASSERT(file.good(), "Failed to write {} bytes to temp file {}", chunk_bytes, temp_path.string()); + downloaded_bytes += chunk_bytes; + disk_space_write_budget -= chunk_bytes; + report_status(false); } - parser.get().body().close(); + + FC_ASSERT(!is_cancelled(), "HTTP file download cancelled"); + file.close(); + FC_ASSERT(file.good(), "Failed to close temp file {} after HTTP download", temp_path.string()); std::error_code rename_ec; std::filesystem::rename(temp_path, final_dest, rename_ec); FC_ASSERT(!rename_ec, "Failed to rename downloaded file: {}", rename_ec.message()); cleanup.cancel(); + transition_to(http_file_download_phase::complete); } boost::asio::io_context _ioc; connection_map _connections; unix_url_split_map _unix_url_paths; + std::function _cancel_check; + std::function _space_available_provider; }; @@ -491,8 +801,18 @@ http_client::http_client() } -void http_client::post_to_file(const url& dest, const variant& payload, const std::filesystem::path& output, const fc::time_point& deadline) { - _my->post_to_file(dest, payload, output, deadline); +void http_client::post_to_file(const url& dest, const variant& payload, const std::filesystem::path& output, + const http_file_download_options& options) { + _my->post_to_file(dest, payload, output, options); +} + +void http_client::set_cancel_check(std::function cancel_check) { + _my->set_cancel_check(std::move(cancel_check)); +} + +void http_client::set_space_available_provider_for_testing( + std::function provider) { + _my->set_space_available_provider_for_testing(std::move(provider)); } variant http_client::post_sync(const url& dest, const variant& payload, const fc::time_point& deadline) { diff --git a/libraries/libfc/test/CMakeLists.txt b/libraries/libfc/test/CMakeLists.txt index ec1cc0e211..a0205e553b 100644 --- a/libraries/libfc/test/CMakeLists.txt +++ b/libraries/libfc/test/CMakeLists.txt @@ -18,6 +18,7 @@ add_executable( test_fc io/test_raw.cpp io/test_tracked_storage.cpp network/ethereum/test_ethereum_client_and_rlp.cpp + network/test_http_client.cpp network/test_json_rpc_client.cpp network/solana/test_solana_client.cpp network/test_message_buffer.cpp diff --git a/libraries/libfc/test/network/test_http_client.cpp b/libraries/libfc/test/network/test_http_client.cpp new file mode 100644 index 0000000000..5ab330c878 --- /dev/null +++ b/libraries/libfc/test/network/test_http_client.cpp @@ -0,0 +1,703 @@ +/** + * @file test_http_client.cpp + * @brief Regression tests for bounded streamed HTTP file downloads. + */ + +#include +#include + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace std::chrono_literals; + +namespace fc { + +/** Private free-space-query injection for HTTP download tests. */ +struct http_client_test_access { + static void set_available_disk_space(http_client& client, uint64_t available_bytes) { + client.set_space_available_provider_for_testing( + [available_bytes](const std::filesystem::path&) { return available_bytes; }); + } +}; + +} // namespace fc + +namespace { + +using tcp = boost::asio::ip::tcp; + +constexpr int64_t normal_timeout_ms = 2'000; +constexpr int64_t cancellation_delay_ms = 100; +constexpr int64_t max_test_elapsed_ms = 1'500; +constexpr size_t exact_body_bytes = 8; +constexpr size_t blocked_request_body_bytes = 16 * 1024 * 1024; +constexpr size_t disk_space_budget_bytes = 64 * 1024 * 1024; +constexpr size_t large_body_chunk_bytes = 1024 * 1024; +constexpr size_t oversized_chunk_extension_bytes = 128 * 1024; +constexpr std::string_view exact_body = "12345678"; + +/** A loopback HTTP server driven by a caller-supplied script for each accepted connection. */ +class scripted_http_server { +public: + using handler_type = std::function; + + /** Start a listener and execute @p handler for each requested connection. */ + explicit scripted_http_server(handler_type handler, bool consume_request_header = true, + size_t connections_to_accept = 1) + : _acceptor(_io, tcp::endpoint(boost::asio::ip::make_address("127.0.0.1"), 0)) + , _socket(_io) + , _port(_acceptor.local_endpoint().port()) + , _handler(std::move(handler)) + , _consume_request_header(consume_request_header) + , _connections_to_accept(connections_to_accept) + , _worker([this] { serve(); }) {} + + scripted_http_server(const scripted_http_server&) = delete; + scripted_http_server& operator=(const scripted_http_server&) = delete; + + /** Stop the response script and release its worker. */ + ~scripted_http_server() { + _stop = true; + boost::system::error_code ec; + _acceptor.close(ec); + _socket.cancel(ec); + _socket.close(ec); + unblock_accept(); + if (_worker.joinable()) { + _worker.join(); + } + } + + /** Return the ephemeral loopback port assigned to this server. */ + uint16_t port() const { return _port; } + + /** Return whether the configured server script has completed. */ + bool finished() const { return _finished.load(); } + +private: + /** Connect to the listener so a platform that does not cancel synchronous accept can exit. */ + void unblock_accept() { + boost::asio::io_context io; + tcp::socket socket(io); + boost::system::error_code ec; + socket.connect(tcp::endpoint(boost::asio::ip::make_address("127.0.0.1"), _port), ec); + } + + /** Accept configured requests, consume each header, and run the response script. */ + void serve() { + boost::system::error_code ec; + for (size_t connection_index = 0; connection_index < _connections_to_accept; ++connection_index) { + if (connection_index != 0) { + _socket = tcp::socket(_io); + } + _acceptor.accept(_socket, ec); + if (ec) { + break; + } + + if (_consume_request_header) { + boost::asio::streambuf request; + boost::asio::read_until(_socket, request, "\r\n\r\n", ec); + if (ec) { + break; + } + } else { + _socket.set_option(tcp::socket::receive_buffer_size(1'024), ec); + if (ec) { + break; + } + } + _handler(_socket, _stop); + if (connection_index + 1 < _connections_to_accept) { + _socket.close(ec); + } + } + _acceptor.close(ec); + _finished = true; + } + + boost::asio::io_context _io; + tcp::acceptor _acceptor; + tcp::socket _socket; + uint16_t _port; + handler_type _handler; + bool _consume_request_header; + size_t _connections_to_accept; + std::atomic_bool _stop{false}; + std::atomic_bool _finished{false}; + std::thread _worker; +}; + +/** Write all of @p bytes unless the peer has already disconnected. */ +bool write_bytes(tcp::socket& socket, std::string_view bytes) { + boost::system::error_code ec; + boost::asio::write(socket, boost::asio::buffer(bytes.data(), bytes.size()), ec); + return !ec; +} + +/** Return a complete fixed-length response header. */ +std::string fixed_length_header(uint64_t body_bytes, std::string_view status = "200 OK") { + std::ostringstream response; + response << "HTTP/1.1 " << status << "\r\n" + << "Content-Type: application/octet-stream\r\n" + << "Content-Length: " << body_bytes << "\r\n" + << "Connection: close\r\n\r\n"; + return response.str(); +} + +/** Return a complete 200 response header for a chunked body. */ +std::string chunked_header() { + return "HTTP/1.1 200 OK\r\n" + "Content-Type: application/octet-stream\r\n" + "Transfer-Encoding: chunked\r\n" + "Connection: close\r\n\r\n"; +} + +/** Return a JSON metadata response that invites a second request on the same connection. */ +std::string keep_alive_metadata_response() { + return "HTTP/1.1 200 OK\r\n" + "Content-Type: application/json\r\n" + "Content-Length: 2\r\n" + "Connection: keep-alive\r\n\r\n{}"; +} + +/** Write @p body_bytes bytes in bounded blocks. */ +bool write_repeated_body(tcp::socket& socket, uint64_t body_bytes) { + const std::string block(large_body_chunk_bytes, 'x'); + while (body_bytes != 0) { + const auto write_bytes_count = std::min(body_bytes, block.size()); + if (!write_bytes(socket, std::string_view(block.data(), write_bytes_count))) { + return false; + } + body_bytes -= write_bytes_count; + } + return true; +} + +/** Write a chunked body in one-MiB decoded blocks. */ +bool write_chunked_body(tcp::socket& socket, uint64_t body_bytes) { + const std::string block(large_body_chunk_bytes, 'x'); + while (body_bytes != 0) { + const auto write_bytes_count = std::min(body_bytes, block.size()); + std::ostringstream chunk_size; + chunk_size << std::hex << write_bytes_count << "\r\n"; + if (!write_bytes(socket, chunk_size.str()) || + !write_bytes(socket, std::string_view(block.data(), write_bytes_count)) || + !write_bytes(socket, "\r\n")) { + return false; + } + body_bytes -= write_bytes_count; + } + return write_bytes(socket, "0\r\n\r\n"); +} + +/** Return finite test limits with a caller-selected response-body maximum. */ +fc::http_file_download_options download_options(uint64_t max_body_bytes) { + return fc::http_file_download_options{ + .max_response_body_bytes = max_body_bytes, + .retry_failed_reused_connection = false, + }; +} + +/** Return the URL for @p server. */ +fc::url server_url(const scripted_http_server& server) { + return fc::url("http://127.0.0.1:" + std::to_string(server.port()) + "/download"); +} + +/** Invoke a bounded empty-payload download into @p output. */ +void download(const scripted_http_server& server, const std::filesystem::path& output, + const fc::http_file_download_options& options, + std::function cancel_check = {}) { + fc::http_client client; + client.set_cancel_check(std::move(cancel_check)); + client.post_to_file(server_url(server), fc::variant(fc::mutable_variant_object()), output, options); +} + +/** Invoke a download with a deterministic free-space query result. */ +void download_with_available_disk_space(const scripted_http_server& server, + const std::filesystem::path& output, + const fc::http_file_download_options& options, + uint64_t available_bytes) { + fc::http_client client; + fc::http_client_test_access::set_available_disk_space(client, available_bytes); + client.post_to_file(server_url(server), fc::variant(fc::mutable_variant_object()), output, options); +} + +/** Request cancellation after a short deterministic delay. */ +class delayed_cancellation { +public: + delayed_cancellation() + : _worker([this]() { + std::this_thread::sleep_for(std::chrono::milliseconds(cancellation_delay_ms)); + _requested = true; + }) {} + + delayed_cancellation(const delayed_cancellation&) = delete; + delayed_cancellation& operator=(const delayed_cancellation&) = delete; + + /** Join the worker that publishes the cancellation request. */ + ~delayed_cancellation() { + if (_worker.joinable()) { + _worker.join(); + } + } + + /** Return a predicate suitable for http_client::set_cancel_check(). */ + std::function check() { + return [this]() { return _requested.load(); }; + } + +private: + std::atomic_bool _requested{false}; + std::thread _worker; +}; + +/** Return the temporary sibling used while @p output is being downloaded. */ +std::filesystem::path partial_path(const std::filesystem::path& output) { + auto partial = output; + partial += ".downloading"; + return partial; +} + +/** Verify that neither the final output nor its temporary sibling exists. */ +void check_download_files_removed(const std::filesystem::path& output) { + BOOST_CHECK(!std::filesystem::exists(output)); + BOOST_CHECK(!std::filesystem::exists(partial_path(output))); +} + +/** Return the complete binary contents of @p path. */ +std::string read_file(const std::filesystem::path& path) { + std::ifstream input(path, std::ios::binary); + return std::string(std::istreambuf_iterator(input), std::istreambuf_iterator()); +} + +} // namespace + +BOOST_AUTO_TEST_SUITE(http_client_file_download_tests) + +/// An attended caller can cancel while a peer accepts but does not read the request. +BOOST_AUTO_TEST_CASE(request_write_can_be_cancelled) { + std::atomic_bool cancellation_requested{false}; + scripted_http_server server([&cancellation_requested](tcp::socket&, const std::atomic_bool& stop) { + std::this_thread::sleep_for(std::chrono::milliseconds(cancellation_delay_ms)); + cancellation_requested = true; + while (!stop.load()) { + std::this_thread::sleep_for(10ms); + } + }, false); + fc::temp_directory temp; + const auto output = temp.path() / "request-write-cancelled.bin"; + auto options = download_options(exact_body_bytes); + const fc::variant payload(std::string(blocked_request_body_bytes, 'a')); + + const auto start = std::chrono::steady_clock::now(); + fc::http_client client; + client.set_cancel_check([&cancellation_requested]() { return cancellation_requested.load(); }); + BOOST_CHECK_EXCEPTION( + client.post_to_file(server_url(server), payload, output, options), + fc::exception, + [](const fc::exception& error) { + return error.to_detail_string().find("Failed to send POST request") != std::string::npos; + }); + const auto elapsed = std::chrono::steady_clock::now() - start; + + BOOST_CHECK_LT(std::chrono::duration_cast(elapsed).count(), max_test_elapsed_ms); + check_download_files_removed(output); +} + +/// Metadata requests using the default unlimited deadline remain interruptible. +BOOST_AUTO_TEST_CASE(unbounded_post_sync_can_be_cancelled) { + scripted_http_server server([](tcp::socket&, const std::atomic_bool& stop) { + while (!stop.load()) { + std::this_thread::sleep_for(10ms); + } + }); + delayed_cancellation cancellation; + fc::http_client client; + client.set_cancel_check(cancellation.check()); + + const auto start = std::chrono::steady_clock::now(); + BOOST_CHECK_EXCEPTION( + client.post_sync(server_url(server), fc::variant(fc::mutable_variant_object())), + fc::exception, + [](const fc::exception& error) { + return error.to_detail_string().find("Failed to read response") != std::string::npos; + }); + const auto elapsed = std::chrono::steady_clock::now() - start; + + BOOST_CHECK_LT(std::chrono::duration_cast(elapsed).count(), max_test_elapsed_ms); +} + +/// Metadata and download requests should reuse one healthy keep-alive connection. +BOOST_AUTO_TEST_CASE(healthy_metadata_connection_is_reused_for_download) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + if (!write_bytes(socket, keep_alive_metadata_response())) { + return; + } + boost::asio::streambuf request; + boost::system::error_code ec; + boost::asio::read_until(socket, request, "\r\n\r\n", ec); + if (!ec) { + write_bytes(socket, fixed_length_header(exact_body_bytes) + std::string(exact_body)); + } + }); + fc::temp_directory temp; + const auto output = temp.path() / "reused-connection.bin"; + fc::http_client client; + auto metadata_deadline = fc::time_point::now(); + metadata_deadline.safe_add(fc::milliseconds(normal_timeout_ms)); + + BOOST_REQUIRE_NO_THROW( + client.post_sync(server_url(server), fc::variant(fc::mutable_variant_object()), metadata_deadline)); + auto options = download_options(exact_body_bytes); + options.retry_failed_reused_connection = true; + BOOST_REQUIRE_NO_THROW( + client.post_to_file(server_url(server), fc::variant(fc::mutable_variant_object()), output, options)); + BOOST_CHECK_EQUAL(read_file(output), exact_body); +} + +/// A cached connection closed after metadata should retry the idempotent download once. +BOOST_AUTO_TEST_CASE(stale_metadata_connection_retries_download_on_fresh_connection) { + std::atomic_size_t connection_index{0}; + scripted_http_server server([&](tcp::socket& socket, const std::atomic_bool&) { + if (connection_index.fetch_add(1) == 0) { + if (write_bytes(socket, keep_alive_metadata_response())) { + boost::system::error_code ec; + socket.shutdown(tcp::socket::shutdown_both, ec); + } + return; + } + write_bytes(socket, fixed_length_header(exact_body_bytes) + std::string(exact_body)); + }, true, 2); + fc::temp_directory temp; + const auto output = temp.path() / "retried-connection.bin"; + fc::http_client client; + auto metadata_deadline = fc::time_point::now(); + metadata_deadline.safe_add(fc::milliseconds(normal_timeout_ms)); + + BOOST_REQUIRE_NO_THROW( + client.post_sync(server_url(server), fc::variant(fc::mutable_variant_object()), metadata_deadline)); + auto options = download_options(exact_body_bytes); + options.retry_failed_reused_connection = true; + BOOST_REQUIRE_NO_THROW( + client.post_to_file(server_url(server), fc::variant(fc::mutable_variant_object()), output, options)); + BOOST_CHECK_EQUAL(connection_index.load(), 2U); + BOOST_CHECK_EQUAL(read_file(output), exact_body); +} + +/// A failed reconnect after stale reuse must unwind without touching an invalid connection iterator. +BOOST_AUTO_TEST_CASE(stale_metadata_reconnect_failure_cleans_up_safely) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + if (write_bytes(socket, keep_alive_metadata_response())) { + boost::system::error_code ec; + socket.shutdown(tcp::socket::shutdown_both, ec); + } + }); + fc::temp_directory temp; + const auto output = temp.path() / "failed-reconnect.bin"; + fc::http_client client; + auto metadata_deadline = fc::time_point::now(); + metadata_deadline.safe_add(fc::milliseconds(normal_timeout_ms)); + + BOOST_REQUIRE_NO_THROW( + client.post_sync(server_url(server), fc::variant(fc::mutable_variant_object()), metadata_deadline)); + for (size_t wait_count = 0; wait_count < 1'000 && !server.finished(); ++wait_count) { + std::this_thread::sleep_for(1ms); + } + BOOST_REQUIRE(server.finished()); + auto options = download_options(exact_body_bytes); + options.retry_failed_reused_connection = true; + BOOST_CHECK_EXCEPTION( + client.post_to_file(server_url(server), fc::variant(fc::mutable_variant_object()), output, options), + fc::exception, + [](const fc::exception& error) { + return error.to_detail_string().find("Failed to connect") != std::string::npos; + }); + check_download_files_removed(output); +} + +/// An attended caller can cancel while a peer never sends a response header. +BOOST_AUTO_TEST_CASE(response_header_wait_can_be_cancelled) { + scripted_http_server server([](tcp::socket&, const std::atomic_bool& stop) { + while (!stop.load()) { + std::this_thread::sleep_for(10ms); + } + }); + fc::temp_directory temp; + const auto output = temp.path() / "header-cancelled.bin"; + auto options = download_options(exact_body_bytes); + delayed_cancellation cancellation; + + const auto start = std::chrono::steady_clock::now(); + BOOST_CHECK_THROW(download(server, output, options, cancellation.check()), fc::exception); + const auto elapsed = std::chrono::steady_clock::now() - start; + + BOOST_CHECK_LT(std::chrono::duration_cast(elapsed).count(), max_test_elapsed_ms); + check_download_files_removed(output); +} + +/// An attended caller can cancel a body that stops after partial progress. +BOOST_AUTO_TEST_CASE(stalled_body_can_be_cancelled_and_removes_partial_file) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool& stop) { + if (!write_bytes(socket, fixed_length_header(2)) || !write_bytes(socket, "1")) { + return; + } + while (!stop.load()) { + std::this_thread::sleep_for(10ms); + } + }); + fc::temp_directory temp; + const auto output = temp.path() / "body-cancelled.bin"; + auto options = download_options(exact_body_bytes); + delayed_cancellation cancellation; + + BOOST_CHECK_THROW(download(server, output, options, cancellation.check()), fc::exception); + check_download_files_removed(output); +} + +/// A continuously progressing transfer has no total deadline but remains interruptible. +BOOST_AUTO_TEST_CASE(progressing_body_has_no_total_deadline_and_can_be_cancelled) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool& stop) { + if (!write_bytes(socket, fixed_length_header(1'000))) { + return; + } + while (!stop.load()) { + if (!write_bytes(socket, "1")) { + return; + } + std::this_thread::sleep_for(40ms); + } + }); + fc::temp_directory temp; + const auto output = temp.path() / "progressing-body-cancelled.bin"; + auto options = download_options(1'000); + delayed_cancellation cancellation; + + BOOST_CHECK_THROW(download(server, output, options, cancellation.check()), fc::exception); + check_download_files_removed(output); +} + +/// Oversized Content-Length is rejected while parsing the header, before a temp file is opened. +BOOST_AUTO_TEST_CASE(oversized_fixed_length_response_is_rejected_before_write) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + write_bytes(socket, fixed_length_header(exact_body_bytes + 1)); + }); + fc::temp_directory temp; + const auto output = temp.path() / "oversized-fixed.bin"; + + BOOST_CHECK_THROW(download(server, output, download_options(exact_body_bytes)), fc::exception); + check_download_files_removed(output); +} + +/// A non-success response includes only a small bounded prefix of its diagnostic body. +BOOST_AUTO_TEST_CASE(error_response_includes_bounded_body_diagnostic) { + const std::string error_prefix = "snapshot not ready: "; + const std::string omitted_suffix = "omitted-tail"; + const std::string error_body = error_prefix + std::string(256, 'x') + omitted_suffix; + scripted_http_server server([&](tcp::socket& socket, const std::atomic_bool&) { + write_bytes(socket, fixed_length_header(error_body.size(), "409 Conflict") + error_body); + }); + fc::temp_directory temp; + const auto output = temp.path() / "error-response.bin"; + + BOOST_CHECK_EXCEPTION( + download(server, output, download_options(error_body.size())), + fc::exception, + [&](const fc::exception& error) { + const auto detail = error.to_detail_string(); + return detail.find("HTTP POST failed with status 409: " + error_prefix) != std::string::npos && + detail.find(omitted_suffix) == std::string::npos; + }); + check_download_files_removed(output); +} + +/// A fixed-length response that ends early must fail without retaining either output file. +BOOST_AUTO_TEST_CASE(truncated_fixed_length_response_is_rejected_and_removed) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + write_bytes(socket, fixed_length_header(exact_body_bytes) + "short"); + boost::system::error_code ec; + socket.shutdown(tcp::socket::shutdown_send, ec); + socket.close(ec); + }); + fc::temp_directory temp; + const auto output = temp.path() / "truncated-fixed.bin"; + + BOOST_CHECK_THROW(download(server, output, download_options(exact_body_bytes)), fc::exception); + check_download_files_removed(output); +} + +/// A chunked response that exceeds the configured maximum is aborted and its temp file is removed. +BOOST_AUTO_TEST_CASE(oversized_chunked_response_is_rejected_and_removed) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + write_bytes(socket, chunked_header() + "9\r\n123456789\r\n0\r\n\r\n"); + }); + fc::temp_directory temp; + const auto output = temp.path() / "oversized-chunked.bin"; + + BOOST_CHECK_THROW(download(server, output, download_options(exact_body_bytes)), fc::exception); + check_download_files_removed(output); +} + +/// An unterminated oversized chunk extension must fail within bounded parser memory and time. +BOOST_AUTO_TEST_CASE(oversized_chunk_extension_is_bounded_and_removed) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + if (!write_bytes(socket, chunked_header())) { + return; + } + write_bytes(socket, "1;" + std::string(oversized_chunk_extension_bytes, 'a')); + }); + fc::temp_directory temp; + const auto output = temp.path() / "oversized-chunk-extension.bin"; + auto options = download_options(exact_body_bytes); + + const auto start = std::chrono::steady_clock::now(); + BOOST_CHECK_EXCEPTION(download(server, output, options), fc::exception, + [](const fc::exception& error) { + return error.to_detail_string().find("buffer overflow") != std::string::npos; + }); + const auto elapsed = std::chrono::steady_clock::now() - start; + + BOOST_CHECK_LT(std::chrono::duration_cast(elapsed).count(), max_test_elapsed_ms); + check_download_files_removed(output); +} + +/// A response exactly equal to the configured byte maximum must complete and be atomically renamed. +BOOST_AUTO_TEST_CASE(exact_maximum_response_succeeds) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + write_bytes(socket, fixed_length_header(exact_body_bytes) + std::string(exact_body)); + }); + fc::temp_directory temp; + const auto output = temp.path() / "exact-maximum.bin"; + std::vector statuses; + auto options = download_options(exact_body_bytes); + options.status_callback = [&](const fc::http_file_download_status& status) { statuses.push_back(status); }; + + BOOST_CHECK_NO_THROW(download(server, output, options)); + BOOST_CHECK(std::filesystem::exists(output)); + BOOST_CHECK(!std::filesystem::exists(partial_path(output))); + BOOST_CHECK_EQUAL(read_file(output), exact_body); + BOOST_REQUIRE(!statuses.empty()); + BOOST_CHECK(std::any_of(statuses.begin(), statuses.end(), [](const auto& status) { + return status.phase == fc::http_file_download_phase::downloading; + })); + const auto& final_status = statuses.back(); + BOOST_CHECK(final_status.phase == fc::http_file_download_phase::complete); + BOOST_CHECK_EQUAL(final_status.downloaded_bytes, exact_body_bytes); + BOOST_REQUIRE(final_status.total_bytes); + BOOST_CHECK_EQUAL(*final_status.total_bytes, exact_body_bytes); +} + +/// Chunked downloads report progress without inventing a total byte count or ETA basis. +BOOST_AUTO_TEST_CASE(chunked_response_reports_unknown_total) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + write_bytes(socket, chunked_header() + "8\r\n" + std::string(exact_body) + "\r\n0\r\n\r\n"); + }); + fc::temp_directory temp; + const auto output = temp.path() / "chunked-progress.bin"; + std::vector statuses; + auto options = download_options(exact_body_bytes); + options.status_callback = [&](const fc::http_file_download_status& status) { statuses.push_back(status); }; + + BOOST_REQUIRE_NO_THROW(download(server, output, options)); + BOOST_REQUIRE(!statuses.empty()); + const auto& final_status = statuses.back(); + BOOST_CHECK(final_status.phase == fc::http_file_download_phase::complete); + BOOST_CHECK_EQUAL(final_status.downloaded_bytes, exact_body_bytes); + BOOST_CHECK(!final_status.total_bytes); +} + +/// A fixed-length response exactly one disk-check budget wide should not require a refill. +BOOST_AUTO_TEST_CASE(exact_disk_space_budget_response_succeeds) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + if (write_bytes(socket, fixed_length_header(disk_space_budget_bytes))) { + write_repeated_body(socket, disk_space_budget_bytes); + } + }); + fc::temp_directory temp; + const auto output = temp.path() / "exact-disk-budget.bin"; + auto options = download_options(disk_space_budget_bytes); + + BOOST_REQUIRE_NO_THROW(download(server, output, options)); + BOOST_CHECK_EQUAL(std::filesystem::file_size(output), disk_space_budget_bytes); +} + +/// A chunked response beyond one disk-check budget must refill and complete successfully. +BOOST_AUTO_TEST_CASE(chunked_response_refills_disk_space_budget) { + constexpr auto response_bytes = disk_space_budget_bytes + large_body_chunk_bytes; + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + if (write_bytes(socket, chunked_header())) { + write_chunked_body(socket, response_bytes); + } + }); + fc::temp_directory temp; + const auto output = temp.path() / "refilled-disk-budget.bin"; + auto options = download_options(response_bytes); + + BOOST_REQUIRE_NO_THROW(download(server, output, options)); + BOOST_CHECK_EQUAL(std::filesystem::file_size(output), response_bytes); +} + +/// A response that cannot fit on the destination filesystem must fail before writing. +BOOST_AUTO_TEST_CASE(insufficient_disk_space_is_rejected_before_write) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + write_bytes(socket, fixed_length_header(1) + "1"); + }); + fc::temp_directory temp; + const auto output = temp.path() / "disk-headroom.bin"; + auto options = download_options(exact_body_bytes); + + BOOST_CHECK_THROW(download_with_available_disk_space(server, output, options, 0), fc::exception); + check_download_files_removed(output); +} + +/// A write that only barely fits must still leave room for the concurrent-consumer margin. +BOOST_AUTO_TEST_CASE(disk_space_budget_requires_concurrency_margin) { + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + write_bytes(socket, fixed_length_header(exact_body_bytes) + std::string(exact_body)); + }); + fc::temp_directory temp; + const auto output = temp.path() / "disk-concurrency-margin.bin"; + auto options = download_options(exact_body_bytes); + + BOOST_CHECK_THROW( + download_with_available_disk_space(server, output, options, exact_body_bytes), fc::exception); + check_download_files_removed(output); +} + +/// Fixed-length preflight must include the margin before opening a temporary file. +BOOST_AUTO_TEST_CASE(fixed_length_preflight_includes_concurrency_margin) { + constexpr auto response_bytes = disk_space_budget_bytes + 1; + constexpr auto headroom_without_full_margin = disk_space_budget_bytes / 2; + scripted_http_server server([](tcp::socket& socket, const std::atomic_bool&) { + write_bytes(socket, fixed_length_header(response_bytes)); + }); + fc::temp_directory temp; + const auto output = temp.path() / "fixed-preflight-margin.bin"; + auto options = download_options(response_bytes); + + BOOST_CHECK_EXCEPTION( + download_with_available_disk_space( + server, output, options, response_bytes + headroom_without_full_margin), + fc::exception, + [](const fc::exception& error) { + return error.to_detail_string().find("Insufficient disk space") != std::string::npos; + }); + check_download_files_removed(output); +} + +BOOST_AUTO_TEST_SUITE_END() diff --git a/plugins/chain_plugin/src/chain_plugin.cpp b/plugins/chain_plugin/src/chain_plugin.cpp index 080b2cfcbb..98b1dcd94e 100644 --- a/plugins/chain_plugin/src/chain_plugin.cpp +++ b/plugins/chain_plugin/src/chain_plugin.cpp @@ -37,7 +37,13 @@ #include #include #include +#include + +#include #include +#include +#include +#include const std::string deep_mind_logger_name("dmlog"); @@ -63,6 +69,97 @@ namespace snapshot_attest { /// provider snapshot interval (_snapshot_provider_block_spacing). constexpr uint32_t snapshot_attestation_grace_blocks = 12500; +constexpr uint64_t bytes_per_mebibyte = 1024 * 1024; + +/// Convert a positive MiB option value to bytes without overflowing uint64_t. +uint64_t checked_mebibytes(uint64_t value, std::string_view option_name) { + constexpr auto max_mebibytes = std::numeric_limits::max() / bytes_per_mebibyte; + SYS_ASSERT(value > 0 && value <= max_mebibytes, sysio::chain::plugin_config_exception, + "{} must be between 1 and {} MiB", option_name, max_mebibytes); + return value * bytes_per_mebibyte; +} + +/** Log attended snapshot-bootstrap phase changes, transfer rate, percentage, and ETA. */ +class snapshot_download_progress_logger { +public: + /** Consume one transport status event. */ + void operator()(const fc::http_file_download_status& status) { + const auto elapsed_seconds = static_cast(status.elapsed.count()) / microseconds_per_second; + if (status.phase != fc::http_file_download_phase::downloading && + status.phase != fc::http_file_download_phase::complete) { + ilog("Snapshot download phase: {} (elapsed {:.1f}s; press Ctrl+C to cancel)", + magic_enum::enum_name(status.phase), elapsed_seconds); + return; + } + + if (!_body_started_at) { + _body_started_at = status.elapsed; + _previous_status_at = status.elapsed; + _previous_downloaded_bytes = status.downloaded_bytes; + } + + const auto body_elapsed = status.elapsed - *_body_started_at; + if (status.phase == fc::http_file_download_phase::complete) { + const auto average_rate = rate_bytes_per_second(status.downloaded_bytes, body_elapsed); + ilog("Snapshot download complete: {} bytes ({:.2f} MiB) in {:.1f}s, average {:.2f} MiB/s", + status.downloaded_bytes, to_mebibytes(status.downloaded_bytes), + static_cast(body_elapsed.count()) / microseconds_per_second, + to_mebibytes(average_rate)); + return; + } + + const auto interval = status.elapsed - _previous_status_at; + const auto interval_bytes = status.downloaded_bytes - _previous_downloaded_bytes; + const auto current_rate = rate_bytes_per_second(interval_bytes, interval); + _previous_status_at = status.elapsed; + _previous_downloaded_bytes = status.downloaded_bytes; + + if (!status.total_bytes) { + ilog("Downloaded {} bytes ({:.2f} MiB) at {:.2f} MiB/s; response size and ETA unavailable", + status.downloaded_bytes, to_mebibytes(status.downloaded_bytes), to_mebibytes(current_rate)); + return; + } + + const auto total_bytes = *status.total_bytes; + const auto percentage = total_bytes == 0 + ? 100.0 + : 100.0 * static_cast(status.downloaded_bytes) / + static_cast(total_bytes); + if (current_rate == 0 || status.downloaded_bytes >= total_bytes) { + ilog("Downloaded {} / {} bytes ({:.1f}%) at {:.2f} MiB/s; ETA unavailable", + status.downloaded_bytes, total_bytes, percentage, to_mebibytes(current_rate)); + return; + } + + const auto remaining_bytes = total_bytes - status.downloaded_bytes; + const auto eta_seconds = static_cast( + std::ceil(static_cast(remaining_bytes) / static_cast(current_rate))); + ilog("Downloaded {} / {} bytes ({:.1f}%) at {:.2f} MiB/s, ETA {}s", + status.downloaded_bytes, total_bytes, percentage, to_mebibytes(current_rate), eta_seconds); + } + +private: + static constexpr double microseconds_per_second = 1'000'000.0; + + /** Convert a byte count to MiB for operator-facing logs. */ + static double to_mebibytes(uint64_t bytes) { + return static_cast(bytes) / static_cast(bytes_per_mebibyte); + } + + /** Calculate whole bytes per second without overflowing intermediate integer arithmetic. */ + static uint64_t rate_bytes_per_second(uint64_t bytes, const fc::microseconds& elapsed) { + if (bytes == 0 || elapsed.count() <= 0) { + return 0; + } + return static_cast( + static_cast(bytes) * microseconds_per_second / static_cast(elapsed.count())); + } + + std::optional _body_started_at; + fc::microseconds _previous_status_at; + uint64_t _previous_downloaded_bytes = 0; +}; + } // anonymous namespace namespace std { @@ -268,9 +365,10 @@ class chain_plugin_impl { /// irreversible-block handler with the just-finalized block; retries on later finalized /// blocks until the attestation record appears or the node catches up to the chain tip. void verify_snapshot_attestation(const signed_block_ptr& lib_block); - /// Download and hash-check a snapshot from a remote provider, leaving snapshot_path + /// Download and hash-check a bounded snapshot from a remote provider, leaving snapshot_path /// pointing at the downloaded file for the normal snapshot-load path to consume. - void fetch_snapshot_from_endpoint(const std::string& endpoint_url); + void fetch_snapshot_from_endpoint(const std::string& endpoint_url, + const fc::http_file_download_options& download_options); private: static void log_guard_exception(const chain::guard_exception& e); @@ -882,7 +980,23 @@ void chain_plugin_impl::plugin_initialize(const variables_map& options) { if (options.count("snapshot-endpoint")) { SYS_ASSERT(!options.contains("snapshot"), plugin_config_exception, "--snapshot-endpoint is incompatible with --snapshot; use one or the other"); - fetch_snapshot_from_endpoint(options.at("snapshot-endpoint").as()); + + const auto max_download_bytes = checked_mebibytes( + options.at("chain-state-db-size-mb").as(), "chain-state-db-size-mb"); + const fc::http_file_download_options download_options{ + .max_response_body_bytes = max_download_bytes, + .retry_failed_reused_connection = true, + .status_callback = [logger = snapshot_download_progress_logger{}]( + const fc::http_file_download_status& status) mutable { logger(status); }, + }; + try { + fetch_snapshot_from_endpoint(options.at("snapshot-endpoint").as(), download_options); + } catch (...) { + if (app().is_quiting()) { + SYS_THROW(chain::interrupt_exception, "Snapshot endpoint bootstrap interrupted"); + } + throw; + } } std::optional chain_id; @@ -1515,7 +1629,8 @@ bool chain_plugin::accept_transactions() const { return my->accept_transactions; } -void chain_plugin_impl::fetch_snapshot_from_endpoint(const std::string& endpoint_url) { +void chain_plugin_impl::fetch_snapshot_from_endpoint( + const std::string& endpoint_url, const fc::http_file_download_options& download_options) { // Check for existing chain data auto shared_mem_path = chain_config->state_dir / "shared_memory.bin"; auto chain_head_path = chain_config->state_dir / chain_head_filename; @@ -1546,6 +1661,7 @@ void chain_plugin_impl::fetch_snapshot_from_endpoint(const std::string& endpoint ilog("Fetching snapshot metadata from endpoint: {}", endpoint_url); fc::http_client http_client; + http_client.set_cancel_check([]() { return app().is_quiting(); }); fc::variant metadata_response; if (request_block_num) { auto url = fc::url(base_url + "/v1/snapshot/by_block"); @@ -1576,7 +1692,8 @@ void chain_plugin_impl::fetch_snapshot_from_endpoint(const std::string& endpoint auto download_url = fc::url(base_url + "/v1/snapshot/download"); fc::mutable_variant_object download_payload; download_payload("block_num", snap_block_num); - http_client.post_to_file(download_url, fc::variant(download_payload), download_dest); + http_client.post_to_file( + download_url, fc::variant(download_payload), download_dest, download_options); } // Quick check: compare the root hash stored in the snapshot footer against diff --git a/plugins/chain_plugin/test/CMakeLists.txt b/plugins/chain_plugin/test/CMakeLists.txt index 9e30f3f891..c77bea0764 100644 --- a/plugins/chain_plugin/test/CMakeLists.txt +++ b/plugins/chain_plugin/test/CMakeLists.txt @@ -8,6 +8,8 @@ add_executable( test_chain_plugin target_link_libraries( test_chain_plugin chain_plugin sysio_testing sysio_chain_wrap version ) set(CHAIN_PLUGIN_TEST_CASES chain_plugin_default_tests + chain_plugin_snapshot_endpoint_option_registration + chain_plugin_snapshot_endpoint_option_validation account_query_db_tests trx_finality_status_processing_test trx_retry_db_test diff --git a/plugins/chain_plugin/test/plugin_config_test.cpp b/plugins/chain_plugin/test/plugin_config_test.cpp index ab05aa42ec..09aacf3ca4 100644 --- a/plugins/chain_plugin/test/plugin_config_test.cpp +++ b/plugins/chain_plugin/test/plugin_config_test.cpp @@ -1,8 +1,44 @@ #include +#include #include #include #include #include +#include +#include +#include +#include + +namespace { + +/** Initialize chain_plugin with one snapshot response-size limit override. */ +sysio::chain::exit_code::exit_code initialize_with_snapshot_size_limit(std::string_view option_name, + std::string_view option_value) { + fc::temp_directory tmp; + sysio::chain::application exe({.enable_resource_monitor = false}); + + const auto tmp_path = tmp.path().string(); + std::vector arguments{ + "test_chain_plugin", + "--snapshot-endpoint", + "http://127.0.0.1:1", + "--config-dir", + tmp_path, + "--data-dir", + tmp_path, + "--" + std::string(option_name), + std::string(option_value), + }; + std::vector argv; + argv.reserve(arguments.size()); + for (auto& argument : arguments) { + argv.push_back(argument.data()); + } + + return exe.init(static_cast(argv.size()), argv.data()); +} + +} // anonymous namespace BOOST_AUTO_TEST_CASE(chain_plugin_default_tests) { fc::temp_directory tmp; @@ -25,6 +61,55 @@ BOOST_AUTO_TEST_CASE(chain_plugin_default_tests) { } +/** Verify snapshot endpoint registration and removal of endpoint-specific resource knobs. */ +BOOST_AUTO_TEST_CASE(chain_plugin_snapshot_endpoint_option_registration) { + sysio::chain::application exe({.enable_resource_monitor = false}); + sysio::chain_plugin plugin; + boost::program_options::options_description cli; + boost::program_options::options_description cfg; + boost::program_options::options_description options; + plugin.set_program_options(cli, cfg); + options.add(cli).add(cfg); + + std::array arguments{ + "test_chain_plugin", + "--snapshot-endpoint", "http://127.0.0.1:1", + }; + boost::program_options::variables_map variables; + boost::program_options::store( + boost::program_options::parse_command_line(arguments.size(), const_cast(arguments.data()), options), + variables); + boost::program_options::notify(variables); + + BOOST_CHECK_EQUAL(variables.at("snapshot-endpoint").as(), "http://127.0.0.1:1"); + + constexpr std::array removed_options{ + "snapshot-endpoint-connect-timeout-ms", + "snapshot-endpoint-header-timeout-ms", + "snapshot-endpoint-idle-timeout-ms", + "snapshot-endpoint-total-timeout-ms", + "snapshot-endpoint-max-download-size-mb", + "snapshot-endpoint-min-disk-free-mb", + }; + for (const auto* option_name : removed_options) { + BOOST_CHECK(options.find_nothrow(option_name, false) == nullptr); + } +} + +/** Verify that the snapshot response-size limit rejects zero and overflow before connecting. */ +BOOST_AUTO_TEST_CASE(chain_plugin_snapshot_endpoint_option_validation) { + constexpr std::array invalid_options{ + std::pair{"chain-state-db-size-mb", "0"}, + std::pair{"chain-state-db-size-mb", "17592186044416"}, + }; + + for (const auto& [option_name, option_value] : invalid_options) { + BOOST_TEST_CONTEXT(option_name << '=' << option_value) { + BOOST_CHECK(initialize_with_snapshot_size_limit(option_name, option_value) != sysio::chain::exit_code::SUCCESS); + } + } +} + #ifdef SYSIO_SYS_VM_OC_RUNTIME_ENABLED /** Verify the default SYS VM OC whitelist when the OC runtime is compiled into this build. */ BOOST_AUTO_TEST_CASE(chain_plugin_default_sys_vm_oc_whitelist) { diff --git a/plugins/snapshot_api_plugin/README.md b/plugins/snapshot_api_plugin/README.md index 9a8aa4fc31..e8ea02d846 100644 --- a/plugins/snapshot_api_plugin/README.md +++ b/plugins/snapshot_api_plugin/README.md @@ -247,6 +247,26 @@ The bootstrap process: `--delete-all-blocks` is required when existing chain data is present. The `--snapshot-endpoint` option is incompatible with `--snapshot` (local file). +### Bootstrap download status and limits + +Snapshot bootstrap is attended and has no automatic network deadlines. While metadata is fetched, the node identifies +the active operation; the file transfer then reports phase changes and, every five seconds, downloaded bytes, +percentage, transfer rate, and ETA when the response supplies `Content-Length`. Pressing Ctrl+C cancels pending +resolver or socket work and removes the partial file. + +The existing chain database size setting supplies the download ceiling: + +| Option | Default | Purpose | +|---|---:|---| +| `--chain-state-db-size-mb` | `1024` | Maximum accepted snapshot response size and chain-state database size | + +The response-size limit must be positive. A fixed-length response that exceeds the maximum is rejected from its +`Content-Length` before a temporary file is opened. Chunked or lengthless responses are stopped at the same byte +ceiling. Available disk space is checked before and throughout the transfer, and a failed transfer removes its +`.downloading` file. +Long transfers recheck after at most 64 MiB or five seconds of progress and retain an additional 64 MiB safety margin +to bound interference from concurrent disk consumers between probes. + ## Reverse Proxy Considerations For production deployments, consider placing a reverse proxy (nginx, caddy) in front of the snapshot endpoint: @@ -256,4 +276,8 @@ For production deployments, consider placing a reverse proxy (nginx, caddy) in f - **Caching / CDN** — offload download bandwidth from the node - **Access control** — restrict by IP or authentication +Configure the proxy with response-size, idle, and total-transfer limits that are no weaker than the origin node's +bootstrap limits. Proxy authentication and TLS reduce interception and unauthorized access risk, but they do not +replace the origin-side resource bounds. + The snapshot files are served uncompressed via zero-copy `sendfile()`. If bandwidth is a concern, configure compression at the proxy layer (e.g., `gzip_static` or `zstd_static` in nginx with pre-compressed files).