From 3a77ebb113bb29db1fb6a845fef8daa45b650c7d Mon Sep 17 00:00:00 2001 From: Reid Price Date: Thu, 26 Jun 2025 14:02:37 -0700 Subject: [PATCH] netflow.rb: cleanup even if there isn't a file This was a change that I've been patching locally for a couple of years. Some details are ommited, but when packets are parsed by logstash using the netflow codec, there are two issues. storage of templates will slowly leak memory The cache ttl is never checked, so it grows without bounds. There is a workaround of doing cache cleanup by specifing a cache file. netflow will stop processing packets in burst This is probably not resolved, since it existed before this change. However, the same issue of (unbounded recv-q) persist when we use a cache file. The issue with the cache file is multi-part: - data is keyed by template but doesn't get passed metadata host/port - this means that sources clobber each other's templates - this is also all for the best, or we would store 3GB of data instead of 2MB - aside: this could even be resolved by having a multi-part lookup template.cache.definitions :: {hash -> template} // 4 * 600 bytes, 2KB template.cache.keys :: {key -> hash} // 3k * 30 bytes, 10KB and then we could even have the different flow exporters not collide - the cache is rewritten on every new template - these come constantly (every X minutes times 2k source) - when the 2MB cache is rewritten it takes a mutex lock - the file just flickers in and out of existence as fast as possible This grinds the entire processing to a halt. Even when changing the cache to only be rewritten if a new key is present it didn't resolve the core issues. --- lib/logstash/codecs/netflow.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/logstash/codecs/netflow.rb b/lib/logstash/codecs/netflow.rb index 3d86201..e02e307 100644 --- a/lib/logstash/codecs/netflow.rb +++ b/lib/logstash/codecs/netflow.rb @@ -647,7 +647,7 @@ def do_load # @see `TemplateRegistry#persist` # @api private def do_persist - return if file_path.nil? + (do_cleanup!; return) if file_path.nil? logger.debug? and logger.debug('Writing templates to template cache', :file_path => file_path)