fluent-plugin-sqs-2.1.0 breaks td-agent-2.3.6

[dwoo4dwoo]

Hello,

The latest fluent-plugin-sqs-2.1.0 breaks td-agent-2.3.5. See below. fluent-plugin-sqs-2.0.0 is fine.

I see commits by @cosmo0920 to support v0.14; however, the gemspec implies this should still work for v0.12.

David

% td-agent
2017-09-12 23:11:31 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2017-09-12 23:11:31 +0000 [info]: starting fluentd-0.12.36
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-kafka' version '0.5.5'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-mongo' version '0.8.0'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-s3' version '0.8.2'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-sqs' version '2.1.0'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-td' version '0.10.29'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2017-09-12 23:11:31 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.7.0'
2017-09-12 23:11:31 +0000 [info]: gem 'fluentd' version '0.12.36'
2017-09-12 23:11:31 +0000 [info]: adding match pattern="docker.**" type="sqs"
/opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- fluent/plugin/output (LoadError)
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-sqs-2.1.0/lib/fluent/plugin/out_sqs.rb:1:in `<top (required)>'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:172:in `block in try_load_plugin'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:170:in `each'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:170:in `try_load_plugin'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:130:in `new_impl'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/plugin.rb:59:in `new_output'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/agent.rb:131:in `add_match'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/agent.rb:64:in `block in configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/agent.rb:57:in `each'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/agent.rb:57:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/root_agent.rb:86:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/engine.rb:129:in `configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/engine.rb:103:in `run_configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:489:in `run_configure'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:160:in `block in start'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:366:in `call'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:366:in `main_process'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:339:in `block in supervise'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:338:in `fork'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:338:in `supervise'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/supervisor.rb:156:in `start'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
        from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.36/bin/fluentd:5:in `<top (required)>'
        from /opt/td-agent/embedded/bin/fluentd:23:in `load'
        from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
        from /sbin/td-agent:7:in `load'
        from /sbin/td-agent:7:in `<main>'
2017-09-12 23:11:31 +0000 [info]: process finished code=256
2017-09-12 23:11:31 +0000 [warn]: process died within 1 second. exit.

[cosmo0920]

@dwoo4dwoo Could you send a PR to fix this issue?
Or you want to help me to fix this issue?

[dwoo4dwoo]

@cosmo0920 - according to https://docs.fluentd.org/v0.14/articles/plugin-update-from-v12:

Plugins which uses v0.14 API don’t work on Fluentd v0.12.x.

So your changes to support v0.14 definitely break v0.12. There must be something wrong with the gemspec, since I can install fluent-plugin-sqs-2.1.0 with fluentd v0.12, and this combination will never work.

@ixixi - the same doc https://docs.fluentd.org/v0.14/articles/plugin-update-from-v12 recommends creating a fluentd-v0.12 branch so you're able to fix bugs and release features for v0.12. Or did you intentionally abandon v0.12?

Also, according to https://docs.fluentd.org/v0.12/articles/plugin-development#plugin-versioning-policy, the major version should have changed when moving to v0.14. It's not at all intuitive that 2.0.0 works with v0.12 whereas 2.1.0 breaks v0.12.

For now, I'm going to hardcode using fluent-plugin-sqs v2.0.0, so I get v2 of the AWS SDK.

[dwoo4dwoo]

It'd also be nice to have something akin to https://github.com/uken/fluent-plugin-elasticsearch#requirements in README.md.

[cosmo0920]

Patch is welcomed.

[dwoo4dwoo]

Meh, I didn't break it, I'm not going to fix it :-)

[cosmo0920]

But you've hit this issue and reported first.
It is enough reason to fix it, though. :-P

[dwoo4dwoo]

For now, I'm going to hardcode using fluent-plugin-sqs v2.0.0, so I get v2 of the AWS SDK.

Sigh, I'm now hitting #30, so I'm going back to v1.8.0 which I know works.

[cosmo0920]

#36 will fix this issue.
Thank you for your report.

[dwoo4dwoo]

#36 will fix this issue.
Awesome - thanks @cosmo0920 !