input-syslog and input-beats have different definitions of the host field

[sinzui]

Beat 6.3.0 redefined the host field from a string an object with many fields. Input-syslog maintains the older definition. When beats and syslog are used to collect log messages intended for the same indices, Logstash/Elasticsearch will drop messages that don't conform to the mapping.

• Version: logstash 6.4.3
• Operating System: Centos 7
• Steps to Reproduce:
  • Configure beats to send host information to logstash: processors: [add_host_metadata]
  • Configure logstash with beat and syslog input
  • Configure the messages to go to the same indice.
  • An error like this will be reported by both Logstash and Elasticsearch

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-cloud-2018.11.18", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x1de564db>], :response=>{"index"=>{"_index"=>"logstash-cloud-2018.11.18", "_type"=>"doc", "_id"=>"tzOkJ2cBdCVEkBSRXuai", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [host] tried to parse field [host] as object, but found a concrete value"}}}}

I implemented this work around as the first step in the logstash filters

    # Beats define [host] as an object, but some inputs may send it as a string.
    if [host] and ! [host][name] {
        mutate {
            rename => { "host" => "hostname" }
        }
    }

[MaartenMol]

Is there any update on this? I have the same issue, but the fix with the filter gives the exact same problem for me:

[2018-12-03T09:46:18,769][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2018.12.03", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x15bc3035>], :response=>{"index"=>{"_index"=>"logstash-2018.12.03", "_type"=>"doc", "_id"=>"ibt2c2cBO_NxfnjWGHim", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [hostname] tried to parse field [hostname] as object, but found a concrete value"}}}}

[sinzui]

@MaartenMol96 That implies you also got a host.name field. Maybe this works for your case

Beats define [host] as an object, but some inputs may send it as a string.

if [host] and "input.syslog" in [tags] {
    mutate {
        rename => { "host" => "hostname" }
    }
}