Receivers

To handle back-pressure, many connection-oriented receivers are blocking. If processing queue is full, it stops handling new event, so the sender can detect it and react appropriately. That is not always a desired effect, so all receivers can take a bean called blocking that disable that feature, as some senders don’t handle back-pressure very well.

loghub.receivers.Beats

It's used to receive events from Elastic's Beats.

Attributes

• clientInactivityTimeout
• maxPayloadSize
• withSSL
• SSLClientAuthentication, can take the values REQUIRED, WANTED, NOTNEEDED, NONE.
• SSLKeyAlias

loghub.receivers.Http

Attributes

• port
• host
• listenBacklog
• poller, Netty's poller. Can take the value NIO, EPOLL, OIO, KPOLL or EPOLL.
• workerThreads
• jaasName
• decoder, default decoder, when no content type is given.
• useJwt
• user
• password
• withSSL
• SSLClientAuthentication, can take the values REQUIRED, WANTED, NOTNEEDED, NONE.
• SSLKeyAlias

loghub.receivers.Jfr

Java Flight Recording is a framework that allows JVM profiling. Using JMC it’s possible to collect events to a jfr file. It’s also possible to connect to a remote JVM. This receiver can handle both case, given a JFR file or a JMX URL.

When doing dynamic collect, the configuration of collected events is done in three optional steps, resolved in the given order of priority.

1. The attribute jfrConfiguration refers to a predefined configuration, that can be found in $JAVA_HOME/lib/jfr, possible values are default and profile.
2. The attribute jfrConfigurationFile refers to the path of jfc files, using the same format that the predefined configuration.
3. The attribute jfrSettings allows to provide a map of settings, using the standard format <event-name> + "#" + <setting-name> and the value provided as a string.

If the distant JVM is restarted, the receiver will detect it and restart the JMX connection.

Attributes

• jfrFile: The path of a JFR file, as generated by JMC.
• jmxUrl: The URL of a JMX connection for a live stream of events.
• jfrConfiguration: The name of a default configuration.
• jfrConfigurationFile: The path (or URI) of an XML jfc file.
• jfrSettings: A Map of individual settings.
• flushInterval: The expected interval in ms betwen expected JFR flush, used to detect remote JVM restart.
• durationUnit: How to handle period received in an event. It’s converted as if it was done the loghub.processors.DurationConvert processor. The default is to keep the duration as a java object.

loghub.receivers.Journald

Journal is an improved logging system used by systemd. It can handle specific log events, syslog events and process kernel messages.

They can be sent to a remote server using systemd-journal-upload.service. LogHub can be configured to receive such events with the following setup:

/***********************
 * The journald source *
 ***********************/
input {
    // HTTP
    loghub.receivers.Journald {
        port: 1522,
    },
} | $prejournald
pipeline[prejournald] {
    $hostresolver |
    [#source] = "journald"
} | $main

In this case, journald should be configured to intercept all syslog and kernel messages. Default journald configuration is to store it in memory, so it's discarded at reboot. But as systemd-journal-upload keep a cursor, it's interesting to keep a small buffer on disk, to keep more information. The configuration of journald is done in /etc/systemd/journald.conf.

[Journal]
Storage=persistent
SystemMaxUse=16M
RuntimeMaxUse=16M

The directory /var/log/journal must have been created, with root:systemd-journal owner and permissions 2755.

The configuration of systemd-journal-upload is done in unit configuration file, in /etc/systemd/system/systemd-journal-upload.service.d/loghub.conf:

[Service]
ExecStart=/usr/lib/systemd/systemd-journal-upload --save-state -u http://loghub:1522 -m --follow --save-state=/var/lib/systemd/journal-upload/state.loghub
Restart=always
RestartSec=5s
StartLimitBurst=100
StartLimitInterval=500s

When installing systemd-journal-upload, the cursor must be reset, to avoid mixing of events and possible dual dispatch. The following scripts does that:

#!/usr/bin/bash
# push the loghub cursor to the current point, we don't want to roleback in time
STATE_FILE="/var/lib/systemd/journal-upload/state.loghub"
# eval is to get ride of "" around the cursor
eval "CURSOR=$(journalctl -o json -n1 | jq .__CURSOR)"
mkdir -p $(dirname "$STATE_FILE")
printf "LAST_CURSOR=$CURSOR\n" > "$STATE_FILE"
chown systemd-journal-upload:systemd-journal-upload $(dirname "$STATE_FILE") "$STATE_FILE"

With a full configuration, rsyslog configuration is optional. It should not listen on the standard /dev/log socket or kernel logging. It's only keep for compatibility and safeguard of logs in /var/log/messages and old logs settings.

It's configuration should now be:

#### MODULES ####
$ModLoad imuxsock.so
$SystemLogSocketName /run/systemd/journal/syslog
 
#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$SystemLogRateLimitInterval 10
$SystemLogRateLimitBurst 2000
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
kern.warn                                             /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none              /var/log/messages
# The authpriv file has restricted access.
authpriv.*                                            /var/log/secure
# Log all the mail messages in one place.
mail.*                                                -/var/log/maillog
# Log cron stuff
cron.*                                                /var/log/cron
# Everybody gets emergency messages
kern.emerg,daemon.emerg                               :omusrmsg:*

Attributes

• port
• host
• listenBacklog
• maxWaitingEvents how many events can be published in a single post from journald-upload. If that limit is reached, old events will start to be dropped.
• poller, Netty's poller. Can take the value NIO, EPOLL, OIO.
• useJwt
• user
• password
• jaasName, incompatible with user and password, needs that the property jaasConfig provides the path to a jaasConfig file.
• withSSL
• SSLClientAuthentication, can take the values REQUIRED, WANTED, NOTNEEDED, NONE
• SSLKeyAlias

loghub.receivers.Kafka

An Apache Kafka consumer that extracts events from a Kafka cluster and inject it for processing. If headers are present, they are injected in an headers attribute.

Attributes

• brokers, an array of hostname(:port) ; port is optional.
• port, the default port to connect, default to 9092.
• topic, the topic to subscribe to.
• group, the group that subscriber is part of, default to loghub.
• keyDeserializer, a class that implements Interface Deserializer<T> ; default to ByteArrayDeserializer

loghub.receivers.SnmpTrap

The MIB modules path is defined in the global property mibdirs, that provides a path to directories that contains modules files.

Attributes

• port.
• listen.
• protocol.
• rcvBuf.

loghub.receivers.TcpLinesStream

Handle a continuous stream of events using a line feed as a separator from a raw socket.

Attributes

• charset
• field
• maxLength
• poller, Netty's poller. It can take the value NIO, EPOLL, OIO, KPOLL or EPOLL.
• separator, a string that can be used as a separator for events.
• transport, a Netty transport. It can take the value TCP (the default value) or UNIX_STREAM.
• withSSL
• SSLClientAuthentication, can take the values REQUIRED, WANTED, NOTNEEDED, NONE
• SSLKeyAlias

loghub.receivers.Udp

Attributes

• bufferSize
• poller, Netty's poller. It can take the value NIO, EPOLL, OIO, KPOLL or EPOLL.
• transport, a Netty’s transport. It can take the value UDP (the default value), UNIX_DGRAM or SCTP.

loghub.receivers.TimeSerie

loghub.receivers.ZMQ

If zapHandler is set to METADATA, the property zmq.keystore should store ZeroMQ Property Language files that map public key the to identity stored in [@context principal name] using the property User-Id. All other generic metadata are forward to the event meta.

Attributes

• type, a ZMQ socket type, see org.zeromq.SocketType for the supported types.
• hwm, the ZMQ High water mark.
• method, can be BIND or CONNECT.
• listen, a ZMQ URL. Will be the listening endpoint if method is BIND, and the peer connection if method is CONNECT. It can be something like tcp://0.0.0.0:0 to listen using TCP transport, on all IP, with a random port.
• serverKey, the peer public key, in the format Curve <ENCODEBASE64 key>.
• security, can be NULL, PLAIN, CURVE.
• topic, a ZMQ topic if type is PUB.
• zapHandler, can be ALLOW, DENY, STRICT, METADATA.