Configure Logstash to forward logs to a syslog server

Overview

This guide explains how to configure Logstash to collect logs from system components and forward them to a remote syslog server. Centralizing logs this way enables remote monitoring and analysis.

Logstash acts as a log processor that ingests data from multiple sources, transforms it, and outputs it to destinations such as syslog servers. This configuration:

• Collects logs from Kafka topics
• Processes them through the Logstash pipeline
• Forwards the processed logs to a remote syslog server over UDP

Before you begin

Before you start, make sure:

• You have write access to /etc/logstash/conf.d/.
• Your remote syslog server is configured to receive UDP traffic on port 514.
• Network connectivity to the syslog server is confirmed.

Configure the syslog output plugin

Create a configuration file that defines the input and output for the syslog pipeline.

File: /etc/logstash/conf.d/rsyslog-persister.conf

Add the following configuration and replace the following placeholders values.

Placeholder	Description
<CLUSTER_VIP>	The cluster virtual IP address of your CubeCOS cluster
<SYSLOG_SERVER_IP>	The IP address of your syslog server
<SYSLOG_PORT>	The port your syslog server listens on (typically 514)
<udp|tcp>	The transport protocol used

/etc/logstash/conf.d/rsyslog-persister.conf

input {
  kafka {
    bootstrap_servers => "<CLUSTER_VIP>:9095"
    topics => "logs"
    group_id => "rsyslog_persister"
    client_id => "rsyslog_persister_cpu014"
    consumer_threads => "2"
    codec => json
  }
}

output {
  syslog {
    host => "<SYSLOG_SERVER_IP>"
    port => <SYSLOG_PORT>
    protocol => "<udp|tcp>"
    rfc        => "rfc5424"
    facility   => "local0"
    severity   => "notice"
    procid     => "-"
    msgid      => "-"
    sourcehost => "%{[host][name]}"
    message    => "%{message}"
  }
}

Register the pipeline

Update the Logstash pipelines configuration file to include your new syslog pipeline.

Add the new pipeline to /etc/logstash/pipelines.yml:

Addition section to the /etc/logstash/pipelines.yml

- pipeline.id: syslog-persister
  path.config: "/etc/logstash/conf.d/rsyslog-persister.conf"

The complete file should include all active pipelines. For reference, a typical configuration includes:

/etc/logstash/pipelines.yml

- pipeline.id: log-transformer
  path.config: "/etc/logstash/conf.d/log-transformer.conf"
- pipeline.id: auditlog-transformer
  path.config: "/etc/logstash/conf.d/auditlog-transformer.conf"
- pipeline.id: hex-event-mapper
  path.config: "/etc/logstash/conf.d/hex-event-mapper.conf"
- pipeline.id: ops-event-mapper
  path.config: "/etc/logstash/conf.d/ops-event-mapper.conf"
- pipeline.id: telegraf-persister
  path.config: "/etc/logstash/conf.d/telegraf-persister.conf"
- pipeline.id: telegraf-hc-persister
  path.config: "/etc/logstash/conf.d/telegraf-hc-persister.conf"
- pipeline.id: telegraf-events-persister
  path.config: "/etc/logstash/conf.d/telegraf-events-persister.conf"
- pipeline.id: ceph-event-mapper
  path.config: "/etc/logstash/conf.d/ceph-event-mapper.conf"
- pipeline.id: kernel-event-mapper
  path.config: "/etc/logstash/conf.d/kernel-event-mapper.conf"
- pipeline.id: rsyslog-persister
  path.config: "/etc/logstash/conf.d/rsyslog-persister.conf"

Propogate the configuration to all control nodes

After updating the primary control node, sync the configuration files to all other control nodes:

Shell

cubectl node -r control rsync /etc/logstash/pipelines.yml
cubectl node -r control rsync /etc/logstash/conf.d/rsyslog-persister.conf

This ensures all control nodes have the same configuration and can process logs consistently.

Restart Logstash

Apply the configuration changes by restarting the Logstash service across all control nodes.

Shell

cubectl node -r control exec -p "systemctl restart logstash"

The -p flag runs the command in parallel on all nodes.

Verify the configuration

Check pipeline status

Verify that the pipeline is running successfully by examining the Logstash service logs.

Shell

grep "rsyslog-persister" /var/log/logstash/logstash-plain.log

A successful result looks like:

Shell

[2025-04-10T14:08:15,997][INFO ][logstash.agent] Pipelines running {:count=>10, :running_pipelines=>[:, :"rsyslog-persister", ...], :non_running_pipelines=>[]}

The rsyslog-persister entry in running_pipelines confirms the pipeline is active.

Next steps

After verifying the configuration, consider:

• Monitoring Logstash resource usage and pipeline performance
• Setting up alerts for pipeline failures
• Configuring log retention policies on the syslog server
• Adding additional filters or transformations to the pipeline as needed