# Configure Logstash to forward logs to a syslog server

> Set up Logstash to collect and forward system logs to a remote syslog server using UDP transport.

## Overview {#overview}

This guide explains how to configure Logstash to collect logs from system components and forward them to a remote syslog server. Centralizing logs this way enables remote monitoring and analysis.

Logstash acts as a log processor that ingests data from multiple sources, transforms it, and outputs it to destinations such as syslog servers. This configuration:

- Collects logs from Kafka topics
- Processes them through the Logstash pipeline
- Forwards the processed logs to a remote syslog server over UDP

## Before you begin {#before-you-begin}

Before you start, make sure:

- You have write access to `/etc/logstash/conf.d/`.
- Your remote syslog server is configured to receive UDP traffic on port 514.
- Network connectivity to the syslog server is confirmed.

## Configure the syslog output plugin {#configure-the-syslog-output-plugin}

Create a configuration file that defines the input and output for the syslog pipeline.

**File:** `/etc/logstash/conf.d/rsyslog-persister.conf`

Add the following configuration and replace the following placeholders values.

| Placeholder          | Description                                            |
| -------------------- | ------------------------------------------------------ |
| `<CLUSTER_VIP>`      | The cluster virtual IP address of your CubeCOS cluster |
| `<SYSLOG_SERVER_IP>` | The IP address of your syslog server                   |
| `<SYSLOG_PORT>`      | The port your syslog server listens on (typically 514) |
| `<udp\|tcp>`         | The transport protocol used                            |

```bash title="/etc/logstash/conf.d/rsyslog-persister.conf"
input {
  kafka {
    # highlight-start
    bootstrap_servers => "<CLUSTER_VIP>:9095"
    # highlight-end
    topics => "logs"
    group_id => "rsyslog_persister"
    client_id => "rsyslog_persister_cpu014"
    consumer_threads => "2"
    codec => json
  }
}

output {
  syslog {
    # highlight-start
    host => "<SYSLOG_SERVER_IP>"
    port => <SYSLOG_PORT>
    protocol => "<udp|tcp>"
    # highlight-end
  }
}
```

## Register the pipeline {#register-the-pipeline}

Update the Logstash pipelines configuration file to include your new syslog pipeline.

Add the new pipeline to `/etc/logstash/pipelines.yml`:

```bash title="Addition section to the /etc/logstash/pipelines.yml"
- pipeline.id: sample-syslog-persister
  path.config: "/etc/logstash/conf.d/rsyslog-persister.conf"
```

The complete file should include all active pipelines. For reference, a typical configuration includes:

```bash title="/etc/logstash/pipelines.yml"
- pipeline.id: log-transformer
  path.config: "/etc/logstash/conf.d/log-transformer.conf"
- pipeline.id: auditlog-transformer
  path.config: "/etc/logstash/conf.d/auditlog-transformer.conf"
- pipeline.id: hex-event-mapper
  path.config: "/etc/logstash/conf.d/hex-event-mapper.conf"
- pipeline.id: ops-event-mapper
  path.config: "/etc/logstash/conf.d/ops-event-mapper.conf"
- pipeline.id: telegraf-persister
  path.config: "/etc/logstash/conf.d/telegraf-persister.conf"
- pipeline.id: telegraf-hc-persister
  path.config: "/etc/logstash/conf.d/telegraf-hc-persister.conf"
- pipeline.id: telegraf-events-persister
  path.config: "/etc/logstash/conf.d/telegraf-events-persister.conf"
- pipeline.id: ceph-event-mapper
  path.config: "/etc/logstash/conf.d/ceph-event-mapper.conf"
- pipeline.id: kernel-event-mapper
  path.config: "/etc/logstash/conf.d/kernel-event-mapper.conf"
  # highlight-start
- pipeline.id: sample-syslog-persister
  path.config: "/etc/logstash/conf.d/rsyslog-persister.conf"
  # highlight-end
```

## Propogate the configuration to all control nodes {#propogate-the-configuration-to-all-control-nodes}

After updating the primary control node, sync the configuration files to all other control nodes:

```bash title="Shell"
cubectl node -r control rsync /etc/logstash/pipelines.yml
cubectl node -r control rsync /etc/logstash/conf.d/rsyslog-persister.conf
```

This ensures all control nodes have the same configuration and can process logs consistently.

## Restart Logstash {#restart-logstash}

Apply the configuration changes by restarting the Logstash service across all control nodes.

```bash title="Shell"
cubectl node -r control exec -p "systemctl restart logstash"
```

The `-p` flag runs the command in parallel on all nodes.

## Verify the configuration {#verify-the-configuration}

### Check pipeline status {#check-pipeline-status}

Verify that the pipeline is running successfully by examining the Logstash service logs.

```bash title="Shell"
grep "sample-syslog-persister" /var/log/logstash/logstash-plain.log
```

A successful result looks like:

```bash title="Shell"
[2025-04-10T14:08:15,997][INFO ][logstash.agent] Pipelines running {:count=>10, :running_pipelines=>[:, :"sample-syslog-persister", ...], :non_running_pipelines=>[]}
```

The `sample-syslog-persister` entry in `running_pipelines` confirms the pipeline is active.

## Next steps {#next-steps}

After verifying the configuration, consider:

- Monitoring Logstash resource usage and pipeline performance
- Setting up alerts for pipeline failures
- Configuring log retention policies on the syslog server
- Adding additional filters or transformations to the pipeline as needed