Skip to content

updating the logstash migration example #11201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

updating the logstash migration example #11201

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 113 additions & 7 deletions _data-prepper/migrating-from-logstash-data-prepper.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,20 +29,126 @@

## Running Data Prepper with a Logstash configuration

1. To install Data Prepper's Docker image, see Installing Data Prepper in [Getting Started with OpenSearch Data Prepper]({{site.url}}{{site.baseurl}}/data-prepper/getting-started#1-installing-data-prepper).
If you have OpenSearch running on your host and want to run Data Prepper Docker container with Logstash configuration, follow these steps:

2. Run the Docker image installed in Step 1 by supplying your `logstash.conf` configuration.
1. Update the `elasticsearch` section of `logstash.conf` to point to your OpenSearch instance. The host name has to match the OpenSearch certificate SANs, for example `node-0.example.com` if demo installation is used.

Check failure on line 34 in _data-prepper/migrating-from-logstash-data-prepper.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.SubstitutionsError] Use 'hostname' instead of 'host name'.

Raw Output:
{"message": "[OpenSearch.SubstitutionsError] Use 'hostname' instead of 'host name'.", "location": {"path": "_data-prepper/migrating-from-logstash-data-prepper.md", "range": {"start": {"line": 34, "column": 100}}}, "severity": "ERROR"}

```
docker run --name data-prepper -p 4900:4900 -v ${PWD}/logstash.conf:/usr/share/data-prepper/pipelines.conf opensearchproject/data-prepper:latest pipelines.conf
```
```
input {
http {
port => 4910 # Note the port used in this example
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
tag_on_failure => []
}
}
output {
# Point this at your OpenSearch/OSD endpoint
elasticsearch {
hosts => ["https://node-0.example.com:9200"] # change to your host:port
index => "logstash-%{+YYYY.MM.dd}"
user => "admin"
password => "<admin_pass>"
ssl => true
ssl_certificate_verification => true
}
}
```
{% include copy-curl.html %}

1. Supply your `logstash.conf` configuration to Data Prepper Docker container, using the following command:

```bash
docker run --rm --name data-prepper \
--add-host node-0.example.com:host-gateway \
-p 4910:4910 \
-v "${PWD}/logstash.conf:/usr/share/data-prepper/logstash.conf" \
--entrypoint bin/data-prepper \
opensearchproject/data-prepper:latest \
/usr/share/data-prepper/logstash.conf \
/usr/share/data-prepper/config/data-prepper-config.yaml
```
{% include copy-curl.html %}

The `logstash.conf` file is converted to `logstash.yaml` by mapping the plugins and attributes in the Logstash configuration to the corresponding plugins and attributes in Data Prepper.
You can find the converted `logstash.yaml` file in the same directory where you stored `logstash.conf`.
You can find the converted `logstash.yaml` file in the same directory where you stored `logstash.conf`. See the converted `logstash.yaml` sample file:

```
logstash-converted-pipeline:
source:
http:
max_connection_count: 500
request_timeout: 10000
port: 4910
processor:
- grok:
match:
message:
- "%{COMBINEDAPACHELOG}"
sink:
- opensearch:
hosts:
- "https://node-0.example.com:9200"
username: "admin"
password: "<admin_pass>"
index: "logstash-%{yyyy.MM.dd}"
```


The following output in your terminal indicates that Data Prepper is running correctly:

```
INFO org.opensearch.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer
INFO org.opensearch.dataprepper.plugins.source.loghttp.HTTPSource - Started http source on port 4910...
```

To test this further, run the following command on your host to push sample data to Data Prepper:

```bash
curl -X POST "http://localhost:4910/log/ingest" \
-H "Content-Type: application/json" \
-d '[{"message":"hello"}]'
```
{% include copy-curl.html %}

After a couple of seconds you can query OpenSearch `logstash-*` index for this document:

```bash
curl -k -uadmin:"<admin_pass>" "https://localhost:9200/logstash-*/_search?pretty"
```
{% include copy-curl.html %}

The sample document will be returned:

```json
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "logstash-2025.10.10",
"_id" : "dHnSzZkBIk7UWjH_Kjxh",
"_score" : 1.0,
"_source" : {
"message" : "hello"
}
}
]
}
}
```

Loading