ELK Stack Setup: Install Elasticsearch, Logstash, Kibana Securely

This article is a localized edition of the Japanese original. Read the Japanese original

The ELK stack combines Elasticsearch (storage and search), Logstash (processing), and Kibana (visualization) into a log platform. Current 8.x releases ship with TLS and user authentication turned on from the moment you install them, so if you work with that design instead of against it, you can build a secure setup without switching security off. This article walks through installing all three on a single Ubuntu/Debian server, collecting Nginx access logs with Filebeat, and searching them in Kibana.

Architecture and what to decide first

The log flow is "Filebeat → Logstash → Elasticsearch → Kibana." Filebeat is a lightweight shipper that reads log files on each server and sends them on. Logstash receives them and breaks each access-log line into fields such as the IP address, request, and status code. Elasticsearch stores the parsed data, and Kibana lets you search and chart it in the browser.

For a small setup you can run everything on one machine, but Elasticsearch uses a lot of memory. If you run all three together, plan on about 8 GB of RAM. It will start with 4 GB, but as the log volume grows, searches slow down and processes are more likely to be killed for lack of memory.

As a rule, keep all three products on the same version. Mixing them, for example Elasticsearch 8.15 with Kibana 8.12, is not supported. (As of October 2026, 9.x releases are also available. The procedure is almost the same as for 8.x; read "8.x" in the repository URL as "9.x.")

Installing Elasticsearch

Register Elastic's official APT repository and install from it.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
sudo apt-get install -y apt-transport-https
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install -y elasticsearch

During installation, the output shows a "Security autoconfiguration information" section, which prints the password for the built-in elastic superuser exactly once. At this point, the TLS certificates for HTTP traffic and node-to-node traffic are also generated automatically and placed in /etc/elasticsearch/certs/. If you missed the password, you can reset it with the following command.

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

Before starting the service, decide on the heap size. Version 8.x sets it automatically from the installed memory, but on a shared server it is more stable to set it explicitly so that memory is left for Logstash and Kibana. As a guideline, use half of the installed memory or less, and set the minimum and maximum to the same value.

# /etc/elasticsearch/jvm.options.d/heap.options
-Xms2g
-Xmx2g

Also run sysctl vm.max_map_count and check that the value is 262144 or higher. If it is too low, the startup checks may stop Elasticsearch. Then start the service and confirm that it responds over HTTPS with authentication.

sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch
sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200

If you enter the password and get back JSON that includes the cluster name and version number, it worked. Do not work around problems here with "I get a certificate error, so I'll add -k" or "it's a hassle, so I'll set xpack.security.enabled: false." The former throws away verification of who you are talking to, and the latter lets anyone who can reach port 9200 read, write, and delete your data. There have been repeated real cases of Elasticsearch exposed without authentication, with the data wiped and a ransom demanded.

Note that the autoconfiguration writes http.host: 0.0.0.0 to /etc/elasticsearch/elasticsearch.yml, so the HTTP API (port 9200) listens on all network interfaces. Traffic is encrypted and requires authentication, but if you only use Elasticsearch from the same server, you can reduce your exposure to attacks by changing this value to localhost or by closing external access to port 9200 in the firewall.

Installing Kibana and connecting it

Install Kibana from the same repository. For the connection to Elasticsearch, use an enrollment token, a temporary one-time secret, and the certificate placement is handled automatically as well.

sudo apt-get install -y kibana
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
sudo /usr/share/kibana/bin/kibana-setup --enrollment-token <the token printed above>
sudo systemctl enable --now kibana

The token is valid for 30 minutes. If it fails because too much time has passed, create a new token.

By default, Kibana listens only on localhost:5601. To use it from a browser, the common approaches are to connect from your own PC through an SSH port forward, or to put Nginx in front as an HTTPS reverse proxy and restrict access by source IP. Avoid opening port 5601 directly to the internet.

If you get warnings about features such as alerting, generate encryption keys with kibana-encryption-keys generate, add the configuration lines it prints to /etc/kibana/kibana.yml, and restart.

When the login screen appears, sign in as the elastic user. However, because elastic has every privilege, it is safer to create a separate user with limited permissions for everyday viewing, under "Users" in "Stack Management."

Creating a write-only user for Logstash

If you give Logstash the password of elastic, a leaked configuration file would do far more damage. Create a role and user that can operate only on the target indices.

CA=/etc/elasticsearch/certs/http_ca.crt
sudo curl --cacert $CA -u elastic -X POST "https://localhost:9200/_security/role/logstash_writer" \
  -H 'Content-Type: application/json' -d '{
  "cluster": ["monitor", "manage_index_templates", "manage_ilm"],
  "indices": [{
    "names": ["nginx-access-*"],
    "privileges": ["write", "create", "create_index", "manage", "manage_ilm"]
  }]
}'
sudo curl --cacert $CA -u elastic -X POST "https://localhost:9200/_security/user/logstash_internal" \
  -H 'Content-Type: application/json' -d '{
  "password": "your-strong-password",
  "roles": ["logstash_writer"]
}'

You can do the same thing from "Roles" and "Users" under "Stack Management" in Kibana.

Configuring the Logstash pipeline

Install Logstash, and copy Elasticsearch's certificate to a location Logstash can read.

sudo apt-get install -y logstash
sudo mkdir -p /etc/logstash/certs
sudo cp /etc/elasticsearch/certs/http_ca.crt /etc/logstash/certs/
sudo chown -R root:logstash /etc/logstash/certs && sudo chmod 640 /etc/logstash/certs/http_ca.crt

Do not write the password in plain text in a configuration file; put it in the Logstash keystore.

sudo /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
sudo /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_PW

Place the pipeline in /etc/logstash/conf.d/. It has three stages: receiving from Beats, parsing Nginx's standard (combined) format, and output to Elasticsearch.

# /etc/logstash/conf.d/nginx.conf
input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    ssl_certificate_authorities => ["/etc/logstash/certs/http_ca.crt"]
    user => "logstash_internal"
    password => "${ES_PW}"
    index => "nginx-access-%{+YYYY.MM.dd}"
  }
}

The date filter makes Logstash use the time written in the log line as the event time (@timestamp). Without it, events are recorded with "the time Logstash received them," so the timeline drifts when logs arrive late or when you import old logs.

Running a syntax check before starting helps you avoid startup failures that are hard to diagnose.

sudo -u logstash /usr/share/logstash/bin/logstash --path.settings /etc/logstash -t
sudo systemctl enable --now logstash

If it prints "Configuration OK," the syntax is fine. Errors after startup appear in /var/log/logstash/logstash-plain.log. For an authentication error (401), recheck the username and the keystore value. For a certificate error, recheck the path in ssl_certificate_authorities and its read permissions.

Sending logs with Filebeat

Install Filebeat on the server that produces the logs. If it is the same server, you can install it right away.

sudo apt-get install -y filebeat

Set the input and output in /etc/filebeat/filebeat.yml as shown below. output.elasticsearch is enabled by default, so comment it out and leave a single output.

filebeat.inputs:
  - type: filestream
    id: nginx-access
    paths:
      - /var/log/nginx/access.log

output.logstash:
  hosts: ["localhost:5044"]

sudo filebeat test config
sudo filebeat test output
sudo systemctl enable --now filebeat

If you send to port 5044 from another server, restrict the source IPs in the firewall and also encrypt the connection between Filebeat and Logstash with TLS. Logs contain personal information such as IP addresses and URLs, so it is safer not to send them in plain text over an external network.

Checking logs in Kibana

In Kibana, go to "Stack Management" → "Data Views" and create a data view with the index pattern nginx-access-* and the time field @timestamp. Then open "Discover" and the ingested logs appear in chronological order.

If no data appears, work through the pipeline from upstream to downstream.

• Whether the index exists in Elasticsearch: curl --cacert ... -u elastic "https://localhost:9200/_cat/indices/nginx-access-*?v"
• Whether the Logstash log shows 401 or certificate errors
• Whether the Filebeat log (journalctl -u filebeat) shows connection errors
• Whether the time range shown in Discover includes the log timestamps

Lines that grok could not parse get the tag _grokparsefailure. If you have customized Nginx's log_format, this tag will appear in large numbers, so adjust the pattern to match your actual format.

Operational tips

Left alone, logs will use up your disk. Elasticsearch stops accepting new writes once disk usage passes a certain level, so the problem shows up as "logs suddenly stopped arriving one day." In Kibana's "Index Lifecycle Policies," create a policy that deletes indices older than, for example, 30 days, and apply it to the target indices.

Upgrade in the order Elasticsearch → Kibana → Logstash → Filebeat, keeping the versions aligned. If only some packages are upgraded first by apt upgrade, you get inconsistencies, so consider pinning the versions with apt-mark hold and upgrading on a planned schedule.

Summary

• With 8.x, TLS and authentication are enabled by default. Connect using the certificate and the auto-generated password, and do not disable them
• Set the Elasticsearch heap minimum and maximum to the same value, and keep it to half of the installed memory or less on a shared server
• Connect Kibana to Elasticsearch with an enrollment token, and do not expose port 5601 directly
• Create a write-only user for Logstash and keep the password in the keystore
• When data does not arrive, check the logs in the order Elasticsearch → Logstash → Filebeat to isolate the problem
• Use a lifecycle policy to delete old indices and prevent running out of disk space

• How to think about Elasticsearch shard design
• How to collect logs with Fluentd

Questions about WordPress? The publisher of this site is based in Japan and replies in Japanese. Contact (Japanese)