Elastic Security Enterprise Guide: Architecture, Deployment, and Hardening

Overview

Elastic Security is Elastic's security operations platform built on Elasticsearch, Kibana, Elastic Agent, and integrations for endpoint, cloud, identity, and network telemetry. Its core purpose is to centralize high-volume security data, normalize it with ECS, detect threats with rules and ML jobs, and accelerate investigation through timelines, cases, and automated response.

Enterprises use Elastic Security because it scales well for heterogeneous environments, supports self-managed and cloud deployment models, and allows deep customization. It is especially strong where teams need to ingest logs from many sources, correlate across endpoint and SIEM telemetry, and keep control over data locality, retention, and detection engineering.

Architecture

Core components

• Elasticsearch stores security telemetry, alerts, cases, and enrichments.
• Kibana provides the Security app, dashboards, rule management, and case workflows.
• Elastic Agent collects endpoint, system, cloud, and network data; it can also provide endpoint prevention and response.
• Fleet Server manages Elastic Agents at scale.
• Integrations ingest data from AWS, Azure, Okta, Microsoft 365, firewalls, EDR tools, and custom sources.

Deployment models

• Elastic Cloud: fastest path, managed upgrades, simpler HA.
• Self-managed on Kubernetes or VMs: best for strict data residency and custom network controls.
• Hybrid: local collection with centralized analysis in Elastic Cloud.

Data flow

1. Elastic Agent or Beats collect telemetry.
2. Fleet policies route data to Elasticsearch data streams.
3. Ingest pipelines parse and map fields to ECS.
4. Detection rules query data streams and create alerts in .alerts-security.alerts-*.
5. Analysts investigate in Kibana, pivot to timelines, and trigger response actions.

Implementation Guide

1. Install Elasticsearch and Kibana with TLS enabled.

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.14.1-amd64.deb
curl -L -O https://artifacts.elastic.co/downloads/kibana/kibana-8.14.1-amd64.deb
sudo dpkg -i elasticsearch-8.14.1-amd64.deb kibana-8.14.1-amd64.deb
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

1. Configure Elasticsearch network, TLS, and API key auth.

# /etc/elasticsearch/elasticsearch.yml
cluster.name: sec-prod
node.name: es01
network.host: 10.10.20.11
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
xpack.security.transport.ssl.enabled: true
discovery.type: single-node

1. Configure Kibana to connect securely.

# /etc/kibana/kibana.yml
server.host: 0.0.0.0
elasticsearch.hosts: ["https://10.10.20.11:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.ssl.verificationMode: full
xpack.encryptedSavedObjects.encryptionKey: "7b8f4f7f9c7e4f1d8d9e1a2b3c4d5e6f"

1. Start services and verify health.

sudo systemctl enable --now elasticsearch kibana
curl -u elastic https://10.10.20.11:9200 -k
curl -I http://10.10.20.12:5601

1. Install Fleet Server and enroll an agent.

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.14.1-linux-x86_64.tar.gz
tar xzf elastic-agent-8.14.1-linux-x86_64.tar.gz
cd elastic-agent-8.14.1-linux-x86_64
sudo ./elastic-agent install --url=https://fleet.example.com:8220 --enrollment-token=<token> --certificate-authorities=/etc/pki/ca.crt

1. In Kibana, enable the Elastic Defend integration, add Windows/Linux/macOS policies, and activate prebuilt detection rules.
2. Create index lifecycle policies for hot-warm retention and tune data tiers for cost control.

Code Examples

curl -u elastic:$(cat /root/elastic.pw) -k -X POST "https://10.10.20.11:9200/_security/api_key" -H "Content-Type: application/json" -d '{"name":"soc-ingest","role_descriptors":{"ingest_role":{"cluster":["monitor"],"index":[{"names":["logs-*","metrics-*"],"privileges":["create_doc","auto_configure"]}]}}}'

# Fleet agent policy excerpt
inputs:
  - type: logfile
    streams:
      - paths:
          - /var/log/auth.log
        data_stream:
          dataset: system.auth
          type: logs
  - type: endpoint
    use_output: default
    meta:
      package:
        name: endpoint
        version: 8.14.1

from elasticsearch import Elasticsearch
es = Elasticsearch("https://10.10.20.11:9200", api_key="<id>:<key>", verify_certs=False)
query = {"query": {"bool": {"filter": [{"range": {"@timestamp": {"gte": "now-15m"}}}, {"term": {"event.category": "authentication"}}, {"term": {"event.outcome": "failure"}}]}}}
resp = es.search(index="logs-*", size=5, query=query["query"])
for hit in resp["hits"]["hits"]:
    print(hit["_source"].get("host", {}).get("name"), hit["_source"].get("user", {}).get("name"))

Security Hardening

• Enforce TLS 1.2+ on HTTP and transport layers; rotate node and client certificates.
• Use SAML/OIDC with MFA for Kibana access; avoid local superuser accounts for daily operations.
• Restrict privileges with spaces, role-based access control, and index-level permissions.
• Store secrets in Elastic Keystore instead of plaintext configs.
• Enable audit logging for Elasticsearch and Kibana.
• Isolate Fleet Server behind reverse proxy controls and allowlist agent egress only.
• Apply ILM, snapshot policies, and immutable backups for forensic retention.

Comparison

Troubleshooting

1. Agent enrollment fails

Log sample:

Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority

Fix: install the issuing CA on the host and pass --certificate-authorities=/path/ca.crt during enrollment.

2. Detection rules show partial failures

Log sample:

[plugins.securitySolution] search_phase_execution_exception: [illegal_argument_exception] Fielddata is disabled on [host.name]

Fix: verify ECS mappings, reindex malformed data, and ensure rules target .keyword fields where required.

3. Kibana cannot create alerts

Log sample:

[error][plugins.alerting] Unauthorized to get a "search" rule for "siem.signals"

Fix: grant the Kibana user manage, read, and alerting privileges in the Security feature space; verify encrypted saved objects key is set.

Best Practices

Do

• Standardize on ECS early to reduce parser drift.
• Separate prod, dev, and lab spaces and agent policies.
• Use data tiers and ILM, for example 7 days hot, 30 days warm, 180 days cold.
• Tune noisy prebuilt rules with host, user, and service account exceptions.

Don't

• Do not give analysts the superuser role; create SOC-specific roles.
• Do not send all debug logs indefinitely; high-cardinality data inflates storage and query cost.
• Do not expose Fleet Server or Elasticsearch directly to the internet without WAF, IP restrictions, and strong auth.
• Do not enable prevention policies broadly without phased testing on pilot groups first.