AWS X-Ray for Enterprise Observability and Distributed Tracing
Prerequisites
- Working knowledge of AWS IAM, Lambda, ECS, or EC2
- AWS CLI configured with permissions to manage IAM and X-Ray
Steps
AWS X-Ray helps enterprises trace requests across distributed applications, identify latency bottlenecks, and improve service reliability. This guide explains its architecture, implementation, security controls, and operational practices for production AWS environments.
Overview
AWS X-Ray is a managed distributed tracing service that records end-to-end request paths through applications built on AWS and hybrid environments. It helps engineering and platform teams understand latency, dependency maps, fault domains, and service behavior across microservices, Lambda functions, containers, and API layers.
Enterprises use X-Ray to reduce mean time to detect and resolve performance issues, validate service dependencies during change windows, and support observability programs alongside CloudWatch, OpenTelemetry, and SIEM tooling. It is especially useful where requests cross multiple AWS services and where teams need a managed option tightly integrated with IAM, ECS, EKS, Lambda, API Gateway, and Elastic Beanstalk.
Architecture
Core components include:
- X-Ray SDK or instrumentation library in the application
- X-Ray daemon or ADOT collector to buffer and send segments
- X-Ray service backend for trace processing, indexing, and service maps
- CloudWatch Logs and metrics for correlated operational telemetry
- IAM roles and policies controlling publish and read access
Deployment models:
- Lambda-native tracing with active tracing enabled
- EC2/ECS/EKS using the X-Ray daemon sidecar or AWS Distro for OpenTelemetry Collector
- Hybrid workloads forwarding traces from on-premises to AWS endpoints
View as text
[Client] ---- 1. request ----> [ALB/API Gateway] ---- 2. invoke ----> [App Service] [App Service] ---- 3. subsegment ----> [RDS] [App Service] ---- 4. subsegment ----> [SQS] [App Service] ---- 5. UDP/TCP ----> [X-Ray Daemon/ADOT Collector] ---- 6. PutTraceSegments ----> [AWS X-Ray] [AWS X-Ray] ---- 7. insights/service map ----> [Operators]
Data flow:
- A request enters an instrumented service.
- The SDK creates a trace header, segment, and subsegments.
- The daemon or collector batches telemetry.
- X-Ray stores and analyzes traces, then renders service maps and latency breakdowns.
Implementation Guide
- Create an IAM policy for publishing traces.
aws iam create-policy --policy-name XRayWriteAccessCustom --policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["xray:PutTraceSegments","xray:PutTelemetryRecords","xray:GetSamplingRules","xray:GetSamplingTargets","xray:GetSamplingStatisticSummaries"],"Resource":"*"}]}'
- Attach the policy to an EC2 instance profile, ECS task role, or EKS service account role.
aws iam attach-role-policy --role-name app-observability-role --policy-arn arn:aws:iam::123456789012:policy/XRayWriteAccessCustom
- Install and start the X-Ray daemon on Amazon Linux 2.
sudo yum install -y https://s3.dualstack.us-east-1.amazonaws.com/aws-xray-assets.us-east-1/xray-daemon/aws-xray-daemon-3.x.rpm
sudo systemctl enable xray
sudo systemctl start xray
sudo systemctl status xray
- Configure daemon binding and region.
Socket:
UDPAddress: "0.0.0.0:2000"
Logging:
LogLevel: "info"
TotalBufferSizeMB: 64
Region: "us-east-1"
LocalMode: false
- Enable tracing for Lambda.
aws lambda update-function-configuration --function-name orders-api --tracing-config Mode=Active
- Query traces.
aws xray get-service-graph --start-time 2026-11-19T08:00:00Z --end-time 2026-11-19T09:00:00Z
aws xray batch-get-traces --trace-ids 1-6901a2b3-cdef0123456789abcdef0123
Code Examples
Python application instrumentation
from aws_xray_sdk.core import xray_recorder, patch_all
from flask import Flask
import requests
patch_all()
xray_recorder.configure(service='orders-service')
app = Flask(__name__)
@app.route('/health')
def health():
with xray_recorder.in_subsegment('call_inventory'):
requests.get('https://inventory.internal/api/check', timeout=2)
return {'status': 'ok'}
ECS sidecar configuration
version: '3'
services:
app:
image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/orders:1.4.2
environment:
AWS_XRAY_DAEMON_ADDRESS: xray-daemon:2000
xray-daemon:
image: public.ecr.aws/xray/aws-xray-daemon:3.3.3
ports:
- "2000:2000/udp"
command: ["-o", "-n", "us-east-1"]
Terraform IAM role attachment
resource "aws_iam_role_policy_attachment" "xray_write" {
role = aws_iam_role.app_role.name
policy_arn = "arn:aws:iam::aws:policy/AWSXRayDaemonWriteAccess"
}
Security Hardening
- Use least privilege IAM: writers need
PutTraceSegmentsandPutTelemetryRecords; readers should be separated. - Prefer private networking with VPC endpoints where supported for related observability paths and restrict outbound egress from collectors.
- Protect trace data because metadata may contain business-sensitive fields. Do not record secrets, tokens, session IDs, or regulated payloads.
- Encrypt surrounding telemetry stores such as CloudWatch Logs with KMS and enforce TLS for application dependencies.
- Use sampling rules to reduce exposure and cost while keeping critical transactions fully traced.
- Restrict console and API access with IAM conditions, MFA, and cross-account read-only roles for operations teams.
Comparison
| Feature | AWS X-Ray | Datadog APM | New Relic Distributed Tracing |
|---|---|---|---|
| Pricing | Pay-per-trace recorded/retrieved, AWS-native billing | Host and ingest based, broader platform pricing | User and ingest based, platform pricing |
| Deployment | Managed AWS service, SDK/daemon/ADOT | SaaS agents and collectors | SaaS agents and collectors |
| Scalability | Strong for AWS-native workloads and serverless | Strong multi-cloud and hybrid support | Strong multi-cloud and enterprise support |
| Security | IAM integration, AWS account boundaries, CloudTrail visibility | RBAC, SaaS controls, tenant isolation | RBAC, SaaS controls, tenant isolation |
| Best fit | AWS-centric enterprises | Cross-platform observability programs | Full-stack telemetry with broad APM features |
Troubleshooting
- Daemon not reachable
2026-11-19T08:14:22Z [ERROR] Failed to send segment: dial udp 127.0.0.1:2000: connect: connection refused
Fix: verify daemon is running, container sidecar DNS is correct, and UDP port 2000 is open.
- Missing IAM permissions
2026-11-19T08:16:03Z [ERROR] AccessDeniedException: User: arn:aws:sts::123456789012:assumed-role/app-observability-role/i-0abc is not authorized to perform: xray:PutTraceSegments
Fix: attach AWSXRayDaemonWriteAccess or equivalent custom policy to the runtime role.
- Segment document too large
2026-11-19T08:21:44Z [WARN] Segment rejected: Segment too large to process; document size exceeded service limit
Fix: reduce annotations and metadata, avoid large payload capture, and split work into smaller subsegments.
Best Practices
Do
- Trace business-critical paths such as checkout, authentication, and order fulfillment.
- Standardize on OpenTelemetry or X-Ray SDK policy across teams.
- Add annotations for searchable fields like
tenant_id,env, andservice_version. - Correlate traces with CloudWatch logs using request IDs.
Don't
- Do not store PII, secrets, or raw request bodies in trace metadata.
- Do not enable 100% sampling globally in high-volume production without cost review.
- Do not give developers broad write and read permissions across all accounts.
Concrete example: sample 100% of POST /checkout in production canary stacks, but only 5% of low-risk GET /catalog traffic in steady state.
Have a project in mind?
Get an instant AI price estimate for it, or talk directly to our team.
One email a month on what we learn building with AI