Azure Monitor Application Insights for Enterprise Observability and Secure Telemetry

Overview

Azure Monitor Application Insights is an application performance management and observability service within Azure Monitor. It collects requests, dependencies, exceptions, traces, availability results, custom events, and distributed traces from web apps, APIs, background services, containers, and client applications.

Enterprises adopt it to centralize telemetry across Azure, on-premises, and multicloud workloads, then query that data with Kusto Query Language (KQL) in a Log Analytics workspace. Key benefits include:

• End-to-end transaction visibility across microservices
• Faster incident triage with correlated logs, metrics, and traces
• Application SLO monitoring using alerts, workbooks, and availability tests
• Governed ingestion and retention for compliance and cost control
• Native integration with Azure Policy, RBAC, Microsoft Sentinel, and automation pipelines

Architecture

Core components

• Instrumentation SDKs / OpenTelemetry: .NET, Java, Node.js, Python, JavaScript, and Azure Monitor OpenTelemetry exporters
• Application Insights resource: logical monitoring endpoint and experience layer
• Log Analytics workspace: stores telemetry when using workspace-based Application Insights
• Ingestion endpoints: receive telemetry over HTTPS/TLS
• Analytics and alerting: KQL, workbooks, metric alerts, log alerts, smart detection

Deployment models

• Workspace-based Application Insights: recommended for enterprises because it supports centralized retention, CMK options via workspace features, RBAC alignment, and cross-resource analytics
• Classic Application Insights: legacy model; avoid for new deployments

Data flow

1. Application emits telemetry through SDK or OpenTelemetry collector.
2. Telemetry is buffered locally and sent to Azure Monitor ingestion endpoints.
3. Data is stored in the linked Log Analytics workspace.
4. Teams analyze data with KQL, dashboards, alerts, and incident tooling.
5. Optional export paths include Event Hub, Storage, or downstream SIEM integration.

Implementation Guide

1. Create a resource group and Log Analytics workspace.
2. Create a workspace-based Application Insights instance.
3. Retrieve the connection string.
4. Instrument the application with OpenTelemetry or native SDK.
5. Configure sampling, retention, alerts, and RBAC.

az group create --name rg-observability-prod --location westeurope
az monitor log-analytics workspace create --resource-group rg-observability-prod --workspace-name law-prod-observability --location westeurope --sku PerGB2018
az monitor app-insights component create --app appi-prod-payments --location westeurope --resource-group rg-observability-prod --workspace law-prod-observability --application-type web
az monitor app-insights component show --app appi-prod-payments --resource-group rg-observability-prod --query connectionString -o tsv

Create an alert for failed requests:

APP_ID=$(az monitor app-insights component show --app appi-prod-payments --resource-group rg-observability-prod --query id -o tsv)
az monitor metrics alert create --name appi-high-failures --resource-group rg-observability-prod --scopes $APP_ID --condition "count requests/failed > 10 where cloud/roleName includes payments-api" --window-size 5m --evaluation-frequency 1m --severity 2

Recommended Python package installation:

pip install azure-monitor-opentelemetry opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-requests
export APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=xxxx;IngestionEndpoint=https://westeurope-5.in.applicationinsights.azure.com/"

Code Examples

1. Bash deployment

az monitor app-insights component create --app appi-prod-orders --location northeurope --resource-group rg-platform-prod --workspace law-platform-prod --application-type web

2. YAML app settings

apiVersion: v1
kind: ConfigMap
metadata:
  name: orders-api-observability
data:
  APPLICATIONINSIGHTS_CONNECTION_STRING: "InstrumentationKey=11111111-2222-3333-4444-555555555555;IngestionEndpoint=https://northeurope-5.in.applicationinsights.azure.com/"
  OTEL_SERVICE_NAME: "orders-api"
  OTEL_TRACES_SAMPLER: "parentbased_traceidratio"
  OTEL_TRACES_SAMPLER_ARG: "0.2"

3. Python OpenTelemetry instrumentation

from fastapi import FastAPI
from azure.monitor.opentelemetry import configure_azure_monitor
import logging

configure_azure_monitor(connection_string="InstrumentationKey=11111111-2222-3333-4444-555555555555;IngestionEndpoint=https://northeurope-5.in.applicationinsights.azure.com/")
app = FastAPI()
logger = logging.getLogger("orders-api")

@app.get("/health")
def health():
    logger.info("health check ok", extra={"custom_dimensions": {"service": "orders-api", "env": "prod"}})
    return {"status": "ok"}

Security Hardening

• Use workspace-based deployments and restrict access with Azure RBAC roles such as Monitoring Reader and Log Analytics Reader.
• Disable broad contributor access; separate platform operators from application developers.
• Send telemetry only over TLS 1.2+ and use approved egress paths or private networking controls where required.
• Protect secrets by storing connection strings in Azure Key Vault or Kubernetes secrets, not source control.
• Apply sampling to reduce sensitive data exposure and cost.
• Mask or avoid collecting PII, tokens, session IDs, and payload bodies.
• Enable diagnostic settings and activity log monitoring for the Application Insights and workspace resources.

Comparison

Troubleshooting

1. No telemetry received

Log sample:

2025-02-14T10:21:33.412Z Warning AzureMonitorExporter Failed to export span batch. StatusCode=403 Response=Forbidden

Fix: verify the connection string, outbound firewall rules, and whether the resource was deleted or moved.

2. Sampling too aggressive

Log sample:

2025-02-14T10:26:11.908Z Info OpenTelemetry TraceIdRatioBased sampler configured with rate=0.01
2025-02-14T10:27:02.117Z Warning Low request volume visible in Application Insights compared to ingress logs

Fix: increase OTEL_TRACES_SAMPLER_ARG to a realistic value such as 0.1 or 0.2 for production APIs.

3. Missing dependency correlation

Log sample:

2025-02-14T10:31:44.551Z Error requests.packages.urllib3.connectionpool HTTPSConnectionPool(host='api.internal', port=443): Max retries exceeded

Fix: enable the correct OpenTelemetry instrumentation package for outbound HTTP libraries and confirm W3C trace context propagation.

Best Practices

Do

• Standardize on OpenTelemetry for new services.
• Use naming conventions like appi-prod-payments and law-prod-observability.
• Create KQL-based alerts for error rate, latency p95, and dependency failures.
• Separate environments into dedicated workspaces when compliance or noisy-neighbor risk requires it.
• Document custom dimensions such as tenantId, service, and deploymentRing.

Don't

• Don't store secrets or customer payloads in traces.
• Don't give developers unrestricted workspace write permissions.
• Don't leave default retention and sampling unreviewed.
• Don't mix production and test telemetry in the same resource without tagging and governance.

A practical enterprise pattern is to use one central Log Analytics workspace per regulated boundary, one Application Insights resource per critical application, OpenTelemetry-based instrumentation, and alert routing into ITSM or Microsoft Sentinel.