Cloud DevOps Monitoring Observability Azure AWS

Cloud Monitoring and Observability

Mind Map Summary

  • Topic: Cloud Monitoring and Observability
  • Definition:
    • Monitoring: The act of collecting and analyzing data to determine if a system is working as expected. It focuses on known unknowns (e.g., CPU usage, memory, network traffic, error rates) and typically involves predefined dashboards and alerts.
    • Observability: The ability to infer the internal state of a system by examining its external outputs. It helps you understand why something is happening, even for unknown unknowns. It’s about being able to ask arbitrary questions about your system without having to deploy new code.
  • Key Concepts:
    • The Three Pillars of Observability:
      • 1. Logs: Discrete, timestamped records of events that happen within a system. They provide detailed context about specific occurrences (e.g., “User logged in,” “Error processing request,” “Database query executed”).
      • 2. Metrics: Numerical values measured over time, aggregated and stored as time-series data. They provide a high-level overview of system health and performance (e.g., CPU utilization, request count, latency, error rate).
      • 3. Traces (Distributed Tracing): Represent the end-to-end journey of a single request or transaction as it flows through multiple services in a distributed system. They show the sequence of operations, their duration, and dependencies, helping to pinpoint latency and errors across microservices.
    • Key Cloud Services:
      • Azure:
        • Azure Monitor: A comprehensive solution for collecting, analyzing, and acting on telemetry from your Azure and on-premises environments.
        • Application Insights: An extension of Azure Monitor specifically for monitoring live web applications. It automatically collects performance metrics, error rates, and user behavior.
        • Log Analytics: A service within Azure Monitor for querying and analyzing logs collected from various sources.
      • AWS:
        • Amazon CloudWatch: Collects monitoring and operational data in the form of logs, metrics, and events. Provides dashboards, alarms, and automated actions.
        • AWS X-Ray: Helps developers analyze and debug distributed applications, providing an end-to-end view of requests.
        • AWS CloudTrail: Records API calls and related events made in your AWS account, providing an audit trail.
        • Managed Prometheus/Grafana: AWS offers managed services for these popular open-source monitoring tools.
    • Alerting: Automated notifications triggered when predefined thresholds for metrics or log patterns are breached, indicating a potential issue.
    • Dashboards: Visual representations of metrics and logs, providing a quick overview of system health and trends.
    • Instrumentation: The process of adding code or configuration to an application to collect telemetry data (logs, metrics, traces).
  • Benefits (Pros):
    • Faster Problem Detection & Resolution: Quickly identify and diagnose issues, reducing Mean Time To Resolution (MTTR).
    • Improved System Reliability & Stability: Proactive identification of potential problems before they impact users.
    • Better Performance Optimization: Pinpoint bottlenecks, inefficient code, and resource hogs.
    • Enhanced User Experience: Minimize downtime and performance degradation, leading to higher user satisfaction.
    • Informed Decision Making: Provides data-driven insights for capacity planning, architectural improvements, and resource allocation.
    • Security & Compliance: Logs and traces can be used for security auditing and compliance requirements.
  • Challenges (Cons):
    • Complexity in Distributed Systems: Setting up comprehensive observability across many microservices can be challenging.
    • Data Volume & Cost: Collecting, storing, and analyzing large amounts of telemetry data can be expensive.
    • Alert Fatigue: Too many alerts (or poorly configured ones) can lead to teams ignoring warnings.
    • Tool Sprawl: Managing multiple monitoring and observability tools can be cumbersome.
    • Instrumentation Overhead: Adding code to collect data can sometimes introduce a slight performance overhead.
    • Data Correlation: Correlating logs, metrics, and traces across different systems can be difficult without proper planning.
  • Practical Use:
    • Ensuring the health and performance of cloud-native applications.
    • Troubleshooting issues in distributed microservices architectures.
    • Capacity planning and resource optimization.
    • Understanding user behavior and application usage patterns.
    • Security auditing and incident response.

Core Concepts

The distinction between monitoring and observability is subtle but important. Monitoring tells you if your system is working (e.g., “CPU is at 80%”). Observability allows you to ask why it’s at 80% (e.g., “Is it a specific user request? A background job? A database query?”). The three pillars (logs, metrics, traces) provide the raw data, and effective tooling (like Azure Monitor/Application Insights or CloudWatch/X-Ray) helps in collecting, correlating, and visualizing this data to gain insights.

Practice Exercise

Instrument an ASP.NET Core application with Application Insights (Azure) or AWS X-Ray. Demonstrate how to trace a request from the front-end, through the API, and to a database query. Set up an alert for when the average response time exceeds a certain threshold.

Answer (Instrumenting ASP.NET Core with Azure Application Insights)

For this exercise, we’ll use Azure Application Insights.

1. Prerequisites

  • An Azure subscription.
  • An Application Insights resource created in Azure Portal.
  • An ASP.NET Core Web API project.

2. Install NuGet Packages

In your ASP.NET Core project, install the Application Insights SDK:

dotnet add package Microsoft.ApplicationInsights.AspNetCore
dotnet add package Microsoft.ApplicationInsights.DependencyCollector # For database/HTTP dependency tracking

3. Configure Application Insights in Program.cs or Startup.cs

For .NET 6+ (Minimal APIs - Program.cs):

using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Configuration; // Needed for Configuration

var builder = WebApplication.CreateBuilder(args);

// Add services to the container.
builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

// Add Application Insights telemetry
builder.Services.AddApplicationInsightsTelemetry(); // Reads instrumentation key from appsettings.json

// Simulate a database repository
builder.Services.AddSingleton<IProductRepository, ProductRepository>();

var app = builder.Build();

// Configure the HTTP request pipeline.
if (app.Environment.IsDevelopment())
{
    app.UseSwagger();
    app.UseSwaggerUI();
}

app.UseHttpsRedirection();
app.UseAuthorization();
app.MapControllers();

app.Run();

// --- ProductRepository and Product classes (same as previous examples) ---
public class Product
{
    public int Id { get; set; }
    public string Name { get; set; }
    public decimal Price { get; set; }
}

public interface IProductRepository
{
    Task<IEnumerable<Product>> GetAllProductsAsync();
    Task<Product> GetByIdAsync(int id);
}

public class ProductRepository : IProductRepository
{
    private static readonly List<Product> _products = new List<Product>
    {
        new Product { Id = 1, Name = "Laptop", Price = 1200.00m },
        new Product { Id = 2, Name = "Mouse", Price = 25.00m },
        new Product { Id = 3, Name = "Keyboard", Price = 75.00m }
    };

    public async Task<IEnumerable<Product>> GetAllProductsAsync()
    {
        // Simulate database latency
        await Task.Delay(500);
        return _products;
    }

    public async Task<Product> GetByIdAsync(int id)
    {
        // Simulate database latency
        await Task.Delay(200);
        return _products.FirstOrDefault(p => p.Id == id);
    }
}

For .NET 5 and earlier (Startup.cs):

// Startup.cs
public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }

    public IConfiguration Configuration { get; }

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddControllers();
        services.AddApplicationInsightsTelemetry(); // Reads instrumentation key from appsettings.json
        services.AddSingleton<IProductRepository, ProductRepository>(); // Your repository
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        // ... other middleware
        app.UseRouting();
        app.UseAuthorization();
        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }
}

4. Add Instrumentation Key to appsettings.json

Get the “Instrumentation Key” or “Connection String” from your Application Insights resource in the Azure Portal.

{
  "ApplicationInsights": {
    "ConnectionString": "InstrumentationKey=YOUR_INSTRUMENTATION_KEY_HERE;IngestionEndpoint=https://eastus-8.in.applicationinsights.azure.com/"
    // Or just "InstrumentationKey": "YOUR_INSTRUMENTATION_KEY_HERE" for older versions
  },
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*"
}

5. Create a Controller to Simulate API Calls and Database Access

using Microsoft.AspNetCore.Mvc;
using System.Collections.Generic;
using System.Threading.Tasks;

[ApiController]
[Route("api/[controller]")]
public class ProductsController : ControllerBase
{
    private readonly IProductRepository _productRepository;

    public ProductsController(IProductRepository productRepository)
    {
        _productRepository = productRepository;
    }

    [HttpGet]
    public async Task<ActionResult<IEnumerable<Product>>> Get()
    {
        // This will be tracked as a dependency call by Application Insights
        var products = await _productRepository.GetAllProductsAsync();
        return Ok(products);
    }

    [HttpGet("{id}")]
    public async Task<ActionResult<Product>> Get(int id)
    {
        var product = await _productRepository.GetByIdAsync(id);
        if (product == null)
        {
            return NotFound();
        }
        return Ok(product);
    }
}

6. Run the Application and Generate Traffic

Run your ASP.NET Core application. Make several requests to /api/products and /api/products/{id}.

7. Observe Telemetry in Azure Portal

Navigate to your Application Insights resource in the Azure Portal.

  • Overview: You’ll see a summary of requests, response times, and failures.
  • Performance:
    • Under “Operations,” you’ll see your API endpoints (e.g., GET /api/Products). Click on one to drill down.
    • Under “Dependencies,” you’ll see calls to your ProductRepository (which Application Insights automatically detects as a dependency, even though it’s simulated Task.Delay). If you were using EF Core or a real database, you’d see SQL queries here.
    • End-to-end transaction details: Click on a sample request. You’ll see a visual representation of the request’s journey, including the incoming HTTP request, the call to your controller, and the simulated database call (ProductRepository.GetAllProductsAsync or GetByIdAsync). This is the distributed tracing in action.
  • Logs (Analytics): Use Kusto Query Language (KQL) to query your telemetry.
    • To see requests: requests
    • To see dependencies: dependencies
    • To see custom logs: traces (if you add ILogger and log messages)

8. Set Up an Alert for Average Response Time

  1. In your Application Insights resource, go to Alerts -> Create alert rule.
  2. Scope: Your Application Insights resource.
  3. Condition:
    • Signal name: Search for “Server response time” (or “Requests (server response time)”).
    • Dimension name: You can filter by “Operation name” (e.g., GET /api/Products) if you want to alert on a specific endpoint.
    • Aggregation type: “Average”.
    • Threshold: “Static”.
    • Operator: “Greater than”.
    • Threshold value: Enter a value in milliseconds (e.g., 500 for 500ms).
    • Frequency of evaluation: How often to check (e.g., “Every 1 minute”).
    • Lookback period: How far back to look for data (e.g., “Last 5 minutes”).
  4. Actions:
    • Create an “Action group” to define who gets notified (email, SMS, webhook, etc.).
  5. Details:
    • Give the alert a name and description.
    • Set severity.
  6. Create alert rule.

Now, if the average response time for your application (or a specific endpoint) exceeds 500ms for the configured lookback period, you will receive an alert.

This demonstrates how Application Insights provides comprehensive monitoring and observability, allowing you to trace requests, analyze performance, and set up proactive alerts.