Telemetry Metrics¶

The Security Event Exporter provides comprehensive telemetry metrics to monitor its performance and operational status. These metrics help you understand the exporter's behavior, identify potential issues, and optimize performance.

Metrics Overview¶

graph TB
    subgraph "Input Metrics"
        A[Logs Received] --> B[Conversion Process]
    end

    subgraph "Processing Metrics"
        B --> C[Events Exported]
        B --> D[Conversion Errors]
    end

    subgraph "Output Metrics"
        C --> E[HTTP Requests]
        E --> F[HTTP Success]
        E --> G[HTTP Errors]
    end

    subgraph "Performance Metrics"
        H[Request Duration] --> I[Performance Analysis]
        J[Batch Size] --> K[Throughput Metrics]
    end

    style A fill:#e3f2fd
    style C fill:#c8e6c9
    style D fill:#ffcdd2
    style F fill:#c8e6c9
    style G fill:#ffcdd2

Core Metrics¶

Input Metrics¶

graph LR
    A[OTEL Collector] --> B[Logs Received]
    B --> C[Total Count]
    B --> D[Rate per Second]

    style B fill:#e3f2fd
    style C fill:#fff3e0
    style D fill:#f3e5f5

Processing Metrics¶

graph TD
    A[Logs Received] --> B{Conversion Success?}
    B -->|Yes| C[Events Exported]
    B -->|No| D[Conversion Errors]

    style C fill:#c8e6c9
    style D fill:#ffcdd2

Output Metrics¶

graph TD
    A[Events Exported] --> B[HTTP Requests]
    B --> C{HTTP Success?}
    C -->|Yes| D[HTTP Success]
    C -->|No| E[HTTP Errors]

    style D fill:#c8e6c9
    style E fill:#ffcdd2

Performance Metrics¶

graph LR
    A[HTTP Request] --> B[Request Duration]
    B --> C[Average Duration]
    B --> D[P95 Duration]
    B --> E[P99 Duration]

    style B fill:#fff3e0
    style C fill:#e3f2fd
    style D fill:#f3e5f5
    style E fill:#ffebee

Metrics Collection Flow¶

sequenceDiagram
    participant Exporter as Security Event Exporter
    participant Metrics as Metrics Store
    participant Logger as Structured Logger
    participant Monitor as Monitoring System

    Exporter->>Metrics: Update logs_received
    Exporter->>Metrics: Update events_exported
    Exporter->>Metrics: Update http_requests
    Exporter->>Metrics: Update http_duration

    Metrics->>Logger: Log metrics
    Logger->>Monitor: Send to monitoring system

    Monitor->>Monitor: Generate alerts
    Monitor->>Monitor: Update dashboards

Metrics Reporting¶

Startup Metrics¶

{
  "level": "debug",
  "msg": "Initialized telemetry metrics",
  "logs_received": 0,
  "events_exported": 0,
  "events_failed": 0,
  "conversion_errors": 0,
  "http_requests": 0,
  "http_errors": 0
}

Batch Processing Metrics¶

{
  "level": "info",
  "msg": "Completed processing logs batch",
  "total_resource_logs": 5,
  "total_log_records": 50,
  "successful_events": 48,
  "failed_events": 2,
  "http_requests": 1
}

Shutdown Metrics¶

{
  "level": "info",
  "msg": "Final telemetry metrics",
  "logs_received": 1000,
  "events_exported": 950,
  "events_failed": 50,
  "conversion_errors": 25,
  "http_requests": 20,
  "http_errors": 5,
  "http_duration_samples": 20
}

Performance Metrics¶

{
  "level": "info",
  "msg": "HTTP request performance metrics",
  "average_duration": "150ms",
  "sample_count": 20
}

Monitoring and Alerting¶

Key Performance Indicators¶

graph TB
    subgraph "KPIs"
        A[Success Rate] --> B[Events Exported / Total Events]
        C[Error Rate] --> D[Failed Events / Total Events]
        E[Throughput] --> F[Events per Second]
        G[Latency] --> H[Average HTTP Duration]
    end

    style B fill:#c8e6c9
    style D fill:#ffcdd2
    style F fill:#e3f2fd
    style H fill:#fff3e0

Alerting Rules¶

# Example alerting rules
alerts:
  - name: HighFailureRate
    condition: events_failed / (events_exported + events_failed) > 0.1
    message: "Security event exporter failure rate is above 10%"
    severity: warning

  - name: HighHTTPErrorRate
    condition: http_errors / http_requests > 0.05
    message: "HTTP error rate is above 5%"
    severity: critical

  - name: SlowResponseTime
    condition: average_duration > "5s"
    message: "Average HTTP response time is above 5 seconds"
    severity: warning

  - name: LowThroughput
    condition: throughput < 10
    message: "Event throughput is below 10 events per second"
    severity: warning

Dashboard Metrics¶

graph TB
    subgraph "Dashboard Panels"
        A[Success Rate Gauge] --> B[95% Target]
        C[Error Rate Gauge] --> D[<5% Target]
        E[Throughput Graph] --> F[Events/sec over time]
        G[Latency Graph] --> H[Response time over time]
    end

    style B fill:#c8e6c9
    style D fill:#ffcdd2
    style F fill:#e3f2fd
    style H fill:#fff3e0

Performance Optimization¶

Metrics-Driven Optimization¶

graph TD
    A[Monitor Metrics] --> B[Identify Bottlenecks]
    B --> C[Analyze Patterns]
    C --> D[Optimize Configuration]
    D --> E[Test Changes]
    E --> F[Deploy Updates]
    F --> A

    style B fill:#fff3e0
    style D fill:#e3f2fd
    style F fill:#c8e6c9

Optimization Strategies¶

1. Batch Size Optimization
2. Monitor batch_size metrics
3. Adjust collector batch processor settings

4. Balance throughput vs. latency

5. HTTP Performance Optimization

6. Monitor http_request_duration
7. Optimize endpoint configuration

8. Adjust timeout settings

9. Error Rate Optimization

10. Monitor conversion_errors and http_errors
11. Identify problematic log patterns
12. Adjust error handling logic

Integration with Monitoring Systems¶

Prometheus Integration¶

graph LR
    A[Structured Logs] --> B[Log Parser]
    B --> C[Metrics Extractor]
    C --> D[Prometheus Metrics]
    D --> E[Grafana Dashboard]

    style A fill:#e3f2fd
    style D fill:#fff3e0
    style E fill:#f3e5f5

Log Parsing for Metrics¶

# Extract metrics from logs
grep "Final telemetry metrics" collector.logs | \
jq -r '.logs_received, .events_exported, .events_failed' | \
awk '{print "security_event_exporter_logs_received_total " $1; print "security_event_exporter_events_exported_total " $2; print "security_event_exporter_events_failed_total " $3}'

Grafana Dashboard¶

{
  "dashboard": {
    "title": "Security Event Exporter Metrics",
    "panels": [
      {
        "title": "Success Rate",
        "type": "stat",
        "targets": [
          {
            "expr": "security_event_exporter_events_exported_total / (security_event_exporter_events_exported_total + security_event_exporter_events_failed_total) * 100"
          }
        ]
      },
      {
        "title": "Throughput",
        "type": "graph",
        "targets": [
          {
            "expr": "rate(security_event_exporter_events_exported_total[5m])"
          }
        ]
      }
    ]
  }
}

Troubleshooting¶

Common Issues and Metrics¶

graph TD
    A[High Conversion Errors] --> B[Check Log Format]
    C[High HTTP Errors] --> D[Verify Endpoint Config]
    E[Slow Response Times] --> F[Check Network Performance]
    G[Low Throughput] --> H[Optimize Batch Settings]

    style B fill:#fff3e0
    style D fill:#fff3e0
    style F fill:#fff3e0
    style H fill:#fff3e0

Debug Mode¶

Enable debug logging to get detailed metrics information:

service:
  telemetry:
    logs:
      level: debug

This provides detailed metrics at each processing step, helping identify bottlenecks and issues.

Performance Analysis¶

graph LR
    A[Collect Metrics] --> B[Analyze Trends]
    B --> C[Identify Issues]
    C --> D[Apply Fixes]
    D --> E[Monitor Results]
    E --> A

    style C fill:#fff3e0
    style D fill:#e3f2fd
    style E fill:#c8e6c9

Future Enhancements¶

Planned enhancements for metrics collection:

graph TB
    subgraph "Future Features"
        A[Native Prometheus Metrics] --> B[Direct Integration]
        C[Real-time Metrics] --> D[Live Dashboard]
        E[Custom Metrics] --> F[User-defined Metrics]
        G[Metrics Export] --> H[External Processors]
    end

    style B fill:#e3f2fd
    style D fill:#fff3e0
    style F fill:#f3e5f5
    style H fill:#c8e6c9

1. Native Prometheus Metrics: Direct integration with Prometheus metrics
2. Real-time Metrics: Live metrics dashboard
3. Custom Metrics: User-defined metrics for specific use cases
4. Metrics Export: Export metrics to external processors