Configure intelligent alerting and notification systems to ensure your team is notified immediately when issues occur.

Alert Setup

Intelligent Alert Configuration: Configure intelligent alerting and notification systems to ensure your team is notified immediately when issues occur.

FortiBlox Monitoring provides comprehensive alerting capabilities to ensure your team is notified immediately when infrastructure issues occur. This guide covers setting up notification channels, configuring alert rules, and creating escalation policies.

Notification Channels Overview

📧 Email

Email alerts and reports with customizable templates

💬 Slack

Real-time alerts directly to Slack channels

🔗 Webhooks

Custom integrations with any HTTP endpoint

📱 SMS

Critical alerts via text message

Quick Start: Setting Up Your First Alert

Step 1: Create Notification Channel

Navigate to Settings → Notification Channels
Click "+ Add Channel" and choose your preferred type
Configure channel settings (email, Slack webhook, etc.)
Test the channel to ensure it works correctly

Step 2: Configure Alert Rules

Go to your monitor configuration
Add the notification channel to the monitor
Set alert conditions (down, degraded, recovery)
Configure alert thresholds and timing

Step 3: Test Alerts

Use the "Send Test Alert" feature
Verify alerts are received correctly
Check alert formatting and content
Adjust settings as needed

Email Notifications

Email Channel Setup

Basic Email Configuration

{
  "type": "email",
  "name": "Operations Team",
  "recipients": [
    "[email protected]",
    "[email protected]"
  ],
  "settings": {
    "sendOnDown": true,
    "sendOnUp": true,
    "sendOnDegraded": true,
    "includeGraphs": true,
    "format": "html"
  }
}

Advanced Email Settings

{
  "emailSettings": {
    "subject": "[{{status}}] {{monitor.name}} Alert",
    "template": "detailed",
    "includeIncidentHistory": true,
    "includeSystemMetrics": true,
    "attachLogs": false,
    "frequency": {
      "maxPerHour": 10,
      "suppressDuplicates": true
    }
  }
}

Email Templates

Custom Email Templates FortiBlox supports custom HTML email templates:

<!-- Custom alert template -->
<div style="font-family: Arial, sans-serif; max-width: 600px;">
  <div style="background: {{status_color}}; color: white; padding: 20px;">
    <h2>{{alert_type}} Alert: {{monitor.name}}</h2>
  </div>
  
  <div style="padding: 20px;">
    <p><strong>Status:</strong> {{status.current}}</p>
    <p><strong>Service:</strong> {{monitor.url}}</p>
    <p><strong>Error:</strong> {{check.error}}</p>
    <p><strong>Duration:</strong> {{incident.duration}}</p>
    
    {{#if check.response_time}}
    <p><strong>Response Time:</strong> {{check.response_time}}ms</p>
    {{/if}}
    
    <div style="margin: 20px 0;">
      <a href="{{dashboard_url}}" style="background: #007cba; color: white; padding: 10px 20px; text-decoration: none; border-radius: 5px;">
        View Dashboard
      </a>
    </div>
  </div>
</div>

Slack Integration

Slack Channel Setup

Slack Webhook Configuration

Create a Slack app in your workspace
Add Incoming Webhooks feature
Generate webhook URL for your channel
Configure webhook in FortiBlox

{
  "type": "slack",
  "name": "Engineering Alerts",
  "webhookUrl": "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX",
  "settings": {
    "channel": "#alerts",
    "username": "FortiBlox Monitor",
    "iconEmoji": ":warning:",
    "notifyChannel": true,
    "threadReplies": true
  }
}

Slack Alert Formatting

Rich Slack Messages

{
  "slackFormat": {
    "color": "danger", // good, warning, danger
    "fields": [
      {
        "title": "Service",
        "value": "{{monitor.name}}",
        "short": true
      },
      {
        "title": "Status",
        "value": "{{status.current}}",
        "short": true
      },
      {
        "title": "Response Time",
        "value": "{{check.response_time}}ms",
        "short": true
      },
      {
        "title": "Error",
        "value": "{{check.error}}",
        "short": false
      }
    ],
    "actions": [
      {
        "type": "button",
        "text": "View Dashboard",
        "url": "{{dashboard_url}}"
      },
      {
        "type": "button",
        "text": "Acknowledge",
        "url": "{{acknowledge_url}}"
      }
    ]
  }
}

Slack Alert Examples

🔴 ALERT: Production API is DOWN
📍 Service: https://api.company.com
⏱️ Duration: 3 minutes
❌ Error: Connection timeout after 30000ms
📊 View Dashboard | ✅ Acknowledge

Webhook Notifications

Webhook Configuration

Basic Webhook Setup

{
  "type": "webhook",
  "name": "PagerDuty Integration",
  "url": "https://events.pagerduty.com/v2/enqueue",
  "method": "POST",
  "headers": {
    "Authorization": "Token token=your-pd-api-key",
    "Content-Type": "application/json"
  },
  "timeout": 10000,
  "retries": 3
}

Custom Webhook Payload

{
  "payload": {
    "routing_key": "your-integration-key",
    "event_action": "{{#if status.is_down}}trigger{{else}}resolve{{/if}}",
    "dedup_key": "{{monitor.id}}-{{incident.id}}",
    "payload": {
      "summary": "{{monitor.name}} is {{status.current}}",
      "severity": "{{#if status.is_down}}error{{else}}info{{/if}}",
      "source": "FortiBlox Monitoring",
      "component": "{{monitor.name}}",
      "group": "{{monitor.team}}",
      "class": "{{monitor.type}}",
      "custom_details": {
        "monitor_url": "{{monitor.url}}",
        "response_time": "{{check.response_time}}",
        "status_code": "{{check.status_code}}",
        "error_message": "{{check.error}}",
        "incident_started": "{{incident.started_at}}",
        "dashboard_link": "{{dashboard_url}}"
      }
    }
  }
}

Popular Webhook Integrations

PagerDuty

{
  "name": "PagerDuty Critical Alerts",
  "url": "https://events.pagerduty.com/v2/enqueue",
  "payload": {
    "routing_key": "{{pagerduty_key}}",
    "event_action": "trigger",
    "payload": {
      "summary": "[{{status.current}}] {{monitor.name}}",
      "severity": "critical",
      "source": "{{monitor.url}}"
    }
  }
}

Microsoft Teams

{
  "name": "Teams Operations Channel",
  "url": "https://outlook.office.com/webhook/...",
  "payload": {
    "@type": "MessageCard",
    "@context": "https://schema.org/extensions",
    "summary": "{{monitor.name}} Alert",
    "themeColor": "{{#if status.is_down}}FF0000{{else}}00FF00{{/if}}",
    "sections": [
      {
        "activityTitle": "{{monitor.name}} - {{status.current}}",
        "activitySubtitle": "{{check.error}}",
        "facts": [
          {
            "name": "Service",
            "value": "{{monitor.url}}"
          },
          {
            "name": "Duration",
            "value": "{{incident.duration}}"
          }
        ]
      }
    ]
  }
}

SMS and Voice Alerts

SMS Configuration

SMS Alert Setup

{
  "type": "sms",
  "name": "Critical SMS Alerts",
  "provider": "twilio",
  "recipients": ["+1234567890", "+1987654321"],
  "settings": {
    "onlyForCritical": true,
    "maxPerHour": 5,
    "quietHours": {
      "enabled": true,
      "start": "22:00",
      "end": "08:00",
      "timezone": "America/New_York"
    }
  }
}

SMS Message Template

[CRITICAL] {{monitor.name}} DOWN
Service: {{monitor.url}}
Duration: {{incident.duration}}
Error: {{check.error}}
Dashboard: {{short_url}}

Voice Call Alerts

Voice Alert Configuration

{
  "type": "voice",
  "name": "Emergency Voice Calls",
  "provider": "twilio",
  "recipients": [
    {
      "phone": "+1234567890",
      "name": "On-Call Engineer",
      "priority": 1
    }
  ],
  "message": "Critical alert for {{monitor.name}}. Service is down for {{incident.duration}}. Please check the dashboard immediately.",
  "settings": {
    "maxRetries": 3,
    "retryDelay": 300000, // 5 minutes
    "onlyForCritical": true
  }
}

Alert Rules and Conditions

Basic Alert Rules

Monitor Status Alerts

{
  "alertRules": [
    {
      "condition": "status.is_down",
      "channels": ["email-ops", "slack-alerts"],
      "priority": "high",
      "suppressTime": 300000 // 5 minutes
    },
    {
      "condition": "status.is_degraded",
      "channels": ["email-ops"],
      "priority": "medium",
      "suppressTime": 900000 // 15 minutes
    },
    {
      "condition": "status.recovered",
      "channels": ["email-ops", "slack-alerts"],
      "priority": "info"
    }
  ]
}

Advanced Alert Conditions

Performance-Based Alerts

{
  "performanceAlerts": [
    {
      "condition": "response_time > 5000", // 5 seconds
      "duration": "5m",  // Sustained for 5 minutes
      "channels": ["slack-performance"],
      "message": "{{monitor.name}} response time is {{check.response_time}}ms"
    },
    {
      "condition": "uptime_7d < 99.5",
      "channels": ["email-management"],
      "frequency": "daily"
    }
  ]
}

Error Rate Alerts

{
  "errorRateAlerts": [
    {
      "condition": "error_rate_5m > 10", // 10% error rate in 5 minutes
      "channels": ["pagerduty-critical"],
      "escalation": true
    }
  ]
}

Escalation Policies

Basic Escalation Setup

Multi-Level Escalation

{
  "escalationPolicy": {
    "name": "Critical Service Escalation",
    "levels": [
      {
        "delay": 0,
        "channels": ["slack-oncall", "email-primary"],
        "requiresAcknowledgment": false
      },
      {
        "delay": 300000, // 5 minutes
        "channels": ["sms-oncall", "pagerduty-escalate"],
        "requiresAcknowledgment": true
      },
      {
        "delay": 900000, // 15 minutes
        "channels": ["voice-manager", "email-leadership"],
        "requiresAcknowledgment": true
      }
    ]
  }
}

Time-Based Escalation

Business Hours Escalation

{
  "timeBasedEscalation": {
    "businessHours": {
      "schedule": "Mon-Fri 09:00-17:00",
      "timezone": "America/New_York",
      "escalation": "standard-escalation"
    },
    "afterHours": {
      "schedule": "Mon-Fri 17:00-09:00, Sat-Sun",
      "escalation": "emergency-escalation"
    },
    "holidays": {
      "escalation": "holiday-escalation"
    }
  }
}

Alert Suppression and Management

Maintenance Windows

Scheduled Maintenance

{
  "maintenanceWindows": [
    {
      "name": "Weekly Deployment",
      "schedule": "0 2 * * SUN", // Sunday 2 AM
      "duration": 3600000, // 1 hour
      "timezone": "UTC",
      "affectedMonitors": ["production-*"],
      "suppressAlerts": true,
      "notifyStart": true,
      "notifyEnd": true
    }
  ]
}

Alert Suppression Rules

Intelligent Suppression

{
  "suppressionRules": [
    {
      "name": "Cascade Suppression",
      "condition": "parent_service_down",
      "suppressDuration": 1800000, // 30 minutes
      "reason": "Dependent service failure"
    },
    {
      "name": "Flapping Detection",
      "condition": "status_changes > 5 in 10m",
      "suppressDuration": 600000, // 10 minutes
      "reason": "Service flapping detected"
    }
  ]
}

Alert Testing and Validation

Testing Notification Channels

Channel Testing

# Test email notifications
curl -X POST https://api.fortiblox.com/v1/notifications/test \
  -H "Authorization: Bearer your-api-token" \
  -H "Content-Type: application/json" \
  -d '{
    "channel_id": "email-ops",
    "test_type": "down_alert",
    "monitor_name": "Test Monitor"
  }'

End-to-End Testing

Create a test monitor with short intervals
Temporarily break the service being monitored
Verify alerts are received through all channels
Test escalation by not acknowledging initial alerts
Verify recovery notifications when service is restored

Best Practices for Alerting

Alert Design Principles

Actionable Alerts: Every alert should require or enable a specific action
Clear Context: Include enough information to understand and respond
Appropriate Urgency: Match alert severity to business impact
Avoid Alert Fatigue: Don't over-alert or send redundant notifications
Test Regularly: Verify alert channels work as expected

Alert Content Best Practices

Essential Information in Alerts

Service name and URL
Current status and error message
Duration of the incident
Impact assessment
Direct links to dashboards and logs
Clear next steps for resolution

Alert Message Templates

Subject: [CRITICAL] {{monitor.name}} - {{status.current}}

🔴 CRITICAL ALERT

Service: {{monitor.name}}
URL: {{monitor.url}}
Status: {{status.current}}
Duration: {{incident.duration}}
Error: {{check.error}}

Last successful check: {{last_success}}
Response time: {{check.response_time}}ms
Status code: {{check.status_code}}

🔗 Dashboard: {{dashboard_url}}
📊 Logs: {{logs_url}}
📞 Escalate: {{escalation_url}}

--- 
This alert was sent by FortiBlox Monitoring

Alert Setup Tip: Start with basic email alerts, then gradually add more sophisticated channels like Slack and webhooks. Always test your alert channels before relying on them for critical monitoring.

🔔 Alert Setup

Alert Setup

Notification Channels Overview

📧 Email

💬 Slack

🔗 Webhooks

📱 SMS

Quick Start: Setting Up Your First Alert

Email Notifications

Email Channel Setup

Email Templates

Slack Integration

Slack Channel Setup

Slack Alert Formatting

Webhook Notifications

Webhook Configuration

Popular Webhook Integrations

SMS and Voice Alerts

SMS Configuration

Voice Call Alerts

Alert Rules and Conditions

Basic Alert Rules

Advanced Alert Conditions

Escalation Policies

Basic Escalation Setup

Time-Based Escalation

Alert Suppression and Management

Maintenance Windows

Alert Suppression Rules

Alert Testing and Validation

Testing Notification Channels

Best Practices for Alerting

Alert Design Principles

Alert Content Best Practices

Next Steps

Notification Channels

Escalation Policies

Monitor Setup

On this page