| 1 |
# PagerDuty |
| 2 |
|
| 3 |
## Overview |
| 4 |
|
| 5 |
PagerDuty is a digital operations management platform that helps teams respond to incidents and outages. The PagerDuty service integrates with PagerDuty's Events API v2 to create, acknowledge, and resolve incidents programmatically. This service enables automated incident management workflows, allowing you to trigger alerts when issues are detected, acknowledge them when investigation begins, and resolve them when problems are fixed. |
| 6 |
|
| 7 |
The service supports rich incident context including custom details, links to related resources, images, and client information to provide comprehensive incident information to on-call responders. |
| 8 |
|
| 9 |
## URL Format |
| 10 |
|
| 11 |
--8<-- "docs/services/incident/pagerduty/config.md" |
| 12 |
|
| 13 |
### URL Components |
| 14 |
|
| 15 |
The PagerDuty service URL follows this structure: |
| 16 |
|
| 17 |
```uri |
| 18 |
pagerduty://[host[:port]]/integration-key[?query-parameters] |
| 19 |
``` |
| 20 |
|
| 21 |
**Components:** |
| 22 |
|
| 23 |
- **Scheme**: `pagerduty://` - Identifies the service type |
| 24 |
- **Host** (optional): PagerDuty API hostname (default: `events.pagerduty.com`) |
| 25 |
- **Port** (optional): API port number (default: `443` for HTTPS) |
| 26 |
- **Integration Key**: 32-character hexadecimal string that routes events to your PagerDuty service |
| 27 |
- **Query Parameters** (optional): Additional configuration parameters |
| 28 |
|
| 29 |
### Examples |
| 30 |
|
| 31 |
!!! example "Default configuration" |
| 32 |
```url |
| 33 |
pagerduty:///eb243592faa24ba2a5511afdf565c889 |
| 34 |
``` |
| 35 |
Uses default host (`events.pagerduty.com`) and port (`443`) |
| 36 |
|
| 37 |
!!! example "Custom host" |
| 38 |
```url |
| 39 |
pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889 |
| 40 |
``` |
| 41 |
Explicitly specifies the PagerDuty Events API host |
| 42 |
|
| 43 |
!!! example "With parameters" |
| 44 |
```url |
| 45 |
pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=critical&source=production-db |
| 46 |
``` |
| 47 |
Includes severity and source parameters |
| 48 |
|
| 49 |
## Setup and Service Integration |
| 50 |
|
| 51 |
### Prerequisites |
| 52 |
|
| 53 |
Before using the PagerDuty service, you need: |
| 54 |
|
| 55 |
1. A PagerDuty account |
| 56 |
2. A service configured in PagerDuty |
| 57 |
3. An Events API v2 integration key |
| 58 |
|
| 59 |
### Creating a Service Integration |
| 60 |
|
| 61 |
Follow these steps to set up PagerDuty integration: |
| 62 |
|
| 63 |
1. **Log in to PagerDuty** |
| 64 |
- Access your PagerDuty account at [pagerduty.com](https://pagerduty.com) |
| 65 |
|
| 66 |
2. **Navigate to Services** |
| 67 |
- Go to **Services** → **Service Directory** in the main navigation |
| 68 |
|
| 69 |
3. **Create or Select a Service** |
| 70 |
- Either create a new service or select an existing one |
| 71 |
- If creating new: Click **+ New Service**, give it a name, and select an escalation policy |
| 72 |
|
| 73 |
4. **Add Events API v2 Integration** |
| 74 |
- In the service details, click **Integrations** tab |
| 75 |
- Click **+ Add another integration** |
| 76 |
- Search for and select **Events API v2** |
| 77 |
- Give the integration a name (e.g., "Shoutrrr Integration") |
| 78 |
- Click **Create Integration** |
| 79 |
|
| 80 |
5. **Copy Integration Key** |
| 81 |
- The integration key will be displayed |
| 82 |
- Copy this 32-character key for use in your Shoutrrr URL |
| 83 |
|
| 84 |
!!! tip "Integration Key Security" |
| 85 |
Keep your integration key secure. It acts as an authentication token for sending events to PagerDuty. |
| 86 |
|
| 87 |
!!! note "Only Integration Keys Supported" |
| 88 |
This service only accepts PagerDuty service integration keys (32 hexadecimal characters matching `/^[a-fA-F0-9]{32}$/`). API keys are not supported and will result in an invalid integration key error. |
| 89 |
|
| 90 |
## Parameters |
| 91 |
|
| 92 |
The PagerDuty service supports the following parameters to customize incident creation and management: |
| 93 |
|
| 94 |
### Core Parameters |
| 95 |
|
| 96 |
| Parameter | Type | Description | Required | Default | Valid Values | |
| 97 |
|-----------|------|-------------|----------|---------|--------------| |
| 98 |
| `severity` | string | The perceived severity of the incident | No | `error` | `info`, `warning`, `error`, `critical` | |
| 99 |
| `source` | string | The unique location of the affected system | No | `default` | Any string identifier | |
| 100 |
| `action` | string | The type of event to send | No | `trigger` | `trigger`, `acknowledge`, `resolve` | |
| 101 |
|
| 102 |
### Additional Parameters |
| 103 |
|
| 104 |
| Parameter | Type | Description | Required | Default | Valid Values | |
| 105 |
|-----------|------|-------------|----------|---------|--------------| |
| 106 |
| `details` | object | Additional custom key-value pairs for the incident | No | - | JSON object or URL-encoded string | |
| 107 |
| `contexts` | array | Additional context such as links or images | No | - | JSON array of context objects (only 'link' and 'image' types supported) | |
| 108 |
| `client` | string | Name of the monitoring client/integration | No | - | Any string | |
| 109 |
| `client_url` | string | URL of the monitoring client dashboard | No | - | Valid URL | |
| 110 |
|
| 111 |
### Parameter Details |
| 112 |
|
| 113 |
#### Severity Levels |
| 114 |
|
| 115 |
- **`info`**: Informational events that don't require immediate action |
| 116 |
- **`warning`**: Potential issues that should be monitored |
| 117 |
- **`error`**: Actual problems affecting system operation |
| 118 |
- **`critical`**: Urgent issues requiring immediate response |
| 119 |
|
| 120 |
#### Event Action Items |
| 121 |
|
| 122 |
- **`trigger`**: Creates a new incident or updates an existing one |
| 123 |
- **`acknowledge`**: Acknowledges an incident (stops escalation) |
| 124 |
- **`resolve`**: Resolves an incident (closes it) |
| 125 |
|
| 126 |
#### Details Parameter |
| 127 |
|
| 128 |
The `details` parameter accepts a JSON object with custom key-value pairs: |
| 129 |
|
| 130 |
```json |
| 131 |
{ |
| 132 |
"component": "database", |
| 133 |
"version": "2.1.0", |
| 134 |
"error_code": "ECONNREFUSED", |
| 135 |
"affected_users": 150 |
| 136 |
} |
| 137 |
``` |
| 138 |
|
| 139 |
#### Contexts Parameter |
| 140 |
|
| 141 |
The `contexts` parameter accepts a JSON array of context objects. Each context must have a `type` field. Only `link` and `image` types are supported. |
| 142 |
|
| 143 |
**Link contexts:** |
| 144 |
|
| 145 |
```json |
| 146 |
{ |
| 147 |
"type": "link", |
| 148 |
"href": "https://example.com/logs", |
| 149 |
"text": "View Application Logs" |
| 150 |
} |
| 151 |
``` |
| 152 |
|
| 153 |
**Image contexts:** |
| 154 |
|
| 155 |
```json |
| 156 |
{ |
| 157 |
"type": "image", |
| 158 |
"src": "https://example.com/error-screenshot.png", |
| 159 |
"href": "https://example.com/dashboard", |
| 160 |
"text": "Error Screenshot" |
| 161 |
} |
| 162 |
``` |
| 163 |
|
| 164 |
## Usage Examples |
| 165 |
|
| 166 |
### Basic Incident Triggering |
| 167 |
|
| 168 |
!!! example "Simple alert" |
| 169 |
```url |
| 170 |
pagerduty:///eb243592faa24ba2a5511afdf565c889 |
| 171 |
``` |
| 172 |
```bash |
| 173 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889' -m 'Database connection lost' |
| 174 |
``` |
| 175 |
|
| 176 |
!!! example "Critical incident with source" |
| 177 |
```url |
| 178 |
pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=critical&source=prod-db-01 |
| 179 |
``` |
| 180 |
```bash |
| 181 |
shoutrrr send -u 'pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=critical&source=prod-db-01' -m 'Production database is unresponsive' |
| 182 |
``` |
| 183 |
|
| 184 |
### Incident Management Workflow |
| 185 |
|
| 186 |
!!! example "Acknowledge incident" |
| 187 |
```url |
| 188 |
pagerduty:///eb243592faa24ba2a5511afdf565c889?action=acknowledge |
| 189 |
``` |
| 190 |
```bash |
| 191 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=acknowledge' -m 'Engineer has acknowledged the incident and is investigating' |
| 192 |
``` |
| 193 |
|
| 194 |
!!! example "Resolve incident" |
| 195 |
```url |
| 196 |
pagerduty:///eb243592faa24ba2a5511afdf565c889?action=resolve |
| 197 |
``` |
| 198 |
```bash |
| 199 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=resolve' -m 'Database connection restored, incident resolved' |
| 200 |
``` |
| 201 |
|
| 202 |
### Advanced Examples |
| 203 |
|
| 204 |
!!! example "With custom details" |
| 205 |
```url |
| 206 |
pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=warning&source=monitoring&details={"component":"api","version":"1.2.3","response_time":"5.2s"} |
| 207 |
``` |
| 208 |
```bash |
| 209 |
shoutrrr send -u 'pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=warning&source=monitoring&details={"component":"api","version":"1.2.3","response_time":"5.2s"}' -m 'API response time degraded' |
| 210 |
``` |
| 211 |
|
| 212 |
!!! example "With contexts and client info" |
| 213 |
```url |
| 214 |
pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=error&source=ci-pipeline&contexts=[{"type":"link","href":"https://github.com/org/repo/actions/runs/123","text":"GitHub Actions Run"},{"type":"image","src":"https://example.com/failure-screenshot.png","text":"Build Failure"}]&client=jenkins&client_url=https://ci.example.com/job/123 |
| 215 |
``` |
| 216 |
```bash |
| 217 |
shoutrrr send -u 'pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=error&source=ci-pipeline&contexts=[{"type":"link","href":"https://github.com/org/repo/actions/runs/123","text":"GitHub Actions Run"},{"type":"image","src":"https://example.com/failure-screenshot.png","text":"Build Failure"}]&client=jenkins&client_url=https://ci.example.com/job/123' -m 'CI/CD pipeline failed' |
| 218 |
``` |
| 219 |
|
| 220 |
## Event Actions |
| 221 |
|
| 222 |
The PagerDuty service supports three primary event actions that correspond to the incident lifecycle: |
| 223 |
|
| 224 |
### Trigger (`action=trigger`) |
| 225 |
|
| 226 |
Creates a new incident or updates an existing one with new information. This is the default action. |
| 227 |
|
| 228 |
**Use cases:** |
| 229 |
|
| 230 |
- System failures or outages |
| 231 |
- Performance degradation |
| 232 |
- Security alerts |
| 233 |
- Application errors |
| 234 |
|
| 235 |
**Behavior:** |
| 236 |
|
| 237 |
- If no incident exists for the service, creates a new one |
| 238 |
- If an incident exists, adds a new trigger event (may escalate if unacknowledged) |
| 239 |
- Includes all provided context, details, and parameters |
| 240 |
|
| 241 |
### Acknowledge (`action=acknowledge`) |
| 242 |
|
| 243 |
Acknowledges an existing incident, stopping escalation and indicating that someone is actively working on it. |
| 244 |
|
| 245 |
**Use cases:** |
| 246 |
|
| 247 |
- Engineer has started investigation |
| 248 |
- Issue has been triaged |
| 249 |
- Response team is assembled |
| 250 |
|
| 251 |
**Behavior:** |
| 252 |
|
| 253 |
- Stops incident escalation |
| 254 |
- Records acknowledgment timestamp |
| 255 |
- Can include update notes in the message |
| 256 |
|
| 257 |
### Resolve (`action=resolve`) |
| 258 |
|
| 259 |
Resolves an incident, marking it as completed and closing the alert cycle. |
| 260 |
|
| 261 |
**Use cases:** |
| 262 |
|
| 263 |
- Issue has been fixed |
| 264 |
- Service has been restored |
| 265 |
- Incident is no longer relevant |
| 266 |
|
| 267 |
**Behavior:** |
| 268 |
|
| 269 |
- Closes the incident |
| 270 |
- Records resolution timestamp |
| 271 |
- Prevents further notifications for this incident |
| 272 |
|
| 273 |
!!! note "Incident Deduplication" |
| 274 |
PagerDuty automatically deduplicates incidents based on the integration key and incident details. Multiple triggers for the same issue will update the existing incident rather than create duplicates. |
| 275 |
|
| 276 |
## Advanced Features |
| 277 |
|
| 278 |
### Custom Details |
| 279 |
|
| 280 |
The `details` parameter allows you to attach structured data to incidents: |
| 281 |
|
| 282 |
```json |
| 283 |
{ |
| 284 |
"environment": "production", |
| 285 |
"component": "web-server", |
| 286 |
"version": "2.1.0", |
| 287 |
"error_code": "500", |
| 288 |
"affected_services": ["api", "database"], |
| 289 |
"metrics": { |
| 290 |
"cpu_usage": 95.2, |
| 291 |
"memory_usage": 87.1 |
| 292 |
} |
| 293 |
} |
| 294 |
``` |
| 295 |
|
| 296 |
These details appear in the PagerDuty incident details and can be used for filtering, reporting, and automation. |
| 297 |
|
| 298 |
### Contexts |
| 299 |
|
| 300 |
Contexts provide additional visual and navigational context to responders: |
| 301 |
|
| 302 |
**Link Contexts:** |
| 303 |
|
| 304 |
- Direct responders to relevant resources |
| 305 |
- Include runbooks, documentation, or monitoring dashboards |
| 306 |
- Support both internal and external links |
| 307 |
|
| 308 |
**Image Contexts:** |
| 309 |
|
| 310 |
- Attach screenshots or graphs |
| 311 |
- Show error visualizations or system diagrams |
| 312 |
- Include clickable links to full-size images |
| 313 |
|
| 314 |
### Client Information |
| 315 |
|
| 316 |
The `client` and `client_url` parameters identify the monitoring system: |
| 317 |
|
| 318 |
- **`client`**: Name of your monitoring tool (e.g., "Prometheus", "DataDog", "Custom Monitor") |
| 319 |
- **`client_url`**: Direct link to the monitoring dashboard or alert details |
| 320 |
|
| 321 |
This helps responders understand where the alert originated and provides quick access to additional monitoring data. |
| 322 |
|
| 323 |
### Incident Grouping |
| 324 |
|
| 325 |
PagerDuty supports incident grouping based on custom details. Use consistent detail keys to group related incidents: |
| 326 |
|
| 327 |
```json |
| 328 |
{ |
| 329 |
"service": "payment-api", |
| 330 |
"region": "us-west-2", |
| 331 |
"cluster": "production" |
| 332 |
} |
| 333 |
``` |
| 334 |
|
| 335 |
## Rate Limiting |
| 336 |
|
| 337 |
The PagerDuty Events API v2 enforces rate limits to ensure system stability: |
| 338 |
|
| 339 |
- **120 events per minute** per integration key (higher ceilings available on AIOps/enterprise tiers) |
| 340 |
- Rate limits are enforced per integration key, not per service |
| 341 |
- Exceeding limits returns HTTP 429 (Too Many Requests) |
| 342 |
|
| 343 |
### Handling Rate Limits |
| 344 |
|
| 345 |
When rate limits are exceeded: |
| 346 |
|
| 347 |
1. **Implement backoff**: Use exponential backoff (start with 1 minute, double each retry) |
| 348 |
2. **Queue messages**: Buffer notifications during high-volume periods |
| 349 |
3. **Use multiple integrations**: Distribute load across multiple integration keys if needed |
| 350 |
4. **Monitor usage**: Track your API usage in PagerDuty's integration settings |
| 351 |
|
| 352 |
!!! tip "Rate Limit Best Practices" |
| 353 |
- Implement retry logic with jitter to avoid thundering herd problems |
| 354 |
- Consider using PagerDuty's enterprise features for higher limits if needed |
| 355 |
- Monitor your integration's usage in PagerDuty's web interface |
| 356 |
|
| 357 |
## Error Handling |
| 358 |
|
| 359 |
The PagerDuty service returns standard HTTP status codes. Handle these appropriately in your applications: |
| 360 |
|
| 361 |
### Success Responses |
| 362 |
|
| 363 |
- **200 OK**: Event processed successfully |
| 364 |
- **202 Accepted**: Event accepted for processing (asynchronous) |
| 365 |
|
| 366 |
### Error Responses |
| 367 |
|
| 368 |
- **400 Bad Request**: Invalid request payload or parameters |
| 369 |
- Check integration key format (must be 32 hex characters) |
| 370 |
- Validate JSON structure for details/contexts |
| 371 |
- Ensure URLs are properly encoded |
| 372 |
|
| 373 |
- **401 Unauthorized**: Invalid or missing integration key |
| 374 |
- Verify the integration key is correct |
| 375 |
- Check that the integration hasn't been disabled |
| 376 |
|
| 377 |
- **403 Forbidden**: Integration lacks required permissions |
| 378 |
- Ensure the integration key has appropriate permissions |
| 379 |
- Check service configuration in PagerDuty |
| 380 |
|
| 381 |
- **429 Too Many Requests**: Rate limit exceeded |
| 382 |
- Implement exponential backoff |
| 383 |
- Consider queuing messages |
| 384 |
|
| 385 |
- **500 Internal Server Error**: PagerDuty service error |
| 386 |
- Retry with backoff |
| 387 |
- Check PagerDuty status page for outages |
| 388 |
|
| 389 |
### Common Issues and Solutions |
| 390 |
|
| 391 |
!!! tip "Integration Key Validation" |
| 392 |
Integration keys must be exactly 32 hexadecimal characters. Common mistakes include: |
| 393 |
- Using API keys instead of integration keys |
| 394 |
- Extra whitespace or characters |
| 395 |
- Using keys from disabled integrations |
| 396 |
|
| 397 |
!!! tip "JSON Parameter Encoding" |
| 398 |
When passing complex parameters like `details` or `contexts` in URLs, ensure proper JSON encoding: |
| 399 |
|
| 400 |
**Correct:** |
| 401 |
|
| 402 |
```bash |
| 403 |
details={"key":"value"} |
| 404 |
``` |
| 405 |
|
| 406 |
**Incorrect - missing quotes:** |
| 407 |
|
| 408 |
```bash |
| 409 |
details={key:value} |
| 410 |
``` |
| 411 |
|
| 412 |
!!! tip "Context Object Validation" |
| 413 |
Each context object must have a valid `type` field. Supported types are `link` and `image`. |
| 414 |
|
| 415 |
## Code Examples |
| 416 |
|
| 417 |
### Go Integration |
| 418 |
|
| 419 |
#### Basic Incident Triggering Implementation |
| 420 |
|
| 421 |
```go |
| 422 |
package main |
| 423 |
|
| 424 |
import ( |
| 425 |
"log" |
| 426 |
"github.com/containrrr/shoutrrr" |
| 427 |
) |
| 428 |
|
| 429 |
func main() { |
| 430 |
url := "pagerduty:///eb243592faa24ba2a5511afdf565c889" |
| 431 |
|
| 432 |
err := shoutrrr.Send(url, "Database connection failed") |
| 433 |
if err != nil { |
| 434 |
log.Fatal(err) |
| 435 |
} |
| 436 |
} |
| 437 |
``` |
| 438 |
|
| 439 |
#### Advanced Configuration with Parameters |
| 440 |
|
| 441 |
```go |
| 442 |
package main |
| 443 |
|
| 444 |
import ( |
| 445 |
"log" |
| 446 |
"github.com/containrrr/shoutrrr" |
| 447 |
"github.com/containrrr/shoutrrr/pkg/types" |
| 448 |
) |
| 449 |
|
| 450 |
func main() { |
| 451 |
url := "pagerduty:///eb243592faa24ba2a5511afdf565c889" |
| 452 |
|
| 453 |
// Trigger critical incident |
| 454 |
params := &types.Params{ |
| 455 |
"severity": "critical", |
| 456 |
"source": "production-db-01", |
| 457 |
"action": "trigger", |
| 458 |
"details": `{"database": "postgres", "error": "connection timeout"}`, |
| 459 |
"client": "monitoring-system", |
| 460 |
"client_url": "https://monitoring.example.com/alerts/123", |
| 461 |
} |
| 462 |
|
| 463 |
err := shoutrrr.Send(url, "Production database is down", params) |
| 464 |
if err != nil { |
| 465 |
log.Fatal(err) |
| 466 |
} |
| 467 |
} |
| 468 |
``` |
| 469 |
|
| 470 |
#### Incident Management Workflow Implementation |
| 471 |
|
| 472 |
```go |
| 473 |
package main |
| 474 |
|
| 475 |
import ( |
| 476 |
"log" |
| 477 |
"time" |
| 478 |
"github.com/containrrr/shoutrrr" |
| 479 |
"github.com/containrrr/shoutrrr/pkg/types" |
| 480 |
) |
| 481 |
|
| 482 |
func handleIncident() { |
| 483 |
baseURL := "pagerduty:///eb243592faa24ba2a5511afdf565c889" |
| 484 |
|
| 485 |
// 1. Trigger incident |
| 486 |
triggerParams := &types.Params{ |
| 487 |
"severity": "critical", |
| 488 |
"source": "web-server-01", |
| 489 |
"action": "trigger", |
| 490 |
} |
| 491 |
|
| 492 |
err := shoutrrr.Send(baseURL, "Web server unresponsive", triggerParams) |
| 493 |
if err != nil { |
| 494 |
log.Printf("Failed to trigger incident: %v", err) |
| 495 |
return |
| 496 |
} |
| 497 |
|
| 498 |
// Simulate investigation time |
| 499 |
time.Sleep(5 * time.Minute) |
| 500 |
|
| 501 |
// 2. Acknowledge incident |
| 502 |
ackParams := &types.Params{ |
| 503 |
"action": "acknowledge", |
| 504 |
} |
| 505 |
|
| 506 |
err = shoutrrr.Send(baseURL, "Engineer investigating server issue", ackParams) |
| 507 |
if err != nil { |
| 508 |
log.Printf("Failed to acknowledge incident: %v", err) |
| 509 |
return |
| 510 |
} |
| 511 |
|
| 512 |
// Simulate resolution time |
| 513 |
time.Sleep(15 * time.Minute) |
| 514 |
|
| 515 |
// 3. Resolve incident |
| 516 |
resolveParams := &types.Params{ |
| 517 |
"action": "resolve", |
| 518 |
} |
| 519 |
|
| 520 |
err = shoutrrr.Send(baseURL, "Web server restarted successfully", resolveParams) |
| 521 |
if err != nil { |
| 522 |
log.Printf("Failed to resolve incident: %v", err) |
| 523 |
return |
| 524 |
} |
| 525 |
} |
| 526 |
``` |
| 527 |
|
| 528 |
#### Using Contexts for Rich Incident Details |
| 529 |
|
| 530 |
```go |
| 531 |
package main |
| 532 |
|
| 533 |
import ( |
| 534 |
"log" |
| 535 |
"github.com/containrrr/shoutrrr" |
| 536 |
"github.com/containrrr/shoutrrr/pkg/types" |
| 537 |
) |
| 538 |
|
| 539 |
func sendRichIncident() { |
| 540 |
url := "pagerduty:///eb243592faa24ba2a5511afdf565c889" |
| 541 |
|
| 542 |
params := &types.Params{ |
| 543 |
"severity": "error", |
| 544 |
"source": "ci-pipeline", |
| 545 |
"action": "trigger", |
| 546 |
"details": `{ |
| 547 |
"pipeline": "deploy-prod", |
| 548 |
"stage": "test", |
| 549 |
"commit": "abc123def", |
| 550 |
"branch": "main" |
| 551 |
}`, |
| 552 |
"contexts": `[ |
| 553 |
{ |
| 554 |
"type": "link", |
| 555 |
"href": "https://github.com/org/repo/commit/abc123def", |
| 556 |
"text": "View Commit" |
| 557 |
}, |
| 558 |
{ |
| 559 |
"type": "link", |
| 560 |
"href": "https://ci.example.com/pipelines/123", |
| 561 |
"text": "Pipeline Details" |
| 562 |
}, |
| 563 |
{ |
| 564 |
"type": "image", |
| 565 |
"src": "https://ci.example.com/screenshots/failure.png", |
| 566 |
"text": "Test Failure Screenshot" |
| 567 |
} |
| 568 |
]`, |
| 569 |
"client": "GitHub Actions", |
| 570 |
"client_url": "https://github.com/org/repo/actions/runs/456", |
| 571 |
} |
| 572 |
|
| 573 |
err := shoutrrr.Send(url, "CI/CD pipeline failed in production deployment", params) |
| 574 |
if err != nil { |
| 575 |
log.Fatal(err) |
| 576 |
} |
| 577 |
} |
| 578 |
``` |
| 579 |
|
| 580 |
### Integration with Monitoring Systems |
| 581 |
|
| 582 |
```go |
| 583 |
package main |
| 584 |
|
| 585 |
import ( |
| 586 |
"encoding/json" |
| 587 |
"log" |
| 588 |
"github.com/containrrr/shoutrrr" |
| 589 |
"github.com/containrrr/shoutrrr/pkg/types" |
| 590 |
) |
| 591 |
|
| 592 |
// AlertData represents alert data from a monitoring system |
| 593 |
type AlertData struct { |
| 594 |
Title string `json:"title"` |
| 595 |
Description string `json:"description"` |
| 596 |
Severity string `json:"severity"` |
| 597 |
Source string `json:"source"` |
| 598 |
Details map[string]interface{} `json:"details"` |
| 599 |
DashboardURL string `json:"dashboard_url"` |
| 600 |
} |
| 601 |
|
| 602 |
func sendMonitoringAlert(data AlertData) error { |
| 603 |
url := "pagerduty:///eb243592faa24ba2a5511afdf565c889" |
| 604 |
|
| 605 |
// Convert details to JSON string |
| 606 |
detailsJSON, err := json.Marshal(data.Details) |
| 607 |
if err != nil { |
| 608 |
return err |
| 609 |
} |
| 610 |
|
| 611 |
params := &types.Params{ |
| 612 |
"severity": data.Severity, |
| 613 |
"source": data.Source, |
| 614 |
"action": "trigger", |
| 615 |
"details": string(detailsJSON), |
| 616 |
"client": "Custom Monitoring", |
| 617 |
"client_url": data.DashboardURL, |
| 618 |
} |
| 619 |
|
| 620 |
return shoutrrr.Send(url, data.Title+": "+data.Description, params) |
| 621 |
} |
| 622 |
``` |
| 623 |
|
| 624 |
## CLI Examples |
| 625 |
|
| 626 |
### Basic Usage |
| 627 |
|
| 628 |
Send a simple alert: |
| 629 |
|
| 630 |
```bash |
| 631 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889' -m 'Server is down' |
| 632 |
``` |
| 633 |
|
| 634 |
Send with severity: |
| 635 |
|
| 636 |
```bash |
| 637 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?severity=critical' -m 'Production outage detected' |
| 638 |
``` |
| 639 |
|
| 640 |
### Incident Lifecycle Management |
| 641 |
|
| 642 |
Trigger incident: |
| 643 |
|
| 644 |
```bash |
| 645 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=trigger&severity=critical&source=web-app' -m 'Application crashed' |
| 646 |
``` |
| 647 |
|
| 648 |
Acknowledge incident: |
| 649 |
|
| 650 |
```bash |
| 651 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=acknowledge' -m 'DevOps team is investigating' |
| 652 |
``` |
| 653 |
|
| 654 |
Resolve incident: |
| 655 |
|
| 656 |
```bash |
| 657 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=resolve' -m 'Application restarted successfully' |
| 658 |
``` |
| 659 |
|
| 660 |
### Advanced Parameters |
| 661 |
|
| 662 |
With custom details: |
| 663 |
|
| 664 |
```bash |
| 665 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?details={"component":"database","error":"connection_failed"}' -m 'Database connectivity issue' |
| 666 |
``` |
| 667 |
|
| 668 |
With contexts: |
| 669 |
|
| 670 |
```bash |
| 671 |
shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?contexts=[{"type":"link","href":"https://monitoring.example.com","text":"Monitoring Dashboard"}]' -m 'High CPU usage detected' |
| 672 |
``` |
| 673 |
|
| 674 |
Full configuration: |
| 675 |
|
| 676 |
```bash |
| 677 |
shoutrrr send -u 'pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=warning&source=load-balancer&client=prometheus&client_url=https://prometheus.example.com/alerts' -m 'Load balancer health check failing' |
| 678 |
``` |
| 679 |
|
| 680 |
### Integration with Scripts |
| 681 |
|
| 682 |
Example script for PagerDuty alerts: |
| 683 |
|
| 684 |
```bash |
| 685 |
#!/bin/bash |
| 686 |
|
| 687 |
INTEGRATION_KEY="eb243592faa24ba2a5511afdf565c889" |
| 688 |
SEVERITY="${1:-error}" |
| 689 |
SOURCE="${2:-unknown}" |
| 690 |
MESSAGE="$3" |
| 691 |
|
| 692 |
if [ -z "$MESSAGE" ]; then |
| 693 |
echo "Usage: $0 [severity] [source] message" |
| 694 |
exit 1 |
| 695 |
fi |
| 696 |
|
| 697 |
URL="pagerduty:///$INTEGRATION_KEY?severity=$SEVERITY&source=$SOURCE" |
| 698 |
|
| 699 |
shoutrrr send -u "$URL" -m "$MESSAGE" |
| 700 |
``` |
| 701 |
|
| 702 |
## Best Practices |
| 703 |
|
| 704 |
### Incident Management |
| 705 |
|
| 706 |
#### Use Appropriate Severity Levels |
| 707 |
|
| 708 |
- Reserve `critical` for issues requiring immediate attention |
| 709 |
- Use `error` for actual problems, `warning` for potential issues |
| 710 |
- Use `info` sparingly for informational events |
| 711 |
|
| 712 |
#### Provide Meaningful Source Identifiers |
| 713 |
|
| 714 |
- Use consistent, descriptive source names |
| 715 |
- Include environment information (e.g., `prod-web-01`, `staging-db`) |
| 716 |
- Avoid generic sources like "unknown" or "default" |
| 717 |
|
| 718 |
#### Include Rich Context |
| 719 |
|
| 720 |
- Always provide `client` and `client_url` for traceability |
| 721 |
- Use `details` for structured data that can be queried |
| 722 |
- Add relevant `contexts` to help responders |
| 723 |
|
| 724 |
### Operational Considerations |
| 725 |
|
| 726 |
#### Rate Limiting Awareness |
| 727 |
|
| 728 |
- Monitor your API usage to avoid hitting limits |
| 729 |
- Implement queuing for high-volume scenarios |
| 730 |
- Use exponential backoff for retries |
| 731 |
|
| 732 |
#### Error Handling Considerations |
| 733 |
|
| 734 |
- Always check for errors when sending notifications |
| 735 |
- Implement proper logging for debugging |
| 736 |
- Handle rate limiting gracefully |
| 737 |
|
| 738 |
#### Security |
| 739 |
|
| 740 |
- Store integration keys securely (environment variables, secrets management) |
| 741 |
- Rotate keys periodically |
| 742 |
- Use least-privilege access |
| 743 |
|
| 744 |
### Integration Patterns |
| 745 |
|
| 746 |
#### Monitoring System Integration |
| 747 |
|
| 748 |
- Map monitoring severity levels to PagerDuty severities |
| 749 |
- Include monitoring system links in `client_url` |
| 750 |
- Use consistent `source` naming conventions |
| 751 |
|
| 752 |
#### CI/CD Pipeline Integration |
| 753 |
|
| 754 |
- Trigger incidents for deployment failures |
| 755 |
- Include build links and commit information |
| 756 |
- Use appropriate severity based on environment |
| 757 |
|
| 758 |
#### Application Integration |
| 759 |
|
| 760 |
- Include error codes, stack traces in `details` |
| 761 |
- Provide links to logs and monitoring |
| 762 |
- Use structured logging for better incident data |
| 763 |
|
| 764 |
### Maintenance |
| 765 |
|
| 766 |
#### Regular Testing |
| 767 |
|
| 768 |
- Test integrations regularly to ensure they work |
| 769 |
- Verify integration keys are valid |
| 770 |
- Test different severity levels and parameters |
| 771 |
|
| 772 |
#### Documentation |
| 773 |
|
| 774 |
- Document your PagerDuty integrations |
| 775 |
- Maintain runbooks for common incident types |
| 776 |
- Keep contact information current |
| 777 |
|
| 778 |
#### Monitoring |
| 779 |
|
| 780 |
- Monitor PagerDuty integration health |
| 781 |
- Set up alerts for integration failures |
| 782 |
- Review incident response times and effectiveness |
| 783 |
|
| 784 |
By following these best practices, you can ensure reliable, effective incident management that helps your team respond quickly and effectively to system issues. |