Debian Codegraph @

docs/services/incident/pagerduty/index.md

raw
1 # PagerDuty
2
3 ## Overview
4
5 PagerDuty is a digital operations management platform that helps teams respond to incidents and outages. The PagerDuty service integrates with PagerDuty's Events API v2 to create, acknowledge, and resolve incidents programmatically. This service enables automated incident management workflows, allowing you to trigger alerts when issues are detected, acknowledge them when investigation begins, and resolve them when problems are fixed.
6
7 The service supports rich incident context including custom details, links to related resources, images, and client information to provide comprehensive incident information to on-call responders.
8
9 ## URL Format
10
11 --8<-- "docs/services/incident/pagerduty/config.md"
12
13 ### URL Components
14
15 The PagerDuty service URL follows this structure:
16
17 ```uri
18 pagerduty://[host[:port]]/integration-key[?query-parameters]
19 ```
20
21 **Components:**
22
23 - **Scheme**: `pagerduty://` - Identifies the service type
24 - **Host** (optional): PagerDuty API hostname (default: `events.pagerduty.com`)
25 - **Port** (optional): API port number (default: `443` for HTTPS)
26 - **Integration Key**: 32-character hexadecimal string that routes events to your PagerDuty service
27 - **Query Parameters** (optional): Additional configuration parameters
28
29 ### Examples
30
31 !!! example "Default configuration"
32 ```url
33 pagerduty:///eb243592faa24ba2a5511afdf565c889
34 ```
35 Uses default host (`events.pagerduty.com`) and port (`443`)
36
37 !!! example "Custom host"
38 ```url
39 pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889
40 ```
41 Explicitly specifies the PagerDuty Events API host
42
43 !!! example "With parameters"
44 ```url
45 pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=critical&source=production-db
46 ```
47 Includes severity and source parameters
48
49 ## Setup and Service Integration
50
51 ### Prerequisites
52
53 Before using the PagerDuty service, you need:
54
55 1. A PagerDuty account
56 2. A service configured in PagerDuty
57 3. An Events API v2 integration key
58
59 ### Creating a Service Integration
60
61 Follow these steps to set up PagerDuty integration:
62
63 1. **Log in to PagerDuty**
64 - Access your PagerDuty account at [pagerduty.com](https://pagerduty.com)
65
66 2. **Navigate to Services**
67 - Go to **Services** → **Service Directory** in the main navigation
68
69 3. **Create or Select a Service**
70 - Either create a new service or select an existing one
71 - If creating new: Click **+ New Service**, give it a name, and select an escalation policy
72
73 4. **Add Events API v2 Integration**
74 - In the service details, click **Integrations** tab
75 - Click **+ Add another integration**
76 - Search for and select **Events API v2**
77 - Give the integration a name (e.g., "Shoutrrr Integration")
78 - Click **Create Integration**
79
80 5. **Copy Integration Key**
81 - The integration key will be displayed
82 - Copy this 32-character key for use in your Shoutrrr URL
83
84 !!! tip "Integration Key Security"
85 Keep your integration key secure. It acts as an authentication token for sending events to PagerDuty.
86
87 !!! note "Only Integration Keys Supported"
88 This service only accepts PagerDuty service integration keys (32 hexadecimal characters matching `/^[a-fA-F0-9]{32}$/`). API keys are not supported and will result in an invalid integration key error.
89
90 ## Parameters
91
92 The PagerDuty service supports the following parameters to customize incident creation and management:
93
94 ### Core Parameters
95
96 | Parameter | Type | Description | Required | Default | Valid Values |
97 |-----------|------|-------------|----------|---------|--------------|
98 | `severity` | string | The perceived severity of the incident | No | `error` | `info`, `warning`, `error`, `critical` |
99 | `source` | string | The unique location of the affected system | No | `default` | Any string identifier |
100 | `action` | string | The type of event to send | No | `trigger` | `trigger`, `acknowledge`, `resolve` |
101
102 ### Additional Parameters
103
104 | Parameter | Type | Description | Required | Default | Valid Values |
105 |-----------|------|-------------|----------|---------|--------------|
106 | `details` | object | Additional custom key-value pairs for the incident | No | - | JSON object or URL-encoded string |
107 | `contexts` | array | Additional context such as links or images | No | - | JSON array of context objects (only 'link' and 'image' types supported) |
108 | `client` | string | Name of the monitoring client/integration | No | - | Any string |
109 | `client_url` | string | URL of the monitoring client dashboard | No | - | Valid URL |
110
111 ### Parameter Details
112
113 #### Severity Levels
114
115 - **`info`**: Informational events that don't require immediate action
116 - **`warning`**: Potential issues that should be monitored
117 - **`error`**: Actual problems affecting system operation
118 - **`critical`**: Urgent issues requiring immediate response
119
120 #### Event Action Items
121
122 - **`trigger`**: Creates a new incident or updates an existing one
123 - **`acknowledge`**: Acknowledges an incident (stops escalation)
124 - **`resolve`**: Resolves an incident (closes it)
125
126 #### Details Parameter
127
128 The `details` parameter accepts a JSON object with custom key-value pairs:
129
130 ```json
131 {
132 "component": "database",
133 "version": "2.1.0",
134 "error_code": "ECONNREFUSED",
135 "affected_users": 150
136 }
137 ```
138
139 #### Contexts Parameter
140
141 The `contexts` parameter accepts a JSON array of context objects. Each context must have a `type` field. Only `link` and `image` types are supported.
142
143 **Link contexts:**
144
145 ```json
146 {
147 "type": "link",
148 "href": "https://example.com/logs",
149 "text": "View Application Logs"
150 }
151 ```
152
153 **Image contexts:**
154
155 ```json
156 {
157 "type": "image",
158 "src": "https://example.com/error-screenshot.png",
159 "href": "https://example.com/dashboard",
160 "text": "Error Screenshot"
161 }
162 ```
163
164 ## Usage Examples
165
166 ### Basic Incident Triggering
167
168 !!! example "Simple alert"
169 ```url
170 pagerduty:///eb243592faa24ba2a5511afdf565c889
171 ```
172 ```bash
173 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889' -m 'Database connection lost'
174 ```
175
176 !!! example "Critical incident with source"
177 ```url
178 pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=critical&source=prod-db-01
179 ```
180 ```bash
181 shoutrrr send -u 'pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=critical&source=prod-db-01' -m 'Production database is unresponsive'
182 ```
183
184 ### Incident Management Workflow
185
186 !!! example "Acknowledge incident"
187 ```url
188 pagerduty:///eb243592faa24ba2a5511afdf565c889?action=acknowledge
189 ```
190 ```bash
191 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=acknowledge' -m 'Engineer has acknowledged the incident and is investigating'
192 ```
193
194 !!! example "Resolve incident"
195 ```url
196 pagerduty:///eb243592faa24ba2a5511afdf565c889?action=resolve
197 ```
198 ```bash
199 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=resolve' -m 'Database connection restored, incident resolved'
200 ```
201
202 ### Advanced Examples
203
204 !!! example "With custom details"
205 ```url
206 pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=warning&source=monitoring&details={"component":"api","version":"1.2.3","response_time":"5.2s"}
207 ```
208 ```bash
209 shoutrrr send -u 'pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=warning&source=monitoring&details={"component":"api","version":"1.2.3","response_time":"5.2s"}' -m 'API response time degraded'
210 ```
211
212 !!! example "With contexts and client info"
213 ```url
214 pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=error&source=ci-pipeline&contexts=[{"type":"link","href":"https://github.com/org/repo/actions/runs/123","text":"GitHub Actions Run"},{"type":"image","src":"https://example.com/failure-screenshot.png","text":"Build Failure"}]&client=jenkins&client_url=https://ci.example.com/job/123
215 ```
216 ```bash
217 shoutrrr send -u 'pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=error&source=ci-pipeline&contexts=[{"type":"link","href":"https://github.com/org/repo/actions/runs/123","text":"GitHub Actions Run"},{"type":"image","src":"https://example.com/failure-screenshot.png","text":"Build Failure"}]&client=jenkins&client_url=https://ci.example.com/job/123' -m 'CI/CD pipeline failed'
218 ```
219
220 ## Event Actions
221
222 The PagerDuty service supports three primary event actions that correspond to the incident lifecycle:
223
224 ### Trigger (`action=trigger`)
225
226 Creates a new incident or updates an existing one with new information. This is the default action.
227
228 **Use cases:**
229
230 - System failures or outages
231 - Performance degradation
232 - Security alerts
233 - Application errors
234
235 **Behavior:**
236
237 - If no incident exists for the service, creates a new one
238 - If an incident exists, adds a new trigger event (may escalate if unacknowledged)
239 - Includes all provided context, details, and parameters
240
241 ### Acknowledge (`action=acknowledge`)
242
243 Acknowledges an existing incident, stopping escalation and indicating that someone is actively working on it.
244
245 **Use cases:**
246
247 - Engineer has started investigation
248 - Issue has been triaged
249 - Response team is assembled
250
251 **Behavior:**
252
253 - Stops incident escalation
254 - Records acknowledgment timestamp
255 - Can include update notes in the message
256
257 ### Resolve (`action=resolve`)
258
259 Resolves an incident, marking it as completed and closing the alert cycle.
260
261 **Use cases:**
262
263 - Issue has been fixed
264 - Service has been restored
265 - Incident is no longer relevant
266
267 **Behavior:**
268
269 - Closes the incident
270 - Records resolution timestamp
271 - Prevents further notifications for this incident
272
273 !!! note "Incident Deduplication"
274 PagerDuty automatically deduplicates incidents based on the integration key and incident details. Multiple triggers for the same issue will update the existing incident rather than create duplicates.
275
276 ## Advanced Features
277
278 ### Custom Details
279
280 The `details` parameter allows you to attach structured data to incidents:
281
282 ```json
283 {
284 "environment": "production",
285 "component": "web-server",
286 "version": "2.1.0",
287 "error_code": "500",
288 "affected_services": ["api", "database"],
289 "metrics": {
290 "cpu_usage": 95.2,
291 "memory_usage": 87.1
292 }
293 }
294 ```
295
296 These details appear in the PagerDuty incident details and can be used for filtering, reporting, and automation.
297
298 ### Contexts
299
300 Contexts provide additional visual and navigational context to responders:
301
302 **Link Contexts:**
303
304 - Direct responders to relevant resources
305 - Include runbooks, documentation, or monitoring dashboards
306 - Support both internal and external links
307
308 **Image Contexts:**
309
310 - Attach screenshots or graphs
311 - Show error visualizations or system diagrams
312 - Include clickable links to full-size images
313
314 ### Client Information
315
316 The `client` and `client_url` parameters identify the monitoring system:
317
318 - **`client`**: Name of your monitoring tool (e.g., "Prometheus", "DataDog", "Custom Monitor")
319 - **`client_url`**: Direct link to the monitoring dashboard or alert details
320
321 This helps responders understand where the alert originated and provides quick access to additional monitoring data.
322
323 ### Incident Grouping
324
325 PagerDuty supports incident grouping based on custom details. Use consistent detail keys to group related incidents:
326
327 ```json
328 {
329 "service": "payment-api",
330 "region": "us-west-2",
331 "cluster": "production"
332 }
333 ```
334
335 ## Rate Limiting
336
337 The PagerDuty Events API v2 enforces rate limits to ensure system stability:
338
339 - **120 events per minute** per integration key (higher ceilings available on AIOps/enterprise tiers)
340 - Rate limits are enforced per integration key, not per service
341 - Exceeding limits returns HTTP 429 (Too Many Requests)
342
343 ### Handling Rate Limits
344
345 When rate limits are exceeded:
346
347 1. **Implement backoff**: Use exponential backoff (start with 1 minute, double each retry)
348 2. **Queue messages**: Buffer notifications during high-volume periods
349 3. **Use multiple integrations**: Distribute load across multiple integration keys if needed
350 4. **Monitor usage**: Track your API usage in PagerDuty's integration settings
351
352 !!! tip "Rate Limit Best Practices"
353 - Implement retry logic with jitter to avoid thundering herd problems
354 - Consider using PagerDuty's enterprise features for higher limits if needed
355 - Monitor your integration's usage in PagerDuty's web interface
356
357 ## Error Handling
358
359 The PagerDuty service returns standard HTTP status codes. Handle these appropriately in your applications:
360
361 ### Success Responses
362
363 - **200 OK**: Event processed successfully
364 - **202 Accepted**: Event accepted for processing (asynchronous)
365
366 ### Error Responses
367
368 - **400 Bad Request**: Invalid request payload or parameters
369 - Check integration key format (must be 32 hex characters)
370 - Validate JSON structure for details/contexts
371 - Ensure URLs are properly encoded
372
373 - **401 Unauthorized**: Invalid or missing integration key
374 - Verify the integration key is correct
375 - Check that the integration hasn't been disabled
376
377 - **403 Forbidden**: Integration lacks required permissions
378 - Ensure the integration key has appropriate permissions
379 - Check service configuration in PagerDuty
380
381 - **429 Too Many Requests**: Rate limit exceeded
382 - Implement exponential backoff
383 - Consider queuing messages
384
385 - **500 Internal Server Error**: PagerDuty service error
386 - Retry with backoff
387 - Check PagerDuty status page for outages
388
389 ### Common Issues and Solutions
390
391 !!! tip "Integration Key Validation"
392 Integration keys must be exactly 32 hexadecimal characters. Common mistakes include:
393 - Using API keys instead of integration keys
394 - Extra whitespace or characters
395 - Using keys from disabled integrations
396
397 !!! tip "JSON Parameter Encoding"
398 When passing complex parameters like `details` or `contexts` in URLs, ensure proper JSON encoding:
399
400 **Correct:**
401
402 ```bash
403 details={"key":"value"}
404 ```
405
406 **Incorrect - missing quotes:**
407
408 ```bash
409 details={key:value}
410 ```
411
412 !!! tip "Context Object Validation"
413 Each context object must have a valid `type` field. Supported types are `link` and `image`.
414
415 ## Code Examples
416
417 ### Go Integration
418
419 #### Basic Incident Triggering Implementation
420
421 ```go
422 package main
423
424 import (
425 "log"
426 "github.com/containrrr/shoutrrr"
427 )
428
429 func main() {
430 url := "pagerduty:///eb243592faa24ba2a5511afdf565c889"
431
432 err := shoutrrr.Send(url, "Database connection failed")
433 if err != nil {
434 log.Fatal(err)
435 }
436 }
437 ```
438
439 #### Advanced Configuration with Parameters
440
441 ```go
442 package main
443
444 import (
445 "log"
446 "github.com/containrrr/shoutrrr"
447 "github.com/containrrr/shoutrrr/pkg/types"
448 )
449
450 func main() {
451 url := "pagerduty:///eb243592faa24ba2a5511afdf565c889"
452
453 // Trigger critical incident
454 params := &types.Params{
455 "severity": "critical",
456 "source": "production-db-01",
457 "action": "trigger",
458 "details": `{"database": "postgres", "error": "connection timeout"}`,
459 "client": "monitoring-system",
460 "client_url": "https://monitoring.example.com/alerts/123",
461 }
462
463 err := shoutrrr.Send(url, "Production database is down", params)
464 if err != nil {
465 log.Fatal(err)
466 }
467 }
468 ```
469
470 #### Incident Management Workflow Implementation
471
472 ```go
473 package main
474
475 import (
476 "log"
477 "time"
478 "github.com/containrrr/shoutrrr"
479 "github.com/containrrr/shoutrrr/pkg/types"
480 )
481
482 func handleIncident() {
483 baseURL := "pagerduty:///eb243592faa24ba2a5511afdf565c889"
484
485 // 1. Trigger incident
486 triggerParams := &types.Params{
487 "severity": "critical",
488 "source": "web-server-01",
489 "action": "trigger",
490 }
491
492 err := shoutrrr.Send(baseURL, "Web server unresponsive", triggerParams)
493 if err != nil {
494 log.Printf("Failed to trigger incident: %v", err)
495 return
496 }
497
498 // Simulate investigation time
499 time.Sleep(5 * time.Minute)
500
501 // 2. Acknowledge incident
502 ackParams := &types.Params{
503 "action": "acknowledge",
504 }
505
506 err = shoutrrr.Send(baseURL, "Engineer investigating server issue", ackParams)
507 if err != nil {
508 log.Printf("Failed to acknowledge incident: %v", err)
509 return
510 }
511
512 // Simulate resolution time
513 time.Sleep(15 * time.Minute)
514
515 // 3. Resolve incident
516 resolveParams := &types.Params{
517 "action": "resolve",
518 }
519
520 err = shoutrrr.Send(baseURL, "Web server restarted successfully", resolveParams)
521 if err != nil {
522 log.Printf("Failed to resolve incident: %v", err)
523 return
524 }
525 }
526 ```
527
528 #### Using Contexts for Rich Incident Details
529
530 ```go
531 package main
532
533 import (
534 "log"
535 "github.com/containrrr/shoutrrr"
536 "github.com/containrrr/shoutrrr/pkg/types"
537 )
538
539 func sendRichIncident() {
540 url := "pagerduty:///eb243592faa24ba2a5511afdf565c889"
541
542 params := &types.Params{
543 "severity": "error",
544 "source": "ci-pipeline",
545 "action": "trigger",
546 "details": `{
547 "pipeline": "deploy-prod",
548 "stage": "test",
549 "commit": "abc123def",
550 "branch": "main"
551 }`,
552 "contexts": `[
553 {
554 "type": "link",
555 "href": "https://github.com/org/repo/commit/abc123def",
556 "text": "View Commit"
557 },
558 {
559 "type": "link",
560 "href": "https://ci.example.com/pipelines/123",
561 "text": "Pipeline Details"
562 },
563 {
564 "type": "image",
565 "src": "https://ci.example.com/screenshots/failure.png",
566 "text": "Test Failure Screenshot"
567 }
568 ]`,
569 "client": "GitHub Actions",
570 "client_url": "https://github.com/org/repo/actions/runs/456",
571 }
572
573 err := shoutrrr.Send(url, "CI/CD pipeline failed in production deployment", params)
574 if err != nil {
575 log.Fatal(err)
576 }
577 }
578 ```
579
580 ### Integration with Monitoring Systems
581
582 ```go
583 package main
584
585 import (
586 "encoding/json"
587 "log"
588 "github.com/containrrr/shoutrrr"
589 "github.com/containrrr/shoutrrr/pkg/types"
590 )
591
592 // AlertData represents alert data from a monitoring system
593 type AlertData struct {
594 Title string `json:"title"`
595 Description string `json:"description"`
596 Severity string `json:"severity"`
597 Source string `json:"source"`
598 Details map[string]interface{} `json:"details"`
599 DashboardURL string `json:"dashboard_url"`
600 }
601
602 func sendMonitoringAlert(data AlertData) error {
603 url := "pagerduty:///eb243592faa24ba2a5511afdf565c889"
604
605 // Convert details to JSON string
606 detailsJSON, err := json.Marshal(data.Details)
607 if err != nil {
608 return err
609 }
610
611 params := &types.Params{
612 "severity": data.Severity,
613 "source": data.Source,
614 "action": "trigger",
615 "details": string(detailsJSON),
616 "client": "Custom Monitoring",
617 "client_url": data.DashboardURL,
618 }
619
620 return shoutrrr.Send(url, data.Title+": "+data.Description, params)
621 }
622 ```
623
624 ## CLI Examples
625
626 ### Basic Usage
627
628 Send a simple alert:
629
630 ```bash
631 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889' -m 'Server is down'
632 ```
633
634 Send with severity:
635
636 ```bash
637 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?severity=critical' -m 'Production outage detected'
638 ```
639
640 ### Incident Lifecycle Management
641
642 Trigger incident:
643
644 ```bash
645 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=trigger&severity=critical&source=web-app' -m 'Application crashed'
646 ```
647
648 Acknowledge incident:
649
650 ```bash
651 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=acknowledge' -m 'DevOps team is investigating'
652 ```
653
654 Resolve incident:
655
656 ```bash
657 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?action=resolve' -m 'Application restarted successfully'
658 ```
659
660 ### Advanced Parameters
661
662 With custom details:
663
664 ```bash
665 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?details={"component":"database","error":"connection_failed"}' -m 'Database connectivity issue'
666 ```
667
668 With contexts:
669
670 ```bash
671 shoutrrr send -u 'pagerduty:///eb243592faa24ba2a5511afdf565c889?contexts=[{"type":"link","href":"https://monitoring.example.com","text":"Monitoring Dashboard"}]' -m 'High CPU usage detected'
672 ```
673
674 Full configuration:
675
676 ```bash
677 shoutrrr send -u 'pagerduty://events.pagerduty.com/eb243592faa24ba2a5511afdf565c889?severity=warning&source=load-balancer&client=prometheus&client_url=https://prometheus.example.com/alerts' -m 'Load balancer health check failing'
678 ```
679
680 ### Integration with Scripts
681
682 Example script for PagerDuty alerts:
683
684 ```bash
685 #!/bin/bash
686
687 INTEGRATION_KEY="eb243592faa24ba2a5511afdf565c889"
688 SEVERITY="${1:-error}"
689 SOURCE="${2:-unknown}"
690 MESSAGE="$3"
691
692 if [ -z "$MESSAGE" ]; then
693 echo "Usage: $0 [severity] [source] message"
694 exit 1
695 fi
696
697 URL="pagerduty:///$INTEGRATION_KEY?severity=$SEVERITY&source=$SOURCE"
698
699 shoutrrr send -u "$URL" -m "$MESSAGE"
700 ```
701
702 ## Best Practices
703
704 ### Incident Management
705
706 #### Use Appropriate Severity Levels
707
708 - Reserve `critical` for issues requiring immediate attention
709 - Use `error` for actual problems, `warning` for potential issues
710 - Use `info` sparingly for informational events
711
712 #### Provide Meaningful Source Identifiers
713
714 - Use consistent, descriptive source names
715 - Include environment information (e.g., `prod-web-01`, `staging-db`)
716 - Avoid generic sources like "unknown" or "default"
717
718 #### Include Rich Context
719
720 - Always provide `client` and `client_url` for traceability
721 - Use `details` for structured data that can be queried
722 - Add relevant `contexts` to help responders
723
724 ### Operational Considerations
725
726 #### Rate Limiting Awareness
727
728 - Monitor your API usage to avoid hitting limits
729 - Implement queuing for high-volume scenarios
730 - Use exponential backoff for retries
731
732 #### Error Handling Considerations
733
734 - Always check for errors when sending notifications
735 - Implement proper logging for debugging
736 - Handle rate limiting gracefully
737
738 #### Security
739
740 - Store integration keys securely (environment variables, secrets management)
741 - Rotate keys periodically
742 - Use least-privilege access
743
744 ### Integration Patterns
745
746 #### Monitoring System Integration
747
748 - Map monitoring severity levels to PagerDuty severities
749 - Include monitoring system links in `client_url`
750 - Use consistent `source` naming conventions
751
752 #### CI/CD Pipeline Integration
753
754 - Trigger incidents for deployment failures
755 - Include build links and commit information
756 - Use appropriate severity based on environment
757
758 #### Application Integration
759
760 - Include error codes, stack traces in `details`
761 - Provide links to logs and monitoring
762 - Use structured logging for better incident data
763
764 ### Maintenance
765
766 #### Regular Testing
767
768 - Test integrations regularly to ensure they work
769 - Verify integration keys are valid
770 - Test different severity levels and parameters
771
772 #### Documentation
773
774 - Document your PagerDuty integrations
775 - Maintain runbooks for common incident types
776 - Keep contact information current
777
778 #### Monitoring
779
780 - Monitor PagerDuty integration health
781 - Set up alerts for integration failures
782 - Review incident response times and effectiveness
783
784 By following these best practices, you can ensure reliable, effective incident management that helps your team respond quickly and effectively to system issues.