Alternative Alert Sources That Can Make a Big Impact Without Heavy Lifting

Observability is vital for any organization, but it is complex and usually expensive. And for a reason: being able to have visibility over your system’s inner status requires sophisticated instrumentation, large-scale data processing, and skilled professionals.

However, no matter how good your observability stack is, there are alerts that you won’t be able to pick up. For example, a customer may be experiencing an issue while performing an edge case nobody thought of. Your telemetry is not picking it up. Nobody will get an alert about it. But there is a real incident impacting your customers.

Having alternative ways to collect alerts can be a valuable safety net for your team, and the best thing is that they require basically no implementation work. Of course, each works better for certain teams than others, so we’re including best practices for each of them. Let’s get to it!

Inbox that generates an alert

If your observability solution is not detecting a flaw in a service, it doesn’t mean that the issue your users are experiencing is not real. Providing customers with a way to bring your attention to a problem can be a good way of developing your relationship with them. Setting up an inbox that creates an alert is especially useful for out-of-working hours.

Setting up emails as an alert source gives you the advantage of being able to integrate them into your regular alert workflows. That includes it being routed to a responder on-call and going through a regular triage. Then you can create an incident and escalate it accordingly if needed. You make use of your incident management workflows instead of having to rely on manual steps and a separate inbox to check.

Best practices when setting up an alert inbox:

• Filter out non-customer addresses: You do not want spam to wake up your responders on-call. You want to start by filtering out any email that does not belong to a customer.
• Compatible with tiered support: If you offer tiered support services, you can enable the alerting connected inbox only for customers who are paying for 24/7 assistance.
• Customer-specific address: You can also enable inboxes dedicated to VIP customers and prioritize their alerts in your on-call solution.
• Automate as much as possible: Email alerts can create significant noise, depending on the type of product or service you’re offering and to whom. You can use automations, including AI, in your alert pipeline to recognize if the email received is a question about how to use the product rather than a problem with your service.

{{subscribe-form}}

Vendor statuses as alerts

You likely rely on dozens of vendors to deliver your software, some more critical than others. Certain external dependencies can compromise your business and overall availability, which makes some SRE teams set up alerts to get upfront notice of potential disruptions.

You can use services like IsDown to aggregate status pages from different vendors and set up notifications to your alerting software using webhooks.

Best practices when setting up vendor status-based alerts:

• Watch only vendors that affect your production environment: You do not want to cause alert fatigue for your customers caused by vendor status updates. Choose only the critical vendors that can directly compromise your service.
• Be specific on which kind of statuses trigger an alert: Not every kind of status update is relevant to you. Make sure you’re being selective enough to avoid flooding your responders with alerts from third-party vendors.

Live call rerouting

Enabling an email inbox that generates alerts is good enough for a lot of users. But if you’re a business-critical partner to your customers, having more immediate contact with you in case something goes wrong can consolidate your commitment to their success.

Usually, you’d offer a fixed phone number where your customer is connected to somebody on-call in your team, either through a call or by leaving a voicemail. Live rerouting means that the fixed number will dynamically route the call or voicemail to the right responder, and even go through an escalation policy if applicable.

To enable adequate on-call rerouting, you’ll need an alerting solution that supports it, such as Rootly On-Call. That way, you’ll make use of your typical rotations and escalation policies, minimizing the work you need to enable this option for your customers.

Best practices when enabling live call rerouting:

• Understand if calls are valuable for your customers: Not everyone finds it useful to make a call to raise an alert over a problem they’re experiencing. Figure out if live call rerouting is a desirable option for your customers.
• Keep contact forms in a centralized place: If you can be contacted through regular tickets, emails, or calls, it can become confusing for customers to know when to use each channel or even remember which ones are available. Keep all contact methods in a centralized place that can be easily accessed from your product UI, for example.
• Provide examples on when it’s best to call: To clarify the action path to customers, explain examples of when calling can be helpful. Take into account the kind of service you’re providing support for and your team preferences.
• Use a modern on-call solution: The last thing you want is to end up having a call-center-like implementation. Rootly On-Call lets you enable live call rerouting for your customers through a few clicks and automates the rest.

Conclusion: make your on-call software work for you

Emails, vendor statuses, and live calls as alert sources can be as easy as clicking a few tickboxes in modern on-call solutions like Rootly On-Call. You can enable valuable use cases without burdening your SRE team with implementation work and improve your position as a partner to your customers.

‍