SUMMARY
From 9pm UTC on October 23, 2023, until 8pm UTC on October 24, 2023, Atlassian customers using the Jira family products that have configured notifications from custom email domains experienced a degradation as email notifications were being sent from the default atlassian.net domain instead of a custom domain. The incident was detected by Atlassian customers as they started to receive notifications from the atlassian.net domain instead of previously configured custom domains. It was mitigated by excluding redundant validations from the custom domain verification job which put Atlassian systems into a known good state. The total time to resolution was about 23 hours.
IMPACT
The overall impact was between October 23, 2023 9pm UTC and October 24 8pm UTC on Jira family products, including Jira Service Management, Jira Software and Jira Work Management. The incident caused service disruption to customers who have notifications from custom email domains enabled for their Jira products as email notifications were being sent from the default atlassian.net domain instead of the custom domain configured by the customer.
ROOT CAUSE
The issue was caused by a change to the service responsible for custom domain email notifications lifecycle that inadvertently enabled SPF record validation as part of a daily validation job. As a result, previously working custom domain configuration transitioned to an unverified state for some Atlassian customers, resulting in notifications from Jira products being sent from atlassian.net domain instead of a previously configured custom domain.
REMEDIAL ACTIONS PLAN & NEXT STEPS
We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident:
- Improve monitoring and alerting for custom domains lifecycle to detect such regressions before they reach production.
We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.
Thanks,
Atlassian Customer Support
The issue has been resolved and the service is operating normally. Additional details about the issue will be shared in our Post Incident Review
We will continue to monitor this issue for 24 hours to ensure it has been resolved and will share an update once confirmed.
A fix is currently being implemented, and we expect this issue to be resolved shortly.
We have identified the root cause and expect recovery shortly.
We will provide more details within the next hour.