Email not being sent

Incident Report for metapulse.com

Postmortem

On July 13, 2023, metapulse.com was unable to send email for several hours. This was due to the Amazon SES authentication keys being rotated while the servers were using the original keys. The extended downtime was due to difficulty getting the servers to recognize the new keys.

After the issue was resolved we were able to resend most emails including notifications and mailings. However some emails were unrecoverable including users registration and resetting passwords.

Due to this issue, we have taken steps to make MetaPulse less dependent on email. A new Data Exports page is now available which lists all data exports and provides download links instead of relying on email.

Posted Jul 20, 2023 - 19:41 EDT

Resolved

This issue has been fixed. We will be sending out missed emails shortly.

Posted Jul 13, 2023 - 15:15 EDT

Update

We are continuing to work on fixing this issue. We will attempt to resend the unsent emails once this issue is fixed.

Posted Jul 13, 2023 - 13:27 EDT

Update

We are continuing to work on a fix for this issue.

Posted Jul 13, 2023 - 10:53 EDT

Identified

Email is not being sent from our web server. We have identified the issue and are working to resolve it as soon as possible.

Posted Jul 13, 2023 - 10:53 EDT

This incident affected: Background Workers and Web Site.