EntrustedMail - Zix Encryption rejecting messages due to Zix Encryption Outage. – Incident details

We have received an update from Zix concerning the outage that took place November 16th.

Please note; the following information is provided from Zix engineering.

Thank you.

Previously you were notified about a major incident we experienced on November 16, 2023, that impacted several of the Zix Services. All impacted services have been restored. The root cause for the issues impacting the Zix Services was a firmware bug affecting the SAN storage application utilized by Zix. Working with our SAN provider, we were able to identify the issue and ultimately replaced the faulty systems. Below is a summary of the services that were impacted and the resolution.

Impact
• Zix Email Encryption customers may have experienced issues sending and receiving emails through the service or delivering emails through the Secure Email Message Portal.
• Zix Email Threat Protection customers may have experienced intermittent filtering disruptions and disruptions in advanced Link Protection, Message Retraction and Attachment Quarantine features.
• Zix Archiving customers using our legacy email archiving solution (console.zixcentral.com) may have experienced a disruption in the archiving of their email communication and may have experienced intermittent issues accessing their archive.

At approximately 10PM E.T. Sunday night,, Zix Engineering resolved the major outage for Zix Encryption service. Apparently, the outage was due to a hard drive(s) failing on a SAN unit.

We will begin rerouting customer outbound email from our stand-by Zix Encryption clusters back over to the Zix Secure Cloud service though-out the day. There will be no interruption of service as a result of our rerouting efforts.

Incident Summary

Last Friday (11/17) morning at 10:30 AM, EntrustedMail restored Zix encryption services to nearly all of our customers by routing email though our stand-by Zix encryption gateway clusters.

Messages sent on Friday, before 10:30 AM ET, that bounced back to the sender as undeliverable throughout the day and/or weekend, will have to be resent by the sender. These messages were queued on the Zix encryption service and were eventually bounced back as a result of the Zix outage.

For nearly all of our customers, messages sent after 10:30 AM ET on Friday, and throughout the weekend, were not impacted by this Zix encryption outage and were encrypted and delivered though the workaround that EntrustedMail implemented.
  
We apologize for any inconvenience that this outage may have caused and will provide a post-mortem report as soon as one is made available from Zix.

The EntrustedMail engineering team has completed the process of moving a subset of customers over to our standby system.

The subset of customers were being severely impacted by the Zix Encryption outage, and could not successful deliver encrypted or unencrypted messages due to the Zix outage.

Please note that messages that have bounced back to the sender will need to be resent.

We apologize for this issue and the inconvenience it has caused.

We will provide updates to the Zix outage as more information comes in from Zix/OpenText.

Thank you,

Andy

The EntrustedMail engineering team is in the process of moving a subset of customers, that have been severely impacted by the Zix Encryption outage, and cannot successful deliver encrypted or unencrypted messages due to the Zix outage, over to our standby data centers gateways so that those customers can send email.

If you are getting bounced messages, your organization is likely within this subset of customers.

We will update the status once we have moved the subset of customers over to our  standby units.

It is important to note that encrypted messages for all customers using Zix encryption maybe undeliverable due to the Zix outage.

Zix has not provided a timeline for a resolution to this outage.