Microsoft fixes ridiculous Y2K22 Exchange bug that disrupted emails around the world

0
98

Microsoft fixes ridiculous Y2K22 Exchange bug that disrupted emails around the world

Getty Images

Microsoft released a fix for a ridiculous Exchange Server bug that stopped local email delivery around the world as the clocks rang for the new year.

The mass disruption resulted from a date checking bug in Exchange Server 2016 and 2019 that made it impossible for the servers to account for the year 2022, leading some to call it the Y2K22 bug. The mail programs saved the date and time as signed integersthat at 2147483647 or 2. Max31 – 1. Microsoft uses the first two numbers of an update version to indicate the year of publication. As long as it was 2021 or earlier, everything worked fine.

“What the hell Microsoft?”

However, when Microsoft released version 220010001 on New Year’s Eve, local servers crashed because they could not interpret the date. As a result, messages got stuck in transport queues. Administrators around the world were desperately trying to fix bugs instead of ringing in the new year with friends and family. All they had to do was make two cryptic log messages that looked like this:

Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 1:03:42 AM 
Event ID: 5300 
Level: Error 
Computer: server1.contoso.com
Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001" to long.
Log Name: Application 
Source: FIPFS 
Logged: 1/1/2022 11:47:16 AM 
Event ID: 1106 
Level: Error 
Computer: server1.contoso.com 
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

“What the hell Microsoft !?” an admin wrote this reddit thread, which was one of the first forums to report on the mass failure. “On New Years Eve !? The first place I look is on Reddit and you’ll save my life before we even get an engineer on the phone.”

The next day, Microsoft released a fix. There are two forms: an automated one PowerShell script, or a manual solution in case the script did not work properly as reported by some administrators. In either case, the fixes must be performed on each on-premises Exchange 2016 and Exchange 2019 server within an affected organization. The automated script can be run on multiple servers in parallel. The software maker said the automated script could “take a while to run,” and urged administrators to be patient.

The date and time check was performed when Exchange was checking the version of FIP-FS, a scanning engine that is part of Exchange anti-malware protection. Once the FIP-FS versions started with the numbers 22, the verification could not be completed and the email delivery stopped abruptly. The fix stops the Microsoft Filtering Management and Microsoft Exchange Transport services, deletes current AV module files, and installs and starts a patched AV module.

On Monday everything returned to normal for many of the organizations affected. It is not clear how long the bad date storage lasted, but judging by the two affected versions, it may have been introduced during the development of Exchange Server 2016.

Source link

Leave a Reply