Microsoft 365 Outage

On 28th September customers started reporting that they couldn’t sign into their Microsoft applications.   

Microsoft had an update to deploy but a latent code defect resulted in an update by passing Microsoft’s normal validation process. Normally Microsoft applies these changes across a validation ring which doesn’t include any customer data. The code defect meant the validation ring did not get tested before deployment to the rings with customer data. This resulted in users not being able to sign into applications which used Azure Active Directory (including office 365 and Microsoft cloud services).   

This was then compounded as Microsoft’s automated rollback failed due to corruption of SDP metadata. They had to manually update the service configuration. 

Microsoft have reported that they’ve fixed the latent code defect, fixed the rollback system and expanded both the scope and frequency of rollback operation drills. They are also working to apply more protections to the Azure Active Directory system.