Guidewire Cloud Platform Status

Check here for service availability updates on Guidewire Cloud Platform (GWCP).

Subscribe to get real-time updates via email, RSS, Slack and more. You can also request support or report a missing incident via https://community.guidewire.com

TeamCity Service Interruption Affecting Multiple NPE Customers in US/Canada

Incident Report for Guidewire

Postmortem

Unable to Build in TeamCity

Issue Summary:

Incident Start Date/Time: 13th Jan 2022 08:14 (PST)

Incident End Date/Time : 13th Jan 2022 15:37 (PST)

Duration: 7 hours and 23 mins

Description of Impact:

Multiple customers were not able to deploy in GCC in the non-production environments.

Root Cause:

The problem was associated with a Git extension called “git lfs” installed on the TeamCity image. TeamCity makes use of the latest version of “git lfs” as large file storage, the newer version of “git lfs” has a defect causing TeamCity failures when pulling code/files from BitBucket, resulted as build and deployments failures in GCC.

Resolution:

Guidewire introduced a hotfix for GCC versions 1.19, 1.20, and 1.21 to resolve the failures in TeamCity.

Corrective & Preventative Measures:

Guidewire GCC Hotfix 1.19, 1.20, and 1.21 downgraded the “git lfs” version to a previous working version 2.13.3 instead of using the latest version that causes TeamCity failures. Guidewire is also planning a TeamCity upgrade in the future.

Posted Jan 24, 2022 - 21:22 UTC

Resolved

This incident has been resolved. Please open a Community case if you are still experiencing any service interruption.

Posted Jan 13, 2022 - 23:37 UTC

Monitoring

All regions have had the fixes deployed. Please open a Community case if you are still experiencing any service interruption.

Posted Jan 13, 2022 - 21:12 UTC

Update

EMEA fixes have been deployed, this region is now operational.

Posted Jan 13, 2022 - 20:10 UTC

Update

The issue has been observed in all regions. Remedial action is still underway.

Posted Jan 13, 2022 - 19:52 UTC

Update

Guidewire Support has identified the cause of this incident impacting Build failures in GCC in NPE, and is actively working to restore service. At this time, we estimate service will be restored in 2 hour 30 minutes at 12:40pm PT. We will post regular updates on our progress.

Posted Jan 13, 2022 - 18:21 UTC

Update

We are continuing to investigate this issue.

Posted Jan 13, 2022 - 16:49 UTC

Update

We are continuing to investigate a potential issue with TeamCity that is causing Build failures in GCC in NPE that may be causing an impact for our users in US/Canada. We thank you for your patience. Our next update will be when new information becomes available.

Posted Jan 13, 2022 - 16:48 UTC

Update

We are continuing to investigate this issue.

Posted Jan 13, 2022 - 16:30 UTC

Investigating

Guidewire Support is aware of an issue with TeamCity that is causing build failures in GCC in NPE. All necessary resources are engaged to restore this service as quickly as possible. We will post regular updates on our progress.

Posted Jan 13, 2022 - 16:27 UTC

This incident affected: GUIDEWIRE CLOUD CONSOLE (GCC) (USA, CANADA, EMEA, APAC).