Service Degradation - High Severity: Avaya One X Agent
Update - WashU IT and BJC technical teams continue to troubleshoot Avaya issues with the vendor.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
May 07, 2024 - 16:31 CDT
Update - WashU IT and BJC technical teams are investigating issues with the Avaya One X Agent. We believe the current fix has mitigated user impact at least temporarily. We are monitoring the environment and will evaluate additional corrective actions next week.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Apr 18, 2024 - 18:03 CDT
Update - The change over the weekend does not appear to have improved performance.

An additional change was performed this morning.

Teams will be performing additional testing to evaluate the effectiveness of the change.
Apr 15, 2024 - 11:45 CDT
Investigating - WashU IT and BJC technical teams are investigating issues with the Avaya One X Agent. Technical teams implemented a possible fix but found that the issue was not resolved. An additional change is being scheduled for Sunday 4/14; the next update will come on Monday 4/15 once the change has been evaluated for effectiveness.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Apr 11, 2024 - 16:56 CDT
Identified - WashU IT and BJC technical teams are investigating issues with the Avaya One X Agent. Technical teams have identified a potential fix and are working to validate it with the vendors.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Apr 10, 2024 - 18:06 CDT
Update - WashU IT and BJC technical teams are investigating issues with the Avaya One X Agent. Technical teams have continued to work with the Avaya vendor to test and troubleshoot the issue but have not yet been able to identify root cause. Our next step will be to escalate our support case with Cisco for help analyzing firewall and network configuration, and we will continue to work with Avaya as they continue their analysis of the data we have provided them today.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Apr 09, 2024 - 18:01 CDT
Update - WashU IT and BJC technical teams are investigating issues with the Avaya One X Agent. After some technical discussion, additional troubleshooting is needed before we can continue working on a resolution. Technical teams will reconvene the call on Tuesday morning at 9am.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Apr 08, 2024 - 17:06 CDT
Investigating - WashU IT and BJC technical teams are investigating issues with the Avaya One X Agent. An upgrade was performed in February and appeared stable initially. Since then, an increasing number of remote users on WashU VPN have been reporting issues with audio dropping and similar issues affecting their Avaya service.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Apr 08, 2024 - 14:53 CDT

About This Site

WashU IT's Service Dashboard gives you real-time updates on system performance issues. If you require assistance, please call our Service Desk at 314-933-3333.

Administrative & Enterprise Systems Operational
Alumni and Development Information System (ADIS) Operational
Research Management System (RMS) Operational
Student Information System (SIS) Operational
Vended Applications ? Operational
Cognos Reporting Operational
Marketplace Operational
Vended Applications ? Operational
Workday ? Operational
SUBS Awards ? Operational
Bear Bucks ? Operational
Back-end Processing (AutoSys) ? Operational
Confluence Operational
Archibus Operational
Mulesoft ? Operational
Clinical Operational
EPIC Operational
Vended Applications Operational
Clinical Websites Operational
Credit Card Systems ? Operational
Communications & Collaboration Operational
Email Operational
Office 365 Operational
Microsoft Teams Operational
Maestro mass communication system Operational
Zoom Operational
Box File Storage System ? Operational
WUSTL Connect ? Operational
Computing and Printing Operational
E-Fax Operational
PaperCut Printing Operational
WashU Cloud Storage Operational
Azure Hosted Infrastructure Operational
WashU IT Services / Cloud Computing ? Operational
Desktop Computing ? Operational
File Services Operational
IT Infrastructure Components ? Operational
JAMF Apple Device Management System ? Operational
Active Directory ? Operational
IT Management Operational
Software Licensing Operational
ServiceNow ? Operational
Splunk ? Operational
Network & Connectivity Operational
Internet Operational
Internet 2 Operational
Phone Operational
VPN Operational
WiFi Operational
Washington University Research Network (WURN) Operational
Vendor Services (Spectrum, AT&T, etc.) ? Operational
Network Services ? Operational
Research Operational
Research Compute Operational
Storage Operational
Security Operational
Identity Management System (IDM Wustl Key) Operational
Spam Filter Operational
Duo Two-Factor Authentication ? Operational
Digital Guardian Operational
Teaching & Learning Operational
Canvas Operational
Learn@Work Operational
Kaltura ? Operational
Curriculum Management Applications ? Operational
EMS (Event Management System) ? Operational
University Websites Operational
WashU IT Hosted Websites Operational
Externally Hosted Websites Operational
BJC Devices and Services Operational
BJC Network Connectivity ? Operational
BJC Clinical Applications Operational
BJC Pharmacy POS Machines ? Operational
BJC Phone System Operational
BJC Handheld Supply Scanners (ATPAR) ? Operational
BJC Administrative Systems ? Operational
BJC Enterprise Client Virtualization (ECV) Operational
Cloud Services Operational
Google Operational
Engineering School Operational
Vendor Service ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Jul 27, 2024

No incidents reported today.

Jul 26, 2024
Service Degradation: Splunk Searches
Resolved - WashU IT Engineers have resolved the issue with Splunk.

We appreciate your patience during this incident.
Jul 26, 13:14 CDT
Monitoring - A fix has been fully implemented and teams are now testing.
Jul 26, 13:11 CDT
Identified - Washington University IT has identified an issue with Splunk searches being slow or not completing for some users.

Teams have identified the issue and a fix is being implemented.
Jul 26, 13:01 CDT
Jul 25, 2024

No incidents reported.

Jul 24, 2024
Service Outage - Low Severity: FPP Domain Controller Offline
Resolved - WashU IT has resolved the issues affecting the FPP (Faculty Plan Practice) domain controllers and PowerBI Reporting.

There were multiple issues stemming from the crowdstrike event that were complicated by out-of-sync FPP domain controllers. It was determined that the Crowdstrike event from last week corrupted some PowerBI Reporting Server configuration files. The corrupt files disrupted access to PowerBI reports used by PBS and FPP staff. Approximately 30 users were affected. The issues have been resolved and access to the PowerBI reports has been restored.

If you continue to experience further issues, please contact us at 314-933-3333. We appreciate your patience during this incident.
Jul 24, 20:16 CDT
Investigating - WashU IT is investigating an issue with our FPP (Faculty Practice Plan) Domain controllers. There is no end user impact from this issue; primary impact will be to the PowerBI reporting team.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 24, 15:30 CDT
Jul 23, 2024

No incidents reported.

Jul 22, 2024
Service Outage - High Severity: Windows Servers and Computers Crashing
Resolved - All WashU IT servers have been remediated. WashU continues working with BJC to assist with their efforts.

If you continue to experience further issues, please contact us at 314-933-3333.

We appreciate your patience during this incident.
Jul 22, 08:33 CDT
Update - Service has been restored to all WashU IT managed servers. BJC IT continues their effort to restore clinical services with the help of WashU IT.

Teams will be monitoring service health throughout the weekend.
Jul 19, 13:26 CDT
Monitoring - WashU IT technical teams have made substantial progress in restoring affected servers, with a small number still being addressed. BJC continues to restore their environment as well. Various clinical apps may be impacted within BJC, but EPIC remains online.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 19, 10:43 CDT
Update - WashU IT technical teams continue to work on bringing our systems back online. BJC is also impacted and continues to restore their environment as well. Various clinical apps may be impacted within BJC, but EPIC remains online.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 19, 09:15 CDT
Update - WashU IT is investigating an issue with multiple Windows servers and computers crashing. The issue was identified as a vendor change affecting a cybersecurity application on our servers (and a small number of pilot users' computers). The vendor has deployed a fix for the issue and WashU technical teams continue to work on bringing our systems back online.

BJC is also impacted and continues to restore their environment as well. Various clinical apps may be impacted within BJC, but EPIC remains online.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 19, 07:14 CDT
Update - WashU IT is investigating an issue with multiple Windows servers and computers crashing. The issue was identified as a vendor change affecting a cybersecurity application on our servers (and a small number of pilot users' computers). The vendor has deployed a fix for the issue and WashU technical teams continue to work on bringing our systems back online.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 19, 05:30 CDT
Update - WashU IT is investigating an issue with multiple Windows servers and computers crashing. The issue was identified as a vendor change affecting a cybersecurity application on our servers (and a small number of pilot users' computers). The vendor has deployed a fix for the issue and WashU technical teams are still monitoring our environment to validate which systems are back online.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 19, 03:38 CDT
Identified - WashU IT is investigating an issue with multiple Windows servers and computers crashing. The issue was identified as a vendor change affecting a cybersecurity application on our servers (and a small number of pilot users' computers). The vendor has deployed a fix for the issue and WashU technical teams are in process of verifying the impact on our environment.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 19, 02:05 CDT
Update - The outage is linked to a global issue affecting many customers due to automated software update. The team is engaging the vendor and developing a plan for remediating WashU Systems
Jul 19, 00:21 CDT
Investigating - WashU IT is investigating an issue with multiple virtual computers and servers crashing. Technical teams are investigating the issue.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 19, 00:08 CDT
Jul 21, 2024

No incidents reported.

Jul 20, 2024

No incidents reported.

Jul 19, 2024
Jul 18, 2024

No incidents reported.

Jul 17, 2024
BJC Service Outage - High Severity: Network Outage @ Siteman Cancer Center (Christian Northwest)
Resolved - BJC technical teams have resolved the network issues at Siteman Cancer Center after rebooting some network equipment. If you continue to experience further issues, please contact us at 314-933-3333.

We appreciate your patience during this incident.
Jul 17, 13:22 CDT
Investigating - BJC Technical teams are investigating a network outage at Siteman Cancer Center (Christian Northwest).

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 17, 12:30 CDT
Jul 16, 2024
Service Outage - High Severity: Mobap Users WUSM-Secure "No Internet"
Resolved - WashU IT and BJC Engineers have resolved the issue affecting WUSM-Secure wireless connectivity for users at Mobap. If you continue to experience further issues, please contact us at 314-933-3333.

We appreciate your patience during this incident.
Jul 16, 17:01 CDT
Investigating - WashU IT is investigating an issue with users at Mobap (multiple floors) getting connected to the WUSM-Secure wifi network and then getting "no internet". Users also reported that credit card terminals are down. These users are on BJC wireless access points, so BJC technical teams are assisting with the troubleshooting.

Teams are working to resolve the issue. Please call 314-933-3333 if you have any questions. We appreciate your patience as we work to restore service.
Jul 16, 16:35 CDT
BJC Service Outage: Telephone Services
Resolved - Phone services have been restored. If you continue to experience further issues, please contact BJC at 314-362-4700.

We appreciate your patience during this incident.
Jul 16, 14:11 CDT
Investigating - BJC IT is investigating a telephony disruption at multiple Location. Some users are unable to receive calls or make outbound calls from their Nortel or Avaya desk phones across multiple locations.
Jul 16, 10:38 CDT
Jul 15, 2024

No incidents reported.

Jul 14, 2024

No incidents reported.

Jul 13, 2024

No incidents reported.

← Incident History Powered by Atlassian Statuspage