MENU

Status Updates


UM System wide outages can be found on the UM IT System Status website.

Current Outages

 UMKC Blackboard Request System unavailable (4/26/2017 9:00:00 AM - 4/27/2017 5:00:00 PM)

Starting at approximately 9:00am this morning, UMKC Blackboard Request system is unavailable for maintenance. The staff are working to restore normal services.

Upcoming Outages

 Applying January oracle patch. (5/5/2017 10:00:00 PM - 5/5/2017 11:59:00 PM)

KC-ISDW-ORAPROD will be down during these hours to apply January Oracle patch.

 Applying January oracle patch. (5/5/2017 10:00:00 PM - 5/5/2017 11:59:00 PM)

KC-DS-AXIUMPROD will be down during these hours to apply January Oracle patch.

 Applying January oracle patch. (5/5/2017 10:00:00 PM - 5/5/2017 11:59:00 PM)

KC-DS-AXIUMDG will be down during these hours to apply January Oracle patch.

 Applying January MySql patch. (5/5/2017 11:00:00 PM - 5/6/2017 1:00:00 AM)

KC-IS-REDCAPDB will be down during these hours to apply MySQL January patch.

 Applying January MySql patch. (5/5/2017 11:00:00 PM - 5/6/2017 1:00:00 AM)

KC-SCE-APPDB01 will be down during these hours to apply Jan MySQL Patch

 Ecommerce server upgrade (5/12/2017 9:00:00 PM - 5/12/2017 10:30:00 PM)

UMKC Ecommerce server will be upgraded with a new version of Oracle client. All the eCommerce websites will be unavailable during this upgrade.

Past Outages (Last 72 Hours)

 Blackboard is experiencing degraded performance and/or availability issues (4/25/2017 9:30:00 AM - 4/25/2017 10:30:00 AM)

Starting at approximately 9:30 this morning, UMKC Blackboard users began noticing slowness or non-responsiveness. The issues have been reported to Blackboard Inc and staff are working urgently to restore normal services.

 Blackboard - Intermittent Outages (4/24/2017 10:50:00 PM - 4/25/2017 2:00:00 AM)

Blackboard is experiencing intermittent sign-on issues that are preventing some users from logging in. We are working to resolve this problem quickly.

Current and Upcoming Change Management Notices

 Storage Array Maintenance (12/21/2015 7:00:00 AM)

We will be performing maintenance on our storage arrays. This could result in reduced performance during the process. Our DR array will be worked on first, followed by the production array.

 vCenter Patching (12/23/2015 7:00:00 AM)

We will be applying patches to our vCenter instance. VMs will be unaffected, but no changes to existing VMs will be possible during the upgrade process.

 Enable DNSsec on UMKC managed DNS zones (5/16/2016 12:00:00 PM)

DNSsec signing will be enabled on DNS zones managed by UMKC. This is the first of two parts to better ensuring UMKC DNS information is not tampered with by end ISPs. The second step is to publish the signing key with our registrars, so that outside sites can validate the signing. This will take place at a later date after we confirm the signing is working. (second step can result in loss of name resolution if the first step has any errors)

 Disable IPv6 Source Routing capability on Windows OS devices (5/17/2016 8:00:00 AM)

IPv6 Source Routing is likely enabled by default on Windows OS devices. This will be disabled via a group policy registry key. This will only take effect after a reboot, after the group policy change is put into place. Source Routing is normally only used for testing and diagnostics, and should not be used for production, and can be abused by attackers trying to get around network security devices.

 Emergency data center firewall reboot (5/17/2016 10:00:00 PM)

Due to performance issues seen on the data center firewall, a reboot will be performed tonight between 10pm and 11pm to clear the data plane CPU. This will result in two 1 minute outages as we fail over the firewalls for the reboot.

We are also working to identify high bandwidth uses of data center resources, and working to set reasonable limits on those systems, to ensure enough bandwidth is available for all other users.

 DNSsec public keys will be posted to DNS domain name registrars (5/20/2016 8:00:00 AM)

Once the initial DNSsec signing has been validated, the DNSsec public keys will be posted to our DNS registrars. At that point enforcement of the signing will take effect. This will be done on May 20th, a few days after the initial signing has started, and once the signing has been validated.

 Domain controller security change (5/25/2016 5:30:00 AM)

A security change will be made between UMKC's domain controller and the Active Directory root. This change is necessary to eliminate the use of RC4 ciphers between domains. This will enable Kerberos tickets that cross domains to use AES encryption.

This change should not cause any outages, and should strengthen the signing of Kerberos tickets.

This is a necessary step to re-attempting to disable RC4 Kerberos ticket signing at UMKC. When we disabled RC4 Kerberos ticket signing, any unauthorized Windows 2003 and XP machines will stop talking to Active Directory.

 Proactive Data Center Firewall reboot (6/4/2016 10:00:00 PM)

In order to avoid any issues that may have to do with long uptime (potential memory or process leaks), a Data Center firewall reboot will be performed roughly every 21 days on a weekend during the network maintenance time window. These firewalls participate in active routing, so even with a failover, there is a 1 minute network outage as routing tables sync up.

 Change in Windows 7 and Windows servers to prevent caching passwords in memory (6/23/2016 8:00:00 AM)

(Note adjusting to 23rd from 22nd due to a calendar issue) A change will be applied as per Microsoft KB: https://support.microsoft.com/en-us/kb/2871997 to prevent storing password in memory. This change already exists on Windows 10 machines deployed widely on campus, and Windows 2012 R2 machines. This will bring Windows 7 and Windows 2008/2008R2/2012 servers up to the same level of security. Since this is already running on newer machines successfully, this should not be a major problem.

 Proactive Data Center Firewall reboot (6/25/2016 10:00:00 PM)

In order to avoid any issues that may have to do with long uptime (potential memory or process leaks), a Data Center firewall reboot will be performed roughly every 21 days on a weekend during the network maintenance time window. These firewalls participate in active routing, so even with a failover, there is a 1 minute network outage as routing tables sync up.

 Campus WiFi controller update (6/26/2016 11:45:00 PM)

Wireless will be down while the controllers reboot a few times as part of applying their new code and the APs will reboot prior to rejoining the controller. The Access Points will not be broadcasting nor allowing any WiFi traffic for a period of time that is usually no greater than an hour. All wired traffic should remain unaffected.

 Proactive Data Center Firewall reboot (7/9/2016 10:00:00 PM)

In order to avoid any issues that may have to do with long uptime (potential memory or process leaks), a Data Center firewall reboot will be performed roughly every 21 days on a weekend during the network maintenance time window. These firewalls participate in active routing, so even with a failover, there is a 1 minute network outage as routing tables sync up.

 Enabling Flashback on KC-ISIA-ORAPRD1 (7/15/2016 12:30:00 AM)

KC-ISIA-ORAPRD1 will be down for like 10 minutes during this time.

 Emergency reboot of campus edge firewalls tonight (7/18/2016 10:00:00 PM)

Due to a problem on the campus edge firewalls, a reboot is needed to reset their licensing status. This should not result in any traffic outage due to the redundant setup of the firewalls, but there is a chance of traffic problems with any restart.

 Demotion and removal of KC-DC03 (7/19/2016 8:00:00 AM)

The Domain Controller KC-DC03 will be demoted and taken out of service. The location this DC is currently in, will no longer be used for data center services. This still leaves 3 Domain Controllers on campus, and two off campus for DR and Exchange Support purposes. Systems should already be using the generic names for finding Active Directory services, so this should be a minor change.

 Shutdown of 134.193.159.52 and 2610:e0:a040:68fd::2 for DNS (7/20/2016 8:00:00 AM)

The IP addresses of 134.193.159.52 and 2610:e0:a040:68fd::2 will be taken out of service as a valid client DNS servers at UMKC. The location this DNS server is in, will no longer be used as a Data Center, and the subnet is not a portable subnet, so it cannot be moved to a new location to keep these IP addresses active.

The remaining client DNS IP addresses are:
134.193.1.2
134.193.83.4
2610:e0:a040:64fc::2
2610:e0:a040:6401::2

Any system manually setting DNS servers, should ensure they are only using the above remaining IP addresses. The DNS server will also be monitored to identify any systems still using the old settings. DHCP enabled machines should already be using the new settings.

 Proactive Data Center Firewall reboot (7/23/2016 10:00:00 PM)

In order to avoid any issues that may have to do with long uptime (potential memory or process leaks), a Data Center firewall reboot will be performed roughly every 21 days on a weekend during the network maintenance time window. These firewalls participate in active routing, so even with a failover, there is a 1 minute network outage as routing tables sync up.

 Starfish Connect Production Instance is being upgraded to version 6.4 (8/6/2016 1:15:00 AM)

The Starfish Connect PRD server is being upgraded to version 6.4 this weekend. This is a SAAS solution and is not an opt-in/opt-out type of upgrade. This CM notice is being posted simply to record this change.

 Upgrade campus data center boundary firewalls and Internet edge firewalls to current software level (8/6/2016 10:00:00 PM)

The data center boundary firewall pair and the Internet edge firewalls will be upgraded to the current software levels. This will hopefully help with resource issues seen on the data center firewalls, but also to ensure current levels of fixes. At this time we will stay in the same software major version and only update to its current fixes version. A major version upgrade will be done later when more time can be planned out on the upgrade.

For the Data Center, there will be two 1-minute outages as the firewall pair is involved in OSPF routing that is slow to converge properly on campus.

For the campus edge pair, there should not be any noticable outage as the failover is very fast due to static routing.

 dropbox.umkc.edu update (9/2/2016 10:00:00 PM)

Dropbox.umkc.edu service will be offline September 2, 2016 Friday night for a system update.

 WebEx Faculty / Staff Site Authentication Changes (9/4/2016 9:00:00 AM)

Information Services will be enabling shibboleth authentication on the WebEx Faculty / Staff site umkc.webex.com on Sunday Sept 4th at 9:00am. These changes are expected to take a few hours to complete and during this time users will not be able to access the umkc.webex.com site.

 Urgent campus WiFi controller update to fix major bugs found in new product line (9/4/2016 11:55:00 PM)

Wireless will be down while the controllers reboot a couple of times as part of applying their new code and the APs will reboot prior to rejoining the controller. The Access Points will not be broadcasting nor allowing any WiFi traffic for a period of time that is usually no greater than an hour. Some wired traffic will be affected, such as the ports on our existing 1810w devices.

 Removal of existing Operations Manager agents on campus computers (9/15/2016 12:00:00 PM)

A bootup script change will be made to remove all old versions of Microsoft Operations Manager from campus computers. This will remove 2012R2 and older versions of the agents.

The Operations Manager server is used to collect antivirus events from campus Windows computers. (servers and workstations run this agent)

A new Operations Manager server will be setup using the new 2016 edition of the software. Once installed, the new agents will be deployed to campus computers. Due to problems with some of the settings of the older configuration, a full removal of the old settings and agents is being done first, to ensure the cleanest move into the new edition.

This change will not affect end-users, this is a behind the scenes change of the agents installed on the machines. These changes will happen at reboot time only.

 Datacenter firewall settings change (9/21/2016 10:00:00 PM)

A settings change will be made on the datacenter firewall, that should not be disruptive, but in case it causes problems is being scheduled.
This change will occur at 10pm, and will be reversed if any problems are seen with traffic flow.
The change is intended to lower the dataplane CPU problem we have been seeing over the past two days, in order to ensure we do not have another outage or network slowdown, and is a recommended change by the vendor support team.

 Firmware upgrade of campus edge firewalls (10/1/2016 10:00:00 PM)

In order to support a new licensing model for VPN connections, the campus edge firewall will be upgraded. This is an intermediate version upgrade, a further version upgrade is still scheduled for December to get to the newest set of capabilities.

New versions of Apple iOS and macOS no longer support PPTP VPN connections. In order to provide a VPN capability, a new type of VPN will be setup on the campus firewall, in hopes of supporting these devices. A firmware upgrade is needed to enable the first level of the licensing needed.

The firewall reboot will upgrade one unit at a time. The edge firewalls are setup so that there should be only a 2 second outage as each unit switches over, so no significant impact on services is expected.

 KMS changes for Windows 2016 and Windows 7 (10/19/2016 4:27:00 PM)

KMS licensing servers have been changed to two KMS servers, due to incompatibility with imaged Windows 7 machines and the newer 2016 KMS server code. New Windows 2016 machines will be pointed to the new KMS server, and all other machines will point at the legacy KMS due to this compatibility issue.

 Parking.umkc.edu System Upgrade (10/23/2016 7:00:00 PM)

The parking.umkc.edu server/web site will be offline on Sunday 23, 2016 at 7PM for a system upgrade.

 Switch active PaloAlto unit for campus Internet edge (11/17/2016 11:00:00 PM)

Due to the an unexpected switch of the active/passive setup on the PaloAlto, this will be switched back tonight. Normally switch-over of which unit is active only affects traffic for 2 to 5 seconds. No other impact is expected.

 Upgrade Filemaker server form v13 to v15. (11/18/2016 6:00:00 PM)

Upgrade Filemaker server form v13 to v15 on KC-ISIA_FMP01

 Functionality change with Papercut (11/21/2016 10:00:00 AM)

We will be changing the functionality where the database throws an error for a alphanumeric input (SSO) by the user. The current functionality is causing security issues.

 Apply Oracle CPU patch on KC-CBORD-GOLDDB (11/22/2016 10:00:00 PM)

Applying July Oracle CPU patch on KC-CBORD-GOLDDB

 Change of the file download source for Windows Desktops and Laptops (12/12/2016 2:00:00 PM)

In order to avoid a bottleneck we are facing with workstation (desktop and laptop) updates downloading from the Data Center, we are changing these machines to instead pull update files directly from Microsoft.

Approvals of updates will still pull from UMKC. So for instance a newly released update from Microsoft would not download to a workstation until it is approved for campus use.

Servers will remain pointing at the local update server for files, as they are inside the data center firewall, and not subject to the same bottleneck.

This change will be made today, as some workstations that are significantly behind on updates are not catching up, and this change will allow them to download the catchup files faster. Also patch Tuesday is tomorrow, and this modification should be in place before we start staging the pre-deployment testing updates.

This change will not adversely affect already up-to-date machines, and the November updates are now past-due, so this will help past-due machines.

 Deploy Operations Manager 2016 agent (12/17/2016)

The agent deployment for Operations Manager 2016 agent will be moved to start on bootups after December 16th. There were some issues with the prior attempt where Operations Manager 2016 was not using Active Directory for agent assignment. Instead agents will now be assigned to a management server based on the OS version and based on Workstation or Server classification, and the assignment will be made when the agent is installed.

This agent picks up event and health information from campus Windows computers. (workstations and servers) This should be a minor impact, and is a replacement of the current Operations Manager 2012R2 configuration.

 Exchange 2016 Migration (12/20/2016 5:00:00 PM)

UMKC will be migrating our Exchange services to new servers on Dec 20th after 5:00pm.

 Campus WiFi controller update (12/22/2016 11:30:00 PM)

Wireless will be down while the controller pair reboots as part of applying the new code. The APs will reboot prior to rejoining the controller. The Access Points will not be broadcasting, nor allowing any WiFi traffic for a period of time that is usually no greater than an hour. All wired traffic should remain unaffected. Typical updates and reboot cycles take 30 minutes for all devices to rejoin, with no more than 10% of units taking the full hour to complete.

 Blackboard Unavailable (12/27/2016 6:00:00 PM)

Blackboard will be undergoing major maintenance, beginning on Wednesday, December 28th. Blackboard will NOT be available during this time, as we implement multiple upgrades and fixes. Full service and access will be restored by midnight on Saturday, December 31st.

 dropbox.umkc.edu update (12/27/2016 10:00:00 PM)

Dropbox.umkc.edu service will be offline Tuesday night December 27th for a system update.

 Spam and Phishing reporting buttons for Outlook on Windows (2/22/2017 8:00:00 AM)

A program will be installed on Windows computers to allow spam and phishing reporting from Outlook. The buttons will appear on the top ribbon bar of Outlook on campus managed computers. This will help ease reporting of these types of email.

An equivalent reporting program is not available for Outlook on Macintosh computers at this time.

 ALEKS schema migration (2/23/2017 11:42:00 AM)

Access to the ALEKS website will be down while the underlying database is migrated to a new server

 The Kerberos TGT account password will be changed (3/18/2017 10:00:00 PM)

As part of a yearly process to ensure the Kerberos TGT account is secure, and to block certain attacks related to pass-the-hash, the password on the Kerberos TGT account will be changed. This has the potential to affect currently logged on accounts, as logons prior to the change will have credentials with the prior encryption material attached. This will be done overnight, to ensure the least disruption if any problems occur. The fix for any issue from this change is to logout and logon again, which mostly affects campus managed Windows computers. (This should not affect remote email access.)

 Campus data center firewall reboot (3/18/2017 10:00:00 PM)

The campus data center firewall will be rebooted, to proactively attempt to reduce high data plane usage seen earlier in the week. This will result in a 1 minute outage of campus server resources, and will also require a reboot of certain phones and dependent systems.

 Campus WiFi controller update (3/26/2017 1:30:00 AM)

Wireless will be down while the controller pair reboots as part of applying the new code. The APs will reboot prior to rejoining the controller. The Access Points will not be broadcasting, nor allowing any WiFi traffic for a period of time that is usually no greater than an hour. All wired traffic should remain unaffected. Typical updates and reboot cycles take 30 minutes for all devices to rejoin, with no more than 10% of units taking the full hour to complete.

 Critical Bug Fix on CAT3850 series (3/28/2017 7:05:00 PM)

The 03.07.02E code for the Catalyst 3800 series switches has a critical security flaw that could leave our configs vulnerable without any direct authentication.

 Med School network upgrade (3/28/2017 8:30:00 PM)

We will be replacing the distribution switches at Med School. This will result in a loss of phone and network during this period.

 Panopto Local Recordings Auto-Deletion Script Deployment (4/17/2017 9:00:00 AM)

This change will implement a script that automatically removes Panopto recordings that are stored locally on ILE machines older than 30 days on shutdown.

 Emergency reboot of campus edge firewalls (4/20/2017 10:00:00 PM)

The live campus edge firewall has become semi-responsive for management changes, and needs a reboot. As traffic is still being filtered and passed correctly, this will be scheduled to be done tonight at 10pm. There will be two brief two-second outages. The first as the main unit is restarted and the secondary unit takes over. The second when the secondary unit is rebooted and traffic resumes running on the primary unit. Internet connectivity should stay active, and phone services should be unaffected by this change.

Without this manual intervention, we cannot make additional changes on the firewall, and a self-reboot at an uncontrolled time is likely to happen. As it is, there is a chance the firewall may select to reboot itself before 10pm if the issue gets worse before the reboot time selected.

 Apply Jan Oracle CPU patch on KC-CHI-DATA (4/28/2017 3:34:00 PM)

Apply Jan Oracle CPU patch on KC-CHI-DATA

 Core switch IOS upgrade (4/29/2017 6:00:00 AM)

I will be upgrade the IOS on Cswitch3 due to a Cisco bug that has been causing memory leaks.

 Apply Oracle CPU patch on KC-ISIA-ORAEXT1 (5/5/2017 9:00:00 PM)

Apply Oracle CPU patch on KC-ISIA-ORAEXT1

 Apply Oracle CPU patch on KC-CHI-HFPROD and KC-CHI-HFTRAIN (5/6/2017 9:00:00 PM)

Apply Oracle CPU patch on KC-CHI-HFPROD and KC-CHI-HFTRAIN. Server will be unavailable during the maintenance

 Apply Oracle CPU patch on KC-SCE-APPDB01 (5/7/2017 9:00:00 PM)

Apply Oracle CPU patch on KC-SCE-APPDB01

 Apply Oracle CPU patch on KC-ISIA-CCORM (5/7/2017 10:00:00 PM)

Apply Oracle CPU patch on KC-ISIA-CCORM

 Apply Oracle CPU patch on KC-ISIA-ORA1 (5/12/2017 9:00:00 PM)

Apply Oracle CPU patch on KC-ISIA-ORA1

 Apply Oracle CPU patch on KC-ISIA-ORA2 (5/13/2017 9:00:00 PM)

Apply Oracle CPU patch on KC-ISIA-ORA2

 Apply Oracle CPU patch on KC-ISIA-ORAPRD1 (5/14/2017 9:00:00 PM)

Apply Oracle CPU patch on KC-ISIA-ORAPRD1

 SolidWorks License Server Upgrade (5/15/2017 9:00:00 AM)

The SolidWorks License Server will be down for a short period of time in order to upgrade to the 2017 Server version.

 Dental school network upgrade (5/16/2017 5:30:00 PM)

We will be replacing the distribution switches at Dental school. This will result in a loss of phone and network during this period.

 vCenter 6.5 Upgrade (5/17/2017 8:00:00 AM)

Our vCenter instance will be upgraded to version 6.5. All VMs will continue to run during the upgrade, but vCenter will be unavailable.

 Apply Oracle CPU patch on kc-cbord-golddb (5/22/2017 7:45:00 AM)

Apply Oracle CPU patch on kc-cbord-golddb

 Unity Connection Server Ugrade (6/2/2017 5:00:00 PM)

We will be upgrading the Unity Connection, voice mail, server from version 10.5.2 to version 11.5.1