MENU

Status Updates


UM System wide outages can be found on the UM IT System Status website.

Current Outages

Currently there are no outages.

Upcoming Outages

There are no upcoming outages.

Past Outages (Last 72 Hours)

There have been no outages over the last 72 hours.

Current and Upcoming Change Management Notices

 Cleanup of old computer objects in AD (10/22/2015 12:00:00 PM)

A cleanup of old computer objects in AD will be performed.

Any non-Mac computer object that has not had a password change in 10 months, which indicates the machine has not talked to Active Directory for at least 9 months, will be removed.

For computers that are lost, stolen, or otherwise locked away longer than 9 months, we can recover the bitlocker contents of these computers when they return to campus with a special recovery key.

 Roll out of Splunk agent to Windows servers at boot time (12/17/2015 8:00:00 AM)

Windows servers at UMKC current log to an old Symantec log collection system. Starting on the 17th, servers at boot time will uninstall the old agent used to collect these logs, and will install a new agent to send logs to our Splunk log collection system. Most Windows server systems will reboot on Monday December 21st at 2am to install patches, and will switch over at that time to the new logging agent. This is currently being testing on IS Security servers.

 Storage Array Maintenance (12/21/2015 7:00:00 AM)

We will be performing maintenance on our storage arrays. This could result in reduced performance during the process. Our DR array will be worked on first, followed by the production array.

 Update to Microsoft encryption cipher suite order (12/21/2015 8:00:00 AM)

Microsoft released some additional encryption cipher suites for Windows. Since we manually manage the cipher suite order list, we must manually add these to campus machines.

In order to facilitate this addition, a change will be made to client computers to allow the new cipher suites.

The cipher suites are briefly described at:
https://technet.microsoft.com/en-us/library/security/3042058.aspx

This will be put in for new bootups after Monday morning December 21st.

This has been initially tested, and no side-effects have been seen.

This will enhance the cipher list for Windows clients and Windows servers.

 vCenter Patching (12/23/2015 7:00:00 AM)

We will be applying patches to our vCenter instance. VMs will be unaffected, but no changes to existing VMs will be possible during the upgrade process.

 Campus edge firewall firmware upgrade (1/9/2016 10:00:00 PM)

The firmware on the campus edge firewall will be upgraded to a new feature set and update version. This is a rolling upgrade, with two expected 20-second Internet outages as the live unit reboots, and during fail-back to live. (secondary unit will be rebooted first to avoid more outages) There is a risk that the upgrade will have complications resulting in a longer outage.

 Set IPv6 enabled servers to use persistent IPv6 addresses (2/19/2016 8:00:00 AM)

A bootup script change will be made, so that IPv6 enabled Windows servers keep the same IPv6 address for the life of the installed OS on their particular virtual machine or physical machine. The specific change is to run these two commands at bootup time:

netsh interface ipv6 set global randomizeidentifiers=disabled
netsh interface ipv6 set privacy state=disabled

which disables two IPv6 privacy functions that can randomize the IPv6 address per day and at each bootup.

This will ensure that campus edge firewall rules will correctly point to the same IPv6 address between subsequent reboots.

This will be made active in group policy on February the 19th, for any reboots that day or later. The campus wide patch installation day is February 22nd, so most Windows servers would pickup the change on that day after installing patches.

This will be activated on IS Security servers now, with actual activation next week when patches are released to pre-deployment testing, or sooner if a particular IS Security server needs a reboot.

This should have no significant impact other than stabilizing IPv6 addresses for Windows servers.

 Emergency campus edge firewall firmware upgrade (2/27/2016 10:00:00 PM)

Due to a pair of vulnerabilities patched this week that will soon be disclosed, we will be doing an emergency upgrade of the campus edge firewalls. The firewalls are setup in a redundant manner, so there should only be two brief 10-second outages as each of the two units is updated.

The brief outage should generally not be noticeable. This will affect inbound traffic to servers on campus from off campus users, and for on campus users accessing content on the Internet.

 Change to set Outlook to consistently use Cached Mode (3/11/2016 8:00:00 AM)

A change in group policy will be applied, that will set Outlook to use Cached Mode consistently. This change helps reduce certain stresses on the Exchange server that live mode has been causing across the UM-System campuses.

This change will be applied on Information Services computers first, and will later be extended to other departments on campus. As the change causes live Outlook to download full mailboxes to switch to cached mode, this change will be rolled out department by department to reduce the impact of a sudden change.

 Windows 10 change to use WSUS (3/22/2016 8:00:00 AM)

Windows 10 systems will be set to use the campus WSUS server for updates in late March. This will ensure that we can remove bad updates, such as a problematic update for Microsoft Office that caused Office 2013 to freeze on Windows 10 machines.

 A change will be made to group policy, to remove DHE ciphers from Windows systems on campus. (3/26/2016 10:00:00 PM)

Of the long list of supported encryption ciphers, the two allowed DHE ciphers are now a concern. Microsoft has a lower DHE bit security level, and so far a way to increase this to a stronger bit level has not been found. So removal of the two DHE ciphers is best. The two ciphers that will be removed are:
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256

As this is a group policy change, it will not take full effect on Windows Servers until April 25th at 2am when the April WSUS updates are applied and systems rebooted. Workstations will have the change on March 28th at 2am as they reboot weekly.

Linux/Unix and Apache/Tomcat administrators are encouraged to retest their sites using: https://www.ssllabs.com/ssltest/index.html and make adjustments to ensure their SSL protection passes at a Grade-A. These systems are each controlled by local settings files, and cannot be centrally managed to use new recommended settings.

 IP address change for KC-DC06 (3/27/2016 7:00:00 AM)

An IP address change will occur on KC-DC06 due to a need to retire a VLAN in Columbia. The new IP address will become 198.209.56.232, and the old retiring IP is 209.106.228.180 . Only Exchange should be affected by this as it uses this server for pulling UMKC data. Other systems should be using DNS to find Active Directory and should update automatically.

 Group policy bootup script to backup drive encryption keys (3/30/2016 12:00:00 PM)

A bootup script is being put into place, to ensure that the bitlocker encryption keys for a machine are properly backed up into Active Directory. Normally this process occurs automatically at the time the drive is encrypted, but we are seeing some corner cases where the backup may fail. This backup is necessary to recover a drive, such as a bad patch that makes a machine unbootable, the key would be necessary to restore the machine to a usable state.

 Yearly password change for PassTheHash protection (4/2/2016 10:00:00 PM)

The krbtgt account will get a yearly password change to protect against certain versions of PassTheHash type attacks.
This was last changed a year ago, and no ill effects occurred.

 Addition of new primary Windows Update server for UMKC (4/5/2016 8:00:00 AM)

Computers at UMKC will be moved from the current WSUS server to a new server at KC-IS-WSUS4. This will reduce the load on the primary management WSUS server. This change will occur between update cycles, and should not otherwise directly impact users or updates. The new server will be setup before the change, and clients will receive the new server name starting April 5th.

 Increase SSL security on Active Directory to TLS 1.0 minimum (4/9/2016 8:00:00 AM)

Active Directory SSL will be increased from a minimum of SSLv3 to TLS 1.0. This will match the requirements of other servers and workstations on campus. The DCs were missed in the last encryption improvement round, and will be brought into compliance with the rest of the campus encryption.

This may affect old LDAP clients such as Oracle and ColdFusion that talk directly to Active Directory, but which should be using TLS encryption. These products should all be up to date, but will be monitored to see if they have any issues with the change.

 Data Center Firewall Change (4/9/2016 10:00:00 PM)

A change will be made to the main data center network, and the firewall for this network. This will result in a likely 2 hour outage on the night of Saturday April 9th. Routing changes will also occur with this change to more successfully apply this change.

Servers on campus will be unreachable at this time, email will still function, but users on campus may have connectivity issues reaching the Internet during this time window.

 Urgent routing change for campus data center (4/14/2016 9:00:00 PM)

Due to some routing loops being seen on campus from the recent data center firewall work, changes will be made to the data center routing. This is likely to cause some 1-minute long network outages as adjustments are made to correct the routing errors. Currently some locations on campus are having problems reaching other areas of campus with this problem active, so it is urgent to correct this routing as quickly as possible.

 Data Center Firewall OS rebuild (4/24/2016 11:00:00 PM)

Due to error found in the OS installation on the Data Center firewalls, the OS will be rebuilt on each. As these units are redundant to each other, the work will be done on each unit while it is set passive or offline, allowing the online unit to remain undisturbed. (then swapping which unit is active to get the peer device up to date)

This should only result in two 1 minute outages as the units switch between active and passive.

There are currently 3 issues on the firewalls we are attempting to fix, this is the known fix for one of the problems. The other two are suspected to be related to this problem, and we are hoping this will resolve these other two issues.

This will be done overnight to avoid disturbing any production traffic during the brief active/passive transition periods.

This process is being tested ahead of time on a stand-alone unit with the same software level as the current units.

 Upgrade to EMET 5.5 (5/16/2016 8:00:00 AM)

The EMET software used to protect campus Windows machines, will be upgraded to version 5.5. This will include additional feature capabilities. This will be released after 8am, so that only machines rebooted after that time will get the update, allowing a controlled roll out over the week.

Information on EMET 5.5 is available at:
https://blogs.technet.microsoft.com/srd/2016/02/02/enhanced-mitigation-experience-toolkit-emet-version-5-5-is-now-available/

Testing will be conducted ahead of time on the upgrade and new feature implementations.

 Enable DNSsec on UMKC managed DNS zones (5/16/2016 12:00:00 PM)

DNSsec signing will be enabled on DNS zones managed by UMKC. This is the first of two parts to better ensuring UMKC DNS information is not tampered with by end ISPs. The second step is to publish the signing key with our registrars, so that outside sites can validate the signing. This will take place at a later date after we confirm the signing is working. (second step can result in loss of name resolution if the first step has any errors)

 Disable IPv6 Source Routing capability on Windows OS devices (5/17/2016 8:00:00 AM)

IPv6 Source Routing is likely enabled by default on Windows OS devices. This will be disabled via a group policy registry key. This will only take effect after a reboot, after the group policy change is put into place. Source Routing is normally only used for testing and diagnostics, and should not be used for production, and can be abused by attackers trying to get around network security devices.

 Emergency data center firewall reboot (5/17/2016 10:00:00 PM)

Due to performance issues seen on the data center firewall, a reboot will be performed tonight between 10pm and 11pm to clear the data plane CPU. This will result in two 1 minute outages as we fail over the firewalls for the reboot.

We are also working to identify high bandwidth uses of data center resources, and working to set reasonable limits on those systems, to ensure enough bandwidth is available for all other users.

 DNSsec public keys will be posted to DNS domain name registrars (5/20/2016 8:00:00 AM)

Once the initial DNSsec signing has been validated, the DNSsec public keys will be posted to our DNS registrars. At that point enforcement of the signing will take effect. This will be done on May 20th, a few days after the initial signing has started, and once the signing has been validated.

 Domain controller security change (5/25/2016 5:30:00 AM)

A security change will be made between UMKC's domain controller and the Active Directory root. This change is necessary to eliminate the use of RC4 ciphers between domains. This will enable Kerberos tickets that cross domains to use AES encryption.

This change should not cause any outages, and should strengthen the signing of Kerberos tickets.

This is a necessary step to re-attempting to disable RC4 Kerberos ticket signing at UMKC. When we disabled RC4 Kerberos ticket signing, any unauthorized Windows 2003 and XP machines will stop talking to Active Directory.

 Proactive Data Center Firewall reboot (6/4/2016 10:00:00 PM)

In order to avoid any issues that may have to do with long uptime (potential memory or process leaks), a Data Center firewall reboot will be performed roughly every 21 days on a weekend during the network maintenance time window. These firewalls participate in active routing, so even with a failover, there is a 1 minute network outage as routing tables sync up.

 Change in Windows 7 and Windows servers to prevent caching passwords in memory (6/23/2016 8:00:00 AM)

(Note adjusting to 23rd from 22nd due to a calendar issue) A change will be applied as per Microsoft KB: https://support.microsoft.com/en-us/kb/2871997 to prevent storing password in memory. This change already exists on Windows 10 machines deployed widely on campus, and Windows 2012 R2 machines. This will bring Windows 7 and Windows 2008/2008R2/2012 servers up to the same level of security. Since this is already running on newer machines successfully, this should not be a major problem.

 Proactive Data Center Firewall reboot (6/25/2016 10:00:00 PM)

In order to avoid any issues that may have to do with long uptime (potential memory or process leaks), a Data Center firewall reboot will be performed roughly every 21 days on a weekend during the network maintenance time window. These firewalls participate in active routing, so even with a failover, there is a 1 minute network outage as routing tables sync up.

 Campus WiFi controller update (6/26/2016 11:45:00 PM)

Wireless will be down while the controllers reboot a few times as part of applying their new code and the APs will reboot prior to rejoining the controller. The Access Points will not be broadcasting nor allowing any WiFi traffic for a period of time that is usually no greater than an hour. All wired traffic should remain unaffected.

 Emergency Database Maintenance on kc-chi-hfprod (7/6/2016 9:00:00 PM)

MEDRESRCH database will be unavailable for Emergency Database maintenance. Need to increase no of max processes/sessions on chihfprd database and it will be unavailable during the maintenance.

 Proactive Data Center Firewall reboot (7/9/2016 10:00:00 PM)

In order to avoid any issues that may have to do with long uptime (potential memory or process leaks), a Data Center firewall reboot will be performed roughly every 21 days on a weekend during the network maintenance time window. These firewalls participate in active routing, so even with a failover, there is a 1 minute network outage as routing tables sync up.

 Enabling Flashback on KC-ISIA-ORAPRD1 (7/15/2016 12:30:00 AM)

KC-ISIA-ORAPRD1 will be down for like 10 minutes during this time.

 Emergency reboot of campus edge firewalls tonight (7/18/2016 10:00:00 PM)

Due to a problem on the campus edge firewalls, a reboot is needed to reset their licensing status. This should not result in any traffic outage due to the redundant setup of the firewalls, but there is a chance of traffic problems with any restart.

 Demotion and removal of KC-DC03 (7/19/2016 8:00:00 AM)

The Domain Controller KC-DC03 will be demoted and taken out of service. The location this DC is currently in, will no longer be used for data center services. This still leaves 3 Domain Controllers on campus, and two off campus for DR and Exchange Support purposes. Systems should already be using the generic names for finding Active Directory services, so this should be a minor change.

 Shutdown of 134.193.159.52 and 2610:e0:a040:68fd::2 for DNS (7/20/2016 8:00:00 AM)

The IP addresses of 134.193.159.52 and 2610:e0:a040:68fd::2 will be taken out of service as a valid client DNS servers at UMKC. The location this DNS server is in, will no longer be used as a Data Center, and the subnet is not a portable subnet, so it cannot be moved to a new location to keep these IP addresses active.

The remaining client DNS IP addresses are:
134.193.1.2
134.193.83.4
2610:e0:a040:64fc::2
2610:e0:a040:6401::2

Any system manually setting DNS servers, should ensure they are only using the above remaining IP addresses. The DNS server will also be monitored to identify any systems still using the old settings. DHCP enabled machines should already be using the new settings.

 Proactive Data Center Firewall reboot (7/23/2016 10:00:00 PM)

In order to avoid any issues that may have to do with long uptime (potential memory or process leaks), a Data Center firewall reboot will be performed roughly every 21 days on a weekend during the network maintenance time window. These firewalls participate in active routing, so even with a failover, there is a 1 minute network outage as routing tables sync up.

 Starfish Connect Production Instance is being upgraded to version 6.4 (8/6/2016 1:15:00 AM)

The Starfish Connect PRD server is being upgraded to version 6.4 this weekend. This is a SAAS solution and is not an opt-in/opt-out type of upgrade. This CM notice is being posted simply to record this change.

 Upgrade campus data center boundary firewalls and Internet edge firewalls to current software level (8/6/2016 10:00:00 PM)

The data center boundary firewall pair and the Internet edge firewalls will be upgraded to the current software levels. This will hopefully help with resource issues seen on the data center firewalls, but also to ensure current levels of fixes. At this time we will stay in the same software major version and only update to its current fixes version. A major version upgrade will be done later when more time can be planned out on the upgrade.

For the Data Center, there will be two 1-minute outages as the firewall pair is involved in OSPF routing that is slow to converge properly on campus.

For the campus edge pair, there should not be any noticable outage as the failover is very fast due to static routing.

 Adding UserTrust Certifictes to Call Manager, IM&P and Unity Connection. (8/21/2016 7:30:00 AM)

We will be adding a new UserTrust Certificate to Call Manager, IM&P and Unity Connection to resolve a certificate trust issue for Jabber clients. .

 dropbox.umkc.edu update (9/2/2016 10:00:00 PM)

Dropbox.umkc.edu service will be offline September 2, 2016 Friday night for a system update.

 Law School Network Upgrade (9/12/2016 8:30:00 PM)

We will be upgrading the access and distribution switches at Law school. This will result in a loss of phone and network capability during this time.

 Upgrade PaloAlto firewalls to 7.1 software train (12/17/2016 10:00:00 PM)

At the start of Winter Intersession, the campus PaloAlto firewalls will be upgraded to the 7.1 firmware train. The final stabilizing builds will occur before this date, allowing us to make use of the new feature sets with reduced impact from new firmware bugs. Confirmations backups will be made ahead of time to help with migrating backwards if major problems arise. Since this is a major upgrade, an outage is likely during the upgrade process of up to an hour for the campus edge and data center.