The access points oconnor-ap4000.chtc.wisc.edu, wright-ap4000.chtc.wisc.edu, and townsend-ap4000.chtc.wisc.edu are back online.
Posted May 04, 2026 - 15:47 CDT
Update
The transfer server (transfer.chtc.wisc.edu) and its Globus Endpoint are back online.
Posted May 04, 2026 - 13:10 CDT
Update
The access points ap2001 and ap2002 are back online. A significant amount of GPU and CPU capacity is also back online. Most users of the shared capacity should be able to login and run jobs. We plan to restore our remaining services early next week.
Posted May 01, 2026 - 17:18 CDT
Update
The HPC system is back online. All functionality of the HPC system should be restored.
Posted May 01, 2026 - 16:21 CDT
Update
We are making good progress in bringing CHTC services back online. We'll have another update end-of-day today.
Posted May 01, 2026 - 15:09 CDT
Update
We are implementing a plan to bring CHTC services back online. We are aiming to have some minimal functionality restored by end of day.
Posted May 01, 2026 - 11:11 CDT
Identified
CHTC services (including HTC and HPC) are offline while we patch a serious Linux vulnerability.
Last evening, a serious Linux exploit was broadly published online. This exploit allows any user to easily obtain root (admin) access, bypassing standard security controls.
We pre-emptively shut down our systems this morning to protect our system and users. We are working to patch the vulnerability on our systems. CHTC servers will remain offline through tomorrow (and possibly longer).
We have expanded the shut down to all user-facing CHTC servers. Users are unable to login and no jobs are running. This is required to ensure the safety and security of our systems. We are actively working to address the core issue.
Posted Apr 30, 2026 - 11:32 CDT
Investigating
All CHTC login nodes (spark-login, ap2001, ap2002, etc.) are being taken down.
Posted Apr 30, 2026 - 09:21 CDT
This incident affected: High Throughput Computing (HTC) System (Access Points, CHTC Pool, External Pools (OSPool, Campus HTCondor Pools)), High Performance Computing (HPC) System (Login Nodes, Cluster Nodes and Jobs), and Data Transfer Tools (Globus Endpoint).