Major CHTC outage

Incident Report for Center for High Throughput Computing

Identified

CHTC services (including HTC and HPC) are offline while we patch a serious Linux vulnerability.

Last evening, a serious Linux exploit was broadly published online. This exploit allows any user to easily obtain root (admin) access, bypassing standard security controls.

We pre-emptively shut down our systems this morning to protect our system and users. We are working to patch the vulnerability on our systems. CHTC servers will remain offline through tomorrow (and possibly longer).

We will share more updates as the situation changes. We will provide the most up-to-date status of the system on our status page: https://status.chtc.wisc.edu/incidents/8zycydmtkf74.

Posted Apr 30, 2026 - 16:30 CDT

Update

We have expanded the shut down to all user-facing CHTC servers. Users are unable to login and no jobs are running.
This is required to ensure the safety and security of our systems.
We are actively working to address the core issue.

Posted Apr 30, 2026 - 11:32 CDT

Investigating

All CHTC login nodes (spark-login, ap2001, ap2002, etc.) are being taken down.

Posted Apr 30, 2026 - 09:21 CDT

This incident affects: High Throughput Computing (HTC) System (Access Points, CHTC Pool, External Pools (OSPool, Campus HTCondor Pools)), High Performance Computing (HPC) System (Login Nodes, Cluster Nodes and Jobs), and Data Transfer Tools (Globus Endpoint).