Identified - The Gurobi license for CHTC has expired. We are working with campus IT to renew the license. In the meantime, user jobs attempting to use the Gurobi license will likely fail due to a "license expired" error.
Apr 02, 2026 - 15:13 CDT
Resolved -
This incident has been resolved.
Apr 2, 10:37 CDT
Monitoring -
We identified the cause of the problem and have applied a fix. Initial tests appear successful. Let us know at chtc@cs.wisc.edu if you continue to encounter problems.
Apr 2, 09:28 CDT
Investigating -
Confirmed user reports that job submission on ap2002.chtc.wisc.edu is failing. We are investigating the issue and will provide updates as they become available.
Apr 2, 09:09 CDT
Completed -
The scheduled maintenance has been completed.
Mar 31, 17:45 CDT
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 31, 13:45 CDT
Scheduled -
Network infrastructure is being updated in one of our server rooms. Jobs will be much slower to match, due to reduced capacity during this time. Jobs already running on hosts in this server room may fail due to timeouts. Any ongoing filetransfer or interactive sessions to these nodes will also be disrupted. Nodes impacted: E40XX, build4001, cxiaogpu4000, xhuangpgu4000
Mar 31, 13:37 CDT
Completed -
The scheduled maintenance has been completed.
Mar 31, 13:00 CDT
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 31, 07:00 CDT
Scheduled -
Network infrastructure is being updated in one of our server rooms. Jobs will be much slower to match, due to reduced capacity during this time. Jobs already running on hosts in this server room may fail due to timeouts. Any ongoing filetransfer or interactive sessions to these nodes will also be disrupted. Nodes impacted: E40XX, build4001, cxiaogpu4000, xhuangpgu4000
Mar 27, 10:41 CDT
Resolved -
This incident has been resolved.
Mar 31, 11:38 CDT
Monitoring -
A fix has been implemented and we are monitoring the results.
Mar 31, 10:26 CDT
Investigating -
Users reported being unable to access or log in to spark-login.chtc.wisc.edu since 7 PM on 3/30/2026. We are currently investigating.
Mar 31, 09:23 CDT
Resolved -
Looks back to normal.
Mar 20, 13:09 CDT
Monitoring -
We believe we've fixed the problem. We'll keep an eye on things to make sure it sticks.
Mar 20, 09:04 CDT
Identified -
A system-wide update to our GPU machines has resulted in a mixup in the configuration of GPU machines prioritized for researchers. We are working to address the problem. In the meantime, you may see reduced or unusually matchmaking behavior for GPU jobs that target prioritized machines, including backfill jobs.
Mar 19, 12:34 CDT