Identified - Some jobs running on gpulab2001 or gpulab2003 may fail with an error "CUDA error: failed call to cuInit: CUDA_ERROR_UNKNOWN". We are working to resolve the issue.
Jun 02, 2026 - 16:53 CDT
The issue was caused by a highly specific and rare bug. We are working to address the underlying cause.
Jun 1, 11:27 CDT
Monitoring -
The issue has been fixed, at least temporarily. We're still not clear on the cause, though, so there is a chance it may reoccur over the weekend.
May 29, 14:13 CDT
Update -
We are still investigating this issue. Our attempted fix did not work so we need to dig deeper into the cause.
May 29, 11:32 CDT
Investigating -
The HTCondor queue on ap2001 is currently down. If you run a command like `condor_q` or `condor_submit`, you'll see a message like:
> Error: Can't find address for schedd ap2001.chtc.wisc.edu
> ERROR: Can't find address of local schedd
We are investigating why the queue is down.
May 29, 08:04 CDT
Completed -
The scheduled maintenance has been completed.
May 27, 18:00 CDT
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 27, 14:00 CDT
Scheduled -
We are bringing ap2001, ap2002, learn, and researcher-owned APs into downtime to apply important updates to the kernel, continuing this morning's maintenance event.
Users will not be able to log in or submit jobs during the downtime.
May 27, 13:43 CDT
Resolved -
This incident has been resolved.
May 22, 10:57 CDT
Update -
We are continuing to monitor for any further issues.
May 22, 10:57 CDT
Monitoring -
We've implemented a fix and are monitoring.
We identified an issue with authentication that was introduced as a side-effect of our recovery from last weekend's outage.
May 21, 15:54 CDT
Investigating -
There is an issue with uploading files to /staging via the OSDF (osdf:///chtc/staging/..). Jobs uploading to /staging via the OSDF will encounter the error "Transfer output files failure ... server returned 403 Forbidden ... ".
Resolved -
This incident has been resolved.
May 22, 10:00 CDT
Monitoring -
A fix has been implemented and we are monitoring the results.
May 21, 10:24 CDT
Investigating -
Any jobs or processes fetching Docker images may fail, with error messages like "Error response from daemon: toomanyrequests: You have reached your unauthenticated pull rate limit." or "While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: ... unexpected status code 500 Internal Server Error"
Resolved -
This incident has been resolved.
May 20, 08:57 CDT
Identified -
Jobs using "pelican://" links to transfer files from Research Drive are failing. Affected jobs, may see jobs go on hold with an error message like:
"Transfer input files failure at execution point ... using protocol pelican. Details: failed to get namespace information for remote URL pelican://chtc.wisc.edu/researchdrive/...: error while querying the director at https://uwdf-director.chtc.wisc.edu: Contact.Director Error: Error code 3001: 404: No sources found for the requested path: no origins found for the requested namespace '/researchdrive/..."
The service that enables the file transfer is currently unavailable. Our team is working to bring it back up. We will post an update to this page once the issue is resolved.
May 19, 11:59 CDT