Investigating - Any jobs or processes fetching Docker images may fail, with error messages like "Error response from daemon: toomanyrequests: You have reached your unauthenticated pull rate limit." or "While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: ... unexpected status code 500 Internal Server Error"
We are currently investigating.
May 20, 2026 - 12:29 CDT
Resolved -
This incident has been resolved.
May 20, 08:57 CDT
Identified -
Jobs using "pelican://" links to transfer files from Research Drive are failing. Affected jobs, may see jobs go on hold with an error message like:
"Transfer input files failure at execution point ... using protocol pelican. Details: failed to get namespace information for remote URL pelican://chtc.wisc.edu/researchdrive/...: error while querying the director at https://uwdf-director.chtc.wisc.edu: Contact.Director Error: Error code 3001: 404: No sources found for the requested path: no origins found for the requested namespace '/researchdrive/..."
The service that enables the file transfer is currently unavailable. Our team is working to bring it back up. We will post an update to this page once the issue is resolved.
May 19, 11:59 CDT
Resolved -
We have addressed the issue for now. As a reminder, CHTC is not a storage service. We do not backup your data or files. Make sure you have the space to backup all of your data and a process for regularly removing stale data from CHTC.
May 19, 11:54 CDT
Investigating -
CHTC is currently experiencing limited availability in /home storage. As a result, we are unable to approve /home quota increase requests at this time.
Our team is actively working on a longer-term storage solution and will provide updates as more information becomes available.
In the meantime, users should reduce /home usage where possible: - Move individual files larger than 1 GB to /staging. - For directories containing many small files, bundle them into a compressed archive before moving them to /staging.
For example: tar -czvf my_directory.tar.gz my_directory/ mv my_directory.tar.gz /staging//
After confirming the archive was created and moved successfully, you may remove the original directory from /home if it is no longer needed there.
Thank you for your patience while we work to expand available storage capacity.
Apr 24, 11:25 CDT
Resolved -
This incident has been resolved.
May 18, 16:08 CDT
Monitoring -
We have implemented a fix. Please release any held jobs affected by this incident with `condor_release `. We will continue to monitor the situation.
May 18, 14:45 CDT
Investigating -
Jobs using the OSDF file transfer protocol are failing with the following error: "error while querying the director at https://osdf-director.osg-htc.org: Contact.Director Error: Error code 3001: 404: No sources found for the requested path: no origins found for the requested namespace".
This error is due to a facilities outage. We are working to bring the system back online.
May 18, 09:29 CDT
Resolved -
Both the HPC and HTC systems are fully operational again
May 16, 12:43 CDT
Investigating -
Parts of the HTC and HPC systems are experiencing an outage due to overnight storms causing power and cooling outages.
May 16, 11:01 CDT
Resolved -
Fix has been deployed. Users should be able to write to their /staging/groups directories as normal.
May 14, 12:03 CDT
Identified -
Confirmed user reports of "Disk quota exceeded" errors for locations in /staging/groups, even though the "get_quotas" command shows usage below the quota limit.
We've identified the cause and are working on implementing a fix.
May 14, 11:55 CDT
Resolved -
We have finished patching, and both the HTC and HPC systems are operational.
May 8, 10:55 CDT
Identified -
We will be rebooting the Access Points/login nodes and temporarily blocking job-matching to implement a critical patch. SSH sessions will be interrupted. Running jobs should not be affected.