Identified - We've confirmed several reports of slow performance of /staging, /project directories. Users may encounter slow file transfers to/from /staging, /projects, and commands that query files in those directories may be slow or hang up entirely.
This issue is related to heavy disk usage in these spaces as a side effect of the ongoing data recovery process. Unfortunately there is not a good workaround at this time, but users are encouraged to move or remove any recovered data (located in /recovery).
We ask that users be patient while we work to resolve this issue.
Jan 23, 2025 - 16:59 CST
Monitoring - The data recovery process for /projects is complete. We believe we have recovered close to 100% of the data that was originally present in these directories. Some of the metadata for files (like file creation date) may be incorrect; we strongly recommend validating any data that you copy from the recovered file system.
Update - The data recovery process for /staging and HTC /software is complete. We believe we have recovered about 50% of the data that was originally present in these directories. Some of the metadata for files (like file creation date) may be incorrect; we strongly recommend validating any data that you copy from the recovered file system.
Update - We are nearly finished recovering data from the /staging directory. We will provide more information in the next day or so as we confirm the recovery process. We are still working on recovering data from the /projects directory and anticipate it will be several weeks before it is ready for users to access.
Dec 10, 2024 - 15:48 CST
Update - We have created new /staging, /projects and /software data spaces. Please email us if you need your group /staging directories, /projects, or /software directories re-created. If any aspects of your jobs relied on these directories and you are currently having issues running jobs, contact us at chtc@cs.wisc.edu.
Dec 06, 2024 - 11:15 CST
Update - All HTC users should now have access to a new staging directory with a default quota of 100GB / 1000 items. This space can be used exactly like the previous /staging directories to run jobs.
Identified - We have identified the issue that was causing file system problems on Thursday. We are able to prevent it from recurring; however, it resulted in significant data loss in /staging, /projects, HTC /software and /squid before CHTC personnel were able to react.
All data in /squid is unrecoverable. Any remaining data in /projects and /staging is currently inaccessible as we work to recover whatever additional data we can. We hope to recover at least 50% of /staging and 60% of /projects.
This week (Nov 25-27), we will create a new data store to serve the “/staging” and “/projects” directories. Initially, there will be no data inside these directories. This new data backend for the /staging and /projects directories will be used for CHTC data storage moving forward and will be usable in jobs once available. We will post on this status page when these directories are available.
Resolved -
This incident has been resolved.
Mar 14, 15:49 CDT
Monitoring -
A fix has been implemented and we are monitoring the results.
Mar 13, 12:54 CDT
Identified -
We have received error reports from users that run jobs with licensed software that rely on license servers hosted at CHTC. These errors include messages such as "Failed to connect to license server", or other connection-related errors. We have identified the underlying issue and are working on a fix.
Mar 13, 10:51 CDT