[HTC] Failure to transfer files between jobs and ResearchDrive using the Pelican plugin

Resolved

This incident has been resolved.
Posted Jun 03, 2025 - 12:37 CDT

Update

We've put in place a hacky and rather temporary fix for the integration. That hopefully means things will work over the weekend, but until we resolve the underlying issue, this problem will reoccur sometime in the near future.
Posted May 23, 2025 - 18:17 CDT

Update

While we are still unable to identify the root cause, it appears to be originating from a combination of software updates and complex credential configurations. We are unlikely to resolve the issue before the weekend, and we will continue to provide updates next week.
Posted May 23, 2025 - 16:56 CDT

Investigating

Apparently the fix did not last for very long. Users may now encounter hold messages along the lines of "Details: Pelican Client Error: from uwdf-pelican-cache.uwdf-prod.chtc.io:8443: server returned 403 Forbidden". Good news is that this is different from before, bad news is that now we have to figure out why its different..
Posted May 20, 2025 - 15:04 CDT

Monitoring

We have implemented a fix and are monitoring the situation.
Posted May 20, 2025 - 13:37 CDT

Investigating

HTC users attempting to transfer files directly to and from their jobs from ResearchDrive using the Pelican plugin will encounter a hold message: "Details: error while querying the director at https://uwdf-director.chtc.wisc.edu: server returned 404 Not Found". We are currently investigating the problem and working on implementing a fix.
Posted May 20, 2025 - 09:53 CDT
This incident affected: High Throughput Computing (HTC) System (File Transfers).