We've deployed some changes to our system that should resolve or at least mitigate the rate of the occurrence of the issue. Please let us know if you encounter the "pull rate limit exceeded" message by emailing us at chtc@cs.wisc.edu .
Posted Jan 17, 2025 - 09:29 CST
Update
Sporadic reports of the "pull rate limit exceeded" hold message for Docker jobs have continued throughout the past week. "Trying again later" is still the best workaround at this time.
To avoid pulling Docker containers at all, consider creating an Apptainer .sif image of the container, as described in our guide here: https://chtc.cs.wisc.edu/uw-research-computing/htc-docker-to-apptainer. You may still run into the error when building the .sif image, but once built, you can use the image as the software environment for your jobs in a similar manner to Docker containers. Except that the .sif image is a standalone file you place in your /staging directory and so you do not have to pull from DockerHub, thus avoiding their pull rate limits.
We are unlikely to resolve this issue until after the holidays, so until we say otherwise, you may assume that this is issue is still ongoing.
Posted Dec 19, 2024 - 17:09 CST
Update
This afternoon, there have been multiple user reports of their Docker jobs going on hold with the "pull rate limit exceeded" message. "Trying again later" is still the best workaround at this time.
Posted Dec 09, 2024 - 16:55 CST
Update
This issue seems to be intermittent in nature. We have not identified the root cause yet, but are still investigating. In the meantime, trying again later seems to be the workaround.
Posted Dec 03, 2024 - 12:22 CST
Investigating
We are encountering errors on the HTC system about the Docker pull rate limit being exceeded. Jobs may be going on hold with such an error message. We are investigating the issue. Note that our system is set up such that this shouldn't be an issue, and it has nothing to do with your account at CHTC or on DockerHub.
Posted Dec 02, 2024 - 16:13 CST
This incident affected: High Throughput Computing (HTC) System (CHTC Pool).