Oct 21, 2026

ICER will be conducting rolling patches and reboots of gateway, development, and compute nodes starting Monday, October 26, 2026. These rolling reboots may impact access to individual gateway/development nodes and increase queue times as individual nodes are scheduled for reboot as this will reduce the total number of nodes available for running jobs. This process will not affect running jobs on the compute nodes, but open sessions on gateways and development nodes will be closed when rebooted. We will post updates to this blog when the process is complete.

Sep 11, 2026

UPDATE: 10/21: Our software vendor has reproduced the issue and developed a patch. We will apply this patch once we complete internal testing.

UPDATE: 10/7: Some other types of job requests were found to still trigger this preemption issue. We have implemented a workaround to ensure scavenger jobs are preempted properly while we work with our software vendor to address the underlying issue.

RESOLVED: 9/22: The scavenger queue is back online. We identified the job requests that trigger the preemption issue and implemented scheduler logic to account for them.

UPDATE: 9/18: We are working with our software vendor to resolve this issue.

We have temporarily paused running jobs through the scavenger queue while we investigate a potential issue with job preemption. If you have any questions, please contact us.

Oct 13, 2026

ICER/HPCC Access and MSU Password Reset Information

The University is requiring a password reset for all accounts accessing MSU systems. If you are having issues accessing ICER/HPCC when using a password to authenticate and have not changed it recently, you will need to reset your password by visiting https://auth.msu.edu and following the procedure presented for your NetID. For those users with an FPID account, if you have issues resetting your password through https://auth.msu.edu, you may need to reactivate your FPID using your PIN, which will reset your password. The MSU IT password management help page is here: https://netid.msu.edu/password/index.html, they would be able to assist with any password reset issues.

HPCC Staff

Sep 30, 2026

UPDATE: 10/27/26 - The failing hardware has been replaced successfully. We are continuing to troubleshoot the IO error root cause with our storage vendor.

UPDATE: 10/26/26 - A failing hardware component has been identified within the research filesystem while troubleshooting the reported IO errors. That component is scheduled to be replaced by our storage vendor the morning of Tuesday, October 27, 2026. No impact to users or workflows is anticipated, and this blog will be updated as soon as the new hardware is installed.

Following the power outage, ICER is aware of some users seeing IO errors when accessing some files on the research filesystem. We are actively working with our storage vendor to resolve these errors, and will provide an update as soon as we are able.