Many jobs failing with Error: Cannot open GBW file: input.gbw

[jtvaughn@albion.edu]

Hello Webmo Team, My apologies if this post is not in the correct forum, but I was hoping to get some help with running jobs on our server. Our server specs are: running Windows Server 2016 latest patches, Orca 4.2.1 and WebMo v20 and we are having some strange issues that we did not have in Webmo v17 and previous versions of Orca. When we run a single job, it almost always completes correctly, however when we have a group of students who queue up maybe 4+ jobs, the jobs do not complete and show this error in the output:


!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!! FATAL ERROR ENCOUNTERED !!!
!!! ----------------------- !!!
!!! I/O OPERATION FAILED !!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

ORCA finished by error termination in SCF
Calling Command: c:\orca\orca_scf input.gbw b
[file orca_tools/qcmsg.cpp, line 458]:
.... aborting the run

[file orca_tools/qcmsg.cpp, line 458]:
.... aborting the run

Executing command: c:\orca\orca.exe input.inp
Reading the input file input.gbw ... [file orca_2mkl/orca_2mkl.cpp, line 77]: Error: Cannot open GBW file: input.gbw

[file orca_2mkl/orca_2mkl.cpp, line 77]: Error: Cannot open GBW file: input.gbw

The system cannot find the file specified.
A duplicate file name exists, or the file
cannot be found.

[schmidt]

I will likely need more information. The tail end of that input fail (aside for the "I/O error" itself) is very generic in terms of a failure message. This happens essentially anytime ORCA encounters an error, even an input error.

Have you tried re-running the IDENTICAL input file again as an isolated job? have you found cases where an isolated job also fails?

I just tested this on my Win 7 / WebMO 20 setup with many simultaneous OCA jobs and no issues. It would thus be very helpful to try and narrow down the set of circumstances so that I can effectively reproduce the issue.

[jtvaughn@albion.edu]

Our Webmo server is a Virtual Machine with 12 vCPU and 16GB of memory. Our Main Professor who uses the server (apologies, I work in IT and handle the hardware and server stuff so I'm not very familiar with the apps) sent me the full output file shared via google drive below. He said he ran the c2h6 job from his Webmo account and it completed, it's just that when we load up many jobs is when it starts to fail.


https://drive.google.com/file/d/1QFAmGP ... sp=sharing


This is what happens when we run a single job, there is still that mention about the file not found at the bottom. I also attached a screenshot of the job manager.

****ORCA TERMINATED NORMALLY****
TOTAL RUN TIME: 0 days 0 hours 0 minutes 23 seconds 913 msec
Executing command: c:\orca\orca.exe input.inp
Reading the input file input.gbw ... done
Writing the output file input.molden.input ... done
1 file(s) moved.
A duplicate file name exists, or the file
cannot be found.

[jtvaughn@albion.edu]

Below is the full output file from the most recent failed job. It's strange that it says "Executing command: c:\orca\orca.exe input.inp" but the input file is called input.gbw


https://drive.google.com/file/d/1r0-zl3 ... sp=sharing


5191 C26H48O2 opt 4 Geometry Optimization - ORCA cbieler webmo 3/25/2021 15:31 Failed 0.0 sec


With this error at the bottomw:

------------------
INITIAL GUESS DONE ( 4.5 sec)
------------------
[file orca_tools/Tool-Misc/qccore.cpp, line 131]: Error (ORCA_MDCI): cannot retrieve the overlap matrix

[file orca_tools/Tool-Misc/qccore.cpp, line 131]: Error (ORCA_MDCI): cannot retrieve the overlap matrix


ORCA finished by error termination in SCF
Calling Command: c:\orca\orca_scf input.gbw b
[file orca_tools/qcmsg.cpp, line 458]:
.... aborting the run

[file orca_tools/qcmsg.cpp, line 458]:
.... aborting the run

Executing command: c:\orca\orca.exe input.inp
Reading the input file input.gbw ... [file orca_2mkl/orca_2mkl.cpp, line 77]: Error: Cannot open GBW file: input.gbw

[file orca_2mkl/orca_2mkl.cpp, line 77]: Error: Cannot open GBW file: input.gbw

The system cannot find the file specified.
A duplicate file name exists, or the file
cannot be found.

[jtvaughn@albion.edu]

Here is the input file for the c2h6 job

[cbieler]

A few more pieces of information that may be helpful.

* multiple jobs run in parallel without a problem until one of the jobs finishes. After that the other jobs Fail with some variation on the file read errors shown in previous posts.

* a single job run via WebMO appears to perform properly and returns information to WebMO. However, the curious file error appears at the bottom of the raw output file.

****ORCA TERMINATED NORMALLY****
TOTAL RUN TIME: 0 days 0 hours 0 minutes 23 seconds 913 msec
Executing command: c:\orca\orca.exe input.inp
Reading the input file input.gbw ... done
Writing the output file input.molden.input ... done
1 file(s) moved.
A duplicate file name exists, or the file
cannot be found.

This last sentence is new since the upgrade to v19 (and v20) and ORCA 4.2.1

[schmidt]

You might try opening up the file "housekeeping.cgi" and commenting out all the lines in the subroutine 'delete_scratch_files' (just add a '#' to the beginning of the corresponding lines). My only crazy hypothesis is that the scratch files for a running job are being deleted erroneously under some point, despite the fact that this does not occur in my testing (or ever having occurred in the past). This is the only rational explanation for why this would fail under load but not for the isolated job.

If that is the case, it may have to due to the strange way ORCA spawns jobs on Windows, which might make WebMO think a job has finished, when in fact it has not.

[jtvaughn@albion.edu]

Big thanks for the recommendation, we will give that a shot once students are done running simulations.

[jtvaughn@albion.edu]

Hi schmidt, Thanks again for all the of the helps. I was not able to locate the housekeeping.cgi file on our windows server. Is that a webmo or orca file?

[jtvaughn@albion.edu]

Oh my mistake, I found it. It's in the apache folder apparently