Just a simple question I don’t understand. I have several experience in PSI4 that the calculation died if I assigned “too much” memory. For example, in a EOM-CC2 calculation, everything is the same but a calculation with “memory 56 Gb” died after running a while, however it finished successfully if I changed it to “memory 32 Gb”. There are enough memory (total 64GB) and there is no warning message of memory in the output. Sometimes it just died quietly, sometimes with an error “Command terminated by signal 9”, which I have no idea what it means. I don’t understand why assigning more memory can be a problem.
This sounds like you actually ran out of memory (even though you seemingly should have enough), since your kernel kills the jobs (signal 9 is the SIGKILL signal). Did you try with just a little less memory (like 50 GB)?
I guess you are using Linux, it uses opportunistic memory allocation.
It can happen that malloc, or new, returns a valid address, even though there is not enough free memory left. So at the time of access the kernel runs out of memory and has to kill a program that uses large amounts of memory. This is a “bug” of malloc, where the return of a valid address does not guarantee the allocation of the memory, and has nothing to do with PSI4.
You can always shut off the opportunistic memory allocation of course.
Further detail at StackOverflow.
To be sure you can also inspect the output of dmesg right after your job was killed.
I hope this helps.