I need to do a large number of small calculations, so I do A LOT of calls to psi4.energy(). However, Psi4 keeps piling up more and more memory.
As a simple test consider the following:
import psi4
psi4.set_memory('1000 MB')
mol = psi4.geometry('''
O
H 1 0.96
H 1 0.96 2 104.5
''')
psi4.set_options({'basis' : 'cc-pVDZ'})
EHF, wfn = psi4.energy('scf', return_wfn='on')
for i in range(10000):
psi4.energy('detci', ref_wfn=wfn)
It repeats the same calculation over and over again. So the memory requirements should always be the same. According to the documentation all PSI variables are cleared when calling psi4.energy, so no data from previous calls should stick around and pile up.
But if I run the above script, the corresponding Python process just keeps using more and more memory.
This also happens when running psi4.energy('scf') in side the loop, however it does not grow as fast as for detci.
I tested this on Psi4 1.1 and 1.2.1.
How many times does DETCI need to run before you see crashes due to lack of memory? grep -c "(a 'D E T C I' module)" output.dat tells me that I’m currently on run 3352 of DETCI, and the computation hasn’t crashed yet.
I’m using the developer version of 1.3, which has many miscellaneous memory error fixes. If it took you fewer iterations to see a crash, we may have already fixed this. If so, I’ll check with the other developers on when we can expect the 1.3 release candidate to appear on conda.
I did not actually run this until it crashed. There is no need to do this, because I can just watch the memory usage while running. Also how many iterations it takes until it crashes is not really the point here. This heavily depends on your hardware (how much memory is available) and probably on the implementation and therefore version (I guess a more memory efficient implementation would leak less). Also note that this is not specific to DETCI.
The point is not how fast it leaks, but that it does leak. There should not be any increase in memory at all, no matter how many times I repeat the same calculation. The 10000 iterations are just some random number I picked, if this runs through on your machine, try more iterations, or a larger basis, or a smaller computer. But it is much easier and faster to see when watching the memory consumption of the corresponding Python process.
I could not test 1.3, because it apparently requires CMake 3.8 and I only have 3.7 available.
My mistake. I can reproduce the leak on the developer version of Psi. I filed an issue for you, as that’s more likely to get the attention of other developers, who are more familiar with DETCI.