Direct is one of our slowest JK algorithms. Running the optimization with density fitting will be much much faster.
Optimizing the geometry of organometallic molecule, problems
In addition to dgasmith’s good advice, if you’re distributing this across 12 threads, then your memory line is almost certainly wrong. In Psi, you don’t tell it the memory per thread. You tell it the total memory, and it distributes that across the threads for you. If you truly have 8GB, I would reconsider a 12-threaded computation. Otherwise, you probably mean to say 96 GB. These two points are important enough that unless there is a long wait time to get on your cluster, I recommend killing your current job and starting a new one at the best geometry so far, with these two changes.
Aha, thanks. I started out with density fitting, but the job crashed with an error message that made me think that the program somehow couldn’t handle the palladium atoms. Now it seems to work better, though. However, the job still crashed because the fourth scf cycle (or third ionic step) didn’t converge after the specified 200 iterations. Any tips (other than just increasing maxiter, that is)? I should probably mention that the slower non-DF calculation is still running after 25 ionic steps.
Ok, thanks for the advice! I haven’t yet run into any memory issues, but this is good to know. I could probably use some more memory per thread.
First, you may very well have been running into memory issues without realizing it. Less memory means more reading and writing from disks which means slower computations.
However, your report that you can’t converge the SCF with the DF approximation is disturbing. Are you absolutely sure you used Psi 1.3.2 for this and added the hydrogens? If you did, please upload the output file so we can investigate. If you didn’t do those things, try again with them.
(Note to self: This may be a problem with SCF reading if the displacements are large, because we need to re-orthogonalize MOs once we figure out orbital reads in DDD.)
1 Like
Yes, psi4 --version now returns 1.3.2, and yes, I’ve added the hydrogens, so now the structure has a total of 69 atoms. I’ve restarted from the last set of displaced coordinates, but now with 8 threads and 8 GB of memory in total. The DF guess converged after 44 iterations and I’ve set maxiter to 500 this time.
Can you send the output file where the DF optimization fails? If you were using the right version of Psi and the right number of hydrogens, this sounds like a bug.
Unfortunately, I didn’t save the original output file. However, perhaps the output file from the restarted job could do, since I still have the same problem. As you can see in the attached file, several scf cycles require quite a lot (>200) of iterations to converge, which makes the calculations pretty slow. problem_output.dat (832.0 KB)
Troubling.
Before I try playing with this example myself: why are you using a UHF reference? It looks like this is a closed-shell species. You can see a large speedup for free by using an RHF reference whenever you work with closed-shell species.
Also, you’re not using density-fitting just because you set df_scf_guess true. Did you look to see what that keyword actually does in the manual? I’m not sure if we need to clarify the documentation, or if you just didn’t read the manual very carefully. For that matter, where did you get scf_type direct from?
Before I added the hydrogens, it must’ve been an open shell molecule since I remember getting an error message that said so (which makes sense since the carbon atoms didn’t have completely filled valence shells then). I simply didn’t think about changing it back into RHF. I’m trying that now.
I read the documentation about density fitting and also some other forum threads. The latter is where I got scf_type direct from, but now that I think about it seems like it would just tell the Psi4 to override the DF guess.
Since I’m not very familiar with density fitting, I do find the documentation a bit difficult to understand. I have to read up more on this. However, could you perhaps tell me which keywords would initiate DF?
I’ve managed to converge the structure using the setup in the original post, but with hydrogen atoms added. The process was reeeeally slow, but a few quick test calculations I did suggest that I could halve the time by using 24 threads instead of 12. My rhf calculations with density fitting (I think) are still not working properly, however.
Interesting, if I had to guess it is Palladium causing issues. Between the sto-3g basis and and incorrect density fitting basis there are likely issues. One item to try is a def2-SVP basis which will likely still be much faster than direct and give you a better quality wavefunction.
Another option to try is scf_type disk as your computation is not that large it may be fairly quick to do.
1 Like
Yes, I think it’s indeed the palladium. Anyway, thanks for your suggestions.
Okay, so I did figure out that scf_type df switches on density fitting, and so I’ve now been able to run my HF calculations using DF-RHF with a STO-3G basis set. One thing all my attempts have in common is that DIIS gets stuck at the same value after a few iterations, as you can see in the snippet below. What could be the problem here? I see the same thing for another palladium containing molecule I’m trying to converge.
@DF-RHF iter 102: -6962.21654729836246 4.54747e-12 1.20210e-05 DIIS
@DF-RHF iter 103: -6962.21654729835700 5.45697e-12 1.20210e-05 DIIS
@DF-RHF iter 104: -6962.21654729836609 -9.09495e-12 1.20210e-05 DIIS
@DF-RHF iter 105: -6962.21654729836246 3.63798e-12 1.20210e-05 DIIS
@DF-RHF iter 106: -6962.21654729837610 -1.36424e-11 1.20210e-05 DIIS
@DF-RHF iter 107: -6962.21654729837337 2.72848e-12 1.20210e-05 DIIS
@DF-RHF iter 108: -6962.21654729834790 2.54659e-11 1.20210e-05 DIIS
@DF-RHF iter 109: -6962.21654729835245 -4.54747e-12 1.20210e-05 DIIS
@DF-RHF iter 110: -6962.21654729835518 -2.72848e-12 1.20210e-05 DIIS
@DF-RHF iter 111: -6962.21654729836428 -9.09495e-12 1.20210e-05 DIIS
@DF-RHF iter 112: -6962.21654729835882 5.45697e-12 1.20210e-05 DIIS
@DF-RHF iter 113: -6962.21654729836700 -8.18545e-12 1.20210e-05 DIIS
@DF-RHF iter 114: -6962.21654729836519 1.81899e-12 1.20210e-05 DIIS
@DF-RHF iter 115: -6962.21654729837246 -7.27596e-12 1.20210e-05 DIIS
@DF-RHF iter 116: -6962.21654729836064 1.18234e-11 1.20210e-05 DIIS
@DF-RHF iter 117: -6962.21654729835427 6.36646e-12 1.20210e-05 DIIS
@DF-RHF iter 118: -6962.21654729836519 -1.09139e-11 1.20210e-05 DIIS
@DF-RHF iter 119: -6962.21654729836428 9.09495e-13 1.20210e-05 DIIS
@DF-RHF iter 120: -6962.21654729836155 2.72848e-12 1.20210e-05 DIIS
@DF-RHF iter 121: -6962.21654729835700 4.54747e-12 1.20210e-05 DIIS
@DF-RHF iter 122: -6962.21654729835791 -9.09495e-13 1.20210e-05 DIIS
@DF-RHF iter 123: -6962.21654729836155 -3.63798e-12 1.20210e-05 DIIS
@DF-RHF iter 124: -6962.21654729835791 3.63798e-12 1.20210e-05 DIIS
@DF-RHF iter 125: -6962.21654729834790 1.00044e-11 1.20210e-05 DIIS
@DF-RHF iter 126: -6962.21654729835245 -4.54747e-12 1.20210e-05 DIIS
@DF-RHF iter 127: -6962.21654729837064 -1.81899e-11 1.20210e-05 DIIS
@DF-RHF iter 128: -6962.21654729836246 8.18545e-12 1.20210e-05 DIIS
@DF-RHF iter 129: -6962.21654729835609 6.36646e-12 1.20210e-05 DIIS
@DF-RHF iter 130: -6962.21654729834790 8.18545e-12 1.20210e-05 DIIS
@DF-RHF iter 131: -6962.21654729836155 -1.36424e-11 1.20210e-05 DIIS
@DF-RHF iter 132: -6962.21654729837064 -9.09495e-12 1.20210e-05 DIIS
@DF-RHF iter 133: -6962.21654729836609 4.54747e-12 1.20210e-05 DIIS
@DF-RHF iter 134: -6962.21654729836246 3.63798e-12 1.20210e-05 DIIS
@DF-RHF iter 135: -6962.21654729835336 9.09495e-12 1.20210e-05 DIIS
@DF-RHF iter 136: -6962.21654729835791 -4.54747e-12 1.20210e-05 DIIS
@DF-RHF iter 137: -6962.21654729836064 -2.72848e-12 1.20210e-05 DIIS
@DF-RHF iter 138: -6962.21654729835700 3.63798e-12 1.20210e-05 DIIS
@DF-RHF iter 139: -6962.21654729836882 -1.18234e-11 1.20210e-05 DIISWhat are those calculations for? Educational to look at MOs?
Convergence problems are common for transition metals. Adding to it, Pd is heavy, meaning relativistic effects play role. The all-electron minimal basis STO-3G is a terrible choice
In short, STO-3G is completely inappropriate here, maybe even HF.
If you want reliable but still ‘cheap’ computations, I’d recommend PBEh-3c.
optimize('pbeh3c/def2-mSVP')
The def2-mSVP basis set is mandatory and comes with ECPs that take care of relativistic effects. Unfortunately, the ECP gradients are done numerically and will be slow.
I tested it and there seem no convergence problems.
1 Like
It’s for a research project where we’re trying to find various quantum mechanical descriptors for a number of different palladium compounds. To do this, I need to find the ground state structure for the compounds first. My idea was to use the approach described in a paper I found on molecular descriptors from first principles calculations, namely, to first use molecular mechanics for a fast but highly approximative ground state search, then Hartree Fock to refine the results, and finally DFT to incorporate both exchange and correlation effects and thus achieve further refinement. This was the most computationally efficient way to do a ground state search according to the authors of the paper. I don’t think they included any organometallic phases, however.
Anyway, I tried the setup found at the bottom of this post, but after three ionic steps the calculation crashed with the following error:
RuntimeError: Command (dftd3 dftd3_geometry.xyz -grad) failed with PATH (:/home/ubuntu/psi4conda/bin:/home/ubuntu/psi4conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin)
scf_type df
guess sad
geom_maxiter 100
}
optimize('pbeh3c/def2-mSVP')odd error. You say you manage to do 3 geometry optimization cycles? Or 3 SCF steps?
Three (or two – I’m not sure if the first one counts) geometry optimization cycles. This is what I get when I use grep ‘~’ output.dat:
Step Total Energy Delta E MAX Force RMS Force MAX Disp RMS Disp ~
--------------------------------------------------------------------------------------------- ~
Convergence Criteria 1.00e-06 * 3.00e-04 * o 1.20e-03 * o ~
--------------------------------------------------------------------------------------------- ~
1 -2196.45623987 -2.20e+03 7.67e-02 1.61e-02 o 1.18e-01 2.44e-02 o ~
2 -2196.57133537 -1.15e-01 3.48e-02 6.92e-03 o 1.12e-01 2.78e-02 o ~
3 -2196.62038153 -4.90e-02 1.30e-02 2.89e-03 o 2.01e-01 3.22e-02 o ~This is where the error message stems from in the code, I think:
File “/home/ubuntu/psi4conda/lib//python3.6/site-packages/psi4/driver/qcdb/intf_dftd3/runner.py”, line 135, in run_dftd3_from_arrays
** raise RuntimeError(err) from err**
Yes, 3 optimization steps. That is good. Obviously the dftd3 program works in principle. Why it would fail mid optimization, I have no idea. The energy looks OK, so the molecule has not exploded which might cause the program to fail.
Any idea @loriab? (psi4 version 1.3.2)
I think the program fail had to do with my disk space running out. I did a test with def2-qzvpp-jkfit/b3lyp yesterday, and this calculation filled up the directory /dev/vda1 to 100 % before even starting the first SCF cycle. Either the ‘pbeh3c/def2-mSVP’ did the same but after a few geometry optimization steps, or another job that I submitted hijacked the entire disk, thus causing all running jobs to fail.
In my mind 90 GB disk space is still a lot, but I may be behind the times, so to speak.
Anyway, I’ve been looking through the list of basis sets to find a suitable one for my molecules. The problem is that there aren’t that many that can handle palladium, and even fewer that are well suited for density fitting (jkfit or RI). Ideally, I’d want a fairly large basis set that can be used for DFT calculations and that can also utilize df.
As I understand it from the Psi4 documentation, the keyword relativistic x2c can be used to incorporate relativistic effects. Is that correct?