Optimizing the geometry of organometallic molecule, problems

I’ve managed to converge the structure using the setup in the original post, but with hydrogen atoms added. The process was reeeeally slow, but a few quick test calculations I did suggest that I could halve the time by using 24 threads instead of 12. My rhf calculations with density fitting (I think) are still not working properly, however.

Interesting, if I had to guess it is Palladium causing issues. Between the sto-3g basis and and incorrect density fitting basis there are likely issues. One item to try is a def2-SVP basis which will likely still be much faster than direct and give you a better quality wavefunction.

Another option to try is scf_type disk as your computation is not that large it may be fairly quick to do.

1 Like

Yes, I think it’s indeed the palladium. Anyway, thanks for your suggestions.

Okay, so I did figure out that scf_type df switches on density fitting, and so I’ve now been able to run my HF calculations using DF-RHF with a STO-3G basis set. One thing all my attempts have in common is that DIIS gets stuck at the same value after a few iterations, as you can see in the snippet below. What could be the problem here? I see the same thing for another palladium containing molecule I’m trying to converge.

   @DF-RHF iter 102: -6962.21654729836246    4.54747e-12   1.20210e-05 DIIS
   @DF-RHF iter 103: -6962.21654729835700    5.45697e-12   1.20210e-05 DIIS
   @DF-RHF iter 104: -6962.21654729836609   -9.09495e-12   1.20210e-05 DIIS
   @DF-RHF iter 105: -6962.21654729836246    3.63798e-12   1.20210e-05 DIIS
   @DF-RHF iter 106: -6962.21654729837610   -1.36424e-11   1.20210e-05 DIIS
   @DF-RHF iter 107: -6962.21654729837337    2.72848e-12   1.20210e-05 DIIS
   @DF-RHF iter 108: -6962.21654729834790    2.54659e-11   1.20210e-05 DIIS
   @DF-RHF iter 109: -6962.21654729835245   -4.54747e-12   1.20210e-05 DIIS
   @DF-RHF iter 110: -6962.21654729835518   -2.72848e-12   1.20210e-05 DIIS
   @DF-RHF iter 111: -6962.21654729836428   -9.09495e-12   1.20210e-05 DIIS
   @DF-RHF iter 112: -6962.21654729835882    5.45697e-12   1.20210e-05 DIIS
   @DF-RHF iter 113: -6962.21654729836700   -8.18545e-12   1.20210e-05 DIIS
   @DF-RHF iter 114: -6962.21654729836519    1.81899e-12   1.20210e-05 DIIS
   @DF-RHF iter 115: -6962.21654729837246   -7.27596e-12   1.20210e-05 DIIS
   @DF-RHF iter 116: -6962.21654729836064    1.18234e-11   1.20210e-05 DIIS
   @DF-RHF iter 117: -6962.21654729835427    6.36646e-12   1.20210e-05 DIIS
   @DF-RHF iter 118: -6962.21654729836519   -1.09139e-11   1.20210e-05 DIIS
   @DF-RHF iter 119: -6962.21654729836428    9.09495e-13   1.20210e-05 DIIS
   @DF-RHF iter 120: -6962.21654729836155    2.72848e-12   1.20210e-05 DIIS
   @DF-RHF iter 121: -6962.21654729835700    4.54747e-12   1.20210e-05 DIIS
   @DF-RHF iter 122: -6962.21654729835791   -9.09495e-13   1.20210e-05 DIIS
   @DF-RHF iter 123: -6962.21654729836155   -3.63798e-12   1.20210e-05 DIIS
   @DF-RHF iter 124: -6962.21654729835791    3.63798e-12   1.20210e-05 DIIS
   @DF-RHF iter 125: -6962.21654729834790    1.00044e-11   1.20210e-05 DIIS
   @DF-RHF iter 126: -6962.21654729835245   -4.54747e-12   1.20210e-05 DIIS
   @DF-RHF iter 127: -6962.21654729837064   -1.81899e-11   1.20210e-05 DIIS
   @DF-RHF iter 128: -6962.21654729836246    8.18545e-12   1.20210e-05 DIIS
   @DF-RHF iter 129: -6962.21654729835609    6.36646e-12   1.20210e-05 DIIS
   @DF-RHF iter 130: -6962.21654729834790    8.18545e-12   1.20210e-05 DIIS
   @DF-RHF iter 131: -6962.21654729836155   -1.36424e-11   1.20210e-05 DIIS
   @DF-RHF iter 132: -6962.21654729837064   -9.09495e-12   1.20210e-05 DIIS
   @DF-RHF iter 133: -6962.21654729836609    4.54747e-12   1.20210e-05 DIIS
   @DF-RHF iter 134: -6962.21654729836246    3.63798e-12   1.20210e-05 DIIS
   @DF-RHF iter 135: -6962.21654729835336    9.09495e-12   1.20210e-05 DIIS
   @DF-RHF iter 136: -6962.21654729835791   -4.54747e-12   1.20210e-05 DIIS
   @DF-RHF iter 137: -6962.21654729836064   -2.72848e-12   1.20210e-05 DIIS
   @DF-RHF iter 138: -6962.21654729835700    3.63798e-12   1.20210e-05 DIIS
   @DF-RHF iter 139: -6962.21654729836882   -1.18234e-11   1.20210e-05 DIIS

What are those calculations for? Educational to look at MOs?

Convergence problems are common for transition metals. Adding to it, Pd is heavy, meaning relativistic effects play role. The all-electron minimal basis STO-3G is a terrible choice
In short, STO-3G is completely inappropriate here, maybe even HF.

If you want reliable but still ‘cheap’ computations, I’d recommend PBEh-3c.


The def2-mSVP basis set is mandatory and comes with ECPs that take care of relativistic effects. Unfortunately, the ECP gradients are done numerically and will be slow.

I tested it and there seem no convergence problems.

1 Like

It’s for a research project where we’re trying to find various quantum mechanical descriptors for a number of different palladium compounds. To do this, I need to find the ground state structure for the compounds first. My idea was to use the approach described in a paper I found on molecular descriptors from first principles calculations, namely, to first use molecular mechanics for a fast but highly approximative ground state search, then Hartree Fock to refine the results, and finally DFT to incorporate both exchange and correlation effects and thus achieve further refinement. This was the most computationally efficient way to do a ground state search according to the authors of the paper. I don’t think they included any organometallic phases, however.

Anyway, I tried the setup found at the bottom of this post, but after three ionic steps the calculation crashed with the following error:

RuntimeError: Command (dftd3 dftd3_geometry.xyz -grad) failed with PATH (:/home/ubuntu/psi4conda/bin:/home/ubuntu/psi4conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin)

scf_type df
guess sad
geom_maxiter 100

odd error. You say you manage to do 3 geometry optimization cycles? Or 3 SCF steps?

Three (or two – I’m not sure if the first one counts) geometry optimization cycles. This is what I get when I use grep ‘~’ output.dat:

   Step     Total Energy     Delta E     MAX Force     RMS Force      MAX Disp      RMS Disp    ~
  --------------------------------------------------------------------------------------------- ~
    Convergence Criteria    1.00e-06 *    3.00e-04 *             o    1.20e-03 *             o  ~
  --------------------------------------------------------------------------------------------- ~
      1   -2196.45623987   -2.20e+03      7.67e-02      1.61e-02 o    1.18e-01      2.44e-02 o  ~
      2   -2196.57133537   -1.15e-01      3.48e-02      6.92e-03 o    1.12e-01      2.78e-02 o  ~
      3   -2196.62038153   -4.90e-02      1.30e-02      2.89e-03 o    2.01e-01      3.22e-02 o  ~

This is where the error message stems from in the code, I think:

File “/home/ubuntu/psi4conda/lib//python3.6/site-packages/psi4/driver/qcdb/intf_dftd3/runner.py”, line 135, in run_dftd3_from_arrays
** raise RuntimeError(err) from err**

Yes, 3 optimization steps. That is good. Obviously the dftd3 program works in principle. Why it would fail mid optimization, I have no idea. The energy looks OK, so the molecule has not exploded which might cause the program to fail.

Any idea @loriab? (psi4 version 1.3.2)

I think the program fail had to do with my disk space running out. I did a test with def2-qzvpp-jkfit/b3lyp yesterday, and this calculation filled up the directory /dev/vda1 to 100 % before even starting the first SCF cycle. Either the ‘pbeh3c/def2-mSVP’ did the same but after a few geometry optimization steps, or another job that I submitted hijacked the entire disk, thus causing all running jobs to fail.

In my mind 90 GB disk space is still a lot, but I may be behind the times, so to speak.

Anyway, I’ve been looking through the list of basis sets to find a suitable one for my molecules. The problem is that there aren’t that many that can handle palladium, and even fewer that are well suited for density fitting (jkfit or RI). Ideally, I’d want a fairly large basis set that can be used for DFT calculations and that can also utilize df.

As I understand it from the Psi4 documentation, the keyword relativistic x2c can be used to incorporate relativistic effects. Is that correct?

I’m testing


again, this time while keeping track of disk space consumption.

Both x2c and DKH relativistic Hamiltonians are available. You would likely need to import appropriate all-electron basis sets from https://www.basissetexchange.org/. There may not be optimal JK fitting basis sets for everything. But appropriate ones could likely be found.

def2-mSVP should not have large disk space requirements. It is printed in in the integral setup section before the SCF iterations.

  ==> Integral Setup <==

  DFHelper Memory: AOs need 7.815 GiB; user supplied 66.388 GiB. Using in-core AOs.```

Wow, that’s quite the basis set treasure trove, thank you :slightly_smiling_face: I only used what I could find on this site.

You’re right, so far so good memory wise with def2-mSVP. It’s very slow, however. The process of adding the ECP gradient terms seems to take forever. Is that what I should expect? I’m planning to calculate properties of thousands of molecules eventually, and so computational efficiency is very important, perhaps even more so than maximum accuracy.

Yes, the numerical ECP gradient is very slow. Our ECP technology is not yet where it should be.

1 Like

So my second attempt to optimize using pbeh3c and def2-mSVP also failed, although this time after four optimization steps instead of three. I have no idea what’s wrong, because the memory did definitely not run out. This is the full error message:

  File "/home/ubuntu/psi4conda/bin/psi4", line 287, in <module>
  File "<string>", line 96, in <module>
  File "/home/ubuntu/psi4conda/lib//python3.6/site-packages/psi4/driver/driver.py", line 1052, in optimize
    G, wfn = gradient(lowername, return_wfn=True, molecule=moleculeclone, **kwargs)
  File "/home/ubuntu/psi4conda/lib//python3.6/site-packages/psi4/driver/driver.py", line 691, in gradient
    wfn = procedures['gradient'][lowername](lowername, molecule=molecule, **kwargs)
  File "/home/ubuntu/psi4conda/lib//python3.6/site-packages/psi4/driver/procrouting/proc.py", line 2070, in run_scf_gradient
    disp_grad = ref_wfn._disp_functor.compute_gradient(ref_wfn.molecule())
  File "/home/ubuntu/psi4conda/lib//python3.6/site-packages/psi4/driver/procrouting/empirical_dispersion.py", line 236, in compute_gradient
  File "/home/ubuntu/psi4conda/lib//python3.6/site-packages/psi4/driver/qcdb/intf_dftd3/runner.py", line 135, in run_dftd3_from_arrays
    raise RuntimeError(err) from err

RuntimeError: Command (`dftd3 dftd3_geometry.xyz -grad`) failed with PATH (:/home/ubuntu/psi4conda/bin:/home/ubuntu/psi4conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin)

Printing out the relevant lines from the Psithon --> Python processed input file:
    core.set_global_option("BASIS", "def2-mSVP")
    core.set_global_option("GUESS", "sad")
    core.set_global_option("GEOM_MAXITER", 100)
--> optimize('pbeh3c')

Ok, so I think I get it: the executable dftd3 can’t find dftd3_geometry.xyz. Where is this file supposed to be found during runtime? Right now it’s located in directory /home/ubuntu/dftd3_3e0bfb10, but I’m guessing that this directory was created after the crash (3e0bfb10 is a memory address, right?). I’m thinking that it’s a temporary geometry file, and thus is written to /tmp.

Btw, when I run dftd3 from /home/ubuntu/psi4conda/bin manually using the command

dftd3 …/…/dftd3_3e0bfb10/dftd3_geometry.xyz -grad

I get the error message “inacceptable version number”. I don’t know if this is relevant information. Perhaps the correct version number is supplied from Psi4? (I’ve looked through the source code of dftd3 and at least I understand that “version” does not refer to which version of dftd3 I have).

dftd3 reads a local parameter file and hence the command line itself is not enough.
The error might be related to https://github.com/psi4/psi4/pull/1725
You could try the development version (conda update psi4 psi4-rt -psi4/label/dev) where the fix is included.

Though I don’t understand why it would only fail after the n-th optimization step and not immediately.

1 Like

@loriab any ideas here?

to make sure we’re on the same page, could you post your conda list, please?

Here it is:

asn1crypto                0.24.0                   py36_0  
atomicwrites              1.3.0                    py36_1  
attrs                     19.1.0                   py36_1  
blas                      1.0                         mkl  
ca-certificates           2019.1.23                     0  
certifi                   2019.3.9                 py36_0  
cffi                      1.12.3           py36h2e261b9_0  
chardet                   3.0.4                    py36_1  
chemps2                   1.8.9                h8c3debe_0    psi4
conda                     4.6.14                   py36_0  
cryptography              2.6.1            py36h1ba5d50_0  
cvxopt                    1.2.0            py36hfa32c7d_0  
dataclasses               0.6                      pypi_0    pypi
decorator                 4.4.0                    py36_1  
deepdiff                  3.3.0                    py36_1    psi4
dftd3                     3.2.1                h84218bc_2    psi4
dkh                       1.2                  h173d85e_2    psi4
future                    0.17.1                   py36_0  
gau2grid                  1.3.1                h035aef0_0    psi4
gcp                       2.0.2                h84218bc_1    psi4
gdma                      2.2.6                h0e1e685_6    psi4
glpk                      4.65                 h3ceedfd_2  
gmp                       6.1.2                h6c8ec71_1  
gsl                       2.4                  h14c3975_4  
h5py                      2.8.0            py36h989c5e5_3  
hdf5                      1.10.2               hba1933b_1  
idna                      2.8                      py36_0  
intel-openmp              2019.3                      199  
jsonpickle                0.9.6                    py36_0    psi4
libedit                   3.1.20181209         hc058e9b_0  
libefp                    1.5.0                h84097be_1    psi4
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
libint                    1.2.1                hb4a4fd4_6    psi4
libstdcxx-ng              8.2.0                hdf63c60_1  
libxc                     4.3.4                h7b6447c_0    psi4
metis                     5.1.0                hf484d3e_4  
mkl                       2019.3                      199  
mkl_fft                   1.0.12           py36ha843d7b_0  
mkl_random                1.0.2            py36hd81dba3_0  
more-itertools            7.0.0                    py36_0  
ncurses                   6.1                  he6710b0_1  
networkx                  2.3                      pypi_0    pypi
numpy                     1.16.3           py36h7e9f1db_0  
numpy-base                1.16.3           py36hde5b4d6_0  
openfermion               0.9              py36hd763080_1    psi4
openfermionpsi4           0.4              py36h934b37d_0    psi4
openssl                   1.1.1b               h7b6447c_1  
pcmsolver                 1.2.1            py36h8733eb9_0    psi4
pint                      0.9                      py36_2    psi4
pip                       19.1.1                   py36_0  
pluggy                    0.11.0                   pypi_0    pypi
psi4                      1.3.2+ecbda83    py36h189457c_0    psi4
psi4-rt                   1.3.2            py36ha6b60c8_0    psi4
py                        1.8.0                    py36_0  
pycosat                   0.6.3            py36h14c3975_0  
pycparser                 2.19                     py36_0  
pydantic                  0.23                     py36_0    psi4
pylibefp                  0.4+f42ffb8      py36h5a86982_1    psi4
pyopenssl                 19.0.0                   py36_0  
pysocks                   1.7.0                    py36_0  
pytest                    4.5.0                    py36_0  
python                    3.6.8                h0371630_0  
qcelemental               0.4.0                    pypi_0    pypi
readline                  7.0                  h7b6447c_5  
requests                  2.21.0                   py36_0  
resp                      0.7              py36ha93d1a2_0    psi4
ruamel_yaml               0.15.46          py36h14c3975_0  
scipy                     1.2.1            py36h7c811a0_0  
setuptools                41.0.1                   py36_0  
simint                    0.7                  h642920c_1    psi4
six                       1.12.0                   py36_0  
snsmp2                    1.0.3            py36he403277_2    psi4
sqlite                    3.28.0               h7b6447c_0  
suitesparse               5.2.0                h9e4a6bb_0  
tbb                       2019.4               hfd86e86_0  
tk                        8.6.8                hbc83047_0  
urllib3                   1.24.2                   py36_0  
v2rdm_casscf              0.8              py36hc3560e4_6    psi4
wcwidth                   0.1.7                    py36_0  
wheel                     0.33.4                   py36_0  
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
zlib                      1.2.11               h7b6447c_3