Job exceeding memory that shouldn't and terminating, memory should be sufficient

LukeW · April 28, 2016, 1:00am

Hi, I have recently had many problems with jobs that say they exceed memory and terminate, yet I am allocating 128gb of memory for a system which theoretically should need less than this. It just randomly terminates even if less memory if being used and I was wondering what the error was.

Here is the input file:

memory 128 Gb
molecule complex {
0 2
C -2.83891 -0.84747 -0.20778
N -2.06959 -0.27604 0.77905
C -0.85937 -0.82512 0.74757
N -0.82980 -1.74677 -0.21226
C -2.05910 -1.77487 -0.83179
C -2.45490 0.86620 1.59959
C 0.35903 -2.49473 -0.61265
F -1.01767 1.46174 -1.11980
B 0.28985 1.63764 -0.63648
F 0.23738 1.82826 0.77419
F 0.89453 2.73957 -1.24020
F 1.05039 0.46734 -0.88673
H -0.02604 -0.54755 1.38339
H -2.27287 -2.45330 -1.64959
H -3.86751 -0.55086 -0.37733
H -3.45079 0.68580 2.01748
H -1.72185 0.98138 2.40249
H -2.44673 1.75972 0.96603
H 0.98379 -1.85263 -1.24323
H 0.91655 -2.78676 0.28358
H 0.03922 -3.39046 -1.15377
C 2.81822 0.12352 1.52490
C 3.26768 -0.78277 0.56145
H 3.28632 0.11977 2.50813
H 2.01109 0.81253 1.27433
N 3.64537 -1.55240 -0.23308
}

set globals {
basis aug-cc-pVQZ
scf_type df
freeze_core True
basis_guess cc-pVDZ
guess sad
S_Orthogonalization=canonical
reference rohf
}

set cachelevel 0
set print 2
energy(‘dfmp2’)

Here is the job file:

#!/bin/bash
#PBS -P k96
#PBS -q normal
#PBS -l mem=128GB
#PBS -l ncpus=32
#PBS -l jobfs=500GB
#PBS -l walltime=4:00:00
#PBS -j oe
#PBS -l wd

module load psicode/4.0b5.1
export OMP_NUM_THREADS=32
export MKL_NUM_THREADS=32
psi4 ch2cnmimbf4conf4conf1avqzsummer.inp ch2cnmimbf4conf4conf1avqzsummer.out

Here is the .o file:

/jobfs/local/pbs/mom_priv/jobs/5883868.r-man2.SC: line 14: 19944 Killed psi4 ch2cnmimbf4conf4conf1avqzsummer.inp ch2cnmimbf4conf4c$
Process psi4 (PID 19944) was killed by kernel for exceeding memory limits. VM: 80570024kB, RSS: 61485776kB, File RSS: 1180kB

======================================================================================
Resource Usage on 2016-04-28 05:39:41:
Job Id: 5883868.r-man2
Project: k96
Exit Status: 137 (Linux Signal 9 SIGKILL Kill, unblockable)
Service Units: 18.07
NCPUs Requested: 32 NCPUs Used: 32
CPU Time Used: 07:56:48
Memory Requested: 128.0GB Memory Used: 54.95GB
Walltime requested: 04:00:00 Walltime Used: 00:33:53
JobFS requested: 500.0GB JobFS used: 581.68MB

This is not system size specific and some large systems have run while smaller systems have not, in general with the 4.0b5.1 psi4 I can get not augmented basis sets to work while augmented ones don’t once larger than double zeta. Does anybody have any suggestions?

LukeW · April 28, 2016, 3:31am

Is there perhaps a way to make it use the low memory algorithm so this error does not occur? I have spoken to people and they think the issue could be that I am using a higher memory algorithm causing it to max out. Any idea if this is possible?

dgasmith · April 28, 2016, 2:19pm

DF-MP2 is simple enough to only require a single algorithm unless you are doing many thousands of basis functions. Can you please post the end of the Psi4 output file starting with the DF-MP2 header?

LukeW · April 28, 2016, 2:37pm

sure! it just randomly ends, sorry if this is quite long.

//>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>//
// DFMP2 //
//<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<//

*** tstart() called on r3437
*** at Sun Apr 24 20:14:27 2016

=> Loading Basis Set <=

Role: BASIS
Keyword: BASIS
Name: CC-PVTZ
atoms 1-4, 6-7, 23, 25, 27 entry C          line   171 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 5, 22, 24, 26        entry N          line   209 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 8-21, 28-29          entry H          line    20 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 30                   entry F          line   285 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs


     ---------------------------------------------------------
                               SCF
        by Justin Turney, Rob Parrish, and Andy Simmonett
                         ROHF Reference
                   32 Threads, 128000 MiB Core
     ---------------------------------------------------------

==> Geometry <==

Molecular point group: c1
Full point group: C1

Geometry (in Angstrom), charge = 0, multiplicity = 2:

   Center              X                  Y                   Z               Mass
------------   -----------------  -----------------  -----------------  -----------------
       C         -1.070238489078    -2.308932598554     0.093130244992    12.000000000000
       C         -0.934488489078    -1.579512598554    -1.266159755008    12.000000000000
       C         -0.936918489078    -0.096542598554    -0.910709755008    12.000000000000
       C         -1.287508489078    -1.202212598554     1.131580244992    12.000000000000
       N         -1.773598489078    -0.012432598554     0.340660244992    14.003074004780
       C         -3.219768489078    -0.160852598554     0.010630244992    12.000000000000
       C         -1.558458489078     1.266277401446     1.072180244992    12.000000000000
       H         -3.378938489078    -1.114012598554    -0.503269755008     1.007825032070
       H         -3.514708489078     0.672317401446    -0.636269755008     1.007825032070
       H         -3.791418489078    -0.138092598554     0.944330244992     1.007825032070
       H         -2.186078489078     1.258027401446     1.970560244992     1.007825032070
       H         -1.840748489078     2.092687401446     0.410990244992     1.007825032070
       H         -0.498148489078     1.330697401446     1.347590244992     1.007825032070
       H         -1.352648489078     0.580077401446    -1.665119755008     1.007825032070
       H          0.065701510922     0.251567401446    -0.626719755008     1.007825032070
       H         -1.767698489078    -1.823902598554    -1.937189755008     1.007825032070
       H          0.001031510922    -1.844392598554    -1.771039755008     1.007825032070
       H         -1.907668489078    -3.017212598554     0.085030244992     1.007825032070
       H         -0.155788489078    -2.864862598554     0.332090244992     1.007825032070
       H         -2.023718489078    -1.429252598554     1.910960244992     1.007825032070
       H         -0.347588489078    -0.882552598554     1.600930244992     1.007825032070
       N          1.671071510922     0.742537401446     2.088560244992    14.003074004780
       C          2.104211510922     0.298847401446     1.091830244992    12.000000000000
       N          2.595321510922    -0.089722598554    -0.065859755008    14.003074004780
       C          2.424341510922    -1.356462598554    -0.398249755008    12.000000000000
       N          2.307121510922    -2.458212598554    -0.779949755008    14.003074004780
       C          0.975841510922     2.909937401446    -0.731989755008    12.000000000000
       H          1.647341510922     2.096917401446    -1.010899755008     1.007825032070
       H          1.021111510922     3.450547401446     0.212680244992     1.007825032070
       F         -0.262708489078     2.821337401446    -1.260599755008    18.998403224000

Running in c1 symmetry.

Rotational constants: A = 0.02222 B = 0.01939 C = 0.01389 [cm^-1]
Rotational constants: A = 666.20841 B = 581.23385 C = 416.53846 [MHz]
Nuclear repulsion = 933.329268115655168

Charge = 0
Multiplicity = 2
Electrons = 107
Nalpha = 54
Nbeta = 53

==> Algorithm <==

SCF Algorithm Type is DF.
DIIS enabled.
MOM disabled.
Fractional occupation disabled.
Guess Type is READ.
Energy threshold = 1.00e-08
Density threshold = 1.00e-08
Integral threshold = 0.00e+00

==> Primary Basis <==

-AO BASIS SET INFORMATION:
Name = CC-PVTZ
Total number of shells = 236
Number of primitives = 492
Number of AO = 730
Number of SO = 644
Maximum AM = 3
Spherical Harmonics = TRUE

-Contraction Scheme:
Atom Type All Primitives // Shells:

   1     C     18s 5p 2d 1f // 4s 3p 2d 1f
   2     C     18s 5p 2d 1f // 4s 3p 2d 1f
   3     C     18s 5p 2d 1f // 4s 3p 2d 1f
   4     C     18s 5p 2d 1f // 4s 3p 2d 1f
   5     N     18s 5p 2d 1f // 4s 3p 2d 1f
   6     C     18s 5p 2d 1f // 4s 3p 2d 1f
   7     C     18s 5p 2d 1f // 4s 3p 2d 1f
   8     H     5s 2p 1d // 3s 2p 1d
   9     H     5s 2p 1d // 3s 2p 1d
  10     H     5s 2p 1d // 3s 2p 1d
  11     H     5s 2p 1d // 3s 2p 1d
  12     H     5s 2p 1d // 3s 2p 1d
  13     H     5s 2p 1d // 3s 2p 1d
  14     H     5s 2p 1d // 3s 2p 1d
  15     H     5s 2p 1d // 3s 2p 1d
  16     H     5s 2p 1d // 3s 2p 1d
  17     H     5s 2p 1d // 3s 2p 1d
  18     H     5s 2p 1d // 3s 2p 1d
  19     H     5s 2p 1d // 3s 2p 1d
  20     H     5s 2p 1d // 3s 2p 1d
  21     H     5s 2p 1d // 3s 2p 1d
  22     N     18s 5p 2d 1f // 4s 3p 2d 1f
  23     C     18s 5p 2d 1f // 4s 3p 2d 1f
  24     N     18s 5p 2d 1f // 4s 3p 2d 1f
  25     C     18s 5p 2d 1f // 4s 3p 2d 1f
  26     N     18s 5p 2d 1f // 4s 3p 2d 1f
  27     C     18s 5p 2d 1f // 4s 3p 2d 1f
  28     H     5s 2p 1d // 3s 2p 1d
  29     H     5s 2p 1d // 3s 2p 1d
  30     F     18s 5p 2d 1f // 4s 3p 2d 1f

==> Pre-Iterations <==

Irrep   Nso     Nmo     Nalpha   Nbeta   Ndocc  Nsocc

 A        644     644       0       0       0       0

Total     644     644      54      53      53       1

=> Loading Basis Set <=

Role: BASIS
Keyword: BASIS
Name: CC-PVTZ
atoms 1-4, 6-7, 23, 25, 27 entry C          line   171 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 5, 22, 24, 26        entry N          line   209 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 8-21, 28-29          entry H          line    20 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 30                   entry F          line   285 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs

OEINTS: Overlap, kinetic, potential, dipole, and quadrupole integrals
stored in file 35.

==> Integral Setup <==

=> Loading Basis Set <=

Role: BASIS
Keyword: BASIS
Name: CC-PVTZ
atoms 1-4, 6-7, 23, 25, 27 entry C          line   171 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 5, 22, 24, 26        entry N          line   209 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 8-21, 28-29          entry H          line    20 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 30                   entry F          line   285 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs
atoms 30                   entry F          line   285 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz.gbs

=> Loading Basis Set <=

Role: JKFIT
Keyword: DF_BASIS_SCF
Name:
atoms 1-4, 6-7, 23, 25, 27 entry C          line   124 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz-jkfit.gbs
atoms 5, 22, 24, 26        entry N          line   176 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz-jkfit.gbs
atoms 8-21, 28-29          entry H          line    50 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz-jkfit.gbs
atoms 30                   entry F          line   280 file /apps/psicode/4.0-Nov19/share/psi/basis/cc-pvtz-jkfit.gbs

hokru · April 28, 2016, 2:48pm

This looks like a kernel (operation system) error and has nothing to do with PSI4.
Try staying at or below 60GB. RSS should be “resident set size”.

Even 60GB is plenty for your job!