Configuration and compilation are completed correctly, ctest passed with all PASSED. I installed PSI4 into my home dir. Some environments I added to .bashrc:
It’s OK when I try to launch the test job from the public server. But as I try to use cluster nodes via SLURM, I receive the next message:
Traceback (most recent call last):
File “/home/ovsyannikov_d/psi/bin/psi4”, line 158, in
import psi4
File “/home/ovsyannikov_d/psi/lib//psi4/init.py”, line 80, in
from .driver import endorsed_plugins
File “/home/ovsyannikov_d/psi/lib//psi4/driver/init.py”, line 30, in
from . import dependency_check
File “/home/ovsyannikov_d/psi/lib//psi4/driver/dependency_check.py”, line 44, in
raise ImportError(msg)
ImportError:
NumPy is a runtime requirement for Psi4. Please install NumPy to proceed.
So if I ssh to the cluster node and launch the job, I receive the next message:
Traceback (most recent call last):
File “/home/ovsyannikov_d/psi/bin/psi4”, line 158, in
import psi4
File “/home/ovsyannikov_d/psi/lib//psi4/init.py”, line 58, in
raise ImportError(“{0}\nLikely cause: GCC >= 4.9 not in [DY]LD_LIBRARY_PATH”.format(err))
ImportError: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.8’ not found (required by /home/ovsyannikov_d/psi/lib//psi4/core.so)
Well, but on nodes there are no numpy, no modules, no *-devel files. Same error I get as I ssh on node and launch import psi4 in python shell.
Is it any cmake options I missed, or anything I did incorrectly? Or, PSI4 is not designed for clusters? Any suggestion appreciated. Thank you in advance.
At first guess: /usr/ usually points to the local file system so while Python on your head node has NumPy your compute nodes do not. Is there some sort of Python module that you can load so that Python comes from shared resources and not local?
Another thing to try is which python while on a compute node to check out where the local Python is coming from, its its pointing somewhere besides /usr/ you can try python psi4 input.dat which will launch python without the /usr/.../python shebang.
On compute nodes there is no numpy module. So I tried to copy it from /usr to $HOME with no success. Numpy works, but PSI4 doesn’t. With same error, GCC>=4.9 …
which pyhton on compute node returns /usr/bin/python, so it’s OK.
Maybe, I have to provide libstdc++ via my $HOME dir?
I trust you’re on Psi4 1.1 (late May 2017 or later)?
Some oddities about the compilation:
there is no ENABLE_MPI b/c no MPI
OpenMP triggered by ENABLE_OPENMP but nevermind b/c its on by default
PSIPATH doesn’t do anything for a default installation. Were you thinking about adding $HOME/psi/bin to envvar PATH?
no setting PSIDATADIR unless you’re an expert developer. this is admittedly a contrast from 1.0
PSI_SCRATCH you want pointing to local hard disk or wherever your cluster wants scratch written.
Now more generally, I agree with Daniel that your cluster really must provide a way to make support software like numpy and gcc libraries on compute nodes. One thing to do is run ldd $HOME/psi/lib/psi4/core.so on the login node and then again on the compute node to see what’s not being found.
Yes, I just cloned PSI4 from github last friday. So I believe it’s 1.1 version.
I tried ldd PSI4 core.so from some node and get the next message:
./core.so: /usr/lib64/libstdc++.so.6: version CXXABI_1.3.8' not found (required by ./core.so) ./core.so: /usr/lib64/libstdc++.so.6: version GLIBCXX_3.4.20’ not found (required by ./core.so)
./core.so: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21’ not found (required by ./core.so)
linux-vdso.so.1 => (0x00007fffe5e37000)
libxc.so => /common/home/ovsyannikov_d/psi/lib/psi4/./…/libxc.so (0x00002b908bc35000)
libgdma.so => /common/home/ovsyannikov_d/psi/lib/psi4/./…/libgdma.so (0x00002b908bf52000)
libderiv.so => /common/home/ovsyannikov_d/psi/lib/psi4/./…/libderiv.so (0x00002b908c202000)
libint.so => /common/home/ovsyannikov_d/psi/lib/psi4/./…/libint.so (0x00002b909a4d1000)
libmkl_rt.so => /common/intel_2017/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64/libmkl_rt.so (0x00002b909d511000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00002b909db01000)
libm.so.6 => /usr/lib64/libm.so.6 (0x00002b909dd1d000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x00002b909e01f000)
libimf.so => not found
libsvml.so => not found
libirng.so => not found
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00002b909e223000)
libiomp5.so => not found
libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00002b909e52b000)
libintlc.so.5 => not found
libc.so.6 => /usr/lib64/libc.so.6 (0x00002b909e741000)
/lib64/ld-linux-x86-64.so.2 (0x00002b90897e7000)
libimf.so => not found
libsvml.so => not found
libirng.so => not found
libintlc.so.5 => not found
libifport.so.5 => not found
libifcoremt.so.5 => not found
libimf.so => not found
libsvml.so => not found
libiomp5.so => not found
libintlc.so.5 => not found
libimf.so => not found
libsvml.so => not found
libirng.so => not found
libintlc.so.5 => not found
libimf.so => not found
libsvml.so => not found
libirng.so => not found
libintlc.so.5 => not found
And on login node PSI4 doesn’t work for now with the next error:
Traceback (most recent call last):
File “/home/ovsyannikov_d/psi/bin/psi4”, line 158, in
import psi4
File “/home/ovsyannikov_d/psi/lib//psi4/init.py”, line 80, in
from .driver import endorsed_plugins
File “/home/ovsyannikov_d/psi/lib//psi4/driver/init.py”, line 31, in
from psi4.driver.molutil import *
File “/home/ovsyannikov_d/psi/lib//psi4/driver/molutil.py”, line 36, in
from psi4.driver.inputparser import process_pubchem_command, pubchemre
File “/home/ovsyannikov_d/psi/lib//psi4/driver/inputparser.py”, line 44, in
from psi4.driver import pubchem
File “/home/ovsyannikov_d/psi/lib//psi4/driver/pubchem.py”, line 53, in
from urllib.request import urlopen, Request
ImportError: No module named request
Conda’s binary works well… but why? maybe, I can link conda’s libraries to the psi4 bin compiled by me? I am so confused…
If you want to get the psi4 you compiled yourself working on the remote nodes, you’ll have to find out how your cluster wants to deliver “system” software to the nodes. Modules, e.g., module list, are a common way, in which case you’ll have to select enough modules to satisfy all those ldd "not found"s. You may have to request through PBS (or whatever your queue) that your jobs be routed to nodes where modules are available.
I thought you said earlier that jobs were working just fine on the cluster head node? The request import error suggests its using a different python than compiled with. You can see the one compiled with as the top line of $HOME/psi/bin/psi4.
The conda works because it brings its dependencies along with it, including python and libgcc_s. It only looks to / for very basic libraries, so it’s not hitting the bleak software landscape of your cluster’s compute nodes.
I faced with another problem: conda’s binary doesn’t work in parallel. I tried to export OMP_NUM_THREADS=16, using command psi4 -n 16, put in input set_num_threads(16) with no effect. Psi4 uses only 1 thread on login node and on compute nodes. If I ssh to compute node and start psi4 from it, the same effect: only 1 process with name ‘psi4’. Ldd conda’s core.so said all libraries are present. Is it some kind of OpenMP error? And what if I send psi4 task to more than 1 node? Would it use all the nodes?
Negative on sending psi4 task to more than 1 node. We’re strictly intra-node parallel.
We’ve had a few issues with math libs and parallelism lately here and here. What’s the version of your binary (psi4 --version)? And does running this show speedup? And what’s the output of python thread.py and psi4 thread.py below?
thread.py
import os
import time
# none for psithon
# good psiapi
import numpy as np
import psi4
# bad psiapi
#import psi4
#import numpy as np
def test_threaded_blas():
threads = 6
times = {}
size = [200, 500, 2000, 4000]
threads = [1, threads]
for th in threads:
psi4.set_num_threads(th)
for sz in size:
nruns = max(1, int(1.e10 / (sz ** 3)))
a = psi4.core.Matrix(sz, sz)
b = psi4.core.Matrix(sz, sz)
c = psi4.core.Matrix(sz, sz)
tp4 = time.time()
for n in range(nruns):
c.gemm(False, False, 1.0, a, b, 0.0)
retp4 = (time.time() - tp4) / nruns
tnp = time.time()
for n in range(nruns):
np.dot(a, b, out=np.asarray(c))
retnp = (time.time() - tnp) / nruns
#retnp = 1.0
print("Time for threads %2d, size %5d: Psi4: %12.6f NumPy: %12.6f" % (th, sz, retp4, retnp))
if sz == 4000:
times["p4-n{}".format(th)] = retp4
times["np-n{}".format(th)] = retnp
assert psi4.get_num_threads() == th
rat1 = times["np-n" + str(threads[-1])] / times["p4-n" + str(threads[-1])]
rat2 = times["p4-n" + str(threads[0])] / times["p4-n" + str(threads[-1])]
print(" NumPy@n%d : Psi4@n%d ratio (want ~1): %.2f" % (threads[-1], threads[-1], rat1))
print(" Psi4@n%d : Psi4@n%d ratio (want ~%d): %.2f" % (threads[0], threads[-1], threads[-1], rat2))
os.system('grep mkl /proc/%d/maps' % os.getpid())
if __name__ == '__main__':
test_threaded_blas()
1.1 binary uses baked-in MKL that we’re not sure adjusts with processor instruct sets. You might get a little better out of the nightly build (-c psi4/label/dev) which does dynamic linking the the runtime lib. But looks like you’re up and running.
Typically, we run 4–8 threads, if you’re considering putting more than one job on a node.
maybe, I can link conda’s libraries to the psi4 bin compiled by me? I am so confused…