Crc32 undefined symbol at runtime when built with PCMSolver (gcc 4.9.4)

If I build psi4 (current git) without PCMSolver, all works fine (make test completes with no failures).

However, if I enable PCMSolver, it still builds fine but if I run it, psi4 bombs out with an undefined symbol for crc32 in core.so.

I’m hoping it’s a quick fix of a known issue but, if not, I’m happy to do some digging to look for the cause.

This is building with gcc 4.9.4, cmake 3.4.1, and linking against MKL. Also, I used a new build directory for each.

Cheers,

Laz

It could be that zlib is not installed, on Ubuntu sudo apt-get install zlib1g-dev should do it.
If this is indeed the fix, I am very surprised that the missing dependency is not caught earlier.

This is on centos 7.3. zlib and zlib-devel both seem to be installed. In fact, a very short chunk of c that just calls crc32 compiles and runs fine (as long as I use -lz). It’s using “modules” so that various different versions of things are available: there is a module for zlib but I think that just pulls in a newer version than the centos default version (and my simple crc32 test program ran fine without using the zlib module).

It built and ran fine with PCMSolver on a different box (under Debian with gcc 6.3 and MKL).

I was expecting “ldd core.so” to point to libz.so somewhere along the line but it doesn’t, even on the working machine.

I’m rebuilding from scratch with one make process so that the build output makes a bit more sense. Once that’s finished, I’ll compare the build output and various bits that cmake leaves behind with that from the working version on the other machine…

-DBUILD_SHARED_LIBS=ON is something to try toggling. Default is OFF, but I haven’t built it that way in a long time.

Hmmm… I changed BUILD_SHARED_LIBS to ON in CMakeLists.txt, removed objdir, reran cmake, and rebuilt. I’m not convinced it actually changed anything! core.so is still huge and I only saw libpcm.a and not libpcm.so as I was expecting (if I’ve understood the option correctly!).
It still failed because crc32 was undefined.

However, editing

psi4-core-prefix/src/psi4-core-build/src/CMakeFiles/core.dir/link.txt

by appending “-lz” fixed it! I’ve not run a full “make test” but ctest within tests/pcmsolver and a few selected at random worked fine with this version including PCMSolver.

Checking link.txt on the other system (where it builds fine with PCMSolver), there is no “-lz” so it must be pulling in zlib implicitly somehow.

Building with PCMSolver:

System 1 (gcc 6.3.0; binutils 2.27: builds and runs fine:
“ldd core.so” output has no mention of libz.so but strace shows that libz.so is loaded at runtime!

System 2 (gcc 4.9.4; binutils 2.25.1): builds fine but fails to run with undefined crc32:
“ldd core.so” output has no mention of libz.so

System 2 with “-lz” added to link.txt: builds and runs fine:
“ldd core.so” output has libz.so

I.e., with the newer compiler and binutils (System 1), zlib seems to be pulled in implicitly at runtime; with the older compiler and binutils (System 2), zlib needs to be explicitly linked into core.so.

It looks like I’ve got it running happily on both systems not but it might be handy to work out why this has happened in case it breaks on other systems.

Glad it’s working for you, though it does sound like there’s a mishandling of libz somewhere, perhaps just for static libpcm.

I think you are understanding BUILD_SHARED_LIBS correctly, and I’m surprised you only saw libpcm.a. Could there have been an installed pcmsolver or psi4+pcmsolver at CMAKE_PREFIX_PATH? And did the reports on externals like below show libpcm.a or libpcm.so?

-- Found LAPACK MKL: /theoryfs2/common/software/intel2016/compilers_and_libraries_2016.3.210/linux/mkl/lib/intel64/libmkl_lapack95_lp64.a;...
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found HDF5: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libhdf5.so;... (found version 1.8.17)
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Found CheMPS2: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libchemps2.so.2 (found version 1.8.3)
-- Found libefp: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libefp.so (found version 1.4.1)
-- Found erd: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/liberd.so (found version 3.0.6)
-- Found gdma: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libgdma.so (found version 2.2.6)
-- Found libint 7: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libint.so (found version 1.1.6)
-- Found PCMSolver: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libpcm.so.1 (found version 1.1.9)
-- Suitable pybind11 could not be located, Building pybind11 instead.
-- Using CheMPS2: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libchemps2.so.2 (version 1.8.3)
-- Disabled dkh
-- Using libefp: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libefp.so (version 1.4.1)
-- Using erd: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/liberd.so (version 3.0.6)
-- Using gdma: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libgdma.so (version 2.2.6)
-- Using libint 7: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libint.so (version 1.1.6)
-- Using PCMSolver: /scratch/cdsgroup/conda-builds/psi4_1490372828952/_b_env_placehold_placehold_pla/lib/libpcm.so.1 (version 1.1.9)

I’ll keep in mind that libz is a potential bug and tackle it if I run into it locally.

I have the same problem now (first time re-enabling PCMsolver/DHK since a long time) and using “-lz” as indicated fixes it, but for my system this:

does not hold true. I still have to link zlib explicitly. I did not re-build everything from scratch though.

Definitely confirmed with libpcm.a (static). Explicit linking to -lz works (as you’ve seen) or enabling CheMPS2, as the hdf5 brings in libz.

I think this should fix it. Thanks!

I hadn’t spotted that it was finding libpcm.a under CMAKE_PREFIX_PATH and so it didn’t bother to rebuild it. Will rebuild after a more thorough cleanout on Monday…

I can confirm that it now builds (and more importantly!) runs fine on the failing system without the need to pass “-lz” by hand. This is including PCMSolver and ChemPS2 and building as shared libraries.

It did take a bit of fighting to stop it finding previously built static libraries and old cmake files under CMAKE_PREFIX_PATH (i.e., I ended up going through and deleting them by hand!). Obviously, it’s a good thing to reuse anything that has already been compiled but is there any way for force cmake to rebuild everything from scratch (for testing purposes, etc.)?

Great that it’s working, and thanks for testing.

Yes, CMake’s hard to reign in. Possible NO_DEFAULT_PATH or some of the other NOs from here would do it. But for practical purposes, I just point CMAKE_PREFIX_PATH to an empty dir, rm -rf objdir/external, rm -rf objdir/stage/.../share/cmake/PCMSolver, rm objdir/stage/.../share/libpcm*, re-issue the cmake command that created objdir (to regenerate objdir/external), and rebuild. It’ll only be a partial rebuild of PCMSolver. Generally I find it simpler to advise ppl to blow the whole thing away.

@laz The problem you reported should not appear with the current master.