Running Psi4 (as a shared library) using (only) valid python

Hello all,

I am slowly working through the source code and hacking away to write a python script which will, using only valid python, run psi4. I want to do this because I need another sub-instance of embedded python in another piece of code to be able to parse the file without throwing syntax errors (or forcing me to write a parsing routine for it). It is possible someone has already done something similar or can point me to anything useful (besides the standard documentation).

Currently in my hacking I’ve tried to feed the input info as a string to the inputparser’s process_input function which seg faults. I’m assuming that is because there are some initial globals and other values or objects that need to be setup before invoking the process_input function. Ideally I would skip the parser and be able to pass python style objects in to define positions and type etc. Any quick tips or a pointer to useful info would be greatly appreciated.

I am no specialist in the Python side of Psi4, but you may want to check out Psi4Numpy, which can already do a lot of things with Psi4 from Python.

Do you already have psi4.so compiled? And are you Mac or Linux?

@loriab Yes the shared object is compiled and working I just don’t know enough about the flow of Psi4 to do what I want. I have also compiled the binary and can run jobs through the binary without issue. I am using Linux(ubuntu).

@jgonthier psithon is great for running python through the Psi4 interpreter but what I want to do is run Psi4 (using the shared object) through python.

Ok, nice. Over the first hurdle. Now, do you want to be feeding a string to process_input, or would you rather do pure python (slightly harder for the quantum-chemist oriented but easier for the python-oriented).

The former I had working once-upon-a-time, and I could get it working again. For the latter, try running the psi4 executable with the psi4 -v flag. It’ll print out at the top the actual python that the exe is running.

@loriab the -v flag is perfect. Exactly what I needed. Thank you. And to answer your question I’m going to try to stick to pure python if possible.

Ok, great. Write back if you have any questions. And this might be helpful, if you haven’t found it already: http://psicode.org/psi4manual/master/autodoc_psimod.html .

So as a starting point I ran the provided scf2 sample input with the verbose output. using the python calls shown in the output as a template I ran the following:

import molutil
import psi4
import p4const
import p4util
psi4_io = psi4.IOManager.shared_object()
psi4.efp_init()
molutil.geometry("""
0 1
H
H 1 0.74
""","blank_molecule_psi4_yo")
#psi4.set_memory(250000000)
psi4.efp_init()
h2o = molutil.geometry("""
O
H 1 1.0
H 1 1.0 2 104.5
""","h2o")
psi4.IO.set_default_namespace("h2o")
psi4.set_global_option_python("BASIS", "cc-pVTZ")
psi4.set_global_option_python("SCF_TYPE", "df")
psi4.set_global_option_python("E_CONVERGENCE", 10)
thisenergy = psi4.energy('scf')

I ran into the following problems:
1.) call to set_memory results in seg fault no exception thrown
2.) The call to energy won’t work as energy(‘scf’) is defined by driver but if I import driver i get the following: Error: option DFT_DISPERSION_PARAMETERS is not in the list of available options. I also get the same error if I try to import gaussian_n (to bypass driver). Same error with importing aliases.

Unfortunately I do not have a clean install to try this out with but I did find at least one change that needs to happen. In

src/bin/psi4/python.cc

beginning around line 1263 you’ll find:

BOOST_PYTHON_MODULE (psi4)
{
#if defined(MAKE_PYTHON_MODULE)
    // Initialize the world communicator
    WorldComm = boost::shared_ptr<LibParallel::ParallelEnvironment>(
      new LibParallel::ParallelEnvironment(0, 0));

    // Setup the environment
    Process::arguments.initialize(0, 0);
    Process::environment.initialize(); // Defaults to obtaining the environment from the global environ variable

    // There is only one timer:
    timer_init();

I know for certain that Process::arguments.initialize(0,0); needs to be removed as it doesn’t exist:

    // Process::arguments.initialize(0, 0);

When dealing with a shared library the linker doesn’t report undefined references until the library is loaded. Give that a try. It explains why DFT_DISPERSION_PARAMETERS isn’t available as program options are set a few lines below the above code.

@jturney Thanks for the suggestion but the behavior remains unchanged. In fact that #if defined(MAKE_PYTHON_MODULE) conditional is not passed (by the compiler even though the makefile says to add the property when make psi4so is called) though it is tested (confirmed by prints). So all setup seems to be handled by main in psi4.cc which has a #if !defined(MAKE_PYTHON_MODULE) and does not contain a process::arguments.initialize(0,0); invocation.

I can’t investigate immediately, but those shoulbe be psi4.set_global_option("BASIS", "cc-pVTZ") not psi4.set_global_option_python("BASIS", "cc-pVTZ") .

@loriab Thank you very much for your help. If I use set_global_option() I get: Error: option BASIS is not contained in the list of available options (I get this error for any property I try to set). I looked in the documentation and noticed a _python variant of the function call and tried it on a whim. I receive no error with the _python variant. Though since you pointed this out I ran a test and fed the _python function a random string as property name and it didn’t complain which is troubling. I will look for what this function actually does in the src.

It seems the main issue right now is that MAKE_PYTHON_MODULE is acting like it isn’t defined for the psi4so build. I see where this property is set using the following command

set_property(TARGET psi4so PROPERTY COMPILE_DEFINITIONS MAKE_PYTHON_MODULE)

in Psi4/psi4public/src/bin/psi4/CMakeLists.txt

The actual file invoked when make psi4so is called I believe is Psi4/psi4public/objdir/CMakeFiles/Makefile2 which calls the build.make file in /objdir/src/bin/psi4/CMakeFiles/psi4so.dir the file containing the make flags (which is included) does indeed contain a definition for MAKE_PYTHON_MODULE.

Despite all this when I invoke functions from the shared library I see code execution up until the ifdef conditionals but not afterwards.

Well, I confirm that I get exactly what you describe on Mac, too. It didn’t used to do that, truly. I know someone who’s used the shared library more recently, so I filed an issue (https://github.com/psi4/psi4/issues/251) to get those changes as a starting point for reviving the poor .so.

Ok, the good news is that thanks to Ben Prichard’s model, I’ve got a psi4.so working again that doesn’t segfault on options. Will check it in tomorrow night.

1 Like

Great. Has the working version already been checked in? I’m not seeing anything about it in my git log.

Sorry, I fell asleep mid-compile. Really will get to it tonight. If you’d rather get working again and don’t mind applying changes yourself, here’s the diff.

diff --git a/cmake/ConfigBoost.cmake b/cmake/ConfigBoost.cmake
index 9d2f55d..103fd95 100644
--- a/cmake/ConfigBoost.cmake
+++ b/cmake/ConfigBoost.cmake
@@ -11,7 +11,7 @@ list(APPEND needed_components filesystem python regex serialization system timer
 if(ENABLE_MPI)
    list(APPEND needed_components mpi)
 endif()
-set(Boost_USE_STATIC_LIBS    ON)
+set(Boost_USE_STATIC_LIBS    OFF) #ON)
 set(Boost_USE_MULTITHREADED  ON)
 set(Boost_USE_STATIC_RUNTIME OFF)
 if(ENABLE_UNIT_TESTS)
diff --git a/src/bin/psi4/CMakeLists.txt b/src/bin/psi4/CMakeLists.txt
index be916c3..5a69f6f 100644
--- a/src/bin/psi4/CMakeLists.txt
+++ b/src/bin/psi4/CMakeLists.txt
@@ -5,10 +5,16 @@ add_custom_target(update_version
     COMMAND ${CMAKE_COMMAND} -E remove ${CMAKE_CURRENT_BINARY_DIR}/gitversion.h.tmp
     COMMENT "Generating Git info")
 
-add_library(versioned_code OBJECT version.cc python.cc psi_start.cc)
+add_library(versioned_code OBJECT version.cc python.cc psi_start.cc psi4.cc)
 add_dependencies(versioned_code update_version)
 add_dependencies(versioned_code mints)
 
+# Versioned code, but for standalone psi4.so (convenient to add psi4.cc to versioned list)
+add_library(versioned_code_so OBJECT EXCLUDE_FROM_ALL version.cc python.cc psi_start.cc psi4.cc)
+set_property(TARGET versioned_code_so PROPERTY COMPILE_DEFINITIONS MAKE_PYTHON_MODULE)
+add_dependencies(versioned_code_so update_version)
+add_dependencies(versioned_code_so mints)
+
 set(headers_list "")
 # List of headers
 list(APPEND headers_list script.h psi4.h)
@@ -24,8 +30,8 @@ list(SORT headers_list)
 
 set(sources_list "")
 # List of sources
-list(APPEND sources_list export_psio.cc export_mints.cc psi_stop.cc export_functional.cc export_oeprop.cc export_plugins.cc export_blas_lapack.cc psi4.cc expor
-list(APPEND sources_list $<TARGET_OBJECTS:versioned_code>)
+list(APPEND sources_list export_psio.cc export_mints.cc psi_stop.cc export_functional.cc export_oeprop.cc export_plugins.cc export_blas_lapack.cc export_cubefi
+#list(APPEND sources_list $<TARGET_OBJECTS:versioned_code>)
 
 # If you want to remove some sources specify them explictly here
 if(DEVELOPMENT_CODE)
@@ -34,6 +40,10 @@ else()
    list(REMOVE_ITEM sources_list "")
 endif()
 
+# Compile sources_list into an object library
+add_library(psi4_objlib OBJECT ${sources_list})
+add_dependencies(psi4_objlib versioned_code)
+
 
 get_property(from_src_bin GLOBAL PROPERTY PSILIB)
 get_property(from_src_lib GLOBAL PROPERTY LIBLIST)
@@ -51,7 +61,8 @@ list(APPEND
      )
 
 # Executable psi4
-add_executable(psi4 ${sources_list})
+#add_executable(psi4 ${sources_list})
+add_executable(psi4 $<TARGET_OBJECTS:psi4_objlib> $<TARGET_OBJECTS:versioned_code>)
 add_dependencies(psi4 update_version)
 if(CUSTOM_BOOST_BUILD)
    add_dependencies(psi4 custom_boost)
@@ -65,11 +76,22 @@ if(ENABLE_PCMSOLVER)
 endif()
 
 # standalone python module psi4.so
-add_executable(psi4so EXCLUDE_FROM_ALL ${sources_list})
+#add_executable(psi4so EXCLUDE_FROM_ALL ${sources_list})
+add_executable(psi4so EXCLUDE_FROM_ALL $<TARGET_OBJECTS:psi4_objlib> $<TARGET_OBJECTS:versioned_code_so>)
+#add_library(psi4so SHARED EXCLUDE_FROM_ALL $<TARGET_OBJECTS:psi4_objlib> $<TARGET_OBJECTS:versioned_code_so>)
 if(CUSTOM_BOOST_BUILD)
    add_dependencies(psi4so custom_boost)
 endif()
-set_property(TARGET psi4so PROPERTY COMPILE_DEFINITIONS MAKE_PYTHON_MODULE)
+#set_property(TARGET psi4so PROPERTY COMPILE_DEFINITIONS MAKE_PYTHON_MODULE)
 set_property(TARGET psi4so PROPERTY LINK_FLAGS "-shared")
 set_property(TARGET psi4so PROPERTY OUTPUT_NAME "psi4${CMAKE_EXECUTABLE_SUFFIX}.so")
 target_link_libraries(psi4so "${LINKLIBS}")
+
+if(ENABLE_CHEMPS2)
+    target_link_libraries(psi4so CHEMPS2::CHEMPS2)
+endif()
+if(ENABLE_PCMSOLVER)
+    target_link_libraries(psi4so PCMSolver::PCMSolver)
+endif()
+#INSTALL(TARGETS psi4so LIBRARY DESTINATION ${CMAKE_INSTALL_PREFIX}/lib)
+INSTALL(TARGETS psi4so RUNTIME DESTINATION ${CMAKE_INSTALL_PREFIX}/bin)
diff --git a/src/bin/psi4/psi_start.cc b/src/bin/psi4/psi_start.cc
index f75f06b..cca01f8 100644
--- a/src/bin/psi4/psi_start.cc
+++ b/src/bin/psi4/psi_start.cc
@@ -172,9 +172,9 @@ int psi_start(int argc, char *argv[])
                 fprefix = optarg;
                 break;
 
-            case 'r': // -r or --restart
-                restart_id = optarg;
-                break;
+//            case 'r': // -r or --restart
+//                restart_id = optarg;
+//                break;
 
             case 'v': // -v or --verbose
                 verbose = true;
@@ -239,8 +239,8 @@ int psi_start(int argc, char *argv[])
         ofname = Process::environment("PSI_OUTPUT");
     if (fprefix.empty() && Process::environment("PSI_PREFIX").size())
         fprefix = Process::environment("PSI_PREFIX");
-    if (restart_id.empty() && Process::environment("PSI_RESTART").size())
-        restart_id = Process::environment("PSI_RESTART");
+//    if (restart_id.empty() && Process::environment("PSI_RESTART").size())
+//        restart_id = Process::environment("PSI_RESTART");
 
     /* if some arguments still not defined - assign default values */
     if (ifname.empty()) ifname = "input.dat";
@@ -287,18 +287,15 @@ int psi_start(int argc, char *argv[])
             return(PSI_RETURN_FAILURE);
         }
     }
-#endif
-
-#if defined(MAKE_PYTHON_MODULE)
-        outfile = boost::shared_ptr<PsiOutStream>(new PsiOutStream());
+    if(ofname == "stdout"){
+        outfile=boost::shared_ptr<PsiOutStream>(new PsiOutStream());
+    }
+    else{
+       outfile=boost::shared_ptr<PsiOutStream>
+          (new OutFile(ofname,(append?APPEND:TRUNCATE)));
+    }
 #else
-        if(ofname == "stdout"){
-            outfile=boost::shared_ptr<PsiOutStream>(new PsiOutStream());
-        }
-        else{
-           outfile=boost::shared_ptr<PsiOutStream>
-              (new OutFile(ofname,(append?APPEND:TRUNCATE)));
-        }
+    outfile = boost::shared_ptr<PsiOutStream>(new PsiOutStream());
 #endif
 
     //if(debug)
diff --git a/src/bin/psi4/python.cc b/src/bin/psi4/python.cc
index a4afc33..cd1d979 100644
--- a/src/bin/psi4/python.cc
+++ b/src/bin/psi4/python.cc
@@ -922,7 +922,7 @@ void py_psi_revoke_local_option_changed(std::string const& module, std::string c
     data.dechanged();
 }
 
-object py_psi_get_local_option(std::string const& module, std::string const& key)
+boost::python::object py_psi_get_local_option(std::string const& module, std::string const& key)
 {
     string nonconst_key = key;
     Process::environment.options.set_current_module(module);
@@ -930,35 +930,35 @@ object py_psi_get_local_option(std::string const& module, std::string const& key
     Data& data = Process::environment.options.get_local(nonconst_key);
 
     if (data.type() == "string" || data.type() == "istring")
-        return str(data.to_string());
+        return boost::python::str(data.to_string());
     else if (data.type() == "boolean" || data.type() == "int")
-        return object(data.to_integer());
+        return boost::python::object(data.to_integer());
     else if (data.type() == "double")
-        return object(data.to_double());
+        return boost::python::object(data.to_double());
     else if (data.type() == "array")
-        return object(data.to_list());
+        return boost::python::object(data.to_list());
 
     return object();
 }
 
-object py_psi_get_global_option(std::string const& key)
+boost::python::object py_psi_get_global_option(std::string const& key)
 {
     string nonconst_key = key;
     Data& data = Process::environment.options.get_global(nonconst_key);
 
     if (data.type() == "string" || data.type() == "istring")
-        return str(data.to_string());
+        return boost::python::str(data.to_string());
     else if (data.type() == "boolean" || data.type() == "int")
-        return object(data.to_integer());
+        return boost::python::object(data.to_integer());
     else if (data.type() == "double")
-        return object(data.to_double());
+        return boost::python::object(data.to_double());
     else if (data.type() == "array")
-        return object(data.to_list());
+        return boost::python::object(data.to_list());
 
     return object();
 }
 
-object py_psi_get_option(std::string const& module, std::string const& key)
+boost::python::object py_psi_get_option(std::string const& module, std::string const& key)
 {
     string nonconst_key = key;
     Process::environment.options.set_current_module(module);
@@ -966,13 +966,13 @@ object py_psi_get_option(std::string const& module, std::string const& key)
     Data& data = Process::environment.options.use_local(nonconst_key);
 
     if (data.type() == "string" || data.type() == "istring")
-        return str(data.to_string());
+        return boost::python::str(data.to_string());
     else if (data.type() == "boolean" || data.type() == "int")
-        return object(data.to_integer());
+        return boost::python::object(data.to_integer());
     else if (data.type() == "double")
-        return object(data.to_double());
+        return boost::python::object(data.to_double());
     else if (data.type() == "array")
-        return object(data.to_list());
+        return boost::python::object(data.to_list());
 
     return object();
 }
@@ -1268,7 +1268,7 @@ BOOST_PYTHON_MODULE (psi4)
           new LibParallel::ParallelEnvironment(0, 0));
 
     // Setup the environment
-    Process::arguments.initialize(0, 0);
+    //Process::arguments.initialize(0, 0);
     Process::environment.initialize(); // Defaults to obtaining the environment from the global environ variable
 
     // There is only one timer:

Thanks! No rush, I was just worried that I wasn’t using git properly (as I am pretty new to git)

When https://github.com/psi4/psi4/pull/268 goes through, you should be able to get your psi4.so.

Thanks for all the help!

Everything seems to be working (I can run scf by feeding it a ref_wfn and get correct energies etc). I have two problems now:

First: strange things happen to the geometry when I try to modify the geometry by Molecule.geometry() followed by Molecule.set_geometry() the geometries get re oriented but not correctly (“bond” distances change as do angles by ~30%) also the no_com and no_reorient settings don’t seem to have any effect on this change. I am using Molecule.geometry() and then creating a numpy array from this matrix object using np.asarray, modifying this array, and then passing back the original object via Molecule.set_geometry()

Second:I cannot seem to exit cleanly. I used the io manager to clean up the temporary files in /tmp and I assumed the finalize() fxn is what I wanted but while this gives me PSI4 exiting successfully message (from psi stop) it causes a seg fault because it assumes we are not using psi4 as a shared library.