Forrtl: severe (174): SIGSEGV, segmentation fault occurred when calculating DFT energy

1

Hey Everyone!
I’am trying to efficiently compute DFT energy for different molecules using a multiprocessing Queue and a process Pool.
First, I put ase.Atoms objects into a queue then compute the energy using the apply_async method of Pool:

def calculate_dft_energy_queue(queue, n_threads, M):
    m = Manager()

    # Create multiprocessing queue
    q = m.Queue()

    # Create a group of parallel writers and start them
    for elem in queue:
        q.put(elem)

    # Create multiprocessing pool
    p = Pool(n_threads)

    # Calculate energy for every item in the queue.
    # To mitigate large variance in dft computation times  
    # for different molecules the size of the queue
    # must be significantly larger then n_threads.
    energy_calculators = []
    for i in range(len(queue)):
        energy_calculators.append(p.apply_async(calculate_dft_energy_item, (q, M)))

    # Wait for the asynchrounous reader threads to finish
    results = [ec.get() for ec in energy_calculators]
    results = sorted(results, key=lambda x:x[0])
    p.terminate()
    p.join()

    return results

The ‘calculate_dft_energy_item’ transforms ase.Atoms object into a psi4.core.Molecule object and computes DFT energy:

def calculate_dft_energy_item(queue):
    # Get molecule from the queue
    ase_atoms, _, idx = queue.get()
    molecule = atoms2psi4mol(ase_atoms)

    # Calculate DFT energy
    energy = get_dft_energy(molecule)
    print(energy)
    
    return (idx, energy)

The ‘get_dft_energy’ function:

def get_dft_energy(mol):
    try:
        energy = psi4.driver.energy(FUNCTIONAL_STRING, **{"molecule": mol, "return_wfn": False})
    except SCFConvergenceError as e:
        # Set energy to some threshold if SOSCF does not converge 
        # Multiply by 627.5 to go from Hartree to kcal/mol
        print("DFT optimization did not converge!")
        return -1000.0 * 627.5
    psi4.core.clean()
    return energy * 627.5

Then I get the following error:

Scratch directory: /tmp/

*** tstart() called on dlils-kad-2gpu
*** at Wed Jan 11 23:18:37 2023

   => Loading Basis Set <=

    Name: DEF2-SVP
    Role: ORBITAL
    Keyword: BASIS
    atoms 1                           entry O          line   130 file /mnt/2tb/tsypin/anaconda3/envs/md_env/share/psi4/basis/def2-svp.gbs 
    atoms 2-3, 5, 7-12, 15-18, 20, 22 entry C          line    90 file /mnt/2tb/tsypin/anaconda3/envs/md_env/share/psi4/basis/def2-svp.gbs 
    atoms 4                           entry S          line   328 file /mnt/2tb/tsypin/anaconda3/envs/md_env/share/psi4/basis/def2-svp.gbs 
    atoms 6, 13-14                    entry N          line   110 file /mnt/2tb/tsypin/anaconda3/envs/md_env/share/psi4/basis/def2-svp.gbs 
    atoms 19, 21                      entry F          line   150 file /mnt/2tb/tsypin/anaconda3/envs/md_env/share/psi4/basis/def2-svp.gbs 
    atoms 23-33                       entry H          line    15 file /mnt/2tb/tsypin/anaconda3/envs/md_env/share/psi4/basis/def2-svp.gbs 

   => WB97X-D: Empirical Dispersion <=

    Chai and Head-Gordon Dispersion Correction
    Chai, J.-D.; Head-Gordon, M. (2010), J. Chem. Phys., 132: 6615-6620

        s6 =       1.000000


         ---------------------------------------------------------
                                   SCF
               by Justin Turney, Rob Parrish, Andy Simmonett
                          and Daniel G. A. Smith
                              RKS Reference
                        1 Threads,   7629 MiB Core
         ---------------------------------------------------------

  ==> Geometry <==

    Molecular point group: c1
    Full point group: C1

    Geometry (in Angstrom), charge = 0, multiplicity = 1:

       Center              X                  Y                   Z               Mass       
    ------------   -----------------  -----------------  -----------------  -----------------
         O           -2.167867914424    -1.962588696865     0.370367951682    15.994914619570
         C           -1.726256554514    -0.922864193944    -0.149864404419    12.000000000000
         C           -0.271956842172    -0.809455125346    -0.508895445941    12.000000000000
         S            0.526637461753    -0.263406843404     1.068910004996    31.972071174400
         C            2.280446301536    -0.154979565447     0.781632164670    12.000000000000
         N            3.179060976186    -1.189271429443     0.968063754384    14.003074004430
         C            4.417025354437    -0.735245254807     0.672485096712    12.000000000000
         C            5.657329012151    -1.346148264937     0.649876622041    12.000000000000
         C            6.816844240870    -0.707928076461     0.290242965568    12.000000000000
         C            6.691746799445     0.629443764239    -0.005635488402    12.000000000000
         C            5.468467523927     1.250287407344     0.014465204913    12.000000000000
         C            4.328757323058     0.559945797839     0.342925027276    12.000000000000
         N            3.039526321674     0.909829599877     0.434147564831    14.003074004430
         N           -2.632811716867     0.148808968265    -0.430559993027    14.003074004430
         C           -4.013605931806     0.161578391499    -0.159427892945    12.000000000000
         C           -4.752150489321    -0.755717653395     0.562628815130    12.000000000000
         C           -6.105751076315    -0.649732549374     0.751885949582    12.000000000000
         C           -6.808369086732     0.392839250554     0.251208606169    12.000000000000
         F           -8.125833248812     0.518601656855     0.445109913212    18.998403162730
         C           -6.130013823267     1.352995821781    -0.466305919285    12.000000000000
         F           -6.842175993804     2.389013087478    -0.926445653242    18.998403162730
         C           -4.746361860982     1.236579912048    -0.648634870780    12.000000000000
         H           -0.163567605548     0.040635315824    -1.205382117854     1.007825032230
         H            0.137134578668    -1.750199516729    -0.854296458713     1.007825032230
         H            5.727494747895    -2.425659953942     0.895557383855     1.007825032230
         H            7.788019560707    -1.206406865598     0.302899204155     1.007825032230
         H            7.618696840841     1.188020575420    -0.269049915072     1.007825032230
         H            5.430489516889     2.327084697066    -0.263429594343     1.007825032230
         H            2.647767672380     1.867907229186     0.241010295921     1.007825032230
         H           -2.161907760510     0.997304042544    -0.894708832110     1.007825032230
         H           -4.250925854811    -1.628236536305     0.955001410991     1.007825032230
         H           -6.625096916710    -1.473899607494     1.326835665672     1.007825032230
         H           -4.246670231684     1.999091381070    -1.226783608770     1.007825032230

  Running in c1 symmetry.

  Rotational constants: A =      0.03642  B =      0.00222  C =      0.00214 [cm^-1]
  Rotational constants: A =   1091.76325  B =     66.70255  C =     64.15705 [MHz]
  Nuclear repulsion = 1693.023622131090860

  Charge       = 0
  Multiplicity = 1
  Electrons    = 164
  Nalpha       = 82
  Nbeta        = 82

  ==> Algorithm <==

  SCF Algorithm Type is DF.
  DIIS enabled.
  MOM disabled.
  Fractional occupation disabled.
  Guess Type is SAD.
  Energy threshold   = 1.00e-06
  Density threshold  = 1.00e-06
  Integral threshold = 1.00e-12

  ==> Primary Basis <==

  Basis Set: DEF2-SVP
    Blend: DEF2-SVP
    Number of shells: 167
    Number of basis functions: 367
    Number of Cartesian functions: 389
    Spherical Harmonics?: true
    Max angular momentum: 2

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
libgdma.so         00007FAA2E4F20DC  for__signal_handl     Unknown  Unknown
libpthread-2.27.s  00007FAABF1AC980  Unknown               Unknown  Unknown
libiomp5.so        00007FAABDF43776  __kmp_acquire_tic     Unknown  Unknown
libiomp5.so        00007FAABDF1443A  __kmpc_set_lock       Unknown  Unknown
core.cpython-39-x  00007FAA25DB4520  _ZN3psi8timer_onE     Unknown  Unknown
core.cpython-39-x  00007FAA1C6E9333  _ZN3psi5VBase10in     Unknown  Unknown
core.cpython-39-x  00007FAA1C6E8219  Unknown               Unknown  Unknown
core.cpython-39-x  00007FAA25FE0FB0  Unknown               Unknown  Unknown
core.cpython-39-x  00007FAA25FC5D7A  Unknown               Unknown  Unknown
core.cpython-39-x  00007FAA25FF4523  Unknown               Unknown  Unknown
core.cpython-39-x  00007FAA262C51BE  Unknown               Unknown  Unknown
core.cpython-39-x  00007FAA26302080  Unknown               Unknown  Unknown
core.cpython-39-x  00007FAA264BD4FD  Unknown               Unknown  Unknown
core.cpython-39-x  00007FAA264BD79B  Unknown               Unknown  Unknown
core.cpython-39-x  00007FAA2612B691  Unknown               Unknown  Unknown
python3.9          000055EE6A253E14  Unknown               Unknown  Unknown
python             000055EE6A20DCAF  _PyObject_MakeTpC     Unknown  Unknown
python3.9          000055EE6A18805B  Unknown               Unknown  Unknown
python             000055EE6A217A7A  _PyObject_Call        Unknown  Unknown
python3.9          000055EE6A17DAD9  Unknown               Unknown  Unknown
python3.9          000055EE6A228E9B  Unknown               Unknown  Unknown
core.cpython-39-x  00007FAA26519557  Unknown               Unknown  Unknown
python             000055EE6A20DCAF  _PyObject_MakeTpC     Unknown  Unknown
python             000055EE6A2ABD89  _PyEval_EvalFrame     Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python             000055EE6A2693C4  _PyFunction_Vecto     Unknown  Unknown
python             000055EE6A21795A  _PyObject_Call        Unknown  Unknown
python             000055EE6A2A99F0  _PyEval_EvalFrame     Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python             000055EE6A2693C4  _PyFunction_Vecto     Unknown  Unknown
python             000055EE6A21795A  _PyObject_Call        Unknown  Unknown
python             000055EE6A2A99F0  _PyEval_EvalFrame     Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python             000055EE6A2693C4  _PyFunction_Vecto     Unknown  Unknown
python             000055EE6A21795A  _PyObject_Call        Unknown  Unknown
python             000055EE6A2A99F0  _PyEval_EvalFrame     Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python             000055EE6A2693C4  _PyFunction_Vecto     Unknown  Unknown
python             000055EE6A21795A  _PyObject_Call        Unknown  Unknown
python             000055EE6A2A99F0  _PyEval_EvalFrame     Unknown  Unknown
python             000055EE6A269284  _PyFunction_Vecto     Unknown  Unknown
python3.9          000055EE6A1CFAE6  Unknown               Unknown  Unknown
python             000055EE6A269284  _PyFunction_Vecto     Unknown  Unknown
python             000055EE6A217A7A  _PyObject_Call        Unknown  Unknown
python             000055EE6A2A99F0  _PyEval_EvalFrame     Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python             000055EE6A269354  _PyFunction_Vecto     Unknown  Unknown
python             000055EE6A217A7A  _PyObject_Call        Unknown  Unknown
python             000055EE6A2A99F0  _PyEval_EvalFrame     Unknown  Unknown
python             000055EE6A269284  _PyFunction_Vecto     Unknown  Unknown
python3.9          000055EE6A1D172F  Unknown               Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python3.9          000055EE6A269DA9  Unknown               Unknown  Unknown
python3.9          000055EE6A1D058E  Unknown               Unknown  Unknown
python             000055EE6A269284  _PyFunction_Vecto     Unknown  Unknown
python3.9          000055EE6A1D172F  Unknown               Unknown  Unknown
python             000055EE6A269284  _PyFunction_Vecto     Unknown  Unknown
python             000055EE6A253ECA  _PyObject_FastCal     Unknown  Unknown
python3.9          000055EE6A25DB79  Unknown               Unknown  Unknown
python             000055EE6A20DD5F  _PyObject_MakeTpC     Unknown  Unknown
python             000055EE6A2A780A  _PyEval_EvalFrame     Unknown  Unknown
python             000055EE6A269284  _PyFunction_Vecto     Unknown  Unknown
python3.9          000055EE6A1D1755  Unknown               Unknown  Unknown
python             000055EE6A269284  _PyFunction_Vecto     Unknown  Unknown
python3.9          000055EE6A1D172F  Unknown               Unknown  Unknown
python             000055EE6A269284  _PyFunction_Vecto     Unknown  Unknown
python3.9          000055EE6A1D1755  Unknown               Unknown  Unknown
python             000055EE6A269284  _PyFunction_Vecto     Unknown  Unknown
python3.9          000055EE6A1D172F  Unknown               Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python             000055EE6A2693C4  _PyFunction_Vecto     Unknown  Unknown
python             000055EE6A253F4E  _PyObject_FastCal     Unknown  Unknown
python3.9          000055EE6A25DB79  Unknown               Unknown  Unknown
python             000055EE6A20DD5F  _PyObject_MakeTpC     Unknown  Unknown
python             000055EE6A2A8735  _PyEval_EvalFrame     Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python3.9          000055EE6A269CA4  Unknown               Unknown  Unknown
python3.9          000055EE6A1CFAE6  Unknown               Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python             000055EE6A2693C4  _PyFunction_Vecto     Unknown  Unknown
python3.9          000055EE6A1D058E  Unknown               Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python3.9          000055EE6A269CA4  Unknown               Unknown  Unknown
python3.9          000055EE6A1D1755  Unknown               Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python             000055EE6A269354  _PyFunction_Vecto     Unknown  Unknown
python3.9          000055EE6A1CFAE6  Unknown               Unknown  Unknown
python3.9          000055EE6A268663  Unknown               Unknown  Unknown
python             000055EE6A31545C  PyEval_EvalCodeEx     Unknown  Unknown
python             000055EE6A26945B  PyEval_EvalCode       Unknown  Unknown
python3.9          000055EE6A31550B  Unknown               Unknown  Unknown
python3.9          000055EE6A345F75  Unknown               Unknown  Unknown
python3.9          000055EE6A1E6987  Unknown               Unknown  Unknown
python             000055EE6A34BA2F  PyRun_SimpleFileE     Unknown  Unknown
python             000055EE6A34C10B  Py_RunMain            Unknown  Unknown
python             000055EE6A34C309  Py_BytesMain          Unknown  Unknown
libc-2.27.so       00007FAABEDCAC87  __libc_start_main     Unknown  Unknown
python3.9          000055EE6A2D30A0  Unknown               Unknown  Unknown

What I also noticed is that when I calculate DFT energy in the parent process everything works just fine.

2

There are lots of pieces to this example. Please provide a minimal working example to help us investigate.

3

@jmisiewicz Thank you for quick response!
Sure, here is a minimal working example to reproduce the bug:

import argparse

from multiprocessing import Manager, Pool
from ase import Atoms

import psi4
from psi4 import SCFConvergenceError
from psi4.driver.p4util.exceptions import OptimizationConvergenceError


psi4.core.set_output_file("/dev/null")


HEADER = "units ang \n nocom \n noreorient \n"
FUNCTIONAL_STRING = "wb97x-d/def2-svp"
psi_bohr2angstroms = 0.52917720859


def xyz2psi4mol(atoms, coordinates):
    molecule_string = HEADER + "\n".join(
        [" ".join([atom, ] + list(map(str, x))) for atom, x in zip(atoms, coordinates)])
    mol = psi4.geometry(molecule_string)
    return mol

def atoms2psi4mol(atoms):
    atomic_numbers = [str(atom) for atom in atoms.get_atomic_numbers().tolist()]
    coordinates = atoms.get_positions().tolist()
    return xyz2psi4mol(atomic_numbers, coordinates)


def get_dft_energy(mol):
    try:
        energy = psi4.driver.energy(FUNCTIONAL_STRING, **{"molecule": mol, "return_wfn": False})
    except SCFConvergenceError as e:
        # Set energy to some threshold if SOSCF does not converge 
        # Multiply by 627.5 to go from Hartree to kcal/mol
        print("DFT optimization did not converge!")
        return -10000.0 * 627.5
    psi4.core.clean()
    return energy * 627.5

def calculate_dft_energy_item(queue, M):
    # Get molecule from the queue
    ase_atoms, _, idx = queue.get()
    molecule = atoms2psi4mol(ase_atoms)

    # Perform DFT minimization
    not_converged = True
    if M > 0:
        psi4.set_options({'geom_maxiter': M})
        try:
            energy = psi4.optimize(FUNCTIONAL_STRING, **{"molecule": molecule, "return_wfn": False})
            not_converged = False
        except OptimizationConvergenceError as e:
            molecule.set_geometry(e.wfn.molecule().geometry())
            energy = e.wfn.energy()
        # Hartree to kcal/mol
        energy *= 627.5
        psi4.core.clean()
    else:
        # Calculate DFT energy
        energy = get_dft_energy(molecule)
    print(energy)
    
    return (idx, not_converged, energy)

def calculate_dft_energy_queue(queue, n_threads, M):
    m = Manager()

    # Create multiprocessing queue
    q = m.Queue()

    # Create a group of parallel writers and start them
    for elem in queue:
        q.put(elem)

    # Create multiprocessing pool
    p = Pool(n_threads)

    # Calculate energy for every item in the queue.
    # To mitigate large variance in dft computation times  
    # for different molecules the size of the queue
    # must be significantly larger then n_threads.
    energy_calculators = []
    for i in range(len(queue)):
        energy_calculators.append(p.apply_async(calculate_dft_energy_item, (q, M)))

    # Wait for the asynchrounous reader threads to finish
    results = [ec.get() for ec in energy_calculators]
    results = sorted(results, key=lambda x:x[0])
    p.terminate()
    p.join()

    return results


def main(num_molecules, num_threads):
    queue = []
    co = Atoms('CO', positions=[(0, 0, 0), (0, 0, 1.1)])
    # Number of atoms = 2
    queue = [[co, 2, i] for i in range(num_molecules)]
    result = calculate_dft_energy_queue(queue, n_threads=num_threads, M=0)
    print(result)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--num_molecules", default=1, type=int)
    parser.add_argument("--num_threads", default=1, type=int)
    args = parser.parse_args()
    main(args.num_molecules, args.num_threads)

To reproduce the bug you can run this code with default arguments.

While I was experimenting with this piece of code, I notices that it works just fine on my MacOS laptop, but fails with “Forrtl: severe (174): SIGSEGV, segmentation fault occurred” on remote server with “Ubuntu 18.04.03 LTS (Bionic Beaver)”. Psi4 package was installed using conda install -c psi4 psi4 on both MacOS and the remote server.

4

Try to set a custom scratch directory, the default /tmp/ can have various limitations and is usually not suitable for large calculations.

psi4_io.set_default_path('/scratch/user')

https://psicode.org/psi4manual/master/external.html#scratch-files

5

Does not help unfortunately :frowning:
Did you manage to reproduce the bug? Do you need any additional info?

6

Unless ase is essential to the bug, this example is not minimal.

Please either:

  1. Rewrite the example to remove use of the ase package (preferred), or
  2. Provide install instructions for ase. Can I get this through pip? conda? Is downloading from GitLab the only solution?
7

The use of ase package is motivated by the fact that it is impossible to put psi4.core.Molecule object into the Queue as psi4.core.Molecule object cannot be pickled.
To install ase just run pip install ase.

8

Thanks for the instructions.

For future reference, a minimal reproducible example contains the minimum amount of code needed to reproduce the bug, even if it’s dissimilar from the original example. All Psi developers are volunteers. Simplifying the thing you’re asking us to debug saves us a lot of valuable time. I was able to whittle down your example to the following input file, which indeed fails on my Linux cluster while passing on my local Mac:

from multiprocessing import Manager, Pool

import psi4

def calculate_dft_energy_item(queue):
    # Get molecule from the queue
    _, _, idx = queue.get()
    mol = psi4.geometry("C 0 0 0\nO 0 0 1.1")

    energy = psi4.driver.energy("SCF/def2-svp", molecule=mol)

    return (idx, True, energy)

if __name__ == "__main__":
    m = Manager()

    # Create multiprocessing queue
    q = m.Queue()

    # Create a group of parallel writers and start them
    q.put([0, 2, 0]) 

    # Create multiprocessing pool
    p = Pool(1)

    # Calculate energy for every item in the queue.
    # To mitigate large variance in dft computation times
    # for different molecules the size of the queue
    # must be significantly larger then n_threads.
    energy_calculators = []
    energy_calculators.append(p.apply_async(calculate_dft_energy_item, (q,)))

    # Wait for the asynchrounous reader threads to finish
    results = [ec.get() for ec in energy_calculators]
    results = sorted(results, key=lambda x:x[0])
    p.terminate()
    p.join()

I’m not familiar with the multiprocessing module. I’ll give some time for others who already know this module to try and solve this, before taking the time to learn it myself.

1 Like
9

Thanks a lot for the effort! Unfortunately, converting from ase.Atoms object was the easiest way I knew to produce a valid molecule. Next time I’ll be a lot more careful when providing a “minimal” example :slight_smile:
Anyways, hope to here from you as soon as possible.

10

I was able to simplify the example a bit more and get rid of the Queue

from multiprocessing import Manager, Pool

import psi4

def calculate_dft_energy_item():
    # Get molecule from the queue
    mol = psi4.geometry("C 0 0 0\nO 0 0 1.1")

    return psi4.driver.energy("SCF/def2-svp", molecule=mol)

if __name__ == "__main__":
    # Create multiprocessing pool
    p = Pool(1)

    # Calculate energy for every item in the queue.
    # To mitigate large variance in dft computation times
    # for different molecules the size of the queue
    # must be significantly larger then n_threads.
    energy_calculators = []
    energy_calculators.append(p.apply_async(calculate_dft_energy_item))
    results = [ec.get() for ec in energy_calculators]

…it looks like this line of code is implicated, which is very strange to me. Even if the problem was in the timer system, why that timer and not an earlier one?

I have to leave debugging here for now. Again, other developers are welcome to drive this forward themselves. I’m not expecting to be able to get back to this for a few days.

11

For the record, I haven’t forgotten this topic, but there have been many help requests this last week, so this keeps getting pushed down my list.

12

I am not sure if it is going to be helpful in your debugging process, but we managed to come up with a workaround that also uses multiprocessing and successfully calculates DFT energy in parallel processes.

import multiprocessing.connection as conn
import psi4

from multiprocessing import get_context, get_all_start_methods 
from psi4 import SCFConvergenceError
from psi4.driver.p4util.exceptions import OptimizationConvergenceError
from typing import List, Optional, Tuple


psi4.set_options({
    "CACHELEVEL": 0,
})
psi4.set_memory("8 GB")
psi4.core.set_output_file("/dev/null")
psi4.core.IOManager.shared_object().set_default_path('/dev/shm/tmp')
psi4.core.set_num_threads(4)

HEADER = "units ang \n nocom \n noreorient \n"
FUNCTIONAL_STRING = "wb97x-d/def2-svp"
psi_bohr2angstroms = 0.52917720859

# Copied from https://github.com/DLR-RM/stable-baselines3 with some changes
def _worker(
    remote: conn.Connection, parent_remote: conn.Connection, M) -> None:
    parent_remote.close()
    cmd, data = remote.recv()
    if cmd == "calc_energy":
        
        # read molecule from data
        molecule = psi4.geometry("C 0 0 0\nO 0 0 1.1")
        energy = psi4.driver.energy("wb97x-d/def2-svp", **{"molecule": mol, "return_wfn": False})
        
        # Psi4 clean
        psi4.core.clean()

        # Send energy and close remote
        remote.send(energy)
        remote.close()
    else:
        raise NotImplementedError(f"`{cmd}` is not implemented in the worker")


class ParallelEnergyCalculator():
    """
    Creates a multiprocess vectorized wrapper for energy calcualtion, distributing each calculator to its own
    process, allowing significant speed up when energy calculation is computationally complex.
    .. warning::
        Only 'forkserver' and 'spawn' start methods are thread-safe,
        which is important when TensorFlow sessions or other non thread-safe
        libraries are used in the parent (see issue #217). However, compared to
        'fork' they incur a small start-up cost and have restrictions on
        global variables. With those methods, users must wrap the code in an
        ``if __name__ == "__main__":`` block.
        For more information, see the multiprocessing documentation.
    :param n_threads: processes to run in parallel
    :param start_method: method used to start the subprocesses.
           Must be one of the methods returned by multiprocessing.get_all_start_methods().
           Defaults to 'forkserver' on available platforms, and 'spawn' otherwise.
    """

    def __init__(self, n_threads: int, M: Optional[int] = 0, start_method: Optional[str] = None):
        self.waiting = False
        self.closed = False
        self.n_threads = n_threads

        if start_method is None:
            # Fork is not a thread safe method (see issue #217)
            # but is more user friendly (does not require to wrap the code in
            # a `if __name__ == "__main__":`)
            forkserver_available = "forkserver" in get_all_start_methods()
            start_method = "forkserver" if forkserver_available else "spawn"
        ctx = get_context(start_method)

        self.remotes, self.work_remotes = zip(*[ctx.Pipe() for _ in range(self.n_threads)])
        self.processes = []
        for work_remote, remote in zip(self.work_remotes, self.remotes):
            args = (work_remote, remote, M)
            # daemon=True: if the main process crashes, we should not cause things to hang
            process = ctx.Process(target=_worker, args=args, daemon=True)  # pytype:disable=attribute-error
            process.start()
            self.processes.append(process)
            work_remote.close()

    def calc_async(self, molecules_list):
        for remote, molecule in zip(self.remotes, molecules_list):
            remote.send(("calc_energy", molecule))
        self.waiting = True

    def calc_wait(self):
        results = [remote.recv() for remote in self.remotes]
        self.waiting = False
        results = sorted(results, key=lambda x:x[0])
        return results

    def calc(self, molecules_list):
        """
        Step the environments with the given action
        :param actions: the action
        :return: observation, reward, done, information
        """
        self.calc_async(molecules_list)
        result = self.calc_wait()
        self.close()
        return result
    
    def close(self) -> None:
        if self.closed:
            return
        for process in self.processes:
            process.join()
        self.closed = True


if __name__ == '__main__':
    queue = [None] * 64
    calc = ParallelEnergyCalculator(64)
    calc.calc(queue)

Basically, every time we need to calculate a batch of energies it spawns a number of processes which calculate the energy in parallel. After the energy has been calculated each process is closed. Not closing the process after the calculations have finished also leads to a segfault (we hypothesize that the it occurs somewhere in destructor).