The analytical solution for a 3x3 matrix did seam tempting at first but it would have involved having to deal with the presence or not of complex solutions! I did find the following link discussing the accuracy of various diagonalisation algorithms for matrices with values spanning massive numerical ranges, and Jacobi’s method is meant to be more stable under those conditions. I don’t think it’s possible to perform diagonalisation directly with LAPACK using a Jacobi algorithm, although it’s probably possible to do so through an SVD function.
I suspect the simplest option is just to zero any matrix elements that are smaller than 1e-14 before diagonalisation in OrientationMgr which shouldn’t cause any other issues. I’ve tested this and all tests pass (apart from psimrcc_ccsd_t-2 test that took 8 more iterations than the standard limit but then passed! This can’t be connected.)
I’ll submit a pull request.