One electron Density Matrix diagonals don't sum to 1

Hi, sorry. I saw an earlier post about what one-electron density matrices represent in the CCSD calculation. It says that the SCF density is what’s returned, which sounds to me like the coefficients times the overlap (phi*phi)a where phi is the basis function and a is the coefficient used to construct the MO. See here:

I tried summing the diagonals wfn.Da() array and got 12 for an Re(0) dimer molecular triplet. I try np.sum of both alpha and beta density matrices and get close to the number of alpha and beta electrons (16 + 14 the matrix sum is 15- for both alpha and beta). I interpret that the missing density is off diagonal because the states are correlated (in Wikipedia, the diagonals are the population of the states whereas off-diagonal elements are coherences)

Am I right?

Hi,

it has been a while since I looked at these, but back in the day when I was playing around with CCSD density matrixes I found that even the trace of HartreeFock SCF density matrix does not add up the number of electrons in PSI4, and the reason was the non-orthogonal basis set.

I had to transform the density matrix with the square root of the overlap matrix:

D_fixed = S^(1/2) x D x S^(1/2)

then Tr(D_fixed) = # of electrons. You can find the procedure to compute the square root of S online, but it involves diagonalizing S. I noticed this because even the eigenvalues of the HF density matrix, which should be the occupation numbers of each molecular orbital, were not integer 1 or 0, as textbook prescribes.
I then applied the same transformation to the CCSD density matrix and it gave the correct number of electrons. In this case the eigenvalues are fractional, but in my limited understanding they should be.

Thanks a lot for getting back to me, Felix. I understand the basis of your fix, but I think you might have missed a negative power to cancel the trivial overlaps of the non-orthogonalized set. I think the solution might be more elegant.
What I tried doing is normalizing using the trace of D. Look at Wikipedia > Definition and Motivation. The diagonal states are meant to sum to occupancy, verifying my method, but also, “Another motivation for the definition of density operators comes from considering local measurements on entangled states […] a partial trace over the Hilbert space makes the operator a convenient tool to calculate the probabilities of these local measurements. It is known as the reduced density matrix” I think that’s pretty much what I’m going for, if you look at the density operator construction with the sigma notation in that same section (I’ll transcribe it: add up states p |psi> <psi| ) then the projective operators though not unique properly represent the probability of observing an electron with respect to its own coordinates uniquely assigned by each of the basis functions. Please tell me what you think.

-Marco

I do not think I missed the negative power. In my past tests I tried:

D_fixed = S^(1/2) x D x S^(1/2)
and
D_fixed = D x S

and in both cases I recovered the correct occupation numbers (from the eigenvalues of D_fixed) and the total number of electrons (from the trace of D_fixed) from any D calculated with HartreeFock. It also worked on CCSD density matrixes.

If the reason for Tr(D) not being the number of electrons is the non-orthogonality of the basis set, then normalizing D using Tr(D) as you suggested, is just going to mess up the matrix elements.

This seems reasonable to me, but unfortunately I am not smart nor competent enough to comment further.

Thanks for trying anyway. It makes sense to me to label states according to a canonical ensemble of entangled states ie we’re comparing to an ideal gas like the radial distribution function (introduction to that wikipedia page). Tr(D) might distort the elements, but it seems like it distorts them towards a physical description. If this worked for you, then I think it works on the same way my way does because multiplying the non-orthogonalized matrix by products between linearly independent vectors only renders the real projection.

Can you post how you did this or what equation you used?

I did not really do it (not smart enough to figure this out!). It was long ago and I was looking into density matrix prediction with machine-learning, and I found some nice pdfs online explaining the different methods. In some of these sources I found that the density matrix from HF is idempotent, the trace is the number of electrons and the eigenvectors are the natural orbitals, and the eigenvalues are the occupation numbers of the natural orbitals. The same properties are expected when the wavefunction is a normalised linear combination of slater determinants (instead of just one in HF), such as in CI and CC. And it puzzled me that none of these were true for any of the 98k molecules I calculated with PSI4 (HF, B3LYP and CCSD), not even idempotency.
Then I saw in one of those sources I cannot find now, that those relationships are valid for an orthonormal basis. It was almost a footnote. And then it all made sense.

Now I found this: The Roothaan-Hall self-consistent field procedure
and if you scroll down you will see this:
N = 2 tr DS (the factor 2 is because of closed shell)

(it is also found in this paper in some form: https://trygvehelgaker.no/Publications/ChemPhysLett_327_397_2000.pdf)

In one of those source I can no longer find I probably saw the symmetric version with S^(1/2) D S^(1/2). I am not knowledgeable enough to tell you why either works. Maybe it is something related to the way a generalised eigensystem is solved. But I tested these with D and S from my PSI4 runs and they worked: got the correct occupations for HF states, and the exact number of electrons down to numerical precision. Occupations for CCSD natural orbitals are all fractional, but they are meant to be

They say nobody understands quantum mechanics, we just get used to it… well I am not used to it yet.
To me the density matrix elements D[i,j] are just numbers to put in front of phi_i(r) phi_j(r), sum them all up and get a nice analytical expression to compute the charge density in real space.

1 Like

Hi, I’m a specialist on density matrices. Sorry for the delay.

There are two different things we call a “reduced density matrix.” They’re related, but they don’t have the same properties.

The first is going to look something like D_pq = <Psi | a^p a_q | Psi>. This formulation is great for methods that give you a specific wavefunction that you can take an expectation value of. This is the form you need to get D(r, r’) = \sum D_pq phi_p(r) phi_q(r’). One of the other great things about this formulation is that computing one-electron properties is, naively, straightforward. The expectation value of a one-electron property is computed simply as I_pq D_pq, where I is your integrals. There are other ways of computing properties, of course, and those lead to variations of the density matrix. This is important for specialists but is beyond the scope of what this question needs.

Implicit in all of this is the assumption that we’re working with orthonormal orbitals. This assumes that your property integrals are expressed in terms of orthonormal orbitals, and this is often an inconvenient thing to assume. What we do in those cases is write I_pq D_pq = I_mn (C_mp C_nq D_pq). This gives us an AO “density matrix” that we can just contract against AO-basis integrals and get the correct numbers. This C_mp C_nq D_pq is also called the AO density matrix. But this isn’t really a density matrix. It’s something that acts like a density matrix for one very specific purpose : contracting against AO-basis integrals to get properties.

Now for Hartree-Fock computations, this “AO density matrix” is the only computationally important property. Only people deriving equations need to worry about the actual density matrix. For every actual computation, the AO density matrix is more convenient to work with because it saves you the trouble of transforming integrals into the MO basis.

So, what does this have to do with electron number? The MO density matrix has the nice property that trace(D) is the number of electrons. What about the AO density matrix?

num_electrons = \sum_{i occupied} <phi_i, phi_i>
= \sum_{i occupied, mu, nu} <C_mu,i chi_mu, C_nu,i chi_nu>
= \sum_{mu, nu} D_mu,nu <chi_mu, chi_nu>
= \sum_{mu, nu} D_mu,nu S_mu, nu
= trace(D * S)

From there, we can use the cyclic property of the trace to rewrite as trace(S^{1/2} * D * S^{1/2}).

Line 1 used the assumption of orthonormal orbitals, and the jump between lines 2 and 3 used the assumption of a single-determinant wavefunction. I’d need to think more to know whether the trace(D*S) formula is still valid when you don’t have the single-determinant assumption, but in those cases, you probably have the MO density matrix anyways and can just compute the electron number from that.

EDIT: Yes, the tr(D * S) should be valid more generally. C * S * C.T is just going to be the overlap matrix in MOs, which had better be the identity matrix by the orthonormality assumption. In MOs, we already know tr(D) = num. el. by other arguments.

1 Like

Thank you very much for this. It’s hard to come by this kind of information, explained in a clear way.

Yeah, that was really useful, thank you

You expedited my work greatly Felix, thank you for the citation. I’ve returned to tell you that I called wfn.Da() and OrbitalSpace.C() which is meant to be an MO coefficient matrix constructed from AOs. Curiously, they have the same pointer but appear to start with different values.