Inconsistent IWL (or PSIO?) reading performance

Hi everyone,

I am in the process of improving the PK integrals code. For that, I am measuring the time spent reading file 33, containing integrals in the IWL format. Briefly, the way it is done is:

do {
// Some code here
} while(!last_buffer)

This whole piece of code takes about 7 seconds to execute in the regular code, where I only measure time spent reading and not any of the time spent in [//Some code].
However, if [//Some code] contains write to multiple other files through IWL buffers, like is necessary for a Yoshimine sort, the total reading time jumps to 40 seconds, and sometimes even 70 seconds.
Is there any obvious reason I am missing that reading file 33 should be much slower when we write to multiple other files ?
Do we expect IO caching on the hard drive to have such a drastic effect, or is there some internal caching in the IWL or PSIO libraries ?

There is no internal caching within PSI itself, but the OS may be caching for you. What size files are you handling where you see these kinds of timing shifts?


It’s mostly with bigger files, i.e. above 5 to 10 Gb. Small files take such a short time to read anyway that it is difficult to get reliable timings.
But yes, it may be OS caching. If that is the case, OS caching is much more efficient than what I would have thought.