In the previous two blogs in this series I showed how solving an apparently simple problem about loading a lot of data into RAM using mmap() also turned out to require a solution that improved CPU use across cores. In this blog, I’ll show how we dealt with the bottleneck problems that ensued, and finally, […]
In Part 1 of this series we solved the problem of using mmap() to load large set of data into RAM all at once. The catch was that it still took too long. In this blog, I describe how we solve this.
We got a call from a bioinformatics group at a European university. The problem appeared simple, how to usemmap() to load a large data set into RAM all at once. OK, no problem I thought...
Today’s complex applications can soak up megabytes, a memory management strategy is crucial for RAM intensive applications like image/video processing, massive databases, and machine learning.