The Design of Software (CLOSED)

A public forum for discussing the design of software, from the user interface to the code architecture. Now closed.

The "Design of Software" discussion group has been merged with the main Joel on Software discussion group.

The archives will remain online indefinitely.

Linux kernel tiered caching question.

the setup:
I have a system with 8GB ram
I have a fast 74GB hard disk (15k SAS)
I have a slow 1TB hard disk (SATA)
Linux OS (probably cent)

My software will at some point need access to all the information on the 1TB drive. I wish to use the fast disk to store a subset of the most frequently accessed files. The issue is that this subset will dynamically change over time, and is unpredictable.

The obvious solution:
Keep track of what files are accessed and move them in and out of the fast storage as necessary.
Problems with that:
-It requires round trips from kernel to user space to perform the copy.
-My program needs to check two locations to find a file
-I have to write the code (instead of the OS handling it automatically for me)

The solution I want:
When the file is read off the disk, its contents are stored in the page cache. When the page cache becomes full, these pages are pruned to make room. Instead of pruning these pages, I want them to be pushed into the swap partition (I will locate the swap on the faster disk). Now when I access the file, and it is not it physical memory, it can be read from the "FAST" virtual memory instead of having to go all the way back to the slow drive.

I was hoping that setting swapiness to 100 would do this. But in my testing that did not seem to work. I assume that this solution will become more useful as disk size increases while disk speed largely does not. A similar solution may also work well with hybrid disks (maybe not, I don't know much about them).

Thursday, January 31, 2008
I don't think that manipulating your swap parameters is going to do what you want. I don't know of any operating system that swaps file buffers out to disk like that.

You might be able to find some software that does hierachical storage management for Linux systems, which would seem to be essentially what you want.

Writing your own caching routine probably wouldn't be much of a problem. Since you presumably know more about your data access patterns than the kernel would, you can be more efficient, anyway.
Mark Bessey Send private email
Thursday, January 31, 2008
Now that I think about it some more, I actually know a guy that implemented almost exactly this as a library that replaced the standard library open(),read(), and related routines with his own version, which copied the file from slow remote storage to fast local storage on first use.

Since it was injected into client apps via dynamic linker trickery, it worked without making any changes to the client applications at all. The code itself was actually pretty simple.

This was on Mac OS X, but I suspect Linux also has a way to force-load a library into an application at startup time.
Mark Bessey Send private email
Friday, February 01, 2008
Create a huge file on the fast disk.  mmap() it into memory.  Read files from the slow disk into this area of memory.  Let the kernel worry about paging it out to the fast disk.

Saturday, February 02, 2008
The issue I have with both of these solutions, is the work is being done in user-space. So the data is copied from an FS, stored in the page cache, copied to a buffer in user space, and back again (from kernel space to user space to kernel space) where the page cache caches it for the second disk. Now the data is in ram 3 times. This make the memory cache less effective, as it is cycled three time faster. And the kernel <-> user space mem-copies are expensive. I would like to see all this happen within kernel space, thus happening quicker and with less memory. Hence my idea for using swap.
slimscsi Send private email
Tuesday, February 05, 2008
I sort of see.  I don't think Linux does what you want.  Data that is in memory from disk never pages to swap because Linux figures it is just as good to pull that data back off disk from its original location.

You could probably change that, but it'd be a patch to the kernel.

Also, swap doesn't usually perform as well as disk because Linux does not have efficient read-ahead during swap-in.  It doesn't have any way to figure what you'll swap in next so it's all very random I/O instead of streaming.
Jonathan Briggs
Sunday, February 17, 2008

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics
Powered by FogBugz