Asynchronous File I/O on Linux

[bogstad-e+AXbWqSrlAAvxtiuMwx3w at public.gmane.org (Bill Bogstad)]

On Tue, May 18, 2010 at 1:19 PM, Richard Pieri <richard.pieri-Re5JQEeQqe8AvxtiuMwx3w at public.gmane.org> wrote:
> On May 18, 2010, at 1:01 PM, Bill Bogstad wrote:
>>
>> He also doesn't want to open the file multiple times which causes a
>> problem since a file descriptor can only have one lseek() position at
>> a time. ? I can imagine scenarios (for example a library might be
>
> This is why you create a bunch of threads, each with its own file handle, each file handle with its own unique pointer. ?You have to open the file multiple times and run each read() in its own thread to truly parallelize the read operations.

Please re-read the end of my last message.   Take a look at pread()
(POSIX) and readahead() (Linux only).
It turns out you do not need separate file handles.    Threads may
still be required to make it non-blocking.

>> Err, CPU cycles are practically free compared to disk seeks. ? That's
>> why disk schedulers implement things like elevator algorithms rather
>> then FIFO:
>
> They're not free if your poll/select loop is waiting for input from the device. ?In other words, you're blocking on the I/O anyway, you're just shifting where you block from the kernel into a loop in your program.

Not necessarily   I would like random chunks of data from this file
(perhaps NEED it at some specific computation point in the future),
but I have some other computation I can do in the meantime.   Please
start the disk IO now.  Don't make me create multiple FDs for a single
file.    At my option, I would like you to:

1. signal() me in some way when the data is available.
2. have a non-blocking operation I could use to check when the data is ready.

Bonus would be if I could submit multiple such requests to the kernel
with a single system call so I can be sure the disk scheduler's
algorithm has all my requests as soon as possible.

I don't see why this is an absurd way to want to do computation.  It
may not be possible in a Linux/POSIX environment, but I don't see why
it wasn't worth thinking about how to come close to it.   If I hadn't,
I would have never learned about pread()/readahead() which look to be
very useful.   If you allow threads, it would appear that just about
everything except
the bonus part is doable.

Bill Bogstad