Parallel video encoding

[me-5yx05kfkO/aqeI1yJSURBw at public.gmane.org (Matthew Gillen)]

Kristian Erik Hermansen wrote:
> In any event, it doesn't really matter since I wasn't planning on
> writing a custom app.  I just wanted to know if it would be more
> efficient to spawn multiple mencoder process at the same time with one
> thread allocated to the process (up to MAX_CORES or MAX_CORES - 1),
> spawn mencoder on MAX_CORES - 1 for one file at a time (that is the
> suggested way, right?), or just invoke mencoder at MAX_FILES_TO_ENCODE
> times with one thread for each.
> 
> I bet option #2 works the best, since spawning multiple mencoder
> processes could be wasteful in terms of caching and memory usage.  I
> bet one mencoder process working on one file with MAX_CORES - 1 would
> be ideal.  If you go higher than MAX_CORES, then you are going to have
> to context switch in and out the threads as they are shared, right?
> Tell me if my logic is flawed here...

Well, there's a school of thought on using 'make' that says you should always
use at least -j(MAX_CORES+1), the idea being that you keep the CPU busier by
always having an "extra" job ready when one thread needs to wait on disk I/O.
 Disk I/O will completely eclipse any context-switching penalty, and even
though video encoding is a cpu-bound task, there's quite a bit of I/O involved
too.  I'm not sure what the optimal answer is for your case.

If you want to tinker with option #1, here's an idea that minimizes context
switching and doesn't involve makefiles:  spawn all the jobs concurrently (ie
more jobs than MAX_CORES), but set their priorities (ie via nice) so that
there is a strict ordering, but even the lowest priority job is more important
than "normal" priority.  That way you don't get context switching overhead,
but you still make the OS scheduler do all the job-management for you.

Have fun, and let us know what you come up with :-)

Matt

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.