Use a file based job queue to prevent timeouts/skips.

[octalmage]

Currently we use a plugin based queue and process each item as a whole. This means we can't process part of a plugin and pick back up on the cron, so we end up with skipped plugins. Instead we should break plugins down by files and process the groups of files.

The scan could be slower due to the overhead of starting the scanner, but if we can make the file groups large enough it shouldn't be a huge issue. We could also provide updates at a higher frequency, and resolve #32.