Parallel Fetching Across Providers to Reduce Queue Idle Time

I’ve been looking at how Keeprix handles video jobs and noticed a possible efficiency improvement in the workflow.

Right now it appears to move through the phases sequentially: Fetching Resources → Decrypting/Encoding → Processing Video. Once a job reaches Processing Video, that time could be used more effectively by already starting the resource fetch for upcoming jobs.

More specifically, there does not seem to be a reason fetching has to remain serialized across different streaming providers. For example, Keeprix should be able to fetch from Netflix, Hulu, and HBO in parallel while another video is already in Processing Video, with concurrency managed independently per provider as needed.

This would not make the actual video processing faster, but it should improve queue efficiency by reducing idle time and making better use of the time spent during processing and thus speeding up the whole process.

Curious if this kind of parallel, provider-aware fetching would fit within the current design. This request is related to other requests for multiple simultaneous downloads, but more focused on pipeline efficiency and provider-level parallelism.

Share update with 0 linked conversations as well

Upvoters
Status

In Review

Board
💡

Feature Request

Date

About 1 month ago

Author

mmarx82

Subscribe to post

Get notified by email when there are changes.