Increasing HPC Utilization with Meta-Queues
Solving problems by the addition of abstractions is a tried and true approach in technology. The management of high-performance computing workflows is no exception.
The Pegasus workflow engine and HTCondor’s DAGman are used to manage workflow dependencies. GridWay and DRIVE route jobs to different resources based on suitability or available capacity. Both of these approaches are important, but they share a key potential drawback: jobs are still treated as distinct units of computation to be scheduled individually by the scheduler.
As we have written previously, the aims of HPC resource administrators and HPC resource users are sometimes at odds. …
Increasing HPC Utilization with Meta-Queues was written by Nicole Hemsoth at The Next Platform.