Insider Brief
- A European research team tested workflow and “malleability” scheduling strategies to improve efficiency in hybrid high-performance computing–quantum computing clusters, finding that both cut idle time compared to static allocation.
- In simulated workloads, workflows used the least HPC node time but incurred more overhead, while malleability allowed faster completion by resizing HPC jobs during quantum phases without re-queuing.
- Results suggest dynamic scheduling will be critical as quantum processors are integrated into supercomputing environments, especially when quantum tasks are long relative to classical ones.
Scientists in Europe have developed and tested new methods to improve how supercomputers and quantum processors share work, aiming to cut waste and speed up results in emerging hybrid computing systems. The study, posted on arXiv, found that “malleability” — or, the ability of software to give up unused computing resources and reclaim them when needed — can reduce idle time and improve job completion times in high-performance computing (HPC) environments that incorporate scarce quantum processors.
The work addresses a growing challenge as quantum computers, still in their early stages, begin to be linked with supercomputers to solve problems that neither can handle alone. The researchers say that without better scheduling, expensive hardware will sit idle while waiting for other parts of a job to finish.
Hybrid Systems, Uneven Resources
Quantum processing units (QPUs) are expected to act as accelerators for certain problem types, such as optimization, materials modeling and quantum simulation, while the bulk of computation still runs on classical HPC systems. But integrating these two worlds is complex.
In current configurations, a hybrid job may reserve both HPC nodes and a QPU for its entire runtime, even though it only needs one type of resource at a time. That means large numbers of CPU cores can sit idle while the quantum hardware is in use, and vice versa. With few QPUs available — sometimes only one or two in a cluster — the risk of bottlenecks grows.
The research team, from E4 Computer Engineering, LINKS Foundation, the Barcelona Supercomputing Center, CINECA and several universities, tested two strategies to tackle this: a workflow-based approach and a malleability-based approach.
Two Ways to Manage Workloads
Workflow management systems (WMS) break jobs into discrete tasks, with dependencies between them, and schedule resources as needed. The team used the StreamFlow WMS to model their hybrid application as a three-step loop: run multiple clustering algorithms in parallel on the HPC nodes, aggregate results via a quantum routine and evaluate quality before repeating or stopping. With this method, quantum resources are only reserved when required, releasing HPC nodes in between.
Malleability takes a different tack by allowing a running application to change the number of computing nodes it uses at runtime, according to the paper. In the team’s tests, the Dynamic Management of Resources (DMR) framework was used to shrink a job’s HPC footprint when computation shifted to the quantum part, keeping only a minimal process alive and then expand back when returning to classical computation. This avoids the overhead of re-queuing tasks but still frees unused cores for other jobs.
A Test Case in Clustering
To compare the approaches, the researchers adapted an existing clustering aggregation algorithm into a hybrid HPC–QC application. The classical phase ran three algorithms — k-means, DBSCAN, and hierarchical clustering — on separate HPC nodes. Their outputs were combined into a graph and turned into a quadratic unconstrained binary optimization (QUBO) problem, which was sent to the quantum phase for solving.
Because the team did not have a QPU in the testbed, they used a “quantum emulator” node to run simulated annealing, with optional artificial delays to mimic the runtime of different quantum technologies.
Experimental Results
Tests ran on a small SLURM-managed cluster — SLURM stands for Simple Linux Utility for Resource Management, for the curious — with three CPU-only compute nodes and one “quantum” node. The researchers compared three setups:
- Baseline: Static allocation of both HPC and quantum resources for the job’s full duration.
- Workflow: Task-based scheduling that only requests quantum resources when needed.
- Malleability: Dynamic resizing of HPC allocation during quantum phases.
For quantum phases simulated to last two minutes — similar to some neutral-atom devices — the workflow method used the least total HPC node time but took longer overall, as it repeatedly requested resources from the scheduler. The baseline was fastest for a single run but least efficient, holding resources idle. Malleability landed between the two, saving substantial resources without the workflow’s repeated scheduling delays.
When two jobs were run at once, the differences sharpened. The baseline approach caused both jobs to finish later and consume more resources, while workflow and malleability allowed better overlap. Malleability had the advantage of resuming computation immediately after the quantum phase, even if not all original HPC nodes were free, cutting completion times further.
For short quantum phases (taking under a second, akin to superconducting QPUs), resource savings were smaller, and total run times were dominated by classical computation. Even so, malleability showed an edge in concurrent job handling.
Implications for HPC–QC Integration
The study suggests that as hybrid workloads become more common, static scheduling will be a poor fit, especially when quantum tasks are long relative to classical ones. In resource-constrained clusters, freeing and reallocating compute nodes dynamically could make better use of expensive HPC hardware and reduce queue wait times.
The researchers also report that workflows and malleability each have trade-offs. Workflow systems require modular application design and familiarity with a workflow language. Malleability is easier to integrate into existing code but adds complexity in managing program state and execution with varying resources.
Limits and Future Work
There’s more work to do, according to the team. The experiments were limited to a small-scale test cluster and simulated quantum workloads. The researchers say larger tests with actual quantum hardware, more diverse workloads and realistic contention scenarios are needed. They also point out that current methods do not yet account for the often substantial wait times in shared HPC environments, a factor that could further favor malleability’s ability to reconfigure on the fly.
The work, however, highlights a broader — potentially more important point in quantum–HPC integration: hardware progress alone will not deliver practical performance gains unless the surrounding software and scheduling systems evolve. Efficient hybrid job handling could be as important as quantum speedups themselves in determining real-world impact.
As the paper puts it, there may be what might be termed a “maturity gap” between HPC and QC technologies, which means orchestration will be a persistent challenge. But by borrowing ideas from decades of supercomputing research — like malleable jobs — and adapting them to quantum’s quirks, researchers can start closing that gap.
For now, these scheduling improvements may only shave seconds or minutes off small-scale hybrid jobs. In future, when national supercomputing centers couple petascale or exascale machines with quantum processors, the same techniques could decide whether those machines run near capacity or spend much of their time waiting.
For a deeper, more technical dive, please review the paper on arXiv. It’s important to note that arXiv is a pre-print server, which allows researchers to receive quick feedback on their work. However, it is not — nor is this article, itself — official peer-review publications. Peer-review is an important step in the scientific process to verify the work.
The following research institutions were involved in the work: E4 Computer Engineering, LINKS Foundation, Barcelona Supercomputing Center, CINECA and Università di Torino.
