Run work on several NPUs in parallel

A server can hold several Q.ANT cards. This example shows one way to use them together. It runs the same batch of matrix-vector products twice: once in a single process on one device, and once with a pool of worker processes, each linked to its own device.

This example is a plain Python script, not a notebook.

Objectives

• List the available NPUs and address each one by its device ID.
• Run a toolkit operation in several processes at once.
• Share large arrays between processes without copying them.

Before you begin

1. Install the toolkit. The CPU emulator works for this example, but it always reports one NPU, so it can’t show a speed-up.

2. Check that Python can import the toolkit:

   python -c "import qant_native_computing_toolkit as q; print(q.generic.get_available_npus())"

   On the emulator this prints {0: 'cpu_backend'}. On a server with cards it lists each NPU.

3. Get the examples repository:

   git clone --recurse-submodules https://github.com/Q-ANT-GmbH/qant_native_computing_toolkit_examples.git
   cd qant_native_computing_toolkit_examples/python/parallel_execution

4. Install the example’s dependencies:

   pip install -r requirements.txt

   This example needs only numpy, ml_dtypes and pytest.

Run the example

Run the script:

python matrix_vector_multiplication.py

How it works

The workload

The script creates ten bfloat16 matrices of 2,000 by 2,000 values and one vector of 2,000 values. Each job multiplies one matrix by the vector with native.linear_fprop.

Find the devices

The script reads the device IDs from get_available_npus and starts one worker for each:

npu_ids = list(qant.generic.get_available_npus().keys())
n_npus = len(npu_ids)
print(f"Found {n_npus} NPUs")

Single process on one device

The baseline runs all ten jobs one after another on device 0:

qant.generic.init_npu(0)
results_singleproc = [multiply_matrix_vector_single(task) for task in task_args]
qant.generic.release_npu(0)

One process per device

For the parallel run, the script copies the matrices and the vector into multiprocessing.shared_memory blocks once. A comment in the script explains why: otherwise the cost of pickling the arrays for each worker is the bottleneck.

It then starts a ProcessPoolExecutor with one worker per NPU. To link each worker to a device, it collects the worker process IDs and maps each one to a device ID. Each worker rebuilds NumPy views on the shared memory and calls the toolkit with its own device:

pid = os.getpid()
wid = pid_to_wid[pid]
print(f"Worker[{wid}] with pid {pid} solves a job")

return qant.native.linear_fprop(vector, matrices[matrix_id], device_id=wid)

Expected output

On the CPU emulator, the script printed:

Found 1 NPUs
Worker[0 with pid 1015] solves a job
...
--------------------
worker pids: [1018]
Worker[0] with pid 1018 solves a job
...
execution time single NPU 0.1
execution time multi NPU 0.2

Each “solves a job” line appears ten times in each run. Your process IDs will differ.

Measure	Value on the CPU emulator
NPUs found	1
Single-process time	0.1 seconds
Multi-process time	0.2 seconds

With one emulated NPU, the pool has one worker, so the parallel run does the same work plus the cost of starting a process. That is why it is slower here.

Run the tests

The example includes a test that runs main() and checks that both approaches give the same results within an absolute tolerance of 0.001:

python -m pytest -v test_matrix_vector_parallel.py

On the emulator the test passed in 1.31 seconds.

What’s next

• Read about device IDs and the device lifecycle in Key concepts.
• Learn which device functions the emulator ignores in Emulator and hardware.