Troubleshooting

Each entry lists the symptom, the cause and the fix. Entries are grouped by when they happen: building, calling the API, or running the examples.

Build and install

Build fails on a missing dlpack.h

Symptom. maturin build or cargo build stops with an error that dlpack.h is missing.

Cause. The repository uses Git submodules, and the DLPack headers live in one of them. A plain git clone doesn’t fetch submodules.

Fix. Fetch the submodules, then build again:

git submodule update --init --recursive

For a fresh clone, use git clone --recurse-submodules. See Install the toolkit.

test_logging fails on the CPU emulator

Symptom. In the toolkit’s own tests, test_logging fails with:

RuntimeError: Rust core error: Toolkit Error: logging not available for cpu-backend

Cause. The CPU emulator doesn’t implement logging. The project marks the test with the logging marker because it needs a driver.

Fix. This failure is expected on the emulator. Exclude the marker:

python -m pytest -q -m "not logging"

On the emulator this gives 109 passed, 1 deselected.

Calling the API

TypeError about the dtype

Symptom. A toolkit call raises:

TypeError: features must be of dtype <class 'ml_dtypes.bfloat16'> but is float32

Cause. Every operation accepts only ml_dtypes.bfloat16 arrays. NumPy creates float32 or float64 arrays by default.

Fix. Convert each input before the call:

from ml_dtypes import bfloat16

x = x.astype(bfloat16)

Convert weights once when you load them. See Key concepts.

Softmax fails on batched input

Symptom. softmax_fprop raises:

RuntimeError: Rust core error: Toolkit Error: does not support batched input.

Cause. softmax_fprop accepts one row at a time. A 2D array with more than one row is rejected. conv_transpose_fprop raises the same error for a 4D input with more than one image.

Fix. Call the operation once per row or image:

rows = [q.ai.softmax_fprop(row) for row in logits]

Operations are slower than expected because of copies

Symptom. Calls take longer than you expect on hardware, even with small inputs.

Cause. The toolkit documentation says the NPU requires input buffers aligned to a 4096-byte page boundary. If a buffer is not aligned, the runtime makes an extra copy before each call.

Fix. Align arrays that you reuse once, with utils.align_ndarray_page_boundary():

weights = q.utils.align_ndarray_page_boundary(weights)

We could not measure this effect, because we have no hardware. See Key concepts.

Running the examples

torchvision import fails with a missing operator

Symptom. Importing torchvision raises:

RuntimeError: operator torchvision::nms does not exist

Cause. The torchvision wheel doesn’t match the installed torch build. This happens when torch comes from the PyTorch CPU index but torchvision comes from PyPI, or when the two versions don’t belong together.

Fix. Reinstall the torchvision version that matches your torch version from the CPU index, without touching other packages:

pip install --force-reinstall --no-deps torchvision==<matching> --index-url https://download.pytorch.org/whl/cpu

Replace <matching> with the torchvision release paired with your torch release. The examples pin torch==2.11.0 with torchvision==0.26.0.

MNIST download returns 404

Symptom. The digit recognition example fails while downloading MNIST, with an HTTP 404 error for a file on ossci-datasets.s3.amazonaws.com.

Cause. The download from this mirror is intermittent. We saw the 404 once, and the same request worked on retry.

Fix. Run the cell or the test again. After a successful download the files stay in /tmp/mnist_files/, and later runs don’t download them again.

The Fourier network runs in PyTorch instead of the toolkit

Symptom. The Fourier network example produces results, but no toolkit calls happen. There is no error.

Cause. The QLinear and QFourier layers only call the toolkit in eval mode. In training mode they use ordinary PyTorch operations.

Fix. Call eval() on each model before inference:

mlp_model.eval()
fourier_model.eval()

See Fourier network.

ResNet pooling runs on device 1

Symptom. None on the emulator. On a server with a single card, the image recognition example may fail at the pooling step or address a device that doesn’t exist.

Cause. This is a bug in the example’s qant_resnet.py, which calls:

x = q_ai.adaptive_avgpool2d_fprop(x, 1, 1)

The signature is adaptive_avgpool2d_fprop(features, output_size, device_id=0), so the third positional argument sets device_id to 1. The intent was probably an output size of 1 by 1. The emulator ignores device_id, so the example still works there. We could not test the behaviour on hardware.

Fix. Pass the output size only:

x = q_ai.adaptive_avgpool2d_fprop(x, 1)

Image segmentation fails to download the model

Symptom. The image segmentation notebook fails at the torch.hub.load cell with HTTP Error 400: Bad Request or a 403 error.

Cause. The pretrained UNet and the sample image come from GitHub. A firewall or proxy that blocks github.com stops the download.

Fix. Run the example on a machine that can reach github.com. See Image segmentation.

What’s next

• Check what the emulator does differently from a card in Emulator and hardware.
• Report a bug in the toolkit on the toolkit issue tracker.