sherpa-onnx/OPENVINO.md

Intel NPU support with OpenVINO

Sherpa-onnx can run ONNX models on Intel NPUs through ONNX Runtime's OpenVINO Execution Provider. This requires ONNX Runtime 1.17 or newer built with the OpenVINO provider; the standard ONNX Runtime package bundled by sherpa-onnx does not include it.

A standalone OpenVINO installation is not sufficient by itself: the ONNX Runtime library must also be built with the OpenVINO Execution Provider, and that build must be compatible with the installed OpenVINO runtime.

Build

Install an OpenVINO-enabled ONNX Runtime and initialize the OpenVINO runtime environment. Then point sherpa-onnx at its C/C++ headers and libraries:

source /path/to/openvino/setupvars.sh
export SHERPA_ONNXRUNTIME_INCLUDE_DIR=/path/to/onnxruntime/include
export SHERPA_ONNXRUNTIME_LIB_DIR=/path/to/onnxruntime/lib

cmake -S . -B build-openvino \
  -DBUILD_SHARED_LIBS=ON \
  -DSHERPA_ONNX_USE_PRE_INSTALLED_ONNXRUNTIME_IF_AVAILABLE=ON
cmake --build build-openvino --parallel

OpenVINOExecutionProvider should appear in the available-provider list when the OpenVINO-enabled ONNX Runtime and its dependencies are discoverable.

Use the Intel NPU for wake-word detection

Set the existing sherpa-onnx provider option to openvino. The default OpenVINO device is NPU:

build-openvino/bin/sherpa-onnx-keyword-spotter \
  --provider=openvino \
  --tokens=/path/to/tokens.txt \
  --encoder=/path/to/encoder.onnx \
  --decoder=/path/to/decoder.onnx \
  --joiner=/path/to/joiner.onnx \
  --keywords-file=/path/to/keywords.txt \
  /path/to/audio.wav

For transducer keyword-spotting models, sherpa-onnx runs the compute-heavy encoder on OpenVINO and keeps the small decoder and joiner on CPU. This avoids unnecessary NPU dispatch overhead and preserves decoder accuracy. The Silero VAD model tested for this integration ran entirely on OpenVINO. Other VAD models may be partitioned by ONNX Runtime, with unsupported nodes assigned to the CPU provider.

For a microphone, use sherpa-onnx-keyword-spotter-microphone (PortAudio) or sherpa-onnx-keyword-spotter-alsa with the same --provider value.

Use the Intel NPU for VAD

VAD uses the --vad-provider option:

build-openvino/bin/sherpa-onnx-vad \
  --vad-provider=openvino \
  --silero-vad-model=/path/to/silero_vad.onnx \
  /path/to/input.wav \
  /path/to/speech-only.wav

The microphone VAD binaries accept the same --vad-provider value.

The same provider string works through the C++, C, Python, Java, Kotlin, Dart, Go, Rust, Swift, C#, and Node.js APIs. For example, Python model configs accept provider="openvino".

If OpenVINO is unavailable or cannot initialize, sherpa-onnx logs the reason and falls back to the ONNX Runtime CPU provider.

Provider options

OpenVINO options use sherpa-onnx's existing provider:config-file syntax. The file contains one key=value entry per line:

# openvino-npu.config
device_type=NPU
enable_qdq_optimizer=True

Pass it by appending the path to the provider name. Use --provider for keyword spotting and --vad-provider for VAD:

--provider=openvino:openvino-npu.config
--vad-provider=openvino:openvino-npu.config

Any OpenVINO V2 provider option can be used. device_type may also be CPU, GPU, or AUTO. Multi-device modes require at least two devices; for example, use HETERO:GPU,CPU or MULTI:GPU,CPU. Bare HETERO and MULTI values are incomplete device selections. AUTO:GPU,NPU,CPU can be used to give automatic selection an explicit device priority. If GraphOptimizationLevel is omitted, sherpa-onnx disables ONNX Runtime graph optimizations as recommended by the OpenVINO Execution Provider. Set it explicitly in the config file to override that behavior.

OpenVINO's NPU support depends on the model's operators and shapes. Dynamic speech-model inputs may need disable_dynamic_shapes and reshape_input bounds tailored to the selected model; unsupported work can execute on the CPU fallback provider.

See the ONNX Runtime OpenVINO Execution Provider documentation for installation packages, supported devices, options, and compatibility.