Troubleshoot: Performance

​

Execution providers don’t seem to register

ort is designed to fail gracefully when an execution provider is not available. It logs failure events through tracing, thus you’ll need a library that subscribes to tracing events to see the logs. The simplest way to do this is to use tracing-subscriber.

[dependencies]
tracing-subscriber = { version = "0.3", features = [ "env-filter", "fmt" ] }

fn main() {
    tracing_subscriber::fmt::init();
}

$env:RUST_LOG = 'ort=debug';
cargo run

​

Inference is slower than expected

There are a few things you could try to improve performance:

• Run onnxsim on the model. Direct graph exports from some frameworks can leave a lot of junk nodes in the graph, which could hinder performance. onnxsim is a neat tool that can be used to simplify the ONNX graph and potentially improve performance.
• Try different execution providers for your hardware.
• Use the transformer optimization tool. This is another neat tool that converts certain transformer-based models to far more optimized graphs.
• Use I/O binding. This can reduce latency caused by copying the session inputs/outputs to/from devices.
• Quantize your model. You could try quantizing your model to 8-bit precision. This comes with a small accuracy loss, but can sometimes provide a large performance boost. If the accuracy loss is too high, you could also use float16/mixed precision.