Values

For ONNX Runtime, a value represents any type that can be given to/returned from a session or operator. Values come in three main types:

• Tensors (multi-dimensional arrays). This is the most common type of Value.
• Maps map a key type to a value type, similar to Rust’s HashMap<K, V>.
• Sequences are homogenously-typed dynamically-sized lists, similar to Rust’s Vec<T>. The only values allowed in sequences are tensors, or maps of tensors.

Creating values

Creating tensors

Tensors can be created with Tensor::from_array from either:

• an ndarray::Array, or
• a tuple of (shape, data), where:
  • shape is one of Vec<I>, [I; N] or &[I], where I is i64 or usize, and
  • data is one of Vec<T> or Box<[T]>.

let tensor = Tensor::from_array(ndarray::Array4::<f32>::zeros((1, 16, 16, 3)))?;
 
let tensor = Tensor::from_array(([1usize, 2, 3], vec![1.0_f32, 2.0, 3.0, 4.0, 5.0, 6.0]))?;

The created tensor will take ownership of the passed data. See Creating views of external data to create temporary tensors referencing borrowed data.

Creating maps & sequences

Maps can be created  from any iterator yielding tuples of (K, V), where K and V are tensor element types.

let mut map = HashMap::<String, f32>::new();
map.insert("one".to_string(), 1.0);
map.insert("two".to_string(), 2.0);
map.insert("three".to_string(), 3.0);
 
let map = Map::<String, f32>::new(map)?;

Maps can also be created from 2 tensors , one containing keys and the other containing values:

let keys = Tensor::<i64>::from_array(([4], vec![0, 1, 2, 3]))?;
let values = Tensor::<f32>::from_array(([4], vec![1., 2., 3., 4.]))?;
 
let map = Map::new_kv(keys, values)?;

Sequences can be created  from any iterator yielding a Value subtype:

let tensor1 = Tensor::<f32>::new(&allocator, [1, 128, 128, 3])?;
let tensor2 = Tensor::<f32>::new(&allocator, [1, 224, 224, 3])?;
 
let sequence: Sequence<Tensor<f32>> = Sequence::new(vec![tensor1, tensor2])?;

Using values

Values can be used as an input in a session’s run function - either by value, by reference, or by view.

let latents = Tensor::<f32>::new(&allocator, [1, 128, 128, 3])?;
let text_embedding = Tensor::<f32>::new(&allocator, [1, 48, 256])?;
let timestep = Tensor::<f32>::new(&allocator, [1])?;
 
let outputs = session.run(ort::inputs![
    "timestep" => timestep,
    "latents" => &latents,
    "text_embedding" => text_embedding.view()
])?;

Extracting data

To access the underlying data of a value directly, the data must first be extracted.

Tensors can either extract to an ndarray::ArrayView via extract_array when the ndarray feature is enabled, or extract to a tuple via extract_tensor of (&Shape, &[T]) (where the second element is the slice of data contained within the tensor).

let array = ndarray::Array4::<f32>::ones((1, 16, 16, 3));
let tensor = TensorRef::from_array_view(&array)?;
 
let extracted: ArrayViewD<'_, f32> = tensor.extract_array();
let (tensor_shape, extracted_data): (&Shape, &[f32]) = tensor.extract_tensor();

Tensors and TensorRefMuts with non-string elements can also be mutably extracted with extract_array_mut and extract_tensor_mut. Mutating the returned types will directly update the data contained within the tensor.

let mut original_array = vec![1_i64, 2, 3, 4, 5];
{
	let mut tensor = TensorRefMut::from_array_view_mut(([original_array.len()], &mut *original_array))?;
	let (extracted_shape, extracted_data) = tensor.extract_tensor_mut();
	extracted_data[2] = 42;
}
assert_eq!(original_array, [1, 2, 42, 4, 5]);

Map and Sequence have Map::extract_map and Sequence::extract_sequence, which emit a HashMap<K, V> and a Vec of value views respectively. Unlike extract_tensor, these types cannot mutably extract their data, and always allocate on each extract call, making them more computationally expensive.

Session outputs return DynValues, which are values whose type is not known at compile time. In order to extract data from a DynValue, you must either downcast it to a strong type or use a corresponding try_extract_* method, which fails if the value’s type is not compatible:

let outputs = session.run(ort::inputs![TensorRef::from_array_view(&input)?])?;
 
let Ok(tensor_output): ort::Result<ndarray::ArrayViewD<f32>> = outputs[0].try_extract_array() else {
    panic!("First output was not a Tensor<f32>!");
}

Views

A view (also called a ref) is functionally a borrowed variant of a value. There are also mutable views, which are equivalent to mutably borrowed values. Views are represented as separate structs so that they can be down/upcasted.

View types are suffixed with Ref or RefMut for shared/mutable variants respectively:

• Tensors have DynTensorRef(Mut) and TensorRef(Mut).
• Maps have DynMapRef(Mut) and MapRef(Mut).
• Sequences have DynSequenceRef(Mut) and SequenceRef(Mut).

These views can be acquired with .view() or .view_mut() on a value type:

let my_tensor: ort::value::Tensor<f32> = Tensor::new(...)?;
 
let tensor_view: ort::value::TensorRef<'_, f32> = my_tensor.view();

Views act identically to a borrow of their type - TensorRef supports extract_tensor, TensorRefMut supports extract_tensor and extract_tensor_mut. The same is true for sequences & maps.

Creating views of external data

You can create TensorRefs and TensorRefMuts from views of external data, like an ndarray array, or a raw slice of data. These types act almost identically to a Tensor - you can extract them and pass them as session inputs - but as they do not take ownership of the data, they are bound to the input’s lifetime.

let original_data = Array4::<f32>::from_shape_vec(...);
let tensor_view = TensorRef::from_array_view(original_data.view())?;
 
let mut original_data = vec![...];
let tensor_view_mut = TensorRefMut::from_array_view_mut(([1, 3, 64, 64], &mut *original_data))?;

Dynamic values

Sessions in ort return a map of DynValues. These are values whose exact type is not known at compile time. You can determine a value’s type  via its .dtype() method.

You can also use fallible methods to extract data from this value - for example, DynValue::try_extract_tensor, which fails if the value is not a tensor. Often times though, you’ll want to reuse the same value which you are certain is a tensor - in which case, you can downcast the value.

Downcasting

Downcasting means to convert a dyn type like DynValue to stronger type like DynTensor. Downcasting can be performed using the .downcast() function on DynValue:

let value: ort::value::DynValue = outputs.remove("output0").unwrap();
 
let dyn_tensor: ort::value::DynTensor = value.downcast()?;

If value is not actually a tensor, the downcast() call will fail.

Stronger types

DynTensor means that the type is a tensor, but the element type is unknown. There are also DynSequences and DynMaps, which have the same meaning - the kind of value is known, but the element/key/value types are not.

The strongly typed variants of these types - Tensor<T>, Sequence<T>, and Map<K, V>, can be directly downcasted to, too:

let dyn_value: ort::value::DynValue = outputs.remove("output0").unwrap();
 
let f32_tensor: ort::value::Tensor<f32> = dyn_value.downcast()?;

If value is not a tensor, or if the element type of the value does not match what was requested (f32), the downcast() call will fail.

Stronger typed values have infallible variants of the .try_extract_* methods:

// We could try to extract a tensor directly from a `DynValue`...
let f32_array: ArrayViewD<f32> = dyn_value.try_extract_array()?;
 
// Or, we can first onvert it to a tensor, and then extract afterwards:
let tensor: ort::value::Tensor<f32> = dyn_value.downcast()?;
let f32_array = tensor.extract_array(); // no `?` required, this will never fail!

Upcasting

Upcasting means to convert a strongly-typed value type like Tensor<f32> to a weaker type like DynTensor or DynValue. This can be useful if you have code that stores values of different types, e.g. in a HashMap<String, DynValue>.

Strongly-typed value types like Tensor<f32> can be converted into a DynTensor using .upcast():

let dyn_tensor = f32_tensor.upcast();
// type is DynTensor

Tensor<f32> or DynTensor can be cast to a DynValue by using .into_dyn():

let dyn_value = f32_tensor.into_dyn();
// type is DynValue

Upcasting a value doesn’t change its underlying type; it just removes the specialization. You cannot, for example, upcast a Tensor<f32> to a DynValue and then downcast it to a Sequence; it’s still a Tensor<f32>, just contained in a different type.

Dyn views

Views also support down/upcasting via .downcast() & .into_dyn() (but not .upcast() at the moment).

You can also directly downcast a value to a stronger-typed view using .downcast_ref() and .downcast_mut():

let tensor_view: ort::value::TensorRef<'_, f32> = dyn_value.downcast_ref()?;
// is equivalent to
let tensor_view: ort::value::TensorRef<'_, f32> = dyn_value.view().downcast()?;

Conversion recap

• DynValue represents a value that can be any type - tensor, sequence, or map. The type can be retrieved with .dtype().
• DynTensor, DynMap, and DynSequence are values with known container types, but unknown element types.
• Tensor<T>, Map<K, V>, and Sequence<T> are values with known container and element types.
• Tensor<T> and co. can be converted from/to their dyn types using .downcast()/.upcast(), respectively.
• Tensor<T>/DynTensor and co. can be converted to DynValues using .into_dyn().