Modern machine learning pipelines often emphasize training reproducibility, but inference reproducibility also matters—especially when models are exported, deployed across heterogeneous machines, or evaluated for regression testing. This post documents a practical, factual approach to making ONNX Runtime inference deterministic by controlling thread parallelism and (where supported) random-related nondeterminism.

### Why inference determinism is tricky

Even with a fixed input, inference can vary across runs due to:
- **Parallel execution**: different thread scheduling can lead to small numerical differences.
- **Non-deterministic operators**: some ONNX operators may be nondeterministic depending on the execution provider and runtime version.
- **Floating-point reduction order**: parallelism can change summation order, yielding tiny output changes.

The core strategy is to reduce nondeterministic degrees of freedom: use **single-thread execution**, and set runtime-level determinism flags where available.

### What this post demonstrates

The example uses:
- An exported **ONNX model** (any feed-forward model with only deterministic ops is appropriate).
- **ONNX Runtime** configured for deterministic behavior:
  - `intra_op_num_threads=1`
  - `inter_op_num_threads=1`
  - provider set to CPU
- An explicit output comparison across repeated runs.

### Install dependencies

```bash
pip install onnxruntime numpy
```

### Deterministic ONNX Runtime inference

```python
import os
import numpy as np
import onnxruntime as ort

# Limit parallelism at the OS level as well (helps in some environments).
os.environ["OMP_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["NUMEXPR_NUM_THREADS"] = "1"

# Path to your exported ONNX model.
model_path = "model.onnx"

# Create deterministic session options.
so = ort.SessionOptions()
so.intra_op_num_threads = 1
so.inter_op_num_threads = 1
so.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL

# Note: onnxruntime has historically used different flags across versions for determinism.
# SessionOptions does not universally expose a single "set deterministic" switch,
# so thread reduction + sequential execution are the reliable baseline.

sess = ort.InferenceSession(
    model_path,
    sess_options=so,
    providers=["CPUExecutionProvider"],
)

# Create a deterministic input.
# Replace input_name and shape to match your model.
input_name = sess.get_inputs()[0].name
input_shape = sess.get_inputs()[0].shape
# Handle symbolic dimensions like None or 'unk__*' by substituting a fixed size.
resolved_shape = []
for d in input_shape:
    resolved_shape.append(1 if isinstance(d, str) or d is None else int(d))

# Use a fixed seed for generating input values.
rng = np.random.default_rng(12345)
x = rng.standard_normal(resolved_shape).astype(np.float32)

inputs = {input_name: x}

# Run inference multiple times and compare outputs.
y_runs = []
for _ in range(5):
    outputs = sess.run(None, inputs)
    y_runs.append(outputs[0])

ref = y_runs[0]
for i, y in enumerate(y_runs[1:], start=1):
    # For deterministic operators on CPU with sequential execution,
    # outputs should be exactly identical for the same input.
    if not np.array_equal(ref, y):
        # Fall back to a tighter diagnostic: max absolute difference.
        max_abs_diff = np.max(np.abs(ref - y))
        raise RuntimeError(f"Non-determinism detected on run {i}. max_abs_diff={max_abs_diff}")

print("Deterministic inference confirmed: all runs produced identical outputs.")
```

### Key implementation details

#### Single-thread + sequential execution
The most robust baseline configuration for deterministic numerics in ONNX Runtime is:

- `so.intra_op_num_threads = 1`
- `so.inter_op_num_threads = 1`
- `so.execution_mode = ort.ExecutionMode.ORT_SEQUENTIAL`

This forces a stable execution order within the runtime, removing variability introduced by parallel scheduling.

#### Fixed input generation
Input generation must also be deterministic. The example uses NumPy’s `default_rng` with a fixed seed and casts to `float32` to match typical model input types.

#### Exact equality vs tolerance
On CPU with deterministic operators and controlled execution order, outputs are often exactly identical. If a model includes numerically sensitive operations (or uses nondeterministic ops), exact equality may fail even with stable execution order. In that case, comparing with a tolerance is common—but this specific snippet enforces exact determinism as a verification step.

### Practical notes about “factual and current” determinism

Determinism depends on:
- **ONNX Runtime version**
- **Execution provider** (CPU vs GPU)
- **Operator set and implementation**
- **Model graph structure**

The approach above targets the most deterministic configuration available in current ONNX Runtime usage patterns: **CPU provider + sequential execution + single-threaded execution**. This reliably eliminates the dominant source of cross-run variation in production inference test suites.

### Conclusion

Deterministic inference in ONNX Runtime is best achieved by constraining execution parallelism and ordering: set single-thread counts and sequential execution mode, then verify by re-running inference multiple times on identical inputs. With deterministic operators on CPU, this yields identical outputs across runs and provides a strong foundation for stable model regression testing and reproducible deployments.

Machine Learning Posts

Deterministic Inference With Onnx Runtime Using Fixed Random Seeds