Dynamic Shapes

Handle dynamic ONNX dimensions without turning model compilation into a runtime bottleneck.

Dynamic shapes are useful, but they are not free. A compiled inference backend works best when it knows the shape range it must optimize.

Preferred Pattern

Use a small set of compiled profiles or artifacts:

model-b1-640.mxr
model-b4-640.mxr
model-b8-640.mxr
model-b16-640.mxr

Select the artifact based on runtime batch size.

Python Shape Map

program = migraphx.parse_onnx(
    "model.onnx",
    map_input_dims={
        "images": [8, 3, 640, 640],
    },
)
program.compile(migraphx.get_target("gpu"))

C++ Startup Checks

for(const auto& name : program.get_parameter_names())
{
    auto shape = program.get_parameter_shape(name);
    // Compare shape against the application's configured batch and image size.
}

Avoid

Avoid these patterns in production:

compiling inside the request loop
accepting arbitrary image sizes without a resizing policy
relying on dynamic output shapes when postprocess expects a fixed tensor
mixing batches of incompatible shapes in one worker

Service Policy

For image/video inference, choose one of these:

fixed input size:
  resize/letterbox everything to one compiled shape

small profile set:
  keep multiple saved programs for known batches or resolutions

fallback:
  reject or route uncommon shapes to a slower path

Preferred Pattern

Python Shape Map

C++ Startup Checks

Avoid

Service Policy

On this page