Manages memory-efficient model inference by automatically batching inputs

Service class that orchestrates memory-efficient inference through automatic batching, flexible offloading (GPU/CPU/disk), and OOM recovery.

Format

An R6Class generator object.

Methods

initialize(enc_name, out_dim, out_no_seq)

Create a new inference manager

configure(...)

Configure inference parameters

forward(forward_fn, inputs, auto_batch, output_repeat)

Run inference

Examples

if (FALSE) { # \dontrun{
mgr <- inference_manager$new(enc_name = "tf_col", out_dim = 128L)
mgr$configure(offload = "auto", verbose = TRUE)
} # }