# Retrieval on the Idle iGPUs

Source: https://blog.derio.net/frank/docs/building/37-igpu-retrieval-tier/
Published: 2026-08-02
Updated: 2026-08-02

An external client came to me with a retrieval problem. Their search kept finding the right document and then burying it: recall was fine, ranking was bad. The fix is a reranker, and it has to run somewhere that can take thousands of calls during a bulk import without becoming the bottleneck.



I had three Intel Arc integrated GPUs sitting idle inside my control plane, and no embedding model and no reranker anywhere in the cluster. Twelve LiteLLM aliases, every one of them chat.



By the end of this post you should be able to claim an integrated GPU for an inference workload through Kubernetes DRA, and know how to prove the accelerator was worth claiming.



## The lay of the land



Three bits of background make the rest legible. Skim whichever you already own.



### How a pod gets to touch a device



Kubernetes has been through three generations of “let this container use that piece of hardware”.



The first was ` hostPath ` plus privilege: mount ` /dev ` into the pod and hope. It works, and it hands the pod far more than the one device.



The second was **device plugins**. A daemon on each node advertises an *extended resource* such as ` gpu.intel.com/i915 `, and pods ask for it in ` resources.limits `, the same way they ask for memory. Allocation is a counter the kubelet keeps. You can see how many are free. You cannot see who holds one.



The third, GA since Kubernetes 1.34 and what I run, is **Dynamic Resource Allocation**. Devices are published as ` ResourceSlice ` objects, selected through a ` DeviceClass `, and claimed by a ` ResourceClaim ` that is an ordinary API object with a lifecycle. That last part is what changes your day:



```bash
kubectl get resourceclaims -A
```



Under device plugins, “is the GPU free?” is a number you read and then interpret. Under DRA it is a query that names holders. Run it on your own cluster now; if it returns ` No resources found `, that is a fact about every namespace at once, and it is the kind of fact you can write a test against.



### Two kinds of retrieval model



Retrieval systems run two model classes, and they are not interchangeable.



A **bi-encoder** turns each document into a vector once, ahead of time, and turns your query into a vector at search time. Ranking is vector similarity. It is fast because the expensive work happened during indexing, and approximate because the query and the document never meet inside the model.



A **cross-encoder** takes a query and one document *together* and scores that pair. Far more accurate, far more expensive, and nothing can be precomputed: every query-document pair is its own forward pass. The standard shape is to retrieve fifty-ish candidates with the bi-encoder, then rerank the top twenty with the cross-encoder.



Serving one as though it were the other is a real failure mode, and it stays silent. An LLM-style reranker pressed into a cross-encoder’s slot returns scores around 1e-9 to 1e-12 with almost no separation between them. Nothing raises an error. You get a ranking, and it is noise. Recognising that signature is worth more than any of the plumbing below.



Concretely: ` bge-m3 ` for embeddings, ` bge-reranker-v2-m3 ` for reranking. Both multilingual, which the client’s corpus needed and an English-centric model had been quietly getting wrong.



### What serves a model on an Intel GPU



This is the choice that shapes everything downstream, so it is worth making explicitly.



Most of the obvious answers are CUDA-only. Hugging Face’s ` text-embeddings-inference `, the reflexive pick for exactly this pair of models, has no Intel GPU backend. Ollama, already running on my discrete card, has no rerank endpoint at any size. That removes the two most familiar options before the comparison starts.



What is left:



| Runtime | Endpoints | Cost |
| --- | --- | --- |
| **OpenVINO Model Server** | ` /v3/embeddings `, ` /v3/rerank ` | Non-standard ` /v3 ` paths; models must be converted to OpenVINO IR ahead of time |
| llama.cpp with the SYCL backend | ` /v1/embeddings `, ` /v1/rerank ` at exactly those paths | One model per process, so two Deployments and two device claims |
| Infinity with an OpenVINO backend | both, and LiteLLM has a native ` infinity ` rerank provider | Its OpenVINO path is the least exercised of the three |



I took OVMS: it is Intel’s own runtime for Intel silicon, one server hosts both models behind one device claim, and its ` /v3/rerank ` already returns ` {"results": [{"index": …, "relevance_score": …}]} `, which is the shape the client was already driving.



**Which pains below are OVMS’s and which are inherent:** the missing conversion toolchain, the four export runs, and the ` /v1/config ` diagnostic are all the price of this runtime. The DRA claim, the dirty-page copy failure, and the residency argument would follow from any runtime on this hardware.



## What you need before starting



To reproduce the claim itself:



- **Kubernetes 1.34 or newer**, for ` resource.k8s.io/v1 `. Mine is 1.35.3. Older clusters have the alpha/beta group names, and most tutorials you will find are written against those.
- **A DRA driver for your hardware**, publishing ` ResourceSlice `s and a ` DeviceClass `. Mine is Intel’s resource driver at ` apps/intel-gpu-driver/ `, which I deployed back in [GPU Compute](https://blog.derio.net/frank/docs/building/04-gpu-compute/).
- **Nodes that actually expose the device**, meaning the ` i915 ` kernel module and ` /dev/dri ` present on the host. On Talos that is a machine-config extension.



What you do **not** need is CDI device discovery in containerd. I happen to have it enabled cluster-wide, and it turns out to matter for a different reason, which the verification step below is built around.



## Choosing where retrieval models live



The obvious home was gpu-1, the one node with a discrete card (an RTX 5070 Ti, 16GB). It is the wrong home, and the reason generalises past my fleet.



Chat models and retrieval models have opposite residency profiles. A chat model is large and tolerant of being swapped, because it runs once per turn and a few seconds of load time vanishes into the response. A retrieval model is small and latency-critical: it runs on every query, and thousands of times during a bulk import.



That card serves Ollama with ` OLLAMA_MAX_LOADED_MODELS=1 `, and the default chat model occupies roughly 14 of its 16GB. Putting an embedder behind the same server means every embedding call evicts the chat model and every chat call evicts the embedder. Two workloads that are each fine, thrashing against each other.



The alternative was the three mini PCs that make up my control plane: Core Ultra 5 machines, 14 cores and 64GB each, with Arc integrated graphics that share that system memory. The two retrieval models are 568M parameters apiece, about 600MB each at int8. They fit with room to spare and stay resident.



**The heuristic that transfers:** place models by residency profile before you place them by raw device speed. A small always-hot model on a slower device beats the same model queued behind a large one.



The cost of that choice is that these nodes are my etcd quorum, which sets up the last section.



## Claiming the device



Here is the entire claim:



```yaml
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: ovms-igpu
  namespace: retrieval
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.intel.com   # published by the Intel DRA driver
            allocationMode: ExactCount
            count: 1
```



` apps/ovms-retrieval/manifests/resourceclaimtemplate.yaml:28 `. The ` DeviceClass ` it names comes from the driver’s chart and selects on ` device.driver == "gpu.intel.com" `.



The pod references it in two places, which surprised me the first time:



```yaml
spec:
  resourceClaims:
    - name: igpu
      resourceClaimTemplateName: ovms-igpu   # pod level: what to allocate
  containers:
    - name: ovms
      resources:
        claims:
          - name: igpu                        # container level: who may use it
```



A ` ResourceClaimTemplate ` ties the allocation to the pod’s lifecycle, so it cannot dangle once the pod is gone. A bare ` ResourceClaim ` outlives its consumer and has to be cleaned up.



The two references are halves of one chain. Pod-level allocates; container-level grants use:



```mermaid

    flowchart TD
  POD["Pod<br/>spec.resourceClaims: igpu"]
  CTR["Container ovms<br/>resources.claims: igpu"]
  RCT["ResourceClaimTemplate<br/>ovms-igpu"]
  DC["DeviceClass<br/>gpu.intel.com"]
  RS["ResourceSlice<br/>published by the DRA driver"]
  DEV["/dev/dri<br/>injected into the container"]
  POD --> CTR
  POD --> RCT
  RCT --> DC
  DC --> RS
  RS --> DEV
  CTR --> DEV
  
```



A claim with no container reference allocates the device and grants it to nobody. A container reference with no pod-level claim is rejected outright.



### Do not ask for capacity



An integrated GPU has no dedicated VRAM. It borrows system memory, and the ` ResourceSlice ` says so plainly:



```yaml
capacity:
  memory:
    value: "0"      # shared with the host; there is no separate pool to filter on
  millicores:
    value: 1k
```



So any claim that filters on memory can never be satisfied. Both spellings below are dead ends, and the second is the dangerous one, because a CEL selector is the form I reached for first and the form that reads as more correct:



```yaml
# Neither of these can ever match on a shared-memory iGPU:
capacity:
  memory: "2Gi"
selectors:
  - cel:
      expression: device.capacity["gpu.intel.com"].memory.compareTo(quantity("2Gi")) >= 0
```



The symptom is a pod that sits ` Pending ` with no event explaining itself, which looks exactly like a pod waiting for ordinary scheduling room. The guard against it (` test_claim_requests_no_capacity `, ` scripts/tests/test_ovms_retrieval_manifests.py:198 `) scans string *values* as well as YAML keys, because the CEL form hides the word ` capacity ` inside an expression and a key-only walker sails straight past it.



### Verify the claim, with a negative control



With the claim in place:



```bash
$ ls -ln /dev/dri/
crw-rw-rw-  1 0 0 226,   0 card0
crw-rw-rw-  1 0 0 226, 128 renderD128
```



Two things worth noticing. The render node is world-accessible, so the usual hunt for the ` render ` group ID to drop into ` supplementalGroups ` is unnecessary here, and OVMS’s uid 5000 can open the device with no extra grants.



Now run the identical pod **without** the claim:



```bash
$ ls -ln /dev/dri/
ls: /dev/dri/: No such file or directory
```



That second run is the one that earns the conclusion. My containerd has CDI device discovery enabled cluster-wide, which makes “the device was going to be there anyway” a completely plausible reading of the first result. It was not: the claim is what injects the device. A positive result on its own would not have told me which mechanism I had tested, only that something worked.



## Getting the weights in



OVMS can pull a model straight from Hugging Face. For these two models that path is a dead end, and it fails in the least useful order.



Run it plainly and it downloads raw safetensors, writes a graph, reports the server healthy, then fails the *model*:



```
Either openvino_tokenizer.xml was not provided or it was not loaded correctly
```



Add ` --weight-format int8 ` to force conversion and the real problem appears:



```
Trying to pull BAAI/bge-reranker-v2-m3 from HuggingFace but missing optimum-intel.
Use the ovms package with optimum-intel installed.
```



No published OVMS image carries ` optimum-intel `. This release ships ` 2026.2.1 ` and ` 2026.2.1-gpu `, and neither will convert anything.



The documented escape hatch is to use an already-converted model from the ` OpenVINO/ ` organisation on Hugging Face. That organisation publishes ` bge-base-en-v1.5 ` and ` bge-reranker-base `: English-only variants, which is the exact model class this work existed to move away from.



**So here is the fork.** Fork the runtime image to add the conversion toolchain, or convert ahead of time and ship the artifacts. I took the second, which keeps the runtime stock: the only thing I maintain is a bag of model files, so an OVMS upgrade becomes a tag bump instead of a rebuild of a forked server.



The build uses OVMS’s own ` export_model.py `, pinned to the same release as the serving image, because it emits everything the server expects in one pass: the IR, the ` openvino_tokenizer.xml ` whose absence caused the first failure, a per-model ` graph.pbtxt `, and the merged ` config.json `.



```dockerfile
FROM python:3.12.13-slim-bookworm AS export
# export_model.py AND its requirements.txt come from the same pinned ref,
# so the script and its dependency list cannot drift apart.
RUN python export_model.py embeddings_ov --source_model BAAI/bge-m3 \
        --weight-format int8 --target_device GPU \
        --config_file_path /out/gpu/config.json --model_repository_path /out/gpu
# ...three more: rerank on GPU, then both again targeting CPU...

FROM busybox:1.38.0-uclibc
COPY --from=export /out /models-src     # artifacts only; the toolchain stays behind
```



Four export runs, because ` target_device ` is baked into each ` graph.pbtxt ` at export time. A CPU repository is a second set of files, not a runtime flag, and building it now is what makes the comparison at the end possible.



One CI note, because it applies to any repo that builds images from a Dockerfile it also tests. Every other image workflow here triggers only on ` push ` to ` main `:



```
$ for f in .github/workflows/build-*.yml; do printf "%-42s " "$(basename $f)"; \
    python3 -c "import yaml,sys;d=yaml.safe_load(open('$f'));print(', '.join(sorted((d.get(True) or d.get('on')).keys())))"; done
build-ai-alert-helper.yml                  push, workflow_dispatch
build-caddy.yml                            push, workflow_dispatch
build-comfyui.yml                          push, workflow_dispatch
build-openrgb.yml                          push, workflow_dispatch
build-ovms-retrieval-models.yml            pull_request, push, workflow_dispatch
```



With a ` push `-only trigger, a Dockerfile change reaches ` main ` having never been compiled: the pull request goes green because the *tests* pass, and the first evidence the image builds arrives after the merge. This workflow builds on ` pull_request ` without touching the registry, and builds-and-publishes on ` push `. That earned itself immediately by failing on a hand-copied requirements list that was missing ` requests `, five seconds into the build.



## Making the pod tell the truth



Three ways this deployment could have run green while broken.



### A health endpoint that answers a different question



OVMS serves ` /v2/health/ready `. It returns **200 while a model sits in ` LOADING_PRECONDITION_FAILED `**, because it reports on the server, not on the model. Point a readiness probe there and you get a Ready pod, a green ArgoCD Application, and every request failing.



So readiness points at a model instead:



```yaml
startupProbe:                      # first boot compiles IR for the iGPU: minutes, not seconds
  httpGet: { path: /v2/models/bge-m3/ready, port: 8000 }
  periodSeconds: 10
  failureThreshold: 60
readinessProbe:
  httpGet: { path: /v2/models/bge-reranker-v2-m3/ready, port: 8000 }
livenessProbe:                     # server level is CORRECT here
  httpGet: { path: /v2/health/live, port: 8000 }
```



Putting one model on startup and the other on readiness gates both with plain ` httpGet `, no shell needed in the stock image. Liveness stays server-level on purpose: restarting a healthy server whose model failed to load would crash-loop away the very thing you need to query.



That thing is ` /v1/config `, which reports per-servable state and is the diagnostic of record:



```bash
$ curl -s http://ovms-retrieval.retrieval.svc.cluster.local:8000/v1/config
"bge-reranker-v2-m3": { "model_version_status": [ { "version": "1",
  "state": "AVAILABLE", "status": { "error_code": "OK" } } ] }
```



### A seed that copies once and never again



The model repository is seeded onto a PVC from the model image by an initContainer (` apps/ovms-retrieval/manifests/deployment.yaml:92 `). The tempting shape is “copy if the directory is missing”, and it is wrong: the next model version never reaches an already-populated volume, the pod stays Ready, and stale weights serve indefinitely. I have that bug on record from another app whose custom-node volume seeded exactly once, then served the same code through three image bumps without complaint.



So the copy is gated on a marker file holding the image revision, and the marker is written **last**, so an interrupted seed re-runs on the next start; a half-copied repository never gets to call itself current.



That initContainer deliberately does not reference the ` igpu ` claim. It copies files; holding a GPU for the whole init phase would block nothing but itself, and would keep the device from any other claimant while it worked.



### A copy that gets OOM-killed in one second



The seed initContainer was killed four times, exit 137, roughly one second each time, with a 512Mi limit against a 2.4GB copy.



One second is too fast for “it gradually used too much memory”. This is dirty-page pressure. Reads come off local overlayfs at NVMe speed while writes go to a Longhorn network volume, so dirty pages accumulate faster than writeback drains them, and cgroup v2 charges page cache to the cgroup. Clean pages get reclaimed under pressure; dirty pages cannot be reclaimed until they are written back, so the limit is reached almost immediately.



Two halves fix it, and both carry load. The copy becomes per-file with a flush between (` deployment.yaml:136 `, inline in the initContainer’s script):



```sh
cd /models-src/gpu
find . -type f | while IFS= read -r f; do
  cp "$f" "/models/$f"
  sync                     # bounds the dirty set to ONE file, not the whole tree
done
```



and the limit goes to 2Gi. Raising the limit alone would fix these two models and break on a larger one; syncing alone would leave the ceiling below a single file. The largest file here is about 600MB, so 2Gi is roughly triple the bound that actually matters, and adding a third model later will not require touching it.



Bare ` sync `, with no file operand. GNU coreutils accepts ` sync FILE `; busybox’s applet does not, and this image is busybox, so ` sync /models/x ` would fail there and abort a seed that had already copied every byte.



## Reaching the models



The service is a ClusterIP on port 8000 and nothing else. No LoadBalancer, no ingress, and deliberately no LiteLLM alias, even though LiteLLM is my gateway for all twelve chat models.



That last one was tempting and I skipped it. LiteLLM’s rerank providers each assume their own URL path, so putting ` /v3/rerank ` behind the gateway means writing a shim, which is real work. Aliasing only the embeddings half would have been a line of config, and would have left half a pair on the gateway and half off it, which is worse than neither. In-cluster callers use the service directly:



```
POST http://ovms-retrieval.retrieval.svc.cluster.local:8000/v3/rerank
{"model": "bge-reranker-v2-m3", "query": "...", "documents": ["...", "..."]}

→ {"results": [{"index": 0, "relevance_score": 0.0395}, ...]}
```



Embeddings are OpenAI-shaped at ` /v3/embeddings `, and the returned vectors are 1024-dimensional. Read that number from the response. It is a property of the model, and the next model will differ.



## Proving the accelerator earned it



I had a number early: reranking twenty candidates took 77ms at p50 on the iGPU. That number on its own justifies nothing, because these are 14-core machines sitting around 95% idle. “An iGPU beats these CPUs for a 568M cross-encoder” is a hypothesis, and the reason the model image builds a CPU repository alongside the GPU one is so it can be tested.



The control arm is a throwaway pod on the same node, from the same image, seeded from ` /models-src/cpu `, holding **no** ` ResourceClaim `:



```bash
kubectl apply -f apps/ovms-retrieval/cpu-arm-pod.yaml
uv run python3 scripts/ovms-retrieval-bench.py --arm cpu --base-url http://<pod-ip>:8000
kubectl delete -f apps/ovms-retrieval/cpu-arm-pod.yaml
kubectl get resourceclaims -A     # residue check
```



Same weights, same node, same server build; the only variable is which repository was seeded. Both arms ran 30 timed iterations after 3 warm-up calls, reranking 20 candidates and embedding batches of 32, over short generated passages.



| Measurement | iGPU | CPU | Ratio |
| --- | --- | --- | --- |
| Rerank, 20 candidates, p50 | **77.3 ms** | 1612.9 ms | **20.9×** |
| Rerank p95 | 87.1 ms | 1702.6 ms | 19.5× |
| Embeddings, batch of 32, p50 | 93.2 ms | 1928.0 ms | 20.7× |
| Embedding throughput, batched | 343 items/s | 16.6 items/s | 20.7× |
| Embeddings, single item, p50 | 28.4 ms | 32.2 ms | **1.1×** |
| Embedding dimension | 1024 | 1024 |  |



The iGPU wins by about twenty times on everything batched. On the last row it is worth almost nothing.



That row is what I would have missed by measuring only the thing I built. A single embedding call is dominated by request overhead rather than by compute, so there is barely any compute for the accelerator to accelerate. Anyone quoting “20× faster” for one-off embedding calls would be wrong, using my own numbers to be wrong with.



One honest limit on that ratio: a cross-encoder’s cost scales with sequence length, and these passages are short. Longer documents will move the number, plausibly in the iGPU’s favour, and I have not measured that — though I have since measured what sequence length does to *memory*, which turned out to matter more than what it does to latency. That is the September update below.



There is a second thing sitting in the CPU column. The original estimate for this workload was 1 to 5 seconds per query, arrived at by arithmetic instead of measurement. The CPU arm lands at 1.6 seconds, inside that range. The estimate had accidentally been a CPU estimate all along.



### Make the harness able to contradict you



A benchmark flag that says ` --arm gpu ` and writes that string into its own output has recorded an intention, not a measurement. The procedure needs two flags changed together, ` --arm ` and ` --base-url `, and forgetting either produces a well-formed, authoritative-looking record that attributes CPU latency to the GPU. Nothing downstream can catch it.



The harness therefore records the base URL, a UTC timestamp, and a snapshot of the server’s ` /v1/config `, all keyword-only with no defaults, so a result cannot be assembled without its evidence. Where the server reports a device, it cross-checks and exits before timing anything.



` /v1/config ` turns out to report servable *state* and not device, so that cross-check cannot fire on this runtime. The docstring says exactly that, and says the device claim rests on which repository was seeded. Recording what you can check, and being explicit about what you cannot, beats a confident field that might be lying.



### What it cost the control plane



These nodes are etcd members, so the last question is whether any of this hurt. I ran 7500 rerank requests over 240 seconds, about 31 per second, from a curl loop on the node itself:



| Signal | Before | Under load |
| --- | --- | --- |
| apiserver p99, ` WATCH `/` CONNECT ` excluded | 23.4 ms | 22.1 ms |
| Node ` Ready ` | True | True |
| MemoryPressure / DiskPressure | False | False |
| ovms working set |  | 2254 Mi of 6 Gi |



That 6 Gi ceiling is 16 Gi now, and the September update below is why.



**A correction I owe this paragraph, added later:** “about 31 per second” is the most expensive sentence in this post, and not because it is wrong — because it is unusable. I did not record what that curl loop sent. Six weeks later a re-run of the same soak measured ~16.5/s, that got read against this figure as a halving, and it cost a session to work out that the two numbers were never comparable. The arithmetic was sitting in this same post the whole time: 31/s from a *sequential* loop is ~32 ms per request, and the table two sections up says twenty candidates cost 77.3 ms. A sequential loop cannot beat its own latency, so whatever that loop sent, it was not the twenty-candidate shape — and the server was never the thing that changed. **Write down the payload shape and the client concurrency next to every throughput number, or you have recorded a rate that belongs to a request you can no longer identify.** The full account is in the September update.



Excluding ` WATCH ` and ` CONNECT ` from that quantile is essential. Include them and the p99 reads 60 seconds flat both before and during, because long-poll watches sit at the apiserver timeout and swamp everything else. It looks like a catastrophe and carries no information.



And a gap I would rather name than paper over: I cannot report etcd leader changes or WAL fsync latency, because I do not scrape my own etcd. The ` etcd_* ` series I do have are the apiserver’s etcd *client* metrics; ` etcd_server_leader_changes_seen_total ` does not exist on this cluster. The evidence I have says no measurable impact, and the strongest signal I would have wanted is missing.



## Update, September 2026: the reranker took the whole server down



A month after this went in, ` /v3/rerank ` started killing the pod. Not slowly — connection closed without response, then about ten seconds of refused connections, then a healthy server again. Both servables live in one process behind one device claim, so every embeddings caller went down with the reranker. The restart counter walked from 1 to 9 over a week of light use.



The obvious reading is that some request is simply too big. That reading is wrong, and finding out cost me two confident wrong conclusions.



**The batch is one tensor, and its width is the longest document in it.** ` rerank_calculator_ov.cc ` builds a single inference input of shape ` B × T `, where ` T ` comes from the longest document in the request and every other document is padded up to it. This reranker is an XLM-RoBERTa-large cross-encoder — 24 layers, 16 heads — so attention is ` B × heads × T × T `. Measured against the served model, reading the cgroup’s own high-water mark:



| documents | tokens/doc | call cost |
| --- | --- | --- |
| 64 | 332 | 1.29 GiB |
| 40 | 498 | 1.69 GiB |
| 50 | 511 | 2.17 GiB |



Cost per row is about ` 0.0202 GiB × (T/332)^1.78 `. Near-quadratic in document *length*, and only linear in document *count*. So the lever that matters is the sequence bound, not the document cap — the reverse of the order I reached for them in, and the reverse of how the bug report framed it. A guard that only counts documents is walked straight past by one long passage.



**And the memory never comes back — because it is pinned GPU memory, not process memory.** The iGPU has no VRAM, so every allocation OpenVINO makes on it is a shmem-backed host page, *unevictable* inside the container’s cgroup. With the server holding 5.11 GiB, ` memory.stat ` reads ` shmem ` 5.11 GiB, ` unevictable ` 5.11 GiB — the same number — ` inactive_file ` and ` active_file ` both zero, ` anon ` 0.55 GiB, and swap disabled. There is no page cache to drop and nowhere to swap, so the kernel may reclaim *nothing* and kills instead. ` limits.memory ` on this container is not a margin around a process: it is the GPU’s memory budget. Baseline after load is 1.71 GiB — the weights, pinned on the GPU — and the pool grows from there toward the largest request shape served, never shrinking.



That is half the bug, and it is the half the report could not see. Take the largest batch this endpoint is expected to serve: it costs 2.93 GiB, so on a freshly restarted pod it totals 5.14 GiB and fits inside the 6 GiB limit with room to spare. It dies only once the floor has risen underneath it. A reranker that works, then doesn’t, then works again after a restart is not a request-size problem, it is a ratchet — and a ratchet explains 1 → 9 restarts across a week of light use, which no single request size does. It also means restarting the pod was a real mitigation the whole time, not a superstition — a restart is the only operation that returns the pool to baseline. So the cap and the restart are not alternatives: the cap bounds what one request may allocate, and only a restart bounds what a long-lived server accumulates. Both ship.



One more thing worth saying plainly, because it cost me three wrong explanations in a row: **none of the obvious metrics can see this.** ` memory.current ` counts pinned shmem *and* page cache, so a high-water reading looks like a leak. ` container_memory_working_set_bytes ` cannot separate them either. And ` container_memory_rss ` counts neither — it sat flat at ~0.45 GiB on busy days and idle days alike, which reads as proof that nothing is accumulating. Only ` /sys/fs/cgroup/memory.stat `, read inside the container, distinguishes them.



Two measurement traps, and I walked into both.



A 20-second metrics scrape cannot see a 0.1 to 2.5 second transient. ` container_memory_working_set_bytes ` reported a comfortable idle figure across calls that had reached the limit. The instrument is ` /sys/fs/cgroup/memory.peak `, read inside the container.



And because that peak is monotonic while the floor ratchets, only a single call on a freshly restarted container measures anything at all. An ascending sweep’s per-size deltas are increments of a high-water mark, not per-call costs. I read them as costs twice, and concluded from it that forty documents at ` T `≈664 was fatal — those calls had each run on a container already holding 3.5 to 4.8 GiB from the call before. The error runs in the direction that makes you cap far too tightly, which is the kind of wrong that ships.



### The fix was two fields that already existed



` RerankCalculatorOVOptions ` has carried ` max_allowed_chunks ` since long before this deployment, defaulted to **10000** and checked three times, the first check before a single token is allocated. ` max_position_embeddings ` sits next to it. Neither appears in the graph template ` export_model.py ` emits, so both defaults applied — four orders of magnitude above what this container’s memory limit can serve. The server had a bouncer; nobody had told him the room holds fifty.



` export_model.py ` will not emit them and I am not forking it, so the model image’s build injects both into every exported ` graph.pbtxt ` — and then re-reads each graph and fails the build if the fields are absent. That second half is the part worth copying. A ` sed ` that silently matched nothing would have published a model revision whose every downstream signal — CI green, ArgoCD Synced, pod Ready, seed marker at the new revision — agreed the guard had shipped.



The pair is ` max_allowed_chunks: 64 ` and ` max_position_embeddings: 640 `, and the container’s ` limits.memory ` went from 6Gi to 10Gi — and then, after the post-merge test plan, to 16Gi. Read that raise the right way round: the ceiling moved so the guard cannot OOM-kill the server it protects, not to make room for a bigger workload. The second raise is the more interesting one, because it corrected a sizing error. I had measured the cap as a **single call on a fresh pool**: 5.67 GiB. The expensive case is an **ascending sequence** — 20 then 40 then 64 documents — which peaks at **8.49 GiB**, because the older pool is still held while the new one is allocated. That is 84.9% of a 10Gi ceiling, against a test-plan threshold of 70%. I could not measure it before merging: the 10Gi limit did not exist yet. These are still etcd members, so ` requests.memory ` stayed at 2Gi and the scheduler’s view of the node did not change — but the honest cost is that this memory is pinned and unevictable, and a third raise should tighten the graph bound instead.



Two costs, both accepted deliberately. A document longer than ` T ` is no longer scored whole: it is split into chunks, scored per chunk, and its chunks count against the same 64 — so relevance scores move for long passages. And a refused request comes back naming the limit, which was the whole point: ` HTTP 400 Chunking failed: exceeding max_allowed_chunks after chunking limit: 64; actual: 100 `.



That 400 is worth a sentence, because I predicted 500 here and was wrong. The reasoning was sound as far as it went — each guard raises ` std::runtime_error `, ` Process() ` catches it into ` absl::InternalError `, and that maps to 500 — but the MediaPipe graph wraps the failure before it reaches the HTTP layer. I had written the number into this post from the source rather than from a request. The test plan *recorded* the status instead of asserting it, which is the only reason this surfaced as a pleasant surprise rather than a failing row. Reading the code tells you what the code says; it does not tell you what the server returns.



### A throughput number I could not use



One more correction belongs here, and it is the one the earlier paragraph pointed at. Six weeks after this layer went in, the acceptance soak was re-run and measured ~16.5/s — against the “about 31 per second” recorded above. That reads as a halving, and the batch guard was the obvious suspect: it had just shipped, and bounding memory plausibly costs throughput.



It cost nothing. Re-measuring the twenty-candidate shape in-container gave **p50 60 ms**, *faster* than the 77.3 ms this post recorded before the guard existed. What the measurement did explain is the ceiling itself:



|  |  |
| --- | --- |
| warm p50, 20 candidates | **60 ms** → 16.6/s sequential |
| 4 calls issued in parallel | **240 ms** — 4 × 60, no overlap at all |
| soak, 16 workers | 16.59/s, p50 787 ms |
| soak, 48 workers | 16.49/s, p50 **2227 ms** |



` export_model.py ` defaults ` --num_streams ` to 1 and I never passed it, so every servable here runs ` plugin_config: '{"NUM_STREAMS": "1" }' `. One stream means inference serialises, throughput is ` 1/latency `, and client concurrency converts itself into queueing delay and nothing else. Tripling the workers tripled the latency and left the rate alone — which is the signature to look for, and much better evidence than my four-parallel probe on its own.



So the guard is exonerated twice: the same shape got faster, and it could not have touched concurrency anyway, because the rerank export command is byte-identical across both model revisions at the same pinned exporter ref. ` NUM_STREAMS ` has been 1 since the first export in August. The regression was never in the server — it was in my notes, and specifically in a rate recorded without the request that produced it.



` --num_streams ` is the real lever and I have not pulled it, because each stream carries its own set of those pinned, unevictable buffers: more streams multiply the peak that 16Gi has to absorb, against a watchdog that restarts the pod at 53% of it. That is a batch-curve measurement, not a config tweak — and nothing has yet driven this endpoint hard enough to need it.



## Missteps



| What Happened | Why It Was Wrong | How We Fixed It | Commit |
| --- | --- | --- | --- |
| **Designed model acquisition around ` ovms --pull `** — the whole seeding path assumed the server could fetch and convert | No published OVMS image carries ` optimum-intel `, and the pre-converted ` OpenVINO/ ` org has English-only variants of both models | Build the OpenVINO IR in CI with OVMS’s own ` export_model.py `; keep the runtime stock | ` 2a6bb5db ` |
| **Readiness probe on ` /v2/health/ready `** | Server-level endpoint: returns 200 while a model sits in ` LOADING_PRECONDITION_FAILED `, giving a Ready pod that serves nothing | Probe ` /v2/models/<name>/ready `; keep liveness server-level | ` 2a6bb5db ` |
| **Seed copied the whole tree with ` cp -R ` under a 512Mi limit** | Dirty-page pressure, not volume: fast overlayfs reads into slow network writes, and cgroup v2 cannot reclaim dirty pages | Per-file ` cp ` with a bare ` sync ` between; limit raised to 2Gi, sized to the largest single file | ` e0089963 ` |
| **Hand-copied upstream’s ` requirements.txt ` with a comment claiming parity** | It was a subset, missing ` requests `; the image build died five seconds in | Fetch the requirements from the same pinned ref as the script itself | ` e0089963 ` |
| **Seed-source test matched its own explanatory comment** | The block scalar carries its ` # ` comments into the scanned string, so flipping the real ` cp ` to the CPU repository still passed | Strip comment lines, assert the executable line | ` 2a6bb5db ` |
| **Read an ascending memory sweep’s per-size deltas as per-call costs** | ` memory.peak ` is monotonic and the resident floor ratchets, so every size after the first was measuring the sizes before it — it put the fatal batch at forty documents when forty is fine on a fresh pod | Measure one call per container start, and restart between sizes | [#793](https://github.com/derio-net/frank/issues/793) |
| **Went looking for ` --max_doc_length ` as the batch cap** | It never reaches ` graph.pbtxt `; it sets the exported tokenizer’s ` model_max_length `, which is a per-document truncation length and says nothing about how many documents a request may carry | Set ` max_allowed_chunks ` and ` max_position_embeddings ` in the graph, which is where the calculator reads them | [#793](https://github.com/derio-net/frank/issues/793) |
| **Published the refusal status as 500, read off the source** | ` std::runtime_error ` → ` absl::InternalError ` → 500 is correct about the code and wrong about the server: the MediaPipe graph wraps the failure before the HTTP layer, and the request returns **400** | Run the request. The test plan recorded the status rather than asserting it, which is why the gap surfaced | [#805](https://github.com/derio-net/frank/issues/805) |
| **Recorded “about 31 per second” without the payload that produced it** | A bare rate is unfalsifiable and uncomparable: six weeks later a ~16.5/s re-run read as a halved-throughput regression, and the suspect was a guard that had in fact made the server *faster* | Record payload shape and client concurrency beside every throughput figure; the ceiling turned out to be ` NUM_STREAMS: 1 `, unchanged since the first export | [#810](https://github.com/derio-net/frank/issues/810) |



## Recovery Path



| Symptom | Cause | Fix |
| --- | --- | --- |
| Pod stuck ` Pending `, no explanatory event | Claim filters on capacity; a shared-memory iGPU reports ` memory: "0" ` | Remove the ` capacity: ` key or CEL memory selector from the ` ResourceClaimTemplate ` |
| Pod ` Ready `, every request fails | Readiness pointed at ` /v2/health/ready ` | Repoint at ` /v2/models/<name>/ready `; diagnose with ` GET /v1/config ` |
| ` seed-models ` exits 137 within seconds | Dirty-page pressure copying to a Longhorn volume | Per-file copy plus bare ` sync `; raise the limit above the largest single file |
| ` ImagePullBackOff ` on first sync, no manifest change helps | GHCR creates a package **private** on first publish, and the pod has no ` imagePullSecret ` | Set the package visibility public, or add a pull secret |
| Stale weights served after a model bump | Seed marker still matches the unchanged ` MODELS_REV ` | Bump ` MODELS_REV ` so the image tag and the marker both move |
| ArgoCD ` Synced/Healthy ` but the change is not live | Synced to a stale revision | Trigger an explicit sync operation with ` syncOptions ` passed explicitly, then assert on the artifact and not on the sync status |
| Rerank kills the pod on a batch that worked yesterday | The resident floor has ratcheted to the high-water of the largest call since the last restart | ` kubectl -n retrieval rollout restart deploy/ovms-retrieval ` resets the floor immediately; the graph’s two bounds are what stop it recurring |
| Rerank returns HTTP 400 on a large batch | The guard refusing the request, before a single token is allocated | Read the message: it names the limit (` exceeding max_allowed_chunks after chunking limit: 64; actual: 100 `). Send fewer or shorter documents; do not raise the cap without re-measuring |
| Adding client workers raises latency and not throughput | ` NUM_STREAMS: 1 ` — inference is serialised, so the ceiling is ` 1/latency ` (~16.6/s on the 20-candidate shape) and concurrency becomes queueing delay | Nothing to fix on the client. The only lever is ` --num_streams ` at export, which multiplies the pinned-buffer peak against the 16Gi ceiling — measure it like a batch curve first |



## What transfers



**An accelerator claim you cannot falsify is a label.** If your benchmark output records the device you told it to use and nothing about what actually answered, you have recorded an intention. Make the artifact carry enough context to contradict its own headline.



**Benchmark the arm you are trying to beat.** A speedup with no control is folklore. The control here cost one extra export inside an image that was being built anyway, and it is the only reason the batched-versus-single distinction showed up in a table now, with time to act on it.



**Ask a health endpoint what it is actually about.** Server health and workload health are different questions, and an HTTP 200 will not distinguish them for you. Find the endpoint that reports on the thing you care about and point readiness at that.



**A copy that dies instantly is about dirty pages, not volume.** Fast reads into slow writes will blow a cgroup limit long before the transferred total looks alarming. Bound the dirty set first, then size the limit against the largest single file.



**A failure that a restart fixes is a ratchet, not a request.** If memory never returns after a peak, the resident floor is the high-water mark of the largest thing you have ever served, and the same request will succeed on a fresh process and die on a warm one. Measure one call per process start, or you are measuring your own sweep.



**Prove device injection with a negative control.** Run the workload without the claim. If the device is still there, something else in your stack is providing it, and the claim you just wrote is decoration.



## References



- [OpenVINO Model Server](https://github.com/openvinotoolkit/model_server) · [rerank endpoint docs](https://docs.openvino.ai/2025/model-server/ovms_docs_rest_api_rerank.html)
- [Kubernetes Dynamic Resource Allocation](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/)
- [Intel resource drivers for Kubernetes](https://github.com/intel/intel-resource-drivers-for-kubernetes)
- [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) · [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3)
- Prior layer: [GPU Compute](https://blog.derio.net/frank/docs/building/04-gpu-compute/) · [Local Inference](https://blog.derio.net/frank/docs/building/10-local-inference/)
- Companion: *Operating on Frank — Inference*