Skip to content
ixian
Optimize your model

Methodology

How runs are measured, compared and published on this site.

01

End-to-end latency

Includes preprocessing & postprocessing.

The measured time covers preprocessing, model inference and postprocessing, from preprocessing start through postprocessing completion for the recorded model call.

This scope applies to both reference and optimized end-to-end measurements. Each result retains its unit, such as milliseconds per action chunk.

02

Reference definition

The reference is the implementation recorded for each run. Its provider, precision and measurement conditions identify the baseline. Speedup is reference latency ÷ optimized latency on the same board, rounded to one decimal.

03

Measurement boundaries

Each run says what one number covers: a policy call at p50, a frame interval, or an action chunk. A frame interval is not single-frame latency. Model-inference and generation timings are labeled separately.

Boundaries must match on both sides of a comparison. When they do not, the row carries a note that says how, and a mismatch that works against the faster path is flagged as conservative.

04

Quality results

Speed is listed with the quality result measured with it: closed-loop success for policies, mask IoU for tracking, token-exact agreement for grounding models, max_abs difference for actor networks.

A recorded quality result does not imply a pass/fail threshold. Where a paired test on common seeds exists, such as McNemar, it is shown next to the point estimate. A point estimate alone does not prove that quality was preserved.

05

Configuration changes

A change to the denoise schedule or step count is a different configuration and is named in the measurement conditions. Read quality results with their evaluation suite. Paths recorded as variants are excluded from the fastest-path comparison.

06

Evidence states

Every value is in exactly one state. A missing value is never shown as zero and never inferred from a neighbouring row.

MeasuredReferenceNot reported
07

Records and reproduction

Open measurement details for the implementation, precision and test conditions behind each result. Source reports are linked when available.

08

What this site is not

It is not a universal ranking and not a hardware support list. It is not physical-robot validation unless a row says so.

Publication grades

A has a quality result recorded; it does not establish matching configurations. B carries a condition note. C is an exploratory result; consult its recorded conditions.

Implementation providers

Each implementation path names its provider, such as the model authors or ZETIC. This is distinct from the model developer shown on library cards and comparison charts.

Model authorsNVIDIA public tutorialZETICCoding agent

Decision models

A decision model answers a question about a robot's situation with a probability for each answer option. These rules apply to its latency, resources and Answer accuracy.

01

Single question

One question per request. This is the headline result.

Each request asks one question about one robot state and is answered in a single pass. Library speedups use this scenario.

Rounds run for a fixed time, so the number of requests in a round reflects speed, not coverage.

02

Robot frames

Requests from the robot-frame dataset.

The robot-frame role datasets, without video, give 6,810 requests. Requests over 3,072 tokens are skipped.

Perplexity Flash-4B, d3-mini-4B and JEV-9B are timed on a stratified third of those requests: every third request of each role dataset, 2,272 in all, the same requests for the original and the ZETIC build.

Each model's tokenizer and prompt count tokens differently, so models can skip a different number of requests, and request and question counts can differ by model. Every Robot frames result states its requests per round, and Robot-frame accuracy states its question count.

03

Measurement boundary

From the raw input to a probability for each option.

Latency covers the whole request: preprocessing the raw input (the question text and any images), running the model, and producing a probability for each answer option. Requests run one at a time (batch 1), and results are in ms per request.

The original and the ZETIC build are timed on the same request set on the same board. Warm-up skips each file's first requests, so the timed count can differ by a few between paths; each result shows its own requests per round.

04

Rounds and p50

Four rounds. p50 is the median of the round medians.

Each result runs four rounds. Every round reports its median latency, and the published p50 is the median of those four values.

Robot frames for Perplexity Flash-4B, d3-mini-4B and JEV-9B runs one round per path on its stratified third; its p50 is that round's median.

The board's memory is cleaned before each round. On AGX Thor Robot frames, the original's round 1 (PyTorch eager and torch.compile) was an earlier full pass, so the clean start holds for its rounds 2 to 4; the ZETIC build's four rounds all start clean.

05

Peak memory

Board memory above idle, including model loading.

Peak memory is the highest rise in the board's memory above its idle level after a memory clean. It includes loading the model. It is not the process's own memory (RSS).

Memory a runtime keeps between requests also counts. Working memory for long or image-heavy requests, kept until the run ends, can put a ZETIC build above the original on Robot frames.

06

Power and energy

Whole-board power, from the same round as peak memory.

Average power is the board's mean power draw over the round, in watts, idle draw included. Energy per request is that power times the round's duration, divided by the requests in the round, in joules.

Memory, power and energy come from one round that started on clean memory, normally the lower of the two middle rounds by latency. Lower power alone does not mean less energy per request if requests take longer.

07

Baseline and torch.compile

PyTorch eager by default. torch.compile can be the baseline instead.

Speed is compared with the original model in PyTorch eager, at the precision shown with each result (BF16 or FP16 here). Answer accuracy and agreement use the FP32 original instead, so the two can name different precisions on purpose. You can choose torch.compile as the speed baseline instead. A ZETIC build is never the baseline.

Every torch.compile round starts with empty compile caches (Inductor and Triton), on every board, so no round reuses kernels compiled in an earlier round.

08

Answer accuracy and agreement

Right answers, and matching the original.

Answer accuracy is the share of questions where the top-choice option matches the gold answer. The original (PyTorch eager, FP32) is scored once per model and question set, with the board it ran on. ZETIC builds are scored on each board.

Agreement with original is the share of questions where a ZETIC build picks the same option as the FP32 original, right or wrong. Probability change (p50, p90) shows how far the chosen option's probability moved. Agreement shows how closely a build keeps the original's answers, so it appears only with ZETIC builds, not on Compare.

Kev-4B's build reports no probability change against the original. It shows Max probability change vs eager instead: the largest change in an option's probability against PyTorch eager on the same requests.

09

Question sets

DecisionBench 1,000: 1,000 text questions from DecisionBench, scored against its gold answers. Models share it, so its accuracy can be compared across models.

Robot frames: robot-frame questions in one question set per role, scored against the dataset's answers: scene understanding (RoboBench and ERQA), success detection (RoboReward and UR5-Fail) and safety judgment (ASIMOV and HomeSafe). Accuracy is read per role and never blended into one figure.

Failure diagnosis is not measured yet. A role is shown once it has two or more independent sources and at least 400 answered questions; the failure set has one source and fewer answered questions.

Publisher development splits, such as Kev-4B's, are a publisher's own question sets. They appear only on that model's page and are not comparable with other models.

10

Originals that do not fit on 8 GB boards

ZETIC measurements stand alone when the original runs out of memory.

On Orin Nano, an 8 GB board, the originals of d1-3B, d1-2B, Perplexity Flash-4B, d3-mini-4B and JEV-9B run out of memory in PyTorch eager and in torch.compile. ZETIC latency, energy and memory are shown when measured. With no baseline there, no speedup can be computed.

A ZETIC build that runs on that board is read on its own. Its accuracy is compared with the original's answers from a board where the original fits.