RESEARCH / NOTES

Runtime

Everything around the calculation

What happens to a fast calculation once I include everything needed to run it and return a usable answer?

Status
Execution and interoperability experiments
Reading time
3 minutes
Filed under
  • Runtime engineering
  • Distributed systems
  • Performance
Updated

The kernel is only one part of the wait

A kernel benchmark starts with data ready and measures the calculation. An execution system has to do more: find resources, prepare inputs, move them, run the operation, check the result and return it to the caller.

I built runtime services to make those steps explicit. They handle object lifetimes, execution requests, result publication and fallback. The harder question was whether the whole service could justify its cost—not merely whether it could run a matrix multiplication correctly.

Keeping data remote was not enough

I compared local execution with cold remote, warm remote and a remote path that kept an operand at the worker. Keeping data there removed some repeated movement, but the complete repeated workload still took about 12.9 seconds at the median against 1.75 seconds locally.

All 960 recorded calls passed their numerical checks. That made the negative result more useful: the problem was not that the worker could not calculate. Uploads, result materialization and required local checking outweighed the compute benefit. The resident route was about 7.4 times slower at that boundary.

Elapsed time · lower is faster
Local execution, median1.75 s
Cold remote execution, median18.3 s
Warm remote execution, median16.3 s
Resident remote execution, median12.9 s
Measurements and how I compared them ↗

How little overhead could a prepared plan add?

I then asked a narrower question. If I prepared and calibrated a plan before the timed operation, could the dispatch machinery stay reasonably close to direct local execution?

For a small eight-call sequence, it took about 228 microseconds at the median against 211 microseconds directly. That stayed inside the overhead allowance I had set, but it was not faster. Every automatic choice in that measured run was local. The forced worker tests used two processes on the same CPU host, not two different kinds of hardware.

First use brought the cost back

The first operation includes the work a prepared measurement leaves out. In the later run, first use took about 8.64 ms through the calibrated path against 0.33 ms locally. The inclusive experiment was also slower.

I keep those results next to the steady ones because they answer different user questions. Someone repeatedly running an established operation may amortize setup; someone waiting for one calculation still has to pay it. A passing overhead target on a prepared path cannot answer both.

Crossing operating systems was another kind of result

I also returned matrix results from a Linux worker to a Windows client and checked object references, stale handles and cleanup. That established a small functional exchange, not a large-message or GPU performance claim.

The installer and desktop work exposed a similar distinction. Installing files, authenticating a request and running a command-line calculation are not the same as a reliable first-run experience. Incomplete GUI attempts remain part of the journal rather than being described as a finished product.

What I measure now

I want the execution system to explain the wait, not conceal it behind a fast kernel. That means separating first use from steady execution, knowing what data is already present, and including the checks needed to trust the returned answer.

The next convincing placement result would be a complete supported workload that beats an equally optimized local path. Until then, choosing local is often the correct outcome of the system I am building.

Technical details and evidence

The four-mode GPU comparison is the historical R14 campaign: thirty paired blocks, eight calls per mode, 960 numerically checked calls, rotated mode order and 10,000 seeded paired bootstrap resamples. Its target was at least 20% lower latency than the local baseline. The complete timer includes required verification; kernel time cannot substitute for it.

The prepared CPU test allowed at most 10% median and 15% p95 overhead. Its observed ratios were about 1.081 and 1.136; the paired mean-ratio interval is not a confidence bound on those quantile ratios. First-use and full-experiment failures remain separate records.

The Windows-to-Linux object test checked sixteen n4 output elements, object references, stale/released handles and return to zero resident bytes. It does not establish reverse-direction execution, Windows GPU support or product-wide interoperability.

The prepared path once it is warm and calibrated

Technical details and evidence
Experiment identifier
rt-prepared-s5-steady-eight-call
Original title
Prepared calibrated steady-state eight-call workload, ratio against the best eligible local baseline
Measurement boundary
The steady-state leg of a prepared multi-resource qualification: a pre-built artifact is attached and calibrated before measurement, so the boundary deliberately excludes preparation and calibration cost. The ratio against the best eligible single target passed the predeclared bounds.
Hardware
Two worker processes on one admitted CPU host, described in the receipt as correlated processes sharing the same hardware. The host is recorded as Ubuntu 24.04 with kernel 6.8 on EPYC-Genoa, with the Rust toolchain at 1.99.0 and one declared thread. The receipt states explicitly that this is not a heterogeneous resource set.
Software
A private immutable prepared plan retaining owned input identity plus live session, backend, memory, capability, affinity and freshness checks; one authoritative placement decision per two-GEMM region; a common optimised runner with a visible fallback trace.
Input
A small-integer N32 two-dependent-GEMM region executed as eight calls per block, with all 1,024 final output elements checked on every measured call.
Samples
1,200 measured calls, 450 warm-ups and 150 phase records across five arms of 30 paired blocks each. 240 calibrated measured rows with valid profiles and ages between 120 and 438 microseconds.
Statistical method
Nearest-rank block median and 95th percentile over 30 blocks, expressed as a ratio against the fixed named best eligible prepared local baseline, with a paired mean-ratio bootstrap of 10,000 resamples at a fixed seed.
Cold / warm state
The qualifying arm is the calibrated prepared steady state. Preparation and calibration costs are charged to a separate first-use boundary and are not folded into the steady figure.
Network
Loopback between the client and the two worker processes on the same host. Directional stream accounting excludes the enclosing transport overhead and private allocator peaks.
Serialization
Traffic records retain actual remote transfer and residency behaviour separately from dispatcher local behaviour. Per-call codec and transfer accounting is carried in the campaign receipts.
Correctness checks
Every one of the 1,200 measured calls reports 1,024 final output elements checked; the source aborts on any mismatch and the launch exit is zero. Independent re-summing of all eight caller times in each of the 150 blocks found zero mismatches, and no failed traffic records appear in the completed phases.
Original measurements — full precision
Original metricExact valueSamplesStatistic
prepared applied whole-workload wait, median0.228 ms30nearest-rank block median, 228 microseconds
prepared applied whole-workload wait, 95th percentile0.259 ms30nearest-rank block 95th percentile, 259 microseconds
best eligible local whole-workload wait, median0.211 ms30nearest-rank block median, 211 microseconds
best eligible local whole-workload wait, 95th percentile0.228 ms30nearest-rank block 95th percentile, 228 microseconds
median ratio against the best eligible local baseline1.0805687204 ratio30observed block median ratio against the 1.10 bound
95th percentile ratio against the best eligible local baseline1.1359649123 ratio30observed block 95th percentile ratio against the 1.15 bound
paired mean-ratio interval, lower bound1.0469289615080855 ratio3010,000 paired resamples at the fixed seed
paired mean-ratio interval, upper bound1.1081226992587965 ratio3010,000 paired resamples at the fixed seed
measured calls1200 calls1501,200 measured calls across 150 phase records
warm-up records450 calls150untimed warm-up records in the campaign
phase records150 blocks150phase records carrying eight measured calls each
calibrated measured rows with valid fresh profiles240 calls150profile ages between 120 and 438 microseconds
forced worker regions executed480 calls150the recorded 240 real regions on each of two forced worker arms, summed

CALCULATED FROM THE MEASUREMENTSbest eligible local whole-workload wait, median ÷ prepared applied whole-workload wait, median = 0.92543860×8.0568720% higher time for prepared applied whole-workload wait, median.

CALCULATED FROM THE MEASUREMENTSbest eligible local whole-workload wait, 95th percentile ÷ prepared applied whole-workload wait, 95th percentile = 0.88030888×13.596491% higher time for prepared applied whole-workload wait, 95th percentile.

Experimental limitations

  • This is the steady leg only. Preparation, calibration, refresh and full-experiment cost are charged elsewhere and fail.
  • All measured dispatcher choices were local; the result measures the cost of the remote path, not a placement win.
  • Real remote execution is proved separately by forced worker baselines and is not claimed here.
  • The paired mean-ratio interval upper endpoint exceeds the 1.10 median bound; the frozen decision rule uses observed quantiles with uncertainty disclosed rather than confidence-bound containment, so no robust population bound is claimed.
  • The result is a thirty-block small-integer experiment and is not a universal numerical or tail guarantee.
  • Retained payload and accounting are not an allocator, resident-set or enclosing transport trace.

In the prepared steady state the dispatcher path costs about 8 percent more than the best eligible local baseline at the median and about 14 percent at the 95th percentile, inside the predeclared 10 and 15 percent bounds. The paired mean-ratio interval is disclosed with an upper endpoint above the median bound. Nothing here is an acceleration claim: every measured choice was local, and the remote path exists only in the forced correctness arms.

Technical details and evidence
Open experiment ↗

What the first use of the prepared path costs

Technical details and evidence
Experiment identifier
rt-prepared-s5-first-use
Original title
Prepared calibrated first use, including preparation and calibration cost
Measurement boundary
The first-use boundary of the same prepared qualification, measured from before the prepared artifact and its attachment through calibration and the first measured call, including the external scalar oracle. This boundary failed the predeclared bounds by more than an order of magnitude.
Hardware
Two worker processes on one admitted CPU host, described in the receipt as correlated processes sharing the same hardware. The host is recorded as Ubuntu 24.04 with kernel 6.8 on EPYC-Genoa, with the Rust toolchain at 1.99.0 and one declared thread. The receipt states explicitly that this is not a heterogeneous resource set.
Software
A private immutable prepared plan retaining owned input identity plus live session, backend, memory, capability, affinity and freshness checks; one authoritative placement decision per two-GEMM region; a common optimised runner with a visible fallback trace.
Input
A small-integer N32 two-dependent-GEMM region executed as eight calls per block, with all 1,024 final output elements checked on every measured call.
Samples
1,200 measured calls, 450 warm-ups and 150 phase records across five arms of 30 paired blocks each. 240 calibrated measured rows with valid profiles and ages between 120 and 438 microseconds.
Statistical method
Nearest-rank block median and 95th percentile over 30 blocks, expressed as a ratio against the fixed named best eligible prepared local baseline, with a paired mean-ratio bootstrap of 10,000 resamples at a fixed seed.
Cold / warm state
The qualifying arm is the calibrated prepared steady state. Preparation and calibration costs are charged to a separate first-use boundary and are not folded into the steady figure.
Network
Loopback between the client and the two worker processes on the same host. Directional stream accounting excludes the enclosing transport overhead and private allocator peaks.
Serialization
Traffic records retain actual remote transfer and residency behaviour separately from dispatcher local behaviour. Per-call codec and transfer accounting is carried in the campaign receipts.
Correctness checks
Every one of the 1,200 measured calls reports 1,024 final output elements checked; the source aborts on any mismatch and the launch exit is zero. Independent re-summing of all eight caller times in each of the 150 blocks found zero mismatches, and no failed traffic records appear in the completed phases.
Original measurements — full precision
Original metricExact valueSamplesStatistic
median ratio of first use against the best eligible local baseline26.9381107492 ratio30observed block median ratio on the inclusive first-use boundary
95th percentile ratio of first use against the best eligible local baseline30.832884097 ratio30observed block 95th percentile ratio on the inclusive first-use boundary
measured calls1200 calls1501,200 measured calls across 150 phase records
phase records150 blocks150phase records carrying eight measured calls each

Experimental limitations

  • The boundary includes artifact preparation and calibration, which are charged once and amortised across later blocks; it is not comparable with the steady figure.
  • Preparation and calibration are nested inside the inclusive first-use total, so the phase quantiles cannot be summed to the total.
  • The external scalar oracle and warm-up are included in this boundary.
  • All measured dispatcher choices were local; the ratio measures the cost of preparing and calibrating the remote path, not a placement win.
  • The historical failure of the first-use boundary is preserved rather than replaced by the steady-state pass.

Preparing, calibrating and first-touching the remote path costs about 27 times the local baseline at the median. Amortising that cost is a property of the workload shape and the preparation design, not evidence that the path is faster; the steady-state pass and this first-use failure are the two ends of the same measurement and both are published.

Technical details and evidence
Open experiment ↗

Taking the first-use cost apart phase by phase

Technical details and evidence
Experiment identifier
rt-s6-first-use
Original title
Preservation first use including preparation and calibration, with the overhead boundaries itemised
Measurement boundary
The first-use boundary of the preservation run, measured from setup through preparation, calibration, warm-up and the first measured call with its external scalar oracle. The overhead phases are published as their own rows so the cost structure is visible rather than hidden inside a single ratio.
Hardware
Two worker processes on one admitted CPU host running Ubuntu 24.04 with kernel 6.8 on EPYC-Genoa, with the Rust toolchain at 1.99.0 and one declared thread. The receipt describes them as correlated processes sharing admitted hardware rather than a heterogeneous resource set.
Software
A separate one-attempt preservation dataset of 1,200 correct measured calls, 450 warm-ups and 150 phases, with 240 calibrated profiles fresh for both bindings and 240 real regions executed on each of two forced workers.
Input
A small-integer N32 two-dependent-GEMM region executed as eight measured calls per block, with every final output element checked on each call.
Samples
1,200 correct measured calls, 450 warm-ups and 150 phases; 240 calibrated measured rows with fresh profiles on both bindings.
Statistical method
Nearest-rank block median and 95th percentile over 30 blocks in microseconds, compared against the optimised local boundary, with a paired mean-ratio bootstrap of 10,000 resamples.
Cold / warm state
Each boundary is measured separately. Setup, preparation and calibration, warm-up, the first measured call with its external scalar oracle, the steady calls and cleanup are charged to distinct boundaries and the quantiles of nested phases cannot be added.
Network
Loopback between the client and the two worker processes on the same host. The harness explicitly excludes process-cold worker launch and any wide-area or operator transport from these totals.
Serialization
Per-phase counters remain fully enabled and charged, separating serial authenticated metadata, execute, fetch, release and statistics calls. Directional accounting excludes the enclosing transport overhead.
Correctness checks
All 1,200 measured calls are recorded correct. The independent gate decision recomputed every comparison and the seeded bootstrap and accepted the source, process, key, listener, size and archive receipts.
Original measurements — full precision
Original metricExact valueSamplesStatistic
inclusive first use, optimised local, median0.328 ms30nearest-rank block median, 328 microseconds
inclusive first use, optimised local, 95th percentile0.456 ms30nearest-rank block 95th percentile, 456 microseconds
inclusive first use, calibrated dispatcher, median8.638 ms30nearest-rank block median, 8,638 microseconds
inclusive first use, calibrated dispatcher, 95th percentile12.16 ms30nearest-rank block 95th percentile, 12,160 microseconds
first-use ratio against the optimised local boundary26.3354 ratio30median ratio on the inclusive first-use boundary
first-use ratio, 95th percentile26.6667 ratio3095th percentile ratio on the inclusive first-use boundary
setup phase, optimised local, median0.06 ms30nearest-rank block median, 60 microseconds
setup phase, calibrated dispatcher, median0.761 ms30nearest-rank block median, 761 microseconds
preparation and calibration phase, optimised local, median0.115 ms30nearest-rank block median, 115 microseconds
preparation and calibration phase, calibrated dispatcher, median7.665 ms30nearest-rank block median, 7,665 microseconds
cleanup phase, optimised local, median0.078 ms30nearest-rank block median, 78 microseconds
cleanup phase, calibrated dispatcher, median0.387 ms30nearest-rank block median, 387 microseconds
measured calls1200 calls1501,200 correct measured calls

CALCULATED FROM THE MEASUREMENTSinclusive first use, optimised local, median ÷ inclusive first use, calibrated dispatcher, median = 0.037971753×2533.5366% higher time for inclusive first use, calibrated dispatcher, median.

CALCULATED FROM THE MEASUREMENTSinclusive first use, optimised local, 95th percentile ÷ inclusive first use, calibrated dispatcher, 95th percentile = 0.037500000×2566.6667% higher time for inclusive first use, calibrated dispatcher, 95th percentile.

Experimental limitations

  • Quantiles of nested boundaries cannot be added; the setup boundary contains the artifact preparation.
  • This boundary is not a native or wide-area start-up measurement and must not be read as one.
  • The wide-area and cold observations in the same receipt are partial and are published separately.
  • All measured dispatcher choices were local; the ratio measures the cost of preparing and calibrating the remote path.
  • The first-use failure is preserved and is not replaced by the steady-state pass.

The remote path costs about 26 times the local boundary at the median on first use. The itemised phases show why: preparation and calibration alone cost roughly 7.7 ms at the median against a local preparation cost of 0.115 ms, while the steady call itself differs by only a fraction of a millisecond. The gap is start-up and calibration debt, not steady-state execution.

Technical details and evidence
Open experiment ↗

The whole experiment, including the wide-area observations

Technical details and evidence
Experiment identifier
rt-s6-full-experiment
Original title
Preservation complete experiment boundary and the partial wide-area observations from the same run
Measurement boundary
The complete experiment boundary of the preservation run, covering setup, preparation, calibration, warm-up, every measured call, the external oracles, instrumentation and cleanup. The same receipt contains partial wide-area and cold observations, which are published here as descriptive rows because they are the only connectivity figures in the dataset.
Hardware
Two worker processes on one admitted CPU host running Ubuntu 24.04 with kernel 6.8 on EPYC-Genoa, with the Rust toolchain at 1.99.0 and one declared thread. The receipt describes them as correlated processes sharing admitted hardware rather than a heterogeneous resource set.
Software
A separate one-attempt preservation dataset of 1,200 correct measured calls, 450 warm-ups and 150 phases, with 240 calibrated profiles fresh for both bindings and 240 real regions executed on each of two forced workers.
Input
A small-integer N32 two-dependent-GEMM region executed as eight measured calls per block, with every final output element checked on each call.
Samples
1,200 correct measured calls, 450 warm-ups and 150 phases; 240 calibrated measured rows with fresh profiles on both bindings.
Statistical method
Nearest-rank block median and 95th percentile over 30 blocks in microseconds, compared against the optimised local boundary, with a paired mean-ratio bootstrap of 10,000 resamples.
Cold / warm state
Each boundary is measured separately. Setup, preparation and calibration, warm-up, the first measured call with its external scalar oracle, the steady calls and cleanup are charged to distinct boundaries and the quantiles of nested phases cannot be added.
Network
Loopback between the client and the two worker processes on the same host. The harness explicitly excludes process-cold worker launch and any wide-area or operator transport from these totals.
Serialization
Per-phase counters remain fully enabled and charged, separating serial authenticated metadata, execute, fetch, release and statistics calls. Directional accounting excludes the enclosing transport overhead.
Correctness checks
All 1,200 measured calls are recorded correct. The independent gate decision recomputed every comparison and the seeded bootstrap and accepted the source, process, key, listener, size and archive receipts.
Original measurements — full precision
Original metricExact valueSamplesStatistic
inclusive full experiment, optimised local, median0.677 ms30nearest-rank block median, 677 microseconds
inclusive full experiment, optimised local, 95th percentile1.232 ms30nearest-rank block 95th percentile, 1,232 microseconds
inclusive full experiment, calibrated dispatcher, median9.43 ms30nearest-rank block median, 9,430 microseconds
inclusive full experiment, calibrated dispatcher, 95th percentile13.002 ms30nearest-rank block 95th percentile, 13,002 microseconds
full experiment ratio against the optimised local boundary13.9291 ratio30median ratio on the inclusive full-experiment boundary
full experiment ratio, 95th percentile10.5536 ratio3095th percentile ratio on the inclusive full-experiment boundary
wide-area local first parent receipt6897 ms1single recorded parent receipt
wide-area partial remote first parent receipt35905 ms1single recorded parent receipt, partial leg
wide-area local eighth call receipt7108 ms1single recorded parent receipt
wide-area remote eighth call receipt0 msnot recordednot recorded; the remote leg produced no receipt
wide-area descriptive API wait, local, median0.025756 ms8median of 8 correlated single-block samples
wide-area descriptive API wait, local, 95th percentile0.115579 ms895th percentile of 8 correlated single-block samples
wide-area descriptive API wait, partial remote, median8374.323146 ms7median of 7 correlated single-block samples
wide-area descriptive API wait, partial remote, 95th percentile11342.190034 ms795th percentile of 7 correlated single-block samples
failed cleanup-inclusive wide-area experiment duration137273 ms1duration of a failed run including cleanup, not a result latency
measured calls1200 calls1501,200 correct measured calls

CALCULATED FROM THE MEASUREMENTSinclusive full experiment, optimised local, median ÷ inclusive full experiment, calibrated dispatcher, median = 0.071792153×1292.9099% higher time for inclusive full experiment, calibrated dispatcher, median.

CALCULATED FROM THE MEASUREMENTSinclusive full experiment, optimised local, 95th percentile ÷ inclusive full experiment, calibrated dispatcher, 95th percentile = 0.094754653×955.35714% higher time for inclusive full experiment, calibrated dispatcher, 95th percentile.

CALCULATED FROM THE MEASUREMENTSwide-area local first parent receipt ÷ wide-area partial remote first parent receipt = 0.19209024×420.58866% higher time for wide-area partial remote first parent receipt.

Experimental limitations

  • Quantiles of nested boundaries cannot be added to the complete total.
  • The wide-area rows are correlated single-block observations with a partial remote leg; the remote eighth-call receipt is not recorded at all.
  • The 137.273 second figure is a failed cleanup-inclusive experiment duration and is not a result latency.
  • The wide-area payload campaign failure was not explained; surviving control connections did not establish a cause.
  • This boundary is not a native or wide-area start-up measurement.
  • The complete-phase failure is preserved alongside the steady-state pass.

The complete experiment costs about 13.9 times the optimised local boundary at the median and 10.6 times at the 95th percentile. The wide-area figures are far larger still and are partial: the local first parent receipt took 6.897 seconds against 35.905 seconds for the partial remote leg, and one remote boundary produced no result at all. Taken together the picture is consistent: the path is roughly as fast as local in steady state and much slower everywhere else.

Technical details and evidence
Open experiment ↗

A Windows client working against a Linux worker

Technical details and evidence
Experiment identifier
rt-s5-windows-linux-object-interop
Original title
Windows client exchanging objects with a Linux worker
Measurement boundary
A live functional exchange across operating systems: a Windows client uploads objects to a Linux worker, references them, executes one multiplication and releases them, with three negative paths. It is a functional interoperability milestone rather than a performance result.
Hardware
A Windows client and a Linux CPU worker on the same admitted host family. The worker process is bounded by an enforced process memory envelope; the accelerator is not involved.
Software
One exact source revision shared by the client and the worker, with the Windows binary and the worker binary identified by digest in the receipts.
Input
A 4 by 4 f32 object set: 16 final output elements checked, 3 references and 1 execution, with stale-epoch, released-handle and wrong-worker negatives.
Samples
One recorded live session with one execution and three references.
Statistical method
Counts of references, executions, checked elements and negatives, plus one recorded end-to-end duration and the directional stream counters. No distribution is published.
Cold / warm state
A single session against a ready worker. No cold-start cost is published.
Network
Loopback between the client and the worker, with 2,053 bytes outbound and 1,141 bytes inbound recorded for the session.
Serialization
The session moved 2,053 bytes outbound and 1,141 bytes inbound, and the worker released all resident bytes at the end.
Correctness checks
The result was recorded as a pass with all 16 final output elements checked. Stale-epoch handle use, released-handle use and a wrong-worker handshake were each rejected, and the worker released to zero resident bytes. The cleanup receipt reported a successful exit with the local key removed and the owned worker absent.
Original measurements — full precision
Original metricExact valueSamplesStatistic
final output elements checked16 cases1all 16 elements of the 4 by 4 f32 output
references issued3 calls1references against the uploaded object set
executions1 calls1one multiplication executed by the worker
negative paths exercised3 cases3stale epoch, released handle and wrong worker
outbound stream bytes2053 bytes1directional stream counter for the session
inbound stream bytes1141 bytes1directional stream counter for the session
end-to-end session duration12463.547 ms1single recorded duration under an explicit absolute deadline
resident bytes after release0 bytes1the worker released to zero resident bytes

Experimental limitations

  • One session and one execution; no distribution and no timing qualification.
  • The object shape is small, so nothing is established about large-message behaviour or throughput.
  • This is a client to worker direction only; the reverse direction is not represented.
  • The result is functional and carries no performance or acceleration meaning.
  • The reviewer note records that the original raw path remains separate malformed-record functional evidence with its own limitations.

The object protocol works across operating systems for the demonstrated path: correct results, three working negative paths, a clean release to zero resident bytes and a clean cleanup. What it does not establish is scale, throughput or the reverse direction, and it is not a performance result.

Technical details and evidence
Open experiment ↗
Technical details and evidence

Follow the next question