BenedictRESEARCH / NOTES
ReSynth
When fewer bytes are worth the work
Can I send a smaller description of an input and reconstruct exactly the bytes the calculation needs?
The network sees bytes, not intentions
Once I started moving numerical work between machines, the size of the input became part of the computation’s cost. A large repeated buffer and a large irregular buffer might contain the same number of values, yet have very different opportunities for compression.
I explored how to choose a representation without changing the data’s meaning. The requirement was lossless: reconstruction must return the exact bytes, not an approximation that merely looks close.
Choosing between sending and rebuilding
I used existing codecs, including LZ4 and Zstandard, and put bounds around the messages and the work needed to decode them. ReSynth brings those pieces together with dictionary identification, sampling and a small cache of recent estimates.
I also compared keeping an unchanged input at the worker with sending it again. These are different ways of avoiding bytes: compression describes the data more compactly; reuse avoids resending something already present. Neither removes the need to identify the correct version.
Structured inputs were the encouraging case
In one structured-data transfer test, the raw route carried about 1.06 MB. The LZ4 route carried about 45 kB. That is a substantial reduction in traffic without changing the decoded result.
But the total time did not fall in proportion to the byte count. Computing, encoding, decoding and exchanging the message still cost time. The mixed-input experiment made the point more sharply: an adaptive choice sent more bytes than its raw baseline. Choosing a representation was itself a decision that could go wrong.
A local connection changed the trade-off
I ran separate codec tests over a local connection, where an uncompressed message was already cheap. Some compressed paths took longer despite sending much less data. The first profiling call could also cost much more than later calls using an existing estimate.
That is why I do not treat a compression ratio as a performance result. The same representation can be useful across a constrained link and wasteful on a cheap one.
Exact reconstruction needs limits
A decoder must not trust the size claimed by an incoming message. I added checks for oversized data, the wrong dictionary, truncated input and extra bytes after a frame. Otherwise a feature intended to save bandwidth can become a way to waste memory or accept the wrong result.
Sampling is another limit. It can suggest that a buffer is worth compressing, but a sample cannot guarantee how the whole buffer will behave. The system needs a way to fall back to sending the original bytes.
The useful result is a cost model, not a magic codec
ReSynth is not a new universal compression algorithm. It is an investigation into selecting and accounting for existing techniques in an execution system.
What I want next is a decision that consistently includes the whole cost: bytes on the link, encoding, reconstruction, memory and the possibility that doing nothing special is cheaper. Sometimes the right representation is simply the original data.
Technical details and evidence
The NCP1 encoding layer has a 64 MiB frame limit and 64 bytes of framing overhead. Dictionary fingerprints, bounded dictionary storage, stratified sampling and an expiring profile cache are implementation details of that layer.
The structured transfer record reports 1,058,842 raw wire bytes versus 45,312 with LZ4. The mixed-input adaptive route reports 664,124 bytes versus 529,796 raw. Framed traffic is not a measurement of all physical network overhead.
The separate XOR-plus-zlib research examples, isolated codec component checks and runtime transfer tests are not interchangeable benchmarks.
A workload that mixes several kinds of operand
Technical details and evidence
- Experiment identifier
- rt-transfer-mixed
- Original title
- Transfer byte and latency accounting, mixed operand regime
- Measurement boundary
- A matched four-call workload combining operand regimes within one workload, measured across the transport and representation modes. The adaptive mode records negative avoided bytes in this regime. Every mode was run against the same logical input demand and the same matched raw workload, so the byte columns are directly comparable across modes and the avoided-byte column is a difference against the raw mode rather than an absolute saving.
- Hardware
- A Linux CPU host running the transport harness with a system CBLAS provider. The host CPU model, core count and memory size are not recorded in this report.
- Software
- The runtime transport with its representation modes, a pinned reference BLAS provider, and an authenticated local tunnel. Compiler and operating system versions are not recorded in this report.
- Input
- The mixed operand regime of the matched four-call workload, with a logical input demand of 786432 bytes in the single-input modes and 786432 bytes for the resident mode of the chain regime.
- Samples
- One matched four-call workload per mode, with the raw mode as the byte-difference reference. No repeated trials or distributions are published for this campaign.
- Statistical method
- Measured application-stream byte totals and total user wait for the matched four-call workload. The avoided-byte figure is the difference against the matched raw workload and can be negative.
- Cold / warm state
- The measured totals include cold setup, calibration and cleanup. The adaptive mode additionally charges its own calibration, which is identified separately in the raw records.
- Network
- The measured byte columns are application-stream bytes only. IP, TCP and tunnel overhead and retransmissions are not recorded and are explicitly unknown.
- Serialization
- Logical, packed, encoded and framed byte counts are kept separate in the raw records; the published column is the framed application-stream total.
- Correctness checks
- Every mode in this class is recorded as passing the correctness check.
| Original metric | Exact value | Samples | Statistic |
|---|---|---|---|
| local execution, no transport: total wire bytes | 0 bytes | 4 | measured application-stream bytes across a matched four-call workload |
| local execution, no transport: total user wait | 8.638 ms | 4 | total user wait across a matched four-call workload |
| local execution, no transport: bytes avoided against the matched raw run | 529796 bytes | 4 | measured stream-byte difference against the matched raw four-call workload |
| raw transport: total wire bytes | 529796 bytes | 4 | measured application-stream bytes across a matched four-call workload |
| raw transport: total user wait | 18.885 ms | 4 | total user wait across a matched four-call workload |
| raw transport: bytes avoided against the matched raw run | 0 bytes | 4 | measured stream-byte difference against the matched raw four-call workload |
| resident-operand transport: total wire bytes | 398002 bytes | 4 | measured application-stream bytes across a matched four-call workload |
| resident-operand transport: total user wait | 15.838 ms | 4 | total user wait across a matched four-call workload |
| resident-operand transport: bytes avoided against the matched raw run | 131794 bytes | 4 | measured stream-byte difference against the matched raw four-call workload |
| lz4-compressed transport: total wire bytes | 197973 bytes | 4 | measured application-stream bytes across a matched four-call workload |
| lz4-compressed transport: total user wait | 18.005 ms | 4 | total user wait across a matched four-call workload |
| lz4-compressed transport: bytes avoided against the matched raw run | 331823 bytes | 4 | measured stream-byte difference against the matched raw four-call workload |
| byte-delta transport: total wire bytes | 398654 bytes | 4 | measured application-stream bytes across a matched four-call workload |
| byte-delta transport: total user wait | 16.65 ms | 4 | total user wait across a matched four-call workload |
| byte-delta transport: bytes avoided against the matched raw run | 131142 bytes | 4 | measured stream-byte difference against the matched raw four-call workload |
| lz4-compressed byte-delta transport: total wire bytes | 231503 bytes | 4 | measured application-stream bytes across a matched four-call workload |
| lz4-compressed byte-delta transport: total user wait | 18.587 ms | 4 | total user wait across a matched four-call workload |
| lz4-compressed byte-delta transport: bytes avoided against the matched raw run | 298293 bytes | 4 | measured stream-byte difference against the matched raw four-call workload |
| adaptive representation selection: total wire bytes | 664124 bytes | 4 | measured application-stream bytes across a matched four-call workload |
| adaptive representation selection: total user wait | 20.83 ms | 4 | total user wait across a matched four-call workload |
| adaptive representation selection: bytes avoided against the matched raw run | -134328 bytes | 4 | measured stream-byte difference against the matched raw four-call workload |
| adaptive selection, which chose the local path: total wire bytes | 0 bytes | 4 | measured application-stream bytes across a matched four-call workload |
| adaptive selection, which chose the local path: total user wait | 9.66 ms | 4 | total user wait across a matched four-call workload |
| adaptive selection, which chose the local path: bytes avoided against the matched raw run | 529796 bytes | 4 | measured stream-byte difference against the matched raw four-call workload |
CALCULATED FROM THE MEASUREMENTSlocal execution, no transport: total user wait ÷ raw transport: total user wait = 0.45740005×118.62700% higher time for raw transport: total user wait.
CALCULATED FROM THE MEASUREMENTSraw transport: total wire bytes ÷ lz4-compressed transport: total wire bytes = 2.6761023×62.632221% lower value for lz4-compressed transport: total wire bytes.
Experimental limitations
- One matched workload per mode with no repeated trials and no distribution.
- Physical packet, tunnel and retransmission overhead are not recorded, so the wire column is an application-stream measure.
- The avoided-byte figure can be negative and is not a saving in every case.
- The timings overlap and must not be summed, and the encode and decode counters cover the call phase only.
- This is a Linux CPU transport measurement and is not a Windows or accelerator measurement.
The published rows let the representation modes be read against each other on the same logical input. The mode that wins depends entirely on the operand regime: a general-purpose codec helps most where the operands are structured and does nothing where they are not, and the adaptive mode does not dominate every regime. No single figure from this table should be quoted without its class.
Technical details and evidence
How the codec stack behaves on friendly and hostile payloads
Technical details and evidence
- Experiment identifier
- rt-cn-codec-link-matrix
- Original title
- Codec matrix and authenticated tunnel timing, structured and random payloads, cold and warm profile
- Measurement boundary
- Two related measurements on one node. The first is a codec matrix over a loopback link, run once for a structured payload and once for a random payload across raw, two zstd levels and lz4. The second is the same stack over an authenticated local tunnel, where the profile cost of the first transfer is compared against later transfers to show the effect of the bounded profile cache.
- Hardware
- A Linux CPU node with 8 dedicated EPYC-Milan vCPUs, 30 GiB of memory and 4 GiB of swap, running Ubuntu 26.04.1 with no accelerator present.
- Software
- The runtime transport with the codec component and its bounded stratified profile cache, an authenticated local tunnel, and a system CBLAS provider. The toolchain version for this campaign is 1.99.1.
- Input
- A fixed pair of payloads: one structured and one random, of the same logical size, transferred through the codec envelope in both directions.
- Samples
- One measured transfer per codec per payload condition in the codec matrix. Three recorded transfers per payload kind and codec in the tunnel measurement, where the first is cold and the third is warm.
- Statistical method
- Single recorded values per codec and condition in the codec matrix. Median-equivalent values in the tunnel log are three recorded transfers, with the first and third published here.
- Cold / warm state
- The profile cache is bounded to a small number of entries with a five minute time to live. On the first recorded transfer the worker-side profile cost is tens of times larger than on the third, which is the cache becoming warm. The codec matrix is a single transfer per condition and is not repeated.
- Network
- Loopback for the codec matrix and an authenticated local tunnel for the second measurement. No external link, encryption or retransmission overhead is involved or recorded.
- Serialization
- The codec envelope carries a fixed header and a length prefix, and upload and download byte counts are recorded separately for every case.
- Correctness checks
- Every case in the codec matrix is recorded as correct, and the tunnel measurement records a per-call worker decode, encode and profile cost alongside the elapsed time.
| Original metric | Exact value | Samples | Statistic |
|---|---|---|---|
| structured payload, raw: upload bytes | 49400 bytes | 1 | single recorded loopback transfer |
| structured payload, raw: download bytes | 16485 bytes | 1 | single recorded loopback transfer |
| structured payload, raw: elapsed | 564.776 ms | 1 | single recorded loopback transfer |
| structured payload, zstd level 1: upload bytes | 311 bytes | 1 | single recorded loopback transfer |
| structured payload, zstd level 1: download bytes | 771 bytes | 1 | single recorded loopback transfer |
| structured payload, zstd level 1: elapsed | 90.311 ms | 1 | single recorded loopback transfer |
| structured payload, zstd level 3: upload bytes | 292 bytes | 1 | single recorded loopback transfer |
| structured payload, zstd level 3: download bytes | 835 bytes | 1 | single recorded loopback transfer |
| structured payload, zstd level 3: elapsed | 101.093 ms | 1 | single recorded loopback transfer |
| structured payload, lz4: upload bytes | 487 bytes | 1 | single recorded loopback transfer |
| structured payload, lz4: download bytes | 1408 bytes | 1 | single recorded loopback transfer |
| structured payload, lz4: elapsed | 106.112 ms | 1 | single recorded loopback transfer |
| random payload, raw: upload bytes | 49400 bytes | 1 | single recorded loopback transfer |
| random payload, raw: download bytes | 16485 bytes | 1 | single recorded loopback transfer |
| random payload, raw: elapsed | 564.055 ms | 1 | single recorded loopback transfer |
| random payload, zstd level 1: upload bytes | 44475 bytes | 1 | single recorded loopback transfer |
| random payload, zstd level 1: download bytes | 15405 bytes | 1 | single recorded loopback transfer |
| random payload, zstd level 1: elapsed | 518.634 ms | 1 | single recorded loopback transfer |
| random payload, zstd level 3: upload bytes | 44451 bytes | 1 | single recorded loopback transfer |
| random payload, zstd level 3: download bytes | 15403 bytes | 1 | single recorded loopback transfer |
| random payload, zstd level 3: elapsed | 529.126 ms | 1 | single recorded loopback transfer |
| random payload, lz4: upload bytes | 49560 bytes | 1 | single recorded loopback transfer |
| random payload, lz4: download bytes | 16646 bytes | 1 | single recorded loopback transfer |
| random payload, lz4: elapsed | 607.937 ms | 1 | single recorded loopback transfer |
| repeated payload, local path: elapsed, first recorded transfer | 0.054 ms | 1 | first recorded transfer of the group |
| repeated payload, local path: upload bytes | 0 bytes | 1 | recorded application-stream bytes |
| repeated payload, local path: download bytes | 0 bytes | 1 | recorded application-stream bytes |
| repeated payload, raw: elapsed, first recorded transfer | 6.533 ms | 1 | first recorded transfer of the group |
| repeated payload, raw: elapsed, last recorded transfer | 0.367 ms | 3 | last recorded transfer of the group (sample 2), after the profile cache is warm |
| repeated payload, raw: upload bytes | 49397 bytes | 1 | recorded application-stream bytes |
| repeated payload, raw: download bytes | 16482 bytes | 1 | recorded application-stream bytes |
| repeated payload, zstd level 1: elapsed, first recorded transfer | 51.921 ms | 1 | first recorded transfer of the group |
| repeated payload, zstd level 1: elapsed, last recorded transfer | 40.641 ms | 3 | last recorded transfer of the group (sample 2), after the profile cache is warm |
| repeated payload, zstd level 1: upload bytes | 302 bytes | 1 | recorded application-stream bytes |
| repeated payload, zstd level 1: download bytes | 626 bytes | 1 | recorded application-stream bytes |
| repeated payload, zstd level 1: worker profile cost, first recorded transfer | 1.396 ms | 1 | worker-side profile cost on the first recorded transfer |
| repeated payload, zstd level 1: worker profile cost, last recorded transfer | 0.022 ms | 1 | worker-side profile cost once the profile cache is warm |
| repeated payload, lz4: elapsed, first recorded transfer | 43.188 ms | 1 | first recorded transfer of the group |
| repeated payload, lz4: elapsed, last recorded transfer | 40.864 ms | 3 | last recorded transfer of the group (sample 2), after the profile cache is warm |
| repeated payload, lz4: upload bytes | 487 bytes | 1 | recorded application-stream bytes |
| repeated payload, lz4: download bytes | 635 bytes | 1 | recorded application-stream bytes |
| repeated payload, lz4: worker profile cost, first recorded transfer | 0.027 ms | 1 | worker-side profile cost on the first recorded transfer |
| repeated payload, lz4: worker profile cost, last recorded transfer | 0.023 ms | 1 | worker-side profile cost once the profile cache is warm |
| random payload, local path: elapsed, first recorded transfer | 0.016 ms | 1 | first recorded transfer of the group |
| random payload, local path: upload bytes | 0 bytes | 1 | recorded application-stream bytes |
| random payload, local path: download bytes | 0 bytes | 1 | recorded application-stream bytes |
| random payload, raw: elapsed, first recorded transfer | 0.375 ms | 1 | first recorded transfer of the group |
| random payload, raw: elapsed, last recorded transfer | 0.346 ms | 3 | last recorded transfer of the group (sample 2), after the profile cache is warm |
| random payload, raw: upload bytes | 49397 bytes | 1 | recorded application-stream bytes |
| random payload, raw: download bytes | 16482 bytes | 1 | recorded application-stream bytes |
| random payload, zstd level 1: elapsed, first recorded transfer | 41.442 ms | 1 | first recorded transfer of the group |
| random payload, zstd level 1: elapsed, last recorded transfer | 41.255 ms | 3 | last recorded transfer of the group (sample 2), after the profile cache is warm |
| random payload, zstd level 1: upload bytes | 29784 bytes | 1 | recorded application-stream bytes |
| random payload, zstd level 1: download bytes | 15583 bytes | 1 | recorded application-stream bytes |
| random payload, zstd level 1: worker profile cost, first recorded transfer | 0.726 ms | 1 | worker-side profile cost on the first recorded transfer |
| random payload, zstd level 1: worker profile cost, last recorded transfer | 0.024 ms | 1 | worker-side profile cost once the profile cache is warm |
| random payload, lz4: elapsed, first recorded transfer | 0.335 ms | 1 | first recorded transfer of the group |
| random payload, lz4: elapsed, last recorded transfer | 0.338 ms | 3 | last recorded transfer of the group (sample 2), after the profile cache is warm |
| random payload, lz4: upload bytes | 33273 bytes | 1 | recorded application-stream bytes |
| random payload, lz4: download bytes | 16775 bytes | 1 | recorded application-stream bytes |
| random payload, lz4: worker profile cost, first recorded transfer | 0.015 ms | 1 | worker-side profile cost on the first recorded transfer |
| random payload, lz4: worker profile cost, last recorded transfer | 0.016 ms | 1 | worker-side profile cost once the profile cache is warm |
CALCULATED FROM THE MEASUREMENTSrandom payload, zstd level 3: upload bytes ÷ random payload, raw: upload bytes = 0.89981781×11.133608% higher value for random payload, raw: upload bytes.
CALCULATED FROM THE MEASUREMENTSstructured payload, zstd level 3: upload bytes ÷ structured payload, raw: upload bytes = 0.0059109312×16817.808% higher value for structured payload, raw: upload bytes.
CALCULATED FROM THE MEASUREMENTSrandom payload, zstd level 3: elapsed ÷ random payload, raw: elapsed = 0.93807519×6.6012632% higher time for random payload, raw: elapsed.
Experimental limitations
- The codec matrix is a single transfer per condition, so the byte figures are reproducible counts but the timings are single observations.
- The tunnel measurement publishes three recorded transfers per group and no distribution.
- These are node loopback baselines and are explicitly not evidence for the frozen useful-workload gate.
- A separate chunked-transfer benchmark in the same campaign is blocked by an admission constraint on that operating system and produced no measurement.
- The structural payload compresses to a few hundred bytes while the random payload barely compresses, so the two conditions are not interchangeable inputs.
The codec stack behaves as a compression stack should and honestly: on a structured payload it removes almost all of the bytes, while on a random payload zstd level 3 saves about a tenth of the upload and level 1 slightly less, and lz4 saves nothing at all. The tunnel measurement then shows the cost side of profiling: the first transfer pays a profile cost tens of times larger than the third, which is the bounded profile cache warming, and that warm-up cost is charged to the first call rather than hidden.
Six ways to compress one megabyte of smooth data
Technical details and evidence
- Experiment identifier
- aspr-codec-smooth-1mib
- Original title
- Lossless codec screening on a 1 MiB smooth buffer
- Measurement boundary
- Measured CPU encode and decode of a 1 MiB smooth byte buffer through six lossless profiles. Coded size is the published result; encode and decode medians are recorded alongside it in the source.
- Hardware
- Windows 11 host; CPU model not recorded.
- Software
- NumPy 2.2.6; exact Python and BLAS build not recorded.
- Input
- One 1048576-byte smooth buffer.
- Samples
- 10 encode and 10 decode samples per profile; 6 profiles.
- Statistical method
- exact coded bytes and median encode time of 10 samples
- Cold / warm state
- Warm in-process buffers; repeated encodes of the same buffer.
- Network
- None; byte buffers only, no transport was exercised.
- Serialization
- Coded byte buffers; no request or response envelope is modeled.
- Correctness checks
- Round-trip correctness is not recorded as a pass or fail field in this file.
| Original metric | Exact value | Samples | Statistic |
|---|---|---|---|
| Raw bytes, no codec | 1048576 bytes | 10 | exact coded size; encode median 0.00555 ms in the source |
| zlib level 1 | 148083 bytes | 10 | exact coded size; encode median 12.79725 ms in the source |
| zlib level 9 | 148474 bytes | 10 | exact coded size; encode median 791.3118 ms in the source |
| lzma preset 1 | 59940 bytes | 10 | exact coded size; encode median 48.127700000000004 ms in the source |
| lzma preset 6 | 23940 bytes | 10 | exact coded size; encode median 496.76265 ms in the source |
| XOR-plane transform plus zlib level 9 | 2397 bytes | 10 | exact coded size; encode median 12.7918 ms in the source |
Experimental limitations
- Byte buffers only; no transport and no authenticated envelope.
- 10 samples per profile on one host.
- Coded size is exact; encode and decode medians are session-local.
- Smooth synthetic data is not a CAD payload.
The raw buffer is 1048576 bytes. zlib level 1 gives 148083 bytes, zlib level 9 148474, lzma preset 1 59940, lzma preset 6 23940 and the XOR-plane transform with zlib level 9 2397. The transform pair is the smallest encoding on this corpus, and the gap between zlib levels shows that a higher level can enlarge the output on smooth data.
Technical details and evidence
The same six codecs on data that will not compress
Technical details and evidence
- Experiment identifier
- aspr-codec-random-1mib
- Original title
- Lossless codec screening on a 1 MiB random-byte buffer
- Measurement boundary
- Measured CPU encode and decode of a 1 MiB random-byte buffer through six lossless profiles. This is the negative case for compression: a codec that helps smooth data may expand or barely help incompressible data.
- Hardware
- Windows 11 host; CPU model not recorded.
- Software
- NumPy 2.2.6; exact Python and BLAS build not recorded.
- Input
- One 1048576-byte random_bytes buffer.
- Samples
- 10 encode and 10 decode samples per profile; 6 profiles.
- Statistical method
- exact coded bytes and median encode time of 10 samples
- Cold / warm state
- Warm in-process buffers; repeated encodes of the same buffer.
- Network
- None; byte buffers only, no transport was exercised.
- Serialization
- Coded byte buffers; no request or response envelope is modeled.
- Correctness checks
- Round-trip correctness is not recorded as a pass or fail field in this file.
| Original metric | Exact value | Samples | Statistic |
|---|---|---|---|
| Raw bytes, no codec | 1048576 bytes | 10 | exact coded size; encode median 0.0948 ms in the source |
| zlib level 1 | 1048902 bytes | 10 | exact coded size; encode median 40.21175 ms in the source |
| zlib level 9 | 1048902 bytes | 10 | exact coded size; encode median 41.77355 ms in the source |
| lzma preset 1 | 674572 bytes | 10 | exact coded size; encode median 255.056 ms in the source |
| lzma preset 6 | 848528 bytes | 10 | exact coded size; encode median 320.27750000000003 ms in the source |
| XOR-plane transform plus zlib level 9 | 558573 bytes | 10 | exact coded size; encode median 29.2665 ms in the source |
Experimental limitations
- Byte buffers only; no transport and no authenticated envelope.
- 10 samples per profile on one host.
- zlib expands this buffer at both levels tested.
- Random synthetic bytes are not a CAD payload.
The raw buffer is 1048576 bytes. zlib level 1 and level 9 both expand it to 1048902 bytes; lzma preset 1 gives 674572 bytes, lzma preset 6 848528 and the XOR-plane transform with zlib level 9 558573. Compression gain on random bytes is small and codec-dependent, so a single corpus cannot justify a codec choice.