igneum/proto-metal/TESTS.md

proto-metal hardening tests

Date: 3 October 2026. Machine: Apple M5 Max, 40 GPU cores, 64 GB unified memory, macOS Darwin 25.6.0, Swift 5.8.1 from Command Line Tools, no Xcode, Metal shaders compiled at runtime. Build: swiftc -O -o igneum-bench main.swift -framework Metal (4 s).

Every number below was produced on this machine on this date by the commands shown.

Later on 3 October 2026 the default dataset became the memory-hard construction of MEMHARD.md. The tables in this file are from the original closed-form dataset and reproduce with --closed-form added to each command. Every test here was re-run on the new dataset with the same commands and passed; those results are in MEMHARD.md section 2.5. The shortcut of section 7 is answered there (section 2.2): the inline kernel is now 4.8x slower than the honest one.

On 4 October 2026 the generator became version 2 (exactly 16 loads per program, fresh-source loads, the acceptance rule of spec 01 section 1.4.6; igneum-pow/src/{generator,accept}.rs, ported into main.swift as generateProgramV2, candidateProgram, acceptProgram). Sections 1 to 7 below describe the version 1 generator and its vectors, which are retired; the loads-per-hash ranges they quote no longer occur. Section 9 holds the version 2 run. The --load-weight and --wide-frac levers select the retired version 1 generator and exist only so MEMHARD.md section 2.4 reproduces.

The tests live in main.swift next to the bench and share its generator, MSL emitter, dataset build and CPU interpreter without modification. The CPU interpreter gained an optional trace hook (cpuWarpTraced) that the bench does not use; cpuWarp calls it with nil.

What the GPU-vs-CPU comparison means: the GPU runs the generated Metal kernel, the CPU runs a plain Swift interpreter over the same instruction list, and every 64-bit output is compared bit for bit. The interpreter never reads the dataset buffer; it recomputes each element from the closed form. So agreement also covers the dataset fill kernel and the address masking, not only the arithmetic.

1. Fuzz: random programs, random warps, three dataset sizes

Command (required run):

./igneum-bench --fuzz 200

For each of N seeds: generate the program, emit MSL, static mask check, compile on the GPU, draw a dataset size from {64 MiB, 256 MiB, 1 GiB}, draw 4 base nonces uniformly from the full 32-bit range (so warps straddle 2^31 and wrap past 2^32), run the GPU, run the CPU interpreter, compare all 128 outputs. The master seed fixes the whole sequence, so any failure is reproducible with --fuzz-seed. The run also asserts the generator's contract that the emitter relies on: rotl immediate in 1..31, shuffle mask in {1,2,4,8,16}, source register never the destination.

Dataset	Programs	Pass	Fail
2^24 words (64 MiB)	63	63	0
2^26 words (256 MiB)	64	64	0
2^28 words (1 GiB)	73	73	0
all	200	200	0

200 programs: pass 200, mismatch 0, compile failures 0, static mask failures 0, contract failures 0. 800 warps compared (25,600 hashes). Loads per hash ranged 64 to 208. Op totals across the 200 programs: load 3204, add 1472, xor 1276, mul 1087, mad 1062, shfl 1038, rotl 892, mulhi 762, sub 742, rotr 739, or 526. Compile (library + pipeline) min 0.0 ms, avg 18.3 ms, max 46.4 ms; the zero is the system shader cache answering for the first 20 seeds, which an earlier smoke run with the same master seed had already compiled. GPU dispatch 156 ms total, CPU interpreter 16 ms total, wall 3.9 s.

Large run, separate master seed so every compile is cold:

./igneum-bench --fuzz 10000 --fuzz-seed igneum-fuzz-large-2026-10-03

Dataset	Programs	Pass	Fail
2^24 words (64 MiB)	3346	3346	0
2^26 words (256 MiB)	3334	3334	0
2^28 words (1 GiB)	3320	3320	0
all	10000	10000	0

10,000 programs: pass 10,000, mismatch 0, compile failures 0, static mask failures 0, contract failures 0. 40,000 warps compared (1,280,000 hashes). Loads per hash ranged 40 to 232. Op totals: load 160,123, add 76,698, xor 64,138, mad 51,577, shfl 51,298, mul 50,919, rotl 44,945, rotr 38,333, mulhi 38,284, sub 38,234, or 25,451. Compile min 13.6 ms, avg 18.5 ms, max 48.5 ms (all cold). GPU dispatch 8.2 s total, CPU interpreter 0.8 s total, wall 196 s. FUZZ: PASS.

Combined with the 200-program run: 10,200 programs, 40,800 warps, 1,305,600 hashes, zero mismatches. The op totals show every one of the 11 instruction families was exercised tens of thousands of times.

2. Edge cases

Command:

./igneum-bench --edge

Hand-built programs, dataset 1 GiB (MASK 0x0fffffff), 4 warps at base nonces 0x00000000, 0x00100000, 0x7ffffff0 and 0xffffffe0, 128 lanes per case. Operand values are forced with r = r - r (zero) followed by a constant add, then proven: the traced CPU interpreter checks the named register value just before the instruction in question, on lane 0 of all 4 warps in all 8 iterations. "Held" means 32 of 32 checks passed.

Case	Instrs	Loads/hash	Preconditions	GPU vs CPU	Result
rotl immediate by 1 and by 31	6	0	none needed	128/128 lanes	PASS
rotr by register == 0	3	0	held	128/128 lanes	PASS
rotr by register == 32 (32 mod 32 = 0)	5	0	held	128/128 lanes	PASS
rotr by register == 0xFFFFFFE0 (-32, 0 mod 32)	5	0	held	128/128 lanes	PASS
rotr by register == 31, 63 and 1	10	0	held	128/128 lanes	PASS
mulhi 0xFFFFFFFF x 0xFFFFFFFF (result 0xFFFFFFFE), 0x80000000 x 2 (result 1), x 0 (result 0)	15	0	held, results checked	128/128 lanes	PASS
shfl_xor every mask 1..16 in sequence	16	0	none needed	128/128 lanes	PASS
load at index 0 via register 0, and via register MASK+1	6	16	held	128/128 lanes	PASS
load at index MASK via register MASK, and via register 0xFFFFFFFF	8	16	held	128/128 lanes	PASS
add wraparound 0xFFFFFFFF + 1 (result 0)	5	0	held, result checked	128/128 lanes	PASS
sub wraparound 0 - 1 (result 0xFFFFFFFF)	6	0	held, result checked	128/128 lanes	PASS
mul 0xFFFFFFFF x 0xFFFFFFFF (low result 1), mad same + 5 (result 6)	15	0	held, results checked	128/128 lanes	PASS
generated program with every load replaced by xor (zero loads)	64	0	none needed	128/128 lanes	PASS
64 loads and nothing else	64	512	none needed	128/128 lanes	PASS
rotl immediate by 0 (outside the generator's 1..31 contract)	2	0	none needed	128/128 lanes	info only: agrees

EDGE: PASS, 14 of 14 counted cases. The last row is informational: the generator never emits a rotate by 0 (the fuzz run asserts this on every instruction), and the MSL rotl_imm would shift by 32 for it. On this GPU and compiler the result happened to equal the CPU's. Nothing may rely on that; the contract stays 1..31. The shuffle case covers masks 3, 5, 6, 7, 9 to 15 that the generator does not emit; simd_shuffle_xor and the interpreter's lane ^ mask agreed for all 16.

3. Output statistics

Command:

./igneum-bench --stats

Three seeds: igneum-genesis, igneum-genesis/stats1, igneum-genesis/stats2. For each, 2^20 consecutive nonces from 0 hashed on the GPU at 1 GiB (warps 0 and 32767 spot-checked against the CPU, both matched), then on the CPU side: (a) ones count per output bit, (b) avalanche from single-bit nonce flips, run on the GPU as pairs of one-warp dispatches, (c) chi-square over 65,536 buckets for each 16-bit window of the output, (d) duplicate count after sorting.

Seed	Loads/hash	Bit freq min..max	Max bit deviation (sigma)	Avalanche 1k mean / std	Avalanche 16k mean / std	Per-output-bit flip prob (16k)	Worst chi2 z of 4 windows	Dups
igneum-genesis	104	0.4990..0.5009	2.07 (bit 38)	32.07 / 4.09	32.035 / 3.99	0.492..0.507	0.42	0
igneum-genesis/stats1	128	0.4986..0.5011	2.90 (bit 3)	32.14 / 3.94	31.987 / 3.98	0.490..0.508	2.26	0
igneum-genesis/stats2	176	0.4989..0.5010	2.29 (bit 9)	31.90 / 3.97	32.035 / 4.01	0.492..0.508	1.65	0

Chi-square detail (df 65535, expected 65535, sigma 362):

Seed	bits 0..15	bits 16..31	bits 32..47	bits 48..63
igneum-genesis	65681 (z 0.40)	65686 (z 0.42)	65513 (z -0.06)	65478 (z -0.16)
igneum-genesis/stats1	65163 (z -1.03)	65768 (z 0.64)	65820 (z 0.79)	66353 (z 2.26)
igneum-genesis/stats2	66132 (z 1.65)	65447 (z -0.24)	65942 (z 1.12)	65960 (z 1.17)

Reading the numbers. (a) With 2^20 samples the sigma on a bit count is 512; the largest deviation over 192 bit positions (3 seeds x 64) was 2.90 sigma, which is what 192 draws from a fair coin produce. (b) An ideal function changes 32 of 64 output bits on average with standard deviation 4. The 1,000-trial means are within 0.14 of 32 (standard error 0.13); the 16,000-trial means are within 0.035 of 32 (standard error 0.03), the standard deviations are 3.98 to 4.01, and every one of the 64 output bits flipped with probability 0.490 to 0.508 (sigma 0.004). The per-input-bit means over 500 flips each ranged 31.5 to 32.5 (sigma 0.18). An earlier 1,000-trial sample for igneum-genesis, drawn with a different RNG salt before the 16,000-trial pass was added, gave mean 31.72; the 16,000-trial figure of 32.035 shows that was sampling noise. (c) All 12 chi-square values are within 2.3 sigma of their expectation. (d) No duplicate among 2^20 outputs; the birthday expectation at 64 bits is 3e-8.

Verdict: the output looks uniform on every measure tried, for all three seeds. STATS: PASS.

This is a sanity check for obvious structural bias. It is not a proof of cryptographic strength, it says nothing about adversarially chosen inputs, and 3 seeds is not a statement about the population of programs.

4. Determinism

Command:

./igneum-bench --determinism

Seed igneum-genesis, 2^20 nonces from 0, 1 GiB dataset. The output buffer is filled with a sentinel before every run so an unwritten lane would show.

Check	Result
Two independent generations of the program give identical MSL text (4,770 bytes)	yes
5 GPU runs of the same pipeline, fingerprint FNV-1a over 8 MiB	933787e8cfefccb7 all 5 runs, 0 outputs differ, 0 unwritten lanes
Second compile of identical source (0.1 ms, served by the system shader cache)	fingerprint identical
Third compile with a comment tag appended so the cache misses (30.0 ms, a real recompile)	fingerprint identical
CPU interpreter on 8 warps of the reference run (first, last, 6 random)	all match
Dataset filled twice, both blitted to shared memory and fingerprinted	0b1a77899ee60493 both times
4,096 sampled dataset words incl. indices 0, 1, MASK-1, MASK vs CPU datasetElem	all match

GPU time per 2^20 batch: 33.0 ms first, 23.2 to 23.8 ms after. DETERMINISM: PASS.

5. Memory safety of dataset indexing

Command:

./igneum-bench --memcheck

Static: the generated MSL for igneum-genesis at 2^20, 2^24 and 2^28 words has 13 dataset[ accesses, all 13 of the exact form dataset[rN & MASK], and the identifier dataset appears 14 times (13 accesses plus the kernel parameter). The CUDA twin from --export-pack has 14 ds[ accesses: 13 of the form ds[rN & mask] and the fill kernel's one write, guarded by if (i < n). The same static check ran on all fuzz programs (section 1) with zero failures.

Dynamic, 4 MiB dataset (MASK 0x000fffff): four full batches of 2^20 nonces from bases 0xfff00000 (last nonce 0xffffffff), 0xffffffe0 (wraps to 0 inside the batch), 0x80000000 and 0 all completed; 4 verification warps at bases 0xffffffe0, 0xffffffff, 0x80000000, 0xfff00000 matched the CPU. In those warps 416 of 416 load indices were above MASK before masking (a random 32-bit value is below 2^20 with probability 2^-12), so the mask was exercised on every load.

Metal does not bounds-check device buffers, so "it did not crash" is weak evidence on its own. The static check is the guarantee: the emitter has exactly one load template and it masks. MEMCHECK: PASS.

6. The bench still works

Command, after all the changes above:

./igneum-bench

Seed	Compile ms	Mhash/s (wall)	Mhash/s (GPU time)	GB/s useful	Loads/hash	CPU verify ms/warp	Verify
igneum-genesis	0.2 (shader cache)	44.56	44.59	18.54	104	0.017	PASS 3/3 warps
igneum-genesis/epoch1	0.3 (shader cache)	47.74	47.77	19.86	104	0.017	PASS 3/3 warps

Dataset fill 5.05 ms GPU (first fill of the process). OVERALL: PASS. The rates match the 3 October table in README.md (45.2 and 48.4) to within 2 percent, so the bench path is unchanged. The compile figures are the system shader cache answering for source compiled earlier today; they are not new compile measurements.

Pack export re-run to a scratch directory and compared byte for byte with the pack written before these changes (../proto-cuda/packs/igneum-genesis):

./igneum-bench --seed igneum-genesis --export-pack <scratch>/pack-genesis
diff -r <scratch>/pack-genesis ../proto-cuda/packs/igneum-genesis

Metal GPU cross-check PASS 3/3 warps, pack written, and diff -r reported no differences: all six files (kernel.cu, program.h, vectors.h, program.json, vectors.json, program.metal) are byte-identical to the pack exported before these changes.

7. Shortcut measurement: the closed-form dataset is not memory-hard

Command:

./igneum-bench --inline-dataset --hours 1

The dataset element is ds_elem(i, d0, d1), six integer operations. A miner can replace every dataset[a & MASK] with ds_elem(a & MASK, d0, d1) and never read memory. --inline-dataset emits exactly that kernel; the CPU verification still passes because the function is unchanged.

Kernel	Dataset	Mhash/s (wall)	Mhash/s (GPU time)	Verify
honest (loads from the 1 GiB buffer), this session	1 GiB	44.6	44.6	PASS 3/3
honest, README sweep of 3 October	4 MiB (cache resident)	569	not recorded	PASS
inline ds_elem, no memory read	1 GiB mask	4,888	6,274	PASS 3/3
inline ds_elem, no memory read	4 MiB mask	5,646	6,024	PASS 3/3

The inline kernel runs about 110x faster than the honest kernel by wall clock (140x by GPU timestamps) and about 9x faster than the honest kernel with a cache-resident dataset. The inline figures are approximate: the 4 timed batches took 2.7 to 3.4 ms in total, so command overhead is visible in the wall figure. The GPU-time figure is consistent with the ALU peak: roughly 1,500 integer instructions per hash at 6.3 Ghash/s is about 9.4 T instructions/s across 40 cores, in the range of this part's compute throughput.

Reading: with a closed-form dataset the prototype hash is compute-bound for anyone who skips the buffer, and the honest kernel's memory traffic is voluntary. Memory-hardness has to come from a dataset element that costs more to derive than to load. RandomX gets this from a cache of roughly 256 MiB expanded by SuperscalarHash into a dataset of roughly 2 GiB (approximate, from memory; vendor/RandomX is not cloned on this machine, so there is no file citation yet). The README already listed an expensive dataset element under "What to try next"; this measurement is why it is not optional.

8. Conclusions

Demonstrated on this machine

1. Bit-exact GPU-versus-CPU agreement of the random-program hash over 10,200 random programs (200 plus the 10,000 run), 4 random warps each across the full 32-bit nonce range, at 64 MiB, 256 MiB and 1 GiB, plus the 7 programs in README.md, plus 14 hand-built edge cases with their operand values proven. Zero mismatches, zero compile failures.
2. Deterministic GPU execution: 5 runs and 3 compiles (one forced cold) of the same program give the same 8 MiB of output; the dataset fill is deterministic and matches the CPU closed form at sampled indices including 0 and MASK.
3. Every dataset index in the emitted MSL and CUDA is masked, by static check on every program tested, and the mask is exercised by essentially every load.
4. No obvious structural bias in the 64-bit output for 3 seeds at 2^20 nonces: bit frequencies, avalanche (mean 32.0, std 4.0, every output bit flips with probability 0.49 to 0.51), chi-square on four 16-bit windows, zero duplicates.
5. The emitter's contract on the generator (rotl 1..31, masks in {1,2,4,8,16}, src != dst) held on every instruction of every fuzzed program.

Not demonstrated

1. Cryptographic security of the construction. Nothing here speaks to preimage, second-preimage or collision resistance, or to an adversary who chooses nonces or influences the epoch seed. The per-register initialisation is splitmix32(nonce ^ seed) ^ seed, which is a bijection of the nonce per register, and the body is add-rotate-xor-multiply with or (which destroys information) and no non-linear table. None of that has been analysed.
2. Resistance to algebraic or structural attacks on weak programs. 3 seeds were measured for bias; the population of programs was not. A program whose or chain saturates a register, or whose load addresses collapse to few values, would be both biased and shortcut-able, and nothing yet rejects such programs or bounds how often they occur.
3. Memory-hardness. Section 7 measures the shortcut directly: with a closed-form dataset the honest kernel's memory traffic is optional. The README's "memory bound at 1 GiB" describes the honest kernel, not the fastest kernel. This is a design gap of the prototype, not a bug in the tests.
4. Vendor independence. Only Apple Metal was run. The CUDA twin exists as text and passed a CPU emulation; it has not run on NVIDIA hardware, where __shfl_xor_sync, __umulhi and shift semantics must be shown to agree bit for bit with these vectors.
5. The 10 ms CPU verification gate against a dataset that is expensive to derive. The current 0.02 ms per warp is with a six-operation element.

The next three tests the real cryptographer must do before the spec is final

1. Weak-program census and rejection filter. Generate at least 10^5 programs. For each, measure on the CPU interpreter (no GPU needed): output bit bias and avalanche on 2^12 nonces, distinct load addresses per hash, fraction of registers saturated by or, and whether any register's final value is independent of the nonce. Report the distribution, define the rejection rule, and bound the advantage of a miner who can grind the epoch seed over candidate blocks.
2. Replace the closed-form dataset with an expensive derivation (RandomX style: a cache of hundreds of MiB expanded by a slow hash, or Argon2-based) and re-run three measurements: the --inline-dataset shortcut ratio (must fall to about 1), the GPU rate at 1 GiB, and the CPU verify time against the 10 ms gate with on-demand element derivation. Also measure distinct cache lines touched per hash so a cache-resident shortcut is ruled out, not assumed.
3. Cross-vendor bit-exactness and a written spec. Run the exported pack and the fuzz vectors on NVIDIA (RTX 5090, CUDA) and on at least one AMD GPU, with the same 4-warp-per-program comparison, including the edge-case programs. Then fix the hash as a specification with test vectors and replace the ad-hoc seed derivation (FNV-1a plus SplitMix) with a standard hash so the seed-to-program mapping is auditable.

9. Generator version 2 (4 October 2026)

Build: swiftc -O -o igneum-bench main.swift -framework Metal (7 s), same machine, idle (load 3).

Fuzz, the required run at a reduced size (the version 1 run was 10,000):

./igneum-bench --fuzz 2000 --fuzz-seed igneum-fuzz-gen2-2026-10-04

Dataset	Programs	Pass	Fail
2^24 words (64 MiB)	647	647	0
2^26 words (256 MiB)	645	645	0
2^28 words (1 GiB)	708	708	0
all	2000	2000	0

2,000 programs: pass 2,000, mismatch 0, compile failures 0, static mask failures 0, contract failures 0. 8,000 warps compared (256,000 hashes), memory-hard dataset. Loads per hash 128 to 128 (every program; version 1 ranged 40 to 232). Op totals: load 32,000, add 15,350, xor 12,802, mad 10,380, mul 10,208, shfl 10,147, rotl 8,983, sub 7,704, rotr 7,632, mulhi 7,596, or 5,198. Compile min 15.7 ms, avg 21.8 ms, max 67.3 ms (all cold). GPU dispatch 1.7 s total, CPU interpreter 17.7 s total (the memory-hard dataset), wall 91.4 s. Every fuzzed program is the accepted attempt of its seed, so the acceptance rule ran 2,000 times inside this run as well. FUZZ: PASS.

Cross-implementation, the Swift generator against the Rust crate (igneum-pow export for the same seeds, closed form unless stated): instruction lists, seed words and all 96 vectors identical on igneum-genesis, igneum-hourly, and the three seeds whose attempt 0 is rejected and attempt 1 accepted (igneum-census-2026-10-03/22, /37, /51); igneum-genesis memory-hard: identical instruction list and 96 vectors between the Swift export (Metal GPU cross-check PASS 3 of 3 warps, cache FNV 48c4f5bf24166b2e) and the checked-in Rust pack proto-cuda/packs/igneum-genesis-mh. The Swift exporter is no longer the pack source; it is the Metal cross-check of the Rust packs (proto-cuda/README.md, "Regenerating a pack"), and its program.json still writes format igneum-program-pack-2 without the generator fields.

Not re-run under version 2: sections 2 (edge, hand-built programs that bypass the generator), 3 (stats), 4 (determinism), 5 (memcheck) and 7 (shortcut). Nothing in them depends on how the instruction list is drawn.

Files

• main.swift: tests under // MARK: - Hardening tests, flags --fuzz, --fuzz-seed, --edge, --stats, --determinism, --memcheck, --inline-dataset; generator version 2 and the acceptance rule under // MARK: - Generator version 2 and the acceptance rule.
• TESTS.md: this file.
• Raw logs of the runs quoted above were kept in the session scratchpad and are not checked in; every table is reproducible with the command above it.