From 6f31657be50eb38f809f76f0191e275f81672cff Mon Sep 17 00:00:00 2001 From: igneum-josh <337424239+igneum-josh@users.noreply.github.com> Date: Mon, 5 Oct 2026 18:55:35 +0100 Subject: [PATCH] release-0.3.9 plan: the ship, the proving rollout, the fee switch (H 210,000, digest ab8847da) and the sweep; restart-hand-nodes.sh: grep -c instead of grep -q under pipefail, lines() never ends the script --- docs/plans/release-0.3.9.md | 171 +++++++++++++++++++++++++++-- infra/devnet/restart-hand-nodes.sh | 5 +- 2 files changed, 165 insertions(+), 11 deletions(-) diff --git a/docs/plans/release-0.3.9.md b/docs/plans/release-0.3.9.md index c20ca9a39..dac1abf10 100644 --- a/docs/plans/release-0.3.9.md +++ b/docs/plans/release-0.3.9.md @@ -214,17 +214,105 @@ in the signer's usage before it writes anything (a 0.3.10 item). |---|---|---| | (1) provers off | Mac: already off (the app's `/api/state`: `enabled false, status off`, 15:46Z). PC 2: run job `prove-off-pc2-039` (the 0.3.8 PowerShell script: `POST /api/prove {"on":false}`, the app state, the node's `igneum_getProvingStatus`), published 16:02:06Z (apps woken, stamp e933f990), started early so the drain overlaps the wait for the `fee-switch` branch | ran 16:02:27 to 16:02:38Z: `api/prove off: {"ok":true}`; PC 2's node pool `{entries 18, failed 0, pending 0, verified 178}`, paidShards 388, tip DAA 113,603 | | (2) empty pool | A 30-s watch on the Mac app node's `igneum_getProvingStatus` that prints every change and every error line: 16:02:36Z `entries 18, pending 0, verified 178`, tip DAA 113,601; then one entry fewer every 30 to 60 s (16:03:07Z 16, 16:07:17Z 10, 16:11:48Z 5), no error line, no new record | OBSERVED empty 16:15:50Z: `entries 0, pending 0, failed 0, verified 178` (13 min 23 s after PC 2's prover went off; the 600-DAA record window at 0.965 blocks/s is 10.4 min) | -| (3) new host on every node | (pending) | | -| (4) `--mode id` | (pending) | | -| (5) provers on | (pending) | | +| (3) new host on every node | The Mac, PC 1, PC 2: by `update-now-039` (published 16:56:12Z through the main script, apps woken, stamp 029c1bb7), see 9. PC 2's WSL tools from the package: `fetch-prove-039` and `rebuild-prover-pc2-039` (7a) | | +| (4) `--mode id` | PC 2 (`/opt/igneum`, in the rebuild job, 16:54:5xZ): shard `0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a`, aggregator `0x474678f3...`, equal to the DMG's host (5d). Mac: the installed app after the update, see 9 | | +| (5) provers on | PC 2: run job `prove-on-pc2-039`; the Mac's prover stays off as before (it was off at 0.3.8 too) | (pending) | -## 8. The ship +### 7a. PC 2's WSL tools from the package -(pending) +| Job | Published | Result | +|---|---|---| +| `fetch-prove-039` (fetch, `--dir prove --extract --extract-dir igneum-prove-wsl2-039 --fresh`; the first add at 16:34:54Z never wrote the job, section 6) | 16:49:21Z | ran 16:49:48Z: the zip (1,530,932 bytes, sha256 ok) extracted into `%LOCALAPPDATA%\igneum\prove\igneum-prove-wsl2-039` | +| `rebuild-prover-pc2-039` (run, PowerShell, the 0.3.8 script with the folder names changed) | 16:54:09Z | ran 16:54:42 to 16:54:53Z as root in Ubuntu-24.04: rsync of `package/` into `~/igneum-prove` (`--delete`, target kept), every source re-stamped, `elf/` listed (the package manifest's shard id `0x2b1a81cb...`), `cargo clean -p igneum-prove-core -p igneum-prove-export -p igneum-prove-host`, `cargo build --release ... --features igneum-prove-host/cuda` (the three crates only, every dependency cached from the 0.3.8 build), installed into `/opt/igneum` (host 73,036,832 bytes 29cc476898c3ab2666110ef65f5b432ac243d1e87e565b57b28e4d140fcbfeee, export 3,602,008 bytes f8b7c19d...; `ldd`: 0 libs missing) | +| `--mode id` on PC 2 (`/opt/igneum/igneum-prove-host --mode id`, in the same job) | | `pinned guests: shard program id 0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a (2832504 bytes, sha256 0x150f4c05a2951fc5) aggregator id 0x474678f35f7545db28055d5e5bbc308231d84a5a072202087a2a8d5b09123896`: equal to the Mac DMG's | -## 9. The machines after the publish +## 8. The ship (16:33 to 16:54Z) -(pending) +`git push -u origin release-0.3.9` at 1269db8 (16:33:55Z), `gh workflow run windows.yml --ref release-0.3.9` -> run +37341810898 (green 16:39:45Z); the script commit 00c7b4f pushed 16:42Z and dispatched again -> run 37342911129 +(started 16:42:42Z, green 16:48:10Z: parse checks; engine, window host, payload, installer, smoke run; the G13 inputs +step against the a24ab01a inputs of 5b). Two more merges of master followed (97b1a74, docs and site only; then the +site rebuilt as ea08798 and the plan as the next commit), so the ship state file got `sha` = 00c7b4f, what the +tool's own commit step would have recorded, and the run resumed with `--from ci`, as the 0.3.6 cut did. + +``` +node tools/ship-app.mjs 0.3.9 --node vendor/igneum-node-036 --branch release-0.3.9 --public \ + --activation-height 84100 --deadline-note "proving v0" --notes "The prover mirrors the fee switch (new pinned guest, shard id 0x2b1a81cb...); node a24ab01a: igneum_exportSegments names the mergeset and every entry's block and body position; the devnet fee-switch runbook" \ + --from ci +``` + +No `--dl-both`: the rotation of 15:25Z left one token folder (`dl-token.next` is gone; the tool would refuse it). +`--public` (coordinator, 16:32Z): the same version into `dl/public/` and the `/public/` aliases. Three false starts +in preflight (16:48:47Z, 16:50:25Z, 16:50:54Z): the pre-push hook's site build had left `site/index.html` and +`site/journey.json` modified after the push, then the plan file (tracked since the 97b1a74 merge) was modified; the +tool refuses any modified tracked file besides the six, which is right. + +| Step | Result | +|---|---| +| preflight | ok: tree ea08798 + the plan commit, 0.3.9 in all 6, fork a24ab01a, gh igneum-josh, live inputs a24ab01a built 16:31:46Z | +| ci | already: run 37342911129 green | +| fetch | Igneum-Miner-Setup-0.3.9.exe 18.9 MB, igneum-windows-app.zip 26.4 MB | +| dmg | already (5e) | +| copy | the DMG into the downloads folder | +| manifest | 0.3.9 mac+windows, signed (key 8f186e37...), verified locally, and in `dl/public/` | +| deploy | one deploy, 16:51Z | +| verify | the three files and the manifest match the local copies; `dl/public/` and the `/public/` aliases serve the local bytes (try 8 of 12: the edge took about 70 s) | +| console | item #360 "Igneum Miner 0.3.9 shipped (mac+windows)"; exit 0 at 16:54:53Z, 211 s | + +| File | sha256 | Size | +|---|---|---| +| Igneum-Miner-0.3.9.dmg | 5e57c5735dc99e44475c43fba8ecffadae9dbd3a150c32ef4858587a98256de3 | 41,333,873 | +| Igneum-Miner-Setup-0.3.9.exe | 1d62a4dbdcef768ebd6fac5e46b199fea4de8854f4cf4c905eee49a6ed9334ce | 19,836,912 | +| igneum-windows-app.zip | 22996a692128a97183c39b37b65275ad33eace4a7d89d83d8f097b315c062adc | 27,683,238 | + +Manifest 0.3.9 published 2026-10-05T16:51:46Z. + +The live manifests, read 16:55:13Z, compared field by field with the 0.3.8 manifest saved before the cut (section 4 of +this plan, read 15:38Z): + +| Field | 0.3.8 (before) | 0.3.9 token folder | 0.3.9 `dl/public/` | 0.3.9 OLD folder | +|---|---|---|---|---| +| version, channel, min_supported_version | 0.3.8, devnet, 0.3.0 | 0.3.9, devnet, 0.3.0 | same | same | +| published_at | 2026-10-05T13:33:22Z | 2026-10-05T16:51:46Z | 2026-10-05T16:51:46Z | 2026-10-05T16:55:1xZ | +| consensus | override {difficulty_v2_activation_daa 33000, proving_v0_activation_daa 84100}, activation_height 84100, deadline_note "proving v0" | identical | identical | identical (carried over from the OLD folder's own 0.3.8 manifest) | +| fees_v1_activation_daa, tuning | absent | absent | absent | absent | +| mac | dmg 53c12b6e..., 41,247,419 | dmg 5e57c573..., 41,333,873, URL in its folder | same bytes, public URL | same bytes, URL in the OLD folder | +| windows | inno-setup 86cffd14..., 19,794,320 | inno-setup 1d62a4db..., 19,836,912, URL in its folder | same bytes, public URL | same bytes, URL in the OLD folder | + +The OLD folder (what Sam's 0.3.5 Mac reads; `rotation-phase-2.md` 8b keeps it until 7 October): the scratchpad +`publish-old-folder.sh` copied the DMG and the installer from the token folder (byte-compared), ran +`publish-manifest.sh --dest --base-url --no-deploy` with the same notes, activation height and deadline +note (the override carried over from the folder's own manifest), one deploy of the site by hand (16:55:18Z; `--deploy` +refuses `--dest`), then `--verify-only` for the OLD base URL: `version 0.3.9, byte-identical, signature OK` (16:55:44Z). + +## 9. The machines after the publish (manifest live 16:51:46Z) + +Baseline 15:35Z: Mac d937c69d app 0.3.8 node 2b6d23ef DAA 111,914 24.0 MH/s; PC 1 ae432dc7 0.3.8 DAA 111,876 110.2 MH/s; +PC 2 1ccfe586 0.3.8 DAA 111,895 122.3 MH/s; Sam's Mac 3a9bf309 silent 5 h (0.3.5); PC 37ba0461 not on the console's +card any more (silent since 13:52Z). A machine counts as updated only when its node reports a DAA score and the miner +a hash rate on 0.3.9. + +| Machine | On 0.3.9 | Its log (`IGNEUM-APP version=0.3.9`) | +|---|---|---| +| PC 2 1ccfe586 (Windows) | `update-now-039` ran 16:56:42Z (30 s after the publish, woken); engine restart 16:56:51Z (unix 1791219412), 39 s after the job; node a24ab01a started 16:56:55Z, `node proof verifier reported: command` +6 s; console 17:30Z: node 2.1.0-a24ab01a, DAA 118,757, 120.6 MH/s | run `win-1ccfe586-20261005-165651` | +| Mac d937c69d | `update-now-039` ran 16:56:43Z; engine restart 16:57:10Z (unix 1791219431); node a24ab01a started (pid 58734), verifier `command` +10 s; console 17:30Z: node a24ab01a, DAA 118,710, 25.4 MH/s | run `mac-d937c69d-20261005-165710`; the node log `node-20261005-165710.log` | +| PC 1 ae432dc7 (Windows) | `update-now-039` ran 16:58:22Z (its wake came 100 s later than the others'); engine restart 16:58:34Z (unix 1791219515); node a24ab01a started (pid 13780), verifier `command` +6 s; a SECOND engine start at 17:26:31Z (run `win-ae432dc7-20261005-172631`, node pid 22280) whose cause is not this rollout (no job of this cut targets PC 1 after 16:56Z; another agent's job or a restart of its own; unverified); console 17:32Z: node a24ab01a, DAA 118,867, 116.8 MH/s | runs `win-ae432dc7-20261005-165834` and `-172631` | +| PC 37ba0461 (Windows, the US laptop) | not reached: silent since 13:52Z (on 0.3.7); the manifest and the job wait for it | | +| Sam's Mac 3a9bf309 | not reached: silent since 09:43Z (0.3.5); reads the OLD folder, which carries 0.3.9 since 16:55Z | | + +The machines watch (`watch-machines.sh`, once a minute from 16:57Z) printed every machine on 0.3.9 with node +a24ab01a and a DAA score from its 17:00Z tick on, but its success test still named the old node commit (2b6d23ef), so +it reported TIMEOUT at 17:33Z instead of success at about 17:00Z: 25 minutes lost to a watcher that was not shown to +fire on a known-finished case first (the CLAUDE.md rule, again; the script is fixed to take the commit as an argument). + +`--mode id` on the installed Mac app (`/Applications/Igneum Miner.app/Contents/Resources/bin/igneum-prove-host`, sha +6846d351..., the DMG's stripped copy): shard `0x2b1a81cb413236cf063077b46ed3111628f6c41036bcf6e23ee4cbbf5679ef7a`, +aggregator `0x474678f3...`, equal to PC 2's `/opt/igneum` host (7a). The Mac app node's pool after its restart: +`{entries 0, failed 0, pending 0, verified 0}`, paidShards 388, tip DAA 119,097 (17:34Z). + +Provers on: `prove-on-pc2-039` published 17:33:32Z, ran 17:35:55 to 17:36:16Z: `api/prove on: {"ok":true}`; the app's +state 20 s later: `backend cuda`, `enabled true`, `available true`, `current "block 77691 shard 0 (0 txs, 0 pgas, open)"`, +keys 10, "proving on the GPU". ## 10. The fee switch @@ -239,7 +327,63 @@ every app, wait for `Calibrated v1 fees from the override file: ... 210000` on e object verbatim whenever it differs from `override.json` (same version or not) and the engine restarts the node at a safe moment. -(pending: the publish itself) +### 10b. The publish (17:37 to 17:42Z) + +Order kept: the proving rollout was complete (the first cross-verified shard under the new id, section 10c) before +anything consensus-bearing was published. The object: +`{"difficulty_v2_activation_daa":33000,"proving_v0_activation_daa":84100,"fees_v1_activation_daa":210000}`; H = 210,000; +tip DAA 119,097 at 17:34Z, so H - tip = 90,903 >= 86,400 at the publish (the 24-hour rule holds; H lands about 19:50Z +on 6 October at 0.965 blocks/s). + +| Step | When | What | +|---|---|---| +| (a) the token folder and `dl/public/` | 17:37:35Z, died at once; again 17:38:34Z, written and deployed | `packaging/ota/publish-manifest.sh --version 0.3.9 --override '' --activation-height 210000 --deadline-note "fees v1" --notes "Calibrated v1 fees from DAA score 210000 ..." --public --deploy` from the worktree. The first run printed only `failed printing to stdout: Broken pipe` and nothing was written (the signer's `embedded` output into `head -1` under `pipefail`, a race the ship tool's own run of the same script did not hit; the second run went through). The second run: mac and windows entries carried over from the folder's 0.3.9 manifest, signed (key 8f186e37...), `dl/public/` rewritten and signed, one deploy; exit 1 only because the public folder's `igneum-downloads.json` (1,317 bytes) was "NOT the local bytes yet" after 12 tries at the edge; the token manifest verified live (`--verify-only`, 17:41Z: byte-identical, signature OK) and the public manifest read live at 17:41:13Z carries the object | +| (b) the OLD folder | 17:37:35Z, verified 17:37:5xZ | `publish-old-folder.sh 0.3.9 "" --override '' --activation-height 210000 --deadline-note "fees v1"`: written, deployed, `version 0.3.9, byte-identical, signature OK` (Sam's 0.3.5 Mac, if it wakes, takes 0.3.9 from this folder and then the object) | +| (c) `update-now` to every app | `update-now-fees-210000` 17:37:51Z (ran against the token manifest BEFORE (a) had landed: a no-op), then `update-now-fees-210000-b` 17:41:34Z (apps woken, stamp 3fa32fec) | the apps re-read the manifest, `ota.rs write_override` writes the object to `override.json` and the engine restarts the node at a safe moment | + +The live manifests after (a), read 17:41:13Z: token folder and `dl/public/` both `0.3.9, published 2026-10-05T17:38:34Z, +consensus {activation_height 210000, deadline_note "fees v1", override {difficulty_v2_activation_daa 33000, +fees_v1_activation_daa 210000, proving_v0_activation_daa 84100}}`; the OLD folder the same object, published 17:37:35Z. + +### 10c. The first cross-verified shard under the new id (before the publish) + +PC 2's prover (key e809e396..., payout 0xCAfc6e74...) took block 77691 shard 0 at 17:35:5xZ; the Mac app node +(`node-20261005-165710.log`): `proof record for block 77691 shard 0 by e809e396... (peer): accepted, verify Command(".../igneum-prove-host")` +at 17:36:28.747Z, `proof of block 77691 shard 0 ...: verified in 0.2 s: RESULT verify: VERIFIED in 0.026 s` at +17:36:29.235Z, `chain block 77730: proof record for block 77691 shard 0 ... paid 1791503640000000000 wei to 0xCAfc6e74...` +at 17:36:31.460Z (1.7915 IGN). Proven on PC 2 under `0x2b1a81cb...`, verified on the Mac under the same id in 26 ms. + +### 10d. The restarts, in the order they happened + +| Node | Restarted with the object | Its lines | +|---|---|---| +| PC 2 1ccfe586 app node | 17:40:55Z (from the first update-now, 17:37:51Z: its app re-read the manifest after (a) had landed at 17:38:34Z) | `node-20261005-165651-r2.log` (digest job `digest-pcs-210000`, 17:43:08Z): `Refusing peer 192.168.68.64:26611 (node 1) and 188.245.5.161:26611 (the seed): consensus params digest mismatch, local ab8847da538dead1dc10e046dfaadab3c1c35928e3748810c4e050d4a886087a remote f10a4eab...`; `override.json` = the object | +| Mac d937c69d app node | 17:42:15Z (`update-now-fees-210000-b` ran 17:42:13Z) | `node-20261005-165710-r2.log`: `Calibrated v1 fees from the override file: ... from DAA score 210000`, digest ab8847da..., `igneumd/2.1.0-a24ab01a`; `override.json` = the object; refusing node 1 and the seed until they moved | +| PC 1 ae432dc7 app node | between 17:46:06Z (its update-now-b) and 17:50Z: its card shows DAA 120,282 with a peer and 115 MH/s at 17:50Z, in step with the network, which a node on the old digest could not be (every peer would refuse it) | its digest line is OWED: `digest-pcs-210000` reported only from PC 2 and `digest-pc1-210000-b` (17:50:49Z) had not reported by 17:55Z (PC 1 took 100 s and 4 min to fetch the two update-now jobs: slow to wake today) | +| The seed | 17:48:27Z (`infra/devnet/restart-seed.sh ''`, the a24ab01a cross-build 7e26e374... re-copied and checked, unit active, MainPID 119125) | journal: `Calibrated v1 fees ... 210000`, digest ab8847da..., `igneumd/2.1.0-a24ab01a` | +| The observer | 17:48:53Z (pid 39882) | `observer-v4.out`: `Calibrated v1 fees ... 210000`, digest ab8847da..., `igneumd/2.1.0-a24ab01a`; peers: the seed and 192.168.68.67 at once | +| Node 1 | 17:49:42Z (pid 40679, caffeinate 40681) | `node1.out`: the same three lines; peers at 17:49:42Z and 17:49:55Z | + +Two script faults on the way, both fixed on this branch: (1) `infra/devnet/restart-hand-nodes.sh` refused the a24ab01a +binary (`strings | grep -q`: `-q` closes the pipe, `pipefail` turns `strings`' SIGPIPE into a failure); now `grep -c` +compared with 0, and `lines()` no longer ends the script when a node's lines are not in the log within 10 s; (2) the +scratchpad script used instead stopped after the observer for that second reason (`set -e` on an empty grep), so +node 1 was restarted by its pid by hand at 17:49:42Z with the same command line and the same file. Between 17:40:55Z and +17:49:42Z the network was split as the runbook predicts (app nodes on ab8847da, the three hand-run nodes on +f10a4eab), 8 min 47 s; node 1 was alone for the last 49 s. + +### 10e. The digest sweep (17:55Z) + +| Node | Digest | Read from | +|---|---|---| +| Mac app node d937c69d | ab8847da538dead1dc10e046dfaadab3c1c35928e3748810c4e050d4a886087a | its node log (above); app state: DAA 120,587, 6 peers, synced, `consensus_switch_daa` 210000 | +| PC 2 app node 1ccfe586 | ab8847da... | its node log through the digest job | +| PC 1 app node ae432dc7 | owed (the card's DAA and peer say the new digest; the line itself not yet read) | | +| The seed | ab8847da... | the journal through `restart-seed.sh` | +| The observer | ab8847da... | `/tmp/igneum-devnet/observer-v4.out` | +| Node 1 | ab8847da... | `/tmp/igneum-devnet/node1.out` | + +Equal to the runbook's digest for the object (read on 2b6d23ef) and to this cut's reading on a24ab01a (5a). ## 10a. The fee-switch branch, as polled @@ -252,4 +396,13 @@ safe moment. ## 11. Open after the cut -(pending) +| Item | State | +|---|---| +| PC 1's digest line | owed: `digest-pc1-210000-b` (17:50:49Z) unreported at 17:55Z; read it with `node tools/jobs.mjs digest-pc1-210000-b` | +| The Mac's miner at 0.0 MH/s since the engine's own restart at 17:45:16Z (its hourly update slot, minute 57; `update check: 0.3.9 is current`) | the node is fine (DAA 120,587, 6 peers, synced); the miner's state after that restart not yet read; not caused by the switch (the node restart at 17:42:15Z was separate and mining was already 0.0 at the 17:47Z card) | +| PC 37ba0461 (0.3.7, silent since 13:52Z) and Sam's Mac (0.3.5, silent since 09:43Z) | take 0.3.9 and the object on their next check (the OLD folder carries both); until then each is refused by every peer from its next restart | +| `dl/public/igneum-downloads.json` | the publisher saw the previous bytes at the edge for its 12 tries at 17:39Z; at 17:42:38Z the live file was byte-identical to the local one (sha 3aa32e75...) | +| The 0.3.9 `publish-jobs.sh` through the worktree before the master merge | left the envelope stale once (16:02Z); every later publish went through the main checkout's script, which rewrote it | +| `make-package.sh` gate | skipped on instruction (`SKIP_GATE=1`); its native half ran on every fixture with the DMG's host (5d); PC 2's CUDA build proved and the Mac verified block 77691 within 40 s of provers-on, the stronger check | +| H = 210,000 at about 19:50Z on 6 October | step 8 of the runbook (budgets, base fees, the first v1 shard) is tomorrow's check | +| Carried from 0.3.8 section 11 | the PC 2 jobs-file signature refusal of 13:19Z (did not recur); the 0.3.6 items | diff --git a/infra/devnet/restart-hand-nodes.sh b/infra/devnet/restart-hand-nodes.sh index d2540a481..a499fcfab 100755 --- a/infra/devnet/restart-hand-nodes.sh +++ b/infra/devnet/restart-hand-nodes.sh @@ -13,11 +13,12 @@ BIN="${IGNEUMD:-/Users/joshm/Projects/igneum/vendor/igneum-node/target-036/relea COMMIT="${IGNEUMD_COMMIT:-a24ab01a}" OV=/tmp/igneum-devnet/override-v3.json echo "$OVJSON" | python3 -c 'import json,sys; o=json.load(sys.stdin); assert isinstance(o, dict) and "fees_v1_activation_daa" in o, "the object must carry fees_v1_activation_daa"; print("override:", json.dumps(o, sort_keys=True))' -strings "$BIN" | grep -q "$COMMIT" || { echo "$BIN is not the $COMMIT node (a 0.3.5 node refuses an override file with fees_v1_activation_daa)"; exit 1; } +# grep -c, not grep -q: -q closes the pipe early and pipefail turns strings' SIGPIPE into a refusal of a good binary (5 October 2026, 17:48Z) +[ "$(strings "$BIN" | grep -c "$COMMIT")" -gt 0 ] || { echo "$BIN is not the $COMMIT node (a 0.3.5 node refuses an override file with fees_v1_activation_daa)"; exit 1; } cp -p "$OV" "$OV.prev-$(date -u +%Y%m%dT%H%M%SZ)" 2>/dev/null || true printf '%s\n' "$OVJSON" > "$OV" stop() { local p; p=$(pgrep -f "$1" | head -1 || true); if [ -n "$p" ]; then kill -INT "$p"; while kill -0 "$p" 2>/dev/null; do sleep 1; done; fi; } -lines() { sleep 10; grep -E "Calibrated v1 fees from the override file|Fees on igneum-devnet|Consensus params digest" "$1" | tail -3; } +lines() { sleep 10; grep -E "Calibrated v1 fees from the override file|Fees on igneum-devnet|Consensus params digest" "$1" | tail -3 || true; } # the observer first stop "igneumd --devnet.*observer-v4" nohup "$BIN" --devnet --nodnsseed --disable-upnp --appdir=/tmp/igneum-devnet/observer-v4 --rpclisten=127.0.0.1:26640 --rpclisten-json=127.0.0.1:28640 --listen=127.0.0.1:26641 \