What actually happened on iron — not a roadmap slide. Each story is
dated, marker-backed, and honest about residuals. Latest close:
2026-08-21 E4 SPA VMLAUNCH on a real R640 (SHELL stub, not Everest).
2026-08-21 · Dell PowerEdge R640 · Cruzer RAYNUV · front USB 2
The guest that came back
After Phase F, the management SPA could list VMs while Linux was
already running. The next honest question was whether
POST /vms/1/start actually launched anything. Not a
table row. A private-EPT guest, on the same CPU, with VMX still on.
That took a day of iron, not a diagram.
The Mac, finally with straight quotes
Lease from COM2 — often 10.99.99.126, never Ubuntu
.124 on the PERC. Bearer
raynu-v-bringup. Spec, sleep two seconds, start:
HTTP/1.1 201 Created
HTTP/1.1 200 OK
Curly quotes in the start URL once hung the Mac in a
dquote> prompt. Start-only after that was 404 —
VM 1 had not been spec’d. The close used ASCII quotes and both
POSTs.
A ladder of EFIs that almost counted
Marker printed is not a close. Several Cruzer flashes proved that
the hard way:
Hang-fix: first SPA VMLAUNCH then
VMPTRLD failed slot=0 and VMXOFF.
G0 relocate by memcpy: Linux had scribbled the live VMCS;
identity-pool VMPTRLD looped forever.
Clone by VMREAD/VMWRITE (98 fields, RIP round-trip): G0 came back;
slot 1 VMPTRLD error 11 (wrong VMCS revision).
VMCLEAR plus rewrite of the incoming VMCS:
error 11 gone; slot 1 re-entry error 7.
Drop the incoming rewrite: first SPA
RAYNU-V-M7-E4-SPA-LAUNCH-OK, then re-entry with
all-zero pin/primary/exit/entry/EPTP/RIP —
Intel error 7, invalid control field. Fail-soft parked SPA and
kept G0. SDM: do not assume VMCLEAR leaves VMCS
data unmodified.
The EFI that closed it (2b795a0) snapshots those 98
fields into a software shadow beforeVMCLEAR,
then VMWRITEs them after VMPTRLD before
any clear-state VMLAUNCH. It also punches the SPA
2 MiB slab out of G0’s identity so Linux cannot scribble the
parked VMCS.
That pair repeated for many quanta. Paste truncated mid-loop.
No error 7. No error 11. No VMXOFF. That is P0-14 /
RAYNU-V-M7-E4-SPA-LAUNCH-OK closed on iron.
The repeating lines are E4 bring-up debug, not the production
console. ADR-011 default is quiet COM2. The next EFI logs the
first G0 re-entry, first SPA re-entry, first restore per slot,
then stays quiet except HTTP, WARN, and markers.
What we claim — and what we don’t
We claim a private-EPT SHELL guest from SPA start,
and that it can leave and come back after VMCLEAR.
We do not claim Mount Everest. The SPA guest is
SHELL CPUID, not a second Linux, not a distro installer, not TLS.
Every switch is still VMLAUNCH (outgoing is
VMCLEAR’d), not VMRESUME. HTTP during
the switch loop is not in the close paste. Residual is guest
console / TLS and a real distro installer.
Ops side-win the same day:
~/projects/raynuv/flashcruzer.sh pulls the latest
green CI EFI onto Cruzer RAYNUV (label + USB id, never
a hardcoded /dev/sdc). Keep APE PHY. Bind LOM
:38. Force Off; F11 Cruzer.
ADR-013 said lifetime management HTTP would not come from firmware
SNP after EBS. It had to come from a host-owned NIC — MMIO, DMA,
our rings — with the APE PHY left alone so iDRAC NCSI stayed up.
2026-08-20 closed two gates on that stack.
E3b: listen after BOOT-OK
Census pick is Broadcom BCM5720 14e4:165f, dual-port.
Dedicated iDRAC NIC is not the host LOM. Host management is jack
:38 / 01:00.0, station
b0:26:28:5c:5a:38. Do not bind APE
:3a. COM2 after BOOT-OK:
Mac on the same LAN as the LOM — not the iDRAC dedicated port —
reached SPA + Bearer. That is HDA E3b /
M7.8. Firmware SNP and Tcp4 stay dead after EBS; we do not chase
them.
Phase F: the same listen with VMX on
Durable HTTP after BOOT-OK is necessary. It is not
sufficient. The product loop needs HTTP while guests run.
EFI 0d06297b polled the native Device on the credit
scheduler quantum. G0 Linux stayed scheduled. G1–G3 parked (their
VMCS regions were not safe to VMPTRLD from the
identity pool). Coexist listen:
Hold COM2 the same evening: twenty-five HTTP-OK exchanges, SOL
stayed up. ADR-013 Stage 1 (Phases 0–G) is closed. Phase G is
the accepted-risk note: host HTTP and guest virtio-net share the
BCM5720 LOM :38. Not 802.1Q, not a second NIC.
Product next stays installer + TLS/console.
One TCP listen slot. A half-open SPA tab holds it; the next
curl times out (curl: (28)). Close the
tabs. Use curl -4 --noproxy '*'.
2026-08-19 · Dell PowerEdge R640 · BCM5720 · APE NCSI
Don’t take the APE PHY
The day before E3b closed, almost every native-NIC EFI printed the
same two letters: bmsr=7949 — link down — then skipped
listen. COM2 looked busy. The jack had no carrier.
Broadcom’s APE firmware talks NCSI to iDRAC over the same copper
the host wants. A Linux-style BMCR_RESET, a PHY lock,
or a reset that drops APE’s phylock, and you steal the analog from
the BMC. Dedicated iDRAC stays up. Host LOM goes dark. SOL
console com2 is not a substitute for a link on
:38.
Several Cruzer prefixes are retired for that reason
(42b42c99, ec08c00f,
1404f055, skip-CORECLK 26573eb1, and
any take-PHY with ape-nophylock=no). The path that
later listened inherited the SNP analog, skipped BMCR when
ape-ncsi=yes, used CORECLK_RESET for DMA
only, and printed:
boot: HINT — keep APE PHY (iDRAC NCSI); will not take phylock
boot: HOST-NIC BCM5720 inherit SNP analog; CORECLK_RESET for DMA (skip BMCR)
boot: HOST-NIC BCM5720 ape-nophylock=yes keep-ape-phy=yes ape-lock=yes
That is not a gate close. It is the rule the next two days stood
on. Keep APE PHY. Bind LOM :38. Never write the PERC.
The day after Cruzer stamps closed, we tried to make the management
SPA live for the whole boot — not just the 45-second PRE-EBS window.
The idea was tidy: park SNP + smoltcp across
ExitBootServices (no CloseProtocol), skip the poll that
already hung, then listen after BOOT-OK. Firmware would
keep owning the NIC. We would keep owning HTTP.
Iron did not care about the diagram.
First EFI: the hang
Poll SNP immediately after EBS and COM2 stops at
boot: smoke frame phys=0x1000000. The first
iface.poll never returns. Guests never start. That
binary (SHA 924af894…) is retired. Do not flash it
again.
Second EFI: the timeout, then the red screen
Skip the immediate poll. Guest path comes back — SHELL, M4,
BOOTED-FROM-DISK, RAYNU-V-R640-BOOT-OK.
Then the idle banner asks the Mac to curl the same lease that
worked an hour earlier in the PRE-EBS window
(10.99.99.133:8443, MAC b0:26:28:5c:5a:3a):
boot: mgmt HTTP listening on 10.99.99.133:8443 (POST-EBS SNP idle)
boot: CURL NOW (post-EBS) → http://10.99.99.133:8443/
Same laptop, same LAN, same token:
curl: (28) Connection timed out after 5002 milliseconds
Then Dell’s red screen: invalid opcode (06), Software UEFI0004 on
the BSP, RIP = 0x17 — a jump into
garbage. That is not a clean hang. That is firmware taking a
pointer we had no right to keep using.
What we still claim — and the ADR that follows
PRE-EBS RAYNU-V-M7-UEFI-HTTP-OK is still true. E2 and
E5 stamps are still true. We do not claim
RAYNU-V-M7-POST-EBS-HTTP-OK. We do not chase Tcp4 or
SNP after EBS.
The same evening Vignesh flashed the WARN-only EFI. COM2 ran through
M4.5, printed RAYNU-V-R640-BOOT-OK three times, then:
boot: WARN — POST-EBS SNP idle skipped; firmware SNP dead after EBS lease=10.99.99.133:8443 (PRE-EBS was the mgmt window; do not chase SNP/Tcp4)
No red screen. That is the close for ADR-013 Phase A
on iron. Lifetime management HTTP needs a host-owned
NIC (MMIO/DMA) and a dedicated mgmt heap. That architecture
is
ADR-013
(Accepted). HDA names it E3b.
Preserve kit:
releases/v0.1.0-adr013-baseline.
Mount Everest does not close on a 45-second window.
Two days later the host-owned NIC listened — see
2026-08-20.
The SPA guest that actually launched is
2026-08-21.
2026-08-16 · Dell PowerEdge R640 · front USB 2 · Cruzer Micro
Two boots on a Cruzer — stamps, not a filesystem
Virtual Floppy is often read-only. The E5 contract needed a stick the
hypervisor could write. Same R640, front USB 2, SanDisk Cruzer Micro:
boot 1 from the SPA Install arm, reboot, boot 2 from the same stick.
Boot 1 printed the persist write, sized the live virtio disk, and
latched reboot-pending:
boot: persist wrote installdisk.bin bytes=1024
boot: E5 install disk bytes=67108864
RAYNU-V-M7-ISO-DISK-WRITTEN
RAYNU-V-M7-ISO-REBOOT-PENDING
An earlier boot 2 failed a prefix mismatch: 1 KiB ESP stamps into a
64 MiB virtio image looked like zeros at DRIVER_OK. The prefix-copy
EFI (SHA 2d931dcd…f4281eed) loaded the stamps into the
front of the larger disk. Boot 2 then printed:
That string is the iron close for HDA E5 stamp persist
(documented equivalent of ISO-INSTALL-OK; host/CI never
print that iron marker). It is still the LBA stamp
contract: 1 KiB of LBA0+LBA1 patterns on the ESP, not a guest
root filesystem and not a distro installer. Live 64 MiB virtio is RAM;
durability is the Cruzer file. Mount Everest stays open.
E3b closed 2026-08-20 — RAYNU-V-M7-HOST-NIC-HTTP-OK
after BOOT-OK on native BCM5720 (Dedicated iDRAC + host
LOM :38). See
HTTP beside VMX
and
don’t take the APE PHY.
E4 SPA VMLAUNCH closed 2026-08-21 — see
the guest that came back.
Residual now is TLS/console and a real distro installer
(not El Torito first).
2026-08-16 · Dell PowerEdge R640 · iDRAC9 · host NIC
The floppy had no Tcp4 — so we used SNP
E2 proved the hypervisor boots on iron. E3 asks a different question:
can an operator on the LAN reach the management SPA without a host OS?
Virtual Floppy firmware on this R640 had no Tcp4 /
Ip4 / Dhcp4. PRE-EBS probes returned all zeros until
ConnectController brought SNP up (snp=12).
From there the residual path was SNP + smoltcp DHCP, then a PRE-EBS
listen on the leased address. COM2 printed the lease, the bind, and
the close marker:
boot: mgmt HTTP listening on 10.99.99.127:8443 (PRE-EBS SNP window)
RAYNU-V-AUDIT: AuthAllowed method_tag=1
RAYNU-V-M7-UEFI-HTTP-OK
…
RAYNU-V-R640-BOOT-OK
From the Mac on the same LAN:
GET / returned the SPA (HTTP 200), and Bearer
raynu-v-bringup on /vms returned
{"ok":true,"listed":true}. That is HDA E3
MVP — bring-up UI, not TLS, not lifetime listen, not a full console.
The day after, we learned SNP does not survive EBS — see
2026-08-17.
Stamp persist on a writable stick closed the same day as HTTP —
see
the Cruzer story.
Closing iron on substance is one thing. Watching COM2 print the claim
string you named the gate after — that is another. After
finish_boot started emitting
RAYNU-V-R640-BOOT-OK (and we stopped shipping a stale
floppy), the Virtual Floppy run ended like this:
boot: VMXOFF ok
boot: E2 marker build=r640-boot-ok-marker
RAYNU-V-R640-BOOT-OK
RAYNU-V-R640-BOOT-OK
boot: M4.5 complete — SMP dual-vCPU path OK
RAYNU-V-R640-BOOT-OK
Same path as the afternoon close — M0 through Linux SHELL, four
guests under the credit scheduler, virtio-blk/net, SMP BSP+AP, then
VMXOFF — plus the stamp r640-boot-ok-marker and the
literal E2 string, three times. That paste is archived as
2026-08-15-boot-ok-marker-com2.txt.
2026-08-15 · Dell PowerEdge R640 · iDRAC9
First light on PowerEdge R640
For months RayNu-V closed gates on a Latitude laptop under QEMU nested
KVM. Mount Everest’s hard gate is different: boot
r640-hypervisor.efi on a real R640 and
read the markers on serial. Today we closed that gate —
loudly, messily, and with the logs still warm.
Morning: an EFI you can pull
We built the release kit on the cloud agent, then put
r640-hypervisor.efi under releases/ so a
normal git pull on the Mac dropped the binary — no Rust
toolchain required on the operator laptop. Media maker packed a FAT
image with \EFI\BOOT\BOOTX64.EFI. iDRAC Virtual Media
mapped the .img as a virtual floppy. One-shot boot.
Virtual console open.
Booting from Virtual Floppy Drive
RayNu-V r640-hypervisor …
RAYNU-V-M0-BOOT-OK
That was first light. The screen then sat on M0 with a blinking cursor.
ConOut had done its job; ExitBootServices was about to take it away.
The console that wasn’t COM2
HTML5 Virtual Console and BIOS serial redirect show
ConOut. Post-M0 RayNu-V prints with port I/O. We had
only written COM1 (0x3F8). Dell iDRAC SOL is
ssh … then console com2 — COM2
(0x2F8). Same freeze on “serial,” different wire.
Kit v0.1.0-com2 mirrored every host diagnostic byte to
both UARTs and added ConOut breadcrumbs through ExitBootServices.
Suddenly the story continued:
boot: M0 complete — entering M1.0 firmware handoff
boot: PE assets embedded (.askern/.asinit) — prefer PE
RAYNU-V-M3-ASSETS-OK
boot: calling ExitBootServices — ConOut/video ends after this line
RAYNU-V-M1-EBS-OK
RAYNU-V-M1-VMXON-OK
RAYNU-V-M3-LOAD-OK
RAYNU-V-M3-BZIMAGE-OK
Three EPT lessons in one afternoon
Large-RAM iron does not forgive assumptions that QEMU’s
-m 512M hid.
Pool at 5 GiB. The bump allocator picked the largest
conventional span. Guest code lived near 0x1402fd000;
identity EPT only maps [0, 512 MiB). Fix: clip the pool
into the precise window (v0.1.0-eptfix).
No virtio BAR hole. Clipping to 512 MiB filled the
whole window; pick_shell_slab_hpa found no unowned 2 MiB
leaf above guest RAM. Fix: cap the pool at 256 MiB so
[256 MiB, 512 MiB) stays free (v0.1.0-barfix).
Host CR3 outside EPT. VMLAUNCH ran; guest shared
host CR3 at 0x47e01000 (~1150 MiB). First page-table walk
EPT-faulted. Fix: build a guest CR3 that identity-maps the precise
window (v0.1.0-cr3fix).
That was mid-afternoon edge: BAR window correct, guest RIP in-window,
CR3 still the UEFI table high in DRAM. The rest of the day was the
ladder — host CR3, RIP-relative GPRs, INVPCID, keep earlyprintk, then
Enable XSAVES — until /init stopped eating its own stack.
Evening: SHELL, M4, claim
Kit v0.1.0-xsavesfix printed
secondary=0x0010100a (XSAVES bit set). After
Run /init as init process:
RAYNU-V-M3-SHELL-OK
RAYNU-V-M3-NOIRQ-OK
RAYNU-V-M4-SHELL-G1
RAYNU-V-M4-2VM-OK
…
RAYNU-V-M4-SMP-OK
boot: VMXOFF ok
A confirming rebuild later the same day — same EFI SHA
c3a688d0…ba28d611, Virtual Floppy again — reproduced the
full chain on COM2. That is the close for
RAYNU-V-R640-BOOT-OK (HDA E2 / M7.5).
What we claim — and what we don’t
We claimRAYNU-V-R640-BOOT-OK on substance
from the xsavesfix / confirming rebuild archives: real R640 COM2 through
Linux SHELL and the M4 probe chain. Later the same day, the marker-fix
rebuild also printed the literal string on COM2 — see
the next story. The day after,
E3 MVP closed as RAYNU-V-M7-UEFI-HTTP-OK — see
the HTTP story.
We still do
not claim Mount Everest closed. E5 stamp persist
closed later the same weekend — see
the Cruzer story.
The next morning, firmware SNP after EBS hit a hang and a red
screen — see
2026-08-17.
Residual at the time was HDA E3b (host-owned NIC, ADR-013),
then remaining E4 polish and a real distro installer. E3b and Phase F
closed 2026-08-20 — see
HTTP beside VMX.
E4 SPA VMLAUNCH closed 2026-08-21 — see
the guest that came back.
Host CI still prints only
RAYNU-V-M7-R640-SCAFFOLD-OK for the iron boot path.
