QUIC restarts, slow problems: udpgrm to the rescue

In this project I wrote quite some eBPF. I was constantly hitting verifier limits, and like everyone, I was initially just reordering variables and sprinkling "inline" or "noinline" everywhere. That wasn't sustainable.

It turns out - there is now some reasonable tooling to understand verifier!

For stack problems, clang accepts `-s` which prints stack requirement per function, like so:

  ** stack usage by function **
  ebpf/ebpf_aes128.c:180  AES_ECB_encrypt        32   static
  ebpf/ebpf_sha256.c:34   sha256_calc_chunk      64   static
  ebpf/ebpf_sha256.c:123  sha256_hmac            40   static
  ebpf/ebpf_quic.c:86     compute_hp_mask        24   static
  ebpf/ebpf_quic.c:242    decrypt_quic           16   static
  ebpf/ebpf_quic.c:193    _do_decrypt_quic_loop  16   static

And for instruction count, I was able to feed the logs from verbose verifier (during loading) into code-coverage tooling, and count stuff up. The reuseorg prog takes 100k verifier instruction count/paths:

  ** verifier instruction count **
  udpgrm_reuseport_prog  processed  103486  insns  
  udpgrm_setsockopt      processed  9260    insns  
  udpgrm_getsockopt      processed  4215    insns  
  udpgrm_bpf_bind6       processed  75      insns

While 100k is lower than 1m instr count limit, it's still a lot. And reordering some loop or introducing some "if" often makes that count baloon. Remember that verifier instructions is not real instructions during run. Rather it's a pessimistic interpretation of how many instr max under pessimistic conditions could possibly be run. I don't think having an actual run of that max is even practically possible. Think about it as upper bound of static analysis.

Anyway - with stack and instr statistics it's way easier to make sense of verifier problems.