A follow-up measurement contradicted the body size implied by the
differential analysis, and the cause is bad span bounds rather than
anything in the format.
Shrub spans were bounded by the next shrub offset, with the last running
to the end of core_data. On level 0 that gives the final shrub class a
925,008-byte span, absorbing most of the lump's tail. The moby-to-tie
boundary is clean by comparison: last moby offset 13414592, first tie
13416192, exactly 1600 bytes apart.
What that invalidates:
- The 3.5M record / 28 MB body figure is an artifact of the oversized final
span and should not be quoted.
- The "payload looks uniform-random" measurement ran over the same
contaminated range, so it says nothing about shrub geometry either way.
- The differential result itself still stands -- it was computed on groups,
and the groups used had bounded spans. The global aggregates built on it
do not.
What is needed is the offset ending the shrub region. It is not in the core
index, and there is no obvious successor boundary the way moby->tie has one.
This is the second confident-sounding number in this format that turned out
to rest on a bad bound, the first being the VIF reading. Aggregates across
many spans are only as good as the span bounds under them, and bounds are
the easy thing to get wrong because nothing errors when you do.
ReaperOfVeriod
aec462df21