Record that shrub span bounds are wrong, and what that invalidates

A follow-up measurement contradicted the body size implied by the
differential analysis, and the cause is bad span bounds rather than
anything in the format.

Shrub spans were bounded by the next shrub offset, with the last running
to the end of core_data. On level 0 that gives the final shrub class a
925,008-byte span, absorbing most of the lump's tail. The moby-to-tie
boundary is clean by comparison: last moby offset 13414592, first tie
13416192, exactly 1600 bytes apart.

What that invalidates:

- The 3.5M record / 28 MB body figure is an artifact of the oversized final
  span and should not be quoted.
- The "payload looks uniform-random" measurement ran over the same
  contaminated range, so it says nothing about shrub geometry either way.
- The differential result itself still stands -- it was computed on groups,
  and the groups used had bounded spans. The global aggregates built on it
  do not.

What is needed is the offset ending the shrub region. It is not in the core
index, and there is no obvious successor boundary the way moby->tie has one.

This is the second confident-sounding number in this format that turned out
to rest on a bad bound, the first being the VIF reading. Aggregates across
many spans are only as good as the span bounds under them, and bounds are
the easy thing to get wrong because nothing errors when you do.
This commit is contained in:
2026-10-11 02:54:19 +02:00
parent 52228c14ed
commit aec462df21
+32
View File
@@ -929,3 +929,35 @@ What survives from this section is the differential finding, which rests on
grouping and diffing rather than on recognising byte patterns: 8-byte records, grouping and diffing rather than on recognising byte patterns: 8-byte records,
6 varying and 2 constant, 144 varying bytes per group regardless of `count`. 6 varying and 2 constant, 144 varying bytes per group regardless of `count`.
The payload's encoding is still unknown. The payload's encoding is still unknown.
### Shrub span bounds are wrong (later still)
A follow-up measurement contradicts the body size implied above, and the
reason is a bug in how spans are bounded rather than anything in the format.
Shrub class spans were bounded by the *next shrub offset*, with the last one
running to the end of `core_data`. On level 0 that makes the final shrub span
**925,008 bytes** -- a single shrub class absorbing most of the tail of the
lump. The moby-to-tie boundary is clean by comparison: last moby offset
13414592, first tie offset 13416192, a gap of exactly 1600 bytes.
Consequences, stated plainly:
- The "3.5 million records / 28 MB of shrub body" figure is an artifact of
that oversized final span and should not be quoted.
- The payload appearing uniform-random across the full u16 range is measured
over that same contaminated range, so it is not evidence about shrub
geometry either way.
- The differential result (8-byte records, 144 varying bytes) was computed on
*groups*, and the specific groups used had bounded spans, so that
measurement stands. The global statistics built on top of it do not.
What is needed: the offset that ends the shrub region. It is not in the core
index, and unlike moby->tie there is no obvious successor boundary. Until
that is found, shrub span sizes are unreliable and anything summed over all
358 spans is suspect.
This is the second time in this format that a confident aggregate hid a bad
bound -- the first was the VIF reading. The pattern is worth noting: aggregates
across many spans are only as trustworthy as the span bounds underneath them,
and those are the easy thing to get wrong because nothing errors.