Record that shrub span bounds are wrong, and what that invalidates
A follow-up measurement contradicted the body size implied by the differential analysis, and the cause is bad span bounds rather than anything in the format. Shrub spans were bounded by the next shrub offset, with the last running to the end of core_data. On level 0 that gives the final shrub class a 925,008-byte span, absorbing most of the lump's tail. The moby-to-tie boundary is clean by comparison: last moby offset 13414592, first tie 13416192, exactly 1600 bytes apart. What that invalidates: - The 3.5M record / 28 MB body figure is an artifact of the oversized final span and should not be quoted. - The "payload looks uniform-random" measurement ran over the same contaminated range, so it says nothing about shrub geometry either way. - The differential result itself still stands -- it was computed on groups, and the groups used had bounded spans. The global aggregates built on it do not. What is needed is the offset ending the shrub region. It is not in the core index, and there is no obvious successor boundary the way moby->tie has one. This is the second confident-sounding number in this format that turned out to rest on a bad bound, the first being the VIF reading. Aggregates across many spans are only as good as the span bounds under them, and bounds are the easy thing to get wrong because nothing errors when you do.
This commit is contained in:
@@ -929,3 +929,35 @@ What survives from this section is the differential finding, which rests on
|
||||
grouping and diffing rather than on recognising byte patterns: 8-byte records,
|
||||
6 varying and 2 constant, 144 varying bytes per group regardless of `count`.
|
||||
The payload's encoding is still unknown.
|
||||
|
||||
### Shrub span bounds are wrong (later still)
|
||||
|
||||
A follow-up measurement contradicts the body size implied above, and the
|
||||
reason is a bug in how spans are bounded rather than anything in the format.
|
||||
|
||||
Shrub class spans were bounded by the *next shrub offset*, with the last one
|
||||
running to the end of `core_data`. On level 0 that makes the final shrub span
|
||||
**925,008 bytes** -- a single shrub class absorbing most of the tail of the
|
||||
lump. The moby-to-tie boundary is clean by comparison: last moby offset
|
||||
13414592, first tie offset 13416192, a gap of exactly 1600 bytes.
|
||||
|
||||
Consequences, stated plainly:
|
||||
|
||||
- The "3.5 million records / 28 MB of shrub body" figure is an artifact of
|
||||
that oversized final span and should not be quoted.
|
||||
- The payload appearing uniform-random across the full u16 range is measured
|
||||
over that same contaminated range, so it is not evidence about shrub
|
||||
geometry either way.
|
||||
- The differential result (8-byte records, 144 varying bytes) was computed on
|
||||
*groups*, and the specific groups used had bounded spans, so that
|
||||
measurement stands. The global statistics built on top of it do not.
|
||||
|
||||
What is needed: the offset that ends the shrub region. It is not in the core
|
||||
index, and unlike moby->tie there is no obvious successor boundary. Until
|
||||
that is found, shrub span sizes are unreliable and anything summed over all
|
||||
358 spans is suspect.
|
||||
|
||||
This is the second time in this format that a confident aggregate hid a bad
|
||||
bound -- the first was the VIF reading. The pattern is worth noting: aggregates
|
||||
across many spans are only as trustworthy as the span bounds underneath them,
|
||||
and those are the easy thing to get wrong because nothing errors.
|
||||
|
||||
Reference in New Issue
Block a user