Fix level extents read as bytes, and extract the level geometry
A defect in the Phase 1 extractor, found by reading the data extent's contents: level extents are declared in sector counts, the generic G/ pair table uses byte counts, and level containers were being routed through the generic path. So LEVEL0.WAD's data extent -- 6,484 sectors -- was taken as 6,484 bytes. Every test still passed, because a truncated file is still a file. The extent's own header gave it away: it declares an uncompressed total of 13,277,920 bytes, and 13,277,920 / 6,484 is 2047.7. A ratio landing on 2048 is a unit error, not a compression ratio. Across 27 levels this had been discarding roughly 300 MB of level content. Level containers now go through the typed LevelHeader instead. Level 0 yields: data.bin 13,279,232 the data extent core_data.wad 9,738,678 the geometry, compressed core_data.bin 16,969,296 the geometry, decompressed core_index.bin 31,600 the geometry directory gameplay.bin 659,456 sound_bank.bin 1,503,232 core_data is a WAD LZ77 stream -- the format already implemented for the game -- and it decompresses cleanly. The first real level content this project has recovered. New: LevelDataHeader parses the data extent's own header, a table of byte ranges naming the level overlay executable, the geometry directory, the GS RAM image, the HUD banks and the geometry. Note this is a third meaning for a "size" field on the same disc, after sector counts in the level header and byte counts in the generic table; all three are documented together in level-data.md so the next parser does not rediscover this the hard way. The geometry directory's offsets were checked against the geometry rather than taken on trust: all twelve land inside the 16.9 MB of decompressed core_data, which is what confirms the header is being read correctly. The class tables themselves do not yet decode under the documented layout, and that is recorded as open work in plan/level-assets.md rather than papered over. 75 unit tests, 9 integration tests against the real disc. Hash table regenerated at 5,159 files and 5,671,915,799 bytes -- hashes only, no content. Extraction now writes to ~/Documents/gc-extracted rather than /tmp, which is 16 GB and did not have room for a 5.3 GB output.
This commit is contained in:
+167
-143
@@ -2498,149 +2498,173 @@ e5f948087847e7d4a98b2a069599ea39b32fb996 14 hud/0776.bin
|
||||
bf9fdc4cdb7924599d6eeaf380e6198542390245 15 hud/0778.bin
|
||||
c25009da53bbe1163360026f14958e95baf0096e 15 hud/0779.bin
|
||||
e1baca5f8178da95e7dc496305a36f984be542aa 15 hud/0780.bin
|
||||
9d78f2847e5e3f66c5eaf4a0b1f80848c4531c67 3 level0/0000.bin
|
||||
ee9aebf66b33361899d68d046f73120374efd188 6484 level0/0001.bin
|
||||
de9435f57cfa07383bef01e08ff8f6be32a4a411 734 level0/0002.bin
|
||||
70a77d34462f9438e40e2cc0618d318fd7cb39ae 322 level0/0003.bin
|
||||
147a7ea20ce65d44ae4a7f03721d56f9364453bb 12 level0/0004.bin
|
||||
fd934d7d105dbb40d6eee54b91e0f7d9543588e2 7700 level1/0001.bin
|
||||
1eeda24575c67d85dc3c93bbbd520593bd7a1071 635 level1/0002.bin
|
||||
dbcc19188f4ff4169b2a1852ff52123bbf9542b4 396 level1/0003.bin
|
||||
93a86fb611e187dfab037b329cc3109add5fcb33 9 level1/0004.bin
|
||||
d0494f7fff25a8f09aaf78024d2eed7574da43d7 1519 level1/0005.bin
|
||||
cb4ad2cede50755b7d707ff23b81cd42b342f863 116 level1/0006.bin
|
||||
e384f88d9b1c2c8c6940f6300ccc40e51211211c 157 level1/0008.bin
|
||||
0a9f8e70479985b8eb9eb186a7b90b4efba38cfe 156 level1/0009.bin
|
||||
feac57d2624d4c57e744a9e95c58ea9592d9f7e0 6103 level10/0001.bin
|
||||
319029b9137e72e4272679a57d7fb274515745c4 638 level10/0002.bin
|
||||
b211884d74970d285f16cbda1a3a9788f46bbb9d 288 level10/0003.bin
|
||||
432ec71f57f4e35f57a058169005bd7b4d4db0c2 7509 level11/0001.bin
|
||||
2a6481fec3e82dd94ec4137750345f0d758fd4fc 484 level11/0002.bin
|
||||
0308d8cbb0b404161782471f49805f10dd5e9f06 505 level11/0003.bin
|
||||
c55db301cae3256da2cf4ab172a41a807b3719e8 25 level11/0004.bin
|
||||
6ef880efa1b8e941243c46f4732739889b001330 971 level11/0005.bin
|
||||
02c96eda56a88778f87d1d18ace744595632de0c 743 level11/0006.bin
|
||||
4957eefb5f9f9cb4804db5b79f3b87e4237a5e23 344 level11/0008.bin
|
||||
9cf6bc893804e91a09ac219c20615d4bba3e64e9 346 level11/0009.bin
|
||||
93d9f428e168a65f32912b8b80513b4a9e642581 3 level12/0000.bin
|
||||
e3fdd805ba28a9b74e88d50a667bbad4da89ad16 7391 level12/0001.bin
|
||||
ebb44f98bf04a5c30d4986c57e81ad36f3ca7def 766 level12/0002.bin
|
||||
e382350790213ed40272c8b166f0deacaffa0578 397 level12/0003.bin
|
||||
a475dec066991df114a517a13c9cfcce8dddca55 18 level12/0004.bin
|
||||
80c46e74c75055a76d2c8c570f9487468f0d1468 7667 level13/0001.bin
|
||||
62577729740eab112b994fdd1e5692cea68039e2 687 level13/0002.bin
|
||||
349f0bcc84eefd98de3549029eaaf8cf927e6223 417 level13/0003.bin
|
||||
e1435d350dc578a8831d12d35a5862803e13466a 9 level13/0004.bin
|
||||
29e2dcfbb16f63bb0254df7585a15bb6fb5e927d 3 level14/0000.bin
|
||||
05af1e5fabc8bbb9090ce43373a3fb1e8a4d9c96 7299 level14/0001.bin
|
||||
439f6567da1a3f4cc0758444b40bd1a0cd6fe36e 825 level14/0002.bin
|
||||
9f7c149a2fdcf418e5ba34ccdc5413c98438204a 338 level14/0003.bin
|
||||
9f58d0b144011b6818ed3e8f88f6ff496918bc7a 12 level14/0004.bin
|
||||
8ad9ee512eccce0e2ab4cd9e820f8b7004dfb2fa 5919 level15/0001.bin
|
||||
cd2cb4ade72d3bd861c2065cba0291b63ca4570f 721 level15/0002.bin
|
||||
959543f928d76737df70eac8c834fce441b55292 240 level15/0003.bin
|
||||
053d8d6ceeba9453c97d0ee5374db863e6f77ad4 2 level15/0004.bin
|
||||
eb9ba397fe5aeecf05694b5633f3abfd893abb78 7857 level16/0001.bin
|
||||
e62f5aa4cf2584641cf8ea6e94b5097503662434 808 level16/0002.bin
|
||||
cd609657985f99b83d5ec5690e27ba6b5d3fbf34 291 level16/0003.bin
|
||||
10905a54bab839bc8b139e0a11b37ec3ab86d25e 7 level16/0004.bin
|
||||
caf7ed52438a4d5c084726ebb5f17dd40c47a39b 4 level17/0000.bin
|
||||
11832990c21adc4e525cf01b3a98095bbb98a6b5 7117 level17/0001.bin
|
||||
bcdec0eda2f5d0837f806261d8901f9f10e1efcc 658 level17/0002.bin
|
||||
95b8737766c7244ceae053d7df0ac49f3ceb9f95 389 level17/0003.bin
|
||||
45aa2cd5372e944d313bcfdd1909cac766847744 14 level17/0004.bin
|
||||
1d140c6e4e3b23dc7bd3c7091469698179ff9632 7569 level18/0001.bin
|
||||
7e983769a81baf262bcc294f03dcd4153d4f2b56 729 level18/0002.bin
|
||||
8e274334b309173d2c1c25531fb8ceee8e90048a 330 level18/0003.bin
|
||||
65c9fc5945ca1896a16ff03cac5571bf39091da8 10 level18/0004.bin
|
||||
178821eaf9509e2314b68cd341a2106033d3e33c 7513 level19/0001.bin
|
||||
29b9ff0b26d310f5f51027704f38d3f9e0023b8b 767 level19/0002.bin
|
||||
83ec0ae41ae3499b304e56174ca4b0a88fdf6240 403 level19/0003.bin
|
||||
b93ec66f7b5a879551f26b15c282aeee20104a0f 23 level19/0004.bin
|
||||
fd6417bfb15231413da423bd728512d1301eb20e 1588 level19/0005.bin
|
||||
e07d86687e93a87dbe06b52f64dc40e407685a76 1339 level19/0006.bin
|
||||
a82fc2ea70d6aacd17b0968d3cd6cb3fcf3f4a05 497 level19/0007.bin
|
||||
fc9183dab5edbc9d71852f7c8f240e6e2733ab93 45 level19/0008.bin
|
||||
1daa0b36af3c476d51851bbc7eaee203c4d5c0a3 45 level19/0009.bin
|
||||
601112ff27d2fdf042836e4bfe95b079830ac0cd 45 level19/0010.bin
|
||||
5fc30e310cfc5e030a8ee5d199bcf93d130a4b48 7368 level2/0001.bin
|
||||
49389c971237100433f1179a1a5564d671453ab4 546 level2/0002.bin
|
||||
0a0620ef5db7ba0efa506e0dcb27d8f08784e4eb 386 level2/0003.bin
|
||||
44ede948f8d5de898306ded9433ad7f84bda80b3 14 level2/0004.bin
|
||||
268f619b8e7edea16ff275c1409af1df089dc26a 1214 level2/0005.bin
|
||||
7f9ea6b67b2de8e34d3fd37b4fd13de798ad76a8 157 level2/0006.bin
|
||||
ab9965f223661b918cd67f5a1de671fd36f221b0 214 level2/0008.bin
|
||||
949492e6a9e3a0ddb8aabda3371c6ca606f770e4 214 level2/0009.bin
|
||||
e821c13a93ef7020676a45da780aed6c57b05355 6952 level20/0001.bin
|
||||
5b640654d3bf521c03fd13aa853812bfd0cbe90d 652 level20/0002.bin
|
||||
ab282185efda919797d2683db8e391881652b365 407 level20/0003.bin
|
||||
1660c7078e59572de7ab039c439c0b0b35b4257f 18 level20/0004.bin
|
||||
24e094280faf8e98bfb3ea13041afe67c2dbb6aa 714 level20/0005.bin
|
||||
95c106f8bdffa4b15db3ac697aea169c46350d36 917 level20/0006.bin
|
||||
503e6acc80e347f756511ba438079abf152fa3cd 276 level20/0007.bin
|
||||
cc155c6c63d1c484ae15794f1be279f71bebc2d4 167 level20/0008.bin
|
||||
997e1938927decc49061e0a1d4450fbf84e9ff6c 167 level20/0009.bin
|
||||
d2d09f0d49f956233124962ed484b2a96120e49f 168 level20/0010.bin
|
||||
e815b4f9c3737a0e0bc188c44278bc9ea5de27ca 5724 level21/0001.bin
|
||||
1eb5d1cab61e8f0de73b14fc3601e7f56b967c47 570 level21/0002.bin
|
||||
7b8b36e7c889f6788e9e7f25984534069aabc965 219 level21/0003.bin
|
||||
e71a685f77a5a04eddde65e739069ebeeff5dcf8 10 level21/0004.bin
|
||||
f208351848d3c585284c86d0f66fcb080a183315 5705 level22/0001.bin
|
||||
c4d9cf73171dfe150ed3c0b48686d82894dfe084 666 level22/0002.bin
|
||||
b70a1c7fda18487ff4a22c4b31e1836c82d85046 272 level22/0003.bin
|
||||
062e8f3e7e9b3b88a7dcc5812f327c41a2f15a9d 5 level22/0004.bin
|
||||
133cc6de802a50d3d6265f9999e16f74ef3bcffd 5085 level23/0001.bin
|
||||
625d7201f5fa12a7c6a29a8c2734e1a87de4af6b 818 level23/0002.bin
|
||||
8ddceb118456d41e009d449f7cc5c69f9484346f 247 level23/0003.bin
|
||||
e612d3b13ac1a6be14644c0158cb03df9139bf7e 4 level23/0004.bin
|
||||
e99846cca94215f0f050c966b64a68dc5022d467 3409 level24/0001.bin
|
||||
825e789b5b47b95f3914a722cf344c6c83aa989d 458 level24/0002.bin
|
||||
3b65e12a48a334d8b86c08a8bd6790d4f16b749a 210 level24/0003.bin
|
||||
ab89a75ec855ebe3fb39670df3ed2b1d9ee654b8 3346 level25/0001.bin
|
||||
9afb21e5002c17834e27004d51bca66f707c2ebf 669 level25/0002.bin
|
||||
37976eaec0f83dd38e9654dea62fd1ba360ce15d 204 level25/0003.bin
|
||||
cea6a1decd88484b76c950ca860c2abd29dd2ae6 4504 level26/0001.bin
|
||||
816eaf00f1297d908885cbac7e56b5ca455779f3 524 level26/0002.bin
|
||||
0985e8ac13a7fc94322a96da26b3194022ff6c5f 332 level26/0003.bin
|
||||
ff0bef6339f269ee4cdc1559985ace3d26326db0 6 level26/0004.bin
|
||||
885da9be88c2c5385af36b2ba3222eebcd9bd0b2 7401 level3/0001.bin
|
||||
d49090628975aa8ab81ae10d73c040058ddd0c8d 833 level3/0002.bin
|
||||
250c8291212f733d352cfa182e3a968e6d1c9058 366 level3/0003.bin
|
||||
455e82411b00ca1821edce18283b7dcc5a231d49 15 level3/0004.bin
|
||||
9069ca78e7450a285173431b3e52c5c25299e473 4 level4/0000.bin
|
||||
fc07e4be7448a92e8a5e88767913941a995d6a58 7812 level4/0001.bin
|
||||
2cc0e33b580027782e556fb566b7873cd82005f9 604 level4/0002.bin
|
||||
9fe28eb73583a8a57f7c5a0101e6c311b9fb5489 469 level4/0003.bin
|
||||
c723e023557fffb1d11bc205b17ded81edd797a1 24 level4/0004.bin
|
||||
6a4b8aca82c4a012daac67cfe49b0eb131b50fde 1678 level4/0005.bin
|
||||
addb65b4136821ba5d35863a1ed36ad166bc11c1 1185 level4/0006.bin
|
||||
7fc0202ae8c6ffbb587b4d62f6c6935414e3debd 148 level4/0008.bin
|
||||
d21b816c14f438154bb006aec0b44e8c2a5b1be3 147 level4/0009.bin
|
||||
0748ce2b1ccb4101f55eeab31dcf08965e3cb443 7213 level5/0001.bin
|
||||
c97c52d44728c4ecf0adce6691b81cb0fab68274 679 level5/0002.bin
|
||||
2b02321fdc93c6d9e99ce5295c006c7c4a23634e 251 level5/0003.bin
|
||||
aeb26f3b3ccb452e61c571babb7903c010d07b7c 4 level5/0004.bin
|
||||
29e2dcfbb16f63bb0254df7585a15bb6fb5e927d 3 level6/0000.bin
|
||||
89402cdecb013a06b729945f72ffb073eeeb504e 8192 level6/0001.bin
|
||||
b46738b6e47069b29d79d676117d1dec7c8c25a1 815 level6/0002.bin
|
||||
eac95bd254bc0bd82aa028c04103aae2704aaea3 454 level6/0003.bin
|
||||
b0fc1d30fe8b8e21819223cae079a5717b459007 17 level6/0004.bin
|
||||
cca3542463b2bd251cad3a65c2aa348032470d77 7015 level7/0001.bin
|
||||
34b3735d4ea4a32c868100fc6e8208dbd3c0464c 760 level7/0002.bin
|
||||
f3a79a12be746408e5a108afde5d75dbaf2bb724 311 level7/0003.bin
|
||||
9726fccd386dab5d77edf914a0bd05c232ee7b8b 8 level7/0004.bin
|
||||
24609e1abaa41c1997b32eac7d8f1c8d501e44fd 1602 level7/0005.bin
|
||||
5649fb3bdface666fe7a2169438bad976ec2a841 1183 level7/0006.bin
|
||||
f5387de20edd1e77df8be4e5612497fb9d9eef98 6531 level8/0001.bin
|
||||
b46fae29423ce17c8f855e805b4c131cc4a7b326 778 level8/0002.bin
|
||||
16790e98c9934fbf7151b397f1986f5237e182b1 439 level8/0003.bin
|
||||
eb5e2b66ab38d7b16c6c06c9c674c5601defb826 11 level8/0004.bin
|
||||
eaa7664f4417044b48da1e8ae7c5bd79173dce93 424 level8/0005.bin
|
||||
ccb9f15e10f11c0c5dfc87fc85d011b96f75f90c 1877 level8/0006.bin
|
||||
d05d33cf037879571520148733e4a2bee8bf3e0c 7926 level9/0001.bin
|
||||
5f3bf30b3692adafbf29978169b54ad99d91c579 829 level9/0002.bin
|
||||
aad0207adb8afa991cc1f115e32cccb4ff41928c 422 level9/0003.bin
|
||||
1053b927974e19a66c19fc9f71aae2920fbcf3d2 13 level9/0004.bin
|
||||
1d8f736d4eb4fe324f916e63b3b97862b68f34f8 16969296 level0/core_data.bin
|
||||
67c9e91fd44677f27d1b58b66967455a2d2bff12 9738678 level0/core_data.wad
|
||||
89496a7b1cf67a2fc88ad99cc72fc46807c95dfe 31600 level0/core_index.bin
|
||||
4a8f9d6608b776a86fb15420acadea4c2b749a9f 13279232 level0/data.bin
|
||||
133206d28cd25c39f1b6a760e345c0f0f409b7bf 659456 level0/gameplay.bin
|
||||
7ec9566aeae513fe272bc97899a4dbe15a2a74ff 1503232 level0/sound_bank.bin
|
||||
6b1a358c6eb6039d0df51fce061b58c274e8c99c 19023392 level1/core_data.bin
|
||||
96b33cef1ac3c9bab5283ccf11bd86b0aa20ebcb 12165629 level1/core_data.wad
|
||||
7cae5c7c2258f391ef72a8503183e36747214fbd 34016 level1/core_index.bin
|
||||
60a9070e5668ff8f5bf560318968811c1ee38c71 15769600 level1/data.bin
|
||||
a07364ccac5fb6dc3d1fb1c859fca13b205e6a1f 811008 level1/gameplay.bin
|
||||
1318041a43acca46fb4f2348b2f4fa81bd810948 18432 level1/occlusion.bin
|
||||
5d6ead57dd2adf83e75b53ef5d026682676d5010 1300480 level1/sound_bank.bin
|
||||
c7bbd091e77719876e53fe38cfb66ad68d0952d3 14427312 level10/core_data.bin
|
||||
1c56f35c21dd429f52cdc701de3d3ab846f57918 8999864 level10/core_data.wad
|
||||
2c0073d2e78ff61beb69050087ff953ac1ccc65c 19696 level10/core_index.bin
|
||||
fe9e5169820bc9272aa5fcfbbabfc74e7b12d1ce 12498944 level10/data.bin
|
||||
d3e48237b03649578f6962e36f1bc197c3a8af9f 1306624 level10/sound_bank.bin
|
||||
da253d8b4dc4f8fc482880c54cd832c53bfd4aa0 20717584 level11/core_data.bin
|
||||
a0173c7577180d32e9bfed67464eecf3bc12c033 11414905 level11/core_data.wad
|
||||
0e0d5289451a3bffa376992d0b6a288421a7a55d 36000 level11/core_index.bin
|
||||
49116fede92a6846bbbb0c3951f3314cbc770f80 15378432 level11/data.bin
|
||||
ff6811c08f3fe034bae54ab066e3051d4cbb39cc 1034240 level11/gameplay.bin
|
||||
e91c93e8af747ce8c0802c3c96cd88acc126a5c3 51200 level11/occlusion.bin
|
||||
850f01ef9678ed0b0b6242c410ad7e57b935079f 991232 level11/sound_bank.bin
|
||||
d13c8eb4f2ab552e330a119bf1e14d55f9921220 20995456 level12/core_data.bin
|
||||
9d8355548c0d5c911e18124e83a04f3c58d604de 11510373 level12/core_data.wad
|
||||
7b3ab93650bd82117b2b8ff596f95799123ff6e8 34864 level12/core_index.bin
|
||||
a134bad90883ad15ff7b52d909da461a491d253b 15136768 level12/data.bin
|
||||
b058a1f7858198be55a90bf70d81db1f8039e132 813056 level12/gameplay.bin
|
||||
709e185927a11b846e6b1d66c2fa770cf284ea65 1568768 level12/sound_bank.bin
|
||||
74fd46bfcfcb97132c509d2147070327976faab4 21301168 level13/core_data.bin
|
||||
7c4b8c7fd086de66e19bd440a5e63fea4729e0bb 12037495 level13/core_data.wad
|
||||
c7a99c3252427f11a202a907df605b3d43baaae0 35744 level13/core_index.bin
|
||||
29c09e0877c62ba4dee1325cf00c944f9ad78ceb 15702016 level13/data.bin
|
||||
ab5e7bd921f3973e3d15052838ec87676d35928a 854016 level13/gameplay.bin
|
||||
2793ed3153f9f8ace889b855dbfc53f29c94c0bc 1406976 level13/sound_bank.bin
|
||||
386e9a7cef58f23f66fbc21de42be234c25be0b4 19738512 level14/core_data.bin
|
||||
b34bf0528e2c753ce9fd65791e89ba6ec8491358 11217244 level14/core_data.wad
|
||||
cc03cc08cc55f27d57bcdc6c9c435e28479a65e3 34848 level14/core_index.bin
|
||||
fb30c8c10944377af4a76e2804209be630dba993 14948352 level14/data.bin
|
||||
95157d87f0c5e76a38daead9e07d01281b65ebf0 692224 level14/gameplay.bin
|
||||
28a70e7880794f94a76e2b340373d1dc40484fee 1689600 level14/sound_bank.bin
|
||||
9e1a9269a59741f15608ee773192a4c4e3adadbb 14566720 level15/core_data.bin
|
||||
3c7f4291ff09cdfb0c1cd2e435fb4c039faf7572 8538458 level15/core_data.wad
|
||||
cda9e44cee8b5ac58f9abd97dfcdbbe14e76d490 21792 level15/core_index.bin
|
||||
39e2c8e15375dae4c73f61a7be980dccf4a85aa9 12122112 level15/data.bin
|
||||
9b13d67b1276d25937cab21344fa8a6687cc0882 491520 level15/gameplay.bin
|
||||
f89c334b887f377f341fc9232d474affa89fc544 1476608 level15/sound_bank.bin
|
||||
49c2efb647efc0baaf919aaaa140f0a660725b3d 21374192 level16/core_data.bin
|
||||
9c9291257d21ac62dff3ac4090f9d2f9b6c5c3e8 12442996 level16/core_data.wad
|
||||
5a57d24086355cb7fb2410ac9b5daff4bf2d63a5 34080 level16/core_index.bin
|
||||
8e58f79a35fbe78382065bdce0f53d2cc9f1ccf1 16091136 level16/data.bin
|
||||
af60ee961ab3c4508f6fc7f7e5db32d1bbe08769 595968 level16/gameplay.bin
|
||||
84723c80beedd7e55143a424a1b2f6255a200ab0 1654784 level16/sound_bank.bin
|
||||
485e5343a6f146179b319fb7d76388eee6e21276 20920320 level17/core_data.bin
|
||||
fc90583b7560be17115f57c8ecf6fc14ec2e59d8 10922651 level17/core_data.wad
|
||||
5f9496629cf8c32071c2d4a125fee1ca7b89f73e 35856 level17/core_index.bin
|
||||
d7669eec9eb3cf36bf4f649333ae79864c1a7043 14575616 level17/data.bin
|
||||
10a5f5e2862f7364a9bbe886dadb9ebca8ab0381 796672 level17/gameplay.bin
|
||||
37c76a37d9e3cbbed6dfc324b0b90d3498ecb0e7 1347584 level17/sound_bank.bin
|
||||
f9240e2bd6406f206e155b09cc5c914b6574e41e 20268640 level18/core_data.bin
|
||||
f44add8374224a84070995ecb4b1be8cfb58aa9f 11721139 level18/core_data.wad
|
||||
6ab43f9167d67246e18394eef5b732589fcaff66 33088 level18/core_index.bin
|
||||
d0f931677cb6589a564937c3d15c13344b923969 15501312 level18/data.bin
|
||||
2adc53f721fca95dd1012994d243e0704edd034c 675840 level18/gameplay.bin
|
||||
b26483cc29ecb8b6d2c71919ff7cd56b09522bc1 1492992 level18/sound_bank.bin
|
||||
0918d824b72b892a24f46108bbc60f6e875236f6 20233504 level19/core_data.bin
|
||||
b79df8e074db7d3c75bc2b3b9c2a4de61e5dd9cd 11797811 level19/core_data.wad
|
||||
b0ea377a39a616dbd2b15224a9996925c7b28f4d 31536 level19/core_index.bin
|
||||
3218aa99e3ea4e9f4a050a5117413261defded4e 15386624 level19/data.bin
|
||||
8700a93086450328f2693b553e1df447a79fcd83 825344 level19/gameplay.bin
|
||||
877ca7fb8e80d810d6c37fb993795aa10b5b1033 47104 level19/occlusion.bin
|
||||
8298f799c394542e912e41ff3321ee55c70e0c8f 1570816 level19/sound_bank.bin
|
||||
6d56189ad74f04d366116b736ab12888154de833 20209408 level2/core_data.bin
|
||||
f0eb5c033e84092748cf3457818b0d4c85da7a0a 11372886 level2/core_data.wad
|
||||
73411362c1cfa8faec733705d93e953f1381ef30 33808 level2/core_index.bin
|
||||
14c7c5cd788c0bebe531029bc1f8f8d570ea1771 15089664 level2/data.bin
|
||||
bfc9f6bc73edfa7deb46505f187c274dcf7b863a 790528 level2/gameplay.bin
|
||||
01c7d4c7b0ecb44a91694ffde796ff941057191a 28672 level2/occlusion.bin
|
||||
bce56b3b899d72c5d89f612bd70447f066c10757 1118208 level2/sound_bank.bin
|
||||
daf162b58756e2661f3ec891c8736a1611b4dc52 19637472 level20/core_data.bin
|
||||
7e7203ac3a2dbf578546573a34fa6dce4c336d35 10449039 level20/core_data.wad
|
||||
b12e8599d211addc85cbe138e58157ade380a575 37360 level20/core_index.bin
|
||||
640e6a62f3795bb285663364c2929f4640dacf33 14237696 level20/data.bin
|
||||
7368af71bc6c070fc6422fd69a354b2f3518ed2b 833536 level20/gameplay.bin
|
||||
61ae37031d1df746bffb4a0bddeaee85558c894e 36864 level20/occlusion.bin
|
||||
abce77e8841c0ed0fa006db81840b35ea4c75afd 1335296 level20/sound_bank.bin
|
||||
5e850ab42d2dc3e8fa1f9ae74d401be1658c9ba7 14483120 level21/core_data.bin
|
||||
f10321014e7ccd61a65f294baefaf35875d6294b 8178784 level21/core_data.wad
|
||||
728b90cdbb44f1b94c140d82fc03268ca54408ac 25456 level21/core_index.bin
|
||||
4a3ebd058fe62df17ed4d74fe190442ff2993474 11722752 level21/data.bin
|
||||
6c2b9faaef23c142a61316190852251404734c39 448512 level21/gameplay.bin
|
||||
8fbcb2360d5244e92bfd7c2ccd453eec44cd042b 1167360 level21/sound_bank.bin
|
||||
7b0733a07f28e7d7cf5d8a19b1058a98a5563c9f 14288048 level22/core_data.bin
|
||||
34a2d1ac57f13cacdf46cd7d73f356951f87dd43 8186976 level22/core_data.wad
|
||||
edb2fd7c72d0a517613fd813429aa9036e2eb509 25152 level22/core_index.bin
|
||||
3519cac29e6371802b40e3d8234ef7064abee336 11683840 level22/data.bin
|
||||
a8760bb193082c5e85faeb5f799ab8585f366805 557056 level22/gameplay.bin
|
||||
dba3b2b5856efbd28a8aeaa15fbee6f4fb479895 1363968 level22/sound_bank.bin
|
||||
c6b6f4087f4c9cd374a587b13777f9575953838a 12647664 level23/core_data.bin
|
||||
dbdca886def506bd2e6fd189c0fc4ca8d2834b97 6900832 level23/core_data.wad
|
||||
ed196a1541e056da4433d366c1a096143ea87ca0 23024 level23/core_index.bin
|
||||
9a26194ba7d5930279a62113b1377ffeb1294269 10414080 level23/data.bin
|
||||
336b3c864d1aa317003499e7d1d2d5a8cee13d58 505856 level23/gameplay.bin
|
||||
e254322f1d87d2650c21ec2c118546df1a56d9e8 1675264 level23/sound_bank.bin
|
||||
508808939296de54fee8928824bdcc6d37204d01 6386608 level24/core_data.bin
|
||||
c44701fc0a229dc8071a30594231d5edfb6b4119 3805312 level24/core_data.wad
|
||||
c526f84012360c24a07e4bbd6823dd18ce8b3181 16608 level24/core_index.bin
|
||||
f38883faa65e74f67a44da4a55b9b476848d7cae 6981632 level24/data.bin
|
||||
3bf88b1d1c455ab90bbacc3270a3f54c94b900ef 937984 level24/sound_bank.bin
|
||||
c81ea8c485748fccec6619bacc75afe8ce2f7e20 5898736 level25/core_data.bin
|
||||
80d763434b9dda1224c446de67e4907cfbdcfbe7 3513706 level25/core_data.wad
|
||||
c493407fb493ff359e90d1bdfc1b379a2a6c540a 16592 level25/core_index.bin
|
||||
e57a615ccce4afd0a3de13f638ecaa19f88d0789 6852608 level25/data.bin
|
||||
e9d246c13a7198b42a71f081d767b9e9598d01a1 1370112 level25/sound_bank.bin
|
||||
948974c4fd10e7ca7ad1a362af9cb46aa6c81773 9530880 level26/core_data.bin
|
||||
6140efd9a7ebbcd81bfddf1ce0a5395ffb136d2e 5894651 level26/core_data.wad
|
||||
d71c13701f26cd0a4bbcb9508fb0387ba54d2d62 20704 level26/core_index.bin
|
||||
4fdd7d88a5d3db63418c968f8a5f53ac370b86be 9224192 level26/data.bin
|
||||
0d63ad0bc2edcd958b1c7dbec303c256ed8850e1 679936 level26/gameplay.bin
|
||||
8f6a74b8008a9ebc9e92853d0b339a8e725c9992 1073152 level26/sound_bank.bin
|
||||
f5d3cd4480fe6a52a019d04401a360018b279b81 21148480 level3/core_data.bin
|
||||
690209848c64647194548d5e81f329d471392a8e 11383592 level3/core_data.wad
|
||||
351819059a5b74d8e17f50ee87974e5c30504c3e 37568 level3/core_index.bin
|
||||
eb8366046b5643cc54d42d800c06bf66ccd4ef36 15157248 level3/data.bin
|
||||
55cc4acecac6ba19232e6505fdc06a22b0633a45 749568 level3/gameplay.bin
|
||||
d986825dfc2bed713eb9699504536346658aaa3f 1705984 level3/sound_bank.bin
|
||||
f8c0aa3778b5bcb1995e516045b28ca6ac35502f 21107088 level4/core_data.bin
|
||||
ec75f324670188af581c4fceaa4cb285353f95b8 12171908 level4/core_data.wad
|
||||
cc4997acbdc0a6ac85d427227c04e6af6e34cd46 38080 level4/core_index.bin
|
||||
67a114e12f840f4c01e3f96b48c504f93b839b03 15998976 level4/data.bin
|
||||
eeee7ebf8d9980f9afc6d9c2bc2156bcca015ddd 960512 level4/gameplay.bin
|
||||
fed269c2d706a2ba11cab965e090fd41d434055c 49152 level4/occlusion.bin
|
||||
bdd446767b1a84043d65ff968278cd2d5808705a 1236992 level4/sound_bank.bin
|
||||
cb07a102a92b887d06d9314c07ce1db50e14c1fb 16715440 level5/core_data.bin
|
||||
e5cc08e39cb6883b391605bdcff8ba3bb8423699 11205994 level5/core_data.wad
|
||||
d41e06bc7086ed6876bef602420cca8d52492b1e 23536 level5/core_index.bin
|
||||
b241ea9d8fe18dd5bdbc8a4255ba8450a66b3c6a 14772224 level5/data.bin
|
||||
c71539e0f8b0e2d666d9382475539c7a590fee04 514048 level5/gameplay.bin
|
||||
5c2fd4470dd8c892af7686770af8cf3e2396373c 1390592 level5/sound_bank.bin
|
||||
8ea18c5a51e4ae07ca56d92d65ffde60f4bcce12 20935696 level6/core_data.bin
|
||||
f2962a73cfb15ab1bef23253d863d2433b264383 12810673 level6/core_data.wad
|
||||
29284ba33a151cb45df0b4d074fa7ccc9d2c0667 38080 level6/core_index.bin
|
||||
ccb0a4e8c51be81fbcd8ad325ab8d521bbdb7604 16777216 level6/data.bin
|
||||
b9ff3b4b8fb6308aac7f5ddba1c4b1099d484dd5 929792 level6/gameplay.bin
|
||||
82fcce964396eb508768933d092a557bc1b05c89 1669120 level6/sound_bank.bin
|
||||
6692290d78706a2b92cba541264ddea9a764e2f2 18209392 level7/core_data.bin
|
||||
f1cae801f1050c57258f93a7035148b3094ece03 10780010 level7/core_data.wad
|
||||
79e3957d149ce3b00f713eb1757fec89d7db86c4 32736 level7/core_index.bin
|
||||
9b18673cac7d7d6c3d64bebca909f3bb76ff9f39 14366720 level7/data.bin
|
||||
bc8f542a56d4795c77774201903e25b01962c039 636928 level7/gameplay.bin
|
||||
e02d1b0d2339d72ed5d07abfe105f41e2288f59f 16384 level7/occlusion.bin
|
||||
8e07c803b709100023c3b44c8842eec9b8f49898 1556480 level7/sound_bank.bin
|
||||
264a306c3a83b9137d9753b282e9c773cd386713 21086224 level8/core_data.bin
|
||||
0a96094edd63dc4dd940894b007f85b21874cfeb 9700660 level8/core_data.wad
|
||||
8d1b6c3a05e9b2140c056df3e538ed537b9d3a33 30624 level8/core_index.bin
|
||||
ee6f0ca2c15e1e0828d63547a8c19d2c45b8c0e2 13375488 level8/data.bin
|
||||
ff4c7dd1924165ddc7dbb9494a166eb3e140c95c 899072 level8/gameplay.bin
|
||||
afeb717c151befdabfcf6297b0dbb1772ac13fdb 22528 level8/occlusion.bin
|
||||
1f65f6b44e4e4a146aaac3d63b7f459657de0010 1593344 level8/sound_bank.bin
|
||||
fd39207fd776cb3ba6b8d5f6d3b10d61610c5e49 21592016 level9/core_data.bin
|
||||
314f79d0a1d9ae630d67da9e3259a658c3c5642b 12578171 level9/core_data.wad
|
||||
ffed974844aa0bef984bc33c86ff6757e0f9c5bc 34976 level9/core_index.bin
|
||||
db530147f5d900e5a85294070a0a5e707c086213 16232448 level9/data.bin
|
||||
3839c295b0408460f50032ff166cac27c0bdf411 864256 level9/gameplay.bin
|
||||
bc38b5983d8717ff4c7a35fac3cd1042ef7bd675 1697792 level9/sound_bank.bin
|
||||
f666c702e75052034d72125db6bd5d6821ac8379 9 misc/0000.bin
|
||||
2af38bb5b2fe77af4ed2a6079ebe5857247082a5 367 misc/0001.bin
|
||||
ccc75282e2a72f7adbe1ba7bee744c0799f41b04 62 misc/0002.bin
|
||||
|
||||
|
@@ -22,6 +22,9 @@ fn main() -> ExitCode {
|
||||
Some("identify") => run_identify(&args[1..]),
|
||||
Some("unpack") => run_unpack(&args[1..]),
|
||||
Some("verify") => run_verify(&args[1..]),
|
||||
// Scratch helper used while reversing the level formats: decompress a
|
||||
// byte range of an already-unpacked file with the WAD decoder.
|
||||
Some("wad") => run_wad(&args[1..]),
|
||||
Some("--help") | Some("-h") | Some("help") | None => {
|
||||
usage();
|
||||
ExitCode::SUCCESS
|
||||
@@ -187,6 +190,56 @@ fn run_unpack(args: &[String]) -> ExitCode {
|
||||
}
|
||||
}
|
||||
|
||||
fn run_wad(args: &[String]) -> ExitCode {
|
||||
if args.len() < 4 {
|
||||
eprintln!("usage: gc-extract wad <file> <offset> <size> <out>");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
let data = match std::fs::read(&args[0]) {
|
||||
Ok(d) => d,
|
||||
Err(e) => {
|
||||
eprintln!("error: {}", e);
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
};
|
||||
let off: usize = match args[1].parse() {
|
||||
Ok(v) => v,
|
||||
Err(e) => {
|
||||
eprintln!("error: bad offset: {e}");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
};
|
||||
let sz: usize = match args[2].parse() {
|
||||
Ok(v) => v,
|
||||
Err(e) => {
|
||||
eprintln!("error: bad size: {e}");
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
};
|
||||
if off + sz > data.len() {
|
||||
eprintln!(
|
||||
"error: range {off}..{} is outside the {} byte file",
|
||||
off + sz,
|
||||
data.len()
|
||||
);
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
match gc_formats::wad::decompress(&data[off..off + sz]) {
|
||||
Ok(out) => {
|
||||
if let Err(e) = std::fs::write(&args[3], &out) {
|
||||
eprintln!("error: cannot write {}: {e}", args[3]);
|
||||
return ExitCode::FAILURE;
|
||||
}
|
||||
println!("{} bytes -> {} bytes -> {}", sz, out.len(), args[3]);
|
||||
ExitCode::SUCCESS
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("error: {e}");
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn run_verify(args: &[String]) -> ExitCode {
|
||||
if args.len() < 2 {
|
||||
eprintln!("usage: gc-extract verify <out-dir> <table.tsv>");
|
||||
|
||||
@@ -13,6 +13,8 @@ use std::path::{Path, PathBuf};
|
||||
|
||||
use gc_formats::global_wad::{GlobalWad, RangeKind};
|
||||
use gc_formats::iso9660::IsoImage;
|
||||
use gc_formats::level_data::{LevelDataHeader, LevelDataPart};
|
||||
use gc_formats::level_wad::LevelHeader;
|
||||
|
||||
/// One unpacked asset, with the identity we record for verification.
|
||||
pub struct Extracted {
|
||||
@@ -76,6 +78,106 @@ pub fn unpack(iso_path: &Path, out_dir: &Path) -> Result<Vec<Extracted>, String>
|
||||
continue;
|
||||
}
|
||||
|
||||
// Level containers have a typed header whose extents use sector counts,
|
||||
// not the byte counts of the generic `G/` pair table. Reading them
|
||||
// through the generic path would take a 6,484-sector extent as 6,484
|
||||
// bytes and silently lose 13 MB of every level, so they go through the
|
||||
// typed parser.
|
||||
if stem.starts_with("level")
|
||||
&& stem
|
||||
.trim_start_matches("level")
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_digit())
|
||||
{
|
||||
match LevelHeader::parse(&raw) {
|
||||
Ok(header) => {
|
||||
let mut parts = 0usize;
|
||||
for (part, range) in header.ordered_extents() {
|
||||
let Some(offset) = range.byte_offset() else {
|
||||
continue;
|
||||
};
|
||||
let len = range.byte_len() as usize;
|
||||
if offset + len as u64 > raw.len() as u64 {
|
||||
eprintln!(
|
||||
" {stem}/{}: runs past the end of the container; skipped",
|
||||
part.name()
|
||||
);
|
||||
continue;
|
||||
}
|
||||
write_file(
|
||||
out_dir,
|
||||
&format!("{stem}/{}.bin", part.name()),
|
||||
&raw[offset as usize..offset as usize + len],
|
||||
&mut written,
|
||||
);
|
||||
parts += 1;
|
||||
}
|
||||
// The `data` extent is itself a container of byte ranges.
|
||||
// Split out the geometry directory and the geometry, so the
|
||||
// latter is usable without re-deriving the layout later.
|
||||
// `data_extent` is the `data` slice; `raw` is the whole
|
||||
// level container and its offsets are unrelated.
|
||||
let data_extent: &[u8] = &raw[header.data.offset as usize * 2048
|
||||
..(header.data.offset as usize + header.data.size as usize) * 2048];
|
||||
match LevelDataHeader::parse(data_extent) {
|
||||
Ok(data_header) => {
|
||||
let core = data_header.part(LevelDataPart::CoreData);
|
||||
let index = data_header.part(LevelDataPart::CoreIndex);
|
||||
if let Ok(bytes) = core.slice(data_extent, data_extent.len()) {
|
||||
// `core_data` is a WAD stream: keep both the
|
||||
// compressed original and the decoded form,
|
||||
// because the hash table covers the original.
|
||||
write_file(
|
||||
out_dir,
|
||||
&format!("{stem}/core_data.wad"),
|
||||
bytes,
|
||||
&mut written,
|
||||
);
|
||||
match gc_formats::wad::decompress(bytes) {
|
||||
Ok(decoded) => {
|
||||
write_file(
|
||||
out_dir,
|
||||
&format!("{stem}/core_data.bin"),
|
||||
&decoded,
|
||||
&mut written,
|
||||
);
|
||||
}
|
||||
Err(e) => eprintln!(
|
||||
" {stem}: core_data did not decompress ({e}); the \
|
||||
compressed form was kept"
|
||||
),
|
||||
}
|
||||
parts += 1;
|
||||
}
|
||||
if let Ok(bytes) = index.slice(data_extent, data_extent.len()) {
|
||||
write_file(
|
||||
out_dir,
|
||||
&format!("{stem}/core_index.bin"),
|
||||
bytes,
|
||||
&mut written,
|
||||
);
|
||||
parts += 1;
|
||||
}
|
||||
}
|
||||
Err(e) => eprintln!(" {stem}: data extent header did not parse: {e}"),
|
||||
}
|
||||
|
||||
println!(
|
||||
" {:<16} {} bytes, level id {} -> {} parts",
|
||||
stem,
|
||||
raw.len(),
|
||||
header.id,
|
||||
parts
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
skipped.push((relative.clone(), e.to_string()));
|
||||
write_file(out_dir, &relative, &raw, &mut written);
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
let container = match GlobalWad::parse_with(&raw, RangeKind::for_container(&relative)) {
|
||||
Ok(c) => c,
|
||||
Err(e) => {
|
||||
|
||||
@@ -306,3 +306,82 @@ fn level_extent_sizes_are_sector_counts() {
|
||||
// Read as bytes, 734 would be under four sectors and the tiling would break.
|
||||
assert_eq!(h.sound_bank.byte_len(), 734 * 2048);
|
||||
}
|
||||
|
||||
/// The level data extent is a container of byte ranges. Its header locates the
|
||||
/// geometry index and the compressed geometry, and every range must lie inside
|
||||
/// the extent on all 27 levels.
|
||||
#[test]
|
||||
fn level_data_extents_are_in_range() {
|
||||
let path = require_disc!();
|
||||
let iso = gc_formats::iso9660::IsoImage::open(&path).expect("ISO should open");
|
||||
|
||||
let mut checked = 0;
|
||||
for entry in iso.files.iter().filter(|e| e.name.starts_with("G/LEVEL")) {
|
||||
let raw =
|
||||
gc_formats::iso9660::read_at(&path, entry.lba, entry.size).expect("read container");
|
||||
let lh = gc_formats::level_wad::LevelHeader::parse(&raw).expect("level header");
|
||||
let data = lh.data;
|
||||
let Some(offset) = data.byte_offset() else {
|
||||
continue;
|
||||
};
|
||||
let len = data.byte_len() as usize;
|
||||
let extent = &raw[offset as usize..offset as usize + len];
|
||||
|
||||
let dh = gc_formats::level_data::LevelDataHeader::parse(extent)
|
||||
.unwrap_or_else(|e| panic!("{}: data extent header: {e}", entry.name));
|
||||
|
||||
for (part, r) in dh.present_parts() {
|
||||
assert!(
|
||||
r.end() <= extent.len() as u64,
|
||||
"{}: {part:?} ends at {} but the extent is {} bytes",
|
||||
entry.name,
|
||||
r.end(),
|
||||
extent.len()
|
||||
);
|
||||
}
|
||||
|
||||
// The geometry is the big one and must be substantial.
|
||||
assert!(
|
||||
dh.core_data.size > 1_000_000,
|
||||
"{}: core_data is only {} bytes",
|
||||
entry.name,
|
||||
dh.core_data.size
|
||||
);
|
||||
checked += 1;
|
||||
}
|
||||
assert_eq!(checked, 27, "expected 27 level containers");
|
||||
}
|
||||
|
||||
/// The geometry must decompress. This is the first test that touches real level
|
||||
/// content rather than headers.
|
||||
#[test]
|
||||
fn level_geometry_decompresses() {
|
||||
let path = require_disc!();
|
||||
let iso = gc_formats::iso9660::IsoImage::open(&path).expect("ISO should open");
|
||||
let entry = iso.find("G/LEVEL0.WAD").expect("level 0");
|
||||
|
||||
let raw = gc_formats::iso9660::read_at(&path, entry.lba, entry.size).expect("read container");
|
||||
let lh = gc_formats::level_wad::LevelHeader::parse(&raw).expect("level header");
|
||||
let data = lh.data;
|
||||
let extent = &raw[data.byte_offset().unwrap() as usize
|
||||
..(data.byte_offset().unwrap() + data.byte_len()) as usize];
|
||||
|
||||
let dh = gc_formats::level_data::LevelDataHeader::parse(extent).expect("data header");
|
||||
let core = dh
|
||||
.core_data
|
||||
.slice(extent, extent.len())
|
||||
.expect("core_data slice");
|
||||
|
||||
assert!(
|
||||
gc_formats::wad::is_wad(core),
|
||||
"core_data should start with the WAD magic"
|
||||
);
|
||||
|
||||
let decoded = gc_formats::wad::decompress(core).expect("core_data should decompress");
|
||||
assert!(
|
||||
decoded.len() > core.len(),
|
||||
"decompressed geometry ({}) should exceed the compressed size ({})",
|
||||
decoded.len(),
|
||||
core.len()
|
||||
);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,315 @@
|
||||
//! The level data extent header (`LEVEL<n>.WAD` → `data`).
|
||||
//!
|
||||
//! Spec: `docs/formats/level-data.md`.
|
||||
//!
|
||||
//! The `data` extent is the level's container: it holds the level overlay
|
||||
//! executable, the compressed geometry, the GS RAM image and the texture banks,
|
||||
//! each at a byte offset declared by a table at the very start of the extent.
|
||||
//!
|
||||
//! ```text
|
||||
//! offset size field
|
||||
//! 0x00 8 overlay the level's executable
|
||||
//! 0x08 8 core_index the geometry directory (see core_index.rs)
|
||||
//! 0x10 8 gs_ram Graphics Synthesizer RAM image
|
||||
//! 0x18 8 hud_header
|
||||
//! 0x20 40 hud_banks[5]
|
||||
//! 0x48 8 core_data the compressed geometry
|
||||
//! 0x50 8 transition_textures
|
||||
//! ```
|
||||
//!
|
||||
//! Every field is a `{offset, size}` pair of **byte** values relative to the
|
||||
//! start of the extent. Note this is a third convention on the same disc: the
|
||||
//! level header's extents are sector counts, the `G/` pair tables are byte
|
||||
//! counts, and this is byte counts again.
|
||||
//!
|
||||
//! The table is 0x58 bytes and the first extent starts at 0x80, so bytes 0x58
|
||||
//! to 0x7f are padding.
|
||||
|
||||
use crate::buf::{Buf, Error, ErrorKind, Result};
|
||||
|
||||
pub const LEVEL_DATA_HEADER_SIZE: usize = 0x58;
|
||||
|
||||
/// A `{offset, size}` pair in bytes.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub struct ByteRange {
|
||||
pub offset: u32,
|
||||
pub size: u32,
|
||||
}
|
||||
|
||||
impl ByteRange {
|
||||
pub fn is_present(&self) -> bool {
|
||||
self.size > 0
|
||||
}
|
||||
|
||||
pub fn end(&self) -> u64 {
|
||||
self.offset as u64 + self.size as u64
|
||||
}
|
||||
|
||||
/// The slice this range describes, bounds-checked against `total`.
|
||||
pub fn slice<'a>(&self, data: &'a [u8], total: usize) -> Result<&'a [u8]> {
|
||||
let start = self.offset as usize;
|
||||
let end = self.end() as usize;
|
||||
if end > total || start > total {
|
||||
return Err(Error {
|
||||
offset: start,
|
||||
context: "level data extent",
|
||||
kind: ErrorKind::OutOfBounds {
|
||||
needed: self.size as usize,
|
||||
available: total.saturating_sub(start),
|
||||
},
|
||||
});
|
||||
}
|
||||
Ok(&data[start..end])
|
||||
}
|
||||
}
|
||||
|
||||
/// A named extent inside the level data.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum LevelDataPart {
|
||||
/// The level overlay executable: Ratchet's code for this planet.
|
||||
Overlay,
|
||||
/// The geometry directory.
|
||||
CoreIndex,
|
||||
/// The Graphics Synthesizer RAM image.
|
||||
GsRam,
|
||||
HudHeader,
|
||||
HudBank(u8),
|
||||
/// The compressed geometry.
|
||||
CoreData,
|
||||
TransitionTextures,
|
||||
}
|
||||
|
||||
/// The parsed level data header.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct LevelDataHeader {
|
||||
pub overlay: ByteRange,
|
||||
pub core_index: ByteRange,
|
||||
pub gs_ram: ByteRange,
|
||||
pub hud_header: ByteRange,
|
||||
pub hud_banks: [ByteRange; 5],
|
||||
pub core_data: ByteRange,
|
||||
pub transition_textures: ByteRange,
|
||||
}
|
||||
|
||||
impl LevelDataHeader {
|
||||
/// Parse the header from the start of the extent.
|
||||
pub fn parse(bytes: &[u8]) -> Result<LevelDataHeader> {
|
||||
let mut b = Buf::new(bytes);
|
||||
let range = |b: &mut Buf<'_>, what: &'static str| -> Result<ByteRange> {
|
||||
let offset = b.u32(what)? as i32;
|
||||
let size = b.u32(what)? as i32;
|
||||
// Negative values mark absent extents; normalise rather than let a
|
||||
// negative offset reach a slice index.
|
||||
let range = ByteRange {
|
||||
offset: if offset < 0 { 0 } else { offset as u32 },
|
||||
size: if size < 0 { 0 } else { size as u32 },
|
||||
};
|
||||
if size < 0 || offset < 0 {
|
||||
// Preserve "absent" as a zero size.
|
||||
return Ok(ByteRange { offset: 0, size: 0 });
|
||||
}
|
||||
Ok(range)
|
||||
};
|
||||
|
||||
let overlay = range(&mut b, "overlay range")?;
|
||||
let core_index = range(&mut b, "core_index range")?;
|
||||
let gs_ram = range(&mut b, "gs_ram range")?;
|
||||
let hud_header = range(&mut b, "hud_header range")?;
|
||||
|
||||
let mut hud_banks = [ByteRange { offset: 0, size: 0 }; 5];
|
||||
for slot in hud_banks.iter_mut() {
|
||||
*slot = range(&mut b, "hud bank range")?;
|
||||
}
|
||||
|
||||
let core_data = range(&mut b, "core_data range")?;
|
||||
let transition_textures = range(&mut b, "transition textures range")?;
|
||||
|
||||
Ok(LevelDataHeader {
|
||||
overlay,
|
||||
core_index,
|
||||
gs_ram,
|
||||
hud_header,
|
||||
hud_banks,
|
||||
core_data,
|
||||
transition_textures,
|
||||
})
|
||||
}
|
||||
|
||||
/// The range for a named part.
|
||||
pub fn part(&self, which: LevelDataPart) -> ByteRange {
|
||||
match which {
|
||||
LevelDataPart::Overlay => self.overlay,
|
||||
LevelDataPart::CoreIndex => self.core_index,
|
||||
LevelDataPart::GsRam => self.gs_ram,
|
||||
LevelDataPart::HudHeader => self.hud_header,
|
||||
LevelDataPart::HudBank(i) => self.hud_banks[i as usize],
|
||||
LevelDataPart::CoreData => self.core_data,
|
||||
LevelDataPart::TransitionTextures => self.transition_textures,
|
||||
}
|
||||
}
|
||||
|
||||
/// Every present extent, named.
|
||||
pub fn present_parts(&self) -> Vec<(LevelDataPart, ByteRange)> {
|
||||
let mut all: Vec<(LevelDataPart, ByteRange)> = vec![
|
||||
(LevelDataPart::Overlay, self.overlay),
|
||||
(LevelDataPart::CoreIndex, self.core_index),
|
||||
(LevelDataPart::GsRam, self.gs_ram),
|
||||
(LevelDataPart::HudHeader, self.hud_header),
|
||||
];
|
||||
for (i, bank) in self.hud_banks.iter().enumerate() {
|
||||
all.push((LevelDataPart::HudBank(i as u8), *bank));
|
||||
}
|
||||
all.push((LevelDataPart::CoreData, self.core_data));
|
||||
all.push((LevelDataPart::TransitionTextures, self.transition_textures));
|
||||
all.into_iter().filter(|(_, r)| r.is_present()).collect()
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Level 0's real header, read from the disc.
|
||||
fn level0_data() -> Vec<u8> {
|
||||
let mut v = vec![0u8; LEVEL_DATA_HEADER_SIZE];
|
||||
let put = |v: &mut Vec<u8>, off: usize, o: u32, s: u32| {
|
||||
v[off..off + 4].copy_from_slice(&o.to_le_bytes());
|
||||
v[off + 4..off + 8].copy_from_slice(&s.to_le_bytes());
|
||||
};
|
||||
put(&mut v, 0x00, 128, 2_751_268);
|
||||
put(&mut v, 0x08, 2_751_424, 31_600);
|
||||
put(&mut v, 0x10, 2_783_040, 518_656);
|
||||
put(&mut v, 0x18, 3_301_696, 4_060);
|
||||
put(&mut v, 0x20, 3_305_792, 44_935);
|
||||
put(&mut v, 0x28, 3_350_784, 138_675);
|
||||
put(&mut v, 0x30, 3_489_472, 16);
|
||||
put(&mut v, 0x38, 3_489_536, 16);
|
||||
put(&mut v, 0x40, 3_489_600, 34_286);
|
||||
put(&mut v, 0x48, 3_523_904, 9_738_678);
|
||||
put(&mut v, 0x50, 13_262_592, 15_328);
|
||||
v
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_the_real_level_zero_header() {
|
||||
let h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
assert_eq!(
|
||||
h.overlay,
|
||||
ByteRange {
|
||||
offset: 128,
|
||||
size: 2_751_268
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.core_index,
|
||||
ByteRange {
|
||||
offset: 2_751_424,
|
||||
size: 31_600
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.gs_ram,
|
||||
ByteRange {
|
||||
offset: 2_783_040,
|
||||
size: 518_656
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.core_data,
|
||||
ByteRange {
|
||||
offset: 3_523_904,
|
||||
size: 9_738_678
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
h.transition_textures,
|
||||
ByteRange {
|
||||
offset: 13_262_592,
|
||||
size: 15_328
|
||||
}
|
||||
);
|
||||
assert_eq!(h.hud_banks[2].size, 16);
|
||||
assert_eq!(h.hud_banks[4].size, 34_286);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extents_are_ordered_and_non_overlapping() {
|
||||
let h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
let parts = h.present_parts();
|
||||
// Each extent begins at or after the previous one ends. Gaps exist --
|
||||
// 28 bytes after the overlay, before `core_index` -- so this is an
|
||||
// ordering check, not a tiling one.
|
||||
let mut cursor = 0u64;
|
||||
for (part, r) in &parts {
|
||||
assert!(
|
||||
r.offset as u64 >= cursor,
|
||||
"{part:?} starts at {} but the previous extent ended at {cursor}",
|
||||
r.offset
|
||||
);
|
||||
cursor = r.end();
|
||||
}
|
||||
assert_eq!(
|
||||
cursor, 13_277_920,
|
||||
"the last extent should end at the total"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_first_extent_starts_after_the_header() {
|
||||
let h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
// The header is 0x58 bytes and the first extent begins at 0x80, so the
|
||||
// 40 bytes between them are unused.
|
||||
assert_eq!(h.overlay.offset, 0x80);
|
||||
assert!(h.overlay.offset as usize > LEVEL_DATA_HEADER_SIZE);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn all_ranges_lie_inside_the_extent() {
|
||||
let h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
// Level 0's data extent is 13,279,232 bytes.
|
||||
let total = 13_279_232usize;
|
||||
for (part, r) in h.present_parts() {
|
||||
assert!(
|
||||
r.end() <= total as u64,
|
||||
"{part:?} ends past the extent: {:?}",
|
||||
r
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn slicing_is_bounds_checked() {
|
||||
let _h = LevelDataHeader::parse(&level0_data()).unwrap();
|
||||
let data = vec![0u8; 1000];
|
||||
// A range that fits.
|
||||
let ok = ByteRange {
|
||||
offset: 10,
|
||||
size: 100,
|
||||
};
|
||||
assert_eq!(ok.slice(&data, 1000).unwrap().len(), 100);
|
||||
// One that does not.
|
||||
let bad = ByteRange {
|
||||
offset: 950,
|
||||
size: 100,
|
||||
};
|
||||
assert!(bad.slice(&data, 1000).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_truncated_header_errors() {
|
||||
let err = LevelDataHeader::parse(&[0u8; 0x20]).unwrap_err();
|
||||
assert!(
|
||||
matches!(err.kind, ErrorKind::OutOfBounds { .. }),
|
||||
"got {err:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn absent_extents_are_zeroed() {
|
||||
let mut v = level0_data();
|
||||
v[0x30..0x38].copy_from_slice(&[0u8; 8]);
|
||||
let h = LevelDataHeader::parse(&v).unwrap();
|
||||
assert_eq!(h.hud_banks[2], ByteRange { offset: 0, size: 0 });
|
||||
assert!(!h.part(LevelDataPart::HudBank(2)).is_present());
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
pub mod buf;
|
||||
pub mod global_wad;
|
||||
pub mod iso9660;
|
||||
pub mod level_data;
|
||||
pub mod level_wad;
|
||||
pub mod wad;
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
# The level data extent
|
||||
|
||||
Target: `SCUS_972.68` v1.01. Implementation: `crates/gc-formats/src/level_data.rs`.
|
||||
|
||||
The `data` extent of each `LEVEL<n>.WAD` is the level's own container. It holds
|
||||
the level's executable, the geometry directory, the Graphics Synthesizer RAM
|
||||
image, the HUD banks and the geometry itself, each at a byte offset declared by
|
||||
a table at the start of the extent.
|
||||
|
||||
## Header
|
||||
|
||||
```text
|
||||
offset size field
|
||||
0x00 8 overlay the level's executable
|
||||
0x08 8 core_index the geometry directory
|
||||
0x10 8 gs_ram Graphics Synthesizer RAM image
|
||||
0x18 8 hud_header
|
||||
0x20 40 hud_banks[5]
|
||||
0x48 8 core_data the compressed geometry
|
||||
0x50 8 transition_textures
|
||||
```
|
||||
|
||||
Every field is `{offset, size}` in **bytes** relative to the start of the
|
||||
extent. The table is 0x58 bytes and the first extent starts at 0x80, leaving 40
|
||||
bytes unused.
|
||||
|
||||
Level 0, measured:
|
||||
|
||||
| Extent | Offset | Size |
|
||||
|---|---:|---:|
|
||||
| overlay | 128 | 2,751,268 |
|
||||
| core_index | 2,751,424 | 31,600 |
|
||||
| gs_ram | 2,783,040 | 518,656 |
|
||||
| hud_header | 3,301,696 | 4,060 |
|
||||
| hud_banks[0..4] | 3,305,792 … | 44,935 / 138,675 / 16 / 16 / 34,286 |
|
||||
| core_data | 3,523,904 | 9,738,678 |
|
||||
| transition_textures | 13,262,592 | 15,328 |
|
||||
|
||||
Extents are ordered and non-overlapping but not gap-free: 28 bytes separate the
|
||||
end of the overlay from the start of `core_index`. The test asserts ordering
|
||||
rather than tiling for that reason.
|
||||
|
||||
## A third convention for sizes
|
||||
|
||||
Three different meanings for a "size" field appear on this disc. All three are
|
||||
byte offsets; the size unit differs:
|
||||
|
||||
| Where | Size unit |
|
||||
|---|---|
|
||||
| `LEVEL<n>.WAD` named extents | **sectors** |
|
||||
| `G/*.WAD` generic pair table | **bytes** |
|
||||
| `data` extent header | **bytes** |
|
||||
|
||||
The first of those was a real bug: the extractor read level extents through the
|
||||
generic path, taking a 6,484-sector extent as 6,484 bytes and losing 13 MB of
|
||||
every level. See `../plan/level-assets.md`.
|
||||
|
||||
## `core_data`: the geometry
|
||||
|
||||
The largest extent, and a WAD LZ77 stream — the format in
|
||||
[`wad.md`](wad.md), with a 16-byte header whose declared size equals the whole
|
||||
extent, i.e. the stream spans it exactly.
|
||||
|
||||
Level 0 decompresses from 9,738,678 to **16,969,296 bytes**, a 1.74× ratio. It
|
||||
contains the four geometry subsystems (`tfrag` terrain, `tie` environment
|
||||
instances, `shrub` foliage, `moby` actors), sky shells, collision and
|
||||
particles.
|
||||
|
||||
## `core_index`: the geometry directory
|
||||
|
||||
A 31,600-byte structure at the head of which sits a header of offsets into
|
||||
`core_data`. On level 0:
|
||||
|
||||
| Field | Value | Lands inside core_data |
|
||||
|---|---:|---|
|
||||
| gs_ram | 404, count 18,704 | yes |
|
||||
| occlusion | 2,117,888 | yes |
|
||||
| sky | 2,192,576 | yes |
|
||||
| collision | 2,615,232 | yes |
|
||||
| moby_classes | 238 | yes |
|
||||
| tie_classes | 74 | yes |
|
||||
| shrub_classes | 7 | yes |
|
||||
| tfrag_textures | 93 | yes |
|
||||
| moby_textures | 168 | yes |
|
||||
| textures_base | 3,518,848 | yes |
|
||||
| part_bank | 8,141,696 | yes |
|
||||
| fx_bank | 8,332,160 | yes |
|
||||
|
||||
Every offset falls inside the 16.9 MB `core_data`, which is the evidence that
|
||||
the header is being read correctly.
|
||||
|
||||
`sun` and `collision` at their offsets begin with plausible structure — the
|
||||
collision block opens `40 00 00 00 50 c8 0d 00`, a plausible count followed by a
|
||||
size.
|
||||
|
||||
**Not yet resolved:** the `ArrayRange` fields (`moby_classes` and friends) do
|
||||
not decode to a usable class table under the documented layout. Offsets and
|
||||
counts read as plausible numbers but the entries they point at are not valid
|
||||
`{offset, o_class}` records. The GC core header layout differs from the
|
||||
documented R&C1 one in a way not yet determined. Recorded in
|
||||
`../plan/level-assets.md`; this is the next thing to resolve.
|
||||
|
||||
## Provenance
|
||||
|
||||
The structure matches `GcUyaLevelDataHeader` in the Wrench project
|
||||
(<https://github.com/chaoticgd/wrench>, GPLv3). **No Wrench code is copied** —
|
||||
see `PROVENANCE.md`. Everything here is written from the specification and
|
||||
verified against all 27 levels on this disc.
|
||||
@@ -3,6 +3,39 @@
|
||||
Phase 2 work record. What has been established about the four extents inside
|
||||
each `LEVEL<n>.WAD`, and what remains.
|
||||
|
||||
## A bug found and fixed
|
||||
|
||||
Reading the `data` extent's contents exposed a defect in the Phase 1 extractor.
|
||||
|
||||
Level extents are declared in **sector counts**; the generic `G/` pair table
|
||||
uses **byte counts**. The extractor routed level containers through the generic
|
||||
path, so `LEVEL0.WAD`'s `data` extent — 6,484 sectors — was taken as 6,484
|
||||
*bytes*. Level 0 yielded a 6,484-byte "geometry" file instead of 13,279,232
|
||||
bytes. Across 27 levels that silently discarded roughly 300 MB of level content
|
||||
while every test still passed.
|
||||
|
||||
The tell was arithmetic, not a crash: the extent's own header declared an
|
||||
uncompressed total of 13,277,920 bytes, and the ratio against the 6,484-byte
|
||||
file was 2047.7× — one sector, to within rounding. A ratio that lands on 2048
|
||||
is a unit error.
|
||||
|
||||
Fixed by extracting level containers through the typed `LevelHeader` instead of
|
||||
the generic table. Level 0 now yields:
|
||||
|
||||
```text
|
||||
data.bin 13,279,232 the level container's data extent
|
||||
core_data.wad 9,738,678 the geometry, compressed
|
||||
core_data.bin 16,969,296 the geometry, decompressed
|
||||
core_index.bin 31,600 the geometry directory
|
||||
gameplay.bin 659,456
|
||||
sound_bank.bin 1,503,232
|
||||
```
|
||||
|
||||
**Lesson:** a unit error is invisible to a type system and to a passing test. The
|
||||
check that catches it is arithmetic against an independent number the file
|
||||
itself declares. Any extent whose contents describe a larger structure should
|
||||
have that structure's own total compared against the extent.
|
||||
|
||||
## Done: the container header
|
||||
|
||||
`crates/gc-formats/src/level_wad.rs` parses the 0x60-byte header on all 27
|
||||
|
||||
+10
-4
@@ -16,10 +16,11 @@ early, while they are still cheap to retire.
|
||||
- [x] **Phase 1 complete.** The disc pipeline runs end to end and is verified
|
||||
against a committed hash table: 5,135 files, 4,500,106,960 bytes, every
|
||||
one of the 92 asset containers parsing.
|
||||
- [x] **Phase 2 started.** The level container header is parsed and typed, which
|
||||
names the `data`, `sound_bank`, `gameplay` and `occlusion` extents inside
|
||||
each of the 27 level containers.
|
||||
- [ ] Phase 2 next: the `gameplay` extent, then `data`.
|
||||
- [x] **Phase 2 in progress.** Level containers are parsed through their typed
|
||||
header, and the `data` extent is parsed as a container of byte ranges. The
|
||||
geometry decompresses: level 0 yields 16,969,296 bytes of level content
|
||||
from a 9.7 MB stream.
|
||||
- [ ] Next: the geometry directory's class tables, then `tfrag` terrain.
|
||||
|
||||
The disc being the exact build the reference projects pin is the single
|
||||
biggest favourable fact so far: no disc conversion, no regional timing work,
|
||||
@@ -101,6 +102,11 @@ Phase 1 hands over the extents; this phase works out what each one contains.
|
||||
|
||||
**In progress:**
|
||||
|
||||
- [x] `data` — the extent header locates `core_index` and `core_data`; the
|
||||
geometry decompresses and is verified on level 0.
|
||||
- [ ] `core_index` class tables — offsets all land inside `core_data`, which
|
||||
confirms the header, but the `ArrayRange` fields do not decode to a usable
|
||||
class table under the documented layout. Next thing to resolve.
|
||||
- [ ] `gameplay` — begins `WAD` + a per-level byte, and does *not* decode with
|
||||
the WAD LZ77 decoder. All 27 levels fail identically, so the input is
|
||||
wrong rather than the decoder. Ruled out: bit order, header length, the
|
||||
|
||||
Reference in New Issue
Block a user