Skip to content

14-the-case-problem.md

Latest commit

 

History

History
261 lines (209 loc) · 9.89 KB

14-the-case-problem.md

File metadata and controls

261 lines (209 loc) · 9.89 KB
Raw

14 — the case problem, four malformed files, and seventeen called - Copy

Measure: all 4,769 file names from notes/sha1-all.txt, extensions recorded as spelt; all 1,321 data files parsed with a strict XML parser; every - Copy file diffed against its original.

the extension, spelt two ways

$ awk -F'\t' '{print $4}' notes/sha1-all.txt | sort | uniq -c | sort -rn
   2323 .meta
   1150 .XML
   1035 .bk2
    176 .xml

1,150 files are spelt .XML and 176 are spelt .xml. One of the 176 is Mono's mconfig.xml, outside the game's data; the other 175 are in ExternalData, mixed in with the 1,150 in the same two directories:

directory .XML .xml
Data/CutsceneSubtitlesData 976 83
Data/Conversations 174 81
Data/GUI/* 0 6
Localization/Strings 0 5
total under ExternalData 1,150 175

Plus one .txt: CutsceneSubtitlesData/StGermaine_1_IA_Tex6_PIECE_A.txt, 279 bytes, which is a CaptionData file that lost its extension:

<CaptionData>
    <caption>
        <stringID>StGermaine_1_IA_Tex6_PIECE_A</stringID>
        <timeStart>0</timeStart><timeEnd>4</timeEnd>
    </caption>
</CaptionData>

Same shape as its neighbours, wrong extension. On Windows it does not matter, because the game builds stem + '.xml' and NTFS does not care about case — and because if the loader globs the directory, .txt will not be in the glob at all and one caption never appears.

why this matters, and it is not pedantry

The code builds file names by concatenation (12). Among the assembly's literals:

'/ExternalData/Data/CutsceneSubtitlesData/'
'.xml'

Lower case. So on a case-sensitive filesystem — Linux, which is what Proton on a Steam Deck runs — a request for Mojo_1_18.xml will not find Mojo_1_18.XML, and 1,150 of 1,325 caption files are spelt the second way. Whether that breaks anything depends on whether the runtime layer folds case, which is a question about Proton's filesystem shim and not about this object. What this object contributes is the count: 1,150 files at risk, 175 not.

The same trap reaches the keys, not just the file names:

CutsceneSubtitlesData/Resort_SC87_3pt13_3K.XML   Resort_SC87_3pt13_3K_PIECE_A
Conversations/Rook_AA_7yrGap.xml                 Rook_AA_7yrGap_Piece_B

_PIECE_ and _Piece_. Case-folding the closure test in 07 recovers 16 lines in every one of the five languages — 16 subtitles whose appearance depends on whether a dictionary lookup is case-sensitive.

how this repository handled it, and the wound it is handling

Two sessions ago, in pc-megaman3-doc, verifying that a .gitignore written for upper-case extensions also caught lower-case ones, the session created test files with lower-case names and then removed them with rm -f. The filesystem is case-insensitive. rm -f mm.exe deleted MM.EXE, the real executable, and rm -f dyna.scn deleted DYNA.SCN. Two of that object's twenty-nine files were destroyed by the command that was verifying they would be ignored.

So this session's .gitignore was verified with no file created and no file removed, on paths that do not exist:

$ git check-ignore -v --no-index \
      TeslaEffect.exe teslaeffect.exe foo.bk2 FOO.BK2 bar.XML bar.xml \
      x.meta X.META x.assets x.resS level0 mainData output_log.txt \
      appmanifest_261510.acf DefaultWsdlHelpGenerator.aspx Tesla_Promo_1.ogv \
      notes/sha1-all.txt notes/census.txt README.md docs/00-predictions.md \
      tools/census.py _work/anything.bin prompt.txt
.gitignore:23:*.[Ee][Xx][Ee]    TeslaEffect.exe
.gitignore:23:*.[Ee][Xx][Ee]    teslaeffect.exe
.gitignore:17:*.[Bb][Kk]2       foo.bk2
.gitignore:17:*.[Bb][Kk]2       FOO.BK2
.gitignore:18:*.[Xx][Mm][Ll]    bar.XML
.gitignore:18:*.[Xx][Mm][Ll]    bar.xml
...
.gitignore:47:!notes/*.TXT      notes/sha1-all.txt
.gitignore:12:_work/            _work/anything.bin

README.md, docs/00-predictions.md and tools/census.py produce no output, which is the correct answer: they are not ignored. No file was created, no file was deleted, and the object is on a drive nothing in this session wrote to.

Every rule is a bracket class — *.[Xx][Mm][Ll], *.[Bb][Kk]2, *.[Mm][Ee][Tt][Aa] — rather than a bare extension, so the rules do not depend on core.ignorecase being true. The one place they do is the exception !notes/*.txt, which is why it is written twice, once for each case.

four files that are not XML

$ python - <<'…'   # ET.parse on all 1,321 data files
data files parsed         1321
not well-formed XML          4
   !! CutsceneSubtitlesData/Mantus_2_Sc_Cinematic.XML       mismatched tag: line 113
   !! CutsceneSubtitlesData/Rook_AA_PhotoMLeonard_triggered.XML  mismatched tag: line 4
   !! CutsceneSubtitlesData/StGermaine_2_4 .XML             no element found: line 7
   !! CutsceneSubtitlesData/Zack_1_3c .XML                  mismatched tag: line 15

Zack_1_3c .XML, entire:

<CaptionData>
                <caption>
                                <stringID>Zack_1_3c_PIECE_A</stringID>
                                <timeStart>0</timeStart><timeEnd>3</timeEnd>
                </caption>
                 <caption>
                                <stringID>Zack_1_3c_PIECE_B</stringID>
                                <timeStart>3</timeStart><timeEnd>6</timeEnd>
                </caption>
                                <stringID>Zack_1_3c_PIECE_C</stringID>
                                <timeStart>6</timeStart><timeEnd>12</timeEnd>
                </caption>
</CaptionData>

The third caption lost its opening tag, leaving an unbalanced </caption>. That is the file behind the <caption> 3,889 versus <stringID> 3,890 discrepancy in 07.

Four files of 1,321 are rejected by a conforming XML parser, which means the game's loader is not one — it is lenient, or it is a regex. tools/closure.py reads these files with a regular expression for exactly that reason, and says so in its docstring: 1,321 files whose entire grammar is three tags do not need a parser, and one of them is not even called .xml.

three names with a space in the wrong place

StGermaine_1_4 .XML      273 B
StGermaine_2_4 .XML      273 B
Zack_1_3c .XML           709 B

A trailing space in the stem. On Windows these are hard to create by accident and hard to type; the shell in this session broke on one of them until the tooling stopped globbing and started walking with os.walk.

And four with a duplicate suffix:

StGermaine_2_4  (2).XML          273 B
StGermaine_2_4  (3).XML          273 B
StGermaine_2_5D8_PIECE_A (2).XML 767 B
StGermaine_2_9G (2).XML          519 B

StGermaine_2_4 .XML, StGermaine_2_4 (2).XML and StGermaine_2_4 (3).XML — note the two spaces in the last two — are three copies of a 273-byte file, one of which is the one that fails to parse.

seventeen files called - Copy, and six of them are not copies

$ grep -ci " - copy" notes/sha1-all.txt
34

Seventeen XML files and their seventeen .meta. All seventeen are Mojo_1_*:

Mojo_1_10F  Mojo_1_11F  Mojo_1_12F  Mojo_1_18   Mojo_1_19I
Mojo_1_1B   Mojo_1_20I  Mojo_1_21I  Mojo_1_22   Mojo_1_23
Mojo_1_25K  Mojo_1_26L  Mojo_1_2B_PIECE_A       Mojo_1_3B
Mojo_1_A    Mojo_1_H    Mojo_1_J

(The briefing said eighteen. It is seventeen; see 19.)

One person, one afternoon, one conversation, selecting seventeen files in Windows Explorer and pressing Ctrl+C, Ctrl+V. The - Copy suffix is Explorer's and nothing else produces it.

Diffing each against its original is where it gets interesting:

identical to the original   11
different                    6
original missing             0

Six of the seventeen are not backups. They are older versions.

Mojo_1_18  vs  Mojo_1_18 - Copy
    -<stringID>Mojo_1_18_PIECE_A</stringID>
    +<stringID>Mojo_1_18</stringID>

Mojo_1_22  vs  Mojo_1_22 - Copy
    -<stringID>Mojo_1_22_PIECE_A</stringID>
    +<stringID>Mojo_1_22</stringID>

Mojo_1_J  vs  Mojo_1_J - Copy
    -<stringID>Mojo_1_J_PIECE_B</stringID>   +<stringID>Mojo_1_J_PIECE_A</stringID>
    -<timeEnd>7</timeEnd>                    +<timeEnd>1</timeEnd>
    -<stringID>Mojo_1_J_PIECE_C</stringID>   +<stringID>Mojo_1_J_PIECE_B</stringID>
    -<stringID>Mojo_1_J_PIECE_D</stringID>   +<stringID>Mojo_1_J_PIECE_C</stringID>

The - Copy versions predate the _PIECE_A naming convention and predate a caption being renumbered. They are a snapshot taken before an edit, kept in the directory the game ships, for two years. Version control implemented as Ctrl+C.

And every one of the seventeen has its own .meta with its own distinct GUID — so at some point the Unity editor saw all seventeen, imported them, and gave each an identity. Nothing rejected them, because nothing was checking.

The same hand appears once more, in 06: four .meta files in the object share a GUID with another .meta, and all four are the Ault_5A / Ault_5BC pair — a folder copied with its sidecars, which is the one thing a Unity project must never do.

the total cost of all of it

17 XML + 17 .meta named "- Copy"          8,164 + 1,581 =  9,745 B
 4 duplicate-suffix XML + their 4 .meta   1,832 +   372 =  2,204 B
 3 trailing-space XML                                       1,255 B
 1 .txt among the .XML                                        279 B
                                                    total  13,483 B

13,483 bytes. Nothing, against 17.5 gigabytes, and that is exactly why they are still there: nobody's disk-space alarm was ever going to go off over nine kilobytes of subtitle backups sitting next to a 560-megabyte video file. They shipped because the thing that would have caught them — a build step that looks at the data directory — did not exist, and 13 KB is under every threshold anybody would have set.