IBM290 mismatching Unicode code points between RFC 1345 and this version

[Rot127]

The Unicode mapping for IBM290 is weird.

RFC 1345 has a different mapping for cp290 than the mapping used here.

The RFC 1345 version (search for IBM290 or here) maps characters to the Katakana, U+30A0 - U+30FF block. While this implementation maps it to the Halfwidth and Fullwidth Forms, U+FF00 - U+FFEF block. Now, which one is it?

It doesn't seem to be documented.

[roskakori]

Generally, the mappings are the same as the ones Java uses. As far as I understand, the IBM mappings mostly stem from IBM themselves. The Encoding documentation from Oracle states that IBM290 is intended to represent:

IBM Japanese Katakana Host Extended SBCS

I am not quite sure what that means.

According to Wikipedia, Katagana has about 48 characters and a few diacritic marks, which technically would fit in an 8 bit encoding.

So all I can offer you here is to go to the source and ask the people maintaining the Java encoding.

[roskakori]

The source mapping from which the Java IBM290 is generated, seems to be located in make/data/charsetmapping/IBM290.map.

@Rot127 Does the Python encoding differ from this one?

[Rot127]

Does the Python encoding differ from this one?

The string encoding used in the Python script is not so much the issue. If the correct Unicode Katagana characters are used (U+FF00-U+FFEF) everything encodes fine.

But there is also the specific "Katakana" Unicode block.. If those ones are in the script and encoded, an exception is thrown.

I think the Katagana block U+30A0 was added maybe later? No idea why they didn't map it to it.
But it is a little in-intuitive to use. Maybe adding a note in the documentation (that only the U+FF00-U+FFEF characters are mapped) is enough.

[roskakori]

I think I understand now. This seems to be a unicode normalization issue. As example take U+FF74. According to its detailed description, this represents the halfwidth form of ｴ, normalized to a single codepoint. There also is a decoomposed form, which takes the fullwidth エ (U+30A8) and adds the "narrow" modifier.

This means in order to process data with IBM290, you need to normalize them properly. I'm not sure how to do that for Katagana, but the unicodedata.normalize() function should be a good starting point. For example:

>>> import unicodedata
>>> ascii(unicodedata.normalize("NFD", "\uff74"))
"'\\uff74'"
>>> ascii(unicodedata.normalize("NFKD", "\uff74"))
"'\\u30a8'"
>>> unicodedata.decomposition("\uff74")
'<narrow> 30A8'

This is a general issue one needs to be aware of when processing Unicode data. For example, the German umlaut "ü" also has different representations depending on the normalization used:

>>> ascii(unicodedata.normalize("NFC", "ü"))
"'\\xfc'"
>>> ascii(unicodedata.normalize("NFD", "ü"))
"'u\\u0308'"

[Rot127]

Nice! I wasn't aware of the unicodedata.normalize() method.
I think it is more then enough to mentioned this in the docs on https://pypi.org/project/ebcdic/.
In the example section.

For me the annoying part was not that it isn't handled (I understand why not), but finding out what the problem was.

[jack980517]

Correct me if I'm wrong, but RFC 1345 doesn't seem to be a definition of code pages. Instead, it's defining "mnemonics", and the code page-to-mnemonic tables are only being given as a reference.

I take that to mean that, it's not defining code points like https://ftp.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/SHIFTJIS.TXT is; instead the mnemonics represent "characters" in a human-facing (i.e. instead of "computer-facing") way. That is to say, U+1D400 𝐀 (MATHEMATICAL BOLD CAPITAL A) would be considered the same "character" as U+0041 A (LATIN CAPITAL LETTER A).

Supporting my interpretation is the fact that, the string "width" doesn't appear in RFC 1345 at all. So it's not like that RFC 1345 specified that half width katakanas should be mapped to other code pages and not IBM 290; rather, the difference between full-width and half-width is simply not considered at all.

All this is to say, the premise that "RFC 1345 maps IBM 290 to full-width katakana instead of half-width katakana" is probably wrong.

SBCS means "single byte character set", in contrast to DBCS (double) or MBCS (multi). If you look at the aforementioned SHIFTJIS.TXT (or the half width-only JIS0201.TXT), you'll see that single-byte katakana is mapped to half width. This is the norm on DOS/Windows. I think (which is to say, I don't have proof) that half width was developed first to accommodate older computers which at first was Latin-only and therefore single-byte- and half-width-only; double-byte and full-width (which actually looks like how the characters should be written on paper) came later. So it would be strange if a single-byte encoding maps katakana to full-width.

[jack980517]

The converter on unicode.org maps IBM 290 to half-width characters (U+FFxx). This should be the authoritative source on the matter.

[roskakori]

If you look at the aforementioned SHIFTJIS.TXT (or the half width-only JIS0201.TXT), you'll see that single-byte katakana is mapped to half width. This is the norm on DOS/Windows. I think (which is to say, I don't have proof) that half width was developed first to accommodate older computers which at first was Latin-only and therefore single-byte- and half-width-only; double-byte and full-width (which actually looks like how the characters should be written on paper) came later. So it would be strange if a single-byte encoding maps katakana to full-width.

I've left this issue open because I am still not sure how to phrase a small section about this in the README.

Any proposals?