st can't handle escape codes emitted by vi or nvim

[UtkarshVerma]

When I open vi, I get the following error from st:

erresc: unknown csi ESC[>4;2m

When I open nvim, I get the following error:

erresc: unknown str ESC]8
erresc: unknown csi ESC[>4;2m
erresc: unknown csi ESC[>4;0m
erresc: unknown str ESC]8

How do I go about fixing these or adding support for these escape sequences?

[bakkeby]

https://invisible-island.net/xterm/ctlseqs/ctlseqs.html#h4-Functions-using-CSI-_-ordered-by-the-final-character-lparen-s-rparen:CSI-Ps-SP-q.1D81

If I'm reading this correctly we have these two:

erresc: unknown csi ESC[>4;2m
erresc: unknown csi ESC[>4;0m

mapping to this:

CSI > Pp ; Pv m
CSI > Pp m
          Set/reset key modifier options (XTMODKEYS), xterm.  Set or
          reset resource-values used by xterm to decide whether to
          construct escape sequences holding information about the
          modifiers pressed with a given key.

          The first parameter Pp identifies the resource to set/reset.
          The second parameter Pv is the value to assign to the
          resource.

          If the second parameter is omitted, the resource is reset to
          its initial value.  Values 3  and 5  are reserved for keypad-
          keys and string-keys.

            Pp = 0  ⇒  modifyKeyboard
            Pp = 1  ⇒  modifyCursorKeys
            Pp = 2  ⇒  modifyFunctionKeys
            Pp = 4  ⇒  modifyOtherKeys

          If no parameters are given, all resources are reset to their
          initial values.

both having a Pp of 4 meaning modify other keys.

https://invisible-island.net/xterm/manpage/xterm.html#VT100-Widget-Resources:modifyOtherKeys

modifyOtherKeys (class ModifyOtherKeys)
               Like modifyCursorKeys, tells xterm to construct an escape
               sequence for ordinary (i.e., "other") keys (such as "2") when
               modified by Shift-, Control-, Alt- or Meta-modifiers.  This
               feature does not apply to special keys, i.e., cursor-, keypad-,
               function- or control-keys which are labeled on your keyboard.
               Those have key symbols which XKB identifies uniquely.

               For example, this feature does not apply to special control-
               keys (e.g., Escape, Tab, Enter, Backspace) Other control keys
               (e.g., Control-I, Control-M, Control-H) may send escape
               sequences when this feature is enabled.

               The default is "0":

               0    disables this feature.

               1    enables this feature for keys except for those with well-
                    known behavior, e.g., Tab, Backarrow and some special
                    control character cases which are built into the X11
                    library, e.g., Control-Space to make a NUL, or Control-3
                    to make an Escape character.

                    Except for those special cases built into the X11 library,
                    the Shift- and Control- modifiers are treated normally.
                    The Alt- and Meta- modifiers do not cause xterm to send
                    escape sequences.  Those modifier keys are interpreted
                    according to other resources, e.g., the metaSendsEscape
                    resource.

               2    enables this feature for keys including the exceptions
                    listed.  Xterm ignores the special cases built into the
                    X11 library.  Any shifted (modified) ordinary key sends an
                    escape sequence.  The Alt- and Meta- modifiers cause xterm
                    to send escape sequences.

               The Xterm FAQ has an extended discussion of this feature, with
               examples:

               https://invisible-island.net/xterm/modified-keys.html

So it sounds like when you enter nvim it wants to enable this feature and it tries to disable it afterwards when you exit nvim.

It sounds xterm specific. If you wanted to you could add dummy case statements that just ignores the escape codes.

This was also raised in #63 but the reporter just ended up commenting out the csidump() call that prints these debug lines.

Then we have:

erresc: unknown str ESC]8

This looks like an operating system command.

ESC ]
     Operating System Command (OSC  is 0x9d).

It is not entirely clear what ]8 means though. On the pages above the above does not match the patterns on the pages referred to earlier.

The escape code is also not listed here:

• https://gist.github.com/fnky/458719343aabd01cfb17a3a4f7296797

but https://man7.org/linux/man-pages/man4/console_codes.4.html suggests that this may be a VT100 console sequence.

       The user can configure xterm(1) to respond to VT220-specific
       control sequences, and it will identify itself as a VT52, VT100,
       and up depending on the way it is configured and initialized.

       It accepts ESC ] (OSC) for the setting of certain resources.

That said ESC ] 8 is not listed in the few OSC control sequences recognized by xterm. So maybe it is something that is not even supported by xterm.

This page for hterm suggests that the escape code may be used in relation to setting a hyperlink.
https://chromium.googlesource.com/apps/libapps/+/nassh-0.8.41/hterm/doc/ControlSequences.md

As there are no additional parameters I wonder if it asks for the hyperlink (if any exists) or if it suggests that the hyperlink should be removed if one exists. This is pure speculation on my part of course.

[UtkarshVerma]

Thanks for your detailed, answer. I was able to find the documentation in NVIM readme for the ESC[>4;2m sequence. Also, I don't get the ESC]8 sequence anymore so we could drop it.

						*tui-modifyOtherKeys* *tui-csiu*
Historically, terminal emulators could not distinguish between certain control
key modifiers and other keys. For example, <C-I> and <Tab> are represented the
same way, as are <Esc> and <C-[>, <CR> and <C-M>, and <NL> and <C-J>. This
meant that Nvim also could not map these keys separately.

Modern terminal emulators are able to distinguish between these pairs of keys
by encoding control modifiers differently. There are two common but distinct
ways of doing this, known as "modifyOtherKeys" and "CSI u". Nvim supports both
encoding methods and at startup will tell the terminal emulator that it
understands these key encodings. If your terminal emulator supports it then
this will allow you to map the key pairs listed above separately.

At startup Nvim will query your terminal to see if it supports the CSI u
encoding by writing the sequence >

	CSI ? u CSI c

If your terminal emulator responds with >

	CSI ? <flags> u

this means your terminal supports the CSI u encoding and Nvim will tell your
terminal to enable it by writing the sequence >

	CSI > 1 u

If your terminal does not support CSI u then Nvim will instead enable the
"modifyOtherKeys" encoding by writing the sequence >

	CSI > 4 ; 2 m

When Nvim exits cleanly it will send the corresponding sequence to disable the
special key encoding. If Nvim does not exit cleanly then your terminal
emulator could be in a bad state. If this happens, simply run "reset".

Based on this documentation, do you think something could be done to st for it to behave properly with NVIM?

[bakkeby]

So this can be summarised as:

• historically terminal emulators were not able to distinguish between certain control key modifiers and other keys; for example Ctrl+i and Tab would be represented the same way
• modern terminal emulators are typically able to distinguish between pairs of keys by encoding control modifiers differently
• there are two common but distinct ways of doing this:
  • CSI u
  • modifyOtherKeys
• nvim supports both encoding methods
• on startup it will tell the terminal emulator that it supports these encodings
• it will first check if the terminal emulator supports CSI u
• if the terminal emulator does not support CSI u then it will fall back to enable modifyOtherKeys by sending CSI > 4 ; 2 m
• the st terminal emulator supports neither methods of distinguishing between control key modifier and it will log that it received an escape sequence that it doesn't support (csi ESC[>4;2m)
• when nvim exits it will try to disable modifyOtherKeys by sending CSI > 4 ; 0 m
• the st terminal emulaotr will again log that it received an escape sequence that it doesn't support (csi ESC[>4;0m)

Good to have all this information in one place.

do you think something could be done to st for it to behave properly with NVIM?

I think the question is whether it is possible to implement CSI u in st. This is likely a question for the maintainers.

The assumption here is of course that nvim misbehaves in some way when used in st due to not supporting CSI u, in which case, and as far as I understand, means that the program can't distinguish between certain key combinations like backspace and shift+backspace.

Would be good to have some practical demonstrations showing that some key combinations work differently in other terminal emulators but do the same thing in st.

I tried to find some details that are specifically on the CSI u escape sequence:

• http://www.leonerd.org.uk/hacks/fixterms/
• https://www.github.com/microsoft/terminal/issues/8719
• https://iterm2.com/documentation-csiu.html
• https://sw.kovidgoyal.net/kitty/keyboard-protocol/
• https://www.github.com/alacritty/alacritty/pull/5358

and modifyOtherKeys:

• https://www.github.com/alacritty/alacritty/issues/3101
• https://news.ycombinator.com/item?id=27219279

If I come across better references then I'll add them above.

[UtkarshVerma]

Thanks for the detailed answer. In my case, I am actually using the CSI u encoding as I have manually added a couple of mappings in config.h for convenience. What do I have to add to st to make it advertise that it supports CSI u encodings?

[bakkeby]

I am actually using the CSI u encoding as I have manually added a couple of mappings in config.h for convenience

Not sure if that is the same thing. That st supports CSI u (or modifyOtherKeys for that matter) would imply that the terminal is able to switch modes where it interprets key combinations in different ways.

But let's leave that aside and pretend that st supports CSI u mode. I have no idea how you do that. I suspect that it may have to do with st.info which is used to generate files under ~/.terminfo/. If we look at the CSI 22, 23 patch for example then that also adds escape codes to that file.

According to the nvim documentation you added they send the escape sequence of CSI ? u CSI c and I am not sure exactly how / where that is handled. I'd have assumed that this would go through the csihandle function, but perhaps not.

As for getting rid of the modifyOtherKeys errors of

erresc: unknown csi ESC[>4;2m
erresc: unknown csi ESC[>4;0m

it is enough to just add a dummy case statement in csihandle. It doesn't actually do anything of course.

	case '>':
		break;

[UtkarshVerma]

Thanks for the suggestion. I'll add the case as you have suggested in csihandle. I had a look at nvim's source code and found that it is relying on this:

https://sourcegraph.com/github.com/neovim/neovim/-/blob/src/nvim/tui/tui.c?L1877&subtree=true
https://sw.kovidgoyal.net/kitty/keyboard-protocol/#detection-of-support-for-this-protocol

Would it be possible for me to handle this in a better way? Something worth adding a patch for?

[bakkeby]

I don't know. If you are able to copy the protocol and key handling from kitty then you may be on to something.