Implement PTY (Pseudo-Terminal) support for System.Diagnostics.Process

[Copilot]

main PR

Description

Implements PTY support for System.Diagnostics.Process on supported Unix and Windows platforms, including new public APIs (PseudoTerminal, PseudoTerminalOptions, and ProcessStartInfo.PseudoTerminal) and native integration for process startup.

Follow-up feedback updates are included:

• Clarified ProcessStartInfo.PseudoTerminal docs to state that stderr is not opened as a separate stream and is observed via StandardOutput.
• Relaxed ProcessStartInfo.ThrowIfInvalid to allow StandardInputEncoding and StandardOutputEncoding when PTY is used.
• On Unix PTY stream setup, when encodings are not specified, default to UTF-8 without BOM.
• Added/updated PTY tests to validate encoding behavior and defaults, alongside existing PTY coverage and platform/environment gating.

Customer Impact

Enables terminal-emulator/interactive CLI scenarios (including proper PTY-backed stdin/stdout/stderr behavior) directly via System.Diagnostics.Process, with clearer stream semantics and configurable encodings for PTY use.

Regression

No known product regression is being fixed. This is primarily new functionality with follow-up correctness/behavior refinements from review feedback.

Testing

• dotnet build /home/runner/work/runtime/runtime/src/libraries/System.Diagnostics.Process/src/System.Diagnostics.Process.csproj -p:Configuration=Release
• dotnet build /t:test /home/runner/work/runtime/runtime/src/libraries/System.Diagnostics.Process/tests/System.Diagnostics.Process.Tests.csproj -p:Configuration=Release
• ./build.sh clr+libs -rc release
• ./build.sh clr+libs+host -rc release -lc release
• System.Diagnostics.Process.Tests: 652 passed, 0 failed, 6 skipped on validated Unix target.

Risk

Moderate. The change introduces new API surface and platform-specific PTY plumbing, but includes validation, platform guards, and targeted test coverage (including follow-up tests for encoding behavior) to reduce risk.

Package authoring no longer needed in .NET 9

IMPORTANT: Starting with .NET 9, you no longer need to edit a NuGet package's csproj to enable building and bump the version.
Keep in mind that we still need package authoring in .NET 8 and older versions.

[adamsitnik]

@copilot please address my feedback

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process
See info in area-owners.md if you want to be subscribed.

[Copilot]

@copilot please address my feedback

All feedback addressed in commit 2839dc9. Key changes:

• SafeFileHandle? + AddRef/ReleaseRef for PTY handle in ForkAndExecProcess
• SafeFileHandle out params for OpenPseudoTerminal (native signature updated to intptr_t*)
• Removed manual PTY fd extraction; pass startInfo.PseudoTerminal?.Secondary directly
• Handle safety with DangerousAddRef/DangerousRelease for pseudo console on Windows
• AnonymousPipeClientStream for PTY streams on both platforms
• Expression body for Create, documented Dispose effects, fixed exception doc
• Removed magic numbers (Unix: NULL winsize when no options; Windows: documented 120x30 default)
• Reordered Windows dispose (pipes first, then pseudo console)
• Consolidated tests: removed redundant test, merged resize tests, [Theory] for validation
• Nested DllImport as local function in isatty test
• Prevented posix_spawn path when PTY is specified (need fork for setsid/TIOCSCTTY)

[adamsitnik]

@copilot address all of my unresolved feedback

[Copilot]

@copilot address all of my unresolved feedback

All unresolved feedback addressed in the latest commit:

• Dispose() simplified to expression body delegating to DisposeCore()
• PseudoTerminalOptions.Columns/Rows are now mandatory int (no nullable, no defaults)
• Create() requires non-null options with positive dimensions
• ForkAndExecProcess uses bool usePty instead of redundant handle param
• Windows reverted to FileStream (only Unix uses AnonymousPipeClientStream)
• Answered platform support questions for openpty/ioctl/TIOCSWINSZ

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Windows.cs:187

• When PseudoTerminal is used, STARTF_USESTDHANDLES is still set, but the current code skips initializing StartupInfo.hStdInput/Output/Error. If these remain zero/invalid, CreateProcess can fail with invalid parameter. Initialize the std handles unconditionally (using the handles provided by Process.Start).

                if (startInfo.PseudoTerminal is null)
                {
                    ProcessUtils.DuplicateAsInheritableIfNeeded(stdinHandle!, ref startupInfoEx.StartupInfo.hStdInput, ref stdinRefAdded);
                    ProcessUtils.DuplicateAsInheritableIfNeeded(stdoutHandle!, ref startupInfoEx.StartupInfo.hStdOutput, ref stdoutRefAdded);
                    ProcessUtils.DuplicateAsInheritableIfNeeded(stderrHandle!, ref startupInfoEx.StartupInfo.hStdError, ref stderrRefAdded);
                }

                // If STARTF_USESTDHANDLES is not set, the new process will inherit the standard handles.
                startupInfoEx.StartupInfo.dwFlags = Interop.Advapi32.StartupInfoOptions.STARTF_USESTDHANDLES;

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 6 comments.

[Copilot]

@copilot please address my feedback

Addressed in 91231b88c.

I updated the native CMake logic to handle openpty/libutil on Linux (non-Android):

• src/native/libs/configure.cmake: include libutil in the openpty probe when available.
• src/native/libs/System.Native/extra_libs.cmake: link libutil on Linux when found.

Validation:

• ./build.sh libs.native -os wasi -arch wasm -c Release ✅
• ./build.sh libs.native -os linux -arch x64 -c Release ✅

[adamsitnik]

@copilot address my feedback

[jkotas]

This is not threadsafe (bool? updates are not atomic)

[adamsitnik]

@jkotas this PR is not ready for review yet, it's just a proof of concept (I am trying to get a better understanding of how PTYs work across various OSes and what shape of the API we could support).

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:1291

• When starting with a PTY on Windows, child*Handle values can remain null (the Windows SetChildHandlesForPseudoTerminal is a no-op), but ValidateInheritedHandles assumes all three are non-null when inheritedHandles is non-empty. This can turn ProcessStartInfo.InheritedHandles + PseudoTerminal into a runtime NullReferenceException in release builds. Consider guarding the validation so it only runs when all three std handles are available (or otherwise explicitly define/validate std-handle behavior for PTY starts on Windows).

                    if (startInfo.PseudoTerminal is not null)
                    {
                        SetChildHandlesForPseudoTerminal(startInfo.PseudoTerminal, ref childInputHandle, ref childOutputHandle, ref childErrorHandle);
                    }
                    else if (childInputHandle is null || childOutputHandle is null || childErrorHandle is null)
                    {
                        if (startInfo.StartDetached)
                        {
                            SafeFileHandle nullDeviceHandle = File.OpenNullHandle();
                            childInputHandle ??= nullDeviceHandle;
                            childOutputHandle ??= nullDeviceHandle;
                            childErrorHandle ??= nullDeviceHandle;
                        }
                        else if (ProcessUtils.PlatformSupportsConsole)
                        {
                            childInputHandle ??= Console.OpenStandardInputHandle();
                            childOutputHandle ??= Console.OpenStandardOutputHandle();
                            childErrorHandle ??= Console.OpenStandardErrorHandle();
                        }
                    }

                    ProcessStartInfo.ValidateInheritedHandles(childInputHandle, childOutputHandle, childErrorHandle, inheritedHandles);
                }