pty: preserve O_NONBLOCK when opening /dev/ptmxHEADmain

ptmx_fdopen() masks the supplied file flags to FREAD | FWRITE before
passing them to pts_alloc(). This discards FNONBLOCK even when the
caller explicitly requests O_NONBLOCK.

As a result, open("/dev/ptmx", O_RDWR | O_NOCTTY | O_NONBLOCK) succeeds,
but the returned file is blocking. F_GETFL reports O_NONBLOCK cleared,
and a subsequent read can block instead of returning EAGAIN when no
data is available.

This causes Linux xfce4-terminal/VTE running under Linuxulator to
freeze. VTE opens the PTY master with O_NONBLOCK and relies on
nonblocking reads while draining terminal output. Once the available
data has been consumed, the main UI thread blocks in ptsdev_read(),
leaving the window unresponsive even though the child shell has
started successfully.

Include FNONBLOCK in the flags passed to pts_alloc(). Blocking opens
retain their existing behavior. No change to posix_openpt(2) flag
validation is required.

The flag loss also reproduces with a native FreeBSD program, so the
problem is in the shared /dev/ptmx implementation rather than Linux
syscall flag translation.

Validation:
- Before the fix, a native FreeBSD reproducer observes O_NONBLOCK
  cleared after opening /dev/ptmx with that flag.
- Explicitly setting O_NONBLOCK with F_SETFL works, confirming that
  nonblocking PTY operation itself is supported.
- After the fix, F_GETFL preserves O_NONBLOCK when requested and
  leaves it cleared for a blocking open.
- xfce4-terminal in the Linux jail displays the shell prompt and
  accepts input.

Reviewed by:	kib
MFC after:	1 week
Differential revision:	https://reviews.freebsd.org/D60700

0 files changed, 0 insertions, 0 deletions