[new release] eio (5 packages) (1.4)

[talex5]

Effect-based direct-style IO API for OCaml

• Project page: https://github.com/ocaml-multicore/eio
• Documentation: https://ocaml-multicore.github.io/eio/

CHANGES:

Breaking changes:

• Report errors from Net.getaddrinfo (@talex5 @haesbaert @avsm Report errors from getaddrinfo ocaml-multicore/eio#866 Net.getaddrinfo: include the query in error messages ocaml-multicore/eio#900).
  Before, Net.getaddrinfo returned [] on error. It now raises an exception with the actual error.

New features:

• Add Eio.Net.Sockopt for setting/getting socket options (@avsm @talex5 @anmonteiro Initial support for setting/getting socket options ocaml-multicore/eio#858 add Eio_unix.Sockopt for setting/getting socket options ocaml-multicore/eio#575 Add portable names for common network options ocaml-multicore/eio#859 eio_posix, eio_windows: map ENOPROTOOPT to Invalid_option ocaml-multicore/eio#874 Be more careful about unterminated string socket options ocaml-multicore/eio#876 Move sockopt stubs to eio.unix ocaml-multicore/eio#889 Raise Eio exceptions from sockopt functions in all cases ocaml-multicore/eio#891).
  This includes all the options from OCaml's Unix module, plus many more (e.g. TCP_KEEPINTVL).

• Support TCP_KEEPIDLE socket option on macOS (@avsm @talex5 Support TCP_KEEPIDLE on macos ocaml-multicore/eio#890).

• Add Path.chmod (@patricoferris @webbunivAdmin @avsm @talex5 Support chmod in Eio_linux and Eio_posix ocaml-multicore/eio#785 chmod: prevent sandbox escape using symlinks ocaml-multicore/eio#872).

• Add Path.chown (@patricoferris @avsm @talex5 Support chown in eio_posix and eio_linux ocaml-multicore/eio#781 use the right flags for chown and add a linux/other test ocaml-multicore/eio#864 chown: prevent sandbox escape using symlinks ocaml-multicore/eio#877).

• Add Path.with_dir_entries for incremental directory reading (@patricoferris @talex5 @avsm Incremental directory reading ocaml-multicore/eio#821 eio_linux: don't return an empty getdents batch before end-of-file ocaml-multicore/eio#897).
  This also provides the type of each entry, saving one stat per entry in most cases.

• Add Eio_unix.Pty for pseudoterminal support (@avsm @talex5 @patricoferris @RyanGibb support for creating ptys and a login_tty fork_action ocaml-multicore/eio#531).
  The Pty module can be used to create and configure pseudoterminals,
  Eio_unix.Process.spawn_unix ?login_tty ... allows connecting them up when spawning a child process.

• Add missing_ok to Eio.Path.unlink (Add missing_ok to Eio.Path.unlink ocaml-multicore/eio#828) (@vog @patricoferris Add missing_ok to Eio.Path.unlink (#828) ocaml-multicore/eio#833).

• Add Flow.null sink/source (@avsm @talex5 add null sink that consumes/discards all data written to it until eof ocaml-multicore/eio#894, suggested by @jonsterling).
  This can be used as sink that consumes and discards all data written to it until end-of-file,
  and as a source that immediately returns end-of-file.

• Add Eio_unix.Process.spawn_unix ?uid ?gid ?pgid options (@patricoferris @talex5 Add process groups to unix backends ocaml-multicore/eio#802 Add setuid and setgid fork action ocaml-multicore/eio#803).
  Allows setting the user ID, group ID and process group of the child.

• Add Eio_unix.Stdenv.override for updating environments (@patricoferris @talex5 Add Eio_unix.Stdenv.override for updating environments ocaml-multicore/eio#823).

• Add Eio_unix.Err module (@talex5 Add Eio_unix.Err module ocaml-multicore/eio#887).
  This is convenient whenever wrapping a function that raises Unix_error.

Linux backend:

• eio_linux now requires Linux 5.18 or later. On older kernels, Eio_main.run will use eio_posix instead.

• Add Low_level.{socket,bind,listen} with uring support (@avsm @talex5 eio_linux: add Low_level.{socket,bind,listen} with uring support ocaml-multicore/eio#884).

• Add ftruncate/fdatasync and use uring for fsync/shutdown (@avsm @talex5 add ftruncate/fdatasync and use uring for fsync/shutdown/socket/bind/listen ocaml-multicore/eio#850).

• Switch fstat to statx on linux and fetch blksize (@avsm @talex5 switch fstat to statx on linux and fetch blksize ocaml-multicore/eio#865).

• uring: update min to 2.15.0+ so we can use new functionality (@avsm uring: update min to 2.15.0+ so we can use new functionality ocaml-multicore/eio#867).

• Add Eio_linux.Low_level.Fixed (@talex5 Add Eio_linux.Low_level.Fixed ocaml-multicore/eio#848).
  This replaces the Uring.Region API.

• eio_linux: use IORING_SETUP_SINGLE_ISSUER (@talex5 eio_linux: use IORING_SETUP_SINGLE_ISSUER ocaml-multicore/eio#841).
  This hints to uring to do some locking optimisations assuming that each ring is only written by one thread.

• eio_linux: use IORING_SETUP_DEFER_TASKRUN for performance (@talex5 eio_linux: use "IORING_SETUP_DEFER_TASKRUN" for performance ocaml-multicore/eio#842).

Bug fixes:

• Allow Eio.Path.load to handle files with zero size (@patricoferris @talex5 Allow Eio.Path.load to handle files with zero size ocaml-multicore/eio#869 Use a sensible minimum buffer size for Path.load ocaml-multicore/eio#878, reported by @copy and @vog).
  This is useful for loading files in /proc/ on Linux.

• Eio.Process: match Unix search behaviour (@talex5 Eio.Process: match Unix search behaviour ocaml-multicore/eio#820).
  Don't search $PATH if the name contains a /.

• eio_posix: on macOS, fall back to select() for FDs where poll() fails (@avsm @talex5 eio_posix: fall back to select() for fds that poll() fails with POLLNVAL ocaml-multicore/eio#893, reported by @aaronjeline).
  macOS poll() returns POLLNVAL for some device FDs such as /dev/tty or /dev/null.

• eio_posix: wait for a writer when first reading from a FIFO (@talex5 eio_posix: wait for a writer when first reading from a FIFO ocaml-multicore/eio#857).
  Before, it would return end-of-file if there was no waiter.

• eio_posix: open confined paths in non-blocking mode too (@talex5 eio_posix: open confined paths in non-blocking mode too ocaml-multicore/eio#855).
  Previously, paths opened using fs were opened in non-blocking mode, but paths opened with cwd were not.
  This probably only affects FIFOs.

• eio_posix: use MSG_CMSG_CLOEXEC when receiving file descriptors (@talex5 @avsm eio_posix: use MSG_CMSG_CLOEXEC when receiving file descriptors ocaml-multicore/eio#843).

• eio_linux: read_upto and write_single didn't hold the FD open (@avsm eio_linux: these read/write functions didnt hold the fd open ocaml-multicore/eio#871).
  This could be a problem if another fiber closed the FD before the operation was submitted.

• eio_linux: handle ENOMEM from io_uring_queue_init (@talex5 eio_linux: handle ENOMEM from io_uring_queue_init ocaml-multicore/eio#852).
  We would already fall back to eio_posix if there wasn't enough lockable memory to add a fixed buffer,
  but didn't cope with not having enough memory for the ring itself.

• fork_action: retry on EINTR when writing to pipe (@avsm fork_action: retry on eintr when writing to pipe ocaml-multicore/eio#870).

• eio_windows: fix removing of FDs from read/write sets (@z-rui @talex5 eio_windows: fix removing of FDs from read/write sets ocaml-multicore/eio#861).
  Could cause high CPU use in some cases.

• eio_windows: remove unused unix_cstruct.ml file (@talex5 Remove unused unix_cstruct.ml file ocaml-multicore/eio#817, reported by @dijkstracula).
  Caused an error if linking with -linkall.

Minor changes:

• Propagate errno back from fork_actions into Eio.Process (@Innf107 @avsm @talex5 Propagate errno back from fork_actions into Eio.Process ocaml-multicore/eio#854).
  Report Unix fork_action failures as Eio exceptions instead of Failure.
  This allows matching on specific errors (e.g. Argument_list_too_long).

• Alias Path.open_dir to open_subtree (@talex5 @avsm Rename Path.open_dir to open_subtree ocaml-multicore/eio#853).
  This more accurately describes what it's for.

• Prioritize returning Fiber.any value over cancelling quickly (@adamchol @talex5 Prioritize returning Fiber.any value over cancelling quickly ocaml-multicore/eio#806).
  This works better with the new combine option.

• Relax return types of Eio.Process.Pipe (@Arogbonlo @patricoferris @talex5 Relax return types of Eio.Process.Pipe ocaml-multicore/eio#775, requested by @rizo).

• Eio.Pool.use: move never_block argument (@talex5 Eio.Pool.use: move never_block argument ocaml-multicore/eio#747).

• eta expand some function applications to support OxCaml (@avsm eta expand function applications to support oxcaml ocaml-multicore/eio#898).

• fork_action: reject ids that don't fit uid_t/gid_t/pid_t (@avsm fork_action: reject ids that don't fit uid_t/gid_t/pid_t ocaml-multicore/eio#895).

Documentation:

• eio_unix: minor typo in Unix_error docstring (@avsm eio_unix: minor typo in Unix_error docstring ocaml-multicore/eio#888).

• Simplify custom "env" type in example (@vog Simplify custom "env" type in example ocaml-multicore/eio#832).

• Fix docs for without_binding (@patricoferris Fix docs for without_binding ocaml-multicore/eio#822).

• Clarify Eio as a possible concurrency library (@patricoferris Clarify Eio as a possible concurrency library ocaml-multicore/eio#879).

Code cleanups:

• Add braces around switch case block (@olafhering add braces around switch case block ocaml-multicore/eio#882).

• eio_linux: use Uring.Res module for safer error handling (@talex5 @avsm eio_linux: use Uring.Res module for safer error handling ocaml-multicore/eio#868).

• eio_linux: move read/write retries out of scheduler (@talex5 @avsm eio_linux: move read/write retries out of scheduler ocaml-multicore/eio#849).

• eio_linux: split out net.ml, process.ml, fs.ml and time.ml (@talex5 eio_linux: split out net.ml, process.ml, fs.ml and time.ml ocaml-multicore/eio#886).
  This more closely matches the layout in eio_posix.

• eio_linux: remove unused code (@talex5 eio_linux: remove unused code ocaml-multicore/eio#851).
  uring should never ask us to retry a job.

• Remove all use of [@@@alert "-unstable"] (@talex5 Remove all use of [@@@alert "-unstable"] ocaml-multicore/eio#885).
  This was needed a long time ago to work around a bug in Merlin.

• Remove some test files after running the tests (@avsm Remove some test files after running the tests ocaml-multicore/eio#883).

• win32: declare unixsupport functions as CAMLextern (@avsm @dra27 win32: declare unixsupport functions as CAMLextern ocaml-multicore/eio#881).
  This is necessary under Windows to prevent flexdll errors.

• Remove unused show_backend_exceptions flag (@talex5 Remove unused show_backend_exceptions flag ocaml-multicore/eio#880).

• Improve formatting of IP addresses in tests (@talex5 Improve formatting of IP addresses in tests ocaml-multicore/eio#847).

• Add lintcstubs-arity as a test dependency (@talex5 Add lintcstubs-arity as a test dependency ocaml-multicore/eio#846).

• Update Dockerfiles to recent versions (@talex5 Update Dockerfiles to recent versions ocaml-multicore/eio#844).

• Skip test_alloc_fixed_or_wait if no fixed buffers are available (@talex5 Skip test_alloc_fixed_or_wait if no fixed buffers are available ocaml-multicore/eio#815).
  Prevents spurious CI failures.

• Make eio_unix_fork_error call _exit automatically (@talex5 Make eio_unix_fork_error call _exit automatically ocaml-multicore/eio#896).

[talex5]

Summary of CI failures:

• Windows: eio_posix doesn't install because iomux doesn't. Probably iomux should be marked as not available on Windows, but for now I just marked eio_posix as unavailable there.
• cohttp-eio: needs an upper bound (fixed upstream in cohttp-eio: update tests for new Eio 1.4 getaddrinfo exception mirage/ocaml-cohttp#1142)
• eio-ssl: needs upper bound (upstream PR: Add {get,set}sockopt functions for Eio 1.4 anmonteiro/eio-ssl#26 @anmonteiro)
• irky-eio, piaf, smaws-clients and smaws-lib depend on eio-ssl.
• ocluster and playwright: errors look unrelated

Also, several platforms failed for an unrelated reason:

"/usr/bin/env" "bash" "-c" "opam init --reinit --config .opamrc-sandbox -ni" failed with exit status 10

I've added the necessary upper-bounds and the tests are running again...

[talex5]

The new test results look good to me - I think this is ready to merge.

The new failures are:

• ubuntu-24.04-ocaml-5.5/riscv64 : "Linux >= 5.18 is required for io_uring support" (expected and OK)
• ez_api.3.0.0 : was only tested due to upper-bound addition in cohttp-eio, and so presumably unrelated (looks like it's using depopts so might or might not be tested against Eio depending on other packages).

(BTW, it would be good to add a log analyser rule for "... failed with exit status" so that you can spot those easily; at the moment they all just show up as "Build failed" in the CI summary)

[avsm]

Thanks!