Spurious layout warning at startup

[bkline]

When emacs is launched from the terminal a message is displayed complaining of illegal calls and warning that something may break in the future.

GNU Emacs 28.1
Copyright (C) 2022 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.
% emacs -Q
2022-09-03 09:35:53.118 Emacs-x86_64-10_14[37072:11711478] It's not
legal to call -layoutSubtreeIfNeeded on a view which is already being
laid out.  If you are implementing the view's -layout method, you can
call -[super layout] instead. Break on void
_NSDetectedLayoutRecursion(void) to debug.  This will be logged only
once.  This may break in the future.

Emacs has been installed by brew install --cask emacs which uses builds hosted by https://emacsformacosx.com/. I tried using brew to install the latest nightly build from the same source, but still got the unwanted message at launch time.

I reported the unwanted behavior to the Emacs team, and I was asked to load Emacs in the debugger and run it with a breakpoint at _NSDetectedLayoutRecursion and provide the backtrace, but that attempt failed with the error message "Not allowed to attach to process." The Emacs for Mac OS X site links to this repository as the source for the scripts used to build the artifacts. So I tried using these scripts to build an Emacs I could debug, but the result does not exhibit the unwanted behavior. So I've been asked to inquire here to find out what options are used in building the artifacts installed by homebrew.

[caldwell]

The x86_64 version of emacs is configured with:

./configure --with-ns '--enable-locallisppath=/Library/Application Support/Emacs/${version}/site-lisp:/Library/Application Support/Emacs/site-lisp' --with-modules

I kind of doubt the --enable-locallisppath option would affect that (and probably not --with-modules either).

What MacOS version are you running? Currently the x86_64 binary is build on 10.11 and 10.14 (the launcher checks your os version and launches the most recent one that isn't ahead of you).

Interestingly, I don't get that particular message when I run on my machine (aarch64), but I get a different one instead.

[bkline]

This laptop (owned by my primary client) is running 12.5.1 on x86_64. On my own MacBook, running 12.5.1 on M1, the same version of Emacs (installed with the identical brew command) emits no message at all.

Can you tell me how you code-sign the executables? And do you know what I would need to do in order to get the debugger to be able to attach to the program?

[bkline]

Taking a little detour, I tried a stab in the dark with the theory that perhaps I would be able to use the combine-and-package script to produce an Emacs which exhibits the behavior we're trying to debug. But that was a bust.

% ./combine-and-package -v Emacs-28.1-12-x86_64.tar.bz2 
+ mkdir -p combined
+ rm -rf combined/Emacs.app
+ rm -rf ./Emacs.app
+ tar xf Emacs-28.1-12-x86_64.tar.bz2
+ cp -r ./Emacs.app combined/Emacs.app {:preserve=>true}
+ rm -rf combined/Emacs.app/Contents/MacOS/bin
+ rm -rf combined/Emacs.app/Contents/MacOS/lib
+ rm -rf combined/Emacs.app/Contents/MacOS/libexec
+ cp ./Emacs.app/Contents/MacOS/Emacs combined/Emacs.app/Contents/MacOS/Emacs-x86_64-12
+ cp -r ./Emacs.app/Contents/MacOS/bin combined/Emacs.app/Contents/MacOS/bin-x86_64-12
+ cp -r ./Emacs.app/Contents/MacOS/lib-x86_64-12 combined/Emacs.app/Contents/MacOS/lib-x86_64-12
+ cp -r ./Emacs.app/Contents/MacOS/libexec combined/Emacs.app/Contents/MacOS/libexec-x86_64-12
+ mv combined/Emacs.app/Contents/MacOS/libexec-x86_64-12/Emacs.pdmp combined/Emacs.app/Contents/MacOS/Emacs-x86_64-12.pdmp
+ cp /Users/bkline/repos/build-emacs/launch combined/Emacs.app/Contents/MacOS/Emacs
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1386:in `initialize': No such file or directory @ rb_sysopen - /Users/bkline/repos/build-emacs/launch (Errno::ENOENT)
	from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1386:in `open'
	from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1386:in `copy_file'
	from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:492:in `copy_file'
	from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:419:in `block in cp'
	from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1558:in `block in fu_each_src_dest'
	from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1574:in `fu_each_src_dest0'
	from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:1556:in `fu_each_src_dest'
	from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fileutils.rb:418:in `cp'
	from /Users/bkline/repos/build-emacs/verbose-shell.rb:89:in `method_missing'
	from ./combine-and-package:83:in `combine'
	from ./combine-and-package:158:in `<main>'

[caldwell]

That error is caused by the launch binary missing. You need Rust to compile it from scratch (the source is in the repo) but you should be able to steal the Emacs binary out of an Emacs.app from emacsformacos.com and use that (cp -a path/to/Emacs.app/Contents/MacOS/Emacs ./launch is all it should take). If you do happen to have Rust installed then make should compile up launch.

[bkline]

Back to the ultimate goal: do you know if there's something I can do to the executable installed by https://emacsformacosx.com/ to make it attachable by the debugger?

[caldwell]

Sadly I don't. I think it has to do with the hardened executable stuff that needs to be there for notarization to work, but I don't know how to strip it or bypass it. It's possible that bypassing Gatekeeper (requires a reboot) might let it work.

[caldwell]

Can you tell me how you code-sign the executables? And do you know what I would need to do in order to get the debugger to be able to attach to the program?

The code signing and notarization code is all in combine-and-package, you should be able to look through there and see what Apple binaries it calls to accomplish that. It won't trigger unless you pass the --sign parameter, and have an Apple developer identity certificate ready in a keychain somewhere (an you can't get on of those without paying their $99 developer fee).

[bkline]

It's starting to look like I can build a program I can debug (but which doesn't exhibit the behavior I'm being asked to debug) or I can install a pre-built program which exhibits the unwanted behavior (but can't be debugged) but I can't create a build of the program which will let me debug the code which is emitting the warning. Oh, well, I'm still better off than when I started, with a build which behaves the way I want it to. I'll just have to live with the fact that I can't give the Emacs team the stack trace they were hoping for. Thanks for your help anyway.

[bkline]

I was able to reproduce this on another Intel MacBook, on which I reformatted the hard disk and reinstalled Mac OS (Monterey). The only third-party application I installed was the current build of Emacs from https://emacsformacosx.com. I launched the program from the terminal and got the spurious warning. So we know that if I run Emacs with the Rust launcher I see the problem, and if I build Emacs using the build-emacs-from-tar script and run the result I don't see the problem. It's tempting to wonder if the Rust launcher might be involved in the problem somehow.

[bkline]

I noticed that this bug was first reported in January of this year, which is when the switch to the Rust loader happened (or at least, the git "author" date for that switch is in January, though the "commit" date is later, reflecting subsequent activity—perhaps, for example, rebases—operating on the original commit). Coincidence?

[caldwell]

You can test whether the launcher is involved by launching the binary directly. It's probably one of:

/Applications/Emacs.app/Contents/MacOS/Emacs-arm64-11
/Applications/Emacs.app/Contents/MacOS/Emacs-x86_64-10_14

The launcher binary sits at /Applications/Emacs.app/Contents/MacOS/Emacs and launches the most appropriate one of the compiled binaries.

I kinda doubt it's the launcher—it only interacts with Cocoa apis if there is an error that it needs to show to the user with a dialog. In the normal working operations it shouldn't be hitting any apis that could print that message. That said, it could be related to the way the launcher execs the main binary, so it's certainly worth ruling out (and with software, anything's possible!).

[bkline]

You're right—I can reproduce the bug by running Emacs-x86_64-10_14 directly. So that does indeed rule out the launcher (and it was in fact a coincidence that the switch to the Rust loader coincided with the first reports of this bug). What we know:

• if I build Emacs using your scripts, it's built for Monterey and I don't see the bug
• when you build emacs we get El Capitan and Mojave versions, with the bug appearing

What we (or at least I) don't know:

• if your binaries are run on pre--MacOS 11 machines, is the bug present?
• if you built an Intel version for MacOS 11 (or higher) would be bug still be present on Monterey?

I'll see if I can determine the answer to the first question (if you don't already know) by trying to get a MacBook to install an older version of MacOS. And I suppose that if nothing else I'll find out the answer to the second question when https://emacsformacosx.com/ hosts Intel builds targeted for MacOS 11 or 12.

[bkline]

It seems that Apple is not interested in letting me install an older version of macOS. I tried following instructions on a MacWorld page, but every time I tried the download got to the end and then displayed "Installation failed. An error occurred while installing the selected updates" (even though the instructions I was following were for downloading an installer which could be put onto removable media). So unless you know of users who are running your emacs on the older macOS I probably won't ever know the answer to the first of the two questions in the previous comment. 🤷‍♂️