BOLT: BOLT is unable to proceed because it couldn't properly understand this function

[itrofimow]

There are some similar issues, but I've decided to open a separate one since I have more context here, probably.

After #117299 commit BOLT fails to create a profile for one of my binaries:

itrofimow@perforator-dev:~/bolt_test$ ~/work/llvm-project/build/bin/llvm-bolt ./binary --perfdata profile.pa --pa --aggregate-only --profile-format yaml -o profile.yaml --thread-count 16
PERF2BOLT: Starting data aggregation job for profile.pa
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 7530e707afc60014624eb62bda9557cdc7f28dd0
BOLT-INFO: first alloc address is 0x200000
BOLT-INFO: creating new program header table at address 0x28a00000, offset 0x28800000
BOLT-INFO: enabling relocation mode
BOLT-INFO: enabling strict relocation mode for aggregation purposes
...
... a bunch of likely unrelated BOLT-INFO/BOLT-WARNING message
...
BOLT-WARNING: reference in the middle of instruction detected in function __KeccakF1600/1(*2) at offset 0x33
BOLT-ERROR: cannot find BB containing branch destination.
FATAL BOLT-ERROR: =======================================
BOLT is unable to proceed because it couldn't properly understand this function.
If you are running the most recent version of BOLT, you may want to report this and paste this dump.
Please check that there is no sensitive contents being shared in this dump.

Offending function: v1_RC4_options

Function contents (
  0000: 488D0559 0000008B 15D36BC7 070FBAE2  |H..Y......k.....|
  0010: 14720C0F BAE21E73 0A4883C0 19F3C348  |.r.....s.H.....H|
  0020: 83C00CF3 C3662E0F 1F840000 00000066  |.....f.........f|
  0030: 2E0F1F84 00000000 00662E0F 1F840000  |.........f......|
  0040: 00000066 2E0F1F84 00000000 00662E0F  |...f.........f..|
  0050: 1F840000 00000066 0F1F8400 00000000  |.......f........|
  0060: 72633428 38782C69 6E742900 72633428  |rc4(8x,int).rc4(|
  0070: 38782C63 68617229 00726334 28313678  |8x,char).rc4(16x|
  0080: 2C696E74 29005243 3420666F 72207838  |,int).RC4 for x8|
  0090: 365F3634 2C204352 5950544F 47414D53  |6_64, CRYPTOGAMS|
  00A0: 20627920 3C617070 726F406F 70656E73  | by <appro@opens|
  00B0: 736C2E6F 72673E00 662E0F1F 84000000  |sl.org>.f.......|
  00C0: 0000662E 0F1F8400 00000000 662E0F1F  |..f.........f...|
  00D0: 84000000 0000662E 0F1F8400 00000000  |......f.........|
)

Binary Function "v1_RC4_options"  {
  Number      : 31778
  State       : disassembled
  Address     : 0x20aa6420
  Size        : 0xe0
  MaxSize     : 0xe0
  Offset      : 0x208a6420
  Section     : .text
  Orc Section : .local.text.v1_RC4_options
  LSDA        : 0x0
  IsSimple    : 1
  IsMultiEntry: 1
  IsSplit     : 0
  BB Count    : 17
  Secondary Entry Points : __ENTRY_v1_RC4_options@0x20aa6480
}
DWARF CFI Instructions:
    <empty>
End of Function "v1_RC4_options"

ERROR: disassembly failed - inconsistent branch found.

---

The offending function is from openssl1.1.1g: https://github.com/openssl/openssl/blob/e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72/crypto/rc4/asm/rc4-x86_64.pl#L527
I assume it's some template language there in openssl repo, in its final form (at least what we have vendored) it looks like this

.globl	RC4_options
.type	RC4_options,@function
.align	16
RC4_options:
.cfi_startproc	
	leaq	.Lopts(%rip),%rax
	movl	OPENSSL_ia32cap_P(%rip),%edx
	btl	$20,%edx
	jc	.L8xchar
	btl	$30,%edx
	jnc	.Ldone
	addq	$25,%rax
	.byte	0xf3,0xc3
.L8xchar:
	addq	$12,%rax
.Ldone:
	.byte	0xf3,0xc3
.cfi_endproc	
.align	64
.Lopts:
.byte	114,99,52,40,56,120,44,105,110,116,41,0
.byte	114,99,52,40,56,120,44,99,104,97,114,41,0
.byte	114,99,52,40,49,54,120,44,105,110,116,41,0
.byte	82,67,52,32,102,111,114,32,120,56,54,95,54,52,44,32,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0
.align	64
.size	RC4_options,.-RC4_options

and it disassembles to

(lldb) dis -a 0x20aa6420 -b
binary`v1_RC4_options:
binary[0x20aa6420] <+0>:   48 8d 05 59 00 00 00           leaq   0x59(%rip), %rax ; <+96>
binary[0x20aa6427] <+7>:   8b 15 d3 6b c7 07              movl   0x7c76bd3(%rip), %edx ; v1_OPENSSL_ia32cap_P
binary[0x20aa642d] <+13>:  0f ba e2 14                    btl    $0x14, %edx
binary[0x20aa6431] <+17>:  72 0c                          jb     0x20aa643f     ; <+31>
binary[0x20aa6433] <+19>:  0f ba e2 1e                    btl    $0x1e, %edx
binary[0x20aa6437] <+23>:  73 0a                          jae    0x20aa6443     ; <+35>
binary[0x20aa6439] <+25>:  48 83 c0 19                    addq   $0x19, %rax
binary[0x20aa643d] <+29>:  f3 c3                          rep    retq
binary[0x20aa643f] <+31>:  48 83 c0 0c                    addq   $0xc, %rax
binary[0x20aa6443] <+35>:  f3 c3                          rep    retq
binary[0x20aa6445] <+37>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa644f] <+47>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa6459] <+57>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa6463] <+67>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa646d] <+77>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa6477] <+87>:  66 0f 1f 84 00 00 00 00 00     nopw   (%rax,%rax)
binary[0x20aa6480] <+96>:  72 63                          jb     0x20aa64e5     ; <+197>
binary[0x20aa6482] <+98>:  34 28                          xorb   $0x28, %al
binary[0x20aa6484] <+100>: 38 78 2c                       cmpb   %bh, 0x2c(%rax)
binary[0x20aa6487] <+103>: 69 6e 74 29 00 72 63           imull  $0x63720029, 0x74(%rsi), %ebp ; imm = 0x63720029 
binary[0x20aa648e] <+110>: 34 28                          xorb   $0x28, %al
binary[0x20aa6490] <+112>: 38 78 2c                       cmpb   %bh, 0x2c(%rax)
binary[0x20aa6493] <+115>: 63 68 61                       movslq 0x61(%rax), %ebp
binary[0x20aa6496] <+118>: 72 29                          jb     0x20aa64c1     ; <+161>
binary[0x20aa6498] <+120>: 00 72 63                       addb   %dh, 0x63(%rdx)
binary[0x20aa649b] <+123>: 34 28                          xorb   $0x28, %al
binary[0x20aa649d] <+125>: 31 36                          xorl   %esi, (%rsi)
binary[0x20aa649f] <+127>: 78 2c                          js     0x20aa64cd     ; <+173>
binary[0x20aa64a1] <+129>: 69 6e 74 29 00 52 43           imull  $0x43520029, 0x74(%rsi), %ebp ; imm = 0x43520029 
binary[0x20aa64a8] <+136>: 34 20                          xorb   $0x20, %al
binary[0x20aa64aa] <+138>: 66 6f                          outsw  (%rsi), %dx
binary[0x20aa64ac] <+140>: 72 20                          jb     0x20aa64ce     ; <+174>
binary[0x20aa64ae] <+142>: 78 38                          js     0x20aa64e8     ; <+200>
binary[0x20aa64b0] <+144>: 36 5f                          popq   %rdi
binary[0x20aa64b2] <+146>: 36 34 2c                       xorb   $0x2c, %al
binary[0x20aa64b5] <+149>: 20 43 52                       andb   %al, 0x52(%rbx)
binary[0x20aa64b8] <+152>: 59                             popq   %rcx
binary[0x20aa64b9] <+153>: 50                             pushq  %rax
binary[0x20aa64ba] <+154>: 54                             pushq  %rsp
binary[0x20aa64bb] <+155>: 4f 47 41 4d 53                 pushq  %r11
binary[0x20aa64c0] <+160>: 20 62 79                       andb   %ah, 0x79(%rdx)
binary[0x20aa64c3] <+163>: 20 3c 61                       andb   %bh, (%rcx,%riz,2)
binary[0x20aa64c6] <+166>: 70 70                          jo     0x20aa6538     ; __KeccakF1600 + 56
binary[0x20aa64c8] <+168>: 72 6f                          jb     0x20aa6539     ; __KeccakF1600 + 57
binary[0x20aa64ca] <+170>: 40 6f                          outsl  (%rsi), %dx
binary[0x20aa64cc] <+172>: 70 65                          jo     0x20aa6533     ; __KeccakF1600 + 51
binary[0x20aa64ce] <+174>: 6e                             outsb  (%rsi), %dx
binary[0x20aa64cf] <+175>: 73 73                          jae    0x20aa6544     ; __KeccakF1600 + 68
binary[0x20aa64d1] <+177>: 6c                             insb   %dx, %es:(%rdi)
binary[0x20aa64d2] <+178>: 2e 6f                          outsl  %cs:(%rsi), %dx
binary[0x20aa64d4] <+180>: 72 67                          jb     0x20aa653d     ; __KeccakF1600 + 61
binary[0x20aa64d6] <+182>: 3e 00 66 2e                    addb   %ah, %ds:0x2e(%rsi)
binary[0x20aa64da] <+186>: 0f 1f 84 00 00 00 00 00        nopl   (%rax,%rax)
binary[0x20aa64e2] <+194>: 66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa64ec] <+204>: 66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa64f6] <+214>: 66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)

---

Also, BOLT fails to exit gracefully and crashes:

 #0 0x0000564e0470204d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x29cd04d)
 #1 0x0000564e047025fb PrintStackTraceSignalHandler(void*) Signals.cpp:0:0
 #2 0x0000564e04700156 llvm::sys::RunSignalHandlers() (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x29cb156)
 #3 0x0000564e04703725 SignalHandler(int) Signals.cpp:0:0
 #4 0x00007f4abd57a520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #5 0x00007f4abd5ce9fc pthread_kill (/lib/x86_64-linux-gnu/libc.so.6+0x969fc)
 #6 0x00007f4abd57a476 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x42476)
 #7 0x00007f4abd5607f3 abort (/lib/x86_64-linux-gnu/libc.so.6+0x287f3)
 #8 0x0000564e04693504 llvm::report_fatal_error(llvm::Twine const&, bool) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x295e504)
 #9 0x0000564e047070b0 llvm::detail::scope_exit<std::decay<llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::optional<unsigned int>)::$_0>::type> llvm::make_scope_exit<llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::optional<unsigned int>)::$_0>(llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::optional<unsigned int>)::$_0&&) Threading.cpp:0:0
#10 0x0000564e04707144 llvm::llvm_thread_join_impl(unsigned long) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x29d2144)
#11 0x0000564e046fc9dc llvm::thread::join() CrashRecoveryContext.cpp:0:0
#12 0x0000564e0536108d llvm::StdThreadPool::~StdThreadPool() (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x362c08d)
#13 0x0000564e053612f9 llvm::StdThreadPool::~StdThreadPool() (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x362c2f9)
#14 0x0000564e0535f40c std::default_delete<llvm::ThreadPoolInterface>::operator()(llvm::ThreadPoolInterface*) const ParallelUtilities.cpp:0:0
#15 0x0000564e0535c9e3 std::unique_ptr<llvm::ThreadPoolInterface, std::default_delete<llvm::ThreadPoolInterface>>::~unique_ptr() ParallelUtilities.cpp:0:0
#16 0x00007f4abd57d495 (/lib/x86_64-linux-gnu/libc.so.6+0x45495)
#17 0x00007f4abd57d610 on_exit (/lib/x86_64-linux-gnu/libc.so.6+0x45610)
#18 0x0000564e04908022 llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&)::operator()(llvm::bolt::BOLTError const&) const RewriteInstance.cpp:0:0
#19 0x0000564e04907f77 llvm::Error llvm::ErrorHandlerTraits<void (&)(llvm::bolt::BOLTError&)>::apply<llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&)>(llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&)&&, std::unique_ptr<llvm::ErrorInfoBase, std::default_delete<llvm::ErrorInfoBase>>) RewriteInstance.cpp:0:0
#20 0x0000564e04907e8a llvm::Error llvm::handleErrorImpl<llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&)>(std::unique_ptr<llvm::ErrorInfoBase, std::default_delete<llvm::ErrorInfoBase>>, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&)&&) RewriteInstance.cpp:0:0
#21 0x0000564e04907df3 llvm::Error llvm::handleErrors<llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&)>(llvm::Error, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&)&&) RewriteInstance.cpp:0:0
#22 0x0000564e04907bb0 void llvm::handleAllErrors<llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&)>(llvm::Error, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&)&&) RewriteInstance.cpp:0:0
#23 0x0000564e04907ab7 llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&, unsigned short) const RewriteInstance.cpp:0:0
#24 0x0000564e04907a58 void std::__invoke_impl<void, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&, llvm::bolt::BinaryFunction&, unsigned short>(std::__invoke_other, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&, llvm::bolt::BinaryFunction&, unsigned short&&) RewriteInstance.cpp:0:0
#25 0x0000564e049079f5 std::enable_if<is_invocable_r_v<void, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&, llvm::bolt::BinaryFunction&, unsigned short>, void>::type std::__invoke_r<void, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&, llvm::bolt::BinaryFunction&, unsigned short>(llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&, llvm::bolt::BinaryFunction&, unsigned short&&) RewriteInstance.cpp:0:0
#26 0x0000564e0490791d std::_Function_handler<void (llvm::bolt::BinaryFunction&, unsigned short), llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0>::_M_invoke(std::_Any_data const&, llvm::bolt::BinaryFunction&, unsigned short&&) RewriteInstance.cpp:0:0
#27 0x0000564e0535e968 std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>::operator()(llvm::bolt::BinaryFunction&, unsigned short) const ParallelUtilities.cpp:0:0
#28 0x0000564e0535db6f llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0::operator()(std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned short) const ParallelUtilities.cpp:0:0
#29 0x0000564e0535f048 void std::__invoke_impl<void, llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, unsigned int&>(std::__invoke_other, llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, unsigned int&) ParallelUtilities.cpp:0:0
#30 0x0000564e0535ef6d std::__invoke_result<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, unsigned int&>::type std::__invoke<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, unsigned int&>(llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>&, unsigned int&) ParallelUtilities.cpp:0:0
#31 0x0000564e0535ef37 void std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0 (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned int)>::__call<void, 0ul, 1ul, 2ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul>) ParallelUtilities.cpp:0:0
#32 0x0000564e0535eea6 void std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0 (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned int)>::operator()<void>() ParallelUtilities.cpp:0:0
#33 0x0000564e0535ee75 void std::__invoke_impl<void, std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0 (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned int)>&>(std::__invoke_other, std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0 (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned int)>&) ParallelUtilities.cpp:0:0
#34 0x0000564e0535ee35 std::enable_if<is_invocable_r_v<void, std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0 (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned int)>&>, void>::type std::__invoke_r<void, std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0 (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned int)>&>(std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0 (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned int)>&) ParallelUtilities.cpp:0:0
#35 0x0000564e0535ec3d std::_Function_handler<void (), std::_Bind<llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function<void (llvm::bolt::BinaryFunction&, unsigned short)>, std::function<bool (llvm::bolt::BinaryFunction const&)>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, bool, unsigned int)::$_0 (std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, std::_Rb_tree_iterator<std::pair<unsigned long const, llvm::bolt::BinaryFunction>>, unsigned int)>>::_M_invoke(std::_Any_data const&) ParallelUtilities.cpp:0:0
#36 0x0000564e0493f515 std::function<void ()>::operator()() const DWARFRewriter.cpp:0:0
#37 0x0000564e0493f4d5 void std::__invoke_impl<void, std::function<void ()>>(std::__invoke_other, std::function<void ()>&&) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2c0a4d5)
#38 0x0000564e0493f495 std::__invoke_result<std::function<void ()>>::type std::__invoke<std::function<void ()>>(std::function<void ()>&&) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2c0a495)
#39 0x0000564e0493f46d void std::thread::_Invoker<std::tuple<std::function<void ()>>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) DWARFRewriter.cpp:0:0
#40 0x0000564e0493f415 std::thread::_Invoker<std::tuple<std::function<void ()>>>::operator()() DWARFRewriter.cpp:0:0
#41 0x0000564e0493f3dc std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>::operator()() const DWARFRewriter.cpp:0:0
#42 0x0000564e0493f2f0 std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter> std::__invoke_impl<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>&>(std::__invoke_other, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>&) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2c0a2f0)
#43 0x0000564e0493f278 std::enable_if<is_invocable_r_v<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>&>, std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>>::type std::__invoke_r<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>&>(std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>&) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2c0a278)
#44 0x0000564e0493f190 std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>>::_M_invoke(std::_Any_data const&) DWARFRewriter.cpp:0:0
#45 0x0000564e04848e88 std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const JITLinkLinker.cpp:0:0
#46 0x0000564e04848be9 std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) JITLinkLinker.cpp:0:0
#47 0x0000564e04848e1f void std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2b13e1f)
#48 0x0000564e04848d8d std::__invoke_result<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>::type std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2b13d8d)
#49 0x0000564e04848d54 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::'lambda'()::operator()() const JITLinkLinker.cpp:0:0
#50 0x0000564e04848d24 std::once_flag::_Prepare_execution::_Prepare_execution<void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::'lambda'()>(void (std::__future_base::_State_baseV2::*&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*))::'lambda'()::operator()() const JITLinkLinker.cpp:0:0
#51 0x0000564e04848cf1 std::once_flag::_Prepare_execution::_Prepare_execution<void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::'lambda'()>(void (std::__future_base::_State_baseV2::*&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*))::'lambda'()::__invoke() JITLinkLinker.cpp:0:0
#52 0x00007f4abd5d1ee8 (/lib/x86_64-linux-gnu/libc.so.6+0x99ee8)
#53 0x0000564e04848c9b __gthread_once(int*, void (*)()) JITLinkLinker.cpp:0:0
#54 0x0000564e04848b99 void std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2b13b99)
#55 0x0000564e04848955 std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) JITLinkLinker.cpp:0:0
#56 0x0000564e04940e2b std::__future_base::_Deferred_state<std::thread::_Invoker<std::tuple<std::function<void ()>>>, void>::_M_complete_async() DWARFRewriter.cpp:0:0
#57 0x0000564e0484a3bd std::__future_base::_State_baseV2::wait() JITLinkLinker.cpp:0:0
#58 0x0000564e049414aa std::__basic_future<void>::wait() const DWARFRewriter.cpp:0:0
#59 0x0000564e04941475 std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()::operator()() const DWARFRewriter.cpp:0:0
#60 0x0000564e04941455 void std::__invoke_impl<void, std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()&>(std::__invoke_other, std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()&) DWARFRewriter.cpp:0:0
#61 0x0000564e04941415 std::enable_if<is_invocable_r_v<void, std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()&>, void>::type std::__invoke_r<void, std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()&>(std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()&) DWARFRewriter.cpp:0:0
#62 0x0000564e0494128d std::_Function_handler<void (), std::shared_future<void> llvm::ThreadPoolInterface::asyncImpl<void>(std::function<void ()>, llvm::ThreadPoolTaskGroup*)::'lambda'()>::_M_invoke(std::_Any_data const&) DWARFRewriter.cpp:0:0
#63 0x0000564e0493f515 std::function<void ()>::operator()() const DWARFRewriter.cpp:0:0
#64 0x0000564e05360202 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x362b202)
#65 0x0000564e05363cd5 llvm::StdThreadPool::grow(int)::$_0::operator()() const ThreadPool.cpp:0:0
#66 0x0000564e05363c59 auto void llvm::thread::GenericThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*)::'lambda'(auto&&, auto&&...)::operator()<llvm::StdThreadPool::grow(int)::$_0&>(auto&&, auto&&...) const ThreadPool.cpp:0:0
#67 0x0000564e05363c2d auto std::__invoke_impl<void, void llvm::thread::GenericThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*)::'lambda'(auto&&, auto&&...), llvm::StdThreadPool::grow(int)::$_0&>(std::__invoke_other, void llvm::thread::GenericThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*)::'lambda'(auto&&, auto&&...)&&, llvm::StdThreadPool::grow(int)::$_0&) ThreadPool.cpp:0:0
#68 0x0000564e05363bdd std::__invoke_result<auto, auto...>::type std::__invoke<void llvm::thread::GenericThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*)::'lambda'(auto&&, auto&&...), llvm::StdThreadPool::grow(int)::$_0&>(auto&&, auto&&...) ThreadPool.cpp:0:0
#69 0x0000564e05363bad decltype(auto) std::__apply_impl<void llvm::thread::GenericThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*)::'lambda'(auto&&, auto&&...), std::tuple<llvm::StdThreadPool::grow(int)::$_0>&, 0ul>(auto&&, std::tuple<llvm::StdThreadPool::grow(int)::$_0>&, std::integer_sequence<unsigned long, 0ul>) ThreadPool.cpp:0:0
#70 0x0000564e05363b3d decltype(auto) std::apply<void llvm::thread::GenericThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*)::'lambda'(auto&&, auto&&...), std::tuple<llvm::StdThreadPool::grow(int)::$_0>&>(auto&&, std::tuple<llvm::StdThreadPool::grow(int)::$_0>&) ThreadPool.cpp:0:0
#71 0x0000564e05363b0e void llvm::thread::GenericThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
#72 0x0000564e05363835 void* llvm::thread::ThreadProxy<std::tuple<llvm::StdThreadPool::grow(int)::$_0>>(void*) ThreadPool.cpp:0:0
#73 0x00007f4abd5ccac3 (/lib/x86_64-linux-gnu/libc.so.6+0x94ac3)
#74 0x00007f4abd65e850 (/lib/x86_64-linux-gnu/libc.so.6+0x126850)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Aborted

for which I don't really care that much, but sharing with you guys anyway since I have the stacktrace symbolized.

---

I do care for the "BOLT is unable to proceed because it couldn't properly understand this function" error, however:
we at Perforator (https://github.com/yandex/perforator) are working on generating BOLT-profiles, and we've got it working with llvm-bolt-19,
but this error with llvm-bolt-20 is not only frightening, but also manifests itself with a lot of binaries we have, which blocks upgrading to llvm-20 for us.

I would be happy to provide any additional info/cooperate on this, if needed

[llvmbot]

@llvm/issue-subscribers-bolt

Author: None (itrofimow)

Details

There are some similar issues, but I've decided to open a separate one since I have more context here, probably.

After #117299 commit BOLT fails to create a profile for one of my binaries:

itrofimow@<!-- -->perforator-dev:~/bolt_test$ ~/work/llvm-project/build/bin/llvm-bolt ./binary --perfdata profile.pa --pa --aggregate-only --profile-format yaml -o profile.yaml --thread-count 16
PERF2BOLT: Starting data aggregation job for profile.pa
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: 7530e707afc60014624eb62bda9557cdc7f28dd0
BOLT-INFO: first alloc address is 0x200000
BOLT-INFO: creating new program header table at address 0x28a00000, offset 0x28800000
BOLT-INFO: enabling relocation mode
BOLT-INFO: enabling strict relocation mode for aggregation purposes
...
... a bunch of likely unrelated BOLT-INFO/BOLT-WARNING message
...
BOLT-WARNING: reference in the middle of instruction detected in function __KeccakF1600/1(*2) at offset 0x33
BOLT-ERROR: cannot find BB containing branch destination.
FATAL BOLT-ERROR: =======================================
BOLT is unable to proceed because it couldn't properly understand this function.
If you are running the most recent version of BOLT, you may want to report this and paste this dump.
Please check that there is no sensitive contents being shared in this dump.

Offending function: v1_RC4_options

Function contents (
  0000: 488D0559 0000008B 15D36BC7 070FBAE2  |H..Y......k.....|
  0010: 14720C0F BAE21E73 0A4883C0 19F3C348  |.r.....s.H.....H|
  0020: 83C00CF3 C3662E0F 1F840000 00000066  |.....f.........f|
  0030: 2E0F1F84 00000000 00662E0F 1F840000  |.........f......|
  0040: 00000066 2E0F1F84 00000000 00662E0F  |...f.........f..|
  0050: 1F840000 00000066 0F1F8400 00000000  |.......f........|
  0060: 72633428 38782C69 6E742900 72633428  |rc4(8x,int).rc4(|
  0070: 38782C63 68617229 00726334 28313678  |8x,char).rc4(16x|
  0080: 2C696E74 29005243 3420666F 72207838  |,int).RC4 for x8|
  0090: 365F3634 2C204352 5950544F 47414D53  |6_64, CRYPTOGAMS|
  00A0: 20627920 3C617070 726F406F 70656E73  | by &lt;appro@<!-- -->opens|
  00B0: 736C2E6F 72673E00 662E0F1F 84000000  |sl.org&gt;.f.......|
  00C0: 0000662E 0F1F8400 00000000 662E0F1F  |..f.........f...|
  00D0: 84000000 0000662E 0F1F8400 00000000  |......f.........|
)

Binary Function "v1_RC4_options"  {
  Number      : 31778
  State       : disassembled
  Address     : 0x20aa6420
  Size        : 0xe0
  MaxSize     : 0xe0
  Offset      : 0x208a6420
  Section     : .text
  Orc Section : .local.text.v1_RC4_options
  LSDA        : 0x0
  IsSimple    : 1
  IsMultiEntry: 1
  IsSplit     : 0
  BB Count    : 17
  Secondary Entry Points : __ENTRY_v1_RC4_options@<!-- -->0x20aa6480
}
DWARF CFI Instructions:
    &lt;empty&gt;
End of Function "v1_RC4_options"

ERROR: disassembly failed - inconsistent branch found.

---

The offending function is from openssl1.1.1g: https://github.com/openssl/openssl/blob/e2e09d9fba1187f8d6aafaa34d4172f56f1ffb72/crypto/rc4/asm/rc4-x86_64.pl#L527
I assume it's some template language there in openssl repo, in its final form (at least what we have vendored) it looks like this

.globl	RC4_options
.type	RC4_options,@<!-- -->function
.align	16
RC4_options:
.cfi_startproc	
	leaq	.Lopts(%rip),%rax
	movl	OPENSSL_ia32cap_P(%rip),%edx
	btl	$20,%edx
	jc	.L8xchar
	btl	$30,%edx
	jnc	.Ldone
	addq	$25,%rax
	.byte	0xf3,0xc3
.L8xchar:
	addq	$12,%rax
.Ldone:
	.byte	0xf3,0xc3
.cfi_endproc	
.align	64
.Lopts:
.byte	114,99,52,40,56,120,44,105,110,116,41,0
.byte	114,99,52,40,56,120,44,99,104,97,114,41,0
.byte	114,99,52,40,49,54,120,44,105,110,116,41,0
.byte	82,67,52,32,102,111,114,32,120,56,54,95,54,52,44,32,67,82,89,80,84,79,71,65,77,83,32,98,121,32,60,97,112,112,114,111,64,111,112,101,110,115,115,108,46,111,114,103,62,0
.align	64
.size	RC4_options,.-RC4_options

and it disassembles to

(lldb) dis -a 0x20aa6420 -b
binary`v1_RC4_options:
binary[0x20aa6420] <+0>:   48 8d 05 59 00 00 00           leaq   0x59(%rip), %rax ; <+96>
binary[0x20aa6427] <+7>:   8b 15 d3 6b c7 07              movl   0x7c76bd3(%rip), %edx ; v1_OPENSSL_ia32cap_P
binary[0x20aa642d] <+13>:  0f ba e2 14                    btl    $0x14, %edx
binary[0x20aa6431] <+17>:  72 0c                          jb     0x20aa643f     ; <+31>
binary[0x20aa6433] <+19>:  0f ba e2 1e                    btl    $0x1e, %edx
binary[0x20aa6437] <+23>:  73 0a                          jae    0x20aa6443     ; <+35>
binary[0x20aa6439] <+25>:  48 83 c0 19                    addq   $0x19, %rax
binary[0x20aa643d] <+29>:  f3 c3                          rep    retq
binary[0x20aa643f] <+31>:  48 83 c0 0c                    addq   $0xc, %rax
binary[0x20aa6443] <+35>:  f3 c3                          rep    retq
binary[0x20aa6445] <+37>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa644f] <+47>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa6459] <+57>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa6463] <+67>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa646d] <+77>:  66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa6477] <+87>:  66 0f 1f 84 00 00 00 00 00     nopw   (%rax,%rax)
binary[0x20aa6480] <+96>:  72 63                          jb     0x20aa64e5     ; <+197>
binary[0x20aa6482] <+98>:  34 28                          xorb   $0x28, %al
binary[0x20aa6484] <+100>: 38 78 2c                       cmpb   %bh, 0x2c(%rax)
binary[0x20aa6487] <+103>: 69 6e 74 29 00 72 63           imull  $0x63720029, 0x74(%rsi), %ebp ; imm = 0x63720029 
binary[0x20aa648e] <+110>: 34 28                          xorb   $0x28, %al
binary[0x20aa6490] <+112>: 38 78 2c                       cmpb   %bh, 0x2c(%rax)
binary[0x20aa6493] <+115>: 63 68 61                       movslq 0x61(%rax), %ebp
binary[0x20aa6496] <+118>: 72 29                          jb     0x20aa64c1     ; <+161>
binary[0x20aa6498] <+120>: 00 72 63                       addb   %dh, 0x63(%rdx)
binary[0x20aa649b] <+123>: 34 28                          xorb   $0x28, %al
binary[0x20aa649d] <+125>: 31 36                          xorl   %esi, (%rsi)
binary[0x20aa649f] <+127>: 78 2c                          js     0x20aa64cd     ; <+173>
binary[0x20aa64a1] <+129>: 69 6e 74 29 00 52 43           imull  $0x43520029, 0x74(%rsi), %ebp ; imm = 0x43520029 
binary[0x20aa64a8] <+136>: 34 20                          xorb   $0x20, %al
binary[0x20aa64aa] <+138>: 66 6f                          outsw  (%rsi), %dx
binary[0x20aa64ac] <+140>: 72 20                          jb     0x20aa64ce     ; <+174>
binary[0x20aa64ae] <+142>: 78 38                          js     0x20aa64e8     ; <+200>
binary[0x20aa64b0] <+144>: 36 5f                          popq   %rdi
binary[0x20aa64b2] <+146>: 36 34 2c                       xorb   $0x2c, %al
binary[0x20aa64b5] <+149>: 20 43 52                       andb   %al, 0x52(%rbx)
binary[0x20aa64b8] <+152>: 59                             popq   %rcx
binary[0x20aa64b9] <+153>: 50                             pushq  %rax
binary[0x20aa64ba] <+154>: 54                             pushq  %rsp
binary[0x20aa64bb] <+155>: 4f 47 41 4d 53                 pushq  %r11
binary[0x20aa64c0] <+160>: 20 62 79                       andb   %ah, 0x79(%rdx)
binary[0x20aa64c3] <+163>: 20 3c 61                       andb   %bh, (%rcx,%riz,2)
binary[0x20aa64c6] <+166>: 70 70                          jo     0x20aa6538     ; __KeccakF1600 + 56
binary[0x20aa64c8] <+168>: 72 6f                          jb     0x20aa6539     ; __KeccakF1600 + 57
binary[0x20aa64ca] <+170>: 40 6f                          outsl  (%rsi), %dx
binary[0x20aa64cc] <+172>: 70 65                          jo     0x20aa6533     ; __KeccakF1600 + 51
binary[0x20aa64ce] <+174>: 6e                             outsb  (%rsi), %dx
binary[0x20aa64cf] <+175>: 73 73                          jae    0x20aa6544     ; __KeccakF1600 + 68
binary[0x20aa64d1] <+177>: 6c                             insb   %dx, %es:(%rdi)
binary[0x20aa64d2] <+178>: 2e 6f                          outsl  %cs:(%rsi), %dx
binary[0x20aa64d4] <+180>: 72 67                          jb     0x20aa653d     ; __KeccakF1600 + 61
binary[0x20aa64d6] <+182>: 3e 00 66 2e                    addb   %ah, %ds:0x2e(%rsi)
binary[0x20aa64da] <+186>: 0f 1f 84 00 00 00 00 00        nopl   (%rax,%rax)
binary[0x20aa64e2] <+194>: 66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa64ec] <+204>: 66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)
binary[0x20aa64f6] <+214>: 66 2e 0f 1f 84 00 00 00 00 00  nopw   %cs:(%rax,%rax)

---

Also, BOLT fails to exit gracefully and crashes:

 #<!-- -->0 0x0000564e0470204d llvm::sys::PrintStackTrace(llvm::raw_ostream&amp;, int) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x29cd04d)
 #<!-- -->1 0x0000564e047025fb PrintStackTraceSignalHandler(void*) Signals.cpp:0:0
 #<!-- -->2 0x0000564e04700156 llvm::sys::RunSignalHandlers() (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x29cb156)
 #<!-- -->3 0x0000564e04703725 SignalHandler(int) Signals.cpp:0:0
 #<!-- -->4 0x00007f4abd57a520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #<!-- -->5 0x00007f4abd5ce9fc pthread_kill (/lib/x86_64-linux-gnu/libc.so.6+0x969fc)
 #<!-- -->6 0x00007f4abd57a476 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x42476)
 #<!-- -->7 0x00007f4abd5607f3 abort (/lib/x86_64-linux-gnu/libc.so.6+0x287f3)
 #<!-- -->8 0x0000564e04693504 llvm::report_fatal_error(llvm::Twine const&amp;, bool) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x295e504)
 #<!-- -->9 0x0000564e047070b0 llvm::detail::scope_exit&lt;std::decay&lt;llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::optional&lt;unsigned int&gt;)::$_0&gt;::type&gt; llvm::make_scope_exit&lt;llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::optional&lt;unsigned int&gt;)::$_0&gt;(llvm::llvm_execute_on_thread_impl(void* (*)(void*), void*, std::optional&lt;unsigned int&gt;)::$_0&amp;&amp;) Threading.cpp:0:0
#<!-- -->10 0x0000564e04707144 llvm::llvm_thread_join_impl(unsigned long) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x29d2144)
#<!-- -->11 0x0000564e046fc9dc llvm::thread::join() CrashRecoveryContext.cpp:0:0
#<!-- -->12 0x0000564e0536108d llvm::StdThreadPool::~StdThreadPool() (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x362c08d)
#<!-- -->13 0x0000564e053612f9 llvm::StdThreadPool::~StdThreadPool() (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x362c2f9)
#<!-- -->14 0x0000564e0535f40c std::default_delete&lt;llvm::ThreadPoolInterface&gt;::operator()(llvm::ThreadPoolInterface*) const ParallelUtilities.cpp:0:0
#<!-- -->15 0x0000564e0535c9e3 std::unique_ptr&lt;llvm::ThreadPoolInterface, std::default_delete&lt;llvm::ThreadPoolInterface&gt;&gt;::~unique_ptr() ParallelUtilities.cpp:0:0
#<!-- -->16 0x00007f4abd57d495 (/lib/x86_64-linux-gnu/libc.so.6+0x45495)
#<!-- -->17 0x00007f4abd57d610 on_exit (/lib/x86_64-linux-gnu/libc.so.6+0x45610)
#<!-- -->18 0x0000564e04908022 llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&amp;)::operator()(llvm::bolt::BOLTError const&amp;) const RewriteInstance.cpp:0:0
#<!-- -->19 0x0000564e04907f77 llvm::Error llvm::ErrorHandlerTraits&lt;void (&amp;)(llvm::bolt::BOLTError&amp;)&gt;::apply&lt;llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&amp;)&gt;(llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&amp;)&amp;&amp;, std::unique_ptr&lt;llvm::ErrorInfoBase, std::default_delete&lt;llvm::ErrorInfoBase&gt;&gt;) RewriteInstance.cpp:0:0
#<!-- -->20 0x0000564e04907e8a llvm::Error llvm::handleErrorImpl&lt;llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&amp;)&gt;(std::unique_ptr&lt;llvm::ErrorInfoBase, std::default_delete&lt;llvm::ErrorInfoBase&gt;&gt;, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&amp;)&amp;&amp;) RewriteInstance.cpp:0:0
#<!-- -->21 0x0000564e04907df3 llvm::Error llvm::handleErrors&lt;llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&amp;)&gt;(llvm::Error, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&amp;)&amp;&amp;) RewriteInstance.cpp:0:0
#<!-- -->22 0x0000564e04907bb0 void llvm::handleAllErrors&lt;llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&amp;)&gt;(llvm::Error, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const::'lambda'(llvm::bolt::BOLTError const&amp;)&amp;&amp;) RewriteInstance.cpp:0:0
#<!-- -->23 0x0000564e04907ab7 llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const RewriteInstance.cpp:0:0
#<!-- -->24 0x0000564e04907a58 void std::__invoke_impl&lt;void, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&amp;, llvm::bolt::BinaryFunction&amp;, unsigned short&gt;(std::__invoke_other, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&amp;, llvm::bolt::BinaryFunction&amp;, unsigned short&amp;&amp;) RewriteInstance.cpp:0:0
#<!-- -->25 0x0000564e049079f5 std::enable_if&lt;is_invocable_r_v&lt;void, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&amp;, llvm::bolt::BinaryFunction&amp;, unsigned short&gt;, void&gt;::type std::__invoke_r&lt;void, llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&amp;, llvm::bolt::BinaryFunction&amp;, unsigned short&gt;(llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&amp;, llvm::bolt::BinaryFunction&amp;, unsigned short&amp;&amp;) RewriteInstance.cpp:0:0
#<!-- -->26 0x0000564e0490791d std::_Function_handler&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short), llvm::bolt::RewriteInstance::buildFunctionsCFG()::$_0&gt;::_M_invoke(std::_Any_data const&amp;, llvm::bolt::BinaryFunction&amp;, unsigned short&amp;&amp;) RewriteInstance.cpp:0:0
#<!-- -->27 0x0000564e0535e968 std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;::operator()(llvm::bolt::BinaryFunction&amp;, unsigned short) const ParallelUtilities.cpp:0:0
#<!-- -->28 0x0000564e0535db6f llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0::operator()(std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, unsigned short) const ParallelUtilities.cpp:0:0
#<!-- -->29 0x0000564e0535f048 void std::__invoke_impl&lt;void, llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, unsigned int&amp;&gt;(std::__invoke_other, llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, unsigned int&amp;) ParallelUtilities.cpp:0:0
#<!-- -->30 0x0000564e0535ef6d std::__invoke_result&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, unsigned int&amp;&gt;::type std::__invoke&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, unsigned int&amp;&gt;(llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;&amp;, unsigned int&amp;) ParallelUtilities.cpp:0:0
#<!-- -->31 0x0000564e0535ef37 void std::_Bind&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0 (std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, unsigned int)&gt;::__call&lt;void, 0ul, 1ul, 2ul&gt;(std::tuple&lt;&gt;&amp;&amp;, std::_Index_tuple&lt;0ul, 1ul, 2ul&gt;) ParallelUtilities.cpp:0:0
#<!-- -->32 0x0000564e0535eea6 void std::_Bind&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0 (std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, unsigned int)&gt;::operator()&lt;void&gt;() ParallelUtilities.cpp:0:0
#<!-- -->33 0x0000564e0535ee75 void std::__invoke_impl&lt;void, std::_Bind&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0 (std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, unsigned int)&gt;&amp;&gt;(std::__invoke_other, std::_Bind&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0 (std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, unsigned int)&gt;&amp;) ParallelUtilities.cpp:0:0
#<!-- -->34 0x0000564e0535ee35 std::enable_if&lt;is_invocable_r_v&lt;void, std::_Bind&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0 (std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, unsigned int)&gt;&amp;&gt;, void&gt;::type std::__invoke_r&lt;void, std::_Bind&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0 (std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, unsigned int)&gt;&amp;&gt;(std::_Bind&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0 (std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, unsigned int)&gt;&amp;) ParallelUtilities.cpp:0:0
#<!-- -->35 0x0000564e0535ec3d std::_Function_handler&lt;void (), std::_Bind&lt;llvm::bolt::ParallelUtilities::runOnEachFunctionWithUniqueAllocId(llvm::bolt::BinaryContext&amp;, llvm::bolt::ParallelUtilities::SchedulingPolicy, std::function&lt;void (llvm::bolt::BinaryFunction&amp;, unsigned short)&gt;, std::function&lt;bool (llvm::bolt::BinaryFunction const&amp;)&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt;&gt;, bool, unsigned int)::$_0 (std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, std::_Rb_tree_iterator&lt;std::pair&lt;unsigned long const, llvm::bolt::BinaryFunction&gt;&gt;, unsigned int)&gt;&gt;::_M_invoke(std::_Any_data const&amp;) ParallelUtilities.cpp:0:0
#<!-- -->36 0x0000564e0493f515 std::function&lt;void ()&gt;::operator()() const DWARFRewriter.cpp:0:0
#<!-- -->37 0x0000564e0493f4d5 void std::__invoke_impl&lt;void, std::function&lt;void ()&gt;&gt;(std::__invoke_other, std::function&lt;void ()&gt;&amp;&amp;) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2c0a4d5)
#<!-- -->38 0x0000564e0493f495 std::__invoke_result&lt;std::function&lt;void ()&gt;&gt;::type std::__invoke&lt;std::function&lt;void ()&gt;&gt;(std::function&lt;void ()&gt;&amp;&amp;) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2c0a495)
#<!-- -->39 0x0000564e0493f46d void std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;::_M_invoke&lt;0ul&gt;(std::_Index_tuple&lt;0ul&gt;) DWARFRewriter.cpp:0:0
#<!-- -->40 0x0000564e0493f415 std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;::operator()() DWARFRewriter.cpp:0:0
#<!-- -->41 0x0000564e0493f3dc std::__future_base::_Task_setter&lt;std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt;, std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;::operator()() const DWARFRewriter.cpp:0:0
#<!-- -->42 0x0000564e0493f2f0 std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt; std::__invoke_impl&lt;std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt;, std::__future_base::_Task_setter&lt;std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt;, std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;&amp;&gt;(std::__invoke_other, std::__future_base::_Task_setter&lt;std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt;, std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;&amp;) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2c0a2f0)
#<!-- -->43 0x0000564e0493f278 std::enable_if&lt;is_invocable_r_v&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt;, std::__future_base::_Task_setter&lt;std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt;, std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;&amp;&gt;, std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt;&gt;::type std::__invoke_r&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt;, std::__future_base::_Task_setter&lt;std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt;, std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;&amp;&gt;(std::__future_base::_Task_setter&lt;std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt;, std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;&amp;) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2c0a278)
#<!-- -->44 0x0000564e0493f190 std::_Function_handler&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; (), std::__future_base::_Task_setter&lt;std::unique_ptr&lt;std::__future_base::_Result&lt;void&gt;, std::__future_base::_Result_base::_Deleter&gt;, std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;&gt;::_M_invoke(std::_Any_data const&amp;) DWARFRewriter.cpp:0:0
#<!-- -->45 0x0000564e04848e88 std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;::operator()() const JITLinkLinker.cpp:0:0
#<!-- -->46 0x0000564e04848be9 std::__future_base::_State_baseV2::_M_do_set(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*) JITLinkLinker.cpp:0:0
#<!-- -->47 0x0000564e04848e1f void std::__invoke_impl&lt;void, void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(std::__invoke_memfun_deref, void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2b13e1f)
#<!-- -->48 0x0000564e04848d8d std::__invoke_result&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;::type std::__invoke&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2b13d8d)
#<!-- -->49 0x0000564e04848d54 void std::call_once&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(std::once_flag&amp;, void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;)::'lambda'()::operator()() const JITLinkLinker.cpp:0:0
#<!-- -->50 0x0000564e04848d24 std::once_flag::_Prepare_execution::_Prepare_execution&lt;void std::call_once&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(std::once_flag&amp;, void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;)::'lambda'()&gt;(void (std::__future_base::_State_baseV2::*&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*))::'lambda'()::operator()() const JITLinkLinker.cpp:0:0
#<!-- -->51 0x0000564e04848cf1 std::once_flag::_Prepare_execution::_Prepare_execution&lt;void std::call_once&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(std::once_flag&amp;, void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;)::'lambda'()&gt;(void (std::__future_base::_State_baseV2::*&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*))::'lambda'()::__invoke() JITLinkLinker.cpp:0:0
#<!-- -->52 0x00007f4abd5d1ee8 (/lib/x86_64-linux-gnu/libc.so.6+0x99ee8)
#<!-- -->53 0x0000564e04848c9b __gthread_once(int*, void (*)()) JITLinkLinker.cpp:0:0
#<!-- -->54 0x0000564e04848b99 void std::call_once&lt;void (std::__future_base::_State_baseV2::*)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*&gt;(std::once_flag&amp;, void (std::__future_base::_State_baseV2::*&amp;&amp;)(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*, bool*), std::__future_base::_State_baseV2*&amp;&amp;, std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;*&amp;&amp;, bool*&amp;&amp;) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x2b13b99)
#<!-- -->55 0x0000564e04848955 std::__future_base::_State_baseV2::_M_set_result(std::function&lt;std::unique_ptr&lt;std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter&gt; ()&gt;, bool) JITLinkLinker.cpp:0:0
#<!-- -->56 0x0000564e04940e2b std::__future_base::_Deferred_state&lt;std::thread::_Invoker&lt;std::tuple&lt;std::function&lt;void ()&gt;&gt;&gt;, void&gt;::_M_complete_async() DWARFRewriter.cpp:0:0
#<!-- -->57 0x0000564e0484a3bd std::__future_base::_State_baseV2::wait() JITLinkLinker.cpp:0:0
#<!-- -->58 0x0000564e049414aa std::__basic_future&lt;void&gt;::wait() const DWARFRewriter.cpp:0:0
#<!-- -->59 0x0000564e04941475 std::shared_future&lt;void&gt; llvm::ThreadPoolInterface::asyncImpl&lt;void&gt;(std::function&lt;void ()&gt;, llvm::ThreadPoolTaskGroup*)::'lambda'()::operator()() const DWARFRewriter.cpp:0:0
#<!-- -->60 0x0000564e04941455 void std::__invoke_impl&lt;void, std::shared_future&lt;void&gt; llvm::ThreadPoolInterface::asyncImpl&lt;void&gt;(std::function&lt;void ()&gt;, llvm::ThreadPoolTaskGroup*)::'lambda'()&amp;&gt;(std::__invoke_other, std::shared_future&lt;void&gt; llvm::ThreadPoolInterface::asyncImpl&lt;void&gt;(std::function&lt;void ()&gt;, llvm::ThreadPoolTaskGroup*)::'lambda'()&amp;) DWARFRewriter.cpp:0:0
#<!-- -->61 0x0000564e04941415 std::enable_if&lt;is_invocable_r_v&lt;void, std::shared_future&lt;void&gt; llvm::ThreadPoolInterface::asyncImpl&lt;void&gt;(std::function&lt;void ()&gt;, llvm::ThreadPoolTaskGroup*)::'lambda'()&amp;&gt;, void&gt;::type std::__invoke_r&lt;void, std::shared_future&lt;void&gt; llvm::ThreadPoolInterface::asyncImpl&lt;void&gt;(std::function&lt;void ()&gt;, llvm::ThreadPoolTaskGroup*)::'lambda'()&amp;&gt;(std::shared_future&lt;void&gt; llvm::ThreadPoolInterface::asyncImpl&lt;void&gt;(std::function&lt;void ()&gt;, llvm::ThreadPoolTaskGroup*)::'lambda'()&amp;) DWARFRewriter.cpp:0:0
#<!-- -->62 0x0000564e0494128d std::_Function_handler&lt;void (), std::shared_future&lt;void&gt; llvm::ThreadPoolInterface::asyncImpl&lt;void&gt;(std::function&lt;void ()&gt;, llvm::ThreadPoolTaskGroup*)::'lambda'()&gt;::_M_invoke(std::_Any_data const&amp;) DWARFRewriter.cpp:0:0
#<!-- -->63 0x0000564e0493f515 std::function&lt;void ()&gt;::operator()() const DWARFRewriter.cpp:0:0
#<!-- -->64 0x0000564e05360202 llvm::StdThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/home/itrofimow/work/llvm-project/build/bin/llvm-bolt+0x362b202)
#<!-- -->65 0x0000564e05363cd5 llvm::StdThreadPool::grow(int)::$_0::operator()() const ThreadPool.cpp:0:0
#<!-- -->66 0x0000564e05363c59 auto void llvm::thread::GenericThreadProxy&lt;std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&gt;(void*)::'lambda'(auto&amp;&amp;, auto&amp;&amp;...)::operator()&lt;llvm::StdThreadPool::grow(int)::$_0&amp;&gt;(auto&amp;&amp;, auto&amp;&amp;...) const ThreadPool.cpp:0:0
#<!-- -->67 0x0000564e05363c2d auto std::__invoke_impl&lt;void, void llvm::thread::GenericThreadProxy&lt;std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&gt;(void*)::'lambda'(auto&amp;&amp;, auto&amp;&amp;...), llvm::StdThreadPool::grow(int)::$_0&amp;&gt;(std::__invoke_other, void llvm::thread::GenericThreadProxy&lt;std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&gt;(void*)::'lambda'(auto&amp;&amp;, auto&amp;&amp;...)&amp;&amp;, llvm::StdThreadPool::grow(int)::$_0&amp;) ThreadPool.cpp:0:0
#<!-- -->68 0x0000564e05363bdd std::__invoke_result&lt;auto, auto...&gt;::type std::__invoke&lt;void llvm::thread::GenericThreadProxy&lt;std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&gt;(void*)::'lambda'(auto&amp;&amp;, auto&amp;&amp;...), llvm::StdThreadPool::grow(int)::$_0&amp;&gt;(auto&amp;&amp;, auto&amp;&amp;...) ThreadPool.cpp:0:0
#<!-- -->69 0x0000564e05363bad decltype(auto) std::__apply_impl&lt;void llvm::thread::GenericThreadProxy&lt;std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&gt;(void*)::'lambda'(auto&amp;&amp;, auto&amp;&amp;...), std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&amp;, 0ul&gt;(auto&amp;&amp;, std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&amp;, std::integer_sequence&lt;unsigned long, 0ul&gt;) ThreadPool.cpp:0:0
#<!-- -->70 0x0000564e05363b3d decltype(auto) std::apply&lt;void llvm::thread::GenericThreadProxy&lt;std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&gt;(void*)::'lambda'(auto&amp;&amp;, auto&amp;&amp;...), std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&amp;&gt;(auto&amp;&amp;, std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&amp;) ThreadPool.cpp:0:0
#<!-- -->71 0x0000564e05363b0e void llvm::thread::GenericThreadProxy&lt;std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&gt;(void*) ThreadPool.cpp:0:0
#<!-- -->72 0x0000564e05363835 void* llvm::thread::ThreadProxy&lt;std::tuple&lt;llvm::StdThreadPool::grow(int)::$_0&gt;&gt;(void*) ThreadPool.cpp:0:0
#<!-- -->73 0x00007f4abd5ccac3 (/lib/x86_64-linux-gnu/libc.so.6+0x94ac3)
#<!-- -->74 0x00007f4abd65e850 (/lib/x86_64-linux-gnu/libc.so.6+0x126850)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Aborted

for which I don't really care that much, but sharing with you guys anyway since I have the stacktrace symbolized.

---

I do care for the "BOLT is unable to proceed because it couldn't properly understand this function" error, however:
we at Perforator (https://github.com/yandex/perforator) are working on generating BOLT-profiles, and we've got it working with llvm-bolt-19,
but this error with llvm-bolt-20 is not only frightening, but also manifests itself with a lot of binaries we have, which blocks upgrading to llvm-20 for us.

I would be happy to provide any additional info/cooperate on this, if needed

[aaupov]

Thanks for investigating it. Have you tried -skip-funcs=v1_RC4_options? Does the issue reproduce on the trunk version?

[itrofimow]

Fails in the exact same way on f295617

-skip-funcs=v1_RC4_options works, which is nice (at least I have a way to make bolt proceed now), thanks.

Is there something wrong with v1_RC4_options (openssl is sometimes creative with asm, so that wouldn't surprise me), or is BOLT not able to parse some of openssl's shenanigans?
I'm afraid we won't be able to -skip-funcs all the problematic functions, and would prefer to fix the pattern in out code, if there is any pattern in openssl which leads to BOLT failures

[yavtuk]

We skip ~10 functions, where there is data inside functions

[yavtuk]

But it depends on the openssl version and target platform

[aaupov]

Also, while you're on it:

1. I can see that you're running an aggregation job (llvm-bolt --aggregate-only). We recommend using perf2bolt explicitly – this enables profile stats reporting (-show-density flag for llvm-bolt). Note perf2bolt also allows an optional heatmap generation (--heatmap=<filename>) with some profile stats (section utilization/hotness). Assuming you also profile BOLTed binaries, you can run perf2bolt with heatmap (or heatmap directly) and these metrics will tell the optimization profile representativeness.
2. Since you're working with pre-aggregated profile, I recommend that you get up-to-date with the recent changes in the format that significantly improve the profile quality and can bring extra performance:

• [BOLT] Add pre-aggregated trace support #127125 - parse LBRs into traces/T-type pre-aggregated entries. Enables efficiencies across sample aggregation and parsing, directly improves profile quality and performance.
• [BOLT] Impute missing trace fall-through #145258 - use with perf2bolt, pass -impute-trace-fall-through. Preserve LBR top-of-stack entry (without a fall-through), passing it as branch-only/B-type pre-aggregated entry. Further improves the profile quality with statistical means.
• [BOLT] Support pre-aggregated returns #143296 - preserve branch type information (-j save_type with perf record, PERF_SAMPLE_BRANCH_TYPE_SAVE with perf_event_open/ebpf) and mark returns as return trace/R-type entries. Improves profile quality, especially important for external (DSO) returns. Branch type also enables filtering syscall returns present even with requested user-space branch filtering (issue seen on ICL) which improves the profile quality.
• [BOLT] Directly use call count in buildCallGraph #134966 – use with llvm-bolt, pass -fix-block-counts=0. Improves call graph profile quality.

[itrofimow]

Thanks for the heads up about format changes/deprecation, @aaupov. I had a look at the PR's you've linked: looks great, will try to catch up to the most recent BOLT.

---

About the original issue at hand: AFAIU BOLT fails to understand functions with data inside of them, which is a somewhat common thing in openssl (for instance). I don't think the -skip-funcs solution is the most convenient one, given that

• the functions to skip are unknown upfront, and one could only skip them when the BOLT fails, which doesn't really work that well with "generate-BOLT-profile-as-a-service" kinda thing we run at Perforator
• there could be a lot of such functions, an unlimited amount, basically

What do you think about something in the lines of -skip-funcs-by-path flag, which would allow to skip all the functions coming from files matching the path[s] provided? - that way I would be able to ignore openssl (and some other asm-heavy libraries) altogether

[itrofimow]

Hi @aaupov ! What do you think about the idea above?

[aaupov]

Hi @itrofimow, sorry for a delay, still catching up after my recharge.

What do you think about something in the lines of -skip-funcs-by-path flag, which would allow to skip all the functions coming from files matching the path[s] provided? - that way I would be able to ignore openssl (and some other asm-heavy libraries) altogether

Where should the path come from? If you mean debug information, it's missing for assembly code where we need it the most.

there could be a lot of such functions, an unlimited amount, basically

I guess there's no generic solution for non-conformant code, but it's in a minority and in practice we can get away with a limited set of functions to skip. It would also help raise such issues upstream, which we do occasionally (e.g. pq-crystals/kyber#75). Additionally, we have openssl binary test running on a buildbot (rafaelauler/bolt-tests@93bdc8b).

[maksfb]

Hi @itrofimow, is it possible for you to use a custom linker script to place .text from openssl into a special section? If yes, we can add a new flag to BOLT to skip this section altogether.

[paschalis-mpeis]

Hey folks,

I haven't fully followed this discussion, but at the last office hours I raised the topic of marking code regions that binary analysis / post-link opt would be wise ignore.

Quick update: we are thinking that defining special symbols could provide such hints. As this is target-agnostic, it doesn't belong into any ABI. But to help establish some standarization we are considering to add it in our Conventions for Binary Analysis and reserve those symbols.

What are your thoughts on such an approach?

[maksfb]

Hi @paschalis-mpeis, that's a good point.

Using a pair of symbols with designated names is definitely an option. E.g., we currently use __hot_start/__hot_end to identify code for huge pages. Similarly, we can use sets of pairs of symbols (like __fixed_start/__fixed)_end) to identify regions of fixed code, i.e. code not meant for modification beyond relocation processing. Note, that ideally we should be able to relocate a fixed region, and we will have to process relocations against other code left by a linker under --emit-relocs. I expect a code region corresponding to a section from .o to satisfy such requirements.

Another option, is to have fixed code being placed in a section with a designated name scheme or ELF type. We can support multiple sections per binary.

If we will have a designated metadata section for describing a binary for post-link analysis, we can also add fixed region information in there. This could complementary to the options above.

[paschalis-mpeis]

Note, that ideally we should be able to relocate a fixed region,..

Meaning, relocating the region as a whole?

Thanks for sharing your thoughts.
I think we can review past use-cases and define a marking protocol, ie,
we could provide information on the region, what action to take, possibly with a hint on the reason/area.

Example actions:

• ignore function fully or just disable optimizations

Example regions:

• a whole section
• a function or many functions
• can we go at a lower granularity? (ie set of blocks not to be split apart?)

A designated metadata section for post-link analysis sounds good.
I think we can expand these directions during the upcoming Office Hours?

[maksfb]

Note, that ideally we should be able to relocate a fixed region,..

Meaning, relocating the region as a whole?

Exactly. We should have the same flexibility as the linker. But we have to know the boundaries of the relocated entity (i.e. a section), a critical piece of information that the linker has and BOLT does not.

Thanks for sharing your thoughts. I think we can review past use-cases and define a marking protocol, ie, we could provide information on the region, what action to take, possibly with a hint on the reason/area.

Example actions:

• ignore function fully or just disable optimizations

Example regions:

• a whole section
• a function or many functions
• can we go at a lower granularity? (ie set of blocks not to be split apart?)

A designated metadata section for post-link analysis sounds good. I think we can expand these directions during the upcoming Office Hours?

I believe having a region-based granularity in addition to existing function-level granularity should cover most interesting cases, from the optimization point of view. Basic block-level is too fine to be important at large scale. Even excluding a set of functions will unlikely have a measurable performance effect in isolation, i.e. excluding possible side effects.