[Bug]: Some tests fail on 32-bit PowerPC [90% tests passed, 74 tests failed out of 729]

[barracuda156]

Contact Details

@barracuda156

What happened?

Build from ac845f0

90% tests passed, 74 tests failed out of 729

Label Time Summary:
flaky       = 544.25 sec*proc (19 tests)
unittest    = 567.74 sec*proc (713 tests)

Total Test time (real) = 1113.89 sec

The following tests did not run:
	499 - Future.AsyncSpecifyNewThread (Disabled)
	500 - Future.AsyncNotDeferredSpecifyNewThread (Disabled)
	501 - Future.AsyncNotAsyncSpecifyNewThread (Disabled)

The following tests FAILED:
	  5 - ChunkedFor.SimpleLoopStatic (Failed)
	 11 - ChunkedFor.SimpleLoopSmallRange (SEGFAULT)
	 14 - ChunkedFor.LoopSmallRangeWithStateWithExternalWait (SEGFAULT)
	465 - ForEach.Vector (SEGFAULT)
	467 - ForEach.Deque (Bus error)
	469 - ForEach.UnorderedMap (Bus error)
	473 - ForEach.References (SEGFAULT)
	474 - ForEach.Cascade (SEGFAULT)
	475 - ForEachN.Vector (SEGFAULT)
	477 - ForEachN.Deque (Bus error)
	479 - ForEachN.UnorderedMap (Bus error)
	496 - Future.AsyncSpecifyThreadPool (Bus error)
	567 - GreedyForRanges.LoopWithDequeState (Bus error)
	568 - GreedyForRanges.LoopWithVectorState (Bus error)
	569 - GreedyForRanges.LoopWithListState (Failed)
	570 - GreedyForRanges.ConcurrentLoopNoCoordination (Bus error)
	571 - GreedyForRanges.CoordinatedLoops (Bus error)
	572 - GreedyForRanges.CoordinatedConcurrentLoops (Bus error)
	578 - GreedyForRanges.AvoidOverflow2 (Bus error)
	579 - GreedyForRanges.EmptyLoopsWaitIfToldTo (Failed)
	580 - GreedyForRanges.SingleLoopWaitIfToldTo (Failed)
	585 - GreedyFor.LoopWithDequeState (Bus error)
	586 - GreedyFor.LoopWithVectorState (Bus error)
	587 - GreedyFor.LoopWithListState (Failed)
	589 - GreedyFor.CoordinatedLoops (Bus error)
	590 - GreedyFor.CoordinatedConcurrentLoops (Bus error)
	591 - GreedyFor.OptionsMaxThreads (SEGFAULT)
	596 - GreedyFor.AvoidOverflow2 (SEGFAULT)
	597 - GreedyFor.EmptyLoopsWaitIfToldTo (Failed)
	598 - GreedyFor.SingleLoopWaitIfToldTo (Failed)
	646 - ResourcePool.SameNumBuffersAsThreadsTest (Bus error)
	647 - ResourcePool.FewerBuffersThanThreadsTest (Bus error)
	648 - ResourcePool.MoreBuffersThanThreadsTest (Subprocess aborted)
	656 - SmallBufferAllocator.ThreadsSimpleSmall (SEGFAULT)
	657 - SmallBufferAllocator.ThreadsSimpleMedium (SEGFAULT)
	658 - SmallBufferAllocator.ThreadsSimpleLarge (SEGFAULT)
	659 - SmallBufferAllocator.ThreadsHandoffSmall (SEGFAULT)
	660 - SmallBufferAllocator.ThreadsHandoffMedium (SEGFAULT)
	661 - SmallBufferAllocator.ThreadsHandoffLarge (Bus error)
	664 - TaskSet.Recursive (Bus error)
	667 - TaskSet.CascadingCancelOne (Bus error)
	669 - TaskSet.CascadingManyCancel (SEGFAULT)
	670 - TaskSet.CascadingMany (SEGFAULT)
	672 - TaskSet.ExceptionNoForceQueuing (SEGFAULT)
	674 - TaskSet.ExceptionNoForceQueuingTryWait (SEGFAULT)
	676 - ConcurrentTaskSet.DoTree (SEGFAULT)
	681 - TaskSetTestParameters/TaskSetTest.MixedWork/0 (SEGFAULT)
	682 - TaskSetTestParameters/TaskSetTest.MixedWork/1 (SEGFAULT)
	683 - TaskSetTestParameters/TaskSetTest.MixedWork/2 (Bus error)
	684 - TaskSetTestParameters/TaskSetTest.MultiWait/0 (SEGFAULT)
	685 - TaskSetTestParameters/TaskSetTest.MultiWait/1 (SEGFAULT)
	686 - TaskSetTestParameters/TaskSetTest.MultiWait/2 (SEGFAULT)
	687 - TaskSetTestParameters/TaskSetTest.MultiSet/0 (SEGFAULT)
	688 - TaskSetTestParameters/TaskSetTest.MultiSet/1 (Bus error)
	689 - TaskSetTestParameters/TaskSetTest.MultiSet/2 (SEGFAULT)
	690 - TaskSetTestParameters/TaskSetTest.MultiSetTryWait/0 (SEGFAULT)
	691 - TaskSetTestParameters/TaskSetTest.MultiSetTryWait/1 (SEGFAULT)
	692 - TaskSetTestParameters/TaskSetTest.MultiSetTryWait/2 (Bus error)
	697 - ThreadPool.ResizeConcurrent (Bus error)
	698 - ThreadPool.ResizeMoreConcurrent (Bus error)
	699 - ThreadPool.SetSignalingWakeConcurrent (Bus error)
	700 - ThreadPool.ResizeCheckApproxActualRunningThreads (SEGFAULT)
	701 - ThreadPool.SetSignalingWakeConcurrentZeroLatency (Bus error)
	703 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWork/1 (SEGFAULT)
	704 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWork/2 (SEGFAULT)
	706 - ThreadPoolTestParameters/ThreadPoolTest.MixedWork/1 (SEGFAULT)
	707 - ThreadPoolTestParameters/ThreadPoolTest.MixedWork/2 (SEGFAULT)
	708 - ThreadPoolTestParameters/ThreadPoolTest.CrossPoolTest/0 (SEGFAULT)
	709 - ThreadPoolTestParameters/ThreadPoolTest.CrossPoolTest/1 (Bus error)
	710 - ThreadPoolTestParameters/ThreadPoolTest.CrossPoolTest/2 (SEGFAULT)
	711 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWorkZeroLatencyPoll/0 (Bus error)
	712 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWorkZeroLatencyPoll/1 (SEGFAULT)
	713 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWorkZeroLatencyPoll/2 (SEGFAULT)
	714 - Priorty.PriorityGetsCycles (INTERRUPT)
Errors while running CTest

Of course, there is no implication that this must be fixed specifically for PowerPC, I rather report the bug for the record.

Version

latest (Edge)

Code of Conduct

• I agree to follow this project's Code of Conduct

Log from tests:
LastTest.log

(I tried to run binaries via wrappers to fix malloc errors, but apparently here it does not help. I think I got the same results with and without wrapper scripts.)

[graphicsMan]

Shoot, this is quite a bummer. It is making me rethink removing the insistent warning. I'm now wondering if this is exactly the error encountered. I just didn't know why it would work for x86(64), riscv, and arm64, but not ppc. Unfortunately I don't have access to a ppc32 machine. I'm guessing that this will prevent your use of the library. If you can help me understand the crashes (which are all likely the same one or two), I can try to figure out a fix. I'm also very happy with any contribution you might be willing to make.

[barracuda156]

AFAIK there are typically two reasons for this sort of errors: a clash of C++ runtime libraries (old OS one and new GCC one) and actual limitation of 32-bit arch. The first is more common, and normally can be fixed on our end (this is not a bug in a software tested, but an inherent issue of a set-up). In most cases, once fixed, it actually works robustly.

However, if something is hitting the limit of 32-bit, solution should be different. I suspect it may be this case here.
Take a look, we had a similar case with blavaan, which was successfully resolved: stan-dev/rstan#1095 (comment)

[graphicsMan]

I'd be very surprised if more than one or two tests exceed 2GB runtime RAM usage. I doubt this would occur in 32-bit x86, though I should be able to test that.

It could be system setup, but presumably you are building and running similar code, so that seems unlikely.

I think there is likely some #else somewhere not handling ppc platform, and the result is that moodycamel queue crashes.

[graphicsMan]

@barracuda156 Do you have any advice for getting an old laptop that might match your test case? I might be interested in purchasing something. I'd love to support dispenso on as many platforms as possible, but I only have so many platforms available to me

[barracuda156]

@barracuda156 Do you have any advice for getting an old laptop that might match your test case? I might be interested in purchasing something. I'd love to support dispenso on as many platforms as possible, but I only have so many platforms available to me

@graphicsMan Laptop options are somewhat constrained, sadly, and until Linux ppc laptop is out, the best we got is PowerBook G4 1.67 GHz from 2005. It is tolerable to use and can be upgraded with SSD, but it is single core, and G4 does not support more than 2 GB RAM.
If you see it cheap, it is a very neat machine to have though. Slow by modern standards, but a piece of tech art.

If you have a space to accommodate a desktop, late 2005 models of PowerMacs G5 are pretty capable (and you can run some modern BSD or Linux too, not just an old MacOS). They often are sold cheap (and sometimes given out for free), but they are heavy – international shipping is a no-go.

Aside of native hardware, there are two other options:

Modern Talos II workstations on POWER 9 can run emulate G4/G5 hardware and run MacOS (or Linux, of course): https://www.talospace.com/2018/08/making-your-talos-ii-into-power-mac.html

Finally, it is pretty easy to set up 10.5.8 or 10.6.8 (preferably Server versions of those) in a VM and run on any modern Intel hardware. Older Intel Macs can run 10.6.8 natively: I think the last models to do that out of the box are 2011 generation, but at least 2012 ones are able to do that with minor hacks. Then, you could build and run ppc binaries via Apple Rosetta (for practical purposes as-if natively).
10.6.8 was the last MacOS to support running ppc binaries (only 32-bit ppc).
Parallels can be used for this, and apparently UTM or maybe Qemu as well.
UTM and Qemu are supposed to be able to emulate PowerPC and therefore boot and run ppc version of 10.5.8, but I do not know how usable it is. I can confirm that 10.6.8 with Rosetta is very usable on a decent Intel MacMini. Not as fast as late G5, but faster than G4.

P. S. I think only OpenBSD currently provides pre-built ports for powerpc. This is not a big deal with a reasonably fast hardware, but building everything from sources on a PowerBook may be a bit of a pain.

[graphicsMan]

I have reproduced some issues on i386. It seems to fail fewer tests, but possibly fixing them would cover a lot of the failures for PPC as well. It may take me some time to fix these, but I will eventually slog through :)

[barracuda156]

I have reproduced some issues on i386. It seems to fail fewer tests, but possibly fixing them would cover a lot of the failures for PPC as well. It may take me some time to fix these, but I will eventually slog through :)

Great!

[graphicsMan]

I submitted changes a few days ago that fixed many 32-bit issues (some test, some in the library) for x86. Can you please try again with ppc 32?

[barracuda156]

I submitted changes a few days ago that fixed many 32-bit issues (some test, some in the library) for x86. Can you please try again with ppc 32?

Sure, will do it tonight, thank you.

[barracuda156]

From a721f22 commit:

92% tests passed, 60 tests failed out of 756

Label Time Summary:
flaky       =  11.47 sec*proc (19 tests)
unittest    = 804.53 sec*proc (741 tests)

Total Test time (real) = 818.48 sec

The following tests did not run:
	505 - Future.AsyncSpecifyNewThread (Disabled)
	506 - Future.AsyncNotDeferredSpecifyNewThread (Disabled)
	507 - Future.AsyncNotAsyncSpecifyNewThread (Disabled)
	522 - Future.NewThreadInvoker (Disabled)

The following tests FAILED:
	 11 - ChunkedFor.SimpleLoopSmallRange (SEGFAULT)
	 14 - ChunkedFor.LoopSmallRangeWithStateWithExternalWait (SEGFAULT)
	530 - Future.WhenAllEmptyTuple (Bus error)
	576 - GreedyForRanges.ConcurrentLoopNoCoordination (SEGFAULT)
	582 - GreedyForRanges.OptionsMaxThreadsBigPoolAutoChunkingNonBlocking (SEGFAULT)
	583 - GreedyForRanges.OptionsMaxThreadsSmallPoolStaticChunkingBlocking (SEGFAULT)
	584 - GreedyForRanges.OptionsMaxThreadsSmallPoolStaticChunkingNonBlocking (Bus error)
	585 - GreedyForRanges.OptionsMaxThreadsSmallPoolAutoChunkingBlocking (SEGFAULT)
	586 - GreedyForRanges.OptionsMaxThreadsSmallPoolAutoChunkingNonBlocking (SEGFAULT)
	605 - GreedyFor.ConcurrentLoopNoCoordination (Bus error)
	608 - GreedyFor.OptionsMaxThreadsBigPoolStaticChunkingBlocking (Bus error)
	612 - GreedyFor.OptionsMaxThreadsSmallPoolStaticChunkingBlocking (SEGFAULT)
	613 - GreedyFor.OptionsMaxThreadsSmallPoolStaticChunkingNonBlocking (SEGFAULT)
	614 - GreedyFor.OptionsMaxThreadsSmallPoolAutoChunkingBlocking (SEGFAULT)
	624 - GreedyFor.AvoidOverflow2 (SEGFAULT)
	629 - GreedyFor.SimpleLoopFewerItemsThanThreads (SEGFAULT)
	674 - ResourcePool.SameNumBuffersAsThreadsTest (Bus error)
	675 - ResourcePool.FewerBuffersThanThreadsTest (Bus error)
	676 - ResourcePool.MoreBuffersThanThreadsTest (ILLEGAL)
	684 - SmallBufferAllocator.ThreadsSimpleSmall (SEGFAULT)
	685 - SmallBufferAllocator.ThreadsSimpleMedium (SEGFAULT)
	686 - SmallBufferAllocator.ThreadsSimpleLarge (Bus error)
	687 - SmallBufferAllocator.ThreadsHandoffSmall (SEGFAULT)
	688 - SmallBufferAllocator.ThreadsHandoffMedium (SEGFAULT)
	689 - SmallBufferAllocator.ThreadsHandoffLarge (SEGFAULT)
	692 - TaskSet.Recursive (Bus error)
	693 - TaskSet.OneChildCancels (SEGFAULT)
	697 - TaskSet.CascadingManyCancel (SEGFAULT)
	698 - TaskSet.CascadingMany (SEGFAULT)
	703 - TaskSet.ExceptionCancels (SEGFAULT)
	704 - ConcurrentTaskSet.DoTree (Bus error)
	705 - ConcurrentTaskSet.Exception (SEGFAULT)
	709 - TaskSetTestParameters/TaskSetTest.MixedWork/0 (SEGFAULT)
	710 - TaskSetTestParameters/TaskSetTest.MixedWork/1 (SEGFAULT)
	711 - TaskSetTestParameters/TaskSetTest.MixedWork/2 (SEGFAULT)
	712 - TaskSetTestParameters/TaskSetTest.MultiWait/0 (SEGFAULT)
	713 - TaskSetTestParameters/TaskSetTest.MultiWait/1 (SEGFAULT)
	714 - TaskSetTestParameters/TaskSetTest.MultiWait/2 (SEGFAULT)
	715 - TaskSetTestParameters/TaskSetTest.MultiSet/0 (SEGFAULT)
	716 - TaskSetTestParameters/TaskSetTest.MultiSet/1 (SEGFAULT)
	717 - TaskSetTestParameters/TaskSetTest.MultiSet/2 (SEGFAULT)
	718 - TaskSetTestParameters/TaskSetTest.MultiSetTryWait/0 (SEGFAULT)
	719 - TaskSetTestParameters/TaskSetTest.MultiSetTryWait/1 (Bus error)
	720 - TaskSetTestParameters/TaskSetTest.MultiSetTryWait/2 (Bus error)
	725 - ThreadPool.ResizeConcurrent (Bus error)
	726 - ThreadPool.ResizeMoreConcurrent (Bus error)
	727 - ThreadPool.SetSignalingWakeConcurrent (Subprocess aborted)
	728 - ThreadPool.ResizeCheckApproxActualRunningThreads (Bus error)
	729 - ThreadPool.SetSignalingWakeConcurrentZeroLatency (SEGFAULT)
	730 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWork/0 (SEGFAULT)
	731 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWork/1 (SEGFAULT)
	733 - ThreadPoolTestParameters/ThreadPoolTest.MixedWork/0 (SEGFAULT)
	734 - ThreadPoolTestParameters/ThreadPoolTest.MixedWork/1 (SEGFAULT)
	735 - ThreadPoolTestParameters/ThreadPoolTest.MixedWork/2 (SEGFAULT)
	736 - ThreadPoolTestParameters/ThreadPoolTest.CrossPoolTest/0 (SEGFAULT)
	737 - ThreadPoolTestParameters/ThreadPoolTest.CrossPoolTest/1 (SEGFAULT)
	739 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWorkZeroLatencyPoll/0 (SEGFAULT)
	740 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWorkZeroLatencyPoll/1 (Bus error)
	741 - ThreadPoolTestParameters/ThreadPoolTest.SimpleWorkZeroLatencyPoll/2 (SEGFAULT)
	742 - Priorty.PriorityGetsCycles (SEGFAULT)

[graphicsMan]

Slightly disappointing considering I crushed dozens of 32-bit x86 failed test cases ;) How much memory does your machine have? I typically work in a land of ample gigabytes, but if you are under 2 gigabytes, it might be possible that some tests might run out of memory.

[barracuda156]

Slightly disappointing considering I crushed dozens of 32-bit x86 failed test cases ;) How much memory does your machine have? I typically work in a land of ample gigabytes, but if you are under 2 gigabytes, it might be possible that some tests might run out of memory.

@graphicsMan Machine has 16 GB, but at the moment restricted to 4 GB (for unrelated reasons, I am just lazy to reset boot args every time, another OS needs 4 GB strictly). I can make 16 GB available and re-run tests.