More granular failure messages for collection comparisons, arrays in particular

[delebedev]

Consider the following examples:

expect([1,2,3,4]).to(equal([1,2,3,4,5]))
// produces "expected to equal <[1, 2, 3, 4, 5]>, got <[1, 2, 3, 4]>"

It is kinda readable, but the thing is that in my experience arrays could contain long strings and / or enums with associated values, and it makes it impossible to understand what actually went wrong.

In this particular case failure message could be "expected array to have 5 elements, but got 4".

Another case:

expect([1,2,3,4]).to(equal([1,4,3,4])
// produces "expected to equal <[1, 4, 3, 4]>, got <[1, 2, 3, 4]>"

Which is again, readable but only if we operate with short values.

In this case message could be "expected to have "4" at index 1, got "2"".

Of course, one could resort to per-element comparison, but it does not feel right (or, maybe I'm just overthinking, please let me know)

guard array.count == 3 else {
  fail("blabla")
}// we need to make sure that we don't overflow in per-index comparison

expect(array[0]) == x
expect(array[1]) == y
expect(array[2]) == z

Exact messages text could be reconsidered, but I hope you get the idea.

[delebedev]

Here is more real example I'm seeing:

expected to equal <[field(title: "Amount", value: "<positive>+</positive> 60.00 GBP"), field(title:
 "Date completed", value: "Monday, January 1, 2001")]>, got <[field(title: "Amount", value: "
<positive>+</positive> 60.00 GBP"), field(title: "topup.details.dateCmpleted", value: "Monday, 
January 1, 2001"), action(title: "Repeat this pay-in", type: 
CustomControls.TWButtonStyle.secondary, action: 
Transfer.ActivityDetailsViewModel.Action.repeatTransfer(1)), field(title: "Transfer number", value:
 "#1")]>

[modocache]

I agree this could be improved, excellent suggestion.

• For the cases in which the count is different, we could print that, and then which elements exists in one array but not the other.
• Pretty-printing the arrays would also be nice.
• Beyond that, some more advanced diffing would be cool, too. I bet there's a lot of prior art out there, in other matcher frameworks and diffing programs.

[chronodm]

I've got a similar issue with Realm objects, and the error messages are pretty monstrous:

/Users/david/Projects/OpenRoget/Tests/OpenRogetTests/OpenRogetSpec.swift:100: error: -[OpenRogetTests_macOS.OpenRogetSpec thesaurusFrom_realmFileURL____success__head_parsing__children__extracts_the_words] : expected to equal <SemicolonGroup {paragraph = Paragraph {partOfSpeech = PartOfSpeech {head = Head {headGroup = HeadGroup {subsection = <Maximum depth exceeded>;number = 5;first = 217;last = 218;heads = <Maximum depth exceeded>;};number = 218;name = Shallowness;origNum = #209.;classId = 0;sectionId = 0;subsectionId = 0;headGroupId = 0;partsOfSpeech = List<PartOfSpeech> <0x7f86dca98200> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>);};number = 3;type = ADJ.;paragraphs = List<Paragraph> <0x7f86dca98020> ([0] Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;});};number = 1;name = shallow;semicolonGroups = List<SemicolonGroup> <0x7f86dca97ea0> ([0] SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 1;words = List<Word> <0x7f86dca99c20> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);},[1] SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 2;words = List<Word> <0x7f86dca9af40> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);},[2] SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 3;words = List<Word> <0x7f86dca9b030> ([0] <Maximum depth exceeded>);},[3] SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 4;words = List<Word> <0x7f86dca9b140> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>);});};number = 2;words = List<Word> <0x7f86dc8ae8b0> ([0] Word {semicolonGroup = SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 2;words = List<Word> <0x7f86dcaa07c0> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);};value = skin deep;number = 1;paraRef = (null);posRef = (null);headRef = (null);headRefId = (null);},[1] Word {semicolonGroup = SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 2;words = List<Word> <0x7f86dcaa1870> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);};value = ankle deep;number = 2;paraRef = (null);posRef = (null);headRef = (null);headRefId = (null);},[2] Word {semicolonGroup = SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 2;words = List<Word> <0x7f86dcaa33d0> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);};value = knee deep;number = 3;paraRef = (null);posRef = (null);headRef = (null);headRefId = (null);});}>, got <SemicolonGroup {paragraph = Paragraph {partOfSpeech = PartOfSpeech {head = Head {headGroup = HeadGroup {subsection = <Maximum depth exceeded>;number = 5;first = 217;last = 218;heads = <Maximum depth exceeded>;};number = 218;name = Shallowness;origNum = #209.;classId = 0;sectionId = 0;subsectionId = 0;headGroupId = 0;partsOfSpeech = List<PartOfSpeech> <0x7f86dcaa5ed0> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>);};number = 3;type = ADJ.;paragraphs = List<Paragraph> <0x7f86dcaa5d00> ([0] Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;});};number = 1;name = shallow;semicolonGroups = List<SemicolonGroup> <0x7f86dcaa5b80> ([0] SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 1;words = List<Word> <0x7f86dcaa78f0> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);},[1] SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 2;words = List<Word> <0x7f86dcaa8bd0> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);},[2] SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 3;words = List<Word> <0x7f86dcaa8cc0> ([0] <Maximum depth exceeded>);},[3] SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 4;words = List<Word> <0x7f86dcaa8db0> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>);});};number = 2;words = List<Word> <0x7f86dcaa05b0> ([0] Word {semicolonGroup = SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 2;words = List<Word> <0x7f86dcaae440> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);};value = skin deep;number = 1;paraRef = (null);posRef = (null);headRef = (null);headRefId = (null);},[1] Word {semicolonGroup = SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 2;words = List<Word> <0x7f86dcaaf590> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);};value = ankle deep;number = 2;paraRef = (null);posRef = (null);headRef = (null);headRefId = (null);},[2] Word {semicolonGroup = SemicolonGroup {paragraph = Paragraph {partOfSpeech = <Maximum depth exceeded>;number = 1;name = shallow;semicolonGroups = <Maximum depth exceeded>;};number = 2;words = List<Word> <0x7f86dcab1120> ([0] <Maximum depth exceeded>,[1] <Maximum depth exceeded>,[2] <Maximum depth exceeded>);};value = knee deep;number = 3;paraRef = (null);posRef = (null);headRef = (null);headRefId = (null);});}>

I feel like the issue here isn't arrays per se, but anything with a long and unhelpful string representation.

For reference, here's what Java's AssertJ does (comparing two vavr Array objects, first the arrays themselves, compared as a generic Iterable, then their string representations:

org.junit.ComparisonFailure: expected:<...12, 13, 14, 15, 16, [-]17, 18, 19)> but was:<...12, 13, 14, 15, 16, []17, 18, 19)>
Expected :Array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, -17, 18, 19)
Actual   :Array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)

org.junit.ComparisonFailure: expected:<...12, 13, 14, 15, 16, [-]17, 18, 19)"> but was:<...12, 13, 14, 15, 16, []17, 18, 19)">
Expected :"Array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, -17, 18, 19)"
Actual   :"Array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)"

In both cases, you get, first, an abbreviated version that concentrates on the spots that differ, and then the full version of expected and actual on separate lines, so they're (relatively) easy to compare.

[SeanROlszewski]

I'm presently running into an issue using Nimble to equate strings that are quite large, and the output that's produced is pretty unusable. I wanted to throw my support for this style of diffing to be added to nimble.

I have some spare bandwidth to make these changes if no one else is going to be working on this.

[ikesyo]

ref: #474

[teogeos]

IMO the best would be to fix this in a generic way with diffs as discussed in #899

[aodhol]

Is there any plan to fix this problem?

[younata]

This is low on my priority list. Getting Nimble to work with swift concurrency is a higher priority.