Name for "expect failure" feature

[JakobJingleheimer]

Related to nodejs/node#60669

Time-boxed discussion ending EoD (CET) 10 Jan 2026.

I can't figure out to create a poll here (a Discussions-specific feature 😕), so please add new options as a new thread below; then we can discuss each option within that thread, upvoting the ones you like, and 👎 ones to object (-1 from a collaborator will be taken as a block on the option that either needs to be resolved or overturned).

[ljharb]

(linking to nodejs/node#60669)

[ljharb]

Number 1 is this feature; number 2 is already clear by the choice of assertion (which is inside a test).

Number 3 is already a “todo” if you don’t want failure to fail the build, and you don’t want success to fail the build. This feature is for when you want failure to pass the build and success to fail the build.

[vassudanagunta]

By number 1 I meant a regular test failure that must be fixed. By number 3 I meant this feature. Clearly all this is ripe for confusion 😹

Imagine the summary at the end of a test report:

Passed: 32
Failed: 1
Failed as expected: 3
Todo: 7
Skipped: 1

I would not want the 3 failing as expected to be silently counted under Passed.

[JakobJingleheimer]

I would not want the 3 failing as expected to be silently counted under Passed.

Hmm, I thought about that when I implemented it but settled on users probably don't care about the differentiation in the summary because of its nature: it happens to need xfail (or whatever) now due to reasons (but at current, it's confirmed that there is no action needed—unlike skip and todo).

If they do want it reported separately, they can still get the info (and it is surfaced to the reporter, so a custom / non-builtin reporter could do that).

[ljharb]

They absolutely should be silently counted under passed; that’s almost the entire purpose of the feature.

[vassudanagunta]

that’s almost the entire purpose of the feature

Again I wonder whether we all have the same mental model of this feature.

[avivkeller]

Throwing out some ideas:

• expectFail / expectFailure
• invert (this one's probably the most unintuitive option, but I'm just spitballing. Invert, as failure = success and success = failure)
• shouldFail

---

xfail

+1. Also note that Discussions have a built-in polling feature, iirc.

knownFailure

-1. IMO saying something is a known failure is different than saying it's expected to fail. A flakey test can be considered a known failure, but we shouldn't expect it to fail.

[JakobJingleheimer]

I can't figure out how to set up a poll

[vassudanagunta]

expectError

• prior art: typescript (@ts-expect-error)
• same semantics

[vassudanagunta]

Can be confused with "a test that asserts that an operation will fail per specification". See #1 (reply in thread)

[JakobJingleheimer]

I don't think so—I think that's conflating the test-runner's features for the test-cases themselves (the test-cases themselves should indicate the behaviour). These flags (only, skip, todo, xfail) are merely orchestration.

[vassudanagunta]

expectedFailure

• prior art: pytest
• searchability: high

[vassudanagunta]

Representing expectations about the thing under test (aka the spec) happens at the granularity of assertions.

Annotations about the test case itself (not about the spec) are made against the test ("granularity of the test):

• This test case is todo because it has yet to be written/completed.

• This test case is to be skipped because it is not applicable in this environment/platform.

• This test case is xfail (expected to fail) because of a known issue that is allowed for now.

  or

  This test case coincides with a known issue which is allowed for now, but if the known issue goes away stop everything because we need to either remove the known issue annotation and close the related bug report, or undo the change because we cannot fix this bug until <some SemVer Minor/Major version change>.

As @JakobJingleheimer says, this new feature is not there as a lazy way to avoid properly representing the spec at the assertion level. It is not syntactic sugar. This is an entirely new and valuable semantic.

[JakobJingleheimer]

It's simply inverting the outcome of the test - if it failed, the overarching test passes; if it passed, the overarching test failed.

Technically that's true, but it would be an abuse of the feature (and confusing AF). If you want that, call it inverted or similar (but in that case, we've been having 2 different conversations).

[ljharb]

I don't think it's an abuse - it's the most valuable thing about the feature. Why is "a known issue that is allowed for now" restricted to an assertion? That's what skips and todos are currently abused for in test cases, and this feature would solve it much better.

I also don't think this is a "lazy" way - I think this is the proper way to indicate "this test case SHOULD be passing, but is known to fail right now - by design or for other reasons - and when that changes, it should be changed to a normal test case". It allows the test case to be written in the "ideal" way, minimizing the diff when the test case no longer fails.

I think that if there's this level of core disagreement still about what the feature is for, then we should definitely not land anything until that's figured out.

I'll try to join the test runner meeting tomorrow, but either way I think delay is far less costly than shipping the wrong thing.

[JakobJingleheimer]

I think discussing in the meeting would be very advantageous 🙂

[JakobJingleheimer]

As discussed in today's team meeting: We have consensus on this name (a slightly different variation because what I wrote here was actually a typo of the original proposal): expectFailure.

[vassudanagunta]

failing

• prior art: ava/jest
• searchability: low / zero

[vassudanagunta]

"Failing test" intuitively refers to any test that is not passing, so this option will introduce a lot of confusion in discussions about tests, in test reports, as well as in node:test documentation.

But let me devil's advocate this option (sometimes the devil wins):

1. Ava and Jest precedent.
2. It works ok in the sample documentation, as long as it is qualified with "flag" or "mark". i.e. a "failing test" vs "a test flagged as failing".

[vassudanagunta]

Imagine the summary at the end of a test report:

Passed: 32
Failed: 1
Failing: 3
Todo: 7
Skipped: 1

I would not want the 3 failing (as expected) to be silently counted under Passed.

[JakobJingleheimer]

I would not want xfail to be reported in the summary at all because it doesn't matter—all is well and there's nothing to do.

[JakobJingleheimer]

fails

• prior art: none
• searchability: low / zero

[vassudanagunta]

knownFailure

• prior art: ?
• searchability: high

[vassudanagunta]

Imagine the summary at the end of a test report:

Passed: 32
Failed: 1
Known Failures: 3
Todo: 7
Skipped: 1

I would not want the 3 known failures to be silently counted under Passed.

[ljharb]

As mentioned elsewhere, it makes no sense to count known failures any differently than passing tests. The test is “it fails”, and if it passes, it’s “passed”, end of story.

[JakobJingleheimer]

The time-box is ending in later today; this (knownFailure) is currently tied for first place with the incumbent xfail but this has an undisputed block from a collaborator. Just raising that in case proponents of this knownFailure want to discuss 🙂

[ljharb]

@avivkeller can you elaborate on your thumbs down?

[ljharb]

ah, i see above:

IMO saying something is a known failure is different than saying it's expected to fail. A flakey test can be considered a known failure, but we shouldn't expect it to fail.

[vassudanagunta]

mustFail

• prior art: C++
• searchability: high
• name suggests it should fail forever

[vassudanagunta]

Too similar to BDD style foo.shouldThrow, so easy to confused with "a test that asserts that an operation will fail per specification". See #1 (reply in thread).

[vassudanagunta]

shouldFail

• prior art: none
• searchability: high

[vassudanagunta]

Too similar to BDD style foo.shouldThrow, so easy to confused with "a test that asserts that an operation will fail per specification". See #1 (reply in thread).

xfail

• prior art: common outside JS
• searchability: very high
• a bit convoluted/obscure name

invert

• prior art: none?
• searchability: medium?

[JakobJingleheimer]

knownIssue

• prior art: common label/tag for bug reports
• searchability: medium? high?

[JakobJingleheimer]

@avivkeller is there anything in particular you dislike about it?

[avivkeller]

-1. IMO saying something is a known failure is different than saying it's expected to fail. A flakey test can be considered a known failure, but we shouldn't expect it to fail.

[JakobJingleheimer]

Hmm. I did see that and agree; but I think this is slightly different.

I'm thinking to also introduce a flaky flag that accepts either boolean or a number (where a number is the count to try for a success). If that exists, would that sufficiently help differentiate how this should be used?

[vassudanagunta]

"known issue" is well understood to mean a known bug that exists in released/production code, which is exactly the thing one would represent with this feature.

[ljharb]

Assertions that things will fail aren’t necessarily or inherently an issue - assert.throws, for example. This feature may be used for known issues, but it is perfectly valid to use it for intended failures as well.