A better way to deal with tabular test cases

[dabrahams]

Description

Here's an example I'm working with:

let regularCases: [String: [(input: String, expected: Bool)]] = [
  // Basic cases
  "": [("", true), ("x", false), ("xy", false)],
  "x": [("", false), ("x", true), ("xy", false)],
  "x+": [("", false), ("x", true), ("xy", false), ("xx", true)],
  "x*": [("", true), ("x", true), ("xy", false), ("xx", true)],
  "x?": [("", true), ("x", true), ("xy", false), ("xx", false)],
  "x|y": [("", false), ("x", true), ("y", true), ("xx", false)],
  
  // Nested groups
  "(xy)+": [("", false), ("xy", true), ("xyxy", true), ("x", false)],
  "(x|y)*": [("", true), ("x", true), ("y", true), ("xy", true), ("yx", true), ("xyxy", true)],
  
  // Complex combinations
  "x(y|z)+": [("", false), ("x", false), ("xy", true), ("xz", true), ("xyz", true), ("xyzyz", true)],
  "(ab|cd)*": [("", true), ("ab", true), ("cd", true), ("abcd", true), ("cdab", true), ("abc", false)],
  
  // Multiple alternatives
  "a|b|c": [("", false), ("a", true), ("b", true), ("c", true), ("d", false), ("ab", false)],
  "(x|y)(a|b)": [("xa", true), ("xb", true), ("ya", true), ("yb", true), ("xx", false), ("ab", false)]
]


@Test func nfaToDfa() async throws {

  for (pattern, expectations) in regularCases {
    let n = TestNFA(pattern)
    let d = EquivalentDFA(n)
    for (input, expectedMatch) in expectations {
      #expect(n.recognizes(input) == expectedMatch, "pattern: \(pattern), input: \(input), nfa:\n\(n)")
      #expect(d.recognizes(input) == expectedMatch, "pattern: \(pattern), input: \(input), dfa:\n\(d)")
    }
  }
}

When one of these fails, it of course points to one of the two expect lines; that's not really what I want at all. Plus, I have to add these comments to get useful information out. Maybe there's a different idiom that would work better, in which case document it? Or maybe different library features are needed.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[grynspan]

Sorry, but it's not clear what you're asking for here. Please clarify—if there's something actionable here, we can reopen this issue.

[grynspan]

Might #840 help?

[stmontgomery]

I think the feature you're looking for is what we call Parameterized Tests. See https://swiftpackageindex.com/swiftlang/swift-testing/6.0.3/documentation/testing

[grynspan]

Oh hey yeah. I think my mind filled in arguments: [...] automatically.

[dabrahams]

OK, I tried that, but it's not really satisfactory. There are two problems:

1. (minor) I get output like ◇ Passing 1 argument patternAndExpectations → (key: "a|b|c", value: [(input: "", expected: false), (input: "a", expected: true), (input: "b", expected: true), (input: "c", expected: true), (input: "d", expected: false), (input: "ab", expected: false)]) to nfaToDfa(patternAndExpectations:) for each of the cases; that's very noisy!
2. (major) I still have to loop over the expectations for each pattern. The arguments: parameter gives me cross-products of elements, which isn't what I need.

[grynspan]

If you're using swift test, you can pass --quiet to reduce output. In Xcode, look at the test report.

If you don't want all combinations of inputs, you can use zip() or just pass tuples of inputs you want to test—or pass most any other collection, if you prefer. Please review the documentation for parameterized testing for more info.

[grynspan]

(.flatMap may be your friend here.)

[dabrahams]

Exactly how could that help me?

[dabrahams]

BTW, I read the documentation. I don't think you are understanding the problem here. The looping is unavoidable in this case. Clone the project yourself and see if it isn't obvious from GitHub. Once you start looping over things in a test case, the feedback you get about what failed and why is poor, so you end up writing large "comment" parameters to the #expect to try to extract information. That would seem to be the very reason you implemented parameterized testing. But it's only structured to work on cross products of collections, not a collection with a dependent collection for each element (the data structure is a tree).