Approaches to testing macros (original) (raw)

Hey everyone! I created MetaCodable to handle common Codable implementation generation use-cases. Currently the issue I face is better testability with the huge set of features this library supports, one of the thing I feel sorely lacking in macro testing approaches is asserting generated macro code compiles.

To address this I was thinking of creating snippets for each tests and grouping snippets based on test cases. Going through the Snippets pitch , I couldn't find any commands to run all the snippets or run snippets in parallel. Is running all the snippets possible with a single command?

What are the approaches you follow to test macro generated code compiles?

rauhul (Rauhul Varma) June 10, 2024, 2:31am 2

In swift-mmio I include sample usages of the macros in a test target to ensure the expanded code type checks and compiles properly: swift-mmio/Tests/MMIOTests/ExpansionTypeCheckTests.swift at main · apple/swift-mmio · GitHub

Personally I find myself wanting to see @pointfree.co's macro testing support added to swift-syntax. I like that their library can autofill the expanded code which reduces the mechanical work needed for complex macros.

soumyamahunt (Soumya Ranjan Mahunt) June 19, 2024, 6:04pm 3

Thanks @rauhul, but I wanted to exactly avoid the approach suggested as the seer amount of combinations supported by MetaCodable will require multiple different samples all with different names. This will also be difficult to maintain as the library scales with more features.

With the new Swift testing framework here is the approach I went with structuring tests:

struct TestCase {
    @Test
    func testErrorsWithMacro() {}

    struct MacroUseCase {
        struct Sample {}

        @Test
        func assertSampleExpansion() {}
    }
}

• Here TestCase represents a specific macro/use-case scenario I am validating.
• It can contain multiple testErrorsWithMacro which asserts proper diagnostic errors/warnings generated by the macro in invalid usages.
• For each usage scenario of the macro one MacroUseCase type can be created.
  • It will contain a simple Sample representing the usage scenario, this will allow verification of any build errors.
  • It will contain one assertSampleExpansion that will assert the expanded code, allowing verification of the usage scenario.
  • Optionally additional tests can be added verifying the generated code behaves as expected.

mbrandonw (Brandon Williams) June 19, 2024, 6:43pm 4

We have found that a hybrid approach typically needs to be taken, as @rauhul mentioned. We will have a file that just lists out a bunch of usages of our macros to make sure that they at least compile, though there is no verification they are "correct". And then another file that uses our assertMacro helper to ensure that the expanded macro code looks correct.

I'm not sure if there is really much one can do to improve on this hybrid approach, but if there is I'd love to see it!

soumyamahunt (Soumya Ranjan Mahunt) June 20, 2024, 9:55am 5

@mbrandonw I am not denying hybrid approach needs to be taken which is what I want to have as well. I have updated my answer with more details how it allows me improve upon the hybrid approach suggested.