$ the-wire · showcase
Elixir reworks test manifest for parameterized failures
By RepoJournal · Filed · About Elixir & Phoenix
4 people shipped this
A fix for parameterized test manifests changes how failures are tracked, ensuring all variants get rerun when any single one fails.
The test manifest now records parameterized test failures differently [1]. Previously, when variant 1, 2, 3, and 4 were run and only 2 failed, the manifest keyed on {module, test} would allow a later pass of a different variant to overwrite the failure, meaning the failing variant might not be rerun. Now "all parameterized tests marked as failing" are kept, so all variants get rerun, preventing sibling failures from being masked. This change conflicts with an earlier approach in [2] that aimed to solve the same issue; the newer PR supersedes it.
Phoenix has reorganized its integration test suites [3]. Tests previously lived in 7 monolithic files, which caused low concurrency at startup and idle worker tails. The suites are now split into focused async modules, better utilizing parallel test runners. Additionally, CI has been improved with "explicit database readiness checks" and more granular phasing [4].
Phoenix also fixes compiler warnings on older Elixir versions [5]. Code in `Phoenix.CodeReloader.Server` used `@requires_consolidation`, which on Elixir < 1.19 always evaluated to true, triggering warnings. The attribute is replaced with version-gated helper functions to eliminate the warnings. Elixir itself begins validating `@enforce_keys` against declared struct fields [6], which will likely make invalid usage a compile-time error in the future.
Action items
- → Monitor the test manifest fix in Elixir; expect a change in how parameterized failures are rerun. elixir-lang/elixir [monitor]
- → If you compile Phoenix with Elixir < 1.19, check that the new version-gated helpers work for your setup. phoenixframework/phoenix [plan]
References
- [1] fix: Test Manifest, parameterized do not override sibling failures ↗ elixir-lang/elixir
- [2] Do not overlap sibling failures in parameterized tests failures manifest ↗ elixir-lang/elixir
- [3] Split integration test suites into focused async modules ↗ phoenixframework/phoenix
- [4] Improve integration test CI visibility, healthchecks, and reliability ↗ phoenixframework/phoenix
- [5] Fix Elixir 1.15 compiler warnings ↗ phoenixframework/phoenix
- [6] Improve defstruct @enforce_keys ↗ elixir-lang/elixir
