$ the-wire · showcase
Spark 4.0 daemon workers and RDD execution land
By RepoJournal · Filed · About .NET
dotnet/spark enables default Linux daemon mode for Spark 4.0 .NET workers and adds RDD NON_UDF execution, while dotnet/runtime completes Wasm jump stub emission.
The Spark 4.0 .NET worker path advances in two pieces. A new daemon implementation for .NET 8 on Linux launches worker subprocesses with real OS PIDs, supports worker reuse and targeted cancellation, and cleans up child processes on shutdown or parent exit, removing the need to set `spark.python.use.daemon=false` [1]. Separately, the RDD command path now accepts `NON_UDF` (`PythonEvalType = 0`) worker commands while retaining `SQL_BATCHED_UDF` support and existing Spark 2.x/3.x behavior, with protocol unit and RDD E2E coverage added [2].
In dotnet/runtime, the Wasm `JumpStubNode` no longer throws `NotImplementedException` during emission. The stub now computes the lowered Wasm function type, forwards all arguments via `local.get`, and transfers control to the target, enabling address-taken method stubs [3]. Separately, the JIT now expands `REFANYTYPE` QMARK nodes before physical promotion, adding a debug check for candidate stores and retbuf definitions in QMARK arms, with regression coverage [4].
The wasm performance pipeline gains a dedicated CoreCLR browser-WASM R2R microbenchmark lane on V8, publishing per-assembly R2R results with a distinct `R2RType=r2r` dimension and failing fast if R2R settings are not applied [5]. This depends on runtime fixes that stage the complete same-build CoreCLR browser-WASM package cohort, resolving earlier workload installation failures [6].
Action items
- → Review Spark 4.0 daemon and NON_UDF changes if you run .NET workers on Spark 4.0 dotnet/spark [plan]
- → Verify Wasm jump stub emission in your runtime workload dotnet/runtime [monitor]
- → Monitor wasm-perf lane for CoreCLR R2R results dotnet/performance [monitor]
References
- [1] Add Spark 4.0 daemon worker lifecycle and reuse support ↗ dotnet/spark
- [2] Enable Spark 4.0 RDD NON_UDF execution ↗ dotnet/spark
- [3] Implement wasm JumpStubNode with signature-matched argument forwarding ↗ dotnet/runtime
- [4] JIT: Expand REFANYTYPE QMARK before physical promotion ↗ dotnet/runtime
- [5] Add CoreCLR WASM R2R performance lane ↗ dotnet/performance
- [6] [perf][wasm] fix wasm-perf lane, stage complete CoreCLR toolchain cohort ↗ dotnet/runtime
