runtime (original) (raw)

Backport of #107333 to release/9.0-staging

Customer Impact

Customer reported
Found internally

Users running tracing in high-throughput scenarios may experience lock contention, which can lead to performance degradation in their applications or services.

The issue was reported by the ASP.NET/Aspire team while enabling CRUD operations over a PostgreSQL database with OpenTelemetry, resulting in a 15% drop in requests per second. Benchmark numbers are available in dotnet/runtime issue #107158.
A customer also reported that in their high-throughput services, lock contention was contributing significantly (around 10–20%) to service latency. The issue Consider releasing #107333 into dotnet 9 #109060 has more details and benchmark numbers.

Regression

While this isn't a regression, it appears that users have started enabling scenarios that lead to this issue. We may expect this get reported more in the near future if we don't service this fix.

Testing

Manual testing, #107333 (comment) has some numbers before and after fix. Also, we have provided a private built package to the customer who reported the issue, and they have confirmed the fix resolve their issue. #109060 (comment)

Risk

Medium to low risk. The whole change is touching the type we use for caching which was causing the lock contention. The fix didn't change any actual functional behavior more than avoiding the lock when notifying the listeners.

IMPORTANT: If this backport is for a servicing release, please verify that:

The PR target branch is release/X.0-staging, not release/X.0.
If the change touches code that ships in a NuGet package, you have added the necessary package authoring and gotten it explicitly reviewed.