[release/9.0] Fix duplicate error.type on kestrel.connection.duration by github-actions[bot] · Pull Request #57581 · dotnet/aspnetcore (original) (raw)

Backport of #57561 to release/9.0

/cc @JamesNK

Fix duplicate error.type on kestrel.connection.duration

kestrel.connection.duration is a metric introduced in .NET 8. It captures how long a connection lasted along with connection metadata. In .NET 9, the metric now records now more information about connection errors.

The issue is the error.type tag could be added to the metric twice, which causes some tooling in the OTEL ecosystem to error.

The fix is to ensure that only one error.type tag is present.

Fixes #57558

Customer Impact

Tooling that uses metrics, such as Prometheus, will error if there are duplicate tag. The customer must be forced to disable that metric.

This issue was introduced in the last .NET 9 preview and there is already one customer reporting it.

Regression?

Although the extra detail in error.type is new in .NET 9, customers successfully using kestrel.connection.duration in .NET 8 will be broken after upgrading to .NET 9.

Risk

Product change is to double check if error.type is present before adding it.

Verification

Manual testing:

Before: (two error.type values per line in result)

Timestamp,Provider,Counter Name,Counter Type,Mean/Increment
1/9/2024 10:00:36 pm,Microsoft.AspNetCore.Server.Kestrel,kestrel.connection.duration (s)[error.type=invalid_request_line;error.type=invalid_request_line;network.protocol.name=http;network.protocol.version=1.1;network.transport=tcp;network.type=ipv4;server.address=127.0.0.1;server.port=5001;Percentile=50],Metric,0.0296630859375
1/9/2024 10:00:36 pm,Microsoft.AspNetCore.Server.Kestrel,kestrel.connection.duration (s)[error.type=invalid_request_line;error.type=invalid_request_line;network.protocol.name=http;network.protocol.version=1.1;network.transport=tcp;network.type=ipv4;server.address=127.0.0.1;server.port=5001;Percentile=95],Metric,0.0296630859375
1/9/2024 10:00:36 pm,Microsoft.AspNetCore.Server.Kestrel,kestrel.connection.duration (s)[error.type=invalid_request_line;error.type=invalid_request_line;network.protocol.name=http;network.protocol.version=1.1;network.transport=tcp;network.type=ipv4;server.address=127.0.0.1;server.port=5001;Percentile=99],Metric,0.0296630859375

After: (one error.type value per line in result)

Timestamp,Provider,Counter Name,Counter Type,Mean/Increment
1/9/2024 9:57:02 pm,Microsoft.AspNetCore.Server.Kestrel,kestrel.queued_connections ({connection})[network.transport=tcp;network.type=ipv4;server.address=127.0.0.1;server.port=5001],Metric,0
1/9/2024 9:57:02 pm,Microsoft.AspNetCore.Server.Kestrel,kestrel.connection.duration (s)[error.type=invalid_request_line;network.protocol.name=http;network.protocol.version=1.1;network.transport=tcp;network.type=ipv4;server.address=127.0.0.1;server.port=5001;Percentile=50],Metric,0.0088043212890625
1/9/2024 9:57:02 pm,Microsoft.AspNetCore.Server.Kestrel,kestrel.connection.duration (s)[error.type=invalid_request_line;network.protocol.name=http;network.protocol.version=1.1;network.transport=tcp;network.type=ipv4;server.address=127.0.0.1;server.port=5001;Percentile=95],Metric,0.0088043212890625
1/9/2024 9:57:02 pm,Microsoft.AspNetCore.Server.Kestrel,kestrel.connection.duration (s)[error.type=invalid_request_line;network.protocol.name=http;network.protocol.version=1.1;network.transport=tcp;network.type=ipv4;server.address=127.0.0.1;server.port=5001;Percentile=99],Metric,0.0088043212890625

Packaging changes reviewed?

When servicing release/2.1