Make SocketManager worker count configurable by naile · Pull Request #1115 · StackExchange/StackExchange.Redis (original) (raw)

Sure. Our current scenario is a message processing system. We're trying to achieve a good balance for throughput/CPU usage. We process many small messages concurrently. Each message will incur ~5-10 Redis operations, all async. When trying to find the sweet spot for how many consumers to run concurrently with different settings these are our findings. Processed messages / second is the metric we care about. Was measured on our production setup, Azure P3 Redis, not clustered. Processing on a 4 CPU machine, dotnet core 2.2 console app using latest version of StackExchange.Redis. When testing we are the sole consumer of this Redis instance.

Single Multiplexer: (using the 10 worker Shared SocketManager)
1 consumer = 67/s
2 consumers = 150/s
3 consumers = 155/s
4 consumers = 215/s
6 consumers = 230/s
8 consumers = 270/s
12 consumers = 290/s
16 consumers = 320/s

2 Multiplexers (using the 10 worker Shared SocketManager)
4 consumers = 280/s
8 consumers = 450/s
16 consumers = 530/s

2 Multiplexers (using the 5 worker specific SocketManager)
8 consumers = frequent crashes with TimeoutExceptions
16 consumers = frequent crashes with TimeoutExceptions

4 Multiplexers (using the 5 worker specific SocketManager)
8 consumers = 480/s
10 consumers = 540/s <----- sweetspot? Had very high CPU usage though.
16 consumers = frequent crashes with TimeoutExceptions

Single Multiplexer with specific SocketManager with 16 workers (created via reflection)
4 consumers = 460/s
8 consumers = 630/s
16 consumers = 1020/s <---- same CPU usage as 10 consumers with 4 multiplexers

So for our use case it seems a single Multiplexer with a SocketManager that has a larger thread pool is a lot more performant. It's not TimeoutExceptions that's our main issue, even though we can get them when using a smaller SocketManager worker pool.
When we get them they usually looks like this with 0 available mgr:

The timeout was reached before the message could be written to the output buffer, and it was not sent, timeout: 5000, outbound: 0KiB, inbound: 0KiB, inst: 1, qu: 1, qs: 1, aw: True, bw: RecordingTimeout, rs: ReadAsync, ws: Idle, in: 0, serverEndpoint: Unspecified/OMITTED-URL:6380, mgr: 0 of 5 available, clientName: Cint.Statistics.Service, IOCP: (Busy=0,Free=1000,Min=16,Max=1000), WORKER: (Busy=2,Free=681,Min=16,Max=682), v: 2.0.593.37019 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)