Vectorize rotate even better by AlexGuteniev · Pull Request #5525 · microsoft/STL (original) (raw)

Follow up for #5502.

Reasons to consider follow up:

• Some cases have no improvement
• After thinking about it, discovered that there are even cases with slight degradation

Weakness of the original approach

It deals well with extremes:

• Small rotation
• Rotation close to the middle

The worst case is when the rotation is small, but still large enough to not engage the small rotation branch.

Mitigation approaches

Generally, we need to do multi-range rotating swap to make fewer element assignments. From the original PR:

A hypothetical functions like swap_3_ranges, swap_4_ranges, etc could reduce the number of assignments for more cases. But going further in optimization will result in less and less improvement for more and more code added, and at some point will cause the complex decisions to take noticeable amount of time, resulting in negative improvement, so we need to stop somewhere. Probably stopping on just small rotation and two ranges swap strategy would be a good idea.

So how we can do some improvement while avoiding unnecessary complication:

• Implement few of swap_3_ranges, swap_4_ranges, etc, but no more than two of them as separate functions.
  • ❌ Will not squeeze away unnecessary assignments too hard
  • ✅ This should be the easiest thing to do
• Spawn many swap_N_ranges using single source and metaprogramming, pick the best one at runtime
  • ✅ Will squeeze away unnecessary assignments harder
  • ❌ Complex metaprogramming to do that using template fold expression or macros
  • ❌ In case of template implementation will heavily rely on compiler optimization
  • 🤮 Macro implementation is just not a good thing
  • ❌ Will add a lot of machine code, which will make binary bigger, and there will be a lot of "cold" code at runtime
• Implement single swap_N_ranges that would work with variable at runtime number of ranges
  • ✅ Will squeeze away unnecessary assignments to the maximum
  • ❓ Swapping too many ranges at the same time is likely to break prefetch, need to see the impact of that
  • ❓ Will have additional runtime cost for iterating over pointers to iterate
  • ⚠️ Large power-of-two stride will cause cache conflict eviction, if N exceeds CPU cache associativity, which would result in dramatic performance degradation
  • ❌ Will have complex flow at runtime

This makes me think that it would be good to:

• First try the simple approach of having one or two additional swap functions
• If there's a strong indication of success in this direction, try the runtime-variable swap_N_ranges

The code chages

So I've tried _Swap_3_ranges, It resulted in at most 1.40 speedup, and that fixed the slightly regressed cases.
I think it is indication to both that the approach is good enough to use, and not too good to try something more complex.

I've moved _Rotating closer to __std_swap_ranges_trivially_swappable_noalias to make the similarity between that and _Swap_3_ranges more obvious.

Coverage

Tests were lacking too long arrays to execute the ranges swapping properly. I've expanded the test to have more elements; to save some run time, I've did this for one of 8-bit elements only. The algorithm does not distinguish element sizes internally anyway.

The same for benchmark, I've added just two examples of the case that became worse.

Benchmark results

Before #5502 / After #5502 may slightly wary from the previous PR description, I've ran the benchmarks again.