An optimized RDMA QP communication mechanism for hyperscale AI infrastructure (original) (raw)

References

  1. Chen, G., Lu, Y., Li, B., et al.: MP-RDMA: enabling RDMA with multi-path transport in datacenters. IEEE/ACM Trans. Netw. 27(6), 2308–2323 (2019)
    Article MATH Google Scholar
  2. Chen, Y., Tian, C., Dong, J., et al.: SWING: providing long-range lossless RDMA via PFC-relay. IEEE Trans. Parallel Distrib. Syst. 34(1), 63–75 (2023). https://doi.org/10.1109/TPDS.2022.3215517
    Article Google Scholar
  3. Choi, M., Lee, S., Kim, Y.: UD-assisted multi-path transport in RDMA. In: 2022 13th International Conference on Information and Communication Technology Convergence (ICTC), 2022, pp. 127–129. IEEE (2022a)
  4. Choi, M., Lee, S., Kim, Y.: UD-assisted multi-path transport in RDMA. In: 2022 13th International Conference on Information and Communication Technology Convergence (ICTC), 2022, pp. 127–129 (2022b). https://doi.org/10.1109/ICTC55196.2022.9952631
  5. Fent, P., van Renen. A., Kipf, A., et al.: Low-latency communication for fast DBMS using RDMA and shared memory. In: 2020 IEEE 36th International Conference on Data Engineering (ICDE), 2020, pp. 1477–1488. IEEE (2020)
  6. Gao, Y.X., Tian, C., Chen, W., et al.: Analyzing and optimizing packet corruption in RDMA network. J. Comput. Sci. Technol. 37(4), 743–762 (2022). https://doi.org/10.1007/s11390-022-2123-8
    Article MATH Google Scholar
  7. Guo, Z., Liu, S., Zhang, Z.L.: Traffic control for RDMA-enabled data center networks: a survey. IEEE Syst. J. 14(1), 677–688 (2019)
    Article MATH Google Scholar
  8. He, Z., Chen, Y., Hua, B.: RoUD: scalable RDMA over UD in lossy data center networks. In: 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid), 2023, pp. 36–46. IEEE (2023)
  9. Hu, J., Zeng, C., Wang, Z., et al.: Load balancing in PFC-enabled datacenter networks. In: Proceedings of the 6th Asia–Pacific Workshop on Networking, 2022, pp. 21–28 (2022)
  10. Hu, J., Zeng, C., Wang, Z., et al.: Load balancing with multi-level signals for lossless datacenter networks. IEEE/ACM Trans. Netw. (2024). https://doi.org/10.1109/TNET.2024.3366336
    Article MATH Google Scholar
  11. Infiniband: Infiniband Architecture Specification, vol. 1. Infiniband (2022) . https://cw.infinibandta.org/document/dl/8567
  12. Jia, C., Liu, J., Jin, X., et al.: Improving the performance of distributed tensorflow with RDMA. Int. J. Parallel Program. 46, 674–685 (2018)
    Article MATH Google Scholar
  13. Kang, N., Wang, Z., Yang, F., et al.: csRNA: connection-scalable RDMA NIC architecture in datacenter environment. In: 2022 IEEE 40th International Conference on Computer Design (ICCD), 2022, pp. 398–406. IEEE (2022)
  14. Lee, S., Kim, Y., Woo, H., et al.: Efficient user-level multi-path utilization in RDMA networks. IEEE Access 9, 127619–127629 (2021)
    Article MATH Google Scholar
  15. Li, M., Wang, S., Huang, T., et al.: MPTD: optimizing multi-path transport with dynamic target delay in datacenters. Clust. Comput. (2024). https://doi.org/10.1007/s10586-024-04470-y
    Article MATH Google Scholar
  16. Li, Z., Huang, J., Wang, S., et al.: Achieving low latency for multipath transmission in RDMA based data center network. IEEE Trans. Cloud Comput. 12(1), 337–346 (2024). https://doi.org/10.1109/TCC.2024.3365075
    Article MATH Google Scholar
  17. Lu, Y., Chen, G., Li, B., et al.: Multi-path transport for RDMA in datacenters. In: 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), 2018, pp. 357–371 (2018)
  18. Ma, S., Ma, T., Chen, K., et al.: A survey of storage systems in the RDMA era. IEEE Trans. Parallel Distrib. Syst. 33(12), 4395–4409 (2022)
    Article MATH Google Scholar
  19. Ma, T., Chen, K., Ma, S., et al.: Thinking more about RDMA memory semantics. In: 2021 IEEE International Conference on Cluster Computing (CLUSTER), 2021, pp. 456–467. IEEE (2021)
  20. NVIDIA: Mellanox Adapters Programmer’s Reference Manual (PRM). NVIDIA (2022). https://network.nvidia.com/related-docs/usermanuals/Ethernet_Adapters_Programming_Manual.pdf
  21. Park, J., Son, Y., Yeom, H.Y., et al.: SoftDC: software-based dynamically connected transport. Clust. Comput. J. Netw. Softw. Tools Appl. 23(1, SI), 347–357 (2020). https://doi.org/10.1007/s10586-019-02926-0
    Article Google Scholar
  22. Park, J., Son, Y., Yeom, H.Y., et al.: SoftDC: software-based dynamically connected transport. Clust. Comput. 23, 347–357 (2020)
    Article Google Scholar
  23. Pathak, A.R., Pandey, M., Rautaray, S.S.: Approaches of enhancing interoperations among high performance computing and big data analytics via augmentation. Clust. Comput. J. Netw. Softw. Tools Appl. 23(2), 953–988 (2020). https://doi.org/10.1007/s10586-019-02960-y
    Article MATH Google Scholar
  24. Shen, D., Luo, J., Dong, F., et al.: Enabling distributed and optimal RDMA resource sharing in large-scale data center networks: modeling, analysis, and implementation. IEEE/ACM Trans. Netw. 31(6), 2745–2760 (2023)
    Article MATH Google Scholar
  25. Shu, J., Chen, Y., Wang, Q., et al.: TH-DPMS: design and implementation of an RDMA-enabled distributed persistent memory storage system. ACM Trans. Storage 16(4), 1–31 (2020)
    Article MATH Google Scholar
  26. Stewart, I.A., Erickson, A.: The influence of datacenter usage on symmetry in datacenter network design. J. Supercomput. 74(6), 2276–2313 (2018). https://doi.org/10.1007/s11227-017-2217-1
    Article MATH Google Scholar
  27. Sur, S., Jin, H.W., Chai, L., et al.: RDMA read based rendezvous protocol for MPI over Infiniband: design alternatives and benefits. In: Proceedings of the Eleventh ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, 2006, pp. 32–39 (2006)
  28. Tang, J., Wang, X., Dai, H.: Scalable RDMA transport with efficient connection sharing. In: IEEE INFOCOM 2023—IEEE Conference on Computer Communications, 2023, pp 1–10. IEEE (2023)
  29. Taranov, K., Di Girolamo, S., Hoefler, T.: CoRM: compactable remote memory over RDMA. In: Proceedings of the 2021 International Conference on Management of Data, 2021, pp. 1811–1824 (2021)
  30. Wang, J., Huang, J.: ERD (2023). https://www.github.com/WangJunliang1/ERD
  31. Wang, J., Lin, B.: RDMA reliability evaluation model for large-scale data center networks. In: 2023 IEEE 3rd International Conference on Computer Communication and Artificial Intelligence (CCAI), 2023, pp. 342–347. IEEE (2023)
  32. Wang, X., Chen, G., Yin, X., et al.: StaR: Breaking the scalability limit for RDMA. In: 2021 IEEE 29th International Conference on Network Protocols (ICNP), 2021, pp. 1–11. IEEE (2021)
  33. Wang, Z., Luo, L., Ning, Q., et al.: SRNIC: a scalable architecture for RDMA NICs. In: 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023, pp. 1–14 (2023)
  34. Yang, H., Xu, C., Han, Y., et al.: An efficient cloud-based elastic RDMA protocol for HPC applications. CCF Trans. High Perform. Comput. 6(1), 45–53 (2024). https://doi.org/10.1007/s42514-023-00170-y
    Article MATH Google Scholar
  35. Zhang, J., Wang, Y., Zhong, X., et al.: PACC: a proactive CNP generation scheme for datacenter networks. IEEE–ACM Trans. Netw. (2024). https://doi.org/10.1109/TNET.2024.3361771
    Article MATH Google Scholar
  36. Zhang, Z., Liu, Z., Jiang, Q., et al.: RDMA-based apache storm for high-performance stream data processing. Int. J. Parallel Program. 49(5, SI), 671–684 (2021). https://doi.org/10.1007/s10766-021-00696-0
    Article MATH Google Scholar
  37. Zhuge, Q., Zhang, H., Sha, E.H.M., et al.: Exploring efficient architectures on remote in-memory NVM over RDMA. ACM Trans. Embed. Comput. Syst. (2021). https://doi.org/10.1145/3477004
    Article MATH Google Scholar

Download references