Parallel convolutional processing using an integrated photonic tensor core (original) (raw)

References

1. Batra, G., Jacobson, Z., Madhav, S., Queirolo, A. & Santhanam, N. Artificial-intelligence hardware: new opportunities for semiconductor companies. https://www.mckinsey.com/industries/semiconductors/our-insights/artificial-intelligence-hardware-new-opportunities-for-semiconductor-companies (McKinsey & Company, 2019).
2. Ben-Nun, T. & Hoefler, T. Demystifying parallel and distributed deep learning: an in-depth concurrency analysis. ACM Comput. Surv. 52, https://doi.org/10.1145/3320060 (2019).
3. Herr, T. et al. Temporal solitons in optical microresonators. Nat. Photon. 8, 145–152 (2014).
   ADS CAS Google Scholar
4. Herr, T., Gorodetsky, M. L. & Kippenberg, T. J. Dissipative Kerr solitons in optical microresonators. In Nonlinear Optical Cavity Dynamics From Microresonators to Fiber Lasers (ed. Grelu, P.) Vol. 8083, Ch. 6, 129–162 (Wiley, 2015).
5. Raja, A. S. et al. Electrically pumped photonic integrated soliton microcomb. Nat. Commun. 10, 680 (2019).
   ADS CAS PubMed PubMed Central Google Scholar
6. Pfeiffer, M. H. P. et al. Photonic damascene process for integrated high-Q microresonator based nonlinear photonics. Optica 3, 20–25 (2016).
   ADS CAS Google Scholar
7. Liu, J. et al. Ultralow-power chip-based soliton microcombs for photonic integration. Optica 5, 1347–1353 (2019).
   ADS Google Scholar
8. Machine Learning on AWS https://aws.amazon.com/machine-learning/ (accessed 12 October 2020).
9. Google Cloud AI And Machine Learning Products https://cloud.google.com/products/machine-learning/ (accessed 12 October 2020).
10. Zhang, C. et al. Optimizing FPGA-based Accelerator Design for Deep Convolutional Neural Networks. In ACM/SIGDA Int. Symp. Field-Programmable Gate Arrays (FPGA ’15) https://doi.org/10.1145/2684746.2689060 (2015).
11. Jouppi, N. P. et al. In-datacenter performance analysis of a tensor processing unit. Proc. ISCA ’17 https://doi.org/10.1145/3079856.3080246 (2017).
12. Wang, P. S., Liu, Y., Guo, Y. X., Sun, C. Y. & Tong, X. O-CNN: octree-based convolutional neural networks for 3D shape analysis. ACM Trans. Graph. 36, https://doi.org/10.1145/3072959.3073608 (2017).
13. Miller, D. A. B. Attojoule optoelectronics for low-energy information processing and communications. J. Lightwave Technol. 35, 346–396 (2017).
    ADS CAS Google Scholar
14. Agrawal, S. R. et al. A many-core architecture for in-memory data processing. In Proc. 50th Annu. IEEE/ACM Int. Symp. Microarchitecture (MICRO-50 ’17) 245–258, https://doi.org/10.1145/3123939.3123985 (IEEE/ACM, 2017).
15. Miller, D. A. B. Are optical transistors the logical next step? Nat. Photon. 4, 3–5 (2010).
    ADS CAS Google Scholar
16. Ielmini, D. & Wong, H. S. P. In-memory computing with resistive switching devices. Nat. Electron. 1, 333–343 (2018).
    Google Scholar
17. Le Gallo, M. et al. Mixed-precision in-memory computing. Nat. Electron. 1, 246–253 (2018).
    Google Scholar
18. Boybat, I. et al. Neuromorphic computing with multi-memristive synapses. Nat. Commun. 9, 2514 (2018).
    ADS PubMed PubMed Central Google Scholar
19. Sebastian, A., Le Gallo, M., Khaddam-Aljameh, R. & Eleftheriou, E. Memory devices and applications for in-memory computing. Nat. Nanotechnol. 15, 529–544 (2020).
    ADS CAS PubMed Google Scholar
20. Hu, M. et al. Dot-product engine for neuromorphic computing: programming 1T1M crossbar to accelerate matrix-vector multiplication. In Proc. 53rd Annu. Design Automation Conf. (DAC ’16) https://doi.org/10.1145/2897937.2898010 (ACM Digital Library, 2016).
21. Gong, N. et al. Signal and noise extraction from analog memory elements for neuromorphic computing. Nat. Commun. 9, 2102 (2018).
    ADS CAS PubMed PubMed Central Google Scholar
22. Joshi, V. et al. Accurate deep neural network inference using computational phase-change memory. Nat. Commun. 11, 2473 (2020).
    ADS CAS PubMed PubMed Central Google Scholar
23. Yang, T. Y., Park, I. M., Kim, B. J. & Joo, Y. C. Atomic migration in molten and crystalline Ge2Sb2Te5 under high electric field. Appl. Phys. Lett. 95, 032104 (2009).
    ADS Google Scholar
24. Koelmans, W. W. et al. Projected phase-change memory devices. Nat. Commun. 6, 8181 (2015).
    ADS PubMed Google Scholar
25. Kim, S. et al. A phase change memory cell with metallic surfactant layer as a resistance drift stabilizer. In 2013 IEEE Int. Electron Devices Meeting https://doi.org/10.1109/IEDM.2013.6724727 (IEEE, 2013).
26. Bell, T. E. Optical computing: a field in flux: a worldwide race is on to develop machines that compute with photons instead of electrons but what is the best approach? IEEE Spectr. 23, 34–38 (1986).
    ADS Google Scholar
27. Hamerly, R., Bernstein, L., Sludds, A., Soljačić, M. & Englund, D. Large-scale optical neural networks based on photoelectric multiplication. Phys. Rev. X 9, 021032 (2018).
    Google Scholar
28. Silva, A. et al. Performing mathematical operations with metamaterials. Science 343, 160–163 (2014).
    ADS MathSciNet CAS PubMed MATH Google Scholar
29. Lin, X. et al. All-optical machine learning using diffractive deep neural networks. Science 361, 1004–1008 (2018).
    ADS MathSciNet CAS PubMed MATH Google Scholar
30. Colburn, S., Chu, Y., Shilzerman, E. & Majumdar, A. Optical frontend for a convolutional neural network. Appl. Opt. 58, 3179–3186 (2019).
    ADS PubMed Google Scholar
31. Shen, Y. et al. Deep learning with coherent nanophotonic circuits. Nat. Photon. 11, 441–446 (2017).
    ADS CAS Google Scholar
32. Tait, A. N. et al. Silicon photonic modulator neuron. Phys. Rev. Appl. 11, 064043 (2019).
    ADS CAS Google Scholar
33. Pérez, D. et al. Multipurpose silicon photonics signal processor core. Nat. Commun. 8, 636 (2017).
    ADS PubMed PubMed Central Google Scholar
34. Galal, S. & Horowitz, M. Energy-efficient floating-point unit design. IEEE Trans. Comput. 60, 913–922 (2011).
    MathSciNet MATH Google Scholar
35. Bangari, V. et al. Digital electronics and analog photonics for convolutional neural networks (DEAP-CNNs). IEEE J. Sel. Top. Quantum Electron. 26, https://doi.org/10.1109/JSTQE.2019.2945540 (2020).
36. LeCun, Y., Cortes, C. & Borges, C. J. C. The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist.
37. Stern, B., Ji, X., Okawachi, Y., Gaeta, A. L. & Lipson, M. Battery-operated integrated frequency comb generator. Nature 562, 401–405 (2018).
    ADS CAS PubMed Google Scholar
38. Jones, R. et al. Heterogeneously integrated InP/silicon photonics: fabricating fully functional transceivers. IEEE Nanotechnol. Mag. 13, 17–26 (2019).
    Google Scholar
39. Marin-Palomo, P. et al. Microresonator-based solitons for massively parallel coherent optical communications. Nature 546, 274–279 (2017).
    ADS CAS PubMed Google Scholar
40. Spencer, D. T. et al. An optical-frequency synthesizer using integrated photonics. Nature 557, 81–85 (2018).
    ADS CAS PubMed Google Scholar
41. Riemensberger, J. et al. Massively parallel coherent laser ranging using soliton microcombs. Nature 581, 164–170 (2019).
    ADS Google Scholar
42. Moss, D. J., Morandotti, R., Gaeta, A. L. & Lipson, M. New CMOS-compatible platforms based on silicon nitride and Hydex for nonlinear optics. Nat. Photon. 7, 597–607 (2013).
    ADS CAS Google Scholar
43. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In 2016 Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90 (IEEE, 2016).
44. Simonyan, K. & Zisserman, A. Very deep convolutional networks for large-scale image recognition. In 3rd Int. Conf. Learning Representations (ICLR 2015) (eds Bengio, Y. & LeCun, Y.) 4 (2015); https://arxiv.org/abs/1409.1556.
45. Al-Ashrafy, M., Salem, A. & Anis, W. An efficient implementation of floating point multiplier. In 2011 Saudi Int. Electronics, Communications and Photonics Conf. (SIECPC) https://doi.org/10.1109/SIECPC.2011.5876905 (2011).
46. Gao, L., Chen, P. Y. & Yu, S. Demonstration of convolution kernel operation on resistive cross-point array. IEEE Electron Device Lett. 37, 870–873 (2016).
    ADS Google Scholar
47. Shafiee, A. et al. ISAAC: a convolutional neural network accelerator with in-situ analog arithmetic in crossbars. In Proc. 2016 43rd Int. Symp. Computer Architecture (ISCA 2016) https://doi.org/10.1109/ISCA.2016.12 (2016).
48. Li, X. et al. Fast and reliable storage using a 5 bit, nonvolatile photonic memory cell. Optica 6, 1–6 (2019).
    ADS Google Scholar
49. Ríos, C. et al. Integrated all-photonic non-volatile multi-level memory. Nat. Photon. 9, 725–732 (2015).
    ADS Google Scholar
50. Feldmann, J. et al. Calculating with light using a chip-scale all-optical abacus. Nat. Commun. 8, 1256 (2017).
    ADS CAS PubMed PubMed Central Google Scholar
51. Gehring, H. et al. Low-loss fiber-to-chip couplers with ultrawide optical bandwidth. APL Photon. 4, 010801 (2019).
    ADS Google Scholar
52. Gehring, H., Eich, A., Schuck, C. & Pernice, W. H. P. Broadband out-of-plane coupling at visible wavelengths. Opt. Lett. 44, 5089 (2019).
    ADS CAS PubMed Google Scholar
53. Nahmias, M. A. et al. Photonic multiply-accumulate operations for neural networks. IEEE J. Sel. Top. Quantum Electron. https://doi.org/10.1109/jstqe.2019.2941485 (2019).
    Article Google Scholar
54. Gehring, H., Blaicher, M., Hartmann, W. & Pernice, W. H. P. Python based open source design framework for integrated nanophotonic and superconducting circuitry with 2D-3D-hybrid integration. OSA Continuum 2, 3091–3101 (2019).
    CAS Google Scholar
55. Guo, H. et al. Universal dynamics and deterministic switching of dissipative Kerr solitons in optical microresonators. Nat. Phys. 13, 94–102 (2017).
    CAS Google Scholar
56. Karpov, M. et al. Dynamics of soliton crystals in optical microresonators. Nat. Phys. 15, 1071–1077 (2019).
    CAS Google Scholar
57. Fialka, O. & Čadík, M. FFT and convolution performance in image filtering on GPU. In Proc. 10th Int. Conf. Information Visualisation (IV’06) https://doi.org/10.1109/IV.2006.53 (IEEE, 2006).
58. Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional neural networks. Commun. ACM 60, https://doi.org/10.1145/3065386 (2017).
59. Szegedy, C. et al. Going deeper with convolutions. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2015.7298594 (IEEE, 2015).
60. Ríos, C. et al. In-memory computing on a photonic platform. Sci. Adv. 5, eaau5759 (2019).
    ADS PubMed PubMed Central Google Scholar
61. Gaeta, A. L., Lipson, M. & Kippenberg, T. J. Photonic-chip-based frequency combs. Nat. Photon. 13, 158–169 (2019).
    ADS CAS Google Scholar
62. Ma, Y. et al. Ultralow loss single layer submicron silicon waveguide crossing for SOI optical interconnect. Opt. Express 21, 29374–29382 (2013).
    ADS PubMed Google Scholar
63. Lu, Z. et al. Broadband silicon photonic directional coupler using asymmetric-waveguide based phase control. Opt. Express 23, 3795–3808 (2015).
    ADS CAS PubMed Google Scholar
64. Farmakidis, N. et al. Plasmonic nanogap enhanced phase change devices with dual electrical-optical functionality. Sci. Adv. 5, eaaw2687 (2019).
    ADS CAS PubMed PubMed Central Google Scholar
65. Zhang, H. et al. Miniature multilevel optical memristive switch using phase change material. ACS Photon. 6, 2205–2212 (2019).
    CAS Google Scholar
66. Atabaki, A. H. et al. Integrating photonics with silicon nanoelectronics for the next generation of systems on a chip. Nature 556, 349–354 (2018).
    ADS CAS PubMed Google Scholar
67. Wang, X. & Liu, J. Emerging technologies in Si active photonics. J. Semicond. 39, 061001 (2018).
    ADS Google Scholar
68. Sun, J., Timurdogan, E., Yaacobi, A., Hosseini, E. S. & Watts, M. R. Large-scale nanophotonic phased array. Nature 493, 195–199 (2013).
    ADS CAS PubMed Google Scholar

Download references