Pongpatapee Peerapatanapokin - Academia.edu (original) (raw)

Papers by Pongpatapee Peerapatanapokin

Research paper thumbnail of Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability

arXiv (Cornell University), Mar 4, 2023

Training deep neural networks (DNNs) takes significant time and resources. A practice for expedit... more Training deep neural networks (DNNs) takes significant time and resources. A practice for expedited deployment is to use pre-trained deep neural networks (PTNNs), often from model zoos-collections of PTNNs; yet, the reliability of model zoos remains unexamined. In the absence of an industry standard for the implementation and performance of PTNNs, engineers cannot confidently incorporate them into production systems. As a first step, discovering potential discrepancies between PTNNs across model zoos would reveal a threat to model zoo reliability. Prior works indicated existing variances in deep learning systems in terms of accuracy. However, broader measures of reliability for PTNNs from model zoos are unexplored. This work measures notable discrepancies between accuracy, latency, and architecture of 36 PTNNs across four model zoos. Among the top 10 discrepancies, we find differences of 1.23%-2.62% in accuracy and 9%-131% in latency. We also find mismatches in architecture for well-known DNN architectures (e.g., ResNet and AlexNet). Our findings call for future works on empirical validation, automated tools for measurement, and best practices for implementation. CCS CONCEPTS • Software and its engineering → Reusability; • Computing methodologies → Neural networks.

Research paper thumbnail of Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability

arXiv (Cornell University), Mar 4, 2023

Research paper thumbnail of Discrepancies among pre-trained deep neural networks: a new threat to model zoo reliability

Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Training deep neural networks (DNNs) takes significant time and resources. A practice for expedit... more Training deep neural networks (DNNs) takes significant time and resources. A practice for expedited deployment is to use pre-trained deep neural networks (PTNNs), often from model zoosÐcollections of PTNNs; yet, the reliability of model zoos remains unexamined. In the absence of an industry standard for the implementation and performance of PTNNs, engineers cannot confidently incorporate them into production systems. As a first step, discovering potential discrepancies between PTNNs across model zoos would reveal a threat to model zoo reliability. Prior works indicated existing variances in deep learning systems in terms of accuracy. However, broader measures of reliability for PTNNs from model zoos are unexplored. This work measures notable discrepancies between accuracy, latency, and architecture of 36 PTNNs across four model zoos. Among the top 10 discrepancies, we find differences of 1.23%ś2.62% in accuracy and 9%ś131% in latency. We also find mismatches in architecture for well-known DNN architectures (e.g., ResNet and AlexNet). Our findings call for future works on empirical validation, automated tools for measurement, and best practices for implementation. CCS CONCEPTS • Software and its engineering → Reusability; • Computing methodologies → Neural networks.

Research paper thumbnail of Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability

arXiv (Cornell University), Mar 4, 2023

Training deep neural networks (DNNs) takes significant time and resources. A practice for expedit... more Training deep neural networks (DNNs) takes significant time and resources. A practice for expedited deployment is to use pre-trained deep neural networks (PTNNs), often from model zoos-collections of PTNNs; yet, the reliability of model zoos remains unexamined. In the absence of an industry standard for the implementation and performance of PTNNs, engineers cannot confidently incorporate them into production systems. As a first step, discovering potential discrepancies between PTNNs across model zoos would reveal a threat to model zoo reliability. Prior works indicated existing variances in deep learning systems in terms of accuracy. However, broader measures of reliability for PTNNs from model zoos are unexplored. This work measures notable discrepancies between accuracy, latency, and architecture of 36 PTNNs across four model zoos. Among the top 10 discrepancies, we find differences of 1.23%-2.62% in accuracy and 9%-131% in latency. We also find mismatches in architecture for well-known DNN architectures (e.g., ResNet and AlexNet). Our findings call for future works on empirical validation, automated tools for measurement, and best practices for implementation. CCS CONCEPTS • Software and its engineering → Reusability; • Computing methodologies → Neural networks.

Research paper thumbnail of Discrepancies among Pre-trained Deep Neural Networks: A New Threat to Model Zoo Reliability

arXiv (Cornell University), Mar 4, 2023

Research paper thumbnail of Discrepancies among pre-trained deep neural networks: a new threat to model zoo reliability

Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Training deep neural networks (DNNs) takes significant time and resources. A practice for expedit... more Training deep neural networks (DNNs) takes significant time and resources. A practice for expedited deployment is to use pre-trained deep neural networks (PTNNs), often from model zoosÐcollections of PTNNs; yet, the reliability of model zoos remains unexamined. In the absence of an industry standard for the implementation and performance of PTNNs, engineers cannot confidently incorporate them into production systems. As a first step, discovering potential discrepancies between PTNNs across model zoos would reveal a threat to model zoo reliability. Prior works indicated existing variances in deep learning systems in terms of accuracy. However, broader measures of reliability for PTNNs from model zoos are unexplored. This work measures notable discrepancies between accuracy, latency, and architecture of 36 PTNNs across four model zoos. Among the top 10 discrepancies, we find differences of 1.23%ś2.62% in accuracy and 9%ś131% in latency. We also find mismatches in architecture for well-known DNN architectures (e.g., ResNet and AlexNet). Our findings call for future works on empirical validation, automated tools for measurement, and best practices for implementation. CCS CONCEPTS • Software and its engineering → Reusability; • Computing methodologies → Neural networks.