Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

Chunwei Xia; Jiacheng Zhao; Qianqi Sun; Zheng Wang; Yuan Wen; Teng Yu; Xiaobing Feng; Huimin  Cui

doi:10.1145/3617232.3624858

Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

Chunwei Xia
, Jiacheng Zhao^* (Corresponding Author)
, Qianqi Sun
, Zheng Wang
, Yuan Wen
, Teng Yu
, Xiaobing Feng
, Huimin Cui

^*Corresponding author for this work

Computing Science

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

11 Citations (Scopus)

3 Downloads (Pure)

Abstract

Optimizing deep neural network (DNN) execution is important but becomes increasingly difficult as DNN complexity grows. Existing DNN compilers cannot effectively exploit optimization opportunities across operator boundaries, leaving room for improvement. To address this challenge, we present Souffle, an open-source compiler that optimizes DNN inference across operator boundaries. Souffle creates a global tensor dependency graph using tensor expressions, traces data flow and tensor information, and partitions the computation graph into subprograms based on dataflow analysis and resource constraints. Within a subprogram, Souffle performs local optimization via semantic-preserving transformations, finds an optimized program schedule, and improves instruction-level parallelism and data reuse. We evaluated Souffle using six representative DNN models on an NVIDIA A100 GPU. Experimental results show that Souffle consistently outperforms six state-of-the-art DNN optimizers by delivering a geometric mean speedup of up to 3.7× over
TensorRT and 7.8× over Tensorflow XLA.

Original language	English
Title of host publication	ASPLOS '24: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems
Publisher	ACM
Pages	286 - 301
Number of pages	15
Volume	1
ISBN (Print)	9798400703720
DOIs	https://doi.org/10.1145/3617232.3624858
Publication status	Published - 17 Apr 2024

Bibliographical note

We thank our shepherd, Vinod Grover, and the anonymous reviewers for their constructive feedback.
For the purpose of open access, the authors have applied a Creative Commons Attribution (CCBY) license to any Author Accepted Manuscript versionarising from this submission.

Funding

This work was supported in part by the National Key R&D Program of China under grant agreement 2021ZD0110101, the National Natural Science Foundation of China (NSFC) under grant agreements T2222026, 22003073, 62232015, and 62090024, the Innovation Funding of ICT CAS under grant agreement E361010, a Beijing Nova Program, and the UK Engineering and Physical Sciences Research Council (EPSRC) under grant agreement EP/X018202/1.

Access to Document

10.1145/3617232.3624858Licence: CC BY

Xia_etal_A_Optimizing_Deep_Learning_Vor
https://creativecommons.org/licenses/by/4.0/
Final published version, 1.97 MBLicence: CC BY

https://eprints.whiterose.ac.uk/203681/Licence: CC BY

Data from: Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions
Xia, C. (Creator), Zhao, J. (Creator), Sun, Q. (Creator), Wang, Z. (Creator), Wen, Y. (Creator), Yu, T. (Creator), Feng, X. (Creator) & Cui, H. (Creator), Zenodo, 3 Oct 2023
DOI: 10.5281/zenodo.8404599, https://zenodo.org/records/8404599
Dataset

Cite this

Xia, C., Zhao, J., Sun, Q., Wang, Z., Wen, Y., Yu, T., Feng, X., & Cui, H. (2024). Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions. In ASPLOS '24: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (Vol. 1, pp. 286 - 301). ACM. https://doi.org/10.1145/3617232.3624858

Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions. / Xia, Chunwei; Zhao, Jiacheng (Corresponding Author); Sun, Qianqi et al.
ASPLOS '24: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. Vol. 1 ACM, 2024. p. 286 - 301.

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

@inproceedings{3c9e3cc777174b6e852a7620d1ea6b0e,

title = "Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions",

abstract = "Optimizing deep neural network (DNN) execution is important but becomes increasingly difficult as DNN complexity grows. Existing DNN compilers cannot effectively exploit optimization opportunities across operator boundaries, leaving room for improvement. To address this challenge, we present Souffle, an open-source compiler that optimizes DNN inference across operator boundaries. Souffle creates a global tensor dependency graph using tensor expressions, traces data flow and tensor information, and partitions the computation graph into subprograms based on dataflow analysis and resource constraints. Within a subprogram, Souffle performs local optimization via semantic-preserving transformations, finds an optimized program schedule, and improves instruction-level parallelism and data reuse. We evaluated Souffle using six representative DNN models on an NVIDIA A100 GPU. Experimental results show that Souffle consistently outperforms six state-of-the-art DNN optimizers by delivering a geometric mean speedup of up to 3.7× over TensorRT and 7.8× over Tensorflow XLA.",

author = "Chunwei Xia and Jiacheng Zhao and Qianqi Sun and Zheng Wang and Yuan Wen and Teng Yu and Xiaobing Feng and Huimin Cui",

note = "We thank our shepherd, Vinod Grover, and the anonymous reviewers for their constructive feedback. For the purpose of open access, the authors have applied a Creative Commons Attribution (CCBY) license to any Author Accepted Manuscript versionarising from this submission. ",

year = "2024",

month = apr,

day = "17",

doi = "10.1145/3617232.3624858",

language = "English",

isbn = "9798400703720",

volume = "1",

pages = "286 -- 301",

booktitle = "ASPLOS '24: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems",

publisher = "ACM",

}

TY - GEN

T1 - Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

AU - Xia, Chunwei

AU - Zhao, Jiacheng

AU - Sun, Qianqi

AU - Wang, Zheng

AU - Wen, Yuan

AU - Yu, Teng

AU - Feng, Xiaobing

AU - Cui, Huimin

N1 - We thank our shepherd, Vinod Grover, and the anonymous reviewers for their constructive feedback. For the purpose of open access, the authors have applied a Creative Commons Attribution (CCBY) license to any Author Accepted Manuscript versionarising from this submission.

PY - 2024/4/17

Y1 - 2024/4/17

N2 - Optimizing deep neural network (DNN) execution is important but becomes increasingly difficult as DNN complexity grows. Existing DNN compilers cannot effectively exploit optimization opportunities across operator boundaries, leaving room for improvement. To address this challenge, we present Souffle, an open-source compiler that optimizes DNN inference across operator boundaries. Souffle creates a global tensor dependency graph using tensor expressions, traces data flow and tensor information, and partitions the computation graph into subprograms based on dataflow analysis and resource constraints. Within a subprogram, Souffle performs local optimization via semantic-preserving transformations, finds an optimized program schedule, and improves instruction-level parallelism and data reuse. We evaluated Souffle using six representative DNN models on an NVIDIA A100 GPU. Experimental results show that Souffle consistently outperforms six state-of-the-art DNN optimizers by delivering a geometric mean speedup of up to 3.7× over TensorRT and 7.8× over Tensorflow XLA.

AB - Optimizing deep neural network (DNN) execution is important but becomes increasingly difficult as DNN complexity grows. Existing DNN compilers cannot effectively exploit optimization opportunities across operator boundaries, leaving room for improvement. To address this challenge, we present Souffle, an open-source compiler that optimizes DNN inference across operator boundaries. Souffle creates a global tensor dependency graph using tensor expressions, traces data flow and tensor information, and partitions the computation graph into subprograms based on dataflow analysis and resource constraints. Within a subprogram, Souffle performs local optimization via semantic-preserving transformations, finds an optimized program schedule, and improves instruction-level parallelism and data reuse. We evaluated Souffle using six representative DNN models on an NVIDIA A100 GPU. Experimental results show that Souffle consistently outperforms six state-of-the-art DNN optimizers by delivering a geometric mean speedup of up to 3.7× over TensorRT and 7.8× over Tensorflow XLA.

U2 - 10.1145/3617232.3624858

DO - 10.1145/3617232.3624858

M3 - Published conference contribution

SN - 9798400703720

VL - 1

SP - 286

EP - 301

BT - ASPLOS '24: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems

PB - ACM

ER -

Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

Abstract

Bibliographical note

Funding

Access to Document

Fingerprint

Datasets

Data from: Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

Cite this