Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

Chunwei Xia; Jiacheng Zhao; Qianqi Sun; Zheng Wang; Yuan Wen; Teng Yu; Xiaobing Feng; Huimin  Cui

doi:10.1145/3617232.3624858

Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

Chunwei Xia, Jiacheng Zhao^* (Corresponding Author), Qianqi Sun, Zheng Wang, Yuan Wen, Teng Yu, Xiaobing Feng, Huimin Cui

^*Corresponding author for this work

Computing Science

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

2 Citations (Scopus)

1 Downloads (Pure)

Abstract

Optimizing deep neural network (DNN) execution is important but becomes increasingly difficult as DNN complexity grows. Existing DNN compilers cannot effectively exploit optimization opportunities across operator boundaries, leaving room for improvement. To address this challenge, we present Souffle, an open-source compiler that optimizes DNN inference across operator boundaries. Souffle creates a global tensor dependency graph using tensor expressions, traces data flow and tensor information, and partitions the computation graph into subprograms based on dataflow analysis and resource constraints. Within a subprogram, Souffle performs local optimization via semantic-preserving transformations, finds an optimized program schedule, and improves instruction-level parallelism and data reuse. We evaluated Souffle using six representative DNN models on an NVIDIA A100 GPU. Experimental results show that Souffle consistently outperforms six state-of-the-art DNN optimizers by delivering a geometric mean speedup of up to 3.7× over
TensorRT and 7.8× over Tensorflow XLA.

Original language	English
Title of host publication	The ACM International Conference on Architectural Support for Programming Languages and Operating Systems
Publisher	ACM
Pages	286 - 301
Number of pages	15
DOIs	https://doi.org/10.1145/3617232.3624858
Publication status	Published - 17 Apr 2024

Bibliographical note

We thank our shepherd, Vinod Grover, and the anonymous reviewers for their constructive feedback.
For the purpose of open access, the authors have applied a Creative Commons Attribution (CCBY) license to any Author Accepted Manuscript versionarising from this submission.

Access to Document

10.1145/3617232.3624858Licence: CC BY

Xia_etal_A_Optimizing_Deep_Learning_Vor
https://creativecommons.org/licenses/by/4.0/
Final published version, 1.97 MBLicence: CC BY

https://eprints.whiterose.ac.uk/203681/Licence: CC BY

Data from: Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions
Xia, C. (Creator), Zhao, J. (Creator), Sun, Q. (Creator), Wang, Z. (Creator), Wen, Y. (Creator), Yu, T. (Creator), Feng, X. (Creator) & Cui, H. (Creator), Zenodo, 3 Oct 2023
DOI: 10.5281/zenodo.8404599, https://zenodo.org/records/8404599
Dataset

Cite this

Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions. / Xia, Chunwei; Zhao, Jiacheng (Corresponding Author); Sun, Qianqi et al.
The ACM International Conference on Architectural Support for Programming Languages and Operating Systems. ACM, 2024. p. 286 - 301.

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

@inproceedings{3c9e3cc777174b6e852a7620d1ea6b0e,

title = "Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions",

abstract = "Optimizing deep neural network (DNN) execution is important but becomes increasingly difficult as DNN complexity grows. Existing DNN compilers cannot effectively exploit optimization opportunities across operator boundaries, leaving room for improvement. To address this challenge, we present Souffle, an open-source compiler that optimizes DNN inference across operator boundaries. Souffle creates a global tensor dependency graph using tensor expressions, traces data flow and tensor information, and partitions the computation graph into subprograms based on dataflow analysis and resource constraints. Within a subprogram, Souffle performs local optimization via semantic-preserving transformations, finds an optimized program schedule, and improves instruction-level parallelism and data reuse. We evaluated Souffle using six representative DNN models on an NVIDIA A100 GPU. Experimental results show that Souffle consistently outperforms six state-of-the-art DNN optimizers by delivering a geometric mean speedup of up to 3.7× overTensorRT and 7.8× over Tensorflow XLA.",

author = "Chunwei Xia and Jiacheng Zhao and Qianqi Sun and Zheng Wang and Yuan Wen and Teng Yu and Xiaobing Feng and Huimin Cui",

note = "We thank our shepherd, Vinod Grover, and the anonymous reviewers for their constructive feedback. For the purpose of open access, the authors have applied a Creative Commons Attribution (CCBY) license to any Author Accepted Manuscript versionarising from this submission. ",

year = "2024",

month = apr,

day = "17",

doi = "10.1145/3617232.3624858",

language = "English",

pages = "286 -- 301",

booktitle = "The ACM International Conference on Architectural Support for Programming Languages and Operating Systems",

publisher = "ACM",

}

TY - GEN

T1 - Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

AU - Xia, Chunwei

AU - Zhao, Jiacheng

AU - Sun, Qianqi

AU - Wang, Zheng

AU - Wen, Yuan

AU - Yu, Teng

AU - Feng, Xiaobing

AU - Cui, Huimin

N1 - We thank our shepherd, Vinod Grover, and the anonymous reviewers for their constructive feedback. For the purpose of open access, the authors have applied a Creative Commons Attribution (CCBY) license to any Author Accepted Manuscript versionarising from this submission.

PY - 2024/4/17

Y1 - 2024/4/17

N2 - Optimizing deep neural network (DNN) execution is important but becomes increasingly difficult as DNN complexity grows. Existing DNN compilers cannot effectively exploit optimization opportunities across operator boundaries, leaving room for improvement. To address this challenge, we present Souffle, an open-source compiler that optimizes DNN inference across operator boundaries. Souffle creates a global tensor dependency graph using tensor expressions, traces data flow and tensor information, and partitions the computation graph into subprograms based on dataflow analysis and resource constraints. Within a subprogram, Souffle performs local optimization via semantic-preserving transformations, finds an optimized program schedule, and improves instruction-level parallelism and data reuse. We evaluated Souffle using six representative DNN models on an NVIDIA A100 GPU. Experimental results show that Souffle consistently outperforms six state-of-the-art DNN optimizers by delivering a geometric mean speedup of up to 3.7× overTensorRT and 7.8× over Tensorflow XLA.

AB - Optimizing deep neural network (DNN) execution is important but becomes increasingly difficult as DNN complexity grows. Existing DNN compilers cannot effectively exploit optimization opportunities across operator boundaries, leaving room for improvement. To address this challenge, we present Souffle, an open-source compiler that optimizes DNN inference across operator boundaries. Souffle creates a global tensor dependency graph using tensor expressions, traces data flow and tensor information, and partitions the computation graph into subprograms based on dataflow analysis and resource constraints. Within a subprogram, Souffle performs local optimization via semantic-preserving transformations, finds an optimized program schedule, and improves instruction-level parallelism and data reuse. We evaluated Souffle using six representative DNN models on an NVIDIA A100 GPU. Experimental results show that Souffle consistently outperforms six state-of-the-art DNN optimizers by delivering a geometric mean speedup of up to 3.7× overTensorRT and 7.8× over Tensorflow XLA.

U2 - 10.1145/3617232.3624858

DO - 10.1145/3617232.3624858

M3 - Published conference contribution

SP - 286

EP - 301

BT - The ACM International Conference on Architectural Support for Programming Languages and Operating Systems

PB - ACM

ER -

Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

Abstract

Bibliographical note

Access to Document

Fingerprint

Datasets

Data from: Optimizing Deep Learning Inference via Global Analysis and Tensor Expressions

Cite this