ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct Indexing

Shuhong  Huang; Shizhi  Tang; Yuan Wen; Huanqi  Cao; Ruibai  Tang; Yidong  Chen; Jiping  Yu; Yang Li; Chao  Jiang; Limin  Xiao; Jidong  Zhai

doi:10.1145/3774934

ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct Indexing

Shuhong Huang^* (Corresponding Author)
, Shizhi Tang^* (Corresponding Author)
, Yuan Wen
, Huanqi Cao
, Ruibai Tang
, Yidong Chen
, Jiping Yu
, Yang Li
, Chao Jiang
, Limin Xiao
, Jidong Zhai^* (Corresponding Author)

^*Corresponding author for this work

Tsinghua University
Qingcheng.AI
Wuhan University
Lenovo

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Abstract

Automatic Differentiation (AD) is a technique that computes the derivatives of numerical programs by systematically applying the chain rule, playing a critical role in domains such as machine learning, simulation, and control systems. However, parallelizing differentiated programs remains a significant challenge due to the conflict between tapes (a data structure for intermediate variable storage) and summations: the differentiation process inherently introduces inter-thread summation patterns, which require prohibitively expensive atomic operations; and traditional tape designs tightly couple data retrieval with the program’s control flow, preventing code restructuring needed to eliminate these costly dependencies.
To address these challenges, we present ParDiff, a novel AD system with a direct-indexed tape design, which enables summation-aware loop transformations and various parallel schemes for differentiated programs. This results in a higher degree of parallelization, less synchronization, and reduced inter-thread data movement. We conduct comprehensive experiments on both multi-core CPUs and GPUs. Results show that ParDiff delivers up to 483.21× (geometric mean: 30.88×) speedup over the state-of-the-art fully-AD system, Enzyme. It also achieves a speedup of 2.05× and 2.06× over PyTorch on CPU and GPU, respectively. The source code is publicly available at https://github.com/roastduck/FreeTensor.

Original language	English
Title of host publication	PPoPP '26: Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming
Editors	Tony Hosking, Madan Musuvathi, Kenjiro Taura
Place of Publication	New York
Publisher	ACM
Pages	452 - 465
Number of pages	14
ISBN (Electronic)	9798400723100
DOIs	https://doi.org/10.1145/3774934
Publication status	Published - 31 Jan 2026
Event	ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming - Sydney, Australia Duration: 30 Jan 2026 → 4 Feb 2026 https://ppopp26.sigplan.org/

Conference

Conference	ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming
Abbreviated title	PPoPP 2026
Country/Territory	Australia
City	Sydney
Period	30/01/26 → 4/02/26
Internet address	https://ppopp26.sigplan.org/

Bibliographical note

The source code is publicly available at https://github.com/roastduck/FreeTensor

Funding

Wewould like to thank the anonymous reviewers for their insightful comments. This work is supported by the National Key R&D Program of China under Grant 2024YFE0210800, NSFC forDistinguished Young Scholar under Grant 62225206, National Natural Science Foundation of China under Grants 62532006, 62495062, U23A6007, 6250070824, Beijing Natural Science Foundation under Grant L242017, and China Postdoctoral Science Fund (NO. 2024M761688).

Funders	Funder number
National Key Research and Development Program of China	2024YFE0210800
National Natural Science Foundation of China	62225206, 62532006, 62495062, U23A6007, 6250070824

Access to Document

10.1145/3774934Licence: CC BY

Huang_etal_PPoPP_ParDiff_Efficiently_Parallelizing_Reverse-Mode_VOR
This work is licensed under a Creative Commons Attribution 4.0 International License. https://creativecommons.org/licenses/by/4.0/
Final published version, 1.18 MBLicence: CC BY

Cite this

Huang, S., Tang, S., Wen, Y., Cao, H., Tang, R., Chen, Y., Yu, J., Li, Y., Jiang, C., Xiao, L., & Zhai, J. (2026). ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct Indexing. In T. Hosking, M. Musuvathi, & K. Taura (Eds.), PPoPP '26: Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming (pp. 452 - 465). ACM. https://doi.org/10.1145/3774934

ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct Indexing. / Huang, Shuhong (Corresponding Author); Tang, Shizhi (Corresponding Author); Wen, Yuan et al.
PPoPP '26: Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming. ed. / Tony Hosking; Madan Musuvathi; Kenjiro Taura. New York: ACM, 2026. p. 452 - 465.

Research output: Chapter in Book/Report/Conference proceeding › Published conference contribution

Huang, S, Tang, S, Wen, Y, Cao, H, Tang, R, Chen, Y, Yu, J, Li, Y, Jiang, C, Xiao, L & Zhai, J 2026, ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct Indexing. in T Hosking, M Musuvathi & K Taura (eds), PPoPP '26: Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming. ACM, New York, pp. 452 - 465, ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Sydney, New South Wales, Australia, 30/01/26. https://doi.org/10.1145/3774934

@inproceedings{fdabb948575d4f9b88433c933d9bcf34,

title = "ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct Indexing",

abstract = "Automatic Differentiation (AD) is a technique that computes the derivatives of numerical programs by systematically applying the chain rule, playing a critical role in domains such as machine learning, simulation, and control systems. However, parallelizing differentiated programs remains a significant challenge due to the conflict between tapes (a data structure for intermediate variable storage) and summations: the differentiation process inherently introduces inter-thread summation patterns, which require prohibitively expensive atomic operations; and traditional tape designs tightly couple data retrieval with the program{\textquoteright}s control flow, preventing code restructuring needed to eliminate these costly dependencies. To address these challenges, we present ParDiff, a novel AD system with a direct-indexed tape design, which enables summation-aware loop transformations and various parallel schemes for differentiated programs. This results in a higher degree of parallelization, less synchronization, and reduced inter-thread data movement. We conduct comprehensive experiments on both multi-core CPUs and GPUs. Results show that ParDiff delivers up to 483.21{\texttimes} (geometric mean: 30.88{\texttimes}) speedup over the state-of-the-art fully-AD system, Enzyme. It also achieves a speedup of 2.05{\texttimes} and 2.06{\texttimes} over PyTorch on CPU and GPU, respectively. The source code is publicly available at https://github.com/roastduck/FreeTensor.",

author = "Shuhong Huang and Shizhi Tang and Yuan Wen and Huanqi Cao and Ruibai Tang and Yidong Chen and Jiping Yu and Yang Li and Chao Jiang and Limin Xiao and Jidong Zhai",

note = "The source code is publicly available at https://github.com/roastduck/FreeTensor; ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP 2026 ; Conference date: 30-01-2026 Through 04-02-2026",

year = "2026",

month = jan,

day = "31",

doi = "10.1145/3774934",

language = "English",

pages = "452 -- 465",

editor = "Tony Hosking and Madan Musuvathi and Kenjiro Taura",

booktitle = "PPoPP '26: Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming",

publisher = "ACM",

url = "https://ppopp26.sigplan.org/",

}

TY - GEN

T1 - ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct Indexing

AU - Huang, Shuhong

AU - Tang, Shizhi

AU - Wen, Yuan

AU - Cao, Huanqi

AU - Tang, Ruibai

AU - Chen, Yidong

AU - Yu, Jiping

AU - Li, Yang

AU - Jiang, Chao

AU - Xiao, Limin

AU - Zhai, Jidong

N1 - The source code is publicly available at https://github.com/roastduck/FreeTensor

PY - 2026/1/31

Y1 - 2026/1/31

N2 - Automatic Differentiation (AD) is a technique that computes the derivatives of numerical programs by systematically applying the chain rule, playing a critical role in domains such as machine learning, simulation, and control systems. However, parallelizing differentiated programs remains a significant challenge due to the conflict between tapes (a data structure for intermediate variable storage) and summations: the differentiation process inherently introduces inter-thread summation patterns, which require prohibitively expensive atomic operations; and traditional tape designs tightly couple data retrieval with the program’s control flow, preventing code restructuring needed to eliminate these costly dependencies. To address these challenges, we present ParDiff, a novel AD system with a direct-indexed tape design, which enables summation-aware loop transformations and various parallel schemes for differentiated programs. This results in a higher degree of parallelization, less synchronization, and reduced inter-thread data movement. We conduct comprehensive experiments on both multi-core CPUs and GPUs. Results show that ParDiff delivers up to 483.21× (geometric mean: 30.88×) speedup over the state-of-the-art fully-AD system, Enzyme. It also achieves a speedup of 2.05× and 2.06× over PyTorch on CPU and GPU, respectively. The source code is publicly available at https://github.com/roastduck/FreeTensor.

AB - Automatic Differentiation (AD) is a technique that computes the derivatives of numerical programs by systematically applying the chain rule, playing a critical role in domains such as machine learning, simulation, and control systems. However, parallelizing differentiated programs remains a significant challenge due to the conflict between tapes (a data structure for intermediate variable storage) and summations: the differentiation process inherently introduces inter-thread summation patterns, which require prohibitively expensive atomic operations; and traditional tape designs tightly couple data retrieval with the program’s control flow, preventing code restructuring needed to eliminate these costly dependencies. To address these challenges, we present ParDiff, a novel AD system with a direct-indexed tape design, which enables summation-aware loop transformations and various parallel schemes for differentiated programs. This results in a higher degree of parallelization, less synchronization, and reduced inter-thread data movement. We conduct comprehensive experiments on both multi-core CPUs and GPUs. Results show that ParDiff delivers up to 483.21× (geometric mean: 30.88×) speedup over the state-of-the-art fully-AD system, Enzyme. It also achieves a speedup of 2.05× and 2.06× over PyTorch on CPU and GPU, respectively. The source code is publicly available at https://github.com/roastduck/FreeTensor.

U2 - 10.1145/3774934

DO - 10.1145/3774934

M3 - Published conference contribution

SP - 452

EP - 465

BT - PPoPP '26: Proceedings of the 31st ACM SIGPLAN Annual Symposium on Principles and Practice of Parallel Programming

A2 - Hosking, Tony

A2 - Musuvathi, Madan

A2 - Taura, Kenjiro

PB - ACM

CY - New York

T2 - ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

Y2 - 30 January 2026 through 4 February 2026

ER -

ParDiff: Efficiently Parallelizing Reverse-Mode Automatic Differentiation with Direct Indexing

Abstract

Conference

Bibliographical note

Funding

Access to Document

Fingerprint

Cite this