+<html lang="en"><head><meta charset="UTF-8"/><meta name="viewport" content="width=device-width, initial-scale=1.0"/><title>Mathematical model · DifferentiationInterface.jl</title><meta name="title" content="Mathematical model · DifferentiationInterface.jl"/><meta property="og:title" content="Mathematical model · DifferentiationInterface.jl"/><meta property="twitter:title" content="Mathematical model · DifferentiationInterface.jl"/><meta name="description" content="Documentation for DifferentiationInterface.jl."/><meta property="og:description" content="Documentation for DifferentiationInterface.jl."/><meta property="twitter:description" content="Documentation for DifferentiationInterface.jl."/><script data-outdated-warner src="../../assets/warner.js"></script><link href="https://cdnjs.cloudflare.com/ajax/libs/lato-font/3.0.0/css/lato-font.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/juliamono/0.050/juliamono.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/fontawesome.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/solid.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.4.2/css/brands.min.css" rel="stylesheet" type="text/css"/><link href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.16.8/katex.min.css" rel="stylesheet" type="text/css"/><script>documenterBaseURL="../.."</script><script src="https://cdnjs.cloudflare.com/ajax/libs/require.js/2.3.6/require.min.js" data-main="../../assets/documenter.js"></script><script src="../../search_index.js"></script><script src="../../siteinfo.js"></script><script src="../../../versions.js"></script><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../../assets/themes/catppuccin-mocha.css" data-theme-name="catppuccin-mocha"/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../../assets/themes/catppuccin-macchiato.css" data-theme-name="catppuccin-macchiato"/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../../assets/themes/catppuccin-frappe.css" data-theme-name="catppuccin-frappe"/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../../assets/themes/catppuccin-latte.css" data-theme-name="catppuccin-latte"/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../../assets/themes/documenter-dark.css" data-theme-name="documenter-dark" data-theme-primary-dark/><link class="docs-theme-link" rel="stylesheet" type="text/css" href="../../assets/themes/documenter-light.css" data-theme-name="documenter-light" data-theme-primary/><script src="../../assets/themeswap.js"></script><link href="../../assets/favicon.ico" rel="icon" type="image/x-icon"/></head><body><div id="documenter"><nav class="docs-sidebar"><a class="docs-logo" href="../../"><img src="../../assets/logo.svg" alt="DifferentiationInterface.jl logo"/></a><div class="docs-package-name"><span class="docs-autofit"><a href="../../">DifferentiationInterface.jl</a></span></div><button class="docs-search-query input is-rounded is-small is-clickable my-2 mx-auto py-1 px-2" id="documenter-search-query">Search docs (Ctrl + /)</button><ul class="docs-menu"><li><a class="tocitem" href="../../">Home</a></li><li><span class="tocitem">Tutorials</span><ul><li><a class="tocitem" href="../../tutorials/basic/">Basic tutorial</a></li><li><a class="tocitem" href="../../tutorials/advanced/">Advanced tutorial</a></li></ul></li><li><span class="tocitem">Explanation</span><ul><li><a class="tocitem" href="../../explanation/operators/">Operators</a></li><li><a class="tocitem" href="../../explanation/backends/">Backends</a></li><li><a class="tocitem" href="../../explanation/advanced/">Advanced features</a></li></ul></li><li><span class="tocitem">FAQ</span><ul><li><a class="tocitem" href="../../faq/limitations/">Limitations</a></li><li><a class="tocitem" href="../../faq/differentiability/">Differentiability</a></li></ul></li><li><a class="tocitem" href="../../api/">API</a></li><li><span class="tocitem">Development</span><ul><li><a class="tocitem" href="../internals/">Internals</a></li><li class="is-active"><a class="tocitem" href>Mathematical model</a><ul class="internal"><li><a class="tocitem" href="#Setting-and-hypotheses"><span>Setting and hypotheses</span></a></li><li><a class="tocitem" href="#Forward-mode"><span>Forward mode</span></a></li><li><a class="tocitem" href="#Reverse-mode"><span>Reverse mode</span></a></li><li><a class="tocitem" href="#Implementation"><span>Implementation</span></a></li></ul></li><li><a class="tocitem" href="../contributing/">Contributing</a></li></ul></li></ul><div class="docs-version-selector field has-addons"><div class="control"><span class="docs-label button is-static is-size-7">Version</span></div><div class="docs-selector control is-expanded"><div class="select is-fullwidth is-size-7"><select id="documenter-version-selector"></select></div></div></div></nav><div class="docs-main"><header class="docs-navbar"><a class="docs-sidebar-button docs-navbar-link fa-solid fa-bars is-hidden-desktop" id="documenter-sidebar-button" href="#"></a><nav class="breadcrumb"><ul class="is-hidden-mobile"><li><a class="is-disabled">Development</a></li><li class="is-active"><a href>Mathematical model</a></li></ul><ul class="is-hidden-tablet"><li class="is-active"><a href>Mathematical model</a></li></ul></nav><div class="docs-right"><a class="docs-navbar-link" href="https://github.com/JuliaDiff/DifferentiationInterface.jl" title="View the repository on GitHub"><span class="docs-icon fa-brands"></span><span class="docs-label is-hidden-touch">GitHub</span></a><a class="docs-navbar-link" href="https://github.com/JuliaDiff/DifferentiationInterface.jl/blob/main/DifferentiationInterface/docs/src/dev/math.md" title="Edit source on GitHub"><span class="docs-icon fa-solid"></span></a><a class="docs-settings-button docs-navbar-link fa-solid fa-gear" id="documenter-settings-button" href="#" title="Settings"></a><a class="docs-article-toggle-button fa-solid fa-chevron-up" id="documenter-article-toggle-button" href="javascript:;" title="Collapse all docstrings"></a></div></header><article class="content" id="documenter-page"><h1 id="Mathematical-model"><a class="docs-heading-anchor" href="#Mathematical-model">Mathematical model</a><a id="Mathematical-model-1"></a><a class="docs-heading-anchor-permalink" href="#Mathematical-model" title="Permalink"></a></h1><p>This page recaps the mathematical model of automatic differentiation used by DI, which justifies how preparation results are constructed. It is inspired by</p><ul><li>the <a href="https://chalk-lab.github.io/Mooncake.jl/stable/understanding_mooncake/rule_system/">documentation</a> of <a href="https://github.com/chalk-lab/Mooncake.jl">Mooncake.jl</a></li><li><a href="https://discourse.julialang.org/t/do-i-understand-enzyme-properly/97760">this Discourse answer</a> about <a href="https://github.com/EnzymeAD/Enzyme.jl">Enzyme.jl</a></li></ul><h2 id="Setting-and-hypotheses"><a class="docs-heading-anchor" href="#Setting-and-hypotheses">Setting and hypotheses</a><a id="Setting-and-hypotheses-1"></a><a class="docs-heading-anchor-permalink" href="#Setting-and-hypotheses" title="Permalink"></a></h2><p>Consider a mathematical function <span>$f(x, c, s) = y$</span> where</p><ul><li><p class="math-container">\[x \in \mathcal{X}\]</p>is the active argument (the one being differentiated)</li><li><p class="math-container">\[c \in \mathcal{C}\]</p>is a constant argument (corresponds to <a href="../../api/#DifferentiationInterface.Constant"><code>Constant</code></a> contexts)</li><li><p class="math-container">\[s \in \mathcal{S}\]</p>is a scratch argument (corresponds to <a href="../../api/#DifferentiationInterface.Cache"><code>Cache</code></a> contexts)</li><li><p class="math-container">\[y \in \mathcal{Y}\]</p>is the output</li></ul><p>In Julia code, some of the input arguments might be mutated, while the output may be written to as well. Therefore, the proper model is a function <span>$\phi(x_0, c_0, s_0, y_0) = (x_1, c_1, s_1, y_1)$</span> where <span>$a_0$</span> is the state of argument <span>$a$</span> before <span>$f$</span> is run, while <span>$a_1$</span> is its state after <span>$a$</span> is run.</p><p>DI makes the following hypotheses on the implementation of <span>$f$</span> (aka the behavior of <span>$\phi$</span>):</p><ol><li>The active argument <span>$x$</span> is not mutated, so <span>$x_1 = x_0$</span></li><li>The constant argument <span>$c$</span> is not mutated, so <span>$c_1 = c_0$</span></li><li>The initial value of the scratch argument <span>$s_0$</span> does not matter</li><li>The initial value of the output <span>$y_0$</span> does not matter</li></ol><h2 id="Forward-mode"><a class="docs-heading-anchor" href="#Forward-mode">Forward mode</a><a id="Forward-mode-1"></a><a class="docs-heading-anchor-permalink" href="#Forward-mode" title="Permalink"></a></h2><p>We want to compute a Jacobian-Vector Product (JVP) <span>$\dot{y} = \left(\frac{\partial f}{\partial x}\right) \dot{x}$</span> where <span>$\dot{x} \in \mathcal{X}$</span> is an input tangent.</p><p>To do that, we run our AD backend on <span>$\phi$</span> with input tangents <span>$(\dot{x}_0, \dot{c}_0, \dot{s}_0, \dot{y}_0)$</span> and obtain <span>$(\dot{x}_1, \dot{c}_1, \dot{s}_1, \dot{y}_1)$</span>. The interesting value is <span>$\dot{y}_1 = \frac{\partial y_1}{\partial x_0} \dot{x}_0 + \frac{\partial y_1}{\partial c_0} \dot{c}_0 + \frac{\partial y_1}{\partial s_0} \dot{s}_0 + \frac{\partial y_1}{\partial y_0} \dot{y}_0$</span></p><p>Thanks to our hypotheses 3 and 4 on the function's implementation, <span>$\frac{\partial y_1}{\partial s_0} = 0$</span> and <span>$\frac{\partial y_1}{\partial y_0} = 0$</span>, so we are left with: <span>$\dot{y}_1 = \frac{\partial y_1}{\partial x_0} \dot{x_0} + \frac{\partial y_1}{\partial c_0} \dot{c_0}$</span></p><p>Thus, as long as <span>$\dot{c}_0 = 0$</span>, the output tangent <span>$\dot{y}_1$</span> contains the correct JVP. Let us now look at <span>$\dot{s}_1$</span> with the help of hypothesis 2: <span>$\dot{c}_1 = \frac{\partial c_1}{\partial x_0} \dot{x}_0 + \frac{\partial c_1}{\partial c_0} \dot{c}_0 + \frac{\partial c_1}{\partial s_0} \dot{s}_0 + \frac{\partial c_1}{\partial y_0} \dot{y}_0 = \dot{c}_0$</span></p><p>The tangent of <span>$c$</span> will always be preserved by differentiation.</p><h2 id="Reverse-mode"><a class="docs-heading-anchor" href="#Reverse-mode">Reverse mode</a><a id="Reverse-mode-1"></a><a class="docs-heading-anchor-permalink" href="#Reverse-mode" title="Permalink"></a></h2><p>We want to compute a Vector-Jacobian Product (VJP) <span>$\bar{x} = \left(\frac{\partial f}{\partial x}\right)^* \bar{y}$</span> where <span>$\bar{y} \in \mathcal{Y}$</span> is an output sensivity.</p><p>To do that, we run our AD backend on <span>$\phi$</span> with output sensitivities <span>$(\bar{x}_1, \bar{c}_1, \bar{s}_1, \bar{y}_1)$</span> and obtain <span>$(\bar{x}_0, \bar{c}_0, \bar{s}_0, \bar{y}_0)$</span>. The interesting value is <span>$\bar{x}_0 = \left(\frac{\partial x_1}{\partial x_0}\right)^* \bar{x}_1 + \left(\frac{\partial c_1}{\partial x_0}\right)^* \bar{c}_1 + \left(\frac{\partial s_1}{\partial x_0}\right)^* \bar{s}_1 + \left(\frac{\partial y_1}{\partial x_0}\right)^* \bar{y}_1$</span></p><p>Thanks to our hypotheses 1 and 2 on the function's implementation, <span>$\frac{\partial x_1}{\partial x_0} = I$</span> and <span>$\frac{\partial c_1}{\partial x_0} = 0$</span>, so we are left with: <span>$\bar{x}_0 = \bar{x}_1 + \left(\frac{\partial s_1}{\partial x_0}\right)^* \bar{s}_1 + \left(\frac{\partial y_1}{\partial x_0}\right)^* \bar{y}_1$</span></p><p>Thus, as long as <span>$\bar{x}_1 = 0$</span> and <span>$\bar{s}_1 = 0$</span>, the input sensitivity <span>$\bar{x}_0$</span> contains the correct VJP. Let us now look at <span>$\bar{s}_0$</span> with the help of hypothesis 3:</p><p class="math-container">\[\bar{s}_0 = \left(\frac{\partial x_1}{\partial s_0}\right)^* \bar{x}_1 + \left(\frac{\partial c_1}{\partial s_0}\right)^* \bar{c}_1 + \left(\frac{\partial s_1}{\partial s_0}\right)^* \bar{s}_1 + \left(\frac{\partial y_1}{\partial s_0}\right)^* \bar{y}_1 = 0\]</p><p>The sensitivity of <span>$s$</span> will always be set to <span>$0$</span> by differentiation.</p><h2 id="Implementation"><a class="docs-heading-anchor" href="#Implementation">Implementation</a><a id="Implementation-1"></a><a class="docs-heading-anchor-permalink" href="#Implementation" title="Permalink"></a></h2><p>DI's preparation mechanism allows pre-allocating the memory for tangents and sensitivities, inside a <code>prep</code> object. This object is then reused across several AD calls.</p><p>For mutable objects, each AD call performs the following transformations on the provided shadow/dual storage (<code>Duplicated</code> for Enzyme, <code>Dual</code> / <code>CoDual</code> for Mooncake):</p><ul><li>In forward mode, <span>$\dot{a}$</span> is updated from <span>$\dot{a}_0$</span> to <span>$\dot{a}_1$</span></li><li>In reverse mode, <span>$\bar{a}$</span> is updated from <span>$\bar{a}_1$</span> to <span>$\bar{a}_0$</span></li></ul><h3 id="At-initialization"><a class="docs-heading-anchor" href="#At-initialization">At initialization</a><a id="At-initialization-1"></a><a class="docs-heading-anchor-permalink" href="#At-initialization" title="Permalink"></a></h3><p>How to initialize shadow/dual memory inside <code>prep</code>?</p><ul><li>In forward mode, make sure that <span>$\dot{c} = 0$</span>.</li><li>In reverse mode, make sure that <span>$\bar{x} = 0$</span> and <span>$\bar{s} = 0$</span>.</li></ul><h3 id="At-every-call"><a class="docs-heading-anchor" href="#At-every-call">At every call</a><a id="At-every-call-1"></a><a class="docs-heading-anchor-permalink" href="#At-every-call" title="Permalink"></a></h3><p>Should the shadow/dual memory inside <code>prep</code> be reset before every AD call?</p><ul><li>In forward mode, no need (<span>$\dot{c}$</span> will remain <span>$0$</span> if it is initialized to <span>$0$</span>)</li><li>In reverse mode, just set <span>$\bar{x} = 0$</span> (<span>$\bar{s}$</span> will be reset to <span>$0$</span> at every AD call)</li></ul></article><nav class="docs-footer"><a class="docs-footer-prevpage" href="../internals/">« Internals</a><a class="docs-footer-nextpage" href="../contributing/">Contributing »</a><div class="flexbox-break"></div><p class="footer-message">Powered by <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> and the <a href="https://julialang.org/">Julia Programming Language</a>.</p></nav></div><div class="modal" id="documenter-settings"><div class="modal-background"></div><div class="modal-card"><header class="modal-card-head"><p class="modal-card-title">Settings</p><button class="delete"></button></header><section class="modal-card-body"><p><label class="label">Theme</label><div class="select"><select id="documenter-themepicker"><option value="auto">Automatic (OS)</option><option value="documenter-light">documenter-light</option><option value="documenter-dark">documenter-dark</option><option value="catppuccin-latte">catppuccin-latte</option><option value="catppuccin-frappe">catppuccin-frappe</option><option value="catppuccin-macchiato">catppuccin-macchiato</option><option value="catppuccin-mocha">catppuccin-mocha</option></select></div></p><hr/><p>This document was generated with <a href="https://github.com/JuliaDocs/Documenter.jl">Documenter.jl</a> version 1.15.0 on <span class="colophon-date" title="Thursday 13 November 2025 18:19">Thursday 13 November 2025</span>. Using Julia version 1.12.1.</p></section><footer class="modal-card-foot"></footer></div></div></div></body></html>
0 commit comments