delicatessen.derivative.auto_differentiation

auto_differentiation(xk, f)

Forward-mode automatic differentiation. Automatic differentiation offers a way to compute the exact derivative, rather than numerically approximate it. Forward-mode automatic differentiation iteratively applies the chain rule through recursive calls to the function.

Note

This functionality is only intended for use behind the scenes in delicatessen.

This is accomplished by the PrimalTangentPairs class, which is a special data type in delicatessen that stores pairs of the original evaluation and the corresponding derivative for a variety of different mathematical operations. This is what allows for the exact derivative calculation. The auto_differentiation function is a wrapper to access and use this class object as it is intended for derivative computations.

Parameters

  • xk (ndarray, list, shape (n, )) – Point(s) or coordinate vector to evaluate the gradient at.

  • f (callable) – Function of which to estimate the gradient of.

Returns

Corresponding array of the pairwise derivatives for all different input x values.

Return type

numpy.array

Examples

Loading necessary functions

>>> import numpy as np
>>> from delicatessen.derivative import auto_differentiation

To illustrate use, we will compute the derivative of the following function

\[f(x) = x^2 - x^1 + sin(x + \sqrt{x})\]
>>> def f(x):
>>>     return x**2 - x + np.sin(x + np.sqrt(x))

If you work out the deriative by-hand, you will end up with the following

\[2x - 1 + \left( \frac{1}{2 \sqrt{x}} + 1 \right) \cos(x + \sqrt{x})\]

Instead, we can use automatic differentiation to evaluate the derivative at a specific point. Here, we will evaluate the derivative at \(x=1\)

>>> dy = auto_differentiation(xk=[1, ], f=f)

which returns 0.3757795. This is the same as if you plugged in \(x=1\) into the previous equation.

Note

If a derivative is not defined, then the function will return a NaN.

The derivative of a function with multiple inputs and multiple outputs can also be evaluated. Consider the following example with three inputs and two outputs

>>> def f(x):
>>>     return [x[0]**2 - x[1], np.sin(np.sqrt(x[1]) + x[2]) + x[2]*(x[1]**2)]

>>> dy = auto_differentiation(xk=[0.7, 1.2, -0.9], f=f)

which will return a 2-by-3 array of all the x-y pair derivatives at the given values. Here, the rows correspond to the output and the columns correspond to the inputs.

References

Baydin AG, Pearlmutter BA, Radul AA, & Siskind JM. (2018). Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18, 1-43.

Rall LB & Corliss GF. (1996). An introduction to automatic differentiation. Computational Differentiation: Techniques, Applications, and Tools, 89, 1-18.