本文地址：https://www.cnblogs.com/faranten/p/16028217.html
转载请注明作者与出处
1 分母布局与分子布局
矩阵微分可以认为是多元微分的一种特殊形式，其中最基础的概念是分母布局（denominator layout）和分子布局（nominator layout）的概念，它决定了矩阵微分的结构。对于\(\mathbf x\in\mathbb R^{M}\)与\(y=f(\mathbf x)\in\mathbb R\)而言：
\[\begin{aligned} \text{分母布局}&\quad\frac{\partial y}{\partial\mathbf x}=[\frac{\partial y}{\partial x_1},\frac{\partial y}{\partial x_2},\cdots,\frac{\partial y}{\partial x_M}]^T\quad\text{列向量}\\ \text{分子布局}&\quad\frac{\partial y}{\partial\mathbf x}=[\frac{\partial y}{\partial x_1},\frac{\partial y}{\partial x_2},\cdots,\frac{\partial y}{\partial x_M}]\quad\text{行向量} \end{aligned} \]
而对于\(x\in\mathbb R\)与\(\mathbf y=f(x)\in\mathbb R^{N}\)而言：
\[\begin{aligned} \text{分母布局}&\quad\frac{\partial\mathbf y}{\partial x}=[\frac{\partial y_1}{\partial x},\frac{\partial y_2}{\partial x},\cdots,\frac{\partial y_N}{\partial x}]\quad\text{行向量}\\ \text{分子布局}&\quad\frac{\partial\mathbf y}{\partial x}=[\frac{\partial y_1}{\partial x},\frac{\partial y_2}{\partial x},\cdots,\frac{\partial y_N}{\partial x}]^T\quad\text{列向量} \end{aligned} \]
对于\(\mathbf x\in\mathbb R^M\)与\(\mathbf y=f(\mathbf x)\in\mathbb R^N\)而言，其分母布局的一阶导数：
\[\frac{\partial f(\mathbf x)}{\partial\mathbf x}= \left[ \begin{array} {ccc} \frac{\partial y_1}{\partial x_1}&\cdots&\frac{\partial y_N}{\partial x_1}\\ \vdots&\ddots&\vdots\\ \frac{\partial y_1}{\partial x_M}&\cdots&\frac{\partial y_N}{\partial x_M} \end{array} \right]\in\mathbb R^{M\times N} \]
称为雅可比矩阵（Jacobian Matrix）的转置（因为雅可比矩阵采用分子布局）。对于\(\mathbf x\in\mathbb R^{M}\)与\(y=f(\mathbf x)\in\mathbb R\)而言，其分母布局的二阶导数：
\[\mathbf H=\frac{\partial^2f(\mathbf x)}{\partial\mathbf x^2}= \left[ \begin{array} {ccc} \frac{\partial^2 y}{\partial x_1^2}&\cdots&\frac{\partial y}{\partial x_1x_M}\\ \vdots&\ddots&\vdots\\ \frac{\partial^2 y}{\partial x_Mx_1}&\cdots&\frac{\partial^2 y}{\partial x_M^2} \end{array} \right]\in\mathbb R^{M\times M} \]
称为函数\(f(\mathbf x)\)的Hessian矩阵，也写作\(\nabla^2f(\mathbf x)\)，其中第\(m,n\)个元素为\(\frac{\partial^2y}{\partial x_mx_n}\)。
2 导数法则
2.1 加减法则
对于\(\mathbf x\in\mathbb R^{M}\)，\(\mathbf y=f(\mathbf x)\in\mathbb R^N\)，\(\mathbf z=g(\mathbf x)\in\mathbb R^N\)，则
\[\frac{\partial(\mathbf y+\mathbf z)}{\partial\mathbf x}=\frac{\partial\mathbf y}{\partial\mathbf x}+\frac{\partial\mathbf z}{\partial\mathbf x} \]
2.2 乘法法则
对于\(\mathbf x\in\mathbb R^M\)，\(\mathbf y=f(\mathbf x)\in\mathbb R^N\)，\(\mathbf z=g(\mathbf x)\in\mathbb R^N\)，则
\[\frac{\partial\mathbf y^T\mathbf z}{\partial\mathbf x}=\frac{\partial\mathbf y}{\partial\mathbf x}\mathbf z+\frac{\partial\mathbf z}{\partial\mathbf x}\mathbf y\quad\in\mathbb R^M \]
对于\(\mathbf x\in\mathbb R^M\)，\(\mathbf y=f(\mathbf x)\in\mathbb R^S\)，\(\mathbf z=g(\mathbf x)\in\mathbb R^T\)且\(\mathbf A\in\mathbb R^{S\times T}\)，则
\[\frac{\partial\mathbf y^T\mathbf A\mathbf z}{\partial\mathbf x}=\frac{\partial\mathbf y}{\partial\mathbf x}\mathbf A\mathbf z+\frac{\partial\mathbf z}{\partial\mathbf x}\mathbf A^T\mathbf y\quad\in\mathbb R^M \]
对于\(\mathbf x\in\mathbb R^M\)，\(y=f(\mathbf x)\in\mathbb R\)，\(\mathbf z=g(\mathbf x)\in\mathbb R^N\)，则
\[\frac{\partial y\mathbf z}{\partial\mathbf x}=y\frac{\partial\mathbf z}{\partial\mathbf x}+\frac{\partial y}{\partial\mathbf x}\mathbf z^T\quad\in\mathbb R^{M\times N} \]
2.3 链式法则
在形式上和普通的链式法则一样。
3 完整定义
可以看出分母布局和分子布局的区别仅在于转置。
3.1 图1：\(\partial\text{向量}/\partial\text{向量}\)
3.2 图2：\(\partial\text{标量}/\partial\text{向量}\)
3.3 图3：\(\partial\text{向量}/\partial\text{标量}\)
3.4 图4：\(\partial\text{标量}/\partial\text{矩阵}\)
3.5 图5：\(\partial\text{矩阵}/\partial\text{标量}\)
3.6 图6：\(\partial\text{标量}/\partial\text{标量}\)链式法则结合矩阵
3.7 图7：\(\partial\text{标量}/\partial\text{标量}\)链式法则结合矩阵
3.8 图8：\(d(\text{矩阵})\)
3.9 图9：\(d(\text{矩阵})\)
3.10 图10：\(d/d\)形式
4 参考资料
