[深度对齐] paddle.nn.functional.gelu

[cszdrg]

PR Category

Operator Mechanism

PR Types

New features

Description

pytorch公式计算位置：pytorch/aten/src/ATen/native/cuda/ActivationGeluKernel.cu

官网公式描述：

paddle修改前后公式分别为：

• 前向近似计算

$\displaystyle kAlpha = 2/\sqrt{\pi} \times 1/\sqrt{2}$
$\displaystyle kBeta = kAlpha \times GELUCONSTANT \times 3$
$\displaystyle cubeX = x^3$
$\displaystyle tanhOut = \tanh(kAlpha \times (GELUCONSTANT \times cubeX + x))$
$$ansA = \tfrac{1}{2}\Big[(1 + tanhOut) + (1 - tanhOut^2)\times (x\times kAlpha + cubeX\times kBeta)\Big]$$

---转化为--->

$\displaystyle kAlpha = \sqrt{2} \times 2/\sqrt{\pi} \times 0.5$
$\displaystyle kBeta = GELUCONSTANT$
$\displaystyle xSeq = x^2$
$\displaystyle cubeX = x^3$
$\displaystyle tanhOut = \tanh(kAlpha \times (kBeta \times cubeX + x))$
$$ansB = \tfrac{1}{2}(1 + tanhOut) + \tfrac{1}{2}x(1 - tanhOut^2)\times (kAlpha \times (1 + 3kBeta xSeq))$$

• 前向不做近似计算

$kBeta = \left(\frac{2}{\sqrt{\pi}}\right)\left(\frac{1}{\sqrt{2}}\right)\times 0.5$
$cdf = \mathrm{normcdf}(x)$
$pdf = \exp\big(-0.5\times x\times x\big)\times kBeta$

---转化为--->

$kBeta = \left(\frac{2}{\sqrt{\pi}}\right)\left(\frac{1}{\sqrt{2}}\right)\times 0.5$
$kAlpha = \frac{1}{\sqrt{2}}$
cdf = 0.5 * (1 + erf(x * kAlpha))
$pdf = \exp\big(-0.5\times x\times x\big)\times kBeta$

paddle 修改后公式为：

• 反向近似计算

$kAlpha = sqrt(1/2) * (2 / sqrt(pi))$
$tanhout = tanh(kAlpha * x * (1 + GELUCONSTANT * x * x))$
$out = x * 0.5 * (1 + tanhout)$

---转化为--->

$kAlpha = sqrt(2) * (2 / sqrt(pi)) * 0.5$
$tanhout = tanh(kAlpha * (x + GELUCONSTANT * (x * x * x)))$
$out = 0.5 * x * (1 + tanhout)$

• 反向不做近似计算

$out = x * normcdf(x)$

---转化为--->

$kAlpha = 1 / sqrt(2)$
$out = x * 0.5 * (1 + erf(x * kAlpha))$

修改gelu的前向和反向计算公式 使得计算过程与pytorch保持对齐，修改后 所有测试均可通过

[paddle-bot]

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[cszdrg]

/re-run all-failed

[A-nnonymous]

LGTM

[zrr1999]

可以在描述里把具体公式的修改情况列一下

[wanghuancoder]

LGTM