(一)线性回归
1.模型
线性回归假设输出与各个输入之间是线性关系,下面是一个简单的线性回归模型:
y ^ = x 1 w 1 + x 2 w 2 + b \hat{y} = x_1w_1+x_2w_2+b
y ^ = x 1 w 1 + x 2 w 2 + b
2.损失函数
索引为i的样本误差:
ℓ ( i ) ( w 1 , w 2 , b ) = 1 2 ( y ^ ( i ) − y ( i ) ) 2 \ell^{(i)}(w_1,w_2,b)=\frac{1}{2}\Big(\hat{y}^{(i)}-y^{(i)}\Big)^2
ℓ ( i ) ( w 1 , w 2 , b ) = 2 1 ( y ^ ( i ) − y ( i ) ) 2
常数1 2 \frac{1}{2} 2 1
ℓ ( w 1 , w 2 , b ) = 1 n ∑ i = 1 n ℓ ( i ) ( w 1 , w 2 , b ) = 1 n ∑ i = 1 n 1 2 ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) 2 \ell(w_1,w_2,b)=\frac1n\sum_{i=1}^n\ell^{(i)}(w_1,w_2,b)=\frac1n\sum_{i=1}^n\frac12\Big(x_1^{(i)}w_1+x_2^{(i)}w_2+b-y^{(i)}\Big)^2
ℓ ( w 1 , w 2 , b ) = n 1 i = 1 ∑ n ℓ ( i ) ( w 1 , w 2 , b ) = n 1 i = 1 ∑ n 2 1 ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) 2
3.优化算法
小批量随机梯度下降进行优化:
w 1 ← w 1 − η ∣ B ∣ ∑ i ∈ B ∂ ℓ ( i ) ( w 1 , w 2 , b ) ∂ w 1 = w 1 − η ∣ B ∣ ∑ i ∈ B x 1 ( i ) ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) w 2 ← w 2 − η ∣ B ∣ ∑ i ∈ B ∂ ℓ ( i ) ( w 1 , w 2 , b ) ∂ w 2 = w 2 − η ∣ B ∣ ∑ i ∈ B x 2 ( i ) ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) b ← b − η ∣ B ∣ ∑ i ∈ B ∂ ℓ ( i ) ( w 1 , w 2 , b ) ∂ b = b − η ∣ B ∣ ∑ i ∈ B ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) \begin{gathered}
w_{1}\leftarrow w_{1}-\frac{\eta}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\frac{\partial\ell^{(i)}(w_{1},w_{2},b)}{\partial w_{1}}=w_{1}-\frac{\eta}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}x_{1}^{(i)}\left(x_{1}^{(i)}w_{1}+x_{2}^{(i)}w_{2}+b-y^{(i)}\right) \\
w_{2}\leftarrow w_{2}-\frac{\eta}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\frac{\partial\ell^{(i)}(w_{1},w_{2},b)}{\partial w_{2}}=w_{2}-\frac{\eta}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}x_{2}^{(i)}\left(x_{1}^{(i)}w_{1}+x_{2}^{(i)}w_{2}+b-y^{(i)}\right)\\
b\leftarrow b-\frac\eta{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\frac{\partial\ell^{(i)}(w_{1},w_{2},b)}{\partial b}=b-\frac\eta{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\left(x_{1}^{(i)}w_{1}+x_{2}^{(i)}w_{2}+b-y^{(i)}\right)
\end{gathered} w 1 ← w 1 − ∣ B ∣ η i ∈ B ∑ ∂ w 1 ∂ ℓ ( i ) ( w 1 , w 2 , b ) = w 1 − ∣ B ∣ η i ∈ B ∑ x 1 ( i ) ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) w 2 ← w 2 − ∣ B ∣ η i ∈ B ∑ ∂ w 2 ∂ ℓ ( i ) ( w 1 , w 2 , b ) = w 2 − ∣ B ∣ η i ∈ B ∑ x 2 ( i ) ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) b ← b − ∣ B ∣ η i ∈ B ∑ ∂ b ∂ ℓ ( i ) ( w 1 , w 2 , b ) = b − ∣ B ∣ η i ∈ B ∑ ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) )
B \mathcal{B} B η \eta η 这篇回答 :
不是所有的函数都可以根据导数求出取得0值的点的, 现实的情况可能是:
4.矢量表示
广义上,当数据样本为n,表达式变为:
y ^ = X w + b \hat{\boldsymbol{y}}=\boldsymbol{X}\boldsymbol{w}+b
y ^ = X w + b
损失函数变为:
ℓ ( θ ) = 1 2 n ( y ^ − y ) ⊤ ( y ^ − y ) \ell(\boldsymbol{\theta})=\frac{1}{2n}(\boldsymbol{\hat{y}}-\boldsymbol{y})^\top(\boldsymbol{\hat{y}}-\boldsymbol{y})
ℓ ( θ ) = 2 n 1 ( y ^ − y ) ⊤ ( y ^ − y )
迭代步骤变成:
θ ← θ − η ∣ B ∣ ∑ i ∈ B ∇ θ ℓ ( i ) ( θ ) \theta\leftarrow\theta-\frac{\eta}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\nabla_{\boldsymbol{\theta}}\ell^{(i)}(\boldsymbol{\theta})
θ ← θ − ∣ B ∣ η i ∈ B ∑ ∇ θ ℓ ( i ) ( θ )
∇ θ ℓ ( i ) ( θ ) = [ ∂ ℓ ( i ) ( w 1 , w 2 , b ) ∂ w 1 ∂ ℓ ( i ) ( w 1 , w 2 , b ) ∂ w 2 ∂ ℓ ( i ) ( w 1 , w 2 , b ) ∂ b ] = [ x 1 ( i ) ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) x 2 ( i ) ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ] = [ x 1 ( i ) x 2 ( i ) 1 ] ( y ^ ( i ) − y ( i ) ) \nabla_{\boldsymbol{\theta}}\ell^{(i)}(\boldsymbol{\theta})=\begin{bmatrix}\frac{\partial\ell^{(i)}(w_1,w_2,b)}{\partial w_1}\\\frac{\partial\ell^{(i)}(w_1,w_2,b)}{\partial w_2}\\\frac{\partial\ell^{(i)}(w_1,w_2,b)}{\partial b}\end{bmatrix}=\begin{bmatrix}x_1^{(i)}(x_1^{(i)}w_1+x_2^{(i)}w_2+b-y^{(i)})\\x_2^{(i)}(x_1^{(i)}w_1+x_2^{(i)}w_2+b-y^{(i)})\\x_1^{(i)}w_1+x_2^{(i)}w_2+b-y^{(i)}\end{bmatrix}=\begin{bmatrix}x_1^{(i)}\\x_2^{(i)}\\1\end{bmatrix}(\hat{y}^{(i)}-y^{(i)})
∇ θ ℓ ( i ) ( θ ) = ⎣ ⎢ ⎢ ⎢ ⎡ ∂ w 1 ∂ ℓ ( i ) ( w 1 , w 2 , b ) ∂ w 2 ∂ ℓ ( i ) ( w 1 , w 2 , b ) ∂ b ∂ ℓ ( i ) ( w 1 , w 2 , b ) ⎦ ⎥ ⎥ ⎥ ⎤ = ⎣ ⎢ ⎢ ⎡ x 1 ( i ) ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) x 2 ( i ) ( x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ) x 1 ( i ) w 1 + x 2 ( i ) w 2 + b − y ( i ) ⎦ ⎥ ⎥ ⎤ = ⎣ ⎢ ⎢ ⎡ x 1 ( i ) x 2 ( i ) 1 ⎦ ⎥ ⎥ ⎤ ( y ^ ( i ) − y ( i ) )
5.为什么选择均方损失
均方损失函数可以用于线性回归的一个原因是:假设了观测中包含噪声,其中噪声服从正态分布:y = w ⊤ x + b + ϵ y=\mathbf{w}^\top\mathbf{x}+b+\epsilon y = w ⊤ x + b + ϵ ϵ ∼ N ( 0 , σ 2 ) \epsilon\sim\mathcal{N}(0,\sigma^2) ϵ ∼ N ( 0 , σ 2 ) n 1 n_1 n 1 n 2 n_2 n 2 n 1 n 1 + n 2 \frac{n_1}{n_1+n_2} n 1 + n 2 n 1 L ( θ ) = θ n 1 ( 1 − θ ) n 2 L(\theta)=\theta^{n_1}(1-\theta)^{n_2} L ( θ ) = θ n 1 ( 1 − θ ) n 2 θ ^ = n 1 n 1 + n 2 \hat{\theta}=\frac{n_1}{n_1+n_2} θ ^ = n 1 + n 2 n 1 梯度下降法 来不断的更新迭代参数，来对参数进行求解。
P ( y ∣ x ) = 1 2 π σ 2 exp ( − 1 2 σ 2 ( y − w ⊤ x − b ) 2 ) P(y\mid\mathbf{x})=\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{1}{2\sigma^2}(y-\mathbf{w}^\top\mathbf{x}-b)^2\right)
P ( y ∣ x ) = 2 π σ 2 1 exp ( − 2 σ 2 1 ( y − w ⊤ x − b ) 2 )
求似然函数的最大(由于历史原因这里取最小):
− log P ( y ∣ X ) = − log ( ∏ i = 1 n p ( y ( i ) ∣ x ( i ) ) ) = ∑ i = 1 n ( 1 2 log ( 2 π σ 2 ) + 1 2 σ 2 ( y ( i ) − w ⊤ x ( i ) − b ) 2 ) -\log P(\mathbf{y}\mid\mathbf{X})=-\log (\prod_{i=1}^np(y^{(i)}|\mathbf{x}^{(i)}))=\sum_{i=1}^n \left(\frac{1}{2}\log(2\pi\sigma^2)+\frac{1}{2\sigma^2}\left(y^{(i)}-\mathbf{w}^\top\mathbf{x}^{(i)}-b\right)^2\right)
− log P ( y ∣ X ) = − log ( i = 1 ∏ n p ( y ( i ) ∣ x ( i ) ) ) = i = 1 ∑ n ( 2 1 log ( 2 π σ 2 ) + 2 σ 2 1 ( y ( i ) − w ⊤ x ( i ) − b ) 2 )
后一项说明在高斯噪声的假设下最小均方误差等价于对线性模型的极大似然估计.
(二)线性回归的从零开始实现
1.生成数据集
1 2 3 4 5 6 7 8 9 10 11 12 13 14 %matplotlib inline import randomimport torchfrom d2l import torch as d2ldef synthetic_data (w, b, num_examples ): """生成y=Xw+b+噪声""" X = torch.normal(0 , 1 , (num_examples, len (w))) y = torch.matmul(X, w) + b y += torch.normal(0 , 0.01 , y.shape) return X, y.reshape((-1 , 1 )) true_w = torch.tensor([2 , -3.4 ]) true_b = 4.2 features, labels = synthetic_data(true_w, true_b, 1000 )
2.读取数据集
以下是一个获取小批量数据的代码:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 def data_iter (batch_size, features, labels ): num_examples = len (features) indices = list (range (num_examples)) random.shuffle(indices) for i in range (0 , num_examples, batch_size): batch_indices = torch.tensor( indices[i: min (i + batch_size, num_examples)]) yield features[batch_indices], labels[batch_indices] batch_size = 10 for X, y in data_iter(batch_size, features, labels): print (X, '\n' , y) break
3.模型与模型参数
从均值为0、标准差为0.01的正态分布中采样随机数来初始化权重， 并将偏置初始化为0.初始化后后续更新这些参数来拟合数据.
1 2 w = torch.normal(0 , 0.01 , size=(2 ,1 ), requires_grad=True ) b = torch.zeros(1 , requires_grad=True )
定义模型:
1 2 3 def linreg (X, w, b ): """线性回归模型""" return torch.matmul(X, w) + b
定义损失函数:
1 2 3 def squared_loss (y_hat, y ): """均方损失""" return (y_hat - y.reshape(y_hat.shape)) ** 2 / 2
定义优化算法(小批量随机梯度下降):
1 2 3 4 5 6 def sgd (params, lr, batch_size ): """小批量随机梯度下降""" with torch.no_grad(): for param in params: param -= lr * param.grad / batch_size param.grad.zero_()
4.训练
在每次迭代中，读取一小批量训练样本，并通过模型来获得一组预测。 计算完损失后，开始反向传播，存储每个参数的梯度。 最后，调用优化算法sgd来更新模型参数。在机器学习中，需要多次遍历整个训练数据集（即多个epoch），在每个迭代周期（epoch）中，使用data_iter函数遍历整个数据集， 并将训练数据集中所有样本都使用一次（假设样本数能够被批量大小整除）,num_epochs和学习率lr都是超参数，分别设为3和0.03(设置超参数需要反复试验调整)。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 lr = 0.03 num_epochs = 3 net = linreg loss = squared_loss for epoch in range (num_epochs): for X, y in data_iter(batch_size, features, labels): l = loss(net(X, w, b), y) l.sum ().backward() sgd([w, b], lr, batch_size) with torch.no_grad(): train_l = loss(net(features, w, b), labels) print (f'epoch {epoch + 1 } , loss {float (train_l.mean()):f} ' )
(三)线性回归的简洁实现
利用框架提供的一些API,能够对线性回归进行简单实现.首先 还是生成数据集.然后 读取数据集使用Pytorch提供的API进行封装:
1 2 3 4 5 6 7 def load_array (data_arrays, batch_size, is_train=True ): """构造一个PyTorch数据迭代器""" dataset = data.TensorDataset(*data_arrays) return data.DataLoader(dataset, batch_size, shuffle=is_train) batch_size = 10 data_iter = load_array((features, labels), batch_size)
这里的is_train表示是否希望数据迭代器对象在每个迭代周期内打乱数据,与上节的data_iter不同,这里我们使用iter构造Python迭代器,并使用next从迭代器中获取第一项.然后 定义模型和模型参数,使用Pytorch提供的Sequential类使用层来构造模型(其实可以不使用,但后续许多模型是多层的也会用到),使用Linear类来输入全连接层:
1 2 3 from torch import nnnet = nn.Sequential(nn.Linear(2 , 1 ))
损失函数:
优化算法:
1 trainer = torch.optim.SGD(net.parameters(), lr=0.03 )
最后 训练:
1 2 3 4 5 6 7 8 9 num_epochs = 3 for epoch in range (num_epochs): for X, y in data_iter: l = loss(net(X) ,y) trainer.zero_grad() l.backward() trainer.step() l = loss(net(features), labels) print (f'epoch {epoch + 1 } , loss {l:f} ' )
(四)softmax回归
回归可以用于预测多少的问题,也可以用于分类问题.softmax 回归 (softmax regression)其实是 logistic 回归的一般形式，logistic 回归用于二分类，而 softmax 回归用于多分类 .
1.softmax简单介绍
对于输入数据{ ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x m , y m ) } \{(x_1,y_1),(x_2,y_2),\ldots,(x_m,y_m)\} { ( x 1 , y 1 ) , ( x 2 , y 2 ) , … , ( x m , y m ) } y i ∈ { 1 , 2 , … , k } y_{i}\in\{1,2,\ldots,k\} y i ∈ { 1 , 2 , … , k } x i x_i x i
h θ ( x i ) = [ p ( y i = 1 ∣ x i ; θ ) p ( y i = 2 ∣ x i ; θ ) ⋮ p ( y i = k ∣ x i ; θ ) ] = 1 ∑ j = 1 k e θ j T x i [ e θ 1 T x i e θ 2 T x i ⋮ e θ h T x i ] \begin{gathered}h_{\theta}\left(x_{i}\right)=\begin{bmatrix}p\left(y_{i}=1|x_{i};\theta\right)\\p\left(y_{i}=2|x_{i};\theta\right)\\\vdots\\p\left(y_{i}=k|x_{i};\theta\right)\end{bmatrix}=\frac{1}{\sum_{j=1}^{k}e^{\theta_{j}^{T}x_{i}}}\begin{bmatrix}e^{\theta_{1}^{T}x_{i}}\\e^{\theta_{2}^{T}x_{i}}\\\vdots\\e^{\theta_{h}^{T}x_{i}}\end{bmatrix}\end{gathered}
h θ ( x i ) = ⎣ ⎢ ⎢ ⎢ ⎢ ⎡ p ( y i = 1 ∣ x i ; θ ) p ( y i = 2 ∣ x i ; θ ) ⋮ p ( y i = k ∣ x i ; θ ) ⎦ ⎥ ⎥ ⎥ ⎥ ⎤ = ∑ j = 1 k e θ j T x i 1 ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎡ e θ 1 T x i e θ 2 T x i ⋮ e θ h T x i ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎤
其中θ 1 , θ 2 , … , θ k ∈ θ \theta_1,\theta_2,\ldots,\theta_k \in \theta θ 1 , θ 2 , … , θ k ∈ θ 1 ∑ j = 1 k e θ j T x i \frac{1}{\sum_{j=1}^{k}e^{\theta_{j}^{T}x_{i}}} ∑ j = 1 k e θ j T x i 1 [ 0 , 1 ] [0,1] [ 0 , 1 ]
L ( θ ) = − 1 m [ ∑ i = 1 m ∑ j = 1 k 1 { y i = j } log e θ j T x i ∑ l = 1 k e θ l T x i ] L(\theta)=-\frac{1}{m}\left[\sum_{i=1}^{m}\sum_{j=1}^{k}1\left\{y_{i}=j\right\}\log\frac{e^{\theta_{j}^{T}x_{i}}}{\sum_{l=1}^{k}e^{\theta_{l}^{T}x_{i}}}\right]
L ( θ ) = − m 1 [ i = 1 ∑ m j = 1 ∑ k 1 { y i = j } log ∑ l = 1 k e θ l T x i e θ j T x i ]
其中1 { ⋅ } 1\{\cdot\} 1 { ⋅ } 1 { 值为真的表达式 } = 1 1\{值为真的表达式\}=1 1 { 值 为 真 的 表 达 式 } = 1 1 { 值为假的表达式 } = 0 1\{值为假的表达式\}=0 1 { 值 为 假 的 表 达 式 } = 0 softmax回归原理与实现 .
2.softmax回归的损失函数（交叉熵损失）
上述代价公式为什么是这样的形式？这里运用了信息论中的一个叫做交叉熵的知识.p = [ 0.279 , 0.119 , 0.359 , 0.014 , 0.008 , 0.002 , 0.011 , 0.145 , 0.036 , 0.025 ] p=[0.279,0.119,0.359,0.014,0.008,0.002,0.011,0.145,0.036,0.025] p = [ 0 . 2 7 9 , 0 . 1 1 9 , 0 . 3 5 9 , 0 . 0 1 4 , 0 . 0 0 8 , 0 . 0 0 2 , 0 . 0 1 1 , 0 . 1 4 5 , 0 . 0 3 6 , 0 . 0 2 5 ] y = [ 0 , 0 , 1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ] y=[0,0,1,0,0,0,0,0,0,0] y = [ 0 , 0 , 1 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ] 同一个随机变量X 有两个单独的概率分布P(x), Q(x)，可以使用KL散度来衡量这两个概率分布之间的差异.
D K L ( p ∣ ∣ q ) = ∑ i = 1 n p ( x i ) log ( p ( x i ) q ( x i ) ) D_{KL}\left(p||q\right)=\sum_{i=1}^np\left(x_i\right)\log\left(\frac{p\left(x_i\right)}{q\left(x_i\right)}\right)
D K L ( p ∣ ∣ q ) = i = 1 ∑ n p ( x i ) log ( q ( x i ) p ( x i ) )
这个公式展开之后:
D K L ( p ∣ ∣ q ) = ∑ i = 1 n p ( x i ) log ( p ( x i ) q ( x i ) ) = ∑ i = 1 n p ( x i ) l o g ( p ( x i ) ) − ∑ i = 1 n p ( x i ) l o g ( q ( x i ) ) = − H ( p ( x ) ) + [ − ∑ i = 1 n p ( x i ) l o g ( q ( x i ) ) ] \begin{gathered}
D_{KL}\left(p||q\right)=\sum_{i=1}^np\left(x_i\right)\log\left(\frac{p\left(x_i\right)}{q\left(x_i\right)}\right) \\
=\sum_{i=1}^np\left(x_i\right)log\left(p\left(x_i\right)\right)-\sum_{i=1}^np\left(x_i\right)log\left(q\left(x_i\right)\right) \\
=-H\left(p\left(x\right)\right)+\left[-\sum_{i=1}^np\left(x_i\right)log\left(q\left(x_i\right)\right)\right]
\end{gathered} D K L ( p ∣ ∣ q ) = i = 1 ∑ n p ( x i ) log ( q ( x i ) p ( x i ) ) = i = 1 ∑ n p ( x i ) l o g ( p ( x i ) ) − i = 1 ∑ n p ( x i ) l o g ( q ( x i ) ) = − H ( p ( x ) ) + [ − i = 1 ∑ n p ( x i ) l o g ( q ( x i ) ) ]
前半部分也有定义,叫做信息熵,对于信息熵的解释是: 信息量的大小与信息发生的概率成反比。概率越大，信息量越小。概率越小，信息量越大 .用I ( x ) = − l o g ( P ( x ) ) I(x)=-log(P(x)) I ( x ) = − l o g ( P ( x ) ) H ( x ) = − ∑ P ( x i ) l o g ( P ( x i ) ) H(x)=- \sum P(x_i)log(P(x_i)) H ( x ) = − ∑ P ( x i ) l o g ( P ( x i ) ) 【损失函数系列】交叉熵做损失函数理论知识_交叉熵函数如何改为损失函数 ,写的很易懂.
3.softmax损失函数导数
softmax的矢量表达式:
O = X W + b Y ^ = softmax ( O ) \begin{aligned}
& \mathbf{O}=\mathbf{X} \mathbf{W}+\mathbf{b} \\
& \hat{\mathbf{Y}}=\operatorname{softmax}(\mathbf{O})
\end{aligned}
O = X W + b Y ^ = s o f t m a x ( O )
对于损失函数对于o j o_j o j
l ( y , y ^ ) = − ∑ j = 1 q y j log exp ( o j ) ∑ k = 1 q exp ( o k ) = ∑ j = 1 q y j log ∑ k = 1 q exp ( o k ) − ∑ j = 1 q y j o j = log ∑ k = 1 q exp ( o k ) − ∑ j = 1 q y j o j . ∴ ∂ o j l ( y , y ^ ) = exp ( o j ) ∑ k = 1 q exp ( o k ) − y j = softmax ( o ) j − y j . \begin{aligned}
l(\mathbf{y}, \hat{\mathbf{y}})
& =-\sum_{j=1}^q y_j \log \frac{\exp \left(o_j\right)}{\sum_{k=1}^q \exp \left(o_k\right)} \\
& =\sum_{j=1}^q y_j \log \sum_{k=1}^q \exp \left(o_k\right)-\sum_{j=1}^q y_j o_j \\
& =\log \sum_{k=1}^q \exp \left(o_k\right)-\sum_{j=1}^q y_j o_j . \\
\therefore \quad \partial_{o_j} l(\mathbf{y}, \hat{\mathbf{y}})
& =\frac{\exp \left(o_j\right)}{\sum_{k=1}^q \exp \left(o_k\right)}-y_j=\operatorname{softmax}(\mathbf{o})_j-y_j .
\end{aligned} l ( y , y ^ ) ∴ ∂ o j l ( y , y ^ ) = − j = 1 ∑ q y j log ∑ k = 1 q exp ( o k ) exp ( o j ) = j = 1 ∑ q y j log k = 1 ∑ q exp ( o k ) − j = 1 ∑ q y j o j = log k = 1 ∑ q exp ( o k ) − j = 1 ∑ q y j o j . = ∑ k = 1 q exp ( o k ) exp ( o j ) − y j = s o f t m a x ( o ) j − y j .
从上式可以看出，softmax回归的损失函数的导数是我们softmax模型分配的概率与实际发生的情况（由独热标签向量表示）之间的差异.
(五)图像分类数据集
1.获取数据
本节使用的是Fashion-MNIST数据集 (Xiao et al. , 2017 ),MNIST(Modified National Institute of Standards and Technology database)数据集 (LeCun et al. , 1998 ) 是图像分类中广泛使用的数据集之一,Fashion-MNIST数据集更复杂.
1 2 3 4 5 6 7 8 9 10 11 from torchvision import tansformstrans = transforms.ToTensor() import torchvisionmnist_train = torchvision.datasets.FashionMNIST( root="../data" , train=True , transform=trans, download=True ) mnist_test = torchvision.datasets.FashionMNIST( root="../data" , train=False , transform=trans, download=True ) mnist_train[0 ][0 ].shape
训练数据集有6000张，测试数据集有1000张。测试数据集不用于训练用于评估模型性能，另外这些图像也不以一般格式存储所以你是不能直接打开的，使用的可能是IDX文件格式.
2.显示数据（图像）
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 def show_images (imgs, num_rows, num_cols, titles=None , scale=1.5 ): ''' 绘制图像列表 Parameters: imgs:图像列表 num_rows:要显示图像的行数 num_cols:要显示的图像的列数 title:为每个图像设置标题 scale:缩放因子 ''' figsize = (num_cols * scale, num_rows * scale) _, axes = d2l.plt.subplots(num_rows, num_cols, figsize=figsize) axes = axes.flatten() for i, (ax, img) in enumerate (zip (axes, imgs)): if torch.is_tensor(img): ax.imshow(img.numpy()) else : ax.imshow(img) ax.axes.get_xaxis().set_visible(False ) ax.axes.get_yaxis().set_visible(False ) if titles: ax.set_title(titles[i]) return axes def get_fashion_mnist_labels (labels ): """返回Fashion-MNIST数据集的文本标签""" text_labels = ['t-shirt' , 'trouser' , 'pullover' , 'dress' , 'coat' , 'sandal' , 'shirt' , 'sneaker' , 'bag' , 'ankle boot' ] return [text_labels[int (i)] for i in labels] from torch.utils import dataX, y = next (iter (data.DataLoader(mnist_train, batch_size=18 ))) show_images(X.reshape(18 , 28 , 28 ), 2 , 9 , titles=get_fashion_mnist_labels(y));
3.读取小批量数据
这里直接使用了内置的数据迭代器不是自己写读取函数.
1 2 3 4 5 6 7 8 9 batch_size = 256 def get_dataloader_workers (): """使用4个进程来读取数据""" return 4 train_iter = data.DataLoader(mnist_train, batch_size, shuffle=True , num_workers=get_dataloader_workers())
4.获取数据并读取
其实就是一个整合.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 def load_data_fashion_mnist (batch_size, resize=None ): """下载Fashion-MNIST数据集，然后将其加载到内存中""" trans = [transforms.ToTensor()] if resize: trans.insert(0 , transforms.Resize(resize)) trans = transforms.Compose(trans) mnist_train = torchvision.datasets.FashionMNIST( root="../data" , train=True , transform=trans, download=True ) mnist_test = torchvision.datasets.FashionMNIST( root="../data" , train=False , transform=trans, download=True ) return (data.DataLoader(mnist_train, batch_size, shuffle=True , num_workers=get_dataloader_workers()), data.DataLoader(mnist_test, batch_size, shuffle=False , num_workers=get_dataloader_workers()))
(六)softmax回归从零开始实现
1.初始化和定义softmax
获取数据:
1 2 3 from IPython import displaybatch_size = 256 train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)
因为图像是28 × 28 28 \times 28 2 8 × 2 8 W \mathrm{W} W
1 2 3 4 5 num_inputs = 784 num_outputs = 10 W = torch.normal(0 , 0.01 , size=(num_inputs, num_outputs), requires_grad=True ) b = torch.zeros(num_outputs, requires_grad=True )
回顾一下sum函数,keepdim参数为True的时候能够保持原始张量轴数:
1 2 3 4 5 6 7 X = torch.tensor([[1.0 , 2.0 , 3.0 ], [4.0 , 5.0 , 6.0 ]]) X.sum (0 , keepdim=True ), X.sum (1 , keepdim=True )
定义softmax函数如下:
1 2 3 4 def softmax (X ): X_exp = torch.exp(X) partition = X_exp.sum (1 , keepdim=True ) return X_exp / partition
示例:
1 2 3 4 5 6 X = torch.normal(0 , 1 , (2 , 5 )) X_prob = softmax(X) X_prob, X_prob.sum (1 )
注意这里代码没有考虑到数值上溢或下溢,这里算不够完善.
2.定义模型
1 2 def net (X ): return softmax(torch.matmul(X.reshape((-1 , W.shape[0 ])), W) + b)
3.定义损失函数和精度
1 2 3 4 5 6 7 8 9 10 def cross_entropy (y_hat, y ): return - torch.log(y_hat[range (len (y_hat)), y]) y = torch.tensor([0 , 2 ]) y_hat = torch.tensor([[0.1 , 0.3 , 0.6 ], [0.3 , 0.2 , 0.5 ]]) cross_entropy(y_hat, y)
分类精度 :正确预测数量与总预测数量之比.也定义一个函数:
1 2 3 4 5 6 7 8 9 def accuracy (y_hat, y ): """计算预测正确的数组量""" if len (y_hat.shape) > 1 and y_hat.shape[1 ] > 1 : y_hat = y_hat.argmax(axis=1 ) cmp = y_hat.type (y.dtype) == y return float (cmp.type (y.dtype).sum ())
评估模型的精度有框架函数定义:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 class Accumulator : """在n个变量上累加""" def __init__ (self, n ): self.data = [0.0 ] * n def add (self, *args ): self.data = [a + float (b) for a, b in zip (self.data, args)] def reset (self ): self.data = [0.0 ] * len (self.data) def __getitem__ (self, idx ): return self.data[idx] def evaluate_accuracy (net, data_iter ): """计算在指定数据集上模型的精度""" if isinstance (net, torch.nn.Module): net.eval () metric = Accumulator(2 ) with torch.no_grad(): for X, y in data_iter: metric.add(accuracy(net(X), y), y.numel()) return metric[0 ] / metric[1 ]
4.训练
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 def train_epoch_ch3 (net, train_iter, loss, updater ): """训练模型一个迭代周期（定义见第3章）""" if isinstance (net, torch.nn.Module): net.train() metric = Accumulator(3 ) for X, y in train_iter: y_hat = net(X) l = loss(y_hat, y) if isinstance (updater, torch.optim.Optimizer): updater.zero_grad() l.mean().backward() updater.step() else : l.sum ().backward() updater(X.shape[0 ]) metric.add(float (l.sum ()), accuracy(y_hat, y), y.numel()) return metric[0 ] / metric[2 ], metric[1 ] / metric[2 ]
一个工具类Animator在动画中绘制数据:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 class Animator : """在动画中绘制数据""" def __init__ (self, xlabel=None , ylabel=None , legend=None , xlim=None , ylim=None , xscale='linear' , yscale='linear' , fmts=('-' , 'm--' , 'g-.' , 'r:' 1 , ncols=1 , figsize=(3.5 , 2.5 ): if legend is None : legend = [] d2l.use_svg_display() self.fig, self.axes = d2l.plt.subplots(nrows, ncols, figsize=figsize) if nrows * ncols == 1 : self.axes = [self.axes, ] self.config_axes = lambda : d2l.set_axes( self.axes[0 ], xlabel, ylabel, xlim, ylim, xscale, yscale, legend) self.X, self.Y, self.fmts = None , None , fmts def add (self, x, y ): if not hasattr (y, "__len__" ): y = [y] n = len (y) if not hasattr (x, "__len__" ): x = [x] * n if not self.X: self.X = [[] for _ in range (n)] if not self.Y: self.Y = [[] for _ in range (n)] for i, (a, b) in enumerate (zip (x, y)): if a is not None and b is not None : self.X[i].append(a) self.Y[i].append(b) self.axes[0 ].cla() for x, y, fmt in zip (self.X, self.Y, self.fmts): self.axes[0 ].plot(x, y, fmt) self.config_axes() display.display(self.fig) display.clear_output(wait=True )
多次迭代:
1 2 3 4 5 6 7 8 9 10 11 12 13 def train_ch3 (net, train_iter, test_iter, loss, num_epochs, updater ): """训练模型（定义见第3章）""" animator = Animator(xlabel='epoch' , xlim=[1 , num_epochs], ylim=[0.3 , 0.9 ], legend=['train loss' , 'train acc' , 'test acc' ]) for epoch in range (num_epochs): train_metrics = train_epoch_ch3(net, train_iter, loss, updater) test_acc = evaluate_accuracy(net, test_iter) animator.add(epoch + 1 , train_metrics + (test_acc,)) train_loss, train_acc = train_metrics assert train_loss < 0.5 , train_loss assert train_acc <= 1 and train_acc > 0.7 , train_acc assert test_acc <= 1 and test_acc > 0.7 , test_acc
如果使用自己的updater,以下是一个实例:
1 2 3 4 5 6 7 8 lr = 0.1 def updater (batch_size ): return d2l.sgd([W, b], lr, batch_size) num_epochs = 10 train_ch3(net, train_iter, test_iter, cross_entropy, num_epochs, updater)
得到结果大致如图:
5.预测
1 2 3 4 5 6 7 8 9 10 11 def predict_ch3 (net, test_iter, n=6 ): """预测标签（定义见第3章）""" for X, y in test_iter: break trues = d2l.get_fashion_mnist_labels(y) preds = d2l.get_fashion_mnist_labels(net(X).argmax(axis=1 )) titles = [true +'\n' + pred for true, pred in zip (trues, preds)] d2l.show_images( X[0 :n].reshape((n, 28 , 28 )), 1 , n, titles=titles[0 :n]) predict_ch3(net, test_iter)
结果大致如图:
(七)softmax简洁实现
基于框架能够简洁实现.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 import torchfrom torch import nnfrom d2l import torch as d2lbatch_size = 256 train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size) net = nn.Sequential(nn.Flatten(), nn.Linear(784 , 10 )) def init_weights (m ): if type (m) == nn.Linear: nn.init.normal_(m.weight, std=0.01 ) net.apply(init_weights);
在之前提到过上溢和下溢的问题,在框架中已经解决好了.这里利用的是如果每个常数o k o_k o k o j − m a x ( o k ) o_j-max(o_k) o j − m a x ( o k )
y ^ = exp ( o j − max ( o k ) ) exp ( max ( o k ) ) ∑ k exp ( o k − max ( o k ) ) exp ( max ( o k ) ) = exp ( o j − max ( o k ) ) ∑ k exp ( o k − max ( o k ) ) \begin{aligned}
\hat{y}& =\frac{\exp(o_j-\max(o_k))\exp(\max(o_k))}{\sum_k\exp(o_k-\max(o_k))\exp(\max(o_k))} \\
&=\frac{\exp(o_{j}-\max(o_{k}))}{\sum_{k}\exp(o_{k}-\max(o_{k}))}
\end{aligned} y ^ = ∑ k exp ( o k − max ( o k ) ) exp ( max ( o k ) ) exp ( o j − max ( o k ) ) exp ( max ( o k ) ) = ∑ k exp ( o k − max ( o k ) ) exp ( o j − max ( o k ) )
而避免下溢(主要是l o g ( exp ( o j − max ( o k ) ) ) log\left(\exp(o_j-\max(o_k))\right) l o g ( exp ( o j − max ( o k ) ) )
log ( y ^ j ) = log ( exp ( o j − max ( o k ) ) ∑ k exp ( o k − max ( o k ) ) ) = log ( exp ( o j − max ( o k ) ) ) − log ( ∑ k exp ( o k − max ( o k ) ) ) = o j − max ( o k ) − log ( ∑ k exp ( o k − max ( o k ) ) ) . \begin{aligned}
\log(\hat{y}_{j})& =\log\left(\frac{\exp(o_{j}-\max(o_{k}))}{\sum_{k}\exp(o_{k}-\max(o_{k}))}\right) \\
&=\log\left(\exp(o_j-\max(o_k))\right)-\log\left(\sum_k\exp(o_k-\max(o_k))\right) \\
&=o_j-\max(o_k)-\log\left(\sum_k\exp(o_k-\max(o_k))\right).
\end{aligned} log ( y ^ j ) = log ( ∑ k exp ( o k − max ( o k ) ) exp ( o j − max ( o k ) ) ) = log ( exp ( o j − max ( o k ) ) ) − log ( k ∑ exp ( o k − max ( o k ) ) ) = o j − max ( o k ) − log ( k ∑ exp ( o k − max ( o k ) ) ) .
1 2 3 loss = nn.CrossEntropyLoss(reduction='none' ) trainer = torch.optim.SGD(net.parameters(), lr=0.1 )
最后训练:
1 2 num_epochs = 10 d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer)
