Continuous Bag-of-Word(CBOW)顾名思义，即连续词袋模型，即文本以单个词为最小单位，像“support vector machine”词组也会被当做三个独立的词考虑，且是连续词袋，即目标词的前后词也作为因素考虑。
One-word context
模型结构
下图为网络模型例子，词汇表大小为 V V V N N N V V V w w w ( x 1 , x 2 , … , x V ) (x_1,x_2,\dots,x_V) ( x 1 , x 2 , … , x V ) x w x_w x w 1 1 1 0 0 0 y y y
输入层和隐藏层间的权重可由一个 V × N V\times N V × N W W W W W W N N N v w v_w v w
W = [ w 11 w 12 … w 1 N w 21 w 22 … w 2 N … … … … w V 1 w V 2 … w V N ] W = \left[ \begin{matrix} w_{11} \ \ w_{12} \ \ \dots\ \ w_{1N}\\ w_{21} \ \ w_{22} \ \ \dots\ \ w_{2N}\\ \dots \ \ \dots \ \ \dots\ \ \dots\\ w_{V1} \ \ w_{V2} \ \ \dots\ \ w_{VN}\\ \end{matrix} \right] W = w 11 w 12 … w 1 N w 21 w 22 … w 2 N … … … … w V 1 w V 2 … w V N
W W W i i i v w T v_w^T v w T x k = 1 x_k=1 x k = 1 x k ′ = 0 x_{k'}=0 x k ′ = 0 k ′ ≠ k k'\neq k k ′ = k k k k 1 1 1
h = W T x = W ( k , ⋅ ) T : = v w I T h=W^Tx=W^T_{(k,\cdot)}:=v^T_{w_I} h = W T x = W ( k , ⋅ ) T := v w I T
其实就是将 W W W k k k h h h x x x k k k 1 1 1 W W W v w I v_{w_I} v w I w I w_I w I
隐藏层到输出层的权重可用一个 N × V N\times V N × V W ′ = { w i j ′ } W'=\{w'_{ij}\} W ′ = { w ij ′ }
W ′ = [ w 11 ′ w 12 ′ … w 1 N ′ w 21 ′ w 22 ′ … w 2 N ′ … … … … w V 1 ′ w V 2 ′ … w V N ′ ] W' = \left[ \begin{matrix} w'_{11} \ \ w'_{12} \ \ \dots\ \ w'_{1N}\\ w'_{21} \ \ w'_{22} \ \ \dots\ \ w'_{2N}\\ \dots \ \ \dots \ \ \dots\ \ \dots\\ w'_{V1} \ \ w'_{V2} \ \ \dots\ \ w'_{VN}\\ \end{matrix} \right] W ′ = w 11 ′ w 12 ′ … w 1 N ′ w 21 ′ w 22 ′ … w 2 N ′ … … … … w V 1 ′ w V 2 ′ … w V N ′
基于权重，我们对于每一个词汇表里的词可计算一个分数 u j u_j u j
u j = v w j ′ T h u_j=v_{w_j}'^T h u j = v w j ′ T h
其中 v w j ′ v'_{w_j} v w j ′ W ′ W' W ′ j j j
p ( w j ∣ w I ) = y j = exp ( u j ) ∑ j ′ = 1 V exp ( u j ′ ) p(w_j|w_I)=y_j=\frac{\exp(u_j)}{\sum\limits_{j'=1}^V\exp (u_{j'})} p ( w j ∣ w I ) = y j = j ′ = 1 ∑ V e x p ( u j ′ ) e x p ( u j )
其中 y j y_j y j j j j h = W T x = W ( k , ⋅ ) T : = v w I T h=W^Tx=W^T_{(k,\cdot)}:=v^T_{w_I} h = W T x = W ( k , ⋅ ) T := v w I T u j = v w j ′ T h u_j=v_{w_j}'^T h u j = v w j ′ T h
p ( w j ∣ w I ) = exp ( v w j ′ T ) v w I ∑ j ′ = 1 V exp ( v w j ′ ′ ′ T v w I ) p(w_j|w_I)=\frac{\exp(v_{w_j}'^T)v_{w_I}}{\sum\limits_{j'=1}^V\exp(v_{w_{j'}'}'^Tv_{w_I})} p ( w j ∣ w I ) = j ′ = 1 ∑ V e x p ( v w j ′ ′ ′ T v w I ) e x p ( v w j ′ T ) v w I
这里 v w v_w v w v w ′ v_w' v w ′ w w w v w v_w v w W W W v w ′ v_w' v w ′ W ′ W' W ′ v w v_w v w v w ′ v'_w v w ′
模型目标是最大化 p ( w j ∣ w I ) = exp ( v w j ′ T ) v w I ∑ j ′ = 1 V exp ( v w j ′ ′ ′ T v w I ) p(w_j|w_I)=\frac{\exp(v_{w_j}'^T)v_{w_I}}{\sum\limits_{j'=1}^V\exp(v_{w_{j'}'}'^Tv_{w_I})} p ( w j ∣ w I ) = j ′ = 1 ∑ V e x p ( v w j ′ ′ ′ T v w I ) e x p ( v w j ′ T ) v w I w I w_I w I w O w_O w O j ∗ j^* j ∗ y y y y y y k k k 1 1 1 0 0 0 k k k 1 1 1 0 0 0
模型训练
1）隐藏层到输出层权重更新
训练目标即最大化 p ( w j ∣ w I ) = exp ( v w j ′ T ) v w I ∑ j ′ = 1 V exp ( v w j ′ ′ ′ T v w I ) p(w_j|w_I)=\frac{\exp(v_{w_j}'^T)v_{w_I}}{\sum\limits_{j'=1}^V\exp(v_{w_{j'}'}'^Tv_{w_I})} p ( w j ∣ w I ) = j ′ = 1 ∑ V e x p ( v w j ′ ′ ′ T v w I ) e x p ( v w j ′ T ) v w I w I w_I w I w O w_O w O
max p ( w O ∣ w I ) = max y j ∗ \max p(w_O|w_I)=\max y_{j^*} max p ( w O ∣ w I ) = max y j ∗
= max log y j ∗ =\max \log y_{j^*} = max log y j ∗
= u j ∗ − log ∑ j ′ = 1 V exp ( u j ′ ) : = − E = u_{j^*}-\log \sum\limits_{j'=1}^V\exp(u_{j'}):= -E = u j ∗ − log j ′ = 1 ∑ V exp ( u j ′ ) := − E
上式给了损失函数的定义，即 E = − log p ( w O ∣ w I ) E = -\log p(w_O|w_I) E = − log p ( w O ∣ w I ) E E E u j ∗ u_{j^*} u j ∗ j ∗ j^* j ∗
首先我们对损失函数 E E E u j u_j u j
∂ E ∂ u j = y j − t j : = e j \frac{\partial E}{\partial u_j}=y_j-t_j:=e_j ∂ u j ∂ E = y j − t j := e j
上式给出了 e j e_j e j t j t_j t j j j j y y y 1 1 1 0 0 0 e j e_j e j
我们根据链式法则求出损失函数 E E E W ′ W' W ′ w i j ′ w'_{ij} w ij ′
∂ E ∂ w i j ′ = ∂ E ∂ u j ∂ u j ∂ w i j ′ = e j ⋅ h i \frac{\partial E}{\partial w'_{ij}}=\frac{\partial E}{\partial u_j}\frac{\partial u_j}{\partial w'_{ij}}=e_j\cdot h_i ∂ w ij ′ ∂ E = ∂ u j ∂ E ∂ w ij ′ ∂ u j = e j ⋅ h i
因此，用随机梯度下降法，我们可以得到隐藏层到输出层的权重更新公式：
w i j ′ ( n e w ) = w i j ′ ( o l d ) − η ⋅ e j ⋅ h i w'^{(new)}_{ij}=w'^{(old)}_{ij}-\eta\cdot e_j\cdot h_i w ij ′ ( n e w ) = w ij ′ ( o l d ) − η ⋅ e j ⋅ h i
或者 v w j ′ ( n e w ) = v w j ′ ( o l d ) − η ⋅ e j ⋅ h f o r j = 1 , 2 , … , V v'^{(new)}_{w_j}=v'^{(old)}_{w_j}-\eta\cdot e_j\cdot h\ \ \ \ for\ j=1,2,\dots,V v w j ′ ( n e w ) = v w j ′ ( o l d ) − η ⋅ e j ⋅ h f or j = 1 , 2 , … , V
其中 η > 0 \eta>0 η > 0 e j = y j − t j e_j=y_j-t_j e j = y j − t j h i h_i h i i i i v w j ′ v'_{w_j} v w j ′ w j w_j w j y j y_j y j t j t_j t j
（1）如果 y j > t j y_j>t_j y j > t j v w j ′ v'_{w_j} v w j ′ h h h v w I v_{w_I} v w I v w j ′ v'_{w_j} v w j ′
就会与向量 v w I v_{w_I} v w I
（2）如果 y j < t j y_j<t_j y j < t j t j = 1 t_j=1 t j = 1 w j = w O w_j=w_O w j = w O h h h
的一部分加入 v w O ′ v'_{w_O} v w O ′ v w O ′ v'_{w_O} v w O ′ v w I v_{w_I} v w I
（3）如果 y j y_j y j t j t_j t j e j = y j − t j e_j = y_j-t_j e j = y j − t j 0 0 0
2）输入层到隐藏层权重更新
我们继续对损失函数 E E E h i h_i h i
∂ E ∂ h i = ∑ j = 1 V ∂ E ∂ u j ∂ u j ∂ h i = ∑ j = 1 V e j w i j ′ : = E H i \frac{\partial E}{\partial h_i}=\sum\limits_{j=1}^V\frac{\partial E}{\partial u_j}\frac{\partial u_j}{\partial h_i}=\sum\limits_{j=1}^Ve_jw'_{ij}:=EH_i ∂ h i ∂ E = j = 1 ∑ V ∂ u j ∂ E ∂ h i ∂ u j = j = 1 ∑ V e j w ij ′ := E H i
其中 h i h_i h i i i i u j u_j u j j j j e j = y j − t j e_j = y_j-t_j e j = y j − t j j j j E H EH E H N N N e j e_j e j w i j ′ w'_{ij} w ij ′ j = 1 j=1 j = 1 V V V
接下来，我们需要求出损失函数 E E E W W W h i h_i h i x x x
h i = ∑ k = 1 V x k ⋅ w k i h_i = \sum\limits_{k=1}^Vx_k\cdot w_{ki} h i = k = 1 ∑ V x k ⋅ w ki
因此对于权重矩阵 W W W E E E
∂ E ∂ w k i = ∂ E ∂ h i ∂ h i ∂ w k i = E H i ⋅ x k \frac{\partial E}{\partial w_{ki}}=\frac{\partial E}{\partial h_i}\frac{\partial h_i}{\partial w_{ki}}=EH_i\cdot x_k ∂ w ki ∂ E = ∂ h i ∂ E ∂ w ki ∂ h i = E H i ⋅ x k
因此我们利用张量乘积的方式，便可得到
∂ E ∂ W = x ⊗ E H = x E H T \frac{\partial E}{\partial W}=x\otimes EH=xEH^T ∂ W ∂ E = x ⊗ E H = x E H T
我们再次得到了一个 N × V N\times V N × V x x x 0 0 0 ∂ E ∂ W \frac{\partial E}{\partial W} ∂ W ∂ E N N N 0 0 0 E H T EH^T E H T W W W
v w I ( n e w ) = v w I ( o l d ) − η E H T v^{(new)}_{w_I}=v^{(old)}_{w_I}-\eta EH^T v w I ( n e w ) = v w I ( o l d ) − η E H T
其中 v w I v_{w_I} v w I W W W W W W 0 0 0 v w I v_{w_I} v w I W W W 0 0 0
与矩阵 W ′ W' W ′
（1）如果过高地估计了某个单词 w j w_j w j y j > t j y_j>t_j y j > t j
w I w_I w I w j w_j w j
（2） 如果相反，某个单词 w j w_j w j y j < t j y_j<t_j y j < t j w I w_I w I
入向量与单词 w j w_j w j
（3） 如果对于单词 w I w_I w I
因此，上下文单词 w I w_I w I
Multi-word context
根据字面意思我们就可以看出，基于multi-word context的CBOW模型就是利用多个上下文单词来推测中心单词target word的一种模型。比如下面这段话，我们的上下文大小取值为4，特定的这个词是"Learning"，也就是我们需要的输出词向量,上下文对应的词有8个，前后各4个，这8个词是我们模型的输入。由于CBOW使用的是词袋模型，因此这8个词都是平等的，也就是不考虑他们和我们关注的词之间的距离大小，只要在我们上下文之内即可。
模型结构
其隐藏层的输出值的计算过程为：首先将输入的上下文单词（context words）的向量叠加起来并取其平均值，接着与输入层到隐藏层的权重矩阵相乘，作为最终的结果，公式如下：
h = 1 C W T ( x 1 + x 2 + ⋯ x C ) h = \frac{1}{C}W^T(x_1+x_2+\cdots x_C) h = C 1 W T ( x 1 + x 2 + ⋯ x C )
= 1 C ( v w 1 + v w 2 + ⋯ + v w C ) T =\frac{1}{C}(v_{w_1}+v_{w_2}+\cdots+v_{w_C})^T = C 1 ( v w 1 + v w 2 + ⋯ + v w C ) T
其中 C C C w 1 , w 2 , … , w C w_1,w_2,\dots,w_C w 1 , w 2 , … , w C v w v_w v w w w w
E = − log p ( w O ∣ w I , 1 , … , w I , C ) E = -\log p(w_O|w_{I,1},\dots,w_{I,C}) E = − log p ( w O ∣ w I , 1 , … , w I , C )
= − u j ∗ + log ∑ j ′ = 1 V exp ( u j ′ ) =-u_{j^*}+\log \sum\limits_{j'=1}^V\exp(u_{j'}) = − u j ∗ + log j ′ = 1 ∑ V exp ( u j ′ )
= − v w O ′ T ⋅ h + log ∑ j ′ = 1 V exp ( v w j ′ T ⋅ h ) = -v_{w_O}'^T\cdot h+\log\sum\limits_{j'=1}^V\exp(v_{w_j}'^T\cdot h) = − v w O ′ T ⋅ h + log j ′ = 1 ∑ V exp ( v w j ′ T ⋅ h )
网络结构如下：
模型训练
1）隐藏层到输出层权重更新
同样，由隐藏层到输出层的权重更新公式与One-word context模型下的一模一样，即类似于公式（11），我们直接写在下面：
v w j ′ ( n e w ) = v w j ′ ( o l d ) − η ⋅ e j ⋅ h f o r j = 1 , 2 , … , V v'^{(new)}_{w_j}=v'^{(old)}_{w_j}-\eta\cdot e_j\cdot h\ \ \ \ for\ j=1,2,\dots,V v w j ′ ( n e w ) = v w j ′ ( o l d ) − η ⋅ e j ⋅ h f or j = 1 , 2 , … , V
2）输入层到隐藏层权重更新
由输入层到隐藏层的权重矩阵更新公式与公式（16）类似，只不过现在我们需要对每一个上下文单词 w I , c w_{I,c} w I , c
v w I , c ( n e w ) = v w I , c ( o l d ) − 1 C η E H T f o r c = 1 , 2 , … , C v^{(new)}_{w_{I,c}}=v^{(old)}_{w_{I,c}}-\frac{1}{C}\eta EH^T \ \ \ \ for\ c=1,2,\dots,C v w I , c ( n e w ) = v w I , c ( o l d ) − C 1 η E H T f or c = 1 , 2 , … , C
其中 v w I , c v_{w_{I,c}} v w I , c c c c η \eta η E H = ∂ E ∂ h i EH = \frac{\partial E}{\partial h_i} E H = ∂ h i ∂ E
Source
