![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918144730732.png\")
Deep Learning<\/h1>\n\n
Deep Learning<\/h1>\ncreate by Arwin Yu<\/p>\n
\n ![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918144800208.png\")
\n<\/p>\n
![](\"https:\/\/img.icons8.com\/bubbles\/50\/000000\/checklist.png\")
Agenda<\/h3>\n![](\"https:\/\/img.icons8.com\/cute-clipart\/64\/000000\/alarm.png\")
The Attention Mechanism<\/h2>\nTransformer\u6a21\u578b\u4e2d\u6700\u5173\u952e\u90e8\u5206\u5c31\u662f\u81ea\u6ce8\u610f\u529b\uff08Self-Attention\uff09\u673a\u5236\uff0c\u6b63\u5982 Transformer \u7684\u8bba\u6587\u7684\u6807\u9898\u662f\u201cAttention Is All You Need\u201d\uff01\u4ee5\u6587\u672c\u95ee\u9898\u4e3a\u4f8b\u6765\u8bb2\u89e3\u8fd9\u4e2a\u673a\u5236\u3002\u5728\u5904\u7406\u6587\u672c\u95ee\u9898\u65f6\uff0c\u81ea\u6ce8\u610f\u529b\u673a\u5236\u4f1a\u544a\u8bc9\u6a21\u578b\uff1a\u5728\u5904\u7406\u53e5\u5b50\u4e2d\u7684\u6bcf\u4e2a\u5355\u8bcd\u65f6\uff0c\u7279\u522b\u5173\u6ce8\u67d0\u4e9b\u91cd\u8981\u7684\u5355\u8bcd\uff0c\u5e76\u6216\u591a\u6216\u5c11\u5730\u5ffd\u7565\u5176\u5b83\u5355\u8bcd\u3002\u7b80\u5355\u6765\u8bf4\uff0c\u5c31\u662f\u7ed9\u53e5\u5b50\u4e2d\u4e0d\u540c\u5355\u8bcd\u5206\u914d\u4e0d\u540c\u7684\u6743\u91cd\u3002\u8fd9\u662f\u7b26\u5408\u5e38\u7406\u7684\uff0c\u56e0\u4e3a\u4e00\u53e5\u8bdd\u4e2d\u7684\u6bcf\u4e2a\u5355\u8bcd\u91cd\u8981\u7a0b\u5ea6\u662f\u4e0d\u4e00\u6837\u7684\uff0c\u4ece\u8bed\u6cd5\u89d2\u5ea6\u8bf4\uff0c\u4e3b\u8c13\u5bbe\u8bed\u6bd4\u5176\u5b83\u53e5\u5b50\u6210\u5206\u66f4\u91cd\u8981\uff0cself-attention\u673a\u5236\u5c31\u662f\u6a21\u578b\u5c1d\u8bd5\u5b66\u4e60\u53e5\u5b50\u6210\u5206\u91cd\u8981\u7a0b\u5ea6\u7684\u65b9\u6cd5\u3002<\/p>\n
self-attention\u53ef\u4ee5\u901a\u8fc7\u5b66\u4e60\u53e5\u5b50\u6210\u5206\u91cd\u8981\u7a0b\u5ea6\u66f4\u597d\u7684\u7406\u89e3\u8bed\u8a00\u7684\u4e0a\u4e0b\u6587\uff0c\u800c\u4e0a\u4e0b\u6587\u5bf9\u4e8e\u8bed\u8a00\u6a21\u578b\u6765\u8bf4\u662f\u81f3\u5173\u91cd\u8981\u7684\u3002\u4f8b\u5982\uff0c\u770b\u4e00\u4e0b\u673a\u5668\u4eba\u7b2c\u4e8c\u5b9a\u5f8b\uff1a<\/p>\n
\u673a\u5668\u4eba\u7b2c\u4e8c\u5b9a\u5f8b\uff1a\u673a\u5668\u4eba\u5fc5\u987b\u670d\u4ece\u4eba\u7c7b\u53d1\u51fa\u7684\u547d\u4ee4\uff0c\u9664\u975e\u8fd9\u4e9b\u547d\u4ee4\u4e0e\u7b2c\u4e00\u5b9a\u5f8b\u76f8\u51b2\u7a81\u3002<\/strong><\/p>\n \u5f53\u6a21\u578b\u5904\u7406\u8fd9\u53e5\u8bdd\u65f6\uff0c\u5b83\u5fc5\u987b\u80fd\u591f\u77e5\u9053\uff1a<\/p>\n \u5b83\u6307\u7684\u662f\u673a\u5668\u4eba<\/p>\n \u8fd9\u79cd\u547d\u4ee4\u6307\u7684\u662f\u6cd5\u5f8b\u7684\u524d\u534a\u90e8\u5206\uff0c\u5373\u201c\u4eba\u7c7b\u53d1\u51fa\u7684\u547d\u4ee4\u201d<\/p>\n \u7b2c\u4e00\u5b9a\u5f8b\u6307\u7684\u662f\u6574\u4e2a\u7b2c\u4e00\u5b9a\u5f8b\u7b49\u7b49<\/p>\n \u6ce8\u610f\u529b\u5c42\u7684\u8f93\u5165\u88ab\u79f0\u4e3a\u67e5\u8be2\u5411\u91cf$q$\u3002$q$\u662f\u6211\u4eec\u5e0c\u671b\u5bf9\u5176\u8fdb\u884c\u6ce8\u610f\u529b\u5904\u7406\u7684\u8f93\u5165\u4fe1\u606f\u3002<\/p>\n<\/li>\n \u5bf9\u4e8e\u6bcf\u4e2a\u67e5\u8be2\u5411\u91cf$q$\uff0c\u6ce8\u610f\u529b\u673a\u5236\u4f1a\u6839\u636e\u8bb0\u5fc6\u8fd4\u56de\u4e00\u4e2a\u8f93\u51fa\u3002\u8bb0\u5fc6\u662f\u4e00\u7ec4\u5728\u6ce8\u610f\u529b\u5c42\u4e2d\u7f16\u7801\u7684\u952e-\u503c\u5bf9(key-value, \u7b80\u79f0$k$\u5411\u91cf\uff0c$v$\u5411\u91cf)\u3002\u8fd9\u4e9b\u952e-\u503c\u5bf9\u5e2e\u52a9\u6a21\u578b\u627e\u5230\u4e0e\u67e5\u8be2\u76f8\u5173\u7684\u4fe1\u606f\u3002<\/p>\n<\/li>\n<\/ul>\n \u6ce8\u610f\u529b\u4e3b\u8981\u6709\u4e24\u79cd\u7c7b\u578b\uff1a<\/p>\n \u81ea\u6211\u6ce8\u610f\u529b<\/strong>\u6307\u7684\u662f\u5728\u540c\u4e00\u4e2a\u8f93\u5165\u5e8f\u5217\u4e2d\uff0c\u67e5\u8be2\u3001\u952e\u548c\u503c\u90fd\u662f\u6765\u81ea\u540c\u4e00\u7ec4\u5e8f\u5217\u3002\u4e5f\u5c31\u662f\u8bf4\uff0c\u6a21\u578b\u5728\u5904\u7406\u6bcf\u4e2a\u5143\u7d20\u65f6\uff0c\u90fd\u4f1a\u67e5\u770b\u6574\u4e2a\u5e8f\u5217\u6765\u8ba1\u7b97\u6bcf\u4e2a\u5143\u7d20\u4e4b\u95f4\u7684\u76f8\u4f3c\u6027\u3002<\/p>\n \u4ea4\u53c9\u6ce8\u610f\u529b<\/strong>\u5219\u662f\u6307\u67e5\u8be2\u548c\u952e-\u503c\u5bf9\u6765\u81ea\u4e0d\u540c\u7684\u8f93\u5165\u5e8f\u5217\u3002\u4f8b\u5982\uff0c\u5728\u673a\u5668\u7ffb\u8bd1\u4e2d\uff0c\u4e00\u4e2a\u5e8f\u5217\u53ef\u80fd\u662f\u6e90\u8bed\u8a00\u53e5\u5b50\uff0c\u53e6\u4e00\u4e2a\u5e8f\u5217\u662f\u76ee\u6807\u8bed\u8a00\u53e5\u5b50\u3002<\/p>\n \n \u591a\u5934\u6ce8\u610f\u529b<\/strong>\u6307\u7684\u662f\u901a\u8fc7\u5e76\u884c\u8ba1\u7b97\u591a\u4e2a\u6ce8\u610f\u529b\u673a\u5236\u6765\u6355\u6349\u8f93\u5165\u5e8f\u5217\u4e2d\u4e0d\u540c\u90e8\u5206\u7684\u4e0d\u540c\u7279\u5f81\uff0c\u8fd9\u6837\u6a21\u578b\u53ef\u4ee5\u66f4\u5168\u9762\u5730\u7406\u89e3\u548c\u5904\u7406\u5e8f\u5217\u6570\u636e\u3002<\/p>\n \n \u8fd9\u4e09\u4e2a\u5411\u91cf\u7684\u521b\u5efa\u8fc7\u7a0b\u5728\u6a21\u578b\u5b9e\u73b0\u65f6\u975e\u5e38\u7b80\u5355\uff0c\u901a\u8fc7\u795e\u7ecf\u7f51\u7edc\u5c42\u7684\u6620\u5c04\u5373\u53ef\u5f97\u5230\u3002\u5177\u4f53\u6765\u8bf4\uff0c\u8f93\u5165\u6570\u636e\u4e3atoken\u672c\u8eab\uff08\u5047\u8bbe64\u7ef4\uff09\uff0c\u800c\u6620\u5c04\u540e\u7684\u8f93\u5165\u5411\u91cf\u53ef\u4ee5\u662f192\u7ef4\uff0c\u6b64\u65f6\u7b2c0-63\u7ef4\u4f5c\u4e3aq\u5411\u91cf\uff0c64-127\u7ef4\u4f5c\u4e3ak\u5411\u91cf\uff0c\u800c128-192\u7ef4\u4f5c\u4e3av\u5411\u91cf\u3002\u8bf7\u6ce8\u610f\uff0c\u67e5\u8be2\u5411\u91cf\u3001\u952e\u5411\u91cf\u548c\u503c\u5411\u91cf\u662f\u4e3a\u8ba1\u7b97\u548c\u601d\u8003\u6ce8\u610f\u529b\u673a\u5236\u800c\u62bd\u8c61\u51fa\u7684\u6982\u5ff5\uff0c\u6216\u8005\u8bf4\u662f\u6211\u4eec\u5bf9\u6a21\u578b\u7684\u5b66\u4e60\u671f\u671b\u3002\u56e0\u4e3a\u8fd9\u4e09\u4e2a\u65b0\u5411\u91cf\u5728\u521a\u521b\u5efa\u65f6\u662f\u968f\u673a\u521d\u59cb\u5316\u7684\uff0c\u6ca1\u6709\u7279\u6b8a\u542b\u4e49\uff0c\u662f\u7ecf\u8fc7\u6a21\u578b\u8bad\u7ec3\u5206\u522b\u5f97\u5230\u4e86\u7c7b\u4f3c\u67e5\u8be2\u3001\u56de\u590d\u3001\u5b58\u503c\u7b49\u5411\u91cf\u529f\u80fd\uff0c\u4e00\u4e2a\u8bcd\u5411\u91cf\u53ef\u4ee5\u901a\u8fc7\u5b83\u4eec\u4e0e\u5176\u5b83\u8bcd\u5411\u91cf\u8fdb\u884c\u4e92\u52a8\u6765\u5efa\u6a21\u8bcd\u4e0e\u8bcd\u4e4b\u95f4\u7684\u76f8\u5173\u6027\u3002\u5728\u8bfb\u8005\u9605\u8bfb\u5b8c\u63a5\u4e0b\u6765\u7684\u5168\u90e8\u8ba1\u7b97\u8fc7\u7a0b\u4e4b\u540e\uff0c\u5c31\u4f1a\u660e\u767d\u5b83\u4eec\u540d\u5b57\u7684\u7531\u6765\u3002<\/p>\n self-attention\u7684\u4e00\u4e2a\u7c97\u7565\u7684\u7c7b\u6bd4\u662f\u5c06\u5176\u60f3\u8c61\u4e3a\u5728\u6587\u4ef6\u67dc\u4e2d\u641c\u7d22\u3002\u67e5\u8be2\u5411\u91cf$q$\u5c31\u50cf\u4e00\u5f20\u4fbf\u7b7e\u7eb8\uff0c\u4e0a\u9762\u5199\u7740\u60a8\u6b63\u5728\u7814\u7a76\u7684\u4e3b\u9898\u3002$k$\u5411\u91cf\u5c31\u50cf\u67dc\u5b50\u5185\u6587\u4ef6\u5939\u7684\u6807\u7b7e\u3002\u5f53\u4f60\u5c06\u6807\u7b7e\u4e0e\u4fbf\u7b7e\u5339\u914d\u65f6\uff0c\u6211\u4eec\u53d6\u51fa\u8be5\u6587\u4ef6\u5939\u7684\u5185\u5bb9\uff0c\u8fd9\u4e9b\u5185\u5bb9\u5c31\u662f\u503c\u5411\u91cf$v$\u3002<\/p>\n \n \u6bcf\u4e2a\u6587\u4ef6\u5939\u7684\u6743\u91cd\u5206\u6570\u662f\u901a\u8fc7\u67e5\u8be2\u5411\u91cf\u4e0e\u6b63\u5728\u8bc4\u5206\u7684\u76f8\u5e94\u5355\u8bcd\u7684\u952e\u5411\u91cf\u7684\u70b9\u79ef\u8ba1\u7b97\u5f97\u51fa\u7684\u3002\u70b9\u79ef\u7684\u516c\u5f0f\uff1a $a \\times b=|a| \\times|b| \\times \\cos \\theta$ \u3002\u5176\u610f\u4e49\u5c31\u662f\u6bd4\u8f83\u4e24\u4e2a\u5411\u91cf\u7684\u76f8\u5173\u7a0b\u5ea6\uff0c\u76f8\u5173\u6027\u8d8a\u9ad8\uff0c\u5206\u6570\u8d8a\u5927\u3002\u6ce8\u610f\uff0c\u70b9\u79ef\u540e\u9700\u8981\u5bf9\u7ed3\u679c\u8fdb\u884csoftmax\u6620\u5c04\u5f97\u5230\u6743\u91cd\u5206\u6570\uff0cSoftmax\u6620\u5c04\u540e\u7684\u5206\u6570\u51b3\u5b9a\u4e86\u6bcf\u4e2a\u8bcd\u5728\u53e5\u5b50\u4e2d\u67d0\u4e2a\u4f4d\u7f6e\u7684\u91cd\u8981\u6027\u3002<\/p>\n \n \u6211\u4eec\u5c06\u6bcf\u4e2a\u503c\u5411\u91cf\u4e58\u4ee5\u5b83\u7684\u6743\u91cd\u5206\u6570\u5e76\u6c42\u548c\u2014\u2014\u5f97\u5230\u6211\u4eec\u7684\u81ea\u6ce8\u610f\u529b\u7ed3\u679c\u3002<\/p>\n \n Image Source<\/a><\/p>\n \u603b\u7ed3\u4e00\u4e0b\uff0cself-attention\u516c\u5f0f\u5206\u4e3a\u4e09\u4e2a\u6b65\u9aa4:<\/p>\n \n Image by Lena Voita<\/a><\/p>\n \uff081\uff09\u8ba1\u7b97 $Q$ \u548c $K$ \u4e4b\u95f4\u7684\u76f8\u4f3c\u5ea6\uff0c\u5373 $Q K^{\\mathrm{T}}$ \u3002<\/p>\n \uff082\uff09\u7531\u4e8e $Q$ \u548c $K$ \u7684\u7ef4\u5ea6\u53ef\u80fd\u5f88\u5927, \u56e0\u6b64\u9700\u8981\u5c06\u5176\u9664\u4ee5$\\sqrt{d_k}$ \u6765\u7f29\u653e\u3002\u8fd9\u6709\u52a9\u4e8e\u907f\u514d\u5728 Softmax \u8ba1\u7b97\u65f6\u51fa\u73b0\u68af\u5ea6\u6d88\u5931\u6216\u68af\u5ea6\u7206\u70b8\u7684\u95ee\u9898\u3002<\/p>\n \uff083\uff09\u5bf9\u76f8\u4f3c\u5ea6\u77e9\u9635\u8fdb\u884c Softmax \u64cd\u4f5c, \u5f97\u5230\u6bcf\u4e2a\u67e5\u95f4\u5411\u91cf\u4e0e\u6240\u6709\u952e\u5411\u91cf\u7684\u6743\u91cd\u5206\u5e03\u3002\u7136\u540e, \u5c06\u8fd9\u4e9b\u6743\u91cd\u4e0e\u503c\u77e9\u9635 $V$ \u76f8\u4e58\u5e76\u76f8\u52a0, \u5f97\u5230\u81ea\u6ce8\u610f\u529b\u673a\u5236\u7684\u8f93\u51fa\u77e9\u9635\u3002<\/p>\n \u201c\u591a\u5934\u6ce8\u610f\u529b\u201d\uff08Multi-heads self-attention\uff09\u7684\u673a\u5236\u8fdb\u4e00\u6b65\u7ec6\u5316\u4e86\u81ea\u6ce8\u610f\u529b\u5c42\u3002\u5bf9\u4e8e\u591a\u5934\u6ce8\u610f\u529b\uff0c\u5176\u4e2d\u6709\u591a\u7ec4\u67e5\u8be2\u5411\u91cf\u3001\u952e\u5411\u91cf\u548c\u503c\u5411\u91cf\uff0c\u8fd9\u91cc\u628a\u4e00\u7ec4q, k, v\u79f0\u4e4b\u4e3a\u4e00\u4e2a\u5934\uff0cTransformer\u539f\u8bba\u6587\u4e2d\u4f7f\u7528\u516b\u4e2a\u6ce8\u610f\u529b\u5934\u3002\u6bcf\u7ec4\u6ce8\u610f\u529b\u5934\u90fd\u662f\u53ef\u8bad\u7ec3\u7684\uff0c\u7ecf\u8fc7\u8bad\u7ec3\u53ef\u4ee5\u6269\u5c55\u6a21\u578b\u5173\u6ce8\u4e0d\u540c\u4f4d\u7f6e\u7684\u80fd\u529b\u3002<\/p>\n \u4e3e\u4e00\u4e2a\u5f62\u8c61\u7684\u7c7b\u6bd4\uff1a\u628a\u6ce8\u610f\u529b\u5934\u7c7b\u6bd4\u6210\u5c0f\u5b66\u751f\uff0c\u90a3\u4e48\u591a\u4e2a\u5c0f\u5b66\u751f\u5728\u5b66\u4e60\u8fc7\u7a0b\u4e2d\u4f1a\u5f62\u6210\u4e0d\u540c\u7684\u601d\u7ef4\u6a21\u5f0f\uff0c\u5bf9\u540c\u6837\u7684\u95ee\u9898\u4f1a\u4ea7\u751f\u4e0d\u540c\u7684\u7406\u89e3\u3002\u8fd9\u5c31\u662f\u4e3a\u4ec0\u4e48\u8981\u4f7f\u7528\u591a\u5934\u7684\u539f\u56e0\uff0c\u5c31\u662f\u5e0c\u671b\u6a21\u578b\u53ef\u4ee5\u4ece\u4e0d\u540c\u7684\u89d2\u5ea6\u601d\u8003\u8f93\u5165\u4fe1\u606f\uff0c\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n \n Image Source<\/a><\/p>\n \u4f46\u662f\uff0c\u591a\u5934\u6ce8\u610f\u529b\u673a\u5236\u4e5f\u7ed9\u5e26\u6765\u4e86\u4e00\u4e2a\u95ee\u9898\u3002\u5982\u679c\u4f7f\u7528\u516b\u4e2a\u5934\uff0c\u7ecf\u8fc7\u591a\u5934\u6ce8\u610f\u529b\u673a\u5236\u540e\u4f1a\u5f97\u52308\u4e2a\u8f93\u51fa\uff0c\u4f46\u662f\uff0c\u5b9e\u9645\u4e0a\u53ea\u9700\u8981\u4e00\u4e2a\u8f93\u51fa\u7ed3\u679c\u3002\u6240\u4ee5\u9700\u8981\u4e00\u79cd\u65b9\u6cd5\u5c06\u8fd9\u516b\u4e2a\u8f93\u51fa\u538b\u7f29\u6210\u4e00\u4e2a\u77e9\u9635\uff0c\u65b9\u6cd5\u4e5f\u5f88\u7b80\u5355\uff0c\u5c06\u5b83\u4eec\u4e58\u4ee5\u4e00\u4e2a\u989d\u5916\u7684\u6743\u91cd\u77e9\u9635\u5373\u53ef\u3002\u8fd9\u4e2a\u64cd\u4f5c\u53ef\u4ee5\u901a\u8fc7\u4e00\u4e2a\u795e\u7ecf\u7f51\u7edc\u5c42\u7684\u6620\u5c04\u5b8c\u6210\uff0c\u5982\u56fe\uff1a<\/p>\n \n \n Image Source<\/a><\/p>\n Transformer<\/strong>\uff1a\u57fa\u4e8e\u6ce8\u610f\u529b\u673a\u5236\u7684\u7f16\u7801\u5668-\u89e3\u7801\u5668\u67b6\u6784\uff0c\u65e8\u5728\u7ed3\u5408\u524d\u9988\u7f51\u7edc\uff08FFN\uff0c\u57fa\u672c\u4e0a\u662f\u5b9e\u73b0\u4e00\u7ef4\u5377\u79ef\u7684 MLP\uff09\u548c RNN \u7684\u4f18\u52bf\u3002<\/p>\n RNN \u88ab\u591a\u5934\u6ce8\u610f\u529b\u5c42\u53d6\u4ee3\uff0c\u901a\u8fc7\u4f4d\u7f6e\u7f16\u7801<\/strong>\u6574\u5408\u4f4d\u7f6e\u4fe1\u606f\uff0c\u5e76\u5e94\u7528\u5c42\u89c4\u8303\u5316<\/strong>\u3002<\/p>\n \u53ef\u5e76\u884c\u8ba1\u7b97\uff0c\u8bad\u7ec3\u65f6\u95f4\u660e\u663e\u7f29\u77ed\u3002<\/p>\n \u6211\u4eec\u4ec0\u4e48\u65f6\u5019\u9700\u8981\u7f16\u7801\u5668-\u89e3\u7801\u5668\uff0c\u4ec0\u4e48\u65f6\u5019\u53ef\u4ee5\u53ea\u4f7f\u7528\u7f16\u7801\u5668\u6216\u89e3\u7801\u5668\uff1f<\/strong><\/p>\n \n \u6bcf\u4e2a\u7f16\u7801\u5668\u4e2d\u7684\u81ea\u6ce8\u610f\u529b\u5c42\u5468\u56f4\u90fd\u6709\u4e00\u4e2a\u6b8b\u5dee\u8fde\u63a5\uff0c\u7136\u540e\u662f\u5c42\u5f52\u4e00\u5316\u6b65\u9aa4\u3002\u5f52\u4e00\u5316\u7684\u8f93\u51fa\u518d\u901a\u8fc7\u524d\u9988\u7f51\u7edc\uff08Feed Forward Network\uff0cFFN\uff09\u8fdb\u884c\u6620\u5c04\uff0c\u4ee5\u8fdb\u884c\u8fdb\u4e00\u6b65\u5904\u7406\u3002\u524d\u9988\u7f51\u7edc\u672c\u8d28\u4e0a\u5c31\u662f\u51e0\u5c42\u795e\u7ecf\u7f51\u7edc\u5c42\uff0c\u5176\u4e2d\u95f4\u91c7\u7528ReLU\u6fc0\u6d3b\u51fd\u6570\uff0c\u4e24\u5c42\u4e4b\u95f4\u91c7\u7528\u6b8b\u5dee\u8fde\u63a5\u3002<\/p>\n \n Image Source<\/a><\/p>\n \u5047\u8bbe $X \\in \\mathbb{R}^{l \\times d}$ :<\/p>\n \u4f4d\u7f6e\u7f16\u7801\u77e9\u9635 $P \\in \\mathbb{R}^{l \\times d}$ :<\/p>\n \u4f4d\u7f6e\u7f16\u7801\u7684\u8ba1\u7b97\u516c\u5f0f:<\/p>\n \u5176\u4e2d\uff0c<\/p>\n \u4f4d\u7f6e\u7f16\u7801\u77e9\u9635 $P$ \u4f1a\u52a0\u5230\u5d4c\u5165\u77e9\u9635 $X$ \u4e0a\uff0c\u5f62\u6210\u65b0\u7684\u8868\u793a\u77e9\u9635 $X+P$ \u3002\u8fd9\u4e2a\u65b0\u8868\u793a\u77e9\u9635\u65e2\u5305\u542b\u4e86\u8bcd\u7684\u8bed\u4e49\u4fe1\u606f\uff08\u6765\u81ea $X$ \uff09\uff0c\u4e5f\u5305\u542b\u4e86\u8bcd\u7684\u4f4d\u7f6e\u4fe1\u606f\uff08\u6765\u81ea $P$ \uff09\uff0c\u4ece\u800c\u589e\u5f3a\u4e86\u6a21\u578b\u5bf9\u5e8f\u5217\u6570\u636e\u7684\u7406\u89e3\u3002\u6b63\u5f26\u548c\u4f59\u5f26\u51fd\u6570\u751f\u6210\u7684\u7f16\u7801\u786e\u4fdd\u4e86\u4e0d\u540c\u4f4d\u7f6e\u4e0a\u7684\u8bcd\u80fd\u591f\u88ab\u533a\u5206\uff0c\u540c\u65f6\u4e5f\u5305\u542b\u4e86\u76f8\u5bf9\u4f4d\u7f6e\u4fe1\u606f\u3002<\/p>\n \n \n \u4e0a\u56fe\u7684\u7ef4\u5ea6dim\uff0c\u5bf9\u5e94\u8f93\u5165\u5e8f\u5217\u6570\u636e\u4e2d\u7684\u6bcf\u4e00\u4e2a\u5143\u7d20\u3002<\/p>\n \u7b2c 4 \u548c 6 \u7ef4\u5ea6\u4f7f\u7528 $\\sin$ \uff0c\u7b2c 5 \u548c 7 \u7ef4\u5ea6\u4f7f\u7528 $\\cos$ \u3002<\/p>\n \u5bf9\u4e8e\u8f83\u5c0f\u7684 j\uff08\u4f8b\u59824\uff0c5\uff09\uff0c\u7f29\u653e\u56e0\u5b50\u8f83\u5c0f\uff0c\u4f4d\u7f6e\u7f16\u7801\u7684\u53d8\u5316\u8f83\u5feb\uff08\u9ad8\u9891\u53d8\u5316\uff09\u3002\u5bf9\u4e8e\u8f83\u5927\u7684 j\uff08\u4f8b\u59826\uff0c7\uff09\uff0c\u7f29\u653e\u56e0\u5b50\u8f83\u5927\uff0c\u4f4d\u7f6e\u7f16\u7801\u7684\u53d8\u5316\u8f83\u6162\uff08\u4f4e\u9891\u53d8\u5316\uff09\u3002<\/p>\n \n \u201cTransformer \u89e3\u7801\u5668\u5757\u4e0e\u7f16\u7801\u5668\u5757\u975e\u5e38\u76f8\u4f3c\uff0c\u4f46\u6709\u4e00\u4e9b\u5173\u952e\u533a\u522b\u3002\u9664\u4e86\u591a\u5934\u6ce8\u610f\u5c42\u548c\u4f4d\u7f6e\u7f16\u7801\u7f51\u7edc\uff0c\u89e3\u7801\u5668\u5757\u8fd8\u5305\u542b\u4e00\u4e2a\u989d\u5916\u7684\u5b50\u5c42\uff0c\u5373\u5c06\u591a\u5934\u6ce8\u610f\u529b\u5e94\u7528\u4e8e\u7f16\u7801\u5668\u8f93\u51fa\u7684\u5b50\u5c42\u3002<\/p>\n \u4ea4\u53c9\u6ce8\u610f\uff1a\u4ea4\u53c9\u6ce8\u610f\u673a\u5236\u4e0e\u81ea\u6ce8\u610f\u673a\u5236\u7c7b\u4f3c\uff0c\u4f7f\u7528\u67e5\u8be2\u3001\u952e\u548c\u503c\u7684\u8bbe\u7f6e\u3002\u4f46\u5728\u89e3\u7801\u5668\u4e2d\uff0c\u8f93\u5165\u7a0d\u5fae\u590d\u6742\u4e00\u4e9b\u3002\u89e3\u7801\u5668\u63a5\u6536\u7684\u6570\u636e\u70b9 $y_i$ \u9996\u5148\u7ecf\u8fc7\u81ea\u6ce8\u610f\u548c add-norm \u5757\uff0c\u7136\u540e\u5230\u8fbe\u4ea4\u53c9\u6ce8\u610f\u5757\u3002\u5728\u8fd9\u4e2a\u5757\u4e2d\uff0c\u89e3\u7801\u5668\u7684\u8f93\u5165\u4f5c\u4e3a\u67e5\u8be2\uff0c\u800c\u7f16\u7801\u5668\u7684\u8f93\u51fa $h^{Enc}$\uff08\u7531\u6240\u6709\u5148\u524d\u7684\u8f93\u5165 $x_1, ..., x_t$ \u8ba1\u7b97\u5f97\u51fa\uff09\u4f5c\u4e3a\u952e\u548c\u503c\u3002<\/p>\n \u4ee5\u201c\u6211\u662f\u5b66\u751f\u201d\u2192 \u201ci am a student\u201d \u7684\u8bed\u8a00\u7ffb\u8bd1\u4efb\u52a1\u4e3a\u4f8b\uff0c\u5982\u4e0b\u56fe\u6240\u793a\u3002<\/p>\n \n \u5728\u8bed\u8a00\u7ffb\u8bd1\u7684\u4f8b\u5b50\u4e2d\uff0c\u89e3\u7801\u5668\u751f\u6210\u7684\u67e5\u8be2\u8868\u793a\u5f53\u524d\u65f6\u95f4\u6b65\u7684\u7ffb\u8bd1\u9700\u6c42\uff0c\u800c\u7f16\u7801\u5668\u7684\u952e\u548c\u503c\u63d0\u4f9b\u4e86\u4ece\u6e90\u8bed\u8a00\u53e5\u5b50\u7684\u6240\u6709\u4e4b\u524d\u65f6\u95f4\u6b65\u8f93\u5165\u4e2d\u63d0\u53d6\u5230\u7684\u4fe1\u606f\u3002<\/p>\n \u4f8b\u5982\uff0c\u5982\u679c\u6211\u4eec\u5728\u7ffb\u8bd1\u4e00\u4e2a\u53e5\u5b50\uff0c\u5f53\u89e3\u7801\u5668\u5728\u751f\u6210\u76ee\u6807\u8bed\u8a00\u53e5\u5b50\u7684\u67d0\u4e2a\u5355\u8bcd\u65f6\uff0c\u5b83\u4f1a\u4ea7\u751f\u4e00\u4e2a\u67e5\u8be2\uff0c\u8868\u793a\u5b83\u9700\u8981\u4ece\u6e90\u8bed\u8a00\u53e5\u5b50\u7684\u54ea\u4e2a\u90e8\u5206\u83b7\u53d6\u4fe1\u606f\u3002\u7f16\u7801\u5668\u7684\u8f93\u51fa\uff08\u952e\u548c\u503c\uff09\u5305\u542b\u4e86\u6e90\u8bed\u8a00\u53e5\u5b50\u7684\u6240\u6709\u4fe1\u606f\uff0c\u6bd4\u5982\u8bed\u6cd5\u7ed3\u6784\u548c\u4e0a\u4e0b\u6587\u542b\u4e49\u3002\u8fd9\u4e9b\u4fe1\u606f\u901a\u8fc7\u952e\u548c\u503c\u63d0\u4f9b\u7ed9\u89e3\u7801\u5668\u3002<\/p>\n \u901a\u8fc7\u8fd9\u79cd\u65b9\u5f0f\uff0c\u89e3\u7801\u5668\u80fd\u591f\u5728\u751f\u6210\u76ee\u6807\u8bed\u8a00\u53e5\u5b50\u7684\u4e0b\u4e00\u4e2a\u5355\u8bcd\u65f6\uff0c\u53c2\u8003\u6e90\u8bed\u8a00\u53e5\u5b50\u7684\u5168\u5c40\u4fe1\u606f\u3002\u8fd9\u79cd\u4ea4\u53c9\u6ce8\u610f\u673a\u5236\u4f7f\u5f97\u6a21\u578b\u5728\u7ffb\u8bd1\u8fc7\u7a0b\u4e2d\u80fd\u591f\u66f4\u597d\u5730\u7406\u89e3\u548c\u5229\u7528\u6e90\u8bed\u8a00\u7684\u4e0a\u4e0b\u6587\uff0c\u4ece\u800c\u63d0\u9ad8\u7ffb\u8bd1\u7684\u51c6\u786e\u6027\u548c\u8fde\u8d2f\u6027\u3002\u4f8b\u5982\uff0c\u5728\u7ffb\u8bd1\u4e00\u4e2a\u957f\u53e5\u5b50\u65f6\uff0c\u89e3\u7801\u5668\u53ef\u4ee5\u901a\u8fc7\u4ea4\u53c9\u6ce8\u610f\u673a\u5236\u786e\u4fdd\u6bcf\u4e2a\u751f\u6210\u7684\u5355\u8bcd\u90fd\u4e0e\u6e90\u8bed\u8a00\u53e5\u5b50\u7684\u76f8\u5173\u90e8\u5206\u76f8\u5bf9\u5e94\uff0c\u4ece\u800c\u751f\u6210\u4e00\u4e2a\u8fde\u8d2f\u3001\u51c6\u786e\u7684\u8bd1\u6587\u3002<\/p>\n \u5728 Transformer \u6a21\u578b\u7684\u80cc\u666f\u4e0b\uff0c\u6559\u5e08\u5f3a\u5236\uff08Teacher Forcing\uff09<\/strong>\u662f\u4e00\u79cd\u8bad\u7ec3\u7b56\u7565\uff0c\u7528\u4e8e\u63d0\u9ad8\u5e8f\u5217\u5230\u5e8f\u5217\u6a21\u578b\uff08\u5982\u673a\u5668\u7ffb\u8bd1\u3001\u6587\u672c\u751f\u6210\u7b49\uff09\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u7684\u6548\u7387\u548c\u51c6\u786e\u6027\u3002<\/p>\n \u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\uff0c\u6a21\u578b\u9700\u8981\u5b66\u4e60\u5982\u4f55\u6839\u636e\u8f93\u5165\u5e8f\u5217\u751f\u6210\u8f93\u51fa\u5e8f\u5217\u3002\u5bf9\u4e8e Transformer \u6a21\u578b\u6765\u8bf4\uff0c\u8fd9\u5305\u62ec\u7f16\u7801\u5668\u548c\u89e3\u7801\u5668\u4e24\u4e2a\u90e8\u5206\u3002\u7f16\u7801\u5668\u5c06\u8f93\u5165\u5e8f\u5217\u7f16\u7801\u6210\u9690\u85cf\u8868\u793a\uff0c\u89e3\u7801\u5668\u5219\u6839\u636e\u8fd9\u4e9b\u9690\u85cf\u8868\u793a\u751f\u6210\u8f93\u51fa\u5e8f\u5217\u3002<\/p>\n<\/li>\n \u5728\u6ca1\u6709\u6559\u5e08\u5f3a\u5236\u7684\u60c5\u51b5\u4e0b\uff0c\u89e3\u7801\u5668\u5728\u6bcf\u4e00\u6b65\u751f\u6210\u4e0b\u4e00\u4e2a\u5355\u8bcd\u65f6\uff0c\u90fd\u4f1a\u4f7f\u7528\u5b83\u81ea\u5df1\u524d\u4e00\u6b65\u751f\u6210\u7684\u5355\u8bcd\u4f5c\u4e3a\u8f93\u5165\u3002\u8fd9\u79cd\u65b9\u5f0f\u7684\u95ee\u9898\u662f\uff0c\u5982\u679c\u6a21\u578b\u5728\u67d0\u4e00\u6b65\u751f\u6210\u4e86\u9519\u8bef\u7684\u5355\u8bcd\uff0c\u9519\u8bef\u4f1a\u4e0d\u65ad\u79ef\u7d2f\uff0c\u4f7f\u5f97\u540e\u7eed\u7684\u9884\u6d4b\u66f4\u52a0\u4e0d\u51c6\u786e\u3002<\/p>\n<\/li>\n \u6559\u5e08\u5f3a\u5236\u901a\u8fc7\u4f7f\u7528\u771f\u5b9e\u7684\u76ee\u6807\u5355\u8bcd\uff08\u5373\u8bad\u7ec3\u6570\u636e\u4e2d\u7684\u6b63\u786e\u5355\u8bcd\uff09\u4f5c\u4e3a\u89e3\u7801\u5668\u5728\u6bcf\u4e00\u6b65\u7684\u8f93\u5165\uff0c\u6765\u89e3\u51b3\u8fd9\u4e2a\u95ee\u9898\u3002\u5728\u6bcf\u4e00\u6b65\u751f\u6210\u65f6\uff0c\u89e3\u7801\u5668\u4e0d\u4f7f\u7528\u5b83\u81ea\u5df1\u751f\u6210\u7684\u5355\u8bcd\uff0c\u800c\u662f\u4f7f\u7528\u8bad\u7ec3\u6570\u636e\u4e2d\u5bf9\u5e94\u7684\u6b63\u786e\u5355\u8bcd\u3002\u8fd9\u6837\u53ef\u4ee5\u907f\u514d\u9519\u8bef\u7684\u7d2f\u79ef\uff0c\u52a0\u901f\u6a21\u578b\u6536\u655b\uff0c\u5e76\u63d0\u9ad8\u8bad\u7ec3\u9636\u6bb5\u7684\u51c6\u786e\u6027\u3002<\/p>\n<\/li>\n<\/ul>\n \n Animation by Jay Alammar<\/a><\/p>\n \u5176\u7f3a\u70b9\u662f\u5728\u5b9e\u9645\u63a8\u7406\uff08\u9884\u6d4b\uff09\u65f6\uff0c\u6a21\u578b\u4e0d\u4f1a\u6709\u771f\u5b9e\u7684\u76ee\u6807\u5355\u8bcd\u53ef\u7528\uff0c\u53ef\u80fd\u5bfc\u81f4\u6a21\u578b\u5728\u8bad\u7ec3\u548c\u63a8\u7406\u9636\u6bb5\u8868\u73b0\u4e0d\u4e00\u81f4\u3002\u5728\u5b9e\u9645\u5e94\u7528\u4e2d\uff0c\u7814\u7a76\u4eba\u5458\u53ef\u80fd\u4f1a\u7ed3\u5408\u6559\u5e08\u5f3a\u5236\u4e0e\u5176\u4ed6\u8bad\u7ec3\u6280\u5de7\uff0c\u5982Scheduled Sampling\uff0c\u4ee5\u7f13\u89e3\u4e0a\u8ff0\u95ee\u9898\u3002<\/p>\n Scheduled Sampling \u662f\u4e00\u79cd\u5177\u4f53\u5b9e\u73b0\u4e0a\u8ff0\u968f\u673a\u6027\u7684\u7b56\u7565\u3002\u5176\u57fa\u672c\u601d\u60f3\u662f\uff0c\u5728\u8bad\u7ec3\u521d\u671f\uff0c\u6559\u5e08\u5f3a\u5236\u7684\u4f7f\u7528\u6982\u7387\u8f83\u9ad8\uff0c\u800c\u968f\u7740\u8bad\u7ec3\u7684\u8fdb\u884c\uff0c\u8fd9\u4e2a\u6982\u7387\u9010\u6e10\u964d\u4f4e\u3002\u4ee5\u4e0b\u662f\u5176\u5177\u4f53\u6b65\u9aa4\uff1a<\/p>\n \u9ad8\u6982\u7387\u4f7f\u7528\u6b63\u786e\u7684\u76ee\u6807\u5355\u8bcd\u3002 \u968f\u7740\u8bad\u7ec3\u8fdb\u5c55\uff0c\u9010\u6e10\u964d\u4f4e\u4f7f\u7528\u6b63\u786e\u5355\u8bcd\u7684\u6982\u7387\u3002 \u6700\u7ec8\uff0c\u4f7f\u5f97\u6a21\u578b\u51e0\u4e4e\u5b8c\u5168\u4f9d\u8d56\u81ea\u5df1\u751f\u6210\u7684\u5355\u8bcd\u3002 \n Image Source<\/a><\/p>\n \n Animation by Jay Alammar<\/a><\/p>\n HF Models Hub<\/a><\/p>\n \n \u6211\u4eec\u53ef\u4ee5\u5c06\u56fe\u50cf\u5757\u89c6\u4e3a\u6211\u4eec\u7684\u201c\u5355\u8bcd\u201d\uff0c\u5373token\u3002<\/p>\n<\/li>\n \u8fd9\u4f7f\u5f97\u6211\u4eec\u80fd\u591f\u5c06 Transformer \u67b6\u6784\u7528\u4e8e\u89c6\u89c9\u4efb\u52a1\uff01<\/p>\n<\/li>\n \u9996\u5148\uff0c\u5c06\u56fe\u50cf\u5206\u5272\u6210\u56fa\u5b9a\u5927\u5c0f\u7684\u5757\uff0c\u7136\u540e\u5c06\u6bcf\u4e2a\u5757\u7ebf\u6027\u5d4c\u5165\u3002\u7136\u540e\uff0c\u6dfb\u52a0 2D \u4f4d\u7f6e\u5d4c\u5165\uff0c\u5e76\u5c06\u5f97\u5230\u7684\u5411\u91cf\u5e8f\u5217\u9988\u9001\u5230\u6807\u51c6 Transformer \u7f16\u7801\u5668\u3002<\/p>\n<\/li>\n \u4e3a\u4e86\u6267\u884c\u5206\u7c7b\uff0c\u5c06\u4e00\u4e2a\u989d\u5916\u7684\u53ef\u5b66\u4e60\u201c\u5206\u7c7b\u6807\u8bb0\u201d\u6dfb\u52a0\u5230\u5e8f\u5217\u4e2d\uff0c\u7c7b\u4f3c\u4e8e\u57fa\u4e8e Transformer \u7684 NLP \u4efb\u52a1\u3002<\/p>\n<\/li>\n Transformers \u9700\u8981\u5927\u91cf\u6570\u636e\u624d\u80fd\u83b7\u5f97\u9ad8\u7cbe\u5ea6\uff0c\u56e0\u6b64\uff0c\u5728\u6570\u636e\u8f83\u5c11\u7684\u60c5\u51b5\u4e0b\uff0cCNN \u901a\u5e38\u6bd4 Transformers \u8868\u73b0\u66f4\u597d\u3002<\/p>\n<\/li>\n \u4e3a\u4e86\u8fbe\u5230 ViT \u7684\u9ad8\u6027\u80fd\uff0c\u901a\u5e38\u4f7f\u7528\u5927\u5c3a\u5bf8\u6570\u636e\u96c6\u8fdb\u884c\u9884\u8bad\u7ec3\uff0c\u56e0\u4e3a\u5b83\u5bf9\u5927\u6570\u636e\u96c6\u7684\u4f9d\u8d56\u88ab\u89e3\u91ca\u4e3a\u7531\u4e8e\u4f4e\u5c40\u90e8\u5f52\u7eb3\u504f\u5dee\uff0c\u8fd9\u662f CNN \u7684\u4e00\u4e2a\u91cd\u8981\u7279\u6027\u3002<\/p>\n<\/li>\n Official ViT Pre-trained Models in PyTorch<\/a>.<\/p>\n<\/li>\n ViT Models and Examples with PyTorch<\/a>.<\/p>\n<\/li>\n<\/ul>\n \n Image Source<\/a><\/p>\n Swin Transformer\uff08Shifted Window Transformer\uff09\u662f\u4e00\u79cd\u4e13\u95e8\u7528\u4e8e\u8ba1\u7b97\u673a\u89c6\u89c9\u4efb\u52a1\u7684 Transformer \u6a21\u578b\u3002\u4e0e\u4f20\u7edf\u7684 Vision Transformer (ViT) \u4e0d\u540c\uff0cSwin Transformer \u901a\u8fc7\u5f15\u5165\u6ed1\u52a8\u7a97\u53e3\u673a\u5236\uff0c\u63d0\u5347\u4e86\u5bf9\u56fe\u50cf\u7684\u5efa\u6a21\u80fd\u529b\u548c\u8ba1\u7b97\u6548\u7387\u3002<\/p>\n \n \n \n \n
\n\n
\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918144938489.png\")
\n<\/p>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918145028778.png\")
\n<\/p>\n\n
\u81ea\u6ce8\u610f\u529b\u673a\u5236\u8be6\u89e3<\/h3>\n
\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918145316731.png\")
\n<\/p>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918145420897.png\")
\n<\/p>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918145505496.png\")
\n<\/p>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918145557491.png\")
\n<\/p>\n\u591a\u5934\u81ea\u6ce8\u610f\u529b\u673a\u5236<\/h3>\n
\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918145939147.png\")
\n<\/p>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918150019565.png\")
\n<\/p>\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918150100576.gif\")
\n<\/p>\nimport torch.nn as nn\nclass MultiHeadAttention(nn.Module):\n def __init__(self, d_model, num_heads, dropout, d_input=None):\n super().__init__()\n self.num_heads = num_heads\n self.d_model = d_model\n if d_input is None:\n d_xq = d_xk = d_xv = d_model\n else:\n d_xq, d_xk, d_xv = d_input\n\n # Make sure that the embedding dimension of model is a multiple of number of heads\n assert d_model % self.num_heads == 0\n\n self.d_k = d_model \/\/ self.num_heads # here d is divided between the heads\n # each head has hidden dimension d\n\n # These are still of dimension d_model. They will be split into number of heads \n self.W_q = nn.Linear(d_xq, d_model, bias=False)\n self.W_k = nn.Linear(d_xk, d_model, bias=False)\n self.W_v = nn.Linear(d_xv, d_model, bias=False)\n\n # Outputs of all sub-layers need to be of dimension d_model\n self.W_h = nn.Linear(d_model, d_model)\n\n self.dropout = nn.Dropout(dropout)\n\n def scaled_dot_product_attention(self, Q, K, V):\n batch_size = Q.size(0) \n k_length = K.size(-2) \n\n # Scaling by d_k so that the soft(arg)max doesn't saturate\n Q = Q \/ np.sqrt(self.d_k) # (bs, n_heads, q_length, dim_per_head)\n scores = torch.matmul(Q, K.transpose(2, 3)) # (bs, n_heads, q_length, k_length)\n\n A = torch.softmax(scores, dim=-1) # (bs, n_heads, q_length, k_length)\n A = self.dropout(A)\n\n # Get the weighted average of the values\n H = torch.matmul(A, V) # (bs, n_heads, q_length, dim_per_head)\n\n return H, A \n\n def split_heads(self, x, batch_size):\n """\n Split the last dimension into (heads X depth)\n Return after transpose to put in shape (batch_size X num_heads X seq_length X d_k)\n """\n return x.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)\n\n def group_heads(self, x, batch_size):\n """\n Combine the heads again to get (batch_size X seq_length X (num_heads times d_k))\n """\n return x.transpose(1, 2).contiguous().view(batch_size, -1, self.num_heads * self.d_k)\n\n def forward(self, X_q, X_k, X_v):\n batch_size, seq_length, dim = X_q.size() # dim = embedding dimension\n\n # After transforming, split into num_heads \n Q = self.split_heads(self.W_q(X_q), batch_size) # (bs, n_heads, q_length, dim_per_head)\n K = self.split_heads(self.W_k(X_k), batch_size) # (bs, n_heads, k_length, dim_per_head)\n V = self.split_heads(self.W_v(X_v), batch_size) # (bs, n_heads, v_length, dim_per_head)\n\n # Calculate the attention weights for each of the heads\n H_cat, A = self.scaled_dot_product_attention(Q, K, V)\n\n # Put all the heads back together by concat\n H_cat = self.group_heads(H_cat, batch_size) # (bs, q_length, dim)\n\n # Final linear layer \n H = self.W_h(H_cat) # (bs, q_length, dim)\n\n return H, A<\/code><\/pre>\n![](\"https:\/\/img.icons8.com\/color\/96\/000000\/transformer.png\"){kind=spacer}
The Transformer<\/h2>\n
\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918150158694.png\")
\n<\/p>\n<\/li>\n<\/ul>\nTransformer's Encoder Module<\/h4>\n
\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918150240291.png\")
\n<\/p>\n"""\nFeed Forward Network (FFN): an MLP with one hidden layer and ReLU activation applied to each and every element in the set.\n"""\nclass FFN(nn.Module):\n def __init__(self, d_model, hidden_dim_multiplier=4, resid_pdrop=0.1):\n super().__init__()\n self.fc_1 = nn.Linear(d_model, hidden_dim_multiplier * d_model)\n self.act = nn.ReLU(True) # inplace=True, saves a little bit of memory\n self.proj = nn.Linear(hidden_dim_multiplier * d_model, d_model)\n self.dropout = nn.Dropout(resid_pdrop)\n\n def forward(self, x):\n # x: [batch_size, seq_len, embed_dim]\n x = self.dropout(self.proj(self.act(self.fc_1(x)))) # [batch_size, seq_len, embed_dim]\n return x<\/code><\/pre>\nimport torch\nffn = FFN(d_model=4, hidden_dim_multiplier=2)\nffn.eval()\nffn(torch.ones((2, 3, 4)))[0] # batch_size = 2, seq_len = 3, embed_dim = 4\n# note that the FFN only operates on the last dimension\n# PyTorch's nn.Linear know how to handle tensors of shape [batch_size, seq_len, embed_dim]<\/code><\/pre>\ntensor([[ 0.2905, -0.0280, -0.2088, -0.0021],\n [ 0.2905, -0.0280, -0.2088, -0.0021],\n [ 0.2905, -0.0280, -0.2088, -0.0021]], grad_fn=Positional Encoding<\/h4>\n
\n\n
sin<\/code> \u548c cos<\/code> \u51fd\u6570\u6765\u6dfb\u52a0\u8be5\u4fe1\u606f\u3002<\/li>\n<\/ul>\n\n
\n
\n
\n$$
\nP_{i, 2 j}=\\sin \\left(\\frac{i}{10000^{2 j \/ d}}\\right)
\n$$<\/li>\n
\n$$
\nP_{i, 2 j+1}=\\cos \\left(\\frac{i}{10000^{2 j \/ d}}\\right)
\n$$<\/li>\n<\/ul>\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918151051367.png\")
\n<\/p>\n \u4e3a\u4ec0\u4e48\u4e0d\u80fd\u53ea\u7528 sin \u6216 cos\uff1f<\/h4>\n
\nclass PositionalEncoding(nn.Module):\n def __init__(self, num_hiddens, dropout, max_len=1000):\n super(PositionalEncoding, self).__init__()\n self.dropout = nn.Dropout(dropout)\n\n self.P = torch.zeros((1, max_len, num_hiddens))\n X = torch.arange(0, max_len, dtype=torch.float32).reshape(-1, 1)\n X = X \/ torch.pow(10_000, torch.arange(0, num_hiddens, 2, dtype=torch.float32) \/ num_hiddens)\n self.P[:, :, 0::2] = torch.sin(X)\n self.P[:, :, 1::2] = torch.cos(X) \n\n def forward(self, X):\n X = X + self.P[:, :X.shape[1], :].to(X.device)\n # if using learned embeddings:\n # X = X + self.positional_embeddings[:, :X.shape[1]] # [bs, seq_len, embed_dim]\n return self.dropout(X)<\/code><\/pre>\nimport matplotlib.pyplot as plt\nimport numpy as np\npe = PositionalEncoding(num_hiddens=20, dropout=0)\npe.eval()\nY = pe(torch.zeros((1, 100, 20))).data.cpu().numpy() # 1 example, 100 words with embedding dim of 20\nfig = plt.figure(figsize=(8, 4))\nax = fig.add_subplot(111)\nfor p in [4, 5, 6, 7]:\n ax.plot(np.arange(100), Y[0, :, p].T, label=f'dim {p}')\nax.legend()\nax.grid()<\/code><\/pre>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918151154269.png\")
\n<\/p>\n# Embeddings class: sequences -> features\n\nclass Embeddings(nn.Module):\n def __init__(self, d_model, vocab_size, max_position_embeddings, dropout=0):\n super().__init__()\n self.dropout = dropout\n self.word_embeddings = nn.Embedding(vocab_size, d_model, padding_idx=1)\n self.position_embeddings = PositionalEncoding(num_hiddens=d_model, dropout=self.dropout,\n max_len=max_position_embeddings)\n self.LayerNorm = nn.LayerNorm(d_model, eps=1e-12)\n self.d_model = d_model\n\n def forward(self, input_ids):\n seq_length = input_ids.size(1)\n\n # Get word embeddings for each input id\n word_embeddings = self.word_embeddings(input_ids) # (bs, max_seq_length, dim)\n # Get position embeddings for the word embeddings and add them \n embeddings = self.position_embeddings(word_embeddings) # (bs, max_seq_length, dim)\n\n # Layer norm \n embeddings = self.LayerNorm(embeddings) # (bs, max_seq_length, dim)\n return embeddings<\/code><\/pre>\n# Transformer encoder\nclass EncoderLayer(nn.Module):\n def __init__(self, d_model, num_heads, hidden_dim_mult=4, dropout=0.1):\n super().__init__()\n\n self.dropout = dropout\n self.mha = MultiHeadAttention(d_model, num_heads, dropout=dropout)\n self.ffn = FFN(d_model, hidden_dim_mult, dropout)\n\n self.layernorm1 = nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n self.layernorm2 = nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n\n def forward(self, x):\n\n # Multi-head attention \n attn_output, _ = self.mha(x, x, x) # (batch_size, input_seq_len, d_model)\n\n # Layer norm after adding the residual connection \n out1 = self.layernorm1(x + attn_output) # (batch_size, input_seq_len, d_model)\n\n # Feed forward \n ffn_output = self.ffn(out1) # (batch_size, input_seq_len, d_model)\n\n # Second layer norm after adding residual connection \n out2 = self.layernorm2(out1 + ffn_output) # (batch_size, input_seq_len, d_model)\n\n return out2\n\nclass TransformerEncoder(nn.Module):\n def __init__(self, num_layers, d_model, num_heads, ff_hidden_dim_mult, input_vocab_size,\n maximum_position_encoding, dropout=0.1):\n super().__init__()\n\n self.d_model = d_model\n self.num_layers = num_layers\n self.dropout = dropout\n\n self.embedding = Embeddings(d_model, input_vocab_size, maximum_position_encoding, dropout)\n\n self.enc_layers = nn.ModuleList()\n for _ in range(num_layers):\n self.enc_layers.append(EncoderLayer(d_model, num_heads, ff_hidden_dim_mult, self.dropout))\n\n def forward(self, x):\n x = self.embedding(x) # Transform to (batch_size, input_seq_length, d_model)\n\n for i in range(self.num_layers):\n x = self.enc_layers[i](x)\n\n return x # (batch_size, input_seq_len, d_model)<\/code><\/pre>\n# Transormer classifier for sentiment analysis\nclass TransformerClassifier(nn.Module):\n def __init__(self, num_layers, d_model, num_heads, ff_hidden_dim_mult, input_vocab_size, num_answers):\n super().__init__()\n\n self.encoder = TransformerEncoder(num_layers, d_model, num_heads, ff_hidden_dim_mult, input_vocab_size,\n maximum_position_encoding=10000)\n self.dense = nn.Linear(d_model, num_answers)\n\n def forward(self, x):\n x = self.encoder(x) # [batch_size, seq_len, d_model]\n # pooling\n x, _ = torch.max(x, dim=1) # [batch_size, d_model], can also use torch.mean(dim=1) or just x[:, -1]\n x = self.dense(x) # [batch_size, num_answers]\n return x<\/code><\/pre>\nimport pandas as pd\nimport torch \n\n# \u52a0\u8f7dIMDB\u6570\u636e\u96c6\u7684CSV\u6587\u4ef6\ncsv_path = '.\/datasets\/imdb\/IMDB Dataset.csv' \n\n# \u8bfb\u53d6CSV\u6587\u4ef6\nds = pd.read_csv(csv_path)\n\n# \u6253\u5370\u6570\u636e\u96c6\u4fe1\u606f\nprint(f"Dataset size: {len(ds)}")<\/code><\/pre>\nDataset size: 50000<\/code><\/pre>\n# \u521b\u5efa\u8fed\u4ee3\u5668\ndata_iter = iter(ds.itertuples(index=False, name=None))\nlabels = [sentiment for review, sentiment in data_iter] # Extracting sentiments from the second column\nnum_data = len(labels)\nclasses = set(labels)\nnum_class = len(classes)\nprint(f'Total examples: {num_data}, Classes: {classes}')\n\n# \u6253\u5370\u4e00\u4e2a\u793a\u4f8b\ndata_iter = iter(ds.itertuples(index=False, name=None))\nprint("Example data entry:", next(data_iter)) # [review, sentiment]<\/code><\/pre>\nTotal examples: 50000, Classes: {'positive', 'negative'}\nExample data entry: (\"One of the other reviewers has mentioned that after watching just 1 Oz episode you'll be hooked. They are right, as this is exactly what happened with me.
The first thing that struck me about Oz was its brutality and unflinching scenes of violence, which set in right from the word GO. Trust me, this is not a show for the faint hearted or timid. This show pulls no punches with regards to drugs, sex or violence. Its is hardcore, in the classic use of the word.
It is called OZ as that is the nickname given to the Oswald Maximum Security State Penitentary. It focuses mainly on Emerald City, an experimental section of the prison where all the cells have glass fronts and face inwards, so privacy is not high on the agenda. Em City is home to many..Aryans, Muslims, gangstas, Latinos, Christians, Italians, Irish and more....so scuffles, death stares, dodgy dealings and shady agreements are never far away.
I would say the main appeal of the show is due to the fact that it goes where other shows wouldn't dare. Forget pretty pictures painted for mainstream audiences, forget charm, forget romance...OZ doesn't mess around. The first episode I ever saw struck me as so nasty it was surreal, I couldn't say I was ready for it, but as I watched more, I developed a taste for Oz, and got accustomed to the high levels of graphic violence. Not just violence, but injustice (crooked guards who'll be sold out for a nickel, inmates who'll kill on order and get away with it, well mannered, middle class inmates being turned into prison bitches due to their lack of street skills or prison experience) Watching Oz, you may become comfortable with what is uncomfortable viewing....thats if you can get in touch with your darker side.\", 'positive')<\/code><\/pre>\nimport pandas as pd\nimport torch\nfrom torchtext.vocab import build_vocab_from_iterator\nfrom torchtext.data.utils import get_tokenizer\nimport torchtext \n\n# \u521d\u59cb\u5316\u5206\u8bcd\u5668\ntokenizer = get_tokenizer("basic_english")\nprint("Tokenizer initialized.")\n\ndef yield_tokens(data_iter):\n for review, sentiment in data_iter:\n yield tokenizer(review)\n\nprint("Tokenized example:", list(yield_tokens([('This is an example review.', 'positive')])))\n\n# \u6784\u5efa\u8bcd\u6c47\u8868\nmax_vocab_size = 55_000\nvocab = build_vocab_from_iterator(yield_tokens(iter(ds.itertuples(index=False, name=None))), specials=["<unk>", "<pad>"], max_tokens=max_vocab_size)\nvocab.set_default_index(vocab["<unk>"])\nprint("Vocabulary built successfully.")\n\n# \u67e5\u770b\u8bcd\u6c47\u8868\u793a\u4f8b\nprint("Vocabulary example:", vocab(['here', 'is', 'an', 'example']))\n<\/code><\/pre>\nTokenizer initialized.\nTokenized example: [['this', 'is', 'an', 'example', 'review', '.']]\nVocabulary built successfully.\nVocabulary example: [136, 10, 41, 472]<\/code><\/pre>\ndevice = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")\nvocab_size = len(vocab)\nmodel = TransformerClassifier(num_layers=1, d_model=32, num_heads=2, \n ff_hidden_dim_mult=4, input_vocab_size=vocab_size, num_answers=2)\nmodel.to(device)<\/code><\/pre>\nTransformerClassifier(\n (encoder): TransformerEncoder(\n (embedding): Embeddings(\n (word_embeddings): Embedding(55000, 32, padding_idx=1)\n (position_embeddings): PositionalEncoding(\n (dropout): Dropout(p=0.1, inplace=False)\n )\n (LayerNorm): LayerNorm((32,), eps=1e-12, elementwise_affine=True)\n )\n (enc_layers): ModuleList(\n (0): EncoderLayer(\n (mha): MultiHeadAttention(\n (W_q): Linear(in_features=32, out_features=32, bias=False)\n (W_k): Linear(in_features=32, out_features=32, bias=False)\n (W_v): Linear(in_features=32, out_features=32, bias=False)\n (W_h): Linear(in_features=32, out_features=32, bias=True)\n (dropout): Dropout(p=0.1, inplace=False)\n )\n (ffn): FFN(\n (fc_1): Linear(in_features=32, out_features=128, bias=True)\n (act): ReLU(inplace=True)\n (proj): Linear(in_features=128, out_features=32, bias=True)\n (dropout): Dropout(p=0.1, inplace=False)\n )\n (layernorm1): LayerNorm((32,), eps=1e-06, elementwise_affine=True)\n (layernorm2): LayerNorm((32,), eps=1e-06, elementwise_affine=True)\n )\n )\n )\n (dense): Linear(in_features=32, out_features=2, bias=True)\n)<\/code><\/pre>\n# collate_fn processes sample from DataLoader according to the data processing pipelines declared previously. \n# label is a tensor saving the labels of individual text entries.\nlabel_pipeline = lambda x: 1 if x == "positive" else 0\ntext_pipeline = lambda x: vocab(tokenizer(x))\n\nmax_seq_len = 200\n\ndef collate_batch(batch):\n label_list, text_list = [], []\n for _text, _label in batch:\n label_list.append(label_pipeline(_label))\n processed_text = torch.tensor(text_pipeline(_text)[:max_seq_len], dtype=torch.int64)\n if processed_text.shape[0] < max_seq_len:\n pad = vocab(['<pad>'])[0] * torch.ones(max_seq_len - len(processed_text), dtype=torch.int64, device=processed_text.device)\n processed_text = torch.cat([processed_text, pad])\n text_list.append(processed_text)\n label_list = torch.tensor(label_list, dtype=torch.int64)\n text_list = torch.stack(text_list, dim=0)\n return label_list.to(device), text_list.to(device)\n<\/code><\/pre>\nimport time \n\ndef train(dataloader):\n model.train()\n total_acc, total_count = 0, 0\n log_interval = 500\n start_time = time.time()\n\n for idx, (label, text) in enumerate(dataloader):\n predicted_label = model(text)\n loss = criterion(predicted_label, label)\n optimizer.zero_grad()\n loss.backward()\n optimizer.step()\n total_acc += (predicted_label.argmax(1) == label).sum().item()\n total_count += label.size(0)\n if idx % log_interval == 0 and idx > 0:\n elapsed = time.time() - start_time\n print(f"| epoch {epoch:3d} | {idx:5d}\/{len(dataloader):5d} batches| accuracy {total_acc \/ total_count:8.3f}")\n total_acc, total_count = 0, 0\n start_time = time.time()\n\ndef evaluate(dataloader):\n model.eval()\n total_acc, total_count = 0, 0\n\n with torch.no_grad():\n for idx, (label, text) in enumerate(dataloader):\n predicted_label = model(text)\n total_acc += (predicted_label.argmax(1) == label).sum().item()\n total_count += label.size(0)\n return total_acc \/ total_count\n<\/code><\/pre>\n# hyper-parameters\nbatch_size = 128\nepochs = 10\nlr = 1e-3\noptimizer = torch.optim.AdamW(model.parameters(), lr=lr)\ncriterion = torch.nn.CrossEntropyLoss()<\/code><\/pre>\nfrom torch.utils.data import DataLoader, random_split\n# Split dataset into train, validation, and test sets\ntrain_ratio = 0.8\nvalid_ratio = 0.1\ntest_ratio = 0.1\n\ntrain_size = int(train_ratio * num_data)\nvalid_size = int(valid_ratio * num_data)\ntest_size = num_data - train_size - valid_size\n\ntrain_dataset, valid_dataset, test_dataset = random_split(ds.to_numpy(), [train_size, valid_size, test_size])\n\n# Create DataLoader instances\nbatch_size = 64\ntrain_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, collate_fn=collate_batch)\nvalid_dataloader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False, collate_fn=collate_batch)\ntest_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False, collate_fn=collate_batch)<\/code><\/pre>\n\n# train loop\ntotal_accu = None\nfor epoch in range(epochs):\n epoch_start_time = time.time()\n train(train_dataloader)\n accu_val = evaluate(valid_dataloader)\n total_accu = accu_val\n print("-" * 59)\n print(f"| end of epoch {epoch:3d} | time: {time.time() - epoch_start_time:5.2f}s | valid accuracy {accu_val:8.3f}")\n print("-" * 59)<\/code><\/pre>\n| epoch 0 | 500\/ 625 batches| accuracy 0.615\n-----------------------------------------------------------\n| end of epoch 0 | time: 13.31s | valid accuracy 0.728\n-----------------------------------------------------------\n| epoch 1 | 500\/ 625 batches| accuracy 0.765\n-----------------------------------------------------------\n| end of epoch 1 | time: 12.07s | valid accuracy 0.801\n-----------------------------------------------------------\n| epoch 2 | 500\/ 625 batches| accuracy 0.825\n-----------------------------------------------------------\n| end of epoch 2 | time: 12.19s | valid accuracy 0.827\n-----------------------------------------------------------\n| epoch 3 | 500\/ 625 batches| accuracy 0.854\n-----------------------------------------------------------\n| end of epoch 3 | time: 11.99s | valid accuracy 0.838\n-----------------------------------------------------------\n| epoch 4 | 500\/ 625 batches| accuracy 0.872\n-----------------------------------------------------------\n| end of epoch 4 | time: 11.99s | valid accuracy 0.851\n-----------------------------------------------------------\n| epoch 5 | 500\/ 625 batches| accuracy 0.887\n-----------------------------------------------------------\n| end of epoch 5 | time: 12.10s | valid accuracy 0.857\n-----------------------------------------------------------\n| epoch 6 | 500\/ 625 batches| accuracy 0.900\n-----------------------------------------------------------\n| end of epoch 6 | time: 12.10s | valid accuracy 0.854\n-----------------------------------------------------------\n| epoch 7 | 500\/ 625 batches| accuracy 0.910\n-----------------------------------------------------------\n| end of epoch 7 | time: 12.12s | valid accuracy 0.857\n-----------------------------------------------------------\n| epoch 8 | 500\/ 625 batches| accuracy 0.921\n-----------------------------------------------------------\n| end of epoch 8 | time: 12.14s | valid accuracy 0.854\n-----------------------------------------------------------\n| epoch 9 | 500\/ 625 batches| accuracy 0.929\n-----------------------------------------------------------\n| end of epoch 9 | time: 12.46s | valid accuracy 0.864\n-----------------------------------------------------------<\/code><\/pre>\n# evaluation on test set\naccu_test = evaluate(test_dataloader)\nprint(f"test accuracy {accu_test:8.3f}")<\/code><\/pre>\ntest accuracy 0.867<\/code><\/pre>\nTransformer's Decoder Module<\/h4>\n
\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918151318423.png\")
\n<\/p>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918151921196.png\")
\n<\/p>\n![](\"https:\/\/img.icons8.com\/plasticine\/100\/000000\/teacher.png\")
Teacher Forcing<\/h3>\n
\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918152016113.gif\")
\n<\/p>\n\n
\n\u4f8b\u5982\uff0c90% \u7684\u65f6\u95f4\u6b65\u4f7f\u7528\u6b63\u786e\u7684\u5355\u8bcd\uff0c10% \u7684\u65f6\u95f4\u6b65\u4f7f\u7528\u6a21\u578b\u751f\u6210\u7684\u5355\u8bcd\u3002<\/p>\n\n
\n\u4f8b\u5982\uff0c50% \u7684\u65f6\u95f4\u6b65\u4f7f\u7528\u6b63\u786e\u7684\u5355\u8bcd\uff0c50% \u7684\u65f6\u95f4\u6b65\u4f7f\u7528\u6a21\u578b\u751f\u6210\u7684\u5355\u8bcd\u3002<\/p>\n\n
\n\u4f8b\u5982\uff0c10% \u7684\u65f6\u95f4\u6b65\u4f7f\u7528\u6b63\u786e\u7684\u5355\u8bcd\uff0c90% \u7684\u65f6\u95f4\u6b65\u4f7f\u7528\u6a21\u578b\u751f\u6210\u7684\u5355\u8bcd\u3002
\n\u901a\u8fc7\u8fd9\u79cd\u65b9\u5f0f\uff0c\u6a21\u578b\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u65e2\u80fd\u5feb\u901f\u5b66\u4e60\u5230\u6b63\u786e\u7684\u6a21\u5f0f\uff0c\u53c8\u80fd\u9010\u6e10\u9002\u5e94\u5b9e\u9645\u63a8\u7406\u573a\u666f\uff0c\u4ece\u800c\u63d0\u9ad8\u6a21\u578b\u7684\u6cdb\u5316\u80fd\u529b\u548c\u5b9e\u9645\u5e94\u7528\u6548\u679c\u3002<\/p>\nTransformer Architecture Summary<\/h4>\n
\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918152110311.png\")
\n<\/p>\n![](\"https:\/\/img.icons8.com\/cotton\/64\/000000\/torch.png\")
Native Transformer in PyTorch<\/h3>\n
\n\n
torch.nn.Transformer(d_model=512, nhead=8, num_encoder_layers=6, num_decoder_layers=6, dim_feedforward=2048, dropout=0.1, activation='relu', custom_encoder=None, custom_decoder=None, layer_norm_eps=1e-05, batch_first=False, norm_first=False, device=None, dtype=None)<\/code><\/li>\n![](\"https:\/\/img.icons8.com\/emoji\/96\/000000\/woman-lifting-weights.png\")
Pretrained Models - BERT and GPT<\/h2>\n
\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918152408388.gif\")
\n<\/p>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918152458350.png\")
\n<\/p>\n![](\"https:\/\/img.icons8.com\/bubbles\/50\/null\/picture.png\")
Vision Transformer (ViT)<\/h2>\n
\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918152543565.gif\")
\n<\/p>\n# code skeleton from: https:\/\/lightning.ai\/docs\/pytorch\/latest\/notebooks\/course_UvA-DL\/11-vision-transformer.html\n\ndef img_to_patch(x, patch_size, flatten_channels=True):\n """\n Inputs:\n x - Tensor representing the image of shape [B, C, H, W]\n patch_size - Number of pixels per dimension of the patches (integer)\n flatten_channels - If True, the patches will be returned in a flattened format\n as a feature vector instead of a image grid.\n """\n B, C, H, W = x.shape\n x = x.reshape(B, C, H \/\/ patch_size, patch_size, W \/\/ patch_size, patch_size)\n x = x.permute(0, 2, 4, 1, 3, 5) # [B, H', W', C, p_H, p_W]\n x = x.flatten(1, 2) # [B, H'*W', C, p_H, p_W]\n if flatten_channels:\n x = x.flatten(2, 4) # [B, H'*W', C*p_H*p_W]\n return x\n\nclass VisionTransformer(nn.Module):\n def __init__(\n self,\n embed_dim,\n hidden_dim,\n num_channels,\n num_heads,\n num_layers,\n num_classes,\n patch_size,\n num_patches,\n dropout=0.0,\n ):\n """\n Inputs:\n embed_dim - Dimensionality of the input feature vectors to the Transformer\n hidden_dim - Dimensionality of the hidden layer in the feed-forward networks\n within the Transformer\n num_channels - Number of channels of the input (3 for RGB)\n num_heads - Number of heads to use in the Multi-Head Attention block\n num_layers - Number of layers to use in the Transformer\n num_classes - Number of classes to predict\n patch_size - Number of pixels that the patches have per dimension\n num_patches - Maximum number of patches an image can have\n dropout - Amount of dropout to apply in the feed-forward network and\n on the input encoding\n """\n super().__init__()\n\n self.patch_size = patch_size\n\n # Layers\/Networks\n self.input_layer = nn.Linear(num_channels * (patch_size**2), embed_dim)\n self.transformer = nn.Sequential(\n *(AttentionBlock(embed_dim, hidden_dim, num_heads, dropout=dropout) for _ in range(num_layers))\n )\n self.mlp_head = nn.Sequential(nn.LayerNorm(embed_dim), nn.Linear(embed_dim, num_classes))\n self.dropout = nn.Dropout(dropout)\n\n # Parameters\/Embeddings\n self.cls_token = nn.Parameter(torch.randn(1, 1, embed_dim))\n self.pos_embedding = nn.Parameter(torch.randn(1, 1 + num_patches, embed_dim))\n\n def forward(self, x):\n # Preprocess input\n x = img_to_patch(x, self.patch_size)\n B, T, _ = x.shape\n x = self.input_layer(x)\n\n # Add CLS token and positional encoding\n cls_token = self.cls_token.repeat(B, 1, 1)\n x = torch.cat([cls_token, x], dim=1)\n x = x + self.pos_embedding[:, : T + 1]\n\n # Apply Transforrmer\n x = self.dropout(x)\n x = x.transpose(0, 1)\n x = self.transformer(x)\n\n # Perform classification prediction\n cls = x[0]\n out = self.mlp_head(cls)\n return out<\/code><\/pre>\n Swin Transformer<\/h2>\n
\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918152638737.png\")
\n<\/p>\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918152722956.png\")
\n<\/p>\n<\/li>\n<\/ul>\n![](\"https:\/\/img.icons8.com\/external-others-maxicons\/62\/null\/external-magic-medieval-others-maxicons.png\")
How to Tame Your Transformer?<\/h2>\n
\n\n
![](\"https:\/\/img.icons8.com\/clouds\/100\/rocket.png\")
Initialization<\/h3>\n
\n\n
\n
\n
# GPT initialization example\n# https:\/\/github.com\/karpathy\/minGPT\/blob\/master\/mingpt\/model.py\ndef _init_weights(self, module):\n if isinstance(module, nn.Linear):\n torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)\n if module.bias is not None:\n torch.nn.init.zeros_(module.bias)\n elif isinstance(module, nn.Embedding):\n torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)\n elif isinstance(module, nn.LayerNorm):\n torch.nn.init.zeros_(module.bias)\n torch.nn.init.ones_(module.weight)<\/code><\/pre>\n![](\"https:\/\/img.icons8.com\/external-flaticons-lineal-color-flat-icons\/64\/null\/external-workout-running-flaticons-lineal-color-flat-icons-4.png\"){kind=icon}
Learning Rate Warm-Up<\/h3>\n
\n\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918152824502.png\")
\n<\/p>\n\n
import numpy as np\nimport matplotlib.pyplot as plt\n\nclass CosineWarmupScheduler:\n def __init__(self, optimizer, warmup, max_iters):\n self.optimizer = optimizer\n self.warmup = warmup\n self.max_num_iters = max_iters\n\n def get_lr_factor(self, epoch):\n lr_factor = 0.5 * (1 + np.cos(np.pi * epoch \/ self.max_num_iters))\n if epoch <= self.warmup:\n lr_factor *= epoch * 1.0 \/ self.warmup\n return lr_factor\n\n def get_lr(self):\n lr_factor = self.get_lr_factor(epoch=self.last_epoch)\n return [base_lr * lr_factor for base_lr in self.base_lrs]\n\n# Parameters for visualization\nwarmup = 300 # Example warmup period\nmax_iter = 5000 # Example total iterations\n\n# Initialize scheduler\nscheduler = CosineWarmupScheduler(optimizer=None, warmup=warmup, max_iters=max_iter)\n\n# Generate learning rate factors for each epoch\nepochs = np.arange(max_iter)\nlr_factors = [scheduler.get_lr_factor(epoch) for epoch in epochs]\n\n# Plot the learning rate schedule\nplt.figure(figsize=(10, 6))\nplt.plot(epochs, lr_factors, label='Learning Rate Factor')\nplt.axvline(x=warmup, color='r', linestyle='--', label='End of Warmup')\nplt.title('Cosine Warmup Scheduler')\nplt.xlabel('Iteration')\nplt.ylabel('Learning Rate Factor')\nplt.legend()\nplt.grid(True)\nplt.show()\n<\/code><\/pre>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918153113291.png\")
\n<\/p>\n![](\"https:\/\/img.icons8.com\/external-flaticons-lineal-color-flat-icons\/64\/external-activation-media-agency-flaticons-lineal-color-flat-icons.png\"){kind=icon}
GLU Variants Activations<\/h3>\n
\n
![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918153003956.png\")
\n<\/p>\n<\/li>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918153046364.png\")
\n<\/p>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918153157728.png\")
\n<\/p>\n![](\"https:\/\/img.icons8.com\/nolan\/64\/replace.png\")
Alternatives to (Post) Layer Normalization<\/h3>\n![](\"https:\/\/gnnclub-1311496010.cos.ap-beijing.myqcloud.com\/wp-content\/uploads\/2024\/09\/20240918153241539.png\")
\n<\/p>\n