【循環(huán)神經(jīng)網(wǎng)絡(luò)簡介1500字】

圖9所示：圖2.SEQ圖2.\*ARABIC8LSTM模型圖Fig2.SEQFig2.\*ARABIC8StructurediagramofRNNmodelLSTM的計算公式如下：(SEQ____________\*ARABIC2.13)(SEQ____________\*ARABIC2.14)(SEQ____________\*ARABIC2.15)(SEQ____________\*ARABIC2.16)(SEQ____________\*ARABIC2.17)(SEQ____________\*ARABIC2.18)下面本文詳細介紹LSTM增加的三個門控單元：(1).遺忘門LSTM中的第一步是決定從細胞狀態(tài)中丟棄什么信息。這個決定通過一個稱為忘記門的門控單元完成。該門會讀取?t?1和xt，輸出一個在0到1之間的數(shù)值給每個在細胞狀態(tài)(SEQ____________\*ARABIC2.19)其中?t?1表示的是上一個神經(jīng)元的輸出，xt(2).輸入門下一步是決定讓多少新的信息加入到神經(jīng)元的狀態(tài)中來。實現(xiàn)這個需要包括兩個步驟：首先，一個叫做“輸入門控單元”的sigmoid層決定哪些信息需要更新；一個tanh層生成一個向量，也就是備選的用來更新的內(nèi)容Ct(SEQ____________\*ARABIC2.20)(SEQ____________\*ARABIC2.21)(3).輸出門如下圖所示，輸出門根據(jù)細胞狀態(tài)決定輸出的值。sigmoid層來確定細胞狀態(tài)的哪個部分將輸出出去，接著細胞狀態(tài)通過tanh函數(shù)處理，得到一個（-1，1）之間的值，并將它和sigmoid門的輸出相乘，得到神經(jīng)網(wǎng)絡(luò)隱藏層的輸出結(jié)果。(SEQ____________\*ARABIC2.22)(SEQ____________\*ARABIC2.23)1.3GRU模型由于標準的RNN網(wǎng)絡(luò)在訓練的過程中容易出現(xiàn)梯度消失和梯度爆炸等問題，導致標準的RNN模型缺乏捕獲用戶簽到的長序列的依賴性。為了彌補RNN的缺陷，一些改進的RNN網(wǎng)絡(luò)被提出，如LSTMADDINCSL_CITATION{"citationItems":[{"id":"ITEM-1","itemData":{"DOI":"10.1162/neco.19735","ISSN":"08997667","PMID":"9377276","abstract":"Learningtostoreinformationoverextendedtimeintervalsbyrecurrentbackpropagationtakesaverylongtime,mostlybecauseofinsufficient,decayingerrorbackflow.WebrieflyreviewHochreiter's(1991)analysisofthisproblem,thenaddressitbyintroducinganovel,efficient,gradient-basedmethodcalledlongshort-termmemory(LSTM).Truncatingthegradientwherethisdoesnotdoharm,LSTMcanlearntobridgeminimaltimelagsinexcessof1000discrete-timestepsbyenforcingconstanterrorflowthroughconstanterrorcarouselswithinspecialunits.Multiplicativegateunitslearntoopenandcloseaccesstotheconstanterrorflow.LSTMislocalinspaceandtime;itscomputationalcomplexitypertimestepandweightisO(1).Ourexperimentswithartificialdatainvolvelocal,distributed,real-valued,andnoisypatternrepresentations.Incomparisonswithreal-timerecurrentlearning,backpropagationthroughtime,recurrentcascadecorrelation,Elmannets,andneuralsequencechunking,LSTMleadstomanymoresuccessfulruns,andlearnsmuchfaster.LSTMalsosolvescomplex,artificiallong-time-lagtasksthathaveneverbeensolvedbypreviousrecurrentnetworkalgorithms.","author":[{"dropping-particle":"","family":"Hochreiter","given":"Sepp","non-dropping-particle":"","parse-names":false,"suffix":""},{"dropping-particle":"","family":"Schmidhuber","given":"Jürgen","non-dropping-particle":"","parse-names":false,"suffix":""}],"container-title":"NeuralComputation","id":"ITEM-1","issue":"8","issued":{"date-parts":[["1997"]]},"page":"1735-1780","title":"LongShort-TermMemory","type":"article-journal","volume":"9"},"uris":["/documents/?uuid=8dadfe80-bc95-4c9b-ae91-0b2ec69b7bf2"]}],"mendeley":{"formattedCitation":"^[48]","plainTextFormattedCitation":"[48]","previouslyFormattedCitation":"^[47]"},"properties":{"noteIndex":0},"schema":"/citation-style-language/schema/raw/master/csl-citation.json"}[48]和GRUADDINCSL_CITATION{"citationItems":[{"id":"ITEM-1","itemData":{"ISBN":"9781510810587","abstract":"Inthiswork,weproposeanovelrecurrentneuralnetwork(RNN)architecture.TheproposedRNN,gated-feedbackRNN(GF-RNN),extendstheexistingapproachofstackingmultiplerecurrentlayersbyallowingandcontrollingsignalsflowingfromupperrecurrentlayerstolowerlayersusingaglobalgatingunitforeachpairoflayers.Therecurrentsignalsexchangedbetweenlayersaregatedadaptivelybasedontheprevioushiddenstatesandthecurrentinput.WeevaluatedtheproposedGF-RNNwithdifferenttypesofrecurrentunits,suchastanh,longshort-termmemoryandgatedrecurrentunits,onthetasksofcharacter-levellanguagemodelingandPythonprogramevaluation.OurempiricalevaluationofdifferentRNNunits,revealedthatinbothtasks,theGF-RNNoutperformstheconventionalapproachestobuilddeepstackedRNNs.WesuggestthattheimprovementarisesbecausetheGF-RNNcanadaptivelyassigndifferentlayerstodifferenttimescalesandlayer-to-layerinteractions(includingthetop-downoneswhicharenotusuallypresentinastackedRNN)bylearningtogatetheseinteractions.","author":[{"dropping-particle":"","family":"Chung","given":"Junyoung","non-dropping-particle":"","parse-names":false,"suffix":""},{"dropping-particle":"","family":"Gulcehre","given":"Caglar","non-dropping-particle":"","parse-names":false,"suffix":""},{"dropping-particle":"","family":"Cho","given":"Kyunghyun","non-dropping-particle":"","parse-names":false,"suffix":""},{"dropping-particle":"","family":"Bengio","given":"Yoshua","non-dropping-particle":"","parse-names":false,"suffix":""}],"container-title":"32ndInternationalConferenceonMachineLearning,ICML2015","id":"ITEM-1","issued":{"date-parts":[["2015"]]},"note":"GRU","page":"2067-2075","title":"Gatedfeedbackrecurrentneuralnetworks","type":"paper-conference","volume":"3"},"uris":["/documents/?uuid=1ef2b1ec-8119-4445-815a-6071c616e576"]}],"mendeley":{"formattedCitation":"^[49]","plainTextFormattedCitation":"[49]","previouslyFormattedCitation":"^[48]"},"properties":{"noteIndex":0},"schema":"/citation-style-language/schema/raw/ma