From eda301c2f690471163b9eef79970b0eeaaa6e251 Mon Sep 17 00:00:00 2001 From: Arthur 'Grizzly' Grisel-Davy Date: Thu, 19 May 2022 11:10:27 -0400 Subject: [PATCH] feat: add section about perceptron and add all images and sources --- .gitignore | 3 + images/2neurones.pdf | Bin 0 -> 4161 bytes images/2neurones.svg | 250 +++++++++++++++++++++ images/logreg_illustration.pdf | Bin 0 -> 2118 bytes images/logreg_illustration.svg | 151 +++++++++++++ images/neuron_model.pdf | Bin 0 -> 7514 bytes images/neuron_model.svg | 239 ++++++++++++++++++++ images/percepton_problem.pdf | Bin 0 -> 5539 bytes images/percepton_problem.svg | 392 +++++++++++++++++++++++++++++++++ images/slr_illustration.pdf | Bin 0 -> 2071 bytes images/slr_illustration.svg | 133 +++++++++++ main.tex | 124 ++++++++++- 12 files changed, 1291 insertions(+), 1 deletion(-) create mode 100644 images/2neurones.pdf create mode 100644 images/2neurones.svg create mode 100644 images/logreg_illustration.pdf create mode 100644 images/logreg_illustration.svg create mode 100644 images/neuron_model.pdf create mode 100644 images/neuron_model.svg create mode 100644 images/percepton_problem.pdf create mode 100644 images/percepton_problem.svg create mode 100644 images/slr_illustration.pdf create mode 100644 images/slr_illustration.svg diff --git a/.gitignore b/.gitignore index 7270eda..4880327 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,6 @@ # vim swap file *.swp + +#include images: +!images/* diff --git a/images/2neurones.pdf b/images/2neurones.pdf new file mode 100644 index 0000000000000000000000000000000000000000..f2ac15511347047a174881013cc46f087b50ab20 GIT binary patch literal 4161 zcma)=cT`hJ-+)niK472@oKz^w0$fp`#Qj(v>O!>0Oslr1z?b z^s-Wg&^sUcxqJ4UJ@0$od*;l{Z)WcA%shAgc`l!Uy2fqD9Z3LR^+w4FKmrH`I$J*m zC@26yI!Gsbv;$D`atHKqJ+WHqN$4fRYlx9qoohI0C%W^DMMu z*zeQfzm*n<4>KMG{8kU-ay6IXzFsi6?jY%IF0(!(fCq6`7QVL}hg$Y(&fRa*fUUbpjk!ZK{d1~8Ai(aRnxDe?Y0gT z_xyXO*f?cs4%brG@iE;5;T-*8rnfh-l!@rdw7N?AGkGMbQSOPn1zZ?C$6sj>DYZrkC0Cw_|aRfsbspjrcpqp%tJZM0`;vr>0|=w6YV=R4+!t(35GIqqVr}Sz%$X z?&wq3=MT#hu$UQL`BIv5_@IB`-d?2j`uMPJIfZ49a%rKir`b}!dtqjdG@jdMZ1ZJm z=IHbK_$>9qSlZKxm)?F!qSoO9x^6Q{!@)*(vNLUig(2l{W^X)X%B#xg6j_Ff(quLWH5Z(RS^6o7-l}f6lBLdQ;#G=9Xno+lyTpU=g_#m{j zp;lq>^RuW8!7Gd8Bcb}?uVog??<5Z}@Cdy_=N2+9Em_pTo4!L`eXPx5cj0M!`&2fh zsxPy6Ti8C^MUaQ6OXk|ev^q=#bT)8#9nMd6Uj+>ve^FMYF>q{ zYqVgB`xr@w_#8~CNu(W(6OB!g^uD!1&|CVbxVEQk%3P@c=sBYcC@5jYRlZZPO7ux$ zl6Nyr6yM9=jT(u%nwU5EWC}l znW(Z2B#p!KfGQ{7eNb%+<)ef@R|;ory+ktCCUqMxm(=1JQ8HKwlt`2*;mBZM8}=a= zt;?%2I(GmLLatC|>>&}R{tsA-NaJVUF|}{K1Mv@6;}fP2{v0^2$ZT=A<308&QgWk4M4qP>YzWdaZ*Cg62EKnWMn5d0$rFrq*wt+uUW8UZ; zpOw(Oz8ubAzr?X@bmO}-g;0T#Q-%+XzgT40_`E_~@Wh4|y&u33dDVPg$V#Q_4@2K8 zsIUH>t-H7OtT{U4OiqH4=_{U@E_lIay))HfFBUDv9j@{+#_)UwE_|_ZT+>5tdUx@| zaC2`@%J}5>2!-LqgmtO)et4~>RKil+&WE0P5<>%{KNJ=#lk}sOocLwwkepedBJIs! zh4~VJT+T)-)O@uX+Jf)ErINnepwZ%%gkkXjvDlR*GzRJ~!8r2$f%TR#)?-~2Qt;S4 zWSp0K@evPtffGXks>CZ6(K|b>-xeHc2PBkK(9&0HMs-;S=@@u|uY7vr92iFl9x$`Z$G+#X zE}A5-?{4RoD2})aGjbF%X6=GISu2?+^rnsQARfm0YV0ZwT~t&h_Y%!3Zx{G5Uc2b# zTuh)@B_8_CeU6EO|HflXIBHP| zch|usuf||*ufCH^k>!-Y+hB`uC1w2Wb&49kCL-|56Ehpd1D=+WWsdd<;x2D&QB|k9 zB%SxX?E>!Y#Zk7e+7?zP@1|RmEr}>~7PO(cji1$LvyL0)yd%eAI94DZl#y9EE|4at zBV*(ND4o4Rnli*Vw`|}!{kyfThJ4rWw^G)tzdo$U0OrZ*Jr|O8PFsyXuIO8Isxg|I zY^yRG;VgZ4H}lJ%cuWK7@i};#i7qy*wMx)0u+{J}dGLykZLE4^z$n5XcwO_DO}iJf z1*S+T9!E93=%Jmy#oh^?#_YzXN9v^RNgSW3xt5;-sGo-1bW7W^Ws_u;ZY4fxDN}w> zV6BK&DSlE4&KEl~b#RXwPKkggwHl7m#*$98I#}4jCEnec7-p0$l7a6k`09K1P3Gxr zRXyi=$;@FVs*+_VEIc|DtPj2BT8Nr&FrJ%&M(BiM?Oz;C@N+waoePB*m@KpLDBqOF z34Z2#Ax|c^!ME8SZohV8gZ?0?opM_N^2`f27m}kynhui0Q468wqpaW%n>N0U+c52? z;%q0m&LML!Vu80;!}whW+XIiJopIMwSzVxztK4XqIwoOKsFLB}QxM^1<__LCTqq$KN9*rtXS$MiPRGhav>cbGa7T7Wim9Nk!VJQh!Gg?4I|mMKpI)ZHII#r+ha;8c)Z zxPFFdIH;9vUSGJ=k_G*TVKMG8(=BoSeE3NGSU}3Ak;nBQ#RukEs8r{`8wx?2=~aw~ z-$e7)7(zU9;-%<`jvqIXxqYpJ-%jXYGR(<*HEyrbJW9gq5n8Ms!yjKz2x*IGgx}!m z9KuOlFQHJC;3D6fKi2>klTTno!NQWA!BPfB+G$$GNr1Meg$e9*_EMsJg>b3`=58;` zg~XRtX&NnWB*nD8YSBcwms=65$Kx#%@3K$?%b{ynMc9cD(Z)KmCemrmMU)sJEVDA* zvRe!lSr_gmrSAG9A?V&HZ+~2r+F|w?eF7=>7Z|&Z)NYfhV+%QCDL}t;2U~L-%R;#iou9rwP;n=C1Z%<8};Y1Em|Q&n5mx27Zf*|T7BPv;jJXuVv85Nh{oX;NtBYxw5oH)R^nC z*Vf2Yr-q|G0a{TiCR74w8t2gh97i5DZ{ui!yqR`ELUD+N^+gi#RbX$n*%7oWSa0L& zD(N3es+%{?<|A{JL@{kiI$3ib^B*mX(5Y~(8?Ww}u{w##OIwfXMjD2PI4w%%sq*ZJ z6w707U58KY`tn1uQ)8d^S_eT#0#}k%ZqV2eF_7+AdN8M!6+9jOU5|KX<6`^%w3YTT zCsFJ%G5%v}xXCArQK=(Jgjt=0W~Q7?J%0?5a0{+2f=Gr;wXUQLzBy66dc|;d+_05A zkpE`9y0BU7Nj;@|9v7j+E2R|yq|Ka3Yku>3 z4$#FyVg0Ifs!`;4@gr%rYo@m*U7-f|KIl{FY`^vB44CM*HTw8|qj|fdsKeV!x&DHv zLrgIqfON9`B@marAL{YLH-5C2#vu+7llrmXM=1Wcnf!DSs5kna5&F_YfC>sf1}+~E z0(=<&L3dSw5Fp3~aamv5$>qdleOZ3xk@`Ci5M=D*f&_x}tsfhq9e-y1r99GqD~}!! z@^eB1<>rnCO8n~ocMVePgU|$l8A)a{9=S$|RMiCWUL#JXC1=1owPTv15LvmTJQve7 zWF05l-={c4VSq0G42rI5e>%@ux>;S)1z! z(jlI1>YHx$MeY1+^-vE<-qWq4T2^#Un$bwK=XTH3djebK0|)KI4{}3(%LO>?7%1Jj zwC$d?Tm>y+gup_AlkELB?;nT##Tz0i`xhpe{{a&O@()hp|KJ3X`kz4Q#wh*(1wA;A z{6LA_t)>ePEnBw!?YS#M7}?Dy5(!lvTQgL9JS7JWJJX6+Ta!8?D1&m-O_&Y|vJP7f zt5x_IW6^1Q`j{c5(^y#U_Nv}BF*3i%mEAIGYc zDoeCS4%ptBl=0MTIO9zgzQ*_~FIt2n-g3iNL(T zVB+lgV&Svo<@}iN7?q$HN_EY^1Q#$kh=N}ufaVSBGX-K|qHi3ek^erUI~w7J_IA76 RzPPOT + + + + + + + + + + + + X + + + + + + + + + X + + + + + + + + w1 + w2 + xi + P1 + y + P2 + + Oi + + diff --git a/images/logreg_illustration.pdf b/images/logreg_illustration.pdf new file mode 100644 index 0000000000000000000000000000000000000000..656bb0501ebf58dc0721eb43ab2dbbd4aff03e1c GIT binary patch literal 2118 zcma)-3pA8zAHa#3n9xYRu|#ifOJe2~8kZ4+u`an!6N3y5L#|_3w{@MuVsmRcEgQK+ zEQygpCZ=54P2adAbirm>F-aulw!_BhoX+>1eb0NI>-+xC|M|Vo`+HOz%yF8amM%=C zkypD4(*aOGh<5-uoI*>#oo0EM)&XHlp#;|Z2 zl}rkPMHNO|vH9tR9^7Ku9daUD+jZ?$d!pd}IWeLAVh}34crQ<^{s%U0Pzg&$JE2K`1~w)qM>mcwH}lP6wDQ>m>Z;yZEHp2vJ=a z_kH|If+xw25J2aKz49?V#(*}fIjW14hQu`wu*mb1(T}+WSrVyGwLP(uv?Hu4L(%;_ zk82sOkJ+*JM2gTA)kmk++E4w%Yz z1$_^Z+ShW|m$d%ddCYID?JV8CaG=rMKflFFv925UgeuP&bV>oaaH6$?gQ%RuUuNBw z*!RVpcReFw?VTsEGaj;OTZtCYk?4w9=7`Tt)%11+!(YT95v1j8AfF}KZ*a6`)*wjf zeou?E%0IaxgthL0)|>q%bG1vUs}EQ9m}lP1yUG(dU@GJ6$UYe9{`&ANbWZICXo;y* zb`d&SXKII7b$=-_Fw$gvuAngH`yXy&Ow(#~+gs!@ifJ7Rxcmu@ii-hBYFC#82 zN9Rh*eIp@mFTC6{gAK)x`CW>=GVRT1n16}neEWEYdIPlSQ}XQYE;vC7dRur@tc8v0 zT_>ttb9T@*t~@zvaG{G>;B}EcVoiyA9~OvZ1)r=P2R;3=8|)Y;=Lb*?-#nY=`l`Hr zqccvW)xQ*ePjsbCLG36?Ul7(XrLS6$|wq2Rr$t2sE0 zUu0^A;ckuULXodnlp>0`;aGhaRO7Zxc}AjVoVc>dn`pmsacRZiUzyWRV;60FNcg0z zT9{ll&HGnxvooFLnL`p`ImUH%h<$BiQ$ZdDb#b^D=MD;yVW&rF^3S^J**trPtmLu0 zI6lbFR#J-LSnE|enqIP-?J$Bl(7P}8?2s(7bTMXZdA4|g9YlyZB_DoElk~D8`%Z7w z5h6FrH9t@_`!y$y@}b}4QO#282ho)>eGis#Cp&&g1oNkL#rUSyC;PI^*^N(ALc*I( zzi%`oe3-2cTJL{)sCg#hTF#Pw&MWr?=9(f7KUIEUb)NI_s*_-tr*nTY+jdHMV8-xP zI(C)HR-i^1UBH%F*yNSkb@T3KHoh;~$XNX&UNHD>UmlDceD1SF_^mDfY)NOU=4%H; zp^tApu+=@jtEX)tSwzt+ooRd-0Y*k!1N;j~Lh(BQ(##Y90i+L!U-A{jpWv7L`*R$< z-8cZ!B|4N0Anm;aoM}PZQ9pa{#IE=30C0N(=T8l%0Xm=C|KEj}-ZoeY>L6tNMFrZ1 zkae>J8_G?lsU3V)LxaakB9uO=zvmT{4KKt!58uBxgKk0{JB_8vt8xVWg;$Cx;CCI| zy)5o|1=)`s_a)8nZ&^JJx#4f`{+N*qdQN;}M{JfM4E1dA?5tX`B8THPG@&nMrd-4` zJlEf`cySjU8Owt$)lzNKIiG45&ODt+uy=FQ&FumM4L0^bD7w90^WIwQ4sTuk9Zcwd zfe8e^;B@>8PN3eu0(I(As{{uH7miKnj8uj!j2NZB#dXqDu{0ABMB}ze*tvq_d*6l$ z^u2lUoF@YR9aK^k$@L9vgPv@26ZfpNj)f*3e$@L)Oq3(}R3?bmAPNs`NH?FD#_agol>tK*X&Bd zZ^OP7$+&u=Rjx-w&H%MOe-S)oX+G053N+wNF@e$xYS6xCOEw__{T3r%Q`}~;gAtA1 z!3b)9#qo?2*>@Yy&LNBbpRl$zwX=T=836yGO8|%$fD%dv! z(BJNUc{KZYd97>~r6;8%C8Z~2B_-vM<+HuU_ujsdk``i;oF#AGlbj!lLM4l+;v&x$ k%f!NjgajtLArW7%F`P!C(xRyR^g*<~HcUmu!X5|v8zE$fs{jB1 literal 0 HcmV?d00001 diff --git a/images/logreg_illustration.svg b/images/logreg_illustration.svg new file mode 100644 index 0000000..0c6b364 --- /dev/null +++ b/images/logreg_illustration.svg @@ -0,0 +1,151 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/images/neuron_model.pdf b/images/neuron_model.pdf new file mode 100644 index 0000000000000000000000000000000000000000..60875a04195a88d75b3fa8bfd64b9faee68cc468 GIT binary patch literal 7514 zcma)>Wl&tpy08iE9)biICTJiq!{CI$H3{y)1{etL8r)q&7~E|jID`;1SRexg5(anI z;KBK__qWffQ+4Xzx~san`+a+@x4NITeypm8O@kKchX&9WDpAaI%CGLZO6i?yhi{1EF_ziP5)& zXX2VUn|ga-O@|v#-yv**Xz3nuuoOcM)K-Ut9`^=Xhb9!y`HZx|{=Ku{1{_NC76m>| zsJg&E|E1V?Uy_o`lBT({dggI+iGMRDd3(F+bpP!6-9?_FZ|AC4a-zr&g z_q@Ms(G@bvZaJ_EMD0VXZ_iyQT6&T?1C5S<-EE9)3g5j~oqR`Wmc4nO-buLvndTP{ z^ekt0JQDsmdpX;`e{)+~e38{yq9F=U5LaIgtH&kH&OuG(&YwCpLr~Q-48WO!oKz^Y z)w9h8*Y9#Lbv3I21qTTR@cOiDC+zCP-pBMSu@fRX*Hd9{EpZ`-b*l3Th)>dDz;QX_ zHAl^2*{v9VRl8-v?!&nUT8ajU$Hx%j5Nj@M?vF97Tk@!_OTbWD8Vf2|1A1ft>|PKo zN?(KcVzDT&k}-p#5!eJw#f6hH*sKpk(wfb9%)j-u#)AeZ{G%Hvg+@sCfim$$sC^Q=p6R3?|EDbW3>FsepW6N%N~!9fXM81q2g=e z81d;N2E#g4mjO>o3wmf2rvZ1E5!*b_?sRrF6$sTC(B-MbzT#lQqEfu}7hw#?s6ey5$JsDEccpB6 z=OBxN$hABUp4;!(ym1Dd2I!^`8}Hd}VBo97)h3GM^5a+~@y<=3ItrGu$$l$mQF+96 z9b>}vk+l9rK=B@8`7}()rZL+EMXVJn=Q(2)RDw=L4#K|2m4%u1B3MgG-asj%l7w=m zW=a+st!O}xwdi50AaKrE%aQ`~Q7idiue3NE{~PRf^vIU zd?V9xujd=nwHQ_ae9>>@DguOS>)UfGLJA}tRdcIT<(u2~HKAl^Psojw23WnC zS>H#Iy4kM_h1meoqb%96Z=Y}QR9;v;sEzR>kfWe(_iC}3odA3%}HFn1xu3bGwxLk-9j%`%NCyjR9up4(+TX<=dKpjxdOd05K2 z!IWOB)K@o{>h8GG!M%1?l!C%dxxR*Dasd?Ps2#O0r1NOSBP#4SH<$9g7ZQGMhsvFD zq>p@F4=VkAE1jZjs8~vpIWFgZ*Tj~nD8y7-=$r?|MvO&+>6ql=s)8os;5SK2?1vfZ1D_k?xJBUzvq5V} zbOssF#BD#<06(>KyH#g>DJn?WW43?gXw-|N4@>Ddd>2^n-R4qnG4|lZ5EhL zxa!R3XUJYwpEaz~Q9m9VJu-GUE2SVcSC}`&gi0I9;9x3rj# zqB5KGW1S8@ENfkT>SI-(XT3ou=8QMZv+04+z|kr4QO8S(R&$f3sY+lnIzIrZYQR3M zRZz;d@j5{x07va=P&r=tYeIAdFL85ImkO;7x7qajigD2z*KN=ungGv>L*Rm_s_7~$ zx;uIulkMr!3Z+Mmto)l-T)Av&Ppxu|o7UN~R;_(9^1fLW$bBeAf$G`ShFs5P(>dUD zM!!TeSiW$+R<+W*W~yN&cIN#|G^mU`eaADf{{CI+vtE^mnOTpEP zgqa5ZB4U~L6n;f}H{^A7xjx4`V$1#g)P?1=WMlj(5~k~x4sE@5k&GDZVP{V_{Bd0;A{>Crw<3GI8Ep7aIXk?qb{RwSu zSz}J2Mf8dI5NATq1yk^IokASY2)nSzjh~e|nVAPZBF;V{^n4@zvl6$(d*k`DS z$7b>D=t@Je^j=fK$+R!YoOHQ$Xq_Jtp|pX)Js9;aYfO{;BDTwQ`BV2@w>?{stFK4Ji4ZQoVfwQt!4IlrMn$=K!_fgelQ_DgwU-rrSFYM@ zZ{KA;89&LUd-T()RYj5AU-*VYzahIZsU}MX@{M2avlRLFzT3GOf?|V_flCL?Lqdz z(H`Gyz@hiFBw82v*#>e1CaydJ1lwmZPC(<&T3AXY)Z(IkP^J(=OzdU2TUg<-6TbFe zs!(`*u*Hd^hM2t@-JU+~ZT_YSWXO)TXVZxhoXAQI!{aFG&LovP=8Z~u{X#EETG|mw407+W%+c}EzaZI$q3zbWfTo{_H7KTJ-m>g6o@r=As@W3(u{ZsI5 zh8byXJ2K2y_r>*|lCKF&8_tv#{s}<<@%NFfV(s3TYCrLot+2J4wvSDq3lN2GPrOks z3t@MBm8K~F0%ZC`*qBa;5>(#_|KSuGg6v=Su6L?pmdA z91}M#+qq9INU2jm<552)H>+}{LTNx(RHuJI^bDv_BkjtoRY%6Mu+h0(do1-sMwit3 zrz1?Pu<8uj5b-CrZj8VL{NxrTJhiRIx^;-jGr=!Ezg5)VZi5b(2!Z_~m}gN>g4p-{Q2mo_+bVcvCS6XI!ChDutFba< zM}H}HoY3N0G<&V!p-VWw_W^!%8!xKR_o)pq;67h?OOx;mNNFurkV%1SAm5uRMz`OC z1VqGSFj0`6?&W;ws$RM*h(E>r0>Z$v*i8P-B+25Z{H%*C8~-N@d&ar(qS#l$gh3^P zdv+xZ(ruBvB%xF|hzyy86ua!qSgpgBR}szehp~bEpXrfgV^0{$kqw3h7Omh^qxnUU z76YCYc)6Y03O)ZMeCn+iaZWx=+q8->1{dNM!z5D|Ru$B(9t-Tlx)yPJk>&-jq`Yv% z;QYKy!uLxdxeX<(XYMa|y~R$Vs3^nQ-@9A+h5`Aak#&QzQY**tacyPHNApPst@!U2 zd_P6C&0zArPk)$g43k!N@f)?%CsQDcpGQ(H;@L>O(wghirjX6SU`j5t4>j%}(o#o0 z7x>H%KI&Vi9x?becJ7e2<{c}D5OxPixe_p&=%ftQu;3eoW$6htnwhQxia`eB7L1i+ zE8KaRoM>LP;G&ZmxkZ~4WYu?Sm$x=HTzZarA)QNu`csb~y_Ahz_>_;4WkF@0udtFA zI^6gG`n?d{q`Ts-M>5CT;c58-df}urab;^n-yEotKW*+t5bzX^Jfu|XZo*5f*MiG^ zWh!<_;c6g;TgZj4O!hr(gQcQvip6M6aOXx6or(t7H)A#!fy;y2c( zE+AGu{3D_Z6u5cX)jM*}xPFgCg(I}$rpK*}ppScErkakjPq9jm2~H9!GD;%5vWQ3U z!s$;Hg0OzNM}CZZ&gEMaN0w1&(ve`ffi+x5ULC3FBpWf>^#X@;ROL~G*PfqVCo+hu z+Nd`+=;*y}_rtVfDy!HNPMq<5J+XN{9ir}33yg*`mhDkg8z$tPX?-0fEd2}VeOu1-^DWQ``nva-GMPVQg+?L@ zlwa60>^S9j87}dSkjO4Eci&k_$Dh#JrjaCK5G5``5&p|AD3v)vQ{RA#NkX@aOWr^H zaJCJQzP>|c;;qgZ=x%E_?MS$8gq50M@`)PYAk8mq%!-$8ZYzV65jL#dL%pdY=jR*C`p0qBp zC4_5WmFXr;%H5$PV$7&r1w$XLSH)#&(-^2Mq^TUrmt>7GPX?`tO$_7L?v+B@+k~_} z?AUXRlG#)aMChZRQ$$;JKWs(i5f~~v;-u%3!&d7C`9H+Q2({a-T9XvaQ5kG4BsiV% z)>YV13yiqT7b3m-j6arVUdlD{Y6wbRdq%c)fn%TEdiIrTsFBQ)*cPBkY+*OJI_(1< zCI!U(;Du`Sh`JX8o{`99Pist+*TpU=oR$VFojUT$5Y320$@H|!ru`nrigDwq2^cW) z;-s$H4cM0FhFF}n%F;y$cKJj;?Ws8kh(xv|tlgKUYUHNyPj@cnsxhazJjLMOmRBc?=vCx)pn5Lb z}mTU+zxwTDM`pki;{W% zGJ6Hfxnz&DAN2w$)jo7s%}nP2Wo}@D$1@YplHNfVGA=x%>S9u#BTW;dyBHJgicgn3mS zuU|JeV!miCA`?S~%*Vvw8U3gO!G?U#hy!>+eskqyQ6;e#h8BNU^#E+GYyO42hCyt+ ziBl0M-Wvc!wPwg2IYX$6?R(-~=j6KaSZxW6bEjmih>svoK3vUIiMz^gvCKhv8)(yh7KQb;Ih;HwKg+wk9ly|-Lph~%v)3+kYOsXtJJb#)*cXSwTVGl1(m>3O zTAC8X_>T|L-TPT`;BsBjS>BP9PaAHH4r7A_BE`Umk{nq$sexl(gn~8X(tjp+a`|yl zoJCJ^fKw27;Ir7ra^*O#-@heIi~W}P^vdH0K@}Vm9jjnQH@+N-+Ge?+r<{g=Ws6x# z!Aaj6e*+D|LCLAPCmp9|DR30>q;`OY(u*{QMPj9z71PK-uSNX`52B^uXspm(z)38% z>q>L7itEB_HKHQ-2u4AWWeMG6BV*`|x%mYJgCJGACXKP1OBo6K@QJ1+*~JBg^Bady zEdPz&h(TUp^P303jtmEygle^4=+Df?fWxCYfS?s?KYWGA(w>2zP}S|nnB3Ug`i;nG z$`5U)*fZD~Q^wJS*u&(?uw%aUHU+q9pA;$7d(mgKnhtGn{es3`b*SpHzQ6_%cl1Fn$y2uB$ zlY8Cu^xwCVy>!kgfY(^PptkE5*T3gQ)?A#Qd7+5kyAAH<`aE>1;=ds9{UiDdm%ht1vLa zL35h?c&c2VPnY_rPJIkarGAHKzlfL;e|uyQJ7P;?K0z8qlfp5Dsdi?$&XmYdXbg$s zx=Y$Fku{RMgbH<9@Sv<^I~vVy#G;QTIq7$O49@pdHtS(TDp>^^{BWo>D{2d zylVbVfn!4WD~oi@SiQ=i0@IL|8&Lrt2;iwX>26HX@u8rc@x38XMcI*tSW9DZB!-un z{7j1OG}r~@DBA78st?iZOazGPe_=KnPQOrWlr@+;fPG}t-QBHG_pZ*- zEYTuG$6B8{5K1LJ0^Pp2Y^jHZ6#6&`u)kA?sh@FBw;6gV^m0O`Bf}h5M;MIPL&ygz zb7k_m$WhSfRT+&?ZWSIjz@w7}Z_uA!<9V`#jFeL!T?0dm0`;0V5mBIIKe!_Wk8V>0{>h}9z-%iIBBjXPpBFJ`zW>6=4L{_$)WNi|$aG*Go@Ms{zB=0= zc3JeM4*nN!(f|iqr+{_L9OnsK9ZC2Fg);GL>l8gebQAO|GnqIW!qRi6ScJy{Y8lo;DM2^6D)7t&)cUve+ z|Az)i815c7J0e)J1O?nti7agBa9AuZjYd{2@AIJPuq2P&@#cnG+E|Rz*Q;Kk*z2<0 z^b#!Sdz5 zf7b!p-1_}S$2B`wTe=jK5xVpIm1pb--?}kWX7)%xdqQ+jz zF?EUMey8bfL)#T5f}nMWkYbw>znE?8``Y>OL-$cf+uMO|elCqUU0F`v)!16uj0a$S zO(r(l2yAQL5sOO67CdwE!V$8EmA{ulHsfM3yoxGkY3aO%)os(g><>RU%8s%--~eDu{`Fyy~s|8>%TV1osP z{>4b-|G)?g{trly|9}J*`ae;6mG~DW^6=kxF&_y+PYvZm!Mv@skId(K3%YR`NZ1@0 zZ+^>YtX>^-eEv;9&YeU>g{>_5x~5yLMm01YIhOli3D50JL^i~^mrddIwLKR0tUXXe z?5#g~hNU9Sg$79V%|3XIMrKnld!W$;`8i|88ZBremg{1OGiz~ueqcAzxj*=_@G5x& zLYu#n$KO0(k)OP$n__8dOs$=iJzOS`M_#gMTZGcZIjeouTSVJy2H|OcaCasD{l6*x z&Ej7eMMVCE5zPNz9Q8HfR)6F9mvzbwrTet@yUg_Hi@hjep?xw?D1{@K4Uzlbm)8=JhE0^$Dy8QlWW literal 0 HcmV?d00001 diff --git a/images/neuron_model.svg b/images/neuron_model.svg new file mode 100644 index 0000000..eaa3212 --- /dev/null +++ b/images/neuron_model.svg @@ -0,0 +1,239 @@ + + + + + + + + + + + + + + + + + + + + Activation Function + + + + x1 + x2 + x3 + w1 + w2 + w3 + y + Σ + + diff --git a/images/percepton_problem.pdf b/images/percepton_problem.pdf new file mode 100644 index 0000000000000000000000000000000000000000..8775a20801b6da317f727875e8394ba1c0e2f7bc GIT binary patch literal 5539 zcma)=cQjmE+rYKxeY7OX=%SBNhA6@4eS+wM8NEb{E_!rENf@0WN=S4rq6MS3h-lG! z^iIm>=DYW5Kx4T8yrC_D@*L@<$~}vq;TxbkRBsmAhu{#7<-@5(BT`VfEA^nX&PBUy0MgixeuNfZM#kM$Nw9i7<}K=( z@6{$Brd>VSj!|IB%S>*;ZRF(5ukT5#i@)YrUi^@wAEg}Jy1IP%{gmM9MB-wzo$=>s z->;^???;z=grCM<3{sZGgExFX2A}(WmoCW;kn<$(yEwA!%X3Dx|2i`|=@)TyD(Hu{ zySKi3DmRd?#FaMwVdJS>(!}UQ1+YFx;K^M>ZnLDo#iZ!x8s?N2yIIKKD+%M#hT+#t z>9X3E15}6(R7zT~erLOIE8hr$G6OJwUa3-5HmxYSjnKQMCt86kg_=RJZeUYU^GP*FmSWqq6tY z?VaS0>jgTe*(9g-h?`dD&AzaGM^API1AkO1p0Oyy=lQj?X~Bgb(JnR#@8LPCvy?|B zu1;?X4AoXv_Kj(pE~G*O(#5+bWzYA%2HY$cHWem3KOHC@z<#iur=Hz$!KT^upw9N^ zb-7m(!1&#<_!x8_$#&8vL+ZWzHbY5Aht63ava3Ceudx=g)T1Ck0^nPY++H)bZ-#-p%w9ZN^-E zwFsp~Pdyz?W$#i5)BM)iT?wBlI88L6} zZ^F$LC-%tEZ1hXOgPH}JMb;FJoog5zGm&gz-K_v+1WV-+37`zg9JG4w$c+lh^Nl}7 zjPW+O9rog(4smmH^tb{j;Nn_~k9*pH1hs8c_B9`#Tp41hjXX6Q*_})t`=z)nYoa&X zFz0BrkDBoVDtn(!6zvua=vVrYO|G8axOa<#PfwwcqD+qZ7-O~|r>*VBLD4W)%o?}B zVK#8mT0>LDj8NTqgDF+?%ZP@2A=bKltvbo?lZBF(VB>@l>EN}$NNadg92eeVksoAH zF0DRZvpy{mfc&ARU+`3AJ3Ol)6CcyeqP}L!!DSls67%RDk%%_UB+0H0jCWA@{t`JJ zMfLZT+#-spvZlh#nc}E{$KPUh}>ge+qG12a8ZlKFb+p&a>k1|w9H08zx z{tSLO;aE^M(R{5=@e`;|r^v4wa5jUIqXG=~Fv^zo{quF?I8u6@Ddp*DVGEO;-mXqS zgu{0#tvAH$+2HNY6X_N_=pp~SB+!VG5|CUFn19nYM&6$!p_*nrOo_~u1h0E)!rHc8 zD(yCn&O=hX4?XxuErv((7!SWg9GqB>N7?Gbo z-nldlqxL9JX4eRv?~g2{W~UdavmnIT#6iw5y~f_f#duIF^r}G+q-WvDp`fGpi*Blv30%GEU=@>gv6Dq=v`#hYvE6Pgk2G3;Kv4WQiB2 z;yB^hZ2fSG>XD%3Jj%p(>^Ql0TAm6bsx-Ffb~8>u<@G@d#n8%(C6o@`pk$jaH$~#{ z~e}ck6oK#;?6PWw{@ZtDu{*h|4M0LIXH1B^)T(WDRgx>HAG{t}_RdaoHaa zF$$-`wmuVNa}hR)H(G=bC{|9ZJR{M?<<1C{NkMx=xDZm_915$!UB(R{1FFviJ#|k# z?`zQ}Yoy<=2M90ei7Aoo_^1ij)A_vW*VGivV~g83h#beN_V#u!XlMJes$Zn#L?47! zNl(>m8_YMH##M-lqmrW53oIw+ZG8|i|3hCs2L}h$>-`a3%t>x;_o%{_zMBYyWRqbQTQM-R+*4PA*pReA3w)>G;$l}w>ToUC+l-KHk1^ZW(M zV77NnV?efOzA!^R5$pAJ084VIk}hhRHJr{2Dt!sTl~Gg-jE;o>u*r!V zJ!h~`F+JWZ-I3O0`#TRE)$!2#pJLNRMBitA4Uc zSL87FpFT!b_6L(;5fN`QMk=46b!HLV@7BjCF=jGN9l~5sI_T1TEuq%E$74||<{BSo zIO6~Y^ZWwF%$ns^Axc*g4y0hlDXEh?V`0L9JUWDLDI}Kz>F&ng^QhElO_N11^PCVt zTVo-GP*N?iXFr}%Enl&jf((I#iN@b7$zSxzBToipljzXJmT!qQe1>fxEt6yT^BfwO zoY}oQdYsgyX!?j~cpEmJo05h&6^3&n&0@y=U5vHiE;pLOm?(f3)5L^`#O9n?D>n#c zEsUutW*-uLHpih}QWZcIu65j|yf?CMUK)rrN!oAV9`ILC;W1l~Rk8(3q*95X=uC^} zLIT(&3QcnqGdMRG8!MZLeM|ur^O`@l;&&^<$3~`m0uPx_x#!kQP86*7drNRf*)eTS zO*|B7f>Hr|I1G6%Vk)X@L5 zK5!RSC|-s&2G@awr`wB>!+)7=N4ru}S3z~Cj>)oTXRcG%hg5vP{1ysWJ`|x>5fu$x z8jVby>-XXOyefNkJWpbK%ScQ}sRbiH5LFNPaJqAX;p|BtqfIMH#6F3$tFE5FXpo^z zj^n{&U^~C0un_Spwv-cJU!3Z2*5oW-PkQ&!A&^1Q=)}-!IVl_fjmND{%XdPzS)ZXY zh=Kxa@&^IH< zx2bsL$}NFrR0Mfbq6j-}6fKm6DodhxMqVjd_7&i){7c|D53q|aZs2^W?u|V{|Fx@3 zl*PhNwCQQ~AjfMXe4Ku3-2QCyWi&^>QG1&BsJ_Yt3`GX%DpXq8&j>X=|hEl_IcQQ_bruOa(x(A1Mho+ zS)B!@oqLq6Yc_e^@+7WG^Z4*w%4yt`yO7MFhwy|*s7<%qa0L-n_e`17-NGGP3l3yX z5~t)K|(#b$v$Cs0v!7B=Ck)=VlmSQ6;4iEJRi8SZ#sJzYy;dGpKwgoeM1lj(*69L>B8VXUX%|_gi&?lxxWbtMRZ7 zPMcojiQCEkk!5v6jYCZ*nNmHdr^2_C#jdz7aHQTY!DT+CXA|#G6;~Zv_LSuBC{Q2_U?EZ7yK}U&PpjWDZ1a6 zuCr|EiEpPR8#jsnsV2Gt`C+@q-L^6(hF%-%F!njyDYPVdusGn;(5TGkQdh)lDVjE5 zI13}44Tr{i1(3d0WIm@%z%XNsr65sGw^XT;U}LE%!D183)@4;fcXhx4M@1sZgjk8| z2)-6+8eme=WIfXZ)i6sb(+|SiEA)=9MJxL$r{N(c7*;tg?!om3Q$*`ckIzQFrzrNE zg2zy!#+V0Can-rNZ8NV*H)bruzbB0}o0 z`_m;J0>VsI;|lM`)R_wQ-g&x8ZQuF866yA$r)1Z+jI7)Yv=X2W*5 zDT4vHSeC2&C(^S?hQZkbYebj#~z z{vS&*yE&d8R}B`+*oye%t_Qt8yD53N1crBH_cdKAmpamI)^RU6k?3#RQ zwx7#|O7^)6-M^H7q7P}{O$b~u!Q|2^T4uHZQQTvPe^fmfF|g0lD{A01wv;dkN+-`G zGnl&I<=Dv|)|ObwqTVdHkL%RbcDZF2)g5=h$vnFx!y*OHUC!%ETp9#on)E~Bvq%Qs z+UKUpe;faD(R?QeFt9ZmG?w)k>F51GBwE+?nw8u!ds;bvrY9Yk`A+d(yMLHw!z}Sk zc#Ll8?9VEKa*OUq$`%qW@zZ|{pq*T3ZCbY@=8Z&gTZZSq!SE|Su{Uok0TPqH)z2lr z^i5tS!I=+#miz39g|U6Z*yoA3;Z@i5U>s8~DV+7E>oR&#QRzFSm4Y8yp%Qpb){d#m zDQLH0*V5h160LUqN1s-5_}BU$)l$1=Zg(InUW0_Q_;su(fC756`2ROeGQ%K}aA6v}r3FiOjVEH& zdJ#+cceZtT8cer=+lprK940&MZYkD%H@+8%>?^EC6YG9>vK!~LS{nw(GTnyPw^EL{^U>{PeIN83B|A1-90f@^A7 z4ChAd_-%`2#P)Zf2YPzbpP4IozaL@@Rma~;>FKr?hrOus1!awzjm%u zoZwmIhjJMjC$7;c9B7Q#%t}oyYOaEJXGL>HAeCWE@6CNK67GE^hjY3AL~ei?3Qpt@ z&W#50>@%-Y)^yEj^rOu-JWYn8C*11QY>UGDzlg8sky_W?ef + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + -1 + -1 + +1 + +1 + +1 + 1st line + 2nd line + + diff --git a/images/slr_illustration.pdf b/images/slr_illustration.pdf new file mode 100644 index 0000000000000000000000000000000000000000..9649976e19e40b5790b1e4ea00963f16780d3445 GIT binary patch literal 2071 zcmY!laBlNxq`rxG)T^*i=bNbq4%j{TEc{qIOzoZS)Mv+M|9;3c^}yawQXkKk z|9_eN;r{>97%Nk~edb*|M8r<>T6vyJqOoU5(#N3Nnos*IGy}zM zO3Xas*SOl+Imt2f_C$j{zrrqAXdAhen z%;tzYOoeF$?FXklWIJ=?OHJ#^i|%j!&G>Z8gQa?zXkj{spUCnps(kyJdJbQ4+L|_* zp-<$}mh3=1rr|l-;2z#a(WOHCahw83!=M6#|WjlMrI|@0bEN@NY zIxy48`EJ>S?is;4ys0|g!9Nm`qW0dd+SwI0DLU#|dQ^U$vB1+CwMR9l>rB@wi=OK! zoo`b!xAW_xa}z=`EdyCwrIKV8oiQo-6}36&2Gb7Tp2^L;HzVV@%!Mz%x_slbDbM;0 z_RagRdrDpuKG)0eXv%{%-*-v*9WB(_@k&IAx2JjDZRP@n&-2!L96f%l(o|~hYt;u2 zW^M_#D0{*E-evvH?R%E*ZJ#=Qa);f_XKkFX*PGW%OyOJ}yTNXUN~0n7u0t6Lna(ja zjC)t4l)F8+bLB?=p=04ag%UAbf^*$e8x~t#Q4pVJtizpJHf>qKXT!I4dk?M`c~u!E zrW#!~)q(kG3`3#zW77uP&L2(jCvPfT%E)}1oX))9@`d)?ZBC)`?GCCNqI0e0m9cU! z_;l{Wh7aHNtFGQ`|7@Smy|)WqKdRH_xi0#<;rZQSiOA9ug6DVrG|S{@o2`HT-YMxt zii>{Po)z6HxcT(i>;0NJHAf}dXY!lB$(|a~KVj={-4m_h^RMYlxfwYN7AxS3F8=vzKK<|df9?D4ox5&+?}|Y5^_cnXw&%oC z52`t@KD<|aW|#S}p)tpb`aCjDTohL1JiNYz|TY;u^yX z9B>iiT2bO2Tmmd!6l`oj0zjajXaM9W=sPl{{`=Fr*ftX(rjxayEl+Ok=dj+u@xk$O&cO+b zxC<5*zRLc2@Ni()Byq#XlGaMr8)i(kxnFE^I8{0Q^)9xf`|o7!)Zlp7e18AI1=4rt zys9%h#n1oxSbD_X3)7^^OBa1N3AlbuZnf5*_a{!c@~2E__dOnexyy7aW^jZ23JY#i z3s@*w5D`j-hDgC=j1)|UW<-UPZ~sodLk0pa)%8b9R9HXWn&a5fX41i?XtiKN)6|Fs z7Mtca?N>K!&q)9N_MEW7v}89Q0hO~is(9TM?cZJV$WzS}TnGq?)Dy1B2;|t|3pIW)iZ0Y$mt1o@2J(ek&)BkqXZvD{z z+tiIaPxqV5KgbaKPcwP86Kn612+v8ir4ug~DLpOnxhxp?YwfpbH&-o?+tyVf(kzkF zwp5&f$@MB`D1yTR7K|3~U^Fzs2*-$^)HHDLz-mnkoPh;O)4`clsS1XW&<$6JHdHWD zFp3512+Ged0p<{R9C+rX2mH+?% literal 0 HcmV?d00001 diff --git a/images/slr_illustration.svg b/images/slr_illustration.svg new file mode 100644 index 0000000..a88d33d --- /dev/null +++ b/images/slr_illustration.svg @@ -0,0 +1,133 @@ + + + + + + + + + + + + + + + + + + + + + + + diff --git a/main.tex b/main.tex index 387cc7c..bafe0ef 100644 --- a/main.tex +++ b/main.tex @@ -74,7 +74,14 @@ Descriptive Models provide new information to describe the data in a new way. Cl The difference between classification and regression is that classification provide a categorical output (one element of a pre-deternime finite set, for example a label) when regression provide a continuouse output as a real number. +\newpage \section{Simple Linear Regression} + +\begin{figure}[h] + \centering + \includegraphics[width=\textwidth]{images/slr_illustration.pdf} +\end{figure} + We are interested in finding a function that represent best the non-functional relationship\footnote{Meaning that the output cannot be expressed as a mathematical function of the input.} between the input and the output. We suppose that we have a dataset of $n$ input/output points. We can first try to determine if there is a statistical relationship between input and output. For that we look at the covariance and the corelation. With $\bar{Y}$ the average of the outputs and $\bar{X}$ the average of the inputs and $S$ the standard deviation: @@ -94,9 +101,16 @@ This is equivalent to using the Maximum Likelihood Estimator (MLE) with the (rea This works well for monovariate problems. In the case of multivariate problems, you have to take each input individually and evaluate the linear regression from this input to the output and the problem becomes way more complex (not detailed here). +\newpage \section{Logistic Regression} -gistic regression is fundamentally different from Linear Regression in the sens that it returns the probability of an input belongign to one class. This probability can be compared to a threshold (usually $0.5$) to express the predicted class of the input. In this sense, the logistic regression does not return a real number but a categorical output. In this context, the "training" data consist of inputs that are real number and output that are binary (either 0 or 1). The logistic regression can be viewed as an extension of the linear regression where the affine curve is "bent" to be confined between 0 and 1. We start by expressing the logistic curve as the probability of the input $x$ belonging to the class 1. +\begin{figure}[h] + \centering + \includegraphics[width=\textwidth]{images/logreg_illustration.pdf} +\end{figure} + + +The Loggistic regression is fundamentally different from Linear Regression in the sens that it returns the probability of an input belongign to one of two classes. This probability can be compared to a threshold (usually $0.5$) to express the predicted class of the input. In this sense, the logistic regression does not return a real number but a categorical output. In this context, the "training" data consist of inputs that are real number and output that are binary (either 0 or 1). The logistic regression can be viewed as an extension of the linear regression where the affine curve is "bent" to be confined between 0 and 1. We start by expressing the logistic curve as the probability of the input $x$ belonging to the class 1. \begin{equation} p(x) = \dfrac{1}{1+e^{-y}} = \dfrac{1}{1+e^{-(\beta_1x+\beta_0)}} @@ -114,4 +128,112 @@ L_k = $$ The issue now is that this function is non-linear so we cannot simply derive it and solve for 0 to get the optimum. Numerical methods such as gradient descent or Newton–Raphson need to be leverage to approximate the optimum $\beta$ value. + +\section{Fundamentals of Artificial Neural Networks} + + +\begin{definition}{Artificial Neural Network}{ann} + An Artificial Neural Network (ANN) is a systems that can acquire, store and utilize experimental knowledge. It is a set of parallelized and distributed computational elements characterisez by there architecture, learning paradigme, and activation functions. +\end{definition} + +ANNs are based on \textit{what and how we think} brain's neurons look like and function. Each neurone is a computational element that receivs impulsion or signal from one or more other neurons and, based on a set of intrinsec parameters (the weights) and a threshold/bias (the activation function), transmit a signal to one or more subsequent neurons. The ANNs can be categorized using multiple criteria: + +Topologies: +\begin{itemize} + \item Feed Forward: The information flow goes from the first to the last layer in only one direction. + \item Recurent: There is no main information flow direction. At some point the information goes out of the network. +\end{itemize} + +Activations functions: +\begin{itemize} + \item Steps. + \item Sigmoid. + \item Sigmoid derivativ. + \item Linear. + \item $\dots$ +\end{itemize} + +Learning paradigmes: +\begin{itemize} + \item Supervised learning: The learning is guided by the error. + \item Unsupervised learning. + \item Reinforcement learning: Values or penalize sequences of steps based on the result. + \item $\dots$ +\end{itemize} + +\newpage +\subsection{McCulloch-Pitts Model} +The first model of a neurone was proposed to represent the computing process of a biological neurone with mathematical elements. + + +\begin{figure}[h] + \centering + \includegraphics[width=0.7\textwidth]{images/neuron_model.pdf} +\end{figure} + +This model does not include any learning mechanism. The weights and activation function parameters need to be set manually in order to procude a meaningfull result. In itseft, this model is pretty but not very usefull. In order to make it learn, we turn it into a perceptron by updating the weights iteratively depending on the output generated by the neurone on our training set of datapoitns. + +\subsection{Perceptron} +\begin{figure}[h] + \centering + \includegraphics[width=0.5\textwidth]{images/percepton_problem.pdf} + \label{fig:perceptron_problem} + \caption{Example of a problem solvable with the perceptron.} +\end{figure} + +The perceptron can learn and adjust its parameter using trainig data. The aim of this problem is to find a hyperplan (a line in this case) that separates the $+1$ and $-1$ points. If each class is on one and only one side of the line then the problem is solved. There is no notion of \textit{best separation line} (like there would be for an SVM for example). To solve this problem, we use a one-neurone model. This neurone has weights $\omega = [\omega_1,\omega_2]^T$ and a threshold $\theta$. The equation of the output $o_i$ from the input $x_i = [x_1,x_2]$ is given by: +\begin{equation} + o_i = [\omega_1,\omega_2,\theta] \cdot [x_1,x_2,1]^T = \omega \cdot x_i + \theta +\end{equation} + +Without any better idea of where to start, we initialize the parameters to random (but reasonable) values: $\omega_1=-1, \omega_2=1, \theta=-1$. This set of parameters give the first line presented in figure \ref{fig:perceptron_problem}. + +The first line is not very good because the lower $-1$ point is missclassified. We can use this point to update the line. Let's define the the update function using $\eta$ the learning rate (used to reduce the variation between two iterations in order to stay in a stable state). + +\begin{align*} + \Delta\omega &= 2\eta t x_i\\ + \Delta\theta &= -2 \eta t \textrm{ if $t \neq o$, oterwise $\Delta\theta = 0$} +\end{align*} + +With this update function and $\eta=0.1$, we can update the weights to $\omega = [-3,-1], \theta=0$. This produces the second line that perfectly. + +The one-neurone perceptron can learn and solve linearily separable problems. There is one hyper parameter to tune but other than that it should converge to a solution if the learning rate is not to big. The stopping criteria is is either finding a solution or reaching a number of iterations. + +\subsection{Multi-Layer Perceptron} +One neurone is good but it is very limited. For example, it is limited to linearily separable problems. The XOR problem where two classes are spread along the both diagonalsof the plan is an example of a very simple and yet non-linearily separable problem where a single-neurone perceptron will fail. We need to connect multiple neurones together now to tackle this kind of problems. We make a few changes to the neurone to prepare it to the use in a network. + +\begin{itemize} + \item The neurone has now a trainable bias which is just a third parameter along with the weights. + \item The training function is now defined as $\Delta \omega = \eta \left( \dfrac{\partial P}{\partial \omega_1} + \dfrac{\partial P}{\partial \omega_2}\right)$ + \item The performance function is define as $P = -\dfrac{1}{2} (t-o)^2$. The square is to make it smooth and derivable, the half is because we know we are going to derivate it at some point, and the negative sign is to make it a maximisation problem (no real use, maximizing performance is the same as minimizing loss or cost). +\end{itemize} + +We can now conenct the two neurones and this minimal network will produce an output $O_i$ for each input $x_i$. + +\begin{figure}[h] + \centering + \includegraphics[width=0.8\textwidth]{images/2neurones.pdf} + \label{fig:2neurones} + \caption{Minimal multi-layers perceptron with two neurones..} +\end{figure} + +We want to maximise the performance value $P$ using the inputs $x_i$. This maximisation problem requires to determine the derivative of $P$ with respect to each parameter and use it to update the parameter (gradient descent method). In order to obtain the derivative of $P$ with respect to both $\omega_i$, we leverage the chain rule to get from the end of the network to the begining using the derivative at each step: + + +\[\arraycolsep=4pt\def\arraystretch{2.2} +\left\{ + \begin{array}{ll} + \dfrac{\partial P}{\partial \omega_2} &= \dfrac{\partial P}{\partial O_i} \dfrac{\partial O_i}{\partial P_2} \dfrac{\partial P_2}{\partial \omega_2}\\ + \dfrac{\partial P}{\partial \omega_1} &= \dfrac{\partial P}{\partial \omega_2} \dfrac{\partial \omega_2}{\partial y} \dfrac{\partial y}{\partial P_1} \dfrac{\partial P_1}{\partial \omega_1}\\ +\end{array} +\right. +\] + +Each derivative is fairly easy to get. We can express our deltas as: +\begin{align*} + \dfrac{\partial P}{\partial \omega_2} &= (t_i - O_i) \times (1-O_i)O_i \times y\\ + \dfrac{\partial P}{\partial \omega_1} &= (t_i - O_i) \times (1-O_i)O_i \times \omega_2 \times(1-y)y\times x_i\\ +\end{align*} + +These derivatives can be used now to update the valeus of $\omega_1$ and $\omega_2$ according to the learning rate $\eta$. This chaining of derivative from the output all the way to the input of the network is the idea behind backtracking. \end{document}