(original) (raw)

%!PS-Adobe-3.0 %%BoundingBox: (atend) %%Pages: (atend) %%PageOrder: (atend) %%DocumentFonts: (atend) %%Creator: Frame 4.0 %%DocumentData: Clean7Bit %%EndComments %%BeginProlog % % Frame ps_prolog 4.0, for use with Frame 4.0 products % This ps_prolog file is Copyright (c) 1986-1993 Frame Technology % Corporation. All rights reserved. This ps_prolog file may be % freely copied and distributed in conjunction with documents created % using FrameMaker, FrameBuilder and FrameViewer as long as this % copyright notice is preserved. % % Frame products normally print colors as their true color on a color printer % or as shades of gray, based on luminance, on a black-and white printer. The % following flag, if set to True, forces all non-white colors to print as pure % black. This has no effect on bitmap images. /FMPrintAllColorsAsBlack false def % % Frame products can either set their own line screens or use a printer's % default settings. Three flags below control this separately for no % separations, spot separations and process separations. If a flag % is true, then the default printer settings will not be changed. If it is % false, Frame products will use their own settings from a table based on % the printer's resolution. /FMUseDefaultNoSeparationScreen true def /FMUseDefaultSpotSeparationScreen true def /FMUseDefaultProcessSeparationScreen false def % % For any given PostScript printer resolution, Frame products have two sets of % screen angles and frequencies for printing process separations, which are % recomended by Adobe. The following variable chooses the higher frequencies % when set to true or the lower frequencies when set to false. This is only % effective if the appropriate FMUseDefault...SeparationScreen flag is false. /FMUseHighFrequencyScreens true def % % PostScript Level 2 printers contain an "Accurate Screens" feature which can % improve process separation rendering at the expense of compute time. This % flag is ignored by PostScript Level 1 printers. /FMUseAcccurateScreens true def % % The following PostScript procedure defines the spot function that Frame % products will use for process separations. You may un-comment-out one of % the alternative functions below, or use your own. % % Dot function /FMSpotFunction {abs exch abs 2 copy add 1 gt {1 sub dup mul exch 1 sub dup mul add 1 sub } {dup mul exch dup mul add 1 exch sub }ifelse } def % % Line function % /FMSpotFunction { pop } def % % Elipse function % /FMSpotFunction { dup 5 mul 8 div mul exch dup mul exch add % sqrt 1 exch sub } def % % /FMversion (4.0) def /FMLevel1 /languagelevel where {pop languagelevel} {1} ifelse 2 lt def /FMPColor FMLevel1 { false /colorimage where {pop pop true} if } { true } ifelse def /FrameDict 400 dict def systemdict /errordict known not {/errordict 10 dict def errordict /rangecheck {stop} put} if % The readline in PS 23.0 doesn't recognize cr's as nl's on AppleTalk FrameDict /tmprangecheck errordict /rangecheck get put errordict /rangecheck {FrameDict /bug true put} put FrameDict /bug false put mark % Some PS machines read past the CR, so keep the following 3 lines together! currentfile 5 string readline 00 0000000000 cleartomark errordict /rangecheck FrameDict /tmprangecheck get put FrameDict /bug get { /readline { /gstring exch def /gfile exch def /gindex 0 def { gfile read pop dup 10 eq {exit} if dup 13 eq {exit} if gstring exch gindex exch put /gindex gindex 1 add def } loop pop gstring 0 gindex getinterval true } bind def } if /FMshowpage /showpage load def /FMquit /quit load def /FMFAILURE { dup = flush FMshowpage /Helvetica findfont 12 scalefont setfont 72 200 moveto show FMshowpage FMquit } def /FMVERSION { FMversion ne { (Frame product version does not match ps_prolog!) FMFAILURE } if } def /FMBADEPSF { (PostScript Lang. Ref. Man., 2nd Ed., H.2.4 says EPS must not call X ) dup dup (X) search pop exch pop exch pop length 4 -1 roll putinterval FMFAILURE } def /FMLOCAL { FrameDict begin 0 def end } def /concatprocs { /proc2 exch cvlit def/proc1 exch cvlit def/newproc proc1 length proc2 length add array def newproc 0 proc1 putinterval newproc proc1 length proc2 putinterval newproc cvx }def FrameDict begin /FMnone 0 def /FMcyan 1 def /FMmagenta 2 def /FMyellow 3 def /FMblack 4 def /FMcustom 5 def /FrameNegative false def /FrameSepIs FMnone def /FrameSepBlack 0 def /FrameSepYellow 0 def /FrameSepMagenta 0 def /FrameSepCyan 0 def /FrameSepRed 1 def /FrameSepGreen 1 def /FrameSepBlue 1 def /FrameCurGray 1 def /FrameCurPat null def /FrameCurColors [ 0 0 0 1 0 0 0 ] def /FrameColorEpsilon .001 def /eqepsilon { sub dup 0 lt {neg} if FrameColorEpsilon le } bind def /FrameCmpColorsCMYK { 2 copy 0 get exch 0 get eqepsilon { 2 copy 1 get exch 1 get eqepsilon { 2 copy 2 get exch 2 get eqepsilon { 3 get exch 3 get eqepsilon } {pop pop false} ifelse }{pop pop false} ifelse } {pop pop false} ifelse } bind def /FrameCmpColorsRGB { 2 copy 4 get exch 0 get eqepsilon { 2 copy 5 get exch 1 get eqepsilon { 6 get exch 2 get eqepsilon }{pop pop false} ifelse } {pop pop false} ifelse } bind def /RGBtoCMYK { 1 exch sub 3 1 roll 1 exch sub 3 1 roll 1 exch sub 3 1 roll 3 copy 2 copy le { pop } { exch pop } ifelse 2 copy le { pop } { exch pop } ifelse dup dup dup 6 1 roll 4 1 roll 7 1 roll sub 6 1 roll sub 5 1 roll sub 4 1 roll } bind def /CMYKtoRGB { dup dup 4 -1 roll add 5 1 roll 3 -1 roll add 4 1 roll add 1 exch sub dup 0 lt {pop 0} if 3 1 roll 1 exch sub dup 0 lt {pop 0} if exch 1 exch sub dup 0 lt {pop 0} if exch } bind def /FrameSepInit { 1.0 RealSetgray } bind def /FrameSetSepColor { /FrameSepBlue exch def /FrameSepGreen exch def /FrameSepRed exch def /FrameSepBlack exch def /FrameSepYellow exch def /FrameSepMagenta exch def /FrameSepCyan exch def /FrameSepIs FMcustom def setCurrentScreen } bind def /FrameSetCyan { /FrameSepBlue 1.0 def /FrameSepGreen 1.0 def /FrameSepRed 0.0 def /FrameSepBlack 0.0 def /FrameSepYellow 0.0 def /FrameSepMagenta 0.0 def /FrameSepCyan 1.0 def /FrameSepIs FMcyan def setCurrentScreen } bind def /FrameSetMagenta { /FrameSepBlue 1.0 def /FrameSepGreen 0.0 def /FrameSepRed 1.0 def /FrameSepBlack 0.0 def /FrameSepYellow 0.0 def /FrameSepMagenta 1.0 def /FrameSepCyan 0.0 def /FrameSepIs FMmagenta def setCurrentScreen } bind def /FrameSetYellow { /FrameSepBlue 0.0 def /FrameSepGreen 1.0 def /FrameSepRed 1.0 def /FrameSepBlack 0.0 def /FrameSepYellow 1.0 def /FrameSepMagenta 0.0 def /FrameSepCyan 0.0 def /FrameSepIs FMyellow def setCurrentScreen } bind def /FrameSetBlack { /FrameSepBlue 0.0 def /FrameSepGreen 0.0 def /FrameSepRed 0.0 def /FrameSepBlack 1.0 def /FrameSepYellow 0.0 def /FrameSepMagenta 0.0 def /FrameSepCyan 0.0 def /FrameSepIs FMblack def setCurrentScreen } bind def /FrameNoSep { /FrameSepIs FMnone def setCurrentScreen } bind def /FrameSetSepColors { FrameDict begin [ exch 1 add 1 roll ] /FrameSepColors exch def end } bind def /FrameColorInSepListCMYK { FrameSepColors { exch dup 3 -1 roll FrameCmpColorsCMYK { pop true exit } if } forall dup true ne {pop false} if } bind def /FrameColorInSepListRGB { FrameSepColors { exch dup 3 -1 roll FrameCmpColorsRGB { pop true exit } if } forall dup true ne {pop false} if } bind def /RealSetgray /setgray load def /RealSetrgbcolor /setrgbcolor load def /RealSethsbcolor /sethsbcolor load def end /setgray { FrameDict begin FrameSepIs FMnone eq { RealSetgray } { FrameSepIs FMblack eq { RealSetgray } { FrameSepIs FMcustom eq FrameSepRed 0 eq and FrameSepGreen 0 eq and FrameSepBlue 0 eq and { RealSetgray } { 1 RealSetgray pop } ifelse } ifelse } ifelse end } bind def /setrgbcolor { FrameDict begin FrameSepIs FMnone eq { RealSetrgbcolor } { 3 copy [ 4 1 roll ] FrameColorInSepListRGB { FrameSepBlue eq exch FrameSepGreen eq and exch FrameSepRed eq and { 0 } { 1 } ifelse } { FMPColor { RealSetrgbcolor currentcmykcolor } { RGBtoCMYK } ifelse FrameSepIs FMblack eq {1.0 exch sub 4 1 roll pop pop pop} { FrameSepIs FMyellow eq {pop 1.0 exch sub 3 1 roll pop pop} { FrameSepIs FMmagenta eq {pop pop 1.0 exch sub exch pop } { FrameSepIs FMcyan eq {pop pop pop 1.0 exch sub } {pop pop pop pop 1} ifelse } ifelse } ifelse } ifelse } ifelse RealSetgray } ifelse end } bind def /sethsbcolor { FrameDict begin FrameSepIs FMnone eq { RealSethsbcolor } { RealSethsbcolor currentrgbcolor setrgbcolor } ifelse end } bind def FrameDict begin /setcmykcolor where { pop /RealSetcmykcolor /setcmykcolor load def } { /RealSetcmykcolor { 4 1 roll 3 { 3 index add 0 max 1 min 1 exch sub 3 1 roll} repeat setrgbcolor pop } bind def } ifelse userdict /setcmykcolor { FrameDict begin FrameSepIs FMnone eq { RealSetcmykcolor } { 4 copy [ 5 1 roll ] FrameColorInSepListCMYK { FrameSepBlack eq exch FrameSepYellow eq and exch FrameSepMagenta eq and exch FrameSepCyan eq and { 0 } { 1 } ifelse } { FrameSepIs FMblack eq {1.0 exch sub 4 1 roll pop pop pop} { FrameSepIs FMyellow eq {pop 1.0 exch sub 3 1 roll pop pop} { FrameSepIs FMmagenta eq {pop pop 1.0 exch sub exch pop } { FrameSepIs FMcyan eq {pop pop pop 1.0 exch sub } {pop pop pop pop 1} ifelse } ifelse } ifelse } ifelse } ifelse RealSetgray } ifelse end } bind put FMLevel1 not { /patProcDict 5 dict dup begin <0f1e3c78f0e1c387> { 3 setlinewidth -1 -1 moveto 9 9 lineto stroke 4 -4 moveto 12 4 lineto stroke -4 4 moveto 4 12 lineto stroke} bind def <0f87c3e1f0783c1e> { 3 setlinewidth -1 9 moveto 9 -1 lineto stroke -4 4 moveto 4 -4 lineto stroke 4 12 moveto 12 4 lineto stroke} bind def <8142241818244281> { 1 setlinewidth -1 9 moveto 9 -1 lineto stroke -1 -1 moveto 9 9 lineto stroke } bind def <03060c183060c081> { 1 setlinewidth -1 -1 moveto 9 9 lineto stroke 4 -4 moveto 12 4 lineto stroke -4 4 moveto 4 12 lineto stroke} bind def <8040201008040201> { 1 setlinewidth -1 9 moveto 9 -1 lineto stroke -4 4 moveto 4 -4 lineto stroke 4 12 moveto 12 4 lineto stroke} bind def end def /patDict 15 dict dup begin /PatternType 1 def /PaintType 2 def /TilingType 3 def /BBox [ 0 0 8 8 ] def /XStep 8 def /YStep 8 def /PaintProc { begin patProcDict bstring known { patProcDict bstring get exec } { 8 8 true [1 0 0 -1 0 8] bstring imagemask } ifelse end } bind def end def } if /combineColor { FrameSepIs FMnone eq { graymode FMLevel1 or not { [/Pattern [/DeviceCMYK]] setcolorspace FrameCurColors 0 4 getinterval aload pop FrameCurPat setcolor } { FrameCurColors 3 get 1.0 ge { FrameCurGray RealSetgray } { FMPColor graymode and { 0 1 3 { FrameCurColors exch get 1 FrameCurGray sub mul } for RealSetcmykcolor } { 4 1 6 { FrameCurColors exch get graymode { 1 exch sub 1 FrameCurGray sub mul 1 exch sub } { 1.0 lt {FrameCurGray} {1} ifelse } ifelse } for RealSetrgbcolor } ifelse } ifelse } ifelse } { FrameCurColors 0 4 getinterval aload FrameColorInSepListCMYK { FrameSepBlack eq exch FrameSepYellow eq and exch FrameSepMagenta eq and exch FrameSepCyan eq and FrameSepIs FMcustom eq and { FrameCurGray } { 1 } ifelse } { FrameSepIs FMblack eq {FrameCurGray 1.0 exch sub mul 1.0 exch sub 4 1 roll pop pop pop} { FrameSepIs FMyellow eq {pop FrameCurGray 1.0 exch sub mul 1.0 exch sub 3 1 roll pop pop} { FrameSepIs FMmagenta eq {pop pop FrameCurGray 1.0 exch sub mul 1.0 exch sub exch pop } { FrameSepIs FMcyan eq {pop pop pop FrameCurGray 1.0 exch sub mul 1.0 exch sub } {pop pop pop pop 1} ifelse } ifelse } ifelse } ifelse } ifelse graymode FMLevel1 or not { [/Pattern [/DeviceGray]] setcolorspace FrameCurPat setcolor } { graymode not FMLevel1 and { dup 1 lt {pop FrameCurGray} if } if RealSetgray } ifelse } ifelse } bind def /savematrix { orgmatrix currentmatrix pop } bind def /restorematrix { orgmatrix setmatrix } bind def /dmatrix matrix def /dpi 72 0 dmatrix defaultmatrix dtransform dup mul exch dup mul add sqrt def /freq dpi dup 72 div round dup 0 eq {pop 1} if 8 mul div def /sangle 1 0 dmatrix defaultmatrix dtransform exch atan def /dpiranges [ 2540 2400 1693 1270 1200 635 600 0 ] def /CMLowFreqs [ 100.402 94.8683 89.2289 100.402 94.8683 66.9349 63.2456 47.4342 ] def /YLowFreqs [ 95.25 90.0 84.65 95.25 90.0 70.5556 66.6667 50.0 ] def /KLowFreqs [ 89.8026 84.8528 79.8088 89.8026 84.8528 74.8355 70.7107 53.033 ] def /CLowAngles [ 71.5651 71.5651 71.5651 71.5651 71.5651 71.5651 71.5651 71.5651 ] def /MLowAngles [ 18.4349 18.4349 18.4349 18.4349 18.4349 18.4349 18.4349 18.4349 ] def /YLowTDot [ true true false true true false false false ] def /CMHighFreqs [ 133.87 126.491 133.843 108.503 102.523 100.402 94.8683 63.2456 ] def /YHighFreqs [ 127.0 120.0 126.975 115.455 109.091 95.25 90.0 60.0 ] def /KHighFreqs [ 119.737 113.137 119.713 128.289 121.218 89.8026 84.8528 63.6395 ] def /CHighAngles [ 71.5651 71.5651 71.5651 70.0169 70.0169 71.5651 71.5651 71.5651 ] def /MHighAngles [ 18.4349 18.4349 18.4349 19.9831 19.9831 18.4349 18.4349 18.4349 ] def /YHighTDot [ false false true false false true true false ] def /PatFreq [ 10.5833 10.0 9.4055 10.5833 10.0 10.5833 10.0 9.375 ] def /screenIndex { 0 1 dpiranges length 1 sub { dup dpiranges exch get 1 sub dpi le {exit} {pop} ifelse } for } bind def /getCyanScreen { FMUseHighFrequencyScreens { CHighAngles CMHighFreqs} {CLowAngles CMLowFreqs} ifelse screenIndex dup 3 1 roll get 3 1 roll get /FMSpotFunction load } bind def /getMagentaScreen { FMUseHighFrequencyScreens { MHighAngles CMHighFreqs } {MLowAngles CMLowFreqs} ifelse screenIndex dup 3 1 roll get 3 1 roll get /FMSpotFunction load } bind def /getYellowScreen { FMUseHighFrequencyScreens { YHighTDot YHighFreqs} { YLowTDot YLowFreqs } ifelse screenIndex dup 3 1 roll get 3 1 roll get { 3 div {2 { 1 add 2 div 3 mul dup floor sub 2 mul 1 sub exch} repeat FMSpotFunction } } {/FMSpotFunction load } ifelse 0.0 exch } bind def /getBlackScreen { FMUseHighFrequencyScreens { KHighFreqs } { KLowFreqs } ifelse screenIndex get 45.0 /FMSpotFunction load } bind def /getSpotScreen { getBlackScreen } bind def /getCompositeScreen { getBlackScreen } bind def /FMSetScreen FMLevel1 { /setscreen load }{ { 8 dict begin /HalftoneType 1 def /SpotFunction exch def /Angle exch def /Frequency exch def /AccurateScreens FMUseAcccurateScreens def currentdict end sethalftone } bind } ifelse def /setDefaultScreen { FMPColor { orgrxfer cvx orggxfer cvx orgbxfer cvx orgxfer cvx setcolortransfer } { orgxfer cvx settransfer } ifelse orgfreq organgle orgproc cvx setscreen } bind def /setCurrentScreen { FrameSepIs FMnone eq { FMUseDefaultNoSeparationScreen { setDefaultScreen } { getCompositeScreen FMSetScreen } ifelse } { FrameSepIs FMcustom eq { FMUseDefaultSpotSeparationScreen { setDefaultScreen } { getSpotScreen FMSetScreen } ifelse } { FMUseDefaultProcessSeparationScreen { setDefaultScreen } { FrameSepIs FMcyan eq { getCyanScreen FMSetScreen } { FrameSepIs FMmagenta eq { getMagentaScreen FMSetScreen } { FrameSepIs FMyellow eq { getYellowScreen FMSetScreen } { getBlackScreen FMSetScreen } ifelse } ifelse } ifelse } ifelse } ifelse } ifelse } bind def end /gstring FMLOCAL /gfile FMLOCAL /gindex FMLOCAL /orgrxfer FMLOCAL /orggxfer FMLOCAL /orgbxfer FMLOCAL /orgxfer FMLOCAL /orgproc FMLOCAL /orgrproc FMLOCAL /orggproc FMLOCAL /orgbproc FMLOCAL /organgle FMLOCAL /orgrangle FMLOCAL /orggangle FMLOCAL /orgbangle FMLOCAL /orgfreq FMLOCAL /orgrfreq FMLOCAL /orggfreq FMLOCAL /orgbfreq FMLOCAL /yscale FMLOCAL /xscale FMLOCAL /edown FMLOCAL /manualfeed FMLOCAL /paperheight FMLOCAL /paperwidth FMLOCAL /FMDOCUMENT { array /FMfonts exch def /#copies exch def FrameDict begin 0 ne /manualfeed exch def /paperheight exch def /paperwidth exch def 0 ne /FrameNegative exch def 0 ne /edown exch def /yscale exch def /xscale exch def FMLevel1 { manualfeed {setmanualfeed} if /FMdicttop countdictstack 1 add def /FMoptop count def setpapername manualfeed {true} {papersize} ifelse {manualpapersize} {false} ifelse {desperatepapersize} {false} ifelse { (Can't select requested paper size for Frame print job!) FMFAILURE } if count -1 FMoptop {pop pop} for countdictstack -1 FMdicttop {pop end} for } {{1 dict dup /PageSize [paperwidth paperheight]put setpagedevice}stopped { (Can't select requested paper size for Frame print job!) FMFAILURE } if {1 dict dup /ManualFeed manualfeed put setpagedevice } stopped pop } ifelse FMPColor { currentcolorscreen cvlit /orgproc exch def /organgle exch def /orgfreq exch def cvlit /orgbproc exch def /orgbangle exch def /orgbfreq exch def cvlit /orggproc exch def /orggangle exch def /orggfreq exch def cvlit /orgrproc exch def /orgrangle exch def /orgrfreq exch def currentcolortransfer FrameNegative { 1 1 4 { pop { 1 exch sub } concatprocs 4 1 roll } for 4 copy setcolortransfer } if cvlit /orgxfer exch def cvlit /orgbxfer exch def cvlit /orggxfer exch def cvlit /orgrxfer exch def } { currentscreen cvlit /orgproc exch def /organgle exch def /orgfreq exch def currenttransfer FrameNegative { { 1 exch sub } concatprocs dup settransfer } if cvlit /orgxfer exch def } ifelse end } def /pagesave FMLOCAL /orgmatrix FMLOCAL /landscape FMLOCAL /pwid FMLOCAL /FMBEGINPAGE { FrameDict begin /pagesave save def 3.86 setmiterlimit /landscape exch 0 ne def landscape { 90 rotate 0 exch dup /pwid exch def neg translate pop }{ pop /pwid exch def } ifelse edown { [-1 0 0 1 pwid 0] concat } if % 0 0 moveto paperwidth 0 lineto paperwidth paperheight lineto % 0 paperheight lineto 0 0 lineto 1 setgray fill xscale yscale scale /orgmatrix matrix def gsave } def /FMENDPAGE { grestore pagesave restore end showpage } def /FMFONTDEFINE { FrameDict begin findfont ReEncode 1 index exch definefont FMfonts 3 1 roll put end } def /FMFILLS { FrameDict begin dup array /fillvals exch def dict /patCache exch def end } def /FMFILL { FrameDict begin fillvals 3 1 roll put end } def /FMNORMALIZEGRAPHICS { newpath 0.0 0.0 moveto 1 setlinewidth 0 setlinecap 0 0 0 sethsbcolor 0 setgray } bind def /fx FMLOCAL /fy FMLOCAL /fh FMLOCAL /fw FMLOCAL /llx FMLOCAL /lly FMLOCAL /urx FMLOCAL /ury FMLOCAL /FMBEGINEPSF { end /FMEPSF save def /showpage {} def % See Adobe's "PostScript Language Reference Manual, 2nd Edition", page 714. % "...the following operators MUST NOT be used in an EPS file:" (emphasis ours) /banddevice {(banddevice) FMBADEPSF} def /clear {(clear) FMBADEPSF} def /cleardictstack {(cleardictstack) FMBADEPSF} def /copypage {(copypage) FMBADEPSF} def /erasepage {(erasepage) FMBADEPSF} def /exitserver {(exitserver) FMBADEPSF} def /framedevice {(framedevice) FMBADEPSF} def /grestoreall {(grestoreall) FMBADEPSF} def /initclip {(initclip) FMBADEPSF} def /initgraphics {(initgraphics) FMBADEPSF} def /initmatrix {(initmatrix) FMBADEPSF} def /quit {(quit) FMBADEPSF} def /renderbands {(renderbands) FMBADEPSF} def /setglobal {(setglobal) FMBADEPSF} def /setpagedevice {(setpagedevice) FMBADEPSF} def /setshared {(setshared) FMBADEPSF} def /startjob {(startjob) FMBADEPSF} def /lettertray {(lettertray) FMBADEPSF} def /letter {(letter) FMBADEPSF} def /lettersmall {(lettersmall) FMBADEPSF} def /11x17tray {(11x17tray) FMBADEPSF} def /11x17 {(11x17) FMBADEPSF} def /ledgertray {(ledgertray) FMBADEPSF} def /ledger {(ledger) FMBADEPSF} def /legaltray {(legaltray) FMBADEPSF} def /legal {(legal) FMBADEPSF} def /statementtray {(statementtray) FMBADEPSF} def /statement {(statement) FMBADEPSF} def /executivetray {(executivetray) FMBADEPSF} def /executive {(executive) FMBADEPSF} def /a3tray {(a3tray) FMBADEPSF} def /a3 {(a3) FMBADEPSF} def /a4tray {(a4tray) FMBADEPSF} def /a4 {(a4) FMBADEPSF} def /a4small {(a4small) FMBADEPSF} def /b4tray {(b4tray) FMBADEPSF} def /b4 {(b4) FMBADEPSF} def /b5tray {(b5tray) FMBADEPSF} def /b5 {(b5) FMBADEPSF} def FMNORMALIZEGRAPHICS [/fy /fx /fh /fw /ury /urx /lly /llx] {exch def} forall fx fw 2 div add fy fh 2 div add translate rotate fw 2 div neg fh 2 div neg translate fw urx llx sub div fh ury lly sub div scale llx neg lly neg translate /FMdicttop countdictstack 1 add def /FMoptop count def } bind def /FMENDEPSF { count -1 FMoptop {pop pop} for countdictstack -1 FMdicttop {pop end} for FMEPSF restore FrameDict begin } bind def FrameDict begin /setmanualfeed { %%BeginFeature *ManualFeed True statusdict /manualfeed true put %%EndFeature } bind def /max {2 copy lt {exch} if pop} bind def /min {2 copy gt {exch} if pop} bind def /inch {72 mul} def /pagedimen { paperheight sub abs 16 lt exch paperwidth sub abs 16 lt and {/papername exch def} {pop} ifelse } bind def /papersizedict FMLOCAL /setpapername { /papersizedict 14 dict def papersizedict begin /papername /unknown def /Letter 8.5 inch 11.0 inch pagedimen /LetterSmall 7.68 inch 10.16 inch pagedimen /Tabloid 11.0 inch 17.0 inch pagedimen /Ledger 17.0 inch 11.0 inch pagedimen /Legal 8.5 inch 14.0 inch pagedimen /Statement 5.5 inch 8.5 inch pagedimen /Executive 7.5 inch 10.0 inch pagedimen /A3 11.69 inch 16.5 inch pagedimen /A4 8.26 inch 11.69 inch pagedimen /A4Small 7.47 inch 10.85 inch pagedimen /B4 10.125 inch 14.33 inch pagedimen /B5 7.16 inch 10.125 inch pagedimen end } bind def /papersize { papersizedict begin /Letter {lettertray letter} def /LetterSmall {lettertray lettersmall} def /Tabloid {11x17tray 11x17} def /Ledger {ledgertray ledger} def /Legal {legaltray legal} def /Statement {statementtray statement} def /Executive {executivetray executive} def /A3 {a3tray a3} def /A4 {a4tray a4} def /A4Small {a4tray a4small} def /B4 {b4tray b4} def /B5 {b5tray b5} def /unknown {unknown} def papersizedict dup papername known {papername} {/unknown} ifelse get end statusdict begin stopped end } bind def /manualpapersize { papersizedict begin /Letter {letter} def /LetterSmall {lettersmall} def /Tabloid {11x17} def /Ledger {ledger} def /Legal {legal} def /Statement {statement} def /Executive {executive} def /A3 {a3} def /A4 {a4} def /A4Small {a4small} def /B4 {b4} def /B5 {b5} def /unknown {unknown} def papersizedict dup papername known {papername} {/unknown} ifelse get end stopped } bind def /desperatepapersize { statusdict /setpageparams known { paperwidth paperheight 0 1 statusdict begin {setpageparams} stopped end } {true} ifelse } bind def /DiacriticEncoding [ /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /space /exclam /quotedbl /numbersign /dollar /percent /ampersand /quotesingle /parenleft /parenright /asterisk /plus /comma /hyphen /period /slash /zero /one /two /three /four /five /six /seven /eight /nine /colon /semicolon /less /equal /greater /question /at /A /B /C /D /E /F /G /H /I /J /K /L /M /N /O /P /Q /R /S /T /U /V /W /X /Y /Z /bracketleft /backslash /bracketright /asciicircum /underscore /grave /a /b /c /d /e /f /g /h /i /j /k /l /m /n /o /p /q /r /s /t /u /v /w /x /y /z /braceleft /bar /braceright /asciitilde /.notdef /Adieresis /Aring /Ccedilla /Eacute /Ntilde /Odieresis /Udieresis /aacute /agrave /acircumflex /adieresis /atilde /aring /ccedilla /eacute /egrave /ecircumflex /edieresis /iacute /igrave /icircumflex /idieresis /ntilde /oacute /ograve /ocircumflex /odieresis /otilde /uacute /ugrave /ucircumflex /udieresis /dagger /.notdef /cent /sterling /section /bullet /paragraph /germandbls /registered /copyright /trademark /acute /dieresis /.notdef /AE /Oslash /.notdef /.notdef /.notdef /.notdef /yen /.notdef /.notdef /.notdef /.notdef /.notdef /.notdef /ordfeminine /ordmasculine /.notdef /ae /oslash /questiondown /exclamdown /logicalnot /.notdef /florin /.notdef /.notdef /guillemotleft /guillemotright /ellipsis /.notdef /Agrave /Atilde /Otilde /OE /oe /endash /emdash /quotedblleft /quotedblright /quoteleft /quoteright /.notdef /.notdef /ydieresis /Ydieresis /fraction /currency /guilsinglleft /guilsinglright /fi /fl /daggerdbl /periodcentered /quotesinglbase /quotedblbase /perthousand /Acircumflex /Ecircumflex /Aacute /Edieresis /Egrave /Iacute /Icircumflex /Idieresis /Igrave /Oacute /Ocircumflex /.notdef /Ograve /Uacute /Ucircumflex /Ugrave /dotlessi /circumflex /tilde /macron /breve /dotaccent /ring /cedilla /hungarumlaut /ogonek /caron ] def /ReEncode { dup length dict begin { 1 index /FID ne {def} {pop pop} ifelse } forall 0 eq {/Encoding DiacriticEncoding def} if currentdict end } bind def FMPColor { /BEGINBITMAPCOLOR { BITMAPCOLOR} def /BEGINBITMAPCOLORc { BITMAPCOLORc} def /BEGINBITMAPTRUECOLOR { BITMAPTRUECOLOR } def /BEGINBITMAPTRUECOLORc { BITMAPTRUECOLORc } def } { /BEGINBITMAPCOLOR { BITMAPGRAY} def /BEGINBITMAPCOLORc { BITMAPGRAYc} def /BEGINBITMAPTRUECOLOR { BITMAPTRUEGRAY } def /BEGINBITMAPTRUECOLORc { BITMAPTRUEGRAYc } def } ifelse /K { FMPrintAllColorsAsBlack { dup 1 eq 2 index 1 eq and 3 index 1 eq and not {7 {pop} repeat 0 0 0 1 0 0 0} if } if FrameCurColors astore pop combineColor } bind def /graymode true def /bwidth FMLOCAL /bpside FMLOCAL /bstring FMLOCAL /onbits FMLOCAL /offbits FMLOCAL /xindex FMLOCAL /yindex FMLOCAL /x FMLOCAL /y FMLOCAL /setPatternMode { FMLevel1 { /bwidth exch def /bpside exch def /bstring exch def /onbits 0 def /offbits 0 def freq sangle landscape {90 add} if {/y exch def /x exch def /xindex x 1 add 2 div bpside mul cvi def /yindex y 1 add 2 div bpside mul cvi def bstring yindex bwidth mul xindex 8 idiv add get 1 7 xindex 8 mod sub bitshift and 0 ne FrameNegative {not} if {/onbits onbits 1 add def 1} {/offbits offbits 1 add def 0} ifelse } setscreen offbits offbits onbits add div FrameNegative {1.0 exch sub} if /FrameCurGray exch def } { pop pop dup patCache exch known { patCache exch get } { dup patDict /bstring 3 -1 roll put patDict 9 PatFreq screenIndex get div dup matrix scale makepattern dup patCache 4 -1 roll 3 -1 roll put } ifelse /FrameCurGray 0 def /FrameCurPat exch def } ifelse /graymode false def combineColor } bind def /setGrayScaleMode { graymode not { /graymode true def FMLevel1 { setCurrentScreen } if } if /FrameCurGray exch def combineColor } bind def /normalize { transform round exch round exch itransform } bind def /dnormalize { dtransform round exch round exch idtransform } bind def /lnormalize { 0 dtransform exch cvi 2 idiv 2 mul 1 add exch idtransform pop } bind def /H { lnormalize setlinewidth } bind def /Z { setlinecap } bind def /PFill { graymode FMLevel1 or not { gsave 1 setgray eofill grestore } if } bind def /PStroke { graymode FMLevel1 or not { gsave 1 setgray stroke grestore } if stroke } bind def /fillvals FMLOCAL /X { fillvals exch get dup type /stringtype eq {8 1 setPatternMode} {setGrayScaleMode} ifelse } bind def /V { PFill gsave eofill grestore } bind def /Vclip { clip } bind def /Vstrk { currentlinewidth exch setlinewidth PStroke setlinewidth } bind def /N { PStroke } bind def /Nclip { strokepath clip newpath } bind def /Nstrk { currentlinewidth exch setlinewidth PStroke setlinewidth } bind def /M {newpath moveto} bind def /E {lineto} bind def /D {curveto} bind def /O {closepath} bind def /n FMLOCAL /L { /n exch def newpath normalize moveto 2 1 n {pop normalize lineto} for } bind def /Y { L closepath } bind def /x1 FMLOCAL /x2 FMLOCAL /y1 FMLOCAL /y2 FMLOCAL /R { /y2 exch def /x2 exch def /y1 exch def /x1 exch def x1 y1 x2 y1 x2 y2 x1 y2 4 Y } bind def /rad FMLOCAL /rarc {rad arcto } bind def /RR { /rad exch def normalize /y2 exch def /x2 exch def normalize /y1 exch def /x1 exch def mark newpath { x1 y1 rad add moveto x1 y2 x2 y2 rarc x2 y2 x2 y1 rarc x2 y1 x1 y1 rarc x1 y1 x1 y2 rarc closepath } stopped {x1 y1 x2 y2 R} if cleartomark } bind def /RRR { /rad exch def normalize /y4 exch def /x4 exch def normalize /y3 exch def /x3 exch def normalize /y2 exch def /x2 exch def normalize /y1 exch def /x1 exch def newpath normalize moveto mark { x2 y2 x3 y3 rarc x3 y3 x4 y4 rarc x4 y4 x1 y1 rarc x1 y1 x2 y2 rarc closepath } stopped {x1 y1 x2 y2 x3 y3 x4 y4 newpath moveto lineto lineto lineto closepath} if cleartomark } bind def /C { grestore gsave R clip setCurrentScreen } bind def /CP { grestore gsave Y clip setCurrentScreen } bind def /FMpointsize FMLOCAL /F { FMfonts exch get FMpointsize scalefont setfont } bind def /Q { /FMpointsize exch def F } bind def /T { moveto show } bind def /RF { rotate 0 ne {-1 1 scale} if } bind def /TF { gsave moveto RF show grestore } bind def /P { moveto 0 32 3 2 roll widthshow } bind def /PF { gsave moveto RF 0 32 3 2 roll widthshow grestore } bind def /S { moveto 0 exch ashow } bind def /SF { gsave moveto RF 0 exch ashow grestore } bind def /B { moveto 0 32 4 2 roll 0 exch awidthshow } bind def /BF { gsave moveto RF 0 32 4 2 roll 0 exch awidthshow grestore } bind def /G { gsave newpath normalize translate 0.0 0.0 moveto dnormalize scale 0.0 0.0 1.0 5 3 roll arc closepath PFill fill grestore } bind def /Gstrk { savematrix newpath 2 index 2 div add exch 3 index 2 div sub exch normalize 2 index 2 div sub exch 3 index 2 div add exch translate scale 0.0 0.0 1.0 5 3 roll arc restorematrix currentlinewidth exch setlinewidth PStroke setlinewidth } bind def /Gclip { newpath savematrix normalize translate 0.0 0.0 moveto dnormalize scale 0.0 0.0 1.0 5 3 roll arc closepath clip newpath restorematrix } bind def /GG { gsave newpath normalize translate 0.0 0.0 moveto rotate dnormalize scale 0.0 0.0 1.0 5 3 roll arc closepath PFill fill grestore } bind def /GGclip { savematrix newpath normalize translate 0.0 0.0 moveto rotate dnormalize scale 0.0 0.0 1.0 5 3 roll arc closepath clip newpath restorematrix } bind def /GGstrk { savematrix newpath normalize translate 0.0 0.0 moveto rotate dnormalize scale 0.0 0.0 1.0 5 3 roll arc closepath restorematrix currentlinewidth exch setlinewidth PStroke setlinewidth } bind def /A { gsave savematrix newpath 2 index 2 div add exch 3 index 2 div sub exch normalize 2 index 2 div sub exch 3 index 2 div add exch translate scale 0.0 0.0 1.0 5 3 roll arc restorematrix PStroke grestore } bind def /Aclip { newpath savematrix normalize translate 0.0 0.0 moveto dnormalize scale 0.0 0.0 1.0 5 3 roll arc closepath strokepath clip newpath restorematrix } bind def /Astrk { Gstrk } bind def /AA { gsave savematrix newpath 3 index 2 div add exch 4 index 2 div sub exch normalize 3 index 2 div sub exch 4 index 2 div add exch translate rotate scale 0.0 0.0 1.0 5 3 roll arc restorematrix PStroke grestore } bind def /AAclip { savematrix newpath normalize translate 0.0 0.0 moveto rotate dnormalize scale 0.0 0.0 1.0 5 3 roll arc closepath strokepath clip newpath restorematrix } bind def /AAstrk { GGstrk } bind def /x FMLOCAL /y FMLOCAL /w FMLOCAL /h FMLOCAL /xx FMLOCAL /yy FMLOCAL /ww FMLOCAL /hh FMLOCAL /FMsaveobject FMLOCAL /FMoptop FMLOCAL /FMdicttop FMLOCAL /BEGINPRINTCODE { /FMdicttop countdictstack 1 add def /FMoptop count 7 sub def /FMsaveobject save def userdict begin /showpage {} def FMNORMALIZEGRAPHICS 3 index neg 3 index neg translate } bind def /ENDPRINTCODE { count -1 FMoptop {pop pop} for countdictstack -1 FMdicttop {pop end} for FMsaveobject restore } bind def /gn { 0 { 46 mul cf read pop 32 sub dup 46 lt {exit} if 46 sub add } loop add } bind def /str FMLOCAL /cfs { /str sl string def 0 1 sl 1 sub {str exch val put} for str def } bind def /ic [ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0223 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0223 0 {0 hx} {1 hx} {2 hx} {3 hx} {4 hx} {5 hx} {6 hx} {7 hx} {8 hx} {9 hx} {10 hx} {11 hx} {12 hx} {13 hx} {14 hx} {15 hx} {16 hx} {17 hx} {18 hx} {19 hx} {gn hx} {0} {1} {2} {3} {4} {5} {6} {7} {8} {9} {10} {11} {12} {13} {14} {15} {16} {17} {18} {19} {gn} {0 wh} {1 wh} {2 wh} {3 wh} {4 wh} {5 wh} {6 wh} {7 wh} {8 wh} {9 wh} {10 wh} {11 wh} {12 wh} {13 wh} {14 wh} {gn wh} {0 bl} {1 bl} {2 bl} {3 bl} {4 bl} {5 bl} {6 bl} {7 bl} {8 bl} {9 bl} {10 bl} {11 bl} {12 bl} {13 bl} {14 bl} {gn bl} {0 fl} {1 fl} {2 fl} {3 fl} {4 fl} {5 fl} {6 fl} {7 fl} {8 fl} {9 fl} {10 fl} {11 fl} {12 fl} {13 fl} {14 fl} {gn fl} ] def /sl FMLOCAL /val FMLOCAL /ws FMLOCAL /im FMLOCAL /bs FMLOCAL /cs FMLOCAL /len FMLOCAL /pos FMLOCAL /ms { /sl exch def /val 255 def /ws cfs /im cfs /val 0 def /bs cfs /cs cfs } bind def 400 ms /ip { is 0 cf cs readline pop { ic exch get exec add } forall pop } bind def /rip { bis ris copy pop is 0 cf cs readline pop { ic exch get exec add } forall pop pop ris gis copy pop dup is exch cf cs readline pop { ic exch get exec add } forall pop pop gis bis copy pop dup add is exch cf cs readline pop { ic exch get exec add } forall pop } bind def /wh { /len exch def /pos exch def ws 0 len getinterval im pos len getinterval copy pop pos len } bind def /bl { /len exch def /pos exch def bs 0 len getinterval im pos len getinterval copy pop pos len } bind def /s1 1 string def /fl { /len exch def /pos exch def /val cf s1 readhexstring pop 0 get def pos 1 pos len add 1 sub {im exch val put} for pos len } bind def /hx { 3 copy getinterval cf exch readhexstring pop pop } bind def /h FMLOCAL /w FMLOCAL /d FMLOCAL /lb FMLOCAL /bitmapsave FMLOCAL /is FMLOCAL /cf FMLOCAL /wbytes { dup dup 24 eq { pop pop 3 mul } { 8 eq {pop} {1 eq {7 add 8 idiv} {3 add 4 idiv} ifelse} ifelse } ifelse } bind def /BEGINBITMAPBWc { 1 {} COMMONBITMAPc } bind def /BEGINBITMAPGRAYc { 8 {} COMMONBITMAPc } bind def /BEGINBITMAP2BITc { 2 {} COMMONBITMAPc } bind def /COMMONBITMAPc { /r exch def /d exch def gsave 3 index 2 div add exch 4 index 2 div add exch translate rotate 1 index 2 div neg 1 index 2 div neg translate scale /h exch def /w exch def /lb w d wbytes def sl lb lt {lb ms} if /bitmapsave save def r /is im 0 lb getinterval def ws 0 lb getinterval is copy pop /cf currentfile def w h d [w 0 0 h neg 0 h] {ip} image bitmapsave restore grestore } bind def /BEGINBITMAPBW { 1 {} COMMONBITMAP } bind def /BEGINBITMAPGRAY { 8 {} COMMONBITMAP } bind def /BEGINBITMAP2BIT { 2 {} COMMONBITMAP } bind def /COMMONBITMAP { /r exch def /d exch def gsave 3 index 2 div add exch 4 index 2 div add exch translate rotate 1 index 2 div neg 1 index 2 div neg translate scale /h exch def /w exch def /bitmapsave save def r /is w d wbytes string def /cf currentfile def w h d [w 0 0 h neg 0 h] {cf is readhexstring pop} image bitmapsave restore grestore } bind def /ngrayt 256 array def /nredt 256 array def /nbluet 256 array def /ngreent 256 array def /gryt FMLOCAL /blut FMLOCAL /grnt FMLOCAL /redt FMLOCAL /indx FMLOCAL /cynu FMLOCAL /magu FMLOCAL /yelu FMLOCAL /k FMLOCAL /u FMLOCAL FMLevel1 { /colorsetup { currentcolortransfer /gryt exch def /blut exch def /grnt exch def /redt exch def 0 1 255 { /indx exch def /cynu 1 red indx get 255 div sub def /magu 1 green indx get 255 div sub def /yelu 1 blue indx get 255 div sub def /k cynu magu min yelu min def /u k currentundercolorremoval exec def % /u 0 def nredt indx 1 0 cynu u sub max sub redt exec put ngreent indx 1 0 magu u sub max sub grnt exec put nbluet indx 1 0 yelu u sub max sub blut exec put ngrayt indx 1 k currentblackgeneration exec sub gryt exec put } for {255 mul cvi nredt exch get} {255 mul cvi ngreent exch get} {255 mul cvi nbluet exch get} {255 mul cvi ngrayt exch get} setcolortransfer {pop 0} setundercolorremoval {} setblackgeneration } bind def } { /colorSetup2 { [ /Indexed /DeviceRGB 255 {dup red exch get 255 div exch dup green exch get 255 div exch blue exch get 255 div} ] setcolorspace } bind def } ifelse /tran FMLOCAL /fakecolorsetup { /tran 256 string def 0 1 255 {/indx exch def tran indx red indx get 77 mul green indx get 151 mul blue indx get 28 mul add add 256 idiv put} for currenttransfer {255 mul cvi tran exch get 255.0 div} exch concatprocs settransfer } bind def /BITMAPCOLOR { /d 8 def gsave 3 index 2 div add exch 4 index 2 div add exch translate rotate 1 index 2 div neg 1 index 2 div neg translate scale /h exch def /w exch def /bitmapsave save def FMLevel1 { colorsetup /is w d wbytes string def /cf currentfile def w h d [w 0 0 h neg 0 h] {cf is readhexstring pop} {is} {is} true 3 colorimage } { colorSetup2 /is w d wbytes string def /cf currentfile def 7 dict dup begin /ImageType 1 def /Width w def /Height h def /ImageMatrix [w 0 0 h neg 0 h] def /DataSource {cf is readhexstring pop} bind def /BitsPerComponent d def /Decode [0 255] def end image } ifelse bitmapsave restore grestore } bind def /BITMAPCOLORc { /d 8 def gsave 3 index 2 div add exch 4 index 2 div add exch translate rotate 1 index 2 div neg 1 index 2 div neg translate scale /h exch def /w exch def /lb w d wbytes def sl lb lt {lb ms} if /bitmapsave save def FMLevel1 { colorsetup /is im 0 lb getinterval def ws 0 lb getinterval is copy pop /cf currentfile def w h d [w 0 0 h neg 0 h] {ip} {is} {is} true 3 colorimage } { colorSetup2 /is im 0 lb getinterval def ws 0 lb getinterval is copy pop /cf currentfile def 7 dict dup begin /ImageType 1 def /Width w def /Height h def /ImageMatrix [w 0 0 h neg 0 h] def /DataSource {ip} bind def /BitsPerComponent d def /Decode [0 255] def end image } ifelse bitmapsave restore grestore } bind def /BITMAPTRUECOLORc { /d 24 def gsave 3 index 2 div add exch 4 index 2 div add exch translate rotate 1 index 2 div neg 1 index 2 div neg translate scale /h exch def /w exch def /lb w d wbytes def sl lb lt {lb ms} if /bitmapsave save def /is im 0 lb getinterval def /ris im 0 w getinterval def /gis im w w getinterval def /bis im w 2 mul w getinterval def ws 0 lb getinterval is copy pop /cf currentfile def w h 8 [w 0 0 h neg 0 h] {w rip pop ris} {gis} {bis} true 3 colorimage bitmapsave restore grestore } bind def /BITMAPTRUECOLOR { gsave 3 index 2 div add exch 4 index 2 div add exch translate rotate 1 index 2 div neg 1 index 2 div neg translate scale /h exch def /w exch def /bitmapsave save def /is w string def /gis w string def /bis w string def /cf currentfile def w h 8 [w 0 0 h neg 0 h] { cf is readhexstring pop } { cf gis readhexstring pop } { cf bis readhexstring pop } true 3 colorimage bitmapsave restore grestore } bind def /BITMAPTRUEGRAYc { /d 24 def gsave 3 index 2 div add exch 4 index 2 div add exch translate rotate 1 index 2 div neg 1 index 2 div neg translate scale /h exch def /w exch def /lb w d wbytes def sl lb lt {lb ms} if /bitmapsave save def /is im 0 lb getinterval def /ris im 0 w getinterval def /gis im w w getinterval def /bis im w 2 mul w getinterval def ws 0 lb getinterval is copy pop /cf currentfile def w h 8 [w 0 0 h neg 0 h] {w rip pop ris gis bis w gray} image bitmapsave restore grestore } bind def /ww FMLOCAL /r FMLOCAL /g FMLOCAL /b FMLOCAL /i FMLOCAL /gray { /ww exch def /b exch def /g exch def /r exch def 0 1 ww 1 sub { /i exch def r i get .299 mul g i get .587 mul b i get .114 mul add add r i 3 -1 roll floor cvi put } for r } bind def /BITMAPTRUEGRAY { gsave 3 index 2 div add exch 4 index 2 div add exch translate rotate 1 index 2 div neg 1 index 2 div neg translate scale /h exch def /w exch def /bitmapsave save def /is w string def /gis w string def /bis w string def /cf currentfile def w h 8 [w 0 0 h neg 0 h] { cf is readhexstring pop cf gis readhexstring pop cf bis readhexstring pop w gray} image bitmapsave restore grestore } bind def /BITMAPGRAY { 8 {fakecolorsetup} COMMONBITMAP } bind def /BITMAPGRAYc { 8 {fakecolorsetup} COMMONBITMAPc } bind def /ENDBITMAP { } bind def end /ALDsave FMLOCAL /ALDmatrix matrix def ALDmatrix currentmatrix pop /StartALD { /ALDsave save def savematrix ALDmatrix setmatrix } bind def /InALD { restorematrix } bind def /DoneALD { ALDsave restore } bind def /I { setdash } bind def /J { [] 0 setdash } bind def %%EndProlog %%BeginSetup (4.0) FMVERSION 1 1 0 0 612 792 0 1 22 FMDOCUMENT 0 0 /Times-Roman FMFONTDEFINE 1 0 /Times-Bold FMFONTDEFINE 2 0 /Times-Italic FMFONTDEFINE 3 1 /Symbol FMFONTDEFINE 4 0 /Times-BoldItalic FMFONTDEFINE 32 FMFILLS 0 0 FMFILL 1 0.1 FMFILL 2 0.3 FMFILL 3 0.5 FMFILL 4 0.7 FMFILL 5 0.9 FMFILL 6 0.97 FMFILL 7 1 FMFILL 8 <0f1e3c78f0e1c387> FMFILL 9 <0f87c3e1f0783c1e> FMFILL 10 FMFILL 11 FMFILL 12 <8142241818244281> FMFILL 13 <03060c183060c081> FMFILL 14 <8040201008040201> FMFILL 16 1 FMFILL 17 0.9 FMFILL 18 0.7 FMFILL 19 0.5 FMFILL 20 0.3 FMFILL 21 0.1 FMFILL 22 0.03 FMFILL 23 0 FMFILL 24 FMFILL 25 FMFILL 26 <3333333333333333> FMFILL 27 <0000ffff0000ffff> FMFILL 28 <7ebddbe7e7dbbd7e> FMFILL 29 FMFILL 30 <7fbfdfeff7fbfdfe> FMFILL %%EndSetup %%Page: "31" 1 %%BeginPaperSize: Letter %%EndPaperSize 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K J 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (31) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 10 Q (Nilsson, Nils, 1986, Probabilistic Logic, Arti\336cial Intelligence 28: 71-88.) 108 713.33 T 1 F ([Ogasawara and Russell, 1993]) 108 697.33 T 0 F (Ogasawara, Gary H. and Russell, Stuart, 1993, Planning using multiple execution architectures, In Proceed-) 108 685.33 T (ings IJCAI 13, IJCAII.) 108 673.33 T 1 F ([Paz, 1971]) 108 657.33 T 0 F (Paz, Azaria, 1971, Introduction to Probabilistic Automata, Academic Press, New Y) 108 645.33 T (ork.) 440.56 645.33 T 1 F ([Pearl, 1988]) 108 629.33 T 0 F (Pearl, Judea, 1988, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Mor-) 108 617.33 T (gan-Kaufmann, Los Altos, California.) 108 605.33 T 1 F ([Pemberton and Korf, 1994]) 108 589.33 T 0 F (Pemberton, Joseph and Korf, Richard, 1994, An incremental search approach to real-time decision making,) 108 577.33 T (In Proceedings of the AAAI Spring Symposium on Decision-Theoretic Planning.) 108 565.33 T 1 F ([Peng and W) 108 549.33 T (illiams, 1993]) 163.38 549.33 T 0 F (Peng, Jing and W) 108 537.33 T (illiams, Ronald J., 1993, Ef) 178.15 537.33 T (\336cient Learning and Planning W) 287.42 537.33 T (ithin the Dyna Framework,) 417.56 537.33 T (Adaptive Behavior 1\0504\051:437-454.) 108 525.33 T 1 F ([Sacerdoti, 1974]) 108 509.33 T 0 F (Sacerdoti, Earl, 1974, Planning in a Hierarchy of Abstraction Spaces, Arti\336cial Intelligence 7:231-272.) 108 497.33 T 1 F ([Schweitzer) 108 481.33 T (, 1984]) 157.06 481.33 T 0 F (Schweitzer) 108 469.33 T (, Paul J., 1984, Aggregation methods for lar) 152.03 469.33 T (ge Markov chains, In G. Iazola, P) 327.11 469.33 T (. J. Coutois, and A.) 460.69 469.33 T (Hordijk, editors, Mathematical Computer Performance and Reliability) 108 457.33 T (, Elsevier Science Publishers.) 388.43 457.33 T 1 F ([Singh et al., 1994]) 108 441.33 T 0 F (Singh, Satinder; Jaakkola, T) 108 429.33 T (ommi; and Jordan, Michael, 1994, Model-Free Reinforcement Learning for) 220.35 429.33 T (Non-Markovian Decision Problems. Learning W) 108 417.33 T (ithout State-Estimation in Partially Observable Markovian) 303.4 417.33 T (Decision Processes, Proceedings of the Eleventh Machine Learning Conference.) 108 405.33 T 1 F ([Smallwood and Sondik, 1973]) 108 389.33 T 0 F (Smallwood, Richard D. and Sondik, Edward J., 1973, The optimal control of partially observable Markov) 108 377.33 T (processes over a \336nite horizon, Operations Research 21:1071-1088.) 108 365.33 T 1 F ([T) 108 349.33 T (ash and Russell, 1994]) 117.08 349.33 T 0 F (T) 108 337.33 T (ash, Jonathan and Russell, Stuart, 1994, Control strategies for a stochastic planner) 113.41 337.33 T (, In Proceedings AAAI-) 441.03 337.33 T (94, AAAI.) 108 325.33 T 1 F ([T) 108 309.33 T (atman and Shachter) 117.08 309.33 T (, 1990]) 202.83 309.33 T 0 F (T) 108 297.33 T (atman, Joseph A. and Shachter) 113.41 297.33 T (, Ross D., 1990, Dynamic programming and in\337uence diagrams, IEEE) 236.32 297.33 T (T) 108 285.33 T (ransactions on Systems, Man, and Cybernetics 20\0502\051:365-379.) 113.76 285.33 T 1 F ([Thiebaux et al., 1994]) 108 269.33 T 0 F (Thiebaux, Sylvie; Hertzber) 108 257.33 T (g, Joachim; Shoaf, W) 216.41 257.33 T (illiam; and Schneider) 302.39 257.33 T (, Moti, 1994, A stochastic model of) 387.54 257.33 T (actions and plans for anytime planning under uncertainty) 108 245.33 T (, In Sandewall, E. and Backstrom, C., editors, Cur-) 335.37 245.33 T (rent T) 108 233.33 T (rends in AI Planning. IOS Press, Amsterdam.) 131.81 233.33 T 1 F ([von W) 108 217.33 T (interfeldt and Edwards, 1986]) 139.21 217.33 T 0 F -0 (von W) 108 205.33 P -0 (interfeldt, Detlof and Edwards, W) 134.54 205.33 P -0 (ard, 1986, Decision Analysis and Behavioral Research, Cambridge) 269.8 205.33 P (University Press, New Y) 108 193.33 T (ork.) 206.43 193.33 T 1 F ([W) 108 177.33 T (ellman, 1987]) 120.78 177.33 T 0 F (W) 108 165.33 T (ellman, Michael P) 116.64 165.33 T (., 1987, Dominance and Subsumption in Constraint-Posting Planning, Proceedings) 188.58 165.33 T (IJCAI 10, Milan, Italy) 108 153.33 T (, IJCAII.) 196.51 153.33 T 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "31" 1 %%Page: "30" 2 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (30) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 10 Q (Howard, Ronald A., 1966, Information value theory) 108 713.33 T (, IEEE T) 315.65 713.33 T (ransactions on Systems Science and Cybernet-) 350.57 713.33 T (ics 2\0501\051:22-26.) 108 701.33 T 1 F ([Kabanza, 1990]) 108 685.33 T 0 F (Kabanza, F) 108 673.33 T (., 1990, Synthesis of reactive plans for multi-path environments, In Proceedings AAAI-90,) 152.74 673.33 T (AAAI, 164-169.) 108 661.33 T 1 F ([Kanazawa and Dean, 1989]) 108 645.33 T 0 F (Kanazawa, Keiji and Dean, Thomas, 1989, A model for projection and action, In Proceedings IJCAI 1) 108 633.33 T (1,) 516.99 633.33 T (IJCAII, 985-990.) 108 621.33 T 1 F ([Karlin and T) 108 605.33 T (aylor) 166.54 605.33 T (, 1975]) 187.84 605.33 T 0 F (Karlin, Samuel and T) 108 593.33 T (aylor) 193.4 593.33 T (, Howard M., 1975, A First Course in Stochastic Processes \050Second Edition\051,) 213.55 593.33 T (Academic Press, New Y) 108 581.33 T (ork.) 204.75 581.33 T 1 F ([Kemeny and Snell, 1960]) 108 565.33 T 0 F (Kemeny) 108 553.33 T (, John G. and Snell, J. Laurie, 1960, Finite Markov Chains, D. V) 141.23 553.33 T (an Nostrand, New Y) 398.44 553.33 T (ork.) 479.64 553.33 T 1 F ([Kirman et al., 1994]) 108 537.33 T 0 F (Kirman, Jak; Nicholson, Ann; Lejter) 108 525.33 T (, Moises; Dean, Thomas; and Santos Jr) 254.25 525.33 T (., Eugene, 1994, Using goals to) 409.8 525.33 T -0.25 (\336nd plans with high expected utility) 108 513.33 P -0.25 (, In Sandewall, E. and Backstrom, C., editors, Current T) 250.29 513.33 P -0.25 (rends in AI Plan-) 472.14 513.33 P (ning, Amsterdam, IOS Press.) 108 501.33 T 1 F ([Knoblock, 1991]) 108 485.33 T 0 F (Knoblock, Craig A., 1991, Search reduction in hierarchical problem solving, In Proceedings AAAI-91,) 108 473.33 T (AAAI, 686-691.) 108 461.33 T 1 F ([Korf, 1987]) 108 445.33 T 0 F (Korf, Richard, 1987, Planning as search: A quantitative approach, Arti\336cial Intelligence 33\0501\051:65-88.) 108 433.33 T 1 F ([Korf, 1988]) 108 417.33 T 0 F (Korf, Richard, 1988, Real-time heuristic search: New results, In Proceedings AAAI-88, AAAI,139-144.) 108 405.33 T 1 F ([Kushmerick et al., 1994]) 108 389.33 T 0 F (Kushmerick, Nicholas; Hanks, Steve; and W) 108 377.33 T (eld, Daniel, 1994, An algorithm for probabilistic planning, In) 286.34 377.33 T (Proceedings AAAI-94, AAAI.) 108 365.33 T 1 F ([Kushner and Kleinman, 1971]) 108 349.33 T 0 F (Kushner) 108 337.33 T (, Harold J. and Kleinman, A. J., 1971, Mathematical programming and the control of Markov) 141.48 337.33 T (chains, International Journal on Control 13\0505\051:801-820.) 108 325.33 T 1 F ([Lin and Dean, 1994]) 108 309.33 T 0 F (Lin, Shieu-Hong and Dean, Thomas, 1994, Exploiting locality in temporal reasoning, In Sandewall, E. and) 108 297.33 T (Backstrom, C., editors, Current T) 108 285.33 T (rends in AI Planning, Amsterdam, IOS Press.) 241.53 285.33 T 1 F ([Luenberger) 108 269.33 T (, 1973]) 160.96 269.33 T 0 F (Luenber) 108 257.33 T (ger) 141.14 257.33 T (, D. G., 1973, Introduction to Linear and Nonlinear Programming, Addison-W) 153.51 257.33 T (esley) 466.56 257.33 T (, Reading,) 486.46 257.33 T (Massachusetts.) 108 245.33 T 1 F ([Marsan et al., 1986]) 108 229.33 T 0 F -0.2 (Marsan, M. Ajmone; Balbo, G.; and Conte, G., 1986, Performance Models of Multiprocessor Systems, MIT) 108 217.33 P (Press, Cambridge, Massachusetts.) 108 205.33 T 1 F ([McAllester and Rosenblitt, 1991]) 108 189.33 T 0 F (McAllester) 108 177.33 T (, David A. and Rosenblitt, David, 1991, Systematic Nonlinear Planning, Proceedings AAAI-91,) 152.59 177.33 T (Anaheim, California, AAAI, 634-639.) 108 165.33 T 1 F ([Monahan, 1982]) 108 149.33 T 0 F (Monahan, Geor) 108 137.33 T (ge E., 1982, A survey of partially observable Markov decision processes: Theory) 170.58 137.33 T (, models,) 494.31 137.33 T (and algorithms, Management Science 28\0501\051:1-16.) 108 125.33 T 1 F ([Nicholson and Kaelbling, 1994]) 108 109.33 T 0 F (Nicholson, Ann and Kaelbling, Leslie Pack, 1994, T) 108 97.33 T (oward approximate planning in very lar) 317 97.33 T (ge stochastic) 475.4 97.33 T (domains, In Proceedings of the AAAI Spring Symposium on Decision-Theoretic Planning.) 108 85.33 T 1 F ([Nilsson, 1986]) 108 69.33 T 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "30" 2 %%Page: "29" 3 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (29) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 1 10 Q ([Dean et al., 1993b]) 108 713.33 T 0 F (Dean, Thomas; Kaelbling, Leslie; Kirman, Jak; and Nicholson, Ann, 1993, Deliberation Scheduling for) 108 701.33 T -0.41 (T) 108 689.33 P -0.41 (ime-Critical Sequential Decision Making, Proceedings of the Ninth Conference on Uncertainty in Arti\336cial) 113.76 689.33 P (Intelligence, W) 108 677.33 T (ashington, D.C., 309-316.) 168.85 677.33 T 1 F ([Derman, 1970]) 108 661.33 T 0 F (Derman, Cyrus, 1970, Finite State Markovian Decision Processes, Academic Press.) 108 649.33 T 1 F ([Draper et al., 1994]) 108 633.33 T 0 F (Draper) 108 621.33 T (, Denise; Hanks, Steve; and W) 135.36 621.33 T (eld, Daniel, 1994, Probabilistic planning with information gathering) 257.04 621.33 T (and contingent execution, In Proceedings of the AAAI Spring Symposium on Decision-Theoretic Planning.) 108 609.33 T 1 F ([Drummond and Br) 108 593.33 T (esina, 1990]) 193.38 593.33 T 0 F (Drummond, Mark and Bresina, John, 1990, Anytime synthetic projection: Maximizing the probability of) 108 581.33 T (goal satisfaction, In Proceedings AAAI-90, AAAI, 138-144.) 108 569.33 T 1 F ([Drummond et al., 1994]) 108 553.33 T 0 F (Drummond, Mark; Swanson, Keith; and Bresina, John, 1994, Robust scheduling and execution for auto-) 108 541.33 T (matic telescopes, In Zweben, M. and Fox, M., editors, Knowledge-Based Scheduling, Mor) 108 529.33 T (gan-Kaufmann,) 470.27 529.33 T (Los Altos, California.) 108 517.33 T 1 F ([Fedor) 108 501.33 T (ov) 136.7 501.33 T (, 1972]) 146.15 501.33 T 0 F (Fedorov) 108 489.33 T (, V) 140.68 489.33 T (., 1972, Theory of Optimal Experimental Design, Academic Press, New Y) 151.61 489.33 T (ork.) 448.06 489.33 T 1 F ([Feldman and Spr) 108 473.33 T (oull, 1977]) 185.61 473.33 T 0 F (Feldman, Jerome and Sproull, Robert, 1977, Decision theory and arti\336cial intelligence II: the hungry) 108 461.33 T (machine, Cognitive Science 1:158-192.) 108 449.33 T 1 F ([Fikes and Nilsson, 1971]) 108 433.33 T 0 F -0.27 (Fikes, Richard and Nilsson, Nils J., 1971, STRIPS: A new approach to the application of theorem proving to) 108 421.33 P (problem solving, Arti\336cial Intelligence 2:189-208.) 108 409.33 T 1 F ([Goldszmidt and Darwiche, 1994]) 108 393.33 T 0 F (Goldszmidt, Moises and Darwiche, Adnan, 1994, Action networks: A framework for reasoning about) 108 381.33 T (actions and change uncertainty) 108 369.33 T (, In Proceedings of the AAAI Spring Symposium on Decision-Theoretic) 230.93 369.33 T (Planning.) 108 357.33 T 1 F ([Gr) 108 341.33 T (eenwald and Dean, 1994]) 123.37 341.33 T 0 F (Greenwald, Lloyd and Dean, Thomas, 1994, Solving time-critical decision-making problems with predict-) 108 329.33 T (able computational demands, In Second International Conference on AI Planning Systems.) 108 317.33 T 1 F ([Gupta and Nau, 1991]) 108 301.33 T 0 F (Gupta, Naresh and Nau, Dana S., 1991, Complexity results for blocks-world planning, In Proceedings) 108 289.33 T (AAAI-91, AAAI, 629-633.) 108 277.33 T 1 F ([Haddawy and Suwandi, 1994]) 108 261.33 T 0 F (Haddawy) 108 249.33 T (, Peter and Suwandi, Meliani, 1994, Decision-theoretic planning using inheritance abstraction, In) 145.67 249.33 T (Proceedings of the AAAI Spring Symposium on Decision-Theoretic Planning.) 108 237.33 T 1 F ([Hanks and McDermott, 1994]) 108 221.33 T 0 F (Hanks, Steve and Drew V) 108 209.33 T (. McDermott, 1994, Modeling a Dynamic and Uncertain W) 210.85 209.33 T (orld I: Symbolic and) 447.51 209.33 T (Probabilistic Reasoning About Change, Arti\336cial Intelligence.) 108 197.33 T 1 F ([Howard and Matheson, 1984]) 108 181.33 T 0 F (Howard, Ronald A. and Matheson, James E., 1984, In\337uence diagrams, In Howard, Ronald A. and Mathe-) 108 169.33 T (son, James E., editors, The Principles and Applications of Decision Analysis, Strategic Decisions Group,) 108 157.33 T (Menlo Park, CA 94025.) 108 145.33 T 1 F ([Howard, 1960]) 108 129.33 T 0 F (Howard, Ronald A., 1960, Dynamic Programming and Markov Processes, MIT Press, Cambridge, Massa-) 108 117.33 T (chusetts.) 108 105.33 T 1 F ([Howard, 1966]) 108 89.33 T 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "29" 3 %%Page: "28" 4 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (28) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (It is worth noting that MDPs have provided foundations for other \336elds of study besides) 108 712 T -0.39 (decision-theoretic planning. In particular) 108 698 P -0.39 (, there is a sub\336eld of adaptive control theory that) 302.3 698 P (concerns MDPs [Bertsekas, 1987] and the study of reinforcement learning has pro\336ted) 108 684 T (enormously from recognizing a basis in MDPs [Barto et al., 1990].) 108 670 T 1 16 Q (10 Bibliography) 108 629.33 T 1 10 Q ([Barto et al., 1990]) 108 607.33 T 0 F (Barto, Andrew G.; Sutton, Richard S.; and W) 108 595.33 T (atkins, Christopher J. C. H., 1990, Learning and sequential) 289.13 595.33 T -0.14 (decision making, In Gabriel, Michael and Moore, John, editors, Learning and Computational Neuroscience:) 108 583.33 P (Foundations of Adaptive Networks, MIT Press, Cambridge, Massachusetts.) 108 571.33 T 1 F ([Barto et al., 1994]) 108 555.33 T 0 F -0.22 (Barto, Andrew G.; Bradtke, Steven J.; and Singh, Satinder P) 108 543.33 P -0.22 (., 1994, Learning to act using real-time dynamic) 346.53 543.33 P (programming, Arti\336cial Intelligence.) 108 531.33 T 1 F ([Bellman, 1957]) 108 515.33 T 0 F (Bellman, Richard, 1957, Dynamic Programming, Princeton University Press, Princeton, New Jersey) 108 503.33 T (.) 508.42 503.33 T 1 F ([Bellman, 1961]) 108 487.33 T 0 F (Bellman, Richard, 1961, Adaptive Control Processes, Princeton University Press, Princeton, New Jersey) 108 475.33 T (.) 525.36 475.33 T 1 F ([Bertsekas and Castanon, 1989]) 108 459.33 T 0 F (Bertsekas, D. P) 108 447.33 T (. and Castanon, D. A., 1989, Adaptive aggregation for in\336nite-horizon dynamic program-) 168.55 447.33 T (ming, IEEE T) 108 435.33 T (ransactions on Automatic Control 34\0506\051:589-598.) 163.48 435.33 T 1 F ([Bertsekas, 1987]) 108 419.33 T 0 F (Bertsekas, Dimitri P) 108 407.33 T (., 1987, Dynamic Programming, Prentice-Hall, Englewood Clif) 188.28 407.33 T (fs, N.J.) 441.69 407.33 T 1 F ([Boutilier and Dearden, 1994]) 108 391.33 T 0 F (Boutilier) 108 379.33 T (, Craig and Dearden, Richard, 1994, Using abstractions for decision-theoretic planning with time) 143.16 379.33 T (constraints, In Proceedings AAAI-94, AAAI.) 108 367.33 T 1 F ([Cassandra et al., 1994]) 108 351.33 T 0 F -0.25 (Cassandra, Anthony R.; Kaelbling, Leslie; and Littman, Michael, 1994, Acting optimally in partially observ-) 108 339.33 P (able stochastic domains, In Proceedings AAAI-94, AAAI.) 108 327.33 T 1 F ([Chavez and Henrion, 1994]) 108 311.33 T 0 F -0.34 (Chavez, T) 108 299.33 P -0.34 (om and Henrion, Max, 1994, Focusing on What Matters in Plan Evaluation: Ef) 148.06 299.33 P -0.34 (\336ciently Estimating) 459.57 299.33 P (the V) 108 287.33 T (alue of Information, Proceedings of the T) 128.83 287.33 T (enth Conference on Uncertainty in Arti\336cial Intelligence,) 293.93 287.33 T (Seattle, W) 108 275.33 T (ashington.) 148.86 275.33 T 1 F ([Darwiche and Pearl, 1994]) 108 259.33 T 0 F (Darwiche, Adnan and Pearl, Judea, 1994, Symbolic causal networks for reasoning about actions and plans,) 108 247.33 T (In Proceedings of the AAAI Spring Symposium on Decision-Theoretic Planning.) 108 235.33 T 1 F ([Davidson and Fehling, 1994]) 108 219.33 T 0 F (Davidson, Ron and Fehling, Michael, 1994, A structured, probabilistic model of action, In Proceedings of) 108 207.33 T (the AAAI Spring Symposium on Decision-Theoretic Planning.) 108 195.33 T 1 F ([Dean and Kanazawa, 1989]) 108 179.33 T 0 F (Dean, Thomas and Kanazawa, Keiji, 1989, A model for reasoning about persistence and causation, Compu-) 108 167.33 T (tational Intelligence 5\0503\051:142-150.) 108 155.33 T 1 F ([Dean and W) 108 139.33 T (ellman, 1991]) 164.12 139.33 T 0 F -0.06 (Dean, Thomas and W) 108 127.33 P -0.06 (ellman, Michael, 1991, Planning and Control, Mor) 194.2 127.33 P -0.06 (gan Kaufmann, San Mateo, Califor-) 396.4 127.33 P (nia.) 108 115.33 T 1 F ([Dean et al., 1993a]) 108 99.33 T 0 F (Dean, Thomas; Kaelbling, Leslie; Kirman, Jak; and Nicholson, Ann, 1993, Planning W) 108 87.33 T (ith Deadlines in Sto-) 457.29 87.33 T (chastic Domains, In Proceedings AAAI-93, W) 108 75.33 T (ashington, D.C., AAAI, 574-579.) 293.55 75.33 T 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "28" 4 %%Page: "27" 5 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (27) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (theory for choosing alternative representations or switching between representations on) 108 712 T (the basis of the current goal or problem instance. Manipulating dynamical models will) 108 698 T (require considerable care if we are to accurately assess the costs and bene\336ts of various) 108 684 T (representational options. The subtle interactions involving computation, observation, and) 108 670 T (prediction de\336ne a rich and lar) 108 656 T (gely unexplored area for future research.) 254.75 656 T 1 16 Q (9 Backgr) 108 615.33 T (ound) 173.49 615.33 T 0 12 Q (Kemeny and Snell [1960] provide a good basic introduction to \336nite Markov chains. T) 108 588 T (ay-) 523.12 588 T -0.35 (lor and Karlin [1975] describe the basic theory involved in stochastic modeling. Marsan et) 108 574 P (al. [1986] provide an introduction to stochastic modeling with an emphasis on applica-) 108 560 T -0.35 (tions to multiprocessor systems. Paz [1971] treatment of stochastic automata is also a very) 108 546 P (useful reference.) 108 532 T (Fikes and Nilsson [1971] introduce the classical model of planning in arti\336cial intelli-) 108 506 T (gence based on operators and search in state and goal space. Nau and Gupta [1989] pro-) 108 492 T (vide complexity results for a variety of classical planning problems.) 108 478 T (The early work of Bellman [1957] [1961] on sequential decision problems and dynamic) 108 452 T (programming \050speci\336cally value iteration\051 is still very readable. Howard [1960] intro-) 108 438 T -0.28 (duces the notion of Markov decision processes and describes policy iteration as an alterna-) 108 424 P (tive to value iteration for solving such problems. Bertsekas [1987] provides a more) 108 410 T -0.2 (modern and comprehensive treatment of dynamic programming with detailed coverage of) 108 396 P (solution methods. Derman [1970] and Kushner and Kleinman [1971] discuss the relation-) 108 382 T (ship between linear programming and Markov decision processes. Luenber) 108 368 T (ger [1973]) 468.71 368 T (describe notions of separability in linear and nonlinear programming.) 108 354 T -0.38 (Howard and Matheson [1984] describe in\337uence diagrams and their application to various) 108 328 P (decision problems. Pearl [1988] provides a comprehensive treatment of Bayesian net-) 108 314 T -0.44 (works which are a special case of in\337uence diagrams without decision variables. Dean and) 108 300 P (Kanazawa [1989] describe how Markov processes can be compactly represented as Baye-) 108 286 T -0.04 (sian networks. V) 108 272 P -0.04 (alue of information was introduced by Howard [1966], but the basic idea) 187.57 272 P -0.2 (has been around for much longer and is central to the \336eld of optimal experimental design) 108 258 P (\050see [Fedoras, 1972]\051. T) 108 244 T (atman and Shachter [1990] consider the notion of time-separable) 222.79 244 T (value in in\337uence diagrams.) 108 230 T (Schweitzer [1984] surveys aggregation techniques for handling lar) 108 204 T (ge Markov chains.) 427.38 204 T (Bertsekas and Castanon [1989] describe an aggregation method for expediting policy) 108 190 T (evaluation that adoptively aggregates and disaggregates states using estimates of the cur-) 108 176 T (rent value of states. Peng and W) 108 162 T (illiams [1993] and Dean et al. [1993a] use similar esti-) 262.49 162 T (mates to restrict attention to a subset of the set of all states.) 108 148 T -0.05 (Monahan [1982] provides a survey of techniques for solving partially observable Markov) 108 122 P (decision processes. Smallwood and Sondik [1973] introduced one of the most ef) 108 108 T (fective) 493.74 108 T (methods \050at that time\051 for solving POMDPs based upon solving linear programs; subse-) 108 94 T (quent improvements on their method are mentioned in Monahan [1982].) 108 80 T 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "27" 5 %%Page: "26" 6 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (26) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q -0.12 (tradeof) 108 712 P -0.12 (fs. In this section, we mention a couple of areas concerned with representation that) 141.77 712 P (seem particularly ripe for progress.) 108 698 T -0.15 (In a conversation following the 1994 Symposium on Decision-Theoretic Planning at Stan-) 108 672 P (ford University) 108 658 T (, one of the participants Daphne Koller remarked that there seems to be a) 180.88 658 T (lot of structure hidden in the way in which we encode our dynamical models. W) 108 644 T (e have) 491.98 644 T (pointed out in this paper that a great deal of the structure is apparent by examining the) 108 630 T (dependencies among state variables, but additional structure may be hidden in the choice) 108 616 T (of state variables and their associated sets of values. Suppose) 108 602 T ( is the state variable for) 413.95 602 T (location where) 108 588 T ( and that) 393.89 588 T ( is the state vari-) 451.21 588 T (able representing weather where) 108 574 T (. Now suppose further) 417.78 574 T (that you are planning to \337y from Boston to Chicago in mid January) 108 560 T (. It would certainly be) 429.88 560 T (advantageous from a computational standpoint if you could replace) 108 546 T ( and) 443.95 546 T ( for deci-) 480.27 546 T (sion-making purposes with something like) 108 532 T ( where) 330.01 532 T ( and) 502.61 532 T ( where) 126.46 518 T (. It is clear that by restricting our actions \050we need) 286.86 518 T -0.39 (not consider \337ying to Dallas or Atlanta\051 we limit the set of reachable states. It is somewhat) 108 504 P (less clear \050but ar) 108 490 T (guably true\051 that a distinction between snowing and not snowing is the) 186.76 490 T -0.19 (only factor critical in evaluating air travel plans between Boston and Chicago in mid Janu-) 108 476 P (ary) 108 462 T (. As Koller remarked, it does seems that there is a lot of useful structure that we have) 122.54 462 T -0.16 (yet to extract from our dynamical models. A general computational theory for \336nding and) 108 448 P (exploiting such structure has yet to be formulated.) 108 434 T (Another area of interest concerns the consequences of modifying) 108 408 T (a representation for a) 422.92 408 T (dynamical system in an attempt to gain some computational, observational, or predictive) 108 394 T (advantage. For example, it is possible to abstract a partially observable Markov decision) 108 380 T -0.11 (process and end up with a completely observable process. In Section) 108 366 P -0.11 (3, we introduced the) 439.47 366 P (variables) 108 352 T ( \050location\051,) 163.99 352 T ( \050battery level\051, and) 229.98 352 T ( \050illumination\051 to describe the state space) 333.94 352 T -0.31 (for a robot control problem. Suppose that the robot cannot observe its location directly but) 108 338 P -0.12 (can sense its battery level. In this case, the stochastic process for the state space) 108 324 P (is only partially observable; however) 108 310 T (, the process for the abstracted state space) 285.49 310 T ( is) 507.52 310 T (completely observable.) 108 296 T -0.49 (Related issues arise regarding the Markov property) 108 270 P -0.49 (. Suppose that the robot is attempting to) 349.57 270 P -0.17 (visit a \336xed sequence of locations and whether or not the robot arrives at the next location) 108 256 P (in the sequence depends on its battery level, which is completely determined by the loca-) 108 242 T (tions visited since the last battery char) 108 228 T (ging. Suppose further that one location is the char) 291.1 228 T (g-) 529.85 228 T (ing station and whenever the robot visits the char) 108 214 T (ging station it leaves with its battery) 343.75 214 T (completely char) 108 200 T (ged and no other location af) 184.76 200 T (fects the robot\325) 318.85 200 T (s battery level. Now suppose) 390.84 200 T (that the robot can observe both its location and battery level, and suppose further that the) 108 186 T -0.29 (stochastic process for the state space) 108 172 P -0.29 ( has the Markov property) 330.83 172 P -0.29 (. Note that the pro-) 450.19 172 P (cess for the abstracted state space) 108 158 T ( does not have the Markov property) 289.9 158 T (, and that an) 460.42 158 T (indeterminate number of past states may be required to predict with accuracy whether or) 108 144 T (not the robot will make it to the next location in the sequence.) 108 130 T (In Section) 108 104 T (3, we considered some alternative methods of representing the state spaces for) 160 104 T (dynamical systems. The literature is full of techniques for generating, evaluating, and) 108 90 T (making use of specialized representations, but there is as yet no coherent computational) 108 76 T 404.28 597 413.95 612 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (L) 405.28 602 T 0 0 612 792 C 181.98 579.55 393.89 598 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 183.11 588 T 2 9 Q (L) 192.78 583.8 T 0 12 Q (B) 225.46 588 T (o) 233.46 588 T (s) 239.46 588 T (t) 244.13 588 T (o) 247.47 588 T (n) 253.47 588 T (C) 265.46 588 T (h) 273.47 588 T (i) 279.47 588 T (c) 282.8 588 T (a) 288.13 588 T (g) 293.46 588 T (o) 299.46 588 T (D) 311.46 588 T (a) 320.12 588 T (l) 325.45 588 T (l) 328.79 588 T (a) 332.12 588 T (s) 337.45 588 T (A) 348.12 588 T (t) 356.78 588 T (l) 360.12 588 T (a) 363.45 588 T (n) 368.78 588 T (t) 374.78 588 T (a) 378.12 588 T 3 F (,) 259.47 588 T (,) 305.46 588 T (,) 342.12 588 T ({) 218.55 588 T (}) 384 588 T 0 F (=) 203.78 588 T 0 0 612 792 C 438.22 583 451.21 598 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (W) 439.21 588 T 0 0 612 792 C 266.6 565.55 417.78 584 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 267.79 574 T 2 9 Q (W) 277.46 569.8 T 0 12 Q (s) 312.63 574 T (n) 317.3 574 T (o) 323.3 574 T (w) 329.3 574 T (f) 343.96 574 T (o) 347.96 574 T (g) 353.96 574 T (r) 365.95 574 T (a) 369.95 574 T (i) 375.28 574 T (n) 378.61 574 T (s) 390.61 574 T (u) 395.28 574 T (n) 401.28 574 T 3 F (,) 337.96 574 T (,) 359.96 574 T (,) 384.61 574 T ({) 305.72 574 T (}) 407.83 574 T 0 F (=) 290.95 574 T 0 0 612 792 C 434.28 541 443.95 556 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (L) 435.28 546 T 0 0 612 792 C 467.28 541 480.27 556 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (W) 468.28 546 T 0 0 612 792 C 315 523.55 330.01 542 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (L) 316.19 532 T 0 9 Q (1) 323.32 527.8 T 0 0 612 792 C 365.33 521.75 502.61 542 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 366.46 532 T 2 9 Q (L) 376.14 527.8 T 0 6 Q (1) 381.48 525.25 T 0 12 Q (B) 412.16 532 T (o) 420.16 532 T (s) 426.16 532 T (t) 430.83 532 T (o) 434.17 532 T (n) 440.17 532 T (C) 452.16 532 T (h) 460.17 532 T (i) 466.17 532 T (c) 469.5 532 T (a) 474.83 532 T (g) 480.16 532 T (o) 486.16 532 T 3 F (,) 446.17 532 T ({) 405.25 532 T (}) 492.71 532 T 0 F (=) 390.48 532 T 0 0 612 792 C 108 509.55 126.46 528 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (W) 109.25 518 T 0 9 Q (1) 119.7 513.8 T 0 0 612 792 C 161.77 507.75 286.86 528 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 162.96 518 T 2 9 Q (W) 172.63 513.8 T 0 6 Q (1) 180.47 511.25 T 0 12 Q (s) 211.14 518 T (n) 215.81 518 T (o) 221.81 518 T (w) 227.81 518 T (s) 251.03 518 T (n) 255.7 518 T (o) 261.7 518 T (w) 267.7 518 T 3 F (\330) 242.47 518 T (,) 236.47 518 T ({) 204.23 518 T (}) 276.91 518 T 0 F (=) 189.46 518 T 0 0 612 792 C 154.32 347 163.99 362 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (L) 155.32 352 T 0 0 612 792 C 219.65 347 229.98 362 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (B) 220.65 352 T 0 0 612 792 C 326.95 347 333.94 362 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (I) 327.95 352 T 0 0 612 792 C 491.15 315.55 537.12 334 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 492.42 324 T 2 9 Q (L) 502.09 319.8 T 3 12 Q (W) 519.68 324 T 2 9 Q (B) 529.35 319.8 T 3 12 Q (\264) 510.09 324 T 0 0 612 792 C 489.1 301.55 507.52 320 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 490.22 310 T 2 9 Q (B) 499.89 305.8 T 0 0 612 792 C 284.86 163.55 330.83 182 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 286.13 172 T 2 9 Q (L) 295.8 167.8 T 3 12 Q (W) 313.39 172 T 2 9 Q (B) 323.06 167.8 T 3 12 Q (\264) 303.8 172 T 0 0 612 792 C 271.94 149.55 289.9 168 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 273.09 158 T 2 9 Q (L) 282.76 153.8 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "26" 6 %%Page: "25" 7 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (25) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (Korf [1988] developed an approach called) 108 712 T 2 F (r) 313.93 712 T (eal-time heuristic sear) 318.16 712 T (ch) 425.7 712 T 0 F ( that addresses the) 437.03 712 T (issues of searching under time pressure. Pemberton and Korf [1994] describe how this) 108 698 T (approach might be extended to real-time decision making. Barto, Bradtke and Singh) 108 684 T ([1994] describe a more general approach called) 108 670 T 2 F (r) 338.92 670 T (eal-time dynamic pr) 343.14 670 T (ogramming) 439.34 670 T 0 F ( that) 494.68 670 T -0.51 (directly addresses the problem of solving Markov decision processes in a real-time setting.) 108 656 P (The learning variant of real-time heuristic search is the deterministic special case of real-) 109 642 T (time dynamic programming.) 108 628 T (The idea of approximating solutions to an in\336nite-horizon problem by solving a \336nite-) 108 602 T -0.42 (horizon problem has been around in adaptive control [Bellman, 1961] for quite some time.) 108 588 P (Kanazawa and Dean [1989] and Ogasawara and Russell [1993] consider some of the) 108 574 T (tradeof) 108 560 T (fs involved in restricting the planning horizon for in\336nite-horizon problems. Since) 141.77 560 T (decision models used for planning vary over time, the graphical models used in these) 108 546 T (approaches are often called) 108 532 T 2 F (dynamic in\337uence diagrams) 241.94 532 T 0 F (. T) 377.26 532 T (atman and Shachter [1990]) 389.75 532 T -0.14 (describe how time separability can be used to speed inference in solving Markov decision) 108 518 P (processes.) 108 504 T -0.33 (Partially observable Markov decision processes open up a whole set of issues that we only) 108 478 P (touch upon. One of the classic approaches to solving such problems is due to Smallwood) 108 464 T (and Sondik [1973]. Cassandra et al. [1994] improve upon Smallwood and Sondik\325) 108 450 T (s basic) 502.31 450 T -0.07 (algorithms and demonstrate how this method might be used to generate very compact rep-) 108 436 P (resentations of policies for partially observable problems. Draper et al. [1994] describe) 108 422 T -0.46 (how extensions of classical planning can be used to address partially observable problems.) 108 408 P (The Draper et al. work builds on the BURIDAN planner described earlier) 108 394 T (.) 460.58 394 T (W) 108 368 T (e focus in this paper on representations for Markov decision processes, but we men-) 118.37 368 T (tioned at the outset that such representations have limited expressive power) 108 354 T (. Much of the) 469.28 354 T (literature on temporal reasoning in arti\336cial intelligence deals with the problems of repre-) 108 340 T (senting the complex ef) 108 326 T (fects of actions \050the) 216.77 326 T 2 F (rami\336cation) 314.08 326 T 0 F ( problem\051, reasoning about what) 372.74 326 T -0.3 (propositions do not change as a consequence of acting \050the) 108 312 P 2 F -0.3 (frame) 390.24 312 P 0 F -0.3 ( problem\051, and reasoning) 418.24 312 P (about what preconditions are required for an action to have a particular ef) 108 298 T (fect \050the) 461.02 298 T 2 F (quali\336-) 503.66 298 T (cation) 108 284 T 0 F ( problem\051. These problems are beyond the scope of this paper) 138 284 T (, but see [Goldszmidt) 433.79 284 T (and Darwiche, 1994] and [Darwiche and Pearl, 1994] for some recent work addressing) 108 270 T (these problems in the context of the factored state transition models discussed in this) 108 256 T (paper) 108 242 T (. Kabanza [1990] describes an approach to dealing with uncertainty in the form of) 133.99 242 T (actions with disjunctive ef) 108 228 T (fects represented in a temporal logic. Thiebaux et al. [1994]) 234.11 228 T (employ a framework much like Dean et al. [1993a] but use probabilistic logic [Nilsson,) 108 214 T -0.52 (1986] to represent actions. These more expressive languages present new opportunities for) 108 200 P (modeling dynamical systems and new challenges to ef) 108 186 T (\336cient decision-theoretic planning.) 368.75 186 T 1 16 Q (8 Further Resear) 108 145.33 T (ch) 229.01 145.33 T 0 12 Q (Despite decades of research on Markov decision processes, we are just beginning to) 108 118 T (understand the issues involved in solving the underlying computational problems. The) 108 104 T (\337edgling \336eld of decision-theoretic planning brings to the table an alternative perspective) 108 90 T (focusing on representation and an alternative set of tools focusing on computational) 108 76 T 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "25" 7 %%Page: "24" 8 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (24) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (The Dean et al. approach borrows from and extends the approach of Drummond and) 108 712 T (Bresina to handle arbitrary Markov chains and value of information calculations to guide) 108 698 T (policy improvement. Dean et al. construct a restricted stochastic automaton de\336ned on a) 108 684 T (subset of) 108 670 T ( and use a variant of policy iteration [Howard, 1960] to compute an optimal) 172.77 670 T -0.46 (policy with respect to the restricted automaton. Execution is allowed to occur concurrently) 108 656 P (with planning and the restricted automaton is modi\336ed over time to suit the current state,) 108 642 T (goal, and policy) 108 628 T (. Decisions as to how to modify the restricted automaton are made on the) 184.21 628 T (basis of expected improvements in the resulting policy) 108 614 T (. Greenwald and Dean [1994]) 369.53 614 T -0.29 (describe how this approach can be extended to handle scheduling problems. T) 108 600 P -0.29 (ash and Rus-) 478.58 600 P (sell [1994] also modify a restricted automaton iteratively but use a dif) 108 586 T (ferent strategy for) 442.74 586 T (determine which states to add to the automaton on each iteration.) 108 572 T -0.17 (Dearden and Boutilier [1994] provide an alternative to the approach of Dean et al. Instead) 108 546 P (of simply restricting attention to a subset of) 108 532 T (, Dearden and Boutilier generate an) 338.76 532 T (abstract state space corresponding to a partition based on a subset of the set of all state) 108 518 T (variables. If) 108 504 T ( is the set of state variables \050boolean variables corresponding propo-) 210.72 504 T (sitions that are either true or false of a state\051, then the subset) 108 490 T ( induces a partition on) 427.03 490 T (the set of all states. From this abstract state space, they generate an abstract Markov deci-) 108 476 T (sion process which they can solve for the optimal policy using any of the standard meth-) 108 462 T -0.29 (ods. T) 108 448 P -0.29 (o assess solution-quality/time-criticality tradeof) 136.87 448 P -0.29 (fs, Dearden and Boutilier are able to) 364.76 448 P (estimate the dif) 108 434 T (ference between the expected utility of an optimal policy generated from) 181.78 434 T (the abstract Markov decision process and the expected utility of an optimal policy gener-) 108 420 T (ated from the original process.) 108 406 T (The abstractions used in Dearden and Boutilier [1994] are closely related to methods) 108 380 T (described by Sacerdoti [1974], Korf [1987], Knoblock [1991], and Lin and Dean [1994]) 108 366 T -0.46 (for planning and temporal reasoning in deterministic environments. There is a long history) 108 352 P -0.18 (of using abstractions in both arti\336cial intelligence and disciplines such as adaptive control) 108 338 P (that deal with Markov decision processes. Davidson and Fehling [1994] and Nicholson) 108 324 T (and Kaelbling [1994] provide examples of approaches that analyze factored state transi-) 108 310 T (tion models to generate abstract representations that serve to expedite planning and deci-) 108 296 T (sion making. Haddawy and Suwandi [1994] construct abstract models by or) 108 282 T (ganizing) 472.07 282 T (actions in a hierarchy of abstractions.) 108 268 T -0.09 (For a survey of abstraction methods \050called) 108 242 P 2 F -0.09 (aggr) 318.33 242 P -0.09 (egation) 340.56 242 P 0 F -0.09 (methods\051 in adaptive control and) 379.47 242 P -0.28 (operations research see [Schweitzer) 108 228 P -0.28 (, 1984]. Bertsekas and Castanon [1989] describe a spe-) 278.29 228 P (ci\336c aggregation technique for expediting the process of evaluating policies for Markov) 108 214 T (decision processes with very lar) 108 200 T (ge state spaces. Assessing the tradeof) 261.08 200 T (fs involved in vari-) 440.16 200 T (ous abstractions can be complicated. W) 108 186 T (e already mentioned that Dearden and Boutilier) 297 186 T (estimate the costs and bene\336ts of their abstractions. Nicholson and Kaelbling use a) 108 172 T (method of evaluation that accounts for the state transition probabilities but not the conse-) 108 158 T (quences of acting. Ideally one could use the notion of value of information [Howard,) 108 144 T (1966] to exactly calculate the cost of adopting an abstract model, but computationally) 108 130 T -0.3 (such exact calculations are infeasible. Chavez and Henrion [1994] describe a Monte Carlo) 108 116 P (method for estimating the value of information for planning abstractions.) 108 102 T 154 661.55 172.77 680 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 155.3 670 T 2 9 Q (X) 164.97 665.8 T 0 0 612 792 C 319.99 523.55 338.76 542 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 321.29 532 T 2 9 Q (X) 330.96 527.8 T 0 0 612 792 C 168.31 499 210.72 514 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 178.32 504 T (Q) 191.65 504 T 3 F (,) 185.65 504 T ({) 171.41 504 T (}) 200.87 504 T 0 0 612 792 C 399.28 485 427.03 500 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 409.29 490 T 3 F ({) 402.37 490 T (}) 417.17 490 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "24" 8 %%Page: "23" 9 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (23) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (immediately following) 108 712 T (. If a proposition is not included in) 231.24 712 T (, then it is assumed not to) 410.47 712 T (af) 108 698 T (fect the outcome of) 117.11 698 T (; if a proposition is not included in) 223.65 698 T (, then it is assumed to be) 403.56 698 T (unchanged by) 108 684 T (. For example, given the following representation for) 188.55 684 T 0 10 Q (\05028\051) 523.34 658 T 0 12 Q (if) 108 633 T ( is true prior to) 128.66 633 T (, nothing is changed following) 213.56 633 T (, but if) 373.46 633 T ( is false prior to) 418.46 633 T (, then) 507.35 633 T (20% of the time) 108 619 T ( is true and 80% of the time) 198.32 619 T ( is false following) 344.64 619 T (. Alternatively) 444.53 619 T (,) 513.08 619 T (given) 108 605 T 0 10 Q (\05029\051) 523.34 579 T 0 12 Q (If) 108 554 T ( is false prior to) 129.32 554 T (, then) 218.22 554 T ( is true immediately after) 258.21 554 T (, but if) 378.38 554 T ( is true prior to) 423.38 554 T (, then) 508.27 554 T (whether or not) 108 540 T ( is true immediately after depends on) 191.31 540 T (. The BURIDAN action repre-) 383.95 540 T (sentation \050see Hanks and McDermott [1994] for details\051 can be used to model arbitrary) 108 526 T -0.47 (Markov chains. In the current implementation, each action is represented as a decision tree) 108 512 P (but it would be easy enough to represent actions using factored state-transition models as) 108 498 T (described earlier) 108 484 T (.) 186.97 484 T -0.08 (BURIDAN searches for candidate plans to evaluate. Plan evaluation consists of assessing) 108 458 P -0.15 (the probability that the plan will achieve the goal. If the probability for a candidate plan is) 108 444 P -0.23 (greater than the threshold, then BURIDAN exits returning the plan, otherwise it continues) 108 430 P (searching. Plan evaluation is a form of probabilistic prediction and can be done using the) 108 416 T (sort of dynamical models described in earlier sections. BURIDAN actually generates a) 108 402 T (partially ordered sequence of actions, but for evaluation purposes all total orders consis-) 108 388 T (tent with the partial order are equivalent. Kirman et al. [1994] describe a method of plan) 108 374 T (evaluation using factored state transition models designed for time-critical applications.) 108 360 T (Drummond and Bresina [1990] have developed an approach for embedded planning that) 108 334 T (searches for closed-loop solutions \050policies\051 maximizing the probability of achieving the) 108 320 T -0.5 (goal. Their approach, called) 108 306 P 2 F -0.5 (anytime synthetic pr) 243.3 306 P -0.5 (ojection) 339.18 306 P 0 F -0.5 (, incrementally improves a partial) 377.85 306 P -0.15 (policy \050a) 108 292 P 2 F -0.15 (partial policy) 153.02 292 P 0 F -0.15 ( is a policy de\336ned on a subset of) 217.87 292 P -0.15 (\051 by \336rst generating an initial) 398.26 292 P (\050degenerate\051 policy corresponding to a sequence of actions that will reach the goal with) 108 278 T (some probability \050perhaps small\051 and then improving the policy by identifying states not) 108 264 T (covered by the partial policy but likely to be encountered during execution. The Drum-) 108 250 T -0.07 (mond and Bresina work was one of the \336rst to directly address the tradeof) 108 236 P -0.07 (fs involving the) 461.86 236 P -0.16 (quality of the constructed policy and the cost of delay) 108 222 P -0.16 (. They have extended their approach) 363.37 222 P (to handling scheduling problems in [Drummond et al., 1994].) 108 208 T (Dean et al. [1993a, 1993b] describe an approach to \336nding a closed-loop solution mini-) 108 182 T (mizing expected time to goal for embedded planning applications. W) 108 168 T (e mentioned in) 439 168 T (Section) 108 154 T (6.2 that one model for in\336nite-horizon problems is to maximize the expected) 147 154 T (cumulative value until a goal state is reached. A simple variant of this model is to assign) 108 140 T ( to any non-goal state and) 123.46 126 T ( to any goal state and assume that every goal state is an) 259.1 126 T (absorbing state \050see Section) 108 112 T (4\051. Let) 244.64 112 T ( be a random variable representing the time that the) 287.9 112 T (system \336rst achieves a goal state for a \336xed policy) 108 98 T (. Dean et al. address a class of in\336nite-) 348.85 98 T (horizon problems in which the objective is to construct a policy maximizing) 108 84 T (.) 503.23 84 T 220.67 707 231.24 722 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 221.67 712 T 0 0 612 792 C 400.23 707 410.47 722 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (j) 401.23 712 T 0 0 612 792 C 213.08 693 223.65 708 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 214.08 698 T 0 0 612 792 C 392.33 693 403.56 708 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (w) 393.33 698 T 0 0 612 792 C 177.98 679 188.55 694 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 178.98 684 T 0 0 612 792 C 445.85 679 456.42 694 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 446.85 684 T 0 0 612 792 C 153.93 653 477.41 668 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 155.39 658 T 2 F (P) 206.12 658 T 3 F ({) 199.21 658 T (}) 214 658 T 0 F (1) 227.76 658 T 3 F (\306) 239.76 658 T (,) 221.76 658 T (,) 233.76 658 T (\341) 191.84 658 T (\361) 249.64 658 T 2 F (P) 284.33 658 T 3 F (\330) 275.77 658 T ({) 268.86 658 T (}) 292.21 658 T 0 F (0) 305.97 658 T (.) 311.97 658 T (2) 314.97 658 T 2 F (P) 335.88 658 T 3 F ({) 328.97 658 T (}) 343.77 658 T (,) 299.97 658 T (,) 320.97 658 T (\341) 261.49 658 T (\361) 351.53 658 T 2 F (P) 386.22 658 T 3 F (\330) 377.66 658 T ({) 370.75 658 T (}) 394.1 658 T 0 F (0) 407.86 658 T (.) 413.86 658 T (8) 416.86 658 T 2 F (P) 446.33 658 T 3 F (\330) 437.77 658 T ({) 430.86 658 T (}) 454.21 658 T (,) 401.86 658 T (,) 422.86 658 T (\341) 363.38 658 T (\361) 461.98 658 T (,) 254.29 658 T (,) 356.18 658 T ({) 183.73 658 T (}) 467.18 658 T 0 F (=) 168.96 658 T 0 0 612 792 C 118.33 628 128.66 643 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 119.33 633 T 0 0 612 792 C 202.99 628 213.56 643 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 203.99 633 T 0 0 612 792 C 362.89 628 373.46 643 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 363.89 633 T 0 0 612 792 C 408.13 628 418.46 643 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 409.13 633 T 0 0 612 792 C 496.78 628 507.35 643 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 497.78 633 T 0 0 612 792 C 187.99 614 198.32 629 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 188.99 619 T 0 0 612 792 C 334.31 614 344.64 629 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 335.3 619 T 0 0 612 792 C 433.96 614 444.53 629 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 434.96 619 T 0 0 612 792 C 140.83 574 490.51 589 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 142.39 579 T 2 F (P) 193.11 579 T (Q) 206.44 579 T 3 F (,) 200.45 579 T ({) 186.2 579 T (}) 215.66 579 T 0 F (1) 229.42 579 T 2 F (P) 250.33 579 T 3 F ({) 243.42 579 T (}) 258.22 579 T (,) 223.42 579 T (,) 235.42 579 T (\341) 178.84 579 T (\361) 265.98 579 T 2 F (P) 292.11 579 T (Q) 314 579 T 3 F (\330) 305.44 579 T (,) 299.44 579 T ({) 285.2 579 T (}) 323.21 579 T 0 F (1) 336.98 579 T 2 F (P) 366.44 579 T 3 F (\330) 357.89 579 T ({) 350.98 579 T (}) 374.33 579 T (,) 330.98 579 T (,) 342.98 579 T (\341) 277.83 579 T (\361) 382.09 579 T 2 F (P) 416.78 579 T 3 F (\330) 408.22 579 T ({) 401.31 579 T (}) 424.66 579 T 0 F (0) 438.42 579 T 3 F (,) 432.42 579 T 2 F (P) 459.33 579 T 3 F ({) 452.42 579 T (}) 467.22 579 T (,) 444.42 579 T (\341) 393.94 579 T (\361) 474.98 579 T (,) 270.64 579 T (,) 386.75 579 T ({) 170.73 579 T (}) 480.19 579 T 0 F (=) 155.96 579 T 0 0 612 792 C 118.99 549 129.32 564 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 119.99 554 T 0 0 612 792 C 207.65 549 218.22 564 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 208.65 554 T 0 0 612 792 C 247.88 549 258.21 564 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 248.88 554 T 0 0 612 792 C 413.05 549 423.38 564 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 414.05 554 T 0 0 612 792 C 497.7 549 508.27 564 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 498.7 554 T 0 0 612 792 C 180.98 535 191.31 550 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (P) 181.98 540 T 0 0 612 792 C 372.29 535 383.95 550 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (Q) 373.29 540 T 0 0 612 792 C 379.48 283.55 398.26 302 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 380.79 292 T 2 9 Q (X) 390.46 287.8 T 0 0 612 792 C 108 121 123.46 136 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (1) 115.46 126 T (\320) 109 126 T 0 0 612 792 C 250.1 121 259.1 136 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (0) 251.1 126 T 0 0 612 792 C 279.64 107 287.9 122 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (t) 280.64 112 T 0 0 612 792 C 476.64 79 503.23 94 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 480.64 84 T 3 F (t) 491.97 84 T 0 F (\050) 487.97 84 T (\051) 497.24 84 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "23" 9 %%Page: "22" 10 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (22) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (the state space and relevant intervals of time thus restricting attention to a subset of the) 108 712 T -0.21 (overall phase space \050the product of states and times\051. W) 108 698 P -0.21 (e call this \336rst step) 373.06 698 P 2 F -0.21 (pr) 465 698 P -0.21 (oblem formu-) 475.22 698 P -0.28 (lation) 108 684 P 0 F -0.28 (. Figure) 136.01 684 P -0.28 (22 depicts a restricted portion of phase space that might be identi\336ed during) 176.06 684 P -0.16 (problem formulation. Second, the system evaluates the current policy \050this corresponds to) 108 670 P (the value determination phase in policy iteration\051 with respect to the identi\336ed phase) 108 656 T (space. The third and last step is concerned with improving the current policy and corre-) 108 642 T (sponds to the policy improvement phase in policy iteration.) 108 628 T 1 10 Q (Figur) 121.75 603.33 T (e 22. Restrictions in phase space) 145.46 603.33 T 1 14 Q (7.1 Existing Appr) 108 406.35 T (oaches) 216.65 406.35 T 0 12 Q (Much of the decision-theoretic planning research in arti\336cial intelligence is concerned) 108 379.68 T (with goal-driven planning. W) 108 365.68 T (e focus on goals of achievement \050reach a state satisfying the) 249.02 365.68 T (goal predicate\051, but goals of prevention \050avoid states satisfying the goal predicate\051 and) 108 351.68 T (goals of maintenance \050remain in states satisfying the goal predicate\051 have also been stud-) 108 337.68 T (ied. Goals of achievement in stochastic environments come in two basic variants: maxi-) 108 323.68 T (mize the probability of achieving the goal and minimize the expected time to the goal.) 108 309.68 T (Planners are also classi\336ed according to whether they generate open- or closed-loop solu-) 108 295.68 T (tions and according to whether they employ \336nite- or in\336nite-horizon decision models.) 108 281.68 T (The \336rst planner we consider is called BURIDAN [Kushmerick et al., 1994]. BURIDAN) 108 255.68 T (generates open-loop solutions \050sequences of actions\051 and attempts to maximize the proba-) 108 241.68 T (bility of achieving a goal for a \336nite-horizon decision model. Actually) 108 227.68 T (, the planner does) 444.18 227.68 T (not attempt to \336nd the optimal open-loop solution, but rather to \336nd a solution whose) 108 213.68 T (probability of achieving the goal is greater than some threshold. BURIDAN employs an) 108 199.68 T -0.33 (extension of the planning algorithm of McAllester and Rosenblitt [1991] to handle actions) 108 185.68 P (with uncertain outcomes. An action) 108 171.68 T ( is modeled as a probability distribution governing) 292.55 171.68 T -0.13 (state transitions. BURIDAN uses a variant of the STRIPS assumption [Fikes and Nilsson,) 108 157.68 P (1971] to keep the action representation compact.) 108 143.68 T (For our purposes, an action is a set of triples of the form) 108 117.68 T ( where) 428.89 117.68 T ( is a set of) 474.44 117.68 T (propositions \050preconditions\051 that describe a subset of) 108 103.68 T (,) 383.74 103.68 T ( is a probability) 399.33 103.68 T (, and) 473.55 103.68 T ( is a) 511.1 103.68 T -0.45 (set of propositions \050postconditions\051 describing another subset of) 108 89.68 P -0.45 (. Semantically) 433.8 89.68 P -0.45 (, if) 501.24 89.68 P -0.45 ( is) 526.9 89.68 P (satis\336ed just prior to) 108 75.68 T (, then with probability) 219.91 75.68 T ( the postconditions in) 339.16 75.68 T ( are satis\336ed) 456.4 75.68 T 108 63 540 720 C 228.49 433.68 419.51 600 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 278.99 558.5 350.99 558.5 350.99 540.5 386.99 540.5 386.99 522.5 368.99 522.5 368.99 504.5 350.99 504.5 350.99 486.5 278.99 486.5 278.99 468.5 260.99 468.5 260.99 522.5 278.99 522.5 14 Y 14 X 0 0 0 1 0 0 0 K V 0.5 H 0 Z 0 X N 305.99 599 305.99 446 2 L 14 X V 2 Z 0 X N 409.49 599 404.99 518 409.49 522.5 404.99 446 418.49 446 418.49 599 6 Y 7 X V 229.49 599 242.99 599 238.49 518 242.99 522.5 238.49 446 229.49 446 6 Y V 90 450 4.5 4.5 269.99 585.5 G 0 Z 0 X 90 450 4.5 4.5 269.99 585.5 A 3 X 90 450 4.5 4.5 269.99 567.5 G 0 X 90 450 4.5 4.5 269.99 567.5 A 7 X 90 450 4.5 4.5 269.99 549.5 G 0 X 90 450 4.5 4.5 269.99 549.5 A 7 X 90 450 4.5 4.5 269.99 531.5 G 0 X 90 450 4.5 4.5 269.99 531.5 A 7 X 90 450 4.5 4.5 287.99 585.5 G 0 X 90 450 4.5 4.5 287.99 585.5 A 7 X 90 450 4.5 4.5 287.99 567.5 G 0 X 90 450 4.5 4.5 287.99 567.5 A 3 X 90 450 4.5 4.5 287.99 531.5 G 0 X 90 450 4.5 4.5 287.99 531.5 A 3 X 90 450 4.5 4.5 287.99 549.5 G 0 X 90 450 4.5 4.5 287.99 549.5 A 7 X 90 450 4.5 4.5 305.99 585.5 G 0 X 90 450 4.5 4.5 305.99 585.5 A 7 X 90 450 4.5 4.5 305.99 567.5 G 0 X 90 450 4.5 4.5 305.99 567.5 A 3 X 90 450 4.5 4.5 305.99 549.5 G 0 X 90 450 4.5 4.5 305.99 549.5 A 3 X 90 450 4.5 4.5 305.99 531.5 G 0 X 90 450 4.5 4.5 305.99 531.5 A 7 X 90 450 4.5 4.5 323.99 585.5 G 0 X 90 450 4.5 4.5 323.99 585.5 A 7 X 90 450 4.5 4.5 323.99 567.5 G 0 X 90 450 4.5 4.5 323.99 567.5 A 7 X 90 450 4.5 4.5 323.99 531.5 G 0 X 90 450 4.5 4.5 323.99 531.5 A 7 X 90 450 4.5 4.5 323.99 549.5 G 0 X 90 450 4.5 4.5 323.99 549.5 A 7 X 90 450 4.5 4.5 341.99 585.5 G 0 X 90 450 4.5 4.5 341.99 585.5 A 7 X 90 450 4.5 4.5 341.99 567.5 G 0 X 90 450 4.5 4.5 341.99 567.5 A 7 X 90 450 4.5 4.5 341.99 549.5 G 0 X 90 450 4.5 4.5 341.99 549.5 A 7 X 90 450 4.5 4.5 341.99 531.5 G 0 X 90 450 4.5 4.5 341.99 531.5 A 7 X 90 450 4.5 4.5 359.99 585.5 G 0 X 90 450 4.5 4.5 359.99 585.5 A 7 X 90 450 4.5 4.5 359.99 567.5 G 0 X 90 450 4.5 4.5 359.99 567.5 A 7 X 90 450 4.5 4.5 359.99 531.5 G 0 X 90 450 4.5 4.5 359.99 531.5 A 7 X 90 450 4.5 4.5 359.99 549.5 G 0 X 90 450 4.5 4.5 359.99 549.5 A 7 X 90 450 4.5 4.5 377.99 585.5 G 0 X 90 450 4.5 4.5 377.99 585.5 A 7 X 90 450 4.5 4.5 377.99 567.5 G 0 X 90 450 4.5 4.5 377.99 567.5 A 7 X 90 450 4.5 4.5 377.99 549.5 G 0 X 90 450 4.5 4.5 377.99 549.5 A 7 X 90 450 4.5 4.5 377.99 531.5 G 0 X 90 450 4.5 4.5 377.99 531.5 A 7 X 90 450 4.5 4.5 395.99 585.5 G 0 X 90 450 4.5 4.5 395.99 585.5 A 7 X 90 450 4.5 4.5 395.99 567.5 G 0 X 90 450 4.5 4.5 395.99 567.5 A 7 X 90 450 4.5 4.5 395.99 531.5 G 0 X 90 450 4.5 4.5 395.99 531.5 A 7 X 90 450 4.5 4.5 395.99 549.5 G 0 X 90 450 4.5 4.5 395.99 549.5 A 3 X 90 450 4.5 4.5 269.99 513.5 G 0 X 90 450 4.5 4.5 269.99 513.5 A 3 X 90 450 4.5 4.5 269.99 495.5 G 0 X 90 450 4.5 4.5 269.99 495.5 A 3 X 90 450 4.5 4.5 269.99 477.5 G 0 X 90 450 4.5 4.5 269.99 477.5 A 7 X 90 450 4.5 4.5 269.99 459.5 G 0 X 90 450 4.5 4.5 269.99 459.5 A 3 X 90 450 4.5 4.5 287.99 513.5 G 0 X 90 450 4.5 4.5 287.99 513.5 A 7 X 90 450 4.5 4.5 287.99 495.5 G 0 X 90 450 4.5 4.5 287.99 495.5 A 7 X 90 450 4.5 4.5 287.99 459.5 G 0 X 90 450 4.5 4.5 287.99 459.5 A 7 X 90 450 4.5 4.5 287.99 477.5 G 0 X 90 450 4.5 4.5 287.99 477.5 A 3 X 90 450 4.5 4.5 305.99 513.5 G 0 X 90 450 4.5 4.5 305.99 513.5 A 7 X 90 450 4.5 4.5 305.99 495.5 G 0 X 90 450 4.5 4.5 305.99 495.5 A 7 X 90 450 4.5 4.5 305.99 477.5 G 0 X 90 450 4.5 4.5 305.99 477.5 A 7 X 90 450 4.5 4.5 305.99 459.5 G 0 X 90 450 4.5 4.5 305.99 459.5 A 7 X 90 450 4.5 4.5 323.99 513.5 G 0 X 90 450 4.5 4.5 323.99 513.5 A 7 X 90 450 4.5 4.5 323.99 495.5 G 0 X 90 450 4.5 4.5 323.99 495.5 A 7 X 90 450 4.5 4.5 323.99 459.5 G 0 X 90 450 4.5 4.5 323.99 459.5 A 7 X 90 450 4.5 4.5 323.99 477.5 G 0 X 90 450 4.5 4.5 323.99 477.5 A 7 X 90 450 4.5 4.5 341.99 513.5 G 0 X 90 450 4.5 4.5 341.99 513.5 A 7 X 90 450 4.5 4.5 341.99 495.5 G 0 X 90 450 4.5 4.5 341.99 495.5 A 7 X 90 450 4.5 4.5 341.99 477.5 G 0 X 90 450 4.5 4.5 341.99 477.5 A 7 X 90 450 4.5 4.5 341.99 459.5 G 0 X 90 450 4.5 4.5 341.99 459.5 A 7 X 90 450 4.5 4.5 359.99 513.5 G 0 X 90 450 4.5 4.5 359.99 513.5 A 7 X 90 450 4.5 4.5 359.99 495.5 G 0 X 90 450 4.5 4.5 359.99 495.5 A 7 X 90 450 4.5 4.5 359.99 459.5 G 0 X 90 450 4.5 4.5 359.99 459.5 A 7 X 90 450 4.5 4.5 359.99 477.5 G 0 X 90 450 4.5 4.5 359.99 477.5 A 7 X 90 450 4.5 4.5 377.99 513.5 G 0 X 90 450 4.5 4.5 377.99 513.5 A 7 X 90 450 4.5 4.5 377.99 495.5 G 0 X 90 450 4.5 4.5 377.99 495.5 A 7 X 90 450 4.5 4.5 377.99 477.5 G 0 X 90 450 4.5 4.5 377.99 477.5 A 7 X 90 450 4.5 4.5 377.99 459.5 G 0 X 90 450 4.5 4.5 377.99 459.5 A 7 X 90 450 4.5 4.5 395.99 513.5 G 0 X 90 450 4.5 4.5 395.99 513.5 A 7 X 90 450 4.5 4.5 395.99 495.5 G 0 X 90 450 4.5 4.5 395.99 495.5 A 7 X 90 450 4.5 4.5 395.99 459.5 G 0 X 90 450 4.5 4.5 395.99 459.5 A 7 X 90 450 4.5 4.5 395.99 477.5 G 0 X 90 450 4.5 4.5 395.99 477.5 A 242.99 599 238.49 518 242.99 522.5 238.49 446 4 L 2 Z N 409.49 599 404.99 518 409.49 522.5 404.99 446 4 L N 7 X 90 450 4.5 4.5 251.99 585.5 G 0 Z 0 X 90 450 4.5 4.5 251.99 585.5 A 7 X 90 450 4.5 4.5 251.99 567.5 G 0 X 90 450 4.5 4.5 251.99 567.5 A 7 X 90 450 4.5 4.5 251.99 531.5 G 0 X 90 450 4.5 4.5 251.99 531.5 A 3 X 90 450 4.5 4.5 251.99 549.5 G 0 X 90 450 4.5 4.5 251.99 549.5 A 3 X 90 450 4.5 4.5 251.99 513.5 G 0 X 90 450 4.5 4.5 251.99 513.5 A 3 X 90 450 4.5 4.5 251.99 495.5 G 0 X 90 450 4.5 4.5 251.99 495.5 A 7 X 90 450 4.5 4.5 251.99 459.5 G 0 X 90 450 4.5 4.5 251.99 459.5 A 3 X 90 450 4.5 4.5 251.99 477.5 G 0 X 90 450 4.5 4.5 251.99 477.5 A 2 9 Q (t) 297.55 437.91 T 0 F (0) 314.11 437.91 T (=) 304.54 437.91 T 108 63 540 720 C 0 0 612 792 C 281.98 166.68 292.55 181.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 282.98 171.68 T 0 0 612 792 C 380.63 112.68 428.89 127.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (j) 388.18 117.68 T (r) 401.42 117.68 T (w) 414.01 117.68 T (,) 395.42 117.68 T (,) 408.01 117.68 T (\341) 382.82 117.68 T (\361) 422.24 117.68 T 0 0 612 792 C 464.2 112.68 474.44 127.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (j) 465.2 117.68 T 0 0 612 792 C 364.97 95.23 383.74 113.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 366.27 103.68 T 2 9 Q (X) 375.94 99.48 T 0 0 612 792 C 389.74 98.68 399.33 113.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (r) 390.74 103.68 T 0 0 612 792 C 499.87 98.68 511.1 113.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (w) 500.87 103.68 T 0 0 612 792 C 415.03 81.23 433.8 99.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 416.33 89.68 T 2 9 Q (X) 426 85.48 T 0 0 612 792 C 516.66 84.68 526.9 99.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (j) 517.66 89.68 T 0 0 612 792 C 209.34 70.68 219.91 85.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (a) 210.34 75.68 T 0 0 612 792 C 329.58 70.68 339.16 85.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (r) 330.58 75.68 T 0 0 612 792 C 445.17 70.68 456.4 85.68 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (w) 446.17 75.68 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "22" 10 %%Page: "21" 11 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (21) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (The underlying dynamical system is modeled as a Markov chain, but the control system) 108 712 T (only uses information regarding the current state and ignores information from prior) 108 698 T -0.13 (states; the system makes no attempt to estimate the state from a sequence of observations.) 108 684 P (The corresponding decision process is said to be non-Markovian. For non-Markovian) 108 670 T (decision processes, it may be that there exists a stochastic policy \050a policy in which the) 108 656 T (control system \337ips coins to decide which action to take in response to a particular obser-) 108 642 T (vation\051 that is better than the best deterministic policy) 108 628 T (.) 365.87 628 T 1 16 Q (7 Decision-Theor) 108 587.33 T (etic Planning) 228.59 587.33 T 0 12 Q -0.12 (Researchers in arti\336cial intelligence working in the area of automated planning have been) 108 560 P (aware of the potential for applying decision-theoretic methods for some time \050see for) 108 546 T (example [Feldman and Sproull, 1977]\051. Until recently) 108 532 T (, however) 365.86 532 T (, computational complex-) 412.69 532 T (ity has prevented most researchers from giving decision-theoretic methods serious atten-) 108 518 T -0.3 (tion. In the last few years there have been a lar) 108 504 P -0.3 (ge number of papers published that claim to) 328.41 504 P (be using decision theory to solve planning problems. Many of these papers address the) 108 490 T (class of problems associated with Markov decision processes \050MDPs\051 without explicitly) 108 476 T (identifying the class or acknowledging the lar) 108 462 T (ge body of related work addressing such) 327.08 462 T (problems. While it is certainly not true that all of decision-theoretic planning can be) 108 448 T -0.38 (described in terms of MDPs \050see for example the work of W) 108 434 P -0.38 (ellman [1987]\051, many existing) 393.18 434 P (techniques can be characterized as attempting to solve MDPs.) 108 420 T (Given that the underlying dynamical systems studied in decision-theoretic planning are) 108 394 T -0.48 (stochastic it is generally useful to construct conditional plans \050policies\051 that indicate which) 108 380 P (actions to take in situations that are likely to be encountered. In many problems, it is) 108 366 T (assumed that the particular task set for the planner will change periodically and that even) 108 352 T (the model for the dynamical system may change from time to time. Given that the under-) 108 338 T -0.15 (lying models have a lar) 108 324 P -0.15 (ge number of states it is generally impractical to construct a policy) 219.17 324 P (to cover all possible situations and all possible tasks.) 108 310 T (Instead of constructing a single policy to cover all possible tasks and situations, most) 108 284 T (researchers are interested in constructing a limited policy constructed online, often con-) 108 270 T -0.15 (current with execution. This limited policy is specially tailored to the particular stochastic) 108 256 P (model and task set for the system. Planning problems in which the control system is con-) 108 242 T (currently planning and executing are said to be) 108 228 T 2 F (embedded) 335.95 228 T 0 F (.) 384.6 228 T (W) 108 202 T (e generally assume that there is some penalty \050opportunity cost\051 associated with delay-) 118.37 202 T (ing execution and so there is some ur) 108 188 T (gency in constructing a policy) 286.09 188 T (. Much of the recent) 429.96 188 T -0.26 (research addresses tradeof) 108 174 P -0.26 (fs involving the quality of the constructed policy and the cost of) 233.2 174 P (delay) 108 160 T (. In some cases, the system may construct a policy) 133.21 160 T (, begin executing it, while at the) 374.06 160 T (same time trying to construct another policy that is either better in terms of overall solu-) 108 146 T (tion quality or that anticipates the demands of execution by planning ahead for situations) 108 132 T (likely to be encountered during a particular interval of time.) 108 118 T (Whether we construct one or several policies in succession, we can think of the basic) 108 92 T (cycle of activity as involving three steps. First, the system identi\336es relevant portions of) 108 78 T 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "21" 11 %%Page: "20" 12 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (20) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 10 Q (\05026\051) 523.34 624.1 T 0 12 Q (Figure) 108 600.76 T (21 shows a partially observable dynamical system with a state estimation compo-) 142.33 600.76 T (nent) 108 586.76 T ( which takes as input the current output) 141.9 586.76 T ( and returns a distribution over) 344.16 586.76 T (notated) 108 572.76 T ( from the set of all such distributions) 158.96 572.76 T (.) 357.87 572.76 T 1 10 Q (Figur) 121.75 548.1 T (e 21. Dynamical system with a Bayesian state estimator) 145.46 548.1 T 0 12 Q (The probability of) 108 406.1 T ( at time) 205.65 406.1 T ( given by the optimal state estimator) 250.98 406.1 T ( is de\336ned as fol-) 455.04 406.1 T (lows.) 108 392.1 T 0 10 Q (\05027\051) 523.34 333.61 T 0 12 Q -0.38 (If observation and control in the dynamical system shown in Figure) 108 274.56 P -0.38 (21 were separable, we) 431.88 274.56 P -0.04 (could just compute the optimal policy for the observable case and couple it to the optimal) 108 260.56 P -0.2 (state estimator) 108 246.56 P -0.2 (. Unfortunately) 176.8 246.56 P -0.2 (, the system in Figure) 249.13 246.56 P -0.2 (21 is not separable and the problem of) 354.99 246.56 P (computing the optimal coupled estimator and regulator is considerably complicated. It) 108 232.56 T -0.46 (turns out that there is a completely observable Markov decision process in which the states) 108 218.56 P (are) 108 204.56 T ( and the transitions are determined by Equation) 145.23 204.56 T (27, but the state space is neither) 375.2 204.56 T (\336nite nor discrete. Fortunately) 108 190.56 T (, this) 252.54 190.56 T (-dimensional continuous space has considerable) 303.53 190.56 T (structure; we do not pursue this potential advantage, but see Smallwood and Sondik) 108 176.56 T ([1972] for details on how this structure might be exploited.) 108 162.56 T (In the decision problem described by Singh et al. [1994], the control system has sensors) 108 136.56 T (that return partial information about the current state. For example, the sensory system) 108 122.56 T (might return the equivalence class of the current state with respect to some equivalence) 108 108.56 T (relation that equates perceptually indistinguishable states.) 108 94.56 T 126.14 624.1 505.2 720 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 133.28 706.27 T 2 F (V) 144.61 706.27 T 0 9 Q (1) 152.4 702.07 T (3) 170.27 702.07 T 3 F (\256) 159.15 702.07 T 0 12 Q (\050) 140.62 706.27 T (\051) 174.77 706.27 T (m) 197.53 706.27 T (a) 206.87 706.27 T (x) 212.2 706.27 T 2 9 Q (d) 218.65 702.07 T 0 6 Q (1) 223.49 699.52 T 0 12 Q (P) 254.23 706.27 T (r) 260.9 706.27 T 2 F (X) 268.89 706.27 T 0 9 Q (1) 276.68 702.07 T 2 12 Q (x) 295.94 706.27 T (D) 306.11 706.27 T 0 9 Q (1) 315.23 702.07 T 2 12 Q (d) 334.49 706.27 T 0 9 Q (1) 340.94 702.07 T 0 12 Q (=) 323.72 706.27 T (=) 285.17 706.27 T (\050) 264.9 706.27 T (\051) 345.44 706.27 T 3 F (\264) 352.44 706.27 T 2 9 Q (x) 227.2 690.32 T 3 F (W) 242.11 690.32 T 2 6 Q (X) 249.36 687.77 T 3 9 Q (\316) 233.45 690.32 T 3 18 Q (\345) 233.7 703.07 T 0 12 Q (=) 184.77 706.27 T 303.66 703.27 303.66 714.07 2 L 0.54 H 2 Z N 2 F (V) 197.36 668.81 T 0 9 Q (1) 205.15 664.61 T 2 12 Q (X) 213.65 668.81 T 0 9 Q (1) 221.43 664.61 T 2 12 Q (x) 240.69 668.81 T 0 F (=) 229.93 668.81 T (\050) 209.65 668.81 T (\051) 246.02 668.81 T (P) 296.78 670.95 T (r) 303.45 670.95 T 2 F (Y) 311.44 670.95 T 0 9 Q (1) 318.57 666.76 T 2 12 Q (y) 337.82 670.95 T (X) 348 670.95 T 0 9 Q (1) 355.79 666.76 T 2 12 Q (x) 375.04 670.95 T 0 F (=) 364.28 670.95 T (=) 327.06 670.95 T (\050) 307.44 670.95 T (\051) 380.37 670.95 T (E) 388.07 670.95 T 2 F (V) 399.4 670.95 T 0 9 Q (2) 407.19 666.76 T (3) 425.07 666.76 T 3 F (\256) 413.94 666.76 T 2 12 Q (D) 439.75 680 T 0 9 Q (1) 448.87 675.8 T 2 12 Q (d) 468.12 680 T 0 9 Q (1) 474.58 675.8 T 0 12 Q (=) 457.36 680 T 2 F (Y) 443.56 659.75 T 0 9 Q (1) 450.69 655.55 T 2 12 Q (y) 469.94 659.75 T 0 F (=) 459.18 659.75 T (\050) 395.4 670.95 T (\051) 483.34 670.95 T 2 9 Q (y) 269.93 655 T 3 F (W) 284.99 655 T 2 6 Q (Y) 292.24 652.45 T 0 9 Q (=) 276.92 655 T 3 18 Q (\345) 276.34 667.75 T 0 12 Q (+) 253.01 668.81 T 345.55 667.95 345.55 678.76 2 L N 431.96 664.51 431.96 678.76 2 L N 440.41 653.3 436.81 653.3 436.81 693.79 3 L N 436.81 693.79 440.41 693.79 2 L N 477.34 653.3 480.94 653.3 480.94 693.79 3 L N 480.94 693.79 477.34 693.79 2 L N 268.78 651.16 265.18 651.16 265.18 695.94 3 L N 265.18 695.94 268.78 695.94 2 L N 487.89 651.16 491.49 651.16 491.49 695.94 3 L N 491.49 695.94 487.89 695.94 2 L N 196.21 651.16 192.61 651.16 192.61 695.94 3 L N 192.61 695.94 196.21 695.94 2 L N 494.44 651.16 498.04 651.16 498.04 695.94 3 L N 498.04 695.94 494.44 695.94 2 L N (E) 194.38 629.1 T 3 F (S) 205.71 629.1 T 0 F (\050) 201.71 629.1 T (\051) 212.81 629.1 T (=) 178.62 629.1 T 0 0 612 792 C 131.66 581.76 141.9 596.76 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (s) 132.66 586.76 T 0 0 612 792 C 333.87 580.12 344.16 595.97 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (y) 334.38 586.76 T 2 9 Q (t) 340.16 582.56 T 0 0 612 792 C 494.81 578.31 513.58 596.76 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 496.11 586.76 T 2 9 Q (X) 505.78 582.56 T 0 0 612 792 C 146.33 564.31 158.96 582.76 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (p) 147.37 572.76 T 2 9 Q (t) 154.41 568.56 T 0 0 612 792 C 338.29 564.31 357.87 582.76 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 339.29 572.76 T 3 9 Q (P) 348.96 568.56 T 0 0 612 792 C 108 63 540 720 C 219.98 430.1 428.02 544.76 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 260.98 506.1 314.98 542.1 R 0.5 H 0 Z 0 X 0 0 0 1 0 0 0 K N 350.98 506.1 386.98 542.1 R N 260.98 434.1 314.98 470.1 R N 343.29 526.3 350.98 524.1 343.29 521.89 343.29 524.1 4 Y V 314.98 524.1 343.29 524.1 2 L 2 Z N 350.98 434.1 386.98 470.1 R 0 Z N 394.67 449.89 386.98 452.1 394.67 454.3 394.67 452.1 4 Y V 386.98 524.1 422.98 524.1 422.98 452.1 394.67 452.1 4 L 2 Z N 322.67 449.89 314.98 452.1 322.67 454.3 322.67 452.1 4 Y V 350.98 452.1 322.67 452.1 2 L N 253.29 535.3 260.98 533.1 253.29 530.89 253.29 533.1 4 Y V 260.98 452.1 224.98 452.1 224.98 533.1 253.29 533.1 4 L N 253.29 517.3 260.98 515.1 253.29 512.89 253.29 515.1 4 Y V 332.98 524.1 332.98 488.1 242.98 488.1 242.98 515.1 253.29 515.1 5 L N 2 12 Q (f) 267.65 522.1 T (x) 278.85 522.1 T 2 9 Q (t) 284.63 517.9 T 2 12 Q (u) 293.13 522.1 T 2 9 Q (t) 299.58 517.9 T 3 12 Q (,) 287.13 522.1 T (\050) 273.7 522.1 T (\051) 302.64 522.1 T 2 F (x) 320.3 532.76 T 2 9 Q (t) 326.08 528.56 T 0 F (1) 338.15 528.56 T (+) 330.83 528.56 T 2 12 Q (h) 355.13 522.76 T (x) 368.99 522.76 T 2 9 Q (t) 374.77 518.56 T 3 12 Q (\050) 363.84 522.76 T (\051) 377.82 522.76 T 2 F (y) 401.33 532.1 T 2 9 Q (t) 407.12 527.9 T 3 12 Q (s) 357.72 450.1 T 2 F (y) 368.95 450.1 T 2 9 Q (t) 374.73 445.9 T 0 12 Q (\050) 364.95 450.1 T (\051) 377.24 450.1 T 3 F (p) 330.7 460.76 T 2 9 Q (t) 337.75 456.56 T 3 12 Q (d) 277.08 450.1 T (p) 287 450.1 T 2 9 Q (t) 294.04 445.9 T 0 12 Q (\050) 283 450.1 T (\051) 296.55 450.1 T 2 F (u) 240.09 460.29 T 2 9 Q (t) 246.55 456.09 T 108 63 540 720 C 0 0 612 792 C 198.32 402.62 205.65 415.3 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (x) 198.82 406.1 T 0 0 612 792 C 244.65 401.1 250.98 416.1 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 245.65 406.1 T 0 0 612 792 C 428.64 397.65 455.04 416.1 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (p) 429.91 406.1 T 2 9 Q (t) 436.95 401.9 T 2 12 Q (x) 443.45 406.1 T 0 F (\050) 439.45 406.1 T (\051) 448.78 406.1 T 0 0 612 792 C 140.51 294.56 490.83 376.1 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (p) 145.01 333.61 T 2 9 Q (t) 152.05 329.41 T 2 12 Q (x) 158.55 333.61 T 0 F (\050) 154.55 333.61 T (\051) 163.88 333.61 T (P) 213.18 357.8 T (r) 219.85 357.8 T 2 F (y) 227.84 357.8 T 2 9 Q (t) 233.63 353.6 T 2 12 Q (X) 240.97 357.8 T 2 9 Q (t) 248.76 353.6 T 2 12 Q (x) 266.02 357.8 T 0 F (=) 255.26 357.8 T (\050) 223.85 357.8 T (\051) 271.35 357.8 T (P) 304.7 357.8 T (r) 311.37 357.8 T 2 F (X) 319.36 357.8 T 2 9 Q (t) 327.15 353.6 T 2 12 Q (x) 344.41 357.8 T 0 F (=) 333.65 357.8 T 2 F (X) 354.58 357.8 T 2 9 Q (t) 362.37 353.6 T 0 F (1) 373.87 353.6 T (\320) 367.12 353.6 T 2 12 Q (x) 393.12 357.8 T 0 F (') 398.45 357.8 T (=) 382.36 357.8 T 2 F (u) 406.61 357.8 T 2 9 Q (t) 413.06 353.6 T 3 12 Q (,) 400.61 357.8 T 0 F (\050) 315.37 357.8 T (\051) 415.57 357.8 T 3 F (p) 420.27 357.8 T 2 9 Q (t) 427.31 353.6 T 0 F (1) 438.81 353.6 T (\320) 432.06 353.6 T 2 12 Q (x) 447.31 357.8 T 0 F (') 452.63 357.8 T (\050) 443.31 357.8 T (\051) 454.79 357.8 T 2 9 Q (x) 276.05 341.84 T 0 F (') 280.05 341.84 T 3 F (W) 292.58 341.84 T 2 6 Q (X) 299.83 339.29 T 3 9 Q (\316) 283.92 341.84 T 3 18 Q (\345) 283.36 354.59 T 0 12 Q (P) 224.29 319.42 T (r) 230.96 319.42 T 2 F (y) 238.96 319.42 T 2 9 Q (t) 244.74 315.22 T 2 12 Q (X) 252.09 319.42 T 2 9 Q (t) 259.88 315.22 T 2 12 Q (x) 277.13 319.42 T 0 F (') 282.46 319.42 T (') 284.62 319.42 T (=) 266.37 319.42 T (\050) 234.96 319.42 T (\051) 286.78 319.42 T (P) 320.13 319.42 T (r) 326.8 319.42 T 2 F (X) 334.8 319.42 T 2 9 Q (t) 342.58 315.22 T 2 12 Q (x) 359.84 319.42 T 0 F (') 365.17 319.42 T (') 367.33 319.42 T (=) 349.08 319.42 T 2 F (X) 374.34 319.42 T 2 9 Q (t) 382.12 315.22 T 0 F (1) 393.62 315.22 T (\320) 386.87 315.22 T 2 12 Q (x) 412.88 319.42 T 0 F (') 418.2 319.42 T (=) 402.11 319.42 T 2 F (u) 426.36 319.42 T 2 9 Q (t) 432.82 315.22 T 3 12 Q (,) 420.36 319.42 T 0 F (\050) 330.8 319.42 T (\051) 435.32 319.42 T 3 F (p) 440.02 319.42 T 2 9 Q (t) 447.07 315.22 T 0 F (1) 458.56 315.22 T (\320) 451.82 315.22 T 2 12 Q (x) 467.06 319.42 T 0 F (') 472.39 319.42 T (\050) 463.06 319.42 T (\051) 474.55 319.42 T 2 9 Q (x) 291.48 302.46 T 0 F (') 295.48 302.46 T 3 F (W) 308.01 302.46 T 2 6 Q (X) 315.27 299.91 T 3 9 Q (\316) 299.35 302.46 T 3 18 Q (\345) 298.79 316.21 T 2 9 Q (x) 186.87 299.61 T 0 F (') 190.87 299.61 T (') 192.49 299.61 T 3 F (W) 205.02 299.61 T 2 6 Q (X) 212.27 297.06 T 3 9 Q (\316) 196.36 299.61 T 3 18 Q (\345) 194.99 312.36 T 0 12 Q (-) 186.87 333.61 T (-) 188.87 333.61 T (-) 190.87 333.61 T (-) 192.87 333.61 T (-) 194.87 333.61 T (-) 196.86 333.61 T (-) 198.86 333.61 T (-) 200.86 333.61 T (-) 202.86 333.61 T (-) 204.86 333.61 T (-) 206.85 333.61 T (-) 208.85 333.61 T (-) 210.85 333.61 T (-) 212.85 333.61 T (-) 214.85 333.61 T (-) 216.84 333.61 T (-) 218.84 333.61 T (-) 220.84 333.61 T (-) 222.84 333.61 T (-) 224.84 333.61 T (-) 226.83 333.61 T (-) 228.83 333.61 T (-) 230.83 333.61 T (-) 232.83 333.61 T (-) 234.83 333.61 T (-) 236.82 333.61 T (-) 238.82 333.61 T (-) 240.82 333.61 T (-) 242.82 333.61 T (-) 244.82 333.61 T (-) 246.81 333.61 T (-) 248.81 333.61 T (-) 250.81 333.61 T (-) 252.81 333.61 T (-) 254.8 333.61 T (-) 256.8 333.61 T (-) 258.8 333.61 T (-) 260.8 333.61 T (-) 262.8 333.61 T (-) 264.8 333.61 T (-) 266.79 333.61 T (-) 268.79 333.61 T (-) 270.79 333.61 T (-) 272.79 333.61 T (-) 274.79 333.61 T (-) 276.78 333.61 T (-) 278.78 333.61 T (-) 280.78 333.61 T (-) 282.78 333.61 T (-) 284.77 333.61 T (-) 286.77 333.61 T (-) 288.77 333.61 T (-) 290.77 333.61 T (-) 292.77 333.61 T (-) 294.77 333.61 T (-) 296.76 333.61 T (-) 298.76 333.61 T (-) 300.76 333.61 T (-) 302.76 333.61 T (-) 304.76 333.61 T (-) 306.75 333.61 T (-) 308.75 333.61 T (-) 310.75 333.61 T (-) 312.75 333.61 T (-) 314.74 333.61 T (-) 316.74 333.61 T (-) 318.74 333.61 T (-) 320.74 333.61 T (-) 322.74 333.61 T (-) 324.73 333.61 T (-) 326.73 333.61 T (-) 328.73 333.61 T (-) 330.73 333.61 T (-) 332.73 333.61 T (-) 334.73 333.61 T (-) 336.72 333.61 T (-) 338.72 333.61 T (-) 340.72 333.61 T (-) 342.72 333.61 T (-) 344.71 333.61 T (-) 346.71 333.61 T (-) 348.71 333.61 T (-) 350.71 333.61 T (-) 352.71 333.61 T (-) 354.7 333.61 T (-) 356.7 333.61 T (-) 358.7 333.61 T (-) 360.7 333.61 T (-) 362.7 333.61 T (-) 364.7 333.61 T (-) 366.69 333.61 T (-) 368.69 333.61 T (-) 370.69 333.61 T (-) 372.69 333.61 T (-) 374.68 333.61 T (-) 376.68 333.61 T (-) 378.68 333.61 T (-) 380.68 333.61 T (-) 382.68 333.61 T (-) 384.67 333.61 T (-) 386.67 333.61 T (-) 388.67 333.61 T (-) 390.67 333.61 T (-) 392.67 333.61 T (-) 394.66 333.61 T (-) 396.66 333.61 T (-) 398.66 333.61 T (-) 400.66 333.61 T (-) 402.66 333.61 T (-) 404.65 333.61 T (-) 406.65 333.61 T (-) 408.65 333.61 T (-) 410.65 333.61 T (-) 412.65 333.61 T (-) 414.64 333.61 T (-) 416.64 333.61 T (-) 418.64 333.61 T (-) 420.64 333.61 T (-) 422.64 333.61 T (-) 424.63 333.61 T (-) 426.63 333.61 T (-) 428.63 333.61 T (-) 430.63 333.61 T (-) 432.63 333.61 T (-) 434.62 333.61 T (-) 436.62 333.61 T (-) 438.62 333.61 T (-) 440.62 333.61 T (-) 442.62 333.61 T (-) 444.61 333.61 T (-) 446.61 333.61 T (-) 448.61 333.61 T (-) 450.61 333.61 T (-) 452.61 333.61 T (-) 454.6 333.61 T (-) 456.6 333.61 T (-) 458.6 333.61 T (-) 460.6 333.61 T (-) 462.6 333.61 T (-) 464.59 333.61 T (-) 466.59 333.61 T (-) 468.59 333.61 T (-) 470.59 333.61 T (-) 472.59 333.61 T (-) 474.58 333.61 T (-) 476.58 333.61 T (-) 478.58 333.61 T (-) 480.58 333.61 T (-) 481.1 333.61 T (=) 173.87 333.61 T 238.53 351.35 238.53 365.6 2 L 0.54 H 2 Z N 352.14 351.35 352.14 365.6 2 L N 249.64 312.97 249.64 327.22 2 L N 371.89 312.97 371.89 327.22 2 L N 223.14 302.01 219.54 302.01 219.54 332.18 3 L N 219.54 332.18 223.14 332.18 2 L N 479.1 302.01 482.7 302.01 482.7 332.18 3 L N 482.7 332.18 479.1 332.18 2 L N 0 0 612 792 C 125.65 196.11 145.23 214.56 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 126.65 204.56 T 3 9 Q (P) 136.32 200.36 T 0 0 612 792 C 278.88 182.11 303.53 200.56 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 283.12 190.56 T 2 9 Q (X) 292.79 186.36 T 281.38 185.31 281.38 199.56 2 L 0.54 H 2 Z N 299.49 185.31 299.49 199.56 2 L N 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "20" 12 %%Page: "19" 13 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (19) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 1 10 Q (Figur) 121.75 713.33 T (e 20. Partially observable decision pr) 145.46 713.33 T (oblem) 302.49 713.33 T 0 12 Q -0.46 (The basic dynamic programming equations start of) 108 511.67 P -0.46 (f very similar to the ones we introduced) 349.68 511.67 P (earlier except for the fact that at each stage we have to account for all past observations) 108 497.67 T (not just the one\050s\051 at the current stage.) 108 483.67 T 0 10 Q (\05024\051) 523.34 411.06 T 0 12 Q (The next equation is more interesting, however) 108 338.64 T (. In computing the expected value for) 333.64 338.64 T -0.04 (stages two through three, we use the prior distribution for the observations given the state) 108 324.64 P -0.21 ( to account for the expected improvement in value from the information gained) 159.74 310.64 P (as a consequence of acting.) 108 296.64 T 0 10 Q (\05025\051) 523.34 134.39 T 0 12 Q -0.07 (W) 108 111.06 P -0.07 (e repeat this value of information calculation in computing the expected value of stages) 118.37 111.06 P (one through three, this time using the prior distribution) 108 97.06 T (.) 424.94 97.06 T 108 63 540 720 C 217.17 535.67 430.83 710 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 378.83 602.75 369.83 589.25 378.83 575.75 387.83 589.25 4 Y 7 X 0 0 0 1 0 0 0 K V 0.5 H 0 Z 0 X N 290.81 675.31 297.83 679.25 293.96 672.19 292.39 673.75 4 Y V 2 Z 127.23 180 9 27 297.83 652.25 A 326.81 675.31 333.83 679.25 329.96 672.19 328.39 673.75 4 Y V 127.23 180 9 27 333.83 652.25 A 322.63 635.56 324.83 643.25 327.04 635.56 324.83 635.56 4 Y V 324.83 616.25 324.83 635.56 2 L N 286.63 635.56 288.83 643.25 291.04 635.56 288.83 635.56 4 Y V 288.83 616.25 288.83 635.56 2 L N 261.83 679.25 279.83 697.25 R 3 X V 0 Z 0 X N 297.83 679.25 315.83 697.25 R 7 X V 0 X N 3 X 90 450 9 9 288.83 652.25 G 0 X 90 450 9 9 288.83 652.25 A 333.83 679.25 351.83 697.25 R 7 X V 0 X N 7 X 90 450 9 9 324.83 652.25 G 0 X 90 450 9 9 324.83 652.25 A 272.14 618.46 279.83 616.25 272.14 614.04 272.14 616.25 4 Y V 252.83 616.25 272.14 616.25 2 L 7 X V 2 Z 0 X N 308.14 618.46 315.83 616.25 308.14 614.04 308.14 616.25 4 Y V 288.83 616.25 308.14 616.25 2 L 7 X V 0 X N 344.14 618.46 351.83 616.25 344.14 614.04 344.14 616.25 4 Y V 324.83 616.25 344.14 616.25 2 L 7 X V 0 X N 7 X 90 450 9 9 252.83 616.25 G 0 Z 0 X 90 450 9 9 252.83 616.25 A 3 X 90 450 9 9 288.83 616.25 G 0 X 90 450 9 9 288.83 616.25 A 7 X 90 450 9 9 324.83 616.25 G 0 X 90 450 9 9 324.83 616.25 A 7 X 90 450 9 9 360.83 616.25 G 0 X 90 450 9 9 360.83 616.25 A 277.57 628.43 279.83 620.75 273.85 626.07 275.71 627.25 4 Y V 2 Z 180 242.75 9 58.5 279.83 679.25 A 313.57 628.43 315.83 620.75 309.85 626.07 311.71 627.25 4 Y V 180 242.75 9 58.5 315.83 679.25 A 349.57 628.43 351.83 620.75 345.85 626.07 347.71 627.25 4 Y V 180 242.75 9 58.5 351.83 679.25 A 342.83 602.75 333.83 589.25 342.83 575.75 351.83 589.25 4 Y 7 X V 0 Z 0 X N 293.1 595.76 297.83 589.25 290.37 592.26 291.74 594.01 4 Y V 2 Z 180 227.37 9 18 297.83 607.25 A 306.83 602.75 297.83 589.25 306.83 575.75 315.83 589.25 4 Y 3 X V 0 Z 0 X N 329.1 595.76 333.83 589.25 326.37 592.26 327.74 594.01 4 Y V 2 Z 180 227.37 9 18 333.83 607.25 A 365.1 595.76 369.83 589.25 362.37 592.26 363.74 594.01 4 Y V 180 227.37 9 18 369.83 607.25 A 414.83 566.75 405.83 553.25 414.83 539.75 423.83 553.25 4 Y 7 X V 0 Z 0 X N 2 Z 180 270 27 22.5 405.83 575.75 A 397.03 556.02 405.83 553.25 396.9 550.93 396.96 553.47 4 Y V 180 261.9 63 22.5 405.83 575.75 A 180 270 99 22.5 405.83 575.75 A 3 12 Q (S) 411.81 550.38 T 2 9 Q (t) 240.53 701.5 T 0 F (0) 257.1 701.5 T (=) 247.53 701.5 T 2 F (t) 312.95 701.5 T 0 F (2) 329.51 701.5 T (=) 319.94 701.5 T 2 F (t) 276.74 701.5 T 0 F (1) 293.31 701.5 T (=) 283.74 701.5 T 2 12 Q (D) 220.17 677.92 T (Y) 220.17 646.81 T (X) 220.17 615.69 T (V) 220.17 584.58 T 2 9 Q (t) 349.16 701.5 T 0 F (3) 365.72 701.5 T (=) 356.15 701.5 T 108 63 540 720 C 0 0 612 792 C 143.39 358.63 487.95 467.67 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 150.23 411.06 T 2 F (V) 161.56 411.06 T 0 9 Q (3) 169.35 406.86 T (3) 187.22 406.86 T 3 F (\256) 176.09 406.86 T 2 12 Q (D) 201.9 440.35 T 0 9 Q (1) 211.02 436.15 T 2 12 Q (d) 230.28 440.35 T 0 9 Q (1) 236.74 436.15 T 0 12 Q (=) 219.52 440.35 T 2 F (Y) 203.24 420.1 T 0 9 Q (1) 210.37 415.9 T 2 12 Q (y) 229.62 420.1 T 0 9 Q (1) 235.4 415.9 T 0 12 Q (=) 218.86 420.1 T 2 F (D) 201.9 399.85 T 0 9 Q (2) 211.02 395.65 T 2 12 Q (d) 230.28 399.85 T 0 9 Q (2) 236.74 395.65 T 0 12 Q (=) 219.52 399.85 T 2 F (Y) 203.24 379.61 T 0 9 Q (2) 210.37 375.41 T 2 12 Q (y) 229.62 379.61 T 0 9 Q (2) 235.4 375.41 T 0 12 Q (=) 218.86 379.61 T (\050) 157.56 411.06 T (\051) 245.49 411.06 T (m) 268.25 411.06 T (a) 277.59 411.06 T (x) 282.92 411.06 T 2 9 Q (d) 289.37 406.86 T 0 6 Q (3) 294.21 404.31 T 0 12 Q (P) 324.95 411.06 T (r) 331.62 411.06 T 2 F (X) 339.61 411.06 T 0 9 Q (3) 347.4 406.86 T 2 12 Q (x) 366.65 411.06 T (D) 382.17 450.47 T 0 9 Q (1) 391.29 446.27 T 2 12 Q (d) 410.54 450.47 T 0 9 Q (1) 417 446.27 T 0 12 Q (=) 399.78 450.47 T 2 F (Y) 383.5 430.22 T 0 9 Q (1) 390.63 426.02 T 2 12 Q (y) 409.88 430.22 T 0 9 Q (1) 415.67 426.02 T 0 12 Q (=) 399.12 430.22 T 2 F (D) 382.17 409.98 T 0 9 Q (2) 391.29 405.78 T 2 12 Q (d) 410.54 409.98 T 0 9 Q (2) 417 405.78 T 0 12 Q (=) 399.78 409.98 T 2 F (Y) 383.5 389.73 T 0 9 Q (2) 390.63 385.53 T 2 12 Q (y) 409.88 389.73 T 0 9 Q (2) 415.67 385.53 T 0 12 Q (=) 399.12 389.73 T 2 F (D) 382.17 369.48 T 0 9 Q (3) 391.29 365.28 T 2 12 Q (d) 410.54 369.48 T 0 9 Q (3) 417 365.28 T 0 12 Q (=) 399.78 369.48 T (=) 355.89 411.06 T (\050) 335.61 411.06 T (\051) 425.76 411.06 T 2 F (V) 430.46 411.06 T 0 9 Q (3) 438.25 406.86 T 2 12 Q (X) 446.74 411.06 T 0 9 Q (3) 454.53 406.86 T 2 12 Q (x) 473.79 411.06 T 0 F (=) 463.02 411.06 T (\050) 442.75 411.06 T (\051) 479.11 411.06 T 2 9 Q (x) 297.92 394.39 T 3 F (W) 312.83 394.39 T 2 6 Q (X) 320.08 391.84 T 3 9 Q (\316) 304.16 394.39 T 3 18 Q (\345) 304.42 407.86 T 0 12 Q (=) 255.49 411.06 T 194.12 404.61 194.12 418.86 2 L 0.54 H 2 Z N 202.56 373.16 198.97 373.16 198.97 454.14 3 L N 198.97 454.14 202.56 454.14 2 L N 239.5 373.16 243.1 373.16 243.1 454.14 3 L N 243.1 454.14 239.5 454.14 2 L N 374.38 408.06 374.38 418.86 2 L N 382.83 363.03 379.23 363.03 379.23 464.27 3 L N 379.23 464.27 382.83 464.27 2 L N 419.76 363.03 423.36 363.03 423.36 464.27 3 L N 423.36 464.27 419.76 464.27 2 L N 0 0 612 792 C 108 300.99 159.74 320.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 109.66 310.64 T (r) 116.33 310.64 T 2 F (Y) 124.32 310.64 T 0 9 Q (2) 131.45 306.44 T 2 12 Q (X) 140.8 310.64 T 0 9 Q (2) 148.59 306.44 T 0 12 Q (\050) 120.33 310.64 T (\051) 153.09 310.64 T 138.35 304.19 138.35 318.44 2 L 0.54 H 2 Z N 0 0 612 792 C 123.04 134.39 508.3 280.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 130.05 244.28 T 2 F (V) 141.38 244.28 T 0 9 Q (2) 149.16 240.08 T (3) 167.04 240.08 T 3 F (\256) 155.91 240.08 T 2 12 Q (D) 181.72 253.32 T 0 9 Q (1) 190.84 249.12 T 2 12 Q (d) 210.1 253.32 T 0 9 Q (1) 216.55 249.12 T 0 12 Q (=) 199.34 253.32 T 2 F (Y) 183.06 233.07 T 0 9 Q (1) 190.18 228.87 T 2 12 Q (y) 209.44 233.07 T 0 9 Q (1) 215.22 228.87 T 0 12 Q (=) 198.68 233.07 T (\050) 137.38 244.28 T (\051) 225.31 244.28 T (m) 248.07 244.28 T (a) 257.41 244.28 T (x) 262.73 244.28 T 2 9 Q (d) 269.19 240.08 T 0 6 Q (2) 274.03 237.53 T 0 12 Q (P) 304.77 244.28 T (r) 311.44 244.28 T 2 F (X) 319.43 244.28 T 0 9 Q (2) 327.22 240.08 T 2 12 Q (x) 346.47 244.28 T (D) 361.98 263.44 T 0 9 Q (1) 371.11 259.24 T 2 12 Q (d) 390.36 263.44 T 0 9 Q (1) 396.82 259.24 T 0 12 Q (=) 379.6 263.44 T 2 F (Y) 363.32 243.19 T 0 9 Q (1) 370.45 238.99 T 2 12 Q (y) 389.7 243.19 T 0 9 Q (1) 395.48 238.99 T 0 12 Q (=) 378.94 243.19 T 2 F (D) 361.98 222.95 T 0 9 Q (2) 371.11 218.75 T 2 12 Q (d) 390.36 222.95 T 0 9 Q (2) 396.82 218.75 T 0 12 Q (=) 379.6 222.95 T (=) 335.71 244.28 T (\050) 315.43 244.28 T (\051) 405.57 244.28 T 3 F (\264) 412.57 244.28 T 2 9 Q (x) 277.74 227.61 T 3 F (W) 292.65 227.61 T 2 6 Q (X) 299.9 225.06 T 3 9 Q (\316) 283.98 227.61 T 3 18 Q (\345) 284.23 241.07 T 0 12 Q (=) 235.3 244.28 T 173.94 237.83 173.94 252.08 2 L 0.54 H 2 Z N 182.38 226.62 178.79 226.62 178.79 267.11 3 L N 178.79 267.11 182.38 267.11 2 L N 219.32 226.62 222.91 226.62 222.91 267.11 3 L N 222.91 267.11 219.32 267.11 2 L N 354.2 241.28 354.2 252.08 2 L N 362.64 216.5 359.05 216.5 359.05 277.24 3 L N 359.05 277.24 362.64 277.24 2 L N 399.58 216.5 403.17 216.5 403.17 277.24 3 L N 403.17 277.24 399.58 277.24 2 L N 2 F (V) 200.88 176.69 T 0 9 Q (2) 208.67 172.49 T 2 12 Q (X) 217.17 176.69 T 0 9 Q (2) 224.96 172.49 T 2 12 Q (x) 244.21 176.69 T 0 F (=) 233.45 176.69 T (\050) 213.17 176.69 T (\051) 249.54 176.69 T (P) 300.3 176.69 T (r) 306.97 176.69 T 2 F (Y) 314.96 176.69 T 0 9 Q (2) 322.09 172.49 T 2 12 Q (y) 341.35 176.69 T (X) 351.52 176.69 T 0 9 Q (2) 359.31 172.49 T 2 12 Q (x) 378.56 176.69 T 0 F (=) 367.8 176.69 T (=) 330.58 176.69 T (\050) 310.97 176.69 T (\051) 383.89 176.69 T (E) 391.6 176.69 T 2 F (V) 402.92 176.69 T 0 9 Q (3) 410.71 172.49 T (3) 428.59 172.49 T 3 F (\256) 417.46 172.49 T 2 12 Q (D) 443.27 205.98 T 0 9 Q (1) 452.39 201.78 T 2 12 Q (d) 471.65 205.98 T 0 9 Q (1) 478.1 201.78 T 0 12 Q (=) 460.89 205.98 T 2 F (Y) 444.6 185.73 T 0 9 Q (1) 451.73 181.53 T 2 12 Q (y) 470.99 185.73 T 0 9 Q (1) 476.77 181.53 T 0 12 Q (=) 460.23 185.73 T 2 F (D) 443.27 165.49 T 0 9 Q (2) 452.39 161.29 T 2 12 Q (d) 471.65 165.49 T 0 9 Q (2) 478.1 161.29 T 0 12 Q (=) 460.89 165.49 T 2 F (Y) 447.08 145.24 T 0 9 Q (2) 454.21 141.04 T 2 12 Q (y) 473.46 145.24 T 0 F (=) 462.7 145.24 T (\050) 398.93 176.69 T (\051) 486.86 176.69 T 2 9 Q (y) 273.45 160.02 T 3 F (W) 288.51 160.02 T 2 6 Q (Y) 295.76 157.47 T 0 9 Q (=) 280.44 160.02 T 3 18 Q (\345) 279.86 173.49 T 0 12 Q (+) 256.54 176.69 T 349.07 173.69 349.07 184.49 2 L N 435.49 170.24 435.49 184.49 2 L N 443.93 138.79 440.33 138.79 440.33 219.78 3 L N 440.33 219.78 443.93 219.78 2 L N 480.86 138.79 484.46 138.79 484.46 219.78 3 L N 484.46 219.78 480.86 219.78 2 L N 272.3 138.79 268.7 138.79 268.7 219.78 3 L N 268.7 219.78 272.3 219.78 2 L N 491.41 138.79 495.01 138.79 495.01 219.78 3 L N 495.01 219.78 491.41 219.78 2 L N 199.73 138.79 196.13 138.79 196.13 219.78 3 L N 196.13 219.78 199.73 219.78 2 L N 497.96 138.79 501.56 138.79 501.56 219.78 3 L N 501.56 219.78 497.96 219.78 2 L N 0 0 612 792 C 374.66 89.33 424.94 106.27 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 375.59 97.06 T (r) 382.27 97.06 T 2 F (Y) 390.26 97.06 T 0 9 Q (1) 397.39 92.86 T 2 12 Q (X) 406.73 97.06 T 0 9 Q (1) 414.52 92.86 T 0 12 Q (\050) 386.26 97.06 T (\051) 419.02 97.06 T 404.28 90.61 404.28 104.86 2 L 0.54 H 2 Z N 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "19" 13 %%Page: "18" 14 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (18) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (W) 108 712 T (e de\336ne the) 118.37 712 T 2 F (optimal expected discounted cumulative value) 176.69 712 T 0 F ( as) 465.28 712 T 0 10 Q (\05021\051) 523.34 686 T 0 12 Q (which can be shown to satisfy the following) 108 656.35 T 2 F (optimality equation) 322.64 656.35 T 0 10 Q (\05022\051) 523.34 625.72 T 0 12 Q (The) 108 584 T 2 F (optimal policy) 129.66 584 T 0 F ( is de\336ned by) 215.54 584 T 0 10 Q (\05023\051) 523.34 558 T 0 12 Q -0.33 (The planning problem for in\336nite-horizon Markov decision processes is to \336nd an optimal) 108 526.62 P (policy or at least a policy that reasonably approximates the optimal expected discounted) 108 512.62 T (cumulative value. For the completely observable case, there are exact algorithms that are) 108 498.62 T (polynomial in the size of the state and action spaces. One of the best known algorithms is) 108 484.62 T (due to Howard [1960] and is known as) 108 470.62 T 2 F (policy iteration.) 297.3 470.62 T 0 F (Policy iteration jumps back and) 376.97 470.62 T (forth between a) 108 456.62 T 2 F (value determination) 185.64 456.62 T 0 F (phase in which the current policy is evaluated and a) 284.96 456.62 T 2 F (policy impr) 108 442.62 T (ovement) 162.55 442.62 T 0 F ( phase in which an attempt is made to improve the current policy) 202.54 442.62 T (. Pol-) 513.05 442.62 T (icy improvement can be performed in) 108 428.62 T ( steps, value determination in) 368.13 428.62 T ( steps by solving a system of linear equations, and there are at most) 154.25 414.62 T ( itera-) 505.2 414.62 T (tions required to \336nd the optimal policy) 108 400.62 T (.) 297.89 400.62 T (An alternative algorithm due to Bellman [1961] is called) 108 374.62 T 2 F (value iteration) 382.97 374.62 T 0 F ( and works by) 453.3 374.62 T (computing the optimal value function for) 108 360.62 T ( stages. The value functions so com-) 347.65 360.62 T (puted are guaranteed to conver) 108 346.62 T (ge in the limit to the value function for the optimal policy) 255.73 346.62 T (and in practice they often conver) 108 332.62 T (ge in a small number of stages. The value function for) 265.4 332.62 T (stage) 108 318.62 T ( is computed from the value function at stage) 144.66 318.62 T ( in) 391.29 318.62 T ( steps. The) 482.8 318.62 T (methods described above for exploiting structure in state and decision spaces can be used) 108 304.62 T (to reduce the computational overhead of both policy and value iteration. The problem of) 108 290.62 T (computing an optimal policy for an in\336nite-horizon Markov decision process can also be) 108 276.62 T (formulated as a linear program [Derman, 1970]. The advantage of such a formulation is) 108 262.62 T -0.05 (that constraints can be added to the linear program to solve a wider class of problems; the) 108 248.62 P (disadvantage is that the formulation does not exploit separability in time.) 108 234.62 T 1 14 Q (6.3 Partially Observable Markov Decision Pr) 108 201.29 T (ocesses) 380.34 201.29 T 0 12 Q -0.32 (Partial observability introduces additional sources of complexity) 108 174.62 P -0.32 (. In the following section,) 415.27 174.62 P (we consider the \336nite-horizon case but the discussion can be easily extended to the in\336-) 108 160.62 T (nite-horizon case. Figure) 108 146.62 T (20 shows a partially observable decision process in which the) 230.64 146.62 T -0.18 (control system can observe) 108 132.62 P -0.18 ( at time) 253.84 132.62 P -0.18 ( but not the state) 298.64 132.62 P -0.18 (, though the state and observa-) 394.26 132.62 P -0.52 (tion are related by the conditional probability distribution) 108 118.62 P -0.52 (. The optimal observer) 430.43 118.62 P (tries to estimate the relevant parts of the state from a sequence of observations, but this) 108 104.62 T (requires quanti\336cation over sequences of observations rather than individual states.) 108 90.62 T 400.98 702.35 465.28 722 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 405.7 712 T 3 F (S) 417.03 712 T 3 9 Q (g) 424.59 707.8 T 2 12 Q (X) 433.14 712 T (i) 455.22 712 T 0 F (=) 444.46 712 T (\050) 413.04 712 T (\051) 458.56 712 T 430.69 705.55 430.69 719.8 2 L 0.54 H 2 Z N 0 0 612 792 C 197.83 676.35 433.51 696 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 201.83 686 T 3 F (S) 213.15 686 T 3 9 Q (g) 220.71 681.8 T 2 12 Q (X) 229.26 686 T (i) 251.35 686 T 0 F (=) 240.59 686 T (\050) 209.16 686 T (\051) 254.68 686 T (m) 277.44 686 T (a) 286.78 686 T (x) 292.11 686 T 3 9 Q (d) 298.56 681.8 T 0 12 Q (E) 306.71 686 T 3 9 Q (d) 314.5 681.8 T 3 12 Q (S) 322.94 686 T 3 9 Q (g) 330.5 681.8 T 2 12 Q (X) 339.05 686 T (i) 361.14 686 T 0 F (=) 350.38 686 T (\050) 318.95 686 T (\051) 364.47 686 T (=) 264.68 686 T 2 F (i) 398.45 686 T 3 F (W) 416.34 686 T 2 9 Q (X) 426.01 681.8 T 3 12 Q (\316) 404.79 686 T 226.81 679.55 226.81 693.8 2 L 0.54 H 2 Z N 336.6 679.55 336.6 693.8 2 L N 0 0 612 792 C 135.26 604 496.08 640.35 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 139.26 625.72 T 3 F (S) 150.59 625.72 T 3 9 Q (g) 158.15 621.52 T 2 12 Q (X) 166.69 625.72 T (i) 188.78 625.72 T 0 F (=) 178.02 625.72 T (\050) 146.59 625.72 T (\051) 192.12 625.72 T (m) 214.88 625.72 T (a) 224.21 625.72 T (x) 229.54 625.72 T 2 9 Q (d) 235.99 621.52 T 2 12 Q (V) 248.35 625.72 T (X) 259.68 625.72 T (i) 281.77 625.72 T 0 F (=) 271.01 625.72 T (\050) 255.68 625.72 T (\051) 285.11 625.72 T 3 F (g) 301.86 625.72 T 2 F (p) 333.04 625.72 T 2 9 Q (i) 339.5 621.52 T (j) 342.53 621.52 T 2 12 Q (d) 349.03 625.72 T 0 F (\050) 345.03 625.72 T (\051) 355.03 625.72 T (E) 362.73 625.72 T 3 9 Q (d) 370.52 621.52 T 3 12 Q (S) 378.96 625.72 T 3 9 Q (g) 386.52 621.52 T 2 12 Q (X) 395.06 625.72 T (j) 417.15 625.72 T 0 F (=) 406.39 625.72 T (\050) 374.96 625.72 T (\051) 420.49 625.72 T 2 9 Q (j) 307.51 609.05 T 3 F (W) 320.92 609.05 T 2 6 Q (X) 328.17 606.5 T 3 9 Q (\316) 312.26 609.05 T 3 18 Q (\345) 313.26 622.51 T 0 12 Q (+) 292.1 625.72 T (=) 202.11 625.72 T 2 F (i) 461.02 625.72 T 3 F (W) 478.91 625.72 T 2 9 Q (X) 488.58 621.52 T 3 12 Q (\316) 467.35 625.72 T 164.25 619.27 164.25 633.52 2 L 0.54 H 2 Z N 392.62 619.27 392.62 633.52 2 L N 247.2 612.17 243.6 612.17 243.6 636.95 3 L N 243.6 636.95 247.2 636.95 2 L N 425.04 612.17 428.64 612.17 428.64 636.95 3 L N 428.64 636.95 425.04 636.95 2 L N 0 0 612 792 C 201.66 579 215.54 598.95 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (d) 202.66 584 T 0 9 Q (*) 209.04 591.2 T 0 0 612 792 C 209.29 546.62 422.05 568 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 213.29 558 T 3 9 Q (d) 221.08 550.87 T 0 6 Q (*) 225.87 556.05 T 3 12 Q (S) 232.86 558 T 3 9 Q (g) 240.42 553.8 T 2 12 Q (X) 248.97 558 T (i) 271.06 558 T 0 F (=) 260.3 558 T (\050) 228.87 558 T (\051) 274.39 558 T (E) 300.15 558 T 3 F (S) 311.48 558 T 3 9 Q (g) 319.04 553.8 T 2 12 Q (X) 327.58 558 T (i) 349.67 558 T 0 F (=) 338.91 558 T (\050) 307.48 558 T (\051) 353.01 558 T (=) 284.39 558 T 2 F (i) 386.99 558 T 3 F (W) 404.88 558 T 2 9 Q (X) 414.55 553.8 T 3 12 Q (\316) 393.32 558 T 246.52 551.55 246.52 565.8 2 L 0.54 H 2 Z N 325.14 551.55 325.14 565.8 2 L N 0 0 612 792 C 291.96 420.17 368.13 438.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (O) 293.38 428.62 T 3 F (W) 308.97 428.62 T 2 9 Q (D) 318.65 424.42 T 3 12 Q (W) 343.61 428.62 T 2 9 Q (X) 353.28 424.42 T 3 12 Q (\264) 331.08 428.62 T 0 F (\050) 302.04 428.62 T (\051) 361.72 428.62 T 307.23 423.37 307.23 437.62 2 L 0.54 H 2 Z N 326.34 423.37 326.34 437.62 2 L N 341.87 423.37 341.87 437.62 2 L N 359.98 423.37 359.98 437.62 2 L N 0 0 612 792 C 108 406.17 154.25 429.57 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (O) 109.3 414.62 T 3 F (W) 124.9 414.62 T 2 9 Q (X) 134.57 410.42 T 0 F (2) 143.46 421.82 T 0 12 Q (\050) 117.96 414.62 T (\051) 147.96 414.62 T 123.16 409.37 123.16 423.62 2 L 0.54 H 2 Z N 141.27 409.37 141.27 423.62 2 L N 0 0 612 792 C 480.55 406.17 505.19 424.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 484.79 414.62 T 2 9 Q (X) 494.46 410.42 T 483.05 409.37 483.05 423.62 2 L 0.54 H 2 Z N 501.15 409.37 501.15 423.62 2 L N 0 0 612 792 C 308.65 355.62 347.65 370.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (1) 309.65 360.62 T (2) 321.65 360.62 T 3 F (\274) 333.65 360.62 T (,) 315.65 360.62 T (,) 327.65 360.62 T 0 0 612 792 C 135.66 313.62 144.66 328.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 136.66 318.62 T 0 0 612 792 C 364.3 313.62 391.29 328.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 365.3 318.62 T 0 F (1) 383.29 318.62 T (\320) 374.29 318.62 T 0 0 612 792 C 406.63 310.17 482.8 328.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (O) 408.04 318.62 T 3 F (W) 423.64 318.62 T 2 9 Q (D) 433.32 314.42 T 3 12 Q (W) 458.27 318.62 T 2 9 Q (X) 467.95 314.42 T 3 12 Q (\264) 445.75 318.62 T 0 F (\050) 416.71 318.62 T (\051) 476.39 318.62 T 421.9 313.37 421.9 327.62 2 L 0.54 H 2 Z N 441.01 313.37 441.01 327.62 2 L N 456.54 313.37 456.54 327.62 2 L N 474.65 313.37 474.65 327.62 2 L N 0 0 612 792 C 240.59 124.18 253.84 142.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (Y) 241.9 132.62 T 2 9 Q (t) 249.03 128.42 T 0 0 612 792 C 292.3 127.62 298.64 142.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 293.3 132.62 T 0 0 612 792 C 380.07 124.18 394.26 142.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 381.52 132.62 T 2 9 Q (t) 389.31 128.42 T 0 0 612 792 C 382.5 108.98 430.43 128.62 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 384.25 118.62 T (r) 390.92 118.62 T 2 F (Y) 398.92 118.62 T 2 9 Q (t) 406.04 114.42 T 2 12 Q (X) 413.39 118.62 T 2 9 Q (t) 421.18 114.42 T 0 12 Q (\050) 394.92 118.62 T (\051) 423.68 118.62 T 410.94 112.17 410.94 126.42 2 L 0.54 H 2 Z N 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "18" 14 %%Page: "17" 15 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (17) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 1 10 Q (Figur) 121.75 713.33 T (e 19. Action-separable decision pr) 145.46 713.33 T (oblem) 289.43 713.33 T 0 12 Q -0.05 (It is worth noting that by restricting the action space at each stage in a sequential decision) 108 473.03 P (problem we also restrict access to states. Much of the work in arti\336cial intelligence seeks) 108 459.03 T (to reduce search in computing policies by exploiting this fact.) 108 445.03 T 1 14 Q (6.2 In\336nite-Horizon Markov Decision Pr) 108 411.7 T (ocesses) 353.52 411.7 T 0 12 Q (So far we have focused on) 108 385.03 T (-stage \050\336nite-horizon\051 sequential decision problems. The) 246.96 385.03 T -0.21 (extension to in\336nite-horizon problems is not as dif) 108 371.03 P -0.21 (\336cult as it might seem. In the most stud-) 348.66 371.03 P -0.04 (ied version of the in\336nite-horizon problem, we assume that the value of a state is indepen-) 108 357.03 P -0.25 (dent of time so that) 108 343.03 P -0.25 ( for any) 248.77 343.03 P -0.25 ( and) 294.69 343.03 P -0.25 (. Since) 327.53 343.03 P -0.25 ( is independent of time, we use) 376.65 343.03 P (to represent that value function for all times. There are three basic decision models for) 108 329.03 T -0.19 (in\336nite-horizon problems: \050i\051 expected value per time step, \050ii\051 expected cumulative value) 108 315.03 P (until some goal state is reached, and \050iii\051 expected discounted cumulative value in which) 108 301.03 T (we assume that the value of being in state) 108 287.03 T ( after) 319.29 287.03 T ( stages is) 356.28 287.03 T ( where) 468.05 287.03 T -0.07 ( is called the) 153.97 273.03 P 2 F -0.07 (discount factor) 217.02 273.03 P 0 F -0.07 (. W) 288.63 273.03 P -0.07 (e concentrate on the discounted case, noting that) 304.93 273.03 P (the other two models involve relatively straight forward extensions.) 108 259.03 T -0.13 (W) 108 233.03 P -0.13 (e assume a discrete state space) 118.37 233.03 P -0.13 ( in which states are represented as integers. For each) 286.33 233.03 P (decision \050action\051, we de\336ne the associated state transition probabilities as follows.) 108 219.03 T 0 10 Q (\05019\051) 523.34 193.03 T 0 12 Q (A policy) 108 163.38 T ( is a mapping from states to decisions. The expected discounted cumulative) 161.59 163.38 T (value with respect to a state) 108 149.38 T ( for a particular policy) 250.31 149.38 T ( and \336xed) 369.54 149.38 T ( is de\336ned by the fol-) 427.8 149.38 T (lowing recurrence equation.) 108 135.38 T 0 10 Q (\05020\051) 523.34 105.65 T 108 63 540 720 C 192.16 497.03 455.84 710 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0.5 H 2 Z 0 X 0 0 0 1 0 0 0 K 0 90 18 36 352.5 561.67 A 0 90 18 36 424.5 561.67 A 372.91 563.59 370.5 554.53 367.74 563.5 370.33 563.55 4 Y V 6.46 90 27 80.14 343.5 554.53 A 445.61 563.55 443.21 554.53 440.47 563.46 443.04 563.51 4 Y V 6.38 90 27.71 80.85 415.5 554.53 A 371.63 646.7 379.49 648.17 373.6 642.76 372.61 644.73 4 Y V 316.5 616.67 372.62 644.73 2 L N 299.63 646.7 307.49 648.17 301.6 642.76 300.62 644.73 4 Y V 244.5 616.67 300.62 644.73 2 L N 267 678.67 285 696.67 R 3 X V 0 Z 0 X N 339 679.67 357 697.67 R 7 X V 0 X N 343.76 650.48 348 643.67 340.79 647.2 342.27 648.84 4 Y V 2 Z 35.09 90 31.5 9 316.5 643.67 A 340.79 622.14 348 625.67 343.76 618.85 342.27 620.5 4 Y V 270 324.91 31.5 9 316.5 625.67 A 340.79 586.14 348 589.67 343.76 582.86 342.27 584.5 4 Y V 270 324.91 31.5 9 316.5 589.67 A 371 554.35 362 540.85 371 527.35 380 540.85 4 Y 3 X V 0 Z 0 X N 443 554.35 434 540.85 443 527.35 452 540.85 4 Y 7 X V 0 X N 407 527.35 398 513.85 407 500.35 416 513.85 4 Y 7 X V 0 X N 390.23 516.73 397.99 513.85 389.89 512.18 390.06 514.45 4 Y V 2 Z 180 252.91 27 13.5 398 527.35 A 424.1 512.18 415.99 513.85 423.76 516.73 423.93 514.45 4 Y V 287.09 360 27 13.5 416 527.35 A 412.79 586.14 420 589.67 415.75 582.86 414.27 584.5 4 Y V 270 324.91 31.5 9 388.5 589.67 A 415.75 650.48 420 643.67 412.79 647.2 414.27 648.84 4 Y V 35.09 90 31.5 9 388.5 643.67 A 412.79 622.14 420 625.67 415.75 618.85 414.27 620.5 4 Y V 270 324.91 31.5 9 388.5 625.67 A 3 12 Q (S) 403.39 510.59 T 299.81 654.87 307.5 652.67 299.81 650.46 299.81 652.67 4 Y V 244.5 652.67 299.81 652.67 2 L N 299.81 618.87 307.5 616.67 299.81 614.46 299.81 616.67 4 Y V 244.5 616.67 299.81 616.67 2 L N 299.81 582.88 307.5 580.67 299.81 578.46 299.81 580.67 4 Y V 244.5 580.67 299.81 580.67 2 L N 371.81 618.87 379.5 616.67 371.81 614.46 371.81 616.67 4 Y V 316.5 616.67 371.81 616.67 2 L N 371.81 654.87 379.5 652.67 371.81 650.46 371.81 652.67 4 Y V 316.5 652.67 371.81 652.67 2 L N 371.81 582.88 379.5 580.67 371.81 578.46 371.81 580.67 4 Y V 316.5 580.67 371.81 580.67 2 L N 267 543.67 285 561.67 R 3 X V 0 Z 0 X N 339 543.67 357 561.67 R 7 X V 0 X N 323.91 576.36 316.5 579.67 324.54 580.79 324.22 578.58 4 Y V 2 Z 0 69.91 22.5 18 316.5 561.67 A 322.99 610.61 316.5 615.67 324.69 614.82 323.84 612.71 4 Y V 0 70.95 22.5 54 316.5 561.67 A 341.63 571.35 339 561.67 336.1 571.28 338.87 571.32 4 Y V 6.15 90 22.5 90 316.5 561.67 A 249.99 645.76 244.5 651.67 252.28 649.57 251.13 647.67 4 Y V 0 72.85 22.5 90 244.5 561.67 A 269.15 570.24 267 561.67 264.28 570.08 266.72 570.16 4 Y V 9.05 90 22.5 54 244.5 561.67 A 251.91 576.36 244.5 579.67 252.54 580.79 252.23 578.58 4 Y V 0 69.91 22.5 18 244.5 561.67 A 299.48 576.57 307.5 575.17 299.96 572.11 299.72 574.34 4 Y V 110.23 180 22.5 13.5 307.5 561.67 A 270 360 22.5 27 316.5 678.67 A 270 360 22.5 63 316.5 678.67 A 336.07 668.87 338.99 678.67 341.7 668.81 338.88 668.84 4 Y V 270 354.3 22.5 99 316.5 678.67 A 270 360 22.5 27 244.5 678.67 A 270 360 22.5 63 244.5 678.67 A 264.07 668.87 267 678.67 269.7 668.81 266.89 668.84 4 Y V 270 354.3 22.5 99 244.5 678.67 A 371.48 576.57 379.5 575.17 371.96 572.11 371.72 574.34 4 Y V 110.23 180 22.5 13.5 379.5 561.67 A 7 X 90 450 9 9 244.5 652.67 G 0 Z 0 X 90 450 9 9 244.5 652.67 A 7 X 90 450 9 9 244.5 616.67 G 0 X 90 450 9 9 244.5 616.67 A 7 X 90 450 9 9 244.5 580.67 G 0 X 90 450 9 9 244.5 580.67 A 3 X 90 450 9 9 316.5 652.67 G 0 X 90 450 9 9 316.5 652.67 A 3 X 90 450 9 9 316.5 616.67 G 0 X 90 450 9 9 316.5 616.67 A 3 X 90 450 9 9 316.5 580.67 G 0 X 90 450 9 9 316.5 580.67 A 7 X 90 450 9 9 388.5 652.67 G 0 X 90 450 9 9 388.5 652.67 A 7 X 90 450 9 9 388.5 616.67 G 0 X 90 450 9 9 388.5 616.67 A 7 X 90 450 9 9 388.5 580.67 G 0 X 90 450 9 9 388.5 580.67 A 352.5 648.17 343.5 634.67 352.5 621.17 361.5 634.67 4 Y 3 X V 0 X N 424.5 648.17 415.5 634.67 424.5 621.17 433.5 634.67 4 Y 7 X V 0 X N 424.5 612.17 415.5 598.67 424.5 585.17 433.5 598.67 4 Y 7 X V 0 X N 300.07 659.82 307.5 656.17 299.26 655.33 299.66 657.58 4 Y V 2 Z 180 249.62 22.5 22.5 307.5 678.67 A 372.07 659.82 379.5 656.17 371.26 655.33 371.66 657.58 4 Y V 180 249.62 22.5 22.5 379.5 678.67 A 351.83 610.17 342.83 596.67 351.83 583.17 360.83 596.67 4 Y 3 X V 0 Z 0 X N 2 9 Q (t) 234.25 701.5 T 0 F (0) 250.82 701.5 T (=) 241.24 701.5 T 2 F (t) 377.92 701.5 T 0 F (2) 394.49 701.5 T (=) 384.92 701.5 T 2 F (t) 306.08 701.5 T 0 F (1) 322.65 701.5 T (=) 313.08 701.5 T 2 12 Q (D) 216.78 678.09 T 0 9 Q (1) 225.9 673.89 T 2 12 Q (X) 216.78 657.01 T 0 9 Q (1) 224.57 652.81 T 2 12 Q (V) 216.78 635.94 T 0 9 Q (1) 224.57 631.74 T 2 12 Q (X) 216.78 614.86 T 0 9 Q (2) 224.57 610.66 T 2 12 Q (X) 216.78 572.71 T 0 9 Q (3) 224.57 568.51 T 2 12 Q (V) 216.78 593.78 T 0 9 Q (2) 224.57 589.59 T 2 12 Q (D) 216.78 551.63 T 0 9 Q (2) 225.9 547.43 T 2 12 Q (W) 216.78 528.68 T 108 63 540 720 C 0 0 612 792 C 237.96 380.03 246.96 395.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 238.96 385.03 T 0 0 612 792 C 202.44 334.58 248.77 353.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (V) 204.05 343.03 T 2 9 Q (t) 211.84 338.83 T 2 12 Q (V) 233.11 343.03 T 2 9 Q (t) 240.89 338.83 T 3 F (\242) 243.93 338.83 T 0 12 Q (=) 220.34 343.03 T 0 0 612 792 C 288.35 338.03 294.69 353.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 289.35 343.03 T 0 0 612 792 C 317.52 338.03 327.53 353.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 318.52 343.03 T 3 F (\242) 322.56 343.03 T 0 0 612 792 C 362.7 334.58 376.65 353.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (V) 364.03 343.03 T 2 9 Q (t) 371.82 338.83 T 0 0 612 792 C 526.92 338.03 537.25 353.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (V) 527.91 343.03 T 0 0 612 792 C 310.97 282.03 319.29 297.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (x) 311.97 287.03 T 0 0 612 792 C 347.28 282.03 356.28 297.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 348.28 287.03 T 0 0 612 792 C 402.61 282.03 468.05 301.98 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (g) 404.96 287.03 T 2 9 Q (n) 410.34 294.23 T 2 12 Q (V) 415.55 287.03 T (X) 430.74 287.03 T (x) 452.83 287.03 T 0 F (=) 442.07 287.03 T 3 F (\050) 425.59 287.03 T (\051) 458.71 287.03 T 0 0 612 792 C 108 268.03 153.97 283.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (0) 109.44 273.03 T 3 F (g) 128.02 273.03 T 0 F (1) 145.54 273.03 T 3 F (\243) 118.43 273.03 T (\243) 135.95 273.03 T 0 0 612 792 C 267.56 224.58 286.33 243.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 268.86 233.03 T 2 9 Q (X) 278.54 228.83 T 0 0 612 792 C 152.4 183.38 478.95 203.03 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (p) 157.18 193.03 T 2 9 Q (i) 163.64 188.83 T (j) 166.67 188.83 T 2 12 Q (d) 173.17 193.03 T 0 F (\050) 169.18 193.03 T (\051) 179.17 193.03 T (P) 201.93 193.03 T (r) 208.6 193.03 T 2 F (X) 216.59 193.03 T 2 9 Q (t) 224.38 188.83 T 2 12 Q (j) 241.64 193.03 T 0 F (=) 230.88 193.03 T 2 F (X) 249.82 193.03 T 2 9 Q (t) 257.61 188.83 T 0 F (1) 269.11 188.83 T (\320) 262.36 188.83 T 2 12 Q (i) 288.36 193.03 T (D) 297.7 193.03 T 2 9 Q (t) 306.82 188.83 T 2 12 Q (d) 324.07 193.03 T 0 F (=) 313.31 193.03 T 3 F (,) 291.7 193.03 T 0 F (=) 277.6 193.03 T (\050) 212.6 193.03 T (\051) 330.07 193.03 T (=) 189.16 193.03 T 2 F (i) 364.05 193.03 T (j) 373.39 193.03 T 3 F (,) 367.39 193.03 T (W) 391.28 193.03 T 2 9 Q (X) 400.95 188.83 T 3 12 Q (\316) 379.72 193.03 T 2 F (d) 436.43 193.03 T 3 F (W) 456.99 193.03 T 2 9 Q (D) 466.66 188.83 T 3 12 Q (\316) 445.43 193.03 T 247.38 186.58 247.38 200.83 2 L 0.54 H 2 Z N 0 0 612 792 C 152.66 158.38 161.59 173.38 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (d) 153.66 163.38 T 0 0 612 792 C 243.97 144.38 250.31 159.38 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (i) 244.97 149.38 T 0 0 612 792 C 360.61 144.38 369.54 159.38 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (d) 361.61 149.38 T 0 0 612 792 C 419.86 144.38 427.8 159.38 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (g) 420.86 149.38 T 0 0 612 792 C 130.96 83.93 500.38 119.38 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 140.15 105.65 T 3 9 Q (d) 147.94 101.45 T 3 12 Q (S) 156.38 105.65 T 3 9 Q (g) 163.94 101.45 T 2 12 Q (X) 172.49 105.65 T (i) 194.57 105.65 T 0 F (=) 183.81 105.65 T (\050) 152.38 105.65 T (\051) 197.91 105.65 T 2 F (V) 220.67 105.65 T (X) 232 105.65 T (i) 254.08 105.65 T 0 F (=) 243.32 105.65 T (\050) 228 105.65 T (\051) 257.42 105.65 T 3 F (g) 274.18 105.65 T 2 F (p) 305.36 105.65 T 2 9 Q (i) 311.81 101.45 T (j) 314.84 101.45 T 3 12 Q (d) 321.34 105.65 T 2 F (X) 331.27 105.65 T (i) 353.35 105.65 T 0 F (=) 342.59 105.65 T (\050) 327.27 105.65 T (\051) 356.69 105.65 T (\050) 317.35 105.65 T (\051) 360.68 105.65 T (E) 368.39 105.65 T 3 9 Q (d) 376.18 101.45 T 3 12 Q (S) 384.62 105.65 T 3 9 Q (g) 392.18 101.45 T 2 12 Q (X) 400.72 105.65 T (j) 422.81 105.65 T 0 F (=) 412.05 105.65 T (\050) 380.62 105.65 T (\051) 426.15 105.65 T 2 9 Q (j) 279.82 88.98 T 3 F (W) 293.23 88.98 T 2 6 Q (X) 300.49 86.43 T 3 9 Q (\316) 284.57 88.98 T 3 18 Q (\345) 285.57 102.45 T 0 12 Q (+) 264.41 105.65 T (=) 207.9 105.65 T 2 F (i) 460.13 105.65 T 3 F (W) 478.02 105.65 T 2 9 Q (X) 487.69 101.45 T 3 12 Q (\316) 466.46 105.65 T 170.04 99.2 170.04 113.45 2 L 0.54 H 2 Z N 398.28 99.2 398.28 113.45 2 L N 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "17" 15 %%Page: "16" 16 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (16) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 10 Q (\05016\051) 523.34 649 T 0 12 Q (The \336rst term requires a summation over) 108 562.18 T (.) 358.6 562.18 T 0 10 Q (\05017\051) 523.34 480.65 T 0 12 Q (The second term requires a summation over) 108 408.08 T (.) 343.74 408.08 T 0 10 Q (\05018\051) 523.34 351.16 T 0 12 Q (The total work for this stage is) 108 301.78 T ( resulting in a small) 430.79 301.78 T (computational saving.) 108 287.78 T (W) 108 261.78 T (e can gain substantial savings if the actions decompose conveniently) 118.37 261.78 T (. Figure) 446.54 261.78 T (19 shows) 486.88 261.78 T (an action-, state- and time-separable sequential decision problem in which the computa-) 108 247.78 T (tions at each stage are) 108 233.78 T ( \050the random variables) 393.26 233.78 T (, and) 302.86 219.78 T ( are all shaded\051.) 347.64 219.78 T 158.83 582.18 472.51 720 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 164.69 649 T 2 F (W) 176.02 649 T 0 9 Q (2) 186.47 644.8 T (2) 204.35 644.8 T 3 F (\256) 193.22 644.8 T 2 12 Q (X) 219.03 668.16 T 0 9 Q (1) 226.82 663.96 T (1) 235.82 663.96 T 3 F (,) 231.32 663.96 T 2 12 Q (x) 255.07 668.16 T 0 9 Q (1) 260.86 663.96 T (1) 269.86 663.96 T 3 F (,) 265.36 663.96 T 0 12 Q (=) 244.31 668.16 T 2 F (X) 219.03 647.91 T 0 9 Q (2) 226.82 643.72 T (1) 235.82 643.72 T 3 F (,) 231.32 643.72 T 2 12 Q (x) 255.07 647.91 T 0 9 Q (2) 260.86 643.72 T (1) 269.86 643.72 T 3 F (,) 265.36 643.72 T 0 12 Q (=) 244.31 647.91 T 2 F (X) 219.03 627.67 T 0 9 Q (3) 226.82 623.47 T (1) 235.82 623.47 T 3 F (,) 231.32 623.47 T 2 12 Q (x) 255.07 627.67 T 0 9 Q (3) 260.86 623.47 T (1) 269.86 623.47 T 3 F (,) 265.36 623.47 T 0 12 Q (=) 244.31 627.67 T (\050) 172.03 649 T (\051) 278.61 649 T (m) 301.37 649 T (a) 310.71 649 T (x) 316.03 649 T 2 9 Q (d) 322.49 644.8 T 0 12 Q (E) 337.85 683.64 T 2 F (V) 349.18 683.64 T 0 9 Q (1) 356.96 679.44 T (2) 365.96 679.44 T (2) 383.84 679.44 T 3 F (\256) 372.71 679.44 T (,) 361.46 679.44 T 2 12 Q (X) 398.52 702.8 T 0 9 Q (1) 406.31 698.6 T (1) 415.3 698.6 T 3 F (,) 410.81 698.6 T 2 12 Q (x) 434.56 702.8 T 0 9 Q (1) 440.35 698.6 T (1) 449.34 698.6 T 3 F (,) 444.85 698.6 T 0 12 Q (=) 423.8 702.8 T 2 F (X) 398.52 682.56 T 0 9 Q (2) 406.31 678.36 T (1) 415.3 678.36 T 3 F (,) 410.81 678.36 T 2 12 Q (x) 434.56 682.56 T 0 9 Q (2) 440.35 678.36 T (1) 449.34 678.36 T 3 F (,) 444.85 678.36 T 0 12 Q (=) 423.8 682.56 T 2 F (D) 408.99 662.31 T 0 9 Q (2) 418.11 658.11 T 2 12 Q (d) 437.37 662.31 T 0 F (=) 426.61 662.31 T (\050) 345.18 683.64 T (\051) 458.1 683.64 T (+) 395.09 638.47 T (E) 337.85 604.23 T 2 F (V) 349.18 604.23 T 0 9 Q (2) 356.96 600.03 T (2) 365.96 600.03 T (2) 383.84 600.03 T 3 F (\256) 372.71 600.03 T (,) 361.46 600.03 T 2 12 Q (X) 398.52 613.27 T 0 9 Q (3) 406.31 609.07 T (1) 415.3 609.07 T 3 F (,) 410.81 609.07 T 2 12 Q (x) 434.56 613.27 T 0 9 Q (3) 440.35 609.07 T (1) 449.34 609.07 T 3 F (,) 444.85 609.07 T 0 12 Q (=) 423.8 613.27 T 2 F (D) 408.99 593.03 T 0 9 Q (2) 418.11 588.83 T 2 12 Q (d) 437.37 593.03 T 0 F (=) 426.61 593.03 T (\050) 345.18 604.23 T (\051) 458.1 604.23 T (=) 288.61 649 T 211.25 642.55 211.25 656.8 2 L 0.54 H 2 Z N 219.69 621.22 216.1 621.22 216.1 681.96 3 L N 216.1 681.96 219.69 681.96 2 L N 272.62 621.22 276.21 621.22 276.21 681.96 3 L N 276.21 681.96 272.62 681.96 2 L N 390.74 677.19 390.74 691.44 2 L N 399.18 655.86 395.58 655.86 395.58 716.6 3 L N 395.58 716.6 399.18 716.6 2 L N 452.1 655.86 455.7 655.86 455.7 716.6 3 L N 455.7 716.6 452.1 716.6 2 L N 390.74 597.78 390.74 612.03 2 L N 399.18 586.58 395.58 586.58 395.58 627.07 3 L N 395.58 627.07 399.18 627.07 2 L N 452.1 586.58 455.7 586.58 455.7 627.07 3 L N 455.7 627.07 452.1 627.07 2 L N 333.69 586.58 330.1 586.58 330.1 716.6 3 L N 330.1 716.6 333.69 716.6 2 L N 462.65 586.58 466.25 586.58 466.25 716.6 3 L N 466.25 716.6 462.65 716.6 2 L N 0 0 612 792 C 306.97 552.78 358.6 571.17 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 307.48 562.18 T 2 9 Q (X) 317.15 557.98 T 0 6 Q (1) 322.99 555.43 T 3 12 Q (W) 338.58 562.18 T 2 9 Q (X) 348.25 557.98 T 0 6 Q (2) 354.09 555.43 T 3 12 Q (\264) 328.99 562.18 T 0 0 612 792 C 113.65 428.08 517.69 546.18 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 120.78 480.65 T 2 F (V) 132.11 480.65 T 0 9 Q (1) 139.9 476.45 T (2) 148.9 476.45 T (2) 166.77 476.45 T 3 F (\256) 155.64 476.45 T (,) 144.4 476.45 T 2 12 Q (X) 181.46 499.81 T 0 9 Q (1) 189.24 495.61 T (1) 198.24 495.61 T 3 F (,) 193.74 495.61 T 2 12 Q (x) 217.5 499.81 T 0 9 Q (1) 223.28 495.61 T (1) 232.28 495.61 T 3 F (,) 227.78 495.61 T 0 12 Q (=) 206.73 499.81 T 2 F (X) 181.46 479.57 T 0 9 Q (2) 189.24 475.37 T (1) 198.24 475.37 T 3 F (,) 193.74 475.37 T 2 12 Q (x) 217.5 479.57 T 0 9 Q (2) 223.28 475.37 T (1) 232.28 475.37 T 3 F (,) 227.78 475.37 T 0 12 Q (=) 206.73 479.57 T 2 F (D) 189.45 459.32 T 0 9 Q (2) 198.57 455.12 T 2 12 Q (d) 217.82 459.32 T 0 9 Q (2) 224.28 455.12 T 0 12 Q (=) 207.06 459.32 T (\050) 128.11 480.65 T (\051) 241.03 480.65 T (P) 300.54 526.89 T (r) 307.21 526.89 T 2 F (X) 315.2 526.89 T 0 9 Q (1) 322.99 522.69 T (2) 331.99 522.69 T 3 F (,) 327.49 522.69 T 2 12 Q (x) 351.24 526.89 T 0 9 Q (1) 357.03 522.69 T (2) 366.02 522.69 T 3 F (,) 361.53 522.69 T 2 12 Q (X) 375.37 526.89 T 0 9 Q (1) 383.16 522.69 T (1) 392.15 522.69 T 3 F (,) 387.66 522.69 T 2 12 Q (x) 411.41 526.89 T 0 9 Q (1) 417.19 522.69 T (1) 426.19 522.69 T 3 F (,) 421.69 522.69 T 0 12 Q (=) 400.65 526.89 T 2 F (X) 436.69 526.89 T 0 9 Q (2) 444.48 522.69 T (1) 453.47 522.69 T 3 F (,) 448.98 522.69 T 2 12 Q (x) 472.73 526.89 T 0 9 Q (2) 478.51 522.69 T (1) 487.51 522.69 T 3 F (,) 483.01 522.69 T 0 12 Q (=) 461.97 526.89 T 3 F (,) 430.69 526.89 T 0 F (=) 340.48 526.89 T (\050) 311.2 526.89 T (\051) 492.01 526.89 T 3 F (\264) 394.98 504.25 T 0 F (P) 308.53 486.25 T (r) 315.2 486.25 T 2 F (X) 323.2 486.25 T 0 9 Q (2) 330.98 482.05 T (2) 339.98 482.05 T 3 F (,) 335.48 482.05 T 2 12 Q (x) 359.24 486.25 T 0 9 Q (2) 365.02 482.05 T (2) 374.02 482.05 T 3 F (,) 369.52 482.05 T 2 12 Q (X) 383.36 486.25 T 0 9 Q (2) 391.15 482.05 T (1) 400.15 482.05 T 3 F (,) 395.65 482.05 T 2 12 Q (x) 419.4 486.25 T 0 9 Q (2) 425.19 482.05 T (1) 434.18 482.05 T 3 F (,) 429.69 482.05 T 0 12 Q (=) 408.64 486.25 T 2 F (D) 444.68 486.25 T 0 9 Q (2) 453.8 482.05 T 2 12 Q (d) 473.06 486.25 T 0 9 Q (2) 479.52 482.05 T 0 12 Q (=) 462.3 486.25 T 3 F (,) 438.68 486.25 T 0 F (=) 348.48 486.25 T (\050) 319.2 486.25 T (\051) 484.02 486.25 T 3 F (\264) 394.98 463.61 T 2 F (V) 325.31 445.61 T 0 9 Q (1) 333.1 441.41 T (2) 342.1 441.41 T 3 F (,) 337.6 441.41 T 2 12 Q (X) 350.59 445.61 T 0 9 Q (1) 358.38 441.41 T (2) 367.38 441.41 T 3 F (,) 362.88 441.41 T 2 12 Q (x) 386.63 445.61 T 0 9 Q (1) 392.42 441.41 T (2) 401.42 441.41 T 3 F (,) 396.92 441.41 T 0 12 Q (=) 375.87 445.61 T 2 F (X) 411.91 445.61 T 0 9 Q (2) 419.7 441.41 T (2) 428.7 441.41 T 3 F (,) 424.2 441.41 T 2 12 Q (x) 447.95 445.61 T 0 9 Q (2) 453.74 441.41 T (2) 462.73 441.41 T 3 F (,) 458.24 441.41 T 0 12 Q (=) 437.19 445.61 T 3 F (,) 405.92 445.61 T 0 F (\050) 346.6 445.61 T (\051) 467.23 445.61 T 2 9 Q (x) 253.79 462.55 T 0 6 Q (1) 258.13 460 T (2) 264.13 460 T 3 F (,) 261.13 460 T 3 9 Q (W) 278.04 462.55 T 2 6 Q (X) 285.29 460 T 0 F (1) 289.19 456.7 T 3 9 Q (\316) 269.38 462.55 T 2 F (x) 253.79 443.96 T 0 6 Q (2) 258.13 441.41 T (2) 264.13 441.41 T 3 F (,) 261.13 441.41 T 3 9 Q (W) 278.04 443.96 T 2 6 Q (X) 285.29 441.41 T 0 F (2) 289.19 438.11 T 3 9 Q (\316) 269.38 443.96 T 3 18 Q (\345) 266.57 477.44 T 0 12 Q (=) 251.03 480.65 T 173.67 474.2 173.67 488.45 2 L 0.54 H 2 Z N 182.11 452.87 178.52 452.87 178.52 513.61 3 L N 178.52 513.61 182.11 513.61 2 L N 235.04 452.87 238.63 452.87 238.63 513.61 3 L N 238.63 513.61 235.04 513.61 2 L N 372.92 520.44 372.92 534.69 2 L N 380.92 479.8 380.92 494.05 2 L N 299.38 432.48 295.79 432.48 295.79 538.49 3 L N 295.79 538.49 299.38 538.49 2 L N 496.56 432.48 500.16 432.48 500.16 538.49 3 L N 500.16 538.49 496.56 538.49 2 L N 0 0 612 792 C 321.62 397.84 343.74 418.08 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 322.92 408.08 T 2 9 Q (X) 332.6 403.89 T 0 6 Q (3) 338.44 401.33 T 0 0 612 792 C 121.98 321.78 509.36 392.08 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 128.78 351.16 T 2 F (V) 140.11 351.16 T 0 9 Q (2) 147.89 346.96 T (2) 156.89 346.96 T (2) 174.77 346.96 T 3 F (\256) 163.64 346.96 T (,) 152.39 346.96 T 2 12 Q (X) 189.45 360.2 T 0 9 Q (3) 197.24 356 T (1) 206.24 356 T 3 F (,) 201.74 356 T 2 12 Q (x) 225.49 360.2 T 0 9 Q (3) 231.27 356 T (1) 240.27 356 T 3 F (,) 235.77 356 T 0 12 Q (=) 214.73 360.2 T 2 F (D) 197.45 339.96 T 0 9 Q (2) 206.57 335.76 T 2 12 Q (d) 225.82 339.96 T 0 9 Q (2) 232.28 335.76 T 0 12 Q (=) 215.06 339.96 T (\050) 136.11 351.16 T (\051) 249.03 351.16 T (P) 318.53 375.66 T (r) 325.2 375.66 T 2 F (X) 333.2 375.66 T 0 9 Q (3) 340.98 371.46 T (2) 349.98 371.46 T 3 F (,) 345.48 371.46 T 2 12 Q (x) 369.24 375.66 T 0 9 Q (3) 375.02 371.46 T (2) 384.02 371.46 T 3 F (,) 379.52 371.46 T 2 12 Q (X) 393.36 375.66 T 0 9 Q (3) 401.15 371.46 T (1) 410.15 371.46 T 3 F (,) 405.65 371.46 T 2 12 Q (x) 429.4 375.66 T 0 9 Q (3) 435.19 371.46 T (1) 444.19 371.46 T 3 F (,) 439.69 371.46 T 0 12 Q (=) 418.64 375.66 T 2 F (D) 454.68 375.66 T 0 9 Q (2) 463.8 371.46 T 2 12 Q (d) 483.06 375.66 T 0 9 Q (2) 489.52 371.46 T 0 12 Q (=) 472.3 375.66 T 3 F (,) 448.69 375.66 T 0 F (=) 358.48 375.66 T (\050) 329.2 375.66 T (\051) 494.02 375.66 T 3 F (\264) 404.98 353.01 T 2 F (V) 365.97 335.01 T 0 9 Q (2) 373.76 330.82 T (2) 382.76 330.82 T 3 F (,) 378.26 330.82 T 2 12 Q (X) 391.25 335.01 T 0 9 Q (3) 399.04 330.82 T (2) 408.04 330.82 T 3 F (,) 403.54 330.82 T 2 12 Q (x) 427.3 335.01 T 0 9 Q (3) 433.08 330.82 T (2) 442.08 330.82 T 3 F (,) 437.58 330.82 T 0 12 Q (=) 416.53 335.01 T (\050) 387.26 335.01 T (\051) 446.58 335.01 T 2 9 Q (x) 271.79 333.78 T 0 6 Q (3) 276.12 331.23 T (2) 282.12 331.23 T 3 F (,) 279.12 331.23 T 3 9 Q (W) 296.04 333.78 T 2 6 Q (X) 303.29 331.23 T 0 F (3) 307.18 327.93 T 3 9 Q (\316) 287.37 333.78 T 3 18 Q (\345) 284.57 347.96 T 0 12 Q (=) 259.02 351.16 T 181.67 344.71 181.67 358.96 2 L 0.54 H 2 Z N 190.11 333.51 186.51 333.51 186.51 374 3 L N 186.51 374 190.11 374 2 L N 243.03 333.51 246.63 333.51 246.63 374 3 L N 246.63 374 243.03 374 2 L N 390.92 369.21 390.92 383.46 2 L N 317.38 326.17 313.78 326.17 313.78 385.83 3 L N 313.78 385.83 317.38 385.83 2 L N 498.57 326.17 502.16 326.17 502.16 385.83 3 L N 502.16 385.83 498.57 385.83 2 L N 0 0 612 792 C 257.65 291.53 430.79 311.78 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (O) 259.96 301.78 T 3 F (W) 282.71 301.78 T 2 9 Q (X) 292.39 297.58 T 0 6 Q (1) 298.23 295.03 T 3 12 Q (W) 313.81 301.78 T 2 9 Q (X) 323.48 297.58 T 0 6 Q (2) 329.32 295.03 T 3 12 Q (\264) 304.23 301.78 T (W) 350.97 301.78 T 2 9 Q (X) 360.64 297.58 T 0 6 Q (3) 366.48 295.03 T 0 12 Q (+) 338.26 301.78 T 3 F ([) 274.63 301.78 T (]) 372.97 301.78 T (W) 382.61 301.78 T 2 9 Q (X) 392.29 297.58 T 3 12 Q (W) 404.37 301.78 T 2 9 Q (D) 414.04 297.58 T 0 12 Q (\050) 268.63 301.78 T (\051) 423.48 301.78 T 280.97 294.73 280.97 310.78 2 L 0.54 H 2 Z N 333.52 294.73 333.52 310.78 2 L N 349.23 294.73 349.23 310.78 2 L N 370.68 294.73 370.68 310.78 2 L N 380.88 296.53 380.88 310.78 2 L N 398.99 296.53 398.99 310.78 2 L N 402.63 296.53 402.63 310.78 2 L N 421.74 296.53 421.74 310.78 2 L N 0 0 612 792 C 216.3 223.53 393.26 245.13 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (O) 218.44 233.78 T 3 F (W) 234.03 233.78 T 2 9 Q (X) 243.71 229.58 T 0 6 Q (1) 249.55 227.03 T 3 12 Q (W) 265.13 233.78 T 2 9 Q (X) 274.8 229.58 T 0 6 Q (2) 280.64 227.03 T 3 12 Q (\264) 255.55 233.78 T 0 9 Q (2) 287.18 237.38 T 3 12 Q (W) 295.33 233.78 T 2 9 Q (D) 305 229.58 T 0 6 Q (1) 311.84 227.03 T 3 12 Q (W) 333.48 233.78 T 2 9 Q (X) 343.16 229.58 T 0 6 Q (3) 349 227.03 T 0 9 Q (2) 355.54 237.38 T 3 12 Q (W) 363.68 233.78 T 2 9 Q (D) 373.35 229.58 T 0 6 Q (2) 380.19 227.03 T 0 12 Q (+) 320.78 233.78 T (\050) 227.1 233.78 T (\051) 386.13 233.78 T 232.29 226.73 232.29 242.78 2 L 0.54 H 2 Z N 284.84 226.73 284.84 242.78 2 L N 293.59 226.73 293.59 242.78 2 L N 316.04 226.73 316.04 242.78 2 L N 331.74 226.73 331.74 242.78 2 L N 353.2 226.73 353.2 242.78 2 L N 361.94 226.73 361.94 242.78 2 L N 384.39 226.73 384.39 242.78 2 L N 0 0 612 792 C 108 211.33 302.86 229.78 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (D) 111.1 219.78 T 0 9 Q (1) 120.22 215.58 T (1) 129.22 215.58 T 3 F (,) 124.72 215.58 T 2 12 Q (D) 139.72 219.78 T 0 9 Q (2) 148.84 215.58 T (1) 157.84 215.58 T 3 F (,) 153.34 215.58 T 2 12 Q (X) 168.34 219.78 T 0 9 Q (1) 176.12 215.58 T (1) 185.12 215.58 T 3 F (,) 180.62 215.58 T 2 12 Q (X) 195.62 219.78 T 0 9 Q (2) 203.41 215.58 T (1) 212.4 215.58 T 3 F (,) 207.91 215.58 T 2 12 Q (X) 222.9 219.78 T 0 9 Q (3) 230.69 215.58 T (1) 239.69 215.58 T 3 F (,) 235.19 215.58 T 2 12 Q (V) 250.18 219.78 T 0 9 Q (1) 257.97 215.58 T (1) 266.97 215.58 T 3 F (,) 262.47 215.58 T 2 12 Q (V) 277.47 219.78 T 0 9 Q (2) 285.26 215.58 T (1) 294.25 215.58 T 3 F (,) 289.76 215.58 T 3 12 Q (,) 133.72 219.78 T (,) 162.34 219.78 T (,) 189.62 219.78 T (,) 216.9 219.78 T (,) 244.19 219.78 T (,) 271.47 219.78 T 0 0 612 792 C 329.18 211.33 347.64 229.78 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (W) 330.44 219.78 T 0 9 Q (1) 340.89 215.58 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "16" 16 %%Page: "15" 17 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (15) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (The expected value function for stages one through four computed in this way is the total) 108 712 T (expected value for the four) 108 698 T (-stage sequential decision problem.) 237.05 698 T 0 10 Q (\05014\051) 523.34 672 T 0 12 Q (The total amount of computation for an) 108 642.35 T (-stage time-separable sequential decision prob-) 308.31 642.35 T (lem using dynamic programming is) 108 628.35 T (. In general, the state space can be) 357.76 628.35 T (quite lar) 108 614.35 T (ge \050) 147.44 614.35 T 2 F (e.g.,) 165.77 614.35 T 0 F (exponential in the number of state variables\051 and so we look for ways of) 189.1 614.35 T (exploiting structure in the state space as well as time.) 108 600.35 T (Figure) 108 574.35 T (18 shows a two-stage sequential decision problem with three state variables and a) 142.33 574.35 T (state- and time-separable value function represented by two \050sub\051 value functions,) 108 560.35 T (, one for each stage, and four sub\051\051 value functions,) 145.91 546.35 T (, two) 500.67 546.35 T (jointly accounting for) 108 532.35 T ( and) 229.28 532.35 T (, and two accounting for) 266.9 532.35 T (.) 401.17 532.35 T 0 10 Q (\05015\051) 523.34 494.31 T 1 F (Figur) 121.75 457.79 T (e 18. State-separable decision pr) 145.46 457.79 T (oblem) 282.77 457.79 T 0 12 Q -0.18 (Again we can use dynamic programming to compute the expected value working our way) 108 218.29 P (backwards from the last stage. Notice in this case, however) 108 204.29 T (, that the calculations decom-) 391.79 204.29 T (pose into the maximum of the sum of two terms that can be computed independently) 108 190.29 T (.) 513.85 190.29 T 249.62 662.35 381.72 682 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 253.62 672 T 2 F (V) 264.94 672 T 0 9 Q (1) 272.73 667.8 T (4) 290.61 667.8 T 3 F (\256) 279.48 667.8 T 2 12 Q (X) 299.95 672 T 0 9 Q (0) 307.74 667.8 T 0 12 Q (\050) 260.95 672 T (\051) 312.24 672 T (E) 338 672 T 3 F (S) 349.33 672 T 2 F (X) 361.28 672 T 0 9 Q (0) 369.07 667.8 T 0 12 Q (\050) 345.33 672 T (\051) 373.57 672 T (=) 322.23 672 T 297.51 665.55 297.51 679.8 2 L 0.54 H 2 Z N 358.83 669 358.83 679.8 2 L N 0 0 612 792 C 300.31 638.88 308.31 651.56 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 300.81 642.35 T 0 0 612 792 C 281.66 619.9 357.76 639.7 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (O) 283.08 628.35 T (n) 295.74 628.35 T 3 F (W) 305.39 628.35 T 2 9 Q (X) 315.06 624.15 T 0 F (2) 324.1 631.95 T 3 12 Q (W) 332.24 628.35 T 2 9 Q (D) 341.92 624.15 T 0 12 Q (\050) 291.74 628.35 T (\051) 351.35 628.35 T 303.65 623.1 303.65 637.35 2 L 0.54 H 2 Z N 321.76 623.1 321.76 637.35 2 L N 330.5 623.1 330.5 637.35 2 L N 349.61 623.1 349.61 637.35 2 L N 0 0 612 792 C 108 539.63 145.91 555.56 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (W) 108.5 546.35 T 0 9 Q (1) 118.96 542.15 T 2 12 Q (W) 129.45 546.35 T 0 9 Q (2) 139.91 542.15 T 3 12 Q (,) 123.46 546.35 T 0 0 612 792 C 395.51 539.63 500.67 555.56 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (V) 396.03 546.35 T 0 9 Q (1) 403.81 542.15 T (1) 412.81 542.15 T 3 F (,) 408.31 542.15 T 2 12 Q (V) 423.31 546.35 T 0 9 Q (1) 431.1 542.15 T (2) 440.09 542.15 T 3 F (,) 435.6 542.15 T 2 12 Q (V) 450.59 546.35 T 0 9 Q (2) 458.38 542.15 T (1) 467.38 542.15 T 3 F (,) 462.88 542.15 T 2 12 Q (V) 477.87 546.35 T 0 9 Q (2) 485.66 542.15 T (2) 494.66 542.15 T 3 F (,) 490.16 542.15 T 3 12 Q (,) 417.31 546.35 T (,) 444.59 546.35 T (,) 471.88 546.35 T 0 0 612 792 C 214.99 525.63 229.28 541.56 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 215.49 532.35 T 0 9 Q (1) 223.28 528.15 T 0 0 612 792 C 252.61 525.63 266.9 541.56 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 253.11 532.35 T 0 9 Q (2) 260.9 528.15 T 0 0 612 792 C 386.88 525.63 401.17 541.56 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 387.38 532.35 T 0 9 Q (3) 395.17 528.15 T 0 0 612 792 C 187.96 476.46 443.38 516.35 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (S) 280.9 506.35 T 2 F (W) 306.77 506.35 T 0 9 Q (1) 317.22 502.15 T 2 12 Q (W) 334.48 506.35 T 0 9 Q (2) 344.94 502.15 T 0 12 Q (+) 324.72 506.35 T (=) 294 506.35 T 2 F (W) 188.96 484.9 T 2 9 Q (i) 199.41 480.71 T 2 12 Q (V) 220.68 484.9 T 0 9 Q (1) 228.47 480.71 T 2 F (i) 237.46 480.71 T 3 F (,) 232.97 480.71 T 2 12 Q (X) 247.82 484.9 T 0 9 Q (1) 255.61 480.71 T 2 F (i) 264.61 480.71 T 3 F (,) 260.11 480.71 T 2 12 Q (X) 273.11 484.9 T 0 9 Q (2) 280.9 480.71 T 2 F (i) 289.89 480.71 T 3 F (,) 285.4 480.71 T 3 12 Q (,) 267.11 484.9 T (\050) 242.68 484.9 T (\051) 292.95 484.9 T 2 F (V) 311.71 484.9 T 0 9 Q (2) 319.5 480.71 T 2 F (i) 328.5 480.71 T 3 F (,) 324 480.71 T 2 12 Q (X) 334.99 484.9 T 0 9 Q (3) 342.78 480.71 T 2 F (i) 351.78 480.71 T 3 F (,) 347.28 480.71 T 0 12 Q (\050) 331 484.9 T (\051) 354.28 484.9 T (+) 301.95 484.9 T (=) 207.91 484.9 T 2 F (i) 388.26 484.9 T 0 F (1) 415.06 484.9 T (2) 427.06 484.9 T 3 F (,) 421.06 484.9 T ({) 408.15 484.9 T (}) 433.61 484.9 T (\316) 394.6 484.9 T 0 0 612 792 C 108 63 540 720 C 202.33 242.29 445.67 454.46 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 363.64 310.79 361.33 301.98 358.62 310.67 361.13 310.73 4 Y 0 X 0 0 0 1 0 0 0 K V 0.5 H 2 Z 7.02 90 27 71.64 334.33 301.97 A 435.64 310.79 433.33 301.98 430.62 310.67 433.13 310.73 4 Y V 7.02 90 27 71.64 406.33 301.97 A 362.51 386.29 370.33 387.95 364.57 382.39 363.54 384.34 4 Y V 307.33 354.51 363.54 384.34 2 L N 290.51 386.29 298.33 387.95 292.57 382.39 291.54 384.34 4 Y V 235.33 354.51 291.54 384.34 2 L N 262.33 421.38 280.33 440.48 R 3 X V 0 Z 0 X N 334.33 421.38 352.33 440.48 R 7 X V 0 X N 343.33 387.95 334.33 373.62 343.33 359.29 352.33 373.62 4 Y 3 X V 0 X N 415.33 387.95 406.33 373.62 415.33 359.29 424.33 373.62 4 Y 7 X V 0 X N 415.33 349.74 406.33 335.41 415.33 321.08 424.33 335.41 4 Y 7 X V 0 X N 334.74 390.25 338.83 383.17 331.61 387.01 333.18 388.63 4 Y V 2 Z 34.86 90 31.5 9.55 307.33 383.17 A 331.61 360.23 338.83 364.07 334.74 356.99 333.18 358.61 4 Y V 270 325.14 31.5 9.55 307.33 364.07 A 331.61 322.02 338.83 325.86 334.74 318.78 333.18 320.4 4 Y V 270 325.14 31.5 9.55 307.33 325.86 A 361.33 301.97 352.33 287.65 361.33 273.32 370.33 287.65 4 Y 3 X V 0 Z 0 X N 433.33 301.97 424.33 287.65 433.33 273.32 442.33 287.65 4 Y 7 X V 0 X N 397.33 273.32 388.33 258.99 397.33 244.66 406.33 258.99 4 Y 7 X V 0 X N 380.73 261.83 388.33 258.99 380.38 257.37 380.55 259.6 4 Y V 2 Z 180 253.25 27 14.33 388.33 273.32 A 414.29 257.37 406.33 258.99 413.94 261.83 414.11 259.6 4 Y V 286.75 360 27 14.33 406.33 273.32 A 403.61 322.02 410.83 325.86 406.74 318.78 405.18 320.4 4 Y V 270 325.14 31.5 9.55 379.33 325.86 A 406.74 390.25 410.83 383.17 403.61 387.01 405.18 388.63 4 Y V 34.86 90 31.5 9.55 379.33 383.17 A 403.61 360.23 410.83 364.07 406.74 356.99 405.18 358.61 4 Y V 270 325.14 31.5 9.55 379.33 364.07 A 3 12 Q (S) 393.7 256.25 T 270 360 27 28.66 307.33 421.38 A 331.65 412.86 334.33 421.38 336.57 412.73 334.11 412.8 4 Y V 270 352.63 27 66.87 307.33 421.38 A 270 360 27 105.08 307.33 421.38 A 270 360 27 28.66 235.33 421.38 A 259.65 412.86 262.33 421.38 264.57 412.73 262.11 412.8 4 Y V 270 352.63 27 66.87 235.33 421.38 A 270 360 27 105.08 235.33 421.38 A 290.64 394.93 298.33 392.72 290.64 390.52 290.64 392.72 4 Y V 235.33 392.72 290.64 392.72 2 L N 290.64 356.72 298.33 354.51 290.64 352.31 290.64 354.51 4 Y V 235.33 354.51 290.64 354.51 2 L N 290.64 318.51 298.33 316.3 290.64 314.1 290.64 316.3 4 Y V 235.33 316.3 290.64 316.3 2 L N 362.64 356.72 370.33 354.51 362.64 352.31 362.64 354.51 4 Y V 307.33 354.51 362.64 354.51 2 L N 362.64 394.93 370.33 392.72 362.64 390.52 362.64 392.72 4 Y V 307.33 392.72 362.64 392.72 2 L N 362.64 318.51 370.33 316.3 362.64 314.1 362.64 316.3 4 Y V 307.33 316.3 362.64 316.3 2 L N 7 X 90 450 9 9.55 235.33 392.72 G 0 Z 0 X 90 450 9 9.55 235.33 392.72 A 3 X 90 450 9 9.55 307.33 392.72 G 0 X 90 450 9 9.55 307.33 392.72 A 7 X 90 450 9 9.55 379.33 392.72 G 0 X 90 450 9 9.55 379.33 392.72 A 7 X 90 450 9 9.55 235.33 354.51 G 0 X 90 450 9 9.55 235.33 354.51 A 3 X 90 450 9 9.55 307.33 354.51 G 0 X 90 450 9 9.55 307.33 354.51 A 7 X 90 450 9 9.55 379.33 354.51 G 0 X 90 450 9 9.55 379.33 354.51 A 7 X 90 450 9 9.55 235.33 316.3 G 0 X 90 450 9 9.55 235.33 316.3 A 3 X 90 450 9 9.55 307.33 316.3 G 0 X 90 450 9 9.55 307.33 316.3 A 7 X 90 450 9 9.55 379.33 316.3 G 0 X 90 450 9 9.55 379.33 316.3 A 348.92 294.92 352.33 287.65 345.59 292.01 347.26 293.47 4 Y V 2 Z 180 235.69 9 33.43 352.33 321.08 A 420.92 294.92 424.33 287.65 417.59 292.01 419.26 293.47 4 Y V 180 235.69 9 33.43 424.33 321.08 A 294.06 328.13 298.33 321.08 290.97 324.79 292.52 326.46 4 Y V 180 251.15 18 100.3 298.33 421.38 A 292.95 365.57 298.33 359.29 290.44 361.76 291.69 363.67 4 Y V 180 248.36 18 62.09 298.33 421.38 A 364.95 365.57 370.33 359.29 362.44 361.76 363.69 363.67 4 Y V 180 248.36 18 62.09 370.33 421.38 A 366.06 328.13 370.33 321.08 362.97 324.79 364.52 326.46 4 Y V 180 251.15 18 100.3 370.33 421.38 A 343.33 349.03 334.33 334.7 343.33 320.37 352.33 334.7 4 Y 3 X V 0 Z 0 X N 2 9 Q (t) 225.17 444.2 T 0 F (0) 241.73 444.2 T (=) 232.16 444.2 T 2 F (t) 368.84 444.2 T 0 F (2) 385.4 444.2 T (=) 375.83 444.2 T 2 F (t) 297 444.2 T 0 F (1) 313.57 444.2 T (=) 304 444.2 T 2 12 Q (D) 207.4 413.39 T (X) 205.58 393.8 T 0 9 Q (1) 213.37 389.6 T 2 12 Q (V) 205.58 372.33 T 0 9 Q (1) 213.37 368.14 T 2 12 Q (X) 205.58 350.87 T 0 9 Q (2) 213.37 346.67 T 2 12 Q (X) 205.58 307.94 T 0 9 Q (3) 213.37 303.74 T 2 12 Q (V) 205.58 329.4 T 0 9 Q (2) 213.37 325.2 T 2 12 Q (W) 206.73 284.6 T 108 63 540 720 C 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "15" 17 %%Page: "14" 18 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (14) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (is unlimited. W) 108 712 T (e return to discuss in\336nite-horizon problems subsequently) 181.38 712 T (. A solution to a) 458.58 712 T (control problem is said to be) 108 698 T 2 F (closed loop) 247.99 698 T 0 F ( if the control system bases its decisions on feed-) 302.99 698 T (back from the environment; the policy summarizes the control system\325) 108 684 T (s response to feed-) 446.96 684 T (back. An) 108 670 T 2 F (open-loop) 154.32 670 T 0 F ( solution does not employ feedback and is represented as a simple) 202.98 670 T (sequence of actions since the control system\325) 108 656 T (s response is not conditioned on the current) 323.96 656 T (state or observations of the current state.) 108 642 T 1 10 Q (Figur) 121.75 617.33 T (e 17. T) 145.46 617.33 T (ime-separable decision pr) 173.88 617.33 T (oblem) 283.14 617.33 T 0 12 Q (By restricting our attention to time-separable problems, we can reduce an) 108 443.88 T (-dimensional) 471.26 443.88 T (problem to) 108 429.88 T ( 1-dimensional problems using dynamic programming. T) 171.33 429.88 T (o illustrate, we) 444.81 429.88 T -0.41 (show how to calculate the expected value for the optimal policy in the decision problem of) 108 415.88 P -0.04 (Figure) 108 401.88 P -0.04 (17. W) 142.33 401.88 P -0.04 (e work our way backward starting at the \336nal stage, calculating the expected) 170.66 401.88 P (value accounting just for stage four) 108 387.88 T (. The calculations involve maximization and taking) 276.97 387.88 T (expectations using the conditional probability distribution of the state given the previous) 108 373.88 T (state and action.) 108 359.88 T 0 10 Q (\0501) 523.71 330.15 T (1\051) 531.67 330.15 T 0 12 Q -0.41 (The notation) 108 290.04 P -0.41 ( represents the expected value of being in state) 267.38 290.04 P -0.41 ( at stage) 498.92 290.04 P ( accounting for the value at stages) 132.33 276.04 T ( through) 304.96 276.04 T (. Given) 354.63 276.04 T (, we can calcu-) 462.95 276.04 T (late the expected value accounting for stages three and four) 108 262.04 T (.) 392.26 262.04 T 0 10 Q (\05012\051) 523.34 232.31 T 0 12 Q (And so on until we have accounted for all four stages.) 108 188.66 T 0 10 Q (\05013\051) 523.34 137.39 T 108 63 540 720 C 219.54 467.88 428.46 614 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 9 Q 0 X 0 0 0 1 0 0 0 K (t) 242.47 603.09 T 0 F (0) 259.03 603.09 T (=) 249.46 603.09 T 2 F (t) 315.87 603.09 T 0 F (2) 332.43 603.09 T (=) 322.86 603.09 T 2 F (t) 279.17 603.09 T 0 F (1) 295.73 603.09 T (=) 286.16 603.09 T 2 F (t) 352.57 603.09 T 0 F (3) 369.14 603.09 T (=) 359.57 603.09 T 273.42 554.49 281.11 552.29 273.42 550.08 273.42 552.29 4 Y V 254.11 552.29 273.42 552.29 2 L 0.5 H 2 Z N 309.42 554.49 317.11 552.29 309.42 550.08 309.42 552.29 4 Y V 290.11 552.29 309.42 552.29 2 L N 345.42 554.49 353.11 552.29 345.42 550.08 345.42 552.29 4 Y V 326.11 552.29 345.42 552.29 2 L N 381.42 554.49 389.11 552.29 381.42 550.08 381.42 552.29 4 Y V 362.11 552.29 381.42 552.29 2 L N 263.11 579.29 281.11 597.29 R 3 X V 0 Z 0 X N 7 X 90 450 9 9 254.11 552.29 G 0 X 90 450 9 9 254.11 552.29 A 299.11 579.29 317.11 597.29 R 7 X V 0 X N 3 X 90 450 9 9 290.11 552.29 G 0 X 90 450 9 9 290.11 552.29 A 335.11 579.29 353.11 597.29 R 7 X V 0 X N 7 X 90 450 9 9 326.11 552.29 G 0 X 90 450 9 9 326.11 552.29 A 371.11 579.29 389.11 597.29 R 7 X V 0 X N 7 X 90 450 9 9 362.11 552.29 G 0 X 90 450 9 9 362.11 552.29 A 7 X 90 450 9 9 398.11 552.29 G 0 X 90 450 9 9 398.11 552.29 A 416.11 538.79 407.11 525.29 416.11 511.79 425.11 525.29 4 Y 7 X V 0 X N 255.82 580.19 263.11 583.79 258.83 576.87 257.33 578.53 4 Y V 2 Z 129.97 180 9 22.5 263.11 561.29 A 291.82 580.19 299.11 583.79 294.83 576.87 293.32 578.53 4 Y V 129.97 180 9 22.5 299.11 561.29 A 327.82 580.19 335.1 583.79 330.83 576.87 329.32 578.53 4 Y V 129.97 180 9 22.5 335.11 561.29 A 363.81 580.19 371.1 583.79 366.83 576.87 365.32 578.53 4 Y V 129.97 180 9 22.5 371.11 561.29 A 276.83 563.7 281.11 556.79 273.82 560.39 275.32 562.04 4 Y V 180 230.03 9 22.5 281.11 579.29 A 312.83 563.7 317.1 556.79 309.82 560.39 311.32 562.04 4 Y V 180 230.03 9 22.5 317.11 579.29 A 348.83 563.7 353.1 556.79 345.82 560.39 347.32 562.04 4 Y V 180 230.03 9 22.5 353.11 579.29 A 384.83 563.7 389.1 556.79 381.81 560.39 383.32 562.04 4 Y V 180 230.03 9 22.5 389.11 579.29 A 380.11 538.79 371.11 525.29 380.11 511.79 389.11 525.29 4 Y 7 X V 0 Z 0 X N 344.11 538.79 335.11 525.29 344.11 511.79 353.11 525.29 4 Y 7 X V 0 X N 308.11 538.79 299.11 525.29 308.11 511.79 317.11 525.29 4 Y 3 X V 0 X N 294.38 531.79 299.11 525.29 291.65 528.3 293.01 530.05 4 Y V 2 Z 180 227.37 9 18 299.11 543.29 A 330.38 531.79 335.11 525.29 327.65 528.3 329.01 530.05 4 Y V 180 227.37 9 18 335.11 543.29 A 366.38 531.79 371.11 525.29 363.65 528.3 365.01 530.05 4 Y V 180 227.37 9 18 371.11 543.29 A 402.37 531.79 407.11 525.29 399.65 528.3 401.01 530.05 4 Y V 180 227.37 9 18 407.11 543.29 A 362.11 498.29 353.11 484.79 362.11 471.29 371.11 484.79 4 Y 7 X V 0 Z 0 X N 344.95 487.63 353.11 484.79 344.69 482.87 344.82 485.25 4 Y V 2 Z 180 259.39 45 27 353.11 511.79 A 348.83 496.2 353.1 489.29 345.82 492.89 347.32 494.55 4 Y V 180 230.03 9 22.5 353.11 511.79 A 378.39 492.89 371.1 489.29 375.38 496.2 376.88 494.55 4 Y V 309.97 360 9 22.5 371.11 511.79 A 379.52 482.87 371.11 484.79 379.26 487.63 379.39 485.25 4 Y V 280.61 360 45 27 371.11 511.79 A 3 12 Q (S) 357.49 481.76 T 2 F (D) 226.21 583.29 T (X) 227.21 551.42 T (V) 227.54 519.55 T 2 9 Q (t) 389.27 603.42 T 0 F (4) 405.84 603.42 T (=) 396.27 603.42 T 108 63 540 720 C 0 0 612 792 C 463.26 440.41 471.26 453.09 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 463.76 443.88 T 0 0 612 792 C 163.33 426.41 171.33 439.09 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 163.83 429.88 T 0 0 612 792 C 134.63 310.04 497.08 343.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 138.63 330.15 T 2 F (V) 149.96 330.15 T 0 9 Q (4) 157.75 325.95 T (4) 175.62 325.95 T 3 F (\256) 164.49 325.95 T 2 12 Q (X) 184.97 330.15 T 0 9 Q (3) 192.76 325.95 T 2 12 Q (x) 212.01 330.15 T 0 F (=) 201.25 330.15 T (\050) 145.96 330.15 T (\051) 217.34 330.15 T (m) 240.1 330.15 T (a) 249.43 330.15 T (x) 254.76 330.15 T 2 9 Q (d) 261.22 325.95 T 0 12 Q (P) 295.07 330.15 T (r) 301.74 330.15 T 2 F (X) 309.74 330.15 T 0 9 Q (4) 317.53 325.95 T 2 12 Q (x) 336.78 330.15 T 0 F (') 342.11 330.15 T 2 F (X) 349.12 330.15 T 0 9 Q (3) 356.9 325.95 T 2 12 Q (x) 376.16 330.15 T 0 F (=) 365.4 330.15 T 2 F (D) 387.49 330.15 T 0 9 Q (4) 396.61 325.95 T 2 12 Q (d) 415.86 330.15 T 0 F (=) 405.1 330.15 T 3 F (,) 381.49 330.15 T 0 F (=) 326.02 330.15 T (\050) 305.74 330.15 T (\051) 421.86 330.15 T 2 F (V) 433.72 330.15 T 0 9 Q (4) 441.5 325.95 T 2 12 Q (X) 450 330.15 T 0 9 Q (4) 457.79 325.95 T 2 12 Q (x) 477.04 330.15 T 0 F (') 482.37 330.15 T (=) 466.28 330.15 T (\050) 446 330.15 T (\051) 484.53 330.15 T 3 F ([) 428.57 330.15 T (]) 489.08 330.15 T 2 9 Q (x) 266.43 315.09 T 0 F (') 270.42 315.09 T 3 F (W) 282.95 315.09 T 2 6 Q (X) 290.21 312.54 T 3 9 Q (\316) 274.29 315.09 T 3 18 Q (\345) 273.73 326.95 T 0 12 Q (=) 227.34 330.15 T 182.52 323.7 182.52 337.95 2 L 0.54 H 2 Z N 346.67 327.15 346.67 337.95 2 L N 0 0 612 792 C 171.17 280.39 267.38 300.04 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 175.17 290.04 T 2 F (V) 186.5 290.04 T 2 9 Q (i) 194.28 285.84 T (j) 210.16 285.84 T 3 F (\256) 199.03 285.84 T 2 12 Q (X) 217.51 290.04 T 2 9 Q (i) 225.3 285.84 T 0 F (1) 236.79 285.84 T (\320) 230.05 285.84 T 2 12 Q (x) 256.05 290.04 T 0 F (=) 245.29 290.04 T (\050) 182.5 290.04 T (\051) 261.38 290.04 T 215.06 283.59 215.06 297.84 2 L 0.54 H 2 Z N 0 0 612 792 C 490.59 285.04 498.92 300.04 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (x) 491.59 290.04 T 0 0 612 792 C 108 271.04 132.33 286.04 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (i) 109 276.04 T 0 F (1) 124.33 276.04 T (\320) 115.33 276.04 T 0 0 612 792 C 298.63 271.04 304.96 286.04 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (i) 299.63 276.04 T 0 0 612 792 C 348.29 271.04 354.63 286.04 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (j) 349.29 276.04 T 0 0 612 792 C 392.95 266.39 462.95 286.04 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 397.64 276.04 T 2 F (V) 408.97 276.04 T 0 9 Q (4) 416.76 271.84 T (4) 434.63 271.84 T 3 F (\256) 423.5 271.84 T 2 12 Q (X) 443.98 276.04 T 0 9 Q (3) 451.77 271.84 T 0 12 Q (\050) 404.97 276.04 T (\051) 456.27 276.04 T 441.53 269.59 441.53 283.84 2 L 0.54 H 2 Z N 0 0 612 792 C 108 208.66 571.08 246.04 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 112 232.31 T 2 F (V) 123.33 232.31 T 0 9 Q (3) 131.12 228.11 T (4) 148.99 228.11 T 3 F (\256) 137.86 228.11 T 2 12 Q (X) 158.34 232.31 T 0 9 Q (2) 166.13 228.11 T 2 12 Q (x) 185.38 232.31 T 0 F (=) 174.62 232.31 T (\050) 119.33 232.31 T (\051) 190.71 232.31 T (m) 213.47 232.31 T (a) 222.8 232.31 T (x) 228.13 232.31 T 2 9 Q (d) 234.59 228.11 T 0 12 Q (P) 268.44 232.31 T (r) 275.11 232.31 T 2 F (X) 283.11 232.31 T 0 9 Q (3) 290.89 228.11 T 2 12 Q (x) 310.15 232.31 T 0 F (') 315.48 232.31 T 2 F (X) 322.48 232.31 T 0 9 Q (2) 330.27 228.11 T 2 12 Q (x) 349.53 232.31 T 0 F (=) 338.77 232.31 T 2 F (D) 360.86 232.31 T 0 9 Q (3) 369.98 228.11 T 2 12 Q (d) 389.23 232.31 T 0 F (=) 378.47 232.31 T 3 F (,) 354.86 232.31 T 0 F (=) 299.39 232.31 T (\050) 279.11 232.31 T (\051) 395.23 232.31 T 2 F (V) 322.08 217.31 T 0 9 Q (3) 329.87 213.11 T 2 12 Q (X) 338.37 217.31 T 0 9 Q (3) 346.16 213.11 T 2 12 Q (x) 365.41 217.31 T 0 F (') 370.74 217.31 T (=) 354.65 217.31 T (\050) 334.37 217.31 T (\051) 372.9 217.31 T (E) 392.66 217.31 T 2 F (V) 403.99 217.31 T 0 9 Q (4) 411.78 213.11 T (4) 429.65 213.11 T 3 F (\256) 418.52 213.11 T 2 12 Q (X) 439 217.31 T 0 9 Q (3) 446.79 213.11 T 2 12 Q (x) 466.04 217.31 T 0 F (') 471.37 217.31 T (=) 455.28 217.31 T (\050) 399.99 217.31 T (\051) 473.53 217.31 T (+) 379.9 217.31 T 3 F ([) 316.94 217.31 T (]) 478.08 217.31 T 2 9 Q (x) 239.8 216.28 T 0 F (') 243.79 216.28 T 3 F (W) 256.32 216.28 T 2 6 Q (X) 263.58 213.73 T 3 9 Q (\316) 247.66 216.28 T 3 18 Q (\345) 247.1 229.1 T 0 12 Q (=) 200.7 232.31 T 155.89 225.86 155.89 240.11 2 L 0.54 H 2 Z N 320.04 229.31 320.04 240.11 2 L N 436.55 210.86 436.55 225.11 2 L N 0 0 612 792 C 108 100.6 571.08 172.66 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (E) 112 158.93 T 2 F (V) 123.33 158.93 T 0 9 Q (2) 131.12 154.73 T (4) 148.99 154.73 T 3 F (\256) 137.86 154.73 T 2 12 Q (X) 158.34 158.93 T 0 9 Q (1) 166.13 154.73 T 2 12 Q (x) 185.38 158.93 T 0 F (=) 174.62 158.93 T (\050) 119.33 158.93 T (\051) 190.71 158.93 T (m) 213.47 158.93 T (a) 222.8 158.93 T (x) 228.13 158.93 T 2 9 Q (d) 234.59 154.73 T 0 12 Q (P) 268.44 158.93 T (r) 275.11 158.93 T 2 F (X) 283.11 158.93 T 0 9 Q (2) 290.89 154.73 T 2 12 Q (x) 310.15 158.93 T 0 F (') 315.48 158.93 T 2 F (X) 322.48 158.93 T 0 9 Q (1) 330.27 154.73 T 2 12 Q (x) 349.53 158.93 T 0 F (=) 338.77 158.93 T 2 F (D) 360.86 158.93 T 0 9 Q (2) 369.98 154.73 T 2 12 Q (d) 389.23 158.93 T 0 F (=) 378.47 158.93 T 3 F (,) 354.86 158.93 T 0 F (=) 299.39 158.93 T (\050) 279.11 158.93 T (\051) 395.23 158.93 T 2 F (V) 322.08 143.93 T 0 9 Q (2) 329.87 139.73 T 2 12 Q (X) 338.37 143.93 T 0 9 Q (2) 346.16 139.73 T 2 12 Q (x) 365.41 143.93 T 0 F (') 370.74 143.93 T (=) 354.65 143.93 T (\050) 334.37 143.93 T (\051) 372.9 143.93 T (E) 392.66 143.93 T 2 F (V) 403.99 143.93 T 0 9 Q (3) 411.78 139.73 T (4) 429.65 139.73 T 3 F (\256) 418.52 139.73 T 2 12 Q (X) 439 143.93 T 0 9 Q (2) 446.79 139.73 T 2 12 Q (x) 466.04 143.93 T 0 F (') 471.37 143.93 T (=) 455.28 143.93 T (\050) 399.99 143.93 T (\051) 473.53 143.93 T (+) 379.89 143.93 T 3 F ([) 316.94 143.93 T (]) 478.08 143.93 T 2 9 Q (x) 239.8 142.4 T 0 F (') 243.79 142.4 T 3 F (W) 256.32 142.4 T 2 6 Q (X) 263.58 139.85 T 3 9 Q (\316) 247.66 142.4 T 3 18 Q (\345) 247.1 155.72 T 0 12 Q (=) 200.7 158.93 T (E) 112 124.25 T 2 F (V) 123.33 124.25 T 0 9 Q (1) 131.12 120.05 T (4) 148.99 120.05 T 3 F (\256) 137.86 120.05 T 2 12 Q (X) 158.34 124.25 T 0 9 Q (0) 166.13 120.05 T 2 12 Q (x) 185.38 124.25 T 0 F (=) 174.62 124.25 T (\050) 119.33 124.25 T (\051) 190.71 124.25 T (m) 213.47 124.25 T (a) 222.8 124.25 T (x) 228.13 124.25 T 2 9 Q (d) 234.59 120.05 T 0 12 Q (P) 268.44 124.25 T (r) 275.11 124.25 T 2 F (X) 283.11 124.25 T 0 9 Q (1) 290.89 120.05 T 2 12 Q (x) 310.15 124.25 T 0 F (') 315.48 124.25 T 2 F (X) 322.48 124.25 T 0 9 Q (0) 330.27 120.05 T 2 12 Q (x) 349.53 124.25 T 0 F (=) 338.77 124.25 T 2 F (D) 360.86 124.25 T 0 9 Q (1) 369.98 120.05 T 2 12 Q (d) 389.23 124.25 T 0 F (=) 378.47 124.25 T 3 F (,) 354.86 124.25 T 0 F (=) 299.39 124.25 T (\050) 279.11 124.25 T (\051) 395.23 124.25 T 2 F (V) 322.08 109.25 T 0 9 Q (1) 329.87 105.05 T 2 12 Q (X) 338.37 109.25 T 0 9 Q (1) 346.16 105.05 T 2 12 Q (x) 365.41 109.25 T 0 F (') 370.74 109.25 T (=) 354.65 109.25 T (\050) 334.37 109.25 T (\051) 372.9 109.25 T (E) 392.66 109.25 T 2 F (V) 403.99 109.25 T 0 9 Q (2) 411.78 105.05 T (4) 429.65 105.05 T 3 F (\256) 418.52 105.05 T 2 12 Q (X) 439 109.25 T 0 9 Q (1) 446.79 105.05 T 2 12 Q (x) 466.04 109.25 T 0 F (') 471.37 109.25 T (=) 455.28 109.25 T (\050) 399.99 109.25 T (\051) 473.53 109.25 T (+) 379.89 109.25 T 3 F ([) 316.94 109.25 T (]) 478.08 109.25 T 2 9 Q (x) 239.8 109.16 T 0 F (') 243.79 109.16 T 3 F (W) 256.32 109.16 T 2 6 Q (X) 263.58 106.61 T 3 9 Q (\316) 247.66 109.16 T 3 18 Q (\345) 247.1 121.05 T 0 12 Q (=) 200.7 124.25 T 155.89 152.48 155.89 166.73 2 L 0.54 H 2 Z N 320.04 155.93 320.04 166.73 2 L N 436.55 137.48 436.55 151.73 2 L N 155.89 117.8 155.89 132.05 2 L N 320.04 121.25 320.04 132.05 2 L N 436.55 102.8 436.55 117.05 2 L N 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "14" 18 %%Page: "13" 19 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (13) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 1 16 Q (6 Exploiting Structur) 108 709.33 T (e in Decision Space) 257.5 709.33 T 0 12 Q -0.1 (In this section, we consider a class of planning problems often referred to as) 108 682 P 2 F -0.1 (discr) 474.89 682 P -0.1 (ete-time) 498.45 682 P -0.47 (sequential decision pr) 108 668 P -0.47 (oblems.) 212.62 668 P 0 F -0.47 (Such problems are said to have) 252.15 668 P 2 F -0.47 (stages) 402.01 668 P 0 F -0.47 ( that correspond to the) 432.01 668 P (points in time at which the control system is allowed to take action. W) 108 654 T (e augment our) 445.02 654 T -0.22 (graphical notation to include boxes that represent) 108 640 P 2 F -0.22 (decisions) 346.06 640 P 0 F -0.22 ( whose possible outcomes) 390.72 640 P (correspond to actions, and diamonds that represent) 108 626 T 2 F (value functions) 354.62 626 T 0 F ( that account, directly) 427.62 626 T (or otherwise, for the costs and bene\336ts of acting. The objective in sequential decision) 108 612 T (problems is to determine a) 108 598 T 2 F (policy) 238.32 598 T 0 F (or conditional plan that maps states to actions so as to) 270.65 598 T -0.11 (maximize the expected value. Figure) 108 584 P -0.11 (16 shows a simple sequential decision problem with) 287.54 584 P -0.42 (four stages and a single value function. The graphical representation shown in Figure) 108 570 P -0.42 (16 is) 514.84 570 P (called an) 108 556 T 2 F (in\337uence diagram) 153.98 556 T 0 F ( [Howard and Matheson, 1984].) 240.97 556 T 1 10 Q (Figur) 121.75 531.33 T (e 16. Sequential decision pr) 145.46 531.33 T (oblem) 261.67 531.33 T 0 12 Q (In Figure) 108 398.19 T (16, the states correspond to the random variables) 155.33 398.19 T (, the deci-) 485.4 398.19 T (sions to) 108 384.19 T ( \050) 224.77 384.19 T ( and) 248.71 384.19 T ( are shaded\051, and there is a single value function of) 288.13 384.19 T -0.11 (the last four state variables, the only ones that can be in\337uenced by the control system.) 108 370.19 P (for) 108 356.19 T ( is dependent on) 168.49 356.19 T (and the arc leading from) 292.67 356.19 T ( to) 439.2 356.19 T ( is meant to) 468.17 356.19 T (indicate that the control system knows the state immediately prior to taking action. The) 108 342.19 T (decision problem shown in Figure) 108 328.19 T (16 is said to be four dimensional or) 274.99 328.19 T (, generally) 444.17 328.19 T (,) 494.04 328.19 T (-) 508.04 328.19 T (dimensional where) 108 314.19 T ( is the number of stages accounted for in determining the total value) 209.98 314.19 T -0.27 (for a particular sequence of actions. This means that the value is an arbitrary function of) 108 300.19 P (ar) 108 286.19 T (guments each of which ranges over) 117.11 286.19 T (.) 308.5 286.19 T (T) 108 260.19 T (ypically) 114.49 260.19 T (, we restrict the value function to be a sum of functions each of which is depen-) 152.38 260.19 T (dent upon a single stage. This restriction is entirely in service to practical and computa-) 108 246.19 T (tional expediency and such problems are said to be) 108 232.19 T 2 F (time separable) 355.62 232.19 T 0 F (. Figure) 426.61 232.19 T (17 shows a) 466.94 232.19 T -0.28 (time-separable decision problem with four \050sub\051 value functions) 108 218.19 P -0.28 (, each one) 489.52 218.19 P (accounting for the value of the state at a speci\336c time \050) 108 204.19 T (,) 387.89 204.19 T (, and) 409.98 204.19 T ( are shaded\051.) 452.17 204.19 T 0 10 Q (\05010\051) 523.34 178.19 T 1 14 Q (6.1 Finite-Horizon Markov Decision Pr) 108 142.41 T (ocesses) 344.95 142.41 T 0 12 Q (A Markov chain together with a set of actions available to the control system and a time-) 108 115.74 T (separable value function is called a) 108 101.74 T 2 F (Markov decision pr) 279.29 101.74 T (ocess) 372.83 101.74 T 0 F (. The sequential decision) 398.82 101.74 T (problems depicted in Figure) 108 87.74 T (16 and Figure) 245.99 87.74 T (17 are) 315.65 87.74 T (-stage or) 357.3 87.74 T 2 F (\336nite-horizon) 401.95 87.74 T 0 F (problems as) 469.62 87.74 T -0.02 (opposed to) 108 73.74 P 2 F -0.02 (inde\336nite duration) 163.28 73.74 P 0 F -0.02 ( or) 252.26 73.74 P 2 F -0.02 (in\336nite-horizon) 268.21 73.74 P 0 F -0.02 ( problems in which the number of stages) 342.22 73.74 P 517.81 631.55 537.22 650 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 518.93 640 T 2 9 Q (D) 528.61 635.8 T 0 0 612 792 C 108 63 540 720 C 211.56 422.19 436.44 528 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 261.2 469.56 268.89 467.35 261.2 465.15 261.2 467.35 4 Y 0 X 0 0 0 1 0 0 0 K V 241.89 467.35 261.2 467.35 2 L 0.5 H 2 Z N 297.2 469.56 304.89 467.35 297.2 465.15 297.2 467.35 4 Y V 277.89 467.35 297.2 467.35 2 L N 333.2 469.56 340.89 467.35 333.2 465.15 333.2 467.35 4 Y V 313.89 467.35 333.2 467.35 2 L N 369.2 469.56 376.89 467.35 369.2 465.15 369.2 467.35 4 Y V 349.89 467.35 369.2 467.35 2 L N 250.89 494.35 268.89 512.35 R 3 X V 0 Z 0 X N 7 X 90 450 9 9 241.89 467.35 G 0 X 90 450 9 9 241.89 467.35 A 286.89 494.35 304.89 512.35 R 7 X V 0 X N 3 X 90 450 9 9 277.89 467.35 G 0 X 90 450 9 9 277.89 467.35 A 322.89 494.35 340.89 512.35 R 7 X V 0 X N 7 X 90 450 9 9 313.89 467.35 G 0 X 90 450 9 9 313.89 467.35 A 358.89 494.35 376.89 512.35 R 7 X V 0 X N 7 X 90 450 9 9 349.89 467.35 G 0 X 90 450 9 9 349.89 467.35 A 7 X 90 450 9 9 385.89 467.35 G 0 X 90 450 9 9 385.89 467.35 A 421.89 453.85 412.89 440.35 421.89 426.85 430.89 440.35 4 Y 7 X V 0 X N 405.06 443.49 412.89 440.36 404.59 438.86 404.83 441.18 4 Y V 2 Z 180 252.62 27 18 412.89 458.35 A 403.59 443.24 412.89 440.36 403.48 437.87 403.53 440.55 4 Y V 180 261.46 63 18 412.89 458.35 A 402.72 443.38 412.89 440.36 402.66 437.52 402.69 440.45 4 Y V 180 264.09 99 18 412.89 458.35 A 402.26 443.46 412.89 440.36 402.23 437.36 402.25 440.41 4 Y V 180 265.48 135 18 412.89 458.35 A 243.6 495.26 250.89 498.86 246.62 491.94 245.11 493.6 4 Y V 129.97 180 9 22.5 250.89 476.35 A 279.6 495.26 286.89 498.86 282.62 491.94 281.11 493.6 4 Y V 129.97 180 9 22.5 286.89 476.35 A 315.6 495.26 322.89 498.86 318.61 491.94 317.11 493.6 4 Y V 129.97 180 9 22.5 322.89 476.35 A 351.6 495.26 358.89 498.86 354.61 491.94 353.11 493.6 4 Y V 129.97 180 9 22.5 358.89 476.35 A 264.62 478.77 268.89 471.86 261.6 475.45 263.11 477.11 4 Y V 180 230.03 9 22.5 268.89 494.35 A 300.62 478.77 304.89 471.86 297.6 475.45 299.11 477.11 4 Y V 180 230.03 9 22.5 304.89 494.35 A 336.61 478.77 340.89 471.86 333.6 475.45 335.11 477.11 4 Y V 180 230.03 9 22.5 340.89 494.35 A 372.61 478.77 376.89 471.86 369.6 475.45 371.11 477.11 4 Y V 180 230.03 9 22.5 376.89 494.35 A 2 12 Q (V) 214.28 434.63 T (X) 214.62 466.01 T (D) 214.28 497.38 T 2 9 Q (t) 232.68 517.15 T 0 F (0) 249.25 517.15 T (=) 239.68 517.15 T 2 F (t) 304.21 517.77 T 0 F (2) 320.78 517.77 T (=) 311.21 517.77 T 2 F (t) 268.45 517.15 T 0 F (1) 285.01 517.15 T (=) 275.44 517.15 T 2 F (t) 339.98 517.15 T 0 F (3) 356.54 517.15 T (=) 346.97 517.15 T 2 F (t) 375.74 517.65 T 0 F (4) 392.31 517.65 T (=) 382.74 517.65 T 108 63 540 720 C 0 0 612 792 C 392.95 389.74 485.4 408.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 395.96 398.19 T 0 9 Q (0) 403.75 393.99 T 2 12 Q (X) 414.25 398.19 T 0 9 Q (1) 422.04 393.99 T 2 12 Q (X) 432.53 398.19 T 0 9 Q (2) 440.32 393.99 T 2 12 Q (X) 450.82 398.19 T 0 9 Q (3) 458.61 393.99 T 2 12 Q (X) 469.11 398.19 T 0 9 Q (4) 476.9 393.99 T 3 12 Q (,) 408.25 398.19 T (,) 426.54 398.19 T (,) 444.82 398.19 T (,) 463.11 398.19 T 0 0 612 792 C 148.01 375.74 224.77 394.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (D) 149.65 384.19 T 0 9 Q (1) 158.77 379.99 T 2 12 Q (D) 169.27 384.19 T 0 9 Q (2) 178.39 379.99 T 2 12 Q (D) 188.89 384.19 T 0 9 Q (3) 198.01 379.99 T 2 12 Q (D) 208.51 384.19 T 0 9 Q (4) 217.63 379.99 T 3 12 Q (,) 163.27 384.19 T (,) 182.89 384.19 T (,) 202.51 384.19 T 0 0 612 792 C 231.77 375.74 248.71 394.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (D) 232.93 384.19 T 0 9 Q (1) 242.05 379.99 T 0 0 612 792 C 272.04 375.74 288.13 394.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 273.44 384.19 T 0 9 Q (1) 281.23 379.99 T 0 0 612 792 C 524.82 363.55 537.11 379.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 525.32 370.19 T 2 9 Q (i) 533.11 365.99 T 0 0 612 792 C 124.99 351.19 168.49 366.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (1) 125.99 356.19 T 2 F (i) 144.57 356.19 T 0 F (4) 160.49 356.19 T 3 F (\243) 134.99 356.19 T (\243) 150.91 356.19 T 0 0 612 792 C 249.82 349.55 263.44 365.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (D) 250.32 356.19 T 2 9 Q (i) 259.44 351.99 T 0 0 612 792 C 266.44 349.45 292.67 365.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 267.17 356.19 T 2 9 Q (i) 274.95 351.99 T 0 F (1) 286.45 351.99 T (\320) 279.7 351.99 T 0 0 612 792 C 412.97 349.45 439.2 365.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 413.7 356.19 T 2 9 Q (i) 421.48 351.99 T 0 F (1) 432.98 351.99 T (\320) 426.23 351.99 T 0 0 612 792 C 454.54 349.55 468.17 365.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (D) 455.04 356.19 T 2 9 Q (i) 464.16 351.99 T 0 0 612 792 C 500.04 324.72 508.04 337.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 500.54 328.19 T 0 0 612 792 C 201.98 310.72 209.98 323.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 202.48 314.19 T 0 0 612 792 C 529.27 296.72 537.27 309.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 529.77 300.19 T 0 0 612 792 C 289.73 277.74 308.5 296.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 291.03 286.19 T 2 9 Q (X) 300.7 281.99 T 0 0 612 792 C 417.07 209.74 489.52 228.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (V) 419.22 218.19 T 0 9 Q (1) 427.01 213.99 T 2 12 Q (V) 437.51 218.19 T 0 9 Q (2) 445.29 213.99 T 2 12 Q (V) 455.79 218.19 T 0 9 Q (3) 463.58 213.99 T 2 12 Q (V) 474.08 218.19 T 0 9 Q (4) 481.87 213.99 T 3 12 Q (,) 431.51 218.19 T (,) 449.79 218.19 T (,) 468.08 218.19 T 0 0 612 792 C 370.94 195.74 387.89 214.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (D) 372.11 204.19 T 0 9 Q (1) 381.23 199.99 T 0 0 612 792 C 393.89 195.74 409.98 214.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 395.29 204.19 T 0 9 Q (1) 403.08 199.99 T 0 0 612 792 C 436.31 195.74 452.17 214.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (V) 437.59 204.19 T 0 9 Q (1) 445.38 199.99 T 0 0 612 792 C 214.21 169.74 417.13 188.19 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (S) 217.95 178.19 T 2 F (V) 243.82 178.19 T 0 9 Q (1) 251.61 173.99 T 2 12 Q (X) 260.1 178.19 T 0 9 Q (1) 267.89 173.99 T 0 12 Q (\050) 256.11 178.19 T (\051) 272.39 178.19 T 2 F (V) 289.15 178.19 T 0 9 Q (2) 296.94 173.99 T 2 12 Q (X) 305.44 178.19 T 0 9 Q (2) 313.23 173.99 T 0 12 Q (\050) 301.44 178.19 T (\051) 317.73 178.19 T 2 F (V) 334.48 178.19 T 0 9 Q (3) 342.27 173.99 T 2 12 Q (X) 350.77 178.19 T 0 9 Q (3) 358.56 173.99 T 0 12 Q (\050) 346.77 178.19 T (\051) 363.06 178.19 T 2 F (V) 379.82 178.19 T 0 9 Q (4) 387.61 173.99 T 2 12 Q (X) 396.1 178.19 T 0 9 Q (4) 403.89 173.99 T 0 12 Q (\050) 392.11 178.19 T (\051) 408.39 178.19 T (+) 279.39 178.19 T (+) 324.72 178.19 T (+) 370.05 178.19 T (=) 231.05 178.19 T 0 0 612 792 C 348.3 82.74 357.3 97.74 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 349.3 87.74 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "13" 19 %%Page: "12" 20 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (12) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 1 10 Q (Figur) 121.75 713.33 T (e 14. State variables can be highly dependent) 145.46 713.33 T 0 12 Q (If the dependencies are complicated or we wish to make long-term predictions, we can) 108 536.38 T (simplify the phase space by disregarding some of the state variables. Figure) 108 522.38 T (15 shows an) 474.28 522.38 T (example in which the variables) 108 508.38 T ( and) 274.6 508.38 T ( are ignored in making predictions regarding) 312.21 508.38 T (. The usual way of ignoring a variable is to get rid of it by taking expectations.) 122.29 494.38 T 0 10 Q (\0509\051) 528.34 464.64 T 0 12 Q -0.39 (In Equation) 108 423.2 P -0.39 (9, we ignore) 166.94 423.2 P -0.39 ( by computing an expectation based on) 242.72 423.2 P -0.39 (. Where do) 485.19 423.2 P -0.31 (we get) 108 409.2 P -0.31 ( from? W) 196.92 409.2 P -0.31 (ell, in the case of determining) 241.33 409.2 P -0.31 ( for) 441.98 409.2 P -0.31 ( the value) 491.27 409.2 P (of) 108 395.2 T ( is given by) 135.29 395.2 T (; in the case of) 206.26 395.2 T ( we can either invent a prior distribution \050) 309.16 395.2 T 2 F (e.g.,) 509.13 395.2 T 0 F (assume a uniform distribution over the values of) 108 381.2 T (\051 or perform a somewhat more) 365.42 381.2 T -0.16 (tedious calculation involving) 108 367.2 P -0.16 ( as well as) 264.15 367.2 P -0.16 (; in either case information is lost and there) 330.46 367.2 P -0.05 (is some loss of accuracy in prediction. Using sensitivity analysis and some computational) 108 353.2 P (ef) 108 339.2 T (fort you can quantify the loss of information, but a discussion of sensitivity analysis is) 117.11 339.2 T -0.07 (beyond the scope of this paper \050see von W) 108 325.2 P -0.07 (interfeldt and Edwards [1986] or Section 6.4 in) 310.92 325.2 P (Pearl [1988] for an introduction to the basic issues and techniques\051.) 108 311.2 T 1 10 Q (Figur) 121.75 286.53 T (e 15. Conditioning r) 145.46 286.53 T (educes complexity with loss of information) 230.29 286.53 T 0 12 Q (At this point, we have investigated some of the sources of structure in dynamical systems) 108 108.03 T -0.42 (modeled as Markov chains. In the next section, we investigate additional structure that can) 108 94.03 P (be had by taking into account the structure of decision making.) 108 80.03 T 108 63 540 720 C 232.75 560.38 415.25 710 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 333.91 562.79 396.91 697.79 18 RR 4 X 0 0 0 1 0 0 0 K V 0.5 H 0 Z 0 X N 273.2 595.59 275.41 603.29 277.61 595.59 275.41 595.59 4 Y V 275.41 576.29 275.41 595.59 2 L 4 X V 2 Z 0 X N 309.2 595.59 311.41 603.29 313.61 595.59 311.41 595.59 4 Y V 311.41 576.29 311.41 595.59 2 L 4 X V 0 X N 345.2 595.59 347.41 603.29 349.61 595.59 347.41 595.59 4 Y V 347.41 576.29 347.41 595.59 2 L 4 X V 0 X N 381.2 595.59 383.41 603.29 385.61 595.59 383.41 595.59 4 Y V 383.41 576.29 383.41 595.59 2 L 4 X V 0 X N 262.53 592.28 266.41 585.29 259.41 589.16 260.97 590.72 4 Y V 239.41 612.29 260.97 590.72 2 L 4 X V 0 X N 298.53 592.28 302.41 585.29 295.41 589.16 296.97 590.72 4 Y V 275.41 612.29 296.97 590.72 2 L 4 X V 0 X N 406.53 592.28 410.41 585.29 403.41 589.16 404.97 590.72 4 Y V 383.41 612.29 404.97 590.72 2 L 4 X V 0 X N 339.03 592.28 342.91 585.29 335.91 589.16 337.47 590.72 4 Y V 315.91 612.29 337.47 590.72 2 L 4 X V 0 X N 370.53 592.28 374.41 585.29 367.41 589.16 368.97 590.72 4 Y V 347.41 612.29 368.97 590.72 2 L 4 X V 0 X N 277.61 628.97 275.41 621.29 273.2 628.97 275.41 628.97 4 Y V 275.41 648.29 275.41 628.97 2 L 4 X V 0 X N 313.61 628.97 311.41 621.29 309.2 628.97 311.41 628.97 4 Y V 311.41 648.29 311.41 628.97 2 L 4 X V 0 X N 349.61 628.97 347.41 621.29 345.2 628.97 347.41 628.97 4 Y V 347.41 648.29 347.41 628.97 2 L 4 X V 0 X N 385.61 628.97 383.41 621.29 381.2 628.97 383.41 628.97 4 Y V 383.41 648.29 383.41 628.97 2 L 4 X V 0 X N 298.53 664.28 302.41 657.29 295.41 661.16 296.97 662.72 4 Y V 275.41 684.29 296.97 662.72 2 L 4 X V 0 X N 334.53 664.28 338.41 657.29 331.41 661.16 332.97 662.72 4 Y V 311.41 684.29 332.97 662.72 2 L 4 X V 0 X N 370.53 664.28 374.41 657.29 367.41 661.16 368.97 662.72 4 Y V 347.41 684.29 368.97 662.72 2 L 4 X V 0 X N 406.53 664.28 410.41 657.29 403.41 661.16 404.97 662.72 4 Y V 383.41 684.29 404.97 662.72 2 L 4 X V 0 X N 262.53 664.28 266.41 657.29 259.41 661.16 260.97 662.72 4 Y V 239.41 684.29 260.97 662.72 2 L 4 X V 0 X N 403.41 599.41 410.41 603.29 406.53 596.29 404.97 597.85 4 Y V 383.41 576.29 404.97 597.85 2 L 4 X V 0 X N 403.41 635.41 410.41 639.29 406.53 632.29 404.97 633.85 4 Y V 383.41 612.29 404.97 633.85 2 L 4 X V 0 X N 295.41 635.41 302.41 639.29 298.53 632.29 296.97 633.85 4 Y V 275.41 612.29 296.97 633.85 2 L 4 X V 0 X N 331.41 635.41 338.41 639.29 334.53 632.29 332.97 633.85 4 Y V 311.41 612.29 332.97 633.85 2 L 4 X V 0 X N 367.41 635.41 374.41 639.29 370.53 632.29 368.97 633.85 4 Y V 347.41 612.29 368.97 633.85 2 L 4 X V 0 X N 259.41 635.41 266.41 639.29 262.53 632.29 260.97 633.85 4 Y V 239.41 612.29 260.97 633.85 2 L 4 X V 0 X N 294.72 686.49 302.41 684.29 294.72 682.08 294.72 684.29 4 Y V 275.41 684.29 294.72 684.29 2 L 4 X V 0 X N 294.72 650.49 302.41 648.29 294.72 646.08 294.72 648.29 4 Y V 275.41 648.29 294.72 648.29 2 L 4 X V 0 X N 295.41 671.41 302.41 675.29 298.53 668.29 296.97 669.85 4 Y V 275.41 648.29 296.97 669.85 2 L 4 X V 0 X N 294.72 614.49 302.41 612.29 294.72 610.08 294.72 612.29 4 Y V 275.41 612.29 294.72 612.29 2 L 4 X V 0 X N 294.72 578.49 302.41 576.29 294.72 574.08 294.72 576.29 4 Y V 275.41 576.29 294.72 576.29 2 L 4 X V 0 X N 295.41 599.41 302.41 603.29 298.53 596.29 296.97 597.85 4 Y V 275.41 576.29 296.97 597.85 2 L 4 X V 0 X N 330.72 686.49 338.41 684.29 330.72 682.08 330.72 684.29 4 Y V 311.41 684.29 330.72 684.29 2 L 4 X V 0 X N 330.72 650.49 338.41 648.29 330.72 646.08 330.72 648.29 4 Y V 311.41 648.29 330.72 648.29 2 L 4 X V 0 X N 331.41 671.41 338.41 675.29 334.53 668.29 332.97 669.85 4 Y V 311.41 648.29 332.97 669.85 2 L 4 X V 0 X N 330.72 614.49 338.41 612.29 330.72 610.08 330.72 612.29 4 Y V 311.41 612.29 330.72 612.29 2 L 4 X V 0 X N 330.72 578.49 338.41 576.29 330.72 574.08 330.72 576.29 4 Y V 311.41 576.29 330.72 576.29 2 L 4 X V 0 X N 331.41 599.41 338.41 603.29 334.53 596.29 332.97 597.85 4 Y V 311.41 576.29 332.97 597.85 2 L 4 X V 0 X N 366.72 686.49 374.41 684.29 366.72 682.08 366.72 684.29 4 Y V 347.41 684.29 366.72 684.29 2 L 4 X V 0 X N 366.72 650.49 374.41 648.29 366.72 646.08 366.72 648.29 4 Y V 347.41 648.29 366.72 648.29 2 L 4 X V 0 X N 367.41 671.41 374.41 675.29 370.53 668.29 368.97 669.85 4 Y V 347.41 648.29 368.97 669.85 2 L 4 X V 0 X N 366.72 614.49 374.41 612.29 366.72 610.08 366.72 612.29 4 Y V 347.41 612.29 366.72 612.29 2 L 4 X V 0 X N 366.72 578.49 374.41 576.29 366.72 574.08 366.72 576.29 4 Y V 347.41 576.29 366.72 576.29 2 L 4 X V 0 X N 367.41 599.41 374.41 603.29 370.53 596.29 368.97 597.85 4 Y V 347.41 576.29 368.97 597.85 2 L 4 X V 0 X N 402.72 686.49 410.41 684.29 402.72 682.08 402.72 684.29 4 Y V 383.41 684.29 402.72 684.29 2 L 4 X V 0 X N 402.72 650.49 410.41 648.29 402.72 646.08 402.72 648.29 4 Y V 383.41 648.29 402.72 648.29 2 L 4 X V 0 X N 403.41 671.41 410.41 675.29 406.53 668.29 404.97 669.85 4 Y V 383.41 648.29 404.97 669.85 2 L 4 X V 0 X N 402.72 614.49 410.41 612.29 402.72 610.08 402.72 612.29 4 Y V 383.41 612.29 402.72 612.29 2 L 4 X V 0 X N 402.72 578.49 410.41 576.29 402.72 574.08 402.72 576.29 4 Y V 383.41 576.29 402.72 576.29 2 L 4 X V 0 X N 258.72 686.49 266.41 684.29 258.72 682.08 258.72 684.29 4 Y V 239.41 684.29 258.72 684.29 2 L 4 X V 0 X N 258.72 650.49 266.41 648.29 258.72 646.08 258.72 648.29 4 Y V 239.41 648.29 258.72 648.29 2 L 4 X V 0 X N 259.41 671.41 266.41 675.29 262.53 668.29 260.97 669.85 4 Y V 239.41 648.29 260.97 669.85 2 L 4 X V 0 X N 258.72 614.49 266.41 612.29 258.72 610.08 258.72 612.29 4 Y V 239.41 612.29 258.72 612.29 2 L 4 X V 0 X N 258.72 578.49 266.41 576.29 258.72 574.08 258.72 576.29 4 Y V 239.41 576.29 258.72 576.29 2 L 4 X V 0 X N 259.41 599.41 266.41 603.29 262.53 596.29 260.97 597.85 4 Y V 239.41 576.29 260.97 597.85 2 L 4 X V 0 X N 234.91 567.29 252.91 693.29 R 7 X V 401.05 566.57 419.05 692.57 R V 90 450 9 9 275.41 684.29 G 0 Z 0 X 90 450 9 9 275.41 684.29 A 7 X 90 450 9 9 275.41 576.29 G 0 X 90 450 9 9 275.41 576.29 A 7 X 90 450 9 9 275.41 612.29 G 0 X 90 450 9 9 275.41 612.29 A 7 X 90 450 9 9 275.41 648.29 G 0 X 90 450 9 9 275.41 648.29 A 7 X 90 450 9 9 311.41 684.29 G 0 X 90 450 9 9 311.41 684.29 A 7 X 90 450 9 9 311.41 576.29 G 0 X 90 450 9 9 311.41 576.29 A 7 X 90 450 9 9 311.41 612.29 G 0 X 90 450 9 9 311.41 612.29 A 7 X 90 450 9 9 311.41 648.29 G 0 X 90 450 9 9 311.41 648.29 A 7 X 90 450 9 9 347.41 684.29 G 0 X 90 450 9 9 347.41 684.29 A 7 X 90 450 9 9 347.41 576.29 G 0 X 90 450 9 9 347.41 576.29 A 7 X 90 450 9 9 347.41 612.29 G 0 X 90 450 9 9 347.41 612.29 A 7 X 90 450 9 9 347.41 648.29 G 0 X 90 450 9 9 347.41 648.29 A 7 X 90 450 9 9 383.41 684.29 G 0 X 90 450 9 9 383.41 684.29 A 7 X 90 450 9 9 383.41 576.29 G 0 X 90 450 9 9 383.41 576.29 A 7 X 90 450 9 9 383.41 612.29 G 0 X 90 450 9 9 383.41 612.29 A 7 X 90 450 9 9 383.41 648.29 G 0 X 90 450 9 9 383.41 648.29 A 369.91 634.79 396.91 661.79 13.5 RR N 2 9 Q (t) 263.63 701.66 T 0 F (0) 280.2 701.66 T (=) 270.62 701.66 T 2 F (t) 337.65 701.66 T 0 F (2) 354.21 701.66 T (=) 344.64 701.66 T 2 F (t) 300.64 701.66 T 0 F (1) 317.2 701.66 T (=) 307.63 701.66 T 2 F (t) 374.65 701.66 T 0 F (3) 391.22 701.66 T (=) 381.65 701.66 T 2 F (X) 239.79 681.73 T 0 F (1) 245.74 676.78 T 2 F (X) 239.79 645.94 T 0 F (2) 245.74 640.99 T 2 F (X) 239.79 610.15 T 0 F (3) 245.74 605.2 T 2 F (X) 239.79 574.36 T 0 F (4) 245.74 569.41 T 108 63 540 720 C 0 0 612 792 C 260.3 501.65 274.59 517.58 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 260.81 508.38 T 0 9 Q (3) 268.59 504.18 T 0 0 612 792 C 297.92 501.65 312.21 517.58 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 298.43 508.38 T 0 9 Q (4) 306.21 504.18 T 0 0 612 792 C 108 487.65 122.29 503.58 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 108.5 494.38 T 0 9 Q (1) 116.29 490.18 T 0 0 612 792 C 141.63 443.2 494.71 478.38 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 146.1 464.64 T (r) 152.77 464.64 T 2 F (X) 160.77 464.64 T 0 9 Q (2) 168.55 460.45 T 2 F (t) 177.55 460.45 T 3 F (,) 173.05 460.45 T 2 12 Q (X) 184.9 464.64 T 0 9 Q (2) 192.69 460.45 T 2 F (t) 201.68 460.45 T 0 F (1) 213.18 460.45 T (\320) 206.43 460.45 T 3 F (,) 197.19 460.45 T 0 12 Q (\050) 156.77 464.64 T (\051) 217.68 464.64 T (P) 270.69 464.64 T (r) 277.36 464.64 T 2 F (X) 285.36 464.64 T 0 9 Q (2) 293.14 460.45 T 2 F (t) 302.14 460.45 T 3 F (,) 297.64 460.45 T 2 12 Q (X) 309.49 464.64 T 0 9 Q (2) 317.28 460.45 T 2 F (t) 326.27 460.45 T 0 F (1) 337.77 460.45 T (\320) 331.02 460.45 T 3 F (,) 321.78 460.45 T 2 12 Q (X) 348.27 464.64 T 0 9 Q (3) 356.06 460.45 T 2 F (t) 365.05 460.45 T 0 F (1) 376.55 460.45 T (\320) 369.8 460.45 T 3 F (,) 360.56 460.45 T 2 12 Q (x) 395.8 464.64 T 0 F (=) 385.04 464.64 T 3 F (,) 342.27 464.64 T 0 F (\050) 281.36 464.64 T (\051) 401.13 464.64 T (P) 417.71 464.64 T (r) 424.39 464.64 T 2 F (X) 432.38 464.64 T 0 9 Q (3) 440.17 460.45 T 2 F (t) 449.16 460.45 T 0 F (1) 460.66 460.45 T (\320) 453.91 460.45 T 3 F (,) 444.67 460.45 T 2 12 Q (x) 479.92 464.64 T 0 F (=) 469.15 464.64 T (\050) 428.38 464.64 T (\051) 485.24 464.64 T 3 F (\264) 408.13 464.64 T 2 9 Q (x) 240.44 451.55 T 3 F (W) 255.34 451.55 T 2 6 Q (X) 262.6 449 T 0 F (3) 266.49 445.7 T 3 9 Q (\316) 246.68 451.55 T 3 18 Q (\345) 248.55 461.44 T 0 12 Q (=) 227.67 464.64 T 182.45 458.2 182.45 472.45 2 L 0.54 H 2 Z N 307.04 458.2 307.04 472.45 2 L N 0 0 612 792 C 228.43 416.48 242.72 432.41 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 228.93 423.2 T 0 9 Q (3) 236.71 419 T 0 0 612 792 C 430.3 416.46 485.19 432.41 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 431.52 423.2 T (r) 438.2 423.2 T 2 F (X) 446.19 423.2 T 0 9 Q (3) 453.98 419 T 2 F (t) 462.97 419 T 0 F (1) 474.47 419 T (\320) 467.72 419 T 3 F (,) 458.48 419 T 0 12 Q (\050) 442.19 423.2 T (\051) 478.97 423.2 T 0 0 612 792 C 142.04 402.46 196.92 418.41 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 143.26 409.2 T (r) 149.93 409.2 T 2 F (X) 157.92 409.2 T 0 9 Q (3) 165.71 405 T 2 F (t) 174.71 405 T 0 F (1) 186.2 405 T (\320) 179.46 405 T 3 F (,) 170.21 405 T 0 12 Q (\050) 153.93 409.2 T (\051) 190.7 409.2 T 0 0 612 792 C 385.11 401.48 441.98 418.41 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 386.05 409.2 T (r) 392.72 409.2 T 2 F (X) 400.71 409.2 T 0 9 Q (1) 408.5 405 T (3) 417.49 405 T 3 F (,) 413 405 T 3 12 Q (q) 426.84 409.2 T 2 9 Q (t) 433.55 405 T 0 12 Q (\050) 396.71 409.2 T (\051) 436.05 409.2 T 424.39 402.75 424.39 417 2 L 0.54 H 2 Z N 0 0 612 792 C 461.35 405.59 491.27 418.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 461.76 409.2 T 0 F (1) 483.86 409.2 T (=) 471.1 409.2 T 0 0 612 792 C 121 388.48 135.29 404.41 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 121.5 395.2 T 0 9 Q (3) 129.29 391 T 0 0 612 792 C 193.96 386.75 206.26 405.2 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (q) 195 395.2 T 2 9 Q (t) 201.71 391 T 0 0 612 792 C 279.24 391.59 309.16 404.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 279.65 395.2 T 0 F (0) 301.75 395.2 T (=) 288.98 395.2 T 0 0 612 792 C 343.31 370.95 365.42 391.2 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 344.61 381.2 T 2 9 Q (X) 354.28 377 T 0 6 Q (3) 360.12 374.45 T 0 0 612 792 C 249.85 360.48 264.15 376.41 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 250.36 367.2 T 0 9 Q (4) 258.14 363 T 0 0 612 792 C 316.17 360.48 330.46 376.41 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 316.67 367.2 T 0 9 Q (3) 324.46 363 T 0 0 612 792 C 108 63 540 720 C 238 132.03 410 283.2 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 261.75 207.2 396.75 270.2 18 RR 4 X 0 0 0 1 0 0 0 K V 0.5 H 0 Z 0 X N 333.75 207.2 396.75 270.2 18 RR N 297.75 207.2 396.75 270.2 18 RR N 369.75 243.2 396.75 270.2 13.5 RR 4 X V 0 X N 297.75 171.2 324.75 198.2 13.5 RR 3 X V 0 X N 261.75 171.2 288.75 198.2 13.5 RR 3 X V 0 X N 261.75 135.2 288.75 162.2 13.5 RR 3 X V 0 X N 403.25 171.82 410.25 175.7 406.37 168.7 404.81 170.26 4 Y V 383.25 148.7 404.81 170.26 2 L 3 X V 2 Z 0 X N 403.25 207.82 410.25 211.7 406.37 204.7 404.81 206.26 4 Y V 383.25 184.7 404.81 206.26 2 L 3 X V 0 X N 295.25 207.82 302.25 211.7 298.37 204.7 296.81 206.26 4 Y V 275.25 184.7 296.81 206.26 2 L 3 X V 0 X N 331.25 207.82 338.25 211.7 334.37 204.7 332.81 206.26 4 Y V 311.25 184.7 332.81 206.26 2 L 3 X V 0 X N 367.25 207.82 374.25 211.7 370.37 204.7 368.81 206.26 4 Y V 347.25 184.7 368.81 206.26 2 L 3 X V 0 X N 259.25 207.82 266.25 211.7 262.37 204.7 260.81 206.26 4 Y V 239.25 184.7 260.81 206.26 2 L 3 X V 0 X N 294.56 258.9 302.25 256.7 294.56 254.49 294.56 256.7 4 Y V 275.25 256.7 294.56 256.7 2 L 3 X V 0 X N 294.56 222.9 302.25 220.7 294.56 218.49 294.56 220.7 4 Y V 275.25 220.7 294.56 220.7 2 L 3 X V 0 X N 295.25 243.82 302.25 247.7 298.37 240.7 296.81 242.26 4 Y V 275.25 220.7 296.81 242.26 2 L 3 X V 0 X N 294.56 186.9 302.25 184.7 294.56 182.49 294.56 184.7 4 Y V 275.25 184.7 294.56 184.7 2 L 3 X V 0 X N 294.56 150.9 302.25 148.7 294.56 146.49 294.56 148.7 4 Y V 275.25 148.7 294.56 148.7 2 L 3 X V 0 X N 295.25 171.82 302.25 175.7 298.37 168.7 296.81 170.26 4 Y V 275.25 148.7 296.81 170.26 2 L 3 X V 0 X N 330.56 258.9 338.25 256.7 330.56 254.49 330.56 256.7 4 Y V 311.25 256.7 330.56 256.7 2 L 3 X V 0 X N 330.56 222.9 338.25 220.7 330.56 218.49 330.56 220.7 4 Y V 311.25 220.7 330.56 220.7 2 L 3 X V 0 X N 331.25 243.82 338.25 247.7 334.37 240.7 332.81 242.26 4 Y V 311.25 220.7 332.81 242.26 2 L 3 X V 0 X N 330.56 186.9 338.25 184.7 330.56 182.49 330.56 184.7 4 Y V 311.25 184.7 330.56 184.7 2 L 3 X V 0 X N 330.56 150.9 338.25 148.7 330.56 146.49 330.56 148.7 4 Y V 311.25 148.7 330.56 148.7 2 L 3 X V 0 X N 331.25 171.82 338.25 175.7 334.37 168.7 332.81 170.26 4 Y V 311.25 148.7 332.81 170.26 2 L 3 X V 0 X N 366.56 258.9 374.25 256.7 366.56 254.49 366.56 256.7 4 Y V 347.25 256.7 366.56 256.7 2 L 3 X V 0 X N 366.56 222.9 374.25 220.7 366.56 218.49 366.56 220.7 4 Y V 347.25 220.7 366.56 220.7 2 L 3 X V 0 X N 367.25 243.82 374.25 247.7 370.37 240.7 368.81 242.26 4 Y V 347.25 220.7 368.81 242.26 2 L 3 X V 0 X N 366.56 186.9 374.25 184.7 366.56 182.49 366.56 184.7 4 Y V 347.25 184.7 366.56 184.7 2 L 3 X V 0 X N 366.56 150.9 374.25 148.7 366.56 146.49 366.56 148.7 4 Y V 347.25 148.7 366.56 148.7 2 L 3 X V 0 X N 367.25 171.82 374.25 175.7 370.37 168.7 368.81 170.26 4 Y V 347.25 148.7 368.81 170.26 2 L 3 X V 0 X N 402.56 258.9 410.25 256.7 402.56 254.49 402.56 256.7 4 Y V 383.25 256.7 402.56 256.7 2 L 3 X V 0 X N 402.56 222.9 410.25 220.7 402.56 218.49 402.56 220.7 4 Y V 383.25 220.7 402.56 220.7 2 L 3 X V 0 X N 403.25 243.82 410.25 247.7 406.37 240.7 404.81 242.26 4 Y V 383.25 220.7 404.81 242.26 2 L 3 X V 0 X N 402.56 186.9 410.25 184.7 402.56 182.49 402.56 184.7 4 Y V 383.25 184.7 402.56 184.7 2 L 3 X V 0 X N 402.56 150.9 410.25 148.7 402.56 146.49 402.56 148.7 4 Y V 383.25 148.7 402.56 148.7 2 L 3 X V 0 X N 258.56 258.9 266.25 256.7 258.56 254.49 258.56 256.7 4 Y V 239.25 256.7 258.56 256.7 2 L 3 X V 0 X N 258.56 222.9 266.25 220.7 258.56 218.49 258.56 220.7 4 Y V 239.25 220.7 258.56 220.7 2 L 3 X V 0 X N 259.25 243.82 266.25 247.7 262.37 240.7 260.81 242.26 4 Y V 239.25 220.7 260.81 242.26 2 L 3 X V 0 X N 258.56 186.9 266.25 184.7 258.56 182.49 258.56 184.7 4 Y V 239.25 184.7 258.56 184.7 2 L 3 X V 0 X N 258.56 150.9 266.25 148.7 258.56 146.49 258.56 148.7 4 Y V 239.25 148.7 258.56 148.7 2 L 3 X V 0 X N 259.25 171.82 266.25 175.7 262.37 168.7 260.81 170.26 4 Y V 239.25 148.7 260.81 170.26 2 L 3 X V 0 X N 234.75 139.7 252.75 265.7 R 7 X V 401.25 139.7 419.25 265.7 R V 90 450 9 9 275.25 256.7 G 0 Z 0 X 90 450 9 9 275.25 256.7 A 7 X 90 450 9 9 275.25 148.7 G 0 X 90 450 9 9 275.25 148.7 A 7 X 90 450 9 9 275.25 184.7 G 0 X 90 450 9 9 275.25 184.7 A 7 X 90 450 9 9 275.25 220.7 G 0 X 90 450 9 9 275.25 220.7 A 7 X 90 450 9 9 311.25 256.7 G 0 X 90 450 9 9 311.25 256.7 A 7 X 90 450 9 9 311.25 148.7 G 0 X 90 450 9 9 311.25 148.7 A 7 X 90 450 9 9 311.25 184.7 G 0 X 90 450 9 9 311.25 184.7 A 7 X 90 450 9 9 311.25 220.7 G 0 X 90 450 9 9 311.25 220.7 A 7 X 90 450 9 9 347.25 256.7 G 0 X 90 450 9 9 347.25 256.7 A 7 X 90 450 9 9 347.25 148.7 G 0 X 90 450 9 9 347.25 148.7 A 7 X 90 450 9 9 347.25 184.7 G 0 X 90 450 9 9 347.25 184.7 A 7 X 90 450 9 9 347.25 220.7 G 0 X 90 450 9 9 347.25 220.7 A 7 X 90 450 9 9 383.25 256.7 G 0 X 90 450 9 9 383.25 256.7 A 7 X 90 450 9 9 383.25 148.7 G 0 X 90 450 9 9 383.25 148.7 A 7 X 90 450 9 9 383.25 184.7 G 0 X 90 450 9 9 383.25 184.7 A 7 X 90 450 9 9 383.25 220.7 G 0 X 90 450 9 9 383.25 220.7 A 2 9 Q (t) 265.09 273.7 T 0 F (0) 281.66 273.7 T (=) 272.09 273.7 T 2 F (t) 339.11 273.7 T 0 F (2) 355.67 273.7 T (=) 346.11 273.7 T 2 F (t) 302.1 273.7 T 0 F (1) 318.67 273.7 T (=) 309.1 273.7 T 2 F (t) 376.12 273.7 T 0 F (3) 392.68 273.7 T (=) 383.11 273.7 T 2 F (X) 241.25 253.77 T 0 F (1) 247.21 248.82 T 2 F (X) 241.25 217.98 T 0 F (2) 247.21 213.03 T 2 F (X) 241.25 182.19 T 0 F (3) 247.21 177.24 T 2 F (X) 241.25 146.4 T 0 F (4) 247.21 141.45 T 108 63 540 720 C 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "12" 20 %%Page: "11" 21 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (1) 320.15 42.62 T (1) 323.85 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (Dependencies such at those shown in Figure) 108 712 T (1) 324.3 712 T (1 can be used to simplify various sorts of) 329.86 712 T -0.25 (inference useful for planning and prediction. Suppose the dynamical system is completely) 108 698 P -0.18 (observable. An observation) 108 684 P -0.18 ( takes the form of vector assigning values to each state vari-) 254.4 684 P (able at time) 108 670 T (.) 172.33 670 T 0 10 Q (\0508\051) 528.34 644 T 0 12 Q (Figure) 108 615.55 T (12 shows the portion of phase space that needs to be accounted for in determining) 142.33 615.55 T ( for) 164.86 601.55 T (. In this particular example, the number of relevant state) 265.08 601.55 T (variables grows to include the entire state space eventually) 108 587.55 T (, but this need not be the case.) 389.16 587.55 T 1 10 Q (Figur) 121.75 562.88 T (e 12. Dependencies for) 145.46 562.88 T (-state pr) 251.49 562.88 T (ediction) 287.13 562.88 T 0 12 Q -0.11 (Figure) 108 386.58 P -0.11 (13 provides an example in which a proper subset of the state variables are indepen-) 142.33 386.58 P -0.13 (dent of the other variables no matter how many steps we consider) 108 372.58 P -0.13 (. In this example, the set) 420.81 372.58 P ( is said to form a) 247.16 358.58 T 2 F (Markov partition) 330.49 358.58 T 0 F (of the state space. Infor-) 415.82 358.58 T (mation regarding) 108 344.58 T ( is suf) 261.43 344.58 T (\336cient for predicting future values of those same) 289.88 344.58 T (variables.) 108 330.58 T 1 10 Q (Figur) 121.75 305.92 T (e 13. Markov partition r) 145.46 305.92 T (educes complexity with no loss of information) 248.88 305.92 T 0 12 Q (Dependencies among state variables can simplify short-term predictions and, in some) 108 129.26 T (cases, long-term predictions with no loss of information or accuracy) 108 115.26 T (. Unfortunately) 434.5 115.26 T (, as) 507.04 115.26 T (Figure) 108 101.26 T (14 illustrates, sometimes the dependencies are hopelessly tangled, making even) 142.33 101.26 T (short-term predictions take into account the entire state space.) 108 87.26 T 242.1 675.55 254.4 694 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (q) 243.14 684 T 2 9 Q (t) 249.85 679.8 T 0 0 612 792 C 166.99 666.53 172.33 679.21 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 167.49 670 T 0 0 612 792 C 231.49 635.55 404.85 654 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (q) 234.25 644 T 2 9 Q (t) 240.95 639.8 T 2 12 Q (X) 268.78 644 T 0 9 Q (1) 276.57 639.8 T 2 F (t) 285.57 639.8 T 3 F (,) 281.07 639.8 T 2 12 Q (x) 302.82 644 T 0 9 Q (1) 308.61 639.8 T 2 F (t) 317.6 639.8 T 3 F (,) 313.11 639.8 T 0 12 Q (=) 292.06 644 T 3 F (\274) 326.1 644 T 2 F (X) 344.1 644 T 2 9 Q (L) 351.89 639.8 T (t) 361.39 639.8 T 3 F (,) 356.89 639.8 T 2 12 Q (x) 378.65 644 T 2 9 Q (L) 384.43 639.8 T (t) 393.93 639.8 T 3 F (,) 389.44 639.8 T 0 12 Q (=) 367.89 644 T 3 F (,) 320.11 644 T (,) 338.1 644 T (\341) 263.42 644 T (\361) 396.44 644 T 0 F (=) 249.45 644 T 0 0 612 792 C 108 593.83 164.86 610.76 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 108.93 601.55 T (r) 115.6 601.55 T 2 F (X) 123.6 601.55 T 0 9 Q (1) 131.38 597.35 T (3) 140.38 597.35 T 3 F (,) 135.88 597.35 T 3 12 Q (q) 149.73 601.55 T 2 9 Q (t) 156.43 597.35 T 0 12 Q (\050) 119.6 601.55 T (\051) 158.93 601.55 T 147.28 595.1 147.28 609.35 2 L 0.54 H 2 Z N 0 0 612 792 C 184.85 596.55 265.09 611.55 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 185.91 601.55 T 0 F (0) 212.72 601.55 T (1) 224.71 601.55 T (2) 236.71 601.55 T (3) 248.71 601.55 T 3 F (,) 218.72 601.55 T (,) 230.71 601.55 T (,) 242.71 601.55 T ({) 205.8 601.55 T (}) 255.26 601.55 T (\316) 192.25 601.55 T 0 0 612 792 C 243.49 559.41 251.49 572.09 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 243.99 562.88 T 0 0 612 792 C 108 63 540 720 C 231.59 410.58 416.41 559.55 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 257.89 411.61 392.89 546.61 18 RR 4 X 0 0 0 1 0 0 0 K V 0.5 H 0 Z 0 X N 399.39 448.23 406.39 452.11 402.51 445.11 400.95 446.67 4 Y V 379.39 425.11 400.95 446.67 2 L 4 X V 2 Z 0 X N 399.39 484.23 406.39 488.11 402.51 481.11 400.95 482.67 4 Y V 379.39 461.11 400.95 482.67 2 L 4 X V 0 X N 291.39 484.23 298.39 488.11 294.51 481.11 292.95 482.67 4 Y V 271.39 461.11 292.95 482.67 2 L 4 X V 0 X N 327.39 484.23 334.39 488.11 330.51 481.11 328.95 482.67 4 Y V 307.39 461.11 328.95 482.67 2 L 4 X V 0 X N 363.39 484.23 370.39 488.11 366.51 481.11 364.95 482.67 4 Y V 343.39 461.11 364.95 482.67 2 L 4 X V 0 X N 255.4 484.23 262.39 488.11 258.51 481.11 256.95 482.67 4 Y V 235.39 461.11 256.95 482.67 2 L 4 X V 0 X N 290.7 535.31 298.39 533.11 290.7 530.9 290.7 533.11 4 Y V 271.39 533.11 290.7 533.11 2 L 4 X V 0 X N 290.7 499.32 298.39 497.11 290.7 494.9 290.7 497.11 4 Y V 271.39 497.11 290.7 497.11 2 L 4 X V 0 X N 291.39 520.23 298.39 524.11 294.51 517.11 292.95 518.67 4 Y V 271.39 497.11 292.95 518.67 2 L 4 X V 0 X N 290.7 463.32 298.39 461.11 290.7 458.9 290.7 461.11 4 Y V 271.39 461.11 290.7 461.11 2 L 4 X V 0 X N 290.7 427.32 298.39 425.11 290.7 422.9 290.7 425.11 4 Y V 271.39 425.11 290.7 425.11 2 L 4 X V 0 X N 291.39 448.23 298.39 452.11 294.51 445.11 292.95 446.67 4 Y V 271.39 425.11 292.95 446.67 2 L 4 X V 0 X N 326.7 535.31 334.39 533.11 326.7 530.9 326.7 533.11 4 Y V 307.39 533.11 326.7 533.11 2 L 4 X V 0 X N 326.7 499.32 334.39 497.11 326.7 494.9 326.7 497.11 4 Y V 307.39 497.11 326.7 497.11 2 L 4 X V 0 X N 327.39 520.23 334.39 524.11 330.51 517.11 328.95 518.67 4 Y V 307.39 497.11 328.95 518.67 2 L 4 X V 0 X N 326.7 463.32 334.39 461.11 326.7 458.9 326.7 461.11 4 Y V 307.39 461.11 326.7 461.11 2 L 4 X V 0 X N 326.7 427.32 334.39 425.11 326.7 422.9 326.7 425.11 4 Y V 307.39 425.11 326.7 425.11 2 L 4 X V 0 X N 327.39 448.23 334.39 452.11 330.51 445.11 328.95 446.67 4 Y V 307.39 425.11 328.95 446.67 2 L 4 X V 0 X N 362.7 535.31 370.39 533.11 362.7 530.9 362.7 533.11 4 Y V 343.39 533.11 362.7 533.11 2 L 4 X V 0 X N 362.7 499.32 370.39 497.11 362.7 494.9 362.7 497.11 4 Y V 343.39 497.11 362.7 497.11 2 L 4 X V 0 X N 363.39 520.23 370.39 524.11 366.51 517.11 364.95 518.67 4 Y V 343.39 497.11 364.95 518.67 2 L 4 X V 0 X N 362.7 463.32 370.39 461.11 362.7 458.9 362.7 461.11 4 Y V 343.39 461.11 362.7 461.11 2 L 4 X V 0 X N 362.7 427.32 370.39 425.11 362.7 422.9 362.7 425.11 4 Y V 343.39 425.11 362.7 425.11 2 L 4 X V 0 X N 363.39 448.23 370.39 452.11 366.51 445.11 364.95 446.67 4 Y V 343.39 425.11 364.95 446.67 2 L 4 X V 0 X N 398.7 535.31 406.39 533.11 398.7 530.9 398.7 533.11 4 Y V 379.39 533.11 398.7 533.11 2 L 4 X V 0 X N 398.7 499.32 406.39 497.11 398.7 494.9 398.7 497.11 4 Y V 379.39 497.11 398.7 497.11 2 L 4 X V 0 X N 399.39 520.23 406.39 524.11 402.51 517.11 400.95 518.67 4 Y V 379.39 497.11 400.95 518.67 2 L 4 X V 0 X N 398.7 463.32 406.39 461.11 398.7 458.9 398.7 461.11 4 Y V 379.39 461.11 398.7 461.11 2 L 4 X V 0 X N 398.7 427.32 406.39 425.11 398.7 422.9 398.7 425.11 4 Y V 379.39 425.11 398.7 425.11 2 L 4 X V 0 X N 254.7 535.31 262.39 533.11 254.7 530.9 254.7 533.11 4 Y V 235.39 533.11 254.7 533.11 2 L 4 X V 0 X N 254.7 499.32 262.39 497.11 254.7 494.9 254.7 497.11 4 Y V 235.39 497.11 254.7 497.11 2 L 4 X V 0 X N 255.4 520.23 262.39 524.11 258.51 517.11 256.95 518.67 4 Y V 235.39 497.11 256.95 518.67 2 L 4 X V 0 X N 254.7 463.32 262.39 461.11 254.7 458.9 254.7 461.11 4 Y V 235.39 461.11 254.7 461.11 2 L 4 X V 0 X N 254.7 427.32 262.39 425.11 254.7 422.9 254.7 425.11 4 Y V 235.39 425.11 254.7 425.11 2 L 4 X V 0 X N 255.4 448.23 262.39 452.11 258.51 445.11 256.95 446.67 4 Y V 235.39 425.11 256.95 446.67 2 L 4 X V 0 X N 232.59 416.11 250.59 542.11 R 7 X V 397.39 416.11 415.39 542.11 R V 90 450 9 9 271.39 533.11 G 0 Z 0 X 90 450 9 9 271.39 533.11 A 7 X 90 450 9 9 271.39 425.11 G 0 X 90 450 9 9 271.39 425.11 A 7 X 90 450 9 9 271.39 461.11 G 0 X 90 450 9 9 271.39 461.11 A 7 X 90 450 9 9 271.39 497.11 G 0 X 90 450 9 9 271.39 497.11 A 7 X 90 450 9 9 307.39 533.11 G 0 X 90 450 9 9 307.39 533.11 A 7 X 90 450 9 9 307.39 425.11 G 0 X 90 450 9 9 307.39 425.11 A 7 X 90 450 9 9 307.39 461.11 G 0 X 90 450 9 9 307.39 461.11 A 7 X 90 450 9 9 307.39 497.11 G 0 X 90 450 9 9 307.39 497.11 A 7 X 90 450 9 9 343.39 533.11 G 0 X 90 450 9 9 343.39 533.11 A 7 X 90 450 9 9 343.39 425.11 G 0 X 90 450 9 9 343.39 425.11 A 7 X 90 450 9 9 343.39 461.11 G 0 X 90 450 9 9 343.39 461.11 A 7 X 90 450 9 9 343.39 497.11 G 0 X 90 450 9 9 343.39 497.11 A 7 X 90 450 9 9 379.39 533.11 G 0 X 90 450 9 9 379.39 533.11 A 7 X 90 450 9 9 379.39 425.11 G 0 X 90 450 9 9 379.39 425.11 A 7 X 90 450 9 9 379.39 461.11 G 0 X 90 450 9 9 379.39 461.11 A 7 X 90 450 9 9 379.39 497.11 G 0 X 90 450 9 9 379.39 497.11 A 365.89 519.61 392.89 546.61 13.5 RR N 329.89 483.61 392.89 546.61 18 RR N 293.89 447.61 392.89 546.61 18 RR N 2 9 Q (t) 259 551.8 T 0 F (0) 275.57 551.8 T (=) 266 551.8 T 2 F (t) 333.02 551.8 T 0 F (2) 349.58 551.8 T (=) 340.01 551.8 T 2 F (t) 296.01 551.8 T 0 F (1) 312.58 551.8 T (=) 303.01 551.8 T 2 F (t) 370.02 551.8 T 0 F (3) 386.59 551.8 T (=) 377.02 551.8 T 2 F (X) 235.16 531.87 T 0 F (1) 241.11 526.92 T 2 F (X) 235.16 496.08 T 0 F (2) 241.11 491.14 T 2 F (X) 235.16 460.29 T 0 F (3) 241.11 455.35 T 2 F (X) 235.16 424.51 T 0 F (4) 241.11 419.55 T 108 63 540 720 C 0 0 612 792 C 108 351.86 247.16 367.79 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 126.35 358.58 T 0 9 Q (1) 134.13 354.38 T 2 12 Q (X) 144.63 358.58 T 0 9 Q (2) 152.42 354.38 T 2 12 Q (X) 162.92 358.58 T 0 9 Q (3) 170.71 354.38 T 3 12 Q (,) 138.63 358.58 T (,) 156.92 358.58 T ({) 119.43 358.58 T (}) 175.76 358.58 T 2 F (X) 198.44 358.58 T 0 9 Q (4) 206.22 354.38 T 2 12 Q (X) 216.72 358.58 T 0 9 Q (5) 224.51 354.38 T 3 12 Q (,) 210.72 358.58 T ({) 191.52 358.58 T (}) 229.56 358.58 T (,) 183.52 358.58 T ({) 110.52 358.58 T (}) 237.88 358.58 T 0 0 612 792 C 193.32 337.86 261.43 353.79 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 202.74 344.58 T 0 9 Q (1) 210.53 340.38 T 2 12 Q (X) 221.03 344.58 T 0 9 Q (2) 228.82 340.38 T 2 12 Q (X) 239.32 344.58 T 0 9 Q (3) 247.1 340.38 T 3 12 Q (,) 215.03 344.58 T (,) 233.32 344.58 T ({) 195.83 344.58 T (}) 252.16 344.58 T 0 0 612 792 C 108 63 540 720 C 247.03 153.26 400.97 302.58 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 283.14 238.45 284.79 244.22 286.45 238.45 284.79 238.45 4 Y 0 X 0 0 0 1 0 0 0 K V 284.79 222.62 284.79 238.45 2 L 0.5 H 2 Z N 311.94 238.45 313.59 244.22 315.24 238.45 313.59 238.45 4 Y V 313.59 222.62 313.59 238.45 2 L N 340.74 238.45 342.39 244.22 344.04 238.45 342.39 238.45 4 Y V 342.39 222.62 342.39 238.45 2 L N 369.54 238.45 371.19 244.22 372.84 238.45 371.19 238.45 4 Y V 371.19 222.62 371.19 238.45 2 L N 300.62 281.88 306.39 280.22 300.62 278.57 300.62 280.22 4 Y V 284.79 280.22 300.62 280.22 2 L N 300.62 253.08 306.39 251.42 300.62 249.77 300.62 251.42 4 Y V 284.79 251.42 300.62 251.42 2 L N 303.48 235.07 306.39 229.82 301.14 232.73 302.31 233.9 4 Y V 284.79 251.42 302.31 233.9 2 L N 301.14 270.11 306.39 273.02 303.48 267.77 302.31 268.94 4 Y V 284.79 251.42 302.31 268.94 2 L N 300.62 224.28 306.39 222.62 300.62 220.97 300.62 222.62 4 Y V 284.79 222.62 300.62 222.62 2 L N 300.62 195.48 306.39 193.82 300.62 192.17 300.62 193.82 4 Y V 284.79 193.82 300.62 193.82 2 L N 300.62 166.68 306.39 165.02 300.62 163.37 300.62 165.02 4 Y V 284.79 165.02 300.62 165.02 2 L N 301.14 183.72 306.39 186.62 303.48 181.38 302.31 182.55 4 Y V 284.79 165.02 302.31 182.54 2 L N 329.42 281.88 335.19 280.22 329.42 278.57 329.42 280.22 4 Y V 313.59 280.22 329.42 280.22 2 L N 329.42 253.08 335.19 251.42 329.42 249.77 329.42 251.42 4 Y V 313.59 251.42 329.42 251.42 2 L N 332.28 235.07 335.19 229.82 329.94 232.73 331.11 233.9 4 Y V 313.59 251.42 331.11 233.9 2 L N 329.94 270.11 335.19 273.02 332.28 267.78 331.11 268.94 4 Y V 313.59 251.42 331.11 268.94 2 L N 329.42 224.28 335.19 222.62 329.42 220.97 329.42 222.62 4 Y V 313.59 222.62 329.42 222.62 2 L N 329.42 195.48 335.19 193.82 329.42 192.17 329.42 193.82 4 Y V 313.59 193.82 329.42 193.82 2 L N 329.42 166.68 335.19 165.02 329.42 163.37 329.42 165.02 4 Y V 313.59 165.02 329.42 165.02 2 L N 329.94 183.71 335.19 186.62 332.28 181.38 331.11 182.54 4 Y V 313.59 165.02 331.11 182.54 2 L N 358.22 281.88 363.99 280.22 358.22 278.57 358.22 280.22 4 Y V 342.39 280.22 358.22 280.22 2 L N 358.22 253.08 363.99 251.42 358.22 249.77 358.22 251.42 4 Y V 342.39 251.42 358.22 251.42 2 L N 361.08 235.07 363.99 229.82 358.74 232.73 359.91 233.9 4 Y V 342.39 251.42 359.91 233.9 2 L N 358.74 270.11 363.99 273.02 361.08 267.77 359.91 268.94 4 Y V 342.39 251.42 359.91 268.94 2 L N 358.22 224.28 363.99 222.62 358.22 220.97 358.22 222.62 4 Y V 342.39 222.62 358.22 222.62 2 L N 358.22 195.48 363.99 193.82 358.22 192.17 358.22 193.82 4 Y V 342.39 193.82 358.22 193.82 2 L N 358.22 166.68 363.99 165.02 358.22 163.37 358.22 165.02 4 Y V 342.39 165.02 358.22 165.02 2 L N 358.74 183.72 363.99 186.62 361.08 181.38 359.91 182.55 4 Y V 342.39 165.02 359.91 182.54 2 L N 387.02 281.88 392.79 280.22 387.02 278.57 387.02 280.22 4 Y V 371.19 280.22 387.02 280.22 2 L N 387.02 253.08 392.79 251.42 387.02 249.77 387.02 251.42 4 Y V 371.19 251.42 387.02 251.42 2 L N 389.88 235.07 392.79 229.82 387.54 232.73 388.71 233.9 4 Y V 371.19 251.42 388.71 233.9 2 L N 387.54 270.11 392.79 273.02 389.88 267.77 388.71 268.94 4 Y V 371.19 251.42 388.71 268.94 2 L N 387.02 224.28 392.79 222.62 387.02 220.97 387.02 222.62 4 Y V 371.19 222.62 387.02 222.62 2 L N 387.02 195.48 392.79 193.82 387.02 192.17 387.02 193.82 4 Y V 371.19 193.82 387.02 193.82 2 L N 387.02 166.68 392.79 165.02 387.02 163.37 387.02 165.02 4 Y V 371.19 165.02 387.02 165.02 2 L N 387.54 183.72 392.79 186.62 389.88 181.38 388.71 182.55 4 Y V 371.19 165.02 388.71 182.54 2 L N 271.82 281.88 277.59 280.22 271.82 278.57 271.82 280.22 4 Y V 255.99 280.22 271.82 280.22 2 L N 271.82 253.08 277.59 251.42 271.82 249.77 271.82 251.42 4 Y V 255.99 251.42 271.82 251.42 2 L N 274.68 235.07 277.59 229.82 272.34 232.73 273.51 233.9 4 Y V 255.99 251.42 273.51 233.9 2 L N 272.34 270.11 277.59 273.02 274.68 267.78 273.51 268.94 4 Y V 255.99 251.42 273.51 268.94 2 L N 271.82 224.28 277.59 222.62 271.82 220.97 271.82 222.62 4 Y V 255.99 222.62 271.82 222.62 2 L N 271.82 195.48 277.59 193.82 271.82 192.17 271.82 193.82 4 Y V 255.99 193.82 271.82 193.82 2 L N 271.82 166.68 277.59 165.02 271.82 163.37 271.82 165.02 4 Y V 255.99 165.02 271.82 165.02 2 L N 272.34 183.71 277.59 186.62 274.68 181.38 273.51 182.54 4 Y V 255.99 165.02 273.51 182.54 2 L N 252.39 157.82 266.79 287.42 R 7 X V 385.59 157.82 399.99 287.42 R V 90 450 7.2 7.2 284.79 280.22 G 0 Z 0 X 90 450 7.2 7.2 284.79 280.22 A 7 X 90 450 7.2 7.2 284.79 165.02 G 0 X 90 450 7.2 7.2 284.79 165.02 A 7 X 90 450 7.2 7.2 284.79 193.82 G 0 X 90 450 7.2 7.2 284.79 193.82 A 7 X 90 450 7.2 7.2 284.79 222.62 G 0 X 90 450 7.2 7.2 284.79 222.62 A 7 X 90 450 7.2 7.2 284.79 251.42 G 0 X 90 450 7.2 7.2 284.79 251.42 A 7 X 90 450 7.2 7.2 313.59 280.22 G 0 X 90 450 7.2 7.2 313.59 280.22 A 7 X 90 450 7.2 7.2 313.59 165.02 G 0 X 90 450 7.2 7.2 313.59 165.02 A 7 X 90 450 7.2 7.2 313.59 193.82 G 0 X 90 450 7.2 7.2 313.59 193.82 A 7 X 90 450 7.2 7.2 313.59 222.62 G 0 X 90 450 7.2 7.2 313.59 222.62 A 7 X 90 450 7.2 7.2 313.59 251.42 G 0 X 90 450 7.2 7.2 313.59 251.42 A 7 X 90 450 7.2 7.2 342.39 280.22 G 0 X 90 450 7.2 7.2 342.39 280.22 A 7 X 90 450 7.2 7.2 342.39 165.02 G 0 X 90 450 7.2 7.2 342.39 165.02 A 7 X 90 450 7.2 7.2 342.39 193.82 G 0 X 90 450 7.2 7.2 342.39 193.82 A 7 X 90 450 7.2 7.2 342.39 222.62 G 0 X 90 450 7.2 7.2 342.39 222.62 A 7 X 90 450 7.2 7.2 342.39 251.42 G 0 X 90 450 7.2 7.2 342.39 251.42 A 7 X 90 450 7.2 7.2 371.19 280.22 G 0 X 90 450 7.2 7.2 371.19 280.22 A 7 X 90 450 7.2 7.2 371.19 165.02 G 0 X 90 450 7.2 7.2 371.19 165.02 A 7 X 90 450 7.2 7.2 371.19 193.82 G 0 X 90 450 7.2 7.2 371.19 193.82 A 7 X 90 450 7.2 7.2 371.19 222.62 G 0 X 90 450 7.2 7.2 371.19 222.62 A 7 X 90 450 7.2 7.2 371.19 251.42 G 0 X 90 450 7.2 7.2 371.19 251.42 A 263.19 211.82 389.19 291.02 18 RR N 263.19 154.22 389.19 204.62 18 RR N 2 9 Q (t) 274.71 294.83 T 0 F (0) 291.27 294.83 T (=) 281.7 294.83 T 2 F (t) 332.39 294.83 T 0 F (2) 348.96 294.83 T (=) 339.39 294.83 T 2 F (t) 303.55 294.83 T 0 F (1) 320.12 294.83 T (=) 310.55 294.83 T 2 F (t) 361.23 294.83 T 0 F (3) 377.8 294.83 T (=) 368.23 294.83 T 2 F (X) 248.03 279.41 T 0 F (1) 253.99 274.46 T 2 F (X) 248.03 251.23 T 0 F (2) 253.99 246.28 T 2 F (X) 248.03 223.05 T 0 F (3) 253.99 218.1 T 2 F (X) 248.03 194.88 T 0 F (4) 253.99 189.93 T 2 F (X) 248.03 166.7 T 0 F (5) 253.99 161.75 T 108 63 540 720 C 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "11" 21 %%Page: "10" 22 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (10) 320 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q -0.2 (independent of the other state variables at time) 108 712 P -0.2 ( and earlier) 339 712 P -0.2 (, given) 392.09 712 P -0.2 ( and) 462.58 712 P -0.2 (, so) 520.74 712 P (the conditional probability distribution) 108 698 T 0 10 Q (\0506\051) 528.34 672.79 T 0 12 Q (tells everything we need to know about) 108 645.05 T ( for purposes of prediction. This distribution) 313.93 645.05 T -0.18 (can be speci\336ed by a table as in T) 108 631.05 P -0.18 (able) 267.69 631.05 P -0.18 (1. If every variable is dependent on at most) 290.68 631.05 P -0.18 ( other) 509.7 631.05 P (variables, then the total space needed for the factored form of the joint distribution is) 108 617.05 T ( instead of) 159.73 603.05 T ( for the case of boolean state variables. It is important to) 244.26 603.05 T (emphasize that the arcs in Figure) 108 589.05 T (1) 269.3 589.05 T (1 represent dependencies between state variables and) 274.86 589.05 T 2 F (not) 108 575.05 T 0 F ( reachability in state space as in the case of state transition diagrams. W) 123.34 575.05 T (e refer to the) 466.31 575.05 T (representation shown in Figure) 108 561.05 T (1) 259.98 561.05 T (1 as a) 265.54 561.05 T 2 F (factor) 295.86 561.05 T (ed state transition model) 324.08 561.05 T 0 F (.) 443.09 561.05 T 1 10 Q (Figur) 121.75 536.39 T (e 1) 145.46 536.39 T (1. Dependencies involving state variables) 156.85 536.39 T (T) 121.75 313.46 T (able 1. T) 127.5 313.46 T (abular speci\336cation for) 163.52 313.46 T 0 12 Q (A state variable at time) 108 128.65 T ( can be dependent upon other state variables at the same time.) 227.65 128.65 T (The only requirement is that the graph of dependencies is acyclic. For example in) 108 114.65 T (Figure) 108 100.65 T (1) 142.33 100.65 T (1,) 147.89 100.65 T ( is dependent upon itself at the previous time and) 174.18 100.65 T ( at the same time.) 427.11 100.65 T 0 10 Q (\0507\051) 528.34 75.44 T 333.67 708.53 339 721.21 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 334.17 712 T 0 0 612 792 C 427.35 705.26 462.58 721.21 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 428.08 712 T 0 9 Q (1) 435.86 707.8 T 2 F (t) 444.86 707.8 T 0 F (1) 456.36 707.8 T (\320) 449.61 707.8 T 3 F (,) 440.36 707.8 T 0 0 612 792 C 485.51 705.26 520.74 721.21 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 486.23 712 T 0 9 Q (2) 494.02 707.8 T 2 F (t) 503.02 707.8 T 0 F (1) 514.51 707.8 T (\320) 507.77 707.8 T 3 F (,) 498.52 707.8 T 0 0 612 792 C 259.11 665.05 377.23 682 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 260.49 672.79 T (r) 267.17 672.79 T 2 F (X) 275.16 672.79 T 0 9 Q (1) 282.95 668.59 T 2 F (t) 291.94 668.59 T 3 F (,) 287.45 668.59 T 2 12 Q (X) 299.29 672.79 T 0 9 Q (1) 307.08 668.59 T 2 F (t) 316.08 668.59 T 0 F (1) 327.57 668.59 T (\320) 320.82 668.59 T 3 F (,) 311.58 668.59 T 2 12 Q (X) 338.07 672.79 T 0 9 Q (2) 345.86 668.59 T 2 F (t) 354.86 668.59 T 0 F (1) 366.35 668.59 T (\320) 359.6 668.59 T 3 F (,) 350.36 668.59 T 3 12 Q (,) 332.07 672.79 T 0 F (\050) 271.16 672.79 T (\051) 370.85 672.79 T 296.84 666.34 296.84 680.59 2 L 0.54 H 2 Z N 0 0 612 792 C 299.64 638.33 313.93 654.26 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 300.14 645.05 T 0 9 Q (1) 307.93 640.85 T 0 0 612 792 C 499.7 627.58 509.7 640.26 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (K) 500.19 631.05 T 0 0 612 792 C 108 598.05 159.73 616.24 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (O) 108.19 603.05 T (L) 120.86 603.05 T 0 F (2) 135.38 603.05 T 2 9 Q (K) 141.99 606.65 T 3 12 Q (\050) 130.24 603.05 T (\051) 148.54 603.05 T 0 F (\050) 116.86 603.05 T (\051) 154.54 603.05 T 0 0 612 792 C 212.72 598.05 244.26 614.4 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (O) 213.86 603.05 T 0 F (2) 226.52 603.05 T 2 9 Q (L) 233.12 606.65 T 0 12 Q (\050) 222.53 603.05 T (\051) 238.13 603.05 T 0 0 612 792 C 108 63 540 720 C 262.26 336.13 385.74 533.05 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 345.3 337.1 377.7 515.3 16.2 RR 7 X 0 0 0 1 0 0 0 K V 0.5 H 0 Z 0 X N 296.7 337.1 329.1 515.3 16.2 RR 7 X V 0 X N 347.73 493.1 353.4 495.05 349.62 490.39 348.68 491.74 4 Y V 312.9 466.7 348.68 491.74 2 L 7 X V 2 Z 0 X N 347.63 500.76 353.4 499.1 347.63 497.45 347.63 499.1 4 Y V 312.9 499.1 347.63 499.1 2 L 7 X V 0 X N 347.63 468.36 353.4 466.7 347.63 465.05 347.63 466.7 4 Y V 312.9 466.7 347.63 466.7 2 L 7 X V 0 X N 347.63 435.96 353.4 434.3 347.63 432.65 347.63 434.3 4 Y V 312.9 434.3 347.63 434.3 2 L 7 X V 0 X N 359.85 452.83 361.5 458.6 363.16 452.83 361.5 452.83 4 Y V 361.5 434.3 361.5 452.83 2 L 7 X V 0 X N 347.63 387.36 353.4 385.7 347.63 384.05 347.63 385.7 4 Y V 312.9 385.7 347.63 385.7 2 L 7 X V 0 X N 349.62 362.02 353.4 357.36 347.73 359.31 348.68 360.66 4 Y V 312.9 385.7 348.68 360.66 2 L 7 X V 0 X N 347.63 354.96 353.4 353.3 347.63 351.65 347.63 353.3 4 Y V 312.9 353.3 347.63 353.3 2 L 7 X V 0 X N 7 X 90 450 8.1 8.1 312.9 499.1 G 0 Z 0 X 90 450 8.1 8.1 312.9 499.1 A 7 X 90 450 8.1 8.1 361.5 466.7 G 0 X 90 450 8.1 8.1 361.5 466.7 A 7 X 90 450 8.1 8.1 361.5 499.1 G 0 X 90 450 8.1 8.1 361.5 499.1 A 7 X 90 450 8.1 8.1 312.9 385.7 G 0 X 90 450 8.1 8.1 312.9 385.7 A 7 X 90 450 8.1 8.1 312.9 466.7 G 0 X 90 450 8.1 8.1 312.9 466.7 A 7 X 90 450 8.1 8.1 312.9 434.3 G 0 X 90 450 8.1 8.1 312.9 434.3 A 7 X 90 450 8.1 8.1 361.5 434.3 G 0 X 90 450 8.1 8.1 361.5 434.3 A 7 X 90 450 8.1 8.1 361.5 385.7 G 0 X 90 450 8.1 8.1 361.5 385.7 A 7 X 90 450 8.1 8.1 312.9 353.3 G 0 X 90 450 8.1 8.1 312.9 353.3 A 7 X 90 450 8.1 8.1 361.5 353.3 G 0 X 90 450 8.1 8.1 361.5 353.3 A 2 12 Q (X) 263.26 496.91 T 0 9 Q (1) 271.05 492.71 T 2 12 Q (X) 263.26 465.51 T 0 9 Q (2) 271.05 461.31 T 2 12 Q (X) 263.26 434.11 T 0 9 Q (3) 271.05 429.91 T 2 12 Q (X) 263.26 383.21 T 2 9 Q (L) 271.05 379.01 T 0 F (1) 285.05 379.01 T (\320) 278.3 379.01 T 2 12 Q (X) 263.26 351.81 T 2 9 Q (L) 271.05 347.61 T 2 12 Q (t) 297.64 523.05 T 0 F (0) 319.74 523.05 T (=) 306.97 523.05 T 2 F (t) 349.52 523.05 T 0 F (1) 371.62 523.05 T (=) 358.85 523.05 T 347.95 427.63 353.62 429.58 349.85 424.92 348.9 426.27 4 Y V 313.13 401.23 348.9 426.27 2 L 7 X V 2 Z 0 X N 314.65 411.73 333.03 408.73 308.65 396.35 306.03 399.35 4 Y 7 X V 287.55 417.35 336.15 409.25 332.1 413.3 384.75 405.2 384.75 401.15 340.2 409.25 344.25 405.2 287.55 413.3 8 Y V 287.55 413.3 344.25 405.2 340.2 409.25 384.75 401.15 4 L 0 X N 287.55 417.35 336.15 409.25 332.1 413.3 384.75 405.2 4 L N 108 63 540 720 C 0 0 612 792 C 108 63 540 720 C 207 150.65 441 310.13 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 208 151.63 440 309.13 R 7 X 0 0 0 1 0 0 0 K V 225.58 290.88 225.58 157.38 2 L 0 X V 0.5 H 0 Z N 308.42 291.38 308.42 156.88 2 L V N 364.58 291.38 364.58 156.88 2 L V N 422.42 290.88 422.42 157.38 2 L V N 225.33 291.13 422.67 291.13 2 L V N 225.33 261.13 422.67 261.13 2 L V N 225.33 235.13 422.67 235.13 2 L V N 225.33 209.13 422.67 209.13 2 L V N 225.33 183.13 422.67 183.13 2 L V N 225.33 157.13 422.67 157.13 2 L V N 318.11 267.49 354.89 280.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 9 Q 0 X 0 0 0 1 0 0 0 K (X) 318.55 273.13 T 0 7 Q (1) 324.39 269.78 T (1) 331.39 269.78 T 3 F (,) 327.89 269.78 T 0 9 Q (0) 348.95 273.13 T (=) 339.38 273.13 T 207 150.65 441 310.13 C 375.11 267.49 411.89 280.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 9 Q 0 X 0 0 0 1 0 0 0 K (X) 375.55 273.13 T 0 7 Q (1) 381.39 269.78 T (1) 388.39 269.78 T 3 F (,) 384.89 269.78 T 0 9 Q (1) 405.95 273.13 T (=) 396.38 273.13 T 207 150.65 441 310.13 C 231.58 239.49 301.65 252.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 9 Q 0 X 0 0 0 1 0 0 0 K (X) 231.97 245.13 T 0 7 Q (1) 237.81 241.78 T (0) 244.81 241.78 T 3 F (,) 241.31 241.78 T 0 9 Q (0) 259.37 245.13 T (=) 251.3 245.13 T 2 F (X) 268.37 245.13 T 0 7 Q (2) 274.21 241.78 T (0) 281.2 241.78 T 3 F (,) 277.71 241.78 T 0 9 Q (0) 295.77 245.13 T (=) 287.7 245.13 T 3 F (,) 263.87 245.13 T 207 150.65 441 310.13 C 329.38 240.88 343.62 252.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 9 Q 0 X 0 0 0 1 0 0 0 K (0) 330.38 245.13 T (.) 334.88 245.13 T (6) 337.12 245.13 T 207 150.65 441 310.13 C 386.38 240.88 400.62 252.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 9 Q 0 X 0 0 0 1 0 0 0 K (0) 387.38 245.13 T (.) 391.88 245.13 T (4) 394.12 245.13 T 207 150.65 441 310.13 C 231.58 213.49 301.65 226.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 9 Q 0 X 0 0 0 1 0 0 0 K (X) 231.97 219.13 T 0 7 Q (1) 237.81 215.78 T (0) 244.81 215.78 T 3 F (,) 241.31 215.78 T 0 9 Q (0) 259.37 219.13 T (=) 251.3 219.13 T 2 F (X) 268.37 219.13 T 0 7 Q (2) 274.21 215.78 T (0) 281.2 215.78 T 3 F (,) 277.71 215.78 T 0 9 Q (1) 295.77 219.13 T (=) 287.7 219.13 T 3 F (,) 263.87 219.13 T 207 150.65 441 310.13 C 329.38 214.88 343.62 226.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 9 Q 0 X 0 0 0 1 0 0 0 K (0) 330.38 219.13 T (.) 334.88 219.13 T (7) 337.12 219.13 T 207 150.65 441 310.13 C 386.38 214.88 400.62 226.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 9 Q 0 X 0 0 0 1 0 0 0 K (0) 387.38 219.13 T (.) 391.88 219.13 T (3) 394.12 219.13 T 207 150.65 441 310.13 C 231.58 187.49 301.65 200.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 9 Q 0 X 0 0 0 1 0 0 0 K (X) 231.97 193.13 T 0 7 Q (1) 237.81 189.78 T (0) 244.81 189.78 T 3 F (,) 241.31 189.78 T 0 9 Q (1) 259.37 193.13 T (=) 251.3 193.13 T 2 F (X) 268.37 193.13 T 0 7 Q (2) 274.21 189.78 T (0) 281.2 189.78 T 3 F (,) 277.71 189.78 T 0 9 Q (0) 295.77 193.13 T (=) 287.7 193.13 T 3 F (,) 263.87 193.13 T 207 150.65 441 310.13 C 329.38 188.88 343.62 200.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 9 Q 0 X 0 0 0 1 0 0 0 K (0) 330.38 193.13 T (.) 334.88 193.13 T (1) 337.12 193.13 T 207 150.65 441 310.13 C 386.38 188.88 400.62 200.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 9 Q 0 X 0 0 0 1 0 0 0 K (0) 387.38 193.13 T (.) 391.88 193.13 T (9) 394.12 193.13 T 207 150.65 441 310.13 C 231.58 161.49 301.65 174.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 9 Q 0 X 0 0 0 1 0 0 0 K (X) 231.97 167.13 T 0 7 Q (1) 237.81 163.78 T (0) 244.81 163.78 T 3 F (,) 241.31 163.78 T 0 9 Q (1) 259.37 167.13 T (=) 251.3 167.13 T 2 F (X) 268.37 167.13 T 0 7 Q (2) 274.21 163.78 T (0) 281.2 163.78 T 3 F (,) 277.71 163.78 T 0 9 Q (1) 295.77 167.13 T (=) 287.7 167.13 T 3 F (,) 263.87 167.13 T 207 150.65 441 310.13 C 329.38 162.88 343.62 174.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 9 Q 0 X 0 0 0 1 0 0 0 K (1) 330.38 167.13 T (.) 334.88 167.13 T (0) 337.12 167.13 T 207 150.65 441 310.13 C 386.38 162.88 400.62 174.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 9 Q 0 X 0 0 0 1 0 0 0 K (0) 387.38 167.13 T (.) 391.88 167.13 T (0) 394.12 167.13 T 207 150.65 441 310.13 C 108 63 540 720 C 0 0 612 792 C 264.92 305.74 361.15 322.67 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 265.85 313.46 T (r) 272.52 313.46 T 2 F (X) 280.52 313.46 T 0 9 Q (1) 288.3 309.27 T (1) 297.3 309.27 T 3 F (,) 292.8 309.27 T 2 12 Q (X) 306.65 313.46 T 0 9 Q (1) 314.43 309.27 T (0) 323.43 309.27 T 3 F (,) 318.93 309.27 T 2 12 Q (X) 333.93 313.46 T 0 9 Q (2) 341.72 309.27 T (0) 350.71 309.27 T 3 F (,) 346.22 309.27 T 3 12 Q (,) 327.93 313.46 T 0 F (\050) 276.52 313.46 T (\051) 355.21 313.46 T 304.2 307.02 304.2 321.27 2 L 0.54 H 2 Z N 0 0 612 792 C 222.31 125.18 227.65 137.86 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 222.81 128.65 T 0 0 612 792 C 159.89 93.93 174.18 109.86 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 160.39 100.65 T 0 9 Q (2) 168.18 96.45 T 0 0 612 792 C 412.82 93.93 427.11 109.86 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 413.33 100.65 T 0 9 Q (3) 421.11 96.45 T 0 0 612 792 C 266.08 67.7 370.26 84.65 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 267.24 75.44 T (r) 273.91 75.44 T 2 F (X) 281.9 75.44 T 0 9 Q (2) 289.69 71.24 T 2 F (t) 298.69 71.24 T 3 F (,) 294.19 71.24 T 2 12 Q (X) 306.04 75.44 T 0 9 Q (3) 313.83 71.24 T 2 F (t) 322.82 71.24 T 3 F (,) 318.33 71.24 T 2 12 Q (X) 331.32 75.44 T 0 9 Q (2) 339.11 71.24 T 2 F (t) 348.11 71.24 T 0 F (1) 359.6 71.24 T (\320) 352.86 71.24 T 3 F (,) 343.61 71.24 T 3 12 Q (,) 325.33 75.44 T 0 F (\050) 277.91 75.44 T (\051) 364.1 75.44 T 303.59 68.99 303.59 83.24 2 L 0.54 H 2 Z N 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "10" 22 %%Page: "9" 23 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (9) 322 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (de\336ne a Markov process and the transition probabilities are not dependent on the starting) 108 712 T (distribution. Figure) 108 698 T (9 shows a lumpable process of only mar) 203.68 698 T (ginal interest as the underly-) 396.43 698 T (ing Markov chain is reducible into two completely independent smaller chains.) 108 684 T 1 10 Q (Figur) 121.75 659.33 T (e 9. Lumpable, r) 145.46 659.33 T (educible, and aperiodic Markov chain) 215.55 659.33 T 0 12 Q -0.21 (Figure) 108 547.67 P -0.21 (10 shows a much more interesting example of a lumpable process in which the par-) 142.33 547.67 P -0.39 (tition) 108 533.67 P -0.39 ( represents an abstraction of the four) 232.63 533.67 P -0.39 (-state process. Now we turn) 405.33 533.67 P (our attention to additional methods for exploiting structure in dynamical systems.) 108 519.67 T 1 10 Q (Figur) 121.75 495 T (e 10. Lumpable, irr) 145.46 495 T (educible, and aperiodic Markov chain) 227.77 495 T 1 16 Q (5 Exploiting Structur) 108 373 T (e in State and Phase Space) 257.5 373 T 0 12 Q (Instead of a single random variable describing the state at time) 108 345.67 T (, we factor) 416.95 345.67 T ( into) 489.69 345.67 T (dimensions) 108 331.67 T ( so that the state space is de\336ned by the product) 273.76 331.67 T 0 10 Q (\0504\051) 528.34 292.41 T 0 12 Q (Let) 108 253.74 T ( be a random variable representing the) 146.29 253.74 T 2 F (state variable) 333.24 253.74 T 0 F ( with values from) 414.19 253.74 T ( at) 522.3 253.74 T (time) 108 239.74 T (. These state variables correspond to \337uents, conditions, or propositions in arti\336cial) 137.67 239.74 T (intelligence representations. For a Markov chain, the state transition probabilities would) 108 225.74 T (be speci\336ed as the following joint probability distribution over) 108 211.74 T (.) 519.75 211.74 T 0 10 Q (\0505\051) 528.34 185.74 T 0 12 Q (Generally) 108 156.09 T (, only a small subset of the state variables at time) 154.54 156.09 T ( are necessary to specify the) 398.51 156.09 T (probabilities for a given state variable at time) 108 142.09 T (. Figure) 352.55 142.09 T (1) 392.88 142.09 T (1 graphically displays the) 398.44 142.09 T (dependencies among state variables for a particular Markov chain. If) 108 128.09 T ( for) 515.67 128.09 T (, we have) 151.18 114.09 T ( with a joint distribution for the transition probabilities of) 252.59 114.09 T (comparable size. Dependencies allow us to factor the joint probability distribution for the) 108 100.09 T -0.42 (transition probabilities into a set of smaller distributions. For example in Figure) 108 86.09 P -0.42 (1) 488.4 86.09 P -0.42 (1,) 493.96 86.09 P -0.42 ( is) 526.83 86.09 P 108 63 540 720 C 120 571.67 528 656 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 176.4 607.34 181.33 600.96 173.78 603.75 175.09 605.54 4 Y 0 X 0 0 0 1 0 0 0 K V 0.5 H 2 Z 55.51 180 20.25 13.5 163.62 594.42 A 151.64 622.12 146.75 628.79 154.43 625.73 153.04 623.93 4 Y V 234.51 360 20.25 13.5 164.79 634.92 A 0 12 Q (0) 273.68 634.63 T (.) 279.68 634.63 T (2) 282.68 634.63 T (0) 293.47 634.63 T (.) 299.47 634.63 T (8) 302.47 634.63 T (0) 317.77 634.63 T (0) 337.57 634.63 T (0) 273.68 617.84 T (.) 279.68 617.84 T (7) 282.68 617.84 T (0) 293.47 617.84 T (.) 299.47 617.84 T (3) 302.47 617.84 T (0) 317.77 617.84 T (0) 337.57 617.84 T (0) 278.18 601.04 T (0) 297.97 601.04 T (0) 313.27 601.04 T (.) 319.27 601.04 T (5) 322.27 601.04 T (0) 333.07 601.04 T (.) 339.07 601.04 T (5) 342.07 601.04 T (0) 278.18 584.24 T (0) 297.97 584.24 T (0) 313.27 584.24 T (.) 319.27 584.24 T (2) 322.27 584.24 T (0) 333.07 584.24 T (.) 339.07 584.24 T (8) 342.07 584.24 T 274.34 581.24 270.74 581.24 270.74 648.43 3 L 0.54 H N 270.74 648.43 274.34 648.43 2 L N 346.33 581.24 349.93 581.24 349.93 648.43 3 L N 349.93 648.43 346.33 648.43 2 L N (1) 216.91 616.52 T (2) 228.91 616.52 T (3) 240.91 616.52 T (4) 252.9 616.52 T 3 F (,) 222.91 616.52 T (,) 234.91 616.52 T (,) 246.91 616.52 T ({) 210 616.52 T (}) 259.46 616.52 T 0 F (1) 361.89 618.19 T (2) 373.89 618.19 T 3 F (,) 367.89 618.19 T ({) 354.98 618.19 T (}) 380.44 618.19 T 0 F (3) 447.38 619.02 T (4) 459.37 619.02 T 3 F (,) 453.38 619.02 T ({) 440.46 619.02 T (}) 465.92 619.02 T 0 F (0) 397.28 623.01 T (.) 403.28 623.01 T (2) 406.28 623.01 T (0) 417.08 623.01 T (.) 423.08 623.01 T (8) 426.08 623.01 T (0) 397.28 606.21 T (.) 403.28 606.21 T (7) 406.28 606.21 T (0) 417.08 606.21 T (.) 423.08 606.21 T (3) 426.08 606.21 T 397.94 603.21 394.35 603.21 394.35 636.8 3 L N 394.35 636.8 397.94 636.8 2 L N 430.34 603.21 433.94 603.21 433.94 636.8 3 L N 433.94 636.8 430.34 636.8 2 L N (0) 481.5 623.01 T (.) 487.5 623.01 T (5) 490.5 623.01 T (0) 501.3 623.01 T (.) 507.3 623.01 T (5) 510.3 623.01 T (0) 481.5 606.21 T (.) 487.5 606.21 T (2) 490.5 606.21 T (0) 501.3 606.21 T (.) 507.3 606.21 T (8) 510.3 606.21 T 482.16 603.21 478.56 603.21 478.56 636.8 3 L N 478.56 636.8 482.16 636.8 2 L N 514.56 603.21 518.16 603.21 518.16 636.8 3 L N 518.16 636.8 514.56 636.8 2 L N 0 10 Q (1) 137.94 634.29 T 176.4 647.83 181.33 641.46 173.78 644.25 175.09 646.04 4 Y V 0.5 H 55.51 180 20.25 13.5 163.62 634.92 A 150.47 581.62 145.58 588.29 153.26 585.23 151.87 583.43 4 Y V 234.51 360 20.25 13.5 163.62 594.42 A 7 X 90 450 6.75 6.75 143.38 634.92 G 0 Z 0 X 90 450 6.75 6.75 143.38 634.92 A 7 X 90 450 6.75 6.75 143.38 594.42 G 0 X 90 450 6.75 6.75 143.38 594.42 A 7 X 90 450 6.75 6.75 183.88 634.92 G 0 X 90 450 6.75 6.75 183.88 634.92 A 7 X 90 450 6.75 6.94 183.88 594.23 G 0 X 90 450 6.75 6.94 183.88 594.23 A 187.06 579.77 183.89 587.2 190.52 582.58 188.79 581.17 4 Y V 2 Z 254.27 450 6.75 6.75 190.62 587.67 A 131.55 641.21 136.62 634.92 128.99 637.57 130.27 639.39 4 Y V 0 199.73 6.75 6.75 136.62 641.67 A (1) 140.35 633.54 T (2) 180.85 633.54 T (3) 140.35 593.04 T (4) 180.85 593.04 T 131.55 600.71 136.62 594.42 128.99 597.07 130.27 598.89 4 Y V 0 199.73 6.75 6.75 136.62 601.17 A 187.05 620.27 183.87 627.7 190.5 623.08 188.78 621.67 4 Y V 254.27 450 6.75 6.75 190.61 628.17 A 123.71 577.42 203.29 611.17 16.88 RR 0 Z N 122.54 618.17 203.46 651.92 16.88 RR N 108 63 540 720 C 0 0 612 792 C 135.95 528.67 232.63 543.67 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (1) 154.78 533.67 T (2) 166.78 533.67 T 3 F (,) 160.78 533.67 T ({) 147.87 533.67 T (}) 173.33 533.67 T 0 F (3) 196.01 533.67 T (4) 208 533.67 T 3 F (,) 202.01 533.67 T ({) 189.09 533.67 T (}) 214.56 533.67 T (,) 181.09 533.67 T ({) 138.95 533.67 T (}) 222.87 533.67 T 0 0 612 792 C 108 63 540 720 C 129.42 407.67 518.58 491.67 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 179.24 433.24 186.7 429.92 178.62 428.78 178.93 431.01 4 Y 0 X 0 0 0 1 0 0 0 K V 0.5 H 2 Z 180 256.68 33.75 40.5 186.7 470.42 A 178.87 470.81 186.95 469.67 179.49 466.35 179.18 468.58 4 Y V 103.32 180 33.75 40.5 186.95 429.17 A 167.54 430.28 159.45 431.42 166.92 434.74 167.23 432.51 4 Y V 283.32 360 33.75 40.5 159.45 471.92 A 180.66 463.32 187.32 468.21 184.26 460.53 182.46 461.93 4 Y V 144.51 270 13.5 20.25 193.45 450.17 A 0 12 Q (0) 285.65 467.87 T (.) 291.65 467.87 T (1) 294.65 467.87 T (0) 305.45 467.87 T (.) 311.45 467.87 T (8) 314.45 467.87 T (0) 329.74 467.87 T (0) 345.04 467.87 T (.) 351.04 467.87 T (1) 354.04 467.87 T (0) 285.65 451.08 T (.) 291.65 451.08 T (7) 294.65 451.08 T (0) 305.45 451.08 T (.) 311.45 451.08 T (2) 314.45 451.08 T (0) 325.24 451.08 T (.) 331.24 451.08 T (1) 334.24 451.08 T (0) 349.54 451.08 T (0) 290.15 434.28 T (0) 305.45 434.28 T (.) 311.45 434.28 T (2) 314.45 434.28 T (0) 325.24 434.28 T (.) 331.24 434.28 T (4) 334.24 434.28 T (0) 345.04 434.28 T (.) 351.04 434.28 T (4) 354.04 434.28 T (0) 290.15 417.48 T (0) 305.45 417.48 T (.) 311.45 417.48 T (2) 314.45 417.48 T (0) 325.24 417.48 T (.) 331.24 417.48 T (2) 334.24 417.48 T (0) 345.04 417.48 T (.) 351.04 417.48 T (6) 354.04 417.48 T 286.31 414.48 282.71 414.48 282.71 481.67 3 L 0.54 H N 282.71 481.67 286.31 481.67 2 L N 358.3 414.48 361.9 414.48 361.9 481.67 3 L N 361.9 481.67 358.3 481.67 2 L N (0) 473.07 454.11 T (.) 479.07 454.11 T (9) 482.07 454.11 T (0) 492.86 454.11 T (.) 498.86 454.11 T (1) 501.86 454.11 T (0) 473.07 437.31 T (.) 479.07 437.31 T (2) 482.07 437.31 T (0) 492.86 437.31 T (.) 498.86 437.31 T (8) 501.86 437.31 T 473.72 434.31 470.13 434.31 470.13 467.9 3 L N 470.13 467.9 473.72 467.9 2 L N 506.12 434.31 509.72 434.31 509.72 467.9 3 L N 509.72 467.9 506.12 467.9 2 L N (1) 387.98 449.22 T (2) 399.98 449.22 T 3 F (,) 393.98 449.22 T ({) 381.07 449.22 T (}) 406.53 449.22 T 0 F (3) 429.2 449.22 T (4) 441.2 449.22 T 3 F (,) 435.2 449.22 T ({) 422.29 449.22 T (}) 447.75 449.22 T (,) 414.29 449.22 T ({) 372.15 449.22 T (}) 456.07 449.22 T 187.48 443.08 192.41 436.71 184.86 439.51 186.17 441.3 4 Y V 0.5 H 55.51 180 20.25 13.5 174.7 430.17 A 159.47 457.12 154.58 463.79 162.26 460.73 160.86 458.93 4 Y V 234.51 360 20.25 13.5 172.62 469.92 A 0 10 Q (1) 147.52 469.79 T 185.98 483.33 190.91 476.96 183.36 479.75 184.67 481.55 4 Y V 55.51 180 20.25 13.5 173.2 470.42 A 160.05 417.13 155.16 423.79 162.84 420.73 161.45 418.93 4 Y V 234.51 360 20.25 13.5 173.2 429.92 A 7 X 90 450 6.75 6.75 152.95 470.42 G 0 Z 0 X 90 450 6.75 6.75 152.95 470.42 A 7 X 90 450 6.75 6.75 152.95 429.92 G 0 X 90 450 6.75 6.75 152.95 429.92 A 7 X 90 450 6.75 6.75 193.45 470.42 G 0 X 90 450 6.75 6.75 193.45 470.42 A 7 X 90 450 6.75 6.94 193.45 429.73 G 0 X 90 450 6.75 6.94 193.45 429.73 A 196.64 415.27 193.47 422.7 200.1 418.08 198.37 416.67 4 Y V 2 Z 254.27 450 6.75 6.75 200.2 423.17 A 141.13 476.71 146.2 470.42 138.57 473.07 139.85 474.89 4 Y V 0 199.73 6.75 6.75 146.2 477.17 A (1) 149.93 469.04 T (2) 190.43 469.04 T (3) 149.93 428.54 T (4) 190.43 428.54 T 141.13 436.21 146.2 429.92 138.57 432.57 139.85 434.39 4 Y V 0 199.73 6.75 6.75 146.2 436.67 A 196.63 455.77 193.45 463.2 200.08 458.58 198.35 457.17 4 Y V 254.27 450 6.75 6.75 200.19 463.67 A 131.41 412.38 213.66 446.13 16.88 RR 0 Z N 132.08 452.88 212.99 486.63 16.88 RR N 0 12 Q (1) 227.21 447.55 T (2) 239.21 447.55 T (3) 251.21 447.55 T (4) 263.21 447.55 T 3 F (,) 233.21 447.55 T (,) 245.21 447.55 T (,) 257.21 447.55 T ({) 220.3 447.55 T (}) 269.76 447.55 T 108 63 540 720 C 0 0 612 792 C 411.61 342.2 416.95 354.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 412.11 345.67 T 0 0 612 792 C 470.92 337.22 489.69 355.67 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 472.22 345.67 T 2 9 Q (X) 481.89 341.47 T 0 0 612 792 C 514.36 342.2 523.04 354.88 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (L) 514.86 345.67 T 0 0 612 792 C 165.67 314.61 273.76 348.47 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 176.58 331.67 T 2 9 Q (X) 186.26 327.47 T 0 6 Q (1) 192.1 324.92 T 3 12 Q (W) 201.09 331.67 T 2 9 Q (X) 210.77 327.47 T 0 6 Q (2) 216.61 324.92 T 3 12 Q (\274) 225.6 331.67 T (W) 243.6 331.67 T 2 9 Q (X) 253.28 327.47 T 2 6 Q (L) 259.12 324.92 T 3 12 Q (,) 195.1 331.67 T (,) 219.61 331.67 T (,) 237.6 331.67 T ({) 169.67 331.67 T (}) 263.01 331.67 T 0 0 612 792 C 280.88 273.74 355.46 315.67 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 282.48 292.41 T 2 9 Q (X) 292.15 288.21 T 3 12 Q (W) 335.68 292.41 T 2 9 Q (X) 345.35 288.21 T 2 6 Q (i) 351.19 285.66 T 2 9 Q (i) 316.41 276.98 T 0 F (1) 329.98 276.98 T (=) 321.91 276.98 T 2 F (L) 322.94 307.92 T 3 18 Q (\325) 318.04 289.15 T 0 12 Q (=) 303.65 292.41 T 0 0 612 792 C 127 247.09 146.29 262.94 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 127.5 253.74 T 2 9 Q (i) 135.29 249.54 T (t) 142.29 249.54 T 3 F (,) 137.79 249.54 T 0 0 612 792 C 401.9 247.09 414.19 262.94 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 402.4 253.74 T 2 9 Q (i) 410.19 249.54 T 0 0 612 792 C 501.52 243.49 522.3 263.74 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 502.82 253.74 T 2 9 Q (X) 512.49 249.54 T 2 6 Q (i) 518.33 246.99 T 0 0 612 792 C 132.34 236.26 137.67 248.94 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 132.84 239.74 T 0 0 612 792 C 411.66 194.68 519.75 228.54 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 422.57 211.74 T 2 9 Q (X) 432.24 207.54 T 0 6 Q (1) 438.08 204.99 T 3 12 Q (W) 447.08 211.74 T 2 9 Q (X) 456.75 207.54 T 0 6 Q (2) 462.6 204.99 T 3 12 Q (\274) 471.59 211.74 T (W) 489.59 211.74 T 2 9 Q (X) 499.26 207.54 T 2 6 Q (L) 505.11 204.99 T 3 12 Q (,) 441.08 211.74 T (,) 465.6 211.74 T (,) 483.59 211.74 T ({) 415.66 211.74 T (}) 508.99 211.74 T 0 0 612 792 C 226.28 176.09 410.06 195.74 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 229.35 185.74 T (r) 236.02 185.74 T 2 F (X) 244.01 185.74 T 0 9 Q (1) 251.8 181.54 T 2 F (t) 260.8 181.54 T 3 F (,) 256.3 181.54 T 3 12 Q (\274) 269.3 185.74 T 2 F (X) 287.3 185.74 T 2 9 Q (L) 295.08 181.54 T (t) 304.58 181.54 T 3 F (,) 300.09 181.54 T 3 12 Q (,) 263.3 185.74 T (,) 281.3 185.74 T 2 F (X) 311.93 185.74 T 0 9 Q (1) 319.72 181.54 T 2 F (t) 328.72 181.54 T 0 F (1) 340.21 181.54 T (\320) 333.47 181.54 T 3 F (,) 324.22 181.54 T 3 12 Q (\274) 350.71 185.74 T 2 F (X) 368.71 185.74 T 2 9 Q (L) 376.5 181.54 T (t) 386 181.54 T 0 F (1) 397.5 181.54 T (\320) 390.75 181.54 T 3 F (,) 381.5 181.54 T 3 12 Q (,) 344.71 185.74 T (,) 362.71 185.74 T 0 F (\050) 240.02 185.74 T (\051) 402 185.74 T 309.49 179.29 309.49 193.54 2 L 0.54 H 2 Z N 0 0 612 792 C 393.18 152.61 398.51 165.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 393.68 156.09 T 0 0 612 792 C 328.63 138.48 352.55 151.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 329.04 142.09 T 0 F (1) 345.14 142.09 T (+) 335.38 142.09 T 0 0 612 792 C 440.9 117.84 515.67 138.09 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 442.2 128.09 T 2 9 Q (X) 451.88 123.89 T 2 6 Q (i) 457.72 121.34 T 0 12 Q (0) 487.06 128.09 T (1) 499.06 128.09 T 3 F (,) 493.06 128.09 T ({) 480.15 128.09 T (}) 505.61 128.09 T 0 F (=) 465.38 128.09 T 0 0 612 792 C 108 110.48 151.18 123.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (1) 108.5 114.09 T 2 F (i) 127.08 114.09 T (L) 143.01 114.09 T 3 F (\243) 117.5 114.09 T (\243) 133.42 114.09 T 0 0 612 792 C 199.82 106.57 252.59 127.28 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 202.94 114.09 T 2 9 Q (X) 212.61 109.89 T 0 12 Q (2) 239.81 114.09 T 2 9 Q (L) 246.41 117.69 T 0 12 Q (=) 227.04 114.09 T 201.2 108.84 201.2 123.09 2 L 0.54 H 2 Z N 219.31 108.84 219.31 123.09 2 L N 0 0 612 792 C 505.54 79.35 526.83 95.29 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 506.04 86.09 T 0 9 Q (1) 513.83 81.89 T 2 F (t) 522.83 81.89 T 3 F (,) 518.33 81.89 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "9" 23 %%Page: "8" 24 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (8) 322 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 1 10 Q (Figur) 121.75 713.33 T (e 6. Aperiodic and irr) 145.46 713.33 T (educible Markov chain) 237.21 713.33 T 0 12 Q (Figure) 108 605.33 T (7 shows a Markov chain with a proper closed subset) 142.33 605.33 T (. Once the system) 434.52 605.33 T (enters state) 108 591.33 T ( it never returns to either state) 172.65 591.33 T ( or) 326.96 591.33 T (.) 350.95 591.33 T 1 10 Q (Figur) 121.75 566.67 T (e 7. Aperiodic and r) 145.46 566.67 T (educible Markov chain) 229.99 566.67 T 0 12 Q (Figure) 108 458.67 T (8 shows a Markov chain such that state) 142.33 458.67 T ( has a period of three.) 341.64 458.67 T 1 10 Q (Figur) 121.75 434 T (e 8. Periodic and irr) 145.46 434 T (educible Markov chain) 230.54 434 T 0 12 Q -0.42 (W) 108 326.83 P -0.42 (e are concerned in this paper with dynamical systems that can be modeled as \336nite-state,) 118.37 326.83 P -0.4 (discrete-time, er) 108 312.83 P -0.4 (godic, stationary Markov chains. While such systems may seem relatively) 185.36 312.83 P (simple, there is still considerable reason for computational concern given that the size of) 108 298.83 T (the state space, while \336nite, can still be, and often is in practice, quite lar) 108 284.83 T (ge. Arti\336cial) 456.4 284.83 T (intelligence researchers have typically been concerned with representations that do not) 108 270.83 T (require explicitly enumerating the set of all states for a given system. Before turning to) 108 256.83 T (techniques for supporting such representations, we consider one additional property of) 108 242.83 T (Markov chains that allows for a powerful sort of abstraction.) 108 228.83 T (A stochastic process is) 108 202.83 T 2 F (lumpable) 219.98 202.83 T 0 F ( with respect to a partition) 264.65 202.83 T ( \050where) 482.4 202.83 T ( and) 188.31 188.83 T (\051 if the distributions) 289.84 188.83 T 108 63 540 720 C 204.83 629.33 443.17 710 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (0) 361.32 686.41 T (.) 367.32 686.41 T (7) 370.32 686.41 T (0) 381.12 686.41 T (.) 387.12 686.41 T (3) 390.12 686.41 T (0) 405.42 686.41 T (0) 425.21 686.41 T (0) 365.82 669.61 T (0) 385.62 669.61 T (0) 400.92 669.61 T (.) 406.92 669.61 T (8) 409.92 669.61 T (0) 420.71 669.61 T (.) 426.71 669.61 T (2) 429.71 669.61 T (0) 361.32 652.82 T (.) 367.32 652.82 T (1) 370.32 652.82 T (0) 381.12 652.82 T (.) 387.12 652.82 T (9) 390.12 652.82 T (0) 405.42 652.82 T (0) 425.21 652.82 T (0) 365.82 636.02 T (0) 385.62 636.02 T (0) 400.92 636.02 T (.) 406.92 636.02 T (6) 409.92 636.02 T (0) 420.71 636.02 T (.) 426.71 636.02 T (4) 429.71 636.02 T 361.98 633.02 358.38 633.02 358.38 700.21 3 L 0.54 H 2 Z N 358.38 700.21 361.98 700.21 2 L N 433.98 633.02 437.57 633.02 437.57 700.21 3 L N 437.57 700.21 433.98 700.21 2 L N (1) 299.55 665.33 T (2) 311.55 665.33 T (3) 323.55 665.33 T (4) 335.55 665.33 T 3 F (,) 305.55 665.33 T (,) 317.55 665.33 T (,) 329.55 665.33 T ({) 292.64 665.33 T (}) 342.1 665.33 T 240.42 646.77 232.33 647.91 239.79 651.23 240.1 649 4 Y V 0.5 H 283.32 360 33.75 40.5 232.33 688.41 A 251.24 689.55 259.33 688.41 251.86 685.09 251.55 687.32 4 Y V 103.32 180 33.75 40.5 259.33 647.91 A 0 10 Q (1) 220.14 687.79 T 258.61 701.33 263.54 694.96 255.98 697.75 257.3 699.54 4 Y V 55.51 180 20.25 13.5 245.83 688.41 A 279.16 655.69 272.83 650.62 275.51 658.28 277.33 656.99 4 Y V 326.51 450 13.5 20.25 266.08 668.16 A 232.68 635.12 227.79 641.78 235.46 638.72 234.07 636.92 4 Y V 234.51 360 20.25 13.5 245.83 647.91 A 212.78 681.31 219.45 686.2 216.39 678.52 214.59 679.92 4 Y V 144.51 270 13.5 20.25 225.58 668.16 A 7 X 90 450 6.75 6.75 225.58 688.41 G 0 Z 0 X 90 450 6.75 6.75 225.58 688.41 A 7 X 90 450 6.75 6.75 225.58 647.91 G 0 X 90 450 6.75 6.75 225.58 647.91 A 7 X 90 450 6.75 6.75 266.08 688.41 G 0 X 90 450 6.75 6.75 266.08 688.41 A 7 X 90 450 6.75 6.94 266.08 647.72 G 0 X 90 450 6.75 6.94 266.08 647.72 A 269.27 633.26 266.09 640.69 272.72 636.07 271 634.67 4 Y V 2 Z 254.27 450 6.75 6.75 272.83 641.16 A 213.76 694.7 218.83 688.41 211.19 691.06 212.47 692.88 4 Y V 0 199.73 6.75 6.75 218.83 695.16 A (1) 222.55 687.04 T (2) 263.05 687.04 T (3) 222.55 646.54 T (4) 263.05 646.54 T 108 63 540 720 C 0 0 612 792 C 396.29 600.33 434.51 615.33 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (3) 406.2 605.33 T (4) 418.2 605.33 T 3 F (,) 412.2 605.33 T ({) 399.29 605.33 T (}) 424.75 605.33 T 0 0 612 792 C 164.65 587.73 172.65 600.52 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (4) 165.15 591.33 T 0 0 612 792 C 318.96 587.73 326.96 600.52 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (1) 319.46 591.33 T 0 0 612 792 C 342.95 587.73 350.95 600.52 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (2) 343.45 591.33 T 0 0 612 792 C 108 63 540 720 C 201.92 482.67 446.08 563.33 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 278.25 509.53 271.92 504.46 274.6 512.11 276.42 510.82 4 Y 0 X 0 0 0 1 0 0 0 K V 0.5 H 2 Z 326.51 450 13.5 20.25 265.17 521.99 A 257.7 514 262.63 507.63 255.07 510.42 256.38 512.21 4 Y V 55.51 180 20.25 13.5 244.92 501.08 A 232.93 528.79 228.04 535.45 235.72 532.4 234.33 530.59 4 Y V 234.51 360 20.25 13.5 246.08 541.58 A 0 10 Q (1) 219.23 540.96 T 257.7 554.5 262.63 548.13 255.07 550.92 256.38 552.71 4 Y V 55.51 180 20.25 13.5 244.92 541.58 A 231.76 488.29 226.87 494.96 234.55 491.89 233.16 490.09 4 Y V 234.51 360 20.25 13.5 244.92 501.08 A 7 X 90 450 6.75 6.75 224.67 541.58 G 0 Z 0 X 90 450 6.75 6.75 224.67 541.58 A 7 X 90 450 6.75 6.75 224.67 501.08 G 0 X 90 450 6.75 6.75 224.67 501.08 A 7 X 90 450 6.75 6.75 265.17 541.58 G 0 X 90 450 6.75 6.75 265.17 541.58 A 7 X 90 450 6.75 6.94 265.17 500.9 G 0 X 90 450 6.75 6.94 265.17 500.9 A 268.36 486.43 265.18 493.86 271.81 489.24 270.08 487.84 4 Y V 2 Z 254.27 450 6.75 6.75 271.92 494.33 A 222.02 555.97 224.67 548.33 218.37 553.41 220.2 554.69 4 Y V 70.27 270 6.75 6.75 217.92 548.33 A (1) 221.64 540.21 T (2) 262.14 540.21 T (3) 221.64 499.71 T (4) 262.14 499.71 T 222.02 515.47 224.67 507.83 218.37 512.91 220.2 514.19 4 Y V 70.27 270 6.75 6.75 217.92 507.83 A 279.82 544.78 272.39 541.6 277.01 548.23 278.41 546.5 4 Y V 344.27 540 6.75 6.75 271.92 548.33 A 0 12 Q (1) 296.97 515.84 T (2) 308.97 515.84 T (3) 320.97 515.84 T (4) 332.96 515.84 T 3 F (,) 302.97 515.84 T (,) 314.97 515.84 T (,) 326.97 515.84 T ({) 290.06 515.84 T (}) 339.52 515.84 T 0 F (0) 363.06 543.47 T (.) 369.06 543.47 T (2) 372.06 543.47 T (0) 382.86 543.47 T (.) 388.86 543.47 T (8) 391.86 543.47 T (0) 407.15 543.47 T (0) 426.95 543.47 T (0) 363.06 526.67 T (.) 369.06 526.67 T (7) 372.06 526.67 T (0) 382.86 526.67 T (.) 388.86 526.67 T (2) 391.86 526.67 T (0) 407.15 526.67 T (0) 422.45 526.67 T (.) 428.45 526.67 T (1) 431.45 526.67 T (0) 367.56 509.88 T (0) 387.36 509.88 T (0) 402.65 509.88 T (.) 408.65 509.88 T (5) 411.65 509.88 T (0) 422.45 509.88 T (.) 428.45 509.88 T (5) 431.45 509.88 T (0) 367.56 493.08 T (0) 387.36 493.08 T (0) 402.65 493.08 T (.) 408.65 493.08 T (2) 411.65 493.08 T (0) 422.45 493.08 T (.) 428.45 493.08 T (8) 431.45 493.08 T 363.72 490.08 360.12 490.08 360.12 557.27 3 L 0.54 H N 360.12 557.27 363.72 557.27 2 L N 435.71 490.08 439.31 490.08 439.31 557.27 3 L N 439.31 557.27 435.71 557.27 2 L N 108 63 540 720 C 0 0 612 792 C 333.64 455.06 341.64 467.85 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (1) 334.14 458.67 T 0 0 612 792 C 108 63 540 720 C 201.92 350.83 446.08 430.67 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 238.22 405.85 230.75 409.17 238.84 410.31 238.53 408.08 4 Y 0 X 0 0 0 1 0 0 0 K V 0.5 H 2 Z 10 76.68 33.75 40.5 230.75 368.67 A 277.83 376.45 271.5 371.38 274.19 379.04 276.01 377.74 4 Y V 326.51 450 13.5 20.25 264.75 388.92 A 213.32 380.04 216 372.38 209.67 377.45 211.5 378.74 4 Y V 115 213.49 13.5 20.25 222.75 389.92 A 0 12 Q (0) 370.23 408.79 T (0) 385.53 408.79 T (.) 391.53 408.79 T (8) 394.53 408.79 T (0) 405.32 408.79 T (.) 411.32 408.79 T (2) 414.32 408.79 T (0) 429.62 408.79 T (0) 370.23 391.99 T (0) 390.03 391.99 T (0) 409.82 391.99 T (1) 425.12 391.99 T (.) 431.12 391.99 T (0) 434.12 391.99 T (0) 370.23 375.19 T (0) 390.03 375.19 T (0) 409.82 375.19 T (1) 425.12 375.19 T (.) 431.12 375.19 T (0) 434.12 375.19 T (1) 365.73 358.39 T (.) 371.73 358.39 T (0) 374.73 358.39 T (0) 390.03 358.39 T (0) 409.82 358.39 T (0) 429.62 358.39 T 366.39 355.39 362.79 355.39 362.79 422.58 3 L 0.54 H N 362.79 422.58 366.39 422.58 2 L N 438.38 355.39 441.98 355.39 441.98 422.58 3 L N 441.98 422.58 438.38 422.58 2 L N 0 10 Q (1) 217.98 408.54 T 256.45 422.08 261.38 415.71 253.82 418.5 255.13 420.29 4 Y V 0.5 H 55.51 180 20.25 13.5 243.67 409.17 A 254.24 358.98 261.92 362.04 257.02 355.37 255.63 357.18 4 Y V 209 305.49 20.25 13.5 243.87 368.17 A 7 X 90 450 6.75 6.75 223.42 409.17 G 0 Z 0 X 90 450 6.75 6.75 223.42 409.17 A 7 X 90 450 6.75 6.75 223.42 368.67 G 0 X 90 450 6.75 6.75 223.42 368.67 A 7 X 90 450 6.75 6.75 263.92 409.17 G 0 X 90 450 6.75 6.75 263.92 409.17 A 7 X 90 450 6.75 6.94 263.92 368.48 G 0 X 90 450 6.75 6.94 263.92 368.48 A (1) 220.39 407.79 T (2) 260.89 407.79 T (3) 220.39 367.29 T (4) 260.89 367.29 T 0 12 Q (1) 298.13 389.84 T (2) 310.12 389.84 T (3) 322.12 389.84 T (4) 334.12 389.84 T 3 F (,) 304.13 389.84 T (,) 316.12 389.84 T (,) 328.12 389.84 T ({) 291.21 389.84 T (}) 340.67 389.84 T 108 63 540 720 C 0 0 612 792 C 393.3 196.11 482.4 212.04 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (A) 402.72 202.83 T 0 9 Q (1) 410.51 198.64 T 2 12 Q (A) 421.01 202.83 T 0 9 Q (2) 428.8 198.64 T 3 12 Q (\274) 439.3 202.83 T 2 F (A) 457.3 202.83 T 2 9 Q (M) 465.08 198.64 T 3 12 Q (,) 415.01 202.83 T (,) 433.3 202.83 T (,) 451.3 202.83 T ({) 395.81 202.83 T (}) 473.13 202.83 T 0 0 612 792 C 108 181.32 188.31 201.44 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (A) 142.39 188.83 T 2 9 Q (i) 150.18 184.64 T (i) 122.53 183.48 T 0 F (1) 136.09 183.48 T (=) 128.02 183.48 T 2 F (M) 122.53 194.88 T 3 18 Q (\310) 108.7 187.15 T 3 12 Q (W) 171.44 188.83 T 2 9 Q (X) 181.11 184.64 T 0 12 Q (=) 158.68 188.83 T 0 0 612 792 C 211.64 182.19 289.84 198.04 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (A) 212.05 188.83 T 2 9 Q (i) 219.84 184.64 T 2 12 Q (A) 237.56 188.83 T 2 9 Q (j) 245.35 184.64 T (i) 257.29 184.64 T 3 F (\271) 250.1 184.64 T 3 12 Q (\307) 225.34 188.83 T (\306) 278.55 188.83 T 0 F (=) 265.78 188.83 T 0 0 612 792 C 108 63 540 720 C 114 98.67 534 184.83 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 296.61 171.63 T (r) 303.28 171.63 T 2 F (x) 311.27 171.63 T 0 9 Q (0) 317.05 167.43 T 2 12 Q (A) 336.11 171.63 T 2 9 Q (i) 343.9 167.43 T 3 12 Q (\316) 324.55 171.63 T 0 F (\050) 307.27 171.63 T (\051) 346.4 171.63 T (P) 276.62 150.18 T (r) 283.29 150.18 T 2 F (x) 291.28 150.18 T 0 9 Q (1) 297.07 145.98 T 2 12 Q (A) 316.12 150.18 T 2 9 Q (j) 323.91 145.98 T 3 12 Q (\316) 304.57 150.18 T 2 F (x) 331.26 150.18 T 0 9 Q (0) 337.04 145.98 T 2 12 Q (A) 356.1 150.18 T 2 9 Q (i) 363.88 145.98 T 3 12 Q (\316) 344.54 150.18 T 0 F (\050) 287.29 150.18 T (\051) 366.39 150.18 T (P) 255.31 127.54 T (r) 261.98 127.54 T 2 F (x) 269.97 127.54 T 0 9 Q (2) 275.76 123.34 T 2 12 Q (A) 294.81 127.54 T 2 9 Q (k) 302.6 123.34 T 3 12 Q (\316) 283.26 127.54 T 2 F (x) 311.44 127.54 T 0 9 Q (0) 317.23 123.34 T 2 12 Q (A) 336.28 127.54 T 2 9 Q (i) 344.07 123.34 T 3 12 Q (\316) 324.72 127.54 T 2 F (x) 352.57 127.54 T 0 9 Q (1) 358.35 123.34 T 2 12 Q (A) 377.4 127.54 T 2 9 Q (j) 385.19 123.34 T 3 12 Q (\316) 365.85 127.54 T (,) 346.57 127.54 T 0 F (\050) 265.98 127.54 T (\051) 387.7 127.54 T 3 F (\274) 317.5 104.89 T 328.81 143.73 328.81 157.98 2 L 0.54 H 2 Z N 308.99 121.09 308.99 135.34 2 L N 108 63 540 720 C 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "8" 24 %%Page: "7" 25 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (7) 322 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 1 16 Q (4 Discr) 108 709.33 T (ete-State Stochastic Pr) 160.14 709.33 T (ocesses) 313.58 709.33 T 0 12 Q -0.12 (In a stochastic process, the resulting state is represented as a random variable) 108 682 P -0.12 ( and state) 492.02 682 P (transitions are governed by a probability distribution, where the state at time) 108 668 T ( is condi-) 482.96 668 T -0.38 (tioned on the state at earlier times. In this paper) 108 654 P -0.38 (, we are concerned with processes in which) 332.08 654 P (the state at) 108 640 T ( is conditionally independent of the states at time) 167.66 640 T ( and earlier given the) 429.9 640 T (state at time) 108 626 T (. W) 192.92 626 T (e write this as) 209.28 626 T 0 10 Q (\0501\051) 528.34 600 T 0 12 Q -0.28 (and refer to stochastic processes satisfying Equation) 108 570.35 P -0.28 (1 as Markov) 359.92 570.35 P 2 F -0.28 ( chains) 419.34 570.35 P 0 F -0.28 (. For the case of a) 453.39 570.35 P (\336nite state space) 108 556.35 T (, we use the following shorthand for describing the one-) 261.35 556.35 T (step transition probabilities for pairs of states.) 108 542.35 T 0 10 Q (\0502\051) 528.34 516.35 T 0 12 Q (and) 108 486.7 T 0 10 Q (\0503\051) 528.34 459.35 T 0 12 Q 0.11 (for describing the) 108 435.7 P 0.11 (-step transition probabilities, where) 203.97 435.7 P 0.11 ( is the probability of ending up) 391.36 435.7 P (in state) 108 421.7 T ( after) 150.67 421.7 T ( steps having started out in state) 186.65 421.7 T (.) 348.31 421.7 T -0.35 (It is not our intent to provide a primer on Markov chains, but there are a few properties that) 108 393.7 P 1.06 (are of particular interest in investigating the structure of dynamical systems modeled as) 108 379.7 P (Markov chains.) 108 365.7 T 0.53 (A subset) 108 337.7 P 0.53 ( of) 163.73 337.7 P 0.53 ( is said to be) 199.55 337.7 P 2 F 0.53 (closed) 265.2 337.7 P 0 F 0.53 (if) 299.38 337.7 P 0.53 ( for all) 348.82 337.7 P 0.53 ( and) 413.29 337.7 P 0.53 (. If a closed set) 465.57 337.7 P 0.88 (consists of a single state, then that state is called an) 108 323.7 P 2 F 0.88 (absorbing) 366.29 323.7 P 0 F 0.88 ( state. A Markov chain is) 414.96 323.7 P 0.01 (called) 108 309.7 P 2 F 0.01 (irreducible) 139.67 309.7 P 0 F 0.01 ( if there exists no nonempty closed set other than) 192.99 309.7 P 0.01 ( itself. If) 449.85 309.7 P 0.01 ( has a) 512.65 309.7 P (proper closed subset, then it is called) 108 295.7 T 2 F (reducible) 287.98 295.7 T 0 F (.) 333.3 295.7 T 0.14 (State) 108 267.7 P 0.14 ( has period) 140.47 267.7 P 0.14 ( if the following two conditions hold: \050i\051) 204.55 267.7 P 0.14 ( unless) 444.64 267.7 P 0.14 ( for) 522.87 267.7 P 0.02 (some positive integer) 108 253.7 P 0.02 (, and \050ii\051) 224.39 253.7 P 0.02 ( is the largest integer with property \050i\051. State) 276.45 253.7 P 0.02 ( is called) 497.29 253.7 P 2 F 0.5 (aperiodic) 108 239.7 P 0 F 0.5 ( when) 154 239.7 P 0.5 (. A state is said to be) 219.57 239.7 P 2 F 0.5 (persistent) 325.7 239.7 P 0 F 0.5 ( if the probability of ultimately re-) 372.37 239.7 P (turning to that state is one. If less than one, then the state is called) 108 225.7 T 2 F (transient) 426.3 225.7 T 0 F (. A state is) 468.97 225.7 T 2 F (positive) 108 211.7 T 0 F ( persistent if the expected number of steps to return to that state is finite.) 145.33 211.7 T (A) 108 185.7 T (\336nite-state Markov chain is) 119.66 185.7 T 2 F (er) 254.32 185.7 T (godic) 263.87 185.7 T 0 F ( if and only if there exists exactly one irreducible) 290.53 185.7 T (closed subset of positive persistent states and all of these states are aperiodic. A discrete-) 108 171.7 T (time Markov chain is said to be) 108 157.7 T 2 F (stationary) 262.32 157.7 T 0 F ( or) 310.99 157.7 T 2 F (homogeneous) 326.99 157.7 T 0 F ( in time if the probability of) 392.98 157.7 T -0.15 (going from one state to another is independent of the time at which the step is being made.) 108 143.7 P (Figure) 108 117.7 T (6 shows an er) 142.33 117.7 T (godic Markov chain with set of states) 207.77 117.7 T (. The) 487.5 117.7 T -0.14 (matrix on the right represents the one-step transition probabilities) 108 103.7 P -0.14 ( where the indices of) 437.72 103.7 P (the matrix are in the obvious correspondence with the states.) 108 89.7 T 479.73 675.36 492.02 691.21 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (X) 480.23 682 T 2 9 Q (t) 488.02 677.8 T 0 0 612 792 C 477.62 664.53 482.96 677.21 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 478.12 668 T 0 0 612 792 C 162.32 636.53 167.66 649.21 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 162.82 640 T 0 0 612 792 C 405.98 636.4 429.9 649.21 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 406.77 640 T 0 F (2) 422.1 640 T (\320) 413.11 640 T 0 0 612 792 C 169 622.4 192.92 635.21 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 169.79 626 T 0 F (1) 185.12 626 T (\320) 176.12 626 T 0 0 612 792 C 223.6 590.35 412.74 610 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (P) 226.82 600 T (r) 233.49 600 T 2 F (X) 241.48 600 T 2 9 Q (t) 249.27 595.8 T 2 12 Q (X) 256.62 600 T 2 9 Q (t) 264.41 595.8 T 0 F (1) 275.9 595.8 T (\320) 269.16 595.8 T 2 12 Q (X) 286.4 600 T 2 9 Q (t) 294.19 595.8 T 0 F (2) 305.68 595.8 T (\320) 298.94 595.8 T 3 12 Q (\274) 316.18 600 T (,) 280.4 600 T (,) 310.18 600 T 0 F (\050) 237.49 600 T (\051) 328.18 600 T (P) 350.94 600 T (r) 357.61 600 T 2 F (X) 365.61 600 T 2 9 Q (t) 373.39 595.8 T 2 12 Q (X) 380.74 600 T 2 9 Q (t) 388.53 595.8 T 0 F (1) 400.02 595.8 T (\320) 393.28 595.8 T 0 12 Q (\050) 361.61 600 T (\051) 404.52 600 T (=) 338.18 600 T 254.17 593.55 254.17 607.8 2 L 0.54 H 2 Z N 378.29 593.55 378.29 607.8 2 L N 0 0 612 792 C 190.32 551.35 261.35 566.35 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (1) 200.63 556.35 T (2) 212.63 556.35 T 3 F (\274) 224.63 556.35 T 2 F (N) 242.63 556.35 T 3 F (,) 206.63 556.35 T (,) 218.63 556.35 T (,) 236.63 556.35 T ({) 193.72 556.35 T (}) 251.19 556.35 T 0 0 612 792 C 215.13 506.7 421.21 526.35 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (p) 219.22 516.35 T 2 9 Q (i) 225.68 512.15 T (j) 228.71 512.15 T 0 12 Q (P) 249.97 516.35 T (r) 256.65 516.35 T 2 F (X) 264.64 516.35 T 2 9 Q (t) 272.43 512.15 T 2 12 Q (j) 289.69 516.35 T 0 F (=) 278.92 516.35 T 2 F (X) 297.87 516.35 T 2 9 Q (t) 305.66 512.15 T 0 F (1) 317.15 512.15 T (\320) 310.4 512.15 T 2 12 Q (i) 336.41 516.35 T 0 F (=) 325.65 516.35 T (\050) 260.64 516.35 T (\051) 339.74 516.35 T (=) 237.21 516.35 T 2 F (i) 373.72 516.35 T (j) 383.06 516.35 T 3 F (,) 377.06 516.35 T (W) 400.95 516.35 T 2 9 Q (X) 410.62 512.15 T 3 12 Q (\316) 389.39 516.35 T 295.42 509.9 295.42 524.15 2 L 0.54 H 2 Z N 0 0 612 792 C 214.83 449.7 421.51 470.7 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (p) 218.92 459.35 T 2 9 Q (i) 225.38 455.15 T (j) 228.41 455.15 T (n) 225.98 462.95 T 0 12 Q (P) 250.27 459.35 T (r) 256.95 459.35 T 2 F (X) 264.94 459.35 T 2 9 Q (t) 272.73 455.15 T 2 12 Q (j) 289.98 459.35 T 0 F (=) 279.22 459.35 T 2 F (X) 298.17 459.35 T 2 9 Q (t) 305.95 455.15 T (n) 317.45 455.15 T 0 F (\320) 310.7 455.15 T 2 12 Q (i) 336.71 459.35 T 0 F (=) 325.95 459.35 T (\050) 260.94 459.35 T (\051) 340.04 459.35 T (=) 237.51 459.35 T 2 F (i) 374.02 459.35 T (j) 383.36 459.35 T 3 F (,) 377.36 459.35 T (W) 401.25 459.35 T 2 9 Q (X) 410.92 455.15 T 3 12 Q (\316) 389.69 459.35 T 295.72 452.9 295.72 467.15 2 L 0.54 H 2 Z N 0 0 612 792 C 195.97 432.23 203.97 444.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 196.47 435.7 T 0 0 612 792 C 377.37 429.06 391.36 448.92 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (p) 377.58 435.7 T 2 9 Q (i) 384.03 431.5 T (j) 387.06 431.5 T (n) 384.63 439.3 T 0 0 612 792 C 145.33 418.23 150.67 430.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (j) 145.83 421.7 T 0 0 612 792 C 178.65 418.23 186.65 430.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 179.15 421.7 T 0 0 612 792 C 342.97 418.23 348.31 430.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (i) 343.47 421.7 T 0 0 612 792 C 153.72 334.23 163.73 346.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (C) 154.22 337.7 T 0 0 612 792 C 180.78 329.26 199.55 347.7 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 182.08 337.7 T 2 9 Q (X) 191.75 333.5 T 0 0 612 792 C 310.24 331.06 348.82 346.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (p) 310.66 337.7 T 2 9 Q (i) 317.11 333.5 T (j) 320.15 333.5 T 0 12 Q (0) 341.41 337.7 T (=) 328.64 337.7 T 0 0 612 792 C 385.4 334.23 413.29 346.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (i) 385.9 337.7 T (C) 403.79 337.7 T 3 F (\316) 392.23 337.7 T 0 0 612 792 C 437.68 334.23 465.57 346.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (j) 438.18 337.7 T (C) 456.07 337.7 T 3 F (\317) 444.52 337.7 T 0 0 612 792 C 431.08 301.26 449.86 319.7 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 432.38 309.7 T 2 9 Q (X) 442.05 305.5 T 0 0 612 792 C 493.88 301.26 512.65 319.7 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 495.18 309.7 T 2 9 Q (X) 504.85 305.5 T 0 0 612 792 C 135.14 264.23 140.47 276.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (j) 135.64 267.7 T 0 0 612 792 C 196.55 264.23 204.55 276.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (d) 197.05 267.7 T 0 0 612 792 C 402.33 259.26 444.64 279.05 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (p) 404.3 267.7 T 2 9 Q (j) 410.76 263.5 T (j) 413.8 263.5 T (n) 411.36 271.3 T 0 12 Q (0) 435.66 267.7 T (=) 422.89 267.7 T 0 0 612 792 C 480.91 264.23 522.87 276.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (n) 481.33 267.7 T (m) 506.09 267.7 T (d) 515.46 267.7 T 0 F (=) 493.32 267.7 T 0 0 612 792 C 213.73 250.23 224.39 262.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (m) 214.23 253.7 T 0 0 612 792 C 268.45 250.23 276.45 262.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (d) 268.95 253.7 T 0 0 612 792 C 491.96 250.23 497.29 262.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (j) 492.46 253.7 T 0 0 612 792 C 186.98 236.1 219.57 248.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (d) 187.39 239.7 T 0 F (1) 212.15 239.7 T (=) 199.39 239.7 T 0 0 612 792 C 390.74 109.25 487.5 127.7 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 392.04 117.7 T 2 9 Q (X) 401.72 113.5 T 0 12 Q (1) 434.89 117.7 T (2) 446.89 117.7 T (3) 458.89 117.7 T (4) 470.89 117.7 T 3 F (,) 440.89 117.7 T (,) 452.89 117.7 T (,) 464.89 117.7 T ({) 427.98 117.7 T (}) 477.44 117.7 T 0 F (=) 413.21 117.7 T 0 0 612 792 C 423.73 97.06 437.72 112.91 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (p) 424.23 103.7 T 2 9 Q (i) 430.68 99.5 T (j) 433.72 99.5 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "7" 25 %%Page: "6" 26 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (6) 322 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (In designing a representation for a dynamical system, we can choose to ignore certain) 108 712 T (states thereby restricting the state space, we can also choose to ignore certain distinctions) 108 698 T (between states. This idea of ignoring distinctions is often referred to as) 108 684 T 2 F (abstraction) 450.95 684 T 0 F (. One) 505.62 684 T (simple form of abstraction consists of simply ignoring variables. For example, we might) 108 670 T (choose to ignore illumination to obtain an abstracted state space) 108 656 T ( or) 463.92 656 T (This form of abstraction consists of projecting a three-dimensional space onto a two-) 108 573.64 T (dimensional space. A standard technique in both control and automated planning is to) 108 559.64 T (construct a solution in some low-dimensional space and then backproject into a higher) 108 545.64 T (-) 522.37 545.64 T (dimensional space to check or perhaps improve the solution. This technique is compli-) 108 531.64 T (cated by the fact that the backprojection is generally a one-to-many mapping.) 108 517.64 T (Another form of abstraction amounts to forming a partition of the set of states. For exam-) 108 491.64 T (ple, we obtain an abstraction) 108 477.64 T ( of) 303.1 477.64 T (, by partitioning the set of locations) 365.06 477.64 T ( into a \336nite set of sets of locations) 125.96 463.64 T ( or) 508.93 463.64 T (and the set of possible values for the battery) 108 375.44 T ( into a \336nite set of ranges of values) 340.71 375.44 T ( or) 340.12 361.44 T (In adaptive control, the term) 108 254.9 T 2 F (aggr) 247.31 254.9 T (egation) 269.53 254.9 T 0 F ( is often used to describe an abstraction. For) 305.53 254.9 T -0.35 (example, we can partition a state space that is isomorphic to the positive integers using the) 108 240.9 P (modulo 3 function to obtain) 108 226.9 T (, an abstrac-) 472.9 226.9 T -0.17 (tion of the positive integers consisting of 3 aggregate states. In solving planning problems) 108 212.9 P -0.25 (corresponding to Markov decision processes, often some function of the value assigned to) 108 198.9 P (states \050perhaps with respect to a particular control law\051 is used to partition the states.) 108 184.9 T (W) 108 158.9 T (e can take sums, products, or arbitrary functions of variables to obtain new variables.) 118.37 158.9 T (For example, we can de\336ne a new variable) 108 144.9 T ( which is a linear combination of illumina-) 329.27 144.9 T (tion and battery) 108 130.9 T (. W) 322.09 130.9 T (e make no use of this technique in the) 338.45 130.9 T (sequel) 108 116.9 T (, but mention it here to underscore the fact that state variables are not sacred. Next) 138.66 116.9 T (we turn to examine dynamical systems represented as stochastic processes.) 108 102.9 T 417.96 647.55 463.92 666 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 419.23 656 T 2 9 Q (L) 428.9 651.8 T 3 12 Q (W) 446.49 656 T 2 9 Q (B) 456.16 651.8 T 3 12 Q (\264) 436.9 656 T 0 0 612 792 C 108 63 540 720 C 242.27 595.64 405.72 652 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 362.05 615.77 356.15 614.67 360.57 618.73 361.31 617.25 4 Y 0 X 0 0 0 1 0 0 0 K V 389.85 633.37 395.75 634.47 391.33 630.41 390.59 631.89 4 Y V 361.31 617.25 390.59 631.89 2 L 0.5 H 0 Z N 375.95 644.37 375.95 604.77 2 L 2 Z N 90 450 0.99 0.99 375.95 639.42 G 0 Z 90 450 0.99 0.99 375.95 639.42 A 90 450 0.99 0.99 375.95 629.52 G 90 450 0.99 0.99 375.95 629.52 A 90 450 0.99 0.99 375.95 634.47 G 90 450 0.99 0.99 375.95 634.47 A 90 450 0.99 0.99 375.95 624.57 G 90 450 0.99 0.99 375.95 624.57 A 90 450 0.99 0.99 375.95 604.77 G 90 450 0.99 0.99 375.95 604.77 A 90 450 0.99 0.99 375.95 609.72 G 90 450 0.99 0.99 375.95 609.72 A 90 450 0.99 0.99 375.95 614.67 G 90 450 0.99 0.99 375.95 614.67 A 90 450 0.99 0.99 375.95 619.62 G 90 450 0.99 0.99 375.95 619.62 A 90 450 0.99 0.99 375.95 644.37 G 90 450 0.99 0.99 375.95 644.37 A 314.28 644.34 M 303.4 649.17 303.4 649.17 274.4 645.95 D 245.4 642.73 245.4 642.73 256.27 633.07 D 267.15 623.4 267.15 623.4 256.27 613.74 D 245.4 604.07 245.4 604.07 278.03 600.85 D 310.65 597.63 310.65 597.63 325.15 604.07 D 339.66 610.51 339.66 610.51 325.15 616.96 D 310.65 623.4 310.65 623.4 317.9 631.46 D 325.15 639.51 325.15 639.51 314.28 644.34 D O 7 X V 0 X N 275.28 615.78 269.38 614.68 273.8 618.74 274.54 617.26 4 Y V 303.08 633.38 308.98 634.48 304.56 630.42 303.82 631.9 4 Y V 274.54 617.26 303.82 631.9 2 L 7 X V 0 X N 269.76 626.78 264.43 629.53 270.41 630.02 270.09 628.4 4 Y V 308.6 622.39 313.93 619.63 307.95 619.14 308.27 620.77 4 Y V 270.09 628.4 308.27 620.76 2 L 7 X V 0 X N 289.18 644.38 289.18 604.78 2 L 7 X V 2 Z 0 X N 90 450 0.99 0.99 289.18 639.43 G 0 Z 90 450 0.99 0.99 289.18 639.43 A 90 450 0.99 0.99 289.18 629.53 G 90 450 0.99 0.99 289.18 629.53 A 90 450 0.99 0.99 289.18 634.48 G 90 450 0.99 0.99 289.18 634.48 A 90 450 0.99 0.99 289.18 624.58 G 90 450 0.99 0.99 289.18 624.58 A 90 450 0.99 0.99 289.18 604.78 G 90 450 0.99 0.99 289.18 604.78 A 90 450 0.99 0.99 289.18 609.73 G 90 450 0.99 0.99 289.18 609.73 A 90 450 0.99 0.99 289.18 614.68 G 90 450 0.99 0.99 289.18 614.68 A 90 450 0.99 0.99 289.18 619.63 G 90 450 0.99 0.99 289.18 619.63 A 90 450 0.99 0.99 289.18 644.38 G 90 450 0.99 0.99 289.18 644.38 A 364.27 641.7 M 371.27 648.33 371.27 648.33 378.27 648.33 D 385.27 648.33 385.27 648.33 392.27 643.91 D 399.27 639.48 399.27 639.48 396.93 628.42 D 394.6 617.36 394.6 617.36 385.27 608.51 D 375.94 599.66 375.94 599.66 366.61 601.87 D 357.28 604.09 357.28 604.09 354.94 610.72 D 352.61 617.36 352.61 617.36 354.94 626.21 D 357.28 635.06 357.28 635.06 364.27 641.7 D O N 3 14 Q (\336) 333.78 620.25 T 108 63 540 720 C 0 0 612 792 C 248.64 469.19 303.1 487.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 249.92 477.64 T 2 9 Q (L) 259.6 473.43 T 3 F (\262) 265.13 473.43 T 3 12 Q (W) 281.41 477.64 T 2 9 Q (B) 291.08 473.43 T 3 F (\262) 297.11 473.43 T 3 12 Q (\264) 271.83 477.64 T 0 0 612 792 C 319.09 469.19 365.06 487.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 320.36 477.64 T 2 9 Q (L) 330.03 473.43 T 3 12 Q (W) 347.62 477.64 T 2 9 Q (B) 357.29 473.43 T 3 12 Q (\264) 338.03 477.64 T 0 0 612 792 C 108 455.19 125.96 473.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 109.14 463.64 T 2 9 Q (L) 118.82 459.43 T 0 0 612 792 C 296.29 455.19 508.93 473.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 297.44 463.64 T 2 9 Q (L) 307.11 459.43 T 3 F (\262) 312.65 459.43 T 0 12 Q (1) 352.93 463.64 T (2) 364.93 463.64 T (3) 376.93 463.64 T 3 F (,) 358.93 463.64 T (,) 370.93 463.64 T ({) 346.02 463.64 T (}) 383.48 463.64 T (\274) 397.24 463.64 T 2 F (n) 424.16 463.64 T 0 F (2) 442.15 463.64 T (\320) 433.15 463.64 T 2 F (n) 454.15 463.64 T 0 F (1) 472.15 463.64 T (\320) 463.15 463.64 T 2 F (n) 484.15 463.64 T 3 F (,) 448.15 463.64 T (,) 478.15 463.64 T ({) 417.25 463.64 T (}) 490.7 463.64 T (,) 391.25 463.64 T (,) 409.24 463.64 T ({) 337.11 463.64 T (}) 499.01 463.64 T 0 F (=) 322.34 463.64 T 0 0 612 792 C 108 63 540 720 C 275.19 397.44 372.81 459.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 289.87 448.78 289.87 409.18 2 L 0.5 H 2 Z 0 X 0 0 0 1 0 0 0 K N 90 450 0.99 0.99 289.88 443.83 G 0 Z 90 450 0.99 0.99 289.88 443.83 A 90 450 0.99 0.99 289.88 433.93 G 90 450 0.99 0.99 289.88 433.93 A 90 450 0.99 0.99 289.88 438.88 G 90 450 0.99 0.99 289.88 438.88 A 90 450 0.99 0.99 289.88 428.98 G 90 450 0.99 0.99 289.88 428.98 A 90 450 0.99 0.99 289.88 409.18 G 90 450 0.99 0.99 289.88 409.18 A 90 450 0.99 0.99 289.88 414.13 G 90 450 0.99 0.99 289.88 414.13 A 90 450 0.99 0.99 289.88 419.08 G 90 450 0.99 0.99 289.88 419.08 A 90 450 0.99 0.99 289.88 424.03 G 90 450 0.99 0.99 289.88 424.03 A 90 450 0.99 0.99 289.88 448.78 G 90 450 0.99 0.99 289.88 448.78 A 297 451.23 M 294.52 456.18 294.52 456.18 289.57 453.7 D 284.62 451.23 284.62 451.23 282.15 441.33 D 279.67 431.43 279.67 431.43 282.15 419.05 D 284.62 406.68 284.62 406.68 289.57 404.2 D 294.52 401.73 294.52 401.73 297 409.15 D 299.47 416.58 299.47 416.58 297 421.53 D 294.52 426.48 294.52 426.48 297 436.38 D 299.47 446.28 299.47 446.28 297 451.23 D O N 3 14 Q (\336) 305.29 425.79 T 90 450 0.99 0.99 335.55 432.62 G 90 450 0.99 0.99 335.55 432.62 A 90 450 0.99 0.99 348.96 422.09 G 90 450 0.99 0.99 348.96 422.09 A 90 450 0.99 0.99 335.55 427.67 G 90 450 0.99 0.99 335.55 427.67 A 90 450 0.99 0.99 348.96 417.14 G 90 450 0.99 0.99 348.96 417.14 A 90 450 0.99 0.99 354.28 433.74 G 90 450 0.99 0.99 354.28 433.74 A 90 450 0.99 0.99 354.28 438.69 G 90 450 0.99 0.99 354.28 438.69 A 90 450 0.99 0.99 354.28 443.64 G 90 450 0.99 0.99 354.28 443.64 A 90 450 0.99 0.99 348.96 412.19 G 90 450 0.99 0.99 348.96 412.19 A 90 450 0.99 0.99 335.55 437.57 G 90 450 0.99 0.99 335.55 437.57 A 335.58 437.45 335.58 427.55 2 L V 2 Z N 348.99 422.11 348.99 412.21 2 L V N 354.31 443.53 354.31 433.63 2 L V N 331.18 428.65 M 328.7 436.08 328.7 436.08 331.18 438.55 D 333.65 441.03 333.65 441.03 336.13 441.03 D 338.61 441.03 338.61 441.03 341.08 436.08 D 343.55 431.13 343.55 431.13 338.61 426.18 D 333.65 421.23 333.65 421.23 331.18 428.65 D O 0 Z N 351.6 424.59 M 349.12 427.06 349.12 427.06 346.65 424.59 D 344.17 422.11 344.17 422.11 344.17 417.17 D 344.17 412.21 344.17 412.21 346.65 409.74 D 349.12 407.27 349.12 407.27 351.6 409.74 D 354.07 412.21 354.07 412.21 354.07 417.17 D 354.07 422.11 354.07 422.11 351.6 424.59 D O N 358.16 441.05 M 358.16 448.48 358.16 448.48 353.21 446 D 348.26 443.53 348.26 443.53 350.73 436.1 D 353.21 428.68 353.21 428.68 355.68 431.15 D 358.16 433.63 358.16 433.63 358.16 441.05 D O N 324.83 433.49 M 324.83 442.07 324.83 442.07 331.94 444.21 D 339.05 446.35 339.05 446.35 348.53 450.64 D 358.01 454.92 358.01 454.92 362.75 450.64 D 367.49 446.35 367.49 446.35 362.75 435.63 D 358.01 424.92 358.01 424.92 358.01 414.2 D 358.01 403.48 358.01 403.48 350.9 403.48 D 343.79 403.48 343.79 403.48 341.42 409.92 D 339.05 416.35 339.05 416.35 331.94 420.63 D 324.83 424.92 324.83 424.92 324.83 433.49 D O N 108 63 540 720 C 0 0 612 792 C 322.3 366.99 340.71 385.44 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 323.42 375.44 T 2 9 Q (B) 333.09 371.24 T 0 0 612 792 C 108 352.99 340.12 371.44 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 109.12 361.44 T 2 9 Q (B) 118.79 357.24 T 3 F (\262) 124.82 357.24 T 3 12 Q (\245) 169.8 361.44 T 0 F (\320) 163.35 361.44 T (0) 184.36 361.44 T 3 F (,) 178.36 361.44 T ([) 158.2 361.44 T (]) 190.91 361.44 T 0 F (0) 210.05 361.44 T 3 F (\050) 204.91 361.44 T 0 F (4) 224.61 361.44 T 3 F (]) 234.31 361.44 T 0 F (4) 253.46 361.44 T 3 F (\050) 248.31 361.44 T 0 F (8) 268.02 361.44 T 3 F (]) 277.72 361.44 T (,) 196.91 361.44 T (,) 218.61 361.44 T (,) 240.31 361.44 T (,) 262.02 361.44 T 0 F (8) 296.87 361.44 T 3 F (\050) 291.72 361.44 T (\245) 311.42 361.44 T (]) 323.68 361.44 T (,) 305.42 361.44 T (,) 283.72 361.44 T ({) 149.29 361.44 T (}) 330.23 361.44 T 0 F (=) 134.52 361.44 T 0 0 612 792 C 108 63 540 720 C 244.77 276.9 403.22 357.44 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 263.61 304.42 257.71 303.32 262.13 307.38 262.87 305.9 4 Y 0 X 0 0 0 1 0 0 0 K V 291.41 322.02 297.31 323.12 292.89 319.06 292.15 320.54 4 Y V 262.87 305.89 292.15 320.54 2 L 0.5 H 0 Z N 260.18 323.11 M 267.61 328.06 267.61 328.06 284.93 328.06 D 302.26 328.06 302.26 328.06 304.73 323.11 D 307.21 318.17 307.21 318.17 299.78 308.27 D 292.36 298.36 292.36 298.36 272.56 298.36 D 252.76 298.36 252.76 298.36 252.76 308.27 D 252.76 318.17 252.76 318.17 260.18 323.11 D O N 347.67 298.38 357.57 303.33 2 L 2 Z N 357.57 308.28 357.57 298.38 2 L N 347.67 303.33 347.67 293.43 2 L N 365.27 330.19 375.17 335.14 2 L N 343.95 318.92 338.05 317.82 342.47 321.87 343.21 320.39 4 Y V 347.95 322.77 343.21 320.39 2 L N 378.67 312.55 384.57 313.65 380.15 309.59 379.41 311.07 4 Y V 374.67 308.7 379.42 311.07 2 L N 347.95 327.71 347.95 317.82 2 L N 365.27 335.14 365.27 325.24 2 L N 374.67 313.65 374.67 303.75 2 L N 375.17 340.09 375.17 330.19 2 L N 342.27 300.08 M 342.27 307.42 342.27 307.42 352.17 309.86 D 362.07 312.3 362.07 312.3 362.07 307.42 D 362.07 302.53 362.07 302.53 359.59 295.2 D 357.12 287.87 357.12 287.87 349.69 290.31 D 342.27 292.75 342.27 292.75 342.27 300.08 D O 0 Z N 358.95 332.23 M 358.95 339.66 358.95 339.66 369.72 342.14 D 380.49 344.62 380.49 344.62 380.49 339.66 D 380.49 334.7 380.49 334.7 377.8 327.27 D 375.11 319.83 375.11 319.83 367.03 322.31 D 358.95 324.79 358.95 324.79 358.95 332.23 D O N 368.63 312.73 M 371.1 317.68 371.1 317.68 381 317.68 D 390.9 317.68 390.9 317.68 388.42 310.26 D 385.95 302.83 385.95 302.83 381 300.36 D 376.05 297.88 376.05 297.88 371.1 302.83 D 366.15 307.78 366.15 307.78 368.63 312.73 D O N 353.54 327.53 M 348.59 332.48 348.59 332.48 341.17 327.53 D 333.74 322.58 333.74 322.58 333.74 317.63 D 333.74 312.68 333.74 312.68 341.17 312.68 D 348.59 312.68 348.59 312.68 353.54 317.63 D 358.49 322.58 358.49 322.58 353.54 327.53 D O N 341.53 335.01 M 350.87 337.35 350.87 337.35 355.54 339.68 D 360.21 342.02 360.21 342.02 364.88 346.68 D 369.55 351.35 369.55 351.35 376.56 349.02 D 383.56 346.68 383.56 346.68 385.9 342.02 D 388.23 337.35 388.23 337.35 385.9 332.68 D 383.56 328.01 383.56 328.01 390.57 323.34 D 397.58 318.67 397.58 318.67 395.24 309.33 D 392.9 299.99 392.9 299.99 381.23 295.32 D 369.55 290.65 369.55 290.65 362.55 285.98 D 355.54 281.31 355.54 281.31 346.2 285.98 D 336.86 290.65 336.86 290.65 336.86 297.65 D 336.86 304.66 336.86 304.66 332.19 309.33 D 327.52 314 327.52 314 329.85 323.34 D 332.19 332.68 332.19 332.68 341.53 335.01 D O N 3 14 Q (\336) 311.98 308.96 T 108 63 540 720 C 0 0 612 792 C 245 221.9 472.9 236.9 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q 0 X 0 0 0 1 0 0 0 K (1) 263.83 226.9 T (4) 275.83 226.9 T (7) 287.83 226.9 T 3 F (\274) 299.83 226.9 T (,) 269.83 226.9 T (,) 281.83 226.9 T (,) 293.83 226.9 T ({) 256.92 226.9 T (}) 312.38 226.9 T 0 F (2) 335.05 226.9 T (5) 347.05 226.9 T (8) 359.05 226.9 T 3 F (\274) 371.05 226.9 T (,) 341.05 226.9 T (,) 353.05 226.9 T (,) 365.05 226.9 T ({) 328.14 226.9 T (}) 383.6 226.9 T 0 F (3) 406.28 226.9 T (6) 418.27 226.9 T (9) 430.27 226.9 T 3 F (\274) 442.27 226.9 T (,) 412.28 226.9 T (,) 424.27 226.9 T (,) 436.27 226.9 T ({) 399.36 226.9 T (}) 454.82 226.9 T (,) 320.14 226.9 T (,) 391.36 226.9 T ({) 248.01 226.9 T (}) 463.14 226.9 T 0 0 612 792 C 316.27 139.9 329.27 154.9 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (W) 317.27 144.9 T 0 0 612 792 C 186.32 122.45 322.09 140.9 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (W) 188.03 130.9 T (C) 223.94 130.9 T 0 9 Q (1) 232.4 126.71 T 2 12 Q (B) 249.49 130.9 T 3 F (\264) 239.9 130.9 T (\050) 218.79 130.9 T (\051) 257.37 130.9 T 2 F (C) 283.28 130.9 T 0 9 Q (2) 291.74 126.71 T 2 12 Q (I) 308.83 130.9 T 3 F (\264) 299.24 130.9 T (\050) 278.14 130.9 T (\051) 313.38 130.9 T 0 F (+) 266.37 130.9 T (=) 204.03 130.9 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "6" 26 %%Page: "5" 27 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (5) 322 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 1 16 Q (3 Sour) 108 709.33 T (ces of Structur) 156.61 709.33 T (e in State Space) 256.74 709.33 T 0 12 Q (As described so far) 108 682 T (, the state space) 199.82 682 T ( has no structure that we can take advantage of) 296.91 682 T (computationally) 108 668 T (. One standard method for imposing structure on a state space is to) 185.22 668 T 2 F (factor) 507.84 668 T 0 F -0.17 (the state space into a product of smaller sets, often referred to as) 108 654 P 2 F -0.17 (dimensions) 417.03 654 P 0 F -0.17 (, much as you) 471.03 654 P -0.29 (might factor a polynomial into a product polynomials of lower degree. W) 108 640 P -0.29 (e return to this in) 456.46 640 P (Section) 108 626 T (5, but some examples now will help to motivate subsequent discussion.) 147 626 T (Suppose that the state of a robot can be described in terms of its location) 108 600 T (, battery level) 468.62 600 T (, and the illumination) 118.33 586 T ( in the robot\325) 230.99 586 T (s immediate surroundings. Note that these three) 292.66 586 T (variables may not tell us everything about the robot and its surroundings, but they tell us) 108 572 T -0.45 (everything that we have chosen to represent. Suppose that battery and illumination are real) 108 558 P (valued so that) 108 544 T ( and location is discrete valued and \336nite so that) 256.26 544 T (. The resulting state space is) 209.95 530 T ( or graphically) 454.45 530 T -0.35 (The choice of variables is open. There is generally more than one useful representation for) 108 442.63 P -0.49 (a given dynamical system. There are also generally many more representations that are not) 108 428.63 P -0.17 (particularly useful. Some factorizations provide a poor representation for a dynamical sys-) 108 414.63 P (tem and other factorizations may include extraneous information. It may be, for example,) 108 400.63 T (that a robot need only concern itself with location and battery level in which case it can) 108 386.63 T (ignore illumination. What constitutes a good representation depends on how the represen-) 108 372.63 T (tation facilitates the robot achieving its assigned tasks.) 108 358.63 T (It is often useful to restrict the set of possible states to ease computational overhead.The) 108 332.63 T (modeler may choose to ignore states, perhaps because they are considered impossible or) 108 318.63 T (unlikely to reach. For example, we might introduce two new variables) 108 304.63 T ( and) 461.95 304.63 T (, restrict) 496.25 304.63 T -0.45 ( to the closed interval) 122.34 290.63 P -0.45 ( and) 304.02 290.63 P -0.45 ( to the closed interval) 337.41 290.63 P -0.45 (, to) 522.57 290.63 P (obtain) 108 276.63 T ( or graphically) 252.03 276.63 T ( is said to be a) 129.32 185.94 T 2 F (subspace) 200.65 185.94 T 0 F ( or) 244.64 185.94 T 2 F (r) 260.64 185.94 T (estriction) 264.86 185.94 T 0 F ( of) 310.2 185.94 T ( given that) 344.97 185.94 T (.) 450.28 185.94 T (Location and illumination are functionally dependent on one another) 108 159.94 T (. For example, if the) 437.29 159.94 T (robot is in a particular closet with the door shut, then the illumination is always either 0) 108 145.94 T -0.35 (\050the single 60 watt light bulb in the closet is not turned on\051 or 10 \050the light is on\051. Note that) 108 131.94 P (functional dependencies can impose restrictions on a state space that cannot be repre-) 108 117.94 T (sented by simply restricting the dimensions of the state space.) 108 103.94 T 278.14 673.55 296.91 692 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 279.44 682 T 2 9 Q (X) 289.11 677.8 T 0 0 612 792 C 458.95 595 468.62 610 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (L) 459.95 600 T 0 0 612 792 C 108 581 118.33 596 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (B) 109 586 T 0 0 612 792 C 224 581 230.99 596 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (I) 225 586 T 0 0 612 792 C 177.66 535.55 256.26 554 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 179.01 544 T 2 9 Q (B) 188.68 539.8 T 3 12 Q (W) 212.94 544 T 2 9 Q (I) 222.61 539.8 T 3 12 Q (\302) 244.37 544 T 0 F (=) 200.18 544 T (=) 231.6 544 T 0 0 612 792 C 108 521.55 209.95 540 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 109.14 530 T 2 9 Q (L) 118.81 525.8 T 0 12 Q (1) 151.49 530 T (2) 163.49 530 T 3 F (\274) 175.49 530 T 2 F (n) 193.49 530 T 3 F (,) 157.49 530 T (,) 169.49 530 T (,) 187.49 530 T ({) 144.58 530 T (}) 200.04 530 T 0 F (=) 129.82 530 T 0 0 612 792 C 348.26 521.55 454.45 540 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 350.05 530 T 2 9 Q (X) 359.72 525.8 T 3 12 Q (W) 383.98 530 T 2 9 Q (L) 393.65 525.8 T 3 12 Q (W) 411.24 530 T 2 9 Q (B) 420.91 525.8 T 3 12 Q (W) 438.99 530 T 2 9 Q (I) 448.67 525.8 T 3 12 Q (\264) 429.41 530 T (\264) 401.65 530 T 0 F (=) 371.21 530 T 0 0 612 792 C 108 63 540 720 C 169.77 464.63 478.23 526 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 246.04 517.69 M 235.16 522.53 235.16 522.53 206.16 519.3 D 177.16 516.08 177.16 516.08 188.03 506.42 D 198.91 496.75 198.91 496.75 188.03 487.09 D 177.16 477.42 177.16 477.42 209.79 474.2 D 242.41 470.98 242.41 470.98 256.91 477.42 D 271.41 483.87 271.41 483.87 256.91 490.31 D 242.41 496.75 242.41 496.75 249.66 504.81 D 256.91 512.86 256.91 512.86 246.04 517.69 D O 7 X 0 0 0 1 0 0 0 K V 0.5 H 0 Z 0 X N 207.04 489.14 201.14 488.04 205.56 492.1 206.3 490.62 4 Y V 234.84 506.73 240.74 507.83 236.32 503.78 235.58 505.26 4 Y V 206.3 490.62 235.58 505.26 2 L 7 X V 0 X N 201.52 500.13 196.19 502.89 202.17 503.38 201.85 501.76 4 Y V 240.35 495.74 245.69 492.99 239.71 492.5 240.03 494.12 4 Y V 201.85 501.76 240.03 494.12 2 L 7 X V 0 X N 220.94 517.73 220.94 478.14 2 L 7 X V 2 Z 0 X N 90 450 0.99 0.99 220.94 512.78 G 0 Z 90 450 0.99 0.99 220.94 512.78 A 90 450 0.99 0.99 220.94 502.89 G 90 450 0.99 0.99 220.94 502.89 A 90 450 0.99 0.99 220.94 507.84 G 90 450 0.99 0.99 220.94 507.84 A 90 450 0.99 0.99 220.94 497.94 G 90 450 0.99 0.99 220.94 497.94 A 90 450 0.99 0.99 220.94 478.14 G 90 450 0.99 0.99 220.94 478.14 A 90 450 0.99 0.99 220.94 483.09 G 90 450 0.99 0.99 220.94 483.09 A 90 450 0.99 0.99 220.94 488.04 G 90 450 0.99 0.99 220.94 488.04 A 90 450 0.99 0.99 220.94 492.99 G 90 450 0.99 0.99 220.94 492.99 A 292.22 501.97 M 292.22 517.23 292.22 517.23 299.64 521.05 D 307.07 524.86 307.07 524.86 310.78 509.6 D 314.49 494.33 314.49 494.33 307.07 479.07 D 299.64 463.8 299.64 463.8 295.93 475.25 D 292.22 486.7 292.22 486.7 292.22 501.97 D O 7 X V 0 X N 339.52 500.45 M 336.22 492.53 336.22 492.53 349.41 486.2 D 362.61 479.86 362.61 479.86 382.4 486.2 D 402.2 492.53 402.2 492.53 389 503.61 D 375.8 514.7 375.8 514.7 359.31 511.53 D 342.82 508.37 342.82 508.37 339.52 500.45 D O N 420.65 490.56 M 417.91 498.64 417.91 498.64 428.87 506.73 D 439.82 514.81 439.82 514.81 453.52 514.81 D 467.22 514.81 467.22 514.81 472.7 506.73 D 478.18 498.64 478.18 498.64 464.48 486.52 D 450.78 474.39 450.78 474.39 437.08 478.43 D 423.39 482.47 423.39 482.47 420.65 490.56 D O N 433.82 487.22 427.93 486.12 432.35 490.18 433.08 488.7 4 Y V 461.63 504.82 467.52 505.92 463.11 501.86 462.37 503.34 4 Y V 433.09 488.7 462.37 503.34 2 L N 348.63 498.84 343.3 501.59 349.28 502.08 348.95 500.46 4 Y V 387.46 494.44 392.8 491.69 386.82 491.2 387.14 492.82 4 Y V 348.96 500.46 387.14 492.82 2 L N 90 450 0.99 0.99 220.94 517.73 G 90 450 0.99 0.99 220.94 517.73 A 301.5 515.62 301.5 476.03 2 L V 2 Z N 90 450 0.99 0.99 301.5 510.68 G 0 Z 90 450 0.99 0.99 301.5 510.68 A 90 450 0.99 0.99 301.5 500.78 G 90 450 0.99 0.99 301.5 500.78 A 90 450 0.99 0.99 301.5 505.73 G 90 450 0.99 0.99 301.5 505.73 A 90 450 0.99 0.99 301.5 495.83 G 90 450 0.99 0.99 301.5 495.83 A 90 450 0.99 0.99 301.5 476.03 G 90 450 0.99 0.99 301.5 476.03 A 90 450 0.99 0.99 301.5 480.98 G 90 450 0.99 0.99 301.5 480.98 A 90 450 0.99 0.99 301.5 485.93 G 90 450 0.99 0.99 301.5 485.93 A 90 450 0.99 0.99 301.5 490.88 G 90 450 0.99 0.99 301.5 490.88 A 90 450 0.99 0.99 301.5 515.62 G 90 450 0.99 0.99 301.5 515.62 A 3 14 Q (\264) 323.71 494.02 T 0 F (=) 272.77 492.35 T 3 F (\264) 404.21 493.52 T 108 63 540 720 C 0 0 612 792 C 447.61 299.64 461.95 314.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (B) 448.78 304.63 T 3 F (\242) 456.82 304.63 T 0 0 612 792 C 485.28 299.64 496.25 314.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (I) 486.43 304.63 T 3 F (\242) 491.13 304.63 T 0 0 612 792 C 108 285.64 122.34 300.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (B) 109.17 290.63 T 3 F (\242) 117.21 290.63 T 0 0 612 792 C 226.39 282.19 304.02 300.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 227.51 290.63 T 2 9 Q (B) 237.18 286.43 T 3 F (\242) 243.21 286.43 T 0 12 Q (0) 271.35 290.63 T (1) 283.35 290.63 T (6) 289.35 290.63 T 3 F (,) 277.35 290.63 T ([) 266.2 290.63 T (]) 295.9 290.63 T 0 F (=) 251.43 290.63 T 0 0 612 792 C 326.44 285.64 337.41 300.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (I) 327.59 290.63 T 3 F (\242) 332.29 290.63 T 0 0 612 792 C 441.47 282.19 522.57 300.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 442.58 290.63 T 2 9 Q (I) 452.25 286.43 T 3 F (\242) 455.78 286.43 T 0 12 Q (0) 483.91 290.63 T (1) 495.91 290.63 T (0) 501.91 290.63 T (0) 507.91 290.63 T 3 F (,) 489.91 290.63 T ([) 478.76 290.63 T (]) 514.46 290.63 T 0 F (=) 464 290.63 T 0 0 612 792 C 138 268.19 252.03 286.64 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 139.58 276.63 T 2 9 Q (X) 149.25 272.43 T 3 F (\242) 155.28 272.43 T 3 12 Q (W) 176.26 276.63 T 2 9 Q (L) 185.93 272.43 T 3 12 Q (W) 203.52 276.63 T 2 9 Q (B) 213.2 272.43 T 3 F (\242) 219.23 272.43 T 3 12 Q (\264) 193.94 276.63 T (W) 234.03 276.63 T 2 9 Q (I) 243.71 272.43 T 3 F (\242) 247.23 272.43 T 3 12 Q (\264) 224.45 276.63 T 0 F (=) 163.5 276.63 T 0 0 612 792 C 108 63 540 720 C 229.77 207.94 418.23 272.63 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 296.51 244.3 M 296.51 259.57 296.51 259.57 303.93 263.38 D 311.35 267.2 311.35 267.2 315.06 251.93 D 318.78 236.67 318.78 236.67 311.35 221.4 D 303.93 206.13 303.93 206.13 300.22 217.58 D 296.51 229.03 296.51 229.03 296.51 244.3 D O 7 X 0 0 0 1 0 0 0 K V 0.5 H 0 Z 0 X N 249.32 240.58 269.12 235.62 2 L 7 X V 2 Z 0 X N 249.32 240.58 259.22 245.53 2 L 7 X V 0 X N 269.12 240.58 269.12 230.67 2 L 7 X V 0 X N 259.22 250.47 259.22 240.58 2 L 7 X V 0 X N 342.75 241.89 362.55 236.94 2 L 7 X V 0 X N 362.55 241.89 362.55 231.99 2 L 7 X V 0 X N 342.75 246.84 342.75 236.94 2 L 7 X V 0 X N 304.37 257.52 304.37 217.92 2 L 7 X V 0 X N 90 450 0.99 0.99 304.37 252.57 G 0 Z 90 450 0.99 0.99 304.37 252.57 A 90 450 0.99 0.99 304.37 242.67 G 90 450 0.99 0.99 304.37 242.67 A 90 450 0.99 0.99 304.37 247.62 G 90 450 0.99 0.99 304.37 247.62 A 90 450 0.99 0.99 304.37 237.72 G 90 450 0.99 0.99 304.37 237.72 A 90 450 0.99 0.99 304.37 217.92 G 90 450 0.99 0.99 304.37 217.92 A 90 450 0.99 0.99 304.37 222.87 G 90 450 0.99 0.99 304.37 222.87 A 90 450 0.99 0.99 304.37 227.82 G 90 450 0.99 0.99 304.37 227.82 A 90 450 0.99 0.99 304.37 232.77 G 90 450 0.99 0.99 304.37 232.77 A 90 450 0.99 0.99 304.37 257.52 G 90 450 0.99 0.99 304.37 257.52 A 249.15 260.68 249.15 221.08 2 L V 2 Z N 90 450 0.99 0.99 249.15 255.73 G 0 Z 90 450 0.99 0.99 249.15 255.73 A 90 450 0.99 0.99 249.15 245.83 G 90 450 0.99 0.99 249.15 245.83 A 90 450 0.99 0.99 249.15 250.78 G 90 450 0.99 0.99 249.15 250.78 A 90 450 0.99 0.99 249.15 240.88 G 90 450 0.99 0.99 249.15 240.88 A 90 450 0.99 0.99 249.15 221.08 G 90 450 0.99 0.99 249.15 221.08 A 90 450 0.99 0.99 249.15 226.03 G 90 450 0.99 0.99 249.15 226.03 A 90 450 0.99 0.99 249.15 230.98 G 90 450 0.99 0.99 249.15 230.98 A 90 450 0.99 0.99 249.15 235.93 G 90 450 0.99 0.99 249.15 235.93 A 90 450 0.99 0.99 249.15 260.68 G 90 450 0.99 0.99 249.15 260.68 A 404.41 244 394.51 239.05 2 L V 2 Z N 404.41 248.95 404.41 239.05 2 L V N 394.51 244 394.51 234.1 2 L V N 338.71 240.79 M 338.71 247.66 338.71 247.66 340.95 249.95 D 343.19 252.25 343.19 252.25 349.91 252.25 D 356.62 252.25 356.62 252.25 363.34 245.37 D 370.06 238.5 370.06 238.5 367.82 233.91 D 365.58 229.33 365.58 229.33 361.1 229.33 D 356.62 229.33 356.62 229.33 347.67 231.62 D 338.71 233.91 338.71 233.91 338.71 240.79 D O 0 Z N 394.13 249.19 M 398.69 253.9 398.69 253.9 403.26 253.9 D 407.83 253.9 407.83 253.9 407.83 249.19 D 407.83 244.49 407.83 244.49 410.11 239.78 D 412.4 235.07 412.4 235.07 405.55 235.07 D 398.69 235.07 398.69 235.07 394.13 230.37 D 389.56 225.66 389.56 225.66 389.56 235.07 D 389.56 244.49 389.56 244.49 394.13 249.19 D O N 252.08 264.59 M 247.37 267.07 247.37 267.07 242.65 262.12 D 237.94 257.17 237.94 257.17 240.29 252.22 D 242.65 247.27 242.65 247.27 242.65 232.42 D 242.65 217.57 242.65 217.57 247.37 215.09 D 252.08 212.62 252.08 212.62 254.44 220.04 D 256.8 227.47 256.8 227.47 266.23 227.47 D 275.66 227.47 275.66 227.47 275.66 237.37 D 275.66 247.27 275.66 247.27 270.95 247.27 D 266.23 247.27 266.23 247.27 261.51 254.69 D 256.8 262.12 256.8 262.12 252.08 264.59 D O N 0 14 Q (=) 283.73 236.12 T 3 F (\264) 325.5 237.79 T (\264) 375.16 236.46 T 108 63 540 720 C 0 0 612 792 C 108 177.49 129.32 195.94 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 109.2 185.94 T 2 9 Q (X) 118.87 181.74 T 3 F (\242) 124.9 181.74 T 0 0 612 792 C 326.2 177.49 344.97 195.94 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 327.5 185.94 T 2 9 Q (X) 337.17 181.74 T 0 0 612 792 C 398.63 177.49 450.28 195.94 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 400.13 185.94 T 2 9 Q (X) 409.8 181.74 T 3 F (\242) 415.83 181.74 T 3 12 Q (W) 432.61 185.94 T 2 9 Q (X) 442.28 181.74 T 3 12 Q (\315) 421.05 185.94 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "5" 27 %%Page: "4" 28 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (4) 322 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 1 10 Q (Figur) 121.75 713.33 T (e 4. Partially observable dynamical system) 145.46 713.33 T 0 12 Q (W) 108 542.5 T (e use the term) 118.37 542.5 T 2 F (contr) 188.35 542.5 T (ol system) 213.24 542.5 T 0 F (to refer to the combination of a state regulator \050control) 260.57 542.5 T (law\051 and a state estimator \050observer\051. As far as we are concerned, a) 108 528.5 T 2 F (plan) 432.88 528.5 T 0 F ( is just another) 454.21 528.5 T (name for a control law and) 108 514.5 T 2 F (planning) 239.96 514.5 T 0 F ( is the process of generating plans.) 282.64 514.5 T -0.5 (Figure) 108 488.5 P -0.5 (5 shows several examples of dynamical systems. Figure) 142.33 488.5 P -0.5 (5.i depicts a simple system) 410.48 488.5 P (with) 108 474.5 T ( and) 262.92 474.5 T (. Figure) 347.63 474.5 T (5.ii has just a little more inter-) 387.96 474.5 T -0 (esting set of actions,) 108 460.5 P -0 (. Figure) 282.37 460.5 P -0 (5.iii and Figure) 322.7 460.5 P -0 (5.iv represent dynamical sys-) 399.36 460.5 P (tems involving uncertainty) 108 446.5 T (. Figure) 235.88 446.5 T (5.iii is a partially observable system in which) 276.22 446.5 T ( but) 204.76 432.5 T (. Figure) 298.15 432.5 T (5.iv represents a completely observable) 338.48 432.5 T (system with a single action) 108 418.5 T ( but the transitions are uncertain. If the system is in state 1,) 249 418.5 T -0.03 (30% of the time action) 108 404.5 P -0.03 ( takes the system to state 2 and 70% of the time) 228.17 404.5 P -0.03 ( leaves the sys-) 467.11 404.5 P -0.25 (tem in state 1. The \336rst three systems in Figure) 108 390.5 P -0.25 (5 are governed by deterministic \336nite state) 334.1 390.5 P (automata. The dynamical system in Figure) 108 376.5 T (5.iv is governed by a) 315.65 376.5 T 2 F (stochastic pr) 418.97 376.5 T (ocess) 480.19 376.5 T 0 F (.) 506.18 376.5 T 1 10 Q (Figur) 121.75 351.83 T (e 5. Examples of dynamical systems) 145.46 351.83 T 108 63 540 720 C 196.5 566.5 451.5 710 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 248.41 660.1 313.21 703.3 R 0.5 H 0 Z 0 X 0 0 0 1 0 0 0 K N 356.41 660.1 399.61 703.3 R N 248.41 573.7 313.21 616.9 R N 348.72 683.91 356.41 681.7 348.72 679.49 348.72 681.7 4 Y V 313.21 681.7 348.72 681.7 2 L 2 Z N 356.41 573.7 399.61 616.9 R 0 Z N 408.26 593.93 400.57 596.14 408.26 598.34 408.26 596.14 4 Y V 400.57 682.53 443.77 682.53 443.77 596.14 408.26 596.14 4 L 2 Z N 320.9 593.1 313.21 595.3 320.9 597.51 320.9 595.3 4 Y V 356.41 595.3 320.9 595.3 2 L N 240.72 694.71 248.41 692.5 240.72 690.29 240.72 692.5 4 Y V 248.41 595.3 205.21 595.3 205.21 692.5 240.72 692.5 4 L N 240.72 673.1 248.41 670.9 240.72 668.7 240.72 670.9 4 Y V 334.81 681.7 334.81 638.5 226.81 638.5 226.81 670.9 240.72 670.9 5 L N 2 12 Q (f) 260.08 680.17 T (x) 271.28 680.17 T 2 9 Q (t) 277.06 675.97 T 2 12 Q (u) 285.56 680.17 T 2 9 Q (t) 292.02 675.97 T 3 12 Q (,) 279.56 680.17 T (\050) 266.13 680.17 T (\051) 295.07 680.17 T 2 F (x) 321.73 692.83 T 2 9 Q (t) 327.52 688.63 T 0 F (1) 339.59 688.63 T (+) 332.27 688.63 T 2 12 Q (h) 365.24 680.17 T (x) 379.1 680.17 T 2 9 Q (t) 384.89 675.97 T 3 12 Q (\050) 373.96 680.17 T (\051) 387.94 680.17 T 2 F (y) 418.04 693.77 T 2 9 Q (t) 423.82 689.57 T 3 12 Q (s) 362.57 594.5 T 2 F (y) 377.67 594.5 T 2 9 Q (t) 383.45 590.3 T 3 12 Q (\050) 372.52 594.5 T (\051) 386.5 594.5 T 2 F (x) 333.53 603.5 T 0 F (\366) 334.19 603.59 T 2 9 Q (t) 339.31 599.3 T 3 12 Q (d) 268.32 591.17 T 2 F (x) 282.11 591.17 T 0 F (\366) 282.77 591.25 T 2 9 Q (t) 287.89 586.97 T 3 12 Q (\050) 276.96 591.17 T (\051) 290.94 591.17 T 2 F (u) 222.35 604.67 T 2 9 Q (t) 228.81 600.47 T 108 63 540 720 C 0 0 612 792 C 132.34 466.05 262.92 484.5 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 133.83 474.5 T 2 9 Q (X) 143.51 470.3 T 3 12 Q (W) 167.76 474.5 T 2 9 Q (Y) 177.44 470.3 T 0 12 Q (1) 210.12 474.5 T (2) 222.12 474.5 T (3) 234.11 474.5 T (4) 246.11 474.5 T 3 F (,) 216.12 474.5 T (,) 228.12 474.5 T (,) 240.11 474.5 T ({) 203.21 474.5 T (}) 252.66 474.5 T 0 F (=) 155 474.5 T (=) 188.44 474.5 T 0 0 612 792 C 286.25 466.05 347.63 484.5 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 287.36 474.5 T 2 9 Q (U) 297.03 470.3 T 2 12 Q (a) 331.21 474.5 T 3 F ({) 324.3 474.5 T (}) 337.76 474.5 T 0 F (=) 309.53 474.5 T 0 0 612 792 C 208.99 452.05 282.37 470.5 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 210.1 460.5 T 2 9 Q (U) 219.77 456.3 T 2 12 Q (a) 253.94 460.5 T (b) 265.94 460.5 T 3 F (,) 259.94 460.5 T ({) 247.03 460.5 T (}) 272.49 460.5 T 0 F (=) 232.26 460.5 T 0 0 612 792 C 108 424.05 204.76 442.5 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 109.3 432.5 T 2 9 Q (X) 118.97 428.3 T 0 12 Q (1) 152.15 432.5 T (2) 164.15 432.5 T (3) 176.14 432.5 T (4) 188.14 432.5 T 3 F (,) 158.15 432.5 T (,) 170.15 432.5 T (,) 182.14 432.5 T ({) 145.24 432.5 T (}) 194.69 432.5 T 0 F (=) 130.47 432.5 T 0 0 612 792 C 226.09 424.05 298.15 442.5 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 227.29 432.5 T 2 9 Q (Y) 236.96 428.3 T 0 12 Q (1) 269.64 432.5 T (0) 281.64 432.5 T 3 F (,) 275.64 432.5 T ({) 262.73 432.5 T (}) 288.19 432.5 T 0 F (=) 247.96 432.5 T 0 0 612 792 C 241 415.03 249 427.71 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (a) 241.5 418.5 T 0 0 612 792 C 220.17 401.03 228.17 413.71 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (a) 220.67 404.5 T 0 0 612 792 C 459.11 401.03 467.11 413.71 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (a) 459.61 404.5 T 0 0 612 792 C 108 63 540 720 C 161.08 102.5 486.92 348.5 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 280.1 336.58 284.75 329.54 277.07 333.05 278.59 334.82 4 Y 0 X 0 0 0 1 0 0 0 K V 0.5 H 2 Z 46.82 180 32.4 21.6 256.42 319.07 A 306.25 263.08 299.61 258.61 302.88 265.92 304.56 264.5 4 Y V 316.82 450 21.6 32.4 288.82 286.67 A 231.88 237.36 227.55 244.46 234.99 240.73 233.43 239.04 4 Y V 224.82 360 32.4 21.6 256.42 254.27 A 207.1 311.2 214.21 315.54 210.48 308.1 208.79 309.65 4 Y V 134.82 270 21.6 32.4 224.02 286.67 A 7 X 90 450 10.8 10.8 224.02 319.07 G 0 Z 0 X 90 450 10.8 10.8 224.02 319.07 A 7 X 90 450 10.8 10.8 224.02 254.27 G 0 X 90 450 10.8 10.8 224.02 254.27 A 7 X 90 450 10.8 10.8 288.82 319.07 G 0 X 90 450 10.8 10.8 288.82 319.07 A 7 X 90 450 10.8 10.8 288.82 254.27 G 0 X 90 450 10.8 10.8 288.82 254.27 A 2 12 Q (a) 249.15 331.17 T (a) 206.08 285.07 T (a) 298.22 285.07 T (a) 249.28 236.47 T 0 F (1) 220.82 316.27 T (2) 285.62 316.27 T (3) 220.82 251.47 T (4) 285.62 251.47 T 375.82 250.16 367.31 251.77 375.38 254.91 375.6 252.54 4 Y V 2 Z 278.83 360 54 64.8 367.32 316.57 A 402.01 318.17 410.51 316.57 402.45 313.42 402.23 315.8 4 Y V 98.83 180 54 64.8 410.52 251.77 A (1) 349.12 315.57 T 412.6 334.08 417.25 327.04 409.57 330.55 411.09 332.32 4 Y V 46.82 180 32.4 21.6 388.92 316.57 A 438.75 260.58 432.11 256.11 435.37 263.41 437.06 262 4 Y V 316.82 450 21.6 32.4 421.32 284.17 A 364.38 234.86 360.05 241.96 367.48 238.23 365.93 236.54 4 Y V 224.82 360 32.4 21.6 388.92 251.77 A 339.6 308.7 346.71 313.04 342.98 305.6 341.29 307.15 4 Y V 134.82 270 21.6 32.4 356.52 284.17 A 7 X 90 450 10.8 10.8 356.52 316.57 G 0 Z 0 X 90 450 10.8 10.8 356.52 316.57 A 7 X 90 450 10.8 10.8 356.52 251.77 G 0 X 90 450 10.8 10.8 356.52 251.77 A 7 X 90 450 10.8 10.8 421.32 316.57 G 0 X 90 450 10.8 10.8 421.32 316.57 A 7 X 90 450 10.8 11.1 421.32 251.47 G 0 X 90 450 10.8 11.1 421.32 251.47 A 422.6 232.26 421.34 240.22 426.63 234.14 424.61 233.2 4 Y V 2 Z 226.04 450 10.8 10.8 432.12 240.97 A 355.81 335.4 356.52 327.37 351.66 333.8 353.74 334.6 4 Y V 42.04 270 10.8 10.8 345.72 327.37 A 2 F (a) 382.98 327.83 T (a) 431.58 282.57 T (a) 339.78 282.57 T (a) 382.98 233.97 T (b) 339.78 326.13 T (b) 427.88 236.87 T (b) 394.18 275.47 T (b) 373.85 290 T 0 F (1) 352.98 314.37 T (2) 417.78 314.37 T (3) 352.98 249.57 T (4) 417.78 249.57 T 280.93 214.09 285.59 207.04 277.91 210.55 279.42 212.32 4 Y V 46.82 180 32.4 21.6 257.25 196.57 A 307.08 140.58 300.45 136.11 303.71 143.42 305.4 142 4 Y V 316.82 450 21.6 32.4 289.65 164.17 A 232.71 114.86 228.38 121.97 235.82 118.23 234.27 116.55 4 Y V 224.82 360 32.4 21.6 257.25 131.77 A 207.94 188.71 215.04 193.04 211.31 185.6 209.63 187.15 4 Y V 134.82 270 21.6 32.4 224.85 164.17 A 7 X 90 450 10.8 10.8 224.85 196.57 G 0 Z 0 X 90 450 10.8 10.8 224.85 196.57 A 7 X 90 450 10.8 10.8 224.85 131.77 G 0 X 90 450 10.8 10.8 224.85 131.77 A 7 X 90 450 10.8 10.8 289.65 196.57 G 0 X 90 450 10.8 10.8 289.65 196.57 A 7 X 90 450 10.8 10.8 289.65 131.77 G 0 X 90 450 10.8 10.8 289.65 131.77 A 2 F (a) 249.98 206.17 T (a) 206.91 162.57 T (a) 299.05 162.57 T (a) 250.95 116.47 T 0 F (1) 221.65 193.77 T (0) 286.45 193.77 T (0) 221.65 128.97 T (1) 286.45 128.97 T 377.82 129.08 369.32 130.69 377.38 133.83 377.6 131.46 4 Y V 2 Z 278.83 360 54 64.8 369.32 195.48 A 404.01 197.1 412.51 195.49 404.45 192.34 404.23 194.72 4 Y V 98.83 180 54 64.8 412.52 130.69 A (1) 351.12 194.48 T 414.6 213 419.25 205.96 411.57 209.47 413.09 211.24 4 Y V 46.82 180 32.4 21.6 390.92 195.48 A 440.75 139.5 434.11 135.03 437.38 142.34 439.06 140.92 4 Y V 316.82 450 21.6 32.4 423.32 163.09 A 366.38 113.78 362.05 120.88 369.49 117.15 367.93 115.47 4 Y V 224.82 360 32.4 21.6 390.92 130.68 A 341.61 187.62 348.71 191.96 344.98 184.52 343.29 186.07 4 Y V 134.82 270 21.6 32.4 358.52 163.09 A 7 X 90 450 10.8 10.8 358.52 195.48 G 0 Z 0 X 90 450 10.8 10.8 358.52 195.48 A 7 X 90 450 10.8 10.8 358.52 130.69 G 0 X 90 450 10.8 10.8 358.52 130.69 A 7 X 90 450 10.8 10.8 423.32 195.48 G 0 X 90 450 10.8 10.8 423.32 195.48 A 7 X 90 450 10.8 11.1 423.32 130.39 G 0 X 90 450 10.8 11.1 423.32 130.39 A 424.6 111.17 423.34 119.14 428.63 113.05 426.62 112.11 4 Y V 2 Z 226.04 450 10.8 10.8 434.12 119.88 A 357.81 214.32 358.52 206.29 353.66 212.72 355.74 213.52 4 Y V 42.04 270 10.8 10.8 347.72 206.28 A 2 F (a) 431.91 116.72 T (a) 430.25 156.48 T (a) 340.12 167.32 T (a) 382.48 115.39 T 0 F (1) 354.98 193.28 T (2) 419.78 193.28 T (3) 354.98 128.49 T (4) 419.78 128.49 T 2 F (a) 378.38 172.39 T (a) 394.58 150.79 T (a) 389.62 207.32 T (a) 342.25 207.32 T 0 F (0) 369.72 200.62 T (.) 375.72 200.62 T (3) 378.72 200.62 T (0) 355.58 216.42 T (.) 361.58 216.42 T (7) 364.58 216.42 T (0) 326.92 127.92 T (.) 332.92 127.92 T (1) 335.92 127.92 T (0) 343.12 149.55 T (.) 349.12 149.55 T (9) 352.12 149.55 T (0) 424.11 172.78 T (.) 430.11 172.78 T (2) 433.11 172.78 T (0) 398.35 179.48 T (.) 404.35 179.48 T (8) 407.35 179.48 T (0) 444.91 130.89 T (.) 450.91 130.89 T (4) 453.91 130.89 T (0) 395.45 120.09 T (.) 401.45 120.09 T (6) 404.45 120.09 T 1 F (i.) 183.75 283.5 T (ii.) 459.58 286 T (iii.) 178.75 160.17 T (iv) 458.75 159.83 T (.) 467.43 159.83 T 108 63 540 720 C 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "4" 28 %%Page: "3" 29 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (3) 322 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (crete-time dynamical systems, including production scheduling in factories, traf) 108 712 T (\336c light) 491.06 712 T (regulation in cities, and gate scheduling for the airlines.) 108 698 T 1 10 Q (Figur) 121.75 673.33 T (e 2. Examples of simple dynamical systems) 145.46 673.33 T 0 12 Q (In order to introduce the notion of control, we need a set of possible inputs to the dynami-) 108 554.17 T (cal system. Let) 108 540.17 T ( correspond to the set of all inputs to the dynamical system. These) 202.72 540.17 T (inputs are often also called) 108 526.17 T 2 F (actions,) 239.64 526.17 T 0 F (and later we refer to them as) 280.31 526.17 T 2 F (decisions.) 419.93 526.17 T 0 F ( The action at) 467.59 526.17 T (time) 108 512.17 T ( is determined by a) 137.67 512.17 T 2 F (contr) 231.99 512.17 T (ol law) 256.88 512.17 T 0 F ( that takes the current state) 298.48 512.17 T ( and returns the cur-) 440.07 512.17 T (rent action) 108 498.17 T (. The interaction between the control law) 220.16 498.17 T (, also called a) 416.33 498.17 T 2 F (state r) 484.65 498.17 T (egu-) 514.54 498.17 T (lator) 108 484.17 T 0 F (, and the state transition function is shown in Figure) 130.86 484.17 T (3. Note that now the state) 383.18 484.17 T (transition function takes two ar) 108 470.17 T (guments: the current state and the current action.) 257.1 470.17 T 1 10 Q (Figur) 121.75 445.5 T (e 3. State r) 145.46 445.5 T (egulator in\337uencing the behavior of a dynamical system) 190.82 445.5 T 0 12 Q -0.26 (In the model described above, we assume that the state regulator is able to observe the cur-) 108 281.33 P (rent state of the dynamical system. In most realistic problems, the state regulator is only) 108 267.33 T (able to observe limited aspects of the current state. Such systems are said to be) 108 253.33 T 2 F (partially) 488.6 253.33 T -0.4 (observable.) 108 239.33 P 0 F -0.4 (Let) 166.26 239.33 P -0.4 ( be the set of observable outputs of the dynamical system and) 202.92 239.33 P -0.35 (be the output at time) 108 225.33 P -0.35 (. W) 213.27 225.33 P -0.35 (e introduce a new function) 229.29 225.33 P -0.35 ( which takes the current state) 366.52 225.33 P -0.35 ( and) 517.36 225.33 P (returns the observable output) 108 211.33 T ( at time) 308.21 211.33 T (. Since the state regulator does not) 352.55 211.33 T (know the current state of the dynamical system, we add a new component called an) 108 197.33 T 2 F (observer) 108 183.33 T 0 F ( or) 149.99 183.33 T 2 F (state estimator) 165.98 183.33 T 0 F ( that tries to recover the state of the system. An observer) 236.99 183.33 T -0.03 (takes the current output and returns an estimate) 108 169.33 P -0.03 ( of the current state. This esti-) 396.53 169.33 P -0.2 (mate might be the observer) 108 155.33 P -0.2 (\325) 238.3 155.33 P -0.2 (s best guess for the current state or a more complicated sort of) 241.63 155.33 P (estimate. Figure) 108 141.33 T (4 shows the interactions involving the state regulator) 188.33 141.33 T (, the state estima-) 453.9 141.33 T -0.36 (tor) 108 127.33 P -0.36 (, and the rest of the dynamical system. In some cases, the state regulator and the state) 134.21 127.33 P (estimator can be designed independently and assuming that each is optimal at its respec-) 108 113.33 T (tive task, regulation or estimation, then the combined control system consisting of the) 108 99.33 T (state regulator coupled to the state estimator is optimal. In such cases, observation and) 108 85.33 T (regulation are said to be) 108 71.33 T 2 F (separable.) 226.31 71.33 T 108 63 540 720 C 184.83 578.17 463.17 670 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 218.93 587.48 235.13 603.68 R 0.5 H 0 Z 0 X 0 0 0 1 0 0 0 K N 218.93 603.68 235.13 619.88 R N 194.63 587.48 239.18 587.48 2 L 2 Z N 0 12 Q (A) 224.53 608.09 T (B) 224.03 591.44 T 315.53 598.28 331.73 614.48 R 0 Z N 295.28 598.28 311.48 614.48 R N 291.23 598.28 335.78 598.28 2 L 2 Z N (A) 300.88 602.69 T (B) 320.63 602.24 T 198.68 645.08 214.88 661.28 R 0 Z N 198.68 628.88 214.88 645.08 R N 194.63 628.88 239.18 628.88 2 L 2 Z N (A) 204.28 633.29 T (B) 203.78 649.04 T 416.93 601.72 408.87 603.22 416.49 606.22 416.71 603.97 4 Y V 286.02 371 21.6 21.6 410.75 624.73 A 428.19 642.77 431.19 635.14 424.6 640.02 426.39 641.39 4 Y V 48.02 137 21.6 21.6 411.95 625.33 A 388.68 629.46 393.44 636.14 392.94 627.95 390.81 628.71 4 Y V 171.02 248 21.6 21.6 412.15 625.33 A 438.85 618.58 433.99 625.01 441.52 622.12 440.18 620.35 4 Y V 254.04 540 10.8 10.8 443.15 630.73 A 389.55 605.58 397.43 607.25 391.64 601.65 390.6 603.61 4 Y V 139.04 451 10.8 10.8 398.75 596.53 A 402.61 652.15 398.7 645.09 398.28 653.14 400.44 652.64 4 Y V 8.04 296 10.8 10.8 389.75 651.13 A (Rain) 365.06 631.58 T (Sun) 434.38 644.58 T (Clouds) 414.05 590.21 T 7 X 90 450 5.4 5.4 397.55 640.33 G 0 Z 0 X 90 450 5.4 5.4 397.55 640.33 A 7 X 90 450 5.4 5.4 432.35 628.93 G 0 X 90 450 5.4 5.4 432.35 628.93 A 7 X 90 450 5.4 5.4 403.55 606.13 G 0 X 90 450 5.4 5.4 403.55 606.13 A 272.51 602.72 264.45 604.22 272.08 607.22 272.29 604.97 4 Y V 2 Z 286.02 371 21.6 21.6 266.33 625.73 A 283.77 643.77 286.77 636.14 280.18 641.02 281.98 642.39 4 Y V 48.02 137 21.6 21.6 267.53 626.33 A 274.5 631.19 282.61 629.98 275.09 626.71 274.8 628.95 4 Y V 108.02 180 21.6 21.6 281.48 608.41 A 294.42 619.58 289.57 626.01 297.1 623.12 295.76 621.35 4 Y V 254.04 540 10.8 10.8 298.73 631.73 A 244.98 605.74 252.69 608.1 247.4 602.01 246.19 603.88 4 Y V 144.04 450 10.8 10.8 254.93 597.53 A 258.19 653.15 254.28 646.09 253.86 654.14 256.02 653.64 4 Y V 8.04 296 10.8 10.8 245.33 652.13 A 255.96 628.3 253.5 636.12 259.73 630.8 257.85 629.55 4 Y V 224.02 314 21.6 21.6 273.38 644.56 A 7 X 90 450 5.4 5.4 253.13 641.33 G 0 Z 0 X 90 450 5.4 5.4 253.13 641.33 A 7 X 90 450 5.4 5.4 287.93 629.93 G 0 X 90 450 5.4 5.4 287.93 629.93 A 7 X 90 450 5.4 5.4 259.13 607.13 G 0 X 90 450 5.4 5.4 259.13 607.13 A 108 63 540 720 C 0 0 612 792 C 183.32 531.72 202.72 550.17 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 184.43 540.17 T 2 9 Q (U) 194.11 535.97 T 0 0 612 792 C 132.34 508.7 137.67 521.38 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 132.84 512.17 T 0 0 612 792 C 289.55 507.17 298.48 522.17 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (d) 290.55 512.17 T 0 0 612 792 C 429.78 505.52 440.07 521.38 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (x) 430.29 512.17 T 2 9 Q (t) 436.07 507.97 T 0 0 612 792 C 161.99 491.52 220.16 507.38 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (d) 162.4 498.17 T 2 F (x) 176.19 498.17 T 2 9 Q (t) 181.97 493.97 T 3 12 Q (\050) 171.04 498.17 T (\051) 185.02 498.17 T 2 F (u) 209.79 498.17 T 2 9 Q (t) 216.24 493.97 T 0 12 Q (=) 197.02 498.17 T 0 0 612 792 C 108 63 540 720 C 241.08 305.33 406.92 442.17 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 292.35 393.7 357.15 436.9 R 0.5 H 0 Z 0 X 0 0 0 1 0 0 0 K N 303.15 312.1 346.35 344.5 R N 284.66 428.3 292.35 426.1 284.66 423.89 284.66 426.1 4 Y V 303.15 328.9 249.15 328.9 249.15 426.1 284.66 426.1 4 L 2 Z N 2 12 Q (u) 280.6 337.65 T 2 9 Q (t) 287.06 333.45 T 354.04 326.7 346.35 328.9 354.04 331.11 354.04 328.9 4 Y V 357.15 415.3 400.35 415.3 400.35 328.9 354.04 328.9 4 L N 284.66 406.7 292.35 404.5 284.66 402.29 284.66 404.5 4 Y V 378.75 415.3 378.75 372.1 270.75 372.1 270.75 404.5 284.66 404.5 5 L N 3 12 Q (d) 310.06 327.03 T 2 F (x) 323.84 327.03 T 2 9 Q (t) 329.63 322.83 T 3 12 Q (\050) 318.7 327.03 T (\051) 332.68 327.03 T 2 F (f) 304.09 413 T (x) 315.29 413 T 2 9 Q (t) 321.07 408.8 T 2 12 Q (u) 329.57 413 T 2 9 Q (t) 336.03 408.8 T 3 12 Q (,) 323.57 413 T (\050) 310.14 413 T (\051) 339.08 413 T 2 F (x) 365.91 426.17 T 2 9 Q (t) 371.69 421.97 T 0 F (1) 383.76 421.97 T (+) 376.44 421.97 T 108 63 540 720 C 0 0 612 792 C 184.85 230.88 202.92 249.33 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 186.05 239.33 T 2 9 Q (Y) 195.72 235.13 T 0 0 612 792 C 496.08 230.88 537.4 249.33 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (y) 497.48 239.33 T 2 9 Q (t) 503.27 235.13 T 3 12 Q (W) 520.32 239.33 T 2 9 Q (Y) 529.99 235.13 T 3 12 Q (\316) 508.77 239.33 T 0 0 612 792 C 207.93 221.86 213.27 234.54 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 208.43 225.33 T 0 0 612 792 C 358.52 221.86 366.52 234.54 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (h) 359.02 225.33 T 0 0 612 792 C 507.08 218.69 517.36 234.54 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (x) 507.58 225.33 T 2 9 Q (t) 513.36 221.13 T 0 0 612 792 C 250.64 204.69 308.21 220.54 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (h) 251.06 211.33 T (x) 264.92 211.33 T 2 9 Q (t) 270.7 207.13 T 3 12 Q (\050) 259.77 211.33 T (\051) 273.75 211.33 T 2 F (y) 298.51 211.33 T 2 9 Q (t) 304.3 207.13 T 0 12 Q (=) 285.75 211.33 T 0 0 612 792 C 347.21 207.86 352.55 220.54 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 347.71 211.33 T 0 0 612 792 C 510.6 178.33 520.83 193.33 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (s) 511.6 183.33 T 0 0 612 792 C 337.73 162.69 396.53 179.69 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (s) 338.14 169.33 T 2 F (y) 353.23 169.33 T 2 9 Q (t) 359.02 165.13 T 3 12 Q (\050) 348.09 169.33 T (\051) 362.07 169.33 T 2 F (x) 386.83 169.33 T 0 F (\366) 387.5 169.42 T 2 9 Q (t) 392.62 165.13 T 0 12 Q (=) 374.07 169.33 T 0 0 612 792 C 444.97 136.33 453.9 151.33 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (d) 445.97 141.33 T 0 0 612 792 C 123.97 122.33 134.2 137.33 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (s) 124.97 127.33 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "3" 29 %%Page: "2" 30 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (2) 322 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 12 Q (i\336cation, frame or rami\336cation problems that plague representations of time and change,) 108 712 T (but neither do they present representational barriers to solving these problems.) 108 698 T (In this paper) 108 672 T (, we explore stochastic processes as a representation for reasoning under) 167.51 672 T -0.39 (uncertainty about change over time. W) 108 658 P -0.39 (e introduce standard notation and terminology from) 291.73 658 P (the literature on the control of stochastic processes. W) 108 644 T (e describe a classical framework) 367.32 644 T (employing stochastic processes for decision making under uncertainty) 108 630 T (, and investigate) 444.17 630 T (how the sort of decomposable representations favored in arti\336cial intelligence can be) 108 616 T -0.45 (added to the framework. Wherever possible we use graphical devices to illustrate concepts) 108 602 P (that, while somewhat complicated notationally) 108 588 T (, are quite easy to visualize. W) 331.54 588 T (e brie\337y sur-) 477.88 588 T (vey existing approaches to decision-theoretic planning under uncertainty and describe) 108 574 T (how they relate to this framework. Finally) 108 560 T (, we conclude with pointers to the literature for) 309.53 560 T (those who wish to obtain additional background.) 108 546 T 1 16 Q (2 Dynamics, Regulation and Observation) 108 505.33 T 0 12 Q (In this section, we consider the structure of time and state. Let) 108 478 T ( represent the set of) 427.37 478 T (time points that we are willing to consider) 108 464 T (.) 309.66 464 T ( can be discrete \050) 333.75 464 T (\051 or) 517.12 464 T -0.19 (continuous \050) 108 450 P -0.19 (\051, \336nite or in\336nite, bounded or not. In this paper) 213.86 450 P -0.19 (, we assume that the) 440.97 450 P (set of time points is isomorphic to the set of positive integers, though much of what we) 108 436 T (have to say applies to cases in which the set of time points is isomorphic to the real num-) 108 422 T -0.08 (bers. Let) 108 408 P -0.08 ( represent the set of possible states we are willing to consider) 171.6 408 P -0.08 (. The set) 463.64 408 P -0.08 ( is) 526.16 408 P (called the) 108 394 T 2 F (state space) 157.32 394 T 0 F (. In this paper) 210.31 394 T (, we assume that the state space is \336nite, though much) 275.82 394 T (of what we say applies to countable state spaces.) 108 380 T 1 10 Q (Figur) 121.75 355.33 T (e 1. Simple discr) 145.46 355.33 T (ete-time dynamical system) 215.27 355.33 T 0 12 Q -0.19 (A) 108 254.5 P 2 F -0.19 (dynamical system) 119.47 254.5 P 0 F -0.19 (is de\336ned by a set of time points, a set of states, and a) 207.07 254.5 P 2 F -0.19 (state transition) 465.03 254.5 P -0.4 (function) 108 240.5 P 0 F -0.4 (that maps states to states. Let) 149.94 240.5 P -0.4 ( represent the state of the dynamical system) 330.56 240.5 P (at time) 108 226.5 T ( and) 149.33 226.5 T ( represent the state of the system at time) 200.69 226.5 T (. Figure) 420.25 226.5 T (1 depicts a sim-) 460.58 226.5 T (ple discrete-time dynamical system. The product of state and time) 108 212.5 T ( is called the) 472.06 212.5 T 2 F (phase space) 108 198.5 T 0 F (for the dynamical system.) 169.32 198.5 T (Pedagogical examples of discrete-time dynamical systems abound. In arti\336cial intelli-) 108 172.5 T -0.14 (gence, the blocks world is often used to motivate planning. In textbooks on stochastic pro-) 108 158.5 P (cesses, examples like the process governing the weather in the land of Oz are used to) 108 144.5 T (motivate reasoning about uncertainty) 108 130.5 T (. Figure) 285.53 130.5 T (2 shows the dynamical systems for a simple) 325.86 130.5 T (case of the blocks world and the weather in the land of Oz. States are depicted as circles) 108 116.5 T -0.23 (and possible transitions as arcs connecting states. The graphs shown in Figure) 108 102.5 P -0.23 (2 are called) 482.39 102.5 P 2 F (state transition diagrams) 108 88.5 T 0 F (. Many problems of practical interest can be represented as dis-) 228.68 88.5 T 409.28 469.55 427.37 488 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 410.49 478 T 2 9 Q (T) 420.16 473.8 T 0 0 612 792 C 315.66 455.55 333.75 474 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 316.87 464 T 2 9 Q (T) 326.54 459.8 T 0 0 612 792 C 415.05 455.55 517.12 474 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 416.25 464 T 2 9 Q (T) 425.92 459.8 T 0 12 Q (0) 458.6 464 T (1) 470.6 464 T 3 F (\274) 482.6 464 T 2 F (n) 500.6 464 T 3 F (,) 464.6 464 T (,) 476.6 464 T (,) 494.6 464 T ({) 451.69 464 T (}) 507.15 464 T 0 F (=) 436.92 464 T 0 0 612 792 C 167.47 441.55 213.86 460 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 168.68 450 T 2 9 Q (T) 178.35 445.8 T 3 12 Q (\302) 202.12 450 T 0 F (=) 189.35 450 T 0 0 612 792 C 152.82 399.55 171.6 418 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 154.12 408 T 2 9 Q (X) 163.8 403.8 T 0 0 612 792 C 507.39 399.55 526.16 418 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 508.69 408 T 2 9 Q (X) 518.36 403.8 T 0 0 612 792 C 108 63 540 720 C 267.75 278.5 380.25 352 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 297.99 311.38 341.19 343.78 R 0.5 H 0 Z 0 X 0 0 0 1 0 0 0 K N 290.3 331.58 297.99 329.38 290.3 327.17 290.3 329.38 4 Y V 341.19 329.38 362.79 329.38 362.79 286.18 276.39 286.18 276.39 329.38 290.3 329.38 6 L 2 Z N 2 12 Q (x) 348.43 339.67 T 2 9 Q (t) 354.21 335.47 T 0 F (1) 366.28 335.47 T (+) 358.96 335.47 T 2 12 Q (f) 307.73 325.15 T (x) 318.93 325.15 T 2 9 Q (t) 324.71 320.95 T 3 12 Q (\050) 313.78 325.15 T (\051) 327.77 325.15 T 108 63 540 720 C 0 0 612 792 C 290.54 232.99 330.56 249.71 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (x) 291.04 240.5 T 2 9 Q (t) 296.83 236.3 T 3 12 Q (W) 313.88 240.5 T 2 9 Q (X) 323.55 236.3 T 3 12 Q (\316) 302.33 240.5 T 0 0 612 792 C 144 223.03 149.33 235.71 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 144.5 226.5 T 0 0 612 792 C 172.66 219.86 200.69 235.71 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (f) 173.16 226.5 T (x) 184.36 226.5 T 2 9 Q (t) 190.14 222.3 T 3 12 Q (\050) 179.21 226.5 T (\051) 193.19 226.5 T 0 0 612 792 C 396.33 222.9 420.25 235.71 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 2 12 Q 0 X 0 0 0 1 0 0 0 K (t) 396.74 226.5 T 0 F (1) 412.84 226.5 T (+) 403.07 226.5 T 0 0 612 792 C 427.62 204.99 472.06 221.49 C 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 3 12 Q 0 X 0 0 0 1 0 0 0 K (W) 428.12 212.5 T 2 9 Q (X) 437.8 208.3 T 3 12 Q (W) 455.88 212.5 T 2 9 Q (T) 465.55 208.3 T 3 12 Q (\264) 446.29 212.5 T 0 0 612 792 C 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "2" 30 %%Page: "1" 31 612 792 0 FMBEGINPAGE [0 0 0 1 0 0 0] [ 0 1 1 0 1 0 0] [ 1 0 1 0 0 1 0] [ 1 1 0 0 0 0 1] [ 1 0 0 0 0 1 1] [ 0 1 0 0 1 0 1] [ 0 0 1 0 1 1 0] 7 FrameSetSepColors FrameNoSep 0 0 0 1 0 0 0 K 108 54 540 54 2 L 0.25 H 2 Z 0 X 0 0 0 1 0 0 0 K N 0 8 Q (1) 322 42.62 T 0 0 0 1 0 0 0 K 0 0 0 1 0 0 0 K 0 24 Q (Decision-Theoretic Planning and) 165.36 704 T (Markov Decision Processes) 190.69 674 T 1 12 Q (Thomas Dean) 288.5 634 T 4 F (Brown University) 280.49 616 T 1 F (Abstract) 301.67 580 T 0 F -0.56 (W) 162 559 P -0.56 (e propose Markov decision processes as a basic representation for) 172.37 559 P (planning under uncertainty) 162 545 T (. W) 290.53 545 T (e present standard notation and ter-) 306.9 545 T -0.03 (minology from the literature on the control of stochastic processes.) 162 531 P (W) 162 517 T (e describe a classical framework employing stochastic processes) 172.37 517 T -0.22 (for decision making under uncertainty) 162 503 P -0.22 (, and investigate how the sort) 343.65 503 P (of decomposable representations favored in arti\336cial intelligence) 162 489 T (can be added to the framework to expedite inference. Finally) 162 475 T (, we) 453.14 475 T (survey existing approaches to decision-theoretic planning under) 162 461 T (uncertainty and describe how they relate to this framework. W) 162 447 T (e) 460.97 447 T (claim that Markov decision processes provide the semantic founda-) 162 433 T (tions for planning under uncertainty in much the same way as the) 162 419 T -0.16 (propositional logic and its associated semantics provide the founda-) 162 405 P (tions for more expressive logics.) 162 391 T 1 16 Q (1 Intr) 108 350.33 T (oduction) 151.26 350.33 T 0 12 Q -0.28 (Any useful answer to the question of what is the best representation for planning under un-) 108 327 P 1.33 (certainty has to address the issue of tradeof) 108 313 P 1.33 (fs involving expressivity and computational) 324.03 313 P 0.14 (complexity) 108 299 P 0.14 (. An expressive representation that is computationally impractical is as much a) 161.22 299 P -0.07 (problem as a representation with desirable computational properties that is not suf) 108 285 P -0.07 (\336ciently) 500.66 285 P (expressive for the task at hand.) 108 271 T (It is well known that \336rst-order predicate logic is more expressive than propositional) 108 245 T (logic, but if the domain of interest is \336nite or can be reasonably approximated as \336nite,) 108 231 T (then propositional logic serves as a suf) 108 217 T (\336ciently expressive representation language. As) 293.76 217 T -0.19 (representations of time and change, deterministic \336nite state automata and their stochastic) 108 203 P (counterparts, which we refer to as, respectively) 108 189 T (, deterministic and stochastic processes,) 333.82 189 T (have limited expressivity) 108 175 T (. However) 227.88 175 T (, such processes are far more powerful than generally) 277.38 175 T (given credit in mainstream arti\336cial intelligence.) 108 161 T -0.2 (In addition to their considerable expressive power) 108 135 P -0.2 (, deterministic and stochastic \336nite state) 346.24 135 P (processes help to clarify computational and semantic issues. In particular) 108 121 T (, we claim that) 458.12 121 T (stochastic processes provide the analog of the propositional logic and its associated T) 108 107 T (ar-) 517.1 107 T -0.29 (skian semantics for planning under uncertainty) 108 93 P -0.29 (. Stochastic processes do not solve the qual-) 330.73 93 P 0 0 0 1 0 0 0 K FMENDPAGE %%EndPage: "1" 31 %%Trailer %%BoundingBox: 0 0 612 792 %%PageOrder: Descend %%Pages: 31 %%DocumentFonts: Times-Roman %%+ Times-Bold %%+ Times-Italic %%+ Symbol %%+ Times-BoldItalic %%EOF