Recursive Markov Process (original) (raw)

Recursive Markov Process for Iterated Games with Markov Strategies

arXiv: Probability, 2015

The dynamics in games involving multiple players, who adaptively learn from their past experience, is not yet well understood. We analyzed a class of stochastic games with Markov strategies in which players choose their actions probabilistically. This class is formulated as a ktextthk^{\text{th}}ktextth order Markov process, in which the probability of choice is a function of kkk past states. With a reasonably large kkk or with the limit ktoinftyk \to \inftyktoinfty, numerical analysis of this random process is unfeasible. This study developed a technique which gives the marginal probability of the stationary distribution of the infinite-order Markov process, which can be constructed recursively. We applied this technique to analyze an iterated prisoner's dilemma game with two players who learn using infinite memory.

A class of self-interacting processes with applications to games and reinforced random walks

SIAM Journal on Control and Optimization, 2010

This paper studies a class of non-Markovian and nonhomogeneous stochastic processes on a finite state space. Relying on a recent paper by Benaïm, Hofbauer, and Sorin [SIAM J. Control Optim., 44 (2005), pp. 328-348] it is shown that, under certain assumptions, the asymptotic behavior of occupation measures can be described in terms of a certain set-valued deterministic dynamical system. This provides a unified approach to simulated annealing type processes and permits the study of new models of vertex reinforced random walks and new models of learning in games such as Markovian fictitious play.

On Markov chains and ltrations

1997

In this paper we rederive some well known results for continuous time Markov processes that live on a nite state space. Martingale techniques are used throughout the paper. Special attention is paid to the construction of a continuous time Markov process, when we start from a discrete time Markov chain. The Markov property here holds with respect to ltrations that need not be minimal.

Sequential games as stochastic processes

Stochastic Processes and their Applications, 1978

This paper examines the stochastic processes generated by sequential games that involve repeated play of a specific game. Such sequential games are viewed as adaptive decision-making processes over time wherein each player updates his "stat? after every play. This revision may involve one's strategy or one's prior distribution on the competitor's strategies. It it shown that results from the theory of discrete time Markov processes can be applied to gain insight into the asymptotic behavior of the game. This is illustrated with a duopoly game in economics. sequential game ergodic theory oligopoly games quasi-compact operators sequential decision making * This paper is based on the author's Ph.D. dissertation submitted to Yale University. The author would like to acknowledge the help given him by Professors M.J. Sobel, W. Whitt, and M. Shubik, and a conversation with Professor S. Kakurani, This research \II as p:lrtially funded by NSF Grant GK-3812 I.

Exact analysis of the M/M/k/setup class of Markov chains via recursive renewal reward

Queueing Systems, 2014

This research was sponsored by the National Science Foundation under grant number NSF-CSR-116282 and Intel Science and Technology Center on Cloud Computing. The views and conclusions contained in this document are those of the author and should not be interpreted as representing the official policies, either expressed or implied, of any sponsoring institution, the U.S. government or any other entity.

A representation result for finite Markov chains

Statistics & Probability Letters, 1998

In this short paper we derive a representation result in terms of a solution to a stochastic differential equation that is valid for both continuous and discrete time Markov processes that live on a finite state space. Martingale techniques are used throughout the paper. @ 1998 Elsevier Science B.V. All rights reserved A MS classification: 60G42; 60G44; 60Jl0; 60J27.

Control problems for markov processes with memory

Cybernetics and Systems Analysis, 1998

Natural ordering of the time argument is often important for the analysis of stochastic processes. Markovian property, martingale, and other notions directly exploit the time-ordering of the argument.

On Markov Chains and Filtrations

1997

In this paper we rederive some well known results for continuous time Markov processes that live on a nite state space. Martingale techniques are used throughout the paper. Special attention is paid to the construction of a continuous time Markov process, when we start from a discrete time Markov c hain. The Markov property here holds with respect to ltrations that need not be minimal.