Polish空间上的折扣马氏过程量子化策略的渐近优化

Abstract

Abstract:

In this paper, we study the asymptotic optimality of the quantized stationary policies for continuous-time Markov decision processes (CTMDPs) with Polish space and state-dependent discount factors. Firstly, the existence and uniqueness of the discounted optimal equation (DOE) and its solution are established. Secondly, the existence of the optimal deterministic stationary policies is proved under appropriate conditions. In addition, in order to discretize the action space, a series of quantization policies are constructed to approximate the optimal stationary policies of the discounted CTMDPs in general state (Polish) space by using the policies in finite action space. Finally, an example is given to illustrate the asymptotic approximation results of this paper.

Key words: Continuous-time Markov decision processes, State-dependent discount factors, Discounted criterion, Quantized stationary policies, Asymptotic optimality

CLC Number:

O211.6

Xiao Wu,Yinying Kong,Zhenbin Guo. Asymptotic Optimality of Quantized Stationary Policies in Continuous-Time Markov Decision Processes with Polish Spaces[J].Acta mathematica scientia,Series A, 2022, 42(2): 594-604.

Trendmd

References 23

1	Journals Doshi B T . Continuous-time control of Markov processes on an arbitrary state space: discounted rewards. Ann Stat, 1976, 4: 1219- 1235
2	Dynkin E B , Yushkevich A A . Controlled Markov Processes. New York: Springer, 1979
3	Feinberg E A . Continuous-time jump Markov decision processes: A discrete-event approach. Math Oper Res, 2004, 29: 492- 524 doi: 10.1287/moor.1040.0089
4	Guo X P . Continuous-time Markov decision processes with discounted rewards: The case of Polish spaces. Math Oper Res, 2007, 32 (1): 73- 87 doi: 10.1287/moor.1060.0210
5	Guo X P . Constrained optimization for average cost continuous-time Markov decision processes. IEEE Trans Autom Control, 2007, 52 (6): 1139- 1143 doi: 10.1109/TAC.2007.899040
6	Guo X P , Hernández-Lerma O . Continuous-time Markov Decision Processes: Theory and Applications. Berlin: Springer, 2009
7	Guo X P , Song X Y . Discounted continuous-time constrained Markov decision processes in Polish spaces. Ann Appl Probab, 2011, 21 (5): 2016- 2049
8	Hernández-Lerma O , Lasserre J B . Discrete-Time Markov Control Processes. New York: Springer, 1996
9	Hernández-Lerma O , Lasserre J B . Further Topics on Discrete-Time Markov Control Processes. New York: Springer, 1999
10	Puterman M L . Markov Decision Processes. New York: Wiley, 1994
11	Feinberg E A , Shwartz A . Markov decision models with weighted discounted criteria. Math Oper Res, 1994, 19 (1): 152- 168 doi: 10.1287/moor.19.1.152
12	González-Hernández J , López-Martínez R R , Pérez-Hernández J R . Markov control processes with randomized discounted cost. Math Methods Oper Res, 2007, 65: 27- 44 doi: 10.1007/s00186-006-0092-2
13	Wu X , Guo X P . First passage optimality and variance minimization of Markov decision processes with varying discount factors. J Appl Probab, 2015, 52 (2): 441- 456 doi: 10.1239/jap/1437658608
14	Wu X , Zhang J Y . Finite approximation of the first passage models for discrete-time Markov decision processes with varying discount factors. Discrete Event Dyn Syst, 2016, 26 (4): 669- 683 doi: 10.1007/s10626-014-0209-3
15	Hinderer K . Foundations of Non Stationary Dynamic Programming with Discrete Time Parameter. New York: Springer, 1970
16	Ye L E , Guo X P . Continuous-time Markov decision processes with state-dependent discount factors. Acta Appl Math, 2012, 121: 5- 27 doi: 10.1007/s10440-012-9669-3
17	Bertsekas D , Tsitsiklis J . Neuro-Dynammic Programming. Boston: Athena Scientific, 1996
18	Cavazos-Cadena R . Finite-state approximations for denumerable state discounted Markov decision processes. Appl Math Optim, 1986, 14: 1- 26 doi: 10.1007/BF01442225
19	Dufour F , Prieto-Rumeau T . Finite linear programming approximations of constrained discounted Markov decision processes. SIAM J Control Optim, 2013, 51 (2): 1298- 1324 doi: 10.1137/120867925
20	Wu X , Guo X P . Convergence of Markov decision processes with constraints and state-action dependent discount factors. Sci China Math, 2020, 63 (1): 167- 182 doi: 10.1007/s11425-017-9292-1
21	Saldi N , Linder T , Yuksel S . Asymptotic optimality and rates of convergence of quantized stationary policies in stochastic control. IEEE Transactions on Automatic Control, 2014, 60 (2): 553- 558
22	Saldi N , Linder T , Yuksel S . Near optimality of quantized policies in stochastic control under weak continuity conditions. J Math Anal Appl, 2016, 435: 321- 337 doi: 10.1016/j.jmaa.2015.10.008
23	Lund R B , Meyn S P , Tweedie R L . Computable exponential convergence rates for stochastically ordered Markov processes. Ann Appl Probab, 1996, 6: 218- 237

[1]	Chen Mi, Nie Changwei, Liu Haiyan. Randomized Dividends in a Discrete Risk Model with Time-Correlated Claims [J]. Acta mathematica scientia,Series A, 2022, 42(2): 631-640.
[2]	Zhiqiang Gao. A Second Order Correction of the Local Limit Theorem for a Branching Random Walk with a Random Environment in Time on ${\mathbb{Z}}^d$ [J]. Acta mathematica scientia,Series A, 2022, 42(1): 282-305.
[3]	Chuanzhou Zhang,Tiantian Li,Fan Jiao. Atomic Aecompositions of $B$-Valued Weak Orlicz $\alpha$-Quasi-Martingale Spaces [J]. Acta mathematica scientia,Series A, 2022, 42(1): 9-17.
[4]	Jiang Li,Guijie Lan,Shuwen Zhang,Chunjin Wei. Dynamics Analysis of a Stochastic Glucose-Insulin Model [J]. Acta mathematica scientia,Series A, 2021, 41(6): 1937-1949.
[5]	Liya Liu,Daqing Jiang. Global Dynamics of a Stochastic Chemostat Model with General Response Function and Wall Growth [J]. Acta mathematica scientia,Series A, 2021, 41(6): 1912-1924.
[6]	Zhonghua Zhang,Qian Zhang. Qualitative Analysis of a Stochastic SIVS Epidemic Model with Nonlinear Perturbations Under Regime Switching [J]. Acta mathematica scientia,Series A, 2021, 41(4): 1218-1234.
[7]	Yi Ding,Jingjun Guo. Pricing Asian Options Under Time-Changed Mixed Fractional Brownian Motion with Transactions Costs [J]. Acta mathematica scientia,Series A, 2021, 41(4): 1135-1146.
[8]	Yunguo Lin. The Analysis of Evolution Process in a Time-Inhomogeneous Two-State Quantum Walk [J]. Acta mathematica scientia,Series A, 2021, 41(4): 1097-1110.
[9]	Xiaofeng Yang,Hua Dong,Hongshuai Dai. Parisian Ruin for Spectrally Negative Lévy Processes Under a Hybrid Observation Scheme [J]. Acta mathematica scientia,Series A, 2021, 41(2): 548-561.
[10]	Xuyan Xiang,Haiqin Fu,Jieming Zhou,Yingchun Deng,Xiangqun Yang. Statistical Identification of Reversible Markov Chain on Cyclic Graph [J]. Acta mathematica scientia,Series A, 2020, 40(6): 1682-1698.
[11]	Linjuan Pu,Xiaozhong Yang,Shuzhen Sun. Numerical Analysis of a Class of Fractional Langevin Equation by Predictor-Corrector Method [J]. Acta mathematica scientia,Series A, 2020, 40(4): 1018-1028.
[12]	Liheng Sang,Zhenlong Chen,Xiaozhen Hao. Smoothness for the Renormalized Self-Intersection Local Time of Bifractional Brownian Motion [J]. Acta mathematica scientia,Series A, 2020, 40(3): 796-810.
[13]	Yingchun Deng,Man Li,Ya Huang,Jieming Zhou. On the Analysis of Ruin-Related Quantities in the Nonhomogeneous Compound Poisson Risk Model [J]. Acta mathematica scientia,Series A, 2020, 40(2): 501-514.
[14]	Zhongwei Cao,Xiangdan Wen,Wei Feng,Li Zu. Dynamics of a Nonautonomous SIRI Epidemic Model with Random Perturbations [J]. Acta mathematica scientia,Series A, 2020, 40(1): 221-233.
[15]	Qikang Ran. SDE Driven by Fractional Brown Motion and Their Coefficients are Locally Linear Growth [J]. Acta mathematica scientia,Series A, 2020, 40(1): 200-211.

Asymptotic Optimality of Quantized Stationary Policies in Continuous-Time Markov Decision Processes with Polish Spaces

RichHTML

PDF (PC)

Abstract

Cite this article

share this article

References 23

Related Articles 15

Metrics

Comments

Recommended 0