Polish空间上的折扣马氏过程量子化策略的渐近优化

数学物理学报 ›› 2022, Vol. 42 ›› Issue (2): 594-604.

Polish空间上的折扣马氏过程量子化策略的渐近优化

吴晓¹,孔荫莹^2,*(),郭圳滨³

¹ 肇庆学院数学与统计学院广东肇庆 526061
² 广东财经大学智能财会管理学院广州 510320
³ 广发证券股份有限公司发展研究中心上海 200120

收稿日期:2021-03-04 出版日期:2022-04-26 发布日期:2022-04-18
通讯作者: 孔荫莹 E-mail:kongcoco@hotmail.com
基金资助:
国家自然科学基金(11961005);中山大学广东省计算科学重点实验室开放基金(2021021);广东省普通高校重点领域(新一代信息技术)基金(2020ZDZX3019);广州市科技计划项目(202102080420)

Asymptotic Optimality of Quantized Stationary Policies in Continuous-Time Markov Decision Processes with Polish Spaces

Xiao Wu¹,Yinying Kong^2,*(),Zhenbin Guo³

¹ School of Mathematics and Statistics, Zhaoqing University, Guangdong Zhaoqing 526061
² School of Intelligence Financial & Accounting Management, Guangdong University of Finance and Economics, Guangzhou 510320
³ Development Research Center, GF Securities Co Ltd, Shanghai 200120

Received:2021-03-04 Online:2022-04-26 Published:2022-04-18
Contact: Yinying Kong E-mail:kongcoco@hotmail.com
Supported by:
the NSFC(11961005);the Opening Project of Guangdong Province Key Laboratory of Computational Science at Sun Yat-sen University(2021021);the Guangdong University (New Generation Information Technology) Key Field Project(2020ZDZX3019);the Guangzhou Science and Technology Plan Project(202102080420)

摘要/Abstract

摘要：

该文研究了Polish空间上、带折扣因子的连续时间马尔可夫决策过程(CTMDPs) 的量子化平稳策略的渐近最优性问题. 首先, 建立了折扣最优方程(DOE) 及其解的存在性和唯一性. 其次, 在适当的条件下证明了最优确定性平稳策略的存在性. 此外, 为了对行动空间进行离散化, 构造了一列量子化策略, 利用有限行动空间的策略来逼近一般(Polish) 空间上的折扣CTMDPs最优平稳策略. 最后, 通过一个例子来说明该文的渐近逼近结果.

关键词: 连续时间马尔可夫决策过程, 依赖状态折扣因子, 折扣准则, 量子化平稳策略, 渐近最优性

Abstract:

In this paper, we study the asymptotic optimality of the quantized stationary policies for continuous-time Markov decision processes (CTMDPs) with Polish space and state-dependent discount factors. Firstly, the existence and uniqueness of the discounted optimal equation (DOE) and its solution are established. Secondly, the existence of the optimal deterministic stationary policies is proved under appropriate conditions. In addition, in order to discretize the action space, a series of quantization policies are constructed to approximate the optimal stationary policies of the discounted CTMDPs in general state (Polish) space by using the policies in finite action space. Finally, an example is given to illustrate the asymptotic approximation results of this paper.

Key words: Continuous-time Markov decision processes, State-dependent discount factors, Discounted criterion, Quantized stationary policies, Asymptotic optimality

中图分类号:

O211.6

吴晓,孔荫莹,郭圳滨. Polish空间上的折扣马氏过程量子化策略的渐近优化[J]. 数学物理学报, 2022, 42(2): 594-604.

Xiao Wu,Yinying Kong,Zhenbin Guo. Asymptotic Optimality of Quantized Stationary Policies in Continuous-Time Markov Decision Processes with Polish Spaces[J]. Acta mathematica scientia,Series A, 2022, 42(2): 594-604.

参考文献 23

1	Journals Doshi B T . Continuous-time control of Markov processes on an arbitrary state space: discounted rewards. Ann Stat, 1976, 4: 1219- 1235
2	Dynkin E B , Yushkevich A A . Controlled Markov Processes. New York: Springer, 1979
3	Feinberg E A . Continuous-time jump Markov decision processes: A discrete-event approach. Math Oper Res, 2004, 29: 492- 524 doi: 10.1287/moor.1040.0089
4	Guo X P . Continuous-time Markov decision processes with discounted rewards: The case of Polish spaces. Math Oper Res, 2007, 32 (1): 73- 87 doi: 10.1287/moor.1060.0210
5	Guo X P . Constrained optimization for average cost continuous-time Markov decision processes. IEEE Trans Autom Control, 2007, 52 (6): 1139- 1143 doi: 10.1109/TAC.2007.899040
6	Guo X P , Hernández-Lerma O . Continuous-time Markov Decision Processes: Theory and Applications. Berlin: Springer, 2009
7	Guo X P , Song X Y . Discounted continuous-time constrained Markov decision processes in Polish spaces. Ann Appl Probab, 2011, 21 (5): 2016- 2049
8	Hernández-Lerma O , Lasserre J B . Discrete-Time Markov Control Processes. New York: Springer, 1996
9	Hernández-Lerma O , Lasserre J B . Further Topics on Discrete-Time Markov Control Processes. New York: Springer, 1999
10	Puterman M L . Markov Decision Processes. New York: Wiley, 1994
11	Feinberg E A , Shwartz A . Markov decision models with weighted discounted criteria. Math Oper Res, 1994, 19 (1): 152- 168 doi: 10.1287/moor.19.1.152
12	González-Hernández J , López-Martínez R R , Pérez-Hernández J R . Markov control processes with randomized discounted cost. Math Methods Oper Res, 2007, 65: 27- 44 doi: 10.1007/s00186-006-0092-2
13	Wu X , Guo X P . First passage optimality and variance minimization of Markov decision processes with varying discount factors. J Appl Probab, 2015, 52 (2): 441- 456 doi: 10.1239/jap/1437658608
14	Wu X , Zhang J Y . Finite approximation of the first passage models for discrete-time Markov decision processes with varying discount factors. Discrete Event Dyn Syst, 2016, 26 (4): 669- 683 doi: 10.1007/s10626-014-0209-3
15	Hinderer K . Foundations of Non Stationary Dynamic Programming with Discrete Time Parameter. New York: Springer, 1970
16	Ye L E , Guo X P . Continuous-time Markov decision processes with state-dependent discount factors. Acta Appl Math, 2012, 121: 5- 27 doi: 10.1007/s10440-012-9669-3
17	Bertsekas D , Tsitsiklis J . Neuro-Dynammic Programming. Boston: Athena Scientific, 1996
18	Cavazos-Cadena R . Finite-state approximations for denumerable state discounted Markov decision processes. Appl Math Optim, 1986, 14: 1- 26 doi: 10.1007/BF01442225
19	Dufour F , Prieto-Rumeau T . Finite linear programming approximations of constrained discounted Markov decision processes. SIAM J Control Optim, 2013, 51 (2): 1298- 1324 doi: 10.1137/120867925
20	Wu X , Guo X P . Convergence of Markov decision processes with constraints and state-action dependent discount factors. Sci China Math, 2020, 63 (1): 167- 182 doi: 10.1007/s11425-017-9292-1
21	Saldi N , Linder T , Yuksel S . Asymptotic optimality and rates of convergence of quantized stationary policies in stochastic control. IEEE Transactions on Automatic Control, 2014, 60 (2): 553- 558
22	Saldi N , Linder T , Yuksel S . Near optimality of quantized policies in stochastic control under weak continuity conditions. J Math Anal Appl, 2016, 435: 321- 337 doi: 10.1016/j.jmaa.2015.10.008
23	Lund R B , Meyn S P , Tweedie R L . Computable exponential convergence rates for stochastically ordered Markov processes. Ann Appl Probab, 1996, 6: 218- 237

[1]	陈密, 聂昌伟, 刘海燕. 一类离散相依索赔风险模型的随机分红问题[J]. 数学物理学报, 2022, 42(2): 631-640.
[2]	高志强. 依时变化的随机环境中的分枝随机游动的局部极限定理的二阶展开[J]. 数学物理学报, 2022, 42(1): 282-305.
[3]	张传洲,李甜甜,焦樊. $B$值弱Orlicz $\alpha$拟鞅空间的原子分解[J]. 数学物理学报, 2022, 42(1): 9-17.
[4]	李江,蓝桂杰,张树文,魏春金. 一类随机葡萄糖-胰岛素模型动力学分析[J]. 数学物理学报, 2021, 41(6): 1937-1949.
[5]	刘丽雅,蒋达清. 具有一般反应函数与贴壁生长现象的随机恒化器模型的全局动力学行为[J]. 数学物理学报, 2021, 41(6): 1912-1924.
[6]	张仲华,张倩. 转换机制下具有非线性扰动的随机SIVS传染病模型的定性分析[J]. 数学物理学报, 2021, 41(4): 1218-1234.
[7]	丁毅,郭精军. 时变混合分数布朗运动下带交易费用的亚式期权定价[J]. 数学物理学报, 2021, 41(4): 1135-1146.
[8]	林运国. 时间非齐次二态量子游荡的演化过程分析[J]. 数学物理学报, 2021, 41(4): 1097-1110.
[9]	杨晓凤,董华,戴洪帅. 混合观测体系下谱负Lévy过程的Parisian破产问题[J]. 数学物理学报, 2021, 41(2): 548-561.
[10]	向绪言,付海琴,周杰明,邓迎春,杨向群. 有环的可逆马氏链的统计确认[J]. 数学物理学报, 2020, 40(6): 1682-1698.
[11]	蒲琳涓,杨晓忠,孙淑珍. 一类分数阶Langevin方程预估校正算法的数值分析[J]. 数学物理学报, 2020, 40(4): 1018-1028.
[12]	桑利恒,陈振龙,郝晓珍. 双分数布朗运动重整化自相交局部时的光滑性[J]. 数学物理学报, 2020, 40(3): 796-810.
[13]	邓迎春,李满,黄娅,周杰明. 非时齐复合Poisson风险模型的破产特征量分析[J]. 数学物理学报, 2020, 40(2): 501-514.
[14]	曹忠威,文香丹,冯微,祖力. 一类具有随机扰动的非自治SIRI流行病模型的动力学行为[J]. 数学物理学报, 2020, 40(1): 221-233.
[15]	冉启康. 带分数Brown运动和局部线性增长的随机微分方程[J]. 数学物理学报, 2020, 40(1): 200-211.