TY - JOUR
T1 - Observer and control design in partially observable finite Markov chains
AU - Clempner, Julio B.
AU - Poznyak, Alexander S.
N1 - Publisher Copyright:
© 2019 Elsevier Ltd
PY - 2019/12
Y1 - 2019/12
N2 - The controllable Partially Observable Markov Decision Process (POMDP) framework has proven to be useful in different domains where one is constrained to provide incomplete information of the structure and the parameters of the problem. Sometimes, it is not easy to track and measure accurately some state variables, and it may be more effective to make decisions based on imprecise information. This paper is focused on the design of an observer (which is unknown) for a class of ergodic homogeneous finite Markov chains with partially observable states. The main goal of the proposed method is the derivation of formulas for computing an observer, and as a result, on optimal control policy. For solving the problem, we introduce a new variable, which involves the product of the policy, the observation kernel and the distribution vector. We derive the formulas to recover the variables of interest. This work considers a dynamic environment for learning the parameters of the POMDP model. The construction of the adaptive policies is based on an identification approach, where we estimate the elements of the transition matrices and utility matrices by counting the number of unobserved experiences. A numerical example is presented to illustrate the practical implications of the theoretical issues applied to a portfolio optimization problem. These findings are important and new to the literature.
AB - The controllable Partially Observable Markov Decision Process (POMDP) framework has proven to be useful in different domains where one is constrained to provide incomplete information of the structure and the parameters of the problem. Sometimes, it is not easy to track and measure accurately some state variables, and it may be more effective to make decisions based on imprecise information. This paper is focused on the design of an observer (which is unknown) for a class of ergodic homogeneous finite Markov chains with partially observable states. The main goal of the proposed method is the derivation of formulas for computing an observer, and as a result, on optimal control policy. For solving the problem, we introduce a new variable, which involves the product of the policy, the observation kernel and the distribution vector. We derive the formulas to recover the variables of interest. This work considers a dynamic environment for learning the parameters of the POMDP model. The construction of the adaptive policies is based on an identification approach, where we estimate the elements of the transition matrices and utility matrices by counting the number of unobserved experiences. A numerical example is presented to illustrate the practical implications of the theoretical issues applied to a portfolio optimization problem. These findings are important and new to the literature.
KW - Markov chains
KW - Observer design
KW - Optimal control
KW - Partially observed
UR - http://www.scopus.com/inward/record.url?scp=85072566801&partnerID=8YFLogxK
U2 - 10.1016/j.automatica.2019.108587
DO - 10.1016/j.automatica.2019.108587
M3 - Artículo
SN - 0005-1098
VL - 110
JO - Automatica
JF - Automatica
M1 - 108587
ER -