Principle of Optimality Continuous Time Process

Alexander S. Poznyak , in Advanced Mathematical Tools for Automatic Control Engineers: Deterministic Techniques, Volume 1, 2008

22.7 Dynamic programing

The dynamic programing method is another powerful approach to solving optimal control problems. It provides sufficient conditions for testing if some control is optimal or not. The basic idea of this approach consists of considering a family of optimal control problems with different initial conditions (times and states) and of obtaining some relationships among them via the, so-called, Hamilton-Jacoby-Bellman equation (HJB) which is a nonlinear first-order partial differential equation. The optimal control can be designed by maximization (or minimization) of the generalized Hamiltonian involved in this equation. If this HJB equation is solvable (analytically or even numerically) then the corresponding optimal controllers turn out to be given by a nonlinear feedback depending on the optimized plant nonlinearity as well as the solution of the corresponding HJB equation. Such approach actually provides the solutions to the whole family of optimization problems, and, in particular, to the original problem. Such a technique is called "invariant embedding". The major drawback of the classical HJB method is that it requires that this partial differential equation admits a smooth enough solution. Unfortunately this is not the case even for some very simple situations. To overcome this problem the so-called viscosity solutions have been introduced (Crandall & Lions 1983). These solutions are some sort of nonsmooth solutions with a key function to replace the conventional derivatives by set-valued super/sub-differentials maintaining the uniqueness of solutions under very mild conditions. These approaches not only save the DPM as a mathematical method, but make it a powerful tool in tackling optimal control. In this section we do not touch on this approach. But we will discuss the gap between necessary (MP) and sufficient (DPM) conditions.

22.7.1 Bellman's principle of optimality

Claim 22.1. (Bellman's principle (BP) of optimality) "Any tail of an optimal trajectory is optimal too."

²

In other words, if some trajectory in the phase space connects the initial $x\left(0\right)$ and terminal $x\left(T\right)$ points and is optimal in the sense of some cost functional, then the sub-trajectory, connecting any intermediate point $x\left(t^{\text{'}}\right)$ of the same trajectory with the same terminal point $x\left(T\right)$ , should also be optimal (see Fig. 22.2).

Fig. 22.2. Illustration of Bellman's principle of optimality.

22.7.2 Sufficient conditions for BP fulfilling

Theorem 22.16. (Sufficient condition for BP fulfilling)

Let

1.

the performance index (a cost functional) $J\left(u(\cdot )\right)$ with $u(\cdot )\in {\mathcal{U}}_{admis}\left[0,T\right]$ be separable for any time $t^{\text{'}}\in \left(0,T\right)$ such that

(22.115) $J\left(u(\cdot )\right)=J_1\left(\begin{array}{cc}{u}_1(\cdot ),& J_2\left(u_2(\cdot )\right)\end{array}\right)$

where $u_1(\cdot )$ is the control within the time interval $\left[0,t^{\text{'}}\right)$ called the initial control strategy and $u_2(\cdot )$ is the control within the time interval $\left[t^{\text{'}},T\right]$ called the terminal control strategy;

2.

the functional $J_1\left(u_1(\cdot ),J_2\left(u_2(\cdot )\right)\right)$ is monotonically nondecreasing with respect to its second argument $J_2\left(u_2(\cdot )\right)$ , that is,

(22.116) $\begin{array}{l}{J}_1\left(u_1(\cdot ),J_2\left(u_2(\cdot )\right)\right)\ge J_1\left(u_1(\cdot ),J_2\left(u_2^{\text{'}}(\cdot )\right)\right)\hfill \\ \begin{array}{cc}if& J_2\left(u_2(\cdot )\right)\ge J_2\left(u_2^{\text{'}}(\cdot )\right)\end{array}\hfill \end{array}$

Then Bellman's principle of optimality takes place for this functional.

Proof

For any admissible control strategies $u_1(\cdot ),u_2(\cdot )$ the following inequality holds

(22.117) $\begin{array}{ll}J*\hfill & :=\underset{u\in {\mathcal{U}}_{admis}\left[0,T\right]}{\inf }J\left(u(\cdot )\right)\hfill \\ \hfill & =\underset{u_1\in {\mathcal{U}}_{admis}\left[0,t^{\text{'}}\right]u_2\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_1\left(u_1(\cdot ),J_2\left(u_2(\cdot )\right)\right)\hfill \\ \hfill & \le J_1\left(u_1(\cdot ),J_2\left(u_2(\cdot )\right)\right)\hfill \end{array}$

Select

(22.118) $u_2(\cdot )=\arg \underset{u_2\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_2\left(u_2(\cdot )\right)$

Then (22.117) and (22.118) imply

(22.119) $J*\le J_1\left(u_1(\cdot )=\underset{u_2\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_2\left(u_2(\cdot )\right)\right)$

So,

(22.120) $u_1(\cdot )=\arg \underset{u_1\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_1\left(u_1(\cdot ),\underset{u_2\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_2\left(u_2(\cdot )\right)\right)$

leads to

(22.121) $J*\le \underset{u_1\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_1\left(u_1(\cdot ),\underset{u_2\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_2\left(u_2(\cdot )\right)\right)$

Since $J_1\left(\begin{array}{cc}{u}_1(\cdot ),& J_2\left(u_2(\cdot )\right)\end{array}\right)$ is monotonically nondecreasing with respect to the second argument, from (22.121) we obtain

(22.122) $\begin{array}{l}\underset{u_1\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_1\left(u_1(\cdot ),\underset{u_2\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_2\left(u_2(\cdot )\right)\right)\hfill \\ \text{undefined}\le \underset{u_1\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }\underset{u_2\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_1\left(u_1(\cdot ),J_2\left(u_2(\cdot )\right)\right)\hfill \\ \text{undefined}=\underset{u_1\in {\mathcal{U}}_{admis}\left[0,T\right]}{\inf }J\left(u(\cdot )\right)=J*\hfill \end{array}$

Combining (22.121) and (22.122), we finally derive that

(22.123) $J*=\underset{u_1\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_1\left(u_1(\cdot ),\underset{u_2\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_2\left(u_2(\cdot )\right)\right)$

This proves the desired result.

Summary 22.3

In strict mathematical form this fact may be expressed as follows: under the assumptions of the theorem above for any time $t^{\text{'}}\in \left(0,T\right)$

(22.124) $\begin{array}{l}\underset{u\in {\mathcal{U}}_{admis}\left[0,T\right]}{\inf }J\left(u(\cdot )\right)\hfill \\ \text{undefined}=\underset{u_1\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_1\left(u_1(\cdot ),\underset{u_2\in {\mathcal{U}}_{admis}\left[t^{\text{'}},T\right]}{\inf }{J}_2\left(u_2(\cdot )\right)\right)\hfill \end{array}$

Corollary 22.7

For the cost functional

$J\left(u(\cdot )\right):=h_0\left(x\left(T\right)\right)+{\displaystyle \underset{t=0}{\overset{T}{\int }}h\left(x\left(t\right),u\left(t\right),t\right)}dt$

given in the Bolza form (22.41) Bellman's principle holds.

Proof

ZFor any $t^{\text{'}}\in \left(0,T\right)$ from (22.41) obviously it follows that

(22.125) $J\left(u(\cdot )\right)=J_1\left(u_1(\cdot )\right)+J_2\left(u_2(\cdot )\right)$

where

(22.126) $\begin{array}{l}{J}_1\left(u_1(\cdot )\right):={\displaystyle \underset{t=0}{\overset{t^{\text{'}}}{\int }}h\left(x\left(t\right),u_1\left(t\right),t\right)dt}\hfill \\ J_2\left(u_2(\cdot )\right):=h_0\left(x\left(T\right)\right)+{\displaystyle \underset{t=t^{\text{'}}}{\overset{T}{\int }}h\left(x\left(t\right),u_1\left(t\right),t\right)dt}\hfill \end{array}$

The representation (22.125) evidently yields the validity (22.115) and (22.116) for this functional.

22.7.3 Invariant embedding

22.7.3.1 System description and basic assumptionsa

Let $\left(s,y\right)\in \left[0,T\right)\times {ℝ}^n$ be "an initial time and state pair" to the following controlled system over $\left[s,T\right]$ :

(22.127) $\begin{array}{l}\begin{array}{ccc}\dot{x}\left(t\right)=f\left(x\left(t\right),u\left(t\right),t\right),& a.e.& t\in \left[s,T\right]\end{array}\hfill \\ x\left(s\right)=y\hfill \end{array}\}$

where $x\in {ℝ}^n$ is its state vector, and $u\in {ℝ}^r$ is the control that may run over a given control region $U\subset {ℝ}^r$ with the cost functional in the Bolza form

(22.128) $J\left(s,y;u(\cdot )\right):=h_0\left(x\left(T\right)\right)+{\displaystyle \underset{t=s}{\overset{T}{\int }}h\left(x\left(t\right),u\left(t\right),t\right)dt}$

containing the integral term as well as the terminal one and with the terminal set $ℳ\subseteq {ℝ}^n$ given by the inequalities (22.42). Here, as before, $u(\cdot )\in {\mathcal{U}}_{admis}\left[s,T\right]$ . For $s=0$ and $y=x_0$ this plant coincides with the original one given by (22.40).

Suppose also that assumption (A1) is accepted and, instead of (A2), its small modification holds:

(A2^′) The maps

(22.129) $\begin{array}{l}f:{ℝ}^n\times U\times \left[0,T\right]\to {ℝ}^n\hfill \\ h:{ℝ}^n\times U\times \left[0,T\right]\to {ℝ}^n\hfill \\ h_0:{ℝ}^n\times U\times \left[0,T\right]\to {ℝ}^n\hfill \\ g_l:{ℝ}^n\to {ℝ}^n\left(l=1,\dots ,L\right)\hfill \end{array}\}$

are uniformly continuous in (x, u, t) including t (before in (A2) they were assumed to be only measurable) and there exists a constant L such that for $\phi =f\left(x,u,t\right),h\left(x,u,t\right),h_0\left(x,u,t\right),g_l\left(x\right)\left(l=1,\dots ,L\right)$ the following inequalities hold:

(22.130) $\begin{array}{l}\Vert \phi \left(x,u,t\right)-\phi \left(\widehat{x},\widehat{u},t\right)\Vert \le L\Vert x-\widehat{x}\Vert \hfill \\ \forall t\in \left[0,T\right],x,\widehat{x}\in {ℝ}^n,u\in U\hfill \\ \Vert \phi \left(0,u,t\right)\Vert \le L\forall u,t\in U\times \left[0,T\right]\hfill \end{array}\}$

It is evident that under assumptions (A1)–(A2^′) for any $\left(s,y\right)\in \left[0,T\right)\times {ℝ}^n$ and any $u(\cdot )\in {\mathcal{U}}_{admis}\left[s,T\right]$ the optimization problem

(22.131) $J\left(s,y;u(\cdot )\right)\to \underset{u(\cdot )\in {\mathcal{U}}_{admis}\left[s,T\right]}{\min }$

formulated for the plant (22.127) and for the cost functional $J\left(s,y;u(\cdot )\right)$ (22.128), admits a unique solution $x(\cdot ):=x\left(\cdot ,s,y;u(\cdot )\right)$ and the functional (22.128) is well defined.

Definition 22.8. (The value function)

The function $V\left(s,y\right)$ defined for any $\left(s,y\right)\in \left[0,T\right)\times {ℝ}^n$

(22.132) $\begin{array}{l}V\left(s,y\right):=\underset{u(\cdot )\in {\mathcal{U}}_{admis}\left[s,T\right]}{\inf }J\left(s,y;u(\cdot )\right)\hfill \\ V\left(T,y\right)=h_0\left(y\right)\hfill \end{array}\}$

is called the value function of the optimization problem (22.131).

22.7.3.2 Dynamic programing equation in the integral form

Theorem 22.17

Under assumptions (A1)–(A2^′)for any $\left(s,y\right)\in \left[0,T\right)\times {ℝ}^n$ the following relation holds

(22.133) $\begin{array}{ll}V\left(s,y\right)\hfill & =\underset{u(\cdot )\in {\mathcal{U}}_{admis}\left[s,T\right]}{\inf }\{\underset{t=s}{\overset{\widehat{s}}{\int }}h\left(x\left(t,s,y,u(\cdot )\right),u\left(t\right),t\right)dt\hfill \\ \hfill & \begin{array}{cc}\text{undefined}+V\left(\widehat{s},x\left(\widehat{s},s,y,u(\cdot )\right)\right)\}& \forall \widehat{s}\in \left[0,T\right]\end{array}\hfill \end{array}$

Proof

The result follows directly from BP of optimality (22.124), but, in view of the great importance of this result, we present the proof again, using the concrete form of the Bolza cost functional (22.128). Denoting the right-hand side of (22.133) by $\overline{V}\left(s,y\right)$ and taking into account the definition (22.132), for any $u(\cdot )\in {\mathcal{U}}_{admis}\left[s,T\right]$ we have

$\begin{array}{l}V\left(s,y\right)\le J\left(s,y;u(\cdot )\right)\hfill \\ \text{undefined}={\displaystyle \underset{t=s}{\overset{\widehat{s}}{\int }}h\left(x\left(t,s,y,u(\cdot )\right),u\left(t\right),t\right)dt+J\left(\widehat{s},x\left(\widehat{s}\right);u(\cdot )\right)}\hfill \end{array}$

and, taking infimum over $u(\cdot )\in {\mathcal{U}}_{admis}\left[s,T\right]$ , it follows that

Hence, for any $\epsilon >0$ there exists a control $u_{\epsilon }(\cdot )\in {\mathcal{U}}_{admis}\left[s,T\right]$ such that for $x(\cdot ):=x\left(\cdot ,s,y;u_{\epsilon }(\cdot )\right)$

(22.135) $\begin{array}{l}V\left(s,y\right)+\epsilon \ge J\left(s,y;u_{\epsilon }(\cdot )\right)\hfill \\ \text{undefined}\ge {\displaystyle \underset{t=s}{\overset{\widehat{s}}{\int }}h\left(x\left(t,s,y,u_{\epsilon }(\cdot )\right),u_{\epsilon }\left(t\right),t\right)dt+V\left(\widehat{s},x_{\epsilon }\left(\widehat{s}\right)\right)\ge V\left(s,y\right)}\hfill \end{array}$

Tending $\epsilon \to 0$ the inequalities (22.134), (22.135) imply the result (22.133) of this theorem.

Finding a solution $V\left(s,y\right)$ to equation (22.133), we would be able to solve the origin optimal control problem putting $s=0$ and $y=x_0$ . Unfortunately, this equation is very difficult to handle because of overcomplicated operations involved on its right-hand side. That's why in the next subsection we will explore this equation further, trying to get another equation for the function $V\left(s,y\right)$ with a simpler and more practically used form.

22.7.4 Hamilton–Jacoby–Bellman equation

To simplify the sequent calculations and following Young & Zhou (1999) we will consider the original optimization problem without any terminal set, that is, $ℳ={ℝ}^n$ . This may be expressed with the constraint function equal to

(22.136) $\begin{array}{cc}g\left(x\right):=0\cdot {\Vert x\Vert }^2-\epsilon \le 0& \left(\epsilon >0\right)\end{array}$

which is true for any $x\in {ℝ}^n$ . Slater's condition (21.88) is evidently valid (also for any $x\in {ℝ}^n$ ). So, we deal here with the regular case. Denote by $C^1\left(\left[0,T\right)\times {ℝ}^n\right)$ the set of all continuously differentiable functions $v:[0,T)\times {ℝ}^n\to ℝ$ .

Theorem 22.18. (The HJB equation)

Suppose that under assumptions (A1)–(A2′) the value function $\upsilon :\left[0,T\right)\times {ℝ}^n\to ℝ$ (22.132) is continuously differentiable, that is $V\in C^1\left(\left[0,T\right)\times {ℝ}^n\right)$ . Then $V\left(s,y\right)$ is a solution to the following terminal value problem of a first-order partial differential equation, named below the Hamilton–Jacoby–Bellman (HJB) equation associated with the original optimization problem (22.131) without terminal set $\left(ℳ={ℝ}^n\right)$ :

(22.137) $\begin{array}{c}-\frac{\partial }{\partial t}V\left(t,x\right)+\underset{u\in U}{\sup }H\left(-\frac{\partial }{\partial t}V\left(t,x\right),x\left(t\right),x\left(t\right),u\left(t\right),t\right)=0\\ \begin{array}{ccc}\left(t,x\right)\in \left[0,T\right)\times {ℝ}^n,& V\left(T,x\right)=h_0\left(x\right)& x\in {ℝ}^n\end{array}\end{array}\}$

where

(22.138) $\begin{array}{c}H\left(\psi ,x,u,t\right):={\psi }^{T}f\left(x,u,t\right)-h\left(x\left(t\right),u\left(t\right),t\right)\\ x,u,t,\psi \in \left[0,T\right]\times {ℝ}^n\times {ℝ}^r\times {ℝ}^n\end{array}$

is the same as in (22.81) with $\mu =1$ corresponding to the regular optimization problem.

Proof

Fixing $u\left(t\right)\equiv u\in U,$ , by (22.133) with $\widehat{s}\downarrow s$ we obtain

$\begin{array}{l}\frac{V\left(s,y\right)-V\left(\widehat{s},x\left(\widehat{s},s,y,u(\cdot )\right)\right)}{\widehat{s}-s}\hfill \\ -\frac{1}{\widehat{s}-s}{\displaystyle \underset{t=s}{\overset{\widehat{s}}{\int }}h\left(x\left(t,s,y,u(\cdot )\right),u\left(t\right),t\right)dt\le 0}\hfill \end{array}$

which implies

$-\frac{\partial }{\partial t}V\left(s,y\right)-\frac{\partial }{\partial t}V{\left(s,y\right)}^{T}f\left(s,y,u\right)-h\left(s,y,u\right)\le 0$

resulting in the following inequality

(22.139) $0\ge -\frac{\partial }{\partial t}V\left(s,y\right)+\underset{u\in U}{\sup }H\left(-\frac{\partial }{\partial t}V\left(t,x\right),x\left(t\right),u\left(t\right),t\right)$

On the other hand, for any $\epsilon >0$ and s, closed to $\widehat{s}$ , there exists a control $u(\cdot ):=u_{\epsilon ,\widehat{s}}(\cdot )\in {\mathcal{U}}_{\text{admis}}\left[s,T\right]$ for which

(22.140) $V\left(s,y\right)+\epsilon \left(\widehat{s}-s\right)\ge {\displaystyle \underset{t=s}{\overset{\widehat{s}}{\int }}h\left(x\left(t,s,y,u(\cdot )\right),u\left(t\right),t\right)dt+V\left(\widehat{s},x\left(\widehat{s}\right)\right)}$

Since $V\in C^1\left(\left[0,T\right)\times {ℝ}^n\right)$ , the last inequality leads to the following

(22.141) $\begin{array}{l}-\epsilon \le -\frac{V\left(\widehat{s},x\left(\widehat{s}\right)\right)-V\left(s,y\right)}{\widehat{s}-s}-\frac{1}{\widehat{s}-s}{\displaystyle \underset{t=s}{\overset{\widehat{s}}{\int }}h\left(x\left(t,s,y,u(\cdot )\right),u\left(t\right),t\right)dt}\hfill \\ \text{undefined}=\frac{1}{\widehat{s}-s}{\displaystyle \underset{t=s}{\overset{\widehat{s}}{\int }}[-\frac{\partial }{\partial t}V\left(t,x\left(t,s,y,u(\cdot )\right)\right)}\hfill \\ \text{undefined}-\frac{\partial }{\partial t}V{\left(t,x\left(t,s,y,u(\cdot )\right)\right)}^{T}f\left(t,x\left(t,s,y,u(\cdot )\right),u\right)\hfill \\ \text{undefined}-h\left(x\left(t,s,y,u(\cdot )\right),u\left(t\right),t\right)]dt\hfill \\ =\frac{1}{\widehat{s}-s}{\displaystyle \underset{t=s}{\overset{\widehat{s}}{\int }}[-\frac{\partial }{\partial t}V\left(t,x\left(t,s,y,u(\cdot )\right)\right)}\hfill \\ \text{undefined}+H\left(\frac{\partial }{\partial t}V\left(t,x\left(t,s,y,u(\cdot )\right)\right),x\left(t,s,y,u(\cdot )\right),u\left(t\right),t\right)]dt\hfill \\ \text{undefined}\le \frac{1}{\widehat{s}-s}{\displaystyle \underset{t=s}{\overset{\widehat{s}}{\int }}[-\frac{\partial }{\partial t}V\left(t,x\left(t,s,y,u(\cdot )\right)\right)}\hfill \\ \text{undefined}+\underset{u\in U}{\sup }H\left(\frac{\partial }{\partial t}V\left(t,x\left(t,s,y,u(\cdot )\right)\right),x\left(t,s,y,u(\cdot )\right),u\left(t\right),t\right)]dt\hfill \end{array}$

which for $\widehat{s}\downarrow s$ gives

(22.142) $-\epsilon \le -\frac{\partial }{\partial t}V\left(s,y\right)+\underset{u\in U}{\sup }H\left(-\frac{\partial }{\partial t}V\left(s,y\right)y,u,s\right)$

Here the uniform continuity property of the functions f and h has been used, namely,

(22.143) $\begin{array}{cc}\underset{t\downarrow s}{\lim }\text{undefined}\underset{y\in {ℝ}^n,u\in U}{\sup }\Vert \phi \left(t,y,u\right)-\phi \left(s,y,u\right)\Vert =0,& \phi =f,h\end{array}$

Combining (22.139) and (22.142) when $\epsilon \to 0$ we obtain (22.137).

The theorem below, representing the sufficient conditions of optimality, is known as the verification rule.

Theorem 22.19.(The verification rule)

Accept the following assumptions:

1.

Let $u*(\cdot ):=u*\left(t,x,\frac{\partial }{\partial x}V\left(t,x\right)\right)$ be a solution to the following optimization problem

(22.144) $H\left(-\frac{\partial }{\partial t}V\left(t,x\right)x,u,t\right)\to \underset{u\in U}{\sup }$

with fixed values x, t and $\frac{\partial }{\partial x}V\left(t,x\right)$ ;

2.

Suppose that we can obtain the solution $V\left(t,x\right)$ to the HJB equation

(22.145) $\begin{array}{l}-\frac{\partial }{\partial t}V\left(t,x\right)+H\left(-\frac{\partial }{\partial t}V\left(t,x\right)x,u*(\cdot ),t\right)=0\hfill \\ \begin{array}{cc}V\left(T,x\right)=h_0\left(x\right)& \left(t,x\right)\in \left[0,T\right)\times {ℝ}^n\end{array}\hfill \end{array}\}$

which for any $\left(t,x\right)\in \left[0,T\right)\times {ℝ}^n$ is unique and smooth, that is, $\left(s,x\right)\in C^1\left(\left[0,T\right)\times {ℝ}^n\right)$ ;

3.

Suppose that for any $\left(s,x\right)\in \left[0,T\right)\times {ℝ}^n$ there exists $\left(a.e.t\in \left[s,T\right]\right)$ a solution $x*\left(s,x\right)$ to the following ODE (ordinary differential equation)

(22.146) $\begin{array}{c}{\dot{x}}^{*}\left(t\right)=f\left(x*\left(t\right),u*\left(t,x*\left(t\right),\frac{\partial }{\partial t}V\left(t,x*\left(t\right)\right)\right),t\right)\\ x*\left(s\right)=x\end{array}$

Then with $\left(s,x\right)=\left(0,x_0\right)$ the pair

(22.147) $\left(x*\left(t\right),u*\left(t,x*\left(t\right),\frac{\partial }{\partial t}V\left(t,x*\left(t\right)\right)\right)\right)$

is optimal, that is, $u*\left(t,x*\left(t\right),\frac{\partial }{\partial x}V\left(t,x*\left(t\right)\right)\right)$ is an optimal control.

Proof

The relations (22.138) and (22.145) imply

(22.148) $\begin{array}{l}\frac{d}{dt}V\left(t,x*\left(t\right)\right)=-\frac{\partial }{\partial t}V\left(t,x*\left(t\right)\right)\hfill \\ \text{undefined}+\frac{\partial }{\partial t}V{\left(t,x*\left(t\right)\right)}^{T}f\left(x*\left(t\right),u*\left(t,x*\left(t\right),\frac{\partial }{\partial t}V\left(t,x*\left(t\right)\right)\right),t\right)\hfill \\ =-h\left(x*\left(t\right),u*\left(t,x*\left(t\right),\frac{\partial }{\partial t}V\left(t,x*\left(t\right)\right)\right),t\right)\hfill \end{array}$

Integrating this equality by t within [s, T] leads to the following relation

$\begin{array}{l}V\left(T,x*\left(T\right)\right)-V\left(s,x*\left(s\right)\right)\hfill \\ =-{\displaystyle \underset{t=s}{\overset{T}{\int }}h\left(x*\left(t\right),u*\left(t,x*\left(t\right),\frac{\partial }{\partial t}V\left(t,x*\left(t\right)\right)\right),t\right)dt}\hfill \end{array}$

which, in view of the identity $V\left(T,x\left(T\right)\right)=h_0\left(x^{*}\left(T\right)\right)$ , is equal to the following one

(22.149) $\begin{array}{ll}V\hfill & \left(s,x^{*}\left(s\right)\right)=h_0\left(x^{*}\left(T\right)\right)\hfill \\ \hfill & +{\displaystyle \underset{t=s}{\overset{T}{\int }}h\left(x^{*}\left(t\right),u^{*}\left(t,x^{*}\left(t\right),\frac{\partial }{\partial x}\left(t,x^{*}\left(t\right)\right)\right),t\right)dt}\hfill \end{array}$

By (22.133), this last equation means exactly that

$\left(x^{*}\left(t\right),u^{*}\left(t,x^{*}\left(t\right),\frac{\partial }{\partial x}V\left(t,x^{*}\left(t\right)\right)\right)\right)$

is an optimal pair and $u^{*}\left(t,x^{*}\left(t\right),\frac{\partial }{\partial x}V\left(t,x^{*}\left(t\right)\right)\right)$ is an optimal control.

hernandezallould.blogspot.com

Source: https://www.sciencedirect.com/topics/engineering/principle-of-optimality#:~:text=2Bellman's%20principle%20of%20optimality%2C%20formulated,resulting%20from%20the%20first%20decision.%E2%80%9D