Stochastic Controls: Hamiltonian Systems and HJB Equations

1. Basic Stochastic Calculus.- 1. Probability.- 1.1. Probability spaces.- 1.2. Random variables.- 1.3. Conditional expectation.- 1.4. Convergence of probabilities.- 2. Stochastic Processes.- 2.1. General considerations.- 2.2. Brownian motions.- 3. Stopping Times.- 4. Martingales.- 5. Itô's Integral.- 5.1. Nondifferentiability of Brownian motion.- 5.2. Definition of Itô's integral and basic properties.- 5.3. Itô's formula.- 5.4. Martingale representation theorems.- 6. Stochastic Differential Equations.- 6.1. Strong solutions.- 6.2. Weak solutions.- 6.3. Linear SDEs.- 6.4. Other types of SDEs.- 2. Stochastic Optimal Control Problems.- 1. Introduction.- 2. Deterministic Cases Revisited.- 3. Examples of Stochastic Control Problems.- 3.1. Production planning.- 3.2. Investment vs. consumption.- 3.3. Reinsurance and dividend management.- 3.4. Technology diffusion.- 3.5. Queueing systems in heavy traffic.- 4. Formulations of Stochastic Optimal Control Problems.- 4.1. Strong formulation.- 4.2. Weak formulation.- 5. Existence of Optimal Controls.- 5.1. A deterministic result.- 5.2. Existence under strong formulation.- 5.3. Existence under weak formulation.- 6. Reachable Sets of Stochastic Control Systems.- 6.1. Nonconvexity of the reachable sets.- 6.2. Noncloseness of the reachable sets.- 7. Other Stochastic Control Models.- 7.1. Random duration.- 7.2. Optimal stopping.- 7.3. Singular and impulse controls.- 7.4. Risk-sensitive controls.- 7.5. Ergodic controls.- 7.6. Partially observable systems.- 8. Historical Remarks.- 3. Maximum Principle and Stochastic Hamiltonian Systems.- 1. Introduction.- 2. The Deterministic Case Revisited.- 3. Statement of the Stochastic Maximum Principle.- 3.1. Adjoint equations.- 3.2. The maximum principle and stochastic Hamiltonian systems.- 3.3. A worked-out example.- 4. A Proof of the Maximum Principle.- 4.1. A moment estimate.- 4.2. Taylor expansions.- 4.3. Duality analysis and completion of the proof.- 5. Sufficient Conditions of Optimality.- 6. Problems with State Constraints.- 6.1. Formulation of the problem and the maximum principle.- 6.2. Some preliminary lemmas.- 6.3. A proof of Theorem 6.1.- 7. Historical Remarks.- 4. Dynamic Programming and HJB Equations.- 1. Introduction.- 2. The Deterministic Case Revisited.- 3. The Stochastic Principle of Optimality and the HJB Equation.- 3.1. A stochastic framework for dynamic programming.- 3.2. Principle of optimality.- 3.3. The HJB equation.- 4. Other Properties of the Value Function.- 4.1. Continuous dependence on parameters.- 4.2. Semiconcavity.- 5. Viscosity Solutions.- 5.1. Definitions.- 5.2. Some properties.- 6. Uniqueness of Viscosity Solutions.- 6.1. A uniqueness theorem.- 6.2. Proofs of Lemmas 6.6 and 6.7.- 7. Historical Remarks.- 5. The Relationship Between the Maximum Principle and Dynamic Programming.- 1. Introduction.- 2. Classical Hamilton-Jacobi Theory.- 3. Relationship for Deterministic Systems.- 3.1. Adjoint variable and value function: Smooth case.- 3.2. Economic interpretation.- 3.3. Methods of characteristics and the Feynman-Kac formula.- 3.4. Adjoint variable and value function: Nonsmooth case.- 3.5. Verification theorems.- 4. Relationship for Stochastic Systems.- 4.1. Smooth case.- 4.2. Nonsmooth case: Differentials in the spatial variable.- 4.3. Nonsmooth case: Differentials in the time variable.- 5. Stochastic Verification Theorems.- 5.1. Smooth case.- 5.2. Nonsmooth case.- 6. Optimal Feedback Controls.- 7. Historical Remarks.- 6. Linear Quadratic Optimal Control Problems.- 1. Introduction.- 2. The Deterministic LQ Problems Revisited.- 2.1. Formulation.- 2.2. A minimization problem of a quadratic functional.- 2.3. A linear Hamiltonian system.- 2.4. The Riccati equation and feedback optimal control.- 3. Formulation of Stochastic LQ Problems.- 3.1. Statement of the problems.- 3.2. Examples.- 4. Finiteness and Solvability.- 5. A Necessary Condition and a Hamiltonian System.- 6. Stochastic Riccati Equations.- 7. Global Solvability of St