Markov Decision Processes: Discrete Stochastic Dynamic Programming (Wiley Series in Probability and Statistics)

Book Description

The Wiley-Interscience Paperback Series consists of selected books that have been made more accessible to consumers in an effort to increase global appeal and general circulation. With these new unabridged softcover volumes, Wiley hopes to extend the lives of these works by making them available to future generations of statisticians, mathematicians, and scientists.
"This text is unique in bringing together so many results hitherto found only in part in other texts and papers. . . . The text is fairly self-contained, inclusive of some basic mathematical results needed, and provides a rich diet of examples, applications, and exercises. The bibliographical material at the end of each chapter is excellent, not only from a historical perspective, but because it is valuable for researchers in acquiring a good perspective of the MDP research potential."
-Zentralblatt fur Mathematik
". . . it is of great value to advanced-level students, researchers, and professional practitioners of this field to have now a complete volume (with more than 600 pages) devoted to this topic. . . . Markov Decision Processes: Discrete Stochastic Dynamic Programming represents an up-to-date, unified, and rigorous treatment of theoretical and computational aspects of discrete-time Markov decision processes."
-Journal of the American Statistical Association

Customer Reviews:

Excellent and detailed, although focusing on exact algorithms only.......2007-06-04

Anyone working with Markov Decision Processes should have this book. It has detailed explanations of several algorithms for MDPs: linear programming, value iteration and policy iteration for finite and infinite horizon; total-reward and average reward criteria, and there's one last chapter on continuous-time MDPs (SMDPs).

However, it does not cover some new ideas like partitioning and some faster approximated algorithms. But still, it is a great book!

Make sure to also get Bertsekas' "Dynamic Programming and Optimal Control".

Deterministic and Stochastic Optimal Control (Stochastic Modelling and Applied Probability)

Average customer rating:

Be advised . . .

Deterministic and Stochastic Optimal Control (Stochastic Modelling and Applied Probability)
Wendell H. Fleming , and Raymond W. Rishel
Manufacturer: Springer
ProductGroup: Book
Binding: Hardcover

General | Engineering | Professional & Technical | Subjects | Books

General | Science | Subjects | Books

General | Mathematics | Science | Subjects | Books

All Amazon Upgrade | Amazon Upgrade | Stores | Books

Engineering | Amazon Upgrade | Stores | Books

Professional & Technical | Amazon Upgrade | Stores | Books

Science | Amazon Upgrade | Stores | Books

All Titles | Qualifying Textbooks - Fall 2007 | Stores | Books

Professional | Qualifying Textbooks - Fall 2007 | Stores | Books

Science | Qualifying Textbooks - Fall 2007 | Stores | Books
Similar Items:

Controlled Markov Processes and Viscosity Solutions (Stochastic Modelling and Applied Probability)
Stochastic Controls: Hamiltonian Systems and HJB Equations (Stochastic Modelling and Applied Probability)
Numerical Methods for Stochastic Control Problems in Continuous Time (Stochastic Modelling and Applied Probability)
Stochastic Integration and Differential Equations
Applied Stochastic Control of Jump Diffusions (Universitext)

ASIN: 0387901558

Book Description

The first part of this book presents the essential topics for an introduction to deterministic optimal control theory. The second part introduces stochastic optimal control for Markov diffusion processes. It also inlcudes two other topics important for applications, namely, the solution to the stochastic linear regulator and the separation principle.

Customer Reviews:

Be advised . . . .......2006-03-17

Not recommended as an introduction -- lacks examples.

Introduction to Stochastic Search and Optimization

Average customer rating:

Great book!!!
Recommended to scholars and graduate students

Introduction to Stochastic Search and Optimization
James C. Spall
Manufacturer: Wiley-Interscience
ProductGroup: Book
Binding: Hardcover

General | Science | Subjects | Books

General | Mathematics | Science | Subjects | Books

All Titles | Qualifying Textbooks - Fall 2007 | Stores | Books

Professional | Qualifying Textbooks - Fall 2007 | Stores | Books

Science | Qualifying Textbooks - Fall 2007 | Stores | Books
Similar Items:

Monte Carlo Statistical Methods (Springer Texts in Statistics)
Stochastic Optimization Methods
Numerical Optimization (Springer Series in Operations Research and Financial Engineering)
Practical Methods of Optimization
Stochastic Approximation and Recursive Algorithms and Applications (Stochastic Modelling and Applied Probability)

ASIN: 0471330523

Book Description

Unique in its survey of the range of topics.
Contains a strong, interdisciplinary format that will appeal to both students and researchers.
Features exercises and web links to software and data sets.

Download Description

Unique in its survey of the range of topics.
Contains a strong, interdisciplinary format that will appeal to both students and researchers.
Features exercises and web links to software and data sets.

Customer Reviews:

Great book!!!.......2004-12-07

A must have for anyone interested in otimization! Extremely well written and objective.

Recommended to scholars and graduate students.......2003-09-23

Introduction to Stochastic Search and Optimization provides comprehensive, current information on methods for real-world problem solving, including stochastic gradient and non-gradient techniques, as well as relatively recent innovations such as simulated annealing, genetic algorithms, and MCMC. It is written to be read and understood by graduate students, industrial practitioners, and experienced researchers in the field. Web links to software and data sets, and an extensive list of references of the book allows the reader to explore deeper into certain topic areas. I also found the index to be very comprehensive and carefully done. The appendices are as a refresher and summary of much of the prerequisite material. The book is somewhat unique in providing a balanced discussion of algorithms, including both their strengths and weaknesses. The book is among very few books that have integrated essential parts of statistical fields with optimization and decision making. The book's inclusion of a chapter on optimal experimental design is an example of such integration. The approaches discussed in the book could be used for financial decision making, forecasting, and quality improvement, among many other areas.

Random Iterative Models (Stochastic Modelling and Applied Probability)

Average customer rating:

Random Iterative Models (Stochastic Modelling and Applied Probability)
Marie Duflo
Manufacturer: Springer
ProductGroup: Book
Binding: Hardcover

Chaos & Systems | Physics | Science | Subjects | Books

General | Mathematics | Science | Subjects | Books

General | Reference | Subjects | Books

All Amazon Upgrade | Amazon Upgrade | Stores | Books

Computers & Internet | Amazon Upgrade | Stores | Books

Professional & Technical | Amazon Upgrade | Stores | Books

Reference | Amazon Upgrade | Stores | Books

Science | Amazon Upgrade | Stores | Books

All Titles | Qualifying Textbooks - Fall 2007 | Stores | Books

Computers & Internet | Qualifying Textbooks - Fall 2007 | Stores | Books

Professional | Qualifying Textbooks - Fall 2007 | Stores | Books

Reference | Qualifying Textbooks - Fall 2007 | Stores | Books

Science | Qualifying Textbooks - Fall 2007 | Stores | Books
ASIN: 3540571000

Book Description

The recent development of computation and automation has lead to quick advances in the theory and practice of recursive methods for stabilization, identification and control of complex stochastic models (guiding a rocket or a plane, organizing multiaccess broadcast channels, self-learning of neural networks ...). This book provides an up-to-date view of a wide range of those methods: stochastic approximation, linear and non-linear models, controlled Markov chains, estimation and adaptive control, learning ...Mathematicians (researchers and also students) and engineers will find here a self-contained account of many approaches to those theories.

Average customer rating:

Very detailed overview of SPNs

Stochastic Petri Nets
Peter J. Haas
Manufacturer: Springer
ProductGroup: Book
Binding: Hardcover

General | Science | Subjects | Books

General | Mathematics | Science | Subjects | Books

Systems Analysis & Design | Computer Science | Computers & Internet | Subjects | Books

General | Computers & Internet | Subjects | Books

All Amazon Upgrade | Amazon Upgrade | Stores | Books

Computers & Internet | Amazon Upgrade | Stores | Books

Engineering | Amazon Upgrade | Stores | Books

Professional & Technical | Amazon Upgrade | Stores | Books

Science | Amazon Upgrade | Stores | Books

All Titles | Qualifying Textbooks - Fall 2007 | Stores | Books

Computers & Internet | Qualifying Textbooks - Fall 2007 | Stores | Books

Professional | Qualifying Textbooks - Fall 2007 | Stores | Books

Science | Qualifying Textbooks - Fall 2007 | Stores | Books
Similar Items:

Petri Nets for Systems Engineering
Discrete, Continuous, and Hybrid Petri Nets

Accessories:

An Introduction to Copulas (Springer Series in Statistics)
Modelling and Control of Robot Manipulators (Advanced Textbooks in Control and Signal Processing)
Numerical Geometry of Images: Theory, Algorithms, and Applications

ASIN: 0387954457

Book Description

This book is about stochastic Petri nets (SPNs), which have proven to be a popular tool for modelling and performance analysis of complex discrete-event stochastic systems. The focus is on methods for modelling a system as an SPN with general firing times and for studying the long-run behavior of the resulting SPN model using computer simulation. Modelling techniques are illustrated in the context of computer, manufacturing, telecommunication, workflow, and transportation systems. The simulation discussion centers on the theory that underlies estimation procedures such as the regenerative method, the method of batch means, and spectral methods.Tying these topics together are conditions on the building blocks of an SPN under which the net is stable over time and specified estimation procedures are valid. In addition, the book develops techniques for comparing the modelling power of different discrete-event formalisms. These techniques provide a means for making principled choices between alternative modelling frameworks and also can be used to extend stability results and limit theorems from one framework to another. As an overview of fundamental modelling, stability, convergence, and estimation issues for discrete-event systems, this book will be of interest to researchers and graduate students in Applied Mathematics, Operations Research, Applied Probability, and Statistics. This book also will be of interest to practitioners of Industrial, Computer, Transportation, and Electrical Engineering, because it provides an introduction to a powerful set of tools both for modelling and for simulation-based performance analysis. Peter J. Haas is a member of the Research Staff at the IBM Almaden Research Center in San Jose, California. He also teaches Computer Simulation at Stanford University and is an Associate Editor (Simulation Area) for Operations Research.

Customer Reviews:

Very detailed overview of SPNs.......2004-03-09

Petri nets have been used in operations research and the mathematical modeling of discrete-event systems ever since they were invented in the early 1960s. The applications of Petri nets are immense, having permeated many different fields, some of these being network engineering, queueing theory, and automated manufacturing. This book gives a very clear introduction to the mathematical theory of stochastic Petri nets (SPNs), which were invented in the 1980s, and which are used to model discrete-event systems which undergo stochastic state transitions occur only at an increasing sequence of random times. The book should be viewed as a monograph rather than a textbook since there are no exercises (unfortunately), but readers could still gain a good understanding of stochastic Petri nets by its perusal. One could make up for the lack of exercises by perhaps thinking of new applications of SPNs. My interest in the book was motivated by my wish to use SPNs to model network and application servers, poker games, and machine curiosity and decision-making in artificial intelligence. I only read chapters 1-5 and chapter 9, and so my review will be confined to these.

The author defines an SPN as a graph composed of a finite set of `places' and a finite set of `transitions'. A subset of these transitions are taken to be `immediate' transitions, and the set of places consists of normal input places, inhibitor input places, and output places, given a particular transition. A (countable) set of markings denoting the number of `tokens' in a place is also defined. In chapter 2, the author gives several examples of SPNs, such as a producer-consumer system, a queue with batch arrivals, a token ring network, a flexible manufacturing system, a particle counter, and a slotted ring network. Some of these examples illustrate the use of marking-dependent transitions, and the fact that SPN representations of discrete-event systems are not unique. The author also briefly discusses the SPSIM simulation language for SPNs. Also discussed briefly are restricted SPNs, wherein the marking set is not specified explicitly, and an accompanying notion of reachability.

The marking process of an SPN is described in terms of an underlying general state-space Markov chain in chapter 3. This allows sample paths to be generated, and one can utilize the results from the theory of Markov chains to study the long-time behavior of SPNs and define performance measures for them. The author gives an explicit algorithm for generating sample paths for the underlying chain and using this, for the marking process itself. This is followed by a discussion of sufficient conditions needed to guarantee infinite lifetimes for the marking process, thus avoiding "explosions", wherein an infinite number of marking changes occur in a finite time interval with probability 1. The author also gives criteria for showing when the marking process is a time-homogeneous continuous-time Markov chain.

In chapter 4, the author discusses to what extent discrete-event systems can be modeled within the SPN framework. He does not answer this in general, claiming that it cannot be, but instead compares the modeling power of SPNs to that of generalized semi-Markov processes (GSMPs). These systems differ, he says, in their event-scheduling and state-transition mechanisms, and the form of the state-space. GSMPs are more general than SPNs, but the author shows that SPNs have at least the modeling power of GSMPs, in that for any GSMP there exists an SPN that `strongly mimics' it: there is a marking process such that both of the processes have the same finite-dimensional distributions using an appropriate mapping between the underlying state spaces. Conversely, for any SNP with both timed and immediate transitions, the author shows that there exists a GSMP that strongly mimics the marking process of the SPN. A very brief but interesting discussion on the ability of Petri nets to mimic a Turing machine is given in the notes to the chapter.

The author turns his attention to stability issues in chapter 5. This attention is dictated by the fact the in order for SPNs to be practical for simulation purposes, their marking processes must have well-defined time-average limits. The stability of an SPN is shown, as expected, with reference to the underlying state-space Markov chain used to define the marking process. In this context, the author uses the notion of "Harris recurrence", wherein Markov chains that have this property repeatedly return to a dense, compact set of states. Criteria for establishing Harris recurrence are given throughout the chapter. Readers will have to know some amount of measure theory in order to read this chapter. The author gives a brief review of it in one of the appendices.

Chapter 9 covers colored stochastic Petri nets (CSPNs), which have myriads of applications and so a thorough reading of it is essential for those involved in those applications. As the author explains, associating colors with tokens and transitions will allow the simplification of Petri nets that have large numbers of places and transitions. The tokens are removed and deposited deterministically, and so CSPNs have less modeling power than SPNs. The tradeoff though is the conciseness of the CSPNs. Noted in the definition of CSPNs is the presence of input and output incidence functions, which determine when a transition is enabled in a color and the number of tokens removed and deposited when a transition fires in a color. Several examples of CSPNs are discussed, including machine repair, a token ring network, a system of cyclic queues with feedback, and one dealing with customer complaint processing. As was the case for SPNs, the marking process of a CSPN is defined in terms of a general state-space Markov chain that describes the CSPN at successive marking changes. The author studies the stability of CSPNs , and considers what are called "symmetric" CSPNs, which are those that remain the same under permutations of its set of colors. The mathematical analysis of symmetric CSPNs is, as expected, simpler than non-symmetric CSPNs.

Introduction to Stochastic Dynamic Programming
Average customer rating:

Good Examples BUT...a little too theoretical

Introduction to Stochastic Dynamic Programming
Sheldon M. Ross
Manufacturer: Academic Press
ProductGroup: Book
Binding: Paperback

General | Introductory & Beginning | Programming | Computers & Internet | Subjects | Books
General | Science | Subjects | Books
General | Mathematics | Science | Subjects | Books
Linear Programming | Applied | Mathematics | Science | Subjects | Books
Probability & Statistics | Applied | Mathematics | Science | Subjects | Books
Linear Programming | Applied | Mathematics | Professional Science | Professional & Technical | Subjects | Books
Statistics | Applied | Mathematics | Professional Science | Professional & Technical | Subjects | Books
All Amazon Upgrade | Amazon Upgrade | Stores | Books
Computers & Internet | Amazon Upgrade | Stores | Books
Professional & Technical | Amazon Upgrade | Stores | Books
Science | Amazon Upgrade | Stores | Books
All Titles | Qualifying Textbooks - Fall 2007 | Stores | Books
Computers & Internet | Qualifying Textbooks - Fall 2007 | Stores | Books
Professional | Qualifying Textbooks - Fall 2007 | Stores | Books
Science | Qualifying Textbooks - Fall 2007 | Stores | Books
Similar Items:

Dynamic Programming

Dynamic Programming: Models and Applications

Applied Probability Models with Optimization Applications (Dover Books on Mathematics)

Dynamic Programming and Optimal Control (2 Vol Set)

Stochastic Models in Operations Research, Vol. II: Stochastic Optimization

ASIN: 0125984219

Customer Reviews:

Good Examples BUT...a little too theoretical.......2000-08-02

I've used this book for a graduate course in Dynamic Programming. Having used many of Mr. Ross's books (undergraduate and graduate), I found this one lacks the detail and lucidity (particularly the end of chapter problems...I believe in "learning by doing"... i.e. solve lots of problems!) that I have come to know of his books (e.g. A First Course in Probability and Introduction to Probability Models). The bright spot of the book is its examples, which are interesting and fairly detailed.

Stochastic Approximation and Its Application (Nonconvex Optimization and Its Applications)

Average customer rating:

Stochastic Approximation and Its Application (Nonconvex Optimization and Its Applications)
Han-Fu Chen
Manufacturer: Springer
ProductGroup: Book
Binding: Hardcover

General | Science | Subjects | Books

General | Mathematics | Science | Subjects | Books

General | Medicine | Subjects | Books

Systems Analysis & Design | Computer Science | Computers & Internet | Subjects | Books

General | Computers & Internet | Subjects | Books

All Amazon Upgrade | Amazon Upgrade | Stores | Books

Computers & Internet | Amazon Upgrade | Stores | Books

Engineering | Amazon Upgrade | Stores | Books

Medicine | Amazon Upgrade | Stores | Books

Professional & Technical | Amazon Upgrade | Stores | Books

Science | Amazon Upgrade | Stores | Books
ASIN: 1402008066

Book Description

This book presents the recent development of stochastic approximation algorithms with expanding truncations based on the TS (trajectory-subsequence) method, a newly developed method for convergence analysis. This approach is so powerful that conditions used for guaranteeing convergence have been considerably weakened in comparison with those applied in the classical probability and ODE methods. The general convergence theorem is presented for sample paths and is proved in a purely deterministic way. The sample-path description of theorems is particularly convenient for applications. Convergence theory takes both observation noise and structural error of the regression function into consideration. Convergence rates, asymptotic normality and other asymptotic properties are presented as well. Applications of the developed theory to global optimization, blind channel identification, adaptive filtering, system parameter identification, adaptive stabilization and other problems arising from engineering fields are demonstrated.
Audience: Researchers and students of both graduate and undergraduate levels in systems and control, optimization, signal processing, communication and statistics.

Stochastic Models in Operations Research, Vol. I: Stochastic Processes and Operating Characteristics (Stochastic Models in Operations Research)

Average customer rating:

Stochastic Models in Operations Research, Vol. I: Stochastic Processes and Operating Characteristics (Stochastic Models in Operations Research)
Matthew J. Sobel , and Daniel P. Heyman
Manufacturer: Dover Publications
ProductGroup: Book
Binding: Paperback

Operations Research | Management & Leadership | Business & Investing | Subjects | Books

General | Science | Subjects | Books

General | Mathematics | Science | Subjects | Books
Similar Items:

Stochastic Models in Operations Research, Vol. II: Stochastic Optimization
Applied Probability Models with Optimization Applications (Dover Books on Mathematics)
Dynamic Programming: Models and Applications
Foundations of Stochastic Inventory Theory (Stanford Business Books)
Markov Decision Processes: Discrete Stochastic Dynamic Programming (Wiley Series in Probability and Statistics)

ASIN: 0486432599

Book Description

This two-volume set of texts explores the central facts and ideas of stochastic processes, illustrating their use in models based on applied and theoretical investigations. They demonstrate the interdependence of three areas of study that usually receive separate treatments: stochastic processes, operating characteristics of stochastic systems, and stochastic optimization. Comprehensive in its scope, they emphasize the practical importance, intellectual stimulation, and mathematical elegance of stochastic models and are intended primarily as a graduate-level texts.

Optimal Stopping and Free-Boundary Problems (Lectures in Mathematics. ETH Zürich)

Average customer rating:

Optimal Stopping and Free-Boundary Problems (Lectures in Mathematics. ETH Zürich)
Goran Peskir , and Albert Shiryaev
Manufacturer: Birkhäuser Basel
ProductGroup: Book
Binding: Hardcover

General | Science | Subjects | Books

General | Mathematics | Science | Subjects | Books

All Titles | Qualifying Textbooks - Fall 2007 | Stores | Books

Professional | Qualifying Textbooks - Fall 2007 | Stores | Books

Science | Qualifying Textbooks - Fall 2007 | Stores | Books
Similar Items:

Stochastic Differential Equations and Applications (Dover Books on Mathematics)
Continuous Martingales and Brownian Motion (Grundlehren der mathematischen Wissenschaften)
Applied Stochastic Control of Jump Diffusions (Universitext)
The Volatility Surface: A Practitioner's Guide (Wiley Finance)
Controlled Markov Processes and Viscosity Solutions (Stochastic Modelling and Applied Probability)

Accessories:

Numerical Optimization (Springer Series in Operations Research and Financial Engineering)
Introduction to the Theory of Nonlinear Optimization
Nonlinear Systems

ASIN: 3764324198

Book Description

The book aims at disclosing a fascinating connection between optimal stopping problems in probability and free-boundary problems in analysis using minimal tools and focusing on key examples.

The general theory of optimal stopping is exposed at the level of basic principles in both discrete and continuous time covering martingale and Markovian methods. Methods of solution explained range from classic ones (such as change of time, change of space, change of measure) to more recent ones (such as local time-space calculus and nonlinear integral equations).

A detailed chapter on stochastic processes is included making the material more accessible to a wider cross-disciplinary audience. The book may be viewed as an ideal compendium for an interested reader who wishes to master stochastic calculus via fundamental examples.

Areas of application where examples are worked out in full detail include financial mathematics, financial engineering, mathematical statistics, and stochastic analysis.

Applications Of Multi-Objective Evolutionary Algorithms (Advances in Natural Computation)

Average customer rating:

Applications Of Multi-Objective Evolutionary Algorithms (Advances in Natural Computation)

Manufacturer: World Scientific Publishing Company
ProductGroup: Book
Binding: Hardcover

General | Computer Science | Computers & Internet | Subjects | Books

General | Mathematics | Science | Subjects | Books

All Titles | Qualifying Textbooks - Fall 2007 | Stores | Books

Computers & Internet | Qualifying Textbooks - Fall 2007 | Stores | Books

Professional | Qualifying Textbooks - Fall 2007 | Stores | Books

Science | Qualifying Textbooks - Fall 2007 | Stores | Books
ASIN: 9812561064

Book Description

This book presents an extensive variety of multi-objective problems across diverse disciplines, along with statistical solutions using multi-objective evolutionary algorithms (MOEAs). The topics discussed serve to promote a wider understanding as well as the use of MOEAs, the aim being to find good solutions for high-dimensional real-world design applications. The book contains a large collection of MOEA applications from many researchers, and thus provides the practitioner with detailed algorithmic direction to achieve good results in their selected problem domain.

Books:

Books Index

Books Home

Recommended Books