In this module we explore techniques for modelling systems that exhibit uncertainty. Markov chains will be the principle tool, but we will also consider some important related processes such as renewal processes and diffusion processes.

Queueing theory is an important application of Markov chains, and this module will build on the material covered in MA0261. The module will also include numerous other applications, for example to problems such as insurance claims, epidemic growth, reliability, inventory management, and stock prices. Many of the exercises will involve simulation or require numerical solution methods, so it is recommended that students have some programming experience.

Prerequisite Modules: MA0261 Operational Research

(In the academic year 2018/19 this module will only be available to students registered on the degree schemes BSc Mathematics, Operational Research and Statistics, MORS Mathematics, Operational Research and Statistics, or BSc Financial Mathematics).

• Understand the principle concepts of Markov chains.
• Classify the states of a Markov chain, and calculate hitting probabilities, expected hitting times, and limiting probabilities.
• Work with the negative exponential and Poisson distributions, and understand their relationship.
• Apply Markov chains to general queues, such as M/G/1 and GI/M/1.
• Use Markov chains to describe a variety of stochastic (random) processes.
• State and interpret the renewal equation.
• Understand Brownian motion as a limit of random walks.

22 fifty-minute lectures

5 fifty-minute exercise classes

Students are also expected to undertake at least 50 hours private study including preparation of worked solutions for exercise classes.

Problem solving, logical thinking and mathematical formulation of real-life situations

Formative assessment is carried out by means of regular exercises.  Feedback to students on their solutions and their progress towards learning outcomes is provided during classes.  There is no summative coursework assessment for this module

Summative assessment is by means of the written examination at the end of the module.  This gives students the opportunity to demonstrate their overall achievement of learning outcomes.  It also allows them to give evidence of the higher levels of knowledge and understanding required for above average marks.

The examination paper has a choice of three from four equally weighted questions.

• Discrete-time Markov chains
• Classification of finite chains
• Infinite chains
• The Poisson process
• Renewal processes
• Continuous-time Markov chains
• Queueing theory
• Brownian motion and diffusion

Sheldon M. Ross, Introduction to Probability Models, Eleventh Edition. Academic Press, 2014.

Geoffrey R. Grimmett and David Stirzaker, Probability and Random Processes, Third Edition. Oxford University Press, 2001.

D.L. Paul Minh, Applied Probability Models. Brooks/Cole, 2000.