ECS 315: Probability and Random Processes

Everything we do, everything that happens around us, obeys the laws of probability. We can no more escape them than we can escape gravity... "Probability," a philosopher (Bishop Butler) once said, "is the very guide of life." We are all gamblers who go through life making countless bets on the outcome of countless actions.

Every field of science is concerned with estimating probability. A physicist calculates the probable path of a particle. A geneticist calculates the chances that a couple will have blue-eyed children. Insurance companies, businessmen, stockbrokers, sociologists, politicians, military experts - all have to be skilled in calculating the probability of the events with which they are concerned.

[Gardner, 1986]

Synopsis

Probability theory is the branch of mathematics that tells us how to estimate degrees of probability. If an event is certain to happen, it is given a probability of 1. If it is certain not to happen, it has a probability of 0.

This course introduces the principles of probability and random processes to undergraduate students in electronics and communication. The topics to be covered include random experiments, events, probability, discrete and continuous random variables, probability density function, cumulative distribution function, functions of random variables, expectations, law of large numbers, central limit theorem, introduction to random processes, Gaussian random process, autocorrelation and power spectral density.

Announcements

The (second) online self-evaluation form is due by the end of Dec 12.
Information regarding the final exam [Last Updated @ 10PM on Dec 10]
- Date: 12 Dec 2017
- Time: 9:00 - 12:00
- ROOM: BKD 2501-2, 2506 and 2602
- Basic calculators, e.g. FX-991MS, are permitted.
- 9 pages ( including the cover page)
- 14+1 = 15 questions. (6+5+16+8+5+1+5+14+12+5+2+2+4+14+1)
- Closed book. Closed notes.
- Two A4 sheets allowed.
  - The first sheet is your midterm study sheet.
    - May add more formulas on the remaining side.
    - Q: Can I make a brand new study sheet?
      A: No.
      However, if you think the old one is in a bad shape (falling apart), you can use a photocopy of your own sheet. (Reduction in size is not allowed though)
    - Q: What can I do if I lose my study sheet?
      A: Dr.Prapun can print out a scanned copy for you. (The writing won’t be as crisp as the original one but it’s better than nothing.)
    - Q: I think the midterm study sheet (and the thing I wrote on it) is cursed (making me get poor score); I don’t want to see or use anything on it anymore. What can I do?
      A: You may bring a brand new sheet for the final exam. However, you can only use one side of it.
  - The second sheet must contain the Phi/Q tables
    - Distributed in class on Dec 7.
    - May reprint if necessary but do not reduce the size.
    - May add more formulas on the remaining side.
    - Indicate your name and ID on the upper-right corner of the sheet (in portrait orientation) on the handwriting side.

Basically, you have two whole pages to write down
Same rules as the midterm:
- Must be hand-written in your own handwriting (except for the Phi/Q tables).
- No small pieces of paper notes glued/attached on top of it.
- Do not modify (,e.g., add/underline/highlight) content on the sheet inside the exam room.
Submit both of your A4 sheets with your exam.

Cover all the materials that we discussed in class and practice in the HWs.

Material Distribution (score-wise): 28 (Ch 7-8) + 21.5 (Ch 9) + 36.5 (Ch 10) + 13 (Ch 11)

For your studying pleasure....

All post-midterm annotated notes combined in one pdf file.
All post-midterm HWs and their solutions
- Checked HWs can be picked up in front of the EC office on the 6th floor.
- Correction: In the Solution for HW 12, Q4b, page 12-5, there should not be the term "14t". They should all be changed to "7t_1+7t_2". The conclusion given in there still holds.
All post-midterm exercises and their solutions

Graded exercises are posted on the SIIT Lecture Note System
If you have valid reason for missing class on the day that we have exercise, please indicate the date, exercise number, and the reason in the (second) online self-evaluation form . Make sure that you also submit/email supporting document/evidence to Dr.Prapun (if you haven't done so).

All post-midterm slides
2013 Final Exam
- You may skip the following parts because they are not covered this year:
- Annotated version from 2015
- Annotated version from 2014
2016 Midterm Exam

Annotated version in 2017

2013 Midterm Exam
2010 Midterm Exam
2011 Final Exam
- We covered quite a different range of topics there because the curriculum was different. However, you can find some relevent problems in there as well.
- Many of the relevant problems have already been included as exercises in the current lecture notes.
2010 Final Exam
- We covered quite a different range of topics there because the curriculum was different. However, you can find some relevent problems in there as well.
Post-midterm summary slides [Posted @ 5:20PM on Oct 24; Updated @ 2:30PM on Nov 7]

Slides for information about the midterm exam [Posted @ 1:30PM on Sep 27; Updated @ 4PM on Oct 9 and @ 2:30PM on Oct 10]
Information regarding the midterm exam [Posted @ 4PM on Sep 26]
- Check this course website regularly for breaking news about the midterm.
- Date: October 5, 2017 (Thursday)
- TIME: 15:00-17:00
- ROOMs: BKD 2506, 2501-2
- Information about the midterm exam:
- Closed book. Closed notes.
- (1 pt) One A4 page allowed.
  - Must be hand-written in your own handwriting.
  - No small pieces of paper notes glued/attached on top of it.
  - Indicate your name and ID on the upper right corner of the sheet (in portrait orientation).
  - Do not modify (,e.g., add/underline/highlight) content on the sheet inside the exam room.
  - Make sure that another side is blank. This will be used for the final exam.
  - Submit your A4 sheet with your exam. (You will get it back before the final exam.)
  - Q: I don't need any formulas. What should I do?
    A: Bring in and submit a blank sheet of paper with your name and ID. Note that you can still only use one side on the final exam.
  - Violating the above instructions will cost you 10 pt.
- Basic SIIT-approved calculators, e.g,. FX-991MS, are permitted, but borrowing is not allowed.
- For your studying pleasure....
We also have a Line group.
Note that we also share the tutorial/make-up session (Wednesday, 09:00-10:20, BKD2601) with ECS332.
This site can be accessed via ecs315.prapun.com.

Welcome to ECS315! Feel free to look around this site.

General Information

Instructor: Asst. Prof. Dr.Prapun Suksompong (prapun@siit.tu.ac.th)
- Office: BKD, 6th floor of Sirindhralai building

Office Hours: See Google calendar below.

Lectures: See Google calendar below.
Course Syllabus [Posted @ 10AM on Aug 10]
Textbook: [Y&G] R. D. Yates and D. J. Goodman, Probability and Stochastic Processes: A Friendly Introduction for Electrical and Computer Engineers, 2nd ed., Wiley, 2004.
- Call No. QA273 Y384 2005. ISBN: 978-0-471-27214-4
- Student Companion Site
References

Lecture notes [Posted @ 11PM on Aug 10, 2017; Updated @ 11PM on Dec 1, 2007]

Caution: The lecture notes will still be updated throughout the semester. Therefore, do not rely on this version for use in class. The arrangement/inclusion/exclusion of topics may be different from the ones posted below and the versions available at the copy center.

Probability and probabilistic reasoning for electrical engineering / Terrence L. Fine. Call No. QA273 F477 2006
Probability and random processes for electrical engineering / Alberto Leon-Garcia. Call No. TK153 L425 1994
Free textbook: Introduction to Probability by Charles M. Grinstead and J. Laurie Snell
Henk Tijms. Understanding Probability: Chance Rules in Everyday Life. Cambridge University Press, 3rd edition, 2012. Call No. QA273 T48 2012
Probability, random variables, and stochastic processes / Athanasios Papoulis, S. Unnikrishna Pillai. Call No. QA273 P2 2002
- Probability, random variables, and stochastic processes / Athanasios Papoulis. Call No. QA273 P2 1991
A first course in probability / Sheldon Ross. 6E Call No. QA273 R83 2002
- A first course in probability / Sheldon Ross. Call No. QA273 R83 1976
- A first course in probability / Sheldon Ross. 9E
Probability models, introduction to / Sheldon M. Ross. 10E Call No. QA273 R84 2010

Probability models, introduction to / Sheldon M. Ross. 6E Call No. QA273 R84 1997
Probability models, introduction to / Sheldon M. Ross. 11E

Handouts and Course Material

Slides: Course Introduction [Posted @ 3PM on Aug 11]
- Full version [Posted @ 5:30PM on Aug 15; Updated @ 9PM on Aug 17]
Part I: Introduction, Set Theory, Classical Probability theory, and Combinatorics [Posted @ 3PM on Aug 11]
- Section 1: Probability and You
- Section 2: Set Theory
- Section 3: Classical Probability
- Section 4: Enumeration / Combinatorics / Counting
Part II: Kolmogorov's Formal Probability Theory and Event-Based Probability Theory [Posted @ 11AM on Aug 24]

Section 5: Probability Foundations

Annotated notes [Posted @ 3PM on Aug 31; Updated @ 2:30PM on Sep 5]
Slides [Posted @ 3:30PM on Aug 31]
Exercise 4 Solution [Posted @ 3PM on Sep 5]

Section 6.1: Conditional Probability
- Annotated notes [Posted @ 4PM on Sep 7; Updated @ 7PM on Sep 14 and @ 3PM on Sep 20]
- Slides [Posted @ 4PM on Sep 7]
- Exercise 5 Solution [Posted @ 4:30PM on Sep 18]
- Exercise 6 Solution [Posted @ 3PM on Sep 27]
- "The Theory That Would Not Die, How Bayes' Rule Cracked the Enigma Code, Hunted Down Russian Submarines & Emerged Triumphant from Two Centuries of Controversy" (Yale University Press, 2011)
Section 6.2: Event-based Independence

Annotated notes [Posted @ 3PM on Sep 20; Updated @ 6:30PM on Sep 21 and @ 2PM on Sep 26]

Section 6.3: Bernoulli Trials

Annotated notes [Posted @ 2PM on Sep 26]

Part III: Discrete Random Variables

Section 7: Random Variables [Posted @ 4PM on Sep 18]

Annotated notes [Posted @ 3PM on Oct 10; Updated @ 8:30AM on Oct 16]
Exercise 7 Solution [Posted @ 10AM on Oct 19]

Sections 8.1-8.2: Discrete RV: PMF and CDF [Posted @ 4PM on Sep 18]

Annotated notes [Posted @ 9PM on Oct 19; Updated @ 11PM on Oct 21 and @ 4:30PM on Oct 24]
Slides [9PM on Oct 19]

Sections 8.3-8.4: Families of Discrete Random Variables [Posted @ 8:30AM on Oct 16]

Annotated notes [Posted @ 4:30PM on Oct 24; Updated @ 2PM on Oct 31 and @ 12AM on Nov 4]
Exercise 8 Solution [Posted @ 2PM on Oct 31]
Exercise 9 Solution [Posted @ 4PM on Nov 13]

Section 9: Expectation and Variance [Posted @ 2PM on Oct 31]

Annotated version [Posted @ 12AM on Nov 4; Updated @ 2:30PM on Nov 7, @ 8PM on Nov 9, and @ 1:30PM on Nov 14]
Slides [Posted @ 2:30PM on Nov 7]
Exercise 10 Solution [Posted @ 4PM on Nov 13]

References: [Y&G] Chapter 2

Part IV: Continuous Random Variables
- Sections 10.1-10.3: pdf [Posted @ 9PM on Nov 5]
- Sections 10.4: Families of Continuous Random Variables [Posted @ 10PM on Nov 14]
- Slides [Posted @ 1:30PM on Nov 14]
- Phi/Q tables [Posted @ 3PM on Nov 24; Distributed in class on Nov 23]
- References
  - From Discrete to Continuous Random Variables: [Y&G] Sections 3.0 to 3.1
  - PDF and CDF: [Y&G] Sections 3.1 to 3.2
  - Expectation and Variance: [Y&G] Section 3.3
  - Families of Continuous Random Variables: [Y&G] Sections 3.4 to 3.5
Part V: Multiple Random Variables
- Section 11[Posted @ 3PM on Nov 21]
- References:
  - A Pair of Random Variables: [Y&G] Sections 4.1 to 4.3 and Section 4.10
  - Extending the Definitions to Multiple RVs: [Y&G] Sections 5.1 to 5.4
  - Function of Discrete Random Variables: [Y&G] Section 4.6 (Theorem 4.9)
  - Expectation of function of discrete random variables: [Y&G] Sections 4.7 and 6.1
  - Linear Dependence: [Y&G] Section 4.7
Part VI: Limiting Theorems
- Sections 12.1-12.2 (LLN and CLT) [Posted @ 3:30PM on Nov 21]
  - Annotated version [Posted @ 4PM on Nov 30]
Part VII: Additional Topics

Section 13 Three Types of Random Variables [Posted @ 3:30PM on Nov 21]

Annotated version [Posted @ 4PM on Nov 30]

Appendix

Section A.3 Calculus [Posted @ 9:30PM on Nov 13]

Annotated version [Posted @ 8:30PM on Nov 16]

Problem Set

HW 1 (Due: Sep 1)
- Solution [Posted @ 2PM on Sep 5]
HW 2 (Due: Sep 7)
- Solution [Posted @ 2PM on Sep 8]
HW 3 (Due: Sep 14)
- Solution [Posted @ 4PM on Sep 17]
HW 4 (Due: Sep 21)
- Annotated version
- Solution [Posted @ 2PM on Sep 26]
HW 5 (Due: Sep 28)
- Solution [Posted @ 4PM on Sep 28]
HW 6 (Not Due)
- Annotated version
- Solution [Posted @ 3PM on Sep 26]
Self-Evaluation Form (1) (Due: Oct 5)
HW 7 (Due: Oct 19)
- Solution [Posted @ 11PM on Oct 25]
HW8 (Due: Oct 31)
- Annotated version [Posted @ 11PM on Oct 25]
- Solution [Posted @ 11AM on Nov 4]
HW9 (Due: Nov 9)
- Annotated version [Posted @ 2:30PM on Nov 8]
- Solution [Posted @ 2PM on Nov 15]
HW10 (Due: Nov 16)
- Annotated version [Posted @ 2:30PM on Nov 8]
- Solution [Posted @ 5PM on Nov 21]
HW11 (Due: Nov 23)
- Annotated version [Posted @ 2PM on Nov 22]
- Solution [Posted @ 3:30PM on Nov 28]
HW12 (Due: Not Due)
- Annotated version [Posted @ 3PM on Nov 29]
- Solution [Posted @ 4PM on Nov 28]
HW13 (Due: Not Due)
- Solution [Posted @ 4PM on Nov 29]
- P_XY_EVarCov.m
- P_XY_marginal_2.m
Self-Evaluation Form (2) (Due: Dec 12)

Calendar

Reading Assignment

Section 1.2 in the lecture notes
2.5 in the lecture notes

More References

Older version of the textbook: Probability and stochastic processes : a friendly introduction for electrical and computer engineers / Roy D. Yates, David J. Goodman. Call No. QA273 Y384 1999
10-page probability cheatsheet compiled from Harvard's Introduction to Probability course

Allen B. Downey, Think Stats: Exploratory Data Analysis, O'Reilly Media, 2014 (free book)
Allen B. Downey, Think Bayes: Bayesian Statistics in Python, O'Reilly Media, 2013 (free book)
Encyclopædia Britannica Online: Probability Theory
Random signals for engineers using MATLAB and Mathcad / Richard C. Jaffe. Call No. TK5102.9 J34 2000
Davenport, W.B., Probability and Random Processes, McGraw-Hill, New York, 1970. (Excellent introductory text.)
Feller, W., An Introduction to Probability Theory and its Applications, Vols. 1, 2, John Wiley, New York, 1950. (Definitive work on probability—requires mature mathematical knowledge.)
- Call No. QA273 F37 1966
Peter Olofsson, Probabilities The Little Numbers That Rule Our Lives, Wiley, 2006
Stochastic processes / Sheldon M. Ross. Call No. QA274 R65 1996
Stochastic processes / Emanuel Parzen. Call No. QA273 P278 1962
MATLAB Primer, 8th edition T. A. Davis. CRC Press, 2010.
- Seventh Edition by T. A. Davis and K. Sigmon: Call No. QA297 D38 2005
- Third Edition by K. Sigmon (Free)
- Second Edition by K. Sigmon (Free)

Misc. Links

Video: Probability 101
More information about theMonty Hall Problem
- Video: The Monty Hall Problem Explained

Video: The Monty Hall Problem
Video: Monty Hall Problem: Numb3rs and 21
Paper: Monty Hall, Monty Fall, Monty Crawl

Articles: How Random is the iPod Shuffle? [HowStuffWorks]; Is iTunes’ Shuffle Mode Truly Random?[About.com]; iTunes: Just how random is random?[CNET.com.au, 2007]; My IPod for a Random Playlist [wired.com, 2005];
Video: It *could* just be coincidence
MV: Bill Nye the Science Guy - "50 Fifty"
Video: Chevalier de Mere's Scandal of Arithmetic
Free educational software: Orstat2000
- Originally developed to promote probability and operations research in the senior forms of Dutch high schools (and early college).
- Contain modules for coin-tossing, central limit theorem, etc.
Probability review from MATH REVIEW for Practicing to Take the GRE General Test
Video: Mlodinow’s talk @ Google
Video: How many ways can you arrange a deck of cards? (There Are More Ways To Arrange a Deck of Cards Than Atoms on Earth)
- Provide nice animation explaining permutation and factorial.
- "Any time you pick up a well shuffled deck, you are almost certainly holding an arrangement of cards that has never before existed and might not exist again."
Video: The Binomial Distribution / Binomial Probability Function
Article: Greenlighting Movies: A High-Risk Game
Video: The Poisson Distribution
If you want to experience probability theory at a more advance level, one standard textbook that you can refer to is "Probability: Theory and Examples" by Prof. Durrett. Currently, the 4th edition of the textbook is available online.
Video: Peter Donnelly shows how stats fool juries (same clip on youtube)
Video: Lies, damned lies and statistics (about TEDTalks): Sebastian Wernicke on TED.com
Article about clinical/medical decision making: Jill G. Klein, "5 pitfalls in diagnosis and prescribing: psychological biases that can lead to poor judgement," 2005.
- Related topics: Pitfall #1 (representatiove heuristic), Pitfall #2(availability heuristic), and Pitfall #5 (illusory correlation).
The Median Isn't the Message by Stephen Jay Gould
Video: Daniel Kahneman: The riddle of experience vs. memory
- Happiness Can Be Bought For About $60,000 Per Year
Articles on risk intelligence
- Dylan Evans, How to Beat the Odds at Judging Risk, The Wall Street Journal, May, 2012
- Alison George, What Gamblers and Weather Forecasters Can Teach Us About Risk: An interview with the creator of the "risk quotient" intelligence scale., pp 30-31, New Scientist, May 19, 2012
Generation of random numbers
- Article: Park, S.K., and K.W. Miller. "Random Number Generators: Good Ones Are Hard to Find." Communications of the ACM, 31(10):1192–1201. 1998.
- Article: Tom McNichol, "Totally Random: How two math geeks with a lava lamp and a webcam are about to unleash chaos on the Internet"
- Article: C. Moler, Random thoughts, "10^435 years is a very long time", MATLAB News and Notes, Fall, 1995
- Article: Ziggurat algorithm generates normally distributed random numbersdescribing the ziggurat algorithm introduced in MATLAB version 5.
Games of chance
- Poker
  - Paper: Cheung, Y. L. "Why Poker is Played with Five Cards." Math. Gaz. 73, 313-315, 1989.
  - Tim Farajian's Texas Hold'Em Poker Analyzer in MATLAB
    - Allow a user to simulate different scenarios in a Texas Hold'Em game.
    - Automatically simulate as many hands as you would like, and display winning probabilities or expected returns.
- Blackjack
  - Cleve Moler's Blackjack in MATLAB + article
    - Paper: Simulating Blackjack with MATLAB
  - Michael Iori's Blackjack in MATLAB
Quotations about Statistics
Video: Statistics - Dream Job of the next decade
Virtual Laboratories in Probability and Statistics
Google Calculator (Cheat Sheet)
Sometimes the easiest way to get information on a counting problem is to compute a few small values of a function, then look for a match at the sequence server; if you find a hit, you can sometimes get citations to the literature.
Prapun's Notes on Probability Theory (Cornell Version)
MATLAB
- MIT OpenCourseWare > Electrical Engineering and Computer Science > 6.094 Introduction to MATLAB (January (IAP) 2009)
- Fundamentals: Academic Tutorial
- Video: Controlling Random Number Generation
- Free online book: Cleve Moler, Experiments with MATLAB, 2008
- Free textbook: Cleve Moler,Numerical Computing with MATLAB, 2004
Learn the Greek Alphabet in less than 10 minutes
The Greek Alphabet Song

ECS315: Probability and Random Processes