A Framework for Information Theory

Sigma Algebras

DEFINITION:

Given a set $\QTR{Large}{S}$ , a set of subsets $\ \QTR{Large}{A}\$ is called a Sigma Algebra on $\QTR{Large}{S}$ if:

The empty set,
If then (The complement of $\QTR{Large}{E}$ )
If is a countable set of subsets in $\QTR{Large}{A}$ then the union is in $\QTR{Large}{A}$ .

Some Comments, Terms, and an Example:

When we use the term countable subset we include finite subsets.
Let and suppose no subset of $\QTR{Large}{E,}$ except is in $\QTR{Large}{A}$ then $\QTR{Large}{E}$ is called an Atom of
If where then $\QTR{Large}{E}$ is called a Singleton of $\QTR{Large}{A.}$ A Singleton is by definition also an Atom.
A Partition of $\QTR{Large}{S}$ is a set such that and . $\QTR{Large}{\ }$
An Example of a Sigma Algebra: The standard deck that has two identical Jokers added. A Sigma Algebra $\QTR{Large}{A}$ is generated as follows:
- Each standard card is a Singleton of $\QTR{Large}{A}.$
  
  The subset $\QTR{Large}{\{}$ Jo $_{\QTR{Large}{1}},$ Jo is an Atom of $\QTR{Large}{A}.$
  
  The empty set,
  
  Any union of Singletons and $\QTR{Large}{\{}$ Jo $_{\QTR{Large}{1}},$ Jo is in
A Simple Example of a Sigma Algebra: $\QTR{Large}{\ }$
- The Atoms? The Singletons?

Two Properties of Sigma Algebras:

Since $\QTR{Large}{S}$
$\QTR{Large}{(}$ $\QTR{Large}{\ }$

Generating Sigma Algebras:

For any set $\QTR{Large}{S}$ the power set of $\QTR{Large}{S,}$ P $(\QTR{Large}{S)}$ , that is the set of all subsets of $\QTR{Large}{S}$ is a Sigma Algebra.

Almost by definition.
Let P $(\QTR{Large}{S)}$ be any collection of subsets of $\QTR{Large}{S}$ , let be the set of all Sigma Algebras with P $(\QTR{Large}{S)}$

then is a Sigma Algebra called the Sigma Algebra generated by $\QTR{Large}{G.}$ Clearly, since for all
Given a Sigma Algebra for a set $\QTR{Large}{S}$ ,a set $\QTR{Large}{T}$ , and a function , then the collection of subsets is a Sigma Algebra of $\QTR{Large}{T.}$

_________________________________________________________________________________________________________

Probability

DEFINITION: Suppose we are given a set $\QTR{Large}{S}$ and a Sigma Algebra, a Probability Measure on $\QTR{Large}{A}$ is a function such that

If is a countable (again could be finite) subset such that for then $\QTR{Large}{P(}$

In the context of Probability, we will call $\QTR{Large}{S}$ the Sample Space.

The Usual Example:

Let $\QTR{Large}{S}$ be a finite set and $\QTR{Large}{A}$ a Sigma Algebra on $\QTR{Large}{S}$ . For any let $\QTR{Large}{n(B)}$ be the number of members in $\QTR{Large}{B}$ , then for all is a Probability Measure on $\QTR{Large}{A}$ .

Properties of Probability Measures:

since $\QTR{Large}{S}$ and
In particular since
In general,

$\hspace{2in}$ since and

and
if then Write
If is a Partition of $\QTR{Large}{A}$ then .

_____________________________________________________________________

Random Variables and Expected Value:

DEFINITION:

Given a set $\QTR{Large}{S}$ , a Sigma Algebra $\QTR{Large}{A}$ on $\QTR{Large}{S}$ , and a countable discrete set $\QTR{Large}{D}$ , a Discrete Random Variable is a function $\QTR{Large}{X}$ $\QTR{Large}{:}$ such that if $\QTR{Large}{x}$ is in the image of $\QTR{Large}{X}$ then is in $\QTR{Large}{A}$ .

If $\QTR{Large}{D}$ , is finite then we will call $\QTR{Large}{X}$ a Finite Random Variable.

We write for
In the setting of 1. , if $\QTR{Large}{X}$ is a Finite Random Variable then

_________________________________________________________________________________________________________

The Framework:

Given a Finite Random Variable $\QTR{Large}{X}$ $\QTR{Large}{:}$ for a Sigma Algebra $\QTR{Large}{A}$ on $\QTR{Large}{S,}$ where . And given a Probability Measure $\QTR{Large}{P}$ on $\QTR{Large}{A}$ . We will be interested in