test1solution.html

Test 1 Information Theory M/CS 4890

1. Shannon's Noiseless Coding Theorem(20 points):

State Shannon's Noiseless Coding Theorem.
Compute the Entropy of the follow alphabet and probability distribution. You can use the approximation

Symbol Probability

A 1/4

B 1 /6

C 1 / 6

D 1 /6

E 1 / 4

Construct the associated Huffman Code.
What does Shannon's Noiseless Coding Theorem say about expected code lengths? Confirm this numerically by computing the

expected lengths of encoded symbols in the Huffman Code.

Answer:

Given a set of symbols with probability distribution any binary prefix code for $\QTR{Large}{S\ }$ satisfies the equation

Symbol Probability

A 1 /4

B 1 / 6

C 1 /6

D 1 /6

E 1 / 4

Symbol Probability

A 1 /4

E 1 /4

B 1 / 6

C 1 / 6

D 1 / 6

Symbol Probability

[C,D] 1 /3

A 1 / 4

E 1 /4

B 1 / 6

Symbol Probability

[E,B] 5 /12

[C,D] 1 /3

A 1 / 4

Symbol Probability

[[C,D],A] 7 /12

[E,B] 5 /12

Symbol Code

[[C,D],A] 0

[E,B] 1

Symbol Code

[C,D] 00

A 01

E 10

B 11

Symbol Code
C 000

D 001

A 01

E 10

B 11

____________________________________________________________________________

2. Shannon's Noisy Coding Theorem(15 points):

Let and $\QTR{Large}{R\ <C}$ , then for sufficiently large integers $\QTR{Large}{n\ \ }$ and we can find:

an code with :.
.
, where is to be read " $\QTR{Large}{y}$ cannot be unambiguously decoded.".

such that

MATH

Letting the approximate value of .:

What does the Noisy Coding Theorem about transmitting copies of The Complete Works of William Shakespeare?

Nothing really it is really about $\QTR{Large}{p}$ not $\QTR{Large}{n.}$ . In particular, for $\QTR{Large}{p}$ close to $\QTR{Large}{1/2}$ . $\QTR{Large}{n}$ may have to be very very large, lots of noise and for $\QTR{Large}{p}$ close to $\QTR{Large}{1}$ . $\QTR{Large}{n,}$ little noise, $\QTR{Large}{n}$ can be close to the code size you actually need.
In general, letting , numerically, what does the Noisy Coding Theorem assert?

For sufficiently large integers $\QTR{Large}{n\ }$ you can find an code with $\QTR{Large}{k/n\ }$ arbitrarily close to $\QTR{Large}{.92}$ and such that the probability of a symbol being decoded incorrectly is less than $\QTR{Large}{.001}$

____________________________________________________________________________________________

3. (20 Points): Do a worst case analysis of the four ball weighing problem, 3 the same and 1 heavier or lighter. What is the Shannon Clue?

Compute the Information provided by the various experiments in your analysis. Does the total worst case Information gained conform to the Shannon Clue

Answer:

There are 4 balls and 1 is heavier or lighter so there are 8 equiprobable outcomes.so the Information gained learning which ball is the odd ball and whether it is heavier or lighter is

Bits of information. So the Shannon Clue is that if somehow I could learn a little more than $\QTR{Large}{1.5}$ Bits of Information in each of 2 weighings, as I had done before, I might be able to solve the problem in two weighings.

The possible experiments are
- one ball on each side two off the balance.
- two balls on each side
  
  This second possibility has two possible outcomes, left side up left side down. It produces $\QTR{Large}{1}$ Bit of Information, not enough Information, given the Clue, it is also easy to see that one additional weighing is not going to be enough.

In the case of one ball on each side two off the balance , there are three possible outcomes, the left side goes up, the left side goes down or the scale balances.

They are not equiprobable. The scale balances in $\QTR{Large}{4\ }$ possible ways. The odd ball could be either of the two off the scale and the odd ball could be heavier or lighter Thus the probability of the scale balancing is

$\QTR{Large}{1/2}$ thus learning that this is the outcome give $\QTR{Large}{1}$ Bit of Information.

The left side could only go down in 2 possible ways. The left side has the odd ball and it is heavier or the right side has the odd ball and it is lighterThus the probability of the left side going down is $\QTR{Large}{1/4}$ and learning that this is the outcome gives $\QTR{Large}{2}$ Bits of Information. and we only need $\QTR{Large}{3\ }$ based on the Shannon Clue ..

Indeed in this case one additional weighing would be enough. Just weigh the ball on the left side against a good ball. If the scale goes down again, the ball on the left is a heavy ball, if the scale balances the ball that was on the right was a light ball.

Similarly for the left side side going up

Since we are considering worst case, in the case that the odd ball was off the scale in the first weighting we know the odd ball is one of the two left but we don't know if it is heavier or lighter. One additional weighting is not enought to determine this. However, knowing it is one of two, we can weight them against each other and use the previous strategy to solve the problem in a second additional weighing, Hence; The worst case is that it takes 3 weighings to solve the 4 ball weighing problem ( no better than the 12 ball problem)

___________________________________________________________________________________________________

4. (20 Points): Letting be define by the matrix ,

and the two code words are certainly $\QTR{Large}{3}$ apart.

Show that it is also a Perfect Code. The Hamming Balls of radius 1 fill up the space.
Define a one bit error checking $\QTR{Large}{\ }$ matrix $\QTR{Large}{H}$ . ( and any )

As an aside, this is the Hamming Code.

Answer:

Besides and , in there are $\QTR{Large}{3}$ vectors with one $\QTR{Large}{1}$ . These are Hamming Distance $\QTR{Large}{1}$ away from There are also $\QTR{Large}{3}$ vectors with two $\QTR{Large}{1}$ s that are Hamming Distance $\QTR{Large}{1}$ away from

, hence is a Perfect Code.

To know we need an even number of $\QTR{Large}{1}$ s in each column and to know any we need a $\QTR{Large}{1}$ in the $\QTR{Large}{i}$ th row of at least 1 column.

MATH

has both of these properties.

________________________________________________________________________________________________

5. (25 Points):Solve the following variant of the Three Doors Problem

The Host hides a money prize behind one of three randomly chosen doors with probability 1/2 for door 1 and 1/4 for doors 2 and 3
.
The Player chooses door 1.knowing the hosts preference
.
The Host opens either door 2 or door 3 using the following rules:
- If the money is behind door 1 open 2 or 3 at random.
  
  If it is a head open 2 and if it is a tail open 3.
- If the money is behind door 2 open 3 and if it is behind 3 open 2.

The Question: Suppose the Host opens door 3, should the Player open door 1 or open door 2.?

Your answer should consist of filling in the following outline.

The Random Variable: The door that the money is hidden behind: $\QTR{Large}{M}$ , which takes on the values

with
The Conditional Distribution:
The Random Variable: The door that the Host opens $\QTR{Large}{O}$ , which takes on the values

with since
The Posterior Distribution:
The Decoding Strategy: either or and either or both are equiprobable
The Marginal Information using any formula version.