Entropy

Information entropy

Introduced by [Shannon, 1948] in the context of information channels.

Railway engineering: to communicate n signals/commands, how many lamps/semaphors are needed?

railway lights

It depends on the informative content (surprise) of the signals.

Wheather news: London vs. Wadi Halfa

Weather forecasts for London are frequent and nuanced

Not so in Wadi Halfa, Sudan, one of the driest cities on Earth

Wadi Halfa

W. in Wadi Halfa has low entropy and needs a smallish communication channel

London needs a high-capacity communication channel

Informative Entropy (Eta)

The expectation to receive information

$H[X] = \sum Pr[X=x_i]\cdot I[x_i]$

Information content:

$I[x_i] = \frac{1}{\log_2 Pr[X=x_i]}$

The log-distribution of surprise

$H[X] = -\sum Pr[X=x_i]\cdot \log_2 Pr[X=x_i]$

Minimal ent.: H[X]=0, the system is deterministic

Max ent.: $H[X]=\log_2 n$ all messages have the same probability

Implementation

def H(distribution):
    '''computes Shannon's entropy of a distribution, given as a numpy array'''

    ent = 0.0

    for dim in distribution:
        if dim==0.0:
            ent+=0.0
        else:
            ent+= dim*math.log(dim, 2)

    return -ent

Ent in Python

Applications

Data compression: we need only $\lceil H(Dist) \rceil$ bits.
How informative a dataset is?
Approximation: what is the model distribution that approximates the observed data while loosing as little information as possible?