1
An alternative construction of Shannon entropy
The underpinning of information theory is Shannon’s formula for the entropy H of a discrete probability distribution P. Derived in 1948 in the paper “A mathematical theory of communication”, this is given by \[ H[P] = -\sum_{i=1}^N p_i \log p_i \] where \( p_i = p(x_i) \) is the probability of sampling \( x_i \) from…