Decipher

Tutorial 5 — Entropy and mutual information

All entropy estimates use the Miller-Madow correction by default. Mutual information is bias-corrected via a permutation null.

Single-stream entropy

from decipher import shannon_entropy

H = shannon_entropy(stream.symbols, alphabet_size=stream.alphabet_size)
print(f"H(X) = {H:.3f} bits")

The Miller-Madow term adds (K - 1) / (2 N ln 2) bits, where K is the number of observed distinct symbols and N is the sequence length. To disable: pass correction="none".

Mutual information between two parallel streams

Suppose you have two stream views over the same units — for instance, two different clusterings, or a clustering vs. an external context label.

from decipher import mutual_information_bits, permutation_null_mi

result = mutual_information_bits(
    s_a=stream_a.symbols, K_a=stream_a.alphabet_size,
    s_b=stream_b.symbols, K_b=stream_b.alphabet_size,
)
print(result["MI"], result["NMI"], result["H_a_given_b"])

The returned dict has H(A), H(B), H(A,B), MI, H(A|B), H(B|A), and NMI. NMI is the arithmetic-mean normalisation: 2 · MI / (H(A) + H(B)), matching scikit-learn's convention.

Bias correction by permutation

null = permutation_null_mi(
    s_a=stream_a.symbols, K_a=stream_a.alphabet_size,
    s_b=stream_b.symbols, K_b=stream_b.alphabet_size,
    n_permutations=500, rng_seed=42,
)
mi_corrected = result["MI"] - null["mean"]
print(f"MI = {result['MI']:.3f} (raw),  null mean = {null['mean']:.3f},  bias-corrected = {mi_corrected:.3f}")

The permutation shuffles s_b while holding s_a fixed. The bias correction is applied identically to observed and shuffled streams, so the null mean captures the plug-in MI bias at this (N, K_a, K_b).

Comparing two clusterings of the same units

from decipher import compare_partitions

cmp = compare_partitions(
    labels_a=labels_clustering_1, K_a=K1,
    labels_b=labels_clustering_2, K_b=K2,
)
print(cmp.nmi, cmp.ari, cmp.mi_bits, cmp.compactness_a, cmp.delta_compactness)

PartitionAlignment packages NMI, adjusted Rand index, mutual information, marginal/joint/conditional entropies, and a "compactness" asymmetry that detects when one partition is a refinement of the other.

Next

Zipf and Menzerath →