decipher.sequence

Sequence-level statistical primitives: linguistic-laws tests, n-gram models, transition counts, segmentation by transitional-probability ratio.

dataclass decipher.sequence.MenzerathResult

@dataclass
class MenzerathResult:
    length_estimate: float       # slope of log(duration) ~ log(length); negative = law holds
    ci_lo: float
    ci_hi: float
    holds: bool                  # True iff entire 95% CI < 0
    n_elements: int
    n_sequences: int
    position_estimate: float     # negative = elements shorten toward sequence end
    position_ci_lo: float
    position_ci_hi: float

dataclass decipher.sequence.ZipfBrevityResult

@dataclass
class ZipfBrevityResult:
    count_estimate: float        # slope of count effect; negative = law holds
    ci_lo: float
    ci_hi: float
    holds: bool
    n_elements: int
    n_types: int

function decipher.sequence.menzerath_test

def menzerath_test(
    sequences: list[list[float]],
) -> MenzerathResult

Mixed-effects model with (1 | sequence) random intercepts. Each inner list is one sequence of element durations.

function decipher.sequence.zipf_brevity_test

def zipf_brevity_test(
    durations: list[float],
    types: list[int | str],
) -> ZipfBrevityResult

Mixed-effects model with (1 | type) random intercepts. durations[i] and types[i] describe the same element.

dataclass decipher.sequence.ZipfFrequencyFit

@dataclass
class ZipfFrequencyFit:
    alpha: float
    intercept: float
    r_squared: float
    n_types: int

function decipher.sequence.zipf_frequency_fit

def zipf_frequency_fit(counts) -> ZipfFrequencyFit

Power-law fit on the rank-frequency distribution of type counts.

dataclass decipher.sequence.TransitionCounts

@dataclass
class TransitionCounts:
    counts: dict[tuple[str, str], int]
    vocab: list[str]

    def probability(
        self, from_sym: str, to_sym: str,
        smoothing_k: float = 0.0,
    ) -> float

    def as_matrix(self) -> tuple[np.ndarray, list[str]]

Bigram transition counts. as_matrix returns (P, vocab) where P[i, j] = P(vocab[j] | vocab[i]).

function decipher.sequence.bigram_transitions

def bigram_transitions(
    sequences: list[list[str]],
) -> TransitionCounts

Count consecutive symbol pairs across multiple sequences.

dataclass decipher.sequence.NgramResult

@dataclass
class NgramResult:
    perplexity: float
    entropy_bits: float
    n: int
    n_tokens: int
    n_types: int

function decipher.sequence.ngram_perplexity

def ngram_perplexity(
    sequences: list[list[str]],
    n: int = 3,
    smoothing_k: float = 0.1,
) -> NgramResult

n-gram language model with add-k smoothing. Returns perplexity (in the model's units) and entropy in bits.

function decipher.sequence.segment_by_tp_ratio

def segment_by_tp_ratio(
    sequence: list[str],
    *,
    direction: str = "forward",     # "forward" | "backward" | "expanded"
    threshold: float = 1.0,
) -> list[list[str]]

Saffran-style segmentation: cut at relative dips in transitional probability. Useful for unsupervised "word"-finding inside a long symbol stream.