Guide — Embeddings

Two embedding families ship with the toolkit: a frozen self-supervised encoder (AVES) and a convolutional autoencoder.

AVESEncoder — frozen SSL embeddings

Wraps the AVES wav2vec2 model from Hagiwara 2023. Loads pretrained weights and a torchaudio config; produces per-frame 768-dim embeddings.

from decipher import AVESEncoder, pool_mean, pool_mean_std

enc = AVESEncoder(
    weights_path="models/aves-base-bio.pt",
    config_path="models/aves-base-bio.json",
    device="cpu",
)

# Per-clip embedding: pool over time
frames = enc.embed_clip(samples_16k, layer=-1)
clip_vec = pool_mean(frames)            # (768,)
clip_vec_ms = pool_mean_std(frames)     # (1536,) — mean + std

# Batched
clips = np.stack([prepare_for_ssl(u.source_samples, u.sr) for u in units])
batch = enc.embed_batch(clips, layer=-1)

Use prepare_for_ssl to resample to 16 kHz and center-pad/crop to a fixed length before embedding.

AutoencoderEmbedder — convolutional autoencoder

Best et al. 2023 architecture: 128-bin log-mel frontend, 5-layer strided CNN encoder to a 256-dim bottleneck, optional decoder for spectrogram reconstruction.

from decipher import AutoencoderEmbedder

ae = AutoencoderEmbedder(
    weights_path="models/ae_best2023.pt",
    sr=22050,
    clip_seconds=2.0,
    device="cpu",
)

z = ae.embed_clip(unit.source_samples)   # (256,)
Z = ae.embed_clips([u.source_samples for u in units])

# Reconstruct the mel spectrogram from a batch of embeddings
mels = ae.reconstruct_mel(Z)

When to use which

Encoder	Strengths	Trade-offs
AVESEncoder	Cross-species transfer, no per-corpus training, well-validated frozen-feature benchmark.	Fixed input sample rate (16 kHz). Embedding dimension is large.
AutoencoderEmbedder	Trains quickly on a single corpus; the decoder lets you visualise what each dimension encodes.	Needs training data; representational geometry differs from SSL (anti-correlated by RSA).

Comparing embedding spaces

To check whether two embeddings carry the same structure over the same units, use the representational-geometry primitives in decipher.stats:

from decipher import rsa, linear_cka, rsa_permutation_null

# Pairwise distance matrices D_a, D_b over the same items
score = rsa(D_a, D_b)
cka_score = linear_cka(X_a, X_b)
null = rsa_permutation_null(D_a, D_b, n_permutations=500)