Rotary Positional Embeddings Module

Rotary embeddings encode positional information in a way that allows continuous rotation of embeddings, enhancing the model's ability to capture long-range dependencies and positional relations.

Usage

RotaryEmbedding(
  dim,
  interleaved = TRUE,
  custom_freqs = NULL,
  freqs_for = "lang",
  theta = 10000,
  max_freq = 10,
  num_freqs = 1L,
  learned_freq = FALSE,
  use_xpos = FALSE,
  xpos_scale_base = 512,
  interpolate_factor = 1,
  theta_rescale_factor = 1,
  seq_before_head_dim = FALSE,
  cache_if_possible = TRUE
)

Arguments

dim

Integer. The dimension of the embeddings.

interleaved

Logical. If TRUE, uses interleaved rotation (default: TRUE).

custom_freqs

Tensor or NULL. Custom frequency tensor (default: NULL).

freqs_for

Character. One of "lang", "pixel", or "constant" (default: "lang").

theta

Numeric. Base scaling factor (default: 10000).

max_freq

Numeric. Maximum frequency for pixel-based embeddings (default: 10).

num_freqs

Integer. Number of frequencies for "constant" mode (default: 1L).

learned_freq

Logical. If TRUE, frequencies are learnable parameters (default: FALSE).

use_xpos

Logical. If TRUE, uses extrapolatable rotary embeddings (XPOS) (default: FALSE).

xpos_scale_base

Numeric. Base scaling factor used for XPOS (default: 512).

interpolate_factor

Numeric. Factor by which sequence length is interpolated (default: 1.0).

theta_rescale_factor

Numeric. Rescaling factor applied to theta (default: 1.0).

seq_before_head_dim

Logical. If TRUE, sequences are handled before head dim (default: FALSE).

cache_if_possible

Logical. If TRUE, caches computed frequencies (default: TRUE).

Value

An nn_module object.