Release 2.1.0 — modern complex-valued architectures

CHANGELOG.md

@@ -5,6 +5,85 @@ All notable changes to `complextorch` are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.1.0]
+
+A large modern-architecture expansion: positional encodings, interference-aware
+attention, state-space models, unitary RNNs, learnable signal front-ends,
+complex KANs, and Steinmetz/analytic networks.
+
+### Added
+
+- **Complex positional encodings** (`complextorch.nn`): `RotaryEmbedding`
+  (relative RoPE — rotates per-head queries/keys by complex phasors so the
+  Hermitian attention score depends only on relative position),
+  `SinusoidalPositionalEncoding` (fixed absolute), and `CoPE` (lightweight
+  learnable absolute). `MultiheadAttention` gains an optional `rotary=`
+  argument and `models.ViT` a `pos_encoding=` selector
+  (`"learned"` / `"sinusoidal"` / `"rotary"`). The native transformer applies
+  no positional encoding on its own, so these fill a real gap. See
+  [Complex positional encodings](concepts/positional-encoding.md).
+- **Holographic (interference-aware) attention**: `HolographicAttention`
+  gates attention logits by the query/key phase discrepancy and performs a
+  coherent (phase-rotated) superposition of the values; selectable inside
+  `MultiheadAttention` via `attention="holographic"`. Adds the companion
+  `HolographicReconstructionLoss` and `phase_smoothness` regularizer
+  (anti-phase-collapse safeguards). After Holographic Transformers
+  ([arXiv:2509.19331](https://arxiv.org/abs/2509.19331)). See
+  [Holographic attention](concepts/holographic-attention.md).
+- **Complex diagonal state-space models**: `S4D` (HiPPO-Lin-initialised
+  diagonal-complex SSM with an FFT long-convolution and an exact recurrent
+  rollout), `DSS` (normalised-kernel variant), `S4DBlock` (residual block), and
+  `MambaBlock` (selective, input-dependent S6 scan). Linear-time long-sequence
+  modelling for the 1-D signals this package targets. After S4D
+  ([arXiv:2206.11893](https://arxiv.org/abs/2206.11893)), DSS
+  ([arXiv:2203.14343](https://arxiv.org/abs/2203.14343)), and Mamba
+  ([arXiv:2312.00752](https://arxiv.org/abs/2312.00752)). See
+  [Complex state-space models](concepts/state-space-models.md).
+- **Unitary complex RNN**: `UnitaryRNN` / `UnitaryRNNCell` — a norm-preserving
+  recurrence whose hidden-to-hidden matrix is the Cayley transform of a
+  learnable skew-Hermitian generator (eigenvalues on the unit circle), with an
+  `AdaptiveModReLU` nonlinearity and `trabelsi_independent_` semi-unitary init.
+  The classic complex-domain fix for vanishing/exploding gradients on long
+  sequences; complements the existing `GRU` / `LSTM`. After uRNN
+  ([arXiv:1511.06464](https://arxiv.org/abs/1511.06464)) and the Cayley/scoRNN
+  line ([arXiv:1707.09520](https://arxiv.org/abs/1707.09520)). See
+  [Unitary complex RNNs](concepts/unitary-rnn.md).
+- **Learnable complex time-frequency front-ends**: `STFT` / `InverseSTFT`
+  (short-time Fourier transform with a learnable analysis/synthesis window and
+  exact window-squared overlap-add reconstruction when the synthesis window is
+  tied to the analysis window) and `ComplexGaborConv1d` / `MorletConv1d`
+  (learnable complex Gabor/Morlet filterbanks — a complex, wavelet-style
+  analogue of SincNet). Differentiable signal front-ends that emit native
+  complex time-frequency representations. See
+  [Learnable time-frequency front-ends](concepts/time-frequency-frontends.md).
+- **Complex-Valued KAN**: `complextorch.nn.CVKANLayer` (a Kolmogorov-Arnold edge
+  layer whose univariate functions are a learnable Gaussian radial-basis
+  expansion over the complex plane, plus a complex linear base) and the
+  `complextorch.models.CVKAN` stack. After CVKAN
+  ([arXiv:2502.02417](https://arxiv.org/abs/2502.02417)). See
+  [Complex-Valued KANs](concepts/kan.md).
+- **Steinmetz & Analytic networks**: `complextorch.models.SteinmetzNetwork`
+  (parallel real-valued subnetworks with coupled outputs) and
+  `complextorch.models.AnalyticNeuralNetwork` (Steinmetz + an analytic-signal
+  consistency penalty that tightens the generalisation bound). Adds the reusable
+  `complextorch.nn.AnalyticSignalLoss` consistency penalty. After Steinmetz
+  Neural Networks ([arXiv:2409.10075](https://arxiv.org/abs/2409.10075)). See
+  [Steinmetz & Analytic networks](concepts/steinmetz.md).
+- **Signal utilities**: `complextorch.signal.analytic_signal` and
+  `complextorch.signal.hilbert` — a differentiable torch port of
+  `scipy.signal.hilbert` (analytic signal / Hilbert transform), reused by the
+  analytic-signal consistency penalty.
+
+### Fixed
+
+- `wFMConvStrict2d` now computes the phase mean as the **circular** (Fréchet)
+  mean on `SO(2)` — averaging the unit phase vectors and recovering the angle
+  via `atan2` — instead of an arithmetic mean of the raw principal-value angles.
+  The previous behaviour (a faithful port of RotLieNet's `ComplexConv2Deffangle`)
+  was wrong across the ±π branch cut; the circular mean is the correct manifold
+  Fréchet mean and makes the layer exactly U(1)-equivariant for any input phase
+  distribution.
+
 ## [2.0.1]
 
 ### Fixed

complextorch/__init__.py

@@ -10,7 +10,7 @@
 
 __author__ = "Josiah W. Smith"
 
-__version__ = "2.0.1"
+__version__ = "2.1.0"
 
 __all__ = ["datasets", "models", "nn", "signal", "transforms"]
 

complextorch/models/__init__.py

@@ -6,12 +6,17 @@
 """
 
 from complextorch.models.cds import CDSMSTAR, CDSEquivariant, CDSInvariant
+from complextorch.models.kan import CVKAN
+from complextorch.models.steinmetz import AnalyticNeuralNetwork, SteinmetzNetwork
 from complextorch.models.vit import ViT, ViTLayer, vit_b, vit_h, vit_l, vit_s, vit_t
 
 __all__ = [
     "CDSMSTAR",
+    "CVKAN",
+    "AnalyticNeuralNetwork",
     "CDSEquivariant",
     "CDSInvariant",
+    "SteinmetzNetwork",
     "ViT",
     "ViTLayer",
     "vit_b",

complextorch/models/kan.py

@@ -0,0 +1,46 @@
+r"""
+Complex-Valued Kolmogorov-Arnold Network (CVKAN)
+================================================
+
+A KAN-style stack of :class:`complextorch.nn.CVKANLayer` edge-function layers.
+See https://arxiv.org/abs/2502.02417.
+"""
+
+import itertools
+
+import torch
+import torch.nn as nn
+
+from complextorch.nn.modules.kan import CVKANLayer
+
+__all__ = ["CVKAN"]
+
+
+class CVKAN(nn.Module):
+    r"""
+    Complex-Valued Kolmogorov-Arnold Network.
+
+    Stacks :class:`complextorch.nn.CVKANLayer` layers according to
+    ``layer_sizes`` (e.g. ``[in, hidden, out]``). Operates on complex tensors of
+    shape ``(..., layer_sizes[0])`` and returns ``(..., layer_sizes[-1])``.
+
+    Args:
+        layer_sizes: feature sizes from input to output (length >= 2).
+        num_grid: grid resolution per axis for each :class:`CVKANLayer`.
+    """
+
+    def __init__(self, layer_sizes: list[int], num_grid: int = 8) -> None:
+        super().__init__()
+        if len(layer_sizes) < 2:
+            raise ValueError("layer_sizes must have at least two entries (in, out)")
+        self.layers = nn.ModuleList(
+            [
+                CVKANLayer(a, b, num_grid=num_grid)
+                for a, b in itertools.pairwise(layer_sizes)
+            ]
+        )
+
+    def forward(self, input: torch.Tensor) -> torch.Tensor:
+        for layer in self.layers:
+            input = layer(input)
+        return input

complextorch/models/steinmetz.py

@@ -0,0 +1,126 @@
+r"""
+Steinmetz & Analytic Neural Networks
+====================================
+
+Architectures that process complex-valued data with **parallel real-valued
+subnetworks** whose outputs are coupled into a complex latent (Steinmetz), with
+an optional **analytic-signal consistency penalty** (Analytic Neural Network)
+that encourages a deterministic, orthogonal relationship between the real and
+imaginary channels and provably tightens the generalisation gap.
+
+Reference:
+
+    - **Venkatasubramanian, Pezeshki, Tarokh. Steinmetz Neural Networks for
+      Complex-Valued Data.** AISTATS 2025. https://arxiv.org/abs/2409.10075
+"""
+
+import torch
+import torch.nn as nn
+
+from complextorch.nn.modules.loss import AnalyticSignalLoss
+
+__all__ = ["AnalyticNeuralNetwork", "SteinmetzNetwork"]
+
+
+class _RealMLP(nn.Module):
+    """Plain real-valued MLP (Linear -> ReLU stack)."""
+
+    def __init__(
+        self, in_features: int, hidden_features: int, out_features: int, depth: int
+    ) -> None:
+        super().__init__()
+        layers: list[nn.Module] = []
+        size = in_features
+        for _ in range(max(depth - 1, 0)):
+            layers += [nn.Linear(size, hidden_features), nn.ReLU()]
+            size = hidden_features
+        layers.append(nn.Linear(size, out_features))
+        self.net = nn.Sequential(*layers)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return self.net(x)
+
+
+class SteinmetzNetwork(nn.Module):
+    r"""
+    Steinmetz Network
+    -----------------
+
+    Two **parallel real-valued** subnetworks consume the stacked
+    ``[Re(z), Im(z)]`` features and produce the real and imaginary parts of a
+    complex latent, which are coupled with :func:`torch.complex`:
+
+    .. math::
+
+        u = f_\Re([\Re z, \Im z]), \quad
+        v = f_\Im([\Re z, \Im z]), \quad
+        \hat{z} = u + j v.
+
+    Args:
+        in_features: number of complex input features.
+        hidden_features: width of the real subnetworks.
+        out_features: number of complex output features.
+        depth: number of linear layers in each subnetwork.
+    """
+
+    def __init__(
+        self,
+        in_features: int,
+        hidden_features: int,
+        out_features: int,
+        depth: int = 2,
+    ) -> None:
+        super().__init__()
+        self.real_branch = _RealMLP(
+            2 * in_features, hidden_features, out_features, depth
+        )
+        self.imag_branch = _RealMLP(
+            2 * in_features, hidden_features, out_features, depth
+        )
+
+    def forward(self, input: torch.Tensor) -> torch.Tensor:
+        r"""Map a complex input to a complex output via the coupled real branches.
+
+        Args:
+            input (torch.Tensor): complex ``(..., in_features)`` tensor.
+
+        Returns:
+            torch.Tensor: complex ``(..., out_features)`` tensor.
+        """
+        feat = torch.cat([input.real, input.imag], dim=-1)
+        return torch.complex(self.real_branch(feat), self.imag_branch(feat))
+
+
+class AnalyticNeuralNetwork(SteinmetzNetwork):
+    r"""
+    Analytic Neural Network
+    -----------------------
+
+    A :class:`SteinmetzNetwork` paired with the analytic-signal consistency
+    penalty (:class:`complextorch.nn.AnalyticSignalLoss`). Add
+    :meth:`consistency_loss` of the network output to the task loss during
+    training to push the latent towards a true analytic signal
+    (:math:`\Im(\hat z) = \mathcal{H}\{\Re(\hat z)\}`), which the paper shows
+    lowers the generalisation-gap bound relative to a generic Steinmetz network.
+
+    Args:
+        in_features, hidden_features, out_features, depth: see
+            :class:`SteinmetzNetwork`.
+        consistency_dim: signal dimension for the Hilbert transform in the
+            consistency penalty.
+    """
+
+    def __init__(
+        self,
+        in_features: int,
+        hidden_features: int,
+        out_features: int,
+        depth: int = 2,
+        consistency_dim: int = -1,
+    ) -> None:
+        super().__init__(in_features, hidden_features, out_features, depth)
+        self.consistency = AnalyticSignalLoss(dim=consistency_dim)
+
+    def consistency_loss(self, z: torch.Tensor) -> torch.Tensor:
+        """Analytic-signal consistency penalty of a (network output) latent."""
+        return self.consistency(z)