https://www.arxiv.org/abs/2105.02716
Develop a theoretical framework studying the geometry of learning dynamics, reveal a key nechanism of explicit symmetry breaking behind the efficiency and stability. Model the discrete learning dynmaics of gradient descent using a continuous time Lagrangian formulation, and identify kinetic symmetry breaking, and generalize Neother's theorem to KSB and show how KSB introduces a mechanisms of implicit adaptive optimization.
https://www.arxiv.org/abs/2111.05529
Studies the generalization benefit of model invariance by introducing the sample cover induced by transformations. Proveide refined generalization bounds for invariant models based on the sample cover.
https://www.arxiv.org/abs/2103.16634
Introduce a feature transform techniques that imposes invariance properties. Enforce GL(n) invariance property with global statistics from a batch.
https://www.arxiv.org/abs/2106.02346
From function space perspective, derive non-zero generalization benefit of incorporating invariance in kernel ridge regression when target is invariant to the action of a compact group.
https://www.arxiv.org/abs/2007.02933
Present a method for learning and encoding equivariances into networks by learning corresponding parameter sharing patterns from data, which can provably represent equivariance inducing parameter sharing for any group of symmetry transformations.
https://www.arxiv.org/abs/1901.09342
Present two main results, any subgroup G of Sn are universal if high-order tensors are allowed, and there are groups that higher-order tensors are unavoidable for universality.
Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks
https://www.arxiv.org/abs/2201.07934
Define adaptive symmetry, which characterizes invariance of variance, where a system explores different pathways of evolution with equal probabilibty, and complex functional structure emerges from accumulation of adaptive symmetry breaking. Characterize the optimization process of DNN system as an extended adaptive-symmetry breaking process.
https://www.arxiv.org/abs/2202.00565
Show that emergence of pattern that FC network learns convolutional structure, is due to non-Gaussian higher-order local structure of inputs. Provide an analytical and numerical characterisation of the pattern formation mechanisms.
https://www.arxiv.org/abs/2202.10638
Develop gradient-based method for selecting the data augmentation. Relies on phrasing data augmentation as an invariance in the prior distribution and learning it using Bayesian model selection. Use a differentiable Kronecker-factored Laplace approximation to the marginal likelihood as objective.
https://www.arxiv.org/abs/2202.12439
Following the previous approach that marginal likelihood offers a way to learn invariances in GP, propose a weight-space equivalent approach, by minimizing a lower bound on the marginal likelihood to learn invariances.
https://www.arxiv.org/abs/2106.10163
Derive G-steerability constraint that completely characterizes partial differential operators is equivariant, and fully solve the constraint for some important groups. Use this solution as equivariant drop-in replacement for convolution layers, and develop a framework for equivariant maps based on Schwartz distributions.
https://www.arxiv.org/abs/2205.09219
Prove a theorem that classifies all G-invariant single-hidden-layer neural network architectures with ReLU activations for any finite orthogonal group G. The proof is based on a correspondence of every G-SNN to a signed permutation representation of G acting on the hidden neurons, which is given in terms of the first cohomology classes of G. Finally prove that architectures with inequivalent cohomology classes in a given cohomology ring coincide only when their weight matrices are zero.
https://www.arxiv.org/abs/2207.03485
Study the operators mapping vector and scalar fields over a manifold that commutes with its group of diffeomorphisms. Prove that for scalar field, these operators correspond to point-wise non-linearities, while the vector fields only allows scalar multiplications. These justify the use of nonlinearities, however also indicates the diffeomorphism is too rich to motivate design over the symmetries of manifold.
https://www.arxiv.org/abs/2208.08707
Study the approximation of functions which are invariant w.r.t. certain permutations, using flow maps. Prove sufficient condition for universal approximation, vieweing the abstraction of deep residual networks with symmetry constraints.
https://www.arxiv.org/abs/2110.06084
Show that L-layer full-width linear, group convolution neural networks trained with gradient descent on binary classification, converge to solutions with low-rank Fourier matrix coefficient. This generalize the results in CNN to G-CNN for all finite groups.
https://www.arxiv.org/abs/2210.05087
Propose a novel structure-preserving neural network to approximate nearly-periodic symplectic maps, which gives rise to a discrete-time adiabatic invariant and a long-time stability.
https://www.arxiv.org/abs/2205.14819
Show the universality of depth-2 group convolutional neural networks based on the ridgelet theory. By formulating the GCNN parameter realizer function, find the analysis operator named ridgelet transform that maps a given function to parameter space. This includes permutation invariant, E(n)-equivariant networks.
https://www.arxiv.org/abs/2008.06952
View the permutation-invariant network as function over probability measures, and establish approximation and generalization bounds under different regularizations.
https://www.arxiv.org/abs/2210.15279
Prove that the invariant functions can be universally approximated by deep neural networks, including complex-valued NN, CNN, and Bayesian NN.
https://www.arxiv.org/abs/2211.07482
Propose to use fusion diagram which simulates SU(2) symmetric quantum many-body problems, to design equivariant component for equivariant NN. Show that these blocks are universal approximators of any continuous equivariant function.
https://www.arxiv.org/abs/2211.08771
Consider the infinitely wide neural network's gradient flow, and study the effect of symmetries on the learned parameters. Describe the class of symmetries, are preserved the dynamics. For special cases, show that odd true function gives linear predictor's dynamics, and low-dimensional structure have lower-dimensional PDE dynamics.
https://www.arxiv.org/abs/2212.08648
Show that Schur-Weyl duality between partition algebra and the symmetric group results theoretical foundation for all possible permutation equivariant neural networks.
https://www.arxiv.org/abs/2212.09507
Study the generalization capacity of group convolutional neural networks, via the VC dimension. For infinite groups and arbitrary convolution kernel, show that two-parameter family have an infinite VC dimension.
https://www.arxiv.org/abs/2209.08371
Prove that the many-channel limit of CNN equivariant w.r.t. two dimensional Euclidean group with vector valued neuron activations and the corresponding equivariant Gaussian processes.
https://www.arxiv.org/abs/2304.14144
Show that category theory can be used to view the group equivariant layer whose layers are tensor power space, and build richer structures with algorithm for quivkly computing the equivariant linear layer.
https://www.arxiv.org/abs/2301.10152
For layers with tensor power of R^n, find a basis of matricse for learnable linear An equivariant layer between such tensor power spaces.
https://www.arxiv.org/abs/2212.08630
Provide full characterisation of equivariant layer on tensor power of R^n, for orthogonal group, special orthogonal group, and the symplectic group, with spanning set of matrices.
https://www.arxiv.org/abs/2303.04198
Considering the linear equivariant steerable neural network under gradient flow, show that thery converges to the unique group-invariant classifier with maximum margin with input group action. Also with unitary assumption, show the equivalence of steerable network and data augmentation, and demonstrate improved margin and generalization bound over non-invariant counterparts.
https://www.arxiv.org/abs/2305.13517
Show that group-invariant has reduced sample complexity, by a power of group size and power depends on the intrinsic dimension of the distribution's support.
https://arxiv.org/abs/2303.04745
Consider the pointwise symmetry instead of symmetry over all data domain, then study the impact of incorrect or extrinsic symmetry on model error, and error lower bound when there exists partially incorrect symmetry.
https://arxiv.org/abs/2303.04198
Show that the linear equivariant steerable network in group-invariant binary classification converges to the solution that is equivalent to data augmentation, which has better margin and generalization bound than non-invariant counterparts due to its equivalence to group averaged dataset.