https://www.arxiv.org/abs/1910.01635
When Euclidean norm of the weight is bounded for infinite width network, show that L1 norm of Radon transform of (d+1)/2 power Laplacian is related, and show that any function in Sobolev space can be represented with bounded norm.
https://www.arxiv.org/abs/1910.02333
Through spline theory, study importance of other activations like leaky-ReLU, weight decay or path norm regularization, and skip connection, using relation between NN training and infninite dimensional optimization over Banach space whose solutions are well known to be fractional and polynomical splines, and the choice of Banach space depends on the choice of activation.
https://www.arxiv.org/abs/2006.05626
In continuous domain linear inverse problems with total-variation-like-regularization in Radon domain, derive a representer theorem that finite width single layer NNs are solution.
https://www.arxiv.org/abs/1902.05040
Show that minimal network norm for representing f with bounded norm is max(int |f''|dx, |f'(-\infty) + f'(+\infty)|), so minimal norm fit is linear spline interpolation.
https://www.arxiv.org/abs/1906.08039
Define the Barron space, and show that it is the right space for two-layer neural network models, in the sense that optimal direct and inverse approixmation theorems hold for functions in the Barron space. For residual network model, construct flow-induced function spcae, and prove direct and inverse approximation theorems. Also show the Rademacher complexity for bounded sets under these norms has the optimal upper bounds.
On the Banach spaces associated with multi-layer ReLU networks: Function representations, approximation theory and gradient descent dynamics
https://www.arxiv.org/abs/2007.15623
Develop Banach spaces for ReLU neural networks, containing all finite fully connected L-layer networks with L2 limiting objects under bounds on natural path norm. Under this norm, the Rademacher complexity on unit ball is low, giving good generalization properties.
https://www.arxiv.org/abs/2005.07275
Explore a formulation of variational inference by exploiting that most PDFs are in Bayesian Hilbert space, and show that variational inference based on KL divergence amounts to an iterative projection of the Bayesian posterior onto a subspace corresponding to the selected approximation family.
https://www.arxiv.org/abs/2004.06989
Show that the function space generated by multi-layer networks with non-expansive activation functions is smooth. Then under band-limited input assumption, provide novel error bounds for univariate neural networks.
https://www.arxiv.org/abs/2102.12238
Provide a function space characterization of the inductive bias resulting from l2 norm of the weights in multi-channel CNNs with linear activations. Show that if the inputs are single channeled, the induced regularizer is independent of the number of output channels and is a norm given by a semidefinite program. For multichannel input, multiple output channel can be necessary to realize all linear functions and the inductive bias does depend on output channels. The regularizer is again given by a SDP that is independent of output channel.
https://www.arxiv.org/abs/2105.03361
Develop a variational framework to understand the properties of functions learned by fitting deep neural networks with rectified linear unit activations to data. Propose a new function space, that captures the compositional structure associated deep neural networks. Derive a representer theorem showing that deep ReLU networks are solutions to regularized data fitting problems over functions from this space.
https://www.arxiv.org/abs/2105.13462
Show that SGD tends to impose constraints on high-order moments of the gradient noise, by a linear analysis of SGD aroung global minima. Identify Sobolev regularization effect of SGD, that SGD regularizes the Sobolev seminorms of the model functions w.r.t. the input data.
https://www.arxiv.org/abs/2211.05020
Barron spaces used to prove generalization bounds for neural network, is not Hilbert space due to the strong nonlinear coupling of the weights. Show that this can be understood as an infinite union of RKHS spaces, and the dual space is again an RKBS with role of data and parameters interchanged.
Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networks
https://www.arxiv.org/abs/2006.07356
Show that for wide enough network, training on the adjusted dataset with gradient descent converges to zero solution error. Moreover, there exists some variational solution which fits the training data while minimizing some curvature penalty for difference of its intiialization, that the difference of it to solution is bounded by n^(-1/2). This formulation applies to other activations, stochastic GD, and multivariate regression also.