Characterize activation functions that are γ-parameter bounding

Determine the class of activation functions beyond the Rectified Linear Unit (ReLU) and the Heaviside step function that are γ-parameter bounding, meaning they permit universal approximation by single-hidden-layer feed-forward neural networks in which every individual scalar parameter (each weight and bias) is bounded within the interval [-γ, γ] for a given γ > 0, with approximation measured in an L1 norm over compact input domains.

Background

The paper introduces the notion of a γ-parameter bounding activation: an activation function that allows universal approximation even when each individual parameter (weights and biases) is bounded in magnitude by γ. The authors prove that ReLU and the Heaviside step function satisfy this property but do not identify other activations, noting the relevance of parameter-bounded approximations to theoretical limits and practical initialization regimes.

Identifying which other activations meet this property would clarify the generality of the results and help bridge to related findings on bandlimited parameters and random networks, informing both theory and practical design choices for bias-learning and masking-based approximations.