This is an online seminar. Registration is required.
【 High-dimensional Causal Analysis Team】
【Date】2023/August/28(Mon) 10:00-11:00(JST)
*【Speaker】Hanna Tseran, Leipzig University *
Title:
Expected Complexity and Gradients of Maxout Networks and Implications to Initialization
Abstract:
Learning with neural networks relies on the complexity of the representable functions but, more importantly, the particular assignment of typical parameters to functions of different complexity. Taking the number of activation regions as an expressivity measure, we show that the practical complexity of networks with maxout activation functions is often far from the theoretical maximum. Continuing the analysis of the expected behavior, we study the expected gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution. We observe that the distribution of the input-output Jacobian depends on the input, which complicates a stable parameter initialization. Nevertheless, based on the moments of the gradients, we formulate parameter initialization strategies that avoid vanishing and exploding gradients in wide networks.
