I'm baffled by "post-normalized vision transformers don't not converge". Thoughts?

[alexander-soare]

I think I'm reading this for the second time now while looking at the CaiT paper. Section 2:

but in our experiments the DeiT training does not converge with post-normalization

The ViT paper has this line alone:

Layernorm (LN) is applied before every block

"Does not converge" is very significant, much more so than "performs slightly worse" would be. If the whole thesis of the vision transformer is so dependent on the placement of the norm layer, it's a BIG deal and needs to be discussed right?

Moreover, I wish I could ask the ViT authors how they figured this out. So it fails to converge the first time they try (because naturally they follow Vaswani). Then I'd love to know what process led them to figure out that going pre-norm fixes it.

[rwightman]

@alexander-soare There original NFNet (characterizing signal propagation) paper might have some insight... perhaps analysing the signal prop would provide clues...

[alexander-soare]

I haven't read it. Thanks for the lead!

[rwightman]

@alexander-soare another very recent paper on the topic, variance of activations in forward, and norm of gradient for backward are what they focus on for comparing normalizers in residual vs non-residual networks ... https://arxiv.org/abs/2106.05956

[alexander-soare]

Ah yes, saw that retweet. I have a longer list of norm questions (as everyone does) so this will be nice.