Yes, for sure, the outer layer is important.
Most of the time I have a LS Santini jersey and a light wind vest with a mesh back.
Or a craft windstopper jacket.
But is is still too much most of the time.

When I go snowboarding, the whole wicking thing works. When I cycle, not.