Nope. You're not losing your mind. Aluminum lacks memory. Meaning that it doesn't snap back into place the same way steel or titanium do. It also doesn't deflect the same way the other two metals do. This is what gives aluminum it's stiffness advantage over damn near every other material out there. It will, however, soften over time. Mind you, for most people who aren't a protour sprinter, we're talking years and years. It'll also depend on the quality of alloy and heat treating. Better quality alloy and heat treating = better memory.
Memory, for real? WTF?
Aluminium is not a stiff material, whatever that means. It is a light material and can be reasonably manipulated into shapes.
No again, aluminium, more appropriately alu alloy, will harden over time due to work. Harden as in brittle. Not soften.
When you see problems with alloy frames, it's usually to do with the welds due to the process at junctions. The material won't just fail out of nowhere.