There is no Six13 in size 53 and there never was. Size 52 horizontal top tube measurement is 53.5mm so that could be the source of confusion.
Only you can make the decision whether or not to downsize, you absolutely should not let anyone on the Internet decide that for you. That said (
) here's some things you might want to take into consideration:
You already rode a Cannondale, it was " a little on the big side but fit decently" - what does this mean? Did you have to use a slammed 90mm stem to get your handlebar where it needs to be, or was the stand-over clearance a bit too low for your liking? Again, was the Six13 a pre-2008 model or not? You say you don't like a large spacer stack - the 50cm CAAD10 will have the head tube some 19mm shorter than the 52cm 2008 Six13, but only 5mm shorter than a pre-2008 model.
You say 51cm Cervélo R3 fit you well - they changed the geometry between seasons 2010 and 2011, extending the head tube. You can easily find geometry sheets for both versions and then compare stack and reach with the CAAD10 geometry. Checking casually, I see the CAAD10 is lower and longer than both versions of Cervélo, so you again have to decide what's the minimum stem length and the maximum spacer stack you're willing to work with.