It's not quite as simple as some are portraying it here, and you may have more position improvement than some see here.
Consider what's happening here. You drop your front end and end up more aerodynamic. But in the process, you are changing both two critical measures, the angle of your hips to your thighs and your weight distribution. On the angle of your hips, this change will in practical measures change the power you deliver, for better or worse. Because it will demand a bit more flexibility in your hamstrings and glutes and lower back muscles, you may not see the improvement right away but be able to find significant improvement as you adapt to the position. That new position will change your saddle height (rotating your hips does that), potentially change your cleat position, and likely change your foot angle at the pedal. For what it's worth, you'll also change your rib cage position with regard to your diaphragm (which changes your lung capacity), and you'll change your shoulder angle (which can improve or worsen shoulder discomfort and also affect how you transition from seated to standing position).
On that last point, note that everyone has two completely different positions, one standing and one sitting. Because standing typically moves you forward, it changes your weight distribution significantly and when you are dropping your bars at the same time, it further accentuates this change. It may be an improvement, may not, depending on how well positioned you are already. Nobody writing a post on this thread is going to know, so don't let anyone dissuade you unless they have data to work with. Even just looking at the standing position, going lower in front will change the center of gravity of your upper body without changing the center of gravity of your lower mass (your hips don't really change location significantly with lower bars). Then three things happen (ignoring aero changes for the moment): First, you have more weight on the front wheel, which generally stabilizes your front wheel at speed and allows you to turn faster; second, you are using your upper body more to support your body, which affects how well your lower body can deliver power to the pedals (again, for better or worse, but only you will be able to test that); and third, you will have changed the position of your diaphragm relative to your thighs and rib cage, and may either improve or diminish effective lung capacity. Typically to maintain your sense of control over the bike, lowering the bars is helped by lengthening the reach (either long stem or greater handlebar reach); this has the added advantage of stretching out your torso slightly, which raises your rib cage and gives your lungs more capacity. There's a limit to this, but most of us aren't at that limit yet. This, by the way, is one reason why pros ride longer than usual stems and also slam them -- I've seen 10-20% improvements in vO2max just from the positional change (at least on the track where I've measured it).
Bottom line? You know yourself, or need to. If you think a smaller frame will work better, then absolutely take it seriously. Just understand that it will lead to a number of positional changes, some of which you'll have to acclimate to, but which can all improve your performance.