ORIGINAL: Prester John
I think that in Europe you see a continuous change in role for mounted troops. From heavy lancers (knights) and mounted men-at-arms changing tactics in response to bodies of pikemen mixed with the first man-portable firearms, you see changes roughly every hundred years. Each major cycle of war in Europe sees new tactics and roles for mounted troops. Only during the golden years of Napoleonics do you see the paper-rock-scissors situation for each branch of arms on the battlefield.
Also I don't want to counter my own point too quickly but cavalry are also the arm in some armies which most resist change due to their cost of re-equiping and traditions. Prussia still had cuirassiers (well trained and equipped) in the Franco Prussian War for example. I think it most likely that in the American wars dragoons were the easiest to train, and there were probably no traditions of "big men on big horses" hacking bits off infantry and gunners.
Added to those observations is the fact that the Cavalry regiments were often home to the higher class of officer - they were see as the elite units of the army, home to the oldest military families with a lineage going back to the mounted knight. This no doubt further entrenched their traditions and made change and adaptation even less likely.