I started builds of armored facilities at a large number of your planets, and found the same disparity in build times for those. Colonies such as Ilyseom 4 or Sulanos 4, with ~100% quality, a 5.7B population, and 135% culture finished when a similar planet with 1.1B population was at 25%. A new colony with 200M ppl on a 66% planet with 40% culture was only 5% done in that same period of time. I suspect the pirate build speed follows the same pattern. They are, after all, corrupting the colony it's being built at, and it would make sense if that's what's happening. Do you recall which planets you saw the <20s build times?
I guess I'd just assumed the bases were supposed to be annoying. Consider this though - the early ones aren't doing much other than increasing corruption at colonies where you don't appear to usually be taxing the planet anyway. Also, they provide an excellent opportunity for you to be getting experience for your troops and that general that has the ground attack strength skill, and to get new troop generals.
On a side note, if you are having financing trouble, you may wish to consider switching to a democracy or something at this stage. Even with the immediate drop in culture from switching governments, when I did that I saw a ~200k difference in your cashflow, and it comes with the 25% research bonus as well. With that much of a difference, you could probably manage 4 or 5 troops per planet (at least the better ones). Or go take over that sluken planet nearby and have them supply your empire with troops.