I loaded up a save and I think I might have figured it out.

I did that go to nearest shipyard to retrofit and they went to a new colony. Which only had a build speed of 55 and I was also building the first spaceport for that new colony.

In latest game I tired to retrofit my construction ships. After two years I deleted them and tried to build three new ones. 2 years after that they were finally built.

Does anybody know why it would be so slow to build or retrofit?