After conqering about half of the country (including Warsaw) and destroying most of their army, Poland surrendered to me.
They surrendered when you captured their capital, which I believe in 1792 is Warsaw. This is WAD.
However, in THIS 1792 game mentioned above, Prussia declared war on Poland and Sweden became its protector. I expected that Austria and I (Prussia) need only inflict enough punishment on Poland to get it to drop out of the war and surrender to me. However, I found myself in a war of total extermination, where they would NOT surrender. Period. Austria (after re-declaring war) and Prussia were forced to take every single provence which belonged to Poland to wipe them out and absorb them into our crowns.
Even when they are a protectorate, you still need to either capture the capital or get the protector (Sweden) to surrender, and then you could ask for provinces in a peace treaty.
To be honest with you, I expected them to surrender to me (having taken their capitol and inflicted the most damage)
If you really did capture the capital, then you would have gotten control of the Polish player, their capital province, plus any other provinces you'd captured. However, any remaining Swedish-controlled provinces would have then become Swedish-owned, i.e. conquered and directly controlled by Sweden. The dotted line should have disappeared, leaving a solid Swedish border only. But only after you captured the Polish capital (again, Warsaw, I think).
Or, perhaps, ideally, a special section of code would be written for the special case of Poland/Larger countries (with "historical" options perhaps for Poland).
Poland is a very large minor power, it's true, but I think it behaves appropriately--if it's a protectorate, its lands will stay in Swedish hands if at all possible.