PBEM Helper

[deMangler]

*** The Server is currently being migrated. It is down for now. I will update here ***


I made some software to simplify PBEM Games.

** Note Please ensure you are using pbem4se client at least version 0.0.11 if you can ** Older versions [s]will[/s] should still work though.


It automatically shares saves between players remotely with minimal user intervention, while playing the game.

It has a windows client which can be downloaded here. It also works in wine.

And a server that links up the clients is hosted at https://www.tgc37.uk/


From the readme:

PBEM4SE

A helper client for playing Shadow Empire PBEM games.


What it does

Enables seamless PBEM game play for Shadow Empire. Once set up, all players have to do is play the game normally.
Saves are exchanged between players automatically.

How it does it

A multiplayer game is configured on the server using a web page, where details such as number of players and turn order are specified. Once configured, the various players clients co-ordinate through the server to maintain game saves.
Each player's client is authorised to their secure account through an SHA-1 token to ensure only they can take their turns and the game saves are safe.

No, really, how does it do it?

It creates a specifically named blank file in your save folder, eg, yourname_b8a8_round1_overwrite_me.se1. When you save after your turn you overwrite this file, it notices, and will upload this to the server.
Then next player will receive this in their save folder as theirname_b8a8_round2_load_me.se1. which they load and play, they will have a file like theirname_b8a8_round2_overwrite_me.se1, to save their game over, and so-on.

Important!!! Make sure you load/overwrite the correct save, especially if you have more than one game on the go. the second part of the filename uniquely identifies the game.

Installing and setting up the client
Unzip to a folder where you have write access. If you are using windows then pbem4se_client.exe should just run.
The python version, once set up should run on most platforms.

You will need to give the client your unique authorisation token from the client settings page on the website. This can be done with "menu->server->change user", and tell it where to look for save game files using "menu->file->Choose Savefile Dir", within the client, or editing the pbem4se_settings.json file directly for the settings to persist.

Everything should be simple enough, the main points are, be a player in an activated game on the server - have your token from the server imported into your client - have your client pointed at a save folder.
Have the client running when you play especially when saving/loading games.

There are three tabs in the client.

Play Game Tab

Here you can select from games that have been created and choose by clicking in the dropown to select one to play.
Also displays state of current game, are we playing or waiting, what round is it, etc.

The Poll Server Checkbox needs to be checked for other players games to be automatically downloaded when they save their game. It can be unchecked to ease strain on the server if you want to keep the client running but are not playing.

If the The Delete Uploaded Saves checkbox is ticked then local copies of uploaded saves will be deleted. This makes sense as in PBEM games you cannot load a turn you have played. Untick to keep them just in case... (Remember, if in doubt always back up saves)

Note: If you are able to save between turns, make sure you do not overwrite the overwrite_me file! When this file is overwritten it is uploaded to the server as the next players download.


Create Game Tab

This takes you to the create game page on the website. You will be able to create games on this tab directly when I get
round to it.

Info Tab

This tells you the current state of the client and has a scrolling text log of recent events, which can help inform in case of difficulty, state of connection to server or issues with settings, etc.


Toubleshooting

Check the text in the info tab for hints if things are not working. Click menu>server>check server status and see what it says.

Make sure the authorisation token is correct.

Make sure the other settings in pbem4se_client.json are correct.

Make sure you have a game selected and the info in the Play Game Tab seems sane.

Make sure The Poll Server Checkbox is checked.


The Website

This is where games are created and shared between players. It acts as a single point of contact between the clients.

There is a bit of help on the website which will be more up-to-date.

There are basically three functions to the website.

1. Create an account and provide authorisation token for the client.

2. Choose people who you can play games with.

3. Create games.


The server itself functions to receive uploaded games from clients and download them. This automatically happens when a game is activated and chosen in a client.

There is no function to communicate with other players on the server. It is assumed that you have arranged for a game on the forum or are otherwise in contact with who you want to play with.

There may be some kind of lobby or player looking for game / starting game thing added if it seems to work.




There is a possibly more up-to-date readme here https://gitlab.com/deMangler/pbem4se-client/-/blob/release/readme.md

There are helpful notes in various places. This is just a thread I am starting so that I have something to link back to in the docs and on the site.

Please feel free to try it out and break things. I have tested it as much as one person can.

I will fix things and respond to feedback here.

The source code for the client is public at https://gitlab.com/deMangler/pbem4se-client. I will make the server source public when it is more finished it is all just getting started.

Edit - updated client link to v0.0.2. Just a small fix
Edit - updated client link to v0.0.3. fixed download url issue to not use debug path.
Edit - updated client link to v0.0.4. UI QOL tweaks.
Edit - updated client link to v0.0.5. tweaks.
Edit - updated client link to v0.0.6. Windows version no longer displays extra console.
Edit - updated client link to v0.0.7. Refactor and better logging. Fixed outdated game dropdown sometimes.
Edit - updated client link to v0.0.8. Reduced error spam in info pane.
Edit - updated client link to v0.0.10. Some optimisation and code clean.
Edit - updated client link to v0.0.11. Better client/server awareness... or something...

Help testing it appreciated.

[Malagant]

A great idea, thanks!

[deMangler]

ORIGINAL: Malagant

A great idea, thanks!

Thanks

I am hoping that if the concept works out and I can get the process ultra-simplified and tested a bit I can post it to the steam discussion. It might bring in some of those people who would only play with friends and to whom e-mail is a stone-age thing.

Give me stone-age any day when it comes to social media....

Anyway - if anyone wants to play a PBEM with me using this software I am totally up for it. I can only play one game at a time though and I warn you I may not be fast.

Then I can iron out the bugs and streamline everything. Also would motivate me to make the website a bit prettier...

[deMangler]

I am thinking that I could implement an ELO rating type system on the server - it would only work if when completing a game, all players agreed to tag the winner, to prevent abuse, and there would need to be a visible record of game history of games that had scored towards a particular ELO score, to prevent sock-puppet exploits.

Yep, I know standard ELO is two player zero-sum. That is why I am throwing the idea out there. There are versions that could work for Shadow Empire if the winning player counts as a win and others counts as a loss. What to so in shared wins?

Anyway, I guess if it is a popular enough idea that will become apparent here

[zgrssd]

ORIGINAL: deMangler

I am thinking that I could implement an ELO rating type system on the server - it would only work if when completing a game, all players agreed to tag the winner, to prevent abuse, and there would need to be a visible record of game history of games that had scored towards a particular ELO score, to prevent sock-puppet exploits.

Yep, I know standard ELO is two player zero-sum. That is why I am throwing the idea out there. There are versions that could work for Shadow Empire if the winning player counts as a win and others counts as a loss. What to so in shared wins?

Anyway, I guess if it is a popular enough idea that will become apparent here

The Youtuber "Spirit of the Law" did Videos on the ELO System as used in Age of Empires 2:
https://www.youtube.com/watch?v=GTaAWtuLHuo
including a recent fix:
https://www.youtube.com/watch?v=uLrfT8l1SYQ

However I do not think can work with anything but Fixed Teams or No Player Diplomacy - players allying would throw a wrench into it.

[deMangler]

ORIGINAL: zgrssd

ORIGINAL: deMangler

I am thinking that I could implement an ELO rating type system on the server - it would only work if when completing a game, all players agreed to tag the winner, to prevent abuse, and there would need to be a visible record of game history of games that had scored towards a particular ELO score, to prevent sock-puppet exploits.

Yep, I know standard ELO is two player zero-sum. That is why I am throwing the idea out there. There are versions that could work for Shadow Empire if the winning player counts as a win and others counts as a loss. What to so in shared wins?

Anyway, I guess if it is a popular enough idea that will become apparent here

The Youtuber "Spirit of the Law" did Videos on the ELO System as used in Age of Empires 2:
https://www.youtube.com/watch?v=GTaAWtuLHuo
including a recent fix:
https://www.youtube.com/watch?v=uLrfT8l1SYQ

However I do not think can work with anything but Fixed Teams or No Player Diplomacy - players allying would throw a wrench into it.

I think you are right.

[BlueTemplar]

Thank you, this should be very useful software !

You might want to contact the Zero-K/Spring devs :
http://zero-k.info/Forum/Thread/32347?p ... 972#230972

They have a *lot* of experience trying to figure out the best way to rank players in games going from 1vs1 to 32vs32 over more than a decade, and all possible combinations of more than 2 teams in-between. (And they'll be probably more available to answer questions than AoE2 devs ?)

But yeah, while IIRC there is *some* support for changeable in-game teams in Zero-K/Spring, I'm not sure that Elo/WHR is up to it ?

But maybe also it's not that much of a problem ? You just consider every player to be in his/her own team at the start of the game, the winner(s) go up, the loser(s) go down, and why shouldn't a player that is particularly good at diplomacy not get a higher multi-team score ?

[eddieballgame]

Where the Elo rating formulas work great for 'head to head' (1v1) competitions; maybe a ranking system similar to what Bridge uses could work out better.
Unlike Bridge, maybe some tweaks to implement losing points based on finishes in a 3 or 4 player game...or not.

& thank you for your work.

[deMangler]

Good input. Thanks.
I am looking forward to it getting used so that I can iron out those bugs that only appear when other people use the software, and then also I will be able to re-write the help so that people other than me can understand it.

I also have no idea if it works in windows 10 - I develop it in linux and build and test it using windows 7 in virtual machines. No reason I can think of why it wouldn't, but you never know...

Anyway - it's all about having fun. Can't wait to writing one of my AAR's from a multiplayer game.

[deMangler]

Having looked into the whole ELO thing a bit more, I think I could easily put something together that would work.
ELO itself is ok for the job really, it is just a matter of deciding how to weigh the outcomes so that the players are treated fairly.

For example, in a 1 v 1 game - player 1 wins, player 2 loses, so you just apply the 1 to player 1 and 0 to player 2 and do the ELO maths. Same for a draw, in this case both player 1 and 2 get 0.5 and you do the ELO maths with that.

With more than 2 players it is not much more complicated than making a decision and sticking to it - so I could say that losers always get .5/number of losers and winners get .5., or that winners always get 1/number of winners and losers get 0. Or something anyway it is not the main difficulty.

The main difficulty is that the starting conditions in Shadow Empire can be so tilted that you would need many, many games played for a system like ELO to start to give a fair appraisal. And Shadow Empire games can take a long time to play.

This does not mean that I won't implement it - it just means that I will implement it with that in mind.

On another note - I am still doing bits and bobs with the client and server, fixing obvious bugs and refactoring and simplifying, which is quite satisfying - like code gardening really.
It isn't really worth me doing much more actual development on it though because I have no idea even if it works as proof of concept, if the flow is correct, or even if the auth-token generation/account process actually works, or a dozen other unknowns. Without knowing those things I might do a lot of work that would need undoing or doesn't need doing at all.
Still there is no rush so I will keep connected to it and respond to all feedback and/or bugs as I think it could make a lot of difference having Shadow Empire multiplayer be pretty much seamless.

[zgrssd]

For example, in a 1 v 1 game - player 1 wins, player 2 loses, so you just apply the 1 to player 1 and 0 to player 2 and do the ELO maths. Same for a draw, in this case both player 1 and 2 get 0.5 and you do the ELO maths with that.

It is more complicated then that. If you compare the ELO scores, you should get a expected win chance for either player.

If Player A is 100 points higher then player B, A is expected to win 64% of the time.
If A wins, the transfer of points is minimal. A performed just about as we expected. B was hit by a poor match, him loosing is to be expected.

If B wins, the transfer of points is large. This is called a "upset win". Either A is rater higher then he should, B is rated lower then he should - or both.

The goal is for all participants to approach the ELO that actually represents their skillevel. There is a full Wikipedia Article on it and all:
https://en.wikipedia.org/wiki/Elo_rating_system

The main difficulty is that the starting conditions in Shadow Empire can be so tilted that you would need many, many games played for a system like ELO to start to give a fair appraisal. And Shadow Empire games can take a long time to play.

The cure to randomness is a large data set.
The more often you roll a die, the more average results you will get.

[deMangler]

We are in agreement I think, I just wasn't being clear. My talk of values between 0 - 1 were examples to show that the main thing is to decide equitably how to distribute the total value between players when there are more than 2, and just stick with that. That is then plugged into the ELO formula.
Yes, ELO is about calculating expected win percentages - that can be done when starting conditions are always the same and the players start with no handicap, or one that can be calculated.
The point I was trying to make is that in Shadow Empire, because the starting conditions vary so much between players and games, that any distortion introduced by making bad decisions about how to distribute the win/lose between more than two players and more than 1 win condition, would be insignificant compared to that, and in any case would need many, many games to become accurate, more so than something like chess or go.
Therefore, it is possible to implement, but only when taking that into account.

[BlueTemplar]

IIRC that's what that constant is for... was it named "K" ?

[deMangler]

ORIGINAL: BlueTemplar

IIRC that's what that constant is for... was it named "K" ?

Yep. Exactly.

However, what with shadow empire being what it is, just picking a reasonably sane starting value and sticking with it is a good place to start, there is no point trying to get it perfect because this approach is always going to be hand-wavey. We just need some consistency and for players to see how each others ratings change over time relative to each other for it to become meaningful. An absolute game-theory kind of meaningfulness isn't the goal, it is about players feeling like they have a basis for comparison that will reflect how their game-play is changing over time, or in relation to other players they may have played.... or something... hand-wavey....

But having said that - I have some ranking code already (thanks to the helpful input from here) that I can plug into the server, but without knowing if what I already have actually works there is little point in complicating the code. I think it is important to have the fundamentals solid and most importantly, secure first. Which I think is in a good place at the moment, this isn't my first client/server project.
I know from experience though that actual real world usage is what exposes the problems.

[deMangler]

Just a quick note to inform that the server is down for an upgrade/fix. Shouldn't be long.

** Edit It's back up now.**

[mroyer]

Geez, I missed this. Thanks deMangler for putting the effort into this.

I see you're using Tcl/Tk!! I use to use Tcl regularly back 25 years ago and loved it then. I'm surprised anyone is still using it; I assumed it was all but a dead language. We use to use it to control cable modem subscriber and infrastructure equipment for physical layer RF testing.

So, I do have one gripe -> both of my regular passwords (and variants thereof) got rejected. One is my simple password I use for gaming where security isn't a big deal - it got rejected for being too simple. The other is a very complex password that normally rates out strong or very-strong on other servers, but failed here due to 'too similar to user name'. It does happen to start with the same too letters, but after that there is little similarity.

As this is just a game, I'd recommend lowering the requirements on the password, a little.

Again, thanks for this!

-Mark R.

Edit: I did finally get in with an offshoot of my 'complex' password. (I try to keep the number of passwords I have to remember to a minimum - because of I still have a shot at entering correct credentials for accounts that are decades old [;)] ).

[mroyer]

Another recommendation:

In the Shadow Empire Generations screen, Passwords and PBEM option must be ticked, and disallow saving during turn should be ticked and the number of players must match

Let disallow saving be a user choice.
For example, it's not uncommon that I need to save during a turn as I will start a turn on one computer and finish on another (my wife or son might need our main office computer while I'm in the middle of play).

-Mark R

[deMangler]

Hi,
Glad the idea is good even if the implementation needs work!

You are the first person to use it apart from me so I apologise for bugs and usage problems. It has literally never been tested and I just knocked it up because it seemd there would be interest.


Having a strong password is preferable I think because of the possibility that games could be compromised. I am just using norms for security atm - no two factor auth.
I will put in other secure methods to make it easier, for example log in with google/discord/github etc.

Having disallow saving during turn is already optional, I just have not updated all the documentation. In some places there are warnings to let people know not to save over the overwrite-me file unless it is end of turn. I will update those docs and so-on.

I used TK because it is so simple and included in the standard library. I wanted it to be 'it just works' across platforms and I only have my linux development box and some VM's to test it.
Also I didn't know if anyone would use it so it is really just a proof of concept as it stands. Now that someone is using it I will start making it friendlier and make changes from what you have said.

Enjoy your game and please keep coming with the ideas and feedback.

[deMangler]

Just a quick update.
There have been a half dozen or so people using this over the last few days and I have had some feedback. Some people have had no problem and others have run into issues with the setup flow.

The first one to address seems to be the process of authorising the client with the server could be more seamless. In the works is a feature to make that easier.

Also, now that it is being used, I plan to improve the website massively - it is awkward to use until you get used to it - however, at the moment it works and I want to make sure nobody's experience is interrupted so I will implement changes very carefully with lots of testing beforehand.

In general the client seems to be working as intended so far so QOL improvements there mostly coming soon.

Thanks for the feedback



*edit* Oh yeah - Someone had a problem with their firewall, so if the server check in the client info tab says OK but player and/or game are not ok - your firewall is not the problem - probably an authorisation thing.
If the server check in the client info tab fails, and the website is loading then it could be the firewall.
click server > check server status to refresh the check.

[macroman247]

I feel as if I am 95% of the way there and just can't get to 100%. I might have a unique situation that is causing the problem. I own SE via Steam and want to play with my wife. Her on a laptop, me on another laptop. We only have the one Steam account and we can run SE just fine concurrently on the laptops...it just kicks the other person off Steam when you launch a second instance of SE, no biggie though...game doesn't close out.

I created 2 accounts on the site and each laptop is logged in and synced. Token is fine, no errors in log. I create the game and validate/activate it. Refresh list, both laptops can see the name of the game. I then start a new game in SE with our preferred setup. 2 human players, no saves during turn, password/pbem option ticked etc etc. Game starts, I set my password and take the turn. When I end the turn, the screen goes back to where you normally see the Player 1/2 it is your turn, login, save/quit yadda yadda...except it is just blank. I have the GUI of SE with the background and the X in the upper right, but that's it. I'm not sure where I have gone wrong here.

Update: I've mostly gotten past the issue with the blank screen. I can move the mouse over where the buttons should be and it causes them to appear. It's worth noting I do not have this issue in regular games and it happened on both laptops. Unfortunately, I'm still not quite able to get it working. I can now Save/Quit after Turn 1 and overwrite the overwrite me save file. It just doesn't seem to pass that file to the other laptop though (by means of downloading from the server). I have the game selected in the dropdown (still no error in log either) but there is no file for me to load.