Findings: Testing Puresim's Default XML

[BryanK]

I created a 16-team, fictional-player league to test puresim's default XML.
To summarize what I (think I) found after 40 years:

1. After the first ten years, puresim generates consistent stats despite the presence of different multipliers in the XML. In general stats are in line with 1990ish MLB baseball
2. HR totals are too high
3. In a number of cases, the distribution of individual player totals for a given statistic are skewed higher than in real life (HR, IP, K), but there are fewer outliers; i.e. no one breaks out and has a special year

Here are the complete findings: http://home.comcast.net/~zbryanknight/T ... im_XML.ppt

Here is the source data: http://home.comcast.net/~zbryanknight/X ... sheets.zip

If you would like the league file, just ask.

What does everyone think?

[KG Erwin]

Despite my earler complaints, I think that the models are good, taken in the long run. In the short term, some weirdness may happen -- for the few guys, like me, who want to manage each game of my chosen team, these anomalies become less noticeableWHEN REAL PLAYERS are chosen.

Maybe the fictional player generation routine needs some tweaking.

[BryanK]

Tweaking the fictional player generation is kinda what I was thinking.
I'm not sure what percentage use real vs. fictional players. I think a lot of Puresim's development lately has focused on real players, but there's undoubtedly some of us who still use fictional players...

That said, given all the interactions that are going on within the models, I think they're simply suberb. If I could build things this good at my job... well, I'd be retired at the ripe old age of 24

[BryanK]

should I make more slides?

[puresimmer]

Holy cow! That is AMAZING work! I am not use to this message board yet and didn't realize the those were links in your message.

I am going to download and save them. All in all overall things look pretty good. I think it looks like it tracks history fairly well in some cases and not in others but I need to do more looking at your numbers. I'd like to see the next 50 years simmed as well

After we get this version out the door and I have room to breathe I am really going to dive in.

Great stuff! Keep it coming!

[BryanK]

Thanks Shaun

I don't think I'll be able to get back to this until the end of the week when I (finally) get some vacation. In the mean time, I'm going to let PS run through until 2005 so that I have some data to work with. As an added validation, I want to start another league in like 1950 and run that one through until 2005 as well. That way it would be possible to tell if things like the ramp up in HR production is due to XML factors or small differences in the types of players PS generates in the initial draft vs. annual june drafts.

At a high level, I think this shows that the game engine Shaun put together is pretty rock solid. Given the potential for interaction effects, it's really cool to see the OBP, AVG, and Runs/Game figures track as close as they do.

The distributions are a little off, but that's a tricky problem to articulate, and no doubt more tough to nail down. Maybe an example would help me get across what I'm thinking. In the major leagues right now the league leader will hit about 45 HR or so, and there's about 1.2 or so players per team who will hit 20+ HR. Now to get a league leader around the 45 HR mark in PS, all the tests that I've done give me about 3 players per team that hit 20+ HR. As a result, the total number of HR hit in the league gets elevated.

I'll get crackin on the additional slides as soon as I can, and hopefully have some stuff by this time next week. In the mean time, if anyone has any questions or ideas about things to test, please let me know


Random thoughts:
1. It seems like the XML is having some effect on the numbers, but the magnitude is smaller than anticipated.
2. 3B seems to track the best, and I think I remember Shaun noting in one of the release notes that he specifically went in and tweaked the 3B totals.
3. Would it make any sense to model pitcher fatigue by year to help replicate the fact that pitchers today throw fewer innnings?