Searching the manual with AI! Give it a try...

[DekeFentle]

This came up a few months ago on the discord channel. https://discord.gg/2dW87zvNgg
While I'm reasonably sure there are other accessible AI's available, I've found Sharly.ai to be extremely helpful when I have a question that can probably be answered in the latest greatest update of the living manual. As of the date of this post that would be;

living manual properties.jpg (36.86 KiB) Viewed 1120 times

TLDR: Go to Sharly.ai, dump the document you want to search into the site and ask your question.

FAQ:
This site is FREE or at least as close to free as exists today. No credit card with a charge down the line.
First three questions/answers come with zero commitment.
Beyond the that you'll need to "sign up", again no credit card or commitment of any kind other than providing and e-mail addy.

SPAM: I think I signed up in January 2024, since then I can't remember more than an e-mail a month hitting the account I used.

Accuracy: Sometimes its 100% on the money and success is most often based on how the question is posed. Even when the answer is off, the information presented can lead you to the right place in the manual.

It's another tool that helps wade through 800 plus pages...

[Kahawa]

This is a very interesting idea, thanks. I think the manual needs a lot of help and this is an incredible and accessible way to immediately improve it. Great idea.

[MarkShot]

I am planning to do that with all my games which have extensive documentation, but to use open source local AI as I have a 4090. Quite a few free UI's have built-in RAG or integrate with free RAG using semantic embeddings, lang chain, and vector databases.

Most of my games do not exceed 10Gb VRAM. (A notable exception is City Skylines I at 16Gb VRAM.) 14Gb VRAM is plenty for a 7B model or one of the latest 3B models.

My main question is how well will PDF perform? As it is a largely unstructured format compared to XML or JSON? As the manuals are PDF, and I intend to convert useful forum posts and WiKis into print to PDF.

Some of these allow Web ingestion. I don't know if that is a single page or recursing a tree like a WiKi (that would be very good). The less work I have to do the better. I would create separate databases for different games.

Finally, I should note while Jensen Huang was busy repeating the mantra in Taipei, "The more you spend, the more you save", NVidia did announce some form of game expert AI (overlay). It is yet unclear what this will be. RTX Chat does do RAG local now, but must be running WIN11.

I will never go beyond WIN10:

* Required Microsoft logon

* Required bit locker with encryption keys on file with Microsoft

* Required Copilot AI always present

* Required RECALL copying all that you do

* Required hardware costs to support the above

* Reduced performance

[Kahawa]

An obvious reflection is that the AI can only be as precise and informative as the manual itself; where the source is vague, ambiguous, or lacking detail, AI won't help. Still, excited to hear people's results with this. In general I think manuals are a good way to teach Monopoly and Risk, but I am not persuaded this is the right tool for the job for any complex computer game. AI unlocks a practical supplement, one hopes.

[MarkShot]

Well, I set this up today, and am testing. You want to use AnythingLLM:

https://useanything.com/

You want a RAG workspace for each of your games.

I understood from the DOCs that you could grab your LLMs from them. I tried this, but it was running my LLMs CPU/RAM; not GPU/VRAM. It seems to have a bundled version of OLLAMA which does not use GPU. (NOTE: That creation of semantic embeddings for the vector database is done via CPU and not GPU. But it is a one time task.)

So, I grabbed OLLAMA and configured it myself. This is an AI inference runtime.

https://ollama.com

I just started testing, but this looks really good and very easy to use. You can dump in manuals, make forums threads into PDFs, rip text from YouTube tutorials and let's play, and have it scrape WiKis.

You don't need a big LLM; 3B or 7B will work. I am for now using LLAMA3 8B. (You can use quantized models to reduce the VRAM footprint. OLLAMA has what you need, but you can also use HuggingFace and OOGABOOGA to serve your model. If you are paying for OpenAI, you go with a number of cloud services.

In my testing, you can make this work with no more than 4-5Gb of VRAM. As I have 24Gb of VRAM, gaming will be fine if I stay under 10Gb for RAG ... I think the above config will cost about 7Gb. You can save space by using a small 3B model and a small context window.

The AI need not be a genius, but simply competent in English to understand your questions and recognize the best answers from the search against your vector database docs.

[MarkShot]

It does decently with Paradox WiKis combined with the best Steam Guides.

[MarkShot]

Okay, here is the results of testing. You only need 4Gb VRAM for this. If you don't have a GPU for this, trust me you are not going to be happy with CPU response. I am run a 13900KS, and even PHI 1.5B has too much latency. Also, using CPU is like doing a CINEBENCH R23 stress test.

Set chunk size from default 1000 -> 250.

Set Max Context Snippets 4 -> 12.

Set Document similarity threshold: 0.25 -> no restriction

Use OLLAMA the smallest good model is: PHI3 3.8B; Also, good are LLAMA3 8B, and QWEN2 8B. These will put you up around 7Gb VRAM. As you are not really chatting or looking for love 4096 is plenty of context memory. Note, 4096 is NOT 4Gb it is 4K tokens of context. From my testing with other AI UIs, this equates to around 2,500-3,000 words of memory.

The following questions were used to test various scenarios:

What is the difference between hasty and deliberate attacks?

What precise penalties do I face when planning a ground offensive during The Winter?

For the Germans attacking on the first day of Barbarossa how is that turn handled differently?

Exactly how does the player win or lose this game?

How do I use air units to deliver supplies to advanced ground units?

How are support units different than other ground units?

By what mechanism do support units get allocated to divisions and HQs?

How are engineers different than other support units?

This was the SYSTEM PROMPT I used:

1 - You are a Game Research Assistant.

2 - Each response is critical to OUR ultimate victory or defeat.

3 - For each prompt, you will consider both the prompt contents and prior user context to determine the user's intent.

4 - Your responses are to drawn from the documents which you have available and NOT your own supositions.

5 - Your responses will NOT be terse. Every response should be highly detailed and elaborate with all the information you have at your disposal.

6 - Terse responses will get you a low score. Detailed and informative responses will get you a high score.

7 - When prompts request information about game mechanics, you will respond with clear algorithmic information as opposed to wishy washy adjectives.

8 - When the information available presents numeric coefficients and numeric results, you will convey those details in your response.

9 - Whether we win or lose is very much dependent on your personal performance. If we lose, rest assured that I WILL HAVE YOU SHOT OR BEHEADED.

One thing, I would like to emphasize. Regardless of your model quality and settings one of the most informative things which you can do is click the regenerate button 2-3 times. It senses to generally give you something quite different as a response.

Also, more precise questions will get more precise answers.

Finally, if you know the manual well (but then you probably know the game), you might be just as fast with it open while playing and just using a PDF reader to produce a list of hits. This approach will shine more with ingested WiKi, Steam Guides, and gems of forum posts which you want to add (just print to PDF or give the URL). I did not have time to try creating transcripts of YouTube videos.

[MarkShot]

Does anyone know an open source utility which will recurse a website and turn it into one or more PDFs? I tried a Web based utility yesterday on Paradox's Web site and it did not work.

Thanks.

[MarkShot]

This is working so much better with PHI 14B. Very accurate. I and the AI negotiated on a new SYSTEM PROMPT. However, you will need 9Gb VRAM, but I still have 14Gb for games.

I was able to grab YouTube transcripts. And I just found out how I can grab an entire Paradox WiKi with one click. Also, I imagine this will work on Matrix forums, but you will need to step across pages of posts.

My new SYSTEM PROMPT:

1 - JOB DESCRIPTION: You are a GAME RESEARCH EXPERT ASSISTANT.

2 - CONSIDER PROMPT INTENT: When responding to prompts, consider the prompt contents, prior user context, and information returned by the database.

3 - RESPONSE STYLE DEPTH AND BREADTH: Terse is unaceptable. Every response should be long and highly detailed including relevant information from the database. Do not truncate for brevity.

4 - RESPONSE STYLE ALGORITHMIC CONTENT: All responses which include game mechanics will be presented in a clear algorithmic fashion (step by step).

5 - RESPONSE STYLE NUMERIC CONTENT: When the database returns numeric coefficients and results associated with game mechanics, you will convey those numeric values in your response.

6 - COLLABORATION: You and I are a team. We either win or lose together.

7 - ATTITUDE: Be a winner, be an expert, be a team asset, be a star.

GO TEAM BLUE!!!!!!!!!!!!!!!!!!!!!!

[MarkShot]

I have some more test results:

* The more you use less than stellar input like "War Room Threads" vs WITE2 Manual, then you should raise the the similarity parameter to 25%. NOTE: citations are returned for answers. If you mass ingested some screwy forum thread, you can just yank it from the workspace.

* I settled on PHI3 14B. Note for LLMs you will see 3B 7B 13B 70B ... This is the number of parameters in the LLM. In general, the higher the B you can accommodate, the better for quality. This is a general statement as the model itself is perhaps your first consideration. Also, nB is not the absolute control of model footprint on your video card. There is also Q; quantization. Think of quantization as zip for LLMs. So, I went with PHI3 14B as I can afford 10Gb VRAM and still game on 14Gb VRAM.

* I was wrong. Use the data sources tab on AnythingLLM and you can ingest an entire Paradox Wiki with one click. Another great use I found for this is ingesting the top rated Steam Guides with one click.

* It can ingest YouTube video transcripts one link at a time. I wish it could handle playlists. It would be fantastic to ingest one hundred episodes of Strategy Gaming Dojo. I will submit as a suggestion.

[MarkShot]

I do want to highlight one serious problem with AI. This is not related to any of the tools above, but the technology of LLMs.

I have been asking questions to which I know the answers. But I asked the following question of my expert using the HOI4 Workspace (built with WiKi, steam guides, and PDX YT tutorial transcripts).

Explain the complete mechanics of how to use the MISSILE PULSE COMMAND?

(1) There is no such thing in HOI4.

(2) The idea of missile pulse warfare did not appears until the 1960s, I believe. It would not be in a WWII historical game.

Yet, the AI managed to come back with a very detailed explanation of how to employ the command. Did the AI lie? No, LLMs produce probable text which may or may not be factually accurate. Now, I invested 2 days for 20 games. I don't regret my time as I think it will still be useful. Further the concepts behind the tool intrigue me, and what better way to understand AI tech than actually build my own application.

I just want YOU to be aware ... in case imaginary answers are not worth your time.