AGElint -- an AGE debugging toolkit

[Chilperic]

Solved. It wasn't lack of a new line but strange invisible characters at the start of the file. I've got the idea to open the file in word to check if this sort of thing had happened. Bingo . Now working

[berto]

Cool.

Sorry that you're the guinea pig, but from experiences like yours, I can code in more flexibility, and make the instructions clearer still.

I'm multitasking (working on my early music website this afternoon), so I'll be available here for further Q&A as needed.

Good luck.

[Chilperic]

Working as a charm.

Time to fix bugs.

Thanks

[berto]

<thumbs up> [8D]

[Chilperic]

ANAAI file fixed. IMHO there are maybe some false bug report, but I will investigate further before signalling them. However, it rports real bugs too

[berto]

AGElint reports

• real errors (of greater or lesser importance)
• possible errors (maybe or maybe not real, due to ambiguity or difference of opinion)
• false positives

False positives might have many different causes. To be discussed later.

Where we determine the false positives, they will be removed or suppressed.

(Historical note: In AGElint's early development, every step of the way, I acknowledged the problem of false positives. AGElint was sometimes attacked for its false positives. But if I am overly diligent at squelching possible false positives, there is a danger of squelching true positives, reports of real bugs. Refining the code is an iterative, approximative, arduous process. "There Will Be Blood." And we must not forget: "Never let the pursuit of perfection obstruct progress." Right? [;)])

[berto]

It deserves repeating:

Please bear in mind that, for all of its current sophistication, AGElint is a work-in-progress, more than a beginning but far from an ending.

The parser (txt.y) still needs much filling out, and deepening. For now, it’s like the parser understands grade school French. I/we need to teach it high school or college French. The deeper AGElint’s understanding of “French”, the more errors it will detect.

Beyond syntax and semantic errors, there are logic errors. For now, AGElint checks for just a small handful of logic errors, like for instance if MaxDate < MinDate (AGElint also already checks for mangled dates, such as 188031/l2). It is checking for logic errors where AGElint will really shine. (For the record, using AGElint, I found four instances in three different AGEOD games where MaxDate < MinDate.)

There are still many, many more things that AGElint might check. Really, the possibilities are almost boundless.

I will have more, possibly much more, to say about this, and about AGElint's inner workings, if there is sufficient interest.

[Chilperic]

And we must not forget: "Never let the pursuit of perfection obstruct progress." Right? [;)])

I don't see what you mean here [:D]

Only point which matter in my mind are:

1) test before assessing about one thing
2) test will always reveal flaws
3) flaws are worth considering
4) but in the end, if test shows flaws don't overpower the advantages, keep the tool.

Even if Agelint would remain unchanged, I would use it. Mispelling is so easy in AGE engine, especially when you use XLS FILES ( unless of course, you're doing one event a month [:D]) and AGELINT is until now unrivaled for squashing them. [8D]

[berto]

broken region links

Another item for the to-do list. It should be possible to code automated checks of inter-regional links. (Indeed, I had begun to do this during ROP's development.)

[Chilperic]

Here a false positive:

EvalRgnWeather: Weather in region 712 Ekaterinodar against 2 - Operator: >= Result: True

EvalRgnWeather is working with numerical aliases located in the various aliases file:

// Weather types
$Fair = 0
$Mud = 1
$Snow = 2
$Frozen = 3
$Freeze = 3
$Blizzard = 4
$HarshWeather = 3
$VHarshWeather = 4
// same with typo included
$Blizard = 4

[berto]

That would be in which file(s)?

That's just what I need, BTW: detailed feedback. Back and forth, give and take. Constant refinement and tweaking of both AGElint code and game data. Getting closer and closer to that (ever?) elusive goal: bug free games!

[Chilperic]

You will have

WHat file? the FY file or the various aliases file? The latter is in the alaiases directory.

[berto]

The FY file. It's easy to locate stuff in the aliases files.

[Chilperic]

ORIGINAL: berto


The FY file. It's easy to locate stuff in the aliases files.

AIAggro_WHI for example

[berto]

Okay, I'll look into it.

What I've been working on over the past 4-5 days (multitasking with the gaming stuff):

Milwaukee Renaissance Band Concert, December 2011

Lest anybody think I don't have a life; that all I do is hang out at the Matrix Forum (and Modder Corner) trading postings with Chliperic/Clovis. [;)]

At Modder Corner, I had written:

Early tomorrow at the Matrix Forum, I will begin a series of posts describing (not too deeply) AGElint’s technical innards. Discussing the problem of false positives. Introducing the KWD keyword usage database. Outlining what has already been done, what still needs to be done, and how we might carry the project forward. And more.

I'm finished for now. I'll be back and doing the promised posts later. Have fun.

[Chilperic]

ORIGINAL: berto


Okay, I'll look into it.

What I've been working on over the past 4-5 days (multitasking with the gaming stuff):

Milwaukee Renaissance Band Concert, December 2011

Lest anybody think I don't have a life; that all I do is hang out at the Matrix Forum (and Modder Corner) trading postings with Chliperic/Clovis. [;)]

At Modder Corner, I had written:

Early tomorrow at the Matrix Forum, I will begin a series of posts describing (not too deeply) AGElint’s technical innards. Discussing the problem of false positives. Introducing the KWD keyword usage database. Outlining what has already been done, what still needs to be done, and how we might carry the project forward. And more.

I'm finished for now. I'll be back and doing the promised posts later. Have fun.

Nice. [:)]

I'm not that impatient too. I've a life also. That's why SVF hasn't progressed as much as I had hoped. And DNO. But in the last week, FY has reached a new level. Even as is, it would be something I would remain proud.

[berto]

txt.l -- the AGElint lexer

Here is a brief, but technical, overview of the AGElint txt.l lexical analyzer. You may skip this discussion entirely if you wish.

What is lexical analysis? Quoting from Wikipedia:

http://en.wikipedia.org/wiki/Lexical_analysis

In computer science, lexical analysis is the process of converting a sequence of characters into a sequence of tokens. A program or function which performs lexical analysis is called a lexical analyzer, lexer or scanner. A lexer often exists as a single function which is called by a parser or another function.

Consider the sentence:

The rain in Spain falls mainly on the plain.

Lexical analysis might tokenize that sentence in many different ways, for example...

• word by word:
  
  The
  rain
  in
  Spain
  falls
  mainly
  on
  the
  plain
• word triplets:
  
  The rain in
  Spain falls mainly
  on the plain
• every fourth character:
  
  The
  rain
  in
  Spai
  n fa
  lls
  main
  ly o
  n th
  e pl
  ain.
• chars separated by the string 'ain':
  
  The r
  ain
  in Sp
  ain
  falls m
  ain
  ly on the pl
  ain

And many, many other possibilities besides.

The AGElint 'make' command inputs txt.l to the flex code generator to output C code -- lex.txt.c -- of hideous complexity. You shouldn't ever need to inspect lext.txt.c!

In the AGElint txt.l, you will see a series of pattern matches (so-called "regular expressions"), with -- bracketed by { and } -- some C code saying what to do with the matched character string.

A simple, and common, example:

---

Code: Select all

MinDate                 { ECHO;
                           RETURNCMD(_MINDATE);
                         }

---

(Note: Apologies for the occasional extra blank lines preceding or following code snippets. The Matrix Forum code tags are quirky. I try to format the display nicely, but I don't always succeed!)

That is, in the data file, if the lexer (compiled txt.l code) sees the character string 'MinDate' ...

• echo (depending) to the screen 'MinDate'
• return to the calling parser the token _MINDATE (where internal to the C code the token _MINDATE is an integer, e.g., 866)

Some cases are a bit more complex, for example:

---

Code: Select all

FlavorName              { ECHO;
                           BEGIN(NM);
                           RETURNCMD(_FLAVORNAME);
                         }
 

---

which adds the instruction

---

Code: Select all

                          BEGIN(NM);

---

which says to begin the "start state" NM (name).

In txt.l, there is just one pattern match with the indicated NM start state:

---

Code: Select all

<NM>{DL}{DLUWP}*{DL}"."*/{nDL} {
                           if (isfac(yytext) && agecmd && !mystrcasestr(agecmd, "name")) {
                             yyless(3);  /* push back all after faction name */
                             *(yytext+3) = '\0';
                             if (isdebug) {
                               fprintf(stdout, "%d %s: #%s#\n", lineno, "FACNAM2", yytext);
                             }
                             ECHO;
                             RETURNSTR(_FACNAM);
                           } else {
                             if (isdebug) {
                               fprintf(stdout, "%d %s: #%s#\n", lineno, "MNYNAM2", yytext);
                             }
                             ECHO;
                             RETURNSTR(_MNYNAM);
                           }
                         }

---

Translating:

When in the NM start state (only), any character string beginning with either a single digit or letter; followed by zero or more digits, letters, underscores, whitespaces, or punctuation marks; followed by a single digit or letter; followed by zero or more . (period); followed by a single non digit or non character (but don't add that single char to yytext) ...

• if the matched text (yytext) signifies a faction (usually a sequence of three capital letters) AND agecmd is set to some char string AND agecmd does not contain the char string "name" (case insensitive) ...
  • push back to the input stream all matched chars except for the first three
  • terminate the yytext string after the third char
  • if in debug mode ...
    • print to screen: <line number> FACNAM2: #<faction name>#
  • echo (depending) to the screen the faction name
  • return to the parser the token _FACNAM, with also the matched text string (yytext)
• else ...
  • if in debug mode ...
    • print to screen: <line number> MNYNAM2: #<the first 3 chars of yytext>#
  • echo (depending) to the screen the first 3 chars of yytext
  • return to the parser the token _MNYNAM, with also the matched text string (yytext)

Whew! That's a complicated case.

Fortunately, few of the pattern match cases are that complicated. By far, most are like the simple MinDate example show above.

How does the lexer exit the NM start state? Under several circumstances, most typically:

---

Code: Select all

<INITIAL,NM>\n.		{ lineno++;
 			  if (isshow_text) {
 				if (isshow_lineno) {
 					fprintf(txtout, "\n%6d  ", lineno);
 				} else {
 					fprintf(txtout, "\n");
 				}
 			  }
 			  BEGIN(INITIAL);
 			  PUTBACK(1);
 			}

---

That is:

Whether or not in the NM start state, for every newline (end of line, \n), followed by a single character (any character) ...

• increment the internal lineno (line number) variable
• if the isshow_text flag is set (to TRUE) ...
  • if the isshow_lineno flag is set (to TRUE) ...
    • print to the txt output stream a newline, then the line number on the immediate next line
  • else
    • just print to the txt output stream a newline
• revert to the INITIAL (default) start state
• push back to the input stream the matched char after the newline

(Note that in this case, nothing is returned to the parser.)

I invite you to look around the txt.l file. If you are really brave, you might try making changes here or there, then redoing the 'make' command to recompile the agelint executable. But beware: Sometimes even the simplest change -- especially those involving the pattern match wildcard '*' (zero or more) or '+' (one or more) -- might totally screw things up, and effectively break the entire lexical analysis. Programming lexers is not for the faint hearted!

But you really don't have to understand any of this. Just know that the AGElint lexer (compiled txt.l code) passes a "tokenized" game data file to the AGElint parser (compiled txt.y code) for subsequent syntactical (and other) analysis.

An overview of the AGElint txt.y parser will follow later ...

[berto]

txt.y -- the AGElint parser

Here is a very brief, but technical, overview of the AGElint txt.y parser. You may skip this discussion entirely if you wish.

What is parsing? Quoting from Wikipedia:

http://en.wikipedia.org/wiki/Parser

In computer science and linguistics, parsing, or, more formally, syntactic analysis, is the process of analyzing a text, made of a sequence of tokens (for example, words), to determine its grammatical structure with respect to a given (more or less) formal grammar...

Consider the sentence:

The rain in Spain falls mainly on the plain.

Grammatically speaking, that sentence may be viewed as a sequence of

• article [The]
• noun [rain]
• prepositional phrase
  • preposition [in]
  • noun [Spain]
• verb [falls]
• adverb [mainly]
• prepositional phrase
  • preposition [on]
  • article [the]
  • noun [plain]

but other organizing schemes are possible.

The formal grammar for English usually specifies that articles precede nouns, that prepositional phrases begin with prepositions and end with nouns, that subject precedes predicate, and so on.

The sentence

rain The in falls Spain mainly on the plain.

violates English grammar, as does

plain The rain in Spain falls mainly on the.

Obviously, there are innumerable ways to mangle English grammar.

Note that English grammar would accept unusual, but still "correct", sentences such as

The rain in Spain on the plain mainly falls.

Although English grammar has been codified in exhaustive detail, mostly we just somehow "know" what is correct grammar or not. But in computer code, "it just doesn't sound right" is inadequate. We need to spell out the grammar as accurately and completely as possible. In AGElint, we do that in the file txt.y.

The AGElint 'make' command inputs txt.y to the bison code generator to output C code -- txt.tab.h, txt.tab.c -- of hideous complexity. You shouldn't ever need to inspect txt.tab.h or txt.tab.c!

In the AGElint txt.y, you will see a list of token declarations (see the earlier txt.l lexer description) (also some other technical stuff), followed by the AGE data file "formal grammar".

The agelint AGE data file formal grammar begins with:

---

Code: Select all

start:            abilities
                 | ais
                 | aliases
                 | diplomacies
                 | ethnics
                 | events
                 | facattribs
                 | facmods
                 | factions
                 | includes
                 | merchandises
                 | models
                 | regions
                 | religions
                 | researches
                 | rgndecisions
                 | rulers
                 | scripts
                 | structures
                 | terrains
                 | units
                 ;

---

That is, an AGE game data file must be one or the other of the listed possibilities (where '|' signifies OR).

Let's look more closely at the aliases case. The aliases stanzas are coded as:

---

Code: Select all

aliases:        aliasthings
                 ;
 
 aliasthings:      aliasthing
                 | aliasthings aliasthing
                 ;
 
 aliasthing:       val eq intvblist
                 | val eq val
                 | error
                 ;

---

That is, a series of alias specifications (aliasthings), where an alias specification might be a val, followed by an eq (equal sign =), followed by either an intvblist or another val; else a specification might be in error (in which case the parser can recover from the mistake and move forward with the parsing).

Here are the specifications for val, eq & intvblist:

---

Code: Select all

val:              alsval
                 | facval
                 ;
 
 eq:             _EQ {
                   agecmdrhs = agecmd;
                   linenorhs = lineno;
                 }
 
 intvblist:        int
                 | intvblist vb int
                 ;

---

with some subsidiary specifications:

---

Code: Select all

alsval:         _ALSVAL {
                     if (islist_aliases || islist_locals) {
                             fprintf(stdout, "%s:%d:%s\n", txtfile, lineno, $1+1);
                     }
                     free($1);
                 }
                 | gmaoptval
                 | abinamval
                 | abitxtval
                 | dinamval
 ...
                 | txtval
                 | unitxtval
                 | ldrval
                 | mdlval
                 ;
 
 facval:         _FACVAL { free($1); }
                 ;
 
 int:              intpos { $$ = $1; }
                 | intneg { $$ = $1; }
                 ;

---

and on and on ...

Look rather complicated? It is!

In the txt.y formal grammar, stanzas such as int: might be viewed as subroutines. It's programming, but programming of a sort different from what you are probably used to.

As you can see, as with txt.l, in txt.y grammar components might include -- bracketed by { and } -- some C code saying what special things to do, if any.

For now, the important thing to understand is: if the pattern of tokens (output by the agelint lexer, the compiled txt.l) doesn't match one of the precisely defined sequences in the formal grammar (specified in txt.y), agelint will report a syntax error (and possibly do other stuff) -- the game data broke the grammar rules!

I invite you to look around the txt.y file. If you are really brave, you might try making changes here or there, then redoing the 'make' command to recompile the agelint executable. If you code a "little" change here or there, in the 'make' output, your conflicts line might say something like:

txt.y: conflicts: 768 shift/reduce, 523 reduce/reduce

rather than the current

txt.y: conflicts: 3 shift/reduce, 2 reduce/reduce

These "conflicts" represent ambiguities in the lexer/parser specification. Ideally you want 0 conflicts of each type, but that is difficult to impossible, depending on the coding pains you take, and possibly the language (in this case, the AGE scripting language) that you are attempting to model.

As with lexer programming, programming parsers can be very weird indeed!

But you really don't have to understand any of this. Just know that the AGElint parser checks the game data file(s) for syntactical (and other) correctness. If agelint reports a syntax error, you know who reports it (the compiled txt.y).

I will have more, possibly much more, to say about the txt.y parser in future posts. The txt.l lexer is more or less settled. It's in txt.y where most of the action (and ambiguity and controversy and mistakes ...) takes place.

[Chilperic]

I will check alias this week. From what I've seen in your posts, most of the bugs you have found in the official games are concerning other scenarios than the GC, and FY GC has been cleaned of some aliases bugs. We'll see soon how much remain in FY.

REAL QA delivery is on move. In FY.[8D]

[berto]

txt.y parser: room for improvement

Consider again the sentence:

The rain in Spain falls mainly on the plain.

Proper English, meaningful (if cliched).

Now consider this sentence:

The thing in somewhere acts sometime on the something.

Still proper English, but meaningful?

Parts of the agelint txt.y parser are still like that: written in a generic sort of way; good for checking the most basic AGE syntax but little else.

Here is an example from txt.y:

---

Code: Select all

changeactorpool: _CHANGEACTORPOOL eq sclist
                 {
                 /*
                 http://www.ageod.net/agewiki/SetActorPool
                 Syntax:  SetActorPool = ActorUID|Identifier(n)|Value(n).....
 
 		yes, ChangeActorPool resolves to above Wiki page for SetActorPool
 		"complex command" -- we	leave this as sclist for now
                 */
                 }
 		;
 
 sclist:		  thing
 		| sclist sc thing
 		;

---

where thing is a generic mishmash of this or that without common theme or sequence. ('sc' refers to semicolon.)

(Again: My apologies for the extra empty space before and after code examples. The Matrix Forum code feature is quite quirky, infuriating even! I have these good things to say about AGEOD: The AGEOD Forum handles code display properly, and their forum is among the most congenial, capable, and easy-to-use anywhere.)

If you visit the AGEWiki URL

http://www.ageod.net/agewiki/SetActorPool

you will see just how "complex" this command really is, how numerous and varied its arguments are, and how precise is the argument sequence. The simple 'changeactorpool: _CHANGEACTORPOOL eq sclist' doesn't begin to capture the complexity of it all.

Now consider this sentence:

The rain in somewhere falls mainly on the something.

Proper English: 'somewhere' (as with 'Spain') and 'something' (as with 'plain') are both nouns. On the surface, the sentence conforms to correct English grammar.

But does it make much better sense?

Some parts of the agelint txt.y are like that too: written in greater specificity, but still lacking in fully useful detail.

Here is an example from txt.y:

---

Code: Select all

objectives:	_OBJECTIVES eq valintsclist
 		;
 
 valintsclist:	  val sc int
 		| valintsclist sc val sc int
 		;
 
 val:		  alsval
 		| facval
 		;

Code: Select all

alsval:		_ALSVAL {
 		    if (islist_aliases || islist_locals) {
 		        fprintf(stdout, "%s:%d:%s\n", txtfile, lineno, $1+1);
 		    }
 		    free($1);
 		}
 		| gmaoptval
 		| abinamval
 		| abitxtval
 		| dinamval
 ...
 		| txtval
 		| unitxtval
 		| ldrval
 		| mdlval
                 ;

---

valintsclist (a semicolon-separated list of val/int data pairs) is more specific than the earlier, and extremely general, sclist. But you can see how the val in valintsclist can be almost anything. (And there is no checking of the int values either.) (I have omitted the facval specification for brevity.)

The txt.y parser still needs much filling out, and deepening. For now, it’s like the parser understands grade school English. I/we need to teach it high school or college English. The deeper AGElint’s understanding of “English”, the more errors it will detect.

Here is an example from txt.y of a well qualified, detailed, fully fleshed out command specification:

---

Code: Select all

selectsubunits: _SELECTSUBUNITS eq selectsubunitsparms
                 {
                 /*
                 http://www.ageod.net/agewiki/SelectSubUnits
                 Syntax:  SelectSubUnits = Region <RgnUID>;Area <AreaUID>;Families <Fam1> <Fam2> ...;Models <Mdl1> <Mdl2> ... ;FactionTags <Tag1> <Tag2> ... ;Domains <_domLand> <_domNav> <_domAir>;<Attributes>;Generations <ModelGen1> <ModelGen2> ...
                 */
                 }
                 ;

Code: Select all

selectsubunitsparms: selectsubunitsparm
                 | selectsubunitsparms sc selectsubunitsparm
                 ;
 
 selectsubunitsparm:
                   _REGION val
                 | _AREA alsval
                 | _FAMILIES alsvallist
                 | _MODELS mdlvallist
                 | _MODELS ldrvallist
                 | _FACTIONTAGS faclist
                 | _DOMAINS alsvallist
                 | selectsubunitsattr
                 | _GENERATIONS gennamlist
                 | _THEATER alsvallist {
                     txterrmsg(_WARNING, TRUE, linenorhs, "suspicious, undocumented usage of Theater");
                   }    
                 ;

Code: Select all

selectsubunitsattr:
 		  _ONLYFIXED
 		| _ONLYNOTFIXED
 		| _ENEMY
 		| _FRIENDONLY
 		| _FRIENDANDSELF
 		| _UNIQUENAME anynam
 		| _ONLYPERMFIXED
 		| _ONLYNPERMFIXED
 		;

---

Although there remains some generality (see the use of alsvallist -- a list of generic alias values), the SelectSubUnits command arguments are spelled out in considerable detail. If for example the Generations keyword is followed by something other than a gennamlist -- a ldrvallist, or an integer, or another keyword, or ... -- agelint will spot the error, and report it. No ambiguity or false positives here. (And hopefully not the converse: errors not detected or reported.)

I estimate that maybe half the txt.y parser is still generic, and half is adequately specific.

Again:

The deeper AGElint’s understanding of “English” [the AGE command language], the more errors it will detect.

Now if only we could teach the agelint txt.y parser to understand "Shakespearean English"! (That is, the AGE command language in all of its detail, richness, and nuance.)

When I write

Please bear in mind that, for all of its current sophistication, AGElint is a work-in-progress, more than a beginning but far from an ending.

this is in part what I mean. There is still much, much more to be done, and more and more bugs to detect!

I will have more to say about the txt.y parser, and other areas of extension and improvement, in the days and weeks ahead.