AGElint -- an AGE debugging toolkit

Chilperic · Post by **Chilperic** » Sun Jan 08, 2012 3:50 pm

BTW what means this:

sh: line 1: 8028 Segmentation fault (core dumped) /cygdrive/c/cygwin/home/Laurent/Games/AGEOD/agelint/agelint +la -g acw "GameData/Models/118USA_Gunboat.mdl" 2> /dev/null

I've a great fun with Memphis, TN [:D].Will fix that.

berto · Post by **berto** » Sun Jan 08, 2012 4:40 pm

AGElint: excludes.???.dat

For all of the AGElint Perl *.pl scripts, you can exclude files from consideration by means of the excludes.???.dat "blacklist" mechanism.

There are six excludes files:

excludes.acw.dat
excludes.ncp.dat
excludes.pon.dat
excludes.rop.dat
excludes.rus.dat
excludes.wia.dat

For example, here is the excludes.rus.dat file:

# rus excludes
(Test|Parse|Samples|CzechLegion)
Scripts/Script.ini
Events/2-CzechLegion1918.sct
Events/2-CzechLegion1918Demo.sct
Events/AigarPolishrussowar.sct
Events/POLAIrpwar.sct
Events/RegistrySamples.sct

The second line is a pattern match. Any file name in part matching any of the strings in the pattern match are bypassed, excluded from the analysis.

So, in the example, all of the following files would be skipped:

Events/2-CzechLegion1918Demo.sct
Events/2-CzechLegion1918.sct
...
Includes/SubfactionsCzechLegion18.inc
Scripts/ScriptCzechLegion.ini

The third line and all following specify exact file names that should be skipped. (In the example, there is really no need to list the two CzechLegion1918 cases, since they would be excluded by the previous pattern match; i.e., listing them is redundant.)

You can exclude entire file categories with the appropriate pattern match. For example, to exclude all Models and Units file from the analysis, your excludes.rus.dat pattern match would look something like

You could exclude the entire GameData folder with

(Test|Parse|Samples|CzechLegion|GameData)

and focus just on Events files with

(It is not necessary to list thing like Fonts, FrontEnd, Graphics, etc., since AGElint at present doesn't consider them anyway.)

In future, we might add an includes.???.dat file, to "whitelist" files.

For the technically inclined, here is the Perl code we use to achieve the file excludes. First, excludes.pl (called in chklint.pl for example by means of the statement 'require "excludes.pl";')

Code: Select all

# excludes.pl - AGElint chk*.pl excludes routines                               
 
 open(EXCLUDES, "$EXCLUDES");
 while(<EXCLUDES>) {
     chomp;
     next if (/^\s*#/);
     $thing = $_;
     print "$thing\n" if $DEBUG;
     if ($thing =~ /^\(.+\)$/) {
         $EXCLPAT = $thing;
     } else {
         $excludes{"./$thing"}++;
     }
 }
 close(EXCLUDES);
 
 ...
 
 1;  # required!

And in, for example, chklint.pl we achieve the bypasses with

Code: Select all

     if ($excludes{$file}) {
         print "skipping $file\n" if $DEBUG;
         next;
     }
 
     if ($EXCLPAT && ($file =~ /($EXCLPAT)/)) {
         print "skipping $file\n" if $DEBUG;
         next;
     }

Chilperic · Post by **Chilperic** » Sun Jan 08, 2012 5:14 pm

Thanks. It will help a lot. [:)]

berto · Post by **berto** » Sun Jan 08, 2012 5:35 pm

Don't forget

reformatting chkaliases.pl output [post #56]

where I discuss various ways of reformatting report output.

I would recommend using the supplied dochk program, but I'm not sure how well the AGElint 1.00 version will work for you. I am in the process of revising dochk to use agelint.conf (and debugging the appearance of ' and spaces in file names [:@]). When AGElint v1.10+ is released, be sure to consider dochk for all of your comprehensive bug checks.

berto · Post by **berto** » Sun Jan 08, 2012 6:58 pm

Here's a trick:

Try running agelint against a buggy file. If you see false positives on lines 15, 244, 532 & 1278, try rerunning the agelint command like, for example:

./agelint -n +w +e +c5 +u +x -S 15 244 532 1278 -g rus "Events/7-Drang1921.sct"

The -S option will squelch error reports for the specified lines. [:)]

And, since you can recall previous commands with cursor-up key presses (cursor-up and cursor-down to review and recall any earlier/later Linux/Cygwin commands), you can continue recalling earlier agelint commands, adding and subtracting line #s to your heart's content.

I will have much more to say about this, and how it factors into the AGElint 'make check' regression tests, soon -- probably by tomorrow morning.

Here's another hint: Look at the file squelches.chk.pl. [;)]

There's probably lots more you still don't know about AGElint!

Chilperic · Post by **Chilperic** » Sun Jan 08, 2012 7:05 pm

Thanks again [:)]

berto · Post by **berto** » Mon Jan 09, 2012 9:05 am

more on false positives

This is an important post, so listen up! [:)]

Here is a typical chklint error report:

ERROR: in Events/3-November1918.sct, at (or near) line 1951: syntax error: #M# (0x4d), #MinDate#

in event evt_nam_Renforts_RED_AmurPartisanArmy:

1942 SetType = $uni_RED_Sup1
1943 SUFlavorName = 1st Co.|2nd Co.|3rd Co.|4th Co.
1944 SetName = Partisans Supply
1945 Apply
1946
1947 EndEvent
1948
1949 SelectFaction = $RED
1950 StartEvent = evt_nam_Renforts_RED_AmurPartisanArmy|1|1|NULL|NULL|NULL|NULL
1951> MinDate = 1919/01/01
1952 MaxDate = 1919/12/01
1953 Probability = 20
1954
1955 Actions
1956
1957 SelectFaction = $REd
1958 SelectFaction = $REd
1959 AddToRegList = $Mukhen
1960

False positive, right?

Not so fast!

Do you see the problem? If so, good for you! If not, it is: There is no 'Conditions' statement. After 'StartEvent ...', it goes immediately to listing conditions.

You may or may not think that's a real problem, but that's not the point. The point is that the

ERROR: in Events/3-November1918.sct, at (or near) line 1951: syntax error: #M# (0x4d), #MinDate#

...

1951> MinDate = 1919/01/01

are misleading. You see the 'MinDate = 1919/01/01', and you think: "What's the problem? There is none. False positive!"

Except that (if you agree with the analysis), no, it's a real error. But the chklint error statement not only misdiagnoses the problem -- "syntax error" -- it also (sometimes) gets the "offending" line number wrong! (Truly it is a syntax error. But just looking at the line 1951 but not at the surrounding context, you fail to see it.)

In my defense, I do state "at (or near) line ..." But that's easily overlooked. It's natural for you to focus just on the reported line number and ignore the surrounding context.

As written, I have removed most of the genericalness from the txt.y parser. In preparation for AGElint v1.10's release, I have begun vetting all (or almost all) reported errors in RUS 1.04 (and some of the other AGEOD games). Soon, I came across this example, and realized (again):

"Houston, we have a problem."

In the AGElint to-do list, I have written:

...
Better pinpoint the exact line number of reported error.
Improve parser 'error' recovery, for detecting all errors etc. in a given data file.
...

These are issues that, in my haste to get a functional AGElint, I had swept under the rug. Folks, there's a lot under that rug!

I am now confronting this issue straight on. I need to focus on the above two to-dos, and add another:

Better describe errors (and not so readily default to the generic, often uninformative "syntax error").

In part, this will require better implementation of the yacc 'error' statement, which deals with parser error recovery.

Let me explain. When the parser encounters an error, when the input violates the syntax rules, the parser gets confused (hopefully just momentarily). It reports an error, and ideally moves on to the next. But sometimes it gets so confused, it is so lost, that it just gives up -- on the entire file. At other times, it remains on the case, continues checking for errors, but glosses over one or more close-by errors until it can reorient itself properly, and resume normal error (warning etc.) checking. Usually, the parser gets it right -- stays with a file, and detects all errors. But in unusual circumstances, it gets it quite wrong.

This is why you can't get to the point of seeing just one last chklint-reported error, fix that error, then move on to the next file. No, it is important that you keep running agelint until, ideally, it reports zero errors. (Possibly with aid of the squelch mechanism, mentioned several posts above, and to be described in greater detail here soon.) The one error might be "masking" the presence of subsequent errors. If you remove the mask (fix the error), it might reveal other follow-on errors. So (ideally) keep looking for errors until you no longer find any.

Another thing I need to do is to have fewer "syntax error:" and more "warning:" and/or use more 'txterrmsg(_ERROR, ..., <explicit error message> ...)' function calls.

I have written that AGElint is "unfinished", "imperfect", a "work-in-progress", etc. I have also written many times about the issue of false positives. If you are frustrated by the problem of false positives, well, so am I! I pledge to keep working hard at this until I Get It Right.

Thank you for your patience!

berto · Post by **berto** » Mon Jan 09, 2012 10:06 am

More on the txt.y parser getting "confused":

Imagine you are learning a foreign language, Chinese say. You are listening to a Chinese speaker. He/she says a word or uses an expression or grammatical construct that you are unfamiliar with, else makes a real mistake, doesn't really matter. The point is that you are momentarily confused, you stop listening to the flow of words while you ponder the mistake (yours, or the speaker's). While you gather your thoughts, you might miss other things (unfamiliar stuff, speaker errors, ...). Sooner or later, you figure it out (or not), and you resume listening intently. (Or not.)

The parser interpreting the input stream: it's kind of like that.

Getting confused while listening to a spoken foreign language is part of the human condition. Similarly, the txt.y parser sometimes getting confused by the input stream is inherent to the (yacc) parsing technology.

We can try to minimize the problem, but we can't universally solve it.

One more reason why there will probably always be false positives and imperfectly precise error reports. Can't help it!

berto · Post by **berto** » Mon Jan 09, 2012 10:23 am

Still more:

If you are a C (or Java or Perl or ...) programmer, you see this parser "confusion" effect if, for instance, you omit a required ; or } or some other "little" thing.

Then, when the program compiles (or runs), you see a spew of compiler errors filling your monitor screen. And maybe not also errors due to later-on coding mistakes.

You go back and fix the "little" mistake, and the thing then compiles cleanly (or reveals the earlier masked errors).

So you see, even the best of the C (and Java and Perl and ...) compilers can get confused. Again, it's an inherent limitation to the available parsing technology. Parser confusion -- can't help it!

Chilperic · Post by **Chilperic** » Mon Jan 09, 2012 11:27 am

When I can't discern first either a false positive or a real bug, I leave as is and reports nothing. A second look a few hours later has given sometimes better results. I've rrun agelint on FY several times too and I plan to do a new inspection on FY with the new version.

Once again, the number of false positive isn't that large and not even an hindrance. After all, once identified as a false positive, it is just a part of the agelint report to bin. The real bugs are anyway found and fixed in a few days rather than a few months ( I'm not kidding).

Missing Condition string is a not uncommon error indeed. Found occurence in AACW.Fixed in SVF [:)]

berto · Post by **berto** » Mon Jan 09, 2012 6:50 pm

AGElint: squelches & ignores

Try running agelint against a buggy file. If you see false positives on lines 15, 244, 532 & 1278, try rerunning the agelint command like, for example:

./agelint -n +w +e +c5 +u +x -S 15 244 532 1278 -g rus "Events/7-Drang1921.sct"

The -S option will squelch error reports for the specified lines.

And, since you can recall previous commands with cursor-up key presses -- cursor-up and cursor-down to review and recall any earlier/later Linux/Cygwin commands; use the standard cursor-left, cursor-right, delete, etc. to edit earlier commands) -- you can continue recalling earlier agelint commands, adding and subtracting line #s to your heart's content.

We use this error (and warning & notice) squelch mechanism in the AGElint 'make check' regression test. In the AGElint Makefile, it says:

check:
./chklint.pl -n +w +e -v -s +u +x +q +c5 -S -g chk 2>&1 | tee ./chklint.chk.out

If you try that command adding the +G debuG option (and piping to 'less' for on-screen display)

./chklint.pl -n +w +e -v -s +u +x +q +c5 +G -S -g chk 2>&1 | less

you will see the -S Squelch mechanism in action:

...

Aliases/uni_Alias.ini
/home/berto/games/ageod/agelint/agelint -n +w +e -v -s +c5 +u +x +q -X -L -S 24 163 400 662 917 -g chk "Aliases/uni_Alias.ini" |

GameData/Research/104-RI_Modern_agricultural_techniques.res
/home/berto/games/ageod/agelint/agelint -n +w +e -v -s +c5 +u +x +q -X -L -S 12 19 25 31 -g chk "GameData/Research/104-RI_Modern_agricultural_techniques.res" |

...

In the CHK/Aliases/uni_Alias.ini file (where CHK is a pseudo install, within the agelintroot folder, for 'make check' regression tests), you will see deliberately buggy lines 24, 163, 400, 662 & 917. Likewise for lines 12, 19, 25 & 31 in the CHK/GameData/Research/104-RI_Modern_agricultural_techniques.res file.

Where do those line #s come from? From the squelches.chk.pl file, shown in part here:

...

# Aliases
$squelches{"Aliases/uni_Alias.ini"} .= "24 ";
$squelches{"Aliases/uni_Alias.ini"} .= "163 ";
$squelches{"Aliases/uni_Alias.ini"} .= "400 ";
$squelches{"Aliases/uni_Alias.ini"} .= "662 ";
$squelches{"Aliases/uni_Alias.ini"} .= "917 ";

...

# Research
$squelches{"GameData/Research/104-RI_Modern_agricultural_techniques.res"} .= "12 ";
$squelches{"GameData/Research/104-RI_Modern_agricultural_techniques.res"} .= "19 ";
$squelches{"GameData/Research/104-RI_Modern_agricultural_techniques.res"} .= "25 ";
$squelches{"GameData/Research/104-RI_Modern_agricultural_techniques.res"} .= "31 ";

...

Whenever chklint.pl runs with the -S Squelch option, it looks for a squelches.???.pl file in the agelintroot directory, where ??? matches the game specified in the -g option.

So, for example, if you were to run the command

./chklint.pl -n +w +e -v -s +u +x +q +c5 -S -g rus 2>&1 | tee ./chklint.rus.out

chklint.pl would look for the file (in agelintroot) squelches.rus.pl, which is currently empty:

# RUS 1.04

1; # required!

It is up for you to add squelch entries in that (and the other game squelches.???.pl files) as you wish.

If you run agelint with the +q (as in '+|-q show/don't show squelches') option, for example

[root@telemann agelint]# ./agelint -n +w +e -v -s +u +x +q +c5 -g chk "Aliases/uni_Alias.ini" 2>&1 | less

you will see something like

ERROR: in ./CHK/Aliases/uni_Alias.ini, at (or near) line 24: syntax error: #$# (0x24), #$uni_RED_Div23#

19 $uni_RED_Div18 = 18
20 $uni_RED_Div19 = 19
21 $uni_RED_Div20 = 20
22 $uni_RED_Div21 = 21
23 $uni_RED_Div22 22
24> $uni_RED_Div23 = 23
25 $uni_RED_Div24 = 24
26 $uni_RED_Div25 = 25
27 $uni_RED_Div26 = 26
28 $uni_RED_Div27 = 27
29 $uni_RED_Div28 = 28

$squelches{"Aliases/uni_Alias.ini"} .= "24 ";

ERROR: in ./CHK/Aliases/uni_Alias.ini, at (or near) line 163: syntax error: #u# (0x75), #uni_RED_VOKhR2#

158 $uni_RED_MilAG = 157
159 $uni_RED_Cav21 = 158
160 $uni_RED_Reb = 159
161 $uni_RED_Tck = 160
162 $uni_RED_VOKhR1 = 161
163> uni_RED_VOKhR2 = 162
164 $uni_RED_Sla = 163
165 $uni_RED_Ins1 = 164
166 $uni_RED_Str1 = 165
167 $uni_RED_Wsu = 166
168 $uni_RED_Mer = 167

$squelches{"Aliases/uni_Alias.ini"} .= "163 ";

...

NOTE: For chklint.pl, you use the -S without line #s. For agelint, you use the -S with line #s.

The agelint +q (show squelches) option displays lines like

$squelches{"Aliases/uni_Alias.ini"} .= "24 ";

for each offending error (or warning or notice) line. You can cut-and-paste these ready-made squelch lines into your customized squelches.???.pl files.

"Hey, this is handy. Why don't I create a customized squelches.rus.pl file stuffed with all those pesky false positives?" you may be thinking to yourself. Be careful! If you add or delete lines from that file, you will offset the numbers in squelches.rus.pl, rendering many or all squelch line #s useless.

That is why agelint doesn't invoke any of the squelches.???.pl files. Only chklint.pl does that. And so far, only the 'make check', which calls chklint.pl on an invariant (to you) squelches.chk.pl file with unchanging line #s.

So, begin populating your customized squelches.???.pl files, and use them with chklint.pl, but use at your own risk!

In addition to the agelint (and chklint.pl) squelch mechanism, there is also the ignore mechanism.

Try editing the file ./agelintroot/CHK/GameData/Religions/8-Protestant.rel, adding a 'Q' before 'UID' in the first line like so:

QUID = 8
Name = Protestant
Alias = relProtestant
Color = colLavenderBlush3

Then run a standard agelint command against that file:

[root@telemann agelint]# ./agelint -n +w +e -v -s +u +x +q +c5 -g chk "GameData/Religions/8-Protestant.rel" 2>&1

ERROR: in ./CHK/GameData/Religions/8-Protestant.rel, at (or near) line 1: syntax error: #=# (0x3d), #=#

1> QUID = 8
2 Name = Protestant
3 Alias = relProtestant
4 Color = colLavenderBlush3

$squelches{"GameData/Religions/8-Protestant.rel"} .= "1 ";

As expected, agelint complains about the unknown 'QUID' in line #1.

Try squelching that error by adding '-S 1' to the command line:

[root@telemann agelint]# ./agelint -n +w +e -v -s +u +x +q +c5 -S 1 -g chk "GameData/Religions/8-Protestant.rel" 2>&1

No error reported. Great, the '-S 1' is working as expected.

Now, instead try ignoring the first line by adding '-I 1' to the command line:

[root@telemann agelint]# ./agelint -n +w +e -v -s +u +x +q +c5 -I 1 -g chk "GameData/Religions/8-Protestant.rel" 2>&1

ERROR: in ./CHK/GameData/Religions/8-Protestant.rel, at (or near) line 2: syntax error: #N# (0x4e), #Name#

1 QUID = 8
2> Name = Protestant
3 Alias = relProtestant
4 Color = colLavenderBlush3

$squelches{"GameData/Religions/8-Protestant.rel"} .= "2 ";

Hmm, what's going on here?

In this case, the '-I 1' says to ignore the first line as if it doesn't exist. The agelint parser simply ignores it entirely. A consequence of that is this Religions file appears to lack the required opening 'UID =' line, which is certainly an error. When the parser sees

2> Name = Protestant

as if it were the very first actual data line (it's ignoring the first line, remember?), the parser "thinks": "Where is the opening 'UID ='? Error!"

If this is all unclear to you, you might try experimenting with the '-S <#s>' squelch option vs. the '-I <#s>' ignore option to gain a better understanding of their differences.

Note that you can just as easily combine '-S' with '-I' in the same agelint command line, as in:

[root@telemann agelint]# ./agelint -n +w +e -v -s +u +x +q +c5 -S 1 -I 4 -g chk "GameData/Religions/8-Protestant.rel" 2>&1

That tells agelint to squelch the report of error in line #1, and totally ignore line #4 (which is also erroneous, since 'Color =' requires an alias parameter; note the missing '$' before 'colLavenderBlush3').

NOTE: You may use -S (without line #s) with chklint.pl, but for that program -I is not a valid option. That is, you may squelch and/or ignore with agelint, but with chklint.pl you may only squelch. (There are also no ignores.???.pl files corresponding to the squelches.???.pl files. That is WAD.)

This may all seem a bit confusing, but play around. Usually squelches are appropriate, but maybe ignores will sometimes suit your purpose. Experiment!

(If you edited the ./agelintroot/CHK/GameData/Religions/8-Protestant.rel to add a 'Q' before 'UID' in the first line, you should undo that edit now. Unless you want to experiment some more.)

Chilperic · Post by **Chilperic** » Mon Jan 09, 2012 9:05 pm

To let aside the question of false positive, should it be possible to check event names? As I use a lot of long chains of events ( until 40, rarely of course [:D]), one of the most frequent error is a mispelling in the event name when used in the EvalEvent condition. This could be a tremendous addition, even for options giving +5 infantry replacements for 3 EPs, which are simpler but sometimes bugged [8D]

berto · Post by **berto** » Mon Jan 09, 2012 9:37 pm

Sure. That's planned. But probably not for the very next AGElint v1.10 (due out later this week).

berto · Post by **berto** » Tue Jan 10, 2012 8:21 am

Consider this example:

ERROR: in Events/3-November1918.sct, at (or near) line 6565: syntax error: #A# (0x41), #AlterCuSubUnit#

in event evt_nam_Renforts_CMN_JapaneseIntervention2:

6556 SelectRegion = $Verkhneudinsk
6557 StartEvent = evt_nam_Renforts_CMN_JapaneseIntervention2|1|1|NULL|NULL|$Verkhneudinsk|NULL
6558
6559 Conditions
6560 MinDate = 1918/11/01
6561 Probability = 10
6562
6563 SelectFaction = $RED
6564 SelectSubUnits = Area $EventArea_JapaneseIntervention2;FactionTags RED
6565> AlterCuSubUnit = ApplytoList;Move $Shilka
6566
6567 Actions^M
6568 DescEvent = evt_desc_Renforts_CMN_JapaneseIntervention2
6569
6570 SelectFaction = $WH3
6571 SelectRegion = $Verkhneudinsk
6572
6573 SelectFaction = $WH3
6574 SelectRegion = $Verkhneudinsk

There is nothing wrong with the AlterCuSubUnit statement in isolation. But looking at the surrounding context, agelint says "syntax error", because the AlterCuSubUnit statement is in Conditions, not in Actions.

A false positive?

There are not a few cases like this: where an action statement (and therefore not a logical test) is found to be in Conditions. Are these genuine errors?

berto · Post by **berto** » Tue Jan 10, 2012 9:45 am

Action statements appearing in the Conditions section -- one way to handle this "problem" (?) is by giving warnings:

Code: Select all

cond:             conditions { txterrmsg(_ERROR, FALSE, lineno, "duplicate Conditions statement"); }
 
                 | actordisabled
                 | actorenabled
 ...
                 | selunqunit
                 | turnindex
 
 		| altercusubunit { txterrmsg(_WARNING, TRUE, linenorhs, "suspicious, unusual use of %s in Conditions section", "AlterCuSubUnit"); }
 ...
 
                 ;

And similarly for condition statements appearing in the Actions section:

Code: Select all

act:              actions { txterrmsg(_ERROR, FALSE, lineno, "duplicate Actions statement"); }
 
                 | absorbfaction
                 | addnationalregions
 ...
 		| winparams
                 | winparamsvp
 
                 | probability { txterrmsg(_WARNING, TRUE, linenorhs, "suspicious, unusual use of %s in Actions section", "Probability"); }
 ...
 
                 ;

By giving explicit warning messages, we avoid the more generic, and possibly confusing, "syntax error" messages.

Chliperic, I genuinely, really, truly appreciate your open-mindedness, cooperativeness, and patience. And your feedback.

As with most things in life, you get one chance to make your impression. With many people, you don't get a second chance.

As I develop AGElint and promote its use, I try to understand the, at best, indifference and, at worst, hostility it sometimes engenders.

I suspect that some people are looking for excuses to write me off, seizing for example the issue of false positives to disparage AGElint. They see false positives, and think: "Berto is a doofus. He doesn't know what he's doing. AGElint sucks." I think: "No, I know what I'm doing, in large part. I just need to do a better job of clarifying the AGElint error/warning statements [among other refinements, improvements, and bug fixes]."

That's what I suspect, anyway.

But thanks again, Chliperic, for your help in working through these difficult, thorny issues. I am grateful for it. Together, and as a community, we debug and fix and mod and build better games.

berto · Post by **berto** » Tue Jan 10, 2012 1:23 pm

ORIGINAL: berto

Still more:

If you are a C (or Java or Perl or ...) programmer, you see this parser "confusion" effect if, for instance, you omit a required ; or } or some other "little" thing.

Then, when the program compiles (or runs), you see a spew of compiler errors filling your monitor screen. And maybe not also errors due to later-on coding mistakes.

You go back and fix the "little" mistake, and the thing then compiles cleanly (or reveals the earlier masked errors).

So you see, even the best of the C (and Java and Perl and ...) compilers can get confused. Again, it's an inherent limitation to the available parsing technology. Parser confusion -- can't help it!

Returning to the C etc. example...

When a C compiler chokes on a coder syntax error, it (a) usually spews forth lots of (b) often cryptic, misleading error messages.

The Perl compiler is no different. The AGElint chkaliases.pl Perl script begins (more or less) like so:

Code: Select all

 #!/usr/bin/perl                                                                 
 
 require "common.pl";
 
 # register all localisations in the %locals hash                                
 open(LOCALS, "cd \"$GAMEDIR\"; find -name \\*.csv -exec cut --delimiter=\\; -f 1 {} \\; | egrep -v \"^//\" | egrep \"[a-zA-Z]\" | sort | uniq |");
 while (<LOCALS>) {
     chomp;
     $local = $_;
     print "LOC1 #$local#\n" if $DEBUG;
     if ($local =~ /^\s*(\S.+\S)\s*$/) {  # strip leading/trailing whitespace    
         $local = $1;
         print "LOC2 #$local#\n" if $DEBUG;
         # now convert to all lower case                                         
         $local = lc($local);
         print "LOC3 #$local#\n" if $DEBUG;
         $locals{$local}++;
     }
 }
 close(LOCALS);

If you remove the ; semi-colon at the end of the 'open(LOCALS, ...)' line, line 6

Code: Select all

 open(LOCALS, "cd \"$GAMEDIR\"; find -name \\*.csv -exec cut --delimiter=\\; -f 1 {} \\; | egrep -v \"^//\" | egrep \"[a-zA-Z]\" | sort | uniq |")

then run the script, you will see something like this:

[root@telemann agelint]# ./chkaliases.pl +i +E -g rus
syntax error at ./chkaliases.pl line 7, near ") {"
syntax error at ./chkaliases.pl line 19, near "}"
Execution of ./chkaliases.pl aborted due to compilation errors.

Wait a minute! The Perl compiler reports errors at lines 7 & 19, and makes no mention of the missing ; semi-colon in line 6!

If you remove the very first line of agelint.h (shown here are the first four lines of that file):

Code: Select all

 /*                                                                              
    agelint.h -- AGElint common header file for all binaries                        
                                                                                 
  */

then try to compile using 'make', you will see something like

Code: Select all

 gcc -O2 -Wall -Wshadow -Wconversion -Wpointer-arith -Wcast-qual -c agelint.c -o agelint.o
 In file included from agelint.c:6:0:
 agelint.h:2:11: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘.’ token
 In file included from agelint.h:10:0,
                  from agelint.c:6:
 /usr/include/sys/types.h:34:18: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘u_char’
 agelint.c: In function ‘checkcfg’:
 agelint.c: In function ‘init’:
 agelint.c: In function ‘getroot’:
 agelint.c: In function ‘date2days’:
 make[1]: *** [agelint.o] Error 1
 make[1]: Leaving directory `/home/berto/games/ageod/agelint'

No mention of the missing '/*' in line #1 in any of that gobbledygook!

So I don't feel so bad if my humble agelint parser obfuscates and misleads when reporting syntax errors. Compared to the mighty and exalted C and Perl compilers, it actually compares favorably.

Having said all that...

I have decided to generalize the solution in post #135 above: I will add all actions statements to the parser cond: (Conditions) section, but give txterrmsg() warnings; and add all conditions statements to the parser act: (Actions) section, and likewise give txterrmsg() warnings.

That should remove any 'syntax error ...' ambiguity. If you use an action statement in Conditions, or vice versa, agelint should report your mistake (with a warning) clearly and explicitly!

Chilperic · Post by **Chilperic** » Tue Jan 10, 2012 7:00 pm

ORIGINAL: berto

Action statements appearing in the Conditions section -- one way to handle this "problem" (?) is by giving warnings:
Code: Select all
cond:             conditions { txterrmsg(_ERROR, FALSE, lineno, "duplicate Conditions statement"); }
 
                 | actordisabled
                 | actorenabled
 ...
                 | selunqunit
                 | turnindex
 
 		| altercusubunit { txterrmsg(_WARNING, TRUE, linenorhs, "suspicious, unusual use of %s in Conditions section", "AlterCuSubUnit"); }
 ...
 
                 ;
And similarly for condition statements appearing in the Actions section:
Code: Select all
act:              actions { txterrmsg(_ERROR, FALSE, lineno, "duplicate Actions statement"); }
 
                 | absorbfaction
                 | addnationalregions
 ...
 		| winparams
                 | winparamsvp
 
                 | probability { txterrmsg(_WARNING, TRUE, linenorhs, "suspicious, unusual use of %s in Actions section", "Probability"); }
 ...
 
                 ;
By giving explicit warning messages, we avoid the more generic, and possibly confusing, "syntax error" messages.

Chliperic, I genuinely, really, truly appreciate your open-mindedness, cooperativeness, and patience. And your feedback.

As with most things in life, you get one chance to make your impression. With many people, you don't get a second chance.

As I develop AGElint and promote its use, I try to understand the, at best, indifference and, at worst, hostility it sometimes engenders.

I suspect that some people are looking for excuses to write me off, seizing for example the issue of false positives to disparage AGElint. They see false positives, and think: "Berto is a doofus. He doesn't know what he's doing. AGElint sucks." I think: "No, I know what I'm doing, in large part. I just need to do a better job of clarifying the AGElint error/warning statements [among other refinements, improvements, and bug fixes]."

That's what I suspect, anyway.

Let me tell you a story, as I've been in the modding field since ten years for several games. One dev of a game was a really gifted one. Unfortunately, he paired himself with a pure asshole, who was as much arrogant than unable to create something of value. he had some competences, which for the far-sighted seemed to be a proof of competence. Of course, he was always right. When he was wrong, he denied the reality, and if needed, modified on the spot offcial docs to hinder the error he made.He finally succeed to empty the communauty of his most dedicated members, and especially modders who were exchanging with the devs. He couldn't tolerate others could get a hand on the dev work. Uncompetence is often tied to vanity.He had always a reason to object, or at last he wanted to be the mandatory channel to authorize the idea.

I will not name this experience. Until I was bored with the engine, I just did my modding stuff a few more months, discovering even in the wargaming field, there are persons struggling for power against those trying to enhance things.

It's a lesson I will always keep in mind. Fatal Years is what can be done both for AI and historicity and I have dancing behind my eyes what will be SVF. I've so much fun to play FY I'm just hooked, and I know by the comments done I'm not the only one. In this year of hard work, I've got betatesters like I rarely saw, help by programer like you, suggestions by people speaking Russian and knwowing a bit the RCW, other having discovered this neglected period of history.

In one word, cheers to all. I'm happy. for those you mentionned, they will never know what freedom and openm ind may give as pleasure. They will live between wishful thinking, half-baked grandiose projects, and this constant necessity to deny their errors on scapegoats.

Let them. Build the games which are worth playing.

berto · Post by **berto** » Tue Jan 10, 2012 8:39 pm

ORIGINAL: berto

Another thing I need to do is to have fewer "syntax error:" and more "warning:" and/or use more 'txterrmsg(_ERROR, ..., <explicit error message> ...)' function calls.

Consider another seeming agelint false positive:

ERROR: in Events/ANMAI.sct, at (or near) line 644: syntax error: #E# (0x45), #EndEvent#

in event evt_nam_ANA_BreakAlliance_Tracker:

635 EvalRgnOwned = $Dobropole;NOT
636
637 Actions
638
639
640
641 SetControl = $Dobropole 5
642
643
644> EndEvent
645
646
647 SelectFaction = $ANM
648 StartEvent = evt_nam_AI_ANMlocalMC11|999|0|NULL|NULL|NULL|NULL
649
650 Conditions
651 EvalEvent = evt_nam_ANA_BreakAlliance_Tracker;=;1
652 CheckAILevel = 1
653 MinDate = 1919/01/01

You might think: "What's the problem? There is an Actions statement, and if you look in the actual Events/ANMAI.sct file, you see too that there are appropriate StartEvent and Conditions statements. Everything seems in order here. False positive!"

Except it's not a false positive, rather yet another misleading ERROR: line.

The problem is the missing ; field separator in the preceding statement

641 SetControl = $Dobropole 5

It should instead be

641 SetControl = $Dobropole;5

as the usage database shows

[root@telemann agelint]# ./kwddat SetControl
1 SetControl = $Achit;80
1 SetControl = $Chaikovskiy;80
1 SetControl = $Ertarskiy;80
1 SetControl = $Kamensk-Uralskiy;80
1 SetControl = $Kataysk;80
1 SetControl = $Krasnoufimsk;80
1 SetControl = $Kueda;80
1 SetControl = $Kungur;80
1 SetControl = $Miass;80
1 SetControl = $Ozersk;80
1 SetControl = $Redva;80
1 SetControl = $Shalya;80
1 SetControl = $Vassilevsko-Chaitanski;80
1 SetControl = $Vishnevogorsk;80
14 SetControl = 35
22 SetControl = 65
47 SetControl = 45
72 SetControl = 55
91 SetControl = 15
106 SetControl = 95
133 SetControl = 5
156 SetControl = 10
157 SetControl = 90
173 SetControl = 70
180 SetControl = 30
180 SetControl = 85
191 SetControl = 25
252 SetControl = 40
276 SetControl = 60
428 SetControl = 20
434 SetControl = 80
483 SetControl = 75
775 SetControl = 50
5487 SetControl = 0
112441 SetControl = 100

and as the AGEWiki page for the SetControl statement suggests. (But note: the AGEWiki is in error, because it stipulates | as the field separator; but again actual usage shows ; instead!)

In keeping with my vow to add clearer, more numerous explicit txterrmsg() errors and warnings, and reduce the frequency of ambiguous, misleading "syntax error" messages, I have modified the setcontrol parser rule to be:

Code: Select all

setcontrol:       _SETCONTROL eq pct
                 | _SETCONTROL eq alsval sc pct
                 | _SETCONTROL eq alsval pct { txterrmsg(_ERROR, TRUE, linenorhs, "missing ; field separator"); }
                 | _SETCONTROL eq alsval { txterrmsg(_ERROR, TRUE, linenorhs, "missing ; field separator, else missing value"); }
                 {
                 /*
                 http://www.ageod.net/agewiki/SetControl
                 Syntax:  SetControl = [RegionUID;]Value
                 */
                 }
                 ;

Why the two different error cases? It's because AGE aliases can contain spaces (<sigh> [:(]), so the txt.l lexer might view the '$Dobropole 5' as an alsval alone, not an alsval pct.

Do you begin (or continue) to see how difficult this all is, how much detailed attention, care, patience, skill and even artistry is necessary to Do It Right?

With that change, and after recompiling agelint and running it against the Events/ANMAI.sct file, we now get:

ERROR: in Events/ANMAI.sct, at (or near) line 641: missing ; field separator, else missing value

632 CheckAILevel = 1
633 MinDate = 1919/01/01
634 MaxDate = 1923/01/01
635 EvalRgnOwned = $Dobropole;NOT
636
637 Actions
638
639
640
641> SetControl = $Dobropole 5
642
643
644 EndEvent
645
646
647 SelectFaction = $ANM
648 StartEvent = evt_nam_AI_ANMlocalMC11|999|0|NULL|NULL|NULL|NULL
649
650 Conditions

Error pinpointed at the offending SetControl statement, no more misdirection to the correct EndEvent statement, "false positive" eradicated, mission accomplished!

I have lots of work ahead of me. In every case where common AGE script and data file coding errors lead to ambiguous, contextual "syntax error" complaints, I will have to account for those errors with new txt.y parser cases with meaningful, informative txterrmsg(_ERROR, ...) calls like the above.

Adding all of that will take time. The next official AGElint release will be delayed, I'm sad to say. [:(]

In order to minimize the wait, I will likely -- as pledged -- focus on RUS, ACW & ROP -- and just ignore the other games for now. Work on the other games will follow in the weeks ahead.

The good news is that I have mainly finished the work of adding all actions statements to the parser cond: (Conditions) section, but give txterrmsg() warnings; and adding all conditions statements to the parser act: (Actions) section, and likewise give txterrmsg() warnings.

Thank you for your continuing patience and interest!

Chilperic · Post by **Chilperic** » Tue Jan 10, 2012 9:05 pm

Grr...fixed this for the ANA, and forgot for their brothers ANM. next version[:D]

Thanks

berto · Post by **berto** » Wed Jan 11, 2012 6:00 am

I am making good progress toward the next official AGElint release.

If you wish, I can post an RC sooner rather than later. This RC would vet the RUS errors & warnings. But it would leave undone the ACW & ROP vetting, also tying up some loose ends. That would follow in the official release.

So, shall I release a RUS-ready RC soon, or just wait for the official?

AGElint -- an AGE debugging toolkit

RE: AGElint: AGEWiki docs vs. actual usage

AGElint: excludes.???.dat

RE: AGElint: excludes.???.dat

AGElint: reformatting report output

RE: AGElint: reformatting report output

RE: AGElint: reformatting report output

AGElint: more on false positives

RE: AGElint: more on false positives

RE: AGElint: more on false positives

RE: AGElint: more on false positives

AGElint: squelches & ignores

RE: AGElint: squelches & ignores

RE: AGElint: squelches & ignores

actions in Conditions: errors or false positives?

RE: actions in Conditions: errors or false positives?

RE: AGElint: more on false positives

RE: actions in Conditions: errors or false positives?

RE: AGElint: more on false positives

RE: AGElint: more on false positives

RE: AGElint: more on false positives