[tex-live] Testing TeX and Metafont
Nelson H. F. Beebe
beebe at math.utah.edu
Thu Mar 18 21:10:55 CET 2004
For several months in the early 1980s, Don Knuth and David Fuchs
collected program-counter data from the installed version of TeX at
the Stanford Computer Science Department, and used that data to figure
out what the time-critical execution paths in TeX are; that analysis
was used to guide some of the changes to the program to make it
efficient.
In some mission-critical defense and aerospace applications, it is a
contractual requirement that software be delivered with sufficient
test cases to demonstrate that either all lines of code have been
exercised at least once, or if not, a convincing argument must be
presented why the unexecuted lines will never be executed. The reason
for this requirement is simple:
Untested code is untrusted code!
BibTeX, TeX, and Metafont certainly do have some code that is not
expected to be executed:
% grep -c "this can't happen" ./*/*.web | grep -v ":0"
./bibtex/bibtex.web:52
./mf/mf.web:18
./tex/tex.web:36
For many years, I've been using the Sun compiler test-coverage
analyzer to instrument programs and produce a post-run-time report on
which lines of code got executed, and how often.
Until recently, most folks without Sun SunOS/Solaris systems and
compilers had no easy way to generate test-coverage analyses.
However, newer 3.x versions of GNU gcc offer the -ftest-coverage and
-fprofile-arcs compile-time options that result in runs that produce
data files that can be analyzed by gcov.
Since Don Knuth devised trip and trap tests for TeX and Metafont many
years ago, before test-coverage analysis was feasible in most
environments, I wondered how well those famous tests actually exercise
the TeX and Metafont code. The answer appears to be: not at all well.
Only 41% of the lines of code are executed at least once (details
below). [NB: 41% is the average of the reported percentages, not the
reported covered lines, which isn't quite the same thing.]
This strongly suggests that we should be developing a large collection
of (small) additional torture tests that cause much more of TeX and
Metafont to be executed. I think that this can be done by a community
effort with pleas posted to texhax and comp.text.tex: e.g., "we need a
small test file that exercises every line of code in section XXX of
TeX the Program" (the one published in Volume B of Computers and
Typesetting). There are enough TeX devotees that many will find it
personally challenging to come up with devious tests that will probe
much more of TeX: who knows, they might even turn up
previously-undetected bugs, for which Don promises in the
ftp://labrea.stanford.edu/pub/tex/DIFFS.june03 file:
% A reward of $327.68 will be paid to the first finder of any remaining bug.
Now that I have a very fast machine (1.4GHz Itanium-2) to make
experiments with, I decided to make a test-coverage report of the trip
and trap runs. gcov produces detailed source listings with the
execution counts, and I give a snippet at the end of this report.
What is of interest now, however, is just the percentages of code that
got executed after a fresh build with "make check", which runs both
trip and trap tests. The Web2c packaging of TeX and Metafont splits
the code into multiple files and directories; I give results here for
the directories web2c/tex, web2c/mf, web2c/lib, and kpathsea.
========================================================================
TeX test coverage
foreach f ( *.c )
echo ======== $f
gcov $f
end
======== iextra.c
51.25% of 160 source lines executed in file iextra.c
Creating iextra.c.gcov.
======== initex.c
62.32% of 2797 source lines executed in file initex.c
Creating initex.c.gcov.
======== itex.c
81.54% of 1495 source lines executed in file itex.c
Creating itex.c.gcov.
======== openinout.c
66.67% of 63 source lines executed in file openinout.c
Creating openinout.c.gcov.
======== patgen.c
Could not open data file patgen.da.
Assuming that all execution counts are zero.
0.00% of 1302 source lines executed in file patgen.c
Creating patgen.c.gcov.
======== tex0.c
48.45% of 1482 source lines executed in file tex0.c
Creating tex0.c.gcov.
======== tex1.c
25.84% of 1405 source lines executed in file tex1.c
Creating tex1.c.gcov.
======== tex2.c
58.25% of 1703 source lines executed in file tex2.c
Creating tex2.c.gcov.
======== tex3.c
69.15% of 1802 source lines executed in file tex3.c
Creating tex3.c.gcov.
======== tex4.c
84.70% of 1484 source lines executed in file tex4.c
Creating tex4.c.gcov.
======== tex5.c
85.98% of 1441 source lines executed in file tex5.c
Creating tex5.c.gcov.
======== tex6.c
71.99% of 1578 source lines executed in file tex6.c
Creating tex6.c.gcov.
======== tex7.c
59.70% of 1444 source lines executed in file tex7.c
Creating tex7.c.gcov.
======== tex8.c
64.14% of 1634 source lines executed in file tex8.c
Creating tex8.c.gcov.
======== tex9.c
62.89% of 159 source lines executed in file tex9.c
Creating tex9.c.gcov.
======== vextra.c
52.91% of 172 source lines executed in file vextra.c
Creating vextra.c.gcov.
========================================================================
Metafont test coverage
foreach f ( *.c )
echo ======== $f
gcov $f
end
======== iextra.c
40.62% of 192 source lines executed in file iextra.c
Creating iextra.c.gcov.
======== imf.c
Could not open data file imf.da.
Assuming that all execution counts are zero.
0.00% of 153 source lines executed in file imf.c
Creating imf.c.gcov.
======== inimf.c
91.53% of 815 source lines executed in file inimf.c
Creating inimf.c.gcov.
======== mf0.c
52.23% of 1505 source lines executed in file mf0.c
Creating mf0.c.gcov.
======== mf10.c
5.65% of 372 source lines executed in file mf10.c
Creating mf10.c.gcov.
======== mf1.c
34.07% of 1265 source lines executed in file mf1.c
Creating mf1.c.gcov.
======== mf2.c
35.59% of 1430 source lines executed in file mf2.c
Creating mf2.c.gcov.
======== mf3.c
23.05% of 1523 source lines executed in file mf3.c
Creating mf3.c.gcov.
======== mf4.c
39.87% of 1515 source lines executed in file mf4.c
Creating mf4.c.gcov.
======== mf5.c
20.08% of 1444 source lines executed in file mf5.c
Creating mf5.c.gcov.
======== mf6.c
54.54% of 1509 source lines executed in file mf6.c
Creating mf6.c.gcov.
======== mf7.c
31.59% of 1472 source lines executed in file mf7.c
Creating mf7.c.gcov.
======== mf8.c
20.16% of 1587 source lines executed in file mf8.c
Creating mf8.c.gcov.
======== mf9.c
37.12% of 1762 source lines executed in file mf9.c
Creating mf9.c.gcov.
======== openinout.c
64.41% of 59 source lines executed in file openinout.c
Creating openinout.c.gcov.
======== vextra.c
Could not open data file vextra.da.
Assuming that all execution counts are zero.
0.00% of 204 source lines executed in file vextra.c
Creating vextra.c.gcov.
========================================================================
Library coverage
foreach f ( *.c )
echo ======== $f
gcov $f
end
======== alloca.c
Could not open data file alloca.da.
Assuming that all execution counts are zero.
0.00% of 34 source lines executed in file alloca.c
Creating alloca.c.gcov.
======== eofeoln.c
88.24% of 17 source lines executed in file eofeoln.c
Creating eofeoln.c.gcov.
======== fprintreal.c
Could not open data file fprintreal.da.
Assuming that all execution counts are zero.
0.00% of 4 source lines executed in file fprintreal.c
Creating fprintreal.c.gcov.
======== getopt1.c
Could not open data file getopt1.da.
Assuming that all execution counts are zero.
No executable code associated with file getopt1.c.
======== getopt.c
Could not open data file getopt.da.
Assuming that all execution counts are zero.
No executable code associated with file getopt.c.
======== inputint.c
Could not open data file inputint.da.
Assuming that all execution counts are zero.
0.00% of 20 source lines executed in file inputint.c
Creating inputint.c.gcov.
======== main.c
33.33% of 12 source lines executed in file main.c
Creating main.c.gcov.
======== openinout.c
Could not open basic block file openinout.bb.
======== ourpaths.c
68.09% of 47 source lines executed in file ourpaths.c
Creating ourpaths.c.gcov.
======== strpascal.c
48.15% of 27 source lines executed in file strpascal.c
Creating strpascal.c.gcov.
======== texmf.c
Could not open basic block file texmf.bb.
======== uexit.c
87.50% of 8 source lines executed in file uexit.c
Creating uexit.c.gcov.
======== version.c
Could not open data file version.da.
Assuming that all execution counts are zero.
No executable code associated with file version.c.
======== xfopen-pas.c
81.82% of 11 source lines executed in file xfopen-pas.c
Creating xfopen-pas.c.gcov.
======== zround.c
81.82% of 11 source lines executed in file zround.c
Creating zround.c.gcov.
========================================================================
kpathsea coverage
======== absolute.c
100.00% of 5 source lines executed in file absolute.c
Creating absolute.c.gcov.
======== concat3.c
0.00% of 8 source lines executed in file concat3.c
Creating concat3.c.gcov.
======== concat.c
100.00% of 6 source lines executed in file concat.c
Creating concat.c.gcov.
======== concatn.c
0.00% of 11 source lines executed in file concatn.c
Creating concatn.c.gcov.
======== debug.c
Could not open data file debug.da.
Assuming that all execution counts are zero.
No executable code associated with file debug.c.
======== default.c
66.67% of 24 source lines executed in file default.c
Creating default.c.gcov.
======== dir.c
40.00% of 10 source lines executed in file dir.c
Creating dir.c.gcov.
======== elt-dirs.c
52.22% of 90 source lines executed in file elt-dirs.c
Creating elt-dirs.c.gcov.
======== expand.c
85.71% of 7 source lines executed in file expand.c
Creating expand.c.gcov.
======== extend-fname.c
0.00% of 6 source lines executed in file extend-fname.c
Creating extend-fname.c.gcov.
======== file-p.c
100.00% of 6 source lines executed in file file-p.c
Creating file-p.c.gcov.
======== find-suffix.c
0.00% of 10 source lines executed in file find-suffix.c
Creating find-suffix.c.gcov.
======== fn.c
36.59% of 41 source lines executed in file fn.c
Creating fn.c.gcov.
======== fontmap.c
0.00% of 69 source lines executed in file fontmap.c
Creating fontmap.c.gcov.
======== gmalloc.c
Could not open basic block file gmalloc.bb.
======== init-path.c
100.00% of 10 source lines executed in file init-path.c
Creating init-path.c.gcov.
======== itoa.c
Could not open data file itoa.da.
Assuming that all execution counts are zero.
0.00% of 5 source lines executed in file itoa.c
Creating itoa.c.gcov.
======== line.c
0.00% of 21 source lines executed in file line.c
Creating line.c.gcov.
======== make-suffix.c
Could not open data file make-suffix.da.
Assuming that all execution counts are zero.
0.00% of 13 source lines executed in file make-suffix.c
Creating make-suffix.c.gcov.
======== path-elt.c
100.00% of 26 source lines executed in file path-elt.c
Creating path-elt.c.gcov.
======== pathsearch.c
84.06% of 69 source lines executed in file pathsearch.c
Creating pathsearch.c.gcov.
======== putenv.c
Could not open data file putenv.da.
Assuming that all execution counts are zero.
No executable code associated with file putenv.c.
======== readable.c
57.14% of 14 source lines executed in file readable.c
Creating readable.c.gcov.
======== rm-suffix.c
0.00% of 12 source lines executed in file rm-suffix.c
Creating rm-suffix.c.gcov.
======== str-list.c
42.86% of 21 source lines executed in file str-list.c
Creating str-list.c.gcov.
======== str-llist.c
62.07% of 29 source lines executed in file str-llist.c
Creating str-llist.c.gcov.
======== tex-font.c
Could not open data file tex-font.da.
Assuming that all execution counts are zero.
0.00% of 18 source lines executed in file ./tex-font.c
Creating tex-font.c.gcov.
======== tex-glyph.c
Could not open data file tex-glyph.da.
Assuming that all execution counts are zero.
0.00% of 127 source lines executed in file tex-glyph.c
Creating tex-glyph.c.gcov.
======== tex-make.c
0.00% of 105 source lines executed in file tex-make.c
Creating tex-make.c.gcov.
======== tilde.c
32.00% of 25 source lines executed in file tilde.c
Creating tilde.c.gcov.
======== truncate.c
0.00% of 17 source lines executed in file truncate.c
Creating truncate.c.gcov.
======== uppercasify.c
0.00% of 8 source lines executed in file uppercasify.c
Creating uppercasify.c.gcov.
======== variable.c
23.91% of 46 source lines executed in file variable.c
Creating variable.c.gcov.
======== version.c
Could not open data file version.da.
Assuming that all execution counts are zero.
No executable code associated with file version.c.
======== xcalloc.c
0.00% of 8 source lines executed in file xcalloc.c
Creating xcalloc.c.gcov.
======== xfopen.c
0.00% of 8 source lines executed in file xfopen.c
Creating xfopen.c.gcov.
======== xmalloc.c
75.00% of 8 source lines executed in file xmalloc.c
Creating xmalloc.c.gcov.
======== xopendir.c
0.00% of 9 source lines executed in file xopendir.c
Creating xopendir.c.gcov.
======== xputenv.c
0.00% of 32 source lines executed in file xputenv.c
Creating xputenv.c.gcov.
======== xrealloc.c
83.33% of 12 source lines executed in file xrealloc.c
Creating xrealloc.c.gcov.
======== xstat.c
Could not open data file xstat.da.
Assuming that all execution counts are zero.
0.00% of 11 source lines executed in file xstat.c
Creating xstat.c.gcov.
======== xstrdup.c
100.00% of 4 source lines executed in file xstrdup.c
Creating xstrdup.c.gcov.
========================================================================
Here is a fragment showing the TeX hot spots (lines executed more than
2M times) in the trip test:
% awk '($1 ~ /^ *[0-9]+$/) && ($1 > 2000000) {print FILENAME ":" FNR ":" $0}' *.gcov
tex0.c.gcov:1245: 2275403 while ( ( mem [ q ] .hh .v.RH == 262143L ) ) {
tex2.c.gcov:655: 5282180 {/* 20 25 21 26 40 10 */ getnext_regmem
tex2.c.gcov:661: 5917006 lab20: curcs = 0 ;
tex2.c.gcov:662: 5917006 if ( curinput .statefield != 0 )
tex2.c.gcov:1021: 4455955 }
tex2.c.gcov:1023: 4455955 else if ( curinput .locfield != 0 )
tex2.c.gcov:1025: 3871909 t = mem [ curinput .locfield ] .hh .v.LH ;
tex2.c.gcov:1026: 3871909 curinput .locfield = mem [ curinput .locfield ] .hh .v.RH ;
tex2.c.gcov:1027: 3871909 if ( t >= 4095 )
tex2.c.gcov:1069: 3829508 }
tex2.c.gcov:1070: 3829508 }
tex2.c.gcov:1075: 5290550 }
tex2.c.gcov:1076: 5290550 if ( curcmd <= 5 )
tex2.c.gcov:1089: 5282179 }
tex2.c.gcov:1090: 5282180 }
tex2.c.gcov:1668: 2304327 {/* 20 30 */ getxtoken_regmem
tex2.c.gcov:1669: 2304327 lab20: getnext () ;
tex2.c.gcov:1670: 2609827 if ( curcmd <= 100 )
tex2.c.gcov:1671: 2296047 goto lab30 ;
tex2.c.gcov:1683: 2304327 lab30: if ( curcs == 0 )
The trap test for Metafont appears to be much less demanding, since
the line counts are more than an order of magnitude lower: here are
the Metafont hotspots for the trap test:
% awk '($1 ~ /^ *[0-9]+$/) && ($1 > 50000) {print FILENAME ":" FNR ":" $0}' *.gcov
iextra.c.gcov:236: 141565 while (last < bufsize && (i = getc (f)) != EOF && i != '\n')
iextra.c.gcov:237: 137807 buffer[last++] = i;
iextra.c.gcov:392: 143910 {
iextra.c.gcov:393: 143910 char temp;
iextra.c.gcov:397: 143910 switch (size)
iextra.c.gcov:417: 179006 while (nitems--)
iextra.c.gcov:419: 89503 SWAP (p[0], p[7]);
iextra.c.gcov:420: 89503 SWAP (p[1], p[6]);
iextra.c.gcov:421: 89503 SWAP (p[2], p[5]);
iextra.c.gcov:422: 89503 SWAP (p[3], p[4]);
iextra.c.gcov:423: 89503 p += size;
iextra.c.gcov:424: 89503 }
iextra.c.gcov:425: 89503 break;
iextra.c.gcov:428: 108814 while (nitems--)
iextra.c.gcov:430: 54407 SWAP (p[0], p[3]);
iextra.c.gcov:431: 54407 SWAP (p[1], p[2]);
iextra.c.gcov:432: 54407 p += size;
iextra.c.gcov:433: 54407 }
iextra.c.gcov:434: 54407 break;
iextra.c.gcov:451: 143910 }
iextra.c.gcov:465: 59987 {
iextra.c.gcov:467: 59987 swap_items (p, nitems, item_size);
iextra.c.gcov:470: 59987 if (fwrite (p, item_size, nitems, out_file) != nitems)
iextra.c.gcov:475: 59987 }
iextra.c.gcov:480: 59987 swap_items (p, nitems, item_size);
mf0.c.gcov:1112: 50887 while ( j < strstart [ s + 1 ] ) {
mf1.c.gcov:941: 57540 while ( h >= 7919 ) h = h - 7919 ;
mf6.c.gcov:317: 57487 while ( charclass [ buffer [ curinput .locfield ] ] == class ) incr (
-------------------------------------------------------------------------------
- Nelson H. F. Beebe Tel: +1 801 581 5254 -
- University of Utah FAX: +1 801 581 4148 -
- Department of Mathematics, 110 LCB Internet e-mail: beebe at math.utah.edu -
- 155 S 1400 E RM 233 beebe at acm.org beebe at computer.org -
- Salt Lake City, UT 84112-0090, USA URL: http://www.math.utah.edu/~beebe -
-------------------------------------------------------------------------------
More information about the tex-live
mailing list