tex-live performance

[Anton Ertl]

On Sun, Apr 04, 2021 at 06:46:27PM +0900, Norbert Preining wrote:
> Hi Anton,
> 
> On Sat, 03 Apr 2021, Anton Ertl wrote:
> > https://www.complang.tuwien.ac.at/franz/latex-bench
> > https://www.complang.tuwien.ac.at/anton/latex-bench/
> 
> Looks interesting.
> 
> > difference?  Other explanations are possible, e.g., the Debian texlive
> > maintainers might build the Testing texlive package with less
> 
> No (I am also maintainer of TL in Debian), we use maximal optimization.
> 
> My guess is that it really depends - as Ulrike said - on the moving
> targets. You are comparing different things, because there are different
> levels:
> - the actual engine (pdfetex, luatex)
> - the format code (latex, context)
> The first one doesn't change a lot. I think there has been only one
> considerable code change in pdf*tex and that is related to how
> case-sensitive/insensitive look-ups are treated. That could be - and in
> some cases turned out to be - a source of increased instructions being
> used.
> 
> Then there is the format, and since nowadays the LaTeX format has
> evolved quite a lot over the last few years (just one thing - utf8 is
> enabled by default) that can of course generate a lot of other activity.
> 
> So I guess the best would be - but that takes probably time and hacking
> - to use the same format files across the tests. AFAIR between 2018 and
> 2020 it should be possible to use the latest files from 2020 and the
> binaries from 2018 (if library wise it works - you could just drop the
> pdfetex binary from 2018/debian stable into /usr/bin of testing and
> rebuild the formats with fmtutil-sys --byengine pdfetex or similar).
> 
> That should give more comparable results at least what the binaries are
> concerned.

Thank you.  I have tried to use the old format with the new binary,
but unfortunately I get:

> latex -fmt /debian8/var/lib/texmf/web2c/pdftex/latex.fmt bench
This is pdfTeX, Version 3.14159265-2.6-1.40.21 (TeX Live 2020/Debian) (preloaded format=/debian8/var/lib/texmf/web2c/pdftex/latex.fmt)
 restricted \write18 enabled.
---! /debian8/var/lib/texmf/web2c/pdftex/latex.fmt made by different executable version
(Fatal format file error; I'm stymied)

So it seems that, to use the benchmark for comparing hardware, we
would need to build a specific release of tex-live, and run that
release on all machines.  Maybe with docker or a virtual machine, but
that's beyond the stuff the web pages above are trying to do.  So what
<https://www.complang.tuwien.ac.at/franz/latex-bench> is showing is
the performance of the combination of hardware and tex-live
installation (including how many packages are in the installation);
that (especially the influence of the number of packages) may be less
helpful than one might wish for, but I'll still leave it at that.

Here are some more data that shows the influence of different versions
and different installations:

insts cyc.  arch  Hardware        installation
1937M 2296M IA-32 Athlon 800 (K7) 2001
1950M  868M AMD64 Core i5-6600K   Debian 8 (2015)
2072M  955M AMD64 Core i5-6600K   Debian 8 more packages
2635M 1179M AMD64 Core i5-6600K   Debian 11 (2021) texlive (only)
3010M 1565M AMD64 Core i5-6600K   Debian 11 texlive-full

It seems that there has been an increase in the number of instructions
between 2001 and 2015, but not very pronounced (it's not that clear
how the switch from IA-32 to AMD64 affected instructions; looking at
the Athlon 64 3200+ results on
<https://www.complang.tuwien.ac.at/franz/latex-bench>, the timing
difference is not big, so the difference in instructions is probably
not big, either).

One interesting aspect is that having more packages increaes the
cycles by a larger proportion than it increases the instructions,
resulting in a slowdown of texlive-full over texlive on Debian 11 by a
factor of 1.33.

- anton