[tex-live] ls-R files and nasty symlinks

[Heiko Oberdiek]

Hello,

On 10/17/2014 07:01 PM, Nelson H. F. Beebe wrote:
> We recently noticed unexpected fileserver loads, and tracked down the
> cause: creation of TeX Live ls-R file in user directory trees when
> their trees contain a file that is a symlink to the root of the
> filesystem, /.  
> 
> In our case, the filesystem contains about 100TB of space, and the
> root / may contain mount points for 30 or so other filesystems.  The
> result is a huge delay in creating the ls-R files, and those are big:
> tens of MB, or even larger (we killed off the offending mktexlsr
> processes before they completed).
> 
> You might wonder how symlinks to / get created.  Here are two examples
> from widely-used software:
> 
> 	 /home/USERNAME/.cxchromium/chromium/dosdevices/z:
> 	 /home/USERNAME/.wine/dosdevices/z:
> 
> In both cases, z: is a symbolic link to `/'.  A scan of our filesystem
> found about 120 such instances among our thousands of users, and about
> 100 ls-R files in user directories.

The files for the TeX system are organized in TDS trees, a root ("texmf"
or similar) with a defined directory structure, see
http://texdoc.net/texmf-dist/doc/generic/tds/tds.pdf

The index files "ls-R" are generated in the root of these TDS
trees. It does not make any sense to include the root of the
file system as symbol link inside the TDS trees, because the
overall root does not follow the TDS structure
conventions. Also indexing lots of unrelated files only
slows down the generating of the index files and the usage
of the TeX system.

The symbol link examples are *outside* of the TDS trees,
therefore they should not be touched by mktexlsr at all.

> Examination of texlive/YYYY/bin/ARCH/mktexlsr shows that it is a shell
> script of about 220 lines, and it appears that the file listing is
> made by this command at line 203 of the 2014 mktexlsr script, which is
> identical across all platforms within a given year:
> 
> 	(cd "$TEXMFLS_R" && \ls -LRa 2>/dev/null) \
> 
> The problem is the -L option, which according to the Solaris manual
> page for ls works like this:
> 
>      -L           If an argument is a symbolic link, this  option
>                   evaluates the file information and file type of
>                   the file or directory that the link references,
>                   rather  than those of the link itself. However,
>                   the name of the link is displayed, rather  than
>                   the referenced file or directory.
> 
> Unlike the Unix "find" command that might have been used for the job
> (although the output would need further processing to make it look
> like the current ls-R format), "ls" has no options like "find"'s
> -mount or -xdev, which prevent descents into directories on other
> filesystems.

Bad idea. Why would someone wants to put a symbol link in a TDS tree?
Certainly the link needs to be dereferenced to find the file or
the directory with its children.
   For example, putting the seldom used "source" subdirectory to a
larger, cheaper, but maybe slower file system would make make sense.
Therefore the index search *must* dereference symbol links and cross
file system boundaries.

> Does anyone have an idea how we might prevent mktexlsr from spending
> large amounts of time listing files that are outside the user's
> personal tree?

It looks like that mktexlsr is called on the wrong directory, which
are not the root of TDS trees (e.g. running on $HOME).
The cause might be a misconfiguration (environment variables,
texmf.cnf) or wrong manual usage.

> Removing the -L option would do that, but that is a change with a big
> enough effect that it could foil TeX Live user expectations at many
> sites.

That would be a bug.

It makes much more sense to fix the underlying cause.
Depending on the configuration TeX and friends might
search the whole tree on each call, if a file could
not be found in the ls-R files, that's not a good idea
either, if the complete file system starting from the
overall root is included in the search.

Yours sincerely
  Heiko Oberdiek