File descriptor error for pdflatex.fmt creation on android 11

[Marcel Fabian Krüger]

On Sun, Apr 04, 2021 at 12:23:03PM +0200, Henrik Grimler wrote:
> Hi,
> 
> I noticed two errors in my small test program (return value of
> open_output was not checked and gzFile was uninitialized). I fixed
> those but am then back at getting the fdsan error with the macro, but
> not with it defined as a function. Compiling and running the attached
> test.c gives the error, but removing the wopenout-macro and un-
> commenting the wopenout-function makes it work.
> 
> Since using a function instead of a macro works for the small test
> program I tried building texlive with wopenout converted to a function
> (see attached wopenout.patch), but pdftex then seem to be unable to
> open the fmtfile, it stops with:
> 

Hi,

the "working" small test program is kind of a red herring: Since it ever
uses the file after openig it, you never realize that the function
doesn't work. The issue is that the macro assigns the opened file handle
to the variable passed as parameter. In a macro this does the expected
thing, but in a function this just changes the local copy of the
parameter leaving the original variable unchanged. So in your example
program `fp` is still NULL after `wopenout` and `gzclose` just happens
to ignore it if a nullptr is passed (Actually it returns the error code
Z_STREAM_ERROR, but your program doesn't test for that)

The zlib documentation recommends using `dup` to copy the descriptor in
such cases to get a descriptor which is independent from the FILE*
managed one (which can then be closed normally) but that is incompatible
with the variable reuse in the current macro. So the macro must first be
rewritten to use separate variables, which is complicated since the
macro is a expression and you can't declare variables in the middle of
an expression.

Best regards,
Marcel

> ```
> (/data/data/com.termux/files/usr/share/texlive/texmf-
> dist/tex/latex/firstaid/la
> tex2e-first-aid-for-external-files.ltx)
>  ) )
> Beginning to dump on file pdflatex.fmt
> ! Could not write 1 4-byte item(s) to pdflatex.fmt.
>  (preloaded format=pdflatex 2021.4.4)
> ```
> 
> Which indicates that I have made some error in the patch, or that there
> is some other error in wopenout. Can anyone see any problems in
> wopenout.patch?
> 
> One workaround that works is to not compress the format file, i.e.
> undef FMT_COMPRESS in texmfmp.h.
> 
> Best regards,
> Henrik Grimler
> 
> On Sat, 2021-04-03 at 20:50 +0200, Henrik Grimler wrote:
> > Hello again, 
> > 
> > I have created a small test program (attached as test.c) that
> > reproduces the problem by condensing the relevant code in
> > texk/web2c/texmfmp.h and texk/web2c/lib/openclose.c.
> > 
> > It compiles without errors with clang-11 on both android (arm) and
> > linux (x86_64) with `clang -lz -o test test_reproduced.c`, and works
> > fine on linux, but when run on android it gives one of those file
> > descriptor errors:
> > 
> > fdsan: attempted to close file descriptor 3, expected to be unowned,
> > actually owned by FILE* 0xacdb000c
> > 
> > The problem seem to have to do with how the format file is opened. In
> > texmfmp.h we have:
> > 
> > ```c
> > #define wopenout(f) (open_output ((FILE**)&(f), FOPEN_WBIN_MODE) \
> >  && (f = gzdopen(fileno((FILE*)f), FOPEN_WBIN_MODE)) \
> >  && (gzsetparams(f, 1, Z_DEFAULT_STRATEGY) == Z_OK))
> > ```
> > 
> > If I convert the macro into a function instead:
> > 
> > ```c
> > int wopenout (gzFile f)
> > {
> >   open_output ((FILE**)&(f), "wb");
> >   f = gzdopen(fileno((FILE*)f), "wb");
> >   return gzsetparams(f, 1, Z_DEFAULT_STRATEGY) == Z_OK;
> > }
> > ```
> > 
> > the error no longer occurs. 
> > 
> > My first thought was that the issue is related to how gzFile is casted
> > to FILE, but since the problem disappears when defined as a function I
> > suppose this might be a bug in the android libc and/or fd sanitizer. 
> > 
> > Could macros, as the one defined for wopenout, be problematic? I.e.
> > would it make sense to convert wopenout into a normal function or is
> > this most likely an android-specific bug?
> > 
> > Best regards,
> > Henrik Grimler
> > 
> > 
> > 
> > On Fri, 2021-04-02 at 15:45 +0200, Henrik Grimler wrote:
> > > Dear all,
> > > 
> > > Android 11 introduced a new file descriptor sanitizer ("fdsan", see
> > > [1]
> > > for docs) which detects possible issues when opening and closing
> > > files
> > > across multiple threads.
> > > 
> > > Installing texlive on android 11 gives a fdsan error when creating
> > > pdflatex.fmt. If I run:
> > > 
> > >   pdftex -ini -jobname=pdflatex -progname=pdflatex -translate-
> > > file=cp227.tcx *pdflatex.ini
> > > 
> > > it aborts with an error
> > > 
> > >   fdsan: attempted to close file descriptor 3, expected to be
> > > unowned,
> > > actually owned by FILE* 0xb6c6100c
> > > 
> > > Does anyone have experience with this type of issue? For debugging it
> > > the docs suggests replacing all `int fd` with a special `unique_fd
> > > fd`
> > > type, but I am not yet sure how to use that for software like texlive
> > > (rather than android apps), so any hints are greatly appreciated.
> > > 
> > > If I run the command above in gdb I get this backtrace (see [2] for
> > > full log):
> > > 
> > > [...]
> > > Hyphenation trie of length 6081 has 183 ops out of 35111
> > >   2 for language 1
> > >   181 for language 0
> > > 0 words of pdfTeX memory
> > > fdsan: attempted to close file descriptor 3, expected to be unowned,
> > > actually owned by FILE* 0xb6c6100c
> > > 
> > > Program received signal SIGABRT, Aborted.
> > > 0xb64ec9e8 in fdsan_error(char const*, ...) () from
> > > /apex/com.android.runtime/lib/bionic/libc.so
> > > (gdb) bt
> > > #0  0xb64ec9e8 in fdsan_error(char const*, ...) () from
> > > /apex/com.android.runtime/lib/bionic/libc.so
> > > #1  0xb64ec6fe in android_fdsan_close_with_tag () from
> > > /apex/com.android.runtime/lib/bionic/libc.so
> > > #2  0xb64ecd6e in close () from
> > > /apex/com.android.runtime/lib/bionic/libc.so
> > > #3  0xb68b18f4 in gzclose_w () from
> > > /data/data/com.termux/files/usr/lib/libz.so.1.2.11
> > > #4  0x7f56abd8 in storefmtfile () at pdftexini.c:3584
> > > #5  0x7f570806 in mainbody () at pdftexini.c:5651
> > > #6  0x7f5609c0 in main (ac=<optimized out>, av=<optimized out>)
> > >     at /home/builder/.termux-build/texlive-
> > > bin/src/texk/web2c/lib/texmfmp.c:1098
> > > 
> > > The above output is for pdftex built from the svn54456 tag. 
> > > Please let me know if I can provide any additional information!
> > > 
> > > Best regards
> > > Henrik Grimler
> > > 
> > > [1]
> > > https://android.googlesource.com/platform/bionic/+/master/docs/fdsan.md
> > > [2] https://grimler.se/logs/pdftex.log
> > > 
> > 
>