How should I deal with packages that require FFI external libraries?

[Max Chernoff]

Hi Christophe,

> 

> 
> 1) When the external library is loaded, I have something like:
> 
> lib = ffi.load('<path to the .so file>')
> 
> For now, the path is hardcoded. Where should I put the .so file so
> that luatex finds it? What is good practice for CTAN/TeXLive? Does
> luatex rely on kpsewhich for that kind of thing?

To get the path to the file, the "proper" way is to use

   kpse.find_file("library-name", "clua")

which will look in

   $ kpsewhich --var-value=CLUAINPUTS
   .:/usr/local/texlive/2023/bin/x86_64-linux/lib/{kpsewhich,unsetengine,}/lua//

by default. If you're distributing your package in a .zip/.tar, that's
not the easiest path to add things to, so something like

   kpse.find_file("library-name.so") or kpse.find_file("library-name.dll") 

will likely be easier to distribute (although not technically
"correct").

> 
> 2) Can I distribute the .so file (or .dll) with the CTAN/TeXLive
> package

My full TL installation doesn't have any shared libraries installed

   $ find /usr/local/texlive/2023 -iname '*.so'

so you probably can't distribute it in TL. TLContrib and MiKTeX _might_
accept it, I have no idea. 

There aren't any .so's on CTAN, but there are a few .dll's. All the
.dll's are over 15 years old though, so I have no idea if they'll accept
any new ones. You'll have to ask them.

> 
> 3) Does FFI work in luatex on all architectures supported by TeXlive?

The documentation

   https://github.com/TeX-Live/luatex/tree/master/source/texk/web2c/luatexdir/luaffi

says that it only supports Linux and macOS, x86 and x86_64. There is a
msvc build file and a bunch of files with "win" in the name, so I'm
pretty sure that it supports Windows too. There are also support files
with "arm" in the name, but they're 5 years old so probably Linux only.

> By the way, I cannot find any reference to the ffi library in the
> luatex manual: I missed something?
> 

You are correct, there is no reference to it at all in the manual. I
didn't even know that LuaTeX included an FFI library until this email.

> 4) Can I compile the library for windows and macos on my linux
> computer? (Sorry, I know it is out of scope.)
> 

You can use mingw from Linux to compile for Windows, and there is
probably a clang-based cross-compiler for macOS. Cross-compiling can
range in difficulty from "replace gcc with x86_64-w64-mingw32-gcc and
compile as normal" to a really complicated and time consuming process.

It looks like CoolProp distributes binaries for Linux, macOS, and
Windows, so you could just use those.

> 5) Regarding security, I don't think it is an issue since ffi est
> shell-escape protected by default in TeXLive. But what do you think?
> 

Well there's the same security issue as always: if you enable shell
escape, then any package can do all sorts of bad things to your
computer. If you don't enable shell escape, then you can't use ffi. But
just adding the shared library to your TEXMF trees won't have any
security implications.

> 6) Regarding the licence: the library has an MIT licence. Is it ok for
> TeXLive?
> 

That should be fine, but (as mentioned above) it's unlikely that you'll
get any binaries into TL.

> I'm currently writing a generic luatex package that relies on an
> external library (https://github.com/CoolProp/CoolProp) called via the
> FFI library in luatex

A much simpler way to do all of this would be to use PythonTeX. CoolProp
distributes binary wheels, so you'd just need to tell your users to run 

   $ pip install CoolProp
   
Then, your package code would look something like

   \pyc{import CoolProp}
   \def\dowhatever#1{\py{CoolProp.do_whatever("#1")}
   
and a document would look like

   \usepackage{coolprop}
   ...
   The value of whatever at 55 is \dowhatever{55}.
   
Here are the docs:

   https://texdoc.org/serve/pythontex/2

> I'm not an expert at
> compiling complex projects (I can read a `README` file and type
> `make`), neither at binding C to lua,

You could also write a Lua module in C directly. This would require you
to write some C code, but it may end up being easier than using the ffi
library since (1) all the CoolProp example code is probably in C, and
(2) the compiler would give you proper warnings and errors so you'll
always be sure that you're calling the functions correctly.

The FFI approach should work, but I think that you'll have better luck
using either PythonTeX or a custom C module.

-- Max
>