Apparent bug in pdftex/xetex (but not luatex)

[Karl Berry]

    first 256 strings are statically initialized for internal purposes

Yeah. Indeed, I see that Knuth used 256 in {tex,mf}.web in his
non-implementation of editing:

    "E": if base_ptr>0 then if input_stack[base_ptr].name_field>=256 then
      begin print_nl("You want to edit file ");
    @.You want to edit file x@>
      slow_print(input_stack[base_ptr].name_field);

Therefore it seems we may as well follow suit?  So I adjusted the patch
and my rather verbose commentary about the code.  I probably messed up
somehow.  If any suggestions, let me know. I'll commit tomorrow barring
problems.

    MetaPost doesn't use the code

Right, I forgot. Living in the past.

    implies that the filename texmfmp.c is at least misleading, 

Yeah, I guess we (I) could/should rename it. --thanks, karl.


Index: lib/ChangeLog
===================================================================
--- lib/ChangeLog	(revision 59012)
+++ lib/ChangeLog	(working copy)
@@ -1,3 +1,16 @@
+2021-04-28  Marcel Fabian Krueger  <tex at 2krueger.de>
+	    Hironori KITAGAWA <h_kitagawa2001 at yahoo.co.jp>
+	    Karl Berry <karl at freefriends.org.
+
+	* texmfmp.c (calledit): Skip \scantokens pseudo-files, indeed all
+	name_field values <=255 (following tex.web), and token lists
+	(state_field==0). Original reports from Don Hosek,
+	https://tex.stackexchange.com/questions/594702
+	and Hironori (plus patch),
+	https://tug.org/pipermail/tex-k/2021-April/003542.html
+	and Marcel (plus patch) for \scantokens,
+	https://tug.org/pipermail/tex-live/2021-April/046846.html.
+
 2021-03-23  Karl Berry  <karl at tug.org>
 
 	* TL'21.
Index: lib/texmfmp.c
===================================================================
--- lib/texmfmp.c	(revision 59012)
+++ lib/texmfmp.c	(working copy)
@@ -2624,29 +2624,49 @@ calledit (packedASCIIcode *filename,
      and a non-file for the insert. https://tex.stackexchange.com/q/552113 
      
      Therefore, we have to traverse down input_stack (not input_file),
-     looking for name_field values >17, which correspond to open
-     files, and then the index_field value of that entry tells us the
+     looking for large enough name_field values corresponding to open
+     files. Then the index_field value of that entry tells us the
      corresponding element of input_file, which is what we need to close.
+     Additionally we have to skip all entries with state_field 0 since these
+     correspond to token lists and not input files.
 
-     We test for >17 because name_field=0 means the terminal,
+     We test for name_field<=255, following tex.web, because the first
+     256 strings are static, initialized by TeX. (Well, many more
+     strings are initialized, but we'll follow tex.web.)
+     
+     For the record, name_field=0 means the terminal,
      name_field=1..16 means \openin stream n - 1,
-     name_field=17 means an invalid stream number (for read_toks).
-     Although ... seems like we should close any opened \openin files also.
-     Whoever is reading this, please implement that? Sigh.
+     name_field=17 means an invalid stream number (for read_toks),
+     name_field=18..19 means \scantokens pseudo-files (except for
+     original TeX of course). But 255 suffices for us.
      
-     Description in modules 300--304 of tex.web: "Input stacks and states."
-     
      Here, we do not have to look at cur_input, the global variable
      which is effectively the top of input_stack, because it will always
      be a terminal (non-file) interaction -- the one where the user
-     typed "e" to start the edit.  */
+     typed "e" to start the edit.
+     
+     In addition, state_field will be zero for token lists. Skip those too.
+     (Does not apply to Metafont.)
+
+     Description in modules 300--304 of tex.web: "Input stacks and states".
+     
+     We should close any opened \openin files also. Whoever is reading
+     this, please implement that?  */
  {  
   int is_ptr; /* element of input_stack, 0 < input_ptr */  
   for (is_ptr = 0; is_ptr < inputptr; is_ptr++) {
-    if (inputstack[is_ptr].namefield <= 17) {
+#ifdef TeX
+    if (inputstack[is_ptr].statefield == 0 /* token list */
+        || inputstack[is_ptr].namefield <= 255) { /* can't be filename */
+#elif defined(MF)
+    if (inputstack[is_ptr].namefield <= 255) {
+#else
+#error "Unable to identify program" /* MetaPost doesn't use this file */
+#endif
         ; /* fprintf (stderr, "calledit: skipped input_stack[%d], ", is_ptr);
-             fprintf (stderr, "namefield=%d <= 17\n",
-                      inputstack[is_ptr].namefield); */
+             fprintf (stderr, "namefield=%d <= 255 or statefield=%d == 0\n",
+                      inputstack[is_ptr].namefield,
+                      inputstack[is_ptr].statefield); */
     } else {
       FILE *f;
       /* when name_field > 17, index_field specifies the element of