[texhax] Strange behavior of \scantokens

Oleg Katsitadze olegkat at gmail.com
Fri Nov 30 22:56:26 CET 2007

Dear TeX hackers,

Can anybody please explain why the following code:

  \catcode`\^^M=13 \scantokens{x\endinput}x\bye

inserts a newline between the two x's?  What makes this really
mysterious is that when I change it to


I don't get a space between the x's.

I'm running this through

  $ pdftex --version
  pdfTeX using libpoppler 3.141592-1.40.3-2.2 (Web2C 7.5.6)
  kpathsea version 3.5.6

TIA for any clues,

P.S.  FWIW, when I emulate \scantokens under vanilla tex, it behaves
exactly the same, so I suspect it's something about how TeX's \input
works.  My emulation is (from Texinfo via Eplain):

  \let\tmpfilebasename = \jobname
  \newwrite\eplain at tmpfile
    \immediate\openout\eplain at tmpfile=\tmpfilebasename\tmpfileextension
    \immediate\write\eplain at tmpfile{\the\toks@}%
    \immediate\closeout\eplain at tmpfile
    \input \tmpfilebasename\tmpfileextension\relax

More information about the texhax mailing list