[XeTeX] Please help me with \XeTeXinterchartoks

[David M. Jones]

> Date: Tue, 2 Dec 2008 13:08:06 +1100
> From: "VAFA KHALIGHI" <vafa.khalighi at students.mq.edu.au>
> 
> Dear Ross and Ulrike, thanks for your wonderful and helpful answers. it
> seems that I need to wait to see what Jonathan says.* **
> 
> *I just want to know if what I want to achive is possible or not? so yes or
> no?
> 
> I will try to give a minimal example from now on, but thanks for letting me
> know that.*
> *

Unfortunately, I don't have a full answer either.  I played around
with something similar to this once and came to the conclusion that
there is an interesting interaction between three different
mechanisms, which I present here in hopes that someone will set me
straight if I got it wrong:

1) Inherent directionality of the Unicode text block.  E.g., if XeTeX
   encounters Arabic or Hebrew characters, it knows they should switch
   to RL mode for those characters.  If I understand correctly, that
   only affects the characters, not the paragraph layout.  So, for
   example, even if a paragraph consists entirely of RL characters,
   the parindent will still be on the right.

2) If you want to override the Unicode directionality, you can use
   these fun Unicode characters:

       \chardef\popDirection"202C
       \chardef\forceLtoR"202D
       \chardef\forceRtoL"202E

   These only affect the order of glyphs, not paragraph-level things.

Neither of those require \TeXXeTstate=1.  That's only required to
enable the third mechanism, which was inherited from e-TeX and
ultimately from TeX-XeT:

3) \beginR, \endR, \beginL, and \endR.   In eTeX, these control both
   the glyph order and the paragraph-level things.  E.g.,

       abc \beginR abc\endR\ abc

   will produce

       abc cba abc

   with the parindent on the left, because TeX is back in LR mode when
   the paragraph ends, but

   \beginR abc

   will produce

                                                               cba

   with the parindent on the right because TeX is in RL mode when the
   paragraph ends.

The problems come in how features (1) and (3) interact in XeTeX.  When
typesetting a non-Unicode font, (1) and (2) are irrelevant and (3)
works just like in eTeX.  But when using a Unicode font, the glyph
order is controlled by (1) and (3) only affects the paragraph-level
things like the parindent.  If you want to override the Unicode
directionality for glyph order, you can use (2).

Here's an unsatisfactory example that shows the limits of what I was
able to accomplish before I lost interest.  It should be possible to
make some improvements, but I don't know if this approach could ever
be made truly robust.

Cheers,
David.