☰

标记列表：标记化逆向谜题：我解答对了吗？如何使用 \ShowTokens 或 \tl_analysis_show:n 来证明我的解答？

tag-icon • tag-icon tex-core latex3
标记列表：标记化逆向谜题：我解答对了吗？如何使用 \ShowTokens 或 \tl_analysis_show:n 来证明我的解答？

我发现了这篇关于代币的有趣文章：

LuaTeX 中的标记来自汉斯·哈根。

宏定义转换为标记如下
[乙][答]

不幸的是，这里缺少宏定义。（pdf 第 3 页（文档第 7 页）右下角）

给定令牌列表：

cmd           chr          id  namedef             
def             0     1114818  def
undefined_cs          1115536  Test  
mac_param      35    
other_char     49  1  
mac_param      35
other_char     50  2
left_brace    123
other_char     91  [
mac_param      35
other_char     50  2
other_char     93  ]
other_char     91  [
mac_param      35
other_char     49  1
other_char     93  ]
right_brace   125
spacer         32
undefined_cs          1115536   Test
left_brace    123
letter         65  A
right_brace   125
left_brace    123
letter         66  B
right_brace   125

   

这是正确的宏定义吗？

\def\Test #1#2{[#2][#1]} \Test {A}{B}

是不是相当于这个呢？

\def\Test#1#2{[#2][#1]} \Test{A}{B}

我怎样才能证明这两个说法？

✔ ted 包：\ShowTokens

✔ expl3 包：\tl_analysis_show:n

✔\ifx比较两个令牌列表

❓ Joseph Wright “有一些很好的答案可以展示代币细分“

所有方法都提供了证明，但\ShowTokens对于初学者来说似乎是最安全的。

答案1

这是\tl_analysis_show:n证明（在加载了包latex的会话中expl3）：

*\catcode`_=11

*\catcode`:=11

*\tl_analysis_show:n {\def\Test #1#2{[#2][#1]} \Test {A}{B}}
The token list contains the tokens:
>  \def (control sequence=\def)
>  \Test (control sequence=undefined)
>  # (macro parameter character #)
>  1 (the character 1)
>  # (macro parameter character #)
>  2 (the character 2)
>  { (begin-group character {)
>  [ (the character [)
>  # (macro parameter character #)
>  2 (the character 2)
>  ] (the character ])
>  [ (the character [)
>  # (macro parameter character #)
>  1 (the character 1)
>  ] (the character ])
>  } (end-group character })
>    (blank space  )
>  \Test (control sequence=undefined)
>  { (begin-group character {)
>  A (the letter A)
>  } (end-group character })
>  { (begin-group character {)
>  B (the letter B)
>  } (end-group character }).
<recently read> }

<*> ...w:n {\def\Test #1#2{[#2][#1]} \Test {A}{B}}

?

*\tl_analysis_show:n {\def\Test#1#2{[#2][#1]} \Test{A}{B}}
The token list contains the tokens:
>  \def (control sequence=\def)
>  \Test (control sequence=undefined)
>  # (macro parameter character #)
>  1 (the character 1)
>  # (macro parameter character #)
>  2 (the character 2)
>  { (begin-group character {)
>  [ (the character [)
>  # (macro parameter character #)
>  2 (the character 2)
>  ] (the character ])
>  [ (the character [)
>  # (macro parameter character #)
>  1 (the character 1)
>  ] (the character ])
>  } (end-group character })
>    (blank space  )
>  \Test (control sequence=undefined)
>  { (begin-group character {)
>  A (the letter A)
>  } (end-group character })
>  { (begin-group character {)
>  B (the letter B)
>  } (end-group character }).
<recently read> }

<*> ...how:n {\def\Test#1#2{[#2][#1]} \Test{A}{B}}

两次调用都返回相同的输出。

答案2

这两个：

\def\Test #1#2{[#2][#1]} \Test {A}{B}
\def\Test#1#2{[#2][#1]} \Test{A}{B}

表示给定的标记列表，但下一个不表示：

c:\>tex & dviout texput.dvi
This is TeX, Version 3.14159265 (TeX Live 2019/W32TeX) (preloaded format=tex)
**\def\Test#1#2{[#2][#1]}\Test{A}       {B}

*\bye

即使视觉结果看起来正确：

在此处输入图片描述

感谢 Marcel Krüger 的提示，这是使用 ted 包生成的令牌列表：

c:\>latex
This is pdfTeX, Version 3.14159265-2.6-1.40.20 (TeX Live 2019/W32TeX) (preloaded format=latex)
 restricted \write18 enabled.
**\RequirePackage{ted}
entering extended mode
LaTeX2e <2018-12-01>

*\ShowTokens*{\def\Test#1#2{[#2][#1]}\Test{A}       {B}}
(c:/texlive/2019/texmf-dist/tex/latex/ted/ted.sty)
--- Begin token decomposition of:
    \def \Test ##1##2{[##2][##1]}\Test {A} {B}
\def (control sequence=\def)
\Test (control sequence=undefined)
# (macro parameter character #)
1 (the character 1)
# (macro parameter character #)
2 (the character 2)
{ (begin-group character {)
[ (the character [)
# (macro parameter character #)
2 (the character 2)
] (the character ])
[ (the character [)
# (macro parameter character #)
1 (the character 1)
] (the character ])
} (end-group character })            <-----  missing space 
\Test (control sequence=undefined)   <-----  missing space
{ (begin-group character {)
A (the letter A)
} (end-group character })
  (blank space  )                    <-----  resulting from extra spaces               
{ (begin-group character {)
B (the letter B)
} (end-group character })
--- End token decomposition.

宏观\ShowTokens证实了 Marcel Krüger 的说法。

答案3

有一些很好的答案可以显示 token 分解，但您可以使用快速回答“它们是否相同”的问题\ifx。我们只需要将 token 列表存储在两个宏中。为了避免担心#重复，我\unexpanded在内部使用\edef

\edef\tempa{\unexpanded{\def\Test #1#2{[#2][#1]} \Test {A}{B}}}
\edef\tempb{\unexpanded{\def\Test#1#2{[#2][#1]} \Test {A}{B}}}
\ifx\tempa\tempb\TRUE\else\FALSE\fi
\bye

或者不使用 e-TeX

\toks0={\def\Test #1#2{[#2][#1]} \Test {A}{B}}
\edef\tempa{\the\toks0}
\toks0={\def\Test#1#2{[#2][#1]} \Test {A}{B}}
\edef\tempb{\the\toks0}
\ifx\tempa\tempb\TRUE\else\FALSE\fi
\bye

上述两点均表明\TRUE：两个宏是相等的，因为两个标记列表是相等的。

相关内容