Raku Core Hacking: QASTalicious

Overview of Q Abstract Syntax Trees + bug fix tutorial

在过去的一个月中,我在 Rakudo 的 QAST 地区花了一些时间写了一些优化,修复了包含警告的错误,并且用一个单一的提交压缩了一个10个thunk范围的bug的怪物蜂巢。 在今天的文章中,我们将详细介绍最后一个专长,以及了解QAST是什么以及如何使用它。

第一部分: QAST

“QAST” 代表 “Q” Abstract Syntax Tree.(“Q” 抽象语法树.) 为什么会有个字母 “Q” 在那里呢, 因为 Q 是 P 的下一个字母, 而 “P” 过去是在 “PAST” 里面的, 代表 “Parrot”(鹦鹉), 是很早之前的一个实验性的 Raku 实现(或者说, 它的虚拟机). 我们来看看什么是 QAST!

Dumping QAST

每个 Rakudo Raku 程序都编译到 QAST 节点树上,如果在编译程序或模块时给 raku 指定 --target=ast--target=optimize 命令行选项,则可以转储该树:

$ raku --target=ast -e 'say "Hello, World!"'
[...]
- QAST::Op(call &say) <sunk> :statement_id<?> say \"Hello, World!\"
  - QAST::Want <wanted> Hello, World!
    - QAST::WVal(Str)
    - Ss
    - QAST::SVal(Hello, World!)
[...]

--target=ast--target=optimize 之间的区别在于,前者在生成后立即显示 QAST 树,而后者在静态优化器执行后显示 QAST 树。

虽然命令行选项为整个程序提供了 QAST(不包括单独预编译的模块),但是每个 QAST::Node 对象都有一个 .dump 方法,可用于从 Rakudo 的源代码中转储特定的 QAST 片段。

例如,为了检查由 statement 标记生成的 QAST,我会在 src/Raku/Actions.nqp 中找到method 语句,并且在 nqp::say('statement QAST: ' ~ $past.dump) 方法。

由于 Rakudo 的编译需要花费几分钟的时间,所以我喜欢在 env 变量上键入我的调试转储,如下所示:

nqp::atkey(nqp::getenvhash(),'ZZ1') && nqp::say('ZZ1: something or other');
...
nqp::atkey(nqp::getenvhash(),'ZZ2') && nqp::say('ZZ2: something else');

然后,我可以执行已编译的 ./raku,就像我没有添加任何内容一样,通过运行 ZZ1=1 ./raku, ZZ2=1 ./raku 或者同时使用 ZZ1=1 ZZ2=1 ./raku

查看 QAST

在终端中查看 --target 转储的输出就足以快速浏览树,但是为了获得额外的帮助,您可以安装带有 q 命令行工具的 CoreHackers::Q 模块。

只需在常规的 raku 调用前面加上 q aq o 前缀以分别生成 --target=ast--target=optimize QAST 转储。该程序将在当前目录中生成 out.html 文件:

$ q a raku -e 'say "Hello, World!"'
$ firefox out.html

弹出打开生成的HTML文件,并获得这些好处:

  • 颜色编码的 QAST 节点
  • 沉没节点的颜色提示
  • Ctrl + 点击任何节点来折叠它
  • 静音的 QAST::Want 替代品,更容易忽略它们

最后,我希望扩展这个工具,使其更有帮助,但在撰写本文时,就是这样。

QAST 森林

rakudo 的源代码 中有四个主要的文件,你可以期待它们使用 QAST 节点:src/Raku/Grammar.nqp, src/Raku/Actions.nqp, src/Raku/World.nqp, 和 src/Raku/Optimizer.nqp。如果您正在使用 Z-Script 工具,甚至可以运行 z q 命令在 Atom 编辑器中打开这四个文件。

Grammar.nqp 是 Raku 语法。 Actions.nqp 是它的动作。 World.nqp 包含 Grammar.nqpActions.nqp 使用的各种有用的例程,它们通过包含 Raku::World 对象的 $*W 动态变量来访问它们。最后,Optimizer.nqp 包含 Rakudo 的静态优化器。

(所有邪恶的)根是 QAST::Node对象,所有其他 QAST 节点是它的子类。让我们来回顾一些流行的:

QAST::Op

QAST::Op 节点是 QAST 世界的主力。:op 命名参数指定 NQP op 的名称或 Rakudo的 NQP扩展操作的名称,其孩子是参数:

下面是 say op 打印一个字符串值:

QAST::Op.new: :op<say>,
  QAST::SVal.new: :value('Hello, World!');

这里是 call op 调用 Raku 的 infix:<+> 操作符的 QAST 节点, 请注意我们调用的例程的名称是如何通过 :name 命名的参数给出的:

QAST::Op.new: :op<call>, :name('&infix:<+>'),
  QAST::IVal.new( :value(2)),
  QAST::IVal.new: :value(2)

QAST::*Val

QAST::SValQAST::IValQAST::NValQAST::WVal 节点分别指定字符串,整数,浮点数和 “World” 对象值。前三个是"未装箱"的原始值,而 World 对象是其他的东西,如 DateTimeBlockStr 对象。

QAST::WANT

一些对象可以用多个 QAST::*Val 节点来表示,其中根据当前上下文中所需的内容使用最合适的值。 QAST::Want 节点包含这些替代方案,与字符串标记交错,指示替代方案是什么。

例如,Raku 中的数值 42 可能需要作为调用某个方法的对象,或者作为要分配给本地 int 变量的原始值。 QAST::Want 节点看起来像这样:

QAST::Want.new:
  QAST::WVal.new(:value($Int-obj))),
  'Ii',
  QAST::IVal.new: :value(42)

上面的 $Int-obj 将包含一个 Int 类型的实例,其值被设置为 42。Ii 标记指示下面的替代是一个整数值,我们提供一个包含它的 QAST::IVal 对象。其他可能的标记是 Nn(float),Ss(string)和 v(void context)。

当这些节点稍后转换为字节码时,将选择最合适的值,第一个子元素为"默认"值,当没有可用的替代方法进行剪切时,将使用该值。

QAST::Var

这些节点用于变量和参数。 :name 具名参数指定变量的名称, :scope 具名参数指定它的作用域:

QAST::Op.new: :op('bind'),
  QAST::Var.new(:name<$x>, :scope<lexical>, :decl<var>, :returns(int)),
  QAST::IVal.new: :value(0)

当节点用于变量的声明(当它不存在时,我们简单地引用该变量)时,会出现 :decl 具名参数,它的值决定了变量的类型: var 表示变量,param 表示例程参数。其他几个 :decl 类型,以及指定变量的额外配置的可选参数存在。你可以在 QAST 文档中找到它们。

QAST::Stmt / QAST::Stmts

这些是语句分组结构。例如,在这里,nqp::if 的 truthy 分支包含三个 nqp::say 语句,所有这些语句都被分组在 QAST::Stmts 中:

QAST::Op.new: :op<if>,
  QAST::IVal.new(:value(42)),
  QAST::Stmts.new(
    QAST::Op.new( :op<say>, QAST::SVal.new: :value<foo>),
    QAST::Op.new( :op<say>, QAST::SVal.new: :value<bar>),
    QAST::Op.new: :op<say>, QAST::SVal.new: :value<ber>),
  QAST::Op.new: :op<say>, QAST::SVal.new: :value<meow>,

单数 QAST::Stmt 是相似的。不同的是它标志着一个寄存器的分配界限,超出这个界限,任何临时对象都可以自由地被重用。如果使用正确,则此替代方法可以导致更好的代码生成。

QAST::Block

这个节点既是一个调用单元,也是一个词法作用域单元。例如,代码 sub foo { say "hello" } 可能编译成 QAST::Block, 像这样:

Block (:cuid(1)) <wanted> :IN_DECL<sub> { say \"hello\" }
[...]
  Stmts <wanted> say \"hello\"
    Stmt <wanted final> say \"hello\"
      Want <wanted>
        Op (call &say) <wanted> :statement_id<?> say \"hello\"
          Want <wanted> hello
            WVal (Str)
            - Ss
            SVal (hello)
        - v
        Op (p6sink)
          Op (call &say) <wanted> :statement_id<?> say \"hello\"
            Want <wanted> hello
              WVal (Str)
              - Ss
              SVal (hello)
[...]

每个块都划分了一个词法作用域边界 - 这个细节在本文的第二部分中有介绍,当我们将要修复一个 bug 的时候。

其它

还有几个 QAST 节点存在。他们超出了本文的范围,但是您可能希望阅读文档,或者由于其中一些文档没有出现在这些文档中,请直接找到源代码

执行 QAST 树

在与 QAST 合作时,与nqp操作(以及 Rakudo 的 nqp 扩展)相当相似。在QAST转储中,一个敏锐的眼睛会注意到许多 QAST::Op 节点对应于 nqp::*op 调用,其中 :op 命名参数指定操作的名称。

在编写大型 QAST 树时,首先使用纯 NQP 操作将其写下来,然后将结果转换为 QAST 节点对象树。我们来看一个简单的例子:

nqp::if(
  nqp::isgt_n(nqp::rand_n(1e0), .5e0),
  nqp::say('Glass half full'),
  nqp::say('Glass half empty'));

我们有 NQP op,所以我们从 QAST::Op 节点开始,使用 'if' 作为 :op 的值。该 op 需要三个位置参数 - 三个 ops 用于有条件的,truthy 分支和 falsy 分支。有些操作符也会使用 float 和 string 的值,所以我们将使用 QAST::NValQAST::SVal 节点。结果是:

QAST::Op.new(:op('if'),
  QAST::Op.new(:op('isgt_n'),
    QAST::Op.new(:op('rand_n'),
      QAST::NVal.new(:value(1e0))
    ),
    QAST::NVal.new(:value(.5e0))
  ),
  QAST::Op.new(:op('say'),
    QAST::SVal.new(:value('Glass half full'))
  ),
  QAST::Op.new(:op('say'),
    QAST::SVal.new(:value('Glass half empty'))
  )
)

我发现只有在必要时才使用括号来跟踪树的嵌套更容易,只要有可能就更喜欢使用冒号方法调用语法:

QAST::Op.new: :op<if>,
  QAST::Op.new(:op<isgt_n>,
    QAST::Op.new(:op<rand_n>,
      QAST::NVal.new: :value(1e0)),
    QAST::NVal.new: :value(.5e0)),
  QAST::Op.new(:op<say>,
    QAST::SVal.new: :value('Glass half full')),
  QAST::Op.new: :op<say>,
    QAST::SVal.new: :value('Glass half empty')

如果.new之后是冒号,则同一级别上不会有更多的节点。如果.new之后是开头括号,那么还有更多的姐妹节点还未到来。

由于Rakudo冗长的编译,可以方便地执行你的QAST树,而不必先将它粘贴到src / Raku / Actions.nqp或类似的文件中。在某种程度上,用普通的Raku程序可以做到这一点。我们只需要在BEGIN块内的Perl * :: W变量中存取Raku :: World对象,然后调用.compile_time_evaluate方法,给它一个空变量作为第一个位置(它需要树的Match对象)和我们的QAST树作为第二个位置:

use QAST:from<NQP>;
BEGIN $*W.compile_time_evaluate: $,
    QAST::Op.new: :op<if>,
      QAST::Op.new(:op<isgt_n>,
        QAST::Op.new(:op<rand_n>,
          QAST::NVal.new: :value(1e0)),
        QAST::NVal.new: :value(.5e0)),
      QAST::Op.new(:op<say>,
        QAST::SVal.new: :value('Glass half full')),
      QAST::Op.new: :op<say>,
        QAST::SVal.new: :value('Glass half empty')

这种方法的一个警告是我们使用成熟的Raku语言,而在src / Raku / Actions.nqp和相关文件中,正如.nqp扩展所示,我们只使用NQP语言。留意怪异的爆炸;有可能你的QAST树会在Raku中爆炸,在纯NQP的领域将会很好的编译。

注解 QAST 节点

所有QAST节点都支持注释,允许您将任意值附加到节点,然后在别处读取该值。要添加注释,请使用.annotate方法,该方法接受两个位置参数 - 一个包含注释名称和附加值的字符串 - 并返回该值。最近版本的NQP也有.annotate_self方法,除了返回QAST节点本身外,

$qast.annotate_self('foo', 42).annotate: 'bar', 'meow';

稍后,您可以使用.ann方法读取该值,该方法将注释的名称作为参数。如果注释不存在,则返回NQPMu:

note($qast.ann: 'foo'); # OUTPUT: «42␤»

您还可以使用返回1(true)或0(false)的.has_ann方法来检查注释是否仅存在:

note($qast.has_ann: 'bar'); # OUTPUT: «1␤»

或者转储节点上的所有注释(为了防止潜在的输出溢出,大多数值将被作为简单的问号转储):

note($qast.dump_annotations); # OUTPUT: « :bar<?> :foo<?>␤»);

最后,要清除节点上的所有注释,只需调用.clear_annotations方法即可。

修改 QAST 节点

QAST节点对象的一个​​方便的事情是将它们变成更好的东西。这基本上是src / Raku / Optimizer.nqp中的所有静态优化器。命名参数可以通过调用方法和提供值来进行变异。例如,$ qast.op(‘callstatic’)将把op的值从无论什么地方改为callstatic。位置参数可以通过重新分配位置索引来进行更改,也可以通过带有这些名称的方法调用或nqp :: ops进行移位,推送,非移位,弹出操作。有些节点也支持对它们的nqp :: elems调用,它比+ @($ qast)的通用模式稍微快一些,可以在所有节点上使用它们来查找节点包含的子节点数量。

作为一个练习,让我们写一个小的优化:一些操作,比如$ foo <$ bar <$ ber编译为nqp :: chain操作。即使我们只有两个孩子,也是如此$ foo <$ bar。在这种情况下,将op重写为nqp :: call具有性能优势:不仅nqp :: call比nqp :: chain快一点,静态优化器知道如何进一步优化nqp ::调用ops。

让我们来看看两个孩子和两个孩子的nqp :: chain链是什么样的:

$ raku --target=ast -e '2 < 3 < 4; 2 < 3'

第一个声明编译到这(我删除QAST ::希望清晰):

- QAST::Op(chain &infix:«<»)  :statement_id<?> <
  - QAST::Op(chain &infix:«<») <wanted> <
    - QAST::IVal(2)
    - QAST::IVal(3)
  - QAST::IVal(4)

第二个编译为:

- QAST::Op(chain &infix:«<»)  :statement_id<?> <
  - QAST::IVal(2)
  - QAST::IVal(3)

因此,要正确定位我们的优化,我们需要确保我们的连锁店的孩子都不是连锁店。另外,我们需要确保我们正在优化的操作本身不是另一个连锁操作系统的孩子。

剔除优化器的代码,我们可以发现链深度已经通过$!chain_depth属性进行了跟踪,所以我们只需要确保我们处于链的第一个链接。代码然后变成:

$qast.op: 'call'
  if nqp::istype($qast, QAST::Op)
  && $qast.op eq 'chain'
  && $!chain_depth == 1
  && ! (nqp::istype($qast[0], QAST::Op) && $qast[0].op eq 'chain')
  && ! (nqp::istype($qast[1], QAST::Op) && $qast[1].op eq 'chain');

一旦我们找到一个链QAST :: Op,我们将它编入索引并使用nqp :: istype来检查kid节点的类型,如果这些节点碰巧是QAST :: Op节点,我们确保:op参数不是一个链运。如果满足所有的条件,我们只需在节点上调用.op方法,将值“call”转换为一个调用操作。

然后,我们把优化的时间放到优化器的.visit_op方法中,并且稍后的部分将进一步优化我们的调用。

一个相当简单和直接的优化,可以带来很多好处。

PART II: A Thunk in The Trunk


Note: it took me three evenings to debug and fix the following tickets. To learn the solution I tried many dead ends that I won’t be covering, to keep you from getting bored, and instead will instantly jump to conclusions. The point I’m making is that fixing core bugs is a lot easier than may seem from reading this article—you just need to be willing to spend some time on them.

注意:我花了三个晚上来调试和修复以下 tickets。为了学习解决方案,我尝试了许多我不会覆盖的死角,以防止你感到厌倦,而是立即跳到结论。我所做的一点是修复核心bug比读这篇文章容易得多 - 你只需要愿意花一些时间在他们身上。


Now that we have some familiarity with QAST, let’s try to fix a bug that existed in Rakudo v2018.01.30.ga.5.c.2398.cc and earlier. The ticket in question is R#1212, that shows the following problem:

$ raku -e 'say <a b c>[$_ xx 2] with 1'

Use of Nil in string context
  in block  at -e line 1
Unable to call postcircumfix [ (Any) ] with a type object
Indexing requires a defined object
  in block <unit> at -e line 1

It looks like the $_ topical variable inside the indexing brackets fails to get the value from with statement modifier and ends up being undefined. Sounds like a challenge! 它看起来像索引括号内的 $_ 主题变量无法从 with 语句修饰符中获取值,并且最终未定义。 听起来像是一个挑战!

It’s A Hive!

Both with and xx operator create thunks (thunks are like blocks of code, without having explicit blocks in the code; this, for example, lets rand xx 10 to produce 10 different random values; rand is thunked and the thunk is called for each iteration). This reminded me of some other tickets I’ve seen, so I went to fail.rakudo.party and looked through open tickets for anything that mentioned thunking or wrong scoping.

``和’xx运算符都创建thunks(thunks就像代码块一样,代码中没有明确的代码块;例如,这可以让rand xx 10产生10个不同的随机值;rand`被thunked并且每次迭代调用thunk)。这让我想起了我见过的其他一些门票,所以我去了[fail.rakudo.party](https://fail.rakudo.party/),并通过开放票查看任何提到thunking或错误范围界定的东西。

I ended up with a list of 7 tickets, and with the help of dogbert++ later increased the number to 9, which with the original Issue gives us a total of 10 different manifestations of a bug. The other tickets are

最后我列出了7张票,在dogbert ++的帮助下,后来增加了9个,与原来的Issue一起给了我们总共10个不同的错误表现。其他门票是

RT#130575, RT#132337, RT#131548, RT#132211, RT#126569, RT#128054, RT#126413, RT#126984, and RT#132172. Quite a bug hive!

Test It Out

Our starting point is to cover each manifestation of the bug with a test. Make all the test pass and you know you’ve fixed the bug, plus you already have something to place into roast, to cover the tickets. My tests ended up looking like this, where I’ve used gather/take duo to capture what the tickets' code printed to the screen: 我们的出发点是用一个测试来覆盖每个bug的表现。 让所有的测试通过,你知道你已经修复了错误,再加上你已经有一些东西要放在[烤](https://github.com/raku/roast/),以涵盖门票。我的测试看起来像这样,我已经使用了[collect /take](https://docs.raku.org/syntax/gather%20take)二重奏来捕获票据打印到屏幕:

use Test;
plan 1;
subtest 'thunking closure scoping' => {
    plan 10;

    # https://github.com/rakudo/rakudo/issues/1212
    is-deeply <a b c>[$_ xx 2], <b b>.Seq, 'xx inside `with`' with 1;

    # RT #130575
    is-deeply gather {
        sub itcavuc ($c) { try {take $c} andthen 42 };
        itcavuc $_ for 2, 4, 6;
    }, (2, 4, 6).Seq, 'try with block and andthen';

    # RT #132337
    is-deeply gather {
        sub foo ($str) { { take $str }() orelse Nil }
        foo "cc"; foo "dd";
    }, <cc dd>.Seq, 'block in a sub with orelse';

    # RT #131548
    is-deeply gather for ^7 {
        my $x = 1;
        1 andthen $x.take andthen $x = 2 andthen $x = 3 andthen $x = 4;
    }, 1 xx 7, 'loop + lexical variable plus chain of andthens';

    # RT #132211
    is-deeply gather for <a b c> { $^v.uc andthen $v.take orelse .say },
        <a b c>.Seq, 'loop + andthen + orelse';

    # RT #126569
    is-deeply gather { (.take xx 10) given 42 }, 42 xx 10,
        'parentheses + xx + given';

    # RT #128054
    is-deeply gather { take ("{$_}") for <aa bb> }, <aa bb>.Seq,
        'postfix for + take + block in a string';

    # RT #126413
    is-deeply gather { take (* + $_)(32) given 10 }, 42.Seq,
        'given + whatever code closure execution';

    # RT #126984
    is-deeply gather {
        sub foo($x) { (* ~ $x)($_).take given $x }; foo(1); foo(2)
    }, ("11", "22").Seq, 'sub + given + whatevercode closure execution';

    # RT #132172
    is-deeply gather { sub {
        my $ver =.lines.uc with "totally-not-there".IO.open
            orelse "meow {$_ ~~ Failure}".take and return 42;
    }() }, 'meow True'.Seq, 'sub with `with` + orelse + block interpolation';
}

When I brought up the first bug in our dev chatroom, jnthn++ pointed out that such bugs are often due to mis-scoped blocks, as p6capturelex op that’s involved needs to be called in the immediate outer of the block it references.

Looking through the tickets, I also spotted skids++’s note that changing a conditional for statement_id in block migrator predicate fixed one of the tickets. This wasn’t the full story of the fix, as the many still-failing tests showed, but it was a good start. 当我在[我们的开发聊天室](https://webchat.freenode.net/?channels=#raku-dev)提出第一个错误时,jnthn ++指出这样的错误通常是由于错误的块,涉及的p6capturelex`需要在它引用的块的外部被调用。

通过查看票据,我还发现滑块++的注意到,在[block migrator predicate]中更改statement_id的条件(https://github.com/rakudo/rakudo/raklob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp#L9195 -L9197)修正了一张门票。 这并不是解决问题的完整故事,因为许多仍然失败的测试显示,但这是一个好的开始。

What’s Your Problem?

In order to find the best solution for a bug, it’s important to understand what exactly is the problem. We know mis-scoped blocks are the cause of the bug, so lets grab each of our tests, dump their QAST (--target=ast), and write out how mis-scoped the blocks are.

To make it easier to match the QAST::Blocks with the QAST::WVals referencing them, I made a modification to QAST::Node.dump to include CUID numbers and statement_id annotations in the dumps.

Going through mosts of the buggy code chunks, we have these results: 为了找到一个错误的最佳解决方案,重要的是要明白究竟是什么问题。我们知道错误的块是错误的原因,所以让我们抓住我们的每个测试,转储他们的QAST(--target = ast),并且写出*块的错误范围。

为了使QAST :: Block与QAST :: WVal引用它们更加容易,我[做了修改](https://github.com/raku/nqp/commit/0264b237930f426f4cba744c55f10813869ac40b) QAST :: Node.dump在转储中包含CUID号码和statement_id注释。

通过大部分的错误代码块,我们有这些结果:

is-deeply <a b c>[$_ xx 2], <b b>.Seq, 'xx inside `with`' with 1;
# QAST for `xx` is ALONGSIDE RHS `andthen` thunk, but needs to be INSIDE

is-deeply gather {
    sub itcavuc ($c) { try {take $c} andthen 42 };
    itcavuc $_ for 2, 4, 6;
}, (2, 4, 6).Seq, 'try with block and andthen';
# QAST for try block is INSIDE RHS `andthen` thunk, but needs to be ALONGSIDE

is-deeply gather {
    sub foo ($str) { { take $str }() orelse Nil }
    foo "cc"; foo "dd";
}, <cc dd>.Seq, 'block in a sub with orelse';
# QAST for block is INSIDE RHS `andthen` thunk, but needs to be ALONGSIDE

is-deeply gather for ^7 {
    my $x = 1;
    1 andthen $x.take andthen $x = 2 andthen $x = 3 andthen $x = 4;
}, 1 xx 7, 'loop + lexical variable plus chain of andthens';
# each andthen thunk is nested inside the previous one, but all need to be
# ALONGSIDE each other

is-deeply gather for <a b c> { $^v.uc andthen $v.take orelse .say },
    <a b c>.Seq, 'loop + andthen + orelse';
# andthen's block is INSIDE orelse's but needs to be ALONGSIDE each other

is-deeply gather { (.take xx 10) given 42 }, 42 xx 10,
    'parentheses + xx + given';
# .take thunk is ALONGSIDE given's thunk, but needs to be INSIDE of it

is-deeply gather { take ("{$_}") for <aa bb> }, <aa bb>.Seq,
    'postfix for + take + block in a string';
# the $_ is ALONGSIDE `for`'s thunk, but needs to be INSIDE

is-deeply gather { take (* + $_)(32) given 10 }, 42.Seq,
    'given + whatever code closure execution';
# the WhateverCode ain't got no statement_id and is ALONGSIDE given
# block but needs to be INSIDE of it

So far, we can see a couple of patterns: 到目前为止,我们可以看到一些模式:

  • xx and WhateverCode thunks don’t get migrated, even though they should
  • andthen thunks get migrated, even though they shouldn’t

The first one is fairly straightforward. Looking at the QAST dump, we see xx thunk has a higher statement_id than the block it was meant to be in. This is what skids++’s hint addresses, so we’ll change the statement_id conditional from == to >= to look for statement IDs higher than our current one as well, since those would be from any substatements, such as our xx inside the positional indexing operator: 第一个很简单。查看QAST转储,我们看到xx thunk的’statement_id比它原本想要的块要高。 这是什么滑动++的提示地址,所以我们会改变[statement_id条件](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp#L9196)从== ````=``查找比我们当前的语句ID更高的语句ID,因为这些语句可能来自任何子语句,比如位置索引操作符中的xx`:

($b.ann('statement_id') // -1) >= $migrate_stmt_id

The cause is very similar for the WhateverCode case, as it’s missing statement_id annotation altogether, so we’ll just annotate the generated QAST::Block with the statement ID. Some basic detective work gives us the location where that node is created: we search src/Raku/Actions.nqp for word "whatever" until we spot whatever_curry method and in its guts we find the QAST::Block we want. For the statement ID, we’ll grep the source for statement_id: 原因与WhateverCode情况非常相似,因为它完全缺少了statement_id注释,所以我们只用语句ID注释生成的QAST :: Block。一些基本的侦探工作给了我们[创建该节点的位置](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp#L9574):我们搜索src / Raku / Actions.nqp用于单词“任何”,直到我们找到whatever_curry方法,并且在它的内核中我们找到我们想要的QAST :: Block`。对于语句ID,我们将grep“statement_id”的源代码:

$ grep -FIRn 'statement_id' src/Raku/
src/Raku/Actions.nqp:1497:            $past.annotate('statement_id', $id);
src/Raku/Actions.nqp:2326:                $_.annotate('statement_id', $*STATEMENT_ID);
src/Raku/Actions.nqp:2488:                -> $b { ($b.ann('statement_id') // -1) == $stmt.ann('statement_id') });
src/Raku/Actions.nqp:9235:                && ($b.ann('statement_id') // -1) >= $migrate_stmt_id
src/Raku/Actions.nqp:9616:            ).annotate_self: 'statement_id', $*STATEMENT_ID;
src/Raku/World.nqp:256:            $pad.annotate('statement_id', $*STATEMENT_ID);

From the output, we can see the ID is stored in $*STATEMENT_ID dynamic variable, so we’ll use that for our annotation on the WhateverCode’s QAST::Block: 从输出中,我们可以看到ID存储在$ * STATEMENT_ID动态变量中,所以我们将使用它作为我们的注释[在 WhateverCode`` QAST :: Block](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp#L9574):

my $block := QAST::Block.new(
    QAST::Stmts.new(), $past
).annotate_self: 'statement_id', $*STATEMENT_ID;

Let’s compile and run our bug tests. If you’re using Z-Script, you can re-compile Rakudo by running z command with no arguments: 让我们编译并运行我们的错误测试。如果你正在使用[Z-Script](https://github.com/zoffixznet/z),你可以通过运行没有参数的“z”命令来重新编译Rakudo:

$ z
[...]
$ ./raku bug-tests.t
1..1
    1..10
    ok 1 - xx inside `with`
    not ok 2 - try with block and andthen
    # Failed test 'try with block and andthen'
    # at bug-tests.t line 10
    # expected: $(2, 4, 6)
    #      got: $(2, 2, 4)
    not ok 3 - block in a sub with orelse
    # Failed test 'block in a sub with orelse'
    # at bug-tests.t line 16
    # expected: $("cc", "dd")
    #      got: $("cc", "cc")
    not ok 4 - loop + lexical variable plus chain of andthens
    # Failed test 'loop + lexical variable plus chain of andthens'
    # at bug-tests.t line 22
    # expected: $(1, 1, 1, 1, 1, 1, 1)
    #      got: $(1, 4, 3, 3, 3, 3, 3)
    not ok 5 - loop + andthen + orelse
    # Failed test 'loop + andthen + orelse'
    # at bug-tests.t line 28
    # expected: $("a", "b", "c")
    #      got: $("a", "a", "a")
    ok 6 - parentheses + xx + given
    ok 7 - postfix for + take + block in a string
    ok 8 - given + whatever code closure execution
    ok 9 - sub + given + whatevercode closure execution
    not ok 10 - sub with `with` + orelse + block interpolation
    # Failed test 'sub with `with` + orelse + block interpolation'
    # at bug-tests.t line 49
    # expected: $("meow True",)
    #      got: $("meow False",)
    # Looks like you failed 5 tests of 10
not ok 1 - thunking closure scoping
# Failed test 'thunking closure scoping'
# at bug-tests.t line 3
# Looks like you failed 1 test of 1

Looks like that fixed half of the issues already. That’s pretty good! 看起来已经是固定的一半了。这很好!

Extra Debugging 额外的调试

Let’s now look at the remaining failures and figure out why block migration isn’t how we want it in those cases. To assists with our sleuthing efforts,let’s make a couple of changes to produce more debugging info.

First, let’s modify QAST::Node.dump method in NQP’s repo to dump the value of in_stmt_mod annotation, by telling it to dump out the value verbatim if the key is in_stmt_mod: 现在我们来看看其余的失败,并找出为什么在这种情况下块迁移不是我们想要的。为了协助我们的侦察工作,让我们做一些改变来产生更多的调试信息。

首先,我们来修改[NQP的repo中的QAST :: Node.dump方法](https://github.com/raku/nqp/blob/d71bd7334c5c9363d49ddf20645e6041af15fa41/src/QAST/Node.nqp#L166) in_stmt_mod注释,通过告诉它如果键是in_stmt_mod就逐字转储出来的值:

if $k eq 'IN_DECL' || $k eq 'BY' || $k eq 'statement_id'
|| $k eq 'in_stmt_mod' {
    ...

Next, let’s go to sub migrate_blocks in Actions.nqp and add a bunch of debug dumps inside most of the conditionals. This will let us track when a block is compared and to see whether migration occurs. As mentioned earlier, I like to key my dumps on env vars using nqp::getenvhash op, so after modifications my migrate_blocks routine looks like this; note the use of .dump method to dump QAST node guts (tip: .dump method also exists on Raku::Grammar’s match objects!): 接下来,我们来到[Actions.nqp中的migrate_blocks](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp#L6535-L6560),并添加一堆在大多数条件下调试转储。 这将让我们跟踪块的比较时间,并查看是否发生迁移。正如前面所提到的,我喜欢用nqp :: getenvhash运算符在env vars上输入密码,所以在修改后我的migrate_blocks例程看起来像这样;请注意使用.dump方法来转储QAST节点的内容(提示:.dump`方法也存在于’Raku :: Grammar’的匹配对象中!

sub migrate_blocks($from, $to, $predicate?) {
    my @decls := @($from[0]);
    my int $n := nqp::elems(@decls);
    my int $i := 0;
    while $i < $n {
        my $decl := @decls[$i];
        if nqp::istype($decl, QAST::Block) {
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: -----------------');
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: trying to grab ' ~ $decl.dump);
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: to move to ' ~ $to.dump);
            if !$predicate || $predicate($decl) {
                nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: grabbed');
                $to[0].push($decl);
                @decls[$i] := QAST::Op.new( :op('null') );
            }
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: -----------------');
        }
        elsif (nqp::istype($decl, QAST::Stmt) || nqp::istype($decl, QAST::Stmts)) &&
              nqp::istype($decl[0], QAST::Block) {
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: -----------------');
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: trying to grab ' ~ $decl[0].dump);
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: to move to ' ~ $to.dump);
            if !$predicate || $predicate($decl[0]) {
                nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: grabbed');
                $to[0].push($decl[0]);
                $decl[0] := QAST::Op.new( :op('null') );
            }
            nqp::atkey(nqp::getenvhash(),'ZZ') && nqp::say('ZZ1: -----------------');
        }
        elsif nqp::istype($decl, QAST::Var) && $predicate && $predicate($decl) {
            $to[0].push($decl);
            @decls[$i] := QAST::Op.new( :op('null') );
        }
        $i++;
    }
}

After making the changes, we need to recompile both NQP and Rakudo. With Z-Script, we can just run z n to do that: 进行更改后,我们需要重新编译NQP和Rakudo。使用[Z-Script](https://github.com/zoffixznet/z),我们可以运行z n来做到这一点:

$ z n
[...]

Now, we’ll grab the first failing code and take a look at its QAST. I’m going to use the CoreHackers::Q tool: 现在,我们将抓住第一个失败的代码,并看看它的QAST。我打算使用[CoreHackers :: Q工具](https://modules.raku.org/dist/CoreHackers::Q):

$ q a ./raku -e '
    sub itcavuc ($c) { try {say $c} andthen 42 };
    itcavuc $_ for 2, 4, 6;'
$ firefox out.html

We can see that our buggy say call lives in QAST::Block with cuid 1, which gets called from within QAST::Block with cuid 3, but is actually located within QAST::Block with cuid 2: 我们可以看到,我们的say调用运行在具有cuid 1QAST :: Block中,该函数从QAST :: Block内用cuid 3调用,但实际上位于QAST ::块cuid 2

- QAST::Block(:cuid(3)) <wanted> :statement_id<1>
        :count<?> :signatured<?> :IN_DECL<sub>
        :in_stmt_mod<0> :code_object<?>
        :outer<?> { try {say $c} andthen 42 }
    [...]
        - QAST::Block(:cuid(2)) <wanted> :statement_id<2>
                :count<?> :in_stmt_mod<0> :code_object<?> :outer<?>
            [...]
            - QAST::Block(:cuid(1)) <wanted> :statement_id<2>
                    :IN_DECL<> :in_stmt_mod<0> :code_object<?>
                    :also_uses<?> :outer<?> {say $c}
                [...]
                - QAST::Op(call &say)  say $c
    [...]
    - QAST::Op(p6typecheckrv)
        [...]
        - QAST::WVal(Block :cuid(1))

Looks like cuid 2 block steals our cuid 1 block. Let’s enable the debug env var and look at the dumps to see why exactly: 看起来像cuid 2块偷了我们的cuid 1块。让我们启用debug env var并查看转储,看看为什么:

$ ZZ=1 ./raku -e '
    sub itcavuc ($c) { try {say $c} andthen 42 };
    itcavuc $_ for 2, 4, 6;'

ZZ1: -----------------
ZZ1: trying to grab - QAST::Block(:cuid(1)) <wanted>
    :statement_id<2> :IN_DECL<> :in_stmt_mod<0> :code_object<?>
    :also_uses<?> :outer<?> {say $c}
[...]

ZZ1: to move to - QAST::Block  :statement_id<2>
    :in_stmt_mod<0> :outer<?>

ZZ1: grabbed
ZZ1: -----------------

We can see the theft in progress. Let’s take a look at our migration predicate again: 我们可以看到正在进行的盗窃。让我们再看看我们的[迁移谓词](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp#L9196):

! $b.ann('in_stmt_mod')
&& ($b.ann('statement_id') // -1) >= $migrate_stmt_id

In the dump we can see in_stmt_mod is false. Were it set to a true value, the block would not be migrated—exactly what we’re trying to accomplish. Let’s investigate the in_stmt_mod annotation, to see when it gets set: 在转储中,我们可以看到in_stmt_mod是错误的。如果它被设置为一个真正的价值,块将不会被迁移 - 正是我们想要完成的。 我们来研究in_stmt_mod注释,看看它什么时候被设置:

$ G 'in_stmt_mod' src/Raku/Actions.nqp
2327:                $_.annotate('in_stmt_mod', $*IN_STMT_MOD);
9206:                !$b.ann('in_stmt_mod') && ($b.ann('statement_id') // -1) >= $migrate_stmt_id

$ G '$*IN_STMT_MOD' src/Raku/Grammar.nqp
1200:        :my $*IN_STMT_MOD := 0;                    # are we inside a statement modifier?
1328:        :my $*IN_STMT_MOD := 0;
1338:        | <EXPR> :dba('statement end') { $*IN_STMT_MOD := 1 }

Looks like it’s a marker for statement modifier conditions. Statement modifiers have a lot of relevance to our andthen thunks, because $foo with $bar gets turned into $bar andthen $foo during parsing. Since, as we can see in src/Raku/Grammar.nqp, in_stmt_mod annotation gets set for with statement modifiers, we can hypothesize that if we turn our buggy andthen into a with, the bug will disappear: 看起来这是语句修饰符条件的标记。语句修饰符与我们的’和’thunk有很多相关性,因为$ foo with $ bar在解析过程中变成$ bar并且$ foo。正如我们在src / Raku / Grammar.nqp中可以看到的,‘in_stmt_mod注释是为``with语句修饰符'设置的,我们可以假设如果我们把我们的bug和``然后``变成with`,bug会消失:

$ ./raku -e 'sub itcavuc ($c) { 42 with try {say $c} };
    itcavuc $_ for 2, 4, 6;'
2
4
6

And indeed it does! Then, we have a way forward: we need to set in_stmt_mod annotation to a truthy value for just the first argument of andthen (and its relatives notandthen and orelse).

Glancing at the Grammar it doesn’t look like it immediatelly offers a similar opportunity for how in_stmt_mod is set for the with statement modifier. Let’s approach it differently. Since we care about this when thunks are created, let’s watch for andthen QAST inside sub thunkity_thunk in Actions, then descend into its first kid and add the in_stmt_mod annotation by cheating and using the past_block annotation on QAST::WVal with the thunk that contains the reference to QAST::Block we wish to annotate. The code will look something like this: 看着[在语法](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Grammar.nqp#L4650-L4651)它看起来并不像它立即提供了一个类似的机会, in_stmt_mod设置为with语句修饰符。让我们以不同的方式处理。由于我们在创建thunk的时候关心这个,所以让我们来看看在动作中的subthunkity_thunk内是否有Qt(https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp #L7170),然后下降到它的第一个孩子,通过欺骗和使用QAST :: WVal上的past_block注释添加in_stmt_mod`注释与包含’QAST :: Block’引用的thunk我们希望注释。代码将如下所示:

sub mark_blocks_as_andnotelse_first_arg($ast) {
    if $ast && nqp::can($ast, 'ann') && $ast.ann('past_block') {
        $ast.ann('past_block').annotate: 'in_stmt_mod', 1;
    }
    elsif nqp::istype($ast, QAST::Op)
    || nqp::istype($ast, QAST::Stmt)
    || nqp::istype($ast, QAST::Stmts) {
        mark_blocks_as_andnotelse_first_arg($_) for @($ast)
    }
}

sub thunkity_thunk($/,$thunky,$past,@clause) {
    [...]

    my $andnotelse_thunk := nqp::istype($past, QAST::Op)
      && $past.op eq 'call'
      && ( $past.name eq '&infix:<andthen>'
        || $past.name eq '&infix:<notandthen>'
        || $past.name eq '&infix:<orelse>');

    while $i < $e {
        my $ast := @clause[$i];
        $ast := $ast.ast if nqp::can($ast,'ast');
        mark_blocks_as_andnotelse_first_arg($ast)
            if $andnotelse_thunk && $i == 0;
        [...]

First, we rake $past argument given to thunkity_thunk for a QAST::Op for nqp::call that calls one of our ops—when we found one, we set a variable to a truthy value. Then, in the loop, when we’re iterating over the first child node ($i == 0) of these ops, we’ll pass its QAST to our newly minted mark_blocks_as_andnotelse_first_arg routine, inside of which we recurse over any ops that can have kids and mark anything that has past_block annotation with truthy in_stmt_mod annotation.

Let’s compile our concoction and give the tests another run. Once again, I’m using Z-Script to recompile Rakudo: 首先,我们为’nqp :: call调用'thunkity_thunk$ past参数,调用nqp :: call调用我们的一个操作 - 当我们找到一个操作时,我们设置一个变量为真值。然后,在循环中,当我们遍历这些操作的第一个子节点($ i == 0)时,我们将把它的QAST传递给我们新建立的mark_blocks_as_andnotelse_first_arg例程,任何可以有孩子的操作,并用truthyin_stmt_mod注释标记任何具有past_block`注解的东西。

让我们编写我们的调和,并再次运行测试。再次,我使用[Z-Script](https://github.com/zoffixznet/z)重新编译Rakudo:

$ z
[...]
$ ./raku bug-tests.t
1..1
    1..10
    ok 1 - xx inside `with`
    ok 2 - try with block and andthen
    ok 3 - block in a sub with orelse
    not ok 4 - loop + lexical variable plus chain of andthens
    # Failed test 'loop + lexical variable plus chain of andthens'
    # at bug-tests.t line 23
    # expected: $(1, 1, 1, 1, 1, 1, 1)
    #      got: $(1, 4, 3, 3, 3, 3, 3)
    ok 5 - loop + andthen + orelse
    ok 6 - parentheses + xx + given
    ok 7 - postfix for + take + block in a string
    ok 8 - given + whatever code closure execution
    ok 9 - sub + given + whatevercode closure execution
    not ok 10 - sub with `with` + orelse + block interpolation
    # Failed test 'sub with `with` + orelse + block interpolation'
    # at bug-tests.t line 50
    # expected: $("meow True",)
    #      got: $("meow False",)
    # Looks like you failed 2 tests of 10
not ok 1 - thunking closure scoping
# Failed test 'thunking closure scoping'
# at bug-tests.t line 4
# Looks like you failed 1 test of 1

We got closer to the goal, with 80% of the tests now passing! In the first remaining failure, we already know from our original examination that chained andthen thunks get nested when they should not—we haven’t done anything to fix that yet. Let’s take care of that first. 我们接近了目标,80%的测试通过了!在剩下的第一次失败中,我们从原来的检查中已经知道,当他们不应该的时候,我们已经把他们连在一起,然后把他们联系起来 - 我们还没有做任何事情来解决这个问题。先来照顾一下吧

Playing Chinese Food Mind Games

Looking back out at the fixes we applied already, we have a marker for when we’re working with andthen or its sister ops: the $andnotelse_thunk variable. It seems fairly straight-forward that if we don’t want the thunks of these ops to migrate, we just need to annotate them appropriately and stick the check for that annotation into the migration predicate.

In Grammar.nqp, we can see our ops are configured with the .b thunky, so we’ll locate that branch in sub thunkity_thunk and pass $andnotelse_thunk variable as a new named param to the make_topic_block_ref block maker: 回想一下我们已经应用的修复,我们有一个标记,当我们正在使用andthen或者它的姐妹操作时:$ andnotelse_thunk变量。看起来很简单,如果我们不希望这些操作符的迁移,我们只需要适当地注释它们,并将该注释的检查插入到迁移谓词中。

Grammar.nqp中,[我们可以看到我们的操作已经配置](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Grammar.nqp#L4007-L4009)与.b thunky,所以我们将找到[分支](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp#L7218-L7223)在sub thunkity_thunk并通过$ andnotelse_thunk“变量作为”make_topic_block_ref“块制造商的新命名参数:

...
elsif $type eq 'b' {  # thunk and topicalize to a block
    unless $ast.ann('bare_block') || $ast.ann('past_block') {
        $ast := block_closure(make_topic_block_ref(@clause[$i],
          $ast, :$andnotelse_thunk,
          migrate_stmt_id => $*STATEMENT_ID));
    }
    $past.push($ast);
}
...

The block maker will shove it into the migration predicate, so our block maker code becomes this: [The block maker](](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp#L9189-L9198))会将其推入到迁移谓词中,所以我们的块生成器代码变成这样:

 sub make_topic_block_ref(
    $/, $past, :$copy, :$andnotelse_thunk, :$migrate_stmt_id,
 ) {
    my $block := $*W.push_lexpad($/);

    # Add annotation to thunks of our ops:
    $block.annotate: 'andnotelse_thunk', 1 if $andnotelse_thunk;

    $block[0].push
        QAST::Var.new( :name('$_'), :scope('lexical'), :decl('var') );
    $block.push($past);
    $*W.pop_lexpad();
    if nqp::defined($migrate_stmt_id) {
        migrate_blocks($*W.cur_lexpad(), $block, -> $b {
               ! $b.ann('in_stmt_mod')

            # Don't migrate thunks of our ops:
            && ! $b.ann('andnotelse_thunk')

            && ($b.ann('statement_id') // -1) >= $migrate_stmt_id
        });
    }
    ...

One more compilation cycle and test run: 再一个编译周期和测试运行:

$ z
[...]
$ ./raku bug-tests.t
1..1
    1..10
    ok 1 - xx inside `with`
    ok 2 - try with block and andthen
    ok 3 - block in a sub with orelse
    ok 4 - loop + lexical variable plus chain of andthens
    ok 5 - loop + andthen + orelse
    ok 6 - parentheses + xx + given
    ok 7 - postfix for + take + block in a string
    ok 8 - given + whatever code closure execution
    ok 9 - sub + given + whatevercode closure execution
    not ok 10 - sub with `with` + orelse + block interpolation
    # Failed test 'sub with `with` + orelse + block interpolation'
    # at bug-tests.t line 50
    # expected: $("meow True",)
    #      got: $("meow False",)
    # Looks like you failed 1 test of 10
not ok 1 - thunking closure scoping
# Failed test 'thunking closure scoping'
# at bug-tests.t line 4
# Looks like you failed 1 test of 1

So close! Just a single test failure remains. Let’s give it a close look. 很近!只剩下一个测试失败。让我们仔细看看。

Within and Without

Let’s repeat our procedure of dumping QASTs as well as enabing the ZZ env var and looking at what’s causing the thunk mis-migration. I’m going to run a slightly simplified version of the failing test, to keep the cruft out of QAST dumps. If you’re following along, when looking at full QAST dump keep in mind what I mentioned earlier: with gets rewritten into andthen op call during parsing. 让我们重复我们的倾销QASTs的程序,并调用ZZ env var并查看导致thunk错误迁移的原因。我将运行一个稍微简化的失败测试版本,以防止QAST转储。如果你一直在跟踪,当看完整的QAST转储记住我刚才提到的:在分析过程中,with会被重写成然后调用。

$ q a ./raku -e '.uc with +"a" orelse "meow {$_ ~~ Failure}".say and 42'
$ firefox out.html

- QAST::Block(:cuid(4)) :in_stmt_mod<0>
    [...]
    - QAST::Block(:cuid(1))  :statement_id<1> :in_stmt_mod<1>
      [...]
      - QAST::Op(chain &infix:<~~>) <wanted> :statement_id<2> ~~
        - QAST::Var(lexical $_) <wanted> $_
        - QAST::WVal(Failure) <wanted> Failure
    - QAST::Block(:cuid(2)) :statement_id<1>
        :in_stmt_mod<1> :andnotelse_thunk<1>
      [...]
      - QAST::Op(callmethod Stringy) <wanted>
        - QAST::Op(call) <wanted> {$_ ~~ Failure}
          - QAST::Op(p6capturelex) <wanted> :code_object<?>
            - QAST::Op(callmethod clone)
              - QAST::WVal(Block)

$ ZZ=1 ./raku -e '.uc with +"a" orelse "meow {$_ ~~ Failure}".say and 42'
[...]
ZZ1: -----------------
ZZ1: trying to grab - QAST::Block(:cuid(1))
  :statement_id<1> :in_stmt_mod<1>
  [...]
ZZ1: to move to - QAST::Block
  :statement_id<1> :andnotelse_thunk<1> :in_stmt_mod<1>
  [...]
ZZ1: -----------------

Although QAST::WVal lacks .past_block annotation and so doesn’t show the block’s CUID in the dump, just by reading the code dumped around that QAST, we can see that the CUID-less block is our QAST::Block :cuid(1), whose immediate outer is QAST::Block :cuid(4), yet it’s called from within QAST::Block :cuid(2). It’s supposed to get migrated, but that migration never happens, as we can see when we use the ZZ env var to enable our debug dumps in the sub migrate_blocks.

We can see why. Here’s our current migration predicate (where $b is the examined block, which in our case is QAST::Block :cuid(1)): 虽然QAST :: WVal缺少.past_block注释,所以在转储中不显示块的CUID,只要读取QAST周围的代码,就可以看到CUID-less块是我们的QAST: :block:cuid(1),它的直接外部是QAST :: Block:cuid(4),但是它在’QAST :: Block:cuid(2)中被调用。它应该被迁移,但是迁移从来没有发生过,就像我们使用'ZZ env var’sub migrate_blocks`启用调试转储时可以看到的那样。

我们可以看到为什么。这里是我们当前的迁移谓词(其中$ b是被检查的块,在我们的例子中是QAST :: Block:cuid(1)):

   ! $b.ann('in_stmt_mod')
&& ! $b.ann('andnotelse_thunk')
&& ($b.ann('statement_id') // -1) >= $migrate_stmt_id

The very first condition prevents our migration, as our block has truthy in_stmt_mod annotation, because it’s part of the with’s condition. At the same time, it does need to be migrated because it’s part of the andthen thunk that’s inside the statement modifier!

Since we already have $andnotelse_thunk variable in the vicinity of the migration predicate we can use it to tell us whether we’re migrating for the benefit of our andthen thunk and not the statement modifier. However, recall that we’ve used the very same in_stmt_mod annotation to mark the first argument of andthen and its brother ops. We need to alter that first.

And so, the sub mark_blocks_as_andnotelse_first_arg we added earlier becomes: 第一个条件阻止了我们的迁移,因为我们的块有in_stmt_mod注释,因为它是’with条件的一部分。 同时,它*需要被迁移,因为它是语句修饰符中和然`thunk的一部分!

由于我们已经在迁移谓词附近有$ andnotelse_thunk变量,所以我们可以用它来告诉我们是否为了我们的’andthen’thunk而不是语句修饰符的迁移。不过请记住,我们已经使用了相同的in_stmt_mod注释来标记andthen和它的兄弟操作符的第一个参数。我们需要先改变它。

因此,我们之前添加的sub mark_blocks_as_andnotelse_first_arg变成:

sub mark_blocks_as_andnotelse_first_arg($ast) {
    if $ast && nqp::can($ast, 'ann') && $ast.ann('past_block') {
        $ast.ann('past_block').annotate: 'in_stmt_mod_andnotelse', 1;
    }
    ...

And then we tweak the migration predicate to watch for this altered annotation and to consider the value of $andnotelse_thunk variable: 然后我们调整[迁移谓词](https://github.com/rakudo/rakudo/blob/a5c2398cc744706eb81b3d73b181cb4233c85a17/src/Raku/Actions.nqp#L9195-L9197)观察这个改变的注释,并考虑$ andnotelse_thunk变量:

migrate_blocks($*W.cur_lexpad(), $block, -> $b {
    (    (! $b.ann('in_stmt_mod_andnotelse') &&   $andnotelse_thunk)
      || (! $b.ann('in_stmt_mod')            && ! $andnotelse_thunk)
    )
    && ($b.ann('statement_id') // -1) >= $migrate_stmt_id
    && ! $b.has_ann('andnotelse_thunk')
});

Thus, we migrate all the blocks with statement_id equal to or higher than ours and are all of the following: 因此,我们迁移所有与statement_id等于或高于我们的块,并且都是以下内容:

  • Not thunks of actual andthen, notandthen, or orelse
  • Not thunks inside a statement modifier, unless they’re inside thunks of andthen or related ops
  • If we’re considering migrating them inside one of the andthen’s thunks, then also not part of the first argument to andthen (or related ops),.

That’s a fancy-pants predicate. Let’s compile and see if it gets the job done:

  • 不是实际的“和”,“不和然”或“orelse”的thunk
  • 不是在声明修饰符中的thunks,除非它们在andthen或相关操作符的内部 如果我们正在考虑将它们迁移到的其中一个thunk中,那么也可以 not *作为andthen(或相关操作)的第一个参数的一部分。

这是一个花哨的谓词。让我们编译一下,看看它是否完成了工作:

$ z
[...]
$ ./raku bug-tests.t
  1..1
    1..10
    ok 1 - xx inside `with`
    ok 2 - try with block and andthen
    ok 3 - block in a sub with orelse
    ok 4 - loop + lexical variable plus chain of andthens
    ok 5 - loop + andthen + orelse
    ok 6 - parentheses + xx + given
    ok 7 - postfix for + take + block in a string
    ok 8 - given + whatever code closure execution
    ok 9 - sub + given + whatevercode closure execution
    ok 10 - sub with `with` + orelse + block interpolation
ok 1 - thunking closure scoping

Success! Now, let’s remove all of the debug statements we added. Then, recompile and run make stresstest, to ensure we did not break anything else. With Z-Script, we can do all that by just running z ss: 成功!现在,让我们删除所有我们添加的调试语句。然后,重新编译并运行make stresstest,确保我们没有破坏别的东西。 使用[Z-Script](https://github.com/zoffixznet/z),我们可以通过运行z ss来完成所有的工作:

$ z ss
[...]
All tests successful.
Files=1287, Tests=153127, 159 wallclock secs (21.40 usr  3.27 sys + 3418.56 cusr 179.32 csys = 3622.55 CPU)
Result: PASS

All is green. We can now commit our fix to Rakudo’s repo, then commit our tests to the roast repo, and all that remains is closing those 10 tickets we fixed!

Job well done. 一切都是绿色的。我们现在可以[提交](https://github.com/rakudo/rakudo/commit/1ee89b54074e80c0753a120d679c6265bd8d5d1f) 我们修复了[Rakudo的回购](https://github.com/rakudo/rakudo/),然后[提交](https://github.com/raku/roast/commit/2f2998733a2d8132ce29a16008cc5b3a50d6567f)我们的测试到[roast回购](https://github.com/raku/roast),剩下的就是关闭我们修复的10张门票!

做得好。

Conclusion

Today, we learned quite a bit about QAST: the Abstract Syntax Trees Raku code compiles to in the Rakudo compiler. We examined the common types of QAST and how to create, annotate, mutate, execute, and dump them for examination.

In the second part of the article, we applied our new knowledge to fix a hive of mis-scoped thunking bugs that plagued various Raku constructs. We introspected the generated QAST nodes to specially annotate them, and then used those annotations to reconfigure migration predicate, so that it migrates the blocks correctly.

Hopefully, this knowledge inspires you to fix the many other bugs we have on the RT tracker as well as our GitHub Issue tracker

今天,我们了解了QAST:抽象语法树Raku代码编译到Rakudo编译器中。我们研究了QAST的常见类型,以及如何创建,注释,变异,执行和转储它们以供检查。

在本文的第二部分,我们应用了我们的新知识来修复困扰各种Raku结构的错误的thunking bug。 我们内置了生成的QAST节点来专门注释它们,然后使用这些注释来重新配置迁移谓词,以便正确地迁移块。

希望这个知识能够激发你去修复我们在RT追踪器(https://fail.rakudo.party/)和我们的GitHub问题追踪器(https://github.com/) rakudo / rakudo /问题)

Raku 

comments powered by Disqus