To tidy up |INVOCATION_LIST_NT| nodes into a list of children under the relevant |RULE_NT| node, and so turn each rule definition into a single subtree.


§1. Initially, the phrases (INVOCATION_LIST_NT) making up a rule (RULE_NT) are simply listed after it in the parse tree, but we want them to become its children: this is the only thing the \(A\)-grammar does with rules, which otherwise wait until later to be dealt with.

The code in this section accomplishes the regrouping: after it runs, every INVOCATION_LIST_NT is a child of the RULE_NT header to which it belongs.

§2. This routine is used whenever new material is added. Whenever it finds a childless RULE_NT followed by a sequence of INVOCATION_LIST_NT nodes, it joins these in sequence as children of the RULE_NT. Since it always acts so as to leave a non-zero number of children, and since it acts only on childless nodes, it cannot ever act on the same node twice.

void RuleSubtrees::register_recently_lexed_phrases(void) {
    if (problem_count > 0) return;  for then the tree is perhaps broken anyway
    SyntaxTree::traverse(Task::syntax_tree(), RuleSubtrees::demote_command_nodes);
    SyntaxTree::traverse(Task::syntax_tree(), RuleSubtrees::detect_loose_command_nodes);
}

§3. Command nodes are demoted to be children of routine nodes:

void RuleSubtrees::demote_command_nodes(parse_node *p) {
    if ((Node::get_type(p) == RULE_NT) && (p->down == NULL)) {
        parse_node *end_def = p;
        while ((end_def->next) && (Node::get_type(end_def->next) == INVOCATION_LIST_NT))
            end_def = end_def->next;
        if (p == end_def) return;  RULE_NT not followed by any INVOCATION_LIST_NTs
         splice so that p->next to end_def become the children of p:
        p->down = p->next;
        p->next = end_def->next;
        end_def->next = NULL;
        RuleSubtrees::parse_routine_structure(p);
    }
}

§4. And just in case:

void RuleSubtrees::detect_loose_command_nodes(parse_node *p) {
    if (Node::get_type(p) == INVOCATION_LIST_NT)
        internal_error("loose COMMAND node outside of rule definition");
}

§5. Parsing Routine Structure. There are now two possible syntaxes to express the structural makeup of a routine. Traditional I7 syntax for blocks is to place them within begin/end markers: the "begin" occurring at the end of the conditional or loop header, and the "end if", "end while", etc., as a phrase of its own at the end of the block. Newer I7 syntax (March 2008) is to use Python-style colons and indentation. Both are allowed, but not in the same routine.

This routine opens with the routine's parse tree consisting of a simple linked list of code-point nodes, one for each phrase. We must work out which syntax is used, decipher it, and turn the list into a proper tree structure in a single unified format.

How much simpler this would all be if we could abolish the old format, but it's kept for the benefit of partially sighted users, who find tabbed indentation difficult to manage with screen-readers.

void RuleSubtrees::parse_routine_structure(parse_node *routine_node) {
    int initial_problem_count = problem_count;

    parse_node *uses_colon_syntax = NULL;
    parse_node *uses_begin_end_syntax = NULL;
    parse_node *mispunctuates_begin_end_syntax = NULL;
    parse_node *requires_colon_syntax = NULL;

    (a.1) See which block syntax is used by conditionals and loops5.1;
    (a.2) Report problems if the two syntaxes are mixed up with each other5.2;
    if (problem_count > initial_problem_count) return;

    if (uses_colon_syntax) (b.1) Annotate the parse tree with indentation levels5.3;
    (b.2) Annotate the parse tree with control structure usage5.4;

    (c) Expand comma notation for blocks5.5;
    if (problem_count > initial_problem_count) return;

    if (uses_colon_syntax) (d) Insert end nodes and check the indentation5.6;
    if (problem_count > initial_problem_count) return;

    (e) Structure the parse tree to match the use of control structures5.7;
    if (problem_count > initial_problem_count) return;

    (f) Police the structure of the parse tree5.8;
    if (problem_count > initial_problem_count) return;

    (g) Optimise out the otherwise if nodes5.9;
    if (problem_count > initial_problem_count) return;

    (h) Remove any end markers as no longer necessary5.11;
    if (problem_count > initial_problem_count) return;

    if (uses_colon_syntax == FALSE)
        (i) Remove any begin markers as no longer necessary5.13;

    (j) Insert code block nodes so that nodes needing to be parsed are childless5.15;
    (k) Insert instead marker nodes5.17;
    (l) Break up say phrases5.19;
}

§5.1. (a.1) See which block syntax is used by conditionals and loops5.1 =

    parse_node *p;
    for (p = routine_node->down; p; p = p->next) {
        control_structure_phrase *csp =
            ControlStructures::detect(Node::get_text(p));
        if (csp) {
            int syntax_used = Annotations::read_int(p, colon_block_command_ANNOT);
            if (syntax_used == FALSE) {  i.e., doesn't end with a colon
                 don't count "if x is 1, let y be 2" — with no block — as deciding it
                if ((csp->subordinate_to == NULL) &&
                    (!(<phrase-beginning-block>(Node::get_text(p)))))
                    syntax_used = NOT_APPLICABLE;
            }
            if (syntax_used != NOT_APPLICABLE) {
                if (syntax_used) {
                    if (uses_colon_syntax == NULL) uses_colon_syntax = p;
                } else {
                    Note what looks like a begin-end piece of syntax5.1.1;
                }
            }
            if ((csp->requires_new_syntax) && (requires_colon_syntax == NULL))
                requires_colon_syntax = p;
        }
        if (ControlStructures::detect_end(Node::get_text(p))) {
            if (uses_begin_end_syntax == NULL)
                uses_begin_end_syntax = p;
        }
    }

§5.1.1. It's possible in oddball cases to mis-punctuate such as to fool us, so:

Note what looks like a begin-end piece of syntax5.1.1 =

    if ((uses_begin_end_syntax == NULL) && (mispunctuates_begin_end_syntax == NULL)) {
        if (<phrase-beginning-block>(Node::get_text(p)))
            uses_begin_end_syntax = p;
        else
            mispunctuates_begin_end_syntax = p;
    }

§5.2. (a.2) Report problems if the two syntaxes are mixed up with each other5.2 =

    if ((uses_colon_syntax) && (mispunctuates_begin_end_syntax)) {
        current_sentence = routine_node;
        Problems::quote_source(1, current_sentence);
        Problems::quote_source(2, mispunctuates_begin_end_syntax);
        StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_BadOldSyntax));
        Problems::issue_problem_segment(
            "The rule or phrase definition %1 seems to use indentation and "
            "colons to group phrases together into 'if', 'repeat' or 'while' "
            "blocks. That's fine, but then this phrase seems to be missing "
            "some punctuation - %2. Perhaps a colon is missing?");
        Problems::issue_problem_end();
        return;
    }

    if ((uses_colon_syntax) && (uses_begin_end_syntax)) {
        current_sentence = routine_node;
        Problems::quote_source(1, current_sentence);
        Problems::quote_source(2, uses_colon_syntax);
        Problems::quote_source(3, uses_begin_end_syntax);
        StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_BothBlockSyntaxes));
        Problems::issue_problem_segment(
            "The rule or phrase definition %1 seems to use both ways of grouping "
            "phrases together into 'if', 'repeat' and 'while' blocks at once. "
            "Inform allows two alternative forms, but they cannot be mixed in "
            "the same definition. %POne way is to end the 'if', 'repeat' or "
            "'while' phrases with a 'begin', and then to match that with an "
            "'end if' or similar. ('Otherwise' or 'otherwise if' clauses are "
            "phrases like any other, and end with semicolons in this case.) "
            "You use this begin/end form here, for instance - %3. %P"
            "The other way is to end with a colon ':' and then indent the "
            "subsequent phrases underneath, using tabs. (Note that any "
            "'otherwise' or 'otherwise if' clauses also have to end with "
            "colons in this case.) You use this indented form here - %2.");
        Problems::issue_problem_end();
        return;
    }

    if ((requires_colon_syntax) && (uses_begin_end_syntax)) {
        current_sentence = routine_node;
        Problems::quote_source(1, current_sentence);
        Problems::quote_source(2, requires_colon_syntax);
        StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_NotInOldSyntax));
        Problems::issue_problem_segment(
            "The construction %2, in the rule or phrase definition %1, "
            "is only allowed if the rule is written in the 'new' format, "
            "that is, with the phrases written one to a line with "
            "indentation showing how they are grouped together, and "
            "with colons indicating the start of such a group.");
        Problems::issue_problem_end();
        return;
    }

§5.3. If we're using Pythonesque notation, then the number of tab stops of indentation of a phrase tells us where it belongs in the structure, so we mark up the tree with that information.

(b.1) Annotate the parse tree with indentation levels5.3 =

    Annotations::write_int(routine_node, indentation_level_ANNOT,
        Lexer::indentation_level(Wordings::first_wn(Node::get_text(routine_node))));
    parse_node *p;
    for (p = routine_node->down; p; p = p->next) {
        int I = Lexer::indentation_level(Wordings::first_wn(Node::get_text(p)));
        Annotations::write_int(p, indentation_level_ANNOT, I);
    }

§5.4. Note that we are a little cautious about recognising phrases which will open blocks, such as "repeat...", because of the dangers of false positives; so we look for the "begin" keyword, or the colon. We're less cautious with subordinate phrases (such as "otherwise") because we know their wonding more certainly, and similarly for "end X" phrases.

(b.2) Annotate the parse tree with control structure usage5.4 =

    for (parse_node *p = routine_node->down; p; p = p->next) {
        control_structure_phrase *csp;
        csp = ControlStructures::detect(Node::get_text(p));
        if (csp) {
            if ((Annotations::read_int(p, colon_block_command_ANNOT)) ||
                (<phrase-beginning-block>(Node::get_text(p))) ||
                (csp->subordinate_to)) {
                Node::set_control_structure_used(p, csp);
                if (csp == case_CSP) Trim a switch case to just the case value5.4.1;
            }
        }
        csp = ControlStructures::detect_end(Node::get_text(p));
        if (csp) Node::set_end_control_structure_used(p, csp);
    }

§5.4.1. At this point anything at all can be a case value: it won't be parsed or type-checked until compilation.

Trim a switch case to just the case value5.4.1 =

    Node::set_text(p, GET_RW(<control-structure-phrase>, 1));

§5.5. "Comma notation" is when a comma is used in an "if" statement to divide off only a single consequential phrase, as in

if the hat is worn, try dropping the hat;

Such a line occupies a single node in its routine's parse tree, and we need to break this up.

(c) Expand comma notation for blocks5.5 =

    for (parse_node *p = routine_node->down; p; p = p->next)
        if (Node::get_control_structure_used(p) == NULL) {
            control_structure_phrase *csp;
            csp = ControlStructures::detect(Node::get_text(p));
            if ((csp == if_CSP) && (<phrase-with-comma-notation>(Node::get_text(p))))
                Effect a comma expansion5.5.1;
        }

§5.5.1. Effect a comma expansion5.5.1 =

    wording BCW = GET_RW(<phrase-with-comma-notation>, 1);  text before the comma
    wording ACW = GET_RW(<phrase-with-comma-notation>, 2);  text after the comma

     First trim and annotate the "if ..." part
    Annotations::write_int(p, colon_block_command_ANNOT, TRUE);  it previously had no colon...
    Node::set_control_structure_used(p, csp);  ...and therefore didn't have its CSP set
    Node::set_text(p, BCW);

     Now make a new node for the "then" part, indenting it one step inward
    parse_node *then_node = Node::new(INVOCATION_LIST_NT);
    Annotations::write_int(then_node, results_from_splitting_ANNOT, TRUE);
    Annotations::write_int(then_node, indentation_level_ANNOT,
        Annotations::read_int(p, indentation_level_ANNOT) + 1);
    Node::set_text(then_node, ACW);

    parse_node *last_node_of_if_construction = then_node, *rest_of_routine = p->next;

     Attach the "then" node after the "if" node:
    p->next = then_node;

    Deal with an immediately following otherwise node, if there is one5.5.1.1;

    if (uses_colon_syntax == FALSE) {
        last_node_of_if_construction->next = RuleSubtrees::end_node(p);
        last_node_of_if_construction->next->next = rest_of_routine;
    } else {
        last_node_of_if_construction->next = rest_of_routine;
    }

§5.5.1.1. Deal with an immediately following otherwise node, if there is one5.5.1.1 =

    if (rest_of_routine)
        if ((uses_colon_syntax == FALSE) ||
            (Annotations::read_int(p, indentation_level_ANNOT) ==
                Annotations::read_int(rest_of_routine, indentation_level_ANNOT))) {
            if (Node::get_control_structure_used(rest_of_routine) == otherwise_CSP)
                Deal with an immediately following otherwise5.5.1.1.1
            else if (ControlStructures::abbreviated_otherwise(Node::get_text(rest_of_routine)))
                Deal with an abbreviated otherwise node5.5.1.1.2;
        }

§5.5.1.1.1. We string a plain "otherwise" node onto the "if" construction.

Deal with an immediately following otherwise5.5.1.1.1 =

    then_node->next = rest_of_routine;
    last_node_of_if_construction = last_node_of_if_construction->next;
    rest_of_routine = rest_of_routine->next;

§5.5.1.1.2. An abbreviated otherwise clause looks like this:

otherwise award 4 points;

and we want to split this, too, into distinct nodes.

Deal with an abbreviated otherwise node5.5.1.1.2 =

    parse_node *otherwise_node = Node::new(CODE_BLOCK_NT);
    Annotations::write_int(otherwise_node, results_from_splitting_ANNOT, TRUE);
    Annotations::write_int(otherwise_node, indentation_level_ANNOT,
        Annotations::read_int(p, indentation_level_ANNOT));
    Node::set_text(otherwise_node,
        Wordings::one_word(Wordings::first_wn(Node::get_text(rest_of_routine))));  extract just the word "otherwise"
    Node::set_control_structure_used(otherwise_node, otherwise_CSP);

    then_node->next = otherwise_node;
    otherwise_node->next = rest_of_routine;

    Node::set_text(rest_of_routine,
        Wordings::trim_first_word(Node::get_text(rest_of_routine)));  to remove the "otherwise"

    Annotations::write_int(rest_of_routine, indentation_level_ANNOT,
        Annotations::read_int(rest_of_routine, indentation_level_ANNOT) + 1);

    last_node_of_if_construction = rest_of_routine;
    rest_of_routine = rest_of_routine->next;

§5.6. If the old-style syntax is used, there are explicit "end if", "end repeat" and "end while" nodes in the list already. But if the Pythonesque syntax is used then we need to create these nodes and insert them into the list; we do these by reading off the structure from the pattern of indentation. It's quite a long task, since this pattern may contain errors, which we have to report more or less helpfully.

(d) Insert end nodes and check the indentation5.6 =

    parse_node *p, *prev, *run_on_at = NULL;
    parse_node *first_misaligned_phrase = NULL, *first_overindented_phrase = NULL;
    int k, indent, expected_indent = 1, indent_misalign = FALSE, indent_overmuch = FALSE,
        just_opened_block = FALSE;

     the blocks open stack holds blocks currently open
    parse_node *blstack_opening_phrase[GROSS_AMOUNT_OF_INDENTATION+1];
    control_structure_phrase *blstack_construct[GROSS_AMOUNT_OF_INDENTATION+1];
    int blstack_stage[GROSS_AMOUNT_OF_INDENTATION+1];
    int blo_sp = 0, suppress_further_problems = FALSE;

    if (Annotations::read_int(routine_node, indentation_level_ANNOT) != 0)
        Issue problem message for failing to start flush on the left margin5.6.1;

    for (prev = NULL, p = routine_node->down, k=1; p; prev = p, p = p->next, k++) {
        control_structure_phrase *csp = Node::get_control_structure_used(p);
        Determine actual indentation of this phrase5.6.2;
        Compare actual indentation to what we expect from structure so far5.6.3;
        Insert begin marker and increase expected indentation if a block begins here5.6.4;
    }

    indent = 1;
    Try closing blocks to bring expected indentation down to match5.6.5;

    if (indent_overmuch) Issue problem message for an excess of indentation5.6.7
    else if (run_on_at) Issue problem message for run-ons within phrase definition5.6.8
    else if (indent_misalign) Issue problem message for misaligned indentation5.6.6;

§5.6.1. Controversially:

Issue problem message for failing to start flush on the left margin5.6.1 =

    current_sentence = routine_node;
    Problems::quote_source_eliding_begin(1, current_sentence);
    StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_NonflushRule));
    Problems::issue_problem_segment(
        "The phrase or rule definition %1 is written using tab indentations "
        "to show how its phrases are to be grouped together. But in that "
        "case the opening line needs to be on the left margin, not indented.");
    Problems::issue_problem_end();
    suppress_further_problems = TRUE;

§5.6.2. Here we set indent to the number of tab-stops in from the margin, or to expected_indent if the text does not appear to be at the start of its own line in the source (because it runs on from a previous phrase, in which case we set the run_on_at flag: except for following on from cases in switches with a non-control-structure, which is allowed, because otherwise the lines often look silly and short).

Determine actual indentation of this phrase5.6.2 =

    indent = expected_indent;
    if (Annotations::read_int(p, indentation_level_ANNOT) > 0)
        indent = Annotations::read_int(p, indentation_level_ANNOT);
    else if (Wordings::nonempty(Node::get_text(p))) {
        switch (Lexer::break_before(Wordings::first_wn(Node::get_text(p)))) {
            case '\n': indent = 0; break;
            case '\t': indent = 1; break;
            default:
                if ((prev) && (csp == NULL)) {
                    control_structure_phrase *pcsp = Node::get_control_structure_used(prev);
                    if ((pcsp) && (pcsp->allow_run_on)) break;
                }
                if ((Annotations::read_int(p, results_from_splitting_ANNOT) == FALSE) &&
                    (run_on_at == NULL)) run_on_at = p;
                break;
        }
    }
    if (indent >= GROSS_AMOUNT_OF_INDENTATION) Record an excess of indentation5.6.2.1;

§5.6.3. We now know the indent level of the line as read, and also the expected_indent given the definition so far. If they agree, fine. If they don't agree, it isn't necessarily bad news — if each line's indentation were a function of the last, there would be no information in it, after all. Roughly speaking, when indent is greater than we expect, that must be wrong — it means indentation has jumped inward as if to open a new block, but blocks are opened explicitly and not by simply raising the indent. But when indent is less than we expect, this may simply mean that the current block(s) has or have been closed, because blocks are indeed closed implicitly just by moving the indentation back in.

Compare actual indentation to what we expect from structure so far5.6.3 =

    if (indent == 0) {
        Record a misalignment of indentation5.6.3.3;
        Record a phrase within current block5.6.3.2;
    } else {
        if ((csp) && (csp->subordinate_to)) {
            Compare actual indentation to what we expect for an intermediate phrase5.6.3.1;
            just_opened_block = TRUE;
        } else {
            if (expected_indent < indent) Record a misalignment of indentation5.6.3.3;
            if (expected_indent > indent)
                Try closing blocks to bring expected indentation down to match5.6.5;
            expected_indent = indent;
            Record a phrase within current block5.6.3.2;
        }
    }
    if (expected_indent < 1) expected_indent = 1;

§5.6.3.1. This is a small variation used for an intermediate phrase like "otherwise". These are required to be at the same indentation as the line which opened the block, rather than being one tab step in from there: in other words they are not deemed part of the block itself. They can also occur in "stages", which is a way to enforce one intermediate phrase only being allowed after another one — for instance, "otherwise if..." is not allowed after an "otherwise" within an "if".

Compare actual indentation to what we expect for an intermediate phrase5.6.3.1 =

    expected_indent--;
    if (expected_indent < indent) {
        Issue problem for an intermediate phrase not matching5.6.3.1.1;
    } else {
        Try closing blocks to bring expected indentation down to match5.6.5;
        if ((blo_sp == 0) ||
            (csp->subordinate_to != blstack_construct[blo_sp-1])) {
            Issue problem for an intermediate phrase not matching5.6.3.1.1;
        } else {
            if (blstack_stage[blo_sp-1] > csp->used_at_stage)
                Issue problem for an intermediate phrase out of sequence5.6.3.1.2;
            blstack_stage[blo_sp-1] = csp->used_at_stage;
        }
    }
    expected_indent++;

§5.6.4. In colon syntax, blocks are explicitly opened; they are only implicitly closed. Here is the opening:

If p is a node representing a phrase beginning a block, and we're in the colon syntax, then it is followed by a word which is the colon: thus if p reads "if x is 2" then the word following the "2" will be ":".

Insert begin marker and increase expected indentation if a block begins here5.6.4 =

    if ((csp) && (csp->subordinate_to == NULL) && (Annotations::read_int(p, colon_block_command_ANNOT))) {
        expected_indent++;
        if (csp->indent_subblocks) expected_indent++;
        blstack_construct[blo_sp] = csp;
        blstack_stage[blo_sp] = 0;
        blstack_opening_phrase[blo_sp++] = p;
        just_opened_block = TRUE;
    }

§5.6.5. Now for the closing of colon-syntax blocks. We know that blocks must be being closed if the indentation has jumped backwards: but it may be that many blocks are being closed at once. (It may also be that the indentation has gone awry.)

Try closing blocks to bring expected indentation down to match5.6.5 =

    if ((just_opened_block) &&
        (blo_sp > 0) &&
        (!(blstack_construct[blo_sp-1]->body_empty_except_for_subordinates)) && (p))
        Issue problem for an empty block5.6.5.2;
    while (indent < expected_indent) {
        parse_node *opening;
        if (blo_sp == 0) {
            Record a misalignment of indentation5.6.3.3;
            indent = expected_indent;
            break;
        }
        if ((blstack_construct[blo_sp-1]->body_empty_except_for_subordinates) &&
            (expected_indent - indent == 1)) {
            indent = expected_indent;
            break;
        }
        expected_indent--;
        if (blstack_construct[blo_sp-1]->indent_subblocks) expected_indent--;
        opening = blstack_opening_phrase[--blo_sp];
        Insert end marker to match the opening of the block phrase5.6.5.1;
    }

§5.6.3.2. Record a phrase within current block5.6.3.2 =

    if ((blo_sp > 0) &&
        (blstack_stage[blo_sp-1] == 0) &&
        (blstack_construct[blo_sp-1]->body_empty_except_for_subordinates)) {
        Issue problem for non-case in a switch5.6.3.2.1;
    }
    just_opened_block = FALSE;

§5.6.5.1. An end marker is a phrase like "end if" which matches the "if... begin" above it: here we insert such a marker at a place where the source text indentation implicitly requires it.

Insert end marker to match the opening of the block phrase5.6.5.1 =

    parse_node *implicit_end = RuleSubtrees::end_node(opening);
    implicit_end->next = prev->next; prev->next = implicit_end;
    prev = implicit_end;

§5.6.3.3. Here we throw what amounts to an exception...

Record a misalignment of indentation5.6.3.3 =

    indent_misalign = TRUE;
    if (first_misaligned_phrase == NULL) first_misaligned_phrase = p;

§5.6.6. ...and catch it with something of a catch-all message:

Issue problem message for misaligned indentation5.6.6 =

    if (suppress_further_problems == FALSE) {
        LOG("$T\n", routine_node);
        current_sentence = routine_node;
        Problems::quote_source_eliding_begin(1, current_sentence);
        Problems::quote_source_eliding_begin(2, first_misaligned_phrase);
        StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_MisalignedIndentation));
        Problems::issue_problem_segment(
            "The phrase or rule definition %1 is written using the 'colon "
            "and indentation' syntax for its 'if's, 'repeat's and 'while's, "
            "where blocks of phrases grouped together are indented one "
            "tab step inward from the 'if ...:' or similar phrase to which "
            "they belong. But the tabs here seem to be misaligned, and I can't "
            "determine the structure. The first phrase going awry in the "
            "definition seems to be %2, in case that helps. %PThis sometimes "
            "happens even when the code looks about right, to the eye, if rows "
            "of spaces have been used to indent phrases instead of tabs.");
        Problems::Using::diagnose_further();
        Problems::issue_problem_end();
    }

§5.6.2.1. And another...

Record an excess of indentation5.6.2.1 =

    indent_overmuch = TRUE;
    if (first_overindented_phrase == NULL) first_overindented_phrase = p;

§5.6.7. ...caught here:

Issue problem message for an excess of indentation5.6.7 =

    if (suppress_further_problems == FALSE) {
        current_sentence = routine_node;
        Problems::quote_source_eliding_begin(1, current_sentence);
        Problems::quote_source_eliding_begin(2, first_overindented_phrase);
        StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_TooMuchIndentation));
        Problems::issue_problem_segment(
            "The phrase or rule definition %1 is written using tab indentations "
            "to show how its phrases are to be grouped together. But the level "
            "of indentation goes far too deep, reaching more than 25 tab stops "
            "from the left margin.");
        Problems::issue_problem_end();
    }

§5.6.8. Issue problem message for run-ons within phrase definition5.6.8 =

    if (suppress_further_problems == FALSE) {
        current_sentence = routine_node;
        Problems::quote_source_eliding_begin(1, current_sentence);
        Problems::quote_source_eliding_begin(2, run_on_at);
        StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_RunOnsInTabbedRoutine));
        Problems::issue_problem_segment(
            "The phrase or rule definition %1 is written using the 'colon "
            "and indentation' syntax for its 'if's, 'repeat's and 'while's, "
            "but that's only allowed if each phrase in the definition "
            "occurs on its own line. So phrases like %2, which follow "
            "directly on from the previous phrase, aren't allowed.");
        Problems::issue_problem_end();
    }

§5.6.5.2. It's a moot point whether the following should be incorrect syntax, but it far more often happens as an accident than anything else, and it's hard to think of a sensible use.

Issue problem for an empty block5.6.5.2 =

    if (suppress_further_problems == FALSE) {
        LOG("$T\n", routine_node);
        current_sentence = routine_node;
        Problems::quote_source_eliding_begin(1, current_sentence);
        Problems::quote_source_eliding_begin(2, prev);
        Problems::quote_source_eliding_begin(3, p);
        StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_EmptyIndentedBlock));
        Problems::issue_problem_segment(
            "The phrase or rule definition %1 is written using the 'colon "
            "and indentation' syntax for its 'if's, 'repeat's and 'while's, "
            "where blocks of phrases grouped together are indented one "
            "tab step inward from the 'if ...:' or similar phrase to which "
            "they belong. But the phrase %2, which ought to begin a block, "
            "is immediately followed by %3 at the same or a lower indentation, "
            "so the block seems to be empty - this must mean there has been "
            "a mistake in indenting the phrases.");
        Problems::issue_problem_end();
    }

§5.6.3.2.1. Issue problem for non-case in a switch5.6.3.2.1 =

    if (suppress_further_problems == FALSE) {
        current_sentence = routine_node;
        Problems::quote_source_eliding_begin(1, current_sentence);
        Problems::quote_source_eliding_begin(2, p);
        StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_NonCaseInIf));
        Problems::issue_problem_segment(
            "In the phrase or rule definition %1, the phrase %2 came as a "
            "surprise since it was not a case in an 'if X is...' but was "
            "instead some other miscellaneous instruction.");
        Problems::issue_problem_end();
    }

§5.6.3.1.1. Issue problem for an intermediate phrase not matching5.6.3.1.1 =

    if ((indent_misalign == FALSE) && (suppress_further_problems == FALSE)) {
        current_sentence = p;
        if (csp->subordinate_to == if_CSP) {
            LOG("$T\n", routine_node);
            StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_MisalignedOtherwise),
                "this doesn't match a corresponding 'if'",
                "as it must. An 'otherwise' must be vertically underneath the "
                "'if' to which it corresponds, at the same indentation, and "
                "if the 'otherwise' uses a colon to begin a block then the "
                "'if' must do the same.");
        }
        if (csp->subordinate_to == switch_CSP)
            StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_MisalignedCase),
                "this seems to be misplaced since it is not a case within an "
                "'if X is...'",
                "as it must be. Each case must be placed one tab stop in from "
                "the 'if X is...' to which it belongs, and the instructions "
                "for what to do in that case should be one tab stop further in "
                "still.");
    }

§5.6.3.1.2. Issue problem for an intermediate phrase out of sequence5.6.3.1.2 =

    if ((indent_misalign == FALSE) && (suppress_further_problems == FALSE)) {
        current_sentence = p;
        if ((csp == default_case_CSP) || (csp == case_CSP))
            StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_DefaultCaseNotLast),
                "'otherwise' must be the last clause if an 'if ... is:'",
                "and in particular it has to come after all the '-- V:' "
                "case values supplied.");
        else
            StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_MisarrangedOtherwise),
                "this seems to be misplaced since it is out of sequence within its 'if'",
                "with an 'otherwise if...' coming after the more general 'otherwise' "
                "rather than before. (Note that an 'otherwise' or 'otherwise if' must "
                "be vertically underneath the 'if' to which it corresponds, at the "
                "same indentation.");
    }

§5.7. And after all that work, the routine's parse tree still consists only of a linked list of nodes; but at least it now contains the same pattern of nodes whichever syntax is used. We finally make a meaningful tree out of it.

(e) Structure the parse tree to match the use of control structures5.7 =

    parse_node *routine_list = routine_node->down;
    parse_node *top_level = Node::new(CODE_BLOCK_NT);

    routine_node->down = top_level;

    parse_node *attach_owners[MAX_BLOCK_NESTING+1];
    parse_node *attach_points[MAX_BLOCK_NESTING+1];
    control_structure_phrase *attach_csps[MAX_BLOCK_NESTING+1];
    int attach_point_sp = 0;

     push the top level code block onto the stack
    attach_owners[attach_point_sp] = NULL;
    attach_csps[attach_point_sp] = NULL;
    attach_points[attach_point_sp++] = top_level;

    parse_node *overflow_point = NULL;  if any overflow is found
    for (parse_node *pn = routine_list, *pn_prev = NULL; pn; pn_prev = pn, pn = pn->next) {
         unstring this node from the old list
        if (pn_prev) pn_prev->next = NULL;
        Attach the node to the routine's growing parse tree5.7.1;
    }
    if (overflow_point) {
        current_sentence = overflow_point;
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_BlockNestingTooDeep),
            "compound phrases have gone too deep",
            "perhaps because many have begun but not been properly ended?");
    }

§5.7.1. Attach the node to the routine's growing parse tree5.7.1 =

    int go_up = FALSE, go_down = FALSE;
    control_structure_phrase *csp = Node::get_end_control_structure_used(pn);
    if (csp) go_up = TRUE;
    else {
        csp = Node::get_control_structure_used(pn);
        if (csp) {
            go_down = TRUE;
            if (ControlStructures::opens_block(csp) == FALSE) {
                go_up = TRUE;
                Node::set_type(pn, CODE_BLOCK_NT);
            }
        }
    }
    if (go_up) Move the attachment point up in the tree5.7.1.1;
    Attach this latest node5.7.1.2;
    if (go_down) Move the attachment point down in the tree5.7.1.3;

§5.7.1.1. Move the attachment point up in the tree5.7.1.1 =

    control_structure_phrase *superior_csp = attach_csps[attach_point_sp-1];
    if ((superior_csp) && (superior_csp->subordinate_to)) Pop the CSP stack5.7.1.1.1;
    if (go_down == FALSE) Pop the CSP stack5.7.1.1.1;

§5.7.1.2. Attach this latest node5.7.1.2 =

    parse_node *to = attach_points[attach_point_sp-1];
    if ((go_up) && (go_down) && (attach_owners[attach_point_sp-1]))
        to = attach_owners[attach_point_sp-1];
    SyntaxTree::graft(Task::syntax_tree(), pn, to);

§5.7.1.3. Move the attachment point down in the tree5.7.1.3 =

    parse_node *next_attach_point = pn;
    if (go_up == FALSE) {
        pn->down = Node::new(CODE_BLOCK_NT);
        next_attach_point = pn->down;
    }
    Push the CSP stack5.7.1.3.1;

§5.7.1.1.1. It's an error to let this underflow, but we'll catch that problem later.

Pop the CSP stack5.7.1.1.1 =

    if (attach_point_sp != 1) attach_point_sp--;

§5.7.1.3.1. An overflow, however, we must catch right here.

Push the CSP stack5.7.1.3.1 =

    if (attach_point_sp <= MAX_BLOCK_NESTING) {
        attach_owners[attach_point_sp] = pn;
        attach_csps[attach_point_sp] = csp;
        attach_points[attach_point_sp++] = next_attach_point;
    } else {
        if (overflow_point == NULL) overflow_point = pn;
    }

§5.8. We now have a neatly structured tree, so from here on anything we do will need a recursive procedure.

Firstly, the tree is certainly neat, but it can still contain all kinds of nonsense: "if" blocks with multiple "otherwise"s, for example. This is where we look for such mistakes.

(f) Police the structure of the parse tree5.8 =

    int n = problem_count;
    RuleSubtrees::police_code_block(routine_node->down, NULL);
    if (problem_count > n) LOG("Local parse tree: $T\n", routine_node);

§6. Which recursively uses the following:

void RuleSubtrees::police_code_block(parse_node *block, control_structure_phrase *context) {
    for (parse_node *p = block->down, *prev_p = NULL; p; prev_p = p, p = p->next) {
        current_sentence = p;

        control_structure_phrase *prior =
            (prev_p)?Node::get_control_structure_used(prev_p):NULL;
        control_structure_phrase *csp = Node::get_end_control_structure_used(p);
        if ((csp) && (csp != prior)) {
            if (prior == NULL) Issue problem for end without begin6.1
            else Issue problem for wrong sort of end6.2;
        }

        csp = Node::get_control_structure_used(p);
        if (csp) {
            if (ControlStructures::opens_block(csp)) {
                if ((p->next == NULL) ||
                    (Node::get_end_control_structure_used(p->next) == NULL))
                    Issue problem for begin without end6.3;
            } else {
                if (context == NULL)
                    Choose a problem for a loose clause6.4
                else if (context != csp->subordinate_to)
                    Choose a problem for the wrong clause6.5
                else if ((csp == otherwise_CSP) && (p->next))
                    Choose a problem for otherwise not occurring last6.6
                else if ((csp == default_case_CSP) && (p->next))
                    Issue a problem for the default case not occurring last6.7;
            }
        }

        if (p->down) RuleSubtrees::police_code_block(p, csp);
    }
}

§6.1. These used to be much-seen problem messages, until Inform moved to Pythonesque structure-by-indentation. Nowadays "end if", "end while" and such are automatically inserted into the stream of commands, always in the right place, and always passing these checks. But the problem messages are kept for the sake of old-format source text, and for refuseniks.

Issue problem for end without begin6.1 =

    StandardProblems::sentence_problem_with_note(Task::syntax_tree(), _p_(PM_EndWithoutBegin),
        "this is an 'end' with no matching 'begin'",
        "which should not happen: every phrase like 'if ... begin;' "
        "should eventually be followed by its bookend 'end if'. "
        "It makes no sense to have an 'end ...' on its own.",
        "Perhaps the problem is actually that you opened several "
        "such begin... end 'blocks' and accidentally closed them "
        "once too many? This is very easily done.");

§6.2. Issue problem for wrong sort of end6.2 =

    Problems::quote_source(1, current_sentence);
    Problems::quote_wide_text(2, prior->keyword);
    Problems::quote_source(3, prev_p);
    StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_WrongEnd));
    Problems::issue_problem_segment(
        "You wrote %1, but the end I was expecting next was 'end %2', "
        "finishing the block you began with %3.");
    Problems::issue_problem_end();

§6.3. Issue problem for begin without end6.3 =

    StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_BeginWithoutEnd),
        "the definition of the phrase ended with no matching 'end' for "
        "this 'begin'",
        "bearing in mind that every begin must have a matching end, and "
        "that the one most recently begun must be the one first to end. For "
        "instance, 'if ... begin' must have a matching 'end if'.");

§6.4. Choose a problem for a loose clause6.4 =

    if (csp == otherwise_CSP)
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_OtherwiseWithoutIf),
            "this is an 'else' or 'otherwise' with no matching 'if' (or 'unless')",
            "which must be wrong.");
    else if (csp == otherwise_if_CSP)
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_OtherwiseIfMisplaced),
            "the 'otherwise if' clause here seems not to be occurring inside "
            "a large 'if'",
            "and seems to be freestanding instead. (Though 'otherwise ...' can "
            "usually be used after simple one-line 'if's to provide an alternative "
            "course of action, 'otherwise if...' is a different matter, and is "
            "used to divide up larger-scale instructions.)");
    else
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(BelievedImpossible),
            "this clause can't occur outside of a control phrase",
            "which suggests that the structure of this routine is wrong.");

§6.5. Choose a problem for the wrong clause6.5 =

    if ((csp == otherwise_CSP) || (csp == otherwise_if_CSP)) {
        Problems::quote_source(1, current_sentence);
        Problems::quote_wide_text(2, context->keyword);
        StandardProblems::handmade_problem(Task::syntax_tree(), _p_(PM_OtherwiseInNonIf));
        Problems::issue_problem_segment(
            "The %1 here did not make sense inside a "
            "'%2' structure: it's provided for 'if' (or 'unless').");
        Problems::issue_problem_end();
    } else
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(BelievedImpossible),
            "this clause is wrong for the phrase containing it",
            "which suggests that the structure of this routine is wrong.");

§6.6. Choose a problem for otherwise not occurring last6.6 =

    int doubled = FALSE, oi = FALSE;
    for (parse_node *p2 = p->next; p2; p2 = p2->next) {
        if (Node::get_control_structure_used(p2) == otherwise_CSP) {
            current_sentence = p2;
            doubled = TRUE;
        }
        if (Node::get_control_structure_used(p2) == otherwise_if_CSP)
            oi = TRUE;
    }
    if (doubled)
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_DoubleOtherwise),
            "that makes two unconditional 'otherwise' or 'else' clauses "
            "for this 'if'",
            "which is forbidden since 'otherwise' is meant to be a single "
            "(optional) catch-all clause at the end.");
    else if (oi)
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_OtherwiseIfAfterOtherwise),
            "this seems to be misplaced since it is out of sequence within its 'if'",
            "with an 'otherwise if...' coming after the more general 'otherwise' "
            "rather than before. (If there's an 'otherwise' clause, it has to be "
            "the last clause of the 'if'.)");
    else
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(BelievedImpossible),
            "'otherwise' must be the last clause",
            "but it seems not to be.");

§6.7. This shouldn't happen because the switch construct requires Python syntax and the structure of that was checked at indentation time, but just in case.

Issue a problem for the default case not occurring last6.7 =

    StandardProblems::sentence_problem(Task::syntax_tree(), _p_(BelievedImpossible),
        "'otherwise' must be the last clause",
        "which must be wrong.");

§5.9. The tree is now known to be correctly structured, and there are no possible problem messages left to issue. It's therefore safe to begin rearranging it. We'll first eliminate one whole construction: "otherwise if whatever: ..." can now become "otherwise: if whatever: ...".

(g) Optimise out the otherwise if nodes5.9 =

    int n = problem_count;
    RuleSubtrees::purge_otherwise_if(routine_node->down);
    if (problem_count > n) LOG("Local parse tree: $T\n", routine_node);

§5.10. We made a similar manoeuvre above, but for one-line "otherwise do something" phrases following one-line "if", not for the wider case of "otherwise if". We didn't handle this back then because to do so would have made it impossible to issue good problem messages for failures to use "otherwise if" correctly.

void RuleSubtrees::purge_otherwise_if(parse_node *block) {
    for (parse_node *p = block->down, *prev_p = NULL; p; prev_p = p, p = p->next) {
        if (Node::get_control_structure_used(p) == otherwise_if_CSP) {
            parse_node *former_contents = p->down;
            parse_node *former_successors = p->next;

             put an otherwise node in the position previously occupied by p
            parse_node *otherwise_node = Node::new(CODE_BLOCK_NT);
            Node::set_control_structure_used(otherwise_node, otherwise_CSP);
             extract just the word "otherwise"
            Node::set_text(otherwise_node, Wordings::one_word(Wordings::first_wn(Node::get_text(p))));
            if (prev_p) prev_p->next = otherwise_node; else block->down = otherwise_node;

             move p to below the otherwise node
            otherwise_node->down = p;
            Node::set_type(p, INVOCATION_LIST_NT);
            Node::set_control_structure_used(p, if_CSP);
            p->next = NULL;
            Node::set_text(p, Wordings::trim_first_word(Node::get_text(p)));

             put the code previously under p under a new code block node under p
            p->down = Node::new(CODE_BLOCK_NT);
            p->down->down = former_contents;

             any further "otherwise if" or "otherwise" nodes after p follow
            p->down->next = former_successors;
        }
        if (p->down) RuleSubtrees::purge_otherwise_if(p);
    }
}

§5.11. End nodes are now redundant: maybe they got here as explicit "end if" phrases in the source text, or maybe they were auto-inserted by the indentation reader, but now that the structure is known to be correct they serve no further purpose. We remove them.

(h) Remove any end markers as no longer necessary5.11 =

    RuleSubtrees::purge_end_markers(routine_node->down);

§5.12.

void RuleSubtrees::purge_end_markers(parse_node *block) {
    for (parse_node *p = block->down, *prev_p = NULL; p; prev_p = p, p = p->next) {
        if (Node::get_end_control_structure_used(p)) {
            if (prev_p) prev_p->next = p->next; else block->down = p->next;
        }
        if (p->down) RuleSubtrees::purge_end_markers(p);
    }
}

§5.13. The "begin" keyword at the end of control constructs in the old-style syntax can now be removed, too.

(i) Remove any begin markers as no longer necessary5.13 =

    RuleSubtrees::purge_begin_markers(routine_node->down);

§5.14.

void RuleSubtrees::purge_begin_markers(parse_node *block) {
    for (parse_node *p = block->down, *prev_p = NULL; p; prev_p = p, p = p->next) {
        if (Node::get_control_structure_used(p))
            if (<phrase-beginning-block>(Node::get_text(p)))
                Node::set_text(p, GET_RW(<phrase-beginning-block>, 1));
        if (p->down) RuleSubtrees::purge_begin_markers(p);
    }
}

§5.15. This all makes a nice tree, but it has the defect that the statements heading block-opening phrases (the ifs, whiles, repeats) have child nodes (the blocks of code consequent on them). We want them to be leaves for now, so that we can append statement-parsing data underneath them later. So we insert blank code block nodes to mark these phrases, and transfer the control structure annotations to them.

(j) Insert code block nodes so that nodes needing to be parsed are childless5.15 =

    RuleSubtrees::insert_cb_nodes(routine_node->down);

§5.16.

void RuleSubtrees::insert_cb_nodes(parse_node *block) {
    for (parse_node *p = block->down, *prev_p = NULL; p; prev_p = p, p = p->next) {
        if (ControlStructures::opens_block(Node::get_control_structure_used(p))) {
            parse_node *blank_cb_node = Node::new(CODE_BLOCK_NT);
            Node::set_control_structure_used(blank_cb_node,
                Node::get_control_structure_used(p));
            Node::set_control_structure_used(p, NULL);
            blank_cb_node->down = p;
            blank_cb_node->next = p->next;
            p->next = p->down;
            p->down = NULL;
            if (prev_p) prev_p->next = blank_cb_node; else block->down = blank_cb_node;
            p = blank_cb_node;
        }
        if (p->down) RuleSubtrees::insert_cb_nodes(p);
    }
}

§5.17. Now:

(k) Insert instead marker nodes5.17 =

    RuleSubtrees::read_instead_markers(routine_node->down);

§5.18.

void RuleSubtrees::read_instead_markers(parse_node *block) {
    for (parse_node *p = block->down, *prev_p = NULL; p; prev_p = p, p = p->next) {
        if (<instead-keyword>(Node::get_text(p))) {
            Node::set_text(p, GET_RW(<instead-keyword>, 1));
            parse_node *instead_node = Node::new(CODE_BLOCK_NT);
            Node::set_control_structure_used(instead_node, instead_CSP);
            instead_node->next = p->next;
            p->next = instead_node;
        }
        if (p->down) RuleSubtrees::read_instead_markers(p);
    }
}

§5.19. Now:

(l) Break up say phrases5.19 =

    RuleSubtrees::break_up_says(routine_node->down);

§7.

void RuleSubtrees::break_up_says(parse_node *block) {
    for (parse_node *p = block->down, *prev_p = NULL; p; prev_p = p, p = p->next) {
        int sf = NO_SIGF;
        wording W = Node::get_text(p);
        if (Annotations::read_int(p, from_text_substitution_ANNOT)) sf = SAY_SIGF;
        else if (<other-significant-phrase>(W)) {
            sf = <<r>>; W = GET_RW(<other-significant-phrase>, 1);
        }
        switch (sf) {
            case SAY_SIGF: {
                parse_node *blank_cb_node = Node::new(CODE_BLOCK_NT);
                Node::set_control_structure_used(blank_cb_node, say_CSP);
                blank_cb_node->next = p->next;
                Node::set_text(blank_cb_node, Node::get_text(p));
                p->next = NULL;
                if (prev_p) prev_p->next = blank_cb_node; else block->down = blank_cb_node;

                current_sentence = p;
                RuleSubtrees::unroll_says(blank_cb_node, W, 0);
                p = blank_cb_node;
                break;
            }
            case NOW_SIGF: {
                Node::set_control_structure_used(p, now_CSP);
                parse_node *cond_node = Node::new(CONDITION_CONTEXT_NT);
                Node::set_text(cond_node, W);
                p->down = cond_node;
                break;
            }
        }
        if (p->down) RuleSubtrees::break_up_says(p);
    }
}

void RuleSubtrees::unroll_says(parse_node *cb_node, wording W, int depth) {
    while (<phrase-with-comma-notation>(W)) {
        wording AW = GET_RW(<phrase-with-comma-notation>, 1);
        wording BW = GET_RW(<phrase-with-comma-notation>, 2);
        W = AW;
        Bite off a say term7.1;
        W = BW;
    }
    Bite off a say term7.1;
}

§7.1. Bite off a say term7.1 =

    if ((Wordings::length(W) > 1) || (Wide::cmp(Lexer::word_text(Wordings::first_wn(W)), L"\"\"") != 0)) {
        if ((Wordings::length(W) == 1) && (Vocabulary::test_flags(Wordings::first_wn(W), TEXTWITHSUBS_MC)) && (depth == 0)) {
            wchar_t *p = Lexer::word_raw_text(Wordings::first_wn(W));
            Check that substitution does not contain suspicious punctuation7.1.1;
            wording A = Feeds::feed_C_string_expanding_strings(p);
            if (<verify-expanded-text-substitution>(A))
                RuleSubtrees::unroll_says(cb_node, A, depth+1);
        } else {
            parse_node *say_term_node = Node::new(INVOCATION_LIST_SAY_NT);
            Node::set_text(say_term_node, W);
            SyntaxTree::graft(Task::syntax_tree(), say_term_node, cb_node);
        }
    }

§7.1.1. Check that substitution does not contain suspicious punctuation7.1.1 =

    int k, sqb = 0;
    for (k=0; p[k]; k++) {
        switch (p[k]) {
            case '[': sqb++; if (sqb > 1) Issue problem message for nested substitution7.1.1.2; break;
            case ']': sqb--; if (sqb < 0) Issue problem message for unopened substitution7.1.1.4; break;
            case ':': if ((k>0) && (Characters::isdigit(p[k-1])) && (Characters::isdigit(p[k+1]))) break;
            case ';':
                if (sqb > 0) Issue PM_TSWithPunctuation problem7.1.1.6;
                break;
            case ',':
                if (sqb > 0) Issue problem message for comma in a substitution7.1.1.1;
                break;
        }
    }
    if (sqb != 0) Issue problem message for unclosed substitution7.1.1.3;

§7.1.1.1. And the more specialised:

Issue problem message for comma in a substitution7.1.1.1 =

    Strings::TextSubstitutions::it_is_not_worth_adding();
    StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_TSWithComma),
        "a substitution contains a comma ','",
        "which is against the rules, because 'say' is a special phrase in "
        "which the comma divides items in a list of things to say, and so it "
        "loses its ordinary meanings. Because of this, no text substitution "
        "can contain a comma. "
        "(If you're trying to use a value produced by a phrase with a phrase "
        "option - say 'the best route from A to B, using even locked doors' - "
        "you'll need to put this in a 'let' variable first and then say that, "
        "or else define a better text substitution to do the job for you.)");
    Strings::TextSubstitutions::it_is_worth_adding();
    return;

§7.1.1.2. Issue problem message for nested substitution7.1.1.2 =

    Strings::TextSubstitutions::it_is_not_worth_adding();
    if ((p[k+1] == 'u') && (p[k+2] == 'n') && (p[k+3] == 'i') && (p[k+4] == 'c') &&
        (p[k+5] == 'o') && (p[k+6] == 'd') && (p[k+7] == 'e') && (p[k+8] == ' ')) {
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_NestedUSubstitution),
            "the text here contains one substitution '[...]' inside another",
            "which is not allowed. Actually, it looks as if you might have got "
            "into this by typing an exotic character as part of the name of a "
            "text substitution - those get rewritten automatically as '[unicode N]' "
            "for the appropriate Unicode character code number N. Either way - "
            "this isn't allowed.");
    } else {
        StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_NestedSubstitution),
            "the text here contains one substitution '[...]' inside another",
            "which is not allowed. (If you just wanted a literal open and closed "
            "square bracket, use '[bracket]' and '[close bracket]'.)");
    }
    Strings::TextSubstitutions::it_is_worth_adding();
    return;

§7.1.1.3. Issue problem message for unclosed substitution7.1.1.3 =

    Strings::TextSubstitutions::it_is_not_worth_adding();
    StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_UnclosedSubstitution),
        "the text here uses an open square bracket '[', which opens a substitution "
        "in the text, but doesn't close it again",
        "so that the result is malformed. (If you just wanted a literal open "
        "square bracket, use '[bracket]'.)");
    Strings::TextSubstitutions::it_is_worth_adding();
    return;

§7.1.1.4. Issue problem message for unopened substitution7.1.1.4 =

    Strings::TextSubstitutions::it_is_not_worth_adding();
    StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_UnopenedSubstitution),
        "the text here uses a close square bracket ']', which closes a substitution "
        "in the text, but never actually opened it",
        "with a matching '['. (If you just wanted a literal close square bracket, "
        "use '[close bracket]'.)");
    Strings::TextSubstitutions::it_is_worth_adding();
    return;

§7.1.1.5. Something devious happens when production (b) of <s-say-phrase> is matched. Double-quoted text is literal if it contains no square brackets, but is expanded if it includes text substitutions in squares. When (b) matches, Inform expands a text such as

"Look, [the noun] said."

into:

"Look, ", the noun, " said."

and then re-parses the result with the following nonterminal; note that we make sure commas are used correctly before handing back to <s-say-phrase> to parse the list.

<verify-expanded-text-substitution> ::=
    *** . *** |    ==> Issue PM_TSWithPunctuation problem7.1.1.6; ==> { fail }
    , *** |        ==> Issue PM_EmptySubstitution problem7.1.1.5.1; ==> { fail }
    *** , |        ==> Issue PM_EmptySubstitution problem7.1.1.5.1; ==> { fail }
    *** , , ***	|  ==> Issue PM_EmptySubstitution problem7.1.1.5.1; ==> { fail }
    ...

§7.1.1.6. So now just the problem messages:

Issue PM_TSWithPunctuation problem7.1.1.6 =

    Strings::TextSubstitutions::it_is_not_worth_adding();
    StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_TSWithPunctuation),
        "a substitution contains a '.', ':' or ';'",
        "which suggests that a close square bracket ']' may have gone astray.");
    Strings::TextSubstitutions::it_is_worth_adding();

§7.1.1.5.1. And:

Issue PM_EmptySubstitution problem7.1.1.5.1 =

    Strings::TextSubstitutions::it_is_not_worth_adding();
    StandardProblems::sentence_problem(Task::syntax_tree(), _p_(PM_EmptySubstitution),
        "the text here contains an empty substitution '[]'",
        "which is not allowed. To say nothing - well, say nothing.");
    Strings::TextSubstitutions::it_is_worth_adding();

§8. That just leaves one utility routine, for manufacturing end nodes which match a given begin node.

parse_node *RuleSubtrees::end_node(parse_node *opening) {
    parse_node *implicit_end = Node::new(INVOCATION_LIST_NT);
    Node::set_end_control_structure_used(implicit_end,
        Node::get_control_structure_used(opening));
    Annotations::write_int(implicit_end, indentation_level_ANNOT,
        Annotations::read_int(opening, indentation_level_ANNOT));
    return implicit_end;
}