• §1. Definition
• §2. Anatomy of a schema
• §15. Token insertion
• §16. Logging
• §17. Lint
• §18. I6S
• §19. Heads and tails
• §20. Compiler
• §20.1. Stage 1
• §20.3. Stage 2
• §32. Operators in I6
• §33. Metadata on inter primitives

§1. Definition. An inter schema is a set of instructions for compiling inter code, while interspersing certain values at given places.

The main use of this is for compiling invocations of inline-defined phrases. For example, the phrase definition:

To adjust (N - a number): (- AdjustThis({N}, 1); -).

results in an inter schema being compiled from this text:

    AdjustThis({N}, 1);

The notation here is essentially Inform 6 code with a few special features in braces: here, {N}, which expands to the value of the token N mentioned in the phrase preamble. The notation being basically I6 was very convenient in the years 2004-17, when Inform generated only I6 code: it became more problematic in 2018. The task then was to generate inter instructions, even though those would then usually be converted back into I6 anyway.

And so the code in this section was written. The first half amounts to a miniature Inform 6 compiler: its task is to take text such as

    AdjustThis({N}, 1);

and convert that into an inter schema. The second half is a code generator, which takes a given schema and generates inter code from it.

§2. Anatomy of a schema. Relatively few inter schemas are generated during Inform's run (typically a few hundred), so they do not need to be stored compactly or compiled quickly.

typedef struct inter_schema {
    struct text_stream *converted_from;  copy of the source notation
    struct inter_schema_node *node_tree;  the structure
    int mid_case;  does this seem to be used inside a switch case?
    int dereference_mode;  emit from this in dereference-pointers mode
    CLASS_DEFINITION
} inter_schema;

• The structure inter_schema is accessed in 2/eis and here.

§3.

inter_schema *InterSchemas::new?Usage of InterSchemas::new:
§20(text_stream *source) {
    inter_schema *sch = CREATE(inter_schema);
    sch->converted_from = Str::duplicate(source);
    sch->node_tree = NULL;
    sch->mid_case = FALSE;
    sch->dereference_mode = FALSE;
    return sch;
}

§4. A schema is a connected tree of nodes, each of which has one of the following types. (Tempting to mark eliminated nodes as OH_NO_IT_ISNT, but the need to do this went away.)

enum EXPRESSION_ISNT from 1
enum CODE_ISNT
enum EVAL_ISNT
enum STATEMENT_ISNT
enum OPERATION_ISNT
enum SUBEXPRESSION_ISNT
enum DIRECTIVE_ISNT
enum ASSEMBLY_ISNT
enum LABEL_ISNT
enum CALL_ISNT
enum MESSAGE_ISNT
enum CALLMESSAGE_ISNT

typedef struct inter_schema_node {
    struct inter_schema *parent_schema;

    struct inter_schema_node *parent_node;
    struct inter_schema_node *child_node;
    struct inter_schema_node *next_node;
    int node_marked;                                 used fleetingly during traverses

    int isn_type;                                    one of the *_ISNT values
    inter_ti isn_clarifier;                          for STATEMENT_ISNT and OPERATION_ISNT only
    int dir_clarifier;                               for DIRECTIVE_ISNT only
    struct inter_schema_token *expression_tokens;    for EXPRESSION_ISNT only

    int semicolon_terminated;                        for EXPRESSION_ISNT only
    int unclosed;                                    for CODE_ISNT only
    int unopened;                                    for CODE_ISNT only

    int blocked_by_conditional;                      used in code generation

    CLASS_DEFINITION
} inter_schema_node;

• The structure inter_schema_node is accessed in 2/eis and here.

§5.

inter_schema_node *InterSchemas::new_node?Usage of InterSchemas::new_node:
§7, §15, §22, §23, §24, §25, §26, §27, §28, §28.1.1, §29(inter_schema *sch, int isnt) {
    inter_schema_node *isn = CREATE(inter_schema_node);
    isn->parent_schema = sch;

    isn->parent_node = NULL;
    isn->child_node = NULL;
    isn->next_node = NULL;
    isn->node_marked = FALSE;

    isn->isn_type = isnt;

    isn->expression_tokens = NULL;
    isn->isn_clarifier = 0;
    isn->dir_clarifier = -1;

    isn->semicolon_terminated = FALSE;
    isn->unclosed = FALSE;
    isn->unopened = FALSE;
    isn->blocked_by_conditional = FALSE;

    return isn;
}

§6. Ordinarily, a CODE_ISNT node represents a complete block of I6 code. For example, in

    if (x == 1) { print "Hello; }

the print statement occurs inside a complete block, which will eventually be represented as a CODE_ISNT. But some inline phrase definitions leave blocks half-open. For example,

    if (x == 1)

is a legal phrase definition: we read it as

    if (x == 1) {

and the schema has to contain a CODE_ISNT marked as having been left unclosed. Similarly, some definitions close an I6 block which is assumed as having been opened by an earlier phrase invocation.

Here we mark a code node as being unclosed. Clearly, if a node is unclosed then any parent code nodes of it must also be unclosed, so we recurse upwards.

void InterSchemas::mark_unclosed?Usage of InterSchemas::mark_unclosed:
§22, §24, §28(inter_schema_node *isn) {
    while (isn) {
        if (isn->isn_type == CODE_ISNT) isn->unclosed = TRUE;
        isn = isn->parent_node;
    }
}

§7. The situation with unopened nodes is different: we must ensure that a single code node is left unopened, even if we have to create a code node at the top of the tree to serve that role.

void InterSchemas::mark_unopened?Usage of InterSchemas::mark_unopened:
§22(inter_schema_node *isn) {
    while (isn) {
        if (isn->isn_type == CODE_ISNT) {
            isn->unopened = TRUE;
            return;
        }
        if (isn->parent_node == NULL) {
            inter_schema *sch = isn->parent_schema;
            inter_schema_node *top = sch->node_tree;
            inter_schema_node *code_isn = InterSchemas::new_node(isn->parent_schema, CODE_ISNT);
            code_isn->child_node = top;
            sch->node_tree = code_isn;
            for (inter_schema_node *n = top; n; n = n->next_node)
                n->parent_node = code_isn;
            code_isn->unopened = TRUE;
            return;
        }
        isn = isn->parent_node;
    }
}

§8. A further complication is that some schemas arise from definitions used only in the middle of switch statements; they contain neither the start nor the end of the construct, so the above mechanisms are not sufficient. For example, the inline definition

    {X}:

implies, by use of the colon, that it's a switch case. We can't conveniently associate this with any single node, so we mark the schema as a whole.

void InterSchemas::mark_case_closed?Usage of InterSchemas::mark_case_closed:
§24(inter_schema_node *isn) {
    if (isn) isn->parent_schema->mid_case = TRUE;
}

§9. EXPRESSION_ISNT nodes carry a linked list of tokens with them. When we begin compiling a schema, we form a single expression node with a long list of tokens: gradually this is transformed into a tree of nodes with more, but much shorter, expression nodes, and in the process most of the token types are eliminated. The following types are present only during the compilation process, and never survive into the final schema:

enum RAW_ISTT from 1          something unidentified as yet
enum WHITE_SPACE_ISTT             a stretch of white space
enum RESERVED_ISTT            am I6 reserved word such as while
enum OPERATOR_ISTT            an I6 operator such as --> or +
enum DIVIDER_ISTT                 a semicolon used to divide I6 statements
enum OPEN_ROUND_ISTT          open round bracket
enum CLOSE_ROUND_ISTT             close round bracket
enum OPEN_BRACE_ISTT          open brace bracket
enum CLOSE_BRACE_ISTT             close brace bracket
enum COMMA_ISTT               comma
enum COLON_ISTT               colon

§10. Whereas these token types do make it into the compiled schema:

enum IDENTIFIER_ISTT          an I6 identifier such as my_var12
enum OPCODE_ISTT              an Inform assembly language opcode such as @pull
enum DIRECTIVE_ISTT           an Inform compiler directive such as #iftrue
enum NUMBER_ISTT              a constant number
enum BIN_NUMBER_ISTT          a constant number
enum HEX_NUMBER_ISTT          a constant number
enum REAL_NUMBER_ISTT             a constant number
enum DQUOTED_ISTT                 a constant piece of text "like this"
enum SQUOTED_ISTT                 a constant piece of text such as 'x'
enum I7_ISTT                  I7 material in (+ ... +) notation
enum INLINE_ISTT              an inline command such as {-my:1}
enum ASM_ARROW_ISTT           the arrow sign -> used in assembly language only
enum ASM_SP_ISTT              the stack pointer pseudo-variable sp
enum ASM_LABEL_ISTT           the label sign ? used in assembly language only
enum ASM_NEGATED_LABEL_ISTT    the label sign ?~ used in assembly language only

typedef struct inter_schema_token {
    struct inter_schema_node *owner;             these form a linked list attached to the owner node
    struct inter_schema_token *next;

    int ist_type;                                one of the *_ISTT values above
    struct text_stream *material;                textual form of token

    inter_ti operation_primitive;                OPERATOR_ISTT only: e.g. PLUS_BIP for +
    int reserved_word;                           RESERVED_ISTT only: which one
    int constant_number;                         NUMBER_ISTT only: if non-negative, value of number
    struct inter_name *as_quoted;                IDENTIFIER_ISTT only: the identified symbol if known
    int inline_command;                          INLINE_ISTT only: one of the *_ISINC values
    int inline_modifiers;
    int inline_subcommand;                       INLINE_ISTT only: one of the *_ISINSC values
    struct text_stream *bracing;
    struct text_stream *command;
    struct text_stream *operand;
    struct text_stream *operand2;
    #ifdef CORE_MODULE
    struct property *extremal_property;
    int extremal_property_sign;
    #endif

    int preinsert;                               fleeting markers only
    int postinsert;

    CLASS_DEFINITION
} inter_schema_token;

• The structure inter_schema_token is accessed in 2/eis and here.

§11.

inter_schema_token *InterSchemas::new_token?Usage of InterSchemas::new_token:
§20.2.2, §20.2.3.1, §20.2.4.1, §20.2.5, §20.2.2.1.4, §21, §26, §29(int type, text_stream *material, inter_ti operation_primitive, int reserved_word, int n) {
    inter_schema_token *t = CREATE(inter_schema_token);
    t->ist_type = type;
    t->material = Str::duplicate(material);
    t->bracing = NULL;
    t->command = NULL;
    t->operand = NULL;
    t->operand2 = NULL;
    #ifdef CORE_MODULE
    t->extremal_property = NULL;  that is, none given
    t->extremal_property_sign = MEASURE_T_EXACTLY;  that is, none given
    #endif
    t->inline_command = no_ISINC;
    t->inline_subcommand = no_ISINSC;
    t->next = NULL;
    t->owner = NULL;
    t->operation_primitive = operation_primitive;
    t->as_quoted = NULL;
    t->reserved_word = reserved_word;
    t->constant_number = n;
    t->preinsert = FALSE;
    t->postinsert = FALSE;
    return t;
}

§12. The value of reserved_word, in a RESERVED_ISTT node, must be one of:

enum IF_I6RW from 1
enum ELSE_I6RW
enum STYLE_I6RW
enum RETURN_I6RW
enum RTRUE_I6RW
enum RFALSE_I6RW
enum FOR_I6RW
enum OBJECTLOOP_I6RW
enum WHILE_I6RW
enum DO_I6RW
enum UNTIL_I6RW
enum PRINT_I6RW
enum PRINTRET_I6RW
enum NEWLINE_I6RW
enum GIVE_I6RW
enum MOVE_I6RW
enum REMOVE_I6RW
enum JUMP_I6RW
enum SWITCH_I6RW
enum DEFAULT_I6RW
enum FONT_I6RW
enum BREAK_I6RW
enum CONTINUE_I6RW
enum QUIT_I6RW
enum RESTORE_I6RW
enum SPACES_I6RW
enum READ_I6RW
enum INVERSION_I6RW
enum IFDEF_I6RW
enum IFNDEF_I6RW
enum IFTRUE_I6RW
enum IFFALSE_I6RW
enum IFNOT_I6RW
enum ENDIF_I6RW

§13. The value of inline_command, in an INLINE_ISTT node, must be one of:

enum no_ISINC from 1
enum primitive_definition_ISINC
enum new_ISINC
enum new_list_of_ISINC
enum printing_routine_ISINC
enum next_routine_ISINC
enum previous_routine_ISINC
enum ranger_routine_ISINC
enum strong_kind_ISINC
enum weak_kind_ISINC
enum backspace_ISINC
enum erase_ISINC
enum open_brace_ISINC
enum close_brace_ISINC
enum label_ISINC
enum counter_ISINC
enum counter_storage_ISINC
enum counter_up_ISINC
enum counter_down_ISINC
enum counter_makes_array_ISINC
enum by_reference_ISINC
enum by_reference_blank_out_ISINC
enum reference_exists_ISINC
enum lvalue_by_reference_ISINC
enum by_value_ISINC
enum box_quotation_text_ISINC
enum try_action_ISINC
enum try_action_silently_ISINC
enum return_value_ISINC
enum return_value_from_rule_ISINC
enum property_holds_block_value_ISINC
enum mark_event_used_ISINC
enum my_ISINC
enum unprotect_ISINC
enum copy_ISINC
enum initialise_ISINC
enum matches_description_ISINC
enum now_matches_description_ISINC
enum arithmetic_operation_ISINC
enum say_ISINC
enum show_me_ISINC
enum segment_count_ISINC
enum final_segment_marker_ISINC
enum list_together_ISINC
enum rescale_ISINC
enum unknown_ISINC
enum substitute_ISINC
enum combine_ISINC

§14. The value of inline_subcommand, in an INLINE_ISTT node, must be one of:

enum no_ISINSC from 1
enum unarticled_ISINSC
enum articled_ISINSC
enum repeat_through_ISINSC
enum repeat_through_list_ISINSC
enum number_of_ISINSC
enum random_of_ISINSC
enum total_of_ISINSC
enum extremal_ISINSC
enum function_application_ISINSC
enum description_application_ISINSC
enum solve_equation_ISINSC
enum switch_ISINSC
enum break_ISINSC
enum verbose_checking_ISINSC

§15. Token insertion.

void InterSchemas::add_token?Usage of InterSchemas::add_token:
§20.2.2, §20.2.3.1, §20.2.4.1, §20.2.5, §20.2.2.1.4(inter_schema *sch, inter_schema_token *t) {
    if (sch->node_tree == NULL)
        sch->node_tree = InterSchemas::new_node(sch, EXPRESSION_ISNT);
    InterSchemas::add_token_to_node(sch->node_tree, t);
}

void InterSchemas::add_token_to_node?Usage of InterSchemas::add_token_to_node:
§26(inter_schema_node *isn, inter_schema_token *t) {
    if (isn->expression_tokens == NULL) isn->expression_tokens = t;
    else {
        inter_schema_token *p = isn->expression_tokens;
        while ((p) && (p->next)) p = p->next;
        p->next = t;
    }
    t->owner = isn;
}

void InterSchemas::add_token_after?Usage of InterSchemas::add_token_after:
§21, §26(inter_schema_token *t, inter_schema_token *existing) {
    if (existing == NULL) internal_error("can't add after null element");
    inter_schema_token *was = existing->next;
    existing->next = t;
    t->next = was;
    t->owner = existing->owner;
}

§16. Logging. It is invaluable to be able to see compiled schemas in the debugging log, so we go to some trouble here.

void InterSchemas::log?Usage of InterSchemas::log:
Building Module - §4.5(OUTPUT_STREAM, void *vis) {
    inter_schema *sch = (inter_schema *) vis;
    if (sch == NULL) LOG("<null schema>\n");
    else if (sch->node_tree == NULL) LOG("<schema without nodes>\n");
    else InterSchemas::log_depth(sch->node_tree, 0);
}

void InterSchemas::log_depth(inter_schema_node *isn, int depth) {
    for (; isn; isn=isn->next_node)
        InterSchemas::log_just(isn, depth);
}
void InterSchemas::log_just?Usage of InterSchemas::log_just:
Emitting Inter Schemas - §2.1(inter_schema_node *isn, int depth) {
    if (isn->blocked_by_conditional) LOG("XX"); else LOG("  ");
    for (int d = 0; d < depth; d++) LOG("    ");
    switch (isn->isn_type) {
        case STATEMENT_ISNT:
            LOG("* (statement) %S\n", Primitives::name(isn->isn_clarifier));
            break;
        case OPERATION_ISNT:
            LOG("* (operation) %S\n", Primitives::name(isn->isn_clarifier));
            break;
        case CODE_ISNT:
            LOG("* (code)");
            if (isn->unclosed) LOG(" <");
            if (isn->unopened) LOG(" >");
            LOG("\n");
            break;
        case EVAL_ISNT:
            LOG("* (eval)\n");
            break;
        case ASSEMBLY_ISNT:
            LOG("* (assembly)\n");
            break;
        case DIRECTIVE_ISNT:
            LOG("* (directive) ");
            switch(isn->dir_clarifier) {
                case IFDEF_I6RW: LOG("#ifdef"); break;
                case IFNDEF_I6RW: LOG("#ifndef"); break;
                case IFTRUE_I6RW: LOG("#iftrue"); break;
                case IFFALSE_I6RW: LOG("#iffalse"); break;
                case IFNOT_I6RW: LOG("#ifnot"); break;
                case ENDIF_I6RW: LOG("#endif"); break;
                default: LOG("<unknown>"); break;
            }
            LOG("\n");
            break;
        case LABEL_ISNT:
            LOG("* (label)\n");
            break;
        case CALL_ISNT:
            LOG("* (call)\n");
            break;
        case MESSAGE_ISNT:
            LOG("* (message)\n");
            break;
        case CALLMESSAGE_ISNT:
            LOG("* (call-message)\n");
            break;
        case SUBEXPRESSION_ISNT:
            LOG("* (subexpression)\n");
            break;
        case EXPRESSION_ISNT:
            LOG("* (expr)");
            if (isn->semicolon_terminated) LOG(" ;");
            if (isn->unclosed) LOG(" <");
            if (isn->unopened) LOG(" >");
            if (isn->expression_tokens == NULL) LOG(" - empty");
            LOG("\n");
            for (inter_schema_token *t = isn->expression_tokens; t; t=t->next) {
                for (int d = 0; d < depth + 1; d++) LOG("    ");
                InterSchemas::log_ist(t);
                if (isn != t->owner) LOG(" !!! ownership incorrect here");
                LOG("\n");
            }
            break;
    }
    InterSchemas::log_depth(isn->child_node, depth+1);
}

void InterSchemas::log_ist(inter_schema_token *t) {
    if (t == NULL) { LOG("<NULL-IST>"); return; }
    switch (t->ist_type) {
        case RAW_ISTT:          LOG("RAW         "); break;
        case OPERATOR_ISTT:     LOG("OPERATOR    "); break;
        case OPCODE_ISTT:       LOG("OPCODE      "); break;
        case DIRECTIVE_ISTT:    LOG("DIRECTIVE   "); break;
        case IDENTIFIER_ISTT:   LOG("IDENTIFIER  "); break;
        case RESERVED_ISTT:     LOG("RESERVED    "); break;
        case NUMBER_ISTT:       LOG("NUMBER      "); break;
        case BIN_NUMBER_ISTT:   LOG("BIN_NUMBER  "); break;
        case HEX_NUMBER_ISTT:   LOG("HEX_NUMBER  "); break;
        case REAL_NUMBER_ISTT:  LOG("REAL_NUMBER "); break;
        case DQUOTED_ISTT:      LOG("DQUOTED     "); break;
        case SQUOTED_ISTT:      LOG("SQUOTED     "); break;
        case WHITE_SPACE_ISTT:  LOG("WHITE_SPACE "); break;
        case DIVIDER_ISTT:      LOG("DIVIDER     "); break;
        case OPEN_ROUND_ISTT:   LOG("OPEN_ROUND  "); break;
        case CLOSE_ROUND_ISTT:  LOG("CLOSE_ROUND "); break;
        case OPEN_BRACE_ISTT:   LOG("OPEN_BRACE  "); break;
        case CLOSE_BRACE_ISTT:  LOG("CLOSE_BRACE "); break;
        case COMMA_ISTT:        LOG("COMMA       "); break;
        case COLON_ISTT:        LOG("COLON       "); break;
        case I7_ISTT:           LOG("I7          "); break;
        case INLINE_ISTT:       LOG("INLINE      "); break;
        case ASM_ARROW_ISTT:    LOG("ASM_ARROW   "); break;
        case ASM_SP_ISTT:       LOG("ASM_SP      "); break;
        case ASM_LABEL_ISTT:    LOG("ASM_LABEL   "); break;
        case ASM_NEGATED_LABEL_ISTT:    LOG("NEGASM_LABEL "); break;
        default: LOG("<unknown>"); break;
    }
    LOG("%S", t->material);
    if (t->inline_modifiers & GIVE_KIND_ID_ISSBM) LOG(" GIVE_KIND_ID");
    if (t->inline_modifiers & GIVE_COMPARISON_ROUTINE_ISSBM) LOG(" GIVE_COMPARISON_ROUTINE");
    if (t->inline_modifiers & DEREFERENCE_PROPERTY_ISSBM) LOG(" DEREFERENCE_PROPERTY");
    if (t->inline_modifiers & ADOPT_LOCAL_STACK_FRAME_ISSBM) LOG(" ADOPT_LOCAL_STACK_FRAME");
    if (t->inline_modifiers & CAST_TO_KIND_OF_OTHER_TERM_ISSBM) LOG(" CAST_TO_KIND_OF_OTHER_TERM");
    if (t->inline_modifiers & BY_REFERENCE_ISSBM) LOG(" BY_REFERENCE");
}

§17. Lint. As can be seen, the inter_schema structure is quite complicated, and there are numerous invariants it has to satisfy. As a precaution, then, we check that all of these invariants hold before shipping out a compiled schema. This is where the check is done:

void InterSchemas::lint?Usage of InterSchemas::lint:
§20(inter_schema *sch) {
    if (sch) {
        text_stream *err = InterSchemas::lint_isn(sch->node_tree, 0);
        if (err) {
            LOG("Lint fail: %S\n$1\n", err, sch);
            internal_error("inter schema failed lint");
        }
    }
}

text_stream *InterSchemas::lint_isn(inter_schema_node *isn, int depth) {
    for (; isn; isn=isn->next_node) {
        if (isn->isn_type == EXPRESSION_ISNT) {
            int asm = FALSE;
            for (inter_schema_token *t = isn->expression_tokens; t; t=t->next) {
                switch (t->ist_type) {
                    case OPCODE_ISTT:       if (t == isn->expression_tokens) asm = TRUE;
                                            else return I"contains loose assembler";
                                            break;
                    case RAW_ISTT:          return I"contains raw node";
                    case OPEN_BRACE_ISTT:   return I"contains open-brace node";
                    case CLOSE_BRACE_ISTT:  return I"contains close-brace node";
                    case OPEN_ROUND_ISTT:   return I"contains open-round node";
                    case CLOSE_ROUND_ISTT:  return I"contains close-round node";
                    case COMMA_ISTT:        return I"contains comma node";
                    case DIVIDER_ISTT:      return I"contains divider node";
                    case RESERVED_ISTT:     return I"contains reserved word node";
                    case COLON_ISTT:        return I"contains colon node";
                    case OPERATOR_ISTT:     return I"contains operator node";
                }
                if ((t->ist_type == INLINE_ISTT) && (t->inline_command == open_brace_ISINC))
                    return I"contains manual open-brace";
                if ((t->ist_type == INLINE_ISTT) && (t->inline_command == close_brace_ISINC))
                    return I"contains manual close-brace";
                if ((t->ist_type == NUMBER_ISTT) && (t->next) && (t->next->ist_type == NUMBER_ISTT) && (asm == FALSE))
                    return I"two consecutive numbers";
                if (isn != t->owner)
                    return I"ownership linkage broken";
            }
            if (isn->child_node) return I"expression has child nodes";
        } else {
            if (isn->expression_tokens) return I"non-expression has elements";
        }
        if ((isn->child_node) && (isn->child_node->parent_node != isn))
            return I"child-parent linkage broken";
        if ((isn->isn_clarifier) && (isn->expression_tokens))
            return I"ambiguous is-node";
        text_stream *R = InterSchemas::lint_isn(isn->child_node, depth+1);
        if (R) return R;
    }
    return NULL;
}

§18. I6S.

dictionary *compiled_i6s_dict = NULL;

inter_schema *InterSchemas::from_i6s(text_stream *prototype, int no_quoted_inames, void **quoted_inames) {
    if (compiled_i6s_dict == NULL) {
        compiled_i6s_dict = Dictionaries::new(512, FALSE);
    }
    dict_entry *de = Dictionaries::find(compiled_i6s_dict, prototype);
    if (de) return (inter_schema *) Dictionaries::read_value(compiled_i6s_dict, prototype);

    inter_schema *result = InterSchemas::from_text(prototype, TRUE, no_quoted_inames, quoted_inames);

    Dictionaries::create(compiled_i6s_dict, prototype);
    Dictionaries::write_value(compiled_i6s_dict, prototype, (void *) result);

    return result;
}

§19. Heads and tails. Most inline-defined phrases compile to a single schema, but some compile to two, the head and the tail (which typically bookend a loop structure). A general inline definition must therefore in principle be compiled to two schemas, not one.

The definition is in a wide C string because it's coming raw from the lexer, as the content of a (- ... -) lexeme, but with the (- and -) removed.

void InterSchemas::from_inline_phrase_definition(wchar_t *p, inter_schema **head, inter_schema **tail) {
    *head = NULL; *tail = NULL;

    text_stream *head_defn = Str::new();
    text_stream *tail_defn = Str::new();
    Fetch the head and tail definitions19.1;

    *head = InterSchemas::from_text(head_defn, FALSE, 0, NULL);
    if (Str::len(tail_defn) > 0)
        *tail = InterSchemas::from_text(tail_defn, FALSE, 0, NULL);
}

§19.1. A tail will only be present if the definition contains {-block}. If it does, we then split the definition into a head and a tail, and again trim white space from each. Note that {-block} is not legal anywhere else.

For example:

To repeat with a King's Court begin -- end loop:

could be given the definition:

    @push {-my:trcount};
    for (trcount=1; trcount<=3; trcount++)
        {-block}
    @pull trcount;

This then repeats what it's given three times, while guaranteeing that the counter is always a local variable called trcount, and that no matter how such operations are nested, they will work. We might then write:

    To say iteration: (- print {-my:trcount}; -).

and then this will work as might be hoped:

    repeat with a King's Court:
        say "[iteration]...";
            repeat with a King's Court:
                say "[iteration]. You play a Shanty Town, getting +2 Actions.";

This is a slightly contrived example, and often {-block} isn't needed. If we didn't care about accessing the iteration count in the body of the loop, for instance, we could simply have defined:

    for ({-my:1}=1; {-my:1}<=3; {-my:1}++)

and Inform would then have allocated a new variable as loop counter each time.

Fetch the head and tail definitions19.1 =

    while (Characters::is_whitespace(*p)) p++;
    WRITE_TO(head_defn, "%w", p);
    int effective_end = 0;
    for (int i=0, L=Str::len(head_defn); i<L; i++)
        if (!(Characters::is_whitespace(Str::get_at(head_defn, i))))
            effective_end = i+1;
    Str::truncate(head_defn, effective_end);

    for (int i=0, L=Str::len(head_defn); i<L; i++)
        if (Str::includes_wide_string_at(head_defn, L"{-block}", i)) {
            int after = i+8, before = i;
            while (Characters::is_whitespace(Str::get_at(head_defn, after))) after++;
            while (Characters::is_whitespace(Str::get_at(head_defn, before-1))) before--;
            Str::copy_tail(tail_defn, head_defn, after);
            Str::truncate(head_defn, before);
            break;
        }

• This code is used in §19.

§20. Compiler. This is a two-stage process.

inter_schema *InterSchemas::from_text?Usage of InterSchemas::from_text:
§18, §19(text_stream *from, int abbreviated, int no_quoted_inames, void **quoted_inames) {
    inter_schema *sch = InterSchemas::new(from);

    if ((Log::aspect_switched_on(SCHEMA_COMPILATION_DA)) ||
        (Log::aspect_switched_on(SCHEMA_COMPILATION_DETAILS_DA)))
        LOG("\n\n------------\nCompiling inter schema from: <%S>\n", from);

    Begin the schema as a single expression node with a linked list of tokens20.2;
    Perform transformations to grow the tree and reduce the token count20.4;

    InterSchemas::lint(sch);

    if ((Log::aspect_switched_on(SCHEMA_COMPILATION_DA)) ||
        (Log::aspect_switched_on(SCHEMA_COMPILATION_DETAILS_DA)))
        LOG("Completed inter schema:\n$1", sch);
    return sch;
}

§20.1. Stage 1. Our method is to tokenise the source code as if it were Inform 6, but to look out for the two extra syntaxes allowed, {-bracing} and (+ Inform 7 interpolation +).

enum NO_TOKSTATE from 1
enum COMMENT_TOKSTATE             currently scanning an I6 comment ! ...
enum DQUOTED_TOKSTATE
enum SQUOTED_TOKSTATE
enum WHITE_TOKSTATE
enum TOK_TOKSTATE                 an actual token

§20.2. Begin the schema as a single expression node with a linked list of tokens20.2 =

    inter_schema_token *preceding_token = NULL;

    int definition_length = Str::len(from);
    text_stream *current_raw = Str::new();
    int tokeniser_state = NO_TOKSTATE;
    int pos = 0;
    if ((abbreviated) && (Str::begins_with_wide_string(from, L"*=-"))) {
        sch->dereference_mode = TRUE; pos = 3;
    }
    for (; pos<definition_length; pos++) {
        int c = Str::get_at(from, pos);
        if (Characters::is_whitespace(c)) {
            if ((tokeniser_state == TOK_TOKSTATE) || (tokeniser_state == NO_TOKSTATE)) {
                Absorb raw material, if any20.2.2;
                tokeniser_state = WHITE_TOKSTATE;
                PUT_TO(current_raw, ' ');
            }
        } else {
            if (tokeniser_state == WHITE_TOKSTATE) {
                Absorb raw material, if any20.2.2;
                tokeniser_state = NO_TOKSTATE;
            }
        }

        switch (tokeniser_state) {
             case DQUOTED_TOKSTATE:
                if (c == '"') {
                    Absorb raw material, if any20.2.2;
                    tokeniser_state = NO_TOKSTATE;
                } else {
                    PUT_TO(current_raw, c);
                }
                break;
             case SQUOTED_TOKSTATE:
                if (c == '\'') {
                    Absorb raw material, if any20.2.2;
                    tokeniser_state = NO_TOKSTATE;
                } else {
                    PUT_TO(current_raw, c);
                }
                break;
             case COMMENT_TOKSTATE:
                if (c == '\n') tokeniser_state = NO_TOKSTATE;
                break;
             case WHITE_TOKSTATE: break;
             default:
                if (c == '!') {
                    Absorb raw material, if any20.2.2;
                    tokeniser_state = COMMENT_TOKSTATE; break;
                }
                if (c == '"') {
                    Absorb raw material, if any20.2.2;
                    tokeniser_state = DQUOTED_TOKSTATE; break;
                }
                if (c == '\'') {
                    Absorb raw material, if any20.2.2;
                    tokeniser_state = SQUOTED_TOKSTATE; break;
                }
                if ((c == '{') && (abbreviated == FALSE))
                    Look for a possible bracing20.2.4
                else if ((c == '*') && (abbreviated == TRUE))
                    Look for a possible abbreviated command20.2.5
                else if ((c == '(') && (Str::get_at(from, pos+1) == '+') && (abbreviated == FALSE))
                    Look for a possible Inform 7 fragment20.2.3
                else Absorb a raw character20.2.1;
                break;
        }
    }
    Absorb raw material, if any20.2.2;

• This code is used in §20.

§20.2.1. Absorb a raw character20.2.1 =

    tokeniser_state = TOK_TOKSTATE;
    PUT_TO(current_raw, c);

• This code is used in §20.2, §20.2.3, §20.2.4, §20.2.5 (twice).

§20.2.2. Absorb raw material, if any20.2.2 =

    if (Str::len(current_raw)) {
        switch (tokeniser_state) {
            case WHITE_TOKSTATE:
                InterSchemas::add_token(sch, InterSchemas::new_token(WHITE_SPACE_ISTT, I" ", 0, 0, -1));
                break;
            case DQUOTED_TOKSTATE:
                InterSchemas::de_escape_text(current_raw);
                InterSchemas::add_token(sch, InterSchemas::new_token(DQUOTED_ISTT, current_raw, 0, 0, -1));
                break;
            case SQUOTED_TOKSTATE:
                InterSchemas::add_token(sch, InterSchemas::new_token(SQUOTED_ISTT, current_raw, 0, 0, -1));
                break;
            default:
                Look for individual tokens20.2.2.1;
                break;
        }
        Str::clear(current_raw);
    }
    tokeniser_state = NO_TOKSTATE;

• This code is used in §20.2 (8 times), §20.2.3, §20.2.4, §20.2.5.

§20.2.3. Material in (+ ... +) notation is an interpolation of I7 source text.

Look for a possible Inform 7 fragment20.2.3 =

    int save_pos = pos, accept = FALSE;
    TEMPORARY_TEXT(source_text_fragment)
    pos += 2;
    while (Str::get_at(from, pos)) {
        if ((Str::get_at(from, pos-1) == '+') && (Str::get_at(from, pos) == ')')) {
            Str::delete_last_character(source_text_fragment);
            accept = TRUE; break;
        }
        PUT_TO(source_text_fragment, Str::get_at(from, pos++));
    }
    if (accept) {
        Absorb raw material, if any20.2.2;
        Expand a fragment of Inform 7 text20.2.3.1;
    } else { int c = '('; Absorb a raw character20.2.1; pos = save_pos; }
    DISCARD_TEXT(source_text_fragment)

• This code is used in §20.2.

§20.2.3.1. The empty I7 interpolation is legal, but produces no result.

Expand a fragment of Inform 7 text20.2.3.1 =

    if (Str::len(source_text_fragment) > 0) {
        InterSchemas::add_token(sch, InterSchemas::new_token(I7_ISTT, source_text_fragment, 0, 0, -1));
    }

• This code is used in §20.2.3.

§20.2.4. Material in braces sometimes indicates an inline command, but not always, because braces often occur innocently in I6 code. So we require the first character after the open-brace not to be white-space, and also not to be a pipe (though I've forgotten why). The text inside the braces is called a "bracing".

Look for a possible bracing20.2.4 =

    int save_pos = pos++, accept = FALSE;
    TEMPORARY_TEXT(bracing)
    while (TRUE) {
        int c = Str::get_at(from, pos);
        if (c == 0) break;
        if (c == '}') { accept = TRUE; break; }
        PUT_TO(bracing, c);
        pos++;
    }
    int first = Str::get_first_char(bracing);
    if ((accept) && (first != ' ') && (first != '\t') && (first != '\n') && (first != '|')) {
        Absorb raw material, if any20.2.2;
        Parse a bracing into an inline command20.2.4.1;
    } else { int c = '{'; Absorb a raw character20.2.1; pos = save_pos; }
    DISCARD_TEXT(bracing)

• This code is used in §20.2.

§20.2.4.1. That's everything, then, except the one thing that counts: how to expand a bracing.

Parse a bracing into an inline command20.2.4.1 =

    inter_schema_token *t = InterSchemas::new_token(INLINE_ISTT, bracing, 0, 0, -1);
    t->bracing = Str::duplicate(bracing);
    t->command = Str::new();
    t->operand = Str::new();
    t->operand2 = Str::new();
    Decompose the bracing20.2.4.1.1;
    if (Str::len(t->command) > 0) {
        if (Str::eq_wide_string(t->command, L"primitive-definition")) {
            t->inline_command = primitive_definition_ISINC;
            if (Str::eq_wide_string(t->operand, L"repeat-through")) t->inline_subcommand = repeat_through_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"repeat-through-list")) t->inline_subcommand = repeat_through_list_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"number-of")) t->inline_subcommand = number_of_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"random-of")) t->inline_subcommand = random_of_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"total-of")) t->inline_subcommand = total_of_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"extremal")) t->inline_subcommand = extremal_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"function-application")) t->inline_subcommand = function_application_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"description-application")) t->inline_subcommand = description_application_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"solve-equation")) t->inline_subcommand = solve_equation_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"switch")) t->inline_subcommand = switch_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"break")) t->inline_subcommand = break_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"verbose-checking")) t->inline_subcommand = verbose_checking_ISINSC;
        } else if (Str::eq_wide_string(t->command, L"new")) t->inline_command = new_ISINC;
        else if (Str::eq_wide_string(t->command, L"new-list-of")) t->inline_command = new_list_of_ISINC;
        else if (Str::eq_wide_string(t->command, L"printing-routine")) t->inline_command = printing_routine_ISINC;
        else if (Str::eq_wide_string(t->command, L"ranger-routine")) t->inline_command = ranger_routine_ISINC;
        else if (Str::eq_wide_string(t->command, L"next-routine")) t->inline_command = next_routine_ISINC;
        else if (Str::eq_wide_string(t->command, L"previous-routine")) t->inline_command = previous_routine_ISINC;
        else if (Str::eq_wide_string(t->command, L"strong-kind")) t->inline_command = strong_kind_ISINC;
        else if (Str::eq_wide_string(t->command, L"weak-kind")) t->inline_command = weak_kind_ISINC;
        else if (Str::eq_wide_string(t->command, L"backspace")) t->inline_command = backspace_ISINC;
        else if (Str::eq_wide_string(t->command, L"erase")) t->inline_command = erase_ISINC;
        else if (Str::eq_wide_string(t->command, L"open-brace")) t->inline_command = open_brace_ISINC;
        else if (Str::eq_wide_string(t->command, L"close-brace")) t->inline_command = close_brace_ISINC;
        else if (Str::eq_wide_string(t->command, L"label")) t->inline_command = label_ISINC;
        else if (Str::eq_wide_string(t->command, L"counter")) t->inline_command = counter_ISINC;
        else if (Str::eq_wide_string(t->command, L"counter-storage")) t->inline_command = counter_storage_ISINC;
        else if (Str::eq_wide_string(t->command, L"counter-up")) t->inline_command = counter_up_ISINC;
        else if (Str::eq_wide_string(t->command, L"counter-down")) t->inline_command = counter_down_ISINC;
        else if (Str::eq_wide_string(t->command, L"counter-makes-array")) t->inline_command = counter_makes_array_ISINC;
        else if (Str::eq_wide_string(t->command, L"by-reference")) t->inline_command = by_reference_ISINC;
        else if (Str::eq_wide_string(t->command, L"by-reference-blank-out")) t->inline_command = by_reference_blank_out_ISINC;
        else if (Str::eq_wide_string(t->command, L"reference-exists")) t->inline_command = reference_exists_ISINC;
        else if (Str::eq_wide_string(t->command, L"lvalue-by-reference")) t->inline_command = lvalue_by_reference_ISINC;
        else if (Str::eq_wide_string(t->command, L"by-value")) t->inline_command = by_value_ISINC;
        else if (Str::eq_wide_string(t->command, L"box-quotation-text")) t->inline_command = box_quotation_text_ISINC;
        else if (Str::eq_wide_string(t->command, L"try-action")) t->inline_command = try_action_ISINC;
        else if (Str::eq_wide_string(t->command, L"try-action-silently")) t->inline_command = try_action_silently_ISINC;
        else if (Str::eq_wide_string(t->command, L"return-value")) t->inline_command = return_value_ISINC;
        else if (Str::eq_wide_string(t->command, L"return-value-from-rule")) t->inline_command = return_value_from_rule_ISINC;
        else if (Str::eq_wide_string(t->command, L"property-holds-block-value")) t->inline_command = property_holds_block_value_ISINC;
        else if (Str::eq_wide_string(t->command, L"mark-event-used")) t->inline_command = mark_event_used_ISINC;
        else if (Str::eq_wide_string(t->command, L"my")) t->inline_command = my_ISINC;
        else if (Str::eq_wide_string(t->command, L"unprotect")) t->inline_command = unprotect_ISINC;
        else if (Str::eq_wide_string(t->command, L"copy")) t->inline_command = copy_ISINC;
        else if (Str::eq_wide_string(t->command, L"initialise")) t->inline_command = initialise_ISINC;
        else if (Str::eq_wide_string(t->command, L"matches-description")) t->inline_command = matches_description_ISINC;
        else if (Str::eq_wide_string(t->command, L"now-matches-description")) t->inline_command = now_matches_description_ISINC;
        else if (Str::eq_wide_string(t->command, L"arithmetic-operation")) t->inline_command = arithmetic_operation_ISINC;
        else if (Str::eq_wide_string(t->command, L"say")) t->inline_command = say_ISINC;
        else if (Str::eq_wide_string(t->command, L"show-me")) t->inline_command = show_me_ISINC;
        else if (Str::eq_wide_string(t->command, L"segment-count")) t->inline_command = segment_count_ISINC;
        else if (Str::eq_wide_string(t->command, L"final-segment-marker")) t->inline_command = final_segment_marker_ISINC;
        else if (Str::eq_wide_string(t->command, L"list-together")) {
            t->inline_command = list_together_ISINC;
            if (Str::eq_wide_string(t->operand, L"unarticled")) t->inline_subcommand = unarticled_ISINSC;
            else if (Str::eq_wide_string(t->operand, L"articled")) t->inline_subcommand = articled_ISINSC;
        } else if (Str::eq_wide_string(t->command, L"rescale")) t->inline_command = rescale_ISINC;
        else t->inline_command = unknown_ISINC;
    }

    InterSchemas::add_token(sch, t);
    preceding_token = t;

• This code is used in §20.2.4.

§20.2.4.1.1. A bracing can take any of the following forms:

    {-command}
    {-command:operand}
    {-command:operand:operand2}
    {-command:operand<property name}
    {-command:operand>property name}
    {some text}
    {-annotation:some text}

We parse this with the command or annotation in command, the "some text" or operand in bracing, the property name (if given) in extremal_property, the direction of the < or > in extremal_property_sign, and the second, optional, operand in operand2.

Decompose the bracing20.2.4.1.1 =

    TEMPORARY_TEXT(pname)
    if (Str::get_first_char(t->bracing) == '-') {
        int portion = 1;
        for (int i=1, L = Str::len(t->bracing); i<L; i++) {
            int c = Str::get_at(t->bracing, i);
            switch(portion) {
                case 1:
                    if (c == ':') portion = 2;
                    else PUT_TO(t->command, c);
                    break;
                case 2:
                    if (c == ':') portion = 3;
                    #ifdef CORE_MODULE
                    else if (c == '<') { t->extremal_property_sign = MEASURE_T_OR_LESS; portion = 4; }
                    else if (c == '>') { t->extremal_property_sign = MEASURE_T_OR_MORE; portion = 4; }
                    #endif
                    else PUT_TO(t->operand, c);
                    break;
                case 3:
                    PUT_TO(t->operand2, c); break;
                case 4:
                    PUT_TO(pname, c); break;
            }
        }
        #ifdef CORE_MODULE
        if (t->extremal_property_sign != MEASURE_T_EXACTLY) {
            wording W = Feeds::feed_text(pname);
            if (<property-name>(W)) t->extremal_property = <<rp>>;
        }
        #endif
        Str::copy(t->bracing, t->operand);
    }
    DISCARD_TEXT(pname)

• This code is used in §20.2.4.1.

§20.2.5. In abbreviated prototypes, *1 and *2 are placeholders, but a number of modifiers are allowed. The full syntax follows:

define GIVE_KIND_ID_ISSBM                  1
define GIVE_COMPARISON_ROUTINE_ISSBM       2
define DEREFERENCE_PROPERTY_ISSBM          4
define ADOPT_LOCAL_STACK_FRAME_ISSBM       8
define CAST_TO_KIND_OF_OTHER_TERM_ISSBM        16
define BY_REFERENCE_ISSBM                  32
define LVALUE_CONTEXT_ISSBM                    64
define STORAGE_AS_FUNCTION_ISSBM            128

Look for a possible abbreviated command20.2.5 =

    int at = pos;
    wchar_t c = Str::get_at(from, ++at);
    int iss_bitmap = 0;
    switch (c) {
        case '!': internal_error("the '*!' schema notation has been abolished"); break;
        case '%': iss_bitmap = iss_bitmap | LVALUE_CONTEXT_ISSBM; c = Str::get_at(from, ++at); break;
        case '$': iss_bitmap = iss_bitmap | STORAGE_AS_FUNCTION_ISSBM; c = Str::get_at(from, ++at); break;
        case '#': iss_bitmap = iss_bitmap | GIVE_KIND_ID_ISSBM; c = Str::get_at(from, ++at); break;
        case '_': iss_bitmap = iss_bitmap | GIVE_COMPARISON_ROUTINE_ISSBM; c = Str::get_at(from, ++at); break;
        case '+': iss_bitmap = iss_bitmap | DEREFERENCE_PROPERTY_ISSBM; c = Str::get_at(from, ++at); break;
        case '|': iss_bitmap = iss_bitmap | (DEREFERENCE_PROPERTY_ISSBM + LVALUE_CONTEXT_ISSBM); c = Str::get_at(from, ++at); break;
        case '?': iss_bitmap = iss_bitmap | ADOPT_LOCAL_STACK_FRAME_ISSBM; c = Str::get_at(from, ++at); break;
        case '<': iss_bitmap = iss_bitmap | CAST_TO_KIND_OF_OTHER_TERM_ISSBM; c = Str::get_at(from, ++at); break;
        case '^': iss_bitmap = iss_bitmap | (ADOPT_LOCAL_STACK_FRAME_ISSBM + BY_REFERENCE_ISSBM); c = Str::get_at(from, ++at); break;
        case '>': iss_bitmap = iss_bitmap | BY_REFERENCE_ISSBM; c = Str::get_at(from, ++at); break;
    }
    if (Characters::isdigit(c)) {
        Absorb raw material, if any20.2.2;
        TEMPORARY_TEXT(T)
        for (int i=pos; i<=at; i++) PUT_TO(T, Str::get_at(from, i));
        inter_schema_token *t = InterSchemas::new_token(INLINE_ISTT, T, 0, 0, -1);
        t->bracing = Str::duplicate(T);
        t->inline_command = substitute_ISINC;
        t->inline_modifiers = iss_bitmap;
        t->constant_number = (int) c - (int) '1';
        InterSchemas::add_token(sch, t);
        preceding_token = t;
        DISCARD_TEXT(T)
        pos = at;
    } else if (c == '&') {
        inter_schema_token *t = InterSchemas::new_token(INLINE_ISTT, I"*&", 0, 0, -1);
        t->bracing = I"*&";
        t->inline_command = combine_ISINC;
        t->inline_modifiers = iss_bitmap;
        InterSchemas::add_token(sch, t);
        preceding_token = t;
        pos = at;
    } else if (c == '-') {
        internal_error("the '*-' schema notation has been abolished");
    } else if (c == '*') {
        int c = '*'; Absorb a raw character20.2.1;
        pos = at;
    } else {
        int c = '{'; Absorb a raw character20.2.1;
    }

• This code is used in §20.2.

§20.2.2.1. That leaves us with just the main case to handle: raw I6 code which is outside of quotation marks and commentary, and which doesn't include bracings or I7 interpolations. That might look like, for instance,

    Frog + 2*Toad(

(there is no reason to suppose that this stretch of code is complete or matches parentheses); we must tokenise it into

Frog W + W 2 * Toad (

where W indicates a white space token. What we do is scan through the text until we reach the start of a new token, and then break off what we scanned through since the last time.

Look for individual tokens20.2.2.1 =

    int L = Str::len(current_raw);
    int c_start = 0, escaped = FALSE;
    for (int p = 0; p < L; p++) {
        wchar_t c1 = Str::get_at(current_raw, p), c2 = 0, c3 = 0;
        if (p < L-1) c2 = Str::get_at(current_raw, p+1);
        if (p < L-2) c3 = Str::get_at(current_raw, p+2);

        if (escaped == FALSE) {
            if ((c1 == '$') && ((p == 0) || (Characters::isalpha(Str::get_at(current_raw, p-1)) == FALSE)))
                Break off here for real, binary or hexadecimal notation20.2.2.1.1;
            if (c1 == '-') Break off here for negative number20.2.2.1.2;
            Break off here for operators20.2.2.1.3;
        }
        if (c1 == 0x00A7) escaped = escaped?FALSE:TRUE;
    }
    if (c_start < L) {
        int x = c_start, y = L-1;
        Break off a token20.2.2.1.4;
    }

• This code is used in §20.2.2.

§20.2.2.1.1. Recall that in I6 notation, a dollar introduces a non-decimal number, and the character after the initial dollar determines which:

    $+3.14159E2
    $$1001001
    $1FE6

Break off here for real, binary or hexadecimal notation20.2.2.1.1 =

    int x = c_start, y = p-1;
    Break off a token20.2.2.1.4;
    switch (c2) {
        case '+': case '-':
            x = p; y = p+1;
            while ((Str::get_at(current_raw, y+1) == '.') ||
                    (Str::get_at(current_raw, y+1) == 'E') ||
                    (Str::get_at(current_raw, y+1) == 'e') ||
                    (Characters::isdigit(Str::get_at(current_raw, y+1))))
                y++;
            Break off a token20.2.2.1.4;
            p = y;
            c_start = p+1;
            continue;
        case '$':
            x = p; y = p+1;
            while ((Str::get_at(current_raw, y+1) == '0') ||
                    (Str::get_at(current_raw, y+1) == '1'))
                y++;
            Break off a token20.2.2.1.4;
            p = y;
            c_start = p+1;
            continue;
        default:
            x = p; y = p;
            while (Characters::isalnum(Str::get_at(current_raw, y+1)))
                y++;
            Break off a token20.2.2.1.4;
            p = y;
            c_start = p+1;
            continue;
    }

• This code is used in §20.2.2.1.

§20.2.2.1.2. A token beginning with a minus sign and continuing with digits may still not be a negative number: it may be the binary subtraction operator. For example, we need to tokenise x-1 as

|x| |-| |1|

and not as

|x| |-1|

This requires context, that is, remembering what the previous token was.

Break off here for negative number20.2.2.1.2 =

    if (((preceding_token == NULL) ||
        (preceding_token->ist_type == OPEN_ROUND_ISTT) ||
        (preceding_token->ist_type == OPERATOR_ISTT) ||
        (preceding_token->ist_type == DIVIDER_ISTT)) &&
        (c_start == p) &&
        (!((abbreviated) && (preceding_token->ist_type == INLINE_ISTT)))) {
        int dc = p+1;
        while (Characters::isdigit(Str::get_at(current_raw, dc))) dc++;
        if (dc > p+1) {
            int x = c_start, y = p-1;
            Break off a token20.2.2.1.4;
            x = p; y = dc - 1;
            Break off a token20.2.2.1.4;
            p = y;
            c_start = p+1;
            continue;
        }
    }

• This code is used in §20.2.2.1.

§20.2.2.1.3. In I6, operators made of non-alphanumeric characters can be up to three characters long, and we take the longest match: thus --> is a trigraph, not the monograph - followed by the digraph ->.

We treat the @ sign as if it were alphanumeric for the sake of assembly language opcodes such as @pull.

Break off here for operators20.2.2.1.3 =

    int monograph = TRUE, digraph = FALSE, trigraph = FALSE;
    if ((Characters::isalnum(c1)) || (c1 == '_') || (c1 == '$')) monograph = FALSE;
    if (c1 == 0x00A7) monograph = FALSE;
    if ((c1 == '#') && (Characters::isalpha(c2))) monograph = FALSE;
    if ((c1 == '_') && (Characters::isalpha(c2))) monograph = FALSE;
    if ((c1 == '#') && (c2 == '#') && (Characters::isalpha(c3))) monograph = FALSE;
    if ((c1 == '@') && (Characters::isalpha(c2))) monograph = FALSE;

    if ((c1 == '+') && (c2 == '+')) digraph = TRUE;
    if ((c1 == '-') && (c2 == '-')) digraph = TRUE;
    if ((c1 == '>') && (c2 == '=')) digraph = TRUE;
    if ((c1 == '<') && (c2 == '=')) digraph = TRUE;
    if ((c1 == '=') && (c2 == '=')) digraph = TRUE;
    if ((c1 == '-') && (c2 == '>')) digraph = TRUE;
    if ((c1 == '.') && (c2 == '&')) digraph = TRUE;
    if ((c1 == '.') && (c2 == '#')) digraph = TRUE;
    if ((c1 == '~') && (c2 == '~')) digraph = TRUE;
    if ((c1 == '~') && (c2 == '=')) digraph = TRUE;
    if ((c1 == '&') && (c2 == '&')) digraph = TRUE;
    if ((c1 == '|') && (c2 == '|')) digraph = TRUE;

    if ((c1 == '-') && (c2 == '-') && (c3 == '>')) trigraph = TRUE;

    if (trigraph) {
        int x = c_start, y = p-1;
        Break off a token20.2.2.1.4;
        x = p; y = p+2;
        Break off a token20.2.2.1.4;
        p += 2;
        c_start = p+1;
        continue;
    }

    if (digraph) {
        int x = c_start, y = p-1;
        Break off a token20.2.2.1.4;
        x = p; y = p+1;
        Break off a token20.2.2.1.4;
        p++;
        c_start = p+1;
        continue;
    }

    if (monograph) {
        int x = c_start, y = p-1;
        Break off a token20.2.2.1.4;
        x = p; y = p;
        Break off a token20.2.2.1.4;
        c_start = p+1;
        continue;
    }

• This code is used in §20.2.2.1.

§20.2.2.1.4. In this code, the new token is between character positions x and y inclusive; we ignore an empty token.

Break off a token20.2.2.1.4 =

    if (y >= x) {
        TEMPORARY_TEXT(T)
        for (int i = x; i <= y; i++) PUT_TO(T, Str::get_at(current_raw, i));

        int is = RAW_ISTT;
        inter_ti which = 0;
        int which_rw = 0, which_number = -1, which_quote = -1;
        Identify this new token20.2.2.1.4.1;

        inter_schema_token *n = InterSchemas::new_token(is, T, which, which_rw, which_number);
        #ifdef CORE_MODULE
        if (which_quote >= 0) n->as_quoted = quoted_inames[which_quote];
        #endif
        InterSchemas::add_token(sch, n);
        if (n->ist_type != WHITE_SPACE_ISTT) preceding_token = n;
        DISCARD_TEXT(T)
    }

• This code is used in §20.2.2.1, §20.2.2.1.1 (four times), §20.2.2.1.2 (twice), §20.2.2.1.3 (6 times).

§20.2.2.1.4.1. Finally, we identify what sort of token we're looking at.

Identify this new token20.2.2.1.4.1 =

    if (Str::get_at(T, 0) == '@') is = OPCODE_ISTT;
    if (Str::get_at(T, 0) == 0x00A7)
        is = IDENTIFIER_ISTT;
    if ((Str::get_at(T, 0) == '#') && (Str::get_at(T, 1) == '#') && (Characters::isalpha(Str::get_at(T, 2)))) {
        is = IDENTIFIER_ISTT;
        LOOP_THROUGH_TEXT(P, T) {
            wchar_t c = Str::get(P);
            if ((c != '_') && (c != '#') && (!Characters::isalnum(c)))
                is = RAW_ISTT;
        }
    }
    if ((Str::get_at(T, 0) == '#') && (Characters::isalpha(Str::get_at(T, 1)))) {
        is = IDENTIFIER_ISTT;
        LOOP_THROUGH_TEXT(P, T) {
            wchar_t c = Str::get(P);
            if ((c != '_') && (c != '#') && (c != '$') && (!Characters::isalnum(c)))
                is = RAW_ISTT;
        }
    }
    if ((Str::get_at(T, 0) == '_') && (Characters::isalpha(Str::get_at(T, 1)))) {
        is = IDENTIFIER_ISTT;
        LOOP_THROUGH_TEXT(P, T) {
            wchar_t c = Str::get(P);
            if ((c != '_') && (c != '#') && (!Characters::isalnum(c)))
                is = RAW_ISTT;
        }
    }
    if (Characters::isalpha(Str::get_at(T, 0))) {
        is = IDENTIFIER_ISTT;
        LOOP_THROUGH_TEXT(P, T) {
            wchar_t c = Str::get(P);
            if ((c != '_') && (!Characters::isalnum(c)))
                is = RAW_ISTT;
        }
        if (Str::begins_with_wide_string(T, L"QUOTED_INAME_0_")) which_quote = 0;
        else if (Str::begins_with_wide_string(T, L"QUOTED_INAME_1_")) which_quote = 1;
        if (Str::eq(T, I"I7_string")) { Str::clear(T); WRITE_TO(T, "I7_String"); }
        if (Str::eq(T, I"COMMA_WORD")) { Str::clear(T); WRITE_TO(T, "comma_word"); }
    }
    if (Characters::isdigit(Str::get_at(T, 0))) {
        is = NUMBER_ISTT;
        LOOP_THROUGH_TEXT(P, T) {
            wchar_t c = Str::get(P);
            if (!Characters::isdigit(c))
                is = RAW_ISTT;
        }
    }
    if (Str::get_at(T, 0) == '$') {
        is = HEX_NUMBER_ISTT;
        wchar_t c = Str::get_at(T, 1);
        if (c == '$') is = BIN_NUMBER_ISTT;
        if (c == '+') is = REAL_NUMBER_ISTT;
        if (c == '-') is = REAL_NUMBER_ISTT;
    }
    if (Str::get_at(T, 0) == '-') is = NUMBER_ISTT;

    if (Str::eq(T, I"false")) { is = NUMBER_ISTT; which_number = 0; }
    if (Str::eq(T, I"true")) { is = NUMBER_ISTT; which_number = 1; }
    if (Str::eq(T, I"nothing")) { is = NUMBER_ISTT; which_number = 0; }

    if (Str::eq(T, I"if")) { is = RESERVED_ISTT; which_rw = IF_I6RW; }
    if (Str::eq(T, I"else")) { is = RESERVED_ISTT; which_rw = ELSE_I6RW; }
    if (Str::eq(T, I"style")) { is = RESERVED_ISTT; which_rw = STYLE_I6RW; }
    if (Str::eq(T, I"return")) { is = RESERVED_ISTT; which_rw = RETURN_I6RW; }
    if (Str::eq(T, I"rtrue")) { is = RESERVED_ISTT; which_rw = RTRUE_I6RW; }
    if (Str::eq(T, I"rfalse")) { is = RESERVED_ISTT; which_rw = RFALSE_I6RW; }
    if (Str::eq(T, I"for")) { is = RESERVED_ISTT; which_rw = FOR_I6RW; }
    if (Str::eq(T, I"objectloop")) { is = RESERVED_ISTT; which_rw = OBJECTLOOP_I6RW; }
    if (Str::eq(T, I"while")) { is = RESERVED_ISTT; which_rw = WHILE_I6RW; }
    if (Str::eq(T, I"do")) { is = RESERVED_ISTT; which_rw = DO_I6RW; }
    if (Str::eq(T, I"until")) { is = RESERVED_ISTT; which_rw = UNTIL_I6RW; }
    if (Str::eq(T, I"print")) { is = RESERVED_ISTT; which_rw = PRINT_I6RW; }
    if (Str::eq(T, I"print_ret")) { is = RESERVED_ISTT; which_rw = PRINTRET_I6RW; }
    if (Str::eq(T, I"new_line")) { is = RESERVED_ISTT; which_rw = NEWLINE_I6RW; }
    if (Str::eq(T, I"give")) { is = RESERVED_ISTT; which_rw = GIVE_I6RW; }
    if (Str::eq(T, I"move")) { is = RESERVED_ISTT; which_rw = MOVE_I6RW; }
    if (Str::eq(T, I"remove")) { is = RESERVED_ISTT; which_rw = REMOVE_I6RW; }
    if (Str::eq(T, I"jump")) { is = RESERVED_ISTT; which_rw = JUMP_I6RW; }
    if (Str::eq(T, I"switch")) { is = RESERVED_ISTT; which_rw = SWITCH_I6RW; }
    if (Str::eq(T, I"default")) { is = RESERVED_ISTT; which_rw = DEFAULT_I6RW; }
    if (Str::eq(T, I"font")) { is = RESERVED_ISTT; which_rw = FONT_I6RW; }
    if (Str::eq(T, I"continue")) { is = RESERVED_ISTT; which_rw = CONTINUE_I6RW; }
    if (Str::eq(T, I"break")) { is = RESERVED_ISTT; which_rw = BREAK_I6RW; }
    if (Str::eq(T, I"quit")) { is = RESERVED_ISTT; which_rw = QUIT_I6RW; }
    if (Str::eq(T, I"restore")) { is = RESERVED_ISTT; which_rw = RESTORE_I6RW; }
    if (Str::eq(T, I"spaces")) { is = RESERVED_ISTT; which_rw = SPACES_I6RW; }
    if (Str::eq(T, I"read")) { is = RESERVED_ISTT; which_rw = READ_I6RW; }
    if (Str::eq(T, I"inversion")) { is = RESERVED_ISTT; which_rw = INVERSION_I6RW; }

    if (Str::eq_insensitive(T, I"#IFDEF")) { is = DIRECTIVE_ISTT; which_rw = IFDEF_I6RW; }
    if (Str::eq_insensitive(T, I"#IFNDEF")) { is = DIRECTIVE_ISTT; which_rw = IFNDEF_I6RW; }
    if (Str::eq_insensitive(T, I"#IFTRUE")) { is = DIRECTIVE_ISTT; which_rw = IFTRUE_I6RW; }
    if (Str::eq_insensitive(T, I"#IFFALSE")) { is = DIRECTIVE_ISTT; which_rw = IFFALSE_I6RW; }
    if (Str::eq_insensitive(T, I"#IFNOT")) { is = DIRECTIVE_ISTT; which_rw = IFNOT_I6RW; }
    if (Str::eq_insensitive(T, I"#ENDIF")) { is = DIRECTIVE_ISTT; which_rw = ENDIF_I6RW; }

    if (Str::eq(T, I",")) is = COMMA_ISTT;
    if (Str::eq(T, I":")) is = COLON_ISTT;
    if (Str::eq(T, I"(")) is = OPEN_ROUND_ISTT;
    if (Str::eq(T, I")")) is = CLOSE_ROUND_ISTT;
    if (Str::eq(T, I"{")) is = OPEN_BRACE_ISTT;
    if (Str::eq(T, I"}")) is = CLOSE_BRACE_ISTT;
    if (Str::eq(T, I";")) is = DIVIDER_ISTT;

    if (Str::eq(T, I".")) { is = OPERATOR_ISTT; which = PROPERTYVALUE_BIP; }
    if (Str::eq(T, I".&")) { is = OPERATOR_ISTT; which = PROPERTYADDRESS_BIP; }
    if (Str::eq(T, I".#")) { is = OPERATOR_ISTT; which = PROPERTYLENGTH_BIP; }

    if (Str::eq(T, I"=")) { is = OPERATOR_ISTT; which = STORE_BIP; }

    if (Str::eq(T, I"+")) { is = OPERATOR_ISTT; which = PLUS_BIP; }
    if (Str::eq(T, I"-")) { is = OPERATOR_ISTT; which = MINUS_BIP; }
    if (Str::eq(T, I"*")) { is = OPERATOR_ISTT; which = TIMES_BIP; }
    if (Str::eq(T, I"/")) { is = OPERATOR_ISTT; which = DIVIDE_BIP; }
    if (Str::eq(T, I"%")) { is = OPERATOR_ISTT; which = MODULO_BIP; }

    if (Str::eq(T, I">")) { is = OPERATOR_ISTT; which = GT_BIP; }
    if (Str::eq(T, I">=")) { is = OPERATOR_ISTT; which = GE_BIP; }
    if (Str::eq(T, I"<")) { is = OPERATOR_ISTT; which = LT_BIP; }
    if (Str::eq(T, I"<=")) { is = OPERATOR_ISTT; which = LE_BIP; }
    if (Str::eq(T, I"==")) { is = OPERATOR_ISTT; which = EQ_BIP; }
    if (Str::eq(T, I"~=")) { is = OPERATOR_ISTT; which = NE_BIP; }

    if (Str::eq(T, I"~~")) { is = OPERATOR_ISTT; which = NOT_BIP; }
    if (Str::eq(T, I"&&")) { is = OPERATOR_ISTT; which = AND_BIP; }
    if (Str::eq(T, I"||")) { is = OPERATOR_ISTT; which = OR_BIP; }
    if (Str::eq(T, I"or")) { is = OPERATOR_ISTT; which = ALTERNATIVE_BIP; }

    if (Str::eq(T, I"ofclass")) { is = OPERATOR_ISTT; which = OFCLASS_BIP; }
    if (Str::eq(T, I"has")) { is = OPERATOR_ISTT; which = HAS_BIP; }
    if (Str::eq(T, I"hasnt")) { is = OPERATOR_ISTT; which = HASNT_BIP; }
    if (Str::eq(T, I"provides")) { is = OPERATOR_ISTT; which = PROVIDES_BIP; }
    if (Str::eq(T, I"in")) { is = OPERATOR_ISTT; which = IN_BIP; }
    if (Str::eq(T, I"notin")) { is = OPERATOR_ISTT; which = NOTIN_BIP; }

    if (Str::eq(T, I"|")) { is = OPERATOR_ISTT; which = BITWISEOR_BIP; }
    if (Str::eq(T, I"&")) { is = OPERATOR_ISTT; which = BITWISEAND_BIP; }
    if (Str::eq(T, I"~")) { is = OPERATOR_ISTT; which = BITWISENOT_BIP; }

    if (Str::eq(T, I"++")) { is = OPERATOR_ISTT; which = POSTINCREMENT_BIP; }
    if (Str::eq(T, I"--")) { is = OPERATOR_ISTT; which = POSTDECREMENT_BIP; }

    if (Str::eq(T, I"->")) { is = OPERATOR_ISTT; which = LOOKUPBYTE_BIP; }
    if (Str::eq(T, I"-->")) { is = OPERATOR_ISTT; which = LOOKUP_BIP; }