CQL Fundamentals

Conditional Set Assignment

An empty set value (e.g. []) does not match the position but it can still be assigned to a variable, and a plain assignment of [] will itself match the position. The =? conditional set assignment filter may be used to assign a Set variable only if the provided value is not the empty set. In other words, = stores empty sets while =? skips them.

For example:

X = []      // X holds the empty set 
X = a1      // X holds the value a1
X =? []     // X is not modified

The =? filter yields true if the variable was assigned (possibly with the same value it already held) and false otherwise. When =? yields false, the variable is left unchanged. This matters when =? appears as one step in a sequence of filters: the assignment may leave the variable unchanged and yet still cause the enclosing sequence to fail to match.

Compound Assignment

Variables can be modified using the simple assignment filter = or the compound assignment filters +=, -=, *=, /=, %=, |=, and &=. Like simple assignment, if the RHS of a compound assignment filter is None, the variable is not modified and the assignment does not match the position.

Unbound Variables

An unbound variable is one whose value is None, i.e. it does not hold a value. A variable of any type except Dictionary type may be unbound. This can occur if the variable has not yet been assigned such as if the initializing declaration is skipped. For example:

if 1 == 2 then X = 1
str(X)      // yields the string "<None>", X is unbound

A variable may be explicitly unbound using the unbind filter which takes the name of a previously declared variable as its only argument, e.g.:

unbind X

The isbound filter takes a single identifier as an argument and yields true if the identifier corresponds to a bound variable and false otherwise (the specified identifier refers to an unbound variable or does not refer to a variable at all). The isunbound filter operates similarly but yields false if the provided identifier is a bound variable and true otherwise. For example:

X = 1
Y = 1
unbind Y
isbound X   // true
isbound Y   // false
isbound Z   // false
isunbound X // false
isunbound Y // true
isunbound Z // true

Dictionary Variables

A dictionary holds a set of key-value pairs. The possible types for keys and values are Numeric, String, and Set. The types of the keys and values are set when the dictionary is declared. A dictionary variable is declared using the dictionary keyword, no initializer is provided with the declaration. An optional type specifier of the form:

{str | int | set} --> {str | int | set}

may be provided where the type indicated on the left of the arrow specifies the key type and the type indicated on the right of the arrow specifies the value type.

For example, the following filter declares a dictionary variable named Dx with keys of Numeric type and values of Set type:

dictionary int --> set Dx

If the optional type specifier is not present, the dictionary will have keys and values of type String. Dictionary variables are persistent, their values are maintained across games.

Dictionary Entry Access and Assignment

Given a dictionary variable Dx, the key access filter Dx[key] yields the value of key stored in Dx or None if key does not exist in Dx. Accessing a missing key never creates it.

The key assignment filter Dx[key]= value inserts key into Dx with the value value, if neither key nor value is None, or replaces the value for key if key already exists. If key or value is None, the key assignment filter does not match the position and Dx is not modified.

In particular, note that:

• Dx[key] on a missing key yields None and does not create an entry.
• Dx[key] = value inserts or replaces an entry unless key or value is None.
• Dx[key] =? value (for Set-valued dictionaries) inserts or replaces an entry only if value is a non-empty set.
• Dx[key] += ... and the other compound assignment operators may create a missing key first; see below.

Compound Assignment and Key Autovivification

The compound assignment operators (+=, -=, *=, /=, %=, |=, and &=) may be used with dictionaries that have appropriate value types. If the specified key does not exist in the dictionary, it will first be created (vivified) with a default value (0 for Numeric values, [] for Set values, and "" for String values) unless the filter will evaluate to false (which occurs when the right-hand side of the compound assignment is None or the right-hand side of /= or %= is zero). For example, Dx[key] += 1 will increment the value of Dx[key], if key did not exist in Dx it will first be added with a value of 0 and then incremented to 1.

This autovivification of entries only occurs when using the compound assignment operators. Simply attempting to access a key that does not exist with Dx[key] will not create a corresponding entry in Dx, and neither will a failing simple assignment or conditional set assignment.

To increment the value of a key that already exists in a dictionary, without creating it if it does not exist, use if Dx[key] then Dx[key] += 1 or Dx[key] = Dx[key] + 1 instead.

Conditional Set Assignment with Dictionaries

The conditional set assignment operator =? can be used with dictionaries having Set values with the same semantics that are employed for regular variables. E.g. Dx[key]=? value will assign value to Dx[key] only if value is not the empty set. If value is empty and key does not exist in Dx, it is not created with a default value.

Removing Keys from Dictionaries

An unbind filter of the form unbind Dx[key] will remove key from Dx. If unbind is applied to a dictionary variable, e.g. unbind Dx, all of the keys are removed from the variable (but the variable itself is not unbound, dictionary variables never have a value of None).

Dictionary Cardinality and Iteration

The number of keys in a dictionary can be obtained using the cardinality filter #, e.g. #Dx will yield the number of key-value pairs present in Dx. The keys in a dictionary may be iterated using the key iteration filter.

Dictionary Access Caveats

For dictionary variables having keys of Set type, care must be taken when attempting to use an unbracketed piece designator as the key in a dictionary access or assignment filter. The parser may otherwise interpret the text as a piece designator instead of a dictionary key.

For example, Dx[a2] += 1 will produce a syntax error because it is interpreted as two separate filters: Dx and [a2] += 1. To force dictionary-key parsing, one of the following disambiguation techniques may be used:

• Enclose the piece type and/or square designator portions in square brackets. For example, instead of Dx[Ka1-8] use Dx[[K]a1-8], Dx[K[a1-8]], or Dx[[K][a1-8]]. This prevents the outside bracketed expression from being confused with a piece designator (since nested brackets are not permitted in piece designators).
• Add a space character before and/or after the piece designator. For example, instead of Dx[Ka1-8] use Dx[ Ka1-8 ].
• Prefix the piece designator with the identity operator. For example, instead of Dx[Ka1-8] use Dx[`Ka1-8].

In short, for a Set-keyed dictionary:

• Dx[a2] is ambiguous and should be avoided.
• Dx[ a2 ], Dx[`a2], and Dx[[a2]] are unambiguous dictionary accesses.

Similarly, if a variable that would be interpreted as a compound piece type designator when enclosed by square brackets without spaces (i.e. one whose name consists entirely of the characters in the set QqBbRrNnKkPpAa_), one of the last two disambiguation techniques above needs to be taken when using the variable as a key in a dictionary access or assignment filter. For example, instead of Dx[Rank] use Dx[ Rank ] or Dx[`Rank].

Persistent Variables

The values of variables are reset to None at the beginning of every game unless declared with the persistent keyword (which immediately precedes the variable name) in which case their value persists across all games. Numeric, Set, and String variables may be declared as persistent. Dictionary variables are always persistent but may not be declared using the persistent keyword. Persistent variables are initialized once, prior to evaluating the first position in the first game. Persistent Numeric variables are initialized to zero, Set variables to the empty set, and String variables to the empty string. Persistent variables may be declared using a compound assignment operator, the type of the variable will be deduced from the RHS of the compound assignment. For example:

persistent $total_positions += 1

declares a persistent Numeric variable named $total_positions that is incremented for every evaluated position. Persistent variables may be unbound using the unbind filter in which case they are never implicitly re-initialized.

The final values of non-dictionary persistent variables are emitted at the end of processing. The option --showdictionaries may be used to cause dictionary variables to be emitted along with other persistent variables.

Persistent and dictionary variables may be declared using the quiet keyword to suppress emission of their value after processing, this may be used to suppress emission of possibly very long string variables. E.g.:

persistent quiet $str = ""

Variable Scopes

Every variable exists within a scope which dictates the portion of the query for which the variable is visible (may be referenced). In CQLi, all variables are placed in either the global scope or a block scope. Block scopes are introduced by function invocations and iteration filters (echo, loop, piece, square, string, and while) and extend to the end of the respective filter.

Names appearing in a function body are resolved when the function is invoked. However, the usual rule still applies at that point: every referenced variable must already have been declared before the reference is processed.

Persistent variables and variables declared outside of a function or iteration filter exist in the global scope. Parameter variables (parameters declared in a function parameter list and the iteration variables of piece, string, echo, and square filters) always exist within their corresponding block scope, shadowing any variable of the same name in an enclosing scope.

Other (non-persistent, non-parameter) variables used in a block scope either refer to a variable in an enclosing scope or create a new variable in the current block scope. While variables within an enclosing scope may be accessed in a block scope, variables created in a block scope may not be accessed after the end of that scope. Block scopes may be nested and all block scopes are enclosed by the global scope.

The following example illustrates the key points described above:

$param = "abc"              // $param created in global scope.
$called = 0                 // $called created in global scope.

function foo($param) {      // $param shadows variable in enclosing scope.
    $called = 1             // Refers to the above $called variable.
    $local = abs $param     // $local is not accessible outside of foo.
    $local + $global        // $global will be accessible at invocation.
}

$global = 10                // $global resides in the global scope.
$result = foo(5)            // foo can access $global from here.

The first two lines declare a String variable named $param and a Numeric variable named $called, both in the global scope. Function foo declares a parameter variable $param, a block scope variable which will shadow the global variable $param within the function invocation. The body of the function is not processed until the function is invoked. When this happens at the end of the example, the first line of the body of foo modifies the global variable $called, it does not create a new variable because the enclosing scope already has a variable named $called and within foo the variable is not a parameter variable.

Since there is not already a variable named $local at the point where foo is invoked, the $local variable will be installed in foo’s block scope and cannot be referenced outside of the function. The variable $global does not exist when foo is defined but does exist when foo is invoked so the reference in foo to $global will be that of the existing global scope variable. Note that if foo was called before $global had been declared a parse error would result. In other words, a function body may refer to a global declared later in the file, but only if that declaration has been processed before the function is invoked.

Piece Type Designators

A simple piece type designator specifies a single class of chess pieces. The table below lists the simple piece type designators used in CQL.

For example, the piece designator Q represents the squares on which a white queen resides in the position currently being evaluated. Multiple simple piece type designators may be combined to form a compound piece type designator which consists of one or more simple piece type designators enclosed in square brackets. For example, [Qq] represents squares occupied by white and black queens and [_a] represents the set of squares not occupied by white pieces, in the current position.

Square Designators

A square designator refers to specific squares on a chessboard by their files and/or ranks. A simple square designator consists of a file designator followed by a rank designator. A file designator is either a file name or two file names separated by a hyphen. A rank designator is either a rank name or two rank names separated by a hyphen. Valid file names are a, b, c, d, e, f, g, and h. Valid rank names are 1, 2, 3, 4, 5, 6, 7, and 8.

Examples of simple square designators include a1, e5, a-h8 (the squares on the 8^th rank), and d-e4-5 (the four center squares). Note that a simple square designator must consist of both a file designator and a rank designator; e.g. b is a piece type designator, not a square designator. Use b1-8 to refer to all the squares in the b file.

Multiple simple square designators may be combined to form a compound square designator which consists of one or more simple square designators, separated by commas, enclosed in square brackets. For example, [a1,h8] represents the squares a1 and h8, while [a1-8,a-h8,d4] represents the union of d4, the a file, and rank 8.

Combining Piece Type and Square Designators

A piece designator may consist of both a piece type designator and a square designator, either of which may be a compound designator, in which the piece type designator precedes the square designator. For example, Ra-h8 represents the squares occupied by white rooks on the 8^th rank and [Kk][a1,a8,h1,h8] represents corner squares occupied by a king of either color.

A piece designator represents the squares that are given by the intersection of its piece type designator and its square designator; if either one is missing, all squares are implied for the missing component. For example, Ra1 is equivalent to R & a1. Compound designators represent the union of each simple designator in the bracketed list, e.g. [a1-8,a-h8,d4] is equivalent to a1-8 | a-h8 | d4 and [Kk][a1,a8,h1,h8] is equivalent to (K | k) & (a1 | a8 | h1 | h8) (except when appearing in shift transforms).

Piece designators are a cornerstone of the CQL language and provide a concise mechanism to articulate a set of squares using both the static nature of the chess board (with square designators) and the dynamic nature of piece occupancy (using piece type designators).

Slicing Assignment

If the left-hand side of a string slicing operation is a variable, the slice may be assigned using the syntax x[ … ]= String where String is an arbitrary string filter. The portion of the substring referenced by the string index expression is replaced with String. String may be a different size than the referenced substring in which case the string value of x is modified to accommodate the replacement. The result of the slicing assignment filter has Boolean type and matches the position unless x is unbound, String does not match the position (i.e. it is None), or the left-hand side of the assignment has the form x[i] and i specifies an index that does not exist in x. If the slicing assignment does not match the position, x is not modified although the reverse is not necessarily true, e.g. given a variable x of length 5, the filter x[10:] = "abc" will match the position even though x is not modified.

As with slicing itself, single-index and range forms behave differently here:

• x[i] = ... fails if i is not a valid existing index.
• x[m:n] = ... may succeed even when the replacement changes nothing, for example because the referenced slice is empty and the replacement is also empty, or because the slice lies entirely beyond the end of the string.

Repetition

Repetition operators allow part of a pattern to optionally match or to match multiple times. The * operator specifies that the previous character is present zero or more times, + matches the previous character one or more times, and ? matches zero or once. For example, in the query:

var ~~ "\d+:\d+"

\d represents any digit and + indicates one or more of what immediately preceded it so \d+ represents one or more digits. The : matches itself so the pattern \d+:\d+ will match if var contains a sequence consisting of one or more digits followed by a colon and one or more digits immediately following the colon, e.g. "Time 1:23" will match the pattern (with the result being "1:23") but "123:" and ":123" will not.

The basic repetition operators (*, +, ?, and {...}) will match as much of the string as possible (i.e. they are greedy matching operators) without causing a match failure (i.e. they are non-possessive). For example:

"ABBB" ~~ "AB*"

will match the entire string and:

"ABBBCABD" ~~ "AB*D"

will match "ABD". The first successful match is always returned regardless of whether a later match would consume a larger portion of the string, e.g.:

"ABABBABBBB" ~~ "AB+"

will match the initial sequence "AB", not a later (and longer) sequence. Repetition operators may be made non-greedy (matching as little as possible) by suffixing them with ?. For example:

"ABBB" ~~ "AB+?"

will match "AB" since the expression B+? requires at least one B and prefers to match the smallest sequence. Non-greedy repetition is useful when trying to match delimited text. For example:

"#A# #B# #C#" ~~ "#.*#"

will match the # character followed by any number of any character (. represents any non-newline character) followed by another #. To extract only the first delimited portion (#A#) the non-greedy version of * may be used:

"#A# #B# #C#" ~~ "#.*?#"

The basic greedy and non-greedy repetition operators are summarized in the table below.

Alternation

The | character is the alternation operator, A|B will match either A or B.

Character Classes

A character class matches any character from the specified square bracket-enclosed set. For example, the regex ch[aio]p will match chap, chip, or chop. Ranges may be created by separating two characters by a dash, e.g. [A-Z] will match any character with a Unicode code point value between (inclusive) the values used to represent A and Z. A negated character class can be specified by using ^ as the first character in the class in which case the class matches anything except the contained values, e.g. [^A-Z] will match any character except A through Z.

Character classes may be nested, e.g. [[A-Z][a-z]] is equivalent to [A-Za-z]. The && and -- class operators may be used to perform set intersection and set subtraction, respectively, to form the resulting class. For example, [\p{S}--\p{Sm}] will match a non-math symbol character and [\p{L}&&\p{script=Cyrl}] will match any Cyrillic letter. Any of the Escape Sequences except for \A, \b, \B, \R, \X, \z, \Z, and backreferences may be used in a character class. The POSIX character classes are also supported, e.g. [[:ascii:]] will match any ASCII character.

Groups, Captures, and Backreferences

Parentheses are used to form a group which is treated as a unit, repetition operators following such a group apply to the entire text matching the group. For example the pattern (\d+:)+ will match a sequence of one or more groups of text, each containing one or more digits followed by a colon.

By default, groups perform captures meaning that their corresponding matching text may be referenced later in the pattern. Text matching a captured group is accessible using backreferences which consist of a backslash followed by an index i (starting at 1) that represents the i^th capture group appearing in the pattern. For example, the pattern \d\d\d will match any three digits but the pattern (\d)\1\1 will only match three identical digits (e.g. 111, 222, etc).

Backreferences may also appear outside of regular expression patterns to extract text matching capture groups from the most recently evaluated ~~ filter. Additionally, \0 may be used outside of a pattern and yields the entire matched text (which is also the result of the ~~ filter). For example, the following query will match the first appearance of a character appearing three or more times in a row with the repeated character available after the match as \1:

"ABCCDEEEEF" ~~ "(.)\1{2,}"
\0 == "EEEE"
\1 == "E"

Named capture groups may be extracted outside of patterns using \{name} which yields the matched text of the named group, and \-{name} which yields its starting index. For example:

"2024-01-15" ~~ "(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})"
\{year} == "2024"
\{month} == "01"
\{day} == "15"
\-{year} == 0
\-{month} == 5

Named capture group names must start with a letter and contain only ASCII letters and digits.

Groups may be nested to an arbitrary depth.

In addition to the basic grouping/capturing parentheses, there are several special parenthetical constructs that introduce various behaviors. These are summarized in the table below.

For example, the patterns (fl|spl)at, (?<prefix>fl|spl)at, and (?:fl|spl)at will all match the same text, the difference being that fl/spl matching prefix will be available with the \1 backreference after matching either of the first two cases and additionally available via the named backreference \k<prefix> later in the same pattern, or via \{prefix} outside of the pattern, in the second case.

Lookahead and lookbehind assertions require specific text to be present at a particular point in a match in order to continue. Use cases for these constructs, as well as possessive matching, are more advanced and outside the scope of this introduction.

Escape Sequences

The backslash \ character is used to escape regex meta characters in patterns and access backreference content of previously matched capture groups. The backslash may also be used to start one of several escape sequences as described in the table below.

Anchored Patterns

Patterns are not anchored by default meaning that the matching substring may occur in any part of the string, the special ^ and $ characters may be used to match the beginning or end of the string, respectively. For example:

var ~~ "^\d+"

will match one or more digits appearing at the start of the string.

Finding all Matches

When the condition of the while filter is a regular expression matching filter, the filter becomes a regex iteration filter. In this case, the LHS of the ~~ operator will be evaluated once after which the pattern provided on the RHS will be applied to the LHS argument until it no longer matches with the body of the while filter being evaluated after each match. For example:

while ("ABC" ~~ ".") {
    message \0
}

will print A on the first iteration, B on the second, and C on the third and final iteration. Note that the pattern provided as the RHS of the ~~ operator will only be processed once, even if the pattern is a string variable whose value changes after iteration begins. The result of this form of the while filter matches the position unless the LHS argument does not, even if the pattern never matches.

Regex Matching Flags

There are several flags that may be embedded in a regular expression to change the default matching behavior in different ways. The table below describes these flags.

The values of the above flags may be set within a regular expression using the syntax:

(? {imswx}[-]{imswx} )

which will set the flag values until the next flag setting expression appears or the end of the pattern is reached. The flags may also be modified just for a sub-pattern using the syntax:

(? {imswx}[-]{imswx} : … )

where … is the pattern that should be subject to the flag modifications. Flags appearing without a preceding - are turned ON, flags appearing after a - are turned OFF.

For example, the following query performs a case-insensitive search for “Michael Jones”:

$name ~~ "(?i)Michael Jones"

The following query performs a case-insensitive search for “Michael” followed by a case-sensitive search for “Jones”:

$name ~~ "(?i)Michael (?-i)Jones"

The same effect could be realized by using the second form above to limit the flag modification to a single sub-pattern:

$name ~~ "(?i:Michael) Jones"

The following pattern shows how multiple flags may be enabled and disabled at once:

(?is-mw)

which will turn the i and s flags ON and turn the m and w flags OFF.