Other Features

Piece Variables

The piece assignment filter is used to define a variable that can store the identity of a piece. The syntax of the piece filter is:

piece name = set

where name is any valid variable name and set is any filter with Set type. If set contains a single square and that square is occupied, the variable name has the value of the identity of the piece occupying that square. Otherwise the piece variable has a value of None. For example:

piece $p = a1

will create a Piece variable $p which holds the identity of the piece residing on square a1 or None if a1 is unoccupied. The piece variable can then be used to represent the piece in any position, including one where the piece is on a different square. For example, the below query will find positions where the a1 rook appears in all four corners of the board at some point in a game:

initial
piece $p = Ra1
(flip count find $p & a1) == 4

When appearing as an argument to the str, comment, or message filters, a piece variable is portrayed with the piece type followed by the square it occupies (e.g. Ke1) or [absent] if the piece is no longer on the board.

Piece variables are automatically converted to Sets when appearing anywhere except as an argument to a user-defined function or the str, comment, message, or pieceid filters.

Piece variables are also created with the piece iteration filter.

The pieceid Filter

The piece ID of any piece can be obtained with the pieceid filter which accepts a single argument which is either a piece variable or a Set filter. If the argument is a Set filter, pieceid yields the numeric pieceID of the piece that resides on the single square represented by the set argument. If the set argument does not consist of a single square, or if the square represented by the argument is not occupied, the pieceid filter yields None. If the argument is a piece variable the pieceid filter yields the numeric piece ID of the corresponding piece, even if the piece is no longer on the board. The pieceid filter is used primarily to determine if two pieces in different positions are the same piece. For example, the following filter will find two positions within a game that are identical except that two or more pieces of the same type and color have swapped places:

echo (source target) {
    source < target
    source & target == .
    $swapped_pieces = square sq in [Aa] {
        source:pieceid sq != pieceid sq
    }         
    $swapped_pieces > 0 
    comment("squares of swapped pieces: " $swapped_pieces)
}

Notes

New pieces are not typically added to the board during play although this can occur for dropped pieces in the Crazyhouse variant and when pieces are added to form imaginary positions with either the imagine filter or when generating reverse captures with the speculative move filter. In such cases, the next available piece ID is assigned to the newly placed piece. Captured pieces that are subsequently dropped in the Crazyhouse variant always have a new piece ID, not one that was associated with a captured piece.

In the Crazyhouse variant, it is possible that a game has multiple variations that each consist of drop moves. In such situations it guaranteed that each dropped piece will have a unique piece ID that is not reused in other variations in the same game.

When new pieces are added with the imagine filter or by exploring reverse moves with the speculative move filter, each new piece is assigned the next available piece ID but these piece IDs may be reused by later placements after the corresponding imaginary position has expired. Piece IDs used in imaginary positions saved with the saveposition filter are not subsequently reused.

A maximum of 65,535 distinct piece IDs are supported per game.

The cqlbegin Filter

The cqlbegin filter takes a single block (compound statement) argument. The block is evaluated one time prior to processing games. When running with multiple threads, this block is evaluated one time by each thread prior to that thread processing any games (note that it is possible for one thread to evaluate its cqlbegin block after another thread has already started processing games). Multiple cqlbegin blocks may be present in which case they are evaluated in the order in which they appear.

The primary purpose of this filter is to perform preparatory tasks such as initializing persistent variables or processing external files that should be completed before processing the first game (such as in this ).

The cqlend Filter

The cqlend filter takes a single block (compound statement) argument. The block is evaluated once after all games have been processed. When running with multiple threads, this block is evaluated after all persistent variables (including dictionaries) have been merged. Multiple cqlend blocks are evaluated in the order in which they appear.

The cqlend filter can be used to perform post-processing tasks such as writing out custom report information to a file via writefile or the message filter. Since such blocks have access to merged persistent data, they can be used to summarize and/or analyze data collected while processing games.

Metadata filters in cqlbegin and cqlend blocks

While all filters are accessible within cqlbegin and cqlend blocks, the evaluation of such blocks occurs outside the context of a real game so some filters are not particularly useful. Within cqlbegin and cqlend blocks, the gamenumber filter yields a value of zero (which never occurs elsewhere), the result "*" filter yields true to indicate an incomplete game, and the rest of the metadata filters yield a value of None.

All filters in CQLi are evaluated in the context of a current position. In the case of cqlbegin and cqlend blocks, this position is the standard starting position which represents the initial and terminal position of a non-existent game.

The readfile Filter

readfile input-filename

The readfile filter accepts a single String argument which represents the name of a file to read. If the provided string is None or the corresponding file cannot be opened for reading, a runtime error is generated and CQLi will terminate. Otherwise the readfile filter yields a String value representing the contents of the specified file. If a relative pathname is provided, it is searched for in the current working directory, the directories specified by CL_PATH are not searched.

The following example will populate the PGN tags ECO and Opening based on the positions reached in each game. The loadEcoFile function reads the tab-delimited files available here which contain opening data carefully curated by Niklas Fiekas (the same data is used by lichess.org to populate these fields for games played on their site). These files contain ECO codes and opening names along with the common set of moves used to reach these positions and a partial FEN string. The data from these files are used to populate two dictionaries, $eco_dict and $opening_dict, whose keys are the first three fields of a FEN string and whose values are the corresponding ECO code and opening name, respectively. The cqlbegin filter is used to ensure that these files are processed one time (per thread) before the first game is analyzed.

cql(silent)
dictionary (min) $eco_dict
dictionary (min) $opening_dict

function loadEcoFile($filename) {
    while ((readfile $filename) ~~ "^(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*)$") {
        $eco = \1
        $opening = \2
        $fen = \5
        $fen ~~ "^(.*? .*? .*?) "
        $partial_fen = \1
        $eco_dict[$partial_fen] = $eco
        $opening_dict[$partial_fen] = $opening
    }
}

cqlbegin {
    $eco_dir = "chess-openings/dist/"
    loadEcoFile($eco_dir + "a.tsv")
    loadEcoFile($eco_dir + "b.tsv")
    loadEcoFile($eco_dir + "c.tsv")
    loadEcoFile($eco_dir + "d.tsv")
    loadEcoFile($eco_dir + "e.tsv")
}

fen ~~ "^(.*? .*? .*?) "
$partial_fen = \1
if ($eco_dict[$partial_fen]) {
    settag("MyECO" $eco_dict[$partial_fen])
    settag("MyOpening" $opening_dict[$partial_fen])
}

The $eco_dir variable will need to be modified to reflect the location of the opening data files.

The result is to set the ECO and Opening tags to the values corresponding to the most advanced position in each game that has a corresponding position in the opening data. The persistent and dictionary variables are defined with arbitrary merge strategies to allow the query to be run in multi-threaded mode. See the commandpipe filter for an alternate implementation which communicates with an external process to perform the positional inquiries.

The writefile Filter

writefile [noclobber] ( output-filename contents )

The writefile filter takes a parenthesized argument list containing two String arguments. The first argument specifies the name of the file to write and the second argument specifies the contents to write to the file. The first time that writefile is used to write to a particular file, that file is opened for writing and any previous contents are replaced with contents unless the noclobber parameter is specified in which case the value of contents is appended to the file. Future evaluations of writefile filters writing to the same file cause the specified contents to be appended to the file.

If the output-filename string is None, a runtime error is generated after which CQLi will terminate. Otherwise, if the contents string is None, the writefile filter will yield false without attempting to write to the file. Otherwise, if the corresponding file cannot be opened for writing, a runtime error is generated and CQLi will subsequently terminate.

For example, to write the FEN string of all matching positions where either side could mate if it was their turn, the following query could be used:

move legal : mate
imagine sidetomove reverse : move legal : mate
writefile("mutual-mates.txt" standardfen + \n)

Notes

The readfile and writefile filters may be used in multi-threaded mode in which case it is guaranteed that no more than one read or write operation will occur at a time. This behavior prevents the possibility of interleaved writes to the same file and inconsistent file state due to race conditions involving reads and writes to the same file but does not provide any guarantees related to the order in which reads or writes are performed which may differ between runs. If the order in which reads and writes are performed is important, multi-threaded mode should not be employed.

The –secure option may be used to forbid the use of these filters.

Unarmored specifiers

The “unarmored” %t and %q specifiers are provided to support use cases where the replacements performed by the corresponding %T and %Q specifiers are undesired. In this case, it is strongly advised that the replace filter be used to unconditionally replace undesired characters from appearing in the expansion of the format specifier. For %t specifiers, this means that the value of the tag is replaced in the primary query using replace and settag. For the %q specifier, the replace filter should be the last filter in the sub-query, replacing all undesired characters with either the empty string or a suitable replacement string. For example, if the tag Site is used with the %t specifier, the primary query should contain a filter that looks something like settag("Site" replace(tag "Site" "bad-character-set" "")).

The replacements performed by the %T and %Q specifiers are equivalent to evaluating the filter replace(string "[^[-A-Za-z0-9_][^[:ascii:]]]" "") where string is the value that would be substituted before replacements. This may be used as a starting point to implement a different character replacement policy. For example, -o 'games/%q{$var = /* ... */ replace($var "[^[-A-Za-z0-9_~$#]]" "")}.pgn' allows the use of ~, $, and # in filenames but prevents the use of non-ASCII characters. To cause offending characters to be replaced with another character (e.g. _), the "" argument to replace can be replaced with the relevant string, e.g. "_".

When deciding what characters to allow in filenames when using %t or %q, it may be helpful to consider the following:

• Windows does not allow the characters <, >, :, ", /, \, |, ?, or * to be present in a file name.
• ASCII control characters (\x00-\x1F) are not allowed in file names for Windows and are not advisable in other filesystems.
• The presence of directory separator characters (e.g. / and \) and . can present security concerns as it could allow parent directory traversal providing a mechanism to access files outside of the intended target directory.
• Space characters may present issues with some programs and can be awkward to handle in the shell.
• Windows does not allow a filename to end with a space or . character.

See also this list of Problematic characters for more information on potential issues that could be encountered when using other symbols in a file name including %, ;, and =.

Filesystem case-sensitivity

It is worth noting that the resulting output files may be impacted by whether the underlying filesystem is case-sensitive or not. On most Windows and macOS systems, file access is case-insensitive meaning that the name test.pgn may be used to access a file with the name test.pgn, Test.pgn, or TEST.PGN. On most Linux systems, the filesystem is case-sensitive which means that attempting to access a file named test.pgn will not access an existing file with the name Test.pgn.

The relevance to CQLi is that when using a dynamic output file specifier that may expand to values that differ only by case, the result will be dependent on the case-sensitivity of the host filesytem. For example, given the option -o 'games/%T{White}.pgn', a game that contains the value "Lasker, Emmanuel" for the White tag, and another game that contains the value "LASKER, EMMANUEL" for the White tag, both games will wind up in the same file on a case-insensitive filesystem but in different files in a case-sensitive filesystem.

To obtain portable results, the lowercase and uppercase filters may be used to ensure consistent case in the resulting expansion. For example, %Q{lowercase(tag "White")} will expand to a value with all uppercase characters converted to lowercase. Similarly, the filter settag("White" lowercase(tag "White")) could be used in the primary query to convert the value of the White tag to all lowercase so that a %T{White} specifier expands to a lowercase value.

Note that the results emitted by the --dryrun option will show case-sensitive filenames which reflect the names that would be used to open the corresponding output files but do not reflect that case-insensitive way in which the resulting files may be accessed.

Persistent Variables and Merge Strategies

Queries that employ persistent variables may not be used with multi-threaded execution unless each persistent variable is defined with a merge strategy that specifies how the final value of the variable is to be formed from the copies used by each thread.

A persistent variable declaration may include an optional parenthesized merge strategy immediately following the persistent keyword. For example, the declaration:

persistent (sum) totalPositions += 1

specifies that when the query is run in multi-threaded mode, the final value of the totalPositions variable will be calculated by summing the value of each thread’s copy of this variable. Despite not using the persistent keyword, dictionary variables are always persistent. A dictionary variable may specify a merge strategy following the dictionary keyword and any optional type specifier.

In the vast majority of cases where persistent variables are used, the variable is used to track an extremal (minimal or maximal) value or as a counter in which case the desired semantics can be obtained in multi-threaded mode using corresponding merge strategies. The table below lists the merge strategies available for each variable type.

When merging dictionary variables, the final result will contain all of the keys that exist in each thread’s copy, the merge strategy is only employed for keys that exist in multiple threads. The final result of persistent variables is never None unless every thread’s copy of the variable was None, i.e. a non-None value trumps a None value regardless of merge strategy.

The below example employs different merge strategies to collect some simple metrics about games in a database:

persistent (sum) totalPositions += 1
persistent (min) earliestPromotion += 0
persistent (max) greatestPly += 0
if move promote A then
    earliestPromotion = min(earliestPromotion ply)
if terminal then
    greatestPly = max(greatestPly ply)

Indeterminate Processing Order

When running in multi-threaded mode, the order in which games are processed will typically vary between runs as the order is dependent on how long it takes to process each game. In most cases, this is not a concern or even noticeable but there are some situations in which differences may be observed when using multi-threaded mode:

• The output produced when using the –limit option may be different between runs on the same database when using multi-threaded execution.
• The order in which messages are emitted via the message filter may differ between runs.
• Reads performed by the readfile filter and the writes performed by writefile are unordered between threads and may differ between runs.
• The order of matching game numbers shown when using the –showmatches option.
• The order in which matching games are written when using the –nosort option.

Command Pipe Considerations

When a commandpipe filter appears in a CQL query running in multi-threaded mode, a separate instance of the corresponding command-pipe program is invoked for each thread. If the command-pipe program needs to maintain state information between processed games the CQL query may not be an appropriate candidate for multi-threaded mode.

Writing Command Pipe Programs

A command-pipe program consists of three parts: 1) an optional initial startup routine, 2) the main loop, and 3) an optional shutdown routine. The main loop reads one line at a time from standard input (stdin) and responds with a single line written to standard output (stdout). Lines read from stdin correspond to requests sent from CQLi during the evaluation of a commandpipe filter, the line written to stdout forms the result of the same commandpipe filter.

Every request sent to the command-pipe program will automatically be terminated by the platform- specific newline sequence, the request string provided in the commandpipe filter must not contain any embedded newline sequences of its own (this would result in the command-pipe program treating the input as two separate requests for which it would send two separate responses causing a stream desynchronization event for which CQLi will detect, produce a fatal error, and terminate).

The command-pipe program must respond to every request with a single line terminated by the platform’s newline sequence. The command-pipe program must ensure that the stdout stream is flushed after every response is written, this is typically accomplished by calling a flush function on the stream. Failure to flush stdout after writing the request will prevent CQLi from being able to read the result causing a timeout (or a hang if timeouts are disabled).

Requests will always be sent to command-pipe programs using UTF-8 encoding and it is expected that the resulting response will likewise be UTF-8 encoded.

Request and response strings are limited to 4096 bytes, including the newline sequence, CQLi will terminate with a fatal error is this limit is exceeded on either end.

When CQLi is shutting down, it will close the write end of the pipe that is connected to the command-pipe’s stdin stream, the command-pipe program will subsequently read EOF from this stream which should terminate the main loop.

A command-pipe program may need to perform initialization at startup such as connecting to a chess engine, loading data from a file or database, etc. and may also need to perform shutdown tasks such as disconnecting from an engine or server, writing results to a file, etc. Since the command-pipe program is not invoked until there is a pending request, the initialization should generally be performed quickly, if the initial response is not received within one second, a timeout will occur (timeouts are adjustable using the --timeout option described below). Shutdown procedures may take longer as CQLi does not wait for command-pipe programs to exit (CQLi may very well have terminated by the time the command-pipe program is notified that there are no more requests to process).

The stdin and stdout streams must not be read, written, or closed by the initialization or shutdown routines. The stderr stream of the command-pipe process will be connected to the stderr stream of CQLi and may be asynchronously written to at any point during the lifetime of the command-pipe program which is useful for debugging purposes and may also be used to write summary data to the screen.

Timeouts

If a command-pipe program does not respond within a specified amount of time CQLi will produce a runtime error and terminate. There are two timeout values that may be configured, the amount of time it takes a newly spawned command-pipe program to respond to its initial request and the amount of time allowed to respond to subsequent requests. The default timeout values are both one second. The --pipetimeout option may be used to specify different timeouts. This option takes one or two numeric non-negative arguments that represent the timeout in milliseconds (1/1000^th of a second), if only one value is provided it applies to both timeouts, otherwise the first value specifies the timeout of initial requests and the second value specifies the response limit of subsequent requests. When two values are provided, the second may not specify a timeout that is larger than the first. A value of zero indicates that no timeout is enforced and may be applied to both timeouts or just the initial timeout. The timeout values are shared by all command-pipe programs.

Note that timeout values represent the wallclock time between sending the request and receiving the response and external resource load could result in spurious timeouts if the specified values are set too low.

Locating the Commandpipe Program

If the provided program-name contains a directory separator, no searching is performed, the program at the provided location will be executed (relative paths will be resolved from the current working directory). If the provided program-name does not contain a directory separator, the specified program is searched for in a platform-specific manner as described in the following sections.

Notes for Windows

If the provided program-name does not contain a directory separator, the program is searched for, in order, in the following locations:

• The directory from which CQLi was launched.
• The Windows system directories.
• The directories specified by the PATH environment variable, in the order in which they appear.

The CL_PATH environment variable is not searched for program-name. If program-name does not contain an extension, a .exe extension is added before the location is resolved.

To execute batch files or scripts, the program-name must specify the coresponding interpreter with the script specified as an argument. For example, to execute a Python script named pipe.py, use:

commandpipe("python" "pipe.py" $request)

To pass commandline arguments to pipe.py, provide them as separate arguments to commandpipe after "pipe.py".

Notes for Linux and macOS

If the provided program-name does not contain a directory separator, the program is searched for in the directories specified by the PATH environment variable, in the order in which they appear. If the PATH environment variable is not defined, an implementation-defined set of default directories is searched. The CL_PATH environment variable is not searched for program-name.

Note that unless the PATH variable contains the current directory (which is typically not the case) the current directory will not be searched. To execute a program in the current directory, prefix the name with ./, e.g. ./test.py. Because the name contains a slash, it will be resolved relative to the current directory without a search.

If the file is marked as executable but is not a valid exectuable format and does not start with a recognizable header specifying an intepreter, the default shell (/bin/sh) will be executed with program-name as the first argument (the remaining arguments will also be passed to the shell which will presumably pass them to the interpreter).

If program-name is a script, e.g. a shell or Python script, and contains a shebang line that identifies the interpreter, the interpreter will automatically be invoked to execute the provided script. Otherwise the interpreter must be provided as program-name with the name of the script and its arguments following the interpreter, e.g.:

commandpipe("python3" "test.py" $request)

Debugging Command Pipe Programs

The most common errors writing or using command-pipe programs are listed below, these should always be the first things to check when troubleshooting the operation of a commandpipe filter:

• Incorrect specification of the program name in the commandpipe filter resulting in an error message.
• Not terminating response strings with a newline sequence in the command-pipe program resulting in a timeout error or a hang.
• Not flushing stdout after writing a response in the command-pipe program resulting in a timeout or a hang.
• Taking too long to respond to a request in the command-pipe program resulting in a timeout or hang.

Other potentially helpful troubleshooting steps include writing event information to stderr from the command-pipe program and using the message filter to write the result of a commandpipe filter to the screen.

Examples

#!/usr/bin/python3

import sys
import chess.engine

def getScore(engine, fen):
    board = chess.Board(fen)
    info = engine.analyse(board, chess.engine.Limit(time=0.5))
    return str(info["score"].white())

# Main loop
with chess.engine.SimpleEngine.popen_uci("/usr/games/stockfish") as engine:
    for line in sys.stdin:
        fen = line.rstrip()
        sys.stdout.write(getScore(engine, fen) + "\n")
        sys.stdout.flush()

cql(quiet)
comment commandpipe("python3" "engine.py" standardfen)

r1bqkbnr/pp1p1ppp/2n1p3/8/3NP3/8/PPP2PPP/RNBQKB1R w KQkq -

B44 Sicilian Defense: Taimanov Variation

#!/usr/bin/python3

import sys

# Initialization: load data from tab-delimited ECO files at startup
eco_dir = "chess-openings/dist/"
eco_files = ("a.tsv", "b.tsv", "c.tsv", "d.tsv", "e.tsv")
eco_dict = dict()                   # Holds FEN -> ECO mappings
opening_dict = dict()               # Holds FEN -> Opening mappings

for eco_file in eco_files:
    with open(eco_dir + eco_file, "r") as fp:
        for line in fp:
            eco_parts = line.strip().split("\t")
            fen = " ".join(eco_parts[4].split()[0:3])
            eco_dict[fen] = eco_parts[0]
            opening_dict[fen] = eco_parts[1]

# Main loop
for line in sys.stdin:
    # fen_part will contain the first three components of the
    # provided FEN string which forms the key into eco_dict.
    fen_part = " ".join(line.strip().split()[0:3])

    # Get ECO and Opening name from partial FEN string.
    # The 'get' method will return the dictionary value that
    # corresponds to the provided key or the empty string.
    eco = eco_dict.get(fen_part, "")
    opening = opening_dict.get(fen_part, "")
    result = (eco + " " + opening).strip() + "\n"

    sys.stdout.write(result)        # Write response with newline
    sys.stdout.flush()              # Flush stdout

cql(quiet)
mainline

$result = commandpipe("python3" "eco-pipe.py" standardfen)
if $result ~~ "^(\S+) (.*)$" {
    settag("ECO" \1)
    settag("Opening" \2)
}

The message Filter

The message filter behaves like the str filter except that the string formed by the concatenation of its arguments is used to form a message that is emitted during processing. message is a Boolean filter which always yields a true value.

By default, the text emitted as a result of evaluating a message filter will be prefixed with the game number and current move information. For example, the query:

initial
not reachableposition
message standardfen

may produce a message that looks like:

Game 32253: move 1(wtm): 6q1/7p/7p/p6p/6kp/p6p/5R1p/5B1K w - - 0 1

If the quiet keyword appears immediately after message, this preamble is omitted:

6q1/7p/7p/p6p/6kp/p6p/5R1p/5B1K w - - 0 1

While message is similar to the comment filter, it does not employ the same Smart Comment semantics that comment does. In particular, a message filter that is evaluated will always result in the message being issued, regardless of whether the position ultimately matches. Comment suppression options do not affect messages.

The message filter can be a useful diagnostic tool by showing the values of variables or other filters at specific points in a query. It can also be used as a convenience utility, e.g. to dump pertinent information without having to consult to matching positions in the corresponding output file. By default, messages are emitted asynchronously but the –noasyncmessages option can be used to disable this behavior which is sometimes useful in multi-threaded mode.

The message filter can also be used in a cqlend block to emit the result of post-processing data that was collected during analysis.

The assert Filter

The assert filter takes a single condition argument. If the condition matches the position, the assert filter yields true. Otherwise a runtime error is emitted and CQLi will terminate prematurely. Information about the game and position being examined at the time of the failed assertion is emitted to aid in debugging efforts.

For example, the filter:

assert isbound numPieces

will produce an error similar to the following when the numPieces variable is unbound at the point where the assert filter is evaluated:

test.cql:5:1 error: assert condition failed at game number 57 positionid 101
assert isbound numPieces
^~~~~~~~~~~~~~~~~~~~~~~~
8 . . ♛ . . ♝ . . 
7 ♖ . ♞ . ♜ ♞ ♚ . 
6 . ♗ ♙ ♟ . ♟ . ♟ 
5 . . . . ♟ ♙ ♟ . 
4 . . . . ♙ . . . 
3 . . . . . . ♙ ♙ 
2 . . . ♘ ♕ . . ♔ 
1 . . ♖ . . . . . 
  a b c d e f g h

Game 57: move 51(btm)
2q2b2/R1n1rnk1/1BPp1p1p/4pPp1/4P3/6PP/3NQ2K/2R5 b - - 8 51

The assert message can be supplemented with additional information by adding or { message  … false } to the condition of the assert filter. For example:

assert x <= 100

will fail if x is greater than 100 but will not include the value of x in the error message. This information can be included by instead using:

assert x <= 100 or { message("x = " x) false }

which will cause the value of x to be emitted immediately before the error triggered by the failed assert condition, e.g:

Game 3487: move 23(wtm): x = 423
test.cql:19:1 error: assert condition failed at game number 348 positionid 0
assert x <= 100 or { message("x = " x) false }}
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

This works because the assert filter first evaluates the LHS of the or condition (x <= 100) and only when that part of the condition fails is the RHS ({ message("x = " x) false}) evaluated because of the short-circuited evaluation of logical operators. During evaluation of the RHS, the value of x is printed by the message filter and the false at the end of the compound filter ensures that the RHS fails as well so that the assertion is still triggered by the failed LHS condition.

Printing the AST

The --parse option will cause CQLi to dump the AST of the parsed CQL query and then exit. This can be useful to determine if the query was parsed as expected. The option can be used with a .cql file or with one or more –cql options (or both). For example, given a file named enpassantecho.cql containing the text:

// Two positions differ only in whether en passant is legal
cql(variations)
move enpassant
echo (source target) {
    sidetomove == source:sidetomove
    not move legal enpassant
    source & target == .
}

when using the --parse option CQLi will emit an AST similar to the following:

QueryContainer {Boolean} <Invalid location>
├─CqlHeader (variations) {Boolean} <enpassantecho.cql:2:1-15>
├─Move (ordinary, enpassant) {Boolean} <enpassantecho.cql:3:1-14>
└─Echo (source slot=0, target slot=1) {Boolean} <enpassantecho.cql:4:1-9:1>
  └─CompoundExpr {Boolean} <enpassantecho.cql:4:22-8:1>
├─EqualToOperator {Numeric} <enpassantecho.cql:5:5-35>
│ ├─SideToMove {Numeric} <enpassantecho.cql:5:5-14>
│ └─ColonOperator {Numeric} <enpassantecho.cql:5:19-35>
│   ├─Identifier (source, slot=0) {Position} <enpassantecho.cql:5:19-24>
│   └─SideToMove {Numeric} <enpassantecho.cql:5:26-35>
├─NotOperator {Boolean} <enpassantecho.cql:6:5-28>
│ └─Move (legal, enpassant) {Boolean} <enpassantecho.cql:6:9-28>
└─EqualToOperator {Boolean} <enpassantecho.cql:7:5-24>
  ├─BitAndOperator {Set} <enpassantecho.cql:7:5-19>
  │ ├─Identifier (source, slot=0) {Position} <enpassantecho.cql:7:5-10>
  │ └─Identifier (target, slot=1) {Position} <enpassantecho.cql:7:14-19>
  └─PieceDesignator '.' {Set} <enpassantecho.cql:7:24>

rotate90 up 1 c3

QueryContainer {Boolean} <Invalid location>
└─Transform (rotate90 4 children) {Set} <test.cql:1:1-16>
  ├─Direction (up) [identity] {Set} <test.cql:1:10-16>
  │ ├─Range {Numeric} <test.cql:1:13>
  │ │ └─Integer '1' {Numeric} <test.cql:1:13>
  │ └─PieceDesignator 'c3' {Set} <test.cql:1:15-16>
  ├─Direction (right) [clockwise90] {Set} <test.cql:1:10-16>
  │ ├─Range {Numeric} <test.cql:1:13>
  │ │ └─Integer '1' {Numeric} <test.cql:1:13>
  │ └─PieceDesignator 'c6' {Set} <test.cql:1:15-16>
  ├─Direction (down) [rotate180] {Set} <test.cql:1:10-16>
  │ ├─Range {Numeric} <test.cql:1:13>
  │ │ └─Integer '1' {Numeric} <test.cql:1:13>
  │ └─PieceDesignator 'f6' {Set} <test.cql:1:15-16>
  └─Direction (left) [counterclockwise90] {Set} <test.cql:1:10-16>
├─Range {Numeric} <test.cql:1:13>
│ └─Integer '1' {Numeric} <test.cql:1:13>
└─PieceDesignator 'f3' {Set} <test.cql:1:15-16>

Colored Output and Unicode

By default, ANSI escape sequences will be used to produce colored effects for the dumped AST tree when using the --parse option. If these sequences do not render properly in the terminal or are undesired (such as when redirecting the output to a file), the --noansicolors option may be used to suppress such sequences from being generated.

Unicode characters are used to represent the chess pieces in the chessboard printed during the presentation of a runtime error and Unicode box-drawing characters are used to print the AST tree when using the --parse option. These characters are emitted by default for Linux and macOS and suppressed by default on Windows. The --consoleunicode and --noconsoleunicode options may be used to enable or disable, respectively, the use of these characters.

The gamenumber Parameter

The gamenumber parameter accepts a range argument with one or two non-negative numeric literals. If two literals are provided, the value of the second must not be less than the value of the first. Processed games are limited to those with game numbers in the provided range. The –gamenumber option may be used to override this parameter.

The input Parameter

The input parameter accepts a single filename argument which may either be a string literal or a series of characters terminated by the first ‘)’ or whitespace character seen. The file designated by the filename argument is used as the input PGN file. The –input option may be used to override this parameter. This parameter is ignored if the –secure option is used.

The matchcount Parameter

The matchcount parameter accepts a range argument with one or two non-negative numeric literals. If two literals are provided, the value of the second must not be less than the value of the first. Only games for which the number of matching positions is within the specified range will be emitted. If zero is included in the specified range, games that do not match the query will be emitted. The –matchcount option may be used to override this parameter.

The matchstring Parameter

The matchstring parameter accepts a single string argument which must be a string literal. The specified string will be used to comment matching positions instead of the default of “CQL”. An empty string may be specified to disable matching comments. The –matchstring option may be used to override this parameter.

The output Parameter

The output parameter accepts a single filename argument which may either be a string literal or a series of characters terminated by the first ‘)’ or whitespace character seen. The file designated by the filename argument is used as the output PGN file. A filename of stdout will cause matching games to be written to standard output. The –output option may be used to override this parameter. Like the --output option, the file specified by this parameter need not have a .pgn extension. This parameter is ignored if the –secure option is used.

The result Parameter

The result parameter accepts a single result argument which is one of the following token sequences: 1-0, 0-1, 1/2-1/2, *, or one of the following string literals: "1-0", "0-1", "1/2-1/2", or "*". Only games that have a result corresponding to the result argument are processed by CQLi. This parameter cannot be overridden by the –result option as the option injects a result filter into the query stream while the result parameter limits the games processed before the CQL query is evaluated.

The quiet Parameter

The quiet parameter does not accept any arguments. Its presence indicates that matching comments and auxiliary comments should not be emitted in matching games. The –silent option may be used to override this parameter.

The silent Parameter

The silent parameter does not accept any arguments. When this option is specified, CQLi will not add any comments to matching games. The –quiet option may be used to override this parameter.

The sort matchcount Parameter

This parameter is identical to the matchcount parameter described above in that it accepts a range which specifies the prerequisite number of matching positions in order for games to be emitted. Additionally, games will be sorted in the output file, in descending order, by the number of matching positions. If the query contains sort filters, those filters take precedence with the match count being the tie-breaker between positions that would otherwise have the same sort order. The –sortmatchcount option provides the equivalent sorting behavior on the commandline. The –matchcount option may be used to override the range associated with this parameter, in such a case the results will still be sorted by the match count.

The variations Parameter

The variations parameter does not accept any arguments. Its presence specifies that processing of variation positions should be enabled. The –mainline option may be used to override this parameter.

Position Attributes

The following Boolean hhdb filter commands may be used to inspect attributes of the current position. The first five commands must be specified as string literals, the last two as keywords.

The comments <cook>, <eg>, <main>, <or>, and <minor_dual> are used in the HHdbVI and HHdbVII databases to mark the positional characteristics described in the above table. The <minor_dual> comment always includes the initials of the person attributed with finding the dual, e.g. <minor_dual MG>. Most <cook> comments will similarly contain the initials of the person credited with discovering the cook. The initials of multiple people may be provided, separated by slashes, e.g. <cook RB/MG/MR>. To extract the initials attached to a <cook> comment, use:

originalcomment ~~ "<cook (.*?)>" \1

Replace cook with minor_dual to accomplish the same for <minor_dual> comments.

Within the HHdbVI and HHdbVII databases, <main> comments are used to mark variations that should be considered part of the mainline for the purpose of the study. An alternate mainline position is one that is not a mainline position but should be treated as one by virtue of a <main> comment appearing in every ancestor position that starts a variation.

Study Attributes

hhdb TE
hhdb theoretical_ending

U2: Zadachy i Etyudy=80 26-6-2020.
U2: Schach/9 1975 U1 U2: Hornecker=S HHdbIII#27364 9-7-2004.
(U2): Ulrichsen=J EG=170 10/2007.

hhdb firstcomment ~~ "\(?U2\)?.*?:(.*?)(\.| \S+:| U\d|$)" \1

hhdb composer

hhdb composer "Arestov=P"

if tag "White" [-3:] == " NN" then
    initialposition: originalcomment ~~ "^([^.]+)"
else
    if initialposition: originalcomment ~~ "^MC:(( [A-Za-z_]+=[A-Z]+)+)" then
        tag "White" + \1
    else tag "White"

initial
hhdb stipulation ~~ "mate in (\d+)"
int \1 > 100

[Black "(+0002.45h8g5) TT (m) U2"]

hhdb gbr == "+0002.45h8g5"
hhdb gbr kings == "h8g5"
hhdb gbr pawns == "45"
hhdb gbr pieces == "0002"
hhdb gbr material == "0002.45"

sort min hhdb sortable

Examples

The documentation for HHdbVII contains several examples of how to use ChessBase to search for certain types of studies. The below table contains equivalent CQLi queries for these examples and the number of matching studies in both the HHdbVI and HHdbVII databases.

HHDB Option Interface

All of the hhdb commands may be accessed from the command line using the --hhdb option which takes one or more arguments consisting of a command and appropriate parameters and injects a corresponding hhdb filter into the composed query. For example:

--hhdb sound

will inject the filter:

hhdb sound

into the query. Commands and parameters that must be presented as string literals in a filter should not be enclosed in quotes when using the --hhdb option, except as necessary for shell escape purposes. For example, use:

--hhdb '<cook>'

instead of:

--hhdb '"<cook>"'

Any commands that yield a String value must be followed by a search parameter (which should not be enclosed in quotes), e.g. to locate studies with a first comment that contains the string “correction” use:

--hhdb firstcomment correction

To inject a filter from the commandline that only matches when there is a first comment use the –cql option instead, e.g.:

--cql 'hhdb firstcomment'