Appendix A: Differences Between CQL 6.1

New Features in CQLi

CQLi Extensions

CQLi-specific enhancements to existing CQL features.

• Persistent variable and dictionary merge strategy specification to support multi-threaded execution of queries employing them.
• CQLi supports dictionaries with keys and values having Numeric and Set types in addition to String type.
• Conditional set assignment and compound assignment for dictionary members.
• Multi-threaded execution support for queries employing the readfile and writefile filters.
• Reverse move generation using the reverse keyword parameter with the move filter.
• The optional restrict keyword parameter for shift transforms.
• Atomic evaluation of nodes involving variable modifications in line constituents.
• Optional nolinearize and nonatomic keyword parameters for the line filter.
• CQLi allows {?} to be used as an alias for the ? quantifier in line constituents.
• The string "*" as an argument to the result filter to indicate an unfinished or incomplete game.
• The move filter allows combining capture and promote with legal and pseudolegal.
• The move filter allows the count parameter to be used with all forms of the move filter, not just legal or pseudolegal.
• Extended Smart Comments which suppresses comments whenever the enclosing expression does not match including many cases not covered by CQL 6.
• Commit Logging for sort and consecutivemoves which suppresses best-value updates and associated comments when an enclosing expression does not match.
• The year filter will attempt to extract the year from the UTCDate tag in the absence of a Date tag.
• Variables declared in loops and functions have block scope.
• The sort filter does not require the use of a doc-string for sort min.
• The consecutivemoves filter allows any position filter for its arguments, CQL 6.1 requires arguments to be position variables.
• The currentfen filter may be used as an alias for the version of the fen filter that does not receive a string literal argument.
• CQLi supports up to 1000 regular expression backreferences in a pattern, CQL 6.1 supports 100.
• The CQLi-specific noclobber keyword parameter may be used with a writefile filter to append to an existing file.
• CQLi supports multiple comments at a position, CQL 6.1 removes all comments except the last.
• CQLi supports the following string regular expression features not supported by CQL 6.1: named capture groups (including extraction via \{name} and \-{name}), lookbehind assertions, in-pattern comments, possessive matches, optional case-insensitive matching, and the backslash sequences \a, \A, \e, \E, \h, \H, \k, \N, \p, \P, \Q, \R, \U, \V, \X, \z, and \Z.
• The nonspecial and sortable parameters of the hhdb filter.

Implementation Defined Behavior

Known differences in implementation-defined behaviors.

• The first and longest matching sequence reported by the line filter may be different between the implementations when there are multiple matching candidate sequences.
• The portion matched by line constituents involving multiple optional regex repetitions may be different between implementations.

Other Observable Differences

• If two transform keywords are followed by a range, the range applies to their composition in CQL 6.1 but to the most nested transform in CQLi. Thus, shift flip count {a1 a7} yields 64 in CQL 6.1 and 8 in CQLi.
• cql() headers are always optional in CQLi, may appear anywhere a function definition is legal, and multiple headers are allowed (all but the last one is ignored).
• In CQL 6.1 1/2-1/2, 1-0 and 0-1 are each individual tokens which means that x = 1-0+1 results in a parse error which can be rectified by placing whitespace on either side of the -. In CQLi these are not tokens which means that 1) x = 1-0 is not a parse error and 2) result 1 - 0 is treated the same as result 1-0.
• CQL 6.1 employs AST node transformations to accommodate certain filters such as !=. The AST produced by CQLi always represents the query as written without performing such transformations. This difference is noticeable in error messages and when viewing the resulting AST via the --parse option.
• The line filter in CQL 6.1 utilizes a traditional backtracking algorithm for filter repetition which can result in extremely long evaluation times for certain combinations of games and patterns due to a phenomenon known as catastrophic backtracking. CQLi avoids this issue by employing a non-backtracking NFA search algorithm.
• CQLi honors the enpassant square and castling rights specified in the FEN tag of a PGN game.
• Extraneous characters at the end of the FEN string in a fen filter result in an error at parse time.
• CQLi honors the fullmove number specified in the FEN tag of a PGN game. The first move of a game with a FEN tag will start at the move number specified instead of 1. The movenumber filter, position portrayal in e.g. comment and message filters, and the move number indicators in the output PGN file reflect this difference.
• The int filter in CQLi will ignore leading whitespace and non-digit characters following a valid number, in CQL 6.1 the int filter will not match the position for such strings.
• CQL 6.1 allows a backslash literal (e.g. \n) to be used in a context where a string literal is required (e.g. the implicit search string for player, site, event, etc.). CQLi does not support this usage.
• CQLi allows dictionaries to be specified as quiet to indicate their values should not be emitted when using --showdictionaries.
• The tag argument to the settag filter need not be a string literal in CQLi and there are no prohibitions on tags that may be modified with this filter.
• CQLi does not require input or output files to have a prescribed file extension, including those operated on by the readfile and writefile filters.
• CQLi preserves one-line comments in PGN files (those that begin with a semicolon) and converts them to braced comments, CQL 6.1 removes one-line comments.
• CQLi recognizes pass as a null move in the move text of a PGN file.
• When count is used with legal or pseudolegal, promotion moves that differ only in the type of the promoted piece are not counted separately in CQL 6.1 but are in CQLi. The CQL 6.1 behavior may be obtained by subtracting 3/4 of the generated promotions, e.g. move count legal - 3 * move count legal promote A / 4.
• In CQLi, the isbound filter always returns false, and the isunbound filter always returns true, for a variable that does not exist in the lexical scope of the filter as identifier resolution is always performed at parse time. This behavior differs from CQL 6.1 where variables are resolved at run time and e.g. isbound X will return true if the variable X holds a value, even if X is not declared until later.
• In CQL 6.1 an hhdb award filter that does not specify special will only consider special awards if the result of the filter is not used in a numeric context. In CQLi, whether or how the result is used does not have any bearing on the behavior of a filter and an hhdb award filter will always match both special and non-special awards unless one of the special or nonspecial parameters are specified.

New Features

• The --limit range option stops processing when the specified number of matching games are found.
• The --warnlevel / -w option accepts a value of 1, 2, or 3 and sets the warning level to the provided value. (1 = errors only, 2 = errors and warnings, 3 = errors, warnings, and infos).
• The --pgnlinewidth width option specifies the maximum line width of output PGN files.
• The --coalescecomments / --nocoalescecomments options specify whether multiple comments for a single position should be emitted as a single combined comment or as individual comments.
• The --compactmoves / --nocompactmoves options control whether space characters separate the move number indicator from the move in output files (e.g. 1.e4 vs 1. e4).
• The --compactvariations / --nocompactvariations options determine whether space characters appear between variation text and the enclosing parentheses in output files (e.g. 1.e4 (1.d4) vs 1.e4 ( 1.d4 )).
• The --compactcomments / --nocompactcomments options specify if spaces should separate comments from the enclosing braces in output files (e.g. 1.e4 {best by test} vs 1.e4 { best by test }).
• PGN and CQL files specified with relative paths on the command line are searched in the colon-separated list of directories in the CL_PATH environment variable if the file cannot be found in the directory from which CQLi was invoked.
• The --showdictionaries option may be used to cause dictionary values to be emitted at the end of processing with other persistent variable values.
• The --noremovetag option suppresses tag removals via the removetag filter.
• The --secure option rejects queries containing readfile, writefile, or commandpipe filters and causes the input and output CQL header parameters to be ignored.
• The --keepallbest option keeps comments associated with all best matches for the line, sort, and consecutivemoves filters.
• The --nosort option reduces memory consumption by emitting matching games immediately.
• The --noclobber option prevents overwriting existing output files.
• The --createdirectories option automatically creates directories as needed for output files.
• CQLi supports expressive diagnostics which include precise location information and query component highlighting in diagnostics and supports non-fatal warning and info messages.
• CQL 6.1 produces syntax errors for many valid, but suspicious, queries. CQLi accepts such queries after producing a non-fatal warning message.

Extensions

• The --nosmartcomments option is an alias for --alwayscomment.
• PGN files are not required to have a .pgn extension and CQL files are not required to have a .cql extension.

Missing Functionality

• The -gui, -guipgnstdin, and -guipgnstdout options are not supported.