Skip to content

6 Automated Testing of Translation Tables

There are a number of automated tests for liblouis and they are proving to be of tremendous value. When changing the code the developers can run the tests to see if anything broke.

The easiest way to test the translation tables is to write a YAML file where you define the table that is to be tested and any number of words or phrases to translate together with their respective expected translation.

The YAML tests are data driven, i.e. you give the test data, a string to translate and the expected output. The data is in a standard format namely YAML. If you have libyaml installed they will automatically be invoked as part of the standard make check command.

6.1 YAML Tests

YAML is a human readable data serialization format that allows for an easy and compact way to define tests.

A YAML file first defines which tables are to be used for the tests. Then it optionally defines flags such as the ‘testmode’. Finally all the tests are defined.

You can repeat the cycle as many times as you like (tables, optional flags, tests). You can also define several rounds of tests for any table, with or without the optional flags. Just remember that the flags are reset to their default values each time you start a new round of tests or load a new set of tables.

Let’s just look at a simple example how tests could be defined:

# comments start with '#' anywhere on a line
# first define which tables will be used for your tests
table: [unicode.dis, en-ueb-g1.ctb]

# then optionally define flags such as testmode. If no flags are
# defined forward translation is assumed

# now define the tests
tests:
  - # each test is a list.
    # The first item is the string to translate. Quoting of strings is
    # optional
    - hello
    # The second item is the expected translation
    - ⠓⠑⠇⠇⠕
  - # optionally you can define additional parameters in a third
    # item such as typeform or expected failure, etc
    - Hello
    - ⠨⠶⠠⠓⠑⠇⠇⠕⠨⠄
    - {typeform: {italic: '++++ '}, xfail: true}
  - # a simple, no-frills test
    - Good bye
    - ⠠⠛⠕⠕⠙ ⠃⠽⠑
  # same as above using "flow style" notation
  - [Good bye,  ⠠⠛⠕⠕⠙ ⠃⠽⠑]

The four basic components of a test file are as follows:

table

A list containing table files, which the tests should be run against. This is usually just one file, but for some situations more than one file can be required. For example:

table: [hu-hu-g1.ctb, hyph_hu_HU.dic]

It is also possible to specify a table inline. Inline definition of tables below explains how to do this.

A third way to specify a table is by its metadata. A table query, which is essentially as list of “features”, is matched against the table metadata defined inside the tables contained in LOUIS_TABLEPATH. Only the best match is used for the test.

The syntax of the query is a variation of the syntax used for the lou_findTable function:

table:
  locale: fr
  grade: 1
display

A display table, which should be used to encode braille in the test. This item is optional. It is only taken into account for the ‘forward’, ‘backward’ and ‘bothDirections’ test modes. If it is present it should be the first item of the file. If it is not present, the braille encoding of each test is determined by the table that is being tested.

The next example shows how to test the en-ueb-g1.ctb table using ASCII notation (as defined in en-ueb-g1.ctb itself):

table: [en-ueb-g1.ctb]

If you wanted to test the en-ueb-g1.ctb table using Unicode dot patterns then you would use the following definition:

display: unicode.dis
table: [en-ueb-g1.ctb]
flags

The flags that apply for all tests in this file. At the moment only the ‘testmode’ flag is supported. It can have four possible values:

forward

This indicates that the tests are for forward translation.

backward

This indicates that the tests are for backward translation.

bothDirections

Checks that forward translating the input yields the expected output, and backward translating the output yields the input again.

display

Checks that a display table maps characters to the expected dot patterns. The input is a string of characters, the output are the corresponding dot patterns, as a string of Unicode braille symbols. Virtual dots in the output are ignored.

hyphenate

This indicates that the tests are for hyphenation.

hyphenateBraille

This indicates that the tests are for hyphenation and the input is braille.

If no flags are defined forward translation is assumed.

tests

A list of tests. Each test consists of a list of two, three or in some cases four items. The first item is the test input. The second item is the expected output. Braille input and output can be either Unicode braille or an ASCII-braille like encoding, depending on the specified test mode, translation mode and display table. Quoting strings is optional. Comments can be inserted almost anywhere using the ‘#’ sign. A simple test would look at follows:

  - # a simple, no-frills test
    - Good bye
    - ⠠⠛⠕⠕⠙ ⠃⠽⠑

Using the more compact “flow style” notation it would look like the following:

  - [Good bye, ⠠⠛⠕⠕⠙ ⠃⠽⠑]

An optional third item can contain additional options for a test such as the typeform, or whether a test is expected to fail. The following shows a typical example:

  -
    - Hello
    - ⠨⠶⠠⠓⠑⠇⠇⠕⠨⠄
    - {typeform: {italic: '++++ '}, xfail: true}
  # same test more compact
  - [Hello, ⠨⠶⠠⠓⠑⠇⠇⠕⠨⠄, {typeform: {italic: '++++ '}, xfail: true}]
  # same test but is only expected to fail for backtranslation
  - [Hello, ⠨⠶⠠⠓⠑⠇⠇⠕⠨⠄, {typeform: {italic: '++++ '}, xfail: {backward: true}}]

The valid additional options for a test are as follows:

xfail

Sometimes it is known that a test is failing. Maybe the table under test doesn’t handle that word correctly yet, or maybe backtranslation has not been implemented. ‘xfail’ is designed to mark tests as expected failures. There are three ways to mark a test as failing:

  1. Simply Mark a test as expected failure by setting ‘xfail’ to ‘true’.
      - [Hello, ⠨, {xfail: true}]
    
  2. Mark a test as expected failure and give a reason for the failure.
      - [Hello, ⠨, {xfail: Test case is not complete}]
    
  3. Mark a test as expected failure just for backward or forward translation using the notation ‘{xfail: {forward: true}}’ or ‘{xfail: {backward: true}}’. If you expect both to fail use ‘{xfail: {forward: true, backward: true}}’.

    To mark just the backward translation of a test as expected failure use the following:

      - [Hello, ⠨, {xfail: {backward: true}}]
    

    Again a reason for the expected failure can be given.

      - [Hello, ⠨, {xfail: {backward: Not implemented}}]
    

    If you expect both forward and backward translation to fail set both ‘forward’ and ‘backward’ to ‘true’ (or give a reason).

      - [Hello, ⠨, {xfail: {forward: true, backward: true}}]
      # above is equivalent to
      - [Hello, ⠨, {xfail: true}]
    
typeform

The typeform used for a translation. It consists of one or more emphasis specifications. For each character in the specifications that is not a space the corresponding emphasis will be set. Valid options for emphasis are ‘italic’, ‘underline’, ‘bold’, ‘computer_braille’, ‘passage_break’, ‘word_reset’, ‘script’, ‘trans_note’, ‘trans_note_1’, ‘trans_note_2’, ‘trans_note_3’, ‘trans_note_4’ or ‘trans_note_5’. The following shows an example where both ‘italic’ and ‘underline’ are specified:

  -
    - Hello
    - ⠨⠶⠠⠓⠑⠇⠇⠕⠨⠄
    - typeform:
        italic:    '++++ '
        underline: '    +'
inputPos

A list of 0-based input positions, one for each output position. Useful when simulating screen reader interaction, to debug contraction and cursor behavior as in the following example. Note that all positions in this and the following examples start at 0. Also note that in these examples the additional options are not passed using the “flow style” notation.

  -
    - went
    - ⠺⠢⠞
    - inputPos: [0,1,3]
outputPos

A list of 0-based output positions, one for each input position. Useful when simulating screen reader interaction, to debug contraction and cursor behavior as in the following example.

  -
    - went
    - ⠺⠢⠞
    - outputPos: [0,1,1,2]
cursorPos

The cursor position for the given translation and optionally an expected cursor position where the cursor is supposed to be after the translation. Useful when simulating screen reader interaction, to debug contraction and cursor behavior:

The cursor position can take two forms: You can either specify a single number or alternatively you can give a tuple of two numbers.

single number (e.g. ‘4’)

When you simply want to specify the cursor position for the given translation you pass a number as in the following example:

  -
    - you went to
    - ⠽ ⠺⠑⠝⠞ ⠞⠕
    - mode: [compbrlAtCursor]
      cursorPos: 4
a tuple (e.g. ‘[4,2]’)

When you expect the cursor to be in a particular position after the translation and you want to check this then pass a tuple of cursor positions as in the following example:

  -
    - you went to
    - ⠽ ⠺⠑⠝⠞ ⠞⠕
    - mode: [compbrlAtCursor]
      cursorPos: [4,2]
mode

A list of translation modes that should be used for this test. If not defined defaults to 0. Valid mode values are ‘noContractions’, ‘compbrlAtCursor’, ‘dotsIO’, ‘compbrlLeftCursor’, ‘ucBrl’, ‘noUndefined’ or ‘partialTrans’.

For a description of the various translation mode flags, please see the function lou_translateString.

maxOutputLength

Define a maximum length of the output. This can be used to test the behavior of liblouis in the face of a limited output buffer, for example the length of the refreshable braille display.

6.1.1 Optional test description

When a test contains three or four items the first item is assumed to be a test description, the second item is the unicode text to be tested and the third item is the expected braille output. Again an optional fourth item can contain additional options for the test. The following shows an example:

  -
    - Number-text-transitions with italic
    - 123abc
    - ⠼⠁⠃⠉⠨⠶⠰⠁⠃⠉⠨⠄
    - {typeform: '000111'}

In case the test fails the description will be printed together with the expected and the actual braille output.

For more examples and inspiration please see the YAML tests (*.yaml) in the tests directory of the source distribution.

6.1.2 Testing multiple tables within the same YAML test file

Sometimes you are more focused on testing a particular feature across several tables rather than just testing one table. For that reason the following is also allowed:

table: ...
tests:
  - [..., ...]
  - [..., ...]
table: ...
tests:
  - [..., ...]
  - [..., ...]

If you specify flags for the tests, remember that the flags are reset to their default values when you specify a new table.

6.1.3 Multiple test sections for each table

You can specify several sections of tests for each table, with or without the optional flags. This is useful e.g. if you want to have various tests for both forward and backward translation for the same set of tables, especially if you are defining the table as part of the yaml file (see next section). This feature is also useful if you simply want to divide your tests into multiple sections for better overview. All flags are reset to their default values when you start a new test section.

Thus, a yaml file might look as follows:

table: ...
tests:
  - [..., ...]
  - [..., ...]

# Some more tests
  tests:
  - [..., ...]
  - [..., ...]

# Some tests for back-translation - same table
flags: {testmode: backward}
  - [..., ...]
  - [..., ...]

6.1.4 Inline definition of tables

When testing very specific opcode combinations it is sometimes tedious to create specific test tables just for that. Hence the YAML tests allow for specification of table definitions inline. Instead of referring to a table by name you just define the table inline by using what the YAML spec calls a Literal Style Block. Start the definition with a ‘|’, then list the opcodes with an indentation. The inline table ends when the indentation ends.

table: |
  sign a 1
  ...
tests:
  - ...
  - ...

6.1.5 Running the same test data on multiple tables

Sometimes you maintain multiple tables which are very similar and basically contain the same test data. Instead of copying the YAML test and changing the table name you can also define multiple tables. This will cause the YAML tests to be checked against both tables.

table: nl-NL
table: nl-BE
tests:
  - [..., ...]
  - [..., ...]