Declarative Type Based Fusion¶

Fusion can be used to combine information from different modalities, e.g. speech and pointing. The DeclarativeTypeBasedFusionRule looks for two tokens that each have a certain type and fuses them into a new token of a third type, which is added to the blackboard. The types need to be specified declaratively.

Example¶

Let's assume a dialog application where the user can ask a robot to bring something to a certain person. The intent and recipient can be conveyed through speech, e.g. by saying "Bring this to Alice", and the object can be selected on a touch screen, e.g. a pizza of a certain sort. The first part of the information is stored in a token of type BringIntent while the second part is stored in a token of type PhysicalObject. These two tokens have to be combined by the fusion rule to create one token containing all the needed information, which will be a token of type BringObject. The result type, in this case BringObject, needs to have two properties whose types match the types of the origin tokens (here: BringIntent and PhysicalObject).

The fusion rule creates a new token that comprises the two origin tokens. See below for an example:

Origin Token 1:

{
    "type": "BringIntent",
    "recipientName": "Alice"
}

{
    "type" : "Pizza",
    "sort" : "Hawaii"
}

{
    "type" : "BringObject",
    "intent": {
                "type": "BringIntent",
                "recipientName": "Alice"
              },
    "object": {
                "type" : "Pizza",
                "sort" : "Hawaii"
              }
}

Assuming that the semantic tree was already defined, the following code snippet shows how such a fusion rule can be defined:

Pattern p1 = new PatternBuilder("BringIntent", this.getKB()).build();
Pattern p2 = new PatternBuilder("PhysicalObject", this.getKB()).build();
Type resultType = this.getKB().getType("BringObject");
long fusionInterval = Duration.ofSeconds(10).toMillis();

Rule rule = new DeclarativeTypeBasedFusionRule(p1, p2, resultType, fusionInterval);
rule.setName("BringObjectFusionRule");
this.getBlackboard().addRule(rule);

The types of the origin tokens are described using a pattern, which could also contain additional constraints apart from the type. The fusion interval specifies a time constraint on the time difference between the two origin tokens. If their time difference is larger than the fusion interval, the tokens will not be fused. This is because two inputs that belong together are usually uttered in a small time frame. If a user says "Bring me this", it would usually not make sense to fuse the intent with an object that the user pointed to 20 minutes ago. A suitable value for the fusion interval might depend on the modalities involved and the given scenario.

Remarks¶

• The fusion rule will only be triggered with the two most recent tokens matching the given types. In the future, there might be an additional option to generate more match combinations, so that the following dialog rules can decide how to interpret the different combinations.
• Tokens that were already used by the fusion rule will not be considered anymore as input tokens for this rule.
• The tokens involved in the fusion process "remember" from which origin tokens they were created or which fusion results were created out of them respectively. This information can be used by dialog rules if desired.