Interpret and Report the Result of Executing the Prolog
It has been a long road: we started with a phrase, parsed it with the ERG, solved the tree, converted to Prolog, and implemented the predicates. This section covers how we actually execute it and decide what to say back.
How To Execute the Prolog
Since we have generated the Prolog and implemented the Prolog, what could possibly be left???
Adding The Proper Verb Predicate
First, as described in the section on writing verb predicates, we have to decide whether the phrase is a query or a command and run either the query predicate or the task predicate for the verb. The ACE Parser provides information about whether a phrase is a command, a proposition, or a query to make detecting the query vs. command cases easy. All that needs to be done is to add the proper predicate right after the event predicate using the conj() predicate. conj just means “conjunction” and does an “and” operation, which just means it executes it the way Prolog normally would:
"there is a rock."
Prolog:
erg(0, d_a_q__xhh(
arg(X9002, var([name(x4), type(reg), pers(3), num(sg)])),
arg(erg(1, d_noun__x(
arg('rock', word('rock')),
arg(X9002, var([name(x4), type(reg), pers(3), num(sg)])))), term),
arg(erg(2, conj(d_be_v_there__ex(
arg(E9003, var([name(e2), index(yes), tense(pres)])),
arg(X9002, var([name(x4), type(reg), pers(3), num(sg)])),
arg(create, var([type(quote)]))),
% <---- This line was added --->
conj(erg(3, query_be_v_there__ex(arg(E9003, var([name(e2), index(yes), tense(pres)]))))))), term),
% ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
arg(Quote9004, var([type(quote)])))).
Handling Free Variables
As described in the section on thing and free variables, we need to wrap freeVariables/2 around any predicates that will try to deserialize a triple containing a variable. Turns out that the verb predicate we decided above is the only place this happens, so we wrap that with freeVariables/2:
"there is a rock."
erg(0, d_a_q__xhh(
arg(X9002, var([name(x4), type(reg), pers(3), num(sg)])),
arg(erg(1, d_noun__x(
arg('rock', word('rock')),
arg(X9002, var([name(x4), type(reg), pers(3), num(sg)])))), term),
arg(erg(2, conj(d_be_v_there__ex(
arg(E9003, var([name(e2), index(yes), tense(pres)])),
arg(X9002, var([name(x4), type(reg), pers(3), num(sg)])),
arg(create, var([type(quote)]))),
% <---- freeVariables was added to this line --->
conj(freeVariables(erg(3, query_be_v_there__ex(arg(E9003, var([name(e2), index(yes), tense(pres)])))), [])))), term),
% ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
arg(Quote9004, var([type(quote)])))).
Adding Error Handling
As described in the section on handling failure, in order to give good responses to the user we need to clear out the “last error” when we run a new query and run the query using a special predicate to capture any new ones. The predicate that does both is called reportAnyErrorStart. It will execute a conjunction (i.e. “and”) of predicates after first clearing the “last error” memory. The above Prolog looks exactly the same except for being surrounded by reportAnyErrorStart([....]):
reportAnyErrorStart([d_a_q__xhh(...)]).
Now…..run the query!
Which of the NxM Solved Trees Was “meant”?
Of the NxM meanings that were generated, there will be a combination of success and failures. At the end, one answer is expected by the user. Here’s the approach I’ve used in Perplexity which has worked out very well to date.
Recall that a single phrase like “every book is in a cave” can have many potential parses and each parse can have many meanings (i.e. scope-resolved trees). That means you can have NxM possible queries that have been run. Which did the user “mean”?
I’ve taken a simple heuristic here that has worked surprisingly well. Go through all NxM queries in the order that the ACE parser returned them – it does a great job of ordering them based on treebank data:
- If the query succeeds (i.e. returns
truein Prolog), then just stop and assume that is what they meant. This works because:- Ace gives a good order so sooner success is more likely to be right
- And anything that actually works really is a valid interpretation of what the user meant so they will at least understand how the system got confused if it turns out it is not quite what they meant
- If we haven’t hit a success yet, remember the result of the first failure. Again, the idea is to rely on ACE’s sorting.
- Keep looping through answers…
If we hit the very end with no successes, report the first failure as the answer.
Next problem: what to report?
What Answer to Give?
I’ve found three cases that I handle differently. The phrase was:
- Not understood or the phrase is in a tense (past, future) that isn’t implemented
- A yes/no question, like a proposition “there is a safe” or a question “is there a safe?”
- A question that needs one or more answers, like “where? what? Who? How many?”
- A command, like “put the book on the table”
Let’s go through each in turn.
1. Answering a Question That Wasn’t Understood
Kind of obvious but: Perplexity just gives an answer that says what the problem is. Easy.
2. Answering a Yes/No Question
All sentences flagged as “propositions” as well as questions that have no free variables (how to detect this is described next) are treated as yes/no questions. Perplexity will see if the query is true or false.
- If true, we return “yes, that is true!”.
- If false, we politely disagree and give the reason. Like: “No, that isn’t true: there isn’t a table on the floor”. The “reason” we give is the one returned by our error handling logic.
Will that be the “right” error? The heuristic Perplexity error handling uses is very good, but not perfect. So, I use the model that, as long as the failure is logically accurate (meaning there are no bugs), go with it. It may not be the best error for a human but it should make logical sense and it is very hard to determine what the best error is.
We return the reason along with “No” because often users ask yes/no questions but expect some detail. “can you see anything?” “No!”. vs “No, it is dark in here”.
To convert the error code into English there is currently a big ol’ if statement that hand-codes the failures as an english sentence and fills in slots with the particular variables or values supplied. Very low tech. It doesn’t use any English generation software in the prototype so answers are often pretty broken English.
Converting the Prolog symbols to English is described later.
3. Answering “Where? What? Who? How Many?” Questions
The current logic for figuring out if a question needs actual answers is to simply look through the predicates for ones which are known to indicate “this requires a list of responses”. The only one I’ve found so far is d_which_q__xhh. If that is found, the variable it introduces is assumed to hold the answers and we fish through the Prolog result and return them. How we convert the items returned to English is described below.
Dealing with logical, but ridiculous, lists of answers
This gets complicated because, as a logic system, some “logical” answers are really verbose and non-sensical to humans.
For example, in response to “what do you see?” you get this:
idBook1, idBook1Bottom, idBook1Inside, idBook1Page1, idBook1Page1Bottom, idBook1Page1Text,
idBook1Page2, idBook1Page2Bottom, idBook1Page2Text, idBook1Top, idCrystal1, idCrystal1Bottom,
idDiamond1, idDiamond1Bottom, idEntrancecave, idEntrancecaveBack, idEntrancecaveBackOpening,
idEntrancecaveBottom, idEntrancecaveInside, idKeyhole1, idKeyhole1Back, idKeyhole1Inside,
idLexi, idLexiBottom, idLexiLeft, idLexiLeftHand, idLexiLeftHandBack, idLexiLeftHandInside,
idLexiRight, idLexiRightHand, idLexiRightHandBack, idLexiRightHandInside, idLock1,
idLock1Back, idLock1Front, idLockFace1, idLockFace1Back, idLockFace1Front, idSafe1,
idSafe1Back, idSafe1Bottom, idSafe1Front, idSafe1Inside, idSafe1Left, idSafe1Right,
idSafe1Top, idTable1, idTable1Bottom, idTable1Top
which includes every surface of every object, as well as the objects themselves. This is technically correct, but…It also exhaustively lists everything as opposed to giving the high level view like a human would. Oh, and there are often duplicates.
So I’ve ended up using a set of heuristics to filter these answers that seems to work well:
- Get rid of duplicates
- If the answer is a mix of concepts and instances, just return the instances
- So you don’t get answers like “A person, a thing and me” when asking “what is Lexi”. You just get “me”
- Don’t list items and their parts (the “what do you see?” problem above)
- Don’t list items and the things inside them (the “just the overview please” problem)
- Finally there are exceptions that always get listed (like doors), so list those regardless
These are the rules that the current prototype uses which work pretty well in practice.
4. Answering Commands
Most commands are just requests to do a thing. If it works out, we respond with “OK.” However, some types of commands like “Look around” may generate answers like query would. The HTN Planner module has HTN Methods that allow these verbs to say “say this __ !” and, in those cases, the system simply does that. Symbols are converted to English using the logic below.
Converting Symbols to English
The items returned from Prolog are just symbols like this:
Where are you?
Returns: idEntrancecave, idWorld1
Perplexity has Prolog predicates that take those symbols and convert them to english using some simple rules:
- Return a proper name
- If there isn’t one return a common name
- If there isn’t one see if anything in its specialization chain has a common name and return that
We’ve Done It!
If you read through all this, hopefully you have some understanding of how to start with an English phrase and actually execute it and “do the right thing” using a logic system like Prolog. It is not perfect as this is an unsolved computer science problem, but, as you’re probably sick of me saying by now, it works surprisingly well in practice!
As I stated at the beginning, there are a lot more challenging areas to explore to actually turn this into a game. Regardless, I was satisfied enough with the result that I plan to do just that.
Stay tuned!