Thursday, May 26, 2016

LUIS and the Bot Framework: A Natural Language Match

My previous Bot Framework posts have concentrated on the sophisticated conversation management facilities of Bot Connector and Bot Builder. While those are amazing capabilities, they don’t offer much more in user interaction than could be handled by a command line utility. For a bot to be truly conversational, it needs to move beyond primitive single word grunts and understand more naturally spoken language.

Language Understanding Intelligent Service (LUIS) is a natural language service that helps your bot be more conversational. It works by building models that classify utterances (language statements) and extracts data from those utterances, providing actionable results for your bot code. The Bot Framework has types that integrate smoothly with LUIS to help your bot understand natural language statements. This post will describe a model I created with LUIS and show you how to integrate that model into a bot.

The LUIS Model

To get started with LUIS, you need to build a model. A LUIS model is a service that uses machine learning to recognize utterances and provide a probability score of how close it thinks that utterance is to Intents and Entities. Intents are things that a user wants to accomplish or obtain information on. Entities are the data points associated with those intents. In your Bot Framework code, you map Intents to methods and use Entities like they were parameters. The LUIS Help pages has an excellent video that explains how to get started, building a LUIS model.

For this demo, I created a ContactInfo model. The purpose of the model is to support a contact management bot. The BotDemos source code for this post includes a LUIS Models folder that contains the ContactInfo.json file that contains the ContactInfo model. This model has a couple intents: None and ChangeInfo. The None intent is the default for catching unrecognized utterances. The ChangeInfo intent supports utterances for when the user wants to change some of their contact information. The ChangeInfo model has a single entity named ContactType, which could be an address, name, phone number, or email. Make sure your train and publish your model before trying to access it via the Bot Framework. After you publish your model, the App Settings on your page will contain an App Id and Subscription Key, which are necessary to add to your Bot code that integrates with LUIS, which is covered next.

Implementing a LuisDialog

To get started with using LUIS, I created a new Bot Framework project. If you don’t know how to do this yet, you can learn from one of my earlier posts, Pig Latin Bot. To start, I created a ContactInfoDialog class that derives from LuisDialog<object> like this:

    [Serializable]
    [LuisModel(
        "<your App ID goes here>",
        "<your subscription key goes here>")]
    public class ContactInfoDialog : LuisDialog<object>
    {
        public const string ContactType = "ContactType";
        string currentEntity = "";
    }

As is typical of Bot Framework dialogs, you need to make it Serializable if you wish to retain state. The LuisModel attribute allows logic in LuisDialog to communicate with LUIS, using the App ID and Subscription Key from your model, ContactInfo in this case. The LuisDialog type parameter is the result type of the dialog, which contains collected information for a stateful bot. Since we aren’t maintaining state in this simple demo bot, I set the LuisDialog type to object. ContactType is the entity we’ll work with and it’s a const to avoid working with strings. Since later code navigates between methods, I’ll keep the name of the current ContactType entity in the currentEntity field. The ContactInfoDialog class has methods that handle intents and you’ll see what happens when LUIS doesn’t recognize an utterance in the next section.

Handling Unrecognized Utterances

You can set up a method in a LuisDialog to handle cases where your LUIS model doesn’t recognize an utterance. Just create the method and decorate it with a LuisIntent attribute containing an empty string. Here’s an example:

        [LuisIntent("")]
        public async Task None(
             IDialogContext context,
             LuisResult result)
        {
            string userUtterance = result.Query;
            await context.PostAsync(
                $"Sorry, I didn't understand \"{userUtterance}\".");
            context.Wait(MessageReceived);
        }

As mentioned, the LuisIntent attribute with the empty string parameter indicates that LUIS didn’t understand what the user said. The None method has IDialogContext and LuisResult parameters. The IDialogContext parameter is the same as used in all dialogs and you can see how the method uses it to call context.PostAsync to let the user know that the bot couldn’t understand them and then context.Wait to wait for the next message. LuisDialog has an implementation of MessageReceived that it uses to interact with LUIS when the next message from the user arrives. You can use LuisResult to extract information about the user’s utterance, as None does by reading result.Query.

Note: You can visit the LUIS model, click on Review Labels, and classify utterances that aren’t part of your training set. This lets you continuously improve your LUIS model with real data.

Next, we’ll handle an utterance that LUIS does recognize.

Handling Valid Intents

To match methods to intents, decorate the method with a LuisIntent attribute containing a parameter matching the intent name from the LUIS model to handle. Then add the logic you need to handle what the user asks for. Here’s an example for the ChangeInfo intent:

        [LuisIntent("ChangeInfo")]
        public async Task ChangeInfo(
            IDialogContext context,
            LuisResult result)
        {
            EntityRecommendation entityRec;
            result.TryFindEntity(ContactType, out entityRec);

            currentEntity = entityRec.Entity;

            PromptDialog.Text(
                context: context,
                resume: ResumeAndHandleTextAsync,
                prompt: 
                    $"What would you like to change your” +
                    $” {currentEntity} to?",
                retry: "I didn't understand. Please try again.");
        }

        public async Task ResumeAndHandleTextAsync(
            IDialogContext context,
            IAwaitable<string> argument)
        {
            string newEntityValue = await argument;

            await context.PostAsync(
                $"Your {currentEntity} is now {newEntityValue}");

            context.Wait(MessageReceived);
        }

The ChangeInfo method handles the LUIS ChangeInfo intent. The IDialogContext and LuisResult parameters serve the same purpose as described earlier, except that this time we need to read the associated entity to implement appropriate logic. Here, the code uses result.TryFIndEntity to obtain a reference to the ContactType entity. In real code, you would want to have more logic for recognition and validation of the entity.

The code performs a prompt for the text to change the bit of contact info to and then waterfalls to ResumeAndHandleTextAsync to work with the user’s response. Again, this code calls context.Wait(MessageRecieved) to continually wait for the user’s input.

Here’s an interaction with ContactInfo bot, showing how it responds to various utterances.

image

Summary

Now you know how to integrate a LUIS model with the Microsoft Bot Framework. Create and publish the model, create a class derived from LuisDialog, add App ID and Subscription Key to the LuisModel attribute, and use the LuisIntent attribute to decorate methods that handle LUIS model intents. For more information on running and testing a bot, you can visit previous posts in my blog. You can also find the code (including the ContactInfo model) on my GitHub site.

 

@JoeMayo

Friday, May 6, 2016

Dynamic FormFlow Forms in Bot Builder

In a previous post on Bot Builder FormFlow, I described how to automatically construct a dialog with just a class, properties, and a few other methods. The demo was a bit rigid because I used enums to define choices. In practice, you won’t always know ahead of time what choices will be available or the choices could change over time. So, you need a data-driven approach. This post shows how to dynamically define those choices and still have the benefits of FormFlow. This is nearly identical to my previous FormFlow post, except for a couple items that I’ll describe in this post.

Specifying the Dynamic Property

The previous FormFlow example defined the Product property as a Product enum. This is a good candidate for a dynamic data-driven property because over time a bug report will need to accommodate all of the products you support. Here’s the updated BugReport class.

    [Serializable]
    public class BugReport
    {
        public string Product { get; set; }

        public List<PlatformOptions> Platform { get; set; }

        public string ProblemDescription { get; set; }
    }

Here you can see the Product is a string so that it can hold the value that the user enters. The following method simulates a data-driven approach to obtaining the values that can be assigned to Product.

        static List<string> GetProducts()
        {
            return new List<string>
            {
                "Office",
                "SQL Server",
                "Visual Studio"
            };
        }

The GetProducts method returns a collection of strings that can be assigned to the Product property. Now, let’s see how to use these strings in a FormFlow dialog.

Adding a Dynamic Field

In this demo, we’ll modify the BuildForm method, which is a member of the BugReport class. In the previous example, the BuildForm implementation was pretty simple by showing a message, specifying an OnCompletionAsync handler, and initiating the dialog. The difference in this demo is that the code specifically directs which fields/properties to display. Here’s the changed BuildForm method.

        public static IForm<BugReport> BuildForm()
        {
            return new FormBuilder<BugReport>()
                    .Message("Welcome to Bug Report bot!")
                    .Field(new FieldReflector<BugReport>(nameof(Product))
                            .SetType(null)
                            .SetDefine((state, field) =>
                            {
                                foreach (var prod in GetProducts())
                                    field
                                        .AddDescription(prod, prod)
                                        .AddTerms(prod, prod);

                                return Task.FromResult(true);
                            }))
                    .Field(nameof(Platform))
                    .AddRemainingFields()
                    .OnCompletionAsync(async (context, bugReport) => 
                     {
                        await context.PostAsync("Thanks for the report!");
                     })
                    .Build();
        }

In this code, the Field methods describe exactly which properties to display and in what order. FormFlow uses reflection, which the FieldReflector class helps with. The SetDefine method dynamically defines the field. It uses the GetProducts method, shown earlier, and iterates through the list of strings. This is just a demo, so AddDescription and AddTerms use the same string as the parameter key and value. If this was a real bot, GetProducts would read from a database table and return a custom class that contained all of the meta-data required to populate description and terms with more useful values.

Since FormBuilder is a fluent interface, you can continue specifying fields to add and optionally use AddRemainingFields for FormFlow to work normally on any following fields. The rest of the program works the same ways as my previous post on FormFlow.

Summary

This post explained why the default FormFlow wasn’t flexible enough for all scenarios. I defined a string type property and a method that returned all the values that could be assigned to that property. Then you saw how to modify the BuildForm method to specify fields and dynamically define a field with custom data. Now, you’re able to make a bot more adaptable to changes with dynamic fields.

The code for this post is available via my BotDemos GitHub repository.

@JoeMayo

Monday, May 2, 2016

Bot Builder FormFlow

In a previous post on Bot Builder Dialogs, I discussed how you could manage conversations with users. While this wasn’t too hard, consider the amount of complexity involved with a larger conversation. To help simplify the conversation, Bot Builder has another library named FormFlow. In this post, I’ll introduce FormFlow and show you how it can help simplify bot conversations.

What is FormFlow?

FormFlow is a Bot Builder SDK library that lets you declare the type of information you need and then it does the bulk of the work of managing the conversation and moving the user from question to question automatically. In Dialogs, you have the responsibility of writing the methods to prompt the user and collect the results, but FormFlow simplifies the entire process. Another benefit of FormFlow is that it automatically formats questions to be more readable and it tries to process partial answers if it can. Here’s a screen shot of the Bug Report Bot.

image

To implement FormFlow, define a class with properties and/or methods, decorate the class and members with attributes for customization, and initialize the form when the user communicates with your bot. Because this is only an intro, I’ll limit the scope of the demo to only show how to get FormFlow working and some basic control flow. The demo program is a modification of the Bug Report example from my Bot Builder Dialogs post. The next section explains how to create a form.

Creating a FormFlow Form

As mentioned earlier, a Form is a class with fields and/or properties to hold the information you need. For the Bug Report demo, I just need to know the product, platform, and description, as shown in the following code.

    [Serializable]
    public class BugReport
    {
        public ProductOptions Product { get; set; }

        public List<PlatformOptions> Platform { get; set; }

        public string ProblemDescription { get; set; }
    }

To properly manage state, your form must be serializable, which is why the BugReport class has a Serializable attribute. The property types are enums, with a member for each valid choice a user could make. Each property holds the value of the choice that the user makes. If the choice type is a List, FormFlow allows the user to enter multiple items for that answer.

One of the benefits of FormFlow is it’s conventions for readable options. e.g. When the bot asks the user for the value it will place into the ProblemDescription property, it will break words by capitalization and print “Problem Description” to the user. Same thing with the enum values.

Configuring the Form

To make the BugReport class a form, instantiate a FormBuilder and tell it the type of form to work with.  Here’s a basic example of how to do that.

        public static IForm<BugReport> BuildForm()
        {
            return new FormBuilder<BugReport>()
                    .Message("Welcome to Bug Report bot!")
                    .OnCompletionAsync(async (context, bugReport) => 
                     {
                        await context.PostAsync("Thanks for the report!");
                     })
                    .Build();
        }

The BuildForm method returns an FormBuilder<BugReport> instance. The FormBuilder gives you a fluent interface, where you can chain several methods together. Message shows a message to the user, which is when the bot starts communicating. There are many methods you could add to this chain to customize the FormFlow conversation and there’s more explanation for those that I won’t cover in this post. FormBuilder will manage the conversation, using the BugReport class.

When the bot finishes, the lambda passed to OnCompletionAsync communicates with the user again. This is where you could take action to do something with the report, like save it in a database or submit to a Web service. The context is the DialogContext, as explained in my Bot Builder Dialogs post and the lambda uses context to send a final message to the user. The lambda’s bugReport parameter holds the BugReport class instance, holding the state that FormFlow collected, which you can use to extract user answers.

I put the BuildForm method inside the BugReport class. The next section shows how to call BuildForm to start bot communication.

Starting the FormFlow Conversation

Now that you have a form (class used as a form) and that form is configured with FormBuilder, you need to start the the conversation. The following example is from the MessageController and shows the flow of control that occurs when the user communicates with your bot.

    public class MessagesController : ApiController
    {
        internal static IDialog<BugReport> MakeRootDialog()
        {
            return Chain.From(() => FormDialog.FromForm(BugReport.BuildForm))
                        .Loop();
        }

        public async Task<Message> Post([FromBody]Message message)
        {
            return await Conversation.SendAsync(message, MakeRootDialog);
        }
    }

Just as with Dialog conversations, the Post handler method in MessagesController calls Conversation.SendAsync. This argument references a method, MakeRootDialog, to instantiate the dialog. Inside the Chain.From lambda, FormDialog.FromForm references the BuildForm method, from the previous section. It’s saying that as the dialog for this conversation, use the BugReport form.

Chain is interesting in that it has convenience methods for managing forms. This example only uses From and Loop, but there are more that help manage more sophisticated conversations. Loop allows the user to start at the beginning again after the current BugReport form has completed.

Summary

Now you’ve seen how FormFlow makes it easy to manage conversations. You create a class with properties, configure that class with a FormBuilder, and create an instance of the FormBuilder when the user communicates with your bot. In addition to knowing how to start the conversation, you also saw how to handle the end of the conversation via the OnCompletionAsync  method.

 

@JoeMayo