Building Boba AI

Templated Prompt

Use a text template to enrich a prompt with context and structure

The first and simplest pattern is using a string templates for the prompts, also
known as chaining. We use Langchain, a library that provides a standard
interface for chains and end-to-end chains for common applications out of
the box. If you’ve used a Javascript templating engine, such as Nunjucks,
EJS or Handlebars before, Langchain provides just that, but is designed specifically for
common prompt engineering workflows, including features for function input variables,
few-shot prompt templates, prompt validation, and more sophisticated composable chains of prompts.

For example, to brainstorm potential future scenarios in Boba, you can
enter a strategic prompt, such as “Show me the future of payments” or even a
simple prompt like the name of a company. The user interface looks like
this:

The prompt template that powers this generation looks something like
this:

You are a visionary futurist. Given a strategic prompt, you will create
{num_scenarios} futuristic, hypothetical scenarios that happen
{time_horizon} from now. Each scenario must be a {optimism} version of the
future. Each scenario must be {realism}.

Strategic prompt: {strategic_prompt}

As you can imagine, the LLM’s response will only be as good as the prompt
itself, so this is where the need for good prompt engineering comes in.
While this article is not intended to be an introduction to prompt
engineering, you will notice some techniques at play here, such as starting
by telling the LLM to Adopt a
Persona,
specifically that of a visionary futurist. This was a technique we relied on
extensively in various parts of the application to produce more relevant and
useful completions.

As part of our test-and-learn prompt engineering workflow, we found that
iterating on the prompt directly in ChatGPT offers the shortest path from
idea to experimentation and helps build confidence in our prompts quickly.
Having said that, we also found that we spent way more time on the user
interface (about 80%) than the AI itself (about 20%), specifically in
engineering the prompts.

We also kept our prompt templates as simple as possible, devoid of
conditional statements. When we needed to drastically adapt the prompt based
on the user input, such as when the user clicks “Add details (signals,
threats, opportunities)”, we decided to run a different prompt template
altogether, in the interest of keeping our prompt templates from becoming
too complex and hard to maintain.

Structured Response

Tell the LLM to respond in a structured data format

Almost any application you build with LLMs will most likely need to parse
the output of the LLM to create some structured or semi-structured data to
further operate on on behalf of the user. For Boba, we wanted to work with
JSON as much as possible, so we tried many different variations of getting
GPT to return well-formed JSON. We were quite surprised by how well and
consistently GPT returns well-formed JSON based on the instructions in our
prompts. For example, here’s what the scenario generation response
instructions might look like:

You will respond with only a valid JSON array of scenario objects.
Each scenario object will have the following schema:
    "title": <string>,       //Must be a complete sentence written in the past tense
    "summary": <string>,   //Scenario description
    "plausibility": <string>,  //Plausibility of scenario
    "horizon": <string>

We were equally surprised by the fact that it could support fairly complex
nested JSON schemas, even when we described the response schemas in pseudo-code.
Here’s an example of how we might describe a nested response for strategy
generation:

You will respond in JSON format containing two keys, "questions" and "strategies", with the respective schemas below:
    "questions": [<list of question objects, with each containing the following keys:>]
      "question": <string>,           
      "answer": <string>             
    "strategies": [<list of strategy objects, with each containing the following keys:>]
      "title": <string>,               
      "summary": <string>,             
      "problem_diagnosis": <string>, 
      "winning_aspiration": <string>,   
      "where_to_play": <string>,        
      "how_to_win": <string>,           
      "assumptions": <string>          

An interesting side effect of describing the JSON response schema was that we
could also nudge the LLM to provide more relevant responses in the output. For
example, for the Creative Matrix, we want the LLM to think about many different
dimensions (the prompt, the row, the columns, and each idea that responds to the
prompt at the intersection of each row and column):

By providing a few-shot prompt that includes a specific example of the output
schema, we were able to get the LLM to “think” in the right context for each
idea (the context being the prompt, row and column):

You will respond with a valid JSON array, by row by column by idea. For example:

If Rows = "row 0, row 1" and Columns = "column 0, column 1" then you will respond
with the following:

[
  {{
    "row": "row 0",
    "columns": [
      {{
        "column": "column 0",
        "ideas": [
          {{
            "title": "Idea 0 title for prompt and row 0 and column 0",
            "description": "idea 0 for prompt and row 0 and column 0"
          }}
        ]
      }},
      {{
        "column": "column 1",
        "ideas": [
          {{
            "title": "Idea 0 title for prompt and row 0 and column 1",
            "description": "idea 0 for prompt and row 0 and column 1"
          }}
        ]
      }},
    ]
  }},
  {{
    "row": "row 1",
    "columns": [
      {{
        "column": "column 0",
        "ideas": [
          {{
            "title": "Idea 0 title for prompt and row 1 and column 0",
            "description": "idea 0 for prompt and row 1 and column 0"
          }}
        ]
      }},
      {{
        "column": "column 1",
        "ideas": [
          {{
            "title": "Idea 0 title for prompt and row 1 and column 1",
            "description": "idea 0 for prompt and row 1 and column 1"
          }}
        ]
      }}
    ]
  }}
]

We could have alternatively described the schema more succinctly and
generally, but by being more elaborate and specific in our example, we
successfully nudged the quality of the LLM’s response in the direction we
wanted. We believe this is because LLMs “think” in tokens, and outputting (ie
repeating) the row and column values before outputting the ideas provides more
accurate context for the ideas being generated.

At the time of this writing, OpenAI has released a new feature called
Function
Calling, which
provides a different way to achieve the goal of formatting responses. In this
approach, a developer can describe callable function signatures and their
respective schemas as JSON, and have the LLM return a function call with the
respective parameters provided in JSON that conforms to that schema. This is
particularly useful in scenarios when you want to invoke external tools, such as
performing a web search or calling an API in response to a prompt. Langchain
also provides similar functionality, but I imagine they will soon provide native
integration between their external tools API and the OpenAI function calling
API.

Real-Time Progress

Stream the response to the UI so users can monitor progress

One of the first few things you’ll realize when implementing a graphical
user interface on top of an LLM is that waiting for the entire response to
complete takes too long. We don’t notice this as much with ChatGPT because
it streams the response character by character. This is an important user
interaction pattern to keep in mind because, in our experience, a user can
only wait on a spinner for so long before losing patience. In our case, we
didn’t want the user to wait more than a few seconds before they started
seeing a response, even if it was a partial one.

Hence, when implementing a co-pilot experience, we highly recommend
showing real-time progress during the execution of prompts that take more
than a few seconds to complete. In our case, this meant streaming the
generations across the full stack, from the LLM back to the UI in real-time.
Fortunately, the Langchain and OpenAI APIs provide the ability to do just
that:

const chat = new ChatOpenAI({
  temperature: 1,
  modelName: 'gpt-3.5-turbo',
  streaming: true,
  callbackManager: onTokenStream ?
    CallbackManager.fromHandlers({
      async handleLLMNewToken(token) {
        onTokenStream(token)
      },
    }) : undefined
});

This allowed us to provide the real-time progress needed to create a smoother
experience for the user, including the ability to stop a generation
mid-completion if the ideas being generated did not match the user’s
expectations:

However, doing so adds a lot of additional complexity to your application
logic, especially on the view and controller. In the case of Boba, we also had
to perform best-effort parsing of JSON and maintain temporal state during the
execution of an LLM call. At the time of writing this, some new and promising
libraries are coming out that make this easier for web developers. For example,
the Vercel AI SDK is a library for building
edge-ready AI-powered streaming text and chat UIs.

Select and Carry Context

Capture and add relevant context information to subsequent action

One of the biggest limitations of a chat interface is that a user is
limited to a single-threaded context: the conversation chat window. When
designing a co-pilot experience, we recommend thinking deeply about how to
design UX affordances for performing actions within the context of a
selection, similar to our natural inclination to point at something in real
life in the context of an action or description.

Select and Carry Context allows the user to narrow or broaden the scope of
interaction to perform subsequent tasks – also known as the task context. This is typically
done by selecting one or more elements in the user interface and then performing an action on them.
In the case of Boba, for example, we use this pattern to allow the user to have
a narrower, focused conversation about an idea by selecting it (eg a scenario, strategy or
prototype concept), as well as to select and generate variations of a
concept. First, the user selects an idea (either explicitly with a checkbox or implicitly by clicking a link):

Then, when the user performs an action on the selection, the selected item(s) are carried over as context into the new task,
for example as scenario subprompts for strategy generation when the user clicks “Brainstorm strategies and questions for this scenario”,
or as context for a natural language conversation when the user clicks Explore:

Depending on the nature and length of the context
you wish to establish for a segment of conversation/interaction, implementing
Select and Carry Context can be anywhere from very easy to very difficult. When
the context is brief and can fit into a single LLM context window (the maximum
size of a prompt that the LLM supports), we can implement it through prompt
engineering alone. For example, in Boba, as shown above, you can click “Explore”
on an idea and have a conversation with Boba about that idea. The way we
implement this in the backend is to create a multi-message chat
conversation:

const chatPrompt = ChatPromptTemplate.fromPromptMessages([
  HumanMessagePromptTemplate.fromTemplate(contextPrompt),
  HumanMessagePromptTemplate.fromTemplate("{input}"),
]);
const formattedPrompt = await chatPrompt.formatPromptValue({
  input: input
})

Another technique of implementing Select and Carry Context is to do so within
the prompt by providing the context within tag delimiters, as shown below. In
this case, the user has selected multiple scenarios and wants to generate
strategies for those scenarios (a technique often used in scenario building and
stress testing of ideas). The context we want to carry into the strategy
generation is collection of selected scenarios:

Your questions and strategies must be specific to realizing the following
potential future scenarios (if any)
  <scenarios>
    {scenarios_subprompt}
  </scenarios>

However, when your context outgrows an LLM’s context window, or if you need
to provide a more sophisticated chain of past interactions, you may have to
resort to using external short-term memory, which typically involves using a
vector store (in-memory or external). We’ll give an example of how to do
something similar in Embedded External Knowledge.

If you want to learn more about the effective use of selection and
context in generative applications, we highly recommend a talk given by
Linus Lee, of Notion, at the LLMs in Production conference: “Generative Experiences Beyond Chat”.

Contextual Conversation

Allow direct conversation with the LLM within a context.

This is a special case of Select and Carry Context.
While we wanted Boba to break out of the chat window interaction model
as much as possible, we found that it is still very useful to provide the
user a “fallback” channel to converse directly with the LLM. This allows us
to provide a conversational experience for interactions we don’t support in
the UI, and support cases when having a textual natural language
conversation does make the most sense for the user.

In the example below, the user is chatting with Boba about a concept for
personalized highlight reels provided by Rogers Sportsnet. The complete
context is mentioned as a chat message (“In this concept, Discover a world of
sports you love…”), and the user has asked Boba to create a user journey for
the concept. The response from the LLM is formatted and rendered as Markdown:

When designing generative co-pilot experiences, we highly recommend
supporting contextual conversations with your application. Make sure to
offer examples of useful messages the user can send to your application so
they know what kind of conversations they can engage in. In the case of
Boba, as shown in the screenshot above, those examples are offered as
message templates under the input box, such as “Can you be more
specific?”

Out-Loud Thinking

Tell LLM to generate intermediate results while answering

While LLMs don’t actually “think”, it’s worth thinking metaphorically
about a phrase by Andrei Karpathy of OpenAI: “LLMs ‘think’ in
tokens.” What he means by this
is that GPTs tend to make more reasoning errors when trying to answer a
question right away, versus when you give them more time (i.e. more tokens)
to “think”. In building Boba, we found that using Chain of Thought (CoT)
prompting, or more specifically, asking for a chain of reasoning before an
answer, helped the LLM to reason its way toward higher-quality and more
relevant responses.

In some parts of Boba, like strategy and concept generation, we ask the
LLM to generate a set of questions that expand on the user’s input prompt
before generating the ideas (strategies and concepts in this case).

While we display the questions generated by the LLM, an equally effective
variant of this pattern is to implement an internal monologue that the user is
not exposed to. In this case, we would ask the LLM to think through their
response and put that inner monologue into a separate part of the response, that
we can parse out and ignore in the results we show to the user. A more elaborate
description of this pattern can be found in OpenAI’s GPT Best Practices
Guide, in the
section Give GPTs time to
“think”

As a user experience pattern for generative applications, we found it helpful
to share the reasoning process with the user, wherever appropriate, so that the
user has additional context to iterate on the next action or prompt. For
example, in Boba, knowing the kinds of questions that Boba thought of gives the
user more ideas about divergent areas to explore, or not to explore. It also
allows the user to ask Boba to exclude certain classes of ideas in the next
iteration. If you do go down this path, we recommend creating a UI affordance
for hiding a monologue or chain of thought, such as Boba’s feature to toggle
examples shown above.

Iterative Response

Provide affordances for the user to have a back-and-forth
interaction with the co-pilot

LLMs are bound to either misunderstand the user’s intent or simply
generate responses that don’t meet the user’s expectations. Hence, so is
your generative application. One of the most powerful capabilities that
distinguishes ChatGPT from traditional chatbots is the ability to flexibly
iterate on and refine the direction of the conversation, and hence improve
the quality and relevance of the responses generated.

Similarly, we believe that the quality of a generative co-pilot
experience depends on the ability of a user to have a fluid back-and-forth
interaction with the co-pilot. This is what we call the Iterate on Response
pattern. This can involve several approaches:

• Correcting the original input provided to the application/LLM
• Refining a part of the co-pilot’s response to the user
• Providing feedback to nudge the application in a different direction

One example of where we’ve implemented Iterative Response
in
Boba is in Storyboarding. Given a prompt (either brief or elaborate), Boba
can generate a visual storyboard, which includes multiple scenes, with each
scene having a narrative script and an image generated with Stable
Diffusion. For example, below is a partial storyboard describing the experience of a
“Hotel of the Future”:

Since Boba uses the LLM to generate the Stable Diffusion prompt, we don’t
know how good the images will turn out–so it’s a bit of a hit or miss with
this feature. To compensate for this, we decided to provide the user the
ability to iterate on the image prompt so that they can refine the image for
a given scene. The user would do this by simply clicking on the image,
updating the Stable Diffusion prompt, and pressing Done, upon which Boba
would generate a new image with the updated prompt, while preserving the
rest of the storyboard:

Another example Iterative Response that we
are currently working on is a feature for the user to provide feedback
to Boba on the quality of ideas generated, which would be a combination
of Select and Carry Context and Iterative Response. One
approach would be to give a thumbs up or thumbs down on an idea, and
letting Boba incorporate that feedback into a new or next set of
recommendations. Another approach would be to provide conversational
feedback in the form of natural language. Either way, we would like to
do this in a style that supports reinforcement learning (the ideas get
better as you provide more feedback). A good example of this would be
Github Copilot, which demotes code suggestions that have been ignored by
the user in its ranking of next best code suggestions.

We believe that this is one of the most important, albeit
generically-framed, patterns to implementing effective generative
experiences. The challenging part is incorporating the context of the
feedback into subsequent responses, which will often require implementing
short-term or long-term memory in your application because of the limited
size of context windows.

Embedded External Knowledge

Combine LLM with other information sources to access data beyond
the LLM’s training set

As alluded to earlier in this article, oftentimes your generative
applications will need the LLM to incorporate external tools (such as an API
call) or external memory (short-term or long-term). We ran into this
scenario when we were implementing the Research feature in Boba, which
allows users to answer qualitative research questions based on publicly
available information on the web, for example “How is the hotel industry
using generative AI today?”:

To implement this, we had to “equip” the LLM with Google as an external
web search tool and give the LLM the ability to read potentially long
articles that may not fit into the context window of a prompt. We also
wanted Boba to be able to chat with the user about any relevant articles the
user finds, which required implementing a form of short-term memory. Lastly,
we wanted to provide the user with proper links and references that were
used to answer the user’s research question.

The way we implemented this in Boba is as follows:

1. Use a Google SERP API to perform the web search based on the user’s query
   and get the top 10 articles (search results)
2. Read the full content of each article using the Extract API
3. Save the content of each article in short-term memory, specifically an
   in-memory vector store. The embeddings for the vector store are generated using
   the OpenAI API, and based on chunks of each article (versus embedding the entire
   article itself).
4. Generate an embedding of the user’s search query
5. Query the vector store using the embedding of the search query
6. Prompt the LLM to answer the user’s original query in natural language,
   while prefixing the results of the vector store query as context into the LLM
   prompt.

This may sound like a lot of steps, but this is where using a tool like
Langchain can speed up your process. Specifically, Langchain has an
end-to-end chain called VectorDBQAChain, and using that to perform the
question-answering took only a few lines of code in Boba:

const researchArticle = async (article, prompt) => {
  const model = new OpenAI({});
  const text = article.text;
  const textSplitter = new RecursiveCharacterTextSplitter({ chunkSize: 1000 });
  const docs = await textSplitter.createDocuments([text]);
  const vectorStore = await HNSWLib.fromDocuments(docs, new OpenAIEmbeddings());
  const chain = VectorDBQAChain.fromLLM(model, vectorStore);
  const res = await chain.call({
    input_documents: docs,
    query: prompt + ". Be detailed in your response.",
  });
  return { research_answer: res.text };
};

The article text contains the entire content of the article, which may not
fit within a single prompt. So we perform the steps described above. As you can
see, we used an in-memory vector store called HNSWLib (Hierarchical Navigable
Small World). HNSW graphs are among the top-performing indexes for vector
similarity search. However, for larger scale use cases and/or long-term memory,
we recommend using an external vector DB like Pinecone or Weaviate.

We also could have further streamlined our workflow by using Langchain’s
external tools API to perform the Google search, but we decided against it
because it offloaded too much decision making to Langchain, and we were getting
mixed, slow and harder-to-parse results. Another approach to implementing
external tools is to use Open AI’s recently released Function Calling
API, which we
mentioned earlier in this article.

To summarize, we combined two distinct techniques to implement Embedded External Knowledge:

1. Use External Tool: Search and read articles using Google SERP and Extract
   APIs
2. Use External Memory: Short-term memory using an in-memory vector store
   (HNSWLib)