How the language model works for dialog applications

Google creating a language model is not something new; in fact, Google LaMDA joins the likes of BERT and MUM as a way for machines to better understand user intent.

Google studied language-based models for several years in hopes of training a model that could essentially hold an insightful and logical conversation on any topic.

So far, Google LaMDA seems to be the closest to reaching this milestone.

What is Google LaMDA?

LaMDA, which stands for Language Models for Dialog Application, was created to enable software to better engage in fluent and natural conversation.

LaMDA is based on the same transformer architecture as other language models such as BERT and GPT-3.

However, thanks to her training, LaMDA can understand nuanced questions and conversations covering several different topics.

With other models, due to the open nature of the conversations, you might end up talking about something completely different, even if you initially focus on one topic.

This behavior can easily confuse most conversational models and chatbots.

At last year’s Google I/O announcement, we saw that LaMDA was designed to overcome these issues.

The demo proved how the model could naturally lead a conversation on a random topic.

Despite the flood of loosely associated questions, the conversation stayed on track, which was amazing to see.

How does LaMDA work?

LaMDA was built on top of Google’s open-source neural network, Transformer, which is used for natural language understanding.

The model is trained to find patterns in sentences, correlations between different words used in those sentences, and even predict which word is likely to come next.

It does this by studying datasets made up of dialogue rather than just individual words.

Although a conversational AI system is similar to chatbot software, there are a few key differences between the two.

For example, chatbots are trained on limited and specific sets of data and can only have a limited conversation based on the exact data and questions they are trained on.

On the other hand, since LaMDA is trained on several different datasets, it can have open conversations.

During the training process, he grasps the nuances of open dialogue and adapts.

It can answer questions on many different topics, depending on how the conversation is going.

Therefore, it enables conversations even more similar to human interaction than chatbots can often provide.

How is LaMDA formed?

Google explained that LaMDA has a two-step training process, including pre-training and fine-tuning.

In total, the model is trained on 1.56 trillion words with 137 billion parameters.

Pre-training

For the pre-training phase, the Google team created a dataset of 1.56 T words from several public web documents.

This dataset is then tokenized (transformed into a string of characters to make sentences) into 2.81T tokens, on which the model is initially trained.

During pre-training, the model uses general and scalable parallelization to predict the next part of the conversation based on previous tokens it has seen.

Fine tuning

LaMDA is trained to perform generation and classification tasks during the debug phase.

Essentially, the LaMDA generator, which predicts the next part of the dialogue, generates multiple relevant responses based on the back and forth conversation.

LaMDA classifiers will then predict the safety and quality scores for each possible answer.

Any response with a low security score is filtered out before the highest rated response is selected to continue the conversation.

Scores are based on safety, sensitivity, specificity and percentages of interest.

Image from Google AI blog, March 2022

The goal is to ensure that the most relevant, high quality and ultimately safest response is provided.

LaMDA Objectives and Key Metrics

Three main objectives for the model have been defined to guide the training of the model.

These are quality, safety and anchoring.

Quality

This is based on three dimensions of the human evaluator:

  • Sensitivity.
  • Specificity
  • Interest.

The quality score is used to ensure that an answer makes sense in the context in which it is used, is specific to the question asked, and is considered insightful enough to create better dialogue.

Security

To ensure safety, the model follows Responsible AI standards. A set of security goals is used to capture and examine model behavior.

This ensures that the output provides no unintended responses and avoids bias.

Rooting

Embeddedness is defined as “the percentage of responses containing statements about the outside world”.

This is used to ensure that answers are as “factually accurate as possible, allowing users to judge the validity of an answer based on the reliability of its source”.

Evaluation

Through an ongoing process of progress quantification, responses from the pre-trained model, refined model, and human raters are examined to assess responses against the aforementioned quality, safety, and anchoring measures.

So far, they have been able to conclude that:

  • Quality metrics improve with the number of parameters.
  • Security improves with fine tuning.
  • Anchorage improves as model size increases.
LaMDA ProgressImage from Google AI blog, March 2022

How will LaMDA be used?

Although still a work in progress with no firm release date, it is expected that LaMDA will be used in the future to improve customer experience and enable chatbots to provide a more human-like conversation.

Also, using LaMDA to navigate search in Google’s search engine is a real possibility.

Implications of LaMDA for SEO

By focusing on linguistic and conversational models, Google offers insight into their vision for the future of search and highlights a shift in how their products are set to develop.

This ultimately means that there could well be a shift in search behavior and how users search for products or information.

Google is constantly working to improve the understanding of users’ search intent to ensure they receive the most useful and relevant results in the SERPs.

The LaMDA model will undoubtedly be a key tool in understanding the questions that researchers may ask.

All of this further underscores the need to ensure content is optimized for humans rather than search engines.

Ensuring content is conversational and written with your target audience in mind means that even as Google progresses, content can continue to perform well.

It’s also essential to refresh ongoing content regularly to ensure it changes with the times and stays relevant.

In an article titled Rethinking research: turning dilettantes into expertsGoogle’s search engineers have shared how they envision AI advancements like LaMDA will further improve “search as a conversation with experts.”

They shared an example around the research question: “What are the health benefits and risks of red wine?”

Currently, Google will show a list of bulleted answers as answers to this question.

However, they suggest that in the future, an answer might well be a paragraph explaining the benefits and risks of red wine, with links to source information.

Therefore, ensuring that content is backed up by expert sources will be more important than ever if Google LaMDA generates search results in the future.

Overcome Challenges

As with any AI model, there are challenges to overcome.

The two main challenges engineers face with Google LaMDA are security and anchoring.

Safety – Avoiding Bias

Since you can pull responses from anywhere on the web, the output may amplify biases, reflecting notions shared online.

It is important that the responsibility goes first with Google LaMDA to ensure that it does not generate unpredictable or harmful results.

To help overcome this, Google has opened up the resources used to analyze and train the data.

This allows diverse groups to participate in creating the datasets used to train the model, help identify any biases that exist, and minimize the sharing of harmful or misleading information.

Evidence base

It is not easy to validate the reliability of the answers produced by the AI ​​models, since the sources are collected all over the web.

To overcome this challenge, the team allows the model to consult multiple external sources, including information retrieval systems and even a calculator, to provide accurate results.

The previously shared Groundedness metric also ensures that answers are based on known sources. These sources are shared to allow users to validate the results given and to avoid the dissemination of false information.

What’s next for Google LaMDA?

Google is clear that open dialog models such as LaMDA have benefits and risks and is committed to improving security and anchoring to ensure a more trustworthy and unbiased experience.

Training LaMDA models on different data, including images or video, is another thing we might see in the future.

This opens up the ability to browse the web even more, using conversation prompts.

Google CEO Sundar Pichai said of LaMDA, “We believe LaMDA’s conversational capabilities have the potential to make information and computing radically more accessible and easier to use.”

Although a rollout date has yet to be confirmed, there’s no doubt that models like LaMDA will be Google’s future.

More resources:


Feature image: Andrey Suslov/Shutterstock

Maria D. Ervin