In the mid-1960s, the Artificial Intelligence Lab at MIT developed an application called ELIZA, which was intended for conversational discourse with humans using simple rules and scripts. In one of its most famous experiments, ELIZA simulated a psychotherapist. Although the system had no actual understanding of the thoughts or feelings being conveyed, many users believed that ELIZA was guided by true machine intelligence.
ELIZA was one of the first examples of Natural Language Processing (NLP) systems. Regarded as an academic curiosity for many years, NLP is finally making its way into practical applications.
What is NLP? It’s a technology that combines linguistics, computer science, artificial intelligence, and statistical-inferencing to enable computers to process unstructured, human-language text (written or spoken) to derive meanings, execute commands, and perform other tasks.
Today, NLP technology can be found in a variety of applications, including virtual assistants such as Siri and Cortana. It can be found in search engines such as Google and “smart home” devices such as Amazon’s Echo line. NLP is also making its way into interactive robots and applications that mine unstructured legal, medical, and social media text.
Let’s describe in more detail what NLP is, how it works, and why it’s becoming important to the enterprise, along with some tips for executing an NLP project of your own:
In this guide, we'll break down a normally complex topic into a few segments and identify key components to implementing Natural Language Processing in the real world.
Background: Artificial Intelligence
NLP is a branch of computer science that’s been around for over half a century. But it’s made significant progress only recently, thanks to advances in artificial intelligence (AI); specifically, artificial neural networks and machine learning techniques. AI attempts to mimic the operation of the human brain in a computer. It is commonly used in pattern-recognition tasks such as identifying objects in images.
It’s somewhat surprising and counterintuitive, but few NLP systems are designed to parse actual sentences word-for-word to divine their meanings the way humans do. They don’t have the rules of grammar or dictionary definitions of the words programmed into them. Instead, they rely on attributes of a block of text such as word count, the co-occurrence of words, and statistics to derive the high-level meaning of the text.
In a machine learning system, the parameters are tweaked automatically. These systems still require large quantities of training data, but they can teach themselves much faster than a designer teach by hand.
Once an ANN system has been trained, it must be tested using input data it has not seen in training, to determine if it has learned its lesson well. If it hasn’t, it’s time for more training, or perhaps a redesign from the ground up.
Many industries and academic pursuits have started to deploy exciting applications powered by AI:
As you might expect, getting a computer to process human language is tricky. There are so many languages and dialects. Even within the same language, you have many ways to convey the same idea. On top of that there are regional variations (“soda” in the U.S. South is known as “pop” in the Midwest), slang terms, and common misspellings (“loose” vs. “lose,” “principle” vs. “principal”).
And then you have language ambiguities that are extremely difficult for computers to parse. Consider the simple sentence, “We can fish.” Does this mean we have the ability to catch fish, or that we are in the process of canning fish that have already been caught? Humans can figure it out from the context. It’s harder for computers, even well-trained AI systems.
Taken together, these issues make NLP extremely challenging. As a result, most NLP applications are quite narrow in scope. As much as you can do with Amazon’s Alexa system, it understands only a handful of commands. It will be a long time before a practical HAL or C-3PO comes along, able to comprehend any statement in any human language.
Despite its complexity and challenges, NLP has an important and growing role in modern business. Today’s enterprises deal with vast quantities of data. Although we have robust tools with which to process data rows and columns, much enterprise data is unstructured text. There are immense value and insight locked in that unstructured text data, so it’s a worthwhile exercise to develop AI-based tools that can extract actionable meaning from it.
NLP is a key component of digital transformation for many businesses. For example: having humans pore through all that unstructured text is labor-intensive (read: expensive) for the company, not to mention tedious and dull for the humans. Having an algorithm do it—even an algorithm that can only identify “positive” or “negative” reviews with 80% accuracy—is more cost-effective.
With the significant advances in NLP capabilities and it’s expanding availability, businesses of all sizes have increased adoption of NLP to identify new opportunities for increased business insight, expanded customer value, and new opportunities for business profitability:
The technical challenges of developing an NLP solution can be many. Fortunately, there are many NLP service providers with API’s that can deliver rapid integration and access to a wide range of NLP functionality including; sentiment determination, intent,
Despite the many API options available today, creating digital products with imbedded NLP functionality can be a challenging process that requires development discipline and NLP expertise in order to select and effectively utilize the appropriate technology in order to meet the specified business application, current technology environment, anticipated deployment model, and the desired business outcome (to name a few).
In short, comparing capabilities and selecting appropriate technologies can be a dauting task for the development teams encountering NLP technologies for the first time. To illustrate, we’ve assembled descriptions of several leading NLP API sources:
SYSTRAN - is a collection of APIs for: Translation, multilingual dictionary lookups, Natural Language Processing (Entity recognition, Morphological analysis, Part of Speech tagging, Language Identification…),text extraction (from documents, audio files or images) SYSTRAN Platform enables utilization and analysis of both structured and unstructured multilingual content, such as user-generated content, social media, Web content and more.
AYLIEN - Text API is a package of Natural Language Processing, information retrieval and machine learning tools that allow developers to extract meaning and insights from documents.
RxNLP – RxNLP’s Text Mining and NLP APIs provide access to some advanced text analytics functionality over the cloud including: sentence clustering, text similarity, topic extraction, and automatic summary evaluation.
IBM - Connect to the IBM Watson Alchemy API to analyze text for sentiment, keywords and broader concepts. With Watson's suite of NLP offerings, including Watson Natural Language Understanding (NLU), you can surface concepts, categories, sentiment, and emotion, and apply knowledge of unique entities in your industry to your data.
Linguakit - Linguakit API helps you analyze and extract information from texts. The API offers technology based on years of research in Natural Language Processing in a very easy and scalable SaaS model through a RESTful API.
Text Summarization - Text Summarization API provides a professional text summarizer service which is based on advanced Natural Language Processing and Machine Learning technologies. This tool can be used to summarize short important texts from the URLs or documents users provide.
Twinword - Twinword Text Analysis provides a single API for many text analysis needs: sentiment analysis, topic tagging, lemmatizer (doing things properly with the use of a vocabulary and morphological analysis of words), and more. Twinword provides multiple NLP tools in a single plan that enables natural language processing to analyze and understand human sentences.
If you don’t already have in-house AI development expertise, in most cases you are better off bringing in consultants rather than trying to hire a full-time AI developer. Once you have an AI project with outside consultants under your belt, you can evaluate whether your future AI projects warrant the cost of a staff AI developer, and you will have a better idea of what to look for in an AI resource.
At AndPlus, we have a wide range of AI expertise on our development staff, and we know what it takes to make your AI projects succeed. From choosing the right AI approach to designing the algorithm and training and testing the system, we your full-service AI resource.
In many ways, NLP is still in its infancy and will continue to improve and mature as researchers and developers find innovative ways to make it more accurate and applicable to a wider range of tasks. Customizable NLP products and services are expected to go mainstream in the short term. Indeed, Amazon Web Services is already providing a service called Amazon Comprehend for text mining—call it “NLP as a service.” Look for more entrants to emerge in this important and lucrative market.
One thing is for certain, our work and personal lives will increasingly be affected by machines that can read and understand the spoken and written word. As this capability is refined and becomes widespread, new opportunities will emerge that deliver new value to customers and businesses alike.
With new use expectations developing as a result of touchless interactions between suppliers and customers, NLP capabilities will certainly play an increasing role in business. Considering this fact, now is the time to explore how NLP can create a competitive advantage to your business and deliver new value to your customers.
With proper, thorough planning, and the right guide to build and take you through an AI Roadmap, a project can go flawlessly, or nearly so. Be aware: You might have only one shot to get it right. A failed adoption of an advanced technology can lead to countless unforeseen consequences as you scale.
AndPlus can be your digital Sherpa. We've done it, we’ve seen it all, and we can do it for you.
A Minimum Viable Product (MVP) has only those features needed to validate its continued development. Its primary goal is to obtain this insight at a lower cost than that needed to develop a product with more features.
Our process begins by identifying the primary goal that will address both our client's business goals and the end user's goals. We select the methods that the MVP will use to accomplish these goals. Our design team then defines the minimum scope of work and use this list of features to map the ideal user journey.
Early product prototypes are often developed at this stage in order to illustrate concepts and ensure that business objectives and user experiences are aligned and optimized.
Once the user journey is mapped, the code starts flowing. Early prototypes/wireframes are brought to life by our engineering team. We use an Agile Scrum process that is custom tailored to our industry. And that's the kicker. We don't just utilize this same Agile framework straight from the textbook, we optimize the development process based upon more than a decade of development experience gained from hundreds of digital development projects.
The fun part! Our sprints run in 2-week increments. You get actual working builds of your product every two weeks. These builds are tested, and iterated upon as the project moves forward. We pride ourselves on iterating these builds to perfection by launch day.
Our deep expertise and custom Agile development process enable AndPlus to iterate quickly, provide transparency, and deliver on time and on budget — helping our clients get to market faster.
AndPlus is a digital transformation agency that applies technologies to create or improve experiences, processes, and products. For more than a decade, our teams of builders, innovators, engineers, and designers have worked the boundaries of mobile, web, IoT, firmware, and advanced technologies including data warehouse, machine learning, AI, and big data.
Put simply, we’re here to guide and assist your organization in its digital transformation journey by delivering the best strategies, expertise, processes, and technology.
Give us a call and let’s get started.