From the course: LLM Foundations: Vector Databases for Caching and Retrieval Augmented Generation (RAG)
Join today to access over 23,200 courses taught by industry experts.
Vectorization in NLP
From the course: LLM Foundations: Vector Databases for Caching and Retrieval Augmented Generation (RAG)
Vectorization in NLP
Vectorization is a key concept in natural language processing or NLP for short. Let's quickly review its importance in NLP. Machine learning algorithms today deal with a lot of text data. Transformer models, which are the foundation for large language models, deal with text data mainly, but machine learning algorithms and architectures can only deal with numeric data. Even in transformers, the inputs and outputs are all numeric values. When using text data for training or inference, they need to be first transformed to equivalent numeric representations before they can be used. When doing so, the original meaning, context, and position information for the text data need to be properly represented in their numeric representations. Vectorization is the process of converting text data into numeric values. Text data is represented as a series of numeric vectors, and vectorization helps generate them. These vectors capture the structure and semantics of the original text. Vector outputs of…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.