CIS625 - LLM Architectures and Embeddings

Course Description

In this course, you will investigate the internal workings of transformer-based language models by exploring how embeddings, attention, and model architecture shape textual outputs. You’ll begin by building a neural search engine that retrieves documents through vector similarity then move on to extracting token-level representations and visualizing attention patterns across different layers and heads.

As you progress, you will analyze how tokens interact with each other in a large language model (LLM), compare encoder-based architecture with decoder-based architectures, and trace how a single word’s meaning can shift from input to output. By mastering techniques like plotting similarity matrices and identifying key influencers in the attention process, you’ll gain insights enabling you to decode model behaviors and apply advanced strategies for more accurate, context-aware text generation.

You are required to have completed the following courses or have equivalent experience before taking this course:

• LLM Tools, Platforms, and Prompts
• Language Models and Next-Word Pronunciation
• Fine-Tuning LLMs
• Language Models and Language Data

Faculty Author

Benefits to the Learner

• Identify the tokens a model focuses on within an input sentence, analyzing how each token influences or is influenced by others
• Extract and visualize attention matrices for sample text
• Compare various LLM architectures to understand how token interactions differ across encoder, decoder, and encoder-decoder models
• Examine intermediate token representations in upper and middle layers, using vector similarity to reveal context-specific shifts in meaning
• Implement and evaluate document retrieval based on embedding vectors, showcasing how learned representations enable effective content search

Target Audience

• Engineers
• Developers
• Analysts
• Data scientists
• AI engineers
• Entrepreneurs
• Data journalists
• Product managers
• Researchers
• Policymakers
• Legal professionals

Applies Towards the Following Certificates

• Large Language Model Fundamentals : Required Courses