Tokenization and embedding of song lyrics “We will, we will…” i know you know how it ends. but have You wondered what would an LLM say ? Let us find out. I want to ask the Clause Sonnet 4.5 about the embeddings, tokenizations and probability of figuring out the lyrics for “We will, we will…” prompt 🙂 Can you show me the tokenization, embeddings, emtadata and probabilities for “We will, we will …” Tokenization Any kind of machine learning uses numbers in the back stage. The text is split into tokens, each mapped to an ID ( example values ) : Token ID Token Text Type Position Length 1234 We…