Jeremie Passerin’s Post

How to run LLM 90% 🧠 cheaper, using less energy 🌳 if you want to profile a website or a report? 1. Downloaded website: 25k tokens, 2. Remove noise and HTML: 3000 tokens, 3. Use ON-DEVICE Digital Hippocampus script, so user's BROWSER is doing pattern recognition, 4. Extract 120 tokens, add prompt (total 200 tokens). 𝐘𝐨𝐮 𝐡𝐚𝐯𝐞 𝐧𝐨𝐰 𝟐𝟎𝟎 𝐭𝐨𝐤𝐞𝐧𝐬 𝐢𝐧𝐬𝐭𝐞𝐚𝐝 𝐨𝐟 𝟑𝟎𝟎𝟎 🤓 GPT can still perform well, doing heavy lifting for the user (smart survey) but for 7% of the cost 💸 Full case study soon! Rafał Jakubowski, PhD Ula Nairne Łukasz Osiński

LLM Latency Benchmarks! 🚀 Hamel H. shared a crispy experiment benchmarking the current open source tools for Large Language Model inference. The focus of this benchmark was to measure latency and compare all of the tools under the same scenario. Quick recap on latency, latency implies how fast can we get replies from our LLM Application. The lower the latency, the higher number of tokens we get can get in a second and the faster your application "feels". Here are the takeaways: - The test measures tokens per second, higher is better - We use Llama-2 7B model for all tests - These are run on an A6000 under fixed parameters - CTranslate2: Best tool both in ease of use and performance - vllm: Second best, but is a bit harder to setup - webui: Worst in performance but best for ease of end-user Like always, whenever Hamel shares his notes on anything, I drop everything and read them with a freshly brewed chai. I suggest you do the same: https://lnkd.in/d_knajWX

Benchmarking LLM Inference Backends https://bit.ly/45l1Hyu

i hate ni.... algorithms section

Wow, thank you for sharing this insightful experiment on LLM latency benchmarks, Hamel! Latency is indeed a key factor in determining the speed and efficiency of our LLM applications. It's great to see that you focused on comparing all the open source tools under the same scenario. I found your takeaways to be quite interesting. The fact that CTranslate2 emerged as the best tool in terms of both ease of use and performance is impressive. And it's good to know that vllm came in second, although it may require a bit more effort to set up. As for webui, it seems like it has its advantages in terms of ease of use for end-users but falls short when it comes to performance. Whenever you share your notes, Hamel, they always bring a fresh perspective. I'll make sure to grab a cup of freshly brewed chai and dive into this benchmark experiment right away! Thank you for sharing the link as well.

Is your code built on a house of cards? This article by Joel Spolsky explores the concept of 'leaky abstractions' - the fancy tools that can sometimes unleash hidden complexities in our code. We all acknowledge the the beauty of abstractions and the simplicity it adds to any complex system. But here's the catch!

Andrej Karpathy is at it again! A new 2 hours tutorial just dropped on how to build the GPT Tokenizer. Tokenizers are a completely separate stage of the LLM pipeline: they have their own training sets, training algorithms (Byte Pair Encoding), and after training implement two fundamental functions: encode() from strings to tokens, and decode() back from tokens to strings. Watch now: https://lnkd.in/dfMccgTJ

A bit technical today, but sharing a really easy & cheap alignment technique for those deploying LLM inference in policy-constrained contexts: (With full credit to my colleague WK for seeding the idea - they don't have a LinkedIn profile, so I won't doxx them): 1) During each round of a multi-turn conversation, after your model has interpreted the tokens in the user prompt, but before you close the prompt, save the model state (including logits, etc.) 2) Add to the end of the prompt, add a guardrail question like "\n\n Is this conversation touching on politically sensitive topics?", then close the prompt, and start the response. 3) Constrain the response to a controlled vocabulary of two tokens, either "yes" or "no", and generate a single token. If the token comes back "yes", kill the user session, and display a nice "please don't press that button again" message to the user. Otherwise, roll the session back to the saved state, pop the logits back in place, and generate the response just like usual. The advantage here is that reading tokens is *way* cheaper than emitting them, so you constrain your guardrail cost to a handful of reads and a single write for each round. In my personal testing with self-hosted models, the cost is negligible, and the quality of the alignment is very good. Photo by Andreea Ch via Pexels

Boost LLM Inference Speed with EAGLE-2: Achieve 5x Faster Results Without Quality Loss! EAGLE-2 enhances the speed and efficiency of large language models by introducing a context-aware dynamic draft tree structure in speculative sampling. Unlike traditional static methods, EAGLE-2 adjusts the draft tree based on context, using confidence scores to approximate acceptance rates. This dynamic adjustment increases the number of accepted tokens per drafting-verification cycle, achieving speedup ratios between 2.5x and 5x. Importantly, EAGLE-2 maintains the original model's parameters and acceptance conditions, ensuring that the generated text quality remains unchanged, making it a reliable and lossless acceleration method. https://lnkd.in/eDyXvmGT

Understand The Problemm --->Decide On Computational Means--->Design An Algorithm--->Prove Correctness--->Analyze The Algorithm--->Code The Algorithm