Otmane Sakhi

𝐍𝐞𝐰 𝐁𝐥𝐨𝐠 𝐏𝐨𝐬𝐭 𝐀𝐥𝐞𝐫𝐭: 𝐂𝐚𝐥𝐢𝐛𝐫𝐚𝐭𝐢𝐨𝐧 𝐓𝐞𝐜𝐡𝐧𝐢𝐪𝐮𝐞𝐬 𝐟𝐨𝐫 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥𝐬🚀 I'm excited to share my latest blog post where we delve into the critical aspect of AI - Calibration Techniques for Language Models. This piece highlights why meticulous calibration is essential and various ways to get calibrated response from LLM models 𝐊𝐞𝐲 𝐦𝐞𝐭𝐡𝐨𝐝𝐬: 𝐕𝐞𝐫𝐛𝐚𝐥𝐢𝐳𝐞𝐝 𝐂𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐜𝐞  - How asking models to rate their confidence can lead to more trustworthy AI. 𝐒𝐞𝐥𝐟-𝐂𝐨𝐧𝐬𝐢𝐬𝐭𝐞𝐧𝐜𝐲-𝐁𝐚𝐬𝐞𝐝 𝐂𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐜𝐞 - Using consistency in model responses to gauge confidence. 𝐋𝐨𝐠𝐢𝐭-𝐁𝐚𝐬𝐞𝐝 𝐀𝐩𝐩𝐫𝐨𝐚𝐜𝐡𝐞𝐬 - Techniques such as Platt Scaling and Temperature Scaling to refine probabilistic outputs. 𝐒𝐮𝐫𝐫𝐨𝐠𝐚𝐭𝐞 𝐌𝐨𝐝𝐞𝐥 𝐚𝐧𝐝 𝐅𝐢𝐧𝐞-𝐓𝐮𝐧𝐢𝐧𝐠 𝐌𝐞𝐭𝐡𝐨𝐝𝐬 - Advanced strategies to enhance prediction reliability and pinpoint uncertainty. Accuracy and confidence in AI responses are crucial, especially in critical applications. Each technique offers a unique approach to better align model outputs with real-world probabilities. For a deeper insight into these techniques and detailed examples, check out the full blog post! 👀 https://lnkd.in/d9emAMEb Do share your experiences and feedback! 💡 #AI #MachineLearning #LanguageModels #ArtificialIntelligence

14

😎 OG ML engineers! Remember the Theano era? Here is my ML origin story 𝗝𝗼𝗶𝗻𝗶𝗻𝗴 𝘁𝗵𝗲 𝗠𝗟 𝗵𝘆𝗽𝗲 𝘁𝗿𝗮𝗶𝗻 Picture this: it's around 2016, and I'm a Bachelor student in Communication Systems at EPFL. As graduation looms, I realize I need to complete a Bachelor project to seal the deal. At the time, machine learning and data science were gaining serious traction, and I couldn't help but take notice. That's when I stumbled upon a project that seemed like a match made in heaven: 𝗳𝗮𝗰𝗲 𝗿𝗲𝗰𝗼𝗴𝗻𝗶𝘁𝗶𝗼𝗻 𝗱𝘂𝗿𝗶𝗻𝗴 𝗯𝗿𝗼𝗮𝗱𝗰𝗮𝘀𝘁 𝘀𝗽𝗼𝗿𝘁 𝗲𝘃𝗲𝗻𝘁𝘀 (basically detecting and identifying players faces). 😅 𝘛𝘩𝘦 𝘤𝘢𝘵𝘤𝘩? 𝘛𝘩𝘦 𝘱𝘳𝘰𝘫𝘦𝘤𝘵 𝘥𝘦𝘴𝘤𝘳𝘪𝘱𝘵𝘪𝘰𝘯 𝘮𝘦𝘯𝘵𝘪𝘰𝘯𝘦𝘥 𝘵𝘩𝘢𝘵 𝘢𝘱𝘱𝘭𝘪𝘤𝘢𝘯𝘵𝘴 𝘴𝘩𝘰𝘶𝘭𝘥 𝘩𝘢𝘷𝘦 𝘴𝘰𝘮𝘦 𝘦𝘹𝘱𝘦𝘳𝘪𝘦𝘯𝘤𝘦 𝘪𝘯 𝘮𝘢𝘤𝘩𝘪𝘯𝘦 𝘭𝘦𝘢𝘳𝘯𝘪𝘯𝘨. 𝗧𝗶𝗺𝗲 𝘁𝗼 𝗮𝗯𝗮𝗻𝗱𝗼𝗻 𝘁𝗵𝗲 𝘀𝗵𝗶𝗽? 𝗻𝗼𝘁 𝘆𝗲𝘁! I applied for the project and started devouring everything I could find on the subject, determined to demonstrate at least a bit of experience. Neural networks, recurrent neural nets, autoencoder, you name it, I read about it. By some miracle (and maybe a bit of charm 😉), I managed to secure an appointment with the lab. And guess what? They gave me the Bachelor project! (To be fair, they were pretty cool and didn't dig too deep into my so-called "experience" in the domain.) With the project set to start after the winter break, I knew I had to level up fast. So, what did I do? I spent my entire holiday break following the famous Andrew Ng's course on Coursera. 𝗚𝗲𝘁𝘁𝗶𝗻𝗴 𝗺𝘆 𝗵𝗮𝗻𝗱𝘀 𝗱𝗶𝗿𝘁𝘆 When the semester (and the project) began, I found myself using the 𝗧𝗵𝗲𝗮𝗻𝗼 framework to implement an autoencoder architecture with triplet loss. The goal was to take two different images of the same athlete, use the autoencoder to build embeddings, and then train a distance model that would give a low number for the same person's face and a high number for different faces. I won't lie, it was a 𝘁𝗼𝘂𝗴𝗵 𝗳𝗶𝗿𝘀𝘁 𝗠𝗟 𝗽𝗿𝗼𝗷𝗲𝗰𝘁, but I managed to get some interesting results in a restricted context: the model worked well on a subset of players and high-quality image samples with clear faces and minimal motion blur. 𝗖𝗵𝗮𝗻𝗴𝗶𝗻𝗴 𝗺𝘆 𝗠𝗮𝘀𝘁𝗲𝗿'𝘀 𝘁𝗼𝗽𝗶𝗰 Inspired by my Bachelor project, I decided to pursue my studies in machine learning. As luck would have it, EPFL had just introduced the Data Science master's program. I jumped at the opportunity, essentially becoming a beta tester for the program. 😎 𝗗𝗮𝗿𝗲 The moral of the story? Grab opportunities even if you don't feel 100% ready. You'll figure it out along the way! This is especially true when applying for job, don't be afraid to overshoot. It's a fantastic way to 𝗽𝗿𝗼𝗴𝗿𝗲𝘀𝘀, as it will put you in situations where you have no choice but to rise to the challenge. 💡 That's my ML origin story! What's yours? #career #machinelearning

23

3 Comments

𝗡𝗲𝘄 𝗯𝗼𝗼𝗸 on engineering end-to-end LLM systems, from data collection and fine-tuning to LLMOps (deployment, monitoring). I kept this one a secret, but in the past months, in collaboration with Packt , Alex Vesa and Maxime Labonne , we started working on the 𝘓𝘓𝘔 𝘌𝘯𝘨𝘪𝘯𝘦𝘦𝘳'𝘴 𝘏𝘢𝘯𝘥𝘣𝘰𝘰𝘬. 𝗔 𝗯𝗼𝗼𝗸 that will walk you through everything you know to build a production-ready LLM project. . I am a big advocate of learning with hands-on examples while being anchored in real-world use cases. That is why this is not the standard theoretical book. While reading the book, you will learn to build a complex LLM project: an LLM Twin. In contrast, theoretical aspects will back everything to understand why we make certain decisions. However, our ultimate goal is to present a framework that can be applied to most LLM projects. . 𝗛𝗲𝗿𝗲 𝗶𝘀 𝗮 𝘀𝗻𝗲𝗮𝗸 𝗽𝗲𝗲𝗸 𝗼𝗳 𝘄𝗵𝗮𝘁 𝘆𝗼𝘂 𝘄𝗶𝗹𝗹 𝗹𝗲𝗮𝗿𝗻 𝘄𝗶𝘁𝗵𝗶𝗻 𝘁𝗵𝗲 𝗟𝗟𝗠 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿'𝘀 𝗛𝗮𝗻𝗱𝗯𝗼𝗼𝗸: - collect unstructured data - create instruction datasets from raw data to fine-tune LLMs - SFT techniques such as LoRA and QLoRA - LLM evaluation techniques - Preference alignment using DPO - inference optimization methods (key optimization, model parallelism, quantization, attention mechanisms) - advanced RAG algorithms using LangChain as our LLM framework and Qdrant as our vector DB - design LLM systems using the FTI architecture - use AWS SageMaker to fine-tune and deploy open-source LLMs - use ZenML to orchestrate all the pipelines and track the data as artifacts - LLMOps patterns such as CT/CI/CD pipelines, model registries and using Comet for experiment tracking and prompt monitoring . The book is still a work in progress, but we are very excited about it! Thank you, Packt, for making this possible and Maxime and Alex for this remarkable collaboration. If you are curious, you can currently pre-order it from Amazon. The whole book should be released by the end of September 2024. ↓↓↓ 🔗 𝘓𝘓𝘔 𝘌𝘯𝘨𝘪𝘯𝘦𝘦𝘳'𝘴 𝘏𝘢𝘯𝘥𝘣𝘰𝘰𝘬: 𝘔𝘢𝘴𝘵𝘦𝘳 𝘵𝘩𝘦 𝘢𝘳𝘵 𝘰𝘧 𝘦𝘯𝘨𝘪𝘯𝘦𝘦𝘳𝘪𝘯𝘨 𝘓𝘢𝘳𝘨𝘦 𝘓𝘢𝘯𝘨𝘶𝘢𝘨𝘦 𝘔𝘰𝘥𝘦𝘭𝘴 𝘧𝘳𝘰𝘮 𝘤𝘰𝘯𝘤𝘦𝘱𝘵 𝘵𝘰 𝘱𝘳𝘰𝘥𝘶𝘤𝘵𝘪𝘰𝘯: https://lnkd.in/dVgFJtzF #machinelearning #mlops #datascience #llms

386

58 Comments

LONGEMBED : Benchmarking Extension of Embedding Models for Long-Context Retrieval In this paper, authors introduced a novel LONGEMBED benchmark comprising of two synthetic tasks and four carefully chosen real-world tasks, featuring documents of varying length and dispersed target information. 𝐍𝐨𝐭𝐚𝐛𝐥𝐞 𝐨𝐮𝐭𝐜𝐨𝐦𝐞𝐬 𝐨𝐟 𝐭𝐡𝐢𝐬 𝐩𝐚𝐩𝐞𝐫 𝐚𝐫𝐞: - observing effective extension of the context window of existing embedding models by several folds, regardless of their original context being 512 or beyond 4k. - For models employing absolute position encoding (APE), showing the possibility of harvesting further improvements via fine-tuning while strictly preserving original behavior within the short context. - For models utilizing Rotary Position embeddings (RoPE) substantial enhancements on LONGEMBED are observed when employing methods that fully leverage RoPE’s advantages, such as NTK and SelfExtend. 𝐋𝐢𝐦𝐢𝐭𝐚𝐭𝐢𝐨𝐧𝐬 𝐨𝐟 𝐄𝐱𝐢𝐬𝐭𝐢𝐧𝐠 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤𝐬 - candidate documents not long enough -  target information to answer user query not uniformly distributed across the document as possible. 𝐓𝐚𝐬𝐤𝐬 𝐢𝐧 𝐋𝐨𝐧𝐠𝐄𝐦𝐛𝐞𝐝 - Synthetic tasks like Personalized Passkey Retrieval and Needle-in-a-haystack Retrieval - Real tasks for evaluations on long-form QA and summarization tasks across datasets like NarrativeQA , 2WikiMultihopQA , QMSum and SummScreenFD 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐖𝐢𝐧𝐝𝐨𝐰 𝐄𝐱𝐭𝐞𝐧𝐬𝐢𝐨𝐧 𝐟𝐨𝐫 𝐀𝐏𝐄-𝐛𝐚𝐬𝐞𝐝 𝐌𝐨𝐝𝐞𝐥𝐬 Strategies used: - 𝐏𝐚𝐫𝐚𝐥𝐥𝐞𝐥 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐖𝐢𝐧𝐝𝐨𝐰𝐬 (𝐏𝐂𝐖) :  PCW divides the long document into multiple short chunks, processes each chunk in parallel, and aggregates their results - 𝐆𝐫𝐨𝐮𝐩𝐞𝐝 𝐏𝐨𝐬𝐢𝐭𝐢𝐨𝐧𝐬 (𝐆𝐏) & 𝐑𝐞𝐜𝐮𝐫𝐫𝐞𝐧𝐭 𝐏𝐨𝐬𝐢𝐭𝐢𝐨𝐧𝐬 (𝐑𝐏) : position reorganization accommodates longer context by reusing the original position ids. - 𝐋𝐢𝐧𝐞𝐚𝐫 𝐏𝐨𝐬𝐢𝐭𝐢𝐨𝐧 𝐈𝐧𝐭𝐞𝐫𝐩𝐨𝐥𝐚𝐭𝐢𝐨𝐧 (𝐏𝐈) : introduces position embeddings via linear interpolation of existing ones. 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐖𝐢𝐧𝐝𝐨𝐰 𝐄𝐱𝐭𝐞𝐧𝐬𝐢𝐨𝐧 𝐟𝐨𝐫 𝐑𝐨𝐏𝐄-𝐛𝐚𝐬𝐞𝐝 𝐌𝐨𝐝𝐞𝐥𝐬 Strategies used: - 𝐒𝐞𝐥𝐟 𝐄𝐱𝐭𝐞𝐧𝐝 (𝐒𝐄): re-introduces normal relative positions within the nearest neighbor window. - 𝐍𝐓𝐊-𝐀𝐰𝐚𝐫𝐞 𝐈𝐧𝐭𝐞𝐫𝐩𝐨𝐥𝐚𝐭𝐢𝐨𝐧 (𝐍𝐓𝐊): scales high frequencies less and low frequencies more to spread out the interpolation pressure across multiple dimensions 𝐑𝐞𝐬𝐮𝐥𝐭𝐬 - APE based models: Plug and play methods like GP, RP, LPI and PCW strategies yield comparable results with no significant disparities. - RoPE-Based Models: RoPE-specific methods including NTK and SE yield significant improvements for both models across all datasets, surpassing PCW, PI and GP by a large margin. 🏆 For my detailed analysis of this paper, please refer to my blogpost on it at: https://lnkd.in/dRMbdH39 #generativeai #largelanguagemodels

15

🚀 Rise of AI Hardware Accelerators: Top Insights from the GPU Optimization Workshop I had the pleasure of attending an amazing workshop on GPU optimization last month featuring experts from PyTorch, NVIDIA, OpenAI, and Voltron Data. Here are some of the key takeaways : 🔥 PyTorch is leveling up performance with kernel fusion (torch.compile), quantization, and custom CUDA kernels. To master CUDA, the CUDA programming guide and the book "Programming Massively Parallel Processors" are essential reads. ⚡ NVIDIA's team shared game-changing optimizations for serving large language models. Techniques like quantization (SmoothQuant, INT4 AWQ), smart batching, paged key-value caches, and speculative decoding can dramatically reduce latency and increase throughput. The TensorRT and Triton Inference Server embody these ideas in practice. 🚀 OpenAI introduced Triton, a cutting-edge programming language that balances flexibility and performance. Its block-based model simplifies writing lightning-fast kernels like FlashAttention and can handle complex algorithms and unique tensor shapes. 📈 Voltron Data showcased how GPUs are revolutionizing big data workloads. Libraries like RAPIDS and analytics engines built on Apache Arrow enable Extremely-fast SQL queries and dataframe operations on GPUs. Using columnar memory formats like Arrow is key for maximizing throughput. Takeaways:- * Boost performance with kernel fusion, quantization, and custom CUDA kernels. * Optimize LLM inference with quantization, smart batching, and speculative decoding etc. * Simplify GPU kernel development using Triton's flexible, high-performance language. * Revolutionize big data on GPUs with RAPIDS, Apache Arrow, and columnar formats. A huge shout-out to the incredible speakers Mark Saroufim(PyTorch), Sharan Chetlur(Nvidia), Philippe Tillet(OpenAI), William Malpica(Voltron Data) and MLOps Learners & Chip Huyen for Hosting.. My strong background in AI and hardware made me truly appreciate the in-depth information shared during this workshop. Can't wait to put these insights into practice! 🌟 #AI #machinelearning #dataengineering #gpuoptimization #HardwareAccelarators

94

H/T Cecile G. Tamura Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs (University of Maryland, June 2024) Paper: https://lnkd.in/e-fTtzfj Abstract: "Large language models can memorize and repeat their training data, causing privacy and copyright risks. To mitigate memorization, we introduce a subtle modification to the next-token training objective that we call the goldfish loss. During training, a randomly sampled subset of tokens are excluded from the loss computation. These dropped tokens are not memorized by the model, which prevents verbatim reproduction of a complete chain of tokens from the training set. We run extensive experiments training billion-scale Llama-2 models, both pre-trained and trained from scratch, and demonstrate significant reductions in extractable memorization with little to no impact on downstream benchmarks."

4

How Much Research Is Being Written by Large Language Models? 17.5% (that is assumed to have been detected) https://buff.ly/3VdqVv0 This will mean all future papers on average will be referencing 3 to 5 papers containing LLM generated content significant error rates.

1

Meta open-sourced Multi-Token Prediction models which are faster and efficient than comparable next-token models 𝗞𝗲𝘆 𝗺𝗼𝗱𝗲𝗹 𝗵𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀: -  In this work, authors suggest that training language models to predict multiple future tokens at once results in higher sample efficiency - these 13B parameter models solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models - Experiments on small algorithmic tasks demonstrate that multi-token prediction is favorable for the development of induction heads and algorithmic reasoning capabilities. - these models trained with 4-token prediction are up to 3 times faster at inference, even with large batch sizes. 𝗥𝗲𝗹𝗲𝗮𝘀𝗲 𝗻𝗼𝘁𝗲𝘀 - pre-trained models released for code completion under a non-commercial/research-only license 𝗠𝗼𝗱𝗲𝗹𝘀 𝗿𝗲𝗹𝗲𝗮𝘀𝗲𝗱 released following four 7B parameter models trained on code: - baseline model (n=1) trained on 200B tokens of code: 7B_200B_1/ - multi-token prediction model (n=4) trained on 200B tokens of code: 7B_200B_4/ - baseline model (n=1) trained on 1T tokens of code: 7B_1T_1/ - multi-token prediction model (n=4) trained on 1T tokens of code: 7B_1T_4/ 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eRZcZnBV 𝗠𝗼𝗱𝗲𝗹 𝗿𝗲𝗹𝗲𝗮𝘀𝗲: https://lnkd.in/eBCjP4S6

59

Meta's Chief AI Scientist's advice to PhD students: ”There is no point working on LLMs”: During a talk at VivaTech2024 in Paris, Yann LeCun stated that he does not recommend students to work on LLMs. Yann views LLMs as an 'off-ramp' from the road to reaching ultimate intelligence. An off-ramp is thinking your new hammer will cut wood. We must remember LLMs' core function—they are next-word prediction tools trained to be conversational by imitating human responses. Therefore, it's hard to surpass human response accuracy and intelligence if LLMs are only trained on humans. As Yann states: ”You should work on the next generation of AI systems that lift the limitation of LLMs.” - - Video source: @tsarnick on X. - - if you found this post helpful! Connect 🤝 Abdul Rahim Roni Follow 👉 Abdul Rahim Roni Notification: Press 🔔 icon Follow Abdul Rahim Roni for more such content. __________ If you liked this and want to get more AI tools, tutorials, and news, join Superhuman - my newsletter that helps readers at companies like and Microsoft get smart about AI & tech:👇 https://lnkd.in/grT4DYuz - - #ai #tech #innovation #NVIDIA #Germany #Europe #Ai #LinkedIn #ChatGPT #Ai #Microsoft #excelskills #excel #freecourses #dataanlytics #dataanalysis #exceltips #business #economics #management #projectmanagement #dataanalysis #businessintelligence #strategy #dataengineering #datascience #softwareengineering #linkedin #amazon #resume #creative #resumesearch #resumewriting #job #jobalert #jobsearch #jobseekers #hiring #jobhunt #amazonjobs

1

Llama 3 by Meta was just released last week .. and it is already on the Chatbot Arena leaderboard 🏆. The battle for the best (open ?) LLM continues stronger than ever ! The 𝐂𝐡𝐚𝐭𝐛𝐨𝐭 𝐀𝐫𝐞𝐧𝐚 is a crowdsourced open platform for blind comparisons of LLMs, developed by LMSYS. It is one of the most popular ways to evaluate LLMs, and companies releasing new models thrive to reach the top of the leaderboard https://lnkd.in/g_sWj5pF. 𝐇𝐨𝐰 𝐝𝐨𝐞𝐬 𝐢𝐭 𝐰𝐨𝐫𝐤 ? ▶ The approach is very simple: users enter prompts, and the platform generates responses from two different LLMs, then users chose the best answer, without knowing which model generated which answer ▶ Scores are calculated with the Elo ranking systems ▶ They have collected hundreds of thousands of votes so far 𝐖𝐡𝐲 𝐢𝐬 𝐭𝐡𝐢𝐬 𝐩𝐨𝐰𝐞𝐫𝐟𝐮𝐥 ? ▶ Realistic Evaluations: traditional benchmarks are not as helpful as they might seem because they can be manipulated and tailored for optimization.. with the Chatbot Arena, LLMs are evaluated through real-world user interactions for more realistic evaluations ▶ The data is always fresh = no risk of data contamination, as compared to static benchmarks ▶ Continuous Feedback: Rankings are updated based on ongoing user votes, to ensure up-to-date performance metrics ▶ Transparent Process: the Elo rating system is a clear and explainable method 𝐒𝐨𝐦𝐞 𝐥𝐢𝐦𝐢𝐭𝐚𝐭𝐢𝐨𝐧𝐬 𝐨𝐟 𝐭𝐡𝐞 𝐂𝐡𝐚𝐭𝐛𝐨𝐭 𝐀𝐫𝐞𝐧𝐚 ▶ Short user interactions might not reveal more advanced model capabilities ▶ Lack of diversity and complexity in the prompts ▶ Crowdsourcing is not representative: it is likely that most of the platform users are in a similar group/race/gender, so their preferences might not fairly reflect the entire population ▶ Users tend to be biased towards more verbose responses 👌 Overall, I love this platform, and it is very user-friendly so if you haven't yet tried it, give it a go! -- #AI #GenAI #Llama3 #LLMs #ChatbotArena

10

2 Comments

One particularly powerful thing about open-source LLMs with Ollama and Timescale's pgai for #Postgres is that you can now host your own models without needing to pay or share private data with closed-sourced providers like OpenAI.

24

⚕🤖 MEDITRON-70B (based on Llama 2) already outperforms GPT-3.5 and Med-PaLM and is within 5% of GPT-4 and 10% of Med-PaLM-2. And now, a new version that is fine-tuned from AI at Meta's Llama 3🦙🦙🦙 8B achieves promising results.

7

[Research] Read if you care about NMF Chordal-NMF with Riemannian Multiplicative Update What : - NMF is not linear but conic algebra - Point-wise distance not good for cone - We propose chrodal / angle distance - We propose a Riemannian gradient method for this NMF - Fun arXiv 2405.12823

14

Fine-Tuning Steps of an LLM for any specific usecase .. using Argilla and MantisNLP

5

Have you also noticed that language models sometimes start their responses alright, then make a mistake, and from that point on they hallucinate and drift off-topic? 🤔 Our paper "Know When to Stop: A Study of Semantic Drift in Text Generation" (which just got accepted at NAACL 2024! 🥳) looks into this phenomenon, now called semantic drift. Here's what you can expect from our paper: 1️⃣ Full definition and a new metric for measuring semantic drift 🚀 2️⃣ Results on LLaMa 2, GPT 4 and Falcon models 🔍 3️⃣ Comparison between early-stopping, resampling and knowledge retrieval methods 📊 Check it out and let's dive deeper into understanding hallucinations in AI text generation! ➡️ https://lnkd.in/ed4m7dCA #SemanticDrift #AIResearch #TextGeneration #LanguageModels #ArtificialIntelligence #MachineLearning #NLP #DeepLearning #AIInnovation #KnowledgeRetrieval #EarlyStopping #Resampling #LLaMa2 #NAACL2024

118

1 Comment

ARC-AGI and the problem with LLMs (even multi-modal models created with the same architecture) Francois Chollet (of Keras fame) and others have put forward a $1 million prize pool on Kaggle (https://lnkd.in/ePQ6VC6n) to support the advancement of efforts towards AGI due to the weakness of the current SOTA models to generalize outside their training set in terms of reasoning and abstraction abilities. The dataset (https://lnkd.in/ec3_fU5U) features relatively simple (someone of 1 sigma should be able to solve >95% of problems and someone of >2 sigma should be able to achieve 100%) visual IQ problems that work on reasoning/abstraction. This highlights one of the issues with current LLMs that they are more akin to giant interactive associative recall structures than anything else; the reasoning and abstraction they exhibit is in very large part hallucinated from the training set. The human like responses act to deceive many users into thinking they are interacting with a human like reasoning system but they would be mistaken in that belief as the above and following illustrates. Indeed recent research allows the exploration of LLMs as interactive giant associative stores by tracing the location of particular knowledge to particular critical MLP (not attention) layers using causal mediation analysis (Locating and Editing Factual Associations in GPT, Meng et al.). Each of the critical MLP layers stores a portion of the memory. Causal mediation analysis works by noising the input and then restoring the state of selected activations to their clean value and thereby observing which layers have the strongest effect on producing the original output. This even allows knowledge/memory editing (MEMIT that computes a delta with the desired output by gradient descent and then spreads the change over the mediating layers for instance) of LLMs. It is perhaps much more instructive to think of LLMs in this way as interactive giant and highly complex associative stores of knowledge from their training sets.

4

Today I presented an updated version of my overview talk on deep generative modelling at the IRI - Institut de Robòtica i Informàtica Industrial, CSIC-UPC. Do you have any tips to prepare the next version ?

57

3 Comments

AGREE : a learning-based framework for improving LLMs by grounding their responses in retrieved passages and by providing citations to make answers more reliable LLMs has major issue towards their widespread deployment in the real world is that they can generated hallucinated answers that are not factual. In this paper, authors propose a new framework, AGREE, Adaptation for GRounding EnhancEment, that provides grounding of answers in retrieved passages by providing citations, making them more reliable and increasing user trust. 𝗞𝗲𝘆 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻𝘀: - Introduces a learning-based approach that adapts a base LLM to include accurate citations in its response, leveraging automatically created data - A test-time adaptation (TTA) method that iteratively improves responses of LLMs based on the citation information. - Extensive experiments on two LLMs over five datasets demonstrating the effectiveness of the proposed AGREE framework for improving grounding and citation generation. 𝗔𝗚𝗥𝗘𝗘 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 i) Tuning LLMs - authors tune the LLM to self-ground the claims in their responses by providing citations to retrieved documents - authors sampled responses from the base model, use an NLI model to add citations to the sampled responses, and tune the base model with the best-grounded response. ii) Test-time adaptation(TTA) -Proposed framework is a form of retrieval augmented generation framework ,using an adapted LLM that is able to answer a query based on a set of given passages retrieved from the corpus, and, more importantly, self-ground its response to add citations to the passages as well as to find supported statements needing further investigation Detailed procedure is as follows: - Given a query Q and corpus D, we first retrieve based on the query to obtain an initial set of working passages - Next, iteratively at each iteration, the LLM generates a response to the query based on the working passages, adds citations to its response, and finds out any unsupported statements that do not have citations - Then, we add the cited passages to the list of relevant passages - Lastly, at each iteration, we update the working passages – if there are unsupported statements, we include additional information retrieved based on the unsupported statements. 𝗘𝘅𝗽𝗲����𝗶𝗺𝗲𝗻𝘁𝗮𝗹 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 - across 5 datasets(NQ, StrategyQA,ASQA etc.), AGREE can generate responses that are better grounded in text corpus and provide accurate citations to its response, substantially outperforming all baselines - improvements obtained from training on in-domain datasets can effectively generalize to OOD datasets that contain different question types - TTA improves llama-2 answer correctness by 3.1 and 3.7 on NQ and ASQA, respectively - AGREE framework fine-tunes the base LLM to enable self-grounding without needing for additional in-context examples or NLI models 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eCGa8AE8 #generativeai #largelanguagemodel

26

1 Comment