Thibaut Issenhuth

LONGEMBED : Benchmarking Extension of Embedding Models for Long-Context Retrieval In this paper, authors introduced a novel LONGEMBED benchmark comprising of two synthetic tasks and four carefully chosen real-world tasks, featuring documents of varying length and dispersed target information. 𝐍𝐨𝐭𝐚𝐛𝐥𝐞 𝐨𝐮𝐭𝐜𝐨𝐦𝐞𝐬 𝐨𝐟 𝐭𝐡𝐢𝐬 𝐩𝐚𝐩𝐞𝐫 𝐚𝐫𝐞: - observing effective extension of the context window of existing embedding models by several folds, regardless of their original context being 512 or beyond 4k. - For models employing absolute position encoding (APE), showing the possibility of harvesting further improvements via fine-tuning while strictly preserving original behavior within the short context. - For models utilizing Rotary Position embeddings (RoPE) substantial enhancements on LONGEMBED are observed when employing methods that fully leverage RoPE’s advantages, such as NTK and SelfExtend. 𝐋𝐢𝐦𝐢𝐭𝐚𝐭𝐢𝐨𝐧𝐬 𝐨𝐟 𝐄𝐱𝐢𝐬𝐭𝐢𝐧𝐠 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤𝐬 - candidate documents not long enough -  target information to answer user query not uniformly distributed across the document as possible. 𝐓𝐚𝐬𝐤𝐬 𝐢𝐧 𝐋𝐨𝐧𝐠𝐄𝐦𝐛𝐞𝐝 - Synthetic tasks like Personalized Passkey Retrieval and Needle-in-a-haystack Retrieval - Real tasks for evaluations on long-form QA and summarization tasks across datasets like NarrativeQA , 2WikiMultihopQA , QMSum and SummScreenFD 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐖𝐢𝐧𝐝𝐨𝐰 𝐄𝐱𝐭𝐞𝐧𝐬𝐢𝐨𝐧 𝐟𝐨𝐫 𝐀𝐏𝐄-𝐛𝐚𝐬𝐞𝐝 𝐌𝐨𝐝𝐞𝐥𝐬 Strategies used: - 𝐏𝐚𝐫𝐚𝐥𝐥𝐞𝐥 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐖𝐢𝐧𝐝𝐨𝐰𝐬 (𝐏𝐂𝐖) :  PCW divides the long document into multiple short chunks, processes each chunk in parallel, and aggregates their results - 𝐆𝐫𝐨𝐮𝐩𝐞𝐝 𝐏𝐨𝐬𝐢𝐭𝐢𝐨𝐧𝐬 (𝐆𝐏) & 𝐑𝐞𝐜𝐮𝐫𝐫𝐞𝐧𝐭 𝐏𝐨𝐬𝐢𝐭𝐢𝐨𝐧𝐬 (𝐑𝐏) : position reorganization accommodates longer context by reusing the original position ids. - 𝐋𝐢𝐧𝐞𝐚𝐫 𝐏𝐨𝐬𝐢𝐭𝐢𝐨𝐧 𝐈𝐧𝐭𝐞𝐫𝐩𝐨𝐥𝐚𝐭𝐢𝐨𝐧 (𝐏𝐈) : introduces position embeddings via linear interpolation of existing ones. 𝐂𝐨𝐧𝐭𝐞𝐱𝐭 𝐖𝐢𝐧𝐝𝐨𝐰 𝐄𝐱𝐭𝐞𝐧𝐬𝐢𝐨𝐧 𝐟𝐨𝐫 𝐑𝐨𝐏𝐄-𝐛𝐚𝐬𝐞𝐝 𝐌𝐨𝐝𝐞𝐥𝐬 Strategies used: - 𝐒𝐞𝐥𝐟 𝐄𝐱𝐭𝐞𝐧𝐝 (𝐒𝐄): re-introduces normal relative positions within the nearest neighbor window. - 𝐍𝐓𝐊-𝐀𝐰𝐚𝐫𝐞 𝐈𝐧𝐭𝐞𝐫𝐩𝐨𝐥𝐚𝐭𝐢𝐨𝐧 (𝐍𝐓𝐊): scales high frequencies less and low frequencies more to spread out the interpolation pressure across multiple dimensions ��𝐞𝐬𝐮𝐥𝐭𝐬 - APE based models: Plug and play methods like GP, RP, LPI and PCW strategies yield comparable results with no significant disparities. - RoPE-Based Models: RoPE-specific methods including NTK and SE yield significant improvements for both models across all datasets, surpassing PCW, PI and GP by a large margin. 🏆 For my detailed analysis of this paper, please refer to my blogpost on it at: https://lnkd.in/dRMbdH39 #generativeai #largelanguagemodels

15

I am proud to be working at Martian, where our AI research team is pushing the boundaries of scaling Mechanistic Interpretability, arguably one of the most important areas of AI research. In recent weeks, we have seen a surge of progress in AI interpretability, driven by both Anthropic and OpenAI. While their recent techniques (linked in our blog post) involve deep and painstaking human inspection of individual neurons, our approach aims to automate this understanding at scale. The Martian research team, led by Shriyash U. and Chaithanya B., outlines a groundbreaking insight in this blog post: to truly automate the understanding of neural networks, we must evolve not only our techniques but also our theoretical foundations. They propose category theory as the key to this evolution. As our founder Etan Ginsberg would say, for more on the technical details, dive into the blog post. For those who are “normies” like me, you might wonder why this is so important! The goal of Mechanistic Interpretability is to make AI models understandable, so we can "read" them like a book. Currently, AI models are largely black boxes—even their developers are often surprised by what they can do. Understanding how to prompt them, what they excel at, and their inherent biases is a process of trial and error. Imagine the Possibilities: -Having detailed insights into an AI model's knowledge and capabilities. -Understanding its biases, strengths, and weaknesses. Knowing how to effectively prompt a model to get the desired answers. This level of understanding would revolutionize our interaction with AI, making it more transparent and predictable. At Martian, this research powers our LLM Router, which predicts the best large language model (LLM) for a given prompt based on a customer's cost, latency, and output quality preferences, a highly practical real-world application of Mechanistic Interpretability.  Follow Martian to be tuned into more exciting developments from our team! P.S. - As I write this, our founders just came out of a meeting with the research team of a major global tech company. They were delving into our technology, and the customer said, “You guys are better than OpenAI.” While I’m not sure of the exact context, it’s still incredibly flattering! #AI #MachineLearning #ArtificialIntelligence #TechInnovation #MechanisticInterpretability #Research #MartianAI #AITransparency

7

Let's evaluate Llama3's RAG performance on an arXiv paper (about RAG evaluation, this is a very meta notebook, puns intended)! In this tutorial, we build on our work earlier this week in combining Unstructured's API for pdf document preprocessing with OpenAI's GPT-4o + ragas for evaluation, Hugging Face for Llama3, and LangChain to integrate all of these systems to evaluate Llama3. Llama3's metrics on RAG for our quick example: context precision: 0.9867, faithfulness: 0.8297, answer relevancy: 0.8643, context_recall: 0.9733 Try this out with your favorite unstructured data by swapping the arXiv pdf URL in the notebook! Colab Notebook: https://lnkd.in/gSndzTuC

26

1 Comment

🚀 Want to employ the power of #PyTorch for your scientific models without a steep learning curve? 👀 Check out our latest blog, where Travis Oliphant discusses how PyTorch's integration with the Array API makes it easy to use with familiar libraries like NumPy and scikit-learn.🔥 https://lnkd.in/dDRkEY3v #Innovation

10

Meta open-sourced Multi-Token Prediction models which are faster and efficient than comparable next-token models 𝗞𝗲𝘆 𝗺𝗼𝗱𝗲𝗹 𝗵𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀: -  In this work, authors suggest that training language models to predict multiple future tokens at once results in higher sample efficiency - these 13B parameter models solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models - Experiments on small algorithmic tasks demonstrate that multi-token prediction is favorable for the development of induction heads and algorithmic reasoning capabilities. - these models trained with 4-token prediction are up to 3 times faster at inference, even with large batch sizes. 𝗥𝗲𝗹𝗲𝗮𝘀𝗲 𝗻𝗼𝘁𝗲𝘀 - pre-trained models released for code completion under a non-commercial/research-only license 𝗠𝗼𝗱𝗲𝗹𝘀 𝗿𝗲𝗹𝗲𝗮𝘀𝗲𝗱 released following four 7B parameter models trained on code: - baseline model (n=1) trained on 200B tokens of code: 7B_200B_1/ - multi-token prediction model (n=4) trained on 200B tokens of code: 7B_200B_4/ - baseline model (n=1) trained on 1T tokens of code: 7B_1T_1/ - multi-token prediction model (n=4) trained on 1T tokens of code: 7B_1T_4/ 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eRZcZnBV 𝗠𝗼𝗱𝗲𝗹 𝗿𝗲𝗹𝗲𝗮𝘀𝗲: https://lnkd.in/eBCjP4S6

59

My take on Yann LeCun vs. Elon Musk: I have more respect for Yann LeCun for his contribution to AI research, especially for making this knowledge accessible and available to everyone. We all should appreciate this part. Yann LeCun said that he published more than 80 papers in 2022 alone while Elon Musk arrogantly said "That is nothing". I think both are wrong here. In my opinion, the number of papers does not matter although this is how acamedia usually works. I rather have one paper that is groundbreaking for many generations to come over 1000s of papers. So the number of papers should not be an indication here of anything. In contrast, ignoring the contribution of top-class researchers who helped to accelerate the development in this field and more importantly sharing their knowledge publically is unjust. Both have different focuses, Elon focuses on the business value which is also fine, while LeCun focuses on the scientific value. By the end of the day, both learn by doing and this is what makes them pioneers in their field. But if one of them were unhealthy, no one of them would be able to do anything. It is always better to be grateful for what we were given such as health, intelligence, and chances that without no one successful would be at where he\she is today.

3

It’s day 4 of our 11 Days of Inference Acceleration Techniques. Today, we’re moving on to runtime level optimization best practices. 𝐅𝐨𝐮𝐫𝐭𝐡 𝐭𝐢𝐩: 𝐓𝐚𝐤𝐞 𝐚𝐝𝐯𝐚𝐧𝐭𝐚𝐠𝐞 𝐨𝐟 𝐠𝐫𝐚𝐩𝐡 𝐜𝐨𝐦𝐩𝐢𝐥𝐚𝐭𝐢𝐨𝐧 📈 Graph compilers such as TVM, Tensor-RT, and OpenVino work by getting a computation graph of a specific model and generating an optimized code adjusted for the target hardware. Graph compilation can optimize the graph structure by merging redundant operations, performing kernel auto-tuning, enhancing memory reuse, preventing cache misses, and more. But few things to be aware of: 📝 Not all models compile equally. 📝 The impact of compilation on model performance can vary. Hence, make sure to check the architecture's ability to compile early on in the process to avoid wasting time and resources on training a model that can’t be optimized for fast inference. – What’s the #11DaysofInferenceAccelerationTechniques? The Deci team is posting, for 11 days, a series of inference acceleration techniques for deep learning applications. If you’re looking for practical tips and best practices for improving inference, follow Deci AI so you won’t miss an update. #deeplearning #machinelearning #neuralnetworks #computervision

9

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

29

🧿 Last week Yann LeCun said something like "LLMs will not reach human intelligence"[1]. BOOM, polemic! It's clear the on-going #deeplearning is not ready for "general AI", a "radical alternative" is necessary to create the “superintelligence”. 👉LeCun is right, of course. The polemic reminds me a paper I loved: "AI and the Everything in the Whole Wide World Benchmark"(2021 - Berkely, Google, UWA) [2]: a faulty logic in AI where many popular benchmarks rely on the same false assumptions inherent to the ridiculous “Everything in the Whole Wide World Museum” that Grover visits (watch the clip). 👉In modern #AI there's a tendency across the different subfields to over-valorize a small collection of influential benchmarks. These benchmarks operate as stand-ins for a range of anointed common problems that are frequently framed as foundational milestones towards the general AI. SOTA performance is widely understood as indicative of progress towards these long-term goals. If you read the paper[2] you can spot a long list of issues, such as: ⛔𝐋𝐢𝐦𝐢𝐭𝐚𝐭𝐢𝐨𝐧𝐬 𝐚𝐧𝐝 𝐂𝐨𝐧𝐬𝐭𝐫𝐮𝐜𝐭 𝐕𝐚𝐥𝐢𝐝𝐢𝐭𝐲 benchmarks in AI are often seen as milestones towards general AI capabilities. However these benchmarks have limitations and suffer from construct validity issues. They cannot fully capture the broad, general-purpose abilities they claim to measure because they are inherently specific, finite, and contextual​. ⛔𝐓𝐚𝐬𝐤 𝐃𝐢𝐯𝐞𝐫𝐬𝐢𝐭𝐲 benchmark approach aims to include a diverse range of tasks, but it falls short compared to the vast diversity of human activity. Despite the intention to cover various scenarios, benchmarks primarily include "easy" tasks like single-sentence classification, detection, segmentation, inference tasks, and so on. ⛔𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤 𝐒𝐮𝐛𝐣𝐞𝐜𝐭𝐢𝐯𝐢𝐭𝐲 all datasets are influenced by the perspectives of their creators, making it impossible for them to be completely neutral or universal. This is evident in the underrepresentation and misrepresentation of non-Western, non-white, and gender-nonconforming individuals in these datasets. ⛔𝐈𝐧𝐚𝐩𝐩𝐫𝐨𝐩𝐫𝐢𝐚𝐭𝐞 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐭𝐲 𝐔𝐬𝐞 AI community often misinterprets benchmarks as measures of progress towards general AI, despite their limited scope. This misinterpretation stems from the desire to associate high performance on these benchmarks with significant advancements in AI, leading to exaggerated claims about the capabilities of AI models​. ⛔𝐆𝐞𝐧𝐞𝐫𝐚𝐥-𝐏𝐮𝐫𝐩𝐨𝐬𝐞 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 Originally the CTF was designed for specific, practical tasks like speech recognition. Modern "generalizing" benchmarks fail to represent real-world objectives and contexts adequately and often do not consider crucial aspect such power consumption (RED AI), GPU requirements, and stability vs. data perturbations. Enjoy the paper and trust the experts. Always. [1]https://lnkd.in/dFraieZS [2]https://lnkd.in/da-7PnVT

47

4 Comments

Benchmarking Federated Learning for Large Language Models with FedLLM-Bench Quick read: https://lnkd.in/g2qf5YBp Paper: https://lnkd.in/gV4JCdYG

26

💬  Learn how to build a custom chatbot with LLMs + RAG and scalable workflows 🤖 RSVP now to join our Chief ML Engineer, Niels Bantilan, for a live workshop online covering: - Using LLMs - Building RAG system for information retrieval - Prompt engineering - Creating scalable workflows using Union.ai (powered by Flyte) Who should attend: Anyone interested in building custom chatbots with LLMs and best MLOps practices should attend. This workshop is designed to be approachable for most skill levels. Familiarity with machine learning and Python is strongly encouraged but not required. By the end of this workshop, you'll be able to build a custom chatbot using LLMs, RAG, and scalable AI workflows for large language models (LLMs). 🔗 https://lnkd.in/gnTajH3Z #llm #llms #rag

13

1 Comment

Exploring the Pros and Cons of Multi-token Prediction in Language Models : A recent study, "Better & Faster Large Language Models via Multi-token Prediction," by researchers from FAIR offers a fascinating look at how predicting multiple tokens simultaneously can significantly enhance the efficiency of large language models like GPT and Llama. The researchers of the paper claim that the models developed via the technique discussed in the paper solves 12 % more problems on HumanEval and 17 % more on MBPP than comparable next-token models Key Highlights: Enhanced Efficiency: The multi-token prediction method boosts sample efficiency without additional training time or memory overhead, presenting a compelling case for its adoption in larger models. Improved Performance: This technique has shown to dramatically improve performance, particularly in complex coding tasks and natural language processing, consistently outperforming traditional models. Pros of Multi-Token Prediction: Scalability and Versatility: The method scales impressively with larger models and demonstrates benefits across various data types, including both code and natural language. Inference Speed: Models trained with this method can be up to 3× faster at inference, making them not only more intelligent but also quicker. Potential Challenges: Complexity in Training Dynamics: The method involves multiple output heads which could complicate model training dynamics and require careful tuning. Optimal for Large-Scale Models: While the benefits are clear for large models, smaller models may not see as significant improvements, possibly limiting its applicability in resource-constrained projects. Generalization across Tasks: The effectiveness across all types of tasks is not fully explored, which may limit its applicability in certain scenarios. Future Directions: The paper discusses promising future research paths, including automating the selection of the number of tokens to predict and optimizing vocabulary sizes for multi-token prediction. This approach represents an exciting advancement in AI training techniques. However, it also poses new challenges that necessitate further exploration to fully understand and mitigate potential drawbacks. Paper link : https://lnkd.in/d69Je8wU #AI #MachineLearning #LanguageModels #Innovation #Research

14

Retrieval-Augmented Generation (RAG) enables answering functional, technical and domain-related questions about large codebases using LLMs, thus supporting developers in onboarding, feature development, or error analysis. In traditional RAG, code fragments relevant to the query are selected using similarity search. However, this method can miss critical code segments that are not directly related to the query but are crucial for correctly answering the question. As part of her master's thesis at itestra, Helena is researching an extension of RAG with enhanced code retrieval capabilities based on LLM agents. This aims to deliver more precise code snippets, significantly improving the quality of the answers. Helena uses Llama 3 70B and runs the entire setup on-premise on our own Nvidia RTX 6000 Ada, making it suitable for projects where the source code should not be processed in the cloud. #SoftwareEngineering #CodeAnalysis #ArtificialIntelligence #GenerativeAI #RetrievalAugmentedGeneration #LargeLanguageModel

27

4 Comments

In this article, Baptiste Lefort shows how to leverage the power of reinforcement learning for feature selection when faced with very large datasets

63

2 Comments

GraphRAG is now on Github ⭐! The approach combines knowledge graphs and LLMs in a fully automated way. It promises to solve the biggest problem in RAG. Here is an example: Imagine your companie sells 1000 different products. For each product, you have one manual. What happens, when you ask a classical RAG system: Which products do we sell? Even if your chunk retrieval and the model were optimal, the answer would only list the products that are in the context. This would be <50 for a typical system. Now, how does GraphRAG work? 1️⃣ The system uses LLMs to extracts entitites and build a knowledge graph 2️⃣ Graph ML methods are used to structure the graph into communities 3️⃣ Use a multi-step process based on the community summaries during each Query In this way, you can automatically model complex relationships. In our previous example you would be able to put all product names in the context even if you don't have a manually-prepared document with a product overview. The main limitation I currently see is that it needs a lot of compute. This means that the query latency is very high and each query is quite expensive. Nonetheless, I think this IMHO a very promising direction and I can't wait to try it out. Btw. the repo got 1500 ⭐ over the last 24h. Check out Github and add yours!

73

6 Comments

📢 Hugging face is directly integrated with GenAI developers favorite tool "langchain"🦜 Get set go!🚀 💡 pip install langchain-huggingface https://lnkd.in/ddHreVrs #huggingface #GenerativeAI #langchain

22

RewardBench : benchmark and leaderboard for evaluation of Reward Models used in RLHF alignment of LLMs Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models and which values are embedded in them. In this paper, authors introduced REWARDBENCH, a benchmark dataset and code-base for evaluation, to enhance scientific understanding of reward models. - REWARDBENCH dataset is a collection of prompt-win-lose trios spanning chat, reasoning, and safety, to benchmark how reward models perform on challenging, structured and out-of-distribution queries. - REWARDBENCH leaderboard, evaluates reward models trained with a variety of methods, such as the direct MLE training of classifiers and the implicit reward modeling of Direct Preference Optimization (DPO), and on a spectrum of datasets 𝗞𝗲𝘆 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻𝘀: - framework for evaluating the many different architectures of reward models, along with tools for visualization, training, and other analysis - Illustrate the differences between DPO and classifier-based reward models across a variety of datasets - Chart the landscape of current state-of-the-art reward models 𝗥𝗘𝗪𝗔𝗥𝗗𝗕𝗘𝗡𝗖𝗛 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 i) REWARDBENCH Dataset - curated four different categories: Chat, Chat Hard, Safety, and Reasoning. ii) REWARDBENCH Scoring - primary scoring metric is accuracy - For each prompt-chosen-rejected trio, it infer the score the reward model assigns for the prompt-chosen and prompt-rejected pairs then assign a true classification label when the chosen score is higher than rejected. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - includes evaluation of many public reward models, ranging in parameter count from 400 million (PairRM) to 70 billion (T¨ulu 2), trained as classifiers or with Direct Preference Optimization (when the reference model is available) - In evaluation there are multiple models trained either with the same or very similar fine-tuning approaches on different base models - Different Shapes of Reward Functions used , as per-prompt scores demonstrate the different magnitudes and distributions of rewards assigned to each reward model over the REWARDBENCH evaluation dataset. 𝗖𝗼𝗱𝗲 𝗮𝗻𝗱 𝗗𝗮𝘁𝗮𝘀𝗲𝘁: https://lnkd.in/eATUa2U4 𝗟𝗲𝗮𝗱𝗲𝗿𝗯𝗼𝗮𝗿𝗱: https://lnkd.in/eEwMe7M3 𝗣𝗮𝗽𝗲𝗿:https://lnkd.in/ekfi9wBF #generativeai #llm

31