Subbarao Kambhampati’s Post

Hey everyone, this blog summarizes the effective strategies and best practices that can be followed while trying to optimize LLMs for specific needs/use cases

Almost everything I read on LLMs is either 1) vendor-driven that doesn't help me build a mental model for tradeoffs, or 2) tool documentation that assumes I know exactly what I want to do, just not how. This is an excellent post smack in the middle. https://applied-llms.org/

Call centers of any kind. A friend of mine has developed a software that allows your agent to have the perfect script, in his or her voice, on every call, every objection at their fingertips, anywhere in the script they can click - or have f1 buttons to go straight to the new script or objection or disclosure. Have 20 disclosures to read- no problem-- read them through the software one time and never read them again- in your voice. No ROBOT- NO AI , Legititmally your voice, on every call, and you're on the line. Just push a button to read those disclosures for you- or objections that you have a hard time with. Let this software close the deal for you-- with your voice. Free 14 day trial. Click here now. https://lnkd.in/gp3_z_DM

What We Learned from a Year of Building with LLMs (Part III): Strategy: We previously shared our insights on the tactics we have honed while operating LLM applications. Tactics are granular: they are the specific actions employed to achieve specific objectives. We also shared our perspective on operations: the higher-level processes in place to support tactical work to achieve objectives. But where do those objectives come from? That is the domain of strategy. Strategy answers […]

We’ve all experienced the “EMERGENCE” of LLMs in suddenly being very useful. Jason Wei et al (https://lnkd.in/g_JgmWki) claim emergence mathematically in the sense of an unexpected change in performance for larger models (FLOPS, # parameters). But Rylan Schaeffer et al, in their NeurIPS 2023 prize winning paper (https://lnkd.in/g54AD5Rp), claim this is an illusion. A few assumptions yield an expression for single token probability as a function of cross-entropy which is empirically seen as decreasing as a power law in # computations. Any criterion like accuracy which depends on a correct decision based on several tokens (eg L tokens) being correct is necessarily going to lead to a product of the probabilities of the single tokens being correct. I’ve taken the liberty of putting it all together, fitting the two constants (⍺,c) to points in the reference data. And yes the behavior shown in the figure below reflects Wei’s results but the fact that these results are predictable counters the claim of emergence in Schaeffer’s mind. Forget these semantics, to me the great joy is in this expression as a design tool. Two constants, (⍺,c), characterize the network type, the accuracy (or a range of others) criterion including L set the performance requirement, and boom we can identify the size of model required.

Here are a list of things I've found interesting to read about LLMs, over the last few months - https://lnkd.in/ek6VrU8m

Once you've fixed all the retrieval errors (RAG), next step is the generation errors (LLM).

🔍 Did you know that a recent paper has provided mathematical proof that #hallucinations are inevitable? Hallucinations, a phenomenon in #LLMs where incorrect information is presented as fact, pose a challenge in leveraging these technologies effectively. So how can you maximize the utility of LLMs while understanding their strengths and limitations? Here are two critical points: ✅ #RAG: Separating knowledge retrieval from generation using an external discovery system like enterprise search grounds LLM responses on real enterprise knowledge, significantly reducing hallucinations. ✅ Traceability and Referenceability: Providing backlinks as references for generated answers allows users to easily verify the source of information. By combining these methods, models can consistently access accurate information and instill users with confidence in the reliability of their results. 💪 In this approach, the heavy-lifting is done by a search engine that must be scalable, secure, robust, and integrate seamlessly with all your document sources while respecting the access rights. That’s what we do at DeepJudge. Contact us to learn how to harness the power of LLMs effectively within your organization, ensuring relevance, recency, and permissions awareness in every interaction. Let's make your results count!

Retrieval Augmented Generation (RAG) is one of the most talked about and effective methods for optimizing LLMs. But does that mean enterprise businesses should be using RAG? Check out our RAG 101 for Enterprise walkthrough to learn more about: ✅ Whether or not enterprise businesses should be considering RAG ✅ How enterprise businesses should be approaching LLM optimization ✅ How RAG works and ways to increase RAG performance ✅ Challenges enterprise businesses will encounter ✅ How Gradient & LlamaIndex can help you overcome these challenges https://lnkd.in/g2v9B8YP #LLMs #Finetuning #RAG #GradientAICloud #GradientAI

Rule-based systems and simple evaluation metrics are the low-hanging fruit for LLM evaluation, but they won’t be enough for use cases that need human input and insight. An LLM evaluation system needs to have direct human evaluation of quality as an integral part of the solution. Optimizing for user interaction, especially on complex outputs, requires optimizing for multiple metrics, including correctness, friendliness, and response context. There won’t always be a single model+prompt that will be best in all of them, so the user will need to make trade-offs, and maybe leverage the balance between them to feed the next iterations. An LLM evaluation system needs to be able to support multiple evaluation criteria in order to be truly “successful.” The Verta Workbench helps users evaluate both OSS and Proprietary LLMs quickly and efficiently so you have more time to spend iterating, innovating, and interacting with your customers and clients. Come build better with us! https://hubs.li/Q02bpXJX0