Sheyda D.

Today, I found a fascinating way to think about “Tokens” in LLMs. It came from a conversation between Jensen Huang and Patrick Collison at a recent Stripe event. When asked about the future of compute capacity in five years, Jensen smartly dodged that question a bit, but then gave an excellent explanation and analogy about why tokens are the new forces that will power the next decades of humanity. He explained that we're now producing something unprecedented (and at scale): floating point numbers that possess “value”, which we now call tokens. These tokens are valuable because they encapsulate “intelligence”. People are now taking these tokens and transforming them into English, French, images, videos, chemicals, proteins, robotic movements, etc. And many are working hard to expand the range of concepts and ideas we can create with these tokens. He then goes on to made a compelling comparison between tokens and electricity. In the past industrial revolution, we successfully found a way to convert “atoms” into “electrons” (by boiling water to power electricity turbines). And now, we've discovered a way to convert "electrons" into "tokens" (by using energy to power data centers that train and run LLMs). When electricity was first introduced, few people understood its value. Today, paying for kilowatts is routine. The same will happen with tokens. Right now, only early adopters and builders are paying tokens. Soon, everyone will be paying for tokens on a daily basis to supercharge productivity and power new products and services. Many new industries will be born from and built on top of tokens. When I first heard about this way of thinking, I got goosebumps. Perhaps it’s because Jensen has a good storytelling skill that helps him sell what he builds, but I have to admit this way of thinking gave me a brief surge of pride. I'm proud of humanity's collective effort and creativity, which have taken us from living under rocks to building machines that can "think." That's nothing short but a miracle.

18

5 Comments

Batch processing for LLM inference activity: "One sign of that is the rise of batch processing. Today, popular consumer AI products like ChatGPT and Perplexity provide users with near instantaneous responses, otherwise known as real-time inference. However, businesses don't always need immediate responses and are often willing to wait hours, days, or even weeks for responses, as long as they don't have to shell out as much money. Founders of cloud and inference providers have told me there's growing pressure from business customers for this kind of flexibility, otherwise known as batch processing." -- Nathaniel Whittemore on "The AI Daily Brief" (with much deserved shout-out to his inspo from Stephanie Palazzolo at The Information) I fall into this bucket as we have a lot of backfilling to do as we're building out the underlying data in our service, and some of that could just as well be done in the wee hours of the night. The new stuff flowing in I want processed ASAFP, but these backfills—I'd mos def take more cost-efficient delayed batch processing. There's also use cases like needing (near) real-time inference for the first inference pass for timeliness / contemporaneousness value, but then a second pass slow-time inference for wider classification exercises over an ever-growing data corpus. I could see us mixing Sonnet & Haiku models in both cases. Just a heads up to Anthropic Mike Krieger as you contemplate this, along with Amazon Web Services (AWS) Jeff Barr Mark Roy Swami Sivasubramanian on the Bedrock side of things...and I don't mean the "batch" mode that doesn't seem to have any real value other than helping AWS manage its own rate limit challenges. (DeepCast is using Bedrock.)

11

2 Comments

Small Language Models (SLM) are being discussed a lot lately, but why? Well for starters LLM's are costly and slow. SLMs are the fields newest solution to the bank account draining capabilities of LLMs, but most of the writings I've seen on SLMs is focused on the technology and use of the SLM itself. The real power of smaller models (even traditional MaxEnt and CRF models) is that they are incredibly fast and practically free in comparison to an LLM. What's been missing in the SLM discussions that I've seen is the idea of a hybrid approach where you use LLM's to feed high quality training data to SLMs so that you have amazing insights that can run on high volume content streams at a very low cost. Something to think about...

31

Who will get crushed by Claude LLM coming to iPhones? Will existing apps (and companies) get steamrolled by this? For those that don’t know Claude is a suite of 3 LLM models: Haiku, Sonnet and Opus. - Opus knocked ChatGPT off the leaderboard on capability when it was introduced. - It can hold a massive 160,000 words “enough for a user to paste in a weighty novel and ask follow-up questions” - Claude is multimodal so you will be able to take pictures or upload images from your iPhone and include that in your prompt Use cases * After a meeting, a business user could snap a photo of a whiteboard diagram and ask Claude to summarise the key points, making it easier to share and act upon important information. * Similarly, a consumer could take a picture of a plant they encounter on a hike and ask Claude to identify the species and provide more information about its characteristics and habitat. ** think the Picture This app ** What other uses do you see for this? https://lnkd.in/gdExfpx9

6

InVision has experienced the biggest fall from glory in Silicon Valley's recent history. $350M raised with a peak valuation of $2B. And now they are shutting down. Not acquihired or "acquired by private equity" or strategic tuck-in. Just a hard stop. It's so easy to s**t on them. They got out-innovated, they were not product driven etc. etc. But, the truth is most startups that raised monster rounds in ZIRP days could face this deadend. You raised too much money: 1/ So you are too expensive to acquire 2/ So you raised your burn too high to finance a big GTM team 2/ So you got complacent on product 3/ So you got lazy on shipping There will be a lot more inVisions in the next 24 months as startups run out of capital. It's sad. The employees & founders at these companies gave it their all, but raising too much money sealed their fate. For the next gen of startup founders, heed this advice: raise little, keep your burn low, ship fast!

29

In the wake of OpenAI's acquisition of Rockset, which BTW is a a great outcome for the Rockset team and a bullish signal for the importance of real-time data for AI in the enterprise, we're finding out that Rockset was pretty darn unique. Rockset's customers have been scrambling to find alternative solutions before the service is sunsetted (for month-to-month customers) on Sept 30. The challenges we've seen folks face fall into 2 buckets, solving for CDC/ETL from NoSQL DBs and solving for some of the unique workloads that Rockset could support, we're mostly seeing ClickHouse as the choice to replace but the overlap is not 100%. I'll address that in another post. Rockset was very good at ingesting from NoSQL databases directly and had a lot of customers using them for MongoDB and DynamoDB data. We're helping folks with MongoDB data already and while there are complications with nested data for many Rockset alternatives, we've already solved this for our ClickHouse destination so it's not been too tricky. CDC DynamoDB is a bit of a different animal, it's not as broadly adopted given it's specific to AWS so there just isn't as much support in the open-source world. We're working on a solution for this and will have one soon but would love to talk to folks about what they are doing. There are 2 basic approaches for streaming updates: DynamoDB Streams - The preferred approach, offers exactly once delivery but is constrained by a 24-hour limit on retention of change events which means careful management of consumption to ensure you're not losing the end of the log and having to re-snapshot/backfill data again Kinesis Data Streams - Fully managed and serverless option if your destination supports Kinesis, if not you will have to connect with Kafka or other solution and deal with additional complexity of multiple streaming platforms. Kinesis data streams only supports at-least-once delivery so you will have to manage duplication. In addition to paying for Kinesis you will have additional fees for DynamoDB CDC with Kinesis. My sense is that this will be limited to businesses that are planning on just using AWS services but I'm interested to hear what folks are doing. Separate from streaming data, you also have to solve for backfilling historical data if you're setting up a new pipeline. For this, you'll need to either use the DynamoDB API to query the DB (best for tables under 5GB) or use the bulk export to S3. If you're working on solving these problems drop a comment or DM, would love to chat. #cdc #changedatacapture #rockset #dynamodb #mongodb #openai #streamingdata #kafka #kinesis #realtimeanalytics #realtimedata #clickhouse #olap #nosql #realtimedatabase #realtimedb

38

7 Comments

Here is a prime (and hilarious) example of terrible use of LLMs. You are trying to sell me a product and the image below is your example. A couple problems: First, our business has nothing to do with "enhancing railways", so whatever "research" your Cohere and Claude Opus models did was worse than useless. The rest of the fluff in the paragraph is based on the misinformation so it begins to sound like buzzword bingo. Second, you've demonstrated that your product will miss the mark so badly, it'll create a massive amount of noise and work for us that we'd lose money trying to use it. "PS: Full disclosure, we let our AI and automation handle the copywriting and outbound process this time. We decided to use Cohere for research and Claude Opus for copy, aiming to one-up the usual ChatGPT antics. If it turns out we've missed the mark, shoot us a message and we'll reevaluate (or atleast give our AIs a pep talk)!" Stop churning out garbage. Building solutions on LLMs requires the same level of thought, planning, and execution as any other project.

10

6 Comments

I wrote a blog post on a thorny but underappreciated problem in ML-based marketing personalization: event attribution. It's how you go from the actions you take and customer behavior you observe to creating training experiences for your model. Good attribution makes a huge difference in driving uplift, there are a lot of degrees of freedom in configuring it, and at the same time, it's rarely discussed in ML/bandits/RL literature.

38

4 Comments

Is AI hype overshadowing true technical innovation? The true measure of AI and machine learning isn't found in the buzz but in the substance of the work behind it. From my own experience, I've noticed a trend: projects that look great in a demo but fall short when showing a proper enterprise use case, i.e. they fall short when it comes to substantive deployment. The key takeaway? As AI continues to evolve, the real game-changers will be those who focus on the bedrock of AI – the fundamentals. I’ve discussed this and other AI-related stuff in our AI-searched podcast with my colleague Tanya Ariana Bendis So go, check it out. https://lnkd.in/dKp39pN9

37

3 Comments

Very insightful work by Anthropic’s interpretability team. And an amazing paper, with outstanding writing and figures. The idea is very simple: interpret LLMs by leveraging sparse autoencoders as surrogate models of the MLP of transformer blocks, which allow one to disambiguate the superposition of features captured by a single neuron. A simple idea, but a very careful and complex execution, as it is often the case in our line of work. The paper goes into many details and provide a large array of insights, although the gist of the implementation remains obfuscated due to the closed source nature of Claude. Too bad, because this is the kind of work that we need to better understand and eventually trust LLMs. This is demonstrated by the authors in the section ‘Influence on Behavior’, where they show that clamping some features to either high or low value during inference is “remarkably effective at modifying model outputs in specific, interpretable ways”. Hopefully this kind of work is going to be replicated and generalized to open-weights models, such that we have new ways to steer their behavior. https://lnkd.in/eVym7f_f #interpretability #xai #explainableai #steerableai #anthropic #claude #anthropic

3

The Stanford Institute for Human-Centered Artificial Intelligence (HAI) just released their 502-page AI Index Report. Some of their most interesting takeaways for AI in 2023: ✅ The number of new large language models doubled in 2023 (over 2022). Of those released, 66% were open-source. ✅ ...But on select AI benchmarks, closed-source models outperform OSS ones with a performance advantage of ~24% (very substantial). ✅ AI can outperform humans on standard tasks like image classification but are far inferior in things like commonsense reasoning. ✅ The cost to pretrain your own large foundation model is still astronomical. Google's Gemini purportedly cost $191M to train. ✅ Of the notable AI models, 61 originated in the US far outpacing other countries including China's 15. This is especially interesting because several years ago there was concern that China would overtake the US AI advantage. The trend seems to have reversed. ✅ The # of AI regulations went up by 56.3% indicating policy makers are becoming more conscious of AI's far-reaching societal impact. ✅ More AI models are being used to generate date to train AI models. This is a huge deal because it suggests the systems are gradually getting to a point where they can self-learn. (Cue ominous "the singularity is near" track...) #ai #llms #foundationmodels #aiindex

21

5 Comments

Day 3/3 GenAI Summit San Francisco Causal AI, causal grounded RAG - cRAG, scaling from POC to production in AWS, a deep dive into the evolution of Llama models and what's coming next, synthetic data in the age of genAI, PII, and domain randomization. I can't count how many times I've heard the words AI, genAI, data, cloud, RAG, LLM, and multimodal in the last 3 intense days. And when I finally went to have lunch at a restaurant, at the adjacent table, a fintech AI startup founder was pitching to a VC, while the people at the table behind me were talking about data engineering. It feels so good to be in SF, where at the airport’s security check-in a big Anthropic advertisement greets you! 😊 The last 3 days were amazing, filled with inspiration and thought-provoking new tech. I can't wait to get back to building DiffuseDrive and working with our customers!

22

Large Language Models (LLMs) as super lossy compression. I have been thinking a lot about demystifying what LLMs do because lots of y'all's are very mystified. In the linked article, I try to reframe them for you, hoping it will help you understand what they good for and what they are not good for. Side note: I am still looking for my next adventure. If communicating complicated concepts so that regular people can grasp them is something you need, you should message me. I write code and stuff too and am usually fun to be around. #WrittenByMe

6

3 Comments

Great example that shows 1) One cap doesn't fit all. You don't need a single LLM for everything 2) With fine tuning you can get better performance from a small model than a huge LLM with trillions of hyperparameters. Result: more efficiency and lower TCO!

9

Almost twenty years ago, in 2000, Eric Brewer introduced the concept of a fundamental trade-off among consistency, availability, and partition tolerance. This concept, now known as the CAP Theorem, has been instrumental in evaluating distributed system architectures for decades. With the rapid proliferation of streaming platforms, there's a growing need for a similar model to evaluate the trade-offs between Cost, Availability, and Performance in data streaming platforms. In response, we introduce the New CAP Theorem tailored for cloud-based data streaming systems. The New CAP Theorem is designed not to rank systems but to emphasize the importance of making informed choices based on a balanced consideration of Cost, Availability, and Performance. We encourage everyone to utilize the New CAP Theorem to select the most suitable technology from various data streaming platform options, including #ApachePulsar, #ApacheKafka, StreamNative, Confluent, and many others, to meet your unique business requirements on your data streaming journey. https://lnkd.in/gcuqViBm

45

4 Comments

Spoilt for choice - Llama this or GPT that? The last 2 weeks have been intense when it comes to LLMs. With so many releases packed into April, it is hard to keep up. Here's the lay of the land (as of today) to help catch up, wherein we shall discuss: 1. Anthropic's Claude Opus: Released mid-March 2. Cohere's Command R+: Released Apr 04 3. Google's Gemini Pro public access: Released Apr 09 4. OpenAI's GPT4-Turbo-2024-04-09: Released Apr 09 5. Mistral AI's Mixtral 8x22B: Released Apr 09 6. X’s Grok 1.5 Vision: Released Apr 12 7. Reka AI's Reka: Released Apr 15 8. Hugging Face's Idefics2: Released Apr 15 9. EleutherAI's Pile T5: Released Apr 15 10. Microsoft's WizardLM-2: Released Apr 15 11. Meta's Llama3: Released Apr 18 Happy Reading! #ai #LLMs #opensource #tech

7

3 Comments

Falkonry (Suhas Mehta and Ashis Khan) and iba AG (Eric Snyder) will present how steel makers are pivoting to AI & Data driven decision support system at AISTech 2024 today (May 6, 2024) - please join us if you are interested. I co-authored this paper with Eric Snyder of iba AG.

66

5 Comments

Very useful insights for folks who are starting to dig further and further into the realm of LLM. An easier GUI approach for fine tuning LLMs.

9