Chris Hazard

Was a bit surprised to see Wired touting RAG as "taking hold in Silicon Valley" like it's the hot new thing - RAG has been around for ages (at least in LLM terms) and I actually think its application space is currently shrinking as context windows grow. Their "reduce AI hallucinations" thing is not really the whole story - I mean, yes, RAG can definitely help in the sense that asking a broad question to the base model (with no input other than the prompt) is likely to result in hallucinations, whereas feeding in concrete data (with RAG or otherwise) will ground the response in that data and therefore produce fewer hallucinations. But RAG is really just a tool for feeding that data - and, yes, for massive search tasks (like broad legal research requests across entire databases), RAG may be the best (if not only) way to do it. But for smaller tasks (like analyzing a single document) it's often better to just feed the input in directly, which the larger context windows of the newest models increasingly allow. I feel like a few months ago (around the time Opus was released) people were asking whether RAG would be obsolete - that was an overreaction, but still it doesn't really fit this narrative of RAG as some hot new thing in the LLM world. Maybe I missed the pendulum swinging the other way again? Not sure. https://lnkd.in/eUEwBmFf

3

Want bounded-error partial updates of your SDF discretizations? See my SIGGRAPH 2023 presentation on Temporal Set Inversion for Animated Implicits!

56

4 Comments

🤯🤯 See Beyond the Pixels with Gemini-1.5-Flash & Pro-Vision-001: GenAI with Eagle Eyes! 🤯🤯 Tired of boring charts and graphs? 😴 Wish your graphical data could tell a more interesting story using LLM knowledge? 🤔 In this medium blog, 'Gemini-1.5-Flash & Pro-Vision-001: GenAI with 20/20 Vision', we're discussing sample fun images related to: 1. Cricket 🏏 World Cup 🏆 2. Fruit Sugar Fights 🍎🍌 3. Battery Blues 🔋 4. Chip Bag Mysteries 🥔, 5. TV Show Characters 📺🎬🎭, and 5. Olympic Host Cities 🏅🏟️ All decoded by the power of Gemini: Multimodal GenAI! 👀 #AI #GenAI #Gemini #Flash Google Cloud

2

Did you hear the news? ✨ With just a single line of code, Astra Vectorize makes it easy to use your data with generative AI, from generating your vectors, storing and indexing them, and them using them for search at lightning speed 💡 You can focus more on the *logic* of your application, rather than the intricacies of aspects such as vector embeddings. Read more here 👇 https://ow.ly/Ni1350RRAtV #MSBuild #GenAI #RAGApplications

10

If you thought software engineers were grumpy before, just wait until they start working with open-ended AI systems. One of the biggest challenges in designing AI systems is establishing guardrails to the behaviors. For example, in traditional software, you can click one of two buttons...two options developers have to deal with. In language-based systems, the inputs are theoretically infinite, which means I have to try to constrain the user to a semi-finite set of options. Without limits, users get frustrated by bad responses, hallucinations, and unintended consequences. In short, engineers now need to enforce policies instead of explicit rules and that is a substantial increase in complexity (and therefore grumpiness). https://buff.ly/3KvNOnq

3

🎙️ Dive into the world of Generative Voice Assistants with us! In our latest episode, we break down the fascinating anatomy of these complex systems and reveal the magic behind the scenes. Curious about the science? Follow us and join the journey as we peel back the layers, one episode at a time! 🔍✨ #VoiceAssistants #TechTalk #BehindTheScenes

22

1 Comment

I was chatting about translation with Andrew Ng at an event several weeks ago. It's fun to see him post a workflow with the seed of the AI Review concept that's live in LILT today. === This is along the path to what many in the industry are doing. Two comments: 1. Replace step 2 with a formalized reflection approach such as AutoMQM. 2. BLEU isn't discriminative at the accuracy levels modern MT systems achieve. Use a trained metrics like BLEURT or COMET to see real differences between systems. 20 years ago we called this idea automatic post-editing, where you'd train a second string transducer to rewrite initial MT output. However, those systems were very difficult to train and cascade. Using LLMs in cascades is significantly more effective.

42

GPT-4o character voices.... impressive, but they do sound a bit like the base voice doing impressions vs unique characters. The intonation and emotional control is impressive. https://lnkd.in/gKz5_dzx #Storytech #AIStorytelling

7

1 Comment

I recently built myself a new PC, it's been sometime since I have created something with a little more oompf. Following up on that I tried out Nvidia's ChatRTX running locally on my RTX4060. Very interesting to see the progress that has been made, as compared to my first encounter with Ai-LLM on my old machine, which was horrific, especially since I never had an RTX and was relying purely on 3rd gen i7 Cores on W10. Yes I know !! silly me. Fast forward to a W11, Ryzen 7, RTX4060 & Nvme world .. my pants are on fire !! With most of my prior use cases in life having low requirements for (SaaS) i've not needed it, until I tried local Ai-LLM's, creating 3d models fro my 3d printing interest and or trying to edit a 4k 60fps 1 hr video !! from my DJi. My production time was cut from 220 minutes !!! down to 10 minutes, that's 22x improvement ... hardware has not given me a cheap thrill for long time ... I am gobsmacked at the more recent hardware capabilities. Although IT management, personal development, coaching and so on are my more current abilities, I do enjoy keeping connected to where it all started, the same goes for programming with the likes of languages such as Python. One thing that has always been cool about Tech, is the nature of the enthusiast within it. IT people can have just a weird a conversation as that had between Jay Leno and Jeremy Clarkson about cars who are likely just as thrilled about steering wheel textures as we can be about GHZ factors. Enjoy your craft.

8

1 Comment

Let the games begin... It appears that one of the few active defenses against AI mimicry that artists posting online could employ has now been circumvented. Though this exercise was considered an act of futility, it is analogous to "a thief taking a safe home to crack it at their leisure." Especially when traditional methods like watermarking, style transfer detection, or other forms of digital rights management are easily removed through recent AI tools. To see the full details around Glaze they have a good post on their website https://lnkd.in/gYxZt4d7 Once art is posted online unless its protection mechanism is able to adapt or continuously change, it would always be a point in time solution. Making it vulnerable to the continuous development of new technology, methods and time itself. A typical cat and mouse game, where the bad actors mimicking artists styles without representation are able to use social media photos of art shows, galleries, and self posted media to fuel an AI able to create other works of art with the artists likeness. At the end of the day, educating yourself as a buyer, engaging directly with artists, and doing your due diligence are the only ways to ensure you are buying art that an artist spent their creative and technical skills to create. I hope we continue to seek solutions to ensure any work can be correctly credited to its creator or better regulations to fairly manage the rights of any works. I also hope this doesn't slow the creativity that exists in various communities or lead to protectionist behaviours where innocent social photography is not allowed in a gallery. More information is available in this great article from ArsTechnica https://lnkd.in/gtF9xKmf If you are interested in Glaze or Nightshade, guide from lifehackers https://lnkd.in/g5tA-9rK There are alternatives to Glaze as well like Mist, this is a paper talking about it. https://lnkd.in/g8CryDUH

1

I do a lot of AI automation work, much of it involving batch processing and reprocessing of vast amounts of text. The low cost and high quality of Claude 3 Haiku (25 cents per million tokens in and $1.25 per million tokens out) has been fantastic. But today, OpenAI released GPT-4o-mini for use in API calls, driving the cost of AI automations even lower: 15 cents per million tokens in and 60 cents per million tokens out! To give you a sense of the savings: I once accidentally ran my batch processor using the high-end GPT-4o model. With 12 million tokens submitted and 2.5 million tokens returned, it cost me $97.50. With GPT-4o-mini, today that cost is only $3.30. That’s over a 96% savings! 😲 I have reviewed the Llama 3 8b + 70b Meta open-source models for these automations. The cost is lower still (the model is free, but hardware time is not). However, in my tests, the quality just isn't the same. Plus, considering the additional effort to host an API to access the Llama 3 models, my engineering time and money have so far been better spent using the low-cost models from OpenAI and Anthropic. Anyone else finding similar results in savings + quality?

27

1 Comment

Cryptographic agility should be part of your mind set as you plan out your risk and threat mitigation strategies. To achieve “security by design”, it is crucial to consider this well in advance. Cryptomathic can help 🫶

18

Incredibly unexpected and super cool fact about NVIDIA GPUs: matrix multiplication speed depends on the content of your matrices! :) Surprised? Read on! (note: it's not only for NVIDIA GPUs but i'll use them as the running example) This is one of those moments when your software stack is a leaky abstraction for the underlying hardware. Conceptually, if we were in the "Platonic world of ideas", the speed (FLOPs / sec) should only be a function of the shapes of your matrices. If matrix A is of shape (m,n) and B is of shape (n,k) then the speed with which we compute C = A @ B (@ == matmul) of shape (m,k) should only be a function of m,n, and k, i.e. f(m,n,k). At least that's what a reasonable ML engineer / scientist expects. But why is this not actually the case? The answer lies in the fact that GPUs are power limited, they are designed to operate under certain power thresholds (see e.g. TDP), and power is a function of your GPU clock frequency and the switching frequency of the underlying transistors. Roughly P = clk_freq * switch_freq. The more the transistors switch (as you feed them alternating voltage that corresponds to your positive / negative numbers), the more current is flowing through your chip and the so-called dynamic dissipation power increases (as opposed to the static dissipation that happens no matter whether the transistors are switching or not). So it turns out that if you multiply a matrix with all 0s you end up with less switching, your GPU doesn't get power-throttled and the clock frequency remains stable (and thus your FLOPs / sec). But if you use a matrix initialized with random numbers (esp. Gaussian) you end up with more switching which causes the GPU to drop the clock frequency in order to stay under the power threshold and this reduces your FLOPs / sec!! Unless you're deep into optimizing your ML/LLM software stack you mostly won't care about this. Similar to how compiler quirks can sometimes impact your software behavior, but the majority of programmers will never ever have to think about it. But it's still fun to know, and as a bonus, you'll be fun at parties. :') (maybe in SF?) Ultimately an amazing metric to track GPU efficiency is how much compute you can squeeze out for a given amount of power: FLOPs / watt. That's the gist of it. If you want to know more Horace wrote a more in-depth explanation of the phenomenon here: https://lnkd.in/dpMmG78K

510

24 Comments

There are many variations of the "Nvidia CEO Jensen Huang..." meme at this point on LinkedIn, so here's a different take: Nvidia was a company that was incredibly lucky that their R&D happened to neatly dovetail with the needs of machine learning. For many years, only PC builders like me cared about the classic Nvidia vs. AMD GPU performance battles, and in that context, most of us were only looking at gaming benchmarks. (*Insert "Can it run Crysis?" meme here*) Then one day, a bunch of Bitcoin Bros with Vapes™ drove up the price of the GPU market because they thought consuming the same amount of energy as the entire Philippines was a good idea to solve math problems. Those math problems were stored in digital-coin form, so international crime syndicates could easily launder money. When money laundering fell out of favor, someone convinced impressionable Technology Influencers with Vapes™ that digital Beanie Babies would be worth hundreds of thousands of dollars someday. So if you buy this JPG of a cartoon ape, all you need to take your actual REAL money and turn it into worthless math problems to buy cartoon apes. What could possibly go wrong? How do you solve those math problems to convert valuable USD into fake money to buy cartoon apes so crime syndicates can continue generating revenue? With an entire server room filled with Nvidia GPU's of course! Then, circa 2021 people got wise. "Wait a second, this s**t isn't real money, and I can still just download this JPG. Also, my grandma just got a call from some dude in India who convinced her to convert her Social Security into BTC and send it to an 'Amazon Refund Department' and literally nothing can be done about it. This sucks." People started dumping their GPU's on the open market. Thankfully, this mega-surge in now cheap GPU's allowed AI researchers to pick them up for pennies on the dollar. When Bitcoin Bros with Vapes™ lose, society wins. Suffice to say, yes Nvidia CEO Jensen Huang seems like a good leader and a smart dude, but his management style isn't really what drove Nvidia to success. All those R&D bucks spent accelerating CUDA cores for gaming nerds just happened to line up with the needs of AI/ML engineers. I know this, because I remember when Jensen Huang would do his big keynotes talking about how Nvidia GPU's performed on different gaming and productivity workloads. (Usually graphic designers or video editors.) The attached photo is an old lady sending money to a scammer on a Bitcoin ATM, generated with Midjourney. The tragic irony here being that it took a lot of little old ladies getting scammed to turn the GPU market into what it is today. The moral of this story is that crime does pay, as long as you're the one selling ammunition. Have a good weekend!

43

12 Comments