Abhijeet Kandalkar

#Day23 and #Day24 of #30daysofcode Learning Outcomes - 1. Climbing Stairs: Dynamic programming for combinatorial problems. Bottom-up approach for efficiency. 2. Edit Distance: Dynamic programming for string manipulation. Optimal substructure in insert, delete, replace operations. 3. Unique Paths: Dynamic programming for grid-based scenarios. Bottom-up approach for optimal path calculation. #CodeJourney #CodingAdventure #CodingChallenge #LearningToCode

14

New blog on Modern Emacs Typescript Web (React) Config with lsp-mode, treesitter, tailwind, TSX & more https://lnkd.in/dMuJ4xnd This one took a while to write. It covers my entire setup for making Emacs a modern, performant IDE for web development.

6

Excited 👾 to share an upgrade for unique identifiers in databases: ULIDs! UUIDs (GUIDs in C#) are a common choice in distributed systems due to their randomness, but they have notable drawbacks: 1. Performance Issues: UUIDs slow down database inserts because their randomness requires frequent rebalancing of the clustered index (a B+ tree). 2. High Storage Costs: UUIDs are 128 bits long, and storing them as strings further increases storage requirements. ULIDs address these problems effectively. They are also 128-bit, making them compatible with UUIDs, but with a key difference: ULIDs are sortable. The first 40 bits of a ULID represent a timestamp, making them monotonically increasing. This leads to better database performance by reducing the need for rebalancing operations.

9

Did you know most open-source LLM models need less than 512 H100 GPUs if the training can span a couple of weeks? With the next-gen Blackwell GPUs, with 2x FLOPs, we need even fewer GPUs for similar training times.   Hyperscalers often build gigantic clusters to run hundreds of training jobs simultaneously and for quick turnaround times for applications needing continuous user data training. However, smaller clusters are more than adequate for most enterprises that either train or fine-tune open-source LLM models.   The network fabric for these clusters could consist of shallow-buffer high-radix switches connected in a fat-tree topology or a modular system that can offer 4x-8x throughput over the shallow buffer switches. For example, the PTX10K8 chassis with #Express5 offers 230Tbps throughput with any-to-any connectivity of 576 x 400G ports where PFEs in the line cards and the fabric switches are connected in a non-blocking CLOS topology - enabling 576 GPUs to be connected with a single system.   PTX10K provides VOQ scheduling to the output ports to eliminate in-cast congestion. Further, packets leaving the PFE towards the switch fabric are broken down into cells, and the cells per destination PFE are evenly sprayed across all available links, achieving over 95% link utilization. Our fabric switch utilizes a buffered crossbar architecture with no head-of-line blocking, sustaining these high link utilizations without backpressure. While cell spraying maximizes link efficiency, it does introduce the challenge of re-ordering and assembling packets from these cells.   How do we tackle reordering? 🤔   Reordering fabric cells is a challenge at which we have always excelled. In fact, our CEO, Rami, who designed the reordering logic for one of our early systems two decades ago, considers it one of his career highlights! At traffic rates of 14.4Tbps per PFE in current systems, with around nine cells to reorder every cycle, the architecture of the cell reorder module is critical to optimize area/power.   As usual, our engineers thrive on such challenges. While I can't disclose all the details of our patented methods, we honed an ingenious approach that writes unordered data into a central buffer while reordering the metadata using a tree-based scheme. This allows for flexible resource sharing across hundreds of reordering contexts, with multiple pipelines working in parallel… The outcome is a compact, highly efficient engine. Combined with our non-blocking fabric switch, this technology enables a true 230Tbps system, connecting GPUs most efficiently in small clusters.    How about large clusters? 🤔 A hybrid approach with modular systems at the spine/super-spine and shallow buffer leaf switches (Juniper's QFX) balances the scale-out and scale-up approaches for better performance, power, and congestion control.   "If your team builds something remarkable, don't hide its light under a bushel" -chatGPT 🙂 #LLMs #genAI #GPUs

226

21 Comments

𝗟𝗹𝗮𝗺𝗮 𝟯 𝘂𝗻𝘃𝗲𝗶𝗹 𝗮 𝗳𝘂𝗻𝗱𝗮𝗺𝗲𝗻𝘁𝗮𝗹 𝗰𝗼𝗺𝗽𝘂𝘁𝗶𝗻𝗴 𝘁𝗿𝗮𝗱𝗲𝗼𝗳𝗳 𝘁𝗵𝗮𝘁 𝗳𝗲𝘄 𝗮𝗿𝗲 𝘀𝗲𝗲𝗶𝗻𝗴⬇️ 🦙Llama 3 just came out and to the surprise of many people, the L3-8B version can match the L2-70B one on several benchmarks. Several people also argued that the chinchilla paper (https://lnkd.in/dXaRqyea) was wrong because the llama 3 scaling is very far from chinchilla-optimal. 🤔 In my opinion, this is due to a misunderstanding of what the chinchilla paper is about and more generally of the effects of scale in deep learning. The bitter lesson (https://lnkd.in/d3sVYYgp) has become common knowledge in the field thanks to the GPT family that popularized it. We hear everywhere that "bigger models are better": Yes it's true, but in what sense? Many people have a mental model along the lines of "bigger models are more capable" meaning they would be able to achieve levels of performance that smaller models can't. ❌ This is not true with LLMs and is mainly because we train only once on tokens (most of them at least) and therefore overfitting is not possible. The 𝘳𝘦𝘢𝘭 advantage of bigger models over smaller ones is that they converge *faster*. In other words, you can reach a given level of NLL (modeling performance) with fewer training iterations. ❔ Then a natural question arises: Should I train a smaller model with more or a bigger model with fewer iterations? That's where the chinchilla paper brings a response. They say that the question they are answering is: "Given a fixed FLOPs budget, how should one trade-off model size and the number of training tokens?" (implying "to achieve the best possible NLL"). 🧐 Coming back to Llama 3, it's obvious that the Llama team is aware of this work and there is a reason why they didn't follow this recipe. My guess is that they are taking into account the inference cost in the balance (which is completely out of the scope of the chinchilla paper). When one "overtrains" an 8B model, one ends up with a worse "NLL to training compute" ratio, but also with a better "NLL to inference compute" ratio. ✅ Now if we think about it from a business perspective, it's a very powerful tradeoff because you can choose to buy performance either at a fixed cost (training) or at a variable one (inference). Hope this post helped you! #Llama3 #Meta #LLM #AI

9

3 Comments

Some of you probably know that saving passwords in your database in plaintext is a very bad idea -can’t stress this enough-. And that you need to hash the password with a random string of bytes called a “salt” that’s unique per password and saved along side the hashed password in your database This achieves multiple things: - if your database -in a worst case scenario- gets leaked, the attacker won’t be able to figure out the plain text passwords - and the attacker won’t be able to use rainbow tables of common passwords, because of the salt being random and unique for each password, I.e even if their plaintext values are the same There’re multiple algorithms to achieve this hashing in the most secure way: - PBKDF2: this one if used with a strong hashing function like SHA256 or 512 and in addition to a high iteration count in excess of 2^16 can be okay ish to use for low sensitive passwords - but there’re much safer hashing algorithms like bcrypt and Argon2. Those can protect against the attacker using a special hardware like GPU’s to crack the hash, i.e the same hardware widely used for cryptocurrency mining operations

1

The Go team does such a good job pushing forward while reasonably balancing backwards compatibility. It's way too easy to be axiomatic here and get swayed by overfocusing in one direction. Here's an accepted proposal for Go 1.24 to increase the minimum Linux kernel version to 3.17. They clearly articulate the security benefits to the project and users, pushback on the inevitable complaints, and find a good tradeoff (3.10 plus backported getrandom support). This is an update to a kernel version released *10 years ago*, and there were still compatibility concerns! I'd expect compatibility windows to start shrinking across the industry. We all know that running old software is insecure, and being able to update regularly (in a controlled fashion) leads to more secure and reliable outcomes with a lower MTTR. The pace of innovation is only going to increase, and supporting older releases is expensive. Two year LTS is quickly going to become the next ten year LTS. https://lnkd.in/eQKCxwXd #golang #go #oss #programming

73

4 Comments

Hot off the press just minutes ago, Confluent just announced Freight clusters in #KafkaSummit Bangalore. They’re a new innovative cluster type that lowers prices by up to 90% (!) 🔥 What is it and how does it work? It’s all about latency. Some workloads like: 🔷telemetry 🔷metrics 🔷logs don’t care whether the latency is 50ms, 500ms or 2s. But the same workloads typically have a TON of data. And when you need to replicate that data 3x and pipe it through the network… the network costs can eat you alive 💀 So what did #CFLT do? They decided to write the data straight to S3! Instead of having Kafka replicate it across brokers, they just write it directly to S3. S3 itself does the replication under the hood (for much cheaper) 💸 The catch? S3 is much slower. But as we mentioned, those workloads don’t care. Similar to a cargo ship, you’re fine letting the cargo (data) take longer to ship as long as the price is good. For the critical use cases - you have planes. ✈️ For everything else - you have Freight. 🛳️

600

36 Comments

RLHF was thought to be the secret behind aligning ChatGPT to give 'smart' responses. Implementing RLHF using PPO optimization is a complex engg task but initially there seemed to be no way around it. 🛠️ The road from the original complex RLHF implementation to the more recent simpler model fine-tuning/alignment strategies is long and unwinding. 👉 The initial RLHF setup used 3 (or 4?) copies of the pretrained model. First is the next-token pretrained model (also called SFT model), one for external reward model and one for the new model being fine-tuned. PPO algorithm was used for learning. Led to many asking do we really need RLHF/PPO ? 👉 DPO got rid of the external reward model by reformulating the loss fn and reduced the copies to two - the original SFT model and the new model being trained. 👉 SPO finally reduced the copies to a single one. They construct a new intrinsic loss function which trains the model while learning from (good, bad) samples. The average log probability of a sequence is used as the implicit reward, same as the usual generation likelihood. Very clean formulation and writing. 🤔 In hindsight, SPO is obvious but why so long? (ORPO was close enough). Two key ideas seem to make it work: length normalization of loss and an extra margin parameter. 🤨 While both DPO and SPO claim full victory on compute and memory efficiency, the jury is still out on whether they yield “better” models than PPO (used in RLHF) setting, on say downstream reasoning tasks like GSM8k. 🎇 I really wish that implementing alignment training becomes as easy as next-token training. Hope that SPO holds its ground and whatever remains of PPO can be brought back to SPO++. --- Stay informed about latest LLM updates with our newsletter: offnote.substack.com

39

3 Comments

One of the problems one will encounter while building LLMs - Failure to get good vector search result due to document size. Try treating chunk size as a tuning parameter and it'll do wonders. Make sure to try different chunking size to see what works best. The OpenAI platform is extremely versatile and provides file retrieval. If you've used it with Assistants then you'll know the beauty of not having to manage your own vector database. In my experience, however, when you try to upload large files (large meaning >50MB, while their website mentions the maximum file size for one file is 512MB (correct me if I am wrong)) I have seen it fail to parse it in one go and even when successfully parsed, I've seen assistants not being able to retrieve information from it after some time. Now I cannot comment on why this happens since I do not have access to how OpenAI parses the files but I suspect it has to do with size and chunk size. So what do you do when you cannot control the chunk as a tuning parameter? One solution that has worked wonders for me is to use the old scatter gather. In this method you break down large document into small files and upload them. How you architect the agent is up to you but I've seen it work beautifully if you make one agent responsible for a couple of smaller documents and a master agent which gathers all the knowledge and gives you the final answer.

4

CPU and GPU - we all known them. But "TPU and DPU" are newer players in the tech world. TPU, also known as "Tensor Processing Unit" are processors used for machine learning applications that can handle large amounts of data and matrix computations. Google Kubernetes Engine (GKE) supports scheduling workloads on Cloud TPUs. DPU, also known as "Data Processing Unit" are processors created to efficiently handle data-related tasks like data transfer, encryption, and network processing. These 2 offload tasks from the CPU/GPU, to improve overall system performance and efficiency. That's my new learning, so scheduling this post for you all. Thank you for reading. Stay focused, because big wins → Take time 😊 Cheers - Mutha Nagavamsi.

257

4 Comments

C++ Tricks here : Ask Senior C++ Tricks Series Check our website: asksenior.in/learn for free video solutions to various problems. In C++, __builtin_popcount(x) returns popcount of a number — the number of ones in the binary representation of x. Use __builtin_popcountll(x) for long longs. There are also __builtin_clz and __builtin_ctz (and their long long versions) for counting the number of leading or trailing zeros in a positive number. Here are some of their applications. Follow Ask Senior for more such tricks

120

3 Comments

Databases use WAL to maintain consistency and durability of transactions, but it can become a bottleneck when we have a massive IO ⚡ This happens because every entry in WAL has a log sequence number that needs to be generated in a single thread to ensure ordering and correctness, hence this is not suitable for multi-threaded CPUs. Because LSN generation lies in the critical path of transactions, high contention for WAL affects the throughput. What if we could log multiple transactions concurrently? This is where decentralized logging comes in ... To understand this better, I read a paper titled DecLog which leverages NVMs (persistent RAM) to make concurrent logging possible. find this paper on - arpitbhayani.me/papershelf ⚡ Admissions for my System Design June 2024 cohort are open, if you are SDE-2 and above and looking to build a rock-solid intuition to design any and every system, check out arpitbhayani.me/course #DatabaseInternals #SystemDesign #AsliEngineering

220

3 Comments

🚀 Bengaluru-based Conversational AI Startup Gnani AI Raises $3M 💰 Gnani AI, India's first voice-first small language model (SLM) for vernacular languages, has secured $3M in funding from tech holding company Info Edge. The investment will fuel the startup's expansion plans. Key Highlights: • Gnani AI is building a series of voice-based SLMs, trained on millions of hours of proprietary audio datasets and billions of conversations in 12 Indian languages • The startup has over 100 customers in banking, financial services, insurance, telecom, automotive, and healthcare industries • Gnani AI's models run on the edge, reducing costs and enabling quick fine-tuning based on enterprise needs • The startup specializes in creating customized models for individual companies, adding real value to their data CEO Ganesh Gopalan emphasizes the importance of multimodal models, focusing on voice and text fusion. By taking their models to enterprises and helping them build on top of it, Gnani AI is revolutionizing the conversational AI landscape in India. #ConversationalAI #VoiceFirst #IndianLanguages #Funding #Expansion

12

1 Comment

PayPal open-sourced their in-mem KV store - JunoDB and I have been skimming the documentation and source to understand it, here are some key details ⚡ It is a distributed key-value store written in Go. It started as a single-threaded C++ project, but then they switched to Go for better concurrency support. At Paypal, it handles 350 billion requests every day and offers a staggering six 9s of system availability which means it sees a downtime of a mere 31.56 seconds every year. I spent 3 days going through the source code to understand how it works and I have compiled my learnings in a 5-video series ⚡ The series is a great primer for anyone to understand distributed databases. Dissecting JunoDB - https://lnkd.in/gTyKfXWb ⚡ I keep writing and sharing these engineering nuggets, so if you are keen on learning them, follow along. youtube.com/c/ArpitBhayani #SystemDesign #DatabaseInternals

1,049

10 Comments

Here is the recording of my REAC talk & subsequent Q&A for those interested: Optimizing the UI’s CPU bound frame rate in Call of Duty. https://lnkd.in/gkZdH6mA

26

4 Comments

Google IO 2024 There are many interesting updates, but some of them caught my attention are 1. **Context Window** - Whopping 1 million context window Gemini 1.5pro rollout and 2 million support is coming. Topping all the LLMs in this category 2. **Trillium** - A 6th generation TPU 3. **Gemini Nano** - supports 36 languages and will steal the world. It's going to break all the communication barriers and make everyone smarter and much more Google is all into AI, and the competition is getting bigger and bigger. I know some models fetched fame for their ability at LLM capacity. I believe the real winner will be someone who packages all the models and offers the Model As Service (MASS) and GenAI ecosystem. In that perspective, again, GenAI space gets consolidated, and the leading cloud providers would continue to dominate. Google Glass was introduced way before Meta and Apple Glass (Vision Pro). I think now is the time but looks like Google gave up on that for good ;) #GoogleIO #AI #LLM #TPU #MASS #GenAI #VisionPro #ContextWindow #Trillium #GoogleNano www.krkaleraj.com

18

5 Comments

RPU / WPU pricing, where costs are calculated like .0001 USD per mn ,seems scam. Many infrastructure software companies are using this "pay-as-you-go" strategy. The problem is, when you get the bill, it's hard to understand what you used or what you could optimize—it's just a random number on your credit card statement. If the bill is low, nobody cares. But if it jumps, people start looking for a more predictable pricing model. From my experience, the best approach is to offer predictable pricing on a monthly, quarterly, or yearly basis. All that complex RPU/WPU stuff should be handled internally to give customers a clear, stable cost. As a customer, I don't want to waste time figuring out how my bill was calculated. I'm interested to know what other developers think about this kind of pricing. Would you rather have simple, predictable costs, or are you okay with the detailed pay-as-you-go models? #engineering #technology #founders #aws #mongodb #azure #developer #softwareengineer #entrepreneurship #paas #gcp

8