Mark Heynen’s Post

The last Practical AI Podcast episode really maps out the emerging problem with AI at work (and what we are solving at Knapsack) well: "It’s a different context from a personal user. Like I mentioned, I’m paying my OpenAI monthly fee and use it all the time for various tasks. However, in the enterprise context, it’s a bit different. There may be similar capabilities, but enterprises are really concerned about their data and information going out. If it’s not their own data, it’s their customers’ data. Using a public API that sends data outside their control poses a huge concern and risk, not only regarding immediate privacy but also around liability and legal issues. Most organizations have a mixture of their own and other organizations’ data, along with a bunch of partnership agreements. In large companies, even if you’re okay with your data going out, of the 50 partners you have, it might be that 36 of them aren’t comfortable with their data, which they have an agreement with you to hold, being sent to a third party. This makes it quite challenging to use third-party APIs in a manner that everyone is comfortable with." https://lnkd.in/gQsvwkCw

Shocker: It is cheaper, easier, and more private to run decentralized AI than to buy AI solutions from large cloud providers like Microsoft, OpenAI, Google, or Amazon. Of course, this is not really a shocker -- anyone who has worked in decentralized finance knows far too well that concentration of market power leads to escalating costs, over-reliance on weak controls, and nasty fine print. Microsoft recently shared that they have seen a $5B lift in ARR in Azure from their AI business. They separately shared that they have 4,500 AI customers, so lets put this at ~$1m per year per customer. They are also spending $3m per customer in capex, so are incentivized to keep these prices high. (source: https://lnkd.in/ghVSRTWs). Meanwhile, this excellent comparison of cloud AI costs by Isaac Lehman with a more baseline use case (read, not the Fortune 100) shows that while cheaper, there is still a hefty premium vs running your AI using a "roll your own" solution decentralized solution like Knapsack Enterprise. ($225K pa vs $70K pa). And there is no capex overhand forcing us to keep prices high. (source: https://lnkd.in/gEs9Msxc) Where things get really interesting is when you roll in the Knapsack desktop app that Cooper, Philip, and I have been building (now in private beta) which can actually carry a fair amount of the load and reduce enterprise cloud costs by another 10x. For those who want the full math: At a rate of 10k chats per day: 500 tok/chat / 20 tok/s = 25 s/chat 25 s/chat * 10k chats = 250,000s 250,000s / 86,400s/gpu-day = 2.89gpus in order to meet the quota of 10k chats per day. Rounding up to 3 gpus, this would cost between $12k and $20k depending on the hardware, or, if you rent from Lambda, $1,650 per month ($20k per year) + $50K in annual software licensing fees = $70K pa. If 90% of those queries can be handled on the edge, then $2K per year + $50K in licenses. Compare to Azure's cheapest option for Llama3 70b @ $18,635 per month ($225,000 per year).

Super proud of the team for their diligent execution on this.

We need to talk about  this #AI “steamroll” myth. On a number of podcasts and blogs I am hearing the notion that many AI applications will ultimately be “steamrolled" by the incumbents, be they Google, Microsoft, Apple, or even OpenAI (which has achieved incumbent status in record time). I think this is a bit of a mind virus.  I can see why these companies may want people to think that way, considering it is taking a while to roll out mainstream applications.  However, history has shown that when a new method of information retrieval is introduced (e.g., Google Search, ICQ, the news feed), it opens up the field to focused players who explore a space on the field that they simply can’t or won’t get to.  If people had that mindset in 2002, no one would have funded Yelp, Zillow, or Kayak, and in 2010 no one would have funded Snap or Instagram. Now I can see you thinking – yes, but didn’t Google dominate in search, and doesn’t Facebook dominate in social?  Yes, but what we didn’t have back then is a lot of the fundamental technology behind this shift open sourced, and widely used.  We also didn’t know the secondary effects of such a concentration of power and data.  People want to build themselves now, to lower both their privacy and SaaS costs. This is where Knapsack comes in. I think David Friedberg said it best on the latest All-In Podcast (deep linked to the relevant section):  https://lnkd.in/gErCdTs2 “What [ChatGPT] doesn't do yet is take all of our internal data and all of our internal tooling and kind of reconfigure how we use that.  So we're still paying for a whole bunch of enterprise software and SaaS licenses for things that we know we don't want to be paying for in the future, and we're now starting to figure out how can we build our own AI based software to replace a lot of the dependency we have on third party software that uses our internal data for our internal consumption.  So there's a lot of that still ahead of us.  And we're pretty progressive so that's why I know it's early innings because I see the way we're going to use it and I know that we're probably well ahead of a big enterprise companies that are more traditional and so I can see like a couple years from now all these big enterprises are going to figure the same thing out and then you're not necessarily going to need to pay for this ChatGPT stuff if there's an internal tool and an internal LLM that you can run."

Like many of you, I do a lot of meetings, and constantly have to context switch. So at Knapsack we have enabled users to get comprehensive briefings on their upcoming meetings from both private and web data -- all running locally on your Mac. No need to trust a new cloud provider, or even Apple. And no, Apple Intelligence does not offer this. Sign up for our private beta on www.knap.ai.

I have been monitoring #Apple rumors closely over the last few weeks.  My predictions for #WWDC announcements, specifically regarding “Apple Intelligence”: - The Siri revamp will mix on-device AI for private / personal queries with online AI with ChatGPT for general queries.  There will be some weirdness with things that fall in the middle, like enterprise data.  Apple will release some APIs that will allow iOS apps to use the on device AI and contribute data to Siri.   Siri may even have a text interface (not just voice). - OS 18 will also have  features across a lot of their apps like summarizing emails, recapping articles, and photo editing.  These are table stakes these days, and will require iCloud accounts / subscriptions to Apple services like Apple News. - MacOS will have more on-device AI capabilities via Siri due to the availability of GPUs on modern hardware.  However, there will be a big split, on both iOS and MacOS, in terms of what is available on new vs old hardware. - There will be a token VisionPro using AI demo to reinvigorate interest.  People are just not using these devices as much as they expected. - There will be no change to Spotlight, and very little in the way of competing head on with Gemini on queries across all of your files and emails. Let me know what you think. We here at Knapsack 🎒 will be watching closely.

The reaction to the Microsoft Copilot Plus PC from expected and unexpected corners shows how attuned people are these days to privacy and second order effects. There is deep seated concern about using a local or cloud AI that is tied to your cloud provider and may be tied to an ad-based model. It also seems people are skeptical of the screenshot recording approach. Casey Newton on Hard Fork yesterday: "I do continue to have trust issues with Microsoft. This is a company that just last month started testing ads in the Windows 11 start menu so ... the more AI is on my computer the less I'm excited about a company that is looking to shove ads into different parts of the interface" (source: https://lnkd.in/gb5TX3Qv) Eva Galperin of the EFF: "If you start logging very detailed and revealing data about what a user is doing at all times, bad actors will attempt to get their hands on it." (source: https://lnkd.in/gaf-QNn7) Kris Shrishak: "People might avoid visiting certain websites and accessing documents, especially confidential documents, when Microsoft is taking screenshots every few seconds." (source: https://lnkd.in/gxmXiAA7) Kevin Beaumont: "Thanks, Microsoft, for your service to enabling malicious hackers.” (source: https://lnkd.in/ggy3Yfw2) There is a better way -- check out Knapsack.

Sometimes people ask me whether large AI incumbents rolling out corporate file sync will be a challenge for Knapsack. Quite the opposite -- it just highlights how infosec concerns and the need for a private solution. Some comments from this post: https://lnkd.in/gQCtVw86