Sona Maniyan

Thrilled to share a fantastic new blog post by Harshal Pagar "Running Databricks Notebook with Airflow." 📖✨ In this detailed guide, Harshal walks us through the process of integrating Databricks notebooks with Apache Airflow, making it easier to orchestrate and manage your data workflows. Whether you're new to Airflow or an experienced user, this post is packed with valuable insights and practical tips. A huge shoutout to Harshal for contributing this amazing resource to our community! 👏👏 #ApacheAirflow #Databricks #DataEngineering #AirflowCommunity #DataOrchestration #TechBlog #DataScience

90

1 Comment

Join PyTorch at Snowflake's #DataCloudDevDay on June 6 in San Francisco to learn how to accelerate GenAI performance with Native PyTorch. Register: https://hubs.la/Q02xQStt0

18

Check out the latest blog from our fellow Data Scientist and Data Surgeon Drew Lehe sharing his thoughts on Graph Data Science (GDS). Graph Data Science is a cutting-edge field helping solve many complex data problems. In this particular blog, Drew demonstrates how GDS helped improve the results of an entity resolution issue for an existing Data Surge client. Click the link below to learn more. At Data Surge innovation is a key piece of what we deliver. That innovation includes AI/ML, MLOps, and GenAI. We apply these innovations to support Network Analytics, Entity Resolution as well as Data Governance and Responsible AI. To learn more about Data Surge's services or book a demo of our solution accelerators, visit our website at www.datasurge.com. #AI #ML #GraphDataScience #GenAI #Data #Modernize #Democratize #Transform

11

Builders Taylor Wagner and Likitha Lokesh have documented four must-know tips for validating data with #Pytest & #PySpark. Check out their insights to boost your data quality game:

14

Senior Consultant and machine learning expert, Kenny Shaevel, breaks down 3 ways Databricks is helping clients implement ML best practices through MLflow. - Practical use case identification: AutoML helps to quickly evaluate the data you already have available to see how well it would work for a machine learning app. - Repeatability of complex models: Feature Store allows you to “register” features you’ve engineered to use in multiple models as needed. - Governance: Model Registry provides governance and traceability to ensure everyone knows how the models were trained and meant to be used Watch Kenny’s 2-min video for the details! #Databricks #MachineLearning #MLFlow #DataScience #AutoML #FeatureEngineering #ModelRegistry #ExpertInsights

28

1 Comment

Want to learn more about how to get started using Snowflake Cortex LLM Functions? Check out this post from Jon Coulter detailing how to use these tools to quickly classify data, empowering Data Engineers with Data Science tools! https://lnkd.in/gMTFT5xQ

14

Now on Databricks Runtime 14.3: the State Reader API, a new capability that allows users to access and analyze Structured Streaming's internal state data. Discover how this new capability is providing much-needed transparency, accessibility, and ease of use: https://dbricks.co/3UeoF6l

215

"The move to Starburst and Iceberg has resulted in a 12X reduction in compute costs versus our previous data warehouse. This efficiency allows us to focus our attention on using analytics for revenue-generating opportunities." — Peter Daniel Lim, Sr. Data Engineer at Yello.co Check out this new case study to learn how Yello adopted Starburst and Apache Iceberg to achieve 98% faster reporting times and reduced their run time from 28 hours to just 4 hours: https://okt.to/wYLAKG #casestudy #CustomerReview #ApacheIceberg

30

1 Comment

AI/BI = Traditional BI + AI Agent powered Chat that continuously learns semantics based on usage patterns. Ali Ghodsi, CEO or Databricks introduces a great way for traditional BI to transition to take advantage of AI while minimizing risk.

2

🎬🔥GenAI in the Enterprise episode 28: https://lnkd.in/grzWT86a Today's episode features Rajiv Shah, a #MachineLearning Engineer and Strategist at Snowflake. Raj built his career talking to various companies and their data teams, showing them how #GenAI (and which tools) could help them reach their goals. Now at Snowflake, he gets to share Snowflake's suite of #GenerativeAI tools, teaching clients how to use them to take their business to the next level. Zach Gardner and Raj dive even deeper into Snowflake, how it works, and why companies may want to take advantage.

6

Have you tried running the DBRX models from Databricks? In our latest blog post we provide a tutorial to get you started with the model on Ori’s public cloud. We were very impressed with DBRX’s performance on the Hugging Face Open LLM Leaderboard before the updates last week. While DBRX Instruct did not score as highly under the new scoring system, it's still one of the best open models for Language Understanding (MMLU), Programming (HumanEval), and Math (GSM8K). We also used BeFOri, the benchmarking framework from Ori to determine DBRX Base performs inference most efficiently between 40 and 256 concurrent requests on one of Ori's 8 x H100 PCIe bare metal servers. Check out the blog to learn how you can use BeFOri to optimise self-hosted LLM performance on any infrastructure! https://lnkd.in/gi55WEks

43

Using Qdrant with Apache Airflow can revolutionize how you automate and manage complex workflows for semantic querying. It makes handling large-scale vector data much simpler, and Airflow's monitoring features ensure everything runs smoothly. In this tutorial, you'll learn to use the Qdrant Airflow provider to write a DAG that generates embeddings in parallel and performs semantic retrieval based on user input. Explore the guide here: https://lnkd.in/eMpqD2tN P.S. How are you using Airflow in your data workflows? Comment below! 🚀

49

3 Comments

💬 See what Wyze's Director of Data Engineering, Wei Zhou has to say about building a warehouse native workflow with RudderStack. Read the case study to find out how Wyze used RudderStack to enable their data team to focus on AL/ML initiatives. 👉 https://bit.ly/3JFgfzj

11

We're just days away from #DataCloudSummit! ❄ If you're attending, be sure to stop by the phData booth (1315) for answers to your toughest data & AI questions including: 1. How to deploy AI application on Snowflake 2. How to realistically achieve your AI goals 3. How to accelerate your data & AI journey through automation 4. How to migrate to Snowflake 5. Much, much more! See you in San Francisco!

29

Clean, high-quality data is essential to making accurate and well-informed decisions. In this article for NextTech Today, Virtualitics' co-head of AI Aakash Indurkhya explores the importance of #dataquality and provides practical strategies to enhance it, including: ◾ Using data in daily business practices ◾ Implementing good data governance ◾ Establishing a culture of data excellence https://lnkd.in/gPF89eXW

15

Top insights from #theCUBE’s real-time coverage at Snowflake’s #DataCloudSummit ❄️ https://lnkd.in/gcFpgcJm Under new CEO Sridhar Ramaswamy, Snowflake is shifting toward an AI-driven strategy, enhancing support for Apache Iceberg, and simplifying AI applications with Cortex. The event highlighted how Snowflake is evolving to meet customer demands for data simplicity and business value. “We use this term, the sixth data platform, so [there were] mainframes, then you had Oracle, you had enterprise data warehouse, you had Hadoop, and then Snowflake separated compute from storage, they put everything in the cloud, made it infinitely scalable and everything was integrated,” said theCUBE Research Chief Analyst David Vellante in his on-site report. “But not any compute can access any data, and we’re seeing a demand for that. Of course, the value proposition of Snowflake is ‘iPhone,’ all totally integrated. This is the nature of the disruption that we’re witnessing today.” These discussions underscore the growing importance of Snowflake's role in data management across various industries. 🌐 From media to retail, Snowflake is revolutionizing how enterprises leverage data for actionable insights. 📺Catch all the insights and watch the interviews now! #DataCloudSummit #theCUBE #AI #TechNews #CloudComputing #DataSimplicity

5

🛑 Data lakehouse promises the best of both data lakes and warehouses. Yet, many still transfer data to traditional warehouses, complicating processes and increasing costs. Sida Shen's latest article explores how optimizing lakehouse architectures can eliminate these inefficiencies, enhance data integrity, and enable real-time analytics directly on the lakehouse. https://lnkd.in/gsCwzsGS #DataAnalytics #DataEngineering #DataLakeAnalytics #DataLake #DataLakehouse

5

🌟 Unfreeze Your Iceberg Lakehouse with SingleStore 🔹 The leading lakehouse vendors have adopted Apache Iceberg as the de facto standard. This breakthrough allows you to write data with one platform and read it seamlessly with another. 🔹 The adoption of Apache Iceberg marks the most significant advancement in data management since the separation of storage and compute. 👩🏽‍💻 Are you developing intelligent applications that demand modern capabilities like sub-second analytics, real-time ingestion, vector search, full-text search, and more? 🎉 With SingleStore’s integration with Apache Iceberg, you can achieve these capabilities without the need for ETL tools or costly queries on the source system! Join Us on June 26 We’ll be unveiling new features in SingleStore that include: - Unfreeze Your Data Lakehouse: Experience SingleStore’s zero-ETL, low-latency operational engine for Iceberg. - Enhanced Search Capabilities: Build intelligent applications with faster vector search and improved full-text search. - Simplified Scaling: Scale your applications, not the complexity, with Autoscaling. - Flexible Deployment: Deploy SingleStore Helios in your own VPC. Mark your calendars for this virtual event and get ahead in building fast, intelligent applications. 👉 Register Now: https://lnkd.in/g8__GWgu 📅 When: June 26, 2024 | 10:00am - 11:30am PDT 📅 June 27, 2024 | 11:30am - 1:00pm IST

12

1 Comment

“Aviso was the only company that could push the data into Aviso from EDW and apply the AI ML out the box, and then return the data back into our EDW.” - New Relic Learn how the pioneer in usage-based pricing integrates Snowflake and leverages #AI for accurate forecasts. https://lnkd.in/gSfarxzc #UBP #consumption #AI #usage #revenueintelligence #revops

10