Robin Moffatt

Have you tried running the DBRX models from Databricks? In our latest blog post we provide a tutorial to get you started with the model on Ori’s public cloud. We were very impressed with DBRX’s performance on the Hugging Face Open LLM Leaderboard before the updates last week. While DBRX Instruct did not score as highly under the new scoring system, it's still one of the best open models for Language Understanding (MMLU), Programming (HumanEval), and Math (GSM8K). We also used BeFOri, the benchmarking framework from Ori to determine DBRX Base performs inference most efficiently between 40 and 256 concurrent requests on one of Ori's 8 x H100 PCIe bare metal servers. Check out the blog to learn how you can use BeFOri to optimise self-hosted LLM performance on any infrastructure! https://lnkd.in/gi55WEks

43

We have often seen many Customers asking one of the very interesting question. 🔥 We have deleted a large amount of data from some of the tables, but we still do not see the space reclaimed to disk. Why ? 🔥 🎇 Some of the database engines may also see performance degradation (no improvement) eventually, as the live tuples/records are scattered across more number of data blocks than actually required. This may be because the data that got deleted was scattered across a lot of data blocks but not really stored sequentially within a few data blocks. Queries on such tables may involve scanning more data blocks for a very few records. We call this as Table fragmentation. 🐘 This concern may not be limited to #postgres, the same is also applicable for #oracle or #sqlserver or any other database engine. 👨‍💼 👩‍💼 Database and Application Architects generally implement techniques like Partitioning and Sharding wherever applicable so that the challenges of data deletion/archival are completely avoided with a simple partition drop instead of a time and resource intensive "delete" statement. We will extend a little more on database scalability sometime later. For now, let us see what can we do to reclaim this space, if you are using PostgreSQL. 💰 Considering the resources and expertise available for #Oracle and #SQLServer like database engines, users might have already been aware of solutions to rebuild tables online in Oracle or SQL Server. 💸 Do note that such options are available only with Enterprise versions for most of the commercial database engines. For users migrating from #oracle to #PostgreSQL or newly deploying #postgres as the database engine, you could use one of the popular OpenSource extension called : pg_repack, that can safely rebuild tables online and reclaim the space back to disk. Manisankar K from HexaCluster has published this article on pg_repack and also compared with solutions available for the Oracle Enterprise versions. If you have a similar challenge where you could not avoid a massive delete but still want to rebuild the table online to reclaim space, read this article to safely implement pg_repack for PostgreSQL. Contact HexaCluster for expert assistance for your PostgreSQL databases and for database migrations to PostgreSQL. #postgres #postgresql #oracle #sqlserver #hexacluster #databasemigration #databasemaintenance

8

New blog post: Access 150k+ Datasets from Hugging Face with DuckDB This blog post, co-authored by the Hugging Face and DuckDB teams, describes how you can use the hf:// prefix in DuckDB to access datasets in Hugging Face repositories. Read more at https://lnkd.in/dYf2PPKk

319

1 Comment

I wrote a new post on the Sympathetic Ink blog to sum up The Deconstructed Database and what makes it composable. Learn more about the role of #Parquet, #Arrow, #DataFusion, Apache Iceberg, #Calcite and OpenLineage

64

4 Comments

Have you ever wondered about the differences between Datafold's Data Diff and #dbt tests? Do data teams really need both? In this post, Elliot G. and Leo Folsom explain: ❓ Why dbt tests prevent some data quality issues, but not all 🤔 How the two tests answer fundamentally different data quality questions 🤝 And why having both in your CI pipeline is essential for complete data quality coverage https://lnkd.in/ggmC97hn

13

1 Comment

🎉 29 Days of Airflow 2.9: Day 23 🎉 📈 Today's feature update in #ApacheAirflow 2.9 focuses on enhancing data visibility: adding datasets to the DAG graph. This improvement, delivered via pull request #37604, integrates datasets directly into the visual #DAG graph in the #Airflow UI. This allows users to see how datasets are connected to various tasks, providing a clearer understanding of how data flows through and impacts workflows. 🔍 Why is this significant? Incorporating datasets into the DAG graph helps users visualize and comprehend the dependencies and relationships between tasks and data more intuitively. This addition is crucial for troubleshooting and optimizing data pipelines, ensuring a more efficient and effective data handling process. 🔗 Learn more about the benefits of visualizing datasets in your DAG graphs: https://bit.ly/458xvqh #Airflow29 #DataVisualization #29daysofairflow29 #DataEngineering

19

When managing database generated events, what’s the difference between using Change Data Capture and Ably LiveSync database connector? In our latest blog post, Ably Engineer Zak Knill explains how the two mechanisms work and why we made some intentional choices about how and why the LiveSync database connector should be different. Read it here 👇

4

Sometimes useful features collide. My latest post covers a couple little "gotchas" you might run into when using Apache NiFi and Progress MarkLogic together. https://lnkd.in/eMjwhxkT #nifi #marklogic #progress #bigdata

6

1 Comment

Azure Cosmos DB design patterns – Part 2: Data Binning This post explores the data binning technique, which aggregates data into specific time intervals or 'bins' for streamlined analysis. https://lnkd.in/ebnWi8WQ

1

YugabyteDB’s automatic sharding overcomes PostgreSQL’s partition key limitations by allowing date range queries without modifying table partitioning.💡 Check out the new blog from Yugabyte expert Franck Pachot for more information, and step-by-step instructions!⬇️ https://hubs.la/Q02s_wKx0 #Yugabytedb #postgresql #sharding #partitionkey #cloudnative #partitioning

14

pgroll - zero-downtime schema changes (keeps 2 version of the schema accessible at the same time), reversible, schema migrations (using JSON format) for Postgres (CLI) https://buff.ly/3VWoEol

16

In todays #XataLaunchWeek demo, Andrew shows us to how add a NOT NULL constraint to an existing column with a multi-version schema migration, accessing both the old and new schema simultaneously until the migration is complete ✨ Watch now: https://lnkd.in/ekcuabS2

2

New blog post by Alex Monahan: Benchmarking Ourselves over Time at DuckDB The DuckDB team's philosophy is to ensure correctness first, then iterate and optimize to improve performance. This blog explores how this happened over the last three years when DuckDB became approximately 3-25x faster and 10x more scalable. Read more at https://lnkd.in/dnQ7Nya8

197

16 Comments

Want to see how easy it is to load data into your data warehouse with Daasity? In this video (https://lnkd.in/gG3aT8JF), I guide you through adding an integration and managing the frequency and subsequent transformation code. Why did we develop Daasity this way? 💰Fair Pricing: Unlike other extraction platforms that charge based on data volume, we believe you should be able to consume as much data as you want. With Daasity, your costs are based on our Gross Merchandise Value and are predictable and manageable. ⚙ Simple Setup: You shouldn’t need a Data Engineer or Developer to configure your data platform. Instead, we believe in empowering your team to focus on insights, not infrastructure. 📊Real-Time Insights: We want you to have the ability to monitor key metrics like sales, conversion, etc., in almost real-time, without any additional costs. That’s why we allow you to customize workflows by choosing the extractors that best suit your needs. And, if managing this seems overwhelming, you can always leverage our Data Agency services to do it for you. This is the power of Daasity - making data management effortless for you. #dtc #analytics #shopify

10

1 Comment

Exponam.Connect now fully supports the latest Delta Sharing changes implemented with Deletion Vectors earlier this month. https://lnkd.in/eC5krg_Y Databricks Delta Lake #deltasharing #D2O #ExponamConnect

1

Databricks users - ready to supercharge your spatial analysis? ⚡ Catch up on our webinar where you'll learn how to use the combined power of Databricks, CARTO & #AI to unlock new possibilities in spatial analytics👇 https://hubs.ly/Q02C4FzB0 #dataaisummit

16