Union.ai

Have you tried using Decks in your ML pipeline yet? The Decks feature enables you to obtain customizable and default visibility into your tasks. Think of it as a visualization tool that you can utilize within your Flyte tasks. Decks are equipped with a variety of renderers, such as FrameRenderer and MarkdownRenderer. These renderers produce HTML files. As an example, FrameRenderer transforms a DataFrame into an HTML table, and MarkdownRenderer converts Markdown text into HTML. Learn more at https://www.union.ai/

This workshop will equip you with the skills to effectively build your AI pipelines and reliable ML workflows using Python, scikit-learn and Flyte/Union. What you'll learn can be transferred to more complex AI pipelines and machine learning libraries. Links to follow along: Union Signup: https://signup.union.ai/ Workshop repo: https://lnkd.in/gnRANzzD Colab Notebook: https://lnkd.in/gZKNX2FS Flyte GitHub: https://lnkd.in/g6sdBDdv Slack Group: https://slack.flyte.org/

The vast majority of AI applications today are powered by batch workloads. From ETA estimation to sequence alignment, image processing, and recommendation systems, batch processing provides an efficient mechanism for running many similar workloads in parallel, saving resources and time while still meeting latency requirements. In this context, this post explores the specific challenges of running batch workloads on Kubernetes (“K8s”) for AI platform engineers and examines how Flyte, the open-source AI orchestrator, is designed to address these issues. Read more in our blog post: Scaling patterns for batch workloads on K8s by Haytham Abuelfutuh! 🔗 https://lnkd.in/gQnpaRBk

We're hiring! 🚀 Check out Tessa Wilder's post to learn more!

We're going live at 10:00 am PT ! Join this fireside chat as Sage Elliott speaks with Leo Parker Dirac Dirac CTO and Co-founder of Groundlight AI about computer vision and robotics!

Artifacts in Union provide a core abstraction that serves as an interface between the different teams that work together to build AI-powered products. Artifacts aren’t just static entities that you use to conveniently fetch data and model objects: they are also the basis of Union’s event-driven orchestration. They enable you to express softly-coupled execution graphs through reactive workflows. These are workflows that can subscribe to a specific artifact and are triggered based on the publication of a new artifact version. Check out this detailed blog post by Niels Bantilan that covers data-aware and Event-driven AI Orchestration with Artifacts. 🔗 https://lnkd.in/gZ3G-Fvm Getting started is easy! from flytekit import Artifact from typing import Annotated MLDataset = Artifact(name="ml-dataset") @task(cache=True, cache_version="1") def create_ml_dataset() -> Annotated[DataFrame, MLDataset]: ...

In this community presentation we’ll see how Flyte is utilized by LinkedIn engineering and Enceladus Bio! The Flyte community presentation series is where practitioners share their experiences and knowledge of building robust and scalable data and machine learning workflows with Flyte. Let us know if you’d like to present. Talk 1: Flyte at LinkedIn LinkedIn leverages Flyte as the core infrastructure for our next-generation ML Training platform. Flyte has provided numerous benefits, enabling ML engineers to iterate much faster than before. We will review the key highlights of adopting Flyte at LinkedIn and discuss features that have proven particularly valuable. Talk 2: Flyte at Enceladus Bio, a gene editing therapeutic startup Enceladus Bio is developing the next generation of gene editing medicines. Computation is deeply integrated in our R&D workflow - from the design of the wet-lab experiments to the analysis of their readouts. As a team composed of academically-trained scientists, we'll share why we chose Flyte to manage our pipelines and our experience of using it as an early stage biotech company. About Flyte: Flyte is the popular open-source orchestration platform used for building end-to-end machine learning pipelines, data processing workflows, and scientific computations. Flyte enables developers, ML engineers, and data scientists to focus on their core tasks without worrying about the underlying infrastructure. Check out the project on GitHub: https://lnkd.in/g6sdBDdv

Reduce the Runtime & Memory Requirements of your Workloads by more than 50% with Accelerated Datasets 📊 In machine learning, model training can re-use the same dataset across multiple runs with different parameters. In data engineering, a large, raw dataset might be processed in multiple ways to fulfill downstream use cases like business analytics. A common use case in bioinformatics is to align DNA sample reads to a large, static reference genome. What if we could cache these files to a semi-persistent data store on the underlying node? This is exactly what we did to help our customers save a huge amount of time, especially when working with homogenous workflows that produce wide map task fanouts. We’re calling this capability Accelerated Datasets. It allows you to pre-load static, read-only datasets into compute nodes and reuse them to reduce a major part of the overhead incurred by using ephemeral compute resources. In the blog post we detail how DelveBio, one of our customers in the medical diagnostics space, found that Accelerated Datasets quadrupled their throughput on the same nodepool. Read the full blog 🔗 https://lnkd.in/gcwG3wSN

Perian enables serverless execution of arbitrary workloads with user-specified hardware and location requirements on multiple cloud providers. Using the Perian Flyte agent, you can run Flyte tasks on multiple clouds through Perian by simply specifying the requirements, including accelerator types, in the task configuration. Join this session even if you're not a Flyte user, everyone is welcome!

We're going live at 9:00am Pacific time!