That's Fresh! Newsletter

FROM THE AI WORLD

I recently discovered Dagster at a conference and was genuinely impressed by its potential. Dagster is an open-source data orchestration platform, allowing data engineers to develop, test, and deploy data pipelines at scale. It can be seen as an alternative to Airflow and wants to enable developers to write data pipelines using Python.

Data pipelines are defined as Directed Acyclic Graphs (DAGs), where each graph is composed of several ops. Ops are units of computations, and they can be written and tested locally using Python functions. One of the most important ideas behind Dagster is that developers can work on pipelines by starting small and gradually scaling up. It also includes a web interface, Dagit, to visualise graphs and ops better.

What interested me and will probably push me to play with Dagster is that the learning curve doesn’t seem steep, especially when thinking of using the tool for small scale projects!

Exploring Dagster platform

This technology is an open-source data orchestration platform for the development, production, and observation of data assets. Do you want to dig deeper?

CLEARBOX AI

Use case: Synthetic Data sandbox

In this use case we show how our technology is helpful to share and move your data inside and outside your organisation, while complying with privacy regulations.

BLOGPOST

Quantifying privacy risks

In the second part of 'Synthetic Data for privacy preservation' series, our Andrea provides a tutorial on how to quantify and prevent re-identification risk.

WEEKLY MEME

Your pals may also like...