Airflow Asset originated from data lineage and evolved into its current state, being used as a scheduling concept (data-aware, event-based scheduling). It has even more potential. This talk discusses how other parts of Airflow, namely Connection and Object Storage, contain concepts related to Asset, and we can tie them all together to make task authoring flow even more naturally.
Planned topics:
- Brief history on Asset and related constructs.
- Current state of Asset concepts.
- Inlets, anyone?
- Finding inspiration from Pydantic et al.
- My next step for Asset.
Tzu-ping Chung
Software Engineer and Architect at Astronomer