Case fileData Engineering
Query architectures for analytical workloads across data platforms and data lakes, using distributed and embedded engines over Azure Data Lake storage.
Analytical workloads needed to query and process data across modern data platforms and data lakes, including federation across sources and direct access to lake storage.
What made it non-trivial
Qzentra designed query architectures over data lake storage using distributed and embedded engines. The engines involved are Trino, DuckDB, and Apache Drill, over Azure Data Lake, with Python and SQL as the processing languages.
Each engine has a distinct role. Trino runs distributed SQL across sources, so one query can federate data that lives in different places. DuckDB runs embedded analytical queries inside Python processes, close to the files, where a cluster would only add overhead. Apache Drill queries files and stores directly, without a fixed schema being defined first.
Which engine serves which workload is decided by the shape of the query and the data. The architecture is performance-aware by design: access patterns, not vendor defaults, decide where processing happens.
What Qzentra was responsible for
Four layers, foundation first. The architecture is described at the level the documentation supports.
Data lives in Azure Data Lake storage and in the platforms around it.
Distributed and embedded engines, Trino, DuckDB, and Apache Drill, process analytical queries over that storage.
Queries reach across sources rather than requiring everything to be copied into one store first.
Analytical workflows in Python and SQL consume the results.
Distributed and embedded engines have different strengths. The architecture uses more than one so that each workload runs on the engine suited to it.
Federation across sources and direct lake access avoid copying everything into one store before a question can be asked.
Azure Data Lake holds the data; engines are layered over it rather than replacing it.
Named last, with the role each played. The capabilities are what the project demonstrates beyond its primary service.
Capabilities demonstrated
Discuss a project
Tell us where the data lives, what the queries look like, and who runs them. Query architectures are scoped from the workload, not from a product list.