Skip to content
Qzentra

Case fileData Engineering

Enterprise Data Query Architecture

Query architectures for analytical workloads across data platforms and data lakes, using distributed and embedded engines over Azure Data Lake storage.

Enters through
02Data Engineering
System type
Distributed and federated query architecture
Also demonstrates
query engines · data federation · data lake access · analytical workflows04Custom AI Applications
Query architecture on the system section.Lake storage at the foundation, query engines on infrastructure, federation through APIs, analytical workflows at the application layer. AI and automation are not part of this system.06  Application05  Automation04  AI03  APIs02  Infrastructure01  Data1234
  1. 1lake storageData
  2. 2query enginesInfrastructure
  3. 3federationAPIs
  4. 4analyticsApplication

01 Context

The challenge

Analytical workloads needed to query and process data across modern data platforms and data lakes, including federation across sources and direct access to lake storage.

What made it non-trivial

  • Data spread across platforms and lake storage rather than one database
  • Analytical processing that has to scale with the workload
  • Different engines suit different query shapes

02 The system

What Qzentra engineered

Qzentra designed query architectures over data lake storage using distributed and embedded engines. The engines involved are Trino, DuckDB, and Apache Drill, over Azure Data Lake, with Python and SQL as the processing languages.

Which engine serves which workload is decided by the shape of the query and the data. The architecture is performance-aware by design: access patterns, not vendor defaults, decide where processing happens.

What Qzentra was responsible for

  1. Data architecture across platforms and lake storage
  2. Query-engine selection and configuration
  3. Data federation across sources
  4. Analytical processing in Python and SQL

03 The route

How the system works

Four layers, foundation first. The architecture is described at the level the documentation supports.

  1. lake storage

    Data

    Data lives in Azure Data Lake storage and in the platforms around it.

  2. query engines

    Infrastructure

    Distributed and embedded engines, Trino, DuckDB, and Apache Drill, process analytical queries over that storage.

  3. federation

    APIs

    Queries reach across sources rather than requiring everything to be copied into one store first.

  4. analytics

    Application

    Analytical workflows in Python and SQL consume the results.

04 Decisions

Engineering decisions

  1. Engine per workload shape

    Distributed and embedded engines have different strengths. The architecture uses more than one so that each workload runs on the engine suited to it.

  2. Query in place before centralizing

    Federation across sources and direct lake access avoid copying everything into one store before a question can be asked.

  3. Lake storage as the foundation

    Azure Data Lake holds the data; engines are layered over it rather than replacing it.

05 Verified outcomes

What was achieved

  • Analytical workloads query across data platforms and data lakes
  • Distributed and federated access without centralizing everything first

06 Technology and capabilities

Technology, after the architecture

Named last, with the role each played. The capabilities are what the project demonstrates beyond its primary service.

  • TrinoDistributed SQL query engine
  • DuckDBEmbedded analytical engine
  • Apache DrillQuery engine over files and stores
  • Azure Data LakeLake storage
  • Python and SQLProcessing and analytical workflows

Capabilities demonstrated

  • data architecture
  • query engines
  • data lake access
  • data federation
  • analytical workflows
  • performance-aware data engineering

07 Related work

Other case files

Systems that share a layer, a capability, or a decision with this one.

  1. 03Case fileData Engineering

    5 TB Zero-Downtime MySQL Migration

    A 24/7 production system on legacy MySQL 5.7 with roughly 5 TB of data, moved to a modern MySQL environment by replication with no service interruption at cutover.

    Verified factApproximately 5 TB, zero service interruption at cutover

    Enters through02Data Engineering

    Read the case file
    5 TB Zero-Downtime MySQL Migration: route through the six layers.06  Application05  Automation04  AI03  APIs02  Infrastructure01  Data12345
    1. 1Compatibility audit of MySQL 5.7 workloadData
    2. 2Deprecated variable remediationInfrastructure
    3. 3Replica build and synchronizationInfrastructure
    4. 4Replica promotion at cutoverInfrastructure
    5. 5Query modernization with CTEsData

08 Discuss a project

Is your analytical data spread across platforms that do not talk to each other?

Tell us where the data lives, what the queries look like, and who runs them. Query architectures are scoped from the workload, not from a product list.

Discuss a ProjectAll work