Amazon Aurora PostgreSQL now allows direct querying of Apache Iceberg and Parquet data lakes stored in Amazon S3, eliminating extract, transform, and load (ETL) pipelines. The capability embeds DuckDB technology directly into Aurora instances, enabling unified queries across live operational data and historical archives using standard PostgreSQL syntax.
How Embedded DuckDB Eliminates S3 Data Duplication
Traditional database architectures force a heavy engineering tax when combining real-time records with cold storage. Building reverse ETL pipelines to sync Amazon S3 data lakes back into relational instances invariably duplicates infrastructure costs, inflates storage footprints, and introduces synchronization lag. As modern software stacks increasingly rely on autonomous agents that require broad access to both live transactions and deep historical context, pre-replicating every conceivable dataset becomes entirely impractical.
The technical breakthrough announced today stems directly from Amazon’s integration of DuckDB, following the DuckLabs team joining AWS. DuckDB now operates natively inside the Aurora PostgreSQL engine. Query processing happens entirely within the database node, meaning zero additional network hops and zero staging tables required to parse open-source file formats. Operational writes—including uncommitted rows—can now be joined inline against analytical data lakes.
Engineers running Aurora PostgreSQL versions 17 (starting with 17.11) and 18 (starting with 18.6) can initialize this functionality immediately. Administrators must attach an IAM role equipped with the AuroraAnalytics feature to grant the cluster access to Amazon S3 and the AWS Glue Data Catalog. Enabling the feature requires a straightforward command:
CREATE EXTENSION aurora_analytics;Once the extension is active, defining access to external datasets requires setting up a foreign table. Aurora automatically infers schemas directly from Parquet metadata, bypassing manual column definition:
CREATE FOREIGN TABLE transaction_history ()
SERVER aurora_analytics_server
OPTIONS (
location 's3://<my-bucket>/finance/transaction_history.parquet',
format 'parquet'
);For large-scale environments, administrators can bypass individual table definitions entirely. Executing a single IMPORT FOREIGN SCHEMA command bulk-creates foreign tables for every Iceberg or Parquet table housed within an AWS Glue Data Catalog database.
Federated Catalogs and Query Optimization Mechanics
Enterprise data rarely lives in a single bucket or catalog. To solve multi-source fragmentation, Aurora PostgreSQL supports external Iceberg REST Catalog (IRC)-compatible catalogs through AWS Glue Data Catalog federation. By registering an external catalog once inside Glue, applications gain a unified view across disparate analytics systems without moving data or discarding legacy storage investments.
Under the hood, query performance is preserved through advanced optimization routines. Aurora applies column pruning and predicate pushdown directly to the analytical scans. This ensures that only relevant data is read, keeping operational workloads responsive as underlying storage grows. Furthermore, frequently accessed lake data is cached locally within the Aurora instance, accelerating subsequent execution loops.
Engineers can audit execution efficiency down to the query level. Invoking aurora_analytics_stat_statements() returns granular metrics detailing rows scanned, bytes read from Amazon S3, and cache hits.
When query patterns demand single-digit-millisecond latency for hot analytical subsets, developers can materialize data lake contents directly into native relational tables using standard commands like CREATE TABLE AS SELECT or MERGE INTO. Because read queries execute across any instance in the cluster—including read replicas—heavy analytical scans can be completely isolated from primary writer nodes.
Production Execution and Cost Structures
Combining recent operational tables with deep historical archives now requires nothing more than standard PostgreSQL syntax. By leveraging a UNION ALL clause, a single query seamlessly merges live transaction tables residing inside Aurora with five-year-old Parquet records sitting in S3:
SELECT merchant, category, amount, transaction_date, 'recent' AS source
FROM recent_transactions
WHERE customer_id = 'C-1001'
UNION ALL
SELECT merchant, category, amount, transaction_date, 'historical' AS source
FROM transaction_history
WHERE customer_id = 'C-1001'
AND transaction_date >= CURRENT_DATE - INTERVAL '5 years'
ORDER BY transaction_date DESC
LIMIT 15;In this execution flow, DuckDB handles the heavy analytical scan of the Parquet files behind the scenes, while Aurora processes the operational data. The resulting dataset stitches both domains together.
This direct-query architecture is available in all commercial AWS Regions and AWS GovCloud (US) Regions at no additional charge. Organizations pay for the incremental Aurora compute the queries consume and Amazon S3 request costs for reading data lake files.
