AWS Glue 6.0 Now Available with 30% Lower Pricing and Apache Iceberg v3 Support

The updated serverless service introduces a 30 percent price reduction alongside comprehensive Apache Iceberg v3 support, aiming to streamline data engineering pipelines and slash infrastructure overhead for enterprise analytics teams.

Modernizing the Spark Runtime and Cutting Costs

Cost pressures in cloud-native data architectures have forced a hard look at backend runtimes. AWS Glue 6.0 addresses this head-on by dropping base pricing by 30 percent compared to legacy versions.

Performance gains are not just architectural theory. The release integrates Arrow-native execution for Python User-Defined Functions (UDFs) and User-Defined Table Functions (UDTFs). By leveraging Apache Arrow, the JVM and Python processes bypass expensive serialization bottlenecks. PySpark transformations that historically choked on inter-process data marshalling now execute with significantly tighter memory footprints.

For pipeline authors, Spark Declarative Pipelines automate the heavy lifting of execution planning. Engineers specify the desired end state of the data transformation rather than manually chaining orchestration steps. The engine handles execution order and optimization natively, reducing boilerplate code.

The Iceberg v3 Spec and the VARIANT Data Type

Handling messy, semi-structured JSON, log files, and event streams has traditionally required brute-force schema flattening or custom parsing logic.

The standout feature is the native VARIANT data type with integrated shredding support. Instead of forcing semi-structured payloads into rigid string columns, the format shreds data into efficient columnar structures. This architecture yields faster query read performance and eliminates the pipeline breakages that typically happen when upstream schemas mutate.

Beyond the VARIANT type, the Iceberg v3 rollout includes several specialized data types and precision controls:

• Geometry and Geography: Native spatial processing capabilities designed for geographic information systems, location analytics, and geospatial pipelines running directly on managed Spark.
• Nanosecond-Precision Timestamps: High-resolution tracking tailored for high-frequency financial ledgers, IoT sensor feeds, and scientific computing workloads requiring accuracy well beyond standard milliseconds.
• Unknown Type Handling: Automated resilience mechanisms that process unexpected schema evolutions without triggering fatal pipeline failures.

Real-Time Streaming and Low-Latency Event Processing

Batch processing no longer cuts it for modern telemetry and live telemetry streams. AWS Glue 6.0 introduces a dedicated real-time streaming mode built on Spark 4.1’s Real-Time Mode architecture, combined with Glue-optimized execution paths.

Photo: releasebytes.com

This mode achieves single-digit millisecond latency for stateless streaming pipelines. Data engineering teams can now route and transform time-sensitive events at scale without spinning up dedicated, always-on streaming clusters outside their serverless ETL workflows.

Migration and Deployment Mechanics

Adopting the new runtime requires zero API alterations.

Photo: aws-news.com

Interactive notebook users can initialize a session by invoking 6.0 in the %glue_version magic command.