Apache

Boost query efficiency by using Apache Iceberg statistics on the AWS Glue Data Catalog | Amazon Web Services

by
Boost query efficiency by using Apache Iceberg statistics on the AWS Glue Data Catalog | Amazon Web Services

AWS Glue Data Catalog now offers column-level aggregation statistics for Apache Iceberg tables, enhancing query performance in Redshift Spectrum. Apache Iceberg is an open table format that supports ACID transactions on data lakes. Enhancements include time-travel, schema evolution, and hidden partitioning. The Data Catalog stores table metadata and supports Iceberg tables, enabling automatic compaction of … Read more

Amazon Web Services introduces support for Apache Airflow version 2.9.2 on MWAA.

by
Amazon Web Services introduces support for Apache Airflow version 2.9.2 on MWAA.

Amazon Managed Workflows for Apache Airflow, known as Amazon MWAA, is a service that enhances security, availability, and reduces the management overhead when setting up data pipelines in the cloud. The announcement of Apache Airflow version 2.9.2 on Amazon MWAA brings new features like logical operators for DAG scheduling, combining data and time-based schedules, and … Read more

AWS now supports Apache Airflow version 2.9 on Amazon MWAA

by
AWS now supports Apache Airflow version 2.9 on Amazon MWAA

Amazon Managed Workflows for Apache Airflow (MWAA) now supports the creation of Apache Airflow version 2.9 environments. This latest release of Apache Airflow brings several enhancements to the popular open source workflow management tool. Amazon MWAA is a managed orchestration service that simplifies the setup and operation of data pipelines in the cloud using Apache … Read more

Amazon Web Services now supports Apache Flink version 1.19 with their Managed Service for Apache Flink

by
Amazon Web Services now supports Apache Flink version 1.19 with their Managed Service for Apache Flink

Apache Flink is an open-source distributed processing engine with support for stateful processing and event time semantics. It offers interfaces for stream and batch processing in multiple programming languages like Java, Python, Scala, and SQL. Amazon Managed Service for Apache Flink now supports Apache Flink version 1.19.1, released by AWS community, bringing bug fixes and … Read more

AWS introduces two new APIs to query operations on Flink applications in Amazon Managed Service for Apache Flink.

by
AWS now supports Apache Airflow version 2.9 on Amazon MWAA

Amazon Managed Service for Apache Flink has introduced new APIs called ListApplicationOperations and DescribeApplicationOperation. These APIs allow users to track the operations performed on their applications, providing details such as start time, current status, success or failure, and whether a rollback was triggered. This enables users to take necessary follow-up actions based on the information … Read more

AWS now offers system-rollback support for Amazon Managed Service on Apache Flink

by
AWS now supports Apache Airflow version 2.9 on Amazon MWAA

Amazon Managed Service for Apache Flink has introduced a new system rollback feature to automatically revert your application to the previous version in case of errors during job submission. This feature helps improve application uptime by identifying errors such as insufficient permissions or incompatible save points that may occur during updates or scaling actions. By … Read more

AWS now supports Apache Flink 1.19 with their Amazon Managed Service for Apache Flink

by
AWS now supports Apache Airflow version 2.9 on Amazon MWAA

Amazon Managed Service for Apache Flink now supports Apache Flink 1.19. This release brings new features to the SQL API, such as TTL state configuration and session window support. Additionally, Python 3.11 support, trace reports for job restarts and checkpoints are included in Flink 1.19. You can easily update to the latest runtime version for … Read more

Create a live streaming generative AI tool with Amazon Bedrock, Amazon Managed Service for Apache Flink, and Amazon Kinesis Data Streams on AWS

by
Create a live streaming generative AI tool with Amazon Bedrock, Amazon Managed Service for Apache Flink, and Amazon Kinesis Data Streams on AWS

Generative artificial intelligence (AI) has seen significant growth in 2024, especially concerning large language models (LLMs) for intelligent chatbot solutions. Amazon Bedrock is a managed service that offers various foundation models (FMs) from renowned AI companies like AI21 Labs, Anthropic, Cohere, and others, allowing users to build generative AI applications securely and responsibly. This technology … Read more

Azure Synapse Runtime for Apache Spark 3.4 is now Generally Available on Microsoft Azure

by
Azure Synapse Runtime for Apache Spark 3.4 is now Generally Available on Microsoft Azure

Microsoft has announced the general availability of Azure Synapse Runtime for Apache Spark 3.4. This release comes after a successful public preview period that started in November 2023. The new runtime is now deemed ready for production workloads. One of the main updates in this latest version is the incorporation of Apache Spark 3.4 and … Read more

Utilize Amazon Managed Service for Apache Flink and Amazon Bedrock for Real-Time Social Media Insights on AWS

by
Utilize Amazon Managed Service for Apache Flink and Amazon Bedrock for Real-Time Social Media Insights on AWS

X (formerly known as Twitter) with over 550 million active users has become a tool for understanding public opinion and spotting trends. Real-time insights play a crucial role for brands to analyze tweet data effectively. Amazon Managed Service for Apache Flink allows real-time analysis of streaming data using Apache Flink with stateful computation and exactly-once … Read more