Enterprise customers are adopting Amazon OpenSearch Ingestion (OSI) to bring data to Amazon OpenSearch for various use cases including log analysis, streaming, security analysis, and searching data. OSI integrates with AWS services like DynamoDB, S3, MSK, and DocumentDB. Support for data ingestion from self-managed accounts, OpenSearch/Elasticsearch, and Apache Kafka Clusters in the AWS environment is now available. A step-by-step guide is provided for users to get started with these sources. Prerequisites include network connectivity, name resolution, certificate verification, accessing AWS Secrets Manager, and IAM role for pipelines. Creating a pipeline with self-managed Kafka or OpenSearch is outlined. Users must configure pipelines, validate them, set network settings, select VPS connection options, specify tags, review, and create the pipelines. Considerations for self-managed OpenSearch data sources are also discussed, emphasizing verifiable certificates, bandwidth considerations, one-time data migration, and faster migration times using OSI over remote reindexing. The post introduces self-managed sources for OpenSearch ingestion and provides insights into the capabilities of OSI, with contributions from search specialist Muthu Pitchaimani and Product Manager Arjun Nambiar in the Austin, Texas, and Seattle, Washington locations respectively.
Article Source
https://aws.amazon.com/blogs/big-data/introducing-self-managed-data-sources-for-amazon-opensearch-ingestion/
