Cribl Stream PrivateLink AMI on AWS

Partner Solution Deployment Guide

February 2023
Kam Amir, Cribl
Tony Bulding, AWS Integration & Automation team

Refer to the GitHub repository to view source files, report bugs, submit feature ideas, and post feedback about this Partner Solution. To comment on the documentation, refer to Feedback.

This Partner Solution was created by Cribl in collaboration with Amazon Web Services (AWS). Partner Solutions are automated reference deployments that help people deploy popular technologies on AWS according to AWS best practices. If you’re unfamiliar with AWS Partner Solutions, refer to the AWS Partner Solution General Information Guide.

Overview

This Partner Solution deploys Cribl Stream PrivateLink AMI on the AWS Cloud. If you are unfamiliar with AWS Partner Solutions, refer to the AWS Partner Solutions General Information Guide.

Costs and licenses

This Partner Solution requires a subscription to the Amazon Machine Image (AMI) for Cribl Stream Single Instance (Free) x86_64, available on AWS Marketplace.

There are no additional licenses required to use this Partner Solution. There is no cost to use this Partner Solution, however you will be billed for the resources deployed. For more information see the AWS Partner Solutions General Information Guide.

Architecture

Deploying this Partner Solution with default parameters builds the following Cribl PrivateLink AMI environment in the AWS Cloud.

Figure 1. Partner Solution architecture for Cribl PrivateLink AMI on AWS

As shown in Figure 1, this Partner Solution sets up the following:

A highly available architecture that spans two Availability Zones in your VPC.
An Application Load Balancer to route user web traffic to Cribl Stream instances.
A PrivateLink VPC endpoint for private access to the Partner Solution VPC from other AWS accounts.
A Network Load Balancer to route traffic from the PrivateLink VPC endpoint to Cribl Stream worker nodes in the public subnets.
In the public subnets, a Cribl Stream leader node and two worker nodes in Auto Scaling groups. Each node is deployed to an Amazon Elastic Compute Cloud (Amazon EC2) instance.*
AWS Identity and Access Management (IAM) for a role and policy for the Cribl Stream instances.

* By default, to give users a better experience when getting started with Cribl Stream, this Partner Solution deploys in a public subnet. If you’re deploying this Partner Solution to a production environment, consider using a private subnet.

Deployment options

This Partner Solution provides the following deployment options:

Deploy Cribl PrivateLink AMI into a new VPC (ARM64). This option builds a new AWS environment that consists of the VPC, subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components. It then deploys Cribl PrivateLink AMI into this new VPC.
Deploy Cribl PrivateLink AMI into an existing VPC (ARM64). This option provisions Cribl PrivateLink AMI in your existing AWS infrastructure.
Deploy {partner-product-short-name into an existing VPC (x86)}. This option builds a new AWS environment that consists of the VPC, subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components. It then deploys Cribl PrivateLink AMI into this new VPC.
Deploy Cribl PrivateLink AMI into an existing VPC (x86). This option provisions Cribl PrivateLink AMI in your existing AWS infrastructure.

This Partner Solution provides separate templates for these options. It also lets you configure Classless Inter-Domain Routing (CIDR) blocks, instance types, and Cribl PrivateLink AMI settings.

Deployment steps

Sign in to your AWS account, and launch this Partner Solution, as described under Deployment options. The AWS CloudFormation console opens with a prepopulated template.
Choose the correct AWS Region, and then choose Next.
On the Create stack page, keep the default setting for the template URL, and then choose Next.

On the Specify stack details page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. When you finish reviewing and customizing the parameters, choose Next.

Unless you’re customizing the Partner Solution templates or are instructed otherwise in this guide’s Predeployment section, don’t change the default settings for the following parameters: QSS3BucketName, QSS3BucketRegion, and QSS3KeyPrefix. Changing the values of these parameters will modify code references that point to the Amazon Simple Storage Service (Amazon S3) bucket name and key prefix. For more information, refer to the AWS Partner Solutions Contributor’s Guide.

On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you finish, choose Next.
On the Review page, review and confirm the template settings. Under Capabilities, select all of the check boxes to acknowledge that the template creates AWS Identity and Access Management (IAM) resources that might require the ability to automatically expand macros.
Choose Create stack. The stack takes about 15 minutes to deploy.
Monitor the stack’s status, and when the status is CREATE_COMPLETE, the Cribl Stream PrivateLink AMI deployment is ready.
To view the created resources, choose the Outputs tab.

Postdeployment steps

Logging in to the Cribl web console

Sign in to the AWS Management Console and open the CloudFormation console.
In the left navigation pane, choose Stacks.
On the Stacks page, choose the Cribl stack.
Choose the Outputs tab. Note the following values:
- Cribl Stream web console URL
- Default web access credentials
Navigate to the Cribl Stream web console URL.
Enter the user name admin and the AMI ID of the Cribl Stream leader instance as the password. Refer to Default web access credentials on the Outputs tab for this password.

Adding other data sources and destinations

You can add data sources, such as Amazon Kinesis, Amazon Simple Queue Service (Amazon SQS), or Amazon S3, to the IAM policy that is included with the deployment. You must have the ARN and IAM policy requirements for any services that you to add. After editing the IAM policy, refer to the following links to integrate the added source or destination with Stream.

Data sources

Data destinations

Additional resources

Troubleshooting

During deployment, you must select two Availability Zones or the deployment will fail.
You can use a git repository to back up your configurations. For more information, refer to GitOps on Cribl.
For troubleshooting common Partner Solution issues visit the AWS Partner Solution General Content Guide or the Troubleshooting CloudFormation page in the AWS documentation.

FAQ

Q. I encountered a CREATE_FAILED error when I launched the Partner Solution.

A. If AWS CloudFormation fails to create the stack, relaunch the template with Rollback on failure set to Disabled. This setting is under Advanced in the AWS CloudFormation console on the Configure stack options page. With this setting, the stack’s state is retained, and the instance keeps running so that you can troubleshoot the issue. (For Windows, look at the log files in %ProgramFiles%\Amazon\EC2ConfigService and C:\cfn\log.)

When you set Rollback on failure to Disabled, you continue to incur AWS charges for this stack. Delete the stack when you finish troubleshooting.

For more information, see Troubleshooting AWS CloudFormation.

Q. I encountered a size-limitation error when I deployed the AWS CloudFormation templates.

A. Launch the Partner Solution templates from the links in this guide or from another S3 bucket. If you deploy the templates from a local copy on your computer or from a location other than an S3 bucket, you might encounter template-size limitations. For more information, see AWS CloudFormation quotas.

Q. Can I deploy this Partner Solution in a different AWS Region other than the default (us-east-1)?

A. You can enter a different Region during deployment for the Partner Solution S3 bucket Region parameter. For any Partner Solution to work in a Region other than its default Region, all the services it deploys must be supported in that Region. You can launch a Partner Solution in any Region and see if it works. If you get an error such as Unrecognized resource type, the Partner Solution is not supported in that Region. For an up-to-date list of AWS Regions and the AWS services they support, see AWS Regional Services.

Q. Can I modify the EC2 IAM policy to include other S3 buckets, Kinesis streams, or other AWS resources for my instance to access?

A. Yes. For more information, refer to S3 and SQS Permissions and Amazon Kinesis Streams.

Q. Where can I find the Cribl URL output when I choose a deployment option into a new VPC? A. Refer to the Cribl LogStream Web URL output of the CloudFormation stack after deployment. To log in, use the Stream instance AMI ID as the password.

Q. After my instance was rebuilt by the Auto Scaling group, I can no longer log in to my instance. What’s the new password? A. When the Auto Scaling group rebuilds an instance, it resets the administrator password to the new instance’s AMI ID.

Customer responsibility

After you deploy a Partner Solution, confirm that your resources and services are updated and configured—including any required patches—to meet your security and other needs. For more information, refer to the Shared Responsibility Model.

Feedback

To submit feature ideas and report bugs, use the Issues section of the GitHub repository for this Partner Solution. To submit code, refer to the Partner Solution Contributor’s Guide. To submit feedback on this deployment guide, use the following GitHub links:

Notices

This document is provided for informational purposes only. It represents current AWS product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS products or services, each of which is provided "as is" without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at https://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "as is" basis, without warranties or conditions of any kind, either expressed or implied. Refer to the License for specific language governing permissions and limitations.