Apache RocketMQ on AWS

Partner Solution Deployment Guide

QS

August 2023
George Gu, Fan Su, Chen Chuan, Apache
Troy Ameigh, Mingwei Leong, AWS

Refer to the GitHub repository to view source files, report bugs, submit feature ideas, and post feedback about this Partner Solution. To comment on the documentation, refer to Feedback.

This Partner Solution was created by Apache in collaboration with Amazon Web Services (AWS). Partner Solutions are automated reference deployments that help people deploy popular technologies on AWS according to AWS best practices. If you’re unfamiliar with AWS Partner Solutions, refer to the AWS Partner Solution General Information Guide.

Overview

This Partner Solution sets up an Amazon Web Services (AWS) architecture that deploys an Apache RocketMQ cluster on AWS. Apache RocketMQ is a unified messaging engine and lightweight data processing platform. This Partner Solution creates and configures Amazon Elastic Compute Cloud (Amazon EC2) instances to host Apache name server and broker nodes. You can configure basic cluster settings such as EC2 instance types during deployment.

Apache RocketMQ is a unified messaging engine and lightweight data processing platform. The Apache RocketMQ Quick Start helps you deploy a configurable and highly available Apache RocketMQ cluster in the AWS Cloud.

Costs and licenses

There is no cost to use this Partner Solution, but you will be billed for any AWS services or resources that this Partner Solution deploys. For more information, refer to the AWS Partner Solution General Information Guide.

Architecture

Deploying this Partner Solution with default parameters builds the following RocketMQ environment in the AWS Cloud.

Architecture
Figure 1. Partner Solution architecture for RocketMQ on AWS

As shown in Figure 1, the Quick Start sets up the following:

  • A highly available architecture that spans up to three Availability Zones.*

  • A VPC configured with public and private subnets, according to AWS best practices, to provide you with your own virtual network on AWS.*

  • In the public subnets:

    • Managed network address translation (NAT) gateways to allow outbound internet access for resources in the private subnets.*

    • Linux bastion hosts in an Auto Scaling group to allow inbound Secure Shell (SSH) access to EC2 instances in public and private subnets.*

  • In the private subnets:

    • Amazon Elastic Compute Cloud (Amazon EC2) instances for Apache RocketMQ NameServers. You can deploy 1–3 NameServer nodes. The default is three (shown).

    • Apache RocketMQ web console running on each Apache NameServer instance.

    • EC2 instances for Apache RocketMQ Broker servers. You can deploy either one or three Broker server nodes. The default is three (shown).

* The template that deploys this Partner Solution into an existing VPC skips the components marked by asterisks and prompts you for your existing VPC configuration.

Deployment options

This Partner Solution provides the following deployment options:

  • Deploy RocketMQ into a new VPC. This option builds a new AWS environment that consists of the VPC, subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components. It then deploys RocketMQ into this new VPC.

  • Deploy RocketMQ into an existing VPC. This option provisions RocketMQ in your existing AWS infrastructure.

This Partner Solution provides separate templates for these options. It also lets you configure Classless Inter-Domain Routing (CIDR) blocks, instance types, and RocketMQ settings.

Deployment steps

  1. Sign in to your AWS account, and launch this Partner Solution, as described under Deployment options. The AWS CloudFormation console opens with a prepopulated template.

  2. Choose the correct AWS Region, and then choose Next.

  3. On the Create stack page, keep the default setting for the template URL, and then choose Next.

  4. On the Specify stack details page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. When you finish reviewing and customizing the parameters, choose Next.

    Unless you’re customizing the Partner Solution templates or are instructed otherwise in this guide’s Predeployment section, don’t change the default settings for the following parameters: QSS3BucketName, QSS3BucketRegion, and QSS3KeyPrefix. Changing the values of these parameters will modify code references that point to the Amazon Simple Storage Service (Amazon S3) bucket name and key prefix. For more information, refer to the AWS Partner Solutions Contributor’s Guide.

  5. On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you finish, choose Next.

  6. On the Review page, review and confirm the template settings. Under Capabilities, select all of the check boxes to acknowledge that the template creates AWS Identity and Access Management (IAM) resources that might require the ability to automatically expand macros.

  7. Choose Create stack. The stack takes about 15-20 minutes to deploy.

  8. Monitor the stack’s status, and when the status is CREATE_COMPLETE, the Apache RocketMQ deployment is ready.

  9. To view the created resources, choose the Outputs tab.

Postdeployment steps

Logging into an Apache RocketMQ NameServer or Broker server node

  1. Add a PEM key pair. For example:

ssh-add -k /Users/username/tmp/rocketmqXXX.pem

  1. Sign into the bastion host.

ssh -A  ec2-user@ecXXXXXXXX.cn-north-1.compute.amazonaws.com.cn

  1. Get the IP address from the Amazon EC2 console.

    1. Open the Amazon EC2 console.

    2. On the Instances screen, select the checkbox for the instance. Note the Private IPs in the Description tab, as shown in Figure 2.

NameServerAddress
Figure 2. EC2 instance private IP address

  1. From the bastion host, log into the node with the IP address. For example:

    (ec2-user@ip-10-0-136-135) $ ssh 10.0.6.235

Last login: Wed Jan  6 04:55:01 2021 from ip-10-0-136-135.cn-north-1.compute.internal

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|
https://aws.amazon.com/amazon-linux-2/

[ec2-user@ip-10-0-6-235 ~]$

  2. Once you are signed into the node, you can use Apache RocketMQ command-line tools to query, manage, and diagnose issues.

Viewing the Apache RocketMQ web console in a browser

The default deployment installs the Apache RocketMQ web console on the name servers. However, as Apache name servers run in the private subnets, they cannot be accessed directly through a browser. To access port 8080 of the private IP address of the Apache name server, configure an SSH connection with the bastion host SSH tunnel. Then use the SSH tunnel to access the web console.

  1. Establish a connection with the bastion host using SSH with the following command. Replace port number, key pair.pem, user name, and host name with your parameters.

    ssh -qTfnN -D port number -i "key pair.pem" user name@host name

    For example:

    ssh -qTfnN -D 40011 -i "rocketMQ.pem" ec2-user@ec2-54-223-36-247.cn-north-1.compute.amazonaws.com.cn

  2. Set up a proxy manager in your browser. There are many proxy manager plug-ins available. These instructions use SwitchyOmega.

    1. Install the SwitchyOmega plug-in.

    2. Open the SwitchyOmega options page and select New Profile in the left sidebar.

      SwitchyOmega
      Figure 3. Add new SwitchyOmega profile

    3. Enter a name and select Create.

      SwitchyOmega
      Figure 4. Profile name

    4. Enter the Protocol, Server, and Port for proxy server. The port should be the local port where you set up the SSH tunnel in step 1.

      SwitchyOmega
      Figure 5. Proxy servers

    5. Select Apply Changes.

    6. Access SwitchyOmega through the extension icon of your browser. Select your created profile in the proxy list. The browser will send all traffic through the local port 40011 proxy to the bastion host.

      SwitchyOmega
      Figure 6. Proxy list

  3. To view the Apache web console on a name server in your browser, navigate to http://10.0.xx.xx:8080. Replace xx.xx with the private IP address of the name server. You can find the private IP address of a name server in the EC2 console.

    ec2
    Figure 7. Private IP address of an Apache name server in the EC2 console

    Figure 8 shows an example of an Apache RocketMQ cluster in the Apache web console.

    console
    Figure 8. Apache web console

Troubleshooting

For troubleshooting common Partner Solution issues, refer to the AWS Partner Solution General Information Guide and Troubleshooting CloudFormation.

  • I cannot access the Apache RocketMQ web console.

    • The Apache RocketMQ web Console runs on an Apache name server in a private subnet. Verify that the SSH tunnel from your desktop to the bastion host is set up to connect to the Apache name server.

    • If the SSH tunnel is setup correctly, but you still cannot open the web console, it is possible that the web console did not start properly. Log in to the name server and check the install log file at /home/ec2-user/rocketmq-deploy/install.log.

  • The Apache RocketMQ broker server cluster does not show up in the Apache RocketMQ web console.

    • Log in to the broker node and check the install log in /home/ec2-user/rocketmq-deploy/install.log. You can submit an issue to the Quick Start GitHub repository for assistance. Include the error log with your submission.

  • How can I access the Apache RocketMQ internal logs?

    • On the name server and brokers, internal Apache RocketMQ logs are found in /home/ec2-user/rocketmq-deploy/.

Apache RocketMQ resources

NameServer nodes

  • Apache RocketMQ installation guide: /home/ec2-user/rocketmq-deploy

  • Apache RocketMQ installation directory: /home/ec2-user/rocketmq-deploy/rocketmq-all-4.7.1-bin-release or /home/ec2-user/rocketmq-deploy/rocketmq-all-4.8.0-bin-release

  • Deployment script installation log used to troubleshoot error messages: /home/ec2-user/rocketmq-deploy/install.log

  • Apache RocketMQ web console: /home/ec2-user/rocketmq-deploy/rocketmq-console-ng-1.0.0.jar

Broker server nodes

  • Apache RocketMQ installation guide: /home/ec2-user/rocketmq-deploy

  • Apache RocketMQ installation directory: /home/ec2-user/rocketmq-deploy/rocketmq-all-4.7.1-bin-release or /home/ec2-user/rocketmq-deploy/rocketmq-all-4.8.0-bin-release

  • Deployment script installation logs: /home/ec2-user/rocketmq-deploy/install.log

  • Configuration file: /home/ec2-user/rocketmq-deploy/rocketMQ-config/

  • Apache RocketMQ local file storage: /home/ec2-user/rocketmq-deploy/rmqstore

Best practices for using RocketMQ on AWS

Customer responsibility

After you deploy a Partner Solution, confirm that your resources and services are updated and configured—including any required patches—to meet your security and other needs. For more information, refer to the Shared Responsibility Model.

Feedback

To submit feature ideas and report bugs, use the Issues section of the GitHub repository for this Partner Solution. To submit code, refer to the Partner Solution Contributor’s Guide. To submit feedback on this deployment guide, use the following GitHub links:

Notices

This document is provided for informational purposes only. It represents current AWS product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS products or services, each of which is provided "as is" without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at https://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "as is" basis, without warranties or conditions of any kind, either expressed or implied. Refer to the License for specific language governing permissions and limitations.