Chainlink Node on AWS

Partner Solution Deployment Guide

September 2021
David Dang, Rodger Johnson, Chainlink Labs
Vijay Krishnan, AWS Solutions Architect
Troy Ameigh, AWS Integration & Automation team

Refer to the GitHub repository to view source files, report bugs, submit feature ideas, and post feedback about this Partner Solution. To comment on the documentation, refer to Feedback.

This Partner Solution was created by Chainlink Labs in collaboration with Amazon Web Services (AWS). Partner Solutions are automated reference deployments that help people deploy popular technologies on AWS according to AWS best practices. If you’re unfamiliar with AWS Partner Solutions, refer to the AWS Partner Solution General Information Guide.

Overview

This guide covers the information you need to deploy the Chainlink Node Partner Solution in the AWS Cloud.

This Partner Solution deploys highly available Chainlink nodes with default parameters and a blockchain to the AWS Cloud. It is for enterprise users who want to run a secure Chainlink node to provide external resources such as external APIs, tamper-proof price data, and verifiable randomness directly to smart contracts stored on the blockchain.

Chainlink is a data oracle that enables smart contracts on any blockchain to take advantage of extensive off-chain resources, such as tamper-proof price data, verifiable randomness, and external APIs.

When deployed to a blockchain, a smart contract is a set of instructions that can be run without intervention from third parties. The code of a smart contract determines how it responds to input, just like the code of any other computer program.

The Chainlink node is middleware that operates between the blockchain and external data. Node operators are able to run their Chainlink nodes with oracle job specifications, or specs, which contain the sequential tasks that the node must perform to produce a final result. The Chainlink node provides external data directly to smart contracts.

Costs and licenses

There is no cost to use this Partner Solution, but you will be billed for any AWS services or resources that this Partner Solution deploys. No licenses are required to deploy this Partner Solution. For more information, refer to the AWS Partner Solution General Information Guide.

Architecture

Deploying this Partner Solution with default parameters builds the following Chainlink Node environment in the AWS Cloud.

Figure 1. Partner Solution architecture for Chainlink Node on AWS

As shown in Figure 1, the Partner Solution sets up the following:

A highly available architecture that spans two Availability Zones.*
A VPC configured with public and private subnets, according to AWS best practices, to provide you with your own virtual network on AWS.*
In the public subnets:
- An internet gateway to allow access to the internet.*
- Managed network address translation (NAT) gateways to allow outbound internet access for the Chainlink node instances in the private subnets.*
- A Linux bastion host in an Auto Scaling group to allow inbound Secure Shell (SSH) access to Amazon Elastic Compute Cloud (Amazon EC2) instances in public and private subnets.
In the private subnets:
- Two Chainlink nodes in an Auto Scaling group that contains security groups for fine-grained inbound access control.
- An Amazon Relational Database Service (Amazon RDS) PostgreSQL managed database instance.
An Elastic Load Balancing (ELB) load balancer to access the Chainlink node user interface.
Amazon CloudWatch logging of Chainlink nodes.
AWS Key Management Service (AWS KMS) resources with both AWS managed and customer managed keys.
AWS Secrets Manager to store credentials.
Amazon DevOps Guru tracking of your Chainlink node stack.

* The template that deploys the Partner Solution into an existing VPC skips the components marked by asterisks and prompts you for your existing VPC configuration.

Deployment options

This Partner Solution provides the following deployment options:

Deploy Chainlink Node into a new VPC. This option builds a new AWS environment that consists of the VPC, subnets, NAT gateways, security groups, bastion hosts, and other infrastructure components. It then deploys Chainlink Node into this new VPC.
Deploy Chainlink Node into an existing VPC. This option provisions Chainlink Node in your existing AWS infrastructure.

This Partner Solution provides separate templates for these options. It also lets you configure Classless Inter-Domain Routing (CIDR) blocks, instance types, and Chainlink node settings.

Predeployment steps

Create an AWS account at https://aws.amazon.com by following the on-screen instructions.
Create a key pair in your preferred AWS Region.
If using a domain and SSL certificate, create a public certificate using AWS Certificate Manager (ACM) in your preferred Region.
Run a blockchain client or use a third-party client. If using a third-party Ethereum client, you can use the web socket endpoint from Infura or Fiews. For other Ethereum clients, see Run an Ethereum Client.

If using an existing VPC, make sure that it contains two public subnets, two private subnets, an internet gateway, NAT gateways, and route tables.

Make sure that the private subnets are in different Availability Zones. These subnets require NAT gateways in their route tables to allow instances to communicate with other Chainlink nodes without exposing them to the internet.

Make sure that you are familiar with networking, DevOps, and running Chainlink nodes.
Familiarize yourself with the following AWS services:

Deployment steps

Sign in to your AWS account, and launch this Partner Solution, as described under Deployment options. The AWS CloudFormation console opens with a prepopulated template.
Choose the correct AWS Region, and then choose Next.
On the Create stack page, keep the default setting for the template URL, and then choose Next.

On the Specify stack details page, change the stack name if needed. Review the parameters for the template. Provide values for the parameters that require input. For all other parameters, review the default settings and customize them as necessary. When you finish reviewing and customizing the parameters, choose Next.

Unless you’re customizing the Partner Solution templates or are instructed otherwise in this guide’s Predeployment section, don’t change the default settings for the following parameters: QSS3BucketName, QSS3BucketRegion, and QSS3KeyPrefix. Changing the values of these parameters will modify code references that point to the Amazon Simple Storage Service (Amazon S3) bucket name and key prefix. For more information, refer to the AWS Partner Solutions Contributor’s Guide.

On the Configure stack options page, you can specify tags (key-value pairs) for resources in your stack and set advanced options. When you finish, choose Next.
On the Review page, review and confirm the template settings. Under Capabilities, select all of the check boxes to acknowledge that the template creates AWS Identity and Access Management (IAM) resources that might require the ability to automatically expand macros.
Choose Create stack. The stack takes about 45 minutes to deploy.
Monitor the stack’s status, and when the status is CREATE_COMPLETE, the Chainlink Node deployment is ready.
To view the created resources, choose the Outputs tab.

Postdeployment steps

Access your Chainlink node

You can access the Chainlink node UI using one of the following options:

Option 1: If using a domain and SSL certificate created through ACM, you can access the Chainlink node UI through the Application Load Balancer endpoint or through your domain by adding the load balancer endpoint to your domain’s DNS record.

Option 2: Allow TCP port forwarding on your bastion host to port forward the Chainlink node UI to your machine. When accessing the Chainlink node through a bastion host, use SSH agent forwarding. Then connect to your bastion host with an SSH agent and TCP port forwarding. Access the Chainlink node UI at http://localhost:6688/.

ssh-add <your_ssh_key>
ssh ec2-user@<bastion_host_public_ip> -A -L 6688:localhost:6688
ssh ec2-user@<chainlink_node_internal_ip> -L 6688:localhost:6688

Fulfill requests with the Chainlink node

After the Partner Solution is deployed, your Chainlink node is ready to interact with smart contracts and fulfill requests. For more information, see Fulfilling Requests.

Work with external adapters

Chainlink enables integration of custom computations and specialized APIs using external adapters. The Chainlink node communicates with external adapter services via an API using a JSON specification. To get started, see External Adapters Introduction.

Manually create files for Chainlink node instances

If the Chainlink node instances are stopped, then you must create new .env, .password, and .api files to start the Chainlink node.

Generate the .env file:

cd /home/ec2-user/.chainlink/ && ./create-env.sh \
${chainNetwork} \
${blockchainNodeUrl} \
${psqlUser} \
$(aws secretsmanager get-secret-value --secret-id DBSecret --query "SecretString" --output text) \
${psqlHostname} \
${psqlPort} \
${psqlDb}

Generate the .password file:

cd /home/ec2-user/.chainlink/ && ./create-password.sh \
$(aws secretsmanager get-secret-value --secret-id WalletSecret --query "SecretString" --output text)

Generate the .api file:

cd /home/ec2-user/.chainlink/ && ./create-api.sh \
${apiUser} \
$(aws secretsmanager get-secret-value --secret-id ApiSecret --query "SecretString" --output text)

Start Chainlink node instances

To run the latest image of the Chainlink node Docker instance, run this command:

latestimage=$(curl -s -S "https://registry.hub.docker.com/v2/repositories/smartcontract/chainlink/tags/" | jq -r '."results"[]["name"]' | head -n 1)
cd /home/ec2-user/.chainlink && docker run -d \
--log-driver=awslogs \
--log-opt awslogs-group=ChainlinkLogs \
--restart unless-stopped \
--name chainlink \
-p 6688:6688 \
-v /home/ec2-user/.chainlink:/chainlink \
--env-file=/home/ec2-user/.chainlink/.env  smartcontract/chainlink:$latestimage local n \
-p /chainlink/.password
-a /chainlink/.api

Best practices for using Chainlink Node on AWS

Failover capabilities

To reduce downtime, failover capabilities are required on both the Chainlink and Ethereum clients so that if one server fails, the service is still online. The Amazon EC2 Auto Scaling group has two Chainlink nodes: one active Chainlink node and one standby Chainlink node. Data from both the Chainlink and Ethereum clients are stored in a highly available PostgreSQL database.

Disaster recovery

If problems occur, the right processes must be in place to reduce downtime. The most common reason for downtime (in the context of Chainlink node operators) is a corrupted Ethereum node that requires a resynchronization.

To recover an Ethereum client and reduce downtime, follow these recommendations:

Take daily snapshots of the Ethereum chain on a separate server than the one connected to the Chainlink node.
Use an Ethereum client start-up process that pulls down the latest template of the chain and syncs it to the latest height.

Active monitoring

To detect issues before or when they occur, use active monitoring in the following areas:

(Minimum required) Ethereum of the wallet address assigned to the node.
Errored job runs.
Operator UI port to be open and responsive (usually 6688).
Ethereum HTTP and web socket ports to be open and responsive (usually 8545 and 8546).
Ethereum client disk, RAM, and CPU usage.

You can use Amazon CloudWatch to monitor Chainlink nodes.

Other resources

For more information, see Best Practices for Deploying Nodes on AWS on the Clainlink website.

Security

The Chainlink node Docker instance requires the .env, .password, and .api file that contains plaintext passwords. Remove the .env, .password, and .api files after the Chainlink node instance is running to prevent potential exposure of sensitive passwords.

Troubleshooting

For troubleshooting common Partner Solution issues, refer to the AWS Partner Solution General Information Guide and Troubleshooting CloudFormation.

I encountered a CREATE_FAILED error when I launched the Partner Solution.

If AWS CloudFormation fails to create the stack, relaunch the template with Rollback on failure set to Disabled. This setting is under Advanced in the AWS CloudFormation console on the Configure stack options page. With this setting, the stack’s state is retained, and the instance keeps running so that you can troubleshoot the issue. (For Windows, look at the log files in %ProgramFiles%\Amazon\EC2ConfigService and C:\cfn\log.)

When you set Rollback on failure to Disabled, you continue to incur AWS charges for this stack. Delete the stack when you finish troubleshooting.

I encountered a size-limitation error when I deployed the AWS CloudFormation templates.

Launch the Partner Solution templates from the links in this guide or from another S3 bucket. If you deploy the templates from a local copy on your computer or from a location other than an S3 bucket, you might encounter template-size limitations. For more information, see AWS CloudFormation quotas.

What other environment variables can I add to my Chainlink node?

For a list of environment variables and their explanations, refer to Configuration Variables.

I encountered an 'Error reading password: open /chainlink/.password: no such file or directory' error message when starting my Chainlink node.

Make sure that the .password files exist inside the Chainlink root directory. If the .password file does not exist, run the following command to create the .password file:

cd /home/ec2-user/.chainlink/ && ./create-password.sh \
$(aws secretsmanager get-secret-value --secret-id WalletSecret --query "SecretString" --output text)

I encountered a 'Creating application: parse "XXXXXX": invalid URI for request' error message when starting my Chainlink node.

This error typically happens when a parsing error exists within the PostgreSQL connection string (postgresql://) in the DATABASE_URL variable. Double-check the .env file for any errors in format or spelling. You can manually edit the connection string or run the following command to create a new .env file:

cd /home/ec2-user/.chainlink/ && ./create-env.sh \
${chainNetwork} \
${blockchainNodeUrl} \
${psqlUser} \
$(aws secretsmanager get-secret-value --secret-id DBSecret --query "SecretString" --output text) \
${psqlHostname} \
${psqlPort} \
${psqlDb}

Customer responsibility

After you deploy a Partner Solution, confirm that your resources and services are updated and configured—including any required patches—to meet your security and other needs. For more information, refer to the Shared Responsibility Model.

Feedback

To submit feature ideas and report bugs, use the Issues section of the GitHub repository for this Partner Solution. To submit code, refer to the Partner Solution Contributor’s Guide. To submit feedback on this deployment guide, use the following GitHub links:

Notices

This document is provided for informational purposes only. It represents current AWS product offerings and practices as of the date of issue of this document, which are subject to change without notice. Customers are responsible for making their own independent assessment of the information in this document and any use of AWS products or services, each of which is provided "as is" without warranty of any kind, whether expressed or implied. This document does not create any warranties, representations, contractual commitments, conditions, or assurances from AWS, its affiliates, suppliers, or licensors. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

The software included with this paper is licensed under the Apache License, version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at https://aws.amazon.com/apache2.0/ or in the accompanying "license" file. This code is distributed on an "as is" basis, without warranties or conditions of any kind, either expressed or implied. Refer to the License for specific language governing permissions and limitations.