Setup a Production-Grade Instance
This is just a single architectural example of how to configure QU4RTET in a production environment and by no means represents an absolute stack of technologies or cloud providers. Again, this is only a representation of a Vantage-curated approach to doing this task. QU4RTET can run on any platform, OS, web-server and database back-end.
There are many approaches to setting up production-grade architectures on AWS once you have an idea of how to configure your application layer to communicate with your UI and database layers. The architecture we will outline here is one of many possibilities and we will be doing tutorials like this on other architectural variations that are possible.
Advanced Setup: Using a Replicated, Clustered Database Backend
Adding a high availability, clustered back-end to QU4RTET is pretty easy. In this section we will guide you along. In no time you'll have a production grade QU4RTET instance ready to go.
The Vantage QU4RTET AMI is pre-configured to look for certain variables in your EC2 instance User Data. Here will will set the PARAMETER_GROUP user data value which will allow your instance to pick up secure environment variables from the Amazon Parameter Store during run time. The EC2 instance will use these parameters to connect to the database we will be creating in forthcoming steps.
Step 1: Stop Your Instance
The first step is to stop the EC2 QU4RTET instance. Right click and select stop as in the diagram below.
Step 2. Add User Data
Next, right click on the stopped instance, and click Instance Settings|View/Change User Data
Add the following data:
Save and then re-start your server by right clicking and choosing Instance State/Start.
Configure Aurora RDS or PostgreSQL
*QU4RTET supports many databases; however, the Vantage bundle is only configured for PostgreSQL. If you require another database, contact Vantage- we can help set one up for you.
Stand up a PostgreSQL (aurora) instance following the first part of the instructions below:
Creating a PostgreSQL DB Instance and Connecting to a Database on a PostgreSQL DB Instance and set up a new database instance sized to your liking.
To follow along with the guide use the following settings when configuring your database. Getting the security groups configured correctly can be tricky- make sure to follow the instructions on this closely.
- DB cluster identifier: qu4rtet-cluster
- database name: qu4rtet
- port: 5432
Other options are at your discretion.
OPTIONAL: Migrating From the Local DB to the New
This step is not required but is useful if you've configured a system and would like to migrate it from current state into a more production ready state.
If you'd like to migrate the local database and all of its data to the new RDS instance (which is optional), execute the following steps.
Note: This will require you to ssh into your instance using the key you downloaded when you first launched or execute the script steps below using the AWS Systems Manager.
Upgrade Your Client (Optional)
If your postgres client applications are less than version 10, execute the following:
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -sc)-pgdg main" > /etc/apt/sources.list.d/PostgreSQL.list' sudo apt-get update sudo apt-get install postgresql-client --upgrade
Switch over to the postgres user account:
sudo -i -u postgres
Optional: Test your connection
psql -h [host name] -U qu4rtet
Back Up the Local Database
pg_dump -h localhost -p 5432 -U postgres -F c -b -v -f /tmp/qu4rtet.backup qu4rtet
Create the qu4rtet Database on RDS
Since your new RDS instance does not have a qu4rtet database created, we will create on here.
createdb -e -E UTF8 -O qu4rtet -U qu4rtet --host=[your host] --port=5432 qu4rtet 'The QU4RTET database backend.
Restore to the RDS Instance
pg_restore -h [put the host name of your RDS instance here] -p 5432 -U qu4rtet -d qu4rtet -v /tmp/qu4rtet.backup
Configure an Inbound Rule
In your RDS instance details, under the Connect heading you will see Security Group Rules. Click on the security group link under the Security Group list- don't worry if there are a few. Just click on the first one.
On the subsequent page, click the Inbound tab at the bottom and click Edit. On the next screen select Add Rule and add the following rule:
|Source||Enter in the security group of your qu4rtet app server. type sg an you should see a group or list of groups to choose from.|
|Description||QU4RTET App Server as a CIDR (with a /32 at the end)|
Note: To allow more than one app server or resource to connect from your VPC , one may add a /16 CIDR value to the security rule.
On the RDS management page, click the Clusters link to the left and then click on the quartet-cluster-n link. This will take you to a details page. Copy the cluster endpoint value.
Your QU4RTET image is already pre-configured to work with the Amazon S3 storage API. It can access this API via values provided in the Amazon EC2 Parameter Store. The steps below will guide you through getting your EC2 instance connected to the Parameter Store.
Create an instance profile for Systems Manager managed instances
A more expansive tutorial on this can be found here... Configuring Access to Systems Manager
Open the IAM console at https://console.aws.amazon.com/iam/.
In the navigation pane, choose Roles, and then choose Create role.
On the Select type of trusted entity page, under AWS Service, choose EC2.
Note: If the Select your use case section appears, choose EC2 Role for Simple Systems Manager, and then choose Next: Permissions.
On the Attached permissions policy page, search and verify that AmazonEC2RoleforSSM is listed, select that and then search and verify that ReadSSMParameters is listed, Select that and then choose Next: Review.
On the Review page, type a name in the Role name box, and then type a description. We recommend QU4RTETInstanceManagement. But you can name it anything.
Note Make a note of the role name. You will choose this role when you create new instances that you want to manage by using Systems Manager.
Choose Create role. The system returns you to the Roles page. When you are done, your role should look like this:
Now go to your EC2 QUARTET instance (if you have one running already) and right click and select Attach/Replace IAM Role.
- Now that your system is able to be managed by SSM and access parameter store values, we are able to configure it to use your RDS instance.
Generate a Secret Key
Your QU4RTET instance uses a random secret key to encrypt sensitive data in the database. It is essential that every QU4RTET instance have it's own unique secret key for this purpose. Use the site below to generate a 50 character randomized string. Copy this value somewhere and save it for the steps below.
Set Up Your System's Parameters
There are a number of parameters that can be set via the Systems Management Parameter Store, the following will get you connected to your RDS instance. There are additional parameters available for the S3 configuration as well which are included in the Configure S3 For Storage
The following assumes the User Data in your QU4RTET instance has been set to PARAMETER_GROUP='GROUP-1' and that you've been following the tutorial using the example values.
|Put the URL to your PostgreSQL cluster server here from the cluster step above.||String|
|/GROUP-1/POSTGRES_USER||The name of the user you created above.||String|
|The password you created for the database user.||SecureString|
|/GROUP-1/DJANGO_SECRET_KEY||Paste in the secret key you generated above.||SecureString|
Your QU4RTET instance comes pre-configured to use Amazon S3 storage to queue inbound and outbound EPCIS messages and other such task data. This allows you to automatically expire data and gives you a flexible and (almost infinite) storage back-end for your system.
Create The QU4RTET Database Schema
Now that your system is configured to communicate with RDS, you can populate it with the QU4RTET database schema. To do this, we will use the AWS Systems Manager to issue a remote command.
Enter the Command Parameters
Select Run a Command
Next, locate the AWS-RunShellScript command document from the list and select it.
After you've selected this, copy this script and paste it into the command parameters box.
workon qu4rtet sudo ./utility/ec2_create_database.sh
Set the Working Directory
In the Working Directory text box, enter the following:
Select The QU4RTET Instance to Run Your Command On
You should see a list of available Targets in the Targets section. If you've just added your EC2 QU4RTET instance to the Systems Manager then it may be a few minutes before it shows up in the lis. Select your QU4RTET instance in the Targets section.
Set the S3 Bucket for Storage (Optional)
If you've set up an S3 bucket you can select that in the Output options to log any errors or output if you want.
Run The Command
Click the Run button at the bottom of the page and wait for the command to complete. Once it is done, you should see a screen like the one below:
IMPORTANT: Inspect The Output
Once your command is done, you can then view the output to ensure that it has executed correctly. Just because a command has run successfully does not mean there were no errors!
Dealing With Issues
502 Bad Gateway Issue
The 502 Bad Gateway issue (if you use a browser directly to test the install) is typically the result of a missing requirement or a postgresql password that has an illegal character in it such as a # sign.
The fastest way to figure out what is going on here is to log onto the server via SSH, navigate to /srv/qu4rtet and run
python manage.py runserver
This should generate an error of some sort that can be looked into.