Visualizing Security Groups with Amazon Elasticsearch

Intro

Elasticsearch is an open-source search and analytics engine for things like log analytics, real-time application monitoring, etc.

Amazon Elasticsearch is a managed service that makes it easy to deploy, operate, and scale Elasticsearch.

Kibana is a tool for visualizing (typically, large amounts of) data indexed by Elasticsearch, using bar graphs, line graphs, scatter plots, pie charts, etc.

About the Lab

See how to use VPC Flow Logs and Amazon Elasticsearch to track port, protocol, and IP address of traffic passing thru' a VPC.

Then configure Kibana to view VPC Flow Logs data and use a bash script to map Security Groups to AWS resources associated with them.

After relationships are established, use Kibana to identify unnecessary ports that are open on an instance.

Launch EC2 instance

1. In the AWS Console, click Services, then click EC2

2. Click Launch Instance

3. Click Select against the Amazon Linux AMI option

4. Choose a t2.micro (or any free tier eligible instance – we don’t need anything beyond that) and click Next: Configure Instance Details

5. For Auto-assign Public IP choose Enable and leave the rest as-is

6. Click on Next: Add Storage

7. Enter 16 under Size column (16GB for storage) and leave the rest as-is

8. Click on Next: Add Tags, then click on Add Tag

9. Under Key enter Name and under Value enter VPC Flow Logs Test

10. Click Next: Configure Secuity Group

11. Choose Create a new security group

12. Enter flowLogsTestSG-[your-user_id] as the security group’s name and something useful in the description (if sharing accounts, adding your name/userid to description could be helpful)

13. Click on Add Rule and choose HTTP (to allow all HTTP traffic in addition to SSL)

14. Click on Review and Launch and review one last time

15. Click on Launch to open the KeyPair modal dialog
    
    Note: If you already have a keypair that you know works with this account and you’re sure that you can login with that keypair, then you can skip the steps for creating and downloading a new keypair. Alternatively, if you have a keypair, now’s a good time to test if you’re able to login to the instance with the keypair.

16. Choose Create a new keypair and give it a descriptive name

17. Click on Download Key Pair (this is the only time you’ll be able to download it)

18. Click on Launch
    File-Name.pem>

This will take a few mins, so let’s proceed. We’ll connect to this EC2 instance later.

IAM Credentials, Roles, and Policies

IAM Credentials

If you already have IAM Credentials, you can skip this section.

You’ll need your own IAM Credentials (that is valid and that works). You’ll need this to authenticate and run AWS APIs via commandline. If you don’t have IAM Credentials, you may be able to create a new set of IAM Credentials yourself (if your AWS Account admin has granted you permissions to iam:CreateAccessKey, iam:DeleteAccessKey, iam:UpdateAccessKey, and iam:ListAccessKeys or request your AWS Account admin to grant you permissions or create them for you.

To Create new IAM Credentials

• In the AWS Console, click Services, then click IAM
• On the left hand pane, click Users
• Select your IAM user
• Click on the Security Credentials tab
• Click Create
• In the modal dialog, click show. This is the only time the secret access key will be available.
• Copy-paste both Access Key ID and Secret access key into a text editor and save it securely.

Create IAM Role and Policy for Flow Logs to CloudWatch

1. In the AWS Console, click Services, then click IAM

2. On the left hand pane, click Policies and click Create policy

3. Choose the JSON tab and enter the policy below

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams",
                "logs:PutLogEvents"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

1. Click Review policy

2. For the name, enter flowLogsPolicy-[your-user-id] and then click Create policy

3. On the left hand pane, click Roles and then click Create role

4. Choose AWS service For Select type of trusted entity

5. Choose EC2 for Choose the service that will use this role

6. Click Next: Permissions and then click Create policy

7. For Filter: enter flowLogsPolicy and choose the check box on the left to attach this policy to the role we are creating

8. Click on Next: Review

9. For Name enter flowLogsRole-[your-user-id]

10. Click on the newly created role and then click on the Trust relationships tab

11. Click on Edit trust relationship and replace ec2.amazonaws.com with vpc-flow-logs.amazonaws.com

12. Click Update Trust Policy

13. Phew!

Create Lambda Role and Policy

Lambda Execution Policy

• From inside the IAM service in the AWS Console, click on Policies in the left pane
• Click Create policy
• Click on the JSON tab
• Enter the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": "es:ESHttpPost",
            "Resource": "arn:aws:es:*:*:*",
            "Effect": "Allow"
        }
    ]
}

• Click on Review policy
• For Name enter lambdaElasticsearchExecutionPolicy-[your-user-id] (enter a description, optionally)
• Click Create policy

• Click on the newly created role and then click on the Trust relationships tab

• Click on Edit trust relationship and replace ec2.amazonaws.com with lambda.amazonaws.com

Lambda Execution Role

• On the left hand pane, click on Roles
• Click Create Role
• Select AWS service
• Select EC2
• Click on Next: Permissions
• In Filter: type lambdaElasticsearchExecutionPolicy (that you created above) and select the associated check box
• Click Next:Review
• For Name enter lambdaElasticsearchExecutionRole-[your-user-id]
• Click Create role

Create EC2 Role and Policy

EC2 Policy

• From inside the IAM service, click on Policies in the left hand pane
• Click Create policy
• Click on the JSON tab
• Enter the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "config:GetResourceConfigHistory"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeVPCs"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "rds:DescribeDBInstances"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "elasticloadbalancing:DescribeLoadBalancers"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "redshift:DescribeClusters"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "ec2:DescribeNetworkInterfaces"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "elasticache:DescribeCacheClusters"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "es:Describe*",
                "es:List*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

• Click on Review policy
• For Name enter ec2Policy-[your-user-id] (enter a description, optionally)
• Click Create policy

EC2 Role

• On the left hand pane, click on Roles
• Click Create Role
• Select AWS service
• Select EC2
• Click on Next: Permissions
• In Filter: type ec2Policy (that you just above created) and select the associated check box
• Click Next:Review
• For Name enter ec2Role
• Click Create role

Create an Elasticsearch Cluster

1. In the AWS Console, click Services then click Elasticsearch Service.

2. Click Get started.

3. For Elasticsearch domain name,type flowlogs-[your-user-id]

4. For Elasticsearch version choose 2.3

5. Click Next.

6. For Instance count, type 2.

7. Check the box labeled Enable zone awareness.

8. Leave all other settings at their defaults, and click Next.
   
   Note These settings instruct the Elasticsearch Service to launch your domain with two nodes. Because you have enabled zone awareness, these nodes will be separated into two different availability zones. If this were a production cluster, you would then use the Elasticsearch API to set up replicas for your cluster. These replicas would then be distributed across the nodes in the two Availability Zones, increasing the availability of the cluster.
   
   Kibana, which is included in Elasticsearch Service clusters, does not directly support authentication. In a production environment, you will need an authentication proxy in front of Kibana, but for this lab you will restrict access by whitelisting your public IP address.

9. For Set the domain access policy to,choose Allow access to the domain from specific IP(s).

10. Choose the Public Access radio button

11. In the IP address dialogbox, enter two IP addresses separated by a comma. Something like this: 73.159.145.254, 35.163.64.127
    
    Enter your computer’s public IP address followed by a comma. Don’t know your IP address? Visit https://www.google.com/search?q=what+is+my+ip+address in a separate tab.
    
    Then enter the Public IP address of your EC2 Instance.
    
    When you enter the IP addresses, it should look something like this:
    73.159.145.254, 35.163.64.127

12. Your cluster’s access policy should now be visible. Click Next to move on to configuration review.

13. After verifying your cluster settings, click Confirm and create to create your Elasticsearch domain.
    
    Note It can take up to 15 minutes to finish creating a domain. While your domain is still creating its Domain status will be listed as loading. While it is creating we will continue setting up other lab components, but you may wish to leave the Elasticsearch console open in a separate browser tab as you will return to it later.

Enable VPC Flow Logs

1. In the AWS Management Console, on the Services menu, click VPC. On the navigation pane, click Your VPCs.

2. Check the box next to your Default VPC to select it.

3. In the VPC Details frame at the bottom of the page, click the Flow
   Logs tab. Delete any existing Flow Logs. Then click Create Flow Log.

4. For Create Flow Log, click on Role and select the newly created
   flowlogsRole.

5. For DestinationLogGroup, type FlowLog-[your-user-id]. This log group
   will contain entries for your VPC traffic.

6. Click Create Flow Log.
   Note VPC Flow Logs can take up to 15 minutes to create the CloudWatch Logs Group and begin delivering flow records. While you wait for the Log Group to be created, continue logging into the EC2 instance provided with the lab.

Connect to EC2 Instance

To connect to your EC2 instance you’ll need its public DNS.

• On the Services menu, click EC2.
• On the navigation pane, click Instances.
• Click the instance you created for the lab and copy the public DNS to a local text editor.

Mac/Linux Users:

This section is for Mac/Linux users only. If you are running Windows, please skip to the next section.

• Enter the below commands substituting the path /filename for the .pem file you downloaded from qwikLABS and pasting ec2-user@ to substitute the example below.

$ chmod 600 ~/Downloads/<EC2-KeyPair-PEM-Format-File-Name.pem>
$ ssh -i ~/Downloads/<C2-KeyPair-PEM-Format-File-Name.pem> ec2-user@<Public

Windows Users:

This section is for Windows users only. If you are running OSX/Linux, please refer to the previous section.

• Download PuTTY from http://www.putty.org and launch putty.exe.

• In the Host Name box, type

ec2-user@<publicDNS>

Paste the public DNS value from your Clipboard.

• In the Category list, expand SSH.

• Click Auth (don’t expand it).

• Use puttygen.exe to convert the PEM file into a PPK file. To see how to do that, see this StackOverflow post: https://stackoverflow.com/questions/3190667/convert-pem-to-ppk-file-format

• In the Private key file for authentication box, browse to the PPK file that you downloaded and double-click it.

• Click Open.

• Click Yes when prompted to allow a first connection to this remote
  SSH server. Because you are using a key pair for authentication, you will not be prompted for a password.

Troubleshooting

If PuTTY fails to connect to your Amazon EC2 instance, verify that:

• You entered ec2-user@ in PuTTY.
• You downloaded the PPK file for this lab from Qwiklabs.
• You are using the downloaded PPK file in PuTTY.
• The network you are on allows for outbound TCP connections to destination port

Install utilities

• Install jq

sudo yum install -y jq

• Install and start httpd

sudo yum install -y httpd
sudo service httpd start

• Make /var/www/html/ writable.
  Not recommended outside this context, but for the purposes of this lab, a quick and dirty sudo chmod -R ugo+r+w /var/www/html/ will also work.

Configure AWS CLI default profile

Now you’ll create (or use) your own IAM Credentials and configure the default aws commandline profile in the EC2 instance to use it. To do this got to the EC@ instance commandline and run:

aws configure

and enter the values asked for by the prompt. It should look something like this

[ec2-user@ip-172-31-30-125 ~]$ aws configure
AWS Access Key ID [None]: AKIA----------------
AWS Secret Access Key [None]: <Need 2 parts vodka, 1 part...>
Default region name [None]: ca-central-1
Default output format [None]: json
[ec2-user@ip-172-31-30-125 ~]$ 
[ec2-user@ip-172-31-30-125 ~]$ 

Verify Elasticsearch Cluster Provisioning

1. Return to the Elasticsearch Service console. In the AWS Management Console, click Services, then click Elasticsearch Service. Verify the Cluster health for your cluster is Green. If your cluster is not yet green, wait a few minutes and refresh the browser page. Once Cluster health is listed as Green, continue to the next part of the lab.

2. On the Services menu, click CloudWatch. Then click Logs in the left CloudWatch Console pane. The log group you created, FlowLogs, should be listed in the Log Groups column. If you see a Getting Started page instead of your log group, wait a few minutes and refresh the page. Once your log group appears under the Log Groups column header, you can continue.

Configure Streaming to Amazon Elasticsearch Service

Note Do not begin these steps until have you have verified that your resources have finished provisioning.

1. In the CloudWatch Log Groups page, check the box next to the log group you created, FlowLogs.

2. Click Actions and select Stream to Amazon Elasticsearch Service.

3. For Amazon ES cluster, select your cluster, flowlogs-[your-user-id].

4. In the Lambda IAM Execution Role field, select the role
   lambdaElasticsearchExecutionRole, then click Next.

5. In the Log Format field, select Amazon VPC Flow Logs. Leave other settings at their defaults and click Next.

6. Review your streaming settings, thenclickNext.ClickStart Streaming.
   You have now created a Lambda function that is triggered whenever new logs are written to CloudWatch Logs from VPC Flow Logs. Each Lambda invocation transforms the new records posted to CloudWatch and inserts them into your ElasticSearch Cluster.

Format Data in the Elasticsearch Cluster

A flow log record is a space-separated string in the following format:

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status

By default, Elasticsearch passes strings through an analyzer before being indexed. The analyzer interprets dashes and periods as field separators and will split fields that have them present. As our logs contain dashes this will create undesired behavior. In order to correct this, we will use the curl command from our instance to configure interface-id, srcaddr, and dstaddr to not_analyzed. This configuration change will cause Elasticsearch to skip the analyzer for these fields and leave their punctuation intact.

• Determine the endpoint for your Elasticsearch cluster.You will substitute this endpoint for ElasticsearchDomainEndpoint in upcoming steps. You can view your endpoint by navigating to the Elasticsearch Console and clicking on your domain, flowlogs. You can also find this information using the AWS CLI from your EC2 instance by running:

aws es describe-elasticsearch-domain --domain-name flowlogs

Your endpoint will look something like this:

search-flowlogs-nihldocymhswk4gzonzrw4j5rq.us-west-2.es.amazonaws.com.

• From your EC2 instance, execute the following command to change your cluster’s index configuration (replace the placeholder text with your endpoint):

curl -XPUT "https://<ElasticsearchDomainEndpoint>/_template/template_1" -d'
{
    "template":"cwl-*","mappings":{
        "FlowLogs": {
            "properties": {
                "interface_id": { "type": "string", "index": "not_analyzed"},
                "srcaddr": { "type": "string", "index": "not_analyzed"},
                "dstaddr": { "type": "string", "index": "not_analyzed"}
            }
        }
    }
}' -H 'Content-Type: application/json'

Note After you add the endpoint URL, make sure you also remove the < > brackets from the placeholder text. (I left that in…)

Delete the old, incorrectly indexed data by running the following command:(replace the placeholder text with your endpoint):

curl -XDELETE 'https://<ElasticsearchDomainEndpoint>/cwl*/'

Import Dashboards and Visualizations

Now configure visualizations in Kibana to view the data in Elasticsearch. Kibana is a tool built into Amazon Elasticsearch used to visualize and search data. An example dashboard has been created here: https://s3.amazonaws.com/awsiammedia/public/sample/securitygroups/SGDashboard.json.

Dashboards are defined in JSON and are normally imported through curl commands against the Elasticsearch endpoint.

Here, however, we will be using a tool called elasticdump to make this process easier.

• Elasticdump requires node and npm, so install them like so:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.32.0/install.sh | bash

. ~/.nvm/nvm.sh

nvm install 8.11.1

• To install elasticdump, go to your EC2 Instance and enter the commands below.

npm install -g elasticdump

• Now download the Kibana dashboard like so:

curl -O https://s3.amazonaws.com/awsiammedia/public/sample/securitygroups/SGDashboard.json

• Now run elasticdump and import this dashboard. Replace with your Elasticsearch endpoint.

elasticdump --input=SGDashboard.json --output=https://<ElasticsearchDomainEndpoint>/.kibana-4

View your Kibana Dashboard

1. Return to the Elasticsearch Console. Click your domain, flowlogs. Click on the link for Kibana.

2. You will be prompted to configure an index pattern. You do not need to do this. Instead, click on the existing pattern you imported in the last step. Click on [cwl-]YYY.MM.DD in the left pane under Index Patterns.

3. Click the green star to set as default index.

4. Click Dashboard (in the top-level menu), then click on the folder icon
   to Load Saved Dashboard. Click FlowLogDash.
   You should now see a visualization of traffic flowing into and out of your
   VPC.

Associating ENI’s to Security Groups

Configuring the ENI mapping script

As you viewed the visualization of your Flow Logs data, you may have noticed that traffic was broken down by ENI, not by Security Group. Security Group information is not provided in Flow Logs, so you will need to generate ENI to Security Group mappings . You will do this through a bash script, sgremediate.sh. Download this from https://s3.amazonaws.com/awsiammedia/public/sample/securitygroups/sgremediate.sh like so:

curl -O https://s3.amazonaws.com/awsiammedia/public/sample/securitygroups/sgremediate.sh

chmod +x sgremediate.sh

You will need to configure two settings in the script: your VpcID and Elasticsearch Endpoint. This information is available from the AWS Console, but we’ll gather it from the AWS CLI instead.

To view your VpcID, in your EC2 instance run the following command:

aws ec2 describe-vpcs --filter Name=isDefault,Values=true | jq -r .'Vpcs[].VpcId'

This command returns JSON data about your Default VPC and parses it with jq. The result is your VpcID which will be needed when you configure the script. It should look something like this: vpc-13303b71.

• To view your Elasticsearch Endpoint, in your EC2 instance run the following command.

aws es describe-elasticsearch-domain --domain-name flowlogs | jq -r .'DomainStatus.Endpoint'

It returns JSON data about your Elasticsearch domain and parses it with jq. The result is your Elasticsearch Endpoint which will be needed when you configure the script. Your endpoint should look something like this: search-flowlogs-nihldocymhswk4gzonzrw4j5rq.us-west-2.es.amazonaws.com.

• In your EC2 instance, open sgremediate.sh with a text editor such as nano or vi

• Edit sgremediate.sh and…
  For vpcID= paste your lab’s VPC ID.
  For ElasticsearchURL= paste your Elasticsearch endpoint. (Note that part of the URL is already filled in for you, so look out for mangled URLs.)

• Save and exit the script file.

Building ENI to Security Group Mapping

Executing the script will list all of the Security Groups in your account and iterate through every resource they could be attached to. When it finds an association, it builds a link to the Kibana dashboard for that Security Group and outputs it as HTML. Your EC2 instance is running a web server that can host the HTML output.

• Run the script and redirect the output to the web server root with the following command:

./sgremediate.sh > /var/www/html/index.html

• In your web browser, visit the IP address for your EC2 instance. Click on the link for your security group to be taken to the Kibana dashboard for that specific Security Group.

Using the Flowlogdash Dashboard

The Kibana dashboard consists of a set of visualizations. Each visualization summarizes data available in your Elasticsearch clustrer for a particular timeframe. The timeframe can be adjusted in the top right corner.
On the FlowLogDash dashboard, the left side consists of three panes.

• The top pane contains a list of the ENIs, a count of records, and the sum of bytes.
• The middle pane shows the percentages of accepts and rejects.
• The pane shows the percentages of protocols in the flow log data.
• The large middle pane of the dashboard contains a pie chart that displays source IP address, protocol, destination port, and destination IP address for VPC network traffic.
• The right pane contains a list of destination ports with item count. The bottom pane contains raw Flow Log records.

Using this dashboard you can visualize what traffic is actually flowing into a particular security group. If the visualization shows that no traffic has passed to a specific port that is open in the security group you have successfully identified an overly permissive security group that can be tightened. If the visualization shows limited traffic to a specific port that is open in the security group, you can further evaluate the source IP of that traffic to determine if it is expected or potentially abusive (e.g. port scaning). If you determine that the port is not intentionally left open, you have identified an overly permissive security group that can be tightened.

Notes

Multiple Security Groups can be associated to an ENI. VPC Flow logs track connections based on ENI, not Security Group, so dashboards created by the sgremediate.sh script will show ALL security groups associated to a particular ENI.

Also note that security groups are stateful. If an instance is initiating traffic to a different location Flow Logs will contain the return traffic.

Conclusion

You’ve successfully enabled VPC Flow Logs, streamed records to Elasticsearch, and visualized them with Kibana. You have also used Kibana dashboards to find overly permissive Security Groups.