Introduction
Edge is a new reconnaissance tool with a unique capability of mapping IP addresses to their cloud providers. It can tell you the data center and service an IP address is hosted with. It automatically downloads all three cloud provider (AWS, Azure, GCP) IP address JSON files, parses and loads them into memory, and can then perform fast lookups at scale. It tells you which cloud provider the IP address belongs to (AWS, Azure, GCP) along with the data center and cloud service (if applicable). This enables a network defender to do cloud provider attribution. This feature can also be correlated with features such as subdomain enumeration and certificate transparency supported by the tool.
Use Cases
Here are a few ideas or use cases for how the tool can be useful.
- Red Team, Bug Bounty, Penetration Testing: Start with a seed value domain and automate discovery of which could services a target is using. With AWS, you can enumerate the data center or region of the target.
- Scope Definition or Verification: In a pentest, the customer provides a cloud provider or set of services that are in scope. The tool can verify based on domain or IP addresses which IP addresses are in scope, or even find out incorrect information on owned IP addresses based on cloud provider.
- Investigations and Forensics: An investigator can lookup an IP address that uses cloud resources and find out which cloud provider, region, and service the suspected incident originated from. For example, adversaries are increasingly using US cloud providers to evade detection (i.e., SolarWinds security incident) and this tool can perform attribution to aid investigators.
- Detection Engineering: A blue team can instrument their SIEM with the tool and enrich their logs to correlate suspected thresholds of attack traffic originating from cloud services. For example, detecting when their Web Application is getting brute forced or password sprayed using a cloud service (i.e., Amazon API Gateway). The SIEM or WAF can be instrumented to detect and correlate this activity and block.
- Detecting Shadow Cloud: A Cloud Security Engineer or Architect can use the tool to help asset management and detect unauthorized cloud services, or be able to “see” what an adversary can see from a black box scenario. For example, start the tool with a seed value of your domain, you detect some S3 buckets that were not authorized.
Examples
Edge always attempts to load the cloud provider JSON files into memory. The magic happens when you pass the -prefix flag, which tells the tool to lookup any IP address against the providers.
Example 1: This example shows running edge in single IP lookup mode. This does cloud attribution lookup against a single IP address and tells you the cloud provider, region, and service if applicable:
edge -single 140.179.144.130 -nd
#1 shows us skipping automatic download of the provider files and using local files, improving performance. Once you have downloaded the files (enabled by default), it isn’t necessary to do this every time.
#2 shows how many AWS prefixes are loaded into memory.
#3 shows matching the IP address to the cloud provider (AWS) and the IP prefix.
#4 shows matching to the service (API Gateway) and region (cn-north-1).
#5 shows the default csv output format. This can then be parsed to use with other security tools.
Example 2: This example shows running edge against a list of IP addresses, enabling performant cloud attribution lookups at scale. The ip address file (ip-hosts.txt) has a single IP address in each line of the file.
edge -ip ip-hosts.txt -prefix -nd
#1 shows detection of an Azure storage service.
#2 shows detection of an AWS EC2 instance running in us-east-2 region.
#3 shows detection of an AWS S3 bucket IP address running in us-east-2 region.
#4 shows detection of a Google Cloud IP address.
Example 3: This example shows classic subdomain enumeration using certificate transparency with crt.sh and DNS. If DNSRecon and GoBuster got together and had a baby, it would be edge. Edge takes the performance of gobuster with the certificate transparency of DNSRecon. The example starts with a certificate transparency query to crt.sh. For each record found, it does a DNS lookup and follows all A and CNAME records. With the prefix directive, all IP addresses are looked up against the providers.
edge -prefix -crt -dns -domain tesla.com -nd
#1 shows enumerating a host via crt.sh. This host will be passed into a DNS lookup and all DNS A and CNAME records will be followed and resolved. You can output to a CSV file and then map out this host further for A and CNAME records.
#2 shows that 340 DNS lookups were performed.
#3 shows 134 hosts were discovered via crt.sh.
#4 shows an IP address was matched to AWS using CloudFront. It’s pretty fascinating to see it automatically resolve and discover all load balancers and CDN hosting and all of the backends hosting SaaS platforms. It can help you learn about how an application is hosted with different cloud services. This shows how you can take a fronted domain like tesla.com and enumerate all of the backend cloud and services used if it includes one of the three US cloud providers.
Project Discovery
There are a lot of useful recon frameworks out there but one of my favorites are the Project Discovery tools. You can use cloud edge together with Project’s Discovery’s tlsx to automate some effective workflows. Let’s step through an example.
The Scenario: Client Pentest: Start with a single IP
Thanks to Josh Wright for providing useful recon information for cloud scanning. I’ve borrowed some of these techniques from his excellent workshop.
In an authorized pentest, your client gives you a list of IP addresses that are “in-scope” instead of a domain. For this learning lesson, I’m picking a single, random IP address out of Amazon’s EC2 services in the region us-east-1. For the story, let’s say that this IP address is one of the IP addresses provided out of the list. That IP address is 35.171.217.10.
Step 1: Run Edge to enumerate cloud and service
Run edge to enumerate a cloud provider and service:
./edge -single 35.171.217.10 -nd
[INF] Single IP prefix lookup of 35.171.217.10
[INF] Matched IP [35.171.217.10] to Cloud Provider via prefix [AWS:35.168.0.0/13]
[INF] Matched IP [35.171.217.10] to Cloud Service [EC2] and Region [us-east-1]
35.171.217.10,Provider:AWS;Prefix:35.168.0.0/13;Region:us-east-1;Service:EC2
From this run we enumerate the IP address (35.171.217.10) is an EC2 system in us-east-1 region. With this information we will get all domains and systems used by this customer using TLS certificate information. I’ll wrap up this quick example showing how we can use masscan together with edge and tlsx to do cloud attribution.
Step 2: Stand up an EC2 instance in same region
In his webcast Josh explains how setting up an instance in the same cloud provider’s region to do scanning against your client is using the “power of the cloud” to carry out security testing. First, it will yield better performance when scanning from within the same physical data center of the cloud provider. Second, researchers are discovering that some systems allow more permissive firewall rules when the scanning is sourced from the same cloud provider (or region). So you might enumerate more open ports (and help your client mitigate risk) for your client with this type of scanning. For this example, I stand up an EC2 instance in the same region (us-east-1) to perform more effective scanning. I’ve set up an Ubuntu 22.04 with t3.medium instance size which can be terminated as soon as I’m done but offers pretty decent performance. You can follow the information on Masscan’s website for benchmarking performance (I’ve determined that a rate of 400,000 pps is going to work well for this instance).
Step 3: Get all IP addresses in this Amazon region and service
Using jq and url, we can pull down all IP prefixes in the us-east-1 region and prepare our scanning box for the next step. These two commands will download the prefix list and then extract and sort a nice list of prefixes for us-east-1 EC2 instances:
wget https://ip-ranges.amazonaws.com/ip-ranges.json
jq -r '.prefixes[] | select(.service=="EC2") | select(.region=="us-east-1") | .ip_prefix' < ip-ranges.json | sort -u > us-east-1.txt
Now we have a file called us-east-1.txt containing all of the IP prefixes for all EC2 instances in the us-east-1 region. In my my most recent testing there were 180 prefixes included in this list.
Step 4: Scan using Masscan and Tlsx
First, run masscan to find all open TCP 443 ports in the list of prefixes:
sudo masscan -iL us-east-1.txt -oL us-east-1-range.masscan -p 443 --rate 400000
Masscan is one of the best tools for the objective of speed. I benchmarked the best rate of 400,000 pps on this t3.medium instance size. We output all enumerated ports to a file. This scan completes in roughly one minute. That’s fast.
Second, extract out all IP address / hosts enumerated as listening on TCP port 443 to an output file called ports-443.txt.
awk '/open/ {print $4}' us-east-1-range.masscan > ports-443.txt
Third, use tlsx to probe for the TLS certificate and extract the Subject Alternative Name (SAN) field from the certificate. This is where the magic happens.
cat ports-443.txt | tlsx -san -o output.txt
tlsx is a very fast TLS prober that “grabs” and validates a TLS certificate. It performs a TLS handshake and then can extract specific fields from the certificate. In our example, we extract the subject alternative name (SAN). The SAN contains the fully qualified DNS entry, which can then be attributed or mapped back to our client in the pentest. We output the results to a file for later analysis.
This technique is so powerful, fast, and fun it is mind-blowing. Tlsx has flexible support for stdin and many different options. It can take DNS host or IP address as input. It can be chained with other tools to create a powerful asset discovery pipeline. This method can be automated to run across all AWS regions and data centers, enabling searching for cloud assets across your clients using the same black box techniques an adversary or bug bounty hunter will use.
Step 5: Analyze results
Finally, search for that IP address in the output.txt file to map this back to your client.
So we find that the IP address can be attributed to the domain splunkcloud.com based on the SAN field of the TLS certificate. Nice analysis and very fast. Then you can search the output.txt file for other keywords or domains owned by the client. Or do an investigation to find other IP addresses that the client didn’t include in the list but still owned by that client.
Video
Here is a demonstration video of running cloud edge against telsa.com using Certificate Transparency with DNS wordlist subdomain enumeration. For any host entries found, edge has the -prefix flag enabled so it will do a lookup of the IP against the cloud provider JSON files.
Video here.
Conclusion
Thanks to Josh Wright for his research and workshops on cloud scanning. Thanks to Moses Frost for tool inspiration for cloud edge and all of the techniques and methods he devised in his Cloud Penetration Testing class.