Install and Test Atomic Red in Kubernetes

In cloud-native environments, where applications scale up and down much faster than traditional monolithic application architectures, the ability to proactively identify and respond to threats in real time is paramount. As more organizations embrace cloud-native architectures for application delivery, more robust security measures need to be introduced. In this blog post, we delve into the dynamic realm of Kubernetes threat detection by exploring how open source Falco can seamlessly detect Atomic Red Team tests in real time within Kubernetes environments.

Atomic Red Team is a powerful framework designed to simulate real-world attacks, providing organizations with a controlled environment to validate the effectiveness of their security measures. We take this a step further by deploying Atomic Red to Kubernetes with a single command, creating a realistic testing ground for evaluating the responsiveness of Falco.

Our journey begins with the effortless deployment of Atomic Red to Kubernetes, showcasing the simplicity and efficiency of orchestrating security testing within containerized environments. Once deployed, we invoke specific Atomic Red Team tests, simulating a range of threat scenarios. The true test lies in Falco's ability to detect these threats in line with the MITRE ATT&CK framework, a globally-recognized matrix mapping adversary techniques to defensive tactics.

This exploration is not just about identifying threats; it's a collaborative effort to enhance Falco's coverage. Should we identify any gaps in detection, we dive deeper, revising the executed techniques and crafting custom rules. This iterative process aims to extend our MITRE ATT&CK coverage, aligning Falco with the industry's best practices for threat detection and mitigation.

Deploying Atomic Red Team

To avoid any potential service disruption in production environments, we recommend installing Atomic Red in a test lab environment, or at least a staging environment of Kubernetes. We have a step-by-step video for installing Atomic Red on Youtube

Before we start the deployment, remember to create the atomic-red network namespace.

kubectl create ns atomic-red

A single pod will be deployed with privileged set to true. Atomic Red requires admin-level securityContext to perform certain actions that require elevated permissions.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: atomicred
  namespace: atomic-red
  labels:
    app: atomicred
spec:
  replicas: 1
  selector:
    matchLabels:
      app: atomicred
  template:
    metadata:
      labels:
        app: atomicred
    spec:
      containers:
      - name: atomicred
        image: issif/atomic-red:latest
        imagePullPolicy: "IfNotPresent"
        command: ["sleep", "3560d"]
        securityContext:
          privileged: true
      nodeSelector:
        kubernetes.io/os: linux
EOF

Note: This creates a pod called 'atomicred' in the 'atomic-red' network namespace. You can check on the state of the installation with the below command:

kubectl get pods -n atomic-red

If you ever want to remove the Atomic Red project from your Kubernetes cluster, simply run:

kubectl delete deployment atomicred -n atomic-red

Familiarize Yourself with Atomic Red Tests

Once deployed, you will need to shell into the Atomic Red pod to perform the following test scenarios. This might seem a little confusing, but Atomic Red was developed with PowerShell in mind, so the below instructions ask the user to shell into a container, and once they are in the running pod, they must run Powershell to import and invoke the various Atomic Test Scenarios.

Once you are familiar with this logic, you’ll find Atomic Red is a truly simple security simulation tool.

kubectl exec -it -n atomic-red deploy/atomicred -- bash

As mentioned, you need to run Powershell once you are in the Atomic Red pod:

pwsh

Now, you can finally load the Atomic Red Team module:

Import-Module "~/AtomicRedTeam/invoke-atomicredteam/Invoke-AtomicRedTeam.psd1" -Force

Check the details of the TTPs:

Invoke-AtomicTest T1070.004 -ShowDetails

Check the prerequisites to ensure the test conditions are right:

Invoke-AtomicTest T1070.004 -GetPreReqs

You can see in the below screenshot, there are no prerequisites required to perform these tests. As a result, we can invoke the bulk file deletion test scenario.

Remove the feature flags to execute the test simulation.

Invoke-AtomicTest T1070.004

This test will attempt to delete individual files or individual directories. When we have Falco installed, this Atomic test should trigger the 'Warning bulk data removed from disk' rule by default. Next, we discuss Falco’s installation.

Congrats! Now that we know how Atomic Red works, let’s install Falco and run it side-by-side against Atomic Red to prove it detects these tests in real time. We will need to open two terminal windows to see the real-time response detections.

Installing & Testing Falco

For this lab guide, we can install Falco via Helm on a fixed version prior to the segregation of rules into different rules feeds, such as 'incubating', 'sandbox' and 'stable'. The reason we are doing this is to ensure all Falco rules are accessible in our lab scenario. To use the latest version of Falco, simply remove the '--version' feature flag from the Helm install script.

helm install falco falcosecurity/falco --namespace falco \
  --create-namespace \
  --set tty=true \
  --version 3.3.0

Just like the Atomic Red deployment, we need to monitor the progress of the Falco installation. The pods will change state a few times during the installation, but should eventually all be in a 'RUNNING' status after about a minute or so.

Please use the below command to check on the status change of Falco pods:

kubectl get pods -n falco -w

Once Falco is installed, we can track the events as they are generated using the following command in the second terminal window.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco

Jump back into the first terminal window and re-run the bulk file deletion Atomic Test - 'T1070.004':

Invoke-AtomicTest T1070.004

You’re going to identify certain noise in the detection rules. For example, all Atomic Tests are run under the 'Root' user, therefore, we will always get a detection for scripts running under root. To ignore this noise, let’s instead just check for the specific Falco rule we are looking to detect:

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Bulk data has been removed from disk'

Hurrah! We see the exact detection matching the context of the Atomic Red test scenario. Let’s move on to the next Atomic Test to invoke. There are a bunch of test scenarios for Linux that you can test out today.

Check out the list on the official Atomic Red Team Github project.

T1556.003 Modify Authentication Process

In this scenario, Atomic Red generates three Pluggable Authentication Modules (PAM): two malicious PAM rules for Linux and FreeBSD, as well as a malicious PAM module for Linux. These programs can be used to open and read sensitive file content, and we can agree that they are non-trusted programs. Again, we have an out-of-the-box rule for these activities:

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Sensitive file opened for reading by non-trusted program'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1556.003

T1036.005 Masquerading: Match Legitimate Name or Location

This test scenario executes a process from a directory masquerading as the current parent directory.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Executing binary not part of base'

Now, it's time to simulate our threat.

Invoke-AtomicTest T1036.005

You can see that in the left terminal window, there is an echo message in the terminal saying '”Hello from the Atomic Red Team.”' Any string output in the command line can also be detected in Falco’s outputs.

T1070.002 Indicator Removal on Host

Adversaries may clear system logs to hide evidence of an intrusion. macOS and Linux both keep track of system or user-initiated actions via system logs. The majority of native system logging is stored under the '/var/log/' directory.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Log files were tampered'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1070.002

T1070.003 Clear Command History

In addition to clearing system logs, an adversary may clear the command history of a compromised account to conceal the actions undertaken during an intrusion. Various command interpreters keep track of the commands users type in their terminals so that users can retrace what they've done.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Shell history had been deleted or renamed'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1070.003

You can see from the below screenshot that four different operations were performed. Therefore, four unique Falco detections were triggered on those individual attempts to clear the command line history.

T1014 Loadable Kernel Module Based Rootkit

Adversaries may use Rootkits to hide the presence of programs, files, network connections, services, drivers, and other system components. Rootkits are programs that hide the existence of malware by intercepting/hooking and modifying operating system API calls that supply system information.

Rootkits may reside at the user or kernel level in the operating system or lower, to include a hypervisor, Master Boot Record, or System Firmware. As such, it’s critical that Falco detects Rootkits in real time.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Linux Kernel Module injection from container detected'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1014

Falco is detecting the Linux kernel module injection attempt, whether it was a successful execution or not.

T1037.004 [CUSTOM RULE] Boot Initialization - RC Scripts

Adversaries may establish persistence by modifying RC scripts that are executed during a Unix-like system’s startup. These files allow system administrators to map and start custom services at startup for different run levels. RC scripts require root privileges to modify.

Command to simulate the 'T1037.004' test:

Invoke-AtomicTest T1037.004

You’ll notice that this is the first test where we don’t get a useful Falco detection related to the threat. As a result, we need to create a 'custom-rules.yaml' file with the custom Falco rule for detecting boot initialization scripts.

cat > custom-rules.yaml <<EOF
customRules:
  custom_rules.yaml: |-
    - rule: Base64-encoded Python Script Execution
      desc: >
        This rule detects base64-encoded Python scripts on command line arguments.
      condition: >
        spawned_process and (
          ((proc.cmdline contains "python -c" or proc.cmdline contains "python3 -c" or proc.cmdline contains "python2 -c") and (proc.cmdline contains "echo" or proc.cmdline icontains "base64")) or ((proc.cmdline contains "import" and proc.cmdline contains "base64" and proc.cmdline contains "decode")))
      output: >
        Potentially malicious Python script encoded on command line
        (proc.cmdline=%proc.cmdline user.name=%user.name proc.name=%proc.name
        proc.pname=%proc.pname evt.type=%evt.type gparent=%proc.aname[2]
        ggparent=%proc.aname[3] gggparent=%proc.aname[4] evt.res=%evt.res
        container.id=%container.id container.name=%container.name file=%fd.name)
      priority: warning
      tags:
        - T1037.004
        - MITRE_Defense_Evasion
      source: syscall
EOF

Adversaries can establish persistence by adding a malicious binary path or shell commands to 'rc.local', 'rc.common', and other RC scripts specific to the Unix-like distribution. Upon reboot, the system executes the script's contents as root, resulting in persistence.

Let’s try upgrading Falco to reflect the 'custom-rules.yaml' file:

helm upgrade falco falcosecurity/falco \
  -n falco \
  --set tty=true \
  --version 3.3.0 \
  --reuse-values \
  -f custom-rules.yaml

Granted, there’s no obvious formatting issues when creating the 'custom-rules.yaml' manifest, you should be able to successfully upgrade Falco and the pods should be in a 'RUNNING' state. If there was an issue with the custom rules file, the Falco pod state will likely change to 'CrashLoopBackOff'.

Let’s see if we detect the Atomic test after upgrading Falco with the newly-created custom rule. Remember to have Falco running in a second terminal window with the following command:

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Potentially malicious Python script'

We can simulate the technique ID 'T1037.004' one last time:

Invoke-AtomicTest T1037.004

Hurrah! We detected the boot initialization scripts with the above command. To recap, adversaries looking to abuse those RC scripts are especially effective for lightweight Unix-like distributions using the root user as default, such as IoT or embedded systems. If you are wondering why the Falco rule was specifically looking at Base64-encoded Python scripts, well, we need to look back at the details associated with the Atomic test simulation.

Invoke-AtomicTest T1037.004 -ShowDetails

We can see from the command that it is using the 'python3' command to run Python scripts. However, the script itself is executed as a base64-encoded string to evade some traditional detection tools.

Conclusion

In conclusion, this article serves as a comprehensive guide aimed at enhancing knowledge in cloud-native threat simulation and detection. Its primary audience includes security practitioners, DevOps teams, and anyone involved in securing Kubernetes environments. The focal point is the utilization of Atomic Red for deploying test simulations aligned with the MITRE ATT&CK framework and leveraging Falco to detect these threats in real time.

The article not only illustrates how Atomic Red can be instrumental in improving the rules shipped with Falco, but also emphasizes its role in identifying potential gaps in coverage. By doing so, it offers a valuable resource for customers seeking alternative methods to validate the effectiveness of their Falco rules and, consequently, fortify the security posture of their Kubernetes deployments.

The notion of creating a realistic testing ground for evaluating the responsiveness of Falco is particularly beneficial for DevOps teams. This facet addresses the need for hands-on experience in threat simulation, adversarial approaches, and understanding the intricacies of the MITRE ATT&CK framework. In providing this practical testing environment, the article empowers DevOps teams to proactively enhance their security measures, ensuring robust coverage and responsiveness in the dynamic landscape of Kubernetes threat detection.

• ←Previous
Next→