From Shellshock to SIEM - Part 3: The Watchtower

Deploying Wazuh SIEM to Catch Containerised Attacks in Real-Time

Part 1 - Building a Vulnerable Docker Container System to Demonstrate Common Mistakes Made During IT Administration.

Part 2 - Full Penetration Test of the Vulnerable System from Initial Access to Privilege Escalation.

Part 3 - Using Wazuh SIEM to Monitor for and Alert on Ongoing Cyber Attacks.

TL;DR: If You Can't See It, You Can't Stop It

This is the 3rd and final part of the From Shellshock to SIEM 3-part lab series. This post covers the defensive side: setting up a 'Security Camera' (Wazuh - SIEM) to monitor our lab Docker container and detect the two previous attacks: SSH credential brute-forcing and Shellshock.

🔦 A cyber threat actor only needs to be right once; cyber defenders, on the other hand, need to be right all the time.

This lab will follow the (1) Preparation (& prevention), (2) Detection/Monitoring, (3)Triage & Analysis, and (5) Eradication phases of the Security Operations Centre (SOC) lifecycle. The lab will skip the (4) Containment, (6) Recovery, and (7) Post-Incident Review & Improvement phases.

Key Takeaways

Visibility Gaps: By default, many systems (including Docker containers) do not send their logs to a central place where security teams can see them. After several attempts, I managed to reconfigure the Docker container's 'logging driver' to ensure the attacks were visible.
The "Silent" Threat: Sophisticated attacks like Shellshock can often hide in normal-looking web traffic. This project highlights that even with a world-class SIEM, you must constantly tune your log generation, decoders, and rules to catch modern, file-less exploits that don't trigger standard alarms.
The Value of Automated Alerts: Wazuh successfully flagged the SSH brute force attack because it recognised the pattern of multiple failed logins followed by a success (Rule 40112).

Phase 1: Preparation - SIEM Deployment

This is the 'Get ready before anything happens' phase, which involves confirming your assets (servers, endpoints, cloud, containers), log source onboarding (firewalls, EDR, IAM, etc.), SIEM configuration (rules, parsers/decoders), baseline setting, playbook & runbook creation, and threat modelling. This lab is simple - there is only one vulnerable server/container we need to monitor, and we already know the attacks targeting the system.

Wazuh SIEM Deployment

Wazuh is an open-source security monitoring platform that provides Extended Detection and Response (XDR) and SIEM (Security Information & Event Management) functionality.

I will not cover the various options for setting up Wazuh as a SIEM; there are several resources online on how to do this, including Wazuh's official deployment guides. This lab utilised the single-node Wazuh Docker option to set up the Wazuh 'Server' (Manager, Indexer, and Dashboard), as three linked, but separate containers.

Wazuh Agent Deployment

The Wazuh Agent is the 'boots on the ground' and is installed on the Ubuntu VM host that runs the Docker vulnerable server/container. The agent, once configured, will monitor both the Ubuntu host VM and any Docker containers running within this VM. Once the Wazuh Server is up and running, log in with the default credentials (admin:SecretPassword), and follow the instructions to deploy the Agent on the Ubuntu VM.

💭 IDEA: I did attempt installing the Wazuh agent (amending the Docker file and direct install via the CLI) within the vulnerable container; however, this created more problems, and is not ideal in a production Docker environment.

Once the Wazuh Agent is installed, give it a few minutes to pass back telemetry to the Server including the CIS Security Configuration Assement data (see image below).

Phase 2: Detection & Monitoring (Solving the Visibility Gap)

The SOC Detection/Monitoring phase involves ensuring the data pipeline (log onboarding) is active and the 'sensors' are tuned to catch the attack vectors we identified in Part 2. By default, Wazuh already has SSH Brute force and Shellshock detection configured.

Wazuh Agent Configuration

Stop and delete any running containers (on the Ubuntu VM): docker stop $(docker ps -aq) && docker rm $(docker ps -aq) (CAUTION: this command shortcut will delete all containers) & delete the respective Docker image: docker rmi <image-id>.

Wazuh includes a Docker module, allowing the agent to interact with the Docker API and monitor container events (start, stop, etc). To enable this module, edit the /var/ossec/etc/ossec.conf file (make a backup copy first) on the Ubuntu VM (this will need to be done as root). Look for the '<wodle name="docker-listener">' entry (or add the following) and ensure the setting is set to 'no'. Do not close the file yet!

By default, the Wazuh agent cannot peer into the Docker container to view SSH login attempts, Apache Logs, or other interesting tidbits. This is because standard Docker containers log to stdout (json-file), which are often isolated from the host's system logs. To ensure Wazuh can 'see' the SSH failures inside the container, I reconfigured the container's logging driver by sending logs to journald, and set the 'tag' sshd. The Ubuntu VM Wazuh Agent can now ingest/forward them just like any other local system log.

Similarly, the container's Apache logs are isolated. Two options - (1) install the Wazuh Agent within the container (not recommended) or (2) mount the container's Apache log directory to the host using a volume (recommended) when spinning up the containers: `-v /var/log/apache2:/var/log/apache2_container:ro`.

After several experiments (destroyed VMs and containers), adding the following entries to the /var/ossec/etc/ossec.conf file allows the agent to retrieve and forward (SSH & Apache) events to the Wazuh Server.

Save the /var/ossec/etc/ossec.conf file and restart the Wazuh Agent: /var/ossec/bin/wazuh-control restart.

Wazuh Server/Manager

It is possible to skip modifying the Wazuh manager once the above Agent configurations have been made. But, in case you need to troubleshoot any attack event detection and display issues, follow these steps (and the steps below: Troubleshooting - Wazuh Ingesting Docker Logs).

View all the running containers: docker ps -a to display all the containers, and note down the Container ID of the Wazuh Manager.
Access the terminal of the Wazuh Manager container: docker exec -ti <container-id> /bin/bash
Docker containers are meant to be lean and will not include a text editor (nano, vim, etc); install one within the container.
Make a backup copy of the /var/ossec/etc/ossec.conf file, e.g., cp /var/ossec/etc/ossec.conf /var/ossec/etc/ossec.conf.backup
Edit the file and ensure the 'logall' & 'logall_json' fields are set to 'yes', and save the file.
Restart the Wazuh Manager: /var/ossec/bin/wazuh-control restart.

⚠️ WARNING: Once you confirm Wazuh can detect the SSH and Shellshock attacks, remember to set <logall>no</logall> in the manager's ossec.conf and restart it. Leaving it on yes can quickly fill up your disk space in a production environment.

Docker Container

After configuring the Wazuh Agent (and Manager), rebuild the vulnerable Docker image and launch the container with two key elements:

Run the container with journald set as the main log driver (SSH event monitoring): --log-driver=journald
Mount the container's Apache log directory to the Ubuntu VM host using a volume: -v /var/log/apache2:/var/log/apache2_container:ro

Execute the following commands to launch the new vulnerable Docker container:

docker build -t ssh-final .
docker run -d --name ssh-container-v2 --log-driver=journald --log-opt tag="sshd" -p 2223:22 -p 8080:8080 -v /var/log/apache2_host:/var/log/apache2 ssh-final

Phase 3: Triage & Analysis (Alert Verification)

During this phase, security teams (SOC analysts) will investigate alerts to determine if they are 'True Positives.'

Analysing SSH Brute Force Attacks

With the above configuration modifications, I managed to get Wazuh to detect and display the SSH brute force attacks from the KALI Linux VM. The default Wazuh Rules were effective in detecting and generating alerts for this particular attack without any modification, and within the Wazuh Dashboard, a sequence of Rule IDs began to light up:

Rule ID 5710 - sshd: Attempt to log in using a non-existent user.
Rule ID 5760 - sshd: authentication failed
Rule ID 2501 - syslog: User authentication failure.
Rule ID 40112 - Multiple authentication failures followed by a success.

Rule ID 5710 - sshd: Attempt to login using a non-existent user; Notice the time difference between each attempted login using the custom Python script.

Rule IDs 2501, 5760, and 40112 - related to the SSH brute force attack. Note there are two successful SSH logins because the 'test' and 'root' accounts, were both active with weak passwords, for this particular testing run.

Successful SSH Brute Force login details - Target & Attack IP addresses and compromised user account.

Analysing Shellshock Attack

This was by far the most difficult attack to forward from the container to the Wazuh manager for the following reasons:

In some Apache setups, headers like Cookies or User-Agents are not always logged by default and will thus not appear within the Apache access.log file being parsed by Wazuh decoders.
The Wazuh Rules only detected the attack after the 'attacker' terminated the reverse shell, not at the point of successful execution.

However, after several trials, I managed to get the event parsed, matched, and an alert generated on the Wazuh Dashboard - Rule ID 31166.

Rule ID 31166 - Shellshock attack attempt.

✏️ TIP: If the Shellshock attack does not seem to be parsed (& detected), attempt modifying the Apache configuration (/etc/apache2/apache2.conf) file LogFormat to actually 'log' both '%{Cookie}i' or '%{User-Agent}i', within the container. Add the following line: LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{Cookie}i\"" combined.

Once edited, both the Cookie and User-Agent headers should be logged, and hopefully passed to the Wazuh manager for rule-matching and alert generation.

Apache logs both the Cookie and User-Agent headers enabling detection of the Shellshock signature characters - () { :;}

Triage & Analysis Summary

Further analysis of the indexed logs within Wazuh reveals the date.sh script was used to launch a bash terminal session for the User ID 'www-data'.

Based on the Indicators of Compromise and Attack (IOC & IOA), a SOC analyst can confirm that an attack took place and the system was breached. In their documentation, they would forward the following to the Incident Responder. Based on the

Attacker IP: 192.168.1.16
Target IP & System: 192.168.1.7 / Ubuntu VM (possibly VM itself or Docker containers that use the VM IP address)
Attacks Performed: SSH-Brute Force & Shellshock
Potentially Compromised Accounts: test & www-data
Rule IDs and MITRE ATT&CK Mapping :
- 5710 - sshd: Attempt to login using a non-existent user.
  - MITRE Techniques (IDs):
    - T1110.001 – Password Guessing: https://attack.mitre.org/techniques/T1110/001/
    - T1021.004 – SSH (Remote Services): https://attack.mitre.org/techniques/T1021/004/
    - T1078 – Valid Accounts: https://attack.mitre.org/techniques/T1078/
  - MITRE Associated Tactics: Credential Access, Lateral Movement, Defence Evasion, Persistence, Privilege Escalation, Initial Access
- 5760 - sshd: authentication failed
  - MITRE Techniques (IDs):
    - T1110.001 – Password Guessing: https://attack.mitre.org/techniques/T1110/001/
    - T1021.004 – SSH: https://attack.mitre.org/techniques/T1021/004/
  - MITRE Associated Tactics: Credential Access, Lateral Movement
- 2501 - syslog: User authentication failure.
  - MITRE Techniques (IDs) & Tactics: No specific MITRE IDs, and is often mapped conceptually to T1110 (Brute Force / Password Guessing)
- 40112 - Multiple authentication failures followed by a success.
  - MITRE Techniques (IDs):
    - T1078 – Valid Accounts (Successful authentication): https://attack.mitre.org/techniques/T1078/
    - T1110 – Brute Force (Credential Access): https://attack.mitre.org/techniques/T1110/
  - MITRE Associated Tactics: Credential Access
- 31166 - Shellshock attack attempt.
  - MITRE Techniques (IDs):
    - T1068 – Exploitation for Privilege Escalation: https://attack.mitre.org/techniques/T1068/
    - T1190 – Exploit Public-Facing Application: https://attack.mitre.org/techniques/T1190/
  - MITRE Associated Tactics: Privilege Escalation, Initial Access

Phase 5: Eradication (Fixing the Vulnerabilities)

This phase of the SOC lifecycle involves removing the 'root cause' of the alerts to ensure the attacker cannot return. Because we intentionally built the Docker container to be vulnerable, defining the required fixes is straightforward.

Software Update: Update the vulnerable version of Bash 4.0 with the latest patched version from the Ubuntu repositories.
Credential Hardening: Change any weak credentials (and disable root remote access), or disable password authentication in sshd_config, enforcing SSH Key-based access only (recommended).
Sudo Policy Review: No user should be allowed to run commands, utilities, or applications with root privileges (especially without a password).
Configuration Lockdown: Disable the cgid module in Apache if CGI scripts are not strictly required for business/production operations.

💡TIPS:

A useful tool used by penetration testers and security teams to evaluate 'low-lying' vulnerabilities within Linux systems is LinPeas; when executed on a system, it identifies the potential Privilege Escalation paths an attack may use, including Sudo policies and weak configurations.
In production environments, Wazuh would be configured to automatically respond and block brute force attacks with minimal input from security teams.
In this lab (and in production environments), systems will not detect and log enumeration scans from tools such as nmap, and thus Wazuh will not generate alerts. It is advised to integrate Wazuh with tools/solutions e.g., Antivirus, EDR (Endpoint Detection and Response), and IDS/IPS (Suricata).

Summary

The conclusion of this series exposes the most dangerous myth in cybersecurity: that installing a SIEM automatically equates to protection. As I highlighted with the Shellshock exploit, a SIEM is only as powerful as the telemetry it is engineered to understand.

The Preparation Foundation: True defense begins long before an alert triggers. Phase 1 of the SOC lifecycle is the most critical; it is the process of mapping your 'digital terrain.' Security teams must move beyond surface-level monitoring to deeply understand the location, format, and behavior of logs produced by critical assets. Without this granular knowledge, the data forwarded to Wazuh is just noise, not intelligence.

Closing the Visibility Gap: The SOC lifecycle is not a linear path, but a relentless feedback loop. The Post-Incident Review is not just a paperwork exercise; it is the fuel for the next cycle of improvement. By identifying what the SIEM was not telling us during the Shellshock attack, I was able to turn a 'silent breach' into a detectable event.

Troubleshooting Tips

The following are tips and tricks I used to troubleshoot the Wazuh section of this three-part lab. Hopefully, you will not face any issues and can ignore this section.

Deleting/Removing Agents

There is no method (that I am aware of) to delete Wazuh agents from the Wazuh Manager. To remove 'dead' agents (decommissioned VMs, endpoints, and servers), you will need to use the Wazuh Manager CLI. To access the Wazuh Manager CLI (if you are not using a Docker-Wazuh implementation, skip the first two steps):

View all the running containers: docker ps -a and note down the Container ID of the Wazuh Manager.
Access the terminal of the Wazuh Manager container: docker exec -ti <container-id> /bin/bash
Execute the following:/var/ossec/bin/manage_agents
Follow the on-screen instructions to view and delete the respective agent.

On the actual endpoint/host where the Wazuh Agent is installed, execute sudo apt remove --purge wazuh-agent (Linux).

Deleting Wazuh Agents from the Manager

Errors Connecting Agent to Server

After deployment, it is common for the Wazuh Agent not to connect to the Wazuh Manager, and thus not appear on the Dashboard GUI. Follow these steps to identify (and resolve) the issues:

On the Agent Host, check the status of the agent to confirm if all services are running - /var/ossec/bin/wazuh-control restart
On the Agent Host, check the state file to see if it is 'Pending' or 'Disconnected' - sudo grep ^status /var/ossec/var/run/wazuh-agentd.state
On the Agent Host, check the logs for specific error codes like 1216 (Connection Refused) or 1208 (Enrollment Failed) - sudo tail -n 50 /var/ossec/logs/ossec.log
On the Manager Host, check if the manager is actually seeing the connection attempt - docker logs <wazuh_manager_container_name> | grep -i "agent"
Allow ports 1514 and 1515 on both hosts of the manager and agent firewalls - sudo ufw allow 1514/tcp and sudo ufw allow 1515/tcp
On the Agent Host, use the agent-auth tool to manually register - sudo /var/ossec/bin/agent-auth -m <MANAGER_IP> (replace with actual Wazuh Manager IP address); useful if you encounter an error similar to the one below.
Restart the Wazuh agent to apply any changes - systemctl restart wazuh-agent or /var/ossec/bin/wazuh-control restart

Wazuh Ingesting & Testing Logs

As mentioned previously, the Wazuh agent cannot peer directly into a Docker container and view the respective logs. Instead, I reconfigured the container's logging driver, sending logs to journald. Once the container is launched and running, you can verify the Docker container logs on the host of the Wazuh agent using the following: sudo journalctl -u docker.service --identifier=sshd -f

During the Wazuh Agent deployment and configuration, I set the logging driver to journald, but did not correctly configure the correct log collection block in the Agent /var/ossec/etc/ossec.conf file. This resulted in the Wazuh Manager receiving logs similar to those highlighted in red in the screenshot below. However, after adding the correct log correction to the Agent configuration file (<filter field="SYSLOG_IDENTIFIER">^sshd$</filter>), the correct syslogs were generated and forwarded to the Wazuh Manager (logs highlighted in green).

Why does this work? I already confirmed that the Wazuh Agent is reading journald and sending it to the Manager; I just needed to ensure the Manager treats those specific journal entries as Syslog so the built-in SSH rules trigger automatically.

To explain this further, I used the Wazuh log testing utility available via CLI on the Manager: /var/ossec/bin/wazuh-logtest, and you can view the results for each log entry in the screenshots below.

In the first screenshot, there is no matching Wazuh decoder for the log entries in red above, and thus, the log cannot be parsed; this would need a custom decoder and rule to generate an alert.

In the second screenshot, Wazuh already has an in-built decoder for this standard syslog, and thus it can be parsed to a corresponding rule to generate an alert.

No matching Wazuh decoder found and therefore a custom decoder and rule would need to be created for this particular log entry

Wazuh decoder can parse the log file and a corresponding rule will flag the log with a severity level 5 causing an alert to be generated.

💡TIP: It is possible to use the syslog driver instead when launching a Docker container - docker run -d --name <my-container> --log-driver=syslog --log-opt syslog-address=udp://<syslog-server-ip:port> --log-opt tag="sshd" <image-name>.

Resources & References

Official Wazuh Documentation: https://documentation.wazuh.com/current/index.html
Docker Container Logging Driver: https://docs.docker.com/engine/logging/configure/
Docker Container Logging Tutorial for Beginners: https://daily.dev/blog/docker-container-logging-tutorial-for-beginners
Photo by Tasha Kostyuk on Unsplash

Page updated

Google Sites

Report abuse