Configuring host-level audit logging for AKS VMSS | MSRC Blog

This blog post runs you through how to enable and configure Linux audit logging on your Azure Kubernetes Service (AKS) Virtual Machine Scale Set (VMSS) using the Linux auditing subsystem, also known as auditd.

Unlike the Kubernetes control plane logs which give you visibility into your cluster’s high-level operation, enabling auditd logging on your cluster gives you visibility into your AKS worker node and container kernel level activity. This includes activity such as process creations, file access and command line activity which can be extremely valuable if you want to find unusual command line activity and programs executing on your cluster. Unlike container-level logging, this provides you with visibility into the workloads running on your underlying worker nodes, not just the containerized workloads running on top of them (including container escapes and other exploits). It is highly configurable and especially valuable if you are running a multi-tenant cluster.

To see examples of how Linux host-level audit logging (and Kubernetes control plane logging) can provide useful visibility into your cluster, check out the threat hunting queries we have provided in our Azure Kubernetes Service (AKS) threat hunting blog post and Jupyter notebook.

Host level audit logging takes advantage of a number of Linux utilities to capture audit logs and ship them to your Log Analytics Workspace for review.

The components used to accomplish this include:

  • auditd is the Linux audit daemon for the Linux auditing system. The audit events written to disk by the daemon are configured by rules defined in auditd.rules.
  • audisp daemon is the audit event multiplexer and can be used to distribute auditd event data to a data collector server like syslog.
  • The syslog daemon forwards logs to the Log Analytics agent for Linux (OMS). The facilities and severities of events forwarded by syslog can be configured through the Azure Portal or managing configuration files on the hosts.
  • auoms is the Microsoft audit collection tool which can be configured to collect and filter auditd/ audisp event data using the processing rules defined in the auoms configuration..
  • A Log Analytics Workspace is where you can explore telemetry from your Azure resources and services using the Kusto Query Language (KQL)

  1. Enable the Linux Operations Management Suite (OMS) Agent (auoms) on your AKS cluster VMSS by running the following

    az vmss extension set 
      --resource-group $RESOURCE_GROUP 
      --vmss-name $VMSS_NAME 
      --name OmsAgentForLinux 
      --publisher Microsoft.EnterpriseCloud.Monitoring 
      --protected-settings '"workspaceKey":"<KEY>"' 
      --settings '"workspaceId":"<ID>","skipDockerProviderInstall":true'
    • The vmss-name for your cluster will start with aks-agentpool and belong to the management resource group beginning with MC_, as shown in the screenshot below.

      Screenshot of the Azure Portal showing the contents of the managed resource group used by AKS, including the virtual machine scale set corresponding to your agent pool, in this case named aks-agentpool-21242312-vmss.

    • Both the workspaceId and workspaceKey can be found in the Agents Management tab within your Log Analytics Workspace, as shown in the below screenshot. The workspaceKey corresponds to the Primary Key highlighted below.

      Screenshot of the Azure Portal&rsquo;s Log Analytics Workspace Agents management blade showing that the workspace ID and primary key can be retrieved here, for inclusion in the deployment command.

  2. Trigger the VM update through the Azure Portal (select each instance and choose to upgrade).

    Screenshot of the Azure Portal&rsquo;s VMSS instances blade showing how one may select a virtual machine instance (or multiple instances) and trigger an upgrade by clicking the upgrade button.

  3. Install auditd on the VMSS worker nodes. To do this, you need to deploy or get a shell to a privileged pod running on your cluster that has access to the underlying worker node’s filesystem and process ID namespace. You can do this by:

    • Deploying a Pod with a specification that mounts a hostPath volume into the container environment.

    • Include the securityContext field in the Pod specification and enable privileged mode.

    • Configure host process ID namespace sharing by including the hostPID field in the Pod specification and
      enabling it.

      apiVersion: 1
      kind: Pod
        name: privileged-pod
        hostPID: true
          - name: priv-host-mount
            image: alpine:latest
              - /bin/sh
              - -c
              - -
              - while true; do sleep 30; done;
              privileged: true
              - name: privmount
                mountPath: /workernode
          - name: privmount
              path: /
              type: Directory
    • Use kubectl exec to get an interactive shell to the running container on the Pod, use chroot to emulate access to the
      underlying virtual machine and install auditd, as shown

      kubectl exec -it $POD /bin/sh
      > chroot /workernode
      > apt-get update && apt-get install auditd -y
  4. Configure the auditd rules to your liking:

    • Add a rule to monitor process executions. You can do this by creating a file called 10-procmon.rules at the following path: /etc/audit/rules.d/10-procmon.rules, and adding the content below. More detailed information on this configuration can be found here.

      -a exit,always -F arch=b64 -S execve -k procmon
      -a exit,always -F arch=b32 -S execve -k procmon
    • Update the auditd daemon by running systemctl restart auditd

  5. Update the syslog plugin facility settings to updated audisp to direct the auditd records to syslog. To do this, you need to update the configuration in /etc/audisp/plugins.d/syslog.conf and set the active field to yes as shown in the screenshot below.

    • By default, the facility enabled for syslog is the user facility. If you want to change this, you can change the second parameter in ARGS (the first parameter is the log level). For example, the following configuration changes the syslog Facility to authpriv.
    active = yes
    direction = out
    path = builtin_syslog
    type = builtin
    format = string
    • Restart the auditd daemon to apply the changes using systemctl restart auditd.
  6. Go to your Log Analytics Workspace in the portal and configure the logging agents to accept syslog events for the authpriv facility at the INFO level. To do this:

    • Navigate to Settings > Agents Configuration
    • Select Syslog
    • Select + Add facility
    • Choose authpriv in the dropdown menu, as shown below.

    A screenshot of the Log Analytics Workspace Agents configuration page, showing that the Syslog authpriv facility has been enabled at the info level.


Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *