Read Azure Logs with Python (NSG)
NSG flow logs are stored in a storage account in block blobs, and block blobs are made up of smaller blocks. Each log is a separate block blob that is generated every hour, and new logs are generated every hour, the logs are updated with new entries every few minutes with the latest data. there are few ways to read logs from the blob and one way is with Python.
You might have set your Azure Vnet, with some NSGs associated to the subnets. You start rolling apps, to the point where you have many VMs and many NSGs. In some situations, you need to make an application upgrade or installs a new application, but traffic is not flowing through. Which NSG is dropping traffic? Which TCP ports should be opened?
One possibility is using Traffic Analytics, a component of Azure Network Watcher, to get an idea of the traffic you have in your Vnet. Traffic Analytics is a two-step process:
- NSG logs are stored in a storage account
- NSG logs from the storage account are processed and made queriable
You can find more information about Azure Traffic Analytics here.
One of the most helpful features of Traffic Analytics in particular and Azure Log Analytics, in general, is being able to query logs with the KQL (Kusto Query Language). For example, you can use this query to find out the dropped flows in the last 3 hours for IP address 10.0.0.0:
AzureNetworkAnalytics_CL
| where TimeGenerated >= ago(1h)
| where SubType_s == “FlowLog”
| where DeniedInFlows_d > 0 or DeniedOutFlows_d > 0
| where SrcIP_s == “10.0.0.0”
| project NSGName=split(NSGList_s, “/”)[2],NSGRules_s,DeniedInFlows_d,DeniedOutFlows_d,SrcIP_s,DestIP_s,DestPort_d,L7Protocol_s
You can find a reference guide for the Kusto Query Language here.
However, you will notice that there is a time lag, and you will not find the very latest logs in Log Analytics. The original NSG Flow logs are stored in a storage account, in JSON format, so an option is getting those logs using the Azure Storage SDK.
That is exactly what the Python script in this Github repository does (Python SDK for storage installed). You can use different flags, like the –help option to get usage information:
$ python3 ./get_nsg_logs.py --help usage: get_nsg_logs.py [-h] [--accountName ACCOUNTNAME] [--displayLB] [--displayAllowed] [--displayDirection DISPLAYDIRECTION] [--displayHours DISPLAYHOURS] [--verbose] Get the latest flow logs in a storage account optional arguments: -h, --help show this help message and exit --accountName ACCOUNTNAME you need to supply an storage account name. You can get a list of your storage accounts with this command: az storage account list -o table --displayLB display or hide flows generated by the Azure LB (default: False) --displayAllowed display as well flows allowed by NSGs (default: False) --displayDirection DISPLAYDIRECTION display flows only in a specific direction. Can be in, out, or both (default in) --displayHours DISPLAYHOURS How many hours to look back (default: 1) --verbose run in verbose mode (default: False)
There is something you need to do before being able to access Azure Blob Storage: finding out the Azure Storage Account key.
The script will read it from the environment variable STORAGE_ACCOUNT_KEY. There are multiple ways of putting the storage account key into an environment variable, I like this command:
export STORAGE_ACCOUNT_KEY=$(az storage account keys list -n your_storage_account_name –query [0].value -o tsv)
Once you have your environment variable set, you can start using the script.
For example, in order to show dropped and allowed traffic with the flag –displayAllowed of ingress NSG logs stored in the storage account ernetworkhubdiag857, and excluding Azure LB probe traffic for the last 6 hours:
$ python3 ./get_nsg_logs.py –accountName “Azure Subscription” –displayHours 4 –displayDirection in –displayAllowed
2019-09-21T09:17:52.3055150Z NVA-NSG DefaultRule_AllowVnetInBound
2019-09-21T09:17:52.3055150Z NVA-NSG DefaultRule_AllowVnetInBound
2019-09-21T09:17:52.3055150Z NVA-NSG DefaultRule_AllowVnetInBound
2019-09-21T09:17:52.3055150Z NVA-NSG DefaultRule_AllowVnetInBound
2019-09-21T09:17:52.3055150Z NVA-NSG DefaultRule_AllowVnetInBound
2019-09-21T09:17:52.3055150Z NVA-NSG DefaultRule_AllowVnetInBound
Note:
- displayDirection- ingress is the default for the flag
- displayLB – that is the default for the flag
Then run the Azure NSG Logs script to read all logs
1 Response
[…] Read Azure NSG Logs With Python […]