All those (vpc) flow logs… Consolidate vpc flow logs using BigQuery

person on watercraft near waterfall — Photo by Jacob Colvin on Pexels.com

VPC FlowLogs

VPC Flow Logs records a sample of network flow. Logs can be used for network monitoring, forensics, real-time security analysis, and expense optimization.

Once the vpc_flows are enabled on all subnets of interest, we can go to Logging to check if we see logs arriving:

Sample below:

[
  {
    "insertId": "1gzzhw9g1wrk28h",
    "jsonPayload": {
      "src_vpc": {
        "subnetwork_name": "network001",
        "vpc_name": "vpc001",
        "project_id": "rtrentin-01"
      },
      "src_instance": {
        "vm_name": "ce-vpc001",
        "project_id": "rtrentin-01",
        "zone": "us-east1-b",
        "region": "us-east1"
      },
      "start_time": "2023-01-21T23:22:17.807967667Z",
      "reporter": "SRC",
      "packets_sent": "64",
      "end_time": "2023-01-21T23:22:17.807967667Z",
      "bytes_sent": "0",
      "connection": {
        "src_ip": "10.11.64.2",
        "protocol": 6,
        "dest_ip": "173.194.217.95",
        "dest_port": 443,
        "src_port": 55106
      }
    },
    "resource": {
      "type": "gce_subnetwork",
      "labels": {
        "subnetwork_id": "7135252347660278790",
        "subnetwork_name": "network001",
        "project_id": "rtrentin-01",
        "location": "us-east1-b"
      }
    },
    "timestamp": "2023-01-21T23:22:42.256060936Z",
    "logName": "projects/rtrentin-01/logs/compute.googleapis.com%2Fvpc_flows",
    "receiveTimestamp": "2023-01-21T23:22:42.256060936Z"
  }
]

BigQuery

Cloud Logging “routes” logs to destinations like buckets, BigQuery or to Pub/Sub:

https://cloud.google.com/logging/docs/routing/overview

We want to consolidate all logs in a centralized location where we can consume the data ingest. We can use BigQuery to accomplish that creating a sink. Sinks control how Cloud Logging routes logs:

We can also set the table to expire after certain number of days:

We want to filter vpc_flows from all logs:

Click preview to validate the inclusion filter:

Checking the sink destination:

A simple BigQuery to show the possibilities we have now:

Another simple example:

Visualization

We can use Looker Studio to explore and visualize the data:

Looker Studio is a free, self-service business intelligence platform that lets users build and consume data visualizations, dashboards, and reports. With Looker Studio, you can connect to your data, create visualizations, and share your insights with others.