SCC as a Data Lake: Leveraging BigQuery for Advanc...

andras · 04-23-2024 06:00 AM

Let's face it, Security Command Center (SCC) is a goldmine of security data. But sometimes, those shiny nuggets of insight need a bit of refining before they reveal their true value. That's where BigQuery comes in, transforming your SCC data into a powerful data lake for supercharged security analytics.

Why SCC + BigQuery = A Security Analyst's Dream

Long-term Retention for Comprehensive Investigations:

By exporting findings to BigQuery, security analysts gain a comprehensive repository of security events. This long-term retention capability enables thorough investigations across an extended timeframe, providing valuable context for incident response and threat analysis.

Unleashing the Power of Custom Queries:

SCC's built-in searches are robust, but sometimes you need more flexibility and customization. Enter BigQuery, where security analysts can unleash their inner SQL ninjas. With custom queries, analysts can explore the data with precision, honing in on specific patterns, indicators of compromise, and threat vectors.

Trend Analysis and Visualization for Informed Decision-Making:

BigQuery's massive data processing capabilities enable security analysts to perform trend analysis and visualization on large volumes of SCC data. This empowers them to identify emerging threats, recurring incidents, and seasonal patterns. Visualizing these insights through dashboards and reports helps decision-makers understand the security landscape and prioritize their response efforts.

Integration with SIEM and Other Security Tools:

BigQuery's open architecture enables seamless integration with SIEM (Security Information and Event Management) systems and other security tools. This integration allows security analysts to correlate data from SCC with other sources, providing a holistic view of the security landscape.

Scalability and Performance for Massive Datasets:

BigQuery is designed to handle massive datasets with lightning-fast performance, ensuring that security analysts can perform complex queries and analysis without encountering bottlenecks. This scalability is crucial for organizations dealing with high volumes of security data.

BigQuery’s integrated AI/ML capabilities:

BigQuery seamlessly integrates powerful AI/ML capabilities directly into your data warehouse. This eliminates the need to move data to specialized platforms. You can build and execute machine learning models using familiar SQL syntax, including models for forecasting, classification, and clustering to drive insights.

In conclusion, the combination of SCC and BigQuery offers security analysts a powerful toolkit to elevate their threat detection and investigation capabilities. By leveraging BigQuery's data lake capabilities, analysts can gain deeper insights, perform custom analysis, and identify emerging threats with unparalleled precision.

Let's Get Practical!

SCC Data Export: We are not going into too much details on the initial configuration itself as this is already documented at the following link with an easy to follow step by step instructions (please note that only future findings are sent to BigQuery from the time the configuration is enabled)

https://cloud.google.com/security-command-center/docs/how-to-analyze-findings-in-big-query

Next have a look at a few examples of common and useful BigQuery Queries:

1: Get all findings for a specific findings class and severity:

SELECT * FROM `<YOURDATASET>` WHERE finding.severity = 'HIGH' AND finding.finding_class = 'MISCONFIGURATION'

2: Get all findings for a specific resource type, severity, and category:

SELECT * FROM `<YOURDATASET>` WHERE resource.type = 'google.compute.Instance' AND finding.severity = 'MEDIUM' AND finding.finding_class = 'MISCONFIGURATION'

3: Get All Findings for a specific project

SELECT * FROM `<YOURDATASET>` WHERE resource.project_display_name = '<YOURPROJECT>'

4: Sort the number of Findings per day

SELECT DATE(finding.event_time) AS finding_date, COUNT(DISTINCT finding_id) AS num_scc_findings FROM `<YOURDATASET>` GROUP BY finding_date ORDER BY num_scc_findings DESC

5, Sort the number of Findings grouped by category, severity and finding class.

SELECT finding.category AS category, finding.severity AS severity, finding.finding_class AS class, COUNT(DISTINCT finding_id) AS num_scc_findings FROM `<YOURDATASET>` GROUP BY category, severity, class ORDER BY num_scc_findings DESC;

Wait, there’s more?

Yes, while getting lost in the amazing world of datasets and SQL queries can be fun, we have to remember that it’s always better to visualize our data wherever possible. Hands up anyone who does not like a nice and shiny pie chart!

With BigQuery you have a range of options on how you can visualize your datasets and results.

Explore your datasets with Sheets!

With just a click of a button, you can explore your data using Google Sheets. This is a great way to get a quick overview of your data and to perform basic analysis tasks.

Easily create charts and graphs to visualize your data.
Use Sheets' built-in functions to perform calculations and analysis.
Collaborate with others on your data analysis.

Here you have the full power of Sheets to work and visualize your data as needed

Explore your dataset with Looker Studio!

Dive deeper into exploring your dataset with Looker. Looker is a powerful business intelligence and data visualization tool that integrates with BigQuery. By using Looker, you gain access to a comprehensive suite of features that empower you to explore and analyze your dataset with ease.

Here are some key benefits of using Looker to explore your dataset:

Intuitive User Interface: Looker makes it easy to navigate and explore your data. Drag-and-drop functionality allows you to create visualizations and dashboards without the need for extensive coding knowledge.
Powerful Visualization Capabilities: Looker offers a wide range of visualization options, including charts, graphs, tables, and maps. With Looker, you can create visually appealing and informative dashboards that effectively communicate insights from your data.
Real-Time Data Exploration: Looker provides real-time data exploration, allowing you to interact with your data and make informed decisions on the fly. You can filter, sort, and drill down into your data to uncover patterns and trends.
Collaboration and Sharing: Looker fosters collaboration by enabling multiple users to simultaneously explore and analyze the same dataset. You can share dashboards and insights with colleagues, fostering a data-driven culture within your organization.

You have the option to explore your dataset with Looker just as you do in Google Sheets.

With the full power of Looker you can create dashboards to visualize your data, just as you want it. You can unlock the full potential of your dataset and gain valuable insights that drive informed decision-making. Looker is a powerful tool that lets businesses leverage their data effectively and achieve their goals.

In this blog, we will refrain from diving deep into the specifics of Looker and Sheets. Each deserves its own dedicated journey.
Now, I know what you're thinking. "That sounds amazing! I want to learn more about Looker and Sheets." In the near future, we'll be dedicating a post to showcasing some of the coolest Looker dashboards out there. So, stay tuned for that.

Think of BigQuery as the trusty sidekick to your SCC superhero. It won't stop threats on its own, but it'll give you the X-ray vision to see patterns, anomalies, and the long-term trends that can make all the difference in protecting your digital assets.

Just remember that BigQuery can enhance your SCC capabilities:

Refine threat detection rules: BigQuery can help you identify patterns suggesting new or evolving attack techniques. For example, you could use BigQuery to analyze SCC logs to look for trends in the types of attacks that are being attempted or the sources of the attacks. This information can then be used to create or update threat detection rules in SCC.
Proactively mitigate risk: BigQuery can help you address recurring vulnerabilities exposed through trend analysis. For example, you could use BigQuery to analyze SCC logs to identify the most common types of vulnerabilities that are being exploited. This information can then be used to prioritize patching and remediation efforts.
Improve incident response: BigQuery can help you leverage historical data for faster and more informed responses to security events. For example, you could use BigQuery to analyze SCC logs to identify the most common types of incidents that occur and the steps that were taken to resolve them. This information can then be used to create playbooks and runbooks for incident response.

Overall, BigQuery can be a valuable tool for enhancing your SCC capabilities. By providing you with the ability to analyze SCC logs in a scalable and efficient manner, BigQuery can help you identify threats, mitigate risk, and improve incident response.

Ready to level up your security analytics game?

Start exploring SCC's BigQuery integration today!

Have an awesome custom query or dashboard you would like to share?

Share them in our community forums below!

Useful links and resources

Setup SCC stream to BigQuery

https://cloud.google.com/security-command-center/docs/how-to-analyze-findings-in-big-query

Running queries in BigQuery

https://cloud.google.com/bigquery/docs/running-queries

BigQuery and Looker

https://cloud.google.com/bigquery/docs/looker

Community Forum Post