Azure Sentinel News
  • Home
    • Home – Layout 1
    • Home – Layout 2
    • Home – Layout 3
  • Security and Compliance
  • SOC
  • Threat Intelligence
  • Security Ochestration & Automated Response
  • SOAR
  • Security Operations
  • Artificial Intelligence
No Result
View All Result
  • Home
    • Home – Layout 1
    • Home – Layout 2
    • Home – Layout 3
  • Security and Compliance
  • SOC
  • Threat Intelligence
  • Security Ochestration & Automated Response
  • SOAR
  • Security Operations
  • Artificial Intelligence
No Result
View All Result
Azure Sentinel News
No Result
View All Result
Home Security and Compliance

Using the VirusTotal V3 API with MSTICPy and Azure Sentinel

Azure Sentinel News Editor by Azure Sentinel News Editor
November 26, 2020
in Security and Compliance
0
Vectra AI and Microsoft partner on security integration
6.8kViews
672 Shares Share on Facebook Share on Twitter

MSTICPy, our CyberSec toolset for Jupyter notebooks, has supported VirusTotal lookups since the very earliest days (the earliest days being only around two years ago!). We recently had a contribution to MSTICPy from Andres Ramirez and Juan Infantes at VirusTotal (VT), which provides a new Python module to access the recently-released version 3 of their API.

As well as a Python module, which provides the interface to lookup IoCs via the API, there is also a sample Jupyter notebook demonstrating how to use it.

As a side note, we’re delighted to get this submission, not just because it brings support for the awesome new VirusTotal API, but it is also first substantial contribution to MSTICPy from anyone outside MSTIC (Microsoft Threat Intelligence Center). Big thanks to Juan and Andres!

The two biggest features of the API are:

  • the ability to query relationships between indicators
  • the ability to visualize these relationships via an interactive network graph.

There is also an easy-to-use Python object interface exposed via the vt_py and vt_graph_api Python libraries (these are both required by the MSTICPy vtlookup3 module). You can read more about the VT Python packages by following the links in the previous sentence.

Why you would want to use VTLookup3

Alerts and incidents in Azure Sentinel will nearly always have a set of entities attached to them. These entities might be things like IP addresses, hosts, file hashes, URLs, etc. This is common to many SOC environments.

Let’s take an example of Microsoft Defender for Endpoint (MDE) alerts ingested into Azure Sentinel.

Note: MDE is the new name for Microsoft Defender Advance Threat Protection (MDATP) so you will probably see both terms kicking around for a while.

If we look at the incident detail and click on the Entities tab we see a list of entities related to the incident.

As part of your SOC triage you will want to check on whether any of these file hashes are known bad and possibly explore other related files, IPs. domains, etc. MSTICPy allows you to do custom correlation with arbitrary external data sources, including VT.

We will use one of the file hashes taken from this incident as we explore VT Lookup capabilities. Although our example below uses a file hash, you could use a URL, domain or IP Address as your starting point.

Getting started

There is a little bit of setup to do before you can get going.

Note: MSTICPy with extras
We’re in the process of moving a lot of the growing list of MSTICPy dependencies into optional installs. These are known as “extras” in the Python setup world. VirusTotal are the first to be “extra”d but we’re not picking on them. It was just, as a new package, it was easiest to start the dependency refactoring with a new library.

Install msticpy with the “vt3” extra

pip install msticpy[vt3]

or just install the vt_py and vt_graph_api packages directly:

pip install vt-py vt-graph-api nest_asyncio

Note: the nest_asyncio package is required for use in notebooks but not if you’re use the vtlookup3 module and VTLookup3 class in Python code.


In the notebook, import the modules:

from msticpy.sectools.vtlookupv3 import VTLookupV3
import nest_asyncio

And create an instance of the VTLookup3 class.

vt_lookup = VTLookupV3(vt_key)

You need to supply your VirusTotal API key when you create the VTLookup3 instance. You can supply this as a string or store it in your msticpyconfig.yaml configuration file.

This code, taken from the notebook will try to find the VT API key in your configuration.

from msticpy.common.provider_settings import get_provider_settings
vt_key = get_provider_settings("TIProviders")["VirusTotal"].args["AuthKey"]

Note: in the configuration file you can specify that the API key value be retrieved from an environment variable or from an Azure Key Vault. See MSTICPy Package Configuration for more details.

Using VTLookup3

VTLookup3 has the following methods:

  • lookup_ioc – for single item lookups
  • loopkup_iocs – for multiple lookups
  • lookup_ioc_relationships – to find IoCs and attributes that are related to the searched-for IoC
  • lookup_iocs_relationships – to find relationships for multiple entities
  • create_vt_graph – to build a graph from a relationship set
  • render_vt_graph – to display the graph in the notebook.

We’ll look at each of these in turn.

Note: we use the terms IoC (indicator of compromise) and observable interchangably in this article. They indicate data items such as IP addresses, URLs, file hashes etc. that may be observed during an attack and thus become indicators of compromise.

Looking up a single IoC/observable

lookup_ioc works much like the other TI providers in MSTICPy except that it returns the results as a pandas DataFrame, even for a single IoC query.

FILE = 'ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa'
example_attribute_df = vt_lookup.lookup_ioc(observable=FILE, vt_type='file')
example_attribute_df

If an entry matching this ID is found in VT it is returned with some basic attributes such as submission times, name, and type.

Since the time stamp is returned as a Unix serial timestamp you might want to re-format into a more readable form using code like this.

example_attribute_df.assign(
    first_submission=pd.to_datetime(example_attribute_df.first_submission_date, unit="s", utc=True),
    last_submission=pd.to_datetime(example_attribute_df.last_submission_date, unit="s", utc=True)
)

This adds two columns to the data that display as human-readable datetimes

Note: we plan to change this soon so that the API will return datetime types directly.

We’ve added a convenience function to the notebook that allows you to return full details for a VirusTotal object. This isn’t done by default in the built-in methods but you can use this to query additional data for specific IDs.

Note: get_object will be exposed in the core VTLookup3 interface in a future version. This simply implements the vt_py client.get_object call. You can use the vt_py method directly or go the VirusTotal site to see the details rendered in a more consumable fashion.

Looking up IoC Relationships

The lookup_ioc_relationships method shows off some of the capability of the new API. From a single indicator you can retrieve any related IoCs (in this case we’re using the same file hash as above). In the example here we’re retrieving any known parent processes for the malware that we are investigating.

example_relationship_df = vt_lookup.lookup_ioc_relationships(
    observable=FILE, 
    vt_type='file', 
    relationship='execution_parents')
example_relationship_df

This returns a DataFrame with the IDs of any related IoCs, their types, and the relationship type to the original observable.

The source column is the ID of our original observable. The target column contains the known execution parents of this file. The results represent a simple graph with the nodes being the source and targets and edges being the relationship_type between them.

Use the limit parameter to this function to restrict how many related entities are returned.

You can look up details of any of the related parents using the get_object() function – but don’t do that just yet until you’ve read the next section.

Looking up multiple observables – lookup_iocs

You can use lookup_iocs to lookup multiple observables in a single call (e.g. IoCs extracted from a set of process events). Assuming that the observables are in a DataFrame called input_df, call lookup_iocs as follows:

results_df = vt_lookup.lookup_iocs(
observables_df=input_df,
observable_column="colname_with_ioc",
observable_type_column="colname_with_vt_type"
)

You can also submit the DataFrame that we generated earlier – example_relationship_df – directly as an input to lookup_iocs. In this case it will default to using the target and target_type columns for the observable_column and observable_type_column parameters, so these do not need to be specified as parameters.

Executing this brings back some basic details about each target item listed in our example_relationship_df DataFrame.

Expanding the Graph – looking up relationships for multiple IoCs

lookup_iocs_relationships is the equivalent of lookup_ioc_relationships but taking multiple IoC/observables as input.

The output from this is shown using the previously generated DataFrame as input. Like lookup_iocs, this is defaulting to using the target and target_type columns for the observable_column and observable_type_column parameters; so we haven’t needed to include these parameters in the call.

You can use this API with an arbitrary DataFrame as input, specifying the observable_column and observable_type_column parameters. The data in both columns must be in the correct format for submission with valid vt_type strings (see the documentation for further details)

Importing the data to networkx

Networkx is probably the most popular Python graphing library. It only includes basic visualization capabilities but it does support a rich variety of graphing analysis and query functions. Since the data structure of the VT relationships is a graph, you can easily import the data into a networkx graph (and also plot it). Having the ability to manipulate the graph in networkx allows you to apply graphing functions and analysis on the data, such as finding the most central nodes, or those having the most neighbors.

We’ve given a simple example of importing the relationships DataFrame into a networkx graph and plotting a simple view of it with Bokeh.

from bokeh.io import output_notebook, show
from bokeh.plotting import figure, from_networkx
from bokeh.models import HoverTool
graph = nx.from_pandas_edgelist(
example_multiple_relationship_df.reset_index(),
source="source",
target="target",
edge_attr="relationship_type",
)
plot = figure(
title="Simple graph plot", x_range=(-1.1, 1.1), y_range=(-1.1, 1.1), tools="hover"
)
g_plot = from_networkx(graph, nx.spring_layout, scale=2, center=(0, 0))
plot.renderers.append(g_plot)
output_notebook()
show(plot)

While this may look cosmic, it isn’t hugely informative and isn’t interactive in any way. On to better things…

Displaying the VT Graph

To see the data in its full glory use the create_vt_graph and render_vt_graph methods:

graph_id = vt_lookup.create_vt_graph(
relationship_dfs=[example_relationship_df, example_multiple_relationship_df],
name="My first Jupyter Notebook Graph",
private=False,
)

You may need to be patient with these APIs since a complex graph can take a while to build and render. In a few moments you should see an interactive graph render into an IFrame in the notebook as seen in the screen shot below.

Exploring the graph on the VirusTotal site brings more capabilities such as being able to view further details about the graph entities, to search within the graph, and to search for, and add, additional nodes.

We very much appreciate this addition to MSTICPy. It brings a lot more of the power of VirusTotal data and exploration to the world of hunting and investigation in Jupyter notebooks.

Reference: https://techcommunity.microsoft.com/t5/azure-sentinel/using-the-virustotal-v3-api-with-msticpy-and-azure-sentinel/ba-p/1893121

Tags: jupyternotebookmsticmsticpypython
Azure Sentinel News Editor

Azure Sentinel News Editor

Related Posts

Vectra AI and Microsoft partner on security integration
Security and Compliance

Replay Now Available – Microsoft Security Insights 036: Azure Sentinel with Rod Trent

February 8, 2021
What’s new: Microsoft Teams connector in Public Preview
Security and Compliance

eBook Available for Managing Azure Sentinel with PowerShell

January 6, 2021
Microsoft is quietly becoming a cybersecurity powerhouse
Security and Compliance

Official Azure Sentinel PowerShell Module Released

January 4, 2021
Next Post
Enriching Windows Security Events with Parameterized Function

Using Azure Data Explorer for long term retention of Azure Sentinel logs

Microsoft Debuts Azure Sentinel SIEM, Threat Experts Service

What's new: Azure Sentinel User and Entity Behavior Analytics in Public Preview!

CRITICALSTART Adds Support for Microsoft Azure Sentinel to MDR Services

What's New: PowerShell+Azure Sentinel notebooks to supercharge your hunting and investigations!

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Follow Us

  • 21.8M Fans
  • 81 Followers

Recommended

Aggregating Insider Risk Management Information via Azure Sentinel

Aggregating Insider Risk Management Information via Azure Sentinel

5 months ago
With new release, CrowdStrike targets Google Cloud, Azure and container adopters

Exporting Events from Disconnected Systems to Ingest into Azure Sentinel

2 months ago
Mimecast® Email Security for Azure Sentinel is now available in the Microsoft Azure Marketplace

Mimecast® Email Security for Azure Sentinel is now available in the Microsoft Azure Marketplace

4 months ago
CriticalStart Boosts MDR With Azure Sentinel Support

CriticalStart Boosts MDR With Azure Sentinel Support

3 months ago

Instagram

    Please install/update and activate JNews Instagram plugin.

Categories

  • AI & ML
  • Artificial Intelligence
  • Incident Response
  • IR
  • KQL
  • Security and Compliance
  • Security Ochestration & Automated Response
  • Security Operations
  • SIEM
  • SOAR
  • SOC
  • Threat Intelligence
  • Uncategorized

Topics

anomaly automation Azure Azure DevOps Azure Security Center Azure Sentinel Azure Sentinel API Azure Sentinel Connector BlueVoyant Call cybersecurity Detection file GitHub Hunting Huntingy IAC incident response Incident Triage infrastructure as code Investigation jupyter LAQueryLogs MDR Microsoft microsoft 365 mssp Multitenancy Notebooks Pester Playbooks PowerShell python Records Security Sentinel Sharing SIEM signin Supply Chain teams Threat hunting Watchlists Workbooks XDR
No Result
View All Result

Highlights

New Items of Note on the Azure Sentinel GitHub Repo

Tuning the MCAS Analytics Rule for Azure Sentinel: System Alerts and Feature Deprecation

New Search Capability for Azure Sentinel Incidents

Follow-up: Microsoft Tech Talks Practical Sentinel : A Day in the Life of a Sentinel Analyst

Changes in How Running Hunting Queries Works in Azure Sentinel

Azure Sentinel can now Analyze All Available Azure Active Directory Log Files

Trending

What’s new: Microsoft Teams connector in Public Preview
IR

How to Generate Azure Sentinel Incidents for Testing

by Azure Sentinel News Editor
February 26, 2021
0

Do you want to generate an Incident in Azure Sentinel for testing/demoing? Here’s a couple easy ways...

What’s new: Microsoft Teams connector in Public Preview

Azure Sentinel Notebooks Loses It’s Preview Tag

February 25, 2021
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

The Holy Grail of Azure Sentinel Data Connections: The Azure Service Diagnostic Setting

February 22, 2021
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

New Items of Note on the Azure Sentinel GitHub Repo

February 18, 2021
Microsoft’s newest sustainable datacenter region coming to Arizona in 2021

Tuning the MCAS Analytics Rule for Azure Sentinel: System Alerts and Feature Deprecation

February 17, 2021

We bring you the best, latest and perfect Azure Sentinel News, Magazine, Personal Blogs, etc. Visit our landing page to see all features & demos.
LEARN MORE »

Recent News

  • How to Generate Azure Sentinel Incidents for Testing February 26, 2021
  • Azure Sentinel Notebooks Loses It’s Preview Tag February 25, 2021
  • The Holy Grail of Azure Sentinel Data Connections: The Azure Service Diagnostic Setting February 22, 2021

Categories

  • AI & ML
  • Artificial Intelligence
  • Incident Response
  • IR
  • KQL
  • Security and Compliance
  • Security Ochestration & Automated Response
  • Security Operations
  • SIEM
  • SOAR
  • SOC
  • Threat Intelligence
  • Uncategorized

[mc4wp_form]

Copyright © 2020 - Azure Sentinel News

No Result
View All Result
  • Home
  • Security and Compliance
  • SOC
  • Threat Intelligence
  • Security Ochestration & Automated Response
  • SOAR
  • Security Operations
  • Artificial Intelligence

Copyright © 2020 Azure Sentinel News