Get-ADAuditLogs - Add same interval logic as M365 #70
Comments
|
The more i think on this, the more i think we could create a function for the acquisition loop, passing in the required command to it. This could hugely simplify the code-reuse we have |
|
Hi @angry-bender, thanks a lot again for the great suggestions and PR :)! I am on holiday so can't check/test will be back in 8 days. |
No problem mate, have a great break |
|
If it helps when the memory issues began to occur a while back I deviated (now heavily lol) from the original codebase in my fork. I decided , in order to keep using delegated permissions, Then semi-manual pagination and some other stuff, and even GC; which is not necessary but I've not had issue since. This is just what worked for me, but the fork I have is significantly altered, especially with removal of assertions Get-AzureADLogs,ps1 using module "$PSScriptRoot\Microsoft-Extractor-Suite.psm1";
# This contains functions for getting Azure AD logging
function Get-ADSignInLogs
{
[CmdletBinding()]
param(
[datetime]$StartDate = (Get-Date).AddDays(-30),
[datetime]$EndDate = Get-Date,
[string]$OutputDir = "$PSScriptRoot\Output\",
[string]$UserIds,
[switch]$MergeOutput,
[string]$Encoding = 'UTF8',
[int]$Interval
)
Write-Log -Message "[INFO] Running Get-ADSignInLogs" -Color "Green"
$dateStamp = Get-Date -Format "yyyyMMddHHmmss"
#if ([string]::IsNullOrWhiteSpace($OutputDir.Split('\')[0])){mkdir -Force "$OutputDir\AzureAD\$date">$null}
#if ([string]::IsNullOrEmpty($UserIds){Write-LogFile -Message "[INFO] UserIDs not specificed."}
if (-not $UserIds)
{
Write-Log -Message "[INFO] UserIDs not specified."
}
$filePath = "$OutputDir$($dateStamp)-AuditLogSignIn.json"
$baseUri = 'https://graph.microsoft.com/v1.0'
# (Find-MgGraphCommand -Command Get-AzureADAuditLogSignIn).CommandI[1]
$resourcePath = (Find-MgGraphCommand -Command Get-MgBetaAuditLogSignIn).URI[1]
$baseUri = "$baseUri$resourcePath`?"
$queryParameters = @()
if ($StartDate) { $queryParameters += "`$filter=activityDateTime ge $StartDate" }
if ($EndDate) { $queryParameters += "activityDateTime le $EndDate" }
if ($UserIds) { $queryParameters += " and initiatedBy/user/id eq $UserIds" }
$filterQuery = $queryParameters -join ' and '
$apiUrl = "$baseUri``$filterQuery"
try
{
do
{
$response = Invoke-MgGraphRequest -Method Get -Uri $apiUrl -ContentType 'application/json'
$logs = $response
$currentDate = Get-Date -Format "yyyy-MM-ddTHH:mm:ss"
$logs.Values | ConvertTo-Json -Depth 100 | Out-File -FilePath $filePath -Encoding utf8BOM
Write-Host "[INFO] Sign-in logs written to $filePath" -ForegroundColor Green
$apiUrl = $response.'@odata.nextLink' # Update the URL to the nextLink for pagination
[System.GC]::Collect()
[System.GC]::WaitForPendingFinalizers()
} while ($response.'@odata.nextLink')
}
catch [Exception]
{
Write-Error "Error fetching data: $_"
}
finally
{
Remove-Variable response -ErrorAction Ignore
Remove-Variable logs -ErrorAction Ignore
[System.GC]::Collect()
[System.GC]::WaitForPendingFinalizers()
}
if ($MergeOutput)
{
try
{
Write-Host '[INFO] Merging output files...' -ForegroundColor Green
$mergedFile = "$OutputDir$($dateStamp)-AuditLogSignIn-MERGED.json
Merge-OutputFiles -OutputDir $outputDir -Encoding $Encoding -mergedFile $mergedFile
}
catch
{
Write-Error "Error fetching data: $_" -ForegroundColor Red
}
finally
{
Write-Host '[INFO] Process completed.' -ForegroundColor Green
}
}
}function Get-ADAuditLogs
{
[CmdletBinding()]
param(
[datetime]$StartDate = (Get-Date).AddDays(-30).ToString('yyyy-MM-ddT00:00:00'),
[datetime]$EndDate = (Get-Date).ToString('yyyy-MM-ddT00:00:00'),
[string]$OutputDir = "$((Get-Location).Path)\Output\",
[string]$UserIds,
[string]$Encoding = 'UTF8'
)
$dateStamp = Get-Date -Format "yyyyMMddHHmmss"
$filePath = "$OutputDir$($dateStamp)-AuditLogDirectoryAudit.json"
Write-Log -Message "[INFO] Collecting the Directory Audit Logs"
$baseUri = 'https://graph.microsoft.com/v1.0'
#(find-MgGraphCommand -Command Get-AzureADAuditLogDirectoryAudit).Command[1] -eq Get-MgBetaAuditLogDirectoryAudit
$resourcePath = (Find-MgGraphCommand -Command Get-MgBetaAuditLogDirectoryAudit).URI[1]
$baseUri = "$baseUri$resourcePath`?"
$queryParameters = @()
if ($StartDate) { $queryParameters += "`$filter=activityDateTime ge $StartDate" }
if ($EndDate) { $queryParameters += "activityDateTime le $EndDate" }
if ($UserIds) { $queryParameters += " and initiatedBy/user/id eq $UserIds" }
$filterQuery = $queryParameters -join ' and '
$apiUrl = "$baseUri$filterQuery"
try
{
do
{
$response = Invoke-MgGraphRequest -Method Get -Uri $apiUrl -ContentType 'application/json'
$logs = $response
$currentDate = Get-Date -Format "yyyy-MM-ddTHH:mm:ss"
$logs.Values | ConvertTo-Json -Depth 100 | Out-File -FilePath $filePath -Encoding utf8BOM
Write-Host "[INFO] Directory logs written to $filePath" -ForegroundColor Green
$apiUrl = $response.'@odata.nextLink' # Update the URL to the nextLink for pagination
[System.GC]::Collect()
} while ($response.'@odata.nextLink')
}
catch [Exception]
{
Write-Error "Error fetching data: $_"
}
finally
{
Remove-Variable response -ErrorAction Ignore
Remove-Variable logs -ErrorAction Ignore
[System.GC]::WaitForPendingFinalizers()
[System.GC]::Collect()
}
Write-Log -Message "[INFO] Directory audit logs written to $filePath" -Color "Green"
} |
|
@Calvindd2f Thanks for providing the code snippets. Looking at your fork, you indeed deviated heavily from this repository💪. You did change and add some cool stuff. You clearly know a lot more about PowerShell than me, haha. We might have to utilize the REST API more and not be dependent on the PowerShell Graph module too much. Your approach seems to be a lot easier. I don't like the approach of acquiring the logs for each interval, similar to how we do with the UAL, because it's fault-prone and feels inefficient. However, I'm not sure what the best approach would be. But I kind of like to use the same approach for all functionalities, so we would need to change it for the rest as well then. |
|
@angry-bender I considered creating a function for the acquisition loop earlier and experimented with a few approaches for the UAL. However, I wasn't sure what the best solution would be. While attempting to build this it didn't necessarily make it easier or simpler, haha. To try and reduce the interval, similar to the M365 scripts (vs sleep and try again). I've had a look at that error handling, but im not quite sure how you implemented it in the UAL acquisition scripts. The main challenge here is that with the Unified Audit Log (UAL), there is a limit of 5000 results per call. To handle this, you can check if there are more than 5000 events and then reduce the interval until the result count falls below this limit. When you run the command
Unfortunately, the Last too do, might be to change the allowable date field, since we can now add time this function too :-) Agreed, should add this! |
|
Just given the functionality of the commandet, my concern would be how long Microsoft takes to return that number, as I think you'd be requesting the same data twice. For the error I didn't get the chance to pull down the exact error message unfortunately to add in some proper logic. But I do agree in that if error then try again rather than just the sleep and try again. I've found this commandlet a lot less fault tolerant than the graph one |
|
@angry-bender I think I will do something like this, make it retry 3 times when an error occurs, waiting 5 seconds before trying again: What would look like this in the output: |
That would work, in my testing I did find the sweet spot is 10 seconds after one error, then it usually resumes for the too many requests error |
|
Hi @angry-bender, Just released the update! All three tasks have been added. Please let me know if anything is missing or if you encounter any issues. |
No problemo, I might do some digging on the restapi when I get the chance. It would be really good if we can get some memory optimisation as per @Calvindd2f 's post. I too, don't like the request coming down and needing to be dropped straight to disk, but Microsoft does keep changing the ball on us folk who just want some logs. |
@angry-bender,Thank you for your appreciation. I'll have to un-gitignore some files for the fork as there were a few missing. I mainly prefer the API because the Authentication module for graph is very inconsistent for me. Below are some of the things I personally did but did not try merge as I feel it strayed to far from original module / unrealistic. It is not exhaustive. Many of the style and formatting changes are not trying to follow some form of arbitrary elitism styling. I say this because they do look a bit fucky, They are proven to be many orders of magnitude faster than alternatives, here are some faster explanations. mslearn : The speeds of
Get-Content $path | Where-Object Length -GT 10This can be an order of magnitude slower than using .NET APIs directly. For example, you can use the .NET [StreamReader] class: try
{
$reader = [System.IO.StreamReader]::new($path)
while (-not $reader.EndOfStream)
{
$line = $reader.ReadLine()
if ($line.Length -gt 10)
{
$line
}
}
}
finally
{
if ($reader)
{
$reader.Dispose()
}
}The reason for the indentation not being the most common
Probably some other stuff I forgot to mention, so here is the gist Memory OptimizationsFor memory optimizations in my own. The very first thing I did was remove all the function Set-OutputEncoding
{
if ($PSVersionTable.PSVersion.Major -lt 6)
{
# For PowerShell versions less than 6, set to UTF8
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
}
else
{
# For PowerShell 6 and above, set to UTF8 without BOM
[System.Text.Encoding]::UTF8NoBOM = New-Object System.Text.UTF8Encoding($false)
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8NoBOM
}
}Then function Merge-OutputFiles
{
param(
[string]$OutputDir,
[string]$Encoding,
[string]$mergedFile,
)
$mergedFilePath = Join-Path -Path $OutputDir -ChildPath $mergedFile
$allLogs = Get-ChildItem -Path $OutputDir -Filter '*.json' | ForEach-Object {
$content = [System.IO.File]::ReadAllText($_.FullName)
[System.Text.Json.JsonSerializer]::Deserialize($content, [object].GetType())
}
$jsonOutput = [System.Text.Json.JsonSerializer]::Serialize($allLogs, [object].GetType(), [System.Text.Json.JsonSerializer]::GetOptions())
[System.IO.File]::WriteAllText($mergedFilePath, $jsonOutput, [System.Text.Encoding]::$Encoding)
Write-Host "[INFO] All logs merged into $mergedFilePath" -ForegroundColor Green
}I could not make up my mind how I wanted to do logging. I knew it had to be refactored as it is called 200+ times. Here are a few revisions of Function Write-Loglol([string]$log,[switch]$show){
[string]$logtime = $((Get-Date -Format "[dd/MM/yyyy HH:mm:ss zz] |").ToString())
foreach($line in $($log -split "`n")){
if($VerbosePreference -eq 'Continue' -or $show -eq $true){Write-Host "$logtime $line"}
$streamWriter = [System.IO.StreamWriter]::Synchronized([System.IO.File]::AppendText("C:\Windows\Temp\log.txt.log"))
$streamWriter.WriteLine("$logtime $line")
$streamWriter.Dispose()
}
}
function Write-LogFile([string]$message, [string]$severity, [string]$logFile = 'Output\LogFile.txt') {
$logEntry = [DateTime]::Now.ToString() + ': ' + $severity.ToUpper() + ' ' + $message
try {
[System.IO.File]::AppendAllText($logFile, $logEntry + [Environment]::NewLine)
} catch {
# exception
}
$foregroundColor = switch ($severity) {
'WARNING' { 'Yellow' }
'W' { 'Yellow' }
'?' { 'Yellow' }
'ERROR' { 'Red' }
'E' { 'Red' }
'!' { 'Red' }
'SUCCESS' { 'Green' }
'S' { 'Green' }
'+' { 'Green' }
Default { 'White' }
}
[console]::ForegroundColor = $foregroundColor
[console]::WriteLine($logEntry)
[console]::ResetColor()
}
function Write-Log
{
param (
[Parameter(Mandatory = $true, Position = 0, HelpMessage = 'Log entry')]
[ValidateNotNullOrEmpty()]
[string]$Entry,
[Parameter(Position = 1, HelpMessage = 'Log file to write into')]
[ValidateNotNullOrEmpty()]
[Alias('LogFile')]
[string]$Logs = 'Output\Log.txt',
[Parameter(Position = 2, HelpMessage = 'Level')]
[ValidateSet('Info', 'Error', 'Process', 'Note', 'Warning')]
[string]$Level = 'Info'
)
$Indicator = switch ($Level) {
'Warning' { 'Warning' }
'Error' { 'Error' }
'Process' { 'Process' }
'Note' { 'Note' }
default { 'Info' }
}
$foregroundColor = switch ($Level) {
'Warning' { 'Yellow' }
'Error' { 'Red' }
'Info' { 'White' }
'Sucess' { 'Green' }
default { 'White' }
}
$message = "$Indicator : $Entry"
[console]::ForegroundColor = $foregroundColor
[console]::WriteLine($message)
[console]::ForegroundColor = 'White'
if (-not $global:NoLog) {
try {
[System.IO.File]::AppendAllText($Log, (Get-Date -Format 'yyyy-MM-dd HH:mm:ss.fff') + ' ' + $message + [Environment]::NewLine)
} catch {
# Handle exception
}
}
}dot sourcing the psm1 so that StartDate , EndDate don't have to be redefined as new functionsThe scripts all dot source the psm1 because this way we do not have to declare startdate/enddate over in new functions . "$PSScriptRoot\Microsoft-Extractor-Suite.psm1";
Function StartDate
{
if ([string]::IsNullOrWhiteSpace($startDate))
{
$daysToAdd = -90
$message = "[INFO] No start date provided by user setting the start date to: {0}" -f ([datetime]::Now.ToUniversalTime().AddDays($daysToAdd).ToString('yyyy-MM-ddTHH:mm:ssK'))
$color = 'Yellow'
}
else
{
$daysToAdd = -30
$message = "[INFO] No start date provided by user setting the start date to: {0}" -f ($startDate -as [datetime]).ToString('yyyy-MM-ddTHH:mm:ssK')
$color = 'Yellow'
}
$Global:StartDate = [datetime]::Now.ToUniversalTime().AddDays($daysToAdd)
write-LogFilew -Message $message -Color $color
return $Global:StartDate;
}
function EndDate
{
if ([string]::IsNullOrWhiteSpace($endDate))
{
$script:EndDate = [datetime]::UtcNow
$message = "[INFO] No end date provided by user; setting the end date to: $($script:EndDate.ToString('yyyy-MM-ddTHH:mm:ssK'))"
$color = 'Yellow'
}
else
{
if (-not ($endDate -as [datetime]))
{
$message = '[WARNING] Not a valid end date and time; make sure to use ___________'
$color = 'Red'
}
else
{
$script:EndDate = $endDate
return
}
}
write-LogFile -Message $message -Color $color
$Global:EndDate = $endDate
return $Global:EndDate;
}I also had a function at one point (I can't find it but it will easily be recreated), where the function would check if startdate or enddate In same function it would check if variable was present If the pattern did not match it would convert the inputted date / datetime to 'yyyy-MM-dd HH:mm:ss.fff' using the [datetime] GetFormats to check what was the input. If a format that did not include time was entered it is assumed that the time value is mid-day 12:00:00:00 UAL
tl;dr Currently the UAL uses the old outlook.com/adminapi still despite Microsoft insisting it is depreciated. I mention this because unfortunately doing the below to have an effective Exchange API requires an app with both exchange manage as app permissions & office 365 management api permissions to even query... The management API can be queried using just App + Permissions but I am not too sure about the completeness offered by it. But it very well could be the true backend. This isn't to say that we cannot manage memory of exchangeonlinemanagement through powershell module, it's just less ideal. But the alternative is quite complex. Querying exchange via the API $conn_id = $([guid]::NewGuid().Guid).ToString()
$conn=$conn_id
$tenant_name='lvin.ie'
$exo_token=''
$exo_token=(ConvertTo-SecureString -String $exo_token -AsPlainText -Force )
Function ExoCommand($conn, $command, [HashTable]$cargs, [int]$retry = 5,[int]$PageSize = 1000)
{
[int]$defaultTimeout = 30;
[int]$MaxRetries = 5 # In total we try 6 times. 1 original try, 5 retry attempts.
$success = $false
$count = 0
$body = @{
CmdletInput = @{
CmdletName="$command"
}
}
if($null -ne $cargs){
$body.CmdletInput += @{Parameters= [HashTable]$cargs}
}
$json = $body | ConvertTo-Json -Depth 5 -Compress
[string]$commandFriendly = $($body.CmdletInput.CmdletName)
for([int]$x = 0 ; $x -le $($body.CmdletInput.Parameters.Count - 1); $x++){
try{$param = " -$([Array]$($body.CmdletInput.Parameters.Keys).Item($x))"}catch{$param = ''}
try{$value = "`"$([Array]$($body.CmdletInput.Parameters.Values).Item($x) -join ',')`""}catch{$value = ''}
$commandFriendly += $("$param $value").TrimEnd()
}
Write-Host "Executing: $commandFriendly"
Write-Host $json
[string]$url = $("https://outlook.office365.com/adminapi/beta/$tenant_name/InvokeCommand")
if(![string]::IsNullOrEmpty($Properties)){
$url = $url + $('?%24select='+$($Properties.Trim().Replace(' ','')))
}
[Array]$Data = @()
do{
try{
do{
## Using HTTPWebRequest library
$request = [System.Net.HttpWebRequest]::Create($url)
$request.Method = "POST";
$request.ContentType = "application/json";
$request.Headers["Authorization"] = "Bearer $($exo_token)"
$request.Headers["x-serializationlevel"] = "Partial"
#$request.Headers["x-clientmoduleversion"] = "2.0.6-Preview6"
$request.Headers["X-AnchorMailbox"] = $("UPN:SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}@$tenant_name")
$request.Headers["X-prefer"] = "odata.maxpagesize=1000"
#$request.Headers["Prefer"] = 'odata.include-annotations="display.*"' v
$request.Headers["X-ResponseFormat"] = "json" ## Can also be 'clixml'
$request.Headers["connection-id"] = "$conn_id"
#$request.Headers["accept-language"] = "en-GB"
$request.Headers["accept-charset"] = "UTF-8"
#$request.Headers["preference-applied"] = ''
$request.Headers["warningaction"] = ""
$request.SendChunked = $true;
$request.TransferEncoding = "gzip"
$request.UserAgent = "Mozilla/5.0 (Windows NT; Windows NT 10.0; en-IE) WindowsPowerShell/5.1.19041.1682"
#$request.Host = "outlook.office365.com"
$request.Accept = 'application/json'
$request.Timeout = $($defaultTimeout*1000)
$requestWriter = New-Object System.IO.StreamWriter $request.GetRequestStream();
$requestWriter.Write($json);
$requestWriter.Flush();
$requestWriter.Close();
$requestWriter.Dispose()
$response = $request.GetResponse();
$reader = new-object System.IO.StreamReader $response.GetResponseStream();
$jsonResult = $reader.ReadToEnd();
$result = $(ConvertFrom-Json $jsonResult)
$response.Dispose();
if(@($result.value).Count -ne 0){
$Data += $($result.value)
Write-Host "Got $($result.value.Count) items"
}
try{$url = $result.'@odata.nextLink'}catch{$url = ''}
if(![string]::IsNullOrEmpty($url)){
Write-Host "Getting next page..."
}
}while(![string]::IsNullOrEmpty($url))
$success = $true
$count = $retry
return @($Data)
} catch {
if($($_.Exception.Message) -like "*timed out*" -or $($_.Exception.Message) -like "*Unable to connect to the remote server*"){
$count++
Write-Warning "TIMEOUT: Will retry in 10 seconds."
Start-Sleep -seconds 10
if($count -gt $retry){throw "Timeout retry limit reached"}
}else{
Write-Warning "Failed to execute Exchange command: $commandFriendly"
Write-Warning $($_.Exception.Message)
throw;
}
}
}while($count -lt $retry -or $success -eq $false)
return $null
}
$user="c@lvin.ie"
Function CheckSuccess($dl, $conn, $user)
{
$members = ExoCommand -conn $conn -Command "Get-Mailbox" -cargs @{ Identity = $user }
foreach($mem in $members)
{
if($mem.WindowsLiveID -eq $user)
{
Write-Host "Success!"
return $true;
}
}
return $false;
}Use API Query to query UAL with backoff: exponential backoff logic function Invoke-ExoCommandWithBackoff {
param (
[string]$CmdletName,
[hashtable]$Parameters,
[int]$MaxRetries = 5,
[int]$PageSize = 1000
)
$conn_id = $([guid]::NewGuid().Guid).ToString()
[int]$BaseDelay = 1000
[Random]$Rnd = [Random]::new()
$Results = @()
$body = @{
CmdletInput = @{
CmdletName = "$CmdletName"
Parameters = $Parameters
}
}
$jsonBody = $body | ConvertTo-Json -Depth 5 -Compress
$encodedBody = [System.Text.Encoding]::UTF8.GetBytes($jsonBody)
[string]$url = "https://outlook.office365.com/adminapi/beta/$tenant_name/InvokeCommand"
[string]$nextPageUri = ''
$Headers = @{
'Accept' = 'application/json'
'Accept-Charset' = 'UTF-8'
'Content-Type' = 'application/json'
'X-CmdletName' = $CmdletName
'client-request-id' = [guid]::NewGuid().Guid
}
$nextPageSize = [Math]::min($PageSize, 1000)
$anotherPagedQuery = $true
while ($anotherPagedQuery) {
$Headers['Prefer'] = "odata.maxpagesize=$nextPageSize"
$isRetryHappening = $true
$isQuerySuccessful = $false
$retryCount = 0
while ($isRetryHappening -and $retryCount -lt $MaxRetries) {
try {
$request = [System.Net.HttpWebRequest]::Create($url)
$request.Method = 'POST'
$request.ContentType = 'application/json;odata.metadata=minimal;odata.streaming=true;'
$request.Headers['Authorization'] = "Bearer $($exo_token)"
$request.Headers['x-serializationlevel'] = 'Partial'
$request.Headers['X-AnchorMailbox'] = $("UPN:SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}@$tenant_name")
$request.Headers['X-prefer'] = 'odata.maxpagesize=1000'
$request.Headers['X-ResponseFormat'] = 'clixml'
$request.Headers['connection-id'] = "$conn_id"
$request.Headers['accept-charset'] = 'UTF-8'
$request.Headers['warningaction'] = ''
$request.SendChunked = $true
$request.TransferEncoding = 'gzip'
$request.UserAgent = 'Mozilla/5.0 (Windows NT; Windows NT 10.0; en-IE) WindowsPowerShell/5.1.19041.1682'
$request.Accept = 'application/json'
$request.Timeout = $defaultTimeout * 1000
$requestWriter = New-Object System.IO.StreamWriter $request.GetRequestStream()
$requestWriter.Write($encodedBody)
$requestWriter.Flush()
$requestWriter.Close()
$requestWriter.Dispose()
$response = $request.GetResponse()
$reader = New-Object System.IO.StreamReader $response.GetResponseStream()
$jsonResult = $reader.ReadToEnd()
$result = $(ConvertFrom-Json $jsonResult)
$response.Dispose()
# Handle result as Clixml
if (@([System.Management.Automation.PSSerializer]::Deserialize($result.value.'_clixml')).Count -ne 0) {
$Results += $([System.Management.Automation.PSSerializer]::Deserialize($result.value.'_clixml'))
Write-Host "Got $($result.value.Count) items"
}
try { $url = $result.'@odata.nextLink' } catch { $url = '' }
if (![string]::IsNullOrEmpty($url)) {
Write-Host 'Getting next page...'
}
$isQuerySuccessful = $true
$isRetryHappening = $false
} catch {
if ($retryCount -lt $MaxRetries) {
$delay = [Math]::Pow(2, $retryCount) * $BaseDelay + $Rnd.Next(0, 1000)
Write-Warning "Attempt $retryCount failed. Retrying in $($delay/1000) seconds."
Start-Sleep -Milliseconds $delay
} else {
Write-Warning "Failed to execute Exchange command: $CmdletName after $MaxRetries attempts."
throw
}
$retryCount++
}
}
if (!$isQuerySuccessful) {
$anotherPagedQuery = $false
}
}
return $Results
}
Function Search-UnifiedAuditLogWithBackoff {
param (
[datetime]$StartDate,
[datetime]$EndDate,
[string]$UserIds
)
$Parameters = @{
StartDate = $StartDate
EndDate = $EndDate
UserIds = $UserIds
}
$results = Invoke-ExoCommandWithBackoff -CmdletName 'Search-UnifiedAuditLog' -Parameters $Parameters
return $results
}Pagination do while loop template {
$URL = $URL.TrimEnd("/")
$success = $false
$WaitTime = 30
$Retry = 5
$RetryCount = 0
$RetryCodes = @(503, 504, 520, 521, 522, 524)
while ($RetryCount -lt $retry -and $success -eq $false) {
try {
$request = [System.Net.HttpWebRequest]::Create("$URL/Companies/$id")
$request.Method = "GET";
$request.ContentType = "application/json";
$request.Headers["ApiIntegrationCode"] = $AT_API_Code;
$request.Headers["Secret"] = $AT_Secret;
$request.Headers["UserName"] = $AT_Username;
$response = $request.GetResponse();
$reader = new-object System.IO.StreamReader $response.GetResponseStream();
$result = $reader.ReadToEnd();
$response.Dispose();
$success = $true
$formatedResult = $(ConvertFrom-Json $result);
return $formatedResult.item;
} catch {
Write-Host "WARNING: $($_.Exception.Message)"
$ErrorCode = $_.Exception.InnerException.Response.StatusCode
if ($ErrorCode -in $RetryCodes){
$RetryCount++
if ($RetryCount -eq $retry) {
Write-host "WARNING: Retry limit reached."
} else {
Write-host "Waiting $WaitTime seconds."
Start-Sleep -seconds $WaitTime
Write-host "Retrying."
}
} else {
return $null;
}
}
}
return $null;
}I've a bit more to say and a script that also creates an App Registration, assigns service principal to App, adds adminagents group to the app registration if it exists [for partner portal Microsoft] For Connect.ps1 I mainly did this because of the This is so that connect.ps1 . "$PSScriptRoot\Microsoft-Extractor-Suite.psm1";
Function Connect-M365
{
versionCheck
Connect-ExchangeOnline > $null
}
Function Connect-Azure
{
versionCheck
Connect-AzureAD > $null
}
Function Connect-AzureAZ
{
versionCheck
Connect-AzAccount > $null
}
##########################################################################
# PR 1
Function Connect-ExtractorSuite([bool]$Application = $false, [bool]$DeviceCode = $false, [bool]$Delegate = $false)
{
versionCheck
if ($Application -eq $true)
{
$appID = "$env:AppId"
$appSecret = "$env:AppSecret"
$appThumbprint = "$env:AppThumbprint"
$tenantID = "$env:TenantId"
Get-Token -scope 'https://graph.microsoft.com/.default'
Check-Token -token $token
}
elseif ($DeviceCode -eq $true)
{
Connect-DeviceCode
}
elseif ($Delegate -eq $true)
{
$delegate_scopes = @('AuditLogsQuery.Read.All', 'UserAuthenticationMethod.Read.All', 'User.Read.All', 'Mail.ReadBasic.All', 'Mail.ReadWrite', 'Mail.Read', 'Mail.ReadBasic', 'Policy.Read.All', 'Directory.Read.All')
Connect-MgGraph -Scopes $delegate_scopes
}
else
{
Connect-DeviceCode
}
}
Function Get-Token($scope = ('https://graph.microsoft.com/.default', 'https://outlook.office365.com/.default'))
{
$body = @{
grant_type = 'client_credentials'
client_id = $appID
client_secret = $appSecret
scope = $scope
}
$tokenEndpoint = "https://login.microsoftonline.com/$tenantID/oauth2/v2.0/token"
$res = Invoke-RestMethod -Uri $tokenEndpoint -Method Post -Body $body
$token = $res.access_token
return $token
}
Function Check-Token($token)
{
try
{
$request = [System.Net.HttpWebRequest]::Create('https://graph.microsoft.com/v1.0/me')
$request.Method = 'GET'
$request.ContentType = 'application/json;odata.metadata=minimal'
$request.Headers['Authorization'] = "Bearer $token"
$response = $request.GetResponse()
$reader = New-Object System.IO.StreamReader $response.GetResponseStream()
$jsonResult = $reader.ReadToEnd()
$response.Dispose()
Write-Host 'MS Graph Token is valid.'
return $true
}
catch
{
Write-Warning 'MS Graph Token is invalid'
return $false
}
}
function Connect-DeviceCode
{
[CmdletBinding()]
param
(
[Parameter(Mandatory = $False)]
[string]$ClientID = '00000003-0000-0000-c000-000000000000',
[Parameter(Mandatory = $False)]
[String]$Resource = 'https://graph.microsoft.com',
[Parameter(Mandatory = $False)]
[ValidateSet('Outlook','Graph', 'AzureCoreManagement', 'AzureManagement', 'MSGraph')]
[String[]]$Client = 'MSGraph'
)
$body = @{
'client_id' = $ClientID
'resource' = $Resource
}
$authResponse = Invoke-RestMethod `
-UseBasicParsing `
-Method Post `
-Uri 'https://login.microsoftonline.com/common/oauth2/devicecode?api-version=1.0' `
-Body $body
Write-Host -ForegroundColor yellow $authResponse.Message
$continue = 'authorization_pending'
while ($continue)
{
$body = @{
'client_id' = $ClientID
'grant_type' = 'urn:ietf:params:oauth:grant-type:device_code'
'code' = $authResponse.device_code
'scope' = 'openid'
}
try
{
$tokens = Invoke-RestMethod -UseBasicParsing -Method Post -Uri 'https://login.microsoftonline.com/Common/oauth2/token?api-version=1.0' -Body $body
if ($tokens)
{
$tokenPayload = $tokens.access_token.Split('.')[1].Replace('-', '+').Replace('_', '/')
while ($tokenPayload.Length % 4) { Write-Verbose 'Invalid length for a Base-64 char array or string, adding ='; $tokenPayload += '=' }
$tokenByteArray = [System.Convert]::FromBase64String($tokenPayload)
$tokenArray = [System.Text.Encoding]::ASCII.GetString($tokenByteArray)
$tokobj = $tokenArray | ConvertFrom-Json
$global:tenantid = $tokobj.tid
Write-Output 'Decoded JWT payload:'
$tokobj
$baseDate = Get-Date -Date '01-01-1970'
$tokenExpire = $baseDate.AddSeconds($tokobj.exp).ToLocalTime()
Write-Host -ForegroundColor Green '["*"] Successful authentication. Access and refresh tokens have been written to the global $tokens variable. To use them with other GraphRunner modules use the Tokens flag (Example. Invoke-DumpApps -Tokens $tokens)'
Write-Host -ForegroundColor Yellow "[!] Your access token is set to expire on: $tokenExpire"
$continue = $null
}
}
catch
{
$details = $_.ErrorDetails.Message | ConvertFrom-Json
$continue = $details.error -eq 'authorization_pending'
Write-Output $details.error
}
}
if ($continue)
{
Start-Sleep -Seconds 3
}
else
{
$global:tokens = $tokens
}
}
@JoeyInvictus thank you very much for that. I will try to make it all a bit more uniform but for the UAL, API calls can't be uniform in the same sense as Microsoft are still using archaic outlook.com adminapi to query the Office365 Management API for UAL. I might be able to make them fairly similar though, I'll make issue / pr if I can get to what seems appropriate. Sorry if this reopens the issue as it likely is off-topic now that the PR is merged, you can close it if this happesn |
|
Thank you so much for the time and effort you put into this detailed post. I learned quite a bit, especially about memory optimizations and performance improvements. Think those are the area's we can improve the most on, since I don't have a development background, I'm just happy when I manage to get something working haha Do you mind if I use some of your ideas for improvements in the next updates? It's interesting how you are querying the audit logs, and it does look quite complex. Have you ever tested or have a rough idea of how much faster it is compared to the slower method we are currently using? I don't like the Feel free to make issues or pull requests if you think something can be improved, we really appreciate those. |
|
Yes of course anything I posted feel free to use or even from the Fork. I'll un-gitignore most stuff then update it when I am home this evening. There is a script for creating app-registrations which is pretty fool-proof as well. I just need to update it to use the Graph SDK full as there are still some calls to AzureAD methods. I'm happy to share perspective with the memory management knowledge, I just wanted to share what works for me. If you need a second look or any input just feel free to at me. As for tests, the Exchange weird way of execution was primarily to have a programmatic way of executing exchange commands because there is no true backend Exchange API , just a powershell interface to query the 365 Management API. The performance difference and speed of start-finish execution is noticeable but there would be delays due to rate-limiting, however catching these with exponential backoffs is much more efficient than using static sleeps. I am still at the office, so when I get home I will try and give you a concrete answer on this. [Here]('https://docs-cortex.paloaltonetworks.com/r/Cortex-XDR/Cortex-XDR-Pro-Administrator-Guide/Ingest-Logs-from-Microsoft-Office-365' is a good reference for the management API, it's for Cortex SOAR EDR, but it details it much better than Microsoft. I appreciate the response on |
|
@Calvindd2f Thanks I appreciate it! |
Hi Again,
Just did a PR for the
Get-AzureADAuditLogsto help acquire large sets of AzureAD Audit Logs in #69However, i think we should also add some logic where if the sytems out of memory error occurs, or this one
To try and reduce the interval, similar to the M365 scripts (vs sleep and try again). I've had a look at that error handling, but im not quite sure how you implemented it in the UAL acquisition scripts.
Once we get the logic on this one, we should also try the same logic in the Graph alternatives. I did code up the changes, but for some reason on my VM Graph API is not accepting the authentication request in V1.8.0 of the GraphAPI for PowerShell.
Last too do, might be to change the allowable date field, since we can now add time this function too :-)
The text was updated successfully, but these errors were encountered: