Intelligent data pipeline for Microsoft Sentinel

DataStream reduces log volume through a layered approach. By default, field-level optimization removes empty values, placeholder fields, and operational metadata that Sentinel analytics rules never reference, achieving 55–60% reduction with zero security risk. For higher savings, optional event-level filtering and statistical sampling can bring total reduction to 70–90%, with security-critical events always protected. Full raw logs are simultaneously routed to low-cost storage (Azure Blob, ADX, Sentinel Data Lake) with a Correlation ID, so analysts can retrieve complete records for forensic investigations when needed. 

Read more: How to Reduce SIEM Costs Without Losing Security Visibility

DCRs and AMA handle ingestion and basic KQL-based transformations within Azure, but they have significant limitations. They can’t collect from agentless sources or systems without AMA support, they don’t normalize data to ASIM across diverse source types, and they send everything to a single destination. DataStream operates before data reaches Azure: it collects from any source via agentless WinRM/SSH or standard protocols, applies vendor-specific ASIM normalization with 170+ processors, optimizes volume before it ever hits Sentinel’s billing meter, and routes different data types to the right destination based on security value.

DataStream handles ASIM mapping automatically: when logs arrive from a supported source, the multi-schema processing engine applies vendor-specific field mappings, validates the output against ASIM schema requirements, and routes the normalized data to the correct Sentinel tables. No manual parser writing or field mapping is required for supported sources.

DataStream supports both agentless and agent-based collection. Agentless collection connects directly via WinRM (Windows) or SSH (Linux, macOS, Solaris, AIX) with no software installation. For network devices, OT/ICS systems, and security appliances, it supports Syslog, CEF, LEEF, and REST APIs. Pre-built content packs cover widely used vendors – Fortinet, Palo Alto, Check Point, CrowdStrike, CyberArk, Zscaler, and more – each activating automatically when logs from that vendor are detected.

DataStream applies policy-based redaction and masking directly in the pipeline, before data leaves your environment. You define rules through a no-code UI to automatically remove or obfuscate sensitive fields: usernames, passwords, tokens, PII such as email addresses and phone numbers, or any custom field. Redaction is applied consistently across all incoming data, eliminating the risk of uneven coverage from manual processes. The structure and security context of each log remain intact, so detection rules and correlations in Sentinel continue to work accurately. Policies are designed to support GDPR, HIPAA, and PCI DSS requirements, and redacted data streams are audit-ready. 

Security Copilot unlocks its full potential when your Sentinel data is ASIM-compliant, normalized to a consistent schema that Copilot’s AI can query across all sources using natural language. Without ASIM, Copilot works with fragmented, inconsistently structured data, limiting the accuracy of detections, investigations, and automated responses. DataStream automatically maps all ingested data to ASIM before it reaches Sentinel, so every source from legacy on-prem systems to cloud services is immediately ready for Copilot’s AI-driven workflows without any additional setup. 

Initial deployment takes under 30 minutes. DataStream’s guided setup automatically handles Azure authentication, Log Analytics workspace creation, and DCR/DCE configuration, no manual Azure setup required. A live demo by our solution engineer shows a complete Sentinel integration in 13 minutes. Watch it on YouTube.