The challenge
Filter noise before it becomes cost
SOC teams are buried under duplicate, redundant, and low-value logs that inflate SIEM costs and hide real threats. Relying on brittle scripts and one-off regex rules creates overhead, slows down detection, and fails to scale in enterprise environments.
The Solution
DataStream – intelligent security data pipeline platform
Filter with confidence
DataStream removes low-value events and redundant fields in real time using clear, transparent policies. You keep the information needed for detections and investigations, while dropping the rest before it becomes costly to store, process, or search.
Remove only what detections never use
DataStream’s Smart Engine removes fields and events that your SIEM detection content never uses. It works from the detection rules you assign, and keeps every field they read.
Three levels of reduction
Field-level filtering removes redundant and low-signal fields, typically cutting ingest by 55–60%. Event-level filtering adds rules you set, such as event IDs, datasets, IPs or known-benign activity, bringing the total to up to 90%. Smart Engine runs both levels from your own detection rules and reaches up to 90% and higher.
No-code configuration
Reduction policies are built in the UI using contextual filters and processors (not regex, brittle rules, or custom scripts). Teams can tune policies quickly as environments evolve, without introducing operational overhead.
Key benefits
Why this approach works
Frequently asked questions
How much reduction can we expect?
With Smart Engine, production deployments reduce SIEM ingest by up to 90% and higher. Teams that prefer manual configuration can apply two-step filtering: field-level trimming typically cuts 55–60%, and event filters bring the total up to 90%. Exact results depend on your sources and policies.
What is Smart Engine?
Smart Engine is a DataStream feature that reduces SIEM ingestion based on your detection rules, removing the fields and events they never use while keeping full detection coverage.
Can we keep certain events unfiltered?
Yes. You can create exceptions/allow-lists by source, dataset, severity, or field to ensure critical logs always pass.
How can we handle sensitive data?
Use policy-based masking and redaction to remove or obfuscate fields like credentials or PII before forwarding.
Do we need to write scripts?
No. Policies and vendor packs control filtering; you tune them in the UI instead of maintaining custom code.
Does filtering impact system performance?
Filtering is applied in the pipeline using vectorized processing. Benchmarks show no measurable latency even at 150k EPS.
Get DataStream in Microsoft Security Store
Deploy DataStream in minutes with Azure Managed Identity support built in. No credential management, no manual setup.