Intelligent data pipeline for Splunk

Splunk licenses on ingested data volume, so cost scales directly with how much you send. DataStream reduces that volume through a layered approach. By default, field-level optimization removes empty values, null fields, and operational metadata that Splunk analytics rules never reference, achieving 55–60% reduction with no security risk. Optional event-level filtering and statistical sampling can push total reduction to 70–80%, with security-critical events always protected. Full raw logs are simultaneously routed to low-cost storage (AWS S3, Azure Blob, SmartStore) with a Correlation ID, so analysts can retrieve complete records for forensic investigations when needed. 

Read more: How to Reduce SIEM Costs Without Losing Security Visibility

Universal Forwarders and props/transforms.conf handle basic filtering and field extraction within the Splunk stack, but they have significant limitations. They can’t collect from agentless sources or systems without a forwarder installed, they don’t normalize logs to CIM across diverse source types automatically, and they send everything to a single Splunk destination.  

DataStream operates before data reaches Splunk: it collects from any source via agentless collection (WinRM/SSH) or agents where needed, applies vendor-specific CIM normalization with no manual TA development, reduces volume before it hits your license meter, and routes different data types to the right destination: Splunk Indexer, SmartStore, AWS S3, or Azure Blob based on security value. 

Splunk’s CIM normalizes logs from different sources to a common schema so that a single correlation search can query e.g. Windows events, Palo Alto firewall logs, and CrowdStrike telemetry using the same field names. Traditionally, this requires writing or maintaining Technology Add-ons (TAs) – a process that can take days per source type and breaks every time a vendor changes their log format. DataStream handles CIM mapping automatically using vendor-specific optimization packs developed through analysis of real-world security operations. Each pack maps source fields to CIM data models at ingest time, with no manual regex authoring or TA maintenance required. The normalization is deterministic and fully auditable: every field mapping decision is documented and traceable, not AI-generated. 

DataStream supports both agentless and agent-based collection. Agentless collection connects directly via WinRM (Windows) or SSH (Linux, macOS, Solaris, AIX) with no software installation. For network devices, OT/ICS systems, and security appliances, DataStream supports standard collection protocols including Syslog, as well as common log formats such as CEF and LEEF, and REST APIs for cloud-based sources. Pre-built content packs cover all widely used vendors such as Fortinet, Palo Alto, Check Point, CrowdStrike, CyberArk, Zscaler, and more; each activating automatically when logs from that vendor are detected, with no manual TA configuration needed. 

DataStream applies policy-based redaction and masking in the pipeline before data leaves your environment. You define rules through a no-code UI to automatically remove or obfuscate sensitive fields: usernames, passwords, tokens, PII such as email addresses and phone numbers, or any custom field. Redaction is applied consistently across all incoming data. The structure and security context of each log remain intact, so Splunk correlation searches and ES detection rules continue to work accurately. Policies are designed to support GDPR, HIPAA, and PCI DSS requirements, and redacted pipelines are audit-ready. 

Yes, multi-destination routing is a core capability. DataStream can simultaneously send CIM-normalized, security-relevant events to Splunk Indexer or Splunk Cloud for real-time analytics; full data volume to Splunk SmartStore; and raw logs in Parquet format to AWS S3 or Azure Blob Storage for long-term retention at a fraction of Splunk licensing cost. Each destination receives the appropriate data tier based on security value, with a Correlation ID linking optimized Splunk data back to complete raw logs in archive storage for forensic investigations. 

Initial deployment takes under 30 minutes. DataStream’s guided setup automatically handles authentication, HEC configuration, and pipeline routing — no manual Splunk infrastructure changes required. A live demo by our solution engineer shows a complete integration in 13 minutes, without cuts. Watch it on YouTube.