Intelligent data pipeline for Elastic

Elasticsearch pricing scales with data volume stored and indexed. DataStream reduces that volume through a layered approach. By default, field-level optimization removes empty values, null fields, and operational metadata that Elasticsearch analytics rules never reference, achieving 55-60% reduction with no security risk. Optional event-level filtering and statistical sampling can push total reduction to 70-90%, with security-critical events always protected. Full raw logs are simultaneously routed to low-cost cloud storage (AWS S3, Azure Blob, Google Cloud Storage) with a Correlation ID, so analysts can retrieve complete records for forensic investigations when needed. 

Read more: How to Reduce SIEM Costs Without Losing Security Visibility

Elasticsearch Ingest Pipelines and Beats handle ingestion and basic transformations within the Elastic Stack, but they have significant limitations. Beats can’t collect from agentless sources or systems without agent installation, Ingest Pipelines don’t normalize data to ECS across diverse source types automatically, and both send everything to a single Elasticsearch cluster. DataStream operates before data reaches Elasticsearch: it collects from any source via agentless collection (WinRM/SSH) or agents where needed, applies vendor-specific ECS normalization using the native Elasticsearch Ingest Pipeline format, reduces volume before it ever hits Elasticsearch’s storage meter, and routes different data types to the right destination based on security value. 

Elastic Common Schema (ECS) provides a standardized structure for security events. DataStream handles ECS mapping automatically: when logs arrive from a supported source, the multi-schema processing engine applies vendor-specific field mappings using the native Elasticsearch Ingest Pipeline format, validates the output against ECS schema requirements, and routes the normalized data to Elasticsearch. No manual parser writing or field mapping is required for supported sources. 

DataStream supports both agentless and agent-based collection. Agentless collection connects directly via WinRM (Windows) or SSH (Linux, macOS, Solaris, AIX) with no software installation. For network devices, OT/ICS systems, and security appliances, it supports Syslog, CEF, LEEF, and REST APIs. Pre-built content packs cover all widely used vendors – Fortinet, Palo Alto, Check Point, CrowdStrike, CyberArk, Zscaler, and more – each activating automatically when logs from that vendor are detected. 

DataStream applies policy-based redaction and masking in the pipeline, before data leaves your environment. You define rules through a no-code UI to automatically remove or obfuscate sensitive fields: usernames, passwords, tokens, PII such as email addresses and phone numbers, or any custom field. Redaction is applied consistently across all incoming data, eliminating the risk of uneven coverage from manual processes. The structure and security context of each log remain intact, so Elasticsearch detection rules and correlations continue to work accurately. Policies are designed to support GDPR, HIPAA, and PCI DSS requirements. 

Yes, multi-destination routing is a core capability. DataStream can simultaneously send ECS-normalized, security-relevant events to Elasticsearch or Elastic Cloud for real-time analytics, full data volume to Elasticsearch storage tiers for long-term retention, and raw logs in JSON or Parquet format to AWS S3, Azure Blob Storage, or Google Cloud Storage for archival at a fraction of Elasticsearch’s storage cost. Each destination receives the appropriate data tier based on security value, with a Correlation ID linking optimized Elasticsearch data back to complete raw logs in archive storage. 

Initial deployment setup takes under 30 minutes. DataStream’s guided configuration automatically handles Elasticsearch authentication, endpoint configuration, and pipeline routing – no manual Elasticsearch cluster reconfiguration required. The actual time-to-value depends on the complexity of your data sources and normalization requirements. Contact our team for a technical consultation on your specific deployment timeline.