Clustered Directors, Pipeline Debugging, and More Integrations

Over the past two months, VirtualMetric DataStream delivered a substantial update cycle focused on resilience, productivity, and platform extensibility. This release strengthens the core architecture, makes pipeline development and troubleshooting significantly easier, and expands integration coverage across schemas, SIEMs, and cloud platforms. 

Let’s take a closer look. 

DataStream Directors now support clustered deployments designed for high-availability environments. Directors can be deployed in odd-number clusters (3, 5, 7, and so on), enabling quorum-based decision making and resilient operation during node failures. 

Configuration is managed centrally at the cluster level and automatically synchronized across all members. If an individual Director becomes unavailable, processing and log collection continue without interruption. This provides the reliability required for mission-critical security pipelines where data loss or downtime is not acceptable. 

Pipeline development and troubleshooting are now significantly more interactive with the introduction of the Pipeline Debugger. 

The debugger provides a dedicated environment where pipelines can be tested in real time without deployment. Teams can load pipelines directly from the Content Hub or test custom configurations, execute them step by step, and inspect how input data is transformed at each stage. Detailed error information highlights exact failure points, making it easier to diagnose parsing issues, transformation errors, or logic flaws. 

This shortens development cycles, reduces trial-and-error deployments, and helps teams move faster when building or refining pipelines. 

Security controls for Director deployments have been strengthened with token-based authentication and network access restrictions. 

Administrators can now manage Director authentication credentials centrally, rotate tokens, and revoke access when needed. Network-level restrictions allow Directors to communicate only with approved platform endpoints, tightening control over distributed deployments and reducing attack surface. 

These enhancements are especially relevant for environments with strict network segmentation or regulated operational models. 

The Content Hub has been significantly expanded with 23 new pipeline packs, extending coverage across schema transformation, encoding, SIEM automation, and platform-specific use cases. 

New schema transformation packs support mappings between ASIM, CSL, CEF, LEEF, OCSF, and UDM, simplifying normalization across heterogeneous environments. Encoding packs make it easier to output data in standard SIEM-friendly formats, while new SIEM automation and normalization packs broaden out-of-the-box support for platforms such as Google Security Operations, Splunk, Elastic, Sumo Logic, Datadog, Rapid7, and others. 

Additional Microsoft Sentinel-specific packs improve normalization and automation for both ASIM and CSL workflows, reducing the need for custom pipelines.

DataStream now supports 16 additional target destinations, extending routing capabilities across cloud storage, streaming platforms, and security operations tools. 

New integrations include object storage platforms such as Alibaba Cloud OSS, Backblaze B2, DigitalOcean Spaces, and Scaleway, along with expanded AWS and Azure ecosystem support. Security-focused targets now include Google Chronicle, Google SecOps, Elastic Security, and Splunk Enterprise Security, enabling more flexible routing strategies across multi-platform SOC environments. 

These additions make it easier to route high-value telemetry to analytics platforms while offloading bulk or compliance data to cost-efficient storage. 

As the number of supported destinations grows, target selection has been refined with vendor-based categorization and filtering. Targets are now grouped by provider (for example, AWS, Azure, Google Cloud, and security platforms), allowing teams to quickly identify relevant destinations and reduce configuration time when building or updating routes. 

These updates focus on making DataStream more resilient, easier to operate, and more adaptable as security architectures continue to diversify. High availability, safer deployments, faster pipeline iteration, and broader integration coverage all support the same goal: keeping security data reliable, usable, and cost-efficient at scale. 

In upcoming releases, we’ll continue expanding automation, refining performance, and adding support for new sources and destinations. 

For a complete list of changes, including fixes and minor improvements not covered here, refer to the documentation and release notes.

As always, feedback from real-world deployments directly shapes what we build next. If you’d like a walkthrough of any of these features or want to suggest improvements, feel free to reach out.