NIST Security Best Practices for Telemetry Data: How to Implement and Automate Them

NIST security best practices for telemetry data treat logs as governed records. Every record has required content, a protection requirement, and a retention period. Eight practices cover the decisions involved, from what you collect through to how you document each filter.

No single standard from the National Institute of Standards and Technology covers telemetry. The requirements sit across a controls catalog, a planning guide, and a framework. Federal agencies work to separate rules from two bodies: the Office of Management and Budget (OMB) and the Cybersecurity and Infrastructure Security Agency (CISA).

One pressure makes these decisions hard. Ingestion pricing pushes you to collect less. At the same time, compliance pushes you to keep more. Eventually, somebody asks whether cutting log volume puts a control at risk. NIST answers that question, and it turns out to allow more room than the question implies.

Three documents carry the requirements, and they work at different levels.

NIST SP 800-53 is the controls catalog. It groups logging requirements into a family called Audit and Accountability, abbreviated AU. Sixteen controls sit in that family, numbered AU-1 through AU-16. Each covers one decision, such as what to log, what a record must contain, or how long to keep it. Every control reference below points here.

NIST SP 800-92 is the log management planning guide. It was published in September 2006 and remains the final version. NIST released a revision as an initial public draft in October 2023, and the comment period closed that November. No final version has followed. Read it for process framing, and treat SP 800-53 as the operative source.

NIST CSF 2.0 states outcomes without listing controls. Subcategory PR.PS-04 covers log records generated and made available for continuous monitoring. Use CSF 2.0 for a board conversation, and SP 800-53 for an engineering decision.

Guidance here is thinner than in other compliance areas. The planning guide predates modern telemetry volumes, and its replacement has sat in draft since 2023. CISA filled part of that gap in August 2026 with a logging reference architecture. It was written for federal agencies, and CISA recommends others use it as a benchmark.

Eight practices. Each names the control that governs it.

1. Define your collection scope, and write down why. AU-2 asks you to identify what your systems can log, then specify the subset you will actually log. It also asks for a rationale explaining why that subset supports investigating an incident after the fact. Review the choice periodically. The rationale is the part teams skip, and it is the part an auditor reads first.

2. Fix the record content that must survive every filter. AU-3 defines what an event must carry to count as an audit record. Decide which fields make a record valid in your environment, then protect them from removal. Any filter that strips one of those fields breaks the control.

3. Reduce at the pipeline edge, before ingestion. Empty values, placeholder patterns, duplicated fields, and events no detection could match all cost money and add nothing. Remove them before your SIEM platform indexes them. Mask tokens, credentials, and secrets at the same point, so downstream tools never receive them.

4. Normalize to a standard schema before ingestion. AU-12(2), titled Standardized Formats, calls for records in formats that support analysis. The Advanced Security Information Model (ASIM), Open Cybersecurity Schema Framework (OCSF), Elastic Common Schema (ECS), Common Information Model (CIM), and Unified Data Model (UDM) are the schemas security teams use. Records an analyst cannot compare across sources are records nobody correlates, which puts AU-6 review quality at risk.

5. Normalize time to a single reference. AU-8 requires time stamps that map to a known reference, so records from different systems can be ordered. Two systems five minutes apart produce a timeline that reads as unrelated events.

6. Protect records in transit, and control who changes pipeline logic. AU-9 covers protection of audit information from unauthorized access, modification, and deletion. Encrypt telemetry in flight. Persist events to disk before processing, so a crash loses nothing. Log every configuration change.

7. Split the stream. AU-4 concerns storage capacity, and AU-11 concerns retention. Reduced, detection-relevant data to the SIEM and the complete stream to cheap object storage satisfies both, at very different unit costs.

8. Document every rule, and keep the decision reviewable. An auditor asks why an event is missing. You need a written answer and the location of the record. AU-6 depends on records an analyst can actually review.

NIST does not expect you to log everything. AU-2 makes your collection set a deliberate subset of what your systems can emit, chosen with a documented rationale.

Four categories are usually safe to remove.

Empty and placeholder values. Vendor log formats carry large numbers of null fields and filler patterns. Values like “-“, “N/A”, and “0x0” occupy storage and add nothing.

Duplicated fields. Many sources emit the same value under two or three field names. One copy is enough.

Fields that no detection or report reads. If no rule, dashboard, or scheduled report references a field, indexing it buys nothing.

Events no rule could match. Some event types will never trigger any detection you run. Verbose informational logging is the usual example.

The last two categories are the ones worth automating. A hand-maintained list of fields and event types falls out of date as detections change.

Detection need and audit-record need are different sets. That distinction is behind many compliance failures in log reduction.

An event that no detection matches can still be a required audit record. Dropping it may be defensible under your AU-2 rationale. It still breaks AU-11 if your retention policy covers that record.

A universal field list would mislead you, because required content depends on your system categorization and your own control baseline. Derive yours in three steps.

Review the list whenever your control baseline changes.

Send the same telemetry to two places, with different content in each. Reduced and normalized data goes to the SIEM. Raw, complete events go to low-cost object storage.

The economics carry the argument. SIEM ingestion is priced per GB indexed. Object storage is priced per GB stored, at a small fraction of that. Sending everything to both places is the expensive option, and it is rarely necessary.

AU-7 is worth reading closely here, because its wording shapes the design. It calls for a reduction capability that does not alter the original content or time ordering of audit records. A reduced view in the SIEM alongside an unaltered original in the archive fits that. A reduced copy with no preserved original does not.

So link the two tiers. An analyst who finds something in a reduced SIEM record needs a path back to the complete original. Without that path, the reduced copy is the only record anyone can reach. Build the link before you enforce any reduction. Adding it later leaves a gap across everything already processed.

Then work source by source.

Two rules make the table usable. Anything in an audit obligation class reaches the SIEM whole. Anything in a detection window class is a candidate for reduction, with your keep-list applied. Record the reasoning per row, because that record becomes your AU-2 rationale.

The table is a policy. Turning it into a running configuration is covered below.

AU-11 sets retention to an organization-defined period, consistent with your records retention policy. NIST specifies no number. Readers expecting one will not find it in SP 800-53.

Three retention classes are usually enough.

Routing decides which class a stream joins. Retention stops being one global setting.

Federal policy gives concrete figures, and they moved this year. OMB Memorandum M-26-14 requires logs searchable for six months and retrievable for one year. It rescinded M-21-31, which required twelve months hot and eighteen months cold. Federal agencies and their suppliers work to further deadlines and a five-level maturity model. Both are set out in M-26-14 and the CISA reference architecture.

One trade-off is worth naming. Mandiant reported a global median dwell time of 14 days for 2025. Espionage and North Korean IT worker cases ran far longer, at a median of 122 days. Some of those intrusions went undetected for over a year. A one-year retrievable window reconstructs a recent incident. It may not reach the start of a slow campaign.

Five steps, in order.

Steps 1 and 2 need no tooling. They are also the first things an auditor asks for.

Meeting NIST logging requirements by hand takes steady effort from a security team. A team member updates the keep-list each time a detection changes. Someone checks that no filter has started dropping a required field. Another person writes the rationale, then reviews it next quarter. The work never finishes, because sources and detections keep changing.

Much of that work can be automated in a layer called a security data pipeline. VirtualMetric DataStream is one example. The platform sits between your log sources and your destinations. It collects telemetry, normalizes it, applies your filters, and routes each record to the right place.

DataStream Smart Engine registers the detection rules your security operations center already maintains. It then forwards only the events and fields those rules could need. When the detections change, the filter policy follows, so there is no separate filter list to update.

Layered reduction runs from 50 to 90% of ingested volume. That range combines field-level filtering, event filtering, and sampling. Field-level filtering alone accounts for roughly 55 to 60%.

Reduction holds up only while the complete original stays reachable. DataStream appends a correlation identifier to each event during initial processing, before any optimization runs. The reduced record in the SIEM and the complete original in the archive carry the same identifier. An analyst who finds something in the SIEM uses that identifier to pull the original.

Want to see how DataStream can help you meet your NIST logging requirements? Book a technical session.

No single standard exists. The operative requirements sit in the Audit and Accountability control family of NIST SP 800-53 Revision 5. SP 800-92 adds process guidance, and the Cybersecurity Framework 2.0 sets outcome-level expectations.

SP 800-92 was published in September 2006 and remains the final version. NIST released Revision 1 as an initial public draft on 11 October 2023. Its comment period closed on 29 November 2023, with no final version since.

Control AU-11 concerns the retention period and leaves the storage tier open. Archived records satisfy it while they stay complete and retrievable inside the window your policy names. Federal agencies work to six months searchable and one year retrievable.

No. The AU controls are technology-neutral and never name a product category. NIST does define security information and event management in its glossary, drawing on SP 800-92 and SP 800-128. No control requires you to run one.

Yes. AU-2 expects your logged set to be a subset of what systems can emit, chosen with a documented rationale. AU-3 and AU-11 set the limits, defining the record content and retention period that must survive.

Not on its own. AU-7 is titled Audit Record Reduction and Report Generation, and the title invites a misreading. The control text calls for a capability that does not alter the original content or time ordering of audit records. It governs records you already hold, and the collection scope is decided under AU-2.