Modern Log Analytics Best Practices: Redaction, Search, and Storage

1. The Compliance Risk of Logging Sensitive Data

Logging database connection strings, client email addresses, authorization headers, or plain-text credentials violates key security compliance standards (SOC2, GDPR, HIPAA). Sending these logs to cloud indexes exposes your organization to severe security and regulatory risks.

A robust log design system must scrub telemetry at the source. Secrets should never be written to disk or sent over network sockets.

2. Setting up Client-Side PII Scrubbing Rules

ObservabilityOS includes a high-performance local scrubbing engine (scrubber.ts). It runs recursive regex algorithms directly on object fields, string parameters, and arrays before they leave the application memory space.

This redacts sensitive objects (like Authorization headers or JWT tokens) at the host level. The cloud indexer only receives sanitized values.

import { createScrubber } from "@observability-os/sdk";

const scrubber = createScrubber({
  redactKeys: ["password", "token", "credit_card"],
  customPatterns: [
    { name: "SocialSecurity", regex: /\d{3}-\d{2}-\d{4}/g }
  ]
});

const cleanPayload = scrubber.scrub({
  msg: "User login failure",
  user: "alex@example.com",
  password: "super_secret_password_123"
});
// Outcome: { msg: "User login failure", user: "alex@example.com", password: "[REDACTED]" }

3. Optimizing High-Throughput Search Indices

When searching through gigabytes of logs, query performance is critical. Instead of executing recursive regex matches across raw tables, leverage Lucene-based search indexes.

Structuring your logs as flat JSON key-value blocks enables faster indexing, lowering query latency from minutes to milliseconds.