Decoupling Audit Logging: Why We Built an Asynchronous Emitter

When building a high-throughput, latency-sensitive access platform like Mezite, every millisecond counts. Establishing an SSH connection, proxying a reverse tunnel, or authenticating a user involves a complex orchestration of cryptography, networking, and state management.

Simultaneously, compliance and security requirements dictate that we maintain a rigorous audit log. Every significant action—login attempts, session starts, certificate issuances, and cluster joins—must be immutably recorded in a database.

If we simply execute a synchronous INSERT into PostgreSQL or SQLite every time an event occurs in the hot path, we introduce a catastrophic bottleneck.

The Blocking Write Problem

Consider the naive approach: a user connects to the Mezite proxy (mezhub). The proxy validates the TLS connection, performs the ALPN handshake, and routes the SSH session to the appropriate agent. It then logs a session.start event by blocking the request goroutine to execute a database write.

If the database is under load, or if there is transient network latency between the proxy and the database server, that synchronous write blocks the entire operation. This means session establishment stalls. In a high-traffic environment, a slow database can cascade into connection timeouts, exhausted connection pools, and a complete system lockup.

We refuse to let a logging system degrade the core user experience.

The Asynchronous Emitter Pattern

To solve this, we architected the Emitter. The Mezite Emitter decouples the generation of an audit event from its persistence in the database.

At its core, the Emitter relies on Go’s powerful concurrency primitives. Instead of writing directly to the database, the caller submits the event to a buffered channel.

// Emitter handles writing audit events to the database.
type Emitter struct {
db          *dbutil.DB
clusterName string
logger      *zap.Logger
events      chan *AuditEvent
// ... synchronization primitives
}

When an event occurs in the fast path, Mezite calls Emit(). This function performs a non-blocking send to the events channel. The size of this channel buffer is intentionally large (defaulting to 4096) to absorb sudden spikes in traffic.

Failing Securely Under Extreme Load

What happens if the system is subjected to an extreme load spike and the 4096-event buffer fills completely?

We have a strict architectural invariant: audit logging must not break routing or access. The Emit() method utilizes a select statement with a default case. If the channel is full, we log a warning via our fast, structured logger (Zap) and intentionally drop the event.

func (e *Emitter) Emit(event *AuditEvent) {
// ... initialization logic
select {
case e.events <- event:
default:
	e.logger.Warn("audit buffer full, dropping event",
		zap.String("type", event.EventType))
}
}

This is a deliberate tradeoff. In a situation where the database is fundamentally broken and cannot accept writes, we prioritize keeping the infrastructure accessible over blocking indefinitely and causing a cluster-wide outage.

Batched Database Writes

A dedicated background goroutine continuously reads from the events channel. It does not perform an INSERT for every single event. Instead, it aggregates them into batches (up to 100 events) or flushes them on a timer (every 500 milliseconds), whichever comes first.

By executing batched database inserts, we dramatically reduce the overhead on PostgreSQL or SQLite, allowing the logging system to scale effortlessly alongside the proxy.

The Exception: EmitSync

While most audit logging benefits from asynchronous batching, there are critical events where we must guarantee persistence before proceeding.

For example, when the Certificate Authority issues a new short-lived SSH certificate, we need absolute certainty that the issuance event is recorded. If the node crashes immediately after issuing the certificate but before the async buffer flushes, we would have an unrecorded, valid certificate existing in the wild.

For these specific scenarios, the Emitter provides an EmitSync() method.

// EmitSync writes an audit event directly to the database, blocking until
// the write completes. Use this for critical events such as login and
// certificate issuance.
func (e *Emitter) EmitSync(ctx context.Context, event *AuditEvent) error {
// ... initialization
if err := e.writeEvent(ctx, event); err != nil {
	e.logger.Error("writing audit event", zap.Error(err))
	return err
}
return nil
}

EmitSync() bypasses the channel entirely and blocks until the database confirms the write. We use it for security-relevant events that must not be lost on buffer overflow — denials, identity and trust mutations, certificate and CA events, and authentication outcomes. These are low-volume tail events, so the latency cost of a synchronous write is a worthwhile trade for guaranteed durability.

By bifurcating our logging strategy—asynchronous batching for the fast path and synchronous writes for critical security operations—the Emitter ensures Mezite remains blisteringly fast without compromising compliance.