Building a Local macOS Endpoint Security Monitor in Swift
A source-led walkthrough of Nick's Endpoint Security extension: client ownership, AUTH deadlines, pointer lifetime, cache-first decisions, event muting, and local XPC delivery.
Apple's Endpoint Security framework gives a system extension structured process and file events close to the point where macOS makes an authorization decision. That power changes the engineering constraints. An ordinary event listener can be slow or drop work; an Endpoint Security client handling an AUTH event can suspend another process while it waits.
This article describes the implementation used by Nick, our open-source macOS security project. It is not a complete antivirus recipe. It is a practical account of the boundaries that made the monitor reliable: own the client explicitly, answer authorization events synchronously, copy C-backed data before returning, and move expensive analysis off the callback path.
The extension boundary
Nick separates the user interface from a system extension. The extension owns the es_client_t, subscribes to events, performs bounded local decisions, and sends value-typed event records to the container app through XPC.
The client wrapper has deliberately narrow responsibilities:
- create and destroy the Endpoint Security client;
- subscribe and unsubscribe from event types;
- mute the extension's own process and selected trusted paths;
- forward incoming messages to the event handler;
- respond to every authorization event using the correct response API.
The narrow wrapper matters because the Endpoint Security API is C-based and stateful. Serializing client operations on one dispatch queue avoids racing subscription, response, and shutdown calls.
let result = es_new_client(&newClient) { [weak self] _, message in
self?.eventHandler?.handle(message: message)
}
guard result == ES_NEW_CLIENT_RESULT_SUCCESS, let newClient else {
return false
}
client = newClient
Client creation can fail when the extension lacks the required entitlement or its development environment is not correctly configured. Nick reports that state to the container instead of pretending monitoring is active.
AUTH and NOTIFY are different contracts
Endpoint Security events fall into two important groups:
AUTHevents ask the client to allow or deny an operation.NOTIFYevents report an operation that has happened.
Nick subscribes to authorization events for execution, opening, memory mapping, copying, creation, renaming, and unlinking. It also observes close, rename, unlink, fork, exit, volume-mount, volume-unmount, and TCC permission-change notifications.
An authorization callback is not a suitable place to hash a large file, compile rules, run behavioral analysis, or wait for a user-interface process. The process performing the operation is suspended until the client responds. Nick therefore uses a cache-first decision path:
- Extract the target path and process metadata.
- Look up an existing local scan result.
- Respond immediately.
- Dispatch scanning, prediction, correlation, and XPC delivery asynchronously using copied values.
For an unknown executable, the current policy is intentionally fail-open: allow the first execution, analyze it off the callback queue, and use the cached result for later authorization decisions. This leaves a first-execution detection gap, but avoids turning the security extension into a source of system-wide launch stalls.
Never retain the raw message pointer
The pointer received by the callback is valid only for the callback's lifetime unless it is explicitly retained through the framework's supported mechanisms. Capturing it in an asynchronous Swift closure is therefore unsafe.
Nick copies the fields it needs synchronously:
let messageValue = message.pointee
let process = messageValue.process.pointee
let processPath = esString(process.executable.pointee.path)
let pid = audit_token_to_pid(process.audit_token)
let parentPID = audit_token_to_pid(process.parent_audit_token)
Only Swift values such as paths, identifiers, flags, and small structs cross onto the worker queue. The UnsafePointer<es_message_t> never does.
This is one of the easiest mistakes to make when wrapping Endpoint Security in Swift: closure syntax makes asynchronous capture look natural even when the underlying C object has a shorter lifetime.
Respond with the correct API
Most authorization events use es_respond_auth_result. AUTH_OPEN is different: it requires es_respond_flags_result, returning the allowed open flags. Treating every authorization event identically can leave the open operation unanswered.
Nick centralizes this distinction in the client wrapper:
if message.pointee.event_type == ES_EVENT_TYPE_AUTH_OPEN {
let flags = allow ? UInt32(bitPattern: message.pointee.event.open.fflag) : 0
es_respond_flags_result(client, message, flags, false)
} else {
let result: es_auth_result_t = allow ? ES_AUTH_RESULT_ALLOW : ES_AUTH_RESULT_DENY
es_respond_auth_result(client, message, result, false)
}
The wrapper also checks the response result and logs failures. A response call is an operation that can fail, not a fire-and-forget declaration.
Cache-first blocking and bounded scanning
Nick's ScanCache records the outcome of earlier local scans. Authorization handlers consult only that in-memory state. Expensive work happens after modified files close or after an unknown executable is first observed.
The scanner combines SHA-256 signature matching with YARA evidence. A YARA match is not automatically equivalent to confirmed malware: lower-confidence behavioral rules remain reviewable findings, while blocking is limited to results whose provenance and policy allow it. Scan timeouts and unreadable files are handled explicitly.
This separation is important:
- the authorization path answers a time-sensitive system question;
- the scanner gathers evidence;
- the policy layer decides whether that evidence may justify blocking;
- the interface explains what happened.
Conflating those responsibilities makes both false positives and deadline failures more likely.
Muting is part of correctness
A monitor can generate enormous volumes of low-value events, including events caused by its own work. Nick mutes its process after creating the client and applies path-prefix mutes for selected high-volume trusted locations after subscription.
Muting is not only a performance optimization. It prevents feedback loops and protects the useful signal from being buried under predictable platform activity. The mute list must remain conservative: broad exclusions can create blind spots, so each prefix should have a documented reason.
XPC is a reporting boundary, not an authorization dependency
The system extension starts its XPC listener before activating Endpoint Security so the container can connect as soon as monitoring begins. But authorization decisions do not depend on that connection. If the UI is closed or disconnected, the extension can still answer events from local state.
That distinction keeps presentation failure from becoming system failure. XPC carries typed event records, status changes, and user-facing evidence; it is not placed in the synchronous decision loop.
What we would keep and what we would change
The strongest decisions in the current implementation are:
- one owner for the ES client lifecycle;
- synchronous extraction and response;
- value-only asynchronous work;
- cache-first authorization;
- explicit fail-open behavior where evidence is incomplete;
- independent UI communication through XPC.
The major trade-off is the first-execution gap for an uncached file. Closing that gap safely requires analysis that is both bounded and fast enough for the authorization deadline, or a stronger trust model that avoids scanning obviously trusted binaries. It should not be "fixed" by performing unbounded hashing and behavioral work inside every AUTH_EXEC callback.
Explore the implementation
The repository contains the extension client, event handler, scanner, correlation tests, integration tests, and documented build requirements. If you are experimenting with Endpoint Security, start by understanding the event contract and pointer lifetime before adding detection logic.