DjangoPlay — audit App

Runtime dependency webapp/core/events/ and webapp/core/execution_context/, webapp/paystream/app_settings/celery.py, and webapp/devtools/management/commands/cleanup_audit_events.py. Verbose name: audit · Registered as "audit.apps.AicoreConfig" in INSTALLED_APPS

#1. Summary

Key properties, enforced directly in code:

• Append-only / immutable: AuditEvent has no FK relationships to domain models (fully denormalized), no soft-delete, no update path used anywhere in the codebase — only create().
• Decoupled from business logic: audit_persistence_subscriber is invoked asynchronously-in-process by the event dispatcher, wrapped in its own try/except so a persistence failure never bubbles into the transaction that triggered it, and dispatch failures are only logged (core/events/runtime/dispatching.py).
• Not every model, not every event is audited — persistence is governed by an explicit policy (core.events.policies.should_persist_event) tied to a curated allow-list of ~40 models (audit.lifecycle.registry.AUDIT_TRACKED_MODELS).
• Fail-closed on bad event names: malformed event names are dropped before dispatch (core.events.emitter.emit).
• Recursion-safe: automatic model-lifecycle tracking uses a ContextVar-based guard so that the audit write itself never re-triggers pre_save/post_save auditing.
• Governance layer for what actually gets stored: sensitive keys are redacted, values are recursively truncated/depth-limited before being written into the metadata JSONField.

#2. Architecture

#2.1 audit is a subscriber, not a service other apps call

DjangoPlay's actual event architecture (verified in core/events/) is a lightweight, in-process pub/sub system, not a queue:

Business code / Django signal
        │
        ▼
core.events.helpers.emit_event(name=..., target=..., metadata=..., category=..., severity=...)
        │
        ▼
core.events.builders.build_event()  →  core.events.contracts.DomainEvent  (dataclass)
        │
        ▼
core.events.emitter.emit(event)
        │  - validates event name via audit.governance.validation.validate_event_name
        │  - iterates core.events.registry.get_subscribers()
        ▼
core.events.runtime.dispatching.safely_dispatch_event()  (per-subscriber try/except + timing + slow-subscriber warning)
        │
   ┌────┼─────────────────────┬───────────────────────────┐
   ▼                          ▼                            ▼
audit.subscribers.persistence   core.events.subscribers.logging   core.telemetry.subscribers
.audit_persistence_subscriber   .logging_subscriber                .telemetry_subscriber
   │                          │                            │
   ▼                          ▼                            ▼
AuditEvent row (DB, if      structured log line          in-memory counters
policy allows persistence)                                (event/category/severity volume,
                                                            anomaly hook)

All three subscribers are registered once, idempotently, in AuditConfig.ready() (audit/apps.py) — this is the actual wiring point, not core's own AppConfig. ready() guards against duplicate registration with both a module-level _SUBSCRIBERS_REGISTERED flag and a get_subscribers() membership check.

#2.2 Two independent ways events reach audit

1. Automatic model lifecycle tracking (audit/lifecycle/tracking.py) — global Django pre_save/post_save receivers (attached to every model, filtered in-handler) that:

   • On pre_save of an existing row: snapshot the "before" state if the model is in AUDIT_TRACKED_MODELS.
   • On post_save: if created, emit <model_name>.created; if updated, diff the before/after snapshots and — only if there's an actual field-level change — emit <model_name>.updated with a changes metadata payload ({field: {before, after}}).
   • Soft-delete/restore are not covered by pre_save/post_save (a soft delete is normally an UPDATE, not a DELETE) — instead audit/signals/lifecycle.py listens on two custom Django signals, post_soft_delete and post_restore (defined in audit/signals/events.py), which other apps' soft-delete mixins presumably fire, and emits <model>.deleted / <model>.restored.

2. Manual/explicit emission for things that aren't a model save:

   • Auth events (audit/security/auth.py) — hooks Django's built-in user_logged_in, user_logged_out, user_login_failed signals → emits auth.login, auth.logout, auth.login_failed (the last one at warning severity, capturing only the attempted username, not the password).
   • Impersonation (audit/security/impersonation.py) — emit_impersonation_started / emit_impersonation_stopped, called by whatever app implements "login as user" (not owned by audit; these are just emitter helpers for that feature to call into).
   • Permission changes (audit/security/permissions.py) — emit_permission_change_event, presumably called from policyengine.
   • Admin actions (audit/admin_events/*.py) — emit_bulk_action_event, emit_export_event, emit_admin_mutation_event; helper functions other apps' Django-admin customizations can call for bulk actions/exports/manual admin edits.

#2.3 The persistence decision is dynamic, not hardcoded per-event

This is the least obvious but most important architectural detail: core.events.policies.should_persist_event() does not hardcode a list of persisted event names. It:

• Rejects an exact-match denylist (email.queued, email.sent, email.delivered, cache.refreshed, task.started, task.completed) and anything containing .email_, .task_, .cache_.
• Otherwise derives a set of allowed prefixes at call time from audit.lifecycle.registry.AUDIT_TRACKED_MODELS by lower-casing each model's class name and appending . (e.g. helpdesk.SupportTicket → supporticket.... actually supportticket.), plus two manually added static prefixes: auth. and admin..
• An event is persisted only if its name starts with one of those derived prefixes.

Practical implication: adding a model to AUDIT_TRACKED_MODELS in audit/lifecycle/registry.py is both what turns on automatic diff-tracking for that model AND what makes any <modelname>.* event (from anywhere in the codebase, not just the automatic tracker) eligible for persistence. The registry is the single source of truth for both concerns.

#2.4 Severity resolution

If a caller doesn't pass severity explicitly to emit_event, core.events.severity.resolve_event_severity() derives it from the suffix of the event name (created/updated/restored → info, deleted/failed/error → warning, login_failed → warning, permission_changed/payment_failed → critical, refund_issued → warning). Unrecognized suffixes default to info. This is a simple lookup table, not a rules engine — new "high-severity" event types must be added here manually or they'll silently default to info.

#3. Implementation Details

#3.1 Package layout

audit/
├── apps.py                # AuditConfig — subscriber registration in ready()
├── constants.py           # AUDIT_ADMIN_ROLES (see §7 — appears unused elsewhere)
├── exceptions.py           # empty
├── models/
│   └── audit_event.py     # AuditEvent — the only model in this app
├── lifecycle/              # automatic model create/update/delete/restore tracking
│   ├── constants.py        # TRACKED_EVENT_* action name constants (declared, not directly referenced elsewhere)
│   ├── registry.py         # AUDIT_TRACKED_MODELS allow-list + is_model_tracked()
│   ├── tracking.py         # pre_save/post_save receivers
│   ├── diffing.py          # calculate_diff(before, after)
│   ├── snapshots.py        # capture_instance_snapshot(instance)
│   ├── serializers.py      # serialize_field_value() — per-field JSON-safe coercion
│   ├── exclusions.py       # AUDIT_EXCLUDED_FIELDS (updated_at, created_at, search_vector, last_login, deleted_at)
│   └── recursion.py        # ContextVar re-entrancy guard
├── governance/              # what actually lands in AuditEvent.metadata
│   ├── constants.py         # SENSITIVE_KEYS, size/depth limits
│   ├── pii.py                # redact_sensitive_value()
│   ├── sanitization.py       # sanitize_metadata() — entry point used by the persistence subscriber
│   ├── serialization.py      # safe_serialize() — recursive, depth/size bounded
│   ├── truncation.py         # truncate_string()
│   ├── retention.py          # delete_expired_audit_events() — 365-day batched cleanup
│   └── validation.py         # validate_event_name()
├── security/                 # manual security-event emitters
│   ├── auth.py                # login/logout/login_failed signal receivers
│   ├── impersonation.py       # emit_impersonation_started/stopped()
│   └── permissions.py         # emit_permission_change_event()
├── signals/
│   ├── events.py              # post_soft_delete, post_restore (custom Django Signal objects)
│   └── lifecycle.py            # receivers for the above → emits deleted/restored events
├── subscribers/
│   └── persistence.py          # audit_persistence_subscriber — the actual DB write
├── admin_events/                # emitter helpers for Django-admin bulk/export/manual actions
│   ├── bulk_actions.py
│   ├── exports.py
│   └── mutations.py
├── admin.py                     # AuditEventAdmin — read-mostly changelist UI
├── tasks.py                     # cleanup_expired_audit_events (Celery task)
├── migrations/                  # 0001–0006
└── tests.py                     # empty — no test coverage currently in this app

#3.2 Django app registration quirk

audit/apps.py contains a fully commented-out second AuditConfig class (older version, verbose name "Logging Journal", only wiring audit.signals.lifecycle) left in the file below the active class. It's dead code, not a toggle — the active class is the one Django loads. Worth cleaning up but not a functional risk.

#3.3 Dependencies

audit depends on:

• core.events.* — the entire pub/sub contract, dispatcher, and policy/severity resolution it relies on.
• core.execution_context.* — reads current-request actor, client_ip, request_id, user_agent via ContextVars populated by core.execution_context.middleware.ExecutionContextMiddleware (or, for management commands, via set_management_command_context(), which stamps a SystemActor and 127.0.0.1).
• core.telemetry.* and core.events.subscribers.logging — siblings on the same event bus, not called directly by audit, but registered alongside it in AuditConfig.ready().
• utilities.admin.* and users.views.ui.errors.custom_403 — for the admin UI only.

Nothing in audit imports business-domain apps (invoices, teamcentral, etc.) directly — the coupling runs the other way, through the AUDIT_TRACKED_MODELS string registry, which is intentionally string-based ("app_label.ModelName") precisely so audit never has to import those apps.

#4. Data Model — AuditEvent

Single model, table audit_event, BigAutoField PK, default ordering -occurred_at.

Indexes beyond the per-field ones: composite (action, occurred_at) and (target_type, target_id) — clearly tuned for "show me the history of this action type over time" and "show me everything that happened to this specific record" queries, which is exactly what the admin changelist filters expose.

Meta.managed = True is explicit (slightly redundant but harmless). Two managers exist: default objects (implicit) is not overridden — the model only explicitly defines all_objects = models.Manager(), which is what the persistence subscriber and admin both use (AuditEvent.all_objects.create(...), self.model.all_objects.all()), i.e. there is no soft-delete manager filtering here; all_objects is really just "the plain manager," named defensively.

#5. Target & Actor Labeling

Two label-resolution systems live in core.events, used by the persistence subscriber to fill actor_* / target_*:

• Actor (core.events.actors.serialize_actor): if the actor has a pk (Django user), returns {id: pk, type: class name, label: str(actor)}. Otherwise falls back to duck-typed id/type/label attributes — this is what lets SystemActor (id=1, type="System", label="System") flow through uniformly for management-command-triggered events.

• Target (core.events.serializers.serialize_target / resolve_target_label): for ORM objects, type is app_label.ModelName, id is the PK, and label is resolved through a priority system:

  1. If the model defines an audit_label() method, use it.
  2. Else look up a per-model field priority tuple in MODEL_LABEL_PRIORITY (e.g. invoices.Invoice → tries invoice_number then description; users.UserIdentity → full_name → email → username).
  3. Else fall back to a generic priority list (full_name, get_full_name, display_name, title, subject, summary, name, label, description, business identifiers, username/email, slug/code/key).
  4. Final fallback: str(target), then the class name if even that raises.

  Field resolution supports plain attributes, @property, and zero-arg callables transparently (safe_getattr).

This is a generic, extensible system — new apps don't need to touch audit to get sensible labels; they either implement audit_label() on the model or add an entry to MODEL_LABEL_PRIORITY.

#6. Governance Layer (what actually gets written)

Before persistence, event.metadata passes through audit.governance.sanitize_metadata():

1. Keys are lower-cased and checked against SENSITIVE_KEYS (password, token, access_token, refresh_token, authorization, api_key, secret, client_secret, session, sessionid, csrftoken) — matches are replaced with the literal string "[REDACTED]" (not removed, so the presence of the key is still visible).
2. Everything else goes through safe_serialize(), which recursively:
   • Passes through primitives (str/int/float/bool), truncating strings over MAX_METADATA_STRING_LENGTH (2000 chars, ...[TRUNCATED] suffix).
   • Stringifies Decimal/UUID, ISO-formats anything with .isoformat() (datetimes).
   • Recurses into dict/list/tuple/set up to MAX_NESTED_DEPTH (5) and MAX_COLLECTION_ITEMS (100 items — collections beyond that are cut with a "[TRUNCATED]" sentinel entry / "__truncated__": True key).
   • Reduces any Django model instance encountered inside metadata to {id, label} rather than serializing its fields.
   • Falls back to truncate_string(str(value)) for anything unrecognized.

This is a defense-in-depth measure against three separate failure modes: secret leakage into logs, unbounded JSONField growth from careless callers, and JSON-serialization crashes from arbitrary Python objects being passed as metadata.

Separately, field-level diffs produced by the automatic lifecycle tracker (audit.lifecycle.diffing.calculate_diff) go through a narrower, earlier-stage safety net: audit.lifecycle.exclusions.AUDIT_EXCLUDED_FIELDS (updated_at, created_at, search_vector, last_login, deleted_at) — these fields are dropped from the snapshot entirely, before diffing, so no-op timestamp churn never produces spurious .updated events. Note this exclusion list is not the same mechanism as SENSITIVE_KEYS — a tracked model's password-named field, for instance, would still be captured in the diff snapshot (via serialize_field_value, which has no redaction logic itself) and would only be redacted downstream by sanitize_metadata() once it's inside event.metadata["changes"]. In other words: redaction does happen for diffs too, but only because diffs are packaged into metadata before persistence, not because the lifecycle layer itself is secret-aware.

#Retention

audit.governance.retention.delete_expired_audit_events():

• Fixed 365-day retention (DEFAULT_RETENTION_DAYS), not currently exposed as a Django setting — it's a hardcoded module constant.

• Deletes in batches of 5000 (DELETE_BATCH_SIZE) via a while loop of SELECT ids LIMIT 5000 + DELETE WHERE id IN (...), explicitly to avoid long-held table locks on what is expected to be a large table.

• Returns a single int (total rows deleted).

• Wired to Celery: audit.tasks.cleanup_expired_audit_events (retries up to 3 times with backoff on any exception) is scheduled daily at 02:00 via CELERY_BEAT_SCHEDULE["cleanup-expired-audit-events"] in paystream/app_settings/celery.py (crontab(hour=2, minute=0)).

• Also exposed as a management command: python manage.py cleanup_audit_events (devtools/management/commands/cleanup_audit_events.py).

  ⚠️ Bug worth flagging: the management command does deleted_count, _ = delete_expired_audit_events() — unpacking two values — but the function actually returns a single int. As written this command will raise TypeError: cannot unpack non-iterable int object every time it's run. The Celery task (audit/tasks.py) calls the same function correctly (deleted_count = delete_expired_audit_events()), so the automatic daily cleanup is unaffected — only the manual CLI entrypoint is broken.

#7. Security & Permissions

• No dedicated audit permission class in this app beyond what BaseAdminPage/Django admin's standard has_change_permission provides. AuditEventAdmin.change_view explicitly blocks edits with a custom 403 page if has_change_permission is false — this is the app's only real "immutability enforcement" in the admin UI; there's nothing at the model/manager level preventing a .save() or .delete() from code (i.e. immutability is a UI/process convention, not a database or model-layer constraint).
• actions = None on AuditEventAdmin — bulk admin actions (including bulk delete) are disabled for this model.
• AUDIT_ADMIN_ROLES (audit/constants.py) declares a default role set (DJGO, CEO, CFO, SSO) intended to gate who can view/manage audit data, overridable via Django setting — but a repo-wide search found no other file referencing AUDIT_ADMIN_ROLES. It's currently a defined-but-unused constant; access control for the audit admin page appears to run entirely through the generic Django admin permission system instead. Worth confirming with whoever owns policyengine/admin role wiring whether this was meant to gate the audit changelist and the wiring was dropped, or whether it's reserved for a not-yet-built feature.
• Login/logout/login-failure auditing captures the attempted username on failure but never a password or token — consistent with the governance layer's SENSITIVE_KEYS redaction, though in this specific path the value simply isn't collected in the first place rather than being redacted after the fact.
• Impersonation start/stop are captured as first-class security events (auth.impersonation_started / _stopped) with the impersonating actor's PK in metadata — but note audit only defines the emitter functions; the actual impersonation feature (session switching logic, who's allowed to impersonate whom) is owned by another app and simply calls these helpers.

#8. Admin UI (AuditEventAdmin)

Registered via the shared AdminIconDecorator.register_with_icon pattern (consistent with the rest of DjangoPlay's admin, see the platform overview doc), on top of the project's BaseAdminPage base class (utilities.admin.base_admin_page) rather than plain ModelAdmin.

• List display: computed actor/event/category/severity/target/timestamp/IP display methods rather than raw fields — actor_id, actor_label, request_id, metadata are explicitly excluded from the list view (list_display_exclude) since audit tables get huge and those are either low-signal in a list or too wide.
• Filters: all major dimensions (action, actor_type, target_type, category, severity, client_ip) are exposed as AJAX-backed filters hitting a shared endpoint (/ops/admin-filters/options/, owned by core.admin_filters) rather than Django's default static list_filter dropdowns — necessary because the cardinality of action/target_id values on a system-wide log would make static filter rendering slow.
• Form layout: read-oriented, grouped into Core / Actor / Target / Request / Metadata tabs via a form_layouts config consumed by BaseAdminPage (a DjangoPlay-wide declarative admin-form-layout convention, not audit-specific).
• Search: actor_label, target_label, request_id, action.
• auto_load_threshold = 500: the changelist template (admin/custom/audit/changelist.html) is custom and clearly designed to avoid loading the full unfiltered table by default once the row count crosses this threshold — changelist_view explicitly computes total_count, filtered_count, and an auto_load_threshold_exceeded flag and pushes them into the template context.
• No custom has_add_permission override — the admin doesn't explicitly forbid manual row creation via /admin/audit/auditevent/add/; immutability there relies on whatever default Django permission the deploying admin role has been granted, which is a small gap relative to the "never mutated except by the system" design intent stated elsewhere in the docs.

#9. Integration Points (cross-app)

#10. Operational Visibility

Runtime dispatch metrics for the whole event bus (not audit-specific, but the primary way to observe whether audit's subscriber is healthy) are exposed at:

• GET /.../metrics/ → core.events.runtime.views.EventRuntimeMetricsAPIView (IsAdminUser-only), returning per-subscriber last-dispatch-duration (dispatch_times, keyed by fully-qualified subscriber name e.g. audit.subscribers.persistence.audit_persistence_subscriber) and total subscriber count. Slow subscribers (≥ SLOW_EVENT_THRESHOLD_SECONDS = 0.5s) are logged as warnings by core.events.runtime.safety/dispatching on every dispatch, not just sampled.
• This is in-memory, per-process state (EVENT_DISPATCH_TIMES = {} module-level dict) — it resets on worker/process restart and isn't aggregated across multiple app server processes, so it's a "is this specific worker's audit subscriber currently slow" signal, not a fleet-wide metric.

#11. Known Gaps

1. No test coverage — audit/tests.py is empty. Given this app underpins compliance/security auditing across ~40 tracked models, this is probably the single highest-value gap to close before treating the audit trail as trustworthy for compliance purposes.

#12. Quick Reference — How to Add Audit Tracking to a New Model

Based on how every existing tracked model is wired (verified pattern, not speculative):

1. Add "app_label.ModelName" to AUDIT_TRACKED_MODELS in audit/lifecycle/registry.py. This alone enables automatic created/updated events with field-level diffs.
2. If the model supports soft delete, ensure its soft-delete/restore code path fires audit.signals.events.post_soft_delete / post_restore — these are not automatic like pre_save/post_save.
3. (Optional but recommended) Add an audit_label() method to the model, or a MODEL_LABEL_PRIORITY entry in core/events/serializers.py, so audit log rows display something more useful than a raw fallback string.
4. Nothing else is required — persistence eligibility is derived automatically from step 1 (see §2.3), and metadata sanitization/truncation happens for free.