DjangoPlay — utilities App

Doc generated by inspecting all files under webapp/utilities/, plus the primary cross-app callers: paystream/custom_site/admin_site.py and paystream/custom_site/admin_console_views.py (consumers of admin/app_builder.py and admin/dashboard_metadata.py), paystream/urlconf/base.py (mounts utilities.urls at the site root), paystream/app_settings/middleware.py, and webapp/configs/*.json (the runtime config files utilities reads).

#1. Summary

1. A custom Django Admin framework (admin/) — BaseAdminPage and its mixins (soft-delete actions, audit-field display, change history, cascading delete/restore, AJAX-populated dropdowns/filters, CSV export) that essentially every ModelAdmin in the platform inherits from instead of django.contrib.admin.ModelAdmin directly.
2. A generic DRF API layer (api/) — BaseListAPIView, BaseDetailAPIView, BaseViewSet, BaseBulkAPIView/BaseBulkUpdateAPIView, Redis-backed response caching, throttling, and reusable filter mixins (date range, FK, name search, trigram search) that other apps' viewsets subclass.
3. The "admin registry" (admin/app_registry.py + configs/admin_registry.json) — the runtime source of truth for which Django apps/models are visible in the custom console UI at all, independent of (and layered on top of) policyengine's role-based permissions.
4. A CSV export subsystem (services/exports/, models/export_job.py) — a generic, any-model CSV export pipeline with per-user/per-IP throttling, wired into the admin via AdminExportMixin.
5. Generic, app-agnostic AJAX endpoints (views/generic/, urls.py) — one filter-options endpoint and one form-options endpoint that serve dropdown/autocomplete data for any Django app/model without per-app registration, plus two narrower endpoints (audit filter options, invoices recipient-address lookup).

Around this, utilities also carries a grab-bag of shared low-level helpers that don't fit neatly into a single domain app: phone/postal-code/address validators used by locations and fincore, a national-ID reference table (constants/national_ids.py), a signed-token generator used by users' verification flows, a domain-allowlist validator for signup emails, and the platform's PlatformAdminService (the one place "superuser" is further narrowed to two specific hardcoded emails).

The app's own README-equivalent (admin/mixins/permissions_bridge.py's docstring) is explicit that policyengine remains the source of truth for permissions and that utilities provides only optional, explicit, opt-in UI-level overrides — in practice, though, utilities ends up being a second, independent access-control layer (the registry) that every request also has to pass, and the two layers are not always kept in sync (see §5 and §6).

#2. Architecture

utilities has no clean single entry point — it's five loosely related subsystems sharing one Django app namespace. This section maps each one.

#2.1 Admin framework (admin/)

admin/mixins/base_admin.py        BaseAdmin (ModelAdmin composition root)
  ├─ admin/mixins/fieldsets.py    FieldsetMixin — injects a "Metadata" audit fieldset
  ├─ admin/mixins/admin_form.py   CustomAdminFormMixin
  ├─ admin/mixins/history.py      HistoryMixin — diffs simple_history records into a display-ready timeline
  ├─ admin/mixins/soft_delete.py  SoftDeleteMixin — soft_delete / restore / hard_delete admin actions
  └─ admin/mixins/queryset.py     OptimizedQuerysetMixin — select_related/prefetch_related hooks

admin/base_console_admin.py       BaseConsoleAdmin — tab/section form-layout resolution for the custom change form UI
admin/export_mixin.py             AdminExportMixin — adds /export-selected/ and /export-all/ admin URLs
admin/base_admin_page.py          BaseAdminPage(AdminExportMixin, BaseAdmin, BaseConsoleAdmin) — what every app's ModelAdmin actually inherits from
admin/cascade.py                  CascadeHandler — generic, reflection-based recursive soft-delete/restore/hard-delete across FK/O2O reverse relations
admin/app_registry.py             build_app_registry() / get_app_registry() — the registry (§2.3)
admin/app_builder.py              build_app_list() — permission-filtered app/model list for the admin UI
admin/dashboard_metadata.py       get_dashboard_apps() / get_global_search_data() — same idea, for the console dashboard + global search
admin/admin_metadata.py           MODEL_ICON_MAP / APP_ICON_MAP — static icon lookup tables
admin/registry_config_loader.py   get_registry_config() — loads configs/admin_registry.json with a 6-second in-memory TTL cache
admin/filters/                    ~15 files of reusable/app-specific admin SimpleListFilter subclasses (§4.5)
admin/export_fields.py            get_export_fields() — reflection-based default export-field-list generator
admin/paginators.py               LimitedAdminPaginator — hard-caps admin changelist pagination at 5 pages
admin/url_utils.py                get_all_subdomains() / get_subdomain() — subdomain URL resolution (§4.6)

BaseAdminPage is the composition root: class BaseAdminPage(AdminExportMixin, BaseAdmin, BaseConsoleAdmin). Every app's ModelAdmin classes (verified via the admin_metadata.py icon map, which lists models from fincore, users, teamcentral, helpdesk, locations, industries, entities, invoices, and the third-party genericissuetracker) are expected to subclass this rather than Django's own ModelAdmin.

#2.2 Generic DRF API layer (api/)

api/generic_views.py       BaseListAPIView, BaseDetailAPIView, BaseFilteredListAPIView,
                            BaseBulkAPIView, BaseHistoryListAPIView
api/generic_viewsets.py    BaseViewSet(SerializerByActionMixin, SoftDeleteMixin, CacheMixin, ModelViewSet)
api/bulk_views.py          BaseBulkUpdateAPIView — PATCH-based bulk field update with history tracking
api/pagination.py          StandardResultsSetPagination (page_size=50, max=100)
api/rate_limits.py         CustomThrottle (50/hr, IP-keyed), CustomSearchThrottle (100/hr), TokenThrottle (50/hr, user-keyed)
api/mixins/redis_cache.py  CacheMixin — raw redis_client-backed get/set/invalidate-by-pattern
api/mixins/soft_delete.py  SoftDeleteMixin — perform_destroy/perform_restore wrapping model.soft_delete()/restore()
api/mixins/serializer_resolution.py  SerializerByActionMixin — read_serializer_class vs write_serializer_class by action
api/filters/                Composable filter mixins: date_range, foreign_key, text_search (icontains), trigram_search (pg_trgm)
api/generic_api_exceptions.py  GenericAPIError — the default error_class every base view/viewset falls back to

Two independent caching mechanisms exist side by side: BaseListAPIView/BaseDetailAPIView (in generic_views.py) use Django's cache framework directly with SHA-256-hashed query-param cache keys; BaseViewSet (in generic_viewsets.py) uses its own CacheMixin, which talks to core.utils.redis_client directly with json.dumps/json.loads. Both default to long TTLs (BaseListAPIView: 172800s / 2 days; BaseDetailAPIView: 86400s / 1 day; BaseViewSet: 172800s / 2 days).

#2.3 The admin registry — a second, independent access-control layer

get_app_registry() (admin/app_registry.py) builds, on every call (no persistent caching beyond the 6-second registry_config_loader TTL), a dict of every installed Django app/model not in a hardcoded exclude_apps/exclude_models set, merged with configs/admin_registry.json. This registry answers two questions independently of Django's permission system:

• is_app_enabled_for_user() / is_model_enabled_for_user() — is this app/model visible at all, per the enabled flag and model_exclusions list in admin_registry.json, for this user? (PlatformAdminService.is_platform_admin() — see §4.4 — always bypasses this.)
• The registry's exclude_apps set (admin, account, allauth, auth, contenttypes, sessions, genericissuetracker, sites, socialaccount, token_blacklist, policyengine, apidocs, mailer, utilities, aicore) determines which apps show up in the console UI's app list / global search / dashboard stats at all, regardless of what Django permissions a user holds.

This is explicitly not the same mechanism as policyengine's Group/Permission system (confirmed by both app_builder.py's and dashboard_metadata.py's own docstrings: "Visibility driven by Django permissions ONLY — Do NOT filter by admin_registry here — Registry enforcement happens in views (404/403)"). In practice this means:

• List-building functions (build_app_list, get_dashboard_apps, get_global_search_data) filter by Django's native has_view_permission()/has_perm() only — a model excluded from the registry can still appear in these lists if there's no explicit registry check in that particular function (see §6, point 3, for a confirmed inconsistency here).
• Actual page access to a given app/model (via the custom console changelist/single-app views, not shown in this app but the consumers of is_model_enabled_for_user) is where registry exclusion is enforced as a 404/403.

services/exports/export_throttle.py and ui/dashboard/services/registry_permissions_service.py both independently consume the registry for their own purposes (§4.3, §5).

#2.4 Export subsystem

models/export_job.py                 ExportJob — status machine: pending → processing → completed | failed
services/exports/export_service.py   ExportService.create_export() / process_export() — orchestrator
services/exports/export_throttle.py  ExportThrottleService — per-email/per-IP limits from configs/throttling.json
services/exports/export_queryset.py  ExportQuerysetService — reflection-based model resolution + filter cleaning
services/exports/export_generator.py ExportGeneratorService.generate_csv() — streaming CSV write via .iterator(chunk_size=2000)
services/exports/export_delivery.py  ExportDeliveryService.deliver() — no-op for "download"; "email" branch is an unimplemented stub
admin/export_mixin.py                AdminExportMixin — /export-selected/ and /export-all/ admin URLs, wired via get_export_fields()

Despite ExportJob modeling an asynchronous job (status, started_at/completed_at/failed_at, processed_records), AdminExportMixin.export_selected_view/export_all_view call ExportService.process_export() synchronously, inline in the admin HTTP request/response cycle — there is no Celery task dispatch here even though the platform has Celery configured (paystream/app_settings/celery.py is loaded in paystream/settings/base.py). For any export large enough to matter, this blocks the admin worker for the full duration of the CSV write. See §6.

#2.5 Generic AJAX endpoints (views/, urls.py)

Four endpoints, all mounted at the site root by path("", include("utilities.urls", namespace="utilities")) in paystream/urlconf/base.py:

The two generic endpoints are deliberately app-agnostic by design (their own module docstrings describe this as a replacement for "the old approach" of per-app FILTER_CHAIN registration) — see §6 for what this means for authentication.

#2.6 Shared low-level helpers

• commons/ — email/username validators, decimal-safe conversion, a domain-allowlist validator (backed by commons/valid_domains.json, not present in this pass), a secure token generator (secrets-based, URL-safe, configurable prefix — used by verification-link flows), and helpers.py (site-name resolution, an identity-state resolver that calls into users.services.identity_query_service.IdentityQueryService).
• constants/ — login-flow status codes, a large static reference table of national ID document names by country (national_ids.py — reference data, not enforced/validated against anywhere in this app), TemplateRegistry (a central catalog of template paths and one permission-aware template selector for apidocs), and email unsubscribe-category configuration.
• utils/ — phone number, postal code, address, and currency validators (utils/general/, utils/locations/), business-entity status-ratio and validation helpers (utils/entities/), and small data-sync utilities (utils/data_sync/) for loading env/paths and company-name normalization.
• signals/disable_signals.py — a context manager for temporarily disconnecting pre_save/post_save receivers for given models (bulk-import / data-migration use case; not invoked anywhere within utilities itself).
• context_processors/report_bug.py — a template context processor gating a "report a bug" UI affordance by URL allowlist or admin-namespace detection.
• services/is_admin.py — PlatformAdminService (§4.4).

#3. Implementation Details

#3.1 admin/app_registry.py::REGISTRY_CONFIG + configs/admin_registry.json

REGISTRY_CONFIG (Python, hardcoded) supplies only exclude_apps/exclude_models. configs/admin_registry.json (loaded via registry_config_loader.get_registry_config(), 6-second TTL cache, falls back to {} silently on any read/parse error) supplies per-app enabled flags, a primary_model, and model_exclusions. The two are merged with {**REGISTRY_CONFIG, **(get_registry_config() or {})} — the JSON file's top-level exclude_apps/exclude_models keys would fully overwrite the Python ones if present, but the current JSON file doesn't define them, so in practice they're additive. As of this pass, admin_registry.json explicitly configures 8 apps (entities, industries, locations, audit, teamcentral, fincore, invoices, users, helpdesk); users is the only one with "enabled": false. It also defines a stats block (order + mapping) that ui/dashboard/services/dashboard_stats_service.py reads for the console dashboard's stat cards.

#3.2 services/is_admin.py::PlatformAdminService

PLATFORM_ADMIN_EMAILS = {"redstar@djangoplay.org", "shekhar@djangoplay.org"}

is_platform_admin(user) = user.is_authenticated and user.is_superuser and user.email in PLATFORM_ADMIN_EMAILS

This is a narrower check than Django's own is_superuser — a Django superuser whose email isn't in this hardcoded set is not a "platform admin" by this app's definition, and therefore does not get the registry-bypass treatment in is_app_enabled_for_user/is_model_enabled_for_user/get_dashboard_apps/ExportThrottleService.allow_export. This two-tier admin model (Django superuser vs. "platform admin") is easy to miss when reasoning about who can see what — see §6.

#3.3 admin/cascade.py::CascadeHandler

Fully reflection-based: walks obj._meta.get_fields() for auto_created and not concrete (i.e., reverse FK/O2O accessors), recurses into children, and — critically — re-queries via model.all_objects (not the default manager) so already-soft-deleted children are still included in the cascade, preventing a cascade restore from silently skipping rows a prior cascade already touched. Soft-delete cascades children first then self; restore cascades self first then children; hard-delete cascades children first then self, calling obj.save() immediately before obj.delete() (to force a final simple_history snapshot pre-deletion). All three set obj._history_user/obj._change_reason on each object rather than passing them as save()/delete() kwargs — this only works if the model's own save/delete logic reads those specific attribute names (verified as the pattern HistoryMixin and SoftDeleteMixin's admin actions expect, not independently verified against every model's own soft_delete()/restore() implementation in this pass).

#3.4 admin/mixins/history.py::HistoryMixin

Reconstructs a human-readable change timeline from simple_history records without a dedicated audit-log table — it diffs consecutive HistoricalRecords rows field-by-field, detects create/soft-delete/restore/hard-delete events by pattern-matching the free-text history_change_reason string ("Soft deleted", "Restored", "Permanently deleted" / "Hard deleted" substrings), separates "activity" fields (currently only last_login) from ordinary field-change history, and supports an optional Redis-cached "snapshot" payload (_history_cache_key) to diff against when there's no previous historical record to compare against (e.g., right after a bulk import that bypassed normal save-triggered history). Explicitly excludes password and standard audit fields (id, slug, created_at, updated_at, created_by, updated_by, deleted_at, deleted_by, is_active) from the diff. On any exception, fails soft — returns an empty history structure and logs a warning rather than breaking the admin page.

#3.5 Export throttling

ExportThrottleService.allow_export() bypasses entirely for PlatformAdminService.is_platform_admin(); otherwise defers to mailer.throttling.flow_throttle.allow_flow() with flow name "export_all" (full export) or "export_selected" (filtered export), configured in configs/throttling.json: export_selected = 2/day per email, 20/day per IP; export_all = 1/day per email, 25/day per IP. This is the same shared throttling mechanism mailer uses for its own flows (configs/email_flow_limits.json follows an identical per-flow schema for signup/password-reset/support-request/bug-report emails) — utilities doesn't own the throttle engine, just calls into it.

#3.6 admin/filters/factory.py::changelist_filter()

A single factory function that dynamically builds a SimpleListFilter subclass in one of three modes based on what's passed in: field_name pointing at a FK (mode="fk", lookups populated from the related model, respecting deleted_at/is_active if present), field_name pointing at a plain field (mode="field", lookups from the field's choices or a distinct-values query), or no field_name at all (mode="pk", lookups directly from the current model's own rows). This is what lets most app-specific filters in admin/filters/{locations,teamcentral,helpdesk,invoices,mailer,users}.py be one-liners like CityFilter = changelist_filter("city") rather than hand-written SimpleListFilter classes.

#4. Cross-App Integration Points

#5. Security-Relevant Observations

1. The two generic AJAX endpoints have no authentication or authorization check in the view code itself. ChangeformOptionsView and ChangelistFilterAPIView (views/generic/) are plain django.views.View subclasses — no permission_classes, no LoginRequiredMixin, no @login_required. They are mounted at the site root (path("", include("utilities.urls", ...))), not under admin_site.urls, so they don't inherit whatever login enforcement Django's AdminSite applies to its own registered views. The platform's global MIDDLEWARE list (paystream/app_settings/middleware.py) contains no login-enforcement middleware (only AuthenticationMiddleware, which attaches request.user but doesn't require it be authenticated). Both views do apply some automatic guardrails by design — they only serve models reachable via django.apps.apps.get_model(), respect deleted_at/is_active if those fields exist, and cap results at 50 — but as written, any unauthenticated caller who can guess an app_label/model_name pair can enumerate id/display-text pairs for any model in the project, including ones a given user has no Django permission to view. AuditFilterOptionsAPIView has the same shape (no auth check), scoped to AuditEvent's 4 allowlisted fields. Worth confirming with the team whether this is intentional (e.g., relying on the URLs being unlinked/undiscoverable) or a gap that should get an IsAuthenticated check.
2. Two-tier "admin" concept. PlatformAdminService.is_platform_admin() requires both is_superuser=True and membership in a two-email hardcoded allowlist (services/is_admin.py). Anywhere this is used as a bypass (registry visibility, export throttling), a Django superuser outside that allowlist is treated as an ordinary user — easy to misread as "superusers always see everything" when reading the registry code in isolation.
3. ExportQuerysetService.build_queryset and views/generic/*.py both default to model.all_objects rather than the model's normal (soft-delete-filtering) default manager, when the model defines it. This is deliberate for the export/lookup use cases described (admins need to export/find soft-deleted rows too), but means any admin with export or dropdown-resolution access can retrieve data on soft-deleted records that wouldn't otherwise surface in list views.
4. Email delivery for exports is an unimplemented stub. ExportDeliveryService.deliver()'s "email" branch is empty (comment: # future: attach export / send email / signed urls / S3 links) — if delivery_method="email" is ever actually set on an ExportJob today, the job silently completes with no email sent and no error surfaced.

#6. Known Gaps / Things Worth Confirming With the Team

1. If an export large enough to noticeably block the web worker actually shows up, that's the point to revisit Celery — with a real "your export is ready" notification UX designed up front, not bolted onto the existing button.

#7. Quick Reference — Tracing "Why Is This App/Model Visible (or Not) Here?"

For a new engineer debugging console UI visibility:

1. Is it in Django's app registry at all? apps.get_app_configs() — if the app isn't installed, nothing else matters.
2. Is it in admin/app_registry.py::REGISTRY_CONFIG["exclude_apps"]? If yes, it's invisible everywhere get_app_registry() is consulted, full stop, regardless of permissions (policyengine, apidocs, mailer, utilities, aicore, plus Django/allauth infra apps, are always excluded).
3. Does it have an entry in configs/admin_registry.json["apps"]? If yes and "enabled": false (currently only users), is_app_enabled_for_user returns False for everyone except PlatformAdminService.is_platform_admin() users. If a model is listed in that app's model_exclusions, is_model_enabled_for_user returns False for it specifically — but only in code paths that actually call it (§6, point 3).
4. Does the requesting user have the relevant Django permission (view_/change_/add_/delete_ + app_label.model_name)? This is what actually gates build_app_list, get_dashboard_apps, and get_global_search_data at the per-model level — check the policyengine app doc for how that permission got assigned to the user in the first place (registry-filtered user_permissions.set() vs. direct .groups.add() — two different mechanisms with different outcomes).
5. Is the user request.user.is_superuser? Bypasses get_dashboard_apps's permission check entirely (shows all registry-listed apps) — but note this is a different, broader bypass than PlatformAdminService.is_platform_admin(), which additionally requires the hardcoded email allowlist and is used by the registry visibility functions instead.
6. For AJAX-populated dropdowns/filters specifically (/admin/form-options/, /admin/filter-options/): these bypass steps 3–5 almost entirely (§5, point 1) — they only check that the model exists and (if present on the model) deleted_at is None/is_active is True. Don't assume "the field is hidden in the admin" means "the underlying data is unreachable via these endpoints."