--- since: 1.2.1 --- # Django File Cleanup System using Celery **Version:** 1.2 **Date:** August 2026 **Status:** Implemented ## 1. ๐Ÿ“Œ Introduction ### 1.1 Purpose This document defines the requirements for the automated file lifecycle management system in the `paystream` project. The system ensures efficient storage management using soft deletion, hard deletion, and age-based retention policies. ### 1.2 Scope The system applies to two DB-tracked models and one third-party, filesystem-only directory: * `helpdesk.FileUpload` โ†’ `media/file-uploads/` * `shared.models.ExportJob` โ†’ `media/exports/` * `media/issues/` (owned by the third-party `genericissuetracker` package โ€” no DB models of ours to query; handled separately, see ยง4.6) It includes: * Time-based cleanup using each model's own age field * Asynchronous processing via Celery * Scheduled execution via Celery Beat ### 1.3 Definitions | Term | Description | | ---------------- | -------------------------------------------------- | | Soft Delete | Logical deletion (mark inactive, retain DB record) | | Hard Delete | Permanent deletion from DB and storage | | Retention Window | Time duration before deletion | | Grace Period | Time between soft and hard delete | --- ## 2. ๐Ÿง  System Overview ### 2.1 Design Principle For `FileUpload` and `ExportJob`: the system MUST NOT scan the filesystem. All operations MUST be database-driven. **Deviation (v1.2):** `media/issues/` is an explicit exception to this principle โ€” see ยง4.6. It's owned by a third-party package with no DB models available to this codebase, so it's handled via a direct, unconditional filesystem wipe instead. #### Source of Truth * `FileUpload.uploaded_at` / `ExportJob.created_at` determine soft-delete eligibility * `deleted_at` (both models) determines hard-delete eligibility #### Processing Mechanism * Django ORM for filtering (`paystream/infra/file_cleanup/policy.py`) * Celery for async execution (`paystream/infra/file_cleanup/tasks.py`) **Deviation (v1.2):** role-based eligibility (ยง4.3 in v1.1) has been **removed**. Production files are already delivered to recipients over email, so nothing stored in `media/exports/` or `media/file-uploads/` needs to be retained based on who created it โ€” age is now the only eligibility signal, applied uniformly to every row. ## 3. ๐Ÿ—๏ธ System Architecture ### 3.1 Components * Django ORM (data layer) โ€” `helpdesk.FileUpload` and `shared.models.ExportJob`, both unchanged by this feature * Celery Worker (background execution) * Celery Beat (scheduler) * Redis (broker, and used directly by the hard-delete cadence gate โ€” see ยง4.5) * Storage backend (local/S3, via `default_storage`) ### 3.2 Code Layout ``` paystream/ app_settings/ file_cleanup.py # settings: retention windows celery.py # CELERY_BEAT_SCHEDULE entries, CELERY_TIMEZONE, # FILE_CLEANUP_DEV_FAST_SCHEDULE toggle infra/ file_cleanup/ policy.py # eligibility querysets (pure, no side effects) # returns {"file_uploads": qs, "exports": qs} tasks.py # Celery tasks: soft_delete_stale_files, # hard_delete_expired_files, wipe_issues_directory tasks.py # re-exports infra tasks for Celery autodiscovery ``` > **Note on task discovery:** Celery's `app.autodiscover_tasks()` only > looks for a `tasks` module directly under each installed app > (`paystream.tasks`), not arbitrary nested paths. `paystream/tasks.py` > exists solely to re-export the tasks from > `paystream/infra/file_cleanup/tasks.py` so they get registered. ### 3.3 Data Flow ``` File Upload / Export Job โ†’ Database Entry โ†’ Time-based evaluation (uploaded_at / created_at, deleted_at) โ†’ Soft delete โ†’ Grace period โ†’ Hard delete ``` `media/issues/` sits entirely outside this flow โ€” see ยง4.6. ## 4. ๐Ÿ“ฆ Functional Requirements ### 4.1 Model Requirements `helpdesk.FileUpload` and `shared.models.ExportJob` already satisfied the fields and behaviors below prior to this feature; no model changes were needed for either. #### 4.1.1 Relevant Fields | Field | FileUpload | ExportJob | | ------------- | ---------------- | ------------- | | Age field | `uploaded_at` | `created_at` | | `is_active` | โœ“ | โœ“ (via `TimeStampedModel`) | | `deleted_at` | โœ“ | โœ“ (via `TimeStampedModel`) | | File field | `file` | `file` | | Owner field | `created_by` | `requested_by` (not used for eligibility โ€” see ยง2.1 deviation) | #### 4.1.2 Model Behavior * `FileUpload` implements its own `soft_delete()` and `delete(hard=True)`, including storage cleanup. * `ExportJob` has no such override โ€” it inherits the base `TimeStampedModel.soft_delete()`, and has no `hard=` support on `delete()` at all. The hard-delete task handles this directly: it removes the file from storage via `default_storage`, then calls the model's plain `.delete()`. ### 4.2 Retention Policy #### 4.2.1 Soft Delete Trigger A row (either model) SHALL be soft deleted if: * age field `< (now - CLEAN_STORAGE_FREQUENCY_DAYS)` * `is_active = True` * `deleted_at` is null #### 4.2.2 Hard Delete Trigger A row (either model) SHALL be permanently deleted if: * `deleted_at` is not null * `deleted_at < (now - HARD_DELETE_AFTER_DAYS)` ### 4.3 Role-Based Logic โ€” **Removed in v1.2** v1.1 restricted eligibility to `USER_ROLES_FOR_CLEANUP` (default `{"SSO", "HR"}`). This has been removed entirely โ€” no role filter is applied. `USER_ROLES_FOR_CLEANUP` and `DEFAULT_CLEANUP_ROLE` settings, and the `resolve_allowed_roles()` / `_role_eligibility_filter()` helpers, no longer exist. ### 4.4 Background Jobs #### 4.4.1 Soft Delete Task `paystream.infra.file_cleanup.tasks.soft_delete_stale_files` * Runs weekly, every Saturday 23:00 IST * Iterates both `policy.get_soft_delete_candidates()` querysets (`file_uploads`, `exports`) * Calls each row's own `.soft_delete()` * Per-row failures are logged and skipped, not fatal to the task #### 4.4.2 Hard Delete Task `paystream.infra.file_cleanup.tasks.hard_delete_expired_files` * Triggers on the same weekly Saturday-23:00 crontab as soft-delete, but only actually executes once every `HARD_DELETE_AFTER_DAYS` (14) โ€” see ยง4.5, "Fortnightly Cadence Gate" * Iterates both `policy.get_hard_delete_candidates()` querysets * `FileUpload` rows: `.delete(hard=True)` * `ExportJob` rows: storage file removed manually via `default_storage`, then `.delete()` * Per-row failures are logged and skipped, not fatal to the task ### 4.5 Scheduling | Task | Frequency | Time | Beat key | | ------------------ | ---------------- | ---------------------- | ----------------------------- | | Soft Delete | Weekly | Saturday 23:00 IST | `file-cleanup-soft-delete` | | Hard Delete | Fortnightly (gated) | Saturday 23:00 IST (every other run) | `file-cleanup-hard-delete` | | Issues Wipe | Quarterly | 1st of Jan/Apr/Jul/Oct, 23:00 IST | `file-cleanup-wipe-issues` | **Fortnightly Cadence Gate (deviation, new in v1.2):** Celery's static `crontab` schedule cannot express "every 2 weeks" directly (only day-of-week / day-of-month / month patterns). `hard_delete_expired_files` is triggered on the same weekly crontab as the soft-delete task, but self-gates via a Redis cache timestamp (`file_cleanup:hard_delete:last_run`) and returns immediately (`{"processed": 0, "failed": 0, "skipped": True}`) if fewer than 13 days have elapsed since its last real run. The 13-day (not 14-day) threshold leaves slack for scheduler jitter. This gate is bypassed entirely when `FILE_CLEANUP_DEV_FAST_SCHEDULE=True` (ยง5.3). **Timezone:** `CELERY_TIMEZONE` is explicitly set to `"Asia/Kolkata"` in `app_settings/celery.py`. Django's own `TIME_ZONE` is `"UTC"`, and Celery does not default to `TIME_ZONE` โ€” without this explicit setting, all crontab hours above would fire in UTC, not IST. ### 4.6 Issues Directory Wipe โ€” **New in v1.2** `paystream.infra.file_cleanup.tasks.wipe_issues_directory` * Runs quarterly (1st of Jan/Apr/Jul/Oct, 23:00 IST) * **Deviates from ยง2.1** โ€” this task scans and clears the filesystem directly (`shutil.rmtree` / `Path.unlink()` on every entry under `MEDIA_ROOT/issues/`), because the directory is populated by the third-party `genericissuetracker` package, which this codebase has no DB models for. * Unconditional: does not check age, does not consult any DB table. Deletes everything present under `media/issues/` at run time, every run. * Per-entry deletion is not individually gated by eligibility, so there is no `processed`/`failed` split โ€” the task returns `{"removed": }`. ### 4.7 Missing-File UX โ€” **New in v1.2** Once files are being routinely deleted, any UI that still links to a now-deleted attachment needs a graceful failure path. `paystream.integrations.issuetracker.views.downloads.attachment.protected_attachment_download` (the only user-facing, template-rendered download endpoint affected by this system โ€” `FileUpload`/`ExportJob` are only ever accessed via the DRF API or Django admin, which handle missing files through their own existing mechanisms, not `django.contrib.messages`): * RBAC-invisible attachment โ†’ unchanged: `Http404` (deliberate 404-masking; must not reveal existence of a resource the requester isn't authorized to see) * File visible in DB but missing from storage (i.e., cleaned up) โ†’ **changed**: instead of `Http404`, the request is redirected to the parent issue's detail page with a `django.contrib.messages.info()` message: *"The resource you are looking for doesn't exist. Please contact support if you need details."* ## 5. โš™๏ธ Configuration Requirements ### 5.1 Settings (`paystream/app_settings/file_cleanup.py`) ```python CLEAN_STORAGE_FREQUENCY_DAYS = env.int("CLEAN_STORAGE_FREQUENCY_DAYS", default=7) HARD_DELETE_AFTER_DAYS = env.int("HARD_DELETE_AFTER_DAYS", default=14) ``` Both are env-overridable, consistent with the rest of `paystream/app_settings/`. `USER_ROLES_FOR_CLEANUP` and `DEFAULT_CLEANUP_ROLE` have been removed (ยง4.3). ### 5.2 Beat Schedule (`paystream/app_settings/celery.py`) See ยง4.5 for the three `CELERY_BEAT_SCHEDULE` entries and the `CELERY_TIMEZONE` setting. ### 5.3 Environment-Driven Dev/Prod Schedule Toggle โ€” **New in v1.2** ```python FILE_CLEANUP_DEV_FAST_SCHEDULE = env.bool("FILE_CLEANUP_DEV_FAST_SCHEDULE", default=False) ``` * `True` โ†’ all three tasks run every 5 minutes, and the fortnightly cadence gate (ยง4.5) is bypassed entirely. **Local development / testing only.** * `False` (default) โ†’ production schedule per ยง4.5. Read directly from the process environment at import time (not from `settings.DEBUG`), because `app_settings/celery.py` loads before Django's `DEBUG` is finalized by `dev.py` / `staging.py` / `prod.py`. Set in each environment's own `.env` โ€” never hardcoded in a settings `.py` file, and never `True` outside local development. | Env var | Local dev | Staging | Production | | ---------------------------------- | ---------- | -------- | ------------ | | `FILE_CLEANUP_DEV_FAST_SCHEDULE` | `True` | `False` | `False` | | `CLEAN_STORAGE_FREQUENCY_DAYS` | 7 (or 0 for immediate testing) | 7 | 7 | | `HARD_DELETE_AFTER_DAYS` | 14 (or 0 for immediate testing) | 14 | 14 | ## 6. ๐Ÿ”„ Process Flow ### 6.1 Lifecycle (FileUpload / ExportJob) ``` Upload โ†“ Stored in DB โ†“ Weekly (Sat 23:00 IST): check age + soft delete โ†“ Wait grace period (HARD_DELETE_AFTER_DAYS = 14) โ†“ Every 2 weeks (gated, Sat 23:00 IST trigger): check deleted_at age + hard delete ``` ## 6.2 Lifecycle (media/issues/) ``` Quarterly (1st of Jan/Apr/Jul/Oct, 23:00 IST): wipe entire directory, unconditionally ``` ## 7. ๐Ÿšจ Edge Cases & Constraints ### 7.1 Edge Cases * File missing in storage โ†’ hard delete does not fail โ€” `FileUpload` checks `default_storage.exists()` first; `ExportJob` handling in `tasks.py` does the same; both additionally wrap each row in try/except * Already soft-deleted rows โ†’ excluded from soft-delete candidates via `deleted_at__isnull=True` * Null `deleted_at` โ†’ excluded from hard-delete candidates via `deleted_at__isnull=False` * Hard-delete task triggered before its 13-day minimum interval has elapsed โ†’ returns immediately with `skipped: True`, no rows touched (ยง4.5) * A `FileUpload`/`ExportJob` row referenced by a UI download link after its file has been cleaned up โ†’ user sees a friendly message, not a raw 404 or stack trace (ยง4.7) ### 7.2 Constraints * No filesystem scanning for `FileUpload` / `ExportJob` โ€” confirmed, all filtering is ORM/DB-only * `media/issues/` is scanned/wiped directly by design (ยง4.6) โ€” this is an intentional, documented exception, not an oversight * System is idempotent for `FileUpload`/`ExportJob` โ€” re-running either task only affects records still matching the eligibility queryset at that moment. `wipe_issues_directory` is idempotent trivially (an empty directory stays empty). ## 8. ๐Ÿ“Š Non-Functional Requirements ### 8.1 Performance * Both DB-driven tasks use `queryset.iterator()` to avoid loading the full candidate set into memory at once * `select_related()` avoids N+1 queries when resolving row ownership (retained even though it's no longer used for eligibility filtering, since owner info is still useful for logging/debugging) ### 8.2 Scalability * Deferred โ€” see ยง11 (unchanged from v1.1) ### 8.3 Reliability * All three tasks retry via `autoretry_for=(Exception,)` (max 3, backoff), matching the existing `audit.tasks.cleanup_expired_audit_events` pattern * Per-row/per-entry failures inside a sweep are caught, logged, and counted โ€” they don't abort the rest of the sweep ### 8.4 Maintainability * Eligibility logic lives entirely in `policy.py`, decoupled from Celery โ€” testable without a worker * Retention configuration centralized in `paystream/app_settings/file_cleanup.py` * Dev/prod schedule cadence controlled by a single env var (`FILE_CLEANUP_DEV_FAST_SCHEDULE`), avoiding hand-edited crontab values in source ## 9. ๐Ÿ” Security Requirements * Deletions only run via scheduled Celery tasks โ€” no new user-facing endpoint was added * RBAC-based 404-masking on the issue attachment download endpoint is preserved unchanged (ยง4.7) โ€” the friendly missing-file message is only shown for rows the requester is already authorized to see ## 10. ๐Ÿงช Testing Requirements ### 10.1 Unit Tests (not yet written โ€” follow-up) * `policy.get_soft_delete_candidates()` / `get_hard_delete_candidates()` filtering for both `FileUpload` and `ExportJob` * `hard_delete_expired_files` cadence gate (skips within 13 days, proceeds after, bypassed under `FILE_CLEANUP_DEV_FAST_SCHEDULE`) * Task per-row failure isolation * `wipe_issues_directory` removes all entries regardless of age ### 10.2 Integration Tests (not yet written โ€” follow-up) * Celery Beat schedule registration (all three entries) * End-to-end soft โ†’ hard delete lifecycle for both models * Attachment download view: missing-file โ†’ message + redirect; RBAC-invisible โ†’ still `Http404` ### 10.3 Manual Local Verification (documented, not automated) Set `FILE_CLEANUP_DEV_FAST_SCHEDULE=True` and drive tasks directly via `python manage.py shell` โ€” backdate rows, call `.run()` synchronously, inspect `policy.get_*_candidates()` counts before/after. See engineering notes for exact snippets. ## 11. ๐Ÿ“ˆ Future Enhancements (Optional) * Batch deletes using `queryset.update()` for soft delete (hard delete must stay per-row due to storage I/O) * File size-based cleanup rules * Audit logging integration * Admin dashboard for monitoring * Unit/integration test suite (ยง10) * If `genericissuetracker` ever exposes its attachment model publicly, revisit ยง4.6 to make the issues cleanup DB-driven like the other two ## 12. โœ… Acceptance Criteria * [x] `FileUpload` and `ExportJob` rows are soft deleted after `CLEAN_STORAGE_FREQUENCY_DAYS` * [x] Rows are hard deleted after `HARD_DELETE_AFTER_DAYS` grace period * [x] No role-based filtering (removed, ยง4.3) * [x] No filesystem scans for `FileUpload` / `ExportJob` * [x] `media/issues/` wiped quarterly via direct filesystem access (documented exception, ยง4.6) * [x] Celery Beat schedule registered and all three tasks discoverable * [x] `CELERY_TIMEZONE` pinned so schedule times are IST, not UTC * [x] Hard-delete fortnightly cadence enforced via cache gate * [x] Dev/prod schedule cadence togglable via `FILE_CLEANUP_DEV_FAST_SCHEDULE` env var * [x] Missing-file downloads show a friendly message instead of a raw 404 (RBAC-masking 404s unaffected) * [ ] Automated test coverage (tracked as follow-up, ยง10.1/ยง10.2)