Django File Cleanup System using Celery

#1. 📌 Introduction

#1.1 Purpose

This document defines the requirements for the automated file lifecycle management system in the paystream project. The system ensures efficient storage management using soft deletion, hard deletion, and age-based retention policies.

#1.2 Scope

The system applies to two DB-tracked models and one third-party, filesystem-only directory:

• helpdesk.FileUpload → media/file-uploads/
• shared.models.ExportJob → media/exports/
• media/issues/ (owned by the third-party genericissuetracker package — no DB models of ours to query; handled separately, see §4.6)

It includes:

• Time-based cleanup using each model's own age field
• Asynchronous processing via Celery
• Scheduled execution via Celery Beat

#1.3 Definitions

#2. 🧠 System Overview

#2.1 Design Principle

For FileUpload and ExportJob: the system MUST NOT scan the filesystem. All operations MUST be database-driven.

Deviation (v1.2): media/issues/ is an explicit exception to this principle — see §4.6. It's owned by a third-party package with no DB models available to this codebase, so it's handled via a direct, unconditional filesystem wipe instead.

Source of Truth

• FileUpload.uploaded_at / ExportJob.created_at determine soft-delete eligibility
• deleted_at (both models) determines hard-delete eligibility

Processing Mechanism

• Django ORM for filtering (paystream/infra/file_cleanup/policy.py)
• Celery for async execution (paystream/infra/file_cleanup/tasks.py)

Deviation (v1.2): role-based eligibility (§4.3 in v1.1) has been removed. Production files are already delivered to recipients over email, so nothing stored in media/exports/ or media/file-uploads/ needs to be retained based on who created it — age is now the only eligibility signal, applied uniformly to every row.

#3. 🏗️ System Architecture

#3.1 Components

• Django ORM (data layer) — helpdesk.FileUpload and shared.models.ExportJob, both unchanged by this feature
• Celery Worker (background execution)
• Celery Beat (scheduler)
• Redis (broker, and used directly by the hard-delete cadence gate — see §4.5)
• Storage backend (local/S3, via default_storage)

#3.2 Code Layout

paystream/
  app_settings/
    file_cleanup.py       # settings: retention windows
    celery.py             # CELERY_BEAT_SCHEDULE entries, CELERY_TIMEZONE,
                           # FILE_CLEANUP_DEV_FAST_SCHEDULE toggle
  infra/
    file_cleanup/
      policy.py            # eligibility querysets (pure, no side effects)
                            # returns {"file_uploads": qs, "exports": qs}
      tasks.py              # Celery tasks: soft_delete_stale_files,
                             # hard_delete_expired_files, wipe_issues_directory
  tasks.py                 # re-exports infra tasks for Celery autodiscovery

Note on task discovery: Celery's app.autodiscover_tasks() only looks for a tasks module directly under each installed app (paystream.tasks), not arbitrary nested paths. paystream/tasks.py exists solely to re-export the tasks from paystream/infra/file_cleanup/tasks.py so they get registered.

#3.3 Data Flow

File Upload / Export Job → Database Entry
           → Time-based evaluation (uploaded_at / created_at, deleted_at)
           → Soft delete
           → Grace period
           → Hard delete

media/issues/ sits entirely outside this flow — see §4.6.

#4. 📦 Functional Requirements

#4.1 Model Requirements

helpdesk.FileUpload and shared.models.ExportJob already satisfied the fields and behaviors below prior to this feature; no model changes were needed for either.

4.1.1 Relevant Fields

4.1.2 Model Behavior

• FileUpload implements its own soft_delete() and delete(hard=True), including storage cleanup.
• ExportJob has no such override — it inherits the base TimeStampedModel.soft_delete(), and has no hard= support on delete() at all. The hard-delete task handles this directly: it removes the file from storage via default_storage, then calls the model's plain .delete().

#4.2 Retention Policy

4.2.1 Soft Delete Trigger

A row (either model) SHALL be soft deleted if:

• age field < (now - CLEAN_STORAGE_FREQUENCY_DAYS)
• is_active = True
• deleted_at is null

4.2.2 Hard Delete Trigger

A row (either model) SHALL be permanently deleted if:

• deleted_at is not null
• deleted_at < (now - HARD_DELETE_AFTER_DAYS)

#4.3 Role-Based Logic — Removed in v1.2

v1.1 restricted eligibility to USER_ROLES_FOR_CLEANUP (default {"SSO", "HR"}). This has been removed entirely — no role filter is applied. USER_ROLES_FOR_CLEANUP and DEFAULT_CLEANUP_ROLE settings, and the resolve_allowed_roles() / _role_eligibility_filter() helpers, no longer exist.

#4.4 Background Jobs

4.4.1 Soft Delete Task

paystream.infra.file_cleanup.tasks.soft_delete_stale_files

• Runs weekly, every Saturday 23:00 IST
• Iterates both policy.get_soft_delete_candidates() querysets (file_uploads, exports)
• Calls each row's own .soft_delete()
• Per-row failures are logged and skipped, not fatal to the task

4.4.2 Hard Delete Task

paystream.infra.file_cleanup.tasks.hard_delete_expired_files

• Triggers on the same weekly Saturday-23:00 crontab as soft-delete, but only actually executes once every HARD_DELETE_AFTER_DAYS (14) — see §4.5, "Fortnightly Cadence Gate"
• Iterates both policy.get_hard_delete_candidates() querysets
• FileUpload rows: .delete(hard=True)
• ExportJob rows: storage file removed manually via default_storage, then .delete()
• Per-row failures are logged and skipped, not fatal to the task

#4.5 Scheduling

Fortnightly Cadence Gate (deviation, new in v1.2): Celery's static crontab schedule cannot express "every 2 weeks" directly (only day-of-week / day-of-month / month patterns). hard_delete_expired_files is triggered on the same weekly crontab as the soft-delete task, but self-gates via a Redis cache timestamp (file_cleanup:hard_delete:last_run) and returns immediately ({"processed": 0, "failed": 0, "skipped": True}) if fewer than 13 days have elapsed since its last real run. The 13-day (not 14-day) threshold leaves slack for scheduler jitter. This gate is bypassed entirely when FILE_CLEANUP_DEV_FAST_SCHEDULE=True (§5.3).

Timezone: CELERY_TIMEZONE is explicitly set to "Asia/Kolkata" in app_settings/celery.py. Django's own TIME_ZONE is "UTC", and Celery does not default to TIME_ZONE — without this explicit setting, all crontab hours above would fire in UTC, not IST.

#4.6 Issues Directory Wipe — New in v1.2

paystream.infra.file_cleanup.tasks.wipe_issues_directory

• Runs quarterly (1st of Jan/Apr/Jul/Oct, 23:00 IST)
• Deviates from §2.1 — this task scans and clears the filesystem directly (shutil.rmtree / Path.unlink() on every entry under MEDIA_ROOT/issues/), because the directory is populated by the third-party genericissuetracker package, which this codebase has no DB models for.
• Unconditional: does not check age, does not consult any DB table. Deletes everything present under media/issues/ at run time, every run.
• Per-entry deletion is not individually gated by eligibility, so there is no processed/failed split — the task returns {"removed": <count>}.

#4.7 Missing-File UX — New in v1.2

Once files are being routinely deleted, any UI that still links to a now-deleted attachment needs a graceful failure path.

paystream.integrations.issuetracker.views.downloads.attachment.protected_attachment_download (the only user-facing, template-rendered download endpoint affected by this system — FileUpload/ExportJob are only ever accessed via the DRF API or Django admin, which handle missing files through their own existing mechanisms, not django.contrib.messages):

• RBAC-invisible attachment → unchanged: Http404 (deliberate 404-masking; must not reveal existence of a resource the requester isn't authorized to see)
• File visible in DB but missing from storage (i.e., cleaned up) → changed: instead of Http404, the request is redirected to the parent issue's detail page with a django.contrib.messages.info() message: "The resource you are looking for doesn't exist. Please contact support if you need details."

#5. ⚙️ Configuration Requirements

#5.1 Settings (paystream/app_settings/file_cleanup.py)

CLEAN_STORAGE_FREQUENCY_DAYS = env.int("CLEAN_STORAGE_FREQUENCY_DAYS", default=7)
HARD_DELETE_AFTER_DAYS = env.int("HARD_DELETE_AFTER_DAYS", default=14)

Both are env-overridable, consistent with the rest of paystream/app_settings/. USER_ROLES_FOR_CLEANUP and DEFAULT_CLEANUP_ROLE have been removed (§4.3).

#5.2 Beat Schedule (paystream/app_settings/celery.py)

See §4.5 for the three CELERY_BEAT_SCHEDULE entries and the CELERY_TIMEZONE setting.

#5.3 Environment-Driven Dev/Prod Schedule Toggle — New in v1.2

FILE_CLEANUP_DEV_FAST_SCHEDULE = env.bool("FILE_CLEANUP_DEV_FAST_SCHEDULE", default=False)

• True → all three tasks run every 5 minutes, and the fortnightly cadence gate (§4.5) is bypassed entirely. Local development / testing only.
• False (default) → production schedule per §4.5.

Read directly from the process environment at import time (not from settings.DEBUG), because app_settings/celery.py loads before Django's DEBUG is finalized by dev.py / staging.py / prod.py. Set in each environment's own .env — never hardcoded in a settings .py file, and never True outside local development.

#6. 🔄 Process Flow

#6.1 Lifecycle (FileUpload / ExportJob)

Upload
  ↓
Stored in DB
  ↓
Weekly (Sat 23:00 IST): check age + soft delete
  ↓
Wait grace period (HARD_DELETE_AFTER_DAYS = 14)
  ↓
Every 2 weeks (gated, Sat 23:00 IST trigger): check deleted_at age + hard delete

#6.2 Lifecycle (media/issues/)

Quarterly (1st of Jan/Apr/Jul/Oct, 23:00 IST): wipe entire directory, unconditionally

#7. 🚨 Edge Cases & Constraints

#7.1 Edge Cases

• File missing in storage → hard delete does not fail — FileUpload checks default_storage.exists() first; ExportJob handling in tasks.py does the same; both additionally wrap each row in try/except
• Already soft-deleted rows → excluded from soft-delete candidates via deleted_at__isnull=True
• Null deleted_at → excluded from hard-delete candidates via deleted_at__isnull=False
• Hard-delete task triggered before its 13-day minimum interval has elapsed → returns immediately with skipped: True, no rows touched (§4.5)
• A FileUpload/ExportJob row referenced by a UI download link after its file has been cleaned up → user sees a friendly message, not a raw 404 or stack trace (§4.7)

#7.2 Constraints

• No filesystem scanning for FileUpload / ExportJob — confirmed, all filtering is ORM/DB-only
• media/issues/ is scanned/wiped directly by design (§4.6) — this is an intentional, documented exception, not an oversight
• System is idempotent for FileUpload/ExportJob — re-running either task only affects records still matching the eligibility queryset at that moment. wipe_issues_directory is idempotent trivially (an empty directory stays empty).

#8. 📊 Non-Functional Requirements

#8.1 Performance

• Both DB-driven tasks use queryset.iterator() to avoid loading the full candidate set into memory at once
• select_related() avoids N+1 queries when resolving row ownership (retained even though it's no longer used for eligibility filtering, since owner info is still useful for logging/debugging)

#8.2 Scalability

• Deferred — see §11 (unchanged from v1.1)

#8.3 Reliability

• All three tasks retry via autoretry_for=(Exception,) (max 3, backoff), matching the existing audit.tasks.cleanup_expired_audit_events pattern
• Per-row/per-entry failures inside a sweep are caught, logged, and counted — they don't abort the rest of the sweep

#8.4 Maintainability

• Eligibility logic lives entirely in policy.py, decoupled from Celery — testable without a worker
• Retention configuration centralized in paystream/app_settings/file_cleanup.py
• Dev/prod schedule cadence controlled by a single env var (FILE_CLEANUP_DEV_FAST_SCHEDULE), avoiding hand-edited crontab values in source

#9. 🔐 Security Requirements

• Deletions only run via scheduled Celery tasks — no new user-facing endpoint was added
• RBAC-based 404-masking on the issue attachment download endpoint is preserved unchanged (§4.7) — the friendly missing-file message is only shown for rows the requester is already authorized to see

#10. 🧪 Testing Requirements

#10.1 Unit Tests (not yet written — follow-up)

• policy.get_soft_delete_candidates() / get_hard_delete_candidates() filtering for both FileUpload and ExportJob
• hard_delete_expired_files cadence gate (skips within 13 days, proceeds after, bypassed under FILE_CLEANUP_DEV_FAST_SCHEDULE)
• Task per-row failure isolation
• wipe_issues_directory removes all entries regardless of age

#10.2 Integration Tests (not yet written — follow-up)

• Celery Beat schedule registration (all three entries)
• End-to-end soft → hard delete lifecycle for both models
• Attachment download view: missing-file → message + redirect; RBAC-invisible → still Http404

#10.3 Manual Local Verification (documented, not automated)

Set FILE_CLEANUP_DEV_FAST_SCHEDULE=True and drive tasks directly via python manage.py shell — backdate rows, call <task>.run() synchronously, inspect policy.get_*_candidates() counts before/after. See engineering notes for exact snippets.

#11. 📈 Future Enhancements (Optional)

• Batch deletes using queryset.update() for soft delete (hard delete must stay per-row due to storage I/O)
• File size-based cleanup rules
• Audit logging integration
• Admin dashboard for monitoring
• Unit/integration test suite (§10)
• If genericissuetracker ever exposes its attachment model publicly, revisit §4.6 to make the issues cleanup DB-driven like the other two

#12. ✅ Acceptance Criteria

• FileUpload and ExportJob rows are soft deleted after CLEAN_STORAGE_FREQUENCY_DAYS
• Rows are hard deleted after HARD_DELETE_AFTER_DAYS grace period
• No role-based filtering (removed, §4.3)
• No filesystem scans for FileUpload / ExportJob
• media/issues/ wiped quarterly via direct filesystem access (documented exception, §4.6)
• Celery Beat schedule registered and all three tasks discoverable
• CELERY_TIMEZONE pinned so schedule times are IST, not UTC
• Hard-delete fortnightly cadence enforced via cache gate
• Dev/prod schedule cadence togglable via FILE_CLEANUP_DEV_FAST_SCHEDULE env var
• Missing-file downloads show a friendly message instead of a raw 404 (RBAC-masking 404s unaffected)
• Automated test coverage (tracked as follow-up, §10.1/§10.2)