It is intended for debugging:

AuthX container configuration
JWT private/public key loading
PostgreSQL connectivity
database schema/table availability
identity lookup
password authentication
/token 401 Unauthorized responses
DjangoPlay → AuthX authentication integration

1. Verify AuthX Settings Are Loading

The settings object is exposed through get_settings().

Check JWT configuration

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings

s = get_settings()

print("JWT_ALGORITHM:", s.jwt_algorithm)
print("JWT_ISSUER:", s.jwt_issuer)
print("JWT_AUDIENCE:", s.jwt_audience)
print("APP_ENV:", s.app_env)
PY
'

Do not print private keys or other secrets.

2. Verify JWT Private Key Formatting

This checks whether the value contains:

real newline characters
literal \n
the expected PEM headers/footers

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings

s = get_settings()
key = s.jwt_private_key

print("type:", type(key))
print("length:", len(key))
print("literal_backslash_n:", "\\n" in key)
print("real_newline:", "\n" in key)
print("starts PEM:", key.startswith("-----BEGIN PRIVATE KEY-----"))
print("ends PEM:", key.endswith("-----END PRIVATE KEY-----"))
print("repr:", repr(key[:100]))
PY
'

Expected:

text

literal_backslash_n: False
real_newline: True
starts PEM: True
ends PEM: True

3. Verify the Private Key Cryptographically

Formatting alone is not sufficient.

Use cryptography to verify that the PEM is actually loadable.

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings
from cryptography.hazmat.primitives.serialization import load_pem_private_key

s = get_settings()
key = s.jwt_private_key

print("PEM length:", len(key))
print("Literal \\\\n:", "\\\\n" in key)
print("Real newline:", "\n" in key)

try:
    load_pem_private_key(
        key.encode(),
        password=None,
    )
    print("PRIVATE KEY: VALID")
except Exception as e:
    print("PRIVATE KEY: INVALID")
    print(type(e).__name__, str(e))
PY
'

Expected:

text

PRIVATE KEY: VALID

If you see:

text

InvalidData(Invalid symbol 92, offset ...)

there is still a literal backslash inside the PEM payload.

4. Verify the Public Key Cryptographically

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings
from cryptography.hazmat.primitives.serialization import load_pem_public_key

s = get_settings()
key = s.jwt_public_key

print("PEM length:", len(key))
print("Literal \\\\n:", "\\\\n" in key)
print("Real newline:", "\n" in key)

try:
    load_pem_public_key(key.encode())
    print("PUBLIC KEY: VALID")
except Exception as e:
    print("PUBLIC KEY: INVALID")
    print(type(e).__name__, str(e))
PY
'

Expected:

text

PUBLIC KEY: VALID

Both keys must pass.

5. Inspect the `.env` Representation

If the cryptographic validation fails, inspect the actual .env representation.

Do not print the complete key.

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from pathlib import Path

p = Path("/app/.env")

print("exists:", p.exists())

if p.exists():
    text = p.read_text()

    for name in ("JWT_PRIVATE_KEY", "JWT_PUBLIC_KEY"):
        for line in text.splitlines():
            if line.startswith(name + "="):
                value = line.split("=", 1)[1]

                print("\n" + name)
                print("raw line length:", len(value))
                print("backslashes:", value.count("\\"))
                print("first 150:", repr(value[:150]))
                break
PY
'

This helps identify malformed values such as:

text

\7
\xB4
\G
\z

inside the Base64 payload.

Those are not valid PEM/Base64 content.

6. Inspect Backslashes in the PEM Loaded by Pydantic

If the .env representation looks suspicious, inspect the actual value after Pydantic processing.

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings

s = get_settings()

for name in ("jwt_private_key", "jwt_public_key"):
    key = getattr(s, name)

    print("\n---", name, "---")
    print("length:", len(key))
    print("backslashes:", key.count("\\"))
    print("real newlines:", key.count("\n"))

    for i, char in enumerate(key):
        if char == "\\":
            start = max(0, i - 20)
            end = min(len(key), i + 25)
            print("BACKSLASH at:", i)
            print("context:", repr(key[start:end]))
PY
'

A correctly formatted PEM should have:

backslashes: 0
real newlines: multiple

7. Verify Database Configuration

Print only non-sensitive database metadata.

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings
from sqlalchemy.engine import make_url

s = get_settings()

url = make_url(s.database_url)

print("=== AUTHX CONFIG ===")
print("DATABASE_HOST:", url.host)
print("DATABASE_PORT:", url.port)
print("DATABASE_NAME:", url.database)
print("JWT_ALGORITHM:", s.jwt_algorithm)
PY
'

Avoid printing:

text

DATABASE_URL
DATABASE_URL_SYNC
passwords
private keys
internal tokens

8. List Database Tables

Use properly quoted PostgreSQL schema names.

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(text("""
            SELECT table_schema, table_name
            FROM information_schema.tables
            WHERE table_schema NOT IN (
                '\''pg_catalog'\'',
                '\''information_schema'\''
            )
            ORDER BY table_schema, table_name
        """))

        for row in result:
            print(f"{row[0]}.{row[1]}")

asyncio.run(main())
PY
'

Expected AuthX tables included:

text

public.alembic_version
public.refresh_token
public.user_identity

Important SQL mistake

This is wrong:

sql

WHERE table_schema NOT IN (pg_catalog, information_schema)

PostgreSQL interprets those as identifiers/column names.

Correct:

sql

WHERE table_schema NOT IN ('pg_catalog', 'information_schema')

9. Inspect `user_identity` Columns

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(text("""
            SELECT column_name, data_type
            FROM information_schema.columns
            WHERE table_schema = '\''public'\''
              AND table_name = '\''user_identity'\''
            ORDER BY ordinal_position
        """))

        for row in result:
            print(f"{row[0]} ({row[1]})")

asyncio.run(main())
PY
'

Important authentication columns:

text

email
password_hash
is_active
is_verified
deleted_at
last_login

10. Find Identities Without Exposing Password Hashes

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(text("""
            SELECT
                id,
                email,
                username,
                is_active,
                is_verified,
                deleted_at,
                CASE
                    WHEN password_hash IS NULL THEN '\''NULL'\''
                    WHEN password_hash = '\'''\'' THEN '\''EMPTY'\''
                    ELSE '\''PRESENT'\''
                END AS password_status
            FROM public.user_identity
            ORDER BY id
        """))

        rows = result.fetchall()

        if not rows:
            print("NO USER_IDENTITY RECORDS FOUND")
        else:
            for row in rows:
                print(dict(row._mapping))

asyncio.run(main())
PY
'

Expected healthy account:

text

is_active: True
is_verified: True
deleted_at: None
password_status: PRESENT

Do not print the complete password_hash.

11. Verify the Exact Identity Lookup Logic

AuthX currently uses:

python

async def _get_by_email(self, email: str) -> UserIdentity | None:
    result = await self.db.execute(
        select(UserIdentity).where(
            UserIdentity.email == email.lower().strip(),
            UserIdentity.deleted_at.is_(None),
        )
    )
    return result.scalar_one_or_none()

Therefore an email lookup can be tested directly:

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio

from sqlalchemy import select
from authx.db.session import AsyncSessionLocal
from authx.models.user_identity import UserIdentity

EMAIL = "YOUR_EMAIL"

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(
            select(UserIdentity).where(
                UserIdentity.email == EMAIL.lower().strip(),
                UserIdentity.deleted_at.is_(None),
            )
        )

        identity = result.scalar_one_or_none()

        if not identity:
            print("IDENTITY: NOT FOUND")
            return

        print("IDENTITY: FOUND")
        print("id:", identity.id)
        print("email:", identity.email)
        print("is_active:", identity.is_active)
        print("is_verified:", identity.is_verified)
        print("deleted_at:", identity.deleted_at)
        print("password_hash exists:", bool(identity.password_hash))

asyncio.run(main())
PY
'

12. Verify Password Independently

If the identity exists, test the password hash directly.

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio

from sqlalchemy import text
from authx.db.session import AsyncSessionLocal
from authx.core.security import verify_password

EMAIL = "YOUR_EMAIL"
PASSWORD = "YOUR_PASSWORD"

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(
            text("""
                SELECT password_hash
                FROM public.user_identity
                WHERE lower(email) = lower(:email)
                  AND deleted_at IS NULL
            """),
            {"email": EMAIL},
        )

        row = result.mappings().first()

        if not row:
            print("IDENTITY: NOT FOUND")
            return

        hashed = row["password_hash"]

        if not hashed:
            print("PASSWORD HASH: EMPTY")
            return

        try:
            result = verify_password(PASSWORD, hashed)
            print("PASSWORD VERIFY:", result)
        except Exception as exc:
            print(
                "PASSWORD VERIFY ERROR:",
                type(exc).__name__,
                str(exc),
            )

asyncio.run(main())
PY
'

Expected:

text

PASSWORD VERIFY: True

If:

text

PASSWORD VERIFY: False

the password supplied to the test does not match the stored password.

If an exception occurs, investigate the Passlib/bcrypt configuration.

13. Verify Passlib and bcrypt Versions

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import bcrypt
import passlib

print("bcrypt:", getattr(bcrypt, "__version__", "unknown"))
print("passlib:", getattr(passlib, "__version__", "unknown"))
PY
'

For the environment tested during this debugging session:

text

bcrypt: 4.0.1
passlib: 1.7.4

14. Test the Entire Authentication Service

This is the most useful final diagnostic.

It reproduces the actual AuthX authentication service rather than testing individual pieces.

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio

from authx.db.session import AsyncSessionLocal
from authx.services.identity_service import IdentityService

EMAIL = "YOUR_EMAIL"
PASSWORD = "YOUR_PASSWORD"

async def main():
    async with AsyncSessionLocal() as db:
        print("=== AUTHENTICATE ===")

        try:
            service = IdentityService(db)

            response = await service.authenticate(
                EMAIL,
                PASSWORD,
            )

            print("AUTHENTICATE: SUCCESS")
            print("response type:", type(response).__name__)

            if hasattr(response, "access_token"):
                print(
                    "access_token present:",
                    bool(response.access_token),
                )

            if hasattr(response, "refresh_token"):
                print(
                    "refresh_token present:",
                    bool(response.refresh_token),
                )

        except Exception as exc:
            print("AUTHENTICATE: ERROR")
            print(type(exc).__name__, str(exc))

asyncio.run(main())
PY
'

15. Understand the Authentication Flow

The relevant AuthX flow is:

text

POST /token
    │
    ▼
IdentityService.authenticate()
    │
    ▼
_get_by_email()
    │
    ├── identity not found ───────────────► 401
    │
    └── identity found
            │
            ├── inactive/deleted ─────────► 401
            │
            ├── password_hash missing ────► 401
            │
            ▼
       verify_password()
            │
            ├── False ────────────────────► 401
            │
            ▼
       _issue_tokens()
            │
            ▼
       create_access_token()
            │
            ▼
       jwt.encode()

This distinction is important:

`Invalid credentials`

Usually means the request failed before successful token issuance.

`Unable to load PEM file`

Means authentication reached JWT signing and the key material was invalid.

These are different layers of the system.

16. When DjangoPlay Returns `401`

If DjangoPlay reports:

text

HTTP/1.1 401 Unauthorized

and AuthX returns:

json

{
  "detail": "Invalid credentials."
}

debug in this order:

text

1. Does user_identity exist?
        ↓
2. Does email exactly match after lower().strip()?
        ↓
3. is_active == True?
        ↓
4. deleted_at == NULL?
        ↓
5. password_hash present?
        ↓
6. verify_password() == True?
        ↓
7. IdentityService.authenticate() succeeds?
        ↓
8. JWT signing succeeds?

Do not start with JWT debugging if the response is already:

text

Invalid credentials.

JWT signing happens after successful password authentication.

17. Container Restart After Configuration Changes

After modifying .env, rebuild/recreate the AuthX container so the new environment is actually loaded.

For a normal Docker Compose workflow:

bash

docker compose build authx
docker compose up -d authx

If both DjangoPlay and AuthX need to be recreated:

bash

docker compose up -d --build authx djangoplay

Then verify:

bash

docker compose ps

and:

bash

docker logs authx-identity-authx-1 --tail 100

18. Basic Health Check

After restarting AuthX:

bash

curl -i http://localhost:8100/health

Expected:

text

HTTP/1.1 200 OK

This confirms that the service is running.

It does not prove that:

database authentication works
password authentication works
JWT signing works
DjangoPlay integration works

Those require the deeper checks above.

19. Recommended Debugging Order

For future AuthX authentication problems, use this sequence:

text

┌───────────────────────────────┐
│ 1. Container running?         │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 2. /health returns 200?       │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 3. Settings loaded correctly? │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 4. PEM keys cryptographically │
│    valid?                     │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 5. Correct database?          │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 6. user_identity exists?      │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 7. Account active/not deleted?│
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 8. Password hash present?     │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 9. Password verifies?         │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 10. IdentityService succeeds? │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 11. DjangoPlay integration?   │
└───────────────────────────────┘

Most probable issues

PEM problem

The JWT keys could contain invalid literal backslashes inside their Base64 payload.

The important validation is not merely:

text

real_newline: True

but:

text

cryptography.load_pem_private_key() → VALID
cryptography.load_pem_public_key()  → VALID

Database problem

The original diagnostic queried:

text

identities

but the actual SQLAlchemy model uses:

text

user_identity

The model definition is the authoritative source.

SQL diagnostic problem

This:

sql

NOT IN (pg_catalog, information_schema)

is invalid because those values need string literals.

Use:

sql

NOT IN ('pg_catalog', 'information_schema')

Authentication problem

A 401 Invalid credentials does not automatically indicate a JWT problem.

The authentication service performs password verification before issuing the JWT.

Quick Reference

For better debugging, these are the highest-value commands:

Health

bash

curl -i http://localhost:8100/health

Check JWT PEM

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings
from cryptography.hazmat.primitives.serialization import (
    load_pem_private_key,
    load_pem_public_key,
)

s = get_settings()

try:
    load_pem_private_key(s.jwt_private_key.encode(), None)
    print("PRIVATE KEY: VALID")
except Exception as e:
    print("PRIVATE KEY: INVALID:", e)

try:
    load_pem_public_key(s.jwt_public_key.encode())
    print("PUBLIC KEY: VALID")
except Exception as e:
    print("PUBLIC KEY: INVALID:", e)
PY
'

List tables

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(text("""
            SELECT table_schema, table_name
            FROM information_schema.tables
            WHERE table_schema NOT IN (
                '\''pg_catalog'\'',
                '\''information_schema'\''
            )
            ORDER BY table_schema, table_name
        """))

        for row in result:
            print(f"{row[0]}.{row[1]}")

asyncio.run(main())
PY
'

Test password

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal
from authx.core.security import verify_password

EMAIL = "YOUR_EMAIL"
PASSWORD = "YOUR_PASSWORD"

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(
            text("""
                SELECT password_hash
                FROM public.user_identity
                WHERE lower(email) = lower(:email)
                  AND deleted_at IS NULL
            """),
            {"email": EMAIL},
        )

        row = result.mappings().first()

        if not row:
            print("IDENTITY: NOT FOUND")
            return

        print(
            "PASSWORD VERIFY:",
            verify_password(PASSWORD, row["password_hash"]),
        )

asyncio.run(main())
PY
'

Test complete AuthX authentication

bash

docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from authx.db.session import AsyncSessionLocal
from authx.services.identity_service import IdentityService

EMAIL = "YOUR_EMAIL"
PASSWORD = "YOUR_PASSWORD"

async def main():
    async with AsyncSessionLocal() as db:
        try:
            response = await IdentityService(db).authenticate(
                EMAIL,
                PASSWORD,
            )
            print("AUTHENTICATE: SUCCESS")
            print("response:", type(response).__name__)
        except Exception as exc:
            print("AUTHENTICATE: ERROR")
            print(type(exc).__name__, str(exc))

asyncio.run(main())
PY
'

Use the last command when you want to determine quickly whether the problem is actually inside AuthX authentication or somewhere in the DjangoPlay → AuthX integration.

AuthX Identity — Authentication Debugging Runbook

#1. Verify AuthX Settings Are Loading

#Check JWT configuration

2. Verify JWT Private Key Formatting

3. Verify the Private Key Cryptographically

4. Verify the Public Key Cryptographically

5. Inspect the .env Representation

6. Inspect Backslashes in the PEM Loaded by Pydantic

7. Verify Database Configuration

8. List Database Tables

#Important SQL mistake

9. Inspect user_identity Columns

10. Find Identities Without Exposing Password Hashes

11. Verify the Exact Identity Lookup Logic

12. Verify Password Independently

13. Verify Passlib and bcrypt Versions

14. Test the Entire Authentication Service

15. Understand the Authentication Flow

#Invalid credentials

#Unable to load PEM file

16. When DjangoPlay Returns 401

17. Container Restart After Configuration Changes

18. Basic Health Check

19. Recommended Debugging Order

#Most probable issues

#PEM problem

#Database problem

#SQL diagnostic problem

#Authentication problem

Quick Reference

#Health

#Check JWT PEM

#List tables

#Test password

#Test complete AuthX authentication

1. Verify AuthX Settings Are Loading

Check JWT configuration

5. Inspect the `.env` Representation

Important SQL mistake

9. Inspect `user_identity` Columns

`Invalid credentials`

`Unable to load PEM file`

16. When DjangoPlay Returns `401`

Most probable issues

PEM problem

Database problem

SQL diagnostic problem

Authentication problem

Health

Check JWT PEM

List tables

Test password

Test complete AuthX authentication