---
since: 1.1.1
---

# AuthX Identity — Authentication Debugging Runbook

This runbook covers the diagnostic commands.

It is intended for debugging:

- AuthX container configuration
- JWT private/public key loading
- PostgreSQL connectivity
- database schema/table availability
- identity lookup
- password authentication
- `/token` `401 Unauthorized` responses
- DjangoPlay → AuthX authentication integration

---

## 1. Verify AuthX Settings Are Loading

The settings object is exposed through `get_settings()`.

### Check JWT configuration

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings

s = get_settings()

print("JWT_ALGORITHM:", s.jwt_algorithm)
print("JWT_ISSUER:", s.jwt_issuer)
print("JWT_AUDIENCE:", s.jwt_audience)
print("APP_ENV:", s.app_env)
PY
'
```

Do **not** print private keys or other secrets.

---

# 2. Verify JWT Private Key Formatting

This checks whether the value contains:

- real newline characters
- literal `\n`
- the expected PEM headers/footers

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings

s = get_settings()
key = s.jwt_private_key

print("type:", type(key))
print("length:", len(key))
print("literal_backslash_n:", "\\n" in key)
print("real_newline:", "\n" in key)
print("starts PEM:", key.startswith("-----BEGIN PRIVATE KEY-----"))
print("ends PEM:", key.endswith("-----END PRIVATE KEY-----"))
print("repr:", repr(key[:100]))
PY
'
```

Expected:

```text
literal_backslash_n: False
real_newline: True
starts PEM: True
ends PEM: True
```

---

# 3. Verify the Private Key Cryptographically

Formatting alone is not sufficient.

Use `cryptography` to verify that the PEM is actually loadable.

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings
from cryptography.hazmat.primitives.serialization import load_pem_private_key

s = get_settings()
key = s.jwt_private_key

print("PEM length:", len(key))
print("Literal \\\\n:", "\\\\n" in key)
print("Real newline:", "\n" in key)

try:
    load_pem_private_key(
        key.encode(),
        password=None,
    )
    print("PRIVATE KEY: VALID")
except Exception as e:
    print("PRIVATE KEY: INVALID")
    print(type(e).__name__, str(e))
PY
'
```

Expected:

```text
PRIVATE KEY: VALID
```

If you see:

```text
InvalidData(Invalid symbol 92, offset ...)
```

there is still a literal backslash inside the PEM payload.

---

# 4. Verify the Public Key Cryptographically

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings
from cryptography.hazmat.primitives.serialization import load_pem_public_key

s = get_settings()
key = s.jwt_public_key

print("PEM length:", len(key))
print("Literal \\\\n:", "\\\\n" in key)
print("Real newline:", "\n" in key)

try:
    load_pem_public_key(key.encode())
    print("PUBLIC KEY: VALID")
except Exception as e:
    print("PUBLIC KEY: INVALID")
    print(type(e).__name__, str(e))
PY
'
```

Expected:

```text
PUBLIC KEY: VALID
```

Both keys must pass.

---

# 5. Inspect the `.env` Representation

If the cryptographic validation fails, inspect the actual `.env` representation.

Do not print the complete key.

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from pathlib import Path

p = Path("/app/.env")

print("exists:", p.exists())

if p.exists():
    text = p.read_text()

    for name in ("JWT_PRIVATE_KEY", "JWT_PUBLIC_KEY"):
        for line in text.splitlines():
            if line.startswith(name + "="):
                value = line.split("=", 1)[1]

                print("\n" + name)
                print("raw line length:", len(value))
                print("backslashes:", value.count("\\"))
                print("first 150:", repr(value[:150]))
                break
PY
'
```

This helps identify malformed values such as:

```text
\7
\xB4
\G
\z
```

inside the Base64 payload.

Those are **not valid PEM/Base64 content**.

---

# 6. Inspect Backslashes in the PEM Loaded by Pydantic

If the `.env` representation looks suspicious, inspect the actual value after Pydantic processing.

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings

s = get_settings()

for name in ("jwt_private_key", "jwt_public_key"):
    key = getattr(s, name)

    print("\n---", name, "---")
    print("length:", len(key))
    print("backslashes:", key.count("\\"))
    print("real newlines:", key.count("\n"))

    for i, char in enumerate(key):
        if char == "\\":
            start = max(0, i - 20)
            end = min(len(key), i + 25)
            print("BACKSLASH at:", i)
            print("context:", repr(key[start:end]))
PY
'
```

A correctly formatted PEM should have:

- backslashes: `0`
- real newlines: multiple

---

# 7. Verify Database Configuration

Print only non-sensitive database metadata.

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings
from sqlalchemy.engine import make_url

s = get_settings()

url = make_url(s.database_url)

print("=== AUTHX CONFIG ===")
print("DATABASE_HOST:", url.host)
print("DATABASE_PORT:", url.port)
print("DATABASE_NAME:", url.database)
print("JWT_ALGORITHM:", s.jwt_algorithm)
PY
'
```

Avoid printing:

```text
DATABASE_URL
DATABASE_URL_SYNC
passwords
private keys
internal tokens
```

---

# 8. List Database Tables

Use properly quoted PostgreSQL schema names.

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(text("""
            SELECT table_schema, table_name
            FROM information_schema.tables
            WHERE table_schema NOT IN (
                '\''pg_catalog'\'',
                '\''information_schema'\''
            )
            ORDER BY table_schema, table_name
        """))

        for row in result:
            print(f"{row[0]}.{row[1]}")

asyncio.run(main())
PY
'
```

Expected AuthX tables included:

```text
public.alembic_version
public.refresh_token
public.user_identity
```

### Important SQL mistake

This is wrong:

```sql
WHERE table_schema NOT IN (pg_catalog, information_schema)
```

PostgreSQL interprets those as identifiers/column names.

Correct:

```sql
WHERE table_schema NOT IN ('pg_catalog', 'information_schema')
```

---

# 9. Inspect `user_identity` Columns

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(text("""
            SELECT column_name, data_type
            FROM information_schema.columns
            WHERE table_schema = '\''public'\''
              AND table_name = '\''user_identity'\''
            ORDER BY ordinal_position
        """))

        for row in result:
            print(f"{row[0]} ({row[1]})")

asyncio.run(main())
PY
'
```

Important authentication columns:

```text
email
password_hash
is_active
is_verified
deleted_at
last_login
```

---

# 10. Find Identities Without Exposing Password Hashes

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(text("""
            SELECT
                id,
                email,
                username,
                is_active,
                is_verified,
                deleted_at,
                CASE
                    WHEN password_hash IS NULL THEN '\''NULL'\''
                    WHEN password_hash = '\'''\'' THEN '\''EMPTY'\''
                    ELSE '\''PRESENT'\''
                END AS password_status
            FROM public.user_identity
            ORDER BY id
        """))

        rows = result.fetchall()

        if not rows:
            print("NO USER_IDENTITY RECORDS FOUND")
        else:
            for row in rows:
                print(dict(row._mapping))

asyncio.run(main())
PY
'
```

Expected healthy account:

```text
is_active: True
is_verified: True
deleted_at: None
password_status: PRESENT
```

Do **not** print the complete `password_hash`.

---

# 11. Verify the Exact Identity Lookup Logic

AuthX currently uses:

```python
async def _get_by_email(self, email: str) -> UserIdentity | None:
    result = await self.db.execute(
        select(UserIdentity).where(
            UserIdentity.email == email.lower().strip(),
            UserIdentity.deleted_at.is_(None),
        )
    )
    return result.scalar_one_or_none()
```

Therefore an email lookup can be tested directly:

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio

from sqlalchemy import select
from authx.db.session import AsyncSessionLocal
from authx.models.user_identity import UserIdentity

EMAIL = "YOUR_EMAIL"

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(
            select(UserIdentity).where(
                UserIdentity.email == EMAIL.lower().strip(),
                UserIdentity.deleted_at.is_(None),
            )
        )

        identity = result.scalar_one_or_none()

        if not identity:
            print("IDENTITY: NOT FOUND")
            return

        print("IDENTITY: FOUND")
        print("id:", identity.id)
        print("email:", identity.email)
        print("is_active:", identity.is_active)
        print("is_verified:", identity.is_verified)
        print("deleted_at:", identity.deleted_at)
        print("password_hash exists:", bool(identity.password_hash))

asyncio.run(main())
PY
'
```

---

# 12. Verify Password Independently

If the identity exists, test the password hash directly.

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio

from sqlalchemy import text
from authx.db.session import AsyncSessionLocal
from authx.core.security import verify_password

EMAIL = "YOUR_EMAIL"
PASSWORD = "YOUR_PASSWORD"

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(
            text("""
                SELECT password_hash
                FROM public.user_identity
                WHERE lower(email) = lower(:email)
                  AND deleted_at IS NULL
            """),
            {"email": EMAIL},
        )

        row = result.mappings().first()

        if not row:
            print("IDENTITY: NOT FOUND")
            return

        hashed = row["password_hash"]

        if not hashed:
            print("PASSWORD HASH: EMPTY")
            return

        try:
            result = verify_password(PASSWORD, hashed)
            print("PASSWORD VERIFY:", result)
        except Exception as exc:
            print(
                "PASSWORD VERIFY ERROR:",
                type(exc).__name__,
                str(exc),
            )

asyncio.run(main())
PY
'
```

Expected:

```text
PASSWORD VERIFY: True
```

If:

```text
PASSWORD VERIFY: False
```

the password supplied to the test does not match the stored password.

If an exception occurs, investigate the Passlib/bcrypt configuration.

---

# 13. Verify Passlib and bcrypt Versions

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import bcrypt
import passlib

print("bcrypt:", getattr(bcrypt, "__version__", "unknown"))
print("passlib:", getattr(passlib, "__version__", "unknown"))
PY
'
```

For the environment tested during this debugging session:

```text
bcrypt: 4.0.1
passlib: 1.7.4
```

---

# 14. Test the Entire Authentication Service

This is the most useful final diagnostic.

It reproduces the actual AuthX authentication service rather than testing individual pieces.

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio

from authx.db.session import AsyncSessionLocal
from authx.services.identity_service import IdentityService

EMAIL = "YOUR_EMAIL"
PASSWORD = "YOUR_PASSWORD"

async def main():
    async with AsyncSessionLocal() as db:
        print("=== AUTHENTICATE ===")

        try:
            service = IdentityService(db)

            response = await service.authenticate(
                EMAIL,
                PASSWORD,
            )

            print("AUTHENTICATE: SUCCESS")
            print("response type:", type(response).__name__)

            if hasattr(response, "access_token"):
                print(
                    "access_token present:",
                    bool(response.access_token),
                )

            if hasattr(response, "refresh_token"):
                print(
                    "refresh_token present:",
                    bool(response.refresh_token),
                )

        except Exception as exc:
            print("AUTHENTICATE: ERROR")
            print(type(exc).__name__, str(exc))

asyncio.run(main())
PY
'
```

---

# 15. Understand the Authentication Flow

The relevant AuthX flow is:

```text
POST /token
    │
    ▼
IdentityService.authenticate()
    │
    ▼
_get_by_email()
    │
    ├── identity not found ───────────────► 401
    │
    └── identity found
            │
            ├── inactive/deleted ─────────► 401
            │
            ├── password_hash missing ────► 401
            │
            ▼
       verify_password()
            │
            ├── False ────────────────────► 401
            │
            ▼
       _issue_tokens()
            │
            ▼
       create_access_token()
            │
            ▼
       jwt.encode()
```

This distinction is important:

### `Invalid credentials`

Usually means the request failed before successful token issuance.

### `Unable to load PEM file`

Means authentication reached JWT signing and the key material was invalid.

These are **different layers of the system**.

---

# 16. When DjangoPlay Returns `401`

If DjangoPlay reports:

```text
HTTP/1.1 401 Unauthorized
```

and AuthX returns:

```json
{
  "detail": "Invalid credentials."
}
```

debug in this order:

```text
1. Does user_identity exist?
        ↓
2. Does email exactly match after lower().strip()?
        ↓
3. is_active == True?
        ↓
4. deleted_at == NULL?
        ↓
5. password_hash present?
        ↓
6. verify_password() == True?
        ↓
7. IdentityService.authenticate() succeeds?
        ↓
8. JWT signing succeeds?
```

Do not start with JWT debugging if the response is already:

```text
Invalid credentials.
```

JWT signing happens **after** successful password authentication.

---

# 17. Container Restart After Configuration Changes

After modifying `.env`, rebuild/recreate the AuthX container so the new environment is actually loaded.

For a normal Docker Compose workflow:

```bash
docker compose build authx
docker compose up -d authx
```

If both DjangoPlay and AuthX need to be recreated:

```bash
docker compose up -d --build authx djangoplay
```

Then verify:

```bash
docker compose ps
```

and:

```bash
docker logs authx-identity-authx-1 --tail 100
```

---

# 18. Basic Health Check

After restarting AuthX:

```bash
curl -i http://localhost:8100/health
```

Expected:

```text
HTTP/1.1 200 OK
```

This confirms that the service is running.

It does **not** prove that:

- database authentication works
- password authentication works
- JWT signing works
- DjangoPlay integration works

Those require the deeper checks above.

---

# 19. Recommended Debugging Order

For future AuthX authentication problems, use this sequence:

```text
┌───────────────────────────────┐
│ 1. Container running?         │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 2. /health returns 200?       │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 3. Settings loaded correctly? │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 4. PEM keys cryptographically │
│    valid?                     │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 5. Correct database?          │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 6. user_identity exists?      │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 7. Account active/not deleted?│
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 8. Password hash present?     │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 9. Password verifies?         │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 10. IdentityService succeeds? │
└──────────────┬────────────────┘
               ▼
┌───────────────────────────────┐
│ 11. DjangoPlay integration?   │
└───────────────────────────────┘
```

---

## Most probable issues

### PEM problem

The JWT keys could contain invalid literal backslashes inside their Base64 payload.

The important validation is not merely:

```text
real_newline: True
```

but:

```text
cryptography.load_pem_private_key() → VALID
cryptography.load_pem_public_key()  → VALID
```

### Database problem

The original diagnostic queried:

```text
identities
```

but the actual SQLAlchemy model uses:

```text
user_identity
```

The model definition is the authoritative source.

### SQL diagnostic problem

This:

```sql
NOT IN (pg_catalog, information_schema)
```

is invalid because those values need string literals.

Use:

```sql
NOT IN ('pg_catalog', 'information_schema')
```

### Authentication problem

A `401 Invalid credentials` does not automatically indicate a JWT problem.

The authentication service performs password verification **before** issuing the JWT.


---

# Quick Reference

For better debugging, these are the highest-value commands:

### Health

```bash
curl -i http://localhost:8100/health
```

### Check JWT PEM

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
from authx.core.settings import get_settings
from cryptography.hazmat.primitives.serialization import (
    load_pem_private_key,
    load_pem_public_key,
)

s = get_settings()

try:
    load_pem_private_key(s.jwt_private_key.encode(), None)
    print("PRIVATE KEY: VALID")
except Exception as e:
    print("PRIVATE KEY: INVALID:", e)

try:
    load_pem_public_key(s.jwt_public_key.encode())
    print("PUBLIC KEY: VALID")
except Exception as e:
    print("PUBLIC KEY: INVALID:", e)
PY
'
```

### List tables

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(text("""
            SELECT table_schema, table_name
            FROM information_schema.tables
            WHERE table_schema NOT IN (
                '\''pg_catalog'\'',
                '\''information_schema'\''
            )
            ORDER BY table_schema, table_name
        """))

        for row in result:
            print(f"{row[0]}.{row[1]}")

asyncio.run(main())
PY
'
```

### Test password

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from sqlalchemy import text
from authx.db.session import AsyncSessionLocal
from authx.core.security import verify_password

EMAIL = "YOUR_EMAIL"
PASSWORD = "YOUR_PASSWORD"

async def main():
    async with AsyncSessionLocal() as db:
        result = await db.execute(
            text("""
                SELECT password_hash
                FROM public.user_identity
                WHERE lower(email) = lower(:email)
                  AND deleted_at IS NULL
            """),
            {"email": EMAIL},
        )

        row = result.mappings().first()

        if not row:
            print("IDENTITY: NOT FOUND")
            return

        print(
            "PASSWORD VERIFY:",
            verify_password(PASSWORD, row["password_hash"]),
        )

asyncio.run(main())
PY
'
```

### Test complete AuthX authentication

```bash
docker exec authx-identity-authx-1 sh -lc '
python - <<'\''PY'\''
import asyncio
from authx.db.session import AsyncSessionLocal
from authx.services.identity_service import IdentityService

EMAIL = "YOUR_EMAIL"
PASSWORD = "YOUR_PASSWORD"

async def main():
    async with AsyncSessionLocal() as db:
        try:
            response = await IdentityService(db).authenticate(
                EMAIL,
                PASSWORD,
            )
            print("AUTHENTICATE: SUCCESS")
            print("response:", type(response).__name__)
        except Exception as exc:
            print("AUTHENTICATE: ERROR")
            print(type(exc).__name__, str(exc))

asyncio.run(main())
PY
'
```

**Use the last command when you want to determine quickly whether the problem is actually inside AuthX authentication or somewhere in the DjangoPlay → AuthX integration.**