Secrets Management

All production and development secrets live in AWS Secrets Manager. ECS Fargate workloads (backend + bg-tasks) consume them via the bundle JSON model: ECS injects the entire JSON-formatted secret as a single env var (_BUNDLE_JSON) at task startup, and the application's bootstrap-env.ts parses it into process.env before any other module loads. Jitsi (the only remaining EC2 workload) still uses the older fetch-secrets.sh shell-script model.

Secret Paths

Secret Path	Environment	Used By
aiqlick-backend/production	Production	ECS backend-prod, bg-tasks-prod (single bundle covers both services)
aiqlick-backend/development	Development	ECS backend-dev, bg-tasks-dev
jitsi/production	Production	Jitsi EC2 (XMPP passwords, JWT secret)
support-backend-rust/production	Production	support-backend-rust service
support-backend-rust/development	Development	support-backend-rust service
support-backend-rust/bootstrap-admin-prod	Production	One-time bootstrap admin credentials for support-backend

One bundle per env covers both backend + bg-tasks

The aiqlick-backend/{environment} secrets contain the union of variables both backend and bg-tasks need (DATABASE_URL, JWT_SECRET, Stripe keys, Bedrock config, queue URLs, etc.). Each ECS task definition references the same secret ARN — one source of truth, one update path.

How ECS workloads load secrets (the bundle model)

Task definition

Each ECS task def declares one entry under secrets[], mapping the secret's full JSON into a single env var named _BUNDLE_JSON:

"secrets": [
  {
    "name": "_BUNDLE_JSON",
    "valueFrom": "arn:aws:secretsmanager:eu-north-1:842697652860:secret:aiqlick-backend/development"
  }
]

ECS pulls the secret at task startup (using the execution role's secretsmanager:GetSecretValue permission, NOT the task role) and injects the JSON string into the container env.

Bootstrap-env loader

The application has a synchronous loader that runs before any other import. For backend (NestJS):

// aiqlick-backend/src/bootstrap-env.ts
const bundle = process.env._BUNDLE_JSON;
if (bundle) {
  const parsed = JSON.parse(bundle);
  for (const [key, value] of Object.entries(parsed)) {
    if (process.env[key] === undefined || process.env[key] === '') {
      process.env[key] = String(value);
    }
  }
}

This runs before import 'dotenv' and before any service module loads. Modules that read env vars at import time (stripe.service.ts, prisma.config.ts) see the populated values. For local dev, .env takes precedence (the loader treats '' as not-set so the empty defaults baked into the Dockerfile don't block bundle hydration).

Background-tasks has the equivalent in Python:

# background-tasks/app/bootstrap_env.py
import json, os
bundle = os.environ.get("_BUNDLE_JSON")
if bundle:
    for k, v in json.loads(bundle).items():
        if not os.environ.get(k):
            os.environ[k] = str(v)

Why bundle-and-load instead of per-secret env vars

Two reasons:

1. One IAM call, one task-def entry. Adding a new secret variable doesn't require a task-def revision or a new IAM grant — just put the new key into the JSON bundle.
2. Hydration ordering is explicit. Modules that read process.env.STRIPE_SECRET_KEY at import time would race against aws-sdk async secret resolution; bootstrap-env is synchronous JSON parsing of an already-populated env var, so there's no race.

Reading a value from inside a running task

When debugging via ECS Exec, the bundle is in env. To get one value:

echo "$_BUNDLE_JSON" | python3 -c \
  'import json,sys; print(json.loads(sys.stdin.read()).get("DATABASE_URL", ""))'

Don't print the entire bundle to logs — it contains every secret.

How Jitsi EC2 loads secrets (legacy fetch-secrets.sh)

The Jitsi EC2 still uses the older shell-script model. There's a fetch-secrets.sh script on the instance that reads from Secrets Manager and writes a .env file the Docker Compose stack consumes.

aws ssm start-session --profile Administrator-842697652860 --region eu-north-1 \
  --target i-0620d2e23695f5bfc

# Inside the session:
sudo /home/ec2-user/fetch-secrets.sh
sudo systemctl restart jitsi-meet  # or docker compose up -d for the affected services

warning

docker restart doesn't pick up new env-file values

Docker captures the env-file contents at container creation time, not at restart time. After fetch-secrets.sh, you must docker compose up -d --force-recreate (or stop + rm + run) for the new values to take effect. Documented bug from a Stripe go-live where new keys were in the env-file but the container kept using the old test keys.

IAM Permissions

ECS execution roles (where Secrets Manager read happens)

aiqlick-ecs-execution-role-{dev,prod} carries the secretsmanager:GetSecretValue grant for the bundle ARN:

{
  "Effect": "Allow",
  "Action": ["secretsmanager:GetSecretValue"],
  "Resource": [
    "arn:aws:secretsmanager:eu-north-1:842697652860:secret:aiqlick-backend/development*",
    "arn:aws:secretsmanager:eu-north-1:842697652860:secret:aiqlick-backend/production*"
  ]
}

This is the execution role, not the task role — it's the role Fargate assumes to launch the container, separate from the role the application uses at runtime.

Jitsi EC2

Carries secretsmanager:GetSecretValue on jitsi/production* via the instance profile.

Frontend Secrets (Amplify)

The Next.js frontend on AWS Amplify uses a separate mechanism since Amplify can't read _BUNDLE_JSON from a Fargate context:

1. Each Amplify branch has a SECRET_NAME env var set in the Amplify console.
2. At build time, Amplify reads the secret named in SECRET_NAME (e.g., aiqlick-backend/production).
3. The build process fetches the secret from Secrets Manager and injects values as build-time env vars.
4. Only NEXT_PUBLIC_* prefixed variables are exposed to the client bundle.

Amplify Branch	SECRET_NAME	Domain
dev	aiqlick-backend/development	dev.aiqlick.com
main	aiqlick-backend/production	www.aiqlick.com

warning

Frontend secrets are baked into the build at deploy time. After updating a secret that affects the frontend, trigger a new Amplify build for the changes to take effect.

GitHub Actions Secrets vs AWS Secrets Manager

Source	Purpose	Examples
GitHub Actions Secrets	CI/CD pipeline authentication (only AWS_ACCOUNT_ID remains; the SSH keys were removed when ECS replaced SSH-deploy)	AWS_ACCOUNT_ID
AWS Secrets Manager	Application runtime secrets	DATABASE_URL, JWT_SECRET, STRIPE_SECRET_KEY, etc.

GitHub Actions authenticates with AWS via OIDC — no static AWS access keys in GitHub. The github-actions-role IAM role trusts the GitHub OIDC provider:

- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::842697652860:role/github-actions-role
    aws-region: eu-north-1

Updating a Secret

# View current secret (use with caution — output contains everything)
aws secretsmanager get-secret-value \
  --profile Administrator-842697652860 --region eu-north-1 \
  --secret-id aiqlick-backend/production \
  --query SecretString --output text | jq .

# Update with the FULL JSON object (Secrets Manager stores it as a single string)
aws secretsmanager put-secret-value \
  --profile Administrator-842697652860 --region eu-north-1 \
  --secret-id aiqlick-backend/production \
  --secret-string '{"DATABASE_URL":"...", "JWT_SECRET":"...", ...}'

warning

Secrets Manager stores the entire secret as a single JSON string. When updating, you must provide the complete JSON object, not just the changed fields. Fetch the current value first, modify it, then put the updated version back.

Picking up the new value on ECS

ECS reads the secret at task startup, so a service redeploy is enough:

aws ecs update-service \
  --profile Administrator-842697652860 --region eu-north-1 \
  --cluster aiqlick --service backend-prod --force-new-deployment

The deployment circuit breaker rolls back automatically on health-check failure. No need to ssh anywhere or recreate containers manually — the old task drains, the new task starts with the fresh bundle.

Picking up the new value on Jitsi

aws ssm start-session --profile Administrator-842697652860 --region eu-north-1 \
  --target i-0620d2e23695f5bfc
sudo /home/ec2-user/fetch-secrets.sh
docker compose -f /opt/jitsi/docker-compose.yml up -d --force-recreate

Key Secret Categories (in aiqlick-backend/{env})

Category	Variables
Database	DATABASE_URL, DATABASE_URL_ASYNCPG
Auth	JWT_SECRET, JWT_EXPIRES_IN, GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET
Stripe	STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, STRIPE_PUBLISHABLE_KEY
Email (SES & SMTP)	MAIL_TRANSPORT, SES_FROM_ADDRESS, SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASSWORD, SMTP_FROM
S3	S3_BUCKET_NAME, S3_PUBLIC_ENDPOINT, S3_REGION
Jitsi	JITSI_MEET_URL, JITSI_MEET_APP_ID, JITSI_MEET_SECRET, JITSI_JWT_SECRET
Inbound Email (IMAP, prod only)	EMAIL_IMAP_HOST, EMAIL_IMAP_USER, EMAIL_IMAP_PASSWORD
Bedrock	BEDROCK_REGION, BEDROCK_MODEL_ID, BEDROCK_OCR_MODEL_ID, BEDROCK_HAIKU_MODEL_ID
Internal service	INTERNAL_SERVICE_SECRET, INTERNAL_ADMIN_TOKEN, WEBSOCKET_AUTH_TOKEN
Cross-service URLs	AI_BACKEND_GRAPHQL_URL, AI_BACKEND_GRAPHQL_WS_URL, AI_BACKEND_VOICE_WS_URL, BACKEND_URL, FRONTEND_URL
AWS messaging	AWS_EVENT_BUS_NAME, BG_EVENTS_QUEUE_URL, MAIL_QUEUE_URL, AGENT_DOCS_QUEUE_URL, NOTIFICATIONS_*_QUEUE_URL, INBOUND_*_QUEUE_URL
Application	NODE_ENV, PORT, LOG_LEVEL, LOG_FORMAT, etc.

Security Notes

• Secrets are never baked into Docker images — they live entirely in Secrets Manager and are pulled at task/container startup.
• ECS task execution roles + EC2 instance profiles use IAM, not static credentials.
• OIDC federation handles GitHub Actions; no AWS keys stored in GitHub.
• The bundle JSON is in container env at runtime — anyone with ecs:ExecuteCommand can read it. Treat ECS Exec access as production-secret access.
• Frontend secrets are exposed only via the NEXT_PUBLIC_* prefix gate; everything else stays server-side at build time.