Inbound email — prod cutover & runbook

This runbook captures the May 2026 production cutover from the legacy IMAP-polling inbound path (imap.one.com → background-tasks polling) to the AWS-native SES → S3 → SQS path. It also covers the day-2 operations: provisioning a new env, slug backfills, and the IMAP shutoff procedure.

For the full architectural context see AWS-Native Migration. For per-company addressing semantics see Inbound Email Processing.

Provisioning a new environment

Prod was provisioned with this exact sequence on 2026-05-02. The order matters — dependencies flow downward.

1. SQS queues + DLQs

Create the DLQ first, then the main queue with a RedrivePolicy pointing at it. Mirror the dev shape (visibility 60s for mail, 120s for inbound, retention 4 days, maxReceiveCount=3). The 7 prod queues are:

Main queue	DLQ	Visibility
aiqlick-mail-prod	aiqlick-mail-prod-dlq	60s
aiqlick-inbound-jobs-prod	aiqlick-inbound-jobs-prod-dlq	120s
aiqlick-inbound-cvs-prod	aiqlick-inbound-cvs-prod-dlq	120s
aiqlick-notifications-delivery-prod	aiqlick-notifications-delivery-prod-dlq	120s
aiqlick-notifications-fanout-prod	aiqlick-notifications-fanout-prod-dlq	120s
aiqlick-bg-events-prod	aiqlick-bg-events-prod-dlq	120s
aiqlick-agent-docs-prod	aiqlick-agent-docs-prod-dlq	120s

The --attributes shorthand of aws sqs create-queue doesn't accept nested JSON for RedrivePolicy. Use a per-queue JSON file:

{
  "VisibilityTimeout": "120",
  "MessageRetentionPeriod": "345600",
  "RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:eu-north-1:842697652860:aiqlick-inbound-jobs-prod-dlq\",\"maxReceiveCount\":3}"
}

aws sqs create-queue --queue-name aiqlick-inbound-jobs-prod \
  --attributes file:///tmp/attrs-inbound-jobs.json \
  --region eu-north-1 --profile Administrator-842697652860

The two inbound queues additionally need a queue policy allowing EventBridge to SendMessage:

{
  "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"AllowEventBridge\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Action\":\"sqs:SendMessage\",\"Resource\":\"arn:aws:sqs:eu-north-1:842697652860:aiqlick-inbound-jobs-prod\",\"Condition\":{\"StringEquals\":{\"aws:SourceAccount\":\"842697652860\"}}}]}"
}

2. S3 bucket + EventBridge bus

Bucket name pattern: aiqlick-inbound-emails-{env}. Apply: AES256 SSE, lifecycle rule "expire after 30d", EventBridge notifications enabled, and a bucket policy granting ses.amazonaws.com s3:PutObject with an aws:SourceAccount confused-deputy guard.

aws s3api create-bucket --bucket aiqlick-inbound-emails-prod \
  --region eu-north-1 \
  --create-bucket-configuration LocationConstraint=eu-north-1 \
  --profile Administrator-842697652860

aws s3api put-bucket-encryption --bucket aiqlick-inbound-emails-prod ...
aws s3api put-bucket-lifecycle-configuration --bucket aiqlick-inbound-emails-prod ...
aws s3api put-bucket-notification-configuration \
  --bucket aiqlick-inbound-emails-prod \
  --notification-configuration '{"EventBridgeConfiguration":{}}' ...
aws s3api put-bucket-policy --bucket aiqlick-inbound-emails-prod ...

Bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowSESPuts",
    "Effect": "Allow",
    "Principal": {"Service": "ses.amazonaws.com"},
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::aiqlick-inbound-emails-prod/*",
    "Condition": {"StringEquals": {"aws:SourceAccount": "842697652860"}}
  }]
}

Then create the custom EventBridge bus (separate from the default bus — Phase 5.x billing-cache invalidator publishes here):

aws events create-event-bus --name aiqlick-events-prod \
  --region eu-north-1 --profile Administrator-842697652860

3. EventBridge rules + SQS targets

Two rules on the default bus (S3 ObjectCreated events come from there, not from the custom bus), one per prefix:

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": {
    "bucket": {"name": ["aiqlick-inbound-emails-prod"]},
    "object": {"key": [{"prefix": "jobs/"}]}
  }
}

aws events put-rule --name aiqlick-inbound-jobs-rule-prod \
  --event-pattern file:///tmp/eb-rule-jobs-prod.json \
  --state ENABLED --region eu-north-1 --profile Administrator-842697652860

aws events put-targets --rule aiqlick-inbound-jobs-rule-prod \
  --targets 'Id=sqs-jobs-prod,Arn=arn:aws:sqs:eu-north-1:842697652860:aiqlick-inbound-jobs-prod' \
  --region eu-north-1 --profile Administrator-842697652860

Same for cvs/. The inbound queue policy from step 1 lets EventBridge actually deliver.

4. SES receipt rules

Append to the existing active rule set aiqlick-receipt-rules — don't create a new rule set, SES allows only one active set per account.

{
  "Name": "jobs-to-s3-prod",
  "Enabled": true,
  "TlsPolicy": "Optional",
  "Recipients": ["jobs@inbound.aiqlick.com"],
  "Actions": [{
    "S3Action": {
      "BucketName": "aiqlick-inbound-emails-prod",
      "ObjectKeyPrefix": "jobs/"
    }
  }],
  "ScanEnabled": true
}

aws ses create-receipt-rule --rule-set-name aiqlick-receipt-rules \
  --rule file:///tmp/ses-rule-jobs-prod.json \
  --region eu-north-1 --profile Administrator-842697652860

Same for cvs-to-s3-prod matching cv@inbound.aiqlick.com.

tip

SES preserves plus-addressing during delivery — the receipt rule matches the base local-part (jobs@, cv@), but mail sent to jobs+acme-corp@inbound.aiqlick.com keeps the +acme-corp tag in the To: header. The BG parser extracts the tag from there.

5. IAM updates

Historical — these policies no longer exist

The six policies below were deleted in the Phase 7 cleanup of 2026-05-14 along with the legacy EC2 roles. The same grants now live in the inline BackendTaskPolicy (on aiqlick-ecs-backend-task-role-{dev,prod}) and BgTaskPolicy (on aiqlick-ecs-bg-task-role-{dev,prod}). Both inline policies already include the prod + dev queue ARNs, so a new env's queues land in one task-def edit per service. The table below is preserved as the historical record of the Phase 4 cutover.

Six policies need the prod ARNs added alongside the dev ones (the same role serves both envs). Each gets a new default version via aws iam create-policy-version --set-as-default:

Policy (historical)	Adds	Replaced by (post-Phase-7)
aiqlick-backend-mail-policy	aiqlick-mail-prod + DLQ	BackendTaskPolicy on aiqlick-ecs-backend-task-role-prod
aiqlick-backend-events-policy	notifications-prod queues + aiqlick-events-prod PutEvents	same
aiqlick-backend-inbound-policy	inbound-prod queues + aiqlick-inbound-emails-prod	same
aiqlick-ai-inbound-policy	inbound-prod queues + S3	BgTaskPolicy on aiqlick-ecs-bg-task-role-prod
aiqlick-ai-events-policy	aiqlick-events-prod PutEvents	same
aiqlick-ai-bg-events-policy	aiqlick-bg-events-prod + aiqlick-agent-docs-prod	same

IAM caps each policy at 5 versions; if you hit the limit, delete the oldest non-default version first.

6. Secrets Manager — additive keys

Edit aiqlick-backend/production (NOT the env file directly — fetch-secrets.sh regenerates /home/ec2-user/.env from this secret on every deploy):

aws secretsmanager get-secret-value --secret-id aiqlick-backend/production \
  --region eu-north-1 --profile Administrator-842697652860 \
  --query SecretString --output text > /tmp/secret.json
# … merge in the new keys …
aws secretsmanager put-secret-value --secret-id aiqlick-backend/production \
  --secret-string file:///tmp/secret.json \
  --region eu-north-1 --profile Administrator-842697652860

Keys to add (mirror dev shape, swap -dev → -prod):

MAIL_BACKEND=sqs
MAIL_TRANSPORT=smtp           # SMTP via send.one.com until SES production access lands
MAIL_QUEUE_URL=https://sqs.eu-north-1.amazonaws.com/842697652860/aiqlick-mail-prod
SES_FROM_ADDRESS=no-reply@aiqlick.com
NOTIFICATIONS_BACKEND=sqs
NOTIFICATIONS_DELIVERY_QUEUE_URL=…/aiqlick-notifications-delivery-prod
NOTIFICATIONS_FANOUT_QUEUE_URL=…/aiqlick-notifications-fanout-prod
INBOUND_EMAIL_BACKEND=sqs
INBOUND_CV_BACKEND=sqs
INBOUND_EMAIL_QUEUE_URL=…/aiqlick-inbound-jobs-prod
INBOUND_CV_QUEUE_URL=…/aiqlick-inbound-cvs-prod
INBOUND_EMAIL_S3_BUCKET=aiqlick-inbound-emails-prod
INBOUND_EMAIL_MAILBOX_ADDRESS=jobs@inbound.aiqlick.com
INBOUND_CV_MAILBOX_ADDRESS=cv@inbound.aiqlick.com
PUBSUB_BACKEND=inprocess
BILLING_PUBSUB_BACKEND=eventbridge
AWS_EVENT_BUS_NAME=aiqlick-events-prod
BG_EVENTS_QUEUE_URL=…/aiqlick-bg-events-prod
AGENT_DOCS_QUEUE_URL=…/aiqlick-agent-docs-prod
XRAY_ENABLED=true
XRAY_SERVICE_NAME=aiqlick-backend

Keep INBOUND_EMAIL_ENABLED=true and INBOUND_CV_ENABLED=true for the soak window — both paths run in parallel until you do the IMAP cutover (step 11).

7. Container restart (stop + rm + run, not docker restart)

docker restart does not re-read --env-file — the new keys won't be picked up. The deploy template uses stop + remove + run:

ssh prod  # or via SSM
CURRENT_IMAGE=$(docker inspect aiqlick-backend --format "{{.Config.Image}}")
/home/ec2-user/fetch-secrets.sh
aws ecr get-login-password --region eu-north-1 | docker login \
  --username AWS --password-stdin 842697652860.dkr.ecr.eu-north-1.amazonaws.com
docker stop aiqlick-backend && docker rm aiqlick-backend
docker run -d --name aiqlick-backend --restart unless-stopped \
  -p 4001:4001 --env-file /home/ec2-user/.env "$CURRENT_IMAGE"

Same pattern for background-tasks (port 8000). Verify both with curl https://api.aiqlick.com/health and curl https://ai.aiqlick.com/health returning 200.

8. Verify in container logs

The startup banner is the receipt:

[SqsMailConsumer] Mail transport=smtp host=send.one.com:465 from=no-reply@aiqlick.com
[SqsMailConsumer] Consumer started (queue=…aiqlick-mail-prod)
[SqsNotificationConsumer] Consumer started (queue=…aiqlick-notifications-delivery-prod)
SQS inbound email consumer started (Phase 2)
SQS inbound CV consumer started (Phase 2)
[SqsInboundEmail.jobs] Consumer started (queue=…aiqlick-inbound-jobs-prod)
[SqsInboundEmail.cv]   Consumer started (queue=…aiqlick-inbound-cvs-prod)
SqsBillingCacheInvalidator: Consumer started (queue=…aiqlick-bg-events-prod)
Agent docs SQS consumer started (Phase 5.1)

If any consumer reports AccessDenied: User: arn:aws:sts::…:assumed-role/<role>… is not authorized to perform: sqs:ReceiveMessage, the IAM update from step 5 didn't land — re-check policy versions and docker restart the container (this time a plain restart works because the IAM role attaches at the IMDS level, not via env-file).

Company.slug backfill

The plus-addressing recipient (jobs+acme-corp@inbound.aiqlick.com) resolves to a company by exact match on Company.slug. Companies created before slugs were enforced may have slug = NULL or slug = '' and won't resolve.

Pre-flight count + dump first:

PGPASSWORD=… psql -h localhost -p 15434 -U aiqlick -d aiqlick_db -At \
  -c 'SELECT count(*) FROM "Company" WHERE slug IS NULL OR slug = '\'''\'';'

PGPASSWORD=… psql -h localhost -p 15434 -U aiqlick -d aiqlick_db -A \
  -c 'SELECT id, "companyName", slug FROM "Company" WHERE slug IS NULL OR slug = '\'''\'';' \
  > /tmp/slugs-before.csv

Backfill using the same regex as normalize_company_slug() in background-tasks/app/services/inbound_email/normalization.py:

UPDATE "Company"
SET slug = trim(both '-' from regexp_replace(lower("companyName"), '[^a-z0-9]+', '-', 'g'))
WHERE slug IS NULL OR slug = ''
RETURNING id, "companyName", slug;

Use SSM port-forward for the SSH-less RDS connection (see SSM Port-Forwarding).

Prod ran this on 2026-05-02 and updated 4 rows: Pineappeal Agency, CeeQ Sweden AB, Creative Art Design, Pactoria.

Sentinel e2e on prod

To verify a freshly-cutover env without touching real customer pipelines:

1. Create a sentinel test company on prod via createCompany mutation as super-admin: companyName: "AIQLick QA", slug: "aiqlick-qa", size: "1-10", status: "ACTIVE". Save the returned id — you'll need it for cleanup.
2. Run the inbound matrix with that slug only:

ID	Recipient	Expected
Plain text JD	jobs+aiqlick-qa@inbound.aiqlick.com	JOB_CREATED
PDF JD attachment	same	JOB_CREATED
Invalid slug	jobs+definitely-not-a-real-slug@…	FAILED + company_not_found
No tag	jobs@inbound.aiqlick.com	IGNORED + missing_company_tag
Real CV PDF	cv+aiqlick-qa@…	CV_CREATED (User+Talent+CV chain)
No attachment	cv+aiqlick-qa@… (text only)	IGNORED + no_cv_attachment

3. Run outbound with rzeraat.tur@gmail.com and rezdev97+prod-<ts>@gmail.com recipients only — never real customer emails:

Trigger	Expected
signUp	Email.status=SENT
requestPasswordReset	Email.status=SENT
createCollaborationInvitation	Email.status=SENT
createSupportTicket + sendSupportReply	both SENT

4. Verify all queue depths + DLQ depths are 0 throughout.
5. Tear down — delete the sentinel Company, the test Users, their Talents, CVs, Jobs, and InboundEmail rows. The cascade order matters because of foreign-key constraints; the canonical sequence is recorded inline in the cutover commit log. Snapshot \copy the soon-to-be-deleted rows to /tmp/sentinel-*.csv first.

A super-admin token on prod can be minted directly from JWT_SECRET (HS256) without a password, using the super-admin's user ID — useful when the prod password isn't to hand:

import json, base64, hmac, hashlib, time
JWT_SECRET = "…"  # from aiqlick-backend/production secret
USER_ID = "<rzeraat.tur user id>"
EMAIL = "rzeraat.tur@gmail.com"
now = int(time.time())
header = {"alg": "HS256", "typ": "JWT"}
payload = {"sub": USER_ID, "email": EMAIL, "iat": now, "exp": now + 3600}
def b64(s): return base64.urlsafe_b64encode(s).decode().rstrip("=")
h = b64(json.dumps(header, separators=(",", ":")).encode())
p = b64(json.dumps(payload, separators=(",", ":")).encode())
sig = hmac.new(JWT_SECRET.encode(), f"{h}.{p}".encode(), hashlib.sha256).digest()
print(f"{h}.{p}.{b64(sig)}")

IMAP cutover

Once the SES path has soaked (~24h on prod is plenty), turn off the legacy IMAP poller:

1. Update the secret: set INBOUND_EMAIL_ENABLED=false and INBOUND_CV_ENABLED=false.
2. (Optional but recommended) strip the now-unused EMAIL_IMAP_*, CV_IMAP_*, INBOUND_EMAIL_RETRY_*, INBOUND_CV_RETRY_*, and any leftover REDIS_* keys.
3. Stop + remove + run the BG container so --env-file is re-read.
4. Confirm BG logs show:

   Inbound email worker disabled (INBOUND_EMAIL_ENABLED=false)
   Inbound CV worker disabled (INBOUND_CV_ENABLED=false)
   SQS inbound email consumer started (Phase 2)
   SQS inbound CV consumer started (Phase 2)

   The IMAP-side Found N new messages log lines should stop entirely.

The IMAP mailboxes themselves (jobs@aiqlick.com, cv@aiqlick.com on imap.one.com) keep existing — they're decoupled from the platform now. Retire them at one.com whenever you're ready.

Rollback

Each stage is independently reversible:

Stage	Rollback
1–4 (AWS resources)	Delete the queues / bucket / rules; idempotent reverts
5 (IAM)	Roll the policy version back via aws iam set-default-policy-version --version-id <prior>
6 (Secrets)	put-secret-value writes a new version; update-secret-version-stage flips AWSCURRENT back
7 (Container)	CI/CD has a :previous image tag; manual rollback is docker stop && docker rm && docker run … :previous
8 (Slug backfill)	Restore from /tmp/slugs-before.csv (this is why the snapshot exists — the regex is technically lossless but the raw nulls are not recoverable)
11 (IMAP cutover)	Set INBOUND_EMAIL_ENABLED=true, restart BG. The IMAP creds are still on imap.one.com if you didn't retire them.

Related

• AWS-Native Migration — full architectural context
• Inbound Email Processing — user-facing routing semantics
• Mail & Email — outbound + inbound code internals
• SSM Port-Forwarding — RDS access for the slug backfill
• Prisma 7 migration — Docker build gotcha that bit during this cutover