Idempotency Keys and policy admin system Envelopes: How InsightUW Pushes Manuscripts to the Policy Admin System Without Creating Phantom Records

From "the network blipped, the worker retried, now policy admin system has three duplicate cancellation transactions on the same policy" to "the same idempotency key on every retry, the PAS de-dupes server-side, and the provider record id is written back the moment the first call succeeds" — through one column on the message table, two provider adapters that wrap or path-map the body, and a generic outbox that mirrors the email-send pattern for the same operational reasons.

The Problem

A UW finalizes a mid-term endorsement on Acme Construction's policy. The system queues a push to the carrier's policy admin system (policy admin system, in this case). The worker dispatches; a 502 comes back from the policy admin system's API gateway during what was actually a successful write. The worker retries. policy admin system now has two mid-term endorsement records on the policy. Three weeks later, finance reconciliation flags the duplicate; the carrier issues a cancellation transaction to retract one. The PAS audit log carries six entries for what was supposed to be one event.

Worse: the Sarah-the-UW story. She finalized an amendment, got a "queued" notification, came back next morning, saw "still queued" — clicked "push now," same row went out twice. Same problem.

The usual fixes don't fix:

• Catch the 502 and don't retry. Now real network blips become user-visible failures. UWs hit retry manually, same problem, fewer rows.
• Trust the PAS to dedupe. Some do; most don't. policy admin system dedupes if you send Idempotency-Key; policy admin system doesn't honor it; older SOAP-based PASs check external id if you pass it. Inconsistent.
• A "stop dispatching while one's in flight" lock. Holds queue throughput hostage to the slowest PAS in the fleet.
• Hand-write a retry guard per provider. Eight providers, eight subtly-different rules, the ninth gets it wrong on a Friday.

The root cause: the system is the idempotent party (it knows "this is the same logical event"), but it's leaving the deduplication to the PAS — which knows what it knows. Without an explicit idempotency contract on the wire, retry loops become duplicate-record loops.

The InsightUW Approach

Generate a deterministic idempotency key once, send it on every retry, and let the PAS de-dupe server-side. Per-provider adapters handle envelope quirks; the dispatcher stays provider-agnostic.

The idempotency key

Four inputs, all stable: which connector it's going to, which entity is being pushed, what kind of push it is, and the source's version (so a v1→v2 amendment gets a fresh key while a v1 retry uses the same one).

The key sits on Pas Message with a unique constraint. queue push checks for an existing row with that key first — re-calling for the same version returns the original message rather than spawning a duplicate. Worker retries reuse the same row, send the same key, the PAS dedupes.

Adapters: thin wrappers on the generic case

Three adapters today; a fourth costs ~150 LOC.

generic rest — covers any PAS that takes the connector's field mapping JSON at face value. POSTs {base_url}/{message_type}, sends Idempotency-Key header, handles api_key / oauth2 / basic auth from connector.auth_config. Has a dry-run mode (env var PAS_DRY_RUN=1) so dev / CI can exercise the full pipeline without a live PAS.

dragon — policy admin system namespaces all writes under /api/v1/transactions and expects a wrapped envelope:
Returns the record id under data.recordId, which the adapter pulls out into AdapterResult.provider_record_id.

majesco — policy admin system's REST gateway path-maps the message type:
Returns the record id under result.id.

Each adapter implements one method:
AdapterResult carries ok, provider record id, response, error, transient. The dispatcher reads those four fields and updates the outbox accordingly. Adding a new provider = subclass GenericRestAdapter, override _endpoint and extract record id. ~150 LOC.

Field mapping: config, not code

Policy System Connector.field_mapping is a JSON dot-path map:
apply field mapping walks the dot-paths and produces the target dict. Adapters consume the post-mapping body verbatim. Different PAS, different mapping, same dispatcher.

Write-back: provider_record_id → Policy

The first successful push writes provider record id back to two places:
1. Policy.policy_system_number — the column the existing Quick Links deep-links read. Now the "Open in Policy Admin" link in the workstation actually opens the policy admin system record.
2. Policy Manuscript Amendment.pas_provider_record_id — for amendment pushes, so the editor can show "Provider id: AMEND-DRG-2026-094218" inline.

This is the moment InsightUW becomes a real PAS-of-record adjacent system instead of a parallel ledger.

Worker: same shape as email outbox

Same retry curve as Email Send Service.process_outbox. Same dead-letter terminal state. One worker pattern, two channels — the operational story (oncall debugging, retry tuning, DLQ inspection) is uniform.

Worked Example: Acme Construction Cancellation Endorsement

Sarah is processing a cancellation endorsement for Acme Construction's $50M Excess Casualty policy. The carrier is policy admin system.

Step 1 — Author the amendment

She opens /uw/policies/POL-2026-EX-0188, clicks Amendments, + New amendment. Picks cancellation, effective date 2026-05-15, reason "Insured request — sold operations to Buyer LLC effective 5/15." Editor opens with the seeded body, she edits the cancellation language, clicks Finalize & push.

Step 2 — Finalize

Policy Amendment Service.finalize_amendment:
1. Renders body markdown → HTML → DOCX bytes.
2. Stores as Submission Document (category=policy amendment).
3. Sets amendment_status='finalized', is_locked=True, attached_document_guid=<doc>.
4. Auto-enqueues PAS push via queue amendment push.

Inside queue push:
- Resolves connector: policy.policy_system_source = 'dragon' → Policy System Connector row for policy admin system-US.
- Builds source context: build amendment context returns {policy: {...}, insured: {...}, broker: {...}, amendment: {transaction_type: 'cancellation', transaction_effective_date: '2026-05-15', ...}}.
- Applies field mapping: apply field mapping → post-mapped dict.
- Computes idempotency key: sha256 → a4b8c9d2…
- Pas Message row with idempotency_key=a4b8c9d2…, payload_json=<post-mapped>, send_status=queued.
- Pas Outbox row with send_status=pending, next_try_at=now.

UI shows the amendment status panel: "queued · policy admin system · idempotency a4b8c9d2…."

Step 3 — Worker dispatches (first try, network blip)

Cron triggers process outbox. Sweeps the row. Calls DragonAdapter.dispatch:
1. Wraps payload: {"transactionType": "amendment_push", "idempotencyKey": "a4b8c9d2…", "data": {...}}
2. POSTs to https://dragon-us.api.example/api/v1/transactions with Idempotency-Key: a4b8c9d2… header (and the wrapped envelope's idempotencyKey field, belt-and-braces — policy admin system honors both).
3. Network: 502 from the gateway. Adapter returns AdapterResult(ok=False, transient=True, error="HTTP 502").
4. Dispatcher: attempt_count=1, next_try_at=now + 30s, send_status=pending. Stays queued.

Sarah's UI polls /refresh-status every 4 seconds (capability #8 polling pattern). She sees: "queued · 1 attempt · last error HTTP 502 · retry in 30s." She closes the editor, goes to lunch.

Step 4 — Worker retries (success)

30 seconds later, cron sweeps again. Same row, same idempotency key, same payload. POSTs to policy admin system. This time the gateway is fine; policy admin system receives the request. policy admin system checks its idempotency cache — sees a4b8c9d2… matches a request it received and persisted 30 seconds ago (the request that did succeed despite the 502 response). Returns the original record:
Adapter: AdapterResult(ok=True, provider_record_id="AMEND-DRG-2026-094218").

Dispatcher:
- outbox.send_status='sent', msg.send_status='sent', msg.sent_at=now, msg.provider_record_id="AMEND-DRG-2026-094218".
- write back policy record id: writes Policy.policy_system_number = "AMEND-DRG-2026-094218" (or skips if already set).
- update amendment on send: writes Policy Manuscript Amendment.pas_provider_record_id and pas pushed at; sets amendment_status='sent'.
- _audit: writes pas message audit row "sent".

Net result on the policy admin system side: one record, not two. The "duplicate retry" never created a phantom because the idempotency key short-circuited the second write.

Step 5 — Sarah sees the result

She comes back from lunch, opens the amendment. Status panel:

sent · policy admin system · sent_at 2026-05-12 11:23 · provider id AMEND-DRG-2026-094218 · idempotency a4b8c9d2…

She clicks the "Open in Policy Admin" Quick Link on the policy detail. The link template {{ system.dragon_base_url }}/policy/{{ submission.policy_system_number }} resolves with the newly-written record id; policy admin system opens the right page.

Step 6 — A week later, the field-mapping admin notices a typo

The connector's field mapping had "policy.premium": "gross_premium" but the policy admin system's API actually wants "gross_premium_usd". The amendment push went through (with the wrong field name silently dropped on the policy admin system's side — policy admin system ignored the unknown field). The admin updates the mapping. Future pushes use the new mapping. The original push isn't re-sent — idempotency key would dedupe even if it were.

The right fix: re-version the amendment (which writes a new idempotency key) and push it as a correction. The PAS sync admin page lets the admin manually trigger the re-push with Push now on the new message row.

What's Deferred (Phase 2)

A few things that came up in the design conversation and were scoped out:

• Inbound reconciliation. "Pull policy state from policy admin system every morning, compare to Policy, flag drift." Different problem, different module. Outbound-only for v1.
• Live FX feed for the payload. Quote.fx_rate_to_usd_at_issue is pinned (capability #13); the payload uses the pinned rate. Live rate at push time would change PAS-side economics relative to what the broker saw on the quote letter — that's a regression, not a feature.
• Per-provider rate-card mapping. Some PASs want premium in cents; some in dollars; some as decimal. Currently apply field mapping just dot-path copies. A "value transform" layer ({{ premium | to_int_cents }}) is a quarter's work and not blocking the current customer set.
• Webhook ingestion of provider status updates. Today the dispatcher gets a synchronous response. If the PAS goes async ("we received it, will process") the worker has no callback channel. Add when a real customer needs it.

What This Means for Underwriters

1. Idempotency is not the PAS's problem. Generate a deterministic key, send it on every retry, sleep at night.
2. Provider quirks live in adapters. the policy admin system's wrapper envelope, policy admin system's path mapping — both isolated. The dispatcher stays provider-agnostic.
3. Field mapping is config, not code. Admins author Policy System Connector.field_mapping JSON. New connector onboarding doesn't require a code release.
4. Dry-run mode keeps dev / CI honest. PAS_DRY_RUN=1 exercises the full queue → message → outbox → adapter → write-back path without a live PAS, recording exactly what would have been posted.
5. Write-back closes the loop. First successful push writes provider record id to Policy.policy_system_number. Quick Links work. Audit trail names the PAS record.
6. One worker pattern, two channels. The PAS outbox mirrors the email outbox: same retry curve, same DLQ semantics, same admin queue UI. Oncall doesn't have to learn a second system.
7. Failure is graceful. No live connector? dead letter immediately with "no connector configured" — the row exists, audit captures it, an admin can configure and re-queue. Better than silent drops.
8. Manual override always available. Sarah's editor has a "Push now" button that bypasses next try at. Used when an earlier failure is blocking and ops want to retry immediately without waiting on the worker tick.

What's Next

Next: BMD Pegged, CAD Pinned: How InsightUW Quotes in Three Currencies Without Forking the Rating Engine

Want to see how a single idempotency contract turns retry storms into noise instead of duplicates? Request a demo.