Engineering Blog · Post #83

Pipeline Management, End-to-End: From Snapshot to Stream

Most underwriting pipelines are screens that happen to show data. InsightUW's pipeline is a stream — one that keeps every workbench, forecast, and personal dashboard in sync with what's true right now, without a single F5.

The Problem

A pipeline answers five questions a day:

What's moving, right now, across the whole book?
What's mine?
Where is revenue likely to land this week, this quarter?
Which brokers and regions are producing — and which are slipping?
What did we win, what did we lose, why?

In most workstations, each question is a different screen, populated by a different query, frozen at a different point in time. The manager looks at a weekly pipeline export. The UW looks at a stale queue. The forecaster looks at last Friday's numbers. Nobody is looking at the same system.

The InsightUW Approach

One streaming pipeline + a small set of derived views. Every screen reads from the same subsystem; the only difference is what it filters and rolls up.

graph TB subgraph Source["Source of truth"] S["Submission Queue + Insured + Policy + Broker"] end subgraph Stream["Streaming layer"] SSE["SSE workbench stream hashed-diff emit 25s heartbeat"] end subgraph Views["Derived views"] V1["Real-time workbench filters + saved views"] V2["Weekly forecast w/ overrides"] V3["Broker book broker × region × product"] V4["My Accounts / Lines / Renewals / Prior"] end S --> SSE SSE --> V1 S --> V2 S --> V3 S --> V4 V1 -.same URL-synced filters.-> V2 V1 -.same URL-synced filters.-> V3 V1 -.same URL-synced filters.-> V4

1. Real-Time Workbench

One Angular page — UwWorkbenchComponent — subscribes to /api/uw_workflow/workbench/underwriting/stream. The server holds a single async loop that:

Queries submissions + renewals + referrals,
Enriches with BOR status, missing-info counts, and stage progress,
Hashes the payload, and
Emits an SSE frame only when the hash changes — plus a heartbeat every 25s.

The browser renders a LIVE · Updated HH:MM:SS chip, auto-reconnects if the connection drops, falls back to a non-streaming GET if the EventSource errors for 10 seconds. No polling. No stale pipeline. A teammate's assignment change on a different machine is visible to every open workbench within 5 seconds.

2. Filters + Saved Views

The pipeline is multi-dimensional — LOB, broker, team, stage, SLA bucket, urgency, premium band, rush flag. Every filter binds to a URL query param via WorkstationFiltersService. Share-a-link works. Browser-back works. Saved View persists filter + sort + column visibility per UW so "my D&O rush queue" is one click, not six.

3. Weekly Forecasts (with Overrides)

Weekly Forecast stores, per (UW, week, LOB), a forecasted bound count and premium — computed from open submissions × win probability + known renewal subs. UWs override any row ("I'm 90% sure Solaris closes this Friday, not next"). Override preservation is the hard part: if the UW overrides Friday to $800K and a submission later moves stage, the override survives until explicitly cleared. uw weekly forecast service runs a nightly recompute that never clobbers overrides; a manager-only bulk-reset exists for quarter-end hygiene.

4. Broker Book — Broker × Region × Product

A matrix view: rows = broker, columns = (region × product), cells = YTD GWP / submissions / hit ratio / YoY delta. Computed in uw broker book service from a single SQL with conditional aggregates; cell-level drilldown lands on the filtered workbench for that broker/region/product.

5. Personal Dashboards — My Accounts, My Lines, My Renewals, Prior

My Accounts — cards for every insured where Insured.assigned_uw = me, ranked by composite urgency score (SLA × renewal window × BOR × missing-info).
My Lines on an Account — when multiple UWs share an insured, this view respects co-underwriter ownership: my policies + my in-flight submissions only, plus a banner showing which other UWs carry other lines.
My Renewals (Upcoming) — Policy (expiring, assigned to me) joined to Submission Queue (the renewal submission, via the prior policy id FK) with 9 action buckets (Overdue, No submission, Awaiting broker, In progress, Awaiting quote, Quoted, Negotiating, Won, Lost).
Prior Renewals — won/lost history with retention %, rate delta on wins, loss-reason breakdown.

All four share a ?viewer=<uw_id> role-gated manager switch — managers see any UW's dashboard at the same URL.

The Data Plumbing That Makes This Cheap

Insured.assigned_uw — single source of truth for account ownership. One-shot backfill derives it from the latest assigned submission or active policy.
Submission Queue.prior_policy_id — FK that makes policy → renewal-submission joins index-only.
Bulk queries, no N+1 — my-accounts loads the entire list with 4 aggregate queries regardless of card count.
Action buckets derived server-side — from stage + status + days-to-expiry. UI is presentational.
Hashed SSE diffs — a quiet minute costs nothing; a busy minute sends one frame, not a query-per-client.

What a UW Actually Sees

Every dot fills in live. Every badge pushes. Every filter is in the URL.

A Day in the Life

08:55 — UW lands on /uw/home. Tiles show 14 actions on her plate.
09:02 — Jumps to /uw/my-accounts. Three cards in "Needs attention." Clicks the top one.
09:10 — Workspace opens. Checklist panel says 4 of 11 done. NDA banner green.
09:30 — Broker replies by email. Missing-info count on the workbench drops from 3 to 1 — she sees it from the sidebar badge without switching.
10:00 — Checks /uw/my-renewals. Two rows in the "No submission" bucket. Multi-selects, clicks Generate renewal submissions — both get created.
11:30 — Opens /uw/weekly-forecast. Overrides Friday to +$800K, commits. The forecast row survives the 2am recompute.
14:15 — Manager pings: "Send me the broker book for Marsh D&O YTD." UW hits /uw/broker-book?broker=marsh&lob=D%26O&ytd=true, clicks Print → PDF in the file in 4 seconds.
16:40 — New rush lands. Workbench sorts it to the top without reload.

Why It Matters

No F5 pipeline. Streaming makes the workbench the system of record, not a screenshot of one.
Same primitives, many questions. Workbench, my-accounts, forecasts, broker book, renewals — all join a handful of tables through URL-synced filters.
Overrides are respected. Forecasts stay intentional.
Personal dashboards cut through the noise. "My plate" is a real URL, not a column you squint at.
Co-underwriters are first-class. Accounts are not single-UW when they're not.

Capability Deep-Dives

Want a pipeline you actually trust at 10:42:17? Request a demo.