SkillUp: one door, any replica, every front end

SkillUp: one door, any replica, every front end

A voice coach that works in a demo is a model and a microphone. A voice coach that works for thousands of people, through deploys, restarts and a front end built by another company, is a platform. We designed and built that platform for SkillUp Coalition. The AI engine that does the coaching was built by a specialist partner, Medius Cortex. This is the story of everything around it: how a conversation gets in, how it stays whole when the machine under it changes, and how a front end team ships without ever touching the backend.

The challenge

SkillUp serves workers without four-year degrees. Its coach talks. A person speaks, the coach listens, and a dashboard of interests, training plans and goals builds itself while the conversation happens.

That shape sets four hard requirements for the platform.

The engine must be private. It holds conversation memory and personal data. Nothing on the internet should be able to reach it, not even by mistake.

The platform must scale sideways. More people means more copies of each service. A conversation cannot belong to one copy, because copies come and go: a deploy replaces them, autoscaling adds and removes them, and a failed health check kills one.

A reply must survive the loss of a connection. A voice reply streams token by token. If the connection that carries it breaks halfway, the person must not hear half a sentence and then silence.

The front end must be someone else's. SkillUp's production front end is built by a partner team in its own repository. That team needs a contract it can build against on day one, test without our servers, and upgrade on its own schedule.

Why it matters

In a chat window, a dropped token is a typo. In a voice conversation, it is a stutter, a pause, or a sentence that never ends. The person on the other side does not see a retry banner. They hear the product fail.

The same is true of scale. A demo runs on one machine with one user. The first real deploy under live traffic is where a design that pins conversations to machines shows its cost: every session on the replaced machine drops at once.

And a front end that needs the backend team for every change is a bottleneck with a salary. The value of a clean contract is measured in the number of meetings it removes.

The technical approach

One public door

The platform has exactly one public entry point: the API service. Every request from a browser goes through Cloudflare to a public load balancer, and that load balancer accepts traffic from Cloudflare's address ranges only. Behind it, the API does the work that belongs at the edge. It verifies the person's sign-in token, applies rate limits, creates voice rooms, handles WebRTC signalling and runs the voice pipeline.

The AI engine has no load balancer at all. It exists on the private network under a service-mesh name, and its security group accepts traffic from one group only. The API is the only member of that group. The engine is not hidden behind a firewall rule that someone could loosen. It has no public address to reach.

Inside that boundary the API tells the engine who the user is in a trusted header. We chose that on purpose, and we wrote down why. A shared secret between the two services would protect against one threat only: another service on the mesh calling the engine. No such service exists. The design record names the trigger for change instead: before any second service joins the mesh, the key goes in. That is a decision a reviewer can check, not a gap someone has to discover.

The division of work follows the same line. The API owns the edge: identity, rooms, signalling and limits. The engine owns the conversation: sessions, memory and the database. The two services scale on their own, with different sizes and different limits, because they do different work.

A reply crosses two hops

A reply travels twice. The engine streams tokens to the API. The API turns them into speech and captions and sends them to the browser.

The browser side is one WebRTC connection through Daily. Audio travels on it, and so do the control messages: typed text, dashboard updates and session signals. We put control on the same connection as the audio for one reason. A control message can never arrive on a connection that the audio has already left.

The engine side is gRPC, one bidirectional stream per conversation, called Converse. The API opens the stream and puts the session ID and user ID in the call metadata, not in the frames. That lets the engine route and authorise the stream before it reads a single frame. A draining server, a missing session, an unknown user or someone else's session each ends the call with its own status code, at once.

Each direction has one envelope. The API sends Subscribe or SubmitTurn. The engine sends events. New kinds of event fit inside the envelope without a change to the contract, so the two services can be released on their own.

Two small rules carry a lot of weight. The API mints the ID of every turn, so it can sort streamed tokens by turn from the first token, and the engine rejects a turn without one. And every frame the engine sends carries a cursor: the ID of that event in a Redis stream. The cursor is what makes the next section work.

Any replica can serve any turn

We ruled out sticky sessions early, and the design record says why. A pinned conversation dies with its machine. And a load-balancer cookie cannot follow the internal hop from the API to the engine. So every piece of state that crosses a request lives in Redis or Postgres, and any replica can serve any turn.

The key structure is one Redis stream per session. As the engine produces a reply, it appends one entry per token to that stream, with a one-hour time to live that each write refreshes. The stream is capped at about 4,096 entries. The ID of each entry is the cursor that travels in the gRPC frame.

When the API's connection to an engine replica breaks mid-reply, the API reconnects through the service mesh. It will often land on a different replica. It sends the last cursor it handled in its Subscribe frame. The new replica reads the stream from just after that cursor, replays what the API missed, and then switches to live reads from the tip. Replay never drops an entry. Live delivery may shed load under pressure, but the missed part of a reply always arrives in full.

The size of that cap was a design decision, not a default. The first version kept 1,000 entries. A reply can be up to 1,024 tokens long, so a long reply trimmed its own first tokens while it was still streaming. We raised the cap to 4,096 so the window always holds at least one whole reply. We also looked at replay by whole message and rejected it: a message-level log cannot rebuild a reply that was only partly streamed.

The edge cases have names. If a cursor is so old that it has been trimmed away, the engine sends an explicit resync frame and replays the whole surviving window. If the session itself has expired, the caller gets a clear "gone" reply. A reconnect never ends in silence.

The same pattern runs through the rest of the platform. The first write binds a session to its user, on whichever replica gets there first, and every later request checks that binding. An epoch value stops a slow teardown on one replica from deleting a newer session that another replica just built. On the API side, Redis records which task owns each live voice room, so a reconnect, a teardown or a transcript request can land on any task and still find the right one. A lock covers the handoff when a voice room nears its expiry, and a short-lived ledger holds any turn that could not be delivered, so it can be replayed after the handoff.

Redis holds what may expire. Work that must happen does not live there. Memory extraction and profile updates after each turn go into a Postgres table, and workers claim rows with FOR UPDATE SKIP LOCKED. That gives retry, dead-lettering and crash recovery with no extra infrastructure. A Redis queue or a cloud queue was one more moving part with its own way to fail.

Deploys that nobody hears

Scaling sideways only helps if a replica can leave cleanly. On shutdown the engine stops taking new streams and tells callers to reconnect elsewhere. It gives open streams 12 seconds, then gives turns in flight 8 more, all inside a 24-second overall deadline. The container platform allows 30 seconds before it forces a stop, and the budget was worked out to fit inside it.

Load shedding follows the same care. When the engine is saturated, it refuses new sessions with a Retry-After header. It never reports itself unhealthy to shed load, because an unhealthy health check makes the platform replace the task, and that drops every live conversation on it. A two-replica test harness, with a shared Redis behind round-robin routing, proves the cross-replica paths before any environment changes transport.

A front end that needs nothing from us

The contract between the platform and any front end is a package: @skilluporg/conversation-kit. It is a versioned React library, published to a private registry, not a folder that teams copy.

The kit exports providers for the session, voice, memory and profile, about 25 hooks, a set of conversation and voice components, and mock services. The mocks matter most for independence. A front end team can build and test a whole conversation screen with no backend running at all.

The kit also owns the hard parts that every front end would otherwise rebuild badly. It owns session continuity. It retries the WebRTC offer with jittered backoff, and only one retry runs at a time. It re-offers before a voice room expires. It restores the conversation history after a reload, and it buffers messages sent during a handoff. The host app passes in one callback that returns a sign-in token, and the kit does the rest. The in-house coach app carries a comment that says it plainly: the kit is the only retry authority, so never add one in the app.

The kit sends its own version with every offer, so the platform always knows which client is calling. Release rules keep the contract honest. A pull request that changes a package without a new version number fails. Packages publish when a release is promoted to staging, after the backend they depend on, so a front end can never install a kit that expects a server that does not exist yet.

Key decisions

No stickiness, anywhere. Every conversation can move between replicas at any moment, on both services. That single rule forced the Redis stream design, the cursor in every frame and the ownership records for voice rooms. It is also why a deploy, a scale-in or a crashed task is a non-event for the person talking.

Size the replay window to the longest reply. The window is a product rule, not a cache setting. Its job is to hold a whole reply, so its size comes from the longest reply the engine can produce.

Network isolation as the internal trust, with a written trigger. The engine is reachable by one caller. We did not add a secret that protects against a caller that does not exist. We wrote down the moment when that changes.

Durable work in Postgres, disposable state in Redis. Streams and ownership records can expire. Memory updates cannot be lost. Each kind of state lives in the store whose failure behaviour fits it.

Ship the front-end contract as a package. A published, versioned kit with mocks lets a partner team build on its own schedule. The cost is a strict release order, backend before kit, on every promotion. We accepted it.

Results

The platform runs the API and the engine as separate services on AWS, in private subnets, each scaling from its own floor to ten tasks. The engine has no public route. Any replica of either service can serve any turn of any conversation, and a two-replica harness gates the cross-replica paths in every environment.

Every architectural choice above is recorded. The repository holds more than 50 decision records, each with its reason and its rejected alternatives. The same container image is built once and promoted unchanged from dev to staging to production, with separate AWS accounts for each. Eighty-five alarms watch the system, including one on the time to the first spoken token.

The partner's production front end builds on the kit from the registry. The in-house coach app, a browser extension and a web client use the same package from the monorepo. None of them contains its own retry logic.

Lessons

The model is one service. The quality of a voice product depends as much on what happens between the model and the ear as on the model itself. Streams, cursors, drains and retries are where the conversation is won or lost.

State that crosses a request needs a home that outlives the machine. Once that rule holds, scaling, deploys and failures stop being special cases.

Write the reason down, and write the trigger down too. A decision record that says when it should be revisited turns a shortcut into a plan.

A contract that ships with mocks is a contract people use. The fastest way to make a front end independent is to let it run without the backend at all.