Add prefex aware routing proposal #602
Conversation
|
Skipping CI for Draft Pull Request. |
k8s-ci-robot
added
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: liu-cong The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
k8s-ci-robot
added
the
size/L
|
|
||
| 1. **Prefix affinity consistent hashing** | ||
|
|
||
| This goes a step beyond the session affinity by using a prefix aware hash function to route requests with similar prefixes to the same or similar servers. A naive hash function can be just taking the hash of the first N characters/tokens of the request, and therefore all requests with the same first N characters/tokens will be routed to the same server. The [vLLM production stack](https://github.com/vllm-project/production-stack/issues/59) is exploring this strategy using simhash, and preliminary experiments showed mixed results. KubeAI uses a simple strategy to only hash request prefix up to a configurable `prefixCharLength`. Its effectiveness is likely highly dependent on the input length distribution. |
There was a problem hiding this comment.
is that a moving window of up to prefixCharLength? or does it always has exactly prefixCharLength characters?
|
|
||
| Pros: | ||
|
|
||
| * Easy to explain (compared to hashing) and likely more effective than hashing strategy. |
There was a problem hiding this comment.
you mean "than consistent hashing strategy"?
| 1. Prefix affinity needs to be aware of the server load, otherwise we will create hot spots. We can use queue length and k-v cache utilization to understand the server load. This is similar to the [queue depth threshold](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/2a615e981228aa6ffc2a89219c986ac863dde776/pkg/epp/scheduling/scheduler.go#L40) for LoRA affinity. | ||
|
|
||
|
|
||
| ## Proposal |
There was a problem hiding this comment.
+1 to start with this approach since it seems relatively simple to implement, but also in theory should be more resilient than the other two options.
|
Cong's PoC is in main...liu-cong:llm-instance-gateway:prefix-poc (or at least, a version of it is) for those interested. |
|
|
||
| ## Design Options | ||
|
|
||
| 1. **Session affinity** |
There was a problem hiding this comment.
Consider switching options to Header 3, its easy to have the options blend together as is, and since they have sub-bullets using 1. resets the count and they all ahve the value of 1.
| 1. **Session affinity** | |
| ### **Session affinity** |
This proposal was initially discussed in #498