Complete the InferencePool documentation #673

nicolexin · 2025-04-09T21:17:07Z

Added section for inference pool configuration and example.
Have a subsection for overlap with Service
Added section for replacing an inference pool / base model
Added extensionReference to the inference pool spec

k8s-ci-robot · 2025-04-09T21:17:17Z

Hi @nicolexin. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

netlify · 2025-04-09T21:17:24Z

✅ Deploy Preview for gateway-api-inference-extension ready!

Name	Link
🔨 Latest commit	`4fdfadf`
🔍 Latest deploy log	https://app.netlify.com/sites/gateway-api-inference-extension/deploys/6802de2593a71f0008779ab5
😎 Deploy Preview	https://deploy-preview-673--gateway-api-inference-extension.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

site-src/api-types/inferencepool.md

site-src/reference/spec.md

nirrozenbaum · 2025-04-10T03:36:44Z

/ok-to-test

kfswain · 2025-04-10T22:56:05Z

/approve

This LGTM, but will let the previous reviewers give the final stamp. Thanks!

robscott

Thanks @nicolexin, this is great!

site-src/api-types/inferencepool.md

Co-authored-by: Rob Scott <rob.scott87@gmail.com>

elevran · 2025-04-16T13:48:12Z

site-src/api-types/inferencepool.md


-It is expected for the InferencePool to:
+An InferencePool is expected to be bundled with an [Endpoint Picker](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/pkg/epp) extension. This extension is responsible for tracking key metrics on each model server (i.e. the KV-cache utilization, queue length of pending requests, active LoRA adapters, etc.) and routing incoming inference requests to the optimal model server replica based on these metrics.


Can an EPP be associated with multiple pools?
Can a Pod belong to more than one pool?

Yes EPP is associated with the inferencepool, so it is associated with multiple pods covered by this inference pool. You can have one pod belong to multiple pools.

thanks @nicolexin.
I wanted to clarify that a single EPP can be associated with multiple InferencePools, not Pods.
Is the mental model for Pool and EPP supporting that and what changes would need to be done in the EPP to support dispatching to the right Pool (e.g., based on the model name present in the request body)?

Ah sorry about misreading your first question. Yes a single EPP can be associated with multiple InferencePools.

The mental model depends on how you define your gateway, for example:

You can configure two separate gateways with different frontend IP, each backed by a different inference pool. Both of the pool sharing the same EPP. In this case if you hit one of the gateway IP, the callout extension should only send available endpoints belong to that gateway to the EPP, and EPP will choose an optimal endpoint based on that.

Similarly, you could have one gateway but separate HTTPRoute/rules, so that on one path (e.g. /v1) requests go to the v1 inference pool, and on the other path (e.g. /v2) requests go to the v2 inference pool. Again in this case the callout extension should only send available endpoints belong to that particular routing rules to the EPP based on where the request was sent.

If there are multiple inference pools sharing the same serving path then EPP will received a list of endpoints belong to all of these inference pools. EPP needs to probe each of the model server endpoints for available model names and choose one of the endpoints based on the model name present in the request body. (Note that this is a common use case even if you want to support just one InferencePool with a common based model + LoRA adapters)

Yes a single EPP can be associated with multiple InferencePools.

An EPP can only be associated with a single InferencePool. The associated InferencePool is specified by the poolName and poolNamespace flags. An HTTPRoute can have multiple backendRefs that reference the same InferencePool and therefore routes to the same EPP. An HTTPRoute can have multiple backendRefs that reference different InferencePools and therefore routes to different EPPs. The implementation guide states:

InferencePool represents a set of Inference-focused Pods and an extension that will be used to route to them.

The " Inference-focused Pods" define the pool of model servers that share common attributes, and "an extension" defines the EPP responsible for picking which model server in the pool the request should be routed to.

@nicolexin this question has surfaced multiple times recently, so you may want to include documentation in this PR that resolves this confusion.

xref: #145

Multi-tenancy comes up a lot. I think at minimum we should be very cautious about it, and potentially even advise against it.

Multi-tenancy would make a single point of failure for multiple sets of accelerators(pools), so an outage would be incredibly painful (loss of revenue + cost of essentially all their expensive accelerators).

Multi-tenancy also puts pressure on any scale issues we may come across, which we currently do not have a good read on.

but just by looking at our InferencePool API, I don't think other implementations are restricted by one EPP associated with a single inference pool.

PTAL at #145 referenced above. InferencePool can only reference a single EPP config, e.g. Service. On the EPP side, it can only take a single poolName and poolNamespace flag. This is a singular bi-directional binding.

On the EPP side, it can only take a single poolName and poolNamespace flag

I'd argue that this is an implementation detail of EPP, and not necessarily a long term or intentional limitation.

I'd argue that this is an implementation detail of EPP, and not necessarily a long term or intentional limitation.

I agree, and that's why I created #252 to provide additional flexibility for the EPP<>InferencePool(s) binding. My comments are specific to what is capable today.

I think the revised wording is great here, thanks @nicolexin! It's reflective of what we have today, and we can use #252 to follow up on more flexible mapping. I think everything else has been resolved, and this is a huge improvement to our docs, so I think we should go ahead and merge.

site-src/reference/spec.md

robscott

Thanks @nicolexin!

site-src/guides/replacing-inference-pool.md

robscott · 2025-04-18T23:01:19Z

site-src/api-types/inferencepool.md

+
+- An InferencePool named `vllm-llama3-8b-instruct` is created in the `default` namespace.
+- It will select Pods that have the label `app: vllm-llama3-8b-instruct`.
+- Traffic routed to this InferencePool will call out to the EPP service `vllm-llama3-8b-instruct-epp` on port `9002` for making routing decisions. If EPP fails to pick an endpoint, or is not responsive, the request will be dropped.


I think a few of these comments got hidden by GitHub UI, hopefully this helps keep them from disappearing.

robscott · 2025-04-18T23:01:25Z

site-src/api-types/inferencepool.md

+- An InferencePool named `vllm-llama3-8b-instruct` is created in the `default` namespace.
+- It will select Pods that have the label `app: vllm-llama3-8b-instruct`.
+- Traffic routed to this InferencePool will call out to the EPP service `vllm-llama3-8b-instruct-epp` on port `9002` for making routing decisions. If EPP fails to pick an endpoint, or is not responsive, the request will be dropped.
+- Traffic routed to this InferencePool will be forwarded to the port `8000` on the selected Pods.


I think a few of these comments got hidden by GitHub UI, hopefully this helps keep them from disappearing.

k8s-ci-robot · 2025-04-18T23:01:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kfswain, nicolexin, robscott

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kfswain]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Co-authored-by: Rob Scott <rob.scott87@gmail.com>

robscott · 2025-04-23T00:36:10Z

Thanks for all the work on this @nicolexin, this is great!

/lgtm

nicolexin added 2 commits April 9, 2025 11:50

Initial guide for inference pool

ca9a110

Add extensionReference to the InferencePool spec

66c0860

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 9, 2025

k8s-ci-robot requested review from ahg-g and danehans April 9, 2025 21:17

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 9, 2025

Fix list formatting

c4abfe5

danehans reviewed Apr 9, 2025

View reviewed changes

site-src/api-types/inferencepool.md Outdated Show resolved Hide resolved

site-src/reference/spec.md Show resolved Hide resolved

site-src/reference/spec.md Outdated Show resolved Hide resolved

site-src/reference/spec.md Outdated Show resolved Hide resolved

Remove unused labels

63a60a2

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 10, 2025

Autogenerate the spec

3ebb03f

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 10, 2025

robscott reviewed Apr 11, 2025

View reviewed changes

nicolexin and others added 11 commits April 11, 2025 09:28

Update site-src/api-types/inferencepool.md

fac37a9

Co-authored-by: Rob Scott <rob.scott87@gmail.com>

Update site-src/api-types/inferencepool.md

5953c37

Co-authored-by: Rob Scott <rob.scott87@gmail.com>

Update site-src/api-types/inferencepool.md

41fa708

Co-authored-by: Rob Scott <rob.scott87@gmail.com>

Update site-src/api-types/inferencepool.md

c22d962

Co-authored-by: Rob Scott <rob.scott87@gmail.com>

Update site-src/api-types/inferencepool.md

bbda2b5

Co-authored-by: Rob Scott <rob.scott87@gmail.com>

Update site-src/api-types/inferencepool.md

6848995

Co-authored-by: Rob Scott <rob.scott87@gmail.com>

Rename llm-pool names in rollout example

1c14218

Add use cases for replacing an inference pool

68a49a2

Rewording the background section

3e4285a

Create replacing-inference-pool.md

a2945e9

Replace instructions with a link for how to replace an inference pool

7520f6f

nicolexin added 3 commits April 11, 2025 10:36

Update replacing-inference-pool.md

860e121

Update mkdocs.yml

53e16b8

Update replacing-inference-pool.md

ca7e02e

elevran reviewed Apr 16, 2025

View reviewed changes

nicolexin added 4 commits April 16, 2025 09:18

Merge branch 'kubernetes-sigs:main' into inferencepool-ref

4127b08

Update inferencemodel_types.go

85c9311

Merge branch 'kubernetes-sigs:main' into inferencepool-ref

cce8c0b

Update inferencepool.md

ee580da

robscott approved these changes Apr 18, 2025

View reviewed changes

Update site-src/guides/replacing-inference-pool.md

4fdfadf

Co-authored-by: Rob Scott <rob.scott87@gmail.com>

k8s-ci-robot assigned robscott Apr 23, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 23, 2025

k8s-ci-robot merged commit 7d238dd into kubernetes-sigs:main Apr 23, 2025
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Complete the InferencePool documentation #673

Complete the InferencePool documentation #673

nicolexin commented Apr 9, 2025

k8s-ci-robot commented Apr 9, 2025

netlify bot commented Apr 9, 2025 •

edited

Loading

nirrozenbaum commented Apr 10, 2025

kfswain commented Apr 10, 2025 •

edited

Loading

robscott left a comment

elevran Apr 16, 2025

nicolexin Apr 16, 2025

elevran Apr 17, 2025 •

edited

Loading

nicolexin Apr 17, 2025

danehans Apr 18, 2025

kfswain Apr 21, 2025 •

edited

Loading

danehans Apr 21, 2025

robscott Apr 21, 2025

danehans Apr 22, 2025

robscott Apr 23, 2025

robscott left a comment

robscott Apr 18, 2025

robscott Apr 18, 2025

k8s-ci-robot commented Apr 18, 2025

robscott commented Apr 23, 2025


		It is expected for the InferencePool to:
		An InferencePool is expected to be bundled with an [Endpoint Picker](https://github.com/kubernetes-sigs/gateway-api-inference-extension/tree/main/pkg/epp) extension. This extension is responsible for tracking key metrics on each model server (i.e. the KV-cache utilization, queue length of pending requests, active LoRA adapters, etc.) and routing incoming inference requests to the optimal model server replica based on these metrics.

Complete the InferencePool documentation #673

Complete the InferencePool documentation #673

Conversation

nicolexin commented Apr 9, 2025

k8s-ci-robot commented Apr 9, 2025

netlify bot commented Apr 9, 2025 • edited Loading

✅ Deploy Preview for gateway-api-inference-extension ready!

nirrozenbaum commented Apr 10, 2025

kfswain commented Apr 10, 2025 • edited Loading

robscott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elevran Apr 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kfswain Apr 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robscott left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 18, 2025

robscott commented Apr 23, 2025

netlify bot commented Apr 9, 2025 •

edited

Loading

kfswain commented Apr 10, 2025 •

edited

Loading

elevran Apr 17, 2025 •

edited

Loading

kfswain Apr 21, 2025 •

edited

Loading