Skip to content

[BUG] The health status of chat api server cannot be queried when the chat api server is generating responses. #29

Open
@tjtanaa

Description

@tjtanaa

Describe the bug

The health status of chat api server cannot be queried when the chat api server is generating responses.

Activity

reopened this on Sep 5, 2024
tjtanaa

tjtanaa commented on Sep 25, 2024

@tjtanaa
ContributorAuthor

The issue has been identified:
Since the server is launched using a single worker, this means that when the model is streaming or generating results, the worker will be occupied. Any incoming requests to the server will need to be queued until the one and only worker is free.

Welcome more suggestions by replying to this comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type: bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @tjtanaa@szeyu

        Issue actions

          [BUG] The health status of chat api server cannot be queried when the chat api server is generating responses. · Issue #29 · EmbeddedLLM/embeddedllm