Skip to content

[Feature] Realtime API from OpenAI working #545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Gemfile
Original file line number Diff line number Diff line change
@@ -5,6 +5,8 @@ gemspec

gem "byebug", "~> 11.1.3"
gem "dotenv", "~> 2.8.1"
gem "eventmachine", "~> 1.2.7"
gem "faye-websocket", "~> 0.11.3"
gem "rake", "~> 13.2"
gem "rspec", "~> 3.13"
gem "rubocop", "~> 1.50.2"
9 changes: 9 additions & 0 deletions Gemfile.lock
Original file line number Diff line number Diff line change
@@ -21,13 +21,17 @@ GEM
diff-lcs (1.5.1)
dotenv (2.8.1)
event_stream_parser (1.0.0)
eventmachine (1.2.7)
faraday (2.8.1)
base64
faraday-net_http (>= 2.0, < 3.1)
ruby2_keywords (>= 0.0.4)
faraday-multipart (1.0.4)
multipart-post (~> 2)
faraday-net_http (3.0.2)
faye-websocket (0.11.3)
eventmachine (>= 0.12.0)
websocket-driver (>= 0.5.1)
hashdiff (1.1.0)
json (2.6.3)
multipart-post (2.3.0)
@@ -74,13 +78,18 @@ GEM
addressable (>= 2.8.0)
crack (>= 0.3.2)
hashdiff (>= 0.4.0, < 2.0.0)
websocket-driver (0.7.6)
websocket-extensions (>= 0.1.0)
websocket-extensions (0.1.5)

PLATFORMS
ruby

DEPENDENCIES
byebug (~> 11.1.3)
dotenv (~> 2.8.1)
eventmachine (~> 1.2.7)
faye-websocket (~> 0.11.3)
rake (~> 13.2)
rspec (~> 3.13)
rubocop (~> 1.50.2)
4 changes: 4 additions & 0 deletions lib/openai.rb
Original file line number Diff line number Diff line change
@@ -18,6 +18,7 @@
require_relative "openai/audio"
require_relative "openai/version"
require_relative "openai/batches"
require_relative "openai/real_time"

module OpenAI
class Error < StandardError; end
@@ -46,11 +47,13 @@ class Configuration
:log_errors,
:organization_id,
:uri_base,
:websocket_uri_base,
:request_timeout,
:extra_headers

DEFAULT_API_VERSION = "v1".freeze
DEFAULT_URI_BASE = "https://api.openai.com/".freeze
DEFAULT_WEBSOCKET_URI_BASE = "wss://api.openai.com/".freeze
DEFAULT_REQUEST_TIMEOUT = 120
DEFAULT_LOG_ERRORS = false

@@ -61,6 +64,7 @@ def initialize
@log_errors = DEFAULT_LOG_ERRORS
@organization_id = nil
@uri_base = DEFAULT_URI_BASE
@websocket_uri_base = DEFAULT_WEBSOCKET_URI_BASE
@request_timeout = DEFAULT_REQUEST_TIMEOUT
@extra_headers = {}
end
5 changes: 5 additions & 0 deletions lib/openai/client.rb
Original file line number Diff line number Diff line change
@@ -10,6 +10,7 @@ class Client
log_errors
organization_id
uri_base
websocket_uri_base
request_timeout
extra_headers
].freeze
@@ -95,6 +96,10 @@ def batches
@batches ||= OpenAI::Batches.new(client: self)
end

def real_time
@real_time ||= OpenAI::RealTime.new(client: self)
end

def moderations(parameters: {})
json_post(path: "/moderations", parameters: parameters)
end
7 changes: 7 additions & 0 deletions lib/openai/http_headers.rb
Original file line number Diff line number Diff line change
@@ -22,6 +22,13 @@ def openai_headers
}.compact
end

def openai_realtime_headers
{
"Authorization" => "Bearer #{@client.access_token}",
"OpenAI-Beta" => "realtime=v1"
}
end

def azure_headers
{
"Content-Type" => "application/json",
31 changes: 31 additions & 0 deletions lib/openai/real_time.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
require "faye/websocket"
require "eventmachine"

module OpenAI
class RealTime
include HTTPHeaders

def initialize(client:)
@client = client
@websocket = nil
@on_message = nil
end

def on_message(&block)
@on_message = block
end

def connect(model: "gpt-4o-realtime-preview-2024-10-01")
uri = "#{File.join(@client.websocket_uri_base, @client.api_version, 'realtime')}?model=#{model}"
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably this uri shouldn't be here but I was not sure where to put it


EM.run do
@websocket = Faye::WebSocket::Client.new(uri, nil, headers: openai_realtime_headers)
@websocket.on :message, @on_message
end
Comment on lines +21 to +24
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we should replace Eventmachine since it last release was 6 yeas ago. I think we could use async. I have tried and it works fine with something like this:

    Async do
      endpoint = Async::HTTP::Endpoint.parse(uri, alpn_protocols: Async::HTTP::Protocol::HTTP11.names)
      Async::WebSocket::Client.connect(endpoint, headers: @headers) do |connection|
        @websocket = connection

        while (message = connection.read)
          @on_message
        end
      end
    end

end

def send_event(event)
@websocket.send(event)
end
end
end