bob_llm.backend_clients

Classes

OpenAICompatibleClient

Client for OpenAI-compatible APIs.

Module Contents

class bob_llm.backend_clients.OpenAICompatibleClient(api_url: str, api_key: str, model: str, logger, temperature: float = 0.7, top_p: float = 1.0, max_tokens: int = 0, stop: list = None, presence_penalty: float = 0.0, frequency_penalty: float = 0.0, timeout: float = 60.0, response_format: dict = None)

Client for OpenAI-compatible APIs.

api_url

api_key

model

logger

headers

temperature = 0.7

top_p = 1.0

max_tokens = 0

stop = None

presence_penalty = 0.0

frequency_penalty = 0.0

timeout = 60.0

response_format = None

_build_payload(history: list, tools: list = None, tool_choice: str = 'auto', stream: bool = False) → dict

Construct the JSON payload for an API request.

Parameters:

history – The list of messages in the chat history.
tools – An optional list of tool definitions.
tool_choice – The choice for tool calling (e.g., ‘auto’, ‘none’, ‘required’).
stream – A boolean indicating whether to enable streaming.

Returns:

A dictionary representing the complete JSON payload.

process_prompt(history: list, tools: list = None, tool_choice: str = 'auto') → Tuple[bool, dict]: Send a prompt to the LLM and wait for the full response.

process_prompt_stream(history: list, tools: list = None, tool_choice: str = 'auto'): Send a prompt and stream response tokens using zero-buffering manual parsing.