SambaNova inference APIs are designed to be compliant with OpenAI client libraries to simplify the adoption of our inference technologies to enhance your AI applications.

​
Download the library

Run the command below to download the library. 
pip install openai

​
Use SambaNova APIs with OpenAI client libraries

Configuring your OpenAI client libraries to use SambaNova inference APIs is as simple as setting two values: the base_url and your api_key, as shown below.
Don’t have a SambaNova API key? Get yours from the API keys and URLs page.
from openai import OpenAI

client = OpenAI(
    base_url="your-sambanova-base-url", 
    api_key="your-sambanova-api-key"
)
Now you can make an API request to a model and choose how to receive your output. 

​
Non-streaming example

The following code demonstrates using the OpenAI Python client for non-streaming completions. 
completion = client.chat.completions.create(
  model="Meta-Llama-3.1-8B-Instruct",
  messages = [
      {"role": "system", "content": "Answer the question in a couple sentences."},
      {"role": "user", "content": "Share a happy story with me"}
    ]
)

print(completion.choices[0].message)

​
Streaming example

The following code demonstrates using the OpenAI Python client for streaming completions. 
completion = client.chat.completions.create(
  model="Meta-Llama-3.1-8B-Instruct",
  messages = [
      {"role": "system", "content": "Answer the question in a couple sentences."},
      {"role": "user", "content": "Share a happy story with me"}
    ],
  stream= True
)

for chunk in completion:
  print(chunk.choices[0].delta.content)
In streaming mode, the API returns chunks that contain multiple tokens. When calculating metrics like tokens per second or time per output token, ensure that you account for all tokens in each chunk.

​
Currently unsupported OpenAI features

The following features are not yet supported and will be ignored:
  • logprobs
  • top_logprobs
  • n
  • presence_penalty
  • frequency_penalty
  • logit_bias
  • seed

​
Feature differences

temperature: The SambaNova API supports a value between 0 and 1 whereas OpenAI supports values between 0 and 2.

​
SambaNova API features not supported by OpenAI clients

The SambaNova API supports the top_k parameter, which is not supported by the OpenAI client libraries.