Installation

Installation and Usage Guide for Float16.cloud

OpenAI

To use Float16.cloud with the OpenAI, follow these steps:

• Install the OpenAI package:

pip install openai

• Use the following Python code snippet (For example, Llama2-7B model):

import httpx
import openai

FLOAT16_BASE_URL = "https://api.float16.cloud/v1/"
FLOAT16_API_KEY = "<your API key>"

client = openai.OpenAI(
    api_key=FLOAT16_API_KEY,
    base_url=FLOAT16_BASE_URL,
)
client._base_url = httpx.URL(FLOAT16_BASE_URL)

# Streaming chat:
messages = [{"role": "system", "content": "You are truly awesome."}]

while True:
    content = input(f"User:")
    messages.append({"role": "user", "content": content})
    print(f"Assistant:", sep="", end="", flush=True)
    content = ""

    for chunk in client.chat.completions.create(
        messages=messages,
        model="Llama2-7B",
        stream=True,
    ):
        delta_content = chunk.choices[0].delta.content
        if delta_content:
            print(delta_content, sep="", end="", flush=True)
            content += delta_content
    
    messages.append({"role": "assistant", "content": content})
    print("\n")

LangChain

To use Float16.cloud with the LangChain, follow these steps:

• Install the LangChain package:

pip install langchain

or

conda install langchain -c conda-forge

• Use the following Python code snippet (For example, Llama2-7B model):

from langchain.chat_models import ChatOpenAI
from langchain.schema import HumanMessage

FLOAT16_BASE_URL = "https://api.float16.cloud/v1/"
FLOAT16_API_KEY = "<your API key>"

chat = ChatOpenAI(
    model="Llama2-7B",
    api_key=FLOAT16_API_KEY,
    base_url=FLOAT16_BASE_URL,
    streaming=True,
)

# Simple invocation:
print(chat.invoke([HumanMessage(content="Hello")]))

# Streaming invocation:
for chunk in chat.stream("Write me a blog about how to start to raise cats"):
    print(chunk.content, end="", flush=True)

Curl

To quickly try the API using cURL, use the following command:

curl -X POST https://api.float16.cloud/v1/chat/completions -d '{"model": "Llama2-7B",  "messages": [{"role":"system", "content":"You are truly awesome."}, {"role":"user", "content":"Howdy"}]}' -H "Content-Type: application/json"  -H "Authorization: Bearer <your API key>"

Upcoming Features

• HayStack

• VS Code Extension

For Further Assistance:

If you need additional help, feel free to contact us at [e-mail].