To use Float16.cloud with the LangChain, follow these steps:
Install the LangChain package:
pipinstalllangchain
or
condainstalllangchain-cconda-forge
Use the following Python code snippet (For example, Llama2-7B model):
from langchain.chat_models import ChatOpenAIfrom langchain.schema import HumanMessageFLOAT16_BASE_URL ="https://api.float16.cloud/v1/"FLOAT16_API_KEY ="<your API key>"chat =ChatOpenAI( model="Llama2-7B", api_key=FLOAT16_API_KEY, base_url=FLOAT16_BASE_URL, streaming=True,)# Simple invocation:print(chat.invoke([HumanMessage(content="Hello")]))# Streaming invocation:for chunk in chat.stream("Write me a blog about how to start to raise cats"):print(chunk.content, end="", flush=True)
Curl
To quickly try the API using cURL, use the following command:
curl -X POST https://api.float16.cloud/v1/chat/completions -d '{"model": "Llama2-7B", "messages": [{"role":"system", "content":"You are truly awesome."}, {"role":"user", "content":"Howdy"}]}' -H "Content-Type: application/json" -H "Authorization: Bearer <your API key>"