Tonight we're going to hook up the Zephyr Large Language Model (LLM) to Kari's Python Kit.

As you may be aware, Python is the preferred programming language for developing top-tier LLM chatbots. With the introduction of Kari 6 Pro, we now have the capability to seamlessly integrate any LLM with Kari. It's essential to note that the performance of LLMs is significantly influenced by the computational power of the host system. Without a robust GPU (graphics processing unit), response times can be frustratingly slow. To address this concern, we've carefully selected the Zephyr LLM, boasting only 7 billion parameters. Despite its low parameter count, the Zephyr LLM offers exceptional conversational abilities, positioning it as a formidable competitor even over ChatGPT, which operates on a supercomputer and boasts a staggering 1.76 trillion parameters. Let's delve into the details and get started on this exciting journey.

If you'd like to visit the official Zephyr HuggingFace page, go here.


1. First, we need to install Anaconda. Go to https://www.anaconda.com/download and click the download button.

After downloading the Anaconda setup file, just run it.

Install Anaconda using the default settings in the installer. Just click Next until you get to this page:



Go ahead and check Add Anaconda3 to my PATH environment variable. Ignore the warning. Click Next. At the end, don't run the Navigator. Just exit the installer.

Great, now that Anaconda is installed, we can use it to run Python in a separate environment. This means that anything we install within the Anaconda environment won't have any impact on your main system.

Now, choose a location on your work drive and make a folder. I'm going to call mine "Zephyr" located in "D:\AI\Zephyr".

Now, run a Command Prompt in Administrator mode. You can find the Command Prompt by going to Search in your task bar and typing "Command Prompt". Once you locate the icon, right-click it and choose Run as Administrator.

In the command prompt, navigate to your folder:

Type this into the Command Prompt:

    
D:
    

    
cd D:\AI\Zephyr
    

Now we'll create the zephyr environment in Anaconda. Type this into the Command Prompt:

    
conda create --name zephyr
    

When it asks, type "y" for yes.

When it completes, we will activate and enter the environment. Type this into the Command Prompt:

    
conda activate zephyr
    

The Command Prompt will change to reflect that we are now inside the zephyr environment: (zephyr)

Now we will install the Python requirement scripts to run Zephyr LLM. Type this into the Command Prompt one line at a time:

    
conda install -c conda-forge accelerate
pip install transformers
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
    

It's time to test Zephyr. Do not close the Command Prompt yet. Download these Python scripts I made to access Zephyr: [Link]. Unzip the file and put the scripts in "D:\AI\Zephyr" or whatever folder you're working from.

Run the test script like this in the Command Prompt:

    
python test1.py "What is the history of apples?"
    

The first thing the script will do is download the LLM files. It's a large download but it only happens the first time you run it.

After a while, Zephyr will spit out its first response. Congratulations! It's working!

Now we'll connect Zephyr to Kari.

First, make sure we're running Kari Pro 6.04 or later. So head over to the Update webpage [HERE] and update your Kari.

Now run Kari Pro 6 and open the Python Kit by going to Menu > Python Kit.

First, we need to set the location of the Anaconda activate.bat batch file. Click the "SET ANACONDA activate.bat LOCATION" button. Now navigate to "C:\Users\your user name\anaconda3\Scripts\activate.bat" and click Open.

Now click the "ADD SCRIPT" button. Navigate to "D:\AI\Zephyr\zephyr1.py" and click Open.

Now click the "SET WORKING DIRECTORY" button. Navigate to and select "D:\AI\Zephyr" or wherever you installed the Zephyr scripts.

Under the "Python Scripts" list, select the "zephyr1" script by clicking to make it selected. Now click the "SET ANACONDA ENVIRONMENT FOR SELECTED SCRIPT" button. Choose the Anaconda environment we created, which is "zephyr".

Now set the small slider below all the way to the right towards "Python Kit". That will make Kari use the script in chat all the time.

Now put this string into the argument list:

    
"#InMessage" "#PreviousOutMessage" "#PreviousInMessage" "#PrevPreviousOutMessage" "#PrevPreviousInMessage" 60
    

The above arguments are the messages that will be fed into the zephyr1 script. For example, "#InMessage" will be replaced with the user's chat input or "in" message. And "out" keywords like "#PreviousOutMessage" are what comes out of the script, the output, the response. We feed all these messages into the Zephyr script to give it as much data about the current conversation. The number at the end is important. It controls the number of max tokens and how long the response will be. 60 tokens is about 60 words.

Now click "EXECUTE SCRIPT" button to test the script. We can see the output appear in the Output textbox.

Once it's done, look at the end of the output. The main response is between the word "<|assistant|>" and the word "#end#". Let Kari know how to parse the response by putting the words "<|assistant|>" and "#end#" into the textboxes. Now it should say: Parse output between... this text: <|assistant|> ... and this text: #end#

If you click "EXECUTE SCRIPT" button again you will now get the correct response in the Parsed Output Text textbox.

Now under the Python Scripts list, make sure the "Active" checkbox for the script is checked so that Kari knows to run the script when you chat.

Now exit the Python Kit and go to the main Chat Window. Start chatting!

This tutorial offers insights on connecting Large Language Models (LLMs) to Kari. LLMs primarily serve as chatbot assistants but hold potential for broader applications as AI evolves. The future promises versatility and innovation in LLMs, expanding beyond chatbots, even into uncensored virtual companions. The AI community is excited about their potential for diverse uses, making it an exciting time to be involved in this evolving field.