Proposed: Calling Open Source LLM with Aria

The Context

Access to OpenAI’s ChatGPT-4 cannot last forever.

Problem Statement

Using the assistant name Aria, I want to call an Apache2 licensed large language model (LLM) with open weights in my code editor, Visual Studio Code by Microsoft.

Proposed Solution

Install LM Studio: A tool for managing and deploying large language models.
Download a quantized version of Hermes-2-Pro-Mistral-7B: This model is available at Hugging Face Model Card.
Install Visual Studio Code: The popular code editor by Microsoft.
Install the Genie extension: This lets you interact with AI models directly in Visual Studio Code. ai-genie.chatgpt-vscode

Set Genie to use the assistant name “Aria”: This will give the AI a unique identity within Visual Studio Code.

{
  "genie.assistantName": "Aria",
  "genie.apiUrl": "http://localhost:1234/v1/chat/completions",
  "genie.model": {
    "id": "nousresearch/hermes-2-pro-mistral-7b",
    "tokenLimit": 8192
  }
}

Start using Aria in Visual Studio Code: With the above settings, you can now access and interact with the open source large language model (LLM) named Hermes-2-Pro-Mistral-7B through the Genie extension.

Benefits

Access to an AI assistant large language model.
Ability to develop and fine-tune your data models on open-source datasets.

Downsides

Requires a good understanding of natural language processing (NLP) and machine learning concepts.
May require additional hardware resources for running the LLM locally.

Road Not Taken

Using proprietary AI models that are not open-source or freely available, such as OpenAI’s ChatGPT-4.
Relying on cloud-based services for AI interactions instead of local deployment.

Infrequent Use Case

In cases where real-time performance is critical, using an LLM might not be the optimal solution due to its computational requirements.

Status

Status: Proposed

Decision Makers

V-Sekai development team
Fire