Skip to content
This repository was archived by the owner on Sep 12, 2024. It is now read-only.
This repository was archived by the owner on Sep 12, 2024. It is now read-only.

What files are compatible? #124

@bedcoding

Description

@bedcoding

I have read the document below.
https://llama-node.vercel.app/docs/start

So, I looked for a model to use in the source code at the site below.
https://huggingface.co/models?search=ggml

Then an error like this appears. What files should I get?

llama.cpp: loading model from model/llama-2-7b-chat.ggmlv3.q2_K.bin
error loading model: unrecognized tensor type 10

llama_init_from_file: failed to load model
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^

[Error: Failed to initialize LLama context from file: model/llama-2-7b-chat.ggmlv3.q2_K.bin] {
  code: 'GenericFailure'
}

code:

import { LLM } from "llama-node";
import { LLamaCpp } from "llama-node/dist/llm/llama-cpp.js";

const model = "model/llama-2-7b-chat.ggmlv3.q2_K.bin";
const llama = new LLM(LLamaCpp);
const config = {
    modelPath: model,
    enableLogging: true,
    nCtx: 1024,
    seed: 0,
    f16Kv: false,
    logitsAll: false,
    vocabOnly: false,
    useMlock: false,
    embedding: false,
    useMmap: true,
    nGpuLayers: 0
};

const template = `How are you?`;
const prompt = `A chat between a user and an assistant.
USER: ${template}
ASSISTANT:`;

const run = async () => {
  await llama.load(config);

  await llama.createCompletion({
      nThreads: 4,
      nTokPredict: 2048,
      topK: 40,
      topP: 0.1,
      temp: 0.2,
      repeatPenalty: 1,
      prompt,
  }, (response) => {
      process.stdout.write(response.token);
  });
}

run();

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions