[User] Load config from kcpps during startup

Right now it's possible to save and load config files (.kcpps) using the GUI. It would be very practical if one could supply a kcpps as an command-line argument to koboldcpp.py during startup.

Something like python koboldcpp.py config/my-config.kcpps (but in a way that doesn't break compatibility).

I have looked everywhere but can't seem to find any information on this .kcpps format?
When I press the "Save" button in the UI it offers to save a .json file:

Loading the save_story.json config file works in the UI but not if I provide it as an argument to the --config parameter, when running KoboldAI Lite from command line as:
python koboldcpp.py models/llama-2-13b-chat.Q5_K_M.gguf 8008 --config save_story.json --stream --multiuser --highpriority --nommap --hordeconfig Llama-2-13B

The .kcpps file represets launcher configs such as number of layers, GPU settings, thread counts etc. You load it via the GUI launcher.

For the json saves they must be loaded via the UI after launching it.

The .kcpps file represets launcher configs such as number of layers, GPU settings, thread counts etc. You load it via the GUI launcher.

For the json saves they must be loaded via the UI after launching it.

Many thanks for your prompt reply.

I am running koboldAI Lite from command line on a Linux RHEL9 serve without graphical interface, hence I cannot access the GUI launcher. Is there any way I can download the default .kcpps file to see how it is structured?

By the way, what I need is to be able to load by default from the command line graphical parameters for the main GUI such as, for e.g.:

Format: Chat Mode
UI Style Select: Aesthetic
Multiline Replies: Yes
AI Name: FriLLM
...

Is it possible to specify these graphical parameters in the .kcpps file?
If not, how can I do it?

In my case, KoboldAI Lite will be accessible on a server with access restricted to University employees and we want to provide a graphical interface that will be displayed with the same parameters no matter the computer they use. Currently, if I load graphical interface parameters with a .json file on my computer, these parameters do not apply to others...

Best,
C.

Here's an example config file, save it as a .kcpps file and you can load it with --config. The parameters match the ones used inside the argparse - for info run it with --help

{"model": null, "model_param": "E:/LLaMA/models/airoboros-l2-7b-2.2.1.Q4_K_S.gguf", "port": 5001, "port_param": 5001, "host": "", "launch": false, "lora": null, "config": null, "threads": 5, "blasthreads": null, "psutil_set_threads": false, "highpriority": false, "contextsize": 2048, "blasbatchsize": 512, "ropeconfig": [0.0, 10000.0], "stream": false, "smartcontext": false, "unbantokens": false, "bantokens": null, "usemirostat": null, "forceversion": 0, "nommap": false, "usemlock": false, "noavx2": false, "debugmode": false, "skiplauncher": false, "hordeconfig": null, "noblas": false, "useclblast": null, "usecublas": ["normal", "0", "mmq"], "gpulayers": 99, "tensor_split": null, "onready": "", "multiuser": true, "foreground": false}

Currently the WebUI is completely independent from the server, and thus has its own individual configurations (such as chat mode, story settings etc) that are not configurable from the server. Would it be possible for you to provide your users with the .json containing your desired settings, as a separate download link for them to load?

Remote save file loading is something I am considering, but have no immediate plans to add right now.

Here's an example config file, save it as a .kcpps file and you can load it with --config. The parameters match the ones used inside the argparse - for info run it with --help

{"model": null, "model_param": "E:/LLaMA/models/airoboros-l2-7b-2.2.1.Q4_K_S.gguf", "port": 5001, "port_param": 5001, "host": "", "launch": false, "lora": null, "config": null, "threads": 5, "blasthreads": null, "psutil_set_threads": false, "highpriority": false, "contextsize": 2048, "blasbatchsize": 512, "ropeconfig": [0.0, 10000.0], "stream": false, "smartcontext": false, "unbantokens": false, "bantokens": null, "usemirostat": null, "forceversion": 0, "nommap": false, "usemlock": false, "noavx2": false, "debugmode": false, "skiplauncher": false, "hordeconfig": null, "noblas": false, "useclblast": null, "usecublas": ["normal", "0", "mmq"], "gpulayers": 99, "tensor_split": null, "onready": "", "multiuser": true, "foreground": false}

Many thanks for the explanations!

Currently the WebUI is completely independent from the server, and thus has its own individual configurations (such as chat mode, story settings etc) that are not configurable from the server. Would it be possible for you to provide your users with the .json containing your desired settings, as a separate download link for them to load?

Remote save file loading is something I am considering, but have no immediate plans to add right now.

Very well, I wasn't aware of the distinction between the server and the WebUI. Unfortunately, I cannot provide them with a .json file as our institution has >1000 collaborators (often with short-term contracts)...
I will thus be waiting for your implementation of the remote save file loading, can you keep me updated in this thread?
Many thanks and best regards,
C.

I would love it if the last used settings were saved. So if I reopened the GUI, it would use the last used settings instead of default.

I keep 99% of my settings the same, but change only 1 or two, so this would be a huge time saver, as I don't want to be having to "load" a setting every time or make dozens of different settings to load via the GUI or launch command.

Hmm... I'd prefer not to automatically write to the user's directory. Especially since the pyinstaller runs from a temporary directory. Loading a config should only take one click though, then you can change the setting you need.

I would love it if the last used settings were saved. So if I reopened the GUI, it would use the last used settings instead of default.

I keep 99% of my settings the same, but change only 1 or two, so this would be a huge time saver, as I don't want to be having to "load" a setting every time or make dozens of different settings to load via the GUI or launch command.

As I suggested, being able to load the .json GUI config file when running koboldcpp from command line might also save you some time :)

@CorentinWicht

Hi @CorentinWicht not sure if you're still following this project, but this feature has finally been added in 1.49. You can now specify a .json file which the server will host and all clients will download on launch with --preloadstory

@CorentinWicht

Hi @CorentinWicht not sure if you're still following this project, but this feature has finally been added in 1.49. You can now specify a .json file which the server will host and all clients will download on launch with --preloadstory

Hi @LostRuins many thanks for implementing the feature in 1.49, it works wonders !

Best wishes,
C.

Navigation Menu

Repository navigation

[User] Load config from kcpps during startup #388

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[User] Load config from kcpps during startup #388

Description

Activity

CorentinWicht commented on Oct 4, 2023

LostRuins commented on Oct 4, 2023

CorentinWicht commented on Oct 5, 2023

LostRuins commented on Oct 5, 2023

CorentinWicht commented on Oct 6, 2023

alicat22 commented on Oct 18, 2023

LostRuins commented on Oct 18, 2023

CorentinWicht commented on Oct 19, 2023

LostRuins commented on Nov 11, 2023

CorentinWicht commented on Nov 15, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions