Merge pull request #42 from A-Baji/dev

changelog, handle new naming system

Author: A-Baji

CHANGELOG.md

@@ -0,0 +1,58 @@
+# Changelog
+
+Observes [Semantic Versioning](https://semver.org/spec/v2.0.0.html) standard and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) convention.
+
+## [2.0.0] - 06-15-2023
+
+### Added
+
+- a changelog
+- confirmation dialogue for model deletion
+
+### Changed
+
+- flag to use an existing dataset for model creation to allow manual revision
+- user input for model creation updated to work with new discord naming system
+- tweaked readme
+
+## [1.2.2] - 02-22-2023
+
+### Fixed
+
+- bug with end of thought punctuation
+
+## [1.2.1] - 02-19-2023
+
+### Fixed
+
+- bug with min and max thought length
+
+## [1.2.0] - 02-17-2023
+
+### Added
+
+- events flag for job status command
+- min and max thought length parameters for model creation
+
+## [1.1.0] - 02-11-2023
+
+### Changed
+
+- replaced `subprocess` instances with appropriate openAI python API methods
+
+### Removed
+
+- dataset clean up step with openAI fine tune API
+
+## [1.0.1] - 02-11-2023
+
+### Changed
+
+- switched to `pathlib` for file path parsing
+
+[2.0.0]: https://github.com/A-Baji/discordAI-modelizer/compare/1.2.2...2.0.0
+[1.2.2]: https://github.com/A-Baji/discordAI-modelizer/compare/1.2.1...1.2.2
+[1.2.1]: https://github.com/A-Baji/discordAI-modelizer/compare/1.2.0...1.2.1
+[1.2.0]: https://github.com/A-Baji/discordAI-modelizer/compare/1.1.0...1.2.0
+[1.1.0]: https://github.com/A-Baji/discordAI-modelizer/compare/1.0.1...1.1.0
+[1.0.1]: https://github.com/A-Baji/discordAI-modelizer/compare/1.0.0...1.0.1

README.md

@@ -1,5 +1,5 @@
 # DiscordAI Modelizer
-DiscordAI Modelizer is a python package that can generate custom openai models based on a discord user's chat history in a discord channel. It uses [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) to download the logs of a channel. Then, after the logs are processed into a usable dataset, it uses [openAI's API](https://beta.openai.com/docs/introduction) to create a customized model. It also wraps some of the tools from the openAI API to help with managing customizations.
+DiscordAI Modelizer is a python package that can generate custom openai models based on a discord user's chat history in a discord channel. It uses [DiscordChatExporter](https://github.com/Tyrrrz/DiscordChatExporter) to download the logs of a channel, processes the logs into a usable dataset, and then uses [openAI's API](https://beta.openai.com/docs/introduction) to create a customized model. It also wraps some of the tools from the openAI API to help with managing customizations.
 
 DiscordAI Modelizer is primarily used as a subcomponent of [DiscordAI](https://github.com/A-Baji/discordAI), but may also be used independently.
 
@@ -14,25 +14,26 @@ DiscordAI Modelizer is primarily used as a subcomponent of [DiscordAI](https://g
 ### Model
 Commands related to your openAI models
 #### `discordai_modelizer model list`
-List your openAi customized models
+* List your openAI models
 #### `discordai_modelizer model create`
-Create a new openAI customized model by downloading the specified chat logs, parsing them into a usable dataset, and then training a customized model using openai
-
-For a proper usage, see the [guide](https://github.com/A-Baji/discordAI#create-a-new-customized-openai-model) for DiscordAI.
+* Create a new openAI customized model by downloading the specified chat logs, parsing them into a usable dataset, and then training a customized model using openai
+* For proper usage, see the [DiscordAI guide](https://github.com/A-Baji/discordAI#create-a-new-customized-openai-model)
 #### `discordai_modelizer model delete`
-Delete an openAI customized model
+* Delete an openAI model
 ### Job
 Commands related to your openAI jobs
 #### `discordai_modelizer job list`
-List your openAI customization jobs
+* List your openAI customization jobs
 #### `discordai_modelizer job follow`
-Follow an openAI customization job
+* Follow an openAI customization job
 #### `discordai_modelizer job status`
-Get an openAI customization job's status
+* Get an openAI customization job's status
 #### `discordai_modelizer job cancel`
-Cancel an openAI customization job
+* Cancel an openAI customization job
 
 ## Disclaimer
 This application allows users to download the chat history of any channel for which they have permission to invite a bot, and then use those logs to create an openai model based on a user's chat messages. It is important to note that this application should only be used with the consent of all members of the channel. Using this application for malicious purposes, such as impersonation, or without the consent of all members is strictly prohibited.
 
-By using this application, you agree to use it responsibly. The developers of this application are not responsible for any improper use of the application or any consequences resulting from such use. We strongly discourage using this application for any unethical purposes.
+By using this application, you agree to use it responsibly. The developers of this application are not responsible for any improper use of the application or any consequences resulting from such use. We strongly discourage using this application for any unethical purposes.
+
+This application is not affiliated with or endorsed by Discord, Inc. The use of the term "Discord" in our product name is solely for descriptive purposes to indicate compatibility with the Discord platform.

discordai_modelizer/customize.py

@@ -58,8 +58,7 @@ def create_model(bot_token: str, openai_key: str, channel_id: str, user_id: str,
             file=open(full_dataset_path, "rb"),
             purpose='fine-tune'
         )
-        file_id = upload_response.id
-        fine_tune=openai.FineTune.create(api_key=openai_key, training_file=file_id, model=base_model, suffix=user_id)
+        fine_tune=openai.FineTune.create(api_key=openai_key, training_file=upload_response.id, model=base_model, suffix=user_id)
         print(f"INFO: Fine tune job id: {fine_tune.id}")
         print("INFO: This may take a few minutes to hours depending on the size of the dataset and the selected base model")
         print("INFO: Use the `job status` command to check on the status of job process") 

discordai_modelizer/gen_dataset.py

@@ -4,16 +4,22 @@
 from datetime import timedelta
 from dateutil import parser
 from string import punctuation
+from os import path
 import pathlib
 
 def parse_logs(file: str, channel:str, user: str, thought_time=10, thought_max: int = None, thought_min=4):
     files_path = pathlib.Path(user_data_dir(appname="discordai"))
     dataset = open(files_path / f"{channel}_{user}_data_set.jsonl", 'w')
     thought_max = 999999 if not thought_max else thought_max
+    try:
+        username, user_id = user.split('#') 
+    except ValueError:
+        username, user_id = user, None
     with open(file, 'r', encoding='utf-8') as data_file:
         data = load(data_file)
         messages = [msg for msg in data['messages']
-                    if f"{msg['author']['name']}#{msg['author']['discriminator']}" == user]
+                    if f'''{msg['author']['name']}{f"#{msg['author']['discriminator']}" if user_id else ''}''' == 
+                    f"{username}{f'#{user_id}' if user_id else ''}"]
         thought = ''
         for i, msg in enumerate(messages):
             msg['content'] = sub(
@@ -46,6 +52,8 @@ def parse_logs(file: str, channel:str, user: str, thought_time=10, thought_max:
                             dumps(
                                 {'prompt': '', 'completion': f'{thought}'
                                  if thought[-1] == '.' else f'{thought}.'}) + "\n")
+    if path.getsize(files_path / f"{channel}_{user}_data_set.jsonl") == 0:
+        print("WARNING: The resulting dataset is empty. Please double check your parameters.")
     dataset.close()
 
 

discordai_modelizer/version.py

@@ -1 +1 @@
-__version__ = "1.2.2"
+__version__ = "2.0.0"

docker-compose.yaml

@@ -1,7 +1,7 @@
 # VERSION=$(cat discordai_modelizer/version.py | grep -oP '\d+\.\d+\.\d+') docker compose up --build
 # discordai_modelizer model list --simple
 # discordai_modelizer model create -d $DISCORD_TOKEN -c $CHANNEL_ID -u "$USERNAME"
-# discordai_modelizer model delete -m "text-babbage:001"
+# discordai_modelizer model delete -m "text-babbage-001"
 # discordai_modelizer job list --simple
 # discordai_modelizer job follow -j ft-V31oOgRGZaVZJvZNQFSvSRBl
 # discordai_modelizer job status -j ft-V31oOgRGZaVZJvZNQFSvSRBl --events