Divine Automated In-Game Tickets

[Instadev]

Alrighty, got enough stuff to post another update.
THERE ARE 2 SYSTEMS IN PLACE AT THE MOMENT. The first one is bound to a dataset with the ability to learn through a command that specifically determines what whole sentence belongs to which category. WITHOUT THE DATASET - THE SYSTEM WILL NOT WORK.
You can look at my previous post (the image one) and see what I'm referring to. You can also look at the MySQL table structure in my previous posts throughout this thread.

The one I'm going to release (for premiums only) is an active-learning one, which includes a command to teach and reload the AI's dataset, which is basically a MySQL table that contains all the "accurate" stuff you guys can utilize.

The best part of this system is the fact it can be used with literally any language, so if you have an English-speaking server, a Spanish-speaking server, or a Chinese-speaking server, it won't matter. As long as your dataset fits, it's good to go.



It is important to note: action-taking expansion will come later on. The initial release is for categorization only (and for inspection & review of course). The system's only limitation is how expanded your dataset is.

There is no ETA on when I can release this as this is a side project at the moment and will take as many polishes as it needs before an official release.

The passive-learning system (as in, self-learning) is still under development (way more work over there and a lot more to go in comparison to the active one) but in theory, it will be based on the exact same dataset (plus a few more tables for better accuracy) and it will be able to learn by itself what categories to add over time and how to handle (and give an appropriate score) for each one. It's an extremely complicated process that requires a lot of time, work, and imports, as well as a huge dataset to work on.
So, in order to process it faster, I ask for your assistance.

As I said, I will share the active system with you guys but everyone who gets a copy please share your dataset so I can progress faster with the passive system. We will need all sorts of variations, from perfectly grammared sentences to messy ones of non-native English speakers. A slight change of word here and there and so forth. These are all super important and will require a lot of eyes to look after. The more - the better.

 

[splicho]

Absolutely amazing, love it.

 

[RedLine]

Not sure if anyone mentioned this since im on Mobil rn and couldn't see all the replies, but in game ticket is not something that people use these days! Almost all server have discord server and use a ticket bots since its 100 time easier to track the conversation than in game.

 

[Instadev]

That's definitely something that the community needs, however I'm wondering if it would be possible to merge this kinda system into Discord for a quicker response time and a more comfortable environment to answer tickets...

Regardless it seems like a good idea.

Definitely possible, and it was actually something I was thinking about myself while building our new ticketing system for Discord.

Begs the question, should this be integrated into an en Emulator directly, or be a standalone API with a library so it can be integrated easier, as if it's a standalone API more robust ticketing systems could be built too, (cross site, such as Discord, Ingame and forum could all be coupled into a single ticketing system)

The concept is to have answers provided instantly, not to have an extended gm capabilities regarding support.
Assuming you take that instant reply concept and perfect it upon multiple major fields, you eliminate the need to integrate it via Discord. This is exactly why everyone's support is needed as there are a lot of commonly asked issues and every server can have its own utilization of this service. The more data the more accurate this system becomes, and the ability to sync the data for a specific realm could also exist, so you can just select the type of realm you're having and go wild.

I could look into creating an in-game API for Discord at a later stage and for other topics.

Not sure if anyone mentioned this since im on Mobil rn and couldn't see all the replies, but in game ticket is not something that people use these days! Almost all server have discord server and use a ticket bots since its 100 time easier to track the conversation than in game.

That is correct, and don't forget the fact people also have the ability to create a ticket on a website as well. And still, nothing has come even close to implementing what this is meant to accomplish long-term: full automation.
The point of this system is not to have direct communication with players but to provide them with an answer instantly based on predefined answers you set up that match your specific server. DO NOT wait for a GM to get online (even from your mobile) and help you out, let a trained bot do this for you and truly provide 24/7 support (plus automated tasks per demand).

Yes, integrating a special API to be based on Discord is something I may consider doing in the future, but it is not as good as using a ready system in-game and making it actually useful for your specific needs.

I make this system with one idea in mind: not to hire a ridiculous number of staff members to try and cover 24/7 support. Those who are interested in this are welcome to give it a go. Those who prefer other ways are welcome to stick to their beliefs.

I hope this clarifies things a bit.

 

[Instadev]

After 5 intense weeks from the initial thread, I finally have something to update you guys.

I've successfully implemented a whole neural network into this classification system, which means we will soon have machine learning within TrinityCore. I've exported the project into a console application, and it's not officially within TrinityCore YET! From user input (console text), it can certainly detect the proper category (assuming our data is accurate enough) and provide a realistic prediction score (on a scale of 0 to 1, which will be multiplied by 100 for a visual "percentage" feel). However, it is, like with every neural network, prone to overfit or underfit, which basically means: "if I put 100 training rows related to VIPs and 3 for regular players, it might consider all players as VIPs": overfit for VIPs, underfit for regular players.
The system works in such a way that you give the network a list of pre-defined sentences ("tickets"), and a predefined category for the ticket (in numbers, which I will provide later on), and from there the system does the rest.
From this point onwards, with every new ticket, you will be able to see the predicted category and the certainty score.

What do we have at the moment?
1. A functional text classification & scoring system
2. The ability to save a pre-trained model & save a new one automatically
3. The ability to fetch data from the database so the integration will be smooth
4. Predefined categories & some sampling data

What can we expect from the system?
You can expect the system to be an actual, real, fully functional artificial intelligence system, which will have the ability to save a significant amount of time by integrating a sort of Frequently Asked Questions system into your own server, with the ability to take automated actions based on certainty.
It will come with comfy commands, to teach the model, predict a certain category (out of a given "test" ticket without actually creating a ticket), delete a certain pre-trained row (index number), and to re-train the entire model from 0.
The model at the moment is in extremely early stages and does not even support multi-threading, which makes the training process long & tedious.

Examples of usage
Example #1
New ticket: "Hey GM, I'm not sure who here can help me but I bought <Epic Sword of Legends> on the website and the item hasn't arrived yet. Care to take a look?"
Automated system: classification -> Failed Mailing Delivery
Automated system: check classification: 89% certainty
Automated system: [if permitted by the owner to take automatic action]: reply: "Hello <PlayerName>, I have viewed your ticket about a failed item delivery. I'm checking our logs right now, please stand by!"
Automated system: <literally checking the logs for last purchases from <PlayerGuid: {GUID}>
Automated system: <if found & data confirmed>: initiate .additem function, document the process in its own logs table in the database, message the player which item was found, disallow additional tickets of this sort based on the existing logs to prevent duplication & repetition
Automated system: <if not found or data mismatch>: "I was unable to verify your request with our logs. Please keep this ticket open and a Game Master will contact you ASAP!"

Example #2
New ticket: "yo gm why is my alt ACC banned? wtf is this I haven't done nothing unban me!!!!"
Automated system: classification -> unjustified ban
Automated system: check classification: 78% certainty
Automated system: [if permitted by the owner to take automatic action]: reply: "Hello <PlayerName>, I have viewed your ticket about the unjustified ban. I'm reviewing the data I have right now, please stand by!"
Automated system: <checking the current IP and matching accounts bound to this IP. If found: check ban history and cache it>
Automated system: <if found & data confirmed>: reply: "I've found X accounts from your existing IP. <Account #1: {AccountName} is banned for {duration} (unban date at {unban_date}), by {banning_gm} for {ban_reason}. If you wish to appeal or shorten your ban, please visit us at: discord/website link>
Automated system: <if no data found>: reply: "I'm sorry, I have found no accounts linked to the IP you are connected from. Sadly, I cannot provide details about another account that I cannot know for sure is yours, therefore the ticket has been assigned to a GM."
It is important to say: the system can have an extension where it logs all the logins on the server, thereby allowing the system to have access to more details a normal GM will be too lazy to check or will not think twice before taking action if given enough power, hereby generating a slight chance for misplaced action

Future plans
1. The system will be multi-threaded, so the training stage will not consume a significant amount of time even when using it
2. Automatic actions: upon reaching a certain certainty score, the system will know automatically which reply to generate, which action to take, and what else needs to be done to automatically close the ticket, give the player what he asked for, and all without GM interaction at any stage.
3. For every X amount of new sentences implemented into the training data (which will be supervised by an administrator), the system will re-train itself in a different thread (and with it, generate new cached data to pre-load from later on).
4. The ability to obtain new categories by itself, thereby making it truly self-sufficient (for example, if we missed a few critical categories, such as "VIP Activation", or "Mailbox Issues", or basically whatever) - the system will know how to define them as their own categories based on the context of the sentence over time if requested a lot through support tickets, as it will save the data into its own database and will use frequently used keywords.
5. Good AI vs bad AI - feedback for improvement and further self-learning with positive or negative reinforcement from the players themselves.
6. Multiple decent-scored predictions handling: allow the system to work according to all of its predicted categories and certainty scores, not just the top one. For example mailbox delivery issues & VIP activation.
7. Multiple extensions of the system, from written text recognition to more user-friendly in-game bot support NPC where you can literally talk via /say and have the same outcome as if done through a ticket or a GM
8. Text trimming & slang implementation without the need for dataset: "u" = you, "m8" = mate, detect slang when used
9. Template integration: popular database tables will be used to detect an item when mentioned partially or in full, such as "Shadowmourne" which will let the system the ability to load the item into its ticket memory and utilize it in its loading and its replies. Same for quests and loot responses, for example: "hey gm what are the rates for gold if I kill guards in orgrimmar?", or "gm what do I get if I kill <worldboss #1>"?

Last but not least: when
When it's ready, I will post a video of the system in-depth and will explain every detail about it. No ETA. I will update here when I make progress with the system obviously.

A short FAQ:
- Yes, I am thinking of Discord integration for this system
- Yes, I am going to release a version of this
- Yes, I am going to maintain this
- No, I do not think it's useless, given a few rather simple examples of what it is capable of
- No, I do not think it will replace or remove the need for GMs at all, but it will shrink it dramatically enough that the main focus should be development, not support
- Yes, I do think this system will help every server and can only improve over time
- No, the system will not be rewritten for AzerothCore anytime soon, other emulators are not even being considered at the moment
- Yes, WoWEmu premiums will get access to a version of this system sooner & for free


Feel free to ask me questions and give me feedback on what you think about this system. I do not have a video at the moment and it will take me a while to complete everything I have planned, but like with every major system - the hardest part is the initial implementation of the foundation, the rest is, while difficult, still less difficult than the initial implementation with thorough testing.

 

[JadaDev]

Such an amazing work and i love it's progress, the whole idea is amazing and i do really love that you'll publishing it on wowemu for premium accounts.
I'll be first to put this on my server hopefully it'll work as intended!
Cheers @Instadev Great Work my friend!

 

[Tommy]

Looking good!

I do have a few questions:
Are you using a regular expressions library for the content data?
Does the AI take into account missing punctuation but accounts for the first X words for the content to become a statement, question, etc. to further the probability of it assigning the ticket for further action?
Are you taking into account typos and other potential data that might lead to failure within the AI? E.g. (You could possibly select the data entry based on the closest wording content) Another example: "Hep my account was baned" and it would select "Help my account was banned" based on the closest entry to that potential question, etc.

 

[Instadev]

Looking good!

I do have a few questions:
Are you using a regular expressions library for the content data?
Does the AI take into account missing punctuation but accounts for the first X words for the content to become a statement, question, etc. to further the probability of it assigning the ticket for further action?
Are you taking into account typos and other potential data that might lead to failure within the AI? E.g. (You could possibly select the data entry based on the closest wording content) Another example: "Hep my account was baned" and it would select "Help my account was banned" based on the closest entry to that potential question, etc.

At the moment the model is very "primitive" and does not cover any missing punctuations and/or typos of any sort. However, having a neural network makes me want to believe it will work just fine based on context and similarities, even if we "risk" a lower score. For now of course, in the future, it will be covered more thoroughly if needed.
However it might be worth noting, the model is very dynamic and the dataset can be altered at basically any moment, so when you notice something being misspelled regularly you might want to add that to the training data, and just for the sake of it, try and put all the ideas of "how to mess this question up" by default within the training data. If you use this method for all the training sets, it is less likely to overfit or underfit anything as it will be done equally across your entire dataset.
Goes without saying, I'm still testing things out and I will update here as soon as I find solutions to multiple problems will come over time.

For example, I was able to get a 88% score with 1 letter missing, and a 90% score with no letters missing. That gives me the estimate it's gonna be alright, even though it is important to note: I am working with a rather small dataset.

As for the libraries, I only use some of the default libraries, nothing specially imported from outside and nothing that requires modification to the stock TrinityCore sourcecode, except for the tickets themselves of course.

Currently tweaking the hidden layer's number of neurons to really test this out with what I currently have, which is 15 categories and a dataset of ~160 sentences so far.