Register    Login    Forum    Search    FAQ Awesomenauts



Post new topic Reply to topic

Author Message
 Post subject: Why IgnotisAI is Impossible
 Post Posted: Sun Jul 19, 2015 2:06 am 
Offline
User avatar

Joined: Wed Jul 30, 2014 3:15 pm
Posts: 1160
Location: 3 miles under in a text file
A month ago, I launched Awesomenauts to test a Leon bot. It was a patchwork bot designed to test IgnotisAI concepts. The bot was hastily written and full of bugs, but astonished me with his performance. You see, when LeonAI got his team's first three kills, I was excited. He was performing well. LeonAI went on to score six, nine, eleven kills in a row, then crown them all with the enemy base. That base was mine, and I had only lost a bot match in a few cases before. Even so, I found him trivial to deal with. He was buggy and easily exploitable.

Why, then, did LeonAI perform so well? I opened the debugger to check. Something quite unexpected was going on. Two of LeonAI's states interacted to lure enemy bots away from safety, then kill them before they had a chance to escape the wrath of my pet project's team. My own bot was actively exploiting the rest of the crop.

What did Leon have that the other bots did not? Was he more advanced? No, LeonAI had only 220 KB of code, dependencies and all. All other bots' code is measured in megabytes. Leon was doing nothing clever, either. His code had no functionality that GeneralAI bots did not. In fact, he didn't even have logic to retreat from dangerous 'Nauts. Rather, his advantage was that he had less logic than any other bot, and therefore a significantly smaller attack surface.

That's right, I said attack surface. For a while, it has occured to me that simpler bots tend to out-perform more complex ones, but the correlation only hit me just now. AI modding is actually very similar to computer systems security. Thousands of instances of bots are run each day, and players are actively attacking each one. Bugs are found and exploited. Up to now, no bot has made its way out unscathed.

It's clear we need to shift our focus. Instead of looking towards features, we should set our sights on security. Bots should be thoroughly tested from the ground up. They should be as simple as possible. Code should be compartmentalized and isolated. They should expose as little logic as possible. This paradigm of least-complexity rules out the use of a GeneralAI. By catering to everyone, even the simplest would add unnecessary complexity.

That means there can be no IgnotisAI. It also means that the best bots will be written from scratch. They will have only the functionality that is proven to improve them. Finally, they will be written defensively, to prevent exploits. Bot makers should know exactly how their bot functions and defend against any and all corner cases. They should isolate each state from the other, so no bugs spill over onto the rest of the AI, causing issues such as permanent stuck bugs. They should assume the worst in all cases. Finally, bots should randomize as much as possible to make them unpredictable, because unpredictable bots are significantly harder to fight.

There may well be a new paradigm on the horizon. Let's try defending against players rather than adding new features. Let's make bots that can't be broken, rather than ones that are excellent only when they perform well.

- Ignotis, overthinker, theorycrafter, botmaker

_________________
"If it works well the first time, something is really, really wrong."

Veanko wrote:
Ignotis the hedgehog confirmed


STEAM


Top 
 Profile  
 
 Post subject: Re: Why IgnotisAI is Impossible
 Post Posted: Sun Jul 19, 2015 7:45 pm 
Offline
User avatar

Joined: Mon Sep 30, 2013 12:13 am
Posts: 236
I feel like you might be missing a point here. Focusing on relative performance to other bots can only result in a zero sum arms race where the focus is on just what you describe: security over features.

I feel that bots should be about improving the player experience. The best bots are not those that beat other bots, but the ones that work best at temporarily filling a player role in online games. The optimal bot does not hog all of the solar and health creeps and does not focus on getting the most kills. Instead, I feel the optimal bot optimizes human players by whittling down the enemy players (thus setting up humans for kills) while avoiding dying (feeding), thus making the game for the human players on the same team more bearable while they wait for a human to replace the bot.

_________________
certified vaporware peddler


Top 
 Profile  
 
 Post subject: Re: Why IgnotisAI is Impossible
 Post Posted: Sun Jul 19, 2015 8:40 pm 
Offline
User avatar

Joined: Wed Jul 30, 2014 3:15 pm
Posts: 1160
Location: 3 miles under in a text file
chaorace wrote:
I feel like you might be missing a point here. Focusing on relative performance to other bots can only result in a zero sum arms race where the focus is on just what you describe: security over features.

I feel that bots should be about improving the player experience. The best bots are not those that beat other bots, but the ones that work best at temporarily filling a player role in online games. The optimal bot does not hog all of the solar and health creeps and does not focus on getting the most kills. Instead, I feel the optimal bot optimizes human players by whittling down the enemy players (thus setting up humans for kills) while avoiding dying (feeding), thus making the game for the human players on the same team more bearable while they wait for a human to replace the bot.


I agree that bots should be about fun. There's no doubt. My concern is not getting kills. It's about reducing deaths and stuck bugs. Take Super Sentry as an example. He can be a great support in bot matches and does a fine job setting up kills. But, he's a disaster in pub matches. Like other bots, you need to guard him to prevent deaths. He also occasionally gets stuck on Ribbit and Aiguillon, sometimes permanently. I'm willing to remove some of his more advanced features if it means I can protect him from that. I don't see performance as the most important function of a bot. I see consistency.

That being said, your post does have a very good point. Bots should improve their teammates' function. The most fun (and reliable!) bots would probably be the ones that stuck with their teammates, rather than those that split pushed all the time.

_________________
"If it works well the first time, something is really, really wrong."

Veanko wrote:
Ignotis the hedgehog confirmed


STEAM


Top 
 Profile  
 
 Post subject: Re: Why IgnotisAI is Impossible
 Post Posted: Sun Jul 19, 2015 10:24 pm 
Offline
User avatar

Joined: Mon Dec 31, 2012 7:03 pm
Posts: 3234
I assume you know of Occam's razor? Writing too much specialized bot code results in overfitting which in result makes the bot perform bad in situations he's not be "trained in" (=the Bot creator didn't think of). That's all more or less basic machine learning theory. :thumb:
And as soon as the bot creator starts "fixing" the bots bugs by adding more instead of fixing the existing code, you get a big mess.

Regardless, I still think you overemphasis the whole aspect of bots and AIs in Awesomenauts, but that's just me.

_________________
I respectfully disagree.


Top 
 Profile  
 
 Post subject: Re: Why IgnotisAI is Impossible
 Post Posted: Mon Jul 20, 2015 2:39 am 
Offline
User avatar

Joined: Sat Sep 07, 2013 5:56 pm
Posts: 539
Location: Somewhere in a Canadian laboratory
r0estir0bbe wrote:
And as soon as the bot creator starts "fixing" the bots bugs by adding more instead of fixing the existing code, you get a big mess.


VinAI... never again...

_________________
Myah!

Bots I've made:

[In-Game] Derpl viewtopic.php?f=23&t=35642 :drool:

http://imgur.com/5P0Ais3


Top 
 Profile  
 
 Post subject: Re: Why IgnotisAI is Impossible
 Post Posted: Mon Jul 20, 2015 4:39 pm 
Offline
User avatar

Joined: Tue Aug 07, 2012 1:43 am
Posts: 1495
I don't know the particular of bot AI since the time I spent on AI for this game doesn't pass the 60 minute mark. However I can tell you that while you said some things right, you made the wrong conclusions.

Test case.
Your test case is flawed. And unfortunately the "right" one requires human resources.
In your case, having your bot face other bots does not prove that your bot is better vs humans, it only proves that it's better vs other bots.

Most professionally written code has test units that test if specific sections of code are working properly. In AI this can still be done for small segments of code, but testing if the overall "product" is working is not possible. This is because the problem faced is similar to "is this level fun?". One can test if a specific section of the level is working properly, but one cannot tell if the whole level achieves it's entertainment target.

This does not mean that bots are untestable. There's ways to approximate it by dividing the game play into small test "situations" and measuring how the bot works in those situations. For example, you could have a situation where your bot is facing 2 enemy players pushing your turret and you can measure the efficiency of the bot by how much hp is left on the turret after the enemies retreat. The theory here is that you can create a bunch of these small situations with measurable output and at the end have a rating for each one.

Unfortunately I don't think we have the tools to create these. And bots are becoming less of a priority as the game evolves, so we're stuck with good old fashioned, design it and test in on humans.

IgnotisAI
That's some crazy jump on conclusions that you made, it's the first time I heard of someone that's given up on something because it was successful. Your AI exploiting the behavior of other AIs is only a good thing, and why would bots gets stuck because of IgnotisAI? I thought it was a framework that other bot creators can use to read the state of the game. Unless IgnotisAI is outputting wildly different outcomes every second I don't see how bots could get stuck. That would be a bug in IgnotisAI anyway.

In any case, let's say you create a bot from scratch, then you need to figure out a way to know what lanes to push, if you're gonna end up copy pasting trees from the last bot you/someone else made to figure out if they should push a lane, what's wrong with having it in a file that every bot includes? If IgnotisAI doesn't work for specific bots because it causes them to do crazy stuff the solution is simple, fork it and fix it. Unless the use case is wildly different, it's always easier to adapt existing code to your use case than to create something for scratch.


Top 
 Profile  
 
 Post subject: Re: Why IgnotisAI is Impossible
 Post Posted: Tue Jul 21, 2015 6:39 am 
Offline
User avatar

Joined: Thu Aug 14, 2014 10:25 am
Posts: 3689
Your Leon-bot sounds like it outperformed every other bot, even though you personally found it simple to exploit, which is why I was also confused that you were treating it as a disappointment.

Your points are quite interesting, though. Complexity can unnecessarily obfuscate, if it is not structured well. However, a well-written general AI is capable of preventing the indecision you described, unless a writer (un)intentionally overrides it. Documentation (or an understanding of the general AI) would be useful in that case.

Bots should take the best course of action in a given scenario. Perhaps that will make them more predictable, but if it yields the best result, I do not see why they should be subject to randomization.

I'm still not sure why you're dismissing even the possibility of IgnotisAI, when your progress-screenshots and posts seemed to indicate that it was proceeding well.

_________________
lorem ipsum dolor sit amet


Top 
 Profile  
 
 Post subject: Re: Why IgnotisAI is Impossible
 Post Posted: Tue Jul 21, 2015 11:53 am 
Offline
User avatar

Joined: Wed Jul 30, 2014 3:15 pm
Posts: 1160
Location: 3 miles under in a text file
The bot was successful in that it was good. However, it also proved to me that further developing it into a GenaralAI would be harmful to the bot. IgnotitisAI was designed to be a GeneralAI with a very high strength ceiling. However, I came to realize that just using a GeneralAI added too much complexity to a bot for it to be bug-free. In other words, the bot was successful because it was not generalized, but was unsuccessful versus players because it was designed hastily, rather than carefully. LeonAI performed well in that case because he was built to be Leon, not yet an everyman.

That being said, a GeneralAI is not necessary to make bots better or botmaking more convenient. The purpose of a GeneralAI is to keep everyone from rethinking the wheel. However, it also stifles certain innovations. Instead of that, we could use a set of protocols. A botmaker might run across a hard problem, then realize it has already been solved. Likewise, such protocols may offer ways to improve a bot, such as stuck-free movement logic.

A GeneralAI that is both good and convenient to use has turned out to be a failure. But, it brought new concepts to light which may be of use to botmaking. My current endeavor is a GeneralAI-less, waypoint-less (unfortunately, waypoints are a big source of bugs), finite-state bot. Perhaps that will bring real success.

_________________
"If it works well the first time, something is really, really wrong."

Veanko wrote:
Ignotis the hedgehog confirmed


STEAM


Top 
 Profile  
 
 Post subject: Re: Why IgnotisAI is Impossible
 Post Posted: Wed Jul 22, 2015 11:23 pm 
Offline
User avatar

Joined: Mon Sep 30, 2013 12:13 am
Posts: 236
Ignotis wrote:
waypoint-less


Down that way lies madness.

I've tried it, with some real success, but it's an inhuman grind to do. You might have something amazing and incredible in mind, but please, for the sake of your sanity, don't just try to bruteforce it

_________________
certified vaporware peddler


Top 
 Profile  
 
 Post subject: Re: Why IgnotisAI is Impossible
 Post Posted: Wed Jul 22, 2015 11:30 pm 
Offline
User avatar

Joined: Mon Sep 02, 2013 5:16 pm
Posts: 1605
Location: Dænmårk
Inspired by this thread, I set myself to create a GeneralAI, and I'm actually very close to having a product that could work.

It's focused on:
- Not dying
- Killing droids
- Staying with team
- Not facetanking


Top 
 Profile  
 
Display posts from previous:  Sort by  
 
Post new topic Reply to topic