Register    Login    Forum    Search    FAQ Awesomenauts



Post new topic Reply to topic Go to page Previous  1 ... 5, 6, 7, 8, 9, 10, 11, 12  Next

Author Message
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Sun Oct 08, 2017 9:49 am 
Offline
User avatar

Joined: Thu Aug 31, 2017 11:42 pm
Posts: 40
Duskitty wrote:
Testing a game takes longer than you would think. Testing one upgrade doesn't take that long, yeah, but all the time spent testing adds up in the end, and there's a lot of upgrades in the game.
As I said before, Ronimo doesn't have a lot of time to do everything themselves, and as I also said before, the QA company is hired to test and find bugs for them. Except they don't even do the job they're supposed to do, and as such, patches have a ton of issues.

but what about when we report these bugs to them and they still go into the live game?

_________________
Interests: cute gf who likes videogames


Top 
 Profile  
 
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Sun Oct 08, 2017 2:25 pm 
Offline
User avatar

Joined: Tue Dec 09, 2014 7:35 pm
Posts: 803
Location: Grenoble, France
Duskitty wrote:
Testing a game takes longer than you would think. Testing one upgrade doesn't take that long, yeah, but all the time spent testing adds up in the end, and there's a lot of upgrades in the game.
As I said before, Ronimo doesn't have a lot of time to do everything themselves, and as I also said before, the QA company is hired to test and find bugs for them. Except they don't even do the job they're supposed to do, and as such, patches have a ton of issues.

You must be aware that as developers, they must have ways to test the game more efficiently than us. I would imagine some kind of debug menu where they can spawn any character they want, with any upgrade, without having to restart a game for every character. Along with a dummy target that shows the exact damage being dealt, the amount of time required to test (i mean very basic testing - cost + damage improvement) would be minimal. This is something they can do as developers.

_________________
Admin of the Rollcage steam group
GRIP - Intense futuristic combat racer


Top 
 Profile  
 
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Sun Oct 08, 2017 2:45 pm 
Offline
User avatar

Joined: Sun Jul 14, 2013 11:33 am
Posts: 59
There are many approaches to improving the quality control within Ronimo, without having to rely on an external company, who will, no doubt, not be able to test every single combination of characters, upgrades, modes, system environments, etc., especially when their only view of the game is a "black box" (not knowing about the internals of a software system makes it rather tricky to identify probable problem spots).

One of those approaches is automated testing at code unit level, trying to run as much code as possible in a controlled test environment and comparing the actual state during/after the test run to a predefined "expected" state. It is entirely possible to automate software testing, even for a project as large as Awesomenauts with loosely-defined, frequently changing requirements. The benefit would be that any refactoring or other code change that breaks things will immediately trigger an alert from the unit testing system, indicating that something is wrong (at least, if test cases are consistently run and/or continuous integration is used).

However, the developers at Ronimo have decided against this type of testing:

Joost's Dev Blog wrote:
One thing that's surprisingly missing in our Coding Methodology is unit testing. We have a strong focus on testing our own code extensively, but the document doesn't say you need to write unit tests. This is because I think gameplay is often too chaotic and unpredictable to test well with unit tests. Certain things are testable with unit tests, but the bugs we encounter are often not things where I can imagine how a unit test would have found them. Often it's things that function fine but result in undesired gameplay.

[...]

I do think we ought to use unit tests more for things like server architecture. Unit testing isn't in our blood at all and it probably should be at least a little bit.

(source)


Not wanting to write unit tests for gameplay features is understandable, as the development costs for covering much of the codebase with tests is presumably outweigh the benefits gained from being able to identify certain types of technical regressions more easily. Automatically testing the actual gameplay itself is not trivial, but certainly possible and practical, as other game developers have demonstrated. Of course, this is much easier when the game's codebase is designed and written from the ground up to be tested, and the time to make the Awesomenauts client auto-testable within reasonable development costs (beyond simulating random user input to detect hard crashes) has probably passed, as the game client has long grown beyond any manageable bounds for an engine-wide implementation of unit tests.

Nevertheless, a few very basic gameplay-oriented unit tests for upgrades to ensure that their core functionality is intact (such as "Does upgrade SpyBoothClone1 cause a CreepSpy to spawn upon secondary skill use?") may have helped prevent some of the regressions introduced in 4.4 for a manageable amount of development time.

The server architecture for Awesomenauts, however, should have much tighter constraints for correctness than the client, in addition to having less frequently changing requirements. This would make unit tests much more straightforward and less costly to write and maintain. Unit tests for the game server could possibly have helped prevent issues like the game coordinator incorrectly detecting won matches as 'abandoned', or locking players into an infinite loop for choosing a random character.


An orthogonal approach to improving code quality is the inclusion of a mandatory "code review" step before additions, changes or removals to any section of the codebase (client-side C++, server-side C#, .settings files, AI XMLs, etc.) are allowed to be merged into the main branch. Seeing newly written code from a different perspective can often help with discovering unintended interactions with other parts of the codebase that might result in bugs.

For instance, if a designer adjusts a character setting to make balance changes (e.g. reworking an upgrade or implementing a new mechanic), but unintentionally triggers a previously unhandled edge case in the engine code, a bug could sneak into the release if the manual testing carried out is not sufficiently rigorous (or outright missing). Letting a developer who has worked on the part of the engine that handles settings at least skim over the changes could give them a chance to catch the bug before it makes it into the release.


I don't know how consistently these principles are applied within Ronimo, or if they are even used at all. Some time has passed since some of the blog posts I quoted were written, and no positive or negative mention regarding code reviews has been made anywhere as far as I was able to tell. However, as much as the balance changes and the new game mode are appreciated, 4.4 has shown rather gaping technical oversights that were discovered the community within hours after its release.

Lastly, it appears that quite a few of the client-side bugs were introduced as a result of changes made between the last beta and 4.4's live version. Perhaps this update could have benefitted from a "code-freeze" phase, where an additional week would be dedicated to extensive testing of the final release candidate, without allowing for any further last-minute changes to be made that are not strictly bugfixes.


Top 
 Profile  
 
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Sun Oct 08, 2017 4:29 pm 
Offline
User avatar

Joined: Mon Nov 25, 2013 6:41 pm
Posts: 804
From my own experience (9 years of game dev & counting), I agree with Joost.

Unit tests for game development are difficult to do right and a waste of time. They make tons of sense for "real world" programming where limited user input is accepted. If you actually want to go all in on unit tests for a game, you're looking at adding an additional programmer or two because the amount of added work it creates is non trivial - code bases for games are huge with lots of moving parts.

The bugs from the recent patch are pretty indicative of a QA fail, either at the time of implementation (not testing yr *!), or actual testers not catching bugs (poor test plan/limited game specific knowledge). A lot of devs ive worked with barely test their own stuff. They'd rather throw it over the fence for QA to find bugs, confident they've probably done the right work. Not saying that's what happened here, but maybe. Bugs happen.

I do agree with the frustration about found beta bugs not being addressed. The typical reason to do a beta is to weed out bugs. But it seems like they're mostly used here to spin up hype about new characters


Top 
 Profile  
 
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Sun Oct 08, 2017 5:01 pm 
Offline
User avatar

Joined: Sun Jul 14, 2013 11:33 am
Posts: 59
Indeed, it would not make much sense to implement engine-wide test cases for Awesomenauts' game client, especially this late in its development.

While I don't entirely agree that unit tests are a waste of time for game clients, I'm primarily suggesting the use of unit tests to ensure the correctness of the game server. Joost admitted that Ronimo's server architecture could use some unit tests, and I brought up examples of issues that arose in 4.4 which could (presumably) have been prevented through automated testing of the server-side code, which meets your definition of a "real world" system, accepting only very limited user input.

It appears that the reason unit tests currently are not used at Ronimo is based on a lack of momentum ("Unit testing isn't in our blood at all"), suggesting that the developers are simply not used to writing test cases for their code.


Top 
 Profile  
 
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Sun Oct 08, 2017 5:12 pm 
Offline
User avatar

Joined: Mon Nov 25, 2013 6:41 pm
Posts: 804
Yeah unit tests make a lot of sense for server stuff. Sorry if I jumped to conclusions there. Most of your post seemed to be about game client unit tests which I have beef with :)


Top 
 Profile  
 
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Sun Oct 08, 2017 10:07 pm 
Offline

Joined: Tue Aug 18, 2015 3:50 am
Posts: 1402
yall should hire marukyu lol

_________________
Mains: :thumb: :derp: :worship: :think: :facepalm: :shady: :rocco: :jimmy:
steamcommunity.com/id/MrPillow92/
stream: twitch.tv/mrpillowthegreat
Awesomenauts Rumble Tournament: http://steamcommunity.com/groups/AwesomenautsRumble


Top 
 Profile  
 
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Mon Oct 09, 2017 1:30 pm 
Offline
User avatar

Joined: Thu Nov 01, 2012 5:29 am
Posts: 1102
Location: Kitchen
Duskitty wrote:
Badass Ketchup wrote:
If I * up with my game, I deserve and accept all the * coming my way from my playerbase, and I actually communicate with them instead of going into full damage control mode. Ronimo is a small-medium sized company, where as it's a 1-2 man operation with myself, there's no excuse for this patch.

I guess it could be the fact that I actually test my changes and fix bugs reported to me.

My point was not that the hate directed at Ronimo is undeserved; the point is simply that everyone working at Ronimo is a human being, and as such, they have emotions. They aren't robots simply programmed to do their job and nothing more, and seeing all this backlash and harsh, abrasive language directed at them indeed makes them feel discouraged, bitter, upset even.

Would you not feel any sort of negative emotion, seeing someone make an angry, harsh review of your game and the work you put into it? It may not seem like it to you, but Ronimo does put effort and time into their game, and I presume it must seem like a slap in the face to see all this negativity.
Surely, in their place, you would feel the same. Humans are not immune to their own emotions like that.

I have had many negative responses to my game such as its repetitive nature and difficulty (both too hard and too easy depending on the type of player). I take pride in what I do for a living and strive to improve upon it by taking those negative responses and figuring out ways to elevate them. Emotionally it depends it depends on the topic, on game mechanics I tend to be more emotionally attached and stubborn about changing them compared to anything else which I don't really get too much, if any, negative emotions from them. If they don't like the style of game there's not much I can do about that. If they say a gun is bad, I check the gun out myself and get other people's feedback and change it if necessary.

But ultimately no, I don't typically get this level of backlash from any of my updates I make to my game. In fact, I don't recall any large amount of negativity to any update I've ever done ever, could maybe be the fact that I make sure my updates actually work, have a good understanding of my own game, and get feedback from the community when needed. Just some food for thought. But if I hypothetically did get a large amount of criticism from an update, it would motivate me to resolve it ASAP rather than make me discouraged.

Duskitty wrote:
Testing a game takes longer than you would think. Testing one upgrade doesn't take that long, yeah, but all the time spent testing adds up in the end, and there's a lot of upgrades in the game.
As I said before, Ronimo doesn't have a lot of time to do everything themselves, and as I also said before, the QA company is hired to test and find bugs for them. Except they don't even do the job they're supposed to do, and as such, patches have a ton of issues.

I don't think you seem to understand that upgrades don't change if you don't change them yourself. There's no magic fairy that comes in the middle of the night and messes with all their settings files. I'm currently managing 67 different weapons, 51 perks and 374 different upgrades in my game without any bugs or issues involved. Normally I charge a lot for my secrets but I'm feeling a bit more generous than usual today so I'll let you in for free.

I TEST MY CHANGES AFTER I CHANGE STUFF TO MAKE SURE MY CHANGES WORK

It's extremely easy to test changes with their setup, a lot more easier than with my game, as they can change and test pretty much everything WHILE playing the game. I've even had the great pleasure of experiencing their development tools through ASM, which is more or less a lite version of their own settings manager. Which brings me to your final point about people not doing their jobs. While this may be a highly subjective opinion of mine, as I'm a very strong believer in people doing what they're being paid to do, as you, yourself seem to be on board with on the topic of the QA testers. How would it NOT be the design team's job to ensure a change is, at the very least, functional? I'm not even getting into the land of the change breaking under certain conditions, I'm just talking about just a single test to ensure they are fulfilling their job description. Oh dear, seems like the problem is one step above the QA testers (Which are still god awful and I've got absolutely no idea why Ronimo would even pay them for anything (at least PC related). I personally rarely hear anything good about any paid QA tester company, but each to their own I guess).

_________________

2 Balance Rules

1) No upgrade should be mandatory
2) No upgrade should outclass or be overly reliant on another upgrade

My steam
Leader of Team HDTV

Jasper wrote:
Well Lashes and Sunballs are about as strong


Top 
 Profile  
 
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Mon Oct 09, 2017 1:38 pm 
Online
User avatar

Joined: Wed Mar 06, 2013 12:40 pm
Posts: 1722
Quote:
There's no magic fairy that comes in the middle of the night and messes with all their settings files.


Let me introduce you to my beloved friend, Ribombee:
Image

_________________
Whether it is sound or sight, true beauty resides in curiousity.

~ Embrace your dreams.


Top 
 Profile  
 
 Post subject: Re: NOW LIVE: Update 4.4: Cirque du Brawlee Hotfix 1
 Post Posted: Mon Oct 09, 2017 2:01 pm 
Offline
User avatar

Joined: Wed Dec 02, 2015 10:35 pm
Posts: 177
A lot of developers here who think they know better than ronimo how to test their game.
They might be right, but as a (non-game) developer myself, I know how you might think a change won't cause any bugs but it ends up being critical. And they're a relatively small company, try to cut them some slack.
They probably got developers working on Nauts right now that weren't involved with the game at the start so they don't have all the required knowledge to be able to tell 'if I change feature X, I will break feature Y'. And we all know that documentation never gets properly done, we all hate it :tongue:

Anyway, I'm hoping for a quick second hotfix. (hopefully later today)
Yoolip without laboratory centrifuge = yoolip hard mode. Damn, I never realised how much I relied upon that upgrade.


Top 
 Profile  
 
Display posts from previous:  Sort by  
 
Post new topic Reply to topic Go to page Previous  1 ... 5, 6, 7, 8, 9, 10, 11, 12  Next