Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for not critical tests in the fbtest framework [QA598] #601

Open
firebird-automations opened this issue Feb 18, 2015 · 1 comment

Comments

@firebird-automations
Copy link

Submitted by: @dyemanov

Allow some tests to be specified as non-critical at the definition level (.fbt file), individually for different FB versions. In the case of actual vs expected mismatch they should be reported as warnings instead of failures and if there are no other critical failures the overall test run should be treated as "passed".

This would allow tests to be written early, as soon as some bug is registered. This is especially important for some well-known (including legacy) issues that remain alive for many years. They are not regressions, so they should not invalidate test runs. But automagically reminding developers about them would be a good thing.

I believe this is a much better approach than remembering what failures are actually OK for the given version.

@firebird-automations
Copy link
Author

Modified by: @dyemanov

assignee: Pavel Cisar [ pcisar ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants