SOBAC Wiki - User contributions [en]

Software Testing Additional Notes

2019-05-03T02:29:00Z

NicholasCollins:

== A lot of people end up in software testing via an oblique path. ==
*Possibly from a technical background, working as developers
*Possibly a business background, get to know a company's products
*You can take courses on it, but often as part of a larger program, rather than the main major
*Will that change in the future?
*Result could be why there is variety in terminology

== Different project managers take different approaches. ==
*Some sit in on code reviews, test plan reviews, dig in to details
*Others get their team to report to them at a summary level
*Project manager will more directly intervene when a high priority situation arises, i.e. testing falling behind schedule, often a good project manager is someone who can keep an eye on testing at a high level and quickly assess an important situation in detail when necessary

== Developers are often expected to do at least some testing of their own ==

*For some technical projects like software upgrades, developers might be recruited to work as testers
*Developers often do unit testing, sometimes document test plans ahead of time
*That unit test plan and results are given to testers when they do additional testing - integration testing, regression testing, load testing

== For testers, one challenge is the need to be thorough and deliver on time ==
*Sometimes, the first, fairly simple tests reveal a number of bugs; this creates concern about staying on schedule
*Advantage is that at least those bugs were found early
*It can be good to push hard, maybe put in overtime to get through the first pass of testing, get an idea of where bugs are, start fixing them early. This can avoid worse time crunch near the end of a project. Sometimes that can still happen anyway!

== Writing test plans can take a significant amount of time ==
*Does the project schedule allow for that?
*How much time will it save alter, if there is a test plan that is thorough and has been peer reviewed and edited? Often this makes it worth the effort
*In some situations, time constraints don't allow it, have to jump in to testing with minimal planning or documenting
*Some projects end up as a mixture. A detailed test plan is written and carefully peer reviewed early in the project
*During the project, unexpected problems and changes in requirements lead to many changes to the code and test plan
*At that point, there is limited time left in the project schedule, test plans are rough, possibly not peer reviewed
*It then helps if you have experience, when you're new to the position it's harder to "improvise" testing

== Reporting bugs to developers requires tact ==
*People have worked very hard on the code; you are finding problems with it
*You are all on the same team with the goal of delivering a good quality product
*Keep e-mails and bug reports in business style; just describe what happens, provide the steps to re-create the error
*When reporting bugs, avoid the use of pronouns. Say, "When I do x, the system does y", never say "Your code has a problem"
*But, do use pronouns when giving thanks for good work, acknowledge others in meetings, "Yes, this was a tough bug, thankfully Jon got it fixed yesterday."
*Einstein is believed to have said, "Make everything as simple as possible but no simpler". May not have been the exact words
*When doing bug reports, get to the main point quickly, but include the technical details necessary. Make sure YOU can reliably recreate the error following the same steps

Calculator

Basic format:
Steps to Re-create:

Press "pi"

Press Clear

Press "1"

Press "+"

Press "1"

Press "="

Expected Results:

Display shows "2"

Actual Results

Display shows "2.00000001"

System should be declaring input and response as integers, possibly using float or another type and not doing garbage collection from previous operation?

*Being very explicit about what was pressed, in what order, matters sometimes - people have habits. On a keyboard, some use the number pad, others the main keyboard. If someone turns num lock on or off during bootup as a habit and someone else was on the computer, you could get different results.
*When describing a sequence of menu choices, the "->" characters are helpful, i.e. select Main Menu->Submenu1->SubSubmenu1
*Making comments, suggestions does not hurt if it only takes a short time. If it's similar to another known bug, point that out, too.
*Does the system have different states? What state was it in when the test was done?
* Sounds simple but can be tricky, can be easy to forget an important detail - in a web-based system, which browser were you using? Sometimes you switch between Edge, FireFox and Chrome throughout the day, remember which one you found a bug in, check and see if it happens in other browsers.
* https://obsproject.com/ - software that can be used to capture videos, including drop menus. Windows 10's Game Capture does not include drop menus.

== Other tasks follow testing ==
*Technical writers have to produce documentation like manuals. They may want to look at some of the existing documentation. That may not be the test plan, it could be developer notes or requirements. Sometimes the tester has worked with those documents a lot, and is asked to assemble them into one place for the technical writer, or directing those people where to find things.
*The technical writer might want to also be a tester. In order to write documentation that explains things to the user, the writer has to understand it well, getting their hands on a test system and trying some things helps them do this.
*The technical writer will have questions for the tester, may want assistance in configuring a system
*Sometimes testing was done to cover many situations, but the first sales are to customers with specific needs, the technical writer will be focused on that, configuring a system that way
*It helps to label things, the tester might know about the system configuration but the technical writer doesn't.
*Sometimes the technical writer has industry experience, ends up trying some additional tests and finds bugs. Then the tester follows up to write up the bug and get it fixed.

== Some professionals actively dislike textbooks ==
*Some textbooks for software testing, as is the case for books in other technical fields, have a tendency towards "perfect worldism"
*They will describe techniques and show examples of using those techniques that are very time consuming to do.
*They might be good techniques that will capture lots of bugs if you apply them
*On a real project, time is limited, you have to make judgment calls about prioritizing things to meet deadlines
*Don't want to tell your manager you spent hours on this marvellous documentation and planning technique but got no actual tests done
*So why pay attention to text books?
*Even if you can't completely apply the techniques they describe exactly as outlined, you might get ideas you can partially apply that do help raise the quality of what you deliver
*Such as - Different types of program maps - even a partial one might get you thinking about test cases you want to do, maybe apply these techniques in depth for especially important or complex parts of a system if not everything
*You may adapt/adjust these techniques to suit your situation
*Can also give you the idea of the number of test there are in theory, including paths

== On YouTube, James Bach has some interesting talks ==
*https://www.youtube.com/watch?v=ILkT_HV9DVU
*Asks people how they would test things, pushes hard for people to explain why
*Sometimes testers have ideas/intuitions that are on the right track, can you explain why?
*He talks about a system that will work 100 - 250 VAC
*So why test, say, 90 V? It's not in the specs, we know it won't work
*But will it "fail gracefully"?
*(There is not always time for that in the workplace, but if you just follow your intuition you might find important bugs)

== Bach also goes through an example and asks how many test cases would you do? ==
*Raises questions about what the project does
*Decision testing, boundary testing, predicate testing
*With more experience, you can often think of more tests right off, i.e. "improvise"
*Emphasizes the fact that a flowchart, document, etc. is a representation, not the actual system

== When do you feel pride/satisfaction in your work? ==
*Some testers say they "high-five" each other when they find a really nasty, obscure, or complicated bug
*That is when they feel they are adding a lot of value
*Not all testers agree
*For some, the time for "high-fives" are after the bug has been found, reported, fixed, and the system successfully retested, and the "high-five" is shared with the developers
*Everyone on the team wants the project to be a success!
*Also fits in with having tact and consideration for others

== Load testing ==
*After testing different parts of a system and making sure they work together well, you may want to do load testing
*Maximize the amount of transactions occurring
*Planning can be very helpful here, keep track of the order you did things, test systematically, expected results, otherwise it's too easy to get lost in a big pile of data
*This means making sure no transactions are dropped
*Or, in financial systems, totals add up, reports balance
*Testing Tools can be used for this
*Ironically, Testing Tools are not always very well tested
*They might be "quick and dirty", but as long as the tester who made them can use them to serve a main purpose well enough, then the real, main tests get done

== "Monkey Testing" ==
*Can be important
*Bach talks about examples, able to break right into a computer, produce error boxes with no text
*Using Ctrl-A Ctrl-C Ctrl-V, put a massive amount of text into an input field, see what happens
*Try moving really fast pressing buttons
*There is one big catch to this kind of testing
*Can you reliably recreate the error? If you were frantically pressing buttons fast, and timing is part of the issue, that can be very hard
[[File:3 - EmptyErrorResized.png]]

[[File:4 - SystemEntryExampleResized.png]]

== Testing similar things at once ==
*Suppose you have a system that allows you to set countdown alarms, like scheduling employees' breaks
*Each employee has a timer object
*Each object is therefore distinct
*But suppose management has a screen they use to see all employee schedules and minutes left until their breaks
*Executive can shorten or lengthen that time, they have a screen Management has access to
*That screen loads an employee day schedule object
*Copy that data into a report for that manager's daily activity, i.e.

Schedule Adjustments:
PayRoll Clerk - 10 minute delay
Production manager - 15 minute delay

Local variables cleared in between these two transactions
What if there's just one break room, an alert is sounded when your break is over.
If you have a flag that says "BreakTimeOver = true", it has to be reset

== What if functions DON'T appear to have much in common? ==
*Some functions, like above, are about tracking time
*Others keep track of that day's production, how many units produced
*Very separate things, but all part of the same system
*Run them all with a heavy load for a while, see if the system responds well, there is enough memory
*Different developers may have worked on these parts, may not have tested all together

== Button availability ==
*Many companies will use this, document it as part of a project plan
*Is important to test
*Even a small number of buttons can create many paths
*Especially important for security, don't allow login attempts without credentials being entered
*You don't have to have lots of buttons in a window for this to become time consuming and a lot of hard work

== Consistent look and feel ==
*Sometimes errors occur that are hard to spot - fonts might be very similar, but slightly different, just one size larger or smaller, shadow effects

== Resizing windows ==
*This can cause data to become lost, off screen, is a good idea to stretch windows, maximize, minimize, resize

== Logs ==
*What information is supposed to be in them? What is definitely NOT supposed to be there? Certain security things should NOT be there, check and make sure. Sometimes developers log things for debug purposes but this should be cleaned up, can be very important!

== Time ==
*Do you leave a system running overnight? Over a weekend? Over a holiday weekend? For weeks?
*Some systems do have to be running around the clock.
*Do you leave it running with a lot of activity for a long time? Is that realistic?
*If something goes wrong in the night, how hard is it to track down the cause of a problem hours later, when you get to work in the morning?

== Updating during a project ==
*Testers might be given whatever's available to start testing. This might be prototypes, partly developed items
*Updates are provided during a project
*Installing updates can take significant amounts of time
*It is important to work with developers to manage time. If several updates are coming soon, maybe wait until several are ready, install them, then resume testing rather than have multiple interruptions to install several updates
*If an update has to be applied to many components, try a few first, do at least some basic testing before investing time in updating whole system

== Graceful failures ==
*What if a component breaks, does the rest of the system respond gracefully?
*If a load test does cause errors, how well is it handled? Load tests can make systems slow, resulting in a system appearing to "hang" rather than fail gracefully

[[Category:NPSA]]

File:4 - SystemEntryExampleResized.png

2019-05-03T02:27:31Z

NicholasCollins:

File:3 - EmptyErrorResized.png

2019-05-03T02:26:23Z

NicholasCollins:

Software Testing/Meeting Notes 2019-04-08

2019-05-03T02:25:38Z

NicholasCollins:

{{:Software Testing}}

__TOC__

==== Resources ====

* [https://www.techsoupcanada.ca/en/directory/334 TechSoup Canada catalogue: Developer Software]
* [https://obsproject.com/ OBS Project] - software that is good for capturing videos of tests including drop menus

One of James Bach's [https://www.youtube.com/watch?v=ILkT_HV9DVU talks on YouTube]
{{#evu:https://www.youtube.com/watch?v=ILkT_HV9DVU | dimensions=640 | alignment=center}}

* [[Software Testing Additional Notes]] - Nicholas Collins' presentation notes

==== Meeting Notes ====
* Introductions

* Nicholas Collins
** Software tester for a few years, knowledge of how ''his'' company works
** But two years isn't a long time compared to some software testers
** Nick has prepared notes, will be presenting slightly differently from other KWNPSA sessions
** SysAdmin in insurance industry; laid off (as are many of us); back to school to upgrade IT skills
** Uses Visual Studio, C#, other languages
** People he's met were developers, or business-specific skills; when software testers are needed these people are thrust into the role
** This might change as more universities offer software testing as a major
** There are very few courses or certificates in software testing, more prevalent in the US
*** but Fanshawe college in London has a certificate program
** Some institutions have a couple of courses in tech writing, project management, quality management; maybe a night course in software quality testing
** Without academic rigour, different people use different terminology, nomenclature
*** "Should I know what all these different terms mean?" But it's fairly common with other software testers Nick has spoken to
** At Microsoft, developers use their development skills to write tests. Needs more skills than just coding
*** Microsoft has internal courses to train testers how to test software
*** Get promoted to full developer once you've proven you can write tests
** people use Terminology like "Post-mortem" (although nobody dies), mix up "Milestone" and "Benchmark", &c.
** Software testing is the start for a developer's career, then to DevOps
*** Does this mean the most junior, inexperienced programmers are responsible for testing software?
** Nick: large companies use junior testers to run tests, senior testers to supervise
** During an upgrade Nick (a programmer at the time) did testing for the Database Analyst
*** But a junior intern was assigned to that role as well, just to gain experience.
*** Worked out details at a high level, then applied tests to get results

* Project Managers take different approaches

* '''You can always think of more tests'''
** It's a fine balance between staying on schedule and being thorough
** Walkthroughs and working in a team can be helpful

* Some testing instructors do not like teaching from texts
** eg. "Software Testing" by Yogesh Singh
** But Nicholas gets good ideas from texts, doesn't agree with those testing instructors
** THe problem is that the authors suffer from "Perfect Worldism"
*** A world where there is unlimited time and money, and the perfect tests can be developed
** Nicholas has experience with sticky problems, gets ideas from texts to adapt to his problem

* Party talk: Software tester does not lead to stimulating conversation
** YouTube presenter on software testing is not dull! https://www.youtube.com/watch?v=ILkT_HV9DVU

* Even the simplest test "is A < 70 ?" can have seven or eight tests
[[File:FlowchartExampleResized.png]]
** Test results, but also overflows, boundaries, different data types
** Input validation can require many tests

* Working with other people, eg. technical writers
** For them to understand the software they'll play with the software, and may create unanticipated conditions
** Everyone can be a software tester to some degree: Project manager, developer, writer. Even sales?
** Sometimes testers find problems with usability as they're running tests; not part of the test suite

* How effective are some of these ad-hoc testers?
** Is there a bias? Do they have some incentive to pass tests even when there are problems?
*** Sometimes a QA will hold back tests that would have been better to give to the developer in the first place
** Accessibility testing is a new skill for QA, may become a testing requirement
** Business Analyst (BA), developer and tester make a good team
*** Sometimes the process of testing will identify the need for more testing

* Reporting bugs
** Requires consideration, tact

* Test plans may need to be developed quickly
** But near the end of a project when time is tight there may not be time to develop tests
** So quality of code may suffer near the end of the project
*** Breaking things during testing that no-one has time to fix

* Automated testing?
** Nick has experience with automated regression testing
** Automated regression testing reduces the introduction of new bugs

* Open Broadcaster Software
** Used to catch all activity during user testing
** Also use Virtual Box recorder uses host to capture all the output on the VM screen

* "Monkey Testing"
** Also "fuzz testing" or "fuzzing"
** Fill all fields, try to overflow, pound on the keyboard, click as fast as possible
*** But this this does not lead to reproducible errors (fine timing errors)
*** Although some testers claim they can reproduce

* Pride in finding bugs?
** Nick finds that the "high five" time should occur only after the entire team has identified, reported, documented, and fixed the error, and re-tested

* Load testing
** Hitting a system with a large number of transactions, &c.
** But a bogged down system may not be writing to logs, making analysis difficult
** A benefit in load testing is adding assertions, find issues with threads
*** Assertions and Singletons...
** Be sure to validate the output even when just testing for capacity

* Nick has written a test for XML testing
** But the code Nick wrote was not well tested at all! Oh, the irony!

* Q: Do you use debugger software like GDB to examine the flow of code?
* A: Not common, but becoming more prevalent.
** Certainly having a debugger to throw at the code is nice to have
** But much testing is done with the software under test as a black box, just examine the input and the expected output

* Nick speaks of the complexity of software testing.
** One thing works fine by itself, and other thing does too, but do they work together?
** Different software on different platform needs to interoperate, but sometimes differences in date formats causes problems
*** although each platform by itself passed all tests
** Dealing with currencies, eg. USD and CAD, and GBP
** Dealing with leap years and 29 February
** General rule: Anything date sensitive needs to test for leap years
*** and time zones! Anything dealing with calendars needs to worry about time zones

* What happens internationally when different countries need to interoperate?
** Companies have service contracts that define how the service is implemented
*** If the system is changed, the contract defines who is responsible for continued interoperation
*** If I make a change and it breaks your system, it's your fault for not defining the contract accurately
*** called "spring contracts"

* Nick gives an example from James ---- YouTube video ("nominal input voltage is 100VAC to 250VAC")
[[File:VoltageExampleResized.png]]
** "Test the nominal range" is an incomplete answer
** Also need to test outside the range
** The user manual may give advice not to go outside the nominal range, but users don't necessarily read the manual
** So, does the system fail gracefully outside the nominal range?
** This is the function of the software tester, to design the test to ensure that software or equipment is failsafe
*** eg. for medical equipment
*** How much money is available to fry the device under test? Some prototypes may be '''really''' expensive
*** Many examples of people damaging electronics with incorrect application of voltage!
** It's good for testers to think outside the parameters of the system

* Testing to ensure system has a consistent look and feel
** eg. fonts on some menus were different
*** Is that a software testers responsibility? Sometimes as an additional task
*** There are tools (overlays, templates) to find these issues
** Window resizing can make the application fail, but there need to be limits for those tests
** Testing for "greyed out" functions can be time consuming
*** When a function is available when it shouldn't be can result in errors
** These are general things for a tester to keep in mind

* Systems that have features which have little to do with each other
** Easy to test they're not contending for resources, &c.
** But still important to run these features simultaneous to shake loose bugs, eg. memory allocation, concurrent DB access
** Perhaps a simple monitor with limited functions: But what if something goes wrong, does the device report an error?

* Client-side data validation: All testing needs to be duplicated at the server to ensure malusers don't bypass client-side validation
** But that increases load on the server

* Logging
** Logs may indicate problems with the way the code executes, eg. repeated log entries indicate an invalid loop
** Circular reasoning: How can the logs from software under test be considered
*** Logs are only one step, begin the process of analysis
*** NewRelic will test user experience (surveillance software)

* Nick has found bugs because the test suites are well designed
** But at least half the time the bugs discovered were found in spite of the test, which was not designed to find that kind of bug

* Q&A
** Is the developer + tester model usable?
*** May be a bit scary for shops not set up for that collaborative arrangement
** Nick says to just forge ahead.
*** Having experience is good, but can also develop that experience in-house
** Worries about the coming requirements for accessibility for software
*** May take changes in coding practices (use POSH: Plain Ol' Semantic HTML instead of Javascripted forms)
*** Jurisdictional differences may be difficult to deal with

Back to: [[Software Testing]]

[[Category:KWNPSA Meeting Notes]]

File:FlowchartExampleResized.png

2019-05-03T02:24:40Z

NicholasCollins:

Software Testing/Meeting Notes 2019-04-08

2019-05-03T02:23:51Z

NicholasCollins:

{{:Software Testing}}

__TOC__

==== Resources ====

* [https://www.techsoupcanada.ca/en/directory/334 TechSoup Canada catalogue: Developer Software]
* [https://obsproject.com/ OBS Project] - software that is good for capturing videos of tests including drop menus

One of James Bach's [https://www.youtube.com/watch?v=ILkT_HV9DVU talks on YouTube]
{{#evu:https://www.youtube.com/watch?v=ILkT_HV9DVU | dimensions=640 | alignment=center}}

* [[Software Testing Additional Notes]] - Nicholas Collins' presentation notes

==== Meeting Notes ====
* Introductions

* Nicholas Collins
** Software tester for a few years, knowledge of how ''his'' company works
** But two years isn't a long time compared to some software testers
** Nick has prepared notes, will be presenting slightly differently from other KWNPSA sessions
** SysAdmin in insurance industry; laid off (as are many of us); back to school to upgrade IT skills
** Uses Visual Studio, C#, other languages
** People he's met were developers, or business-specific skills; when software testers are needed these people are thrust into the role
** This might change as more universities offer software testing as a major
** There are very few courses or certificates in software testing, more prevalent in the US
*** but Fanshawe college in London has a certificate program
** Some institutions have a couple of courses in tech writing, project management, quality management; maybe a night course in software quality testing
** Without academic rigour, different people use different terminology, nomenclature
*** "Should I know what all these different terms mean?" But it's fairly common with other software testers Nick has spoken to
** At Microsoft, developers use their development skills to write tests. Needs more skills than just coding
*** Microsoft has internal courses to train testers how to test software
*** Get promoted to full developer once you've proven you can write tests
** people use Terminology like "Post-mortem" (although nobody dies), mix up "Milestone" and "Benchmark", &c.
** Software testing is the start for a developer's career, then to DevOps
*** Does this mean the most junior, inexperienced programmers are responsible for testing software?
** Nick: large companies use junior testers to run tests, senior testers to supervise
** During an upgrade Nick (a programmer at the time) did testing for the Database Analyst
*** But a junior intern was assigned to that role as well, just to gain experience.
*** Worked out details at a high level, then applied tests to get results

* Project Managers take different approaches

* '''You can always think of more tests'''
** It's a fine balance between staying on schedule and being thorough
** Walkthroughs and working in a team can be helpful

* Some testing instructors do not like teaching from texts
** eg. "Software Testing" by Yogesh Singh
** But Nicholas gets good ideas from texts, doesn't agree with those testing instructors
** THe problem is that the authors suffer from "Perfect Worldism"
*** A world where there is unlimited time and money, and the perfect tests can be developed
** Nicholas has experience with sticky problems, gets ideas from texts to adapt to his problem

* Party talk: Software tester does not lead to stimulating conversation
** YouTube presenter on software testing is not dull! https://www.youtube.com/watch?v=ILkT_HV9DVU

* Even the simplest test "is A < 70 ?" can have seven or eight tests
** Test results, but also overflows, boundaries, different data types
** Input validation can require many tests

* Working with other people, eg. technical writers
** For them to understand the software they'll play with the software, and may create unanticipated conditions
** Everyone can be a software tester to some degree: Project manager, developer, writer. Even sales?
** Sometimes testers find problems with usability as they're running tests; not part of the test suite

* How effective are some of these ad-hoc testers?
** Is there a bias? Do they have some incentive to pass tests even when there are problems?
*** Sometimes a QA will hold back tests that would have been better to give to the developer in the first place
** Accessibility testing is a new skill for QA, may become a testing requirement
** Business Analyst (BA), developer and tester make a good team
*** Sometimes the process of testing will identify the need for more testing

* Reporting bugs
** Requires consideration, tact

* Test plans may need to be developed quickly
** But near the end of a project when time is tight there may not be time to develop tests
** So quality of code may suffer near the end of the project
*** Breaking things during testing that no-one has time to fix

* Automated testing?
** Nick has experience with automated regression testing
** Automated regression testing reduces the introduction of new bugs

* Open Broadcaster Software
** Used to catch all activity during user testing
** Also use Virtual Box recorder uses host to capture all the output on the VM screen

* "Monkey Testing"
** Also "fuzz testing" or "fuzzing"
** Fill all fields, try to overflow, pound on the keyboard, click as fast as possible
*** But this this does not lead to reproducible errors (fine timing errors)
*** Although some testers claim they can reproduce

* Pride in finding bugs?
** Nick finds that the "high five" time should occur only after the entire team has identified, reported, documented, and fixed the error, and re-tested

* Load testing
** Hitting a system with a large number of transactions, &c.
** But a bogged down system may not be writing to logs, making analysis difficult
** A benefit in load testing is adding assertions, find issues with threads
*** Assertions and Singletons...
** Be sure to validate the output even when just testing for capacity

* Nick has written a test for XML testing
** But the code Nick wrote was not well tested at all! Oh, the irony!

* Q: Do you use debugger software like GDB to examine the flow of code?
* A: Not common, but becoming more prevalent.
** Certainly having a debugger to throw at the code is nice to have
** But much testing is done with the software under test as a black box, just examine the input and the expected output

* Nick speaks of the complexity of software testing.
** One thing works fine by itself, and other thing does too, but do they work together?
** Different software on different platform needs to interoperate, but sometimes differences in date formats causes problems
*** although each platform by itself passed all tests
** Dealing with currencies, eg. USD and CAD, and GBP
** Dealing with leap years and 29 February
** General rule: Anything date sensitive needs to test for leap years
*** and time zones! Anything dealing with calendars needs to worry about time zones

* What happens internationally when different countries need to interoperate?
** Companies have service contracts that define how the service is implemented
*** If the system is changed, the contract defines who is responsible for continued interoperation
*** If I make a change and it breaks your system, it's your fault for not defining the contract accurately
*** called "spring contracts"

* Nick gives an example from James ---- YouTube video ("nominal input voltage is 100VAC to 250VAC")
[[File:VoltageExampleResized.png]]
** "Test the nominal range" is an incomplete answer
** Also need to test outside the range
** The user manual may give advice not to go outside the nominal range, but users don't necessarily read the manual
** So, does the system fail gracefully outside the nominal range?
** This is the function of the software tester, to design the test to ensure that software or equipment is failsafe
*** eg. for medical equipment
*** How much money is available to fry the device under test? Some prototypes may be '''really''' expensive
*** Many examples of people damaging electronics with incorrect application of voltage!
** It's good for testers to think outside the parameters of the system

* Testing to ensure system has a consistent look and feel
** eg. fonts on some menus were different
*** Is that a software testers responsibility? Sometimes as an additional task
*** There are tools (overlays, templates) to find these issues
** Window resizing can make the application fail, but there need to be limits for those tests
** Testing for "greyed out" functions can be time consuming
*** When a function is available when it shouldn't be can result in errors
** These are general things for a tester to keep in mind

* Systems that have features which have little to do with each other
** Easy to test they're not contending for resources, &c.
** But still important to run these features simultaneous to shake loose bugs, eg. memory allocation, concurrent DB access
** Perhaps a simple monitor with limited functions: But what if something goes wrong, does the device report an error?

* Client-side data validation: All testing needs to be duplicated at the server to ensure malusers don't bypass client-side validation
** But that increases load on the server

* Logging
** Logs may indicate problems with the way the code executes, eg. repeated log entries indicate an invalid loop
** Circular reasoning: How can the logs from software under test be considered
*** Logs are only one step, begin the process of analysis
*** NewRelic will test user experience (surveillance software)

* Nick has found bugs because the test suites are well designed
** But at least half the time the bugs discovered were found in spite of the test, which was not designed to find that kind of bug

* Q&A
** Is the developer + tester model usable?
*** May be a bit scary for shops not set up for that collaborative arrangement
** Nick says to just forge ahead.
*** Having experience is good, but can also develop that experience in-house
** Worries about the coming requirements for accessibility for software
*** May take changes in coding practices (use POSH: Plain Ol' Semantic HTML instead of Javascripted forms)
*** Jurisdictional differences may be difficult to deal with

Back to: [[Software Testing]]

[[Category:KWNPSA Meeting Notes]]

File:VoltageExampleResized.png

2019-05-03T02:20:57Z

NicholasCollins:

Software Testing Additional Notes

2019-05-03T02:04:52Z

NicholasCollins:

== A lot of people end up in software testing via an oblique path. ==
*Possibly from a technical background, working as developers
*Possibly a business background, get to know a company's products
*You can take courses on it, but often as part of a larger program, rather than the main major
*Will that change in the future?
*Result could be why there is variety in terminology

== Different project managers take different approaches. ==
*Some sit in on code reviews, test plan reviews, dig in to details
*Others get their team to report to them at a summary level
*Project manager will more directly intervene when a high priority situation arises, i.e. testing falling behind schedule, often a good project manager is someone who can keep an eye on testing at a high level and quickly assess an important situation in detail when necessary

== Developers are often expected to do at least some testing of their own ==

*For some technical projects like software upgrades, developers might be recruited to work as testers
*Developers often do unit testing, sometimes document test plans ahead of time
*That unit test plan and results are given to testers when they do additional testing - integration testing, regression testing, load testing

== For testers, one challenge is the need to be thorough and deliver on time ==
*Sometimes, the first, fairly simple tests reveal a number of bugs; this creates concern about staying on schedule
*Advantage is that at least those bugs were found early
*It can be good to push hard, maybe put in overtime to get through the first pass of testing, get an idea of where bugs are, start fixing them early. This can avoid worse time crunch near the end of a project. Sometimes that can still happen anyway!

== Writing test plans can take a significant amount of time ==
*Does the project schedule allow for that?
*How much time will it save alter, if there is a test plan that is thorough and has been peer reviewed and edited? Often this makes it worth the effort
*In some situations, time constraints don't allow it, have to jump in to testing with minimal planning or documenting
*Some projects end up as a mixture. A detailed test plan is written and carefully peer reviewed early in the project
*During the project, unexpected problems and changes in requirements lead to many changes to the code and test plan
*At that point, there is limited time left in the project schedule, test plans are rough, possibly not peer reviewed
*It then helps if you have experience, when you're new to the position it's harder to "improvise" testing

== Reporting bugs to developers requires tact ==
*People have worked very hard on the code; you are finding problems with it
*You are all on the same team with the goal of delivering a good quality product
*Keep e-mails and bug reports in business style; just describe what happens, provide the steps to re-create the error
*When reporting bugs, avoid the use of pronouns. Say, "When I do x, the system does y", never say "Your code has a problem"
*But, do use pronouns when giving thanks for good work, acknowledge others in meetings, "Yes, this was a tough bug, thankfully Jon got it fixed yesterday."
*Einstein is believed to have said, "Make everything as simple as possible but no simpler". May not have been the exact words
*When doing bug reports, get to the main point quickly, but include the technical details necessary. Make sure YOU can reliably recreate the error following the same steps

Calculator

Basic format:
Steps to Re-create:

Press "pi"

Press Clear

Press "1"

Press "+"

Press "1"

Press "="

Expected Results:

Display shows "2"

Actual Results

Display shows "2.00000001"

System should be declaring input and response as integers, possibly using float or another type and not doing garbage collection from previous operation?

*Being very explicit about what was pressed, in what order, matters sometimes - people have habits. On a keyboard, some use the number pad, others the main keyboard. If someone turns num lock on or off during bootup as a habit and someone else was on the computer, you could get different results.
*When describing a sequence of menu choices, the "->" characters are helpful, i.e. select Main Menu->Submenu1->SubSubmenu1
*Making comments, suggestions does not hurt if it only takes a short time. If it's similar to another known bug, point that out, too.
*Does the system have different states? What state was it in when the test was done?
* Sounds simple but can be tricky, can be easy to forget an important detail - in a web-based system, which browser were you using? Sometimes you switch between Edge, FireFox and Chrome throughout the day, remember which one you found a bug in, check and see if it happens in other browsers.
* https://obsproject.com/ - software that can be used to capture videos, including drop menus. Windows 10's Game Capture does not include drop menus.

== Other tasks follow testing ==
*Technical writers have to produce documentation like manuals. They may want to look at some of the existing documentation. That may not be the test plan, it could be developer notes or requirements. Sometimes the tester has worked with those documents a lot, and is asked to assemble them into one place for the technical writer, or directing those people where to find things.
*The technical writer might want to also be a tester. In order to write documentation that explains things to the user, the writer has to understand it well, getting their hands on a test system and trying some things helps them do this.
*The technical writer will have questions for the tester, may want assistance in configuring a system
*Sometimes testing was done to cover many situations, but the first sales are to customers with specific needs, the technical writer will be focused on that, configuring a system that way
*It helps to label things, the tester might know about the system configuration but the technical writer doesn't.
*Sometimes the technical writer has industry experience, ends up trying some additional tests and finds bugs. Then the tester follows up to write up the bug and get it fixed.

== Some professionals actively dislike textbooks ==
*Some textbooks for software testing, as is the case for books in other technical fields, have a tendency towards "perfect worldism"
*They will describe techniques and show examples of using those techniques that are very time consuming to do.
*They might be good techniques that will capture lots of bugs if you apply them
*On a real project, time is limited, you have to make judgment calls about prioritizing things to meet deadlines
*Don't want to tell your manager you spent hours on this marvellous documentation and planning technique but got no actual tests done
*So why pay attention to text books?
*Even if you can't completely apply the techniques they describe exactly as outlined, you might get ideas you can partially apply that do help raise the quality of what you deliver
*Such as - Different types of program maps - even a partial one might get you thinking about test cases you want to do, maybe apply these techniques in depth for especially important or complex parts of a system if not everything
*You may adapt/adjust these techniques to suit your situation
*Can also give you the idea of the number of test there are in theory, including paths

== On YouTube, James Bach has some interesting talks ==
*https://www.youtube.com/watch?v=ILkT_HV9DVU
*Asks people how they would test things, pushes hard for people to explain why
*Sometimes testers have ideas/intuitions that are on the right track, can you explain why?
*He talks about a system that will work 100 - 250 VAC
*So why test, say, 90 V? It's not in the specs, we know it won't work
*But will it "fail gracefully"?
*(There is not always time for that in the workplace, but if you just follow your intuition you might find important bugs)

== Bach also goes through an example and asks how many test cases would you do? ==
*Raises questions about what the project does
*Decision testing, boundary testing, predicate testing
*With more experience, you can often think of more tests right off, i.e. "improvise"
*Emphasizes the fact that a flowchart, document, etc. is a representation, not the actual system

== When do you feel pride/satisfaction in your work? ==
*Some testers say they "high-five" each other when they find a really nasty, obscure, or complicated bug
*That is when they feel they are adding a lot of value
*Not all testers agree
*For some, the time for "high-fives" are after the bug has been found, reported, fixed, and the system successfully retested, and the "high-five" is shared with the developers
*Everyone on the team wants the project to be a success!
*Also fits in with having tact and consideration for others

== Load testing ==
*After testing different parts of a system and making sure they work together well, you may want to do load testing
*Maximize the amount of transactions occurring
*Planning can be very helpful here, keep track of the order you did things, test systematically, expected results, otherwise it's too easy to get lost in a big pile of data
*This means making sure no transactions are dropped
*Or, in financial systems, totals add up, reports balance
*Testing Tools can be used for this
*Ironically, Testing Tools are not always very well tested
*They might be "quick and dirty", but as long as the tester who made them can use them to serve a main purpose well enough, then the real, main tests get done

== "Monkey Testing" ==
*Can be important
*Bach talks about examples, able to break right into a computer, produce error boxes with no text
*Using Ctrl-A Ctrl-C Ctrl-V, put a massive amount of text into an input field, see what happens
*Try moving really fast pressing buttons
*There is one big catch to this kind of testing
*Can you reliably recreate the error? If you were frantically pressing buttons fast, and timing is part of the issue, that can be very hard

== Testing similar things at once ==
*Suppose you have a system that allows you to set countdown alarms, like scheduling employees' breaks
*Each employee has a timer object
*Each object is therefore distinct
*But suppose management has a screen they use to see all employee schedules and minutes left until their breaks
*Executive can shorten or lengthen that time, they have a screen Management has access to
*That screen loads an employee day schedule object
*Copy that data into a report for that manager's daily activity, i.e.

Schedule Adjustments:
PayRoll Clerk - 10 minute delay
Production manager - 15 minute delay

Local variables cleared in between these two transactions
What if there's just one break room, an alert is sounded when your break is over.
If you have a flag that says "BreakTimeOver = true", it has to be reset

== What if functions DON'T appear to have much in common? ==
*Some functions, like above, are about tracking time
*Others keep track of that day's production, how many units produced
*Very separate things, but all part of the same system
*Run them all with a heavy load for a while, see if the system responds well, there is enough memory
*Different developers may have worked on these parts, may not have tested all together

== Button availability ==
*Many companies will use this, document it as part of a project plan
*Is important to test
*Even a small number of buttons can create many paths
*Especially important for security, don't allow login attempts without credentials being entered
*You don't have to have lots of buttons in a window for this to become time consuming and a lot of hard work

== Consistent look and feel ==
*Sometimes errors occur that are hard to spot - fonts might be very similar, but slightly different, just one size larger or smaller, shadow effects

== Resizing windows ==
*This can cause data to become lost, off screen, is a good idea to stretch windows, maximize, minimize, resize

== Logs ==
*What information is supposed to be in them? What is definitely NOT supposed to be there? Certain security things should NOT be there, check and make sure. Sometimes developers log things for debug purposes but this should be cleaned up, can be very important!

== Time ==
*Do you leave a system running overnight? Over a weekend? Over a holiday weekend? For weeks?
*Some systems do have to be running around the clock.
*Do you leave it running with a lot of activity for a long time? Is that realistic?
*If something goes wrong in the night, how hard is it to track down the cause of a problem hours later, when you get to work in the morning?

== Updating during a project ==
*Testers might be given whatever's available to start testing. This might be prototypes, partly developed items
*Updates are provided during a project
*Installing updates can take significant amounts of time
*It is important to work with developers to manage time. If several updates are coming soon, maybe wait until several are ready, install them, then resume testing rather than have multiple interruptions to install several updates
*If an update has to be applied to many components, try a few first, do at least some basic testing before investing time in updating whole system

== Graceful failures ==
*What if a component breaks, does the rest of the system respond gracefully?
*If a load test does cause errors, how well is it handled? Load tests can make systems slow, resulting in a system appearing to "hang" rather than fail gracefully

[[Category:NPSA]]

Software Testing Additional Notes

2019-05-03T02:03:55Z

NicholasCollins:

== A lot of people end up in software testing via an oblique path. ==
*Possibly from a technical background, working as developers
*Possibly a business background, get to know a company's products
*You can take courses on it, but often as part of a larger program, rather than the main major
*Will that change in the future?
*Result could be why there is variety in terminology

== Different project managers take different approaches. ==
*Some sit in on code reviews, test plan reviews, dig in to details
*Others get their team to report to them at a summary level
*Project manager will more directly intervene when a high priority situation arises, i.e. testing falling behind schedule, often a good project manager is someone who can keep an eye on testing at a high level and quickly assess an important situation in detail when necessary

== Developers are often expected to do at least some testing of their own ==

*For some technical projects like software upgrades, developers might be recruited to work as testers
*Developers often do unit testing, sometimes document test plans ahead of time
*That unit test plan and results are given to testers when they do additional testing - integration testing, regression testing, load testing

== For testers, one challenge is the need to be thorough and deliver on time ==
*Sometimes, the first, fairly simple tests reveal a number of bugs; this creates concern about staying on schedule
*Advantage is that at least those bugs were found early
*It can be good to push hard, maybe put in overtime to get through the first pass of testing, get an idea of where bugs are, start fixing them early. This can avoid worse time crunch near the end of a project. Sometimes that can still happen anyway!

== Writing test plans can take a significant amount of time ==
*Does the project schedule allow for that?
*How much time will it save alter, if there is a test plan that is thorough and has been peer reviewed and edited? Often this makes it worth the effort
*In some situations, time constraints don't allow it, have to jump in to testing with minimal planning or documenting
*Some projects end up as a mixture. A detailed test plan is written and carefully peer reviewed early in the project
*During the project, unexpected problems and changes in requirements lead to many changes to the code and test plan
*At that point, there is limited time left in the project schedule, test plans are rough, possibly not peer reviewed
*It then helps if you have experience, when you're new to the position it's harder to "improvise" testing

== Reporting bugs to developers requires tact ==
*People have worked very hard on the code; you are finding problems with it
*You are all on the same team with the goal of delivering a good quality product
*Keep e-mails and bug reports in business style; just describe what happens, provide the steps to re-create the error
*When reporting bugs, avoid the use of pronouns. Say, "When I do x, the system does y", never say "Your code has a problem"
*But, do use pronouns when giving thanks for good work, acknowledge others in meetings, "Yes, this was a tough bug, thankfully Jon got it fixed yesterday."
*Einstein is believed to have said, "Make everything as simple as possible but no simpler". May not have been the exact words
*When doing bug reports, get to the main point quickly, but include the technical details necessary. Make sure YOU can reliably recreate the error following the same steps

Calculator

Basic format:
Steps to Re-create:

Press "pi"

Press Clear

Press "1"

Press "+"

Press "1"

Press "="

Expected Results:

Display shows "2"

Actual Results

Display shows "2.00000001"

System should be declaring input and response as integers, possibly using float or another type and not doing garbage collection from previous operation?

*Being very explicit about what was pressed, in what order, matters sometimes - people have habits. On a keyboard, some use the number pad, others the main keyboard. If someone turns num lock on or off during bootup as a habit and someone else was on the computer, you could get different results.
*When describing a sequence of menu choices, the "->" characters are helpful, i.e. select Main Menu->Submenu1->SubSubmenu1
*Making comments, suggestions does not hurt if it only takes a short time. If it's similar to another known bug, point that out, too.
*Does the system have different states? What state was it in when the test was done?
* Sounds simple but can be tricky, can be easy to forget an important detail - in a web-based system, which browser were you using? Sometimes you switch between Edge, FireFox and Chrome throughout the day, remember which one you found a bug in, check and see if it happens in other browsers.
* https://obsproject.com/ - software that can be used to capture videos, including drop menus. Windows 10's Game Capture does not include drop menus.

== Other tasks follow testing ==
*Technical writers have to produce documentation like manuals. They may want to look at some of the existing documentation. That may not be the test plan, it could be developer notes or requirements. Sometimes the tester has worked with those documents a lot, and is asked to assemble them into one place for the technical writer, or directing those people where to find things.
*The technical writer might want to also be a tester. In order to write documentation that explains things to the user, the writer has to understand it well, getting their hands on a test system and trying some things helps them do this.
*The technical writer will have questions for the tester, may want assistance in configuring a system
*Sometimes testing was done to cover many situations, but the first sales are to customers with specific needs, the technical writer will be focused on that, configuring a system that way
*It helps to label things, the tester might know about the system configuration but the technical writer doesn't.
*Sometimes the technical writer has industry experience, ends up trying some additional tests and finds bugs. Then the tester follows up to write up the bug and get it fixed.

== Some professionals actively dislike textbooks ==
*Some textbooks for software testing, as is the case for books in other technical fields, have a tendency towards "perfect worldism"
*They will describe techniques and show examples of using those techniques that are very time consuming to do.
*They might be good techniques that will capture lots of bugs if you apply them
*On a real project, time is limited, you have to make judgment calls about prioritizing things to meet deadlines
*Don't want to tell your manager you spent hours on this marvellous documentation and planning technique but got no actual tests done
*So why pay attention to text books?
*Even if you can't completely apply the techniques they describe exactly as outlined, you might get ideas you can partially apply that do help raise the quality of what you deliver
*Such as - Different types of program maps - even a partial one might get you thinking about test cases you want to do, maybe apply these techniques in depth for especially important or complex parts of a system if not everything
*You may adapt/adjust these techniques to suit your situation
*Can also give you the idea of the number of test there are in theory, including paths

== On YouTube, James Bach has some interesting talks ==
*https://www.youtube.com/watch?v=ILkT_HV9DVU
*Asks people how they would test things, pushes hard for people to explain why
*Sometimes testers have ideas/intuitions that are on the right track, can you explain why?
*He talks about a system that will work 100 - 250 VAC
*So why test, say, 90 V? It's not in the specs, we know it won't work
*But will it "fail gracefully"?
*(There is not always time for that in the workplace, but if you just follow your intuition you might find important bugs)

== Bach also goes through an example and asks how many test cases would you do? ==
*Raises questions about what the project does
*Decision testing, boundary testing, predicate testing
*With more experience, you can often think of more tests right off, i.e. "improvise"
*Emphasizes the fact that a flowchart, document, etc. is a representation, not the actual system

== When do you feel pride/satisfaction in your work? ==
*Some testers say they "high-five" each other when they find a really nasty, obscure, or complicated bug
*That is when they feel they are adding a lot of value
*Not all testers agree
*For some, the time for "high-fives" are after the bug has been found, reported, fixed, and the system successfully retested, and the "high-five" is shared with the developers
*Everyone on the team wants the project to be a success!
*Also fits in with having tact and consideration for others

== Load testing ==
*After testing different parts of a system and making sure they work together well, you may want to do load testing
*Maximize the amount of transactions occurring
*Planning can be very helpful here, keep track of the order you did things, test systematically, expected results, otherwise it's too easy to get lost in a big pile of data
*This means making sure no transactions are dropped
*Or, in financial systems, totals add up, reports balance
*Testing Tools can be used for this
*Ironically, Testing Tools are not always very well tested
*They might be "quick and dirty", but as long as the tester who made them can use them to serve a main purpose well enough, then the real, main tests get done

== "Monkey Testing" ==
*Can be important
*Bach talks about examples, able to break right into a computer, produce error boxes with no text
*Using Ctrl-A Ctrl-C Ctrl-V, put a massive amount of text into an input field, see what happens
*Try moving really fast pressing buttons
*There is one big catch to this kind of testing
*Can you reliably recreate the error? If you were frantically pressing buttons fast, and timing is part of the issue, that can be very hard

== Testing similar things at once ==
*Suppose you have a system that allows you to set countdown alarms, like scheduling employees' breaks
*Each employee has a timer object
*Each object is therefore distinct
*But suppose management has a screen they use to see all employee schedules and minutes left until their breaks
*Executive can shorten or lengthen that time, they have a screen Management has access to
*That screen loads an employee day schedule object
*Copy that data into a report for that manager's daily activity, i.e.

Schedule Adjustments:
PayRoll Clerk - 10 minute delay
Production manager - 15 minute delay

Local variables cleared in between these two transactions
What if there's just one break room, an alert is sounded when your break is over.
If you have a flag that says "BreakTimeOver = true", it has to be reset

== What if functions DON'T appear to have much in common? ==
*Some functions, like above, are about tracking time
*Others keep track of that day's production, how many units produced
*Very separate things, but all part of the same system
*Run them all with a heavy load for a while, see if the system responds well, there is enough memory
*Different developers may have worked on these parts, may not have tested all together

== Button availability ==
*Many companies will use this, document it as part of a project plan
*Is important to test
*Even a small number of buttons can create many paths
*Especially important for security, don't allow login attempts without credentials being entered
*You don't have to have lots of buttons in a window for this to become time consuming and a lot of hard work

== Consistent look and feel ==
*Sometimes errors occur that are hard to spot - fonts might be very similar, but slightly different, just one size larger or smaller, shadow effects

== Resizing windows ==
*This can cause data to become lost, off screen, is a good idea to stretch windows, maximize, minimize, resize

== Logs ==
*What information is supposed to be in them? What is definitely NOT supposed to be there? Certain security things should NOT be there, check and make sure. Sometimes developers log things for debug purposes but this should be cleaned up, can be very important!

== Time ==
*Do you leave a system running overnight? Over a weekend? Over a holiday weekend? For weeks?
*Some systems do have to be running around the clock.
*Do you leave it running with a lot of activity for a long time? Is that realistic?
*If something goes wrong in the night, how hard is it to track down the cause of a problem hours later, when you get to work in the morning?

== Updating during a project ==
*Testers might be given whatever's available to start testing. This might be prototypes, partly developed items
*Updates are provided during a project
*Installing updates can take significant amounts of time
*It is important to work with developers to manage time. If several updates are coming soon, maybe wait until several are ready, install them, then resume testing rather than have multiple interruptions to install several updates
*If an update has to be applied to many components, try a few first, do at least some basic testing before investing time in updating whole system

== Graceful failures ==
*What if a component breaks, does the rest of the system respond gracefully?
*If a load test does cause errors, how well is it handled? Load tests can make systems slow, resulting in a system appearing to "hang" rather than fail gracefully

[[File:1 - VoltageExample.png]]

[[Category:NPSA]]

File:1 - VoltageExample.png

2019-05-03T02:02:30Z

NicholasCollins:

Software Testing Additional Notes

2019-04-16T02:01:32Z

NicholasCollins:

== A lot of people end up in software testing via an oblique path. ==
*Possibly from a technical background, working as developers
*Possibly a business background, get to know a company's products
*You can take courses on it, but often as part of a larger program, rather than the main major
*Will that change in the future?
*Result could be why there is variety in terminology

== Different project managers take different approaches. ==
*Some sit in on code reviews, test plan reviews, dig in to details
*Others get their team to report to them at a summary level
*Project manager will more directly intervene when a high priority situation arises, i.e. testing falling behind schedule, often a good project manager is someone who can keep an eye on testing at a high level and quickly assess an important situation in detail when necessary

== Developers are often expected to do at least some testing of their own ==

*For some technical projects like software upgrades, developers might be recruited to work as testers
*Developers often do unit testing, sometimes document test plans ahead of time
*That unit test plan and results are given to testers when they do additional testing - integration testing, regression testing, load testing

== For testers, one challenge is the need to be thorough and deliver on time ==
*Sometimes, the first, fairly simple tests reveal a number of bugs; this creates concern about staying on schedule
*Advantage is that at least those bugs were found early
*It can be good to push hard, maybe put in overtime to get through the first pass of testing, get an idea of where bugs are, start fixing them early. This can avoid worse time crunch near the end of a project. Sometimes that can still happen anyway!

== Writing test plans can take a significant amount of time ==
*Does the project schedule allow for that?
*How much time will it save alter, if there is a test plan that is thorough and has been peer reviewed and edited? Often this makes it worth the effort
*In some situations, time constraints don't allow it, have to jump in to testing with minimal planning or documenting
*Some projects end up as a mixture. A detailed test plan is written and carefully peer reviewed early in the project
*During the project, unexpected problems and changes in requirements lead to many changes to the code and test plan
*At that point, there is limited time left in the project schedule, test plans are rough, possibly not peer reviewed
*It then helps if you have experience, when you're new to the position it's harder to "improvise" testing

== Reporting bugs to developers requires tact ==
*People have worked very hard on the code; you are finding problems with it
*You are all on the same team with the goal of delivering a good quality product
*Keep e-mails and bug reports in business style; just describe what happens, provide the steps to re-create the error
*When reporting bugs, avoid the use of pronouns. Say, "When I do x, the system does y", never say "Your code has a problem"
*But, do use pronouns when giving thanks for good work, acknowledge others in meetings, "Yes, this was a tough bug, thankfully Jon got it fixed yesterday."
*Einstein is believed to have said, "Make everything as simple as possible but no simpler". May not have been the exact words
*When doing bug reports, get to the main point quickly, but include the technical details necessary. Make sure YOU can reliably recreate the error following the same steps

Calculator

Basic format:
Steps to Re-create:

Press "pi"

Press Clear

Press "1"

Press "+"

Press "1"

Press "="

Expected Results:

Display shows "2"

Actual Results

Display shows "2.00000001"

System should be declaring input and response as integers, possibly using float or another type and not doing garbage collection from previous operation?

*Being very explicit about what was pressed, in what order, matters sometimes - people have habits. On a keyboard, some use the number pad, others the main keyboard. If someone turns num lock on or off during bootup as a habit and someone else was on the computer, you could get different results.
*When describing a sequence of menu choices, the "->" characters are helpful, i.e. select Main Menu->Submenu1->SubSubmenu1
*Making comments, suggestions does not hurt if it only takes a short time. If it's similar to another known bug, point that out, too.
*Does the system have different states? What state was it in when the test was done?
* Sounds simple but can be tricky, can be easy to forget an important detail - in a web-based system, which browser were you using? Sometimes you switch between Edge, FireFox and Chrome throughout the day, remember which one you found a bug in, check and see if it happens in other browsers.
* https://obsproject.com/ - software that can be used to capture videos, including drop menus. Windows 10's Game Capture does not include drop menus.

== Other tasks follow testing ==
*Technical writers have to produce documentation like manuals. They may want to look at some of the existing documentation. That may not be the test plan, it could be developer notes or requirements. Sometimes the tester has worked with those documents a lot, and is asked to assemble them into one place for the technical writer, or directing those people where to find things.
*The technical writer might want to also be a tester. In order to write documentation that explains things to the user, the writer has to understand it well, getting their hands on a test system and trying some things helps them do this.
*The technical writer will have questions for the tester, may want assistance in configuring a system
*Sometimes testing was done to cover many situations, but the first sales are to customers with specific needs, the technical writer will be focused on that, configuring a system that way
*It helps to label things, the tester might know about the system configuration but the technical writer doesn't.
*Sometimes the technical writer has industry experience, ends up trying some additional tests and finds bugs. Then the tester follows up to write up the bug and get it fixed.

== Some professionals actively dislike textbooks ==
*Some textbooks for software testing, as is the case for books in other technical fields, have a tendency towards "perfect worldism"
*They will describe techniques and show examples of using those techniques that are very time consuming to do.
*They might be good techniques that will capture lots of bugs if you apply them
*On a real project, time is limited, you have to make judgment calls about prioritizing things to meet deadlines
*Don't want to tell your manager you spent hours on this marvellous documentation and planning technique but got no actual tests done
*So why pay attention to text books?
*Even if you can't completely apply the techniques they describe exactly as outlined, you might get ideas you can partially apply that do help raise the quality of what you deliver
*Such as - Different types of program maps - even a partial one might get you thinking about test cases you want to do, maybe apply these techniques in depth for especially important or complex parts of a system if not everything
*You may adapt/adjust these techniques to suit your situation
*Can also give you the idea of the number of test there are in theory, including paths

== On YouTube, James Bach has some interesting talks ==
*https://www.youtube.com/watch?v=ILkT_HV9DVU
*Asks people how they would test things, pushes hard for people to explain why
*Sometimes testers have ideas/intuitions that are on the right track, can you explain why?
*He talks about a system that will work 100 - 250 VAC
*So why test, say, 90 V? It's not in the specs, we know it won't work
*But will it "fail gracefully"?
*(There is not always time for that in the workplace, but if you just follow your intuition you might find important bugs)

== Bach also goes through an example and asks how many test cases would you do? ==
*Raises questions about what the project does
*Decision testing, boundary testing, predicate testing
*With more experience, you can often think of more tests right off, i.e. "improvise"
*Emphasizes the fact that a flowchart, document, etc. is a representation, not the actual system

== When do you feel pride/satisfaction in your work? ==
*Some testers say they "high-five" each other when they find a really nasty, obscure, or complicated bug
*That is when they feel they are adding a lot of value
*Not all testers agree
*For some, the time for "high-fives" are after the bug has been found, reported, fixed, and the system successfully retested, and the "high-five" is shared with the developers
*Everyone on the team wants the project to be a success!
*Also fits in with having tact and consideration for others

== Load testing ==
*After testing different parts of a system and making sure they work together well, you may want to do load testing
*Maximize the amount of transactions occurring
*Planning can be very helpful here, keep track of the order you did things, test systematically, expected results, otherwise it's too easy to get lost in a big pile of data
*This means making sure no transactions are dropped
*Or, in financial systems, totals add up, reports balance
*Testing Tools can be used for this
*Ironically, Testing Tools are not always very well tested
*They might be "quick and dirty", but as long as the tester who made them can use them to serve a main purpose well enough, then the real, main tests get done

== "Monkey Testing" ==
*Can be important
*Bach talks about examples, able to break right into a computer, produce error boxes with no text
*Using Ctrl-A Ctrl-C Ctrl-V, put a massive amount of text into an input field, see what happens
*Try moving really fast pressing buttons
*There is one big catch to this kind of testing
*Can you reliably recreate the error? If you were frantically pressing buttons fast, and timing is part of the issue, that can be very hard

== Testing similar things at once ==
*Suppose you have a system that allows you to set countdown alarms, like scheduling employees' breaks
*Each employee has a timer object
*Each object is therefore distinct
*But suppose management has a screen they use to see all employee schedules and minutes left until their breaks
*Executive can shorten or lengthen that time, they have a screen Management has access to
*That screen loads an employee day schedule object
*Copy that data into a report for that manager's daily activity, i.e.

Schedule Adjustments:
PayRoll Clerk - 10 minute delay
Production manager - 15 minute delay

Local variables cleared in between these two transactions
What if there's just one break room, an alert is sounded when your break is over.
If you have a flag that says "BreakTimeOver = true", it has to be reset

== What if functions DON'T appear to have much in common? ==
*Some functions, like above, are about tracking time
*Others keep track of that day's production, how many units produced
*Very separate things, but all part of the same system
*Run them all with a heavy load for a while, see if the system responds well, there is enough memory
*Different developers may have worked on these parts, may not have tested all together

== Button availability ==
*Many companies will use this, document it as part of a project plan
*Is important to test
*Even a small number of buttons can create many paths
*Especially important for security, don't allow login attempts without credentials being entered
*You don't have to have lots of buttons in a window for this to become time consuming and a lot of hard work

== Consistent look and feel ==
*Sometimes errors occur that are hard to spot - fonts might be very similar, but slightly different, just one size larger or smaller, shadow effects

== Resizing windows ==
*This can cause data to become lost, off screen, is a good idea to stretch windows, maximize, minimize, resize

== Logs ==
*What information is supposed to be in them? What is definitely NOT supposed to be there? Certain security things should NOT be there, check and make sure. Sometimes developers log things for debug purposes but this should be cleaned up, can be very important!

== Time ==
*Do you leave a system running overnight? Over a weekend? Over a holiday weekend? For weeks?
*Some systems do have to be running around the clock.
*Do you leave it running with a lot of activity for a long time? Is that realistic?
*If something goes wrong in the night, how hard is it to track down the cause of a problem hours later, when you get to work in the morning?

== Updating during a project ==
*Testers might be given whatever's available to start testing. This might be prototypes, partly developed items
*Updates are provided during a project
*Installing updates can take significant amounts of time
*It is important to work with developers to manage time. If several updates are coming soon, maybe wait until several are ready, install them, then resume testing rather than have multiple interruptions to install several updates
*If an update has to be applied to many components, try a few first, do at least some basic testing before investing time in updating whole system

== Graceful failures ==
*What if a component breaks, does the rest of the system respond gracefully?
*If a load test does cause errors, how well is it handled? Load tests can make systems slow, resulting in a system appearing to "hang" rather than fail gracefully

Software Testing Additional Notes

2019-04-16T01:37:57Z

NicholasCollins: Created page with " == A lot of people end up in software testing via an oblique path. == *Possibly from a technical background, working as developers *Possibly a business background, get to kn..."

== A lot of people end up in software testing via an oblique path. ==
*Possibly from a technical background, working as developers
*Possibly a business background, get to know a company's products
*You can take courses on it, but often as part of a larger program, rather than the main major
*Will that change in the future?
*Result could be why there is variety in terminology

== Different project managers take different approaches. ==
*Some sit in on code reviews, test plan reviews, dig in to details
*Others get their team to report to them at a summary level
*Project manager will more directly intervene when a high priority situation arises, i.e. testing falling behind schedule, often a good project manager is someone who can keep an eye on testing at a high level and quickly assess an important situation in detail when necessary

== Developers are often expected to do at least some testing of their own ==

*For some technical projects like software upgrades, developers might be recruited to work as testers
*Developers often do unit testing, sometimes document test plans ahead of time
*That unit test plan and results are given to testers when they do additional testing - integration testing, regression testing, load testing

== For testers, one challenge is the need to be thorough and deliver on time ==
*Sometimes, the first, fairly simple tests reveal a number of bugs; this creates concern about staying on schedule
*Advantage is that at least those bugs were found early
*It can be good to push hard, maybe put in overtime to get through the first pass of testing, get an idea of where bugs are, start fixing them early. This can avoid worse time crunch near the end of a project. Sometimes that can still happen anyway!

== Writing test plans can take a significant amount of time ==
*Does the project schedule allow for that?
*How much time will it save alter, if there is a test plan that is thorough and has been peer reviewed and edited? Often this makes it worth the effort
*In some situations, time constraints don't allow it, have to jump in to testing with minimal planning or documenting
*Some projects end up as a mixture. A detailed test plan is written and carefully peer reviewed early in the project
*During the project, unexpected problems and changes in requirements lead to many changes to the code and test plan
*At that point, there is limited time left in the project schedule, test plans are rough, possibly not peer reviewed
*It then helps if you have experience, when you're new to the position it's harder to "improvise" testing

== Reporting bugs to developers requires tact ==
*People have worked very hard on the code; you are finding problems with it
*You are all on the same team with the goal of delivering a good quality product
*Keep e-mails and bug reports in business style; just describe what happens, provide the steps to re-create the error
*When reporting bugs, avoid the use of pronouns. Say, "When I do x, the system does y", never say "Your code has a problem"
*But, do use pronouns when giving thanks for good work, acknowledge others in meetings, "Yes, this was a tough bug, thankfully Jon got it fixed yesterday."
*Einstein is believed to have said, "Make everything as simple as possible but no simpler". May not have been the exact words
*When doing bug reports, get to the main point quickly, but include the technical details necessary. Make sure YOU can reliably recreate the error following the same steps

Calculator

Basic format:
Steps to Re-create:

Press "pi"

Press Clear

Press "1"

Press "+"

Press "1"

Press "="

Expected Results:

Display shows "2"

Actual Results

Display shows "2.00000001"

System should be declaring input and response as integers, possibly using float or another type and not doing garbage collection from previous operation?

*Being very explicit about what was pressed, in what order, matters sometimes - people have habits. On a keyboard, some use the number pad, others the main keyboard. If someone turns num lock on or off during bootup as a habit and someone else was on the computer, you could get different results.
*When describing a sequence of menu choices, the "->" characters are helpful, i.e. select Main Menu->Submenu1->SubSubmenu1
*Making comments, suggestions does not hurt if it only takes a short time. If it's similar to another known bug, point that out, too.
*Does the system have different states? What state was it in when the test was done?
* Sounds simple but can be tricky, can be easy to forget an important detail - in a web-based system, which browser were you using? Sometimes you switch between Edge, FireFox and Chrome throughout the day, remember which one you found a bug in, check and see if it happens in other browsers.
* https://obsproject.com/ - software that can be used to capture videos, including drop menus. Windows 10's Game Capture does not include drop menus.

== Other tasks follow testing ==
*Technical writers have to produce documentation like manuals. They may want to look at some of the existing documentation. That may not be the test plan, it could be developer notes or requirements. Sometimes the tester has worked with those documents a lot, and is asked to assemble them into one place for the technical writer, or directing those people where to find things.
*The technical writer might want to also be a tester. In order to write documentation that explains things to the user, the writer has to understand it well, getting their hands on a test system and trying some things helps them do this.
*The technical writer will have questions for the tester, may want assistance in configuring a system
*Sometimes testing was done to cover many situations, but the first sales are to customers with specific needs, the technical writer will be focused on that, configuring a system that way
*It helps to label things, the tester might know about the system configuration but the technical writer doesn't.
*Sometimes the technical writer has industry experience, ends up trying some additional tests and finds bugs. Then the tester follows up to write up the bug and get it fixed.

== Some professionals actively dislike textbooks ==
*Some textbooks for software testing, as is the case for books in other technical fields, have a tendency towards "perfect worldism"
*They will describe techniques and show examples of using those techniques that are very time consuming to do.
*They might be good techniques that will capture lots of bugs if you apply them
*On a real project, time is limited, you have to make judgment calls about prioritizing things to meet deadlines
*Don't want to tell your manager you spent hours on this marvellous documentation and planning technique but got no actual tests done
*So why pay attention to text books?
*Even if you can't completely apply the techniques they describe exactly as outlined, you might get ideas you can partially apply that do help raise the quality of what you deliver
*Such as - Different types of program maps - even a partial one might get you thinking about test cases you want to do, maybe apply these techniques in depth for especially important or complex parts of a system if not everything
*You may adapt/adjust these techniques to suit your situation
*Can also give you the idea of the number of test there are in theory, including paths

== On YouTube, James Bach has some interesting talks ==
*https://www.youtube.com/watch?v=ILkT_HV9DVU
*Asks people how they would test things, pushes hard for people to explain why
*Sometimes testers have ideas/intuitions that are on the right track, can you explain why?
*He talks about a system that will work 100 - 250 VAC
*So why test, say, 90 V? It's not in the specs, we know it won't work
*But will it "fail gracefully"?
*(There is not always time for that in the workplace, but if you just follow your intuition you might find important bugs)

== Bach also goes through an example and asks how many test cases would you do? ==
*Raises questions about what the project does
*Decision testing, boundary testing, predicate testing
*With more experience, you can often think of more tests right off, i.e. "improvise"
*Emphasizes the fact that a flowchart, document, etc. is a representation, not the actual system

== When do you feel pride/satisfaction in your work? ==
*Some testers say they "high-five" each other when they find a really nasty, obscure, or complicated bug
*That is when they feel they are adding a lot of value
*Not all testers agree
*For some, the time for "high-fives" are after the bug has been found, reported, fixed, and the system successfully retested, and the "high-five" is shared with the developers
*Everyone on the team wants the project to be a success!
*Also fits in with having tact and consideration for others

== Load testing ==
*After testing different parts of a system and making sure they work together well, you may want to do load testing
*Maximize the amount of transactions occurring
*Planning can be very helpful here, keep track of the order you did things, test systematically, expected results, otherwise it's too easy to get lost in a big pile of data
*This means making sure no transactions are dropped
*Or, in financial systems, totals add up, reports balance
*Testing Tools can be used for this
*Ironically, Testing Tools are not always very well tested
*They might be "quick and dirty", but as long as the tester who made them can use them to serve a main purpose well enough, then the real, main tests get done

== "Monkey Testing" ==
*Can be important
*Bach talks about examples, able to break right into a computer, produce error boxes with no text
*Using Ctrl-A Ctrl-C Ctrl-V, put a massive amount of text into an input field, see what happens
*Try moving really fast pressing buttons
*There is one big catch to this kind of testing
*Can you reliably recreate the error? If you were frantically pressing buttons fast, and timing is part of the issue, that can be very hard

== Testing similar things at once ==
*Suppose you have a system that allows you to set countdown alarms, like scheduling employees' breaks
*Each employee has a timer object
*Each object is therefore distinct
*But suppose management has a screen they use to see all employee schedules and minutes left until their breaks
*Executive can shorten or lengthen that time, they have a screen Management has access to
*That screen loads an employee day schedule object
*Copy that data into a report for that manager's daily activity, i.e.

Schedule Adjustments:
PayRoll Clerk - 10 minute delay
Production manager - 15 minute delay

Local variables cleared in between these two transactions
What if there's just one break room, an alert is sounded when your break is over.
If you have a flag that says "BreakTimeOver = true", it has to be reset

== What if functions DON'T appear to have much in common? ==
*Some functions, like above, are about tracking time
*Others keep track of that day's production, how many units produced
*Very separate things, but all part of the same system
*Run them all with a heavy load for a while, see if the system responds well, there is enough memory
*Different developers may have worked on these parts, may not have tested all together

== Button availability ==
*Many companies will use this, document it as part of a project plan
*Is important to test
*Even a small number of buttons can create many paths
*Especially important for security, don't allow login attempts without credentials being entered
*You don't have to have lots of buttons in a window for this to become time consuming and a lot of hard work

== Consistent look and feel ==
*Sometimes errors occur that are hard to spot - fonts might be very similar, but slightly different, just one size larger or smaller, shadow effects

== Resizing windows ==
*This can cause data to become lost, off screen, is a good idea to stretch windows, maximize, minimize, resize

== Logs ==
*What information is supposed to be in them? What is definitely NOT supposed to be there? Certain security things should NOT be there, check and make sure. Sometimes developers log things for debug purposes but this should be cleaned up, can be very important!

== Time ==
*Do you leave a system running overnight? Over a weekend? Over a holiday weekend? For weeks?
*Some systems do have to be running around the clock.
*Do you leave it running with a lot of activity for a long time? Is that realistic?
*If something goes wrong in the night, how hard is it to track down the cause of a problem hours later, when you get to work in the morning?

== Updating during a project ==
*Testers might be given whatever's available to start testing. This might be prototypes, partly developed items
*Updates are provided during a project
*Installing updates can take significant amounts of time
*It is important to work with developers to manage time. If several updates are coming soon, maybe wait until several are ready, install them, then resume testing rather than have multiple interruptions to install several updates
*If an update has to be applied to many components, try a few first, do at least some basic testing before investing time in updating whole system

== Graceful failures ==
*What if a component breaks, does the rest of the system respond gracefully?
*If a load test does cause errors, how well is it handled? Load tests can make systems slow, resulting in a system appearing to "hang" rather than fail gracefully

Software Testing

2019-04-11T01:35:35Z

NicholasCollins:

=== [[Software Testing]] === 

; Date: Monday, 8 April 2019 Year from 7:00pm to 9:00pm 
; Meetup Event: https://www.meetup.com/NetSquared-Kitchener-Waterloo/events/260073069/
; Location: Room 1300 -- Conrad Grebel University College, 140 Westmount Rd. N., Waterloo, Ontario {{map|url=https://osm.org/go/ZXnbg2DSE--?m=}}
 

If you work in software development, how much of your job involves testing? If you're a project manager, do you work closely with the testers to keep an eye on their results and bug reports as a project progresses? If you are a developer, do you do your own unit testing and work with testers on test plan reviews and fix the bugs they find when they do integration and regression testing? If you are a software tester, how do you balance the need to be thorough with the need to deliver on time? How does that affect writing test plans? When a project team has to deal with a lot of changes in the middle of development and testing, how do you cope with updating test plans with limited time? Since you should be considerate when reporting bugs in software, especially when people have worked hard on it, how can you do this tactfully? How do testers work with people like technical writers who use test plans as a reference when writing documentation like user manuals?

Nicholas Collins, a long-time member of KWNSPA and a professional software tester will give us an overview of the deep art of testing software.

--Bob Jonkman & Marc Paré

* [[Software Testing/Meeting Notes 2019-04-08]]

==== Resources ==== 

[https://www.techsoupcanada.ca/en/directory/334 TechSoup Canada catalogue: Developer Software]

[https://obsproject.com/ OBS Project] - software that is good for capturing videos of tests including drop menus

One of James Bach's [https://www.youtube.com/watch?v=ILkT_HV9DVU talks on YouTube]

==== Meeting Notes ====
* Introductions

* Nicholas Collins
** Software tester for a few years, knowledge of how ''his'' company works
** But two years isn't a long time compared to some software testers
** Nick has prepared notes, will be presenting slightly differently from other KWNPSA sessions
** SysAdmin in insurance industry; laid off (as are many of us); back to school to upgrade IT skills
** Uses Visual Studio, C#, other languages
** People he's met were developers, or business-specific skills; when software testers are needed these people are thrust into the role
** This might change as more universities offer software testing as a major
** There are very few courses or certificates in software testing, more prevalent in the US
*** but Fanshawe college in London has a certificate program
** Some institutions have a couple of courses in tech writing, project management, quality management; maybe a night course in software quality testing
** Without academic rigour, different people use different terminology, nomenclature
*** "Should I know what all these different terms mean?" But it's fairly common with other software testers Nick has spoken to
** At Microsoft, developers use their development skills to write tests. Needs more skills than just coding
*** Microsoft has internal courses to train testers how to test software
*** Get promoted to full developer once you've proven you can write tests
** people use Terminology like "Post-mortem" (although nobody dies), mix up "Milestone" and "Benchmark", &c.
** Software testing is the start for a developer's career, then to DevOps
*** Does this mean the most junior, inexperienced programmers are responsible for testing software?
** Nick: large companies use junior testers to run tests, senior testers to supervise
** During an upgrade Nick (a programmer at the time) did testing for the Database Analyst
*** But a junior intern was assigned to that role as well, just to gain experience.
*** Worked out details at a high level, then applied tests to get results

* Project Managers take different approaches
**

* '''You can always think of more tests'''
** It's a fine balance between staying on schedule and being thorough
** Walkthroughs and working in a team can be helpful

* Some testing instructors do not like teaching from texts
** eg. "Software Testing" by Yogesh Singh
** But Nicholas gets good ideas from texts, doesn't agree with those testing instructors
** THe problem is that the authors suffer from "Perfect Worldism"
*** A world where there is unlimited time and money, and the perfect tests can be developed
** Nicholas has experience with sticky problems, gets ideas from texts to adapt to his problem

* Party talk: Software tester does not lead to stimulating conversation
** YouTube presenter on software testing is not dull! (**********Need link!***********)

* Even the simplest test "is A < 70 ?" can have seven or eight tests
** Test results, but also overflows, boundaries, different data types
** Input validation can require many tests

* Working with other people, eg. technical writers
** For them to understand the software they'll play with the software, and may create unanticipated conditions
** Everyone can be a software tester to some degree: Project manager, developer, writer. Even sales?
** Sometimes testers find problems with usability as they're running tests; not part of the test suite

* How effective are some of these ad-hoc testers?
** Is there a bias? Do they have some incentive to pass tests even when there are problems?
*** Sometimes a QA will hold back tests that would have been better to give to the developer in the first place
** Accessibility testing is a new skill for QA, may become a testing requirement
** Business Analyst (BA), developer and tester make a good team
*** Sometimes the process of testing will identify the need for more testing

* Reporting bugs
** Requires consideration, tact

* Test plans may need to be developed quickly
** But near the end of a project when time is tight there may not be time to develop tests
** So quality of code may suffer near the end of the project
*** Breaking things during testing that no-one has time to fix

* Automated testing?
** Nick has experience with automated regression testing
** Automated regression testing reduces the introduction of new bugs

* Open Broadcaster Software
** Used to catch all activity during user testing
** Also use Virtual Box recorder uses host to capture all the output on the VM screen

* "Monkey Testing"
** Also "fuzz testing" or "fuzzing"
** Fill all fields, try to overflow, pound on the keyboard, click as fast as possible
*** But this this does not lead to reproducible errors (fine timing errors)
*** Although some testers claim they can reproduce

* Pride in finding bugs?
** Nick finds that the "high five" time should occur only after the entire team has identified, reported, documented, and fixed the error, and re-tested

* Load testing
** Hitting a system with a large number of transactions, &c.
** But a bogged down system may not be writing to logs, making analysis difficult
** A benefit in load testing is adding assertions, find issues with threads
*** Assertions and Singletons...
** Be sure to validate the output even when just testing for capacity

* Nick has written a test for XML testing
** But the code Nick wrote was not well tested at all! Oh, the irony!

* Q: Do you use debugger software like GDB to examine the flow of code?
* A: Not common, but becoming more prevalent.
** Certainly having a debugger to throw at the code is nice to have
** But much testing is done with the software under test as a black box, just examine the input and the expected output

* Nick speaks of the complexity of software testing.
** One thing works fine by itself, and other thing does too, but do they work together?
** Different software on different platform needs to interoperate, but sometimes differences in date formats causes problems
*** although each platform by itself passed all tests
** Dealing with currencies, eg. USD and CAD, and GBP
** Dealing with leap years and 29 February
** General rule: Anything date sensitive needs to test for leap years
*** and time zones! Anything dealing with calendars needs to worry about time zones

* What happens internationally when different countries need to interoperate?
** Companies have service contracts that define how the service is implemented
*** If the system is changed, the contract defines who is responsible for continued interoperation
*** If I make a change and it breaks your system, it's your fault for not defining the contract accurately
*** called "spring contracts"

* Nick gives an example from James ---- YouTube video ("nominal input voltage is 100VAC to 250VAC")
** "Test the nominal range" is an incomplete answer
** Also need to test outside the range
** The user manual may give advice not to go outside the nominal range, but users don't necessarily read the manual
** So, does the system fail gracefully outside the nominal range?
** This is the function of the software tester, to design the test to ensure that software or equipment is failsafe
*** eg. for medical equipment
*** How much money is available to fry the device under test? Some prototypes may be '''really''' expensive
*** Many examples of people damaging electronics with incorrect application of voltage!
** It's good for testers to think outside the parameters of the system

* Testing to ensure system has a consistent look and feel
** eg. fonts on some menus were different
*** Is that a software testers responsibility? Sometimes as an additional task
*** There are tools (overlays, templates) to find these issues
** Window resizing can make the application fail, but there need to be limits for those tests
** Testing for "greyed out" functions can be time consuming
*** When a function is available when it shouldn't be can result in errors
** These are general things for a tester to keep in mind

* Systems that have features which have little to do with each other
** Easy to test they're not contending for resources, &c.
** But still important to run these features simultaneous to shake loose bugs, eg. memory allocation, concurrent DB access
** Perhaps a simple monitor with limited functions: But what if something goes wrong, does the device report an error?

* Client-side data validation: All testing needs to be duplicated at the server to ensure malusers don't bypass client-side validation
** But that increases load on the server

* Logging
** Logs may indicate problems with the way the code executes, eg. repeated log entries indicate an invalid loop
** Circular reasoning: How can the logs from software under test be considered
*** Logs are only one step, begin the process of analysis
*** NewRelic will test user experience (surveillance software)

* Nick has found bugs because the test suites are well designed
** But at least half the time the bugs discovered were found in spite of the test, which was not designed to find that kind of bug

* Q&A
** Is the developer + tester model usable?
*** May be a bit scary for shops not set up for that collaborative arrangement
** Nick says to just forge ahead.
*** Having experience is good, but can also develop that experience in-house
** Worries about the coming requirements for accessibility for software
*** May take changes in coding practices (use POSH: Plain Ol' Semantic HTML instead of Javascripted forms)
*** Jurisdictional differences may be difficult to deal with

[[Category:NPSA]]
[[Category:Events]]