Software Testing Heuristics and mnemonics.

Software Testing Heuristics

Heuristics are simply experience-based techniques for problem-solving, learning, and discovery. Where an exhaustive search is impractical, heuristic methods are used to speed up the process of finding a satisfactory solution. Examples of this method include using a rule of thumb, an educated guess, an intuitive judgment, or common sense.

When you face a problem, you try some solutions that may work. For instance, if you suddenly find your mobile hang, what do you do? You may restart your mobile first. If that doesn’t work, then you may check & update the latest software and uninstall suspicious apps.if that also doesn’t work, then you reset your phone setting ( Factory reset). Most of us try a number of steps to solve our problems. These steps may not always work but we try them because we know that they may work.

Following heuristics can apply to absolutely any kind of system with any kind of interface.

 1.Abstract

 Remove details in a model. Simplify by omitting low-level, granular bits so that the model focuses less on specifics and more on concepts. For example, with an interaction diagram you could replace details about a set of communications with a single line named for the essential interaction that the communications represent. If you were working on an email client, the steps that the client goes through to send an email include connecting to the server, authenticating, and transmitting the data. Instead of identifying each of these steps in your model, you can remove the details and raise the level of abstraction by simply labeling the entire client/server interaction as “Send Email.” 

2.Never and Always

 The things that the software should always do (such as ensure that accounts balance) or never do (such as destroy user data). To discover the Nevers and Alwayses for your system, talk with subject matter experts and business stakeholders.

 3.Beginning, Middle, End 

Vary the position of an element. You could vary where a given element appears in a sequence, or you could act on an element in a given position. Examples of applying this heuristic include the following: 

• Deleting an item from the beginning, middle, or end of a list

• Pasting text at the beginning, middle, or end of a line

• Placing a special character at the beginning, middle, or end of a value 

• Performing an action at the beginning, middle, or end of a sequence 

Watch for indications that position makes a difference. Also be alert for bad behavior, such as data appended at the end being truncated or data inserted at the beginning or in the middle causing other contents to be truncated.

4.Centralize Everything

If things are scattered, subdivided, or organized in separate bins, put them all in one place. Examples:

• For software where items can be organized into folders: Take a large collection of folders with items in them and move everything into a single folder or a single deep hierarchy of folders.

• For software where ownership of an item can be transferred between accounts: Transfer ownership of a large number of things from varying accounts into a single account.

5.Change the Model

Take a model of the system you’re testing and translate it into a different representation of the model. Examples:

• Change a state diagram with circles and arrows representing the states and transitions into a table showing states as columns, events as rows, and resulting states as data in the cells.

• Change a linear outline into a mindmap, or a mind map into a linear

outline.

6.CRUD

Create. Read. Update. Delete. These are the essential actions you can take on any data element within a system. Use CRUD in conjunction with other heuristics. For example:

• With Beginning, Middle, End: Create an element at the beginning of a list, in the middle of a list, or at the end of a list.

• With Goldilocks: Update a data element with a value that is too small, too big, or just right.

• With Zero, One, Many: Delete an item with no child records, with one child record, or with many child records.

7.Decentralize Everything

If things are all in one place, scatter them, subdivide them, or organize them into separate bins. Also, make connections across distributed systems. Examples:

• For software where items can be organized into folders: Create a large number of folders, each with a few items.

• For software that can be configured to work in a distributed environment: Configure the system so that each part exists on its own separate machine and separated from the others with subnets or firewalls.

8.Follow the Data

Perform a sequence of actions involving data, verifying the data integrity at each step. This is appropriate for any system where users can perform activities on data like entering, searching, reporting, importing, exporting, updating, and viewing. Following the data is particularly useful when combined with other data-related heuristics. For example:

• With data that has special characters, including SQL injection or JavaScript injection attacks: Import it. Search on it. View it. Run reports that include it. Export it.

• Create data that violates domain-specific rules and then import it. Sometimes the logic that prevents bad data from being entered directly is missing from batch import utilities.

• With Zero, One, Many: Perform data-centric actions such as running reports or exporting data with entities that have no child records, one child record, or many child records.

9.Goldilocks

Too big, too small, or just right. This can apply to anything that has a valid range of sizes, such as a range of dates, a numerical value, the length of a string or text, or the size of a file. Behavior to watch for includes the following:

• Unhelpful error messages, such as raw exceptions with stack traces or mysterious error codes

• Truncating large data without notice

• Failing to save data without notice

10.Interrupt

Find ways to interrupt key processes, such as logging off, shutting down the machine, rebooting the machine, killing the process from the operating system, disconnecting the machine from the network, forcing the machine into hibernation, letting the session timeout, and cancelling the operation. Watch for misbehavior, such as the following:

• Unhelpful error messages, such as raw exceptions with stack traces or mysterious error codes

• Inability to start the process (ever) again

• Unrecoverable data loss or corruption

11.Reverse

Do things in reverse order. Undo everything. Skip to the end and work backward. Examples:

• For software with an Undo feature: Perform a wide variety of actions, then undo all your work step by step.

• For software with a defined workflow: Accept all the default values to get to the end of the workflow; then use the back button to go back to each screen and change the values.

12.Some, None, All

Some of the items in a set, none of them, all of them. Related to 0, 1, Many but for defined sets of items or attributes such as permissions, configuration options, tags, multi select fields, and checkboxes. Behavior to watch for includes the following:

• None treated the same as All—for example, the system treating a user with no permissions the same as a superuser with all permissions

• Problems with count or percentage calculations, including divide-by zero and off-by-one errors

• Display issues, particularly with None or All

• Data truncation with All

13.Starve

Starve the program of any required resource, including CPU, memory,

network bandwidth, or disk space. Watch for these responses:

• Unrecoverable data loss or corruption

• Unhelpful error messages, such as raw exceptions with stack traces or mysterious error codes

• Unexpected program or session termination

14.Too Few

Applies when you have counts of things. Create conditions with fewer things than the software expected. For example, have too few items to populate a list or too few line items on an invoice.

15.Too Many

Applies when you have counts of things. Create conditions with too many things, such as too many active sessions or connections.

16.Useful Approximations

Especially if you are dealing with a complex domain, it may be impossible to determine exactly what the expected result is for a given experiment. In that case, you can use an approximation to assess correctness. You can evaluate results against a range of possible values to see if it falls inside the range. You might also check the characteristics of the results, such as whether the computed values are increasing or decreasing.

17.Violate Data Format Rules

The software expects data to conform to rules. Ages should never be negative. IP addresses should always contain four numbers ranging from 0 to 255. Email addresses should always have a username separated from the domain by an @ sign. Custom file formats have a specific structure. Try violating these rules and see how gracefully (or not) the software handles the badly formatted data.

18.Zero

Cem Kaner’s law of zero is that if there is a zero, something will attempt to divide by it. It also turns out that when software is expecting a set of things, it frequently does not handle the empty set (zero elements) well.

19.Zero, One, Many

Zero things, one thing, many things. The things you are counting might include results returned in a search, characters in a string, bytes in a data stream, descendants in a hierarchy, a number of files, lines within a file, accounts, activities in a log, or any domain-specific object or attribute. Behavior to watch for includes these issues:

• Problems with plurals, such as “0 record found” or “1 records found”

• Problems with count or percentage calculations, including divide-by zero and off-by-one errors • Indications of performance issues as the count of things increases

 20.Zoom 

Focus on the concrete details, narrowing your field of vision if necessary.  For example, with a state model, you could identify an event in your model and identify substrates by breaking down the steps the software takes in transitioning from one state to the next. For instance, with software that allows users to create and  update items, you could take a Save event and break it down into steps such as transmitting data, validating data, and responding. Each of these steps is a state with events that trigger the transition between them.

Software Testing Mnemonics

According to Wikipedia “ A mnemonic device is a mind memory and/or learning aid. Mnemonics rely on associations between easy-to-remember constructs which can be related back to the data that is to be remembered.” Following are the examples of some testing mnemonics.

HEENA

• History

• Explore

• Experiment

• Experience

• Note Taking

• Analyse

CRUSSPIC STMPL – by James Bach

Operational Criteria - CRUSSPIC

• Capability

• Reliability

• Usability

• Security

• Scalability

• Performance

• Installability

• Compatibility

Development Criteria – STMPL

• Supportability

• Testability

• Maintainability

• Portability

• Localizability

SCAMPER

A mnemonic useful for questioning existing products and coming up with creative ideas. 

• Substitute

• Combine

• Adapt

• Modify

• rePurpose

• Eliminate

• Rearrange

DUFFSSCRA – Test Techniques Heuristics by James Bach

• Domain

• User

• Function

• Flow

• Stress

• Scenario

• Claims

• Risk

• Automatic

HICCUPPSF – Test Oracles by Michael Bolton

This mnemonic is particularly useful in the context of exploratory testing or any situation where the behaviour of the software is not well specified.

• History

• Image

• Comparable Product

• Claims

• User Expectations

• Product

• Purpose

• Standards

• Familiarity

RIMGEA – Bug Advocacy Mnemonic by Cem Kaner which is useful to communicate bug related information to stakeholders

• Replicate it

• Isolate it

• Maximize it

• Generalize it

• Externalize it

• And Say it Clearly and Dispassionately

MCOASTER – Test Reporting Heuristics by Michael D Kelly

• Mission

• Coverage

• Obstacles

• Audience

• Status

• Techniques

• Environment

• Risk

FAILURE – Error Handling Heuristics by Ben Simo.

• Functional

• Appropriate

• Impact

• Log

• UI

• Recovery

• Emotions

RCRCRC – Regression Testing Heuristics by Karen N. Johnson helpful to generate ideas during regression testing

• Recent – Think about the recent changes

• Core – Think about the core functionality that should continue to work

• Risk – Think about the areas that are risky due to the recent changes

• Configuration sensitive – Think about environment dependencies during testing

• Repaired – Think about areas that require testing due to recently fixed defects

• Chronic – Think about areas that have issues seen frequently

W5HE – Requirements Analysis and Feedback Mnemonic by Darren McMillan.

• Who

• What

• When

• Where

• Why

• How

• Experience

GRATEDD SCRIPTS – Test Strategy Mnemonic by Jared Quinert

• Goals – What must work in the product?

• Risks – What are the risky areas in the product?

• Approach – What approach would we take in testing the product?

• Tradeoffs – What tradeoffs are we prepared to make?

• Environments – What environments do we have and what do we plan to use for testing?

• Dependencies – We need to understand any external and internal dependencies during testing.

• Data – Where will data come from?

• Stakeholders – Understand who the stakeholders are for a project

• Coverage – Thing about how the test coverage is going to be made visible to the project stakeholders.

• Resources – Are the available resources enough for testing a feature?

• Information needs – What are we trying to know from the testing task?

• Prioritisation – What is most important?

• Tooling – What are the tools we plan to use during testing? Are they available in house?

SFDPOT (San Francisco Depot) – Useful mnemonic by James Bach for exploratory testing under pressure

• Structure (what the product is)

• Function (what the product does)

• Data (what it processes)

• Platform (what it depends upon)

• Operations (how it will be used)

• Time (what happens when)

PROOF – Session Based Test Reporting Mnemonic by Jon Bach

• Past

• Results

• Obstacles

• Outlook

• Feelings