Writing automatic functional tests for Rogo
- Neill Magill
Feature files
Where feature files should go
A sub-directory of: testing/behat/features
It is recommended that you place them in a sub directory based on the main area of Rogo you are testing, for example if you were testing that the details of a paper could be updated and had the desired effect you might place it in the papers sub-directory.
Feature file coding standards
All Feature files must have a Feature definition, all tests in the file must fit into this description. We prefer more files to unrelated tests in the same file.
Indenting in the tests must be two spaces per level of indentation.
There should be no trailing white space
@feature_wide_tags
Feature: Short descriptive title
Description of what the users want to do
As a type of user
Description of what the user needs to be able to do
Background:
Given this step runs in all scenarios in the file
@scenario_specific_tags
Scenario: Descriptive scenario title
Given the following "users" exist:
| username |
| student1 |
When I login as "student1"
Then I should see "No exams found" "content"
For more details about the Gerkin language used in feature files see http://docs.behat.org/en/v3.0/guides/1.gherkin.html
Tags
Tags can be used to define the area of functionality a feature or scenario is testing, feel free to use them liberally. For example if your test is about the creation of creating Likert questions you may tag the test with @question and @likert, this would allow us to easily run any tests for questions in general or Likert questions in particular.
There are some tags that have special meanings beyond this:
| Tag | Description |
|---|---|
| @javascript | The Scenario will use the Selenium2 webdriver and be able to use Javascript.Tests that use this tag will be slower than tests that do not. This tag should not be mixed with the @backend tag |
| @backend | The test will be run by the backend TestSuite. |
| @attachfile | Any test that uploads files should use this tag. It will allow us not to run them if we are doing testing in an environment that cannot do uploads (or a driver that does not support attaching files). |
| @iframe | Any tests that run on pages within rogo that have iframes should have this tag. It will allow us not to run them if we are doing testing using a driver that does not support iframes. |
| @jsevaluation | Any tests that use steps that require javascript evaluation. It will allow us not to run them if we are doing testing using a driver that does not support javascript evaluation. |
| @mousemanipulation | Any tests that use steps that require mouse manipulation. It will allow us not to run them if we are doing testing using a driver that does not support mouse manipulation. |
Finding available steps
There are two commands you may use to find available steps:
The just lists the commands:
vendor\bin\behat --config <path_to_rogo>\testing\behat\config\behat.yml -dl
The output first tells you which test suite the step belongs to and then after the pipe tells you the step. note that Given, Then, And, But, etc. are interchangeable. If a step starts with /^ and ends with $/ it is a regular expression.
The second gives a detailed list that includes information about what the step does:
vendor\bin\behat --config <path_to_rogo>\testing\behat\config\behat.yml -di
The first line of each block has the same meaning as the simple list, the next few lines describe what the step does.
Test suites
The version of Behat used in Rogo allows us to define multiple test suites we have two defined.
frontend
This is the default test suite, it uses a browser to simulate student interaction with the user interface of Rogo, either via a real browser or a headless browser
backend
This suite is used if the test is tagged with @backend. It allows test to use PHP unit functions. It should also allow steps to share data between each other using variables that are automatically cleaned between tests with a custom data store
Using the custom data store
The backend test suite will automatically interpret any calls to any non-class defined properties as being a call to the custom data store. So for example if in a step you do:
$this->myvariable = 5;
In a later step of the same test you could call $this->myvariable and it would return 5
isset() and unset() will work on variables stored in the custom data store
Step definitions
The steps definitions hold the code that makes the feature files work.
In Rogo we define our steps in PHP traits
Where step definitions should be located
A sub-directory of: testing/behat/steps
Currently there are 3 sub-directories:
| Directory | Notes |
|---|---|
| backend | The steps that use PHP Unit functions to test code directly should be placed here |
| database | Steps that generate data for Rogo by adding it directly into the database should be placed here |
| frontend | The steps that interact with Rogo via a web browser should be placed here |
Each file in the sub directory should contain a single trait that defines steps related to a specific area, for example the frontend forms trait should contain steps for interacting with html forms.
The frontend and backend sub directory also have an include_ trait that should use all the other traits.
Creating data
There is a special step that can be used to add data to the Rogo database.
Given the following "users" exist: | username | | student1 |
The data type is passed as a parameter i.e. "users" above.
The table passed must include data for all required fields for the data generator, it may include any number of optional fields. If an optional field is omitted a default or random value should be generated for it.
Currently the following data generators are available via this method:
| Name | Required fields | Optional fields |
|---|---|---|
| users | username | surname, first_names, title, email, roles, gender, special_needs, yearofstudy, user_deleted, password_expire, grade, initials, password |
| questions | user, type + question type specific requirements | are question type specific from the following list: theme, leadin, notes, display_method, ownerID, q_media_id, q_media, q_media_width, q_media_height, q_media_alt, q_media_owner, q_media_num, creation_date, last_edited, bloom, scenario, scenario_plain, leadin_plain, checkout_time, checkout_authorID, deleted, locked, std, status, q_option_order, score_method, settings, quid, keywords, options |
| modules | moduleid, fullname | active, schoolid, vle_api, sms_api, selfEnroll, peer, external, stdset, mapping, neg_marking, ebel_grid_template, sms_import, timed_exams, exam_q_feedback, add_team_members, map_level, academic_year_start, externalID |
| module team members | moduleid, username | not applicable |
| module keywords | moduleid | keyword |
| papers | papertitle, papertype, paperowner, modulename | startdate, enddate, labs, duration, session, timezone, externalid, externalsys, calendaryear, remote, password |
Where data is expected as an error in a data generator or step we pass this in the feature file as a json string i.e.
@question_true_false
Scenario: Create a true_false question
Given I login as "teacher"
And I follow "TEST1001"
And I select a "true_false" question type
And The upload source path is "questions/media"
And I create a new "true_false" question:
| theme | textbox theme |
| notes | textbox notes |
| scenario | textbox scenario |
| leadin | textbox leadin |
| true | 0 |
| file | {"filename":"frog.jpg","alt":"alternate text"} |
When I click "Add to Bank" "button"
Then I should see "Module: TEST1001" "paper_title"