Package thelper

import "gitlab.com/zoralab/cerigo/testing/thelper"

thelper is a test helper package complimenting the standard “testing” package.

OBJECTIVES

Its primary goal is to extend the “testing” package by:

  1. facilitating large scale unit testing (>1000 test cases)
  2. respecting test codes like the source codes
  3. simplify any common assertions.

PROBLEMS

The biggest problem for testing in Go is that no-one mentions about large scale testing and continuous test development. Most of the tutorial guides only mention table-driven approach and shows the basic of it.

In reality, package does grow incrementally and as it grows larger, the existing test approaches becomes a nightmare to maintain.

APPROACHES

thelper is still using table-driven test approach like any other gophers. The differences are:

  1. how thelper organizes the source codes and segments of test codes
  2. use simulation test approach instead of direct testing approach
  3. the use of switches

This way, it does not limits tester’s flexibility in testing while still able some systematic test implementations across various packages.

Example

t := &testing.T{}

h := THelper{
Controller:t,
FailKeyword:"TESTFAIL",
QuietMode:false,
}

h.Logf("This is a log message")
h.Errorf("This is an error message that will fail the test")

h.Conclude()

a := []byte("Hello World")
h.ExpectExists("demo bytes", a, true)
h.ExpectExists("demo nil", nil, true)
h.ExpectExists("demo nil", nil, false)

err := fmt.Errorf("demo error")
h.ExpectError(err, true)
h.ExpectError(nil, true)
h.ExpectError(nil, false)

start := h.SnapTime()
fmt.Printf("%v jumps over %v and made it to the %v",
"James",
"Lina",
"Mountain")
stop := h.SnapTime()
myDuration := int64(300000)
minimum, maximum := h.CalculateTimeLimits(start, myDuration, 25000)
h.ExpectInTime(stop, minimum, maximum)

bSliceA := []byte{0xDE, 0xAD, 0xBE, 0xEF}
bSliceB := []byte{0xde, 0xad, 0xbe, 0xef}
h.ExpectSameBytesSlices("target", &bSliceA, "reference", &bSliceB)

sSliceA := []string{"Hello", "world"}
sSliceB := []string{"Bonjour", "World"}
h.ExpectSameStringSlices("target", &sSliceA, "reference", &sSliceB)

h.ExpectSameStrings("subject", "Hello", "reference", "Bonjour")

h.ExpectStringHasKeywords("subject", "Hello World", "keyword", "World")

h.ExpectSameBool("subject", true, "reference", true)

Constants

const (
	// AllType is the Flush configuration input that tells
	// Flush(...) to clean both Log and Failed containers.
	AllType = uint(1)

	// LogType is the Flush configuration input that tells
	// Flush(...) to clean only the Log containers.
	LogType = uint(2)

	// FailedType is the Flush configuration input that tells
	// Flush(...) to clean only the Failed containers.
	FailedType = uint(3)
)

Scenario

type Scenario struct {
	UID         int
	TestType    string
	Description string
	Switches    map[string]bool
}

Upon tested, the test suite uses its own assertion to determine the test results. Scenario relies heavily on the following standard test processes:

  1. generate all the scenarios in the list (table driven approach).
  2. loop through each scenarios: 2.1. check the TestType is meant for the test script (algorithm) 2.1.1. skip the test case if not compatible using ‘continue’ 2.2. prepare simulation environments 2.2.1. prepare happy path parameters 2.2.2. alters the parameters accordingly based on Switches. 2.2.3. generates BOTH the test inputs and the expected output. 2.3. run the test 2.3.1. execute the test based on the generated test inputs. 2.3.2. capture the test outputs. 2.4. assert the result using the generated expected output and test output. 2.5. log every details for easier assertion statement.
  3. repeat step 1 for another test script (algorithm)

To configure the simulation environment, the test suite uses the map[string]bool Switches field. It allows the tester to dynamically configure the simulation preparation functions, case by case basis.

Also, Scenario facilitates isolation for future test algorithm developments for large scale unit testing.

TController

type TController interface {
	Errorf(format string, args ...interface{})
	Logf(format string, args ...interface{})
}

TController is the interface that allows THelper to report out all the messages.

It takes 2 basic inputs:

  1. Errorf - printing by failed cases
  2. Logf - printing by passed cases

THelper

type THelper struct {
	Controller  TController
	FailKeyword string
	QuietMode   bool
	// contains filtered or unexported fields
}

THelper is a test helper structure offering various assertions, generator and some calculator functions for simplifying the testing.

This structure has 3 public elements:

  1. Controller - holding TController interface like * testing.T
  2. FailKeyword - the “FAIL” announcement that allows you to quickly search the failing cases one by one. It can be unique to you.
  3. QuietMode - instructs the helper to only print failed cases, leaving passed cases quiet.

This structure has private elements so use NewTHelper(…) function to create one instead of using the conventional structre{} method.

Func NewTHelper(t *testing.T) *THelper

NewTHelper is to create a properly defined THelper object.

It takes 1 input:

  1. t - your t *testing.T test case instructor

By default, the public interfacing elements are configured in the following manners. They are customizable after the THelper object is created and initialized.

  1. Controller - the given *testing.T object
  2. FailKeyword - “TESTFAIL”
  3. QuietMode - false

Func (h *THelper) CalculateDuration(start *time.Time, stop *time.Time) time.Duration

CalculateDuration is a function to calculate time duration from the given start and stop time.

It calculates by taking the stop time subtracting the start time. Hence, if the 2 inputs are misplaced, the return value should be negative.

It takes 2 inputs:

  1. start - start time
  2. stop - stop time

It returns:

  1. duration - time duration

Func (h *THelper) CalculateTimeLimits(start *time.Time, duration int64, ranges int64) (minimum *time.Time, maximum *time.Time)

It first calculates the expected stop time which is by adding the expected duration time (duration) to the given start time (start).

Then, It calculates:

  1. the minimum limit by subtracting the stop time from the limit range (range).
  2. the maximum limit by adding the limit range (range) into the stop time.

Providing a nil to the start time will cause the function to do nothing. returning nil for both minimum and maximum values.

It takes 2 inputs:

  1. start - the reference time
  2. duration - the expected duration for the stop time in nanoseconds
  3. ranges - the symmetric time range between the stop time in nanoseconds.

It returns:

  1. minimum - the timestamp at the stop time’s minimum limit
  2. maximum - the timestamp at the stop time’s maximum limit

Func (h *THelper) Conclude()

Conclude processes the message containers and pretty print them out under a single output message.

Conclude uses the internal TController interface to perform the printing. The outcome can be either:

  1. using the Errorf(…) when there is any logged error/failed messages
  2. using the Logf(…) for normal passed cases' messages

If QuietMode is true for the #2, Conclude() will not print anything.

Upon successful output, the THelper structure automatically runs Flush(AllType) function, setting itself clean for next test run.

Func (h *THelper) Errorf(format string, a ...interface{})

Errorf is to print an error message reporting to the T testing controller.

It works similarly with testing.T.Errorf(…) function from testing package with a few differences:

  1. messages are queued, only printed out when THelper.Conclude() (a.k.a. concluding the test) is made.

Func (h *THelper) ExpectError(subject error, expected bool) int

ExpectError is an assertion to compare an error object existence.

It takes 2 inputs:

  1. subject - the error test subject
  2. expected - the expectation for the subject existence.

If the test subject is behaving outside of the given expectation, this function automatically logs the error message into the THelper.

It returns:

  1. 0 - passed assertion
  2. non-0 - failed assertion

Func (h *THelper) ExpectExists(label string, subject interface{}, expected bool) int

ExpectExists is an assertion to check a given object is nil or exist.

It takes 3 inputs:

  1. label - for error logging purposes, naming the asserted object.
  2. subject - the test subject being asserted.
  3. expected - the expectation for the subject existence. True means it is expected to exist.

This function automatically logs the error message into the THelper if the test subject is behaving outside of expectation.

It returns:

  1. 0 - passed assertion
  2. non-0 - failed assertion

Func (h *THelper) ExpectInTime(subject *time.Time, minimum *time.Time, maximum *time.Time) int

ExpectInTime is an assertion to check a given time is within a given set of time limits.

It takes 1 main input and 2 optional inputs. They are:

  1. subject - the given test subject
  2. minimum - the minimum time limit (optional)
  3. maximum - the maximum time limit (optional)

Depending on the availability of minimum and maximum, ExpectInTime carries out the assertion differently:

  1. If no limit is available, it always return true since the test subject is always in range.
  2. If only maximum is available, it will check the test subject is always below tha maximum limit. It will assert failed if it went over.
  3. If only minimum is available, it will check the test subject is always above the minimum limit. It will assert failed if it went under.
  4. If both maximum and minimum are available, it will check the test subject falls within range. It will assert failed if it went outside.
  5. If test subject is not available, it will assert an error.

ExpectInTime automatically logs the error message when meeting a failed assertion. If everything is good, ExpectInTime does nothing.

It returns:

  1. 0 - assertion is a passed case
  2. not 0 - assertion is a failed case

Func (h *THelper) ExpectSameBigFloat(labelA string, subjectA *big.Float, labelB string, subjectB *big.Float) int

ExpectSameBigFloat is an assertion to compare 2 math/big.Float are same.

If the assertion fails (such as the string pair are not the same), this function automatically logs an error message using the 2 labels.

It takes 4 inputs:

  1. labelA - label for first (a) bool
  2. a - the first *big.Float
  3. labelB - label for second (b) bool
  4. b - the second *big.Float

It returns:

  1. 0 - passed assertion (including a == b == nil)
  2. 1 - failed assertion

Func (h *THelper) ExpectSameBool(labelA string, subjectA bool, labelB string, subjectB bool) int

ExpectSameBool is an assertion to compare 2 given bools are having the same contents.

If the assertion fails (such as the string pair are not the same), this function automatically logs an error message using the 2 labels.

It takes 4 inputs:

  1. labelA - label for first (a) bool
  2. a - the first bool
  3. labelB - label for second (b) bool
  4. b - the second bool

It returns:

  1. 0 - passed assertion
  2. 1 - failed assertion

Func (h *THelper) ExpectSameBytesSlices(labelA string, a *[]byte, labelB string, b *[]byte) int

ExpectSameBytesSlices is an assertion to check 2 given bytes slices are the same.

It checks for nil, length, and lastly byte by byte before returning a passing result. It automatically log the error message when encountering a failed assertion.

If a and b bytes slices are nil, ExpectSameBytesSlices passes the assertion immediately since nil is always the same as nil.

It takes 4 inputs:

  1. labelA - label for first slice (A) reporting purpose
  2. a - the first byte slice (A)
  3. labelB - label for second slice (B) reporting purpose
  4. b - the second byte slice (B)

It returns:

  1. 0 - passed assertion
  2. 1 - A is nil but B is not nil
  3. 2 - A is not nil but B is nil
  4. 3 - both A and B has different length
  5. 4 - incorrect data matching during byte-to-byte comparison

Func (h *THelper) ExpectSameFloat32(labelA string, subjectA float32, labelB string, subjectB float32) int

ExpectSameFloat32 is an assertion to compare 2 given float32 are same.

If the assertion fails (such as the string pair are not the same), this function automatically logs an error message using the 2 labels.

It takes 4 inputs:

  1. labelA - label for first (a) bool
  2. a - the first float32
  3. labelB - label for second (b) bool
  4. b - the second float32

It returns:

  1. 0 - passed assertion
  2. 1 - failed assertion

Func (h *THelper) ExpectSameFloat64(labelA string, subjectA float64, labelB string, subjectB float64) int

ExpectSameFloat64 is an assertion to compare 2 given float64 are same.

If the assertion fails (such as the string pair are not the same), this function automatically logs an error message using the 2 labels.

It takes 4 inputs:

  1. labelA - label for first (a) bool
  2. a - the first float64
  3. labelB - label for second (b) bool
  4. b - the second float64

It returns:

  1. 0 - passed assertion
  2. 1 - failed assertion

Func (h *THelper) ExpectSameStringSlices(labelA string, a *[]string, labelB string, b *[]string) int

ExpectSameStringSlices is an assertion to check 2 given string slices' contents are the same.

It checks for nil, length, and lastly line by line before returning a passing result. It automatically logs the error message when encountering a failed assertion.

If a and b bytes slices are nil, ExpectSameStringSlices(…) passes the assertion immediately since nil is always the same as nil.

It takes 4 inputs:

  1. labelA - label for first slice (A) reporting purpose
  2. a - the first string slice (A)
  3. labelB - label for second slice (B) reporting purpose
  4. b - the second string slice (B)

It returns:

  1. 0 - passed assertion
  2. 1 - A is nil but B is not nil
  3. 2 - A is not nil but B is nil
  4. 3 - both A and B has different length
  5. 4 - incorrect data matching during line-by-line comparison

Func (h *THelper) ExpectSameStrings(labelA string, subjectA string, labelB string, subjectB string) int

ExpectSameStrings is an assertion to compare 2 given strings are having the same contents.

If the assertion fails, this function automatically logs an error message when using the 2 labels.

It takes 4 inputs:

  1. labelA - label for first (a) string
  2. a - the first string
  3. labelB - label for second (b) string
  4. b - the second string

It returns:

  1. 0 - passed assertion
  2. 1 - failed assertion

Func (h *THelper) ExpectStringHasKeywords(labelA string, subject string, labelB string, substring string) int

ExpectStringHasKeywords is an assertion to check a given sub-string is inside the string subject.

If the assertion fails (such as missing substring), this function automatically logs an error message using the 2 labels.

It takes 4 inputs:

  1. labelA - label for the subject
  2. subject - the subject string
  3. labelB - label for the substring
  4. substring - the sub-string

It returns:

  1. 0 - passed assertion
  2. 1 - failed assertion

Func (h *THelper) ExpectUIDCorrectness(index int, uid int, startFromZero bool) (verdict int)

ExpectUIDCorrectness is to check the current index against the test case uid.

If there is a mismatch, it automatically log a failed message.

It takes 3 inputs:

  1. index - the current index number of the test cases (for loop)
  2. uid - the uid of the test case from the table driven testing
  3. startFromZero - to state whether the uid starts from 0.

It returns:

  1. 0 - successful assertion
  2. non-zero - failed assertion

Func (h *THelper) Flush(t uint)

Flush is to reset all message containers in a THelper structure to initial conditions for next run.

Flush is made available in case of manual interventions.

Func (h *THelper) LogScenario(s Scenario, data map[string]interface{})

LogScenario is to log a compatible Scenario object holding the test case meta information.

It utilizes THelper.Logf(…) to printout the the scenario elements in a standardized way. If a test suite defined a Scenario derivative types, the Scenario type conversion should happen in that test suite itself.

LogScenario facilitates data map parameter that allows tester to print out any data alongside the logging process. The string label of the list is the field name while the value can be anything since it is a interface{}.

The data value access is unsafe (without using any mutexes). Hence, it is the test suite responsibility to ensure the data concurrency safety before feeding into LogScenario data field.

If data map list is not available (nil), LogScenario will stop printing after switches.

Func (h *THelper) Logf(format string, a ...interface{})

Logf is to print a log message reporting to the T testing controller.

It works similarly with testing.T.Logf(…) function from testing package with a few differences:

  1. messages are queued, only printed out when THelper.Conclude() (a.k.a. concluding the test) is made.

Func (h *THelper) SnapTime() *time.Time

SnapTime is a function to snap the current time.

It returns:

  1. Time - time.Now() object.