Testing framework

[ladc]

Agree on a testing library and a test runner with minimal dependencies, preferably Lua/shell only.
Test cases should still be easy to run without any (big) dependencies.

[agentzh]

As mentioned in #5, I hope that with the data driven approach, we can provide an official test runner (or test harness) but also allow 3rd-party more sophisticated test runners to run exactly the same test suite without modifications. The test cases themselves should be test framework agnostic and stay very clean and self-contained. This also has the advantage that there is no way that a Lua test framework library itself can affect the behaviors of the test cases in any way if we wish.

[ladc]

@agentzh I've made a proof-of-concept test runner in the spirit of TestML and following the guidelines by @MikePall in his comment to issue #5. I've adapted some tests as an example (in the directories test/libs and test/libs_ext).

Like ldoc it considers the --- as special, in this case a marker for a new test. The first line defines the test name. Subsequent comments preceded by -- contain the description, and tags are added in the description +this +way. The test file's path is also tokenized and added to the tags.

The tags allow you to select or exclude certain tests, this allows for 'tiers' (slow/fast/lua52 etc.).

It also considers the code before the first --- identifier as a prelude which should be included before each subsequent test.

It's here: https://github.com/ladc/LuaJIT-test-cleanup
Try running

luajit test.lua test/libs +table -lua52

or

luajit test.lua --help

Let me know what you think.

It still has a lot of TODOs of course, like the filenames used to extract failing code snippets to (currently uses os.tmpname()), etc.

[fsfod]

It seems kind of fragile trying to parse out the bodys of the test and append what you hope the helper functions are instead of just rewriting all the individual tests in a file into functions that you can just selectively call

[ladc]

@fsfod Thank you for your feedback. I tried to find a solution that's minimally invasive to the tests, while avoiding the need to include a YAML parser or the like, and while allowing the tests to be run as-is.

Of course a declarative framework like this would need a minimal specification for the tests' format, which would include that the chunk before the first --- contains the helper functions. And some tests would need to be adapted to comply to this.

[Wiladams]

I wonder why lua itself could not be the test format? Why rely on something else, when lua is perfectly capable of doing the job?

Sent from my Windows Phone

---

From: Lesley De Cruzmailto:notifications@github.com
Sent: ‎2/‎22/‎2016 1:49 PM
To: LuaJIT/LuaJIT-test-cleanupmailto:LuaJIT-test-cleanup@noreply.github.com
Subject: Re: [LuaJIT-test-cleanup] Testing framework (#3)

@fsfod Thank you for your feedback. I tried to find a solution that's minimally invasive to the tests, while avoiding the need to include a YAML parser or the like, and while allowing the tests to be run as-is.

Of course a declarative framework like this would need a minimal specification for the tests' format, which would include that the chunk before the first --- contains the helper functions. And some tests would need to be adapted to comply to this.

---

Reply to this email directly or view it on GitHub:
#3 (comment)

[ladc]

@Wiladams That would require more invasive changes to the tests, and no longer make them runnable as-is. Or it would require putting every (sub)test into a different file, which is not ideal either.

[CapsAdmin]

Coming up with names for all the subtests seems a bit difficult if they were to be split into files but I guess you could just name them table/insert1.lua table/insert2.lua etc instead.

I think parsing a test for subtests is fine. I would match subtests by balanced do end and leave the top part of the test as shared code. For instance https://github.com/LuaJIT/LuaJIT-test-cleanup/blob/master/test/ffi/ffi_new.lua has 10 sub tests where line 1 to 11 would be each subtests header. Then moving your comment config thing inside do end or left ofdo` while adding a symbol of some sort to tell the people reading the code that this is something that's being parsed would make it seem a little less "fragile".

I don't know how important it is to not have dependencies but if you're going to use a stable version of luajit to run the tests you could just use ffi to iterate directories instead of relying on lfs. But that's something that should probably be looked at later.

[ladc]

OK, the --- divisor is indeed a bit fragile since the subtests can still leak (local) variables if they don't have do ... end delimiters. How would you match this in a robust way without adding too much machinery?

It's indeed better to drop the dependency on lfs in favour of an ffi solution (see ljsyscall and TINN).

[ladc]

I wrote

It's indeed better to drop the dependency on lfs in favour of an ffi solution (see ljsyscall and TINN).

OTOH if you're not testing a stable LuaJIT (or if you're in the process of porting it) you may appreciate a test runner that works in vanilla Lua...

[Wiladams]

Or minilua

Sent from my Windows Phone

---

From: Lesley De Cruzmailto:notifications@github.com
Sent: ‎2/‎26/‎2016 3:22 PM
To: LuaJIT/LuaJIT-test-cleanupmailto:LuaJIT-test-cleanup@noreply.github.com
Cc: William Adamsmailto:william_a_adams@msn.com
Subject: Re: [LuaJIT-test-cleanup] Testing framework (#3)

I wrote

It's indeed better to drop the dependency on lfs in favour of an ffi solution (see ljsyscall and TINN).

OTOH if you're not testing a stable LuaJIT (or if you're in the process of porting it) you may appreciate a test runner that works in vanilla Lua...

---

Reply to this email directly or view it on GitHub:
#3 (comment)

[ladc]

@Wiladams I also considered minilua. But keep in mind that minilua lacks some core libraries and functions. If I understand correctly from genminilua.lua the following list is not included:

  collectgarbage dofile gcinfo getfenv getmetatable load print rawequal rawset
  select tostring xpcall
  foreach foreachi getn maxn setn
  popen tmpfile seek setvbuf __tostring
  clock date difftime execute getenv rename setlocale time tmpname
  dump gfind len reverse

[CapsAdmin]

@ladc instead of trying to come up with a parser that matches do end you do something like

--#header foo {
local ffi = require("ffi")
local bit = require("bit")

dofile("../common/ffi_util.inc")

ffi.cdef([[
typedef struct { int a,b,c; } foo1_t;
typedef int foo2_t[?];
void *malloc(size_t size);
void free(void *ptr);
]])
--#}

do 
  --#subtest test {
  --#include foo
  assert(ffi.sizeof("foo1_t") == 12)
  local cd = ffi.new("foo1_t")
  assert(ffi.sizeof(cd) == 12)
  local foo1_t = ffi.typeof("foo1_t")
  assert(ffi.sizeof(foo1_t) == 12)
  cd = foo1_t()
  assert(ffi.sizeof(cd) == 12)
  --#}  
end 

do --#subtest test2 {
  --#include foo
  assert(ffi.sizeof("foo2_t", 3) == 12)
  local cd = ffi.new("foo2_t", 3)
  assert(ffi.sizeof(cd) == 12)
  local foo2_t = ffi.typeof("foo2_t")
  fails(ffi.sizeof, foo2_t)
  assert(ffi.sizeof(foo2_t, 3) == 12)
  cd = foo2_t(3)
  assert(ffi.sizeof(cd) == 12)
  --#}  
end 

do --#subtest test2 {
  --#include foo
  local tpi = ffi.typeof("int")
  local tpb = ffi.typeof("uint8_t")
  local t = {}
  for i=1,200 do t[i] = tpi end
  t[100] = tpb
  local x = 0
  for i=1,200 do x = x + tonumber(ffi.new(t[i], 257)) end
  assert(x == 199*257 + 1)
  --#}  
end

I think the prefix symbol makes it clear that this is something more than a comment.

Instead of making a pseudo language you could use lua itself somehow too which at that point it becomes a generic macro thing. (--#header = [[ ... --#]] ... --#test1 = [[ ... --#include(header) ... --#]]) but I'm not sure if there's going to be much benefit or if it'll make things easier.

[Wiladams]

Here's an example of what I'm thinking. Just deal with the test file as if it were a database of individual test cases.

This is all pure lua, so a lua parser can deal with it. And if you want to split thing out, you can easily do that by just

querying the table, and only running what you want.

The added benefit is that you get more meta data associated with tests. So, if you want to run tests that are related to a particular area, you can easily do that.

I just want to separate the difference between data, and the mechanism to run the tests. The data itself can easily be represented in lua itself, minimizing the need to create a different 'language' simply to represent the data.

local testCases = {
{

id = "constov1",

desc = "test case to catch issue 2345",

author = "williamaadams",

issue = "2345",


[[
  local t = { "local x\n" }
   for i=2,65537 do t[i] = "x="..i..".5\n" end
  assert(loadstring(table.concat(t)) ~= nil)
   t[65538] = "x=65538.5"
   assert(loadstring(table.concat(t)) == nil)

]]

},

{

id = "constov2",

desc = "test case to catch issue 2346",

issue = "2346",

[[

  local t = { "local x\n" }


  for i=2,65537 do t[i] = "x='"..i.."'\n" end


  assert(loadstring(table.concat(t)) ~= nil)


  t[65538] = "x='65538'"


  assert(loadstring(table.concat(t)) == nil)

]]

}
}

if not os.getenv("SLOWTEST") then return end

-- Run all the test cases

for tcase in ipairs(testCases) do

for idx, tcase in ipairs(tcase) do

dostring(tcase)

end

end

=============================== - Shaping clay is easier than digging it out of the ground.

Date: Sat, 27 Feb 2016 06:08:52 -0800
From: notifications@github.com
To: LuaJIT-test-cleanup@noreply.github.com
CC: william_a_adams@msn.com
Subject: Re: [LuaJIT-test-cleanup] Testing framework (#3)

@ladc instead of trying to come up with a parser that matches do end you do something like

--#header foo {
local ffi = require("ffi")
local bit = require("bit")

dofile("../common/ffi_util.inc")

ffi.cdef([[
typedef struct { int a,b,c; } foo1_t;
typedef int foo2_t[?];
void *malloc(size_t size);
void free(void *ptr);
]])
--#}

do
--#subtest test {
--#include foo
assert(ffi.sizeof("foo1_t") == 12)
local cd = ffi.new("foo1_t")
assert(ffi.sizeof(cd) == 12)
local foo1_t = ffi.typeof("foo1_t")
assert(ffi.sizeof(foo1_t) == 12)
cd = foo1_t()
assert(ffi.sizeof(cd) == 12)
--#}
end

do --#subtest test2 {
--#include foo
assert(ffi.sizeof("foo2_t", 3) == 12)
local cd = ffi.new("foo2_t", 3)
assert(ffi.sizeof(cd) == 12)
local foo2_t = ffi.typeof("foo2_t")
fails(ffi.sizeof, foo2_t)
assert(ffi.sizeof(foo2_t, 3) == 12)
cd = foo2_t(3)
assert(ffi.sizeof(cd) == 12)
--#}
end

do --#subtest test2 {
--#include foo
local tpi = ffi.typeof("int")
local tpb = ffi.typeof("uint8_t")
local t = {}
for i=1,200 do t[i] = tpi end
t[100] = tpb
local x = 0
for i=1,200 do x = x + tonumber(ffi.new(t[i], 257)) end
assert(x == 199*257 + 1)
--#}
end

I think the prefix symbol makes it clear that this is something more than a comment.

Instead of making a pseudo language you could use lua itself somehow too which at that point it becomes a generic macro thing. (--#header = [[ ... --#]] ... --#test1 = [[ ... --#include(header) ... --#]]) but I'm not sure if there's going to be much benefit or if it'll make things easier.

—
Reply to this email directly or view it on GitHub.

[CapsAdmin]

That's probably how I would do it personally too but if it's important (for some reason?) to keep the tests as is in single lua files then I would do what I said.