recutils - Quick Testing Session

There are paths through the system: rec2csv -> csv -> csv2rec -> … recins -> recset -> recdel -> …

• Diff
• Timestamps
• LibreOffice
• recutils integrity checks

• recins does not create non-existent files, in spite claiming it does.
• recins changes the modified date of the file, even if (apparently) the file's contents do not change.
• recset segfaults with only file argument.
• Mixing decimal and hexadecimal numbers allows you to violate uniqueness constraints.
• How do I get uuids?

• Different date formats
• Different %rec in one file cause problems?
• Greater range of characters, e.g. I was having problems with control characters (which may or may not be due to recutils)
• Are there code injection vectors for C?

Id = 1

Location = 'loaned'

Running:

recset -n0 -fRating -aExcellent test.rec

Segmentation fault

Curious. The file has two types of records. I wonder if I only select one:

recset -tBook -n0 -fRating -aExcellent test.rec

recset: error: operation aborted due to integrity failures.
recset: error: use --verbose to get a detailed report.
recset: error: use --force to skip the integrity check.

Checking the file, the first record has:

%type: Location enum loaned home unknown
...
Location:
+ home
+ work

Perhaps Location can only have one value. I can use recset to fix this.

recset -tBook -n0 -fLocation -shome test.rec

And try again.

recset -tBook -n0 -fRating -aExcellent test.rec

Successful.

Imagine if I have lots of books at work, change my job, and so move all my books back to home. There did not seem to be a way of removing a single value from a field, but perhaps using pipes…. Or, can I duplicate a record to represent the fact I have two copies?

For the present, I want to modify an existing field to escape the newlines.

modified=$(recsel -e"Id = 1" -PNotes sessions.rec | sed -z 's/\n/ \\\n/g' | sed 's/^+/ /g')
recset -e"Id = 1" -fNotes -s"$modified" sessions.rec

Not successful. It starts each line with "+". It seems to be able to be able to recognise multiline input and deal with it appropriately.

Would probably be smarter to use sed to remove newlines entirely.

modified=$(recsel -e"Id = 1" -PNotes sessions.rec | sed -z 's/\n/ /g' | sed 's/^+ //g')
recset -e"Id = 1" -fNotes -s"$modified" sessions.rec

Id = 1

I wonder if it is possible to go back the other way? Used org-table-export to convert the above into a csv file.

csv2rec -t Session sessions.csv > sessions_imported.rec

sessions_import.rec is different from the original file in the whitespace of the "Notes" field, the record descriptors and the all apart from the first "Issues" field has numbers appended, e.g. "Issue_2".

I speculate that the whitespace is being lost by org-table or by org-table-export, not recutils. If I copy the field directly from the original file, I can search and find it in the above org-table. However, I cannot find it in the csv file produced by org-table-export, which suggests the string has been changed at this point and no longer matches.

recset --verbose -tSession -e"Id = 2" -fId -s02 sessions_evil.rec

Id = 2

Id: 02
Charter: Analyse Norbert's session report and decide what to do next.
Issue: Prevented from testing a lot of functionality due to recins bug
Notes: 
+ [Started investigating recins bug ("error renaming /tmp/...") but
+ decided to continue with the planned testing.]
+ 
+ Start by using recinf to see what I can about Norbert's session. Some
+ of the options (-n and -S) I couldn't get to work, at least not on
+ anonymous records.
+ 
+ recsel seems more likely. I couldn't get --collapse to work, if I have
+ understood it correctly.
+ 
+ Seem to be able to achive most if not all of
+ http://www.satisfice.com/sbtm/demo/reports/status.htm and its
+ sub-sections.
+ 
+ Tricky might be to store individual session reports in the way they
+ have in
+ http://www.satisfice.com/sbtm/demo/reports/s_by_datetime.htm. Either
+ we could just use recsel's functionality to find individual ones, or a
+ bit of bash-foo to generate separate files, or use recutils templates
+ to create HTML files with anchors for each record. Difficulty of
+ latter is that there isn't an easy way to print an entire record in a
+ template. Would have to put every field into the template and then not
+ sure how to deal with repeated fields like "Bug" or "Issue".
+ 
+ What next?
+ 
+ I could make a finished SBTM reporting product. But I think we already
+ know it is possible, or that any shortfalls we might have are going to
+ be pretty minimal.
+ 
+ I could attempt to achieve the same affect through:
+ * The C library
+ * Lisp
+ * Org
+ 
+ [But, is that a realistic scenario? Is this test team already happy
+ enough with what they have? And would I be getting any more
+ "coverage"? Or would I be covering the same ground but through
+ different interfaces?]
+ 
+ I could attempt other techniques to test recutils, e.g. coverage-based
+ or risk-based. I think I would need to get to the bottom of the
+ recins/recset etc. bug in order to test other parts of the product. [I
+ would also be introducing new members to the team and increasing the
+ amount of data in the project, which is more realistic.]
Bug: recsel --collapse did not collapse the fields
Bug: recinf -n and -S didn't change recinf's output
Bug: Constraint or type validation: "stdin:69: error: value too large.
+ Expected a size <= 62." Could this be nicer? Give the actual fieldname
+ at least?

With recfile starting:

%rec: Naughty String

recins --verbose -fFoo -vBar blns.rec

blns.rec:1: error: could not read external descriptor from file String.
recins: error: use --force to skip the integrity check.

From the info manual:

For those cases it is more convenient to use a "external descriptor". External descriptors can be built appending a file path to the '%rec' field value, like:

%rec: FSD_Entry /path/to/file.rec

The previous example indicates that a record descriptor describing the 'FSD_Entry' records shall be read from the file '/path/to/file.rec'.

For:

%typedef: num_t range -0xffffffff 0xffffffff
%type: Num num_t

recins --verbose -tNaughty_String -fNum -v0 blns.rec

(null):: error: expected an integer between 1 and -1.
recins: error: use --force to skip the integrity check.

recins --force -tNaughty_String -fNum -v0 blns.rec

Recfile now has:

Id: 0x0

Id: 0
Num: 0

Id = 0

Id: 0x0

Id: 0
Num: 0

Violating uniqueness constraint. Although only after using "–force".

Try to fix this:

recset --verbose -tNaughty_String -e"Id = 0x0" -fId -s2 blns.rec

blns.rec:13: error: duplicated key value in field 'Id' in record
blns.rec:15: error: duplicated key value in field 'Id' in record
blns.rec:16: error: expected an integer between 1 and -1.
blns.rec:19: error: expected an integer between 1 and -1.
recset: error: use --force to skip the integrity check.

I am unable to select either the record Id=0 or Id=0x0, as it violates integrity checks.

recins --verbose -tNaughty_String -fId -v0x05 -fNum -v0 blns.rec

recins --verbose -tNaughty_String -fNum -v0 blns.rec

Id = 5

Id: 0x05
Num: 0

Id: 5
Num: 0

recins --verbose -tNaughty_String -fId -v0x06 -fNum -v0 blns.rec

recfix --auto blns.rec

I cannot get uuids working. Perhaps a compile flag? Cannot see anything in the INSTALL, README or info documentation.

recset -fUuid -d blns.rec

Are dates normalised? Does it care about timezones, etc?

Id = 1

recset --verbose -e"Id = 0" -fNum -S1.00 blns.rec

recset --verbose -e"Id = 1" -fString -Snull blns.rec

recset --verbose -e"Id = 1" -fNum -s1/2 blns.rec

strings=(0
1
1.00
$1.00
1/2
1E2
1E02
1E+02
-1
-1.00
-$1.00
-1/2
-1E2
-1E02
-1E+02
1/0
0/0
-2147483648/-1
-9223372036854775808/-1
-0
-0.0
+0
+0.0
0.00
0..0
.
0.0.0
0,00
0,,0
,
0,0,0
0.0/0
1.0/0.0
0.0/0.0
1,0/0,0
0,0/0,0
--1
-
-.
-,
999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999999
NaN
Infinity
-Infinity
INF
1#INF
-1#IND
1#QNAN
1#SNAN
1#IND
0x0
0xffffffff
0xffffffffffffffff
0xabad1dea
123456789012345678901234567890123456789
1,000.00
1 000.00
1'000.00
1,000,000.00
1 000 000.00
1'000'000.00
1.000,00
1 000,00
1'000,00
1.000.000,00
1 000 000,00
1'000'000,00
01000
08
09
2.2250738585072011e-308)

for string in ${strings[@]}
do
    recins --verbose -tNaughty_String -fNum -v$string blns.rec
done

> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > drw@drw-t60:~/projects/testing/recutils$ drw@drw-t60:~/projects/testing/recutils$ > > > (null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: expected an integer between -255 and 255.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
Segmentation fault
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: expected an integer between -255 and 255.
recins: error: use --force to skip the integrity check.
(null):: error: expected an integer between -255 and 255.
recins: error: use --force to skip the integrity check.
(null):: error: expected an integer between -255 and 255.
recins: error: use --force to skip the integrity check.
(null):: error: expected an integer between -255 and 255.
recins: error: use --force to skip the integrity check.
(null):: error: expected an integer between -255 and 255.
recins: error: use --force to skip the integrity check.
(null):: error: expected an integer between -255 and 255.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.
(null):: error: invalid 'range' value.
recins: error: use --force to skip the integrity check.

Num < 0 || Num > 0 || Num = 0

Date_Added: Wed, 18 Jul 2018 10:21:14 +0100
Id: 0
Num: 1.00
String: (null)

Date_Added: Wed, 18 Jul 2018 10:21:14 +0100
Id: 1
Num: 2.2250738585072011e-308
String: null

Date_Added: Wed, 18 Jul 2018 10:21:14 +0100
Id: 2
Num: 0

Date_Added: Wed, 18 Jul 2018 10:21:14 +0100
Id: 3
Num: 0

Date_Added: Wed, 18 Jul 2018 10:21:14 +0100
Id: 4
Num: 0

Date_Added: Wed, 18 Jul 2018 10:21:14 +0100
Id: 0x05
Num: 0

Date_Added: Wed, 18 Jul 2018 10:21:14 +0100
Id: 5
Num: 0

Date_Added: Wed, 18 Jul 2018 10:21:14 +0100
Id: 6
Num: 0

Date_Added: Wed, 18 Jul 2018 10:21:14 +0100
Id: 0x06
Num: 0

Id: 7
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: 0

Id: 8
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: 1

Id: 9
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: 1.00

Id: 11
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: -1

Id: 12
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: -1.00

Id: 17
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: -0

Id: 18
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: -0.0

Id: 19
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: 0.00

Id: 35
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 0x0

Id: 36
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 0xffffffff

Id: 37
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 0xffffffffffffffff

Id: 38
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 123456789012345678901234567890123456789

Id: 40
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 1

Id: 41
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 000.00

Id: 43
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 1

Id: 44
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 000

Id: 45
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 000.00

Id: 47
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 1

Id: 50
Date_Added: Wed, 18 Jul 2018 11:06:06 +0100
Num: 1

Id: 51
Date_Added: Wed, 18 Jul 2018 11:06:06 +0100
Num: 000

Id: 53
Date_Added: Wed, 18 Jul 2018 11:06:06 +0100
Num: 2.2250738585072011e-308

Id=10,13-16 is missing; doesn't recognise "1/2" (which it doesn't claim to, although it does accept it). Id=20-34 missing; various odd formats. Id=39; Num=1,000.00 looks legit… Similarly Id=42,46,49,52?

Num < 0

Id: 11
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: -1

Id: 12
Date_Added: Wed, 18 Jul 2018 11:06:04 +0100
Num: -1.00

Id: 36
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 0xffffffff

Id: 37
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 0xffffffffffffffff

Id: 38
Date_Added: Wed, 18 Jul 2018 11:06:05 +0100
Num: 123456789012345678901234567890123456789

Last 3 are incorrect.

Still a problem when the range is increased? Or type=int?

strStrings=(",./;'[]\-="
         "<>?:\"{}|_+"
         "\!@#$%^\&*()\`~"
         "Ω≈ç√∫˜µ≤≥÷"
         "ЁЂЃЄЅІЇЈЉЊЋЌЍЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюя"
         "⁰⁴⁵"
         "田中さんにあげて下さい")

for strString in "${strStrings[@]}"
do
    recins --verbose -tNaughty_String -fString -v$strString blns.rec
done

cat blns.rec | recfmt -fblns.tmpl

ID: 0
Num: 1.00
String: (null)
Date: Wed, 18 Jul 2018 10:21:14 +0100
ID: 1
Num: 2.2250738585072011e-308
String: null
Date: Wed, 18 Jul 2018 10:21:14 +0100
ID: 2
Num: 0
String: 
Date: Wed, 18 Jul 2018 10:21:14 +0100
ID: 3
Num: 0
String: 
Date: Wed, 18 Jul 2018 10:21:14 +0100
ID: 4
Num: 0
String: 
Date: Wed, 18 Jul 2018 10:21:14 +0100
ID: 0x05
Num: 0
String: 
Date: Wed, 18 Jul 2018 10:21:14 +0100
ID: 5
Num: 0
String: 
Date: Wed, 18 Jul 2018 10:21:14 +0100
ID: 6
Num: 0
String: 
Date: Wed, 18 Jul 2018 10:21:14 +0100
ID: 0x06
Num: 0
String: 
Date: Wed, 18 Jul 2018 10:21:14 +0100
ID: 7
Num: 0
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 8
Num: 1
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 9
Num: 1.00
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 10
Num: 1/2
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 11
Num: -1
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 12
Num: -1.00
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 13
Num: -1/2
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 14
Num: 1/0
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 15
Num: 0/0
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 16
Num: -9223372036854775808/-1
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 17
Num: -0
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 18
Num: -0.0
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 19
Num: 0.00
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 20
Num: 0..0
String: 
Date: Wed, 18 Jul 2018 11:06:04 +0100
ID: 21
Num: 0.0.0
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 22
Num: 0,00
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 23
Num: 0,,0
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 24
Num: 0,0,0
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 25
Num: 0.0/0
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 26
Num: 1.0/0.0
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 27
Num: 0.0/0.0
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 28
Num: 1,0/0,0
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 29
Num: 0,0/0,0
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 30
INF
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 31
IND
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 32
QNAN
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 33
SNAN
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 34
IND
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 35
Num: 0x0
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 36
Num: 0xffffffff
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 37
Num: 0xffffffffffffffff
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 38
Num: 123456789012345678901234567890123456789
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 39
Num: 1,000.00
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 40
Num: 1
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 41
Num: 000.00
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 42
Num: 1,000,000.00
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 43
Num: 1
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 44
Num: 000
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 45
Num: 000.00
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 46
Num: 1.000,00
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 47
Num: 1
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 48
Num: 000,00
String: 
Date: Wed, 18 Jul 2018 11:06:05 +0100
ID: 49
Num: 1.000.000,00
String: 
Date: Wed, 18 Jul 2018 11:06:06 +0100
ID: 50
Num: 1
String: 
Date: Wed, 18 Jul 2018 11:06:06 +0100
ID: 51
Num: 000
String: 
Date: Wed, 18 Jul 2018 11:06:06 +0100
ID: 52
Num: 000,00
String: 
Date: Wed, 18 Jul 2018 11:06:06 +0100
ID: 53
Num: 2.2250738585072011e-308
String: 
Date: Wed, 18 Jul 2018 11:06:06 +0100
ID: 54
Num: 
String: ,./;'[]\-=
Date: Wed, 18 Jul 2018 11:43:34 +0100
ID: 55
Num:
?:"{}|_+
Date: Wed, 18 Jul 2018 11:43:34 +0100
ID: 56
Num:
$%^\&*()`~
Date: Wed, 18 Jul 2018 11:43:34 +0100
ID: 57
Num: 
String: Ω≈ç√∫˜µ≤≥÷
Date: Wed, 18 Jul 2018 11:43:34 +0100
ID: 58
Num: 
String: ЁЂЃЄЅІЇЈЉЊЋЌЍЎЏАБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯабвгдежзийклмнопрстуфхцчшщъыьэюя
Date: Wed, 18 Jul 2018 11:43:34 +0100
ID: 59
Num: 
String: ⁰⁴⁵
Date: Wed, 18 Jul 2018 11:43:34 +0100
ID: 60
Num: 
String: 田中さんにあげて下さい
Date: Wed, 18 Jul 2018 11:43:34 +0100

What happened to Id=55 and 56? Actually, the full output from the shell is:

ID: 55
Num: 
String: <>?:"{}|_+
Date: Wed, 18 Jul 2018 11:43:34 +0100
ID: 56
Num: 
String: \!@#$%^\&*()`~
Date: Wed, 18 Jul 2018 11:43:34 +0100

So some sort of problem with org-babel.