Record-heterogeneity

Contents: • For I/O
• CSV and pretty-print
• Key-value-pair, vertical-tabular, and index-numbered formats
• For processing

We think of CSV tables as rectangular: if there are 17 columns in the header then there are 17 columns for every row, else the data have a formatting error.

But heterogeneous data abound (today’s no-SQL databases for example). Miller handles this.

For I/O

CSV and pretty-print

Miller simply prints a newline and a new header when there is a schema change. When there is no schema change, you get CSV per se as a special case. Likewise, Miller reads heterogeneous CSV or pretty-print input the same way. The difference between CSV and CSV-lite is that the former is RFC4180-compliant, while the latter readily handles heterogeneous data (which is non-compliant). For example:

$ cat data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false

$ mlr --ocsvlite cat data/het.dkvp
resource,loadsec,ok
/path/to/file,0.45,true

record_count,resource
100,/path/to/file

resource,loadsec,ok
/path/to/second/file,0.32,true

record_count,resource
150,/path/to/second/file

resource,loadsec,ok
/some/other/path,0.97,false

$ mlr --opprint cat data/het.dkvp
resource      loadsec ok
/path/to/file 0.45    true

record_count resource
100          /path/to/file

resource             loadsec ok
/path/to/second/file 0.32    true

record_count resource
150          /path/to/second/file

resource         loadsec ok
/some/other/path 0.97    false

You may also find Miller’s group-like feature handy (see also Reference):

$ mlr --ocsvlite group-like data/het.dkvp
resource,loadsec,ok
/path/to/file,0.45,true
/path/to/second/file,0.32,true
/some/other/path,0.97,false

record_count,resource
100,/path/to/file
150,/path/to/second/file

$ mlr --opprint group-like data/het.dkvp
resource             loadsec ok
/path/to/file        0.45    true
/path/to/second/file 0.32    true
/some/other/path     0.97    false

record_count resource
100          /path/to/file
150          /path/to/second/file

Key-value-pair, vertical-tabular, and index-numbered formats

For these formats, record-heterogeneity comes naturally:

$ cat data/het.dkvp
resource=/path/to/file,loadsec=0.45,ok=true
record_count=100,resource=/path/to/file
resource=/path/to/second/file,loadsec=0.32,ok=true
record_count=150,resource=/path/to/second/file
resource=/some/other/path,loadsec=0.97,ok=false

$ mlr --onidx --ofs ' ' cat data/het.dkvp
/path/to/file 0.45 true
100 /path/to/file
/path/to/second/file 0.32 true
150 /path/to/second/file
/some/other/path 0.97 false

$ mlr --oxtab cat data/het.dkvp
resource /path/to/file
loadsec  0.45
ok       true

record_count 100
resource     /path/to/file

resource /path/to/second/file
loadsec  0.32
ok       true

record_count 150
resource     /path/to/second/file

resource /some/other/path
loadsec  0.97
ok       false

$ mlr --oxtab group-like data/het.dkvp
resource /path/to/file
loadsec  0.45
ok       true

resource /path/to/second/file
loadsec  0.32
ok       true

resource /some/other/path
loadsec  0.97
ok       false

record_count 100
resource     /path/to/file

record_count 150
resource     /path/to/second/file

For processing

Miller operates on specified fields and takes the rest along: for example, if you are sorting on the count field then all records in the input stream must have a count field but the other fields can vary, and moreover the sorted-on field name(s) don’t need to be in the same position on each line:

$ cat data/sort-het.dkvp
count=500,color=green
count=600
status=ok,count=250,hours=0.22
status=ok,count=200,hours=3.4
count=300,color=blue
count=100,color=green
count=450

$ mlr sort -n count data/sort-het.dkvp
count=100,color=green
status=ok,count=200,hours=3.4
status=ok,count=250,hours=0.22
count=300,color=blue
count=450
count=500,color=green
count=600