The REPL¶
The Miller REPL (read-evaluate-print loop) is an interactive counterpart to record-processing using the put
/filter
language. (A REPL is anything that evaluates what you type into it -- like python
with no arguments, or Ruby's irb
, or node
with no arguments, etc.)
Miller's REPL isn't a source-level debugger which lets you execute one source-code statement at a time -- however, it does let you operate on one record at a time. Further, it lets you use "immediate expressions", namely, you can interact with the Miller programming language without having to provide data from an input file.
mlr repl
[mlr] 1 + 2 3 [mlr] apply([1,2,3,4,5], func(e) {return e ** 3}) [1, 8, 27, 64, 125] [mlr] :open example.csv [mlr] :read [mlr] $* { "color": "yellow", "shape": "triangle", "flag": "true", "k": 1, "index": 11, "quantity": 43.6498, "rate": 9.8870 } [mlr] :context FILENAME="example.csv",FILENUM=1,NR=1,FNR=1
Using Miller without the REPL¶
Using put
and filter
, you can do the following as we've seen above:
- Specify input format (e.g.
--icsv
), output format (e.g.--ojson
), etc. using command-line flags. - Specify filenames on the command line.
- Define
begin {...}
blocks which are executed before the first record is read. - Define
end {...}
blocks which are executed after the last record is read. - Define user-defined functions/subroutines using
func
andsubr
. - Specify statements to be executed on each record -- which are anything outside of
begin
/end
/func
/subr
. - Example:
mlr --icsv --ojson --from example.csv head -n 2 \ then put 'begin {print "HELLO"} $qr = $quantity / $rate; end {print "GOODBYE"}'
HELLO [ { "color": "yellow", "shape": "triangle", "flag": "true", "k": 1, "index": 11, "quantity": 43.6498, "rate": 9.8870, "qr": 4.414868008496004 }, { "color": "red", "shape": "square", "flag": "true", "k": 2, "index": 15, "quantity": 79.2778, "rate": 0.0130, "qr": 6098.292307692308 }GOODBYE ]
Using Miller with the REPL¶
Using the REPL, by contrast, you get interactive control over those same steps:
- Specify input format (e.g.
--icsv
), output format (e.g.--ojson
), etc. using command-line flags. - REPL-only statements (non-DSL statements) start with
:
, such as:help
or:quit
or:open
. - Specify filenames either on the command line or via
:open
at the Miller REPL. - Read records one at a time using
:read
. - Write the current record (maybe after you've modified it with things like
$z = $x + $y
) using:write
. This goes to the terminal; you can use:> {filename}
to make writes go to a file, or:>> {filename}
to append. - You can type
:reopen
to go back to the start of the same file(s) you specified with:open
. - Skip ahead using statements
:skip 10
or:skip until NR == 100
or:skip until $status_code != 200
. - Similarly, but processing records rather than skipping past them, using
:process
rather than:skip
. Like:write
, these go to the screen; use:> {filename}
or:>> {filename}
to log to a file instead. - Define
begin {...}
blocks; invoke them at will using:begin
. - Define
end {...}
blocks; invoke them at will using:end
. - Define user-defined functions/subroutines using
func
/subr
; call them from other statements. - Interactively specify statements to be executed immediately on the current record.
- Load any of the above from Miller-script files using
:load
.
The input "record" by default is the empty map but you can do things like
$x=3
, or unset $y
, or $* = {"x": 3, "y": 4}
to populate it. Or, :open
foo.dat
followed by :read
to populate it from a data file.
Non-assignment expressions, such as 7
or true
, operate as filter conditions
in the put
DSL: they can be used to specify whether a record will or won't be
included in the output-record stream. But here in the REPL, they are simply
printed to the terminal, e.g. if you type 1+2
, you will see 3
.
Entering multi-line statements¶
- To enter multi-line statements, enter
<
on a line by itself, then the code (taking care for semicolons), then>
on a line by itself. These will be executed immediately. - If you enter
<<
on a line by itself, then the code, then>>
on a line by itself, the statements will be remembered for executing on records with:main
, as if you had done:load
to load statements from a file.
Examples¶
Use the REPL to look at arithmetic:
mlr repl
[mlr] 6/3 2 [mlr] 6/5 1.2 [mlr] typeof(6/3) int [mlr] typeof(6/5) float
Read the first record from a small file:
mlr repl
[mlr] :open foo.dat [mlr] :read [mlr] :context FILENAME="foo.dat",FILENUM=1,NR=1,FNR=1 [mlr] $* { "a": "eks", "b": "wye", "i": 4, "x": 0.38139939387114097, "y": 0.13418874328430463 } [mlr] $z = $x + $i [mlr] :write a=eks,b=wye,i=4,x=0.38139939387114097,y=0.13418874328430463,z=4.381399393871141
Skip until deep into a larger file, then inspect a record:
mlr repl --csv
[mlr] :open data/colored-shapes.csv [mlr] :skip until NR == 10000 [mlr] :r [mlr] $* { "color": "yellow", "shape": "circle", "flag": 1, "i": 496422, "u": 0.6530503199545348, "v": 0.23908588907834516, "w": 0.4799125551304738, "x": 6.379888206335166 }
Parsing and operator precedence¶
You can invoke mlr repl
with the -v
or -d
flags to show parse trees for expressions you enter.
(The -v
and -d
flags differ only in the format they use to present the parse trees.)
For example, if you have any questions about operator precedence, you can check the
operator-precedence section -- or, you can
try it out and see for yourself:
mlr repl -v
Miller 6.0.0-rc1 REPL for darwin/amd64/go1.16.5 Docs: https://miller.readthedocs.io Type ':h' or ':help' for online help; ':q' or ':quit' to quit. [mlr] x = 1 * 2 + 3 * statement block * assignment "=" * local variable "x" * operator "+" * operator "*" * int literal "1" * int literal "2" * int literal "3" [mlr] x = 1 + 2 * 3 * statement block * assignment "=" * local variable "x" * operator "+" * int literal "1" * operator "*" * int literal "2" * int literal "3"
mlr repl -d
$ mrpl -d Miller 6.0.0-rc1 REPL for darwin/amd64/go1.16.5 Docs: https://miller.readthedocs.io Type ':h' or ':help' for online help; ':q' or ':quit' to quit. [mlr] x = 1 * 2 + 3 (statement-block (= x (+ (* 1 2) 3 ) ) ) [mlr] x = 1 + 2 * 3 (statement-block (= x (+ 1 (* 2 3) ) ) )
History-editing¶
No command-line-history-editing feature is built in but rlwrap mlr repl is a
delight. You may need brew install rlwrap
, sudo apt-get install rlwrap
,
etc. depending on your platform.
Suggestion: alias mrpl='rlwrap mlr repl'
in your shell's startup file.
Online help¶
After mlr repl
, type :help
to see more about your options. In particular, :help examples
.