Building from source¶
Please also see Installation for information about pre-built executables.
Miller license¶
Two-clause BSD license https://github.com/johnkerl/miller/blob/master/LICENSE.txt.
From release tarball using autoconfig¶
Miller allows you the option of using GNU autoconfigure
to build portably.
Grateful acknowledgement: Miller’s GNU autoconfig work was done by the generous and expert efforts of Thomas Klausner.
Obtain
mlr-i.j.k.tar.gz
from https://github.com/johnkerl/miller/tags, replacingi.j.k
with the desired release, e.g.2.2.1
.tar zxvf mlr-i.j.k.tar.gz
cd mlr-i.j.k
Install the following packages using your system’s package manager (
apt-get
,yum install
, etc.): flexVarious configuration options of your choice, e.g.
./configure
./configure --prefix=/usr/local
./configure --prefix=$HOME/pkgs
./configure CC=clang
./configure --disable-shared
(to make a statically linked executable)./configure 'CFLAGS=-Wall -std=gnu99 -O3'
etc.
make
creates thec/mlr
executablemake check
make install
copies thec/mlr
executable to your prefix’sbin
subdirectory.
From git clone using autoconfig¶
git clone https://github.com/johnkerl/miller
cd miller
Install the following packages using your system’s package manager (
apt-get
,yum install
, etc.): automake autoconf libtool flexRun
autoreconf -fiv
. (This is necessary when building from head as discussed in https://github.com/johnkerl/miller/issues/131.)Then continue from “Install the following … ” as above.
Without using autoconfig¶
GNU autoconfig is familiar to many users, and indeed plenty of folks won’t bother to use an open-source software package which doesn’t have autoconfig support. And this is for good reason: GNU autoconfig allows us to build software on a wide diversity of platforms. For this reason I’m happy that Miller supports autoconfig.
But, many others (myself included!) find autoconfig confusing: if it works without errors, great, but if not, the ./configure && make
output can be exceedingly difficult to decipher. And this also can be a turn-off for using open-source software: if you can’t figure out the build errors, you may just keep walking. For this reason I’m happy that Miller allows you to build without autoconfig. (Of course, if you have any build errors, feel free to contact me at mailto:kerl.john.r+miller@gmail.com – or, better, open an issue with “New Issue” at https://github.com/johnkerl/miller/issues.)
Steps:
Obtain a release tarball or git clone.
cd
into thec
subdirectory.Edit the
INSTALLDIR
inMakefile.no-autoconfig
.To change the C compiler, edit the
CC=
lines inMakefile.no-autoconfig
anddsls/Makefile.no-autoconfig
.make -f Makefile.no-autoconfig
creates themlr
executable and runs unit/regression tests (i.e. the equivalent of bothmake
andmake check
using autoconfig).make install
copies themlr
executable to your install directory.
The Makefile.no-autoconfig
is simple: little more than gcc *.c
. Customzing is less automatic than autoconfig, but more transparent. I expect this makefile to work with few modifications on a large fraction of modern Linux/BSD-like systems: I’m aware of successful use with gcc
and clang
, on Ubuntu 12.04 LTS, SELinux, Darwin (MacOS Yosemite), and FreeBSD.
Windows¶
Disclaimer: I’m now relying exclusively on Appveyor for Windows builds; I haven’t built from source using MSYS in quite a while.
Miller has been built on Windows using MSYS2: http://www.msys2.org/. You can install MSYS2 and build Miller from its source code within MSYS2, and then you can use the binary from outside MSYS2. You can also use a precompiled binary (see above).
You will first need to install MSYS2: http://www.msys2.org/. Then, start an MSYS2 shell, e.g. (supposing you installed MSYS2 to C:\msys2\
) run C:\msys2\mingw64.exe
. Within the MSYS2 shell, you can run the following to install dependent packages:
pacman -Syu
pacman -Su
pacman -S base-devel
pacman -S msys2-devel
pacman -S mingw-w64-x86_64-toolchain
pacman -S mingw-w64-x86_64-pcre
pacman -S msys2-runtime
The list of dependent packages may be also found in appveyor.yml in the Miller base directory.
Then, simply run msys2-build.sh which is a thin wrapper around ./configure && make
which accommodates certain Windows/MSYS2 idiosyncracies.
There is a unit-test false-negative issue involving the semantics of the mkstemp
library routine but a make -k
in the c
subdirectory has been producing a mlr.exe
for me.
Within MSYS2 you can run mlr
: simply copy it from the c
subdirectory to your desired location somewhere within your MSYS2 $PATH
. To run mlr
outside of MSYS2, just as with precompiled binaries as described above, you’ll need msys-2.0.dll
. One way to do this is to augment your path:
C:\> set PATH=%PATH%;\msys64\mingw64\bin
Another way to do it is to copy the Miller executable and the DLL to the same directory:
C:\> mkdir \mbin
C:\> copy \msys64\mingw64\bin\msys-2.0.dll \mbin
C:\> copy \msys64\wherever\you\installed\miller\c\mlr.exe \mbin
C:\> set PATH=%PATH%;\mbin
In case of problems¶
If you have any build errors, feel free to contact me at mailto:kerl.john.r+miller@gmail.com – or, better, open an issue with “New Issue” at https://github.com/johnkerl/miller/issues.
Dependencies¶
Required external dependencies¶
These are necessary to produce the mlr
executable.
gcc
,clang
, etc. (or presumably other compilers; please open an issue or send me a pull request if you have information for me about other 21st-century compilers)The standard C library
flex
automake
,autoconf
, andlibtool
, if you build with autoconfig
Optional external dependencies¶
This documentation pageset is built using Sphinx. Please see ./README.md for details.
Internal dependencies¶
These are included within the Miller source tree and do not need to be separately installed (and in fact any separate installation will not be picked up in the Miller build):
Mersenne Twister for pseudorandom-number generation: C implementation by Nishimura and Matsumoto with license terms respected.
MinUnit for unit-testing, with as-is-no-warranty license http://www.jera.com/techinfo/jtns/jtn002.html#License, https://github.com/johnkerl/miller/blob/master/c/lib/minunit.h.
The Lemon parser-generator, the author of which explicitly disclaims copyright.
The udp JSON parser, with BSD2 license.
The sheredom UTF-8 library, which is free and unencumbered software released into the public domain.
The NetBSD
strptime
(needed for the Windows/MSYS2 port since MSYS2 lacks this), with BSD license.
Creating a new release: for developers¶
At present I’m the primary developer so this is just my checklist for making new releases.
In this example I am using version 3.4.0; of course that will change for subsequent revisions.
Update version found in
mlr --version
andman mlr
:Edit
configure.ac
,c/mlrvers.h
, andmiller.spec
from3.3.2-dev
to3.4.0
.Do a fresh
autoreconf -fiv
and commit the output.make -C c -f Makefile.no-autoconfig installhome && make -C doc -f Makefile.no-autoconfig all installhome
Commit and push.
ISP upload so http://johnkerl.org/miller/doc/index.html will have updated head-docs.
Create the release tarball and SRPM:
On buildbox:
./configure && make distcheck
On buildbox: make SRPM as in https://github.com/johnkerl/miller/blob/master/README-RPM.md
On all buildboxes:
cd c
andmake -f Makefile.no-autoconfig mlr.static
. Then copymlr.static
to../mlr.{arch}
. (This may require as prerequisitesudo yum install glibc-static
or the like.)For static binaries, please do
ldd mlr.static
and make sure it saysnot a dynamic executable
.Then
mv mlr.static ../mlr.linux_x86_64
Pull back release tarball
mlr-3.4.0.tar.gz
and SRPMmiller-3.4.0-1.el6.src.rpm
from buildbox, andmlr.{arch}
binaries from whatever buildboxes.Download
mlr.exe
andmsys-2.0.dll
from https://ci.appveyor.com/project/johnkerl/miller/build/artifacts.
Create the Github release tag:
Don’t forget the
v
inv3.4.0
Write the release notes
Attach the release tarball, SRPM, and binaries. Double-check assets were successfully uploaded.
Publish the release
Create the release-specific docs:
Pull the Github-generated source
.tgz
from Github to ISP miller-releases directory and unpack andcd
into it.cd
intodoc
subdir. Edittemplate.html
’stitleinbody
tag fromMiller
toMiller 3.4.0
and poki-build.ISP upload
At the ISP site, update the latest-revision symlink
ln -s pub_http_internet/miller-releases/miller-3.4.0 pub_http_internet/miller
. (This is the main doc URL linked to from the top of the Github repo so it must point to this newly published revision.)Back in the Github repo, edit
doc/content-for-release-docs.html
to include new release, poki-build, push/commit, and ISP upload.
Notify:
Submit
brew
pull request; notify any other distros which don’t appear to have autoupdated since the previous release (notes below)Similarly for
macports
: https://github.com/macports/macports-ports/blob/master/textproc/miller/Portfile.Social-media updates.
git remote add upstream https://github.com/Homebrew/homebrew-core # one-time setup only
git fetch upstream
git rebase upstream/master
git checkout -b miller-3.4.0
shasum -a 256 /path/to/mlr-3.4.0.tar.gz
edit Formula/miller.rb
# Test the URL from the line like
# url "https://github.com/johnkerl/miller/releases/download/v3.4.0/mlr-3.4.0.tar.gz"
# in a browser for typos
# A '@BrewTestBot Test this please' comment within the homebrew-core pull request will restart the homebrew travis build
git add Formula/miller.rb
git commit -m 'miller 3.4.0'
git push -u origin miller-3.4.0
(submit the pull request)
Afterwork:
Edit
configure.ac
andc/mlrvers.h
to change version from3.4.0
to3.4.0-dev
.make -C c -f Makefile.no-autoconfig installhome && make -C doc -f Makefile.no-autoconfig all installhome
Commit and push.
Misc. development notes¶
I use terminal width 120 and tabwidth 4.