Comparing Bitfield/Scripts and Risor as alternatives to shell scripts
I don’t like writing shell scripts. I like uniformity and predictability, easy to grok interfaces, and, if I can get it, type-hinting. And shell scripts have none of that, often written quick with little deference to the system they’re participating in or the future maintainer.
So it’s unfortunate to me that shell scripts are the glue of automation, connecting disparate components and data across Linux to form pipelines. It’s equally unfortunate is that, like glue, shell scripts are not always applied with care.
When I came across Risor and “script” (github.com/bitfield/scripts), two projects with the goal of making script writing easier in Go, I had a lot of questions. How easy was it to write a script from scratch? What features are available? How easy was it to share scripts across hosts and pipelines? Could these really replace shell scripts?
To answer those questions, I wanted to run a few tests to see how well each could perform and determine if either could perform as well as a regular shell script, and whether the drawbacks outweighed the benefits.
I had 4 criteria I wanted to evaluate with each:
- Getting started:
How difficult was it to install the tool or library and start executing a script? - Performance:
How well does the tool or library perform compared to standard shell scripts? - Distribution:
How difficult is it to share and execute scripts in different environments? - Maintenance:
How difficult is it to learn the tool or library to contribute or troubleshoot?
Weighing those together, not in any scientific or numerical scale, will at least give an idea of how these projects compare to each other, how they could potentially replace bash in existing workflows, and when they might be worth considering.
Risor
Risor is a CLI tool and Go package that reads and executes files written in Risor’s DSL.
Installing and Getting Started with Risor
There are two ways of using Risor: As an executable and as a package.
Installing the executable is simple for Mac users: Risor has a package available through Homebrew (brew install risor) that handles installing the tool and any dependencies.
On other platforms, there isn’t a precompiled binary (that I could find in the docs), so the tool needs to be built before it can be used. That means installing Go and configuring the environment, cloning the project, and running the go install. Not especially difficult, but more work than required for MacOS.
Using the Risor package is the same process on all platforms, requiring Go’s build tools, and is pulled down to the environment in the same method as any other Go package.
Risor modules
Functionality within Risor is implemented by compiling and installing various modules along with the project and then exposing as the DSL at the script later. Modules are included for interacting with the OS, DNS, JSON, and other tools and applications like Kubernetes.
While there are many cases where the DSL doesn’t really provide a benefit, like reading files or printing text where shell languages are already simple enough, the more advanced and specific modules provide a much simpler interface.
For example, the tablewriter module can take arrays of data and print nice looking tables to the shell. I can see uses where I’m trying to display a table of data during a pipeline and implementing it in Risor is much easier than in other languages.
Risor also has documentation for implementing custom modules and contributing them back to the project or distributing them privately.
Creating an example Risor script
For the first test, I created a temporary directory to start from and installed Risor.
$ cd $(mktemp -d) $ brew install risor
Second, I created a script read_file to read a file and print the contents to stdout.
$ <<EOT >> read_file #!/usr/bin/env risor -- my_file := os.args()[1] printf(cat(my_file)) EOT $ chmod u+x ./read_file
Lastly, I created an example file and executed the script to read it.
$ <<EOT >> test.txt Hello, Risor! EOT $ ./read_file ./test.txt Hello, Risor!
All together, a pretty simple process from start to finish, albeit by only performing a very simple task.
Comparing Risor performance to shell
To compare against the shell performance, I wrote that same script in Zsh, reading a file from an argument and printing to standard out.
$ <<EOT >> shell_script
#!/bin/zsh
my_file="\${1}";
printf "%s" "\$(cat \${my_file})"
EOT
$ chmod u+x ./shell_scriptTo generate the comparison, I started a timer and loop and executed each script 1000 times to get a little easier number to compare.
# Shell script
$ time zsh -c 'for i in {1..1000}; do ( ./shell_script ./test.txt >> /dev/null ); done'
zsh -c 0.02s user 0.15s system 3% cpu 4.420 total
# Risor
$ time zsh -c 'for i in {1..1000}; do ( ./read_file ./test.txt >> /dev/null ); done'
zsh -c 'for i in {1..1000}; do ( ./read_file ./test.txt >> /dev/null ); done' 0.04s user 0.20s system 1% cpu 18.706 totalPerforming the file read 1000 times with Risor took 18.7 seconds. Performing that same operation with just Zsh took only 4.4 seconds.
Side note: Assigning the argument to a variable in the shell script slowed it down by almost half (47%), whereas assigning the argument to a variable in Risor made almost no different (2%).
Is it enough to be noticeable? Probably. Considering that this is a very small example, reading and printing a file, performed a thousand times, the difference of an individual run is milliseconds, and in isolation, may never be noticed. But, for more complex scripts reading multiple files and performing string transforms, or multiple scripts run at different points in a pipeline, those delays start adding up.
Distributing Risor scripts
Risor executes these scripts when executed by reading the script file, parsing the DSL, and executing it.
There are a few available methods for distributing and executing Risor scripts across hosts:
- Install Risor and custom modules anywhere the scripts will need to run, following the normal installation steps,
- Create and build a Go project that utilizes the Risor package and distribute those binaries from a repository,
- Or compile the scripts using another a separate tool, like com/rubiojr/rsx, and distribute the binaries.
Compared to sharing shell scripts, none of these solutions are particularly simple. All require the installation of outside tools or complicated methods of packaging. I had really hoped there would be a way to compile scripts using the same toolset that is used to execute them, but it’s either not possible or not yet implemented.
Bitfield/Scripts
script is a Go package that exposes shell executables and functionality as Go-like objects.
Getting started with script
Since script is a Go package, installing and getting started doesn’t require applications outside of the Go toolset, and is identical for each platform
- Install and configure Go environment
- Create a new module for code go mod init go.example.com/script
- Fetch and add dependency go get github.com/bitfield/script
While altogether not difficult, I’ve always lamented Go’s requirement that everything be a module. It’s difficult to work quickly to prototype a solution because of the requirement to initiate a module, define dependencies, etc., when all I want to do is test if my code works.
For me, the benefits of using script get lost quickly because it doesn’t provide anything that couldn’t be gained by just writing in Shell or Python. Because scripts must be compiled by script before they can be run, leaving go run … aside because that still requires the entire environment, the quick nature of writing them is lost and writing them is similar to any other Go-compiled executable.
Creating a script test
I implemented the same functionality in the script example that I did for Risor, so I started by creating a new temp directory and a new module.
$ cd $(mktemp -d) $ go mod init go.example.com/script go: creating new go.mod: module go.example.com/script
Then I add the script dependency and created the new script.
$ go get github.com/bitfield/script
go: added github.com/bitfield/script v0.24.0
go: added github.com/itchyny/gojq v0.12.13
go: added github.com/itchyny/timefmt-go v0.1.5
go: added mvdan.cc/sh/v3 v3.7.0
$ mkdir -p cmd/read_file
$ <<EOT >> cmd/read_file/main.go
package main
import (
"strings"
"github.com/bitfield/script"
)
func main() {
my_file := script.Args().First(1)
filename, err := my_file.String()
if err != nil {
panic(err)
}
script.File(strings.TrimSpace(filename)).Stdout()
}
EOTNext, I built the project and created my test file.
$ go build ./... $ <<EOT >> test.txt Hello, script! EOT
And ran my script.
$ ./read_file ./test.txt Hello, script!
Not too bad, either. There’s much more code involved, and it admittedly took a bit of debugging to figure out .String() would always append a newline and there wasn’t a way to access arguments individually, but once figured out it was simple to get going.
script performance vs. shell
Since my script is a compiled Go executable, I hoped that the performance would be improved, as well.
To test, I performed the same test as Risor and ran the script through a loop 1000 times to get a bigger number to compare between the two.
# Shell script results (from earlier)
$ time zsh -c 'for i in {1..1000}; do ( ./shell_script ./test.txt >> /dev/null ); done'
zsh -c 0.02s user 0.15s system 3% cpu 4.420 total
$ time zsh -c 'for i in {1..1000}; do ( ./read_file ./test.txt >> /dev/null ); done'
zsh -c 'for i in {1..1000}; do ( ./read_file ./test.txt >> /dev/null ); done' 0.02s user 0.15s system 4% cpu 3.800 totalscript completed 1000 iterations in 3.8 seconds, beating the Zsh script by 16%! In my previous experience, Go executables have rarely outperformed pure shell scripts.
Will it make a noticeable different in execution time in real-world use? I doubt it. It’s a difference of half a second across a thousand operations. But, it being close means performance is not a reason to avoid using script.
Distributing script executables
script projects produce executable binaries, similar to the binaries produced by any Go project. So, an executable produced by script will be able to run on similar hosts without needing to install additional dependencies. This is familiar for Go projects, and one of the main benefits of Go.
That does come with it’s own challenges, though. Go builds binaries for a target CPU architecture, unless given a target architecture to build. So, when distributing a script, the build pipeline will need to build and push a version for each target architecture.
The other option left is to generate the binary on the host that will run it. Doing that loses out on one of the major benefits of Go, requiring the Go toolset be installed and configured everywhere it will be run. Building a project for multiple architectures is a familiar requirement for Go development, so it’s not inherently a negative.
script interpreter
I like that the scripts can be compiled, but I wished there was a way to execute the script without needing to compile it. Being able to view the source and execute it, without other dependencies, is something that Shell scripts do well, and can speed up debugging broken pipelines.
Within the script documentation, it provides an example script to run script projects using just the provided code. This script will take the provided source, generate a temporary Go project, and compile and execute the script.
I can see some issues with that right away, but I created a goscript.sh within the same project to test it. I’ll save pasting the full goscript.sh here, but it can be found in the original post linked by the documentation.
I created a .goscript file using the using the code from the main.go.
$ <<EOT >> read_file.goscript
#!$(pwd)/goscript.sh
my_file := script.Args().First(1)
filename, err := my_file.String()
if err != nil {
panic(err)
}
script.File(strings.TrimSpace(filename)).Stdout()
EOT
$ chmod u+x ./read_file.goscriptAnd then executed the file.
$ ./read_file.goscript # command-line-arguments ./script.go:15:13: undefined: strings
So, immediately, there’s a problem. Without altering the goscript.sh, the interpreter script doesn’t include any package dependencies, meaning it’s limited to only the functionality exposed by the github.com/bitfield/script package or built-in to Go.
It’s an interesting project, but the limitations are too imposing: The full Go toolset needs to be installed, and the script can only use the script package and API.
Comparing the tools
In the beginning, I laid out 4 criteria I was going to use to assess these projects:
- Getting started
- Performance
- Distribution
- Maintenance
Risor vs. script
First, I’ll compare the two projects to each other, and then compare the “winner” to just using Shell languages.
1. Getting started
Under the “Getting Started” criteria, I give it to Risor. Risor provided me with an easier starting process. No project needed to be created and installing the tool was a single command (thanks to Homebrew). script was a bit more involved, requiring everything needed of a Go project, including Go source, regardless of platform it was developed on.
Risor 1, script 0.
2. Performance
“Performance”-wise, the clear winner is script. script was over 400% faster than Risor, and was able to perform simple operations more efficiently than the same test in Risor.
Risor 1, script 1.
3. Distribution
I give the “Distributing” category again to `script`. Risor is more flexible, allowing scripts to be interpreted and run on any platform that the Risor executable is installed on. But, those Risor scripts will always require installing external dependencies to run. Compiling for multiple architectures may be more work, but `script` projects can be installed and run as a single file, so sharing them amongst platforms doesn’t require vetting additional dependencies to install.
Risor 1, script 2.
4. Maintenance
For long-term maintenance, I give the point to script. Both projects require some knowledge of the API, Risor using a custom DSL and script using Go structs, interfaces, and functions. But, like introducing a new language to an environment, adopting a DSL should always be a conscious, deliberate choice. It takes time for a developer to learn an API and a project, and adding a DSL eliminates the benefit of knowing that language. script also can be deployed to hosts without outside dependencies, so there are no additional packages to keep updated.
Risor 1, script 3.
Better than shell?
After weighing all that, the question is, “is script better than writing shell scripts”?
In my non-scientific, completely arbitrary position: Not particularly.
I can see specific use cases where I will use script in the future. In places where maybe I would put a shell script to perform structured operations, like querying known sources and generating JSON for pipeline inputs, I may look at script as an alternative.
But, that assumes that I know ahead of time that the team I’m working with can take it over. If the maintainers don’t know Go, then I might as well hand them a jigsaw puzzle. Shell languages being so ubiquitous means that it’s likely most developers have enough knowledge to trouble shoot the issues, without needing to learn an entire other language and toolset. And what’s provided by using script needs to be compelling enough to give up that simplicity.
The performance of script was surprising, though, and I am excited to implement script for personal projects. I’d held off from writing those personal projects in Go because the startup times were painful and it didn’t provide enough of a benefit to rewrite them. But I’m eager to see if I can gain the distribution improvements and only have to download a single file rather than a whole library.
If you would like to learn more or have a conversation about go-based scripting, contact us.
 | Christopher Gerber, Solutions Architect Chris brings extensive experience in building developer and data platforms on AWS and GCP. As part of our technical pre-sales team, he supports sales efforts and leads projects focused on AWS, GCP, Ansible, CloudFormation, and container implementations. With a passion for enhancing developer experiences and creating resilient platforms, Chris is dedicated to helping clients achieve their goals through innovative solutions. Like what you read? Follow Christopher on LinkedIn |