In my current role, we've been using Ansible for our orchestration and configuration management. Ansible is okay, but after several months wrestling with its extension API and being frustrated by YAML syntax ideas started popping into my head for something better.
Chopsticks (docs) represents my vision of Pythonic orchestration. It's not an orchestration framework in itself. It's more the transport layer for other orchestration systems. It's a remote procedure call (RPC) system that relies on no agent on the remote host: the agent is built dynamically on the remote host by feeding code over the SSH tunnel to the system Python.
For example, you can create an SSH Tunnel to a remote host:
Then you can pass a function (any pickleable Python function), to be called on the remote host. Here I'm just calling the standard time.time() function.
Of course, you might want to do this in parallel on a number of hosts, and this is also built-in to Chopsticks:
from chopsticks.group import Group from chopsticks.facts import ip group = Group([ 'web1.example.com', 'web2.example.com', 'web3.example.com', ]) for host, t in group.call(time.time).items(): print('%s time:' % host, t) for host, addr in group.call(ip).items(): print('%s ip:' % host, addr)
Note that the code for chopsticks.facts does not need to be installed on the remote hosts. They will load it on demand from the orchestration host.
Effectively, Chopsticks gives you the ability to write Python programs that straddle a number of machines, all sharing a single codebase.
See the README for a summary of how this works.
SSH tunnels are not the only connection type Chopsticks supports. It communicates over stdin/stdout pipes, so can work with any system that supports these without inteference. Such as Docker (this is on Github, but not PyPI yet):
from chopsticks.tunnel import Docker from chopsticks.group import Group from chopsticks.facts import python_version group = Group([ Docker('worker-1', image='python:3.4'), Docker('worker-2', image='python:3.5'), Docker('worker-3', image='python:3.6'), ]) for host, python_version in group.call(python_version).items(): print('%s Python version:' % host, python_version)
Chopsticks gives fine control at a distance - like chopsticks do.
Chopsticks vs ...
It's natural to draw comparisons between Chopsticks and various existing tools, but I would point out that Chopsticks is a library, not an orchestration framework, and I'd invite you to think whether other tools could benefit from using and building on it.
Perhaps the immediate comparison is with Ansible, because it is frustrations with this that inspired Chopsticks.
Ansible feels a lot like Bash scripting across hosts, but in a warty YAML syntax. So first and foremost, I'm attracted to the idea of describing plays in nice, clean Python code. Python code is also more easily testable, and there are great documentation tools you can use.
Ansible's remote execution model involves dropping scripts, calling them, and deleting them. In Ansible 2.1, some of Ansible's support code for Python-based Ansible plugins gets shipped over SSH as part of a zipped bundle; but this doesn't extend to your own code extentions. So Chopsticks is more easily and naturally extensible: write your code how you like and let Chopsticks deal with getting it running on the remote machine.
Fabric is perhaps more similar to Chopsticks - it's a thin framework around the SSH transport, that allows scripting across hosts in Python syntax.
The big difference between Fabric and Chopsticks is that Fabric will only execute shell commands on the remote host, not Python callables. Of course you can drop Python scripts and call them, but then you're back in Ansible territory.
The difference in concept goes deeper: Fabric tries to be "of SSH", exploiting all the cool SSH tunnelling features. Chopsticks doesn't care about SSH specifically; it only cares about Python and pipes. This is what allows it to work identically with Docker as with remote SSH hosts.
As I was sharing Chopsticks on the Twitters, people pointed out the similarity to execnet, which I had not heard of.
Chopsticks has similarity to execnet, but from what little I've read it works in a very different way (by shipping selected code fragments), and will not allow importing arbitrary code from the orchestration host (ie. full import hooks).
Future of Chopsticks
Chopsticks is open source under the Apache 2 license, and at the time of writing, is at a very early stage - barely more than a proof-of-concept - but under very active development.
It currently has support for:
- SSH, Docker and subprocess tunnels
- Python 2.6-2.7 and 3.3-3.6
- Parallel execution
- Error handling
- Proxying of stderr (with hostname prepended)
- Send/receive file streams
- Higher-level orchestration functions
If Chopsticks looks interesting to you, I'd appreciate your feedback, and I welcome any pull requests.
Updated 2016-07-24: Updated to reflect improvements since original posting.