Chopsticks - a Pythonic orchestration library

In my current role, we've been using Ansible for our orchestration and configuration management. Ansible is okay, but after several months wrestling with its extension API and being frustrated by YAML syntax ideas started popping into my head for something better.

Chopsticks (docs) represents my vision of Pythonic orchestration. It's not an orchestration framework in itself. It's more the transport layer for other orchestration systems. It's a remote procedure call (RPC) system that relies on no agent on the remote host: the agent is built dynamically on the remote host by feeding code over the SSH tunnel to the system Python.

For example, you can create an SSH Tunnel to a remote host:

from chopsticks.tunnel import Tunnel
tun = Tunnel('ns1.example.com')

Then you can pass a function (any pickleable Python function), to be called on the remote host. Here I'm just calling the standard time.time() function.

import time
print('Time on %s:' % tun.host, tun.call(time.time))

Of course, you might want to do this in parallel on a number of hosts, and this is also built-in to Chopsticks:

from chopsticks.group import Group
from chopsticks.facts import ip

group = Group([
    'web1.example.com',
    'web2.example.com',
    'web3.example.com',
])
for host, t in group.call(time.time).items():
    print('%s time:' % host, t)
for host, addr in group.call(ip).items():
    print('%s ip:' % host, addr)

Note that the code for chopsticks.facts does not need to be installed on the remote hosts. They will load it on demand from the orchestration host.

Effectively, Chopsticks gives you the ability to write Python programs that straddle a number of machines, all sharing a single codebase.

See the README for a summary of how this works.

SSH tunnels are not the only connection type Chopsticks supports. It communicates over stdin/stdout pipes, so can work with any system that supports these without inteference. Such as Docker (this is on Github, but not PyPI yet):

from chopsticks.tunnel import Docker
from chopsticks.group import Group
from chopsticks.facts import python_version

group = Group([
    Docker('worker-1', image='python:3.4'),
    Docker('worker-2', image='python:3.5'),
    Docker('worker-3', image='python:3.6'),
])

for host, python_version in group.call(python_version).items():
    print('%s Python version:' % host, python_version)