Gitingest

Include files under: 50kB

Private Repository NEW

Summary

Repository: puppetlabs/tasks-playground
Files analyzed: 87

Estimated tokens: 18.7k

Download

Directory Structure

Directory structure:

└── puppetlabs-tasks-playground/

    ├── README.md

    ├── LICENSE

    ├── ansible-inventory/

    │   ├── README.md

    │   └── inventory

    ├── awk-task/

    │   ├── README.md

    │   └── modules/

    │       └── sample/

    │           └── tasks/

    │               └── init

    ├── bash-task/

    │   ├── README.md

    │   └── modules/

    │       └── sample/

    │           └── tasks/

    │               └── init.sh

    ├── bolt-running-tests/

    │   ├── README.md

    │   ├── bashcheck

    │   └── ports.bats

    ├── boltmap/

    │   ├── README.md

    │   ├── boltmap.rb

    │   └── boltmap.yaml

    ├── docker-image/

    │   ├── README.md

    │   └── Dockerfile

    ├── docker-ssh/

    │   ├── README.md

    │   ├── docker-compose.yml

    │   ├── osquery/

    │   │   └── Dockerfile

    │   └── ssh/

    │       └── Dockerfile

    ├── dynamic-nodes/

    │   ├── README.md

    │   └── nodes.py

    ├── go-task/

    │   ├── README.md

    │   ├── main.go

    │   └── Makefile

    ├── install-puppet/

    │   └── README.md

    ├── junos/

    │   ├── README.md

    │   └── tasks/

    │       └── init.sh

    ├── librarian-tasks/

    │   ├── README.md

    │   ├── Gemfile

    │   ├── Puppetfile

    │   ├── Puppetfile.lock

    │   └── .librarian/

    │       └── puppet/

    │           └── config

    ├── lua-task/

    │   ├── README.md

    │   └── modules/

    │       └── sample/

    │           └── tasks/

    │               └── init.lua

    ├── nodejs-task/

    │   ├── README.md

    │   └── modules/

    │       └── sample/

    │           └── tasks/

    │               └── init.js

    ├── nodes-from-file/

    │   ├── README.md

    │   └── nodes

    ├── osquery-examples/

    │   ├── README.md

    │   └── modules/

    │       └── osquery/

    │           └── tasks/

    │               ├── init.sh

    │               ├── listening.sh

    │               ├── logins.sh

    │               ├── processes.sh

    │               ├── schema.sh

    │               ├── tables.sh

    │               └── users.sh

    ├── perl-task/

    │   ├── README.md

    │   └── modules/

    │       └── sample/

    │           └── tasks/

    │               └── init.pl

    ├── php-task/

    │   ├── README.md

    │   ├── Dockerfile

    │   └── modules/

    │       └── sample/

    │           └── tasks/

    │               ├── box.json

    │               └── init.php

    ├── process/

    │   ├── README.md

    │   ├── metadata.json

    │   └── tasks/

    │       ├── nix.json

    │       ├── nix.sh

    │       ├── win.json

    │       └── win.ps1

    ├── ps1exec/

    │   ├── README.md

    │   ├── metadata.json

    │   └── tasks/

    │       ├── init.json

    │       └── init.ps1

    ├── puppet-task/

    │   ├── README.md

    │   └── manifest.pp

    ├── python-task/

    │   ├── README.md

    │   ├── Dockerfile

    │   ├── setup.py

    │   ├── modules/

    │   │   └── sample/

    │   │       └── tasks/

    │   │           ├── init.py

    │   │           ├── main.py

    │   │           └── samplepkg.pex

    │   └── samplepkg/

    │       ├── __init__.py

    │       └── main.py

    ├── r10k-manage-tasks/

    │   ├── README.md

    │   ├── Gemfile

    │   └── Puppetfile

    ├── run-different-tasks-on-different-operating-systems/

    │   ├── README.md

    │   └── modules/

    │       └── example8/

    │           ├── functions/

    │           │   └── print_node_result.pp

    │           ├── plans/

    │           │   └── packages.pp

    │           └── tasks/

    │               ├── dpkg.py

    │               └── rpm.sh

    └── task/

        ├── README.md

        ├── metadata.json

        └── tasks/

            ├── kill.json

            ├── kill.py

            ├── stderr.json

            └── stderr.py

Files Content

================================================
FILE: README.md
================================================
# Tasks Playground

A series of experiments, examples and general play with `bolt`.

The intention of this repository is to:

* Collect examples of usage in one place
* Have a place for hacks, tips and tricks
* Provide a source of ideas which could be implemented in `bolt` itself
* Stretch what's possible with `bolt`

Adding new things to this repository should be simple and encouraged, nothing that works is out of scope and no idea a bad idea. This isn't about supported solutions or best practices but about the art of the possible from which those practices tend to grow.

If you've been experimenting with `bolt` feel free to include your experiments here.



================================================
FILE: LICENSE
================================================
   Bolt Playground - Puppet Task examples

   Copyright (C) 2017 Puppet, Inc.

   Puppet, Inc. can be contacted at: info@puppet.com

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       https://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.



================================================
FILE: ansible-inventory/README.md
================================================
# Targetting hosts with bolt using an Ansible inventory

Note that the following is a demonstration of what's possible, rather than any sort of best practice. 

Ansible supports listing hosts in an inventory file. At the simple end this is just an INI or YAML file, but the format supports various dynamic flavours, variable expansion, ranges, etc. This makes parsing arbitrary ansible inventories non-trivial.

Bolt takes a `--nodes` argument of a command-separated list of hosts. It would in some cases be handy for existing ansible users trying out bolt to reuse the inventory. Here is an example of how to do that.

Take the following static inventory:

```ini
mail.example.com

[webservers]
foo.example.com
bar.example.com

[dbservers]
one.example.com
two.example.com
three.example.com
```

Ansible supports the ability to list hosts for a particular group with `--list-hosts`. With a bit of shell foo we can convert that to the format required by `bolt`:

```
ansible <group> -i inventory --list-hosts | tail -n +2 | awk -vORS=, '{ print $1 }' | sed 's/,$/\n/'
```

For example:

```
$ ansible dbservers -i inventory --list-hosts | tail -n +2 | awk -vORS=, '{ print $1 }' | sed 's/,$/\n/'
one.example.com,two.example.com,three.example.com
```

```
$ ansible all -i inventory --list-hosts | tail -n +2 | awk -vORS=, '{ print $1 }' | sed 's/,$/\n/'
mail.example.com,foo.example.com,bar.example.com,one.example.com,two.example.com,three.example.com
```

This could be used inline with `bolt` like so:

```
bolt command run <command> --nodes `ansible all -i inventory --list-hosts | tail -n +2 | awk -vORS=, '{ print $1 }' | sed 's/,$//'`
```



================================================
FILE: ansible-inventory/inventory
================================================
mail.example.com

[webservers]
foo.example.com
bar.example.com

[dbservers]
one.example.com
two.example.com
three.example.com



================================================
FILE: awk-task/README.md
================================================
# Writing Puppet Tasks with Awk

Awk can be used as a scripting language, though it's hard to imagine a reason to do that. Nonetheless...

```awk
#!/usr/bin/awk -f

{
  printf "%s\n", ENVIRON["PT_message"]
}
```

Now it can be run on any node that has `awk` installed.

```
$ bolt task run sample message='Hello world' --nodes <nodes> --modules ./modules
bolt_ssh_1:

Hello world

Ran on 1 node in 2.98 seconds
```



================================================
FILE: awk-task/modules/sample/tasks/init
================================================
#!/usr/bin/awk -f

{
  printf "%s\n", ENVIRON["PT_message"]
}



================================================
FILE: bash-task/README.md
================================================
# Tasks Hello World

Tasks allow you to share commonly used commands as Puppet modules. This means they can be uploaded and downloaded from the Forge, as well as managed using all the existing Puppet tools. You can also just use them from GitHub or as a way of organising regularly used commands locally. 

By default tasks take arguments as environment variables, prefixed with `PT` (short for Puppet Tasks). Tasks are stored in the `tasks` directory of a module, a module just being a directory with a unique name. You can have several tasks per module, but the `init` task is special and will be run by default if a task name is not specified. All of that will make more sense if we see it in action.

Note that tasks can be implemented in any language which will run on the target nodes. We'll use `sh` here purely as a simple demonstration, but you could use Perl, Python, Lua, Javascript, etc. as long as it can read environment variables or take content on stdin.

Save the following file to `./modules/sample/tasks/init.sh`:

```
#!/bin/sh

echo $(hostname) got passed the message: $PT_message
```

We can then run that task using `bolt` like so. Note the `message` argument. This will be expanded to the `PT_message` environment variable expected by our task.

```
bolt task run sample message=hello <nodes> --modules ./modules
```

This should result in output similar to:

```
bolt_ssh_1:

bolt_ssh1 got passed the message: hello

Ran on 1 node in 0.39 seconds
```

Try running the `bolt` command with a different value for `message` and you should see the expected results.



================================================
FILE: bash-task/modules/sample/tasks/init.sh
================================================
#!/bin/sh

echo $(hostname) got passed the message: $PT_message



================================================
FILE: bolt-running-tests/README.md
================================================
# Using Bolt to run tests against nodes

One of the useful things we can do with bolt is run a set of tests against a large number of nodes at the same time. Those tests might be written in a formal test framework, or just handy scripts which make assertions against the state of the system. 

## Bats

`bats` is handy testing framework written in bash we can use to demonstrate this. You could write tests for all sorts of policy type assertions you want to make, and then easily demonstrate to colleagues that all nodes are correctly configured.

Note that for the following examples nodes need to have [bats](https://github.com/sstephenson/bats) installed. The `bats` documentation has a section for [installation instructions](https://github.com/sstephenson/bats/wiki/Install-Bats-Using-a-Package).  

First lets write some simple `bats` tests. We'll start with checking whether anything is listening on certain ports:

```bash
#!/usr/bin/env bats

@test "Test that nothing is listening on port 80" {
    run lsof -i :80
    [ "$status" -eq 1 ]
}

@test "Test that something is listening on port 22" {
    run lsof -i :22
    [ "$status" -eq 0 ]
}
```

`bolt` can upload the above script and execute it on the remote nodes using the `script` subcommand:

```
bolt script run something.bats -n <nodes>
```

This should result in output like the following, assuming we run against a single node which has something listening on port 22 and nothing listening on port 80.

```
bolt_ssh_1:

 ✓ Test that nothing is listening on port 80
 ✓ Test that something is listening on port 22

2 tests, 0 failures

Ran on 1 node in 1.19 seconds
```

## Shellshock

We might also want to use scripts written by third parties, for instance scripts that check nodes for known vulerabilities. Lets use [bashcheck](https://raw.githubusercontent.com/hannob/bashcheck/master/bashcheck) as an example. It checks nodes for the [ShellShock](https://en.wikipedia.org/wiki/Shellshock_(software_bug)) vulnerability.

```
bolt script run bashcheck -n <nodes>
```

This should hopefully result in a response like the following. If the script indicates issues you know you have work to do to upgrade the affected nodes.

```
bolt_ssh_1:

Testing /bin/bash ...
Bash version 4.3.11(1)-release

Variable function parser pre/suffixed [%%, upstream], bugs not exploitable
Not vulnerable to CVE-2014-6271 (original shellshock)
Not vulnerable to CVE-2014-7169 (taviso bug)
Not vulnerable to CVE-2014-7186 (redir_stack bug)
Test for CVE-2014-7187 not reliable without address sanitizer
Not vulnerable to CVE-2014-6277 (lcamtuf bug #1)
Not vulnerable to CVE-2014-6278 (lcamtuf bug #2)

Ran on 1 node in 0.64 seconds
```



================================================
FILE: bolt-running-tests/bashcheck
================================================
#!/bin/bash

warn() {
	if [ "$scary" == "1" ]; then
		echo -e "\033[91mVulnerable to $1\033[39m"
	else
		echo -e "\033[93mFound non-exploitable $1\033[39m"
	fi
}

good() {
	echo -e "\033[92mNot vulnerable to $1\033[39m"
}

tmpdir=`mktemp -d -t tmp.XXXXXXXX`

[ -n "$1" ] && bash=$(which $1) || bash=$(which bash)
echo -e "\033[95mTesting $bash ..."
$bash -c 'echo "Bash version $BASH_VERSION"'
echo -e "\033[39m"

#r=`a="() { echo x;}" $bash -c a 2>/dev/null`
if [ -n "$(env 'a'="() { echo x;}" $bash -c a 2>/dev/null)" ]; then
	echo -e "\033[91mVariable function parser active, maybe vulnerable to unknown parser bugs\033[39m"
	scary=1
elif [ -n "$(env 'BASH_FUNC_a%%'="() { echo x;}" $bash -c a 2>/dev/null)" ]; then
	echo -e "\033[92mVariable function parser pre/suffixed [%%, upstream], bugs not exploitable\033[39m"
	scary=0
elif [ -n "$(env 'BASH_FUNC_a()'="() { echo x;}" $bash -c a 2>/dev/null)" ]; then
	echo -e "\033[92mVariable function parser pre/suffixed [(), redhat], bugs not exploitable\033[39m"
	scary=0
elif [ -n "$(env '__BASH_FUNC<a>()'="() { echo x;}" $bash -c a 2>/dev/null)" ]; then
	echo -e "\033[92mVariable function parser pre/suffixed [__BASH_FUNC<..>(), apple], bugs not exploitable\033[39m"
	scary=0
else
	echo -e "\033[92mVariable function parser inactive, bugs not exploitable\033[39m"
	scary=0
fi


r=`env x="() { :; }; echo x" $bash -c "" 2>/dev/null`
if [ -n "$r" ]; then
	warn "CVE-2014-6271 (original shellshock)"
else
	good "CVE-2014-6271 (original shellshock)"
fi

pushd $tmpdir > /dev/null
env x='() { function a a>\' $bash -c echo 2>/dev/null > /dev/null
if [ -e echo ]; then
	warn "CVE-2014-7169 (taviso bug)"
else
	good "CVE-2014-7169 (taviso bug)"
fi
popd > /dev/null

$($bash -c "true $(printf '<<EOF %.0s' {1..80})" 2>$tmpdir/bashcheck.tmp)
ret=$?
grep AddressSanitizer $tmpdir/bashcheck.tmp > /dev/null
if [ $? == 0 ] || [ $ret == 139 ]; then
	warn "CVE-2014-7186 (redir_stack bug)"
else
	good "CVE-2014-7186 (redir_stack bug)"
fi


$bash -c "`for i in {1..200}; do echo -n "for x$i in; do :;"; done; for i in {1..200}; do echo -n "done;";done`" 2>/dev/null
if [ $? != 0 ]; then
	warn "CVE-2014-7187 (nested loops off by one)"
else
	echo -e "\033[96mTest for CVE-2014-7187 not reliable without address sanitizer\033[39m"
fi

$($bash -c "f(){ x(){ _;};x(){ _;}<<a;}" 2>/dev/null)
if [ $? != 0 ]; then
	warn "CVE-2014-6277 (lcamtuf bug #1)"
else
	good "CVE-2014-6277 (lcamtuf bug #1)"
fi

if [ -n "$(env x='() { _;}>_[$($())] { echo x;}' $bash -c : 2>/dev/null)" ]; then
	warn "CVE-2014-6278 (lcamtuf bug #2)"
elif [ -n "$(env BASH_FUNC_x%%='() { _;}>_[$($())] { echo x;}' $bash -c : 2>/dev/null)" ]; then
	warn "CVE-2014-6278 (lcamtuf bug #2)"
elif [ -n "$(env 'BASH_FUNC_x()'='() { _;}>_[$($())] { echo x;}' $bash -c : 2>/dev/null)" ]; then
	warn "CVE-2014-6278 (lcamtuf bug #2)"
else
	good "CVE-2014-6278 (lcamtuf bug #2)"
fi

rm -rf $tmpdir



================================================
FILE: bolt-running-tests/ports.bats
================================================
#!/usr/bin/env bats

@test "Test that nothing is listening on port 80" {
    run lsof -i :80
    [ "$status" -eq 1 ]
}

@test "Test that something is listening on port 22" {
    run lsof -i :22
    [ "$status" -eq 0 ]
}



================================================
FILE: boltmap/README.md
================================================
# boltmap: Sourcing bolt nodes from a YAML file

A super simple Ruby script that can be used to output a list of nodes in a format that's acceptable to the bolt cli from YAML, inspired by [ansible-inventory](../ansible-inventory).  Not actually compatible with Ansible Inventory though.  Can provide specific YAML file on the cli or provide it via STDIN.

### Usage

`boltmap.rb [options] file`

`-g`, `--group=` `Map group to run`

### Example

`` bolt command run hostname --nodes `boltmap.rb --group=webservers ~/mymap.yaml` ``



================================================
FILE: boltmap/boltmap.rb
================================================
#!/usr/bin/env ruby

require 'yaml'
require 'optparse'

options = { }

OptionParser.new do |opts|
  opts.banner = "Usage: boltmap.rb [options] file"

  opts.on("-g", "--group=", "Map group to run") do |g|
    options[:group] = g
  end
end.parse!

boltyaml = YAML.load(ARGF)

hosts = []
if options[:group].nil?
  boltyaml.keys.each { |b| hosts << boltyaml[b] }
else
  hosts = boltyaml[options[:group]]
end

puts hosts.flatten.join(',')



================================================
FILE: boltmap/boltmap.yaml
================================================
---
servers:
  - '192.168.0.139'
  - '192.168.0.138'
  - '192.168.0.137'
  - '192.168.0.136'
  - '192.168.0.135'
db:
  - '192.168.0.134'
  - '192.168.0.133'
  - '192.168.0.132'
  - '192.168.0.131'
  - '192.168.0.130'



================================================
FILE: docker-image/README.md
================================================
# Run Bolt from Docker

Rather than install a full Ruby development environment locally in order to install `bolt` you can use it from Docker. The Dockerfile in this directory needs to be run from within the source directory of bolt, so you'll first need to check that out. The following commands should work if run from this directory:

```
git clone https://github.com/puppetlabs/bolt.git
cp Dockerfile bolt
cd bolt
docker build -t puppet/bolt .
```

Using bolt from Docker is as simple as:

```
docker run puppet/bolt
```

From there you can mount directories (of modules or scripts you want to run, or ssh keys you have available) or pass any relevant environment variables required for configuration.



================================================
FILE: docker-image/Dockerfile
================================================
FROM alpine:3.5

MAINTAINER Gareth Rushgrove "gareth@puppet.com"

LABEL org.label-schema.vendor="Puppet" \
      org.label-schema.url="https://github.com/puppetlabs/bolt" \
      org.label-schema.name="Bolt" \
      org.label-schema.license="Apache-2.0" \
      org.label-schema.schema-version="1.0" \
      com.puppet.dockerfile="/Dockerfile"

RUN apk add --update --no-cache \
      build-base \
      libffi-dev \
      ca-certificates \
      ruby \
      ruby-io-console \
      ruby-irb \
      ruby-rdoc \
      ruby-dev \
      git && \
      gem install bundler

RUN addgroup -S bolt && adduser -S -g bolt bolt

# As of https://github.com/moby/moby/pull/34263 Docker supports
# setting ownership on Copy, but it's only available since the
# 17.09 edge release of Moby. We'll wait until it's in the
# main release
# COPY --chown=bolt:bolt . /usr/src/bolt
COPY . /usr/src/bolt
RUN chown bolt:bolt /usr/src/bolt
WORKDIR /usr/src/bolt

USER bolt

RUN bundle install --path .bundle

ENTRYPOINT ["bundle", "exec", "bolt"]
CMD ["--help" ]

COPY Dockerfile /



================================================
FILE: docker-ssh/README.md
================================================
# Ephemeral SSH servers

When experimenting with `bolt` it's useful to have a bunch of machines running SSH to hand. What better way to grab a large number of SSH servers locally than with Docker? The advantage of this approach is speed (we can launch 100 SSH servers in seconds) and low overhead (we can do all of that locally). It's a great way to experiment with `bolt` across a large number of nodes, without needing to provision new infrastructure or jump straight to production.


The following assumes you have Docker installed along with `docker-compose`. Compose will build an SSH server image with Puppet installed, so you can also experiment with the tasks which require the Puppet runtime to be present. You can launch with:

```
docker-compose up -d --scale ssh=<num>
```

Just provide a suitable value for the number of SSH nodes you would like to launch. 

```
docker-compose up -d --scale ssh=100
```

With your new SSH servers running you probably want to connect to them. For that we'll use the `bolt` Docker image described elsewhere, so we can connect to the Docker network automatically created by Compose. You'll need to [build your own image](../docker-image/) before running the following commands.

```
docker run -it --net bolt_default puppet/bolt command run uptime -n bolt_ssh_1 -u root -p root
```

We can get fancy as well and hit all of the containers we just started. On Windows we can do something like the following to provide the full list of sequential container names to bolt:

```
for /f %a in ('powershell -Command "'bolt_ssh_' + (1..100 -join ',bolt_ssh_')"') do docker run -rm -it --net bolt_default puppet/bolt command run uptime -n %a -u root -p root
```

Finally remember to stop all of the containers you started up:

```
docker-compose rm -fs
```





================================================
FILE: docker-ssh/docker-compose.yml
================================================
version: '3'
services:
  ssh-extras:
    build: ssh
  ssh:
    image: rastasheep/ubuntu-sshd
  ssh-osquery:
    build: osquery



================================================
FILE: docker-ssh/osquery/Dockerfile
================================================
FROM kolide/osquery

RUN apt-get update && \
    apt-get install -y openssh-server

RUN mkdir /var/run/sshd

RUN echo 'root:root' |chpasswd

RUN sed -ri 's/^PermitRootLogin\s+.*/PermitRootLogin yes/' /etc/ssh/sshd_config
RUN sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config

EXPOSE 22

CMD ["/usr/sbin/sshd", "-D"]



================================================
FILE: docker-ssh/ssh/Dockerfile
================================================
FROM rastasheep/ubuntu-sshd

ENV PUPPET_AGENT_VERSION="5.2.0" UBUNTU_CODENAME="trusty"

RUN apt-get update && \
    apt-get install --no-install-recommends -y wget ca-certificates lsb-release && \
    wget https://apt.puppetlabs.com/puppet5-release-"$UBUNTU_CODENAME".deb && \
    dpkg -i puppet5-release-"$UBUNTU_CODENAME".deb && \
    rm puppet5-release-"$UBUNTU_CODENAME".deb && \
    apt-get update && \
    apt-get install --no-install-recommends -y puppet-agent="$PUPPET_AGENT_VERSION"-1"$UBUNTU_CODENAME" && \
    apt-get remove --purge -y wget

RUN apt-get update && \
    apt-get install --no-install-recommends -y lsof software-properties-common python-software-properties && \ 
    add-apt-repository ppa:duggan/bats && \
    apt-get update && \
    apt-get install --no-install-recommends -y bats

RUN apt-get update && \
    apt-get install php5 -y 

ENV PATH=/opt/puppetlabs/server/bin:/opt/puppetlabs/puppet/bin:/opt/puppetlabs/bin:$PATH



================================================
FILE: dynamic-nodes/README.md
================================================
# Dynamic node lists

`bolt` takes a list of nodes as a comma-separated list of host names. But what if you want to determine that list dynamically? Maybe based on some CMDB or other database, or based on an API query? You can do this using standard shell, depending on your operating system.

On Windows we can use PowerShell. Here is a simple example

``` powershell
# Call an API or query a CMDB. For this example just output a simply array
$nodelist = (1..10) | % { Write-Output "test$_"}

bolt command run <command> --nodes ($nodeList -join ',')
```

This would be the equivalent of running the following, with nodes up to `test10`:

``` powershell
PS> bolt command run <command> -n test1,test2,test3...
```


You are likely to want to do something more complex, in which case moving the collection of nodes to run against into a separate script is likely useful. This could be written in any language you prefer but lets look at a simple Python example:

```python
nodes = []
for i in range(125):
    nodes.append("test%s" % i)

print(','.join(nodes))
```

This prints the nodes `test0` up to `test124`. You can have that data used by `bolt` like so. On Windows:


```
for /f %a in ('python nodes.py') do bolt command run <command> -n %a
```

Or on Linux:

```
bolt command run <command> -n `python nodes.py`
```



================================================
FILE: dynamic-nodes/nodes.py
================================================
nodes = []
for i in range(125):
    nodes.append("test%s" % i)

print(','.join(nodes))



================================================
FILE: go-task/README.md
================================================
# Writing Puppet Tasks in Go

Tasks for `bolt` can be written in any language, including compiled languages. Let's have a look at writing simple Go tasks to illustrate this.

Create an empty directory tree of `modules/sample/tasks`. `sample` is the name of our module, it will be used in the `bolt` user interface in a moment. Then save the following code as `main.go`:

```go
package main

import (
    "fmt"
    "os"
)

func main() {
    fmt.Println(os.Getenv("PT_message"))
}
```

Note that this is just a standard Go programme. It's using the standard library `fmt` module and then printing out the `PE_message` environment variable. Task arguments are automatically passed along to tasks and prefixed with `PT_` (short for Puppet Tasks).

We need to compile the task into a standalone executable for the target platform. Go has excelent support for cross-compilation. Let's assume we're targetting a Linux host. On macOS or Linux you can run: 

```bash
GOOS=linux GOAARCH=amd64 go build -o modules/sample/tasks/main
```

On Windows the syntax for setting the environment variables is slightly different, but you can run the following:

```powershell
set GOOS=linux
set GOAARCH=amd64
go build -o modules/sample/tasks/main
```

Both of the above will produce an executable called `main` in the tasks directory of the module we created above. That's all that's required to be able to run the task.

```
bolt task run sample::main message=hello --nodes <nodes> --modules ./modules
```

Note:

* the `--modules` flag points at the modules diretory we created above with out `sample` module in
* the argument `message` matches the name of our environment variable above
* `sample` is the name of our module and `main` is the name of our executable. These could be any valid module name and task name

This is a simple way of running any Go code on a large number of remote nodes, and packaging that code up so it's easy to share with the rest of your team or the wider community.





================================================
FILE: go-task/main.go
================================================
package main

import (
	"fmt"
	"os"
)

func main() {
	fmt.Println(os.Getenv("PT_message"))
}



================================================
FILE: go-task/Makefile
================================================
build:
	env GOOS=linux GOAARCH=amd64 go build -o $(CURDIR)/modules/sample/tasks/main



================================================
FILE: install-puppet/README.md
================================================
# Installing Puppet with Bolt using puppet-install-shell

[puppet-install-shell](https://github.com/petems/puppet-install-shell) is a project to maintain a set of scripts which install Puppet on a range of different Linux flavours.

The script can be downloaded and piped to `sh` as indicated in the documentation, alternatively you can download the script locally and then use `bolt script` to upload and run it.

```
wget https://raw.githubusercontent.com/petems/puppet-install-shell/master/install_puppet_5_agent.sh
bolt script run install_puppet_5_agent.sh --nodes <nodes>
```

That should output various installation steps and result in Puppet being installed from the official Puppet packages.You can verify that with bolt itself.

```
$ bolt command run "/opt/puppetlabs/bin/puppet --version" --nodes <nodes>
bolt_ssh_1:

5.2.0

Ran on 1 node in 0.68 seconds
```



================================================
FILE: junos/README.md
================================================
# junos: Simple show, set and delete for JUNOS

A really simple module that allows the showing, setting and deletion of arbitary items from Juniper Switches, Routers and Firewalls.
Requires my hacked up branch of bolt located at https://github.com/jonnytdevops/bolt/tree/junos
 
### Example

`bolt task run junos show='system login' set='system login user test_user1' delete='system login user test_user2' commit=true --nodes pdx-626c6-coretest.ops.puppetlabs.net`



================================================
FILE: junos/tasks/init.sh
================================================
# Show Commands
if ( $?PT_show ) then
  cli -c "show configuration $PT_show"
endif

# Set Commands
if ( $?PT_set ) then
  cli -c "edit; set $PT_set"
endif

# Delete Commands
if ( $?PT_delete ) then
  cli -c "edit; delete $PT_delete"
endif

# Delete Commands
if ( $?PT_commit ) then
  cli -c "edit; commit confirmed 1"
endif




================================================
FILE: librarian-tasks/README.md
================================================
# Installing tasks with librarian-puppet

Tasks are distributed as Puppet modules, which means existing tooling that works with Puppet modules already supports tasks. We can demonstrate that by using [librarian-puppet](https://github.com/voxpupuli/librarian-puppet) and a `Puppetfile` to download and manage a set of modules.

The following assumes you have a Ruby environment available. First install `librarian-puppet` and it's dependencies.

```
gem install librarian-puppet
```

We then need a `Puppetfile` which describes the modules we want to download. We're referencing the GitHub repositories for the modules for now because they modules are currently private and only available internally.

```
forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs-package',
  :git => "https://github.com/puppetlabs/puppetlabs-package.git"

mod 'puppetlabs-service',
  :git => "https://github.com/puppetlabs/puppetlabs-service.git"
```

We then run `librarian-puppet` to download the modules to the `./modules` folder.

```
librarian-puppet install
```

This should create a `./modules` folder if one doesn't already exist and download the `package` and `service` module.

```
$ ls modules
package  service
```

With those modules downloaded we can use them with `bolt` using the `--modules` flag.

```
bolt task run package action=status package=vim --modules ./modules --nodes <nodes>
```

Assuming we run this against an Ubuntu machine without vim installed you can expect to see output like the following:

```
bolt_ssh_1:

{"status":"absent","latest":"2:7.4.052-1ubuntu3.1"}

Ran on 1 node in 4.40 seconds
```



================================================
FILE: librarian-tasks/Gemfile
================================================
source "https://rubygems.org"

gem "librarian-puppet"



================================================
FILE: librarian-tasks/Puppetfile
================================================
forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs-package',
  :git => "https://github.com/puppetlabs/puppetlabs-package.git"

mod 'puppetlabs-service',
  :git => "https://github.com/puppetlabs/puppetlabs-service.git"



================================================
FILE: librarian-tasks/Puppetfile.lock
================================================
GIT
  remote: https://github.com/puppetlabs/puppetlabs-package.git
  ref: master
  sha: b68ea3d5003ff21406014824cf1398a1817a330a
  specs:
    puppetlabs-package (0.1.1)

GIT
  remote: https://github.com/puppetlabs/puppetlabs-service.git
  ref: master
  sha: 7abfd859a0870a8b3be0926f82808025499ac38d
  specs:
    puppetlabs-service (0.1.1)

DEPENDENCIES
  puppetlabs-package (>= 0)
  puppetlabs-service (>= 0)




================================================
FILE: librarian-tasks/.librarian/puppet/config
================================================
---
LIBRARIAN_PUPPET_DESTRUCTIVE: 'false'
LIBRARIAN_PUPPET_USE_V1_API: '1'



================================================
FILE: lua-task/README.md
================================================
# Writing Puppet Tasks with Lua

Lua is an interpreted language that also takes advantage of some runtime compilation features. This is an example task only to prove that you can write tasks in Lua, too!

## Write it!
You might do something like this (don't forget that shebang) :
```lua
#!/usr/bin/lua

print(os.getenv('PT_message'))

```

Or just go nuts for no reason:
```lua
#!/usr/bin/lua

msg = function() local message = os.getenv('PT_message'); return message end

print(msg())
```

## Run it!
Try running your Lua task on any node with Lua installed.

```
$ bolt task run sample message='hello world' --nodes <nodes> --modules ./modules
node:

hello world


Ran on 1 node in 1.23 seconds
```



================================================
FILE: lua-task/modules/sample/tasks/init.lua
================================================
#!/usr/bin/lua

msg = function() local message = os.getenv('PT_message'); return message end

print(msg())



================================================
FILE: nodejs-task/README.md
================================================
# Writing Puppet Tasks with Node.js

[Node.js](https://nodejs.org) provides a runtime for Javascript, and makes a nice environment for writing Puppet Tasks with. One approach to using Node is to ensure the runtime is already installed on on of the nodes you want to run tasks on. If that's the case then the simplest task would be a single `.js` file.

Save the following as `modules/sample/tasks/init.js`:

```javascript
#!/usr/bin/env node
console.log(process.env.PT_message);
```

The above creates a module called `sample` with a default task (indicated by the name `init`). We can run that with `bolt` like so:

```
$ bolt task run sample message=hello --nodes <nodes> --modules ./modules
bolt_ssh_1:

hello

Ran on 1 node in 1.37 seconds
```


## Removing the need to install Node.js on nodes

As noted, the above assumes the node runtime is already installed on the nodes. We can remove even that limitation if we use [pkg](https://github.com/zeit/pkg). `pkg` packages up out node application, it's dependencies and the runtime itself into a single executable that's perfect as a Puppet Task.

First install `pkg` with:

```
npm install -g pkg
```

Then package up our above script. Note that:

* We're only building the linux executable here, but `pkg` supports Windows and macOs as well
* We're outputting a single executable called `full`, but the name could be any valid task name

```
cd modules/sample/tasks
pkg -o full -t node8-linux-x64 init.js
```

This should create a file called `full`. Because `bolt` can automatically detect tasks based on unique filenames we can just run that from the top level directory like any other task.

```
bolt task run sample::full message=hello --nodes <nodes> --modules ./modules
```



================================================
FILE: nodejs-task/modules/sample/tasks/init.js
================================================
#!/usr/bin/env node
console.log(process.env.PT_message);



================================================
FILE: nodes-from-file/README.md
================================================
# Listing nodes from a text file

Rather than specifying all of the nodes on the command line you may like to store them in an accompanying test file. Let's start with a file like so:


```
web1.example.com
web2.example.com
web3.example.com
db.example.com
```

On a typical Linux machine (with `cat` and `paste` available) you can access that list of nodes with `bolt` using the following:

```
bolt command run uptime --nodes `cat nodes | paste -sd "," -`
```

On Windows you can do the same using PowerShell:

```powershell
bolt command run uptime --nodes ((Get-Content .\nodes) -join ',')
```



================================================
FILE: nodes-from-file/nodes
================================================
web1.exmaple.com
web2.exmaple.com
web3.exmaple.com
db.exmaple.com



================================================
FILE: osquery-examples/README.md
================================================
# Building your own user interface for existing tools

A handy use of `bolt` is to provide a shared user interface to existing tools. By encapsulating commonly run commands in a Puppet Task it's easier to discover and easier to share within a team.

As an example let's look at [osquery](https://osquery.io/), a tool for querying the state of a host via a SQL interface. The init task for the osquery provides a shortcut into the `osqueryi` command. But with `bolt` we can easily run those commands across all of you nodes.

Note that you can find all the code for the `osquery` tasks in `modules/osquery/tasks`. They consist of only a few lines of bash in this case.

Here's a quick example of listing all of the network routes for all the specified hosts.


```
$ bolt task run osquery query="SELECT * FROM routes" --nodes <nodes> --modules ./modules
bolt_ssh-osquery_1:

+-----------------+---------+------------+------------+-------+-----------+-----+--------+-----------+
| destination     | netmask | gateway    | source     | flags | interface | mtu | metric | type      |
+-----------------+---------+------------+------------+-------+-----------+-----+--------+-----------+
| 0.0.0.0         | 0       | 172.18.0.1 |            | 0     | eth0      | 0   | 0      | gateway   |
| 172.18.0.0      | 16      |            | 172.18.0.4 | 0     | eth0      | 0   | 0      | gateway   |
| 127.0.0.0       | 0       |            | 127.0.0.1  | 0     | lo        | 0   | 0      | broadcast |
| 127.0.0.0       | 8       |            | 127.0.0.1  | 0     | lo        | 0   | 0      | local     |
| 127.0.0.1       | 0       |            | 127.0.0.1  | 0     | lo        | 0   | 0      | local     |
| 127.255.255.255 | 0       |            | 127.0.0.1  | 0     | lo        | 0   | 0      | broadcast |
| 172.18.0.0      | 0       |            | 172.18.0.4 | 0     | eth0      | 0   | 0      | broadcast |
| 172.18.0.4      | 0       |            | 172.18.0.4 | 0     | eth0      | 0   | 0      | local     |
| 172.18.255.255  | 0       |            | 172.18.0.4 | 0     | eth0      | 0   | 0      | broadcast |
| 0.0.0.0         | 0       |            |            | 0     | lo        | 0   | -1     | other     |
| 0.0.0.0         | 0       |            |            | 0     | lo        | 0   | -1     | other     |
+-----------------+---------+------------+------------+-------+-----------+-----+--------+-----------+

Ran on 1 node in 0.37 seconds
```


The above example relies on your knowing the SQL query to run. That's obviously great if you're an `osquery` expert, but less useful when you're in a hurry or new to the tooling. In those cases it's nice to store oft-used queries as separate tasks. Rather than taking the query as an argument we've embedded a few queries in the tasks. Here are a few examples, the first one lists the processes listening on open ports:

```
$ bolt task run osquery::listening --nodes <nodes> --modules ./modules
bolt_ssh-osquery_1:

+------+------+-----+
| name | port | pid |
+------+------+-----+
| sshd | 22   | 1   |
+------+------+-----+

Ran on 1 node in 0.40 seconds
```

`osquery::processes` lists all running processes along with details of the command running:

```
$ bolt task run osquery::processes --nodes <nodes> --modules ./modules
bolt_ssh-osquery_1:

+-----+-------------+-------------------------------------------------+
| pid | name        | cmdline                                         |
+-----+-------------+-------------------------------------------------+
| 1   | sshd        | /usr/sbin/sshd -D                               |
| 260 | sshd        | sshd: root@notty                                |
| 263 | sftp-server | /usr/lib/openssh/sftp-server                    |
| 265 | bash        | bash /tmp/tmp.1Lnag0jU0t/processes.sh           |
| 266 | osqueryi    | osqueryi SELECT pid,name,cmdline FROM processes |
+-----+-------------+-------------------------------------------------+

Ran on 1 node in 0.41 seconds
```

`osquery::users` will list all users installed available on each node:

```
$ bolt task run osquery::users --nodes <nodes> --modules ./modules
bolt_ssh-osquery_1:

+-------+-------+----------+-----------+------------------------------------+
| uid   | gid   | username | groupname | description                        |
+-------+-------+----------+-----------+------------------------------------+
| 0     | 0     | root     | root      | root                               |
| 1     | 1     | daemon   | daemon    | daemon                             |
| 2     | 2     | bin      | bin       | bin                                |
| 3     | 3     | sys      | sys       | sys                                |
| 4     | 65534 | sync     | nogroup   | sync                               |
| 5     | 60    | games    | games     | games                              |
| 6     | 12    | man      | man       | man                                |
| 7     | 7     | lp       | lp        | lp                                 |
| 8     | 8     | mail     | mail      | mail                               |
| 9     | 9     | news     | news      | news                               |
| 10    | 10    | uucp     | uucp      | uucp                               |
| 13    | 13    | proxy    | proxy     | proxy                              |
| 33    | 33    | www-data | www-data  | www-data                           |
| 34    | 34    | backup   | backup    | backup                             |
| 38    | 38    | list     | list      | Mailing List Manager               |
| 39    | 39    | irc      | irc       | ircd                               |
| 41    | 41    | gnats    | gnats     | Gnats Bug-Reporting System (admin) |
| 65534 | 65534 | nobody   | nogroup   | nobody                             |
| 100   | 101   | libuuid  | libuuid   |                                    |
| 101   | 104   | syslog   | syslog    |                                    |
| 102   | 65534 | sshd     | nogroup   |                                    |
+-------+-------+----------+-----------+------------------------------------+

Ran on 1 node in 0.39 seconds
```



================================================
FILE: osquery-examples/modules/osquery/tasks/init.sh
================================================
#!/usr/bin/env bash

if [ -z "$PT_query" ]; then
  echo "Need to pass a query to run in the query argument"
  exit 1
fi

osqueryi "${PT_query}"



================================================
FILE: osquery-examples/modules/osquery/tasks/listening.sh
================================================
#!/usr/bin/env bash

osqueryi "SELECT DISTINCT processes.name, listening_ports.port, processes.pid FROM listening_ports JOIN processes USING (pid)   WHERE listening_ports.address = '0.0.0.0';"



================================================
FILE: osquery-examples/modules/osquery/tasks/logins.sh
================================================
#!/usr/bin/env bash

osqueryi "select username, pid, host from last;"



================================================
FILE: osquery-examples/modules/osquery/tasks/processes.sh
================================================
#!/usr/bin/env bash

osqueryi "SELECT pid,name,cmdline FROM processes"



================================================
FILE: osquery-examples/modules/osquery/tasks/schema.sh
================================================
#!/usr/bin/env bash

"${PT_table:?Need to specify a table, for example table=users}"

osqueryi ".schema ${PT_table}"



================================================
FILE: osquery-examples/modules/osquery/tasks/tables.sh
================================================
#!/usr/bin/env bash

osqueryi .tables



================================================
FILE: osquery-examples/modules/osquery/tasks/users.sh
================================================
#!/usr/bin/env bash

osqueryi "SELECT u.uid, u.gid, u.username, g.groupname, u.description FROM users u LEFT JOIN groups g ON (u.gid = g. gid)"



================================================
FILE: perl-task/README.md
================================================
# Writing Puppet Tasks with Perl

Perl can be a useful language to write tasks in for some organisations. Especially in a Linux environment the Perl runtime is very likely to already be installed on the node. And lots of organisations have a collection of useful Perl scripts which can quickly be moved into a module and then easily run across a collection of nodes.

Here's a very simple Perl script which reads the value of an environment variable and prints it out. Note the `PT_` prefix used by Puppet Tasks to indicate arguments passed to the task by the task runner. Save the following to `modules/sample/tasks/init.pl`

```perl
#!/usr/bin/env perl
$userName =  $ENV{'PT_message'};
print "Hello, $userName\n";
```

You can then run the above task like so, assuming the target nodes do indeed have the Perl runtime installed.

```
$ bolt task run sample message=bolt --nodes <nodes> --modules ./modules
bolt_ssh_1:

Hello, bob

Ran on 1 node in 0.40 seconds
```



================================================
FILE: perl-task/modules/sample/tasks/init.pl
================================================
#!/usr/bin/env perl 
$userName =  $ENV{'PT_message'}; 
print "Hello, $userName\n"; 



================================================
FILE: php-task/README.md
================================================
# Writing Puppet Tasks using PHP

If you're a PHP user you'll be happy to know you can write your Puppet Tasks in PHP. You'll need PHP installed on the nodes you want to target. As a hello world example save the following as `modules/sample/tasks/init.php`:

```php
#!/usr/bin/env php
<?php
echo getenv('PT_message');
?>
```

You can then run the task like so:

```
bolt task run sample message=hello --nodes <nodes> ./modules
```

For more complex tasks, maybe with multiple files or dependencies on third party libraries, it's useful to wrap tasks up as a `.phar` (PHP archive). There are several tools that can help you do that but [box](https://box-project.github.io/box2/) is nice and simple to experiment with.

First we need to create a `box.json` configuration file. Store that alongside the `init.php` file from above in the `tasks` folder.

```json
{
    "main": "init.php",
    "output": "example.phar",
    "stub": true
}
```

Note in the above we're going to package our existing `init.php` script as `example.phar`. This will make it available as the `example` task in our `sample` module, which is referenced with `sample::example`.

We then just need to install `box` and run `box build`. If you don't have PHP and Box installed locally you can use the Dockerfile provided along with this README like so.

```
docker build -t box .
docker run -it -v ${PWD}/modules/sample/tasks:/source box build
```

This will create a file at `modules/sample/tasks/example.phar`. You can run that task like so:

```
bolt task run sample::example message=hello --nodes <nodes> ./modules
```

See the full [box documentation](https://box-project.github.io/box2/) for how to bundle libraries or add additional files to the package.



================================================
FILE: php-task/Dockerfile
================================================
FROM php

RUN curl -LSs https://box-project.github.io/box2/installer.php | php
RUN mv /box.phar /usr/local/bin/box

RUN echo "phar.readonly = off" > /usr/local/etc/php/php.ini

WORKDIR /source

ENTRYPOINT ["/usr/local/bin/box"]
CMD [""]



================================================
FILE: php-task/modules/sample/tasks/box.json
================================================
{
    "main": "init.php",
    "output": "example.phar",
    "stub": true
}



================================================
FILE: php-task/modules/sample/tasks/init.php
================================================
#!/usr/bin/env php
<?php
echo getenv('PT_message');
?>



================================================
FILE: process/README.md
================================================
# process

#### Table of Contents

1. [Description](#description)
1. [Usage - examples](#usage)
    * [See all running processes](#see-all-running-processes)
    * [Query running process](#query-running-process)

## Description

This module provides tasks for querying system processes. It also provides an example of a powershell task.

## Usage

### See all running processes

See all processes running on a Windows node

```
$ puppet-task run --nodes=example.com process::win
```

A similar example works on other platforms

```
$ puppet-task run --nodes=example.com process::nix
```

### Querry running process

Get structured data about one or more running processes

```
$puppet-task run <target> process::win name='IIS*'
```

A similar example works on other platforms

```
$ puppet-task run <target> process::nix name='.*d'
```

Note that the patterns used to match process names may differ on Windows vs other
platforms. Windows uses `Get-Process`, others use `grep`.



================================================
FILE: process/metadata.json
================================================
{
  "name": "michaelsmith-process",
  "version": "0.1.0",
  "author": "michaelsmith",
  "summary": "Tasks for querying process info",
  "license": "Apache-2.0",
  "source": "https://github.com/puppetlabs/puppetlabs-process",
  "project_page": "https://github.com/puppetlabs/puppetlabs-process",
  "issues_url": "https://github.com/puppetlabs/puppetlabs-process/issues",
  "dependencies": [],
  "data_provider": null
}




================================================
FILE: process/tasks/nix.json
================================================
{
  "description": "Query running processes",
  "parameters": {
    "name": {
      "description": "A pattern to match against process names",
      "type": "Optional[String]"
    }
  }
}



================================================
FILE: process/tasks/nix.sh
================================================
#!/usr/bin/env bash

if [[ $PT_name ]]; then
    processes=`ps -ef | grep "${PT_name}"`
    if [[ $processes ]]; then
        echo '{"_items": ['
        numlines=`echo "$processes" | wc -l`
        count=0
        echo "$processes" | while read line; do
            output="{ \"Name\": \"`echo $line | cut -d' ' -f8-`\", "
            output+=$(echo $line | awk '{split($0,a," "); print "\"CPU\": \"" a[7] "\", \"Id\":" a[2] "}"}')
            let count++
            if [[ $count -ne $numlines ]]; then
                output+=","
            fi
            echo "  $output"
        done
        echo ']}'
    fi
else
    ps -ef
fi




================================================
FILE: process/tasks/win.json
================================================
{
  "description": "Query running processes",
  "parameters": {
    "name": {
      "description": "A pattern to match against process names",
      "type": "Optional[String]"
    }
  }
}



================================================
FILE: process/tasks/win.ps1
================================================
#!/usr/bin/env powershell

[CmdletBinding()]
Param(
  [Parameter(Mandatory = $False)] [String] $Name
  )

if ($Name -eq $null -or $Name -eq "") {
  Get-Process
} else {
  $processes = Get-Process -Name $Name
  $result = @()
  foreach ($process in $processes) {
    $result += @{"Name" = $process.ProcessName;
                 "CPU" = $process.CPU;
                 "Memory" = $process.WorkingSet;
                 "Path" = $process.Path;
                 "Id" = $process.Id}
  }
  if ($result.Count -eq 1) {
    ConvertTo-Json -InputObject $result[0] -Compress
  } elseif ($result.Count -gt 1) {
    ConvertTo-Json -InputObject @{"_items" = $result} -Compress
  }
}



================================================
FILE: ps1exec/README.md
================================================
# ps1exec

#### Table of Contents

1. [Description](#description)
1. [Usage - examples](#usage)
    * [Execute command](#execute-command)

## Description

This module provides tasks for running a PowerShell command.

## Usage

### Execute command

Execute a powershell command

```
$ puppet-task run --nodes=example.com ps1exec command='Get-Process'
```



================================================
FILE: ps1exec/metadata.json
================================================
{
  "name": "michaelsmith-powershell",
  "version": "0.1.0",
  "author": "michaelsmith",
  "summary": "Tasks for running a PowerShell command",
  "license": "Apache-2.0",
  "source": "https://github.com/puppetlabs/puppetlabs-powershell",
  "project_page": "https://github.com/puppetlabs/puppetlabs-powershell",
  "issues_url": "https://github.com/puppetlabs/puppetlabs-powershell/issues",
  "dependencies": [],
  "data_provider": null
}




================================================
FILE: ps1exec/tasks/init.json
================================================
{
  "description": "Execute a PowerShell command",
  "parameters": {
    "command": {
      "description": "The PowerShell command",
      "type": "String"
    }
  }
}



================================================
FILE: ps1exec/tasks/init.ps1
================================================
#!/usr/bin/env powershell

Param([Parameter(Mandatory=$True)] [String] $Command)

Invoke-Expression $Command



================================================
FILE: puppet-task/README.md
================================================
# Run ad-hoc Puppet manifests with Bolt

Bolt can be used to upload and run arbitrary scripts on remote nodes. If those nodes have Puppet installed on them then you can easily run arbitrary Puppet code using this feature.

First lets create a simple manifest. Note the shebang line of `#!/opt/puppetlabs/puppet/bin/puppet apply`. This means when the script is executed it should be run with `puppet apply`. The manifest itself it just a simple Puppet example of ensuring a file is present with some specific content.

```puppet
#!/opt/puppetlabs/puppet/bin/puppet apply

file { '/tmp/bolt':
  ensure  => present,
  content => 'bolt hello world',
}
```

We can run this against a list of nodes like so:

```
bolt script run manifest.pp -n <nodes>
```

This should result in a response something like the following. We see the standard Puppet output and a notice about the newly created file.

```
bolt_ssh_1:

Notice: Compiled catalog for cea827639e13 in environment production in 0.02 seconds
Notice: /Stage[main]/Main/File[/tmp/bolt]/ensure: defined content as '{md5}74d9194aec98b3f9d75912afdca19732'
Notice: Applied catalog in 0.02 seconds

Ran on 1 node in 5.63 seconds
```

We can use `bolt` to check on the file as well:

```
$ bolt command run "cat /tmp/bolt"
bolt_ssh_1:

bolt hello world
Ran on 1 node in 0.07 seconds
```

If we ran the same manifest again, against the same node, we should see the idempotent nature of Puppet in that the file already exists with the content and so no notice is present indicating a change occured.

```
bolt_ssh_1:

Notice: Compiled catalog for cea827639e13 in environment production in 0.02 seconds
Notice: Applied catalog in 0.02 seconds

Ran on 1 node in 5.65 seconds
```



================================================
FILE: puppet-task/manifest.pp
================================================
#!/opt/puppetlabs/puppet/bin/puppet apply

file { '/tmp/bolt':
  ensure  => present,
  content => 'bolt hello world',
}



================================================
FILE: python-task/README.md
================================================
# Writing Puppet Tasks in Python

Tasks for `bolt` can be written in any language, they just require the runtime to be available on the target node. Let's have a look at writing simple Python tasks to illustrate this.

Save the following as `modules/sample/tasks/init.py`.

Note that the `sample` part is the name of our module. We could have used anything here but note it's used to build out the user interface as we'll see in a moment.

```python
#!/usr/bin/env python
import os
print os.environ['PT_message']
```

Note as well that this is just a standard Python script. It's using the standard library `os` module and then printing out the `PE_message` environment variable. Task arguments are automatically passed along to tasks and prefixed with `PT_` for Puppet Tasks.

The name `init` is special. It means this task is the default for this module, so if you don't explicitly run a named task the `init` task will be run. Here's an example of running that task.

```
bolt task run sample message=hello --nodes <nodes> --modules ./modules
```

Note:

* the `--modules` flag points at the modules diretory we created above with out `sample` module in
* the argument `message` matches the name of our environment variable above
* `sample` is the name of our module. We're not naming a task explicitly so bolt will run the `init` task

This is a simple way of running any arbitrary Python code on a large number of remote nodes, and packaging that code up so it's easy to share with the rest of your team or the wider community.


## Managing dependencies

The above example just uses the Python standard library. But what if our task wants to use a third party library? We could just install that library on our remote machines, but that requires upfront configuration which might not always be possible. It would be nice if we could rely just on the runtime and nothing else. Lucily with [Pex](https://github.com/pantsbuild/pex) we can do exactly that.

Pex is a packaging format for Python which packages up our code along with any required libraries and can create a single executable. Perfect for distribution with `bolt`. Let's see a very simple example of doing that.

To create a Pex package we need to have a `setup.py` file. A simple example might look like the following:

```python
from distutils.core import setup

setup(
    name='samplepkg',
    version='0.1',
    packages=['samplepkg'],
)
```

All of the code samples needed to run this example accompany this README.

Next we need to write the code for our task. We'll save that to `samplepkg/main.py`.

```python
#!/usr/bin/env python

import os
import colored
from colored import stylize

print(stylize(os.environ['PT_message'], colored.fg("red")))
```

We'll also need a blank file at `samplepkg/__init__.py` to indicate that `samplepkg` is a Python package.

Note in the above we're using the [colored](https://pypi.python.org/pypi/colored) Python package from PyPi. We're using that to style the printing of the output in red.

Finally we need to build the pex package. For that you'll need to install Pex:

```
pip install pex
```

And finally build the intermediary wheel and save the `.pex` file to our tasks folder.

```
pip wheel -w . .
pex -f ${PWD} samplepkg colored -e samplepkg.main -o modules/sample/tasks/samplepkg.pex
```

We can now run our task. Note the `::samplepkg` part of the command below. `::` separates the module name from the task name. `samplepkg` is the filename of our `pex` file above.

```
bolt task run sample::samplepkg message=color --nodes <nodes> --modules ./modules
```

This should print out the message in red rather than the default terminal color. Obviously printing output in a different color is just a simple example of the kinds of things you can do. This approach makes it trivial to use the huge Python ecosystem of libraries to help write tasks to automate your infrastructure.



================================================
FILE: python-task/Dockerfile
================================================
FROM python:2

RUN pip install pex




================================================
FILE: python-task/setup.py
================================================
from distutils.core import setup

setup(
    name='samplepkg',
    version='0.1',
    packages=['samplepkg'],
)



================================================
FILE: python-task/modules/sample/tasks/init.py
================================================
#!/usr/bin/env python

import os
print os.environ['PT_message']



================================================
FILE: python-task/modules/sample/tasks/main.py
================================================
#!/usr/bin/env python

import os
print os.environ['PT_message']



================================================
FILE: python-task/modules/sample/tasks/samplepkg.pex
================================================
[Binary file]


================================================
FILE: python-task/samplepkg/__init__.py
================================================
[Empty file]


================================================
FILE: python-task/samplepkg/main.py
================================================
#!/usr/bin/env python

import os
import colored
from colored import stylize

print(stylize(os.environ['PT_message'], colored.fg("red")))



================================================
FILE: r10k-manage-tasks/README.md
================================================
# Installing tasks with r10k

Tasks are distributed as Puppet modules, which means existing tooling that works with Puppet modules already supports tasks. We can demonstrate that by using [librarian-puppet](https://github.com/puppetlabs/r10k) and a `Puppetfile` to download and manage a set of modules.

The following assumes you have a Ruby environment available. First install `r10k` and it's dependencies.

```
gem install r10k
```

We then need a `Puppetfile` which describes the modules we want to download. We're referencing the GitHub repositories for the modules for now because they modules are currently private and only available internally.

```
forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs-package',
  :git => "https://github.com/puppetlabs/puppetlabs-package.git"

mod 'puppetlabs-service',
  :git => "https://github.com/puppetlabs/puppetlabs-service.git"
```

We then run `r10k` to download the modules to the `./modules` folder.

```
r10k puppetfile install --moduledir modules
```

This should create a `./modules` folder if one doesn't already exist and download the `package` and `service` module.

```
$ ls modules
package  service
```

With those modules downloaded we can use them with `bolt` using the `--modules` flag.

```
bolt task run package action=status package=vim --modules ./modules --nodes <nodes>
```

Assuming we run this against an Ubuntu machine without vim installed you can expect to see output like the following:

```
bolt_ssh_1:

{"status":"absent","latest":"2:7.4.052-1ubuntu3.1"}

Ran on 1 node in 4.40 seconds
```



================================================
FILE: r10k-manage-tasks/Gemfile
================================================
source "https://rubygems.org"

gem "r10k"



================================================
FILE: r10k-manage-tasks/Puppetfile
================================================
forge "https://forgeapi.puppetlabs.com"

mod 'puppetlabs-package',
  :git => "https://github.com/puppetlabs/puppetlabs-package.git"

mod 'puppetlabs-service',
  :git => "https://github.com/puppetlabs/puppetlabs-service.git"



================================================
FILE: run-different-tasks-on-different-operating-systems/README.md
================================================
# Run different tasks on different operating systems

It is common that plans need to make decisions based on the operating
system installed on the remote node. If those nodes have Puppet installed,
you can just run `facter --json` on those nodes with the `run_command`
function you learned about in the last exercise.

If those nodes do not have Puppet installed, you can use the
[`minifact`](https://github.com/puppetlabs/task-modules/blob/master/minifact/tasks/init.sh)
task from the [task-modules](https://github.com/puppetlabs/task-modules)
repo to gather some very basic facts. To do that, check out that repo with
`git clone https://github.com/puppetlabs/task-modules.git`; we will refer
to the location where you checked that repo out as `$task_modules`. You can
now run `minifact` directly using

```bash
bolt --modules $task_modules --nodes <nodes> task run minifact
```

When we need to access the result of running `minifact` inside a plan, we
first assign that result to a variable, and can then access the output from
each node. The result of running a task acts like a hash, where the node
name is the key, and the associated value is the output from the task:

```puppet
   $minifacts = run_task('minifact', $nodes)
   $minifacts.each |$nd, $out| {
     # $nd is the name of the node, and $out contains the output
     # that that node produced
   }
```

As a simple example of using task results in a plan, we will just print the
output from `minifact` more concisely using a plan. The
[`minifact::info`](https://github.com/puppetlabs/task-modules/blob/master/minifact/plans/info.pp)
plan simply runs `minifact` and then loops over the result:

```puppet
plan minifact::info(String $nodes) {
  $all = $nodes.split(",")
  $minifacts = run_task('minifact', $all)
  $minifacts.each |$nd, $out| {
    util::print("${nd}: ${out[os][name]} ${out[os][release][full]} (${out[os][family]})")
  }
  undef
}
```

Since `minifact` returns a JSON object, Puppet automatically parses that
and makes it available as the Hash that we access with `$out`. If the task
does not return JSON, the entire output can be found in `$out[output]`.

The `minifact::info` plan can be run with

```bash
    bolt --modules $task_modules plan run minifact::info nodes=node1,node2,...
```


In heterogeneous environments, it is often necessary to run different tasks
to achieve the same result. As an example, we will write a plan that
collects the versions of installed packages across Debian and Red Hat
machines, but uses different tasks to do so, one using `dpkg` and one using
`rpm`.

To follow along with the example, you will need to copy the `example8`
module to your `$task_modules` directory:

```bash
    cp -pr modules/example8 $task_modules
```

The `tasks` directory in `example8` contains two tasks: `dpkg.py`, a Python
script that runs `dpkg` to get information about a package, and `rpm.sh`, a
shell script that does the same using `rpm`. The important thing about them
is that they expect the same input (a package name in `$PT_package`) and
produce a JSON object with the exact same format as output.

The [`example::packages`](modules/example8/plans/packages.pp) plan first
runs `minifact` on all nodes and uses the `os.family` fact to select nodes
from Debian-like and from RedHat-like nodes. It then runs the
`example8::dpkg` task and the `example8::rpm` task on them and prints
information returned by these. The function `example8::print_node_result`
is a small helper function that just prints the package result from one
node.

```puppet
    plan example8::packages(String $nodes, String $package = "openssl") {
      $all = $nodes.split(",")
      $minifacts = run_task('minifact', $all)

      $debian = $all.filter |$nd| { $minifacts[$nd][os][family] == "Debian" }
      $redhat = $all.filter |$nd| { $minifacts[$nd][os][family] == "RedHat" }

      $rpms = run_task('example8::rpm', $redhat, package => $package)
      $debs = run_task('example8::dpkg', $debian, package => $package)

      $rpms.each |$nd, $out| { example8::print_node_result($nd, $out, "rpm") }
      $debs.each |$nd, $out| { example8::print_node_result($nd, $out, "deb") }

      undef
    }
```

You can run this plan with
```
    bolt --modules $task_modules plan run example8::packages nodes=<nodes> package=openssl
```



================================================
FILE: run-different-tasks-on-different-operating-systems/modules/example8/functions/print_node_result.pp
================================================
function example8::print_node_result(String $nd, $out, String $kind) {
  $out[result].each |$res| {
    util::print("$nd: ${res[name]} ${res[version]} ${res[arch]} $kind")
  }
}



================================================
FILE: run-different-tasks-on-different-operating-systems/modules/example8/plans/packages.pp
================================================
plan example8::packages(String $nodes, String $package = "openssl") {
  $all = $nodes.split(",")
  $minifacts = run_task('minifact', $all)

  $debian = $all.filter |$nd| { $minifacts[$nd][os][family] == "Debian" }
  $redhat = $all.filter |$nd| { $minifacts[$nd][os][family] == "RedHat" }

  $rpms = run_task('example8::rpm', $redhat, package => $package)
  $debs = run_task('example8::dpkg', $debian, package => $package)

  $rpms.each |$nd, $out| { example8::print_node_result($nd, $out, "rpm") }
  $debs.each |$nd, $out| { example8::print_node_result($nd, $out, "deb") }

  undef
}



================================================
FILE: run-different-tasks-on-different-operating-systems/modules/example8/tasks/dpkg.py
================================================
#! /usr/bin/python

import os
import json


def split_line(line):
    split = line.split()
    return { "name": split[1], "version": split[2], "arch": split[3] }

command = 'dpkg -l %s | grep "^ii"' % os.getenv("PT_package")

lines = os.popen(command).read().split('\n')

pkgs = [ split_line(line) for line in lines if len(line) > 0]
print json.dumps({ "result": pkgs})



================================================
FILE: run-different-tasks-on-different-operating-systems/modules/example8/tasks/rpm.sh
================================================
#! /bin/bash

echo '{ "result": ['

first=yes

rpm -q --qf '%{NAME} %{VERSION} %{ARCH}\n' $PT_package |
    while read name version arch
    do
        if [[ "$first" = yes ]]
        then
            first=no
        else
            echo ","
        fi
        cat <<EOF
{
   "name": "$name",
   "version": "$version",
   "arch": "$arch" }
EOF
    done
echo "] }"



================================================
FILE: task/README.md
================================================
# task

#### Table of Contents

1. [Description](#description)
1. [Setup - The basics of getting started with task](#setup)
    * [Beginning with task](#beginning-with-task)
1. [Usage - examples](#usage)
    * [Kill Tasks](#kill-a-running-task)
    * [Retrieve stderr](#retrieve-stderr)

## Description

This module provides tasks for interacting with other orchestrator tasks. It
also provides an example of a python task.

## Usage

### Kill a running task

If you need to kill a running task on some nodes you can use the `task::kill`
task to kill those tasks by job_id

```
$ puppet-task run --nodes=example.com task::kill job_id=42
```

### Retreive stderr

If you need to retrieve the stderr output from the execution of a task you can
use the `task::stderr task.

```
$puppet-task run <target> task::stderr job_id=42
```



================================================
FILE: task/metadata.json
================================================
{
  "name": "adreyer-task",
  "version": "0.1.0",
  "author": "adreyer",
  "summary": "Tasks for interacting with tasks",
  "license": "Apache-2.0",
  "source": "https://github.com/puppetlabs/puppetlabs-task",
  "project_page": "https://github.com/puppetlabs/puppetlabs-task",
  "issues_url": "https://github.com/puppetlabs/puppetlabs-task/issues",
  "dependencies": [],
  "data_provider": null
}




================================================
FILE: task/tasks/kill.json
================================================
{
  "input_method": "stdin",
  "description": "Kill a running task",
  "parameters": {
    "job_id": {
      "description": "The job_id of the task to kill",
      "type": "String"
    },
    "signal": {
      "description": "The signal to send to the task process.(default: kill)",
      "type": "Optional[Enum[kill, term]]"
    },
    "timeout": {
      "description": "How many seconds to poll for",
      "type": "Optional[Integer]"
    }
  },
  "output": {
    "status": {
      "description": "The status of the task process",
      "type": "Enum[running, finished, killed]"
    }
  }
}



================================================
FILE: task/tasks/kill.py
================================================
#!/usr/bin/env python

import json
import os
import signal
import sys
import time

SIGNALS = {
  'term': signal.SIGTERM,
  'kill': signal.SIGKILL,
}

params = json.load(sys.stdin)

job_id = params['job_id']
signal = SIGNALS[params.get('signal', 'kill')]
timeout = params.get('timeout', 1)

def task_file(job_id, filename):
  "Traverse up from the task file -> the spool"
  return os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', 'spool', job_id, filename))

def make_error(kind, msg, details=None):
  "Make and task_error response"
  return {
     "_error": {
       "kind": kind,
       "msg": msg,
       "details": details or {},
       }
     }

pid_file = task_file(job_id, 'pid')
try:
  with open(pid_file) as fh:
    pid = int(fh.read())
except IOError:
  json.dump(make_error("file-not-found", 'could not find pid file: %s' % pid_file, {"path": pid_file}), sys.stdout)
  exit(5)

status = "running"
for i in range(timeout):
  try:
    os.kill(pid, signal)
  except OSError as e:
    # This is the error when the pid isn't found
    if e[0] == 3:
      if i == 0:
        status = "finished"
      else:
        status = "killed"
      break
    else:
      json.dump(make_error("process_error", e[1], {"pid": pid}), sys.stdout)
      exit(1)
  time.sleep(1)

json.dump({"status":  status}, sys.stdout)



================================================
FILE: task/tasks/stderr.json
================================================
{
  "input_method": "stdin",
  "description": "Retrieve stderr from a task",
  "parameters": {
    "job_id": {
      "description": "The job_id of the task to kill",
      "type": "String"
    }
  },
  "output": {
    "stderr": {
      "description": "The output of stderr",
      "type": "String"
    }
  }
}



================================================
FILE: task/tasks/stderr.py
================================================
#!/usr/bin/env python
import json
import os
import sys

params = json.load(sys.stdin)

job_id = params['job_id']


def task_file(job_id, filename):
  "Traverse up from the task file -> the spool"
  return os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', 'spool', job_id, filename))

def error(code=1, kind="runtime-error", msg="unexpected error running task", details=None):
  "Make and task_error response"
  result =  {
     "_error": {
       "kind": kind,
       "msg": msg,
       "details": details or {},
       }
     }
  json.dump(result, sys.stdout)
  exit(code)

stderr_file = task_file(job_id, 'stderr')
try:
  with open(stderr_file) as fh:
    stderr = fh.read()
except IOError:
  error(5, "file-not-found", 'could not find file: %s' % pid_file, {"path": stderr_file})

json.dump({"stderr": stderr}, sys.stdout)