Nanobox vs Docker Swarm & Kubernetes


This article is taken from conversation in the Nanobox Slack channel. Someone asked a question about how Nanobox's orchestration model differs from others like Docker Swarm and Kubernetes. It's a question we get asked fairly often and thought it would make a good article.


Generally speaking, Nanobox treats containers as lightweight VMs, so many concepts from Docker Swarm or Kubernetes won’t translate directly with our system. If you’re familiar with Solaris zones, the way we treat containers is very much like a Solaris zone.

At a high level, Docker (Swarm) and Kubernetes move the process management, logging, supervision, networking, etc., out of the container and into the orchestration layer. Details aside, conceptually they are moving aggressively towards the notion of running simple processes inside of a kernel namespace and a filesystem chroot, then having the orchestration layer handle everything else.

Ultimately, the developer is left to define the service details in every implementation, like which commands to run, etc. I don’t necessarily think that’s a bad thing, in fact I think the implementation of Kubernetes’ pods is pretty clever. That's just the direction they're going.

Our system was designed differently – on the notion of autonomy. Essentially, a Docker image should have everything it needs to fulfill a responsibility across different roles and states. The orchestration framework does two things:

  1. Runs the container (lightweight VM).
  2. Transitions the container between states and roles.

Conceptual Example

Let's use Postgres as an example. A Postgres database can be a standalone node, part of a cluster, the leader or follower in a failover setup, or it can simply be a backup node. Our system was designed to allow a container to come online and inform Nanobox of its capabilities. Later, Nanobox can simply tell it to change to a new state or transition into a new role and it will do it. Nanobox doesn’t have to know anything about how the container should behave in the new role. It can just say, "Hey database, become the leader" and it will do so.

All the logic to implement that behavior and functionality is encapsulated within the Docker image itself, and consequently, within the container that uses the image. Nanobox ultimately becomes a powerful control plane that completely alters the form of an entire cluster just by starting containers and changing their state or re-assigning roles.

Patterned After Human Organizations

We designed Nanobox to mimic an organization of real people. You can (hopefully) tell someone on your team that something needs to be done and, assuming that individual has that skill set, they will do it. You don’t have to setup the perfect ecosystem for them to be able to carry out simple responsibilities, then spoon-feed them instructions.

I’m not saying that Kubernetes and Docker Swarm are going about it the wrong way, just a different way than Nanobox. They're ultimately following the same path as systemd – do as much as possible in one big layer. We tend to prefer the Unix philosophy, even at a service orchestration level.

Real-World Example

For a real-world example, I’ll show you the Nanobox Redis image, since both it and Redis are crazy simple.

First off, the Dockerfile...You’ll notice that it only installs software. It doesn’t make any assumptions about how and when it will be used.

# -*- mode: Dockerfile; tab-width: 4;indent-tabs-mode: nil;-*-
# vim: ts=4 sw=4 ft=Dockerfile et: 1
FROM nanobox/runit

# Create directories
RUN mkdir -p /var/log/gonano

# Install arping
RUN apt-get update -qq && \
    apt-get install -y iputils-arping cron && \
    apt-get clean all && \
    rm -rf /var/lib/apt/lists/*

USER gonano

# Install binaries
RUN rm -rf /data/var/db/pkgin && \
    /data/bin/pkgin -y up && \
    /data/bin/pkgin -y in \
        luvit-0.10.0 \
        flip \
        flip-scripts \
        py27-rdbtools \
        redundis \
        redis-4.0 && \
    rm -rf /data/var/db/pkgin/cache

USER root

RUN rm -rf /var/gonano/db/pkgin && \
    /opt/gonano/bin/pkgin -y up && \
    /opt/gonano/bin/pkgin -y in \
        jq \
        rsync && \
    rm -rf /var/gonano/db/pkgin/cache

RUN /opt/gonano/bin/gem install remote_syslog_logger

# Install hooks
RUN mkdir -p /opt/nanobox/hooks && \
    mkdir -p /var/nanobox && \
    curl \
      -f \
      -k \
      https://s3.amazonaws.com/tools.nanobox.io/hooks/redis-stable.tgz \
        | tar -xz -C /opt/nanobox/hooks && \
    curl \
      -f \
      -k \
      -o /var/nanobox/hooks.md5 \
      https://s3.amazonaws.com/tools.nanobox.io/hooks/redis-stable.md5

# Cleanup disk
RUN rm -rf \
        /var/lib/apt/lists/* \
        /tmp/* \
        /var/tmp/*

WORKDIR /data

# Run runit automatically
CMD [ "/opt/gonano/bin/nanoinit" ]

One of the things it installs are hooks, which allow Nanobox to "hook" into the service. The first hook being the "plan" hook. The plan hook tells Nanobox what behaviors and topologies the image can be used for.

#!/usr/bin/env ruby

# hookit is installed as a bundled app, so we need bundler to load it for us
$:.unshift  '/opt/gonano/hookit/vendor/bundle'
require 'bundler/setup'

# load hookit/setup to bootstrap hookit and import the dsl
require 'hookit/setup'

require 'json'

plan = {
  redundant: false,
  horizontal: false,
  users: [],
  ips: [:default],
  port: 6379,
  behaviors: [:migratable, :backupable]
}

puts JSON.pretty_generate(plan)

By default, when the container comes online, Redis doesn't even start. When Nanobox needs Redis to start, it runs the "start" hook. Nanobox doesn’t know how to start Redis. It's up to the Redis image to know how to start the Redis process.

#!/usr/bin/env ruby

# hookit is installed as a bundled app, so we need bundler to load it for us
$:.unshift  '/opt/gonano/hookit/vendor/bundle'
require 'bundler/setup'

# load hookit/setup to bootstrap hookit and import the dsl
require 'hookit/setup'

# Import service (and start)
directory '/etc/service/cache' do
  recursive true
end

directory '/etc/service/cache/log' do
  recursive true
end

template '/etc/service/cache/log/run' do
  mode 0755
  source 'log-run.erb'
  variables ({ svc: "cache" })
end

template '/etc/service/cache/run' do
  mode 0755
  variables ({ exec: "redis-server /data/etc/redis/redis.conf 2>&1" })
end

sleep 5

service "cache" do
  action :enable
end

ensure_socket 'cache' do
  port '6379'
  action :listening
end

# Attach the IP if provided
if payload[:ips]
  # set up persistance
  file "/etc/nanoinit.d/eth00" do
    content <<-EOF
#!/bin/bash
case $1 in
  start)
    if [[ ! $(ifconfig) =~ eth0:0 ]]; then
      ifconfig eth0:0 #{payload[:ips][:default]}
      arping -A -c 10 -I eth0 #{payload[:ips][:default]}
    fi
    ;;
  stop)
    if [[ $(ifconfig) =~ eth0:0 ]]; then
      ifconfig eth0:0 down
    fi
    ;;
  *)
    echo "$0 start|stop"
    exit 1
    ;;
esac
EOF
    mode 0755
  end

  # bring up interface
  execute "bring up vip" do
    command <<-EOF
      /etc/nanoinit.d/eth00 start
    EOF
  end
end

if Dir.exist? "/opt/nanobox/cron"
  directory '/etc/service/cron'

  hook_file '/etc/service/cron/run' do
    source 'cron'
    mode 0755
  end

  service 'cron' do
    action :enable
    only_if { File.exist?('/etc/service/cron/run') }
  end
end

Same with configuration. Nanobox will provide the "configure" hook some generic data and Redis will use that (or ignore it) however it needs.

#!/usr/bin/env ruby

# hookit is installed as a bundled app, so we need bundler to load it for us
$:.unshift  '/opt/gonano/hookit/vendor/bundle'
require 'bundler/setup'

# load hookit/setup to bootstrap hookit and import the dsl
require 'hookit/setup'

include Hooky::Redis

config = converge( Hooky::Redis::CONFIG_DEFAULTS, payload[:config] || {} )

if payload[:platform] != 'local'
  # Setup root keys for data migrations
  directory '/root/.ssh' do
    recursive true
  end

  file '/root/.ssh/id_rsa' do
    content payload[:ssh][:admin_key][:private_key]
    mode 0600
  end

  file '/root/.ssh/id_rsa.pub' do
    content payload[:ssh][:admin_key][:public_key]
  end

  file '/root/.ssh/authorized_keys' do
    content payload[:ssh][:admin_key][:public_key]
  end

  # Create some ssh host keys
  execute "ssh-keygen -f /opt/gonano/etc/ssh/ssh_host_rsa_key -N '' -t rsa" do
    not_if { ::File.exists? '/opt/gonano/etc/ssh/ssh_host_rsa_key' }
  end

  execute "ssh-keygen -f /opt/gonano/etc/ssh/ssh_host_dsa_key -N '' -t dsa" do
    not_if { ::File.exists? '/opt/gonano/etc/ssh/ssh_host_dsa_key' }
  end

  execute "ssh-keygen -f /opt/gonano/etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa" do
    not_if { ::File.exists? '/opt/gonano/etc/ssh/ssh_host_ecdsa_key' }
  end

  execute "ssh-keygen -f /opt/gonano/etc/ssh/ssh_host_ed25519_key -N '' -t ed25519" do
    not_if { ::File.exists? '/opt/gonano/etc/ssh/ssh_host_ed25519_key' }
  end
end

# make sure the env dir exists
directory "/data/etc/env.d" do
  recursive true
end

# and that it's owned by gonano
execute "chown gonano /data/etc/env.d"

(payload[:env] || {}).each do |key, value|
  file "/data/etc/env.d/#{key}" do
    content value
    owner 'gonano'
    group 'gonano'
    mode 0444
  end
end

if ['default', 'primary', 'secondary'].include? payload[:member][:role]
  directory '/data/var/db/redis' do
    recursive true
  end

  if payload[:platform] == 'local'
    maxmemory = 128
    appname   = 'nanobox'
  else
    total_mem = `vmstat -s | grep 'total memory' | awk '{print $1}'`.to_i
    cgroup_mem = `cat /sys/fs/cgroup/memory/memory.limit_in_bytes`.to_i
    maxmemory = [ total_mem / 1024, cgroup_mem / 1024 / 1024 ].min
    appname   = 'nanobox'
  end

  # chown data/var/db/redis for gonano
  execute 'chown /data/var/db/redis' do
    command 'chown -R gonano:gonano /data/var/db/redis'
  end

  directory '/data/etc/redis' do
    recursive true
  end

  # Configure redis
  template '/data/etc/redis/redis.conf' do
    mode 0755
    source 'redis.conf.erb'
    variables ({ config: config, maxmemory: maxmemory })
  end

  if payload[:logvac_host]
    # Configure narc
    template '/opt/gonano/etc/narc.conf' do
      variables ({ uid: payload[:component][:uid], logvac: payload[:logvac_host] })
    end
    # ensure log files are created
    ["/var/log/gonano/cache/current"].each do |log_file|
      if not ::File.exists? "#{log_file}"
        parent = File.expand_path("..", "#{log_file}")
        
        # create the parent directory
        directory parent do
          owner 'gonano'
          group 'gonano'
          recursive true
        end
        
        # create the log_file
        file "#{log_file}" do
          owner 'gonano'
          group 'gonano'
          mode  0644
        end
      end
    end

    directory '/etc/service/narc'

    template '/etc/service/narc/run' do
      mode 0755
      source 'run-root.erb'
      variables ({ exec: "/opt/gonano/bin/narcd /opt/gonano/etc/narc.conf" })
    end
  end
end

# Install extra packages

# Add extra paths
if payload[:extra_path_dirs] && payload[:extra_path_dirs].length > 0
  directory "/data/etc/env.d" do
    recursive true
    owner 'gonano'
    group 'gonano'
  end

  file "/data/etc/env.d/EXTRA_PATHS" do
    content payload[:extra_path_dirs].join(":")
    owner 'gonano'
    group 'gonano'
    mode 0444
  end
end

if payload[:extra_packages]
  execute "Install packages" do
    command "/data/bin/pkgin -y in #{payload[:extra_packages].join(' ')}"
    path "/data/sbin:/data/bin:/opt/gonano/sbin:/opt/gonano/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    user 'gonano'
  end
end

if payload[:extra_steps]
  payload[:extra_steps].each_with_index do |cmd, i|

    execute "Extra Step: #{cmd}" do
      command "bash -i -l -c \"#{escape cmd}\""
      cwd "/data"
      path "/data/sbin:/data/bin:/opt/gonano/sbin:/opt/gonano/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      user 'gonano'
    end
  end
end

# Set up cron
if payload[:member][:uid] == 1 && ! payload[:cron_jobs].nil?
  include Hookit::Helper::Cron
  include Hookit::Helper::Shell

  directory '/opt/nanobox/cron/'

  payload[:cron_jobs].each_with_index do |job, id|
    template "/opt/nanobox/cron/#{job[:id]}" do
      owner 'gonano'
      source 'cron.erb'
      group 'gonano'
      mode 0755
      variables ({
        component_uid: payload[:component][:uid],
        member_uid: payload[:member][:uid],
        logvac_host: payload[:logvac_host],
        command: escape_shell_string(job[:command]),
        cron_id: id + 1
      })
    end
  end

  template '/etc/crontab' do
    mode 0644
    variables ({ crons: payload[:cron_jobs] })
  end

end

The Framework

Containers are simply agents or actors. They are smart enough to implement the service, behaviors, and topologies on their own. Nanobox is responsible for telling them what they should be and when, but not how to do it. Essentially, we're encapsulating behavior (hooks) inside services (containers).

The end result is a very flexible, automatable system.

In Conclusion

In a lot of ways, comparing Nanobox to Kubernetes or Swarm is difficult because they really want you to define a service with specific run commands (IE: this service == these containers with these commands). Nanobox is much more than that. Nanobox expects to be able to compose services across various states and configurations. Meaning (and you’ll see this in the coming weeks), Nanobox will let you take a database, convert it into a redundant cluster, and then back into a single node. The data will stay around while that happens. Also, (something you’ll see this in the coming months), Nanobox will be able to move an entire app to a different datacenter. All of that is possible because Nanobox can remote-control services in realtime.

Hopefully that paints a clear picture of how Nanobox compares with Swarm and Kubernetes. If you have thoughts to share, post them in the comments below. We'd love to hear them.

Posted in Nanobox, Docker, Kubernetes