Getting Social with Mastodon and Nanobox

It’s hard to use the Internet these days without using social media at some point along the way. It’s a great place to stay connected​ with friends, family, and even customers. Some even use it as a personal blog, instead of something like WordPress or Blogger, and most who don’t still post links to their personal blog entries on their social media so their followers will know about them.

You may have heard of the latest new social media platform, an OSocial implementation called Mastodon. It’s an open source project hosted on GitHub, and powered by Ruby on Rails, ReactJS, and Redux. Because it supports OSocial, you can install your own instance anywhere, and connect to other instances throughout the OSocial Federation. A full exploration of Mastodon is beyond the scope of this article, but you can read more here.

The biggest hurdle to launching your own Mastodon instance is all the moving pieces involved. The official repo has everything you need to use Docker Compose, Vagrant, Heroku, or Scalingo, but each of those has its own limitations and learning curves to optimize it for the platform. And none is really geared toward using the same tooling in both development and production, so if you want to work on any code customizations, even for something as simple as changing the styling, you have to set up one environment (usually Vagrant), then deploy using another, and then work out any differences in the configuration between your development and production environments manually. It’s a lot of work, and most of it is actually unnecessary.

Mastadon and Nanobox

Nanobox is a toolset for spinning up disposable but reproducible development environments, then deploying those same environments to production, with little to no reconfiguration. Any configuration that is needed - or, to account for any unavoidable differences between environments (hot-reload versus compiled assets, running background operations in a separate environment from web requests, etc), reconfiguration - is performed automatically by specifying the necessary components and steps in a configuration file called boxfile.yml in the project root.

Setting Up For Nanobox

I recently spent a few days getting Mastodon set up with Nanobox. Most of that time was spent tracking down fixes for Ruby Gems with external dependencies on system libraries (and becoming more familiar with Rails itself in the process), and the rest was tuning. The end result was merged into the main codebase, but you can still see the PR (and each of the changes it introduced) here, if you’re interested. In the interest of seeing how to set up other projects using Nanobox, let’s take a look at each of those changes, and what each of them does in terms of getting Mastodon up and running.

NOTE: Jump ahead to Deploying An Instance if you just want to get started. This section isn't required to understand the rest.

boxfile.yml

As mentioned above, this is the core configuration for using Nanobox with your projects. As a simple YAML file, it’s pretty straightforward, but some of the terms may be a little unfamiliar, so I’ll give a quick overview of what the file tells Nanobox to do.

  • The file starts with the runtime configuration, which is essentially every aspect of the project which should be identical between development and production. Nearly everything you want to configure is specified here.

    • The first block of the file specifies that this is a Ruby project, and which version to use.

    • Next up is a block that specifies a number of extra packages to install, which are dependencies of the project, but not included with Ruby itself. It is broken into subsections with comments explaining what each package is included for.

    • The next block defines a dependency cache, a part of the project tree that is preserved between builds to decrease build times.

    • Next up is an instruction to look for commands given during the build in a custom location, in addition to any other places the system is already configured to look.

    • The next block is one of the more important ones. It tells Nanobox which files to check to determine if it should rebuild the project before it deploys (or runs the development environment; more on this later). If none of these files have changed, the build step can safely be skipped.

    • Now we come to the actual build process itself. We copy the Nanobox-specific .env template into place for development mode (more on that later), then install Bundler, configure a couple of packages that have external dependencies on system libraries, and then install all the project’s dependencies, both for Ruby and for JavaScript. These steps are performed after all the other binaries we specified earlier in the file have been installed.

    • The last line in the run.config section tells Nanobox to watch for changes to project files when run in development, and notify the development container about them so hot-reloading features will trigger in your project. It’s one of the most often forgotten options for the boxfile.yml - I’ve fielded a number of questions about it on the Nanobox Slack server. So remember, always include fs_watch: true in your boxfile.yml if you want code changes to be noticed by your running code.

  • The deploy configuration section supports a number of different phases; we’ll be using three of them here.

    • First, on your local machine, we’ll precompile assets and (unless you’ve created one of your own) create a production .env from our template, though ensuring HTTPS is enabled.

    • Then, on your production server, we rebuild the .env file to contain the actual values for your environment (the .env parser used by the Node.js portions of Mastodon only supports literal values, not references to other variables), and ensure the production log file exists for Rails so it won’t complain about not being able to write to it later when the application starts. We also process the Nginx configuration files for the web and streaming servers so they use the correct Content Security Policies - Nginx doesn’t make it easy to process environment variables, so we use erb to pull these values in for us before Nginx is actually started.

    • Finally, right after the code is deployed, but right before the deploy goes live, we run database migrations.

    • All these steps ensure everything is properly prepared before the application is even exposed to the public Internet, so your users won’t experience a broken deployment.

  • The next two sections define web server components.

    • The start block tells the server what commands to run in this container to be considered “online”. Nanobox will monitor these processes and restart them if anything crashes.

    • The routes block describes which requests should be sent to each web component. If we only had one web component, we could leave this out, but since we have two, this is vital.

    • For security reasons, everything in your production codebase is read-only. We want to override this for the tmp subfolder in our Mastodon project, so we use the writable_dirs block for that.

    • Nanobox reports everything in your application’s output to the logs page on your dashboard, but sometimes you also want to include the contents of log files. We use the log_watch block for that.

    • The network_dirs block lets us share some storage space between multiple components, and to persist the contents across deploys.

  • The following section defines a worker component. Most of its options are the same as those for the web components, but workers don’t have routes - they run entirely in the background. Instead, we include a cron block, which sets up a number of tasks to run at regular intervals. Note the two cron jobs which are commented out - you can use these to schedule open registration periods at regular intervals, which can be useful for preventing your instance from being overwhelmed all at once.

  • The last three blocks define data components; in this case one for Postgres, one for Redis, and the third for our shared persistent storage.

nanobox/nginx-local.conf
nanobox/nginx-web.conf.erb
nanobox/nginx-stream.conf.erb

These files are a synthesis of the example Nginx configurations for both Mastodon and Ruby on Rails with Nanobox. The first, nginx-local, is for local development; I’ll discuss more on how to use it later on. The other two (nginx-web and nginx-stream) are used by the production components web.web and web.stream, respectively, to proxy traffic between the Nanobox load balancer (more on that in a moment) and the Rails app or the Node.js websocket (again respectively). As you can see, nginx-local contains everything the other two do, but is set up to run both proxies in one environment, while the other two are designed to be isolated in their own separate containers. It also leaves out the Content Security Policy and other HTTPS-related settings, because local development doesn’t support HTTPS, yet. Again, more on that later.

.nanoignore

This file tells Nanobox to leave certain files and folders out when preparing to deploy. Its format is identical to the exclude file format used by rsync, and in fact Nanobox actually uses rsync under the hood to perform this step. Any file or folder listed here will not exist in production, unless your deploy process recreates them (as ours actually does with node_modules/, public/assets/, and vendor/bundle/).

.env.nanobox

Here we create a new .env template file for use with Nanobox. We set the values within to point to the environment variables that Nanobox will automatically create for us, making it so we don’t have to mess with them at all later on. A few values don’t come directly from Nanobox itself, though, so we configure these using sane defaults, and leave most of the rest as they are in the production sample. Note that we do have HTTPS disabled by default - this is because local development is done on your own system, and (as mentioned a moment ago) HTTPS isn’t actually (yet) supported by Nanobox in development mode. Don’t worry about changing this, though - as mentioned above, the boxfile.yml already includes a command that handles changing this for us when we deploy to production.

lib/tasks/db.rake

This file is adapted from the Nanobox Ruby on Rails guide. The only real difference, here, is that the task is named db:migrate:setup instead of db:setup_or_migrate, mostly to keep it more consistent with other, existing Rake tasks.

Deploying An Instance

With all these changes in place (which has been the case since version 1.4rc4), we can deploy an instance really easily. There’s a bit of setup involved, but you only have to do it once, and then your instance can be kept up to date with only two simple commands. Let’s walk through the first deploy, and then I’ll show you how simple the upgrade process is.

The first step is to create a new app in your Nanobox dashboard. If you don’t yet have an account, go ahead and register one, then download and install the Nanobox Desktop application, before you continue. There are other guides for actually creating a new app, so I won’t cover that here.

Once your new app exists, let’s set it up some, so that when we deploy, everything will Just Work™. We’ll start on the Config tab:

Config Tab

From here we can add the handful of environment variables Mastodon needs to run properly. We want to add these before we deploy so that the app is built properly from the moment it goes online - part of the deploy process relies on these variables being set correctly. First, every Mastodon app needs both RAILS_ENV and NODE_ENV set to production:

RAILS_ENV and NODE_ENV set

We also need to set the three secrets - PAPERCLIP_SECRET, OTP_SECRET, and SECRET_KEY_BASE - to random strings. You should use something like:

# run in directory where Mastodon repo is cloned
# using terminal, command prompt, Powershell, etc.
#
# NOTE: git bash is not currently supported by Docker,
#       so it won't work, here, either
#
nanobox run bundle exec rake secret

to generate these values, but in the screenshot below, I’ve used a dummy value instead.

NOTE: If you are migrating an existing instance, you'll want to use the values you already have in your old instance's .env or .env.production, here. When you import the database contents, later on, they'll be keyed to those existing values, so if you use different ones, everything will break.

These are the bare minimum required values to get Mastodon to work properly on your initial deploy:

All required environment variables set

Optionally, there are a few other values you might want to set as well. LOCAL_DOMAIN should be set to your own domain name if you have one you’d like to use instead of [app-name].nanoapp.io (the default value, which Nanobox provides for free with your app). Set SINGLE_USER_MODE to true if you want to deploy a single-user instance, instead of letting others sign up on it to create a larger community. You probably want to set the SMTP_* values as well, so that emails can be sent properly (the Mastodon .env example file mentions MailGun, which I’ve had lots of luck with personally, and SparkPost, which I’ve not tried, as having good free options for sending email). I recommend looking through the .env.nanobox file to see what other values are available, and what each does, so you can set any that feel appropriate. For this walkthrough, though, I won’t be setting any of these other values.

NOTE: Again, if you're migrating your instance, it's a good idea to copy over as many of the values from your .env.production file as you can. The only exceptions, here, are your REDIS_* and DB_* values, and DATABASE_URL if you have that set. Nanobox generates these values automatically to work within the new environment it will be creating for you in a few steps, so they need to be left as-is.

Once the environment variables are all set to your liking, we’ll proceed to the Network tab. The initial page here has your DNS settings. You can use these A and CNAME values to point your own domain name at your new app. You have to create an A record in your domain’s DNS records with the A value if you want your root domain (the one you paid for, without any subdomains) to load your Mastodon instance. Otherwise, if you’re pointing a subdomain at your Mastodon instance instead, you should create a CNAME record, since that won’t change even if (for some reason, such as scaling) your instance’s IP address does. Go ahead and set up your DNS records now, and double-check you set LOCAL_DOMAIN as described in the previous step. This is vital to do before the next step, since your app needs to respond to requests for your domain in order for the next step to work. (The process of actually setting your DNS records is beyond the scope of this article, as there are about as many DNS services as domain registrars, and each does things differently. I recommend CloudFlare, mostly because it has a number of additional benefits beyond simple DNS, but whatever you use is really up to you.)

If you’re OK using the default domain, or if you’ve already set up your DNS records appropriately, let’s proceed to the SSL/TLS Certificates page (still on the Network tab):

SSL/TLS Certificates Page

HTTPS support requires a certificate to securely identify your instance as actually being what it says it is. This is where you can create (or add, if you happen to already have one) such a certificate, and associate it with your Mastodon app. The instructions at the bottom of the page (as shown in the screenshot above) should be enough to get the gist of what to do, but let’s walk through it here anyway. First, click New SSL/TLS Certificate:

New Certificate Page

The information here is pretty straightforward, as you can see above. Remember to use the value of LOCAL_DOMAIN as your Domain, here, so that requests will be correctly secured. Feel free to repeat this process with [app-name].nanoapp.io as well, if you like, so that requests can come in there securely, too.

NOTE: If you get an error saying your domain isn't properly formatted, let us know! IANA added a whole bunch of new TLDs recently, and we may have missed a few by mistake.

Once you’ve filled out everything needed here, click Create Certificate to continue:

Certificate created

Now that you’ve entered your certificate data, it’s time to actually create the certificate bundle. Click New Bundle:

New Bundle Form

I personally recommend choosing LetsEncrypt, here, as it’s free, automatically trusted by most browsers, and Nanobox will automatically renew it for you when it’s about to expire. You might have reasons to choose one of the other options, though. For this walkthrough, I’m just going to use LetsEncrypt. Select your preference, then click Create Bundle:

Bundle generated

After some processing in the background (you should see a progress indicator at the top of the page), you’ll be presented with a screen very much like the one above. All we need to do now is click Activate on our new bundle:

Bundle activated

And that’s it. HTTPS is all set up and ready to go! Additionally, we’re done using the dashboard for the moment - you can close it, or go back to the Dash tab to watch the progress of the deploy once it leaves your desktop.

The next step is to ensure you have the latest (tagged) version of Mastodon available to deploy from. Clone the repo from GitHub, then checkout the latest tag:

git checkout $(git tag | tail -n 1)

Note that versions prior to 1.4rc4 don’t include the Nanobox files, so you need to be using at least that version to continue.

Now we’ll connect the local repo with your shiny new Nanobox app we just configured.

nanobox remote add [appname]

Replace [appname] with the actual name of your app.

Then follow the prompts to log in and connect your app. Once this is done, you’re ready to deploy!

nanobox deploy

Sit back while Nanobox handles the rest! (No, really, relax for a while. The initial deploy will probably take several minutes to complete, depending on the processing power of your local computer, your Internet connection speeds between your local computer and your app server, and the amount of processing power your app server has available.)

Final Setup

Once your new instance is deployed, you’ll want to do a couple of things to set it up the rest of the way. The first, of course, is to create your account. Start by connecting to your app’s main web component like so:

nanobox console web.web

Next, start a Rails console:

bundle exec rails c

Enter the following code, replacing values as needed to actually set up your own account:

account = Account.create!(username: '[username]')
user = User.create!(email: '[email@example.com]', password: '[password]', account: account)
user.confirm
account.save!
user.save!
exit

Now you need to set your new user as an admin, so you can properly manage your new instance. While still in the main web component’s shell, use the following command to upgrade your account’s access:

bundle exec rails mastodon:make_admin USERNAME=[username]

Remember to use your own username here.

Now we can go to the web interface and log in, click Preferences to open the settings section, Administration to access the admin section (you can configure your personal settings first, if you like, but they’re outside the scope of this article), and Site Settings to get to the instance configuration page. It’s a bit of work to get here, but this is where we will configure the public information about the instance, such as the instance name, contact info for the instance admin(s), instance descriptions (which usually contain short versions of the code of conduct and such), and whether or not registration is enabled (plus the message to display if not). Fill these out with whatever makes sense for your instance:

Instance Settings

Deploying Updates

So your instance is running smoothly, and a new version of Mastodon is tagged for release. Updating to it couldn’t be simpler. First you update the repo to pull in the release:

git fetch --all && git checkout [tag]

Then simply deploy again:

nanobox deploy

Everything else is automated. Nanobox will spin up new containers for each web and worker component, deploy the updated code to each, then transfer requests to the new containers before destroying the old ones. All of this happens without any interaction from you, and results in little to no downtime. Couldn’t be simpler, right?

Scaling

At some point, you’ll probably need to adjust how much your instance can handle. Nanobox makes scaling your app very simple from your dashboard. Let’s walk through scaling up the main web component, so you can see how this works.

Starting from your app’s Dash tab, choose App Components from below your server’s status block. Once your components are loaded and displayed, select Move from below your web.web component’s status block. It should look something like this:

Scaling modes for web.web

We’re gonna stick with the default, here, and set up a horizontal cluster of main web components, so just click Next:

Sizing options for web.web on AWS

Select a size for the new VMs (this will vary depending on what host you’re deploying to; this example shows AWS options), and a number of VMs to create (these will run in parallel, and the Nanobox load balancer will direct traffic to each in an intelligent manner automatically), and click Next again:

Scaling options review for web.web on AWS

Look over all the information here and make sure it’s all what you intended it to be. It should be, but it always helps to check when you’re going to be spending money based on what you’ve entered. If everything looks good, go ahead and click Submit. It’ll take a minute or two to set up the new server and spin up the web component on it, but once it’s done, your app will be scaled:

Newly scaled web.web, outside the App Components for aws.1

Nanobox In Development

I’ve mentioned several times, above, that you can use Nanobox in development, as well as production. Each time I promised I’d cover that later - well, now it’s time to deliver on that promise. Let’s take a look at how to make and test changes right on your desktop, before ever deploying to a server.

For this example, we’re going to adjust the theme of our instance to this one by @soft_chomps@mastodon.social. It’s a pretty simple example, but it seems a lot of instance admins want to tweak the theme almost as soon as their instance goes live, so it feels like a good one to provide.

So, the first step is to make sure you have the Mastodon repository cloned locally and the version you intend to deploy/work on checked out. For this example, you’ll want the latest tag (see above for a quick way to do this), but if you’re making changes to Mastodon itself, you’ll want to use master instead, since that’s where pull requests will be accepted and merged.

Note: There are some fixes to make this all work which have been merged into master, but not yet tagged, so until v1.4.2 is released, you'll have to rely on master, or apply these changes to v1.4.1 manually.

Once that’s in place, set up a local domain name alias:

nanobox dns add local mastodon.dev

You can replace mastodon.dev with your own domain alias if you like.

This will let us avoid using the IP address of the development container directly in the browser, which makes things a lot easier to work with. Then, to ensure Mastodon plays nice with our alias, run:

nanobox evar add local LOCAL_DOMAIN=mastodon.dev

While we’re here, let’s also set up the three secret keys Mastodon requires. I covered these in more detail in the production deployment section, above, but you’ll need to use the nanobox evar command to set each of these for development.

nanobox evar add local PAPERCLIP_SECRET=value OTP_SECRET=value SECRET_KEY_BASE=value

Replace each value with the appropriate secure key value.

Finally, we need to set up the local database:

nanobox run bundle exec rake db:migrate:setup

This last command will run Nanobox’s build task first (which actually creates the container and installs all the required software to make your app work) if you haven’t already run it by deploying or doing other development work since the last change to the boxfile.yml or one of the other build_triggers. So it might take a while, especially for the first build.

Next, we’ll use three separate consoles: one each to start the streaming API, Rails, and WebPack. Here are the commands to run in each console:

nanobox run 'bash -c "envsubst < .env.nanobox > .env ; yarn run start"'
nanobox run bundle exec puma -C /app/config/puma.rb
nanobox run bin/webpack-dev-server

With the servers running, go ahead and pull up your browser and go to http://mastodon.dev:3000/ (again replacing mastodon.dev with your actual domain alias). You should be presented with the signup page:

Development Signup Page

Click Log in, and enter admin@mastodon.dev as the Email address, and mastodonadmin as the password. This account is created automatically when you set up the database, but only in development - this isn't an account that will leave your production instance(s) vulnerable. Click "Log In" to get to the Getting Started page, complete with the new user intro:

New User Intro

Go ahead and skip the intro if you like. Now that we can see our site's current theming properly, it's time to go make adjustments. Mastodon no longer has a reserved location for adding theme customizations, so we'll need to tell WebPack where to look. Create a new file in app/javascript/packs called custom.js, and place the following code inside:

// import customization styles
require.context('../styles/', false, /custom.*\.scss$/);

Now we can create the actual theme file, in app/javascript/styles, called custom-white.scss, and paste the new theme's CSS inside it. To get the new CSS, go to the theme page on Stylish, find and click the Show CSS Code button, then copy everything inside the site selector (find the first {, and copy everything after that except the very last }). Once you've pasted that into place, save the file. The WebPack process should notice the change and recompile your assets, then tell your browser to reload so it will get the updates:

Your New, White Theme

Feel free, if you're comfortable working with CSS, to play around with the settings until you get everything looking the way you like. Every time you save the CSS file, your browser should refresh to show the new version. Once you're happy with what you see (or if you aren't comfortable with CSS), you can easily deploy your new theme by following the production steps above (just nanobox deploy if you've already deployed from this computer before).

Obviously you can do much more complex edits to everything in the Mastodon code base if you like. Hopefully this walkthrough has given you a good idea of how to go about doing it.

Backups

I’ve saved this topic for last, because it’s a little bit advanced, as Nanobox tasks go. Any time you’re storing data, especially data generated by and for other people, you want to ensure it won’t all be lost if something unexpected happens to your app’s servers. That means making regular backups of your database and persistent storage. But Nanobox doesn’t automate this for you (yet - there have been discussions about how this might be automated in a sane way, so it might be added in the future), so you have to set up your own backups. Let’s walk through one way to do this - which is the officially recommended approach.

First, ensure you have the Postgres client tools installed locally on your computer (or on a computer dedicated to making and/or storing your backups). We’ll be using pg_dump to run the actual backup process, and you’ll want pg_restore to recover in case of problems. If you’re using a dedicated backup system, you’ll also need Nanobox installed, and a local copy of your application (or at least the boxfile.yml), so you can add your app as a remote (see above for how to do that).

To get your production database password, run the following in your project directory:

nanobox evar ls | grep DATA_DB_PASS

Then establish a secure tunnel to your production Postgres server:

nanobox tunnel data.db

In another window or tab, run:

pg_dump -U nanobox -W -h localhost -p 5432 -Fc -O gonano > [filename].bak

Enter the password you just looked up. This will probably take a while, as your entire database is transferred to a local file across the tunnel, but once it’s done, you can close the tunnel by switching back to the console it’s running in and using Ctrl-C to end the process.

In the unlikely case you have to restore from your backup, it’s essentially the same process. Look up the current DB password, establish the tunnel, and then use pg_restore -U nanobox -W -h localhost -p 5432 -Fc -O [filename].bak to transfer the data back to your server, before closing the tunnel once the restoration is done.

There are likely other methods you could use, but this should be enough information to get started. Other approaches are a bit outside the scope of this article, which is already pretty lengthy.

Other Stuff

If you have any further questions, feel free to join us on the Nanobox Slack server, where someone (usually either myself or one of the staff) will be more than happy to help you out. Hopefully, though, this article should be enough to get you up and running with Mastodon, and keep you going for quite some time. I strongly feel that Mastodon and Nanobox are both tools of the future, and that they just make sense together. I hope you feel that, too!

Posted in Nanobox, Deployment, Ruby, Rails, DevOps, Development, React.js, Cloud, JavaScript, Automated Deployment, OpenSource, Mastodon

Daniel Hunsaker

Daniel Hunsaker

Author, Father, Programmer, Nut. Dan contributes to so many projects he sometimes gets them mixed up. He'll happily help you out on the Nanobox Slack server when the staff are offline.

@sendoshin Idaho, USA
Read More