First Impressions of Self-hosted Forem on AWS

If you're looking to run your own forem here are some of my personal thoughts.
I'm only covering the technical aspects in this first impression.

TL;DR: It was easy, and is a great fit if you're on a budget and are comfortable with everything running on a single machine (no different than running a Wordpress)

How hard was it to deploy?

I chose to deploy to AWS because I'm an AWS Community Hero and I'm launching a Forem for AWS.

• I think you can get it running in 15 mins.
• It took me 2 hours because of Python3 and OpenSSL issues on mac OS which is not related to Forem code-base, just my own problems.

@coffeecraftcode was there to help troubleshoot problems with me to get it running.

How much does it cost to run?

Designed for cost-effectiveness

Forem on AWS is configured to be deployed to a single EC2 instance and the default size is t3a.small, so that will run you about $15 USD per month.

I don't generally recommend running a Ruby on Rails apps anything less than 4 GB which would be ~$30 USD per month on a t2.medium or t3.medium.

The Self-hosted Forem deploys a containerized monolith meaning everything is deployed old-school to the same server. That includes the app, the database, background jobs, redis, and few other things.

This is the most cost effective model, and this is ideal if you are starting small.

You will be limited to vertical scaling (increasing the size of the instance) but this is going to work well for most communities.

Hidden costs

Forem has optional third-party cloud services you can integrate by passing along API keys for:

• Honeybadger (bug tracking)
• DataDog (APM, Infrastructure Monitoring)
• Cloudinary (Image and Video storage and delivery)
• Recaptcha (Are you bot? This is by Google, I think they have generous free-tier)
• Pusher (real-time push updates for websockets, probably used for chat)
• SendGrid (transactional emails)
• MailChimp (If you want a newsletter)

Some of these have free tiers. I have used all of the services at some-point in my career so I would say if you are paying its about $100 USD additional. Bringing our total operational cost to $130 USD per month.

Is it easy to say swap SendGrid for SES or HoneyBadger for Rollbar? I'd say no, not unless you want to open the code, and you risk deviating from the original codebase.

I think I would like to see better plug-and-play for cloud service provider (CPS) native services because they have more generous free-tiers and honestly lots of startups get ahold of credits from their CSPs and allows them to operate a year for free and non-profits can perpetually obtain free credits from CSPs.

I didn't see Fastly or ElasticSearch in the Ansible configuration files which were key services part of the codebase prior, so I imagine those have been abstracted away. Possibly ElasticSearch is now have been replaced with FullText search for the community edition of Forem.

The proper cost

If you are planning to build a community at scale, and it was deployed to AWS, these are the services you would be using to be positioned for growth:

• Elastic Load Balancer (ELB)- ~$15 USD / month
• Relational Database Service (RDS) - ~$15 USD / month
• ELastiCache - ~$15 USD / month
• Elastic Compute Cloud (EC2) Instance ~$30 USD / USD month

There would be hidden costs like EBS snapshots and manual RDS backup snapshots but overall to be position for scale would bring our grand total to $75 USD per month including the optional third party service its $275 USD per month.

Architecture

Here is just some of my thoughts about the architecture.

Configuration Management vs Infrastructure as Code

The way self-hosted Forem works is the server is provisioned using Ansible.
I imagine the design choice was to use a tool that could be utilized across multiple hosting providers and basically anything that is running a Virtual Machine.

The small problem with using Ansible is that means we do not get Infrastructure as Code. So its up to you keep track of your resources (which lets be fair here is just a single server)

I think what I would have liked to have seen was CloudFormation, though given the cloud agnostic nature I would say probably better Terraform. Cloud-Init could have been used for for the configuration of a linux VM server since it is generally available on all hosting providers. (though its not fun writing Cloud-Init files so who can blame them really)

Considering a VM Image in the marketplace

I think Forem may want to consider offering Virtual Machine (VM) image via the AWS Marketplace (GCP and Azure also has Marketplaces) because as a user I don't have to use any kind of configuration management.

Forem could even charge a subscription via Marketplace images, which I think I would pay for if it means they will keep the VM image harden and up to date.

If you have ever launched a Bitnami Wordpress from the AWS Marketplace you know the ease of that experience.

Upgrading

The Ansible playbook appears to only be setup for provisioning, how you would deploy an updated codebase, uncertain. Maybe future playbooks are coming.

Since all the data resides on the server, you can't just provision a new EC2 instance, you'd have to first backup your data.

Backups

Since the postgres database is not running on a managed database service you will need to devise your own backup strategy. So you would want to do a pg_dump and then copy the dumped data to a cloud storage service. You could use a cron job but if you're on AWS you could write uses Systems Manager to write a Run Command to run on a schedule. Run commands are just remote bash scripts.

The database (and the server data) would reside on the Elastic Block Storage (EBS) volume attached to the EC2 so you can alternatively just snapshot the EBS volume on a schedule using EventBridge (also known as CloudWatch Events)

If you wanted to use a managed service you'll have to SSH (or use Sessions Manager) to connect to the instance. You'll need to update the config/credentials.yml.en via rails credentials:edit and provide a DATABASE_URL postgres connection string and restart the forem-rails service.

SystemD Services

The Ansible playbook creates multiples systemd services:

• docker-to-podman-sock.service
• fcos-python-3.service
• forem-container.service
• forem-pod.service
• forem-imgproxy.service
• forem-postgresql.service
• forem-redis.service
• forem.service
• forem-rails.service
• forem-worker.service
• forem-openresty.service
• forem-traefik.service
• rpm-ostree-install.service

So if you need to restart something you can do things like: sudo service forem restart. Could also be systemctl I get my systemd, sysvinit and upstart mixed up all the time.

Scaling

We only have vertical scaling. (make the instance bigger) and you should want to backup your database first via cronjob to cloud storage or take an EBS snapshot.

You could horizontally scale (add more same size servers) if you create a Launch Template by creating an Amazon Machine Image (AMI) from your running EC2 instance and then use that Launch Template in an Auto Scaling Group, however this is not going to work if you continue to use the database on the EC2 instance because you'd end up with two databases. You would have to use a managed database service eg. RDS.

Platform as a Service

I think that Forem would have been better suited with the self-hosted on Platform as a Service (PaaS) offerings than on the IaaS VM offerings eg:

• AWS Elastic Beanstalk
• Azure App Services
• GCP App Engine
• Digital Ocean App Platform
• Heroku

Since all of these have natural scaling paths, easy backup strategies, and easier to connect to native managed services.

Final Thoughts

I think the self-hosted solution is a success.

You'll get alot out of a monolith on a single VM. Before cloud services providers, I had built a Ruby on Rails startup that had 100K daily active users and it ran on a single machine.

By design it is not out-of-the-box to positioned for scale and while the Forem team could remedy that, I think as a consumer I'd much prefer a SaaS offering where when I need scale, I can just bring my self-hosted data over to a fully-managed Forem service.