Dependency failed for Forem OpenResty and Worker Service

#selfhost #help

While running this command on my DigitalOcean install:

$ sudo systemctl restart forem-traefik.service

The command is stalling.

I've checked the forem services:

$ systemctl list-units forem* --all
  UNIT                     LOAD   ACTIVE     SUB          JOB   DESCRIPTION             
  forem-container.service  loaded inactive   dead               Forem container service
  forem-imgproxy.service   loaded active     running            Forem Imgproxy Service
  forem-openresty.service  loaded inactive   dead         start Forem OpenResty Service
  forem-pod.service        loaded active     running            Forem pod service
  forem-postgresql.service loaded active     running            Forem Postgresql Service
  forem-rails.service      loaded activating auto-restart start Forem Rails Service
  forem-redis.service      loaded active     running            Forem Redis Service
  forem-traefik.service    loaded active     running            Forem Traefik Service
  forem-worker.service     loaded inactive   dead         start Forem Worker Service
  forem.service            loaded active     exited             Forem Service

For Openresty:

$ sudo journalctl -u forem-openresty.service -n 200 --no-pager
... (all the same)
systemd[1]: Dependency failed for Forem OpenResty Service.
systemd[1]: forem-openresty.service: Job forem-openresty.service/start failed with result 'dependency'.
...

For Worker:

$ sudo journalctl -u forem-worker.service -n 200 --no-pager
systemd[1]: Dependency failed for Forem Worker Service.
systemd[1]: forem-worker.service: Job forem-worker.service/start failed with result 'dependency'.

I tried to use sudo systemctl start forem-container.service.
But it doesn't help.

Please help me.

Top comments (12)

andy george • Aug 31 '21

Hey @tien thanks for trying out Selfhost!

Can you show us some logs from forem-container.service? That's what handles retrieving the Forem container, and if there are issues pulling that container, nothing will really work.

Tien • Sep 1 '21 • Edited

Hi, thanks for your fast comment.

This is logs from forem-container.service.

systemd[1]: Starting Forem container service...
forem-container[763]: Using Forem container tag (quay.io/forem/forem:latest) from /opt/forem/version
forem-container[763]: The remote container quay.io/forem/forem:latest repository or tag does not exist. Please provide a valid container repository and tag.
forem-container[763]: Example: foremimg quay.io/forem/forem:latest
systemd[1]: forem-container.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: forem-container.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Forem container service.
systemd[1]: forem-container.service: Scheduled restart job, restart counter is at 1.
systemd[1]: Stopped Forem container service.
systemd[1]: Starting Forem container service...
forem-container[2876]: Using Forem container tag (quay.io/forem/forem:latest) from /opt/forem/version
forem-container[2876]: Pulling quay.io/forem/forem:latest
forem-container[2985]: 77037df6f09452964c898e092b25e000d764101eaec2e50644e06199a7da62d7
podman[2985]: 2021-09-01 00:22:54.897617957 +0000 UTC m=+0.064896844 image pull  quay.io/forem/forem:latest
forem-container[2876]: The Forem container SHA256 has not changed.
forem-container[2876]: quay.io/forem/forem:latest (77037df6f094)
forem-container[2876]: sha256:61127e9b3162f7750713f5f3e628898e740858fa8e9980acd24dd832eaa53cfd
forem-container[2876]: The current tag will not be updated!
forem-container[2876]: The current tag localhost/forem/forem:current is pointed at quay.io/forem/forem:latest (77037df6f094)
systemd[1]: forem-container.service: Deactivated successfully.
systemd[1]: Finished Forem container service.
...
systemd[1]: Starting Forem container service...
forem-container[3657]: Using Forem container tag (quay.io/forem/forem:latest) from /opt/forem/version
forem-container[3657]: Pulling quay.io/forem/forem:latest
forem-container[3666]: 77037df6f09452964c898e092b25e000d764101eaec2e50644e06199a7da62d7
podman[3666]: 2021-09-01 00:24:55.859277169 +0000 UTC m=+0.126298997 image pull  quay.io/forem/forem:latest
forem-container[3657]: The Forem container SHA256 has not changed.
forem-container[3657]: quay.io/forem/forem:latest (77037df6f094)
forem-container[3657]: sha256:61127e9b3162f7750713f5f3e628898e740858fa8e9980acd24dd832eaa53cfd
forem-container[3657]: The current tag will not be updated!
forem-container[3657]: The current tag localhost/forem/forem:current is pointed at quay.io/forem/forem:latest (77037df6f094)
systemd[1]: forem-container.service: Deactivated successfully.
systemd[1]: Finished Forem container service.

andy george • Sep 1 '21

Ok, that's fine; looks like it's correctly pulling down the quay.io/forem/forem:latest container.

Can you show your logs for forem-rails.service as well? Looks like it's restarting, wonder if there are any obvious errors in there.

Tien • Sep 1 '21 • Edited

Hi, this is my logs for forem-rails.service :)

systemd[1]: Starting Forem Rails Service...
podman[12001]: 2021-09-01 02:32:05.760964154 +0000 UTC m=+0.167483134 image pull  localhost/forem/forem:current
podman[12001]: 2021-09-01 02:32:05.879917716 +0000 UTC m=+0.286436649 volume create 8fb2b040495a339a43ac0b59d79e683398582df2ced821568956b6937cfc1ee4
podman[12001]: 2021-09-01 02:32:05.897400108 +0000 UTC m=+0.303919070 container create c59ca07bf1afb69e3e08da3a0c8c2afd60bbdd81f79727469e19fd1f3b7eec84 (image=localhost/forem/forem:current, name=forem-rails-bootstrap, maintainer=Forem Systems Engineering <systems@forem.com>, name=fedora, vendor=Fedora Project, version=34, PODMAN_SYSTEMD_UNIT=forem-rails.service, io.buildah.version=1.21.0, license=MIT)
podman[12001]: 2021-09-01 02:32:07.507108275 +0000 UTC m=+1.913627199 container init c59ca07bf1afb69e3e08da3a0c8c2afd60bbdd81f79727469e19fd1f3b7eec84 (image=localhost/forem/forem:current, name=forem-rails-bootstrap, maintainer=Forem Systems Engineering <systems@forem.com>, name=fedora, vendor=Fedora Project, version=34, PODMAN_SYSTEMD_UNIT=forem-rails.service, io.buildah.version=1.21.0, license=MIT)
podman[12001]: 2021-09-01 02:32:07.53855414 +0000 UTC m=+1.945073066 container start c59ca07bf1afb69e3e08da3a0c8c2afd60bbdd81f79727469e19fd1f3b7eec84 (image=localhost/forem/forem:current, name=forem-rails-bootstrap, vendor=Fedora Project, version=34, PODMAN_SYSTEMD_UNIT=forem-rails.service, io.buildah.version=1.21.0, license=MIT, maintainer=Forem Systems Engineering <systems@forem.com>, name=fedora)
podman[12001]: 2021-09-01 02:32:07.539144937 +0000 UTC m=+1.945663867 container attach c59ca07bf1afb69e3e08da3a0c8c2afd60bbdd81f79727469e19fd1f3b7eec84 (image=localhost/forem/forem:current, name=forem-rails-bootstrap, name=fedora, vendor=Fedora Project, version=34, PODMAN_SYSTEMD_UNIT=forem-rails.service, io.buildah.version=1.21.0, license=MIT, maintainer=Forem Systems Engineering <systems@forem.com>)
conmon[12049]: Running rake app_initializer:setup...
forem-rails[12001]: Running rake app_initializer:setup...
conmon[12049]: I, [2021-09-01T02:32:15.968700 #5]  INFO -- honeybadger: ** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.9.0 framework=rails level=1 pid=5
forem-rails[12001]: I, [2021-09-01T02:32:15.968700 #5]  INFO -- honeybadger: ** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.9.0 framework=rails level=1 pid=5
conmon[12049]: 
forem-rails[12001]: == Preparing database ==
conmon[12049]: == Preparing database ==
conmon[12049]: I, [2021-09-01T02:32:27.025697 #7]  INFO -- honeybadger: ** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.9.0 framework=rails level=1 pid=7
forem-rails[12001]: I, [2021-09-01T02:32:27.025697 #7]  INFO -- honeybadger: ** [Honeybadger] Initializing Honeybadger Error Tracker for Ruby. Ship it! version=4.9.0 framework=rails level=1 pid=7
conmon[12049]: I, [2021-09-01T02:32:29.615228 #7]  INFO -- : Migrating to DropUserOptionalFields (20200731033002)
forem-rails[12001]: I, [2021-09-01T02:32:29.615228 #7]  INFO -- : Migrating to DropUserOptionalFields (20200731033002)
conmon[12049]: == 20200731033002 DropUserOptionalFields: migrating ===========================
forem-rails[12001]: == 20200731033002 DropUserOptionalFields: migrating ===========================
conmon[12049]: [strong_migrations] DANGER: No lock timeout set
forem-rails[12001]: [strong_migrations] DANGER: No lock timeout set
conmon[12049]: -- drop_table(:user_optional_fields)
forem-rails[12001]: -- drop_table(:user_optional_fields)
conmon[12049]:    -> 0.0068s
conmon[12049]: -- index_exists?(:user_optional_fields, [:label, :user_id], {:unique=>true})
forem-rails[12001]:    -> 0.0068s
forem-rails[12001]: -- index_exists?(:user_optional_fields, [:label, :user_id], {:unique=>true})
conmon[12049]: E, [2021-09-01T02:32:29.685878 #7] ERROR -- honeybadger: ** [Honeybadger] Unable to send error report: API key is missing. id=5f7ec137-ed25-4310-919c-0141274e7844 level=3 pid=7
forem-rails[12001]: E, [2021-09-01T02:32:29.685878 #7] ERROR -- honeybadger: ** [Honeybadger] Unable to send error report: API key is missing. id=5f7ec137-ed25-4310-919c-0141274e7844 level=3 pid=7
conmon[12049]: rails aborted!
conmon[12049]: StandardError: An error has occurred, this and all later migrations canceled:
conmon[12049]: 
conmon[12049]: wrong number of arguments (given 3, expected 2)
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:99:in `index_exists?'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:929:in `block in method_missing'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:897:in `block in say_with_time'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:897:in `say_with_time'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:918:in `method_missing'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/hypershield-0.2.2/lib/hypershield/migration.rb:11:in `method_missing'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/strong_migrations-0.7.8/lib/strong_migrations/migration.rb:13:in `block in method_missing'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/strong_migrations-0.7.8/lib/strong_migrations/checker.rb:339:in `perform'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/strong_migrations-0.7.8/lib/strong_migrations/migration.rb:12:in `method_missing'
conmon[12049]: /opt/apps/forem/db/migrate/20200731033002_drop_user_optional_fields.rb:7:in `up'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:870:in `public_send'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:870:in `exec_migration'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:850:in `block in migrate'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:849:in `migrate'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/strong_migrations-0.7.8/lib/strong_migrations/migration.rb:5:in `migrate'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1037:in `migrate'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:209:in `transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1380:in `ddl_transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1302:in `each'
forem-rails[12001]: rails aborted!
forem-rails[12001]: StandardError: An error has occurred, this and all later migrations canceled:
forem-rails[12001]: wrong number of arguments (given 3, expected 2)
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:99:in `index_exists?'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:929:in `block in method_missing'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:897:in `block in say_with_time'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:897:in `say_with_time'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:918:in `method_missing'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/hypershield-0.2.2/lib/hypershield/migration.rb:11:in `method_missing'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/strong_migrations-0.7.8/lib/strong_migrations/migration.rb:13:in `block in method_missing'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/strong_migrations-0.7.8/lib/strong_migrations/checker.rb:339:in `perform'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/strong_migrations-0.7.8/lib/strong_migrations/migration.rb:12:in `method_missing'
forem-rails[12001]: /opt/apps/forem/db/migrate/20200731033002_drop_user_optional_fields.rb:7:in `up'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:870:in `public_send'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:870:in `exec_migration'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:851:in `block (2 levels) in migrate'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:850:in `block in migrate'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1302:in `migrate_without_lock'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:849:in `migrate'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/strong_migrations-0.7.8/lib/strong_migrations/migration.rb:5:in `migrate'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1037:in `migrate'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1329:in `block in execute_migration_in_transaction'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1380:in `block in ddl_transaction'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1251:in `block in migrate'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:209:in `transaction'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1380:in `ddl_transaction'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1328:in `execute_migration_in_transaction'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1302:in `each'
...
...
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:363:in `block (3 levels) in <main>'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:359:in `each'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/railties/databases.rake:359:in `block (2 levels) in <main>'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/honeycomb-beeline-2.6.0/lib/honeycomb/integrations/rake.rb:21:in `block in execute'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/honeycomb-beeline-2.6.0/lib/honeycomb/client.rb:62:in `start_span'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/honeycomb-beeline-2.6.0/lib/honeycomb/integrations/rake.rb:16:in `execute'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1416:in `block in with_advisory_lock_connection'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/command.rb:50:in `invoke'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/commands.rb:18:in `<main>'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
forem-rails[12001]: Caused by:
forem-rails[12001]: ArgumentError: wrong number of arguments (given 3, expected 2)
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:462:in `with_connection'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:99:in `index_exists?'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:929:in `block in method_missing'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:897:in `block in say_with_time'
...
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1086:in `up'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/command.rb:50:in `invoke'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/railties-6.1.4.1/lib/rails/commands.rb:18:in `<main>'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `require'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:23:in `block in require_with_bootsnap_lfi'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/loaded_features_index.rb:92:in `register'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:22:in `require_with_bootsnap_lfi'
forem-rails[12001]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/bootsnap-1.8.1/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:31:in `require'
forem-rails[12001]: Tasks: TOP => db:prepare
forem-rails[12001]: (See full trace by running task with --trace)
conmon[12049]: /opt/apps/forem/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:1061:in `migrate'
...
conmon[12049]: Tasks: TOP => db:prepare
conmon[12049]: (See full trace by running task with --trace)
podman[12001]: 2021-09-01 02:32:29.839959818 +0000 UTC m=+24.246478789 container died c59ca07bf1afb69e3e08da3a0c8c2afd60bbdd81f79727469e19fd1f3b7eec84 (image=localhost/forem/forem:current, name=forem-rails-bootstrap)
podman[12001]: 2021-09-01 02:32:30.078505663 +0000 UTC m=+24.485024597 container remove c59ca07bf1afb69e3e08da3a0c8c2afd60bbdd81f79727469e19fd1f3b7eec84 (image=localhost/forem/forem:current, name=forem-rails-bootstrap, version=34, PODMAN_SYSTEMD_UNIT=forem-rails.service, io.buildah.version=1.21.0, license=MIT, maintainer=Forem Systems Engineering <systems@forem.com>, name=fedora, vendor=Fedora Project)
podman[12001]: 2021-09-01 02:32:30.176020232 +0000 UTC m=+24.582539290 volume remove 8fb2b040495a339a43ac0b59d79e683398582df2ced821568956b6937cfc1ee4
systemd[1]: forem-rails.service: Control process exited, code=exited, status=1/FAILURE
systemd[1]: forem-rails.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Forem Rails Service.
systemd[1]: forem-rails.service: Scheduled restart job, restart counter is at 9.
systemd[1]: Stopped Forem Rails Service.
systemd[1]: Starting Forem Rails Service...

andy george • Sep 1 '21

Not sure what's going on there - can you reboot your droplet and see if you're still getting errors on forem-rails.service? Thanks!

Tien • Sep 2 '21

Thanks! But unfortunately it still doesn't work :(.

derekenos • Sep 2 '21

I recently ran into this same issue when building the Docker config at github/forem/forem.

This is not very helpful, but my temporary "fix" was to comment out the offending index_exists? invocations in that particular migration and then everything was ok..ish:

$ git diff db/migrate/20200731033002_drop_user_optional_fields.rb
diff --git a/db/migrate/20200731033002_drop_user_optional_fields.rb b/db/migrate/20200731033002_drop_user_optional_fields.rb
index c66dcd6..8928fad 100644
--- a/db/migrate/20200731033002_drop_user_optional_fields.rb
+++ b/db/migrate/20200731033002_drop_user_optional_fields.rb
@@ -4,7 +4,7 @@ class DropUserOptionalFields < ActiveRecord::Migration[6.0]
   def up
     drop_table :user_optional_fields

-    remove_index(*INDEX) if index_exists?(*INDEX)
+    # remove_index(*INDEX) if index_exists?(*INDEX)
   end

   def down
@@ -16,7 +16,6 @@ def down
       t.timestamps
     end

-    add_index(*INDEX) unless index_exists?(*INDEX)
+    # add_index(*INDEX) unless index_exists?(*INDEX)
   end
 end
-